Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Fix/tracing across workers (#860)

Louis Choquel258e324130d2
diff --git a/.claude/skills/temporal-e2e-validate/SKILL.md b/.claude/skills/temporal-e2e-validate/SKILL.md
index ac5bdb835..fd1e69242 100644
--- a/.claude/skills/temporal-e2e-validate/SKILL.md
+++ b/.claude/skills/temporal-e2e-validate/SKILL.md
@@ -25,6 +25,9 @@ allowed-tools:
- Bash(open *)
- Bash(cat *)
- Bash(.venv/bin/python *)
+ - Bash(grep *)
+ - Bash(sort *)
+ - Bash(head *)
---
# Temporal E2E Validation Suite
@@ -532,6 +535,69 @@ wait $PID_BETA && echo "Beta: PASS" || echo "Beta: FAIL"
After this, tell the user both results.
+### Step 5b: Tier 8 — Cross-worker usage emission (writer_id fallback)
+
+This validates that `UsageReportEvent`s emitted from inference activities running on
+the runner process land in the same NDJSON partition as the router-side trace events,
+stamped with a runner-process `writer_id` (`act_{pid}_{uuid8}`). It's the e2e
+counterpart to the Phase 4 integration test
+`tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py`.
+
+**Important — dry-run does NOT exercise this path.** In dry-run mode, `PipeLLM`
+(and friends) instantiate `ContentGeneratorDry()` directly inside the workflow
+body — on the router process — and never dispatch `act_llm_gen_text` to the
+runner. So a vanilla `pipelex run bundle --temporal --dry-run` against
+router+runner workers will emit ALL `usage_report` events with
+`writer_id="primary"`, never `act_*`. To deterministically observe runner-side
+fallback in dry-run, run the Phase 4 integration test, which substitutes the
+inference activity with a wrapper that synthesizes a real `LLMJob` server-side:
+
+```bash
+.venv/bin/pytest -x -v \
+ tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py \
+ -m temporal --temporal-server local 2>&1
+```
+
+Expected: both test cases pass —
+`test_runner_usage_event_lands_in_same_ndjson_dir` (lands in same dir, with an
+`__w_act_*` file) and `test_no_double_emit_in_split_worker_pool` (no
+double-counting between fast path and fallback).
+
+**To observe `act_*` writer files via the CLI**, run live mode (real LLM call —
+costs money) so `act_llm_gen_text` is actually dispatched to the runner:
+
+```bash
+.venv/bin/pipelex run bundle \
+ tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds \
+ --pipe native_text_sequence \
+ --temporal --no-logo --graph
+```
+
+After completion, list the trace files for the run:
+
+```bash
+RUN_ID=$(ls -t .pipelex/traces/ | head -1)
+ls -la .pipelex/traces/$RUN_ID/
+grep -l '"event_kind":"usage_report"' .pipelex/traces/$RUN_ID/*.ndjson
+grep -hoE '"writer_id":"[^"]+"' .pipelex/traces/$RUN_ID/*.ndjson | sort -u
+```
+
+Expect:
+
+- At least one `wf_*__w_act_{pid}_{uuid}.ndjson` file alongside the
+ router-side `wf_*.ndjson` files.
+- The runner-side file contains `usage_report` events with
+ `writer_id` starting `act_`.
+- Router-side files contain `pipe_start` / `pipe_end_success` events with
+ `writer_id="primary"`.
+
+If only `writer_id="primary"` appears, the inference activity ran in the
+router process — check `worker_config.inference_task_queue` and confirm both
+workers are running with the latest code (restart them if they predate the
+Phase 2 runner-side fallback commit).
+
+---
+
### Step 6: StoragePayloadCodec tests — does the codec work end-to-end?
These tests validate Phase 5: large payloads are transparently offloaded to external
@@ -658,6 +724,7 @@ ls results/*/reactflow.html
| Tier 5: Image flow | Generated image flows as input to next pipe step | PASS/FAIL | path | yes/no |
| Tier 6: Codec transparency | Existing pipelines work unchanged with codec enabled | PASS/FAIL | path | — |
| Tier 7: Large payload | Multi-step pipeline with codec stress test | PASS/FAIL | path | — |
+| Tier 8: Cross-worker usage | Runner-side `UsageReportEvent` lands in same NDJSON dir with `act_*` writer_id (live mode or integration test) | PASS/FAIL | — | — |
| Concept isolation (alpha) | Conflicting `Result` concepts don't cross-contaminate | PASS/FAIL | path | — |
| Concept isolation (beta) | (same test, other side) | PASS/FAIL | path | — |
| Pipe isolation (alpha) | Conflicting `shared_step` pipes use correct version | PASS/FAIL | path | — |
diff --git a/.gitignore b/.gitignore
index 36d6030fe..c9fe49d82 100644
--- a/.gitignore
+++ b/.gitignore
@@ -85,4 +85,5 @@ package-lock.json
# Claude plans
.claude/plans/
+.claude/scheduled_tasks.lock
.gstack/
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 230acb044..177ba5ec6 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -19,6 +19,7 @@
- **`StoragePayloadCodec`**: offloads Temporal payloads larger than a configurable threshold (default 1 MiB) to external storage (S3 or local) and replaces them with lightweight URI references. Configurable via `[temporal.payload_codec_config]`. New `pipelex codec serve` CLI launches the codec server consumed by the Temporal Web UI for payload inspection.
- **Schema-based dynamic model construction (`pipelex/cogt/content_generation/schema_to_model_factory.py`)**: rebuilds dynamic Pydantic models on a Temporal worker from the JSON schema embedded in `ObjectAssignment`, instead of relying on cross-process class identity. Cache bounded by an LRU (`_SCHEMA_CACHE_MAX_SIZE = 1024`); codegen serialized through a file lock to kill thundering-herd duplication on long-running workers receiving many distinct schemas. Adds `datamodel-code-generator>=0.55.0` as a runtime dependency.
- **Worker scopes**: `pipelex worker --scope ...` lets a worker subscribe to a subset of activities/workflows so deployments can specialize workers (e.g. inference workers vs orchestration workers).
+- **`WorkerConfig.inference_task_queue`**: optional task-queue override for `act_llm_gen_text` so router and inference activities can be deployed on separate worker pools.
- **Pre-generated `pipeline_run_id` support**: `pipeline_run_setup()`, `PipelineFactory`, and `PipelineManager` accept an optional `pipeline_run_id` parameter for external run-record creation.
- **`workflow_id` in start response**: `PipelexPipelineStartResponse` includes `workflow_id`.
- **Temporal `callbacks` passthrough**: `WorkflowExecutor.start_workflow()` accepts optional Temporal `Callback` objects.
@@ -27,7 +28,12 @@
- **`EventLogProtocol` with pluggable backends**: `NdjsonEventLog`, `DynamoDBEventLog`, `InMemoryEventLog`, `BufferingEventLog`. Emits `TraceEvent`s as pipes start/finish across processes. Configured via `[pipelex.tracing_config]` (`backend = "ndjson" | "dynamodb"`). Tracing is **enabled by default** and decoupled from Temporal — single-process runs also produce a trace.
- **`pipelex[dynamodb]` extra**: `pip install "pipelex[dynamodb]"` for the DynamoDB event log backend.
-- **`writer_id` on `TraceEvent`**: every event records which writer (worker / process) produced it, propagated through every backend. Enables proper cross-worker dedup and per-writer aggregation.
+- **`writer_id` on `TraceEvent`** (default `"primary"`): every event records which writer (worker / process) produced it, propagated through every backend. Standalone activity worker processes use a distinct `writer_id` so they can emit into the same backend partition as workflow workers without colliding on `(workflow_id, sequence)`. Stamped at construction time (no copy-on-emit). Enables cross-worker dedup and per-writer aggregation.
+- **NDJSON multi-writer key scheme**: dedup key is `(workflow_id, writer_id, type, sequence)`; sort key is `(workflow_id, sequence, writer_id)` — sequence is primary so the runner's `UsageReportEvent` does not sort before the router's `PipeStartEvent`. The file-handle cache key includes `writer_id`, so two writers for the same `(pipeline_run_id, workflow_id)` cannot share a stale handle.
+- **DynamoDB sort key format**: `EVENT#{workflow_id}#{writer_id}#{sequence:010d}`.
+- **Cross-worker `UsageReportEvent` emission from runner processes**: runner processes that never execute `WfPipeRouter.run()` (so `_event_log_contexts` has no entry for the workflow's `lookup_key`) emit usage events through a process-local event log built from `tracing_config`. The fallback log is cached per process via a `threading.Lock`-guarded factory in `pipelex.tracing.activity_event_log` and keyed by a stable per-process `writer_id = f"act_{pid}_{uuid8}"`. A single one-shot warning per process records that the fallback engaged. Specific exceptions (`OSError`, `MissingDependencyError`, `PipelexConfigError`, `botocore.ClientError`) are caught and logged at WARNING — never silently dropped, never re-raised into the inference path.
+- **`ReportingManager` registry lookup split into `_get_registry_strict` and `_get_or_create_registry`**: strict raises on miss and is used by `_report_*_job`, so runner processes silently skip the registry add when `open_registry` was never called; or-create is used by `inject_tokens_usages`, the console cost-report path, and `generate_report` for never-opened runs. Activity workers do not accumulate orphan registries.
+- **`ContentGeneratorDry` emits `UsageReportEvent`s**: invokes `report_inference_job` with a synthetic `LLMTokensUsage` so dry-run mode produces real events observable through the same backend path as live runs.
- **`GraphSpecAssembler`**: rebuilds a complete `GraphSpec` offline from a stream of `TraceEvent`s — equivalence with the live `GraphTracer` is pinned by tests.
- **`UsageAggregator`**: aggregates per-pipeline LLM/extract/img-gen usage across workers from the event log.
- **Hub DI**: `PipelexHub.set_event_log()` / `get_event_log()` for dependency injection of custom event log backends. `make_event_log()` factory selects the backend from config or hub injection; the 3 prior call sites that hardcoded `NdjsonEventLog` now go through it.
@@ -63,7 +69,7 @@
- **Cold start trimmed**: heavy SDK imports (`boto3`, `huggingface_hub`, etc.) deferred behind `TYPE_CHECKING` or function-local guards across multiple plugins.
- **Graph viewer updated to mthds-ui v0.3.4**: bumped from v0.3.0 — additional polish atop the resizable detail panel, escape-to-close, sticky header, prompt expand/collapse with copy button, concept refinement display.
- **Default models**: small-vision and creative defaults now point to `gemini-3.0-flash-preview`. Added `claude-4.6-sonnet` and Bedrock token-auth support.
-- **`kajson` upgraded** from `0.3.1` to `0.4.2`.
+- **`kajson` upgraded** from `0.3.1` to `0.5.0`.
- **README install instructions**: replaced step-by-step Claude Code setup with single copy-paste messages for Claude Code and Codex, added manual install section.
### Fixed
diff --git a/pipelex/cogt/content_generation/content_generator_dry.py b/pipelex/cogt/content_generation/content_generator_dry.py
index 19d94ff51..f9db0d907 100644
--- a/pipelex/cogt/content_generation/content_generator_dry.py
+++ b/pipelex/cogt/content_generation/content_generator_dry.py
@@ -1,3 +1,4 @@
+from datetime import datetime
from typing import Any
from typing_extensions import override
@@ -12,8 +13,11 @@ from pipelex.cogt.image.generated_image import GeneratedImageRawDetails
from pipelex.cogt.image.image_size import ImageSize
from pipelex.cogt.img_gen.img_gen_job_components import ImgGenJobConfig, ImgGenJobParams
from pipelex.cogt.img_gen.img_gen_prompt import ImgGenPrompt
+from pipelex.cogt.llm.llm_job import LLMJob
+from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobReport
from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_prompt_factory_abstract import LLMPromptFactoryAbstract
+from pipelex.cogt.llm.llm_report import LLMTokensUsage
from pipelex.cogt.llm.llm_setting import LLMSetting
from pipelex.cogt.templating.template_category import TemplateCategory
from pipelex.cogt.templating.templating_style import TemplatingStyle
@@ -22,11 +26,56 @@ from pipelex.core.stuffs.image_content import ImageContent
from pipelex.core.stuffs.page_content import PageContent
from pipelex.core.stuffs.text_and_images_content import TextAndImagesContent
from pipelex.core.stuffs.text_content import TextContent
+from pipelex.hub import get_report_delegate
from pipelex.pipeline.job_metadata import JobMetadata
from pipelex.tools.jinja2.jinja2_parsing import check_jinja2_parsing
from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar
from pipelex.urls import URLs
+_DRY_RUN_INFERENCE_MODEL_NAME = "dry_run"
+_DRY_RUN_INFERENCE_MODEL_ID = "dry_run"
+
+
+def _report_dry_llm_job(job_metadata: JobMetadata, llm_setting: LLMSetting, llm_prompt: LLMPrompt) -> None:
+ """Build a synthetic LLMJob with zero-token usage and report it.
+
+ This makes the runner-side cross-worker emission path observable via the
+ existing dry-run e2e harness without forcing assertions to run live LLM.
+ The synthetic LLMTokensUsage carries zero counts and a "dry_run" model id
+ so consumers can distinguish dry-run usage from real-inference usage.
+
+ The synthetic job_metadata copy gets completed_at set so report_inference_job
+ can format the duration without crashing on None — matching what
+ LLMJob.llm_job_after_complete does in the live path. JobMetadata.started_at
+ can arrive either offset-naive (from the default factory) or offset-aware
+ (e.g. callers in pipe_abstract.py use datetime.now(timezone.utc)), so we
+ build the synthetic ``now`` with the same tzinfo as the incoming started_at
+ — mixing aware/naive raises TypeError when JobMetadata.duration subtracts.
+ """
+ tz = job_metadata.started_at.tzinfo if job_metadata.started_at is not None else None
+ now = datetime.now(tz)
+ synthetic_metadata = job_metadata.model_copy(
+ update={
+ "started_at": job_metadata.started_at or now,
+ "completed_at": now,
+ },
+ )
+ tokens_usage = LLMTokensUsage(
+ job_metadata=synthetic_metadata,
+ inference_model_name=_DRY_RUN_INFERENCE_MODEL_NAME,
+ inference_model_id=_DRY_RUN_INFERENCE_MODEL_ID,
+ unit_costs={},
+ nb_tokens_by_category={},
+ )
+ synthetic_job = LLMJob(
+ job_metadata=synthetic_metadata,
+ llm_prompt=llm_prompt,
+ job_params=llm_setting.make_llm_job_params(),
+ job_config=LLMJobConfig(max_retries=1),
+ job_report=LLMJobReport(llm_tokens_usage=tokens_usage),
+ )
+ get_report_delegate().report_inference_job(inference_job=synthetic_job)
+
class ContentGeneratorDry(ContentGeneratorProtocol):
"""This class is used to generate mock content for testing purposes.
@@ -60,6 +109,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):
func_name = "make_llm_text"
log.verbose(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")
prompt_truncated = llm_prompt_for_text.desc(truncate_text_length=self._text_gen_truncate_length)
+ _report_dry_llm_job(job_metadata=job_metadata, llm_setting=llm_setting_main, llm_prompt=llm_prompt_for_text)
return f"DRY RUN: {func_name} • llm_setting={llm_setting_main.desc()} • prompt={prompt_truncated}"
@override
@@ -76,6 +126,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):
# We run validators to ensure mock data is valid. Fields with format constraints
# (snake_case, PascalCase, etc.) should have `examples` defined in their Field()
# so polyfactory uses those instead of random strings.
+ _report_dry_llm_job(job_metadata=job_metadata, llm_setting=llm_setting_for_object, llm_prompt=llm_prompt_for_object)
return object_factory.build()
@override
diff --git a/pipelex/graph/graph_tracer.py b/pipelex/graph/graph_tracer.py
index bcca1e633..7471f59e7 100644
--- a/pipelex/graph/graph_tracer.py
+++ b/pipelex/graph/graph_tracer.py
@@ -171,6 +171,17 @@ class GraphTracer(GraphTracerProtocol):
self._event_sequence += 1
return seq
+ def _event_writer_id(self) -> str:
+ """Writer-id to stamp on every constructed TraceEvent.
+
+ Must only be called when self._event_log is not None. Reading from
+ the event log instance keeps the writer_id invariant local to the
+ emit site and avoids drift between the log's writer_id and the
+ events it stores.
+ """
+ assert self._event_log is not None
+ return self._event_log.writer_id
+
def _make_node_id(self) -> str:
"""Generate a node ID, including workflow_id when in Temporal mode."""
if self._workflow_id != "direct":
@@ -453,6 +464,7 @@ class GraphTracer(GraphTracerProtocol):
self._emit_event(
BatchItemEvent(
pipeline_run_id=self._event_pipeline_run_id,
+ writer_id=self._event_writer_id(),
workflow_id=self._workflow_id,
timestamp=datetime.now(timezone.utc),
sequence=self._next_event_sequence(),
@@ -495,6 +507,7 @@ class GraphTracer(GraphTracerProtocol):
self._emit_event(
BatchAggregateEvent(
pipeline_run_id=self._event_pipeline_run_id,
+ writer_id=self._event_writer_id(),
workflow_id=self._workflow_id,
timestamp=datetime.now(timezone.utc),
sequence=self._next_event_sequence(),
@@ -536,6 +549,7 @@ class GraphTracer(GraphTracerProtocol):
self._emit_event(
ParallelCombineEvent(
pipeline_run_id=self._event_pipeline_run_id,
+ writer_id=self._event_writer_id(),
workflow_id=self._workflow_id,
timestamp=datetime.now(timezone.utc),
sequence=self._next_event_sequence(),
@@ -597,6 +611,7 @@ class GraphTracer(GraphTracerProtocol):
self._emit_event(
PipeStartEvent(
pipeline_run_id=self._event_pipeline_run_id,
+ writer_id=self._event_writer_id(),
workflow_id=self._workflow_id,
timestamp=started_at,
sequence=self._next_event_sequence(),
@@ -646,6 +661,7 @@ class GraphTracer(GraphTracerProtocol):
self._emit_event(
ExecutionDataEvent(
pipeline_run_id=self._event_pipeline_run_id,
+ writer_id=self._event_writer_id(),
workflow_id=self._workflow_id,
timestamp=datetime.now(timezone.utc),
sequence=self._next_event_sequence(),
@@ -704,6 +720,7 @@ class GraphTracer(GraphTracerProtocol):
self._emit_event(
PipeEndSuccessEvent(
pipeline_run_id=self._event_pipeline_run_id,
+ writer_id=self._event_writer_id(),
workflow_id=self._workflow_id,
timestamp=ended_at,
sequence=self._next_event_sequence(),
@@ -747,6 +764,7 @@ class GraphTracer(GraphTracerProtocol):
self._emit_event(
ControllerOutputEvent(
pipeline_run_id=self._event_pipeline_run_id,
+ writer_id=self._event_writer_id(),
workflow_id=self._workflow_id,
timestamp=datetime.now(timezone.utc),
sequence=self._next_event_sequence(),
@@ -786,6 +804,7 @@ class GraphTracer(GraphTracerProtocol):
self._emit_event(
PipeEndErrorEvent(
pipeline_run_id=self._event_pipeline_run_id,
+ writer_id=self._event_writer_id(),
workflow_id=self._workflow_id,
timestamp=ended_at,
sequence=self._next_event_sequence(),
@@ -827,6 +846,7 @@ class GraphTracer(GraphTracerProtocol):
self._emit_event(
EdgeEvent(
pipeline_run_id=self._event_pipeline_run_id,
+ writer_id=self._event_writer_id(),
workflow_id=self._workflow_id,
timestamp=datetime.now(timezone.utc),
sequence=self._next_event_sequence(),
diff --git a/pipelex/reporting/reporting_manager.py b/pipelex/reporting/reporting_manager.py
index 490227793..750a3cad8 100644
--- a/pipelex/reporting/reporting_manager.py
+++ b/pipelex/reporting/reporting_manager.py
@@ -7,6 +7,7 @@ from pydantic import Field, RootModel
from typing_extensions import override
from pipelex import log
+from pipelex.base_exceptions import PipelexConfigError
from pipelex.cogt.exceptions import ReportingManagerError
from pipelex.cogt.extract.extract_job import ExtractJob
from pipelex.cogt.img_gen.img_gen_job import ImgGenJob
@@ -16,14 +17,22 @@ from pipelex.cogt.llm.llm_report import LLMTokensUsage
from pipelex.cogt.search.search_job import SearchJob
from pipelex.cogt.usage.cost_registry import CostRegistry
from pipelex.config import get_config
+from pipelex.graph.graph_context import GraphContext
from pipelex.pipeline.pipeline_models import SpecialPipelineId
from pipelex.reporting.reporting_protocol import ReportingProtocol
from pipelex.reporting.reporting_types import AnyTokensUsage, TokensUsage
+from pipelex.system.exceptions import MissingDependencyError
from pipelex.tools.misc.file_utils import ensure_path, get_incremental_file_path
from pipelex.tools.typing.pydantic_utils import empty_list_factory_of
+from pipelex.tracing.activity_event_log import ActivityEventLogCache
from pipelex.tracing.event_log_protocol import EventLogProtocol
from pipelex.tracing.trace_events import UsageReportEvent
+try:
+ from botocore.exceptions import ClientError as _BotoClientError # type: ignore[import-untyped]
+except ImportError:
+ _BotoClientError = None # type: ignore[assignment, misc]
+
if TYPE_CHECKING:
from pathlib import Path
@@ -107,16 +116,43 @@ class ReportingManager(ReportingProtocol):
# Private methods
############################################################
- def _get_registry(self, pipeline_run_id: str) -> UsageRegistry:
- if pipeline_run_id not in self._usage_registries:
- # Auto-create registry for unknown pipeline IDs. This happens when
- # Activities report inference jobs on a Temporal worker where
- # open_registry() was never called (it runs on the API process).
+ def _get_registry_strict(self, pipeline_run_id: str) -> UsageRegistry:
+ """Return the registry for ``pipeline_run_id`` or raise ``KeyError`` on miss.
+
+ Used by ``_report_*_job``: on the runner, the registry is never opened
+ because ``open_registry`` runs on the API process. Callers swallow
+ ``KeyError`` and skip the local-add — the runner-side
+ ``UsageReportEvent`` emission (Phase 2 fallback) is independent of
+ the registry, so distributed reassembly still gets the data.
+ """
+ return self._usage_registries[pipeline_run_id]
- # TODO: replace with proper distributed reporting system
+ def _get_or_create_registry(self, pipeline_run_id: str) -> UsageRegistry:
+ """Return the registry for ``pipeline_run_id``, creating it on miss.
+
+ Used by ``inject_tokens_usages`` (the P1 cross-worker assembly path),
+ the console cost-report path, and ``generate_report`` when called for
+ a specific run that wasn't opened on this process.
+ """
+ if pipeline_run_id not in self._usage_registries:
self._usage_registries[pipeline_run_id] = UsageRegistry()
return self._usage_registries[pipeline_run_id]
+ def _try_add_to_registry(self, pipeline_run_id: str, tokens_usage: AnyTokensUsage) -> None:
+ """Add to the registry if it exists; silently skip if not.
+
+ Skipping on miss is the runner-process behavior — the registry was
+ never opened here. The local-skip is intentional: the
+ ``UsageReportEvent`` emitted by ``_emit_usage_event`` carries the
+ data into the shared backend partition, and the P1 readback path
+ will assemble it on the process that owns ``inject_tokens_usages``.
+ """
+ try:
+ registry = self._get_registry_strict(pipeline_run_id)
+ except KeyError:
+ return
+ registry.add_tokens_usage(tokens_usage)
+
def _report_llm_job(self, llm_job: LLMJob):
llm_tokens_usage = llm_job.job_report.llm_tokens_usage
@@ -125,7 +161,7 @@ class ReportingManager(ReportingProtocol):
return
pipeline_run_id = llm_job.job_metadata.pipeline_run_id
- self._get_registry(pipeline_run_id).add_tokens_usage(llm_tokens_usage)
+ self._try_add_to_registry(pipeline_run_id, llm_tokens_usage)
self._emit_usage_event(llm_job, llm_tokens_usage)
if self._reporting_config.is_log_costs_to_console:
@@ -140,7 +176,7 @@ class ReportingManager(ReportingProtocol):
return
pipeline_run_id = img_gen_job.job_metadata.pipeline_run_id
- self._get_registry(pipeline_run_id).add_tokens_usage(img_gen_tokens_usage)
+ self._try_add_to_registry(pipeline_run_id, img_gen_tokens_usage)
self._emit_usage_event(img_gen_job, img_gen_tokens_usage)
if self._reporting_config.is_log_costs_to_console:
@@ -155,7 +191,7 @@ class ReportingManager(ReportingProtocol):
return
pipeline_run_id = extract_job.job_metadata.pipeline_run_id
- self._get_registry(pipeline_run_id).add_tokens_usage(extract_tokens_usage)
+ self._try_add_to_registry(pipeline_run_id, extract_tokens_usage)
self._emit_usage_event(extract_job, extract_tokens_usage)
if self._reporting_config.is_log_costs_to_console:
@@ -170,7 +206,7 @@ class ReportingManager(ReportingProtocol):
return
pipeline_run_id = search_job.job_metadata.pipeline_run_id
- self._get_registry(pipeline_run_id).add_tokens_usage(search_tokens_usage)
+ self._try_add_to_registry(pipeline_run_id, search_tokens_usage)
self._emit_usage_event(search_job, search_tokens_usage)
if self._reporting_config.is_log_costs_to_console:
@@ -198,30 +234,51 @@ class ReportingManager(ReportingProtocol):
pipeline_run_id: The pipeline run to add usage data to.
tokens_usages: Token usage records to inject.
"""
- registry = self._get_registry(pipeline_run_id)
+ registry = self._get_or_create_registry(pipeline_run_id)
for tokens_usage in tokens_usages:
registry.add_tokens_usage(tokens_usage)
def _emit_usage_event(self, inference_job: InferenceJobAbstract, tokens_usage: AnyTokensUsage) -> None:
- """Emit a UsageReportEvent if event log is configured for this job's context."""
+ """Emit a UsageReportEvent for this job.
+
+ Fast path: when set_event_log was registered for this graph context's
+ lookup_key (router process or direct mode), emit through the cached
+ per-context event log.
+
+ Fallback: when context lookup misses (runner process — set_event_log
+ was never called here), emit through the per-process activity event
+ log so the event still lands in the same backend partition as the
+ rest of the run. See _emit_usage_event_runner_fallback for details.
+ """
graph_context = inference_job.job_metadata.graph_context
if graph_context is None:
return
context = self._event_log_contexts.get(graph_context.lookup_key)
- if context is None:
+ if context is not None:
+ self._emit_via_registered_context(context, graph_context, tokens_usage)
return
- # Determine the node_id from graph context (the pipe that dispatched this inference)
- node_id: str = "unknown"
- if graph_context.parent_node_id is not None:
- node_id = graph_context.parent_node_id
+ self._emit_usage_event_runner_fallback(
+ inference_job=inference_job,
+ tokens_usage=tokens_usage,
+ graph_context=graph_context,
+ )
+ @staticmethod
+ def _emit_via_registered_context(
+ context: _EventLogContext,
+ graph_context: GraphContext,
+ tokens_usage: AnyTokensUsage,
+ ) -> None:
+ """Fast-path emit through a context registered via set_event_log."""
+ node_id = graph_context.parent_node_id or "unknown"
seq = context.event_log.next_sequence()
event = UsageReportEvent(
pipeline_run_id=context.pipeline_run_id,
workflow_id=context.workflow_id,
+ writer_id=context.event_log.writer_id,
timestamp=datetime.now(timezone.utc),
sequence=seq,
node_id=node_id,
@@ -229,6 +286,71 @@ class ReportingManager(ReportingProtocol):
)
context.event_log.emit(event)
+ def _emit_usage_event_runner_fallback(
+ self,
+ inference_job: InferenceJobAbstract,
+ tokens_usage: AnyTokensUsage,
+ graph_context: GraphContext,
+ ) -> None:
+ """Emit through a per-process activity event log when no context was registered.
+
+ On the runner, ``set_event_log`` was never called — the workflow only
+ registered a context on the router process. We fall back to a
+ process-local event log built from ``tracing_config``, stamped with a
+ stable per-process writer_id of the form ``act_{pid}_{uuid8}``.
+
+ Documented over-counting risk (R2): retried activities re-emit a fresh
+ event at sequence N+1 instead of overwriting the original at N, so the
+ same usage may be counted twice if the activity is retried by Temporal.
+ Suppression of retried-emit duplicates is a separate, harder problem
+ (deferred follow-up).
+
+ Specific exceptions caught and dropped with WARNING:
+ - ``OSError``: NDJSON dir unwritable, file system errors.
+ - ``MissingDependencyError``: ``boto3`` missing for the DynamoDB backend.
+ - ``PipelexConfigError``: factory misconfigured.
+ - ``botocore.exceptions.ClientError`` (when boto3 is installed):
+ DynamoDB throttle / auth fail at PutItem time.
+ Other exceptions propagate.
+ """
+ tracing_config = get_config().pipelex.tracing_config
+ if not tracing_config.is_enabled:
+ return
+
+ try:
+ process_event_log = ActivityEventLogCache.get_or_create(tracing_config)
+ except (OSError, MissingDependencyError, PipelexConfigError) as exc:
+ log.warning(f"Runner-side activity event log construction failed; dropping usage event: {exc}")
+ return
+
+ if process_event_log is None:
+ return
+
+ workflow_id = graph_context.tracer_key or graph_context.graph_id
+ node_id = graph_context.parent_node_id or "unknown"
+
+ ActivityEventLogCache.warn_once_runner_fallback_engaged(workflow_id=workflow_id, writer_id=process_event_log.writer_id)
+
+ seq = process_event_log.next_sequence()
+ event = UsageReportEvent(
+ pipeline_run_id=inference_job.job_metadata.pipeline_run_id,
+ workflow_id=workflow_id,
+ writer_id=process_event_log.writer_id,
+ timestamp=datetime.now(timezone.utc),
+ sequence=seq,
+ node_id=node_id,
+ tokens_usage=tokens_usage,
+ )
+
+ emit_exceptions: tuple[type[BaseException], ...] = (OSError,)
+ if _BotoClientError is not None:
+ emit_exceptions = (*emit_exceptions, _BotoClientError)
+
+ try:
+ process_event_log.emit(event)
+ except emit_exceptions as exc:
+ log.warning(f"Runner-side usage event emit failed; dropping: {exc}")
+
@override
def report_inference_job(self, inference_job: InferenceJobAbstract):
log.verbose(f"Inference job '{inference_job.job_metadata.unit_job_id}' completed in {inference_job.job_metadata.duration:.2f} seconds")
@@ -256,7 +378,7 @@ class ReportingManager(ReportingProtocol):
registries_to_process: dict[str, UsageRegistry] = {}
if pipeline_run_id:
- registries_to_process = {pipeline_run_id: self._get_registry(pipeline_run_id)}
+ registries_to_process = {pipeline_run_id: self._get_or_create_registry(pipeline_run_id)}
else:
registries_to_process = self._usage_registries
diff --git a/pipelex/temporal/config_temporal.py b/pipelex/temporal/config_temporal.py
index 4444decc4..059cd3ed9 100644
--- a/pipelex/temporal/config_temporal.py
+++ b/pipelex/temporal/config_temporal.py
@@ -175,6 +175,7 @@ class WorkerConfig(ConfigModel):
"""Configuration model for workflow execution settings."""
task_queue: str
+ inference_task_queue: str | None = None
workflow_execution_timeout: timedelta = Field(strict=False)
run_timeout: timedelta | None = Field(default=None, strict=False)
task_timeout: timedelta | None = Field(default=None, strict=False)
diff --git a/pipelex/temporal/tprl_content_generation/wf_make_llm_text.py b/pipelex/temporal/tprl_content_generation/wf_make_llm_text.py
index f635e5bed..48f296c57 100644
--- a/pipelex/temporal/tprl_content_generation/wf_make_llm_text.py
+++ b/pipelex/temporal/tprl_content_generation/wf_make_llm_text.py
@@ -27,6 +27,7 @@ class WfMakeLLMText(WorkflowClass[LLMAssignment, str]):
arg=workflow_arg,
start_to_close_timeout=worker_config.workflow_execution_timeout,
retry_policy=worker_config.retry_policy,
+ task_queue=worker_config.inference_task_queue,
)
except ActivityError as exc:
if isinstance(exc.cause, ApplicationError):
diff --git a/pipelex/tracing/activity_event_log.py b/pipelex/tracing/activity_event_log.py
new file mode 100644
index 000000000..262051c66
--- /dev/null
+++ b/pipelex/tracing/activity_event_log.py
@@ -0,0 +1,128 @@
+"""Process-local cached event log for runner-side activity emission.
+
+When activities run on a separate worker pool from the workflow router,
+the router's `set_event_log` never registers a context on the runner's
+ReportingManager — the runner-side `_event_log_contexts` dict is permanently
+empty for every workflow that runs there. This module provides the per-process
+backstop: a lazily-constructed event log, stamped with a stable
+`act_{pid}_{uuid8}` writer_id, that emits into the same backend partition as
+the rest of the run.
+
+The cache-and-writer-id construction is guarded by `threading.Lock` with
+double-checked locking so concurrent first-emitters from N activity threads
+agree on a single writer_id and reuse the same backend instance.
+"""
+
+import atexit
+import os
+import threading
+from typing import ClassVar
+from uuid import uuid4
+
+from pipelex import log
+from pipelex.system.configuration.configs import TracingConfig
+from pipelex.tracing.event_log_factory import make_event_log
+from pipelex.tracing.event_log_protocol import EventLogProtocol
+
+
+class ActivityEventLogCache:
+ """Process-local cache holding the runner-side activity event log.
+
+ All state is stored as ``ClassVar`` so the cache is naturally shared across
+ every activity thread inside the worker process, with no module-level
+ globals. ``_lock`` guards cache construction and the one-shot WARNING flag.
+ """
+
+ _lock: ClassVar[threading.Lock] = threading.Lock()
+ _cached_event_log: ClassVar[EventLogProtocol | None] = None
+ _writer_id: ClassVar[str | None] = None
+ _warning_emitted: ClassVar[bool] = False
+ _atexit_registered: ClassVar[bool] = False
+
+ @classmethod
+ def get_or_create(cls, tracing_config: TracingConfig) -> EventLogProtocol | None:
+ """Return the per-process event log for runner-side emission.
+
+ Returns None when tracing is disabled — callers must skip emit. On the
+ first successful call, generates a stable per-process writer_id of the
+ form ``act_{pid}_{uuid8}`` and constructs the backend via
+ :func:`pipelex.tracing.event_log_factory.make_event_log`. Subsequent
+ calls reuse the cached instance.
+
+ Thread-safety: cache check + creation is wrapped in a class-level
+ :class:`threading.Lock` (double-checked) so concurrent first-callers
+ from N activity threads observe the same writer_id and the same
+ backend instance.
+
+ The caller is responsible for catching specific exceptions that
+ ``make_event_log`` may raise (``OSError``, ``MissingDependencyError``,
+ ``PipelexConfigError``); this method does not swallow them so the
+ runner-side emit site can log a single WARNING with the right context.
+ """
+ if not tracing_config.is_enabled:
+ return None
+ if cls._cached_event_log is not None:
+ return cls._cached_event_log
+ with cls._lock:
+ if cls._cached_event_log is not None:
+ return cls._cached_event_log
+ new_writer_id = f"act_{os.getpid()}_{uuid4().hex[:8]}"
+ new_event_log = make_event_log(tracing_config, writer_id=new_writer_id)
+ cls._writer_id = new_writer_id
+ cls._cached_event_log = new_event_log
+ if not cls._atexit_registered:
+ atexit.register(cls._close_atexit)
+ cls._atexit_registered = True
+ return cls._cached_event_log
+
+ @classmethod
+ def warn_once_runner_fallback_engaged(cls, workflow_id: str, writer_id: str) -> None:
+ """Log a single WARNING the first time the runner fallback engages.
+
+ The warning is per-process; subsequent fallback emissions from the same
+ process are silent. Operators see one unmistakable signal in the worker
+ logs that runner-side emission is in use, without WARNING spam at high
+ activity throughput.
+ """
+ if cls._warning_emitted:
+ return
+ with cls._lock:
+ if cls._warning_emitted:
+ return
+ cls._warning_emitted = True
+ log.warning(
+ f"Runner-side usage event emission engaged "
+ f"(workflow_id={workflow_id}, writer_id={writer_id}). "
+ "Activity is emitting into the per-process activity event log because "
+ "no _event_log_contexts entry was registered in this process. "
+ "This is expected when activities run on a separate worker pool from the workflow router."
+ )
+
+ @classmethod
+ def reset_for_tests(cls) -> None:
+ """Reset cache state. For test fixtures only — never call from production code.
+
+ Closes any cached event log, clears the writer_id, and resets the one-shot
+ warning flag so subsequent tests observe a fresh process-like state. The
+ atexit registration flag is intentionally left alone — the registration
+ is process-global and the handler is a no-op once the cache is cleared.
+ """
+ if cls._cached_event_log is not None:
+ try:
+ cls._cached_event_log.close()
+ except OSError:
+ pass
+ cls._cached_event_log = None
+ cls._writer_id = None
+ cls._warning_emitted = False
+
+ @classmethod
+ def _close_atexit(cls) -> None:
+ """Best-effort close of the cached event log at interpreter shutdown."""
+ if cls._cached_event_log is None:
+ return
+ try:
+ cls._cached_event_log.close()
+ except OSError:
+ # Interpreter shutdown — file handles may already be torn down. Drop quietly.
+ pass
diff --git a/pipelex/tracing/buffering_event_log.py b/pipelex/tracing/buffering_event_log.py
index 95fd0c642..ff55bd1d8 100644
--- a/pipelex/tracing/buffering_event_log.py
+++ b/pipelex/tracing/buffering_event_log.py
@@ -4,6 +4,8 @@ Collects trace events in a list with zero I/O. Events are flushed to the
real backend (DynamoDB) via a Temporal activity after pipe execution.
"""
+import threading
+
from typing_extensions import override
from pipelex.tracing.event_log_protocol import EventLogProtocol
@@ -18,16 +20,30 @@ class BufferingEventLog(EventLogProtocol):
flush them to the real backend via act_flush_trace_events.
"""
- def __init__(self) -> None:
+ def __init__(self, writer_id: str = "primary") -> None:
self._buffer: list[TraceEvent] = []
self._sequence: int = 0
+ self._sequence_lock = threading.Lock()
+ self._writer_id = writer_id
+
+ @property
+ @override
+ def writer_id(self) -> str:
+ return self._writer_id
@override
def next_sequence(self) -> int:
- """Return the next sequence number. Shared by all emitters."""
- seq = self._sequence
- self._sequence += 1
- return seq
+ """Return the next sequence number. Shared by all emitters.
+
+ The Temporal workflow sandbox is single-threaded so this lock is
+ practically uncontended, but the contract — "implementations must be
+ safe under concurrent emitters" — is the same for every backend, so
+ we keep the guard for symmetry.
+ """
+ with self._sequence_lock:
+ seq = self._sequence
+ self._sequence += 1
+ return seq
@override
def emit(self, event: TraceEvent) -> None:
diff --git a/pipelex/tracing/dynamodb_event_log.py b/pipelex/tracing/dynamodb_event_log.py
index f95d81440..368cd91f9 100644
--- a/pipelex/tracing/dynamodb_event_log.py
+++ b/pipelex/tracing/dynamodb_event_log.py
@@ -2,13 +2,16 @@
Stores trace events in a DynamoDB table with the schema:
PK: PIPELINE_RUN#{pipeline_run_id}
- SK: EVENT#{workflow_id}#{sequence:010d}
+ SK: EVENT#{workflow_id}#{writer_id}#{sequence:010d}
payload: dict (full event via model_dump)
+Legacy emissions write writer_id="primary" so existing rows remain valid.
+
Compatible with the pipelex-api-infra TraceEventDynamoDBAdapter schema.
Requires: pip install "pipelex[dynamodb]"
"""
+import threading
from typing import Any
from pydantic import TypeAdapter, ValidationError
@@ -43,7 +46,7 @@ class DynamoDBEventLog(EventLogProtocol):
the deadlock detector. Use BufferingEventLog + act_flush_trace_events instead.
"""
- def __init__(self, table_name: str, region: str) -> None:
+ def __init__(self, table_name: str, region: str, writer_id: str = "primary") -> None:
if boto3 is None:
lib_name = "boto3"
lib_extra_name = "dynamodb"
@@ -52,24 +55,36 @@ class DynamoDBEventLog(EventLogProtocol):
self._table_name = table_name
self._region = region
+ self._writer_id = writer_id
self._sequence: int = 0
+ self._sequence_lock = threading.Lock()
dynamodb: Any = boto3.resource("dynamodb", region_name=self._region) # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType]
self._table: Any = dynamodb.Table(self._table_name) # pyright: ignore[reportUnknownMemberType]
+ @property
+ @override
+ def writer_id(self) -> str:
+ return self._writer_id
+
@override
def next_sequence(self) -> int:
- """Return the next sequence number. Shared by all emitters."""
- seq = self._sequence
- self._sequence += 1
- return seq
+ """Return the next sequence number. Shared by all emitters.
+
+ Guarded by a per-instance lock so concurrent activity threads
+ sharing this backend cannot collide on the dedup key.
+ """
+ with self._sequence_lock:
+ seq = self._sequence
+ self._sequence += 1
+ return seq
@staticmethod
def _make_pk(pipeline_run_id: str) -> str:
return f"PIPELINE_RUN#{pipeline_run_id}"
@staticmethod
- def _make_sk(workflow_id: str, sequence: int) -> str:
- return f"EVENT#{workflow_id}#{sequence:010d}"
+ def _make_sk(workflow_id: str, writer_id: str, sequence: int) -> str:
+ return f"EVENT#{workflow_id}#{writer_id}#{sequence:010d}"
def _key_condition(self, pipeline_run_id: str) -> Any:
"""Build a KeyConditionExpression for querying by pipeline_run_id."""
@@ -80,9 +95,10 @@ class DynamoDBEventLog(EventLogProtocol):
"""Write a single event to DynamoDB. Synchronous and idempotent."""
item: dict[str, Any] = {
"PK": self._make_pk(event.pipeline_run_id),
- "SK": self._make_sk(event.workflow_id, event.sequence),
+ "SK": self._make_sk(event.workflow_id, event.writer_id, event.sequence),
"pipeline_run_id": event.pipeline_run_id,
"workflow_id": event.workflow_id,
+ "writer_id": event.writer_id,
"sequence": event.sequence,
"payload": event.model_dump_json(),
}
diff --git a/pipelex/tracing/event_log_factory.py b/pipelex/tracing/event_log_factory.py
index b9f145a2b..d4715f03c 100644
--- a/pipelex/tracing/event_log_factory.py
+++ b/pipelex/tracing/event_log_factory.py
@@ -7,14 +7,19 @@ from pipelex.tracing.event_log_protocol import EventLogProtocol
from pipelex.tracing.ndjson_event_log import NdjsonEventLog
-def make_event_log(tracing_config: TracingConfig) -> EventLogProtocol:
- """Create an event log backend from tracing configuration."""
+def make_event_log(tracing_config: TracingConfig, writer_id: str = "primary") -> EventLogProtocol:
+ """Create an event log backend from tracing configuration.
+
+ The writer_id is stamped on every TraceEvent emitted through the resulting
+ log. Defaults to "primary" so existing call sites keep emitting into the
+ legacy single-writer NDJSON file / DDB sort-key shape.
+ """
match tracing_config.backend:
case TracingBackend.NDJSON:
if tracing_config.ndjson is None:
msg = "ndjson config is required when backend is 'ndjson'"
raise PipelexConfigError(msg)
- return NdjsonEventLog(traces_dir=tracing_config.ndjson.traces_dir)
+ return NdjsonEventLog(traces_dir=tracing_config.ndjson.traces_dir, writer_id=writer_id)
case TracingBackend.DYNAMODB:
if tracing_config.dynamodb is None:
msg = "dynamodb config is required when backend is 'dynamodb'"
@@ -22,6 +27,7 @@ def make_event_log(tracing_config: TracingConfig) -> EventLogProtocol:
return DynamoDBEventLog(
table_name=tracing_config.dynamodb.table_name,
region=tracing_config.dynamodb.region,
+ writer_id=writer_id,
)
case TracingBackend.TEMPORAL_DYNAMODB:
if tracing_config.temporal_dynamodb is None:
@@ -30,4 +36,5 @@ def make_event_log(tracing_config: TracingConfig) -> EventLogProtocol:
return DynamoDBEventLog(
table_name=tracing_config.temporal_dynamodb.table_name,
region=tracing_config.temporal_dynamodb.region,
+ writer_id=writer_id,
)
diff --git a/pipelex/tracing/event_log_protocol.py b/pipelex/tracing/event_log_protocol.py
index fa70c90a4..1006f3cfb 100644
--- a/pipelex/tracing/event_log_protocol.py
+++ b/pipelex/tracing/event_log_protocol.py
@@ -16,14 +16,37 @@ class EventLogProtocol(Protocol):
Implementations must guarantee that emit() is synchronous and durable:
when emit() returns, the event must be persisted. This invariant ensures
all events from a child workflow are flushed before it returns to its parent.
+
+ Each event log instance carries an immutable `writer_id` that emitters
+ stamp on every TraceEvent they construct via this log. Two writers
+ sharing the same (pipeline_run_id, workflow_id) partition must use
+ distinct writer_ids so the read-side dedup key
+ (workflow_id, writer_id, type, sequence) keeps their events apart.
"""
+ @property
+ def writer_id(self) -> str:
+ """Stable identifier for this event log instance.
+
+ Emitters read this to stamp every constructed TraceEvent. Defaults
+ to "primary" for the legacy single-writer path so existing NDJSON
+ files / DDB rows are not invalidated by code change.
+ """
+ ...
+
def next_sequence(self) -> int:
"""Return the next monotonically increasing sequence number.
All emitters sharing the same event log must use this method to
- obtain sequence numbers, ensuring events never collide on the
- (workflow_id, sequence) key regardless of storage backend.
+ obtain sequence numbers. Combined with writer_id, this guarantees
+ events never collide on the (workflow_id, writer_id, sequence) key
+ regardless of storage backend.
+
+ Implementations MUST be safe to call from multiple threads
+ concurrently — the runner-side activity event log is cached
+ per-process and shared by every activity thread on the worker, so
+ unguarded reads/writes of the underlying counter would cause
+ duplicate sequence numbers and silent dedup-key collisions.
"""
...
diff --git a/pipelex/tracing/in_memory_event_log.py b/pipelex/tracing/in_memory_event_log.py
index 639104e16..2fcf3bfa9 100644
--- a/pipelex/tracing/in_memory_event_log.py
+++ b/pipelex/tracing/in_memory_event_log.py
@@ -1,5 +1,7 @@
"""In-memory event log implementation for unit tests and direct mode."""
+import threading
+
from typing_extensions import override
from pipelex.tracing.event_log_protocol import EventLogProtocol
@@ -13,16 +15,28 @@ class InMemoryEventLog(EventLogProtocol):
Provides the same deduplication and ordering guarantees as NdjsonEventLog.
"""
- def __init__(self) -> None:
+ def __init__(self, writer_id: str = "primary") -> None:
self._events: list[TraceEvent] = []
self._sequence: int = 0
+ self._sequence_lock = threading.Lock()
+ self._writer_id = writer_id
+
+ @property
+ @override
+ def writer_id(self) -> str:
+ return self._writer_id
@override
def next_sequence(self) -> int:
- """Return the next sequence number. Shared by all emitters."""
- seq = self._sequence
- self._sequence += 1
- return seq
+ """Return the next sequence number. Shared by all emitters.
+
+ Guarded by a per-instance lock so concurrent emitters cannot
+ collide on the dedup key.
+ """
+ with self._sequence_lock:
+ seq = self._sequence
+ self._sequence += 1
+ return seq
@override
def emit(self, event: TraceEvent) -> None:
@@ -33,20 +47,22 @@ class InMemoryEventLog(EventLogProtocol):
def read_events(self, pipeline_run_id: str) -> list[TraceEvent]:
"""Return events for a pipeline run, deduplicated and sorted.
- Deduplicates by (workflow_id, sequence), keeping the first occurrence.
- Sorts by (workflow_id, sequence) for deterministic ordering.
+ Deduplicates by (workflow_id, writer_id, type, sequence), keeping
+ the first occurrence. Sorts by (workflow_id, sequence, writer_id) —
+ sequence is primary so two writers emitting into the same workflow
+ partition do not get reordered by writer-id lexicographic sort.
"""
filtered = [evt for evt in self._events if evt.pipeline_run_id == pipeline_run_id]
- seen: set[tuple[str, str, int]] = set()
+ seen: set[tuple[str, str, str, int]] = set()
deduped: list[TraceEvent] = []
for event in filtered:
- dedup_key = (event.workflow_id, type(event).__name__, event.sequence)
+ dedup_key = (event.workflow_id, event.writer_id, type(event).__name__, event.sequence)
if dedup_key not in seen:
seen.add(dedup_key)
deduped.append(event)
- deduped.sort(key=lambda evt: (evt.workflow_id, evt.sequence))
+ deduped.sort(key=lambda evt: (evt.workflow_id, evt.sequence, evt.writer_id))
return deduped
@override
diff --git a/pipelex/tracing/ndjson_event_log.py b/pipelex/tracing/ndjson_event_log.py
index 00deb7e82..07155820b 100644
--- a/pipelex/tracing/ndjson_event_log.py
+++ b/pipelex/tracing/ndjson_event_log.py
@@ -7,6 +7,7 @@ One JSON event per line, one file per workflow, organized by pipeline run:
import json
import os
import shutil
+import threading
from pathlib import Path
from typing import IO
@@ -25,20 +26,59 @@ class NdjsonEventLog(EventLogProtocol):
Write path: appends one JSON line per event, flushed immediately.
Read path: globs all .ndjson files in the run directory, parses,
- deduplicates by (workflow_id, sequence), and sorts deterministically.
+ deduplicates by (workflow_id, writer_id, type, sequence), and sorts
+ by (workflow_id, sequence, writer_id) — sequence primary so a
+ runner-side writer's events do not sort before earlier router events.
+
+ Multi-writer file naming: events from writer_id="primary" land in the
+ legacy file `wf_{workflow_id}.ndjson`; events from any other writer
+ land in `wf_{workflow_id}__w_{writer_id}.ndjson`. The file-handle cache
+ key is `(pipeline_run_id, workflow_id, writer_id)` so two writers
+ emitting concurrently never share a stale handle.
+
+ For multi-process / multi-host deployments, traces_dir must be a
+ filesystem visible to all writer processes (NFS/EFS); use the
+ DynamoDB backend for fully separated hosts.
"""
- def __init__(self, traces_dir: str) -> None:
+ def __init__(self, traces_dir: str, writer_id: str = "primary") -> None:
self._traces_dir = traces_dir
- self._file_handles: dict[tuple[str, str], IO[str]] = {}
+ self._file_handles: dict[tuple[str, str, str], IO[str]] = {}
self._sequence: int = 0
+ self._sequence_lock = threading.Lock()
+ self._writer_id = writer_id
+
+ @property
+ @override
+ def writer_id(self) -> str:
+ return self._writer_id
@override
def next_sequence(self) -> int:
- """Return the next sequence number. Shared by all emitters."""
- seq = self._sequence
- self._sequence += 1
- return seq
+ """Return the next sequence number. Shared by all emitters.
+
+ The increment is guarded by a per-instance lock so concurrent activity
+ threads sharing this backend (via ``get_or_create_activity_event_log``)
+ cannot read the same value before either increments — duplicate
+ sequence numbers would collide on the
+ ``(workflow_id, writer_id, type, sequence)`` dedup key and silently
+ drop one event.
+ """
+ with self._sequence_lock:
+ seq = self._sequence
+ self._sequence += 1
+ return seq
+
+ @staticmethod
+ def _file_name_for(workflow_id: str, writer_id: str) -> str:
+ """File name for a (workflow_id, writer_id) pair.
+
+ The legacy single-writer name is preserved when writer_id="primary"
+ so existing files continue to be written and read correctly.
+ """
+ if writer_id == "primary":
+ return f"wf_{workflow_id}.ndjson"
+ return f"wf_{workflow_id}__w_{writer_id}.ndjson"
# ------------------------------------------------------------------
# Write
@@ -49,15 +89,16 @@ class NdjsonEventLog(EventLogProtocol):
"""Append event as a JSON line, flush immediately.
Creates the run directory on first write. Caches file handles
- to avoid repeated open() calls for high-frequency events.
+ keyed by (pipeline_run_id, workflow_id, writer_id) to avoid
+ repeated open() calls for high-frequency events.
"""
- cache_key = (event.pipeline_run_id, event.workflow_id)
+ cache_key = (event.pipeline_run_id, event.workflow_id, event.writer_id)
handle = self._file_handles.get(cache_key)
if handle is None:
run_dir = os.path.join(self._traces_dir, event.pipeline_run_id)
os.makedirs(run_dir, exist_ok=True)
- file_path = os.path.join(run_dir, f"wf_{event.workflow_id}.ndjson")
+ file_path = os.path.join(run_dir, self._file_name_for(event.workflow_id, event.writer_id))
handle = open(file_path, "a", encoding="utf-8") # noqa: SIM115
self._file_handles[cache_key] = handle
@@ -83,7 +124,7 @@ class NdjsonEventLog(EventLogProtocol):
ndjson_files = sorted(Path(run_dir).glob("*.ndjson"))
- seen: set[tuple[str, str, int]] = set()
+ seen: set[tuple[str, str, str, int]] = set()
events: list[TraceEvent] = []
for ndjson_path in ndjson_files:
@@ -99,7 +140,7 @@ class NdjsonEventLog(EventLogProtocol):
log.warning(f"Skipping corrupt line in {file_path}:{line_number} — {exc}")
continue
- dedup_key = (event.workflow_id, type(event).__name__, event.sequence)
+ dedup_key = (event.workflow_id, event.writer_id, type(event).__name__, event.sequence)
if dedup_key not in seen:
seen.add(dedup_key)
events.append(event)
@@ -108,7 +149,7 @@ class NdjsonEventLog(EventLogProtocol):
# workflow ID, not execution order. In parent/child workflow topologies this can cause
# incorrect producer map overwrites in GraphSpecAssembler. Consider timestamp-based
# or topology-aware ordering.
- events.sort(key=lambda evt: (evt.workflow_id, evt.sequence))
+ events.sort(key=lambda evt: (evt.workflow_id, evt.sequence, evt.writer_id))
return events
# ------------------------------------------------------------------
@@ -119,9 +160,9 @@ class NdjsonEventLog(EventLogProtocol):
def cleanup(self, pipeline_run_id: str) -> None:
"""Close cached file handles and remove the run directory."""
keys_to_remove = [key for key in self._file_handles if key[0] == pipeline_run_id]
- for key in keys_to_remove:
- self._file_handles[key].close()
- del self._file_handles[key]
+ for cache_key in keys_to_remove:
+ self._file_handles[cache_key].close()
+ del self._file_handles[cache_key]
run_dir = os.path.join(self._traces_dir, pipeline_run_id)
if os.path.isdir(run_dir):
diff --git a/pipelex/tracing/trace_events.py b/pipelex/tracing/trace_events.py
index 0445db9d3..00edb687b 100644
--- a/pipelex/tracing/trace_events.py
+++ b/pipelex/tracing/trace_events.py
@@ -37,6 +37,12 @@ class TraceEvent(BaseModel):
Shared fields:
- pipeline_run_id: identifies the pipeline execution
- workflow_id: Temporal workflow ID or "direct" for in-process mode
+ - writer_id: identifies the emitting event-log instance, so two writers
+ sharing the same (pipeline_run_id, workflow_id) partition can emit
+ independent sequence streams without colliding. Defaults to "primary",
+ which preserves the legacy single-writer NDJSON file naming and DDB
+ sort-key shape; non-default values come from the per-process activity
+ event log used by separate-process workers.
- event_kind: discriminator for the event subclass (defined on each subclass as Literal)
- timestamp: UTC, for display/debugging only
- sequence: per-writer monotonic counter, for ordering and deduplication
@@ -44,6 +50,7 @@ class TraceEvent(BaseModel):
pipeline_run_id: str
workflow_id: str
+ writer_id: str = "primary"
timestamp: datetime
sequence: int
diff --git a/tests/integration/pipelex/temporal/tracing/helpers.py b/tests/integration/pipelex/temporal/tracing/helpers.py
index b85bb33ec..b171c4a39 100644
--- a/tests/integration/pipelex/temporal/tracing/helpers.py
+++ b/tests/integration/pipelex/temporal/tracing/helpers.py
@@ -2,16 +2,30 @@
import uuid
from collections import defaultdict
+from collections.abc import AsyncIterator
+from contextlib import asynccontextmanager
+from datetime import datetime
from pathlib import Path
+from temporalio import activity
from temporalio.client import Client as TemporalClient
+from pipelex.cogt.content_generation.assignment_models import LLMAssignment
+from pipelex.cogt.llm.llm_job import LLMJob
+from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobReport
+from pipelex.cogt.llm.llm_report import LLMTokensUsage
+from pipelex.cogt.usage.cost_category import CostCategory
+from pipelex.cogt.usage.token_category import TokenCategory
+from pipelex.config import get_config
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.graph.graph_config import DataInclusionConfig
from pipelex.graph.graph_context import GraphContext
from pipelex.graph.graphspec import EdgeKind, EdgeSpec, GraphSpec, NodeStatus, PipelineRef
+from pipelex.hub import get_report_delegate
from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.reporting.reporting_manager import ReportingManager
from pipelex.temporal.temporal_hub import get_task_manager
+from pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_text
from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
from pipelex.tracing.graphspec_assembler import GraphSpecAssembler
from pipelex.tracing.ndjson_event_log import NdjsonEventLog
@@ -133,3 +147,129 @@ def ndjson_files_for_run(traces_dir: str, pipeline_run_id: str) -> list[Path]:
def node_pipe_codes(graph_spec: GraphSpec) -> set[str | None]:
"""Extract the set of pipe_codes from all nodes in a GraphSpec."""
return {node.pipe_code for node in graph_spec.nodes}
+
+
+_RUNNER_FAKE_INFERENCE_MODEL_NAME = "split_runner_fake"
+_RUNNER_FAKE_INFERENCE_MODEL_ID = "split_runner_fake_id"
+_RUNNER_FAKE_RESPONSE_TEXT = "split-worker fake response"
+
+
[email protected](name="act_llm_gen_text")
+async def _runner_isolated_act_llm_gen_text(llm_assignment: LLMAssignment) -> str: # noqa: RUF029
+ """Substitute for `act_llm_gen_text` that exercises the Phase-2 runner-side fallback.
+
+ Within a single pytest process the router worker registers a context via
+ `set_event_log` on the process-wide `ReportingManager` singleton; a real
+ runner activity in the same process would otherwise hit that same dict and
+ take the fast path, masking the cross-worker bug. Instead this activity:
+
+ - Clears `_event_log_contexts` on entry, simulating a cold runner process.
+ - Synthesizes an `LLMJob` with non-zero usage and reports it via
+ `report_inference_job`. The reporting path runs synchronously in the
+ activity, so `_emit_usage_event` sees the cleared contexts dict and
+ takes the fallback (writes to the per-process activity event log).
+ - Returns a fixed string so the calling workflow can finish without
+ attempting a real LLM call.
+
+ This avoids the LIVE/DRY-mode dilemma: the existing `pipe_job_from_bundle`
+ fixture defaults to DRY mode, where the LLM activity is normally bypassed
+ by `ContentGeneratorDry` reporting inline inside the workflow. Subbing
+ `act_llm_gen_text` with this version forces the activity to fire from the
+ workflow's `start_activity` call (because we set
+ `inference_task_queue=q_runner`), exercising the cross-worker path even
+ in DRY mode and keeping the test hermetic.
+ """
+ delegate = get_report_delegate()
+ if isinstance(delegate, ReportingManager):
+ delegate._event_log_contexts.clear() # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ now = datetime.now()
+ synthetic_metadata = llm_assignment.job_metadata.model_copy(
+ update={
+ "started_at": llm_assignment.job_metadata.started_at or now,
+ "completed_at": now,
+ },
+ )
+ tokens_usage = LLMTokensUsage(
+ job_metadata=synthetic_metadata,
+ inference_model_name=_RUNNER_FAKE_INFERENCE_MODEL_NAME,
+ inference_model_id=_RUNNER_FAKE_INFERENCE_MODEL_ID,
+ unit_costs={CostCategory.INPUT: 0.0, CostCategory.OUTPUT: 0.0},
+ nb_tokens_by_category={TokenCategory.INPUT: 1, TokenCategory.OUTPUT: 1},
+ )
+ synthetic_job = LLMJob(
+ job_metadata=synthetic_metadata,
+ llm_prompt=llm_assignment.llm_prompt,
+ job_params=llm_assignment.llm_setting.make_llm_job_params(),
+ job_config=LLMJobConfig(max_retries=1),
+ job_report=LLMJobReport(llm_tokens_usage=tokens_usage),
+ )
+ delegate.report_inference_job(inference_job=synthetic_job)
+ return _RUNNER_FAKE_RESPONSE_TEXT
+
+
+@asynccontextmanager
+async def make_split_workers(
+ temporal_client: TemporalClient,
+ q_router: str,
+ q_runner: str,
+) -> AsyncIterator[None]:
+ """Open two scoped workers on two task queues in the current process.
+
+ - `q_router`: workflows + `act_flush_trace_events`. We don't use the bare
+ `router` scope (`disable_all_activities=True`) here because
+ `WfPipeRouter.run` dispatches `act_flush_trace_events` without a
+ `task_queue` argument — the activity lands on the workflow's own queue
+ and would never be picked up if the router registered no activities.
+ - `q_runner`: activity-only (runner scope, `disable_all_workflows=True`),
+ with `act_llm_gen_text` substituted by the isolation wrapper that clears
+ the in-process `_event_log_contexts` cache so the runner cannot
+ accidentally use the router's registered context.
+
+ Pair this with `worker_config.inference_task_queue = q_runner` so the
+ workflow on `q_router` actually dispatches `act_llm_gen_text` to `q_runner`.
+ """
+ worker_scopes = get_config().temporal.worker_scopes
+ base_router = worker_scopes.scopes["router"]
+ runner_scope = worker_scopes.scopes["runner"]
+
+ # Override the bare router scope to also register `act_flush_trace_events`
+ # so the workflow's flush call does not deadlock waiting for an activity
+ # worker on q_router. We keep `disable_all_activities=True` and add the
+ # activity back via `required_activities` — but resolution clears
+ # `disable_all_activities` last, so we instead use an empty exclude set
+ # against the runner scope's activity surface.
+ router_scope = base_router.model_copy(
+ update={
+ "disable_all_activities": False,
+ "required_activities": ["act_flush_trace_events"],
+ "excluded_activities": [
+ "act_llm_gen_text",
+ "act_llm_gen_object",
+ "act_llm_gen_object_list",
+ "act_img_gen_images",
+ "act_jinja2_gen_text",
+ "act_extract_gen_extract_pages",
+ "act_render_page_views",
+ "act_assemble_graph",
+ "act_deliver",
+ ],
+ },
+ )
+
+ async with (
+ get_task_manager().make_worker(
+ temporal_client,
+ task_queue=q_router,
+ is_not_sandboxed=True,
+ scope=router_scope,
+ ),
+ get_task_manager().make_worker(
+ temporal_client,
+ task_queue=q_runner,
+ is_not_sandboxed=True,
+ scope=runner_scope,
+ substitute_activities={act_llm_gen_text: _runner_isolated_act_llm_gen_text},
+ ),
+ ):
+ yield
diff --git a/tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py b/tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py
new file mode 100644
index 000000000..bf53616fb
--- /dev/null
+++ b/tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py
@@ -0,0 +1,190 @@
+"""Integration tests for cross-worker UsageReportEvent emission via two task queues.
+
+Validates the Phase 2 runner-side fallback: when `WfMakeLLMText` dispatches
+`act_llm_gen_text` to a separate task queue (the production deployment topology
+where router and runner are physically separated), the runner-side worker —
+with no `_event_log_contexts` entry registered for the workflow — emits its
+`UsageReportEvent` into the same `traces_dir` partition via the per-process
+activity event log, stamped with a stable `act_*` `writer_id`.
+"""
+
+import uuid
+from collections import defaultdict
+from collections.abc import Generator
+from pathlib import Path
+from typing import TYPE_CHECKING
+
+import pytest
+from temporalio.client import Client as TemporalClient
+
+from pipelex.config import get_config
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
+from pipelex.tracing.activity_event_log import ActivityEventLogCache
+from pipelex.tracing.ndjson_event_log import NdjsonEventLog
+from pipelex.tracing.trace_events import PipeStartEvent, UsageReportEvent
+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle
+from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output
+from tests.integration.pipelex.temporal.tracing.helpers import (
+ inject_graph_context,
+ make_split_workers,
+ ndjson_files_for_run,
+)
+from tests.integration.pipelex.temporal.tracing.test_data import SequenceTracingTestData
+
+if TYPE_CHECKING:
+ from pipelex.core.pipes.pipe_output import PipeOutput
+
+
[email protected](loop_scope="class")
+class TestSplitWorkerUsageEmission:
+ """Cross-process emission: router on q_router, runner on q_runner.
+
+ Each test uses its own pair of task queues (UUID-named) so concurrent
+ executions don't share queue state, and resets the per-process activity
+ event log cache so the writer_id is regenerated and the warn-once flag
+ re-arms for each test.
+ """
+
+ @pytest.fixture
+ def split_queues(self) -> tuple[str, str]:
+ """A fresh (q_router, q_runner) pair per test."""
+ return (f"q_router_{uuid.uuid4().hex[:8]}", f"q_runner_{uuid.uuid4().hex[:8]}")
+
+ @pytest.fixture
+ def live_sequence_tracing_job(self, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:
+ """Like the class-scoped `sequence_tracing_job` but in LIVE mode.
+
+ DRY mode short-circuits the `act_llm_gen_text` activity dispatch
+ (`ContentGeneratorDry` reports inline inside the workflow), so the
+ cross-worker activity hop never fires. LIVE mode forces the workflow
+ to dispatch through `WfMakeLLMText` → `act_llm_gen_text`, which is
+ what the runner-side fallback pinned in this suite is supposed to
+ exercise. The fake `act_llm_gen_text` substitute installed by
+ `make_split_workers` returns a stub string and synthesizes the
+ usage report, so no real LLM call is made.
+ """
+ yield from pipe_job_from_bundle(
+ bundle_file=SequenceTracingTestData.BUNDLE_FILE,
+ pipe_code=SequenceTracingTestData.PIPE_CODE,
+ pipe_run_mode=PipeRunMode.LIVE,
+ isolated_registry=is_class_registry_isolated,
+ )
+
+ @pytest.fixture
+ def configure_inference_task_queue(self, split_queues: tuple[str, str]) -> Generator[None, None, None]:
+ """Route `act_llm_gen_text` to `q_runner` for the duration of the test.
+
+ Without this override, `WfMakeLLMText` would dispatch the activity to
+ whichever task queue the workflow is running on (the router queue),
+ defeating the split-worker topology this suite is supposed to exercise.
+ """
+ _q_router, q_runner = split_queues
+ worker_config = get_config().temporal.worker_config
+ original = worker_config.inference_task_queue
+ worker_config.inference_task_queue = q_runner
+ yield
+ worker_config.inference_task_queue = original
+
+ @pytest.fixture(autouse=True)
+ def reset_activity_event_log(self) -> Generator[None, None, None]:
+ """Clear the per-process activity event log cache between tests.
+
+ The cache (writer_id, event log instance, warn-once flag) is class-level
+ on `ActivityEventLogCache`. Without this reset, two tests in the same
+ module would share a writer_id and the second test would find a stale,
+ possibly closed, event log handle.
+ """
+ ActivityEventLogCache.reset_for_tests()
+ yield
+ ActivityEventLogCache.reset_for_tests()
+
+ async def _execute_split(
+ self,
+ live_sequence_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ split_queues: tuple[str, str],
+ ) -> str:
+ """Run the workflow on q_router; activities dispatch to q_runner."""
+ q_router, q_runner = split_queues
+ execution_run_id = f"split_exec_{uuid.uuid4().hex[:12]}"
+ execution_job = inject_graph_context(live_sequence_tracing_job, execution_run_id)
+ workflow_id = f"wf_{uuid.uuid4().hex[:12]}"
+
+ async with make_split_workers(temporal_client, q_router=q_router, q_runner=q_runner):
+ pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfPipeRouter.run,
+ arg=execution_job,
+ id=workflow_id,
+ task_queue=q_router,
+ )
+ rehydrate_pipe_output(pipe_output)
+ return execution_run_id
+
+ async def test_runner_usage_event_lands_in_same_ndjson_dir(
+ self,
+ live_sequence_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ tracing_tmp_dir: Path,
+ split_queues: tuple[str, str],
+ configure_inference_task_queue: None, # noqa: ARG002
+ ) -> None:
+ """Router on q_router dispatches inference activity to q_runner.
+
+ The runner — with no `_event_log_contexts` entry — emits its
+ `UsageReportEvent` into the same `traces_dir` partition via the
+ Phase-2 fallback. Both the router-written `wf_{wf_id}.ndjson`
+ (from `act_flush_trace_events`) and at least one runner-written
+ `wf_{wf_id}__w_act_*.ndjson` are present.
+ """
+ run_id = await self._execute_split(live_sequence_tracing_job, temporal_client, split_queues)
+
+ ndjson_files = ndjson_files_for_run(str(tracing_tmp_dir), run_id)
+ primary_files = [path for path in ndjson_files if "__w_" not in path.name]
+ runner_files = [path for path in ndjson_files if "__w_act_" in path.name]
+ assert primary_files, f"Expected a primary wf_*.ndjson router file in {[path.name for path in ndjson_files]}"
+ assert runner_files, f"Expected at least one wf_*__w_act_*.ndjson runner file in {[path.name for path in ndjson_files]}"
+
+ reader = NdjsonEventLog(traces_dir=str(tracing_tmp_dir))
+ try:
+ events = reader.read_events(run_id)
+ finally:
+ reader.close()
+
+ primary_pipe_starts = [evt for evt in events if isinstance(evt, PipeStartEvent) and evt.writer_id == "primary"]
+ runner_usage_events = [evt for evt in events if isinstance(evt, UsageReportEvent) and evt.writer_id.startswith("act_")]
+ assert primary_pipe_starts, "Expected at least one PipeStartEvent with writer_id='primary' from the router"
+ assert runner_usage_events, "Expected at least one UsageReportEvent with writer_id='act_*' from the runner"
+
+ async def test_no_double_emit_in_split_worker_pool(
+ self,
+ live_sequence_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ tracing_tmp_dir: Path,
+ split_queues: tuple[str, str],
+ configure_inference_task_queue: None, # noqa: ARG002
+ ) -> None:
+ """Exactly-once invariant: `_emit_usage_event` takes either the fast
+ path OR the fallback path, never both. Submit one workflow; assert
+ exactly one `UsageReportEvent` per `(node_id, workflow_id)` across
+ all NDJSON files.
+ """
+ run_id = await self._execute_split(live_sequence_tracing_job, temporal_client, split_queues)
+
+ reader = NdjsonEventLog(traces_dir=str(tracing_tmp_dir))
+ try:
+ events = reader.read_events(run_id)
+ finally:
+ reader.close()
+
+ usage_events = [evt for evt in events if isinstance(evt, UsageReportEvent)]
+ assert usage_events, "Expected at least one UsageReportEvent across the run"
+
+ counts: dict[tuple[str, str], int] = defaultdict(int)
+ for usage_event in usage_events:
+ counts[usage_event.node_id, usage_event.workflow_id] += 1
+
+ for key, count in counts.items():
+ assert count == 1, f"Duplicate UsageReportEvent for (node_id, workflow_id)={key}: count={count}"
diff --git a/tests/unit/pipelex/cogt/content_generation/test_content_generator_dry_reporting.py b/tests/unit/pipelex/cogt/content_generation/test_content_generator_dry_reporting.py
new file mode 100644
index 000000000..8796bb314
--- /dev/null
+++ b/tests/unit/pipelex/cogt/content_generation/test_content_generator_dry_reporting.py
@@ -0,0 +1,55 @@
+"""Pin that ContentGeneratorDry invokes report_inference_job with a synthetic LLMJob.
+
+This is a Phase-2 prerequisite for the Phase-6 e2e cross-worker emission tier:
+without the dry generator emitting a UsageReportEvent, the Tier 8 assertions in
+.claude/skills/temporal-e2e-validate/SKILL.md cannot run dry — they would have
+to require live LLM calls to observe the runner-side fallback.
+"""
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.cogt.content_generation.content_generator_dry import ContentGeneratorDry
+from pipelex.cogt.llm.llm_job import LLMJob
+from pipelex.cogt.llm.llm_prompt import LLMPrompt
+from pipelex.cogt.llm.llm_setting import LLMSetting
+from pipelex.pipeline.job_metadata import JobMetadata
+
+
+class TestContentGeneratorDryReporting:
+ """Pin the ContentGeneratorDry → ReportingManager.report_inference_job hook."""
+
+ @pytest.mark.asyncio
+ async def test_make_llm_text_invokes_report_inference_job(self, mocker: MockerFixture) -> None:
+ """make_llm_text builds a synthetic LLMJob and reports it through get_report_delegate()."""
+ captured_jobs: list[LLMJob] = []
+
+ def _capture(inference_job: LLMJob) -> None:
+ captured_jobs.append(inference_job)
+
+ mock_delegate = mocker.MagicMock()
+ mock_delegate.report_inference_job.side_effect = _capture
+ mocker.patch(
+ "pipelex.cogt.content_generation.content_generator_dry.get_report_delegate",
+ return_value=mock_delegate,
+ )
+
+ generator = ContentGeneratorDry()
+ job_metadata = JobMetadata(
+ user_id="test_user",
+ pipeline_run_id="run_dry_001",
+ )
+ llm_setting = LLMSetting(model="gpt-4o", temperature=0.5)
+ llm_prompt = LLMPrompt()
+
+ await generator.make_llm_text(
+ job_metadata=job_metadata,
+ llm_setting_main=llm_setting,
+ llm_prompt_for_text=llm_prompt,
+ )
+
+ assert len(captured_jobs) == 1
+ reported_job = captured_jobs[0]
+ assert isinstance(reported_job, LLMJob)
+ assert reported_job.job_metadata.pipeline_run_id == "run_dry_001"
+ assert reported_job.job_report.llm_tokens_usage is not None
diff --git a/tests/unit/pipelex/cogt/content_generation/test_content_generator_dry_tz.py b/tests/unit/pipelex/cogt/content_generation/test_content_generator_dry_tz.py
new file mode 100644
index 000000000..d919e22f7
--- /dev/null
+++ b/tests/unit/pipelex/cogt/content_generation/test_content_generator_dry_tz.py
@@ -0,0 +1,68 @@
+"""Pin that the dry generator copes with both naive and aware ``started_at``.
+
+The synthetic ``completed_at`` must share the tzinfo of the incoming
+``started_at`` so ``JobMetadata.duration`` can subtract them without crossing
+naive and aware datetimes (which raises ``TypeError``). Callers in the codebase
+do construct aware datetimes (e.g. ``pipe_abstract.py``), so the dry path must
+not crash when one of those flows in.
+"""
+
+from datetime import datetime, timezone
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.cogt.content_generation.content_generator_dry import ContentGeneratorDry
+from pipelex.cogt.llm.llm_job import LLMJob
+from pipelex.cogt.llm.llm_prompt import LLMPrompt
+from pipelex.cogt.llm.llm_setting import LLMSetting
+from pipelex.pipeline.job_metadata import JobMetadata
+
+
+class TestContentGeneratorDryTimezone:
+ @pytest.mark.parametrize(
+ "started_at",
+ [
+ pytest.param(datetime(2020, 1, 1, 10, 0, 0), id="naive"),
+ pytest.param(datetime(2020, 1, 1, 10, 0, 0, tzinfo=timezone.utc), id="aware_utc"),
+ pytest.param(None, id="none_uses_default_factory"),
+ ],
+ )
+ @pytest.mark.asyncio
+ async def test_duration_is_finite_for_any_started_at_tz(
+ self,
+ started_at: datetime | None,
+ mocker: MockerFixture,
+ ) -> None:
+ captured_jobs: list[LLMJob] = []
+
+ def _capture(inference_job: LLMJob) -> None:
+ captured_jobs.append(inference_job)
+
+ mock_delegate = mocker.MagicMock()
+ mock_delegate.report_inference_job.side_effect = _capture
+ mocker.patch(
+ "pipelex.cogt.content_generation.content_generator_dry.get_report_delegate",
+ return_value=mock_delegate,
+ )
+
+ generator = ContentGeneratorDry()
+ job_metadata = JobMetadata(
+ user_id="test_user",
+ pipeline_run_id="run_dry_tz",
+ started_at=started_at,
+ )
+ llm_setting = LLMSetting(model="gpt-4o", temperature=0.5)
+ llm_prompt = LLMPrompt()
+
+ await generator.make_llm_text(
+ job_metadata=job_metadata,
+ llm_setting_main=llm_setting,
+ llm_prompt_for_text=llm_prompt,
+ )
+
+ assert len(captured_jobs) == 1
+ reported_metadata = captured_jobs[0].job_metadata
+ duration = reported_metadata.duration
+ assert duration is not None
+ assert duration >= 0.0
diff --git a/tests/unit/pipelex/reporting/test_emit_runner_fallback.py b/tests/unit/pipelex/reporting/test_emit_runner_fallback.py
new file mode 100644
index 000000000..87bdcfb75
--- /dev/null
+++ b/tests/unit/pipelex/reporting/test_emit_runner_fallback.py
@@ -0,0 +1,388 @@
+"""Tests for the runner-side activity event log fallback in ReportingManager._emit_usage_event.
+
+Pins the behavior introduced in Phase 2 of the cross-worker tracing P0 plan:
+when context lookup misses (runner process never had set_event_log called for
+this workflow), the manager emits the usage event through a per-process
+activity event log instead of dropping silently.
+"""
+
+import logging
+import threading
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+from typing import Any
+
+import pytest
+from botocore.exceptions import ClientError
+from pytest_mock import MockerFixture
+
+from pipelex.base_exceptions import PipelexConfigError
+from pipelex.cogt.llm.llm_job import LLMJob
+from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobParams, LLMJobReport
+from pipelex.cogt.llm.llm_prompt import LLMPrompt
+from pipelex.cogt.llm.llm_report import LLMTokensUsage
+from pipelex.cogt.usage.cost_category import CostCategory
+from pipelex.cogt.usage.token_category import TokenCategory
+from pipelex.config import get_config
+from pipelex.graph.graph_config import DataInclusionConfig
+from pipelex.graph.graph_context import GraphContext
+from pipelex.pipeline.job_metadata import JobMetadata, UnitJobId
+from pipelex.reporting.reporting_manager import ReportingManager
+from pipelex.system.configuration.configs import NdjsonTracingConfig, TracingBackend
+from pipelex.system.exceptions import MissingDependencyError
+from pipelex.tracing import activity_event_log
+from pipelex.tracing.activity_event_log import ActivityEventLogCache
+from pipelex.tracing.in_memory_event_log import InMemoryEventLog
+from pipelex.tracing.ndjson_event_log import NdjsonEventLog
+from pipelex.tracing.trace_events import UsageReportEvent
+
+DATA_INCLUSION_OFF = DataInclusionConfig(
+ pipe_and_concept_registry=False,
+ stuff_json_content=False,
+ stuff_text_content=False,
+ stuff_html_content=False,
+ error_stack_traces=False,
+)
+
+
+def _make_llm_job(
+ pipeline_run_id: str,
+ graph_context: GraphContext | None,
+ nb_input_tokens: int = 100,
+ nb_output_tokens: int = 50,
+) -> LLMJob:
+ now = datetime.now(timezone.utc)
+ job_metadata = JobMetadata(
+ user_id="test_user",
+ pipeline_run_id=pipeline_run_id,
+ graph_context=graph_context,
+ started_at=now,
+ completed_at=now + timedelta(seconds=1),
+ unit_job_id=UnitJobId.LLM_GEN_TEXT,
+ )
+ tokens_usage = LLMTokensUsage(
+ job_metadata=job_metadata,
+ inference_model_name="test-model",
+ inference_model_id="test-model-id",
+ unit_costs={CostCategory.INPUT: 1.0, CostCategory.OUTPUT: 2.0},
+ nb_tokens_by_category={TokenCategory.INPUT: nb_input_tokens, TokenCategory.OUTPUT: nb_output_tokens},
+ )
+ return LLMJob(
+ job_metadata=job_metadata,
+ llm_prompt=LLMPrompt(),
+ job_params=LLMJobParams(temperature=0.5),
+ job_config=LLMJobConfig(max_retries=1),
+ job_report=LLMJobReport(llm_tokens_usage=tokens_usage),
+ )
+
+
+def _make_graph_context(
+ graph_id: str,
+ parent_node_id: str | None = "g:node_0",
+ tracer_key: str | None = None,
+) -> GraphContext:
+ return GraphContext(
+ graph_id=graph_id,
+ tracer_key=tracer_key,
+ parent_node_id=parent_node_id,
+ node_sequence=0,
+ data_inclusion=DATA_INCLUSION_OFF,
+ )
+
+
[email protected](autouse=True)
+def _reset_activity_event_log_state() -> Any: # pyright: ignore[reportUnusedFunction]
+ """Class-level state on ActivityEventLogCache persists across tests; reset it."""
+ ActivityEventLogCache.reset_for_tests()
+ yield
+ ActivityEventLogCache.reset_for_tests()
+
+
+def _enable_ndjson_tracing(mocker: MockerFixture, traces_dir: Path) -> None:
+ """Patch the live tracing config to enabled NDJSON pointing at traces_dir."""
+ cfg = get_config().pipelex.tracing_config
+ mocker.patch.object(cfg, "is_enabled", True)
+ mocker.patch.object(cfg, "backend", TracingBackend.NDJSON)
+ mocker.patch.object(cfg, "ndjson", NdjsonTracingConfig(traces_dir=str(traces_dir)))
+
+
+class TestEmitRunnerFallback:
+ """Pins the runner-side fallback emission contract."""
+
+ def test_fallback_engages_when_context_missing_and_tracing_enabled(
+ self,
+ tmp_path: Path,
+ mocker: MockerFixture,
+ ) -> None:
+ """When set_event_log is never called and tracing is enabled, the fallback emits an NDJSON usage event."""
+ _enable_ndjson_tracing(mocker, tmp_path)
+
+ manager = ReportingManager()
+ manager.setup()
+ graph_context = _make_graph_context(graph_id="run_abc", tracer_key="wf_xyz", parent_node_id="g:node_2")
+ llm_job = _make_llm_job("run_abc", graph_context=graph_context)
+
+ manager.report_inference_job(llm_job)
+
+ run_dir = tmp_path / "run_abc"
+ ndjson_files = list(run_dir.glob("*.ndjson"))
+ assert len(ndjson_files) == 1
+ assert ndjson_files[0].name.startswith("wf_wf_xyz__w_act_"), ndjson_files[0].name
+
+ reader = NdjsonEventLog(traces_dir=str(tmp_path))
+ events = reader.read_events("run_abc")
+ usage_events = [evt for evt in events if isinstance(evt, UsageReportEvent)]
+ assert len(usage_events) == 1
+ assert usage_events[0].workflow_id == "wf_xyz"
+ assert usage_events[0].pipeline_run_id == "run_abc"
+ assert usage_events[0].node_id == "g:node_2"
+ assert usage_events[0].writer_id.startswith("act_")
+
+ def test_runner_fallback_uses_tracer_key_when_set(
+ self,
+ tmp_path: Path,
+ mocker: MockerFixture,
+ ) -> None:
+ """workflow_id on the emitted event is graph_context.tracer_key when set, else graph_id."""
+ _enable_ndjson_tracing(mocker, tmp_path)
+
+ manager = ReportingManager()
+ manager.setup()
+
+ # Case 1: tracer_key set → workflow_id == tracer_key
+ ctx_with_tracer = _make_graph_context(graph_id="run_abc", tracer_key="wf_xyz")
+ manager.report_inference_job(_make_llm_job("run_abc", graph_context=ctx_with_tracer))
+
+ # Case 2: tracer_key None → workflow_id == graph_id
+ ctx_no_tracer = _make_graph_context(graph_id="run_def", tracer_key=None)
+ manager.report_inference_job(_make_llm_job("run_def", graph_context=ctx_no_tracer))
+
+ reader = NdjsonEventLog(traces_dir=str(tmp_path))
+ events_abc = [evt for evt in reader.read_events("run_abc") if isinstance(evt, UsageReportEvent)]
+ events_def = [evt for evt in reader.read_events("run_def") if isinstance(evt, UsageReportEvent)]
+
+ assert len(events_abc) == 1
+ assert events_abc[0].workflow_id == "wf_xyz"
+ assert len(events_def) == 1
+ assert events_def[0].workflow_id == "run_def"
+
+ def test_fallback_caches_event_log_per_process(
+ self,
+ tmp_path: Path,
+ mocker: MockerFixture,
+ ) -> None:
+ """make_event_log is called only once per process — the second emission reuses the cached log."""
+ _enable_ndjson_tracing(mocker, tmp_path)
+ spy = mocker.spy(activity_event_log, "make_event_log")
+
+ manager = ReportingManager()
+ manager.setup()
+ graph_context = _make_graph_context(graph_id="run_abc", tracer_key="wf_xyz")
+
+ manager.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))
+ manager.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))
+
+ assert spy.call_count == 1
+
+ def test_concurrent_first_call_yields_single_writer_id(
+ self,
+ tmp_path: Path,
+ mocker: MockerFixture,
+ ) -> None:
+ """Concurrent first-callers from N threads observe the same writer_id and event log instance.
+
+ Pins the threading.Lock fix: without double-checked locking, two threads could each
+ construct their own backend, generating two writer_ids and racing on the file.
+ """
+ _enable_ndjson_tracing(mocker, tmp_path)
+ cfg = get_config().pipelex.tracing_config
+
+ nb_threads = 16
+ barrier = threading.Barrier(nb_threads)
+ seen_logs: list[Any] = []
+ seen_lock = threading.Lock()
+
+ def call_fn() -> None:
+ barrier.wait()
+ log_instance = ActivityEventLogCache.get_or_create(cfg)
+ with seen_lock:
+ seen_logs.append(log_instance)
+
+ threads = [threading.Thread(target=call_fn) for _ in range(nb_threads)]
+ for thread in threads:
+ thread.start()
+ for thread in threads:
+ thread.join(timeout=10)
+
+ assert len(seen_logs) == nb_threads
+ # All threads must observe the same backend instance and writer_id.
+ for log_instance in seen_logs:
+ assert log_instance is seen_logs[0]
+ writer_ids = {log_instance.writer_id for log_instance in seen_logs}
+ assert len(writer_ids) == 1
+
+ def test_disabled_tracing_skips_emit(
+ self,
+ tmp_path: Path,
+ mocker: MockerFixture,
+ ) -> None:
+ """When tracing_config.is_enabled=False, the fallback returns without writing any file."""
+ cfg = get_config().pipelex.tracing_config
+ mocker.patch.object(cfg, "is_enabled", False)
+
+ manager = ReportingManager()
+ manager.setup()
+ graph_context = _make_graph_context(graph_id="run_silent", tracer_key="wf_silent")
+
+ manager.report_inference_job(_make_llm_job("run_silent", graph_context=graph_context))
+
+ # No NDJSON files should have been created.
+ assert list(tmp_path.glob("**/*.ndjson")) == []
+
+ def test_no_graph_context_skips_emit(
+ self,
+ tmp_path: Path,
+ mocker: MockerFixture,
+ ) -> None:
+ """When job_metadata.graph_context is None, neither fast path nor fallback engages."""
+ _enable_ndjson_tracing(mocker, tmp_path)
+
+ manager = ReportingManager()
+ manager.setup()
+
+ manager.report_inference_job(_make_llm_job("run_abc", graph_context=None))
+
+ assert list(tmp_path.glob("**/*.ndjson")) == []
+
+ @pytest.mark.parametrize(
+ ("exc_type", "exc_args"),
+ [
+ (OSError, ("disk full",)),
+ (MissingDependencyError, ("boto3", "dynamodb")),
+ (PipelexConfigError, ("misconfigured",)),
+ ],
+ ids=["oserror", "missing_dep", "config_error"],
+ )
+ def test_explicit_log_when_emit_path_unavailable(
+ self,
+ tmp_path: Path,
+ mocker: MockerFixture,
+ caplog: pytest.LogCaptureFixture,
+ exc_type: type[BaseException],
+ exc_args: tuple[Any, ...],
+ ) -> None:
+ """Specific exception types from make_event_log are caught, logged WARNING, and dropped.
+
+ Implementation must NOT use except Exception; each type is named explicitly.
+ """
+ _enable_ndjson_tracing(mocker, tmp_path)
+ mocker.patch(
+ "pipelex.tracing.activity_event_log.make_event_log",
+ side_effect=exc_type(*exc_args),
+ )
+
+ manager = ReportingManager()
+ manager.setup()
+ graph_context = _make_graph_context(graph_id="run_abc", tracer_key="wf_xyz")
+
+ with caplog.at_level(logging.WARNING):
+ # Must not raise — the failure is logged and dropped.
+ manager.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))
+
+ assert any(record.levelno == logging.WARNING for record in caplog.records)
+
+ def test_explicit_log_when_emit_raises_client_error(
+ self,
+ tmp_path: Path,
+ mocker: MockerFixture,
+ caplog: pytest.LogCaptureFixture,
+ ) -> None:
+ """A botocore ClientError raised by event_log.emit() is caught, logged WARNING, and dropped."""
+ _enable_ndjson_tracing(mocker, tmp_path)
+
+ # Force the event log to raise ClientError on emit.
+ stub_log = InMemoryEventLog(writer_id="act_test")
+ client_error = ClientError(
+ error_response={"Error": {"Code": "ProvisionedThroughputExceededException"}},
+ operation_name="PutItem",
+ )
+ mocker.patch.object(stub_log, "emit", side_effect=client_error)
+ mocker.patch(
+ "pipelex.tracing.activity_event_log.make_event_log",
+ return_value=stub_log,
+ )
+
+ manager = ReportingManager()
+ manager.setup()
+ graph_context = _make_graph_context(graph_id="run_abc", tracer_key="wf_xyz")
+
+ with caplog.at_level(logging.WARNING):
+ manager.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))
+
+ assert any(record.levelno == logging.WARNING for record in caplog.records)
+
+ def test_warning_emitted_once_per_process_when_fallback_engages(
+ self,
+ tmp_path: Path,
+ mocker: MockerFixture,
+ caplog: pytest.LogCaptureFixture,
+ ) -> None:
+ """The 'runner fallback engaged' warning fires only on the first emission per process."""
+ _enable_ndjson_tracing(mocker, tmp_path)
+
+ manager = ReportingManager()
+ manager.setup()
+ graph_context = _make_graph_context(graph_id="run_abc", tracer_key="wf_xyz")
+
+ with caplog.at_level(logging.WARNING):
+ for _ in range(100):
+ manager.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))
+
+ engaged_records = [record for record in caplog.records if "runner-side" in record.message.lower()]
+ assert len(engaged_records) == 1
+
+ def test_warning_emitted_once_even_with_multiple_managers(
+ self,
+ tmp_path: Path,
+ mocker: MockerFixture,
+ caplog: pytest.LogCaptureFixture,
+ ) -> None:
+ """Module-level once-flag survives multiple ReportingManager instances."""
+ _enable_ndjson_tracing(mocker, tmp_path)
+
+ graph_context = _make_graph_context(graph_id="run_abc", tracer_key="wf_xyz")
+
+ with caplog.at_level(logging.WARNING):
+ manager_a = ReportingManager()
+ manager_a.setup()
+ manager_a.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))
+
+ manager_b = ReportingManager()
+ manager_b.setup()
+ manager_b.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))
+
+ engaged_records = [record for record in caplog.records if "runner-side" in record.message.lower()]
+ assert len(engaged_records) == 1
+
+ def test_retried_activity_emits_duplicate_usage_event_documenting_r2(
+ self,
+ tmp_path: Path,
+ mocker: MockerFixture,
+ ) -> None:
+ """Pins R2: retried activities re-emit; today's behavior is two events at sequence N and N+1.
+
+ Documents over-counting risk and prevents silent regression of this contract.
+ """
+ _enable_ndjson_tracing(mocker, tmp_path)
+
+ manager = ReportingManager()
+ manager.setup()
+ graph_context = _make_graph_context(graph_id="run_abc", tracer_key="wf_xyz")
+
+ manager.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))
+ manager.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))
+
+ reader = NdjsonEventLog(traces_dir=str(tmp_path))
+ events = [evt for evt in reader.read_events("run_abc") if isinstance(evt, UsageReportEvent)]
+ assert len(events) == 2
+ assert sorted(evt.sequence for evt in events) == [0, 1]
diff --git a/tests/unit/pipelex/reporting/test_no_orphan_registries.py b/tests/unit/pipelex/reporting/test_no_orphan_registries.py
new file mode 100644
index 000000000..65e24ad75
--- /dev/null
+++ b/tests/unit/pipelex/reporting/test_no_orphan_registries.py
@@ -0,0 +1,120 @@
+"""Pin the three-method registry split contract.
+
+Phase 3 of the cross-worker tracing P0 plan replaces the single auto-creating
+``_get_registry`` with two explicit methods:
+
+- ``_get_registry_strict`` raises ``KeyError`` when the registry was never
+ opened. Used by the ``_report_*_job`` paths so the runner-process case
+ (no ``open_registry`` call) does not silently accumulate orphan registries.
+- ``_get_or_create_registry`` creates on miss. Used by ``inject_tokens_usages``
+ (the P1 cross-worker assembly path) and ``generate_report`` for runs not
+ opened on this process.
+"""
+
+from datetime import datetime, timedelta, timezone
+
+import pytest
+
+from pipelex.cogt.llm.llm_job import LLMJob
+from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobParams, LLMJobReport
+from pipelex.cogt.llm.llm_prompt import LLMPrompt
+from pipelex.cogt.llm.llm_report import LLMTokensUsage
+from pipelex.cogt.usage.cost_category import CostCategory
+from pipelex.cogt.usage.token_category import TokenCategory
+from pipelex.pipeline.job_metadata import JobMetadata, UnitJobId
+from pipelex.reporting.reporting_manager import ReportingManager
+
+
+def _make_llm_job(pipeline_run_id: str) -> LLMJob:
+ now = datetime.now(timezone.utc)
+ job_metadata = JobMetadata(
+ user_id="test_user",
+ pipeline_run_id=pipeline_run_id,
+ started_at=now,
+ completed_at=now + timedelta(seconds=1),
+ unit_job_id=UnitJobId.LLM_GEN_TEXT,
+ )
+ tokens_usage = LLMTokensUsage(
+ job_metadata=job_metadata,
+ inference_model_name="test-model",
+ inference_model_id="test-model-id",
+ unit_costs={CostCategory.INPUT: 1.0, CostCategory.OUTPUT: 2.0},
+ nb_tokens_by_category={TokenCategory.INPUT: 100, TokenCategory.OUTPUT: 50},
+ )
+ return LLMJob(
+ job_metadata=job_metadata,
+ llm_prompt=LLMPrompt(),
+ job_params=LLMJobParams(temperature=0.5),
+ job_config=LLMJobConfig(max_retries=1),
+ job_report=LLMJobReport(llm_tokens_usage=tokens_usage),
+ )
+
+
+class TestNoOrphanRegistries:
+ """Phase 3 contract: runner-process reporting must not silently auto-create registries."""
+
+ def test_runner_does_not_accumulate_registries(self) -> None:
+ """When open_registry was never called, _report_llm_job must not insert a registry.
+
+ The runner process gets activities reported with pipeline_run_ids it has never seen.
+ Pre-Phase-3 behavior auto-created an empty registry on every miss, accumulating
+ memory for every distinct pipeline_run_id. Phase 3 makes _report_*_job silently skip
+ the add (the usage event still emits via the runner fallback path).
+ """
+ manager = ReportingManager()
+ manager.setup()
+
+ manager.report_inference_job(_make_llm_job("never_opened"))
+
+ assert "never_opened" not in manager._usage_registries # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ def test_direct_mode_still_accumulates_registry(self) -> None:
+ """When open_registry was called for this run, _report_llm_job must add to it."""
+ manager = ReportingManager()
+ manager.setup()
+ manager.open_registry("run_x")
+
+ manager.report_inference_job(_make_llm_job("run_x"))
+
+ registry = manager._get_or_create_registry("run_x") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+ usages = registry.get_current_tokens_usage()
+ assert len(usages) == 1
+ assert usages[0].nb_tokens_by_category[TokenCategory.INPUT] == 100
+
+ def test_inject_tokens_usages_still_creates_on_miss(self) -> None:
+ """inject_tokens_usages preserves auto-create behavior via _get_or_create_registry.
+
+ This is the P1 cross-worker assembly path: usage records collected from trace
+ events get injected into a registry that may not have been opened on this
+ process — the registry must be created on demand.
+ """
+ manager = ReportingManager()
+ manager.setup()
+
+ tokens_usage = LLMTokensUsage(
+ job_metadata=_make_llm_job("new_run").job_metadata,
+ inference_model_name="test-model",
+ inference_model_id="test-model-id",
+ unit_costs={CostCategory.INPUT: 1.0, CostCategory.OUTPUT: 2.0},
+ nb_tokens_by_category={TokenCategory.INPUT: 50, TokenCategory.OUTPUT: 25},
+ )
+ manager.inject_tokens_usages(pipeline_run_id="new_run", tokens_usages=[tokens_usage])
+
+ registry = manager._get_or_create_registry("new_run") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+ assert len(registry.get_current_tokens_usage()) == 1
+
+ def test_get_registry_strict_raises_when_missing(self) -> None:
+ """_get_registry_strict raises KeyError on miss — distinct from _get_or_create_registry."""
+ manager = ReportingManager()
+ manager.setup()
+
+ with pytest.raises(KeyError):
+ manager._get_registry_strict("never_opened") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ def test_generate_report_for_unopened_run_creates_empty_registry_and_renders(self) -> None:
+ """generate_report on a never-opened run does not crash; renders empty cost report."""
+ manager = ReportingManager()
+ manager.setup()
+
+ # Must not raise.
+ manager.generate_report(pipeline_run_id="never_opened")
diff --git a/tests/unit/pipelex/reporting/test_reporting_event_emission.py b/tests/unit/pipelex/reporting/test_reporting_event_emission.py
index e8cdff7c7..51e9ac53b 100644
--- a/tests/unit/pipelex/reporting/test_reporting_event_emission.py
+++ b/tests/unit/pipelex/reporting/test_reporting_event_emission.py
@@ -7,12 +7,15 @@ isolation for concurrent workflows.
from datetime import datetime, timedelta, timezone
+from pytest_mock import MockerFixture
+
from pipelex.cogt.llm.llm_job import LLMJob
from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobParams, LLMJobReport
from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_report import LLMTokensUsage
from pipelex.cogt.usage.cost_category import CostCategory
from pipelex.cogt.usage.token_category import TokenCategory
+from pipelex.config import get_config
from pipelex.graph.graph_config import DataInclusionConfig
from pipelex.graph.graph_context import GraphContext
from pipelex.pipeline.job_metadata import JobMetadata, UnitJobId
@@ -159,7 +162,7 @@ class TestReportingEventEmission:
tokens_usages=[tokens_usage],
)
- registry = manager._get_registry(self.PIPELINE_RUN_ID) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+ registry = manager._get_or_create_registry(self.PIPELINE_RUN_ID) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
usages = registry.get_current_tokens_usage()
assert len(usages) == 1
assert usages[0].nb_tokens_by_category[TokenCategory.INPUT] == 200
@@ -182,7 +185,7 @@ class TestReportingEventEmission:
tokens_usages=[tokens_usage],
)
- registry = manager._get_registry("new_run") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+ registry = manager._get_or_create_registry("new_run") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
usages = registry.get_current_tokens_usage()
assert len(usages) == 1
@@ -304,3 +307,25 @@ class TestReportingEventEmission:
events_b = [evt for evt in event_log_b.read_events("run_b") if isinstance(evt, UsageReportEvent)]
assert [evt.sequence for evt in events_a] == [0, 1, 2]
assert [evt.sequence for evt in events_b] == [0]
+
+ def test_no_emit_when_no_event_log_set_and_no_fallback_yet(self, mocker: MockerFixture) -> None:
+ """Pin the tracing-disabled silent baseline.
+
+ When set_event_log is never called and tracing_config.is_enabled=False,
+ report_inference_job must not write any event anywhere — even after the
+ Phase 2 fallback lands. The fallback path is only allowed to engage when
+ tracing is enabled.
+ """
+ # Force tracing to disabled regardless of process config.
+ mocker.patch.object(get_config().pipelex.tracing_config, "is_enabled", False)
+
+ manager = ReportingManager()
+ manager.setup()
+ manager.open_registry("run_silent")
+
+ graph_context = _make_graph_context(graph_id="run_silent", tracer_key="wf_silent", parent_node_id="g:node_0")
+ llm_job = _make_test_llm_job("run_silent", graph_context=graph_context)
+ manager.report_inference_job(llm_job)
+
+ # No event log was ever set; no event should have been emitted anywhere.
+ assert len(manager._event_log_contexts) == 0 # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
diff --git a/tests/unit/pipelex/tracing/test_event_log_sequence_threadsafe.py b/tests/unit/pipelex/tracing/test_event_log_sequence_threadsafe.py
new file mode 100644
index 000000000..a69abc2ce
--- /dev/null
+++ b/tests/unit/pipelex/tracing/test_event_log_sequence_threadsafe.py
@@ -0,0 +1,85 @@
+"""Concurrent sequence allocation must yield unique, contiguous values across threads."""
+
+import sys
+import threading
+from collections.abc import Iterator
+from typing import TYPE_CHECKING, Any
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.tracing.buffering_event_log import BufferingEventLog
+from pipelex.tracing.dynamodb_event_log import DynamoDBEventLog
+from pipelex.tracing.in_memory_event_log import InMemoryEventLog
+from pipelex.tracing.ndjson_event_log import NdjsonEventLog
+
+if TYPE_CHECKING:
+ from pipelex.tracing.event_log_protocol import EventLogProtocol
+
+
+def aggressive_gil_switching() -> Iterator[None]:
+ """Lower the GIL switch interval to force frequent thread interleaving.
+
+ Without this, CPython batches many bytecodes between GIL releases and the
+ `seq = self._sequence; self._sequence += 1` race effectively never triggers
+ in short test runs. Restored after the test.
+ """
+ original_interval = sys.getswitchinterval()
+ sys.setswitchinterval(1e-9)
+ try:
+ yield
+ finally:
+ sys.setswitchinterval(original_interval)
+
+
[email protected]("aggressive_gil_switching")
+class TestEventLogSequenceThreadSafety:
+ @pytest.mark.parametrize(
+ "backend_name",
+ ["ndjson", "in_memory", "buffering", "dynamodb"],
+ )
+ def test_concurrent_next_sequence_returns_unique_contiguous_values(
+ self,
+ backend_name: str,
+ tmp_path: Any,
+ mocker: MockerFixture,
+ ) -> None:
+ nb_threads = 16
+ nb_calls = 2000
+ total_calls = nb_threads * nb_calls
+
+ event_log: EventLogProtocol
+ match backend_name:
+ case "ndjson":
+ event_log = NdjsonEventLog(traces_dir=str(tmp_path))
+ case "in_memory":
+ event_log = InMemoryEventLog()
+ case "buffering":
+ event_log = BufferingEventLog()
+ case "dynamodb":
+ mocker.patch("pipelex.tracing.dynamodb_event_log.boto3")
+ event_log = DynamoDBEventLog(table_name="t", region="us-east-1")
+ case _:
+ msg = f"Unknown backend_name: {backend_name}"
+ raise ValueError(msg)
+
+ results: list[int] = []
+ results_lock = threading.Lock()
+ barrier = threading.Barrier(nb_threads)
+
+ def worker() -> None:
+ barrier.wait()
+ local: list[int] = [event_log.next_sequence() for _ in range(nb_calls)]
+ with results_lock:
+ results.extend(local)
+
+ threads = [threading.Thread(target=worker) for _ in range(nb_threads)]
+ for thread in threads:
+ thread.start()
+ for thread in threads:
+ thread.join()
+
+ assert len(results) == total_calls
+ assert len(set(results)) == total_calls, "next_sequence returned duplicate values under concurrency"
+ assert set(results) == set(range(total_calls)), "next_sequence skipped or repeated values under concurrency"
diff --git a/tests/unit/pipelex/tracing/test_writer_id_schema.py b/tests/unit/pipelex/tracing/test_writer_id_schema.py
new file mode 100644
index 000000000..cbf6bacbc
--- /dev/null
+++ b/tests/unit/pipelex/tracing/test_writer_id_schema.py
@@ -0,0 +1,220 @@
+"""Writer-id schema tests.
+
+Pins the contract that every TraceEvent carries a writer_id field, that backends
+expose a writer_id property, and that the read-side dedup/sort/file-routing keys
+all incorporate writer_id correctly.
+"""
+
+import json
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+from pytest_mock import MockerFixture
+
+from pipelex.graph.graphspec import NodeKind
+from pipelex.tracing.buffering_event_log import BufferingEventLog
+from pipelex.tracing.dynamodb_event_log import DynamoDBEventLog
+from pipelex.tracing.in_memory_event_log import InMemoryEventLog
+from pipelex.tracing.ndjson_event_log import NdjsonEventLog
+from pipelex.tracing.trace_events import PipeStartEvent
+
+_TS = datetime(2025, 6, 15, 10, 30, 0, tzinfo=timezone.utc)
+_PIPELINE_RUN_ID = "run_writer_id_001"
+_WORKFLOW_ID = "wf_writer_id_xyz"
+
+
+def _make_pipe_start(sequence: int, writer_id: str = "primary", workflow_id: str = _WORKFLOW_ID) -> PipeStartEvent:
+ return PipeStartEvent(
+ pipeline_run_id=_PIPELINE_RUN_ID,
+ workflow_id=workflow_id,
+ writer_id=writer_id,
+ timestamp=_TS,
+ sequence=sequence,
+ node_id=f"g:{workflow_id}:node_{sequence}",
+ pipe_code="pipe_x",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ )
+
+
+class TestWriterIdSchema:
+ """Tests pinning the writer_id contract across TraceEvent and event log backends."""
+
+ def test_trace_event_has_writer_id_default_primary(self) -> None:
+ """A TraceEvent built without writer_id round-trips with writer_id="primary"."""
+ event = PipeStartEvent(
+ pipeline_run_id=_PIPELINE_RUN_ID,
+ workflow_id=_WORKFLOW_ID,
+ timestamp=_TS,
+ sequence=0,
+ node_id=f"g:{_WORKFLOW_ID}:node_0",
+ pipe_code="pipe_x",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ )
+
+ json_str = event.model_dump_json()
+ payload: dict[str, Any] = json.loads(json_str)
+
+ assert payload["writer_id"] == "primary"
+ restored = PipeStartEvent.model_validate_json(json_str)
+ assert restored.writer_id == "primary"
+
+ def test_legacy_ndjson_without_writer_id_field_reads_as_primary(self, tmp_path: Path) -> None:
+ """An NDJSON line missing the writer_id field reads back with writer_id="primary".
+
+ Pins backwards compatibility with files written before the writer_id field landed.
+ """
+ run_dir = tmp_path / _PIPELINE_RUN_ID
+ run_dir.mkdir(parents=True)
+ ndjson_file = run_dir / f"wf_{_WORKFLOW_ID}.ndjson"
+
+ legacy_payload: dict[str, Any] = {
+ "pipeline_run_id": _PIPELINE_RUN_ID,
+ "workflow_id": _WORKFLOW_ID,
+ "timestamp": _TS.isoformat(),
+ "sequence": 0,
+ "event_kind": "pipe_start",
+ "node_id": f"g:{_WORKFLOW_ID}:node_0",
+ "pipe_code": "pipe_x",
+ "pipe_type": "PipeLLM",
+ "node_kind": NodeKind.OPERATOR,
+ }
+ ndjson_file.write_text(json.dumps(legacy_payload) + "\n", encoding="utf-8")
+
+ event_log = NdjsonEventLog(traces_dir=str(tmp_path))
+ events = event_log.read_events(_PIPELINE_RUN_ID)
+
+ assert len(events) == 1
+ assert events[0].writer_id == "primary"
+
+ def test_two_writers_same_workflow_dedup_keeps_both(self) -> None:
+ """Read-side dedup distinguishes events by writer_id.
+
+ Two events with same (workflow_id, type, sequence) but different writer_id
+ must both survive dedup.
+ """
+ event_log = InMemoryEventLog()
+ event_log.emit(_make_pipe_start(sequence=0, writer_id="a"))
+ event_log.emit(_make_pipe_start(sequence=0, writer_id="b"))
+
+ events = event_log.read_events(_PIPELINE_RUN_ID)
+ writer_ids = {evt.writer_id for evt in events}
+
+ assert len(events) == 2
+ assert writer_ids == {"a", "b"}
+
+ def test_ndjson_writer_id_in_filename_for_non_primary(self, tmp_path: Path) -> None:
+ """NDJSON file naming includes writer_id when non-primary; legacy name when primary."""
+ primary_log = NdjsonEventLog(traces_dir=str(tmp_path))
+ runner_log = NdjsonEventLog(traces_dir=str(tmp_path), writer_id="act_pid42")
+
+ primary_log.emit(_make_pipe_start(sequence=0))
+ runner_log.emit(_make_pipe_start(sequence=0, writer_id="act_pid42"))
+
+ run_dir = tmp_path / _PIPELINE_RUN_ID
+ ndjson_files = sorted(path.name for path in run_dir.glob("*.ndjson"))
+
+ assert ndjson_files == [f"wf_{_WORKFLOW_ID}.ndjson", f"wf_{_WORKFLOW_ID}__w_act_pid42.ndjson"]
+
+ def test_two_writers_same_workflow_get_separate_handles(self, tmp_path: Path) -> None:
+ """Two NdjsonEventLogs with different writer_ids must not share the file-handle cache slot.
+
+ Pins the (pipeline_run_id, workflow_id, writer_id) cache key. With a 2-tuple key,
+ the second writer would steal the first writer's file handle and corrupt routing.
+ """
+ log_a = NdjsonEventLog(traces_dir=str(tmp_path), writer_id="writer_a")
+ log_b = NdjsonEventLog(traces_dir=str(tmp_path), writer_id="writer_b")
+
+ log_a.emit(_make_pipe_start(sequence=0, writer_id="writer_a"))
+ log_b.emit(_make_pipe_start(sequence=0, writer_id="writer_b"))
+ log_a.close()
+ log_b.close()
+
+ run_dir = tmp_path / _PIPELINE_RUN_ID
+ a_lines = (run_dir / f"wf_{_WORKFLOW_ID}__w_writer_a.ndjson").read_text(encoding="utf-8").splitlines()
+ b_lines = (run_dir / f"wf_{_WORKFLOW_ID}__w_writer_b.ndjson").read_text(encoding="utf-8").splitlines()
+
+ assert len(a_lines) == 1
+ assert len(b_lines) == 1
+ assert json.loads(a_lines[0])["writer_id"] == "writer_a"
+ assert json.loads(b_lines[0])["writer_id"] == "writer_b"
+
+ def test_ndjson_dedup_uses_writer_id(self, tmp_path: Path) -> None:
+ """Read-side dedup for NDJSON keeps events with same (workflow_id, sequence) but different writer_id."""
+ log_a = NdjsonEventLog(traces_dir=str(tmp_path), writer_id="writer_a")
+ log_b = NdjsonEventLog(traces_dir=str(tmp_path), writer_id="writer_b")
+
+ log_a.emit(_make_pipe_start(sequence=0, writer_id="writer_a"))
+ log_b.emit(_make_pipe_start(sequence=0, writer_id="writer_b"))
+ log_a.close()
+ log_b.close()
+
+ reader = NdjsonEventLog(traces_dir=str(tmp_path))
+ events = reader.read_events(_PIPELINE_RUN_ID)
+
+ assert len(events) == 2
+ assert {evt.writer_id for evt in events} == {"writer_a", "writer_b"}
+
+ def test_ndjson_sort_order_is_sequence_primary_writer_id_secondary(self, tmp_path: Path) -> None:
+ """Sort order is (workflow_id, sequence, writer_id) — sequence is primary.
+
+ Without this fix, sorting by (workflow_id, writer_id, sequence) would put
+ runner-side `act_*` events before router-side `primary` events for the same
+ workflow even though the router's events came earlier in time.
+ """
+ primary_log = NdjsonEventLog(traces_dir=str(tmp_path))
+ runner_log = NdjsonEventLog(traces_dir=str(tmp_path), writer_id="act_x")
+
+ # Emit out of order to exercise the sort.
+ primary_log.emit(_make_pipe_start(sequence=2))
+ runner_log.emit(_make_pipe_start(sequence=0, writer_id="act_x"))
+ primary_log.emit(_make_pipe_start(sequence=1))
+ primary_log.close()
+ runner_log.close()
+
+ reader = NdjsonEventLog(traces_dir=str(tmp_path))
+ events = reader.read_events(_PIPELINE_RUN_ID)
+
+ assert [evt.sequence for evt in events] == [0, 1, 2]
+ assert events[0].writer_id == "act_x"
+ assert events[1].writer_id == "primary"
+ assert events[2].writer_id == "primary"
+
+ def test_dynamodb_sk_includes_writer_id(self, mocker: MockerFixture) -> None:
+ """DynamoDB SK encodes writer_id between workflow_id and sequence."""
+ captured_items: list[dict[str, Any]] = []
+
+ class _StubTable:
+ def put_item(self, *, Item: dict[str, Any]) -> None: # noqa: N803 # boto3 kwarg name
+ captured_items.append(Item)
+
+ class _StubResource:
+ @staticmethod
+ def Table(_name: str) -> _StubTable: # noqa: N802 # boto3 method name
+ return _StubTable()
+
+ mocker.patch(
+ "pipelex.tracing.dynamodb_event_log.boto3.resource",
+ return_value=_StubResource(),
+ )
+
+ event_log = DynamoDBEventLog(table_name="my_table", region="us-east-1", writer_id="act_pid42")
+ event_log.emit(_make_pipe_start(sequence=0, writer_id="act_pid42"))
+
+ assert len(captured_items) == 1
+ assert captured_items[0]["SK"] == f"EVENT#{_WORKFLOW_ID}#act_pid42#0000000000"
+
+ def test_event_log_exposes_writer_id_property(self, tmp_path: Path) -> None:
+ """Every event log backend exposes its writer_id via a read-only property.
+
+ Emitters use this to stamp writer_id at event construction time.
+ """
+ in_mem = InMemoryEventLog()
+ ndjson = NdjsonEventLog(traces_dir=str(tmp_path), writer_id="act_x")
+ buffering = BufferingEventLog()
+
+ assert in_mem.writer_id == "primary"
+ assert ndjson.writer_id == "act_x"
+ assert buffering.writer_id == "primary"
diff --git a/wip/02-master-plan.md b/wip/02-master-plan.md
index 583337bd9..41672d5ef 100644
--- a/wip/02-master-plan.md
+++ b/wip/02-master-plan.md
@@ -12,7 +12,9 @@ The big direction now is to actually run **activities on standalone Worker pools
| # | Item | Status | Owner file |
|---|---|---|---|
-| **P0** | **Tracing & cost reporting across separate-process workers** | Not started — top priority | this file (problem statement) |
+| **P0** | **Tracing & cost reporting across separate-process workers** | Done — shipped on `fix/Tracing-across-workers`; see [tracing-cost-reporting-as-built.md](tracing-cost-reporting-as-built.md) (T1 marked fixed) | this file (problem statement) |
+| **P0.1** | **Dry-run through activity dispatch (testing affordance)** | Not started — surfaced during P0 Phase 6 validation | this file (problem statement) |
+| **P0.2** | **Deferred follow-ons from P0** | Not started — surfaced during P0 eng review + PR #860 SWE-agent review | this file (problem statement) |
| **P1** | **Cross-worker cost report assembly wiring** | Not started — depends on P0 design | this file (problem statement) |
| **P2** | Phase 6a — Local cross-package dependencies in crate | Not started | [phase6a-local-cross-package-deps.md](phase6a-local-cross-package-deps.md) |
| **P3** | Phase 6b — Remote dependencies from GitHub | Not started — needs P2 | [phase6b-remote-deps-from-github.md](phase6b-remote-deps-from-github.md) |
@@ -23,7 +25,9 @@ Sibling tracks (separate branches, not in this plan): worker error-handling Phas
---
-## P0 — Tracing & cost reporting across separate-process workers
+## P0 — Tracing & cost reporting across separate-process workers — DONE
+
+Shipped on `fix/Tracing-across-workers`. The full implementation history (six phases, eng-review notes, decisions, test inventory, atomic-commit list) is preserved in the branch's git log and summarized in [tracing-cost-reporting-as-built.md](tracing-cost-reporting-as-built.md). The architectural notes below are kept for context. Follow-ons that were explicitly deferred during implementation are tracked under §P0.2.
### Why this is top priority
@@ -41,13 +45,56 @@ The whole point of the distributed-execution work is to allow **activities on st
We're not designing the fix yet — that comes when we pick up P0. The two natural starting points are (a) the original Phase 4.5 Step 6 `TracingActivityInboundInterceptor` design (now in `archive/00-master-plan.md` and `archive/01-master-plan.md`), and (b) plumbing tracing config + workflow id through `JobMetadata` so each activity can construct its own event log directly. Either way the activity needs request-scoped tracing data, not process-scoped state.
-### Acceptance criteria (sketch — refine when planning)
+### Acceptance criteria — all met
-- [ ] Activities deployed on standalone Worker pools (separate process from workflow Workers) emit `UsageReportEvent`s that land in the same backend partition as the rest of the run.
-- [ ] No reliance on `WfPipeRouter` having executed in the activity's process.
-- [ ] No silent drops — if tracing is enabled, an activity that fails to emit raises or logs explicitly.
-- [ ] Direct mode and the current single-bundle Worker mode keep working unchanged.
-- [ ] Tests covering: (1) standalone activity Worker, (2) mixed worker pool, (3) tracing disabled, (4) backend = NDJSON, (5) backend = DynamoDB.
+- [x] Activities deployed on standalone Worker pools (separate process from workflow Workers) emit `UsageReportEvent`s that land in the same backend partition as the rest of the run.
+- [x] No reliance on `WfPipeRouter` having executed in the activity's process.
+- [x] No silent drops — if tracing is enabled, an activity that fails to emit raises or logs explicitly.
+- [x] Direct mode and the current single-bundle Worker mode keep working unchanged.
+- [x] Tests covering: standalone activity Worker, mixed worker pool, tracing disabled, NDJSON backend, DynamoDB backend (unit-level via stubbed boto3; full DDB e2e gated on `pytest -m dynamodb`).
+
+---
+
+## P0.1 — Dry-run through activity dispatch (testing affordance)
+
+Today, `--dry-run` instantiates `ContentGeneratorDry()` directly inside the workflow body (`pipe_llm.py:515`, and similar sites in `pipe_extract.py`, `pipe_compose.py`, `pipe_img_gen.py`). The activity (`act_llm_gen_text` etc.) is never dispatched. As a result, the cross-worker `UsageReportEvent` emission path that P0 ships cannot be exercised in dry-run mode against router+runner workers — the runner-side fallback only fires when the activity actually dispatches across processes.
+
+This was discovered while validating P0 Phase 6.1: a vanilla `pipelex run bundle --temporal --dry-run --mock-inputs` against a 2-worker (router+runner) topology produces only `writer_id="primary"` events; no `wf_*__w_act_*.ndjson` files appear. The Phase 4 integration test (`TestSplitWorkerUsageEmission`) works around this by substituting `act_llm_gen_text` with a wrapper that synthesizes an `LLMJob` server-side, but that's test-internal scaffolding, not a general-purpose mode.
+
+### What's needed
+
+A run mode that combines "no real LLM call" with "still go through the activity dispatch path", so a `pipelex run bundle` against router+runner workers exercises the same code path as live mode, just with mocked inference inside the activity. Concretely:
+
+- `act_llm_gen_text` is dispatched to the runner as in live mode.
+- Inside the activity, the inference is mocked (use `ContentGeneratorDry` *inside* the activity body, not in place of dispatching it).
+- All cross-process surfaces — `_event_log_contexts` lookup miss, runner-side fallback, `writer_id="act_*"` emission — fire normally.
+
+### Why this matters beyond P0
+
+Same affordance unlocks dry-run e2e for any future cross-process work (P1 cost report assembly, distributed graph assembly, payload codec stress testing, etc.) without needing real LLM spend.
+
+### Acceptance criteria (sketch)
+
+- [ ] A `--mock-inference` (or similar) flag — distinct from `--dry-run` — that keeps activity dispatch but mocks inside the activity. Or: redefine `--dry-run` to mean "dispatch activities, mock inside" and rename the current behavior.
+- [ ] `temporal-e2e-validate` Tier 8 can run in this mode and surface `wf_*__w_act_*.ndjson` files deterministically.
+- [ ] Equivalent mocking sites in `pipe_extract.py`, `pipe_compose.py`, `pipe_img_gen.py` updated.
+
+---
+
+## P0.2 — Deferred follow-ons from P0
+
+Items surfaced during the P0 eng review and the post-merge SWE-agent review of PR #860. Each is independently scoped and not blocking for P0.1 / P1 / P2 / P3. Pick up individually as priorities and signals dictate.
+
+### From eng review (logged at the time of P0 implementation)
+
+- [ ] **`tracing_config.strict_mode: bool`** — opt-in flag that raises (instead of WARNING + drop) when the runner-side emit path fails. Useful for compliance/audit deployments where missing trace data is a hard error. (Eng review R6.)
+- [ ] **DynamoDB `BatchWriteItem` for runner emission** — current path does one `PutItem` per usage event. At high-concurrency runners that's one boto3 call per emit; `BatchWriteItem` batches up to 25. Defer until actual throughput shows it matters. (Eng review R3.)
+- [ ] **NDJSON shared-filesystem invariant enforcement** — the NDJSON backend assumes `traces_dir` is visible to all writer processes (router pool + runner pool + `act_flush_trace_events`). For multi-host deployments this needs NFS/EFS or equivalent. Follow-up: a startup check that warns if `traces_dir` looks like a local-only path on a multi-worker deployment. (Eng review TODO-2.)
+
+### From PR #860 SWE-agent review (post-merge triage of `fix/Tracing-across-workers`)
+
+- [ ] **Per-thread `writer_id` for activity event log** — the surgical lock fix from PR #860 closes the duplicate-sequence race on `next_sequence()`, but a per-thread writer_id would eliminate the contention entirely and align the design with how Temporal's worker pool already partitions activities. Each thread would emit into its own writer namespace; dedup naturally drops to per-thread, no lock needed in the hot path.
+- [ ] **Project-wide migration of `JobMetadata.started_at` to timezone-aware datetimes** — the PR #860 patch builds the synthetic `now` with the same `tzinfo` as the incoming `started_at`, which removes the immediate `TypeError` crash but doesn't fix the underlying inconsistency: `JobMetadata.started_at` default factory is naive (`datetime.now`) while several callers (e.g. `pipe_abstract.py:428`) construct aware datetimes (`datetime.now(timezone.utc)`). A clean migration would standardize on `datetime.now(timezone.utc)` everywhere, document the tz contract on `JobMetadata`, and let `duration` subtract without defensive tzinfo matching at every site.
---
@@ -94,6 +141,9 @@ In one line: extend the blueprint collector from P2 with a remote (GitHub) resol
```
P0 (Tracing across separate workers)
+ │
+ ├─► P0.1 (Dry-run through activity dispatch — testing affordance)
+ ├─► P0.2 (Deferred follow-ons — independent, pick up individually)
│
▼
P1 (Cross-worker cost report assembly)
@@ -104,4 +154,4 @@ P2 (Phase 6a — local cross-package deps)
P3 (Phase 6b — remote deps from GitHub)
```
-P0/P1 and P2/P3 are independent tracks; P0 blocks P1, P2 blocks P3.
+P0/P0.1/P0.2/P1 and P2/P3 are independent tracks; P0 unlocks P0.1, P0.2, and P1; P2 blocks P3. P0.1 is a parallel testing affordance that improves P0 / P1 / future cross-process work confidence. P0.2 items are independent of each other — pick them up individually as needs and signals dictate.
diff --git a/wip/tracing-cost-reporting-as-built.md b/wip/tracing-cost-reporting-as-built.md
index e4c471fac..caed9acb0 100644
--- a/wip/tracing-cost-reporting-as-built.md
+++ b/wip/tracing-cost-reporting-as-built.md
@@ -34,21 +34,25 @@ The original Step 6 plan called for a `TracingActivityInboundInterceptor` to set
## What is broken or missing
-### Issue T1 — Activities on standalone (separate-process) workers lose usage events
+### Issue T1 — Activities on standalone (separate-process) workers lose usage events — **FIXED (P0)**
-**Severity**: Blocks the whole point of distributed activity workers.
+**Severity (was)**: Blocked the whole point of distributed activity workers.
-`tasks.py` currently bundles workflows + activities into the same `TaskPack`, so this issue is latent. But the explicit design intent is to split activities onto standalone Worker pools (separate processes from the workflow Worker pool). When that split happens:
+**Status**: Resolved on `fix/Tracing-across-workers` (P0 in [02-master-plan.md](02-master-plan.md)). The full implementation history (six phases, eng-review notes, decisions, test inventory, atomic-commit list) is preserved in the branch's git log.
-- The activity's process never runs `WfPipeRouter.run()`, so its `ReportingManager` has no entry in `_event_log_contexts` for the workflow's `lookup_key`.
-- `ReportingManager._emit_usage_event` returns silently when the context lookup misses (`reporting_manager.py:211-213`).
-- `ReportingManager._get_registry` auto-creates a worker-local `UsageRegistry` (with the explicit TODO at `reporting_manager.py:111-117`: *"Auto-create registry for unknown pipeline IDs. This happens when Activities report inference jobs on a Temporal worker where open_registry() was never called (it runs on the API process). TODO: replace with proper distributed reporting system"*).
-- The worker-local registry is never read back by anyone — usage data is lost on the activity worker.
+The fix has two independent parts that ship together:
-**Note**: The `BufferingEventLog` + `act_flush_trace_events` pattern is structurally tied to a workflow lifecycle; it cannot serve standalone activity workers as-is. A different mechanism is needed (the original interceptor design, or per-activity event log construction from tracing config + JobMetadata).
+- **Runner-side fallback in `_emit_usage_event`.** When the activity's `ReportingManager` has no entry in `_event_log_contexts` for the workflow's `lookup_key`, it now falls back to a process-local event log built from `tracing_config`. The event log is cached per process via `pipelex.tracing.activity_event_log.get_or_create_activity_event_log` (guarded by a module-level `threading.Lock` so concurrent first-emitters from multiple activity threads agree on a single writer_id and instance). The fallback uses `graph_context.tracer_key or graph_context.graph_id` as `workflow_id` (no new `JobMetadata` field needed — `tracer_key` is already populated by `WfPipeRouter.run`). Specific exceptions (`OSError`, `MissingDependencyError`, `PipelexConfigError`, `botocore.ClientError`) are caught at WARNING — never silently dropped, never `except Exception`. A one-shot warning per process records that the fallback engaged.
+- **Writer-id disambiguation in `TraceEvent`.** A new `writer_id: str = "primary"` field makes `(workflow_id, sequence)` collisions impossible across writers. NDJSON file naming becomes `wf_{workflow_id}__w_{writer_id}.ndjson` for non-primary writers; the `(pipeline_run_id, workflow_id, writer_id)` cache key prevents two writers from sharing a stale file handle. Read-side dedup key is `(workflow_id, writer_id, type, sequence)`; sort key is `(workflow_id, sequence, writer_id)` — sequence primary so a runner-side `UsageReportEvent` does not sort before earlier router events. DynamoDB SK becomes `EVENT#{workflow_id}#{writer_id}#{sequence:010d}`.
+
+The `_get_registry` orphan-accumulation TODO at `reporting_manager.py` is gone: the method was split into `_get_registry_strict` (used by `_report_*_job` — runner processes silently skip the registry add) and `_get_or_create_registry` (used by `inject_tokens_usages`, the console cost-report path, and `generate_report`).
+
+The known retry-related over-counting case (R2) is documented and pinned by `test_retried_activity_emits_duplicate_usage_event_documenting_r2`. A `tracing_config.strict_mode` flag (raise instead of WARNING + drop) is deferred — see [02-master-plan.md §P0.2](02-master-plan.md).
### Issue T2 — Cross-worker cost report assembly is not wired
+**Status**: Still open. P0 (T1 above) ensures the events now land in the same backend partition from every worker; P1 is the remaining plumbing change to read them back into a cost report.
+
**Severity**: High — events are persisted but never turned into a report.
The pieces all exist:
@@ -63,6 +67,8 @@ Net effect: even when usage events ARE captured (i.e., the same-worker case in t
### Issue T3 — `set_event_log` is on a process-singleton `ReportingManager`
+**Status**: Partially mitigated by P0. The runner-side fallback now bypasses `_event_log_contexts` entirely when the lookup misses, so the singleton-+-context-dict pattern no longer blocks standalone activity workers. The deeper "request-scoped tracing data instead of process-scoped state" refactor remains open and is the natural follow-up alongside P1.
+
**Severity**: Design smell that magnifies T1.
`ReportingManager` is a process-singleton accessed via `hub.get_report_delegate()`. The per-context `_event_log_contexts` dict + `lookup_key` scheme works because the workflow process is the same process that emits events from activities (today). The moment activity execution moves off-process, the singleton + context-dict pattern stops being the right shape — the activity needs to pull its event log from request-scoped data (e.g., `JobMetadata` / `TracingContext`), not from a process-local dict that was never populated on its process.
@@ -71,6 +77,6 @@ Net effect: even when usage events ARE captured (i.e., the same-worker case in t
## How this connects to 02-master-plan
-- T1 is the **top priority** in [02-master-plan.md](02-master-plan.md) — the whole point of distributed execution is to allow activities on standalone Worker pools.
-- T2 is the **second priority** — once T1 is solved, the assembly wiring is the small remaining step that makes cost reporting actually report.
-- T3 is the architectural shape that the T1 + T2 work will refactor.
+- T1 was the **top priority** in [02-master-plan.md](02-master-plan.md) — now resolved on `fix/Tracing-across-workers`.
+- T2 is the **next priority** — events now land in the same partition from every worker; the small remaining step is wiring `read events → UsageAggregator → inject_tokens_usages → generate_report`.
+- T3 is the architectural shape; the T1 fix mitigates the immediate impact, the deeper refactor will land alongside T2 / further follow-ups.