Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
fix(temporal): workflow determinism (#987)
Louis Choquela7e44c5b2c28
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 4bc057732..f84b134da 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,8 @@- **`--dry-run` now honors the configured backend — the mock moved from the operators to the cogt leaf.** `run_mode` rides a new `CogtRunParams` carrier (derived from `PipeRunParams.run_mode`, stamped on every cogt assignment, serialized across the Temporal wire), so a DRY run on a Temporal backend dispatches the REAL `act_llm_gen_*` / extract / img-gen / search / render activities and the leaf mocks **inside** the activity — testing dispatch, scheduling, serialization, routing, and cross-worker propagation at zero AI cost, with no API keys and no storage IO (the img/extract DRY branches sit above the store step). `ContentGeneratorDry` is deleted: operators no longer swap generators (the base operator's dry path simply reuses the live path), and the in-process validation scopes pass an inline `ContentGenerator`. Object mocks are now schema-built on every backend — identical mock on direct and Temporal; classes with exotic format constraints should declare `examples`/`mock_format` (a re-validation failure surfaces as a typed `DryRunObjectFidelityError`, and a deterministic mock-build failure as the new non-retryable `DryRunMockBuildError`). The acceptance gate is Tier 17 of the `temporal-e2e-validate` skill (GREEN on all leaf families + keyless worker, RED-proven) plus a Mode-1 pytest.- **Unified dry run — `--mock-inference` removed (breaking).** There is now exactly one non-live run mode: `--dry-run`. The mock-inference mode is retired; its capability — non-zero synthetic usage so the cost report renders, used to validate cross-worker cost-report assembly cheaply — survives as an **internal** `is_mock_usage` sub-flag of DRY on `CogtRunParams` (replacing `is_mock_inference`; setting it on a LIVE run is a validation error). It is exposed on the Python surface (`PipelexRunner` / `execute_pipeline` / `prepare_pipe_job` via `is_mock_usage=...`) and as a hidden test-only CLI trigger, deliberately undocumented. Dry-run leaf coverage was already uniform across every operator, so `MockInferenceUnsupportedError` and its img-gen/extract/search guards are deleted; `MockInferenceObjectFidelityError` is renamed `DryRunObjectFidelityError` (mock-built objects are DRY-only now); the `mock_inference` usage sentinel is renamed `mock_usage`.- **Keyless boot (`needs_inference=False`) now forces every run to DRY instead of installing a mock generator.** Generator selection is purely backend-keyed (a keyless Temporal submitter still dispatches and mocks inside activities); the forced-DRY flag is consumed at `PipeRunParamsFactory.make_run_params` — the single writer of `run_mode` — so every execution entry point is covered, including the runtime bridge.+- **Temporal runs without graph events skip the guaranteed-empty trace-flush activity.** Post-fix, the workflow's in-memory trace buffer is populated only by inline (workflow-thread) emissions — and with the unified dry run mocking inside the leaf activities, the only inline source left is graph events when graph tracing is on. Every usage emission (LIVE, DRY, or mock-usage) happens activity-side through the per-process event log, so scheduling `act_flush_trace_events` with an empty payload was a wasted activity round-trip per workflow in the fan-out (history events, task-queue dispatch, worker slot — including every child workflow spawned by controllers). The schedule is now gated payload-purely on `emit_graph_events`, so the decision replays identically and the command stream stays a pure function of the payload.+- **The "runner-side usage emission engaged" log is now INFO and states the actual contract.** Since activity-side usage emissions always go through the per-process activity event log (by design, co-located or split deployment alike), the old per-process WARNING — which claimed the fallback was specific to split deployments — fired on every standard deployment with a false explanation. Renamed to `log_once_runner_fallback_engaged`, logged once per process at INFO with the writer id.### Removed@@ -14,6 +16,14 @@### Fixed+- **Resubmitting a `pipeline_run_id` after its run finished no longer fails permanently.** `PipelineManager` was the one per-run registry with no per-run removal: every run permanently stranded its key, so a second submission of the same `pipeline_run_id` against a long-lived server process raised `PipelineManagerAlreadyExistsError` — surfacing as an unrecoverable 500 from the hosted runner API until process restart. The registry entry is now freed on every exit path: `execute_pipeline`'s `finally` removes it after each run (success or failure), and `pipeline_run_setup` removes its own registration when setup fails after registering (the caller never learns the id on that path, so setup must self-clean). Serial resubmission of a completed/failed run id now succeeds; only genuinely concurrent same-id runs still collide — deliberately, because the collision raise fires before `open_tracer` and shields the live direct-mode tracer (keyed by the caller-suppliable `pipeline_run_id`) from `open_tracer`'s stale-key pop-and-replace healing. The hosted API maps the remaining concurrent-duplicate case to 409 Conflict (pipelex-api change).+- **The temporalio extra is no longer imported on every pipelex boot.** `reporting_manager.py` (imported by `pipelex.pipelex` on every `Pipelex.make()` / CLI invocation) eagerly imported `temporalio.activity` at module level to detect in-activity usage emissions — putting the entire temporalio import (Rust bridge, protobuf, ~130ms measured) on every boot's critical path wherever `pipelex[temporal]` is installed, including processes that never touch Temporal, and silently breaking the runtime bridge's documented lazy-import contract. The gate now sniffs `sys.modules` instead: a process that never imported `temporalio.activity` cannot be inside a Temporal activity (the activity context is set by temporalio's own machinery, which requires the module). Worker processes are unaffected. A subprocess boot-cost guard pins the lazy-import contract.++- **Per-run Temporal worker-local state is keyed by run id, not workflow id.** The per-workflow library (`wf_{run_id}`), the graph tracer key, and the report-delegate event-log context key now use `workflow.info().run_id` — replay-stable but unique per run — instead of the workflow id. Workflow ids are reused across workflow-level `retry_policy` attempts, Temporal reset, and resubmission of the same `pipeline_run_id` (workflow ids derive deterministically from it), so workflow-id keying let a closed predecessor run's late eviction cleanup tear down a live successor run's library, tracer, and usage context on the same worker — failing the successor inline. With run-id keying such cross-run collisions are structurally impossible; `open_fresh_library` remains the same-run self-heal. `LibraryManager.teardown` also now forgets a library entry even if the library's own teardown raises (pop-first + `finally`), so a raising teardown can never turn into permanent worker-local poison, and the stale-entry healing path is tolerant of concurrent teardown.+- **An evicted Temporal workflow no longer poisons same-worker reruns through a leaked per-workflow library.** `WfPipeRouter`'s worker-local cleanup (per-workflow library + crate fingerprint, report-delegate event-log context) sat in the `finally` block AFTER the awaited `act_flush_trace_events` activity. An eviction-time interruption raised at that await (`_WorkflowBeingEvictedError` is a `BaseException`) escaped the `except Exception` around the flush and aborted the rest of the `finally`, leaking the library and its crate fingerprint in the worker-local `LibraryManager` under the deterministic `wf_{workflow_id}`. On a same-worker replay or re-run of that workflow id, `open_library` returned the stale library, a fresh `ClassRegistry` was attached, and `load_from_crate` was fingerprint-skipped — so the crate's dynamic classes never landed in the new registry and inline hydration failed where history recorded success: command-stream divergence, and with dynamic-class results on a single-worker queue, a persistent silent hang. Two complementary fixes: `WfPipeRouter` now opens its per-workflow library via the new `LibraryManager.open_fresh_library`, which force-tears-down any pre-existing library under the id (a leftover there can only be an interrupted predecessor), and the workflow `finally` now runs all worker-local cleanup BEFORE the awaited flush activity, which moved last — an eviction can only skip the flush itself, never the teardown. Guarded by an integration test that installs the leaked predecessor state and re-runs the workflow under the same id.+- **Costs-only Temporal runs no longer hang on a routine sticky-cache eviction.** In costs-only tracing mode (`emit_usage_events=True, emit_graph_events=False`), the workflow's in-memory trace buffer was populated exclusively by co-located activity completions writing cross-thread through the process-global `ReportingManager`, and the `act_flush_trace_events` schedule was gated on that buffer being non-empty. Activities don't re-execute on replay, so after a sticky-cache eviction the recreated buffer stayed empty, the recorded flush command was never re-emitted, and Temporal raised `[TMPRL1100]` — a silently hung workflow, on a single worker, with no config drift. Two changes make the command stream a pure function of payload + history: the flush-activity schedule is now gated purely on the payload field `emit_graph_events` (see the Changed entry above; runs without graph events skip the guaranteed-empty flush entirely) — and activity-side usage emissions always go through the per-process runner-fallback event log — the same path used when activities run on a remote worker — instead of the workflow's in-sandbox buffer, even when the activity is co-located with the workflow worker. Guarded by a replay regression test that records a costs-only history and replays it through `temporalio.worker.Replayer`.+- **Per-workflow tracing setup no longer depends on worker-local leak state.** `GraphTracerManager.open_tracer` raised `ValueError` when a stale tracer key was left behind by a prior interrupted execution on the same worker (e.g. a workflow evicted before its cleanup ran), and `WfPipeRouter` papered over that with a best-effort `except Exception` that silently disabled tracing — dropping the recorded flush command on replay and diverging the trace context carried by all downstream commands. `open_tracer` now pops and replaces a stale tracer (tearing it down, with a WARNING), and the catch-all around tracing setup is gone: the setup block is pure in-memory state, so any failure is a real bug that surfaces deterministically.+- **Temporal workflows no longer hang when worker tracing config differs across replays.** `WfPipeRouter` decided whether to set up per-workflow tracing — and therefore whether to schedule `act_flush_trace_events` — by reading `tracing_config.is_enabled` from worker-local config inside the workflow body. Replaying a workflow on a worker whose tracing config differed from the recorder's (worker restart, rolling deploy, scale-in) produced a divergent command stream, and Temporal's `[TMPRL1100]` nondeterminism error surfaced as a silently hung workflow stuck in workflow-task retries. The workflow now gates tracing setup purely on the presence of `trace_context` in the payload — the submitter's decision, recorded in history — and the worker-local `is_enabled` check happens only inside the flush activity, where reading local config is deterministic-safe (it already no-ops there when tracing is disabled). Side benefit: per-workflow graph assembly no longer requires the worker to have a tracing backend enabled. Guarded by a replay regression test that flips the worker's tracing config mid-workflow.- **`dry_run_pipeline` now owns its graph transport — graph generation no longer depends on the host's `tracing_config`.** The function requests a graph explicitly (`generate_graph=True`) but relied on the host's configured tracing backend as the emit/assemble channel, so a host with `tracing_config.is_enabled = false` (e.g. pipelex-api's `/validate` in direct mode, where the web-app's Dry Run button lands) always got "Pipeline execution did not produce a graph spec" and a response without `graph_spec`. The run now traces through a scoped `InMemoryEventLog` — the graph comes back under any tracing config, and validation dry-runs stop writing NDJSON/DynamoDB trace events as a side effect when tracing IS enabled. The no-graph contract violation is now a typed `DryRunGraphNotProducedError` (was a bare `PipelexError`, ambiguous in host logs that record only the exception type) — raised by both `dry_run_pipeline` and `dry_run_pipe_in_process`.- **tz-aware payloads no longer hang Temporal callers on hosts without a timezone database — kajson bumped to 0.7.0.** kajson ≤ 0.6.0 decoded timezone-aware datetimes via `ZoneInfo(...)`, which needs the IANA tz database: system tz files or the `tzdata` package. On hosts with neither (slim containers, uv-managed standalone Pythons — including the CI test runner, where `tzdata` only arrived transitively via pandas on Python < 3.11), decoding any aware-datetime payload crossing the Temporal boundary (e.g. the `GraphSpec` on `act_dry_validate`'s result) raised `ZoneInfoNotFoundError` *inside the workflow task* — which Temporal retries forever, so the submitter hung instead of failing. Fixed upstream in kajson 0.7.0: its self-sufficient timezone wire format carries the UTC offset (decoding degrades gracefully to a fixed offset instead of raising, and plain UTC decodes with zero tz-database dependence), and kajson now declares `tzdata` as its own runtime dependency, so a tz database is always available for named zones.diff --git a/TODOS.md b/TODOS.mdindex 10218c19a..d91bd6a86 100644--- a/TODOS.md+++ b/TODOS.md@@ -1,259 +1,55 @@-# Dry-run + validation as one in-process Temporal activity (in-memory tracing)+# Burn down the nondeterminism review follow-ups-> **Branch:** `feature/Dry-run-as-temporal-activity`. This file is the **executable plan + progress tracker** for **Mode 1** (in-memory activity). Update the checkboxes and the checkpoint handoff blocks as you go.->-> **Read first, in order:** (1) [`wip/dry-run-refactor/dry-run-modes-master-plan.md`](wip/dry-run-refactor/dry-run-modes-master-plan.md) — the big picture (both dry-run modes, what they share); (2) [`wip/dry-run-refactor/followup-temporal-validation-activity.md`](wip/dry-run-refactor/followup-temporal-validation-activity.md) — this mode's design / why; (3) the "Cold-start context" section just below. **The other mode (full-distribution, leaf-mocks) is out of scope here** — return to the master plan for it after Checkpoint F.->-> **Goal:** when Temporal is enabled, the API dispatches the whole `/validate` job — validation sweep **+** graph-producing dry-run — to a worker as **one activity** that runs entirely **in-process** and traces the graph in an **in-memory** event log (no DynamoDB round-trip, no NDJSON files, no usage/cost reporting). Direct mode stays in-process and unchanged.+> **Cold-start brief for a new session.** Mission: go through every item in [`wip/distributed-execution/nondeterminism-fix-review-follow-ups.md`](wip/distributed-execution/nondeterminism-fix-review-follow-ups.md), re-verify each against the current code, **fix the obvious ones**, **dismiss the wrong ones** (record the dismissal + reason in the follow-ups doc, don't silently delete), and **surface the judgment-call ones** in a new `.md` for discussion with the user. Do not start new feature work.----+## Context (read first, in this order)-## Cold-start context (read this first in a fresh session)+1. `wip/distributed-execution/workflow-nondeterminism-audit.md` — the verified audit. H1, H2, M1 are FIXED; the MEDIUM/LOW drift table is open but **out of scope for this session** (it has its own fix-order, steps 3–5).+2. `wip/distributed-execution/nondeterminism-fix-review-follow-ups.md` — the work list. Items came from two recall-biased multi-agent review rounds: round 1 reviewed the committed H1/M1 fixes, round 2 reviewed the H2 fix. Every item was verified once by a review agent, but **re-verify before acting** — line numbers drift and some verdicts may not survive contact with the current code.-The five facts that shape the whole plan — verify by symbol before relying on them, but they were true at branch start:+Branch state when this brief was written: `fix/Config`, H1/M1 fixes committed; the H2 fix (new `LibraryManager.open_fresh_library` + eviction-safe `finally` reorder in `WfPipeRouter`) was uncommitted in the working tree. **Check `git status` first** — the user may have committed it since. Full suite was green (`make agent-test`) and `make agent-check` clean at that point.-1. **The validation sweep is already Temporal-safe and in-process.** `BundleValidator.validate_pipes` wraps its per-pipe loop in `scoped_pipe_router(self._pipe_router)` (PR #976) and runs a locally-constructed `PipeRun` — never `get_pipe_run()`. DRY mocks inline at the pipe level (`ContentGeneratorDry`). So the sweep half is ready to wrap; **Part B is NOT a prerequisite** (it's the opposite cell — a top-level DRY run that *should* hit the worker path).+## Triage protocol-2. **The graph dry-run is the real work.** `pipelex-api`'s `/validate` (`../pipelex-api/api/routes/pipelex/validate.py`) calls `validate_bundle()` (sweep) **and** `dry_run_pipeline()` (graph). `dry_run_pipeline` (`pipelex/pipe_run/dry_run_pipeline.py`) goes through `PipelexRunner.execute_pipeline` → `get_pipe_run()` → `TemporalPipeRun` under a Temporal hub → dispatches a top-level workflow to the worker, which traces to `temporal_dynamodb` and assembles the `GraphSpec`. That works today (single dispatch, no collision). **Do not move it in-process *in the API*** — the API runs tracing-off; that would break graph generation. The new design moves it in-process *inside the activity, on the worker*.+For each item: re-verify → then one of:-3. **The two-instance problem (the crux).** Emit and assemble build *separate* event logs from config: write side `pipeline_run_setup.py` (`make_event_log(tracing_config)`, ~line 200, handed to the tracer); read side `pipelex/pipe_run/tracing_assembly.py` `assemble_tracing` (`make_event_log(tracing_config)` again, ~line 95, then `read_events`), triggered from `PipeRun.run`'s `finally` (`pipelex/pipe_run/pipe_run.py` ~line 76). NDJSON/DynamoDB bridge the two instances via an external store; a plain in-memory log can't. **Fix = a shared instance scoped to the run.**+- **FIX** — obvious, low-risk, verdict holds: implement, with tests where the repo's TDD norm applies (red first for behavior changes; no tests for trivial comment/message fixes).+- **DISMISS** — wrong, stale, or unreachable: edit the follow-ups doc to mark it dismissed with the evidence (file:line), so it doesn't get re-chased.+- **SURFACE** — real but a judgment call (design tradeoff, behavior change with ripple, multiple defensible shapes): write it up in a new `wip/distributed-execution/nondeterminism-follow-ups-decisions-needed.md` with the options and a recommendation, and leave the code alone.-4. **`InMemoryEventLog` already exists** (`pipelex/tracing/in_memory_event_log.py`, implements `EventLogProtocol`, used in unit tests). We don't write it — we *wire* it via a scope.+## The items, with expected triage (verify anyway)-5. **The scope pattern to mirror** lives in `pipelex/hub.py`: `scoped_current_library` (~line 518) and `scoped_pipe_router` (~line 635) — `ContextVar` + `@contextmanager` + a `get_*` accessor that prefers the override. The in-process "force DIRECT under a Temporal worker" primitive to reuse is `pipelex/runtime_bridge/bridge.py` `_run_direct` (~line 318) / `run_pipe_via_bridge` (~line 111, already accepts a `trace_context` honored in DIRECT mode).+Check the box when the item reaches its terminal state (FIXED / DISMISSED / SURFACED), and append the outcome to the line — e.g. `→ FIXED (commit abc123)`, `→ DISMISSED (unreachable, see follow-ups doc)`, `→ SURFACED (decisions doc §2)`.-### File map+- [x] **0 — PRIORITY: rekey per-run worker-local state by `run_id`, not `workflow_id`** → FIXED. `wf_run_id` keys library/tracer/event-log context (and event/node-id stamps — one identity); `open_fresh_library` made pop-based (atomic, concurrent-teardown tolerant). New RED-first guard `test_wf_pipe_router_run_scoped_state_keying.py`; eviction-leak test converted to start-then-install under `wf_{run_id}`; `temporal-integration.md` keying story updated.+- [x] **0bis — cancellation test pinning the `finally`-ordering invariant** → FIXED, premise corrected: cancellation at an activity await surfaces as `ActivityError` (an `Exception`, swallowed by the best-effort except) — it CANNOT abort the finally. The guard instead forces eviction via worker shutdown: `test_wf_pipe_router_eviction_cleanup_ordering.py`, verified RED against a reordered finally.+- [x] **1 — stale runner-fallback warning message** → FIXED. Renamed `log_once_runner_fallback_engaged`, INFO level, message states the post-H1 contract; module docstring + the two unit tests updated.+- [x] **2 — dead `_event_log_contexts.clear()`** → FIXED. Deleted in `tracing/helpers.py` AND the same dead clear in `test_split_worker_real_inference_cost.py`; docstrings rewritten.+- [x] **3 — test-scaffolding dedup** → FIXED (two sub-points DISMISSED with evidence in the follow-ups doc: the enable-tracing fixture has only one copy; the extract-pages scan needs `(name, activity_id)` pairs). Shared: `make_synthetic_usage_llm_job`, `act_flush_noop`, `scheduled_activity_names` in `tracing/helpers.py`; `make_prepared_greeting_job` in `library_crate/helpers.py`.+- [x] **4 — `open_tracer` stale-eviction → delegate to `close_tracer`** → FIXED with the membership-gated warning.+- [x] **5 — skip the guaranteed-empty flush activity in costs-only LIVE** → FIXED. Determinism argument re-derived (gate inputs payload-pure; costs-only LIVE buffer provably empty; Temporal unshipped). `schedule_flush` computed in setup, consumed in finally; costs-only test arm now asserts flush ABSENT + clean replay.+- [x] **6 — trim redundant null conjuncts in the `finally`** → FIXED (folded into 5/9 rewrite; sentinels kept).+- [x] **8 — document `set_event_log` overwrite as load-bearing** → FIXED (contract comment naming the three idioms).+- [x] **9 — single-block `finally` simplification** → FIXED (sentinel + TYPE_CHECKING import gone; temporal suite green).+- [x] **10 — shared scoped-library helper for the two uuid-keyed `runtime_bridge` copies** → FIXED. Sync `scoped_library_for_crate` in `runtime_bridge/primitives/scoped_library.py`; `wf_pipe_router` left bespoke.+- [x] **11 — `LibraryManager.teardown`: forget the entry even if `library.teardown()` raises** → FIXED (`_pop_and_teardown_library`, RED-first unit tests).+- [x] **7 — graph events lack an H1-style guard** (design note) → SURFACED (decisions doc §1, with options + recommendation; ties into T3).-| Concern | File |-|---|---|-| Event-log protocol / impls | `pipelex/tracing/{event_log_protocol,in_memory_event_log,ndjson_event_log,dynamodb_event_log,buffering_event_log}.py` |-| Event-log factory | `pipelex/tracing/event_log_factory.py` (`make_event_log`) |-| Scope pattern to mirror | `pipelex/hub.py` (`scoped_current_library`, `scoped_pipe_router`, `get_pipe_router`) |-| Write-side wiring | `pipelex/pipeline/pipeline_run_setup.py` (~200) |-| Read-side wiring | `pipelex/pipe_run/tracing_assembly.py` (~95), called from `pipelex/pipe_run/pipe_run.py` (~76) |-| Graph dry-run (to make in-process/in-memory) | `pipelex/pipe_run/dry_run_pipeline.py` |-| In-process run primitive to reuse | `pipelex/runtime_bridge/bridge.py` (`_run_direct`, `run_pipe_via_bridge`, `build_pipe_job_from_input`) |-| Validation sweep | `pipelex/pipeline/bundle_validator.py` |-| Tracer lifecycle | `pipelex/graph/graph_tracer_manager.py` (`open_tracer`, `close_tracer`) |-| Activity error boundary | `pipelex/temporal/tprl/activity_error_boundary.py` (`convert_pipelex_errors`) |-| Existing activities + registration | `pipelex/temporal/tprl_pipe/act_*.py`; `pipelex/temporal/tasks.py` (`PackName.PIPE` activity_list) |-| API route (cross-repo) | `../pipelex-api/api/routes/pipelex/validate.py` |+## Constraints and gotchas----+- **Do not modify** `tests/integration/pipelex/temporal/test_wf_pipe_router_eviction_library_leak.py` to make anything pass — it's the H2 regression guard. Extending it (item 0's successor-direction case) is fine.+- Repo error-handling rules apply (`.claude/rules/python-standards.md`): no new bare `except Exception`, `try/finally` for required cleanup.+- The `finally` block in `wf_pipe_router.py` is the determinism boundary: any edit there must keep ALL synchronous worker-local cleanup before the single flush await, and the flush gate payload-pure. Re-read the block's comments before touching it.+- Items 5, 6, 9 (and 0 partially) all edit that same block — do them as one coherent change, not three diffs.+- `worker_scopes` live only in base `pipelex.toml` — a recurring false-positive bot finding; don't "fix" it.-## Pre-flight — ALL DECIDED (2026-06-09)+## Acceptance-All decisions are resolved up front so a cold-start session runs autonomously through Checkpoint F. There is **no remaining human gate** — the dispatch question that used to gate this was resolved in favor of the wrapper workflow (no `temporalio` bump).+- [x] Every checklist item above is checked with its outcome appended (**FIXED** / **DISMISSED (reason)** / **SURFACED → decisions doc**), and the follow-ups doc reflects the same status per item.+- [x] `wip/distributed-execution/nondeterminism-follow-ups-decisions-needed.md` exists iff anything was surfaced, and the track `README.md` links it.+- [x] Targeted suites green along the way (temporal integration + unit libraries/reporting/tracing/graph + runtime_bridge).+- [x] `make agent-check` clean at the end.+- [x] Full `make agent-test` green at the end.+- [x] CHANGELOG `[Unreleased]` updated for any behavior change; `docs/under-the-hood/temporal-integration.md` updated if item 0 changes the keying story.-- [x] **(DECIDED) In-memory exposure = scope-only.** Add `scoped_event_log` only; do **not** add a `TracingBackend.IN_MEMORY` config enum (insufficient alone — the two-instance problem — and it would change the global default instead of being opt-in per call).-- [x] ~~**(DECIDED) Graph dry-run shape = bridge DIRECT reuse.**~~ ⚠️ **REVISED BY ENG REVIEW (2026-06-09) — see GSTACK REVIEW REPORT, decision D-C1.** The bridge **cannot** carry `run_mode=DRY` + `mock_inputs`: `PipelexPipeRunInput` has no run-mode/mock-input field and `build_pipe_job_from_input` (`bridge.py:216`) calls `make_run_params()` → `PipeRunMode.LIVE` with no mock-input synthesis. Running the main pipe through `run_pipe_via_bridge` DIRECT would produce a **LIVE run with no inputs** that fails before reaching the graph. **New shape: use the `prepare_pipe_job` seam** (`execution_seams.py:199`) with `pipe_run_mode=PipeRunMode.DRY` + `execution_config(mock_inputs=True, generate_graph=True)` (what `dry_run_pipeline.py` already does), running a **local `PipeRun`** under `scoped_event_log` + `scoped_pipe_router` + `scoped_content_generator`. Still do **not** refactor `PipelexRunner`; still no bridge wire-contract change.-- [x] **(DECIDED) One activity (status + graph).** One activity returns `{status map, GraphSpec, signature_check_error}`; the graph dry-run is **best-effort inside the activity** — a graph failure still returns a successful validation with `graph=None`, preserving today's `/validate` semantics. (Not two separate activities.)-- [x] **(DECIDED) Dispatch = one-step wrapper workflow (no `temporalio` bump).** The API dispatches a one-step workflow that runs the single activity and returns — works on the current `temporalio` (`1.23.0`), no SDK bump, no Temporal Cloud/server verification, **no hard gate**. Functionally identical to the caller. A true standalone activity is a **later optional optimization** (Phase G0, deferred — not on this branch's critical path).-- [x] **(DECIDED: BUILD THE SCOPE NOW) Force the inline content generator.** Today the in-process sweep/dry-run is safe because DRY mocks at the **pipe** level (`ContentGeneratorDry` inline) and never calls `get_content_generator()`. But under a Temporal-enabled hub `get_content_generator()` is `ContentGeneratorInWorkflow` **globally** (boot-time, `pipelex.py:370-385`). The moment **Part B** ([`wip/dry-run-refactor/followup-leaf-run-mode-mock.md`](wip/dry-run-refactor/followup-leaf-run-mode-mock.md)) relocates the DRY mock to the leaf, this activity's leaf would dispatch `act_llm_gen_*` and break the in-process guarantee. **DECISION (2026-06-09): add the inline content-generator scope now** (`scoped_content_generator`, mirroring `scoped_pipe_router`) in Phases 2–3, so the activity is correct regardless of Part-B ordering and the shared seam is built once. Rejected the "rely on the pipe-level mock, defer the scope" alternative as fragile (would silently regress when Part B lands). The Phase-2/3 zero-dispatch test must simulate the leaf-level mock (not only today's pipe-level mock) so the guard isn't a no-op until then.- - This scope is **orthogonal to** the req-1 "full-distribution leaf-mock" mode (Mode C) — that mode deliberately *does* dispatch and mocks inside the activity. The two coexist (proof: `is_mock_inference` already ships the LLM slice of Mode C alongside the DRY path). The scope is per-call/ContextVar, so it doesn't disturb Mode C. See [`wip/dry-run-refactor/dry-run-modes-master-plan.md`](wip/dry-run-refactor/dry-run-modes-master-plan.md).--> ### ✅ Former HUMAN GATE — RESOLVED (2026-06-09): wrapper workflow, no bump->-> The `temporalio` standalone-activity question that used to gate dispatch is **resolved**: we use the **one-step wrapper workflow** on the current `temporalio` (`1.23.0`). No SDK bump, no Temporal Cloud/server verification, nothing to ask. The cold-start session proceeds through all phases without stopping for human input. (Standalone activity is a later optional optimization — Phase G0, deferred.)-------## Status at a glance--| Phase | Title | Status |-|---|---|---|-| 1 | `scoped_event_log` + shared in-memory tracing (in-repo, no Temporal) | ☑ done |-| | **⛔ CHECKPOINT 1 — in-memory tracing verified in direct mode** | ☑ done (suite green, committed) |-| 2 | In-process, in-memory graph dry-run safe under a Temporal hub (+ `scoped_content_generator`) | ☑ done |-| | **⛔ CHECKPOINT 2 — graph dry-run verified under a Temporal hub** | ☑ done (suite green, committed) |-| 3 | The `act_dry_validate` activity + wrapper-workflow dispatch + worker registration + isolation test | ☑ done |-| | **⛔ CHECKPOINT 3 — activity registered + isolation-tested** | ☑ done (suite green, Tier 2d GREEN+RED, committed) |-| 4 | API dispatch (cross-repo `pipelex-api`) | ☑ done |-| | **⛔ CHECKPOINT F — all requirements met** | ☑ done (both suites green, Tier 2d both arms GREEN, committed) |-| G0 | *(optional, deferred)* `temporalio` bump → true standalone activity | ☐ later |--Status legend: ☐ not started · ◐ in progress · ☑ done. **No human gate remains** (dispatch = wrapper workflow).-------## Phase 1 — `scoped_event_log` + shared in-memory tracing--Pure in-repo capability, no Temporal, no cross-repo. Fully testable in direct mode. **TDD: write the failing tests first.**--- [x] *Tests first* (`tests/unit/pipelex/tracing/test_scoped_event_log.py` + `tests/integration/pipelex/pipeline/test_scoped_in_memory_tracing.py`): assert that running a dry-run-with-graph under `with scoped_event_log(InMemoryEventLog())` (a) produces a non-empty, correct `GraphSpec`, (b) writes **no** NDJSON file and touches **no** configured backend, and (c) emit and assemble hit the **same** instance. Concurrency/nesting tests included (unit: ContextVar isolation + nesting + exception restore; integration: two concurrent scoped dry-runs stay isolated).-- [x] Added to `pipelex/hub.py` (mirroring `scoped_pipe_router`): `_event_log_override: ContextVar["EventLogProtocol | None"]`, `scoped_event_log(event_log)` `@contextmanager` (save/set/restore), and accessor `get_event_log_override()`.-- [x] Write side prefers the override: `pipeline_run_setup.py` (`event_log = get_event_log_override()`, factory only when `None` and `is_enabled`). D1: a set override implies tracing-enabled.-- [x] Read side prefers the override: `tracing_assembly.py::assemble_tracing` — the `is_enabled` early-return is bypassed when an override is set (D1); the override is read **without** `close()` (the scope owner keeps the instance's lifecycle).-- [x] `EventLogProtocol` surface confirmed sufficient — no protocol change.-- [x] `make agent-check` clean · new tracing tests green (7/7).--> ### ⛔ CHECKPOINT 1 — after Phase 1 — **MANDATORY STOP**->-> A self-contained, separately reviewable in-repo capability. Natural place to split sessions and/or land alone.->-> **Verify:** `make agent-check` clean · `make agent-test` green · the new in-memory-tracing tests pass (same-instance emit+read, zero file/backend, concurrency-safe) · commit.->-> **Handoff (filled in, 2026-06-09):**->-> - **Scope API (`pipelex/hub.py`, right after `get_pipe_router`):** `scoped_event_log(event_log: EventLogProtocol)` — `@contextmanager`, ContextVar save/set/restore, mirrors `scoped_pipe_router`. Accessor: `get_event_log_override() -> EventLogProtocol | None`. ContextVar: `_event_log_override`.-> - **Write-side edit:** `pipeline_run_setup.py::pipeline_run_setup` — `event_log = get_event_log_override()`; falls back to `make_event_log(tracing_config)` only when no override AND `tracing_config.is_enabled`.-> - **Read-side edit:** `tracing_assembly.py::assemble_tracing` — early-return is now `if event_log_override is None and not tracing_config.is_enabled`; with an override, `read_events` is called on it directly and it is NOT `close()`d (scope owner keeps lifecycle; the no-override branch keeps its `make_event_log` + `close()` shape — D2 symmetric primitive).-> - **`is_enabled` vs override (D1):** a set override **implies tracing-enabled** at both guards; regression-tested (`test_override_implies_enabled_when_tracing_disabled`).-> - **`EventLogProtocol`:** unchanged.-> - **Other `make_event_log` call sites left alone (intentional):** `tracing/activity_event_log.py` (Temporal worker per-process usage log — Phase-3 activity never reaches it) and `runtime_bridge/primitives/trace_flush.py` (bridge DIRECT flush — off-path since D-C1 dropped the bridge).->-> **Next: Phase 2.**-------## Phase 2 — In-process, in-memory graph dry-run safe under a Temporal hub--Make the graph-producing dry-run run fully in-process even under a Temporal-enabled hub, tracing into the Phase-1 in-memory log. Reuse the bridge DIRECT primitive (it already forces in-process via `scoped_pipe_router` + a local `PipeRun`, and `run_pipe_via_bridge` already honors a `trace_context` in DIRECT mode).--- [x] *Tests first* (`tests/integration/pipelex/temporal/test_dry_run_graph_in_process.py`): under the suite's **Temporal-enabled** hub (real `TemporalPipeRouter` + `ContentGeneratorInWorkflow` as hub defaults, asserted as preconditions), the in-process graph dry-run (a) returns a correct `GraphSpec` covering the full `temporal_parallel` controller topology, (b) dispatches **zero** workflows (spy `WorkflowExecutor.execute_workflow`), and (c) touches no file/DDB transport (`make_event_log` forbidden via patch).-- [x] Added `dry_run_pipe_in_process(pipe, *, library_id)` in `pipelex/pipe_run/dry_run_pipeline.py`: opens a `GraphTracerManager` tracer against an `InMemoryEventLog`, runs via `prepare_pipe_job` (DRY + mock_inputs + generate_graph) + a local `PipeRun` under the three scopes (`scoped_event_log` + `scoped_pipe_router` + `scoped_content_generator`) — not the bridge, not `PipelexRunner` (D-C1). `GraphSpec` rides back on `PipeOutput`. Caller owns the library lifecycle (mirrors `validate_pipes`, ready for D4).-- [x] Added `scoped_content_generator(content_generator)` in `hub.py` (ContextVar + contextmanager + `get_content_generator()` prefers the override). Leaf-level-mock simulation test (`test_leaf_level_mock_stays_in_process`) forces DRY through `get_content_generator()` and stays zero-dispatch; **RED-proven**: with the scope removed, the leaf reached `ContentGeneratorInWorkflow` and failed with `_NotInWorkflowEventLoopError`.-- [x] Tracer-key alignment (D-C7): tracer opened at `graph_id=pipeline_run_id` and closed by `pipeline_run_id` in `PipeRun.run`'s finally — aligned by construction, no divergent `tracer_key` parameter exposed; pinned by the non-empty-graph assertions (a divergent key would assemble an empty graph).-- [x] `dry_run_pipeline`'s existing (worker-workflow + DynamoDB) path left intact — the API still uses it until Phase 4.-- [x] `make agent-check` clean · zero-dispatch / in-memory-graph tests green.--> ### ⛔ CHECKPOINT 2 — after Phase 2 — **MANDATORY STOP**->-> **Verify:** under a Temporal-enabled hub the in-process graph dry-run yields a `GraphSpec` with **zero** nested dispatch and **zero** file/DDB I/O · `make agent-test` green · commit.->-> **Handoff (filled in, 2026-06-09):**->-> - **Entry point:** `dry_run_pipe_in_process(pipe: PipeAbstract, *, library_id: str) -> GraphSpec` in `pipelex/pipe_run/dry_run_pipeline.py`. Takes a resolved pipe + an already-open library (caller owns the library lifecycle — fits D4's load-once activity shape). Raises `PipelexError` when the run fails or yields no graph.-> - **Tracer lifecycle:** `open_tracer(graph_id=pipeline_run_id, event_log=InMemoryEventLog, workflow_id="direct", emit_graph_events=True, emit_usage_events=False)` before the run; `trace_context` threaded via `prepare_pipe_job(trace_context=...)` (no bridge involved — D-C1); closed by `pipeline_run_id` inside `PipeRun.run`'s `finally` (which also assembles from the scoped log onto `PipeOutput`); a safety-net `close_tracer(pipeline_run_id)` in the entry's own `finally` covers pre-run failures (idempotent pop).-> - **Tracer-key alignment:** `graph_id == pipeline_run_id == close key`, by construction; no `tracer_key` parameter exposed. A unique `dry_run_graph_<uuid>` run id is minted per call.-> - **Scopes:** `scoped_event_log(event_log)` + `scoped_pipe_router(PipeRouter(observer=ObserverNoOp()))` + `scoped_content_generator(ContentGeneratorDry())` wrap `prepare_pipe_job` + `pipe_run.run`. `scoped_content_generator` is new in `hub.py`; `get_content_generator()` now prefers the ContextVar override.-> - **Divergence from plan:** none of substance. No `PipelineManager.add_new_pipeline` registration and no report-delegate usage-event wiring (mirrors the sweep, not `pipeline_run_setup`) — deliberate: no usage/cost in this mode.->-> **Next: Phase 3.**-------## Phase 3 — The `act_dry_validate` activity + wrapper-workflow dispatch + registration + isolation test--Wrap the sweep + the in-memory graph dry-run in one activity, dispatched via a one-step wrapper workflow (DECIDED — no `temporalio` bump). No gate; runs straight through.--- [x] Activity defined (`pipelex/temporal/tprl_pipe/act_dry_validate.py`): body runs **`validate_bundle`** (the exact function the direct-mode route calls — sweep → status map AND the categorized `ValidateBundleError` error cascade, so both backends surface the identical 422 contract) **and** `dry_run_pipe_in_process` (→ `GraphSpec`, against the same library `validate_bundle` left loaded+current — D4), teardown once in the activity's `finally`. The scopes live in the callees (the Phase-2 entry carries all three; the sweep carries `scoped_pipe_router` + `scoped_content_generator` itself). **Graph best-effort per D5 (widened to cross-backend parity in the post-implementation review, R-D1):** catches `PipelexError` + `ValidationError`/`FactoryException` — the same catch `_classify_pipe` and the direct route use, and pipe resolution sits inside it — → `graph=None`; non-Pipelex bugs propagate. Decorated with `convert_pipelex_errors`.-- [x] Input `DryValidateArg` (`mthds_contents`/`allow_signatures`/`pipe_code` — no `bundle_uris` and, post-review R-D3, no `library_dirs`: no caller passes them and `library_dirs` was an unconstrained worker-filesystem read surface); output `DryValidateResult` (`dry_run_outputs: dict[str, DryRunOutput]` + `graph_spec: GraphSpec | None`). Per D3: no `signature_check_error` field — signature refusals raise (wrapped as `ValidateBundleError`, same as direct mode).-- [x] One-step wrapper workflow `wf_dry_validate.py` (`WfDryValidate`) runs the single activity with explicit `start_to_close_timeout=5min` + `RetryPolicy(maximum_attempts=2, non_retryable_error_types=["ValidateBundleError"])` (D-C5; post-review the list is just the one name that actually crosses — `validate_bundle` wraps the other deterministic failures into `ValidateBundleError` and the graph arm degrades instead of raising). Submitter helper `dry_validate_dispatch.py::dispatch_dry_validate` is the one-round-trip call the API (Phase 4) and the Tier-2d script share.-- [x] Registered both in `pipelex/temporal/tasks.py` (`PackName.PIPE`).-- [x] *Isolation tests* (`tests/integration/pipelex/temporal/test_dry_validate_activity_in_memory.py`): real worker against the in-process server; status map + GraphSpec correct in one round-trip; **zero nested dispatch asserted on the Temporal history** (exactly one `ActivityTaskScheduled`, zero child workflows — D6, stronger than the wrapper spy); in-memory tracing (`make_event_log` forbidden, no NDJSON); best-effort graph (`graph=None`, validation OK) + bug-propagates arm (D5); `SignaturesNotAllowedError` crosses as structured `ErrorReport` with offender + signature refs intact (T3) and lenient-mode counterpart; concurrent invocations isolated.-- [x] **Tier 2d added** to `.claude/skills/temporal-e2e-validate/` (scenario in `references/mode-2-tiers.md` after Tier 2c, submitter script `scripts/submit_dry_validate.py`, Step-7 master-table row) **and RUN against a real 3-process setup** (server + split router/runner workers): GREEN (exit 0, 5/5 SUCCESS, GraphSpec 5 nodes/6 edges, server history = 1 activity + 0 children, no NDJSON partition, activity ran on the runner process), best-effort sub-case (no main_pipe → `GRAPH: None`, exit 0), concurrency (2 parallel dispatches, distinct graph_ids, no cross-contamination), RED (dropping `scoped_event_log` → exit 1 `did not produce a graph spec` — the two-instance regression fails loudly; doc updated to the observed behavior; the content-generator RED arm is the Mode-1 leaf-mock test from Phase 2), restored + GREEN re-confirmed.-- [x] `make agent-check` clean · activity isolation tests green (7/7) · Tier 2d GREEN+RED.--> ### ⛔ CHECKPOINT 3 — after Phase 3 — **MANDATORY STOP**->-> **Verify:** activity + wrapper workflow registered + isolation-tested (status map + GraphSpec correct, zero nested dispatch, in-memory tracing, best-effort graph, structured error on validation failure) · **Tier 2d (activity arm) GREEN and RED-proven** in `temporal-e2e-validate` · `make agent-test` green · Temporal e2e green · commit.->-> **Handoff (filled in, 2026-06-09):**->-> - **Activity:** `act_dry_validate(arg: DryValidateArg) -> DryValidateResult` in `pipelex/temporal/tprl_pipe/act_dry_validate.py`, `@activity.defn(name="act_dry_validate")` over `@convert_pipelex_errors`. `DryValidateArg{mthds_contents, allow_signatures, pipe_code}` (no `library_dirs` — dropped post-review, R-D3); `DryValidateResult{dry_run_outputs: dict[str, DryRunOutput], graph_spec: GraphSpec | None}`. **The sweep half is `validate_bundle` itself** (not `validate_current_library`) — chosen for cross-backend 422 parity: failures cross as the categorized `ValidateBundleError` (`error_domain=input`, refs in the caller-facing message), exactly what the direct route raises; the API problem document is built from these `ErrorReport` fields (the structured arrays on `ValidateBundleError` never reached the problem document even in direct mode — verified). Graph target = `arg.pipe_code or` the first blueprint's qualified `main_pipe`.-> - **Wrapper workflow:** `WfDryValidate` (`wf_dry_validate.py`, Temporal name `wf_dry_validate`) — one `workflow.execute_activity(act_dry_validate, start_to_close_timeout=5min, RetryPolicy(maximum_attempts=2, non_retryable_error_types=[ValidateBundleError]))` (post-review trim: the other deterministic names never cross — `validate_bundle` wraps them).-> - **Submitter helper (new, for Phase 4):** `dispatch_dry_validate(arg, *, task_queue=None, should_auto_connect_temporal=True)` in `tprl_pipe/dry_validate_dispatch.py` — mints the workflow id (`dry_validate_<uuid>` through `make_workflow_id`) and awaits the wrapper workflow; failures surface as `WorkflowExecutionError` carrying the recovered `ErrorReport`. The API route should call THIS.-> - **Registration:** `tasks.py` `PackName.PIPE` — `WfDryValidate` appended to `workflow_list`, `act_dry_validate` to `activity_list` (split workers pick them up by scope automatically — verified in the 3-process run: workflow on router, activity on runner).-> - **Divergences from plan:** (1) the activity body holds no scopes itself — `dry_run_pipe_in_process` owns its three, and `validate_pipes` now owns `scoped_pipe_router` + `scoped_content_generator` (the Part-B guard now also covers direct CLI sweeps, not just the activity); (2) graph "no main_pipe and no pipe_code" → `graph=None` silently (no error), matching the route's separate main-pipe precondition; (3) ~~the event-log-drop RED fails loudly (exit 1)~~ post-review (R-D1 parity) the event-log-drop regression degrades to `GRAPH: None` in the activity — the loud CI guard is the Mode-1 `test_dry_run_graph_in_process.py` direct-call assertions; Tier-2d doc updated.->-> **Next: Phase 4.**-------## Phase 4 — API dispatch (cross-repo `pipelex-api`)--- [x] `../pipelex-api/api/routes/pipelex/validate.py` split into `_validate_direct` (today's body, unchanged) + `_validate_via_temporal`, gated on `get_config().temporal.is_enabled`. Temporal branch: **dispatch FIRST** (before any API-side parsing, so every validation failure surfaces through the worker's `validate_bundle` cascade with the identical categorized `ValidateBundleError` 422 — `WorkflowExecutionError.to_error_report()` returns the recovered report and the existing global handler renders it; zero route-side catch), then re-parse blueprints for the envelope, check the `main_pipe` precondition, and build `pipe_structures` from a local **load-only** `acquire_library` (no sweep, no dry-run, no tracing API-side). Best-effort graph rides back as `graph_spec` on the activity result.-- [x] Both backends tested (`tests/unit/test_validate_temporal_dispatch.py`): Temporal-enabled returns the envelope from one round-trip (graph + blueprint + structures, dispatch awaited exactly once with the request's contents/allow_signatures), best-effort `graph_spec=null` → 200, validation failure renders the same 422 (`ValidateBundleError`/`input`, refs in `detail`), and direct mode **never** touches the dispatch helper. Existing direct-path tests (`test_allow_signatures` etc.) green against the refactored route.-- [x] **Tier 2d API arm RUN live** (real route + real dispatch via TestClient in the pipelex-api venv, against the running server + split workers): 200 with worker-assembled `graph_spec` (3 nodes/3 edges) in one round-trip; signature bundle → 422 `ValidateBundleError`/`input` through the real activity→workflow→submitter→handler chain; **found+fixed in the process:** the config-default workflow retry policy re-ran a deterministically-failed validation (same workflow id listed twice) — `dispatch_dry_validate` now pins `RetryPolicy(maximum_attempts=1)` on the wrapper workflow (D-C5 at the workflow tier), re-verified (failed workflow appears exactly once). API-arm procedure added to the Tier-2d skill scenario.-- [x] Docs + CHANGELOG `[Unreleased]` in both repos (pipelex: tracing doc + temporal-integration doc + CHANGELOG, done in Phases 1–3; pipelex-api: `docs/pipe-validate.md` "Execution Backends" section + CHANGELOG entry noting the pipelex version requirement).--> ### ⛔ CHECKPOINT F — after Phase 4 — **ALL REQUIREMENTS MET**->-> req 2 (production dry-run + validation as one in-process Temporal activity, in-memory tracing) · req 3 (direct in-process unchanged). req 1 is delivered by Part B, not here.->-> **Verify:** `pipelex` `make agent-test` green · Temporal e2e green · **Tier 2d (activity + API arms) GREEN** · `pipelex-api` tests green on **both** backends · commit.->-> **Handoff (filled in, 2026-06-09) — as-built summary:**->-> - **Phase 1** — `pipelex/hub.py`: `scoped_event_log` / `get_event_log_override` (ContextVar scope); write side `pipeline_run_setup.py` and read side `tracing_assembly.py::assemble_tracing` prefer the override (override implies tracing-enabled, read side doesn't `close()` it). The two-instance fix.-> - **Phase 2** — `pipelex/pipe_run/dry_run_pipeline.py::dry_run_pipe_in_process(pipe, *, library_id)`: graph dry-run with zero dispatch under the three scopes (`scoped_event_log` + `scoped_pipe_router` + new `hub.scoped_content_generator`, which `get_content_generator()` prefers); tracer keys aligned by construction.-> - **Phase 3** — `pipelex/temporal/tprl_pipe/act_dry_validate.py` (`DryValidateArg`/`DryValidateResult`, activity composes `validate_bundle` + `dry_run_pipe_in_process`, D4 load-once, D5 narrow best-effort graph), `wf_dry_validate.py::WfDryValidate` (one-step wrapper, D-C5 activity retry bounds), `dry_validate_dispatch.py::dispatch_dry_validate` (submitter helper, `maximum_attempts=1` at the workflow tier), registered in `tasks.py`; `BundleValidator.validate_pipes` also scopes the dry content generator. Tier 2d (skill scenario + `submit_dry_validate.py` + master-table row) GREEN+RED on a real 3-process stack.-> - **Phase 4** — `pipelex-api` route split (`_validate_direct` unchanged / `_validate_via_temporal` dispatch-first), unit tests for both backends, live API-arm verification, docs+CHANGELOG both repos.-> - **How the API dispatches now:** `dispatch_dry_validate(DryValidateArg(mthds_contents, allow_signatures))` → `wf_dry_validate` → one `act_dry_validate` on the worker → `{dry_run_outputs, graph_spec}` in one round-trip; failures cross as `WorkflowExecutionError` carrying the recovered `ValidateBundleError` report → existing 422 handler.-> - **Cross-repo shipping note:** the `pipelex-api` branch (`feature/Update-dry-run-api`) imports `pipelex.temporal.tprl_pipe.act_dry_validate` — it needs a pipelex release/rev **newer than v0.32.1** (bump the `pipelex` pin + `[tool.uv.sources]` rev) before its CI can pass; local verification used `uv pip install -e ../_dry`.-> - **Follow-ups (deferred):** retire the old worker-workflow graph path for `/validate` (now unused by the API; `dry_run_pipeline` still serves the direct CLI/API path) · Phase G0 standalone activity (`temporalio` bump) · D-plan §7 endpoint unification · consider moving `pipe_structures` into the activity result to drop the API-side load-only library acquisition (today it double-loads: worker + API).->-> Folded into `wip/dry-run-refactor/consolidation-as-built.md` (§ "Part C as built") and the README workstream table.-------## Phase G0 — *(optional, deferred — do AFTER Checkpoint F, only if wanted)* true standalone activity--The wrapper workflow ships the feature with no SDK bump. Replacing it with a true standalone activity is a pure runtime optimization (one fewer workflow per call), not a requirement — do it later, on its own branch/PR, only if the round-trip cost matters. Independent regression surface (the SDK underpins the whole worker/runtime).--- [ ] Confirm the target `temporalio` version (past `1.23.0`) and that our Temporal Cloud / server supports standalone-activity execution. *(This is the only step that needs infra input — and it's no longer blocking anything.)*-- [ ] Bump `temporalio` in `pyproject.toml` (+ `uv.lock` via `uv`); run the **full** Temporal e2e suite on the new SDK (`temporal-e2e-validate`) as a separately-reviewable step.-- [ ] Swap the API dispatch from the wrapper workflow to the standalone activity; keep the wrapper-workflow path as fallback if useful.-------## Distributed verification — `temporal-e2e-validate` (Mode 1 / Tier 2d)--Mode 1's contract must be proven in a **real distributed context**, not only unit/integration. Add a tier to the repo's own skill at `.claude/skills/temporal-e2e-validate/`, following the **Tier 2c precedent** (#976 added: a scenario in `references/mode-2-tiers.md` Step 3, a Mode-1 pytest, and a Step-7 master-table row). Build it as part of Phase 3 (activity arm) and Phase 4 (API arm).--**Tier 2d — dry-run + validation runs as ONE in-process, in-memory activity.** Sibling to Tier 2c: where 2c proves the *direct* `/validate` sweep doesn't leak to Temporal, 2d proves the *Temporal-dispatched* path runs the whole sweep **+** graph dry-run inside a single activity, in memory, returning `{status map, GraphSpec}`.--**Mode 2 (3-process) GREEN** — boot Temporal-enabled, split workers up (`mode-2-setup.md`); dispatch the wrapper-workflow→`act_dry_validate` over a controller bundle (`temporal_parallel.mthds` — interesting graph):--- exit 0; per-pipe status map present (all `SUCCESS`); non-empty `GraphSpec` → `reactflow.html` assembles.-- **Strong check (the point):** during the run the worker ran the wrapper workflow + exactly one `act_dry_validate` and **nothing else** — NO child `WfPipeRouter`/`WfPipeRun`, NO `act_llm_gen_*`, NO `act_assemble_tracing`/`act_flush_trace_events`. Capture both worker sessions and grep — expect none of those for this run. (Mirrors Tier 2c's worker-idle check, but here the activity itself is expected; what must be absent is everything *nested*.)-- **In-memory tracing:** no new NDJSON partition appears under `.pipelex/traces/` for the activity's internal graph dry-run, and no DynamoDB write — the `GraphSpec` rode back on the activity result, assembled from the in-memory log.-- **No usage/cost:** no cost table, no `usage_report` events.-- **Best-effort graph sub-case:** a bundle whose graph dry-run fails (a pipe needing un-mockable input) still returns exit 0 with the status map and `graph=None` (no `reactflow.html`); validation still OK.-- **Concurrency:** two concurrent dispatches return distinct `GraphSpec`s with no shared/merged trace events.--**Mode 2 RED (prove it bites):** in the activity body, drop `scoped_content_generator(inline)` → the leaf reaches the hub `ContentGeneratorInWorkflow` and the activity tries to dispatch `act_llm_gen_*` from inside an activity (illegal) / the strong check shows dispatch; **or** drop `scoped_event_log` → the `GraphSpec` comes back empty (the two-instance regression). Restore immediately.--**Mode 1 (pytest) companion (CI-cheap)** — `tests/integration/pipelex/temporal/test_dry_validate_activity_in_memory.py`: run the wrapper-workflow→activity against the in-process server; spy `WorkflowExecutor.execute_activity`/`execute_workflow` and assert no nested dispatch during the activity; assert the event-log backend received no writes (in-memory only); assert best-effort graph returns `graph=None` on a failing graph dry-run.--**Step-7 master-table row to add:** `Tier 2d: Dry-run+validate as one in-memory activity | the Temporal-dispatched /validate runs the whole sweep + graph dry-run inside ONE in-process activity (zero nested dispatch), traces the graph in memory (no NDJSON/DDB), returns {status, GraphSpec}; best-effort graph → None on failure | PASS/FAIL | path | — `.--## GSTACK REVIEW REPORT--_Eng review (FULL) 2026-06-09 · commit 6434b6e09 · outside voice: Codex._--| Review | Trigger | Why | Runs | Status | Findings |-|--------|---------|-----|------|--------|----------|-| CEO Review | `/plan-ceo-review` | Scope & strategy | 0 | — | — |-| Codex Review | `/codex review` | Independent 2nd opinion | 1 | issues_found | 4 substantive (C1/C2/C3 resolved, C5 build-now), C6/C7 folded to tests |-| Eng Review | `/plan-eng-review` | Architecture & tests (required) | 1 | issues_open | 7 issues, 1 critical-gap-class (T3) |-| Design Review | `/plan-design-review` | UI/UX gaps | 0 | — | — |-| DX Review | `/plan-devex-review` | Developer experience gaps | 0 | — | — |--- **CODEX:** caught the Phase-2 `bridge DIRECT` flaw (cannot carry `run_mode=DRY` + `mock_inputs` — verified at `bridge.py:216`), the output-contract contradiction, and the library-lifecycle gap — all missed by the in-review reasoning, all folded as decisions.-- **CROSS-MODEL:** the review and Codex agree the plan's foundation (scoped in-memory event log, one-activity dispatch via wrapper workflow, best-effort graph) is sound. The one tension — Phase 2 seam — resolved in Codex's favor (`prepare_pipe_job`, not the bridge), verified in code.-- **VERDICT:** ENG CLEARED — findings folded as decisions; T1–T10 backlogged (tasks JSONL in gstack project dir). Scope accepted as-is (Step 0). Lake score 8/8. Implement with the decisions below.--**Decisions locked in this review (amend the phases accordingly):**--- **D1 — Phase 1 `is_enabled`:** a set `scoped_event_log` override **implies tracing-enabled** at all three guards (`pipeline_run_setup.py:199`, `tracing_assembly.py:84` early-return, `:95`). Regression test: `is_enabled=False` + override → GraphSpec still assembles in-memory. (T2)-- **D2 — Phase 1 write side:** keep **both** `make_event_log` patches for a symmetric primitive (write side is off this feature's bridge path but works for any `execute_pipeline` caller); test the full path. (T7)-- **D-C1 — Phase 2 seam:** use **`prepare_pipe_job`** (`pipe_run_mode=DRY`, `mock_inputs=True`, `generate_graph=True`) + a **local `PipeRun`** under the three scopes — **not** the bridge (the bridge runs LIVE/no-mock). (T1)-- **D3 — Phase 3 output contract:** output = `{status map (success only), GraphSpec | None}`. Unexpected pipe failures + signature refusals **raise** → `ErrorReport` → API 422. Drop the `signature_check_error` output field. **Mandatory: verify the structured offending-signatures survive the `ErrorReport` crossing** (else 422 degrades to a plain message — the one critical-gap-class). (T3)-- **D4 — Phase 3 library lifecycle:** **load once**, run sweep **and** graph against the same `library_id` (borrow-the-open-library inner sweep, no per-step teardown), tear down once in the activity's `finally`. (T4)-- **D5 — Phase 3 best-effort catch:** **narrow** to the specific dry-run-failure `PipelexError` subclasses → `graph=None`; all other `PipelexError`s and every non-`PipelexError` bug **propagate** (mirrors `assemble_tracing:69`). Test the bug-propagates arm. (T5)-- **D-C5 — Phase 3 retry/timeout:** set explicit `start_to_close_timeout` + `non_retryable_error_types` on `act_dry_validate` so deterministic failures don't retry 3×. (T6)-- **D6 — tests:** harden the Mode-1 spy to assert at the `workflow.execute_activity` SDK site / Temporal history, not the `WorkflowExecutor` wrapper (T8); forbid divergent `tracer_key` (T9).-- **Dismissed:** activity-result payload-size guard — existing S3 payload offload already handles large results.--NO UNRESOLVED DECISIONS--### Post-implementation /review (2026-06-09, on the completed branch)--Pre-landing review (structured pass + specialist subagents + Claude adversarial + Red Team + Codex adversarial + Codex structured review, all on commit 6ba8d8137). Codex structured: clean. Four decisions taken and applied on-branch:--- **R-D1 — graph-arm cross-backend parity (supersedes D5's narrow catch).** The direct route catches base `PipelexError` around the graph dry-run (best-effort for ANY domain failure); the activity's narrow tuple made the same bundle 200 on direct and 422-after-retry on Temporal. Widened to `(PipelexError, ValidationError, FactoryException)` — the `_classify_pipe` catch — with pipe resolution moved inside (unknown `pipe_code` → `graph=None`). `WfDryValidate.non_retryable_error_types` trimmed to `["ValidateBundleError"]` (the only deterministic name that actually crosses). Consequence: the event-log-drop regression degrades to `GRAPH: None` in the activity; the loud CI guard is the Mode-1 direct-call test.-- **R-D2 — cheap hardening.** Activity `finally` mirrors `PipeRun.run`'s suppress-secondary pattern (a teardown raise no longer replaces the user's real error); `dispatch_dry_validate` pins `workflow_execution_timeout=12min` (interactive bound, not the 1-hour batch default); `scoped_event_log` lifecycle docs corrected (the write-side tracer DOES `close()` its log at teardown — a scoped log's `close()` must be idempotent/no-op); `dry_run_pipe_in_process` constructs its primitives before `open_tracer` (no tracer-entry leak window). Design-tradeoff items deferred to [`wip/dry-run-refactor/followup-dry-validate-hardening.md`](wip/dry-run-refactor/followup-dry-validate-hardening.md).-- **R-D3 — `DryValidateArg.library_dirs` dropped.** No caller passed it; it was an unconstrained worker-filesystem read surface. Re-add with allowlist confinement if a real caller appears.-- **R-D4 — test lakes added.** No-retry assertion (D-C5 pinned), no-main-pipe arm, unknown-pipe_code parity arm, `scoped_content_generator` unit tests, restore-prev-library direct-call test, `dispatch_dry_validate` retry/timeout unit test.+Update THIS file as you go (checkpoint style): check boxes and append outcomes as each item lands, so a further session can resume cold.diff --git a/docs/under-the-hood/temporal-integration.md b/docs/under-the-hood/temporal-integration.mdindex ed026c7b6..fa4f81e98 100644--- a/docs/under-the-hood/temporal-integration.md+++ b/docs/under-the-hood/temporal-integration.md@@ -148,11 +148,11 @@ Each `WfPipeRouter.run()` creates its own scoped state:1. **ClassRegistry**: A new `ClassRegistry` pre-seeded from the global registry (which has base classes from `PIPELEXPATH`). Dynamic classes from the crate are registered here — not in the global registry.-2. **Library**: A new `Library` instance opened via `library_manager.open_library()`, with the `ClassRegistry` attached as a `PrivateAttr`. The library is set as the current library via the `_library_id` `ContextVar`.+2. **Library**: A new `Library` instance opened via `library_manager.open_fresh_library()`, with the `ClassRegistry` attached as a `PrivateAttr`. The library is set as the current library via the `_library_id` `ContextVar`. The library id is keyed by the run id (`wf_{run_id}`) — replay-stable but unique per run. Keying by workflow id would collide across runs of a reused workflow id (workflow-level retry policy, Temporal reset, resubmission of the same `pipeline_run_id`): a closed predecessor run's late eviction cleanup could tear down a live successor run's library on the same worker. With run-id keying, a pre-existing library under the id can only be this same run's own leftover (an interrupted execution whose cleanup never ran), so `open_fresh_library` tears it down before opening: reusing it would fingerprint-skip the crate load against the fresh registry — the crate's dynamic classes would never land in it. The stale teardown is best-effort: the manager forgets the entry pop-first, and a raising teardown is logged and never fails the fresh open (worker-local leak state must not decide setup success). The same run-id keying applies to the rest of the per-run worker-local state: the graph tracer key and the report-delegate event-log context key.3. **ClassRegistry lookup chain**: `hub.get_class_registry()` reads `_library_id` from the `ContextVar`, gets the library's attached `ClassRegistry`. Falls back to the global registry if no library is set.-4. **Cleanup**: `library_manager.teardown(library_id)` deletes the library and its `ClassRegistry`. The `ContextVar` is reset via `clear_current_library()`. No manual GC needed — the `ClassRegistry` is garbage-collected with the `Library`.+4. **Cleanup**: `library_manager.teardown(library_id)` deletes the library and its `ClassRegistry`. The `ContextVar` is reset via `clear_current_library()`. No manual GC needed — the `ClassRegistry` is garbage-collected with the `Library`. In the workflow's `finally` block, this worker-local cleanup runs BEFORE the awaited `act_flush_trace_events` activity: the await is a suspension point where an eviction-time `BaseException` can abort the rest of the block, so ordering cleanup first guarantees an eviction can only skip the flush itself, never the teardown.### Kajson integrationdiff --git a/pipelex/graph/graph_tracer_manager.py b/pipelex/graph/graph_tracer_manager.pyindex 5f0afe54b..df25ae27f 100644--- a/pipelex/graph/graph_tracer_manager.py+++ b/pipelex/graph/graph_tracer_manager.py@@ -3,6 +3,7 @@from datetime import datetimefrom typing import Any+from pipelex import logfrom pipelex.graph.graph_config import DataInclusionConfigfrom pipelex.graph.graph_tracer import GraphTracerfrom pipelex.graph.graph_tracer_protocol import GraphTracerProtocol@@ -125,14 +126,27 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):Returns:Initial TraceContext to pass through JobMetadata.-- Raises:- ValueError: If a tracer for this key already exists."""key = tracer_key or graph_id+ # Collision-proof pop-and-replace: a stale tracer under this key can only be the+ # leftover of a prior interrupted execution (e.g. a Temporal workflow evicted before+ # its finally-block close_tracer ran). Raising would let worker-local leak state+ # decide whether tracing setup succeeds — a replay-determinism breach (audit+ # finding M1). Delegate the eviction to close_tracer (the single teardown path).+ # The warning is gated on key membership, not on close_tracer's return value: that+ # value is ambiguous (None both when no tracer existed and when a costs-only tracer+ # legitimately tears down without a GraphSpec).if key in self._tracers:- msg = f"Tracer for key '{key}' already exists"- raise ValueError(msg)+ log.warning(f"Tracer for key '{key}' already exists; replacing stale tracer left by a prior interrupted execution")+ try:+ self.close_tracer(key)+ except Exception as stale_teardown_exc: # noqa: BLE001+ # Best-effort: the stale tracer's teardown runs graph assembly over arbitrary+ # half-built state from the interrupted execution — its failure must never+ # fail the fresh run's setup (that would be the M1 class again: worker-local+ # leak state deciding setup success). close_tracer pops before tearing down,+ # so the key is free either way.+ log.warning(f"Stale tracer teardown for key '{key}' failed; replacing anyway: {stale_teardown_exc}")tracer = GraphTracer()diff --git a/pipelex/libraries/library_manager.py b/pipelex/libraries/library_manager.pyindex 9d6f67bb9..40bd4c535 100644--- a/pipelex/libraries/library_manager.py+++ b/pipelex/libraries/library_manager.py@@ -1,4 +1,5 @@import uuid+from contextlib import ExitStackfrom graphlib import CycleError, TopologicalSorterfrom pathlib import Pathfrom typing import TYPE_CHECKING@@ -110,30 +111,56 @@ class LibraryManager(LibraryManagerAbstract):def setup(self) -> None:pass- @override- def teardown(self, library_id: str | None = None) -> None:- if library_id:- if library_id not in self._libraries:- msg = f"Trying to teardown a library that does not exist: '{library_id}'"- raise LibraryError(msg)- library = self._libraries[library_id]+ def _pop_and_teardown_library(self, library_id: str) -> bool:+ """Atomically forget a library entry, then tear it down.++ The entry is popped BEFORE ``library.teardown()`` runs and the associated maps are+ cleared in a ``finally``, so the manager forgets the library even if its teardown+ raises — a kept entry would turn ``open_fresh_library`` into a deterministic+ re-raise on every rerun of that library id on this worker (permanent worker-local+ poison). The pop also makes the operation tolerant of a concurrent teardown of the+ same id from another workflow thread: whichever caller pops the entry tears it+ down, the other sees a clean miss.++ Returns:+ True when an entry existed and was removed, False when there was none.+ """+ library = self._libraries.pop(library_id, None)+ if library is None:+ return False+ try:# Remove source map entries for pipes in this libraryfor pipe_ref in library.pipe_library.root:self._pipe_source_map.pop(pipe_ref, None)library.teardown()- del self._libraries[library_id]+ finally:self._blueprints.pop(library_id, None)self._crate_cache.pop(library_id, None)self._loaded_fingerprints.pop(library_id, None)+ return True++ @override+ def teardown(self, library_id: str | None = None) -> None:+ if library_id:+ if not self._pop_and_teardown_library(library_id=library_id):+ msg = f"Trying to teardown a library that does not exist: '{library_id}'"+ raise LibraryError(msg)return- for library in self._libraries.values():- library.teardown()+ # Same forget-even-on-raise contract as the single-id branch, extended to the loop:+ # all bookkeeping is dropped upfront, and ExitStack guarantees EVERY library's own+ # teardown is attempted even when one of them raises — the first failure propagates+ # only after the rest have run, so neither a kept entry nor a skipped sibling+ # teardown can result from a raising library.+ libraries = list(self._libraries.values())self._libraries = {}self._pipe_source_map = {}self._blueprints = {}self._crate_cache = {}self._loaded_fingerprints = {}+ with ExitStack() as teardown_stack:+ for library in libraries:+ teardown_stack.callback(library.teardown)@overridedef reset(self) -> None:@@ -153,6 +180,28 @@ class LibraryManager(LibraryManagerAbstract):return library_id, the_library+ @override+ def open_fresh_library(self, library_id: str) -> Library:+ try:+ removed_existing = self._pop_and_teardown_library(library_id=library_id)+ except Exception as stale_teardown_exc: # noqa: BLE001+ # Best-effort: the stale library's teardown runs over arbitrary half-built state+ # from an interrupted execution — its failure must never fail the fresh open+ # (worker-local leak state deciding setup success would be the M1 class again).+ # _pop_and_teardown_library forgets the entry pop-first, so the id is free either way.+ log.warning(+ f"open_fresh_library: stale teardown of pre-existing library '{library_id}' raised; "+ f"continuing with a fresh library: {stale_teardown_exc}"+ )+ else:+ if removed_existing:+ log.warning(+ f"open_fresh_library: tore down pre-existing library '{library_id}' — "+ f"leftover of an interrupted execution whose cleanup never ran"+ )+ _library_id, the_library = self.open_library(library_id=library_id)+ return the_library+############################################################# Public library accessors############################################################diff --git a/pipelex/libraries/library_manager_abstract.py b/pipelex/libraries/library_manager_abstract.pyindex aa1f2ad4c..1e587b445 100644--- a/pipelex/libraries/library_manager_abstract.py+++ b/pipelex/libraries/library_manager_abstract.py@@ -30,6 +30,17 @@ class LibraryManagerAbstract(ABC):def open_library(self, library_id: str | None = None) -> tuple[str, "Library"]:"""Open a library with the given library_id. Creates it if it doesn't exist. If no library_id is provided, it creates one."""+ @abstractmethod+ def open_fresh_library(self, library_id: str) -> "Library":+ """Open a library guaranteed to be empty, tearing down any pre-existing library under this id.++ For callers that reuse a deterministic library_id across executions (e.g. a Temporal+ workflow keyed by its workflow id): a pre-existing library under such an id can only be+ the leftover of an interrupted predecessor execution whose cleanup never ran. Reusing it+ is poison — its crate fingerprints would dedup-skip a fresh crate load while a freshly+ attached ClassRegistry no longer holds the crate's dynamic classes.+ """+@abstractmethoddef get_library(self, library_id: str) -> "Library":"""Get the Library object for a specific library_id."""diff --git a/pipelex/pipeline/pipeline_manager.py b/pipelex/pipeline/pipeline_manager.pyindex 040a059a0..47faa7a9d 100644--- a/pipelex/pipeline/pipeline_manager.py+++ b/pipelex/pipeline/pipeline_manager.py@@ -44,3 +44,7 @@ class PipelineManager(PipelineManagerAbstract, RootModel[PipelineManagerRoot]):pipeline = PipelineFactory.make_pipeline(pipeline_run_id=pipeline_run_id)self._set_pipeline(pipeline_run_id=pipeline.pipeline_run_id, pipeline=pipeline)return pipeline++ @override+ def remove_pipeline(self, pipeline_run_id: str) -> None:+ self.root.pop(pipeline_run_id, None)diff --git a/pipelex/pipeline/pipeline_manager_abstract.py b/pipelex/pipeline/pipeline_manager_abstract.pyindex c87d5f7ed..3e756cefe 100644--- a/pipelex/pipeline/pipeline_manager_abstract.py+++ b/pipelex/pipeline/pipeline_manager_abstract.py@@ -23,3 +23,11 @@ class PipelineManagerAbstract(ABC):@abstractmethoddef add_new_pipeline(self, pipe_code: str | None, pipeline_run_id: str | None = None) -> Pipeline:pass++ @abstractmethod+ def remove_pipeline(self, pipeline_run_id: str) -> None:+ """Forget a run's entry so the same ``pipeline_run_id`` can be resubmitted later.++ Tolerant: a no-op when the entry is absent (setup may have failed before+ the registration committed).+ """diff --git a/pipelex/pipeline/pipeline_run_setup.py b/pipelex/pipeline/pipeline_run_setup.pyindex 26fa4cca6..ebf2d9c5e 100644--- a/pipelex/pipeline/pipeline_run_setup.py+++ b/pipelex/pipeline/pipeline_run_setup.py@@ -2,6 +2,7 @@ from typing import TYPE_CHECKINGfrom mthds.models.pipeline_inputs import PipelineInputs+from pipelex import logfrom pipelex.config import get_configfrom pipelex.core.memory.working_memory import WorkingMemoryfrom pipelex.graph.graph_tracer_manager import GraphTracerManager@@ -151,19 +152,21 @@ async def pipeline_run_setup(# acquire_library's own load-failure teardown).prev_library_id = get_current_library_id_or_none()- # Seam 1: open the library and load dirs + blueprints. Owns its own load-failure teardown.- library_id, qualified_main_pipe = acquire_library(- library_id=library_id,- library_dirs=library_dirs,- mthds_contents=mthds_contents,- bundle_uris=bundle_uris,- )-library_manager = get_library_manager()+ library_acquired = Falsetrace_context: TraceContext | None = Noneevent_log: EventLogProtocol | None = Nonesuccess = Falsetry:+ # Seam 1: open the library and load dirs + blueprints. Owns its own load-failure teardown.+ library_id, qualified_main_pipe = acquire_library(+ library_id=library_id,+ library_dirs=library_dirs,+ mthds_contents=mthds_contents,+ bundle_uris=bundle_uris,+ )+ library_acquired = True+# Resolve the pipe to execute against the now-open library.pipe: PipeAbstractif mthds_contents:@@ -286,25 +289,50 @@ async def pipeline_run_setup(return pipe_job, pipeline_run_id, library_idfinally:if not success:- # Error-path-only cleanup for failures after the library was acquired: tear down the graph- # tracer, event-log state, and the library, and restore the outer current-library, then let- # the exception propagate. Uses try/finally (not except) so a BaseException — e.g.- # asyncio.CancelledError — is covered too. acquire_library owns teardown for load-time- # failures (before this try); this block owns the post-acquire window.- if trace_context is not None:- tracer_manager = GraphTracerManager.get_instance()- if tracer_manager is not None:- tracer_manager.close_tracer(pipeline_run_id)- if event_log is not None:- get_report_delegate().clear_event_log(context_key=pipeline_run_id)- # Restore the caller's outer current-library FIRST so the guarantee survives a teardown raise,- # then tear the library down — mirroring validate_bundle. set_current_library cannot take None,- # so route the "no outer was set" case through clear_current_library. The `!= library_id` guard- # covers the collision validate_bundle never hits (it always opens a fresh uuid): when the caller- # passed a library_id equal to its own outer current-library, "restoring" it would leave the- # ContextVar pointing at the library we are about to tear down — so clear instead of dangling.- if prev_library_id is not None and prev_library_id != library_id:- set_current_library(library_id=prev_library_id)- else:- clear_current_library()- library_manager.teardown(library_id=library_id)+ # Error-path-only cleanup for failures after the run was registered: tear down the graph+ # tracer and event-log state, free the pipeline-manager entry, and restore + tear down the+ # library, then let the exception propagate. Uses try/finally (not except) so a+ # BaseException — e.g. asyncio.CancelledError — is covered too. acquire_library owns+ # library teardown for its own load-time failures (the `library_acquired` guard); this+ # block owns the post-acquire window.+ try:+ # close_tracer must run before remove_pipeline: while the entry is registered,+ # add_new_pipeline's collision raise shields the live direct-mode tracer (keyed by the+ # caller-suppliable pipeline_run_id) from open_tracer's stale-key pop-and-replace healing.+ # It can also raise — GraphTracer.teardown closes the event log, and a file-backed+ # transport's close() flushes to disk. The known OSError is suppressed below so it cannot+ # mask the original setup failure (callers type-match: runner.py wraps PipelexError into+ # PipelineExecutionError, the API maps error types to status codes — mirrors PipeRun.run).+ # The registry free and library restore still live in the inner finally as a guarantee+ # against any other teardown raise. Safe even then: close_tracer pops the tracer BEFORE+ # teardown, so no stale tracer remains under the key when the id is freed.+ if trace_context is not None:+ tracer_manager = GraphTracerManager.get_instance()+ if tracer_manager is not None:+ try:+ tracer_manager.close_tracer(pipeline_run_id)+ except OSError as tracer_close_error:+ log.error(+ f"close_tracer also failed for pipeline_run_id={pipeline_run_id} "+ f"during setup-failure cleanup; raising original setup error. "+ f"Suppressed tracer close error: {tracer_close_error}"+ )+ finally:+ if event_log is not None:+ get_report_delegate().clear_event_log(context_key=pipeline_run_id)+ # Free the per-run registry entry so the same pipeline_run_id can be resubmitted after+ # this failure. Placed before the library block so a teardown raise cannot strand the+ # entry (remove_pipeline itself is a pop-with-default and cannot raise).+ get_pipeline_manager().remove_pipeline(pipeline_run_id=pipeline_run_id)+ if library_acquired:+ # Restore the caller's outer current-library FIRST so the guarantee survives a teardown raise,+ # then tear the library down — mirroring validate_bundle. set_current_library cannot take None,+ # so route the "no outer was set" case through clear_current_library. The `!= library_id` guard+ # covers the collision validate_bundle never hits (it always opens a fresh uuid): when the caller+ # passed a library_id equal to its own outer current-library, "restoring" it would leave the+ # ContextVar pointing at the library we are about to tear down — so clear instead of dangling.+ if prev_library_id is not None and prev_library_id != library_id:+ set_current_library(library_id=prev_library_id)+ else:+ clear_current_library()+ library_manager.teardown(library_id=library_id)diff --git a/pipelex/pipeline/runner.py b/pipelex/pipeline/runner.pyindex 71a4a4175..2176f3775 100644--- a/pipelex/pipeline/runner.py+++ b/pipelex/pipeline/runner.py@@ -16,6 +16,7 @@ from pipelex.hub import (get_current_library_id_or_none,get_library_manager,get_pipe_run,+ get_pipeline_manager,get_report_delegate,get_telemetry_manager,set_current_library,@@ -216,6 +217,17 @@ class PipelexRunner(RunnerProtocol["PipeOutput"]):if pipeline_run_id is not None:get_report_delegate().clear_event_log(context_key=pipeline_run_id)+ # Free the per-run registry entry so a later run can resubmit the same explicit+ # pipeline_run_id (serial resubmission is a supported scenario; only genuinely+ # concurrent same-id runs should collide in add_new_pipeline). Must come after+ # close_tracer: while the entry is registered, the collision raise shields the live+ # direct-mode tracer (keyed by the caller-suppliable pipeline_run_id) from+ # open_tracer's stale-key pop-and-replace healing. The pop is tolerant — when+ # pipeline_run_setup failed it already removed its own registration (and+ # pipeline_run_id is None here anyway).+ if pipeline_run_id is not None:+ get_pipeline_manager().remove_pipeline(pipeline_run_id=pipeline_run_id)+# Only teardown library if it was successfully createdif library_id_resolved is not None:# Restore the caller's outer current-library FIRST so the guarantee survives a teardowndiff --git a/pipelex/reporting/reporting_manager.py b/pipelex/reporting/reporting_manager.pyindex 50e6607de..5c2b067c6 100644--- a/pipelex/reporting/reporting_manager.py+++ b/pipelex/reporting/reporting_manager.py@@ -1,3 +1,4 @@+import sysfrom datetime import datetime, timezonefrom typing import NamedTuple, cast@@ -19,6 +20,24 @@ from pipelex.tracing.activity_event_log import ActivityEventLogCachefrom pipelex.tracing.event_log_protocol import EventLogProtocolfrom pipelex.tracing.trace_events import UsageReportEvent++def _is_in_temporal_activity() -> bool:+ """True when the current call stack runs inside a Temporal activity.++ ``sys.modules`` sniff instead of importing temporalio: if ``temporalio.activity`` was never+ imported in this process, no activity context can exist — the context is set by temporalio's+ own machinery, which requires the module to be imported. Importing it here would put the+ entire temporalio extra (Rust bridge, protobuf) on every boot's critical path wherever the+ ``pipelex[temporal]`` extra is installed, even for processes that never touch Temporal.+ Worker processes are unaffected: by the time any activity runs, ``temporalio.activity`` is+ necessarily in ``sys.modules``.+ """+ activity_module = sys.modules.get("temporalio.activity")+ if activity_module is None:+ return False+ return cast("bool", activity_module.in_activity())++# DynamoDB PutItem failures come in two sibling botocore base classes (neither subclasses the other):# ClientError (service-side throttle / auth) and BotoCoreError (transport / credential / timeout, e.g.# EndpointConnectionError, ReadTimeoutError, NoCredentialsError). Both are imported together (present or@@ -80,9 +99,19 @@ class ReportingManager(ReportingProtocol):Args:context_key: Unique key for this context (trace_context.lookup_key).event_log: The event log backend for emitting UsageReportEvents.- workflow_id: Temporal workflow ID or "direct".+ workflow_id: Run-scoped execution identity stamped into events+ (Temporal run ID, or "direct" outside Temporal).pipeline_run_id: Pipeline run ID for event correlation."""+ # The silent overwrite-on-existing-key below is intentional and load-bearing.+ # Shared contract: worker-local state keyed by a deterministic per-run id must be+ # self-healing on open/set, because a leaked entry from a prior interrupted+ # execution remains possible (deadlock-detector thread abandonment skips finally+ # entirely; worker kill between set and clear). The same idiom exists in+ # LibraryManager.open_fresh_library and GraphTracerManager.open_tracer (both+ # explicit, WARNING); here the bare dict assignment is the healing mechanism.+ # Adding "context already exists -> raise" collision detection would reintroduce+ # the eviction-poison class on the cost path (pre-M1 open_tracer did exactly that).self._event_log_contexts[context_key] = _EventLogContext(event_log=event_log,workflow_id=workflow_id,@@ -150,13 +179,23 @@ class ReportingManager(ReportingProtocol):"""Emit a UsageReportEvent for this job.Fast path: when set_event_log was registered for this trace context's- lookup_key (router process or direct mode), emit through the cached+ lookup_key (workflow-thread or direct mode), emit through the cachedper-context event log.+ Activity path: an emission coming from inside a Temporal activity NEVER+ takes the fast path, even when the activity runs co-located with the+ workflow worker. The registered context there is the workflow's+ in-sandbox BufferingEventLog, and activities do not re-execute on+ replay — a cross-thread write into that buffer makes the workflow's+ buffer content depend on whether activities actually ran, breaking+ replay determinism (audit finding H1). Activities must behave+ identically whether co-located or remote: per-process fallback.+Fallback: when context lookup misses (runner process — set_event_log- was never called here), emit through the per-process activity event- log so the event still lands in the same backend partition as the- rest of the run. See _emit_usage_event_runner_fallback for details.+ was never called here) or the emission comes from an activity, emit+ through the per-process activity event log so the event still lands in+ the same backend partition as the rest of the run. See+ _emit_usage_event_runner_fallback for details."""trace_context = inference_job.job_metadata.trace_contextif trace_context is None:@@ -173,10 +212,11 @@ class ReportingManager(ReportingProtocol):if not trace_context.emit_usage_events:return- context = self._event_log_contexts.get(trace_context.lookup_key)- if context is not None:- self._emit_via_registered_context(context, trace_context, tokens_usage)- return+ if not _is_in_temporal_activity():+ context = self._event_log_contexts.get(trace_context.lookup_key)+ if context is not None:+ self._emit_via_registered_context(context, trace_context, tokens_usage)+ returnself._emit_usage_event_runner_fallback(inference_job=inference_job,@@ -226,12 +266,15 @@ class ReportingManager(ReportingProtocol):tokens_usage: AnyTokensUsage,trace_context: TraceContext,) -> None:- """Emit through a per-process activity event log when no context was registered.+ """Emit through the per-process activity event log.- On the runner, ``set_event_log`` was never called — the workflow only- registered a context on the router process. We fall back to a- process-local event log built from ``tracing_config``, stamped with a- stable per-process writer_id of the form ``act_{pid}_{uuid8}``.+ This is the universal path for activity-side usage emissions (audit finding+ H1): every emission from inside a Temporal activity routes here, co-located+ or remote alike, so the workflow's in-sandbox buffer stays a pure function+ of inline execution. It also covers the original fallback case — a runner+ process where ``set_event_log`` was never called. The event log is+ process-local, built from ``tracing_config``, stamped with a stable+ per-process writer_id of the form ``act_{pid}_{uuid8}``.Documented over-counting risk (R2): retried activities re-emit a freshevent at sequence N+1 instead of overwriting the original at N, so the@@ -263,7 +306,7 @@ class ReportingManager(ReportingProtocol):workflow_id = trace_context.tracer_key or trace_context.graph_idnode_id = trace_context.parent_node_id or "unknown"- ActivityEventLogCache.warn_once_runner_fallback_engaged(workflow_id=workflow_id, writer_id=process_event_log.writer_id)+ ActivityEventLogCache.log_once_runner_fallback_engaged(workflow_id=workflow_id, writer_id=process_event_log.writer_id)seq = process_event_log.next_sequence()event = UsageReportEvent(diff --git a/pipelex/runtime_bridge/bridge.py b/pipelex/runtime_bridge/bridge.pyindex bc1f68339..7ffbfbb6c 100644--- a/pipelex/runtime_bridge/bridge.py+++ b/pipelex/runtime_bridge/bridge.py@@ -14,21 +14,15 @@ The Temporal extra is lazy-imported only inside the temporal-mode branches.from __future__ import annotations-from contextlib import asynccontextmanager-from typing import TYPE_CHECKING, Any, AsyncGenerator, Callable, cast-from uuid import uuid4+from typing import TYPE_CHECKING, Any, Callable, castimport shortuuid-from kajson.class_registry import ClassRegistry-from kajson.kajson_manager import KajsonManagerfrom pydantic import BaseModel, ConfigDict, Field, ValidationErrorfrom pipelex.core.memory.working_memory import MAIN_STUFF_NAMEfrom pipelex.core.memory.working_memory_factory import WorkingMemoryFactoryfrom pipelex.hub import (- get_library_manager,get_required_pipe,- scoped_current_library,scoped_pipe_router,)from pipelex.libraries.library_crate import LibraryCrate@@ -47,6 +41,7 @@ from pipelex.runtime_bridge.exceptions import (PipelexBridgeDispatchError,)from pipelex.runtime_bridge.execution_mode import PipelexExecutionMode+from pipelex.runtime_bridge.primitives.scoped_library import scoped_library_for_cratefrom pipelex.system.telemetry.otel_constants import OTelConstantsif TYPE_CHECKING:@@ -134,7 +129,7 @@ async def run_pipe_via_bridge(delivery_assignment = _decode_delivery_assignment(input_payload.delivery_assignment_dump)_validate_input(input_payload, delivery_assignment=delivery_assignment)- async with _scoped_library_for_crate(library_crate):+ with scoped_library_for_crate(library_crate, library_id_prefix="runtime_bridge"):# trace_context is honored for DIRECT only. The Temporal modes have their# own event-log infrastructure (via pipeline_run_setup); forwarding a host# trace_context there would make WfPipeRouter open its tracer under the@@ -270,51 +265,6 @@ def _decode_delivery_assignment(delivery_assignment_dump: dict[str, Any] | None)raise PipelexBridgeDispatchError(msg) from exc-@asynccontextmanager-async def _scoped_library_for_crate(library_crate: LibraryCrate | None) -> AsyncGenerator[str | None, None]: # noqa: RUF029- """Open a per-call scoped library for the duration of a pipe run.-- When ``library_crate`` is None, this is a no-op: callers fall back to the- library that was loaded into the active class registry at boot. When- provided, opens a fresh library, loads the crate into it, sets it as the- current library for the duration of the pipe execution, and tears it down- on the way out.- """- if library_crate is None:- yield None- return-- library_manager = get_library_manager()- library_id = f"runtime_bridge_{uuid4().hex}"-- # Pre-seed a per-call ClassRegistry from the global one so classes generated- # from the crate's inline structured concepts register into this scoped- # registry (discarded on teardown) rather than leaking into / colliding in- # the global Kajson registry. Mirrors the Temporal worker hydration path- # (see wf_pipe_router.py).- global_registry = KajsonManager.get_class_registry()- scoped_registry = ClassRegistry()- scoped_registry.register_classes_dict(global_registry.get_classes_dict())- # open_library + set_class_registry live inside the try so a throw between- # opening the library and reaching the yield still tears it down (the entry- # is registered in the manager the moment open_library returns). Mirrors the- # library_opened guard in submitter_hydration.rehydrate_pipe_output_with_crate.- library_opened = False- try:- _opened_library_id, library = library_manager.open_library(library_id=library_id)- library_opened = True- library.set_class_registry(scoped_registry)- # scoped_current_library captures and restores the prior current-library- # ContextVar, so a bridge call made from within an already-scoped library- # doesn't clobber the caller's context.- with scoped_current_library(library_id=library_id):- library_manager.load_from_crate(library_id=library_id, crate=library_crate)- yield library_id- finally:- if library_opened:- library_manager.teardown(library_id=library_id)--async def _run_direct(pipe_job: PipeJob,delivery_assignment: DeliveryAssignment | None,diff --git a/pipelex/runtime_bridge/primitives/scoped_library.py b/pipelex/runtime_bridge/primitives/scoped_library.pynew file mode 100644index 000000000..11365ad64--- /dev/null+++ b/pipelex/runtime_bridge/primitives/scoped_library.py@@ -0,0 +1,65 @@+"""Per-call scoped library for crate-based execution/hydration outside Temporal workers."""++import uuid+from collections.abc import Generator+from contextlib import contextmanager++from kajson.class_registry import ClassRegistry+from kajson.kajson_manager import KajsonManager++from pipelex import log+from pipelex.hub import get_library_manager, scoped_current_library+from pipelex.libraries.library_crate import LibraryCrate+++@contextmanager+def scoped_library_for_crate(library_crate: LibraryCrate | None, library_id_prefix: str) -> Generator[str | None, None, None]:+ """Open a per-call scoped library loaded from ``library_crate`` for the duration of the block.++ When ``library_crate`` is None this is a no-op (yields None): callers fall back to the+ library that was loaded into the active class registry at boot. Otherwise: opens a+ fresh library under a uuid-suffixed id, attaches a scoped ``ClassRegistry`` pre-seeded+ from the global one (so classes generated from the crate's inline structured concepts+ register into the scoped registry — discarded on teardown — rather than leaking into+ or colliding in the global Kajson registry), sets it as the current library, loads the+ crate, and tears everything down on the way out.++ The per-call uuid keying makes leaked-predecessor collisions structurally impossible,+ so plain ``open_library`` is correct here — ``open_fresh_library`` is only needed on+ the Temporal worker path, whose library ids are deterministic per run. This helper is+ deliberately NOT used by ``WfPipeRouter``: its teardown sequencing relative to the+ flush await IS the eviction-safety fix (H2), and folding it into a context manager+ would fight that ordering.+ """+ if library_crate is None:+ yield None+ return++ library_manager = get_library_manager()+ library_id = f"{library_id_prefix}_{uuid.uuid4().hex}"+ # open_library + set_class_registry live inside the try so a throw between opening+ # the library and reaching the yield still tears it down (the entry is registered in+ # the manager the moment open_library returns). scoped_current_library captures and+ # restores the prior current-library ContextVar, so a call made from within an+ # already-scoped library doesn't clobber the caller's context.+ library_opened = False+ try:+ _opened_library_id, library = library_manager.open_library(library_id=library_id)+ library_opened = True+ global_registry = KajsonManager.get_class_registry()+ scoped_registry = ClassRegistry()+ scoped_registry.register_classes_dict(global_registry.get_classes_dict())+ library.set_class_registry(scoped_registry)+ with scoped_current_library(library_id=library_id):+ library_manager.load_from_crate(library_id=library_id, crate=library_crate)+ yield library_id+ finally:+ if library_opened:+ try:+ library_manager.teardown(library_id=library_id)+ except Exception as teardown_exc: # noqa: BLE001+ # Best-effort: teardown in the finally must neither mask the body's in-flight+ # exception nor fail an otherwise-successful call. LibraryManager forgets the+ # entry pop-first even when the library's own teardown raises, so no poisoned+ # state remains.+ log.warning(f"scoped_library_for_crate: failed to tear down scoped library '{library_id}': {teardown_exc}")diff --git a/pipelex/runtime_bridge/primitives/submitter_hydration.py b/pipelex/runtime_bridge/primitives/submitter_hydration.pyindex 64c8d40e0..ff6883802 100644--- a/pipelex/runtime_bridge/primitives/submitter_hydration.py+++ b/pipelex/runtime_bridge/primitives/submitter_hydration.py@@ -17,15 +17,11 @@ from the global, load the crate into it, hydrate inside the scope, then teardown — leaving the submitter's global registry untouched."""-import uuidfrom typing import TYPE_CHECKING-from kajson.class_registry import ClassRegistry-from kajson.kajson_manager import KajsonManager-from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.hub import get_library_manager, scoped_current_libraryfrom pipelex.runtime_bridge.primitives.hydration import hydrate_working_memory+from pipelex.runtime_bridge.primitives.scoped_library import scoped_library_for_crateif TYPE_CHECKING:from pipelex.libraries.library_crate import LibraryCrate@@ -49,33 +45,8 @@ def rehydrate_pipe_output_with_crate(if pipe_output.working_memory_raw is None:return pipe_output- if library_crate is None:+ with scoped_library_for_crate(library_crate, library_id_prefix="rehydrate"):pipe_output.working_memory = hydrate_working_memory(pipe_output.working_memory_raw)pipe_output.working_memory_raw = None- return pipe_output-- library_manager = get_library_manager()- rehydration_library_id = f"rehydrate_{uuid.uuid4().hex}"- # open_library + set_class_registry live inside the try so a throw before the yield- # still tears the library down. scoped_current_library captures and restores the- # prior current-library ContextVar, so this rehydration doesn't clobber the caller's- # context. Mirrors bridge._scoped_library_for_crate.- library_opened = False- try:- _opened_library_id, rehydration_library = library_manager.open_library(library_id=rehydration_library_id)- library_opened = True-- global_registry = KajsonManager.get_class_registry()- scoped_registry = ClassRegistry()- scoped_registry.register_classes_dict(global_registry.get_classes_dict())- rehydration_library.set_class_registry(scoped_registry)-- with scoped_current_library(library_id=rehydration_library_id):- library_manager.load_from_crate(library_id=rehydration_library_id, crate=library_crate)- pipe_output.working_memory = hydrate_working_memory(pipe_output.working_memory_raw)- pipe_output.working_memory_raw = None- finally:- if library_opened:- library_manager.teardown(library_id=rehydration_library_id)return pipe_outputdiff --git a/pipelex/temporal/tprl_pipe/wf_pipe_router.py b/pipelex/temporal/tprl_pipe/wf_pipe_router.pyindex 6a6637403..eb580d69e 100644--- a/pipelex/temporal/tprl_pipe/wf_pipe_router.py+++ b/pipelex/temporal/tprl_pipe/wf_pipe_router.py@@ -10,7 +10,6 @@ with workflow.unsafe.imports_passed_through():from kajson.kajson_manager import KajsonManagerfrom pipelex.base_exceptions import PipelexError, iter_cause_chain- from pipelex.config import get_configfrom pipelex.core.pipes.pipe_output import PipeOutputfrom pipelex.graph.graph_tracer_manager import GraphTracerManagerfrom pipelex.hub import clear_current_library, get_library_manager, get_report_delegate, set_current_library@@ -72,14 +71,23 @@ class WfPipeRouter(WorkflowClass[PipeJob, PipeOutput]):pipe = workflow_arg.pipeworkflow_log.verbose(f"Routing {pipe.__class__.__name__} pipe '{workflow_arg.pipe.code}': {pipe.description}")+ # Run-scoped identity for ALL per-run worker-local state (library id, tracer key,+ # event-log context key) and for event/node-id stamps: run_id is replay-stable but+ # unique per run, so a reused workflow id (workflow-level retry_policy, Temporal+ # reset, resubmission of the same pipeline_run_id) can never collide with a live+ # successor run's state on the same worker.+ wf_run_id = workflow.info().run_id+# Set up per-workflow library if a library crate is presentlibrary_crate = workflow_arg.library_cratewf_library_id: str | None = None- # Per-workflow tracing state (declared before try for finally block access)+ # Per-workflow tracing state (declared before try for finally block access).+ # schedule_flush is the payload-pure flush gate, computed in the tracing setup+ # block below (see the determinism comments there and in the finally).event_log = None- wf_graph_tracer_manager: GraphTracerManager | None = Nonewf_tracer_key: str | None = None+ schedule_flush: bool = Falsetrace_context = workflow_arg.job_metadata.trace_contextpipe_output: PipeOutput | None = None@@ -91,10 +99,21 @@ class WfPipeRouter(WorkflowClass[PipeJob, PipeOutput]):workflow_registry = ClassRegistry()workflow_registry.register_classes_dict(global_registry.get_classes_dict())- # 2. Open library and attach registry to it+ # 2. Open library and attach registry to it. The library id is keyed by run_id —+ # replay-stable (a replay of this run sees the same run_id) but unique per run.+ # Keying by workflow_id would collide across runs: workflow ids are reused by+ # workflow-level retry_policy, Temporal reset, and resubmission of the same+ # pipeline_run_id (make_workflow_id is deterministic), so a closed predecessor+ # run's late eviction cleanup could tear down a live successor run's library.+ # With run_id keying, a pre-existing library under this id can only be THIS run's+ # own leftover — an evicted/interrupted execution whose finally never ran (H2):+ # reusing it would fingerprint-skip the crate load below against the fresh+ # registry, so the crate's dynamic classes never land in it and inline hydration+ # fails where history recorded success. open_fresh_library tears any such+ # leftover down — pure in-memory, no commands emitted, so it is replay-safe.library_manager = get_library_manager()- wf_library_id = f"wf_{workflow.info().workflow_id}"- _wf_library_id, wf_library = library_manager.open_library(library_id=wf_library_id)+ wf_library_id = f"wf_{wf_run_id}"+ wf_library = library_manager.open_fresh_library(library_id=wf_library_id)wf_library.set_class_registry(workflow_registry)set_current_library(library_id=wf_library_id)@@ -106,68 +125,74 @@ class WfPipeRouter(WorkflowClass[PipeJob, PipeOutput]):workflow_arg.working_memory = hydrate_working_memory(workflow_arg.working_memory_raw)workflow_arg.working_memory_raw = None- # Set up per-workflow graph tracing if enabled+ # Set up per-workflow graph tracing if enabled.pipeline_run_id = workflow_arg.job_metadata.pipeline_run_id- wf_workflow_id = workflow.info().workflow_id- tracing_config = get_config().pipelex.tracing_config- if tracing_config.is_enabled and trace_context is not None:- try:- # Use BufferingEventLog inside workflows (no I/O allowed).- # Events are flushed to the real backend via act_flush_trace_events.- event_log = BufferingEventLog()- wf_graph_tracer_manager = GraphTracerManager.get_or_create_instance()- wf_tracer_key = wf_workflow_id- wf_trace_context = wf_graph_tracer_manager.open_tracer(- graph_id=trace_context.graph_id,- data_inclusion=trace_context.data_inclusion,- # D5: only feed the event log to the tracer when graph events are wanted. In- # costs-only mode the tracer still mints node ids but emits no graph events;- # usage events flow via the report delegate's set_event_log below.- event_log=event_log if trace_context.emit_graph_events else None,- workflow_id=wf_workflow_id,+ # Determinism: whether tracing is set up — and therefore whether the finally+ # block schedules act_flush_trace_events — must be a pure function of the+ # workflow payload, never of worker-local state (which can differ between+ # the worker that recorded the history and the one replaying it, causing a+ # [TMPRL1100] nondeterminism error). The presence of trace_context IS the+ # submitter's decision; the worker-local tracing_config.is_enabled check+ # lives in the flush activity, where reading local config is legal. No+ # best-effort guard around this block: every step is pure in-memory state+ # (no I/O, no config reads) and open_tracer is collision-proof against+ # tracer keys leaked by a prior interrupted execution, so a failure here+ # is a real bug that must surface — and being inline-deterministic, it+ # re-fires identically on replay.+ if trace_context is not None:+ # Use BufferingEventLog inside workflows (no I/O allowed).+ # Events are flushed to the real backend via act_flush_trace_events.+ event_log = BufferingEventLog()+ wf_tracer_key = wf_run_id+ # Payload-pure flush gate: the buffer is populated exclusively by inline+ # (workflow-thread) emissions, and since the unified dry run moved leaf+ # mocking inside the activities, the only inline source left is graph+ # events from the tracer when emit_graph_events is on. Every usage+ # emission — LIVE, DRY, or mock-usage — happens activity-side via the+ # per-process runner fallback, so with graph events off the buffer is+ # deterministically empty and the flush round-trip is skipped. The gate+ # input rides in the payload, so the decision replays identically.+ schedule_flush = trace_context.emit_graph_events+ wf_trace_context = GraphTracerManager.get_or_create_instance().open_tracer(+ graph_id=trace_context.graph_id,+ data_inclusion=trace_context.data_inclusion,+ # D5: only feed the event log to the tracer when graph events are wanted. In+ # costs-only mode the tracer still mints node ids but emits no graph events;+ # usage events flow via the report delegate's set_event_log below.+ event_log=event_log if trace_context.emit_graph_events else None,+ workflow_id=wf_run_id,+ pipeline_run_id=pipeline_run_id,+ tracer_key=wf_tracer_key,+ # Threaded in so the returned context is born with the correct flags (no emit-flag+ # footgun in the model_copy below).+ emit_graph_events=trace_context.emit_graph_events,+ emit_usage_events=trace_context.emit_usage_events,+ )+ # Update job_metadata with the per-workflow trace_context (carries tracer_key + emit+ # flags), but preserve parent_node_id from the incoming context so CONTAINS edges link+ # back to the parent workflow's controller node.+ wf_trace_context = wf_trace_context.model_copy(+ update={"parent_node_id": trace_context.parent_node_id},+ )+ workflow_arg.job_metadata = workflow_arg.job_metadata.model_copy(+ update={"trace_context": wf_trace_context},+ )+ # Configure the report delegate for usage event emission — only when cost reporting+ # is on. In graph-only mode no usage context is registered, so usage events are+ # suppressed (the runner fallback also gates on emit_usage_events). Only emissions+ # from the workflow thread itself would land in this buffer — and since the unified+ # dry run moved leaf mocking inside the activities, no such inline source currently+ # exists: ReportingManager routes activity-side emissions to its per-process+ # fallback even when the activity runs co-located, so the buffer content stays a+ # deterministic function of inline execution and re-fires identically on replay.+ if trace_context.emit_usage_events:+ get_report_delegate().set_event_log(+ context_key=wf_tracer_key,+ event_log=event_log,+ workflow_id=wf_run_id,pipeline_run_id=pipeline_run_id,- tracer_key=wf_tracer_key,- # Threaded in so the returned context is born with the correct flags (no emit-flag- # footgun in the model_copy below).- emit_graph_events=trace_context.emit_graph_events,- emit_usage_events=trace_context.emit_usage_events,- )- # Update job_metadata with the per-workflow trace_context (carries tracer_key + emit- # flags), but preserve parent_node_id from the incoming context so CONTAINS edges link- # back to the parent workflow's controller node.- wf_trace_context = wf_trace_context.model_copy(- update={"parent_node_id": trace_context.parent_node_id},)- workflow_arg.job_metadata = workflow_arg.job_metadata.model_copy(- update={"trace_context": wf_trace_context},- )- # Configure the report delegate for usage event emission — only when cost reporting- # is on. In graph-only mode no usage context is registered, so usage events are- # suppressed (the runner fallback also gates on emit_usage_events).- if trace_context.emit_usage_events:- get_report_delegate().set_event_log(- context_key=wf_workflow_id,- event_log=event_log,- workflow_id=wf_workflow_id,- pipeline_run_id=pipeline_run_id,- )- except Exception as exc: # noqa: BLE001- # Best-effort: per-workflow tracing setup must never fail the workflow — log and continue without it.- workflow_log.warning(f"Failed to set up per-workflow tracing, continuing without: {exc}")- # Clean up partially initialized resources before nulling (the finally block- # won't be able to clean up after we null these references)- if wf_graph_tracer_manager is not None and wf_tracer_key is not None:- try:- wf_graph_tracer_manager.close_tracer(wf_tracer_key)- except Exception as tracer_exc: # noqa: BLE001- # Best-effort cleanup: closing a partially-initialized tracer must not mask the setup failure.- workflow_log.warning(f"Failed to close partially initialized tracer: {tracer_exc}")- if event_log is not None:- event_log.close()- get_report_delegate().clear_event_log(context_key=wf_workflow_id)- event_log = None- wf_graph_tracer_manager = Noneworking_memory = workflow_arg.get_working_memory()pipe_output = await pipe.run_pipe(@@ -197,23 +222,66 @@ class WfPipeRouter(WorkflowClass[PipeJob, PipeOutput]):raiseraise TemporalError.from_message_exception(exc=exc, force_non_retryable=True) from excfinally:- # Close per-workflow graph tracer (collects in-memory graph spec). F1: only assign the spec- # when graph events were requested — in costs-only mode close_tracer returns None (teardown- # skips the spec build), and this guard keeps the contract explicit even if that changes.- if wf_graph_tracer_manager is not None and wf_tracer_key is not None:+ # ALL worker-local cleanup in this finally is synchronous and runs before the+ # awaited flush activity at the end. Eviction safety (H2): the flush await is a+ # suspension point where a BaseException can be raised (_WorkflowBeingEvictedError+ # on sticky-cache eviction) — it escapes the `except Exception` around the flush+ # and aborts the rest of this finally block. Worker-local cleanup (tracer,+ # event-log context, per-workflow library + crate fingerprint) must therefore run+ # BEFORE that await, so an eviction can no longer leak state keyed by the+ # deterministic wf_{run_id} into the worker-local singletons and poison a+ # same-worker replay. (Workflow cancellation is NOT such an interruption: at an+ # activity await the SDK surfaces it as an ActivityError — an Exception — which+ # the best-effort except around the flush swallows.)++ # Close per-workflow graph tracer (collects in-memory graph spec). In costs-only+ # mode close_tracer returns None by contract (teardown skips the spec build), so+ # the graph_spec gate alone keeps costs-only outputs spec-free.+ if wf_tracer_key is not None:try:- graph_spec = wf_graph_tracer_manager.close_tracer(wf_tracer_key)- if graph_spec is not None and pipe_output is not None and trace_context is not None and trace_context.emit_graph_events:+ graph_spec = GraphTracerManager.get_or_create_instance().close_tracer(wf_tracer_key)+ if graph_spec is not None and pipe_output is not None:pipe_output.graph_spec = graph_specexcept Exception as tracer_exc: # noqa: BLE001# Best-effort: tracer close in the finally block must never fail the workflow — log and continue.workflow_log.warning(f"Failed to close per-workflow tracer: {tracer_exc}")+ # Clear stale event log state from the report delegate, under the SAME key+ # set_event_log registered it with (clear_event_log is a no-op pop when no+ # context was registered, e.g. graph-only mode).+ get_report_delegate().clear_event_log(context_key=wf_tracer_key)++ if wf_library_id is not None:+ try:+ get_library_manager().teardown(library_id=wf_library_id)+ except Exception as teardown_exc: # noqa: BLE001+ # Best-effort: library teardown in the finally block must never fail the+ # workflow or skip the remaining worker-local cleanup below — log and+ # continue. LibraryManager forgets the entry pop-first even when the+ # library's own teardown raises, so no poisoned state remains.+ workflow_log.warning(f"Failed to tear down per-workflow library: {teardown_exc}")+ finally:+ clear_current_library()- # Flush trace events and clean up event logif event_log is not None:- # Drain buffered events and flush to the real backend via activitybuffered_events = event_log.drain()- if buffered_events:+ event_log.close()++ # The awaited flush comes LAST: an interruption here can only skip the flush+ # itself, never the worker-local cleanup above — and on replay the buffer is+ # rebuilt deterministically from inline execution, so the flush re-fires with+ # the same content.+ #+ # Determinism (H1): the schedule is a pure function of the payload — the+ # event_log sentinel (trace_context presence) and schedule_flush+ # (emit_graph_events, a payload field). It must NOT be gated on the buffer+ # CONTENT: anything written into the buffer from outside the workflow+ # thread would not be re-written on replay (activities do not re-execute),+ # so a content-gated schedule recorded in history could vanish from the+ # replayed command stream after a routine sticky-cache eviction+ # ([TMPRL1100]). The payload-pure schedule_flush gate only skips+ # graph-events-off runs, where the buffer is deterministically empty (see+ # the tracing setup block). The activity no-ops on an empty list either way.+ if schedule_flush:try:await workflow.execute_activity(act_flush_trace_events,@@ -224,18 +292,12 @@ class WfPipeRouter(WorkflowClass[PipeJob, PipeOutput]):except Exception as flush_exc: # noqa: BLE001# Best-effort: trace-event flush in the finally block must never fail the workflow — log and continue.workflow_log.warning(f"Failed to flush trace events: {flush_exc}")-- event_log.close()-- # Clear stale event log state from the report delegate- if wf_tracer_key is not None:- get_report_delegate().clear_event_log(context_key=wf_tracer_key)-- if wf_library_id is not None:- try:- get_library_manager().teardown(library_id=wf_library_id)- finally:- clear_current_library()+ elif buffered_events:+ # Tripwire, not a guard: with graph events off the buffer is deterministically+ # empty by the invariant above, so this branch is unreachable today. If a+ # future inline emission path breaks that invariant, these events would be+ # silently discarded — turn that into a log line instead.+ workflow_log.warning(f"Discarding {len(buffered_events)} buffered trace events: flush skipped by the graph-events-off gate")# Dehydrate PipeOutput for Temporal transit: serialize WorkingMemory to# raw dict so the parent's data converter can deserialize without needingdiff --git a/pipelex/tracing/activity_event_log.py b/pipelex/tracing/activity_event_log.pyindex 262051c66..7df6165b8 100644--- a/pipelex/tracing/activity_event_log.py+++ b/pipelex/tracing/activity_event_log.py@@ -1,12 +1,13 @@-"""Process-local cached event log for runner-side activity emission.+"""Process-local cached event log for activity-side usage emission.-When activities run on a separate worker pool from the workflow router,-the router's `set_event_log` never registers a context on the runner's-ReportingManager — the runner-side `_event_log_contexts` dict is permanently-empty for every workflow that runs there. This module provides the per-process-backstop: a lazily-constructed event log, stamped with a stable-`act_{pid}_{uuid8}` writer_id, that emits into the same backend partition as-the rest of the run.+Every usage emission that happens inside a Temporal activity goes through this+per-process event log — by design, whether the activity runs co-located with+the workflow router or on a separate worker pool. Activities do not re-execute+on replay, so routing their emissions into the workflow's in-sandbox buffer+would make the buffer content depend on whether activities actually ran+(replay-determinism breach, audit finding H1). The per-process log, stamped+with a stable `act_{pid}_{uuid8}` writer_id, emits into the same backend+partition as the rest of the run.The cache-and-writer-id construction is guarded by `threading.Lock` withdouble-checked locking so concurrent first-emitters from N activity threads@@ -36,7 +37,7 @@ class ActivityEventLogCache:_lock: ClassVar[threading.Lock] = threading.Lock()_cached_event_log: ClassVar[EventLogProtocol | None] = None_writer_id: ClassVar[str | None] = None- _warning_emitted: ClassVar[bool] = False+ _fallback_logged: ClassVar[bool] = False_atexit_registered: ClassVar[bool] = False@classmethod@@ -76,26 +77,28 @@ class ActivityEventLogCache:return cls._cached_event_log@classmethod- def warn_once_runner_fallback_engaged(cls, workflow_id: str, writer_id: str) -> None:- """Log a single WARNING the first time the runner fallback engages.-- The warning is per-process; subsequent fallback emissions from the same- process are silent. Operators see one unmistakable signal in the worker- logs that runner-side emission is in use, without WARNING spam at high- activity throughput.+ def log_once_runner_fallback_engaged(cls, workflow_id: str, writer_id: str) -> None:+ """Log a single INFO the first time activity-side emission engages.++ The log is per-process; subsequent emissions from the same process are+ silent. This is the normal path for every activity-side usage emission+ (co-located or split deployment alike) — INFO level, one line, so+ operators can see the writer_id in the worker logs without log spam at+ high activity throughput."""- if cls._warning_emitted:+ if cls._fallback_logged:returnwith cls._lock:- if cls._warning_emitted:+ if cls._fallback_logged:return- cls._warning_emitted = True- log.warning(- f"Runner-side usage event emission engaged "+ cls._fallback_logged = True+ log.info(+ f"Activity-side usage event emission engaged "f"(workflow_id={workflow_id}, writer_id={writer_id}). "- "Activity is emitting into the per-process activity event log because "- "no _event_log_contexts entry was registered in this process. "- "This is expected when activities run on a separate worker pool from the workflow router."+ "Activities always emit through the per-process activity event log, by design: "+ "activity emissions must never land in the workflow's replay-rebuilt buffer "+ "(replay determinism), whether the activity runs co-located with the workflow "+ "router or on a separate worker pool.")@classmethod@@ -103,7 +106,7 @@ class ActivityEventLogCache:"""Reset cache state. For test fixtures only — never call from production code.Closes any cached event log, clears the writer_id, and resets the one-shot- warning flag so subsequent tests observe a fresh process-like state. The+ log flag so subsequent tests observe a fresh process-like state. Theatexit registration flag is intentionally left alone — the registrationis process-global and the handler is a no-op once the cache is cleared."""@@ -114,7 +117,7 @@ class ActivityEventLogCache:passcls._cached_event_log = Nonecls._writer_id = None- cls._warning_emitted = False+ cls._fallback_logged = False@classmethoddef _close_atexit(cls) -> None:diff --git a/tests/integration/pipelex/pipeline/test_pipeline_run_id_resubmission.py b/tests/integration/pipelex/pipeline/test_pipeline_run_id_resubmission.pynew file mode 100644index 000000000..3002f85b1--- /dev/null+++ b/tests/integration/pipelex/pipeline/test_pipeline_run_id_resubmission.py@@ -0,0 +1,231 @@+"""Per-run ``PipelineManager`` entry lifecycle across a full run — serial resubmission support.++Pins that the registry entry created by ``pipeline_run_setup`` is freed on every exit path,+so a later run can resubmit the same explicit ``pipeline_run_id`` (the hosted runner threads a+client-supplied id into ``pipeline_run_setup``; before this fix the entry was process-permanent+and every resubmission raised ``PipelineManagerAlreadyExistsError``):++- ``execute_pipeline`` success: the runner's ``finally`` removes the entry.+- ``execute_pipeline`` failure (pipe-run raises after setup succeeded): same ``finally`` removes it.+- ``pipeline_run_setup`` failure after registration (pipe missing from the bundle): setup removes+ its OWN registration — the runner never learns the id on this path, so setup must self-clean —+ and a resubmission of the same explicit id then succeeds.++Only genuinely concurrent same-id runs still collide in ``add_new_pipeline`` — that raise is+load-bearing (it shields the live direct-mode tracer from ``open_tracer``'s pop-and-replace+healing) and is pinned by ``tests/unit/pipelex/pipeline/test_pipeline_manager.py``.++These cases run fully dry (``PipeRunMode.DRY`` + ``mock_inputs``) so no inference happens.+"""++from __future__ import annotations++from typing import TYPE_CHECKING++import pytest+from typing_extensions import override++from pipelex.config import get_config+from pipelex.graph.graph_tracer import GraphTracer+from pipelex.hub import (+ clear_current_library,+ get_current_library_id_or_none,+ get_library_manager,+ get_pipeline_manager,+)+from pipelex.libraries.pipe.exceptions import PipeNotFoundError+from pipelex.pipe_run.exceptions import PipeRouterError+from pipelex.pipe_run.pipe_run_mode import PipeRunMode+from pipelex.pipe_run.pipe_run_protocol import PipeRunProtocol+from pipelex.pipeline.exceptions import PipelineExecutionError+from pipelex.pipeline.pipeline_factory import PipelineFactory+from pipelex.pipeline.pipeline_run_setup import pipeline_run_setup+from pipelex.pipeline.runner import PipelexRunner++if TYPE_CHECKING:+ from pytest_mock import MockerFixture++ from pipelex.core.pipes.pipe_output import PipeOutput+ from pipelex.pipe_run.delivery_assignment import DeliveryAssignment+ from pipelex.pipe_run.pipe_job import PipeJob+ from pipelex.system.configuration.configs import PipelineExecutionConfig++_RESUBMISSION_DOMAIN = "run_id_resubmission"+_RESUBMISSION_MTHDS = f"""+domain = "{_RESUBMISSION_DOMAIN}"+description = "Minimal bundle for pipeline_run_id resubmission tests"++[concept.Topic]+description = "A topic"++[concept.Topic.structure]+name = {{ type = "text", description = "Topic name" }}++[pipe.echo_topic]+type = "PipeLLM"+description = "Pipe used only to exercise the per-run registry lifecycle"+inputs = {{ subject = "Text" }}+output = "Topic"+prompt = "Echo the $subject as a topic"+"""+++def _dry_mock_config() -> PipelineExecutionConfig:+ return get_config().pipelex.pipeline_execution_config.with_execution_overrides(+ generate_graph=False,+ mock_inputs=True,+ )+++class _FailingPipeRun(PipeRunProtocol):+ """A PipeRun whose ``run`` always raises, so the failure lands in ``execute_pipeline``'s+ finally AFTER setup has succeeded and registered the run — exercising the runner-side removal.+ """++ @override+ async def run(self, pipe_job: PipeJob, delivery_assignment: DeliveryAssignment | None = None) -> PipeOutput:+ msg = "Injected pipe-run failure to exercise registry-entry removal."+ raise PipeRouterError(+ message=msg,+ run_mode=PipeRunMode.DRY,+ pipe_code=pipe_job.pipe.code,+ output_name=pipe_job.output_name,+ pipe_stack=[],+ )++[email protected](loop_scope="class")+class TestPipelineRunIdResubmission:+ async def test_success_clears_registry_entry(self) -> None:+ runner = PipelexRunner(+ pipe_run_mode=PipeRunMode.DRY,+ execution_config=_dry_mock_config(),+ )+ response = await runner.execute_pipeline(+ pipe_code="echo_topic",+ mthds_contents=[_RESUBMISSION_MTHDS],+ )+ assert get_pipeline_manager().get_optional_pipeline(pipeline_run_id=response.pipeline_run_id) is None++ async def test_failure_clears_registry_entry(self, mocker: MockerFixture) -> None:+ # Spy on the factory to capture the run id minted during the failing run — the+ # PipelineExecutionError doesn't carry it (the manager is a pydantic RootModel, which+ # rejects spy attribute injection).+ manager = get_pipeline_manager()+ mint_spy = mocker.spy(PipelineFactory, "make_pipeline")+ runner = PipelexRunner(+ pipe_run_mode=PipeRunMode.DRY,+ execution_config=_dry_mock_config(),+ pipe_run=_FailingPipeRun(),+ )+ with pytest.raises(PipelineExecutionError):+ await runner.execute_pipeline(+ pipe_code="echo_topic",+ mthds_contents=[_RESUBMISSION_MTHDS],+ )+ minted_run_id = mint_spy.spy_return.pipeline_run_id+ # The runner's finally removed the failing run's entry — same id can resubmit.+ assert manager.get_optional_pipeline(pipeline_run_id=minted_run_id) is None++ async def test_setup_failure_self_cleans_and_same_id_resubmits(self) -> None:+ explicit_run_id = "resubmission-test-run-id"+ # First submission fails AFTER registration: the pipe is absent from the bundle, so+ # get_required_pipe raises inside pipeline_run_setup's try — the runner never learns the+ # id on this path, so setup must remove its own registration.+ with pytest.raises(PipeNotFoundError):+ await pipeline_run_setup(+ execution_config=_dry_mock_config(),+ mthds_contents=[_RESUBMISSION_MTHDS],+ pipe_code="absent_pipe",+ pipe_run_mode=PipeRunMode.DRY,+ pipeline_run_id=explicit_run_id,+ )+ assert get_pipeline_manager().get_optional_pipeline(pipeline_run_id=explicit_run_id) is None++ # Resubmission of the SAME explicit id now succeeds (before the fix: process-permanent+ # PipelineManagerAlreadyExistsError).+ pipe_job, pipeline_run_id, library_id = await pipeline_run_setup(+ execution_config=_dry_mock_config(),+ mthds_contents=[_RESUBMISSION_MTHDS],+ pipe_code="echo_topic",+ pipe_run_mode=PipeRunMode.DRY,+ pipeline_run_id=explicit_run_id,+ )+ try:+ assert pipeline_run_id == explicit_run_id+ assert pipe_job.pipe.code == "echo_topic"+ assert get_pipeline_manager().get_optional_pipeline(pipeline_run_id=explicit_run_id) is not None+ finally:+ # Caller-side teardown (pipeline_run_setup leaves the library open and the run+ # registered on success — execute_pipeline's finally owns this in production).+ get_pipeline_manager().remove_pipeline(pipeline_run_id=explicit_run_id)+ get_library_manager().teardown(library_id=library_id)+ clear_current_library()++ async def test_setup_failure_frees_entry_even_when_tracer_teardown_raises(self, mocker: MockerFixture) -> None:+ """A raise inside the error-path tracer teardown must not strand the registry entry nor mask the original failure.++ ``close_tracer`` runs before ``remove_pipeline`` in setup's failure cleanup (that ordering+ is load-bearing — see the module docstring), and ``GraphTracer.teardown`` can raise: it+ closes the event log, and a file-backed transport's ``close()`` flushes to disk. If that+ raise skipped ``remove_pipeline`` and the library restore, the entry would be+ process-permanent again — exactly the bug this module exists to pin. And it must not+ replace the original setup failure as the propagating exception either: callers type-match+ (the runner wraps ``PipelexError`` into ``PipelineExecutionError``, the API maps error+ types to status codes), so the flush error is logged and suppressed, mirroring+ ``PipeRun.run``'s close_tracer handling.+ """+ explicit_run_id = "resubmission-teardown-raise-run-id"+ graph_config = get_config().pipelex.pipeline_execution_config.with_execution_overrides(+ generate_graph=True,+ mock_inputs=True,+ )+ outer_library_id = get_current_library_id_or_none()++ # Fail setup AFTER open_tracer so the cleanup path actually closes a tracer.+ prepare_failure_msg = "Injected post-tracer setup failure"+ prepare_mock = mocker.patch(+ "pipelex.pipeline.pipeline_run_setup.prepare_pipe_job",+ side_effect=RuntimeError(prepare_failure_msg),+ )+ # Simulate the event-log flush failing on close (e.g. NDJSON file handle on a full disk).+ teardown_mock = mocker.patch.object(+ GraphTracer,+ "teardown",+ side_effect=OSError("Injected event-log flush failure on close"),+ )++ # The ORIGINAL setup failure propagates — the flush error must not replace it.+ with pytest.raises(RuntimeError, match=prepare_failure_msg):+ await pipeline_run_setup(+ execution_config=graph_config,+ mthds_contents=[_RESUBMISSION_MTHDS],+ pipe_code="echo_topic",+ pipe_run_mode=PipeRunMode.DRY,+ pipeline_run_id=explicit_run_id,+ )++ # The suppression path was actually exercised (the cleanup did close a tracer and the+ # injected flush failure did fire) — otherwise the RuntimeError assert above is vacuous.+ teardown_mock.assert_called_once()+ # The teardown raise must not strand the entry nor clobber the caller's current-library.+ assert get_pipeline_manager().get_optional_pipeline(pipeline_run_id=explicit_run_id) is None+ assert get_current_library_id_or_none() == outer_library_id++ # Resubmission of the SAME explicit id succeeds once the fault is gone.+ mocker.stop(prepare_mock)+ mocker.stop(teardown_mock)+ pipe_job, pipeline_run_id, library_id = await pipeline_run_setup(+ execution_config=_dry_mock_config(),+ mthds_contents=[_RESUBMISSION_MTHDS],+ pipe_code="echo_topic",+ pipe_run_mode=PipeRunMode.DRY,+ pipeline_run_id=explicit_run_id,+ )+ try:+ assert pipeline_run_id == explicit_run_id+ assert pipe_job.pipe.code == "echo_topic"+ finally:+ get_pipeline_manager().remove_pipeline(pipeline_run_id=explicit_run_id)+ get_library_manager().teardown(library_id=library_id)+ clear_current_library()diff --git a/tests/integration/pipelex/temporal/conftest.py b/tests/integration/pipelex/temporal/conftest.pyindex 880f2967e..75d23e027 100644--- a/tests/integration/pipelex/temporal/conftest.py+++ b/tests/integration/pipelex/temporal/conftest.py@@ -21,6 +21,7 @@ from pipelex.temporal.temporal_hub import temporal_hubfrom pipelex.temporal.temporal_task_manager import TemporalTaskManagerfrom pipelex.temporal.tprl.namespace_check import RegistrationFailure, ensure_required_search_attributes_registeredfrom pipelex.test_extras.shared_pytest_plugins import ClassRegistryMode+from pipelex.tracing.activity_event_log import ActivityEventLogCacheTEMPORAL_SERVER_NONE = "none"TEMPORAL_SERVER_TIME_SKIPPING = "time-skipping"@@ -158,6 +159,23 @@ def boot_temporal(reset_pipelex_config_fixture: None) -> Generator[None, None, Nget_plugin_manager().plugin_sdk_registry.teardown()[email protected](autouse=True)+def reset_activity_event_log_cache() -> Generator[None, None, None]:+ """Reset the per-process activity event log cache around every temporal test.++ ReportingManager routes every activity-side usage emission through the per-process+ fallback (even co-located activities — see audit finding H1), and the fallback's+ event log is process-cached on ``ActivityEventLogCache`` with the traces_dir it was+ first built with. Without this reset, a cache created by an earlier test would keep+ writing into that test's directory and the current test's events would silently+ land elsewhere. Hoisted to this conftest because any temporal test whose activities+ report usage engages the cache, not just the ones under ``tracing/``.+ """+ ActivityEventLogCache.reset_for_tests()+ yield+ ActivityEventLogCache.reset_for_tests()++@pytest_asyncio.fixture(scope="session") # pyright: ignore[reportUntypedFunctionDecorator, reportUnknownMemberType]async def env(request: FixtureRequest) -> AsyncGenerator[WorkflowEnvironment, None]:"""Temporal test environment, shared across all temporal tests.diff --git a/tests/integration/pipelex/temporal/library_crate/helpers.py b/tests/integration/pipelex/temporal/library_crate/helpers.pyindex 5a7347b9a..bc419d6b7 100644--- a/tests/integration/pipelex/temporal/library_crate/helpers.py+++ b/tests/integration/pipelex/temporal/library_crate/helpers.py@@ -5,12 +5,41 @@ from typing import Anyfrom temporalio.client import Client as TemporalClient+from pipelex.core.memory.working_memory import WorkingMemoryfrom pipelex.core.pipes.pipe_output import PipeOutputfrom pipelex.core.stuffs.structured_content import StructuredContent+from pipelex.core.stuffs.stuff import Stuff+from pipelex.core.stuffs.stuff_content_factory import StuffContentFactoryfrom pipelex.core.stuffs.text_content import TextContent+from pipelex.hub import get_required_conceptfrom pipelex.pipe_run.pipe_job import PipeJobfrom pipelex.runtime_bridge.primitives.submitter_hydration import rehydrate_pipe_output_with_cratefrom pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter+from tests.integration.pipelex.temporal.test_data import DeferredHydrationTestData+++def make_prepared_greeting_job(pipe_job: PipeJob, stuff_code: str) -> PipeJob:+ """Copy ``pipe_job`` with an input WorkingMemory carrying a dynamic-concept Greeting+ stuff, dehydrated for Temporal transit via ``prepare_for_temporal``.++ The Greeting concept comes from the deferred-hydration bundle, so the receiving+ workflow must hydrate the stuff inline after loading the crate — the deferred+ hydration path.+ """+ greeting_concept = get_required_concept(concept_ref=f"{DeferredHydrationTestData.DOMAIN}.Greeting")+ greeting_content = StuffContentFactory.make_stuff_content_from_concept_required(+ concept=greeting_concept,+ value={"message": "Bonjour le monde", "language": "French"},+ )+ greeting_stuff = Stuff(+ stuff_code=stuff_code,+ stuff_name="greeting_result",+ concept=greeting_concept,+ content=greeting_content,+ )+ input_memory = WorkingMemory()+ input_memory.root["greeting_result"] = greeting_stuff+ return pipe_job.model_copy(update={"working_memory": input_memory}).prepare_for_temporal()def rehydrate_pipe_output(pipe_output: PipeOutput, pipe_job: PipeJob | None = None) -> PipeOutput:diff --git a/tests/integration/pipelex/temporal/library_crate/test_wf_deferred_hydration.py b/tests/integration/pipelex/temporal/library_crate/test_wf_deferred_hydration.pyindex a04f3574e..2a8cbae48 100644--- a/tests/integration/pipelex/temporal/library_crate/test_wf_deferred_hydration.py+++ b/tests/integration/pipelex/temporal/library_crate/test_wf_deferred_hydration.py@@ -11,18 +11,14 @@ from collections.abc import Generatorimport pytestfrom temporalio.client import Client as TemporalClient-from pipelex.core.memory.working_memory import WorkingMemoryfrom pipelex.core.pipes.pipe_output import PipeOutputfrom pipelex.core.stuffs.structured_content import StructuredContent-from pipelex.core.stuffs.stuff import Stuff-from pipelex.core.stuffs.stuff_content_factory import StuffContentFactoryfrom pipelex.core.stuffs.text_content import TextContent-from pipelex.hub import get_required_conceptfrom pipelex.pipe_run.pipe_job import PipeJobfrom pipelex.temporal.temporal_hub import get_task_managerfrom pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouterfrom tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle-from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output+from tests.integration.pipelex.temporal.library_crate.helpers import make_prepared_greeting_job, rehydrate_pipe_outputfrom tests.integration.pipelex.temporal.test_data import DeferredHydrationTestData@@ -102,26 +98,9 @@ class TestWfDeferredHydration:is serialized via prepare_for_temporal(), sent through Temporal, and correctlyhydrated on the worker before the pipe sequence runs."""- # Build a Stuff with the dynamic Greeting concept content- greeting_concept = get_required_concept(concept_ref=f"{DeferredHydrationTestData.DOMAIN}.Greeting")- greeting_content = StuffContentFactory.make_stuff_content_from_concept_required(- concept=greeting_concept,- value={"message": "Bonjour le monde", "language": "French"},- )- greeting_stuff = Stuff(- stuff_code="test_input",- stuff_name="greeting_result",- concept=greeting_concept,- content=greeting_content,- )-- # Create a WorkingMemory with the dynamic-concept Stuff pre-populated- input_wm = WorkingMemory()- input_wm.root["greeting_result"] = greeting_stuff-- # Copy the pipe job with pre-populated WM, then prepare for Temporal dispatch- pipe_job_with_input = pipe_job_with_dynamic_concept.model_copy(update={"working_memory": input_wm})- prepared_job = pipe_job_with_input.prepare_for_temporal()+ # Build a WorkingMemory with a dynamic Greeting stuff pre-populated, then+ # prepare the pipe job for Temporal dispatch+ prepared_job = make_prepared_greeting_job(pipe_job_with_dynamic_concept, stuff_code="test_input")# Verify prepare_for_temporal converted WM to raw dictassert prepared_job.working_memory is None, "prepare_for_temporal should set working_memory to None"diff --git a/tests/integration/pipelex/temporal/test_wf_pipe_router_costs_only_flush_nondeterminism.py b/tests/integration/pipelex/temporal/test_wf_pipe_router_costs_only_flush_nondeterminism.pynew file mode 100644index 000000000..3bbe1eb3c--- /dev/null+++ b/tests/integration/pipelex/temporal/test_wf_pipe_router_costs_only_flush_nondeterminism.py@@ -0,0 +1,211 @@+"""Regression guard for the [TMPRL1100] costs-only flush nondeterminism in ``WfPipeRouter``.++The bug (finding H1 in ``wip/distributed-execution/workflow-nondeterminism-audit.md``): in+costs-only tracing mode (``emit_usage_events=True, emit_graph_events=False``) the workflow's+``BufferingEventLog`` is populated exclusively by co-located *activity* completions — the+workflow registers the buffer in the process-global ``ReportingManager`` via ``set_event_log``,+and the activity-side ``report_inference_job`` writes into it from the activity thread. The+workflow's ``finally`` block then gates the ``act_flush_trace_events`` schedule on+``if buffered_events:``, a value that is NOT a pure function of the workflow payload + history.+Live execution drains a non-empty buffer (the co-located activity ran and emitted), so history+records the flush schedule. On replay, activity results come from history without executing,+the freshly recreated buffer stays empty, and no flush command is emitted — the command streams+diverge and Temporal raises ``[TMPRL1100] Nondeterminism error``. In production this fires on a+routine sticky-cache eviction on the SAME worker, no config drift or fleet heterogeneity+required: the silent-hang class.++How the reproduction works:++1. A leaf PipeLLM job runs in LIVE mode with a costs-only trace context. ``act_llm_gen_text``+ is substituted with a stub that synthesizes token usage and reports it through+ ``report_inference_job`` — co-located with the workflow, so the emission takes the+ ReportingManager fast path into the workflow's registered buffer, exactly what a real+ co-located worker does. ``act_flush_trace_events`` is substituted with a no-op so the test+ does not depend on a tracing backend; its *schedule* is what matters.+2. The workflow runs to completion on a normal sticky-cache worker, so the live continuation+ drains a non-empty buffer and schedules the flush. The test asserts from the fetched+ history that the flush was actually scheduled, so it cannot pass vacuously.+3. The recorded history is replayed through ``temporalio.worker.Replayer`` — the exact code+ path a worker takes after a sticky-cache eviction. With the bug, the replay's buffer is+ empty, the flush command is never emitted, and the replay raises a nondeterminism error.+ The test asserts the replay succeeds: RED while the bug is present (TDD).++The ``graph-and-usage`` control arm runs the identical setup with graph events enabled: inline+tracer emissions re-fire deterministically on replay, the buffer is non-empty on both sides,+and the replay succeeds even with the bug present — proving that the costs-only buffer gating,+not the replay harness, is what breaks the command stream.++Fix shape (as landed): activity-side usage events never route through the workflow's+in-sandbox buffer (they take the per-process runner fallback), and the flush schedule is a+pure function of the payload. The payload-pure gate goes further than "always schedule": in+costs-only LIVE mode the buffer is deterministically empty (no graph events, no inline usage+emissions), so the flush activity is skipped entirely — the costs-only arm therefore asserts+the flush is ABSENT from history and that the recorded history still replays cleanly. The+``graph-and-usage`` arm keeps the original vacuity guard (flush present in history).+"""++import uuid+from collections.abc import Generator+from datetime import timedelta++import pytest+from temporalio import activity+from temporalio.client import Client as TemporalClient+from temporalio.worker import Replayer, UnsandboxedWorkflowRunner++from pipelex.cogt.content_generation.assignment_models import LLMAssignment+from pipelex.config import get_config+from pipelex.core.pipes.pipe_output import PipeOutput+from pipelex.graph.graph_tracer_manager import GraphTracerManager+from pipelex.hub import get_report_delegate+from pipelex.pipe_run.pipe_job import PipeJob+from pipelex.pipe_run.pipe_run_mode import PipeRunMode+from pipelex.temporal.temporal_data_converter import data_converter+from pipelex.temporal.temporal_hub import get_task_manager+from pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_text+from pipelex.temporal.tprl_pipe.act_flush_trace_events import act_flush_trace_events+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle+from tests.integration.pipelex.temporal.tracing.helpers import (+ act_flush_noop,+ inject_trace_context,+ make_synthetic_usage_llm_job,+ route_activities_to,+ scheduled_activity_names,+)+from tests.integration.pipelex.temporal.tracing.test_data import SequenceTracingTestData++_LEAF_PIPE_CODE = "step_one"+_FLUSH_ACTIVITY_NAME = "act_flush_trace_events"+_FAKE_INFERENCE_MODEL_NAME = "costs_only_replay_fake"+_FAKE_INFERENCE_MODEL_ID = "costs_only_replay_fake_id"+_FAKE_RESPONSE_TEXT = "costs-only replay guard fake response"++[email protected](name="act_llm_gen_text")+async def _act_llm_gen_text_reports_usage(llm_assignment: LLMAssignment) -> str: # noqa: RUF029+ """Substitute for ``act_llm_gen_text`` that emits usage from a CO-LOCATED activity.++ The point is that ``report_inference_job`` runs on the activity thread in the same+ process as the workflow's registered ``BufferingEventLog`` — exactly the co-located+ emission that pre-H1 took the ReportingManager fast path into the workflow's buffer+ (the cross-thread population that replay cannot reproduce). Post-H1 it must take the+ per-process activity event log instead.+ """+ synthetic_job = make_synthetic_usage_llm_job(+ llm_assignment=llm_assignment,+ inference_model_name=_FAKE_INFERENCE_MODEL_NAME,+ inference_model_id=_FAKE_INFERENCE_MODEL_ID,+ )+ get_report_delegate().report_inference_job(inference_job=synthetic_job)+ return _FAKE_RESPONSE_TEXT++[email protected](scope="class")+def live_leaf_tracing_job() -> Generator[PipeJob, None, None]:+ """LIVE-mode PipeJob for a leaf PipeLLM.++ LIVE pins the arm under test: costs-only LIVE is the shape whose flush schedule+ used to be gated on worker-local buffer content and hung on replay. (Since the+ unified dry run, DRY also dispatches ``act_llm_gen_text`` and emits activity-side,+ but the regression being guarded was recorded against LIVE.)+ """+ yield from pipe_job_from_bundle(+ bundle_file=SequenceTracingTestData.BUNDLE_FILE,+ pipe_code=_LEAF_PIPE_CODE,+ pipe_run_mode=PipeRunMode.LIVE,+ )++[email protected](loop_scope="class")+class TestWfPipeRouterCostsOnlyFlushNondeterminism:+ @pytest.fixture(autouse=True)+ def clear_graph_tracer_manager(self) -> Generator[None, None, None]:+ """Reset the process-lifetime tracer singleton so arms cannot leak tracer keys."""+ yield+ GraphTracerManager.clear_instance()++ @pytest.fixture(scope="class", autouse=True)+ def redirect_traces_dir(self, tmp_path_factory: pytest.TempPathFactory) -> Generator[None, None, None]:+ """Point the NDJSON traces_dir at a temp directory for this class.++ The stub activity's ``report_inference_job`` takes the per-process activity+ event log path, which lazily builds an NDJSON writer in the configured+ traces_dir — without this redirect the test writes stray trace files into the+ repo's default ``.pipelex/traces``.+ """+ ndjson_config = get_config().pipelex.tracing_config.ndjson+ original_dir = ndjson_config.traces_dir if ndjson_config else ""+ if ndjson_config:+ ndjson_config.traces_dir = str(tmp_path_factory.mktemp("traces"))+ yield+ if ndjson_config:+ ndjson_config.traces_dir = original_dir++ @pytest.mark.parametrize("emit_graph_events", [True, False], ids=["graph-and-usage", "costs-only"])+ async def test_recorded_history_replays_deterministically(+ self,+ temporal_client: TemporalClient,+ live_leaf_tracing_job: PipeJob,+ emit_graph_events: bool,+ ) -> None:+ """A history recorded with co-located activity usage emission must replay cleanly.++ With the bug, the ``costs-only`` arm raises a nondeterminism error from+ ``Replayer.replay_workflow``: the flush schedule recorded in history is not+ re-emitted because the replay's buffer is empty. Flush scheduling must be a+ pure function of the workflow payload + history.+ """+ execution_run_id = f"costs_only_replay_guard_{uuid.uuid4().hex[:12]}"+ execution_job = inject_trace_context(+ live_leaf_tracing_job,+ execution_run_id,+ emit_graph_events=emit_graph_events,+ emit_usage_events=True,+ )+ task_queue = f"q_costs_only_replay_{uuid.uuid4().hex[:8]}"+ workflow_id = f"wf_costs_only_replay_{uuid.uuid4().hex[:8]}"++ with route_activities_to(task_queue, [act_llm_gen_text.__name__]):+ async with get_task_manager().make_worker(+ temporal_client,+ task_queue=task_queue,+ is_not_sandboxed=True,+ substitute_activities={+ act_llm_gen_text: _act_llm_gen_text_reports_usage,+ act_flush_trace_events: act_flush_noop,+ },+ ):+ workflow_handle = await temporal_client.start_workflow( # pyright: ignore[reportUnknownMemberType]+ workflow=WfPipeRouter.run,+ arg=execution_job,+ id=workflow_id,+ task_queue=task_queue,+ # Safety net: a regression that turns the divergence into a+ # workflow-task retry loop must fail the test, not hang it.+ execution_timeout=timedelta(seconds=60),+ )+ pipe_output = await workflow_handle.result()+ assert isinstance(pipe_output, PipeOutput)+ history = await workflow_handle.fetch_history()++ scheduled_names = scheduled_activity_names(history)+ if emit_graph_events:+ # Vacuity guard for the replay assertion: the graph arm must record the flush+ # schedule (inline graph events populate the buffer deterministically).+ assert _FLUSH_ACTIVITY_NAME in scheduled_names, f"history must record the flush schedule, got: {scheduled_names}"+ else:+ # Costs-only LIVE: the buffer is deterministically empty (activity-side usage+ # emissions take the per-process runner fallback, never the workflow buffer),+ # so the payload-pure gate skips the guaranteed-empty flush round-trip.+ assert _FLUSH_ACTIVITY_NAME not in scheduled_names, f"costs-only LIVE must skip the empty flush schedule, got: {scheduled_names}"++ # Replay the recorded history — the exact code path of a post-eviction worker.+ # With the bug, the costs-only arm raises here with a nondeterminism error.+ replayer = Replayer(+ workflows=[WfPipeRouter],+ workflow_runner=UnsandboxedWorkflowRunner(),+ data_converter=data_converter,+ )+ await replayer.replay_workflow(history)diff --git a/tests/integration/pipelex/temporal/test_wf_pipe_router_eviction_cleanup_ordering.py b/tests/integration/pipelex/temporal/test_wf_pipe_router_eviction_cleanup_ordering.pynew file mode 100644index 000000000..ddb70e179--- /dev/null+++ b/tests/integration/pipelex/temporal/test_wf_pipe_router_eviction_cleanup_ordering.py@@ -0,0 +1,156 @@+"""Pin the eviction-safe ``finally`` ordering invariant of ``WfPipeRouter`` (H2 fix).++The invariant: in the workflow's ``finally`` block, ALL synchronous worker-local cleanup+(tracer close, report-delegate event-log context clear, per-run library teardown) runs+BEFORE the awaited ``act_flush_trace_events`` activity. The flush await is a suspension+point where eviction raises a ``BaseException`` (``_WorkflowBeingEvictedError``) that+escapes the ``except Exception`` around the flush and aborts the rest of the ``finally``.+If any cleanup sat after the await, an eviction there would leak state keyed by the+deterministic ``wf_{run_id}`` into the worker-local singletons.++Why eviction and not cancellation: workflow cancellation delivered at an activity await+is converted by the SDK into an ``ActivityError`` (an ``Exception``), which the+best-effort ``except`` around the flush swallows — the workflow then completes normally,+running whatever code follows. Cancellation therefore cannot abort the ``finally`` and+cannot exercise the invariant. Eviction is the real BaseException-at-suspension-point,+and worker shutdown evicts every cached run deterministically.++The reproduction:++1. ``act_flush_trace_events`` is substituted with a stub that signals entry and then+ blocks, parking the workflow exactly at the flush await — by which point the ordering+ invariant says all worker-local cleanup must already have run.+2. The test shuts the worker down (exits the worker context). Shutdown evicts the cached+ run: the workflow coroutine is resumed at the flush await with the eviction+ ``BaseException``, aborting the rest of the ``finally`` — exactly the H2 interruption.+3. The test then asserts the worker-local singletons are clean: no per-run library in+ ``LibraryManager``, no tracer under the run id, no report-delegate event-log context.++A regression that moves any cleanup after the flush await turns these assertions red.+"""++import asyncio+import uuid+from collections.abc import Generator+from datetime import timedelta+from typing import ClassVar++import pytest+from temporalio import activity+from temporalio.client import Client as TemporalClient++from pipelex.graph.graph_tracer_manager import GraphTracerManager+from pipelex.hub import get_library_manager, get_report_delegate+from pipelex.libraries.exceptions import LibraryError+from pipelex.pipe_run.pipe_job import PipeJob+from pipelex.reporting.reporting_manager import ReportingManager+from pipelex.temporal.temporal_hub import get_task_manager+from pipelex.temporal.tprl_pipe.act_flush_trace_events import FlushTraceEventsArg, act_flush_trace_events+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle+from tests.integration.pipelex.temporal.tracing.helpers import inject_trace_context, scheduled_activity_names+from tests.integration.pipelex.temporal.tracing.test_data import SequenceTracingTestData++_LEAF_PIPE_CODE = "step_one"+++class _FlushBlockSignals:+ """Events shared between the test coroutine and the blocking flush stub."""++ entered: ClassVar[asyncio.Event | None] = None+ release: ClassVar[asyncio.Event | None] = None++[email protected](name="act_flush_trace_events")+async def _act_flush_blocking(arg: FlushTraceEventsArg) -> None: # noqa: ARG001+ """Flush substitute that parks the workflow at its flush await."""+ assert _FlushBlockSignals.entered is not None+ assert _FlushBlockSignals.release is not None+ _FlushBlockSignals.entered.set()+ await _FlushBlockSignals.release.wait()++[email protected](scope="class")+def leaf_tracing_job() -> Generator[PipeJob, None, None]:+ """DRY-mode PipeJob for a leaf PipeLLM: the trace flush is the only activity."""+ yield from pipe_job_from_bundle(+ bundle_file=SequenceTracingTestData.BUNDLE_FILE,+ pipe_code=_LEAF_PIPE_CODE,+ )++[email protected](loop_scope="class")+class TestWfPipeRouterEvictionCleanupOrdering:+ @pytest.fixture(autouse=True)+ def clear_graph_tracer_manager(self) -> Generator[None, None, None]:+ """Reset the process-lifetime tracer singleton so runs cannot leak tracer keys."""+ yield+ GraphTracerManager.clear_instance()++ async def test_eviction_at_flush_await_leaks_no_worker_local_state(+ self,+ temporal_client: TemporalClient,+ leaf_tracing_job: PipeJob,+ ) -> None:+ """Evicting the run while it sits at the flush await must leave the worker-local+ singletons clean: all synchronous cleanup precedes the only await.+ """+ execution_job = inject_trace_context(leaf_tracing_job, f"eviction_ordering_guard_{uuid.uuid4().hex[:12]}")+ assert execution_job.library_crate is not None, "the job must carry a library crate so the workflow opens a per-run library"++ task_queue = f"q_eviction_ordering_{uuid.uuid4().hex[:8]}"+ _FlushBlockSignals.entered = asyncio.Event()+ _FlushBlockSignals.release = asyncio.Event()++ library_manager = get_library_manager()+ tracer_manager = GraphTracerManager.get_or_create_instance()+ report_delegate = get_report_delegate()+ assert isinstance(report_delegate, ReportingManager)++ try:+ async with get_task_manager().make_worker(+ temporal_client,+ task_queue=task_queue,+ is_not_sandboxed=True,+ substitute_activities={act_flush_trace_events: _act_flush_blocking},+ ):+ workflow_handle = await temporal_client.start_workflow( # pyright: ignore[reportUnknownMemberType]+ workflow=WfPipeRouter.run,+ arg=execution_job,+ id=f"wf_eviction_ordering_{uuid.uuid4().hex[:8]}",+ task_queue=task_queue,+ execution_timeout=timedelta(seconds=60),+ )+ run_id = workflow_handle.first_execution_run_id+ assert run_id is not None++ # The workflow now sits at the flush await — the only await in the+ # finally. Per the ordering invariant, all worker-local cleanup has+ # already run by the time the flush activity starts.+ await asyncio.wait_for(_FlushBlockSignals.entered.wait(), timeout=30)++ # Vacuity guard while the worker is still up: the flush schedule must be+ # in history, proving the workflow is parked at that await.+ history = await workflow_handle.fetch_history()+ assert "act_flush_trace_events" in scheduled_activity_names(history)++ # Exiting the worker context shuts the worker down, which evicts the+ # cached run: the coroutine resumes at the flush await with the eviction+ # BaseException and the rest of the finally is aborted — the H2 shape.++ # Post-eviction: no worker-local state may have leaked. With the bug+ # (cleanup ordered after the flush await), the per-run library, tracer,+ # and event-log context would all still be registered here.+ with pytest.raises(LibraryError):+ library_manager.get_library(library_id=f"wf_{run_id}")+ assert tracer_manager.get_tracer(run_id) is None, "tracer must be closed before the flush await"+ contexts = report_delegate._event_log_contexts # noqa: SLF001 # pyright: ignore[reportPrivateUsage]+ assert run_id not in contexts, "event-log context must be cleared before the flush await"++ # Tidy up the abandoned execution (no worker is listening anymore).+ await workflow_handle.terminate(reason="eviction-ordering guard done")+ finally:+ _FlushBlockSignals.release.set()+ _FlushBlockSignals.entered = None+ _FlushBlockSignals.release = Nonediff --git a/tests/integration/pipelex/temporal/test_wf_pipe_router_eviction_library_leak.py b/tests/integration/pipelex/temporal/test_wf_pipe_router_eviction_library_leak.pynew file mode 100644index 000000000..e90fc65c8--- /dev/null+++ b/tests/integration/pipelex/temporal/test_wf_pipe_router_eviction_library_leak.py@@ -0,0 +1,163 @@+"""Regression guard for the eviction-time library leak that poisons same-worker replay in ``WfPipeRouter``.++The bug (finding H2 in ``wip/distributed-execution/workflow-nondeterminism-audit.md``): the+worker-local library teardown sits in the workflow's ``finally`` block AFTER the awaited+``act_flush_trace_events`` activity. An eviction-time interruption raised at that await+(``_WorkflowBeingEvictedError`` is a ``BaseException``) bypasses the ``except Exception``+around the flush and aborts the rest of the ``finally``: the per-workflow library and its+crate fingerprint leak in the worker-local ``LibraryManager`` singleton, keyed by the+*deterministic, replay-stable* ``wf_{run_id}``. When the same worker then replays that+run, ``open_library`` returns the stale library, ``set_class_registry`` installs a+FRESH registry seeded only from the global one, and ``load_from_crate`` is+fingerprint-skipped — so the crate's dynamic classes are never registered into the new+registry and inline hydration fails where history recorded success: command-stream+divergence, and with dynamic-class activity results on a single-worker queue, a persistent+silent hang.++How the reproduction works:++Forcing a real eviction exactly during the flush await is racy, so the test installs the+*post-eviction leaked state* directly: it starts the workflow with no worker listening+(so the run id is known but nothing executes yet), performs the same setup steps the+interrupted predecessor execution would have performed — open the library under+``wf_{run_id}``, attach a scoped class registry, ``load_from_crate`` — and deliberately+skips teardown, leaving the library and the crate fingerprint behind in the+``LibraryManager``. Only then does it open the worker, so the workflow executes against+that leaked state with a payload whose+``working_memory_raw`` carries a dynamic-concept stuff (the deferred-hydration path).+With the bug, the fingerprint skip leaves the dynamic class unregistered in the fresh+per-workflow registry and the workflow fails inline at hydration. The test asserts the+workflow completes and hydrates correctly: RED while the bug is present (TDD).++The job is built with an isolated class registry on purpose: in shared mode the dynamic+classes also live in the global registry, the workflow's fresh per-workflow registry+inherits them on seeding, and the leak is masked.++Fix shape: make the per-workflow library setup self-healing against a leaked predecessor+under the same deterministic id — scope the fingerprint dedup to the registry instance,+or force-teardown any pre-existing library before opening — and additionally make the+``finally`` cleanup eviction-safe.+"""++import uuid+from collections.abc import Generator+from contextlib import suppress+from datetime import timedelta++import pytest+from kajson.class_registry import ClassRegistry+from kajson.kajson_manager import KajsonManager+from temporalio.client import Client as TemporalClient++from pipelex.core.pipes.pipe_output import PipeOutput+from pipelex.core.stuffs.structured_content import StructuredContent+from pipelex.hub import get_current_library, get_library_manager, set_current_library+from pipelex.libraries.exceptions import LibraryError+from pipelex.pipe_run.pipe_job import PipeJob+from pipelex.temporal.temporal_hub import get_task_manager+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle+from tests.integration.pipelex.temporal.library_crate.helpers import make_prepared_greeting_job, rehydrate_pipe_output+from tests.integration.pipelex.temporal.test_data import DeferredHydrationTestData++[email protected](scope="class")+def isolated_hydration_job() -> Generator[PipeJob, None, None]:+ """PipeJob from the dynamic-concept bundle with a library-scoped class registry.++ Isolation is required for the reproduction (not parametrized on purpose): with the+ shared registry the workflow's per-workflow registry inherits the dynamic classes+ from the global one and the fingerprint-skip bug cannot manifest.+ """+ yield from pipe_job_from_bundle(+ bundle_file=DeferredHydrationTestData.BUNDLE_FILE,+ pipe_code=DeferredHydrationTestData.PIPE_CODE,+ isolated_registry=True,+ )++[email protected](loop_scope="class")+class TestWfPipeRouterEvictionLibraryLeak:+ async def test_workflow_recovers_from_leaked_predecessor_library(+ self,+ temporal_client: TemporalClient,+ isolated_hydration_job: PipeJob,+ ) -> None:+ """A leaked predecessor library under the same ``wf_{run_id}`` must not poison the run.++ With the bug, ``load_from_crate`` is fingerprint-skipped against the leaked+ library while ``set_class_registry`` already swapped in a fresh registry, so+ deferred hydration of the dynamic Greeting stuff fails inline and the workflow+ fails terminally instead of completing.+ """+ # Build an input WorkingMemory carrying a dynamic-concept stuff, then dehydrate it+ # for Temporal transit so the workflow must hydrate it inline after crate loading.+ prepared_job = make_prepared_greeting_job(isolated_hydration_job, stuff_code="leak_input")+ assert prepared_job.working_memory_raw is not None, "prepare_for_temporal must populate working_memory_raw"+ library_crate = prepared_job.library_crate+ assert library_crate is not None, "the dynamic-concept job must carry a library crate"++ workflow_id = f"wf_eviction_leak_{uuid.uuid4().hex[:8]}"+ task_queue = f"q_eviction_leak_{uuid.uuid4().hex[:8]}"++ # Start the workflow with NO worker listening on the task queue: the server+ # assigns the run id immediately but nothing executes until the worker opens+ # below. The per-run library id is derived from that run id (run-scoped keying:+ # workflow ids are reused across retry/reset/resubmission, run ids are not).+ workflow_handle = await temporal_client.start_workflow( # pyright: ignore[reportUnknownMemberType]+ workflow=WfPipeRouter.run,+ arg=prepared_job,+ id=workflow_id,+ task_queue=task_queue,+ # With the bug this fails the workflow: the inline hydration failure is+ # converted to a terminal TemporalError by the workflow's fail-safe floor.+ # The execution timeout is a safety net in case a regression turns the+ # inline failure into a workflow-task retry loop.+ execution_timeout=timedelta(seconds=60),+ )+ run_id = workflow_handle.first_execution_run_id+ assert run_id is not None+ leaked_library_id = f"wf_{run_id}"++ # Install the post-eviction leaked state: replay the interrupted predecessor's+ # setup steps (mirroring WfPipeRouter.run) and deliberately skip teardown, so the+ # library AND the crate fingerprint stay behind in the worker-local LibraryManager.+ library_manager = get_library_manager()+ fixture_library_id = get_current_library()+ global_registry = KajsonManager.get_class_registry()+ predecessor_registry = ClassRegistry()+ predecessor_registry.register_classes_dict(global_registry.get_classes_dict())+ _opened_library_id, leaked_library = library_manager.open_library(library_id=leaked_library_id)+ leaked_library.set_class_registry(predecessor_registry)+ set_current_library(library_id=leaked_library_id)+ try:+ library_manager.load_from_crate(library_id=leaked_library_id, crate=library_crate)+ finally:+ set_current_library(library_id=fixture_library_id)++ try:+ async with get_task_manager().make_worker(+ temporal_client,+ task_queue=task_queue,+ is_not_sandboxed=True,+ ):+ pipe_output: PipeOutput = await workflow_handle.result()+ finally:+ # The workflow's own finally tears the library down on most paths; suppress+ # covers early failures so the leak never escapes into other tests.+ with suppress(LibraryError):+ library_manager.teardown(library_id=leaked_library_id)++ assert isinstance(pipe_output, PipeOutput)+ rehydrate_pipe_output(pipe_output)+ working_memory = pipe_output.working_memory+ assert working_memory is not None++ # The hydrated dynamic-concept input must have survived the full round-trip.+ assert working_memory.is_stuff_exists("greeting_result"), "greeting_result should be in output WM"+ greeting_out = working_memory.get_stuff("greeting_result")+ assert isinstance(greeting_out.content, StructuredContent), (+ f"Expected StructuredContent after hydration, got {type(greeting_out.content).__name__}"+ )+ assert working_memory.is_stuff_exists("summary_result"), "summary_result should be in output WM"diff --git a/tests/integration/pipelex/temporal/test_wf_pipe_router_run_scoped_state_keying.py b/tests/integration/pipelex/temporal/test_wf_pipe_router_run_scoped_state_keying.pynew file mode 100644index 000000000..eaf16f363--- /dev/null+++ b/tests/integration/pipelex/temporal/test_wf_pipe_router_run_scoped_state_keying.py@@ -0,0 +1,177 @@+"""Regression guard for the cross-run worker-local state collision in ``WfPipeRouter``.++The hazard (priority follow-up of the H2 fix, see+``wip/distributed-execution/nondeterminism-fix-review-follow-ups.md`` item 0): the+per-workflow library, the tracer key, and the report-delegate event-log context are+worker-local singleton entries. If they are keyed by ``workflow.info().workflow_id``,+two *runs* of the same workflow id collide: workflow ids are reused across workflow-level+``retry_policy`` attempts, Temporal reset, and resubmission of the same+``pipeline_run_id`` (``make_workflow_id`` is deterministic). The H2 fix made eviction+reliably execute the ``finally``'s synchronous cleanup — which is destructive cross-run:+a closed predecessor run's late eviction would tear down a live successor run's library,+pop its tracer, and clear its event-log context, failing the successor inline.++The structural fix: key all per-run worker-local state by ``workflow.info().run_id`` —+replay-stable, but unique per run — so cross-run collisions are impossible by+construction. This test pins that keying.++How it works: ``act_llm_gen_text`` is substituted with a stub that blocks on an+``asyncio.Event``, holding the workflow mid-execution with its worker-local state live.+The test then probes the worker-local singletons directly (same process): the library,+tracer, and report-delegate context must exist under run-id-derived keys and must NOT+exist under workflow-id-derived keys. RED while the state is keyed by workflow_id (TDD).+"""++import asyncio+import uuid+from collections.abc import Generator+from datetime import timedelta+from typing import ClassVar++import pytest+from temporalio import activity+from temporalio.client import Client as TemporalClient++from pipelex.cogt.content_generation.assignment_models import LLMAssignment+from pipelex.core.pipes.pipe_output import PipeOutput+from pipelex.graph.graph_tracer_manager import GraphTracerManager+from pipelex.hub import get_library_manager, get_report_delegate+from pipelex.libraries.exceptions import LibraryError+from pipelex.pipe_run.pipe_job import PipeJob+from pipelex.pipe_run.pipe_run_mode import PipeRunMode+from pipelex.reporting.reporting_manager import ReportingManager+from pipelex.temporal.temporal_hub import get_task_manager+from pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_text+from pipelex.temporal.tprl_pipe.act_flush_trace_events import act_flush_trace_events+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle+from tests.integration.pipelex.temporal.tracing.helpers import act_flush_noop, inject_trace_context, route_activities_to+from tests.integration.pipelex.temporal.tracing.test_data import SequenceTracingTestData++_LEAF_PIPE_CODE = "step_one"+_FAKE_RESPONSE_TEXT = "run-scoped keying guard fake response"+++class _BlockSignals:+ """Events shared between the test coroutine and the blocking activity stub.++ Created fresh by the test on the running loop; class-level so the module-level+ activity stub can reach them.+ """++ entered: ClassVar[asyncio.Event | None] = None+ release: ClassVar[asyncio.Event | None] = None++[email protected](name="act_llm_gen_text")+async def _act_llm_gen_text_blocking(llm_assignment: LLMAssignment) -> str: # noqa: ARG001+ """Substitute for ``act_llm_gen_text`` that parks the workflow mid-execution.++ While this activity is blocked, the workflow's worker-local state (library,+ tracer, report-delegate context) is live and probe-able from the test.+ """+ assert _BlockSignals.entered is not None+ assert _BlockSignals.release is not None+ _BlockSignals.entered.set()+ await _BlockSignals.release.wait()+ return _FAKE_RESPONSE_TEXT++[email protected](scope="class")+def live_leaf_keying_job() -> Generator[PipeJob, None, None]:+ """LIVE-mode PipeJob for a leaf PipeLLM, carrying a library crate.++ LIVE mode is required so the workflow dispatches ``act_llm_gen_text`` as an+ activity the test can block on (DRY mode short-circuits it inline).+ """+ yield from pipe_job_from_bundle(+ bundle_file=SequenceTracingTestData.BUNDLE_FILE,+ pipe_code=_LEAF_PIPE_CODE,+ pipe_run_mode=PipeRunMode.LIVE,+ )++[email protected](loop_scope="class")+class TestWfPipeRouterRunScopedStateKeying:+ @pytest.fixture(autouse=True)+ def clear_graph_tracer_manager(self) -> Generator[None, None, None]:+ """Reset the process-lifetime tracer singleton so runs cannot leak tracer keys."""+ yield+ GraphTracerManager.clear_instance()++ async def test_worker_local_state_is_keyed_by_run_id_not_workflow_id(+ self,+ temporal_client: TemporalClient,+ live_leaf_keying_job: PipeJob,+ ) -> None:+ """While the workflow is in flight, its worker-local state must live under+ run-id-derived keys, and nothing may be registered under workflow-id keys.+ """+ execution_job = inject_trace_context(+ live_leaf_keying_job,+ f"run_scoped_keying_guard_{uuid.uuid4().hex[:12]}",+ emit_graph_events=False,+ emit_usage_events=True,+ )+ assert execution_job.library_crate is not None, "the job must carry a library crate so the workflow opens a per-run library"++ task_queue = f"q_run_scoped_keying_{uuid.uuid4().hex[:8]}"+ workflow_id = f"wf_run_scoped_keying_{uuid.uuid4().hex[:8]}"+ _BlockSignals.entered = asyncio.Event()+ _BlockSignals.release = asyncio.Event()++ library_manager = get_library_manager()+ tracer_manager = GraphTracerManager.get_or_create_instance()+ report_delegate = get_report_delegate()+ assert isinstance(report_delegate, ReportingManager)++ try:+ with route_activities_to(task_queue, [act_llm_gen_text.__name__]):+ async with get_task_manager().make_worker(+ temporal_client,+ task_queue=task_queue,+ is_not_sandboxed=True,+ substitute_activities={+ act_llm_gen_text: _act_llm_gen_text_blocking,+ act_flush_trace_events: act_flush_noop,+ },+ ):+ workflow_handle = await temporal_client.start_workflow( # pyright: ignore[reportUnknownMemberType]+ workflow=WfPipeRouter.run,+ arg=execution_job,+ id=workflow_id,+ task_queue=task_queue,+ execution_timeout=timedelta(seconds=60),+ )+ run_id = workflow_handle.first_execution_run_id+ assert run_id is not None++ await asyncio.wait_for(_BlockSignals.entered.wait(), timeout=30)++ # Library: present under wf_{run_id}, absent under wf_{workflow_id}.+ library_manager.get_library(library_id=f"wf_{run_id}")+ with pytest.raises(LibraryError):+ library_manager.get_library(library_id=f"wf_{workflow_id}")++ # Tracer: registered under run_id, not workflow_id.+ assert tracer_manager.get_tracer(run_id) is not None, "tracer must be keyed by run_id"+ assert tracer_manager.get_tracer(workflow_id) is None, "tracer must not be keyed by workflow_id"++ # Report-delegate event-log context: keyed by run_id, not workflow_id.+ contexts = report_delegate._event_log_contexts # noqa: SLF001 # pyright: ignore[reportPrivateUsage]+ assert run_id in contexts, "event-log context must be keyed by run_id"+ assert workflow_id not in contexts, "event-log context must not be keyed by workflow_id"++ _BlockSignals.release.set()+ pipe_output = await workflow_handle.result()+ assert isinstance(pipe_output, PipeOutput)++ # After completion, the run-scoped state must be fully cleaned up.+ with pytest.raises(LibraryError):+ library_manager.get_library(library_id=f"wf_{run_id}")+ assert tracer_manager.get_tracer(run_id) is None+ assert run_id not in contexts+ finally:+ _BlockSignals.entered = None+ _BlockSignals.release = Nonediff --git a/tests/integration/pipelex/temporal/test_wf_pipe_router_tracing_config_nondeterminism.py b/tests/integration/pipelex/temporal/test_wf_pipe_router_tracing_config_nondeterminism.pynew file mode 100644index 000000000..f2258da7f--- /dev/null+++ b/tests/integration/pipelex/temporal/test_wf_pipe_router_tracing_config_nondeterminism.py@@ -0,0 +1,159 @@+"""Regression guard for the [TMPRL1100] tracing-config nondeterminism in ``WfPipeRouter``.++The bug: ``wf_pipe_router.py`` reads ``get_config().pipelex.tracing_config`` inside the+workflow body to decide whether to set up tracing — and therefore whether the ``finally``+block schedules ``act_flush_trace_events``. That config is worker-local mutable state, so+activity scheduling is not a pure function of the workflow history. In production this+bites when a workflow recorded by a tracing-enabled worker is replayed by a worker whose+tracing config differs (worker restart, rolling deploy, scale-in): the replay generates a+different command stream and Temporal raises a nondeterminism error.++The ``config-flips`` arm reproduces that scenario in a single process against the+in-process Temporal server — no distributed setup, no frozen history fixture — and+asserts the workflow nevertheless completes. It is RED while the bug is present (TDD):+the workflow fails with ``[TMPRL1100] Nondeterminism error`` instead of completing.++How the reproduction works:++1. Tracing is enabled and a ``trace_context`` is injected on a DRY-mode leaf PipeLLM job,+ so the workflow's first task buffers trace events and ends by scheduling+ ``act_flush_trace_events`` (the only activity a dry leaf workflow schedules).+2. The worker is built with ``max_cached_workflows=0``: the workflow is evicted from the+ cache after every workflow task, so each subsequent task replays the full history from+ scratch — the in-process equivalent of the workflow landing on a *different* worker.+3. ``act_flush_trace_events`` is substituted with a stub that flips+ ``tracing_config.is_enabled`` to ``False`` — the in-process equivalent of "the worker+ that picks up the next workflow task has a different tracing config". Activities run+ in the worker process, so the flip lands between the workflow task that scheduled the+ flush and the one that follows its completion: a deterministic, race-free injection+ point.+4. The post-flush workflow task replays from the start under tracing-off config. With the+ bug, the workflow skips tracing setup, schedules no flush, and tries to complete — but+ the history records an ``act_flush_trace_events`` schedule at that position. Command+ streams diverge; the SDK raises ``workflow.NondeterminismError``.++The ``config-stable`` control arm runs the identical setup with a no-op flush stub that+leaves the config alone: it passes even with the bug present, proving the config flip —+not the forced replays — is what breaks the command stream. Both arms assert from the+fetched history that ``act_flush_trace_events`` was actually scheduled, so neither can+pass vacuously.++``NondeterminismError`` is added to ``workflow_failure_exception_types`` so a divergence+fails the workflow terminally and surfaces in the test. Without it, Temporal's default+turns nondeterminism into a workflow-task failure that retries forever — which is exactly+the production symptom (a silently hung workflow), but useless in a test.+"""++import uuid+from collections.abc import Generator+from datetime import timedelta++import pytest+from temporalio import activity, workflow+from temporalio.client import Client as TemporalClient+from temporalio.worker import UnsandboxedWorkflowRunner, Worker++from pipelex.config import get_config+from pipelex.core.pipes.pipe_output import PipeOutput+from pipelex.graph.graph_tracer_manager import GraphTracerManager+from pipelex.pipe_run.pipe_job import PipeJob+from pipelex.temporal.temporal_hub import get_task_manager+from pipelex.temporal.tprl_pipe.act_flush_trace_events import FlushTraceEventsArg, act_flush_trace_events+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle+from tests.integration.pipelex.temporal.tracing.helpers import act_flush_noop, inject_trace_context, scheduled_activity_names+from tests.integration.pipelex.temporal.tracing.test_data import SequenceTracingTestData++_LEAF_PIPE_CODE = "step_one"+_FLUSH_ACTIVITY_NAME = "act_flush_trace_events"++[email protected](name=_FLUSH_ACTIVITY_NAME)+async def _act_flush_flips_worker_tracing_config(arg: FlushTraceEventsArg) -> None: # noqa: RUF029, ARG001+ """Substitute flush activity simulating a worker whose tracing config differs.++ Flipping the live config here (the activity runs in the worker process) lands the+ change exactly between the workflow task that scheduled this activity and the replay+ triggered by its completion — deterministically reproducing what a rolling deploy or+ config drift does across real workers.+ """+ get_config().pipelex.tracing_config.is_enabled = False+++def enable_tracing_restored() -> Generator[None, None, None]:+ """Enable tracing for the test; restore it even though the stub activity flips it.++ Function-scoped on purpose: the ``config-flips`` arm leaves the flag False, and the+ other arm must start from True again.+ """+ tracing_config = get_config().pipelex.tracing_config+ original_enabled = tracing_config.is_enabled+ tracing_config.is_enabled = True+ yield+ tracing_config.is_enabled = original_enabled+ GraphTracerManager.clear_instance()++[email protected](scope="class")+def leaf_tracing_job() -> Generator[PipeJob, None, None]:+ """DRY-mode PipeJob for a leaf PipeLLM: no inference activity, so the trace flush is+ the only activity the workflow schedules.+ """+ yield from pipe_job_from_bundle(+ bundle_file=SequenceTracingTestData.BUNDLE_FILE,+ pipe_code=_LEAF_PIPE_CODE,+ )++[email protected](loop_scope="class")+class TestWfPipeRouterTracingConfigNondeterminism:+ @pytest.mark.parametrize("worker_config_flips", [False, True], ids=["config-stable", "config-flips"])+ async def test_replay_survives_worker_tracing_config_flip(+ self,+ temporal_client: TemporalClient,+ enable_tracing_restored: None, # noqa: ARG002 - enables tracing so the workflow schedules the flush+ leaf_tracing_job: PipeJob,+ worker_config_flips: bool,+ ) -> None:+ execution_job = inject_trace_context(leaf_tracing_job, f"tracing_replay_guard_{uuid.uuid4().hex[:12]}")+ flush_substitute = _act_flush_flips_worker_tracing_config if worker_config_flips else act_flush_noop++ # Build the worker directly (not via make_worker) for the two knobs the+ # reproduction needs: cache eviction after every workflow task, and terminal+ # workflow failure on nondeterminism.+ workflows, activities = get_task_manager().workflows_and_activities(+ substitute_activities={act_flush_trace_events: flush_substitute},+ )+ async with Worker(+ temporal_client,+ task_queue=f"q_tracing_replay_guard_{uuid.uuid4().hex[:8]}",+ workflows=workflows,+ activities=activities,+ workflow_runner=UnsandboxedWorkflowRunner(),+ max_cached_workflows=0,+ workflow_failure_exception_types=[workflow.NondeterminismError],+ ) as worker:+ workflow_handle = await temporal_client.start_workflow( # pyright: ignore[reportUnknownMemberType]+ workflow=WfPipeRouter.run,+ arg=execution_job,+ id=f"wf_tracing_replay_guard_{uuid.uuid4().hex[:8]}",+ task_queue=worker.task_queue,+ # Safety net: if nondeterminism ever stops being terminal here, the+ # workflow-task retry loop would hang the test forever without this.+ execution_timeout=timedelta(seconds=60),+ )+ # With the bug, the config-flips arm raises WorkflowFailureError here, caused+ # by "[TMPRL1100] Nondeterminism error". Flush scheduling must be a pure+ # function of the workflow payload/history, so the worker-side config flip+ # must not prevent completion.+ pipe_output = await workflow_handle.result()+ assert isinstance(pipe_output, PipeOutput)++ # Guard against a vacuous pass: the recorded history must contain the flush+ # schedule, otherwise there is nothing for a config-divergent replay to+ # diverge on (and the control arm would prove nothing).+ history = await workflow_handle.fetch_history()+ scheduled_names = scheduled_activity_names(history)+ assert _FLUSH_ACTIVITY_NAME in scheduled_names, f"history must record the flush schedule, got: {scheduled_names}"diff --git a/tests/integration/pipelex/temporal/tracing/helpers.py b/tests/integration/pipelex/temporal/tracing/helpers.pyindex 3bd374799..2774bac1e 100644--- a/tests/integration/pipelex/temporal/tracing/helpers.py+++ b/tests/integration/pipelex/temporal/tracing/helpers.py@@ -9,6 +9,7 @@ from pathlib import Pathfrom temporalio import activityfrom temporalio.client import Client as TemporalClient+from temporalio.client import WorkflowHistoryfrom pipelex.cogt.content_generation.assignment_models import LLMAssignmentfrom pipelex.cogt.llm.llm_job import LLMJob@@ -23,10 +24,10 @@ from pipelex.graph.graphspec import EdgeKind, EdgeSpec, GraphSpec, NodeStatus, Pfrom pipelex.graph.trace_context import TraceContextfrom pipelex.hub import get_report_delegatefrom pipelex.pipe_run.pipe_job import PipeJob-from pipelex.reporting.reporting_manager import ReportingManagerfrom pipelex.temporal.config_temporal import ActivityRouteConfigfrom pipelex.temporal.temporal_hub import get_task_managerfrom pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_text+from pipelex.temporal.tprl_pipe.act_flush_trace_events import FlushTraceEventsArgfrom pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouterfrom pipelex.tracing.graphspec_assembler import GraphSpecAssemblerfrom pipelex.tracing.ndjson_event_log import NdjsonEventLog@@ -187,38 +188,33 @@ def node_pipe_codes(graph_spec: GraphSpec) -> set[str | None]:return {node.pipe_code for node in graph_spec.nodes}-_RUNNER_FAKE_INFERENCE_MODEL_NAME = "split_runner_fake"-_RUNNER_FAKE_INFERENCE_MODEL_ID = "split_runner_fake_id"-_RUNNER_FAKE_RESPONSE_TEXT = "split-worker fake response"+def scheduled_activity_names(history: WorkflowHistory) -> list[str]:+ """Activity type names scheduled in a fetched workflow history, in event order."""+ return [+ event.activity_task_scheduled_event_attributes.activity_type.name+ for event in history.events+ if event.HasField("activity_task_scheduled_event_attributes")+ ][email protected](name="act_llm_gen_text")-async def _runner_isolated_act_llm_gen_text(llm_assignment: LLMAssignment) -> str: # noqa: RUF029- """Substitute for `act_llm_gen_text` that exercises the Phase-2 runner-side fallback.-- Within a single pytest process the router worker registers a context via- `set_event_log` on the process-wide `ReportingManager` singleton; a real- runner activity in the same process would otherwise hit that same dict and- take the fast path, masking the cross-worker bug. Instead this activity:-- - Clears `_event_log_contexts` on entry, simulating a cold runner process.- - Synthesizes an `LLMJob` with non-zero usage and reports it via- `report_inference_job`. The reporting path runs synchronously in the- activity, so `_emit_usage_event` sees the cleared contexts dict and- takes the fallback (writes to the per-process activity event log).- - Returns a fixed string so the calling workflow can finish without- attempting a real LLM call.-- Note: since Part B (leaf-level dry mock), DRY mode also dispatches- `act_llm_gen_text` (the leaf mocks inside the activity with zero-token,- suppressed usage), so this substitute is what guarantees a *reportable*- non-zero usage event lands on the runner regardless of the fixture's run- mode — keeping the cross-worker assertion hermetic and deterministic.[email protected](name="act_flush_trace_events")+async def act_flush_noop(arg: FlushTraceEventsArg) -> None:+ """No-op flush substitute: keeps a test independent of any tracing backend.++ Only the *schedule* of this activity matters to the command stream under test."""- delegate = get_report_delegate()- if isinstance(delegate, ReportingManager):- delegate._event_log_contexts.clear() # noqa: SLF001 # pyright: ignore[reportPrivateUsage]++def make_synthetic_usage_llm_job(+ llm_assignment: LLMAssignment,+ inference_model_name: str,+ inference_model_id: str,+) -> LLMJob:+ """Synthesize an ``LLMJob`` with non-zero token usage from an activity's assignment.++ For activity substitutes that need ``report_inference_job`` to emit a real+ ``UsageReportEvent`` without performing an actual LLM call.+ """now = datetime.now()synthetic_metadata = llm_assignment.job_metadata.model_copy(update={@@ -228,19 +224,51 @@ async def _runner_isolated_act_llm_gen_text(llm_assignment: LLMAssignment) -> st)tokens_usage = LLMTokensUsage(job_metadata=synthetic_metadata,- inference_model_name=_RUNNER_FAKE_INFERENCE_MODEL_NAME,- inference_model_id=_RUNNER_FAKE_INFERENCE_MODEL_ID,+ inference_model_name=inference_model_name,+ inference_model_id=inference_model_id,unit_costs={CostCategory.INPUT: 0.0, CostCategory.OUTPUT: 0.0},nb_tokens_by_category={TokenCategory.INPUT: 1, TokenCategory.OUTPUT: 1},)- synthetic_job = LLMJob(+ return LLMJob(job_metadata=synthetic_metadata,llm_prompt=llm_assignment.llm_prompt,job_params=llm_assignment.llm_setting.make_llm_job_params(),job_config=LLMJobConfig(schema_reask_max_attempts=1),job_report=LLMJobReport(llm_tokens_usage=tokens_usage),)- delegate.report_inference_job(inference_job=synthetic_job)+++_RUNNER_FAKE_INFERENCE_MODEL_NAME = "split_runner_fake"+_RUNNER_FAKE_INFERENCE_MODEL_ID = "split_runner_fake_id"+_RUNNER_FAKE_RESPONSE_TEXT = "split-worker fake response"++[email protected](name="act_llm_gen_text")+async def _synthetic_usage_act_llm_gen_text(llm_assignment: LLMAssignment) -> str: # noqa: RUF029+ """Substitute for `act_llm_gen_text` that emits synthetic usage from the activity.++ Synthesizes an `LLMJob` with non-zero usage and reports it via+ `report_inference_job`, then returns a fixed string so the calling workflow+ can finish without attempting a real LLM call. The emission unconditionally+ takes the per-process activity event log: `_emit_usage_event` checks+ `_is_in_temporal_activity()` before any context lookup, so an in-activity+ emission never touches a workflow's registered buffer — co-located or split+ deployment alike (audit finding H1).++ Since the unified dry run (leaf-level dry mock), DRY mode also dispatches+ `act_llm_gen_text` (the leaf mocks inside the activity with zero-token,+ suppressed usage), so this substitute is what guarantees a *reportable*+ non-zero usage event lands on the runner regardless of the fixture's run+ mode (we route `act_llm_gen_text` to `q_runner` via+ `worker_config.activity_queues`) — keeping the cross-worker assertion+ hermetic and deterministic.+ """+ synthetic_job = make_synthetic_usage_llm_job(+ llm_assignment=llm_assignment,+ inference_model_name=_RUNNER_FAKE_INFERENCE_MODEL_NAME,+ inference_model_id=_RUNNER_FAKE_INFERENCE_MODEL_ID,+ )+ get_report_delegate().report_inference_job(inference_job=synthetic_job)return _RUNNER_FAKE_RESPONSE_TEXT@@ -249,7 +277,7 @@ async def make_split_workers(temporal_client: TemporalClient,q_router: str,q_runner: str,- runner_act_llm_gen_text: Callable[[LLMAssignment], Awaitable[str]] = _runner_isolated_act_llm_gen_text,+ runner_act_llm_gen_text: Callable[[LLMAssignment], Awaitable[str]] = _synthetic_usage_act_llm_gen_text,) -> AsyncGenerator[None, None]:"""Open two scoped workers on two task queues in the current process.@@ -260,11 +288,11 @@ async def make_split_workers(and would never be picked up if the router registered no activities.- `q_runner`: activity-only (runner scope, `disable_all_workflows=True`),with `act_llm_gen_text` substituted by ``runner_act_llm_gen_text``. The- default isolation wrapper clears the in-process `_event_log_contexts`- cache (so the runner cannot accidentally use the router's registered- context) and synthesizes usage instead of calling a real LLM. Pass a- different substitute — e.g. one that clears the cache and then runs real- inference — to exercise the fallback with real provider token counts.+ default substitute synthesizes usage instead of calling a real LLM; the+ in-activity emission always lands in the per-process activity event log+ (never the router's registered context — audit finding H1). Pass a+ different substitute — e.g. one that runs real inference — to exercise+ the same path with real provider token counts.Pair this with `worker_config.activity_queues[act_llm_gen_text.__name__] =ActivityRouteConfig(default=q_runner, by_handle={})` so the workflow ondiff --git a/tests/integration/pipelex/temporal/tracing/test_split_worker_cross_child_usage.py b/tests/integration/pipelex/temporal/tracing/test_split_worker_cross_child_usage.pyindex 26f4c8e03..81588e923 100644--- a/tests/integration/pipelex/temporal/tracing/test_split_worker_cross_child_usage.py+++ b/tests/integration/pipelex/temporal/tracing/test_split_worker_cross_child_usage.py@@ -28,7 +28,6 @@ from pipelex.pipe_run.pipe_run_mode import PipeRunModefrom pipelex.temporal.config_temporal import ActivityRouteConfigfrom pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_textfrom pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from pipelex.tracing.activity_event_log import ActivityEventLogCachefrom pipelex.tracing.ndjson_event_log import NdjsonEventLogfrom pipelex.tracing.trace_events import UsageReportEventfrom pipelex.tracing.usage_aggregator import UsageAggregator@@ -82,12 +81,6 @@ class TestSplitWorkerCrossChildUsage:else:worker_config.activity_queues[activity_name] = original_entry- @pytest.fixture(autouse=True)- def reset_activity_event_log(self) -> Generator[None, None, None]:- ActivityEventLogCache.reset_for_tests()- yield- ActivityEventLogCache.reset_for_tests()-async def _execute_split(self,live_parallel_tracing_job: PipeJob,diff --git a/tests/integration/pipelex/temporal/tracing/test_split_worker_real_inference_cost.py b/tests/integration/pipelex/temporal/tracing/test_split_worker_real_inference_cost.pyindex fdf2d2b15..cfbb43489 100644--- a/tests/integration/pipelex/temporal/tracing/test_split_worker_real_inference_cost.py+++ b/tests/integration/pipelex/temporal/tracing/test_split_worker_real_inference_cost.py@@ -8,10 +8,10 @@ eyeballing. This test closes that blind spot end-to-end.It is marked ``inference``/``llm`` so the default (no-spend) lanes skip it; runit explicitly when validating real-inference cost capture in distributed mode.-The runner activity clears the in-process event-log context cache (cold-runner-simulation) and then runs the genuine ``llm_gen_text`` — so the real LLM call-happens on the runner queue and its usage report takes the fallback path, just-as a physically separate runner process would.+The runner activity runs the genuine ``llm_gen_text`` — the real LLM call happens+on the runner queue and its usage report takes the per-process activity event log+unconditionally (in-activity emissions never touch a workflow's registered+context), just as a physically separate runner process would."""import uuid@@ -28,14 +28,11 @@ from pipelex.cogt.content_generation.llm_generate import llm_gen_textfrom pipelex.cogt.usage.cost_registry import CostRegistryfrom pipelex.cogt.usage.token_category import TokenCategoryfrom pipelex.config import get_config-from pipelex.hub import get_report_delegatefrom pipelex.pipe_run.pipe_job import PipeJobfrom pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.reporting.reporting_manager import ReportingManagerfrom pipelex.temporal.config_temporal import ActivityRouteConfigfrom pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_textfrom pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from pipelex.tracing.activity_event_log import ActivityEventLogCachefrom pipelex.tracing.ndjson_event_log import NdjsonEventLogfrom pipelex.tracing.trace_events import UsageReportEventfrom pipelex.tracing.usage_aggregator import UsageAggregator@@ -50,15 +47,14 @@ if TYPE_CHECKING:@activity.defn(name="act_llm_gen_text")async def _real_runner_act_llm_gen_text(llm_assignment: LLMAssignment) -> str:- """Cold-runner real-inference substitute: clear contexts, then run the genuine LLM call.+ """Real-inference substitute: run the genuine LLM call from the activity.- Clearing ``_event_log_contexts`` makes the synchronous usage report take the runner-side- fallback (the router's registered context lives in the same process during a single-process- test); ``llm_gen_text`` performs the real provider call so the captured token counts are real.+ The synchronous usage report takes the per-process activity event log+ unconditionally — ``_emit_usage_event`` checks ``_is_in_temporal_activity()``+ before any context lookup, so the router's registered context (which lives in+ the same process during a single-process test) is never touched; ``llm_gen_text``+ performs the real provider call so the captured token counts are real."""- delegate = get_report_delegate()- if isinstance(delegate, ReportingManager):- delegate._event_log_contexts.clear() # noqa: SLF001 # pyright: ignore[reportPrivateUsage]return await llm_gen_text(llm_assignment=llm_assignment)@@ -95,12 +91,6 @@ class TestSplitWorkerRealInferenceCost:else:worker_config.activity_queues[activity_name] = original_entry- @pytest.fixture(autouse=True)- def reset_activity_event_log(self) -> Generator[None, None, None]:- ActivityEventLogCache.reset_for_tests()- yield- ActivityEventLogCache.reset_for_tests()-async def test_real_provider_usage_aggregates_to_nonzero_cost(self,live_sequence_tracing_job: PipeJob,diff --git a/tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py b/tests/integration/pipelex/temporal/tracing/test_split_worker_usage.pyindex 1560f27f5..534e91b73 100644--- a/tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py+++ b/tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py@@ -32,7 +32,6 @@ from pipelex.pipe_run.pipe_run_mode import PipeRunModefrom pipelex.temporal.config_temporal import ActivityRouteConfigfrom pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_textfrom pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from pipelex.tracing.activity_event_log import ActivityEventLogCachefrom pipelex.tracing.ndjson_event_log import NdjsonEventLogfrom pipelex.tracing.trace_events import PipeStartEvent, UsageReportEventfrom pipelex.tracing.usage_aggregator import UsageAggregator@@ -55,9 +54,9 @@ class TestSplitWorkerUsageEmission:"""Cross-process emission: router on q_router, runner on q_runner.Each test uses its own pair of task queues (UUID-named) so concurrent- executions don't share queue state, and resets the per-process activity- event log cache so the writer_id is regenerated and the warn-once flag- re-arms for each test.+ executions don't share queue state. The per-process activity event log+ cache is reset around every test by the temporal conftest's autouse+ ``reset_activity_event_log_cache`` fixture."""@pytest.fixture@@ -105,19 +104,6 @@ class TestSplitWorkerUsageEmission:else:worker_config.activity_queues[activity_name] = original_entry- @pytest.fixture(autouse=True)- def reset_activity_event_log(self) -> Generator[None, None, None]:- """Clear the per-process activity event log cache between tests.-- The cache (writer_id, event log instance, warn-once flag) is class-level- on `ActivityEventLogCache`. Without this reset, two tests in the same- module would share a writer_id and the second test would find a stale,- possibly closed, event log handle.- """- ActivityEventLogCache.reset_for_tests()- yield- ActivityEventLogCache.reset_for_tests()-async def _execute_split(self,live_sequence_tracing_job: PipeJob,diff --git a/tests/unit/pipelex/graph/test_graph_tracer_manager_stale_key_healing.py b/tests/unit/pipelex/graph/test_graph_tracer_manager_stale_key_healing.pynew file mode 100644index 000000000..8c97e9e5f--- /dev/null+++ b/tests/unit/pipelex/graph/test_graph_tracer_manager_stale_key_healing.py@@ -0,0 +1,91 @@+"""Guard for ``GraphTracerManager.open_tracer``'s stale-key healing (audit fix M1).++A stale tracer under the requested key can only be the leftover of a prior interrupted+execution (e.g. a Temporal workflow evicted before its finally-block ``close_tracer``+ran). Pre-M1 this raised ``ValueError`` — letting worker-local leak state decide whether+a fresh run's tracing setup succeeds, a replay-determinism breach. ``WfPipeRouter``+dropped its best-effort guard around tracing setup on the strength of the pop-and-replace+contract pinned here.+"""++import logging++import pytest+from pytest_mock import MockerFixture++from pipelex.graph.graph_config import DataInclusionConfig+from pipelex.graph.graph_tracer_manager import GraphTracerManager++_DATA_INCLUSION_OFF = DataInclusionConfig(+ pipe_and_concept_registry=False,+ stuff_json_content=False,+ stuff_text_content=False,+ stuff_html_content=False,+ error_stack_traces=False,+)++_TRACER_KEY = "run_under_test"+++class TestGraphTracerManagerStaleKeyHealing:+ @pytest.fixture(autouse=True)+ def fresh_singleton(self):+ GraphTracerManager.clear_instance()+ yield+ GraphTracerManager.clear_instance()++ def _open_tracer(self, manager: GraphTracerManager) -> None:+ manager.open_tracer(+ graph_id="pipeline_run_x",+ data_inclusion=_DATA_INCLUSION_OFF,+ workflow_id=_TRACER_KEY,+ pipeline_run_id="pipeline_run_x",+ tracer_key=_TRACER_KEY,+ emit_graph_events=False,+ emit_usage_events=True,+ )++ def test_open_tracer_replaces_stale_tracer_under_same_key(self, caplog: pytest.LogCaptureFixture) -> None:+ """Opening over a stale key must self-heal: warn, evict the stale tracer, register a fresh one."""+ manager = GraphTracerManager.get_or_create_instance()+ self._open_tracer(manager)+ stale_tracer = manager.get_tracer(_TRACER_KEY)+ assert stale_tracer is not None++ with caplog.at_level(logging.WARNING):+ # Pre-M1 this raised ValueError; it must now pop-and-replace.+ self._open_tracer(manager)++ fresh_tracer = manager.get_tracer(_TRACER_KEY)+ assert fresh_tracer is not None+ assert fresh_tracer is not stale_tracer, "the stale tracer must be evicted, not reused"+ assert "already exists" in caplog.text++ def test_open_tracer_heals_even_when_stale_teardown_raises(self, mocker: MockerFixture, caplog: pytest.LogCaptureFixture) -> None:+ """A stale tracer's raising teardown must not fail the fresh run's setup.++ The stale teardown runs graph assembly over arbitrary half-built state from an+ interrupted execution; letting its exception propagate would make a fresh run's+ tracing setup depend on leaked predecessor state — the M1 class all over again.+ """+ manager = GraphTracerManager.get_or_create_instance()+ self._open_tracer(manager)+ stale_tracer = manager.get_tracer(_TRACER_KEY)+ assert stale_tracer is not None+ mocker.patch.object(stale_tracer, "teardown", side_effect=RuntimeError("half-built state"))++ with caplog.at_level(logging.WARNING):+ # Must not raise despite the stale teardown blowing up.+ self._open_tracer(manager)++ fresh_tracer = manager.get_tracer(_TRACER_KEY)+ assert fresh_tracer is not None+ assert fresh_tracer is not stale_tracer, "the stale tracer must be evicted even when its teardown raises"++ def test_open_tracer_fresh_key_does_not_warn(self, caplog: pytest.LogCaptureFixture) -> None:+ """The healing warning must be gated on key membership: a fresh key opens silently."""+ manager = GraphTracerManager.get_or_create_instance()+ with caplog.at_level(logging.WARNING):+ self._open_tracer(manager)+ assert "already exists" not in caplog.text+ assert manager.get_tracer(_TRACER_KEY) is not Nonediff --git a/tests/unit/pipelex/libraries/test_library_manager_teardown_forgets_entry.py b/tests/unit/pipelex/libraries/test_library_manager_teardown_forgets_entry.pynew file mode 100644index 000000000..b3abde865--- /dev/null+++ b/tests/unit/pipelex/libraries/test_library_manager_teardown_forgets_entry.py@@ -0,0 +1,84 @@+"""Guards for ``LibraryManager`` teardown bookkeeping when ``library.teardown()`` raises.++Today every sub-teardown is a plain dict reassignment that cannot raise, so the raising+path is unreachable — but "teardown never raises" is an accident, not a contract. If a+future ``Library.teardown`` does real work and raises, the manager must still forget the+entry: a kept entry would turn ``open_fresh_library`` into a deterministic re-raise on+every rerun of that library id on this worker (permanent worker-local poison).+"""++import pytest+from pytest_mock import MockerFixture++from pipelex.libraries.exceptions import LibraryError+from pipelex.libraries.library import Library+from pipelex.libraries.library_manager import LibraryManager++_POISONED_LIBRARY_ID = "poisoned_lib"+++class TestLibraryManagerTeardownForgetsEntry:+ def test_teardown_forgets_entry_when_library_teardown_raises(self, mocker: MockerFixture) -> None:+ """The entry must be forgotten even when ``library.teardown()`` raises."""+ manager = LibraryManager()+ manager.open_library(library_id=_POISONED_LIBRARY_ID)+ mocker.patch.object(Library, "teardown", side_effect=RuntimeError("boom"))++ with pytest.raises(RuntimeError, match="boom"):+ manager.teardown(library_id=_POISONED_LIBRARY_ID)++ # The entry must be gone: a second teardown reports a missing library+ # instead of re-raising the poisoned library's teardown error.+ with pytest.raises(LibraryError):+ manager.teardown(library_id=_POISONED_LIBRARY_ID)++ def test_open_fresh_library_proceeds_when_stale_teardown_raises(self, mocker: MockerFixture) -> None:+ """A raising stale teardown must not fail the fresh open itself — worker-local+ leak state must never decide whether a fresh run's setup succeeds (M1 class).+ """+ manager = LibraryManager()+ _library_id, poisoned_library = manager.open_library(library_id=_POISONED_LIBRARY_ID)+ mocker.patch.object(Library, "teardown", side_effect=RuntimeError("boom"))++ fresh_library = manager.open_fresh_library(library_id=_POISONED_LIBRARY_ID)++ assert fresh_library is not poisoned_library+ assert manager.get_library(library_id=_POISONED_LIBRARY_ID) is fresh_library++ def test_open_fresh_library_recovers_after_raising_teardown(self, mocker: MockerFixture) -> None:+ """A raising teardown must not poison subsequent ``open_fresh_library`` calls."""+ manager = LibraryManager()+ _library_id, poisoned_library = manager.open_library(library_id=_POISONED_LIBRARY_ID)+ mocker.patch.object(Library, "teardown", side_effect=RuntimeError("boom"))++ with pytest.raises(RuntimeError, match="boom"):+ manager.teardown(library_id=_POISONED_LIBRARY_ID)++ mocker.stopall()+ fresh_library = manager.open_fresh_library(library_id=_POISONED_LIBRARY_ID)+ assert fresh_library is not poisoned_library+ assert manager.get_library(library_id=_POISONED_LIBRARY_ID) is fresh_library++ def test_full_teardown_forgets_all_entries_when_a_library_teardown_raises(self, mocker: MockerFixture) -> None:+ """The full-teardown branch (``library_id=None``) shares the forget-even-on-raise contract,+ AND attempts every library's own teardown — a raise from one library must not skip the+ resource-closing of the others.+ """+ manager = LibraryManager()+ manager.open_library(library_id="lib_a")+ manager.open_library(library_id="lib_b")+ teardown_mock = mocker.patch.object(Library, "teardown", side_effect=RuntimeError("boom"))++ with pytest.raises(RuntimeError, match="boom"):+ manager.teardown()++ # Every library's teardown must have been ATTEMPTED, not just the first raising one.+ assert teardown_mock.call_count == 2, "a raising teardown must not skip the remaining libraries' teardowns"++ # And ALL entries must be gone — a raising teardown mid-loop must not strand the+ # remaining libraries (or the bookkeeping maps) in the manager.+ mocker.stopall()+ with pytest.raises(LibraryError):+ manager.teardown(library_id="lib_a")+ with pytest.raises(LibraryError):+ manager.teardown(library_id="lib_b")diff --git a/tests/unit/pipelex/pipeline/test_pipeline_manager.py b/tests/unit/pipelex/pipeline/test_pipeline_manager.pynew file mode 100644index 000000000..8651fdc2a--- /dev/null+++ b/tests/unit/pipelex/pipeline/test_pipeline_manager.py@@ -0,0 +1,52 @@+"""Unit tests for ``PipelineManager`` per-run entry lifecycle.++Pins the two-sided contract around explicit ``pipeline_run_id`` reuse:++- Serial resubmission: once a run's entry is removed (the runner's ``finally`` does this+ after every run), re-registering the same ``pipeline_run_id`` succeeds.+- Concurrent duplicate: while a run's entry is registered, ``add_new_pipeline`` with the+ same ``pipeline_run_id`` raises. This raise is load-bearing — it fires before+ ``GraphTracerManager.open_tracer`` is reached in ``pipeline_run_setup``, shielding a+ LIVE direct-mode tracer (keyed by the caller-suppliable ``pipeline_run_id``) from+ ``open_tracer``'s stale-key pop-and-replace healing. Do not relax it without gating+ that healing to run-unique keys.+"""++import pytest++from pipelex.pipeline.exceptions import PipelineManagerAlreadyExistsError+from pipelex.pipeline.pipeline_manager import PipelineManager+++class TestPipelineManagerRemoval:+ def test_remove_then_resubmit_same_run_id_succeeds(self) -> None:+ """Serial resubmission: after removal, the same ``pipeline_run_id`` registers again."""+ manager = PipelineManager()+ manager.add_new_pipeline(pipe_code="some_pipe", pipeline_run_id="run-1")+ manager.remove_pipeline(pipeline_run_id="run-1")+ pipeline = manager.add_new_pipeline(pipe_code="some_pipe", pipeline_run_id="run-1")+ assert pipeline.pipeline_run_id == "run-1"++ def test_remove_is_tolerant_of_absent_key(self) -> None:+ """Removal of an unregistered id is a no-op — the entry may legitimately be absent+ when setup failed before the registration committed.+ """+ manager = PipelineManager()+ manager.remove_pipeline(pipeline_run_id="never-registered")++ def test_remove_only_targets_the_given_run(self) -> None:+ manager = PipelineManager()+ manager.add_new_pipeline(pipe_code="some_pipe", pipeline_run_id="run-1")+ manager.add_new_pipeline(pipe_code="some_pipe", pipeline_run_id="run-2")+ manager.remove_pipeline(pipeline_run_id="run-1")+ assert manager.get_optional_pipeline(pipeline_run_id="run-1") is None+ assert manager.get_optional_pipeline(pipeline_run_id="run-2") is not None++ def test_concurrent_duplicate_still_raises(self) -> None:+ """While the first run's entry is live, the same ``pipeline_run_id`` must collide+ loudly (tracer shield — see module docstring).+ """+ manager = PipelineManager()+ manager.add_new_pipeline(pipe_code="some_pipe", pipeline_run_id="run-1")+ with pytest.raises(PipelineManagerAlreadyExistsError):+ manager.add_new_pipeline(pipe_code="some_pipe", pipeline_run_id="run-1")diff --git a/tests/unit/pipelex/reporting/test_emit_runner_fallback.py b/tests/unit/pipelex/reporting/test_emit_runner_fallback.pyindex 0d4d5a600..c5e509404 100644--- a/tests/unit/pipelex/reporting/test_emit_runner_fallback.py+++ b/tests/unit/pipelex/reporting/test_emit_runner_fallback.py@@ -323,27 +323,27 @@ class TestEmitRunnerFallback:assert any(record.levelno == logging.WARNING for record in caplog.records)- def test_warning_emitted_once_per_process_when_fallback_engages(+ def test_engaged_log_emitted_once_per_process(self,tmp_path: Path,mocker: MockerFixture,caplog: pytest.LogCaptureFixture,) -> None:- """The 'runner fallback engaged' warning fires only on the first emission per process."""+ """The 'activity-side emission engaged' INFO log fires only on the first emission per process."""_enable_ndjson_tracing(mocker, tmp_path)manager = ReportingManager()manager.setup()trace_context = _make_trace_context(graph_id="run_abc", tracer_key="wf_xyz")- with caplog.at_level(logging.WARNING):+ with caplog.at_level(logging.INFO):for _ in range(100):manager.report_inference_job(_make_llm_job("run_abc", trace_context=trace_context))- engaged_records = [record for record in caplog.records if "runner-side" in record.message.lower()]+ engaged_records = [record for record in caplog.records if "activity-side usage event emission engaged" in record.message.lower()]assert len(engaged_records) == 1- def test_warning_emitted_once_even_with_multiple_managers(+ def test_engaged_log_emitted_once_even_with_multiple_managers(self,tmp_path: Path,mocker: MockerFixture,@@ -354,7 +354,7 @@ class TestEmitRunnerFallback:trace_context = _make_trace_context(graph_id="run_abc", tracer_key="wf_xyz")- with caplog.at_level(logging.WARNING):+ with caplog.at_level(logging.INFO):manager_a = ReportingManager()manager_a.setup()manager_a.report_inference_job(_make_llm_job("run_abc", trace_context=trace_context))@@ -363,7 +363,7 @@ class TestEmitRunnerFallback:manager_b.setup()manager_b.report_inference_job(_make_llm_job("run_abc", trace_context=trace_context))- engaged_records = [record for record in caplog.records if "runner-side" in record.message.lower()]+ engaged_records = [record for record in caplog.records if "activity-side usage event emission engaged" in record.message.lower()]assert len(engaged_records) == 1def test_retried_activity_emits_duplicate_usage_event_documenting_r2(diff --git a/tests/unit/pipelex/reporting/test_reporting_event_emission.py b/tests/unit/pipelex/reporting/test_reporting_event_emission.pyindex 8efa6c992..86b08df40 100644--- a/tests/unit/pipelex/reporting/test_reporting_event_emission.py+++ b/tests/unit/pipelex/reporting/test_reporting_event_emission.py@@ -219,6 +219,29 @@ class TestReportingEventEmission:assert events_b[0].workflow_id == "wf_b"assert events_b[0].pipeline_run_id == "run_b"+ def test_set_event_log_silently_overwrites_existing_key(self) -> None:+ """Re-registering a context key must replace, never raise — the silent overwrite is+ the load-bearing self-heal for contexts leaked by a prior interrupted execution+ (see the contract comment on ``ReportingManager.set_event_log``).+ """+ stale_log = InMemoryEventLog()+ fresh_log = InMemoryEventLog()+ manager = ReportingManager()+ manager.setup()++ manager.set_event_log(context_key="wf_a", event_log=stale_log, workflow_id="wf_a", pipeline_run_id="run_stale")+ # Same key again — must not raise, must replace.+ manager.set_event_log(context_key="wf_a", event_log=fresh_log, workflow_id="wf_a", pipeline_run_id="run_fresh")++ ctx = _make_trace_context(graph_id="run_fresh", tracer_key="wf_a")+ manager.report_inference_job(_make_test_llm_job("run_fresh", trace_context=ctx))++ fresh_events = [evt for evt in fresh_log.read_events("run_fresh") if isinstance(evt, UsageReportEvent)]+ assert len(fresh_events) == 1, "emission must route to the newest registered log"+ assert fresh_events[0].pipeline_run_id == "run_fresh"+ stale_events = list(stale_log.read_events("run_stale")) + list(stale_log.read_events("run_fresh"))+ assert len(stale_events) == 0, "the replaced log must no longer receive emissions"+def test_clear_event_log_removes_only_target_context(self) -> None:"""Clearing one context does not affect another."""event_log_a = InMemoryEventLog()diff --git a/tests/unit/pipelex/reporting/test_temporal_activity_gate_lazy_import.py b/tests/unit/pipelex/reporting/test_temporal_activity_gate_lazy_import.pynew file mode 100644index 000000000..f81dec116--- /dev/null+++ b/tests/unit/pipelex/reporting/test_temporal_activity_gate_lazy_import.py@@ -0,0 +1,54 @@+"""Lazy-import contract of the Temporal-activity gate in ``reporting_manager``.++``_is_in_temporal_activity`` must detect the activity context WITHOUT importing temporalio:+a module-level import would put the entire temporalio extra (Rust bridge, protobuf, ~130ms)+on every pipelex boot's critical path wherever ``pipelex[temporal]`` is installed — including+processes that never touch Temporal (CLI validate runs, direct-mode servers, the hosted+runner image). The gate sniffs ``sys.modules`` instead: a process that never imported+``temporalio.activity`` cannot be inside a Temporal activity, because the activity context is+set by temporalio's own machinery, which requires the module to be imported.+"""++import subprocess # noqa: S404+import sys++import pytest+from pytest_mock import MockerFixture++from pipelex.reporting.reporting_manager import _is_in_temporal_activity # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]+++class TestTemporalActivityGateLazyImport:+ def test_false_when_temporalio_activity_not_imported(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """No ``temporalio.activity`` in ``sys.modules`` → no activity context can exist."""+ monkeypatch.delitem(sys.modules, "temporalio.activity", raising=False)+ assert _is_in_temporal_activity() is False++ def test_delegates_to_in_activity_when_module_present(self, monkeypatch: pytest.MonkeyPatch, mocker: MockerFixture) -> None:+ """With ``temporalio.activity`` imported, the gate delegates to its ``in_activity()``."""+ fake_activity_module = mocker.MagicMock()+ fake_activity_module.in_activity.return_value = True+ monkeypatch.setitem(sys.modules, "temporalio.activity", fake_activity_module)+ assert _is_in_temporal_activity() is True+ fake_activity_module.in_activity.assert_called_once_with()++ fake_activity_module.in_activity.return_value = False+ assert _is_in_temporal_activity() is False++ def test_importing_reporting_manager_does_not_import_temporalio(self) -> None:+ """Boot-cost guard: the module import must not pull temporalio into ``sys.modules``.++ Subprocess check so the assertion sees a pristine interpreter (the test process+ itself may have temporalio loaded via the integration fixtures).+ """+ probe = "import sys; import pipelex.reporting.reporting_manager; raise SystemExit(2 if 'temporalio' in sys.modules else 0)"+ result = subprocess.run( # noqa: S603+ [sys.executable, "-c", probe],+ capture_output=True,+ text=True,+ check=False,+ timeout=120,+ )+ assert result.returncode == 0, (+ f"importing pipelex.reporting.reporting_manager pulled temporalio into sys.modules (exit {result.returncode}); stderr: {result.stderr}"+ )diff --git a/tests/unit/pipelex/runtime_bridge/primitives/test_scoped_library_best_effort_teardown.py b/tests/unit/pipelex/runtime_bridge/primitives/test_scoped_library_best_effort_teardown.pynew file mode 100644index 000000000..4d5c5b6e4--- /dev/null+++ b/tests/unit/pipelex/runtime_bridge/primitives/test_scoped_library_best_effort_teardown.py@@ -0,0 +1,58 @@+"""Best-effort teardown contract of ``scoped_library_for_crate``.++The teardown in the ``finally`` must neither mask the body's in-flight exception nor+fail an otherwise-successful call: ``LibraryManager`` forgets the entry pop-first even+when the library's own teardown raises, so suppressing the teardown error is safe.+"""++import pytest+from pytest_mock import MockerFixture++from pipelex.hub import get_library_manager+from pipelex.libraries.library_crate import LibraryCrate+from pipelex.runtime_bridge.primitives.scoped_library import scoped_library_for_crate+++def _run_scope_with_failing_body(fake_crate: LibraryCrate, captured_library_ids: list[str]) -> None:+ with scoped_library_for_crate(fake_crate, library_id_prefix="test_scope") as library_id:+ assert library_id is not None+ captured_library_ids.append(library_id)+ msg = "body boom"+ raise ValueError(msg)+++class TestScopedLibraryBestEffortTeardown:+ def test_teardown_failure_does_not_fail_successful_body(self, mocker: MockerFixture) -> None:+ fake_crate = mocker.MagicMock(spec=LibraryCrate)+ library_manager = get_library_manager()+ mocker.patch.object(library_manager, "load_from_crate", return_value=None)+ teardown_mock = mocker.patch.object(library_manager, "teardown", side_effect=RuntimeError("teardown boom"))++ scoped_library_id: str | None = None+ with scoped_library_for_crate(fake_crate, library_id_prefix="test_scope") as library_id:+ assert library_id is not None+ scoped_library_id = library_id++ assert teardown_mock.call_count == 1++ # The mocked teardown left the scoped entry registered on the singleton — clean it up.+ mocker.stopall()+ assert scoped_library_id is not None+ library_manager.teardown(library_id=scoped_library_id)++ def test_teardown_failure_does_not_mask_body_exception(self, mocker: MockerFixture) -> None:+ fake_crate = mocker.MagicMock(spec=LibraryCrate)+ library_manager = get_library_manager()+ mocker.patch.object(library_manager, "load_from_crate", return_value=None)+ teardown_mock = mocker.patch.object(library_manager, "teardown", side_effect=RuntimeError("teardown boom"))++ captured_library_ids: list[str] = []+ with pytest.raises(ValueError, match="body boom"):+ _run_scope_with_failing_body(fake_crate, captured_library_ids)++ assert teardown_mock.call_count == 1++ # The mocked teardown left the scoped entry registered on the singleton — clean it up.+ mocker.stopall()+ assert len(captured_library_ids) == 1+ library_manager.teardown(library_id=captured_library_ids[0])diff --git a/wip/distributed-execution/README.md b/wip/distributed-execution/README.mdindex ef28506f0..479ee2041 100644--- a/wip/distributed-execution/README.md+++ b/wip/distributed-execution/README.md@@ -14,7 +14,11 @@ This folder is the **distributed-execution track**: running MTHDS methods as Tem- [temporal-exception-model-revamp.md](temporal-exception-model-revamp.md) — deferred proposal to reparent workflow exceptions onto `ApplicationError`.- [schema-reconstruction-hardening.md](schema-reconstruction-hardening.md) — deferred hardening of cross-process Pydantic-schema reconstruction.- [unit-testing-worker-sandbox-validation.md](unit-testing-worker-sandbox-validation.md) — deferred bug: a sandboxed worker can't boot under `--is-unit-testing` because the registered test workflows fail temporalio sandbox validation.-- [bridge-changes-sibling-repo-reconciliation.md](bridge-changes-sibling-repo-reconciliation.md) — **OPEN handoff:** how the `_bridge` dev-merge changed the runtime-bridge surface (param renames, deleted graph-assembly primitive, `graph_context→trace_context`, #967/#968) and what the downstream `pipelex-mistralai-workflows` + `_workflows` repos need reconciled. Greps/diagnosis not yet done.+- [bridge-changes-sibling-repo-reconciliation.md](bridge-changes-sibling-repo-reconciliation.md) — **OPEN handoff:** how the `_bridge` dev-merge changed the runtime-bridge surface (param renames, deleted graph-assembly primitive, `graph_context→trace_context`, #967/#968) and what the downstream `pipelex-mistralai-workflows` + `_workflows` repos need reconciled. Diagnosis DONE — it is the authoritative breakage inventory; now also carries item 7, the PR #987 reporting-path behavioral break (activity-side usage emissions bypass the plugin's registered buffer).+- [workflow-nondeterminism-audit.md](workflow-nondeterminism-audit.md) — **diagnosis, HIGHs fixed:** verified audit of worker-local inputs leaking into the workflow command stream (post-PR-#984). Both HIGH findings (flush activity gated on an activity-fed buffer; library/fingerprint leak on eviction) and M1 are now FIXED with regression guards; the open remainder is config-derived dispatch options and inline randomness drifting command arguments. Carries the suggested fix order.+- [nondeterminism-fix-review-follow-ups.md](nondeterminism-fix-review-follow-ups.md) — **BURNED DOWN:** verified findings from the code reviews of the H1/M1 and H2 fixes, now all resolved (per-item statuses recorded inline). The priority item — keying per-run worker-local state by `run_id` instead of `workflow_id` — is FIXED with regression guards, along with the eviction-ordering test, the flush-skip in costs-only LIVE, the `finally` simplification, helper consolidation, and the smaller cleanups.+- [nondeterminism-pre-merge-fixes.md](nondeterminism-pre-merge-fixes.md) — **PRE-MERGE, ACTIONABLE:** the two findings from PR #987's xhigh review to fix before merge — `PipelineManager` 500s resubmission of the same `pipeline_run_id` process-permanently (and its collision raise is load-bearing for the direct-mode tracer keyspace — read the coupling warning before touching it), and the module-level temporalio import in `reporting_manager.py` that puts ~130ms on every pipelex boot. Cold-start ready: evidence, fix shapes, red-first tests, acceptance.+- [nondeterminism-follow-ups-decisions-needed.md](nondeterminism-follow-ups-decisions-needed.md) — **DECISIONS NEEDED:** judgment calls deferred from the burn-down and from PR #987's pre-landing review army. Headliners: the graph-event path has no H1-style guard against in-activity tracer mutation (ties into T3), tracing-disabled workers silently drop submitter-requested events, a non-PipelexError in the unguarded tracing-setup block retries forever, and the run-id-keying residual leak/zombie-thread risks. Options + recommendations inside.- [validate-sweep-temporal-leak-repro.md](validate-sweep-temporal-leak-repro.md) — **bug brief + e2e repro (DONE):** the `/validate` dry-run sweep leaked nested controller sub-pipes to Temporal (HTTP 422 on a standalone `PipeBatch`). Fixed (commit `3377babb`); all three guard layers now in place — in-process sentinel test, Mode-1 pytest (`test_validate_sweep_stays_in_process.py`), and the deployment-faithful Mode-2 scenario (Tier 2c in the `temporal-e2e-validate` skill, using the `pipelex validate bundle --temporal` flag).- [temporal-e2e-validate-skill.md](temporal-e2e-validate-skill.md) — validation status of the `/temporal-e2e-validate` skill and the Step 9 (queue-options) skill bugs patched in its `queue-options-battery.md`. Full dry + live run all green; open follow-ups are the vestigial `act_jinja2_gen_text` removal and a stale Mode 1 "Known xfails" note.- [mistral-workflows/](mistral-workflows/) — sub-track for landing the `pipelex-mistralai-workflows` plugin (the MISTRAL_NATIVE backend) onto current `dev`. Its [README](mistral-workflows/README.md) is the hub; verdict is split — land the bridge, hold the `mistralai` 1.x→2.x bump (blocked on instructor PR #2298). Carries the two ready-to-paste keep-alive prompts.diff --git a/wip/distributed-execution/bridge-changes-sibling-repo-reconciliation.md b/wip/distributed-execution/bridge-changes-sibling-repo-reconciliation.mdindex c74e53805..294fd1191 100644--- a/wip/distributed-execution/bridge-changes-sibling-repo-reconciliation.md+++ b/wip/distributed-execution/bridge-changes-sibling-repo-reconciliation.md@@ -9,7 +9,7 @@- **`pipelex-mistralai-workflows`** (`/Users/lchoquel/repos/Pipelex/pipelex-mistralai-workflows`) — the published package (PyPI `pipelex-mistralai-workflows`, import root `pipelex_mistralai_workflows`). The Mistral Workflows plugin that invokes Pipelex pipes from inside Mistral Workflows activities.- **`_workflows`** (`/Users/lchoquel/repos/Pipelex/_workflows`) — a pipelex worktree on branch `feature/Mistral-workflows-merge-4` (the bigger Mistral-Workflows-integration line, PR #954 family).-**The break is deferred, not live.** Both consumers are pinned to / based on a *pre-rename* pipelex, so nothing is broken at this instant. The break materializes the moment either consumer is moved onto post-#966 pipelex. The fix is mechanical: the Mistral-native graph/usage path is a mirror of the Temporal path, and it needs the **exact same** `act_assemble_graph → act_assemble_tracing` rewiring the Temporal path already received in #967 — plus it gains cost reporting it never had.+**The break is deferred, not live.** Both consumers are pinned to / based on a *pre-rename* pipelex, so nothing is broken at this instant. The break materializes the moment either consumer is moved onto post-#966 pipelex. (Item 7 below — added 2026-06-11 — has a different trigger: it's a behavioral break from PR #987's reporting-path change, and it lands via the package's `pipelex>=0.27.0` floor on the next pipelex release containing #987, no rev bump required.) The fix is mechanical: the Mistral-native graph/usage path is a mirror of the Temporal path, and it needs the **exact same** `act_assemble_graph → act_assemble_tracing` rewiring the Temporal path already received in #967 — plus it gains cost reporting it never had.## Dependency reality (why it's deferred)@@ -35,6 +35,10 @@ All paths under `pipelex_mistralai_workflows/`. Split into hard breaks (won't im6. The Mistral-native path **never assembled usage/cost** — `wf_pipe_run.py` only ever set `graph_spec`. Post-#967 cost rides on `PipeOutput.tokens_usages` (+ `usage_assembly_error`). When you rewrite (1)+(2) to use `assemble_tracing`, you get usage assembly **for free** from the same single event read; copy `tokens_usages` + `usage_assembly_error` onto `pipe_output` too. This is the "Mistral-native gains cost reporting" win, not just a break-fix.+### Behavioral break added after this diagnosis (PR #987, 2026-06-11): activity-side usage emissions bypass the plugin's registered buffer++7. **`primitives/wf_pipe_router.py` (usage-buffer contract) + deployment config.** PR #987 (`fix/Config`, the workflow-determinism fixes) added an H1 routing gate to `pipelex/reporting/reporting_manager.py::_emit_usage_event` (~line 210): any emission made **inside a temporalio activity** unconditionally skips the registered-context fast path and goes to the per-process activity event log (`act_{pid}_{uuid8}` writer). Mistral Workflows **is temporalio underneath** — its `@activity` decorator wraps `temporalio.activity.defn` (`mistralai/workflows/core/activity.py:181/:187`) — so every usage report from a plugin leaf activity (`act_pipelex_leaf.py`) has `in_activity() == True`. Consequence for the plugin's pre-H1 co-located contract (`wf_pipe_router.py:175-180` registers a `BufferingEventLog` via `set_event_log`; `:208-212` flushes gated on `if buffered_events:`): the buffer **silently stops receiving usage events**. Not a compile break, and not dead code — graph events are tracer-emitted from the workflow body and still land in the buffer — but the flush stops carrying usage. In default configs this is a **diversion, not a loss**: the fallback stamps `workflow_id = tracer_key or graph_id` (same partition) and writes through the backend built from `tracing_config`, which the multi-writer assembly tolerates. It degrades to **actual loss** when the activity-side process's `tracing_config` is disabled or unbuildable (the fallback bare-returns). The diversion incidentally gives the plugin the same H1 replay-determinism protection pipelex's own `WfPipeRouter` adopted — but the plugin never opted in, and its writer_id/sequence expectations change. Trigger condition: any pipelex containing #987 — the package's `pipelex>=0.27.0` floor means a fresh non-editable install picks this up the moment such a release ships, independent of the #966 rename breaks above. Fix during step 3: mirror post-#987 pipelex `wf_pipe_router.py` — drop the usage half of the buffer contract (usage rides the per-process activity log; make the flush schedule payload-pure) or consciously keep the buffer for graph events only and document the usage diversion; either way, ensure the deployment enables `tracing_config` where the activities run. Cross-ref: `nondeterminism-follow-ups-decisions-needed.md` §6 (the generic mechanism — live here, not latent).+### Confirmed NOT broken (don't touch — already verified to resolve against post-#966 pipelex)- `streaming_event_forwarder.py`, `streaming.py`, `wf_pipe_router.py`, `pipe_run.py`, `pipe_router.py`, `act_pipelex_leaf.py`, `act_pipelex_deliver.py`, `act_pipelex_flush_trace_events.py`, `scoped_library.py`, `_kajson_codec.py`, `dependency.py`, `activities.py` import these, all of which **survive**: `pipelex.graph.graph_config.DataInclusionConfig`, `pipelex.tracing.event_log_protocol.EventLogProtocol`, `pipelex.graph.graph_tracer_manager.GraphTracerManager`, `pipelex.runtime_bridge.bootstrap.ensure_pipelex_booted`, and the bridge primitives `trace_flush.flush_trace_events_to_backend` / `hydration.hydrate_working_memory` / `pipe_classification.is_controller_pipe` / `submitter_hydration.rehydrate_pipe_output_with_crate` / `delivery.execute_delivery`.@@ -54,6 +58,7 @@ Mistral-package edits, concretely:- Rename `act_pipelex_assemble_graph.py` → `act_pipelex_assemble_tracing.py`: `AssembleTracingArg(pipeline_run_id, domain_code, main_pipe_code, assemble_graph, assemble_usage)`; body `return assemble_tracing(...)` returning `TracingAssembly`; keep the Mistral `@activity(...)` decorator.- In `wf_pipe_run.py`: gate on the run's emit flags (the Mistral parent has the `PipeJob`, hence `job_metadata.trace_context`; if `trace_context` is None, default both to True to preserve today's always-assemble-graph behavior — or read the config gate the way DIRECT does, decide during impl), call the new activity, copy `graph_spec` / `graph_assembly_error` / `tokens_usages` / `usage_assembly_error`.- In `wf_pipe_router.py` + `streaming.py`: the `graph_context` → `trace_context` renames in the breakage inventory above.+- In `wf_pipe_router.py`: resolve item 7 (post-#987 usage routing) — mirror pipelex's post-#987 `pipelex/temporal/tprl_pipe/wf_pipe_router.py` (payload-pure flush gate; usage events ride the per-process activity log instead of the registered buffer), or keep the buffer for graph events only and document the usage diversion.- Update `primitives/__init__.py` + `registration.py` exports.## `_workflows` (merge-4) reconciliationdiff --git a/wip/distributed-execution/nondeterminism-fix-review-follow-ups.md b/wip/distributed-execution/nondeterminism-fix-review-follow-ups.mdnew file mode 100644index 000000000..f7b3296d0--- /dev/null+++ b/wip/distributed-execution/nondeterminism-fix-review-follow-ups.md@@ -0,0 +1,116 @@+# Nondeterminism fixes (H1/M1/H2) — review follow-ups++> **BURN-DOWN COMPLETE.** Every item below reached a terminal state — see the **Status** line under each heading. Summary: 0/0bis/1/2/3/4/5/6/8/9/10/11 FIXED (with one sub-point of 3 dismissed); 7 SURFACED to [nondeterminism-follow-ups-decisions-needed.md](nondeterminism-follow-ups-decisions-needed.md).++Source: code reviews of branch `fix/Config`. Round 1 reviewed the committed H1/M1 fixes (flush activity and tracing setup made pure functions of payload + history); its H2-related findings were handled by the H2 fix session. Round 2 (below, at the end) reviewed the H2 fix itself — its top finding is the priority item in this doc.++Companion diagnosis: [workflow-nondeterminism-audit.md](workflow-nondeterminism-audit.md).++## PRIORITY — from the H2-fix review (round 2)++### 0. Key per-run worker-local state by `run_id`, not `workflow_id` — `wf_pipe_router.py`++**This is a correctness hazard, not a cleanup.** The per-workflow library (`wf_{workflow_id}`), the tracer key, and the report-delegate event-log context are all keyed by `workflow.info().workflow_id`. The H2 fix guarantees an evicted predecessor instance's `finally` now runs its synchronous cleanup (verified against temporalio source: eviction resumes the coroutine, the `finally` executes, and the new flush command raises `_WorkflowBeingEvictedError` at the await). That reliable cleanup is destructive cross-run: workflow id X run A is cached on worker W; A is closed server-side (workflow-level `retry_policy` in `workflow_caller.py` reuses the id — as do Temporal reset and resubmission of the same `pipeline_run_id` via the deterministic `make_workflow_id`); successor run B starts on W and `open_fresh_library` opens fresh `wf_X`; the worker later processes A's eviction, and A's `finally` tears down B's LIVE library, pops B's tracer, and clears B's event-log context. B then fails inline (registry fallback / `LibraryError` from its own teardown → workflow-task retry loop). Pre-fix only the tracer pop was reachable at eviction; the reorder widened the blast radius to the library and usage-event context. The root defect (workflow_id-keyed worker-local state) predates the H2 fix.++Fix: key the per-run worker-local state by `workflow.info().run_id` (replay-stable, unique per run, new on retry/reset/continue-as-new) — e.g. `wf_{run_id}` for the library id and run-scoped tracer/event-log context keys. `open_fresh_library` stays as the same-run self-heal (eviction leak of run A heals when run A itself replays), and cross-run collisions become structurally impossible. Also make `open_fresh_library`'s check-then-teardown tolerant of concurrent teardown (the membership check and the teardown are not atomic across workflow threads). The existing eviction-leak test covers the leak-then-rerun direction only; add the late-eviction-destroys-successor direction if feasible.++**Status: FIXED.** All per-run worker-local state in `wf_pipe_router.py` is keyed by `wf_run_id = workflow.info().run_id` (library `wf_{run_id}`, tracer key, event-log context key), and the run-scoped identity is also what gets stamped into events/node ids (the runner fallback already stamped `tracer_key`, so stamps and keys stay one identity). `open_fresh_library` is now pop-based (`LibraryManager._pop_and_teardown_library`): the membership check IS the atomic pop, so concurrent teardown resolves to a clean miss. Tests: new `test_wf_pipe_router_run_scoped_state_keying.py` (parks the workflow mid-activity and asserts library/tracer/context exist under run-id keys and NOT under workflow-id keys — RED against the old keying); the eviction-leak test converted to start-then-install so the leaked state sits under the actual `wf_{run_id}`. The destroys-successor direction is pinned structurally by the keying test (no workflow-id-keyed state exists to destroy).++### 0bis. Pin the finally-ordering invariant with a cancellation test++The ordering half of the H2 fix ("no await before worker-local cleanup in the `finally`") is enforced only by comments; the regression test pins the `open_fresh_library` self-heal, not the ordering. Deterministic guard shape (verified constructible with existing infra): `substitute_activities` with a flush stub blocked on an `asyncio.Event` (pattern already in `test_wf_pipe_router_tracing_config_nondeterminism.py`), start the workflow, `handle.cancel()` while it sits at the flush await (`CancelledError` is the same BaseException-at-suspension-point shape as eviction), then assert the `LibraryManager` holds no per-run library and the report-delegate context is clear. Land this before or with item 5 below, which plans to edit the flush schedule.++**Status: FIXED (test landed), with one premise corrected empirically.** Cancellation is NOT the same shape as eviction: workflow cancellation delivered at an activity await is surfaced by the SDK as an `ActivityError` (an `Exception`), which the best-effort `except` around the flush swallows — the workflow then completes normally, so cancellation cannot abort the `finally` (good news: it cannot leak either). The guard test instead forces a real eviction: park the workflow at the blocked flush, shut the worker down (shutdown evicts every cached run, resuming the coroutine with the eviction `BaseException` at the await), then assert library/tracer/context are all clean. Verified RED against a deliberately reordered `finally` (cleanup moved after the await). Test: `test_wf_pipe_router_eviction_cleanup_ordering.py`. The router's `finally` comment was updated to drop the incorrect cancellation claim.++## Do anytime (independent of H2)++### 1. Fix the stale runner-fallback warning — `pipelex/tracing/activity_event_log.py`++`warn_once_runner_fallback_engaged` (~line 93) still says the fallback engages "because no `_event_log_contexts` entry was registered in this process" and "is expected when activities run on a separate worker pool from the workflow router". After the H1 fix, the runner fallback is the **universal** path for all activity-side usage emissions (`reporting_manager.py` routes every in-activity emission there, co-located or not), so this WARNING now fires once per worker process on every standard co-located deployment with a false explanation, and it no longer signals anything anomalous about split deployments.++Fix: rewrite the message to describe the actual post-H1 contract (activity emissions always use the per-process runner log, by design, for replay determinism), and consider downgrading to DEBUG/INFO since it is now the normal path — or keep a WARNING only for the genuinely anomalous case if one still exists.++**Status: FIXED.** Renamed to `log_once_runner_fallback_engaged`, downgraded to INFO, message now states the post-H1 contract (activities always emit through the per-process log, by design, co-located or split alike). No anomalous case remains distinguishable at that call site, so no WARNING arm. Module docstring of `activity_event_log.py` rewritten to match; the two unit tests updated to assert the INFO once-flag.++### 2. Remove the dead `_event_log_contexts.clear()` dance — `tests/integration/pipelex/temporal/tracing/helpers.py`++`_runner_isolated_act_llm_gen_text` (~line 220) clears `ReportingManager._event_log_contexts` (with `SLF001`/`reportPrivateUsage` suppressions) to force the fallback path, and its docstring explains that mechanism. Post-fix, `_emit_usage_event` checks `_is_in_temporal_activity()` **before** the contexts lookup, so the clear no longer changes which path is taken — it is dead code with a stale docstring, and the private-state mutation can interfere with other contexts registered in the same process.++Fix: delete the clear + the isinstance scaffolding, rewrite the docstring to say the fallback is now taken unconditionally in activities. This also collapses the helper toward the duplicated stub in item 3.++**Status: FIXED.** Clear + isinstance scaffolding deleted, docstring rewritten (helper renamed `_synthetic_usage_act_llm_gen_text`). The same dead clear existed in `test_split_worker_real_inference_cost.py::_real_runner_act_llm_gen_text` — removed there too, with its docstring and the module docstring corrected.++### 3. Deduplicate the new test scaffolding into shared helpers++The two new regression tests (`test_wf_pipe_router_costs_only_flush_nondeterminism.py`, `test_wf_pipe_router_tracing_config_nondeterminism.py`) plus the eviction test carry copy-pasted blocks, all verified line-for-line:++- the synthetic `LLMTokensUsage`/`LLMJob`/`report_inference_job` construction duplicates `_runner_isolated_act_llm_gen_text` in `tracing/helpers.py` (only the constant strings differ, and after item 2 the bodies become identical modulo constants) → extract a `make_synthetic_usage_llm_job(...)` builder;+- an identical `_act_flush_noop` `@activity.defn(name="act_flush_trace_events")` stub in both new files → move one copy into `tracing/helpers.py`;+- an identical scheduled-activity-names proto scan over fetched history in both new files (a near-variant also pre-exists in `tracing/test_split_worker_extract_pages.py`) → a `scheduled_activity_names(history)` helper;+- the eviction test's Greeting-stuff/`prepare_for_temporal` setup duplicates `library_crate/test_wf_deferred_hydration.py` down to the same payload → a shared builder in `library_crate` helpers.++Also: the new files re-implement the enable/restore-tracing fixture from `tracing/conftest.py` because they sit one directory up — hoisting a parametrizable fixture (or moving the tests under `tracing/`) removes that copy too.++**Status: FIXED**, except two sub-points dismissed on re-verification:++- Extracted `make_synthetic_usage_llm_job(...)`, a shared `act_flush_noop` stub, and `scheduled_activity_names(history)` into `tracing/helpers.py`; all router-level guard tests now consume them. Extracted `make_prepared_greeting_job(...)` into `library_crate/helpers.py`, consumed by both the eviction-leak test and `test_wf_deferred_hydration.py`.+- DISMISSED (fixture hoist): only ONE copy of the enable/restore-tracing fixture exists (`test_wf_pipe_router_tracing_config_nondeterminism.py::enable_tracing_restored`) — the costs-only test never enables tracing (its flush is stubbed and post-H1 the schedule doesn't read config). One user = nothing to dedup; moving the tests under `tracing/` would also subject them to that directory's `gha_disabled` CI skip, which would be a regression.+- DISMISSED (extract-pages scan): the "near-variant" in `tracing/test_split_worker_extract_pages.py` scans `(activity_type.name, activity_id)` pairs and `start_to_close_timeout`s, not bare names — different shape, not servable by `scheduled_activity_names` without contortion.++### 4. `open_tracer` stale-eviction re-implements `close_tracer` — `pipelex/graph/graph_tracer_manager.py`++The stale-tracer path (~line 131) inlines `pop` + `teardown()`, which is exactly `close_tracer`'s body. Delegate to keep one eviction path. **Caveat for the refactor:** gate the "evicting stale tracer" warning on `key in self._tracers` *before* delegating — `close_tracer`'s return value is ambiguous (`None` both when no tracer existed and when a costs-only tracer legitimately tears down without a `GraphSpec`).++**Status: FIXED** exactly as specified (membership-gated warning, then `self.close_tracer(key)`).++## Do with / after the H2 session (same code, avoid conflicts)++> The H2 fix has landed (eviction-safe `finally` ordering + `open_fresh_library`), so these are now unblocked — the `finally` block has its final shape.++These touch the exact `finally` block the H2 fix restructured; they were deferred to avoid churn while that fix was pending.++### 5. Skip the guaranteed-empty flush activity in costs-only LIVE runs — `wf_pipe_router.py`++Post-H1, the workflow's `BufferingEventLog` is deterministically empty in costs-only LIVE mode (the tracer gets `event_log=None`; all activity emissions go to the per-process fallback), yet every `WfPipeRouter` — including every child workflow spawned per sub-pipe by the controllers — schedules `act_flush_trace_events` with an empty payload: a wasted activity round-trip (history events, task-queue dispatch, worker slot) per workflow in the fan-out. The H1 rationale only forbids gating on **buffer content** (activity-fed, hence nondeterministic); a payload-pure gate is fine. Gate the schedule on `trace_context.emit_graph_events or run-mode-is-DRY` — both ride in the payload, so the decision replays identically.++**Status: FIXED.** Determinism argument re-derived independently before trusting it: (1) gate inputs (`emit_graph_events`, `pipe_run_params.run_mode`) are pure payload, identical on replay; (2) in costs-only LIVE the buffer's only possible feeders — inline tracer graph events and inline dry-run usage reporting — are both absent, and every LIVE usage emission is activity-side (always routed to the per-process fallback post-H1), so the skipped flush provably loses nothing; (3) Temporal is unshipped, so no in-flight histories exist to diverge against. Implemented as a `schedule_flush` bool computed in the tracing setup block (where `trace_context` is narrowed) and consumed in the `finally`. The costs-only regression test's vacuity guard inverted accordingly: the costs-only arm now asserts the flush is ABSENT from history and still replays cleanly; the graph arm keeps the original presence guard.++### 6. Trim the redundant null conjuncts in the `finally` block — `wf_pipe_router.py`++Verified redundant given the tracing setup block's atomicity: `and wf_tracer_key is not None` (~line 200), `and trace_context is not None` (~line 203), and the nested `if wf_tracer_key is not None` (~line 233) can never differ from their enclosing conditions. **Do not** go further and collapse the sentinels onto `trace_context is not None`: the sentinels are load-bearing — library-crate setup earlier in the `try` can raise before the tracing block runs, leaving them `None` while `trace_context` is set, and a `trace_context`-gated `finally` would then crash on `event_log.drain()` and unbound `wf_workflow_id`, masking the original error.++**Status: FIXED** (folded into the item 5/9 rewrite of the block). The tracer-close gate is now the single sentinel `wf_tracer_key is not None` (the `wf_graph_tracer_manager` variable is gone — the singleton getter is called in the finally, pure in-memory); the spec-assignment gate is `graph_spec is not None and pipe_output is not None` (costs-only returns `None` by `close_tracer`'s contract, noted in the comment); the nested `wf_tracer_key` re-check is gone (`clear_event_log` takes the always-bound `wf_run_id`, now assigned before the `try`). Sentinels (`wf_library_id`, `event_log`, `wf_tracer_key`) kept, as required.++## Smaller items from the H2-fix review (round 2)++### 8. Document `set_event_log`'s overwrite as load-bearing — `pipelex/reporting/reporting_manager.py`++The leaked-predecessor self-heal now exists as three uncoordinated idioms: `open_fresh_library` (explicit, WARNING), `open_tracer` pop-and-replace (explicit, WARNING), and `set_event_log`'s bare dict assignment (incidental, silent, undocumented). The silent overwrite is genuinely load-bearing: a leaked `_event_log_contexts` entry under the deterministic workflow id remains possible post-H2 (deadlock-detector thread abandonment skips `finally` entirely; worker kill between set and clear), and the overwrite is what heals it. A future hardening pass that adds "context already exists → raise" collision detection — exactly what pre-M1 `open_tracer` did — would reintroduce the eviction-poison class on the cost path with no test catching it. Fix: a comment on `set_event_log` stating the overwrite-on-existing-key is intentional and load-bearing for leaked-predecessor healing; ideally also a shared contract note ("worker-local state keyed by a deterministic workflow id must be self-healing on open/set") tying the three idioms together.++**Status: FIXED.** Comment added on the assignment in `set_event_log`, stating the shared contract and naming the two sibling idioms. (With item 0's run-id keying the leak window is same-run only, but the healing contract stands unchanged.)++### 9. Simplify the `finally` to one event-log block — `wf_pipe_router.py`++Verified equivalent and strictly simpler: move the library-teardown block BEFORE the event-log block, then keep `drain → close → clear_event_log → await flush` together under one `if event_log is not None:` with the await last inside it. Library teardown and event-log cleanup are mutually independent (teardown touches only `LibraryManager` dicts; `clear_event_log` is a bare `dict.pop`; nothing between drain and flush suspends), and the eviction-safety invariant is identical — all synchronous cleanup still precedes the only await. Deletes the `buffered_events` sentinel, the second gate, the TYPE_CHECKING-only `TraceEvent` import, and resolves the comment tension where "the flush activity is scheduled unconditionally" sits directly above a conditional. Natural to fold into item 5/6 since they edit the same block.++**Status: FIXED** (one rewrite together with 5 and 6). `buffered_events` sentinel, second gate, and the TYPE_CHECKING `TraceEvent` import all gone; the eviction-ordering guard test (item 0bis) pins the resulting invariant and was verified RED against a reordered block.++### 10. Extract a shared scoped-library helper for the two uuid-keyed copies — `runtime_bridge`++The seed-registry → open → `set_class_registry` → set-current → `load_from_crate` → teardown-in-finally ceremony exists in three hand-rolled copies. Verified: the two `runtime_bridge` copies (`bridge._scoped_library_for_crate`, `submitter_hydration.rehydrate_pipe_output_with_crate`) use per-call uuid4 ids, so they structurally cannot hit the leaked-predecessor collision and correctly do NOT need `open_fresh_library` — this is consolidation-for-maintenance only. Extract one shared async context manager for those two; leave `wf_pipe_router` bespoke — its teardown sequencing relative to the flush await IS the H2 fix, and folding it into a `with`-helper would fight that ordering.++**Status: FIXED.** One shared SYNC context manager `scoped_library_for_crate(library_crate, library_id_prefix)` in `pipelex/runtime_bridge/primitives/scoped_library.py` (the bridge's was only nominally async — `# noqa: RUF029` — so going sync removed the noqa too). Both copies replaced; id prefixes preserved (`runtime_bridge_*`, `rehydrate_*`); `wf_pipe_router` left bespoke as required, with the why documented on the helper.++### 11. One-line hardening: make `LibraryManager.teardown` forget the entry even if `library.teardown()` raises++Today all sub-teardowns are plain dict reassignments that provably cannot raise, so this is unreachable — but "teardown never raises" is an accident, not a contract. If a future teardown does real work (closing resources, a model gaining `validate_assignment`), a raising teardown would leave the entry in `_libraries` and turn `open_fresh_library` into a deterministic re-raise on every rerun of that id on that worker (permanent worker-local poison). Pop the entry first, or `del` in a `finally`.++**Status: FIXED** (folded into item 0's `_pop_and_teardown_library`): the entry is popped before `library.teardown()` runs and the associated maps are cleared in a `finally`. Unit tests in `tests/unit/pipelex/libraries/test_library_manager_teardown_forgets_entry.py` (RED first): a raising teardown still forgets the entry, and `open_fresh_library` recovers afterwards.++## Deferred design note (no action yet)++### 7. Graph events have no equivalent of the H1 usage-path guard++H1's fix enforces "only workflow-thread emissions land in the workflow buffer" for **usage** events only (via `ReportingManager`'s in-activity check). `GraphTracerManager` is imported in `wf_pipe_router.py` under `workflow.unsafe.imports_passed_through()`, so the singleton **is** shared across the sandbox boundary on co-located workers, and the workflow's tracer key rides into activity args via `job_metadata.trace_context`. No current activity-side code calls tracer hooks, so this is latent — but the first activity that does (bridge instrumentation, future activity-side tracing) would mutate the workflow's tracer from outside the workflow thread, corrupting replay-rebuilt graph data (the H1 mechanism on the graph path; post-fix it corrupts data payloads rather than the command stream, so it is data corruption, not TMPRL1100). The structural fix is the same request-scoped tracing-state direction already tracked as T3 in [tracing-cost-reporting.md](tracing-cost-reporting.md); until then, a guard or an explicit "activities must never call tracer hooks" contract note on the tracer manager would prevent the regression.++**Status: SURFACED** → [nondeterminism-follow-ups-decisions-needed.md](nondeterminism-follow-ups-decisions-needed.md) (options + recommendation; ride along with T3).diff --git a/wip/distributed-execution/nondeterminism-follow-ups-decisions-needed.md b/wip/distributed-execution/nondeterminism-follow-ups-decisions-needed.mdnew file mode 100644index 000000000..7bfd23c3b--- /dev/null+++ b/wip/distributed-execution/nondeterminism-follow-ups-decisions-needed.md@@ -0,0 +1,59 @@+# Nondeterminism follow-ups — decisions needed++Items from [nondeterminism-fix-review-follow-ups.md](nondeterminism-fix-review-follow-ups.md) that are real but a judgment call: design tradeoffs with several defensible shapes, surfaced here for discussion instead of being fixed unilaterally. Everything fixable in that doc has been fixed (statuses recorded there).++## 1. Graph events have no equivalent of the H1 usage-path guard (follow-ups item 7)++H1's fix enforces "only workflow-thread emissions land in the workflow buffer" for **usage** events only: `ReportingManager._emit_usage_event` checks `_is_in_temporal_activity()` before any context lookup, so an in-activity emission can never write into the workflow's replay-rebuilt `BufferingEventLog`.++The **graph** path has no such guard. `GraphTracerManager` is imported in `wf_pipe_router.py` under `workflow.unsafe.imports_passed_through()`, so the singleton IS shared across the sandbox boundary on co-located workers, and the workflow's tracer key rides into activity args via `job_metadata.trace_context`. No current activity-side code calls tracer hooks, so the hazard is latent — but the first activity that does (bridge instrumentation, future activity-side tracing) would mutate the workflow's tracer from outside the workflow thread, corrupting replay-rebuilt graph data. Post-H1/H2 this corrupts **data payloads** (the assembled GraphSpec), not the command stream, so it is silent data corruption rather than a [TMPRL1100] failure — harder to notice, not easier.++### Options++- **A. Structural fix (T3 direction):** request-scoped tracing state instead of the process-singleton `GraphTracerManager` — already tracked as Issue T3 in [tracing-cost-reporting.md](tracing-cost-reporting.md). Solves the class, not just this instance. Largest change; touches every tracer lookup site.+- **B. H1-style guard on the tracer hooks:** make the tracer-hook entry points check `_is_in_temporal_activity()` and refuse (or no-op + WARNING) in-activity calls against a workflow-keyed tracer. Small, targeted, mirrors the proven usage-path mechanism — but adds temporal-awareness to `pipelex/graph/`, which is otherwise temporal-free.+- **C. Contract note only:** an explicit "activities must never call tracer hooks" note on `GraphTracerManager`. Zero code risk, but nothing enforces it — exactly the kind of cross-file invariant that rots.++### Recommendation++B as the cheap interim guard if any activity-side tracing work is planned before T3 lands; otherwise C now and fold the real fix into T3. A is the right end state either way — this item should ride along with T3 rather than spawn a separate effort.++> Status: no action taken in the fix/Config burn-down (carried here from the follow-ups doc, which marked it "no action yet"). Decide when T3 is scheduled.++---++The items below were surfaced by the pre-landing review army on PR #987 (gstack `/review`: specialists + adversarial passes). Same rule as above: real, but judgment calls — deferred rather than fixed unilaterally. The review's clear wins (full-teardown forget-even-on-raise hardening with every sibling teardown still attempted, stale-key healing unit tests, best-effort containment of the stale tracer's teardown inside the healing branch, conftest cache-reset hoist, doc/changelog corrections, finally-block key symmetry + discarded-buffer tripwire) were applied directly on the branch and are NOT listed here.++## 2. Tracing-disabled workers silently drop submitter-requested trace/cost events++The worker-local `tracing_config.is_enabled` check deliberately lives activity-side (deterministic-safe), but both drop sites are silent: `flush_trace_events_to_backend` bare-returns on a NON-empty event list when tracing is disabled, and `_emit_usage_event_runner_fallback` bare-returns before building the event log — post-H1 the path for every LIVE usage emission. A submitter that requested tracing (payload carries `trace_context` with emit flags) against a tracing-disabled worker fleet gets a green workflow and an empty event stream: costs undercounted, no operator signal. This is intentional for deliberately-disabled fleets, which is exactly why a warning is a judgment call — it would fire once per flush/process on every such deployment.++**Options:** (a) one-shot-per-process WARNING at each drop site when the payload requested emission; (b) a config-drift metric/health signal instead of logs; (c) keep silent (current), treat disabled-worker fleets as fully intentional. Leans (a) — cheap and aligned with "a run's terminal reporting must not silently vanish" — but decide alongside the broader config-drift observability story.++## 3. Non-`PipelexError` raised in the now-unguarded tracing-setup block retries forever++The catch-all around tracing setup was removed on purpose (M1: worker-local state must not decide setup success; failures must surface). But the workflow's inline fail-safe floor catches `PipelexError` only — a non-PipelexError from the setup block (e.g. a pydantic `ValidationError` from the `model_copy` calls after a refactor) escapes both except clauses, fails the workflow task, and Temporal retries it forever: the silent-hang failure mode this whole series eliminates, reintroduced for one error class. Being inline-deterministic it re-fires identically on every retry. **Option:** widen the inline fail-safe floor (or wrap just the setup block) to convert any inline exception deterministically into a terminal `TemporalError`. This is one instance of the already-open "should deterministic converter/workflow-task errors be fail-fast instead of retry-forever" question (see the dry-validate hardening follow-up) — decide them together.++## 4. Residual worker-local-state risks under run-id keying (follow-ups item 0 residuals)++Two accepted-but-undocumented residuals of the run-id rekeying, plus one perf note, recorded so they're findable when symptoms appear:++- **Permanent leak on finally-skipping paths:** run ids are unique, so state leaked by a path that skips the `finally` entirely (e.g. deadlock-detector thread abandonment) is never reclaimed by the overwrite/self-heal idioms that workflow-id reuse used to trigger. Unbounded growth of `LibraryManager._libraries` / tracers / event-log contexts over a long-lived worker's life, on rare paths. Mitigation candidates: TTL sweep, size-bounded eviction, worker-eviction callbacks.+- **Zombie abandoned thread vs the same run's replay:** if an abandoned (deadlock-detector) thread later unblocks, its `finally` tears down `wf_{run_id}` state while a retried replay of the SAME run (same run id) is live on the same worker — the replay then fails inline with a `LibraryError`, which the fail-safe floor converts to a terminal failure. Low probability (abandonment + later resume + timing window); structural fix would be epoch/ownership tokens on the per-run entries.+- **Blocking backend write per usage emission on the activity event loop** (`_emit_best_effort` → e.g. boto3 `put_item`): pre-existing on the split path, now universal post-H1. Dwarfed by inference latency today; offload to a thread or batch if emission frequency grows.++## 5. `LibraryManager._pipe_source_map` is a shared keyspace across concurrent libraries (pre-existing)++`load_from_crate` writes `_pipe_source_map[pipe_ref] = source` keyed by bare `pipe_ref`, and teardown pops the dying library's refs — semantics unchanged by this branch (verified against the pre-branch code) but newly load-bearing now that per-run libraries are the norm: two concurrent runs loading the same crate share keys, so run A's teardown deletes source attribution run B still needs, and loads are last-writer-wins. Blast radius is error-traceability diagnostics (`get_pipe_source`), not execution. **Fix shape:** scope the map per `library_id` (nested dict or move it onto `Library`) — touches `get_pipe_source` callers, so it didn't belong in this PR.++## 6. `_is_in_temporal_activity()` is a process-global signal, not a pipelex-run boundary (contract note)++Two latent edges of the H1 routing check: (a) a host app running a DIRECT pipelex pipeline inside its **own** temporalio activity (legit runtime-bridge embedding) gets its registered direct-mode context bypassed — emissions divert to the per-process fallback with a restamped `workflow_id`, ignoring a custom event-log backend; (b) an emission from a thread spawned inside an activity without contextvar propagation (the `run_in_executor(None, ...)` pattern that exists in `bedrock_client_boto3.py`) would report `in_activity()=False` and resurrect the H1 cross-thread buffer write. All current `report_inference_job` call sites run in the activity's own context, so both are latent **within this repo** — same class as item 1; fold into the same T3-adjacent decision. **Update 2026-06-11:** edge (a) is NOT latent for `pipelex-mistralai-workflows` — Mistral Workflows wraps temporalio activities, so the plugin's registered `BufferingEventLog` usage path is bypassed by this gate on any pipelex release containing PR #987. Recorded as breakage item 7 in [bridge-changes-sibling-repo-reconciliation.md](bridge-changes-sibling-repo-reconciliation.md); fix it there, not here.++## 7. `PipelineManager` keying/lifecycle and the `/pipeline/start` submitter-state gap++Deferred from the pre-merge resubmission fix ([nondeterminism-pre-merge-fixes.md](nondeterminism-pre-merge-fixes.md), fix 1). What landed: a tolerant `remove_pipeline` called from `execute_pipeline`'s `finally` and from `pipeline_run_setup`'s own failure path, plus a 409 mapping in `pipelex-api`. What was deliberately NOT decided:++- **Should `PipelineManager` adopt run-unique keying / self-heal like the three patched siblings (library, tracer, event-log context)?** Today its collision raise is load-bearing: it fires at the top of `pipeline_run_setup`, before `open_tracer`, and is the only thing preventing two concurrent same-id direct-mode runs from cross-popping each other's LIVE tracer via `open_tracer`'s stale-key pop-and-replace healing (the tracer key in direct mode is the caller-suppliable `pipeline_run_id`, not a run-unique Temporal run id). Any move to self-heal/overwrite must simultaneously gate that healing to run-scoped keys or make direct-mode tracer keys run-unique. Note the registry is otherwise write-only — `get_pipeline`/`get_optional_pipeline` have no production callers — so the long-term question is really "what is this registry for".+- **The fire-and-forget `/pipeline/start` path has no submitter-side terminal hook.** `ApiRunner.start_pipeline` (pipelex-api) calls `pipeline_run_setup` and hands off to Temporal; nothing in the submitter process runs when the workflow later completes, so the manager entry (and any submitter-side state keyed by the run id) stays registered for the process lifetime. Consequence after the pre-merge fix: serial resubmission of a completed run's id works on the `execute` path, but on the `start` path it still collides — now a clean 409 instead of a 500, but still process-permanent for that id. Deciding where the start path's terminal cleanup belongs (post-handoff removal? completion webhook? TTL sweep alongside item 4's leak mitigation?) interacts with the tracer-shield coupling above and with what submitter-side state the start path actually needs once the workflow owns delivery.+diff --git a/wip/distributed-execution/nondeterminism-pre-merge-fixes.md b/wip/distributed-execution/nondeterminism-pre-merge-fixes.mdnew file mode 100644index 000000000..8bfa5ab3e--- /dev/null+++ b/wip/distributed-execution/nondeterminism-pre-merge-fixes.md@@ -0,0 +1,101 @@+# PR #987 pre-merge fixes — PipelineManager resubmission 500 + eager temporalio import++> **STATUS: BOTH FIXES IMPLEMENTED (2026-06-11).** As built:+>+> - **Fix 1:** `remove_pipeline` (tolerant pop) added to `PipelineManagerAbstract` + `PipelineManager`; called from `execute_pipeline`'s `finally` (after `close_tracer` — tracer-shield ordering) AND from `pipeline_run_setup`'s own failure path. The setup-side removal goes beyond the original fix shape: the runner can't clean setup failures (it never learns the id when setup raises), and the window between `add_new_pipeline` and the old `try` (i.e. `acquire_library`) also stranded the entry — the setup `try` now starts before `acquire_library` with a `library_acquired` guard preserving the old teardown ownership split. Concurrent-duplicate raise kept (load-bearing tracer shield), pinned by unit test. Tests: `tests/unit/pipelex/pipeline/test_pipeline_manager.py`, `tests/integration/pipelex/pipeline/test_pipeline_run_id_resubmission.py` (red-first).+> - **Fix 1 caveat:** the fire-and-forget `/pipeline/start` path (pipelex-api `ApiRunner.start_pipeline`) has no submitter-side terminal hook, so a started run's id stays registered for the submitter process lifetime — resubmission there now gets a clean 409 instead of a 500, but is still process-permanent. Recorded with the deeper keying/lifecycle question as item 7 in [nondeterminism-follow-ups-decisions-needed.md](nondeterminism-follow-ups-decisions-needed.md).+> - **409 mapping:** landed in `pipelex-api` on branch `fix/pipeline-run-id-conflict-409` (override entry + route-level tests + changelog + `docs/error-responses.md` status-code section, which also gained the previously-undocumented 501 override). Uncommitted in that repo's working tree, agent-check + full suite green.+> - **Fix 2:** module-level temporalio import replaced by a `sys.modules` sniff in `_is_in_temporal_activity`; `bridge.py`'s lazy-import docstring claim is true again (no edit needed there). Tests: `tests/unit/pipelex/reporting/test_temporal_activity_gate_lazy_import.py` incl. subprocess boot-cost guard (red-first); temporal integration True-branch suites green.+>+> Original brief below, kept for context.++> **Cold-start brief.** Two findings from the pre-merge xhigh review of PR #987 (`fix/Config`, "fix(temporal): workflow determinism") that should be fixed before merge. Everything else the review surfaced is either already captured in [nondeterminism-follow-ups-decisions-needed.md](nondeterminism-follow-ups-decisions-needed.md), recorded as breakage item 7 in [bridge-changes-sibling-repo-reconciliation.md](bridge-changes-sibling-repo-reconciliation.md) (the `pipelex-mistralai-workflows` reporting-path change), or is low-priority cleanup. Both fixes below were adversarially verified against the working tree on 2026-06-11 — line numbers were exact then; re-verify before editing, they drift.+>+> Repo norms apply: red-first TDD for behavior changes, `make agent-check` clean, CHANGELOG `[Unreleased]` entries, no new bare `except Exception`.++## Fix 1 — `PipelineManager` makes resubmission of the same `pipeline_run_id` a process-permanent 500++### The bug++`PipelineManager` is the one per-run registry PR #987 did NOT make self-healing, and unlike the three it patched (library / tracer / event-log context), it has **no per-run removal at all**:++- `pipelex/pipeline/pipeline_manager.py:40-46` — `add_new_pipeline` raises `PipelineManagerAlreadyExistsError` when `pipeline_run_id` is already in `self.root`. This is the exact pre-M1 raise-on-collision shape the PR removed from `GraphTracerManager.open_tracer`.+- `pipelex/pipeline/pipeline_manager.py:20-21` — the only cleanup is whole-process `teardown()` (`self.root.clear()`). Grep confirms no `remove`/`pop`/per-run forget anywhere; the abstract has only setup/teardown/get/add.+- `pipelex/pipeline/runner.py:206` — the runner's `finally` meticulously closes the tracer, clears the event-log context, and tears down the per-run library, but **never touches the PipelineManager entry**. Every run permanently strands its key.++### Trigger (real today, not latent)++`pipelex/pipeline/pipeline_run_setup.py:141` calls `add_new_pipeline(pipe_code=pipe_code, pipeline_run_id=pipeline_run_id)` **before anything else** (before `acquire_library` at ~153 and `open_tracer` at ~205). The hosted runner API threads a **client-supplied** id into it: `pipelex-api/api/routes/pipelex/pipeline.py:83-99` passes `pipeline_run_id` from the request (`extras.pipeline_run_id`, populated externally, e.g. by an API gateway). So: two POSTs to the start endpoint with the same `pipeline_run_id` against the same long-lived server process → the second raises `PipelineManagerAlreadyExistsError` → `pipelex-api/api/exception_handlers.py` has **no** AlreadyExists/409 mapping → **unhandled 500** — and since the key is never removed, every retry of that id 500s until process restart.++This directly contradicts PR #987's own changelog, which names "resubmission of the same `pipeline_run_id`" as a real, supported scenario (it's one of the motivations for the run-id rekeying). Worker-side retry/reset/replay does NOT re-enter `add_new_pipeline` (it's submitter-side), which is why the PR's worker-side fixes don't cover it.++Aggravating detail: the registry is effectively write-only — `get_pipeline`/`get_optional_pipeline` have zero production callers in `pipelex/` or `pipelex-api/`. Today the dict exists to mint ids and raise on collision.++### ⚠️ Coupling — do NOT just make it self-healing++The raise at `pipeline_manager.py:41` is currently the **only** thing shielding direct mode from a second PR-#987 finding: `GraphTracerManager.open_tracer`'s new pop-and-replace healing (`pipelex/graph/graph_tracer_manager.py:139-149`) assumes "an existing key can only be a stale leftover", which is proven only for run-unique Temporal `run_id` keys. In direct mode the tracer key is the caller-suppliable `pipeline_run_id` (`pipeline_run_setup.py` ~205 opens with `tracer_key=None`, so `key = pipeline_run_id`). Two **concurrent** runs with the same id would have run 2 silently pop run 1's LIVE tracer (cross-wired graphs, then run 1's `finally` `close_tracer` pops run 2's tracer) — today that can't happen only because `add_new_pipeline` raises first, at line 141, before `open_tracer` is reached. A bare overwrite-on-collision in `add_new_pipeline` would unshield that path.++### Recommended fix shape++Keep the collision raise for **concurrent** duplicates (it's load-bearing, see above); fix the **process-permanence** and the API surface:++1. **Per-run removal in the runner's `finally`** (`pipelex/pipeline/runner.py:206` block): add a `remove_pipeline(pipeline_run_id)` (tolerant pop) to `PipelineManagerAbstract` + `PipelineManager` and call it alongside the existing tracer/event-log/library cleanup. Serial resubmission of a completed/failed run id then works; only genuinely concurrent same-id runs still collide — which is the correct, loud behavior given the tracer coupling.+2. **Map the collision to a 4xx in `pipelex-api`**: add `PipelineManagerAlreadyExistsError` → 409 in `pipelex-api/api/exception_handlers.py` so a true concurrent duplicate is a client-visible conflict, not a 500. (This is a `pipelex-api` change — per workspace rules it lands in that repo, with its own changelog entry.)+3. Leave the deeper question — whether `PipelineManager` should adopt run-unique keying / self-heal like the three patched siblings, and whether `open_tracer`'s healing should be gated to run-scoped keys — to the decisions doc; don't solve it in this PR. If you DO decide to self-heal here instead, you must simultaneously harden `open_tracer` (heal only run-scoped keys, or make direct-mode keys run-unique at registration).++### Tests (red-first)++- Unit (`tests/unit/pipelex/pipeline/`): resubmitting a `pipeline_run_id` after the previous run's cleanup ran → `add_new_pipeline` succeeds (RED before fix 1). A second `add_new_pipeline` while the first is still registered → still raises (pins the concurrent-duplicate guard and the tracer shield).+- Integration: a full `execute_pipeline` run followed by a second run with the same explicit `pipeline_run_id` succeeds end-to-end; the runner `finally` removes the entry even when the pipe fails (assert on the manager after a failing run).+- Note: removal must be tolerant (pop, not raise) — the entry may legitimately be absent if setup failed before line 141 committed it.++## Fix 2 — module-level `temporalio` import puts ~130ms + the Rust bridge on every pipelex boot++### The bug++`pipelex/reporting/reporting_manager.py:25-28`:++```python+try:+ from temporalio import activity as _temporal_activity+except ImportError:+ _temporal_activity = None # type: ignore[assignment]+```++`reporting_manager.py` is imported by `pipelex/pipelex.py` (module import chain), i.e. on **every** `Pipelex.make()` / CLI invocation. Wherever the `pipelex[temporal]` extra is installed — including processes that never touch Temporal (CLI `validate` runs, direct-mode servers, the `pipelex-api` runner image) — this now pays the full temporalio import at boot: **measured ~131ms** in this venv (`.venv/bin/python -c 'import time;t=time.perf_counter();from temporalio import activity;print(time.perf_counter()-t)'`), plus the Rust-bridge/protobuf memory footprint. It also silently breaks the documented lazy-import contract: `pipelex/runtime_bridge/bridge.py`'s module docstring still says "The Temporal extra is lazy-imported only inside the temporal-mode branches" — now false.++### Recommended fix shape++Replace the eager import with a `sys.modules` sniff — strictly better than a lazy cached import, because a process that never imported temporalio **cannot** be inside a Temporal activity (the activity context is set by temporalio's own machinery, which requires the module to be imported):++```python+import sys++def _is_in_temporal_activity() -> bool:+ """True when the current call stack runs inside a Temporal activity.++ sys.modules sniff instead of importing temporalio: if temporalio.activity was never+ imported in this process, no activity context can exist, and importing it here would+ put the entire temporalio extra (Rust bridge, protobuf) on every boot's critical path.+ """+ activity_module = sys.modules.get("temporalio.activity")+ if activity_module is None:+ return False+ return activity_module.in_activity()+```++Delete the module-level try/except entirely (no top-level `temporalio` name should remain). Worker processes are unaffected: by the time any activity runs, `temporalio.activity` is necessarily in `sys.modules`. Keep the gate's behavior contract identical — the existing unit suite (`tests/unit/pipelex/reporting/test_emit_runner_fallback.py`) pins it.++### Tests (red-first)++- Unit: `_is_in_temporal_activity()` returns False when `temporalio.activity` is absent from `sys.modules` (monkeypatch `sys.modules` with the key removed) and delegates to `in_activity()` when present — RED while the module-level import exists (the monkeypatch test can't even simulate "not imported" today since module import already bound the name).+- Boot-cost guard (the actual regression): a subprocess check asserting `import pipelex.reporting.reporting_manager` does **not** pull `temporalio` into `sys.modules`. Cheap and pins the lazy-import contract the docstring promises. Suggested home: `tests/unit/pipelex/reporting/`.+- Existing temporal integration suite must stay green — the H1 routing tests (`test_wf_pipe_router_costs_only_flush_nondeterminism.py`, split-worker tracing tests) exercise the True branch for real.++## Acceptance++- [x] Both fixes implemented with their red-first tests; the concurrent-duplicate raise and the tracer-shield coupling explicitly pinned by a test.+- [x] `pipelex-api` 409 mapping landed in that repo (branch `fix/pipeline-run-id-conflict-409`, uncommitted working tree, suite green).+- [x] `pipelex/runtime_bridge/bridge.py` module docstring's lazy-import claim is true again.+- [x] CHANGELOG `[Unreleased]` entries for both (pipelex), plus pipelex-api's changelog for the 409 mapping.+- [x] `make agent-check` clean; targeted suites (unit reporting + pipeline, temporal integration) green along the way; full `make agent-test` green at the end.diff --git a/wip/distributed-execution/tracing-cost-reporting.md b/wip/distributed-execution/tracing-cost-reporting.mdindex c79b9de0d..b20147189 100644--- a/wip/distributed-execution/tracing-cost-reporting.md+++ b/wip/distributed-execution/tracing-cost-reporting.md@@ -18,7 +18,7 @@ The original Step 6 plan called for a `TracingActivityInboundInterceptor` to set- `BufferingEventLog` — in-memory buffer used **inside** Temporal workflow code (`pipelex/tracing/buffering_event_log.py`). Synchronous I/O is forbidden in workflow context, so workflows buffer here and flush via an activity.- `InMemoryEventLog` — for tests.- Backend chosen at runtime via `make_event_log(tracing_config)` in `pipelex/tracing/event_log_factory.py` (`TracingBackend.NDJSON | DYNAMODB | TEMPORAL_DYNAMODB`).-- **Workflow → activity flush, not interceptor.** `WfPipeRouter.run()` (`pipelex/temporal/tprl_pipe/wf_pipe_router.py`) wires a `BufferingEventLog` into the per-workflow `GraphTracerManager` and into the `ReportingManager` via `set_event_log(context_key=workflow_id, ...)`. After pipe execution, buffered events are drained and persisted by `act_flush_trace_events` (`pipelex/temporal/tprl_pipe/act_flush_trace_events.py`), which runs the synchronous boto3 / file writes off the workflow thread.+- **Workflow → activity flush, not interceptor.** `WfPipeRouter.run()` (`pipelex/temporal/tprl_pipe/wf_pipe_router.py`) wires a `BufferingEventLog` into the per-workflow `GraphTracerManager` and into the `ReportingManager` via `set_event_log(context_key=run_id, ...)` — all per-run worker-local state is keyed by `workflow.info().run_id` (replay-stable, unique per run; workflow ids are reused across retry/reset/resubmission). After pipe execution, buffered events are drained and persisted by `act_flush_trace_events` (`pipelex/temporal/tprl_pipe/act_flush_trace_events.py`), which runs the synchronous boto3 / file writes off the workflow thread. Replay determinism (audit finding H1): the flush schedule is a pure function of the payload — gated on `trace_context` presence AND (`emit_graph_events` or DRY run mode), so costs-only LIVE runs (whose buffer is deterministically empty) skip the round-trip while every potentially-non-empty case schedules it (the activity no-ops on an empty list) — and only workflow-thread emissions (inline tracer graph events, dry-run inline usage) land in the buffer — `ReportingManager` routes activity-side usage emissions to the per-process fallback even when the activity is co-located with the workflow worker, so the buffer content and the command stream are pure functions of payload + history.- **Per-context keying** — `ReportingManager` holds a dict of `_event_log_contexts` keyed by `graph_context.lookup_key`, so concurrent workflows don't trample each other.- **Direct mode unchanged** — `pipeline_run_setup.py:282` calls the same `set_event_log` path, so the same `EventLogProtocol` machinery runs in-process for non-Temporal execution.- **Tracing assembly across workers (graph + cost)** — `pipelex/pipe_run/tracing_assembly.py` (`assemble_tracing` / `assemble_tracing_on_output`) and `pipelex/temporal/tprl_pipe/act_assemble_tracing.py` read all events for a `pipeline_run_id` from the configured backend **once** and feed them into both `GraphSpecAssembler` (when `--graph`) and `UsageAggregator` (when `--costs`), setting `PipeOutput.graph_spec` and `PipeOutput.tokens_usages` respectively. (Renamed from `graph_assembly.py` / `act_assemble_graph` when T2 landed — see below.)@@ -28,7 +28,7 @@ The original Step 6 plan called for a `TracingActivityInboundInterceptor` to set| Scenario | Captured? | How ||---|---|---|| Direct mode (single process) | ✅ | `pipeline_run_setup` configures event log on `ReportingManager`; events emitted in-process. |-| Single-worker Temporal (workflows + activities on same Worker bundle) | ✅ | Activities share the workflow's process; usage events emit through the singleton `ReportingManager` whose `set_event_log` was configured by `WfPipeRouter`. |+| Single-worker Temporal (workflows + activities on same Worker bundle) | ✅ | Activity-side usage events emit through the per-process fallback — same path as split workers, never the workflow's in-sandbox buffer (replay determinism, audit finding H1). Workflow-thread emissions (dry-run inline usage, graph events) use the buffer registered by `WfPipeRouter`. || Parent + child workflows on different worker processes | ✅ | Each workflow's process buffers its own events and flushes its own slice. All slices land in the same NDJSON dir / DynamoDB partition keyed by `pipeline_run_id`. || Cross-worker **graph** assembly | ✅ | `assemble_tracing_on_output` / `act_assemble_tracing` reads all events for the `pipeline_run_id` and runs `GraphSpecAssembler`. || Cross-worker **cost** assembly + report | ✅ | Same single event read feeds `UsageAggregator` → `PipeOutput.tokens_usages`; the submitter renders one cost report from it. (T2 — fixed.) |@@ -43,7 +43,7 @@ The original Step 6 plan called for a `TracingActivityInboundInterceptor` to setThe fix has two independent parts that ship together:-- **Runner-side fallback in `_emit_usage_event`.** When the activity's `ReportingManager` has no entry in `_event_log_contexts` for the workflow's `lookup_key`, it now falls back to a process-local event log built from `tracing_config`. The event log is cached per process via `pipelex.tracing.activity_event_log.get_or_create_activity_event_log` (guarded by a module-level `threading.Lock` so concurrent first-emitters from multiple activity threads agree on a single writer_id and instance). The fallback uses `graph_context.tracer_key or graph_context.graph_id` as `workflow_id` (no new `JobMetadata` field needed — `tracer_key` is already populated by `WfPipeRouter.run`). Specific exceptions (`OSError`, `MissingDependencyError`, `PipelexConfigError`, `botocore.ClientError`) are caught at WARNING — never silently dropped, never `except Exception`. A one-shot warning per process records that the fallback engaged.+- **Runner-side fallback in `_emit_usage_event`.** When the activity's `ReportingManager` has no entry in `_event_log_contexts` for the workflow's `lookup_key`, it now falls back to a process-local event log built from `tracing_config`. The event log is cached per process via `pipelex.tracing.activity_event_log.get_or_create_activity_event_log` (guarded by a module-level `threading.Lock` so concurrent first-emitters from multiple activity threads agree on a single writer_id and instance). The fallback uses `graph_context.tracer_key or graph_context.graph_id` as `workflow_id` (no new `JobMetadata` field needed — `tracer_key` is already populated by `WfPipeRouter.run`). Specific exceptions (`OSError`, `MissingDependencyError`, `PipelexConfigError`, `botocore.ClientError`) are caught at WARNING — never silently dropped, never `except Exception`. A one-shot INFO log per process records that activity-side emission engaged (it is the normal path for every in-activity emission, co-located or split).- **Writer-id disambiguation in `TraceEvent`.** A new `writer_id: str = "primary"` field makes `(workflow_id, sequence)` collisions impossible across writers. NDJSON file naming becomes `wf_{workflow_id}__w_{writer_id}.ndjson` for non-primary writers; the `(pipeline_run_id, workflow_id, writer_id)` cache key prevents two writers from sharing a stale file handle. Read-side dedup key is `(workflow_id, writer_id, type, sequence)`; sort key is `(workflow_id, sequence, writer_id)` — sequence primary so a runner-side `UsageReportEvent` does not sort before earlier router events. DynamoDB SK becomes `EVENT#{workflow_id}#{writer_id}#{sequence:010d}`.The `_get_registry` orphan-accumulation TODO at `reporting_manager.py` is moot: T2 removed the submitter-side `UsageRegistry` entirely (see below), so there is no longer any per-run registry to accumulate or leak. The four `_report_*_job` methods now only emit a `UsageReportEvent` onto the event log; nothing buffers usage in the manager.@@ -71,7 +71,7 @@ Net effect: usage events emitted by runner/worker processes are now aggregated i**Severity**: Design smell that magnifies T1.-`ReportingManager` is a process-singleton accessed via `hub.get_report_delegate()`. The per-context `_event_log_contexts` dict + `lookup_key` scheme works because the workflow process is the same process that emits events from activities (today). The moment activity execution moves off-process, the singleton + context-dict pattern stops being the right shape — the activity needs to pull its event log from request-scoped data (e.g., `JobMetadata` / `TracingContext`), not from a process-local dict that was never populated on its process.+`ReportingManager` is a process-singleton accessed via `hub.get_report_delegate()`. The per-context `_event_log_contexts` dict + `lookup_key` scheme now serves only workflow-thread and direct-mode emissions: activity-side emissions always take the per-process fallback, whether the activity is co-located or remote (the H1 determinism fix made co-location irrelevant to the emit path). The deeper shape issue stands — request-scoped tracing data (e.g., `JobMetadata` / `TracingContext`) would be a better home than a process-local dict.---diff --git a/wip/distributed-execution/workflow-nondeterminism-audit.md b/wip/distributed-execution/workflow-nondeterminism-audit.mdnew file mode 100644index 000000000..eed830e82--- /dev/null+++ b/wip/distributed-execution/workflow-nondeterminism-audit.md@@ -0,0 +1,124 @@+# Workflow Nondeterminism Audit — worker-local inputs leaking into the command stream++**Status: H1, H2, and M1 fixed; remaining findings are diagnosis only.** Multi-agent audit of all code reachable inline from `@workflow.defn` bodies, run on top of the PR #984 fix (commit `db669ea62`, which gated `act_flush_trace_events` on the payload's `trace_context` instead of worker-local `tracing_config.is_enabled`). Every finding below was adversarially verified against the actual code; refuted candidates are listed at the end so they don't get re-chased.++**The breach definition used throughout:** any input read in code executing inline on the workflow thread (the `@workflow.run` body and everything it calls outside `@activity.defn` functions) that is not derived from the workflow payload or workflow APIs (`workflow.now()`, `workflow.uuid4()`, history), and that influences the command stream — whether/which/how many activities or child workflows get scheduled, their order, arguments, timeouts, retry policies, or task queues. Config reads inside activity bodies or at submission time are fine.++**Key calibration from verification (matters for prioritization):** Temporal's replay checker compares command *sequence, types, and IDs* — **not input payloads**. So worker-local inputs that flip *whether a command is emitted* cause the TMPRL1100 silent-hang class; inputs that only change *command arguments* (activity args, timeouts, retry policies, task queues of not-yet-scheduled commands) cause silent skew, not hangs. Both are breaches; only the first class reproduces the PR #984 symptom.++## Inline/activity boundary (map facts)++Everything from `WfPipeRouter.run` → `pipe.run_pipe` → controller/operator `_live_run_*` paths (Jinja prompt rendering, model-deck resolution, class-registry lookups, PipeFunc user functions) executes inline on the workflow thread. The activity boundary is exactly the `workflow.execute_activity(...)` calls inside `ContentGeneratorInWorkflow.make_*`. Controller pipes (`PipeSequence`/`PipeParallel`/`PipeCondition`/`PipeBatch`) dispatch every sub-pipe as a child `WfPipeRouter` via `hub.get_pipe_router()` → `TemporalPipeRouter`; operator pipes are *not* turned into activities wholesale — each runs inline inside its own child workflow and only its inference/templating/extract/img/search leaves cross to activities. `TemporalPipeRun` and `WorkflowExecutorFactory` are submitter-only (no in-workflow branch). The dry-run path (`run_mode=DRY`, `ContentGeneratorDry`) is reachable inline — `run_mode` comes from the payload and nothing on the Temporal dispatch path gates DRY out.++---++## HIGH — can reproduce the TMPRL1100 silent hang (command-count divergence)++These two finish what PR #984 started: scheduling around the flush activity and library teardown in `wf_pipe_router.py` is still not a pure function of payload + history.++### H1. Flush activity gated on a buffer that activities populate cross-thread — empty on replay — **FIXED**++> **Fixed:** `WfPipeRouter` now schedules `act_flush_trace_events` unconditionally whenever the payload carries a `trace_context` (the activity no-ops on an empty list), and `ReportingManager._emit_usage_event` routes any emission coming from inside a Temporal activity (`activity.in_activity()`) to the per-process runner fallback — co-located activities never write into the workflow's in-sandbox buffer. Guarded by `tests/integration/pipelex/temporal/test_wf_pipe_router_costs_only_flush_nondeterminism.py`.++`pipelex/reporting/reporting_manager.py:176` + `pipelex/temporal/tprl_pipe/wf_pipe_router.py:221`++`WfPipeRouter` registers its `BufferingEventLog` in the process-global `ReportingManager._event_log_contexts` keyed by `wf_{workflow_id}`. The single Worker registers workflows and activities in one process, so co-located **activity** completions (`report_inference_job`) hit the fast path and mutate the workflow's buffer from the activity thread. In costs-only mode (`emit_usage_events=True, emit_graph_events=False` — a first-class payload option, see `pipeline_run_setup.py`) the tracer gets `event_log=None`, so those activity-side emissions are the *only* buffer content. Live execution drains a non-empty buffer and schedules `act_flush_trace_events`; on replay, activity results come from history without executing, the recreated buffer stays empty, and the `if buffered_events:` gate emits no flush command. A `ScheduleActivityTask` present in history but absent on replay → TMPRL1100 **from a routine sticky-cache eviction on the same worker** — no config skew or fleet heterogeneity needed. Symmetrically, routing activities to a different queue/worker, or `is_reporting_enabled=false` on the replaying worker (a `ReportingNoOp` never populates the buffer), flips the same command. Graph+usage mode is safe for the boolean (inline tracer emissions re-fire deterministically on replay) — only costs-only mode flips command emission.++**Fix:** gate the flush dispatch purely on the payload (`trace_context` presence / emit flags) and let the activity no-op on an empty event list; stop routing activity-side usage events through the workflow's in-sandbox buffer (co-located activities should use the same runner-fallback per-process event log they use when remote).++### H2. Eviction skips library teardown → leaked fingerprint poisons same-worker replay — **FIXED**++> **Fixed:** two complementary moves. (1) Self-healing setup: `WfPipeRouter` opens its per-workflow library via the new `LibraryManager.open_fresh_library`, which force-tears-down any pre-existing library under the deterministic `wf_{workflow_id}` — a leftover there can only be an interrupted predecessor, and a clean teardown removes the whole stale-state class (fingerprint, pipes, concepts, registry), which is strictly stronger than scoping the fingerprint dedup (re-loading a crate into a half-stale library would hit duplicate-add errors). (2) Eviction-safe cleanup ordering: the workflow `finally` now runs ALL worker-local cleanup (event-log drain/close, report-delegate context clear, library teardown) BEFORE the awaited flush activity, which moved last — a `BaseException` raised at that await can only skip the flush itself, never the teardown. Guarded by `tests/integration/pipelex/temporal/test_wf_pipe_router_eviction_library_leak.py`.++`pipelex/temporal/tprl_pipe/wf_pipe_router.py:96` (setup) and `:241` (teardown)++Worker-local cleanup (`get_library_manager().teardown(wf_library_id)`) sits in the `finally` block *after* the awaited flush activity. Eviction-time interruptions (`_WorkflowBeingEvictedError` — a `BaseException` subclass raised by `_assert_not_read_only` when the workflow is being deleted; also `CancelledError` paths) bypass the `except Exception` at line 229 and abort the `finally` before teardown runs, leaking `_libraries` and `_loaded_fingerprints` in the worker-local `LibraryManager` singleton, keyed by the *deterministic* `wf_{workflow_id}`. On same-worker sticky replay: `open_library` returns the stale library, `set_class_registry` installs a **fresh** registry seeded only from the global one, and `load_from_crate` is fingerprint-skipped (`library_manager.py:388-393`) — so the crate's dynamic classes are never registered into the new registry. `hydrate_working_memory` then raises `PipeJobError` inline where history recorded successful pipe execution → the workflow takes the error path instead of re-emitting recorded commands → command-stream divergence. Triggers are routine: cache eviction under load + tracing-on payload + same-worker replay. The common variant self-heals on the next task retry (that replay's synchronous `finally` tears the leak down), but the persistent-hang variant exists with dynamic-class activity results on a single-worker queue.++**Fix:** scope the fingerprint dedup to the registry instance (clear `_loaded_fingerprints` when `set_class_registry` swaps the registry), or have `WfPipeRouter` force-teardown any pre-existing library under its deterministic id before opening; additionally move worker-local cleanup before the awaited flush activity or into an eviction-safe guard.++---++## MEDIUM — command-stream breaches (which/how many commands, or known-residual divergence)++### M1. The known residual: best-effort tracing-setup `except Exception` — **FIXED**++> **Fixed:** `GraphTracerManager.open_tracer` is now collision-proof (pop-and-replace of a stale tracer key with a WARNING, instead of raising), and the best-effort `except Exception` around tracing setup in `WfPipeRouter` was removed entirely — the setup block is pure in-memory state, so any failure is a real bug that surfaces deterministically (and re-fires identically on replay).++`pipelex/temporal/tprl_pipe/wf_pipe_router.py:160`++The catch-all around tracing setup nulls `event_log` on any worker-local failure, so the flush command at line 223 is never emitted while history contains it. Reachable with identical config on both workers: `GraphTracerManager.open_tracer` raises `ValueError` on a stale tracer key leaked by a prior interrupted execution (`graph_tracer_manager.py:133-135`, process-lifetime singleton keyed by `workflow_id`). Verification narrowed severity: fresh/restarted workers have empty singletons so routine churn does *not* trigger it — a leak requires a stuck eviction or `disable_safe_eviction` plus a same-`workflow_id` rerun on the poisoned worker. Aggravator: on *success* the block rewrites `workflow_arg.job_metadata` with `wf_trace_context`, which rides inside every child-workflow `PipeJob` and every content-generation activity argument — so setup success/failure also diverges the arguments of all subsequent commands. **Fix:** payload-pure flush gating (H1's fix) plus make `open_tracer` collision-proof (pop-and-replace instead of raise).++### M2. `PipeBatch` fan-out chunking from worker config++`pipelex/pipe_controllers/batch/pipe_batch.py:141` — `get_config().pipelex.pipeline_execution_config.max_concurrency` bounds `gather_bounded` over branches that each `workflow.execute_child_workflow`. The chunk size controls how many `StartChildWorkflowExecution` commands are emitted per workflow task and their grouping; a config edit rolled out while a batch workflow is in flight diverges replay (TMPRL1100) on any batch spanning multiple workflow tasks. **Fix:** carry `max_concurrency` in `pipe_run_params` (payload), stamped from config at the submission boundary.++### M3. Dry-run mock fan-out length from worker config++`pipelex/cogt/content_generation/content_generator_dry.py:94` (also `nb_extract_pages` at lines 233/262, `nb_list_items` again at 305, image params) — `dry_run_config.nb_list_items` sets the *length* of mock lists built inline; a downstream `PipeBatch` fans out one child workflow per item, so the recorded command count equals a worker-local config value. DRY runs reach Temporal: `dry_run_pipeline()` dispatches via `TemporalPipeRun` under a Temporal-enabled hub with no DRY gate on the path (PR #976 only localized the BundleValidator sweep). **Fix:** move dry-run synthesis into the activity (aligned with the dry-run-as-temporal-activity direction) or carry the dry-run params in the payload.++### M4. `PipeCondition` pipe resolution on the no-crate path++`pipelex/pipe_controllers/condition/pipe_condition.py:254` — the expression evaluation is deterministic from `WorkingMemory`, but the resolved pipe object selects *which* child workflow gets scheduled (and is embedded in its `PipeJob` argument). On the crate-backed path resolution is payload-pure (per-workflow library rebuilt from the crate); the **no-crate** path is real and supported (`PipeJob.library_crate` is `Optional`; `worker_cli.py` boot-loads a `worker_base` library), and there library drift between workers changes the child command or replaces it with an inline PipeNotFound error path. **Fix:** require crate-derived resolution for Temporal dispatch, or carry the resolved sub-pipe selection in the payload.++### M5. `PipeFunc` runs arbitrary user code inline; sync branch is broken outright++`pipelex/pipe_operators/func/pipe_func.py:183,189` — `PipeFunc` has no temporal-aware dispatch: the worker-local module-level `func_registry` read and the user function both execute inline on the workflow thread. Registry presence (function registered on one worker but not another — `LibraryCrate` carries blueprints, not functions) flips fail-vs-complete command paths. The coroutine branch runs unconstrained user async code inline, where nondeterministic success-vs-failure flips the command stream and user `asyncio` calls emit nondeterministically-parameterized timer commands. **Bonus functional bug found during verification:** the sync branch's `asyncio.to_thread` raises `NotImplementedError` under Temporal's workflow event loop (no `run_in_executor`) — sync PipeFunc functions cannot work inside workflows at all today; it fails deterministically, but it fails. **Fix:** execute PipeFunc user functions inside a dedicated activity (`act_run_func`) when in Temporal mode.++---++## MEDIUM/LOW — command-argument drift (silent skew, not hangs)++Temporal does not compare input payloads on replay, so these cannot TMPRL1100 — but recorded vs re-emitted command attributes diverge, and any not-yet-completed activity rescheduled after a config edit runs with different routing/args/policies than history recorded.++| Site | Worker-local input | What drifts | Fix |+|---|---|---|---|+| `tprl_content_generation/content_generator_in_workflow.py:113` (same pattern in every `make_*`: 154, 195, 265, 311, 350, 391, 429, 458, 491, 525) | `get_config().temporal.worker_config.resolve_dispatch` + `temporal.queue_options` at command-emission time | task_queue, all four timeouts, full RetryPolicy of every `ScheduleActivityTask` | move-to-payload: resolve `DispatchOptions` at the submission boundary and carry the per-activity dispatch bundle in `PipeJob`. Note `wf_pipe_run.py` already states and follows this exact principle for its child dispatch — the content generator violates it |+| `pipe_operators/llm/pipe_llm.py:196` (same pattern: `pipe_structure.py:125`, img-gen/extract operators) | worker-local model deck singleton (`get_model_deck()`; deck/config files, not in the crate) | resolved handle is both the activity arg (`LLMAssignment.llm_setting`) **and** the `routing_key` for `resolve_dispatch` → task-queue selection via `by_handle` | move-to-payload |+| `pipe_operators/llm/pipe_llm.py:219` | `pipelex.prompting_config` **plus mutation of `self.llm_prompt_spec` on the library-resident shared pipe object** (the `not templating_style` guard makes the first run on a worker win — sticky, order-dependent) | inline Jinja rendering → the LLM activity's prompt argument | move-to-payload for the style; separately, stop mutating library-resident pipe objects per-run |+| `pipe_operators/llm/pipe_llm.py:275` + `pipe_operators/structure/pipe_structure.py:132` | `cogt.llm_config` structure-prompt flag + template text | whether/how the structure-schema section is rendered into the activity-arg prompt | move-into-activity |+| `tprl_content_generation/content_generator_in_workflow.py:256` (and 302) | `cogt.img_gen_config` defaults | baked into `ImgGenAssignment` activity argument | move-into-activity |+| `tprl_content_generation/content_generator_in_workflow.py:385` (and 452) | `cogt.extract_config.default_page_views_dpi` (the default path always has `page_views_dpi=None`, so the read is always live) | `RenderPageViewsAssignment` activity argument | move-into-activity |+| `tprl/observability.py:104` | `temporal.search_attributes` (enabled flag + attribute subset) | `search_attributes` on `StartChildWorkflowExecution` (called from `wf_pipe_run.py:63` and `temporal_pipe_router.py:88`; the helper's docstring claims purity from `pipe_job` — the config read breaks that claim) | move-to-payload |+| `pipe_operators/compose/pipe_compose.py:317` | `pipeline_execution_config.is_mock_inputs` | raise vs mock-and-continue in the dry-run construct-mode except path — diverges the remaining command stream (low: needs a `StructuredContentComposerValueError` first) | move-to-payload |++### Randomness / clock cluster++Root cause: `pipelex/core/stuffs/stuff_factory.py:44` — `make_stuff_code()` is `shortuuid.uuid()[:5]`, and every operator's output stuff is minted inline on the workflow thread. The codes reach command arguments two ways: working memory serialized into child `PipeJob` args (`pipe_batch.py:138` branch codes → `final_stuff_code` on each branch's `PipeRunParams`; `sub_pipe.py:119` synthetic batch stuff; `pipe_batch.py:218` aggregate output), and the IOSpec digest inside trace events passed literally as the `act_flush_trace_events` argument. Adjacent instances:++- `pipelex/core/pipes/pipe_abstract.py:627` — `PipelineFactory.make_pipe_run_id()` is plain `uuid.uuid4()` per pipe run, stamped into the `child_metadata` that rides in every activity and child-workflow argument. Note the *child workflow id* was already fixed with `workflow.uuid4` (`temporal_pipe_router.py:72`) — the run id needs the same treatment.+- `pipelex/core/pipes/pipe_abstract.py:650` — OTel span creation inline: worker-local telemetry singleton decides span-vs-None and the SDK's random span-id generator feeds `child_metadata.otel_context` (low).+- `pipelex/graph/graph_tracer.py:250` — `datetime.now(timezone.utc)` timestamps in every trace event → flush activity argument (low; timestamps never gate emission).++**Fix shape for the whole cluster:** an injectable replay-safe id/clock provider that returns `workflow.uuid4()` / `workflow.now()` when inside a Temporal workflow, plain `shortuuid`/`uuid4`/`datetime.now` outside.++---++## LOW — test workflows (`test_extras/`, flaky-test risk only)++- `wf_test_child_pipe.py:43` — mutates the worker-global hub pipe library inline with hardcoded pipe codes; `add_new_pipe` raises on duplicates, so any replay or second run on the same worker fails inline before recorded commands. One verifier found these two workflows are currently **orphaned** (registered in `PIPELEX_TEMPORAL_TEST_WORKFLOWS` but not exercised by any test), which caps urgency.+- `wf_test_child_pipe.py:48` (and the other test workflows) — `JobMetadata` constructed inline, so `started_at`'s `default_factory=datetime.now` (`pipeline/job_metadata.py:91`) fires on the workflow thread and rides into every activity argument. The root is the naive `default_factory` itself — already tracked as a P0.2 follow-on in [README.md](README.md).+- `wf_test_content_generator_child.py:63,71` — worker-local model-deck alias resolution (`$testing-text` / `$testing-structured`) inline; a worker missing the alias raises before any downstream activity is scheduled.++---++## Refuted candidates (verified non-breaches — don't re-chase)++- **`temporal_error.py:198` config read of non-retryable name lists** — reachable workflow-side via `from_app_error`, but only sets log severity and the `non_retryable` flag on an unconditionally-raised error; no command divergence.+- **Kajson global `ClassRegistry` accumulation (`wf_pipe_router.py:89`)** — the only writes to the global registry happen at deterministic worker boot; per-workflow registries are seeded fresh.+- **`conditional_worker.py` INTERNAL branch (shortuuid + instance state)** — unreachable inside workflow bodies; the hub router that workflow-thread code resolves is always built EXTERNAL.+- **`wf_pipe_router.py:241` teardown raising on missing library** — `open_library` is idempotent and runs at line 96 on every replay before teardown can be reached.+- **`SubPipe` hub pipe lookup (`sub_pipe.py:45`)** — payload-derived on every supported crate-backed dispatch (the genuine gap is the `PipeCondition` no-crate path, M4).+- **`pipe_run_params_factory.py:19` `pipe_stack_limit` fallback** — production params are fully payload-derived; only the orphaned test workflows hit the config fallback.+- **`pipe_compose.py:292` dry generator selection** — the chosen generator is never invoked on any `PipeCompose` path.+- **`submitter_hydration.py:58` `uuid4`** — names a worker-local scoped library only; never reaches a command.+- **`pipe_abstract.py:474` `datetime.now` in `_run_pipe_traced`** — timestamps land only in trace-event activity args, never gate emission (subsumed by the clock-cluster note above).++---++## Suggested fix order++1. ~~**H1 + M1 together**~~ — **DONE**: `act_flush_trace_events` scheduling is a pure function of `trace_context` (always scheduled when present; activity no-ops on empty events), `open_tracer` is collision-proof, and activity-side usage events never route through the in-sandbox buffer. This finishes PR #984.+2. ~~**H2**~~ — **DONE**: `open_fresh_library` force-teardown of leaked predecessor state, plus eviction-safe cleanup ordering in the `finally` (worker-local cleanup before the awaited flush).+3. **Dispatch options to payload** — resolve `DispatchOptions` at the submission boundary and carry them in `PipeJob` (`ContentGeneratorInWorkflow` then reads only the payload), closing M2's sibling and the biggest argument-drift surface in one move.+4. **Replay-safe id/clock provider** for the randomness cluster.+5. **M3/M4/M5** individually as the dry-run-as-activity and PipeFunc-as-activity work lands.