Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Fix/temporal failsafe (#975)
Louis Choquel855f5833ffc1
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 507de1054..f94cd234f 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -36,6 +36,8 @@ This cycle reworks **cost reporting** so it survives distributed execution and s- **`UsageRegistry` success-path leak.** The per-run cost registry was opened during run setup but closed only on the failure path, so every successful run leaked its registry — and in a long-lived process (e.g. the Pipelex API) reusing a `pipeline_run_id` could collide on the orphaned registry. The registry is removed entirely, so the leak is structurally impossible.+- **Temporal silent-hang fail-safe.** A pipelex domain error raised *inline* in workflow code — never dispatched through an activity — was neither an `ActivityError` nor an `ApplicationError`, so Temporal treated it as a workflow-task failure and retried it indefinitely: a silent, resource-burning hang that only surfaced (as a generic timeout) after the workflow execution timeout. `WfPipeRouter` and `WfPipeRun` now end their boundary handling with a `PipelexError` catch-all that converts a genuine inline error to a terminal, classified failure (the parent also still fires the FAILED webhook), and the worker registers `PipelexError` in `workflow_failure_exception_types` as a backstop so any uncaught domain error fails the workflow terminally instead of hanging. Scoped to domain errors only — transient Temporal/infra errors keep their (correct) task-retry behavior.+## [v0.31.0] - 2026-06-04This release hardens Pipelex at its edges. The headliners: a full **error-handling overhaul** that gives every error a stable, RFC 7807-shaped identity and carries it all the way out to webhooks and external surfaces; **lenient validation with pipe signatures**, so you can sketch and dry-run a whole pipeline top-down before a single pipe is implemented; and a first, **experimental cut of CSV tabular support** for reading and writing typed lists straight from `.csv`.diff --git a/docs/under-the-hood/error-model.md b/docs/under-the-hood/error-model.mdindex 3e0728006..7dca3b5fb 100644--- a/docs/under-the-hood/error-model.md+++ b/docs/under-the-hood/error-model.md@@ -256,7 +256,17 @@ When no report dict is found in the chain — a non-Pipelex exception, a workerThe recovered report is carried on `WorkflowExecutionError(error_report=...)`, whose `to_error_report()` override returns it. Since `WorkflowExecutionError` is a `PipelexError`, `PipelineExecutionError` inherits the classification natively.-**Net effect:** a pipe failing on a Temporal worker reaches the CLI and HTTP adapters with the *same* `error_category` / `retryable` / `model` / `provider` / `user_action` as the identical failure run locally.+### Workflow-Level Fail-Safe Floor++The activity bridge converts every error that crosses an *activity* boundary. But a `PipelexError` raised **inline in workflow code** — an operator that runs its leaf inline instead of dispatching it as an activity, or an inline setup error — is neither an `ActivityError` nor an `ApplicationError`. Temporal's default for an unhandled exception in workflow code that is *not* a recognized failure type is to treat it as a **workflow-task failure** and retry the task indefinitely (bounded only by the workflow execution timeout). That is the single worst outcome a durable-execution system can produce: invisible, resource-consuming, and surfaced only after ~the timeout as a generic timeout error rather than the real cause. The fail-safe floor closes that hole on two levels:++- **Workflow-level catch-all (rich path).** `WfPipeRouter.run()` and `WfPipeRun.run()` each end their boundary handling with an `except PipelexError` clause. `WfPipeRouter` converts a genuine inline error to a terminal `TemporalError` via `from_message_exception` (same conversion the activity bridge does — the structured report rides in `ApplicationError.details`). `WfPipeRun` routes it through its deferred-delivery path so the FAILED webhook still fires, then chains a details-carrying `TemporalError` so the classification also survives to the submitter. The catch-all only converts *genuine* inline errors: an escaping `PipelexError` that already carries a Temporal failure in its `__cause__` chain (e.g. a nested child-dispatch that already wrapped a sub-pipe failure as `WorkflowExecutionError`) is already terminal and recoverable, so it propagates untouched rather than being flattened to a generic report.++- **Worker-level floor (belt-and-suspenders).** The worker registers `workflow_failure_exception_types=[WorkflowExecutionError, PipelexError]`. Any domain error that slips past the catch-alls still fails the workflow **terminally** instead of hanging — degrading to a synthesized `UnrecoverableWorkflowFailureError` report (message preserved, classification floored) rather than a silent retry loop. The scope is deliberately `PipelexError`, not `Exception`: genuinely transient Temporal/infra errors and deterministic-replay glitches are not domain errors, and workflow-task retry is the *correct* behavior for them — so they keep Temporal's default.++The rule of thumb: in a durable-execution system the default for "unexpected exception in business logic inside a workflow" is the most dangerous behavior available (retry forever), so "we convert all the errors we know about" is not enough — the floor must hold for the errors, and the code paths, that nobody enumerated.++**Net effect:** a pipe failing on a Temporal worker reaches the CLI and HTTP adapters with the *same* `error_category` / `retryable` / `model` / `provider` / `user_action` as the identical failure run locally — and a failure that escapes inline fails loud and bounded instead of hanging.---diff --git a/pipelex/base_exceptions.py b/pipelex/base_exceptions.pyindex 57dd465eb..212e13036 100644--- a/pipelex/base_exceptions.py+++ b/pipelex/base_exceptions.py@@ -1,3 +1,4 @@+from collections.abc import Iteratorfrom typing import Any, ClassVarfrom pydantic import BaseModel, ConfigDict@@ -7,6 +8,28 @@ from pipelex.tools.misc.string_utils import pascal_case_to_kebab, pascal_case_tofrom pipelex.types import StrEnumfrom pipelex.urls import URLs++def iter_cause_chain(exc: BaseException) -> Iterator[BaseException]:+ """Yield ``exc`` and each error along its ``__cause__`` chain, exactly once.++ The single cycle-guarded ``__cause__`` walk shared across the error-reporting+ paths — classification recovery (``find_inference_error_category_in_chain``),+ report extraction and message synthesis (``_find_error_report_dict`` /+ ``_message_from_exc``), and the inline fail-safe guard+ (``_carries_temporal_failure``). The ``id()`` set makes a cyclic ``__cause__``+ chain terminate instead of spinning forever. That guard runs *on the error+ path*, so getting it wrong would turn the failure being reported into the very+ hang the reporting exists to surface — centralizing it here means it is+ written, and audited, once.+ """+ node: BaseException | None = exc+ seen: set[int] = set()+ while node is not None and id(node) not in seen:+ seen.add(id(node))+ yield node+ node = node.__cause__++# Placeholder ``message`` substituted into a STRICT-mode serialization of a# report whose ``message`` is not caller-facing copy. A report flagged# ``caller_facing_message`` keeps its original ``message`` instead.@@ -504,13 +527,8 @@ class PipelexError(Exception):# Guard against a cyclic __cause__ chain: if self is reachable from cause, recursing# into cause.to_error_report() would never terminate. Bail out with the enrichment# gathered so far rather than raising a RecursionError from the error-reporting path.- node: BaseException | None = cause- seen: set[int] = set()- while node is not None and id(node) not in seen:- if node is self:- return report- seen.add(id(node))- node = node.__cause__+ if any(node is self for node in iter_cause_chain(cause)):+ return reportcause_report = cause.to_error_report()# Only the cause-merged classification fields are updated; the wrapper-wins# fields (error_type, message, title, type_uri, caller_facing_message)diff --git a/pipelex/cli/agent_cli/commands/agent_output.py b/pipelex/cli/agent_cli/commands/agent_output.pyindex 441304a6f..22c3980a0 100644--- a/pipelex/cli/agent_cli/commands/agent_output.py+++ b/pipelex/cli/agent_cli/commands/agent_output.py@@ -26,7 +26,7 @@ from typing import Any, NoReturn, castimport typer-from pipelex.base_exceptions import PipelexError+from pipelex.base_exceptions import PipelexError, iter_cause_chainfrom pipelex.pipeline.exceptions import ValidateBundleErrorfrom pipelex.tools.misc.json_utils import clean_json_dumpsfrom pipelex.types import StrEnum@@ -197,11 +197,9 @@ def _build_error_source(exc: BaseException) -> list[str]:List of source location strings, outermost first."""sources: list[str] = []- current: BaseException | None = exc- while current is not None:+ for current in iter_cause_chain(exc):if current.__traceback__ is None:sources.append(f"{type(current).__name__} (no traceback)")- current = current.__cause__continuetbe = traceback.extract_tb(current.__traceback__)if tbe:@@ -214,7 +212,6 @@ def _build_error_source(exc: BaseException) -> list[str]:sources.append(location)else:sources.append(f"{type(current).__name__} (no traceback)")- current = current.__cause__return sourcesdiff --git a/pipelex/cogt/exceptions.py b/pipelex/cogt/exceptions.pyindex 4c125e556..cb11e294a 100644--- a/pipelex/cogt/exceptions.py+++ b/pipelex/cogt/exceptions.py@@ -4,7 +4,7 @@ from typing import TYPE_CHECKINGfrom typing_extensions import override-from pipelex.base_exceptions import ErrorReport, PipelexError+from pipelex.base_exceptions import ErrorReport, PipelexError, iter_cause_chainfrom pipelex.cogt.inference.error_classification import ProviderErrorMetadata, UserAction, UserActionKindfrom pipelex.system.pipelex_service.types import RemoteConfigSourcefrom pipelex.types import StrEnum@@ -122,17 +122,12 @@ def find_inference_error_category_in_chain(exc: BaseException) -> InferenceError(``non_retryable``) from the underlying failure's category. A ``CogtError`` carrying nocategory is skipped — the walk continues to the first one that actually classifies the failure.- The ``id()`` set guards against a cyclic ``__cause__`` chain: without it a cycle would- spin this loop forever — and it runs on the error path, so the failure being classified- would be lost to a hang rather than reported.+ Walks via ``iter_cause_chain``, which owns the cyclic-``__cause__`` guard so a cycle+ terminates the walk instead of hanging the error path."""- current: BaseException | None = exc- seen: set[int] = set()- while current is not None and id(current) not in seen:- seen.add(id(current))- if isinstance(current, CogtError) and current.error_category is not None:- return current.error_category- current = current.__cause__+ for node in iter_cause_chain(exc):+ if isinstance(node, CogtError) and node.error_category is not None:+ return node.error_categoryreturn Nonediff --git a/pipelex/pipe_operators/llm/pipe_llm.py b/pipelex/pipe_operators/llm/pipe_llm.pyindex 2c32c1238..3ec77b613 100644--- a/pipelex/pipe_operators/llm/pipe_llm.py+++ b/pipelex/pipe_operators/llm/pipe_llm.py@@ -4,6 +4,7 @@ from pydantic import ValidationErrorfrom typing_extensions import overridefrom pipelex import log+from pipelex.base_exceptions import iter_cause_chainfrom pipelex.cogt.content_generation.content_generator_dry import ContentGeneratorDryfrom pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocolfrom pipelex.cogt.exceptions import LLMCompletionError@@ -382,12 +383,10 @@ class PipeLLM(PipeOperator[PipeLLMOutput]):def _format_llm_error(self, exc: LLMCompletionError, settings: list[LLMSetting]) -> str:"""Format an LLMCompletionError, extracting and formatting any ValidationError in the chain."""error_details = str(exc)- current_exc: BaseException | None = exc- while current_exc is not None:+ for current_exc in iter_cause_chain(exc):if isinstance(current_exc, ValidationError):error_details += f"\n{format_pydantic_validation_error(current_exc)}"break- current_exc = current_exc.__cause__return f"{error_details}\nLLM settings: {settings}"@overridediff --git a/pipelex/temporal/temporal_task_manager.py b/pipelex/temporal/temporal_task_manager.pyindex 3b7f21cc4..714e46f88 100644--- a/pipelex/temporal/temporal_task_manager.py+++ b/pipelex/temporal/temporal_task_manager.py@@ -10,6 +10,7 @@ from temporalio.worker.workflow_sandbox import (from typing_extensions import overridefrom pipelex import log+from pipelex.base_exceptions import PipelexErrorfrom pipelex.config import get_configfrom pipelex.hub import get_class_registryfrom pipelex.system.runtime import WorkerMode, runtime_manager@@ -135,17 +136,15 @@ class TemporalTaskManager(TaskManager):workflows=workflows,activities=activities,workflow_runner=workflow_runner,- # Register WorkflowExecutionError as a workflow-failure type so a- # workflow re-raising it (e.g. WfPipeRun re-raising the- # execution_error from a failed child) surfaces as a terminal- # workflow failure instead of being treated as an- # unhandled-exception workflow task failure (which retries- # indefinitely). WfPipeRun and TemporalPipeRouter both catch- # ChildWorkflowError from their workflow.execute_child_workflow- # calls and wrap it in-place as WorkflowExecutionError; without- # this registration Temporal cannot tell that re-raise apart from- # a programmer bug.- workflow_failure_exception_types=[WorkflowExecutionError],+ # Make an escaping domain error a terminal workflow failure instead of Temporal's+ # default "unhandled exception = workflow-task failure", which retries indefinitely —+ # a silent hang (see docs/under-the-hood/error-model.md "Workflow-Level Fail-Safe+ # Floor"). PipelexError is the floor: any domain error that slips past the workflow-+ # level catch-alls in WfPipeRouter / WfPipeRun still fails terminally (degraded to a+ # synthesized report) rather than hanging. WorkflowExecutionError — the child-dispatch+ # wrapper raised by those workflows — is listed explicitly for intent. Scoped to+ # PipelexError, not Exception: transient Temporal/infra errors keep their task-retry.+ workflow_failure_exception_types=[WorkflowExecutionError, PipelexError],max_cached_workflows=profile.max_cached_workflows,max_concurrent_workflow_tasks=profile.max_concurrent_workflow_tasks,max_concurrent_activities=profile.max_concurrent_activities,diff --git a/pipelex/temporal/tprl/temporal_error.py b/pipelex/temporal/tprl/temporal_error.pyindex bfaa4d01c..cf20ec444 100644--- a/pipelex/temporal/tprl/temporal_error.py+++ b/pipelex/temporal/tprl/temporal_error.py@@ -5,7 +5,7 @@ from pydantic import ValidationErrorfrom temporalio import activityfrom temporalio.exceptions import ApplicationError-from pipelex.base_exceptions import ErrorReport, PipelexError+from pipelex.base_exceptions import ErrorReport, PipelexError, iter_cause_chainfrom pipelex.cogt.exceptions import find_inference_error_category_in_chainfrom pipelex.config import get_configfrom pipelex.temporal.exceptions import UnrecoverableWorkflowFailureError@@ -53,15 +53,11 @@ def _find_error_report_dict(exc: BaseException) -> dict[str, Any] | None:its failure via ``ChildWorkflowError.cause`` rather than ``__cause__``, soits caller passes ``exc.cause`` straight in."""- node: BaseException | None = exc- seen: set[int] = set()- while node is not None and id(node) not in seen:+ for node in iter_cause_chain(exc):if isinstance(node, ApplicationError):report_dict = error_report_dict_from_details(node.details)if report_dict is not None:return report_dict- seen.add(id(node))- node = node.__cause__return None@@ -79,14 +75,10 @@ def _message_from_exc(exc: BaseException) -> str:win and the synthesized preamble would be visually broken."""deepest_message = ""- node: BaseException | None = exc- seen: set[int] = set()- while node is not None and id(node) not in seen:- seen.add(id(node))+ for node in iter_cause_chain(exc):text = str(node)if text.strip():deepest_message = text- node = node.__cause__return deepest_message or repr(exc)@@ -237,19 +229,32 @@ class TemporalError(ApplicationError):)@classmethod- def from_message_exception(cls, exc: PipelexError) -> Self:- """Convert a Pipelex exception raised inside an activity into a ``TemporalError``.-- When the exception's ``__cause__`` chain carries a ``CogtError`` with an- ``InferenceErrorCategory``, retryability flows from ``category.is_retryable``- — recovered even when wrapper exceptions (``PipeRunError``, ``PipeRouterError``,- ``PipelineExecutionError``) sit on top. A chain with no categorized ``CogtError``- falls back to the configured ``non_retryable_error_types`` class-name list.+ def from_message_exception(cls, exc: PipelexError, force_non_retryable: bool = False) -> Self:+ """Convert a Pipelex exception into a ``TemporalError``.++ **Activity-side (default).** Retryability is derived from the ``__cause__``+ chain: a ``CogtError`` carrying an ``InferenceErrorCategory`` sets it from+ ``category.is_retryable`` — recovered even when wrapper exceptions+ (``PipeRunError``, ``PipeRouterError``, ``PipelineExecutionError``) sit on+ top. A chain with no categorized ``CogtError`` falls back to the configured+ ``non_retryable_error_types`` class-name list.++ **Workflow-side inline (``force_non_retryable=True``).** The inline fail-safe+ in ``WfPipeRouter`` / ``WfPipeRun`` converts a domain error raised directly in+ workflow code. It forces ``non_retryable=True`` for two reasons: an inline+ error must fail the workflow *terminally* rather than trigger a blunt whole-+ workflow retry that re-runs already-completed inline work (retry belongs at+ the activity boundary, where it is scoped and observable); and forcing the+ flag short-circuits ``_is_non_retryable`` — the only ``get_config()`` read in+ this method — so the conversion stays deterministic inside workflow code (a+ config-derived decision is replay-fragile across deploys). The report's own+ ``retryable`` field is config-free and preserved either way, as informational+ metadata."""message = exc.messageerror_type = exc.__class__.__name__error_report = exc.to_error_report().to_dict()- non_retryable = cls._is_non_retryable(exc=exc, error_type=error_type)+ non_retryable = force_non_retryable or cls._is_non_retryable(exc=exc, error_type=error_type)if non_retryable:cls._log_critical(f"Non retryable error from PipelexError[{error_type}]: {message}")else:diff --git a/pipelex/temporal/tprl_pipe/wf_pipe_router.py b/pipelex/temporal/tprl_pipe/wf_pipe_router.pyindex 2ec319777..6a6637403 100644--- a/pipelex/temporal/tprl_pipe/wf_pipe_router.py+++ b/pipelex/temporal/tprl_pipe/wf_pipe_router.py@@ -2,13 +2,14 @@ from datetime import timedeltafrom temporalio import workflowfrom temporalio.common import RetryPolicy-from temporalio.exceptions import ActivityError, ApplicationError+from temporalio.exceptions import ActivityError, ApplicationError, FailureErrorfrom typing_extensions import overridewith workflow.unsafe.imports_passed_through():from kajson.class_registry import ClassRegistryfrom kajson.kajson_manager import KajsonManager+ from pipelex.base_exceptions import PipelexError, iter_cause_chainfrom pipelex.config import get_configfrom pipelex.core.pipes.pipe_output import PipeOutputfrom pipelex.graph.graph_tracer_manager import GraphTracerManager@@ -22,6 +23,39 @@ with workflow.unsafe.imports_passed_through():from pipelex.tracing.buffering_event_log import BufferingEventLog+def _carries_temporal_failure(exc: BaseException) -> bool:+ """True when ``exc`` or any error in its ``__cause__`` chain is a Temporal ``FailureError``.++ This is the hinge of the inline fail-safe: ``True`` ⇒ leave ``exc`` untouched,+ ``False`` ⇒ convert it with ``from_message_exception``. Carrying a Temporal+ ``FailureError`` *is* the definition of "already a terminal Temporal failure",+ so it is the predicate, not a proxy for a narrower type.++ The reachable case it protects: a controller pipe (e.g. ``PipeSequence``) runs a+ sub-pipe as a child workflow; the sub-pipe fails; ``TemporalPipeRouter`` wraps the+ ``ChildWorkflowError`` as ``WorkflowExecutionError`` (``temporal_pipe_router.py``),+ which escapes the parent's ``pipe.run_pipe`` into the ``except PipelexError`` below.+ That ``WorkflowExecutionError`` carries no report of its own — the rich leaf+ classification lives deeper, in the child's ``ApplicationError.details``, reachable+ only by ``recover_error_report``'s ``__cause__`` walk at the submitter (which+ normalizes ``ChildWorkflowError.cause`` into the chain). A worker-side+ ``from_message_exception`` here would flatten it, because ``to_error_report``'s+ enrichment stops at the non-``PipelexError`` ``ChildWorkflowError``. So we propagate.++ Invariant this relies on: disciplined ``raise … from`` usage. The one way it can+ misattribute is "recover-then-rechain" — pipe code that handles a side failure and+ then raises a *fresh, unrelated* error ``from`` the one it already recovered. The+ fresh error would then carry that side ``FailureError`` and propagate untouched,+ surfacing the side failure's classification instead of its own. Don't do that: a+ fresh error must only be chained ``from`` the error that actually caused it.++ Walks via ``iter_cause_chain`` (single cyclic-guard). See+ ``docs/under-the-hood/error-model.md`` "Workflow-Level Fail-Safe Floor" for the+ design narrative.+ """+ return any(isinstance(node, FailureError) for node in iter_cause_chain(exc))++@workflow.defn(name="wf_pipe_router")class WfPipeRouter(WorkflowClass[PipeJob, PipeOutput]):@override@@ -147,6 +181,21 @@ class WfPipeRouter(WorkflowClass[PipeJob, PipeOutput]):if isinstance(exc.cause, ApplicationError):raise TemporalError.from_app_error(exc=exc.cause) from excraise+ except PipelexError as exc:+ # Inline fail-safe floor: a domain error raised inline in workflow code (never via an+ # activity) is neither an ActivityError nor an ApplicationError, so without this clause+ # it escapes as a non-terminal workflow-task failure and retries indefinitely — a silent+ # hang (see docs/under-the-hood/error-model.md "Workflow-Level Fail-Safe Floor"). Convert+ # a genuine inline error to a terminal, classified TemporalError; leave one that already+ # carries a Temporal failure untouched for the submitter to recover. The propagate-vs-+ # convert decision and the invariant it relies on live in _carries_temporal_failure.+ # Scoped to PipelexError: transient Temporal/infra errors keep Temporal's task-retry.+ # force_non_retryable: an inline error must fail terminally, not trigger a blunt whole-+ # workflow retry that re-runs completed inline work — retry belongs at the activity+ # boundary. It also keeps this workflow-side conversion config-free (deterministic).+ if _carries_temporal_failure(exc):+ raise+ raise TemporalError.from_message_exception(exc=exc, force_non_retryable=True) from excfinally:# Close per-workflow graph tracer (collects in-memory graph spec). F1: only assign the spec# when graph events were requested — in costs-only mode close_tracer returns None (teardowndiff --git a/pipelex/temporal/tprl_pipe/wf_pipe_run.py b/pipelex/temporal/tprl_pipe/wf_pipe_run.pyindex 02d3ad389..108d15876 100644--- a/pipelex/temporal/tprl_pipe/wf_pipe_run.py+++ b/pipelex/temporal/tprl_pipe/wf_pipe_run.py@@ -6,14 +6,14 @@ from temporalio.exceptions import ActivityError, ChildWorkflowErrorfrom typing_extensions import overridewith workflow.unsafe.imports_passed_through():- from pipelex.base_exceptions import ErrorReport # noqa: TC001 # must traverse the workflow sandbox+ from pipelex.base_exceptions import ErrorReport, PipelexError # must traverse the workflow sandboxfrom pipelex.core.pipes.pipe_output import PipeOutputfrom pipelex.pipe_run.delivery_assignment import DeliveryStatusfrom pipelex.runtime_bridge.primitives.pipe_run_arg import PipeRunArgfrom pipelex.temporal.exceptions import WorkflowExecutionErrorfrom pipelex.temporal.log_temporal import WorkflowLogfrom pipelex.temporal.tprl.observability import build_search_attributes, build_static_summary- from pipelex.temporal.tprl.temporal_error import recover_error_report+ from pipelex.temporal.tprl.temporal_error import TemporalError, recover_error_reportfrom pipelex.temporal.tprl.workflow_caller import WorkflowClassfrom pipelex.temporal.tprl_pipe.act_assemble_tracing import AssembleTracingArg, act_assemble_tracingfrom pipelex.temporal.tprl_pipe.act_deliver import DeliveryActivityArg, act_deliver@@ -80,6 +80,27 @@ class WfPipeRun(WorkflowClass[PipeRunArg, PipeOutput]):execution_error = WorkflowExecutionError("WfPipeRouter failed", error_report=error_report)execution_error.__cause__ = excworkflow_log.error(f"WfPipeRouter failed: {exc}")+ except PipelexError as exc:+ # Inline fail-safe floor, parent side: a domain error raised inline in this workflow+ # (e.g. building the child's search attributes / static summary, evaluated as arguments+ # to execute_child_workflow) is not a ChildWorkflowError, so without this clause it+ # escapes as a non-terminal workflow-task failure and hangs (see error-model.md+ # "Workflow-Level Fail-Safe Floor"). Route it through the same deferred re-raise as a+ # child failure so act_deliver still fires the FAILED webhook on the failure path, then+ # re-raise terminally below. Scoped to PipelexError: transient infra errors keep task-retry.+ status = DeliveryStatus.FAILED+ error_report = exc.to_error_report()+ execution_error = WorkflowExecutionError(exc.message, error_report=error_report)+ # Mint a details-carrying TemporalError as the cause so the classification survives the+ # workflow -> submitter serialization. The child-failure path gets this for free (the+ # child's TemporalError rides the ChildWorkflowError chain); the inline path has no such+ # child failure, so without this carrier the submitter would floor to a synthesized+ # UnrecoverableWorkflowFailureError. error_report (above) still feeds the webhook directly.+ # force_non_retryable: here the flag does not change WfPipeRun's own retry (the raised+ # WorkflowExecutionError drives that, like the child-failure path) — it is set to keep+ # this workflow-side conversion config-free (deterministic), matching the router.+ execution_error.__cause__ = TemporalError.from_message_exception(exc=exc, force_non_retryable=True)+ workflow_log.error(f"WfPipeRun inline failure: {exc}")# Step 2: Assemble full graph + usage from trace events (cross-worker)# Runs as an activity because DynamoDB reads are I/O forbidden in workflows. The dispatch isdiff --git a/tests/integration/pipelex/temporal/test_wf_pipe_run_failure_path.py b/tests/integration/pipelex/temporal/test_wf_pipe_run_failure_path.pyindex 631480c4c..b1392972e 100644--- a/tests/integration/pipelex/temporal/test_wf_pipe_run_failure_path.py+++ b/tests/integration/pipelex/temporal/test_wf_pipe_run_failure_path.py@@ -25,6 +25,7 @@ from temporalio.common import RetryPolicyfrom temporalio.exceptions import ApplicationErrorfrom temporalio.worker import UnsandboxedWorkflowRunner, Worker+from pipelex.base_exceptions import PipelexErrorfrom pipelex.core.pipes.pipe_output import PipeOutputfrom pipelex.pipe_run.delivery_assignment import DeliveryAssignment, DeliveryStatusfrom pipelex.pipe_run.pipe_job import PipeJob@@ -105,10 +106,11 @@ class TestWfPipeRunFailurePath:activities=[stub_act_deliver],workflow_runner=UnsandboxedWorkflowRunner(),# Mirror the production ``make_worker`` config: register- # ``WorkflowExecutionError`` as a workflow failure type so- # ``WfPipeRun`` re-raising it ends the workflow execution- # terminally instead of triggering indefinite task-failure retry.- workflow_failure_exception_types=[WorkflowExecutionError],+ # ``WorkflowExecutionError`` (the wrapped child failure) and the+ # ``PipelexError`` fail-safe floor as workflow failure types so a+ # workflow re-raising either ends the execution terminally instead+ # of triggering indefinite task-failure retry.+ workflow_failure_exception_types=[WorkflowExecutionError, PipelexError],):with pytest.raises(WorkflowFailureError):# ``maximum_attempts=1`` disables the workflow-level retry so thediff --git a/tests/integration/pipelex/temporal/test_wf_pipe_run_request_id_logging.py b/tests/integration/pipelex/temporal/test_wf_pipe_run_request_id_logging.pyindex d2d2b7a98..c4f77f286 100644--- a/tests/integration/pipelex/temporal/test_wf_pipe_run_request_id_logging.py+++ b/tests/integration/pipelex/temporal/test_wf_pipe_run_request_id_logging.py@@ -25,6 +25,7 @@ from temporalio.exceptions import ApplicationErrorfrom temporalio.worker import UnsandboxedWorkflowRunner, Workerfrom typing_extensions import override+from pipelex.base_exceptions import PipelexErrorfrom pipelex.core.pipes.pipe_output import PipeOutputfrom pipelex.pipe_run.delivery_assignment import DeliveryAssignmentfrom pipelex.pipe_run.pipe_job import PipeJob@@ -114,7 +115,7 @@ class TestWfPipeRunRequestIdLogging:workflows=[WfPipeRun, WfPipeRouterFailingStub],activities=[stub_act_deliver],workflow_runner=UnsandboxedWorkflowRunner(),- workflow_failure_exception_types=[WorkflowExecutionError],+ workflow_failure_exception_types=[WorkflowExecutionError, PipelexError],):with pytest.raises(WorkflowFailureError):# ``maximum_attempts=1`` keeps the failure terminal and single — seediff --git a/tests/integration/pipelex/temporal/test_workflow_inline_error_failsafe.py b/tests/integration/pipelex/temporal/test_workflow_inline_error_failsafe.pynew file mode 100644index 000000000..b0836fb0d--- /dev/null+++ b/tests/integration/pipelex/temporal/test_workflow_inline_error_failsafe.py@@ -0,0 +1,319 @@+"""Regression tests for the Temporal workflow-level fail-safe floor.++The error-handling bridge converts every error that crosses an *activity*+boundary into a terminal, classified Temporal failure. But a pipelex domain+error raised **inline in workflow code** — never dispatched through an activity —+is neither an ``ActivityError`` nor an ``ApplicationError``. Without a backstop+Temporal treats it as a *workflow-task* failure and retries the task+indefinitely: a silent, resource-burning hang that only surfaces (as the wrong,+generic timeout error) after the workflow execution timeout.++These tests pin the floor that closes that hole:++- ``WfPipeRouter`` converts an escaping ``PipelexError`` to a terminal+ ``TemporalError`` carrying the structured ``ErrorReport`` (rich path).+- ``WfPipeRun`` routes an inline ``PipelexError`` through its deferred-delivery+ path so the FAILED webhook still fires, then re-raises terminally.+- The worker's ``workflow_failure_exception_types`` includes ``PipelexError`` so+ even an *uncaught* domain error fails the workflow terminally (the floor) with+ a synthesized report, rather than hanging.++Every case sets a short ``workflow_execution_timeout`` so a regression that+reopens the hole fails fast (and on the wrong, timeout-shaped classification)+instead of hanging the suite for the full default budget.+"""++import uuid+from collections.abc import Generator+from datetime import timedelta+from typing import Any++import pytest+from pytest_mock import MockerFixture+from temporalio import workflow+from temporalio.client import Client as TemporalClient+from temporalio.common import RetryPolicy++from pipelex.config import get_config+from pipelex.core.pipes.pipe_abstract import PipeAbstract+from pipelex.core.pipes.pipe_output import PipeOutput+from pipelex.pipe_run.delivery_assignment import DeliveryAssignment, DeliveryStatus, WebhookTarget+from pipelex.pipe_run.pipe_job import PipeJob+from pipelex.pipe_run.pipe_run_mode import PipeRunMode+from pipelex.runtime_bridge.primitives.pipe_run_arg import PipeRunArg+from pipelex.temporal.exceptions import WorkflowExecutionError, WorkflowInputError+from pipelex.temporal.temporal_hub import get_task_manager+from pipelex.temporal.tprl.temporal_error import TemporalError+from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter+from pipelex.temporal.tprl_pipe.wf_pipe_run import WfPipeRun+from tests.integration.pipelex.error_handling.test_data import ErrorReportParityTestData+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle++# Bounds the regression hang: with the fail-safe in place every case fails+# near-instantly; only a regression that reopens the workflow-task-retry hole+# would consume this budget before surfacing a (wrong, timeout-shaped) error.+_FAILSAFE_EXECUTION_TIMEOUT = timedelta(seconds=30)++_RAW_INLINE_MESSAGE = "raw inline pipelex error, never caught"++[email protected](name="wf_raw_pipelex_error_stub")+class WfRawPipelexErrorStub:+ """Workflow that raises a raw, uncaught ``PipelexError`` inline.++ Exercises the worker-level floor (``workflow_failure_exception_types``+ contains ``PipelexError``) — no workflow-level catch-all is involved, so the+ only thing that can make this terminal instead of a retrying task failure is+ the registration on the production ``make_worker``.+ """++ @workflow.run+ async def run(self, workflow_arg: PipeJob) -> PipeOutput: # noqa: ARG002 - signature pinned by the WorkflowClass protocol; raises immediately+ raise WorkflowInputError(_RAW_INLINE_MESSAGE)++[email protected](loop_scope="class")+class TestWorkflowInlineErrorFailsafe:+ """A pipelex error escaping workflow code fails loud and classified, never hangs."""++ @pytest.fixture+ def temporal_enabled(self) -> Generator[None, None, None]:+ """Enable ``temporal.is_enabled`` so ``WorkflowExecutor.temporal_client()`` connects."""+ config = get_config()+ previous = config.temporal.is_enabled+ config.temporal = config.temporal.model_copy(update={"is_enabled": True})+ yield+ config.temporal = config.temporal.model_copy(update={"is_enabled": previous})++ @pytest.fixture+ def failing_pipe_job(self) -> Generator[PipeJob, None, None]:+ """A PipeJob in LIVE mode (the workflow runs the pipe instead of short-circuiting in dry mode)."""+ yield from pipe_job_from_bundle(+ bundle_file=ErrorReportParityTestData.BUNDLE_FILE,+ pipe_code=ErrorReportParityTestData.PIPE_CODE,+ pipe_run_mode=PipeRunMode.LIVE,+ )++ async def test_router_inline_pipelex_error_surfaces_classified_terminally(+ self,+ temporal_client: TemporalClient,+ mocker: MockerFixture,+ temporal_enabled: None, # noqa: ARG002 - enables temporal.is_enabled for the duration+ failing_pipe_job: PipeJob,+ ) -> None:+ """A ``PipelexError`` raised inline by the pipe (no activity) surfaces fully classified, not as a hang."""+ # Make the pipe raise *inline* in the workflow event loop — the escape path the activity+ # bridge never sees. ``run_pipe`` is @final on PipeAbstract, so patching the base covers+ # whatever concrete pipe the bundle resolves to.+ mocker.patch.object(+ PipeAbstract,+ "run_pipe",+ new=mocker.AsyncMock(side_effect=ErrorReportParityTestData.make_failing_llm_error()),+ )++ task_queue = f"q_inline_router_{uuid.uuid4().hex[:8]}"+ workflow_id = f"wf_inline_router_{uuid.uuid4().hex[:8]}"+ executor: WorkflowExecutor[PipeJob, PipeOutput] = WorkflowExecutor(+ temporal_client=temporal_client,+ task_queue=task_queue,+ retry_policy=RetryPolicy(maximum_attempts=1),+ workflow_execution_timeout=_FAILSAFE_EXECUTION_TIMEOUT,+ )++ async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):+ with pytest.raises(WorkflowExecutionError) as exc_info:+ await executor.execute_workflow(+ workflow_class=WfPipeRouter,+ workflow_arg=failing_pipe_job,+ workflow_id=workflow_id,+ )++ error = exc_info.value+ assert error.error_report is not None+ report = error.to_error_report()+ # The rich classification of the *inline* error survived — proving it went through the+ # fail-safe conversion, not the regression timeout path (which would synthesize an+ # UnrecoverableWorkflowFailureError instead).+ assert report.error_type == "LLMCompletionError"+ assert ErrorReportParityTestData.FAILURE_MESSAGE in report.message+ assert report.error_category == ErrorReportParityTestData.FAILURE_CATEGORY+ assert report.retryable == ErrorReportParityTestData.EXPECTED_RETRYABLE+ assert report.model == ErrorReportParityTestData.FAILURE_MODEL+ assert report.provider == ErrorReportParityTestData.FAILURE_PROVIDER+ assert report.user_action is not None+ assert report.user_action.kind == ErrorReportParityTestData.EXPECTED_USER_ACTION_KIND++ async def test_router_already_terminal_error_propagates_untouched_preserving_leaf_classification(+ self,+ temporal_client: TemporalClient,+ mocker: MockerFixture,+ temporal_enabled: None, # noqa: ARG002 - enables temporal.is_enabled for the duration+ failing_pipe_job: PipeJob,+ ) -> None:+ """An escaping error that ALREADY carries a Temporal failure propagates untouched.++ Pins the ``_carries_temporal_failure`` True branch — the reachable case where a controller+ pipe's sub-pipe fails as a child workflow and ``TemporalPipeRouter`` wraps the+ ``ChildWorkflowError`` as ``WorkflowExecutionError``, which then escapes the parent's+ ``pipe.run_pipe``. The rich leaf report rides an ``ApplicationError`` (a Temporal+ ``FailureError``) in the ``__cause__`` chain and is recoverable only by+ ``recover_error_report`` at the submitter. A regression that negated or deleted the guard+ would convert it via ``from_message_exception`` instead, flattening ``error_type`` to+ ``WorkflowExecutionError`` and dropping model / provider / category — so these assertions+ fail loudly on exactly that regression (the convert-branch sibling test above would still+ pass, which is why this branch needs its own coverage).+ """+ # The structured leaf report the nested failure carries, built like the activity bridge:+ # an LLMCompletionError -> ErrorReport -> dict, packed into ApplicationError.details.+ leaf_report_dict = ErrorReportParityTestData.make_failing_llm_error().to_error_report().to_dict()+ # The already-terminal carrier: a TemporalError IS an ApplicationError (a Temporal+ # FailureError) holding the leaf report in details — what sits in the chain after a child+ # WfPipeRouter fails. Wrapped via ``from`` as WorkflowExecutionError, mirroring+ # TemporalPipeRouter's ``raise WorkflowExecutionError(...) from ChildWorkflowError``.+ already_terminal = TemporalError(+ message=ErrorReportParityTestData.FAILURE_MESSAGE,+ error_type="LLMCompletionError",+ non_retryable=True,+ error_report=leaf_report_dict,+ )+ nested_failure = WorkflowExecutionError("nested sub-pipe failed")+ nested_failure.__cause__ = already_terminal+ mocker.patch.object(+ PipeAbstract,+ "run_pipe",+ new=mocker.AsyncMock(side_effect=nested_failure),+ )++ task_queue = f"q_inline_passthrough_{uuid.uuid4().hex[:8]}"+ workflow_id = f"wf_inline_passthrough_{uuid.uuid4().hex[:8]}"+ executor: WorkflowExecutor[PipeJob, PipeOutput] = WorkflowExecutor(+ temporal_client=temporal_client,+ task_queue=task_queue,+ retry_policy=RetryPolicy(maximum_attempts=1),+ workflow_execution_timeout=_FAILSAFE_EXECUTION_TIMEOUT,+ )++ async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):+ with pytest.raises(WorkflowExecutionError) as exc_info:+ await executor.execute_workflow(+ workflow_class=WfPipeRouter,+ workflow_arg=failing_pipe_job,+ workflow_id=workflow_id,+ )++ report = exc_info.value.to_error_report()+ # The LEAF classification survived end-to-end: error_type is the leaf's, NOT+ # "WorkflowExecutionError" — proof the guard propagated the error untouched and let the+ # submitter recover the original report, rather than re-wrapping it into a generic one.+ assert report.error_type == "LLMCompletionError"+ assert ErrorReportParityTestData.FAILURE_MESSAGE in report.message+ assert report.error_category == ErrorReportParityTestData.FAILURE_CATEGORY+ assert report.retryable == ErrorReportParityTestData.EXPECTED_RETRYABLE+ assert report.model == ErrorReportParityTestData.FAILURE_MODEL+ assert report.provider == ErrorReportParityTestData.FAILURE_PROVIDER++ async def test_wf_pipe_run_inline_error_fires_failed_webhook_and_surfaces_classification(+ self,+ temporal_client: TemporalClient,+ mocker: MockerFixture,+ temporal_enabled: None, # noqa: ARG002 - enables temporal.is_enabled for the duration+ failing_pipe_job: PipeJob,+ ) -> None:+ """An inline ``PipelexError`` in the parent workflow still fires the FAILED webhook and surfaces terminally."""+ # Force an inline domain error inside WfPipeRun itself — build_search_attributes is+ # evaluated as an argument to execute_child_workflow, before any child/activity exists.+ inline_error = WorkflowInputError("simulated inline search-attribute failure")+ mocker.patch(+ "pipelex.temporal.tprl_pipe.wf_pipe_run.build_search_attributes",+ side_effect=inline_error,+ )++ captured_webhook_payloads: list[dict[str, Any]] = []+ mock_httpx_client = mocker.AsyncMock()+ mock_response = mocker.MagicMock()+ mock_response.raise_for_status = mocker.Mock()+ mock_httpx_client.__aenter__ = mocker.AsyncMock(return_value=mock_httpx_client)+ mock_httpx_client.__aexit__ = mocker.AsyncMock(return_value=False)++ def capture_post(url: str, **kwargs: Any) -> Any: # noqa: ARG001 - url is positional, unused+ captured_webhook_payloads.append(kwargs["json"])+ return mock_response++ mock_httpx_client.post = mocker.AsyncMock(side_effect=capture_post)+ mocker.patch("pipelex.pipe_run.delivery_executor.httpx.AsyncClient", return_value=mock_httpx_client)++ delivery_assignment = DeliveryAssignment(webhooks=[WebhookTarget(url="https://test.example.com/hook")])+ pipe_run_arg = PipeRunArg(pipe_job=failing_pipe_job, delivery_assignment=delivery_assignment).prepare_for_temporal()++ task_queue = f"q_inline_run_{uuid.uuid4().hex[:8]}"+ workflow_id = f"wf_inline_run_{uuid.uuid4().hex[:8]}"+ executor: WorkflowExecutor[PipeRunArg, PipeOutput] = WorkflowExecutor(+ temporal_client=temporal_client,+ task_queue=task_queue,+ retry_policy=RetryPolicy(maximum_attempts=1),+ workflow_execution_timeout=_FAILSAFE_EXECUTION_TIMEOUT,+ )++ async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):+ with pytest.raises(WorkflowExecutionError) as exc_info:+ await executor.execute_workflow(+ workflow_class=WfPipeRun,+ workflow_arg=pipe_run_arg,+ workflow_id=workflow_id,+ )++ # The FAILED webhook fired even though the failure never reached pipe execution — a terminal+ # failure must always notify the receiver, including on an inline workflow-setup error.+ assert len(captured_webhook_payloads) == 1, f"expected exactly one FAILED webhook, got {len(captured_webhook_payloads)}"+ payload = captured_webhook_payloads[0]+ assert payload["status"] == DeliveryStatus.FAILED+ assert "error" in payload, "FAILED delivery webhook must include the structured error report"+ assert payload["error"]["error_type"] == "WorkflowInputError"+ assert "simulated inline search-attribute failure" in payload["error"]["message"]++ # The submitter sees the same inline error, classified — not a generic timeout.+ submitter_error = exc_info.value+ assert submitter_error.error_report is not None+ assert submitter_error.error_report.error_type == "WorkflowInputError"+ assert "simulated inline search-attribute failure" in submitter_error.message++ async def test_uncaught_pipelex_error_fails_terminally_on_production_worker(+ self,+ temporal_client: TemporalClient,+ temporal_enabled: None, # noqa: ARG002 - enables temporal.is_enabled for the duration+ failing_pipe_job: PipeJob,+ ) -> None:+ """The worker-level floor: a raw, uncaught ``PipelexError`` fails terminally instead of retrying forever."""+ task_queue = f"q_floor_{uuid.uuid4().hex[:8]}"+ workflow_id = f"wf_floor_{uuid.uuid4().hex[:8]}"+ executor: WorkflowExecutor[PipeJob, PipeOutput] = WorkflowExecutor(+ temporal_client=temporal_client,+ task_queue=task_queue,+ retry_policy=RetryPolicy(maximum_attempts=1),+ workflow_execution_timeout=_FAILSAFE_EXECUTION_TIMEOUT,+ )++ # Use the production make_worker so the assertion is against the real+ # workflow_failure_exception_types config, with the stub registered alongside it.+ async with get_task_manager().make_worker(+ temporal_client,+ task_queue=task_queue,+ is_not_sandboxed=True,+ test_workflows=[WfRawPipelexErrorStub],+ ):+ with pytest.raises(WorkflowExecutionError) as exc_info:+ await executor.execute_workflow(+ workflow_class=WfRawPipelexErrorStub,+ workflow_arg=failing_pipe_job,+ workflow_id=workflow_id,+ )++ error = exc_info.value+ assert error.error_report is not None+ # No catch-all converted it, so the floor degrades to a synthesized report — but it is+ # terminal and preserves the original message, proving the workflow failed loud rather+ # than hanging (a hang would have surfaced a timeout message instead).+ assert error.error_report.error_type == "UnrecoverableWorkflowFailureError"+ assert _RAW_INLINE_MESSAGE in error.messagediff --git a/tests/unit/pipelex/temporal/test_temporal_error_bridge.py b/tests/unit/pipelex/temporal/test_temporal_error_bridge.pyindex 73f7536db..e80bde467 100644--- a/tests/unit/pipelex/temporal/test_temporal_error_bridge.py+++ b/tests/unit/pipelex/temporal/test_temporal_error_bridge.py@@ -156,6 +156,42 @@ class TestTemporalErrorBridge:assert temporal_error.error_report is not Noneassert temporal_error.error_report["error_type"] == "CogtError"+ def test_force_non_retryable_is_deterministic_and_preserves_report_retryable(+ self,+ log_mocks: tuple[Any, Any],+ mocker: MockerFixture,+ ) -> None:+ """The inline workflow-side conversion forces ``non_retryable=True``.++ Workflow-inline callers (``WfPipeRouter`` / ``WfPipeRun``) pass+ ``force_non_retryable=True`` so an inline domain error fails the workflow+ terminally instead of triggering a blunt whole-workflow retry that re-runs+ already-completed inline work — retry belongs at the activity boundary. The+ forced path must also be config-free: it short-circuits the+ ``_is_non_retryable`` decision (the only ``get_config()`` read in the+ conversion), so the conversion stays deterministic for Temporal replay. The+ report keeps its own ``retryable`` field as informational metadata,+ independent of the Temporal retry flag.+ """+ _ = log_mocks # silences the bridge log helpers; severity is asserted elsewhere+ # A genuinely retryable error: a TRANSIENT CogtError reports retryable=True.+ exc = CogtError("transient boom", error_category=InferenceErrorCategory.TRANSIENT)++ # Default (activity-side) path still decides retryability normally — retryable here.+ default_side = TemporalError.from_message_exception(exc=exc)+ assert default_side.non_retryable is False++ # Forced (workflow-inline) path skips the retry-decision logic entirely: no config read.+ decision_spy = mocker.spy(TemporalError, "_is_non_retryable")+ forced = TemporalError.from_message_exception(exc=exc, force_non_retryable=True)+ decision_spy.assert_not_called()++ # The Temporal retry flag is terminal regardless of the error's own retryability ...+ assert forced.non_retryable is True+ # ... while the report preserves the informational retryable metadata (still True).+ assert forced.error_report is not None+ assert forced.error_report["retryable"] is True+@pytest.mark.parametrize(("error_category", "expected_non_retryable"),[diff --git a/tests/unit/pipelex/test_iter_cause_chain.py b/tests/unit/pipelex/test_iter_cause_chain.pynew file mode 100644index 000000000..f94eb5457--- /dev/null+++ b/tests/unit/pipelex/test_iter_cause_chain.py@@ -0,0 +1,42 @@+"""Unit tests for ``iter_cause_chain`` — the centralized, cycle-guarded ``__cause__`` walk.++This primitive is load-bearing on the error-reporting path: ``_carries_temporal_failure``,+``find_inference_error_category_in_chain``, ``_find_error_report_dict``, ``_message_from_exc``+and the is-self cycle check in ``_enrich_error_report_from_cause`` all delegate to it. A walk+that started at ``exc.__cause__`` instead of ``exc`` would silently break the predicates that+inspect the seed node, while the indirect coverage (``test_message_from_exc``) stayed green.+These tests pin the two contract properties that nothing else exercises: the seed is yielded+first, and every node is visited exactly once even on a cyclic chain.+"""++from pipelex.base_exceptions import iter_cause_chain+++class TestIterCauseChain:+ def test_yields_seed_first_then_each_cause_in_order(self) -> None:+ """The walk starts at ``exc`` itself, then follows ``__cause__`` outermost-first."""+ leaf = ValueError("leaf")+ middle = RuntimeError("middle")+ outer = RuntimeError("outer")+ middle.__cause__ = leaf+ outer.__cause__ = middle+ assert list(iter_cause_chain(outer)) == [outer, middle, leaf]++ def test_single_exception_yields_only_itself(self) -> None:+ """An exception with no ``__cause__`` yields exactly the seed."""+ exc = RuntimeError("alone")+ assert list(iter_cause_chain(exc)) == [exc]++ def test_self_loop_yields_seed_once(self) -> None:+ """A node that is its own ``__cause__`` terminates after a single yield."""+ exc = RuntimeError("self")+ exc.__cause__ = exc+ assert list(iter_cause_chain(exc)) == [exc]++ def test_two_cycle_yields_each_node_exactly_once(self) -> None:+ """A 2-cycle yields both nodes once, in walk order, without spinning forever."""+ first = RuntimeError("first")+ second = RuntimeError("second")+ first.__cause__ = second+ second.__cause__ = first+ assert list(iter_cause_chain(first)) == [first, second]diff --git a/wip/error-handling/README.md b/wip/error-handling/README.mdindex 93596a882..ccb59def2 100644--- a/wip/error-handling/README.md+++ b/wip/error-handling/README.md@@ -13,7 +13,7 @@ Each `track-*.md` describes **what the code does today**. What is still open is| Extract / Classify / Render | Landed — every inference worker's `except` block now collapses to `extract → classify → raise render(...) from exc`. **Classify** (`pipelex/cogt/inference/error_classify.py`) and **Render** (`error_render.py`) are single shared functions; only the `extract_*_metadata` functions stay per-provider. A new `tests/unit/pipelex/cogt/inference/test_provider_classification_parity.py` walks every `ProviderName` against the extract-fn registry to catch unwired providers. | [track-extract-classify-render.md](track-extract-classify-render.md) || Retry & resilience | Landed — direct (non-Temporal) execution is a single pipeline-level attempt with no application-level retry loop; resilience is the Temporal track's job (activity `RetryPolicy` keyed off `InferenceErrorCategory`). `PipeBatch` bounds fan-out via `gather_bounded` (chunked, `max_concurrency`, default 8) — admission control, not retry. Tier 1 (transport retry) is an explicit, uniform policy: `cogt.transport_max_retries` is wired into every inference SDK client factory, the SDK-less `azure_rest` image-gen path has a `tenacity` transport-retry floor, and `instructor`'s structured-output retry is confined to schema re-ask. | [track-retry-and-resilience.md](track-retry-and-resilience.md) || CLI delivery | Landed — agent CLI emits markdown by default for `run`/`validate`/`init` (and `models`/`check-model`/`doctor`), `--format json` for the structured payload; the error path follows the same option via a per-invocation `ContextVar`. `ErrorReport` carries an authoritative `error_domain` → HTTP-status mapping. `display_error_panel` collapses the field-shaped Rich handlers onto one helper. | [track-cli-delivery.md](track-cli-delivery.md) |-| Temporal integration | Landed — `TemporalError.from_message_exception()` derives retryability from `InferenceErrorCategory.is_retryable` for category-carrying `CogtError`s (the `non_retryable_error_types` list is the fallback) and packs the full `ErrorReport` into `ApplicationError.details`. Every in-scope activity is decorated with `@convert_pipelex_errors`, so the bridge runs in production. The symmetric workflow → submitter and child-workflow → parent recovery is landed: `recover_error_report` reads the report back out of `ApplicationError.details`, so a pipe failing on a Temporal worker reaches the CLI and HTTP adapters with the same classification as a local run. | [track-temporal-integration.md](track-temporal-integration.md) |+| Temporal integration | Landed — `TemporalError.from_message_exception()` derives retryability from `InferenceErrorCategory.is_retryable` for category-carrying `CogtError`s (the `non_retryable_error_types` list is the fallback) and packs the full `ErrorReport` into `ApplicationError.details`. Every in-scope activity is decorated with `@convert_pipelex_errors`, so the bridge runs in production. The symmetric workflow → submitter and child-workflow → parent recovery is landed: `recover_error_report` reads the report back out of `ApplicationError.details`, so a pipe failing on a Temporal worker reaches the CLI and HTTP adapters with the same classification as a local run. A later pass added the **workflow-level fail-safe floor** (an escaping *inline* `PipelexError` — never dispatched through an activity — now fails the workflow terminally and classified instead of hanging on indefinite task-retry); landed on `fix/Temporal-failsafe`. Design rationale + open review backlog live in the three `temporal-failsafe-*.md` docs — see [What's still open](#whats-still-open) item 4. | [track-temporal-integration.md](track-temporal-integration.md) || Testing | Landed — worker-level classification tests, the `agent_output.py` dict-drift test, and the full-chain worker→runner→CLI integration test (JSON + markdown) plus Rich-panel snapshot tests are all in place. | [track-testing.md](track-testing.md) || API readiness (error-handling companion) | Landed — `PipelexError.title()` / `type_uri()`, `request_id` on `JobMetadata`, `DisclosureMode` with `to_dict(disclosure_mode=)` / `to_problem_document()`, total `recover_error_report`, `ErrorReport` as a `BaseModel` threaded to the webhook, and per-class error doc pages. Webhook signing is the last open item — see [What's still open](#whats-still-open). | [api-readiness.md](api-readiness.md) |@@ -26,6 +26,7 @@ The bulk of the error-handling work has landed. What remains:1. **Webhook signing** — the last unshipped piece of the API-readiness companion work. Cross-repo: the pipelex side and the API side land in lockstep. Authoritative plan: [../webhook-signing.md](../webhook-signing.md).2. **Metadata-model long tail** — a handful of `CogtError` subclasses still carry no class-level `error_category`, and several non-inference `PipelexError` subclasses still depend on the `agent_output.py` fallback dicts for `hint` / `error_domain` rather than carrying class-level metadata. See the Followups in [track-metadata-model.md](track-metadata-model.md).3. **`request_id` correlation on delivery error paths** — `request_id` is on the success-path completion logs but not on the failure-path messages of `StorageDeliveryError` / `WebhookDeliveryError`, nor on the `pipe_run.py:79` debug entry log. Small but real. See [track-delivery-error-path-request-id.md](track-delivery-error-path-request-id.md).+4. **Temporal fail-safe floor — review backlog (parked, gated on Temporal productionization).** The silent-hang hole is *closed* (workflow-level catch-alls + `PipelexError` worker floor, landed on `fix/Temporal-failsafe`), but the code review left follow-ups open: a `SecurityError`-contract decision (Finding 3), small cleanups (5, 8, 10), a product question (12), and two findings from a later `/review` pass — the `_carries_temporal_failure` predicate's third fragility mode (Finding 13) and a pre-existing `build_search_attributes` replay-determinism hazard (Finding 14). **Live backlog:** [temporal-failsafe-review-followups.md](temporal-failsafe-review-followups.md). Fix rationale (✅ resolved, read-only): [temporal-error-handling-failsafe-gap.md](temporal-error-handling-failsafe-gap.md) (the "why it was the real defect" brief; also tracks deferred hardening C/D/F — submitter deadline, CI dispatch-guard, stuck-workflow observability) and [temporal-failsafe-guard-hardening.md](temporal-failsafe-guard-hardening.md) (the guard/consolidation findings 1/6/7/9).## Suggested read order@@ -37,7 +38,7 @@ Tracks are independent for implementation purposes, but there is a natural onram4. [track-extract-classify-render.md](track-extract-classify-render.md) — landed refactor: decomposes the per-worker pipeline into one shared `extract → classify → render` chain.5. [track-cli-delivery.md](track-cli-delivery.md) — Layer 4 → 5, where the classified errors get rendered for humans and agents.6. [track-retry-and-resilience.md](track-retry-and-resilience.md) — builds on classification and metadata to drive retry decisions.-7. [track-temporal-integration.md](track-temporal-integration.md) — extends the same model across the activity → workflow boundary.+7. [track-temporal-integration.md](track-temporal-integration.md) — extends the same model across the activity → workflow boundary. For the fail-safe floor that backstops it, the deep dive is the three `temporal-failsafe-*.md` docs (start at [temporal-failsafe-review-followups.md](temporal-failsafe-review-followups.md) for live work; the other two are resolved rationale).8. [track-testing.md](track-testing.md) — cross-cutting, comes last because it verifies the others.This is reading order, not implementation order — any track can be picked up independently.diff --git a/wip/error-handling/temporal-error-handling-failsafe-gap.md b/wip/error-handling/temporal-error-handling-failsafe-gap.mdnew file mode 100644index 000000000..57d957420--- /dev/null+++ b/wip/error-handling/temporal-error-handling-failsafe-gap.md@@ -0,0 +1,114 @@+# The real defect: the Temporal integration can hang silently instead of failing loud++> **Cold start:** this doc is the *why* — resolved rationale, read-only. The live open backlog is in [`temporal-failsafe-review-followups.md`](./temporal-failsafe-review-followups.md); the directory index is [`README.md`](./README.md). Read here only when you need the rationale behind the fix or the deferred-hardening list (C/D/F) below.++**Status:** ✅ **FIXED** (fixes A + B + E landed on `fix/Temporal-failsafe`). The fail-safe floor is in place; the silent-hang hole is closed. Deferred hardening (C, D, F) is tracked under "Deferred follow-ups" below — none of it is required to close the hole.+**Severity:** high — this is a class of bug, not one bug. The missing Search activity (see [`search-operator-missing-temporal-activity.md`](./search-operator-missing-temporal-activity.md)) is only the first trigger. Any future code path where an exception escapes pipe execution without having been converted to a Temporal terminal failure would hang the runner the same way. Fixing Search closes the instance; the fix below closes the hole.+**Companion brief:** read the Search brief first for the concrete instance; this one is about why the system let that instance become a silent hang.++## Resolution (implemented)++The fix establishes the fail-safe floor the design lacked, scoped to pipelex **domain** errors so transient Temporal/infra errors keep Temporal's (correct) task-retry behavior:++- **Fix A — workflow-level catch-all.** `WfPipeRouter.run()` and `WfPipeRun.run()` each gained a final `except PipelexError` boundary. `WfPipeRouter` converts a *genuine inline* error to a terminal `TemporalError` via `from_message_exception` (rich report in `ApplicationError.details`, exactly as the activity bridge does). `WfPipeRun` routes the inline error through its deferred-delivery path so the FAILED webhook still fires, then chains a details-carrying `TemporalError` so the classification also reaches the submitter. The catch-all converts only genuine inline errors: an escaping `PipelexError` that already carries a Temporal failure in its `__cause__` chain (e.g. a nested `TemporalPipeRouter` child-dispatch that already wrapped a sub-pipe failure as `WorkflowExecutionError`) is already terminal and recoverable, so it propagates untouched (guarded by `_carries_temporal_failure`) rather than being flattened to a generic report.+- **Fix B — worker-level floor.** `workflow_failure_exception_types` is now `[WorkflowExecutionError, PipelexError]` (`pipelex/temporal/temporal_task_manager.py`). Any domain error that slips past the catch-alls fails the workflow terminally instead of hanging — degrading to a synthesized `UnrecoverableWorkflowFailureError` (message preserved) rather than a silent retry loop.+- **Fix E — defensive tests.** `tests/integration/pipelex/temporal/test_workflow_inline_error_failsafe.py` pins the escape path: (1) an inline `PipelexError` in `WfPipeRouter` surfaces fully classified; (2) an inline `PipelexError` in `WfPipeRun` fires the FAILED webhook and reaches the submitter classified; (3) a raw uncaught `PipelexError` on the *production* worker fails terminally with the message preserved. Each sets a short `workflow_execution_timeout` so a regression that reopens the hole fails fast (on the wrong, timeout-shaped classification) instead of hanging the suite. The two contract tests that mirror the production worker config were updated to the broadened list. Full temporal suite green.++Docs updated: `docs/under-the-hood/error-model.md` gains a "Workflow-Level Fail-Safe Floor" subsection.++## Deferred follow-ups++These harden the system further but are **not** required to close the silent-hang hole (A + B already do). They are deferred because each is a distinct concern — cross-repo, a different bug class, or ops:++- **C — separate short submitter-side deadline for the synchronous API path.** Genuinely cross-repo: the sync `/execute` endpoint lives in `pipelex-api`, and giving it a bounded result-wait distinct from the 1h durable `workflow_execution_timeout` requires a caller-side change there (plus a config knob plumbed from `WorkerConfig` here). With A + B the silent hang is already closed — any domain error now fails terminally and fast — so C is defense-in-depth against *other* causes of slowness (genuinely slow pipes, infra hangs), not this defect. The brief's "no `pipelex-api` change" framing rules C out of a pipelex-only fix.+- **D — CI guard for the activity-dispatch invariant.** This is the regression guard for the *Search-class trigger* (an inference operator forgetting to route its leaf through an activity), tracked by the companion Search brief — not the systemic hole this brief is about. The floor (A + B) already degrades any such omission to a loud, classified failure instead of a hang, so D is a quality guard for the trigger, deferrable independently.+- **F — observability on stuck workflows.** A metric/alert on workflow-task-failure accumulation (or workflows open past an interactive threshold). Ops/infra concern; with A + B nothing should reach the stuck state for a domain error, so this is a backstop-for-the-backstop.++## Where the fix happens++Entirely in the `pipelex` repo (the open-source runtime). This brief is filed in `pipelex-api/wip/` because the symptom showed up there, but no `pipelex-api` change is involved. The next session is expected to run from a `pipelex` worktree.++**Path convention:** every `pipelex/...` path is relative to the **pipelex repo root** = your worktree root. Work inside the worktree; do not reach into a sibling `../pipelex/` checkout.++## One-sentence statement of the defect++The error-handling architecture (`pipelex/wip/error-handling/`) makes **terminal** Temporal failures cross the boundary beautifully — but it has **no backstop for the non-terminal "workflow task failure" mode**, which is exactly what any unconverted exception inside pipe code triggers, and which Temporal's default turns into a silent, ~1-hour hang that finally surfaces as the *wrong* error.++## Why this is the actual problem (not "Search forgot an activity")++The error-handling work invested heavily in the **terminal-failure recovery chain**:++- activity → workflow: `convert_pipelex_errors` makes leaf errors `TemporalError(ApplicationError)`; `from_app_error` recovers them in workflow code.+- workflow → submitter: `recover_error_report` + the `except WorkflowFailureError` clause rebuild the `ErrorReport` on the caller.+- child → parent: `WfPipeRun`'s `except ChildWorkflowError` lifts the report out of `exc.cause`.++Every one of these handlers fires **only when the inner unit fails *terminally* with a recognized failure type** (`ApplicationError` / `FailureError` / `WorkflowExecutionError`). That is the unstated, load-bearing precondition of the entire design.++Temporal has a second failure mode that the design never accounts for: a **workflow task failure**. When workflow code raises any exception that is *not* an `ApplicationError`/`FailureError` and *not* in the worker's `workflow_failure_exception_types`, Temporal does **not** fail the workflow. It assumes a transient/code-deploy bug, and **retries the workflow task** — re-running the workflow code from the last consistent point, indefinitely, bounded only by the workflow execution timeout.++A workflow-task failure reaches **none** of the recovery handlers above (there is no terminal failure to recover). The child never produces a `ChildWorkflowError`; the parent's `await execute_child_workflow(...)` simply never returns; the submitter's `execute_workflow(...)` simply never returns. The polished recovery chain is bypassed wholesale, and the system's behavior degrades to its worst possible quadrant: **invisible + unbounded-ish + resource-consuming.**++## The exact chain (Search as the worked example)++> Pre-fix walkthrough: line numbers are dropped and present-tense handler claims are framed as the pre-fix state, since the fix (the two `except PipelexError` catch-alls + the broadened worker registration) changed exactly the code this describes.++1. `WfPipeRouter.run()` executes the pipe inline (its `pipe.run_pipe` call). At the time its only handler was `except ActivityError`.+2. A `SearchJobFailureError` (a `CogtError`/`PipelexError`) is raised inline — **not** an `ActivityError`, **not** an `ApplicationError`, **not** a `WorkflowExecutionError`. It escapes the child workflow raw.+3. Worker then registered `workflow_failure_exception_types=[WorkflowExecutionError]` only (`pipelex/temporal/temporal_task_manager.py`). The escaping type isn't in it → **child workflow-task failure → child retries its task**, re-executing the inline search call for real on each retry (real provider spend + log spam; no result is recorded in history because there is no activity).+4. The child never fails terminally, so `WfPipeRun`'s `except ChildWorkflowError` (`pipelex/temporal/tprl_pipe/wf_pipe_run.py`) — the handler that would have produced a classified `WorkflowExecutionError` and fired the failure webhook — **never runs**. The parent's `await workflow.execute_child_workflow(...)` blocks.+5. The submitter's `execute_workflow(...)` (`pipelex/temporal/tprl/workflow_caller.py`) is awaiting the parent's result with `execution_timeout = workflow_execution_timeout` (configured **1h**). The child has **no** execution timeout of its own (deliberately omitted for replay determinism — see `WfPipeRun.run`'s child-dispatch comment), so the whole tree is bounded only by the parent's 1h.+6. After ~1h the parent execution times out. The submitter gets a `WorkflowFailureError(TimeoutError)`; `recover_error_report` finds no `ApplicationError` report in a timeout chain and synthesizes an `UnrecoverableWorkflowFailureError` (`error_domain=RUNTIME`). The API returns **that opaque timeout** — not the `SearchJobFailureError` that was plainly in the worker logs the entire hour.++Net: a sync `/execute` request hangs for up to an hour and then returns the wrong, generic error, while the worker silently burns retries. The "improve error handling so it behaves well for the API" goal is fully defeated by this single uncovered mode.++## The systemic gaps (each is independent of the Search feature)++1. **The "every leaf error is converted at an activity boundary" invariant is unenforced.** It holds today only by manual enumeration — the hand-maintained `Tasks` activity list (`pipelex/temporal/tasks.py`) and `@convert_pipelex_errors` applied by hand per activity. Nothing fails when a new operator skips activity dispatch. Enumerated coverage rots; Search is the proof.++2. **No workflow-level catch-all that makes escaping domain exceptions terminal.** `WfPipeRouter` catches only `ActivityError`; `WfPipeRun` catches only `ChildWorkflowError`. Neither has a final clause that wraps an escaping `PipelexError` into a `WorkflowExecutionError` (the one type that *is* terminal here). Such a backstop would have turned the Search hang into an immediate, classified, surfaced failure — i.e. exactly the promised behavior — for any cause, known or not.++3. **`workflow_failure_exception_types` is set to the narrowest possible value and the default is the dangerous one.** It's `[WorkflowExecutionError]`. Everything depends on code paths *manually* arriving at that one type. The safer posture is to make Temporal fail the workflow terminally for any pipelex **domain** error by default (e.g. include `PipelexError`), so a missed conversion degrades to "fails loud with a slightly-less-rich report" instead of "hangs for an hour and lies."++4. **The synchronous API path has no short deadline and mis-prioritizes the surfaced error.** Two distinct faults: (a) a sync HTTP endpoint inherits the **1h durable** workflow execution timeout — there is no separate, short submitter-side result-wait for the interactive path; (b) on timeout the caller surfaces an opaque `UnrecoverableWorkflowFailureError` rather than the real, already-observed failure.++5. **Worst-quadrant failure with no observability.** During the hang there is no terminal event, no surfaced error, and (presumably) no alert — only an accumulating count of workflow-task failures visible in the Temporal UI if someone looks. A silent, resource-consuming, unbounded-until-timeout state is the single worst outcome a durable-execution system can produce, and nothing flags it.++6. **Tests pin the *terminal-classification* path, not the *escape* path.** The parity integration tests (`tests/integration/pipelex/temporal/test_workflow_error_report_full_chain.py` ↔ `.../error_handling/test_error_report_local_full_chain.py`) mock an activity to fail and assert `ErrorReport` parity — i.e. they exercise the path where conversion already happened. The defensive case that actually broke — "an unconverted exception escapes the workflow" — is untested. The test suite encoded the architecture's blind spot.++## Recommended fixes (systemic; do these even after Search is wired)++> **Status:** A, B, E are implemented — see [Resolution](#resolution-implemented) above. C, D, F are [deferred](#deferred-follow-ups). The original analysis is kept below for rationale.++Ordered by leverage. A and C are the floor; the rest harden and prevent regression.++**A. Workflow-level fail-safe (highest leverage).** In both `WfPipeRouter.run()` and `WfPipeRun.run()`, add a final boundary that converts an escaping pipelex **domain** exception into a terminal `WorkflowExecutionError` carrying `recover_error_report(...)` classification — mirroring the existing `ActivityError`/`ChildWorkflowError` handlers, but for the "raised inline, never went through an activity" case. This guarantees no pipe-code exception can become a silent workflow-task-retry hang again, for *any* operator.+ - **Scope it carefully.** Do **not** blanket-convert `Exception`. Workflow-task retry is the *correct* behavior for genuinely transient Temporal/infra errors and for deterministic-replay glitches — making those terminal would throw away durability's whole point. Convert pipelex **domain** errors (`PipelexError` and friends), and let Temporal-internal exceptions keep their default. Per repo standards, a workflow-root `except` is one of the two sanctioned `except Exception` sites only if you immediately re-raise as terminal *and* you've reasoned about the transient case; prefer `except PipelexError` here.++**B. Broaden `workflow_failure_exception_types` to include `PipelexError`.** Cheap belt-and-suspenders behind A: even a path that slips past the catch-all fails the workflow terminally instead of hanging. (`pipelex/temporal/temporal_task_manager.py`.) Verify interaction with the existing `WfPipeRun` contract test `test_wf_pipe_run_failure_path` (it pins the current `[WorkflowExecutionError]` value).++**C. Separate, short submitter-side deadline for synchronous execution.** Give the sync `/execute` path a bounded result-wait (config, seconds-to-low-minutes) distinct from the 1h durable `workflow_execution_timeout` that the async `/start` path legitimately needs. On deadline, surface a clear, classified timeout — never leave the HTTP request hanging on the durable budget. (Submitter side: `pipelex/temporal/tprl/workflow_caller.py` + the timeout config plumbed from `WorkerConfig`.)++**D. Enforce the activity-dispatch invariant in CI.** A test that walks the inference `PipeOperator` subclasses and asserts each routes its leaf through `get_content_generator()` (equivalently: each inference leaf has a protocol method + a registered activity). The next operator that forgets the wiring then fails CI, not production. (This is the regression guard for the Search-class bug.)++**E. Negative/defensive test for the escape path.** Assert that a raw `PipelexError` raised *inside pipe code* within a workflow (not via an activity) results in a terminal, classified failure promptly surfaced to the submitter — the exact invariant that was silently violated. Pair it across local/Temporal like the existing parity tests.++**F. Observability on stuck workflows.** A metric/alert on workflow-task-failure accumulation (or workflows open past a sane interactive threshold), so a path that still slips through A–E is visible instead of silent.++## The meta-lesson worth recording++The error-handling project optimized the **fail-loud** path to a high polish but never established a **fail-safe floor**. It assumed terminal classification everywhere and built rich recovery on top of that assumption, without a guard for the case where the assumption doesn't hold. In a durable-execution system the default for "unexpected exception in business logic running inside a workflow" is the most dangerous behavior available (retry forever), so "we convert all the errors we know about" is not enough — the system needs a backstop that holds for the errors, and the code paths, that nobody enumerated. Search exposed it; the next gap will too unless the floor exists.++## Reference file map (relative to pipelex repo root / your worktree root)++> Symbolic refs (clause / function names), not line numbers — the fix shifted the lines and the catch-alls below now exist, so line-pinned navigation would mislead. Where the *gap* code differs from current reality it is noted inline.++- Workflow that runs pipe code inline (`WfPipeRouter.run`'s `pipe.run_pipe` call): `pipelex/temporal/tprl_pipe/wf_pipe_router.py` — its `except ActivityError` handler, now backstopped by the `except PipelexError` inline fail-safe this fix added.+- Parent workflow (`WfPipeRun.run`), `except ChildWorkflowError` plus the `except PipelexError` inline fail-safe this fix added; child started without its own execution timeout (deliberate, for replay determinism): `pipelex/temporal/tprl_pipe/wf_pipe_run.py`+- Worker `workflow_failure_exception_types` (now `[WorkflowExecutionError, PipelexError]`; was `[WorkflowExecutionError]` pre-fix): `pipelex/temporal/temporal_task_manager.py`+- Submitter wait + 1h `execution_timeout` applied (`WorkflowExecutor.execute_workflow`): `pipelex/temporal/tprl/workflow_caller.py`+- Timeout config source (`workflow_execution_timeout`, `run_timeout`, `rpc_timeout`): `pipelex/temporal/config_temporal.py` (`WorkerConfig`) + `pipelex/pipelex.toml` temporal section (~line 489)+- Activity error conversion (works for activity-dispatched leaves only): `pipelex/temporal/tprl/activity_error_boundary.py`+- Submitter-side report recovery (terminal-failure path only): `pipelex/temporal/tprl/temporal_error.py` (`recover_error_report`)+- Existing parity tests (terminal path; escape path untested): `tests/integration/pipelex/temporal/test_workflow_error_report_full_chain.py`, `tests/integration/pipelex/error_handling/test_error_report_local_full_chain.py`+- Design docs this critiques/extends: `pipelex/wip/error-handling/track-temporal-integration.md`, `.../track-retry-and-resilience.md`+- The concrete first instance: [`search-operator-missing-temporal-activity.md`](./search-operator-missing-temporal-activity.md)diff --git a/wip/error-handling/temporal-failsafe-guard-hardening.md b/wip/error-handling/temporal-failsafe-guard-hardening.mdnew file mode 100644index 000000000..ee689ca10--- /dev/null+++ b/wip/error-handling/temporal-failsafe-guard-hardening.md@@ -0,0 +1,131 @@+# Temporal fail-safe — harden & consolidate the inline-error guard++> **Cold start:** this doc is resolved rationale, read-only (findings 1/6/7/9, all handled). The live open backlog is in [`temporal-failsafe-review-followups.md`](./temporal-failsafe-review-followups.md); the directory index is [`README.md`](./README.md). Note: Finding 1's "resolved — kept the broad predicate" conclusion is *reopened* by Finding 13 in the followups doc (a third fragility mode the analysis here missed).++**Status:** ✅ **RESOLVED** on `fix/Temporal-failsafe`. All four findings handled; the chain-walk and fail-safe suites are green. Summary of decisions (the original analysis is preserved below for the record):++- **Finding 7 (done).** Extracted one cycle-guarded `iter_cause_chain(exc)` primitive in `base_exceptions.py`; the five hand-rolled `__cause__` walks (`_carries_temporal_failure`, `find_inference_error_category_in_chain`, `_find_error_report_dict`, `_message_from_exc`, and the cycle check in `_enrich_error_report_from_cause` — the doc missed this fifth one) now delegate to it.+- **Finding 9 (done).** Added `test_router_already_terminal_error_propagates_untouched_preserving_leaf_classification` for the guard's `True` branch. Proven meaningful: forcing the guard to `False` flips the submitter's `error_type` from the leaf `LLMCompletionError` to the generic `WorkflowExecutionError`, which the new test catches — while the convert-branch sibling stays green, which is exactly why this branch needed its own coverage.+- **Finding 1 (kept, *not* rewritten).** Verification upgraded the doc's own risk note: the guard's `True` branch is **reachable**, not theoretical — a controller sub-pipe failing as a child workflow is wrapped by `TemporalPipeRouter` as `WorkflowExecutionError(msg)` (no `error_report`) and escapes the parent's `pipe.run_pipe`. Its rich leaf report lives only in the child's `ApplicationError.details`, recoverable solely by `recover_error_report` at the submitter; any worker-side `from_message_exception` flattens it. **Both proposed rewrites regress this** — direction (a) (direct-cause check) is equal-or-worse (a deeper `FailureError` in a legitimate nested wrapper would now flatten), and direction (b) (always-convert) flattens the reachable case outright. So the broad "carries a Temporal `FailureError`" predicate is kept — it *is* the definition of "already terminal", not a proxy for a narrower type — refactored onto the primitive, with its docstring now stating the reachable case and the `raise … from` invariant (the over-inclusive "recover-then-rechain" failure requires an anti-pattern no operator does). The Finding 9 test pins it.+- **Finding 6 (light touch).** The two catch-alls' control flow is irreducibly different (router raises immediately; parent defers so `act_deliver` still fires the FAILED webhook), so they were **not** folded into one helper — that would force a dead, unreachable guarded branch into `WfPipeRun`. Instead the duplicated *rationale prose* (Finding 11) was deduped: the full argument lives once in `_carries_temporal_failure`'s docstring (code contract) + `docs/under-the-hood/error-model.md` "Workflow-Level Fail-Safe Floor" (design narrative); the router / parent / worker-registration comments carry a short statement plus a pointer.++**Left open (followups doc):** Finding 4 (parent lacks the guard) stays open but is assessed harmless — `WfPipeRun`'s `except PipelexError` can only catch errors from the pure `build_search_attributes` / `build_static_summary` argument evaluation, which cannot carry a `FailureError`, so a guarded branch there would be dead code. Findings 2, 3, 5, 8, 10, 12 are untouched (out of this doc's scope).++---++**Original review follow-up (for the record).** These were the highest-value findings from the code review of the fail-safe floor that landed on `fix/Temporal-failsafe`. None blocked the merge — the silent-hang hole *is* closed — but all four concern the single load-bearing mechanism the fix relies on, and they were best done together as one small refactor.++**Companion docs:** the fix itself is described in [`temporal-error-handling-failsafe-gap.md`](./temporal-error-handling-failsafe-gap.md) (the "why" and "what landed"); the landed behavior is summarized in [`track-temporal-integration.md`](./track-temporal-integration.md) under "Workflow-level fail-safe floor". The other, lower-priority review findings are in [`temporal-failsafe-review-followups.md`](./temporal-failsafe-review-followups.md).++**Finding numbers** (1, 6, 7, 9) are kept from the original review so the two follow-up docs cross-reference cleanly.++---++## Background (cold-start)++The fail-safe floor closes a hole where a pipelex **domain** error raised *inline in workflow code* — never dispatched through an activity — was neither an `ActivityError` nor an `ApplicationError`, so Temporal treated it as a *workflow-task* failure and retried it indefinitely: a silent, resource-burning hang surfaced only as a generic timeout after the execution timeout.++The fix has two layers:++- **Workflow-level catch-all.** `WfPipeRouter.run()` (`pipelex/temporal/tprl_pipe/wf_pipe_router.py`) and `WfPipeRun.run()` (`.../wf_pipe_run.py`) each end their boundary handling with an `except PipelexError` clause that converts a *genuine inline* error into a terminal, classified `TemporalError` (an `ApplicationError` carrying the structured `ErrorReport` in `details`).+- **Worker-level floor.** The worker registers `workflow_failure_exception_types=[WorkflowExecutionError, PipelexError]` (`pipelex/temporal/temporal_task_manager.py`), so any domain error that slips past the catch-alls still fails the workflow terminally instead of hanging.++The "genuine inline" decision is made by **`_carries_temporal_failure(exc)`** in `wf_pipe_router.py:26`. It walks the exception's `__cause__` chain and returns `True` if any node is a Temporal `FailureError`. `True` ⇒ the error already originated at a Temporal boundary (activity / child workflow), is already terminal, and its report is recoverable from the chain by `recover_error_report` at the submitter — so the catch-all leaves it untouched. `False` ⇒ a genuine inline error that never crossed a boundary — convert it via `TemporalError.from_message_exception`.++**Key fact established during review (why none of these is a live bug yet):** the real inline-leaf path, `pipelex/temporal/tprl_content_generation/content_generator_in_workflow.py`, dispatches activities and, on `ActivityError`, raises `TemporalError.from_app_error(exc=exc.cause) from exc` — a `FailureError`, wired with `from`. So `_carries_temporal_failure` classifies it correctly. Genuine inline errors (the Search-operator-missing-activity case, crate-load, hydration setup) carry no `FailureError` and are converted correctly.++> **Correction (this session).** The original review claimed "no confirmed live trigger" for the guard's `True` branch. That was wrong, and it reshaped Finding 1: the `True` branch **is** reachable — a controller sub-pipe failing as a child workflow is wrapped by `TemporalPipeRouter` (`temporal_pipe_router.py:92-94`) as `WorkflowExecutionError(msg)` and escapes the parent's `pipe.run_pipe`. So the guard is load-bearing on a real path, not just a latent safeguard. What *remains* latent are only the two **misclassification** fragilities below (under-/over-inclusive), each of which requires a `raise … from` discipline violation that no operator currently makes.++---++## ✅ Finding 1 — `_carries_temporal_failure` uses chain-membership as a proxy for "already terminal", which is fragile both ways++> **Resolved — kept the guard, not rewritten.** Both proposed directions regress the *reachable* nested-failure case (worker-side conversion flattens a report only the submitter can recover). The broad "carries a Temporal `FailureError`" predicate is correct — it *is* the definition of "already terminal" — and now sits on `iter_cause_chain` with a docstring stating the reachable case + the `raise … from` invariant. Pinned by the Finding 9 test. (Detail in the top summary.)++**Where:** `pipelex/temporal/tprl_pipe/wf_pipe_router.py:26` (the helper), `:195` (the guard call), `:197` (the conversion it gates).++**What:** the rule "any `FailureError` anywhere in the `__cause__` chain ⇒ propagate untouched" is a *proxy* for the real question, which is "is **this** error already a terminal, report-carrying Temporal failure?". The proxy is both over- and under-inclusive:++- **Under-inclusive (flattening).** A Temporal-failure-carrying error wrapped **without** `from` (implicit `__context__` only, not `__cause__`) reads as "no failure in chain" → the guard returns `False` → the already-classified error is re-converted by `from_message_exception`, flattening its rich leaf report (`model` / `provider` / `error_category`) to the wrapper's generic classification. Pipelex's coding standard is `raise … from exc`, so this requires a *missing* `from` somewhere on a workflow-inline path — the guard is silently fragile to that mistake.+- **Over-inclusive (misattribution).** A genuinely fresh inline error that is chained `from` an unrelated, *already-recovered* side failure (pipe code dispatches a side activity, it fails as a `TemporalError`, the code catches and recovers, then later raises a different fresh domain error `from` the recovered one) carries a `FailureError` in its chain → the guard returns `True` → it propagates untouched and carries **no report of its own**. At the submitter, `recover_error_report` walks the chain and surfaces the **side** failure's `error_type` / `error_category` / `model` — i.e. the run is reported as the wrong failure.++**Why it matters:** this guard is the hinge of the whole fix. The failure modes are latent today (see the key fact above), but a single missing `from`, or one operator that recovers-then-rechains, silently degrades the exact classification fidelity the error-handling project exists to deliver — and there is no test on the `True` branch to catch a regression (see Finding 9).++**Proposed direction:** replace the membership heuristic with a decision about *this* error's own terminality/recoverability. Options to weigh:++- Convert only when the error is **not itself** a `FailureError` *and* does not **directly** wrap one as its immediate `__cause__` (rather than anywhere in the chain) — tightens the over-inclusive case.+- Or: always convert (mint a fresh `TemporalError` from `exc`), but make `from_message_exception` *prefer an already-present report in the chain* over re-deriving — so flattening becomes impossible regardless of `from`/`__cause__` shape. This removes the guard's load-bearing role entirely and may be the deeper fix.++**Open questions:**++1. Do we tighten the predicate (direct-cause check) or eliminate it by making the conversion report-preserving (prefer the chain's existing report)?+2. Is "recover-then-rechain" a pattern we ever want in pipe/operator code, or should we forbid it (and lean on `from`-discipline + a lint)? That answer decides whether the over-inclusive case is worth engineering against at all.++---++## ✅ Finding 7 — `_carries_temporal_failure` is the 4th hand-rolled `__cause__`-chain walk++> **Resolved — done.** Extracted `iter_cause_chain(exc)` in `base_exceptions.py`; all five `__cause__` walks (the four named here + the cycle check in `_enrich_error_report_from_cause`) plus two unguarded strays found while sweeping (`agent_output._build_error_source`, `pipe_llm._format_llm_error` — both previously missing the cycle guard) now delegate to it.++**Where:** `pipelex/temporal/tprl_pipe/wf_pipe_router.py:26`. Siblings that re-implement the same walk: `find_inference_error_category_in_chain` (`pipelex/cogt/exceptions.py`, ~`:129`), `_find_error_report_dict` (`pipelex/temporal/tprl/temporal_error.py:41`), `_message_from_exc` (`.../temporal_error.py:68`).++**What:** all four are the identical loop — `node = exc; seen: set[int] = set(); while node is not None and id(node) not in seen: <predicate>; seen.add(id(node)); node = node.__cause__` — differing only in the per-node predicate. Each re-rolls the cycle guard (the `id()` set that prevents an infinite spin on a cyclic `__cause__` chain) by hand.++**Why it matters:** on the error-reporting path a cycle-guard bug would itself become a hang — the very failure the fail-safe exists to prevent. Four hand-copies mean a fifth will get it subtly wrong. This is pure reuse debt, but it sits on the most safety-critical path.++**Proposed fix:** extract one primitive — e.g. `find_first_in_cause_chain(exc, predicate) -> BaseException | None` (and/or `any_in_cause_chain`) — in `pipelex/base_exceptions.py` (or a small `pipelex/tools` exception util) with the cycle guard written once. Then:++- `_carries_temporal_failure(exc)` collapses to `any_in_cause_chain(exc, lambda n: isinstance(n, FailureError))`.+- `_find_error_report_dict`, `_message_from_exc`, `find_inference_error_category_in_chain` all delegate to it.++**Open question:** where does the primitive live so both `cogt/` and `temporal/` can import it without a layering inversion or circular import? `base_exceptions.py` is the natural home (it already defines `PipelexError` and a cycle-guarded `_enrich_error_report_from_cause` walk) — confirm that placement.++---++## ✅ Finding 6 — Two near-identical catch-alls were copy-pasted into two workflows; the worker-floor registration is the actual hole-closer (altitude)++> **Resolved — light touch.** Catch-alls kept separate (their control flow is irreducibly different: router raises immediately, parent defers so `act_deliver` fires the FAILED webhook). The duplicated *rationale prose* was deduped onto `_carries_temporal_failure`'s docstring (code contract) + `error-model.md` (design); the router / parent / worker-registration comments now carry a statement + pointer. No shared helper — that would have forced a dead guarded branch into `WfPipeRun` (followups Finding 4).++**Where:** the `except PipelexError` clause in `wf_pipe_router.py:174` and in `wf_pipe_run.py:83`; the registration in `temporal_task_manager.py:159`.++**What:** the single line `workflow_failure_exception_types=[…, PipelexError]` already closes the hang for **every** workflow, present and future, with zero per-workflow code — it guarantees an escaping `PipelexError` fails terminally instead of hanging. The two hand-written catch-alls only *upgrade the report* from a synthesized `UnrecoverableWorkflowFailureError` to the original classification. That richness is real and worth having, but it means:++- the actual hole-closer is one config line, while the design doc / CHANGELOG / three multi-paragraph inline comment blocks present the catch-alls as the headline fix;+- the "convert genuine inline error / leave already-terminal alone" decision is split across two ~20-line blocks that are implemented **divergently** — the router guards with `_carries_temporal_failure` and raises immediately; the parent omits the guard and defers via the delivery path (see Finding 4 in the followups doc);+- every future workflow author must remember to copy the clause to get the rich report.++**Why it matters:** maintenance weight and drift. A change to the conversion policy must be applied in N places, and the two existing copies already disagree.++**Proposed fix (right altitude):** hoist the policy into one shared helper, e.g. `convert_inline_pipelex_error(exc) -> TemporalError` that owns the `_carries_temporal_failure` short-circuit (or its Finding-1 replacement) and the `from_message_exception` conversion. Both workflows call it; each chooses only *immediate-raise* (router) vs *deferred-delivery* (parent). The policy then lives once, the guard becomes non-optional, and Finding 4's asymmetry disappears for free.++**Open questions:**++1. Is the per-workflow rich-report richness worth keeping at all, or do we accept the worker floor's synthesized `UnrecoverableWorkflowFailureError` and delete the catch-alls? (The floor already prevents the hang; the question is purely "rich classification vs less code".) Recommendation: keep the richness but centralize it.+2. If we keep it, should the shared helper be a plain function both sites call, or a workflow-root decorator/util applied uniformly to all `@workflow.defn` run methods so new workflows get it automatically?++---++## ✅ Finding 9 — The `_carries_temporal_failure` `True` branch is untested++> **Resolved — done.** Added `test_router_already_terminal_error_propagates_untouched_preserving_leaf_classification`. Proven meaningful: forcing the guard to `False` flips the submitter's `error_type` from the leaf `LLMCompletionError` to the generic `WorkflowExecutionError`, which the new test catches while the convert-branch sibling stays green.++**Where:** `tests/integration/pipelex/temporal/test_workflow_inline_error_failsafe.py`.++**What:** all three new tests inject errors with **no** `FailureError` in the chain (`make_failing_llm_error()` builds a clean `LLMCompletionError`; `WorkflowInputError` is raw). So they exercise only the `False` branch (convert). The `True` branch — "the error already carries a Temporal failure, do **not** re-wrap" — is the trickier, more error-prone arm and has **zero** coverage. Negate or delete the guard and the suite stays green.++**Why it matters:** the guard from Finding 1 is the fix's hinge, and its hardest branch is the one with no regression net. Combined with the retryable-inline gap (Finding 2 in the followups doc), the test suite currently pins the easy paths and leaves the subtle ones open.++**Proposed fix:** add a test that raises, inline in `WfPipeRouter`, a `PipelexError` whose `__cause__` chain already contains a report-carrying `TemporalError`/`ApplicationError` (simulating an already-terminal escapee), and assert it propagates **untouched** — i.e. the submitter recovers the *original* leaf classification, not a re-wrapped generic one. If Finding 1 is reworked to be report-preserving, this test should assert the preserved leaf report either way.++**Open question:** the new tests run **unsandboxed** (`is_not_sandboxed=True`), like the rest of the temporal suite. The catch-all now calls `from_message_exception` (→ `get_config()`, `find_inference_error_category_in_chain`, `ErrorReport.to_dict`, `workflow_log`) inside workflow code under the production `SandboxedWorkflowRunner` — a surface no test covers under the sandbox. Do we add one sandboxed smoke test for the conversion path, or is the existing unsandboxed coverage + the passthrough-module config (`temporal_task_manager.py` `with_passthrough_modules(...)`) sufficient assurance? (This is a pre-existing gap for the whole suite, not introduced here, but the floor is now one more untested-under-sandbox surface.)++---++## What was done (final sequencing)++All four landed as one unit of work:++1. **Finding 7** — extracted `iter_cause_chain` (a generator, not `find_first_in_cause_chain`: it also subsumes the accumulator walk `_message_from_exc` and the is-self cycle check, which a first-match primitive could not). Migrated all five named walks plus two unguarded strays.+2. **Finding 1** — *kept* the guard rather than rewriting it: verification showed the `True` branch is reachable and both proposed rewrites regress it. Refactored the predicate onto the primitive and documented the invariant.+3. **Finding 6** — light touch: deduped the rationale prose, kept the two catch-alls separate (no shared helper — see followups Finding 4's resolution).+4. **Finding 9** — added the `True`-branch test and proved it meaningful via a deliberate guard-regression check. The sandboxed-smoke open question (above) was left as the pre-existing suite-wide gap it is.diff --git a/wip/error-handling/temporal-failsafe-review-followups.md b/wip/error-handling/temporal-failsafe-review-followups.mdnew file mode 100644index 000000000..1b0e01781--- /dev/null+++ b/wip/error-handling/temporal-failsafe-review-followups.md@@ -0,0 +1,155 @@+# Temporal fail-safe — secondary review follow-ups++> **Gated on Temporal productionization — parked, not abandoned.** The silent-hang hole this branch closed is *done and shipped*; everything below is latent because Temporal is not in production (Findings 13/14 have no trigger until workflows actually run; C/D/F are pre-prod hardening; 3/12 are decisions, not bugs). **Do not execute piecemeal now** — run it as one focused pass when Temporal is scheduled to ship, with the system exercisable so the fixes are verifiable rather than speculative. The only "now" exception: fold the opportunistic one-liners (5, 8, 10) in *if* their file is touched for another reason.++**Status:** 🔎 open backlog. Lower-priority findings from the code review of the fail-safe floor that landed on `fix/Temporal-failsafe`. None blocks the merge. **Resolved in the guard-hardening session and removed from this doc:** Finding 4 (catch-all asymmetry — won't fix, harmless: a guarded branch in `WfPipeRun` would be dead code) and Finding 11 (rationale duplication — deduped onto `_carries_temporal_failure`'s docstring + `error-model.md`). Finding 12's comment overstatement was fixed there too; only its product question survives below. **Resolved in the review-agent triage session** (independently re-flagged by codex + cubic): Finding 2 — the inline conversions now force `non_retryable=True` (deterministic + terminal); see its resolution banner below. What remains open: a contract decision (Finding 3), small cleanups (5, 8, 10), and two findings added by a later `/review` pass — **Finding 13** (the `_carries_temporal_failure` predicate floors inline errors chained from a report-less `FailureError` — deepens the Finding 1 fragility analysis) and **Finding 14** (a pre-existing replay-determinism hazard: `build_search_attributes` reads config on the recorded child-start command).++**Companion docs:** the fix is described in [`temporal-error-handling-failsafe-gap.md`](./temporal-error-handling-failsafe-gap.md); the landed behavior in [`track-temporal-integration.md`](./track-temporal-integration.md). The four highest-value findings (the guard's correctness, consolidation, reuse, and missing test) are in [`temporal-failsafe-guard-hardening.md`](./temporal-failsafe-guard-hardening.md) — **read its Background section first**; this doc assumes that context.++**Finding numbers** (2, 3, 5, 8, 10, 12) are kept from the original review. Findings 4 and 11 were resolved and removed (see Status). **Findings 13 and 14** were added by a later `/review` pass (adversarial subagent + verification); they continue this doc's numeric scheme. (The gap doc owns the `A`–`F` letters for its fix-plan items, so there is deliberately no letter/number overlap across the two docs.)++---++## Background (cold-start, condensed)++The fail-safe floor makes a pipelex **domain** error raised *inline in workflow code* (never via an activity) fail the workflow **terminally and classified** instead of hanging on indefinite workflow-task retry. Two layers:++- `WfPipeRouter.run()` and `WfPipeRun.run()` each end with an `except PipelexError` clause. The router converts a genuine inline error to a terminal `TemporalError` via `from_message_exception` and raises immediately. The parent (`WfPipeRun`) routes it through its deferred-delivery path so the FAILED webhook still fires, then re-raises terminally.+- The worker registers `workflow_failure_exception_types=[WorkflowExecutionError, PipelexError]` as a backstop.++`from_message_exception` (`pipelex/temporal/tprl/temporal_error.py:232`) builds the terminal `TemporalError`: it sets `error_type = exc.__class__.__name__`, packs `exc.to_error_report().to_dict()` into `ApplicationError.details`, and derives `non_retryable` from the inference error category (or the `non_retryable_error_types` name-list fallback). The full background and the `_carries_temporal_failure` guard are explained in the companion hardening doc.++---++## Finding 2 — Retryable inline errors still re-run the inline leaf on retry (behavioral tradeoff + test gap)++> **✅ Resolved on `fix/Temporal-failsafe`** (review-agent triage; independently re-flagged by codex + cubic). The inline catch-alls now pass `force_non_retryable=True` to `from_message_exception` (the `wf_pipe_router.py` line-195 conversion and the `wf_pipe_run.py` inline carrier). Open question #1 is answered **"force `non_retryable=True`"**: an inline domain error fails the workflow terminally instead of re-running already-completed inline work on a blunt whole-workflow retry — retry belongs at the activity boundary, where it is scoped and observable. Forcing the flag also short-circuits the config-reading `_is_non_retryable`, so the workflow-side conversion is now **config-free / deterministic** — which also resolves the separate review concern that `from_message_exception` read `get_config()` inside workflow code (a replay-fragility risk). The report's own `retryable` field is config-free and preserved as informational metadata. Open question #3 (test the arm) is closed: `test_temporal_error_bridge.py::TestTemporalErrorBridge::test_force_non_retryable_is_deterministic_and_preserves_report_retryable` pins `non_retryable=True`, the preserved `report.retryable`, and that `_is_non_retryable` is not called when forced. Original analysis kept below for the record.++**Where:** `wf_pipe_router.py:195` (the conversion) → `from_message_exception` retryability logic in `temporal_error.py`.++**What:** `from_message_exception` sets `non_retryable=False` for a transient/uncategorized-not-listed inline domain error. A *retryable* `ApplicationError` raised in workflow code is then subject to the workflow's `RetryPolicy` — configured for top-level workflows via `WorkflowExecutorFactory`, and the child `WfPipeRouter` is dispatched **without** an explicit retry policy (deliberately, for replay determinism — see `wf_pipe_run.py:50-57`), so it inherits Temporal's default. The consequence: a retryable inline error re-runs `pipe.run_pipe` — **including any real provider/inference call it makes inline** — on each attempt, up to the execution timeout.++**Why it matters:** this is strictly *better* than the pre-fix outcome (an unbounded workflow-task-retry hang that also re-ran the inline work), so it is not a regression. But the fix does **not** make retryable inline errors fail fast — it trades an unbounded hang for a bounded-but-still-resource-burning execution retry. And the tests only exercise the **non-retryable** arm (`make_failing_llm_error()` is a non-retryable `CONFIGURATION`-class error), so the retryable path's behavior is unverified.++**Open questions:**++1. Is execution-level retry of an inline leaf acceptable, or should the inline catch-all force `non_retryable=True` regardless of category? Rationale for forcing: if an operator is running its leaf *inline* (rather than via an activity with its own `RetryPolicy`), retrying the whole workflow execution is a blunt, expensive instrument — the operator was supposed to dispatch an activity precisely so that retry is scoped and observable. Rationale against: a genuinely transient inline error (rare) would then fail instead of self-healing.+2. This overlaps with deferred follow-up **C** (a short submitter-side deadline for the synchronous API path) and **D** (CI guard that every inference operator routes its leaf through an activity) in [`temporal-error-handling-failsafe-gap.md`](./temporal-error-handling-failsafe-gap.md). If **D** lands, "retryable inline error" becomes a should-never-happen state and this question is moot. Decide whether to engineer #1 now or defer behind **D**.+3. Add a test for the retryable-inline arm regardless of the above (it is currently a blind spot).++---++## Finding 3 — `SecurityError` is now routed through `except PipelexError` and the worker floor, contradicting its own contract++**Where:** the catch-alls (`wf_pipe_router.py:184`, `wf_pipe_run.py:83`) and the registration (`temporal_task_manager.py:147`). The contract: `pipelex/base_exceptions.py:567`.++**What:** `SecurityError`'s docstring states it is "kept distinct from domain errors so security signals are not silently swallowed by domain-level `except` handlers (e.g. `except PipelexError`)." `SecurityError` is a `PipelexError`, so the new `except PipelexError` clauses catch it and the `PipelexError` registration floors it — exactly the pattern the docstring warns against.++**Nuance:** the new code does **not** *silence* it — it re-raises it terminally with the type preserved in the report (`error_type` survives via `from_message_exception`). And making a security violation a terminal failure (rather than an indefinite retry) is arguably *correct*. So this may be a benign — even desirable — interaction. But it is the literal anti-pattern the contract calls out, and a maintainer relying on that contract would be surprised.++**Open questions:**++1. Should the catch-alls add an explicit `except SecurityError: raise` (or `except SecurityError` that converts but flags/audits differently) **before** the `except PipelexError`, to honor the contract literally?+2. Or do we update the `SecurityError` docstring/contract to acknowledge that the Temporal workflow boundary legitimately floors it to a terminal failure (because hanging on a security violation is worse)? If so, is there any place a `SecurityError` *should* get differentiated handling at the boundary (e.g. suppressed from the webhook payload, or routed to an audit sink)?+3. Are there other `PipelexError` subclasses with special "do not swallow" intent that the broad registration now captures? (Audit the hierarchy under `base_exceptions.py` for similar contracts.)++---++## Finding 5 — The inline path drops the original exception from the worker-side cause chain++**Where:** `wf_pipe_run.py:99` — `execution_error.__cause__ = TemporalError.from_message_exception(exc=exc)`.++**What:** the minted `TemporalError` itself has `__cause__ = None` (`from_message_exception` returns `cls(...)` and never chains `from exc`). So after `execution_error.__cause__` is set to it, the **original** inline `exc` and its traceback are no longer reachable from `execution_error`. This is asymmetric with:++- the sibling child-failure path (`wf_pipe_run.py:81`: `execution_error.__cause__ = exc`, preserving the `ChildWorkflowError`), and+- the router catch-all (`wf_pipe_router.py:195`: `raise TemporalError.from_message_exception(exc=exc) from exc`, preserving `exc`).++**Why it matters:** low. The classification and message survive (they ride in the `TemporalError.details` report and are logged via `workflow_log.error(f"WfPipeRun inline failure: {exc}")`), and the submitter recovers the report fine. The loss is the original exception object / traceback / any deeper `__cause__` on the worker side — a debuggability and consistency gap, not a behavioral break.++**Open question:** preserve the original by chaining it — set the minted `TemporalError`'s `__cause__` to `exc` before assigning it to `execution_error`, giving `execution_error → TemporalError → exc`. Confirm this does not confuse `recover_error_report`'s walk (it stops at the first report-carrying `ApplicationError`, so a deeper `exc` is harmless). Standalone fix now — the Finding-6 shared helper that would have owned the chaining was not built (the catch-alls stayed separate; see the hardening doc's light-touch resolution).++---++## Finding 8 — `exc.to_error_report()` is computed twice on the inline path (efficiency + drift risk)++**Where:** `wf_pipe_run.py:92` (`error_report = exc.to_error_report()`, for the webhook) and `:99` (`from_message_exception`, which internally recomputes `exc.to_error_report().to_dict()` at `temporal_error.py:243`).++**What:** `to_error_report()` is not trivial — it walks and enriches from the `__cause__` chain (`base_exceptions.py:482` + `_enrich_error_report_from_cause`). On every inline failure it is paid twice. Worse, the webhook gets the object from `:92` while the submitter gets the dict packed at `:99`; if the two derivations ever diverge, the webhook and submitter disagree on classification.++**Why it matters:** small wasted work on the failure path; the drift risk is the more interesting part (single-source-of-truth for the report).++**Proposed fix:** compute the report once and have `from_message_exception` accept a precomputed `ErrorReport` (or expose a variant taking the dict), so the webhook payload and the wire details come from one source.++**Open question:** does `from_message_exception`'s signature change ripple to the activity-side caller (`activity_error_boundary.py:52`)? Prefer an additive optional `error_report=` parameter that defaults to recomputing, so the activity path is unaffected.++---++## Finding 10 — `workflow_failure_exception_types=[WorkflowExecutionError, PipelexError]` is redundant++**Where:** `temporal_task_manager.py:147`.++**What:** `WorkflowExecutionError` is a subclass of `PipelexError` (via `TemporalFlowError`), so the second entry fully subsumes the first. The list is redundant. The previously-misleading comment was already trimmed in the guard-hardening session — it now says `WorkflowExecutionError` is "listed explicitly for intent" rather than claiming it is separately load-bearing — so what remains is purely the cosmetic list decision below.++**Why it matters:** harmless at runtime; a maintenance trap. A future reader who trusts the comment might narrow the broad `PipelexError` entry believing `WorkflowExecutionError` still covers the child-wrap case — when in fact removing `PipelexError` silently reopens the floor for every non-`WorkflowExecutionError` domain error, and removing `WorkflowExecutionError` changes nothing.++**Open question:** drop to `[PipelexError]` and simplify the comment, **or** keep `WorkflowExecutionError` explicit purely for documentation (signaling the originally-intended terminal type) with a comment that says so? Either is defensible; pick one and make the comment match reality.++---++## Finding 12 — Should a terminal failure with no delivery target still notify?++**Where:** `wf_pipe_run.py:89` (the webhook comment) and `:146` (`if delivery_assignment is not None:`).++**Comment half — done.** The guard-hardening trim dropped the "a terminal failure must always notify the receiver" overstatement; the comment now reads "fires the FAILED webhook on the failure path," which is accurate. Only the product question below remains.++**What:** when `delivery_assignment is None`, Step 3 is skipped (`wf_pipe_run.py:146`) and no webhook fires. This matches direct-mode `PipeRun` semantics (no target ⇒ no delivery), so the current behavior is intentional. The only test always supplies a webhook target, so the no-assignment branch is unexercised.++**Open question:** is "no target ⇒ no notification" the right product contract, or should a terminal failure always notify *somewhere* even without a `delivery_assignment` (cf. the memory note that a run's terminal failure webhook must always fire)? If the latter, the gap is in the code, not the comment. Current lean: keep as-is (parity with direct mode); revisit only if a "must-always-notify" product invariant is adopted.++---++## Finding 13 — `_carries_temporal_failure` floors inline errors chained from a non-`ApplicationError` `FailureError` (deepens Finding 1)++> **Provenance:** surfaced by a `/review` adversarial pass and verified against the live Temporal SDK hierarchy. Latent today, not a merge blocker. Extends [`temporal-failsafe-guard-hardening.md`](./temporal-failsafe-guard-hardening.md) Finding 1 with a third fragility mode its analysis did not consider.++**Where:** `wf_pipe_router.py:56` (`_carries_temporal_failure`), gating the conversion at `:198`.++**What:** the predicate is `any(isinstance(node, FailureError) for node in iter_cause_chain(exc))`. Verified: `TimeoutError`, `CancelledError`, `TerminatedError`, `ActivityError`, and `ChildWorkflowError` are **all** `FailureError` subclasses but **not** `ApplicationError` — and only `ApplicationError` carries the `details` payload where the structured `ErrorReport` rides. So an inline `PipelexError` chained `from` a bare Temporal failure that carries no recoverable report — e.g. pipe code that awaits an activity, the activity times out, and the code does `raise SomeDomainError(...) from timeout_exc` — returns `True` and is **propagated untouched**. At the submitter, `recover_error_report` walks `__cause__` for a report-bearing `ApplicationError`, finds none (a `TimeoutError`/`CancelledError` has no `details`), and synthesizes an `UnrecoverableWorkflowFailureError` — **dropping the domain error's own `error_type` / message** that `from_message_exception` (the convert branch) would have preserved. The guard's docstring assumes the chain's `FailureError` is a report-carrier ("the rich leaf classification lives deeper, in the child's `ApplicationError.details`"); that assumption is false for non-`ApplicationError` `FailureError`s.++**Why it matters:** this is the exact classification-fidelity loss the error-handling project exists to prevent — surfacing neither the timeout nor the domain error, but a generic synthesized report. **Latent today:** the one real inline-leaf path (`content_generator_in_workflow.py`) raises `TemporalError.from_app_error(...)` — an `ApplicationError`-with-report — so it is classified correctly, and genuine inline errors (Search-missing-activity, crate-load, hydration) carry no `FailureError` and convert correctly. The gap opens the moment any operator catches a Temporal timeout/cancel and re-raises a *plain* `PipelexError` `from` it inline.++**Why the obvious fix regresses (and what to do instead):** tightening the predicate to "carries a recoverable report" (`isinstance(node, ApplicationError) and error_report_dict_from_details(node.details) is not None`) **breaks the reachable case** Finding 1 protects: the controller-sub-pipe-as-child-workflow failure escapes as `WorkflowExecutionError` → `ChildWorkflowError`, whose leaf report lives in `ChildWorkflowError.cause` (recovered only by the submitter's `.cause`-normalizing walk), **not** in `__cause__` — so a worker-side `__cause__`-only report check would not find it and would wrongly convert. This is third confirmation that the broad-vs-narrow predicate is fundamentally fragile, and it strengthens the case for **Finding 1's deferred option (b)**: stop using the guard as the load-bearing decision — instead always mint a `TemporalError` but have `from_message_exception` *prefer an already-present report in the chain* over re-deriving, so flattening is impossible regardless of the `from`/`__cause__`/`.cause` shape.++**Open questions:**++1. Adopt Finding 1 option (b) (report-preserving conversion, guard demoted)? That subsumes both this finding and Finding 1's original two modes.+2. Until then, is it worth a narrow guard tweak that treats a chain whose only `FailureError`s are report-less (`TimeoutError`/`CancelledError`/`TerminatedError`) as convert-eligible, while still propagating `ChildWorkflowError`/report-bearing `ApplicationError`? (More surface, partial fix — option (b) is cleaner.)+3. Add a `True`-branch test variant where the chain carries a report-less `FailureError`, asserting the domain error's own classification survives (it currently does **not**).++---++## Finding 14 — `build_search_attributes` reads `get_config()` on the recorded child-start command (replay-determinism hazard, pre-existing)++> **Provenance:** surfaced by a `/review` adversarial pass; verified that `build_search_attributes` reads config. **Pre-existing — not introduced by this branch** (`pipelex/temporal/tprl/observability.py` is untouched here). Filed here per the repo's "flag pre-existing bugs" principle because it directly undercuts the fail-safe's anti-hang purpose. Better tracked in a dedicated Temporal wip doc if one is opened.++**Where:** `observability.py:104` (`config = get_config().temporal.search_attributes`) feeding `wf_pipe_run.py:63` and `temporal_pipe_router.py` (`search_attributes=build_search_attributes(pipe_job)` as an argument to `execute_child_workflow`).++**What:** the `search_attributes` value is part of the recorded `StartChildWorkflowExecution` command. Because `build_search_attributes` derives it from `get_config().temporal.search_attributes` (`.enabled`, `.attributes`) — process/deploy state, not workflow input — editing that config while a workflow is live would make a later replay re-derive a *different* attribute set on the child-start command → a Temporal non-determinism mismatch. This contradicts the same file's load-bearing comment (`wf_pipe_run.py:50-57`), which deliberately keeps the child-dispatch command "a pure function of the workflow input" precisely so "replay after a config edit" cannot diverge — the comment enumerates `execution_timeout` / `retry_policy` / `task_queue` but overlooks that `search_attributes` is on the same command and is config-derived. The function's docstring calls itself pure because it "reads everything off `pipe_job`," which is inaccurate.++**Why it matters:** a config-edit-during-live-workflow → stuck/failed replay is the exact silent-hang class the fail-safe exists to eliminate, sitting unguarded on the dispatch path. **Latent:** narrow trigger (a long-running workflow surviving a worker redeploy with edited search-attribute config) and Temporal is not yet in production.++**Open questions:**++1. Stamp the resolved search attributes onto `pipe_job`/`PipeRunArg` at submit time (submitter-side, before the durable boundary) so the workflow reads them off its input instead of `get_config()` — matching how the file already handles `execution_timeout` / `retry_policy` / `task_queue`.+2. Or confirm (with a versioning/patch strategy) that search-attribute config is frozen for a workflow's lifetime, and correct the `build_search_attributes` docstring's "pure" claim either way.+3. Does `build_static_summary` (also on the command) have any config dependency? (Spot-checked: no `get_config()` read — appears input-only, but confirm.)++---++## Suggested handling++- **Finding 2** is resolved (see its banner — inline conversions forced non-retryable + deterministic). **Finding 3** remains a genuine decision — surface it before the next Temporal release; it may be answered by the deferred **C/D** follow-ups in the gap doc.+- **Findings 5 and 8** are cheap standalone cleanups (the Finding-6 shared helper that would have absorbed them was not built — see the hardening doc's light-touch resolution). Best folded in opportunistically the next time `wf_pipe_run.py` is touched; **5** is the higher-value of the two.+- **Finding 10** is a one-line cleanup (drop the redundant entry, or keep it for intent) — batch it whenever `temporal_task_manager.py` is next edited. **Finding 12**'s comment is already fixed; only its product question lingers.+- **Finding 13** is the higher-value of the two new ones: it converts Finding 1 from "resolved — kept the broad predicate" to "the broad predicate has a third, unhandled failure mode," and points at Finding 1 option (b) (report-preserving conversion) as the resolution that subsumes all of it. Decide 13 and Finding 1 together, before the next Temporal release. Latent today, so not urgent — but it is a correctness gap, not a cosmetic one.+- **Finding 14** is pre-existing and out of this branch's scope; fold it into a dedicated Temporal wip doc (or fix it at submit time per its open question 1) when `observability.py` / the child-dispatch path is next touched.diff --git a/wip/error-handling/track-temporal-integration.md b/wip/error-handling/track-temporal-integration.mdindex bba98c83e..f5dc2ad97 100644--- a/wip/error-handling/track-temporal-integration.md+++ b/wip/error-handling/track-temporal-integration.md@@ -54,6 +54,15 @@ Net effect: a pipe failure on a Temporal worker now reaches every `to_error_repoThe child-workflow boundary (`execute_child_workflow` / `start_child_workflow` in `pipelex/temporal/tprl/workflow_caller.py`) now applies the same recovery. `ChildWorkflowError` exposes the deserialized failure via `.cause`; when that cause is an `ApplicationError`, the handler runs `recover_error_report(exc.cause)` and, on a recovered report, raises `WorkflowExecutionError(error_report.message, error_report=...)` — mirroring `execute_workflow`. A cause with no report payload, malformed details, or a non-`ApplicationError` cause all fall back to the prior generic error. These wrappers are not on Pipelex's traced execution paths (in-workflow child spawns call `workflow.execute_child_workflow` directly to stay replay-deterministic), so this closes a latent gap on the public surface.+### Workflow-level fail-safe floor (landed)++The bridge above converts every error that crosses an *activity* boundary. The remaining gap was an error raised **inline in workflow code** that never went through an activity — neither an `ActivityError` nor an `ApplicationError`, so Temporal treated it as a *workflow-task* failure and retried it indefinitely (a silent hang surfaced only as a generic timeout after the execution timeout). The floor closes that:++- `WfPipeRouter.run()` and `WfPipeRun.run()` each end with an `except PipelexError` boundary. `WfPipeRouter` converts a genuine inline error to a terminal `TemporalError` via `from_message_exception`; `WfPipeRun` routes it through deferred delivery (FAILED webhook still fires) and chains a details-carrying `TemporalError` so the submitter stays classified. Both convert only *genuine* inline errors — an escaping `PipelexError` already carrying a Temporal failure in its `__cause__` chain (e.g. a nested `TemporalPipeRouter` child-dispatch wrapping a sub-pipe failure as `WorkflowExecutionError`) is already terminal and recoverable, so it propagates untouched (`_carries_temporal_failure` guard) rather than being flattened.+- The worker now registers `workflow_failure_exception_types=[WorkflowExecutionError, PipelexError]` (`pipelex/temporal/temporal_task_manager.py`). Anything that slips past the catch-alls still fails terminally (degrading to a synthesized `UnrecoverableWorkflowFailureError`) instead of hanging. Scoped to `PipelexError`, not `Exception`: transient Temporal/infra errors keep Temporal's default task-retry.++**Coverage:** `tests/integration/pipelex/temporal/test_workflow_inline_error_failsafe.py` pins the escape path across the router catch-all, the `WfPipeRun` inline-delivery path, and the worker-level floor, each bounded by a short execution timeout so a regression fails fast instead of hanging. The systemic analysis and deferred hardening (sync-API deadline, activity-dispatch CI guard, stuck-workflow observability) live in [`temporal-error-handling-failsafe-gap.md`](./temporal-error-handling-failsafe-gap.md).+### Deferred review followupsTwo optional refinements from the activity-boundary review (tracked in the deferred backlog) remain deliberate scope choices: the integration test verifies the converted payload rather than Temporal's retry-engine behavior, and `error_report_dict_from_details` identifies the report by dict shape. Optional to revisit.