Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feature/Step-problem-misses-carrier · L-261025-a3cd22 — step problem misses carrier code (#1487)

Louis Choquelbf31d71d82d2
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 0e83b06b8..0b879a6d3 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,6 +6,10 @@
- **The API server refuses a method's unmet requirements as an invalid method (Breaking)**: `/v1/execute` and `/v1/start` refuse a method whose `requirements.txt` the PipeFunc sandbox does not meet, or holds a line the standard refuses, with a `422` `method.invalid` whose `errors` hold one `method.dependency_missing` or `method.package_invalid` item per refused line, located at `requirements.txt` and the line, the problem a local run of the method raises and the items `/v1/validate` gives, where they answered a bare `method.dependency_missing` or `method.package_invalid` with no item and no location; a client reads the refused requirement from the item's `params` and `location`.
+### Fixed
+
+- **A failed step's trace records the failure's own code**: a failed node of the execution graph, and the `pipe_end_error` trace event, now carry the problem the run fails with when the step's error wraps a carrier of a problem recovered across a transport boundary, such as `pipe.function_raised` for a `PipeFunc` whose function raised in a sandbox, located at the step, where they carried the carrier's catch-all `runtime.unexpected`. The step's own message names that recovered code too, `Error: <carrier class> (pipe.function_raised)`, and `reported_failure_code` (`pipelex.pipe_run.located_failure`) reads the code a failure reports without building its problem.
+
## [v0.91.0] - 2026-10-25
### Added
diff --git a/docs/under-the-hood/error-model.md b/docs/under-the-hood/error-model.md
index 41179e946..42d369f9f 100644
--- a/docs/under-the-hood/error-model.md
+++ b/docs/under-the-hood/error-model.md
@@ -106,7 +106,7 @@ A message reaches the operator's diagnostics, the error tracker and every log li
- **A validator's own sentence** is kept only when it raises `InputFreeValueError` (`pipelex/tools/typing/validation_exceptions.py`), the `ValueError` by which a validator declares that its message names what it expected and never the value. The native contents' validators raise it, a date's, a time's, a number's, an image's and a JSON object's; any other validator's message may quote the value it rejected, so it is written as its type, `value_error`.
- **A URL** is the input's value, and possibly a signed link: a refusal names the input, never the URL, and the scheme only when it is one Pipelex reads, `http`, `https`, `s3`, `gs` or `pipelex-storage`, since any other is the caller's own text too, and a URL that does not parse is refused without the part httpx quotes. The host of a URL that could not be fetched, and the input it belongs to, are facts `RemoteFileFetchError` holds, not parameters its raise site gives: `input.url_unreachable` takes both, and a catch site that codes the failure otherwise, as a provider's file does with `provider.response_invalid`, is given neither, rather than a parameter its code would drop with a warning. The fetch knows the URL alone, so a catch site that knows the input names it with `with_input()`, or wraps the fetch in `fetching_for_input()` (`pipelex/tools/misc/file_fetch_utils.py`): an extract worker does, a prompt's image or document carries the input it came from as its `input_name`, `photos[2]`, which its preparation names, and a document rendered as page views carries it on its assignment.
- **A table an input gives** is the caller's file: a CSV refusal names the file by the input it was given as, `the CSV file given for input 'people'`, with the row, the field and the column's position, never the file's path, a cell, or a column the concept does not declare, and gives the input as the problem's `input` parameter and its `location.input`, which the site that knows the input names with `CsvError.with_input()`, the codec not knowing it: the four sites of `StuffFactory.try_make_csv_list_content`, the three refusals before the table is read and the codec's own refusals it re-raises.
-- **A function a `PipeFunc` calls** is named by the class of what it raised, `The function raised KeyError`: its text is the function's own, and `pipe.function_raised` declares no parameter for it. Any other failure of the executor is named by its class, and a Pipelex error by its code too, `PipeFuncExecutionError (runtime.unexpected)`, since an out-of-process executor's error, or a transport's, can carry the function's text in its own.
+- **A function a `PipeFunc` calls** is named by the class of what it raised, `The function raised KeyError`: its text is the function's own, and `pipe.function_raised` declares no parameter for it. Any other failure of the executor is named by its class, and a Pipelex error by the code it reports too, `PipeFuncExecutionError (runtime.unexpected)`, since an out-of-process executor's error, or a transport's, can carry the function's text in its own. A host plugin's carrier of a problem recovered across a transport boundary is named by that problem's code, `pipe.function_raised` for a function that raised in a sandbox, never by the catch-all `runtime.unexpected` its class declares (`reported_failure_code()` in `pipelex/pipe_run/located_failure.py`, which the run's own failure line reads as well).
- **A template that fails to render** is described by a fixed sentence naming the class Jinja2 or Python raised and the template's line, `the template reads a value that is not defined (UndefinedError, at line 2 of the template)`, never by that exception's text, which quotes what the template reached for, a key of the run's data included, nor by the template source. A name is kept only when the parsed template spells it as a variable or an attribute (`template_spelled_names()` in `pipelex/tools/jinja2/jinja2_rendering.py`), never because the same word appears in the template's text: the variable an undefined names, the attribute the sandbox refused to read, the name it refused to store a callable under in a namespace, the method it refused to call, the method whose call overran the render budget. The sandbox words each refusal with those names and without them (`SandboxRefusalError`), as the render budget words its refusal of a call (`RenderBudgetExceededError`), and a refusal whose names the template does not all spell is written without them, `A template may not read an unsafe name on a 'object' value.`, `calling a method of a 'str' value needs about 1,000,000,000,000`. A name a replacement field reads in a format string counts as spelled too, but only where the literal is the string a `.format` or `.format_map` call formats, such as the `_x` that `'{0._x}'.format(v)` reads, since the template's own literal wrote the access being refused; a word any other string literal holds, as `mode == 'debug _token'` does, and the plain text outside the expressions never count, so the sandbox and the budget read one rule (`_spells_every_name()`). A template that cannot render as written is described by the kind of fault and its line, `syntax error at line 3.`, never by the token Jinja2 quotes, and a filter or a test its environment lacks is named when the template applies it by that name, `No filter named 'frob' at line 1.`. Parsing a template to find its variables is refused the same way, by the kind of fault and its line. Pipelex's own filters refuse what they were handed without quoting it: an unknown text format by the formats they know, an undefined value given a tag without the tag's name.
- **A structured generation `instructor` gave up on** is classified from what ended its loop. When its re-asks end on a refused answer, the recovered `ValidationError` is written by its reasons, located by the requested class (`extract_openai_metadata()` and its siblings take it as `validated_type`), and any other failure to parse the answer by its class alone, such as the `AssertionError` `instructor`'s OpenAI parser raises with a model's refusal in its text: each worker passes its SDK's own error classes as `sdk_error_types`, and only those keep the text, which is the provider's; when nothing can be recovered, the failure is named by the class of the wrapper's cause, or of the wrapper itself (`describe_instructor_failure()`), never by a text or a representation, since a retry error's is a `Future`'s, a memory address included. `instructor`'s own text, which is the last validation error's and quotes the model's answer, is never used, and no sink writes `instructor`'s log records (see [Logging](../tools/logging.md)).
- **A foreign exception** a pipe raised stands as a `PipelexUnexpectedError` whose message is its class, `KeyError`, never its text, which can be a key of the run's data or a value a function was handed; a `ValidationError` adds its reasons. A failure that follows the pipe's own, a delivery's or the graph tracer's, is logged by its class and its code, never its text, which names the caller's webhook URL or the trace's file, and a webhook's transport failure is named by its class. A trace event log's refusal, and the tracing assembly's errors on the run's output, are written by their reasons, located by the event models' schema.
@@ -437,7 +437,7 @@ When a pipe fails during a run, the error a surface receives sits under several
So a run failing at a nested pipe reads, in the caller view, `"location": {"pipe": "summarize", "path": ["two_steps", "summarize"]}` beside the code's own sentence about it, and the location is never parsed out of a sentence. The wrappers' own messages, which an operator meets in a traceback and the CLIs print, still read `Pipe 'summarize' failed (two_steps → summarize): Model handle 'x' was not found in the model deck.` for a nested pipe and `Pipe 'flow' failed: …` for the entry pipe; `PipeRouterError` and `PipelineExecutionError` also carry the location structurally, as `pipe_code` and `pipe_stack`.
-**A recovered problem is taken as it is.** A distributed submitter receives a problem that was already located on the worker. When it raises that problem inside a `PipelexError` whose `recovered_problem()` returns it, which its `to_problem()` then reports in the view asked for with the location it is handed, and no `PipeRouterError` sits on the submitter's own chain, the runner's `PipelineExecutionError` reports it unchanged: its `pipe_code` is the entry pipe, its `pipe_stack` is empty, and it adds no location of its own. A located problem must never reach a router this way, since the router would locate it again: that is why a host router's transport packs the root fault's own problem. Only that hook makes a root fault report its own problem as it is: an inner `PipeRouterError` standing for a foreign exception, under a wrapper that attributed the failure to a model, is reported as the runner's wrapper reports it, so the model and the backend reach the problem. A bridge's own sentence (`Pipe execution failed in DIRECT mode …`) appears nowhere in the problem, so a run gives the same problem whether it ran in process, through the in-process orchestrator or on a remote worker.
+**A recovered problem is taken as it is.** A distributed submitter receives a problem that was already located on the worker. When it raises that problem inside a `PipelexError` whose `recovered_problem()` returns it, which its `to_problem()` then reports in the view asked for with the location it is handed, and no `PipeRouterError` sits on the submitter's own chain, the runner's `PipelineExecutionError` reports it unchanged: its `pipe_code` is the entry pipe, its `pipe_stack` is empty, and it adds no location of its own. A located problem must never reach a router this way, since the router would locate it again: that is why a host router's transport packs the root fault's own problem. Only that hook makes a root fault report its own problem as it is: an inner `PipeRouterError` standing for a foreign exception, under a wrapper that attributed the failure to a model, is reported as the runner's wrapper reports it, so the model and the backend reach the problem. A bridge's own sentence (`Pipe execution failed in DIRECT mode …`) appears nowhere in the problem, so a run gives the same problem whether it ran in process, through the in-process orchestrator or on a remote worker. A failed node of the execution graph is located the same way, at its own pipe (`pipelex/graph/node_problem.py`), so a step whose failure a carrier recovered records the recovered code on its trace record, as the run does.
**Where the remedy goes.** A problem's next step is its code's, the same on every surface, a hosted run's stored error included. Advice that only a local reader can act on belongs to the local CLI: `pipelex run` renders the model panel of a `PipeOperatorModelAvailabilityError` found anywhere on the chain, and the panel's tip is where the local model deck remedy lives.
diff --git a/docs/under-the-hood/execution-graph-tracing.md b/docs/under-the-hood/execution-graph-tracing.md
index 00b2a0d5d..8bdc1483c 100644
--- a/docs/under-the-hood/execution-graph-tracing.md
+++ b/docs/under-the-hood/execution-graph-tracing.md
@@ -225,7 +225,7 @@ Every `EdgeSpec` also carries an `optional` boolean (default `false`). On a `DAT
A `SKIPPED` node also carries `skip_reason` (a human-readable sentence naming the absent input, e.g. `skipped because input 'source' is absent`). A lifted pipe with a plural output still writes a real empty-list value, so its node registers that output and downstream `DATA` edges resolve normally; a lifted singular output is a recorded absence with no output spec.
-A failed node's `error` is the [problem document](error-model.md#the-problem-document) of the error that ended it, in the caller view, since the graph is read by whoever reads the run: the code, its sentences rendered from the parameters a caller may read, and never the error's own message. Its `location` names the node's own pipe and that pipe's path from the entry pipe, whatever pipe below it the failure happened in, so each failed node of a nested failure is located at itself while all of them carry the root fault's code. An exception that is not a Pipelex error records `runtime.unexpected`. Beside it, `error_stack` holds the stack trace of the error when the graph's data inclusion asks for it, capped at `MAX_STACK_LENGTH` (`cap_error_stack()`) both on the node and on the `PipeEndErrorEvent` that every trace subscriber reads before any node is built. Tracing never changes how a run fails: the problem is built while the pipe's error is on its way to the caller (`pipelex/graph/node_problem.py`), so when building it raises, the fault is logged as an error under the message `The problem of a failed pipe could not be built for its trace`, the node records `runtime.unexpected`, and the run fails with the pipe's own error.
+A failed node's `error` is the [problem document](error-model.md#the-problem-document) of the error that ended it, in the caller view, since the graph is read by whoever reads the run: the code, its sentences rendered from the parameters a caller may read, and never the error's own message. Its `location` names the node's own pipe and that pipe's path from the entry pipe, whatever pipe below it the failure happened in, so each failed node of a nested failure is located at itself while all of them carry the root fault's code. The problem is the error's own, whose code is the one its cause chain resolves to, the code the run fails with: an error that takes its cause's code, such as the `PipeRunError` a step wraps its failure in, carries its root fault's code, and an error that implies its code, such as an inference error built with a category, keeps the code it implies. When the fault deciding that code is a carrier, below the error, of a problem recovered across a transport boundary, a step run on a far worker or in a sandbox, the node reports that recovered problem as the pipe router does (`located_problem()` in `pipelex/pipe_run/located_failure.py`), so it records `pipe.function_raised` for a function that raised in a sandbox, never its carrier's `runtime.unexpected`: the node reads the code the run fails with. An exception that is not a Pipelex error records `runtime.unexpected`. Beside it, `error_stack` holds the stack trace of the error when the graph's data inclusion asks for it, capped at `MAX_STACK_LENGTH` (`cap_error_stack()`) both on the node and on the `PipeEndErrorEvent` that every trace subscriber reads before any node is built. Tracing never changes how a run fails: the problem is built while the pipe's error is on its way to the caller (`pipelex/graph/node_problem.py`), so when building it raises, the fault is logged as an error under the message `The problem of a failed pipe could not be built for its trace`, the node records `runtime.unexpected`, and the run fails with the pipe's own error.
### Node Paths
diff --git a/pipelex/graph/node_problem.py b/pipelex/graph/node_problem.py
index a8575d701..9e1532fdf 100644
--- a/pipelex/graph/node_problem.py
+++ b/pipelex/graph/node_problem.py
@@ -5,6 +5,7 @@ from mthds.protocol.problem.registry import CoreCode
from pipelex import log
from pipelex.base_exceptions import PipelexError
from pipelex.errors.problem import PipelexProblem, ProblemView, make_location, make_problem
+from pipelex.pipe_run.located_failure import find_root_fault, located_problem
from pipelex.tools.log.error_fields import error_fields
NODE_PROBLEM_FAULT_MESSAGE = "The problem of a failed pipe could not be built for its trace"
@@ -19,6 +20,14 @@ def make_node_problem(*, error: BaseException, pipe_code: str, pipe_path: list[s
path from the entry pipe, whatever pipe below it the failure happened in. An exception that is not a Pipelex error
is a fault of ours, `runtime.unexpected`.
+ The problem is the error's own (`PipelexError.to_problem`), whose code is the one its cause chain resolves to, the
+ code the run fails with. One shape needs more: when the fault deciding that code is a carrier, below the error, of a
+ problem recovered across a transport boundary, a step run on a far worker or in a sandbox, the error's own problem
+ would stop at the carrier's class code, `runtime.unexpected`. The node then reports the carrier's recovered problem
+ as the pipe router does (`pipelex.pipe_run.located_failure.located_problem`), so it reads the code the run fails
+ with. The fault is the one `find_root_fault` finds from the error itself, not from below it, so an error that implies
+ its code, such as an inference error built with a category, keeps that code rather than its cause's fallback.
+
Tracing never changes how a run fails. This runs while the pipe's own error is being handled, on its way to the
caller, so building the problem must not raise in its place: a problem that cannot be built is logged as an error,
and the node records `runtime.unexpected`.
@@ -34,6 +43,11 @@ def make_node_problem(*, error: BaseException, pipe_code: str, pipe_path: list[s
location = make_location(pipe=pipe_code, path=pipe_path or [pipe_code])
if isinstance(error, PipelexError):
try:
+ root_fault = find_root_fault(error=error)
+ if root_fault is not None and root_fault is not error and root_fault.recovered_problem() is not None:
+ return located_problem(
+ wrapper=error, root_fault=root_fault, view=ProblemView.CALLER, request_id=None, pipeline_run_id=None, failure_location=location
+ )
return error.to_problem(view=ProblemView.CALLER, failure_location=location)
except Exception as problem_fault: # ruff: ignore[blind-except]
# Unbounded code: `to_problem` reads hooks overridden across the whole error hierarchy, the errors plugins
diff --git a/pipelex/pipe_operators/func/pipe_func.py b/pipelex/pipe_operators/func/pipe_func.py
index 094cab1c5..55456b9f1 100644
--- a/pipelex/pipe_operators/func/pipe_func.py
+++ b/pipelex/pipe_operators/func/pipe_func.py
@@ -42,6 +42,7 @@ from pipelex.pipe_operators.func.pipe_func_python import (
locate_on_import_path,
)
from pipelex.pipe_operators.pipe_operator import PipeOperator
+from pipelex.pipe_run.located_failure import reported_failure_code
from pipelex.pipe_run.pipe_run_params import PipeRunParams
from pipelex.runtime_hub import get_class_registry
from pipelex.system.job_metadata import JobMetadata
@@ -55,9 +56,11 @@ def _describe_executor_failure(*, exc: BaseException) -> str:
"""What the executor's failure was, named by classes and a code, never by a text.
What the function raised is named by its class, which the in-process executor marks by raising `PipeFuncRaisedError`
- from it. Any other failure is named by its class, and a Pipelex error by its code as well: an out-of-process
- executor reports a coded Pipelex error whose message can hold the function's own text, and a foreign exception, a
- transport's, can quote it too. The text of either can quote the run's inputs, and no message carries those.
+ from it. Any other failure is named by its class, and a Pipelex error by the code it reports as well: an
+ out-of-process executor reports a coded Pipelex error whose message can hold the function's own text, and a foreign
+ exception, a transport's, can quote it too. The text of either can quote the run's inputs, and no message carries
+ those. A carrier of a problem recovered across a transport boundary is named by that problem's code, such as
+ `pipe.function_raised` for a function that raised in a sandbox, never by its own class's `runtime.unexpected`.
"""
if isinstance(exc, PipeFuncRaisedError):
function_exc = exc.__cause__
@@ -65,7 +68,7 @@ def _describe_executor_failure(*, exc: BaseException) -> str:
return "The function raised an exception"
return f"The function raised {type(function_exc).__name__}"
if isinstance(exc, PipelexError):
- return f"{type(exc).__name__} ({exc.resolve_error_code()})"
+ return f"{type(exc).__name__} ({reported_failure_code(failure=exc)})"
return type(exc).__name__
diff --git a/pipelex/pipe_run/located_failure.py b/pipelex/pipe_run/located_failure.py
index 32702001b..107c464a9 100644
--- a/pipelex/pipe_run/located_failure.py
+++ b/pipelex/pipe_run/located_failure.py
@@ -62,6 +62,21 @@ def find_root_fault(*, error: BaseException) -> PipelexError | None:
return implying_fault or root_fault
+def reported_failure_code(*, failure: PipelexError) -> str:
+ """The registry code `failure` reports, read without building its problem.
+
+ The code its cause chain resolves to (`PipelexError.resolve_error_code`), or, when its root fault carries a problem
+ recovered across a transport boundary (`PipelexError.recovered_problem`), that problem's code, as the problem a
+ located wrapper of `failure` reports does (`located_problem`). A carrier's own class declares the catch-all
+ `runtime.unexpected`, so a reader of the resolved code alone would name the carrier rather than the failure.
+ """
+ root_fault = find_root_fault(error=failure)
+ recovered = root_fault.recovered_problem() if root_fault is not None else None
+ if recovered is not None:
+ return recovered.code
+ return str(failure.resolve_error_code())
+
+
def find_foreign_fault(*, error: BaseException) -> BaseException | None:
"""Return the non-Pipelex exception a run failure stands in for, or `None`.
@@ -169,7 +184,8 @@ def located_problem(
`PipeRouterError` standing for a foreign exception under the runner's `PipelineExecutionError`.
Args:
- wrapper: The located wrapper, a `PipeRouterError` or a `PipelineExecutionError`.
+ wrapper: The located wrapper, a `PipeRouterError` or a `PipelineExecutionError`, or the error that ended a node
+ of the execution graph, which the node locates at its pipe (`pipelex.graph.node_problem`).
root_fault: The root fault below it (`located_root_fault`).
view: The view of the reader the problem is built for.
request_id: The request that carried the failure, when known.
diff --git a/pipelex/pipe_run/pipe_run.py b/pipelex/pipe_run/pipe_run.py
index a90131388..41defb497 100644
--- a/pipelex/pipe_run/pipe_run.py
+++ b/pipelex/pipe_run/pipe_run.py
@@ -15,7 +15,7 @@ from pipelex.interpreter_hub import get_concept_library, get_own_pipes
from pipelex.pipe_run.delivery_assignment import DeliveryAssignment, DeliveryStatus
from pipelex.pipe_run.delivery_executor import DeliveryExecutor
from pipelex.pipe_run.exceptions import DeliveryError
-from pipelex.pipe_run.located_failure import find_root_fault, make_unexpected_failure
+from pipelex.pipe_run.located_failure import make_unexpected_failure, reported_failure_code
from pipelex.pipe_run.pipe_run_protocol import PipeRunProtocol
from pipelex.pipe_run.tracing_assembly import RunTraceEvents, assemble_tracing_on_output, read_run_trace_events
from pipelex.pipeline.build_pipe_io_artifacts import build_pipe_io_artifacts
@@ -54,11 +54,7 @@ def _failure_code(*, error: Exception) -> str:
"""
if not isinstance(error, PipelexError):
return CoreCode.RUNTIME_UNEXPECTED
- root_fault = find_root_fault(error=error)
- recovered = root_fault.recovered_problem() if root_fault is not None else None
- if recovered is not None:
- return recovered.code
- return str(error.resolve_error_code())
+ return reported_failure_code(failure=error)
WEBHOOK_PROBLEM_FAULT_MESSAGE = "The problem of a failed run could not be built for its webhooks"
diff --git a/tests/helpers/recovered_problem.py b/tests/helpers/recovered_problem.py
index af02594af..809725d1e 100644
--- a/tests/helpers/recovered_problem.py
+++ b/tests/helpers/recovered_problem.py
@@ -29,6 +29,12 @@ class RecoveredProblemError(PipelexError):
return self.carried_problem
+def make_recovered_function_raised(*, pipe: str, function: str, message: str = "a sandbox reported the function's failure") -> RecoveredProblemError:
+ """A function a `PipeFunc` called that raised in a sandbox, as the sandbox's report reaches the submitter: `pipe.function_raised`."""
+ problem = make_problem(code=CoreCode.PIPE_FUNCTION_RAISED, view=ProblemView.OPERATOR, params={"pipe": pipe, "function": function})
+ return RecoveredProblemError(message, problem=problem)
+
+
def make_recovered_url_refusal() -> RecoveredProblemError:
"""A recovered refusal of an input URL: an `input` problem a retry can clear, unlike the carrier's own code."""
problem = make_problem(code=CoreCode.INPUT_URL_UNREACHABLE, view=ProblemView.OPERATOR, params={"input": "document", "host": "example.com"})
diff --git a/tests/unit/pipelex/graph/test_node_problem.py b/tests/unit/pipelex/graph/test_node_problem.py
index 48e2c7e3c..e5af04947 100644
--- a/tests/unit/pipelex/graph/test_node_problem.py
+++ b/tests/unit/pipelex/graph/test_node_problem.py
@@ -4,8 +4,15 @@ import pytest
from mthds.protocol.problem.registry import CoreCode
from pytest_mock import MockerFixture
+from pipelex.cogt.exceptions import InferenceErrorCategory, LLMCompletionError
+from pipelex.core.pipes.exceptions import PipeRunError
from pipelex.core.stuffs.exceptions import StuffFactoryError
+from pipelex.errors.problem import ProblemView
from pipelex.graph.node_problem import NODE_PROBLEM_FAULT_MESSAGE, make_node_problem
+from pipelex.pipe_run.exceptions import PipeRouterError
+from pipelex.pipe_run.located_failure import reported_failure_code
+from pipelex.system.pipe_run_mode import PipeRunMode
+from tests.helpers.recovered_problem import make_recovered_function_raised
class TestNodeProblem:
@@ -42,6 +49,61 @@ class TestNodeProblem:
assert problem.location is not None
assert problem.location.as_wire() == {"pipe": "flow", "path": ["flow"]}
+ def test_a_wrapper_above_a_recovered_problem_records_that_problem(self) -> None:
+ """A step's `PipeRunError` above a carrier of a recovered problem records the carried code, not the carrier's class code."""
+ carrier = make_recovered_function_raised(pipe="explode", function="funcs.explode.explode", message="secret-ish text the sandbox wrote")
+ wrapper = PipeRunError(message="PipeFunc 'explode' failed during execution", run_mode=PipeRunMode.LIVE, pipe_code="explode")
+ wrapper.__cause__ = carrier
+
+ problem = make_node_problem(error=wrapper, pipe_code="explode", pipe_path=["flow", "explode"])
+
+ assert carrier.own_error_code == CoreCode.RUNTIME_UNEXPECTED
+ assert problem.code == CoreCode.PIPE_FUNCTION_RAISED
+ assert problem.location is not None
+ assert problem.location.as_wire() == {"pipe": "explode", "path": ["flow", "explode"]}
+ assert problem.params is not None
+ assert problem.params.as_wire() == {"pipe": "explode", "function": "funcs.explode.explode"}
+ # The caller's view: no diagnostics, and none of the carrier's or the wrapper's own text.
+ assert problem.diagnostics is None
+ wire = str(problem.to_wire())
+ assert "secret-ish text" not in wire
+ assert "failed during execution" not in wire
+
+ def test_a_wrapper_records_the_code_the_run_fails_with(self) -> None:
+ """The node and the run read one answer: the node's problem has the code of the run's located failure."""
+ carrier = make_recovered_function_raised(pipe="explode", function="funcs.explode.explode")
+ wrapper = PipeRunError(message="PipeFunc 'explode' failed during execution", run_mode=PipeRunMode.LIVE, pipe_code="explode")
+ wrapper.__cause__ = carrier
+ located = PipeRouterError.make_located(
+ failure=wrapper, run_mode=PipeRunMode.LIVE, pipe_code="explode", output_name=None, pipe_stack=["flow", "explode"]
+ )
+ located.__cause__ = wrapper
+
+ node_problem = make_node_problem(error=wrapper, pipe_code="explode", pipe_path=["flow", "explode"])
+ run_problem = located.to_problem(view=ProblemView.CALLER)
+
+ assert node_problem.to_wire() == run_problem.to_wire()
+
+ def test_an_error_implying_its_code_records_that_code(self) -> None:
+ """An inference error implying its category's code, above a cause that carries none, records the implied code.
+
+ The error carries no code of its own, so a walk below it alone would stop at its cause's fallback,
+ `runtime.unexpected`, while the run reports the code the error implies: the node reads that code too.
+ """
+ cause = PipeRunError(message="the inner step failed", run_mode=PipeRunMode.LIVE, pipe_code="inner")
+ error = LLMCompletionError(message="the completion failed", error_category=InferenceErrorCategory.TRANSIENT)
+ error.fill_model_and_provider("gpt-4o-mini", backend_name="openai")
+ error.__cause__ = cause
+
+ problem = make_node_problem(error=error, pipe_code="summarize", pipe_path=["flow", "summarize"])
+
+ assert error.own_error_code is None
+ assert error.implied_error_code == CoreCode.PROVIDER_UNAVAILABLE
+ assert reported_failure_code(failure=error) == CoreCode.PROVIDER_UNAVAILABLE
+ assert problem.code == CoreCode.PROVIDER_UNAVAILABLE
+ assert problem.location is not None
+ assert problem.location.as_wire() == {"pipe": "summarize", "path": ["flow", "summarize"]}
+
def test_a_foreign_exception_is_a_fault_of_ours(self) -> None:
problem = make_node_problem(error=RuntimeError("The worker was lost"), pipe_code="flow", pipe_path=["flow"])
diff --git a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_failure_code.py b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_failure_code.py
index 737991876..ea6f4f1ab 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_failure_code.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_failure_code.py
@@ -12,6 +12,7 @@ from pipelex.core.pipes.exceptions import PipeRunError
from pipelex.core.stuffs.text_content import TextContent
from pipelex.errors.error_codes import ErrorCode
from pipelex.errors.problem import ProblemView
+from pipelex.graph.node_problem import make_node_problem
from pipelex.interpreter_hub import scoped_pipe_func_executor
from pipelex.kernel.exceptions import FuncOutputTypeError
from pipelex.pipe_machinery.pipe_factory import PipeFactory
@@ -27,6 +28,7 @@ from pipelex.system.exceptions import RunParameterError
from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
from pipelex.tools.storage.exceptions import StorageS3Error
+from tests.helpers.recovered_problem import make_recovered_function_raised
def _raises_a_request_refusal(working_memory: WorkingMemory) -> TextContent:
@@ -227,8 +229,14 @@ class TestPipeFuncFailureCode:
"Error: PipeFuncExecutionError (runtime.unexpected)",
),
(ConnectionResetError(f"The sandbox closed the call: {_FUNCTION_TEXT}"), "Error: ConnectionResetError"),
+ (
+ make_recovered_function_raised(
+ pipe="read_line", function="ledger_funcs.lines.read_line", message=f"The function raised in the sandbox: {_FUNCTION_TEXT}"
+ ),
+ "Error: RecoveredProblemError (pipe.function_raised)",
+ ),
],
- ids=["function_raised_far_away", "coded_seam_failure", "foreign_transport_failure"],
+ ids=["function_raised_far_away", "coded_seam_failure", "foreign_transport_failure", "recovered_function_raised"],
)
@pytest.mark.asyncio
async def test_an_out_of_process_failure_is_named_without_its_text(
@@ -247,6 +255,24 @@ class TestPipeFuncFailureCode:
assert step_failure.message.endswith(expected_error)
assert _FUNCTION_TEXT not in step_failure.message
+ @pytest.mark.asyncio
+ async def test_a_function_that_raised_in_a_sandbox_is_the_step_s_traced_failure(
+ self, load_empty_library: Callable[[], str], mocker: MockerFixture
+ ) -> None:
+ """The carrier of the sandbox's report declares `runtime.unexpected`: the step's trace record reads the code it carries."""
+ load_empty_library()
+ mocker.patch("pipelex.pipe_operators.func.pipe_func.is_pipe_func_sandbox_hosted", return_value=True)
+ carrier = make_recovered_function_raised(pipe="read_line", function="ledger_funcs.lines.read_line")
+
+ step_failure = await _failure_of(pipe_func=_make_pipe_func(), executor=_OutOfProcessExecutor(failure=carrier), function=_raises_a_plain_error)
+ node_problem = make_node_problem(error=step_failure, pipe_code="read_line", pipe_path=["flow", "read_line"])
+
+ assert step_failure.__cause__ is carrier
+ assert node_problem.code == CoreCode.PIPE_FUNCTION_RAISED
+ assert node_problem.location is not None
+ assert node_problem.location.as_wire() == {"pipe": "read_line", "path": ["flow", "read_line"]}
+ assert node_problem.diagnostics is None
+
@pytest.mark.builds_problems_without_required_params(
reason="the coercion of a function's return knows neither the step's output concept nor a field of it"
)