Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
feature/Security-refusal-event · L-261015-9b61e2 (#1449)
Louis Choquelb6b36e4e1b18
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 45a6542d0..325db720a 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -12,6 +12,7 @@- **`PipelexError.with_items()`**: a raise site that finds several faults of one kind gives one problem item per fault, so a `PipeParallel` whose branches give several results of a multiplicity their fields do not declare names each one in its `pipe.output_multiplicity` problem's `errors`, beside the first in its sentence.- **`PluginRegistrar.declare_error_codes`**: a plugin, a closed one included, declares the public codes it raises while it registers, and a code neither the MTHDS core nor the Pipelex registry defines refuses the boot with `PluginErrorCodeNotRegisteredError` (`pipelex.plugin.broken`), naming the plugin and the code.- **`PipelexRequestError`**: the runtime's own refusal of a request it cannot serve is a `PipelineRequestError` of the mthds package carrying the core code `request.invalid` (or `request.unsupported`) and its problem document, so a client reads the same refusal whether the runtime or the standard's client raised it.+- **The security event**: a refusal coded in the `security.` or `pipelex.security.` family fails its run under that code and logs one record on its own logger, `pipelex.errors.security_event`, at warning level, with the message `A security rule refused a request` and `event.name` set to `pipelex.security.refusal`, naming the code, its domain, the kind of target refused, the class of destination when the rule knows it (`loopback`, `private`, `link_local`, `metadata`, `scheme`, `foreign_storage_key`), the run, the request and each of the run's extras as `run.extras.<key>`, and never the URL, path, template or any input value. The process that raises a refusal logs it, once: the pipe run for a refusal raised while it executes, its delivery included, the runner for one before the run exists and for one raised in its own process that no pipe run logged, and the API server for one answered before any run, the refusals it authors itself included (a reserved object key, a callback URL the SSRF rule refuses, a resource path or a method file path leaving the method: absolute, naming a drive or a network share, or climbing with `..`, read with either separator, while a path of the wrong shape is `request.invalid` and logs none), which keep their response, while a failure rebuilt from another process's report is never logged again, since the orchestrator that rebuilds it marks it, and no event names a run id a request body chose; `pipelex.errors.security_event` gives a host `emit_security_event()`, `mark_security_event_emitted()` and `log_security_event()`, and a raise site states the class of destination with `PipelexError.with_security_target_class()`.### Changed@@ -33,7 +34,8 @@- **A truncated completion says which limit cut it**: `LLMCompletionTruncatedError` carries `model.output_truncated` when the method's `max_tokens` cut the answer and `model.context_exceeded` when the context window did, both the method's to change.- **`ModelDeckPresetValidationError` is spelt right (Breaking)**: the error a model deck preset that fails validation raises was `ModelDeckPresetValidatonError`, and its reports carried that `error_type`; a consumer that branched on the old name branches on the new one, or on its code, `pipelex.config.invalid`.- **`pipelex-agent` reads `retryable` from the code (Breaking)**: the agent CLI's error envelope takes `retryable` from the error's code, and its table of retryable error types is gone, as are the domain-table entries of classes that now carry a code. The envelope names that code as `error_code`, the member a hosted refusal already carried, so an agent branches on the code whichever runner failed, and its `hint`, when the error's chain advises nothing and the table names no hint for its class, is its code's next step as the operator reads it.-- **`Jinja2TemplateSecurityError` is a `SecurityError` (Breaking)**: the template sandbox's refusal at render time now derives from `SecurityError`, like `UriReadRefusedError`, `SsrfBlockedError` and `UnsafeSchemaError`, instead of `ToolError`, so a host catching `SecurityError` catches it too and an `except ToolError` no longer does; it stays a type of its own, apart from `Jinja2TemplateRenderError`, and its class name, which the operator reads as `diagnostics.error_class`, its title and its classification are unchanged.+- **`Jinja2TemplateSecurityError` is a `SecurityError` (Breaking)**: the template sandbox's refusal at render time now derives from `SecurityError`, like `UriReadRefusedError`, `SsrfBlockedError` and `UnsafeSchemaError`, instead of `ToolError`, so a host catching `SecurityError` catches it too and an `except ToolError` no longer does; it stays a type of its own, apart from `Jinja2TemplateRenderError`, and its class name, which the operator reads as `diagnostics.error_class`, and its code, `security.template_refused` on a method's template, are unchanged.+- **A security refusal is never a validation verdict (Breaking)**: a refusal a raise site coded in the `security.` family is let through by the bundle-validation cascade whatever its class, as a `SecurityError` always was, so a `PipeDocGen` step whose `template_file` climbs above its method tree's root fails `validate` and a run with `security.method_path_refused` and logs its security event, where it was a `method.invalid` verdict naming the pipe.- **`mthds` 0.22.1 and `pipelex-sdk` 0.46.0 (Breaking)**: the exact pins move from `mthds==0.22.0` and `pipelex-sdk==0.45.0` to `mthds==0.22.1`, the release whose errors carry the MTHDS problem document read by its registry code, and `pipelex-sdk==0.46.0`, the release built on it; everyone downstream inherits those exact versions.### Fixed@@ -56,6 +58,10 @@- **The per-class error pages (Breaking)**: the generated page every `PipelexError` class had under `docs/errors/`, `pipelex-dev generate-error-pages` (`make gep`) and `pipelex.errors.error_pages_generator` are gone; a problem's `type` is now its code's page, and docs.pipelex.com redirects each retired page to the page of its class's code, or to the Error Codes index when the class is gone or raises no single code.- **Class-level error classification (Breaking)**: an error class no longer declares `error_domain` or `retryable`, `PipelexError.title()` and `PipelexError.type_uri()` are gone, and so is `InferenceErrorCategory.error_domain`: the class declares its `error_code`, which decides all four, and an inference failure's category, which stays inside the runtime, picks the code its worker raises. `as_caller_fault()` is gone too, since a caller reads the sentences a code renders and never an error's own message.+### Security++- **Callback URLs are held to the runtime's SSRF rule**: the runner refuses a `/v1/start` callback URL with the rule the webhook delivery applies when it fires, `pipelex.tools.network.host_rules`, so a cloud metadata name spelled in capitals or with a trailing dot, and an address that is not publicly routable such as a carrier-grade NAT one, are refused with a 422 `InvalidCallbackUrls` before the run, where the runner's own copy of the rule let them through to the delivery.+## [v0.89.0] - 2026-10-17### Addeddiff --git a/api/pipelex_api/bundle_resources.py b/api/pipelex_api/bundle_resources.pyindex 00282f24c..7be215ad3 100644--- a/api/pipelex_api/bundle_resources.py+++ b/api/pipelex_api/bundle_resources.py@@ -11,8 +11,9 @@ the tree's rule judges them as it judges the tree's own files.Refusals, in the order they are checked:- more `resources` than `MAX_RESOURCES_PER_REQUEST` → 413 `PayloadTooLarge`;-- a path that cannot name a file inside the bundle (absolute, a `..` segment, a backslash, a colon, empty, or a- second spelling of a path already submitted) → 422 `InvalidResourcePath`, before any value is decoded;+- a path that cannot name a file inside the bundle (absolute, a drive, a `..` segment, a backslash, a colon, empty, or+ a second spelling of a path already submitted) → 422 `InvalidResourcePath`, before any value is decoded, which also+ logs a security event when the path leaves the bundle, as a key of a method tree does;- a file whose encoded length already exceeds the per-file ceiling → 413 `PayloadTooLarge`, before decoding;- a value that is not valid base64 → 400 `InvalidBase64`, as `bundle_b64` is;- a decoded file over `MAX_RESOURCE_FILE_BYTES`, or a running total over `MAX_RESOURCES_TOTAL_BYTES` → 413.@@ -28,7 +29,7 @@ from pipelex.core.bundle_resources.exceptions import BundleResourcePathErrorfrom pipelex.tools.log.error_fields import error_fieldsfrom pipelex_api.error_types import ErrorType-from pipelex_api.errors import raise_bad_request, raise_payload_too_large, raise_validation_error+from pipelex_api.errors import raise_bad_request, raise_payload_too_large, raise_validation_error, security_refusal_offrom pipelex_api.limits import MAX_RESOURCE_FILE_BYTES, MAX_RESOURCES_PER_REQUEST, MAX_RESOURCES_TOTAL_BYTES# Base64 inflates by 4/3, so a file within the ceiling encodes to at most this many characters (+4 covers padding).@@ -57,7 +58,10 @@ def decode_bundle_resources(*, resources: dict[str, str] | None) -> dict[str, bytry:canonical_path = canonical_resource_path(path=path)except BundleResourcePathError as exc:- raise_validation_error(message=exc.message, error_type=ErrorType.INVALID_RESOURCE_PATH)+ # A path that leaves the bundle is a security rule's refusal (`security.input_path_refused`), whose security event+ # the response's handler logs; one of the wrong shape is the request's (`request.invalid`), and logs none. The+ # response keeps its error type either way.+ raise_validation_error(message=exc.message, error_type=ErrorType.INVALID_RESOURCE_PATH, security_refusal=security_refusal_of(error=exc))if canonical_path in sent_by_path:msg = f"Two files submitted with the bundle name the same file '{canonical_path}'."raise_validation_error(message=msg, error_type=ErrorType.INVALID_RESOURCE_PATH)diff --git a/api/pipelex_api/errors.py b/api/pipelex_api/errors.pyindex d619f030a..e99633ff4 100644--- a/api/pipelex_api/errors.py+++ b/api/pipelex_api/errors.py@@ -16,14 +16,43 @@ itself."""from collections.abc import Mapping-from typing import Any, NoReturn+from typing import Any, NamedTuple, NoReturn-from pipelex.base_exceptions import ErrorDomain+from pipelex.base_exceptions import ErrorDomain, PipelexError+from pipelex.errors.error_codes import ErrorCode+from pipelex.errors.security_event import log_security_event, security_target_kind+from pipelex.errors.security_target import SecurityTargetClassfrom pipelex_api.error_types import ErrorTypefrom pipelex_api.problem_document import build_problem_document_from_api_error+class ApiSecurityRefusal(NamedTuple):+ """The refusal of a security rule an API-authored error answers, which its handler logs as a security event.++ It holds what the event names and never what was refused: the refusal's code, the class of the error that refused+ (or the response's `error_type` when no error class did), and the class of destination when the rule knows it.+ """++ code: ErrorCode+ error_type: str+ target_class: SecurityTargetClass | None = None+++def security_refusal_of(*, error: BaseException | None) -> ApiSecurityRefusal | None:+ """The security refusal `error` reports, when it is a `PipelexError` whose code is a security refusal, else `None`.++ For a route that answers a pipelex refusal with an API-authored problem document, which keeps the response the+ API has always given while the refusal still logs its security event.+ """+ if not isinstance(error, PipelexError):+ return None+ code = error.resolve_error_code()+ if security_target_kind(code=code) is None:+ return None+ return ApiSecurityRefusal(code=code, error_type=type(error).__name__, target_class=error.security_target_class)++class ApiError(Exception):"""An API-authored error carrying a ready-rendered RFC 7807 problem document.@@ -36,13 +65,33 @@ class ApiError(Exception):`Request`."""- def __init__(self, *, status_code: int, document: dict[str, Any], headers: dict[str, str] | None = None) -> None:+ def __init__(+ self,+ *,+ status_code: int,+ document: dict[str, Any],+ headers: dict[str, str] | None = None,+ security_refusal: ApiSecurityRefusal | None = None,+ ) -> None:self.status_code = status_codeself.document = documentself.headers: dict[str, str] = headers or {}+ self.security_refusal = security_refusalsuper().__init__(str(document.get("detail", "")))+def log_api_security_event(*, api_error: ApiError, extras: Mapping[str, str] | None) -> None:+ """Log the security event of an API-authored error that answers a security rule's refusal, and nothing otherwise.++ The event names the request through the log context and the run's extras the body carried, never a run id, since+ no run exists yet, and never what was refused.+ """+ refusal = api_error.security_refusal+ if refusal is None:+ return+ log_security_event(code=refusal.code, error_type=refusal.error_type, target_class=refusal.target_class, extras=extras)++def _raise_api_error(*,error_type: ErrorType,@@ -51,6 +100,7 @@ def _raise_api_error(error_domain: ErrorDomain,headers: dict[str, str] | None = None,members: Mapping[str, Any] | None = None,+ security_refusal: ApiSecurityRefusal | None = None,) -> NoReturn:"""Build the RFC 7807 document and raise `ApiError`.@@ -72,21 +122,44 @@ def _raise_api_error(if members:# Extension members of RFC 7807 (section 3.2), such as `missing_blobs`: data a client acts on, beside the prose.document.update(members)- raise ApiError(status_code=status, document=document, headers=headers)+ raise ApiError(status_code=status, document=document, headers=headers, security_refusal=security_refusal)-def raise_validation_error(message: str, error_type: ErrorType = ErrorType.VALIDATION_ERROR) -> NoReturn:- """Raise a 422 RFC 7807 problem response for invalid caller input."""- _raise_api_error(error_type=error_type, message=message, status=422, error_domain=ErrorDomain.INPUT)+def raise_validation_error(+ message: str,+ error_type: ErrorType = ErrorType.VALIDATION_ERROR,+ *,+ security_refusal: ApiSecurityRefusal | None = None,+) -> NoReturn:+ """Raise a 422 RFC 7807 problem response for invalid caller input.++ `security_refusal` names the security rule the input broke, when it broke one, so the response's handler logs+ the security event of the refusal; the response is the same either way.+ """+ _raise_api_error(error_type=error_type, message=message, status=422, error_domain=ErrorDomain.INPUT, security_refusal=security_refusal)-def raise_validation_problem(*, message: str, error_type: ErrorType, members: Mapping[str, Any]) -> NoReturn:+def raise_validation_problem(+ *,+ message: str,+ error_type: ErrorType,+ members: Mapping[str, Any],+ security_refusal: ApiSecurityRefusal | None = None,+) -> NoReturn:"""Raise a 422 RFC 7807 problem response for invalid caller input, carrying extension members a client acts on.The members join the document beside `detail`, such as the `missing_blobs` a sender uploads before it sends the- request again, or the `tree_refusal` naming the standard's refusal code and the paths it names.+ request again, or the `tree_refusal` naming the standard's refusal code and the paths it names. `security_refusal`+ is `raise_validation_error`'s."""- _raise_api_error(error_type=error_type, message=message, status=422, error_domain=ErrorDomain.INPUT, members=members)+ _raise_api_error(+ error_type=error_type,+ message=message,+ status=422,+ error_domain=ErrorDomain.INPUT,+ members=members,+ security_refusal=security_refusal,+ )def raise_bad_request(message: str, error_type: ErrorType = ErrorType.BAD_REQUEST) -> NoReturn:diff --git a/api/pipelex_api/exception_handlers.py b/api/pipelex_api/exception_handlers.pyindex 0f99e63b3..f76e9eb7c 100644--- a/api/pipelex_api/exception_handlers.py+++ b/api/pipelex_api/exception_handlers.py@@ -42,6 +42,7 @@ from pipelex.base_exceptions import ErrorDomain, PipelexErrorfrom pipelex.cogt.exceptions import find_inference_error_category_in_chainfrom pipelex.errors.error_codes import error_code_spec, parse_error_codefrom pipelex.errors.problem import PipelexProblem, ProblemView, caller_view_of+from pipelex.errors.security_event import emit_security_eventfrom pipelex.pipe_run.located_failure import make_unexpected_failurefrom pipelex.plugins.registrar import HttpErrorMapperFnfrom pipelex.system.telemetry.otel_constants import OTelLogAttr@@ -49,7 +50,7 @@ from pipelex.tools.log.error_fields import ERROR_TYPE_FIELDfrom starlette.requests import ClientDisconnectfrom pipelex_api.error_types import ErrorType-from pipelex_api.errors import ApiError+from pipelex_api.errors import ApiError, log_api_security_eventfrom pipelex_api.middleware import request_id_offrom pipelex_api.problem_document import (PROBLEM_JSON_MEDIA_TYPE,@@ -113,6 +114,16 @@ def _pipeline_run_id_of(request: Request) -> str | None:return getattr(request.state, "pipeline_run_id", None)+def _run_extras_of(request: Request) -> dict[str, str] | None:+ """Return the run's extras when `_parse_request` bound them on the request.++ The opaque labels a host attaches to a run, which the hosted plane fills with the organization. They ride+ the security event a refusal answered here logs, so a refusal before any run exists still names whose+ request it was. `None` for a route that parses no run request, or a body whose extras did not validate.+ """+ return getattr(request.state, "run_extras", None)++def _request_fields(request: Request) -> dict[str, Any]:"""Return the request-scoped attributes every error record carries.@@ -341,6 +352,11 @@ async def handle_pipelex_error(request: Request, exc: Exception) -> Response:that plugin's own mapper-backed handler instead."""error = cast("PipelexError", exc)+ # A security refusal logs its security event, once: a run's boundary already reported a refusal it saw, and+ # one refused before any run exists, or by a route around the runner, is reported here. The request id is+ # the log context's, which `RequestIdMiddleware` binds for the whole request. No run id: the one a body names+ # is the caller's claim, and no run the runtime assigned one to exists yet.+ emit_security_event(error=error, extras=_run_extras_of(request))return problem_response(operator_problem_of(error, request=request), error=error, request=request)@@ -405,8 +421,11 @@ async def handle_api_error(request: Request, exc: Exception) -> Response:and emits the structured error record so an API-owned 500 from anyroute is observable (a `raise_internal_server_error` site like `version.py`'smissing-package case is the canonical example — it has no preceding- `log.error` at the call site). `exc` is typed `Exception` to match Starlette's- handler contract; FastAPI only routes an `ApiError` here, so the cast is sound.+ `log.error` at the call site). An `ApiError` answering a security rule's refusal,+ a reserved object key, a callback URL or a path leaving the method, also logs its+ security event here, with the run's extras the body carried and no run id.+ `exc` is typed `Exception` to match Starlette's handler contract; FastAPI only+ routes an `ApiError` here, so the cast is sound."""api_error = cast("ApiError", exc)document = with_request_context(@@ -415,6 +434,7 @@ async def handle_api_error(request: Request, exc: Exception) -> Response:request_id=request_id_of(request),)_log_api_authored_error(document=document, status=api_error.status_code, request=request)+ log_api_security_event(api_error=api_error, extras=_run_extras_of(request))return JSONResponse(status_code=api_error.status_code,content=document,diff --git a/api/pipelex_api/exceptions.py b/api/pipelex_api/exceptions.pyindex b4d5e08b0..282255cea 100644--- a/api/pipelex_api/exceptions.py+++ b/api/pipelex_api/exceptions.py@@ -10,6 +10,7 @@ The API-authored refusals (``pipelex_api.errors.ApiError``, built from ``pipelexseparate family with no pipelex error behind them."""+from mthds.protocol.problem.registry import CoreCodefrom pipelex.base_exceptions import PipelexErrorfrom pipelex.errors.registry.pipelex_codes import PipelexErrorCode@@ -24,3 +25,14 @@ class InvalidAuthModeError(PipelexError):"""Raised at startup when `AUTH_MODE` holds a value that is not an `AuthMode` member."""error_code = PipelexErrorCode.CONFIG_INVALID+++class CallbackUrlRefusedError(PipelexError, ValueError):+ """A completion callback URL the SSRF rules refuse: a scheme other than http or https, or a disallowed host.++ Also a ``ValueError``, since the check runs inside the request model's validator, which turns only a ``ValueError``+ into a validation error. The route answering the validation error reads it from there to log its security event,+ with the class of destination it states and never the URL.+ """++ error_code = CoreCode.SECURITY_INPUT_URL_REFUSEDdiff --git a/api/pipelex_api/method_tree_request.py b/api/pipelex_api/method_tree_request.pyindex 3011698c1..57a16fce6 100644--- a/api/pipelex_api/method_tree_request.py+++ b/api/pipelex_api/method_tree_request.py@@ -52,6 +52,7 @@ from mthds.package.manifest.schema import MethodsManifestfrom mthds.protocol.problem.registry import CoreCodefrom pipelex.cli.installed_methods import PROVENANCE_FILENAMEfrom pipelex.config import is_pipe_func_sandbox_hosted+from pipelex.core.bundle_resources.bundle_resource import root_escape_refusalfrom pipelex.libraries.written_trees import WRITTEN_METHOD_TREES, write_method_treefrom pipelex.methods.structures_check import ensure_no_structured_content_in_library_sources, ensure_no_structured_content_pythonfrom pipelex.mthds_parsing.parser import MthdsParser@@ -63,7 +64,7 @@ from pipelex_api.blob_store import RunnerBlobStorefrom pipelex_api.bundle import decode_bundle_archivefrom pipelex_api.bundle_resources import refuse_two_files_under_one_pathfrom pipelex_api.error_types import ErrorType-from pipelex_api.errors import raise_forbidden, raise_validation_error, raise_validation_problem+from pipelex_api.errors import ApiSecurityRefusal, raise_forbidden, raise_validation_error, raise_validation_problemfrom pipelex_api.limits import (MAX_METHOD_DEPENDENCIES,MAX_METHOD_REQUEST_TOTAL_BYTES,@@ -371,9 +372,52 @@ def _refuse_map(*, refusal_error: MethodTreeRefusedError, label: str) -> NoReturmessage=f"{label} is not a method tree. {refusal_error.message}",error_type=ErrorType.INVALID_METHOD_FILES,members={"tree_refusal": {"code": refusal.code, "names": [wire_name(name) for name in refusal.names]}},+ security_refusal=_tree_path_security_refusal(refusal_error=refusal_error),)+def _tree_path_security_refusal(*, refusal_error: MethodTreeRefusedError) -> ApiSecurityRefusal | None:+ """The security refusal a tree refusal is, when a path it names tries to leave the tree, else `None`.++ The standard's reader refuses a key that is not a tree path without saying why, so the runtime's one rule for a+ caller's path that leaves its root decides (`root_escape_refusal`): a path that is absolute, names a drive or a+ network share, or climbs with a `..` segment, read with either separator, is a traversal,+ `security.input_path_refused`, whether the caller sent it as a key of `files` or as a member of `bundle_b64`, which+ the same reader judges, and as it is under `resources`. Every other path the reader refuses (an empty segment, a+ backslash, a colon, a path not in NFC) is the shape of how the caller packed the method, a refusal of the request+ and no security event.+ """+ refusal = refusal_error.refusal+ match refusal.code:+ case TreeRefusalCode.INVALID_TREE_PATH:+ if any(root_escape_refusal(path=name) is not None for name in refusal.names):+ return ApiSecurityRefusal(code=CoreCode.SECURITY_INPUT_PATH_REFUSED, error_type=type(refusal_error).__name__)+ return None+ case (+ TreeRefusalCode.MISSING_BLOB+ | TreeRefusalCode.SYMBOLIC_LINK+ | TreeRefusalCode.SPECIAL_FILE+ | TreeRefusalCode.NORMALIZATION_COLLISION+ | TreeRefusalCode.PATH_COLLISION+ | TreeRefusalCode.EMPTY_TREE+ | TreeRefusalCode.LONE_FILE_EXCLUDED+ | TreeRefusalCode.GITIGNORE_NOT_UTF8+ | TreeRefusalCode.DUPLICATE_KEY+ | TreeRefusalCode.EXCLUDED_NAME+ | TreeRefusalCode.MALFORMED_ENTRY+ | TreeRefusalCode.REFERENCE_INVALID+ | TreeRefusalCode.REFERENCE_LEAVES_TREE+ | TreeRefusalCode.REFERENCE_NOT_A_TREE_PATH+ | TreeRefusalCode.REFERENCE_NOT_IN_TREE+ | TreeRefusalCode.REFERENCE_IN_ONE_FILE_TREE+ | TreeRefusalCode.FUNCTION_NAME_WITHOUT_MODULE+ | TreeRefusalCode.FUNCTION_NAME_INVALID_SEGMENT+ | TreeRefusalCode.MODULE_NOT_IN_TREE+ | TreeRefusalCode.MODULE_AMBIGUOUS+ ):+ return None++async def _held_blobs(*, blob_store: RunnerBlobStore, stated_sizes: dict[str, set[int]], label: str) -> BlobStore:"""The bytes a map's hash entries name that the blob store holds, looked up by size before any is read.diff --git a/api/pipelex_api/routes/pipelex/pipeline.py b/api/pipelex_api/routes/pipelex/pipeline.pyindex e9e668ae7..f3b5e0d23 100644--- a/api/pipelex_api/routes/pipelex/pipeline.py+++ b/api/pipelex_api/routes/pipelex/pipeline.py@@ -37,7 +37,7 @@ from pipelex_api.api_config import ApiConfig, get_api_config, is_callers_overridfrom pipelex_api.blob_store import RunnerBlobStorefrom pipelex_api.bundle_resources import decode_bundle_resourcesfrom pipelex_api.error_types import ErrorType-from pipelex_api.errors import raise_bad_request, raise_forbidden, raise_validation_error+from pipelex_api.errors import ApiSecurityRefusal, raise_bad_request, raise_forbidden, raise_validation_error, security_refusal_offrom pipelex_api.json_body import decode_json_bodyfrom pipelex_api.method_source import fetched_methodfrom pipelex_api.method_tree_request import (@@ -687,7 +687,7 @@ def _find_reserved_object_key(payload: object) -> str | None:def _decode_body(body: bytes) -> dict[str, Any]:- """Parse the run body as plain JSON and confirm it is an object free of class markers. Raises 422 if not.+ """Parse the run body as plain JSON and confirm it is an object. Raises 422 if not.The body is parsed as plain JSON by `decode_json_body`, never with kajson. kajson reads a`{"__class__": ..., "__module__": ...}` object as an order to import that module and@@ -696,8 +696,10 @@ def _decode_body(body: bytes) -> dict[str, Any]:wire is plain JSON, and no client sends class markers.Parsing as plain data is not enough on its own: a marker accepted as an ordinary dict comes- back to life the moment the runtime round-trips the inputs through kajson. So a key a kajson- decoder treats as a marker is refused at any depth with a 422 `ReservedObjectKey`.+ back to life the moment the runtime round-trips the inputs through kajson. So the caller,+ `_parse_request`, refuses a key a kajson decoder treats as a marker at any depth with a 422+ `ReservedObjectKey` (`_refuse_reserved_object_keys`), once the run's extras are bound for the+ security event that refusal logs.The parse is `decode_json_body`'s, so its failures are caller mistakes answered with a 422`InvalidJSON` rather than a sanitized 500: a body nested deeper than `MAX_JSON_NESTING_DEPTH`,@@ -710,7 +712,15 @@ def _decode_body(body: bytes) -> dict[str, Any]:message="Request body must be a JSON object",error_type=ErrorType.INVALID_JSON,)- request_data = cast("dict[str, Any]", decoded)+ return cast("dict[str, Any]", decoded)+++def _refuse_reserved_object_keys(request_data: dict[str, Any]) -> None:+ """Refuse a body holding an object key a kajson decoder reads as a class marker, at any depth, with a 422.++ The refusal is a security rule's, `pipelex.security.reserved_key`, so it logs a security event, which names the+ kind of thing refused and never the key.+ """reserved_key = _find_reserved_object_key(request_data)if reserved_key is not None:# The key is the caller's, as long as the body allows (any `__kajson…` key matches), and the@@ -722,8 +732,8 @@ def _decode_body(body: bytes) -> dict[str, Any]:f"no object in it may carry a '__class__' or '__module__' key, or a key starting with '{_RESERVED_OBJECT_KEY_PREFIX}'."),error_type=ErrorType.RESERVED_OBJECT_KEY,+ security_refusal=ApiSecurityRefusal(code=PipelexErrorCode.SECURITY_RESERVED_KEY, error_type=ErrorType.RESERVED_OBJECT_KEY),)- return request_data# The `error_type` a 422 on one extras field carries. A field absent from this map@@ -765,9 +775,36 @@ def _validate_extras(request_data: dict[str, Any]) -> PipelineApiExtras:raise_validation_error(message=str(exc),error_type=_extras_error_type(exc),+ security_refusal=_security_refusal_in(exc),)+def _security_refusal_in(exc: ValidationError) -> ApiSecurityRefusal | None:+ """The refusal of a security rule an extras validator raised, such as a callback URL aimed at a private address.++ A validator's error rides its entry's context, where pydantic keeps the exception it raised. A body that also fails+ on another field is still the security rule's refusal, so the first one found is reported whatever else failed.+ """+ for error in exc.errors():+ refusal = security_refusal_of(error=error.get("ctx", {}).get("error"))+ if refusal is not None:+ return refusal+ return None+++def _validated_run_extras_or_none(request_data: dict[str, Any]) -> dict[str, str] | None:+ """The body's `analytics_groups` when they pass the runtime's rules on their own, else `None`.++ Read before anything else is refused, so that the security event of a refusal before the extras are validated,+ such as a reserved object key or a callback URL, still names whose request it was, with values the runtime's+ own rules accepted.+ """+ try:+ return PipelineApiExtras.model_validate({"analytics_groups": request_data.get("analytics_groups")}).analytics_groups+ except ValidationError:+ return None++# Per-field bound applied at the request.state binding site so an oversized# caller-supplied `pipe_code` cannot blow up the size of every record the request emits.# `RunRequest.pipe_code` carries no Pydantic `max_length`; this is the@@ -807,7 +844,7 @@ async def _parse_request(request: Request) -> _ParsedRunRequest:Splits the body into:1. The upstream `RunRequest` (pipe_code, mthds_contents, inputs, …),- parsed as plain JSON by `_decode_body`, which refuses class markers.+ parsed as plain JSON by `_decode_body`, then refused by `_refuse_reserved_object_keys` if it carries a class marker.2. `PipelineApiExtras` (pipeline_run_id, callback_urls, orchestration_mode,storage_scope, analytics_groups) validated by Pydantic — callback_urlsare checked for scheme + private/loopback hosts to harden against SSRF.@@ -816,6 +853,9 @@ async def _parse_request(request: Request) -> _ParsedRunRequest:"""body = await request.body()request_data = _decode_body(body)+ # The run's opaque extras, for the security event a refusal of this request logs before any run carries them.+ request.state.run_extras = _validated_run_extras_or_none(request_data)+ _refuse_reserved_object_keys(request_data)# Bind body-derived correlation identifiers onto `request.state` as soon as# the raw dict is in hand — before `_validate_extras` or `from_body`, so a# later validation failure (a rejected callback URL, a Pydantic coerciondiff --git a/api/pipelex_api/schemas/models.py b/api/pipelex_api/schemas/models.pyindex 40838b6d5..46a7b7e81 100644--- a/api/pipelex_api/schemas/models.py+++ b/api/pipelex_api/schemas/models.py@@ -8,22 +8,24 @@ one — owns it. `RunRequest` / `StartRequest` are defined here, not imported.from __future__ import annotations-from ipaddress import ip_addressfrom typing import Annotated, Any, Selffrom urllib.parse import urlparsefrom mthds.protocol.pipe_output import VariableMultiplicityfrom mthds.protocol.pipeline_inputs import PipelineInputsfrom pipelex.core.pipes.pipe_output import PipeOutput+from pipelex.errors.security_target import SecurityTargetClassfrom pipelex.methods.fetching import MethodProvenancefrom pipelex.pipeline.pipeline_response import PipelexRunResultExecute, PipelexRunResultStartfrom pipelex.pipeline.request_exceptions import PipelexRequestErrorfrom pipelex.reporting.usage_records import TokensUsageRecordfrom pipelex.system.run_extras import validate_run_extrasfrom pipelex.system.storage_scope import validate_read_scope, validate_storage_scope+from pipelex.tools.network.host_rules import classify_disallowed_host, is_disallowed_hostfrom pydantic import BaseModel, ConfigDict, Field, field_validator, model_validatorfrom pydantic.functional_validators import SkipValidation+from pipelex_api.exceptions import CallbackUrlRefusedErrorfrom pipelex_api.limits import (DEFAULT_MAX_RESOURCE_FILE_KIB,DEFAULT_MAX_RESOURCES_PER_REQUEST,@@ -396,22 +398,15 @@ _ALLOWED_CALLBACK_SCHEMES = frozenset({"http", "https"})def _is_disallowed_host(host: str) -> bool:- """True if `host` looks like a private/loopback/link-local address.+ """True if `host` is a destination a completion callback must not reach: the runtime's own SSRF rule.- Used to harden /start callback_urls against SSRF — a malicious- client could otherwise aim webhooks at internal services or cloud metadata- endpoints (e.g. 169.254.169.254). Best-effort: hostnames that resolve to- private addresses at fire time aren't blocked here, only literal IPs.+ Used to harden /start callback_urls against SSRF — a malicious client could otherwise aim webhooks at internal+ services or cloud metadata endpoints (e.g. 169.254.169.254). The rule is the one the delivery-time guard applies+ when the webhook fires, `pipelex.tools.network.host_rules`, so a spelling the guard refuses at fire time (an+ uppercase or trailing-dot alias, a carrier-grade NAT address) is refused here first. Best-effort: hostnames that+ resolve to private addresses at fire time aren't blocked here, only literal IPs and the known internal names."""- if not host:- return True- if host in {"localhost", "metadata.google.internal", "metadata"}:- return True- try:- addr = ip_address(host)- except ValueError:- return False- return addr.is_private or addr.is_loopback or addr.is_link_local or addr.is_multicast or addr.is_reserved or addr.is_unspecified+ return is_disallowed_host(host)class PipelineApiExtras(BaseModel):@@ -493,12 +488,22 @@ class PipelineApiExtras(BaseModel):msg = f"callback URL exceeds {MAX_CALLBACK_URL_LEN} chars"raise ValueError(msg)parsed = urlparse(url)+ # A URL with no scheme or no host is malformed, the shape of the field, which a plain `ValueError` refuses+ # with no security event. A scheme other than http or https, and a host the SSRF rule refuses, are a security+ # rule's refusals, so each states the class of destination its security event names.+ if not parsed.scheme:+ msg = "callback URL must be an absolute http or https URL, starting with its scheme"+ raise ValueError(msg)if parsed.scheme not in _ALLOWED_CALLBACK_SCHEMES:msg = f"callback URL scheme must be http or https (got {parsed.scheme!r})"+ raise CallbackUrlRefusedError(msg).with_security_target_class(SecurityTargetClass.SCHEME)+ host = parsed.hostname+ if not host:+ msg = "callback URL must name a host"raise ValueError(msg)- if _is_disallowed_host(parsed.hostname or ""):+ if _is_disallowed_host(host):msg = f"callback URL host {parsed.hostname!r} is not allowed (private/loopback/metadata addresses are blocked)"- raise ValueError(msg)+ raise CallbackUrlRefusedError(msg).with_security_target_class(classify_disallowed_host(host))return valuediff --git a/api/tests/unit/test_bundle_resources_routes.py b/api/tests/unit/test_bundle_resources_routes.pyindex 48dde6b0a..d41c3f1ac 100644--- a/api/tests/unit/test_bundle_resources_routes.py+++ b/api/tests/unit/test_bundle_resources_routes.py@@ -275,7 +275,8 @@ class TestBundleResourcesRoutes:("/etc/invoice.docx", "absolute"),("../invoice.docx", "'..' segment"),("templates\\invoice.docx", "backslash"),- ("C:invoice.docx", "colon"),+ ("C:invoice.docx", "drive"),+ ("invoice:v2.docx", "colon"),],)def test_a_path_that_cannot_name_a_file_inside_the_bundle_is_refused(self, path: str, reason: str) -> None:diff --git a/api/tests/unit/test_security_event_routes.py b/api/tests/unit/test_security_event_routes.pynew file mode 100644index 000000000..d4cf4771a--- /dev/null+++ b/api/tests/unit/test_security_event_routes.py@@ -0,0 +1,460 @@+"""A security refusal answered by the run routes fails under its security code and logs one security event naming the request.++`/execute` runs through the base runner, whose own boundary reports a refusal before the run; `/start` builds its job+itself, so a refusal before its run reaches the global handler unreported, which reports it there. A refusal the server+authors itself, a reserved object key, a callback URL aimed at a private address or a path leaving the method's tree,+is reported by the handler of its problem response. Either way the event is logged once, with the run's extras the body+carried and the request id, never a run id the caller chose, and never what was refused.+"""++import base64+import io+import json+import logging+import zipfile+from collections.abc import Generator+from contextlib import contextmanager+from typing import Any, NamedTuple++import pytest+from fastapi import FastAPI+from fastapi.testclient import TestClient+from mthds.protocol.problem.registry import CoreCode+from pipelex.errors.error_codes import ErrorCode+from pipelex.errors.registry.pipelex_codes import PipelexErrorCode+from pipelex.errors.security_event import SECURITY_EVENT_MESSAGE, SECURITY_EVENT_NAME+from pipelex.pipe_run.delivery_assignment import DeliveryAssignment+from pipelex.pipe_run.pipe_job import PipeJob+from pipelex.plugins.orchestrator_registry import OrchestratorRegistry+from pipelex.runtime_bridge.payloads import PipelexPipeDispatchAck, PipelexPipeRunOutput+from pipelex.tools.log.json_log_sink import JsonLogSink+from pytest_mock import MockerFixture++from pipelex_api.api_config import ApiConfig+from pipelex_api.exception_handlers import register_exception_handlers+from pipelex_api.middleware import REQUEST_ID_HEADER, RequestIdMiddleware+from pipelex_api.routes import router as api_router++_PIPELINE_NS = "pipelex_api.routes.pipelex.pipeline"+_MODE = "temporal"+_SECURITY_EVENT_LOGGER = "pipelex.errors.security_event"+_LOCAL_PATH = "/etc/passwd"+_REQUEST_ID = "req-security-event-1"+_METHODS_TOML = """\+[package]+address = "github.com/pipelex-tests/security-event"+version = "1.0.0"+description = "Prints a letter from a template file"+"""+# A method whose step names, as its template file, a path that climbs out of its tree to a file of the server's disk.+_TEMPLATE_OUTSIDE_THE_TREE_MTHDS = """\+domain = "security_event"+main_pipe = "print_letter"++[concept.Letter]+description = "A letter to print"++[concept.Letter.structure]+body = { type = "text", description = "The letter's text", required = true }++[concept.LetterFile]+description = "The letter as a printable file"+refines = "Document"++[pipe.print_letter]+type = "PipeDocGen"+description = "Print the letter"+inputs = { letter = "Letter" }+output = "LetterFile"+format = "docx"+template_file = "../../etc/passwd.docx"+"""+# Paths that leave the tree in a spelling other than a leading slash or a `..` between slashes, each with what of it the+# event must never carry: a backslash traversal, a drive and a network share.+_ESCAPING_PATHS: list[tuple[str, str, str]] = [+ ("..\\secrets\\key.pem", "key.pem", "backslash_dot_dot"),+ ("C:/Windows/win.ini", "win.ini", "drive"),+ ("\\\\server\\share\\key.pem", "key.pem", "network_share"),+]+_DESCRIBE_IMAGE_MTHDS = """\+domain = "security_event"+main_pipe = "describe_image"++[pipe.describe_image]+type = "PipeLLM"+description = "Describe an image"+inputs = { photo = "Image" }+output = "Text"+prompt = "Describe this image. $photo"+"""+++def _bundle_b64(entries: dict[str, str]) -> str:+ """A `bundle_b64` archive holding each entry under its name as written, a backslash included."""+ buffer = io.BytesIO()+ with zipfile.ZipFile(buffer, "w", zipfile.ZIP_DEFLATED) as archive:+ for name, text in entries.items():+ archive.writestr(name, text)+ return base64.b64encode(buffer.getvalue()).decode("ascii")+++def _b64(text: str) -> str:+ return base64.b64encode(text.encode("utf-8")).decode("ascii")+++class _ServerRefusal(NamedTuple):+ """A refusal of a security rule the server authors itself, and the security event it logs."""++ route: str+ body: dict[str, Any]+ error_type: str+ code: ErrorCode+ event_error_type: str+ target: str+ target_class: str | None+ refused_content: str+ """Something the request carried that the event must never hold."""+++class _UnreachedOrchestrator:+ """An async-capable orchestrator no refused request reaches, so a route refusing before the run dispatches nothing."""++ supports_fire_and_forget = True++ async def execute(self, *, pipe_job: PipeJob, delivery_assignment: DeliveryAssignment | None) -> PipelexPipeRunOutput: # noqa: ARG002+ msg = "a refused request must not reach the orchestrator"+ raise AssertionError(msg)++ async def start(self, *, pipe_job: PipeJob, delivery_assignment: DeliveryAssignment | None) -> PipelexPipeDispatchAck: # noqa: ARG002+ msg = "a refused request must not reach the orchestrator"+ raise AssertionError(msg)+++def _build_client(mocker: MockerFixture) -> TestClient:+ config = ApiConfig(orchestration_mode=_MODE, allow_request_orchestration_mode_override=False)+ mocker.patch(f"{_PIPELINE_NS}.get_api_config", return_value=config)+ mocker.patch(f"{_PIPELINE_NS}.get_orchestrator_registry", return_value=OrchestratorRegistry({_MODE: _UnreachedOrchestrator()}))+ app = FastAPI()+ app.include_router(api_router, prefix="/v1")+ register_exception_handlers(app)+ return TestClient(RequestIdMiddleware(app))+++def _run_body() -> dict[str, Any]:+ """A run body that passes every check but the one a test breaks, naming the organization in its extras."""+ return {+ "mthds_contents": [_DESCRIBE_IMAGE_MTHDS],+ "inputs": {"photo": "https://example.com/photo.png"},+ "storage_scope": "org_abc/mt_1/run_1",+ "read_scope": "org_abc",+ "analytics_groups": {"org_id": "org_abc"},+ }+++@contextmanager+def _security_event_lines() -> Generator[list[dict[str, Any]]]:+ """The security events logged in the block, as the `json` sink the server selects serializes them."""+ buffer = io.StringIO()+ handler = JsonLogSink(stream=buffer).handler+ logger = logging.getLogger(_SECURITY_EVENT_LOGGER)+ previous_level = logger.level+ logger.setLevel(logging.WARNING)+ logger.addHandler(handler)+ lines: list[dict[str, Any]] = []+ try:+ yield lines+ finally:+ logger.removeHandler(handler)+ logger.setLevel(previous_level)+ handler.close()+ lines.extend(json.loads(line) for line in buffer.getvalue().splitlines() if line)+++class TestSecurityEventRoutes:+ @pytest.mark.parametrize(+ ("route", "pipeline_run_id"),+ [+ pytest.param("/v1/execute", None, id="execute"),+ pytest.param("/v1/start", "run-security-event-1", id="start"),+ ],+ )+ def test_a_refused_input_answers_its_security_code_and_logs_one_event(+ self, mocker: MockerFixture, route: str, pipeline_run_id: str | None+ ) -> None:+ client = _build_client(mocker)+ body: dict[str, Any] = {+ "mthds_contents": [_DESCRIBE_IMAGE_MTHDS],+ "inputs": {"photo": _LOCAL_PATH},+ "storage_scope": "org_abc/mt_1/run_1",+ "read_scope": "org_abc",+ "analytics_groups": {"org_id": "org_abc"},+ }+ if pipeline_run_id is not None:+ body["pipeline_run_id"] = pipeline_run_id++ with _security_event_lines() as lines:+ response = client.post(route, json=body, headers={REQUEST_ID_HEADER: _REQUEST_ID})++ assert response.status_code == 422, response.text+ assert response.json()["code"] == CoreCode.SECURITY_INPUT_URL_REFUSED+ (line,) = lines+ assert line["message"] == SECURITY_EVENT_MESSAGE+ assert line["event.name"] == SECURITY_EVENT_NAME+ assert line["error.code"] == CoreCode.SECURITY_INPUT_URL_REFUSED+ assert line["error.domain"] == "input"+ assert line["security.target"] == "url"+ assert line["security.target_class"] == "scheme"+ assert line["run.extras.org_id"] == "org_abc"+ assert line["request_id"] == _REQUEST_ID+ # A run id the body names is the caller's claim, not a run of this runtime, before any run exists.+ assert "pipeline_run_id" not in line+ assert _LOCAL_PATH not in json.dumps(line)++ @pytest.mark.parametrize(+ "case",+ [+ pytest.param(+ _ServerRefusal(+ route="/v1/execute",+ body={"inputs": {"photo": {"__class__": "subprocess.Popen", "args": ["id"]}}},+ error_type="ReservedObjectKey",+ code=PipelexErrorCode.SECURITY_RESERVED_KEY,+ event_error_type="ReservedObjectKey",+ target="object_key",+ target_class=None,+ refused_content="__class__",+ ),+ id="reserved_object_key",+ ),+ pytest.param(+ _ServerRefusal(+ route="/v1/start",+ body={"callback_urls": ["http://169.254.169.254/latest/meta-data/"]},+ error_type="InvalidCallbackUrls",+ code=CoreCode.SECURITY_INPUT_URL_REFUSED,+ event_error_type="CallbackUrlRefusedError",+ target="url",+ target_class="metadata",+ refused_content="169.254.169.254",+ ),+ id="callback_metadata_endpoint",+ ),+ pytest.param(+ _ServerRefusal(+ route="/v1/start",+ body={"callback_urls": ["https://10.0.0.5/hooks/done"]},+ error_type="InvalidCallbackUrls",+ code=CoreCode.SECURITY_INPUT_URL_REFUSED,+ event_error_type="CallbackUrlRefusedError",+ target="url",+ target_class="private",+ refused_content="10.0.0.5",+ ),+ id="callback_private_address",+ ),+ pytest.param(+ _ServerRefusal(+ route="/v1/start",+ body={"callback_urls": ["http://METADATA.google.internal./computeMetadata/v1/"]},+ error_type="InvalidCallbackUrls",+ code=CoreCode.SECURITY_INPUT_URL_REFUSED,+ event_error_type="CallbackUrlRefusedError",+ target="url",+ target_class="metadata",+ refused_content="computeMetadata",+ ),+ id="callback_metadata_alias_spelled_otherwise",+ ),+ pytest.param(+ _ServerRefusal(+ route="/v1/start",+ body={"callback_urls": ["file:///etc/passwd"]},+ error_type="InvalidCallbackUrls",+ code=CoreCode.SECURITY_INPUT_URL_REFUSED,+ event_error_type="CallbackUrlRefusedError",+ target="url",+ target_class="scheme",+ refused_content="/etc/passwd",+ ),+ id="callback_scheme",+ ),+ pytest.param(+ _ServerRefusal(+ route="/v1/execute",+ body={"resources": {"../secrets/key.pem": base64.b64encode(b"x").decode("ascii")}},+ error_type="InvalidResourcePath",+ code=CoreCode.SECURITY_INPUT_PATH_REFUSED,+ event_error_type="BundleResourcePathError",+ target="path",+ target_class=None,+ refused_content="key.pem",+ ),+ id="resource_path_leaving_the_bundle",+ ),+ pytest.param(+ _ServerRefusal(+ route="/v1/execute",+ body={"mthds_contents": None, "files": {"main.mthds": _DESCRIBE_IMAGE_MTHDS, "../secrets/key.pem": "x"}},+ error_type="InvalidMethodFiles",+ code=CoreCode.SECURITY_INPUT_PATH_REFUSED,+ event_error_type="MethodTreeRefusedError",+ target="path",+ target_class=None,+ refused_content="key.pem",+ ),+ id="file_leaving_the_method_tree",+ ),+ pytest.param(+ _ServerRefusal(+ route="/v1/execute",+ body={"mthds_contents": None, "files": {"main.mthds": _DESCRIBE_IMAGE_MTHDS, "/etc/key.pem": "x"}},+ error_type="InvalidMethodFiles",+ code=CoreCode.SECURITY_INPUT_PATH_REFUSED,+ event_error_type="MethodTreeRefusedError",+ target="path",+ target_class=None,+ refused_content="key.pem",+ ),+ id="absolute_file_path",+ ),+ *[+ pytest.param(+ _ServerRefusal(+ route="/v1/execute",+ body={"mthds_contents": None, "files": {"main.mthds": _DESCRIBE_IMAGE_MTHDS, path: "x"}},+ error_type="InvalidMethodFiles",+ code=CoreCode.SECURITY_INPUT_PATH_REFUSED,+ event_error_type="MethodTreeRefusedError",+ target="path",+ target_class=None,+ refused_content=refused_content,+ ),+ id=f"file_{case_id}",+ )+ for path, refused_content, case_id in _ESCAPING_PATHS+ ],+ *[+ pytest.param(+ _ServerRefusal(+ route="/v1/execute",+ body={"mthds_contents": None, "bundle_b64": _bundle_b64({"main.mthds": _DESCRIBE_IMAGE_MTHDS, path: "x"})},+ error_type="InvalidMethodFiles",+ code=CoreCode.SECURITY_INPUT_PATH_REFUSED,+ event_error_type="MethodTreeRefusedError",+ target="path",+ target_class=None,+ refused_content=refused_content,+ ),+ id=f"archive_member_{case_id}",+ )+ for path, refused_content, case_id in _ESCAPING_PATHS+ ],+ *[+ pytest.param(+ _ServerRefusal(+ route="/v1/execute",+ body={"resources": {path: _b64("x")}},+ error_type="InvalidResourcePath",+ code=CoreCode.SECURITY_INPUT_PATH_REFUSED,+ event_error_type="BundleResourcePathError",+ target="path",+ target_class=None,+ refused_content=refused_content,+ ),+ id=f"resource_{case_id}",+ )+ for path, refused_content, case_id in _ESCAPING_PATHS+ ],+ ],+ )+ def test_a_refusal_the_server_authors_logs_one_event_without_its_content(self, mocker: MockerFixture, case: _ServerRefusal) -> None:+ """The server's own refusal of a security rule keeps its response and logs the event its code calls for."""+ client = _build_client(mocker)+ request_body = {**_run_body(), "pipeline_run_id": "run-chosen-by-the-caller", **case.body}+ request_body = {key: value for key, value in request_body.items() if value is not None}++ with _security_event_lines() as lines:+ response = client.post(case.route, json=request_body, headers={REQUEST_ID_HEADER: _REQUEST_ID})++ assert response.status_code == 422, response.text+ assert response.json()["error_type"] == case.error_type+ (line,) = lines+ assert line["message"] == SECURITY_EVENT_MESSAGE+ assert line["event.name"] == SECURITY_EVENT_NAME+ assert line["error.code"] == case.code+ assert line["error.domain"] == "input"+ assert line["error.type"] == case.event_error_type+ assert line["security.target"] == case.target+ assert line.get("security.target_class") == case.target_class+ assert line["run.extras.org_id"] == "org_abc"+ assert line["request_id"] == _REQUEST_ID+ assert "pipeline_run_id" not in line+ assert case.refused_content not in json.dumps(line)++ def test_a_method_naming_a_file_outside_its_tree_fails_under_its_security_code_and_logs_one_event(self, mocker: MockerFixture) -> None:+ """A path the method names that climbs out of its tree is the method's security refusal, never a verdict about the method."""+ client = _build_client(mocker)+ body = {+ **_run_body(),+ "mthds_contents": None,+ "pipe_code": "print_letter",+ "inputs": {"letter": {"concept": "security_event.Letter", "content": {"body": "Dear reader"}}},+ "files": {"METHODS.toml": _METHODS_TOML, "main.mthds": _TEMPLATE_OUTSIDE_THE_TREE_MTHDS},+ }++ with _security_event_lines() as lines:+ response = client.post("/v1/execute", json=body, headers={REQUEST_ID_HEADER: _REQUEST_ID})++ assert response.status_code == 422, response.text+ assert response.json()["code"] == CoreCode.SECURITY_METHOD_PATH_REFUSED, response.text+ (line,) = lines+ assert line["event.name"] == SECURITY_EVENT_NAME+ assert line["error.code"] == CoreCode.SECURITY_METHOD_PATH_REFUSED+ assert line["error.domain"] == "method"+ assert line["error.type"] == "PipeDocGenFactoryError"+ assert line["security.target"] == "path"+ assert line["run.extras.org_id"] == "org_abc"+ assert line["request_id"] == _REQUEST_ID+ assert "passwd" not in json.dumps(line)++ @pytest.mark.parametrize(+ ("route", "body", "error_type"),+ [+ pytest.param("/v1/start", {"callback_urls": [""]}, "InvalidCallbackUrls", id="empty_callback_url"),+ pytest.param("/v1/start", {"callback_urls": ["https://hooks.example.com/" + "a" * 5000]}, "InvalidCallbackUrls", id="long_callback_url"),+ pytest.param("/v1/start", {"callback_urls": ["hooks.example.com/done"]}, "InvalidCallbackUrls", id="callback_url_without_scheme"),+ pytest.param("/v1/start", {"callback_urls": ["http:///done"]}, "InvalidCallbackUrls", id="callback_url_without_host"),+ pytest.param(+ "/v1/execute",+ {"mthds_contents": None, "files": {"main.mthds": _DESCRIBE_IMAGE_MTHDS, "notes//a.txt": "x"}},+ "InvalidMethodFiles",+ id="empty_segment_in_a_file_path",+ ),+ pytest.param(+ "/v1/execute",+ {"mthds_contents": None, "files": {"main.mthds": _DESCRIBE_IMAGE_MTHDS, "notes\\a.txt": "x"}},+ "InvalidMethodFiles",+ id="backslash_in_a_file_path",+ ),+ pytest.param(+ "/v1/execute",+ {"mthds_contents": None, "files": {"main.mthds": _DESCRIBE_IMAGE_MTHDS, "notes:v2.txt": "x"}},+ "InvalidMethodFiles",+ id="colon_in_a_file_path",+ ),+ pytest.param("/v1/execute", {"resources": {"notes\\a.txt": _b64("x")}}, "InvalidResourcePath", id="backslash_in_a_resource_path"),+ pytest.param("/v1/execute", {"resources": {"notes:v2.txt": _b64("x")}}, "InvalidResourcePath", id="colon_in_a_resource_path"),+ pytest.param("/v1/execute", {"resources": {"": _b64("x")}}, "InvalidResourcePath", id="empty_resource_path"),+ ],+ )+ def test_a_refusal_of_the_request_shape_logs_no_event(self, mocker: MockerFixture, route: str, body: dict[str, Any], error_type: str) -> None:+ """A malformed field is the caller's mistake, not an attempt to reach what a security rule forbids."""+ client = _build_client(mocker)+ request_body = {key: value for key, value in {**_run_body(), **body}.items() if value is not None}++ with _security_event_lines() as lines:+ response = client.post(route, json=request_body, headers={REQUEST_ID_HEADER: _REQUEST_ID})++ assert response.status_code == 422, response.text+ assert response.json()["error_type"] == error_type+ assert lines == []diff --git a/docs/api-server/logging.md b/docs/api-server/logging.mdindex 54423f16d..e96945e6b 100644--- a/docs/api-server/logging.md+++ b/docs/api-server/logging.md@@ -73,6 +73,20 @@ Every field in that table is a **record attribute**, which is a different thingDisposition follows the HTTP status, not the error domain: a `4xx` is a caller mistake and logs at `WARNING` without a traceback; a `5xx` is a server fault and logs at `ERROR` with one. See [Error Responses](error-responses.md) for the response side of the same failure.+## Security events++A request a security rule refused writes one more line beside its error line: the **security event**, so that whoever runs the server learns that someone tried to reach what a rule forbids, a private address, another tenant's stored file, a path on the server's disk, a template reaching for a private name. It is a record of its own kind, which a query or an alert selects without reading any text: its `logger` is `pipelex.errors.security_event`, its `message` is `A security rule refused a request`, and its `event.name` is `pipelex.security.refusal`. It is written at `WARNING`, with no traceback.++```json+{"time": "2026-10-17T09:12:03.512Z", "severity": "WARNING", "logger": "pipelex.errors.security_event", "message": "A security rule refused a request", "request_id": "01M2TCP02W04RZG6DTM8AR508C", "event.name": "pipelex.security.refusal", "error.code": "security.input_url_refused", "error.domain": "input", "error.type": "UriReadRefusedError", "security.target": "url", "security.target_class": "scheme", "run.extras.org_id": "org_abc"}+```++Its keys name the refusal and never its content: the refusal's code under `error.code`, the code's domain under `error.domain`, the class of the error under `error.type`, the kind of thing refused under `security.target`, the class of destination under `security.target_class` when the rule could tell it, the run and the request under `pipeline_run_id` and `request_id`, and each of the run's extras, the body's `analytics_groups`, under a key of its own, `run.extras.<key>`. Never the URL, the path, the template or an input value, which are the caller's content or the method's. The Pipelex documentation's [error model](../under-the-hood/error-model.md#security-refusals-and-the-security-event) lists every value.++The server writes it for the refusals its own boundary decides too, which keep the response they always had: a run body carrying a reserved object key (`pipelex.security.reserved_key`, `security.target` `object_key`), a callback URL whose scheme is not http or https or whose host is a loopback, private, link-local or cloud metadata address (`security.input_url_refused`, with the class of destination), and a resource path or a key of a method tree, `files` or a `bundle_b64` member, that leaves the method: absolute, naming a drive or a network share, or climbing out with a `..` segment, read with either separator, so `..\secrets\key.pem`, `C:/Windows/win.ini` and `\\server\share` are among them (`security.input_path_refused`). A refusal of the shape of a field, an empty callback URL, one with no scheme or no host, or a path that is empty or holds an empty segment, a backslash or a colon, is the caller's mistake and writes no event, under `resources` as under `files`. The event never carries a `pipeline_run_id` a body names: before any run exists, that id is the caller's claim.++A refusal is written once, by the process that raised it, whichever part of that process saw it: during a run executed in-process, before the run while its inputs are read or its method loads, while an orchestration plugin prepares a run in this process before it dispatches it, or in the handler answering the request. A run dispatched to a worker under a distributed `orchestration_mode` is refused in the worker's process, so its event is in the worker's logs, written there by its orchestration plugin, and never in this server's: the failure the server receives back, or never receives for a `/start` run, is not logged a second time. The Pipelex documentation's [error model](../under-the-hood/error-model.md#security-refusals-and-the-security-event) states the rule an orchestration plugin follows.+## Authentication linesThe error line above is the record of every refused request, a `401` included, at `WARNING`. The authentication check before it says why it refused, at a level that depends on who has to act:diff --git a/docs/api-server/pipe-run.md b/docs/api-server/pipe-run.mdindex 5bb0217af..726968cf6 100644--- a/docs/api-server/pipe-run.md+++ b/docs/api-server/pipe-run.md@@ -183,6 +183,8 @@ Start a pipeline execution and get its `pipeline_run_id` back with a `202` ack.}```+A callback URL is refused before anything runs with a **422** `InvalidCallbackUrls` when its scheme is not `http` or `https`, or when its host is one the runtime's SSRF rule refuses: `localhost`, a cloud metadata name such as `metadata.google.internal` in any case and with or without a trailing dot, or a literal address that is not publicly routable, a loopback, private, link-local, carrier-grade NAT or cloud metadata address among them. It is the rule the delivery applies again when the webhook fires, resolving the name, and such a refusal also writes a [security event](logging.md#security-events). A callback URL that is not a URL at all, empty, longer than the limit, with no scheme or naming no host, is refused with the same 422 and writes none.+When the pipeline completes, each URL in the list receives a POST carrying:- A completion notice in the body, not the run's output: `pipeline_run_id`, the delivery `state` (`"COMPLETED"` or `"FAILED"`, also sent under the legacy `status` key), `result_url` (when results were stored: the storage key prefix under which the result files sit, not a fetchable URL) and `error` on failure, the run's problem document in the view of whoever started the run (see [Error Responses](error-responses.md#async-callbacks-webhook-payload)). The receiver reads the results from storage under that prefix.diff --git a/docs/api-server/pipe-validate.md b/docs/api-server/pipe-validate.mdindex 92fd1083b..7c3301f40 100644--- a/docs/api-server/pipe-validate.md+++ b/docs/api-server/pipe-validate.md@@ -179,7 +179,7 @@ A bundle sent as content has no directory, so a file one of its steps reads from}```-- **Each key is the path a step names**, relative to the bundle and written with forward slashes, exactly as in its `template_file`. `.` segments and repeated slashes collapse, so `./templates/invoice.docx` names the same file; every bundle of the request is read as sitting in one directory. An empty or absolute path, a `..` segment, a backslash, a colon, or two keys naming the same file is a **422** `InvalidResourcePath`, refused before any value is decoded. A method sent as a tree may hold the file instead, beside the bundle whose step names it; beside a tree, each key of `resources` joins it at that path from the tree's root, and a file the tree already holds must have the same bytes (see [Pipe Run](pipe-run.md#sending-a-method-as-a-tree)).+- **Each key is the path a step names**, relative to the bundle and written with forward slashes, exactly as in its `template_file`. `.` segments and repeated slashes collapse, so `./templates/invoice.docx` names the same file; every bundle of the request is read as sitting in one directory. An empty or absolute path, a drive or a network share, a `..` segment, a backslash, a colon, or two keys naming the same file is a **422** `InvalidResourcePath`, refused before any value is decoded. A method sent as a tree may hold the file instead, beside the bundle whose step names it; beside a tree, each key of `resources` joins it at that path from the tree's root, and a file the tree already holds must have the same bytes (see [Pipe Run](pipe-run.md#sending-a-method-as-a-tree)).- **Each value is the file's bytes** in standard base64 (RFC 4648, with padding). A value that does not decode is a **400** `InvalidBase64`.- **Beside inline contents, every file must be one a step names.** A file no step of the bundle reads is a **422** problem with the code `request.invalid`, whose operator view's message lists the unnamed paths: a misspelled key fails loudly rather than leaving its step without its file. This is a request error, never an `is_valid: false` verdict. Beside a tree, a file no step names is kept like any other file of the tree.- **The sizes are bounded**: at most `MAX_RESOURCES_PER_REQUEST` files (32 by default), each at most `MAX_RESOURCE_FILE_KIB` once decoded (5 MiB by default), and at most `MAX_RESOURCES_TOTAL_KIB` across the request (10 MiB by default). A request over any of them is a **413** `PayloadTooLarge`, an oversized file refused on its encoded length before it is decoded. The whole body stays under `MAX_REQUEST_BODY_MIB` as well.diff --git a/docs/building-methods/pipes/pipe-operators/PipeDocGen.md b/docs/building-methods/pipes/pipe-operators/PipeDocGen.mdindex 4de6d77d6..561e87200 100644--- a/docs/building-methods/pipes/pipe-operators/PipeDocGen.md+++ b/docs/building-methods/pipes/pipe-operators/PipeDocGen.md@@ -81,7 +81,7 @@ To print the same invoice on the plugin's `pipelex-pdf`, the step names it: `mod| `format` | string | `pdf`, `xlsx`, `docx` or `pptx`. It sets the file's type and suffix. | Yes || `model` | string | The document engine that prints the file, a `doc_gen` model, alias or preset, such as `reportlab-pdf` or, with the plugin, `pipelex-pdf`. It defaults to the model deck's engine for the format and source. | No || `template` | string | An inline HTML and Jinja2 template, for `pdf` only. A `pdf` with a template is printed by the plugin. | No |-| `template_file` | string | A template file relative to the bundle's file, written with forward slashes: `.html` for `pdf`, `.xlsx`, `.docx` or `.pptx`. An empty or absolute path, a URL, a backslash and a colon are refused. When the bundle is read from a folder, the file is the one its [method tree](../../method-trees.md#file-references) holds at that path, a `..` climbing to another folder of the tree included, and a file the tree keeps out, such as a gitignored one, is refused; a bundle file run alone with no `METHODS.toml` at or above it reads no other file. When the bundle is loaded from a string, the file is one of the files sent with it, each keyed by a path inside the bundle, so a path no such file could be keyed by, one holding a `..` segment or a NUL character, or naming no file, is refused when the bundle is parsed, as a validation error at the step that does not repeat the path. It excludes `template`. `pptx` requires one. | No |+| `template_file` | string | A template file relative to the bundle's file, written with forward slashes: `.html` for `pdf`, `.xlsx`, `.docx` or `.pptx`. An empty or absolute path, a URL, a backslash and a colon are refused. When the bundle is read from a folder, the file is the one its [method tree](../../method-trees.md#file-references) holds at that path, a `..` climbing to another folder of the tree included, and a file the tree keeps out, such as a gitignored one, is refused; a path that climbs above the tree's root is refused as a security matter, `security.method_path_refused`, which fails validation and the run rather than reporting a validation error, and logs a security event; a bundle file run alone with no `METHODS.toml` at or above it reads no other file. When the bundle is loaded from a string, the file is one of the files sent with it, each keyed by a path inside the bundle, so a path no such file could be keyed by, one holding a `..` segment or a NUL character, or naming no file, is refused when the bundle is parsed, as a validation error at the step that does not repeat the path. It excludes `template`. `pptx` requires one. | No || `filename` | string | A Jinja expression over the inputs for the file's name, without its suffix, such as `invoice-{{ invoice.number }}`. It defaults to the pipe's code. The rendered name is made into one every storage reader serves, the hosted API's included, spelled in ASCII with hyphens for the rest: see [The file's name](#the-files-name). | No |### The file's namediff --git a/docs/tools/logging.md b/docs/tools/logging.mdindex 691db39e9..e45911458 100644--- a/docs/tools/logging.md+++ b/docs/tools/logging.md@@ -422,6 +422,12 @@ for (operation, model_handle), group in groups.items():A test runs a small method live, with stand-in workers answering every model call on two models, and computes the cost and the latency of each operation and model from the `json` output alone.+## The security event++A refusal a security rule makes, an address the SSRF guard blocks, a stored file outside the run's read scope, a path escaping its root, a template the sandbox refuses, fails the run under a `security.` code and logs one **security event** beside it. The event is a record of its own kind, on its own logger, `pipelex.errors.security_event`, at **WARNING**, with the fixed message `A security rule refused a request` and the field `event.name` set to `pipelex.security.refusal`, so a host routes it by its logger or by that field without reading any text: to an alert on a hosted deployment, to the client's own sinks on-premises. Its fields name the refusal's code and domain, the kind and class of target refused, the run, the request and each of the run's extras as `run.extras.<key>`, and never the URL, the path, the template or any input value; it carries no traceback. A refusal is logged once however many boundaries it crosses. The [error model](../under-the-hood/error-model.md#security-refusals-and-the-security-event) lists every field and value, and where the event is logged.++The event follows its logger's level like any other record, so a host that raises `pipelex` above `WARNING` with `package_log_levels` drops it too; one that quiets the runtime and still wants its security events sets `pipelex-errors-security_event = "WARNING"` beside that key.+## Log-call conventionsPipelex's own log calls, in `pipelex/` and in the API server's `api/pipelex_api/`, follow these conventions, so that a line written today can be grouped, counted and filtered in a log store tomorrow. The [log-call guard](../contribute/log-calls.md), `make check-log-calls`, holds every call to the ones that can be read off the source: the fixed message at INFO and above, the absence of markup, the mechanical rules of [wording a message](#wording-a-message), and no handled exception spliced into the text; the rest is for review. Code of your own that logs through `log` is welcome to follow them too.diff --git a/docs/under-the-hood/error-model.md b/docs/under-the-hood/error-model.mdindex 8d41adbbd..ad3a6c135 100644--- a/docs/under-the-hood/error-model.md+++ b/docs/under-the-hood/error-model.md@@ -443,6 +443,37 @@ See [Runtime Bridge & Transport](./runtime-bridge-and-transport.md) for the boun---+## Security Refusals and the Security Event++A refusal a security rule makes is a failed run and a security event. The run fails under a code of the `security.` family of the core registry, or of Pipelex's own `pipelex.security.` family, and that code's domain says who caused the refusal: `security.input_url_refused` and `security.input_path_refused` are the caller's, `security.method_url_refused`, `security.method_path_refused`, `security.template_refused`, `security.schema_refused` and `security.integrity_mismatch` are the method's, and `security.provider_url_refused` is the runtime's, a URL a model's answer named. A refusal is coded where it is raised, as every error is, and every wrapper above it takes its cause's code, so the failure a host receives reports the security code however many boundaries it crossed. A security refusal, a `SecurityError` or any error its raise site coded in a security family, such as a step's template file that climbs out of its method tree, is never absorbed into a verdict or a domain answer on the way. One refusal is coded otherwise on purpose: the sandbox refusing one of Pipelex's own templates is our bug, `runtime.unexpected`, and so it is no security refusal and logs no event.++Beside the failure, the deployment logs one **security event**, so that whoever runs it learns that someone tried to reach what a rule forbids. It is a structured log record of its own kind, routable without reading its text: it is logged at warning level on its own logger, `pipelex.errors.security_event`, with the fixed message `A security rule refused a request`, and its `event.name` field is `pipelex.security.refusal`. It goes through the deployment's log sinks like any other record, so a hosted deployment routes it to its alerts and an on-premises one receives it in its own sinks, and nothing leaves the deployment because of it. Its fields name the refusal and never its content:++| Field | What it carries |+|-------|-----------------|+| `event.name` | `pipelex.security.refusal` |+| `error.code` | The refusal's code, such as `security.method_url_refused` |+| `error.domain` | The code's domain, who caused it: `input`, `method` or `runtime` |+| `error.type` | The class of the error that carries or implies the code, such as `SsrfBlockedError`, never a wrapper above it; for a refusal the API server authors with no error class behind it, the `error_type` its response names, such as `ReservedObjectKey` |+| `security.target` | The kind of thing refused, which the code decides: `url`, `path`, `template`, `schema`, `dependency`, `object_key` or `result_location` |+| `security.target_class` | The class of destination refused, when the raise site knows it: `loopback`, `private`, `link_local`, `metadata` (a cloud metadata endpoint), `scheme` (a scheme the run does not read, a `file://` URL or a path on the server's disk among them) or `foreign_storage_key` (a storage key outside the run's read scope) |+| `pipeline_run_id`, `request_id` | The run, only when the runtime assigned it one, never a run id a request body names before any run exists, and the request, when there is one |+| `run.extras.<key>` | Each of the run's opaque extras, one field per key, which the hosted plane fills with the organization |++Never the URL, the path, the template, the schema, a prompt or an input value, nor the error's message, which may quote any of them, and never a traceback: the failure itself is reported where it is handled, as any other failure is. The runtime does not know organizations, so the event names one only through the run's extras, which it carries without reading.++The class of destination is stated where the refusal is decided, with `PipelexError.with_security_target_class()`. The SSRF guard classifies the host it refused, or the first disallowed address a name resolved to, through `classify_disallowed_host` in `pipelex/tools/network/host_rules.py`, and a read the run's read scope refuses takes its class from its reason, a storage key of another tenant or a scheme the run does not read. A refusal that states no class, such as a schema or a template refusal, has no `security.target_class`.++**Where it is logged.** `pipelex/errors/security_event.py` holds the event, and **the process that raises a refusal logs it**: whoever executes a run owns the events of the refusals raised in it, and the side that dispatched the run to another process never logs a failure it receives from there. Three boundaries call it, each with what it knows of the run:++- the pipe run, `PipeRun`, for a refusal raised while it executes, whether one of its pipes raised it or its tail did, such as a webhook the SSRF guard refuses at fire time, with the run's identity, its extras included;+- the runner, `PipelexMTHDSProtocol.execute`, for a refusal before the run exists, at the input seam or while the method loads, with the runner's extras, and for a run's refusal raised in this process that the pipe run left unreported, with the run's identity: a pipe run that hands the job to another process can be refused here first, such as an input URL the SSRF guard refuses while it uploads the inputs before it dispatches. A refusal the pipe run logged, `PipeRun` in this process, is found reported, and so is a failure an orchestrator rebuilt from a remote worker's report, which it marks where it rebuilds it;+- the API server's handler of every `PipelexError`, for a refusal answered before any run, and its handler of the problem responses it authors itself, for the refusals of a security rule it decides at its own boundary: a reserved object key in a run body (`pipelex.security.reserved_key`), a callback URL whose scheme or host the SSRF rule refuses (`security.input_url_refused`), and a resource path or a key of a method tree that leaves the method (`security.input_path_refused`). Both log with the extras the request body carried and no run id. One rule tells a path that leaves its root from one of the wrong shape, `root_escape_refusal` in `pipelex/core/bundle_resources/bundle_resource.py`: a path that is absolute, names a drive or a network share, or holds a `..` segment, read with either separator, leaves it, and any other refused path, empty or holding a backslash or a colon, is `request.invalid` and logs no event. It decides for a key of `resources`, a file submitted beside a method, and a key of a method tree, which the standard's reader refuses without saying why.++A refusal is reported once per process, however many of these boundaries it crosses: the event marks the errors of the chain it reported, and a boundary that receives the same failure again, wrapped, finds the mark and logs nothing. The refusal is the error of the chain that carries or implies the code the failure reports, so a wrapper chained to it with `from`, which takes its code, restates it, while a new refusal is reported on its own, even of the same code, whether it was raised while a reported one was being handled or chained to one with `from`: a run whose input URL the SSRF guard refused, and whose webhook it then refuses, logs two events. Across processes, the rule above keeps it to one event whatever the transport rebuilds: a distributed orchestrator logs the refusal on the worker that raised it, at the boundary where the failure leaves that process, and marks every error it rebuilds from another process's report with `mark_security_event_emitted()`, so that neither its own boundaries nor the runner log it again, since the runner logs a run failure it finds unreported; [Orchestrator Plugins](./orchestrator-plugins.md#security-refusals-are-logged-where-they-are-raised) states what an orchestrator owes. A `/start` run, whose failure the dispatching side never receives, is covered the same way. A host that decides a refusal itself, such as a gateway's refusal of a private address or a tenant-boundary refusal of its own, logs the same record with `log_security_event()`. A program embedding the kernel without the runner (see [The Pipelex Kernel](./pipelex-kernel.md)) calls `emit_security_event()` on the refusals it receives.++---+## Interfaces### CLI@@ -533,9 +564,9 @@ Exception├── PipelexConfigError → one of pipelex.config.invalid, .syntax, .file_missing; gives its migration as a fact├── PipelexSetupError → inherits its cause's code, else runtime.misconfigured├── SecurityError base_exceptions.py — abstract; the policy refusals never absorbed into a domain answer- │ ├── UriReadRefusedError / SsrfBlockedError ← security.input_url_refused, or the code the site that knows the URL's author chooses- │ ├── UnsafeSchemaError → security.schema_refused- │ └── Jinja2TemplateSecurityError ← the template sandbox's refusal at render time, security.template_refused+ │ ├── UriReadRefusedError / SsrfBlockedError → security.input_url_refused, .method_url_refused (and .provider_url_refused)+ │ ├── UnsafeSchemaError → security.schema_refused+ │ └── Jinja2TemplateSecurityError → security.template_refused, else runtime.unexpected on Pipelex's own template├── CogtError cogt/exceptions.py — abstract; error_category, provider_metadata, problem_facts()│ ├── LLMCompletionError ← a set the worker's classification picks from│ ├── ImgGenGenerationError ← a set the worker's classification picks from@@ -544,7 +575,7 @@ Exception└── ... (one exceptions.py per package) ...```-`SecurityError` is the abstract base class of the policy refusals that are never absorbed into a domain answer: `UriReadRefusedError` (a read outside the run's scope), `SsrfBlockedError` (an outbound request the SSRF guard blocks), `UnsafeSchemaError` (a schema carrying Python code-generation extensions) and `Jinja2TemplateSecurityError` (a template the sandbox refuses at render time). The handlers that turn a refusal of the caller's input into a verdict, the bundle-validation cascade and the library load, let a `SecurityError` through as it is, and a host that treats this family alike, a durable-execution plugin marking it non-retryable for instance, names that base class. Not every refusal belongs to it: the template sandbox's render-budget refusal, `Jinja2TemplateBudgetError`, is a `ToolError` coded `pipe.template_budget_exceeded`, and `TemplateUrlRefusedError`, coded `security.method_path_refused`, derives from `PipelexError` directly, so each is handled by its code like any other refusal.+`SecurityError` is the abstract base class of the policy refusals that are never absorbed into a domain answer: `UriReadRefusedError` (a read outside the run's scope), `SsrfBlockedError` (an outbound request the SSRF guard blocks), `UnsafeSchemaError` (a schema carrying Python code-generation extensions) and `Jinja2TemplateSecurityError` (a template the sandbox refuses at render time). The handlers that turn a refusal of the caller's input into a verdict, the bundle-validation cascade and the library load, let a `SecurityError` through as it is, and the cascade also lets through any refusal coded in a security family, whatever its class (`is_security_refusal` in `pipelex/errors/security_event.py`), and a host that treats this family alike, a durable-execution plugin marking it non-retryable for instance, names that base class. Not every refusal belongs to it: the template sandbox's render-budget refusal, `Jinja2TemplateBudgetError`, is a `ToolError` coded `pipe.template_budget_exceeded`, and `TemplateUrlRefusedError`, coded `security.method_path_refused`, derives from `PipelexError` directly, so each is handled by its code like any other refusal.### Factory-time vs Runtime@@ -603,6 +634,8 @@ InferenceErrorCategory.TRANSIENT.is_retryable # True — only TRANSIENT| `pipelex/errors/error_codes.py` | `ErrorCode`, `error_code_spec()`: what the registries declare about each code, as generated tables, and `first_fault_code()` || `pipelex/errors/bundle_validation_failure.py` | `BundleValidationFailure`, a bundle validator's invalid arm: the refusal's problem and its validation items || `pipelex/errors/foreign_raises.py` | The scan behind the foreign-raise check |+| `pipelex/errors/security_event.py` | The security event: `emit_security_event()`, `emit_security_event_for_run()`, `log_security_event()`, `mark_security_event_emitted()`, `is_security_refusal()`, and the kind of target each security code refuses |+| `pipelex/errors/security_target.py` | `SecurityTargetKind` and `SecurityTargetClass`, the vocabulary of what a security refusal refused || `pipelex/errors/retired_error_pages.py` | The static redirects of the retired per-class error pages || `pipelex/errors/registry/pipelex-error-registry.yaml` | The Pipelex error registry, the single source of the `pipelex.` codes; its snapshot, its typed codes (`pipelex_codes.py`) and its pages are generated from it || `pipelex/errors/registry/registry_document.py` | The typed model of an MTHDS error registry, and `load_pipelex_error_registry()` |@@ -651,6 +684,7 @@ InferenceErrorCategory.TRANSIENT.is_retryable # True — only TRANSIENT| Unknown or ambiguous entry `pipe_code` (a CLI argument, a run request's field, the `--pipe` / `pipe_ref` slice selector of bundle validation) | `EntryPipeNotFoundError` / `EntryPipeAmbiguousError` → `UserAction(CHANGE_INPUT)`; `request.pipe_not_found`, `request.pipe_unspecified` or `request.pipe_ambiguous` → **HTTP 422**. The in-body lookups (`get_optional_pipe` / `get_required_pipe`) keep raising `PipeNotFoundError` / `PipeLibraryError`, whose codes are the method's: a ref written inside a bundle is not the caller's input || A template naming a filter or a test that does not exist, found only when it renders | `Jinja2TemplateInvalidError` at the pipe, naming the filter or test and none of the template: `method.template_invalid` → **HTTP 422** for a method's `PipeCondition` expression, `PipeLLM` prompt or `PipeDocGen` template, and `runtime.unexpected` for one of Pipelex's own templates || A URL or a storage key refused by the SSRF guard or the run's read scope | A `security.*` code saying whose the URL was: `security.input_url_refused` for one of the run's inputs, the default when the guard cannot tell, `security.method_url_refused` for one the method names and for a read refused inside a step, whose inputs were authorized when they were normalized, both → **HTTP 422**, and `security.provider_url_refused` for one in a model's generated answer, the runtime's → HTTP 502; a traversal in a storage key the caller supplied is `security.input_path_refused`, which `PipelineInputContentError` takes from its cause |+| Any refusal coded in the `security.` or `pipelex.security.` family | The run fails under that code, and the deployment logs one security event on `pipelex.errors.security_event`, naming the code, its domain, the kind and class of target refused, the run, the request and the run's extras, never the target or any content; a refusal crossing several boundaries is logged once || A method of the host's own library directories refuses to load | `HostLibraryLoadRefusalError`, `pipelex.config.invalid` → HTTP 500, its message the operator's alone; the same refusal in the caller's own content stays the caller's verdict || Wrapper exception (no own category) | Takes its cause's code, and its parameters and facts from the chain || A pipe fails during a run | `PipeRouterError` → `PipelineExecutionError`, both answering the root fault's problem, its `location` naming the failing pipe and its path from the entry pipe; their own messages, for the operator, are the root fault's prefixed with `Pipe '<code>' failed (<path>): ` |diff --git a/docs/under-the-hood/orchestrator-plugins.md b/docs/under-the-hood/orchestrator-plugins.mdindex aed3cb53a..365af88fd 100644--- a/docs/under-the-hood/orchestrator-plugins.md+++ b/docs/under-the-hood/orchestrator-plugins.md@@ -56,6 +56,10 @@ registrar.add_orchestrator(mode="temporal", orchestrator=TemporalOrchestrator())Constructing the orchestrator instance must be **import-light** — it must not import the host-runtime SDK (`temporalio`, …) at module scope or in its `__init__`. The heavy import happens lazily inside `execute` (and a friendly `MissingOrchestratorError` is raised there if the mode's extra is absent), so discovering and registering the plugin never pulls the SDK. This is what keeps boot import-light even on a process that will never use the mode.+### Security refusals are logged where they are raised++A refusal a security rule makes during a run logs one [security event](./error-model.md#security-refusals-and-the-security-event), in the process that raised it. The runner that dispatched the job logs a run failure only when nobody reported it: a refusal `PipeRun` raised and logged in this process is found reported, and so must be a failure rebuilt from a worker's report, whose worker logged it, while a refusal raised in this process that the orchestrator did not log, such as an input URL the SSRF guard refuses while it uploads the inputs before it dispatches, is logged by the runner, naming the run. A `/start` run, whose failure the dispatching side never sees, is covered by the same rule. An orchestrator that runs the job in this process to its end, as core's DIRECT does through `PipeRun`, owes nothing more. One that runs it in another process owes two things there: it calls `emit_security_event_for_run` (or `emit_security_event` with the run's identity) on every failure at the boundary where it leaves the process that raised it, before converting it for the wire, and it calls `mark_security_event_emitted` on every error it rebuilds from another process's report, a batch branch's included, on the outermost error of the chain it raises, so that neither its own boundaries nor the runner log a refusal a second time: an error the orchestrator raises unmarked is one the runner logs, and an error carrying a code of its own chained above a marked one is a new refusal. Both functions live in `pipelex.errors.security_event`.+### A missing orchestrator is a generic, plugin-decoupled errorA token with no registered orchestrator (its plugin is not installed) raises `MissingOrchestratorError(mode=...)` (`pipelex/runtime_bridge/exceptions.py`). The message names the token but **no orchestrator** — *"No orchestrator is registered for orchestration mode '{mode}'; is its plugin installed?"* — so core stays fully decoupled from its plugins (it never spells out the Temporal or Mistral Workflows backend). The one special case is the core `"direct"` token: its orchestrator is always present, so a miss there reports a boot/discovery fault, `runtime.unexpected`. The message rides only the operator view; a caller reads the sentences of the error's code: `pipelex.plugin.not_found` for a mode the deployment configured, and `runtime.unsupported` for a mode the request named that this runtime does not offer.@@ -205,7 +209,7 @@ What an out-of-tree orchestrator imports *is* a contract. The SPI is a documente| Area | Modules / symbols ||---|---|| Boundary serialization + boot | `pipelex.runtime_bridge.serialization` (`serialize_pipe_output`, `serialize_completed_output`, `PIPE_DISPATCH_ERRORS`), `pipelex.runtime_bridge.payloads` (`PipelexPipeRunInput`, `PipelexPipeRunOutput`, `PipelexPipeDispatchAck`), `pipelex.runtime_bridge.bootstrap` (`ensure_pipelex_booted`) |-| Mode + delivery + errors | `pipelex.runtime_bridge.orchestration_mode` (`OrchestrationMode`, `DIRECT_ORCHESTRATION_MODE`), `pipelex.runtime_bridge.delivery_mode` (`DeliveryMode`), `pipelex.runtime_bridge.exceptions` (`MissingOrchestratorError`, `PipelexBridgeDispatchError`) |+| Mode + delivery + errors | `pipelex.runtime_bridge.orchestration_mode` (`OrchestrationMode`, `DIRECT_ORCHESTRATION_MODE`), `pipelex.runtime_bridge.delivery_mode` (`DeliveryMode`), `pipelex.runtime_bridge.exceptions` (`MissingOrchestratorError`, `PipelexBridgeDispatchError`), `pipelex.errors.security_event` (`emit_security_event_for_run`, `emit_security_event`, `mark_security_event_emitted`) || Working-memory hydration | `pipelex.runtime_bridge.primitives.hydration` (re-hydrate `working_memory_raw` → typed `WorkingMemory`; stayed open because it is host-agnostic — used by core delivery and the open `pipelex-api` runner, and re-used across the boundary by the commercial plugins) || Plugin contract | `pipelex.plugins.contract` (`PipelexPlugin`, `PLUGIN_API_VERSION`), `pipelex.plugins.registrar` (`PluginRegistrar` menu: `add_orchestrator`, `add_bundle_validator`, `add_http_error_mapper`, `claim_*`, `add_teardown`; read accessor: `get_http_error_mappers`), `pipelex.plugins.orchestrator_registry` (`OrchestratorProtocol`), `pipelex.plugins.bundle_validator_registry` (`BundleValidatorProtocol`, `BundleValidationVerdict`) || Execution protocols | `PipeRouterProtocol` (incl. the [`run_batch_branch`](./pipe-routing-and-execution.md#the-batch-branch-hook) fan-out hook — concrete default, override it only to isolate batch branches), `PipeRunProtocol`, `ContentGeneratorProtocol`, the task-manager protocol |diff --git a/docs/under-the-hood/template-sandbox.md b/docs/under-the-hood/template-sandbox.mdindex 075e0db74..8a727a822 100644--- a/docs/under-the-hood/template-sandbox.md+++ b/docs/under-the-hood/template-sandbox.md@@ -44,9 +44,9 @@ Many refusals are visible in the template itself, and validation reports them beWhat validation cannot see, because it depends on the values at run time (a method call, a name computed while rendering, or a bracketed key, which is legitimate on a plain dict), is refused where it happens: the render raises `Jinja2TemplateSecurityError`, with the code `security.template_refused`, whose message names the refused attribute or callable and the type it was reached on. A refusal never renders as an empty string, so a prompt is never sent with a hole in it.-`Jinja2TemplateSecurityError` is a `SecurityError`, like `UriReadRefusedError`, `SsrfBlockedError` and `UnsafeSchemaError`, the other policy refusals that are never absorbed into a domain answer. A host that treats that family alike, never retrying any of it, catches that base class, and an `except ToolError` around a render does not catch this refusal. The sandbox's budget refusal is a separate error, `Jinja2TemplateBudgetError`, which is a `ToolError` (see below). `Jinja2TemplateSecurityError` stays a type of its own, apart from `Jinja2TemplateRenderError`, which a template that fails for any other reason raises.+`Jinja2TemplateSecurityError` is a `SecurityError`, like `UriReadRefusedError`, `SsrfBlockedError` and `UnsafeSchemaError`, the other policy refusals that are never absorbed into a domain answer. A host that treats that family alike, never retrying any of it, catches that base class, and an `except ToolError` around a render does not catch this refusal. `Jinja2TemplateSecurityError` stays a type of its own, apart from `Jinja2TemplateRenderError`, which a template that fails for any other reason raises.-A render that would overdraw its budget raises `Jinja2TemplateBudgetError`, with the code `pipe.template_budget_exceeded`, whose message names the operation that was refused and what it would have spent, never the template or a value: for example, *the operator '\*' needs about 1,000,000,000, and 134,217,062 are left*. A template whose macros or includes nest deeper than Python allows raises the same error.+A render that would overdraw its budget raises `Jinja2TemplateBudgetError`, with the code `pipe.template_budget_exceeded`, whose message names the operation that was refused and what it would have spent, never the template or a value: for example, *the operator '\*' needs about 1,000,000,000, and 134,217,062 are left*. A template whose macros or includes nest deeper than Python allows raises the same error. It is a `ToolError`, not a `SecurityError`: a budget is a limit on what a render may spend rather than a policy refusal, so it is handled by its code like any other fault of the method.A template that names a filter or a test Jinja2 does not know inside a conditional branch compiles, and only the render of that branch finds the name. The render raises `Jinja2TemplateInvalidError`, with the code `method.template_invalid`, whose message names the unknown name and never quotes the template. A `PipeCondition` expression and a `PipeLLM` prompt answer it as a fault of the method, located on the pipe and the field, with the code's step for the method's author, which the caller view carries as well.diff --git a/pipelex/base_exceptions.py b/pipelex/base_exceptions.pyindex ee67fecf5..1fd5fcc3e 100644--- a/pipelex/base_exceptions.py+++ b/pipelex/base_exceptions.py@@ -33,6 +33,7 @@ from pipelex.errors.problem import (runtime_version,)from pipelex.errors.registry.pipelex_codes import PipelexErrorCode+from pipelex.errors.security_target import SecurityTargetClassfrom pipelex.suggested_fix import SuggestedFixfrom pipelex.system.telemetry.current_span import current_span_context_for_logs, pipelex_span_context_for_logsfrom pipelex.validation_error_types import ValidationErrorType@@ -279,6 +280,9 @@ class PipelexError(Exception):# The code the raise site chose with :meth:`with_error_code`, from the codes the class declares._chosen_error_code: ErrorCode | None = None+ # The class of destination a security refusal refused, stated by the raise site with+ # :meth:`with_security_target_class`; the security event carries it and nothing else about the target.+ _security_target_class: SecurityTargetClass | None = None# The parameters the raise site gave with :meth:`with_params`._given_params: dict[str, Any] | None = None# The items the raise site gave with :meth:`with_items`, one per fault of a refusal of several.@@ -324,6 +328,28 @@ class PipelexError(Exception):self._chosen_error_code = error_codereturn self+ def with_security_target_class(self, target_class: SecurityTargetClass | None) -> Self:+ """State, where a security refusal is decided, which class of destination it refused.++ The security event (:mod:`pipelex.errors.security_event`) names that class, a loopback or a private address,+ a cloud metadata endpoint, a scheme the runtime does not read, and never the destination itself. Only the+ raise site knows it: the SSRF guard has the address in hand, the read scope the reason. ``None`` leaves the+ class unknown, for a site that refused something it cannot classify, such as a name that resolved to nothing.++ Args:+ target_class: The class of destination refused, or ``None`` when the site cannot tell.++ Returns:+ This error, so that the raise site reads ``raise SomeError(msg).with_security_target_class(target_class)``.+ """+ self._security_target_class = target_class+ return self++ @property+ def security_target_class(self) -> SecurityTargetClass | None:+ """The class of destination this security refusal refused, ``None`` when nobody stated it."""+ return self._security_target_class+def with_params(self, **params: Any) -> Self:"""Give, where this error is raised, the parameters its code's sentences are rendered from.diff --git a/pipelex/core/bundle_resources/bundle_resource.py b/pipelex/core/bundle_resources/bundle_resource.pyindex 1845685df..cee2a45ed 100644--- a/pipelex/core/bundle_resources/bundle_resource.py+++ b/pipelex/core/bundle_resources/bundle_resource.py@@ -8,34 +8,55 @@ it would in a folder, and the run's crate names that tree by its digest like any"""from collections.abc import Mapping-from pathlib import PurePosixPath+from pathlib import PurePosixPath, PureWindowsPathfrom mthds.protocol.problem.registry import CoreCodefrom pipelex.core.bundle_resources.exceptions import BundleResourcePathError+def root_escape_refusal(*, path: str) -> str | None:+ r"""Why `path`, written relative to a root, would reach outside that root, or None when it stays below it.++ The one rule telling a path that tries to leave its root, a security refusal (``security.input_path_refused`` for a+ caller's path), from a path of the wrong shape, a refusal of the request: it is held by the key of a file submitted+ with a bundle, by a file submitted beside a method, and by a key of a method tree a request carries, which the+ standard's reader refuses without saying why. The path is read with either separator, as a system that follows a+ backslash would read it: it leaves its root when it is absolute or rooted (`/etc/passwd`, `\etc`), when it names a+ drive or a network share (`C:/Windows`, `C:secrets`, `\\server\share`), or when one of its segments is `..`. Any+ other path that is refused, empty, holding a backslash, a colon or a NUL character, is of the wrong shape. The+ reason never repeats the path.+ """+ windows_path = PureWindowsPath(path)+ if windows_path.drive:+ return "it names a drive or a network share; write it relative to the bundle"+ if windows_path.root:+ return "it is absolute; write it relative to the bundle"+ if ".." in windows_path.parts:+ return "it leaves the bundle through a '..' segment"+ return None++def resource_path_refusal(*, path: str) -> str | None:"""Why `path` cannot name a file inside a bundle, or None when it can.The one rule for a bundle-relative path, held by the key of a file submitted with a bundle and by the path a stepnames, such as a `PipeDocGen` step's `template_file`, so any file a step names can be submitted with its bundle.- The reason never repeats the path, which a host library's step would otherwise put in front of a caller.+ A path that leaves the bundle says so first (`root_escape_refusal`). The reason never repeats the path, which a host+ library's step would otherwise put in front of a caller."""if not path.strip():return "it is empty"+ escape = root_escape_refusal(path=path)+ if escape is not None:+ return escapeif "\\" in path:return "it holds a backslash; write the path with forward slashes"if ":" in path:return "it holds a colon"if "\x00" in path:return "it holds a NUL character"- pure_path = PurePosixPath(path)- if pure_path.is_absolute():- return "it is absolute; write it relative to the bundle"- if ".." in pure_path.parts:- return "it leaves the bundle through a '..' segment"- if not pure_path.parts:+ if not PurePosixPath(path).parts:return "it names no file"return None@@ -53,20 +74,28 @@ def canonical_resource_path_or_none(*, path: str) -> str | None:def canonical_resource_path(*, path: str) -> str:"""The one spelling of a bundle-relative path, the key a submitted file is held and looked up under.- POSIX separators only, relative, and inside the bundle: an absolute path, a `..` segment, a backslash, a colon- and an empty path are refused. `.` segments and repeated slashes collapse, so `./invoice.xlsx` and+ POSIX separators only, relative, and inside the bundle: an absolute path, a drive, a `..` segment, a backslash, a+ colon and an empty path are refused. `.` segments and repeated slashes collapse, so `./invoice.xlsx` and`invoice.xlsx` name one file.Raises:- BundleResourcePathError: the path cannot name a file inside the bundle.+ BundleResourcePathError: the path cannot name a file inside the bundle: ``security.input_path_refused`` for a+ path that leaves it (`root_escape_refusal`), ``request.invalid`` for a path of the wrong shape."""refusal = resource_path_refusal(path=path)if refusal is not None:msg = f"The file '{path}' submitted with the bundle cannot be one of its files: {refusal}."- raise BundleResourcePathError(msg)+ raise BundleResourcePathError(msg).with_error_code(path_refusal_code(path=path))return PurePosixPath(path).as_posix()+def path_refusal_code(*, path: str) -> CoreCode:+ """The code of the refusal of a caller's path: a security refusal when it leaves its root, else a refusal of the request."""+ if root_escape_refusal(path=path) is not None:+ return CoreCode.SECURITY_INPUT_PATH_REFUSED+ return CoreCode.REQUEST_INVALID++def canonical_submitted_files(*, files: Mapping[str, bytes]) -> dict[str, bytes]:"""The files submitted with bundles, keyed by their canonical bundle-relative path.diff --git a/pipelex/core/bundle_resources/exceptions.py b/pipelex/core/bundle_resources/exceptions.pyindex 5ac796307..878f26e02 100644--- a/pipelex/core/bundle_resources/exceptions.py+++ b/pipelex/core/bundle_resources/exceptions.py@@ -12,8 +12,9 @@ class BundleResourcePathError(PipelexError):the same file, or a key the tree of the submitted files cannot hold, such as one colliding with another oncecase-folded, or naming a file already submitted, from a library directory or with the request, with other bytes.The path is the caller's own, so the message is theirs to read, and the same request is refused again. A path that- cannot name a file inside the bundle is ``security.input_path_refused``; paths that each could but that cannot- make one tree together choose ``request.invalid``.+ leaves the bundle, absolute, naming a drive or climbing with `..`, is ``security.input_path_refused``; a path of+ the wrong shape, and paths that each could name a file but that cannot make one tree together, choose+ ``request.invalid`` (``pipelex.core.bundle_resources.bundle_resource.path_refusal_code``)."""error_code = ErrorCodeSet(default=CoreCode.SECURITY_INPUT_PATH_REFUSED, others=(CoreCode.REQUEST_INVALID,))diff --git a/pipelex/errors/security_event.py b/pipelex/errors/security_event.pynew file mode 100644index 000000000..68df09bc0--- /dev/null+++ b/pipelex/errors/security_event.py@@ -0,0 +1,276 @@+"""The security event: one structured record for each refusal a security rule makes, carrying no content.++A refusal for safety fails the run under a code of the ``security.`` family, or of Pipelex's own+``pipelex.security.`` family, whose domain says who caused it. Beside that failure the deployment logs one security+event, so that whoever runs it learns that someone tried to reach what a rule forbids: the hosted plane routes it to+its alerts and its digest, and an on-premises deployment receives it in its own sinks. It is a record of its own+kind, routable without reading its text: it is logged on this module's logger, ``pipelex.errors.security_event``,+with the fixed message :data:`SECURITY_EVENT_MESSAGE`, and its ``event.name`` field is :data:`SECURITY_EVENT_NAME`.++Its fields name the refusal and never its content:++- ``error.code``, ``error.domain`` and ``error.type``: the refusal's code, the code's domain, and the class of the+ error that carries or implies the code, never a wrapper above it;+- ``security.target``: the kind of thing refused, a URL, a path, a template, a schema, which the code decides;+- ``security.target_class``: the class of destination refused, a loopback or a private address, a cloud metadata+ endpoint, a scheme, when the raise site stated it (:class:`~pipelex.errors.security_target.SecurityTargetClass`);+- ``pipeline_run_id`` and ``request_id``, when there is a run or a request;+- ``run.extras.<key>``: each of the run's opaque extras, the labels a host attaches to a run, which the hosted+ plane fills with the organization. The runtime never reads one by name.++Never the URL, the path, the template, the schema, a prompt, an input value, nor the error's message, which may+quote any of them; never a traceback either, since the failure itself is logged and reported where it is handled.++A refusal is reported once per process, however many boundaries it crosses: the event marks the errors of the+chain it reported, and a boundary the same failure reaches again, wrapped, finds the refusal marked and logs nothing.+A refusal is the error that carries or implies the code its failure reports, so a wrapper chained to it with ``from``,+which takes its code, restates it, while a new refusal is reported on its own, even of the same code, even raised+while a reported one was being handled or chained to one with ``from``.+"""++from __future__ import annotations++import weakref+from typing import TYPE_CHECKING, Any++from mthds.protocol.problem.registry import CoreCode++from pipelex.base_exceptions import PipelexError, iter_cause_chain+from pipelex.errors.error_code_declaration import InheritsCause+from pipelex.errors.error_codes import ErrorCode, error_code_spec+from pipelex.errors.registry.pipelex_codes import PipelexErrorCode+from pipelex.errors.security_target import SecurityTargetClass, SecurityTargetKind+from pipelex.tools.log.error_fields import ERROR_TYPE_FIELD+from pipelex.tools.log.log import log+from pipelex.tools.log.log_context import PIPELINE_RUN_ID_FIELD, REQUEST_ID_FIELD++if TYPE_CHECKING:+ from collections.abc import Mapping++ from pipelex.system.job_metadata import RunMetadata++#: The message every security event is logged with, which a query or an alert selects the events by.+SECURITY_EVENT_MESSAGE = "A security rule refused a request"++#: The value of the event's ``event.name`` field, which names the kind of record whatever its logger.+SECURITY_EVENT_NAME = "pipelex.security.refusal"++#: The field naming the kind of record, the OpenTelemetry semantic-convention key for a log event's name.+EVENT_NAME_FIELD = "event.name"++#: The field carrying the refusal's registry code.+ERROR_CODE_FIELD = "error.code"++#: The field carrying the code's domain, who caused the refusal.+ERROR_DOMAIN_FIELD = "error.domain"++#: The field carrying the kind of thing refused.+SECURITY_TARGET_FIELD = "security.target"++#: The field carrying the class of destination refused, present only when the raise site stated it.+SECURITY_TARGET_CLASS_FIELD = "security.target_class"++#: The prefix of the fields carrying the run's extras, one field per key.+RUN_EXTRAS_FIELD_PREFIX = "run.extras."++#: The kind of thing each security code refuses. Every code of the two security families has an entry, which a test+#: checks against both registries, so a new security code cannot ship without saying what it refuses.+SECURITY_TARGET_KIND_BY_CODE: dict[ErrorCode, SecurityTargetKind] = {+ CoreCode.SECURITY_INPUT_URL_REFUSED: SecurityTargetKind.URL,+ CoreCode.SECURITY_METHOD_URL_REFUSED: SecurityTargetKind.URL,+ CoreCode.SECURITY_PROVIDER_URL_REFUSED: SecurityTargetKind.URL,+ CoreCode.SECURITY_INPUT_PATH_REFUSED: SecurityTargetKind.PATH,+ CoreCode.SECURITY_METHOD_PATH_REFUSED: SecurityTargetKind.PATH,+ CoreCode.SECURITY_TEMPLATE_REFUSED: SecurityTargetKind.TEMPLATE,+ CoreCode.SECURITY_SCHEMA_REFUSED: SecurityTargetKind.SCHEMA,+ CoreCode.SECURITY_INTEGRITY_MISMATCH: SecurityTargetKind.DEPENDENCY,+ PipelexErrorCode.SECURITY_RESERVED_KEY: SecurityTargetKind.OBJECT_KEY,+ PipelexErrorCode.SECURITY_RESULT_OUTSIDE_SCOPE: SecurityTargetKind.RESULT_LOCATION,+}++# The errors whose refusal has been reported, held weakly so the mark never keeps an error alive.+_REPORTED_ERRORS: weakref.WeakSet[PipelexError] = weakref.WeakSet()+++def security_target_kind(*, code: ErrorCode) -> SecurityTargetKind | None:+ """The kind of thing a code refuses, ``None`` for a code that is no security refusal.++ Args:+ code: A registered code.++ Returns:+ The kind for a code of the ``security.`` or ``pipelex.security.`` family, else ``None``.+ """+ return SECURITY_TARGET_KIND_BY_CODE.get(code)+++def is_security_refusal(*, error: BaseException) -> bool:+ """Whether a failure reports a code of the ``security.`` or ``pipelex.security.`` family, wherever it was coded.++ A refusal coded so at its raise site is a security refusal whatever its class, a ``SecurityError`` or not, so a+ handler that answers the caller's input with a domain answer, a validation verdict included, lets it through as it+ is rather than replacing its code with its own.++ Args:+ error: The failure, the outermost error of its chain.++ Returns:+ Whether its code is a security refusal's.+ """+ return isinstance(error, PipelexError) and security_target_kind(code=error.resolve_error_code()) is not None+++def mark_security_event_emitted(*, error: BaseException) -> None:+ """Record that the security event of ``error``'s failure was logged, so no boundary logs it again in this process.++ :func:`emit_security_event` marks what it reports by itself. A host calls this for a failure whose event was+ logged elsewhere, such as the error a transport rebuilds from a worker's report, so that the boundary it reaches+ here does not report the same refusal a second time.++ Args:+ error: The failure, the outermost error of its chain.+ """+ for node in iter_cause_chain(error):+ if isinstance(node, PipelexError):+ _REPORTED_ERRORS.add(node)+++def emit_security_event(+ *,+ error: BaseException,+ pipeline_run_id: str | None = None,+ request_id: str | None = None,+ extras: Mapping[str, str] | None = None,+) -> bool:+ """Log the security event of a failure, when its code is a security refusal and nobody reported it yet.++ The code is the one the failure reports (:meth:`~pipelex.base_exceptions.PipelexError.resolve_error_code`), so a+ refusal wrapped by every boundary it crossed is still recognized. A failure whose refusal, the error of its chain+ that carries or implies that code, was reported already logs nothing; the errors of the chain are marked once the+ event is logged.++ Args:+ error: The failure, the outermost error of its chain. One that is not a ``PipelexError`` reports no code+ and logs nothing.+ pipeline_run_id: The run the refusal failed, when there is one.+ request_id: The request the refusal answered, when there is one. Either identifier left out is taken+ from the log context bound where the event is logged, if any.+ extras: The run's opaque extras, each logged as a field of its own.++ Returns:+ Whether a security event was logged.+ """+ if not isinstance(error, PipelexError):+ return False+ code = error.resolve_error_code()+ target_kind = security_target_kind(code=code)+ if target_kind is None:+ return False+ refusal = _code_carrier(error=error)+ if refusal in _REPORTED_ERRORS:+ return False+ log_security_event(+ code=code,+ error_type=type(refusal).__name__,+ target_class=_stated_target_class(error=error),+ pipeline_run_id=pipeline_run_id,+ request_id=request_id,+ extras=extras,+ )+ mark_security_event_emitted(error=error)+ return True+++def emit_security_event_for_run(*, error: BaseException, run_metadata: RunMetadata) -> bool:+ """Log the security event of a run's failure, naming the run, its request and its extras.++ Args:+ error: The run's failure, the outermost error of its chain.+ run_metadata: The run's identity.++ Returns:+ Whether a security event was logged.+ """+ return emit_security_event(+ error=error,+ pipeline_run_id=run_metadata.pipeline_run_id,+ request_id=run_metadata.request_id,+ extras=run_metadata.extras,+ )+++def log_security_event(+ *,+ code: ErrorCode,+ error_type: str,+ target_class: SecurityTargetClass | None = None,+ pipeline_run_id: str | None = None,+ request_id: str | None = None,+ extras: Mapping[str, str] | None = None,+) -> None:+ """Log one security event from its parts, for a refusal that is not a ``PipelexError`` of this process.++ :func:`emit_security_event` derives the parts from a failure and calls this. A host calls it directly for a+ refusal it decides itself, such as one a gateway answered, and then owes the event's rule itself: no content in+ any part.++ Args:+ code: The refusal's code. A code outside the security families still logs, with no ``security.target``.+ error_type: The class name of the error that refused, never its message.+ target_class: The class of destination refused, when the refusing site knows it.+ pipeline_run_id: The run the refusal failed, when there is one.+ request_id: The request the refusal answered, when there is one.+ extras: The run's opaque extras, each logged as a field of its own.+ """+ fields: dict[str, Any] = {+ EVENT_NAME_FIELD: SECURITY_EVENT_NAME,+ ERROR_CODE_FIELD: code,+ ERROR_DOMAIN_FIELD: error_code_spec(code).domain,+ ERROR_TYPE_FIELD: error_type,+ }+ target_kind = security_target_kind(code=code)+ if target_kind is not None:+ fields[SECURITY_TARGET_FIELD] = target_kind+ if target_class is not None:+ fields[SECURITY_TARGET_CLASS_FIELD] = target_class+ if pipeline_run_id is not None:+ fields[PIPELINE_RUN_ID_FIELD] = pipeline_run_id+ if request_id is not None:+ fields[REQUEST_ID_FIELD] = request_id+ for extras_key, extras_value in (extras or {}).items():+ fields[f"{RUN_EXTRAS_FIELD_PREFIX}{extras_key}"] = extras_value+ log.warning(SECURITY_EVENT_MESSAGE, fields=fields)+++def _code_carrier(*, error: PipelexError) -> PipelexError:+ """The error of the chain that names the code ``error`` reports, the refusal the wrappers above it report.++ It is read in the walk :meth:`~pipelex.base_exceptions.PipelexError.resolve_error_code` makes, so the two cannot+ disagree: the first error down the chain that carries a code by itself, where that walk stops; when none does, the+ deepest error that implies a code, else the deepest that names its fallback, whose code the chain then reports; and+ the outermost error when none names one. It is the refusal's identity: two failures report the same refusal when+ they name the same error here, which is how a wrapper is told from a new refusal of the same code.+ """+ implied_carrier: PipelexError | None = None+ fallback_carrier: PipelexError | None = None+ for node in iter_cause_chain(error):+ if not isinstance(node, PipelexError):+ break+ if node.own_error_code is not None:+ return node+ if node.implied_error_code is not None:+ implied_carrier = node+ if isinstance(type(node).error_code, InheritsCause):+ fallback_carrier = node+ return implied_carrier or fallback_carrier or error+++def _stated_target_class(*, error: PipelexError) -> SecurityTargetClass | None:+ """The class of destination the first error down the chain states, ``None`` when none states one."""+ for node in iter_cause_chain(error):+ if not isinstance(node, PipelexError):+ break+ target_class = node.security_target_class+ if target_class is not None:+ return target_class+ return Nonediff --git a/pipelex/errors/security_target.py b/pipelex/errors/security_target.pynew file mode 100644index 000000000..d126086c1--- /dev/null+++ b/pipelex/errors/security_target.py@@ -0,0 +1,42 @@+"""What a security refusal refused, named without naming it: the vocabulary the security event carries.++A security event (:mod:`pipelex.errors.security_event`) says what kind of thing a security rule refused, and, for a+destination, which class of destination it was, and never the thing itself: the URL, the path, the template or the+schema is the caller's content or the method's. The kind follows from the refusal's code alone; the class is known+only where the refusal is decided, so a raise site states it on the error with+:meth:`~pipelex.base_exceptions.PipelexError.with_security_target_class`.++This module imports nothing from Pipelex, so the base error class can name the class without a cycle.+"""++from enum import StrEnum+++class SecurityTargetKind(StrEnum):+ """The kind of thing a security rule refused, which the refusal's code decides."""++ URL = "url"+ PATH = "path"+ TEMPLATE = "template"+ SCHEMA = "schema"+ DEPENDENCY = "dependency"+ OBJECT_KEY = "object_key"+ RESULT_LOCATION = "result_location"+++class SecurityTargetClass(StrEnum):+ """The class of destination a refused URL or storage key points at, as the raise site classifies it.++ The first five are the values the error registry gives the ``target_class`` parameter of the URL refusals:+ a loopback address, a private address (any other address that is not publicly routable), a link-local+ address, a cloud metadata endpoint, and a scheme the runtime does not read, a ``file://`` URL or a bare+ path on the server's disk among them. ``foreign_storage_key`` is a storage key outside the run's read scope,+ which another tenant's run wrote; the registry has no value for it yet.+ """++ LOOPBACK = "loopback"+ PRIVATE = "private"+ LINK_LOCAL = "link_local"+ METADATA = "metadata"+ SCHEME = "scheme"+ FOREIGN_STORAGE_KEY = "foreign_storage_key"diff --git a/pipelex/libraries/library_manager.py b/pipelex/libraries/library_manager.pyindex 1c1328b81..5c773384c 100644--- a/pipelex/libraries/library_manager.py+++ b/pipelex/libraries/library_manager.py@@ -21,7 +21,7 @@ from pipelex import logfrom pipelex.base_exceptions import ErrorDomain, PipelexError, SecurityError, iter_cause_chainfrom pipelex.cogt.exceptions import ModelChoiceNotFoundErrorfrom pipelex.config import is_pipe_func_sandbox_hosted-from pipelex.core.bundle_resources.bundle_resource import canonical_submitted_files+from pipelex.core.bundle_resources.bundle_resource import canonical_submitted_files, path_refusal_codefrom pipelex.core.bundle_resources.exceptions import BundleResourcePathErrorfrom pipelex.core.concepts.concept_blueprint import ConceptBlueprintfrom pipelex.core.concepts.concept_factory import ConceptFactory@@ -638,7 +638,7 @@ class LibraryManager(LibraryManagerAbstract):violation = tree_path_violation(path)if violation is not None:msg = f"The file '{path}' submitted beside the method cannot be a file of its tree: {violation}."- raise BundleResourcePathError(msg)+ raise BundleResourcePathError(msg).with_error_code(path_refusal_code(path=path))already_held = joined.get(path)if already_held is not None and already_held.sha256 != held_file.sha256:msg = (diff --git a/pipelex/pipe_run/pipe_run.py b/pipelex/pipe_run/pipe_run.pyindex c3b3afc45..ed8037846 100644--- a/pipelex/pipe_run/pipe_run.py+++ b/pipelex/pipe_run/pipe_run.py@@ -9,6 +9,7 @@ from typing_extensions import overridefrom pipelex import logfrom pipelex.base_exceptions import PipelexErrorfrom pipelex.errors.problem import PipelexProblem, ProblemView, make_problem+from pipelex.errors.security_event import emit_security_event_for_runfrom pipelex.graph.graph_tracer_manager import GraphTracerManagerfrom pipelex.interpreter_hub import get_concept_library, get_own_pipesfrom pipelex.pipe_run.delivery_assignment import DeliveryAssignment, DeliveryStatus@@ -83,7 +84,14 @@ class PipeRun(PipeRunProtocol):# The run's identifiers ride the payload; here is where a direct-mode run binds them onto every# record it emits, the delivery's included. The binding is released when the run returns.with pipe_job.job_metadata.log_context():- return await self._run_bound(pipe_job=pipe_job, delivery_assignment=delivery_assignment)+ try:+ return await self._run_bound(pipe_job=pipe_job, delivery_assignment=delivery_assignment)+ except PipelexError as exc:+ # This run owns the security events of the refusals raised while it executes, in this process; the runner+ # that dispatched it logs none. A refusal its pipes raised was logged already; one its tail raised, such as a+ # webhook the SSRF guard refuses at fire time after the run completed, is logged here.+ emit_security_event_for_run(error=exc, run_metadata=pipe_job.job_metadata.run_metadata)+ raiseasync def _run_bound(self,@@ -128,6 +136,8 @@ class PipeRun(PipeRunProtocol):"The execution of a pipe failed",fields={"pipe_code": pipe_job.pipe.code, "pipe_ref": pipe_job.pipe.pipe_ref, **error_fields(exc=exc)},)+ # A security refusal also logs its security event, once, naming the run and never what it refused.+ emit_security_event_for_run(error=exc, run_metadata=pipe_job.job_metadata.run_metadata)finally:tracer_manager = GraphTracerManager.get_instance()if tracer_manager is not None:diff --git a/pipelex/pipe_run/pipe_run_protocol.py b/pipelex/pipe_run/pipe_run_protocol.pyindex dc8c3f7f5..f7acd3192 100644--- a/pipelex/pipe_run/pipe_run_protocol.py+++ b/pipelex/pipe_run/pipe_run_protocol.py@@ -27,5 +27,11 @@ class PipeRunProtocol(Protocol):When `delivery_assignment` is None, no storage or webhook delivery happens.Pass `DeliveryAssignment(storage=StorageTarget())` for default storage-only delivery.++ An implementation owns the security events of the refusals raised while the job executes: each is logged+ once, in the process that raised it (`pipelex.errors.security_event`). One that hands the job to another+ process logs there, and marks with `mark_security_event_emitted` every error it rebuilds from that process's+ report, the outermost error of the chain it raises: the runner that called `run` logs any failure it raises+ unmarked, as a refusal raised in the runner's own process that nobody logged."""...diff --git a/pipelex/pipeline/runner.py b/pipelex/pipeline/runner.pyindex 80bcd6b4c..95104ab6d 100644--- a/pipelex/pipeline/runner.py+++ b/pipelex/pipeline/runner.py@@ -17,6 +17,7 @@ from typing_extensions import overridefrom pipelex.base_exceptions import PipelexErrorfrom pipelex.cogt.models.model_listing import ModelCategory, list_modelsfrom pipelex.config import get_config+from pipelex.errors.security_event import emit_security_event, emit_security_event_for_runfrom pipelex.graph.graph_tracer_manager import GraphTracerManagerfrom pipelex.interpreter_hub import (clear_current_library,@@ -372,9 +373,18 @@ class PipelexMTHDSProtocol(MTHDSProtocol["PipeOutput"]):pipe_output = await effective_pipe_run.run(pipe_job, delivery_assignment=delivery_assignment)except PipelexError as exc:# If pipe_job is None, the error occurred during pipeline_run_setup, before the job existed:- # it is not a run failure, and it propagates untouched.+ # it is not a run failure, and it propagates untouched. A security refusal raised there, in this+ # process, logs its security event here.if pipe_job is None:+ emit_security_event(error=exc, request_id=request_id, extras=self.extras)raise+ # A refusal raised during the run is logged in the process that raised it, by whoever saw it first: `PipeRun`+ # logged one its run raised in this process, and a remote orchestrator's worker one raised there, whose+ # failure reaches this process rebuilt from its report and marked by the orchestrator that rebuilt it. Either+ # way it is reported, and this logs nothing. What is left is a refusal raised in this process that the pipe+ # run did not log, such as an input URL the SSRF guard refused while a remote pipe run uploaded the inputs+ # before it dispatched: it is logged here, naming the run, and the handler above logs it no more.+ emit_security_event_for_run(error=exc, run_metadata=pipe_job.job_metadata.run_metadata)properties = {EventProperty.PIPELINE_RUN_ID: pipeline_run_id,EventProperty.PIPE_TYPE: pipe_job.pipe.pipe_type,diff --git a/pipelex/pipeline/validate_bundle_translation.py b/pipelex/pipeline/validate_bundle_translation.pyindex 4eac1a3c9..058df655d 100644--- a/pipelex/pipeline/validate_bundle_translation.py+++ b/pipelex/pipeline/validate_bundle_translation.py@@ -32,6 +32,7 @@ from pipelex.core.pipes.exceptions import ()from pipelex.core.validation import report_validation_errorfrom pipelex.errors.error_codes import code_is_method_domain+from pipelex.errors.security_event import is_security_refusalfrom pipelex.interpreter_hub import get_library_managerfrom pipelex.libraries.exceptions import LibraryError, LibraryLoadingError, MethodTreeLoadErrorfrom pipelex.libraries.library_utils import get_pipelex_mthds_files_from_dirs@@ -101,9 +102,10 @@ def translate_to_validate_bundle_error() -> Generator[None, None, None]:invalid. A failure of the tool or its environment, a refusal of another actor than the method's author,anything that is not a ``PipelexError``, propagates as no verdict, along with the refusals that mustkeep their own class: ``PipeNotFoundError`` (its- dedicated not-found handler), a ``SecurityError`` (never absorbed by a domain handler), a- ``ValidateBundleError`` already produced (never re-wrapped), and a ``BundleResourceNotNamedError`` or a- ``BundleResourcePathError``, which refuse the files submitted with the bundle rather than the bundle.+ dedicated not-found handler), a security refusal, a ``SecurityError`` or any refusal coded in a security+ family (never absorbed by a domain handler), a ``ValidateBundleError`` already produced (never re-wrapped),+ and a ``BundleResourceNotNamedError`` or a ``BundleResourcePathError``, which refuse the files submitted+ with the bundle rather than the bundle."""try:yield@@ -267,7 +269,11 @@ def _make_refusal_verdict(*, refusal: PipelexError) -> ValidateBundleError | Non``None`` — no verdict — unless the refusal's code is the method's: a refusal another actor must act on,the caller's input, the account, the operator or the runtime, is never the bundle's fault, and keeps- its own code. The item is ``pipe_validation`` when the library load located the+ its own code. A security refusal has no verdict either, whatever its class and its domain: a refusal coded in+ a security family, such as a step's template file that climbs out of its method tree+ (``security.method_path_refused``), keeps its code and logs its security event, never a verdict's.++ The item is ``pipe_validation`` when the library load located therefusal on the pipe it was building (``PipeLoadRefusalError``, with the pipe's code, domain and file),and ``blueprint_validation`` otherwise. It carries no ``error_type``: a refusal with a closed codehas its own arm above, and naming one here would claim a diagnosis the refusal does not make.@@ -275,7 +281,7 @@ def _make_refusal_verdict(*, refusal: PipelexError) -> ValidateBundleError | NonThe item's text is the sentence the refusal's code renders for a caller, never the refusal's own message:the verdict is read by the caller, and the message is the operator's diagnostics."""- if not code_is_method_domain(code=refusal.resolve_error_code()):+ if is_security_refusal(error=refusal) or not code_is_method_domain(code=refusal.resolve_error_code()):return Nonemessage = refusal_detail(refusal=refusal)if isinstance(refusal, PipeLoadRefusalError):diff --git a/pipelex/tools/network/host_rules.py b/pipelex/tools/network/host_rules.pyindex 555176abc..45424bfc3 100644--- a/pipelex/tools/network/host_rules.py+++ b/pipelex/tools/network/host_rules.py@@ -7,13 +7,38 @@ rule here means the API server and the worker classify the same address the sameway."""-from ipaddress import ip_address+from ipaddress import IPv4Address, IPv6Address, ip_address++from pipelex.errors.security_target import SecurityTargetClass++# The cloud metadata-server aliases: ``metadata`` / ``metadata.google.internal`` are GCP's.+_METADATA_HOSTNAMES: frozenset[str] = frozenset({"metadata.google.internal", "metadata"})++# The loopback alias.+_LOOPBACK_HOSTNAMES: frozenset[str] = frozenset({"localhost"})# Hostnames that never resolve to a public destination but aren't literal IPs, so-# the IP-privateness check below can't catch them. ``metadata`` /-# ``metadata.google.internal`` are the GCP metadata-server aliases; ``localhost``-# is the loopback alias.-_DISALLOWED_HOSTNAMES: frozenset[str] = frozenset({"localhost", "metadata.google.internal", "metadata"})+# the IP-privateness check below can't catch them.+_DISALLOWED_HOSTNAMES: frozenset[str] = _METADATA_HOSTNAMES | _LOOPBACK_HOSTNAMES++# The well-known cloud metadata and credential endpoints by address, each in its canonical spelling. Every one is+# already refused as an address that is not publicly routable; this set only names its class for the security event.+_METADATA_IPS: frozenset[str] = frozenset(+ {+ # The instance metadata service AWS, GCP, Azure and Oracle Cloud Infrastructure all serve, and AWS's IPv6 one.+ "169.254.169.254",+ "fd00:ec2::254",+ # AWS ECS: the task metadata endpoint and the container credentials endpoint.+ "169.254.170.2",+ # AWS EKS Pod Identity: the credentials agent, and its IPv6 address.+ "169.254.170.23",+ "fd00:ec2::23",+ # Alibaba Cloud's instance metadata service.+ "100.100.100.200",+ # Oracle Cloud's legacy metadata service.+ "192.0.0.192",+ }+)def is_disallowed_ip(host: str) -> bool:@@ -64,3 +89,40 @@ def is_disallowed_host(host: str) -> bool:if normalized_host in _DISALLOWED_HOSTNAMES:return Truereturn is_disallowed_ip(normalized_host)+++def classify_disallowed_host(host: str) -> SecurityTargetClass | None:+ """The class of destination a disallowed host is, for the security event its refusal emits.++ A cloud metadata endpoint, by alias or by address, is ``metadata``; ``localhost`` and a loopback address are+ ``loopback``; any other link-local address is ``link_local``; and every other address that is not publicly+ routable (private ranges, carrier-grade NAT, multicast, reserved, the unspecified address) is ``private``.+ The host is normalized as :func:`is_disallowed_host` normalizes it, and an IPv4-mapped IPv6 address+ (``::ffff:169.254.169.254``) is classified as the IPv4 address it carries, which is where a dual-stack socket+ connects it: ``ipaddress`` reads the mapped form's loopback and link-local properties differently across Python+ versions, and never as the metadata address it spells.++ Args:+ host: A host :func:`is_disallowed_host` refuses, a literal IP or a hostname.++ Returns:+ The host's class, or ``None`` for a host that is neither a known internal name nor a literal disallowed+ IP, an empty one among them, which the refusal names no class for.+ """+ normalized_host = host.rstrip(".").lower()+ if normalized_host in _METADATA_HOSTNAMES:+ return SecurityTargetClass.METADATA+ if normalized_host in _LOOPBACK_HOSTNAMES:+ return SecurityTargetClass.LOOPBACK+ if not is_disallowed_ip(normalized_host):+ return None+ addr: IPv4Address | IPv6Address = ip_address(normalized_host)+ if isinstance(addr, IPv6Address) and addr.ipv4_mapped is not None:+ addr = addr.ipv4_mapped+ if str(addr) in _METADATA_IPS:+ return SecurityTargetClass.METADATA+ if addr.is_loopback:+ return SecurityTargetClass.LOOPBACK+ if addr.is_link_local:+ return SecurityTargetClass.LINK_LOCAL+ return SecurityTargetClass.PRIVATEdiff --git a/pipelex/tools/network/ssrf_guard.py b/pipelex/tools/network/ssrf_guard.pyindex b61839d54..88f4ef5e6 100644--- a/pipelex/tools/network/ssrf_guard.py+++ b/pipelex/tools/network/ssrf_guard.py@@ -26,7 +26,7 @@ import httpxfrom typing_extensions import overridefrom pipelex.tools.network.exceptions import SsrfBlockedError-from pipelex.tools.network.host_rules import is_disallowed_host, is_disallowed_ip+from pipelex.tools.network.host_rules import classify_disallowed_host, is_disallowed_host, is_disallowed_ipif TYPE_CHECKING:from collections.abc import Iterable@@ -58,7 +58,7 @@ async def resolve_to_allowed_ips("""if is_disallowed_host(host):msg = f"Refusing to connect to {host!r}: destination is a disallowed (private/loopback/metadata) address."- raise SsrfBlockedError(msg)+ raise SsrfBlockedError(msg).with_security_target_class(classify_disallowed_host(host))loop = asyncio.get_running_loop()getaddrinfo_coro = loop.getaddrinfo(host, port, type=socket.SOCK_STREAM)try:@@ -79,9 +79,11 @@ async def resolve_to_allowed_ips(if not resolved_ips:msg = f"Refusing to connect to {host!r}: name resolved to no addresses."raise SsrfBlockedError(msg)- if any(is_disallowed_ip(resolved_ip) for resolved_ip in resolved_ips):+ disallowed_ips = [resolved_ip for resolved_ip in resolved_ips if is_disallowed_ip(resolved_ip)]+ if disallowed_ips:msg = f"Refusing to connect to {host!r}: it resolved to a disallowed (private/loopback/metadata) address."- raise SsrfBlockedError(msg)+ # The first disallowed address classifies the refusal for its security event, which never names the address.+ raise SsrfBlockedError(msg).with_security_target_class(classify_disallowed_host(disallowed_ips[0]))return resolved_ipsdiff --git a/pipelex/tools/uri/exceptions.py b/pipelex/tools/uri/exceptions.pyindex 9781edebb..816874cbd 100644--- a/pipelex/tools/uri/exceptions.py+++ b/pipelex/tools/uri/exceptions.py@@ -6,6 +6,7 @@ from typing_extensions import overridefrom pipelex.base_exceptions import SecurityErrorfrom pipelex.errors.error_code_declaration import ErrorCodeSet+from pipelex.errors.security_target import SecurityTargetClassclass UriReadRefusalReason(StrEnum):@@ -22,6 +23,15 @@ class UriReadRefusalReason(StrEnum):case UriReadRefusalReason.LOCAL_PATH:return "it names a file on the server's own disk, or a URL in a scheme this runtime does not read, and a hosted run reads neither"+ @property+ def security_target_class(self) -> SecurityTargetClass:+ """The class of destination the refusal names in its security event: another tenant's key, or a scheme the run does not read."""+ match self:+ case UriReadRefusalReason.FOREIGN_STORAGE_KEY:+ return SecurityTargetClass.FOREIGN_STORAGE_KEY+ case UriReadRefusalReason.LOCAL_PATH:+ return SecurityTargetClass.SCHEME+@propertydef target_class(self) -> str:"""The kind of target the refused URL points at, as the refusal's sentence names it."""@@ -55,6 +65,8 @@ class UriReadRefusedError(SecurityError):def __init__(self, message: str, *, reason: UriReadRefusalReason):super().__init__(message)self.reason = reason+ # The reason is the class of destination refused, so the security event names it without the raise site saying so.+ self.with_security_target_class(reason.security_target_class)@overridedef problem_params(self) -> dict[str, Any]:diff --git a/subject_grants.toml b/subject_grants.tomlindex 8b6320fba..d744fd1d3 100644--- a/subject_grants.toml+++ b/subject_grants.toml@@ -8,6 +8,10 @@ rationale = "Converts its one operand, a registry domain: ErrorDomain.from_problparam = "error_code"rationale = "Chained at every raise site on its one operand, the chosen code: raise SomeError(msg).with_error_code(CoreCode.INPUT_MISSING)."+["pipelex/base_exceptions.py::PipelexError.with_security_target_class"]+param = "target_class"+rationale = "Chained at every raise site on its one operand, the class of destination refused: raise SsrfBlockedError(msg).with_security_target_class(target_class)."+["pipelex/base_exceptions.py::error_domain_is_input"]param = "error_domain"rationale = "Predicate on the error domain; single operand."@@ -6376,6 +6380,10 @@ rationale = "Parses/reads the path; single operand."param = "data"rationale = "Transforms the data; single operand."+["pipelex/tools/network/host_rules.py::classify_disallowed_host"]+param = "host"+rationale = "Reads as classify(host): its one operand is the host the SSRF guard refused, as is_disallowed_host takes it."+["pipelex/tools/network/host_rules.py::is_disallowed_host"]param = "host"rationale = "Predicate on the host; single operand."diff --git a/tests/integration/pipelex/pipeline/test_data.py b/tests/integration/pipelex/pipeline/test_data.pyindex b78c132d1..43bc8132e 100644--- a/tests/integration/pipelex/pipeline/test_data.py+++ b/tests/integration/pipelex/pipeline/test_data.py@@ -700,6 +700,98 @@ prompt = "Read the label from $note""""+class SecurityRefusalTestData:+ """Runs a security rule refuses, mid-run, before the run and at a template's render, and what none of their events may quote."""++ READ_SCOPE: ClassVar[str] = "org_abc"+ STORAGE_SCOPE: ClassVar[str] = "org_abc/mt_1/run_1"+ EXTRAS: ClassVar[dict[str, str]] = {"org_id": "org_abc"}++ # Another tenant's storage key, which a construct turns into an image URL the next step shows a model.+ FOREIGN_KEY: ClassVar[str] = "pipelex-storage://org_other/mt_y/run_9/assets/payroll.png"+ FORGED_IMAGE_MTHDS: ClassVar[str] = """+domain = "security_refusal_forged_image"+description = "Forge an image URL from plain text, then show the image to a model"+main_pipe = "describe_forged_photo"++[concept.Photo]+description = "A photo with a caption"++[concept.Photo.structure]+caption = { type = "text", description = "The caption" }+picture = { type = "concept", concept_ref = "native.Image", description = "The picture" }++[pipe.describe_forged_photo]+type = "PipeSequence"+description = "Forge a photo from a text, then describe it"+inputs = { where = "Text" }+output = "Text"+steps = [+ { pipe = "forge_photo", result = "photo" },+ { pipe = "describe_photo", result = "description" },+]++[pipe.forge_photo]+type = "PipeCompose"+description = "Build a photo whose URL is the text it was given"+inputs = { where = "Text" }+output = "Photo"++[pipe.forge_photo.construct]+caption = "forged"++[pipe.forge_photo.construct.picture]+url = { template = "{{ where }}" }++[pipe.describe_photo]+type = "PipeLLM"+description = "Describe the photo"+inputs = { photo = "Photo" }+output = "Text"+prompt = \"\"\"+Describe this image.+$photo.picture+\"\"\"+"""++ # A path on the server's disk passed as an image input, which the input seam refuses before the run exists.+ LOCAL_PATH: ClassVar[str] = "/etc/passwd"+ DESCRIBE_IMAGE_MTHDS: ClassVar[str] = """+domain = "security_refusal_input"+description = "Describe an image input"+main_pipe = "describe_image"++[pipe.describe_image]+type = "PipeLLM"+description = "Describe an image"+inputs = { photo = "Image" }+output = "Text"+prompt = \"\"\"+Describe this image.+$photo+\"\"\"+"""++ # A template reading a private name off a value, which passes the load and which the sandbox refuses at render.+ PRIVATE_READ_TEMPLATE: ClassVar[str] = "Talk about {{ topic['_owner'] }}"+ # A template that fails at render for a reason no security rule has, a method the value lacks.+ MISSING_METHOD_TEMPLATE: ClassVar[str] = "Talk about {{ topic.model_dump_json() }}"+ TOPIC: ClassVar[str] = "the quarterly payroll"+ COMPOSE_MTHDS_SLOT: ClassVar[str] = "TEMPLATE_SLOT"+ COMPOSE_MTHDS: ClassVar[str] = """+domain = "security_refusal_template"+description = "Compose a line about a topic"+main_pipe = "say"++[pipe.say]+type = "PipeCompose"+description = "Say something about the topic"+inputs = { topic = "Text" }+output = "Text"+template = "TEMPLATE_SLOT"+"""++class ResolveBundleTestData:"""Bundles the host-facing resolve cores are pinned with."""diff --git a/tests/integration/pipelex/pipeline/test_security_refusal_event.py b/tests/integration/pipelex/pipeline/test_security_refusal_event.pynew file mode 100644index 000000000..6ddfb7a43--- /dev/null+++ b/tests/integration/pipelex/pipeline/test_security_refusal_event.py@@ -0,0 +1,155 @@+from __future__ import annotations++import io+import json+import logging+from contextlib import contextmanager+from typing import TYPE_CHECKING, Any++import pytest+from mthds.protocol.problem.registry import CoreCode, ProblemDomain++from pipelex.base_exceptions import PipelexError+from pipelex.errors.error_codes import error_code_spec+from pipelex.errors.security_event import SECURITY_EVENT_MESSAGE, SECURITY_EVENT_NAME+from pipelex.errors.security_target import SecurityTargetClass, SecurityTargetKind+from pipelex.pipeline.exceptions import PipelineExecutionError+from pipelex.pipeline.runner import PipelexMTHDSProtocol+from pipelex.system.pipe_run_mode import PipeRunMode+from pipelex.system.storage_scope import LOCAL_STORAGE_SCOPE+from pipelex.tools.log.json_log_sink import JsonLogSink+from pipelex.tools.uri.exceptions import UriReadRefusedError+from tests.integration.pipelex.pipeline.test_data import SecurityRefusalTestData++if TYPE_CHECKING:+ from collections.abc import Generator++ from mthds.protocol.pipeline_inputs import PipelineInputs++SECURITY_EVENT_LOGGER = "pipelex.errors.security_event"+++@contextmanager+def _security_event_lines() -> Generator[list[dict[str, Any]]]:+ """The security events logged in the block, as the ``json`` sink serializes them."""+ buffer = io.StringIO()+ handler = JsonLogSink(stream=buffer).handler+ logger = logging.getLogger(SECURITY_EVENT_LOGGER)+ previous_level = logger.level+ logger.setLevel(logging.WARNING)+ logger.addHandler(handler)+ lines: list[dict[str, Any]] = []+ try:+ yield lines+ finally:+ logger.removeHandler(handler)+ logger.setLevel(previous_level)+ handler.close()+ lines.extend(json.loads(line) for line in buffer.getvalue().splitlines() if line)+++def _scoped_runner() -> PipelexMTHDSProtocol:+ return PipelexMTHDSProtocol(+ pipe_run_mode=PipeRunMode.DRY,+ user_id="test-user",+ storage_scope=SecurityRefusalTestData.STORAGE_SCOPE,+ read_scope=SecurityRefusalTestData.READ_SCOPE,+ extras=SecurityRefusalTestData.EXTRAS,+ )+++async def _refused_run(*, runner: PipelexMTHDSProtocol, bundle: str, inputs: PipelineInputs) -> tuple[PipelexError, list[dict[str, Any]]]:+ with _security_event_lines() as lines, pytest.raises(PipelexError) as exc_info:+ await runner.execute(mthds_contents=[bundle], inputs=inputs)+ return exc_info.value, lines+++def _compose_bundle(*, template: str) -> str:+ return SecurityRefusalTestData.COMPOSE_MTHDS.replace(SecurityRefusalTestData.COMPOSE_MTHDS_SLOT, template)+++def _assert_is_the_event(*, line: dict[str, Any], code: CoreCode, error_type: str, target_kind: SecurityTargetKind) -> None:+ assert line["message"] == SECURITY_EVENT_MESSAGE+ assert line["event.name"] == SECURITY_EVENT_NAME+ assert line["error.code"] == code+ assert line["error.domain"] == error_code_spec(code).domain+ assert line["error.type"] == error_type+ assert line["security.target"] == target_kind+ assert line["run.extras.org_id"] == SecurityRefusalTestData.EXTRAS["org_id"]+ assert "exception" not in line++[email protected](loop_scope="class")+class TestSecurityRefusalEvent:+ async def test_a_refusal_mid_run_fails_the_run_under_its_code_and_logs_one_event(self) -> None:+ """A key of another tenant read by a step: the method's `security.method_url_refused`, one event naming the run, never the key."""+ error, lines = await _refused_run(+ runner=_scoped_runner(),+ bundle=SecurityRefusalTestData.FORGED_IMAGE_MTHDS,+ inputs={"where": SecurityRefusalTestData.FOREIGN_KEY},+ )++ assert isinstance(error, PipelineExecutionError)+ assert error.resolve_error_code() == CoreCode.SECURITY_METHOD_URL_REFUSED+ assert error_code_spec(CoreCode.SECURITY_METHOD_URL_REFUSED).domain == ProblemDomain.METHOD+ # Reported by the run's boundary, and not again by the runner's that wraps the failure.+ (line,) = lines+ _assert_is_the_event(+ line=line, code=CoreCode.SECURITY_METHOD_URL_REFUSED, error_type="UriReadRefusedError", target_kind=SecurityTargetKind.URL+ )+ assert line["security.target_class"] == SecurityTargetClass.FOREIGN_STORAGE_KEY+ assert line["pipeline_run_id"]+ assert SecurityRefusalTestData.FOREIGN_KEY not in json.dumps(line)+ assert "org_other" not in json.dumps(line)++ async def test_a_refusal_before_the_run_logs_its_event_too(self) -> None:+ """A local path passed as an input: the caller's `security.input_url_refused`, refused before any run, still one event."""+ error, lines = await _refused_run(+ runner=_scoped_runner(),+ bundle=SecurityRefusalTestData.DESCRIBE_IMAGE_MTHDS,+ inputs={"photo": SecurityRefusalTestData.LOCAL_PATH},+ )++ assert isinstance(error, UriReadRefusedError)+ assert error.resolve_error_code() == CoreCode.SECURITY_INPUT_URL_REFUSED+ (line,) = lines+ _assert_is_the_event(+ line=line, code=CoreCode.SECURITY_INPUT_URL_REFUSED, error_type="UriReadRefusedError", target_kind=SecurityTargetKind.URL+ )+ assert line["security.target_class"] == SecurityTargetClass.SCHEME+ assert SecurityRefusalTestData.LOCAL_PATH not in json.dumps(line)++ async def test_a_template_the_sandbox_refuses_at_render_logs_its_event(self) -> None:+ """A method's template reading a private name: `security.template_refused`, the method's, and an event quoting no template."""+ runner = PipelexMTHDSProtocol(pipe_run_mode=PipeRunMode.DRY, storage_scope=LOCAL_STORAGE_SCOPE, extras=SecurityRefusalTestData.EXTRAS)+ error, lines = await _refused_run(+ runner=runner,+ bundle=_compose_bundle(template=SecurityRefusalTestData.PRIVATE_READ_TEMPLATE),+ inputs={"topic": SecurityRefusalTestData.TOPIC},+ )++ assert isinstance(error, PipelineExecutionError)+ assert error.resolve_error_code() == CoreCode.SECURITY_TEMPLATE_REFUSED+ assert error_code_spec(CoreCode.SECURITY_TEMPLATE_REFUSED).domain == ProblemDomain.METHOD+ (line,) = lines+ _assert_is_the_event(+ line=line,+ code=CoreCode.SECURITY_TEMPLATE_REFUSED,+ error_type="Jinja2TemplateSecurityError",+ target_kind=SecurityTargetKind.TEMPLATE,+ )+ assert "security.target_class" not in line+ serialized = json.dumps(line)+ assert "_owner" not in serialized+ assert SecurityRefusalTestData.TOPIC not in serialized++ async def test_a_failure_no_security_rule_made_logs_no_event(self) -> None:+ runner = PipelexMTHDSProtocol(pipe_run_mode=PipeRunMode.DRY, storage_scope=LOCAL_STORAGE_SCOPE, extras=SecurityRefusalTestData.EXTRAS)+ error, lines = await _refused_run(+ runner=runner,+ bundle=_compose_bundle(template=SecurityRefusalTestData.MISSING_METHOD_TEMPLATE),+ inputs={"topic": SecurityRefusalTestData.TOPIC},+ )++ assert error.resolve_error_code() == CoreCode.PIPE_TEMPLATE_RENDER_FAILED+ assert lines == []diff --git a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_template_files.py b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_template_files.pyindex 0da5da380..dd3ca07bd 100644--- a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_template_files.py+++ b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_template_files.py@@ -5,6 +5,7 @@ from mthds.method_tree.models import MethodTreefrom mthds.protocol.problem.registry import CoreCode, ProblemDomainfrom pytest_mock import MockerFixture+from pipelex.base_exceptions import PipelexErrorfrom pipelex.cogt.doc_gen.input_shape import InputShapeKindfrom pipelex.cogt.doc_gen.template_check import TemplateFinding, TemplateFindingSeverityfrom pipelex.core.method_trees.method_file_store import LOCAL_METHOD_FILES@@ -48,7 +49,6 @@ class TestPipeDocGenTemplateFiles:("template_file", "expected_code", "expected"),[("invoice.docx", CoreCode.METHOD_RESOURCE_MISSING, "resolves to 'invoice.docx', which is not a file of the tree."),- ("../outside.docx", CoreCode.SECURITY_METHOD_PATH_REFUSED, "climbs above the root of its method tree"),("ignored.docx", CoreCode.METHOD_RESOURCE_MISSING, "which is not a file of the tree: it is kept out by a .gitignore pattern"),(".templates/invoice.docx",@@ -56,15 +56,14 @@ class TestPipeDocGenTemplateFiles:"which is not a file of the tree: it is kept out by a path segment starting with a dot",),],- ids=["missing_file", "outside_the_tree", "gitignored", "hidden"],+ ids=["missing_file", "gitignored", "hidden"],)async def test_a_template_file_is_a_file_of_the_bundle_s_method_tree(self, tmp_path: Path, template_file: str, expected_code: CoreCode, expected: str) -> None:- """A template file must be a file of the method's tree: one outside it, or one its membership rule keeps out, is never read."""+ """A template file must be a file of the method's tree: one its membership rule keeps out is never read."""method_dir = tmp_path / "method"method_dir.mkdir()- (tmp_path / "outside.docx").write_bytes(b"docx")(method_dir / "ignored.docx").write_bytes(b"docx")(method_dir / ".gitignore").write_text("ignored.docx\n", encoding="utf-8")(method_dir / ".templates").mkdir()@@ -77,6 +76,21 @@ class TestPipeDocGenTemplateFiles:assert only_problem_item(exc_info.value).code == expected_codeassert expected in refusal_messages(exc_info.value)+ @pytest.mark.usefixtures("stub_engines")+ async def test_a_template_file_outside_the_tree_is_the_method_s_security_refusal_not_a_verdict(self, tmp_path: Path) -> None:+ """A template file that climbs out of the method's tree is refused under its security code, never absorbed into a verdict."""+ method_dir = tmp_path / "method"+ method_dir.mkdir()+ (tmp_path / "outside.docx").write_bytes(b"docx")+ bundle_path = write_bundle(directory=method_dir, step_fields='format = "docx"\ntemplate_file = "../outside.docx"')++ with pytest.raises(PipelexError) as exc_info:+ await validate_bundle(mthds_file_path=bundle_path)++ assert not isinstance(exc_info.value, ValidateBundleError)+ assert exc_info.value.resolve_error_code() == CoreCode.SECURITY_METHOD_PATH_REFUSED+ assert "climbs above the root of its method tree" in exc_info.value.message+@pytest.mark.usefixtures("stub_engines")async def test_a_template_file_climbs_through_dot_dot_to_another_folder_of_its_tree(self, tmp_path: Path) -> None:"""A `..` may climb out of the bundle's folder to a file elsewhere in its method tree, which the step reads."""diff --git a/tests/unit/pipelex/core/bundle_resources/test_bundle_resource.py b/tests/unit/pipelex/core/bundle_resources/test_bundle_resource.pyindex f0196998c..cd7989dc7 100644--- a/tests/unit/pipelex/core/bundle_resources/test_bundle_resource.py+++ b/tests/unit/pipelex/core/bundle_resources/test_bundle_resource.py@@ -1,4 +1,5 @@import pytest+from mthds.protocol.problem.registry import CoreCodefrom pipelex.core.bundle_resources.bundle_resource import (canonical_resource_path,@@ -25,24 +26,30 @@ class TestCanonicalResourcePath:assert canonical_resource_path_or_none(path=path) == expected@pytest.mark.parametrize(- ("path", "reason"),+ ("path", "reason", "code"),[- ("", "empty"),- (" ", "empty"),- (".", "names no file"),- ("/etc/invoice.docx", "absolute"),- ("../invoice.docx", "'..' segment"),- ("templates/../../invoice.docx", "'..' segment"),- ("templates\\invoice.docx", "backslash"),- ("C:invoice.docx", "colon"),- ("invoice\x00.docx", "NUL"),+ ("", "empty", CoreCode.REQUEST_INVALID),+ (" ", "empty", CoreCode.REQUEST_INVALID),+ (".", "names no file", CoreCode.REQUEST_INVALID),+ ("/etc/invoice.docx", "absolute", CoreCode.SECURITY_INPUT_PATH_REFUSED),+ ("\\etc\\invoice.docx", "absolute", CoreCode.SECURITY_INPUT_PATH_REFUSED),+ ("../invoice.docx", "'..' segment", CoreCode.SECURITY_INPUT_PATH_REFUSED),+ ("templates/../../invoice.docx", "'..' segment", CoreCode.SECURITY_INPUT_PATH_REFUSED),+ ("..\\secrets\\key.pem", "'..' segment", CoreCode.SECURITY_INPUT_PATH_REFUSED),+ ("C:/Windows/invoice.docx", "drive", CoreCode.SECURITY_INPUT_PATH_REFUSED),+ ("C:invoice.docx", "drive", CoreCode.SECURITY_INPUT_PATH_REFUSED),+ ("\\\\server\\share\\invoice.docx", "network share", CoreCode.SECURITY_INPUT_PATH_REFUSED),+ ("templates\\invoice.docx", "backslash", CoreCode.REQUEST_INVALID),+ ("invoice:v2.docx", "colon", CoreCode.REQUEST_INVALID),+ ("invoice\x00.docx", "NUL", CoreCode.REQUEST_INVALID),],)- def test_a_path_that_cannot_name_a_file_inside_the_bundle_is_refused(self, path: str, reason: str) -> None:- """The message says why, and the lenient form answers None."""+ def test_a_path_that_cannot_name_a_file_inside_the_bundle_is_refused(self, path: str, reason: str, code: CoreCode) -> None:+ """The message says why, the lenient form answers None, and only a path leaving the bundle is a security refusal."""with pytest.raises(BundleResourcePathError) as exc_info:canonical_resource_path(path=path)assert reason in exc_info.value.message+ assert exc_info.value.resolve_error_code() == codeassert canonical_resource_path_or_none(path=path) is Nonediff --git a/tests/unit/pipelex/errors/test_security_event.py b/tests/unit/pipelex/errors/test_security_event.pynew file mode 100644index 000000000..2553f94f9--- /dev/null+++ b/tests/unit/pipelex/errors/test_security_event.py@@ -0,0 +1,425 @@+from __future__ import annotations++import io+import json+import logging+from contextlib import contextmanager+from enum import StrEnum+from typing import TYPE_CHECKING, Any++import pytest+from mthds.protocol.problem.registry import CoreCode, ProblemDomain+from typing_extensions import override++from pipelex.base_exceptions import PipelexError, PipelexUnexpectedError+from pipelex.cogt.content_generation.exceptions import UnsafeSchemaError+from pipelex.cogt.doc_gen.exceptions import TemplateUrlRefusedError+from pipelex.core.bundle_resources.exceptions import BundleResourcePathError+from pipelex.errors.error_code_declaration import InheritsCause+from pipelex.errors.error_codes import ErrorCode, error_code_spec+from pipelex.errors.problem import ProblemView+from pipelex.errors.registry.pipelex_codes import PipelexErrorCode+from pipelex.errors.security_event import (+ SECURITY_EVENT_MESSAGE,+ SECURITY_EVENT_NAME,+ SECURITY_TARGET_KIND_BY_CODE,+ emit_security_event,+ log_security_event,+ mark_security_event_emitted,+)+from pipelex.errors.security_target import SecurityTargetClass, SecurityTargetKind+from pipelex.pipeline.exceptions import PipelineInputContentError+from pipelex.tools.jinja2.exceptions import Jinja2TemplateSecurityError+from pipelex.tools.log.json_log_sink import LOGGER_KEY, MESSAGE_KEY, SEVERITY_KEY, TIME_KEY, JsonLogSink+from pipelex.tools.log.log_context import bind_log_context+from pipelex.tools.network.exceptions import SsrfBlockedError+from pipelex.tools.storage.exceptions import StorageInvalidUriError+from pipelex.tools.uri.exceptions import UriReadRefusalReason, UriReadRefusedError++if TYPE_CHECKING:+ from collections.abc import Generator++SECURITY_EVENT_LOGGER = "pipelex.errors.security_event"++# What a refusal's message may quote, which its security event must never carry.+_SECRET_HOST = "billing-db.internal.example"+_SECRET_KEY = "pipelex-storage://org_other/mt_9/run_9/assets/payroll.pdf"+_SECRET_TEMPLATE = "{{ invoice['_owner'] }}"+++@contextmanager+def _security_event_lines() -> Generator[list[dict[str, Any]]]:+ """The security events logged in the block, as the ``json`` sink serializes them."""+ buffer = io.StringIO()+ handler = JsonLogSink(stream=buffer).handler+ logger = logging.getLogger(SECURITY_EVENT_LOGGER)+ previous_level = logger.level+ logger.setLevel(logging.WARNING)+ logger.addHandler(handler)+ lines: list[dict[str, Any]] = []+ try:+ yield lines+ finally:+ logger.removeHandler(handler)+ logger.setLevel(previous_level)+ handler.close()+ lines.extend(json.loads(line) for line in buffer.getvalue().splitlines() if line)+++def _ssrf_refusal() -> SsrfBlockedError:+ message = f"Refusing to connect to '{_SECRET_HOST}': it resolved to a disallowed (private/loopback/metadata) address."+ return SsrfBlockedError(message).with_security_target_class(SecurityTargetClass.PRIVATE)+++class _ImpliedRefusalError(PipelexError):+ """A refusal that carries no code of its own and implies a security code, as an inference error implies its category's."""++ error_code = InheritsCause(fallback=CoreCode.RUNTIME_UNEXPECTED, others=(CoreCode.SECURITY_PROVIDER_URL_REFUSED,))++ @property+ @override+ def implied_error_code(self) -> ErrorCode | None:+ return CoreCode.SECURITY_PROVIDER_URL_REFUSED+++class _FallbackRefusalError(PipelexError):+ """A refusal that takes its cause's code, and names a security code as its fallback when no cause carries one."""++ error_code = InheritsCause(fallback=CoreCode.SECURITY_METHOD_PATH_REFUSED)+++def _chained(*, error: PipelexError, cause: BaseException) -> PipelexError:+ error.__cause__ = cause+ return error+++class _Chaining(StrEnum):+ """How a refusal raised while another is handled holds the first one."""++ IMPLICIT = "implicit"+ FROM_NONE = "from_none"+ FROM_REFUSAL = "from_refusal"+++def _refuse_again_while_handling(*, refusal: PipelexError, chaining: _Chaining) -> None:+ """Raise a new refusal of the same code while handling ``refusal``, as a second destination refused in its handler.++ ``chaining`` says how the new refusal holds the first: only as its ``__context__``, not at all, or as its ``__cause__``.+ """+ try:+ raise refusal+ except PipelexError as exc:+ msg = "Refusing to connect to the callback host."+ second_refusal = SsrfBlockedError(msg).with_security_target_class(SecurityTargetClass.LOOPBACK)+ match chaining:+ case _Chaining.IMPLICIT:+ raise second_refusal # ruff: ignore[raise-without-from-inside-except] — chained only as its context, on purpose+ case _Chaining.FROM_NONE:+ raise second_refusal from None+ case _Chaining.FROM_REFUSAL:+ raise second_refusal from exc+++def _refuse_another_while_handling(*, refusal: SsrfBlockedError) -> None:+ """Raise a refusal of another code while handling ``refusal``, chained to it only as its ``__context__``."""+ try:+ raise refusal+ except SsrfBlockedError:+ msg = "The schema of concept 'Invoice' carries x-python-* extensions."+ raise UnsafeSchemaError(msg) # ruff: ignore[raise-without-from-inside-except] — chained only as its context, on purpose+++def _wrapped(*, refusal: PipelexError) -> PipelineInputContentError:+ """The refusal under a wrapper that takes its cause's code, as every boundary a failure crosses wraps it."""+ try:+ raise refusal+ except PipelexError as exc:+ wrapper = PipelineInputContentError(f"The input 'photo' cannot be read: {exc.message}")+ wrapper.__cause__ = exc+ return wrapper+++class TestSecurityEvent:+ def test_every_security_code_names_the_kind_of_target_it_refuses(self) -> None:+ """Both registries' security families, and nothing else, have a target kind, so a new security code cannot ship without one."""+ security_codes: set[ErrorCode] = {code for code in CoreCode if code.startswith("security.")}+ security_codes |= {code for code in PipelexErrorCode if code.startswith("pipelex.security.")}+ assert security_codes+ assert set(SECURITY_TARGET_KIND_BY_CODE) == security_codes++ @pytest.mark.parametrize(+ ("error", "code", "domain", "error_type", "target_kind", "target_class"),+ [+ pytest.param(+ UriReadRefusedError("The input 'photo' names a file this run may not read.", reason=UriReadRefusalReason.LOCAL_PATH),+ CoreCode.SECURITY_INPUT_URL_REFUSED,+ ProblemDomain.INPUT,+ "UriReadRefusedError",+ SecurityTargetKind.URL,+ SecurityTargetClass.SCHEME,+ id="read_scope_input",+ ),+ pytest.param(+ UriReadRefusedError(+ "Image 1 of pipe 'describe' names a file this run may not read.", reason=UriReadRefusalReason.FOREIGN_STORAGE_KEY+ ).with_error_code(CoreCode.SECURITY_METHOD_URL_REFUSED),+ CoreCode.SECURITY_METHOD_URL_REFUSED,+ ProblemDomain.METHOD,+ "UriReadRefusedError",+ SecurityTargetKind.URL,+ SecurityTargetClass.FOREIGN_STORAGE_KEY,+ id="read_scope_method",+ ),+ pytest.param(+ _ssrf_refusal(),+ CoreCode.SECURITY_INPUT_URL_REFUSED,+ ProblemDomain.INPUT,+ "SsrfBlockedError",+ SecurityTargetKind.URL,+ SecurityTargetClass.PRIVATE,+ id="ssrf_input",+ ),+ pytest.param(+ _ssrf_refusal().with_error_code(CoreCode.SECURITY_PROVIDER_URL_REFUSED),+ CoreCode.SECURITY_PROVIDER_URL_REFUSED,+ ProblemDomain.RUNTIME,+ "SsrfBlockedError",+ SecurityTargetKind.URL,+ SecurityTargetClass.PRIVATE,+ id="ssrf_provider",+ ),+ pytest.param(+ UnsafeSchemaError("The schema of concept 'Invoice' carries x-python-* extensions."),+ CoreCode.SECURITY_SCHEMA_REFUSED,+ ProblemDomain.METHOD,+ "UnsafeSchemaError",+ SecurityTargetKind.SCHEMA,+ None,+ id="unsafe_schema",+ ),+ pytest.param(+ Jinja2TemplateSecurityError("Jinja2 render — refused by the template sandbox: A template may not read '_owner'."),+ CoreCode.SECURITY_TEMPLATE_REFUSED,+ ProblemDomain.METHOD,+ "Jinja2TemplateSecurityError",+ SecurityTargetKind.TEMPLATE,+ None,+ id="template_sandbox",+ ),+ pytest.param(+ BundleResourcePathError("The resource key escapes the bundle's root."),+ CoreCode.SECURITY_INPUT_PATH_REFUSED,+ ProblemDomain.INPUT,+ "BundleResourcePathError",+ SecurityTargetKind.PATH,+ None,+ id="bundle_resource_path",+ ),+ pytest.param(+ TemplateUrlRefusedError("The document template names a file outside the bundle.", refusal_code=None),+ CoreCode.SECURITY_METHOD_PATH_REFUSED,+ ProblemDomain.METHOD,+ "TemplateUrlRefusedError",+ SecurityTargetKind.PATH,+ None,+ id="doc_gen_template_path",+ ),+ pytest.param(+ _wrapped(refusal=StorageInvalidUriError("The storage key holds a path traversal.")),+ CoreCode.SECURITY_INPUT_PATH_REFUSED,+ ProblemDomain.INPUT,+ "StorageInvalidUriError",+ SecurityTargetKind.PATH,+ None,+ id="storage_key_traversal_under_its_wrapper",+ ),+ pytest.param(+ _wrapped(refusal=_ssrf_refusal()),+ CoreCode.SECURITY_INPUT_URL_REFUSED,+ ProblemDomain.INPUT,+ "SsrfBlockedError",+ SecurityTargetKind.URL,+ SecurityTargetClass.PRIVATE,+ id="ssrf_under_its_wrapper",+ ),+ pytest.param(+ _wrapped(refusal=_ImpliedRefusalError("The answer names an address this run may not fetch.")),+ CoreCode.SECURITY_PROVIDER_URL_REFUSED,+ ProblemDomain.RUNTIME,+ "_ImpliedRefusalError",+ SecurityTargetKind.URL,+ None,+ id="implied_code_under_its_wrapper",+ ),+ pytest.param(+ _wrapped(refusal=_chained(error=_FallbackRefusalError("The template names a file outside the bundle."), cause=OSError("denied"))),+ CoreCode.SECURITY_METHOD_PATH_REFUSED,+ ProblemDomain.METHOD,+ "_FallbackRefusalError",+ SecurityTargetKind.PATH,+ None,+ id="fallback_code_under_its_wrapper",+ ),+ ],+ )+ def test_a_refusal_fails_under_its_security_code_and_logs_its_event(+ self,+ error: PipelexError,+ code: ErrorCode,+ domain: ProblemDomain,+ error_type: str,+ target_kind: SecurityTargetKind,+ target_class: SecurityTargetClass | None,+ ) -> None:+ """The failure reports the security code with the domain of whoever caused it, and its event names the same, with the run and the org."""+ problem = error.to_problem(view=ProblemView.CALLER)+ assert problem.code == code+ # The registry domain of the code, who caused the refusal, which the problem document carries.+ assert error_code_spec(code).domain == domain+ assert problem.domain == domain++ with _security_event_lines() as lines:+ assert emit_security_event(error=error, pipeline_run_id="run_1", request_id="req_1", extras={"org_id": "org_abc", "plan": "pro"})++ (line,) = lines+ expected: dict[str, Any] = {+ SEVERITY_KEY: "WARNING",+ LOGGER_KEY: SECURITY_EVENT_LOGGER,+ MESSAGE_KEY: SECURITY_EVENT_MESSAGE,+ "event.name": SECURITY_EVENT_NAME,+ "error.code": code,+ "error.domain": domain,+ "error.type": error_type,+ "security.target": target_kind,+ "pipeline_run_id": "run_1",+ "request_id": "req_1",+ "run.extras.org_id": "org_abc",+ "run.extras.plan": "pro",+ }+ if target_class is not None:+ expected["security.target_class"] = target_class+ line.pop(TIME_KEY)+ # The exact record: nothing else rides it, so neither the message nor a traceback can carry content.+ assert line == expected++ @pytest.mark.parametrize(+ ("error", "secret"),+ [+ pytest.param(_wrapped(refusal=_ssrf_refusal()), _SECRET_HOST, id="ssrf_host"),+ pytest.param(+ UriReadRefusedError(f"The input 'photo' names {_SECRET_KEY}.", reason=UriReadRefusalReason.FOREIGN_STORAGE_KEY),+ _SECRET_KEY,+ id="foreign_storage_key",+ ),+ pytest.param(Jinja2TemplateSecurityError(f"Refused by the template sandbox: {_SECRET_TEMPLATE}"), _SECRET_TEMPLATE, id="template_text"),+ ],+ )+ def test_the_event_carries_no_content_even_when_the_message_does(self, error: PipelexError, secret: str) -> None:+ """A message quoting the URL, the key or the template stays off the event, logged while the failure is being handled."""+ assert secret in str(error) or (error.__cause__ is not None and secret in str(error.__cause__))+ emitted: list[bool] = []+ with _security_event_lines() as lines:+ try:+ raise error+ except PipelexError as exc:+ # Inside the `except`, as every boundary logs it: the exception being handled must not ride the record.+ emitted.append(emit_security_event(error=exc))++ assert emitted == [True]+ (line,) = lines+ serialized = json.dumps(line)+ assert secret not in serialized+ assert "exception" not in line+ assert line[MESSAGE_KEY] == SECURITY_EVENT_MESSAGE++ def test_a_refusal_is_reported_once_however_many_boundaries_it_crosses(self) -> None:+ refusal = _ssrf_refusal()+ wrapper = _wrapped(refusal=refusal)+ outer_wrapper = PipelineInputContentError("The run failed.")+ outer_wrapper.__cause__ = wrapper++ with _security_event_lines() as lines:+ assert emit_security_event(error=wrapper, pipeline_run_id="run_1")+ assert not emit_security_event(error=wrapper, pipeline_run_id="run_1")+ assert not emit_security_event(error=outer_wrapper)+ assert not emit_security_event(error=refusal)++ assert len(lines) == 1++ @pytest.mark.parametrize("chaining", list(_Chaining))+ @pytest.mark.parametrize("reported_as", ["refusal", "wrapper"])+ def test_a_new_refusal_of_the_same_code_is_reported_however_it_is_chained_to_a_reported_one(self, chaining: _Chaining, reported_as: str) -> None:+ """A wrapper restates the refusal it wraps; a new refusal is reported on its own, even of the same code.++ Such as a callback URL the delivery refuses while the run's own refusal of an input URL is being handled: both+ are `security.input_url_refused`, and each is a refusal of its own, whether the second holds the first as its+ context, not at all, or as its cause, and whether the first was reported bare or under a wrapper.+ """+ refusal = _ssrf_refusal()+ reported: PipelexError = _wrapped(refusal=refusal) if reported_as == "wrapper" else refusal+ with _security_event_lines() as lines:+ assert emit_security_event(error=reported)+ with pytest.raises(SsrfBlockedError) as exc_info:+ _refuse_again_while_handling(refusal=reported, chaining=chaining)+ second_refusal = exc_info.value+ assert emit_security_event(error=_wrapped(refusal=second_refusal))+ assert not emit_security_event(error=second_refusal)++ assert [line["security.target_class"] for line in lines] == [SecurityTargetClass.PRIVATE, SecurityTargetClass.LOOPBACK]+ assert [line["error.code"] for line in lines] == [CoreCode.SECURITY_INPUT_URL_REFUSED] * 2++ def test_a_distinct_refusal_raised_while_handling_a_reported_one_is_reported(self) -> None:+ """`__context__` only ties a refusal to the one it restates: another code, raised while handling the first, is its own."""+ refusal = _ssrf_refusal()+ with _security_event_lines() as lines:+ assert emit_security_event(error=refusal)+ with pytest.raises(UnsafeSchemaError) as exc_info:+ _refuse_another_while_handling(refusal=refusal)+ assert emit_security_event(error=exc_info.value)++ assert exc_info.value.__context__ is refusal+ assert [line["error.code"] for line in lines] == [CoreCode.SECURITY_INPUT_URL_REFUSED, CoreCode.SECURITY_SCHEMA_REFUSED]++ def test_a_failure_whose_event_was_logged_elsewhere_is_not_reported_again(self) -> None:+ """A host marks the error a transport rebuilt from a worker's report, whose worker logged the event."""+ rebuilt = _ssrf_refusal()+ mark_security_event_emitted(error=rebuilt)+ with _security_event_lines() as lines:+ assert not emit_security_event(error=rebuilt)+ assert lines == []++ @pytest.mark.parametrize(+ "error",+ [+ pytest.param(PipelexUnexpectedError("An invariant did not hold."), id="not_a_security_code"),+ pytest.param(+ Jinja2TemplateSecurityError("Refused by the template sandbox.").with_error_code(CoreCode.RUNTIME_UNEXPECTED),+ id="pipelex_own_template_is_runtime_unexpected",+ ),+ pytest.param(ValueError("not a Pipelex error"), id="not_a_pipelex_error"),+ ],+ )+ def test_a_failure_that_is_no_security_refusal_logs_no_event(self, error: BaseException) -> None:+ with _security_event_lines() as lines:+ assert not emit_security_event(error=error)+ assert lines == []++ def test_the_bound_log_context_names_the_run_when_the_caller_does_not(self) -> None:+ with _security_event_lines() as lines, bind_log_context(request_id="req_ctx", pipeline_run_id="run_ctx"):+ assert emit_security_event(error=_ssrf_refusal())++ (line,) = lines+ assert line["request_id"] == "req_ctx"+ assert line["pipeline_run_id"] == "run_ctx"++ def test_a_host_logs_an_event_for_a_refusal_it_decides_itself(self) -> None:+ """A host's own tenant-boundary refusal logs the same record, with no target kind outside the security families."""+ with _security_event_lines() as lines:+ log_security_event(code=PipelexErrorCode.ORG_MISMATCH, error_type="CrossOrgStorageAccessError", extras={"org_id": "org_abc"})++ (line,) = lines+ assert line["event.name"] == SECURITY_EVENT_NAME+ assert line["error.code"] == PipelexErrorCode.ORG_MISMATCH+ assert line["run.extras.org_id"] == "org_abc"+ assert "security.target" not in line+ assert "security.target_class" not in linediff --git a/tests/unit/pipelex/pipe_run/test_pipe_run.py b/tests/unit/pipelex/pipe_run/test_pipe_run.pyindex ca42fbbd2..0d774b990 100644--- a/tests/unit/pipelex/pipe_run/test_pipe_run.py+++ b/tests/unit/pipelex/pipe_run/test_pipe_run.py@@ -9,12 +9,17 @@ from typing_extensions import overridefrom pipelex.base_exceptions import PipelexErrorfrom pipelex.errors.problem import PipelexProblem+from pipelex.errors.security_event import SECURITY_EVENT_MESSAGE, emit_security_event+from pipelex.errors.security_target import SecurityTargetClassfrom pipelex.pipe_run import pipe_run as pipe_run_modulefrom pipelex.pipe_run.delivery_assignment import DeliveryAssignment, DeliveryStatus, StorageTarget, WebhookTargetfrom pipelex.pipe_run.exceptions import PipeRouterError, WebhookDeliveryErrorfrom pipelex.pipe_run.pipe_run import PipeRunfrom pipelex.system.job_metadata import JobMetadata, RunMetadatafrom pipelex.system.pipe_run_mode import PipeRunMode+from pipelex.tools.network.exceptions import SsrfBlockedError++_SECURITY_EVENT_LOGGER = "pipelex.errors.security_event"_WEBHOOK_URL = "https://hooks.example.test/runs"@@ -163,6 +168,54 @@ class TestPipeRun:assert record_fields["error.message"] == "router blew up"assert failure_record.exc_info is None+ async def test_a_refusal_the_delivery_raises_logs_its_security_event_once(self, mocker: MockerFixture, caplog: pytest.LogCaptureFixture) -> None:+ """A webhook the SSRF guard refuses at fire time fails a run that completed: the run that raised it logs its event."""+ mock_router = mocker.AsyncMock()+ mock_router.run = mocker.AsyncMock(return_value=mocker.MagicMock())+ refusal = SsrfBlockedError("Refusing to connect to 'hooks.internal.example'.").with_security_target_class(SecurityTargetClass.PRIVATE)+ mock_executor = mocker.patch("pipelex.pipe_run.pipe_run.DeliveryExecutor")+ mock_executor.return_value.execute = mocker.AsyncMock(side_effect=refusal)+ mock_job = mocker.MagicMock()+ mock_job.job_metadata = JobMetadata(+ run_metadata=RunMetadata(+ user_id="pytest", pipeline_run_id="plr-webhook-refused", storage_scope="test/scope", read_scope=None, extras={"org_id": "org_abc"}+ )+ )++ with caplog.at_level(logging.WARNING, logger=_SECURITY_EVENT_LOGGER), pytest.raises(SsrfBlockedError):+ await PipeRun(pipe_router=mock_router).run(pipe_job=mock_job, delivery_assignment=DeliveryAssignment(storage=StorageTarget()))++ (event,) = [record for record in caplog.records if record.getMessage() == SECURITY_EVENT_MESSAGE]+ event_fields = vars(event)+ assert event_fields["error.code"] == CoreCode.SECURITY_INPUT_URL_REFUSED+ assert event_fields["security.target_class"] == SecurityTargetClass.PRIVATE+ assert event_fields["pipeline_run_id"] == "plr-webhook-refused"+ assert event_fields["run.extras.org_id"] == "org_abc"+ assert not emit_security_event(error=refusal)++ async def test_a_run_refused_then_its_webhook_refused_logs_both_refusals(self, mocker: MockerFixture, caplog: pytest.LogCaptureFixture) -> None:+ """A run its input refused, whose FAILED webhook the SSRF guard then refuses: two refusals of one code, two events."""+ input_refusal = SsrfBlockedError("Refusing to connect to 'files.internal.example'.").with_security_target_class(SecurityTargetClass.PRIVATE)+ webhook_refusal = SsrfBlockedError("Refusing to connect to 'hooks.internal.example'.").with_security_target_class(+ SecurityTargetClass.LOOPBACK+ )+ mock_router = mocker.AsyncMock()+ mock_router.run = mocker.AsyncMock(side_effect=input_refusal)+ mock_executor = mocker.patch("pipelex.pipe_run.pipe_run.DeliveryExecutor")+ mock_executor.return_value.execute = mocker.AsyncMock(side_effect=webhook_refusal)+ mock_job = mocker.MagicMock()+ mock_job.job_metadata = JobMetadata(+ run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-both-refused", storage_scope="test/scope", read_scope=None)+ )++ with caplog.at_level(logging.WARNING, logger=_SECURITY_EVENT_LOGGER), pytest.raises(SsrfBlockedError) as exc_info:+ await PipeRun(pipe_router=mock_router).run(pipe_job=mock_job, delivery_assignment=DeliveryAssignment(storage=StorageTarget()))++ assert exc_info.value is webhook_refusal+ events = [vars(record) for record in caplog.records if record.getMessage() == SECURITY_EVENT_MESSAGE]+ assert [event["security.target_class"] for event in events] == [SecurityTargetClass.PRIVATE, SecurityTargetClass.LOOPBACK]+ assert [event["error.code"] for event in events] == [CoreCode.SECURITY_INPUT_URL_REFUSED] * 2+async def test_run_success_no_delivery_when_none(self, mocker: MockerFixture) -> None:"""When delivery_assignment is None, the delivery executor is not called."""mock_output = mocker.MagicMock()diff --git a/tests/unit/pipelex/pipeline/test_runner_execute_error_paths.py b/tests/unit/pipelex/pipeline/test_runner_execute_error_paths.pyindex 842c9f417..c6777ce5d 100644--- a/tests/unit/pipelex/pipeline/test_runner_execute_error_paths.py+++ b/tests/unit/pipelex/pipeline/test_runner_execute_error_paths.py@@ -8,22 +8,29 @@ the finally-block library-restore matrix are pinned by the integration suitefrom __future__ import annotations+import loggingfrom typing import TYPE_CHECKING, NamedTupleimport pytest+from mthds.protocol.problem.registry import CoreCodefrom pydantic import BaseModel, ValidationErrorfrom typing_extensions import override+from pipelex.errors.security_event import SECURITY_EVENT_MESSAGE, emit_security_event, mark_security_event_emitted+from pipelex.errors.security_target import SecurityTargetClassfrom pipelex.pipe_run.pipe_run_protocol import PipeRunProtocolfrom pipelex.pipeline.exceptions import PipeExecutionError, PipelineExecutionErrorfrom pipelex.pipeline.runner import PipelexMTHDSProtocol+from pipelex.system.job_metadata import RunMetadatafrom pipelex.system.pipe_run_mode import PipeRunModefrom pipelex.system.telemetry.events import EventName, EventProperty, Outcome+from pipelex.tools.network.exceptions import SsrfBlockedErrorfrom pipelex.tools.typing.pydantic_utils import format_pydantic_validation_errorif TYPE_CHECKING:from pytest_mock import MockerFixture, MockType+ from pipelex.base_exceptions import PipelexErrorfrom pipelex.core.pipes.pipe_output import PipeOutputfrom pipelex.pipe_run.delivery_assignment import DeliveryAssignmentfrom pipelex.pipe_run.pipe_job import PipeJob@@ -46,6 +53,22 @@ class _RaisingPipeRun(PipeRunProtocol):raise self._exc+class _RebuiltFailurePipeRun(PipeRunProtocol):+ """A pipe run that hands the job to another process and raises the failure it rebuilt from that process's report.++ The other process logged the security event of a refusal where it raised it, so the rebuilt failure is marked as+ reported, as an orchestrator marks every error it rebuilds.+ """++ def __init__(self, exc: PipelexError):+ self._exc = exc++ @override+ async def run(self, pipe_job: PipeJob, *, delivery_assignment: DeliveryAssignment | None = None) -> PipeOutput:+ mark_security_event_emitted(error=self._exc)+ raise self._exc++class _ExecuteEnv(NamedTuple):setup_mock: MockTypetelemetry_manager: MockType@@ -55,6 +78,7 @@ class _ExecuteEnv(NamedTuple):_RUN_ID = "run-id-123"_LIB_ID = "lib-id-456"+SECURITY_EVENT_LOGGER = "pipelex.errors.security_event"@pytest.mark.asyncio(loop_scope="class")@@ -127,6 +151,52 @@ class TestRunnerExecuteErrorPaths:run_metadata=env.pipe_job.job_metadata.run_metadata,)+ async def test_a_refusal_raised_in_this_process_that_the_pipe_run_left_unreported_is_logged_by_the_runner(+ self, mocker: MockerFixture, caplog: pytest.LogCaptureFixture+ ) -> None:+ """A pipe run that hands the job to another process can be refused here first, before it dispatches.++ Such as an input URL the SSRF guard refuses while the pipe run uploads the inputs in this process. The refusal+ never left this process and nobody logged it, so the runner logs it, once, naming the run, and the API handler+ above it finds it reported.+ """+ env = self._patch_env(mocker)+ env.pipe_job.job_metadata.run_metadata = RunMetadata(+ user_id="pytest", pipeline_run_id=_RUN_ID, storage_scope="test/scope", read_scope=None, extras={"org_id": "org_abc"}+ )+ refusal = SsrfBlockedError("Refusing to connect to '10.0.0.5'.").with_security_target_class(SecurityTargetClass.PRIVATE)+ runner = PipelexMTHDSProtocol(execution_config=env.execution_config, pipe_run=_RaisingPipeRun(refusal))++ with caplog.at_level(logging.WARNING, logger=SECURITY_EVENT_LOGGER), pytest.raises(PipelineExecutionError) as exc_info:+ await runner.execute(pipe_code="echo_pipe")++ assert exc_info.value.resolve_error_code() == CoreCode.SECURITY_INPUT_URL_REFUSED+ (event,) = [vars(record) for record in caplog.records if record.getMessage() == SECURITY_EVENT_MESSAGE]+ assert event["error.code"] == CoreCode.SECURITY_INPUT_URL_REFUSED+ assert event["error.type"] == "SsrfBlockedError"+ assert event["pipeline_run_id"] == _RUN_ID+ assert event["run.extras.org_id"] == "org_abc"+ assert not emit_security_event(error=exc_info.value)++ async def test_a_failure_rebuilt_from_another_process_s_report_is_not_logged_again(+ self, mocker: MockerFixture, caplog: pytest.LogCaptureFixture+ ) -> None:+ """A refusal raised in another process was logged there: the failure rebuilt from its report carries the mark.++ The stand-in pipe run marks the failure it raises, as an orchestrator marks every error it rebuilds from a+ worker's report, so neither the runner nor the API handler above it logs the refusal a second time.+ """+ env = self._patch_env(mocker)+ rebuilt = SsrfBlockedError("Refusing to connect to '10.0.0.5'.").with_security_target_class(SecurityTargetClass.PRIVATE)+ runner = PipelexMTHDSProtocol(execution_config=env.execution_config, pipe_run=_RebuiltFailurePipeRun(rebuilt))++ with caplog.at_level(logging.WARNING, logger=SECURITY_EVENT_LOGGER), pytest.raises(PipelineExecutionError) as exc_info:+ await runner.execute(pipe_code="echo_pipe")++ assert exc_info.value.resolve_error_code() == CoreCode.SECURITY_INPUT_URL_REFUSED+ assert [record for record in caplog.records if record.getMessage() == SECURITY_EVENT_MESSAGE] == []+ assert not emit_security_event(error=exc_info.value)+async def test_pipelex_error_during_setup_propagates_unwrapped(self, mocker: MockerFixture) -> None:"""When pipeline_run_setup itself raises (pipe_job is None), the SAME exceptioninstance propagates unwrapped and no failure telemetry event fires.diff --git a/tests/unit/pipelex/tools/network/test_host_rules.py b/tests/unit/pipelex/tools/network/test_host_rules.pyindex 9ebd811ba..35915c082 100644--- a/tests/unit/pipelex/tools/network/test_host_rules.py+++ b/tests/unit/pipelex/tools/network/test_host_rules.py@@ -1,6 +1,7 @@import pytest-from pipelex.tools.network.host_rules import is_disallowed_host, is_disallowed_ip+from pipelex.errors.security_target import SecurityTargetClass+from pipelex.tools.network.host_rules import classify_disallowed_host, is_disallowed_host, is_disallowed_ipclass TestHostRules:@@ -20,6 +21,8 @@ class TestHostRules:("0.0.0.0", True), # ruff: ignore[hardcoded-bind-all-interfaces] # unspecified address — asserting it is disallowed("::1", True), # IPv6 loopback("fe80::1", True), # IPv6 link-local+ ("::ffff:127.0.0.1", True), # IPv4-mapped loopback+ ("::ffff:169.254.169.254", True), # IPv4-mapped metadata endpoint("8.8.8.8", False), # public("1.1.1.1", False), # public("93.184.216.34", False), # public (example.com)@@ -56,3 +59,40 @@ class TestHostRules:)def test_is_disallowed_host(self, host: str, expected: bool) -> None:assert is_disallowed_host(host) is expected++ @pytest.mark.parametrize(+ ("host", "expected"),+ [+ ("169.254.169.254", SecurityTargetClass.METADATA), # the cloud metadata endpoint, link-local but named apart+ ("fd00:ec2::254", SecurityTargetClass.METADATA), # AWS's IPv6 metadata endpoint+ ("169.254.170.2", SecurityTargetClass.METADATA), # AWS ECS task metadata and container credentials+ ("169.254.170.23", SecurityTargetClass.METADATA), # AWS EKS Pod Identity credentials+ ("fd00:ec2::23", SecurityTargetClass.METADATA), # AWS EKS Pod Identity credentials, IPv6+ ("100.100.100.200", SecurityTargetClass.METADATA), # Alibaba Cloud metadata+ ("192.0.0.192", SecurityTargetClass.METADATA), # Oracle Cloud metadata+ ("::ffff:169.254.169.254", SecurityTargetClass.METADATA), # IPv4-mapped: classified as the IPv4 address it carries+ ("::ffff:169.254.170.2", SecurityTargetClass.METADATA),+ ("::ffff:127.0.0.1", SecurityTargetClass.LOOPBACK),+ ("::ffff:169.254.1.1", SecurityTargetClass.LINK_LOCAL),+ ("::ffff:10.0.0.5", SecurityTargetClass.PRIVATE),+ ("metadata", SecurityTargetClass.METADATA),+ ("METADATA.google.internal.", SecurityTargetClass.METADATA), # normalized like is_disallowed_host+ ("localhost", SecurityTargetClass.LOOPBACK),+ ("LocalHost.", SecurityTargetClass.LOOPBACK),+ ("127.0.0.1", SecurityTargetClass.LOOPBACK),+ ("127.8.9.10", SecurityTargetClass.LOOPBACK),+ ("::1", SecurityTargetClass.LOOPBACK),+ ("169.254.1.1", SecurityTargetClass.LINK_LOCAL),+ ("fe80::1", SecurityTargetClass.LINK_LOCAL),+ ("10.0.0.5", SecurityTargetClass.PRIVATE),+ ("192.168.1.1", SecurityTargetClass.PRIVATE),+ ("100.64.0.1", SecurityTargetClass.PRIVATE), # carrier-grade NAT: not publicly routable+ ("224.0.0.1", SecurityTargetClass.PRIVATE), # multicast: refused, and no public destination+ ("0.0.0.0", SecurityTargetClass.PRIVATE), # ruff: ignore[hardcoded-bind-all-interfaces] # unspecified address+ ("", None), # refused, but nothing to classify+ ("8.8.8.8", None), # not refused at all+ ("example.com", None), # a public name is classified only by what it resolves to+ ],+ )+ def test_classify_disallowed_host(self, host: str, expected: SecurityTargetClass | None) -> None:+ assert classify_disallowed_host(host) == expecteddiff --git a/tests/unit/pipelex/tools/network/test_ssrf_guard.py b/tests/unit/pipelex/tools/network/test_ssrf_guard.pyindex 8fdf799f0..76822c762 100644--- a/tests/unit/pipelex/tools/network/test_ssrf_guard.py+++ b/tests/unit/pipelex/tools/network/test_ssrf_guard.py@@ -7,6 +7,7 @@ import httpximport pytestfrom pytest_mock import MockerFixture+from pipelex.errors.security_target import SecurityTargetClassfrom pipelex.tools.network.exceptions import SsrfBlockedErrorfrom pipelex.tools.network.ssrf_guard import SsrfGuardedBackend, SsrfGuardedTransport, resolve_to_allowed_ips@@ -39,13 +40,27 @@ class TestSsrfGuard:return _addrinfo("169.254.169.254", 80)mocker.patch("socket.getaddrinfo", side_effect=fake)- with pytest.raises(SsrfBlockedError):+ with pytest.raises(SsrfBlockedError) as exc_info:await resolve_to_allowed_ips("attacker.example", port=80)-- async def test_resolve_blocks_literal_private_host(self) -> None:- # A literal private IP never needs resolution — blocked by the cheap rule.- with pytest.raises(SsrfBlockedError):- await resolve_to_allowed_ips("10.0.0.5", port=80)+ # The refusal states the class of what the name resolved to, for its security event, and its message never names the address.+ assert exc_info.value.security_target_class == SecurityTargetClass.METADATA+ assert "169.254.169.254" not in exc_info.value.message++ @pytest.mark.parametrize(+ ("host", "target_class"),+ [+ ("10.0.0.5", SecurityTargetClass.PRIVATE),+ ("127.0.0.1", SecurityTargetClass.LOOPBACK),+ ("localhost", SecurityTargetClass.LOOPBACK),+ ("metadata.google.internal", SecurityTargetClass.METADATA),+ ("169.254.7.7", SecurityTargetClass.LINK_LOCAL),+ ],+ )+ async def test_resolve_blocks_literal_private_host(self, host: str, target_class: SecurityTargetClass) -> None:+ # A literal private IP never needs resolution — blocked by the cheap rule, and classified for the security event.+ with pytest.raises(SsrfBlockedError) as exc_info:+ await resolve_to_allowed_ips(host, port=80)+ assert exc_info.value.security_target_class == target_classasync def test_resolve_blocks_mixed_public_and_private(self, mocker: MockerFixture) -> None:# A host resolving to both a public and a private IP is itself a rebinding@@ -54,8 +69,10 @@ class TestSsrfGuard:return _addrinfo("93.184.216.34", 80) + _addrinfo("127.0.0.1", 80)mocker.patch("socket.getaddrinfo", side_effect=fake)- with pytest.raises(SsrfBlockedError):+ with pytest.raises(SsrfBlockedError) as exc_info:await resolve_to_allowed_ips("rebind.example", port=80)+ # The disallowed address among the answers classifies the refusal, not the public one beside it.+ assert exc_info.value.security_target_class == SecurityTargetClass.LOOPBACKasync def test_resolution_failure_maps_to_connect_error(self, mocker: MockerFixture) -> None:# A genuine DNS failure is not a security event — surface httpx's usual connect error.diff --git a/tests/unit/pipelex/tools/uri/test_uri_read_scope.py b/tests/unit/pipelex/tools/uri/test_uri_read_scope.pyindex c88e3e5fb..344e6f9f4 100644--- a/tests/unit/pipelex/tools/uri/test_uri_read_scope.py+++ b/tests/unit/pipelex/tools/uri/test_uri_read_scope.py@@ -5,6 +5,7 @@ from mthds.protocol.problem.registry import CoreCode, ProblemDomainfrom pipelex.errors.error_codes import error_code_specfrom pipelex.errors.problem import ProblemView+from pipelex.errors.security_target import SecurityTargetClassfrom pipelex.tools.uri.exceptions import UriReadRefusalReason, UriReadRefusedErrorfrom pipelex.tools.uri.uri_read_scope import UriReference, authorize_uri_read, authorize_uri_reads@@ -49,6 +50,8 @@ class TestAuthorizeUriRead:with pytest.raises(UriReadRefusedError) as exc_info:authorize_uri_read(uri=uri, read_scope=READ_SCOPE, position="image 1 of the prompt")assert exc_info.value.reason == UriReadRefusalReason.FOREIGN_STORAGE_KEY+ # The security event names the class of what was refused, never the key.+ assert exc_info.value.security_target_class == SecurityTargetClass.FOREIGN_STORAGE_KEY@pytest.mark.parametrize("uri",@@ -68,6 +71,8 @@ class TestAuthorizeUriRead:with pytest.raises(UriReadRefusedError) as exc_info:authorize_uri_read(uri=uri, read_scope=READ_SCOPE, position="the input 'photo'")assert exc_info.value.reason == UriReadRefusalReason.LOCAL_PATH+ # A path on the server's disk, or a scheme the run does not read, is a refused scheme to the security event.+ assert exc_info.value.security_target_class == SecurityTargetClass.SCHEMEdef test_a_multi_segment_scope_compares_every_segment(self) -> None:authorize_uri_read(uri="pipelex-storage://org_abc/team_1/x.png", read_scope="org_abc/team_1", position="p")