Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
feature/Run-metadata-org-id · L-260917-01d281 (#1306)
Thomas Hebrardf4304035dcfe
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 5d4e04c06..3107dc30b 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,6 +9,7 @@- **The `markdown` template filter**: an HTML template turns Markdown held in a text field into HTML with `{{ order.notes | markdown }}`. Raw HTML in the source is shown as text, and only URLs with a scheme become links.- **The `Markdown` native concept**: `Markdown` refines `Text` and holds a text written in Markdown, which a `PipeLLM` with `output = "Markdown"` writes as free text and a `Markdown` input takes as a string or as `{"concept": "Markdown", "content": "…"}`; Pipelex defines it ahead of the MTHDS standard. A `Markdown` value is accepted wherever a `Text` is, while a `Text`, or any concept refining `Text`, is refused where a `Markdown` is required. `pipelex run --save-main-stuff` saves it verbatim as `main_stuff.md`, an HTML template prints `{{ report }}` and `$report` as HTML with raw HTML escaped, and `Stuff.as_markdown` and `PipeOutput.main_stuff_as_markdown` read it from Python.- **The Pipelex API server ships from this repository**: the FastAPI runner published as the `pipelex/pipelex-api` image now lives in the `api/` directory, as the `pipelex-api` member of a uv workspace, and it runs the `pipelex` of the same commit instead of a pinned release. It is released together with `pipelex`, under the same version, so the image tag that follows `0.33.2` is a `pipelex` version; the image keeps its name, its port and its `/root/.pipelex` configuration mount, and `Pipelex/pipelex-api` keeps the server's earlier history.+- **`x-pipelex-metadata` header on Pipelex Manifold calls**: every inference request the Manifold backend makes (chat completions, responses, image generation, extraction, search, and Claude over the Anthropic SDK) carries one JSON header holding the run's `extras` as they are, plus the run's `user_id` and `pipeline_run_id` and the step's `pipe_run_id`, `pipe_code` and `content_generation_job_id` when set, so the service can attribute each call's spend and log it. On a key collision the run's own id wins over the extras entry, and no other backend receives the header.### Changeddiff --git a/docs/setup/telemetry.md b/docs/setup/telemetry.mdindex 28877b418..6c296a963 100644--- a/docs/setup/telemetry.md+++ b/docs/setup/telemetry.md@@ -131,6 +131,12 @@ Anything that names no caller of its own keeps reporting under the `user_id` youThe extras do not depend on that. A run that leaves `user_id` at its default still carries its `extras` onto every capture, under your configured id — knowing which entities a run belongs to and naming its caller are two separate decisions, and you may take one without the other.+The extras also travel with the run's inference calls when they go through Pipelex Manifold. Every request the Manifold backend makes carries one `x-pipelex-metadata` header: a flat JSON object holding the run's extras as they are, plus the run's `user_id` and `pipeline_run_id` and the step's `pipe_run_id`, `pipe_code` and `content_generation_job_id` when it has them, so the service can attribute and log each call. If an extras key has the same name as one of those ids, the run's own value is the one sent, since the extras are yours to set and must not be able to restate whose run it is. No other backend receives this header: a provider you reach directly with your own key gets none of your labels.++```text+x-pipelex-metadata: {"org_id":"org_acme","user_id":"user_42","pipeline_run_id":"run_abc","pipe_run_id":"0123456789abcdef","pipe_code":"summarize_doc"}+```+### Work that is not a run, and crashesA run is not the only thing a host does on a caller's behalf. Validating a bundle dry-runs its pipes without being a run, and an unhandled exception reaches PostHog through the interpreter's exception hook, which is handed nothing but the error. Both are still attributed to the caller when one is known, so your configured `user_id` stands only for work that truly belongs to nobody:diff --git a/pipelex/providers/anthropic/anthropic_llm_worker.py b/pipelex/providers/anthropic/anthropic_llm_worker.pyindex 82de182b3..2216a7e00 100644--- a/pipelex/providers/anthropic/anthropic_llm_worker.py+++ b/pipelex/providers/anthropic/anthropic_llm_worker.py@@ -32,6 +32,7 @@ from pipelex.cogt.llm.llm_utils import (from pipelex.cogt.llm.llm_worker_abstract import LLMWorkerAbstractfrom pipelex.cogt.llm.structured_output import StructureMethodfrom pipelex.cogt.llm.thinking_mode import ThinkingMode+from pipelex.cogt.model_backends.backend import PipelexBackendfrom pipelex.cogt.model_backends.constraints import ListedConstraintfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpecfrom pipelex.config import get_config@@ -42,6 +43,7 @@ from pipelex.providers.anthropic.anthropic_factory import (AnthropicFactory,AnthropicSdkVariant,)+from pipelex.providers.manifold.manifold_metadata import make_manifold_metadata_headersfrom pipelex.reporting.reporting_protocol import ReportingProtocolfrom pipelex.tools.typing.pydantic_utils import BaseModelTypeVar@@ -249,6 +251,7 @@ class AnthropicLLMWorker(LLMWorkerAbstract):max_tokens=max_tokens,thinking=thinking_params.thinking or omit,output_config=thinking_params.output_config or omit,+ **self._request_header_kwargs(llm_job=llm_job),) as stream:final_message: Message = await stream.get_final_message()except (APIStatusError, APIConnectionError) as sdk_exc:@@ -305,6 +308,19 @@ class AnthropicLLMWorker(LLMWorkerAbstract):return full_reply_content+ def _request_header_kwargs(self, *, llm_job: LLMJob) -> dict[str, Any]:+ """The per-request headers this call adds, as SDK keyword arguments: none, except behind Manifold.++ Claude reaches the Pipelex Manifold service over this shared driver rather than over a+ manifold sdk, so this is where the manifold dialect's `x-pipelex-metadata` header joins an+ Anthropic request — per request, because it names the job, where the token is a client+ default. Every other Anthropic backend is a direct provider SDK path and sends nothing of the+ kind: the run's identity and labels are ours to forward to our own service, not to a vendor.+ """+ if self.inference_model.backend_name != PipelexBackend.MANIFOLD:+ return {}+ return {"extra_headers": make_manifold_metadata_headers(job_metadata=llm_job.job_metadata)}+def _structure_method_kwargs(self) -> dict[str, Any]:"""What the structured call sends, beyond what instructor's mode sets, for the model's structure method.@@ -361,6 +377,7 @@ class AnthropicLLMWorker(LLMWorkerAbstract):max_tokens=effective_max_tokens,timeout=float(timeout_seconds), # Explicit timeout disables SDK's long-request protection**self._structure_method_kwargs(),+ **self._request_header_kwargs(llm_job=llm_job),)except InstructorRetryException as instructor_exc:# instructor wraps SDK exceptions during retries; recover the underlyingdiff --git a/pipelex/providers/manifold/manifold_constants.py b/pipelex/providers/manifold/manifold_constants.pyindex 8ecbf7cba..34d062bf2 100644--- a/pipelex/providers/manifold/manifold_constants.py+++ b/pipelex/providers/manifold/manifold_constants.py@@ -1,4 +1,4 @@-"""The manifold dialect's names: its sdk set, its one wire header, and its route paths.+"""The manifold dialect's names: its sdk set, its two wire headers, and its route paths.Everything the manifold plugin needs to name is named here rather than imported from``providers/portkey/`` or ``providers/gateway/``. The duplication is deliberate — it is the@@ -20,6 +20,15 @@ from enum import StrEnum# `backends.toml` must spell it exactly as this constant does.MANIFOLD_AUTH_HEADER = "x-pipelex-api-key"+# The header that says whose call this is, sent on every request beside the token.+#+# Its value is a flat JSON object of string to string: the run's opaque `extras` and the runtime's+# own ids for the run and the step (see `manifold_metadata`). It differs per job, so unlike the token+# it is never a client default: every path adds it per request. The vendor spelling of the same idea,+# `x-portkey-metadata`, is never sent — the gateway reads this name, and the dialect speaks none of+# the vendor's vocabulary it does not have to.+MANIFOLD_METADATA_HEADER = "x-pipelex-metadata"+# The gateway's response headers are named in `cogt/inference/error_classification.py`, not here.# `MANIFOLD_TRACE_ID_HEADER` and `MANIFOLD_VENDOR_TRACE_ID_HEADER` are the two spellings it stamps# its trace id on, and `extract_manifold_metadata` — the native routes' distiller, which is indiff --git a/pipelex/providers/manifold/manifold_extract_worker.py b/pipelex/providers/manifold/manifold_extract_worker.pyindex a73b872e6..f3fcd3a7f 100644--- a/pipelex/providers/manifold/manifold_extract_worker.py+++ b/pipelex/providers/manifold/manifold_extract_worker.py@@ -83,6 +83,7 @@ class ManifoldExtractWorker(ExtractWorkerAbstract):body=request.model_dump(exclude_none=True),family=InferenceErrorFamily.EXTRACT,inference_model=self.inference_model,+ job_metadata=extract_job.job_metadata,)self._record_usage(response_body=response_body, extract_job=extract_job)try:diff --git a/pipelex/providers/manifold/manifold_factory.py b/pipelex/providers/manifold/manifold_factory.pyindex e38a31d1f..2bcf4dc39 100644--- a/pipelex/providers/manifold/manifold_factory.py+++ b/pipelex/providers/manifold/manifold_factory.py@@ -24,6 +24,7 @@ from pipelex.cogt.img_gen.img_gen_job import ImgGenJobfrom pipelex.cogt.llm.llm_job import LLMJobfrom pipelex.providers.manifold.manifold_constants import MANIFOLD_API_VERSION_SEGMENT, MANIFOLD_AUTH_HEADERfrom pipelex.providers.manifold.manifold_exceptions import ManifoldCredentialsError, ManifoldEndpointError+from pipelex.providers.manifold.manifold_metadata import make_manifold_metadata_headersif TYPE_CHECKING:from portkey_ai import AsyncPortkey@@ -135,6 +136,10 @@ class ManifoldFactory:) -> tuple[dict[str, str], dict[str, Any]]:"""The per-request headers and body additions.+ The headers always include `x-pipelex-metadata`, naming the run and the step this call+ serves (see `manifold_metadata`). It is built here, per request, because it differs per job:+ the token is a client default, this is not.+`output_desc` is part of the shared factory signature and is unused on this path: it existsfor the Portkey-path tracing header that names the job, and the manifold dialect sends notracing headers in the beta.@@ -147,6 +152,8 @@ class ManifoldFactory:# None of these is routing: the service refuses any client header that tries to choose a# provider, and passes the rest through to whichever one it picked.extra_headers.update(inference_model.extra_headers)+ # After the catalog's headers, so a catalog entry cannot restate whose call this is.+ extra_headers.update(make_manifold_metadata_headers(job_metadata=inference_job.job_metadata))if isinstance(inference_job, LLMJob) and inference_model.model_id.lower().startswith("mistral-") and inference_job.job_params.seed is None:# Mistral models really want a non-null seed.diff --git a/pipelex/providers/manifold/manifold_img_gen_worker.py b/pipelex/providers/manifold/manifold_img_gen_worker.pyindex c460306c0..dd7c484de 100644--- a/pipelex/providers/manifold/manifold_img_gen_worker.py+++ b/pipelex/providers/manifold/manifold_img_gen_worker.py@@ -45,6 +45,7 @@ from pipelex.cogt.inference.error_classification import UserAction, UserActionKifrom pipelex.cogt.inference.error_classify import classify_inference_errorfrom pipelex.cogt.inference.error_render import InferenceErrorFamily, render_inference_errorfrom pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory+from pipelex.providers.manifold.manifold_metadata import make_manifold_metadata_headersfrom pipelex.providers.manifold.manifold_schemas import ManifoldImgGenAzureGptImagefrom pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error@@ -96,13 +97,21 @@ class ManifoldImgGenWorker(ImgGenWorkerAbstract):)image_files: list[ImageFileTuple] | None = args_dict.pop("image", None)- response_dict = await self._call_images_api(image_files=image_files, args_dict=args_dict)+ extra_headers = make_manifold_metadata_headers(job_metadata=img_gen_job.job_metadata)+ response_dict = await self._call_images_api(image_files=image_files, args_dict=args_dict, extra_headers=extra_headers)self._record_usage(response_dict=response_dict, img_gen_job=img_gen_job)return self._make_generated_images(response_dict=response_dict)- async def _call_images_api(self, *, image_files: list[ImageFileTuple] | None, args_dict: dict[str, Any]) -> dict[str, Any]:+ async def _call_images_api(+ self, *, image_files: list[ImageFileTuple] | None, args_dict: dict[str, Any], extra_headers: dict[str, str]+ ) -> dict[str, Any]:"""One call, one dict back — whichever of the two Images routes the job needs.+ `extra_headers` is the per-request `x-pipelex-metadata` header. It goes through the SDK's+ own `extra_headers` rather than the client's defaults because it differs per job, and the+ vendor client's `metadata` constructor argument is left alone: it would be sent as+ `x-portkey-metadata`, a name the gateway does not read.+There is no `endpoint_path` here and no config id: the manifold dialect puts the model in thebody and lets the gateway decide who serves it, so which *route* is called is the SDKmethod's business and which *provider* answers is the gateway's.@@ -120,9 +129,9 @@ class ManifoldImgGenWorker(ImgGenWorkerAbstract):# `image` parts keeps one and silently drops the rest, so an edit of several images# would come back as a plausible edit of the first.images_arg: Any = image_files[0] if len(image_files) == 1 else list(image_files)- response = await self.portkey_client.images.edit(image=images_arg, **args_dict) # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType]+ response = await self.portkey_client.images.edit(image=images_arg, extra_headers=extra_headers, **args_dict) # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType]else:- response = await self.portkey_client.images.generate(**args_dict) # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType]+ response = await self.portkey_client.images.generate(extra_headers=extra_headers, **args_dict) # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType]except (portkey_exceptions.APIError, portkey_vendored_openai.APIError) as exc:metadata = extract_gateway_metadata(exc)classification = classify_inference_error(metadata)diff --git a/pipelex/providers/manifold/manifold_metadata.py b/pipelex/providers/manifold/manifold_metadata.pynew file mode 100644index 000000000..912dab71b--- /dev/null+++ b/pipelex/providers/manifold/manifold_metadata.py@@ -0,0 +1,73 @@+"""The one header that says whose call this is: `x-pipelex-metadata`, a flat JSON object.++**Why the Manifold dialect sends it.** The Pipelex Manifold service attributes the spend of every+inference call it serves and logs the call, and neither is possible from the service token alone —+one token serves every run a deployment executes, for every caller and every tenant. So each request+the dialect makes carries, beside the token, a small JSON object naming the run and the step it+belongs to, and the host's own labels for it.++**What goes in it.** The run's `extras` entries, verbatim and unread, then the runtime's own ids:+`user_id` and `pipeline_run_id` from the run half of the job metadata, `pipe_run_id`, `pipe_code`+and `content_generation_job_id` from the step half. An id the job does not carry (`None`) is left+out rather than sent as `null`, so every value on the wire is a string.++**The runtime's keys win on a collision, deliberately.** `extras` is caller data: the host fills it,+and its key charset (`RUN_EXTRAS_KEY_PATTERN`, lowercase snake_case up to 32 characters) admits every+one of the runtime's key names. Letting an extras entry named `user_id` or `pipeline_run_id` stand+would let whoever supplies the labels restate the run's identity to the service that bills for it.+Writing the runtime's keys after the extras makes the run's own facts the last word.++**The runtime still never reads a key of `extras` by name** — the rule `pipelex.system.run_extras`+states. The mapping is copied across whole; nothing here knows that the hosted platform puts an+organization id in it, and a deployment that sends other labels is served identically.++**Why the value cannot forge a header line.** Every value reaching this module is already bounded —+the extras charset, the request-id check, the hex run ids — but the encoding does not rely on that.+`json.dumps` with ASCII output escapes every control character (CR and LF included) and every+non-ASCII code point, and the one printable-range character JSON leaves raw that an HTTP field value+refuses, DEL, is escaped here too. What reaches the wire is printable ASCII, whatever went in.+"""++from __future__ import annotations++import json+from typing import TYPE_CHECKING++from pipelex.providers.manifold.manifold_constants import MANIFOLD_METADATA_HEADER++if TYPE_CHECKING:+ from pipelex.system.job_metadata import JobMetadata+++def make_manifold_metadata(*, job_metadata: JobMetadata) -> dict[str, str]:+ """The flat mapping the header carries: the run's extras, then the runtime's own ids over them."""+ run_metadata = job_metadata.run_metadata+ runtime_ids: dict[str, str | None] = {+ "user_id": run_metadata.user_id,+ "pipeline_run_id": run_metadata.pipeline_run_id,+ "pipe_run_id": job_metadata.pipe_run_id,+ "pipe_code": job_metadata.pipe_code,+ "content_generation_job_id": job_metadata.content_generation_job_id,+ }+ metadata: dict[str, str] = dict(run_metadata.extras)+ for key, value in runtime_ids.items():+ if value is None:+ # An absent id is not a colliding one: an extras entry of the same name is dropped+ # rather than left standing in for a fact the runtime does not have.+ metadata.pop(key, None)+ continue+ metadata[key] = value+ return metadata+++def encode_manifold_metadata(*, metadata: dict[str, str]) -> str:+ """The header value: compact JSON in printable ASCII only."""+ encoded = json.dumps(metadata, ensure_ascii=True, separators=(",", ":"))+ # JSON escapes control characters below 0x20 and, with `ensure_ascii`, everything above 0x7E+ # except DEL itself, which an HTTP field value refuses. `\u007f` is the same JSON string.+ return encoded.replace("\x7f", "\\u007f")+++def make_manifold_metadata_headers(*, job_metadata: JobMetadata) -> dict[str, str]:+ """The one-entry header mapping a manifold request adds for the job it serves."""+ return {MANIFOLD_METADATA_HEADER: encode_manifold_metadata(metadata=make_manifold_metadata(job_metadata=job_metadata))}diff --git a/pipelex/providers/manifold/manifold_native_client.py b/pipelex/providers/manifold/manifold_native_client.pyindex 87b77fd2e..ec5bee9a9 100644--- a/pipelex/providers/manifold/manifold_native_client.py+++ b/pipelex/providers/manifold/manifold_native_client.py@@ -7,7 +7,9 @@ are called with plain ``httpx`` rather than through a vendor SDK wrapped around**Building the request by hand is what makes the wire readable.** Two properties of the manifolddialect live in this one method and can be asserted in a test that reads the request it built: theonly header about us is the service token, and the body carries the model with nothing about-routing. A client whose constructor defaults decide the headers cannot say that.+routing. A client whose constructor defaults decide the headers cannot say that. The one other+header is `x-pipelex-metadata`, which says whose call it is rather than anything about us; it+differs per job, so the caller hands the job's metadata to each call rather than to the client."""from __future__ import annotations@@ -23,11 +25,13 @@ from pipelex.cogt.inference.transport_retry import request_with_transport_retryfrom pipelex.config import get_configfrom pipelex.providers.manifold.manifold_exceptions import ManifoldErrorfrom pipelex.providers.manifold.manifold_factory import ManifoldFactory+from pipelex.providers.manifold.manifold_metadata import make_manifold_metadata_headersif TYPE_CHECKING:from pipelex.cogt.inference.error_render import InferenceErrorFamilyfrom pipelex.cogt.model_backends.backend import InferenceBackendfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpec+ from pipelex.system.job_metadata import JobMetadata# Generous, and deliberately so: a deep web search or a many-page extraction is a slow call, and the# runtime's own job-level timeouts are the ones that should bound it. Matches the SDK-less image@@ -49,14 +53,20 @@ class ManifoldNativeClient:body: dict[str, Any],family: InferenceErrorFamily,inference_model: InferenceModelSpec,+ job_metadata: JobMetadata,) -> dict[str, Any]:url = f"{self.base_url}{route}"+ headers = {+ **self.auth_headers,+ **make_manifold_metadata_headers(job_metadata=job_metadata),+ "Content-Type": "application/json",+ }async def _send() -> httpx.Response:async with httpx.AsyncClient() as client:response = await client.post(url,- headers={**self.auth_headers, "Content-Type": "application/json"},+ headers=headers,json=body,timeout=MANIFOLD_NATIVE_TIMEOUT_SECONDS,)diff --git a/pipelex/providers/manifold/manifold_search_worker.py b/pipelex/providers/manifold/manifold_search_worker.pyindex 07b43e36a..6c1516904 100644--- a/pipelex/providers/manifold/manifold_search_worker.py+++ b/pipelex/providers/manifold/manifold_search_worker.py@@ -146,6 +146,7 @@ class ManifoldSearchWorker(SearchWorkerAbstract):body=request.model_dump(exclude_none=True),family=InferenceErrorFamily.SEARCH,inference_model=self.inference_model,+ job_metadata=search_job.job_metadata,)self._record_usage(response_body=response_body, search_job=search_job)return response_bodydiff --git a/pipelex/system/job_metadata.py b/pipelex/system/job_metadata.pyindex ead1dbf8d..a3071884f 100644--- a/pipelex/system/job_metadata.py+++ b/pipelex/system/job_metadata.py@@ -163,7 +163,8 @@ class RunMetadata(BaseModel):request_id: str | None = None# The opaque labels the host attaches to this run. Never read by name;- # telemetry forwards the whole mapping as the groups of each capture+ # telemetry forwards the whole mapping as the groups of each capture, and+ # the Manifold dialect forwards it in its `x-pipelex-metadata` header# — see `pipelex.system.run_extras` for why the host's own concepts# (organization, tenant, plan tier) deliberately do not cross this boundary,# and for the charset and the size bound.diff --git a/pipelex/system/run_extras.py b/pipelex/system/run_extras.pyindex d807bedb1..f9d4ad530 100644--- a/pipelex/system/run_extras.py+++ b/pipelex/system/run_extras.py@@ -19,10 +19,21 @@ is the backend that supports grouping. So whatever is put here is a live groupkey in those projects and a span attribute on every OpenTelemetry export, and thebounds below are shaped by that consumer.-The hosted Pipelex platform fills it with `{"organization": "<org_id>"}`, but-nothing here knows or checks that: no key is privileged, and a deployment that-sends `{"tenant": …, "workspace": …}` is served identically. A single-user-deployment sends nothing at all.+The inference gateway is the other reader. Every call the Pipelex Manifold+dialect makes carries the mapping, whole, in its `x-pipelex-metadata` header,+beside the runtime's own ids for the run and the step, so the gateway can+attribute the call's spend and log it (`pipelex.providers.manifold.manifold_metadata`).+The runtime's ids win when an extras key collides with one of them — every one of+their names (`user_id`, `pipeline_run_id`, `pipe_run_id`, `pipe_code`,+`content_generation_job_id`) fits the key charset below — because this mapping is+caller data and must not be able to restate whose run it is. No other inference+path sends it: a vendor reached directly has no use for a host's labels.++The hosted Pipelex platform fills it with `{"org_id": "<org_id>"}`, plus+`"api_key_id"` on a call made with an API key, but nothing here knows or checks+that: no key is privileged, and a deployment that sends+`{"tenant": …, "workspace": …}` is served identically. A single-user deployment+sends nothing at all.**Why the value is validated on the type.** The mapping is caller data thatarrives over a wire, so its contents are whatever the other side put there. Itdiff --git a/tests/unit/pipelex/providers/manifold/test_data.py b/tests/unit/pipelex/providers/manifold/test_data.pynew file mode 100644index 000000000..777e68871--- /dev/null+++ b/tests/unit/pipelex/providers/manifold/test_data.py@@ -0,0 +1,33 @@+"""Job metadata the manifold tests hand to a request, and the header value it must produce."""++from typing import ClassVar++from pipelex.system.job_metadata import JobMetadata, RunMetadata+++class ManifoldMetadataTestData:+ """A job whose run carries host labels, and the exact `x-pipelex-metadata` value it yields.++ The expected value is written out as the wire string rather than rebuilt from the job, so a+ change to the encoding (key order, separators, escaping) turns a test red instead of moving with it.+ """++ EXTRAS: ClassVar[dict[str, str]] = {"org_id": "org_acme", "api_key_id": "key_123"}++ JOB_METADATA: ClassVar[JobMetadata] = JobMetadata(+ run_metadata=RunMetadata(+ storage_scope="test/scope",+ read_scope=None,+ user_id="user_42",+ pipeline_run_id="run_abc",+ extras={"org_id": "org_acme", "api_key_id": "key_123"},+ ),+ pipe_code="summarize_doc",+ pipe_run_id="0123456789abcdef",+ content_generation_job_id="cgj_1",+ )++ EXPECTED_HEADER_VALUE: ClassVar[str] = (+ '{"org_id":"org_acme","api_key_id":"key_123","user_id":"user_42","pipeline_run_id":"run_abc",'+ '"pipe_run_id":"0123456789abcdef","pipe_code":"summarize_doc","content_generation_job_id":"cgj_1"}'+ )diff --git a/tests/unit/pipelex/providers/manifold/test_manifold_img_gen_worker.py b/tests/unit/pipelex/providers/manifold/test_manifold_img_gen_worker.pyindex 404875a06..81933dcfb 100644--- a/tests/unit/pipelex/providers/manifold/test_manifold_img_gen_worker.py+++ b/tests/unit/pipelex/providers/manifold/test_manifold_img_gen_worker.py@@ -23,6 +23,7 @@ from portkey_ai.api_resources import exceptions as portkey_excfrom pipelex.cogt.exceptions import ImgGenGenerationError, InferenceErrorCategoryfrom pipelex.cogt.inference.error_classification import UserActionKindfrom pipelex.providers.manifold.manifold_img_gen_worker import ManifoldImgGenWorker+from tests.unit.pipelex.providers.manifold.test_data import ManifoldMetadataTestDataif TYPE_CHECKING:from pytest_mock import MockerFixture@@ -62,6 +63,7 @@ def _make_img_gen_job(mocker: MockerFixture) -> Any:job = mocker.MagicMock()job.img_gen_prompt.positive_text = "a manifold, rendered"job.job_report.img_gen_tokens_usage = None+ job.job_metadata = ManifoldMetadataTestData.JOB_METADATAreturn jobdiff --git a/tests/unit/pipelex/providers/manifold/test_manifold_metadata.py b/tests/unit/pipelex/providers/manifold/test_manifold_metadata.pynew file mode 100644index 000000000..7193bb1e6--- /dev/null+++ b/tests/unit/pipelex/providers/manifold/test_manifold_metadata.py@@ -0,0 +1,121 @@+"""The `x-pipelex-metadata` value: its exact shape, who wins a collision, and why it cannot forge a line.++The header is how the Manifold gateway attributes spend and logs a call, so its value is asserted as+the wire string. The collision rule is the security-relevant one: `extras` is caller data whose key+charset admits every one of the runtime's key names, so a label named `user_id` must never restate+the run's identity to the service that bills for it.+"""++from __future__ import annotations++import json++import pytest++from pipelex.providers.manifold.manifold_constants import MANIFOLD_AUTH_HEADER, MANIFOLD_METADATA_HEADER+from pipelex.providers.manifold.manifold_metadata import (+ encode_manifold_metadata,+ make_manifold_metadata,+ make_manifold_metadata_headers,+)+from pipelex.system.job_metadata import JobMetadata, RunMetadata+from pipelex.system.run_extras import RUN_EXTRAS_KEY_PATTERN+from tests.unit.pipelex.providers.manifold.test_data import ManifoldMetadataTestData++_RUNTIME_KEYS = ("user_id", "pipeline_run_id", "pipe_run_id", "pipe_code", "content_generation_job_id")+++def _job_metadata(+ *,+ extras: dict[str, str] | None = None,+ user_id: str = "user_42",+ pipe_code: str | None = None,+ pipe_run_id: str | None = None,+ content_generation_job_id: str | None = None,+) -> JobMetadata:+ return JobMetadata(+ run_metadata=RunMetadata(+ storage_scope="test/scope",+ read_scope=None,+ user_id=user_id,+ pipeline_run_id="run_abc",+ extras=extras or {},+ ),+ pipe_code=pipe_code,+ pipe_run_id=pipe_run_id,+ content_generation_job_id=content_generation_job_id,+ )+++class TestManifoldMetadata:+ def test_the_header_is_named_by_the_dialect_and_never_by_the_vendor(self) -> None:+ assert MANIFOLD_METADATA_HEADER == "x-pipelex-metadata"+ assert MANIFOLD_METADATA_HEADER != MANIFOLD_AUTH_HEADER++ def test_a_job_with_extras_yields_the_exact_wire_value(self) -> None:+ headers = make_manifold_metadata_headers(job_metadata=ManifoldMetadataTestData.JOB_METADATA)++ assert headers == {MANIFOLD_METADATA_HEADER: ManifoldMetadataTestData.EXPECTED_HEADER_VALUE}+ decoded = json.loads(headers[MANIFOLD_METADATA_HEADER])+ assert decoded == {+ "org_id": "org_acme",+ "api_key_id": "key_123",+ "user_id": "user_42",+ "pipeline_run_id": "run_abc",+ "pipe_run_id": "0123456789abcdef",+ "pipe_code": "summarize_doc",+ "content_generation_job_id": "cgj_1",+ }++ def test_every_runtime_key_is_a_legal_extras_key_so_a_collision_is_possible(self) -> None:+ """The premise of the collision rule: the extras charset does not keep these names out."""+ for runtime_key in _RUNTIME_KEYS:+ assert RUN_EXTRAS_KEY_PATTERN.fullmatch(runtime_key), runtime_key++ def test_the_runtime_ids_win_over_colliding_extras_keys(self) -> None:+ job_metadata = _job_metadata(+ extras={"user_id": "spoofed_user", "pipeline_run_id": "spoofed_run", "org_id": "org_acme"},+ pipe_code="real_pipe",+ pipe_run_id="real_pipe_run",+ content_generation_job_id="real_cgj",+ )++ metadata = make_manifold_metadata(job_metadata=job_metadata)++ assert metadata == {+ "org_id": "org_acme",+ "user_id": "user_42",+ "pipeline_run_id": "run_abc",+ "pipe_run_id": "real_pipe_run",+ "pipe_code": "real_pipe",+ "content_generation_job_id": "real_cgj",+ }++ def test_a_none_id_is_omitted_and_a_colliding_extras_key_does_not_stand_in_for_it(self) -> None:+ """An id the job lacks is absent, never `null`, and never filled by the caller's label of that name."""+ job_metadata = _job_metadata(extras={"pipe_code": "spoofed_pipe", "org_id": "org_acme"})++ metadata = make_manifold_metadata(job_metadata=job_metadata)++ assert metadata == {"org_id": "org_acme", "user_id": "user_42", "pipeline_run_id": "run_abc"}+ assert "null" not in make_manifold_metadata_headers(job_metadata=job_metadata)[MANIFOLD_METADATA_HEADER]++ def test_empty_extras_yield_the_runtime_ids_alone(self) -> None:+ headers = make_manifold_metadata_headers(job_metadata=_job_metadata(pipe_code="p", pipe_run_id="r"))++ assert headers == {MANIFOLD_METADATA_HEADER: '{"user_id":"user_42","pipeline_run_id":"run_abc","pipe_run_id":"r","pipe_code":"p"}'}++ @pytest.mark.parametrize(+ "hostile_user_id",+ ["user\r\nx-injected: 1", "user\nforged", "café-☃", "del\x7fchar", "tab\there"],+ ids=["crlf", "lf", "non-ascii", "del", "tab"],+ )+ def test_the_value_is_printable_ascii_whatever_went_in(self, hostile_user_id: str) -> None:+ """`user_id` has no charset of its own, so the encoding is what keeps the header one line."""+ value = make_manifold_metadata_headers(job_metadata=_job_metadata(user_id=hostile_user_id))[MANIFOLD_METADATA_HEADER]++ assert all(0x20 <= ord(char) <= 0x7E for char in value), value+ assert json.loads(value)["user_id"] == hostile_user_id++ def test_the_encoding_is_compact_json(self) -> None:+ assert encode_manifold_metadata(metadata={"a": "1", "b": "2"}) == '{"a":"1","b":"2"}'diff --git a/tests/unit/pipelex/providers/manifold/test_manifold_metadata_scope.py b/tests/unit/pipelex/providers/manifold/test_manifold_metadata_scope.pynew file mode 100644index 000000000..066a5014f--- /dev/null+++ b/tests/unit/pipelex/providers/manifold/test_manifold_metadata_scope.py@@ -0,0 +1,131 @@+"""Only the manifold dialect sends `x-pipelex-metadata`: no direct-provider or Portkey path does.++The run's identity and the host's labels are forwarded to our own gateway, which bills and logs the+call; a vendor reached directly with the caller's own key has no use for them and must not receive+them. The shared Anthropic driver is the path most at risk, because it serves both a direct Anthropic+backend and Claude behind Manifold from the same code.+"""++from __future__ import annotations++from typing import TYPE_CHECKING, Any++import pytest+from anthropic import AsyncAnthropic+from anthropic.types import Message, ToolUseBlock, Usage++from pipelex.cogt.llm.llm_job import LLMJob+from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobParams, LLMJobReport+from pipelex.cogt.llm.llm_prompt import LLMPrompt+from pipelex.cogt.llm.structured_output import StructureMethod+from pipelex.providers.anthropic.anthropic_llm_worker import AnthropicLLMWorker+from pipelex.providers.gateway.gateway_factory import GatewayFactory+from pipelex.providers.manifold.manifold_constants import MANIFOLD_METADATA_HEADER+from pipelex.providers.openai.openai_completions_factory import OpenAICompletionsFactory+from pipelex.providers.openai.openai_responses_factory import OpenAIResponsesFactory+from pipelex.providers.portkey.portkey_factory import PortkeyFactory+from tests.helpers.instructor_test_utils import DummySchema+from tests.unit.pipelex.providers.manifold.test_data import ManifoldMetadataTestData++if TYPE_CHECKING:+ from pytest_mock import MockerFixture+++class _RequestCapturedError(Exception):+ """Raised by a stubbed SDK call once it has recorded its arguments."""+++def _llm_job() -> LLMJob:+ return LLMJob(+ job_metadata=ManifoldMetadataTestData.JOB_METADATA,+ llm_prompt=LLMPrompt(user_text="ping"),+ job_params=LLMJobParams(temperature=0.5),+ job_config=LLMJobConfig(schema_reask_max_attempts=1),+ job_report=LLMJobReport(),+ )+++def _model(mocker: MockerFixture, *, backend_name: str) -> Any:+ model = mocker.MagicMock()+ model.desc = "test-model-desc"+ model.model_id = "claude-test"+ model.name = "claude-test"+ model.backend_name = backend_name+ model.thinking_mode = None+ model.listed_constraints = []+ model.extra_headers = None+ model.structure_method = StructureMethod.INSTRUCTOR_ANTHROPIC_TOOLS+ return model+++def _anthropic_worker(mocker: MockerFixture, *, sdk_client: AsyncAnthropic, backend_name: str) -> AnthropicLLMWorker:+ from instructor import from_anthropic # ruff: ignore[import-outside-top-level]++ worker = object.__new__(AnthropicLLMWorker)+ worker.inference_model = _model(mocker, backend_name=backend_name)+ worker.default_max_tokens = 4096+ worker.anthropic_async_client = sdk_client+ worker.instructor_for_objects = from_anthropic(client=sdk_client, mode=StructureMethod.INSTRUCTOR_ANTHROPIC_TOOLS.as_instructor_mode())+ config = mocker.MagicMock()+ config.inference.llm.anthropic.structured_output_timeout_seconds = 1200+ mocker.patch("pipelex.providers.anthropic.anthropic_llm_worker.get_config", return_value=config)+ return worker+++def _tool_call_message() -> Message:+ return Message(+ id="msg_test",+ type="message",+ role="assistant",+ model="claude-test",+ content=[ToolUseBlock(type="tool_use", id="toolu_test", name="DummySchema", input={"text": "answer"})],+ stop_reason="tool_use",+ stop_sequence=None,+ usage=Usage(input_tokens=1, output_tokens=1),+ )++[email protected](loop_scope="class")+class TestManifoldMetadataScope:+ @pytest.mark.parametrize(+ "factory",+ [+ OpenAICompletionsFactory(is_http_url_enabled=False),+ OpenAIResponsesFactory(is_http_url_enabled=False),+ PortkeyFactory,+ GatewayFactory,+ ],+ ids=["openai-completions", "openai-responses", "portkey", "gateway"],+ )+ async def test_other_backends_extras_carry_no_metadata_header(self, mocker: MockerFixture, factory: Any) -> None:+ model = _model(mocker, backend_name="openai")+ model.model_id = "gpt-4o"++ extra_headers, _ = factory.make_extras(inference_model=model, inference_job=_llm_job(), output_desc="text")++ assert MANIFOLD_METADATA_HEADER not in extra_headers+ assert "x-portkey-metadata" not in extra_headers++ @pytest.mark.parametrize("backend_name", ["anthropic", "bedrock", "pipelex_gateway"])+ async def test_direct_claude_sends_no_headers_on_the_text_call(self, mocker: MockerFixture, backend_name: str) -> None:+ sdk_client = AsyncAnthropic(api_key="test-key")+ stream = mocker.MagicMock(side_effect=_RequestCapturedError)+ mocker.patch.object(sdk_client.messages, "stream", new=stream)+ worker = _anthropic_worker(mocker, sdk_client=sdk_client, backend_name=backend_name)++ with pytest.raises(_RequestCapturedError):+ await worker._gen_text(_llm_job()) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]++ assert "extra_headers" not in stream.call_args.kwargs++ async def test_direct_claude_sends_no_headers_on_the_structured_call(self, mocker: MockerFixture) -> None:+ sdk_client = AsyncAnthropic(api_key="test-key")+ create = mocker.AsyncMock(return_value=_tool_call_message())+ mocker.patch.object(sdk_client.messages, "create", new=create)+ worker = _anthropic_worker(mocker, sdk_client=sdk_client, backend_name="anthropic")++ await worker._gen_object(_llm_job(), schema=DummySchema) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]++ await_args = create.await_args+ assert await_args is not None+ assert MANIFOLD_METADATA_HEADER not in str(await_args.kwargs.get("extra_headers"))diff --git a/tests/unit/pipelex/providers/manifold/test_manifold_metadata_sdk_paths.py b/tests/unit/pipelex/providers/manifold/test_manifold_metadata_sdk_paths.pynew file mode 100644index 000000000..922bc06cb--- /dev/null+++ b/tests/unit/pipelex/providers/manifold/test_manifold_metadata_sdk_paths.py@@ -0,0 +1,230 @@+"""Every path the manifold dialect calls the service on carries `x-pipelex-metadata` for its job.++The header differs per job, so it cannot be a client default: each path adds it per request, and+each is pinned here where the request leaves the runtime — the kwargs handed to the SDK call, or the+headers handed to `httpx` on the native routes. The paths are the OpenAI-substrate chat completions+(text, and image generation over completions) and responses, the vendor-SDK image path, the native+extract and search routes, and Claude over the shared Anthropic driver behind the manifold backend.+"""++from __future__ import annotations++from typing import TYPE_CHECKING, Any++import httpx+import pytest+from anthropic import AsyncAnthropic+from anthropic.types import Message, ToolUseBlock, Usage++from pipelex.cogt.llm.llm_job import LLMJob+from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobParams, LLMJobReport+from pipelex.cogt.llm.llm_prompt import LLMPrompt+from pipelex.cogt.llm.structured_output import StructureMethod+from pipelex.cogt.model_backends.backend import InferenceBackend, PipelexBackend+from pipelex.providers.anthropic.anthropic_llm_worker import AnthropicLLMWorker+from pipelex.providers.manifold.manifold_completions_factory import ManifoldCompletionsFactory+from pipelex.providers.manifold.manifold_constants import MANIFOLD_METADATA_HEADER+from pipelex.providers.manifold.manifold_extract_worker import ManifoldExtractWorker+from pipelex.providers.manifold.manifold_img_gen_worker import ManifoldImgGenWorker+from pipelex.providers.manifold.manifold_native_client import ManifoldNativeClient+from pipelex.providers.manifold.manifold_responses_factory import ManifoldResponsesFactory+from pipelex.providers.manifold.manifold_search_worker import ManifoldSearchWorker+from pipelex.providers.openai.openai_completions_llm_worker import OpenAICompletionsLLMWorker+from tests.helpers.instructor_test_utils import DummySchema+from tests.unit.pipelex.providers.manifold.test_data import ManifoldMetadataTestData++if TYPE_CHECKING:+ from pytest_mock import MockerFixture++_EXPECTED = {MANIFOLD_METADATA_HEADER: ManifoldMetadataTestData.EXPECTED_HEADER_VALUE}+_ORIGIN = "https://manifold.example.com"+++class _RequestCapturedError(Exception):+ """Raised by a stubbed SDK call once it has recorded its arguments, so no response has to be faked."""+++def _llm_job() -> LLMJob:+ return LLMJob(+ job_metadata=ManifoldMetadataTestData.JOB_METADATA,+ llm_prompt=LLMPrompt(user_text="ping"),+ job_params=LLMJobParams(temperature=0.5),+ job_config=LLMJobConfig(schema_reask_max_attempts=1),+ job_report=LLMJobReport(),+ )+++def _model(mocker: MockerFixture, *, backend_name: str = PipelexBackend.MANIFOLD, model_id: str = "gpt-4o") -> Any:+ model = mocker.MagicMock()+ model.desc = "test-model-desc"+ model.model_id = model_id+ model.name = model_id+ model.backend_name = backend_name+ model.thinking_mode = None+ model.listed_constraints = []+ model.extra_headers = None+ return model+++def _native_client() -> ManifoldNativeClient:+ return ManifoldNativeClient(backend=InferenceBackend(name="pipelex_manifold", endpoint=_ORIGIN, api_key="token"))+++def _patch_httpx_post(mocker: MockerFixture, *, json_body: dict[str, Any]) -> Any:+ response = httpx.Response(status_code=200, request=httpx.Request("POST", _ORIGIN), json=json_body)+ return mocker.patch.object(httpx.AsyncClient, "post", new_callable=mocker.AsyncMock, return_value=response)+++def _anthropic_worker(mocker: MockerFixture, *, sdk_client: AsyncAnthropic, backend_name: str) -> AnthropicLLMWorker:+ from instructor import from_anthropic # ruff: ignore[import-outside-top-level]++ worker = object.__new__(AnthropicLLMWorker)+ model = _model(mocker, backend_name=backend_name, model_id="claude-test")+ model.structure_method = StructureMethod.INSTRUCTOR_ANTHROPIC_TOOLS+ worker.inference_model = model+ worker.default_max_tokens = 4096+ worker.instructor_for_objects = from_anthropic(client=sdk_client, mode=StructureMethod.INSTRUCTOR_ANTHROPIC_TOOLS.as_instructor_mode())+ config = mocker.MagicMock()+ config.inference.llm.anthropic.structured_output_timeout_seconds = 1200+ mocker.patch("pipelex.providers.anthropic.anthropic_llm_worker.get_config", return_value=config)+ return worker+++def _tool_call_message() -> Message:+ return Message(+ id="msg_test",+ type="message",+ role="assistant",+ model="claude-test",+ content=[ToolUseBlock(type="tool_use", id="toolu_test", name="DummySchema", input={"text": "answer"})],+ stop_reason="tool_use",+ stop_sequence=None,+ usage=Usage(input_tokens=1, output_tokens=1),+ )++[email protected](loop_scope="class")+class TestManifoldMetadataSdkPaths:+ @pytest.mark.parametrize(+ "factory",+ [ManifoldCompletionsFactory(is_http_url_enabled=False), ManifoldResponsesFactory(is_http_url_enabled=False)],+ ids=["completions", "responses"],+ )+ async def test_the_openai_substrate_extras_carry_the_header(self, mocker: MockerFixture, factory: Any) -> None:+ """`make_extras` feeds every OpenAI-substrate call: completions text and objects, image gen over completions, responses."""+ extra_headers, _ = factory.make_extras(inference_model=_model(mocker), inference_job=_llm_job(), output_desc="text")++ assert extra_headers == _EXPECTED++ async def test_a_catalog_header_of_the_same_name_does_not_override_the_job(self, mocker: MockerFixture) -> None:+ model = _model(mocker)+ model.extra_headers = {"anthropic-beta": "some-beta", MANIFOLD_METADATA_HEADER: '{"user_id":"catalog"}'}++ extra_headers, _ = ManifoldCompletionsFactory(is_http_url_enabled=False).make_extras(+ inference_model=model, inference_job=_llm_job(), output_desc="text"+ )++ assert extra_headers == {"anthropic-beta": "some-beta", **_EXPECTED}++ async def test_the_completions_worker_sends_the_header_on_the_sdk_call(self, mocker: MockerFixture) -> None:+ worker = object.__new__(OpenAICompletionsLLMWorker)+ worker.inference_model = _model(mocker)+ worker.openai_completions_factory = ManifoldCompletionsFactory(is_http_url_enabled=False)+ client = mocker.MagicMock()+ client.chat.completions.create = mocker.AsyncMock(side_effect=_RequestCapturedError)+ worker.openai_client_for_text = client++ with pytest.raises(_RequestCapturedError):+ await worker._gen_text(_llm_job()) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]++ assert client.chat.completions.create.call_args.kwargs["extra_headers"] == _EXPECTED++ @pytest.mark.parametrize(+ ("args", "route"),+ [({"prompt": "draw"}, "generate"), ({"prompt": "edit", "image": [("a.png", b"\x89PNG", "image/png")]}, "edit")],+ ids=["generate", "edit"],+ )+ async def test_the_image_worker_sends_the_header_on_both_images_routes(self, mocker: MockerFixture, args: dict[str, Any], route: str) -> None:+ worker = object.__new__(ManifoldImgGenWorker)+ worker.inference_model = _model(mocker, model_id="gpt-image-1")+ client: Any = mocker.MagicMock()+ client.images.generate = mocker.AsyncMock(side_effect=_RequestCapturedError)+ client.images.edit = mocker.AsyncMock(side_effect=_RequestCapturedError)+ worker.portkey_client = client+ mocker.patch(+ "pipelex.providers.manifold.manifold_img_gen_worker.ImgGenArgsFactory.make_args_for_model",+ new_callable=mocker.AsyncMock,+ return_value=args,+ )+ job = mocker.MagicMock()+ job.job_metadata = ManifoldMetadataTestData.JOB_METADATA++ with pytest.raises(_RequestCapturedError):+ await worker._gen_image_list(img_gen_job=job, nb_images=1) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]++ assert getattr(client.images, route).call_args.kwargs["extra_headers"] == _EXPECTED++ async def test_the_extract_route_sends_the_header_on_the_wire(self, mocker: MockerFixture) -> None:+ post = _patch_httpx_post(mocker, json_body={"pages": {}})+ worker = object.__new__(ManifoldExtractWorker)+ model = _model(mocker, model_id="firecrawl/scrape")+ model.is_web_page_supported_for_extract = True+ worker.inference_model = model+ worker.client = _native_client()+ job = mocker.MagicMock()+ job.job_metadata = ManifoldMetadataTestData.JOB_METADATA+ job.extract_input.document_uri = "https://example.com/page"+ job.job_params.max_nb_images = None+ job.job_params.render_js = None+ job.job_params.include_raw_html = None+ job.job_report.extract_tokens_usage = None++ await worker._extract_pages(job) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]++ assert post.call_args.kwargs["headers"][MANIFOLD_METADATA_HEADER] == ManifoldMetadataTestData.EXPECTED_HEADER_VALUE++ async def test_the_search_route_sends_the_header_on_the_wire(self, mocker: MockerFixture) -> None:+ post = _patch_httpx_post(mocker, json_body={"answer": "a manifold is a space", "sources": []})+ worker = object.__new__(ManifoldSearchWorker)+ worker.inference_model = _model(mocker, model_id="linkup/standard")+ worker.client = _native_client()+ job = mocker.MagicMock()+ job.job_metadata = ManifoldMetadataTestData.JOB_METADATA+ job.query = "what is a manifold"+ job.job_params.search_setting.include_images = False+ job.job_params.search_setting.include_inline_citations = False+ job.job_params.search_setting.max_results = 10+ job.job_params.include_domains = None+ job.job_params.exclude_domains = None+ job.job_params.from_date = None+ job.job_params.to_date = None+ job.job_report.search_tokens_usage = None++ await worker._search_sourced_answer(job) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]++ assert post.call_args.kwargs["headers"][MANIFOLD_METADATA_HEADER] == ManifoldMetadataTestData.EXPECTED_HEADER_VALUE++ async def test_claude_behind_manifold_sends_the_header_on_the_text_call(self, mocker: MockerFixture) -> None:+ sdk_client = AsyncAnthropic(api_key="test-key")+ stream = mocker.MagicMock(side_effect=_RequestCapturedError)+ mocker.patch.object(sdk_client.messages, "stream", new=stream)+ worker = _anthropic_worker(mocker, sdk_client=sdk_client, backend_name=PipelexBackend.MANIFOLD)+ worker.anthropic_async_client = sdk_client++ with pytest.raises(_RequestCapturedError):+ await worker._gen_text(_llm_job()) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]++ assert stream.call_args.kwargs["extra_headers"] == _EXPECTED++ async def test_claude_behind_manifold_sends_the_header_on_the_structured_call(self, mocker: MockerFixture) -> None:+ """Through a real instructor client, so what is asserted is what reaches the SDK's `messages.create`."""+ sdk_client = AsyncAnthropic(api_key="test-key")+ create = mocker.AsyncMock(return_value=_tool_call_message())+ mocker.patch.object(sdk_client.messages, "create", new=create)+ worker = _anthropic_worker(mocker, sdk_client=sdk_client, backend_name=PipelexBackend.MANIFOLD)++ await worker._gen_object(_llm_job(), schema=DummySchema) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]++ await_args = create.await_args+ assert await_args is not None+ assert await_args.kwargs["extra_headers"] == _EXPECTEDdiff --git a/tests/unit/pipelex/providers/manifold/test_manifold_native_client.py b/tests/unit/pipelex/providers/manifold/test_manifold_native_client.pyindex 3e3aac8f2..ccfc2f73e 100644--- a/tests/unit/pipelex/providers/manifold/test_manifold_native_client.py+++ b/tests/unit/pipelex/providers/manifold/test_manifold_native_client.py@@ -3,7 +3,8 @@`POST /v1/pipelex/extract` and `POST /v1/pipelex/search` are not OpenAI-shaped, so they are calledwith plain `httpx`. Building the request by hand is what makes the dialect's two properties readablein a test rather than dependent on a vendor constructor's defaults: the only header about us is the-service token, and the body carries the model with nothing about routing.+service token, the only other one says whose call it is, and the body carries the model with+nothing about routing.The failure arm is the other half. A 2xx whose body is not JSON — an intermediary's HTML error pageis how this happens — left raw is neither a `PipelexError` nor annotated with the model, so it@@ -23,6 +24,7 @@ from pipelex.cogt.inference.error_render import InferenceErrorFamilyfrom pipelex.cogt.model_backends.backend import InferenceBackendfrom pipelex.providers.manifold.manifold_exceptions import ManifoldErrorfrom pipelex.providers.manifold.manifold_native_client import ManifoldNativeClient+from tests.unit.pipelex.providers.manifold.test_data import ManifoldMetadataTestDataif TYPE_CHECKING:from pytest_mock import MockerFixture@@ -68,12 +70,18 @@ class TestManifoldNativeClientRequest:body={"model": "linkup/standard", "query": "what is a manifold"},family=InferenceErrorFamily.SEARCH,inference_model=_inference_model(mocker),+ job_metadata=ManifoldMetadataTestData.JOB_METADATA,)assert post.call_args.args[0] == f"{_ORIGIN}/v1/pipelex/search"headers = post.call_args.kwargs["headers"]- # Asserted whole: a header added later should turn this red rather than widen silently.- assert headers == {"x-pipelex-api-key": _TOKEN, "Content-Type": "application/json"}+ # Asserted whole: a header added later should turn this red rather than widen silently. The+ # metadata header says whose call this is, not anything about us or about routing.+ assert headers == {+ "x-pipelex-api-key": _TOKEN,+ "x-pipelex-metadata": ManifoldMetadataTestData.EXPECTED_HEADER_VALUE,+ "Content-Type": "application/json",+ }assert post.call_args.kwargs["json"] == {"model": "linkup/standard", "query": "what is a manifold"}async def test_a_json_object_body_comes_back_as_a_dict(self, mocker: MockerFixture) -> None:@@ -84,6 +92,7 @@ class TestManifoldNativeClientRequest:body={"model": "linkup/standard", "query": "q"},family=InferenceErrorFamily.SEARCH,inference_model=_inference_model(mocker),+ job_metadata=ManifoldMetadataTestData.JOB_METADATA,)assert result == {"answer": "ok", "sources": []}@@ -100,6 +109,7 @@ class TestManifoldNativeClientFailures:body={"model": "linkup/standard", "query": "q"},family=InferenceErrorFamily.SEARCH,inference_model=_inference_model(mocker),+ job_metadata=ManifoldMetadataTestData.JOB_METADATA,)assert "linkup-sourced-answer" in exc_info.value.message@@ -113,6 +123,7 @@ class TestManifoldNativeClientFailures:body={"model": "linkup/standard", "query": "q"},family=InferenceErrorFamily.SEARCH,inference_model=_inference_model(mocker),+ job_metadata=ManifoldMetadataTestData.JOB_METADATA,)async def test_a_status_error_is_classified_rather_than_raised_raw(self, mocker: MockerFixture) -> None:@@ -124,4 +135,5 @@ class TestManifoldNativeClientFailures:body={"model": "linkup/standard", "query": "q"},family=InferenceErrorFamily.SEARCH,inference_model=_inference_model(mocker),+ job_metadata=ManifoldMetadataTestData.JOB_METADATA,)