Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
fix(search): recognise the sources envelope by the schema's wire names
Louis Choquel73771c49a044
diff --git a/.badges/tests.json b/.badges/tests.jsonindex 3ee947438..bb2d9654d 100644--- a/.badges/tests.json+++ b/.badges/tests.json@@ -1,7 +1,7 @@{"schemaVersion": 1,"label": "tests",- "message": "9369",+ "message": "9371","color": "blue","cacheSeconds": 300}\ No newline at end of filediff --git a/pipelex/cogt/search/structured_search_payload.py b/pipelex/cogt/search/structured_search_payload.pyindex 5b256cb0b..48804a18b 100644--- a/pipelex/cogt/search/structured_search_payload.py+++ b/pipelex/cogt/search/structured_search_payload.py@@ -15,9 +15,9 @@ Neither backend can therefore assume its own request shape decided the responsestops, a worker that demands the envelope rejects every search — a two-sided deploy hazard.Recognising the envelope *structurally* removes both. A response is the envelope only when it carries-exactly ``data`` and ``sources`` **and** the caller's own output structure does not declare those two-names itself — that last clause is what stops a legitimate ``{data, sources}`` output class from being-unwrapped into its own sub-object.+exactly ``data`` and ``sources`` **and** the caller's own output structure does not put those two names+on the wire itself — that last clause is what stops a legitimate ``{data, sources}`` output class from+being unwrapped into its own sub-object."""from typing import Any, cast@@ -49,12 +49,30 @@ def extract_structured_search_payload(*, response: Any, schema: type[BaseModel])def _is_sources_envelope(*, payload: dict[str, Any], schema: type[BaseModel]) -> bool:"""Whether this object is Linkup's sources envelope rather than the structured payload itself.- Field names *and* aliases are checked because the response is keyed by the schema's property names,- which pydantic emits by alias — so an output structure that reaches the wire as ``{data, sources}``- must be recognised as its own payload whichever way it spells those fields in Python.+ The declared names come from the generated JSON schema rather than from ``model_fields``, because the+ schema is what the provider was handed and the response is keyed by its property names. Pydantic emits+ those properties *by alias*, and only the schema knows which of a field's alias forms actually reaches+ the wire: ``Field(validation_alias=...)`` renames the property while leaving ``field.alias`` ``None``,+ whereas ``Field(serialization_alias=...)`` does not rename it at all. Reading the schema keeps this+ guard aligned with the provider by construction, instead of re-deriving pydantic's alias precedence+ here. It is reached only once the payload carries exactly the two envelope keys, so the schema build+ costs nothing on the ordinary path."""if frozenset(payload) != _ENVELOPE_KEYS:return False- declared = set(schema.model_fields)- declared.update(field.alias for field in schema.model_fields.values() if field.alias)- return not _ENVELOPE_KEYS.issubset(declared)+ return not _ENVELOPE_KEYS.issubset(_declared_wire_names(schema=schema))+++def _declared_wire_names(*, schema: type[BaseModel]) -> set[str]:+ """The property names the caller's output structure puts on the wire.++ A schema whose root is a ``$ref`` — what pydantic emits for a self-referencing model — carries its+ properties under ``$defs`` rather than at the top level, so the reference is followed before reading+ them; otherwise such a class would declare nothing and have its own payload unwrapped.+ """+ json_schema = schema.model_json_schema()+ root_ref = json_schema.get("$ref")+ if isinstance(root_ref, str):+ definitions = cast("dict[str, Any]", json_schema.get("$defs", {}))+ json_schema = cast("dict[str, Any]", definitions.get(root_ref.rsplit("/", 1)[-1], {}))+ return set(cast("dict[str, Any]", json_schema.get("properties", {})))diff --git a/tests/unit/pipelex/providers/gateway/test_gateway_structured_search_contract.py b/tests/unit/pipelex/providers/gateway/test_gateway_structured_search_contract.pyindex 771312479..a4099da9e 100644--- a/tests/unit/pipelex/providers/gateway/test_gateway_structured_search_contract.py+++ b/tests/unit/pipelex/providers/gateway/test_gateway_structured_search_contract.py@@ -15,7 +15,7 @@ import jsonfrom typing import TYPE_CHECKING, Anyimport pytest-from pydantic import BaseModel+from pydantic import BaseModel, Fieldfrom pipelex.providers.gateway.gateway_exceptions import GatewaySearchEmptyResultError, GatewaySearchResponseErrorfrom pipelex.providers.gateway.gateway_search_worker import GatewaySearchWorker@@ -36,6 +36,20 @@ class DataAndSources(BaseModel):sources: list[str]+class AliasedDataAndSources(BaseModel):+ """The same shape, spelled through aliases: it reaches the wire as {data, sources} all the same."""++ body: dict[str, str] = Field(validation_alias="data")+ refs: list[str] = Field(validation_alias="sources")+++class SerializationAliasedTopicSummary(BaseModel):+ """A `serialization_alias` renames the dump, never the schema the provider was handed."""++ title: str = Field(serialization_alias="data")+ summary: str = Field(serialization_alias="sources")++def _make_worker(mocker: MockerFixture, *, content: str) -> GatewaySearchWorker:worker = object.__new__(GatewaySearchWorker)mock_model = mocker.MagicMock()@@ -101,6 +115,38 @@ class TestGatewayStructuredSearchContract:assert result == payload, "the caller's own {data, sources} output is its payload, not an envelope around one"+ async def test_an_output_class_whose_wire_names_are_aliases_is_not_unwrapped(self, mocker: MockerFixture) -> None:+ """The same shape as above, reached through `validation_alias` rather than plain field names.++ The provider is handed `model_json_schema()`, which pydantic emits by alias, so this class reaches+ the wire as {data, sources} exactly like `DataAndSources` — while `field.alias` stays `None` on both+ fields. Recognising the envelope from the Python field names alone would mistake this class's own+ payload for one and hand the leaf its `data` member: a silent wrong answer of precisely the kind the+ structural check exists to prevent.+ """+ payload = {"data": {"nested": "value"}, "sources": ["a"]}+ worker = _make_worker(mocker, content=json.dumps(payload))++ result = await worker._search_structured(search_job=_make_search_job(mocker), schema=AliasedDataAndSources) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]++ assert result == payload, "an aliased {data, sources} output class owns those names as much as one that spells them out"+ assert AliasedDataAndSources.model_validate(result).body == {"nested": "value"}++ async def test_a_serialization_alias_does_not_make_a_class_own_the_envelope_names(self, mocker: MockerFixture) -> None:+ """The other direction: only the alias that reaches the *schema* counts.++ A `serialization_alias` renames the dump, not the property the provider sees, so this class does not+ own {data, sources} and a real envelope must still be unwrapped for it. Pinning this stops the guard+ from being "fixed" by folding every alias kind into the declared set.+ """+ payload = {"title": "pipelex", "summary": "a language"}+ content = json.dumps({"data": payload, "sources": [{"name": "src", "url": "https://example.com", "snippet": "…"}]})+ worker = _make_worker(mocker, content=content)++ result = await worker._search_structured(search_job=_make_search_job(mocker), schema=SerializationAliasedTopicSummary) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]++ assert result == payload+async def test_an_envelope_carrying_no_payload_raises_an_empty_result_error(self, mocker: MockerFixture) -> None:"""A search that found nothing is not a malformed response — it must not advise changing model."""worker = _make_worker(mocker, content=json.dumps({"data": None, "sources": []}))diff --git a/wip/refactoring/leaf-conversion-and-search-follow-ups.md b/wip/refactoring/leaf-conversion-and-search-follow-ups.mdindex c8fbbcf46..f51f49fee 100644--- a/wip/refactoring/leaf-conversion-and-search-follow-ups.md+++ b/wip/refactoring/leaf-conversion-and-search-follow-ups.md@@ -15,6 +15,7 @@ Every claim below was re-verified against the tree on **2026-07-31 at `2cc27966a| [RF-5](#rf-5--the-llm-paths-schema-check-is-implicit-and-a-plausible-optimisation-would-delete-it) | The LLM path's schema check is implicit, and a plausible optimisation deletes it | true | small | nothing || [RF-6](#rf-6--dry-search-leaves-report-no-usage) | Dry search leaves report no usage | true, pre-existing | small design | nothing || [RF-7](#rf-7--the-relays-envelope-contract-is-unpinned-on-the-producing-side) | The relay's envelope contract is unpinned on the producing side | true | small, cross-repo | a `pipelex-relay` PR |+| [RF-8](#rf-8--an-envelope-whose-data-is-neither-null-nor-an-object-is-called-empty-rather-than-malformed) | An envelope whose `data` is neither null nor an object is called *empty* rather than *malformed* | true, deliberate | small, but two workers | nothing — decide whether the split is worth it |**What the review already fixed, so you do not re-find it:** the direct Linkup backend silently validating a `{data, sources}` envelope into an all-defaults object; a billed search whose shape was rejected vanishing from the cost report; `pipelex validate` no longer catching an output class that cannot emit a JSON schema; the gateway's positional `.get("data")` unwrap; `JSONDecodeError` escaping unclassified; the misleading "try a different model" advice on an empty result; and `linkup/exceptions.py` not following the `<provider>_exceptions.py` convention.@@ -269,6 +270,46 @@ Worth doing anyway, and cheap: a relay-side regression test pinning `{data, sour---+## RF-8 — An envelope whose `data` is neither null nor an object is called *empty* rather than *malformed*++Raised by a review bot on the v0.42.0 release PR (#1078) against `linkup_search_worker.py`. Verified true as a mechanic, judged **not worth fixing there** — recorded here rather than patched under a release.++### Verified++Both workers make the malformed-vs-empty split the extractor's docstring promises, and both make it the same way:++```bash+sed -n '178,182p' pipelex/providers/linkup/linkup_search_worker.py+# payload = extract_structured_search_payload(response=response, schema=schema)+# if payload is not None: return payload+# if isinstance(response, dict): ← empty; else → malformed+```++`extract_structured_search_payload` returns `None` in exactly two places (`structured_search_payload.py`): the response is not a dict at all, or it *is* the envelope and `payload["data"]` is not a dict. The `isinstance` line excludes the first, so the empty branch is reachable iff an envelope arrived carrying `data` of `None`, `list`, `str`, `int`, `float` or `bool`. Only `data: null` is genuinely "the search found nothing"; a list or a scalar is a provider contract breach — the schema always crosses as `model_json_schema()`, which is always `"type": "object"` — and gets `CHANGE_INPUT` / "try a broader query" advice it does not deserve.++The extractor's own docstring closes that sub-split on purpose: *"an envelope whose `data` is null **or not an object** (the search simply found nothing…)"*. So this is a documented simplification, not an oversight.++### Why it was not fixed on the release PR++- Blast radius is the advice string and the message wording, on a path that already raises a classified error in the right family. Nothing is silently wrong.+- The only shape observed in practice is `data: null`, which is classified correctly today. `data: {}` never reaches the branch at all — it is a dict, so it is returned as the payload and the leaf decides.+- `gateway_search_worker.py` is structurally identical here, so fixing linkup alone would *create* a divergence where there is currently none. The change is two workers plus the shared helper plus both contract suites.++### If you do it++Put the predicate in the module that owns the envelope rule, not in the call sites: a `structured_search_payload` helper returning "envelope present **and** `data` is null", and swap the `isinstance(response, dict)` line in both workers for it. Inlining `response["data"] is None` at each call site instead hard-codes the helper's internals in two places and breaks the day the helper grows a third `None` case. Then tighten the extractor docstring's "null or not an object" → "null", or it becomes the drifted artifact. Make the fall-through message name the offending `data` type rather than the outer `dict`.++**Do not** reach for a three-way outcome enum + result type + `match/case` in both workers. That is the shape the docstring literally describes, but it is a lot of new machinery to change an advice string on a shape never observed.++### Two coverage gaps found while verifying this++Both are real, independent of whether RF-8 itself is ever done:++1. **The shared rule has no direct test.** `pipelex/cogt/search/structured_search_payload.py` is exercised only indirectly, through the two worker contract suites. It is the one piece of logic both backends share, and the alias defect fixed on #1078 lived in it.+2. **The linkup suite is missing the `DataAndSources` case.** `test_gateway_structured_search_contract.py` pins `test_an_output_class_that_declares_data_and_sources_is_not_unwrapped` — "the one shape a positional `.get('data')` unwrap would silently corrupt" — and its linkup counterpart has no equivalent, despite running the same shared rule. The aliased variants added on #1078 landed on the gateway side only, for the same reason.++---+## ProvenanceFindings came from a `/review` pass on `refactor/Follow-ups`: specialist reviewers (testing, maintainability, security, performance, api-contract), a red-team pass, an independent fresh-context adversarial pass, and a Codex adversarial pass. Claims that survived were re-verified by hand — the silent-envelope corruption, the schema-generability regression and the `validate_by_alias` edge were each proven or disproven by running pydantic directly rather than by reading. Several confident-sounding findings did **not** survive and are deliberately absent: `by_alias=True` was attacked with `serialization_alias`≠`validation_alias`, `AliasChoices`, `AliasPath`, `extra="allow"` and the real spec classes without producing a regression, and the "json dump mode breaks strict types" claim is already the deliberately-deferred residue recorded in the branch's own notes.