Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
fix/Defaulted-field-refuses-null · L-261005-835cb8 (#1408)
Louis Choquelb009163d1ba1
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 1e4860182..43b00ee35 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -16,6 +16,7 @@- **Exceptions ride as fields at every level**: a `DEBUG` or `VERBOSE` line about a handled exception, such as a Google client that could not be closed, has a fixed message and carries the exception's class and text as `error.type` and `error.message`, as the lines at `INFO` and above already did, rather than splicing them into its text. The retry line names the class of what the attempt raised under `error.type` rather than `exception_type`.- **`make check-log-calls` holds a message's wording and its exceptions**: the log-call guard also refuses, at every level, a message that starts with a lowercase letter, ends with a period or an ellipsis, holds a backtick or splices a handled exception into its text, through an f-string, a `%`, a `.format()` or a task's `exception()`, and, at `INFO` and above, a message that holds an identifier, a word with an underscore or a call such as `setup()`, or runs past 80 characters. The rules are `lowercase-start`, `trailing-period`, `backtick`, `spliced-exception`, `identifier` and `length`, specified on the log-call guard's page of the contributor documentation, and every call in the tree passes them with an empty baseline.- **A refused enum value names the valid ones**: a validation error for an unknown enum value, in a configuration file or anywhere else Pipelex formats pydantic's errors, now lists the values the field accepts, `invalid enum value 'inptu', expected 'input', 'input_cached', …`, where it used to name only the refused value.+- **A field that never holds nothing refuses `null` (Breaking)**: a structure field with a `default_value` is generated as `X = Field(default=…)` rather than `X | None`, and a required field naming `native.Anything` as `NonNullAny` (`pipelex.core.stuffs.non_null_any`), any value but `None`, so a value setting either to `null`, an input or a model's answer, is refused, and a binding step reading such a field never records an absence its static analysis ruled out. Leave the key out, or leave its CSV cell empty, to get a field's default. The codegen projections follow: a defaulted field is `X = Field(default=…)` in `python-pydantic` and `python-structures` and a bare `.default(…)` in `ts-zod`, typed `T` rather than `T | null`. The JSON schema of a defaulted field, the one a pipe's IO contract publishes as an input's `json_schema` and a model is asked to fill, drops its `null` arm: `{"type": "string", "default": …}` rather than `{"anyOf": [{"type": "string"}, {"type": "null"}], "default": …}`.### Fixed@@ -23,13 +24,13 @@- **A condition whose outcome may be skipped declares it**: a PipeCondition routing to an outcome pipe that takes plainly an input the condition declares optional must declare its own output optional, and is refused with `optional_not_handled` otherwise, since a run without that input skips the outcome and resolves the condition's output absent; an outcome the condition's expression chooses only with that input present declares it forced (`!`) instead, since validation does not read the expression. Every name such an outcome always stores in its caller's memory counts as maybe-absent too, so a sequence binding one into a plain output is refused where it used to validate and then resolve absent; the outcome is listed in `liftable_pipes` under its condition, and a `!` input on an outcome counts in the `optional_force_redundant` lint.- **A sequence step that may be skipped keeps what its run may store**: a step an absent input may lift, whose pipe stores a name on some runs only, now leaves that name maybe-absent when a run may store an absence there, so a sequence binding it into a plain output is refused with `optional_not_handled` rather than validating and then resolving absent.- **A configuration value of the wrong type is a validation error (Breaking)**: a model's `costs`, `listed_constraints`, `valued_constraints` or `rules` in a backend file or in a plugin's internal model, a backend's constraints in `backends.toml`, and `package_log_levels` in `pipelex.toml` no longer crash the boot with a bare `AttributeError` or `TypeError` naming nothing when written as a value of the wrong shape, such as `costs = "free"` or a date; the boot fails with a message naming the field, the model, the backend and the file, or the plugin, and an empty `inference.dry_run.image_urls` is reported the same way. A plugin may set an internal model's `costs` to `None`. Some values that used to be accepted are now refused: a table as `listed_constraints`, a string as `interpreter.scan.excluded_dirs`, a price that is `nan` or infinite, and a `max_tokens` setting that is neither an integer nor `"auto"`, which used to fall back to the model's default.+- **A construct path holding nothing leaves a defaulted field to its default**: a `PipeCompose` construct field fed `{ from = "doc.note" }` where `note` holds nothing is left unset when the target field refuses `null` but has a default, so it takes that default rather than failing the composition; a target field that admits `null` still receives it.+- **A model's `null` on a defaulted field is refused where it can be asked again**: the class a worker rebuilds from a structure's JSON schema keeps a defaulted field non-nullable, so a model answering `null` there is asked again rather than failing when its answer reaches the caller's class. A model answering `null` on a required `native.Anything` field is asked again in process, but across a worker boundary its schema stays open, so the step fails there.+- **A field whose `Literal` lists `None` may hold nothing**: a Python class field typed `Literal["a", None]` or `object` is read as one that may hold nothing, so a binding step reading it declares a possible absence and a `PipeCompose` path holding nothing gives it `None` rather than its default.### Removed- **`ConfigModelError` and `ConfigModel`'s conversion helpers (Breaking)**: `ConfigModel.transform_dict_str_to_enum`, `transform_dict_of_floats_str_to_enum`, `transform_dict_keys_str_to_enum` and `transform_list_of_str_to_enum` are gone, and with them `ConfigModelError`, which only they raised. A configuration model holding enums in a container annotates them with `LaxEnum` from `pipelex.system.configuration.config_model`, `list[LaxEnum[MyEnum]]`, and pydantic converts the strings itself.--### Removed-- **`[inference.gateway_test]` (Breaking)**: the maintainer-only section whose `config_id_substitutions` map swapped the Pipelex Gateway's model configurations is gone, with its `GatewayTestConfig` model and the `gateway_test` field of `InferenceConfig`, as nothing has read it since the Gateway backend left in 0.73.0. **Migration:** run `pipelex migrate`, which deletes the section from the file on disk (ledger entry `pipelex-config@8`).### Securitydiff --git a/docs/building-methods/concepts/inline-structures.md b/docs/building-methods/concepts/inline-structures.mdindex 85aa9eba9..4dbfe2d09 100644--- a/docs/building-methods/concepts/inline-structures.md+++ b/docs/building-methods/concepts/inline-structures.md@@ -57,7 +57,7 @@ Each field can specify:- **type**: The data type (required for detailed definitions)- **description**: Human-readable description- **required**: Whether the field must be present in the payload (defaults to `false`)-- **default_value**: Default value applied when the field is omitted. A field cannot declare both `required = true` and a `default_value` — a default makes absence legal, which contradicts requiring the field, so the pair is rejected at validation+- **default_value**: Default value applied when the field is omitted. A field cannot declare both `required = true` and a `default_value` — a default makes absence legal, which contradicts requiring the field, so the pair is rejected at validation. A defaulted field never holds nothing: a value that sets it to `null` is refused, so leave the key out to get the default- **choices**: For `enum`-like fields, a list of valid values- **item_type**: For `list` fields, the type of list items- **key_type** and **value_type**: For `dict` fields, the types of keys and values@@ -300,6 +300,8 @@ bio = { type = "text", description = "User bio" } # Optional (default)In this example, `username` and `email` are mandatory, while `bio` is optional.+Only an optional field with no `default_value` may hold nothing, and it is the only kind of field a value may set to `null`. A required field refuses `null`, a required field naming `native.Anything` included, which takes any value but `null`, and so does a defaulted one. This is the rule a [binding step](../pipes/understanding-optionality.md#binding-steps-under-absence) reading a field relies on: binding a required or defaulted field never records an absence.+## Generating Python ClassesThe `pipelex build structures` command generates Python classes from your inline definitions. This is useful when you want:diff --git a/docs/building-methods/pipes/csv-input-and-output.md b/docs/building-methods/pipes/csv-input-and-output.mdindex e3b778b1d..88c48f3bc 100644--- a/docs/building-methods/pipes/csv-input-and-output.md+++ b/docs/building-methods/pipes/csv-input-and-output.md@@ -75,9 +75,9 @@ Vint Cerf,Computer Scientist,United States,1943,- A missing **required** column is an error.-- A missing **optional** column is fine — that field is `None` for every row. (Pipelex's own writer always emits every column, so this only matters for hand-authored partial CSVs.)+- A missing **optional** column is fine — a field with a `default_value` takes its default for every row, and any other optional field is `None` for every row. (Pipelex's own writer always emits every column, so this only matters for hand-authored partial CSVs.)-- An **empty cell** becomes `None` before validation, so the field must be optional to accept it.+- An **empty cell** means no value: a field with a `default_value` takes its default, any other optional field becomes `None`, and a required field refuses it.### Type coercion@@ -102,7 +102,7 @@ pipelex run bundle demo.mthds --save-csv out.csvpipelex run method my_method --save-csv out.csv```-The path is **literal and cwd-relative** — it is written exactly where you point it, not under `--output-dir`. Parent directories are created as needed. The output's main stuff must be a flat list; if it is not a list, or its row concept is not flat, you get a clear error rather than a malformed file. The header is derived from the row concept's structure — the class the run produced its rows with, or, for a run that produced no rows, the declared concept resolved by reloading the method — so an empty list still writes a correct header-only file. On write, `None` is rendered as an empty cell and booleans as lowercase `true`/`false`, so a value round-trips back through a read unchanged.+The path is **literal and cwd-relative** — it is written exactly where you point it, not under `--output-dir`. Parent directories are created as needed. The output's main stuff must be a flat list; if it is not a list, or its row concept is not flat, you get a clear error rather than a malformed file. The header is derived from the row concept's structure — the class the run produced its rows with, or, for a run that produced no rows, the declared concept resolved by reloading the method — so an empty list still writes a correct header-only file. On write, `None` is rendered as an empty cell and booleans as lowercase `true`/`false`, so a value round-trips back through a read unchanged, except an empty text: it is written as an empty cell too, so it reads back as the field's default, or as `None` for an optional field without one.## Round-trip examplediff --git a/docs/building-methods/pipes/pipe-operators/PipeCompose.md b/docs/building-methods/pipes/pipe-operators/PipeCompose.mdindex 87b83b487..38a9517d1 100644--- a/docs/building-methods/pipes/pipe-operators/PipeCompose.md+++ b/docs/building-methods/pipes/pipe-operators/PipeCompose.md@@ -166,6 +166,8 @@ The referenced value must be a list, and every item must carry the key attributeA `from` path and a `list_to_dict_keyed_by` name read public fields only: validation refuses a segment starting with an underscore, such as `{ from = "order._stuff" }`, for the reason the [Template Sandbox](../../../under-the-hood/template-sandbox.md) gives.+A `from` path reaching a field that holds nothing, such as an optional `order.note` left unset, gives the target field nothing. A target field with a `default_value` is left unset and takes its default, since it never holds nothing; an optional one holds nothing; and a required one is refused. A Python class field typed `X | None` holds nothing whether it is required or defaulted. With `list_to_dict_keyed_by`, a path holding nothing is refused, since the modifier needs a list.+### Copying Whole Inputs Into Native FieldsThe `from` reference is not limited to dotted paths like `"customer.name"` — it can name a whole input variable. When the referenced input is a native stuff (`Text`, `Number`, `YesNo`, `Date`, `Time`, or a list of them) and the target field is native-typed, the composer automatically converts the content wrapper into the field's native value. This works for required and optional fields alike.diff --git a/docs/under-the-hood/codegen-projections.md b/docs/under-the-hood/codegen-projections.mdindex 8ecc2f3e1..e3bc98e37 100644--- a/docs/under-the-hood/codegen-projections.md+++ b/docs/under-the-hood/codegen-projections.md@@ -31,14 +31,16 @@ For **`python-pydantic`**, no key mapping is ever needed either: wire names are### Optionality: the wire carries explicit nulls-A projection describes the payload the engine actually produces, and that payload spells an unset optional field as an explicit `null` — not as an absent key. The runtime class generated for a concept annotates every non-required field `X | None`, and the transport dump keeps nulls deliberately (`dump_for_transport()` is a `model_dump(serialize_as_any=True)` with no `exclude_none`, and the composer spells `exclude_none=False` at its dump sites), because the shape also feeds the Temporal data converter and the hydrator. A partial payload arriving from a caller, meanwhile, may simply omit the key. **The wire genuinely carries both spellings of "unset", so every projection must accept both.**+A projection describes the payload the engine actually produces, and that payload spells an unset optional field as an explicit `null` — not as an absent key. The runtime class generated for a concept annotates every field that may hold nothing `X | None`, and the transport dump keeps nulls deliberately (`dump_for_transport()` is a `model_dump(serialize_as_any=True)` with no `exclude_none`, and the composer spells `exclude_none=False` at its dump sites), because the shape also feeds the Temporal data converter and the hydrator. A partial payload arriving from a caller, meanwhile, may simply omit the key. **The wire genuinely carries both spellings of "unset", so every projection must accept both.**++A field **may hold nothing** only when it is neither `required` nor defaulted, which is the standard's rule and the one a binding step derives a read's presence from. A defaulted field never holds nothing: the runtime class annotates it `X = Field(default=…)` and refuses an explicit `null` for it, so the engine always puts a value there, and a payload omitting the key takes the default.Per target that means:-- **`python-pydantic` / `python-structures`** — `X | None = Field(default=None)`. Pydantic accepts an omitted key and an explicit `null` alike, so nothing further is needed.-- **`ts-zod`** — `.nullish()` for a non-required field (`T | null | undefined`), and `.nullable().default(…)` for a non-required field carrying a default, since a producer may null it explicitly. A bare `.optional()` would mean `T | undefined` in Zod and would **reject the engine's own output**; a bare `.default(…)` rejects the explicit null the same way. There is deliberately no `.transform()` folding `null` into `undefined`: the binder uses one schema for both `parse` and `serialize`, so a transform would make the pair asymmetric.+- **`python-pydantic` / `python-structures`** — `X | None = Field(default=None)` for a field that may hold nothing, since pydantic accepts an omitted key and an explicit `null` alike, and `X = Field(default=…)` for a defaulted field, which refuses the `null` the runtime refuses.+- **`ts-zod`** — `.nullish()` for a field that may hold nothing (`T | null | undefined`), and a bare `.default(…)` for a defaulted field, which infers `T` and refuses `null`. A bare `.optional()` would mean `T | undefined` in Zod and would **reject the engine's own output**. There is deliberately no `.transform()` folding `null` into `undefined`: the binder uses one schema for both `parse` and `serialize`, so a transform would make the pair asymmetric.-On the TypeScript side the consequence is that a parsed optional field types as `T | null | undefined` rather than `T | undefined` — `??` covers both. For a recursive concept, where the emitter writes an explicit `export type` beside a `z.ZodType<Name>`-annotated schema, the declared type carries the same widening (`next?: Node | null`), because that annotation is checked against the schema's inferred output and stops typechecking otherwise.+On the TypeScript side the consequence is that a parsed optional field types as `T | null | undefined` rather than `T | undefined` — `??` covers both — while a defaulted one types as `T`. For a recursive concept, where the emitter writes an explicit `export type` beside a `z.ZodType<Name>`-annotated schema, the declared type carries the same widening (`next?: Node | null`), because that annotation is checked against the schema's inferred output and stops typechecking otherwise.### Refinement and native basesdiff --git a/pipelex/codegen/emitters/python_pydantic.py b/pipelex/codegen/emitters/python_pydantic.pyindex adf9031f9..099f68ddf 100644--- a/pipelex/codegen/emitters/python_pydantic.py+++ b/pipelex/codegen/emitters/python_pydantic.py@@ -78,7 +78,8 @@ def _base_model(*, imports: set[str]) -> str:def _render_field(concept_field: ResolvedField, *, by_ref: dict[str, ResolvedConcept], imports: set[str]) -> str:annotation = _annotation(concept_field.resolved_type, by_ref=by_ref, imports=imports)- if not concept_field.required:+ # Only a field that may hold nothing is nullable: a defaulted field is never `None`, as in the runtime class.+ if concept_field.may_hold_nothing:annotation = f"{annotation} | None"return field_line(concept_field, annotation=annotation, imports=imports)diff --git a/pipelex/codegen/emitters/python_structures.py b/pipelex/codegen/emitters/python_structures.pyindex 7f454654c..aa71aba60 100644--- a/pipelex/codegen/emitters/python_structures.py+++ b/pipelex/codegen/emitters/python_structures.py@@ -104,7 +104,8 @@ def _structured_content(*, imports: set[str]) -> str:def _render_field(concept_field: ResolvedField, *, by_ref: dict[str, ResolvedConcept], imports: set[str]) -> str:annotation = _annotation(concept_field.resolved_type, by_ref=by_ref, imports=imports)- if not concept_field.required:+ # Only a field that may hold nothing is nullable: a defaulted field is never `None`, as in the runtime class.+ if concept_field.may_hold_nothing:annotation = f"{annotation} | None"return field_line(concept_field, annotation=annotation, imports=imports)diff --git a/pipelex/codegen/emitters/ts_zod.py b/pipelex/codegen/emitters/ts_zod.pyindex 86eb348df..669942233 100644--- a/pipelex/codegen/emitters/ts_zod.py+++ b/pipelex/codegen/emitters/ts_zod.py@@ -139,20 +139,21 @@ def _render_field(def _presence_modifiers(concept_field: ResolvedField) -> list[str]:"""The zod member calls that encode a field's presence, in chain order.- Both non-required spellings are **null-tolerant**, because the runtime serializes an unset optional- field as an explicit `null`: the generated runtime class annotates every non-required field `X | None`- and `dump_for_transport()` keeps nulls on purpose (no `exclude_none`). `.optional()` means- `T | undefined` in zod and rejects that payload, so a schema that spelled it would fail to parse the- very wire the engine produces. The wire genuinely carries both spellings of "unset" — a key a partial- payload omitted, and a key the runtime nulled — and `.nullish()` is the honest description of that.+ A field that may hold nothing is **null-tolerant**, because the runtime serializes an unset optional+ field as an explicit `null`: the generated runtime class annotates such a field `X | None` and+ `dump_for_transport()` keeps nulls on purpose (no `exclude_none`). `.optional()` means `T | undefined`+ in zod and rejects that payload, so a schema that spelled it would fail to parse the very wire the+ engine produces. The wire genuinely carries both spellings of "unset" — a key a partial payload+ omitted, and a key the runtime nulled — and `.nullish()` is the honest description of that.A defaulted field is non-required by construction (the blueprint validator refuses `required = true`- beside a `default_value`, E3), and a producer may set it to `None` explicitly, so `.default(...)` needs- `.nullable()` in front of it for the same reason.+ beside a `default_value`, E3), but it never holds nothing: the runtime class refuses an explicit `None`+ for it, so the engine always serializes a value there, and a payload omitting the key takes the default.+ A bare `.default(...)` says exactly that."""if concept_field.default_value is not None:- return [".nullable()", f".default({_format_default_value(concept_field.default_value)})"]- if not concept_field.required:+ return [f".default({_format_default_value(concept_field.default_value)})"]+ if concept_field.may_hold_nothing:return [".nullish()"]return []@@ -192,7 +193,7 @@ def _render_broken_call(call: str, *, indent: int) -> str:Prettier re-measures every call once the chain is broken, and one that now fits is left flat — thethreshold is exact, `indent + len(call) <= TS_PRINT_WIDTH` stays on the line. Exploding unconditionallyis what a `z.enum` special case looks like from the other side: an ordinary three-choice enum carrying a- default reaches this branch on its `.nullable()` width alone, sits far inside the width at indent 4, and+ default reaches this branch on its `.default(…)` width alone, sits far inside the width at indent 4, andwas being exploded into a shape prettier immediately folds back — a consumer's first format run changesthe bytes and `pipelex codegen check` reports an untouched artifact as hand-edited. Only the enum'sarguments are modelled here, for the same reason as in the single-call branch.@@ -369,10 +370,10 @@ def _render_type_field(type_expr = _ts_type(concept_field.resolved_type, by_ref=by_ref, type_name_by_ref=type_name_by_ref)# Mirror `_presence_modifiers` branch for branch: this declared type is what the annotated# `z.ZodType<Name>` is checked against, so it has to be exactly the schema's inferred output —- # `.nullable().default(…)` infers `T | null`, `.nullish()` infers `T | null | undefined`.+ # `.default(…)` infers `T`, `.nullish()` infers `T | null | undefined`.if concept_field.default_value is not None:- return f" {concept_field.name}: {type_expr} | null;"- if not concept_field.required:+ return f" {concept_field.name}: {type_expr};"+ if concept_field.may_hold_nothing:return f" {concept_field.name}?: {type_expr} | null;"return f" {concept_field.name}: {type_expr};"diff --git a/pipelex/cogt/content_generation/schema_to_model_factory.py b/pipelex/cogt/content_generation/schema_to_model_factory.pyindex 22db018dc..aa669498f 100644--- a/pipelex/cogt/content_generation/schema_to_model_factory.py+++ b/pipelex/cogt/content_generation/schema_to_model_factory.py@@ -192,12 +192,18 @@ class SchemaToModelFactory:# and an LLM filling that schema returns the Python repr# `"Recommendation.Poor_Match"` instead of the value `"Poor Match"`,# which then fails Pydantic validation against the original choice set.+ # `strict_nullable=True` keeps a property nullable only where its schema says so. Without it, a+ # property that is not required and carries a default (a defaulted structure field, which pydantic+ # emits as a plain `{"type": "string", "default": …}`) round-trips as `X | None`, so a model's+ # `null` there passes the rebuilt class and fails only when the result is revalidated into the+ # caller's class, past the point where the model could be asked again.generate(input_=schema_str,input_file_type=InputFileType.JsonSchema,output=output_path,output_model_type=DataModelType.PydanticV2BaseModel,enum_field_as_literal=LiteralType.All,+ strict_nullable=True,formatters=[],)return output_path.read_text(encoding="utf-8")diff --git a/pipelex/core/concepts/concept_structure_blueprint.py b/pipelex/core/concepts/concept_structure_blueprint.pyindex 7638ac1b6..7cb647fef 100644--- a/pipelex/core/concepts/concept_structure_blueprint.py+++ b/pipelex/core/concepts/concept_structure_blueprint.py@@ -47,6 +47,11 @@ class ConceptStructureBlueprintFieldType(StrEnum):CONCEPT = "concept"+def field_may_hold_nothing(*, required: bool, default_value: Any) -> bool:+ """Whether a structure field's value may hold nothing: by the standard's rule, only when it is neither `required` nor defaulted."""+ return not required and default_value is None++class ConceptStructureBlueprint(BaseModel):"""One field of a concept's structure table. Unknown keys are rejected — the field table'skeys are strict, exactly like an input slot table's; hint *content* stays lenient (unknown@@ -74,6 +79,11 @@ class ConceptStructureBlueprint(BaseModel):field as a site. Carried as authored through the crate; the site-over-concept merge is theconsumer's. An empty table is equivalent to no hints and normalizes to absence."""+ @property+ def may_hold_nothing(self) -> bool:+ """Whether a value of this field may hold nothing, by `field_may_hold_nothing`."""+ return field_may_hold_nothing(required=self.required, default_value=self.default_value)+@field_validator("hints", mode="after")@classmethoddef normalize_empty_hints(cls, hints: dict[str, str] | None) -> dict[str, str] | None:diff --git a/pipelex/core/concepts/resolved_fields.py b/pipelex/core/concepts/resolved_fields.pyindex b9600cf37..1571d92c1 100644--- a/pipelex/core/concepts/resolved_fields.py+++ b/pipelex/core/concepts/resolved_fields.py@@ -22,7 +22,7 @@ from typing import Anyfrom pydantic import BaseModel, ConfigDict-from pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprint, ConceptStructureBlueprintFieldType+from pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprint, ConceptStructureBlueprintFieldType, field_may_hold_nothingfrom pipelex.core.concepts.native.concept_native import NativeConceptCodefrom pipelex.core.qualified_ref import QualifiedRef@@ -76,6 +76,11 @@ class ResolvedField(BaseModel):default_value: Any | None = Noneresolved_type: ResolvedType+ @property+ def may_hold_nothing(self) -> bool:+ """Whether a value of this field may hold nothing, by `field_may_hold_nothing`."""+ return field_may_hold_nothing(required=self.required, default_value=self.default_value)+def resolve_structure_fields(structure_blueprint: dict[str, ConceptStructureBlueprint],diff --git a/pipelex/core/concepts/structure_generation/generator.py b/pipelex/core/concepts/structure_generation/generator.pyindex b8f1a265c..edcca4ccf 100644--- a/pipelex/core/concepts/structure_generation/generator.py+++ b/pipelex/core/concepts/structure_generation/generator.py@@ -12,6 +12,8 @@ from pipelex.core.concepts.native.concept_native import NativeConceptCodefrom pipelex.core.concepts.resolved_fields import ResolvedField, ResolvedType, ResolvedTypeKind, resolve_structure_fieldsfrom pipelex.core.concepts.structure_generation.exceptions import ConceptStructureGeneratorError, ConceptStructureValidationError, SyntaxErrorDatafrom pipelex.core.qualified_ref import QualifiedRef+from pipelex.core.stuffs import non_null_any+from pipelex.core.stuffs.non_null_any import NonNullAnyfrom pipelex.core.stuffs.stuff_content import StuffContentfrom pipelex.system.registries.class_registry_access import get_class_registry@@ -20,6 +22,10 @@ from pipelex.system.registries.class_registry_access import get_class_registry_MAX_LINE_LENGTH = 150_WRAP_WIDTH = 120 # conservative: leaves headroom for indentation and syntax overhead+# The annotation of an open field that never holds nothing, imported by name in the generated code.+_NON_NULL_ANY_MODULE = non_null_any.__name__+_NON_NULL_ANY_NAME = "NonNullAny"+class StructureGenerator:"""Generate Pydantic BaseModel classes from concept structure blueprints."""@@ -291,16 +297,25 @@ class StructureGenerator:"""Render one Python field definition from a neutral resolved field."""python_type = self._python_type_from_resolved(resolved_field.resolved_type)- # Make optional if not required. A concept ref renders as a *quoted* forward reference (this- # module, unlike the codegen emitters, has no `from __future__ import annotations`), and- # `"Foo" | None` is a TypeError at runtime — str doesn't implement `|`. Fold the union inside- # the quotes so the whole annotation stays one deferred expression for model_rebuild to eval.- # Only a *bare* quoted ref needs this: `list["Foo"] | None` is a real GenericAlias and works.- if not resolved_field.required:+ # Only a field that may hold nothing admits `None`: by the standard's rule, one that is neither+ # required nor defaulted. A binding step reading a field derives its absence from that rule alone,+ # so a required or defaulted field holding `None` would be an absence its readers ruled out — a+ # defaulted field is therefore `X = Field(default=…)`, and an explicit `null` for it is refused+ # rather than kept. A concept ref renders as a *quoted* forward reference (this module, unlike+ # the codegen emitters, has no `from __future__ import annotations`), and `"Foo" | None` is a+ # TypeError at runtime — str doesn't implement `|`. Fold the union inside the quotes so the whole+ # annotation stays one deferred expression for model_rebuild to eval. Only a *bare* quoted ref+ # needs this: `list["Foo"] | None` is a real GenericAlias and works.+ if resolved_field.may_hold_nothing:if python_type.startswith('"') and python_type.endswith('"'):python_type = f'"{python_type[1:-1]} | None"'else:python_type = f"{python_type} | None"+ elif python_type == "Any":+ # `Any` admits `None` by itself, so an open field that never holds nothing (one naming+ # `native.Anything`, or of an unspecified type) takes the annotation that refuses it.+ self.imports.add(f"from {_NON_NULL_ANY_MODULE} import {_NON_NULL_ANY_NAME}")+ python_type = _NON_NULL_ANY_NAME# Generate Field parameters (default/... first, then description)field_params = [self._format_field_description(resolved_field.description)]@@ -414,6 +429,7 @@ class StructureGenerator:"Any": Any,"Literal": Literal,"Field": Field,+ _NON_NULL_ANY_NAME: NonNullAny,}# Add all native content classes to exec_globals so they're available during validationdiff --git a/pipelex/core/stuffs/non_null_any.py b/pipelex/core/stuffs/non_null_any.pynew file mode 100644index 000000000..4d0646f07--- /dev/null+++ b/pipelex/core/stuffs/non_null_any.py@@ -0,0 +1,41 @@+"""The annotation of an open-typed structure field that never holds nothing.++A field naming `native.Anything`, or one whose type the author left unspecified, is generated as `Any`, and+`Any` admits `None`. The standard says a `required` field never holds nothing, and a binding step reading one+derives no absence from it, so a required open field holding `None` is an absence its readers ruled out.+`NonNullAny` is that field's annotation: any value but `None`.++Its JSON schema stays the open `{}` that `Any` has. The structure class's schema is what a model is asked+for, and `{"not": {"type": "null"}}`, the only spelling of "anything but null" that does not enumerate the+other types, is refused by the strict structured-output modes of several providers, so a model may still+answer `null` there. In process, the answer is validated against the class itself, so that `null` is refused+and the model asked again; across a worker boundary, the class the worker rebuilds from the schema has a bare+`Any` there, so the `null` is only refused when the answer reaches the caller's class, failing the step.+"""++from typing import Annotated, Any++from pydantic import AfterValidator+from pydantic.fields import FieldInfo+from pydantic_core import PydanticCustomError++from pipelex.tools.typing.annotation_utils import annotation_admits_none+++def _refuse_none( # kw-only: ignore -- pydantic calls an AfterValidator positionally+ value: Any,+) -> Any:+ if value is None:+ error_type = "null_value"+ message = "Input should be a value, not null: this field never holds nothing"+ raise PydanticCustomError(error_type, message)+ return value+++NonNullAny = Annotated[Any, AfterValidator(_refuse_none)]+++def field_admits_none(*, field_info: FieldInfo) -> bool:+ """Whether a class field admits `None`, read off its annotation, `NonNullAny` being the one `Any` that does not."""+ is_non_null_any = any(isinstance(constraint, AfterValidator) and constraint.func is _refuse_none for constraint in field_info.metadata)+ return annotation_admits_none(annotation=field_info.annotation) and not is_non_null_anydiff --git a/pipelex/pipe_controllers/binding/binding_concept_resolvers.py b/pipelex/pipe_controllers/binding/binding_concept_resolvers.pyindex 933e43221..774e42f38 100644--- a/pipelex/pipe_controllers/binding/binding_concept_resolvers.py+++ b/pipelex/pipe_controllers/binding/binding_concept_resolvers.py@@ -36,6 +36,7 @@ from pipelex.core.concepts.native.concept_native import NativeConceptCodefrom pipelex.core.concepts.native.pinned_blueprints import make_pinned_native_blueprintfrom pipelex.core.domains.domain import SpecialDomainfrom pipelex.core.qualified_ref import QualifiedRef+from pipelex.core.stuffs.non_null_any import field_admits_nonefrom pipelex.core.stuffs.stuff_content import StuffContentfrom pipelex.libraries.concept.concept_library_abstract import ConceptLibraryAbstractfrom pipelex.libraries.concept.exceptions import ConceptLibraryError@@ -141,9 +142,10 @@ def _scalar_value_kind(*, field_type: ConceptStructureBlueprintFieldType) -> Bindef _walkable_field_from_blueprint(*, name: str, field_blueprint: ConceptStructureBlueprint, domain_code: str, package_alias: str | None) -> WalkableField:- # The standard's rule: a declared field may hold nothing when it is not `required` and has no `default_value`. A- # `required` field is generated as a value that cannot be `None`.- may_hold_nothing = not field_blueprint.required and field_blueprint.default_value is None+ # The standard's rule: a declared field may hold nothing when it is not `required` and has no `default_value`. The+ # structure generator follows the same rule, so a required or defaulted field is generated as a value that cannot+ # be `None`.+ may_hold_nothing = field_blueprint.may_hold_nothingif field_blueprint.type is None:# A field declared by its `choices` alone holds one of them, a text.return WalkableField(name=name, value_kind=BindingValueKind.TEXT, may_hold_nothing=may_hold_nothing)@@ -305,8 +307,9 @@ class LibraryConceptWalkResolver(ConceptWalkResolver):def _walkable_field_from_class_field(self, *, name: str, field_info: FieldInfo, package_alias: str | None) -> WalkableField:# A class states what a field may hold through its annotation: one that admits `None` may hold nothing, whether it- # is required or has a default, and so does one left unrequired with no default.- is_nullable = _admits_none(annotation=field_info.annotation)+ # is required or has a default, and so does one left unrequired with no default. `NonNullAny` is the one `Any` that+ # refuses `None`, as a generated class spells a required open field.+ is_nullable = field_admits_none(field_info=field_info)may_hold_nothing = is_nullable or (not field_info.is_required() and field_info.default is None and field_info.default_factory is None)annotation, _ = _strip_optional(annotation=field_info.annotation)is_list = get_origin(annotation) is list@@ -364,16 +367,6 @@ class LibraryConceptWalkResolver(ConceptWalkResolver):return [concept_key for concept_key in concept_keys if _package_alias_of(concept_key=concept_key) == package_alias] or concept_keys-def _admits_none(*, annotation: Any) -> bool:- """Whether a field annotation admits `None`: `None` itself, `Any`, or a union with a `None` arm."""- if annotation is None or annotation is type(None) or annotation is Any:- return True- origin = get_origin(annotation)- if origin is Union or origin is types.UnionType:- return any(_admits_none(annotation=arm) for arm in get_args(annotation))- return False--def _strip_optional(*, annotation: Any) -> tuple[Any, bool]:"""An annotation without its `None` arm, and whether it had one."""origin = get_origin(annotation)diff --git a/pipelex/pipe_operators/compose/structured_content_composer.py b/pipelex/pipe_operators/compose/structured_content_composer.pyindex a7e12702e..dcba6f85b 100644--- a/pipelex/pipe_operators/compose/structured_content_composer.py+++ b/pipelex/pipe_operators/compose/structured_content_composer.py@@ -8,6 +8,7 @@ from pipelex.cogt.templating.template_rendering import render_templatefrom pipelex.core.memory.working_memory import WorkingMemoryfrom pipelex.core.stuffs.date_content import DateContentfrom pipelex.core.stuffs.list_content import ListContent+from pipelex.core.stuffs.non_null_any import field_admits_nonefrom pipelex.core.stuffs.number_content import NumberContentfrom pipelex.core.stuffs.stuff_content import StuffContentfrom pipelex.core.stuffs.text_content import TextContent@@ -130,15 +131,28 @@ class StructuredContentComposer:"""Resolve all fields in the blueprint to their values.Returns:- Dictionary mapping field names to resolved values+ Dictionary mapping field names to resolved values, a defaulted field refusing `None` left out when its source holds nothing"""field_values: dict[str, Any] = {}for field_name, field_blueprint in self.construct_blueprint.fields.items():- field_values[field_name] = await self._resolve_field(field_blueprint=field_blueprint, field_name=field_name)+ field_value = await self._resolve_field(field_blueprint=field_blueprint, field_name=field_name)+ if field_value is None and self._takes_its_default_over_nothing(field_name=field_name):+ continue+ field_values[field_name] = field_valuereturn field_values+ def _takes_its_default_over_nothing(self, *, field_name: str) -> bool:+ """Whether a target field fed a source holding nothing is left unset, to take its default.++ Only a path reaching a field that holds nothing resolves to `None`. A target field that refuses `None` but has+ a default, as a generated defaulted field does, takes that default. One that admits `None`, as a Python class+ may declare `str | None` with or without a default, keeps it, and a required one refusing it is refused.+ """+ target_field = self.output_class.model_fields.get(field_name)+ return target_field is not None and not target_field.is_required() and not field_admits_none(field_info=target_field)+async def _resolve_field(self, field_blueprint: ConstructFieldBlueprint, *, field_name: str) -> Any:"""Resolve a single field according to its composition method.diff --git a/pipelex/tools/tabular/csv_codec.py b/pipelex/tools/tabular/csv_codec.pyindex 7eeb0898d..d57abc6e9 100644--- a/pipelex/tools/tabular/csv_codec.py+++ b/pipelex/tools/tabular/csv_codec.py@@ -31,6 +31,7 @@ from pipelex.tools.tabular.exceptions import (CsvFlatnessError,CsvReadError,)+from pipelex.tools.typing.annotation_utils import annotation_admits_noneif TYPE_CHECKING:from pydantic_core import ErrorDetails@@ -83,13 +84,6 @@ def _is_flat_annotation(annotation: Any) -> bool:return False-def _annotation_allows_none(annotation: Any) -> bool:- """Whether a field annotation accepts ``None`` (an ``Optional`` / ``... | None`` field)."""- if get_origin(annotation) in {Union, _UnionType}:- return any(arg is _NONE_TYPE for arg in get_args(annotation))- return annotation is _NONE_TYPE--def flat_field_names(row_model: type[StuffContent]) -> list[str]:"""Validate that ``row_model`` is CSV-flat and return its field names in declared order.@@ -320,8 +314,10 @@ def list_content_from_csv(Columns must match ``row_model``'s field names (no implicit remap): an extra columnor a missing *required* column raises ``CsvColumnError``; a missing *optional* column- sets that field to ``None`` for all rows. Empty cells map to ``None`` BEFORE pydantic- validation (so they must target optional fields); remaining strings are coerced via+ sets that field to ``None`` for all rows, or leaves a defaulted field that refuses ``None``+ to its default. Empty cells map to ``None`` BEFORE pydantic validation (so they must target+ optional fields), but are left out for a defaulted field that refuses ``None``, which then+ takes its default; remaining strings are coerced viapydantic's lax validation. A coercion failure raises ``CsvCoercionError`` naming the1-based row/column and field. ``row_model`` must be CSV-flat (see ``flat_field_names``)."""@@ -351,13 +347,25 @@ def list_content_from_csv(# depending on a non-None field default (e.g. ``nickname: str | None = "anon"`` → None, not "anon").# A non-nullable defaulted field (e.g. ``count: int = 0``) is left absent so its own default# applies — forcing None there would fail validation, not honor the column's absence.- omitted_nullable_fields = {name for name in (field_name_set - header_set) if _annotation_allows_none(row_model.model_fields[name].annotation)}+ omitted_nullable_fields = {+ name for name in (field_name_set - header_set) if annotation_admits_none(annotation=row_model.model_fields[name].annotation)+ }+ # An empty cell is "no value" in the same way: a defaulted field that refuses None (a structure field with a+ # `default_value` never holds nothing) is left out of the row so its default applies, as for an omitted column.+ defaulted_non_nullable_fields = {+ name+ for name, field_info in row_model.model_fields.items()+ if not field_info.is_required() and not annotation_admits_none(annotation=field_info.annotation)+ }items: list[StuffContentType] = []for row_number, data_row in data_rows:cell_map = _row_to_dict(data_row, header=header)- # Empty cell -> None BEFORE validation (so it targets an optional field, or fails required).- row_data: dict[str, str | None] = {column: (value or None) for column, value in cell_map.items()}+ # Empty cell -> None BEFORE validation (so it targets an optional field, or fails required), or left out+ # for a defaulted field that refuses None.+ row_data: dict[str, str | None] = {+ column: (value or None) for column, value in cell_map.items() if value or column not in defaulted_non_nullable_fields+ }for omitted_field in omitted_nullable_fields:row_data[omitted_field] = Nonetry:diff --git a/pipelex/tools/typing/annotation_utils.py b/pipelex/tools/typing/annotation_utils.pyindex 01378ff05..a210e9708 100644--- a/pipelex/tools/typing/annotation_utils.py+++ b/pipelex/tools/typing/annotation_utils.py@@ -1,5 +1,5 @@import types-from typing import Any, Union, get_args, get_origin+from typing import Annotated, Any, Literal, Union, get_args, get_origindef unwrap_optional(annotation: Any) -> Any:@@ -25,3 +25,17 @@ def unwrap_optional(annotation: Any) -> Any:if len(non_none_args) < len(args) and len(non_none_args) == 1:return non_none_args[0]return annotation+++def annotation_admits_none(*, annotation: Any) -> bool:+ """Whether an annotation admits `None`: `None` itself, `Any`, `object`, a `Literal` listing `None`, or a union or `Annotated` holding one."""+ if annotation is None or annotation is type(None) or annotation is Any or annotation is object:+ return True+ origin = get_origin(annotation)+ if origin is Annotated:+ return annotation_admits_none(annotation=get_args(annotation)[0])+ if origin is Literal:+ return None in get_args(annotation)+ if origin is Union or origin is types.UnionType:+ return any(annotation_admits_none(annotation=arm) for arm in get_args(annotation))+ return Falsediff --git a/subject_grants.toml b/subject_grants.tomlindex 54d2341dc..b2919c9bb 100644--- a/subject_grants.toml+++ b/subject_grants.toml@@ -6656,10 +6656,6 @@ rationale = "Keyed accessor by the uri; single operand."param = "data"rationale = "Keyed accessor by the data; single operand."-["pipelex/tools/tabular/csv_codec.py::_annotation_allows_none"]-param = "annotation"-rationale = "Validates/checks the annotation; single operand."-["pipelex/tools/tabular/csv_codec.py::_assert_single_char_delimiter"]param = "delimiter"rationale = "Validates/checks the delimiter; single operand."diff --git a/tests/integration/pipelex/pipes/controller/pipe_sequence/test_binding_a_field_that_never_holds_nothing.py b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_binding_a_field_that_never_holds_nothing.pynew file mode 100644index 000000000..177c83aed--- /dev/null+++ b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_binding_a_field_that_never_holds_nothing.py@@ -0,0 +1,108 @@+"""A field that never holds nothing refuses an explicit null, so a binding reading it never records an absence.++The standard's rule says a field that is `required` or carries a `default_value` never holds nothing, and a+binding step derives its static presence from that rule alone: a sequence ending with a binding of such a+field validates with a non-optional output. The runtime class used to admit `None` on a defaulted field+(`X | None = default`) and on a required `native.Anything` field (`Any`), so an input nulling either one was+kept, and the binding recorded a declared absence its own derivation had ruled out. The class now refuses+that `None` where the input enters the run, and leaving the key out binds the default.+"""++from typing import Any++import pytest++from pipelex.config import get_config+from pipelex.core.memory.absence import AbsenceRecord+from pipelex.core.memory.exceptions import InputShapingError+from pipelex.core.stuffs.text_content import TextContent+from pipelex.graph.graph_tracer_manager import GraphTracerManager+from pipelex.interpreter_hub import clear_current_library, get_library_manager, get_pipe_router+from pipelex.pipeline.pipeline_run_setup import pipeline_run_setup+from pipelex.runtime_hub import get_report_delegate++_BINDING_MTHDS = """+domain = "never_nothing"+description = "Bind fields that never hold nothing"++[concept.Inner]+description = "An inner value"++[concept.Inner.structure]+label = { type = "text", description = "a label" }++[concept.Doc]+description = "A document"++[concept.Doc.structure]+note = { type = "text", description = "a note", default_value = "fallback" }+inner = { type = "concept", concept_ref = "Inner", description = "inner", required = true }+payload = { type = "concept", concept_ref = "native.Anything", description = "any value", required = true }++[pipe.bind_note]+type = "PipeSequence"+description = "Bind the defaulted note"+inputs = { doc = "Doc" }+output = "Text"+steps = [{ from = "doc.note", result = "note" }]+"""+++def _cleanup(*, pipeline_run_id: str, library_id: str) -> None:+ """Tear down what `pipeline_run_setup` leaves open on its success path, which its caller owns when it runs the job itself."""+ get_report_delegate().clear_event_log(context_key=pipeline_run_id)+ tracer_manager = GraphTracerManager.get_instance()+ if tracer_manager is not None:+ tracer_manager.close_tracer(pipeline_run_id)+ get_library_manager().teardown(library_id=library_id)+ clear_current_library()+++async def _run_bind_note(*, doc: dict[str, Any]) -> AbsenceRecord | TextContent:+ execution_config = get_config().interpreter.pipeline_execution.with_execution_overrides(generate_graph=False)+ pipe_job, pipeline_run_id, library_id = await pipeline_run_setup(+ storage_scope="test/scope",+ read_scope=None,+ user_id="test-user",+ execution_config=execution_config,+ mthds_contents=[_BINDING_MTHDS],+ pipe_code="bind_note",+ inputs={"doc": doc},+ )+ try:+ pipe_output = await get_pipe_router().run(pipe_job=pipe_job)+ finally:+ _cleanup(pipeline_run_id=pipeline_run_id, library_id=library_id)+ resolved = pipe_output.working_memory.resolve_main_stuff()+ if isinstance(resolved, AbsenceRecord):+ return resolved+ return resolved.content_as(content_type=TextContent)++[email protected](loop_scope="class")+class TestBindingAFieldThatNeverHoldsNothing:+ async def test_an_omitted_defaulted_field_binds_its_default(self):+ bound = await _run_bind_note(doc={"inner": {}, "payload": {"any": "value"}})++ assert isinstance(bound, TextContent)+ assert bound.text == "fallback"++ async def test_a_set_defaulted_field_binds_its_value(self):+ bound = await _run_bind_note(doc={"note": "written", "inner": {}, "payload": [1, 2]})++ assert isinstance(bound, TextContent)+ assert bound.text == "written"++ @pytest.mark.parametrize(+ ("doc", "nulled_field"),+ [+ ({"note": None, "inner": {}, "payload": 1}, "note"),+ ({"inner": {}, "payload": None}, "payload"),+ ],+ )+ async def test_an_explicit_null_on_a_field_that_never_holds_nothing_is_refused(self, doc: dict[str, Any], nulled_field: str):+ with pytest.raises(InputShapingError) as exc_info:+ await _run_bind_note(doc=doc)++ assert exc_info.value.variable_name == "doc"+ assert nulled_field in str(exc_info.value)diff --git a/tests/unit/pipelex/codegen/conftest.py b/tests/unit/pipelex/codegen/conftest.pyindex 1b482443f..8e6f2f960 100644--- a/tests/unit/pipelex/codegen/conftest.py+++ b/tests/unit/pipelex/codegen/conftest.py@@ -309,21 +309,21 @@ def every_type_kind_crate() -> LibraryCrate:choices=["a) strongly agree with the proposal", "b) agree. with some reservations", "c unsure"],required=True,),- # The same unbounded choice list carrying a default — the widest presence spelling- # ts-zod emits (`.nullable().default(…)` on top of a broken `z.enum([…])`), so the- # print-width and prettier guards actually exercise the multi-member chain break.+ # The same unbounded choice list carrying a default — a presence call on top of a broken+ # `z.enum([…])`, so the print-width and prettier guards actually exercise the+ # multi-member chain break."fallback_state": ConceptStructureBlueprint(description="Fallback workflow state",choices=["awaiting_triage", "in_progress_with_owner", "blocked_on_dependency", "ready_for_review"],default_value="awaiting_triage",),- # A short choice list that only overflows once `.nullable().default(…)` is attached.+ # A short choice list that only overflows once `.default(…)` is attached.# Prettier re-measures each call after breaking the chain, so the enum goes back onto# one line at indent 4 — exploding its members regardless is a shape prettier folds# straight back, which is exactly the stamp break the breaking exists to prevent."escalation_severity": ConceptStructureBlueprint(description="How severe the escalation is",- choices=["low", "medium", "high"],+ choices=["low", "medium", "high", "critical"],default_value="medium",),# The overflow that has nothing to do with the *expression*: `z.string()` is as short as@@ -333,7 +333,7 @@ def every_type_kind_crate() -> LibraryCrate:"default_summary_style": ConceptStructureBlueprint(description="How the summary should be written",type=ConceptStructureBlueprintFieldType.TEXT,- default_value="a concise executive summary",+ default_value="a concise executive summary for the board",),# The same overflow reached through a nested type rather than a long default."per_reviewer_summary_style_overrides": ConceptStructureBlueprint(diff --git a/tests/unit/pipelex/codegen/test_ts_zod_emitter.py b/tests/unit/pipelex/codegen/test_ts_zod_emitter.pyindex 738ed1803..e5258f5c9 100644--- a/tests/unit/pipelex/codegen/test_ts_zod_emitter.py+++ b/tests/unit/pipelex/codegen/test_ts_zod_emitter.py@@ -42,22 +42,30 @@ class TestTsZodEmitter:content = emit_ts_zod(resolve_concepts_from_crate(pipeline_crate))[0].content# A concept reference is a forward-safe lazy schema; the literal-with-default is a defaulted enum.assert "score: z.lazy(() => ScoreSchema).nullish()" in content- assert 'status: z.enum(["draft", "final"]).nullable().default("draft")' in content+ assert 'status: z.enum(["draft", "final"]).default("draft")' in content- def test_non_required_fields_are_null_tolerant(self, pipeline_crate: LibraryCrate):- """Both non-required spellings must accept the explicit `null` the runtime puts on the wire.+ def test_a_field_that_may_hold_nothing_is_null_tolerant(self, pipeline_crate: LibraryCrate):+ """A field that may hold nothing must accept the explicit `null` the runtime puts on the wire.An unset optional field is dumped as `"key": null` (the generated runtime class annotates it`X | None` and `dump_for_transport()` carries no `exclude_none`), so a `.optional()` schema —- `T | undefined` in zod — rejects the engine's own payload. `.nullish()` and `.nullable()` are- the two null-tolerant spellings; `.optional()` alone must appear nowhere.+ `T | undefined` in zod — rejects the engine's own payload. `.optional()` must appear nowhere."""content = emit_ts_zod(resolve_concepts_from_crate(pipeline_crate))[0].contentassert ".optional()" not in content# No default: the wire may omit the key *or* null it, and `.nullish()` describes exactly that.assert "rationale: z.string().nullish()" in content- # With a default: absent applies the default, explicit null stays null.- assert 'status: z.enum(["draft", "final"]).nullable().default("draft")' in content++ def test_a_defaulted_field_is_never_null(self, pipeline_crate: LibraryCrate):+ """A defaulted field never holds nothing, so its schema refuses `null` and an omitted key takes the default.++ The runtime class annotates it `X = Field(default=…)` and refuses an explicit `None`, so the engine always+ serializes a value there. A `.nullable()` in front of the default would type the parsed field `T | null`+ and describe a payload the engine never produces.+ """+ content = emit_ts_zod(resolve_concepts_from_crate(pipeline_crate))[0].content+ assert 'status: z.enum(["draft", "final"]).default("draft")' in content+ assert ".nullable()" not in contentdef test_an_overlong_field_breaks_its_whole_member_chain(self, every_type_kind_crate: LibraryCrate):"""Prettier breaks by call count, not by expression, so the break cannot be a `z.enum` special case.@@ -69,7 +77,7 @@ class TestTsZodEmitter:"""content = emit_ts_zod(resolve_concepts_from_crate(every_type_kind_crate))[0].content- assert ' default_summary_style: z\n .string()\n .nullable()\n .default("a concise executive summary"),' in content+ assert ' default_summary_style: z\n .string()\n .default("a concise executive summary for the board"),' in contentassert " per_reviewer_summary_style_overrides: z\n .record(z.string(), z.string())\n .nullish()," in content# A single call has no chain to break: prettier explodes the enum members in place instead.assert ' workflow_state: z.enum([\n "awaiting_triage",' in content@@ -77,8 +85,8 @@ class TestTsZodEmitter:def test_a_broken_chain_keeps_a_call_that_fits_on_its_own_line(self, every_type_kind_crate: LibraryCrate):"""Prettier re-measures every call once the chain is broken, and one that now fits stays flat.- The counterpart to the test above: `.nullable()` costs width on every defaulted field, so an- ordinary three-choice enum reaches the break on nothing but that — and then sits far inside the+ The counterpart to the test above: `.default(…)` costs width on every defaulted field, so an+ ordinary four-choice enum reaches the break on nothing but that — and then sits far inside theprint width at indent 4. Exploding its members anyway produces a shape prettier folds straightback, changing the artifact's bytes on the consumer's first format run. The threshold is exact:a call is left flat while `indent + len(call)` fits, and `fallback_state` above is the same@@ -86,7 +94,7 @@ class TestTsZodEmitter:"""content = emit_ts_zod(resolve_concepts_from_crate(every_type_kind_crate))[0].content- assert ' escalation_severity: z\n .enum(["low", "medium", "high"])\n .nullable()\n .default("medium"),' in content+ assert ' escalation_severity: z\n .enum(["low", "medium", "high", "critical"])\n .default("medium"),' in contentdef test_an_exploded_choice_keeps_its_own_commas(self, every_type_kind_crate: LibraryCrate):"""A choice is authored text and may carry a comma; split on it, the emission stops being TypeScript.@@ -136,9 +144,9 @@ class TestTsZodEmitter:def test_temporal_defaults_emit_iso_wire_strings(self, temporal_defaults_crate: LibraryCrate):content = emit_ts_zod(resolve_concepts_from_crate(temporal_defaults_crate))[0].content- assert 'starts_on: z.string().nullable().default("2026-07-11")' in content- assert 'recorded_at: z.string().nullable().default("2026-07-11T09:30:00")' in content- assert 'starts_at: z.string().nullable().default("09:30:00")' in content+ assert 'starts_on: z.string().default("2026-07-11")' in content+ assert 'recorded_at: z.string().default("2026-07-11T09:30:00")' in content+ assert 'starts_at: z.string().default("09:30:00")' in contentdef test_dict_defaults_are_canonical(self, reordered_dict_default_crates: tuple[LibraryCrate, LibraryCrate]):first_crate, second_crate = reordered_dict_default_crates@@ -190,8 +198,8 @@ class TestTsZodEmitter:assert "export const NodeSchema: z.ZodType<Node> = z.object({" in contentassert "next: z.lazy(() => NodeSchema).nullish()" in content- def test_recursive_defaulted_field_declares_a_nullable_type(self):- """The other explicit-type branch: a defaulted field infers `T | null` with no `?` marker."""+ def test_recursive_defaulted_field_declares_a_non_null_type(self):+ """The other explicit-type branch: a defaulted field infers `T`, with no `?` marker and no `null`."""crate = LibraryCrate(concepts={"graph.Node": ConceptBlueprint(@@ -212,8 +220,8 @@ class TestTsZodEmitter:content = emit_ts_zod(resolve_concepts_from_crate(crate))[0].content- assert "export type Node = {\n next?: Node | null;\n label: string | null;\n depth: number;\n};" in content- assert 'label: z.string().nullable().default("root")' in content+ assert "export type Node = {\n next?: Node | null;\n label: string;\n depth: number;\n};" in content+ assert 'label: z.string().default("root")' in contentassert "depth: z.number().int()," in contentdef test_mutually_recursive_concepts_use_explicit_types_and_annotated_schemas(self):diff --git a/tests/unit/pipelex/codegen/test_ts_zod_wire_agreement.py b/tests/unit/pipelex/codegen/test_ts_zod_wire_agreement.pyindex 40a4b1022..cefd69b65 100644--- a/tests/unit/pipelex/codegen/test_ts_zod_wire_agreement.py+++ b/tests/unit/pipelex/codegen/test_ts_zod_wire_agreement.py@@ -1,20 +1,23 @@"""The ts-zod projection must accept the payload the runtime actually puts on the wire.The two sides read one authored crate. `ConceptFactory` builds the class an interpreted run stores in-memory, and that class annotates every non-required field `X | None`; the transport dump keeps nulls on+memory, and that class annotates every field that may hold nothing `X | None`; the transport dump keeps nulls onpurpose (`dump_for_transport()` is a `model_dump(serialize_as_any=True)` with no `exclude_none`, and thecomposer spells `exclude_none=False` at its dump sites). So an unset optional field reaches a consumer asan explicit `"key": null`. `emit_ts_zod` writes the schema that consumer parses it with — and `.optional()`means `T | undefined` in zod, which *rejects* an explicit null. A projection that spelled it that way would-refuse the engine's own output, with nothing in this repo going red.+refuse the engine's own output, with nothing in this repo going red. A defaulted field is the other+half: it never holds nothing, so the runtime class refuses an explicit `None` for it and the projection+spells it a bare `.default(…)`, which refuses a `null` too.Three layers, on one shared crate:1. **Wire pin** — build the runtime class and dump it the way transport does, asserting the unset keys arepresent and null. This is the fact the projection must accommodate; if the transport dump ever flips to`exclude_none`, this reddens and forces the projection decision to be revisited alongside it.-2. **Projection pin** — emit ts-zod for the same crate and require every non-required field to be- null-tolerant. Paired with (1), the cross-language contract is encoded here even where CI has no node.+2. **Projection pin** — emit ts-zod for the same crate and require every field that may hold nothing to be+ null-tolerant, and the defaulted one not to be. Paired with (1), the cross-language contract is encoded+ here even where CI has no node.3. **Executable round-trip** — feed (1)'s JSON through the emitted schema under a real zod. Mandatorywherever the node toolchain is provisioned (`make test-ts-gates`, which is what CI runs); opportunisticon a machine without it, where the two always-on pins hold the line.@@ -26,6 +29,7 @@ from pathlib import Pathfrom typing import Anyimport pytest+from pydantic import ValidationErrorfrom pipelex.codegen.emitters.ts_zod import emit_ts_zodfrom pipelex.codegen.resolved_concepts import resolve_concepts_from_crate@@ -75,8 +79,8 @@ _AUTHORED: dict[str, ConceptBlueprint] = {),}-# Every non-required field of `Payload`, i.e. every key the runtime can put on the wire as an explicit null.-_NON_REQUIRED_FIELDS = ("note", "detail", "tags", "counts", "status")+# Every field of `Payload` that may hold nothing, i.e. every key the runtime can put on the wire as an explicit null.+_MAY_HOLD_NOTHING_FIELDS = ("note", "detail", "tags", "counts")def _authored_crate() -> LibraryCrate:@@ -88,13 +92,8 @@ def _authored_crate() -> LibraryCrate:class TestTsZodWireAgreement:@pytest.fixture- def wire_payloads(self, load_empty_library: Any) -> list[dict[str, Any]]:- """What the runtime hands to transport for two ordinary instances, unset optionals and all.-- The first leaves every optional unset — the shape the ledger item's hosted `PipeImgGen` run- produced. The second nulls the *defaulted* field explicitly, which a producer may do because the- generated annotation is `X | None`: that is the payload a bare `.default(…)` rejects.- """+ def payload_class(self, load_empty_library: Any) -> type[StuffContent]:+ """The runtime class an interpreted run stores a `Payload` in."""load_empty_library()registry = get_class_registry()classes: dict[str, Any] = {}@@ -108,35 +107,52 @@ class TestTsZodWireAgreement:structure_class.model_rebuild(_types_namespace=classes)# `Detail` exists only to be referenced; `Payload` is the one that gets dumped.- payload_class = classes["Payload"]+ payload_class: type[StuffContent] = classes["Payload"]+ return payload_class++ @pytest.fixture+ def wire_payloads(self, payload_class: type[StuffContent]) -> list[dict[str, Any]]:+ """What the runtime hands to transport for two ordinary instances, unset optionals and all.++ The first leaves every optional unset — the shape the ledger item's hosted `PipeImgGen` run+ produced — and so takes the defaulted field's default. The second sets the defaulted field.+ """return [- payload_class(title="Everything optional left unset").model_dump(serialize_as_any=True),- payload_class(title="A producer nulling the defaulted field", status=None).model_dump(serialize_as_any=True),+ payload_class.model_validate({"title": "Everything optional left unset"}).model_dump(serialize_as_any=True),+ payload_class.model_validate({"title": "A producer setting the defaulted field", "status": "final"}).model_dump(serialize_as_any=True),]def test_the_wire_carries_unset_optionals_as_explicit_nulls(self, wire_payloads: list[dict[str, Any]]):"""The fact the projection has to accommodate — pinned here so it cannot change unnoticed."""- unset, nulled_default = wire_payloads+ unset, set_default = wire_payloadsassert unset["title"] == "Everything optional left unset"- for field_name in ("note", "detail", "tags", "counts"):+ for field_name in _MAY_HOLD_NOTHING_FIELDS:assert field_name in unset, f"{field_name} was dropped from the transport dump"assert unset[field_name] is None, f"{field_name} is {unset[field_name]!r}, not the explicit null the wire carries"- # A defaulted field is not nulled when unset — the default is applied — but it can be nulled explicitly.+ # A defaulted field is never nulled: unset, it holds its default.assert unset["status"] == "draft"- assert nulled_default["status"] is None+ assert set_default["status"] == "final"++ def test_the_runtime_class_refuses_a_null_on_the_defaulted_field(self, payload_class: type[StuffContent]):+ """A defaulted field never holds nothing, so no producer can put a `null` for it on the wire."""+ with pytest.raises(ValidationError) as exc_info:+ payload_class.model_validate({"title": "A producer nulling the defaulted field", "status": None})+ assert [error["loc"] for error in exc_info.value.errors()] == [("status",)]- def test_every_non_required_field_projects_null_tolerant(self):- """`.optional()` alone is `T | undefined` in zod, and would reject the payload pinned above."""+ def test_the_projection_states_presence_as_the_runtime_does(self):+ """`.optional()` alone is `T | undefined` in zod and would reject the nulls pinned above; `.nullable()` would admit a refused one."""content = emit_ts_zod(resolve_concepts_from_crate(normalize_crate(_authored_crate(), mthds_version=CRATE_TEST_VERSION)))[0].contentassert ".optional()" not in content- for field_name in _NON_REQUIRED_FIELDS:+ assert ".nullable()" not in content+ for field_name in _MAY_HOLD_NOTHING_FIELDS:# Every field of this crate is short enough to stay on one line, so a missing match means the# field vanished from the projection rather than that it was broken across lines.line = next((line for line in content.splitlines() if line.strip().startswith(f"{field_name}: ")), None)assert line is not None, f"{field_name} has no field line in the emitted schema"- assert ".nullish()" in line or ".nullable()" in line, f"{field_name} is not null-tolerant: {line.strip()}"+ assert ".nullish()" in line, f"{field_name} is not null-tolerant: {line.strip()}"+ assert 'status: z.enum(["draft", "final"]).default("draft"),' in content# The nested concept's own optional field, reached through the reference rather than declared beside it.assert "weight: z.number().nullish()," in content@@ -161,7 +177,9 @@ class TestTsZodWireAgreement:'import { readFileSync } from "node:fs";\n''import { PayloadSchema } from "./types.ts";\n''const wire = JSON.parse(readFileSync("wire.json", "utf-8"));\n'- "process.stdout.write(JSON.stringify(wire.map((one: unknown) => PayloadSchema.parse(one))));\n",+ "const parsed = wire.map((one: unknown) => PayloadSchema.parse(one));\n"+ 'const nulledDefault = PayloadSchema.safeParse({ title: "A nulled default", status: null }).success;\n'+ "process.stdout.write(JSON.stringify({ parsed, nulledDefault }));\n",encoding="utf-8",)@@ -174,10 +192,13 @@ class TestTsZodWireAgreement:)assert run.returncode == 0, f"the emitted schema rejected the runtime's own payload:\n{run.stdout}\n{run.stderr}"- unset, nulled_default = json.loads(run.stdout)+ result = json.loads(run.stdout)+ unset, set_default = result["parsed"]# Nulls survive as nulls: the schema describes the wire, it does not fold `null` into `undefined`# (the binder uses one schema for both parse and serialize, so a transform would desynchronize them).- for field_name in ("note", "detail", "tags", "counts"):+ for field_name in _MAY_HOLD_NOTHING_FIELDS:assert unset[field_name] is None, f"{field_name} came back as {unset[field_name]!r} instead of null"assert unset["status"] == "draft"- assert nulled_default["status"] is None+ assert set_default["status"] == "final"+ # The schema refuses the `null` the runtime class refuses.+ assert result["nulledDefault"] is Falsediff --git a/tests/unit/pipelex/cogt/content_generation/test_schema_to_model.py b/tests/unit/pipelex/cogt/content_generation/test_schema_to_model.pyindex 93e828659..0040a1aa9 100644--- a/tests/unit/pipelex/cogt/content_generation/test_schema_to_model.py+++ b/tests/unit/pipelex/cogt/content_generation/test_schema_to_model.py@@ -5,7 +5,7 @@ import uuidfrom typing import Any, Literal, get_args, get_originimport pytest-from pydantic import BaseModel, Field+from pydantic import BaseModel, Field, ValidationErrorfrom pytest_mock import MockerFixturefrom pipelex.cogt.content_generation.exceptions import UnsafeSchemaError@@ -56,6 +56,14 @@ def _benign_object_schema() -> dict[str, Any]:return {"title": "Innocent", "type": "object", "properties": {"x": {"type": "integer"}}}+class ModelWithPresenceVariants(BaseModel):+ """A defaulted field, a field that may hold nothing, and a required one."""++ note: str = Field(default="fallback", description="A defaulted note")+ remark: str | None = Field(default=None, description="An optional remark")+ title: str = Field(..., description="A required title")++class TestSchemaToModel:def test_literal_choices_field_round_trips_as_literal_not_enum(self) -> None:"""Round-tripping a ``Literal[str-set]`` field through ``make_from_json_schema``@@ -172,6 +180,23 @@ class TestSchemaToModel:assert instance.name == "Alice" # type: ignore[attr-defined]assert instance.age == 30 # type: ignore[attr-defined]+ def test_reconstructed_model_keeps_each_field_nullable_only_where_the_source_is(self) -> None:+ """A defaulted field refuses `null` on the rebuilt class, as on the class it was rebuilt from.++ The rebuilt class is what a model's answer is validated against on a worker, so a `null` it admitted+ would pass there and fail only at the revalidation into the caller's class, too late to ask again.+ """+ schema = ModelWithPresenceVariants.model_json_schema()+ result_class = SchemaToModelFactory.make_from_json_schema(schema, class_name="ModelWithPresenceVariants")++ instance = result_class.model_validate({"title": "Kept"})+ assert instance.note == "fallback" # type: ignore[attr-defined]+ assert instance.remark is None # type: ignore[attr-defined]+ assert result_class.model_validate({"title": "Kept", "remark": None}).remark is None # type: ignore[attr-defined]+ with pytest.raises(ValidationError) as exc_info:+ result_class.model_validate({"title": "Kept", "note": None})+ assert [error["loc"] for error in exc_info.value.errors()] == [("note",)]+def test_nested_model_reconstruction(self) -> None:"""A model with nested BaseModel fields (producing $defs) can be reconstructed."""schema = PersonWithAddress.model_json_schema()diff --git a/tests/unit/pipelex/core/concepts/structure_generation/test_structure_generator.py b/tests/unit/pipelex/core/concepts/structure_generation/test_structure_generator.pyindex 50e429e77..62385c17d 100644--- a/tests/unit/pipelex/core/concepts/structure_generation/test_structure_generator.py+++ b/tests/unit/pipelex/core/concepts/structure_generation/test_structure_generator.py@@ -52,7 +52,7 @@ class TestModel(StructuredContent):name: str = Field(..., description="Name field")age: int | None = Field(default=None, description="Age field")- active: bool | None = Field(default=True, description="Active status")+ active: bool = Field(default=True, description="Active status")'''assert result == expected_code@@ -643,9 +643,9 @@ from typing import Any, Literalclass PersonWithDefaults(StructuredContent):"""Generated PersonWithDefaults class"""- name: str | None = Field(default="Anonymous", description="Person name")- age: int | None = Field(default=0, description="Person age")- active: bool | None = Field(default=True, description="Is active")+ name: str = Field(default="Anonymous", description="Person name")+ age: int = Field(default=0, description="Person age")+ active: bool = Field(default=True, description="Is active")'''assert result == expected_code@@ -830,8 +830,8 @@ class ComplexItem(StructuredContent):name: str = Field(..., description="Display name")tags: list[str] | None = Field(default=None, description="Associated tags")metadata: dict[str, str] | None = Field(default=None, description="Additional metadata")- active: bool | None = Field(default=True, description="Whether item is active")- priority: Literal["low", "medium", "high", "urgent"] | None = Field(default="medium", description="Priority level")+ active: bool = Field(default=True, description="Whether item is active")+ priority: Literal["low", "medium", "high", "urgent"] = Field(default="medium", description="Priority level")'''assert result == expected_codediff --git a/tests/unit/pipelex/core/concepts/structure_generation/test_structure_generator_escaping.py b/tests/unit/pipelex/core/concepts/structure_generation/test_structure_generator_escaping.pyindex f5b2caad7..ee20002d6 100644--- a/tests/unit/pipelex/core/concepts/structure_generation/test_structure_generator_escaping.py+++ b/tests/unit/pipelex/core/concepts/structure_generation/test_structure_generator_escaping.py@@ -371,7 +371,7 @@ from typing import Any, Literalclass WithDefault(StructuredContent):"""Generated WithDefault class"""- field_with_default: str | None = Field(default="Example: \\"quoted text\\"", description="A field with quoted default")+ field_with_default: str = Field(default="Example: \\"quoted text\\"", description="A field with quoted default")'''assert generated_code == expected_code@@ -419,7 +419,7 @@ from typing import Any, Literalclass DefaultPath(StructuredContent):"""Generated DefaultPath class"""- path: str | None = Field(default="C:\\\\Program Files\\\\App", description="Default path")+ path: str = Field(default="C:\\\\Program Files\\\\App", description="Default path")'''assert generated_code == expected_code@@ -670,7 +670,7 @@ class AnimalRecord(StructuredContent):animal_type: str = Field(..., description="The species or type of animal (e.g., \\"fox\\", \\"penguin\\", \\"octopus\\")")owner_name: str | None = Field(default=None, description="The owner\'s full name")- file_path: str | None = Field(default="C:\\\\default\\\\path", description="Location on disk (e.g., C:\\\\Animals\\\\data.txt)")+ file_path: str = Field(default="C:\\\\default\\\\path", description="Location on disk (e.g., C:\\\\Animals\\\\data.txt)")notes: str | None = Field(default=None, description="Notes with \\"quotes\\" and\\nnewlines")'''diff --git a/tests/unit/pipelex/core/concepts/structure_generation/test_structure_generator_presence.py b/tests/unit/pipelex/core/concepts/structure_generation/test_structure_generator_presence.pynew file mode 100644index 000000000..ae938fd7d--- /dev/null+++ b/tests/unit/pipelex/core/concepts/structure_generation/test_structure_generator_presence.py@@ -0,0 +1,109 @@+"""A generated field admits `None` only when it may hold nothing: when it is neither `required` nor defaulted.++That is the standard's rule, and a binding step derives a read's presence from it alone, so a generated class+that admitted `None` anywhere else would hold an absence its readers ruled out.+"""++from typing import Any++import pytest+from pydantic import ValidationError++from pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprint, ConceptStructureBlueprintFieldType+from pipelex.core.concepts.resolved_fields import resolve_field+from pipelex.core.concepts.structure_generation.generator import StructureGenerator+++def _generate(structure_blueprint: dict[str, ConceptStructureBlueprint]) -> tuple[str, Any]:+ return StructureGenerator(local_domain="presence").generate_from_structure_blueprint(+ class_name="PresenceModel", structure_blueprint=structure_blueprint+ )+++class TestStructureGeneratorPresence:+ @pytest.mark.parametrize(+ ("field_blueprint", "may_hold_nothing"),+ [+ (ConceptStructureBlueprint(description="optional", type=ConceptStructureBlueprintFieldType.TEXT), True),+ (ConceptStructureBlueprint(description="required", type=ConceptStructureBlueprintFieldType.TEXT, required=True), False),+ (ConceptStructureBlueprint(description="defaulted", type=ConceptStructureBlueprintFieldType.TEXT, default_value="x"), False),+ (ConceptStructureBlueprint(description="defaulted false", type=ConceptStructureBlueprintFieldType.BOOLEAN, default_value=False), False),+ (ConceptStructureBlueprint(description="defaulted choice", choices=["a", "b"], default_value="a"), False),+ ],+ )+ def test_the_blueprint_and_the_resolved_field_state_the_same_rule(self, field_blueprint: ConceptStructureBlueprint, may_hold_nothing: bool):+ assert field_blueprint.may_hold_nothing is may_hold_nothing+ assert resolve_field(field_name="field", blueprint=field_blueprint).may_hold_nothing is may_hold_nothing++ def test_a_defaulted_field_takes_its_default_and_refuses_null(self):+ source, generated_class = _generate(+ {"note": ConceptStructureBlueprint(description="a note", type=ConceptStructureBlueprintFieldType.TEXT, default_value="fallback")}+ )++ assert ' note: str = Field(default="fallback", description="a note")' in source+ assert generated_class.model_validate({}).note == "fallback"+ assert generated_class.model_validate({"note": "written"}).note == "written"+ with pytest.raises(ValidationError) as exc_info:+ generated_class.model_validate({"note": None})+ assert [error["loc"] for error in exc_info.value.errors()] == [("note",)]++ def test_a_defaulted_falsy_value_is_still_a_default(self):+ """`default_value = false` is a default like any other, so the field is not nullable either."""+ source, generated_class = _generate(+ {"enabled": ConceptStructureBlueprint(description="on", type=ConceptStructureBlueprintFieldType.BOOLEAN, default_value=False)}+ )++ assert " enabled: bool = Field(default=False" in source+ with pytest.raises(ValidationError):+ generated_class.model_validate({"enabled": None})++ def test_a_field_that_may_hold_nothing_keeps_an_explicit_null(self):+ source, generated_class = _generate({"note": ConceptStructureBlueprint(description="a note", type=ConceptStructureBlueprintFieldType.TEXT)})++ assert " note: str | None = Field(default=None" in source+ assert generated_class.model_validate({"note": None}).note is None+ assert generated_class.model_validate({}).note is None++ @pytest.mark.parametrize("value", ["text", 0, False, [1, "two"], {"key": "value"}, {}])+ def test_a_required_anything_field_holds_any_value_but_null(self, value: Any):+ source, generated_class = _generate(+ {+ "payload": ConceptStructureBlueprint(+ description="any value", type=ConceptStructureBlueprintFieldType.CONCEPT, concept_ref="native.Anything", required=True+ )+ }+ )++ assert "from pipelex.core.stuffs.non_null_any import NonNullAny" in source+ assert " payload: NonNullAny = Field(..., description=" in source+ assert generated_class.model_validate({"payload": value}).payload == value+ with pytest.raises(ValidationError) as exc_info:+ generated_class.model_validate({"payload": None})+ assert [(error["loc"], error["type"]) for error in exc_info.value.errors()] == [(("payload",), "null_value")]++ def test_a_required_anything_field_keeps_an_open_schema(self):+ """The schema is what a model is asked for, and the strict modes that would read `not` refuse it."""+ _, generated_class = _generate(+ {+ "payload": ConceptStructureBlueprint(+ description="any value", type=ConceptStructureBlueprintFieldType.CONCEPT, concept_ref="native.Anything", required=True+ )+ }+ )++ schema = generated_class.model_json_schema()+ assert schema["properties"]["payload"] == {"description": "any value", "title": "Payload"}+ assert schema["required"] == ["payload"]++ def test_an_optional_anything_field_may_hold_nothing(self):+ source, generated_class = _generate(+ {+ "payload": ConceptStructureBlueprint(+ description="any value", type=ConceptStructureBlueprintFieldType.CONCEPT, concept_ref="native.Anything"+ )+ }+ )++ assert " payload: Any | None = Field(default=None" in source+ assert "NonNullAny" not in source+ assert generated_class.model_validate({"payload": None}).payload is Nonediff --git a/tests/unit/pipelex/pipe_controllers/binding/test_library_concept_walk_resolver.py b/tests/unit/pipelex/pipe_controllers/binding/test_library_concept_walk_resolver.pyindex 30089e0e1..9d6bb14c3 100644--- a/tests/unit/pipelex/pipe_controllers/binding/test_library_concept_walk_resolver.py+++ b/tests/unit/pipelex/pipe_controllers/binding/test_library_concept_walk_resolver.py@@ -10,6 +10,7 @@ from pipelex.core.concepts.concept_blueprint import ConceptBlueprintfrom pipelex.core.concepts.concept_factory import ConceptFactoryfrom pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprint, ConceptStructureBlueprintFieldTypefrom pipelex.core.concepts.exceptions import ConceptStructureClassNotFoundError+from pipelex.core.stuffs.non_null_any import NonNullAnyfrom pipelex.core.stuffs.structured_content import StructuredContentfrom pipelex.core.stuffs.stuff_content import StuffContentfrom pipelex.libraries.concept.concept_library_abstract import ConceptLibraryAbstract@@ -58,6 +59,8 @@ class _VesselRecord(StructuredContent):pilot: str | None = "the harbour pilot"escort: str | None = Field(default_factory=_default_escort)draught: float = 4.5+ manifest: NonNullAny+ remarks: Anydef _concept(*, code: str, structure_class_name: str, domain_code: str = "harbour", **kwargs: Any) -> Concept:@@ -190,6 +193,8 @@ class TestLibraryConceptWalkResolver:pytest.param("pilot", BindingValueKind.TEXT, None, False, True, id="nullable-with-a-default"),pytest.param("escort", BindingValueKind.TEXT, None, False, True, id="nullable-with-a-default-factory"),pytest.param("draught", BindingValueKind.NUMBER, None, False, False, id="not-nullable-with-a-default"),+ pytest.param("manifest", BindingValueKind.UNDERIVABLE, None, False, False, id="required-non-null-any"),+ pytest.param("remarks", BindingValueKind.UNDERIVABLE, None, False, True, id="required-but-any"),],)def test_a_class_backed_concept_is_walked_through_its_class_fields(diff --git a/tests/unit/pipelex/pipe_operators/pipe_compose/test_composer_source_holding_nothing.py b/tests/unit/pipelex/pipe_operators/pipe_compose/test_composer_source_holding_nothing.pynew file mode 100644index 000000000..8d3dfdab5--- /dev/null+++ b/tests/unit/pipelex/pipe_operators/pipe_compose/test_composer_source_holding_nothing.py@@ -0,0 +1,128 @@+from typing import Callable, Literal++import pytest+from pydantic import Field++from pipelex.core.concepts.native.concept_native import NativeConceptCode+from pipelex.core.memory.working_memory import WorkingMemory+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory+from pipelex.core.stuffs.structured_content import StructuredContent+from pipelex.core.stuffs.stuff_factory import StuffFactory+from pipelex.interpreter_hub import get_native_concept+from pipelex.pipe_operators.compose.construct_blueprint import ConstructBlueprint+from pipelex.pipe_operators.compose.exceptions import StructuredContentComposerValidationError+from pipelex.pipe_operators.compose.structured_content_composer import StructuredContentComposer+from pipelex.tools.templating.templating_style import TagStyle, TemplatingStyle++_TEMPLATING_STYLE = TemplatingStyle(tag_style=TagStyle.XML)+++class SourceWithOptionalNote(StructuredContent):+ """A source whose note may hold nothing."""++ note: str | None = Field(default=None, description="An optional note")+++class TargetWithPresenceVariants(StructuredContent):+ """A target fed the same source field three ways: defaulted, may hold nothing, required."""++ defaulted_note: str = Field(default="fallback", description="A defaulted note")+ optional_note: str | None = Field(default=None, description="An optional note")+++class TargetWithRequiredNote(StructuredContent):+ """A target whose note is required."""++ required_note: str = Field(description="A required note")+++class TargetWithNullableNotes(StructuredContent):+ """A Python-declared target whose notes admit `None`, one required and one defaulted to a value."""++ required_nullable_note: str | None = Field(description="A required note that admits None")+ defaulted_nullable_note: str | None = Field(default="fallback", description="A defaulted note that admits None")+++class TargetWithNullableLiteral(StructuredContent):+ """A Python-declared target whose note is a `Literal` listing `None`, defaulted to a value."""++ literal_note: Literal["fallback", None] = Field(default="fallback", description="A note whose Literal admits None") # ruff: ignore[redundant-none-literal] — a Literal listing None is the test subject++[email protected](loop_scope="class")+class TestStructuredContentComposerSourceHoldingNothing:+ """A path reaching a field that holds nothing feeds the target field `None`, unless it refuses `None` but has a default.++ A generated defaulted field refuses `None`, so nulling it would fail the composition; left unset, it takes its+ default. A field whose annotation admits `None` keeps it, required or defaulted, and a required one refusing+ it is refused.+ """++ @pytest.fixture+ def working_memory_with_empty_note(self, load_empty_library: Callable[[], None]) -> WorkingMemory:+ load_empty_library()+ return WorkingMemoryFactory.make_from_single_stuff(+ stuff=StuffFactory.make_stuff(+ concept=get_native_concept(NativeConceptCode.TEXT),+ content=SourceWithOptionalNote(),+ name="source",+ ),+ )++ async def test_a_defaulted_target_takes_its_default_and_an_optional_one_holds_nothing(self, working_memory_with_empty_note: WorkingMemory):+ blueprint = ConstructBlueprint.make_from_raw({"defaulted_note": {"from": "source.note"}, "optional_note": {"from": "source.note"}})++ composer = StructuredContentComposer(+ templating_style=_TEMPLATING_STYLE,+ construct_blueprint=blueprint,+ working_memory=working_memory_with_empty_note,+ output_class=TargetWithPresenceVariants,+ )+ result = await composer.compose()++ assert isinstance(result, TargetWithPresenceVariants)+ assert result.defaulted_note == "fallback"+ assert result.optional_note is None++ async def test_a_required_target_is_refused(self, working_memory_with_empty_note: WorkingMemory):+ blueprint = ConstructBlueprint.make_from_raw({"required_note": {"from": "source.note"}})++ composer = StructuredContentComposer(+ templating_style=_TEMPLATING_STYLE,+ construct_blueprint=blueprint,+ working_memory=working_memory_with_empty_note,+ output_class=TargetWithRequiredNote,+ )+ with pytest.raises(StructuredContentComposerValidationError, match="required_note"):+ await composer.compose()++ async def test_a_target_that_admits_none_keeps_it(self, working_memory_with_empty_note: WorkingMemory):+ blueprint = ConstructBlueprint.make_from_raw(+ {"required_nullable_note": {"from": "source.note"}, "defaulted_nullable_note": {"from": "source.note"}}+ )++ composer = StructuredContentComposer(+ templating_style=_TEMPLATING_STYLE,+ construct_blueprint=blueprint,+ working_memory=working_memory_with_empty_note,+ output_class=TargetWithNullableNotes,+ )+ result = await composer.compose()++ assert isinstance(result, TargetWithNullableNotes)+ assert result.required_nullable_note is None+ assert result.defaulted_nullable_note is None++ async def test_a_literal_target_listing_none_keeps_it(self, working_memory_with_empty_note: WorkingMemory):+ blueprint = ConstructBlueprint.make_from_raw({"literal_note": {"from": "source.note"}})++ composer = StructuredContentComposer(+ templating_style=_TEMPLATING_STYLE,+ construct_blueprint=blueprint,+ working_memory=working_memory_with_empty_note,+ output_class=TargetWithNullableLiteral,+ )+ result = await composer.compose()++ assert isinstance(result, TargetWithNullableLiteral)+ assert result.literal_note is Nonediff --git a/tests/unit/pipelex/tools/tabular/test_csv_codec_blank_cells.py b/tests/unit/pipelex/tools/tabular/test_csv_codec_blank_cells.pynew file mode 100644index 000000000..392d222cc--- /dev/null+++ b/tests/unit/pipelex/tools/tabular/test_csv_codec_blank_cells.py@@ -0,0 +1,63 @@+"""An empty CSV cell is "no value": a defaulted field that refuses `None` takes its default, as for an omitted column.++A structure field with a `default_value` never holds nothing, so its generated class refuses `None`, and a blank+cell under it must not become one.+"""++from pathlib import Path+from typing import Any++import pytest++from pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprint, ConceptStructureBlueprintFieldType+from pipelex.core.concepts.structure_generation.generator import StructureGenerator+from pipelex.core.stuffs.structured_content import StructuredContent+from pipelex.tools.tabular.csv_codec import list_content_from_csv+from pipelex.tools.tabular.exceptions import CsvCoercionError+++class PresenceRow(StructuredContent):+ name: str+ note: str = "fallback"+ count: int = 0+ nickname: str | None = "anon"+++def _write_csv(directory: Path, content: str) -> Path:+ path = directory / "data.csv"+ path.write_text(content, encoding="utf-8")+ return path+++class TestCsvCodecBlankCells:+ def test_a_blank_cell_under_a_defaulted_field_refusing_none_takes_its_default(self, tmp_path: Path):+ path = _write_csv(tmp_path, "name,note,count,nickname\nalice,,,\nbob,hello,3,bobby\n")++ rows = list_content_from_csv(path, row_model=PresenceRow).items++ assert [(row.name, row.note, row.count, row.nickname) for row in rows] == [+ ("alice", "fallback", 0, None),+ ("bob", "hello", 3, "bobby"),+ ]++ def test_a_blank_cell_under_a_required_field_is_refused(self, tmp_path: Path):+ path = _write_csv(tmp_path, "name,note\n,hello\n")++ with pytest.raises(CsvCoercionError, match="name"):+ list_content_from_csv(path, row_model=PresenceRow)++ def test_a_blank_cell_under_a_generated_defaulted_field_takes_its_default(self, tmp_path: Path):+ row_class: Any+ _, row_class = StructureGenerator(local_domain="csv_blank").generate_from_structure_blueprint(+ class_name="GeneratedRow",+ structure_blueprint={+ "name": ConceptStructureBlueprint(description="a name", type=ConceptStructureBlueprintFieldType.TEXT, required=True),+ "note": ConceptStructureBlueprint(description="a note", type=ConceptStructureBlueprintFieldType.TEXT, default_value="fallback"),+ "remark": ConceptStructureBlueprint(description="a remark", type=ConceptStructureBlueprintFieldType.TEXT),+ },+ )+ path = _write_csv(tmp_path, "name,note,remark\nalice,,\n")++ rows = list_content_from_csv(path, row_model=row_class).items++ assert [row.model_dump() for row in rows] == [{"name": "alice", "note": "fallback", "remark": None}]diff --git a/tests/unit/pipelex/tools/test_annotation_admits_none.py b/tests/unit/pipelex/tools/test_annotation_admits_none.pynew file mode 100644index 000000000..06db40167--- /dev/null+++ b/tests/unit/pipelex/tools/test_annotation_admits_none.py@@ -0,0 +1,46 @@+from typing import Annotated, Any, Literal, Optional, Union++import pytest++from pipelex.core.stuffs.text_content import TextContent+from pipelex.tools.typing.annotation_utils import annotation_admits_none+++class TestAnnotationAdmitsNone:+ """annotation_admits_none must say whether pydantic accepts `None` for a field so annotated, whatever spelling admits it."""++ @pytest.mark.parametrize(+ "annotation",+ [+ pytest.param(None, id="none_annotation"),+ pytest.param(type(None), id="none_type"),+ pytest.param(Any, id="any"),+ pytest.param(object, id="object"),+ pytest.param(Optional[str], id="typing_optional_str"), # ruff: ignore[non-pep604-annotation-optional] — the typing.Optional spelling is the test subject+ pytest.param(Union[str, None], id="typing_union_str_none"),+ pytest.param(str | None, id="pep604_str_none"),+ pytest.param(str | int | None, id="pep604_union_multi_arm_with_none"),+ pytest.param(TextContent | None, id="pep604_text_content_none"),+ pytest.param(Literal["fallback", None], id="literal_listing_none"), # ruff: ignore[redundant-none-literal] — a Literal listing None is the test subject+ pytest.param(Literal["fallback"] | None, id="pep604_literal_none"),+ pytest.param(str | Any, id="union_with_any"),+ pytest.param(Annotated[str | None, "note"] | int, id="union_with_annotated_optional_arm"),+ pytest.param(Annotated[str | None, "note"], id="annotated_optional"),+ ],+ )+ def test_admits_none(self, annotation: Any):+ assert annotation_admits_none(annotation=annotation)++ @pytest.mark.parametrize(+ "annotation",+ [+ pytest.param(str, id="bare_class"),+ pytest.param(TextContent, id="content_class"),+ pytest.param(list[str | None], id="list_of_optional_items"),+ pytest.param(str | int, id="pep604_union_multi_arm"),+ pytest.param(Literal["fallback", "other"], id="literal_without_none"),+ pytest.param(Annotated[str, "note"] | int, id="union_with_annotated_required_arm"),+ ],+ )+ def test_refuses_none(self, annotation: Any):+ assert not annotation_admits_none(annotation=annotation)