Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Migrator Phase 2: the engine, the gates that prove a ledger, and the first real migration (#1112)
Louis Choquel0aa8913f2f46
diff --git a/.gitignore b/.gitignoreindex 489f50b20..eb14b7eb8 100644--- a/.gitignore+++ b/.gitignore@@ -15,6 +15,7 @@ site/# Unit test / coverage reportshtmlcov/.pytest_cache/+.hypothesis/.coveragecoverage.xmldiff --git a/AGENTS.md b/AGENTS.mdindex 568c783a3..f1bd42f4c 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -9,7 +9,7 @@```bashmake agent-check# If the current system doesn't have the `make` command,- # lookup the "agent-check" target in the Makefile and run the commands one by one (targets fix-unused-imports fix-keyword-only format lint pyright mypy check-keyword-only check-hub-layering drift-check)+ # lookup the "agent-check" target in the Makefile and run the commands one by one (targets fix-unused-imports fix-keyword-only format lint pyright mypy check-ledger check-keyword-only check-hub-layering drift-check)```This runs multiple code quality tools:@@ -17,6 +17,7 @@- Ruff: Fix unused imports, lint, format- Mypy: Static type checker- plxt: Format and lint TOML, MTHDS, and PLX files+ - Migration ledgers: every checked-in ledger must be legal and replay harmlessly over its reference documents (`make check-ledger`, alias `cl`)- Drift contracts: open code↔docs review obligations fail the gate (note the digest reads the git index, so stage your changes for the check to see them)Always fix any issues reported by these tools before proceeding.diff --git a/CHANGELOG.md b/CHANGELOG.mdindex e75581b0c..69e716f97 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -4,12 +4,19 @@### Added-- **Configuration surfaces now have a migration ledger and a gate that forces one to be written**: Each of the three configuration surfaces — `pipelex.toml` and its tiers, `telemetry.toml`, `pipelex_service.toml` — carries a checked-in ledger at `pipelex/migration/ledgers/<surface-id>.toml` recording, as data, every shape change it has ever undergone, plus a golden chain (`fingerprint@N`, `defaults@N`) that pins what the schema looked like at each version. All three start at schema version 1 with no entries: this release cuts the baseline, it migrates nothing. The new `make check-migration-schemas` (alias `cmig`, in `make check`) recomputes each surface's fingerprint and refuses a change that would break a user's file without recording how to repair it — a removed path or a lost enumerated spelling demands a version bump and an operation accounting for it, a destination the new schema does not know is refused by name, an entry that removes a path the schema still has is refused as over-deletion, and a `safe` value remap whose old spelling is still legal is refused because it would rewrite a deliberate choice. Its counterpart `make up-migration-schemas` (alias `umig`, in `make up`) regenerates the goldens; the regeneration diff is the review artifact. One standing invariant is checked whether or not anything changed: every required path must have a value in its surface's defaults layer, because that is what makes an added key absorbable rather than breaking. The contract the whole thing implements is `docs/migration-ledger.md`.+- **A schema change that narrows what a value may *be* is now caught, not waved through as additive**: A change can keep every path and every enumerated spelling and still break a user's file — `int` becomes `str`, a free string becomes an enum, a bound moves from `ge=1` to `ge=5`. Read as a path diff those look like additions, so the coverage gate used to ask for a golden regeneration, demand no version bump and no entry, and let the next boot reject a valid file with a green gate behind it. It now splits its verdict by *direction*: a widened type or a relaxed bound still costs only a regeneration, while a narrowing demands a bump and an entry carrying a `remap_value` for each narrowed path — or marked `unsafe`, which is the only remedy a tightened numeric bound has, since no structural operation can repair a value. The same rule applies to a stored entry, compared by origin so a narrowing hidden behind a rename is still caught, and to a `pre_history` entry, so the flag is not a way past it. To see a bound at all the fingerprint now records one: a **closed whitelist** over `gt`, `ge`, `lt`, `le`, `min_length`, `max_length` and `multiple_of`, read from `annotated_types` (now a declared dependency) and from the field metadata pydantic folds a top-level `Field(ge=1)` into, with anything unrecognized dropped rather than serialized — `pattern` excluded permanently, because regex containment is not decidable and every pattern edit would read as a tightening. One correction rides along: an enumerated type relaxed into a free string no longer reads as the loss of every spelling it had. The contract states in the same change what the gate structurally cannot see — narrowing expressed in a validator, which no projection of the annotations can reach.+- **The telemetry configuration can now be migrated instead of thrown away, and the escape hatch that would have let it ship unverified is closed**: `telemetry.toml` carries the package's first migration entry, `telemetry-config@2`, which lifts a flat first-generation file — `telemetry_mode`, `host`, `project_api_key` and the rest at the document root — into today's `[custom_posthog]` section, keeping every value the user chose; four settings the current shape has no home for are dropped, and the entry says which and why. Because that change predates the first fingerprint, no diff describes it, and such an entry is marked `pre_history` and exempted from the coverage gate's accounting — so the verification that replaces it lands in the same change and neither ships without the other. A pre-history entry must declare the paths it removes (an empty declaration is refused when the ledger is parsed), none of those paths may appear in any fingerprint at or below its own version, the fingerprint pair it sits between must show no removal at all, and every operation must act inside the declaration. It also ships a hand-authored `[email protected]` beside the golden chain, which the transform check migrates and holds against the current shape and the current model — the same three claims every other entry answers. One comparator refinement came with it: a created path is now checked against `defaults@N` **or** `fingerprint@N`, because an optional key defaulting to `None` is a legal destination that no TOML document can carry.+- **Configuration surfaces now have a migration ledger and a gate that forces one to be written**: Each of the three configuration surfaces — `pipelex.toml` and its tiers, `telemetry.toml`, `pipelex_service.toml` — carries a checked-in ledger at `pipelex/migration/ledgers/<surface-id>.toml` recording, as data, every shape change it has ever undergone, plus a golden chain (`fingerprint@N`, `defaults@N`) that pins what the schema looked like at each version. All three start at schema version 1 with no entries: this release cuts the baseline, it migrates nothing. The new `make check-migration-schemas` (alias `cmig`, in `make check`) recomputes each surface's fingerprint and refuses a change that would break a user's file without recording how to repair it — a removed path or a lost enumerated spelling demands a version bump and an operation accounting for it, a destination the new schema does not know is refused by name, and an entry that removes a path the schema still has is refused as over-deletion. Its counterpart `make up-migration-schemas` (alias `umig`, in `make up`) regenerates the goldens; the regeneration diff is the review artifact. One standing invariant is checked whether or not anything changed: every required path must have a value in its surface's defaults layer, because that is what makes an added key absorbable rather than breaking. The contract the whole thing implements is `docs/migration-ledger.md`.- **The migration operation vocabulary**: Fix operations became a discriminated union on `kind`, published as two subsets — every kind for the `.mthds` fix path, and the structural kinds only for ledgers, so an operation that would write a value into a user's file fails when the ledger is parsed rather than at review time. Two new kinds land with it: `move_key`, which relocates a key across tables (the whole subtree travels when the key is table-valued, and a missing destination parent is created), and `remap_value`, which rewrites a value against an explicit old-to-new mapping. A `*` segment addresses every entry of an open mapping, expanded over the keys the file actually holds. A new `CONFLICT` outcome separates "this cannot be done without choosing on your behalf" — a rename or move onto an occupied destination, or an `ensure_table` over a key holding something that is not a table — from the benign `SKIPPED` it used to hide inside; a conflicting operation writes nothing at all.- **`[meta] schema_version` is reserved in configuration files**: Every configuration-surface reader now tolerates and strips it. Nothing writes it — the key exists so a later release can stamp a version into a file without that being a breaking change today.+- **`make check-ledger` (alias `cl`)**: The second migration gate, and the first one that joins `make agent-check` as well as `make check`. It asks what a ledger entry is allowed to *say* and whether replaying it is harmless, where `check-migration-schemas` asks whether a schema change was accounted for. An operation may only act on material some schema version removed — one addressing a path the schema still has would rewrite a perfectly valid file on every run — and it may never name a concrete key beneath an open mapping, whose keys are the user's and unbounded, while the `*` segment that addresses every entry of one is legal exactly there and nowhere else. A `safe` value remap whose old spelling is still legal, or whose target is a free string where staleness cannot be proven, is refused because it would rewrite a deliberate choice; a remap to a spelling the new schema rejects is refused because every migrated file would then fail to load with the tool reporting success. A retired path, or a remapped-away spelling, may never come back. Finally, replaying the whole ledger over each of the surface's two reference documents — the complete packaged configuration and the sparse kit starter template, deliberately different shapes — must apply nothing, report nothing, and hand back the very bytes it was given. It reads checked-in files only and fingerprints no live model, which is what makes it safe to run in the loop agents run constantly: every failure names a file the author wrote.+- **The migration engine**: A surface's ledger can now be replayed over a real file. The engine takes a document's text and returns the text it should now have, plus a per-file plan naming the entries that applied, the entries it would not apply, and why. Two reporting rules keep a plan from being more optimistic than the file: an entry is reported once — a conflicting operation routes the whole entry into the blocked list, carrying whichever of its operations did land — and an `unsafe` entry is rehearsed against the document first, so it stays silent on a file that does not need it rather than warning at every boot forever. Writing is transactional per file and always takes a backup: exactly one per file, named `<file>.bak.<UTC stamp>`, inheriting the source file's mode rather than the umask, written before the replacement and pruned only after it commits. A file that is unparseable, unwritable or changed during the run is reported as blocked while every sibling is migrated normally — unlike the `.mthds` fix loop, whose files only make sense together and which still commits a round all-or-nothing; both now share one set of transaction primitives. Nothing calls this from the command line yet.+- **The transform goldens**: `make check-migration-schemas` now also *performs* each migration it is asked to believe in. For every ledger entry it applies the entry's operations to the frozen reference document of the version before it and compares the result with the frozen document of the version after: every path the migration creates must be one the new document has, every path the two documents share must survive, and the last link's output — read the way a user's file is really read, beneath the current defaults layer — must be accepted by the current model. That closes the one defect the other two checks pass in silence: a rename whose destination is misspelled is accounted for by coverage and skipped by convergence, and then migrates every file to a key the schema rejects while the tool reports success. The comparison is deliberately one-directional in each claim, so a bump that also adds keys, edits comments, flips defaults or ships one more entry in a packaged deck stays green.+- **Replay neutrality is now a property rather than an assertion**: The guarantee a user's machine depends on — replaying a surface's whole ledger over a file already valid at the current schema changes not one byte — is checked over generated documents, not only over the two reference documents. A reference document carries exactly one value per key, so it cannot see an operation that misbehaves on the *other* legal spelling of an enumerated field; a sampler that reaches those spellings can. The sampler proposes each mutation from the surface's fingerprint and lets the models decide whether it lands, because schema membership is settled by validators no type projection can see: a legal enum member can still be rejected for what it requires elsewhere in the document, and a mapping typed with arbitrary string keys can still refuse a key outside a fixed set. Idempotence — replaying twice equals replaying once — and prefix coherence — a file caught halfway lands where a replay from zero lands — are checked alongside it, and each generator carries its own vacuity check, since every one of these properties is trivially true over documents nothing acts on. `hypothesis` joins the development dependencies.### Changed+- **The fix applier moves and renames a table whole even when a descendant is written in several places**: A `[a.b]` … `[a.c]` … `[a.b.d]` layout stores `a.b` as several chunks under one key, and tomlkit's own header-cache invalidation walks the merged dict facade — one chunk per key — so renaming or moving `a` used to leave the later chunks rendering under the old name: the file came out split between the old name and the new, with an `applied` verdict. The applier now refreshes every chunk beneath a renamed or moved table, so the file re-parses to the same value under the new path, in the original order. This reaches every caller of the applier, the `.mthds` fix loop included.- **Templating style is now an authoring decision (Breaking)**: How a pipe's inputs are tagged into a prompt is declared on the pipe — `templating_style` on `PipeLLM`, either a bare tag-style string (`"xml"`, `"ticks"`, `"square_brackets"`, `"no_tag"`) or a full `{ tag_style, text_format }` table — with a single runtime default for everything that declares nothing (`[pipelex.templating_config].default_templating_style`, shipped as `xml`). It is no longer derived from model metadata, so the same method renders the same prompt shape on every model. **Every prompt rendered for an OpenAI-family model changes shape, from back-tick fences to XML tags.** `PipelexKernel.llm_text` and `llm_object` take the same optional argument.- **Compose, image-generation, search and structuring prompts follow the same default (Breaking)**: These paths previously rendered style-less, which the Jinja2 filters silently absorbed as triple-backticks and plain text. They now resolve the same runtime default as the LLM path. Any template of yours that uses `| tag` — including `PipeCompose` templates and construct-mode template fields — changes shape accordingly, whichever model family you run on.- **Templating filters are strict**: The `tag`, `format` and `with_images` Jinja2 filters no longer fall back to a built-in style when the rendering context carries none; a missing tag style or text format raises `Jinja2ContextError`. Every prompt-rendering entry point now supplies a resolved style, so a missing one is a bug rather than a shape nobody chose. An explicit filter argument (`| format("markdown")`) is unaffected, and an unknown format name is now reported as a template error instead of a bare `ValueError`.diff --git a/CLAUDE.md b/CLAUDE.mdindex a47ff0dbf..354ffb36c 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -9,7 +9,7 @@```bashmake agent-check# If the current system doesn't have the `make` command,- # lookup the "agent-check" target in the Makefile and run the commands one by one (targets fix-unused-imports fix-keyword-only format lint pyright mypy check-keyword-only check-hub-layering drift-check)+ # lookup the "agent-check" target in the Makefile and run the commands one by one (targets fix-unused-imports fix-keyword-only format lint pyright mypy check-ledger check-keyword-only check-hub-layering drift-check)```This runs multiple code quality tools:@@ -17,6 +17,7 @@- Ruff: Fix unused imports, lint, format- Mypy: Static type checker- plxt: Format and lint TOML, MTHDS, and PLX files+ - Migration ledgers: every checked-in ledger must be legal and replay harmlessly over its reference documents (`make check-ledger`, alias `cl`)- Drift contracts: open code↔docs review obligations fail the gate (note the digest reads the git index, so stage your changes for the check to see them; resolve with the drift workflow below)Always fix any issues reported by these tools before proceeding.diff --git a/Makefile b/Makefileindex d55d4e4ac..6a5828fd8 100644--- a/Makefile+++ b/Makefile@@ -102,6 +102,8 @@ make generate-mthds-schema - Generate JSON Schema for .mthds filesmake gms - Shorthand -> generate-mthds-schemamake check-mthds-schema - Check MTHDS JSON Schema is up-to-datemake cms - Shorthand -> check-mthds-schema+make check-ledger - Check migration ledgers are legal and replay harmlessly+make cl - Shorthand -> check-ledgermake check-migration-schemas - Check configuration surfaces have accounted for their schema changesmake cmig - Shorthand -> check-migration-schemasmake up-migration-schemas - Regenerate the migration fingerprint and defaults goldens@@ -215,7 +217,7 @@ export HELPli check-unused-imports fix-unused-imports check-TODOs check-uv \docs docs-check docs-serve-versioned docs-list docs-deploy docs-deploy-stable docs-deploy-specific-version docs-delete \generate-mthds-schema generate-mthds-schema-quiet gms check-mthds-schema cms \- check-migration-schemas cmig up-migration-schemas up-migration-schemas-quiet umig \+ check-ledger cl check-migration-schemas cmig up-migration-schemas up-migration-schemas-quiet umig \generate-error-pages generate-error-pages-quiet gep \generate-error-identity generate-error-identity-quiet gei \update-gateway-models update-gateway-models-quiet ugm check-gateway-models cgm up \@@ -414,6 +416,13 @@ check-mthds-schema: envcms: check-mthds-schema@echo "> done: cms = check-mthds-schema"+check-ledger: env+ $(call PRINT_TITLE,"Checking migration ledgers are legal and converge")+ $(VENV_PIPELEX_DEV) check-ledger --quiet++cl: check-ledger+ @echo "> done: cl = check-ledger"+check-migration-schemas: env$(call PRINT_TITLE,"Checking configuration surfaces have accounted for their schema changes")$(VENV_PIPELEX_DEV) check-migration-schemas --quiet@@ -1181,10 +1190,10 @@ cc: cleanderived regenerate-test-models-quiet generate-mthds-schema-quiet updateup: generate-mthds-schema-quiet update-gateway-models-quiet up-kit-configs up-migration-schemas-quiet rules@echo "> done: up = generate-mthds-schema update-gateway-models up-kit-configs up-migration-schemas rules"-check: cleanderived regenerate-test-models-quiet generate-mthds-schema-quiet update-gateway-models-quiet check-unused-imports check-config-sync check-rules check-urls check-gateway-models check-mthds-schema check-migration-schemas check-keyword-only check-hub-layering drift-check format lint pyright mypy pylint+check: cleanderived regenerate-test-models-quiet generate-mthds-schema-quiet update-gateway-models-quiet check-unused-imports check-config-sync check-rules check-urls check-gateway-models check-mthds-schema check-ledger check-migration-schemas check-keyword-only check-hub-layering drift-check format lint pyright mypy pylint@echo "> done: check"-agent-check: fix-unused-imports fix-keyword-only format lint pyright mypy check-keyword-only check-hub-layering drift-check+agent-check: fix-unused-imports fix-keyword-only format lint pyright mypy check-ledger check-keyword-only check-hub-layering drift-check@echo "> done: agent-check"v: validatediff --git a/docs/migration-ledger.md b/docs/migration-ledger.mdindex 9b2d078d1..b4db03dda 100644--- a/docs/migration-ledger.md+++ b/docs/migration-ledger.md@@ -4,7 +4,7 @@ Pipelex configuration files live on users' machines and in users' repositories,This page is the contract: what a ledger may contain, what the engine may do with it, and what is guaranteed to a user whose files are migrated. It is normative. Everything asserted here has, or will have, a test behind it, and where a rule exists to prevent a specific failure the failure is named — a rule whose reason is forgotten is a rule that gets relaxed.-> **Status.** This specification was written before its implementation, deliberately: the guarantees below are what the engine is built against. The data format, the operation vocabulary and the checks land first; `pipelex migrate` and boot tolerance land last. Of the checks, the coverage gate (`make check-migration-schemas`) and its regenerator (`make up-migration-schemas`) exist today; `make check-ledger`, the transform-golden verifier and pre-history entries arrive with the engine. Until the commands exist, this page is built but kept out of the documentation navigation.+> **Status.** This specification was written before its implementation, deliberately: the guarantees below are what the engine is built against. The data format, the operation vocabulary and the checks land first; `pipelex migrate` and boot tolerance land last. Every check described here exists today: the coverage gate (`make check-migration-schemas`), its regenerator (`make up-migration-schemas`), the ledger check (`make check-ledger`), the transform-golden verifier inside the coverage gate, the neutrality property suite, and the verification of a [pre-history entry](#pre-history-entries). The ledgers themselves are nearly empty — `telemetry-config` carries the one entry the package ships, and the other two surfaces sit at schema version 1 — so over those two the neutrality property still exercises its sampler rather than the theorem. Until the commands exist, this page is built but kept out of the documentation navigation.## What migrates, and what does not@@ -28,10 +28,12 @@ File names are relative to a configuration directory, and every surface spans exThe tier set is **open**. Environment and run-mode names are dynamic, so the tier filenames of `pipelex-config` cannot be enumerated in advance — `pipelex_local.toml`, `pipelex_{environment}.toml`, `pipelex_{run_mode}.toml`, `pipelex_override.toml` and `pipelex_temporary_override.toml` are a description of today, not a closed list. That is why the registry matches tiers with a glob, and the glob is what makes the resolution rule necessary:-> **Exact filenames claim before globs, across all surfaces.** A file matched by any surface's exact pattern belongs to that surface and is excluded from every surface's glob. A file claimed by two globs, or by two exact names, is a registry error and is rejected when the registry loads.+> **Exact filenames claim before globs, across all surfaces.** A file matched by any surface's exact pattern belongs to that surface and is excluded from every surface's glob. A file claimed by two globs, or by two exact names, is a registry error.Without the rule, `pipelex_service.toml` is both the base file of one surface and a match for another's `pipelex_*.toml`, and which ledger runs over it becomes an accident of iteration order.+The registry error is raised at the two moments it is decidable, and no earlier. Two surfaces sharing an id, a base file or a *literal* glob string are refused when the registry loads. Two surfaces whose globs merely *could* overlap are not: whether two glob languages intersect is not cheaply decidable, and a registry has no files to look at. That one is refused the moment a real file proves it — resolution stops by name, on the file, rather than picking whichever surface came first.+Each surface also declares its **defaults-layer kind**, because the checks need to know where a current-schema default value comes from: a packaged TOML document merged beneath the user's files, or model-level field defaults. The distinction matters twice — for synthesizing the complete reference document of a surface that has no packaged file, and for refusing an added path that has no default anywhere. A model-defaults surface whose model cannot be built with nothing set has no defaults layer at all, and says so when its reference document is asked for rather than surfacing later as a missing value.The registry and each ledger's `[surface]` block both describe the same thing, and both are read as truth — the ledger's when a migration walks a directory, the registry's when a gate fingerprints a model. A disagreement between them is therefore checked, not trusted: it would otherwise wait silently for the day the two readers meet.@@ -94,9 +96,13 @@ With both premises every operation in a full replay over a current-valid file skThe theorem is checked anyway, twice, because proofs rot where code does not:- **The convergence witness.** A full replay over each of a surface's reference documents — the complete defaults document and the sparse kit template — must produce zero applied operations. Both are at the current schema by construction, so there is no fixture to maintain. Cheap and exact; it runs in `agent-check`.-- **The neutrality property test.** Generated documents that are valid at the current schema — mutations of the packaged defaults within the schema: other enum members, deleted optional keys, altered scalars, tier-shaped sparse subsets — must replay to byte-identical output. This is the executable form of the theorem over the domain the theorem actually claims. It carries a **vacuity meta-test**: every generated document must validate against its surface's model, or the property is passing on garbage.+- **The neutrality property test.** Generated documents that are valid at the current schema must replay to byte-identical output. This is the executable form of the theorem over the domain the theorem actually claims, and the sampler is what makes it worth anything.++The sampler mutates the surface's defaults three ways — dropping a path, swapping an enumerated spelling for another member of its recorded set, flipping a boolean — and **proposes each mutation from the fingerprint while letting the models decide whether it lands**: a proposal the models reject is simply not made and the document keeps the value it had. That division is forced, because [schema membership is decided by validators the fingerprint cannot see](#the-fingerprint). A sampler that consults only the projection can do nothing but drop keys — sound over every surface, since the defaults layer restores whatever a file omits, but never able to reach a different *value*, which is the whole reason a property beats a witness. A sampler that mutates whatever the annotation allows and checks nothing is red only when it happens to pick the wrong path, and the remedy everyone learns for that is to grow an exclusion list. Numbers and free strings are not perturbed at all, and that costs the property nothing: no operation's precondition mentions either, so perturbing one cannot separate a neutral replay from a non-neutral one.++It carries a **vacuity meta-test**: every generated document must validate against its surface's model, or the property is passing on garbage. Under a propose-and-decide sampler that assertion guards what the per-mutation decisions do not — the assembly around them, and any later model change that breaks it — and it is made against the emitted text, which is the artifact the property is quantified over. The other direction needs guarding too, because a sampler that quietly re-emitted the reference document every time would satisfy every property on this page: each mutation kind must be shown to be reachable.-Two further properties hold and are tested with it: replay is **idempotent** (applying a ledger twice equals applying it once) and **prefix-coherent** (a file that has already absorbed some entries lands where a replay from zero lands).+Two further properties hold and are tested with it: replay is **idempotent** (applying a ledger twice equals applying it once) and **prefix-coherent** (a file that has already absorbed some entries lands where a replay from zero lands). Both are claims about files that still have something to migrate, so they are quantified over a second generator — within-schema documents with retired material put back — which needs a vacuity check of its own: a ledger that acts on nothing satisfies both.## The ledger file@@ -157,7 +163,17 @@ The rules that govern the file:A removal that predates the first fingerprint has no observed diff and no snapshot to compare against. Such an entry may carry `pre_history = true`, in which case it **declares its own removed paths** — the reserved-path registry records them from the declaration rather than from a fingerprint diff — and ships a hand-authored `before` document so the transform check has a pair to verify. Convergence and neutrality then verify it exactly like any other entry.-This exception is bounded to entries carrying the flag, and exists for the changes that shipped before the ledger did. It is not a way to avoid a fingerprint diff for a change that has one.+This exception is bounded to entries carrying the flag, and exists for the changes that shipped before the ledger did. It is not a way to avoid a fingerprint diff for a change that has one — and because that is the whole risk of the flag, it is what the four rules below check rather than trust. A pre-history entry buys an exemption from one accounting and pays for it with another.++> **The declaration is not optional.** An entry carrying the flag and declaring nothing is refused when the ledger is parsed: the declaration is what stands in for the diff, so without one the flag exempts the entry from every accounting there is.++> **No declared path may appear in a fingerprint at or below the entry's own version.** A path some snapshot records is material whose removal *is* observable, so it is accounted against that snapshot like any other change. (A *later* version bringing a declared path back is a different failure with a different remedy, and [the reserved-path rule](#reserved-paths-and-names) reports it in those terms.)++> **The fingerprint pair the entry sits between must show no removal and no narrowing.** Stated from the other side by the coverage gate, which is the gate the flag exempts: if something did go away between those two snapshots — a path, a spelling, or the values a path accepts — the entry is not pre-history and the flag would be exempting a real change from the accounting it needs.++> **Every operation's source must be declared material, or lie beneath some.** The ordinary op-legality rule reads the reserved-path registry, which a pre-history entry feeds by declaration; the effect is the same rule and the same refusal. Beneath, because a declaration names the shape that retired and an operation may address one key inside it. The rule reads each source at its literal, pre-entry path — it does not follow the entry's own renames the way [sequential path state](#sequential-path-state) does — so an operation that would address a declared table under a name an earlier operation of the same entry gave it is refused; address the material first, then rename it.++The `before` document lives beside the golden chain as `[email protected]`, and it is the one file there that is neither snapshotted nor regenerated — a regenerated one would describe today's models, which is exactly what it is not about. Write it as a faithful representative of the old shape, carrying every path the entry's operations address, so that the transform check exercises each of them rather than taking them on the entry's word. The link then runs from that document instead of from `defaults@N-1`, and nothing else about the check changes: the same three claims verify it.## The operation vocabulary@@ -188,6 +204,8 @@ Both halves are deliberate and neither is wrong, but they differ, and a reviewerSome configuration nodes are **open dictionaries** — fields typed as a mapping from arbitrary keys to a value schema, where any key is legal, no fingerprint can enumerate the keys, and `extra="forbid"` says nothing. The keys under such a node belong to the user; the value schema belongs to us. So the fingerprint records value-schema paths beneath a `*` segment, and an operation may use `*` **exactly at an open node**, where the applier expands it over the keys the file actually contains. A field renamed inside every entry of an open mapping, or an enum member remapped across every value of one, is then an ordinary `safe` operation rather than an impossibility.+> **`move_key` takes no wildcard on either side.** A wildcard destination names no target: "move each entry's key into *some* entry" has no rule for which entry receives it. A wildcard *source* with the fixed destination that leaves it is many-to-one: on any file where two matched entries carry the key, the second lands on the destination the first just occupied and the whole operation conflicts. Both are refused when the ledger is parsed. The other kinds act on each matched entry in place, which is what makes a wildcard mean something for them.+### OutcomesThe applier reports one outcome per operation:@@ -208,7 +226,9 @@ The guarantees above hold only if entries cannot say certain things. These rules### Op legality-- **Every operation's source must be a path some schema version removed** — recorded as such in the reserved-path registry by the entry that removes it. An operation whose source is still a live path of the current schema would act on a valid file, and replay neutrality would be false.+Enforced by [`make check-ledger`](#the-targets-and-where-the-data-lives), against the entry's [sequential path state](#sequential-path-state) rather than against the literal path an operation spells — an operation acting inside a table an earlier operation of the same entry renamed is judged on the node it really addresses.++- **Every operation's source must be a path some schema version removed** — recorded as such in the reserved-path registry by the entry that removes it. An operation whose source is still a live path of the current schema would act on a valid file, and replay neutrality would be false. A `remap_value` is exempt by construction: what it retires is an enumerated *spelling*, not the path, which survives into the new schema — the remap legality rule is what governs it instead.- **No operation may address a concrete key beneath an open node.** Those keys are the user's, they are unbounded, and no schema change can remove one.- **A `*` segment is legal exactly at an open node**, and nowhere else.@@ -221,6 +241,7 @@ Legality is defined over an entry's **sequential path state**, not operation by- **Every path surviving the walk must be a path of the new fingerprint.** A survivor that is not is either an unaccounted removal or a misspelled destination; the two are the same defect seen from either end, and one containment check catches both.- **Every path of the new fingerprint that the walk removed must be a genuine addition** — that is, absent from the previous fingerprint. One that was present is **over-deletion**: an entry that dropped a parent table where it meant to drop one child.- **Every enumerated spelling lost between the two fingerprints must be remapped, following the path through the entry's own renames.** Members are compared by origin — an enumerated path of the previous schema is looked up at the path the walk carried it to — so a member lost by a path the same entry renamed still demands its `remap_value`, and the remap that supplies it addresses the key by its new name.+- **Every path whose value domain narrowed must carry a remap, or the entry must be `unsafe`.** Compared by origin like the spellings, and for the same reason: a narrowing hidden behind a rename would otherwise read as an unrelated addition and removal. The remedies are the two the [coverage table](#coverage-every-schema-change-is-accounted-for) names, and they are fewer than a removal has, because no structural operation can repair a value.The end state is deliberately *not* compared as "the new fingerprint minus its additions". That formulation reads a rename's destination as an addition and its source as a removal, so it demands that the destination be absent from the end state — which is exactly where a correct rename puts it. Containment plus the over-deletion check is the same guarantee stated in a form renames satisfy.@@ -248,15 +269,28 @@ The fingerprint is a normalized projection of a surface's model tree, checked in- whether the path is required;- the enum member set, where the type is enumerated;- the value in the surface's defaults layer, where one exists;-- an open-node marker, with the value-schema paths beneath it recorded under a `*` segment.+- an open-node marker, with the value-schema paths beneath it recorded under a `*` segment;+- the numeric and length bounds on the value, where there are any.It is serialized in a stable order and is **deliberately not raw `model_json_schema()` output**, which moves for reasons that have nothing to do with our schema — reference layout, titles, ordering, the validation library's own version. A gate that cries wolf gets regenerated reflexively, and that is how a gate dies.-The same reasoning drives two normalizations that look like information loss and are not. A nested model records its type as `table` and an enumerated one as `enum`, with no class name anywhere: renaming a *Python class* changes nothing in anybody's file and must move nothing, while renaming a *field* or an enum *member* — the things a file actually contains — moves the fingerprint and is caught. Constraint metadata is stripped for the same reason, at every level of an annotation rather than only the outermost one, so a validation-library upgrade that reshapes a constraint object cannot move a golden.+The same reasoning drives two normalizations that look like information loss and are not. A nested model records its type as `table` and an enumerated one as `enum`, with no class name anywhere: renaming a *Python class* changes nothing in anybody's file and must move nothing, while renaming a *field* or an enum *member* — the things a file actually contains — moves the fingerprint and is caught. Constraint objects are stripped out of the recorded *type* for the same reason, at every level of an annotation rather than only the outermost one, so a validation-library upgrade that reshapes one cannot move a rendered type.++The bounds are recorded beside the type instead, and the same principle decides what a bound may be: the projection is a **closed whitelist** over `gt`, `ge`, `lt`, `le`, `min_length`, `max_length` and `multiple_of`, read from `annotated_types` — a zero-dependency interchange vocabulary rather than validation-library internals — and **anything unrecognized is dropped rather than serialized**. A strictness flag, a before-validator, a pattern object, a bound expressed over dates, a constraint kind a future release invents: none of them reach the golden. What lands there is a function of our schema, not of the library's representation of it, which is the property the strip above defends.++> **`pattern` is excluded, permanently.** Regex containment is not decidable for real expressions, so a gate that compared patterns would read every pattern edit as a tightening, produce false positives on changes that break nobody, and be waved through until it caught nothing. There are no `pattern` constraints on any surface today, so saying so costs nothing.++A bound is read from two places, because a bound written the ordinary way appears in neither the other: pydantic folds a top-level `Field(ge=1)` into the field's *metadata*, leaving the annotation bare, while a bound inside a union member (`Annotated[int, Field(ge=1)] | Literal["unbounded"]`) stays in the annotation. A generic container's arguments are deliberately not descended into — a bound inside `list[Annotated[int, Field(ge=1)]]` binds the items, not the list. Across union members the widest bound wins, since a union's value domain is the union of its members' and a bound on one never binds the others.++An enumerated member set is collected from both sources that produce one: an enum class and a string literal type. To a TOML file they are the same thing — a closed set of legal spellings — and either can lose one. A member is only *lost*, though, when the new shape stops accepting it — that is, when the new type does not admit `str`: an enumerated type relaxed into a free string records no members afterwards, and reading that raw set difference as the loss of every spelling would demand a bump and a remap for the most benign loosening a configuration model can undergo. The exemption is exactly that shape and not "the type did not change": `Literal['a']` becoming `Literal[1]` renders the same type on both sides and records no members after, and it loses `'a'`. A literal over non-string values records no members at all — its domain is one more thing the projection does not see.-An enumerated member set is collected from both sources that produce one: an enum class and a string literal type. To a TOML file they are the same thing — a closed set of legal spellings — and either can lose one.+Two shapes are recorded as terminal rather than descended into, because no operation can address what lies beneath them: an array of tables, which has no path segment syntax and no wildcard form, and a model reachable from itself, which has no finite path set at all. Terminal means blind: a change to the model beneath an array of tables leaves the fingerprint unchanged, and the telemetry surface's `otlp` exporters are that shape today (see [Limits you will meet](#limits-you-will-meet)).-Two shapes are recorded as terminal rather than descended into, because no operation can address what lies beneath them: an array of tables, which has no path segment syntax and no wildcard form, and a model reachable from itself, which has no finite path set at all.+**What the fingerprint records is what the types say, and a validator can say something else.** Every recorded field is read from an annotation or from the metadata beside it, so a constraint expressed as a validator instead is invisible here — and that is not a corner case, it reaches the enum member set and the open-node marker alike. A telemetry mode is a member of the recorded set and is still rejected unless the document also carries the user id that mode requires. Several nodes typed `dict[str, X]` on the main configuration surface reject any key outside a fixed set, so the open-node marker claims a key space that is not the user's after all. A pair of validators demands every member of an enum as a key of every entry of an open mapping, which makes *adding* an enum member break a user's file — additively, as far as any projection can tell. Nothing here can be made to see any of it.++> **The gate does not claim what it cannot see.** Domain narrowing expressed in a validator — a non-empty check, a cross-field invariant, a completeness rule over user-supplied keys — is not visible from the schema and stays the author's responsibility. The [coverage gate](#coverage-every-schema-change-is-accounted-for) catches the mechanical subset: types and the whitelisted bounds. It does not catch the rest, and the coverage table would overclaim without this sentence.++The consequence for the gates is mild where they compare a fingerprint against a fingerprint, because a blind spot on both sides cancels. The consequence for anything that tries to *decide schema membership* from a fingerprint is total, and it is why [the neutrality property test](#replay-neutrality) asks the models rather than the projection.What each recorded field is for: requiredness and enum members are load-bearing for coverage and for remap legality; defaults are recorded so that a flipped default is *visible in the regeneration diff*, but a default value **never gates on its own** — it changes no shape and no operation can address it. The one place a default is decisive is [a required path that has none](#coverage-every-schema-change-is-accounted-for).@@ -268,11 +302,14 @@ Checks with distinct failure meanings, so that a red gate says which guarantee bRecompute the fingerprint and diff it against the golden for the surface's current schema version:+The verdict turns on **direction**, not on whether something moved: a change that leaves a user's file valid asks for a regeneration, and one that does not asks for a bump. That distinction is what the last row exists for — a narrowing removes no path and no spelling, so read as a raw diff it looks exactly like an addition.+| Diff | Verdict ||---|---|| Unchanged | Pass. |-| Paths or enum members added, or a type or default changed | Regenerate the golden. No version bump and no entry are demanded, because additive changes are absorbed by the defaults layer. |+| Paths or enum members added, a default changed, or a type or bound **widened** | Regenerate the golden. No version bump and no entry are demanded, because additive changes are absorbed by the defaults layer and a widened domain still accepts every value a file already carries. Widening is defined so the fingerprint can decide it: a union whose members are a superset of the old ones, an enumerated type becoming `str`, and a bound dropped or relaxed. || Any path or enum member removed or renamed | Require a schema version bump, an operation accounting for every removed path (wildcard paths included), and a remap or an `unsafe` entry for every removed enum member. Destinations are cross-checked against added paths. |+| A type or bound **narrowed** — `int → str`, `str → SomeEnum`, `list[int] → list[str]`, `ge=1 → ge=5` | Require a schema version bump and an entry, exactly as a removal does. Every path survives and every spelling is still enumerated, and a file carrying an out-of-domain value stops validating all the same. No structural operation can repair a value, so the entry must carry a `remap_value` for each narrowed path — where the old spellings can be enumerated and [remap legality](#replay-neutrality) accepts it — or be marked `unsafe`, which is the only remedy a tightened numeric bound has. A remap answers only for the string values it rewrites — a lost `str`, enum or literal member — and for nothing else: an `int \| literal` field that loses its number, or that loses a spelling *and* raises its floor, still demands `unsafe`, because no mapping reaches a number. |Alongside the diff, one standing invariant is checked on the new fingerprint itself, whether or not anything changed:@@ -286,7 +323,9 @@ Stating it over every required path rather than only over newly added ones costs### Convergence — replay is neutral on the witnesses-A full replay of a surface's ledger over each of its reference documents produces zero applied operations. The same target checks what else is cheap and unambiguous: the ledger parses into the migration-operation subset, versions are contiguous and match ids, every `safe` remap satisfies the legality rule, and no entry reintroduces a reserved path.+A full replay of a surface's ledger over each of its reference documents produces zero applied operations, reports nothing about them, and returns the very bytes it was given. The same target checks what else is cheap and unambiguous: the ledger parses into the migration-operation subset, versions are contiguous and match ids, every operation is legal, every `safe` remap satisfies the legality rule, and no entry reintroduces a reserved path or a remapped-away spelling.++> **This target reads checked-in files and never fingerprints a live model** — the ledgers, the stored golden chain, the two reference documents (one of which, for a surface whose defaults come from its model, is rendered from that model's default values — rendered, not fingerprinted). That restraint is what earns it a place in `make agent-check`: every failure is a statement about a file the author wrote, and every remedy is to fix one. A check that read the models would go red on an ordinary configuration edit with *regenerate the golden* as its remedy, which is the fail-regenerate-fail cycle that keeps the coverage gate out of the loop agents run constantly. The cost is bounded and visible: between a version bump and the `make up-migration-schemas` that snapshots it, the new link does not exist, so the entry checks that need a fingerprint pair report the missing link by name instead of running.### Transform goldens — the operations say what the schema change did@@ -298,15 +337,27 @@ The regenerator writes, for each surface, the snapshot of its **current** schemaFreezing the head instead would rot. An additive change is absorbed by the defaults layer and needs no bump, so a head frozen at the last bump would drift from the live model and eventually stop validating against it — while the whole point of the last link is that it is what the current model reads. Only the sparse kit template is not snapshotted at all: it is a convergence and neutrality witness, and both read it live, so a snapshot of it would have no reader.-The check then asserts, pairwise over the complete-defaults family:+The check then applies entry N to `defaults@N−1` and asserts three things about what comes out, pairwise over the complete-defaults family:-> `paths(apply(entry N, defaults@N−1)) == paths(defaults@N) − added_at_N`, where `added_at_N` is `fingerprint@N` minus `fingerprint@N−1`; plus value equality on every path an operation wrote; plus validation of the last link against the current model.+> **Every path the migration creates, schema version N has** — in `defaults@N` or in `fingerprint@N`; **every path `defaults@N−1` and `defaults@N` share survives the migration**; and **the last link's migrated document is accepted by the current model**, read the way a user's file is read — beneath the current defaults layer, not alone.-The comparator is **not byte-exact and not a subset test**, and both exclusions are deliberate. Byte-exactness fails on any honest version bump, because the same commit adds keys, edits comments and flips unrelated defaults. A subset test is blind to over-deletion — deleting a parent table where the entry meant to delete one child would pass. What survives is a comparator that tolerates additions, comment edits and default flips, and is red on wrong destinations, wrong remap targets, wrong order and over-deletion.+Each claim is one-directional on purpose, and together they are red on wrong destinations, wrong remap targets, wrong order and over-deletion. The first catches a destination that is misspelled or lands where the new shape carries nothing, which is the defect the whole check exists for: coverage accounts for the removed path and convergence skips the absent source, so without this the typo migrates every user file to a key `extra="forbid"` rejects, with the tool reporting success. The second catches an entry that dropped a parent table where it meant to drop one child. The third catches what moves no path at all — a value the schema does not accept, or a key it does not know.++One exemption belongs to the first claim: a created path whose **container** is absent from `defaults@N` is not compared. When a whole entry of an open mapping is dropped from a packaged document between versions, what an operation did inside it says nothing about the operation.++The comparator is **not byte-exact, not a subset test, and not an equality**. Byte-exactness fails on any honest version bump, because the same commit adds keys, edits comments and flips unrelated defaults. A subset test is blind to over-deletion. An equality is blind in the other direction: it would refuse every addition the same commit made. What survives tolerates additions, comment edits and default flips, because nothing asserts them.++> **What the comparator is not: `paths(defaults@N) − added_at_N`.** A raw fingerprint difference counts a rename's *destination* as an addition, so subtracting it would demand the destination be absent from the expected shape — exactly where a correct rename puts it. This is the same defect the [symbolic end-state walk](#sequential-path-state) met and answered with containment, and the answer here is the same one expressed over documents. Subtraction would also be blind by construction to everything a fingerprint cannot see: a model added to a packaged deck lives beneath an open node, where the fingerprint records a value schema and never a key, so that addition would be asserted rather than tolerated and the check would go red on ordinary content.++> **What the first claim reads: `defaults@N` *or* `fingerprint@N`, never the document alone.** An optional key whose default is `None` has no value in any reference document — TOML has no null, and the synthesized document drops the key — while being a perfectly ordinary destination for a migration to move a user's value onto. Against the document alone, the check would refuse the destination the schema most obviously has. A misspelled destination is in neither the document nor the fingerprint, so nothing the claim was protecting is given up, and the fingerprint is checked-in data like everything else the check reads. The fingerprint is read *with its wildcards*: a document names the user's own key where the fingerprint names `*`, so `deck.claude.new_name` is recorded as `deck.*.new_name`, and a rename beneath an open mapping whose destination the reference document happens not to carry under that entry is a correct rename, not a misspelled one.++> **What the comparator does not assert: value equality against `defaults@N`.** A default flipped in the same commit as a rename makes such a check red with no remedy available to anyone — the older link is frozen, the head link tracks, and a migrated file legitimately carries the user's old value where the new reference document carries the new default. What that assertion was for, *the operations produce values the new schema accepts*, is checked where it can be checked soundly: by the last link's validation above, and by [the remap legality rule](#replay-neutrality), which refuses a remap whose target spelling the new schema does not accept.++One kind of entry has no link to check: an **unsafe** entry is reported and never applied, so no document ever makes that transition mechanically, and the vocabulary explicitly grants such an entry the right to be incomplete — an entry with no operations at all is legal precisely when it is unsafe. A **pre-history** entry does have a link, and this is where it is verified: it has no snapshot on the far side of it, which is what the flag means, so the link starts from its hand-authored [`[email protected]`](#pre-history-entries) and answers the same three claims. An entry marked pre-history with no such document is refused by name.The sparse kit template is **not** in the chain: starter-template edits between bumps would go red under equality, and it exercises no operation the complete document does not. It remains a convergence and neutrality witness.-Pairwise checks compose by induction into the full chain — a replay from any historical snapshot lands on the current shape, because prefix entries skip on any snapshot at or past their version, their sources being permanently removed material.+Pairwise checks compose by induction into the full chain — a replay from any historical snapshot lands on the current shape, because prefix entries skip on any snapshot at or past their version, their sources being permanently removed material. That induction is also why only the last link is validated against a model: it is the only link whose model we still have, and every earlier one was the last link when it was authored.### Testing the gates without coupling them to the configuration models@@ -314,11 +365,13 @@ The surface registry is an **injected parameter, never a module constant**. TheWithout this, every legitimate configuration change turns the gate's own suite red alongside the gate, and the fix everyone learns is "regenerate the goldens" — which is how a gate goes permanently green while catching nothing.+[The neutrality property](#replay-neutrality) is the deliberate exception: it is quantified over the real surfaces, because the domain it claims is the one real files live in. What keeps that from costing what the rule warns about is the propose-and-decide sampler — a model that grows a new validator makes a proposed mutation stop landing, rather than making the suite red. Everything the property needs a ledger with entries in it for runs against a synthetic surface, as the rule says.+### The targets, and where the data lives| Target | Alias | Aggregates | What it does ||---|---|---|---|-| `make check-ledger` | `cl` | `check`, `agent-check` | The ledger parses into migration operations, versions are contiguous and match ids, legality and remap legality hold, no entry reintroduces a reserved path, and replay over both reference documents applies nothing. Reads checked-in data and regenerates nothing. |+| `make check-ledger` | `cl` | `check`, `agent-check` | The ledger parses into migration operations, versions are contiguous and match ids, legality and remap legality hold, no entry reintroduces a reserved path, and replay over both reference documents applies nothing. Reads checked-in data only — no live model is fingerprinted and nothing is regenerated. || `make check-migration-schemas` | `cmig` | `check` | Fingerprints every surface, diffs against the goldens, demands a bump and full accounting for every removal, cross-checks destinations, and verifies the pairwise transform-golden chain. || `make up-migration-schemas` | `umig` | `up` | Regenerates the fingerprint goldens, and snapshots the reference documents into the transform-golden chain when a schema version bumped. |@@ -331,9 +384,10 @@ pipelex/migration/ledgers/<surface-id>.toml the ledgerspipelex/migration/goldens/<surface-id>/[email protected] the fingerprint at schema version N[email protected] the complete reference document at schema version N+ [email protected] a pre-history entry's hand-authored starting document```-Two files per version and nothing else. The fingerprint the coverage check diffs against is the chain's head link, `fingerprint@<current>`, rather than a separate always-current copy — one file with one writer cannot disagree with itself. The reserved-path registry is [derived from the chain](#reserved-paths-and-names), and the sparse kit template is read live rather than snapshotted.+Two files per version, and a third only where a [pre-history entry](#pre-history-entries) needs one. The fingerprint the coverage check diffs against is the chain's head link, `fingerprint@<current>`, rather than a separate always-current copy — one file with one writer cannot disagree with itself. The reserved-path registry is [derived from the chain](#reserved-paths-and-names), and the sparse kit template is read live rather than snapshotted.## What the engine reports@@ -342,17 +396,24 @@ Every surface produces the same plan model:```MigrationReport└── MigrationPlan (per file)- surface, file_path- blocked_reason → set when the file itself could not be processed at all- steps[] → id, to_schema_version, title, breaking, safety, ops[], changelog- blocked[] → id, reason, guidance+ surface_id, file_path+ blocked_reason, blocked_detail → set when the file itself could not be processed at all+ backup_path, was_written → set when the run wrote the file+ steps[] → entry_id, to_schema_version, title, description, breaking, safety, applied_ops[]+ blocked[] → entry_id, to_schema_version, reason, detail, guidance, applied_ops[]unexplained[] → path, note```-There is no `from_version` and no trusted-version concept: nothing skips, so nothing needs one. Because a replay walks every entry while usually changing little, the report renders what actually changed — it filters on `APPLIED`, routes `CONFLICT` into `blocked`, and carries the downgrade diagnosis in `unexplained`.+There is no `from_version` and no trusted-version concept: nothing skips, so nothing needs one. Because a replay walks every entry while usually changing little, the report renders what actually changed — a step lists only the operations that fired, not the many that skipped.Two different things can be blocked, and the report keeps them apart. An **entry** is blocked when it cannot be applied — it is `unsafe`, or one of its operations came back `CONFLICT` — and it lands in `blocked[]` with the reason and its guidance, while the rest of the file's entries proceed. A **file** is blocked when it cannot be processed at all: it is unparseable, it is unwritable, or it changed on disk between the read and the write. That reason sits on the plan itself, and a blocked file never stops its siblings — every other file in the surface is migrated and reported normally.+Two rules govern how an entry appears, and both exist so that a report is never more optimistic than the file:++> **An entry is reported once.** An entry with a conflicting operation lands in `blocked[]` and not in `steps[]`, carrying in its own `applied_ops` whichever of its operations did land before the conflict was found. Operation-level atomicity is the applier's — a conflicting operation writes nothing — but an entry is not atomic, and saying it arrived whole when part of it could not would be a lie the next run has to correct.++> **An `unsafe` entry is reported only when it would do something.** It is rehearsed against the document and the result discarded, so an entry with nothing to do on this file stays silent. Reporting every `unsafe` entry regardless would warn every user with a perfectly current file, at every boot, forever — and a warning nobody can act on is a warning everybody learns to ignore.+> **No value read from a user's file is ever rendered** — not in the command's output, not in the structured plan, not in an error. Paths, operation kinds and ledger-supplied values carry everything a plan needs to say.That is a mechanical rule rather than a list of credential-shaped key names, because such a list is a guess that eventually misses one. The single deliberate exception is the backup file, which contains the user's values by definition and is protected by inheriting the source file's mode rather than the process umask.@@ -392,10 +453,24 @@ The rules this encodes:The `.mthds` fix path follows its applier with a canonical reflow of the whole file. That is right for `.mthds` files, which are canonically formatted by CI, and wrong for a user-owned configuration, where a one-key rename must not also rewrite the user's spacing and layout. It is also load-bearing for the guarantees: byte-level replay neutrality only holds if serialization contributes no changes of its own.+Stated more strongly than the library can promise on its own: **a replay in which no operation applies returns the very text it was given** — not a re-serialization of it. Neutrality is therefore a property of the engine rather than of the TOML library it happens to use, and a round-trip can never contribute a change of its own to a file that needed nothing.++### The document is re-read between operations that applied++> **Operations are applied one at a time, and the document is parsed afresh after each one that changed it.**++This is a measured requirement, not caution. The position-preserving rename the applier depends on leaves the node's raw `dict` storage out of step with the body it renders from, for any value that is not a table: everything the library renders or looks up still works, but *addressing that key again in the same in-memory document* raises from inside the library. Always-replay runs many operations over one document, so migration is the caller that meets it — the `.mthds` fix path renames tables, which take the branch the library keeps consistent.++Re-reading is exact, because serialization is byte-faithful, and it costs one parse per **applied** operation — which under always-replay is almost never, since the common case is a current file where everything skips. The behaviour is pinned by a characterization test so that a library upgrade which fixes it makes the workaround removable rather than invisible.+### BackupsAlways back up, before writing. Exactly one backup per file — a successful run replaces the previous one — named with a UTC timestamp, inheriting the source file's mode rather than the default umask, and with its path printed in the report. For files tracked in git, git remains the durable history; the backup covers the untracked ones and the moment between two commits.+The name is the source file's whole name, extension included, followed by `.bak.` and a compact UTC stamp: `pipelex.toml` backs up to `pipelex.toml.bak.20260815T120000Z`. Extension-included so a backup never shadows a real `.toml`, and separator-free so no filesystem objects to it. Pruning matches that whole shape, stamp included — a copy the user made by hand under a name that merely starts the same way (`pipelex.toml.bak.notes`) is theirs, and is never touched.++The order of the three steps is itself a guarantee. The backup is written **first**, so there is never a moment with a rewritten file and no copy of the original; the older backups are pruned **last**, so there is never a moment with no backup at all; and a write that fails before touching the file takes its own fresh backup with it, so a failed run leaves the directory exactly as it found it. A write whose outcome the transaction cannot vouch for — a rollback it could not complete — is the one case that keeps its backup: the report names the copy, because that is precisely the state a backup exists for. Nothing that happens after the file is written can un-write it in the report: an older backup that will not prune, or a temp file the transaction could not remove, is a warning on a migrated file, never a blocked one.+### Per-file transactionsEach file is written transactionally and independently: snapshot, stage, atomic replace, restore on failure. A file that is unparseable, unwritable or changed during the run is reported as blocked while its siblings proceed. No run leaves a partially rewritten file behind.@@ -404,17 +479,18 @@ Each file is written transactionally and independently: snapshot, stage, atomicThese are measured properties of the TOML library the engine uses, not aspirations. They are stated here because a user or an author will run into them, and being surprised by them is worse than being told.-- **Arrays of tables are unaddressable, and reserved.** An `[[entry]]` node is a list, and no `table_path` segment syntax reaches it or anything under it; the `*` wildcard has no array form. No configuration surface contains one today. If one ever appears, addressing it needs a new segment kind and a decision on the day — until then, an operation pointed at one resolves nothing and is reported as a guarded skip rather than raising.+- **Arrays of tables are unaddressable, and reserved.** An `[[entry]]` node is a list, and no `table_path` segment syntax reaches it or anything under it; the `*` wildcard has no array form. One surface carries one today — the telemetry configuration's `otlp` list of exporters — and the fingerprint records it as a terminal `list[table]`: what happens *inside* an exporter entry (a renamed field, a tightened bound, a new required member) is invisible to the coverage gate, so a change there is the author's to account for by hand, exactly as validator-expressed narrowing is. Addressing what lies beneath one needs a new segment kind and a decision on the day — until then, an operation pointed at one resolves nothing and is reported as a guarded skip rather than raising.- **A moved table loses its introducing comment.** A comment block written above a table is stored as trailing trivia of the *previous* sibling, not as part of the table it appears to introduce, so it does not travel with a move. Comments *inside* a moved table travel intact, including trailing comments on individual keys. The consequence is real: a file seeded from a heavily-commented template and then migrated ends up with banner comments labelling sections that have moved away. Nothing in the engine may rely on comment fidelity across a move, and the package's own files are corrected by hand rather than by migration.- **A migration is not byte-minimal.** A rename adds a small amount of whitespace, and a renamed table that was written out of order across several chunks loses its own bare header while a plain table keeps it. Both forms are semantically identical, both are stable under replay, and neither accumulates — but a migration diff is not the minimal diff a human would have written.- **A guarded skip is never an error.** Every operation whose target does not resolve reports itself skipped rather than raising. That guard is what makes always-replay possible, and it is why a misdirected operation is caught by the checks rather than by a crash on a user's machine.+- **A renamed key cannot be addressed again in the same in-memory document.** The position-preserving rename updates the body the document renders from but leaves the node's raw `dict` storage stale, for any value that is not a table. Nothing the library renders or looks up is affected — which is why a single rename is correct, and why the `.mthds` fix path, whose renames target tables, never meets it. The engine [re-reads the document between operations that applied](#the-document-is-re-read-between-operations-that-applied), so nothing downstream has to know about this; it is recorded here because it explains a piece of the engine that would otherwise look like superstition.## Authoring an entryThe gate says what is missing; you should not be composing entries by hand from a diff. The intended loop:1. Make the schema change.-2. Run the coverage check. It refuses the change and names every removed path and enum member that lacks accounting, every destination that does not match an added path, and every added path that has no default.+2. Run the coverage check. It refuses the change and names every removed path and enum member that lacks accounting, every path whose value domain narrowed, every destination that does not match an added path, and every added path that has no default.3. Write the entry — or have the migration-authoring skill derive it from the fingerprint diff, bump the surface's schema version, regenerate the fingerprint golden and snapshot the transform goldens.4. Add the changelog entry. The changelog and the ledger are deliberately separate artifacts saying the same thing to different readers; the release process is where they are checked against each other.diff --git a/pipelex/cli/dev_cli/_dev_cli.py b/pipelex/cli/dev_cli/_dev_cli.pyindex 75f3b0c55..b830bac1c 100644--- a/pipelex/cli/dev_cli/_dev_cli.py+++ b/pipelex/cli/dev_cli/_dev_cli.py@@ -14,6 +14,7 @@ from pipelex.cli.dev_cli.commands.check_config_sync_cmd import LeadingConfig, chfrom pipelex.cli.dev_cli.commands.check_gateway_models_cmd import check_gateway_models_cmdfrom pipelex.cli.dev_cli.commands.check_hub_layering_cmd import check_hub_layering_cmdfrom pipelex.cli.dev_cli.commands.check_keyword_only_cmd import check_keyword_only_cmd+from pipelex.cli.dev_cli.commands.check_ledger_cmd import check_ledger_cmdfrom pipelex.cli.dev_cli.commands.check_migration_schemas_cmd import check_migration_schemas_cmdfrom pipelex.cli.dev_cli.commands.check_mthds_schema_cmd import check_mthds_schema_cmdfrom pipelex.cli.dev_cli.commands.check_rules_sync_cmd import check_rules_sync_cmd@@ -45,6 +46,7 @@ class PipelexDevCLI(TyperGroup):"check-gateway-models","check-hub-layering","check-keyword-only",+ "check-ledger","check-migration-schemas","check-mthds-schema","check-rules",@@ -283,6 +285,28 @@ def check_mthds_schema_command(sys.exit(1)[email protected](name="check-ledger", help="Verify that every migration ledger is legal and replays harmlessly over current documents")+def check_ledger_command(+ quiet: Annotated[+ bool, typer.Option("--quiet", "-q", help="Light output on success (single line); the full issue list still prints on failure")+ ] = False,+) -> None:+ """Verify that every configuration surface's migration ledger is legal and converges."""+ try:+ check_ledger_cmd(quiet=quiet)+ except (typer.Exit, typer.Abort):+ # Typer control-flow exits carry an intended exit code — not a failure. Let them through.+ raise+ except Exception: # noqa: BLE001+ # Dev CLI command root: print a traceback for any unexpected failure and exit non-zero.+ console = get_console()+ console.print()+ console.print("[bold red]Unexpected error occurred[/bold red]")+ console.print()+ console.print(Traceback())+ sys.exit(1)++@app.command(name="check-migration-schemas", help="Verify that every configuration surface has accounted for its schema changes")def check_migration_schemas_command(quiet: Annotated[diff --git a/pipelex/cli/dev_cli/commands/check_ledger_cmd.py b/pipelex/cli/dev_cli/commands/check_ledger_cmd.pynew file mode 100644index 000000000..f1d6fac6a--- /dev/null+++ b/pipelex/cli/dev_cli/commands/check_ledger_cmd.py@@ -0,0 +1,63 @@+"""Command to verify that every migration ledger says only legal things, and replays harmlessly.++This is the static half of the migration gate. It reads the checked-in ledgers, the golden chain+and the two reference documents, and refuses an entry that would act on live material, reach into+a user's own key space, rewrite a value a user may legitimately have chosen, resurrect a retired+name, or fire on a file that is already current.++Unlike `check-migration-schemas`, it joins `make agent-check`: it fingerprints no live model and+regenerates nothing, so every failure is a statement about a file the author wrote and every+remedy is to fix one — never "regenerate the golden and move on".+"""++import sys++from rich.markup import escape++from pipelex.migration.exceptions import MigrationError+from pipelex.migration.ledger_check import LedgerIssue, check_ledgers+from pipelex.migration.surfaces import build_config_surface_registry, packaged_migration_dir+from pipelex.runtime_hub import get_console+++def check_ledger_cmd(*, quiet: bool = False) -> None:+ """Verify that every configuration surface's migration ledger is legal and converges.++ Args:+ quiet: If True, keep the success output to a single line (for Make targets / CI). Quiet+ only trims success output: a failure always lists every issue, because the Make+ targets invoke the check quietly and a red gate has to say what to do.+ """+ console = get_console()++ if not quiet:+ console.print()+ console.print("[bold]Checking migration ledgers...[/bold]")+ console.print()++ registry = build_config_surface_registry()+ try:+ issues = check_ledgers(registry=registry, migration_dir=packaged_migration_dir())+ except MigrationError as exc:+ # An error is always loud — quiet only trims success output, never failures.+ console.print(f"[red]✗ Migration ledger check: FAILED[/red] - {escape(str(exc))}")+ sys.exit(1)++ if not issues:+ surface_count = len(registry.surfaces)+ console.print(f"[green]✓ Migration ledger check: PASSED[/green] - {surface_count} ledgers legal and convergent")+ return++ _print_issues(issues=issues)+ sys.exit(1)+++def _print_issues(*, issues: list[LedgerIssue]) -> None:+ console = get_console()+ console.print(f"[red]✗ Migration ledger check: FAILED[/red] - {len(issues)} issues")+ console.print()+ for issue in issues:+ # Messages quote ledger text and path lists, both of which can look like Rich markup.+ console.print(f" [yellow]{escape(issue.surface_id)}[/yellow] [dim]({issue.kind})[/dim]")+ console.print(f" {escape(issue.message)}")+ console.print()diff --git a/pipelex/cli/dev_cli/commands/check_migration_schemas_cmd.py b/pipelex/cli/dev_cli/commands/check_migration_schemas_cmd.pyindex 0b1abc989..07c591bf3 100644--- a/pipelex/cli/dev_cli/commands/check_migration_schemas_cmd.py+++ b/pipelex/cli/dev_cli/commands/check_migration_schemas_cmd.py@@ -1,21 +1,26 @@"""Command to verify that every configuration surface has accounted for its schema changes.-This is the coverage half of the migration gate. It reads the checked-in ledgers and golden-chains, recomputes each surface's fingerprint, and fails when a schema change would leave a user's-file broken with nothing written down about how to repair it.+This is the golden half of the migration gate, and it asks two questions of every surface. Coverage+reads the checked-in ledgers and golden chains, recomputes each surface's fingerprint, and fails+when a schema change would leave a user's file broken with nothing written down about how to repair+it. The transform goldens then *do* the migration each entry describes, from one frozen reference+document to the next, and fail when what the operations produce is not the shape the new version+actually has.-It lives in `make check` and deliberately **not** in `make agent-check`: it is a golden check, and-in the loop agents run constantly a fail-regenerate-fail cycle with no single right answer is how-a gate goes permanently green while catching nothing.+Both live in `make check` and deliberately **not** in `make agent-check`: they are golden checks,+and in the loop agents run constantly a fail-regenerate-fail cycle with no single right answer is+how a gate goes permanently green while catching nothing."""import sys+from collections.abc import Sequencefrom rich.markup import escapefrom pipelex.migration.coverage import CoverageIssue, check_registryfrom pipelex.migration.exceptions import MigrationErrorfrom pipelex.migration.surfaces import build_config_surface_registry, packaged_migration_dir+from pipelex.migration.transform_check import TransformIssue, check_transformsfrom pipelex.runtime_hub import get_console@@ -35,8 +40,11 @@ def check_migration_schemas_cmd(*, quiet: bool = False) -> None:console.print()registry = build_config_surface_registry()+ migration_dir = packaged_migration_dir()+ issues: list[CoverageIssue | TransformIssue] = []try:- issues = check_registry(registry=registry, migration_dir=packaged_migration_dir())+ issues.extend(check_registry(registry=registry, migration_dir=migration_dir))+ issues.extend(check_transforms(registry=registry, migration_dir=migration_dir))except MigrationError as exc:# An error is always loud — quiet only trims success output, never failures.console.print(f"[red]✗ Migration schema check: FAILED[/red] - {escape(str(exc))}")@@ -51,7 +59,7 @@ def check_migration_schemas_cmd(*, quiet: bool = False) -> None:sys.exit(1)-def _print_issues(*, issues: list[CoverageIssue]) -> None:+def _print_issues(*, issues: Sequence[CoverageIssue | TransformIssue]) -> None:console = get_console()console.print(f"[red]✗ Migration schema check: FAILED[/red] - {len(issues)} issues")console.print()diff --git a/pipelex/kit/agent_rules/codex_commands.md b/pipelex/kit/agent_rules/codex_commands.mdindex 7461088f4..5dafcc587 100644--- a/pipelex/kit/agent_rules/codex_commands.md+++ b/pipelex/kit/agent_rules/codex_commands.md@@ -7,7 +7,7 @@```bashmake agent-check# If the current system doesn't have the `make` command,- # lookup the "agent-check" target in the Makefile and run the commands one by one (targets fix-unused-imports fix-keyword-only format lint pyright mypy check-keyword-only check-hub-layering drift-check)+ # lookup the "agent-check" target in the Makefile and run the commands one by one (targets fix-unused-imports fix-keyword-only format lint pyright mypy check-ledger check-keyword-only check-hub-layering drift-check)```This runs multiple code quality tools:@@ -15,6 +15,7 @@- Ruff: Fix unused imports, lint, format- Mypy: Static type checker- plxt: Format and lint TOML, MTHDS, and PLX files+ - Migration ledgers: every checked-in ledger must be legal and replay harmlessly over its reference documents (`make check-ledger`, alias `cl`)- Drift contracts: open code↔docs review obligations fail the gate (note the digest reads the git index, so stage your changes for the check to see them)Always fix any issues reported by these tools before proceeding.diff --git a/pipelex/kit/agent_rules/commands.md b/pipelex/kit/agent_rules/commands.mdindex 52eba80c6..530458bf3 100644--- a/pipelex/kit/agent_rules/commands.md+++ b/pipelex/kit/agent_rules/commands.md@@ -7,7 +7,7 @@```bashmake agent-check# If the current system doesn't have the `make` command,- # lookup the "agent-check" target in the Makefile and run the commands one by one (targets fix-unused-imports fix-keyword-only format lint pyright mypy check-keyword-only check-hub-layering drift-check)+ # lookup the "agent-check" target in the Makefile and run the commands one by one (targets fix-unused-imports fix-keyword-only format lint pyright mypy check-ledger check-keyword-only check-hub-layering drift-check)```This runs multiple code quality tools:@@ -15,6 +15,7 @@- Ruff: Fix unused imports, lint, format- Mypy: Static type checker- plxt: Format and lint TOML, MTHDS, and PLX files+ - Migration ledgers: every checked-in ledger must be legal and replay harmlessly over its reference documents (`make check-ledger`, alias `cl`)- Drift contracts: open code↔docs review obligations fail the gate (note the digest reads the git index, so stage your changes for the check to see them; resolve with the drift workflow below)Always fix any issues reported by these tools before proceeding.diff --git a/pipelex/migration/backup.py b/pipelex/migration/backup.pynew file mode 100644index 000000000..d6f94399a--- /dev/null+++ b/pipelex/migration/backup.py@@ -0,0 +1,96 @@+"""Backups — the copy taken before a user's configuration file is rewritten.++Always, before writing. Exactly **one** backup per file: a successful run replaces the previous+one, so a directory does not accumulate a decade of copies of a file whose durable history is+already in git. For the untracked files — and for the moment between two commits — this copy is+the whole safety net.++The mode is inherited from the source rather than left to the process umask, and that is the one+deliberate exception to "no value read from a user's file is ever rendered": a backup contains the+user's values by definition, so a `0600` configuration must not become a world-readable `0644`+copy sitting beside it.++See `docs/migration-ledger.md` → "Backups".+"""++import re+from datetime import datetime+from pathlib import Path++from pipelex.pipeline.fixes.file_transaction import FileSnapshot, write_staged_file++BACKUP_INFIX = ".bak."+"""What marks a file as one of our backups. Appended to the whole file name, extension included,+so `pipelex.toml` backs up to `pipelex.toml.bak.<stamp>` and never shadows a real `.toml`."""++BACKUP_STAMP_FORMAT = "%Y%m%dT%H%M%SZ"+"""A UTC stamp with no separators a filesystem dislikes — no colons, which Windows refuses."""++_BACKUP_STAMP_PATTERN = re.compile(r"\d{8}T\d{6}Z")+"""The stamp, as the shape `BACKUP_STAMP_FORMAT` renders — what tells one of our backups from a+file the user named `<file>.bak.notes` themselves."""+++def backup_stamp(*, moment: datetime) -> str:+ """Render a moment as the stamp a backup file name carries.++ The moment is a parameter rather than read here, so that a caller stamps every file of one+ run identically and a test can assert on a name it chose.+ """+ return moment.strftime(BACKUP_STAMP_FORMAT)+++def backup_path_for(*, path: Path, moment: datetime) -> Path:+ return path.with_name(f"{path.name}{BACKUP_INFIX}{backup_stamp(moment=moment)}")+++def existing_backups_of(*, path: Path) -> list[Path]:+ """Every backup of one file currently on disk, oldest name first.++ Matched by name rather than by a recorded list: the backups are the user's files too, and a+ side record of which ones we wrote would be one more piece of untracked state to go stale.+ Matched by the *whole* name we write, stamp included, because a directory can also hold a+ copy the user made by hand under a name that merely starts the same way — and pruning must+ never take one of those.+ """+ # Matched by string prefix rather than by glob, so a file name carrying a glob metacharacter+ # (`telemetry_[eu].toml`, say) can never match a sibling's backups and have them pruned.+ prefix = f"{path.name}{BACKUP_INFIX}"+ return sorted(+ candidate+ for candidate in path.parent.iterdir()+ if candidate.name.startswith(prefix) and _BACKUP_STAMP_PATTERN.fullmatch(candidate.name[len(prefix) :])+ )+++def write_backup(*, snapshot: FileSnapshot, moment: datetime) -> Path:+ """Write the pre-migration copy of one file and return where it landed.++ Staged and atomically renamed like any other write in this codebase, which also gets the mode+ right for free: the staged temp file is created `0600` and `fchmod`-ed to the source's mode+ before it is ever visible under its final name. A rename that fails takes the staged copy with+ it — the user's directory is not where a half-made backup gets left.+ """+ destination = backup_path_for(path=snapshot.path, moment=moment)+ staged_path = write_staged_file(snapshot=snapshot, content=snapshot.content, label="backup")+ try:+ staged_path.replace(destination)+ except OSError:+ staged_path.unlink(missing_ok=True)+ raise+ return destination+++def prune_backups_except(*, path: Path, keep: Path) -> list[Path]:+ """Delete every backup of `path` other than `keep`, returning the ones removed.++ Called only after the new file is committed: pruning first would leave a window with no+ backup at all, which is the one state this helper exists to prevent.+ """+ pruned: list[Path] = []+ for candidate in existing_backups_of(path=path):+ if candidate == keep:+ continue+ candidate.unlink(missing_ok=True)+ pruned.append(candidate)+ return pruneddiff --git a/pipelex/migration/coverage.py b/pipelex/migration/coverage.pyindex 065630802..f6ef386ad 100644--- a/pipelex/migration/coverage.py+++ b/pipelex/migration/coverage.py@@ -4,11 +4,12 @@ This is the half of the gate that needs no applier and no engine: it reads the cand golden chains, recomputes each surface's fingerprint, and refuses a change that would leave auser's file broken with nothing written down about how to repair it.-The centrepiece is the **sequential path state** walk. An entry's operations chain — a parent is-renamed and then keys inside it are renamed — so the intermediate paths belong to neither the old-fingerprint nor the new one, and no ordering avoids that. The walk therefore replays the entry-symbolically over the previous fingerprint's path set and compares the end state with the new-fingerprint. One walk answers every question at once:+The centrepiece is the **sequential path state** walk, which lives in `walk.py` because the ledger+check reads it too. An entry's operations chain — a parent is renamed and then keys inside it are+renamed — so the intermediate paths belong to neither the old fingerprint nor the new one, and no+ordering avoids that. The walk replays the entry symbolically over the previous fingerprint's path+set, and this module compares the end state with the new fingerprint. One walk answers every+question at once:- an operation whose source is absent from the state — or a `delete_table` aimed at a key — is**dead**: it can never fire, and the applier's guarded skip would report success forever;@@ -22,31 +23,33 @@ fingerprint. One walk answers every question at once:a parent table where it meant to drop one child;- an enumerated member lost between the two fingerprints is looked up at the path the walk carriedit to, so a member lost by a path the same entry renamed still demands its remap;+- a path whose value domain **narrowed** — its type stopped accepting what it accepted, or a bound+ tightened — is looked up the same way and demands the same accounting, because a change that+ keeps every path and every spelling still breaks the file that carries an out-of-domain value;- everything the new fingerprint has and the walk does not is a genuine addition, which needs nooperation because the defaults layer absorbs it.+What this module deliberately does **not** check is what an operation may say in the first place —+that its source is removed material, that it does not reach into user key space, that a `safe`+remap cannot fire on a current file. Those are static properties of the ledger against the+checked-in chain, they need no live model, and they live in `ledger_check.py`, which is the half+of the apparatus that runs in `agent-check`.+See `docs/migration-ledger.md` → "Legality rules" and "The checks"."""-from collections.abc import Sequencefrom enum import StrEnumfrom pathlib import Pathfrom pydantic import BaseModel, ConfigDict, Field-from pipelex.migration.fingerprint import PATH_SEPARATOR, TABLE_TYPE, PathFingerprint, SurfaceFingerprint, compute_fingerprint+from pipelex.migration.fingerprint import PATH_SEPARATOR, TABLE_TYPE, SurfaceFingerprint, compute_fingerprintfrom pipelex.migration.goldens import defaults_golden_path, read_fingerprint_goldenfrom pipelex.migration.ledger import MigrationEntry, MigrationLedger, MigrationSafety, load_ledger+from pipelex.migration.narrowing import describe_narrowing, lost_enumerated_spellingsfrom pipelex.migration.surfaces import Surface, SurfaceRegistry-from pipelex.suggested_fix import (- WILDCARD_SEGMENT,- DeleteKeyOp,- DeleteTableOp,- MigrationOp,- MoveKeyOp,- RemapValueOp,- RenameTableKeyOp,-)+from pipelex.migration.walk import EntryWalk, walk_entry+from pipelex.suggested_fix import WILDCARD_SEGMENTclass CoverageIssueKind(StrEnum):@@ -62,8 +65,9 @@ class CoverageIssueKind(StrEnum):UNACCOUNTED_PATH = "unaccounted_path"OVER_DELETION = "over_deletion"ENUM_MEMBER_NOT_REMAPPED = "enum_member_not_remapped"- ILLEGAL_REMAP = "illegal_remap"+ VALUE_DOMAIN_NARROWED = "value_domain_narrowed"REQUIRED_PATH_WITHOUT_DEFAULT = "required_path_without_default"+ PRE_HISTORY_HAS_A_DIFF = "pre_history_has_a_diff"class CoverageIssue(BaseModel):@@ -84,6 +88,10 @@ class FingerprintDiff(BaseModel):changed_paths: list[str] = Field(default_factory=list[str])added_enum_members: dict[str, list[str]] = Field(default_factory=dict[str, list[str]])removed_enum_members: dict[str, list[str]] = Field(default_factory=dict[str, list[str]])+ narrowed_paths: dict[str, list[str]] = Field(default_factory=dict[str, list[str]])+ """Paths whose value domain shrank without losing the path or an enumerated member, mapped to+ the reasons. Every one is also in `changed_paths` — what this adds is the *direction*, which is+ what separates a change a user's file survives from one it does not."""@propertydef is_empty(self) -> bool:@@ -93,6 +101,12 @@ class FingerprintDiff(BaseModel):def has_removals(self) -> bool:return bool(self.removed_paths or self.removed_enum_members)+ def render_removals(self) -> str:+ return str(self.removed_paths + [f"{path}: {members}" for path, members in self.removed_enum_members.items()])++ def render_narrowings(self) -> str:+ return "; ".join(f"'{path}' ({', '.join(reasons)})" for path, reasons in self.narrowed_paths.items())+def diff_fingerprints(*, before: SurfaceFingerprint, after: SurfaceFingerprint) -> FingerprintDiff:before_paths = before.path_names()@@ -101,18 +115,19 @@ def diff_fingerprints(*, before: SurfaceFingerprint, after: SurfaceFingerprint)added_enum_members: dict[str, list[str]] = {}removed_enum_members: dict[str, list[str]] = {}+ narrowed_paths: dict[str, list[str]] = {}changed_paths: list[str] = []for path in shared:before_record = before.paths[path]after_record = after.paths[path]if before_record != after_record:changed_paths.append(path)- before_members = set(before_record.enum_members or [])- after_members = set(after_record.enum_members or [])- if added := sorted(after_members - before_members):+ if added := sorted(set(after_record.enum_members or []) - set(before_record.enum_members or [])):added_enum_members[path] = added- if removed := sorted(before_members - after_members):+ if removed := lost_enumerated_spellings(before=before_record, after=after_record):removed_enum_members[path] = removed+ if narrowing := describe_narrowing(before=before_record, after=after_record):+ narrowed_paths[path] = narrowingreturn FingerprintDiff(added_paths=sorted(after_paths - before_paths),@@ -120,159 +135,14 @@ def diff_fingerprints(*, before: SurfaceFingerprint, after: SurfaceFingerprint)changed_paths=changed_paths,added_enum_members=added_enum_members,removed_enum_members=removed_enum_members,+ narrowed_paths=narrowed_paths,)-class PathState(BaseModel):- """The symbolic state of a path set part-way through an entry's operations.-- Each live path is carried with the path it *originated from* in the previous fingerprint, so- that an operation acting on a path some earlier operation renamed can still be attributed to- the schema path it is really about.- """-- model_config = ConfigDict(extra="forbid")-- origin_by_current: dict[str, str]-- @classmethod- def make_from_fingerprint(cls, *, fingerprint: SurfaceFingerprint) -> "PathState":- return cls(origin_by_current={path: path for path in fingerprint.path_names()})-- def current_paths(self) -> set[str]:- return set(self.origin_by_current)-- def subtree_of(self, *, path: str) -> list[str]:- prefix = path + PATH_SEPARATOR- return [current for current in self.origin_by_current if current == path or current.startswith(prefix)]-- def drop_subtree(self, *, path: str) -> None:- for current in self.subtree_of(path=path):- del self.origin_by_current[current]-- def move_subtree(self, *, source: str, destination: str) -> None:- moved = {destination + current[len(source) :]: self.origin_by_current[current] for current in self.subtree_of(path=source)}- self.drop_subtree(path=source)- self.origin_by_current.update(moved)-- def final_path_of_origin(self, *, origin: str) -> str | None:- for current, recorded_origin in self.origin_by_current.items():- if recorded_origin == origin:- return current- return None---class RecordedRemap(BaseModel):- """A remap that fired during the walk, attributed to the schema path it originated from."""-- model_config = ConfigDict(frozen=True, extra="forbid")-- origin_path: str- old_values: list[str]- new_values: list[str]---class EntryWalk(BaseModel):- """What replaying one entry over the previous fingerprint's path set found out."""-- model_config = ConfigDict(extra="forbid")-- state: PathState- """The end state."""-- dead_ops: list[str] = Field(default_factory=list[str])- """Descriptions of operations that can never fire: the source is absent, or the operation- kind cannot act on what the source is."""-- occupied_destinations: list[str] = Field(default_factory=list[str])- """Descriptions of renames or moves onto a path the state already had. The applier refuses- to clobber an occupied destination, so a file carrying both keys — a valid file at the old- schema — would come back CONFLICT on every run."""-- remaps: list[RecordedRemap] = Field(default_factory=list[RecordedRemap])- """The remaps that fired, attributed to the schema path they originated from."""---def _joined(*, segments: Sequence[str]) -> str:- return PATH_SEPARATOR.join(segments)---def walk_entry(*, entry: MigrationEntry, before: SurfaceFingerprint) -> EntryWalk:- """Replay an entry's operations symbolically over the previous fingerprint's path set."""- walk = EntryWalk(state=PathState.make_from_fingerprint(fingerprint=before))-- for op in entry.ops:- source = _op_source_path(op=op)- if source not in walk.state.origin_by_current:- walk.dead_ops.append(f"{op.kind} on '{source}' — no such path at schema version {before.schema_version}, so it can never fire")- continue- origin_record = before.paths[walk.state.origin_by_current[source]]- if isinstance(op, DeleteTableOp) and not _is_table_like(record=origin_record):- walk.dead_ops.append(f"{op.kind} on '{source}' — that path is a key, not a table, so the applier would skip it forever; use delete_key")- continue- _apply_op_to_state(op=op, source=source, walk=walk)-- return walk---def _is_table_like(*, record: PathFingerprint) -> bool:- """Whether the applier's `delete_table` finds a table at this path.-- An open node counts: `dict[str, X]` is a `[table]` in a file, whatever the fingerprint calls- its value type.- """- return record.value_type == TABLE_TYPE or record.open_node---def _op_source_path(*, op: MigrationOp) -> str:- match op:- case DeleteTableOp():- return _joined(segments=op.table_path)- case DeleteKeyOp() | RenameTableKeyOp() | MoveKeyOp() | RemapValueOp():- return _joined(segments=[*op.table_path, op.key])---def _apply_op_to_state(*, op: MigrationOp, source: str, walk: EntryWalk) -> None:- match op:- case DeleteKeyOp() | DeleteTableOp():- walk.state.drop_subtree(path=source)- case RenameTableKeyOp():- _move_in_state(op=op, source=source, destination=_joined(segments=[*op.table_path, op.new_key]), walk=walk)- case MoveKeyOp():- _move_in_state(op=op, source=source, destination=_joined(segments=[*op.new_table_path, op.new_key]), walk=walk)- case RemapValueOp():- walk.remaps.append(- RecordedRemap(- origin_path=walk.state.origin_by_current[source],- old_values=sorted(op.mapping),- new_values=sorted(set(op.mapping.values())),- )- )---def _move_in_state(*, op: MigrationOp, source: str, destination: str, walk: EntryWalk) -> None:- """Move a subtree, recording a collision when the destination is already occupied.-- The move is performed either way, so that the end-state comparison stays meaningful and the- collision is reported once rather than echoed as an unaccounted source.- """- if walk.state.subtree_of(path=destination):- walk.occupied_destinations.append(- f"{op.kind} moves '{source}' onto '{destination}', which the previous schema already has — "- f"a file carrying both is refused as a conflict on every run, so a safe entry cannot do this"- )- walk.state.move_subtree(source=source, destination=destination)--def check_entry_accounting(*, surface_id: str, entry: MigrationEntry, before: SurfaceFingerprint, after: SurfaceFingerprint) -> list[CoverageIssue]:"""Verify that one entry says exactly what the fingerprint diff shows happened."""if entry.pre_history:- # A removal that predates the first fingerprint has no observed diff to be accounted- # against — that is what the flag means. Such an entry declares its own removed paths and- # ships a hand-authored `before` document, and is verified against those by `check-ledger`- # rather than here. Checking it against a fingerprint pair it never described would- # produce a failure naming the wrong defect.- return []+ return _check_the_pre_history_claim(surface_id=surface_id, entry=entry, before=before, after=after)issues: list[CoverageIssue] = []walk = walk_entry(entry=entry, before=before)@@ -303,23 +173,98 @@ def check_entry_accounting(*, surface_id: str, entry: MigrationEntry, before: Su))- for path in sorted((after_paths - end_paths) & before_paths):+ for origin, landing in sorted(_over_deleted_landings(before_paths=before_paths, walk=walk).items()):+ if landing not in after_paths or landing in end_paths:+ continue+ where = f"'{origin}'" if landing == origin else f"'{origin}' (which is '{landing}' at schema version {after.schema_version})"issues.append(CoverageIssue(surface_id=surface_id,kind=CoverageIssueKind.OVER_DELETION,message=(- f"entry '{entry.id}': its operations remove '{path}', but schema version {after.schema_version} still has it — "+ f"entry '{entry.id}': its operations remove {where}, but schema version {after.schema_version} still has it — "f"an operation targets a parent where it meant to target one child"),))issues.extend(_check_enum_accounting(surface_id=surface_id, entry=entry, before=before, after=after, walk=walk))- issues.extend(_check_remap_legality(surface_id=surface_id, entry=entry, after=after, walk=walk))+ issues.extend(_check_narrowing_accounting(surface_id=surface_id, entry=entry, before=before, after=after, walk=walk))return issues+def _over_deleted_landings(*, before_paths: set[str], walk: EntryWalk) -> dict[str, str]:+ """For every path the entry's operations remove, the path it *would* have had at the new schema.++ Compared by origin, like every other accounting here: a child deleted beneath a table the same+ entry renamed is spelled one way in the previous fingerprint and another in the next, and a+ comparison by current spelling never lines the two up — the deletion of a path the new schema+ still has would pass as the disappearance of one path and the appearance of an unrelated+ other. So each removed origin is carried through its nearest surviving ancestor's rename, and+ it is that landing the caller looks for in the new schema.+ """+ landings: dict[str, str] = {}+ for origin in before_paths:+ if walk.state.final_path_of_origin(origin=origin) is not None:+ continue+ segments = origin.split(PATH_SEPARATOR)+ landing = origin+ for depth in range(len(segments) - 1, 0, -1):+ ancestor = PATH_SEPARATOR.join(segments[:depth])+ ancestor_landing = walk.state.final_path_of_origin(origin=ancestor)+ if ancestor_landing is not None:+ landing = PATH_SEPARATOR.join([ancestor_landing, *segments[depth:]])+ break+ landings[origin] = landing+ return landings+++def _check_the_pre_history_claim(+ *,+ surface_id: str,+ entry: MigrationEntry,+ before: SurfaceFingerprint,+ after: SurfaceFingerprint,+) -> list[CoverageIssue]:+ """A pre-history entry claims its change is invisible here, and that claim is checkable.++ The flag says the change predates the first fingerprint, so there is no diff to account the+ entry against — which is precisely why the accounting above cannot run and precisely why the+ flag must not be taken on trust. What replaces the accounting is the claim itself: **the+ fingerprint pair must show no removal at all.** A removal between these two snapshots is a+ change that *does* have an observed diff, and the flag would be exempting it from the only+ gate that would have demanded an operation for it.++ Additions are not the flag's business, here as everywhere: the defaults layer absorbs them.++ A narrowed value domain is the same defect reached from the other side and is refused the same+ way: the paths all survive, so nothing is *removed*, but a file valid at the previous version+ stops validating at this one — which is a change with an observed diff, and the flag would+ exempt it from the accounting that would have demanded a remap or an `unsafe` marking for it.++ What the declaration itself says, and whether the operations stay inside it, is `check-ledger`'s+ to verify against the checked-in chain; what the entry does to the hand-authored `before`+ document is the transform check's, inside `check-migration-schemas`. Neither needs the diff+ this function reads, and this function exists to look at that diff.+ """+ diff = diff_fingerprints(before=before, after=after)+ if not diff.has_removals and not diff.narrowed_paths:+ return []+ observed = diff.render_removals() if diff.has_removals else diff.render_narrowings()+ return [+ CoverageIssue(+ surface_id=surface_id,+ kind=CoverageIssueKind.PRE_HISTORY_HAS_A_DIFF,+ message=(+ f"entry '{entry.id}' is marked pre_history, but between schema versions {before.schema_version} and "+ f"{after.schema_version} a file already valid at the older one stops validating: {observed}. That change is "+ f"observable here, so the flag would exempt it from the accounting it needs. Drop the flag and account for it; "+ f"the flag is for material that predates the first fingerprint, which by definition no snapshot shows going away"+ ),+ )+ ]++def _check_enum_accounting(*,surface_id: str,@@ -352,7 +297,7 @@ def _check_enum_accounting(# The path does not survive into the new schema; the unaccounted-path or over-deletion# check has already said so in terms the author can act on.continue- removed_members = set(before_record.enum_members) - set(after_record.enum_members or [])+ removed_members = set(lost_enumerated_spellings(before=before_record, after=after_record))unaccounted = sorted(removed_members - remapped_by_origin.get(origin, set()))if unaccounted:issues.append(@@ -368,68 +313,59 @@ def _check_enum_accounting(return issues-def _check_remap_legality(+def _check_narrowing_accounting(*,surface_id: str,entry: MigrationEntry,+ before: SurfaceFingerprint,after: SurfaceFingerprint,walk: EntryWalk,) -> list[CoverageIssue]:- """A `safe` remap must be provably unable to fire on a current-valid file.-- That holds when the target is enumerated at the new schema and every old spelling in the- mapping now falls outside its member set. When an old spelling remains legal — or the path is- a free string, where staleness can never be proven from the schema — the applier cannot tell a- stale value from a deliberate choice, and the entry has to be `unsafe`.+ """Every path whose value domain the entry narrows must carry a remap, or the entry must be unsafe.++ Paths are compared by *origin*, exactly as enumerated members are: a path the same entry+ renamed is looked up where the walk carried it, so a narrowing hidden behind a rename still+ demands its accounting instead of slipping through as an unrelated addition and removal.++ Only two remedies exist, because no structural operation can repair a value the new schema+ refuses. A `remap_value` is the real fix where the old spellings can be enumerated and the+ remap legality rule accepts it — a free string becoming enum-typed is the case that fits.+ Everywhere else, a tightened numeric bound above all, the entry has to say `unsafe`: the+ migration is then reported to the user and never applied, which is the honest answer when the+ tool cannot tell a stale value from a deliberate one."""if entry.safety is MigrationSafety.UNSAFE:+ # An unsafe entry is reported and never applied, so it is allowed to describe a change no+ # operation can make. That is what `unsafe` is for.return []+ remapped_origins = {remap.origin_path for remap in walk.remaps}issues: list[CoverageIssue] = []- for remap in walk.remaps:- final_path = walk.state.final_path_of_origin(origin=remap.origin_path)- record = after.paths.get(final_path) if final_path is not None else None- if record is None:- # The path the remap targets does not survive into the new schema; the unaccounted-path- # or over-deletion check above has already said so in terms the author can act on.+ for origin, before_record in before.paths.items():+ final_path = walk.state.final_path_of_origin(origin=origin)+ after_record = after.paths.get(final_path) if final_path is not None else None+ if after_record is None:+ # The path does not survive into the new schema; the unaccounted-path or over-deletion+ # check has already said so in terms the author can act on.continue- member_set = set(record.enum_members or [])- if not member_set:- issues.append(- CoverageIssue(- surface_id=surface_id,- kind=CoverageIssueKind.ILLEGAL_REMAP,- message=(- f"entry '{entry.id}': remap_value on '{final_path}' is 'safe', but that path is not enumerated at "- f"schema version {after.schema_version} — staleness cannot be proven from the schema, so the entry must be unsafe"- ),- )- )+ # A remap rewrites spellings, so it answers for a string-typed member the type stopped+ # accepting — and for nothing else. A lost numeric member, or a bound tightened on the same+ # path, is still a value no mapping can repair, and must not ride under the remap into a+ # `safe` entry.+ reasons = describe_narrowing(before=before_record, after=after_record, remapped=origin in remapped_origins)+ if not reasons:continue- still_legal = sorted(set(remap.old_values) & member_set)- if still_legal:- issues.append(- CoverageIssue(- surface_id=surface_id,- kind=CoverageIssueKind.ILLEGAL_REMAP,- message=(- f"entry '{entry.id}': remap_value on '{final_path}' is 'safe', but {still_legal} are still legal values at "- f"schema version {after.schema_version} — a user who chose one deliberately would have it rewritten"- ),- )- )- unknown_new = sorted(set(remap.new_values) - member_set)- if unknown_new:- issues.append(- CoverageIssue(- surface_id=surface_id,- kind=CoverageIssueKind.ILLEGAL_REMAP,- message=(- f"entry '{entry.id}': remap_value on '{final_path}' rewrites values to {unknown_new}, which schema version "- f"{after.schema_version} does not accept — every migrated file would be rejected, with the tool reporting success"- ),- )+ issues.append(+ CoverageIssue(+ surface_id=surface_id,+ kind=CoverageIssueKind.VALUE_DOMAIN_NARROWED,+ message=(+ f"entry '{entry.id}': '{final_path}' accepts fewer values than it did — {', '.join(reasons)}. A file "+ f"carrying one of the values it has stopped accepting no longer validates, and no structural operation "+ f"can repair a value, so the entry needs a remap_value for that path, or must be marked unsafe"+ ),)+ )return issues@@ -621,19 +557,39 @@ def _check_head_link(*, surface_id: str, current_version: int, live: SurfaceFingdiff = diff_fingerprints(before=golden, after=live)if diff.is_empty:return []++ breaking: list[CoverageIssue] = []if diff.has_removals:- removed = diff.removed_paths + [f"{path}: {members}" for path, members in diff.removed_enum_members.items()]- return [+ breaking.append(CoverageIssue(surface_id=surface_id,kind=CoverageIssueKind.REMOVAL_NEEDS_A_BUMP,message=(- f"the models no longer have {removed}, which breaks every file that carries them. Bump "+ f"the models no longer have {diff.render_removals()}, which breaks every file that carries them. Bump "f"current_schema_version to {current_version + 1}, add the entry '{surface_id}@{current_version + 1}' "f"accounting for each, then run `make umig`"),)- ]+ )+ if diff.narrowed_paths:+ # A narrowing keeps every path and every enumerated spelling, so the diff looks additive+ # and would otherwise be answered with "regenerate the golden" — while a value a user's+ # file legitimately carries has stopped validating.+ breaking.append(+ CoverageIssue(+ surface_id=surface_id,+ kind=CoverageIssueKind.REMOVAL_NEEDS_A_BUMP,+ message=(+ f"the models still have every path, but they accept fewer values than the golden records: "+ f"{diff.render_narrowings()}. A file valid before this change is not valid after it, so this is a "+ f"removal like any other. Bump current_schema_version to {current_version + 1}, add the entry "+ f"'{surface_id}@{current_version + 1}' carrying a remap_value for each narrowed path — or marked "+ f"unsafe, where no remap can express it — then run `make umig`"+ ),+ )+ )+ if breaking:+ return breakingreturn [CoverageIssue(surface_id=surface_id,diff --git a/pipelex/migration/documents.py b/pipelex/migration/documents.pynew file mode 100644index 000000000..c0ffe08ae--- /dev/null+++ b/pipelex/migration/documents.py@@ -0,0 +1,45 @@+"""Reading a TOML document as a set of addressable paths and values.++The fingerprint projects a *model tree*; this projects a *document*. The two use the same dotted+addressing so a check can compare them, but they are not the same thing and the difference is+load-bearing: a document carries the concrete keys a user chose beneath an open node, where the+fingerprint records only the value schema under a `*`, and a document omits every optional key+that is not set.++Used by the checks that compare a migrated document against the one the new schema expects.+"""++from typing import Any, cast++from pipelex.migration.fingerprint import PATH_SEPARATOR+++def document_paths(*, document: dict[str, Any]) -> set[str]:+ """Every addressable path in a document, tables and leaves alike.++ Tables are included as paths of their own, not only their leaves, so that deleting a whole+ section is visible as the removal of the section rather than only of its contents.+ """+ return set(flatten_document(document=document))+++def flatten_document(*, document: dict[str, Any]) -> dict[str, Any]:+ """A document as a flat path-to-value mapping, table paths included with value `None`.++ An array of tables is recorded as a terminal path and not descended into, matching the+ fingerprint: no `table_path` segment syntax reaches inside one, so nothing can address it.+ """+ flattened: dict[str, Any] = {}+ _flatten_into(mapping=document, prefix=(), flattened=flattened)+ return flattened+++def _flatten_into(*, mapping: dict[str, Any], prefix: tuple[str, ...], flattened: dict[str, Any]) -> None:+ for key, value in mapping.items():+ path = (*prefix, str(key))+ dotted = PATH_SEPARATOR.join(path)+ if isinstance(value, dict):+ flattened[dotted] = None+ _flatten_into(mapping=cast("dict[str, Any]", value), prefix=path, flattened=flattened)+ continue+ flattened[dotted] = valuediff --git a/pipelex/migration/engine.py b/pipelex/migration/engine.pynew file mode 100644index 000000000..643593168--- /dev/null+++ b/pipelex/migration/engine.py@@ -0,0 +1,180 @@+"""The engine — replaying a surface's ledger over one document's text.++This module is deliberately filesystem-free: it takes the text of a file and a parsed ledger and+returns the text the file should now have, plus what happened. Everything about reading, backing+up and writing files lives in `runner.py`, so that the checks — which replay over reference+documents that are never written anywhere — reuse exactly the code a user's migration runs.++**Text in, text out, and one operation at a time.** The document is re-read between operations+that applied, rather than carried as a single long-lived DOM. That is not caution, it is a+measured requirement: tomlkit's position-preserving rename leaves the node's raw `dict` storage+out of step with the body it renders from, so a later operation addressing the renamed key raises+`KeyError` from deep inside the library (see+`tests/unit/pipelex/pipeline/fixes/test_fix_applier_rename_dom_consistency.py`, which pins the+behaviour). Re-reading is exact — serialization is byte-faithful — and it costs a parse of a few+hundred lines per *applied* operation, which under always-replay is almost never.++Three properties this module is responsible for:++- **Every run replays everything.** There is no version resolution step and no state stamp to+ read: each entry is replayed in order and the applier skips whatever is already gone or already+ done. Any side record of "what has been applied" would eventually report "nothing to do" beside+ a broken boot, which is the worst failure this tool can have.+- **Serialization adds nothing.** When no operation applies, the text returned is the text that+ came in — the same string, untouched — so byte-level replay neutrality is a property of this+ function rather than a property of the TOML library we happen to use. When something does+ apply, the output is `tomlkit.dumps` of the mutated document and nothing else: no canonical+ reflow, because a one-key rename must not rewrite a user's spacing.+- **An `unsafe` entry is never written, and is only *reported* when it would do something.** It+ is rehearsed against the text and the result discarded: on a file already at the current schema+ every operation skips, so a correct file stays silent instead of warning at every boot.++See `docs/migration-ledger.md` → "Schema versions, and why every run replays everything",+"Applying", and "What the engine reports".+"""++from collections.abc import Sequence++import tomlkit+from pydantic import BaseModel, ConfigDict, Field++from pipelex.migration.ledger import MigrationEntry, MigrationLedger, MigrationSafety+from pipelex.migration.plan import BlockedEntry, BlockedEntryReason, MigrationStep+from pipelex.pipeline.fixes.applier import FixOpApplication, apply_fix_ops+from pipelex.suggested_fix import MigrationOp+++class OpsApplication(BaseModel):+ """The result of applying a run of operations to one document's text."""++ model_config = ConfigDict(frozen=True, extra="forbid")++ text: str+ applications: list[FixOpApplication]++ @property+ def applied_count(self) -> int:+ return sum(1 for application in self.applications if application.outcome.did_apply)++ @property+ def conflicts(self) -> list[FixOpApplication]:+ return [application for application in self.applications if application.outcome.is_conflict]+++class DocumentReplay(BaseModel):+ """What replaying a whole ledger over one document did to it."""++ model_config = ConfigDict(frozen=True, extra="forbid")++ text: str+ """The document as it should now be. Identical to the input when nothing applied."""++ steps: list[MigrationStep] = Field(default_factory=list[MigrationStep])+ blocked: list[BlockedEntry] = Field(default_factory=list[BlockedEntry])++ @property+ def did_change_document(self) -> bool:+ return any(step.applied_ops for step in self.steps) or any(entry.applied_ops for entry in self.blocked)+++def apply_ops_over_text(*, text: str, ops: Sequence[MigrationOp]) -> OpsApplication:+ """Apply operations in order to a document's text, re-reading it between applied ones.++ The returned text is the *same string* when no operation applied, which is what makes replay+ neutrality a property of this code. Operation-level atomicity is the applier's own: a+ conflicting operation writes nothing, including a wildcard one whose conflict sits in the last+ matched entry, so a conflict leaves the text exactly as it found it too.+ """+ current_text = text+ toml_doc = tomlkit.loads(current_text)+ applications: list[FixOpApplication] = []+ for op in ops:+ applied_now = apply_fix_ops(toml_doc=toml_doc, ops=[op])+ applications.extend(applied_now)+ if not any(application.outcome.did_apply for application in applied_now):+ continue+ current_text = tomlkit.dumps(toml_doc) # pyright: ignore[reportUnknownMemberType]+ toml_doc = tomlkit.loads(current_text)+ return OpsApplication(text=current_text, applications=applications)+++def replay_ledger_over_text(*, ledger: MigrationLedger, text: str) -> DocumentReplay:+ """Replay every entry of a surface's ledger over one document, in order.++ Entries compose in sequence, which is what lets a file coming from schema version 1 land+ correctly when a key was renamed at 2 and moved at 4.+ """+ current_text = text+ steps: list[MigrationStep] = []+ blocked: list[BlockedEntry] = []+ for entry in ledger.migration:+ match entry.safety:+ case MigrationSafety.UNSAFE:+ _rehearse_unsafe_entry(entry=entry, text=current_text, blocked=blocked)+ case MigrationSafety.SAFE:+ current_text = _apply_safe_entry(entry=entry, text=current_text, steps=steps, blocked=blocked)+ return DocumentReplay(text=current_text, steps=steps, blocked=blocked)+++def _rehearse_unsafe_entry(*, entry: MigrationEntry, text: str, blocked: list[BlockedEntry]) -> None:+ """Find out whether an `unsafe` entry has anything to say about this file, writing nothing.++ The rehearsal's text is discarded, so the file is untouched whatever the outcome. An entry+ with nothing to do stays silent: reporting it regardless would warn every user with a+ perfectly current file, at every boot, forever.+ """+ rehearsal = apply_ops_over_text(text=text, ops=entry.ops)+ if not rehearsal.applied_count and not rehearsal.conflicts:+ return+ blocked.append(+ BlockedEntry(+ entry_id=entry.id,+ to_schema_version=entry.to_schema_version,+ reason=BlockedEntryReason.UNSAFE,+ detail=(+ f"this file needs the changes of '{entry.id}', but the entry is marked unsafe: the applier cannot "+ f"tell a stale value from a deliberate choice, so it is reported and never applied"+ ),+ guidance=entry.guidance,+ )+ )+++def _apply_safe_entry(*, entry: MigrationEntry, text: str, steps: list[MigrationStep], blocked: list[BlockedEntry]) -> str:+ """Apply a `safe` entry and fold its operation outcomes into the plan, returning the new text.++ What is decided here is how the *entry* is reported: a conflict anywhere routes the whole+ entry into `blocked[]`, carrying whichever of its operations did apply, so the report never+ claims an entry landed whole when part of it could not.+ """+ application = apply_ops_over_text(text=text, ops=entry.ops)+ applied_ops = [op for op, outcome in zip(entry.ops, application.applications, strict=True) if outcome.outcome.did_apply]+ conflicts = application.conflicts+ if conflicts:+ details = "; ".join(str(conflict.detail) for conflict in conflicts)+ blocked.append(+ BlockedEntry(+ entry_id=entry.id,+ to_schema_version=entry.to_schema_version,+ reason=BlockedEntryReason.CONFLICT,+ detail=f"{len(conflicts)} of {len(application.applications)} operations conflict: {details}",+ guidance=entry.guidance,+ applied_ops=applied_ops,+ )+ )+ return application.text+ if applied_ops:+ steps.append(+ MigrationStep(+ entry_id=entry.id,+ to_schema_version=entry.to_schema_version,+ title=entry.title,+ description=entry.description,+ breaking=entry.breaking,+ safety=entry.safety,+ applied_ops=applied_ops,+ )+ )+ # Skipping is the overwhelmingly common outcome under always-replay and is entirely benign —+ # the target is gone, or the change is already there. Reports suppress it.+ return application.textdiff --git a/pipelex/migration/fingerprint.py b/pipelex/migration/fingerprint.pyindex 1aa100eeb..7e7749ecc 100644--- a/pipelex/migration/fingerprint.py+++ b/pipelex/migration/fingerprint.py@@ -3,8 +3,8 @@The fingerprint is what the coverage gate diffs, so what it records and what it deliberatelyignores is the whole design. It records, per TOML-addressable path: the type, whether the path isrequired within its parent, the enumerated member set where there is one, the value the surface's-defaults layer supplies, and an open-node marker with the value-schema paths beneath it recorded-under a `*` segment.+defaults layer supplies, an open-node marker with the value-schema paths beneath it recorded under+a `*` segment, and a projection of the value's numeric and length bounds.It is **deliberately not raw `model_json_schema()` output**, which moves for reasons that havenothing to do with our schema — reference layout, titles, ordering, the validation library's own@@ -14,19 +14,33 @@ model renders as `table` and an enum as `enum`, so renaming a *Python class* movrenaming a *field* or an enum *member* — the things a user's file actually contains — moves thefingerprint and is caught.+The bound projection is built on the same principle, which is why it is a **closed whitelist**:+only the constraint kinds named in `CONSTRAINT_ATTRIBUTE_BY_CARRIER` are read, and anything else+an annotation carries is dropped rather than serialized. What lands in the golden is then a+function of our schema rather than of the validation library's representation of it — a strictness+flag, a before-validator, a pattern object or a constraint kind invented by a future release moves+nothing. `pattern` is excluded on purpose and permanently: regex containment is not decidable for+real expressions, so every pattern edit would read as a tightening, produce false positives, and+teach everyone to wave the gate through.+See `docs/migration-ledger.md` → "The fingerprint"."""import typesfrom collections.abc import Mapping, Sequence-from enum import Enum+from enum import Enum, StrEnumfrom typing import Annotated, Any, Literal, Union, cast, get_args, get_origin+from annotated_types import Ge, Gt, Le, Lt, MaxLen, MinLen, MultipleOffrom pydantic import BaseModel, ConfigDict, Field+from pydantic.fields import FieldInfofrom pipelex.suggested_fix import WILDCARD_SEGMENTPATH_SEPARATOR = "."+UNION_SEPARATOR = " | "+"""How a union renders in `value_type` — the one definition both the writer here and the parser in+`narrowing.py` read, so the two cannot drift apart."""TABLE_TYPE = "table""""The rendering of a nested model. Class names are deliberately not recorded — see the module docstring."""@@ -34,6 +48,54 @@ TABLE_TYPE = "table"ENUM_TYPE = "enum""""The rendering of an enumerated type. The member set carries the shape; the class name does not."""+LITERAL_TYPE = "literal"+"""The rendering of a `Literal[...]`. Its spellings are recorded as enumerated members, like an enum's."""++STRING_TYPE = "str"+"""The rendering of `str` — the type every enumerated spelling widens into."""+++class ConstraintKind(StrEnum):+ """The closed whitelist of constraint kinds the fingerprint records.++ Each value is also the attribute name the matching `annotated_types` object carries, which is+ what lets one mapping serve both the projection and the golden's key names.+ """++ GT = "gt"+ GE = "ge"+ LT = "lt"+ LE = "le"+ MIN_LENGTH = "min_length"+ MAX_LENGTH = "max_length"+ MULTIPLE_OF = "multiple_of"++ @property+ def widest_is_the_lower_value(self) -> bool:+ """Which direction of this kind admits more values — the merge rule across union members.++ A union's value domain is the union of its members' domains, so a bound found on one+ member never binds the others: the widest of the two is the one the path really has.+ """+ match self:+ case ConstraintKind.GT | ConstraintKind.GE | ConstraintKind.MIN_LENGTH | ConstraintKind.MULTIPLE_OF:+ return True+ case ConstraintKind.LT | ConstraintKind.LE | ConstraintKind.MAX_LENGTH:+ return False+++CONSTRAINT_ATTRIBUTE_BY_CARRIER: dict[type[Any], ConstraintKind] = {+ Gt: ConstraintKind.GT,+ Ge: ConstraintKind.GE,+ Lt: ConstraintKind.LT,+ Le: ConstraintKind.LE,+ MinLen: ConstraintKind.MIN_LENGTH,+ MaxLen: ConstraintKind.MAX_LENGTH,+ MultipleOf: ConstraintKind.MULTIPLE_OF,+}+"""The whitelist itself. `annotated_types` is a zero-dependency interchange vocabulary, not+validation-library internals, which is what makes reading it compatible with the strip above."""+class PathFingerprint(BaseModel):"""What the fingerprint records about one TOML-addressable path."""@@ -54,6 +116,13 @@ class PathFingerprint(BaseModel):"""A mapping from arbitrary user-owned keys to a value schema. The keys belong to the user;the value schema belongs to us and is recorded beneath a `*` segment."""+ constraints: dict[ConstraintKind, int | float] | None = None+ """The whitelisted bounds on the value, or `None` for none recorded. Recorded so that a bound+ a schema change *tightens* is visible to the gate: it keeps every path and every enumerated+ spelling, so nothing else in the projection moves, while a value a user's file legitimately+ carries stops validating. What is deliberately absent is everything outside the whitelist —+ see the module docstring."""+class SurfaceFingerprint(BaseModel):"""A whole surface's model tree, projected and stably ordered."""@@ -116,6 +185,10 @@ def _walk_model(_record_field(path=(*prefix, field_name),annotation=field_info.annotation,+ # A top-level `Annotated[int, Field(ge=1)]` is unwrapped by pydantic before we ever see+ # the annotation: the constraint is folded into the field's metadata instead, so a walk+ # that read the annotation alone would be blind to every bound declared the usual way.+ field_metadata=field_info.metadata,required=field_info.is_required(),defaults_value=_defaults_value(defaults=defaults, key=field_name),ancestry=ancestry,@@ -131,6 +204,7 @@ def _record_field(defaults_value: Any,ancestry: tuple[type[BaseModel], ...],collected: dict[str, PathFingerprint],+ field_metadata: Sequence[Any] = (),) -> None:resolved = _strip_annotated(annotation=annotation)nested_model = _as_nested_model(annotation=resolved)@@ -160,6 +234,7 @@ def _record_field(enum_members=_collect_enum_members(annotation=resolved),default=_json_safe(value=defaults_value),open_node=open_value_type is not None,+ constraints=_collect_constraints(annotation=annotation, field_metadata=field_metadata),)if open_value_type is not None:@@ -229,16 +304,18 @@ def _as_open_mapping_value(*, annotation: Any) -> Any:def _render_type(*, annotation: Any) -> str:# Strip at every level, not just the top one: a union member carrying its own constraint# (`Annotated[int, Field(ge=1)] | Literal["unbounded"]`) would otherwise render the- # constraint object into the golden, where a pydantic upgrade could move it.+ # constraint object into the golden, where a pydantic upgrade could move it. The bounds are not+ # lost — `_collect_constraints` records the whitelisted ones under their own field, in a+ # vocabulary that is ours rather than the library's.annotation = _strip_annotated(annotation=annotation)if annotation is Any:return "any"if annotation is types.NoneType:return "none"if _is_union(annotation=annotation):- return " | ".join(_render_type(annotation=member) for member in get_args(annotation))+ return UNION_SEPARATOR.join(_render_type(annotation=member) for member in get_args(annotation))if get_origin(annotation) is Literal:- return "literal"+ return LITERAL_TYPEorigin = get_origin(annotation)if origin is not None:args = get_args(annotation)@@ -259,6 +336,65 @@ def _render_bare(*, annotation: Any) -> str:return name.lower() if isinstance(name, str) else "unknown"+def _collect_constraints(*, annotation: Any, field_metadata: Sequence[Any]) -> dict[ConstraintKind, int | float] | None:+ """The whitelisted bounds on one path's value, merged across the annotation's union members.++ Two sources are read and no others: the field's own metadata, which is where a top-level+ `Field(ge=1)` ends up, and the `Annotated` wrappers reachable through unions — the shape+ `Annotated[int, Field(ge=1)] | Literal["unbounded"]` that the walk would otherwise miss.++ A generic container's arguments are deliberately **not** descended into. A bound inside+ `list[Annotated[int, Field(ge=1)]]` binds the items, not the list, and recording it at the+ list's path would attribute to one path a constraint belonging to another — the value schema+ beneath an open node gets its own `*` record and its own constraints there.+ """+ carriers: list[Any] = list(field_metadata)+ _gather_constraint_carriers(annotation=annotation, carriers=carriers)+ collected: dict[ConstraintKind, int | float] = {}+ _absorb_constraint_carriers(carriers=carriers, collected=collected)+ return {kind: collected[kind] for kind in sorted(collected)} if collected else None+++def _gather_constraint_carriers(*, annotation: Any, carriers: list[Any]) -> None:+ if get_origin(annotation) is Annotated:+ args = get_args(annotation)+ carriers.extend(args[1:])+ _gather_constraint_carriers(annotation=args[0], carriers=carriers)+ return+ if _is_union(annotation=annotation):+ for member in get_args(annotation):+ _gather_constraint_carriers(annotation=member, carriers=carriers)+++def _absorb_constraint_carriers(*, carriers: Sequence[Any], collected: dict[ConstraintKind, int | float]) -> None:+ """Fold every carrier the whitelist recognizes into the projection, and drop the rest silently.++ Dropping is the whole point: a carrier this function does not recognize — a strictness flag, a+ before-validator, a pattern object, a kind a future release invents — must leave no trace in+ the golden, or the gate starts moving for reasons that are not about our schema.+ """+ for carrier in carriers:+ if isinstance(carrier, FieldInfo):+ # `Field(...)` inside an `Annotated` arrives as a `FieldInfo` carrying its own metadata.+ _absorb_constraint_carriers(carriers=carrier.metadata, collected=collected)+ continue+ kind = CONSTRAINT_ATTRIBUTE_BY_CARRIER.get(cast("type[Any]", type(carrier)))+ if kind is None:+ continue+ value = getattr(carrier, kind, None)+ if isinstance(value, bool) or not isinstance(value, (int, float)):+ # A bound expressed over dates, decimals or anything else is outside the projection:+ # it has no stable rendering here and no comparison this gate could make.+ continue+ existing = collected.get(kind)+ if existing is None:+ collected[kind] = value+ elif kind.widest_is_the_lower_value:+ collected[kind] = min(existing, value)+ else:+ collected[kind] = max(existing, value)++def _collect_enum_members(*, annotation: Any) -> list[str] | None:"""Every enumerated spelling reachable in the annotation, sorted and deduplicated.diff --git a/pipelex/migration/goldens.py b/pipelex/migration/goldens.pyindex 8ae09dae7..4803783a9 100644--- a/pipelex/migration/goldens.py+++ b/pipelex/migration/goldens.py@@ -5,6 +5,10 @@ One directory per surface, holding two files per schema version:pipelex/migration/goldens/<surface-id>/[email protected]pipelex/migration/goldens/<surface-id>/[email protected]+plus, for a pre-history entry only, the hand-authored document it migrates from:++ pipelex/migration/goldens/<surface-id>/[email protected]+**The head link tracks; every link below it is frozen.** `update-migration-schemas` rewrites thesnapshot for the surface's *current* version from live sources on every run, and a version bumpsimply leaves the previous version's files behind as history. Freezing the head instead would@@ -27,6 +31,7 @@ from pipelex.migration.fingerprint import SurfaceFingerprintGOLDENS_DIR_NAME = "goldens"FINGERPRINT_STEM = "fingerprint"DEFAULTS_STEM = "defaults"+PRE_HISTORY_STEM = "before"VERSION_MARKER = "@"@@ -42,6 +47,18 @@ def defaults_golden_path(*, migration_dir: Path, surface_id: str, schema_versionreturn goldens_dir(migration_dir=migration_dir, surface_id=surface_id) / f"{DEFAULTS_STEM}{VERSION_MARKER}{schema_version}.toml"+def pre_history_document_path(*, migration_dir: Path, surface_id: str, schema_version: int) -> Path:+ """The hand-authored document a pre-history entry migrates *from*.++ `[email protected]` sits beside the chain and is the one file here that is neither snapshotted nor+ regenerated: a change that predates the first fingerprint has no `defaults@N-1` to start from,+ because there was no snapshot yet, so its author writes the old shape down instead. Nothing+ regenerates it — a regenerated one would describe today's models, which is exactly what it is+ not about.+ """+ return goldens_dir(migration_dir=migration_dir, surface_id=surface_id) / f"{PRE_HISTORY_STEM}{VERSION_MARKER}{schema_version}.toml"++def read_fingerprint_golden(*, migration_dir: Path, surface_id: str, schema_version: int) -> SurfaceFingerprint | None:"""The stored fingerprint for one schema version, or `None` when it has never been snapshotted.diff --git a/pipelex/migration/goldens/pipelex-config/[email protected] b/pipelex/migration/goldens/pipelex-config/[email protected]index 0e235137f..507d2c72f 100644--- a/pipelex/migration/goldens/pipelex-config/[email protected]+++ b/pipelex/migration/goldens/pipelex-config/[email protected]@@ -91,7 +91,10 @@"value_type": "float","required": true,"default": 2.5,- "open_node": false+ "open_node": false,+ "constraints": {+ "gt": 0+ }},"cogt.img_gen_config.img_gen_param_defaults.is_moderated": {"value_type": "bool",@@ -109,7 +112,10 @@"value_type": "int","required": false,"default": 28,- "open_node": false+ "open_node": false,+ "constraints": {+ "gt": 0+ }},"cogt.img_gen_config.img_gen_param_defaults.quality": {"value_type": "enum",@@ -126,7 +132,11 @@"value_type": "int","required": true,"default": 5,- "open_node": false+ "open_node": false,+ "constraints": {+ "ge": 1,+ "le": 6+ }},"cogt.img_gen_config.img_gen_param_defaults.seed": {"value_type": "int | literal",@@ -387,7 +397,11 @@"value_type": "int","required": true,"default": 3,- "open_node": false+ "open_node": false,+ "constraints": {+ "ge": 1,+ "le": 10+ }},"cogt.model_deck_config": {"value_type": "table",@@ -415,7 +429,11 @@"value_type": "int","required": true,"default": 2,- "open_node": false+ "open_node": false,+ "constraints": {+ "ge": 0,+ "le": 10+ }},"migration": {"value_type": "table",@@ -1180,7 +1198,10 @@"unbounded"],"default": 8,- "open_node": false+ "open_node": false,+ "constraints": {+ "ge": 1+ }},"pipelex.reporting_config": {"value_type": "table",@@ -1209,7 +1230,10 @@"value_type": "float","required": true,"default": 1.0,- "open_node": false+ "open_node": false,+ "constraints": {+ "gt": 0+ }},"pipelex.reporting_config.is_generate_cost_report_file_enabled": {"value_type": "bool",diff --git a/pipelex/migration/goldens/telemetry-config/[email protected] b/pipelex/migration/goldens/telemetry-config/[email protected]new file mode 100644index 000000000..23ef55b33--- /dev/null+++ b/pipelex/migration/goldens/telemetry-config/[email protected]@@ -0,0 +1,21 @@+# The telemetry configuration as it was before the ledger existed — hand-authored, and the one+# document in this directory that is neither snapshotted nor regenerated. Entry `telemetry-config@2`+# is marked `pre_history` because the change it describes predates the first fingerprint, so there+# is no `[email protected]` on its far side to start from; this file is what it starts from instead,+# and the transform check verifies the entry by migrating it and looking at what comes out.+#+# It carries every root key that shipped in the flat format across its lifetime, so that every+# operation of the entry is exercised by that check rather than merely declared. A real file of the+# era carried most of these and not all — the shape is faithful, the census is deliberate.++settings_customized = false+telemetry_mode = "anonymous"+host = "https://eu.i.posthog.com"+project_api_key = "phc_example_project_api_key"+respect_dnt = true+redact = ["prompt", "system_prompt", "response", "file_path", "url"]+geoip_enabled = true+dry_mode_enabled = false+verbose_enabled = false+debug = false+user_id = ""diff --git a/pipelex/migration/goldens/telemetry-config/[email protected] b/pipelex/migration/goldens/telemetry-config/[email protected]new file mode 100644index 000000000..bc3f3c406--- /dev/null+++ b/pipelex/migration/goldens/telemetry-config/[email protected]@@ -0,0 +1,33 @@+otlp = []++[custom_posthog]+mode = "off"+endpoint = "https://us.i.posthog.com"+geoip = true+debug = false+redact_properties = []++[custom_posthog.tracing]+enabled = false++[custom_posthog.tracing.capture]+content = false+pipe_codes = false+output_class_names = false++[custom_portkey]+force_debug_enabled = false+force_tracing_enabled = false++[langfuse]+enabled = false++[telemetry_allowed_modes]+ci = false+cli = true+docker = true+fastapi = true+mcp = true+n8n = true+pytest = false+python = falsediff --git a/pipelex/migration/goldens/telemetry-config/[email protected] b/pipelex/migration/goldens/telemetry-config/[email protected]new file mode 100644index 000000000..1a2ebb466--- /dev/null+++ b/pipelex/migration/goldens/telemetry-config/[email protected]@@ -0,0 +1,264 @@+{+ "surface_id": "telemetry-config",+ "schema_version": 2,+ "paths": {+ "custom_portkey": {+ "value_type": "table",+ "required": false,+ "open_node": false+ },+ "custom_portkey.force_debug_enabled": {+ "value_type": "bool",+ "required": false,+ "default": false,+ "open_node": false+ },+ "custom_portkey.force_tracing_enabled": {+ "value_type": "bool",+ "required": false,+ "default": false,+ "open_node": false+ },+ "custom_posthog": {+ "value_type": "table",+ "required": false,+ "open_node": false+ },+ "custom_posthog.api_key": {+ "value_type": "str",+ "required": false,+ "open_node": false+ },+ "custom_posthog.debug": {+ "value_type": "bool",+ "required": false,+ "default": false,+ "open_node": false+ },+ "custom_posthog.endpoint": {+ "value_type": "str",+ "required": false,+ "default": "https://us.i.posthog.com",+ "open_node": false+ },+ "custom_posthog.geoip": {+ "value_type": "bool",+ "required": false,+ "default": true,+ "open_node": false+ },+ "custom_posthog.mode": {+ "value_type": "enum",+ "required": false,+ "enum_members": [+ "anonymous",+ "identified",+ "off"+ ],+ "default": "off",+ "open_node": false+ },+ "custom_posthog.redact_properties": {+ "value_type": "list[str]",+ "required": false,+ "default": [],+ "open_node": false+ },+ "custom_posthog.tracing": {+ "value_type": "table",+ "required": false,+ "open_node": false+ },+ "custom_posthog.tracing.capture": {+ "value_type": "table",+ "required": false,+ "open_node": false+ },+ "custom_posthog.tracing.capture.content": {+ "value_type": "bool",+ "required": false,+ "default": false,+ "open_node": false+ },+ "custom_posthog.tracing.capture.content_max_length": {+ "value_type": "int",+ "required": false,+ "open_node": false+ },+ "custom_posthog.tracing.capture.output_class_names": {+ "value_type": "bool",+ "required": false,+ "default": false,+ "open_node": false+ },+ "custom_posthog.tracing.capture.pipe_codes": {+ "value_type": "bool",+ "required": false,+ "default": false,+ "open_node": false+ },+ "custom_posthog.tracing.enabled": {+ "value_type": "bool",+ "required": false,+ "default": false,+ "open_node": false+ },+ "custom_posthog.user_id": {+ "value_type": "str",+ "required": false,+ "open_node": false+ },+ "langfuse": {+ "value_type": "table",+ "required": false,+ "open_node": false+ },+ "langfuse.enabled": {+ "value_type": "bool",+ "required": false,+ "default": false,+ "open_node": false+ },+ "langfuse.endpoint": {+ "value_type": "str",+ "required": false,+ "open_node": false+ },+ "langfuse.public_key": {+ "value_type": "str",+ "required": false,+ "open_node": false+ },+ "langfuse.secret_key": {+ "value_type": "str",+ "required": false,+ "open_node": false+ },+ "otlp": {+ "value_type": "list[table]",+ "required": false,+ "default": [],+ "open_node": false+ },+ "pipelex_gateway": {+ "value_type": "table",+ "required": false,+ "open_node": false+ },+ "pipelex_gateway.portkey": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "pipelex_gateway.portkey.force_debug_enabled": {+ "value_type": "bool",+ "required": false,+ "open_node": false+ },+ "pipelex_gateway.portkey.force_tracing_enabled": {+ "value_type": "bool",+ "required": false,+ "open_node": false+ },+ "pipelex_gateway.posthog": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "pipelex_gateway.posthog.api_key": {+ "value_type": "str",+ "required": false,+ "open_node": false+ },+ "pipelex_gateway.posthog.debug": {+ "value_type": "bool",+ "required": false,+ "open_node": false+ },+ "pipelex_gateway.posthog.endpoint": {+ "value_type": "str",+ "required": false,+ "open_node": false+ },+ "pipelex_gateway.posthog.geoip": {+ "value_type": "bool",+ "required": false,+ "open_node": false+ },+ "pipelex_gateway.posthog.mode": {+ "value_type": "enum",+ "required": false,+ "enum_members": [+ "anonymous",+ "identified",+ "off"+ ],+ "open_node": false+ },+ "pipelex_gateway.posthog.redact_properties": {+ "value_type": "list[str]",+ "required": false,+ "open_node": false+ },+ "pipelex_gateway.posthog.tracing": {+ "value_type": "table",+ "required": false,+ "open_node": false+ },+ "pipelex_gateway.posthog.tracing.capture": {+ "value_type": "table",+ "required": false,+ "open_node": false+ },+ "pipelex_gateway.posthog.tracing.capture.content": {+ "value_type": "bool",+ "required": false,+ "open_node": false+ },+ "pipelex_gateway.posthog.tracing.capture.content_max_length": {+ "value_type": "int",+ "required": false,+ "open_node": false+ },+ "pipelex_gateway.posthog.tracing.capture.output_class_names": {+ "value_type": "bool",+ "required": false,+ "open_node": false+ },+ "pipelex_gateway.posthog.tracing.capture.pipe_codes": {+ "value_type": "bool",+ "required": false,+ "open_node": false+ },+ "pipelex_gateway.posthog.tracing.enabled": {+ "value_type": "bool",+ "required": false,+ "open_node": false+ },+ "pipelex_gateway.posthog.user_id": {+ "value_type": "str",+ "required": false,+ "open_node": false+ },+ "telemetry_allowed_modes": {+ "value_type": "dict[str, bool]",+ "required": false,+ "default": {+ "ci": false,+ "cli": true,+ "docker": true,+ "fastapi": true,+ "mcp": true,+ "n8n": true,+ "pytest": false,+ "python": false+ },+ "open_node": true+ },+ "telemetry_allowed_modes.*": {+ "value_type": "bool",+ "required": true,+ "open_node": false+ }+ }+}diff --git a/pipelex/migration/ledger.py b/pipelex/migration/ledger.pyindex e389fc148..eef8c4c79 100644--- a/pipelex/migration/ledger.py+++ b/pipelex/migration/ledger.py@@ -12,6 +12,7 @@ See `docs/migration-ledger.md` → "The ledger file"."""from enum import StrEnum+from functools import cachefrom pathlib import Pathfrom typing import Any@@ -106,6 +107,17 @@ class MigrationEntry(BaseModel):raise ValueError(msg)return self+ @model_validator(mode="after")+ def check_pre_history_declares_what_it_removed(self) -> Self:+ if self.pre_history and not self.declared_removed_paths:+ msg = (+ f"entry '{self.id}': a pre-history entry declares the paths it removes — the flag exempts the entry from "+ f"being accounted against a fingerprint diff, and the declaration is what replaces that diff, so an entry "+ f"carrying the flag and declaring nothing is exempt from every accounting there is"+ )+ raise ValueError(msg)+ return self+@model_validator(mode="after")def check_an_op_free_entry_is_unsafe(self) -> Self:if not self.ops and self.safety is MigrationSafety.SAFE:@@ -180,3 +192,19 @@ def load_ledger(*, migration_dir: Path, surface_id: str) -> MigrationLedger:except ValidationError as exc:msg = f"invalid ledger for surface '{surface_id}' at {path}: {exc}"raise MigrationLedgerError(msg) from exc+++@cache+def load_ledger_cached(*, migration_dir: Path, surface_id: str) -> MigrationLedger:+ """`load_ledger`, parsed once per process.++ Replaying a surface over a directory of files must not re-parse the same TOML per file: the+ ledger is checked-in package data that cannot change under a running process, and the parsed+ model is frozen. Use this everywhere that reads a ledger during a run; `load_ledger` itself+ stays uncached for the gates, which are told which directory to read and are asked to see+ what is on disk right now.++ A raise is not cached — `lru_cache` only records successful calls — so a ledger fixed on disk+ between two calls is re-read rather than failing forever.+ """+ return load_ledger(migration_dir=migration_dir, surface_id=surface_id)diff --git a/pipelex/migration/ledger_check.py b/pipelex/migration/ledger_check.pynew file mode 100644index 000000000..5e67e20bf--- /dev/null+++ b/pipelex/migration/ledger_check.py@@ -0,0 +1,587 @@+"""The ledger check — is what the ledger *says* legal, and is replaying it harmless?++This is the half of the migration apparatus that reads nothing but checked-in files: the ledgers,+the golden chain, and the two reference documents. It never fingerprints a live model, and that+restraint is the whole reason it can live in `make agent-check`. A check that read the models+would go red on an ordinary configuration edit with "regenerate the golden" as its remedy — the+fail-regenerate-fail cycle that keeps the coverage gate out of the loop agents run constantly.+Here, every failure is a statement about files the author wrote, and every remedy is to fix one.++The questions, each with its own failure kind:++- **Op legality.** An operation's source must be material some schema version *removed*. One+ addressing a live path would fire on a perfectly valid current file, and replay neutrality —+ the premise that lets us replay the whole ledger over every file forever — would be false.+- **Open-node addressing.** The keys beneath an open mapping belong to the user and are+ unbounded, so no operation may name one; `*` stands for "each of these" and is legal exactly+ at an open node, nowhere else.+- **Remap legality.** A `safe` remap must be provably unable to fire on a current-valid file:+ its target enumerated at the new schema, and every old spelling now outside the member set.+- **Pre-history declarations.** An entry claiming its change predates the first fingerprint must+ declare material no fingerprint records, and may address nothing outside that declaration —+ otherwise the flag, which exempts an entry from being accounted against a diff, would be a way+ to opt out of accounting for a change that has one.+- **Convergence.** Replaying the whole ledger over each reference document must apply nothing+ and return the very bytes it was given.++Plus one refusal that outlives them all: a **reserved path** — anything a ledger entry ever+removed — may not come back, because that would make removed material legal again.++See `docs/migration-ledger.md` → "Legality rules" and "The checks".+"""++from enum import StrEnum+from pathlib import Path++from pydantic import BaseModel, ConfigDict, Field++from pipelex.migration.engine import replay_ledger_over_text+from pipelex.migration.fingerprint import PATH_SEPARATOR, SurfaceFingerprint+from pipelex.migration.goldens import read_fingerprint_golden+from pipelex.migration.ledger import INITIAL_SCHEMA_VERSION, MigrationEntry, MigrationLedger, MigrationSafety, load_ledger+from pipelex.migration.reserved import ReservedRegistry, derive_reserved_registry+from pipelex.migration.surfaces import Surface, SurfaceRegistry+from pipelex.migration.walk import EntryWalk, WalkedOp, op_source_path, walk_entry+from pipelex.suggested_fix import WILDCARD_SEGMENT, MigrationOp, RemapValueOp+++class LedgerIssueKind(StrEnum):+ """Why a ledger failed the check. Distinct kinds so a red gate says which guarantee broke."""++ CHAIN_INCOMPLETE = "chain_incomplete"+ OP_ACTS_ON_LIVE_MATERIAL = "op_acts_on_live_material"+ CONCRETE_KEY_UNDER_OPEN_NODE = "concrete_key_under_open_node"+ WILDCARD_NOT_AT_OPEN_NODE = "wildcard_not_at_open_node"+ ILLEGAL_REMAP = "illegal_remap"+ RESERVED_PATH_REUSED = "reserved_path_reused"+ RESERVED_VALUE_REUSED = "reserved_value_reused"+ PRE_HISTORY_PATH_IS_RECORDED = "pre_history_path_is_recorded"+ CONVERGENCE_BROKEN = "convergence_broken"+++class LedgerIssue(BaseModel):+ model_config = ConfigDict(frozen=True, extra="forbid")++ surface_id: str+ kind: LedgerIssueKind = Field(strict=False)+ message: str+++def check_ledgers(*, registry: SurfaceRegistry, migration_dir: Path) -> list[LedgerIssue]:+ """Run the ledger check over every surface in a registry."""+ issues: list[LedgerIssue] = []+ for surface in registry.surfaces:+ issues.extend(check_ledger(surface=surface, migration_dir=migration_dir))+ return issues+++def check_ledger(*, surface: Surface, migration_dir: Path) -> list[LedgerIssue]:+ """Run the whole ledger check over one surface.++ Raises:+ MigrationLedgerError: the ledger is missing, unparseable, or internally inconsistent —+ which covers the contract's *parses into migration operations* and *versions are+ contiguous and match ids*, both enforced when the file is read.+ MigrationGoldenError: a golden in the chain cannot be read as the link its name claims.+ """+ surface_id = surface.surface_id+ ledger = load_ledger(migration_dir=migration_dir, surface_id=surface_id)+ chain = _read_chain(surface_id=surface_id, ledger=ledger, migration_dir=migration_dir)+ reserved = derive_reserved_registry(surface_id=surface_id, ledger=ledger, migration_dir=migration_dir)++ issues = _check_chain_is_complete(surface_id=surface_id, ledger=ledger, chain=chain)+ issues.extend(_check_entries(surface_id=surface_id, ledger=ledger, chain=chain, reserved=reserved))+ issues.extend(_check_reserved_paths_stay_retired(surface_id=surface_id, ledger=ledger, chain=chain, reserved=reserved))+ issues.extend(_check_convergence(surface=surface, ledger=ledger))+ return issues+++def _read_chain(*, surface_id: str, ledger: MigrationLedger, migration_dir: Path) -> dict[int, SurfaceFingerprint]:+ """Every stored link of the surface's golden chain, keyed by schema version.++ Only what is on disk: a missing link is absent from the mapping rather than substituted, so+ that no check ever compares an entry against a shape nobody snapshotted.+ """+ chain: dict[int, SurfaceFingerprint] = {}+ for schema_version in range(INITIAL_SCHEMA_VERSION, ledger.surface.current_schema_version + 1):+ stored = read_fingerprint_golden(migration_dir=migration_dir, surface_id=surface_id, schema_version=schema_version)+ if stored is not None:+ chain[schema_version] = stored+ return chain+++def _check_chain_is_complete(*, surface_id: str, ledger: MigrationLedger, chain: dict[int, SurfaceFingerprint]) -> list[LedgerIssue]:+ """Say once, loudly, which links are missing — every entry check below needs a pair of them."""+ current_version = ledger.surface.current_schema_version+ missing = [schema_version for schema_version in range(INITIAL_SCHEMA_VERSION, current_version + 1) if schema_version not in chain]+ if not missing:+ return []+ versions = ", ".join(str(schema_version) for schema_version in missing)+ plural = len(missing) > 1+ return [+ LedgerIssue(+ surface_id=surface_id,+ kind=LedgerIssueKind.CHAIN_INCOMPLETE,+ message=(+ f"no fingerprint golden for schema {'versions' if plural else 'version'} {versions}, so the entries that change "+ f"{'those shapes' if plural else 'that shape'} cannot be checked against them — run `make umig` if the current "+ f"version was just bumped; a gap below version {current_version} is a broken chain and the missing link has to "+ f"be restored"+ ),+ )+ ]+++def _check_entries(+ *,+ surface_id: str,+ ledger: MigrationLedger,+ chain: dict[int, SurfaceFingerprint],+ reserved: ReservedRegistry,+) -> list[LedgerIssue]:+ issues: list[LedgerIssue] = []+ for entry in ledger.migration:+ before = chain.get(entry.to_schema_version - 1)+ after = chain.get(entry.to_schema_version)+ if before is None or after is None:+ # The chain check above already named the missing link and its remedy. Checking the+ # entry against a shape nobody snapshotted would blame the entry for that gap.+ continue+ if entry.pre_history:+ issues.extend(_check_pre_history_entry(surface_id=surface_id, entry=entry, chain=chain, before=before, after=after, reserved=reserved))+ continue+ walk = walk_entry(entry=entry, before=before)+ issues.extend(_check_op_legality(surface_id=surface_id, entry=entry, before=before, walk=walk, reserved=reserved))+ issues.extend(_check_remap_legality(surface_id=surface_id, entry=entry, after=after, walk=walk))+ return issues+++def _check_pre_history_entry(+ *,+ surface_id: str,+ entry: MigrationEntry,+ chain: dict[int, SurfaceFingerprint],+ before: SurfaceFingerprint,+ after: SurfaceFingerprint,+ reserved: ReservedRegistry,+) -> list[LedgerIssue]:+ """What a pre-history entry is verified against, since no fingerprint pair describes it.++ The flag says the change predates the first fingerprint, so the entry's own declaration stands+ in for the diff and the checks are stated over that declaration instead:++ - **Every declared path really is invisible to the chain.** A path some fingerprint records is+ material a snapshot describes, so the change that removed it has an ordinary diff and must be+ accounted against it. Were the declaration allowed to name a recorded path, the flag would be+ a way to opt out of accounting entirely — the escape hatch this contract refuses.+ - **Every operation acts on declared material.** The ordinary rule is that a source must be a+ path some version removed, read from the reserved registry; for a pre-history source the+ registry is fed by the declaration, so the same rule holds and the same failure is reported.+ Without it the entry could reach for a live path, and replay neutrality would be false.++ Remap legality is unchanged and is checked against the literal path: a `remap_value` retires a+ *spelling*, so its path survives into the new schema, and none of this entry's renames can have+ moved it — they act on material no fingerprint records.++ The walk is not run at all. It replays an entry over the previous fingerprint's path set, and a+ pre-history entry addresses paths that set never had, so every operation would be reported dead.+ """+ issues = _check_declared_paths_are_invisible_to_the_chain(surface_id=surface_id, entry=entry, chain=chain)+ for op in entry.ops:+ issues.extend(_check_open_node_addressing(surface_id=surface_id, entry=entry, op=op, origin_path=op_source_path(op=op), before=before))+ issues.extend(_check_pre_history_source_is_declared_material(surface_id=surface_id, entry=entry, op=op, reserved=reserved))+ if entry.safety is MigrationSafety.SAFE:+ for op in entry.ops:+ if isinstance(op, RemapValueOp):+ issues.extend(+ _check_one_remap(+ surface_id=surface_id,+ entry=entry,+ after=after,+ final_path=op_source_path(op=op),+ old_values=sorted(op.mapping),+ new_values=sorted(set(op.mapping.values())),+ )+ )+ return issues+++def _check_declared_paths_are_invisible_to_the_chain(+ *,+ surface_id: str,+ entry: MigrationEntry,+ chain: dict[int, SurfaceFingerprint],+) -> list[LedgerIssue]:+ """No declared path may appear in a fingerprint at or below the entry's own version.++ A later version bringing one back is a different failure with a different remedy, and the+ reserved-path check reports it in those terms.+ """+ issues: list[LedgerIssue] = []+ for path in entry.declared_removed_paths:+ recorded_at = sorted(version for version, fingerprint in chain.items() if version <= entry.to_schema_version and path in fingerprint.paths)+ if recorded_at:+ issues.append(+ LedgerIssue(+ surface_id=surface_id,+ kind=LedgerIssueKind.PRE_HISTORY_PATH_IS_RECORDED,+ message=(+ f"entry '{entry.id}' declares '{path}' as removed before the first fingerprint, but schema "+ f"{'versions' if len(recorded_at) > 1 else 'version'} {', '.join(str(version) for version in recorded_at)} "+ f"records it. A pre-history declaration stands in for a diff nobody can observe, so material a snapshot "+ f"does describe has to be accounted against that snapshot like any other change"+ ),+ )+ )+ return issues+++def _check_pre_history_source_is_declared_material(+ *,+ surface_id: str,+ entry: MigrationEntry,+ op: MigrationOp,+ reserved: ReservedRegistry,+) -> list[LedgerIssue]:+ """A pre-history operation's source must be declared material, or lie beneath some.++ Beneath, because a declaration names the shape that retired and an operation may address one+ key inside it — declaring the parent is the honest record of what went away, and enumerating+ every leaf under it would add nothing a reader could act on.+ """+ if isinstance(op, RemapValueOp):+ return []+ source = op_source_path(op=op)+ removed_at = _reserved_at_or_above(reserved=reserved, path=source)+ if removed_at is not None and removed_at <= entry.to_schema_version:+ return []+ return [+ LedgerIssue(+ surface_id=surface_id,+ kind=LedgerIssueKind.OP_ACTS_ON_LIVE_MATERIAL,+ message=(+ f"entry '{entry.id}': {op.kind} acts on '{source}', which no schema version up to {entry.to_schema_version} "+ f"removes and this entry does not declare either — a pre-history entry's declaration is the only record of "+ f"what it may address, so an operation outside it would fire on material nothing retired"+ ),+ )+ ]+++def _reserved_at_or_above(*, reserved: ReservedRegistry, path: str) -> int | None:+ """The version that retired this path, or the nearest ancestor of it that a version retired."""+ segments = path.split(PATH_SEPARATOR)+ for depth in range(len(segments), 0, -1):+ removed_at = reserved.reserved_at(path=PATH_SEPARATOR.join(segments[:depth]))+ if removed_at is not None:+ return removed_at+ return None+++def _check_op_legality(+ *,+ surface_id: str,+ entry: MigrationEntry,+ before: SurfaceFingerprint,+ walk: EntryWalk,+ reserved: ReservedRegistry,+) -> list[LedgerIssue]:+ """What an operation may address: removed material, and never a user's own key."""+ issues: list[LedgerIssue] = []+ for walked, op in zip(walk.walked_ops, entry.ops, strict=True):+ issues.extend(_check_open_node_addressing(surface_id=surface_id, entry=entry, op=op, origin_path=walked.origin_path, before=before))+ issues.extend(_check_source_is_removed_material(surface_id=surface_id, entry=entry, op=op, walked=walked, reserved=reserved))+ return issues+++def _check_source_is_removed_material(+ *,+ surface_id: str,+ entry: MigrationEntry,+ op: MigrationOp,+ walked: WalkedOp,+ reserved: ReservedRegistry,+) -> list[LedgerIssue]:+ """Every structural operation acts on a path some schema version removed, or it is illegal.++ A `remap_value` is exempt by construction: what it retires is an enumerated *spelling*, not+ the path, which survives into the new schema and must — remap legality is what governs it.++ An operation whose source the fingerprint does not record at all is left alone here: it is+ either dead, which the coverage gate reports against the diff it belongs to, or it reaches+ into user key space, which the open-node check above reports in the terms that matter.+ """+ if isinstance(op, RemapValueOp) or not walked.source_was_recorded:+ return []+ removed_at = reserved.reserved_at(path=walked.origin_path)+ if removed_at is not None and removed_at <= entry.to_schema_version:+ return []+ return [+ LedgerIssue(+ surface_id=surface_id,+ kind=LedgerIssueKind.OP_ACTS_ON_LIVE_MATERIAL,+ message=(+ f"entry '{entry.id}': {op.kind} acts on '{walked.origin_path}', which no schema version up to "+ f"{entry.to_schema_version} removes — so it would fire on a valid current file and rewrite material the "+ f"schema still has. An operation may only act on what a schema version retired"+ ),+ )+ ]+++def _check_open_node_addressing(+ *,+ surface_id: str,+ entry: MigrationEntry,+ op: MigrationOp,+ origin_path: str,+ before: SurfaceFingerprint,+) -> list[LedgerIssue]:+ """`*` exactly at an open node, and never a concrete key beneath one.++ The check runs on the source traced back to the previous fingerprint's spelling, so that an+ operation acting inside a table an earlier operation in the same entry renamed is judged+ against the node it really addresses. A pre-history source has no such spelling to trace back+ to and is judged as written, which is the right answer for both halves: no fingerprint records+ the node, so nothing can confirm that a `*` stands at an open one, and nothing claims the keys+ beneath it are the user's either.++ Sources only, as the contract states them. A *destination* landing in user key space needs no+ rule of its own: a concrete key beneath an open node is never a path of any fingerprint, so+ the coverage gate's containment check reports it as a destination the new schema does not+ have — the same defect, named where the author is already looking.+ """+ issues: list[LedgerIssue] = []+ segments = origin_path.split(PATH_SEPARATOR)+ for index, segment in enumerate(segments):+ parent_path = PATH_SEPARATOR.join(segments[:index])+ parent_record = before.paths.get(parent_path) if index else None+ parent_is_open = parent_record is not None and parent_record.open_node+ if segment == WILDCARD_SEGMENT and not parent_is_open:+ parent_label = f"'{parent_path}'" if parent_path else "the document root"+ issues.append(+ LedgerIssue(+ surface_id=surface_id,+ kind=LedgerIssueKind.WILDCARD_NOT_AT_OPEN_NODE,+ message=(+ f"entry '{entry.id}': {op.kind} addresses '{origin_path}', but {parent_label} is not an open "+ f"mapping at schema version {before.schema_version} — '{WILDCARD_SEGMENT}' stands for every key of an "+ f"open node and means nothing anywhere else"+ ),+ )+ )+ elif segment != WILDCARD_SEGMENT and parent_is_open:+ issues.append(+ LedgerIssue(+ surface_id=surface_id,+ kind=LedgerIssueKind.CONCRETE_KEY_UNDER_OPEN_NODE,+ message=(+ f"entry '{entry.id}': {op.kind} addresses '{origin_path}', but the keys under '{parent_path}' are "+ f"the user's and unbounded, so no schema change can remove one — address every entry with "+ f"'{WILDCARD_SEGMENT}' instead of naming '{segment}'"+ ),+ )+ )+ return issues+++def _check_remap_legality(+ *,+ surface_id: str,+ entry: MigrationEntry,+ after: SurfaceFingerprint,+ walk: EntryWalk,+) -> list[LedgerIssue]:+ """A `safe` remap must be provably unable to fire on a current-valid file.++ That holds when the target is enumerated at the new schema and every old spelling in the+ mapping now falls outside its member set. When an old spelling remains legal — or the path is+ a free string, where staleness can never be proven from the schema — the applier cannot tell a+ stale value from a deliberate choice, and the entry has to be `unsafe`.+ """+ if entry.safety is MigrationSafety.UNSAFE:+ return []++ issues: list[LedgerIssue] = []+ for remap in walk.remaps:+ final_path = walk.state.final_path_of_origin(origin=remap.origin_path)+ if final_path is None:+ continue+ issues.extend(+ _check_one_remap(+ surface_id=surface_id,+ entry=entry,+ after=after,+ final_path=final_path,+ old_values=remap.old_values,+ new_values=remap.new_values,+ )+ )+ return issues+++def _check_one_remap(+ *,+ surface_id: str,+ entry: MigrationEntry,+ after: SurfaceFingerprint,+ final_path: str,+ old_values: list[str],+ new_values: list[str],+) -> list[LedgerIssue]:+ """The rule itself, over one remap whose destination path is already resolved.++ Resolved differently by each caller and for the same reason: an ordinary entry's remap may sit+ on a path that entry renamed, so the walk says where it ended up, while a pre-history entry's+ renames act on material no fingerprint records and therefore cannot have moved it.+ """+ record = after.paths.get(final_path)+ if record is None:+ # The path the remap targets does not survive into the new schema; the coverage gate's+ # unaccounted-path or over-deletion check says so in terms the author can act on.+ return []+ member_set = set(record.enum_members or [])+ if not member_set:+ return [+ LedgerIssue(+ surface_id=surface_id,+ kind=LedgerIssueKind.ILLEGAL_REMAP,+ message=(+ f"entry '{entry.id}': remap_value on '{final_path}' is 'safe', but that path is not enumerated at "+ f"schema version {after.schema_version} — staleness cannot be proven from the schema, so the entry must be unsafe"+ ),+ )+ ]++ issues: list[LedgerIssue] = []+ still_legal = sorted(set(old_values) & member_set)+ if still_legal:+ issues.append(+ LedgerIssue(+ surface_id=surface_id,+ kind=LedgerIssueKind.ILLEGAL_REMAP,+ message=(+ f"entry '{entry.id}': remap_value on '{final_path}' is 'safe', but {still_legal} are still legal values at "+ f"schema version {after.schema_version} — a user who chose one deliberately would have it rewritten"+ ),+ )+ )+ unknown_new = sorted(set(new_values) - member_set)+ if unknown_new:+ issues.append(+ LedgerIssue(+ surface_id=surface_id,+ kind=LedgerIssueKind.ILLEGAL_REMAP,+ message=(+ f"entry '{entry.id}': remap_value on '{final_path}' rewrites values to {unknown_new}, which schema version "+ f"{after.schema_version} does not accept — every migrated file would be rejected, with the tool reporting success"+ ),+ )+ )+ return issues+++def _check_reserved_paths_stay_retired(+ *,+ surface_id: str,+ ledger: MigrationLedger,+ chain: dict[int, SurfaceFingerprint],+ reserved: ReservedRegistry,+) -> list[LedgerIssue]:+ """A retired path, and a remapped-away spelling, may never come back.++ Reuse would make removed material legal again on a current file, and every operation that+ retired it would start firing on files that are not stale at all. There is no escape-hatch+ marker: an author who hits the rule picks another name.+ """+ issues: list[LedgerIssue] = []+ for entry in ledger.migration:+ schema_version = entry.to_schema_version+ fingerprint = chain.get(schema_version)+ if fingerprint is None:+ continue+ for path, record in fingerprint.paths.items():+ reserved_at = reserved.reserved_at(path=path)+ if reserved_at is not None and reserved_at < schema_version:+ issues.append(+ LedgerIssue(+ surface_id=surface_id,+ kind=LedgerIssueKind.RESERVED_PATH_REUSED,+ message=(+ f"schema version {schema_version} has '{path}' again, but schema version {reserved_at} retired it — "+ f"a retired path stays retired, because reusing the name makes removed material legal again. "+ f"Pick another name"+ ),+ )+ )+ for member in record.enum_members or []:+ value_reserved_at = reserved.value_reserved_at(path=path, value=member)+ if value_reserved_at is not None and value_reserved_at < schema_version:+ issues.append(+ LedgerIssue(+ surface_id=surface_id,+ kind=LedgerIssueKind.RESERVED_VALUE_REUSED,+ message=(+ f"schema version {schema_version} accepts '{member}' at '{path}' again, but schema version "+ f"{value_reserved_at} remapped that spelling away — every file still carrying it would be "+ f"rewritten on every run. Pick another spelling"+ ),+ )+ )+ return issues+++def _check_convergence(*, surface: Surface, ledger: MigrationLedger) -> list[LedgerIssue]:+ """Replaying the whole ledger over a current document must do nothing, and say nothing.++ Two witnesses of deliberately different shape: the complete reference document, where every+ key is set, and the sparse kit template, where almost none are. An operation that misbehaves+ on an absent key passes over one and fails over the other.++ Byte identity is asserted as well as emptiness, but only for a replay that reported nothing:+ the engine returns the very string it was given when nothing applies, so a document that came+ back merely equal would mean something applied and was not reported — which is the one failure+ a report cannot describe. Where the replay *did* report something, the changed bytes are that+ finding's consequence and saying it twice adds nothing.+ """+ issues: list[LedgerIssue] = []+ for label, text in surface.reference_documents():+ replay = replay_ledger_over_text(ledger=ledger, text=text)+ applied = [step.entry_id for step in replay.steps]+ if applied:+ issues.append(+ LedgerIssue(+ surface_id=surface.surface_id,+ kind=LedgerIssueKind.CONVERGENCE_BROKEN,+ message=(+ f"replaying the ledger over the {label} applies {applied} — a document already at schema version "+ f"{ledger.surface.current_schema_version} must come through untouched, or every user's current file "+ f"is rewritten on every run"+ ),+ )+ )+ blocked = [blocked_entry.entry_id for blocked_entry in replay.blocked]+ if blocked:+ issues.append(+ LedgerIssue(+ surface_id=surface.surface_id,+ kind=LedgerIssueKind.CONVERGENCE_BROKEN,+ message=(+ f"replaying the ledger over the {label} reports {blocked} as blocked — a document already at the "+ f"current schema version must give an entry nothing to say, or every user is warned about a file "+ f"that is perfectly current"+ ),+ )+ )+ if not applied and not blocked and replay.text is not text:+ issues.append(+ LedgerIssue(+ surface_id=surface.surface_id,+ kind=LedgerIssueKind.CONVERGENCE_BROKEN,+ message=(+ f"replaying the ledger over the {label} returned a re-serialized document rather than the bytes it was "+ f"given — something applied without being reported, which is the one failure the plan cannot describe"+ ),+ )+ )+ return issuesdiff --git a/pipelex/migration/ledgers/telemetry-config.toml b/pipelex/migration/ledgers/telemetry-config.tomlindex 4bdb983c7..70a5e0b14 100644--- a/pipelex/migration/ledgers/telemetry-config.toml+++ b/pipelex/migration/ledgers/telemetry-config.toml@@ -7,5 +7,117 @@ id = "telemetry-config"title = "Telemetry destinations and capture settings"base_file = "telemetry.toml"tier_glob = "telemetry_*.toml"-current_schema_version = 1+current_schema_version = 2min_supported_schema_version = 0++[[migration]]+id = "telemetry-config@2"+to_schema_version = 2+introduced_in = "0.46.0"+breaking = true+safety = "safe"+pre_history = true+title = "Nest the flat telemetry settings under [custom_posthog]"+description = """+The first telemetry configuration was a flat file: `telemetry_mode`, `host` and `project_api_key` \+sat at the document root, because PostHog was the only destination there was. Adding Langfuse and \+OTLP made a section per destination necessary, and the PostHog settings moved into \+`[custom_posthog]` — named for *your* PostHog, to distinguish it from the Gateway's. This entry \+carries a file written before that move onto the current shape.+"""+guidance = """+Four settings of the flat format have no counterpart today and are dropped rather than moved:++- `respect_dnt` — `DO_NOT_TRACK=1` is now honoured unconditionally, so there is nothing to opt out of.+- `dry_mode_enabled` and `verbose_enabled` — replaced by `debug`, which travels.+- `settings_customized` — a marker the first-run prompt wrote; there is no first-run telemetry prompt any more.++If you had turned `respect_dnt` off, nothing restores that behaviour: honouring `DO_NOT_TRACK` is+not configurable.+"""++# Declared rather than derived: this change predates the first fingerprint, so no snapshot shows+# these paths going away and there is no diff to account the entry against. The hand-authored+# document the operations below are verified against is goldens/telemetry-config/[email protected].+declared_removed_paths = [+ "telemetry_mode",+ "host",+ "project_api_key",+ "user_id",+ "geoip_enabled",+ "debug",+ "redact",+ "settings_customized",+ "respect_dnt",+ "dry_mode_enabled",+ "verbose_enabled",+]++[[migration.ops]] # the destination table does not exist in a flat file: this first move creates it+kind = "move_key"+table_path = []+key = "telemetry_mode"+new_table_path = ["custom_posthog"]+new_key = "mode"++[[migration.ops]]+kind = "move_key"+table_path = []+key = "host"+new_table_path = ["custom_posthog"]+new_key = "endpoint"++[[migration.ops]]+kind = "move_key"+table_path = []+key = "project_api_key"+new_table_path = ["custom_posthog"]+new_key = "api_key"++[[migration.ops]]+kind = "move_key"+table_path = []+key = "user_id"+new_table_path = ["custom_posthog"]+new_key = "user_id"++[[migration.ops]]+kind = "move_key"+table_path = []+key = "geoip_enabled"+new_table_path = ["custom_posthog"]+new_key = "geoip"++[[migration.ops]]+kind = "move_key"+table_path = []+key = "debug"+new_table_path = ["custom_posthog"]+new_key = "debug"++[[migration.ops]]+kind = "move_key"+table_path = []+key = "redact"+new_table_path = ["custom_posthog"]+new_key = "redact_properties"++[[migration.ops]] # the four settings the current shape has no home for+kind = "delete_key"+table_path = []+key = "settings_customized"++[[migration.ops]]+kind = "delete_key"+table_path = []+key = "respect_dnt"++[[migration.ops]]+kind = "delete_key"+table_path = []+key = "dry_mode_enabled"++[[migration.ops]]+kind = "delete_key"+table_path = []+key = "verbose_enabled"diff --git a/pipelex/migration/narrowing.py b/pipelex/migration/narrowing.pynew file mode 100644index 000000000..ee62153bb--- /dev/null+++ b/pipelex/migration/narrowing.py@@ -0,0 +1,182 @@+"""Value-domain narrowing — the schema change that keeps every path and still breaks every file.++A change can leave the path set untouched, leave every enumerated spelling in place, and still+make a user's valid file stop validating: `int` becomes `str`, a free string becomes an enum, a+bound moves from `ge=1` to `ge=5`. Read as a fingerprint diff those are additive-looking — nothing+was removed — so without this module the gate would ask for a golden regeneration, demand no+version bump and no entry, and the next boot would reject a file with a green gate behind it.++No operation in the vocabulary can repair an out-of-domain value, which is why the remedies this+names are the only two there are: an `unsafe` entry, which is reported to the user and never+applied, or a `remap_value` where the old spellings can be enumerated and the new schema's member+set makes the remap legal. That is a narrower set of remedies than a removal has, and it is the+honest one.++**What this cannot see.** Domain narrowing expressed in a *validator* — a non-empty check, a+cross-field invariant, a completeness rule over user-supplied keys — is invisible to any+projection of the annotations, so it stays the author's responsibility and the gate does not claim+it. See `docs/migration-ledger.md` → "The fingerprint" for the live instances.++A lost enumerated spelling is a narrowing too, but it is not `describe_narrowing`'s to report: the+coverage gate compares member sets by origin, following the entry's own renames, which nothing here+could do. What this module owns of it is the other direction — `lost_enumerated_spellings` decides+which of the old spellings really stopped being accepted, so that an enum relaxed into a free+string is read as the widening it is instead of as the loss of every member it had.+"""++from fractions import Fraction++from pipelex.migration.fingerprint import ENUM_TYPE, LITERAL_TYPE, STRING_TYPE, UNION_SEPARATOR, ConstraintKind, PathFingerprint++_LOWER_VALUE_BOUND_KINDS: tuple[tuple[ConstraintKind, bool], ...] = ((ConstraintKind.GT, True), (ConstraintKind.GE, False))+_UPPER_VALUE_BOUND_KINDS: tuple[tuple[ConstraintKind, bool], ...] = ((ConstraintKind.LT, True), (ConstraintKind.LE, False))+_LOWER_LENGTH_BOUND_KINDS: tuple[tuple[ConstraintKind, bool], ...] = ((ConstraintKind.MIN_LENGTH, False),)+_UPPER_LENGTH_BOUND_KINDS: tuple[tuple[ConstraintKind, bool], ...] = ((ConstraintKind.MAX_LENGTH, False),)+_STRING_TYPED_MEMBERS: frozenset[str] = frozenset({STRING_TYPE, ENUM_TYPE, LITERAL_TYPE})+++def describe_narrowing(*, before: PathFingerprint, after: PathFingerprint, remapped: bool = False) -> list[str]:+ """Why the values this path accepts are fewer than they were — empty when they are not.++ Every reason is phrased so that it can be read on its own in a gate's output, because that is+ where it lands: the author sees the path and needs to know which half of the record moved.++ ``remapped`` is set when the entry carries a ``remap_value`` on this path. A remap rewrites+ string values, so it answers for a lost string-typed member — ``str``, ``enum``, ``literal`` —+ and for nothing else: a number the old type accepted and the new one does not (``int | literal``+ becoming ``literal``) is a value no mapping reaches, and so is a tightened bound. Both must+ still be reported, or they ride under the remap into a ``safe`` entry.+ """+ reasons: list[str] = []+ exempt = _STRING_TYPED_MEMBERS if remapped else frozenset[str]()+ if before.value_type != after.value_type and not _is_type_widening(before=before.value_type, after=after.value_type, exempt=exempt):+ reasons.append(f"its type went from '{before.value_type}' to '{after.value_type}'")+ reasons.extend(_describe_tightenings(before=before.constraints or {}, after=after.constraints or {}))+ return reasons+++def lost_enumerated_spellings(*, before: PathFingerprint, after: PathFingerprint) -> list[str]:+ """The enumerated spellings the new schema no longer accepts at this path.++ A raw set difference is wrong in one common case: an enumerated type relaxed into a free string+ records no members afterwards, so every spelling it had reads as lost, while in truth every one+ of them still validates. That change is a widening and must ask for nothing — a bump demanded+ there is a gate crying wolf on a change no user's file notices.++ The exemption is exactly that case — the new type admits `str` — and not "the type did not+ narrow": `Literal['a']` to `Literal[1]` renders the same type on both sides and records no+ members after, and every file carrying `'a'` stops validating.+ """+ if not before.enum_members:+ return []+ if not after.enum_members and STRING_TYPE in _union_members(rendered=after.value_type):+ return []+ return sorted(set(before.enum_members) - set(after.enum_members or []))+++def _is_type_widening(*, before: str, after: str, exempt: frozenset[str]) -> bool:+ """Whether every value the old type accepted the new one still accepts.++ Two shapes qualify, and nothing else does. A union that keeps its members and gains more is a+ widening, member by member — a comparison of the whole rendered string would call `int` to+ `int | str` a change and demand a bump for it. And an enumerated type becoming `str` is a+ widening, because the enumerated spellings are strings and `str` accepts them all; the reverse+ is the narrowing this exists to catch. An ``exempt`` old member is one something else answers+ for, and is not asked to be absorbed.+ """+ after_members = _union_members(rendered=after)+ return all(_is_member_absorbed(member=member, after_members=after_members) for member in _union_members(rendered=before) - exempt)+++def _is_member_absorbed(*, member: str, after_members: set[str]) -> bool:+ if member in after_members:+ return True+ return member in {ENUM_TYPE, LITERAL_TYPE} and STRING_TYPE in after_members+++def _union_members(*, rendered: str) -> set[str]:+ """The top-level members of a rendered type.++ Splitting has to respect brackets: `list[int | str]` is one member, not two, and treating it+ as two would let a genuine narrowing inside a container read as a widening.+ """+ members: set[str] = set()+ depth = 0+ current = ""+ for character in rendered:+ if character == "[":+ depth += 1+ elif character == "]":+ depth -= 1+ current += character+ if depth == 0 and current.endswith(UNION_SEPARATOR):+ members.add(current[: -len(UNION_SEPARATOR)])+ current = ""+ members.add(current)+ return members+++def _describe_tightenings(*, before: dict[ConstraintKind, int | float], after: dict[ConstraintKind, int | float]) -> list[str]:+ """Every bound family whose new form admits fewer values than its old one.++ A bound that appears where there was none is a tightening; one that disappears is a widening+ and says nothing. The `gt`/`ge` pair is compared as a single lower bound rather than key by+ key, so swapping `gt=0` for `ge=0` reads as the widening it is instead of one key vanishing+ and another appearing.+ """+ reasons: list[str] = []+ for label, kinds, sign in (+ ("lower bound", _LOWER_VALUE_BOUND_KINDS, 1.0),+ ("upper bound", _UPPER_VALUE_BOUND_KINDS, -1.0),+ ("minimum length", _LOWER_LENGTH_BOUND_KINDS, 1.0),+ ("maximum length", _UPPER_LENGTH_BOUND_KINDS, -1.0),+ ):+ before_bound = _strictest_bound(constraints=before, kinds=kinds, sign=sign)+ after_bound = _strictest_bound(constraints=after, kinds=kinds, sign=sign)+ if after_bound is not None and (before_bound is None or after_bound > before_bound):+ reasons.append(+ f"its {label} tightened from {_render_bounds(constraints=before, kinds=kinds)} to {_render_bounds(constraints=after, kinds=kinds)}"+ )+ reasons.extend(_describe_multiple_of(before=before, after=after))+ return reasons+++def _strictest_bound(+ *,+ constraints: dict[ConstraintKind, int | float],+ kinds: tuple[tuple[ConstraintKind, bool], ...],+ sign: float,+) -> tuple[float, bool] | None:+ """The binding constraint of one family, as a key that sorts by strictness — `None` for none.++ The sign flips an upper bound so that both directions sort the same way: a *higher* threshold+ binds a lower bound harder, a *lower* one binds an upper bound harder, and negating the second+ makes `max` mean "strictest" in both. Exclusivity breaks the tie, since `gt=0` admits one fewer+ value than `ge=0` at the same threshold.+ """+ keys = [(sign * constraints[kind], exclusive) for kind, exclusive in kinds if kind in constraints]+ return max(keys) if keys else None+++def _render_bounds(*, constraints: dict[ConstraintKind, int | float], kinds: tuple[tuple[ConstraintKind, bool], ...]) -> str:+ present = [f"{kind}={constraints[kind]}" for kind, _ in kinds if kind in constraints]+ return ", ".join(present) if present else "unbounded"+++def _describe_multiple_of(*, before: dict[ConstraintKind, int | float], after: dict[ConstraintKind, int | float]) -> list[str]:+ """`multiple_of` narrows unless the new step divides the old one.++ Multiples of four are a subset of multiples of two, so going from two to four is a tightening+ and going the other way is a widening. Steps that divide neither way — two to three — share+ only their common multiples, so most old values stop validating: a tightening, and reported as+ one rather than waved through for not being comparable.+ """+ before_step = before.get(ConstraintKind.MULTIPLE_OF)+ after_step = after.get(ConstraintKind.MULTIPLE_OF)+ if after_step is None or after_step == 0:+ return []+ # Divisibility is decided over exact rationals: `0.3 % 0.1` is not zero in binary floating+ # point, and a relaxed step must not read as a tightened one.+ if before_step is not None and Fraction(str(before_step)) % Fraction(str(after_step)) == 0:+ return []+ return [f"its step tightened from {'unbounded' if before_step is None else f'multiple_of={before_step}'} to multiple_of={after_step}"]diff --git a/pipelex/migration/plan.py b/pipelex/migration/plan.pynew file mode 100644index 000000000..689cea0ac--- /dev/null+++ b/pipelex/migration/plan.py@@ -0,0 +1,146 @@+"""What a migration run reports — one plan per file, one report per surface walk.++There is no `from_version` and no trusted-version concept anywhere in these models, and that is+the design rather than an omission: nothing skips on a version record, so nothing needs one. A+replay walks every entry while usually changing nothing, and the report renders what *this run*+observed and did — which it knows exactly — rather than what some earlier run claims to have done.++Two different things can be blocked and the models keep them apart. An **entry** is blocked when+it cannot be applied: it is `unsafe`, or one of its operations came back `CONFLICT`. It lands in+`blocked[]` with its guidance while the rest of the file's entries proceed. A **file** is blocked+when it cannot be processed at all — unparseable, unwritable, or changed on disk between the read+and the write — and that reason sits on the plan itself, where it never stops a sibling file.++> **No value read from a user's file is ever rendered.** Paths, operation kinds and+> ledger-supplied values carry everything these models need to say. That is a mechanical rule+> rather than a list of credential-shaped key names, because such a list is a guess that+> eventually misses one.++See `docs/migration-ledger.md` → "What the engine reports".+"""++from enum import StrEnum+from pathlib import Path++from pydantic import BaseModel, ConfigDict, Field++from pipelex.migration.ledger import MigrationSafety+from pipelex.suggested_fix import MigrationOp+++class MigrationStep(BaseModel):+ """One ledger entry that changed this file, and the operations of it that fired."""++ model_config = ConfigDict(frozen=True, extra="forbid")++ entry_id: str+ to_schema_version: int+ title: str+ description: str+ """The entry's release-note sentence — ledger text, never anything read from the file."""++ breaking: bool+ safety: MigrationSafety = Field(strict=False)+ applied_ops: list[MigrationOp]+ """Only the operations that actually applied. Under always-replay most operations skip, and+ listing the skipped ones would bury the change in noise."""+++class BlockedEntryReason(StrEnum):+ """Why an entry could not be applied to this file."""++ UNSAFE = "unsafe"+ """The entry is `unsafe`: the applier cannot tell a stale value from a deliberate choice."""++ CONFLICT = "conflict"+ """An operation's destination is already occupied — typically because the user hand-fixed+ part of their file — so applying it would choose on their behalf."""+++class BlockedEntry(BaseModel):+ """An entry that this file needs and the run would not apply."""++ model_config = ConfigDict(frozen=True, extra="forbid")++ entry_id: str+ to_schema_version: int+ reason: BlockedEntryReason = Field(strict=False)+ detail: str+ """Why, in terms of paths and operation kinds only."""++ guidance: str | None = None+ """The entry's own agent-facing Markdown, when it carries any."""++ applied_ops: list[MigrationOp] = Field(default_factory=list[MigrationOp])+ """Operations of this entry that did apply before the conflict was found. Always empty for+ an `unsafe` entry, which is rehearsed on a copy and never written."""+++class FileBlockedReason(StrEnum):+ """Why a file could not be processed at all."""++ UNPARSEABLE = "unparseable"+ UNWRITABLE = "unwritable"+ CHANGED_DURING_RUN = "changed_during_run"+++class UnexplainedPath(BaseModel):+ """A path the current schema does not know and no ledger entry removes.++ Populated by the downgrade diagnosis, which lands with the commands: the file is either+ carrying a typo or was written by a newer pipelex than the one running.+ """++ model_config = ConfigDict(frozen=True, extra="forbid")++ path: str+ note: str+++class MigrationPlan(BaseModel):+ """What one file needs, and what this run did to it."""++ model_config = ConfigDict(frozen=True, extra="forbid")++ surface_id: str+ file_path: Path++ blocked_reason: FileBlockedReason | None = Field(default=None, strict=False)+ """Set when the file itself could not be processed. A blocked file never stops its siblings."""++ blocked_detail: str | None = None++ steps: list[MigrationStep] = Field(default_factory=list[MigrationStep])+ blocked: list[BlockedEntry] = Field(default_factory=list[BlockedEntry])+ unexplained: list[UnexplainedPath] = Field(default_factory=list[UnexplainedPath])++ backup_path: Path | None = None+ """Where the pre-migration copy of this file was written — when the run wrote the file, and also+ when a write failed leaving the file's state uncertain, because then the copy is kept for the user."""++ was_written: bool = False++ @property+ def is_clean(self) -> bool:+ """Whether the file needs nothing: nothing applied, nothing blocked, nothing unexplained."""+ return not (self.steps or self.blocked or self.unexplained or self.blocked_reason)+++class MigrationReport(BaseModel):+ """Every file one run visited."""++ model_config = ConfigDict(frozen=True, extra="forbid")++ plans: list[MigrationPlan] = Field(default_factory=list[MigrationPlan])++ @property+ def written_plans(self) -> list[MigrationPlan]:+ return [plan for plan in self.plans if plan.was_written]++ @property+ def blocked_plans(self) -> list[MigrationPlan]:+ return [plan for plan in self.plans if plan.blocked_reason is not None or plan.blocked]++ @property+ def is_clean(self) -> bool:+ return all(plan.is_clean for plan in self.plans)diff --git a/pipelex/migration/reserved.py b/pipelex/migration/reserved.pynew file mode 100644index 000000000..3218e5e3c--- /dev/null+++ b/pipelex/migration/reserved.py@@ -0,0 +1,89 @@+"""The reserved-path registry — every path and enumerated spelling a ledger has ever removed.++**The registry is derived, not stored.** Every path a schema version removed is `fingerprint@N-1`+minus `fingerprint@N`, so the whole registry is a walk of the golden chain, plus the declarations+of any pre-history entry. A stored copy would be a second source of truth that could disagree with+the chain it summarizes, and the only way to check it would be to recompute it — at which point+the stored copy has no reader left.++Reuse is refused outright, with no escape-hatch marker: reintroducing a removed path would make it+legal again on a current file, and premise one of replay neutrality — *every operation's+precondition mentions only removed material* — would be false. An author who hits the rule picks+another name.++The registry is diagnostic as much as preventive: when a convergence check or a transform golden+fails, it is what turns the failure into a sentence naming the path and the schema version that+reserved it.++See `docs/migration-ledger.md` → "Reserved paths and names".+"""++from pathlib import Path++from pydantic import BaseModel, ConfigDict, Field++from pipelex.migration.goldens import read_fingerprint_golden+from pipelex.migration.ledger import MigrationLedger+from pipelex.migration.walk import op_source_path+from pipelex.suggested_fix import RemapValueOp+++class ReservedRegistry(BaseModel):+ """What a surface's ledger has retired, and at which schema version."""++ model_config = ConfigDict(frozen=True, extra="forbid")++ surface_id: str++ version_by_path: dict[str, int] = Field(default_factory=dict[str, int])+ """Every removed path, mapped to the schema version that removed it."""++ version_by_value: dict[str, int] = Field(default_factory=dict[str, int])+ """Every remapped-away enumerated spelling, keyed as `<path>=<value>`."""++ def reserved_at(self, *, path: str) -> int | None:+ return self.version_by_path.get(path)++ def is_reserved(self, *, path: str) -> bool:+ return path in self.version_by_path++ def value_reserved_at(self, *, path: str, value: str) -> int | None:+ return self.version_by_value.get(_value_key(path=path, value=value))+++def _value_key(*, path: str, value: str) -> str:+ return f"{path}={value}"+++def derive_reserved_registry(*, surface_id: str, ledger: MigrationLedger, migration_dir: Path) -> ReservedRegistry:+ """Walk the golden chain and the ledger, and record what each version retired.++ A link whose `before` golden is missing contributes nothing rather than raising: a broken chain+ is the coverage check's finding to report, in the terms an author can act on, and this function+ exists to *describe* the ledger rather than to judge it.+ """+ version_by_path: dict[str, int] = {}+ version_by_value: dict[str, int] = {}++ for entry in ledger.migration:+ version = entry.to_schema_version+ if entry.pre_history:+ # No fingerprint pair describes a pre-history diff — that is what the flag means — so+ # the entry's own declaration is the record: `check-ledger` verifies the operations stay+ # inside it, and the transform check migrates the hand-authored `before` document.+ for path in entry.declared_removed_paths:+ version_by_path.setdefault(path, version)+ else:+ before = read_fingerprint_golden(migration_dir=migration_dir, surface_id=surface_id, schema_version=version - 1)+ after = read_fingerprint_golden(migration_dir=migration_dir, surface_id=surface_id, schema_version=version)+ if before is not None and after is not None:+ for path in sorted(before.path_names() - after.path_names()):+ version_by_path.setdefault(path, version)++ for op in entry.ops:+ if isinstance(op, RemapValueOp):+ path = op_source_path(op=op)+ for old_value in sorted(op.mapping):+ version_by_value.setdefault(_value_key(path=path, value=old_value), version)++ return ReservedRegistry(surface_id=surface_id, version_by_path=version_by_path, version_by_value=version_by_value)diff --git a/pipelex/migration/runner.py b/pipelex/migration/runner.pynew file mode 100644index 000000000..819ccb145--- /dev/null+++ b/pipelex/migration/runner.py@@ -0,0 +1,189 @@+"""Running a migration over real files — read, replay, back up, write, report.++The scope here is **per file**, and that is the difference from the `.mthds` fix loop this code+shares its transaction primitives with. A bundle's files only make sense together, so that loop+commits a round all-or-nothing; a surface's files are independent user documents, so a file that+is unparseable, unwritable or changed during the run is reported as blocked while every sibling is+migrated and reported normally.++Nothing here decides *whether* to migrate. There is no version resolution and no state stamp: the+engine replays the whole ledger over every file and the applier skips whatever is already done.++See `docs/migration-ledger.md` → "Applying" and "Per-file transactions".+"""++from datetime import UTC, datetime+from pathlib import Path++from tomlkit.exceptions import TOMLKitError++from pipelex import log+from pipelex.migration.backup import prune_backups_except, write_backup+from pipelex.migration.engine import replay_ledger_over_text+from pipelex.migration.ledger import MigrationLedger, load_ledger_cached+from pipelex.migration.plan import FileBlockedReason, MigrationPlan, MigrationReport+from pipelex.migration.surfaces import Surface, SurfaceRegistry+from pipelex.pipeline.exceptions import FixTransactionError, FixWriteConflictError+from pipelex.pipeline.fixes.file_transaction import FileSnapshot, PendingFileUpdate, commit_file_updates, read_file_snapshot+++def migrate_file(*, surface: Surface, ledger: MigrationLedger, file_path: Path, dry_run: bool, moment: datetime) -> MigrationPlan:+ """Replay one surface's ledger over one file, writing it only when something applied."""+ try:+ snapshot = read_file_snapshot(file_path)+ except FileNotFoundError:+ return _blocked_plan(+ surface_id=surface.surface_id,+ file_path=file_path,+ reason=FileBlockedReason.CHANGED_DURING_RUN,+ detail="the file was removed while the migration was being prepared",+ )+ except OSError as exc:+ return _blocked_plan(+ surface_id=surface.surface_id,+ file_path=file_path,+ reason=FileBlockedReason.UNWRITABLE,+ detail=f"the file could not be read: {exc.strerror or exc}",+ )++ try:+ text = snapshot.content.decode("utf-8")+ except UnicodeDecodeError:+ return _blocked_plan(+ surface_id=surface.surface_id,+ file_path=file_path,+ reason=FileBlockedReason.UNPARSEABLE,+ detail="the file is not valid UTF-8",+ )++ try:+ replay = replay_ledger_over_text(ledger=ledger, text=text)+ except TOMLKitError as exc:+ return _blocked_plan(+ surface_id=surface.surface_id,+ file_path=file_path,+ reason=FileBlockedReason.UNPARSEABLE,+ # A parse error names a position and a syntax expectation, never a value.+ detail=f"the file is not valid TOML: {exc}",+ )++ plan = MigrationPlan(+ surface_id=surface.surface_id,+ file_path=file_path,+ steps=replay.steps,+ blocked=replay.blocked,+ )+ if not replay.did_change_document or dry_run:+ return plan+ return _write_migrated_file(plan=plan, snapshot=snapshot, new_content=replay.text, moment=moment)+++def _write_migrated_file(*, plan: MigrationPlan, snapshot: FileSnapshot, new_content: str, moment: datetime) -> MigrationPlan:+ """Back the file up, replace it atomically, then prune the older backups.++ The order is the point. Backing up first means there is never a moment with a rewritten file+ and no copy of the original; pruning last means there is never a moment with no backup at all.+ A commit that fails before touching the file takes its own fresh backup with it, so a failed run+ leaves the directory exactly as it found it. The one exception is a commit whose outcome the+ transaction cannot vouch for: then the backup is the only copy whose provenance is certain, and+ it stays, named in the report.++ Nothing raised in here escapes to the caller: this is the per-file boundary, and an exception+ crossing it would abort every sibling file after this one — the one thing the per-file scope+ exists to rule out. Whatever goes wrong lands on this file's plan, or, once the file is+ written, in a warning, because a written file is written whatever happens to the housekeeping+ around it.+ """+ try:+ backup_path = write_backup(snapshot=snapshot, moment=moment)+ except OSError as exc:+ return plan.model_copy(+ update={+ "blocked_reason": FileBlockedReason.UNWRITABLE,+ "blocked_detail": f"the backup could not be written: {exc.strerror or exc}",+ }+ )++ try:+ commit_file_updates([PendingFileUpdate(snapshot=snapshot, new_content=new_content)])+ except FixWriteConflictError as exc:+ # The primitive refused before touching the target, so the directory is as it was — minus+ # the copy this run just made, which has nothing to back up.+ _discard_backup(backup_path=backup_path)+ return plan.model_copy(update={"blocked_reason": FileBlockedReason.CHANGED_DURING_RUN, "blocked_detail": str(exc)})+ except OSError as exc:+ _discard_backup(backup_path=backup_path)+ return plan.model_copy(+ update={"blocked_reason": FileBlockedReason.UNWRITABLE, "blocked_detail": f"the file could not be written: {exc.strerror or exc}"}+ )+ except FixTransactionError as exc:+ # The primitive raises this when it cannot vouch for the state it leaves behind — a rollback+ # that did not complete, or a replacement that landed but whose temp files could not be+ # removed. The exception cannot say which; the file can, so ask it. Whatever the answer,+ # the backup stays: it is the one copy of the original whose provenance is certain, and a+ # state the primitive cannot vouch for is exactly the state a backup exists for.+ if not _carries(path=snapshot.path, content=new_content):+ return plan.model_copy(+ update={+ "blocked_reason": FileBlockedReason.UNWRITABLE,+ "blocked_detail": f"the file could not be written, and its state is uncertain — the original is kept at '{backup_path}': {exc}",+ "backup_path": backup_path,+ }+ )+ log.warning(f"'{snapshot.path}' was migrated, but the write left something behind: {exc}")++ try:+ prune_backups_except(path=snapshot.path, keep=backup_path)+ except OSError as exc:+ # An older backup that would not go is a housekeeping failure on a file that is already+ # migrated and already backed up. Not the plan's to report as a failure of the file.+ log.warning(+ f"'{snapshot.path}' was migrated and backed up to '{backup_path}', but an older backup could not be pruned: {exc.strerror or exc}"+ )+ return plan.model_copy(update={"backup_path": backup_path, "was_written": True})+++def _discard_backup(*, backup_path: Path) -> None:+ """Remove the copy this run made of a file it then did not rewrite.++ A copy that will not go is a stray file beside an untouched original — worth a warning, never+ worth an exception crossing the per-file boundary and aborting the siblings.+ """+ try:+ backup_path.unlink(missing_ok=True)+ except OSError as exc:+ log.warning(f"the backup '{backup_path}' was made for a write that did not happen and could not be removed: {exc.strerror or exc}")+++def _carries(*, path: Path, content: str) -> bool:+ """Whether the file on disk holds exactly this text — the one question a failed commit leaves open."""+ try:+ return path.read_bytes() == content.encode("utf-8")+ except OSError:+ return False+++def _blocked_plan(*, surface_id: str, file_path: Path, reason: FileBlockedReason, detail: str) -> MigrationPlan:+ return MigrationPlan(surface_id=surface_id, file_path=file_path, blocked_reason=reason, blocked_detail=detail)+++def migrate_directories(+ *,+ registry: SurfaceRegistry,+ migration_dir: Path,+ config_dirs: list[Path],+ dry_run: bool,+ moment: datetime | None = None,+) -> MigrationReport:+ """Migrate every claimed file in every given configuration directory.++ The walk is over the directories a caller names — in practice the global `~/.pipelex/` and the+ project's `.pipelex/`, and only those. A directory that does not exist is skipped.+ """+ stamp = moment if moment is not None else datetime.now(UTC)+ plans: list[MigrationPlan] = []+ for directory in config_dirs:+ for surface, file_path in registry.files_by_surface_in_directory(directory=directory):+ ledger = load_ledger_cached(migration_dir=migration_dir, surface_id=surface.surface_id)+ plans.append(migrate_file(surface=surface, ledger=ledger, file_path=file_path, dry_run=dry_run, moment=stamp))+ return MigrationReport(plans=plans)diff --git a/pipelex/migration/surfaces.py b/pipelex/migration/surfaces.pyindex e573a0872..8cde71cc9 100644--- a/pipelex/migration/surfaces.py+++ b/pipelex/migration/surfaces.py@@ -12,6 +12,7 @@ configuration models"."""from enum import StrEnum+from fnmatch import fnmatchfrom pathlib import Pathfrom typing import Any, cast@@ -30,6 +31,7 @@ PIPELEX_CONFIG_FILE_NAME = "pipelex.toml"# The package directory, `pipelex/` — this module sits one level under it._PACKAGE_ROOT = Path(__file__).resolve().parent.parent+_KIT_CONFIGS_DIR = _PACKAGE_ROOT / "kit" / "configs"def packaged_migration_dir() -> Path:@@ -64,6 +66,15 @@ class Surface(BaseModel):defaults_layer_kind: DefaultsLayerKindpackaged_document_path: Path | None = None+ kit_template_path: Path | None = None+ """The sparse starter file `pipelex init` copies into a configuration directory.++ A **convergence and neutrality witness only**, never a snapshot: it is read live, so there is+ no fixture to maintain and no golden to go stale. It earns its place beside the complete+ reference document because the two are different shapes — the complete document has every key+ set, the template has almost none — and an operation that misbehaves on an absent key would+ pass over one and fail over the other."""+@model_validator(mode="after")def check_defaults_layer_is_reachable(self) -> Self:has_packaged_path = self.packaged_document_path is not None@@ -94,6 +105,18 @@ class Surface(BaseModel):document: str = tomlkit.dumps(self.read_defaults_document()) # pyright: ignore[reportUnknownMemberType]return document+ def reference_documents(self) -> list[tuple[str, str]]:+ """Every document a replay must be neutral over, as `(label, text)` pairs.++ Both are at the current schema by construction — one is the packaged defaults layer or the+ document the model's own defaults synthesize, the other is the starter template the kit+ ships — so neither is a fixture anyone has to keep up to date.+ """+ documents = [("reference document", self.render_reference_document())]+ if self.kit_template_path is not None:+ documents.append(("kit template", self.kit_template_path.read_text(encoding="utf-8")))+ return documents+def read_defaults_document(self) -> dict[str, Any]:"""The surface's complete reference document, as a plain mapping.@@ -178,6 +201,52 @@ class SurfaceRegistry(BaseModel):msg = f"no surface '{surface_id}' in this registry"raise MigrationRegistryError(msg)+ def surface_for_file_name(self, *, file_name: str) -> Surface | None:+ """Which surface owns a file, by its name alone, or `None` when no surface claims it.++ > **Exact filenames claim before globs, across all surfaces.** A file matched by any+ > surface's exact pattern belongs to that surface and is excluded from every surface's+ > glob.++ Without the rule, `pipelex_service.toml` is both the base file of one surface and a match+ for another's `pipelex_*.toml`, and which ledger runs over it becomes an accident of+ iteration order.++ Raises:+ MigrationRegistryError: two surfaces' globs both claim this file. That the registry+ itself cannot decide — whether two glob languages overlap is not decidable+ cheaply, and the registry has no files to look at. Here there is one, so the+ contract's *a file claimed by two globs is a registry error* becomes enforceable,+ by name, on the file that proves it.+ """+ for surface in self.surfaces:+ if surface.base_file == file_name:+ return surface+ claimants = [surface for surface in self.surfaces if surface.tier_glob is not None and fnmatch(file_name, surface.tier_glob)]+ if len(claimants) > 1:+ named = ", ".join(f"'{surface.surface_id}' ({surface.tier_glob})" for surface in claimants)+ msg = f"registry error: '{file_name}' is claimed by the tier globs of {named} — a file belongs to exactly one surface"+ raise MigrationRegistryError(msg)+ return claimants[0] if claimants else None++ def files_by_surface_in_directory(self, *, directory: Path) -> list[tuple[Surface, Path]]:+ """Every file in one configuration directory that a surface claims, with its surface.++ A directory that does not exist is skipped rather than refused: the global `~/.pipelex/`+ and a project's `.pipelex/` are both optional, and a machine that has only one of them is+ an ordinary machine, not a broken one.+ """+ if not directory.is_dir():+ return []+ claimed: list[tuple[Surface, Path]] = []+ for path in sorted(directory.iterdir()):+ if not path.is_file():+ continue+ surface = self.surface_for_file_name(file_name=path.name)+ if surface is not None:+ claimed.append((surface, path))+ return claimed+def _reject_duplicates(*, values: list[str], label: str) -> None:seen: set[str] = set()@@ -205,6 +274,7 @@ def build_config_surface_registry() -> SurfaceRegistry:config_model=PipelexConfig,defaults_layer_kind=DefaultsLayerKind.PACKAGED_DOCUMENT,packaged_document_path=_PACKAGE_ROOT / PIPELEX_CONFIG_FILE_NAME,+ kit_template_path=_KIT_CONFIGS_DIR / PIPELEX_CONFIG_FILE_NAME,),Surface(surface_id="telemetry-config",@@ -213,6 +283,7 @@ def build_config_surface_registry() -> SurfaceRegistry:tier_glob="telemetry_*.toml",config_model=TelemetryConfig,defaults_layer_kind=DefaultsLayerKind.MODEL_DEFAULTS,+ kit_template_path=_KIT_CONFIGS_DIR / TELEMETRY_CONFIG_FILE_NAME,),Surface(surface_id="pipelex-service-config",@@ -220,6 +291,7 @@ def build_config_surface_registry() -> SurfaceRegistry:base_file=PIPELEX_SERVICE_CONFIG_FILE_NAME,config_model=PipelexServiceConfig,defaults_layer_kind=DefaultsLayerKind.MODEL_DEFAULTS,+ kit_template_path=_KIT_CONFIGS_DIR / PIPELEX_SERVICE_CONFIG_FILE_NAME,),])diff --git a/pipelex/migration/transform_check.py b/pipelex/migration/transform_check.pynew file mode 100644index 000000000..2bbbc2711--- /dev/null+++ b/pipelex/migration/transform_check.py@@ -0,0 +1,388 @@+"""The transform goldens — do an entry's operations really say what the schema change did?++Coverage proves an entry *exists* for every removal. Convergence proves replay is *harmless* on a+current file. Neither proves the entry is *right*: a rename whose `new_key` is misspelled passes+both — the removed path is accounted for, and over a current document the source is absent so the+operation skips — and then migrates every user file to a key `extra="forbid"` rejects, with the+tool reporting success. This check closes that, by doing the migration for real on the one pair of+documents the golden chain keeps for it: the reference document of schema version N-1 and the one+of N, with entry N in between.++Three claims per link, and every one of them is one-directional on purpose — a comparator that+demands the two documents agree outright goes red on any honest bump, where the same commit adds+keys, edits comments and flips unrelated defaults:++- **Every path the migration creates, the new shape has** — its reference document or its+ fingerprint. This is the wrong destination, the misordered rename and the operation that lands in+ a table the new shape does not carry. The fingerprint is consulted alongside the document because+ a document cannot express every legal path: an optional key whose default is `None` is absent+ from the reference document, TOML having no null, while being a perfectly ordinary destination —+ and a misspelled destination is in neither, so nothing is weakened. A created path whose+ *container* is gone from the new document is exempt too: when a whole entry of an open mapping+ was dropped from the reference document between versions, nothing can be said about what an+ operation did inside it.+- **Every path the two reference documents share survives the migration.** This is over-deletion —+ an entry that dropped a parent table where it meant to drop one child. A path the new shape no+ longer has is not this check's business: the schema removed it, and coverage is what demands the+ operation that accounts for it.+- **The last link's migrated document is accepted by the current model**, read the way a user's+ file is actually read: beneath the current defaults layer. This is where a wrong value lands —+ a remap rewriting to a spelling the schema rejects, a destination the model does not know.++This is also where a **pre-history** entry is verified, and it is verified by exactly these three+claims: an entry whose change predates the first fingerprint has no `defaults@N-1` to start from,+so it ships a hand-authored `[email protected]` saying what the old shape was, and the link runs from+there. Nothing else about the check changes — which is the point of that exception's shape. What+the entry may declare and address is `check-ledger`'s half.++What the comparator deliberately does not do, and why (each stated in `docs/migration-ledger.md` too):++**`paths(defaults@N)` minus `added_at_N` is not the comparator.** A raw fingerprint difference counts a+rename's *destination* as an addition, so subtracting it would demand the destination be absent+from the expected shape — exactly where a correct rename puts it. That is the same defect the+symbolic end-state check met and answered with containment; the answer here is the same, expressed+over documents: additions are tolerated because nothing asserts them, rather than subtracted.+Tolerating them by subtraction would also be blind by construction to everything a fingerprint+cannot see — a model added to a packaged deck between versions lives beneath an open node, where+the fingerprint records a value schema and never a key.++**Value equality against `defaults@N` is unsound and is not asserted.** A default flipped in the+same commit as a rename would make the check red with no remedy available to anyone: the older+link is frozen, the head link tracks, and a migrated file legitimately carries the user's old+value where the new reference document carries the new default. What the contract wanted from it —+*the operations produce values the new schema accepts* — is checked where it can be checked+soundly: by the last link's validation here, and by `check-ledger`'s remap legality, which refuses+a remap whose target spelling the new schema does not accept.++**A created path is checked against `defaults@N` *or* `fingerprint@N`, not against the document+alone.** An optional key whose default is `None` has no value in any reference document — TOML has+no null and the synthesized document drops it — so a migration that moves a user's value onto such+a key creates a path the document legitimately lacks. Checking the document alone would refuse the+one destination the schema most obviously has. The fingerprint is checked-in data like everything+else here, so consulting it costs the check nothing it was protecting.++See `docs/migration-ledger.md` → "Transform goldens".+"""++from enum import StrEnum+from pathlib import Path+from typing import Any++from pydantic import BaseModel, ConfigDict, Field, ValidationError++from pipelex.migration.documents import document_paths+from pipelex.migration.engine import apply_ops_over_text+from pipelex.migration.exceptions import MigrationGoldenError+from pipelex.migration.fingerprint import PATH_SEPARATOR, SurfaceFingerprint+from pipelex.migration.goldens import defaults_golden_path, pre_history_document_path, read_fingerprint_golden+from pipelex.migration.ledger import MigrationEntry, MigrationLedger, MigrationSafety, load_ledger+from pipelex.migration.surfaces import Surface, SurfaceRegistry+from pipelex.suggested_fix import WILDCARD_SEGMENT+from pipelex.system.configuration.config_surface import strip_reserved_meta+from pipelex.tools.misc.json_utils import deep_update+from pipelex.tools.misc.toml_utils import load_toml_from_content+++class TransformIssueKind(StrEnum):+ """Why a link of the transform chain failed. Distinct kinds so a red gate says what broke."""++ DEFAULTS_GOLDEN_MISSING = "defaults_golden_missing"+ PRE_HISTORY_DOCUMENT_MISSING = "pre_history_document_missing"+ TRANSFORM_CONFLICTED = "transform_conflicted"+ DESTINATION_NOT_IN_NEW_SHAPE = "destination_not_in_new_shape"+ SURVIVING_PATH_REMOVED = "surviving_path_removed"+ MIGRATED_DOCUMENT_REJECTED = "migrated_document_rejected"+++class TransformIssue(BaseModel):+ model_config = ConfigDict(frozen=True, extra="forbid")++ surface_id: str+ kind: TransformIssueKind = Field(strict=False)+ message: str+++def check_transforms(*, registry: SurfaceRegistry, migration_dir: Path) -> list[TransformIssue]:+ """Verify the transform-golden chain of every surface in a registry."""+ issues: list[TransformIssue] = []+ for surface in registry.surfaces:+ issues.extend(check_transform_chain(surface=surface, migration_dir=migration_dir))+ return issues+++def check_transform_chain(*, surface: Surface, migration_dir: Path) -> list[TransformIssue]:+ """Verify one surface's chain, pairwise, one link per ledger entry.++ Pairwise is enough: the links compose by induction into the whole chain, because a replay from+ any historical snapshot skips every entry at or below that snapshot's version — their sources+ are material permanently removed.++ Raises:+ MigrationLedgerError: the ledger is missing, unparseable, or internally inconsistent.+ """+ ledger = load_ledger(migration_dir=migration_dir, surface_id=surface.surface_id)+ issues: list[TransformIssue] = []+ for entry in ledger.migration:+ issues.extend(_check_link(surface=surface, ledger=ledger, entry=entry, migration_dir=migration_dir))+ return issues+++def _check_link(*, surface: Surface, ledger: MigrationLedger, entry: MigrationEntry, migration_dir: Path) -> list[TransformIssue]:+ if entry.safety is MigrationSafety.UNSAFE:+ # An unsafe entry is reported and never applied, so no document ever makes this transition+ # mechanically. Demanding that its operations reach the new shape would demand a+ # completeness the vocabulary explicitly grants it the right to lack: an entry with no+ # operations at all is legal precisely when it is unsafe.+ return []++ version = entry.to_schema_version+ is_head = version == ledger.surface.current_schema_version+ before_text = _read_document_the_entry_migrates_from(surface=surface, entry=entry, migration_dir=migration_dir)+ if before_text is None:+ return [_starting_document_missing(surface=surface, entry=entry, migration_dir=migration_dir)]+ after_text = _read_defaults_golden(migration_dir=migration_dir, surface_id=surface.surface_id, schema_version=version)+ if after_text is None:+ # The head link has no snapshot until `update-migration-schemas` runs, and the coverage+ # gate already asks for exactly that, by name. Saying it twice adds nothing.+ if is_head:+ return []+ return [_defaults_golden_missing(surface=surface, entry=entry, migration_dir=migration_dir, schema_version=version)]++ application = apply_ops_over_text(text=before_text, ops=entry.ops)+ if application.conflicts:+ details = "; ".join(str(conflict.detail) for conflict in application.conflicts)+ return [+ _issue(+ surface=surface,+ kind=TransformIssueKind.TRANSFORM_CONFLICTED,+ message=(+ f"entry '{entry.id}': {len(application.conflicts)} of its operations conflict when applied to "+ f"{_starting_document_label(entry=entry)} — {details}. A document at the version the entry migrates *from* "+ f"is the one document it must be able to migrate, so a conflict here is a conflict on every file in the field"+ ),+ )+ ]++ issues = _check_paths(+ surface=surface,+ entry=entry,+ before_text=before_text,+ after_text=after_text,+ migrated_text=application.text,+ after_fingerprint=read_fingerprint_golden(migration_dir=migration_dir, surface_id=surface.surface_id, schema_version=version),+ )+ if is_head:+ issues.extend(_check_the_migrated_document_is_accepted(surface=surface, entry=entry, after_text=after_text, migrated_text=application.text))+ return issues+++def _check_paths(+ *,+ surface: Surface,+ entry: MigrationEntry,+ before_text: str,+ after_text: str,+ migrated_text: str,+ after_fingerprint: SurfaceFingerprint | None,+) -> list[TransformIssue]:+ version = entry.to_schema_version+ before_paths = _paths_of(text=before_text)+ after_paths = _paths_of(text=after_text)+ migrated_paths = _paths_of(text=migrated_text)+ recorded_paths = after_fingerprint.path_names() if after_fingerprint is not None else set[str]()++ created = {path for path in migrated_paths - before_paths if not _is_recorded(path=path, recorded_paths=recorded_paths)}+ unexpected = {path for path in created - after_paths if _ancestors_are_in(path=path, paths=after_paths)}+ removed = (before_paths & after_paths) - migrated_paths++ issues = [+ _issue(+ surface=surface,+ kind=TransformIssueKind.DESTINATION_NOT_IN_NEW_SHAPE,+ message=(+ f"entry '{entry.id}': migrating {_starting_document_label(entry=entry)} produces '{path}', which schema version "+ f"{version} has nowhere — neither its reference document nor its fingerprint carries that path, so a destination "+ f"is misspelled or lands where the new shape holds nothing. Every file this entry migrates would end up holding "+ f"that key, with the tool reporting success"+ ),+ )+ for path in _shallowest(paths=unexpected)+ ]+ issues.extend(+ _issue(+ surface=surface,+ kind=TransformIssueKind.SURVIVING_PATH_REMOVED,+ message=(+ f"entry '{entry.id}': '{path}' is in {_starting_document_label(entry=entry)} and in the reference document of "+ f"schema version {version} alike, but migrating the first one removes it — an operation targets a parent where it "+ f"meant to target one child, or renames away material the new shape still has"+ ),+ )+ for path in _shallowest(paths=removed)+ )+ return issues+++def _check_the_migrated_document_is_accepted(*, surface: Surface, entry: MigrationEntry, after_text: str, migrated_text: str) -> list[TransformIssue]:+ """The last link's output must be something the current model accepts.++ Read the way a user's file is really read: beneath the surface's current defaults layer, not+ alone. A migrated file is *expected* to lack whatever the new version added — that is what+ makes an additive change absorbable — so validating it on its own would report the defaults+ layer doing its job as a failure.++ Only the last link can be checked this way, and that is not a shortcut: it is the only link+ whose model we still have. Earlier ones are covered by induction, having been the last link+ when they were authored.+ """+ document: dict[str, Any] = load_toml_from_content(after_text)+ deep_update(document, updates=load_toml_from_content(migrated_text))+ strip_reserved_meta(config_dict=document)+ try:+ surface.config_model.model_validate(document)+ except ValidationError as exc:+ return [+ _issue(+ surface=surface,+ kind=TransformIssueKind.MIGRATED_DOCUMENT_REJECTED,+ message=(+ f"entry '{entry.id}': {_starting_document_label(entry=entry)}, migrated by this "+ f"entry and read beneath the current defaults, is not accepted by {surface.config_model.__name__} — {exc}. "+ f"An operation writes a value or a key the schema does not take, so every file it migrates fails to load"+ ),+ )+ ]+ return []+++def _read_defaults_golden(*, migration_dir: Path, surface_id: str, schema_version: int) -> str | None:+ return _read_golden_text(path=defaults_golden_path(migration_dir=migration_dir, surface_id=surface_id, schema_version=schema_version))+++def _read_golden_text(*, path: Path) -> str | None:+ """A golden's text, `None` when it does not exist — and a named refusal when it exists but cannot be read.++ Raises:+ MigrationGoldenError: the file is there and is not readable UTF-8 text. A gate has to say+ which file, the same way `read_fingerprint_golden` does; a raw traceback from deep in a+ comparison names nothing an author can act on.+ """+ if not path.exists():+ return None+ try:+ return path.read_text(encoding="utf-8")+ except (OSError, ValueError) as exc:+ msg = f"unreadable golden at {path}: {exc}"+ raise MigrationGoldenError(msg) from exc+++def _read_document_the_entry_migrates_from(*, surface: Surface, entry: MigrationEntry, migration_dir: Path) -> str | None:+ """The document this link starts from: the previous snapshot, or a hand-authored one.++ A pre-history entry has no previous snapshot by definition — the change predates the first+ fingerprint — so its author writes the old shape down as `[email protected]` and the link is+ verified from there. Everything after that point is identical for both kinds of entry, which is+ the whole reason the exception is worth having: the three claims below are what actually+ verifies a pre-history entry, and they are the same three claims every other entry answers.+ """+ if entry.pre_history:+ return _read_golden_text(path=_pre_history_document_path(surface=surface, entry=entry, migration_dir=migration_dir))+ return _read_defaults_golden(migration_dir=migration_dir, surface_id=surface.surface_id, schema_version=entry.to_schema_version - 1)+++def _pre_history_document_path(*, surface: Surface, entry: MigrationEntry, migration_dir: Path) -> Path:+ return pre_history_document_path(migration_dir=migration_dir, surface_id=surface.surface_id, schema_version=entry.to_schema_version)+++def _starting_document_label(*, entry: MigrationEntry) -> str:+ if entry.pre_history:+ return f"its hand-authored pre-history document (before@{entry.to_schema_version}.toml)"+ return f"the reference document of schema version {entry.to_schema_version - 1}"+++def _starting_document_missing(*, surface: Surface, entry: MigrationEntry, migration_dir: Path) -> TransformIssue:+ if not entry.pre_history:+ return _defaults_golden_missing(surface=surface, entry=entry, migration_dir=migration_dir, schema_version=entry.to_schema_version - 1)+ path = _pre_history_document_path(surface=surface, entry=entry, migration_dir=migration_dir)+ return _issue(+ surface=surface,+ kind=TransformIssueKind.PRE_HISTORY_DOCUMENT_MISSING,+ message=(+ f"entry '{entry.id}' is marked pre_history and there is no {path.name} beside the golden chain. A pre-history entry "+ f"is exempt from being accounted against a fingerprint diff precisely because none describes it, and this document "+ f"is what it is verified against instead — write the old shape it migrates from, by hand, at {path}"+ ),+ )+++def _paths_of(*, text: str) -> set[str]:+ return document_paths(document=load_toml_from_content(text))+++def _is_recorded(*, path: str, recorded_paths: set[str]) -> bool:+ """Whether a concrete document path is one the fingerprint records, wildcard included.++ A document names the user's own key where the fingerprint names `*`, so `deck.claude.new_name`+ is recorded as `deck.*.new_name`. Comparing the two literally would let a rename beneath an+ open mapping read as a misspelled destination whenever the reference document happens not to+ carry that key under that entry.+ """+ if path in recorded_paths:+ return True+ segments = path.split(PATH_SEPARATOR)+ for recorded in recorded_paths:+ recorded_segments = recorded.split(PATH_SEPARATOR)+ if len(recorded_segments) != len(segments):+ continue+ if all(+ _segment_matches(recorded=recorded_segment, concrete=segment)+ for recorded_segment, segment in zip(recorded_segments, segments, strict=True)+ ):+ return True+ return False+++def _segment_matches(*, recorded: str, concrete: str) -> bool:+ return recorded in {WILDCARD_SEGMENT, concrete}+++def _ancestors_are_in(*, path: str, paths: set[str]) -> bool:+ """Whether every table containing this path is present in a path set.++ A created path is only meaningful against a document that still has somewhere to put it: when+ the reference documents disagree about a whole container — an entry of an open mapping dropped+ between versions, say — what an operation did inside it says nothing about the operation.+ """+ segments = path.split(PATH_SEPARATOR)+ return all(PATH_SEPARATOR.join(segments[:depth]) in paths for depth in range(1, len(segments)))+++def _shallowest(*, paths: set[str]) -> list[str]:+ """The topmost path of each affected subtree, sorted.++ A `delete_table` one level too high takes a whole section with it, and naming every leaf it+ took would bury the one path the author has to look at under its own consequences.+ """+ return sorted(path for path in paths if not _has_ancestor_in(path=path, paths=paths))+++def _has_ancestor_in(*, path: str, paths: set[str]) -> bool:+ segments = path.split(PATH_SEPARATOR)+ return any(PATH_SEPARATOR.join(segments[:depth]) in paths for depth in range(1, len(segments)))+++def _defaults_golden_missing(*, surface: Surface, entry: MigrationEntry, migration_dir: Path, schema_version: int) -> TransformIssue:+ path = defaults_golden_path(migration_dir=migration_dir, surface_id=surface.surface_id, schema_version=schema_version)+ return _issue(+ surface=surface,+ kind=TransformIssueKind.DEFAULTS_GOLDEN_MISSING,+ message=(+ f"entry '{entry.id}' cannot be verified against what it did: no reference document at {path.name}. Run `make umig` if "+ f"the current schema version was just bumped; below the current version this is a broken chain, and the missing link "+ f"has to be restored rather than regenerated — a regenerated one would describe today's models, not that version's"+ ),+ )+++def _issue(*, surface: Surface, kind: TransformIssueKind, message: str) -> TransformIssue:+ return TransformIssue(surface_id=surface.surface_id, kind=kind, message=message)diff --git a/pipelex/migration/walk.py b/pipelex/migration/walk.pynew file mode 100644index 000000000..7ae82ae81--- /dev/null+++ b/pipelex/migration/walk.py@@ -0,0 +1,213 @@+"""The sequential path-state walk — replaying an entry's operations symbolically.++An entry's operations chain: a parent is renamed and then keys inside it are renamed, so the+intermediate paths belong to neither the old fingerprint nor the new one, and no ordering avoids+that. Every static check about an entry therefore starts by replaying the entry over the previous+fingerprint's path set, carrying each surviving path together with the path it *originated from*.++The walk lives here rather than inside either gate because both read it, for different questions:+the coverage gate asks what the end state looks like against the new fingerprint, and the ledger+check asks what each individual operation was addressing when it ran. One walk answers both, and+one walk is what keeps the two gates from disagreeing about what an entry does.++See `docs/migration-ledger.md` → "Sequential path state".+"""++from collections.abc import Sequence++from pydantic import BaseModel, ConfigDict, Field++from pipelex.migration.fingerprint import PATH_SEPARATOR, TABLE_TYPE, PathFingerprint, SurfaceFingerprint+from pipelex.migration.ledger import MigrationEntry+from pipelex.suggested_fix import DeleteKeyOp, DeleteTableOp, MigrationOp, MoveKeyOp, RemapValueOp, RenameTableKeyOp+++class PathState(BaseModel):+ """The symbolic state of a path set part-way through an entry's operations.++ Each live path is carried with the path it *originated from* in the previous fingerprint, so+ that an operation acting on a path some earlier operation renamed can still be attributed to+ the schema path it is really about.+ """++ model_config = ConfigDict(extra="forbid")++ origin_by_current: dict[str, str]++ @classmethod+ def make_from_fingerprint(cls, *, fingerprint: SurfaceFingerprint) -> "PathState":+ return cls(origin_by_current={path: path for path in fingerprint.path_names()})++ def current_paths(self) -> set[str]:+ return set(self.origin_by_current)++ def subtree_of(self, *, path: str) -> list[str]:+ prefix = path + PATH_SEPARATOR+ return [current for current in self.origin_by_current if current == path or current.startswith(prefix)]++ def drop_subtree(self, *, path: str) -> None:+ for current in self.subtree_of(path=path):+ del self.origin_by_current[current]++ def move_subtree(self, *, source: str, destination: str) -> None:+ moved = {destination + current[len(source) :]: self.origin_by_current[current] for current in self.subtree_of(path=source)}+ self.drop_subtree(path=source)+ self.origin_by_current.update(moved)++ def final_path_of_origin(self, *, origin: str) -> str | None:+ for current, recorded_origin in self.origin_by_current.items():+ if recorded_origin == origin:+ return current+ return None++ def trace_to_origin(self, *, path: str) -> str:+ """A path as the previous fingerprint would have spelled it, on a best-effort basis.++ The exact lookup only answers for paths the fingerprint records, and an operation may+ legitimately address one it does not — a concrete key beneath an open node is the user's+ and is nowhere in any fingerprint. So the longest recorded prefix is traced back and the+ remaining segments are carried along unchanged, which is enough for the ledger check to+ ask whether the operation is addressing user key space.+ """+ segments = path.split(PATH_SEPARATOR)+ for depth in range(len(segments), 0, -1):+ prefix = PATH_SEPARATOR.join(segments[:depth])+ origin = self.origin_by_current.get(prefix)+ if origin is not None:+ return PATH_SEPARATOR.join([origin, *segments[depth:]])+ return path+++class RecordedRemap(BaseModel):+ """A remap that fired during the walk, attributed to the schema path it originated from."""++ model_config = ConfigDict(frozen=True, extra="forbid")++ origin_path: str+ old_values: list[str]+ new_values: list[str]+++class WalkedOp(BaseModel):+ """One operation, as the walk saw it: what it addressed, and where that came from."""++ model_config = ConfigDict(frozen=True, extra="forbid")++ op_index: int+ source_path: str+ """The path the operation addresses at the moment it runs, after any earlier rename in the+ same entry has moved it."""++ origin_path: str+ """That path traced back to the previous fingerprint's spelling — exact when the fingerprint+ records it, best effort otherwise (see `PathState.trace_to_origin`)."""++ source_was_recorded: bool+ """Whether the fingerprint records the exact source path. False means either a genuinely dead+ operation — which the coverage gate reports — or an operation reaching into user key space,+ which the ledger check reports."""+++class EntryWalk(BaseModel):+ """What replaying one entry over the previous fingerprint's path set found out."""++ model_config = ConfigDict(extra="forbid")++ state: PathState+ """The end state."""++ walked_ops: list[WalkedOp] = Field(default_factory=list[WalkedOp])+ """One record per operation, in order, whether or not the operation could fire."""++ dead_ops: list[str] = Field(default_factory=list[str])+ """Descriptions of operations that can never fire: the source is absent, or the operation+ kind cannot act on what the source is."""++ occupied_destinations: list[str] = Field(default_factory=list[str])+ """Descriptions of renames or moves onto a path the state already had. The applier refuses+ to clobber an occupied destination, so a file carrying both keys — a valid file at the old+ schema — would come back CONFLICT on every run."""++ remaps: list[RecordedRemap] = Field(default_factory=list[RecordedRemap])+ """The remaps that fired, attributed to the schema path they originated from."""+++def joined(*, segments: Sequence[str]) -> str:+ return PATH_SEPARATOR.join(segments)+++def op_source_path(*, op: MigrationOp) -> str:+ """The path an operation acts on. `delete_table`'s `table_path` *is* its target."""+ match op:+ case DeleteTableOp():+ return joined(segments=op.table_path)+ case DeleteKeyOp() | RenameTableKeyOp() | MoveKeyOp() | RemapValueOp():+ return joined(segments=[*op.table_path, op.key])+++def walk_entry(*, entry: MigrationEntry, before: SurfaceFingerprint) -> EntryWalk:+ """Replay an entry's operations symbolically over the previous fingerprint's path set."""+ walk = EntryWalk(state=PathState.make_from_fingerprint(fingerprint=before))++ for op_index, op in enumerate(entry.ops):+ source = op_source_path(op=op)+ recorded_origin = walk.state.origin_by_current.get(source)+ walk.walked_ops.append(+ WalkedOp(+ op_index=op_index,+ source_path=source,+ origin_path=recorded_origin if recorded_origin is not None else walk.state.trace_to_origin(path=source),+ source_was_recorded=recorded_origin is not None,+ )+ )+ if recorded_origin is None:+ walk.dead_ops.append(f"{op.kind} on '{source}' — no such path at schema version {before.schema_version}, so it can never fire")+ continue+ origin_record = before.paths[recorded_origin]+ if isinstance(op, DeleteTableOp) and not _is_table_like(record=origin_record):+ walk.dead_ops.append(f"{op.kind} on '{source}' — that path is a key, not a table, so the applier would skip it forever; use delete_key")+ continue+ _apply_op_to_state(op=op, source=source, walk=walk)++ return walk+++def _is_table_like(*, record: PathFingerprint) -> bool:+ """Whether the applier's `delete_table` finds a table at this path.++ An open node counts: `dict[str, X]` is a `[table]` in a file, whatever the fingerprint calls+ its value type.+ """+ return record.value_type == TABLE_TYPE or record.open_node+++def _apply_op_to_state(*, op: MigrationOp, source: str, walk: EntryWalk) -> None:+ match op:+ case DeleteKeyOp() | DeleteTableOp():+ walk.state.drop_subtree(path=source)+ case RenameTableKeyOp():+ _move_in_state(op=op, source=source, destination=joined(segments=[*op.table_path, op.new_key]), walk=walk)+ case MoveKeyOp():+ _move_in_state(op=op, source=source, destination=joined(segments=[*op.new_table_path, op.new_key]), walk=walk)+ case RemapValueOp():+ walk.remaps.append(+ RecordedRemap(+ origin_path=walk.state.origin_by_current[source],+ old_values=sorted(op.mapping),+ new_values=sorted(set(op.mapping.values())),+ )+ )+++def _move_in_state(*, op: MigrationOp, source: str, destination: str, walk: EntryWalk) -> None:+ """Move a subtree, recording a collision when the destination is already occupied.++ The move is performed either way, so that the end-state comparison stays meaningful and the+ collision is reported once rather than echoed as an unaccounted source.+ """+ if walk.state.subtree_of(path=destination):+ walk.occupied_destinations.append(+ f"{op.kind} moves '{source}' onto '{destination}', which the previous schema already has — "+ f"a file carrying both is refused as a conflict on every run, so a safe entry cannot do this"+ )+ walk.state.move_subtree(source=source, destination=destination)diff --git a/pipelex/pipeline/fixes/applier.py b/pipelex/pipeline/fixes/applier.pyindex d29bb4545..bef2cba68 100644--- a/pipelex/pipeline/fixes/applier.py+++ b/pipelex/pipeline/fixes/applier.py@@ -29,7 +29,7 @@ from pipelex_tools import format_mthdsfrom pydantic import BaseModel, ConfigDict, Fieldfrom tomlkit import TOMLDocumentfrom tomlkit.container import Container, OutOfOrderTableProxy-from tomlkit.items import AbstractTable, Item+from tomlkit.items import AbstractTable, AoT, Item, Tablefrom pipelex.base_exceptions import PipelexUnexpectedErrorfrom pipelex.suggested_fix import (@@ -73,6 +73,19 @@ class FixOpOutcome(StrEnum):case FixOpOutcome.SKIPPED | FixOpOutcome.CONFLICT:return False+ @property+ def is_conflict(self) -> bool:+ """Whether the change could not be made without choosing on the user's behalf.++ A separate question from ``did_apply``: both a skip and a conflict left the document+ untouched, but only one of them is something a caller has to act on.+ """+ match self:+ case FixOpOutcome.CONFLICT:+ return True+ case FixOpOutcome.APPLIED | FixOpOutcome.SKIPPED:+ return False+class FixOpApplication(BaseModel):"""Per-op application report: the op, whether it applied, and why not when skipped."""@@ -109,6 +122,18 @@ def _rename_key_in_place(*, parent_table: dict[str, Any], key: str, new_key: strparent (the old fixer's reordering bug). The golden byte-compare tests are the CI tripwire ifa tomlkit bump ever changes this.+ ⚠ **A rename leaves the node's raw ``dict`` storage stale for a value that is not a Table.**+ ``_replace_at`` leaves the old key in the dict and only writes the new one into it inside its+ table branch, so everything tomlkit renders or looks up stays right (``dumps``, ``in``, ``[]``,+ ``.get()`` all read the authoritative body) while ``dict.__delitem__`` — which+ ``Container.remove`` calls — cannot find the new key. Addressing a renamed **scalar or+ inline-table** key again in the same DOM therefore raises ``KeyError`` from inside the library.+ The ``.mthds`` fix path is unaffected because it renames ``[pipe.*]`` tables, which take the+ branch tomlkit maintains; configuration migration re-reads the document between operations+ that applied, for exactly this reason. Both facts are pinned by+ ``tests/unit/pipelex/pipeline/fixes/test_fix_applier_rename_dom_consistency.py``, which is the+ tripwire if a tomlkit bump ever changes it.+``_resolve_table`` hands back one of three dict-like shapes, each with its own route to the``Container`` that owns the key:@@ -145,7 +170,9 @@ def _rename_key_in_place(*, parent_table: dict[str, Any], key: str, new_key: strdef _replace_key_in_container(*, container: Container, key: str, new_key: str) -> None:- container._replace(key, new_key, cast("Item", container[key])) # pyright: ignore[reportPrivateUsage]+ item = cast("Item", container[key])+ container._replace(key, new_key, item) # pyright: ignore[reportPrivateUsage]+ _refresh_table_headers(item=item)def _as_tomlkit_value(value: TomlValue | None) -> Any:@@ -369,6 +396,34 @@ def _apply_at_table_path(*, toml_doc: TOMLDocument, fix_op: FixOp, table_path: lreturn _OpResult(FixOpOutcome.APPLIED)+def _refresh_table_headers(*, item: Any) -> None:+ """Make every table beneath a renamed or moved ``item`` re-render its header under its new path.++ tomlkit caches each table's rendered header in ``Table.display_name`` and, after a re-key,+ invalidates that cache by walking ``Table.values()`` — the merged dict facade, which yields+ one item per key. A table written in several chunks (``[a.b]`` … ``[a.c]`` … ``[a.b.d]``) is+ several ``Table`` items under one key, so only the first chunk is visited and the others keep+ rendering under the old name: the file comes out split between two tables with an+ ``applied`` verdict. Walking the *body* instead reaches every chunk, so clearing the cache+ here is what lets a rename or move come out whole for any layout the parser accepts.+ """+ if isinstance(item, OutOfOrderTableProxy):+ for chunk in item._tables: # noqa: SLF001 # pyright: ignore[reportPrivateUsage]+ _refresh_table_headers(item=chunk)+ return+ if isinstance(item, Table):+ item.display_name = None+ _refresh_table_headers(item=item.value)+ return+ if isinstance(item, AoT):+ for element in item.body:+ _refresh_table_headers(item=element)+ return+ if isinstance(item, Container):+ for _, child in item.body:+ _refresh_table_headers(item=child)++def _apply_move_key(*, toml_doc: TOMLDocument, fix_op: MoveKeyOp, table_path: list[str]) -> _OpResult:"""Relocate one key, creating whatever destination parents are missing.@@ -395,6 +450,7 @@ def _apply_move_key(*, toml_doc: TOMLDocument, fix_op: MoveKeyOp, table_path: lidel source_table[fix_op.key]destination_table = _create_block_table_path(toml_doc=toml_doc, table_path=fix_op.new_table_path)destination_table[fix_op.new_key] = moved_value+ _refresh_table_headers(item=moved_value)return _OpResult(FixOpOutcome.APPLIED)diff --git a/pipelex/pipeline/fixes/file_transaction.py b/pipelex/pipeline/fixes/file_transaction.pynew file mode 100644index 000000000..e6a00647a--- /dev/null+++ b/pipelex/pipeline/fixes/file_transaction.py@@ -0,0 +1,190 @@+"""Transactional file replacement — snapshot, stage, atomically replace, roll back on failure.++Extracted from the ``.mthds`` fix loop because configuration migration needs exactly the same+primitives with a different *scope*. The two callers differ in one deliberate way, and it is the+whole reason this module is shared rather than duplicated:++- the ``.mthds`` fix loop commits a whole round **all-or-nothing**: its fixes cascade across files+ of one bundle, and a half-written round leaves a bundle that validates as neither the old shape+ nor the new one;+- a migration commits **one file at a time**, because a surface's files are independent user+ documents. A file that is unparseable, unwritable, or changed during the run is reported as+ blocked while every sibling is migrated and reported normally+ (``docs/migration-ledger.md`` → "Per-file transactions").++Both scopes come out of the same call: ``commit_file_updates`` is all-or-nothing over the list it+is handed, so a list of one *is* the per-file scope.+"""++import os+import stat+import sys+import tempfile+from pathlib import Path+from typing import NamedTuple++from pipelex.pipeline.exceptions import FixTransactionError, FixWriteConflictError+++class FileSnapshot(NamedTuple):+ """Bytes and identity read from one target before it is replaced."""++ path: Path+ content: bytes+ mode: int+ device: int+ inode: int+++class PendingFileUpdate(NamedTuple):+ """A fully rendered replacement paired with the snapshot it was derived from."""++ snapshot: FileSnapshot+ new_content: str+++class StagedFileUpdate(NamedTuple):+ """Same-directory temp files for one replacement and its rollback copy."""++ snapshot: FileSnapshot+ replacement_snapshot: FileSnapshot+ rollback_path: Path+++def read_file_snapshot(path: Path) -> FileSnapshot:+ """Read bytes and mode from one open file descriptor, avoiding split stat/read state."""+ with path.open("rb") as file:+ content = file.read()+ file_stat = os.fstat(file.fileno())+ return FileSnapshot(+ path=path,+ content=content,+ mode=stat.S_IMODE(file_stat.st_mode),+ device=file_stat.st_dev,+ inode=file_stat.st_ino,+ )+++def write_staged_file(*, snapshot: FileSnapshot, content: bytes, label: str) -> Path:+ """Write and fsync a same-directory temp file ready for atomic replacement."""+ temp_file = tempfile.NamedTemporaryFile( # noqa: SIM115 - closed before the atomic replace+ mode="wb",+ dir=str(snapshot.path.parent),+ prefix=f".{snapshot.path.name}.pipelex-fix-{label}.",+ suffix=".tmp",+ delete=False,+ )+ temp_path = Path(temp_file.name)+ try:+ try:+ temp_file.write(content)+ os.fchmod(temp_file.fileno(), snapshot.mode)+ temp_file.flush()+ os.fsync(temp_file.fileno())+ finally:+ temp_file.close()+ except OSError:+ temp_path.unlink(missing_ok=True)+ raise+ return temp_path+++def stage_file_update(update: PendingFileUpdate) -> StagedFileUpdate:+ replacement_path = write_staged_file(snapshot=update.snapshot, content=update.new_content.encode("utf-8"), label="new")+ try:+ replacement_snapshot = read_file_snapshot(replacement_path)+ rollback_path = write_staged_file(snapshot=update.snapshot, content=update.snapshot.content, label="rollback")+ except OSError:+ replacement_path.unlink(missing_ok=True)+ raise+ return StagedFileUpdate(snapshot=update.snapshot, replacement_snapshot=replacement_snapshot, rollback_path=rollback_path)+++def file_state_matches(*, current_snapshot: FileSnapshot, expected_snapshot: FileSnapshot) -> bool:+ """Whether content, permissions, and filesystem identity still match an observed file."""+ return (+ current_snapshot.content == expected_snapshot.content+ and current_snapshot.mode == expected_snapshot.mode+ and current_snapshot.device == expected_snapshot.device+ and current_snapshot.inode == expected_snapshot.inode+ )+++def assert_snapshot_unchanged(snapshot: FileSnapshot) -> None:+ """Verify that a path still names the exact file state previously observed."""+ try:+ current_snapshot = read_file_snapshot(snapshot.path)+ except FileNotFoundError as exc:+ msg = f"refusing to overwrite '{snapshot.path}': the file was removed while changes were being prepared"+ raise FixWriteConflictError(msg) from exc+ if not file_state_matches(current_snapshot=current_snapshot, expected_snapshot=snapshot):+ msg = f"refusing to overwrite '{snapshot.path}': the file changed while changes were being prepared"+ raise FixWriteConflictError(msg)+++def _rollback_committed_updates(committed_updates: list[StagedFileUpdate]) -> list[str]:+ """Restore targets still owned by this transaction, returning rollback failures."""+ failures: list[str] = []+ for staged_update in reversed(committed_updates):+ target_path = staged_update.snapshot.path+ try:+ current_snapshot = read_file_snapshot(target_path)+ if not file_state_matches(current_snapshot=current_snapshot, expected_snapshot=staged_update.replacement_snapshot):+ failures.append(f"{target_path}: file changed after the replacement was committed")+ continue+ staged_update.rollback_path.replace(target_path)+ except OSError as exc:+ failures.append(f"{target_path}: {exc}")+ return failures+++def _commit_staged_updates(staged_updates: list[StagedFileUpdate]) -> None:+ committed_updates: list[StagedFileUpdate] = []+ try:+ for staged_update in staged_updates:+ # Accepted portability tradeoff: this check and Path.replace() are not one compare-and-swap operation.+ # An edit in the tiny gap can be overwritten; keep this portable unless real-world evidence justifies serialization.+ assert_snapshot_unchanged(staged_update.snapshot)+ staged_update.replacement_snapshot.path.replace(staged_update.snapshot.path)+ committed_updates.append(staged_update)+ except (FixWriteConflictError, OSError) as exc:+ rollback_failures = _rollback_committed_updates(committed_updates)+ if rollback_failures:+ failures = "; ".join(rollback_failures)+ msg = f"commit failed and rollback was incomplete ({failures}); inspect the named files before retrying"+ raise FixTransactionError(msg) from exc+ raise+++def _cleanup_staged_updates(staged_updates: list[StagedFileUpdate]) -> list[str]:+ """Remove transaction temps and record cleanup failures for the caller to report."""+ failures: list[str] = []+ for staged_update in staged_updates:+ for temp_path in (staged_update.replacement_snapshot.path, staged_update.rollback_path):+ try:+ temp_path.unlink(missing_ok=True)+ except OSError as exc:+ failures.append(f"{temp_path}: {exc}")+ return failures+++def commit_file_updates(updates: list[PendingFileUpdate]) -> None:+ """Stage every update completely, then atomically replace the targets, rolling back on failure.++ All-or-nothing over the list it is handed. A caller that wants per-file independence calls it+ once per file; a caller whose files only make sense together hands it the whole round.+ """+ staged_updates: list[StagedFileUpdate] = []+ try:+ for update in updates:+ staged_updates.append(stage_file_update(update))+ _commit_staged_updates(staged_updates)+ finally:+ cleanup_failures = _cleanup_staged_updates(staged_updates)+ active_exception = sys.exception()+ if cleanup_failures and active_exception is not None:+ active_exception.add_note(f"temporary-file cleanup also failed ({'; '.join(cleanup_failures)})")+ elif cleanup_failures:+ failures = "; ".join(cleanup_failures)+ msg = f"target changes were committed, but temporary-file cleanup failed ({failures})"+ raise FixTransactionError(msg)diff --git a/pipelex/pipeline/fixes/fix_loop.py b/pipelex/pipeline/fixes/fix_loop.pyindex 9fb420926..0786f4242 100644--- a/pipelex/pipeline/fixes/fix_loop.py+++ b/pipelex/pipeline/fixes/fix_loop.py@@ -23,9 +23,6 @@ Two scoping rules bound what gets touched:"""import os-import stat-import sys-import tempfilefrom collections.abc import Sequencefrom pathlib import Pathfrom typing import Any, NamedTuple, cast@@ -38,9 +35,16 @@ from pipelex.config import get_configfrom pipelex.interpreter_hub import resolve_library_dirsfrom pipelex.libraries.library_utils import get_pipelex_mthds_files_from_dirsfrom pipelex.pipe_machinery.pipe_blueprint import SIGNATURE_ONLY_KEYS-from pipelex.pipeline.exceptions import FixTransactionError, FixWriteConflictError, ValidateBundleError+from pipelex.pipeline.exceptions import FixWriteConflictError, ValidateBundleErrorfrom pipelex.pipeline.fixes.applicability import is_safe_fix_for_load_scope, is_target_in_write_scopefrom pipelex.pipeline.fixes.applier import apply_fix_ops, serialize_and_format+from pipelex.pipeline.fixes.file_transaction import (+ FileSnapshot,+ PendingFileUpdate,+ assert_snapshot_unchanged,+ commit_file_updates,+ read_file_snapshot,+)from pipelex.pipeline.validate_bundle import validate_bundlefrom pipelex.pipeline.validation_errors import build_validation_error_itemsfrom pipelex.suggested_fix import DeleteKeyOp, DeleteTableOp, EnsureTableOp, FixOp, MoveKeyOp, RemapValueOp, RenameTableKeyOp, SetKeyOp, SuggestedFix@@ -50,177 +54,17 @@ from pipelex.tools.misc.toml_utils import load_toml_from_path_MAIN_PIPE_KEY = "main_pipe"-class _FileSnapshot(NamedTuple):- """Bytes and permission bits read from one target before fixes are applied."""-- path: Path- content: bytes- mode: int- device: int- inode: int---class _PendingFileUpdate(NamedTuple):- """A fully rendered replacement paired with the snapshot it was derived from."""-- snapshot: _FileSnapshot- new_content: str---class _StagedFileUpdate(NamedTuple):- """Same-directory temp files for one replacement and its rollback copy."""-- snapshot: _FileSnapshot- replacement_snapshot: _FileSnapshot- rollback_path: Path--class _BoundFix(NamedTuple):"""A suggested fix pinned to the source identity snapshotted before validation."""fix: SuggestedFix- snapshot: _FileSnapshot+ snapshot: FileSnapshot@propertydef target_path(self) -> Path:return self.snapshot.path-def _read_file_snapshot(path: Path) -> _FileSnapshot:- """Read bytes and mode from one open file descriptor, avoiding split stat/read state."""- with path.open("rb") as file:- content = file.read()- file_stat = os.fstat(file.fileno())- return _FileSnapshot(- path=path,- content=content,- mode=stat.S_IMODE(file_stat.st_mode),- device=file_stat.st_dev,- inode=file_stat.st_ino,- )---def _write_staged_file(*, snapshot: _FileSnapshot, content: bytes, label: str) -> Path:- """Write and fsync a same-directory temp file ready for atomic replacement."""- temp_file = tempfile.NamedTemporaryFile( # noqa: SIM115 - closed before the atomic replace- mode="wb",- dir=str(snapshot.path.parent),- prefix=f".{snapshot.path.name}.pipelex-fix-{label}.",- suffix=".tmp",- delete=False,- )- temp_path = Path(temp_file.name)- try:- try:- temp_file.write(content)- os.fchmod(temp_file.fileno(), snapshot.mode)- temp_file.flush()- os.fsync(temp_file.fileno())- finally:- temp_file.close()- except OSError:- temp_path.unlink(missing_ok=True)- raise- return temp_path---def _stage_file_update(update: _PendingFileUpdate) -> _StagedFileUpdate:- replacement_path = _write_staged_file(snapshot=update.snapshot, content=update.new_content.encode("utf-8"), label="new")- try:- replacement_snapshot = _read_file_snapshot(replacement_path)- rollback_path = _write_staged_file(snapshot=update.snapshot, content=update.snapshot.content, label="rollback")- except OSError:- replacement_path.unlink(missing_ok=True)- raise- return _StagedFileUpdate(snapshot=update.snapshot, replacement_snapshot=replacement_snapshot, rollback_path=rollback_path)---def _file_state_matches(*, current_snapshot: _FileSnapshot, expected_snapshot: _FileSnapshot) -> bool:- """Whether content, permissions, and filesystem identity still match an observed file."""- return (- current_snapshot.content == expected_snapshot.content- and current_snapshot.mode == expected_snapshot.mode- and current_snapshot.device == expected_snapshot.device- and current_snapshot.inode == expected_snapshot.inode- )---def _assert_snapshot_unchanged(snapshot: _FileSnapshot) -> None:- """Verify that a path still names the exact file state previously observed."""- try:- current_snapshot = _read_file_snapshot(snapshot.path)- except FileNotFoundError as exc:- msg = f"refusing to overwrite '{snapshot.path}': the file was removed while fixes were being prepared"- raise FixWriteConflictError(msg) from exc- if not _file_state_matches(current_snapshot=current_snapshot, expected_snapshot=snapshot):- msg = f"refusing to overwrite '{snapshot.path}': the file changed while fixes were being prepared"- raise FixWriteConflictError(msg)---def _rollback_committed_updates(committed_updates: list[_StagedFileUpdate]) -> list[str]:- """Restore targets still owned by this transaction, returning rollback failures."""- failures: list[str] = []- for staged_update in reversed(committed_updates):- target_path = staged_update.snapshot.path- try:- current_snapshot = _read_file_snapshot(target_path)- if not _file_state_matches(current_snapshot=current_snapshot, expected_snapshot=staged_update.replacement_snapshot):- failures.append(f"{target_path}: file changed after the autofix replacement was committed")- continue- staged_update.rollback_path.replace(target_path)- except OSError as exc:- failures.append(f"{target_path}: {exc}")- return failures---def _commit_staged_updates(staged_updates: list[_StagedFileUpdate]) -> None:- committed_updates: list[_StagedFileUpdate] = []- try:- for staged_update in staged_updates:- # Accepted portability tradeoff: this check and Path.replace() are not one compare-and-swap operation.- # An edit in the tiny gap can be overwritten; keep this portable unless real-world evidence justifies serialization.- _assert_snapshot_unchanged(staged_update.snapshot)- staged_update.replacement_snapshot.path.replace(staged_update.snapshot.path)- committed_updates.append(staged_update)- except (FixWriteConflictError, OSError) as exc:- rollback_failures = _rollback_committed_updates(committed_updates)- if rollback_failures:- failures = "; ".join(rollback_failures)- msg = f"autofix commit failed and rollback was incomplete ({failures}); inspect the named files before retrying"- raise FixTransactionError(msg) from exc- raise---def _cleanup_staged_updates(staged_updates: list[_StagedFileUpdate]) -> list[str]:- """Remove transaction temps and record cleanup failures for the caller to report."""- failures: list[str] = []- for staged_update in staged_updates:- for temp_path in (staged_update.replacement_snapshot.path, staged_update.rollback_path):- try:- temp_path.unlink(missing_ok=True)- except OSError as exc:- failures.append(f"{temp_path}: {exc}")- return failures---def _commit_file_updates(updates: list[_PendingFileUpdate]) -> None:- """Stage a round completely, then atomically replace targets with rollback on failure."""- staged_updates: list[_StagedFileUpdate] = []- try:- for update in updates:- staged_updates.append(_stage_file_update(update))- _commit_staged_updates(staged_updates)- finally:- cleanup_failures = _cleanup_staged_updates(staged_updates)- active_exception = sys.exception()- if cleanup_failures and active_exception is not None:- active_exception.add_note(f"autofix temporary-file cleanup also failed ({'; '.join(cleanup_failures)})")- elif cleanup_failures:- failures = "; ".join(cleanup_failures)- msg = f"autofix target changes were committed, but temporary-file cleanup failed ({failures})"- raise FixTransactionError(msg)--class FixBundleResult(BaseModel):"""Outcome of one fix run: the final verdict, the work done, and what remains."""@@ -286,7 +130,7 @@ def _absolute_source_path(path: Path) -> Path:return Path(os.path.abspath(path))-def _snapshot_loaded_sources(*, entry_source_path: Path, effective_dirs: Sequence[Path]) -> dict[Path, _FileSnapshot]:+def _snapshot_loaded_sources(*, entry_source_path: Path, effective_dirs: Sequence[Path]) -> dict[Path, FileSnapshot]:"""Snapshot every source identity validation is about to load.Keys are lexical absolute source names, while each snapshot is pinned to the destination@@ -298,10 +142,10 @@ def _snapshot_loaded_sources(*, entry_source_path: Path, effective_dirs: Sequenc_absolute_source_path(mthds_path)for mthds_path in get_pipelex_mthds_files_from_dirs(dirs={_absolute_source_path(directory) for directory in effective_dirs}))- snapshots_by_source: dict[Path, _FileSnapshot] = {}+ snapshots_by_source: dict[Path, FileSnapshot] = {}for source_path in sorted(source_paths, key=str):target_path = source_path.resolve()- snapshot = _read_file_snapshot(target_path)+ snapshot = read_file_snapshot(target_path)snapshots_by_source[source_path] = snapshotsnapshots_by_source.setdefault(target_path, snapshot)return snapshots_by_source@@ -311,7 +155,7 @@ def _bind_fixes_to_snapshots(fixes: list[SuggestedFix],*,entry_source_path: Path,- snapshots_by_source: dict[Path, _FileSnapshot],+ snapshots_by_source: dict[Path, FileSnapshot],) -> list[_BoundFix]:"""Bind each fix once to the pre-validation file identity that supplied its source."""bound_fixes: list[_BoundFix] = []@@ -330,7 +174,7 @@ def _bind_fixes_to_snapshots(if current_target != snapshot.path:msg = f"refusing to apply fix from '{absolute_source}': its source was retargeted during validation"raise FixWriteConflictError(msg)- _assert_snapshot_unchanged(snapshot)+ assert_snapshot_unchanged(snapshot)bound_fixes.append(_BoundFix(fix=fix, snapshot=snapshot))return bound_fixes@@ -661,12 +505,12 @@ async def fix_bundle_file(bail_reason=bail_reason,)- fixes_by_target: dict[Path, tuple[_FileSnapshot, list[SuggestedFix]]] = {}+ fixes_by_target: dict[Path, tuple[FileSnapshot, list[SuggestedFix]]] = {}for bound_fix in new_fixes:target_group = fixes_by_target.setdefault(bound_fix.target_path, (bound_fix.snapshot, []))target_group[1].append(bound_fix.fix)- pending_updates: list[_PendingFileUpdate] = []+ pending_updates: list[PendingFileUpdate] = []round_fixes_applied: list[SuggestedFix] = []round_written_paths: list[Path] = []for target_path, (snapshot, target_fixes) in fixes_by_target.items():@@ -678,10 +522,10 @@ async def fix_bundle_file(round_fixes_applied.append(target_fix)any_op_applied = Trueif any_op_applied:- pending_updates.append(_PendingFileUpdate(snapshot=snapshot, new_content=serialize_and_format(toml_doc)))+ pending_updates.append(PendingFileUpdate(snapshot=snapshot, new_content=serialize_and_format(toml_doc)))round_written_paths.append(target_path)- _commit_file_updates(pending_updates)+ commit_file_updates(pending_updates)seen_fingerprints.update(_fix_fingerprint(fix.fix) for fix in new_fixes)fixes_applied.extend(round_fixes_applied)for target_path in round_written_paths:diff --git a/pipelex/suggested_fix.py b/pipelex/suggested_fix.pyindex 1a3a371d7..af2821986 100644--- a/pipelex/suggested_fix.py+++ b/pipelex/suggested_fix.py@@ -164,6 +164,21 @@ class MoveKeyOp(FixOpBase):raise ValueError(msg)return new_table_path+ @field_validator("table_path")+ @classmethod+ def refuse_wildcard_source(cls, table_path: list[str]) -> list[str]:+ """A move cannot start from a wildcard either, and for the same reason read the other way.++ With the destination pinned to one fixed key, "move each entry's key" is many-to-one: on+ any file where two matched entries carry the key, the second lands on a destination the+ first just occupied and the whole operation conflicts. The other kinds act on each matched+ entry in place, which is what makes a wildcard mean something for them.+ """+ if WILDCARD_SEGMENT in table_path:+ msg = f"a move_key source may not contain the wildcard segment '{WILDCARD_SEGMENT}': every matched entry would move onto one destination"+ raise ValueError(msg)+ return table_path+class RemapValueOp(FixOpBase):"""Rewrite ``key``'s value through ``mapping``, doing nothing when it is not a mapped value.diff --git a/pyproject.toml b/pyproject.tomlindex 130fa9246..d1a6f61b9 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -19,6 +19,9 @@ classifiers = [dependencies = ["aiofiles>=23.2.1",+ # The migration fingerprint reads constraint objects from it by type. A zero-dependency+ # interchange vocabulary, declared directly rather than borrowed from pydantic's own tree.+ "annotated-types>=0.7.0","datamodel-code-generator>=0.55.0","filetype>=1.2.0","httpx>=0.23.0,<1.0.0",@@ -110,6 +113,7 @@ docs = [dev = ["boto3-stubs>=1.35.24","cryptography>=46.0.7",+ "hypothesis>=6.140.3","jsonschema>=4.20.0","moto[dynamodb,s3]>=5.0.0","mypy==1.19.1",diff --git a/subject_grants.toml b/subject_grants.tomlindex bf9ef56fe..fbc75a982 100644--- a/subject_grants.toml+++ b/subject_grants.toml@@ -4436,34 +4436,46 @@ rationale = "Verb-object: renders the diagnostic; single operand."param = "toml_doc"rationale = "Single-operand serialization of the TOML document."-["pipelex/pipeline/fixes/fix_loop.py::_absolute_source_path"]-param = "path"-rationale = "Single-operand conversion of the path."+["pipelex/pipeline/fixes/file_transaction.py::_cleanup_staged_updates"]+param = "staged_updates"+rationale = "Verb-object: cleans up the staged updates; single operand."++["pipelex/pipeline/fixes/file_transaction.py::_commit_staged_updates"]+param = "staged_updates"+rationale = "Verb-object: commits the staged updates; single operand."++["pipelex/pipeline/fixes/file_transaction.py::_rollback_committed_updates"]+param = "committed_updates"+rationale = "Verb-object: rolls back the committed updates; single operand."-["pipelex/pipeline/fixes/fix_loop.py::_assert_snapshot_unchanged"]+["pipelex/pipeline/fixes/file_transaction.py::assert_snapshot_unchanged"]param = "snapshot"rationale = "Validator: the snapshot is the single operand checked."+["pipelex/pipeline/fixes/file_transaction.py::commit_file_updates"]+param = "updates"+rationale = "Verb-object: commits the updates; single operand."++["pipelex/pipeline/fixes/file_transaction.py::read_file_snapshot"]+param = "path"+rationale = "Factory from source: reads the snapshot of the file at path."++["pipelex/pipeline/fixes/file_transaction.py::stage_file_update"]+param = "update"+rationale = "Verb-object: stages the update; single operand."++["pipelex/pipeline/fixes/fix_loop.py::_absolute_source_path"]+param = "path"+rationale = "Single-operand conversion of the path."+["pipelex/pipeline/fixes/fix_loop.py::_bind_fixes_to_snapshots"]param = "fixes"rationale = "Verb-object: binds the fixes; the rest is context."-["pipelex/pipeline/fixes/fix_loop.py::_cleanup_staged_updates"]-param = "staged_updates"-rationale = "Verb-object: cleans up the staged updates; single operand."-["pipelex/pipeline/fixes/fix_loop.py::_colliding_op_name"]param = "fix_op"rationale = "Noun-named derivation from the fix op; scopes stay keyword."-["pipelex/pipeline/fixes/fix_loop.py::_commit_file_updates"]-param = "updates"-rationale = "Verb-object: commits the updates; single operand."--["pipelex/pipeline/fixes/fix_loop.py::_commit_staged_updates"]-param = "staged_updates"-rationale = "Verb-object: commits the staged updates; single operand."-["pipelex/pipeline/fixes/fix_loop.py::_fix_fingerprint"]param = "fix"rationale = "Noun-named derivation: the fingerprint of the fix; single operand."@@ -4480,14 +4492,6 @@ rationale = "Verb-object: partitions the fixes; scope params stay keyword."param = "validation_result"rationale = "Noun-named derivation from the validation result; single operand."-["pipelex/pipeline/fixes/fix_loop.py::_read_file_snapshot"]-param = "path"-rationale = "Factory from source: reads the snapshot of the file at path."--["pipelex/pipeline/fixes/fix_loop.py::_rollback_committed_updates"]-param = "committed_updates"-rationale = "Verb-object: rolls back the committed updates; single operand."-["pipelex/pipeline/fixes/fix_loop.py::_safe_fixes"]param = "items"rationale = "Noun-named derivation: filters the error items down to safe fixes; the rest is context."@@ -4496,10 +4500,6 @@ rationale = "Noun-named derivation: filters the error items down to safe fixes;param = "fixes"rationale = "Verb-object: splits the fixes; codes_by_file is context."-["pipelex/pipeline/fixes/fix_loop.py::_stage_file_update"]-param = "update"-rationale = "Verb-object: stages the update; single operand."-["pipelex/pipeline/fixes/fix_loop.py::_validation_error_items"]param = "exc"rationale = "Noun-named derivation from the single error operand."diff --git a/tests/unit/pipelex/migration/conftest.py b/tests/unit/pipelex/migration/conftest.pynew file mode 100644index 000000000..4d7a06d26--- /dev/null+++ b/tests/unit/pipelex/migration/conftest.py@@ -0,0 +1,127 @@+"""Builders shared by the migration engine's unit tests.++Ledgers are built from the models rather than from TOML text on purpose: these tests are about+what a *parsed* ledger does to a document, and the parsing itself has its own module.+"""++from collections.abc import Callable+from pathlib import Path++import pytest+from pydantic import BaseModel++from pipelex.migration.ledger import MigrationEntry, MigrationLedger, MigrationSafety, SurfaceBlock+from pipelex.migration.surfaces import DefaultsLayerKind, Surface, SurfaceRegistry+from pipelex.suggested_fix import MigrationOp++EXAMPLE_SURFACE_ID = "example-config"+EXAMPLE_BASE_FILE = "example.toml"+EXAMPLE_TIER_GLOB = "example_*.toml"++EntryBuilder = Callable[..., MigrationEntry]+LedgerBuilder = Callable[..., MigrationLedger]+SurfaceBuilder = Callable[..., Surface]+++def build_entry() -> EntryBuilder:+ def _build_entry(+ *,+ to_schema_version: int,+ ops: list[MigrationOp],+ surface_id: str = EXAMPLE_SURFACE_ID,+ safety: MigrationSafety = MigrationSafety.SAFE,+ guidance: str | None = None,+ title: str = "Reshape the example surface",+ description: str = "The example surface changed shape.",+ breaking: bool = True,+ pre_history: bool = False,+ declared_removed_paths: list[str] | None = None,+ ) -> MigrationEntry:+ return MigrationEntry(+ id=f"{surface_id}@{to_schema_version}",+ to_schema_version=to_schema_version,+ introduced_in="0.46.0",+ breaking=breaking,+ safety=safety,+ title=title,+ description=description,+ guidance=guidance,+ pre_history=pre_history,+ declared_removed_paths=declared_removed_paths or [],+ ops=ops,+ )++ return _build_entry+++def build_ledger() -> LedgerBuilder:+ def _build_ledger(+ *,+ entries: list[MigrationEntry],+ surface_id: str = EXAMPLE_SURFACE_ID,+ base_file: str = EXAMPLE_BASE_FILE,+ tier_glob: str | None = EXAMPLE_TIER_GLOB,+ ) -> MigrationLedger:+ return MigrationLedger(+ surface=SurfaceBlock(+ id=surface_id,+ title="An example configuration surface",+ base_file=base_file,+ tier_glob=tier_glob,+ current_schema_version=1 + len(entries),+ min_supported_schema_version=0,+ ),+ migration=entries,+ )++ return _build_ledger+++class ExampleConfig(BaseModel):+ """A stand-in configuration model for surfaces the tests invent."""++ label: str = "hello"+++def build_surface() -> SurfaceBuilder:+ def _build_surface(+ *,+ surface_id: str = EXAMPLE_SURFACE_ID,+ base_file: str = EXAMPLE_BASE_FILE,+ tier_glob: str | None = EXAMPLE_TIER_GLOB,+ ) -> Surface:+ return Surface(+ surface_id=surface_id,+ title=surface_id,+ base_file=base_file,+ tier_glob=tier_glob,+ config_model=ExampleConfig,+ defaults_layer_kind=DefaultsLayerKind.MODEL_DEFAULTS,+ )++ return _build_surface+++def write_ledger_file() -> Callable[..., Path]:+ """Write a minimal ledger TOML into a temporary migration directory."""++ def _write_ledger_file(*, migration_dir: Path, surface_id: str, body: str) -> Path:+ ledgers = migration_dir / "ledgers"+ ledgers.mkdir(parents=True, exist_ok=True)+ path = ledgers / f"{surface_id}.toml"+ path.write_text(body, encoding="utf-8")+ return path++ return _write_ledger_file+++def build_registry(build_surface: SurfaceBuilder) -> Callable[..., SurfaceRegistry]:+ def _build_registry(*, surfaces: list[Surface] | None = None) -> SurfaceRegistry:+ return SurfaceRegistry(surfaces=surfaces or [build_surface()])++ return _build_registrydiff --git a/tests/unit/pipelex/migration/document_strategies.py b/tests/unit/pipelex/migration/document_strategies.pynew file mode 100644index 000000000..6776f804c--- /dev/null+++ b/tests/unit/pipelex/migration/document_strategies.py@@ -0,0 +1,269 @@+"""Generating documents that are valid at a surface's current schema.++Replay neutrality is claimed over *every* document the current models accept, and a witness+document cannot stand in for that domain — the packaged defaults and the kit template each carry+exactly one value per key. This module is the sampler that stands in for it instead;+`test_replay_properties.py` is what asks the sampler questions.++**The fingerprint proposes a mutation; the model decides whether it lands.** That division is the+whole design, and it is forced: *schema membership is decided by validators the fingerprint+cannot see*. Three findings, each one a mutation this generator tried and had to give up on:++- `custom_posthog.mode = "identified"` is a perfectly legal member of a closed set, and it is+ rejected unless the document also carries a `user_id` — a requirement no type records.+- Several nodes typed `dict[str, X]` on the main configuration surface — every+ `effort_to_level_map`, `quality_to_steps_maps` — reject a key outside a fixed set, so their key+ space is not the user's at all despite the annotation saying so. `open_node` is a claim about+ the annotation and a validator can contradict it.+- `cogt.img_gen_config.img_gen_param_defaults.safety_tolerance` carries `Field(le=6)`, which the+ fingerprint does not record today — recording numeric constraints is exactly what the+ value-domain-narrowing rule adds.++The two shapes that answer are both wrong. A sampler that consults only the fingerprint is+confined to *dropping keys* — sound for every surface without looking at anything, because a+user's file is read merged beneath the defaults layer, so a dropped key is restored by the merge —+and a drop-only sampler can never reach a different *value*, which is the whole reason a property+beats the convergence witness. A sampler that mutates whatever the annotation allows and does not+check is red only when it happens to pick the wrong path, and the remedy everyone learns for that+is to grow an exclusion list, which is how a gate dies.++So this one proposes from the fingerprint and checks each proposal against the model, **one+mutation at a time**: a proposal the model rejects is simply not made, and the document keeps the+value it had. Checking per mutation rather than filtering whole documents is what keeps the+sampler from quietly shrinking to nothing — every draw yields a document, and `mutations` reports+what actually survived, so a test can tell a rich sampler from a degenerate one.++Three mutations are proposed:++- **Dropping a path** — any path, including a whole table and a user-owned key beneath an open+ node. Sound for the defaults-layer reason above, and the mutation that matters most, because a+ sparse file is the shape most user files actually have.+- **Swapping an enumerated spelling** for another member of the set the fingerprint records. This+ is the one value shape an operation's precondition can mention: the remap legality rule confines+ a `safe` `remap_value` to an enum-typed path.+- **Flipping a boolean**, whose entire domain is the two values it can hold.++Numbers and free strings are not perturbed at all, and that costs the property nothing: no+operation in the vocabulary has a precondition that mentions one, so perturbing a number or a free+string cannot separate a neutral replay from a non-neutral one.++See `docs/migration-ledger.md` → "Replay neutrality" and "The fingerprint".+"""++from enum import StrEnum+from typing import Any, cast++import tomlkit+from hypothesis import strategies as st+from pydantic import BaseModel, ConfigDict, Field, ValidationError++from pipelex.migration.documents import flatten_document+from pipelex.migration.fingerprint import ENUM_TYPE, PATH_SEPARATOR, PathFingerprint, SurfaceFingerprint, compute_fingerprint+from pipelex.migration.surfaces import Surface+from pipelex.suggested_fix import WILDCARD_SEGMENT+from pipelex.system.configuration.config_surface import strip_reserved_meta+from pipelex.system.exceptions import ConfigValidationError+from pipelex.tools.misc.json_utils import deep_update++BOOL_TYPE = "bool"++MAX_DROPPED_PATHS = 6+MAX_SWAPPED_ENUM_MEMBERS = 4+MAX_FLIPPED_BOOLS = 4+"""Bounds per document. Hypothesis shrinks a set toward the empty one, so the smallest+counterexample any failure reports is the reference document with one mutation on it."""+++class DocumentMutation(StrEnum):+ """A kind of within-schema edit the generator made to a reference document."""++ DROPPED_PATH = "dropped_path"+ SWAPPED_ENUM_MEMBER = "swapped_enum_member"+ FLIPPED_BOOL = "flipped_bool"+++class GeneratedDocument(BaseModel):+ """One sampled document, and what actually survived being made to it."""++ model_config = ConfigDict(frozen=True, extra="forbid")++ text: str+ mutations: list[DocumentMutation] = Field(default_factory=list[DocumentMutation])+++def surface_fingerprint(*, surface: Surface) -> SurfaceFingerprint:+ """The surface's model tree projected against its own defaults, at its current shape."""+ return compute_fingerprint(+ surface_id=surface.surface_id,+ schema_version=1,+ config_model=surface.config_model,+ defaults_document=surface.read_defaults_document(),+ )+++def merge_beneath_defaults(*, surface: Surface, document: dict[str, Any]) -> dict[str, Any]:+ """A file's content as the models will really see it: merged beneath the defaults layer.++ Validating a user's file on its own would report the defaults layer doing its job as a+ failure — an absent optional key is the normal shape of a configuration file, not an error.+ """+ merged: dict[str, Any] = surface.read_defaults_document()+ deep_update(merged, updates=document)+ strip_reserved_meta(config_dict=merged)+ return merged+++def merge_text_beneath_defaults(*, surface: Surface, text: str) -> dict[str, Any]:+ """`merge_beneath_defaults` starting from a document's text rather than its mapping."""+ return merge_beneath_defaults(surface=surface, document=tomlkit.loads(text).unwrap())+++def is_accepted_by_the_surface(*, surface: Surface, document: dict[str, Any]) -> bool:+ """Whether the models accept this document — the only authority on schema membership."""+ try:+ surface.config_model.model_validate(merge_beneath_defaults(surface=surface, document=document))+ except (ValidationError, ConfigValidationError):+ # Both spellings of the same answer: a `ConfigRoot` re-raises pydantic's rejection as its+ # own error, and the surfaces are a mix of the two.+ return False+ return True+++def within_schema_documents(*, surface: Surface) -> st.SearchStrategy[GeneratedDocument]:+ """Documents a user could legitimately have, at the surface's current schema."""+ fingerprint = surface_fingerprint(surface=surface)+ flattened = flatten_document(document=surface.read_defaults_document())++ return _sampled_documents(+ surface=surface,+ fingerprint=fingerprint,+ droppable=sorted(flattened),+ swappable=sorted(path for path, value in flattened.items() if _is_swappable_enum(fingerprint=fingerprint, path=path, value=value)),+ flippable=sorted(path for path, value in flattened.items() if _is_flippable_bool(fingerprint=fingerprint, path=path, value=value)),+ )+++def _sampled_documents( # kw-only: ignore — Hypothesis passes `draw` positionally to a composite's body.+ draw: st.DrawFn,+ *,+ surface: Surface,+ fingerprint: SurfaceFingerprint,+ droppable: list[str],+ swappable: list[str],+ flippable: list[str],+) -> GeneratedDocument:+ dropped = draw(_path_subsets(paths=droppable, max_size=MAX_DROPPED_PATHS))+ swapped = draw(_path_subsets(paths=swappable, max_size=MAX_SWAPPED_ENUM_MEMBERS))+ flipped = draw(_path_subsets(paths=flippable, max_size=MAX_FLIPPED_BOOLS))++ document = surface.read_defaults_document()+ mutations: list[DocumentMutation] = []++ for path in sorted(swapped):+ alternatives = [member for member in _members_at(fingerprint=fingerprint, path=path) if member != _value_at(mapping=document, path=path)]+ if not alternatives:+ continue+ if _assign_if_accepted(surface=surface, document=document, path=path, value=draw(st.sampled_from(alternatives))):+ _record(mutations=mutations, mutation=DocumentMutation.SWAPPED_ENUM_MEMBER)++ for path in sorted(flipped):+ if _assign_if_accepted(surface=surface, document=document, path=path, value=not _value_at(mapping=document, path=path)):+ _record(mutations=mutations, mutation=DocumentMutation.FLIPPED_BOOL)++ # Dropping runs last and is never checked, because the defaults layer restores whatever it+ # removes: the document the models see is the same one they just accepted.+ for path in sorted(dropped):+ if _drop(mapping=document, path=path):+ _record(mutations=mutations, mutation=DocumentMutation.DROPPED_PATH)++ text: str = tomlkit.dumps(document) # pyright: ignore[reportUnknownMemberType]+ return GeneratedDocument(text=text, mutations=mutations)+++def _record(*, mutations: list[DocumentMutation], mutation: DocumentMutation) -> None:+ if mutation not in mutations:+ mutations.append(mutation)+++def _path_subsets(*, paths: list[str], max_size: int) -> st.SearchStrategy[set[str]]:+ if not paths:+ return st.just(set[str]())+ return st.sets(st.sampled_from(paths), max_size=max_size)+++def _is_swappable_enum(*, fingerprint: SurfaceFingerprint, path: str, value: Any) -> bool:+ entry = fingerprint_at(fingerprint=fingerprint, path=path)+ if entry is None or entry.value_type != ENUM_TYPE or not entry.enum_members:+ return False+ return isinstance(value, str) and len(entry.enum_members) > 1+++def _is_flippable_bool(*, fingerprint: SurfaceFingerprint, path: str, value: Any) -> bool:+ entry = fingerprint_at(fingerprint=fingerprint, path=path)+ return entry is not None and entry.value_type == BOOL_TYPE and isinstance(value, bool)+++def fingerprint_at(*, fingerprint: SurfaceFingerprint, path: str) -> PathFingerprint | None:+ """What the fingerprint says about a *document* path, `None` when it says nothing.++ A document path and a fingerprint path are not the same language: beneath an open node the+ document carries the user's own key where the fingerprint carries a `*`. Walking segment by+ segment, and substituting `*` the moment the parent is an open node, translates one into the+ other — including through nesting, where `dict[str, dict[str, int]]` addresses `a.*.*`.+ """+ translated: list[str] = []+ for segment in path.split(PATH_SEPARATOR):+ parent = fingerprint.paths.get(PATH_SEPARATOR.join(translated)) if translated else None+ translated.append(WILDCARD_SEGMENT if parent is not None and parent.open_node else segment)+ if PATH_SEPARATOR.join(translated) not in fingerprint.paths:+ return None+ return fingerprint.paths[PATH_SEPARATOR.join(translated)]+++def _members_at(*, fingerprint: SurfaceFingerprint, path: str) -> list[str]:+ entry = fingerprint_at(fingerprint=fingerprint, path=path)+ return list(entry.enum_members) if entry is not None and entry.enum_members else []+++def _assign_if_accepted(*, surface: Surface, document: dict[str, Any], path: str, value: Any) -> bool:+ """Write a value, keep it only if the models still accept the document, and say which."""+ located = _parent_of(mapping=document, path=path)+ if located is None:+ return False+ parent, key = located+ previous = parent[key]+ parent[key] = value+ if is_accepted_by_the_surface(surface=surface, document=document):+ return True+ parent[key] = previous+ return False+++def _parent_of(*, mapping: dict[str, Any], path: str) -> tuple[dict[str, Any], str] | None:+ segments = path.split(PATH_SEPARATOR)+ node = mapping+ for segment in segments[:-1]:+ child = node.get(segment)+ if not isinstance(child, dict):+ return None+ node = cast("dict[str, Any]", child)+ return (node, segments[-1]) if segments[-1] in node else None+++def _value_at(*, mapping: dict[str, Any], path: str) -> Any:+ located = _parent_of(mapping=mapping, path=path)+ return located[0][located[1]] if located is not None else None+++def _drop(*, mapping: dict[str, Any], path: str) -> bool:+ """Remove a path and, when it names a table, everything beneath it.++ A path whose parent an earlier drop already removed is simply absent, which is why this+ reports rather than insists: the drawn set is a set of paths, not a set of independent edits.+ """+ located = _parent_of(mapping=mapping, path=path)+ if located is None:+ return False+ del located[0][located[1]]+ return Truediff --git a/tests/unit/pipelex/migration/test_coverage_gate.py b/tests/unit/pipelex/migration/test_coverage_gate.pyindex 27c3f0451..0f7fb6737 100644--- a/tests/unit/pipelex/migration/test_coverage_gate.py+++ b/tests/unit/pipelex/migration/test_coverage_gate.py@@ -14,8 +14,9 @@ the golden the gate will compare against, hand it a ledger entry, and assert whifrom enum import StrEnumfrom pathlib import Path+from typing import Annotated, Literal-from pydantic import BaseModel+from pydantic import BaseModel, Fieldfrom pipelex.migration.coverage import CoverageIssue, CoverageIssueKind, check_surface, diff_fingerprintsfrom pipelex.migration.fingerprint import compute_fingerprint@@ -70,6 +71,86 @@ class _SchemaTwoTierRenamedAndMemberGone(BaseModel):level: _TierWithoutBasic = _TierWithoutBasic.PREMIUM+class _Section(BaseModel):+ """A table with two keys, one of them optional with no value in any reference document."""++ limit: int | None = None+ name: str = "x"+++class _SchemaOneWithSection(BaseModel):+ label: str = "hello"+ section: _Section = Field(default_factory=_Section)+++class _SchemaTwoSectionRenamed(BaseModel):+ """`section` renamed to `area`, both children kept."""++ label: str = "hello"+ area: _Section = Field(default_factory=_Section)+++class _BoundedOne(BaseModel):+ """A shape with a bounded number beside an ordinary key — the starting point for narrowings."""++ label: str = "hello"+ retries: int = Field(default=3, ge=1)+++class _BoundedTightened(BaseModel):+ """The bound moved up. Every path survives; a file saying `retries = 2` stops validating."""++ label: str = "hello"+ retries: int = Field(default=3, ge=2)+++class _BoundedRelaxed(BaseModel):+ """The bound is gone. Every file that validated before still does."""++ label: str = "hello"+ retries: int = 3+++class _BoundedTightenedAndRenamed(BaseModel):+ """A tightened bound riding along with a rename, the way a real schema version mixes changes."""++ title: str = "hello"+ retries: int = Field(default=3, ge=2)+++class _BoundedOrAuto(BaseModel):+ """A bounded number that may also be spelled `auto` — the `int | literal` shape real surfaces carry."""++ label: str = "hello"+ retries: Annotated[int, Field(ge=1)] | Literal["auto", "unbounded"] = 3+++class _BoundedTightenedOrUnbounded(BaseModel):+ """`auto` is gone *and* the bound moved up: a remap answers for the first, nothing but `unsafe` for the second."""++ label: str = "hello"+ retries: Annotated[int, Field(ge=8)] | Literal["unbounded"] = 8+++class _SpelledOnly(BaseModel):+ """The number is gone altogether: only one spelling survives, and `retries = 4` is out of domain."""++ label: str = "hello"+ retries: Literal["unbounded"] = "unbounded"+++class _FreeStringTier(BaseModel):+ """`tier` accepts any string."""++ tier: str = "basic"+++class _EnumeratedTier(BaseModel):+ """`tier` accepts two spellings and nothing else."""++ tier: _Tier = _Tier.BASIC++class _SchemaOneWithBothNames(BaseModel):"""A starting shape that already has `title`, so a rename of `label` onto it collides."""@@ -141,6 +222,27 @@ new_key = "{new_key}""""+def _pre_history_entry() -> str:+ """An entry about `legacy_mode`, a key no fingerprint in this chain has ever recorded."""+ return f"""+[[migration]]+id = "{SURFACE_ID}@2"+to_schema_version = 2+introduced_in = "0.46.0"+breaking = true+safety = "safe"+title = "Drop a key that predates the chain"+description = "It was gone before anything was snapshotted."+pre_history = true+declared_removed_paths = ["legacy_mode"]++[[migration.ops]]+kind = "delete_key"+table_path = []+key = "legacy_mode"+"""++class TestTheSteadyState:def test_a_surface_whose_golden_matches_its_models_passes(self, tmp_path: Path) -> None:"""The overwhelmingly common state: nothing changed, so nothing is owed."""@@ -211,6 +313,31 @@ class TestTheHeadLink:assert "schema version 1" in issues[0].message+class TestThePreHistoryClaim:+ def test_a_pre_history_entry_over_an_unmoved_pair_is_green(self, tmp_path: Path) -> None:+ """The whole point of the flag: the change it describes happened before any of this was snapshotted."""+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=_pre_history_entry())+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=2)+ assert check_surface(surface=_surface(config_model=_SchemaOne), migration_dir=tmp_path) == []++ def test_a_pre_history_entry_hiding_a_real_removal_is_refused(self, tmp_path: Path) -> None:+ """The flag exempts an entry from accounting, so a change with an observable diff must not carry it."""+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=_pre_history_entry())+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaTwoRemoved, schema_version=2)+ issues = check_surface(surface=_surface(config_model=_SchemaTwoRemoved), migration_dir=tmp_path)+ assert CoverageIssueKind.PRE_HISTORY_HAS_A_DIFF in _kinds(issues)+ assert "label" in "".join(issue.message for issue in issues)++ def test_a_pre_history_entry_beside_an_addition_stays_green(self, tmp_path: Path) -> None:+ """Additions are absorbed by the defaults layer, so they are nobody's accounting — the flag's included."""+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=_pre_history_entry())+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOneWithBothNames, schema_version=2)+ assert check_surface(surface=_surface(config_model=_SchemaOneWithBothNames), migration_dir=tmp_path) == []++class TestEntryAccounting:def test_a_correct_rename_entry_is_green(self, tmp_path: Path) -> None:_write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=_rename_entry())@@ -276,6 +403,63 @@ key = "tier"assert _kinds(issues) == [CoverageIssueKind.OVER_DELETION]assert "tier" in issues[0].message+ def test_an_over_deletion_beneath_a_renamed_table_is_caught(self, tmp_path: Path) -> None:+ """The deleted child is `section.limit` in the old spelling and `area.limit` in the new, so a+ comparison by current spelling never lines the two up. Compared by origin, the entry removes a+ path the new schema still has — and the convergence witness cannot see it either, because+ the child is optional and no reference document carries it.+ """+ entry = f"""+[[migration]]+id = "{SURFACE_ID}@2"+to_schema_version = 2+introduced_in = "0.46.0"+breaking = true+safety = "safe"+title = "Rename the section, and drop too much beneath it"+description = "Deletes area.limit as well, which schema 2 still has."++[[migration.ops]]+kind = "rename_table_key"+table_path = []+key = "section"+new_key = "area"++[[migration.ops]]+kind = "delete_key"+table_path = ["area"]+key = "limit"+"""+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=entry)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOneWithSection, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaTwoSectionRenamed, schema_version=2)+ issues = check_surface(surface=_surface(config_model=_SchemaTwoSectionRenamed), migration_dir=tmp_path)+ assert _kinds(issues) == [CoverageIssueKind.OVER_DELETION]+ assert "'section.limit'" in issues[0].message+ assert "'area.limit'" in issues[0].message++ def test_a_rename_that_keeps_every_child_is_green(self, tmp_path: Path) -> None:+ entry = f"""+[[migration]]+id = "{SURFACE_ID}@2"+to_schema_version = 2+introduced_in = "0.46.0"+breaking = true+safety = "safe"+title = "Rename the section"+description = "Nothing beneath it changes."++[[migration.ops]]+kind = "rename_table_key"+table_path = []+key = "section"+new_key = "area"+"""+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=entry)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOneWithSection, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaTwoSectionRenamed, schema_version=2)+ assert check_surface(surface=_surface(config_model=_SchemaTwoSectionRenamed), migration_dir=tmp_path) == []+def test_an_operation_whose_source_never_existed_is_caught(self, tmp_path: Path) -> None:"""A dead operation is silent forever: it skips on every file and reports success."""entry = f"""@@ -441,60 +625,149 @@ mapping = { basic = "standard" }_snapshot(migration_dir=tmp_path, config_model=_SchemaTwoEnumMemberGone, schema_version=2)assert check_surface(surface=_surface(config_model=_SchemaTwoEnumMemberGone), migration_dir=tmp_path) == []- def test_a_safe_remap_of_a_value_that_is_still_legal_is_refused(self, tmp_path: Path) -> None:- """The remap legality rule, and the reason replay neutrality holds.- `premium` is still a legal tier at schema 2, so a user who chose it deliberately would- have it rewritten on every run — and replay over a current-valid file would not be a no-op.+class TestValueDomainNarrowing:+ """The change that keeps every path and every spelling, and still breaks a user's file.++ Without the direction split these all read as additive — nothing was removed — so the gate+ would answer "regenerate the golden", demand no bump and no entry, and the next boot would+ reject a file that was valid the day before with a green gate behind it.+ """++ def _entry_with_ops(self, *, safety: str, ops: str) -> str:+ return f"""+[[migration]]+id = "{SURFACE_ID}@2"+to_schema_version = 2+introduced_in = "0.46.0"+breaking = true+safety = "{safety}"+title = "Narrow what a value may be"+description = "The domain shrank."+{ops}+"""++ def test_a_tightened_bound_without_a_bump_is_refused(self, tmp_path: Path) -> None:+ _write_ledger(migration_dir=tmp_path, current_schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_BoundedOne, schema_version=1)+ issues = check_surface(surface=_surface(config_model=_BoundedTightened), migration_dir=tmp_path)+ assert _kinds(issues) == [CoverageIssueKind.REMOVAL_NEEDS_A_BUMP]+ assert "lower bound tightened from ge=1 to ge=2" in issues[0].message++ def test_a_free_string_becoming_enumerated_without_a_bump_is_refused(self, tmp_path: Path) -> None:+ """Every spelling the file could carry is now checked against a closed set it may fail."""+ _write_ledger(migration_dir=tmp_path, current_schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_FreeStringTier, schema_version=1)+ issues = check_surface(surface=_surface(config_model=_EnumeratedTier), migration_dir=tmp_path)+ assert _kinds(issues) == [CoverageIssueKind.REMOVAL_NEEDS_A_BUMP]+ assert "its type went from 'str' to 'enum'" in issues[0].message++ def test_a_relaxed_bound_asks_only_for_a_regeneration(self, tmp_path: Path) -> None:+ """The other direction has to stay cheap, or the gate is one an author learns to fight."""+ _write_ledger(migration_dir=tmp_path, current_schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_BoundedOne, schema_version=1)+ issues = check_surface(surface=_surface(config_model=_BoundedRelaxed), migration_dir=tmp_path)+ assert _kinds(issues) == [CoverageIssueKind.FINGERPRINT_DRIFTED]++ def test_an_enumerated_path_relaxed_into_a_free_string_asks_only_for_a_regeneration(self, tmp_path: Path) -> None:+ """The member set empties, and a raw set difference would call that the loss of every spelling.++ This is the widening most likely to be attempted in practice, and demanding a bump and a+ remap for each spelling would be the gate at its most obviously wrong."""+ _write_ledger(migration_dir=tmp_path, current_schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_EnumeratedTier, schema_version=1)+ issues = check_surface(surface=_surface(config_model=_FreeStringTier), migration_dir=tmp_path)+ assert _kinds(issues) == [CoverageIssueKind.FINGERPRINT_DRIFTED]++ def test_a_bumped_entry_that_does_not_account_for_the_narrowing_is_refused(self, tmp_path: Path) -> None:+ """The entry explains the rename it made and says nothing about the bound it moved."""+ ops = """+[[migration.ops]]+kind = "rename_table_key"+table_path = []+key = "label"+new_key = "title"+"""+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=self._entry_with_ops(safety="safe", ops=ops))+ _snapshot(migration_dir=tmp_path, config_model=_BoundedOne, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_BoundedTightenedAndRenamed, schema_version=2)+ issues = check_surface(surface=_surface(config_model=_BoundedTightenedAndRenamed), migration_dir=tmp_path)+ assert _kinds(issues) == [CoverageIssueKind.VALUE_DOMAIN_NARROWED]+ assert "'retries'" in issues[0].message++ def test_a_remap_on_the_narrowed_path_accounts_for_it(self, tmp_path: Path) -> None:+ """The remedy that repairs the file rather than only warning about it, where it applies."""ops = """[[migration.ops]]kind = "remap_value"table_path = []key = "tier"-mapping = { basic = "standard", premium = "standard" }+mapping = { entry = "basic" }"""_write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=self._entry_with_ops(safety="safe", ops=ops))- _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)- _snapshot(migration_dir=tmp_path, config_model=_SchemaTwoEnumMemberGone, schema_version=2)- issues = check_surface(surface=_surface(config_model=_SchemaTwoEnumMemberGone), migration_dir=tmp_path)- assert CoverageIssueKind.ILLEGAL_REMAP in _kinds(issues)- assert "premium" in " ".join(issue.message for issue in issues)+ _snapshot(migration_dir=tmp_path, config_model=_FreeStringTier, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_EnumeratedTier, schema_version=2)+ assert check_surface(surface=_surface(config_model=_EnumeratedTier), migration_dir=tmp_path) == []- def test_a_remap_to_a_spelling_the_new_schema_rejects_is_refused(self, tmp_path: Path) -> None:- """The value-side twin of the misspelled rename destination.+ def test_a_remap_on_a_path_does_not_answer_for_a_bound_tightened_on_that_same_path(self, tmp_path: Path) -> None:+ """A remap rewrites spellings; the number `retries = 4` is not a spelling and no mapping reaches it.- Rewriting to a spelling the new schema does not accept migrates every file to something- the model rejects, with the tool reporting success.+ Without this the remap that retires `auto` would carry the tightened bound into a `safe`+ entry, and a file saying `retries = 4` would fail at boot behind a green gate."""ops = """[[migration.ops]]kind = "remap_value"table_path = []-key = "tier"-mapping = { basic = "standrad" }+key = "retries"+mapping = { auto = "unbounded" }"""_write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=self._entry_with_ops(safety="safe", ops=ops))- _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)- _snapshot(migration_dir=tmp_path, config_model=_SchemaTwoEnumMemberGone, schema_version=2)- issues = check_surface(surface=_surface(config_model=_SchemaTwoEnumMemberGone), migration_dir=tmp_path)- assert CoverageIssueKind.ILLEGAL_REMAP in _kinds(issues)- assert "standrad" in " ".join(issue.message for issue in issues)-- def test_a_safe_remap_of_a_free_string_is_refused(self, tmp_path: Path) -> None:- """Where the schema does not enumerate, staleness cannot be proven and the entry must be unsafe."""+ _snapshot(migration_dir=tmp_path, config_model=_BoundedOrAuto, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_BoundedTightenedOrUnbounded, schema_version=2)+ issues = check_surface(surface=_surface(config_model=_BoundedTightenedOrUnbounded), migration_dir=tmp_path)+ assert _kinds(issues) == [CoverageIssueKind.VALUE_DOMAIN_NARROWED]+ assert "lower bound tightened from ge=1 to ge=8" in issues[0].message+ assert "its type went from" not in issues[0].message++ def test_a_remap_on_a_path_does_not_answer_for_a_non_string_member_that_path_lost(self, tmp_path: Path) -> None:+ """The other way a remap falls short: `int | literal` becomes `literal`. The remap rewrites the+ spelling it names and never touches the number, so `retries = 4` fails at boot — the type+ half of narrowing must still run for a remapped origin, exempting only what a remap can+ rewrite.+ """ops = """[[migration.ops]]kind = "remap_value"table_path = []-key = "label"-mapping = { hello = "hi" }+key = "retries"+mapping = { auto = "unbounded" }"""_write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=self._entry_with_ops(safety="safe", ops=ops))- _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)- _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=2)- issues = check_surface(surface=_surface(config_model=_SchemaOne), migration_dir=tmp_path)- assert CoverageIssueKind.ILLEGAL_REMAP in _kinds(issues)+ _snapshot(migration_dir=tmp_path, config_model=_BoundedOrAuto, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_SpelledOnly, schema_version=2)+ issues = check_surface(surface=_surface(config_model=_SpelledOnly), migration_dir=tmp_path)+ assert _kinds(issues) == [CoverageIssueKind.VALUE_DOMAIN_NARROWED]+ assert "its type went from 'int | literal' to 'literal'" in issues[0].message++ def test_an_unsafe_entry_may_leave_a_narrowing_unremapped(self, tmp_path: Path) -> None:+ """For a tightened numeric bound this is the only remedy there is: no mapping can enumerate+ the values a bound retires, so the migration is reported to the user and never applied.+ """+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=self._entry_with_ops(safety="unsafe", ops=""))+ _snapshot(migration_dir=tmp_path, config_model=_BoundedOne, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_BoundedTightened, schema_version=2)+ assert check_surface(surface=_surface(config_model=_BoundedTightened), migration_dir=tmp_path) == []++ def test_a_pre_history_entry_cannot_hide_a_narrowing_either(self, tmp_path: Path) -> None:+ """The flag exempts an entry from accounting, so it must not be a way past this one."""+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=_pre_history_entry())+ _snapshot(migration_dir=tmp_path, config_model=_BoundedOne, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_BoundedTightened, schema_version=2)+ issues = check_surface(surface=_surface(config_model=_BoundedTightened), migration_dir=tmp_path)+ assert _kinds(issues) == [CoverageIssueKind.PRE_HISTORY_HAS_A_DIFF]+ assert "'retries'" in issues[0].messageclass TestTheDefaultsLayerRule:diff --git a/tests/unit/pipelex/migration/test_engine.py b/tests/unit/pipelex/migration/test_engine.pynew file mode 100644index 000000000..746bc2697--- /dev/null+++ b/tests/unit/pipelex/migration/test_engine.py@@ -0,0 +1,271 @@+"""Unit tests for the replay engine — what a ledger does to one document.++The guarantees under test are the ones a user's machine depends on: every run replays everything+and the applier skips what is already done, an `unsafe` entry is never written and stays silent on+a file that does not need it, and serialization contributes no changes of its own.+"""++from typing import Any, cast++import tomlkit++from pipelex.migration.engine import replay_ledger_over_text+from pipelex.migration.ledger import MigrationSafety+from pipelex.migration.plan import BlockedEntryReason+from pipelex.suggested_fix import DeleteKeyOp, MoveKeyOp, RenameTableKeyOp+from tests.unit.pipelex.migration.conftest import EntryBuilder, LedgerBuilder++OLD_SHAPE = """\+# A user's file, with their own comments and spacing.+[reporting]+output_config = { directory = "out" }+retention = 30++[legacy_reporting]+enabled = true+"""++CURRENT_SHAPE = """\+[reporting]+output = { directory = "out" }++[storage]+retention = 30+"""+++class TestReplayEngine:+ def test_a_safe_entry_applies_and_reports_only_the_operations_that_fired(+ self,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ ) -> None:+ entry = build_entry(+ to_schema_version=2,+ ops=[+ RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output"),+ MoveKeyOp(table_path=["reporting"], key="retention", new_table_path=["storage"], new_key="retention"),+ DeleteKeyOp(table_path=["reporting"], key="never_existed"),+ ],+ )+ ledger = build_ledger(entries=[entry])++ replay = replay_ledger_over_text(ledger=ledger, text=OLD_SHAPE)++ assert not replay.blocked+ assert len(replay.steps) == 1+ step = replay.steps[0]+ assert step.entry_id == "example-config@2"+ assert [op.kind for op in step.applied_ops] == ["rename_table_key", "move_key"]+ assert replay.did_change_document+ assert "output = " in replay.text+ assert "[storage]" in replay.text++ def test_replay_over_an_already_current_file_is_a_byte_level_no_op(+ self,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ ) -> None:+ """Replay neutrality, stated over a file the current schema accepts.++ Every operation's source is material some schema version removed, so on a current file+ every operation skips — and skipping mutates nothing, so the bytes are identical.+ """+ ledger = build_ledger(+ entries=[+ build_entry(+ to_schema_version=2,+ ops=[+ RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output"),+ MoveKeyOp(table_path=["reporting"], key="retention", new_table_path=["storage"], new_key="retention"),+ ],+ )+ ]+ )++ replay = replay_ledger_over_text(ledger=ledger, text=CURRENT_SHAPE)++ assert not replay.steps+ assert not replay.blocked+ assert not replay.did_change_document+ assert replay.text == CURRENT_SHAPE++ def test_replaying_twice_equals_replaying_once(+ self,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ ) -> None:+ ledger = build_ledger(+ entries=[+ build_entry(+ to_schema_version=2,+ ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")],+ )+ ]+ )++ once = replay_ledger_over_text(ledger=ledger, text=OLD_SHAPE)+ twice = replay_ledger_over_text(ledger=ledger, text=once.text)++ assert not twice.steps+ assert twice.text == once.text++ def test_entries_compose_in_sequence_for_a_file_coming_from_the_first_version(+ self,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ ) -> None:+ """A key renamed at 2 and moved at 3 lands correctly for a file still at 1.++ This is also the shape that catches the tomlkit rename-then-address defect the engine+ re-reads the document to work around: without the re-read, the move raises `KeyError`+ from inside the library rather than relocating the key.+ """+ ledger = build_ledger(+ entries=[+ build_entry(+ to_schema_version=2,+ ops=[RenameTableKeyOp(table_path=["reporting"], key="retention", new_key="keep_days")],+ ),+ build_entry(+ to_schema_version=3,+ ops=[MoveKeyOp(table_path=["reporting"], key="keep_days", new_table_path=["storage"], new_key="keep_days")],+ ),+ ]+ )++ replay = replay_ledger_over_text(ledger=ledger, text=OLD_SHAPE)++ assert [step.to_schema_version for step in replay.steps] == [2, 3]+ migrated = cast("dict[str, Any]", tomlkit.loads(replay.text))+ assert migrated["storage"]["keep_days"] == 30+ assert "retention" not in migrated["reporting"]++ def test_an_unsafe_entry_is_reported_without_writing_anything(+ self,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ ) -> None:+ ledger = build_ledger(+ entries=[+ build_entry(+ to_schema_version=2,+ safety=MigrationSafety.UNSAFE,+ guidance="Decide which reporting directory you want.",+ ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")],+ )+ ]+ )++ replay = replay_ledger_over_text(ledger=ledger, text=OLD_SHAPE)++ assert not replay.steps+ assert len(replay.blocked) == 1+ blocked = replay.blocked[0]+ assert blocked.reason is BlockedEntryReason.UNSAFE+ assert blocked.guidance == "Decide which reporting directory you want."+ assert not blocked.applied_ops+ assert not replay.did_change_document+ assert replay.text == OLD_SHAPE++ def test_an_unsafe_entry_with_nothing_to_do_stays_silent(+ self,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ ) -> None:+ """Otherwise every user with a perfectly current file is warned at every boot, forever."""+ ledger = build_ledger(+ entries=[+ build_entry(+ to_schema_version=2,+ safety=MigrationSafety.UNSAFE,+ ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")],+ )+ ]+ )++ replay = replay_ledger_over_text(ledger=ledger, text=CURRENT_SHAPE)++ assert not replay.blocked+ assert not replay.steps++ def test_a_conflicting_entry_is_blocked_and_still_names_what_it_did_apply(+ self,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ ) -> None:+ """A user who hand-fixed half of a migration gets a conflict, not a clobbered file.++ The entry is reported once — in `blocked[]`, carrying the operations that did land — so+ the report never claims an entry arrived whole when part of it could not.+ """+ half_fixed = """\+[reporting]+output_config = { directory = "out" }+output = { directory = "mine" }+retention = 30+"""+ ledger = build_ledger(+ entries=[+ build_entry(+ to_schema_version=2,+ ops=[+ MoveKeyOp(table_path=["reporting"], key="retention", new_table_path=["storage"], new_key="retention"),+ RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output"),+ ],+ )+ ]+ )++ replay = replay_ledger_over_text(ledger=ledger, text=half_fixed)++ assert not replay.steps+ assert len(replay.blocked) == 1+ blocked = replay.blocked[0]+ assert blocked.reason is BlockedEntryReason.CONFLICT+ assert [op.kind for op in blocked.applied_ops] == ["move_key"]+ assert "already present" in blocked.detail++ def test_a_replay_that_applies_nothing_returns_the_very_text_it_was_given(+ self,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ ) -> None:+ """Byte-level neutrality is a property of this code, not of the TOML library.++ A run in which every operation skips returns the input string untouched, so no round-trip+ through the serializer can contribute a change of its own.+ """+ ledger = build_ledger(entries=[build_entry(to_schema_version=2, ops=[DeleteKeyOp(table_path=["nowhere"], key="gone")])])++ replay = replay_ledger_over_text(ledger=ledger, text=OLD_SHAPE)++ assert replay.text is OLD_SHAPE++ def test_a_report_never_echoes_a_value_read_from_the_users_file(+ self,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ ) -> None:+ secret_bearing = """\+[reporting]+output_config = { directory = "out" }+api_key = "sk-live-DO-NOT-RENDER-THIS"+"""+ ledger = build_ledger(+ entries=[+ build_entry(+ to_schema_version=2,+ safety=MigrationSafety.UNSAFE,+ guidance="Move your reporting settings by hand.",+ ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")],+ )+ ]+ )++ replay = replay_ledger_over_text(ledger=ledger, text=secret_bearing)++ # The entry must actually be reported — a report of nothing renders no value trivially.+ assert [blocked.entry_id for blocked in replay.blocked] == ["example-config@2"]+ rendered = replay.model_dump_json(exclude={"text"})+ assert "DO-NOT-RENDER-THIS" not in rendereddiff --git a/tests/unit/pipelex/migration/test_fingerprint.py b/tests/unit/pipelex/migration/test_fingerprint.pyindex 95315fbae..5190c9130 100644--- a/tests/unit/pipelex/migration/test_fingerprint.py+++ b/tests/unit/pipelex/migration/test_fingerprint.py@@ -8,12 +8,13 @@ red on every legitimate configuration change and teach everyone to regenerate ra"""from collections.abc import Mapping+from decimal import Decimalfrom enum import StrEnumfrom typing import Annotated, Any, Literalfrom pydantic import BaseModel, Field-from pipelex.migration.fingerprint import ENUM_TYPE, TABLE_TYPE, compute_fingerprint+from pipelex.migration.fingerprint import ENUM_TYPE, TABLE_TYPE, ConstraintKind, compute_fingerprintclass _Mode(StrEnum):@@ -34,6 +35,10 @@ class _Synthetic(BaseModel):labels: dict[str, str]tags: list[str]bounded: Annotated[int, Field(ge=1)] | Literal["unbounded"] = 1+ retries: int = Field(default=3, ge=0, le=10)+ lenient: str = Field(default="x", strict=False)+ item_bounded: list[Annotated[int, Field(ge=1)]] = Field(default_factory=list[int])+ amount: Decimal = Field(default=Decimal(1), ge=Decimal(0))class _SelfReferential(BaseModel):@@ -77,6 +82,10 @@ class TestFingerprintPaths:"labels.*","tags","bounded",+ "retries",+ "lenient",+ "item_bounded",+ "amount",}def test_paths_are_stably_ordered(self) -> None:@@ -151,6 +160,40 @@ class TestFingerprintRecords:"""`Annotated[int, Field(ge=1)]` renders as `int` — a pydantic upgrade must not move the golden."""assert _fingerprint().paths["bounded"].value_type == "int | literal"+ def test_a_whitelisted_bound_is_recorded_under_its_own_key(self) -> None:+ """A tightened bound breaks a user's file while moving nothing else in the projection."""+ assert _fingerprint().paths["retries"].constraints == {ConstraintKind.GE: 0, ConstraintKind.LE: 10}++ def test_a_bound_pydantic_folded_into_the_field_metadata_is_still_found(self) -> None:+ """`Field(ge=0)` never appears in the annotation — pydantic moves it to the field's metadata,+ so a walk reading the annotation alone would be blind to every bound declared the usual way.+ """+ assert ConstraintKind.GE in (_fingerprint().paths["retries"].constraints or {})++ def test_a_bound_nested_inside_a_union_member_is_found_too(self) -> None:+ """The shape `Annotated[int, Field(ge=1)] | Literal["unbounded"]`, which the type rendering strips."""+ assert _fingerprint().paths["bounded"].constraints == {ConstraintKind.GE: 1}++ def test_metadata_outside_the_whitelist_is_dropped_rather_than_serialized(self) -> None:+ """The closed whitelist is what keeps the golden a function of our schema.++ `Field(strict=False)` is on dozens of configuration fields as a house convention. Recording+ it — or any other carrier the validation library invents — would move goldens for reasons+ that have nothing to do with what a user's file may contain.+ """+ assert _fingerprint().paths["lenient"].constraints is None++ def test_a_bound_on_a_container_item_is_not_attributed_to_the_container(self) -> None:+ """`list[Annotated[int, Field(ge=1)]]` bounds the items, not the list."""+ assert _fingerprint().paths["item_bounded"].constraints is None++ def test_a_bound_over_a_non_numeric_value_is_dropped(self) -> None:+ """A `Decimal` (or a date) bound is outside the projection — the whitelist is over kinds *and* value types."""+ assert _fingerprint().paths["amount"].constraints is None++ def test_a_path_with_no_bound_records_none(self) -> None:+ assert _fingerprint().paths["leaf.name"].constraints is None+def test_a_nested_model_renders_as_a_table(self) -> None:assert _fingerprint().paths["leaf"].value_type == TABLE_TYPEdiff --git a/tests/unit/pipelex/migration/test_ledger_check.py b/tests/unit/pipelex/migration/test_ledger_check.pynew file mode 100644index 000000000..5711ed8c9--- /dev/null+++ b/tests/unit/pipelex/migration/test_ledger_check.py@@ -0,0 +1,506 @@+"""Unit tests for the ledger check — what an entry may say, and whether replaying it is harmless.++Every test here points the check at a **synthetic** surface with a fixture ledger and a fixture+golden chain in a temporary directory, for the reason the whole registry is an injected parameter:+wire the real configuration models in and every legitimate configuration change turns this suite+red alongside the gate, which is how a gate goes permanently green while catching nothing. The real+surfaces get one smoke test, in `test_real_surfaces.py`.++The shape of each test is the same: describe the schema versions as synthetic models, snapshot the+chain the check will read, hand it a ledger entry, and assert which guarantee broke.+"""++from enum import StrEnum+from pathlib import Path++import pytest+from pydantic import BaseModel, Field++from pipelex.migration.exceptions import MigrationLedgerError+from pipelex.migration.fingerprint import compute_fingerprint+from pipelex.migration.goldens import write_fingerprint_golden+from pipelex.migration.ledger import ledgers_dir+from pipelex.migration.ledger_check import LedgerIssue, LedgerIssueKind, check_ledger+from pipelex.migration.surfaces import DefaultsLayerKind, Surface++SURFACE_ID = "synthetic-config"+++class _Tier(StrEnum):+ BASIC = "basic"+ PREMIUM = "premium"+++class _TierWithoutBasic(StrEnum):+ PREMIUM = "premium"+ STANDARD = "standard"+++class _DeckEntry(BaseModel):+ old_name: str = "a"+++class _DeckEntryRenamed(BaseModel):+ new_name: str = "a"+++class _Settings(BaseModel):+ """A plain nested table — deliberately not an open mapping, so `*` has no business here."""++ retries: int = 3+++class _SchemaOne(BaseModel):+ """The starting shape: two keys, one of them enumerated, plus a table and an open mapping."""++ label: str = "hello"+ tier: _Tier = _Tier.BASIC+ settings: _Settings = Field(default_factory=_Settings)+ deck: dict[str, _DeckEntry] = Field(default_factory=dict[str, _DeckEntry])+++class _SchemaTwoRenamed(BaseModel):+ """`label` renamed to `title`."""++ title: str = "hello"+ tier: _Tier = _Tier.BASIC+ settings: _Settings = Field(default_factory=_Settings)+ deck: dict[str, _DeckEntry] = Field(default_factory=dict[str, _DeckEntry])+++class _SchemaTwoDeckFieldRenamed(BaseModel):+ """A field renamed inside every entry of the open mapping."""++ label: str = "hello"+ tier: _Tier = _Tier.BASIC+ settings: _Settings = Field(default_factory=_Settings)+ deck: dict[str, _DeckEntryRenamed] = Field(default_factory=dict[str, _DeckEntryRenamed])+++class _SchemaTwoEnumMemberGone(BaseModel):+ """`basic` is no longer a legal spelling of `tier`."""++ label: str = "hello"+ tier: _TierWithoutBasic = _TierWithoutBasic.PREMIUM+ settings: _Settings = Field(default_factory=_Settings)+ deck: dict[str, _DeckEntry] = Field(default_factory=dict[str, _DeckEntry])+++class _SchemaTwoLabelGone(BaseModel):+ """`label` dropped outright."""++ tier: _Tier = _Tier.BASIC+ settings: _Settings = Field(default_factory=_Settings)+ deck: dict[str, _DeckEntry] = Field(default_factory=dict[str, _DeckEntry])+++class _TierWithBasicBack(StrEnum):+ BASIC = "basic"+ PREMIUM = "premium"+ STANDARD = "standard"+++class _SchemaThreeBasicBack(BaseModel):+ """`basic` is a legal spelling of `tier` again — the reuse of a retired spelling — and `label` is renamed."""++ title: str = "hello"+ tier: _TierWithBasicBack = _TierWithBasicBack.STANDARD+ settings: _Settings = Field(default_factory=_Settings)+ deck: dict[str, _DeckEntry] = Field(default_factory=dict[str, _DeckEntry])+++class _SchemaThreeStaysRetired(BaseModel):+ """Schema three keeps `basic` retired; only `label` is renamed."""++ title: str = "hello"+ tier: _TierWithoutBasic = _TierWithoutBasic.STANDARD+ settings: _Settings = Field(default_factory=_Settings)+ deck: dict[str, _DeckEntry] = Field(default_factory=dict[str, _DeckEntry])+++class _SchemaThreeLabelBack(BaseModel):+ """`label` is back — the reuse of a retired name — and `tier` is renamed to `level`."""++ label: str = "hello"+ level: _Tier = _Tier.BASIC+ settings: _Settings = Field(default_factory=_Settings)+ deck: dict[str, _DeckEntry] = Field(default_factory=dict[str, _DeckEntry])+++def _surface(+ *,+ config_model: type[BaseModel],+ packaged_document_path: Path | None = None,+ kit_template_path: Path | None = None,+) -> Surface:+ defaults_layer_kind = DefaultsLayerKind.PACKAGED_DOCUMENT if packaged_document_path is not None else DefaultsLayerKind.MODEL_DEFAULTS+ return Surface(+ surface_id=SURFACE_ID,+ title="A synthetic surface",+ base_file="synthetic.toml",+ tier_glob="synthetic_*.toml",+ config_model=config_model,+ defaults_layer_kind=defaults_layer_kind,+ packaged_document_path=packaged_document_path,+ kit_template_path=kit_template_path,+ )+++def _write_ledger(*, migration_dir: Path, current_schema_version: int, entries: str = "") -> None:+ directory = ledgers_dir(migration_dir=migration_dir)+ directory.mkdir(parents=True, exist_ok=True)+ body = f"""+[surface]+id = "{SURFACE_ID}"+title = "A synthetic surface"+base_file = "synthetic.toml"+tier_glob = "synthetic_*.toml"+current_schema_version = {current_schema_version}+min_supported_schema_version = 0+{entries}+"""+ (directory / f"{SURFACE_ID}.toml").write_text(body, encoding="utf-8")+++def _snapshot(*, migration_dir: Path, config_model: type[BaseModel], schema_version: int) -> None:+ surface = _surface(config_model=config_model)+ write_fingerprint_golden(+ migration_dir=migration_dir,+ fingerprint=compute_fingerprint(+ surface_id=SURFACE_ID,+ schema_version=schema_version,+ config_model=config_model,+ defaults_document=surface.read_defaults_document(),+ ),+ )+++def _entry(*, ops: str, to_schema_version: int = 2, safety: str = "safe", pre_history: str = "") -> str:+ return f"""+[[migration]]+id = "{SURFACE_ID}@{to_schema_version}"+to_schema_version = {to_schema_version}+introduced_in = "0.46.0"+breaking = true+safety = "{safety}"+title = "Reshape the synthetic surface"+description = "The surface changed shape."+{pre_history}+{ops}+"""+++_RENAME_LABEL_TO_TITLE = """+[[migration.ops]]+kind = "rename_table_key"+table_path = []+key = "label"+new_key = "title"+"""+++_DELETE_LEGACY_MODE = """+[[migration.ops]]+kind = "delete_key"+table_path = []+key = "legacy_mode"+"""+++def _pre_history_entry(*, ops: str, declared: str = 'declared_removed_paths = ["legacy_mode"]') -> str:+ """An entry for material that predates the chain: no fingerprint records `legacy_mode` at all."""+ return _entry(ops=f"{declared}\n{ops}", pre_history="pre_history = true")+++def _kinds(issues: list[LedgerIssue]) -> list[LedgerIssueKind]:+ return [issue.kind for issue in issues]+++class TestTheLedgerCheck:+ def test_an_empty_ledger_at_the_initial_version_is_green(self, tmp_path: Path) -> None:+ """The overwhelmingly common state: a surface that has never changed shape."""+ _write_ledger(migration_dir=tmp_path, current_schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)+ assert check_ledger(surface=_surface(config_model=_SchemaOne), migration_dir=tmp_path) == []++ def test_a_correct_rename_entry_is_green(self, tmp_path: Path) -> None:+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=_entry(ops=_RENAME_LABEL_TO_TITLE))+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaTwoRenamed, schema_version=2)+ assert check_ledger(surface=_surface(config_model=_SchemaTwoRenamed), migration_dir=tmp_path) == []++ def test_a_missing_link_is_named_rather_than_papered_over(self, tmp_path: Path) -> None:+ """An entry checked against a shape nobody snapshotted would be blamed for the gap."""+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=_entry(ops=_RENAME_LABEL_TO_TITLE))+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)+ issues = check_ledger(surface=_surface(config_model=_SchemaTwoRenamed), migration_dir=tmp_path)+ assert _kinds(issues) == [LedgerIssueKind.CHAIN_INCOMPLETE]+ assert "make umig" in issues[0].message++ def test_an_operation_on_material_no_version_removed_is_refused(self, tmp_path: Path) -> None:+ """The premise of always-replay: an operation may only act on what a schema version retired.++ Here the shape does not change at all, so `label` is still live — and an operation that+ renames it would fire on every valid current file, forever.+ """+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=_entry(ops=_RENAME_LABEL_TO_TITLE))+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=2)+ issues = check_ledger(surface=_surface(config_model=_SchemaOne), migration_dir=tmp_path)+ assert LedgerIssueKind.OP_ACTS_ON_LIVE_MATERIAL in _kinds(issues)++ def test_an_operation_naming_a_key_beneath_an_open_node_is_refused(self, tmp_path: Path) -> None:+ """The keys under an open mapping are the user's, so no schema change can remove one."""+ ops = """+[[migration.ops]]+kind = "delete_key"+table_path = ["deck"]+key = "some_model_the_user_chose"+"""+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=_entry(ops=ops))+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=2)+ issues = check_ledger(surface=_surface(config_model=_SchemaOne), migration_dir=tmp_path)+ assert _kinds(issues) == [LedgerIssueKind.CONCRETE_KEY_UNDER_OPEN_NODE]+ assert "some_model_the_user_chose" in issues[0].message++ def test_a_wildcard_anywhere_but_an_open_node_is_refused(self, tmp_path: Path) -> None:+ ops = """+[[migration.ops]]+kind = "rename_table_key"+table_path = ["settings", "*"]+key = "retries"+new_key = "attempts"+"""+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=_entry(ops=ops))+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=2)+ issues = check_ledger(surface=_surface(config_model=_SchemaOne), migration_dir=tmp_path)+ assert _kinds(issues) == [LedgerIssueKind.WILDCARD_NOT_AT_OPEN_NODE]+ assert "'settings'" in issues[0].message++ def test_a_wildcard_at_an_open_node_is_exactly_what_it_is_for(self, tmp_path: Path) -> None:+ """A field renamed inside every entry of an open mapping is an ordinary safe operation."""+ ops = """+[[migration.ops]]+kind = "rename_table_key"+table_path = ["deck", "*"]+key = "old_name"+new_key = "new_name"+"""+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=_entry(ops=ops))+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaTwoDeckFieldRenamed, schema_version=2)+ assert check_ledger(surface=_surface(config_model=_SchemaTwoDeckFieldRenamed), migration_dir=tmp_path) == []++ def test_a_safe_remap_of_a_value_that_is_still_legal_is_refused(self, tmp_path: Path) -> None:+ """The remap legality rule, and the reason replay neutrality holds.++ `premium` is still a legal tier at schema 2, so a user who chose it deliberately would+ have it rewritten on every run — and replay over a current-valid file would not be a no-op.+ """+ ops = """+[[migration.ops]]+kind = "remap_value"+table_path = []+key = "tier"+mapping = { basic = "standard", premium = "standard" }+"""+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=_entry(ops=ops))+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaTwoEnumMemberGone, schema_version=2)+ issues = check_ledger(surface=_surface(config_model=_SchemaTwoEnumMemberGone), migration_dir=tmp_path)+ assert LedgerIssueKind.ILLEGAL_REMAP in _kinds(issues)+ assert "premium" in " ".join(issue.message for issue in issues)++ def test_a_remap_to_a_spelling_the_new_schema_rejects_is_refused(self, tmp_path: Path) -> None:+ """The value-side twin of the misspelled rename destination.++ Rewriting to a spelling the new schema does not accept migrates every file to something+ the model rejects, with the tool reporting success.+ """+ ops = """+[[migration.ops]]+kind = "remap_value"+table_path = []+key = "tier"+mapping = { basic = "standrad" }+"""+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=_entry(ops=ops))+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaTwoEnumMemberGone, schema_version=2)+ issues = check_ledger(surface=_surface(config_model=_SchemaTwoEnumMemberGone), migration_dir=tmp_path)+ assert LedgerIssueKind.ILLEGAL_REMAP in _kinds(issues)+ assert "standrad" in " ".join(issue.message for issue in issues)++ def test_a_safe_remap_of_a_free_string_is_refused(self, tmp_path: Path) -> None:+ """Staleness can never be proven from the schema when any string is legal."""+ ops = """+[[migration.ops]]+kind = "remap_value"+table_path = []+key = "label"+mapping = { hello = "hi" }+"""+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=_entry(ops=ops))+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=2)+ issues = check_ledger(surface=_surface(config_model=_SchemaOne), migration_dir=tmp_path)+ assert LedgerIssueKind.ILLEGAL_REMAP in _kinds(issues)++ def test_an_unsafe_entry_may_remap_a_value_that_is_still_legal(self, tmp_path: Path) -> None:+ """`unsafe` is exactly the answer to a change the applier cannot make on its own."""+ ops = """+[[migration.ops]]+kind = "remap_value"+table_path = []+key = "tier"+mapping = { premium = "standard" }+"""+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=_entry(ops=ops, safety="unsafe"))+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaTwoEnumMemberGone, schema_version=2)+ issues = check_ledger(surface=_surface(config_model=_SchemaTwoEnumMemberGone), migration_dir=tmp_path)+ assert LedgerIssueKind.ILLEGAL_REMAP not in _kinds(issues)++ def test_a_retired_name_that_comes_back_is_refused(self, tmp_path: Path) -> None:+ """And the reuse is not academic: the operation that retired the name starts firing again."""+ delete_label = """+[[migration.ops]]+kind = "delete_key"+table_path = []+key = "label"+"""+ rename_tier = """+[[migration.ops]]+kind = "rename_table_key"+table_path = []+key = "tier"+new_key = "level"+"""+ entries = _entry(ops=delete_label, to_schema_version=2) + _entry(ops=rename_tier, to_schema_version=3)+ _write_ledger(migration_dir=tmp_path, current_schema_version=3, entries=entries)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaTwoLabelGone, schema_version=2)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaThreeLabelBack, schema_version=3)+ issues = check_ledger(surface=_surface(config_model=_SchemaThreeLabelBack), migration_dir=tmp_path)+ assert LedgerIssueKind.RESERVED_PATH_REUSED in _kinds(issues)+ assert LedgerIssueKind.CONVERGENCE_BROKEN in _kinds(issues)++ def test_a_retired_spelling_that_comes_back_is_refused(self, tmp_path: Path) -> None:+ """The value-side twin of the retired name: a remapped-away spelling accepted again would be+ rewritten on every run of every file that still carries it.+ """+ remap_basic = """+[[migration.ops]]+kind = "remap_value"+table_path = []+key = "tier"+mapping = { basic = "standard" }+"""+ entries = _entry(ops=remap_basic, to_schema_version=2) + _entry(ops=_RENAME_LABEL_TO_TITLE, to_schema_version=3)+ _write_ledger(migration_dir=tmp_path, current_schema_version=3, entries=entries)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaTwoEnumMemberGone, schema_version=2)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaThreeBasicBack, schema_version=3)+ issues = check_ledger(surface=_surface(config_model=_SchemaThreeBasicBack), migration_dir=tmp_path)+ assert LedgerIssueKind.RESERVED_VALUE_REUSED in _kinds(issues)+ assert "'basic'" in " ".join(issue.message for issue in issues)++ def test_a_spelling_that_stays_retired_is_not_an_issue(self, tmp_path: Path) -> None:+ remap_basic = """+[[migration.ops]]+kind = "remap_value"+table_path = []+key = "tier"+mapping = { basic = "standard" }+"""+ entries = _entry(ops=remap_basic, to_schema_version=2) + _entry(ops=_RENAME_LABEL_TO_TITLE, to_schema_version=3)+ _write_ledger(migration_dir=tmp_path, current_schema_version=3, entries=entries)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaTwoEnumMemberGone, schema_version=2)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaThreeStaysRetired, schema_version=3)+ issues = check_ledger(surface=_surface(config_model=_SchemaThreeStaysRetired), migration_dir=tmp_path)+ assert LedgerIssueKind.RESERVED_VALUE_REUSED not in _kinds(issues)++ def test_a_reference_document_the_ledger_still_changes_is_refused(self, tmp_path: Path) -> None:+ """A packaged document left behind by its own model would be migrated on every user's run."""+ packaged = tmp_path / "packaged.toml"+ packaged.write_text('label = "hello"\ntier = "basic"\n', encoding="utf-8")+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=_entry(ops=_RENAME_LABEL_TO_TITLE))+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaTwoRenamed, schema_version=2)+ surface = _surface(config_model=_SchemaTwoRenamed, packaged_document_path=packaged)+ issues = check_ledger(surface=surface, migration_dir=tmp_path)+ assert _kinds(issues) == [LedgerIssueKind.CONVERGENCE_BROKEN]+ assert "reference document" in issues[0].message++ def test_the_sparse_kit_template_is_a_witness_of_its_own(self, tmp_path: Path) -> None:+ """The two witnesses are different shapes on purpose: one has every key set, one almost none."""+ packaged = tmp_path / "packaged.toml"+ packaged.write_text('title = "hello"\ntier = "basic"\n', encoding="utf-8")+ template = tmp_path / "template.toml"+ template.write_text('label = "hello"\n', encoding="utf-8")+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=_entry(ops=_RENAME_LABEL_TO_TITLE))+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaTwoRenamed, schema_version=2)+ surface = _surface(config_model=_SchemaTwoRenamed, packaged_document_path=packaged, kit_template_path=template)+ issues = check_ledger(surface=surface, migration_dir=tmp_path)+ assert _kinds(issues) == [LedgerIssueKind.CONVERGENCE_BROKEN]+ assert "kit template" in issues[0].message++ def test_a_pre_history_entry_addressing_only_what_it_declares_is_green(self, tmp_path: Path) -> None:+ """The shape the flag is for: material that predates the chain, so no fingerprint shows it going away."""+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=_pre_history_entry(ops=_DELETE_LEGACY_MODE))+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=2)+ assert check_ledger(surface=_surface(config_model=_SchemaOne), migration_dir=tmp_path) == []++ def test_a_pre_history_entry_declaring_a_path_the_chain_records_is_refused(self, tmp_path: Path) -> None:+ """Otherwise the flag is a way to opt out of accounting for a change that has a diff."""+ _write_ledger(+ migration_dir=tmp_path,+ current_schema_version=2,+ entries=_pre_history_entry(ops=_DELETE_LEGACY_MODE, declared='declared_removed_paths = ["legacy_mode", "label"]'),+ )+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=2)+ issues = check_ledger(surface=_surface(config_model=_SchemaOne), migration_dir=tmp_path)+ assert _kinds(issues) == [LedgerIssueKind.PRE_HISTORY_PATH_IS_RECORDED]+ assert "'label'" in issues[0].message++ def test_a_pre_history_operation_outside_the_declaration_is_refused(self, tmp_path: Path) -> None:+ """The declaration is the entry's only record of what it may address, so it bounds the operations too."""+ ops = (+ _DELETE_LEGACY_MODE+ + """+[[migration.ops]]+kind = "delete_key"+table_path = []+key = "label"+"""+ )+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=_pre_history_entry(ops=ops))+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=2)+ issues = check_ledger(surface=_surface(config_model=_SchemaOne), migration_dir=tmp_path)+ assert LedgerIssueKind.OP_ACTS_ON_LIVE_MATERIAL in _kinds(issues)++ @pytest.mark.parametrize(+ ("current_schema_version", "entries"),+ [+ pytest.param(2, "", id="a version with no entry to produce it"),+ pytest.param(3, _entry(ops=_RENAME_LABEL_TO_TITLE, to_schema_version=3), id="a gap in the entry numbering"),+ ],+ )+ def test_a_ledger_that_does_not_hold_together_is_refused_when_it_is_read(+ self,+ tmp_path: Path,+ current_schema_version: int,+ entries: str,+ ) -> None:+ """Contiguity and matching ids are enforced at parse time — the earliest place they can be."""+ _write_ledger(migration_dir=tmp_path, current_schema_version=current_schema_version, entries=entries)+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)+ with pytest.raises(MigrationLedgerError):+ check_ledger(surface=_surface(config_model=_SchemaOne), migration_dir=tmp_path)diff --git a/tests/unit/pipelex/migration/test_ledger_models.py b/tests/unit/pipelex/migration/test_ledger_models.pyindex e108d729f..94d66aa07 100644--- a/tests/unit/pipelex/migration/test_ledger_models.py+++ b/tests/unit/pipelex/migration/test_ledger_models.py@@ -115,6 +115,13 @@ class TestEntryInvariants:with pytest.raises(ValidationError, match="pre-history entries only"):MigrationLedger.model_validate({"surface": surface, "migration": [entry]})+ def test_a_pre_history_entry_declaring_nothing_is_refused(self) -> None:+ """The declaration is what replaces the diff; without one the flag exempts an entry from everything."""+ surface = {**_SURFACE_BLOCK, "current_schema_version": 2}+ entry = _entry(pre_history=True)+ with pytest.raises(ValidationError, match="declares the paths it removes"):+ MigrationLedger.model_validate({"surface": surface, "migration": [entry]})+def test_nothing_migrates_to_schema_version_one(self) -> None:"""Version 1 is the shape a surface starts at; there is no earlier shape to come from."""surface = {**_SURFACE_BLOCK, "current_schema_version": 1}diff --git a/tests/unit/pipelex/migration/test_narrowing.py b/tests/unit/pipelex/migration/test_narrowing.pynew file mode 100644index 000000000..288d7ef50--- /dev/null+++ b/tests/unit/pipelex/migration/test_narrowing.py@@ -0,0 +1,175 @@+"""Unit tests for the narrowing relation — which schema changes shrink the values a file may carry.++The relation is tested on hand-built `PathFingerprint` records rather than on models, because what+is under test is the *comparison*, not the projection that feeds it. Two directions matter equally+and are asserted equally: a narrowing that goes unreported ships a change that breaks a user's boot+with a green gate behind it, and a widening reported as a narrowing is a gate crying wolf, which is+how a gate gets waved through and then ignored.+"""++import pytest++from pipelex.migration.fingerprint import ConstraintKind, PathFingerprint+from pipelex.migration.narrowing import describe_narrowing, lost_enumerated_spellings+++def _record(+ *,+ value_type: str = "int",+ enum_members: list[str] | None = None,+ constraints: dict[ConstraintKind, int | float] | None = None,+) -> PathFingerprint:+ return PathFingerprint(value_type=value_type, required=True, enum_members=enum_members, constraints=constraints)+++class TestTheNarrowingRelation:+ @pytest.mark.parametrize(+ ("before_type", "after_type"),+ [+ ("int", "str"),+ ("str", "enum"),+ ("list[str]", "list[int]"),+ ("int | str", "int"),+ ("str", "table"),+ ],+ )+ def test_a_type_that_stops_accepting_what_it_accepted_is_a_narrowing(self, before_type: str, after_type: str) -> None:+ reasons = describe_narrowing(before=_record(value_type=before_type), after=_record(value_type=after_type))+ assert reasons == [f"its type went from '{before_type}' to '{after_type}'"]++ @pytest.mark.parametrize(+ ("before_type", "after_type"),+ [+ ("int", "int | str"),+ ("enum", "str"),+ ("literal", "str"),+ ("enum | literal", "str"),+ ("enum", "enum | int"),+ ],+ )+ def test_a_type_that_still_accepts_everything_it_did_is_not(self, before_type: str, after_type: str) -> None:+ """Both widening shapes: a union that gains members, and an enumerated type relaxed into `str`."""+ assert describe_narrowing(before=_record(value_type=before_type), after=_record(value_type=after_type)) == []++ @pytest.mark.parametrize(+ ("before_type", "after_type", "expected"),+ [+ ("str", "enum", []),+ ("enum | literal", "literal", []),+ ("int | literal", "literal", ["its type went from 'int | literal' to 'literal'"]),+ ("int | str", "enum", ["its type went from 'int | str' to 'enum'"]),+ ],+ )+ def test_under_a_remap_only_the_string_typed_members_are_exempt(self, before_type: str, after_type: str, expected: list[str]) -> None:+ """A remap rewrites string values, so it answers for a lost `str`, `enum` or `literal` member and+ for nothing else: a number the old type accepted and the new one does not is still a narrowing.+ """+ assert describe_narrowing(before=_record(value_type=before_type), after=_record(value_type=after_type), remapped=True) == expected++ def test_a_union_inside_a_container_is_one_member_not_two(self) -> None:+ """Splitting a rendered type on `|` without respecting brackets would read this as a widening."""+ reasons = describe_narrowing(before=_record(value_type="list[int | str]"), after=_record(value_type="int"))+ assert reasons == ["its type went from 'list[int | str]' to 'int'"]++ @pytest.mark.parametrize(+ ("before_constraints", "after_constraints", "expected_fragment"),+ [+ ({}, {ConstraintKind.GE: 1}, "its lower bound tightened from unbounded to ge=1"),+ ({ConstraintKind.GE: 1}, {ConstraintKind.GE: 5}, "its lower bound tightened from ge=1 to ge=5"),+ ({ConstraintKind.GE: 0}, {ConstraintKind.GT: 0}, "its lower bound tightened from ge=0 to gt=0"),+ ({ConstraintKind.LE: 10}, {ConstraintKind.LE: 6}, "its upper bound tightened from le=10 to le=6"),+ ({}, {ConstraintKind.MAX_LENGTH: 40}, "its maximum length tightened from unbounded to max_length=40"),+ ({ConstraintKind.MIN_LENGTH: 1}, {ConstraintKind.MIN_LENGTH: 3}, "its minimum length tightened from min_length=1 to min_length=3"),+ ({ConstraintKind.MULTIPLE_OF: 2}, {ConstraintKind.MULTIPLE_OF: 4}, "its step tightened from multiple_of=2 to multiple_of=4"),+ ({ConstraintKind.MULTIPLE_OF: 2}, {ConstraintKind.MULTIPLE_OF: 3}, "its step tightened from multiple_of=2 to multiple_of=3"),+ ],+ )+ def test_a_tightened_bound_is_a_narrowing(+ self,+ before_constraints: dict[ConstraintKind, int | float],+ after_constraints: dict[ConstraintKind, int | float],+ expected_fragment: str,+ ) -> None:+ reasons = describe_narrowing(before=_record(constraints=before_constraints), after=_record(constraints=after_constraints))+ assert reasons == [expected_fragment]++ @pytest.mark.parametrize(+ ("before_constraints", "after_constraints"),+ [+ ({ConstraintKind.GE: 5}, {ConstraintKind.GE: 1}),+ ({ConstraintKind.GE: 1}, {}),+ ({ConstraintKind.GT: 0}, {ConstraintKind.GE: 0}),+ ({ConstraintKind.LE: 6}, {ConstraintKind.LE: 10}),+ ({ConstraintKind.MAX_LENGTH: 40}, {}),+ ({ConstraintKind.MULTIPLE_OF: 4}, {ConstraintKind.MULTIPLE_OF: 2}),+ ({ConstraintKind.MULTIPLE_OF: 2}, {ConstraintKind.MULTIPLE_OF: 2}),+ ],+ )+ def test_a_relaxed_or_dropped_bound_is_not(+ self,+ before_constraints: dict[ConstraintKind, int | float],+ after_constraints: dict[ConstraintKind, int | float],+ ) -> None:+ """A bound that admits more values than it did asks for nothing: every old file survives it."""+ assert describe_narrowing(before=_record(constraints=before_constraints), after=_record(constraints=after_constraints)) == []++ def test_both_halves_are_reported_together(self) -> None:+ """One path can move both ways at once, and the author needs to read both reasons."""+ reasons = describe_narrowing(+ before=_record(value_type="int | str", constraints={ConstraintKind.LE: 10}),+ after=_record(value_type="int", constraints={ConstraintKind.LE: 6}),+ )+ assert reasons == ["its type went from 'int | str' to 'int'", "its upper bound tightened from le=10 to le=6"]++ def test_an_unchanged_record_narrows_nothing(self) -> None:+ record = _record(value_type="int", constraints={ConstraintKind.GE: 1, ConstraintKind.LE: 10})+ assert describe_narrowing(before=record, after=record) == []++ def test_a_spelling_the_new_member_set_lacks_is_lost(self) -> None:+ lost = lost_enumerated_spellings(+ before=_record(value_type="enum", enum_members=["basic", "premium"]),+ after=_record(value_type="enum", enum_members=["premium"]),+ )+ assert lost == ["basic"]++ def test_an_enum_relaxed_into_a_free_string_loses_no_spelling(self) -> None:+ """The set difference says every member vanished; the truth is that `str` accepts them all.++ Left as a raw difference, the most benign loosening a config model can undergo would demand+ a version bump and a remap for every spelling it had.+ """+ lost = lost_enumerated_spellings(+ before=_record(value_type="enum", enum_members=["basic", "premium"]),+ after=_record(value_type="str"),+ )+ assert lost == []++ def test_an_enum_that_became_something_else_entirely_still_loses_its_spellings(self) -> None:+ """The exemption is for widening only — `int` accepts none of the spellings `str` did."""+ lost = lost_enumerated_spellings(+ before=_record(value_type="enum", enum_members=["basic", "premium"]),+ after=_record(value_type="int"),+ )+ assert lost == ["basic", "premium"]++ def test_a_literal_whose_spellings_became_non_strings_still_loses_them(self) -> None:+ """`Literal['a']` to `Literal[1]` renders the same type on both sides and records no members+ after — the one shape where the type comparison alone would wave a lost spelling through.+ """+ lost = lost_enumerated_spellings(+ before=_record(value_type="literal", enum_members=["a"]),+ after=_record(value_type="literal"),+ )+ assert lost == ["a"]++ @pytest.mark.parametrize(+ ("before_step", "after_step"),+ [(0.3, 0.1), (1.0, 0.5), (0.6, 0.2)],+ )+ def test_a_relaxed_float_step_is_read_exactly_rather_than_in_binary(self, before_step: float, after_step: float) -> None:+ """`0.3 % 0.1` is not zero in binary floating point; a widening must not read as a tightening for it."""+ reasons = describe_narrowing(+ before=_record(constraints={ConstraintKind.MULTIPLE_OF: before_step}),+ after=_record(constraints={ConstraintKind.MULTIPLE_OF: after_step}),+ )+ assert reasons == []diff --git a/tests/unit/pipelex/migration/test_real_surfaces.py b/tests/unit/pipelex/migration/test_real_surfaces.pyindex 1abab1938..15af09e3b 100644--- a/tests/unit/pipelex/migration/test_real_surfaces.py+++ b/tests/unit/pipelex/migration/test_real_surfaces.py@@ -11,7 +11,9 @@ that is not about configuration, which teaches the reflex the whole design existfrom pipelex.migration.coverage import check_registryfrom pipelex.migration.ledger import INITIAL_SCHEMA_VERSION, load_ledger+from pipelex.migration.ledger_check import check_ledgersfrom pipelex.migration.surfaces import build_config_surface_registry, packaged_migration_dir+from pipelex.migration.transform_check import check_transformsclass TestTheRealRegistry:@@ -46,3 +48,23 @@ class TestTheCheckedInGoldens:"""issues = check_registry(registry=build_config_surface_registry(), migration_dir=packaged_migration_dir())assert issues == [], "\n".join(f"{issue.surface_id}: {issue.message}" for issue in issues)++ def test_the_real_ledgers_pass_the_ledger_check(self) -> None:+ """The checked-in ledgers say only legal things, and replay over our own documents does nothing.++ The convergence half of this is the one assertion here that is not about the goldens at+ all: it replays every ledger over the packaged configuration and the kit template, which+ is exactly what a user's machine will do to their own file.+ """+ ledger_issues = check_ledgers(registry=build_config_surface_registry(), migration_dir=packaged_migration_dir())+ assert ledger_issues == [], "\n".join(f"{issue.surface_id}: {issue.message}" for issue in ledger_issues)++ def test_the_real_transform_chains_hold(self) -> None:+ """Every entry, applied to the document of the version before it, lands on the next one.++ No longer vacuous: `telemetry-config@2` is a pre-history entry, so this walks its+ hand-authored `[email protected]` through its operations and holds the result against today's+ shape and today's model.+ """+ transform_issues = check_transforms(registry=build_config_surface_registry(), migration_dir=packaged_migration_dir())+ assert transform_issues == [], "\n".join(f"{issue.surface_id}: {issue.message}" for issue in transform_issues)diff --git a/tests/unit/pipelex/migration/test_replay_properties.py b/tests/unit/pipelex/migration/test_replay_properties.pynew file mode 100644index 000000000..dcb59e84a--- /dev/null+++ b/tests/unit/pipelex/migration/test_replay_properties.py@@ -0,0 +1,345 @@+"""Property tests for replay neutrality, and for the two properties that travel with it.++Replay neutrality is the guarantee the whole contract exists to keep true, and it is claimed over+*every* document the current models accept. Example tests cannot express that, and the convergence+witness — a replay over the packaged defaults and the kit template — cannot either: a witness+carries exactly one value per key, so an operation that misbehaves on the *other* legal spelling+of an enumerated field passes it. `test_the_witnesses_cannot_see_an_illegal_remap` is that gap,+demonstrated: the same ledger is green on the witness and red on a generated document.++**Two venues, and the difference between them is the point.** The real surfaces are all at schema+version 1 with no entries at all, so a replay over them has nothing to replay and neutrality holds+for a reason that has nothing to do with the theorem. What does real work over the real surfaces+today is the **vacuity meta-test**: it proves the sampler emits documents the current models+actually accept, so that the property is not passing on garbage on the day the first entry lands —+the reshape's, at the join. Everything that needs a ledger with entries in it runs against a+synthetic surface whose model and ledger are written here and never move.+"""++from enum import StrEnum+from typing import Any, cast++import pytest+import tomlkit+from hypothesis import find, given, settings+from hypothesis import strategies as st+from pydantic import BaseModel, ConfigDict, Field++from pipelex.migration.engine import replay_ledger_over_text+from pipelex.migration.fingerprint import ENUM_TYPE+from pipelex.migration.ledger import MigrationEntry, MigrationLedger, MigrationSafety, SurfaceBlock, load_ledger+from pipelex.migration.surfaces import DefaultsLayerKind, Surface, build_config_surface_registry, packaged_migration_dir+from pipelex.suggested_fix import DeleteKeyOp, DeleteTableOp, MigrationOp, MoveKeyOp, RemapValueOp, RenameTableKeyOp+from tests.unit.pipelex.migration.document_strategies import (+ DocumentMutation,+ GeneratedDocument,+ fingerprint_at,+ merge_text_beneath_defaults,+ surface_fingerprint,+ within_schema_documents,+)++MAX_EXAMPLES = 40+PROPERTY_SETTINGS = settings(max_examples=MAX_EXAMPLES, deadline=None)+"""Bounded on purpose. `deadline=None` because these run under `xdist`, where a per-example+wall-clock deadline measures machine contention rather than anything about the engine."""++REAL_SURFACES = build_config_surface_registry().surfaces+REAL_SURFACE_IDS = [surface.surface_id for surface in REAL_SURFACES]+++class _Tone(StrEnum):+ QUIET = "quiet"+ LOUD = "loud"+ MUTED = "muted"+++class _Deck(BaseModel):+ """The value schema of a genuinely open node — the keys above it belong to the user."""++ model_config = ConfigDict(extra="forbid")++ tone: _Tone = Field(default=_Tone.QUIET, strict=False)+ enabled: bool = True+++class _Section(BaseModel):+ model_config = ConfigDict(extra="forbid")++ tone: _Tone = Field(default=_Tone.QUIET, strict=False)+ enabled: bool = True+++class _SyntheticConfig(BaseModel):+ """A stand-in configuration model, so the properties never move when the real models do.++ It is shaped to carry every mutation the generator can make — an enumerated field whose+ default is not the spelling a migration would target, booleans, and an open node with entries+ in it — because a property over a document with nothing to mutate proves nothing.+ """++ model_config = ConfigDict(extra="forbid")++ title: str = "example"+ verbose: bool = False+ tone: _Tone = Field(default=_Tone.MUTED, strict=False)+ section: _Section = Field(default_factory=_Section)+ decks: dict[str, _Deck] = Field(default_factory=lambda: {"primary": _Deck(), "backup": _Deck(tone=_Tone.LOUD, enabled=False)})+++SYNTHETIC_SURFACE = Surface(+ surface_id="synthetic-config",+ title="A synthetic surface whose ledger actually has entries",+ base_file="synthetic.toml",+ config_model=_SyntheticConfig,+ defaults_layer_kind=DefaultsLayerKind.MODEL_DEFAULTS,+)+++def _entry(*, to_schema_version: int, ops: list[MigrationOp], safety: MigrationSafety = MigrationSafety.SAFE) -> MigrationEntry:+ return MigrationEntry(+ id=f"{SYNTHETIC_SURFACE.surface_id}@{to_schema_version}",+ to_schema_version=to_schema_version,+ introduced_in="0.46.0",+ breaking=True,+ safety=safety,+ title=f"The change that produced schema version {to_schema_version}",+ description="A shape change on the synthetic surface.",+ ops=ops,+ )+++def _ledger(*, entries: list[MigrationEntry]) -> MigrationLedger:+ return MigrationLedger(+ surface=SurfaceBlock(+ id=SYNTHETIC_SURFACE.surface_id,+ title=SYNTHETIC_SURFACE.title,+ base_file=SYNTHETIC_SURFACE.base_file,+ current_schema_version=1 + len(entries),+ min_supported_schema_version=0,+ ),+ migration=entries,+ )+++SYNTHETIC_ENTRIES = [+ _entry(+ to_schema_version=2,+ ops=[+ RenameTableKeyOp(table_path=[], key="heading", new_key="title"),+ MoveKeyOp(table_path=[], key="chatty", new_table_path=["section"], new_key="enabled"),+ RemapValueOp(table_path=[], key="tone", mapping={"shouty": _Tone.LOUD}),+ DeleteKeyOp(table_path=["decks", "*"], key="legacy_flag"),+ ],+ ),+ _entry(to_schema_version=3, ops=[DeleteTableOp(table_path=["legacy"])]),+]+"""Every operation's source is material the current shape no longer has — `heading`, `chatty`,+the spelling `shouty`, `legacy_flag`, the `legacy` table — which is what op legality demands and+what makes the replay a no-op on a current-valid file."""++SYNTHETIC_LEDGER = _ledger(entries=SYNTHETIC_ENTRIES)++_STRATEGIES: dict[str, st.SearchStrategy[GeneratedDocument]] = {}+++def _documents_for(*, surface: Surface) -> st.SearchStrategy[GeneratedDocument]:+ """The sampler for a surface, built once — projecting a model tree is not free."""+ if surface.surface_id not in _STRATEGIES:+ _STRATEGIES[surface.surface_id] = within_schema_documents(surface=surface)+ return _STRATEGIES[surface.surface_id]+++def _documents_at_an_older_shape( # kw-only: ignore — Hypothesis passes `draw` positionally to a composite's body.+ draw: st.DrawFn,+) -> str:+ """A current-valid document with some of the synthetic surface's retired material put back.++ Idempotence and prefix coherence are claims about files that still have something to migrate,+ so they need documents the ledger will actually act on — the neutrality domain is exactly the+ set where it does not.++ Putting a retired key back means taking its successor out: a document carrying both `heading`+ and `title` is not an older shape but a hand-fixed one, on which the rename conflicts and+ writes nothing — and a property quantified over stable conflicts would say nothing about+ a rename.+ """+ generated = draw(_documents_for(surface=SYNTHETIC_SURFACE))+ document = tomlkit.loads(generated.text).unwrap()+ if draw(st.booleans()):+ document.pop("title", None)+ document["heading"] = "a heading from before the rename"+ if draw(st.booleans()):+ cast("dict[str, Any]", document.get("section", {})).pop("enabled", None)+ document["chatty"] = True+ if draw(st.booleans()):+ document["tone"] = "shouty"+ if draw(st.booleans()):+ document["legacy"] = {"debug": True, "level": 3}+ if draw(st.booleans()):+ for deck in cast("dict[str, dict[str, Any]]", document.get("decks", {})).values():+ deck["legacy_flag"] = True+ text: str = tomlkit.dumps(document) # pyright: ignore[reportUnknownMemberType]+ return text+++class TestTheReplayProperties:+ @pytest.mark.parametrize("surface", REAL_SURFACES, ids=REAL_SURFACE_IDS)+ @PROPERTY_SETTINGS+ @given(data=st.data())+ def test_every_generated_document_is_valid_at_the_current_schema(self, surface: Surface, data: st.DataObject) -> None:+ """The vacuity meta-test: without it, a green property says nothing.++ The generator already checks each *proposed mutation* against the model, so what is left+ for this to catch is everything that happens after — a mis-addressed drop, a path+ translated wrongly beneath an open node, a document whose round-trip through TOML is not+ what was validated — and any future model change that breaks the assembly rather than an+ individual value. It is asserted here, on the emitted text, because that is the artifact+ the property is quantified over.++ The document is validated the way a user's file is really read, merged beneath the+ surface's defaults layer: an absent optional key is the ordinary shape of a configuration+ file, not an error.+ """+ generated = data.draw(_documents_for(surface=surface))++ surface.config_model.model_validate(merge_text_beneath_defaults(surface=surface, text=generated.text))++ @pytest.mark.parametrize("surface", REAL_SURFACES, ids=REAL_SURFACE_IDS)+ def test_the_generator_does_not_merely_re_emit_the_reference_document(self, surface: Surface) -> None:+ """The other direction of vacuity. A sampler of one document satisfies every property."""+ found = find(_documents_for(surface=surface), lambda generated: bool(generated.mutations))++ assert found.mutations++ def test_a_document_path_beneath_an_open_node_resolves_to_its_value_schema(self) -> None:+ """The one translation the generator does that nothing else would notice going wrong.++ A document names the user's own key where the fingerprint names a `*`, so without this+ the values inside an open node would silently stop being mutated and every property would+ stay green over a smaller domain.+ """+ fingerprint = surface_fingerprint(surface=SYNTHETIC_SURFACE)++ beneath_an_open_node = fingerprint_at(fingerprint=fingerprint, path="decks.backup.tone")+ assert beneath_an_open_node is not None+ assert beneath_an_open_node.value_type == ENUM_TYPE+ assert beneath_an_open_node.enum_members == sorted(_Tone)+ assert fingerprint_at(fingerprint=fingerprint, path="decks.backup.invented") is None++ @pytest.mark.parametrize("mutation", list(DocumentMutation), ids=list(DocumentMutation))+ def test_the_generator_reaches_every_mutation_kind(self, mutation: DocumentMutation) -> None:+ """Asked of the synthetic surface, whose model is written here to carry all three.++ Asking it of the real surfaces would couple this test to whichever fields they happen to+ have today, and the service surface has no enumerated field at all.+ """+ found = find(_documents_for(surface=SYNTHETIC_SURFACE), lambda generated: mutation in generated.mutations)++ assert mutation in found.mutations++ @pytest.mark.parametrize("surface", REAL_SURFACES, ids=REAL_SURFACE_IDS)+ @PROPERTY_SETTINGS+ @given(data=st.data())+ def test_a_replay_over_a_current_valid_document_returns_the_very_same_text(self, surface: Surface, data: st.DataObject) -> None:+ """Neutrality over the real surfaces — structurally vacuous until the first entry lands.++ Every surface sits at schema version 1 with an empty ledger today, so this passes because+ there is nothing to replay. It is written now, against the real ledgers rather than a+ copy, so that the reshape entry is measured by it the moment it is added.+ """+ generated = data.draw(_documents_for(surface=surface))+ ledger = load_ledger(migration_dir=packaged_migration_dir(), surface_id=surface.surface_id)++ replay = replay_ledger_over_text(ledger=ledger, text=generated.text)++ # The *very* string, not an equal one: neutrality is a property of the engine rather than+ # of tomlkit's round-trip, and identity is what tells the two apart.+ assert replay.text is generated.text+ assert not replay.steps+ assert not replay.blocked++ @PROPERTY_SETTINGS+ @given(data=st.data())+ def test_a_replay_of_a_ledger_with_entries_is_still_neutral_on_a_current_valid_document(self, data: st.DataObject) -> None:+ """The same claim where it is not vacuous: a ledger with five operations across two entries."""+ generated = data.draw(_documents_for(surface=SYNTHETIC_SURFACE))++ replay = replay_ledger_over_text(ledger=SYNTHETIC_LEDGER, text=generated.text)++ assert replay.text is generated.text+ assert not replay.steps+ assert not replay.blocked++ def test_the_older_shape_generator_reaches_documents_the_ledger_acts_on(self) -> None:+ """The vacuity meta-test for the second generator, and it needs one just as much.++ Idempotence and prefix coherence are trivially true of a file with nothing to migrate. If+ the injector ever stopped injecting, both properties would stay green over a sampler of+ documents the ledger never touches.+ """+ found = find(_documents_at_an_older_shape(), lambda text: replay_ledger_over_text(ledger=SYNTHETIC_LEDGER, text=text).did_change_document)++ assert replay_ledger_over_text(ledger=SYNTHETIC_LEDGER, text=found).did_change_document++ def test_the_older_shape_generator_reaches_documents_where_the_rename_and_the_move_really_apply(self) -> None:+ """`did_change_document` is also true of a conflicting entry whose other operations landed.++ A generator that put the retired keys back without taking their successors out would reach+ `did_change_document` through the remap and the deletes alone, while the rename and the move+ conflicted on every draw — and idempotence would then be a claim about stable conflicts.+ """++ def rename_and_move_applied(text: str) -> bool:+ replay = replay_ledger_over_text(ledger=SYNTHETIC_LEDGER, text=text)+ applied = [op for step in replay.steps for op in step.applied_ops]+ return not replay.blocked and any(isinstance(op, RenameTableKeyOp) for op in applied) and any(isinstance(op, MoveKeyOp) for op in applied)++ found = find(_documents_at_an_older_shape(), rename_and_move_applied)++ assert rename_and_move_applied(found)++ @PROPERTY_SETTINGS+ @given(text=_documents_at_an_older_shape())+ def test_replaying_twice_lands_where_replaying_once_landed(self, text: str) -> None:+ """Idempotence. A user who runs the migration twice must not get a second set of changes."""+ once = replay_ledger_over_text(ledger=SYNTHETIC_LEDGER, text=text)+ twice = replay_ledger_over_text(ledger=SYNTHETIC_LEDGER, text=once.text)++ assert twice.text == once.text+ assert not twice.steps++ @PROPERTY_SETTINGS+ @given(text=_documents_at_an_older_shape(), absorbed=st.integers(min_value=0, max_value=len(SYNTHETIC_ENTRIES)))+ def test_a_file_that_already_absorbed_some_entries_lands_where_a_full_replay_lands(self, text: str, absorbed: int) -> None:+ """Prefix coherence. A file caught halfway — by an older release, or by hand — converges.++ This is what makes "every run replays everything" safe without a state stamp: where a file+ starts in the chain cannot change where it ends up.+ """+ partial = replay_ledger_over_text(ledger=_ledger(entries=SYNTHETIC_ENTRIES[:absorbed]), text=text)++ from_the_middle = replay_ledger_over_text(ledger=SYNTHETIC_LEDGER, text=partial.text)+ from_zero = replay_ledger_over_text(ledger=SYNTHETIC_LEDGER, text=text)++ assert from_the_middle.text == from_zero.text++ def test_the_witnesses_cannot_see_an_illegal_remap_and_the_property_can(self) -> None:+ """Why the property exists, shown on the one ledger that separates the two checks.++ A `safe` `remap_value` whose old spelling is still a legal member of the current enum+ breaks neutrality — but only for a file that carries *that* spelling. The reference+ document carries one value per key and it is not that one, so the convergence witness is+ green on this ledger. A generated document reaches the spelling, and the property is red.+ """+ illegal = _ledger(entries=[_entry(to_schema_version=2, ops=[RemapValueOp(table_path=[], key="tone", mapping={_Tone.LOUD: _Tone.QUIET})])])+ witness = SYNTHETIC_SURFACE.render_reference_document()++ assert replay_ledger_over_text(ledger=illegal, text=witness).text is witness++ found = find(+ _documents_for(surface=SYNTHETIC_SURFACE),+ lambda generated: replay_ledger_over_text(ledger=illegal, text=generated.text).text != generated.text,+ )+ assert DocumentMutation.SWAPPED_ENUM_MEMBER in found.mutationsdiff --git a/tests/unit/pipelex/migration/test_runner.py b/tests/unit/pipelex/migration/test_runner.pynew file mode 100644index 000000000..76c8f5287--- /dev/null+++ b/tests/unit/pipelex/migration/test_runner.py@@ -0,0 +1,579 @@+"""Unit tests for the file-level migration runner — backups, per-file scope, and what is written.++The scope is the point. A surface's files are independent user documents, so one that cannot be+processed is reported as blocked while every sibling is migrated normally — unlike the `.mthds`+fix loop, whose files only make sense together and which commits a round all-or-nothing.+"""++import errno+import stat+from collections.abc import Callable+from datetime import UTC, datetime+from pathlib import Path++from pytest_mock import MockerFixture++from pipelex.migration.backup import existing_backups_of, write_backup+from pipelex.migration.plan import FileBlockedReason+from pipelex.migration.runner import migrate_directories, migrate_file+from pipelex.migration.surfaces import SurfaceRegistry+from pipelex.pipeline.exceptions import FixTransactionError+from pipelex.pipeline.fixes.file_transaction import FileSnapshot, PendingFileUpdate, commit_file_updates+from pipelex.suggested_fix import RenameTableKeyOp+from tests.unit.pipelex.migration.conftest import EXAMPLE_SURFACE_ID, EntryBuilder, LedgerBuilder, SurfaceBuilder++MOMENT = datetime(2026, 8, 15, 12, 0, 0, tzinfo=UTC)++OLD_SHAPE = """\+[reporting]+output_config = { directory = "out" }+"""+++class TestMigrationRunner:+ def test_a_stale_file_is_rewritten_and_backed_up_exactly_once(+ self,+ tmp_path: Path,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ build_surface: SurfaceBuilder,+ ) -> None:+ target = tmp_path / "example.toml"+ target.write_text(OLD_SHAPE, encoding="utf-8")+ ledger = build_ledger(+ entries=[+ build_entry(+ to_schema_version=2,+ ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")],+ )+ ]+ )++ plan = migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=False, moment=MOMENT)++ assert plan.was_written+ assert "output = " in target.read_text(encoding="utf-8")+ backups = existing_backups_of(path=target)+ assert backups == [plan.backup_path]+ assert backups[0].read_text(encoding="utf-8") == OLD_SHAPE+ assert backups[0].name == "example.toml.bak.20260815T120000Z"++ def test_a_backup_inherits_the_source_mode_rather_than_the_umask(+ self,+ tmp_path: Path,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ build_surface: SurfaceBuilder,+ ) -> None:+ """A backup holds the user's values by definition, so a private file must not get a+ world-readable copy sitting beside it.+ """+ target = tmp_path / "example.toml"+ target.write_text(OLD_SHAPE, encoding="utf-8")+ target.chmod(0o600)+ ledger = build_ledger(+ entries=[+ build_entry(+ to_schema_version=2,+ ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")],+ )+ ]+ )++ plan = migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=False, moment=MOMENT)++ assert plan.backup_path is not None+ assert stat.S_IMODE(plan.backup_path.stat().st_mode) == 0o600+ assert stat.S_IMODE(target.stat().st_mode) == 0o600++ def test_an_older_backup_is_pruned_only_after_the_new_file_is_committed(+ self,+ tmp_path: Path,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ build_surface: SurfaceBuilder,+ ) -> None:+ target = tmp_path / "example.toml"+ target.write_text(OLD_SHAPE, encoding="utf-8")+ stale_backup = tmp_path / "example.toml.bak.20200101T000000Z"+ stale_backup.write_text("from another era\n", encoding="utf-8")+ ledger = build_ledger(+ entries=[+ build_entry(+ to_schema_version=2,+ ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")],+ )+ ]+ )++ plan = migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=False, moment=MOMENT)++ assert not stale_backup.exists()+ assert existing_backups_of(path=target) == [plan.backup_path]++ def test_a_dry_run_writes_nothing_and_backs_nothing_up(+ self,+ tmp_path: Path,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ build_surface: SurfaceBuilder,+ ) -> None:+ target = tmp_path / "example.toml"+ target.write_text(OLD_SHAPE, encoding="utf-8")+ ledger = build_ledger(+ entries=[+ build_entry(+ to_schema_version=2,+ ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")],+ )+ ]+ )++ plan = migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=True, moment=MOMENT)++ assert not plan.was_written+ assert plan.backup_path is None+ assert len(plan.steps) == 1+ assert target.read_text(encoding="utf-8") == OLD_SHAPE+ assert not existing_backups_of(path=target)++ def test_a_current_file_is_neither_written_nor_backed_up(+ self,+ tmp_path: Path,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ build_surface: SurfaceBuilder,+ ) -> None:+ """The overwhelmingly common case under always-replay, and it must cost the user nothing."""+ current = '[reporting]\noutput = { directory = "out" }\n'+ target = tmp_path / "example.toml"+ target.write_text(current, encoding="utf-8")+ ledger = build_ledger(+ entries=[+ build_entry(+ to_schema_version=2,+ ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")],+ )+ ]+ )++ plan = migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=False, moment=MOMENT)++ assert plan.is_clean+ assert not plan.was_written+ assert target.read_text(encoding="utf-8") == current+ assert not existing_backups_of(path=target)++ def test_a_users_own_backup_like_file_survives_the_pruning(+ self,+ tmp_path: Path,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ build_surface: SurfaceBuilder,+ ) -> None:+ """Only names carrying the stamp we write are ours to prune; anything else beside the file is the user's."""+ target = tmp_path / "example.toml"+ target.write_text(OLD_SHAPE, encoding="utf-8")+ users_own = tmp_path / "example.toml.bak.notes"+ users_own.write_text("my own copy\n", encoding="utf-8")+ stale_backup = tmp_path / "example.toml.bak.20200101T000000Z"+ stale_backup.write_text("from another era\n", encoding="utf-8")+ ledger = build_ledger(+ entries=[build_entry(to_schema_version=2, ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")])]+ )++ plan = migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=False, moment=MOMENT)++ assert plan.was_written+ assert users_own.read_text(encoding="utf-8") == "my own copy\n"+ assert not stale_backup.exists()+ assert existing_backups_of(path=target) == [plan.backup_path]++ def test_a_file_name_carrying_a_glob_metacharacter_never_prunes_a_siblings_backup(+ self,+ tmp_path: Path,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ build_surface: SurfaceBuilder,+ ) -> None:+ """Backups are found by string prefix, not by glob: `example_?.toml` must not see `example_a.toml`'s backups."""+ target = tmp_path / "example_?.toml"+ target.write_text(OLD_SHAPE, encoding="utf-8")+ sibling = tmp_path / "example_a.toml"+ sibling.write_text(OLD_SHAPE, encoding="utf-8")+ siblings_backup = tmp_path / "example_a.toml.bak.20200101T000000Z"+ siblings_backup.write_text("the sibling's only backup\n", encoding="utf-8")+ ledger = build_ledger(+ entries=[build_entry(to_schema_version=2, ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")])]+ )++ plan = migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=False, moment=MOMENT)++ assert plan.was_written+ assert siblings_backup.exists()+ assert existing_backups_of(path=target) == [plan.backup_path]+ assert existing_backups_of(path=sibling) == [siblings_backup]++ def test_a_backup_that_cannot_be_placed_leaves_no_staged_copy_behind(+ self,+ tmp_path: Path,+ mocker: MockerFixture,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ build_surface: SurfaceBuilder,+ ) -> None:+ target = tmp_path / "example.toml"+ target.write_text(OLD_SHAPE, encoding="utf-8")+ ledger = build_ledger(+ entries=[build_entry(to_schema_version=2, ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")])]+ )+ mocker.patch.object(Path, "replace", side_effect=OSError(errno.EACCES, "Permission denied"))++ plan = migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=False, moment=MOMENT)++ assert plan.blocked_reason is FileBlockedReason.UNWRITABLE+ assert not plan.was_written+ assert target.read_text(encoding="utf-8") == OLD_SHAPE+ assert sorted(path.name for path in tmp_path.iterdir()) == ["example.toml"]++ def test_a_transaction_error_after_the_write_landed_reports_the_file_as_written(+ self,+ tmp_path: Path,+ mocker: MockerFixture,+ build_surface: SurfaceBuilder,+ write_ledger_file: Callable[..., Path],+ ) -> None:+ """The primitive can raise after the target was replaced (its own temp cleanup failed). The plan must say what happened+ to the file, not what happened to the temp files — and the sibling must still be migrated.+ """+ config_dir = tmp_path / ".pipelex"+ config_dir.mkdir()+ first = config_dir / "example.toml"+ first.write_text(OLD_SHAPE, encoding="utf-8")+ second = config_dir / "example_local.toml"+ second.write_text(OLD_SHAPE, encoding="utf-8")+ migration_dir = tmp_path / "migration"+ write_ledger_file(migration_dir=migration_dir, surface_id=EXAMPLE_SURFACE_ID, body=_LEDGER_TOML)+ real_commit = commit_file_updates++ def commit_then_fail_cleanup(updates: list[PendingFileUpdate]) -> None:+ real_commit(updates)+ if updates[0].snapshot.path == first:+ msg = "target changes were committed, but temporary-file cleanup failed"+ raise FixTransactionError(msg)++ mocker.patch("pipelex.migration.runner.commit_file_updates", side_effect=commit_then_fail_cleanup)++ report = migrate_directories(+ registry=SurfaceRegistry(surfaces=[build_surface()]), migration_dir=migration_dir, config_dirs=[config_dir], dry_run=False, moment=MOMENT+ )++ assert [plan.was_written for plan in report.plans] == [True, True]+ assert all(plan.blocked_reason is None for plan in report.plans)+ assert "output = " in first.read_text(encoding="utf-8")+ assert existing_backups_of(path=first) == [report.plans[0].backup_path]++ def test_a_transaction_error_before_the_write_landed_blocks_the_file_and_spares_its_siblings(+ self,+ tmp_path: Path,+ mocker: MockerFixture,+ build_surface: SurfaceBuilder,+ write_ledger_file: Callable[..., Path],+ ) -> None:+ config_dir = tmp_path / ".pipelex"+ config_dir.mkdir()+ first = config_dir / "example.toml"+ first.write_text(OLD_SHAPE, encoding="utf-8")+ second = config_dir / "example_local.toml"+ second.write_text(OLD_SHAPE, encoding="utf-8")+ migration_dir = tmp_path / "migration"+ write_ledger_file(migration_dir=migration_dir, surface_id=EXAMPLE_SURFACE_ID, body=_LEDGER_TOML)+ real_commit = commit_file_updates++ def fail_first_commit(updates: list[PendingFileUpdate]) -> None:+ if updates[0].snapshot.path == first:+ msg = "commit failed and rollback was incomplete"+ raise FixTransactionError(msg)+ real_commit(updates)++ mocker.patch("pipelex.migration.runner.commit_file_updates", side_effect=fail_first_commit)++ report = migrate_directories(+ registry=SurfaceRegistry(surfaces=[build_surface()]), migration_dir=migration_dir, config_dirs=[config_dir], dry_run=False, moment=MOMENT+ )++ assert [plan.was_written for plan in report.plans] == [False, True]+ assert report.plans[0].blocked_reason is FileBlockedReason.UNWRITABLE+ assert first.read_text(encoding="utf-8") == OLD_SHAPE+ # The transaction could not vouch for the state it left, so the one certain copy of the+ # original stays, and the report says where it is.+ assert existing_backups_of(path=first) == [report.plans[0].backup_path]+ assert report.plans[0].backup_path is not None+ assert report.plans[0].backup_path.read_text(encoding="utf-8") == OLD_SHAPE+ assert str(report.plans[0].backup_path) in (report.plans[0].blocked_detail or "")+ assert "output = " in second.read_text(encoding="utf-8")++ def test_a_file_edited_between_the_read_and_the_write_is_reported_as_changed_and_left_alone(+ self,+ tmp_path: Path,+ mocker: MockerFixture,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ build_surface: SurfaceBuilder,+ ) -> None:+ """The snapshot/compare design exists for exactly this: a concurrent edit is never overwritten,+ the user's edit is what stays on disk, and the copy this run made of the pre-edit text goes.+ """+ target = tmp_path / "example.toml"+ target.write_text(OLD_SHAPE, encoding="utf-8")+ ledger = build_ledger(+ entries=[build_entry(to_schema_version=2, ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")])]+ )+ edited_shape = OLD_SHAPE + "extra = 1\n"+ real_write_backup = write_backup++ def edit_target_while_backing_up(*, snapshot: FileSnapshot, moment: datetime) -> Path:+ backup_path = real_write_backup(snapshot=snapshot, moment=moment)+ target.write_text(edited_shape, encoding="utf-8")+ return backup_path++ mocker.patch("pipelex.migration.runner.write_backup", side_effect=edit_target_while_backing_up)++ plan = migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=False, moment=MOMENT)++ assert plan.blocked_reason is FileBlockedReason.CHANGED_DURING_RUN+ assert not plan.was_written+ assert target.read_text(encoding="utf-8") == edited_shape+ assert not existing_backups_of(path=target)++ def test_a_write_that_fails_before_touching_the_file_takes_its_backup_with_it(+ self,+ tmp_path: Path,+ mocker: MockerFixture,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ build_surface: SurfaceBuilder,+ ) -> None:+ target = tmp_path / "example.toml"+ target.write_text(OLD_SHAPE, encoding="utf-8")+ ledger = build_ledger(+ entries=[build_entry(to_schema_version=2, ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")])]+ )+ mocker.patch("pipelex.migration.runner.commit_file_updates", side_effect=OSError(errno.EACCES, "Permission denied"))++ plan = migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=False, moment=MOMENT)++ assert plan.blocked_reason is FileBlockedReason.UNWRITABLE+ assert not plan.was_written+ assert target.read_text(encoding="utf-8") == OLD_SHAPE+ assert sorted(path.name for path in tmp_path.iterdir()) == ["example.toml"]++ def test_a_backup_that_will_not_go_after_a_refused_write_is_a_warning_not_an_abort(+ self,+ tmp_path: Path,+ mocker: MockerFixture,+ build_surface: SurfaceBuilder,+ write_ledger_file: Callable[..., Path],+ ) -> None:+ config_dir = tmp_path / ".pipelex"+ config_dir.mkdir()+ first = config_dir / "example.toml"+ first.write_text(OLD_SHAPE, encoding="utf-8")+ second = config_dir / "example_local.toml"+ second.write_text(OLD_SHAPE, encoding="utf-8")+ migration_dir = tmp_path / "migration"+ write_ledger_file(migration_dir=migration_dir, surface_id=EXAMPLE_SURFACE_ID, body=_LEDGER_TOML)+ real_commit = commit_file_updates++ def refuse_first_commit(updates: list[PendingFileUpdate]) -> None:+ if updates[0].snapshot.path == first:+ raise OSError(errno.EACCES, "Permission denied")+ real_commit(updates)++ mocker.patch("pipelex.migration.runner.commit_file_updates", side_effect=refuse_first_commit)+ real_unlink = Path.unlink++ def refuse_backup_unlink(self: Path, missing_ok: bool = False) -> None:+ if ".bak." in self.name:+ raise OSError(errno.EPERM, "Operation not permitted")+ real_unlink(self, missing_ok=missing_ok)++ mocker.patch.object(Path, "unlink", refuse_backup_unlink)++ report = migrate_directories(+ registry=SurfaceRegistry(surfaces=[build_surface()]), migration_dir=migration_dir, config_dirs=[config_dir], dry_run=False, moment=MOMENT+ )++ assert [plan.was_written for plan in report.plans] == [False, True]+ assert report.plans[0].blocked_reason is FileBlockedReason.UNWRITABLE+ assert "output = " in second.read_text(encoding="utf-8")++ def test_a_file_that_is_not_utf8_is_blocked_as_unparseable(+ self,+ tmp_path: Path,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ build_surface: SurfaceBuilder,+ ) -> None:+ target = tmp_path / "example.toml"+ target.write_bytes(b"\xff\xfe[reporting]\n")+ ledger = build_ledger(+ entries=[build_entry(to_schema_version=2, ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")])]+ )++ plan = migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=False, moment=MOMENT)++ assert plan.blocked_reason is FileBlockedReason.UNPARSEABLE+ assert "UTF-8" in (plan.blocked_detail or "")+ assert not existing_backups_of(path=target)++ def test_a_file_that_cannot_be_read_is_blocked_and_nothing_is_written(+ self,+ tmp_path: Path,+ mocker: MockerFixture,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ build_surface: SurfaceBuilder,+ ) -> None:+ target = tmp_path / "example.toml"+ target.write_text(OLD_SHAPE, encoding="utf-8")+ ledger = build_ledger(+ entries=[build_entry(to_schema_version=2, ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")])]+ )+ mocker.patch("pipelex.migration.runner.read_file_snapshot", side_effect=PermissionError(errno.EACCES, "Permission denied"))++ plan = migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=False, moment=MOMENT)++ assert plan.blocked_reason is FileBlockedReason.UNWRITABLE+ assert "could not be read" in (plan.blocked_detail or "")+ assert sorted(path.name for path in tmp_path.iterdir()) == ["example.toml"]++ def test_a_file_removed_before_it_is_read_is_reported_as_changed_during_the_run(+ self,+ tmp_path: Path,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ build_surface: SurfaceBuilder,+ ) -> None:+ target = tmp_path / "example.toml"+ ledger = build_ledger(+ entries=[build_entry(to_schema_version=2, ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")])]+ )++ plan = migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=False, moment=MOMENT)++ assert plan.blocked_reason is FileBlockedReason.CHANGED_DURING_RUN+ assert not plan.was_written++ def test_a_pruning_failure_never_turns_a_written_file_into_a_blocked_one(+ self,+ tmp_path: Path,+ mocker: MockerFixture,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ build_surface: SurfaceBuilder,+ ) -> None:+ target = tmp_path / "example.toml"+ target.write_text(OLD_SHAPE, encoding="utf-8")+ stale_backup = tmp_path / "example.toml.bak.20200101T000000Z"+ stale_backup.write_text("from another era\n", encoding="utf-8")+ ledger = build_ledger(+ entries=[build_entry(to_schema_version=2, ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")])]+ )+ mocker.patch("pipelex.migration.runner.prune_backups_except", side_effect=OSError(errno.EACCES, "Permission denied"))++ plan = migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=False, moment=MOMENT)++ assert plan.was_written+ assert plan.blocked_reason is None+ assert "output = " in target.read_text(encoding="utf-8")+ assert stale_backup.exists()++ def test_an_unparseable_file_is_blocked_and_its_siblings_are_still_migrated(+ self,+ tmp_path: Path,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ build_surface: SurfaceBuilder,+ write_ledger_file: Callable[..., Path],+ ) -> None:+ migration_dir = tmp_path / "migration"+ surface = build_surface()+ ledger = build_ledger(+ entries=[+ build_entry(+ to_schema_version=2,+ ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")],+ )+ ]+ )+ write_ledger_file(migration_dir=migration_dir, surface_id=EXAMPLE_SURFACE_ID, body=_LEDGER_TOML)++ config_dir = tmp_path / ".pipelex"+ config_dir.mkdir()+ broken = config_dir / "example.toml"+ broken.write_text("this is not = = toml\n", encoding="utf-8")+ healthy = config_dir / "example_local.toml"+ healthy.write_text(OLD_SHAPE, encoding="utf-8")++ report = migrate_directories(+ registry=SurfaceRegistry(surfaces=[surface]),+ migration_dir=migration_dir,+ config_dirs=[config_dir, tmp_path / "does-not-exist"],+ dry_run=False,+ moment=MOMENT,+ )++ assert len(report.plans) == 2+ blocked_plan = next(plan for plan in report.plans if plan.file_path == broken)+ assert blocked_plan.blocked_reason is FileBlockedReason.UNPARSEABLE+ assert not blocked_plan.was_written+ healthy_plan = next(plan for plan in report.plans if plan.file_path == healthy)+ assert healthy_plan.was_written+ assert "output = " in healthy.read_text(encoding="utf-8")+ assert ledger.surface.current_schema_version == 2++ def test_a_missing_configuration_directory_is_skipped_rather_than_refused(+ self,+ tmp_path: Path,+ build_surface: SurfaceBuilder,+ write_ledger_file: Callable[..., Path],+ ) -> None:+ migration_dir = tmp_path / "migration"+ write_ledger_file(migration_dir=migration_dir, surface_id=EXAMPLE_SURFACE_ID, body=_LEDGER_TOML)++ report = migrate_directories(+ registry=SurfaceRegistry(surfaces=[build_surface()]),+ migration_dir=migration_dir,+ config_dirs=[tmp_path / "nowhere"],+ dry_run=True,+ moment=MOMENT,+ )++ assert not report.plans+++_LEDGER_TOML = """\+[surface]+id = "example-config"+title = "An example configuration surface"+base_file = "example.toml"+tier_glob = "example_*.toml"+current_schema_version = 2+min_supported_schema_version = 0++[[migration]]+id = "example-config@2"+to_schema_version = 2+introduced_in = "0.46.0"+breaking = true+safety = "safe"+title = "Rename the reporting output table"+description = "The reporting output table lost its suffix."++[[migration.ops]]+kind = "rename_table_key"+table_path = ["reporting"]+key = "output_config"+new_key = "output"+"""diff --git a/tests/unit/pipelex/migration/test_surface_resolution.py b/tests/unit/pipelex/migration/test_surface_resolution.pynew file mode 100644index 000000000..ee1bb17f1--- /dev/null+++ b/tests/unit/pipelex/migration/test_surface_resolution.py@@ -0,0 +1,82 @@+"""Unit tests for resolving which surface owns a file.++Which ledger runs over a file must never be an accident of iteration order. The registry refuses+what it can decide by name alone when it loads; what it cannot decide — whether two glob languages+overlap — is decided here, where there is a real file to point at.+"""++import re+from pathlib import Path++import pytest++from pipelex.migration.exceptions import MigrationRegistryError+from pipelex.migration.surfaces import SurfaceRegistry+from tests.unit.pipelex.migration.conftest import SurfaceBuilder+++class TestSurfaceResolution:+ def test_an_exact_base_file_claims_before_another_surfaces_glob(self, build_surface: SurfaceBuilder) -> None:+ """The real configuration: `pipelex_service.toml` also matches `pipelex_*.toml`."""+ registry = SurfaceRegistry(+ surfaces=[+ build_surface(surface_id="pipelex-config", base_file="pipelex.toml", tier_glob="pipelex_*.toml"),+ build_surface(surface_id="pipelex-service-config", base_file="pipelex_service.toml", tier_glob=None),+ ]+ )++ resolved = registry.surface_for_file_name(file_name="pipelex_service.toml")++ assert resolved is not None+ assert resolved.surface_id == "pipelex-service-config"++ def test_a_tier_file_resolves_to_the_surface_whose_glob_matches(self, build_surface: SurfaceBuilder) -> None:+ registry = SurfaceRegistry(surfaces=[build_surface(surface_id="pipelex-config", base_file="pipelex.toml", tier_glob="pipelex_*.toml")])++ resolved = registry.surface_for_file_name(file_name="pipelex_staging.toml")++ assert resolved is not None+ assert resolved.surface_id == "pipelex-config"++ def test_a_file_no_surface_claims_resolves_to_nothing(self, build_surface: SurfaceBuilder) -> None:+ registry = SurfaceRegistry(surfaces=[build_surface(surface_id="pipelex-config", base_file="pipelex.toml", tier_glob="pipelex_*.toml")])++ assert registry.surface_for_file_name(file_name="plxt.toml") is None++ def test_a_file_two_globs_both_claim_stops_by_name(self, build_surface: SurfaceBuilder) -> None:+ """The registry accepts these globs — deciding overlap between glob *languages* is not its+ job, and it has no files to look at. Here there is one, and it is refused with its name.++ Today's registry has one glob per family and none overlap, so nothing else would catch a+ future one.+ """+ registry = SurfaceRegistry(+ surfaces=[+ build_surface(surface_id="a", base_file="a.toml", tier_glob="pipelex_*.toml"),+ build_surface(surface_id="b", base_file="b.toml", tier_glob="*_local.toml"),+ ]+ )++ with pytest.raises(MigrationRegistryError, match=re.escape("pipelex_local.toml")):+ registry.surface_for_file_name(file_name="pipelex_local.toml")++ def test_a_directory_walk_returns_the_claimed_files_with_their_surfaces(+ self,+ tmp_path: Path,+ build_surface: SurfaceBuilder,+ ) -> None:+ registry = SurfaceRegistry(surfaces=[build_surface(surface_id="example-config", base_file="example.toml", tier_glob="example_*.toml")])+ (tmp_path / "example.toml").write_text("", encoding="utf-8")+ (tmp_path / "example_local.toml").write_text("", encoding="utf-8")+ (tmp_path / "unrelated.toml").write_text("", encoding="utf-8")+ (tmp_path / "example_dir.toml").mkdir()++ claimed = registry.files_by_surface_in_directory(directory=tmp_path)++ assert [path.name for _, path in claimed] == ["example.toml", "example_local.toml"]++ def test_a_missing_directory_is_skipped_rather_than_refused(self, tmp_path: Path, build_surface: SurfaceBuilder) -> None:+ """A machine with only a global configuration directory is an ordinary machine."""+ registry = SurfaceRegistry(surfaces=[build_surface()])++ assert registry.files_by_surface_in_directory(directory=tmp_path / "nowhere") == []diff --git a/tests/unit/pipelex/migration/test_telemetry_back_entry.py b/tests/unit/pipelex/migration/test_telemetry_back_entry.pynew file mode 100644index 000000000..5a2debaea--- /dev/null+++ b/tests/unit/pipelex/migration/test_telemetry_back_entry.py@@ -0,0 +1,61 @@+"""The one migration the package ships today, exercised on the shape it is about.++Everything else about migration behaviour is tested against synthetic surfaces, deliberately. This+module is the exception the entry earns: `telemetry-config@2` is about a real file that real users+still have, and what it must do to that file cannot be asked of a stand-in.++Two hardcoded sniffs still tell such a user to re-initialize their `telemetry.toml` — `doctor`'s+telemetry row and the telemetry validation error handler — which throws away every choice they+made. What is asserted here is what replaces them: the ledger carries a flat file onto the current+shape, keeping the values, and produces something the model accepts.+"""++from typing import Any++from pipelex.migration.engine import replay_ledger_over_text+from pipelex.migration.goldens import pre_history_document_path+from pipelex.migration.ledger import load_ledger+from pipelex.migration.surfaces import packaged_migration_dir+from pipelex.system.telemetry.telemetry_config import PostHogMode, TelemetryConfig+from pipelex.tools.misc.toml_utils import load_toml_from_content++SURFACE_ID = "telemetry-config"+++class TestTheTelemetryBackEntry:+ def _old_shape_document(self) -> str:+ """The hand-authored `[email protected]` — the flat shape, as the entry's own check reads it."""+ path = pre_history_document_path(migration_dir=packaged_migration_dir(), surface_id=SURFACE_ID, schema_version=2)+ return path.read_text(encoding="utf-8")++ def test_replaying_the_ledger_over_a_flat_file_produces_a_document_the_model_accepts(self) -> None:+ ledger = load_ledger(migration_dir=packaged_migration_dir(), surface_id=SURFACE_ID)+ replay = replay_ledger_over_text(ledger=ledger, text=self._old_shape_document())++ assert [step.entry_id for step in replay.steps] == ["telemetry-config@2"]+ assert replay.blocked == []++ migrated: dict[str, Any] = load_toml_from_content(replay.text)+ assert TelemetryConfig.model_validate(migrated).custom_posthog.mode is PostHogMode.ANONYMOUS+ # Every value the flat era set is carried, under the key the current shape gives it, and+ # nothing else is left at the root — data fidelity is what this entry is for, so it is+ # asserted rather than left to `extra="forbid"`.+ assert migrated == {+ "custom_posthog": {+ "mode": "anonymous",+ "endpoint": "https://eu.i.posthog.com",+ "api_key": "phc_example_project_api_key",+ "user_id": "",+ "geoip": True,+ "debug": False,+ "redact_properties": ["prompt", "system_prompt", "response", "file_path", "url"],+ }+ }++ def test_replaying_it_again_changes_nothing(self) -> None:+ """The migrated file is a current file, and a current file is what replay must not touch."""+ ledger = load_ledger(migration_dir=packaged_migration_dir(), surface_id=SURFACE_ID)+ once = replay_ledger_over_text(ledger=ledger, text=self._old_shape_document())+ twice = replay_ledger_over_text(ledger=ledger, text=once.text)+ assert twice.steps == []+ assert twice.text is once.textdiff --git a/tests/unit/pipelex/migration/test_transform_check.py b/tests/unit/pipelex/migration/test_transform_check.pynew file mode 100644index 000000000..090dfa679--- /dev/null+++ b/tests/unit/pipelex/migration/test_transform_check.py@@ -0,0 +1,413 @@+"""Unit tests for the transform goldens — does an entry's operations really do what it claims?++Every test here points the check at a **synthetic** surface whose golden chain is written by hand+into a temporary directory, for the reason the whole registry is an injected parameter: wire the+real configuration models in and every legitimate configuration change turns this suite red+alongside the gate, which is how a gate goes permanently green while catching nothing. The real+surfaces get one smoke test, in `test_real_surfaces.py`.++Writing the reference documents by hand rather than snapshotting them from a model is deliberate+too: a golden chain *is* checked-in text, and half of what this check has to tolerate — a comment+edited between versions, a default flipped, a key added to a packaged document — cannot be+expressed by a model at all.+"""++from enum import StrEnum+from pathlib import Path++import pytest+from pydantic import BaseModel, ConfigDict, Field++from pipelex.migration.exceptions import MigrationGoldenError+from pipelex.migration.fingerprint import compute_fingerprint+from pipelex.migration.goldens import defaults_golden_path, pre_history_document_path, write_defaults_golden, write_fingerprint_golden+from pipelex.migration.ledger import ledgers_dir+from pipelex.migration.surfaces import DefaultsLayerKind, Surface+from pipelex.migration.transform_check import TransformIssue, TransformIssueKind, check_transform_chain++SURFACE_ID = "synthetic-config"+++class _TierAtTwo(StrEnum):+ PREMIUM = "premium"+ STANDARD = "standard"+++class _Renamed(BaseModel):+ """Schema 2 of the rename story: `label` is now `title`."""++ model_config = ConfigDict(extra="forbid")++ title: str = "hello"+++class _RenamedWithMotto(BaseModel):+ """The same bump, which also added a key — the additive change every honest bump carries."""++ model_config = ConfigDict(extra="forbid")++ title: str = "hello"+ motto: str = "onwards"+++class _WithOptional(BaseModel):+ """Schema 2 with an optional destination: legal path, no value any TOML document can carry."""++ model_config = ConfigDict(extra="forbid")++ nickname: str | None = None+++class _DeckEntry(BaseModel):+ model_config = ConfigDict(extra="forbid")++ new_name: str = "a"+++class _Deck(BaseModel):+ model_config = ConfigDict(extra="forbid")++ deck: dict[str, _DeckEntry] = Field(default_factory=dict[str, _DeckEntry])+++class _Settings(BaseModel):+ model_config = ConfigDict(extra="forbid")++ attempts: int = 1+++class _WithSettings(BaseModel):+ model_config = ConfigDict(extra="forbid")++ settings: _Settings = Field(default_factory=_Settings)+++class _Tiered(BaseModel):+ model_config = ConfigDict(extra="forbid")++ tier: _TierAtTwo = _TierAtTwo.PREMIUM+++class _Inner(BaseModel):+ model_config = ConfigDict(extra="forbid")++ inner_new: int = 1+++class _Shell(BaseModel):+ model_config = ConfigDict(extra="forbid")++ shell: _Inner = Field(default_factory=_Inner)+++def _surface(*, config_model: type[BaseModel]) -> Surface:+ return Surface(+ surface_id=SURFACE_ID,+ title="A synthetic surface",+ base_file="synthetic.toml",+ tier_glob="synthetic_*.toml",+ config_model=config_model,+ defaults_layer_kind=DefaultsLayerKind.MODEL_DEFAULTS,+ )+++def _write_ledger(*, migration_dir: Path, entries: str, current_schema_version: int = 2) -> None:+ directory = ledgers_dir(migration_dir=migration_dir)+ directory.mkdir(parents=True, exist_ok=True)+ body = f"""+[surface]+id = "{SURFACE_ID}"+title = "A synthetic surface"+base_file = "synthetic.toml"+tier_glob = "synthetic_*.toml"+current_schema_version = {current_schema_version}+min_supported_schema_version = 0+{entries}+"""+ (directory / f"{SURFACE_ID}.toml").write_text(body, encoding="utf-8")+++def _entry(*, ops: str, to_schema_version: int = 2, safety: str = "safe") -> str:+ return f"""+[[migration]]+id = "{SURFACE_ID}@{to_schema_version}"+to_schema_version = {to_schema_version}+introduced_in = "0.46.0"+breaking = true+safety = "{safety}"+title = "Reshape the synthetic surface"+description = "The surface changed shape."+{ops}+"""+++def _defaults(*, migration_dir: Path, schema_version: int, document: str) -> None:+ write_defaults_golden(migration_dir=migration_dir, surface_id=SURFACE_ID, schema_version=schema_version, document=document)+++def _rename(*, key: str, new_key: str, table_path: str = "[]") -> str:+ return f"""+[[migration.ops]]+kind = "rename_table_key"+table_path = {table_path}+key = "{key}"+new_key = "{new_key}"+"""+++def _kinds(issues: list[TransformIssue]) -> list[TransformIssueKind]:+ return [issue.kind for issue in issues]+++def _messages(issues: list[TransformIssue]) -> str:+ return " ".join(issue.message for issue in issues)+++def _pre_history_entry(*, ops: str) -> str:+ return f"""+[[migration]]+id = "{SURFACE_ID}@2"+to_schema_version = 2+introduced_in = "0.46.0"+breaking = true+safety = "safe"+title = "Carry a shape that predates the chain"+description = "There was no snapshot on the far side of this one."+pre_history = true+declared_removed_paths = ["legacy_label"]+{ops}+"""+++def _pre_history_document(*, migration_dir: Path, document: str) -> None:+ path = pre_history_document_path(migration_dir=migration_dir, surface_id=SURFACE_ID, schema_version=2)+ path.parent.mkdir(parents=True, exist_ok=True)+ path.write_text(document, encoding="utf-8")+++def _fingerprint(*, migration_dir: Path, config_model: type[BaseModel], schema_version: int) -> None:+ write_fingerprint_golden(+ migration_dir=migration_dir,+ fingerprint=compute_fingerprint(+ surface_id=SURFACE_ID,+ schema_version=schema_version,+ config_model=config_model,+ defaults_document=_surface(config_model=config_model).read_defaults_document(),+ ),+ )+++class TestThePreHistoryLink:+ def test_a_pre_history_entry_is_verified_against_its_hand_authored_document(self, tmp_path: Path) -> None:+ """The exception's whole shape: a different starting document, then the same three claims."""+ _write_ledger(migration_dir=tmp_path, entries=_pre_history_entry(ops=_rename(key="legacy_label", new_key="title")))+ _pre_history_document(migration_dir=tmp_path, document='legacy_label = "hello"\n')+ _defaults(migration_dir=tmp_path, schema_version=2, document='title = "hello"\n')+ assert check_transform_chain(surface=_surface(config_model=_Renamed), migration_dir=tmp_path) == []++ def test_a_pre_history_entry_with_no_document_to_start_from_is_refused(self, tmp_path: Path) -> None:+ """Nothing else verifies it, so an absent document is the escape hatch reopening."""+ _write_ledger(migration_dir=tmp_path, entries=_pre_history_entry(ops=_rename(key="legacy_label", new_key="title")))+ _defaults(migration_dir=tmp_path, schema_version=2, document='title = "hello"\n')+ issues = check_transform_chain(surface=_surface(config_model=_Renamed), migration_dir=tmp_path)+ assert _kinds(issues) == [TransformIssueKind.PRE_HISTORY_DOCUMENT_MISSING]+ assert "[email protected]" in _messages(issues)++ def test_a_misspelled_destination_is_caught_on_the_pre_history_link_too(self, tmp_path: Path) -> None:+ """The exception changes where the link starts, and nothing about what it proves."""+ _write_ledger(migration_dir=tmp_path, entries=_pre_history_entry(ops=_rename(key="legacy_label", new_key="titel")))+ _pre_history_document(migration_dir=tmp_path, document='legacy_label = "hello"\n')+ _defaults(migration_dir=tmp_path, schema_version=2, document='title = "hello"\n')+ issues = check_transform_chain(surface=_surface(config_model=_Renamed), migration_dir=tmp_path)+ assert TransformIssueKind.DESTINATION_NOT_IN_NEW_SHAPE in _kinds(issues)+ assert "titel" in _messages(issues)+++class TestTheTransformGoldens:+ def test_a_destination_the_fingerprint_records_but_the_document_cannot_carry_is_tolerated(self, tmp_path: Path) -> None:+ """An optional key defaulting to `None` is a legal path with no value in any document.++ TOML has no null, so the reference document simply lacks it — and a migration moving a+ user's value onto it creates a path that document does not have. Checking the document+ alone would refuse the one destination the schema most obviously has.+ """+ _write_ledger(migration_dir=tmp_path, entries=_entry(ops=_rename(key="label", new_key="nickname")))+ _defaults(migration_dir=tmp_path, schema_version=1, document='label = "hello"\n')+ _defaults(migration_dir=tmp_path, schema_version=2, document="")+ _fingerprint(migration_dir=tmp_path, config_model=_WithOptional, schema_version=2)+ assert check_transform_chain(surface=_surface(config_model=_WithOptional), migration_dir=tmp_path) == []++ def test_a_surface_that_has_never_changed_shape_has_no_link_to_check(self, tmp_path: Path) -> None:+ """The overwhelmingly common state, and the one all three surfaces are in today."""+ _write_ledger(migration_dir=tmp_path, entries="", current_schema_version=1)+ _defaults(migration_dir=tmp_path, schema_version=1, document='label = "hello"\n')+ assert check_transform_chain(surface=_surface(config_model=_Renamed), migration_dir=tmp_path) == []++ def test_a_correct_rename_lands_exactly_on_the_new_shape(self, tmp_path: Path) -> None:+ _write_ledger(migration_dir=tmp_path, entries=_entry(ops=_rename(key="label", new_key="title")))+ _defaults(migration_dir=tmp_path, schema_version=1, document='label = "hello"\n')+ _defaults(migration_dir=tmp_path, schema_version=2, document='title = "hello"\n')+ assert check_transform_chain(surface=_surface(config_model=_Renamed), migration_dir=tmp_path) == []++ def test_a_misspelled_destination_is_refused(self, tmp_path: Path) -> None:+ """The defect the transform goldens exist for: coverage and convergence both pass it.++ The removed path is accounted for, and over a current document the source is absent so the+ operation skips — and every user file is then migrated to a key the schema rejects, with+ the tool reporting success.+ """+ _write_ledger(migration_dir=tmp_path, entries=_entry(ops=_rename(key="label", new_key="titel")))+ _defaults(migration_dir=tmp_path, schema_version=1, document='label = "hello"\n')+ _defaults(migration_dir=tmp_path, schema_version=2, document='title = "hello"\n')+ issues = check_transform_chain(surface=_surface(config_model=_Renamed), migration_dir=tmp_path)+ assert _kinds(issues) == [TransformIssueKind.DESTINATION_NOT_IN_NEW_SHAPE, TransformIssueKind.MIGRATED_DOCUMENT_REJECTED]+ assert "titel" in _messages(issues)++ def test_operations_applied_in_the_wrong_order_are_refused(self, tmp_path: Path) -> None:+ """Renaming the parent first leaves the child operation addressing a table that is gone.++ It skips, silently and forever — the applier cannot tell a dead operation from one whose+ work is already done. What lands is the new parent holding the old child key.+ """+ ops = _rename(key="outer", new_key="shell") + _rename(key="inner_old", new_key="inner_new", table_path='["outer"]')+ _write_ledger(migration_dir=tmp_path, entries=_entry(ops=ops))+ _defaults(migration_dir=tmp_path, schema_version=1, document="[outer]\ninner_old = 1\n")+ _defaults(migration_dir=tmp_path, schema_version=2, document="[shell]\ninner_new = 1\n")+ issues = check_transform_chain(surface=_surface(config_model=_Shell), migration_dir=tmp_path)+ assert TransformIssueKind.DESTINATION_NOT_IN_NEW_SHAPE in _kinds(issues)+ assert "shell.inner_old" in _messages(issues)++ def test_over_deletion_is_refused(self, tmp_path: Path) -> None:+ """A `delete_table` one level too high takes material the new shape still has.++ A subset comparison would pass this, which is why the check is not one.+ """+ ops = """+[[migration.ops]]+kind = "delete_table"+table_path = ["settings"]+"""+ _write_ledger(migration_dir=tmp_path, entries=_entry(ops=ops))+ _defaults(migration_dir=tmp_path, schema_version=1, document="[settings]\nretries = 3\nattempts = 1\n")+ _defaults(migration_dir=tmp_path, schema_version=2, document="[settings]\nattempts = 1\n")+ issues = check_transform_chain(surface=_surface(config_model=_WithSettings), migration_dir=tmp_path)+ assert _kinds(issues) == [TransformIssueKind.SURVIVING_PATH_REMOVED]+ assert "'settings'" in issues[0].message++ def test_a_remap_to_a_spelling_the_new_schema_rejects_is_refused(self, tmp_path: Path) -> None:+ """The value-side defect, which moves no path at all — so only the last link can see it.++ Reading the migrated document beneath the current defaults is what a user's boot does, and+ it is the only oracle that stays sound: comparing values against the new reference document+ would go red whenever a default was flipped in the same commit, with no remedy available.+ """+ ops = """+[[migration.ops]]+kind = "remap_value"+table_path = []+key = "tier"+mapping = { basic = "standrad" }+"""+ _write_ledger(migration_dir=tmp_path, entries=_entry(ops=ops))+ _defaults(migration_dir=tmp_path, schema_version=1, document='tier = "basic"\n')+ _defaults(migration_dir=tmp_path, schema_version=2, document='tier = "premium"\n')+ issues = check_transform_chain(surface=_surface(config_model=_Tiered), migration_dir=tmp_path)+ assert _kinds(issues) == [TransformIssueKind.MIGRATED_DOCUMENT_REJECTED]+ assert "standrad" in issues[0].message++ def test_an_operation_that_conflicts_on_the_reference_document_is_refused(self, tmp_path: Path) -> None:+ """The document at the version an entry migrates *from* is the one it must be able to migrate."""+ _write_ledger(migration_dir=tmp_path, entries=_entry(ops=_rename(key="label", new_key="title")))+ _defaults(migration_dir=tmp_path, schema_version=1, document='label = "hello"\ntitle = "taken"\n')+ _defaults(migration_dir=tmp_path, schema_version=2, document='title = "taken"\n')+ issues = check_transform_chain(surface=_surface(config_model=_Renamed), migration_dir=tmp_path)+ assert _kinds(issues) == [TransformIssueKind.TRANSFORM_CONFLICTED]++ def test_a_key_the_new_document_gained_is_tolerated(self, tmp_path: Path) -> None:+ """Every honest bump adds keys in the same commit; a comparator red on that is worthless."""+ _write_ledger(migration_dir=tmp_path, entries=_entry(ops=_rename(key="label", new_key="title")))+ _defaults(migration_dir=tmp_path, schema_version=1, document='label = "hello"\n')+ _defaults(migration_dir=tmp_path, schema_version=2, document='title = "hello"\nmotto = "onwards"\n')+ assert check_transform_chain(surface=_surface(config_model=_RenamedWithMotto), migration_dir=tmp_path) == []++ def test_an_edited_comment_and_a_flipped_default_are_tolerated(self, tmp_path: Path) -> None:+ """Neither moves a path, and both are ordinary edits to a document we ship."""+ _write_ledger(migration_dir=tmp_path, entries=_entry(ops=_rename(key="label", new_key="title")))+ _defaults(migration_dir=tmp_path, schema_version=1, document='# what to greet with\nlabel = "hello"\n')+ _defaults(migration_dir=tmp_path, schema_version=2, document='# how to greet\ntitle = "goodbye"\n')+ assert check_transform_chain(surface=_surface(config_model=_Renamed), migration_dir=tmp_path) == []++ def test_an_entry_added_beneath_an_open_mapping_is_tolerated(self, tmp_path: Path) -> None:+ """The keys of an open mapping are user key space, and no fingerprint records one.++ A comparator built on the fingerprint diff would be blind to this by construction and go+ red the first time we shipped one more model in a packaged deck.+ """+ _write_ledger(migration_dir=tmp_path, entries=_entry(ops=_rename(key="old_name", new_key="new_name", table_path='["deck", "*"]')))+ _defaults(migration_dir=tmp_path, schema_version=1, document='[deck.claude]\nold_name = "b"\n')+ _defaults(migration_dir=tmp_path, schema_version=2, document='[deck.claude]\nnew_name = "b"\n\n[deck.gpt6]\nnew_name = "c"\n')+ assert check_transform_chain(surface=_surface(config_model=_Deck), migration_dir=tmp_path) == []++ def test_a_destination_beneath_an_open_mapping_that_the_new_document_omits_is_tolerated(self, tmp_path: Path) -> None:+ """The fingerprint records `deck.*.new_name`; the document carries `deck.claude.new_name`, or not.++ A packaged deck entry can drop a key in the same commit that renames it, because the new+ default is what it wanted anyway. The destination is then absent from the reference+ document and present in the fingerprint only under the wildcard — a literal comparison+ would call the correct rename a misspelled destination.+ """+ _write_ledger(migration_dir=tmp_path, entries=_entry(ops=_rename(key="old_name", new_key="new_name", table_path='["deck", "*"]')))+ _defaults(migration_dir=tmp_path, schema_version=1, document='[deck.claude]\nold_name = "b"\n')+ _defaults(migration_dir=tmp_path, schema_version=2, document="[deck.claude]\n")+ _fingerprint(migration_dir=tmp_path, config_model=_Deck, schema_version=2)+ assert check_transform_chain(surface=_surface(config_model=_Deck), migration_dir=tmp_path) == []++ def test_a_reference_document_that_cannot_be_read_is_refused_by_name(self, tmp_path: Path) -> None:+ """A golden that exists but cannot be read is a broken golden, not a crash: the gate names it."""+ _write_ledger(migration_dir=tmp_path, entries=_entry(ops=_rename(key="label", new_key="title")))+ _defaults(migration_dir=tmp_path, schema_version=2, document='title = "hello"\n')+ defaults_golden_path(migration_dir=tmp_path, surface_id=SURFACE_ID, schema_version=1).write_bytes(b"\xff\xfe not utf-8")+ with pytest.raises(MigrationGoldenError, match=r"defaults@1\.toml"):+ check_transform_chain(surface=_surface(config_model=_Renamed), migration_dir=tmp_path)++ def test_an_entry_dropped_from_an_open_mapping_does_not_blame_the_operation(self, tmp_path: Path) -> None:+ """When a whole container is gone from the new document, what happened inside it says nothing.++ The wildcard rename is correct here; only the packaged deck lost one of its entries in the+ same commit. Without the ancestry rule the correct operation would be reported as having+ produced a path the new shape does not have.+ """+ _write_ledger(migration_dir=tmp_path, entries=_entry(ops=_rename(key="old_name", new_key="new_name", table_path='["deck", "*"]')))+ _defaults(migration_dir=tmp_path, schema_version=1, document='[deck.gpt5]\nold_name = "a"\n\n[deck.claude]\nold_name = "b"\n')+ _defaults(migration_dir=tmp_path, schema_version=2, document='[deck.claude]\nnew_name = "b"\n')+ assert check_transform_chain(surface=_surface(config_model=_Deck), migration_dir=tmp_path) == []++ def test_an_unsafe_entry_is_not_held_to_the_new_shape(self, tmp_path: Path) -> None:+ """An unsafe entry is reported and never applied, so no document makes this transition.++ The vocabulary grants it the right to be incomplete — an entry with no operations at all is+ legal precisely when it is unsafe — so demanding its operations reach the new shape would+ contradict the one thing `unsafe` means.+ """+ _write_ledger(migration_dir=tmp_path, entries=_entry(ops="", safety="unsafe"))+ _defaults(migration_dir=tmp_path, schema_version=1, document='label = "hello"\n')+ _defaults(migration_dir=tmp_path, schema_version=2, document='title = "hello"\n')+ assert check_transform_chain(surface=_surface(config_model=_Renamed), migration_dir=tmp_path) == []++ def test_a_frozen_reference_document_that_went_missing_is_named(self, tmp_path: Path) -> None:+ """A link below the head is history: it is restored, never regenerated."""+ _write_ledger(migration_dir=tmp_path, entries=_entry(ops=_rename(key="label", new_key="title")))+ _defaults(migration_dir=tmp_path, schema_version=2, document='title = "hello"\n')+ issues = check_transform_chain(surface=_surface(config_model=_Renamed), migration_dir=tmp_path)+ assert _kinds(issues) == [TransformIssueKind.DEFAULTS_GOLDEN_MISSING]+ assert "[email protected]" in issues[0].message++ def test_the_head_link_waits_for_its_snapshot_in_silence(self, tmp_path: Path) -> None:+ """Between a bump and the `umig` that snapshots it, the coverage gate already says so.++ It names the missing snapshot and the command that writes it; saying it a second time here+ would only make the same bump look like two problems.+ """+ _write_ledger(migration_dir=tmp_path, entries=_entry(ops=_rename(key="label", new_key="title")))+ _defaults(migration_dir=tmp_path, schema_version=1, document='label = "hello"\n')+ assert check_transform_chain(surface=_surface(config_model=_Renamed), migration_dir=tmp_path) == []diff --git a/tests/unit/pipelex/pipeline/fixes/test_fix_applier_config_surface_shapes.py b/tests/unit/pipelex/pipeline/fixes/test_fix_applier_config_surface_shapes.pyindex 8ba53bc6d..ffa88f696 100644--- a/tests/unit/pipelex/pipeline/fixes/test_fix_applier_config_surface_shapes.py+++ b/tests/unit/pipelex/pipeline/fixes/test_fix_applier_config_surface_shapes.py@@ -27,7 +27,7 @@ import tomlkitfrom tomlkit import TOMLDocumentfrom pipelex.pipeline.fixes.applier import FixOpOutcome, apply_fix_ops-from pipelex.suggested_fix import DeleteKeyOp, DeleteTableOp, EnsureTableOp, RenameTableKeyOp+from pipelex.suggested_fix import DeleteKeyOp, DeleteTableOp, EnsureTableOp, MoveKeyOp, RenameTableKeyOp# Every configuration file this repository owns and tracks. The packaged defaults are the complete# witness (every path present); the kit templates are the sparse shape real user files have — and@@ -217,6 +217,61 @@ class TestFixApplierConfigSurfaceShapes:reparsed = tomlkit.parse(rendered)assert reparsed["interpreter"] == original["pipelex"]+ @pytest.mark.parametrize(+ "text",+ [+ pytest.param("[a.b]\nx = 1\n[a.c]\ny = 2\n[a.b.d]\nz = 3\n", id="grandchild continued after a sibling"),+ pytest.param("[a]\n[a.b]\nx = 1\n[a.c]\ny = 2\n[a.b.d]\nz = 3\n", id="same, with an explicit header"),+ pytest.param("[a.b.c]\nx = 1\n[a.b.e]\ny = 1\n[a.b.c.d]\nz = 1\n", id="split two levels down"),+ pytest.param("[a]\nb.x = 1\nc = 2\nb.y = 3\n", id="dotted keys interleaved in one table"),+ pytest.param("[a.b]\nx = 1\n[[a.b.l]]\nq = 1\n[a.c]\ny = 2\n[a.b.d]\nz = 3\n", id="array of tables between the chunks"),+ ],+ )+ def test_a_table_whose_descendant_is_split_across_chunks_moves_and_renames_whole(self, text: str) -> None:+ """A split *descendant* (a child declared, interrupted by a sibling, then continued) is several+ tomlkit tables under one key, and the library's own header-cache invalidation visits only the+ first of them — left alone, the rest would keep rendering under the old name. The applier+ refreshes every chunk, so the rename or move comes out whole and re-parses to the same value+ under the new path, in the original order.+ """+ original = tomlkit.parse(text).unwrap()["a"]+ for op, expected in (+ (RenameTableKeyOp(table_path=[], key="a", new_key="zz"), {"zz": original}),+ (MoveKeyOp(table_path=[], key="a", new_table_path=["q"], new_key="a"), {"q": {"a": original}}),+ ):+ toml_doc = tomlkit.parse(text)+ applications = apply_fix_ops(toml_doc=toml_doc, ops=[op])+ assert [application.outcome for application in applications] == [FixOpOutcome.APPLIED]+ rendered = _dumps(toml_doc)+ assert "[a" not in rendered+ assert tomlkit.parse(rendered).unwrap() == expected++ def test_a_split_descendant_beneath_a_root_proxy_still_renames_whole(self) -> None:+ """The two shapes compose: a table interleaved at its own level *and* carrying a split child+ goes through the proxy path, which renames in every chunk — every child lands under the new name.+ """+ text = "[a.b]\nx = 1\n[other]\nz = 1\n[a.c]\ny = 2\n[a.b.d]\nz = 3\n"+ toml_doc = tomlkit.parse(text)+ applications = apply_fix_ops(toml_doc=toml_doc, ops=[RenameTableKeyOp(table_path=[], key="a", new_key="zz")])+ assert [application.outcome for application in applications] == [FixOpOutcome.APPLIED]+ reparsed = tomlkit.parse(_dumps(toml_doc)).unwrap()+ assert "a" not in reparsed+ assert reparsed["zz"] == {"b": {"x": 1, "d": {"z": 3}}, "c": {"y": 2}}++ def test_a_split_descendant_beneath_a_nested_proxy_still_renames_whole(self) -> None:+ """Same composition one level down: the renamed key lives inside an out-of-order table that is+ itself nested, and one of its chunks carries a split child. The header refresh must reach+ the chunks of a *nested* proxy too — the merged facade would show it one table per key.+ """+ text = "[p.a.b]\nx = 1\n[q]\nw = 0\n[p.a.c]\ny = 1\n[p.a.b.d]\nz = 1\n[p.a.e]\nv = 1\n[p.a.b.f]\nu = 1\n"+ original = tomlkit.parse(text).unwrap()["p"]["a"]+ toml_doc = tomlkit.parse(text)+ applications = apply_fix_ops(toml_doc=toml_doc, ops=[RenameTableKeyOp(table_path=["p"], key="a", new_key="zz")])+ assert [application.outcome for application in applications] == [FixOpOutcome.APPLIED]+ rendered = _dumps(toml_doc)+ assert "[p.a" not in rendered+ assert tomlkit.parse(rendered).unwrap()["p"] == {"zz": original}+def test_rename_is_replay_neutral_once_applied(self) -> None:"""Re-applying a rename over an already-migrated document skips and changes no bytes.diff --git a/tests/unit/pipelex/pipeline/fixes/test_fix_applier_rename_dom_consistency.py b/tests/unit/pipelex/pipeline/fixes/test_fix_applier_rename_dom_consistency.pynew file mode 100644index 000000000..d8d426059--- /dev/null+++ b/tests/unit/pipelex/pipeline/fixes/test_fix_applier_rename_dom_consistency.py@@ -0,0 +1,138 @@+"""Characterization of what a rename leaves behind in the tomlkit DOM.++`_rename_key_in_place` renames through `Container._replace`, tomlkit's own position-preserving+primitive — the only one there is, and the reason a renamed key keeps its place among its siblings+instead of being appended at the bottom of its parent. What it does **not** do, for a value that+is not a `Table`, is update the node's raw `dict` storage: the old key stays in the dict and the+new one is only ever written into it inside the table branch of `_replace_at`.++Everything tomlkit renders or looks up still works — `dumps`, `in`, `[]`, `.get()` all read the+authoritative body — so a single rename followed by serialization is correct, which is why the+`.mthds` fix path never met this: the keys it renames are `[pipe.*]` tables, and tables take the+branch that keeps the dict in step.++A migration is the case that meets it, because always-replay runs many operations over one+document. The consequence is pinned below: addressing a renamed non-table key again in the same+DOM raises `KeyError` from inside the library. `pipelex.migration.engine` works around it by+re-reading the document between operations that applied, and this module is the tripwire — **if+these tests go red after a tomlkit bump, the workaround can be reconsidered**, and the engine's+docstring says so.+"""++from typing import Any, cast++import pytest+import tomlkit+from tomlkit import TOMLDocument++from pipelex.pipeline.fixes.applier import FixOpOutcome, apply_fix_ops+from pipelex.suggested_fix import DeleteKeyOp, RenameTableKeyOp++FLAT = """\+[reporting]+output_config = { directory = "out" }+retention = 30+"""++OUT_OF_ORDER = """\+[pipelex]+alpha = 1++[other]+z = 0++[pipelex.sub]+b = 2+"""+++def _raw_dict_keys(*, node: object) -> set[str]:+ """The keys of a tomlkit node's *raw* dict storage, bypassing its overriding accessors.++ Every public accessor reads the authoritative body instead, which is precisely why the+ staleness this pins is invisible until something calls `dict.__delitem__`.+ """+ raw_mapping = cast("dict[str, Any]", node)+ return {str(key) for key in cast("list[Any]", list(dict.keys(raw_mapping)))} # pyright: ignore[reportUnknownArgumentType, reportUnknownMemberType]+++def _dumped(*, toml_doc: TOMLDocument) -> str:+ dumped: str = tomlkit.dumps(toml_doc) # pyright: ignore[reportUnknownMemberType]+ return dumped+++TABLE_VALUED = """\+[pipe]+[pipe.old_name]+type = "PipeLLM"+"""+++class TestRenameDomConsistency:+ def test_a_renamed_scalar_serializes_correctly_and_reads_back_through_every_accessor(self) -> None:+ toml_doc = tomlkit.loads(FLAT)++ applications = apply_fix_ops(toml_doc=toml_doc, ops=[RenameTableKeyOp(table_path=["reporting"], key="retention", new_key="keep_days")])++ assert applications[0].outcome is FixOpOutcome.APPLIED+ assert "keep_days = 30" in _dumped(toml_doc=toml_doc)+ reporting = cast("dict[str, Any]", toml_doc["reporting"])+ assert "keep_days" in reporting+ assert reporting["keep_days"] == 30++ def test_the_raw_dict_storage_is_left_stale_by_the_rename(self) -> None:+ """The measured defect itself, stated in one assertion so a tomlkit fix is visible."""+ toml_doc = tomlkit.loads(FLAT)+ apply_fix_ops(toml_doc=toml_doc, ops=[RenameTableKeyOp(table_path=["reporting"], key="retention", new_key="keep_days")])++ raw_keys = _raw_dict_keys(node=toml_doc["reporting"])+ assert "retention" in raw_keys+ assert "keep_days" not in raw_keys++ @pytest.mark.parametrize(+ ("text", "table_path", "key", "new_key"),+ [+ (FLAT, ["reporting"], "retention", "keep_days"),+ (OUT_OF_ORDER, ["pipelex"], "alpha", "beta"),+ ],+ )+ def test_addressing_a_renamed_key_again_in_the_same_dom_raises_from_inside_tomlkit(+ self,+ text: str,+ table_path: list[str],+ key: str,+ new_key: str,+ ) -> None:+ """Both parent shapes are affected: a plain table and an out-of-order table proxy.++ This is what the migration engine's re-read exists to avoid. It is deliberately asserted+ as a raise rather than worked around here: the applier's guarded-skip contract is about+ targets that do not resolve, and this target does resolve — the library's own bookkeeping+ is what disagrees with itself.+ """+ toml_doc = tomlkit.loads(text)+ apply_fix_ops(toml_doc=toml_doc, ops=[RenameTableKeyOp(table_path=table_path, key=key, new_key=new_key)])++ with pytest.raises(KeyError):+ apply_fix_ops(toml_doc=toml_doc, ops=[DeleteKeyOp(table_path=table_path, key=new_key)])++ def test_renaming_a_table_valued_key_keeps_the_dom_consistent(self) -> None:+ """The branch tomlkit does maintain — and the reason the `.mthds` fix path never met this."""+ toml_doc = tomlkit.loads(TABLE_VALUED)++ apply_fix_ops(toml_doc=toml_doc, ops=[RenameTableKeyOp(table_path=["pipe"], key="old_name", new_key="new_name")])+ applications = apply_fix_ops(toml_doc=toml_doc, ops=[DeleteKeyOp(table_path=["pipe", "new_name"], key="type")])++ assert applications[0].outcome is FixOpOutcome.APPLIED+ assert "old_name" not in _dumped(toml_doc=toml_doc)++ def test_re_reading_between_operations_is_what_makes_the_sequence_work(self) -> None:+ """The workaround, shown at its smallest: serialize, parse, continue."""+ toml_doc = tomlkit.loads(FLAT)+ apply_fix_ops(toml_doc=toml_doc, ops=[RenameTableKeyOp(table_path=["reporting"], key="retention", new_key="keep_days")])++ re_read = tomlkit.loads(_dumped(toml_doc=toml_doc))+ applications = apply_fix_ops(toml_doc=re_read, ops=[DeleteKeyOp(table_path=["reporting"], key="keep_days")])++ assert applications[0].outcome is FixOpOutcome.APPLIED+ assert "keep_days" not in _dumped(toml_doc=re_read)diff --git a/tests/unit/pipelex/pipeline/fixes/test_fix_file_transaction.py b/tests/unit/pipelex/pipeline/fixes/test_fix_file_transaction.pyindex 3cdef5b87..9257fd9b2 100644--- a/tests/unit/pipelex/pipeline/fixes/test_fix_file_transaction.py+++ b/tests/unit/pipelex/pipeline/fixes/test_fix_file_transaction.py@@ -6,13 +6,15 @@ import pytestfrom pytest_mock import MockerFixturefrom pipelex.pipeline.exceptions import FixTransactionError, FixWriteConflictError+from pipelex.pipeline.fixes.file_transaction import (+ FileSnapshot,+ PendingFileUpdate,+ assert_snapshot_unchanged,+ commit_file_updates,+ read_file_snapshot,+)from pipelex.pipeline.fixes.fix_loop import (- _assert_snapshot_unchanged, # pyright: ignore[reportPrivateUsage] # noqa: PLC2701_bind_fixes_to_snapshots, # pyright: ignore[reportPrivateUsage] # noqa: PLC2701- _commit_file_updates, # pyright: ignore[reportPrivateUsage] # noqa: PLC2701- _FileSnapshot, # pyright: ignore[reportPrivateUsage]- _PendingFileUpdate, # pyright: ignore[reportPrivateUsage] # noqa: PLC2701- _read_file_snapshot, # pyright: ignore[reportPrivateUsage] # noqa: PLC2701_snapshot_loaded_sources, # pyright: ignore[reportPrivateUsage] # noqa: PLC2701)from pipelex.suggested_fix import FixSafety, SuggestedFix@@ -37,11 +39,11 @@ class TestFixFileTransaction:def test_concurrent_edit_is_not_overwritten(self, tmp_path: Path) -> None:target = tmp_path / "bundle.mthds"target.write_text("original\n", encoding="utf-8")- snapshot = _read_file_snapshot(target)+ snapshot = read_file_snapshot(target)target.write_text("user edit\n", encoding="utf-8")with pytest.raises(FixWriteConflictError, match="file changed"):- _commit_file_updates([_PendingFileUpdate(snapshot=snapshot, new_content="fixed\n")])+ commit_file_updates([PendingFileUpdate(snapshot=snapshot, new_content="fixed\n")])assert target.read_text(encoding="utf-8") == "user edit\n"assert list(tmp_path.glob(".*.pipelex-fix-*.tmp")) == []@@ -49,11 +51,11 @@ class TestFixFileTransaction:def test_failed_atomic_replace_preserves_original(self, tmp_path: Path, mocker: MockerFixture) -> None:target = tmp_path / "bundle.mthds"target.write_text("original\n", encoding="utf-8")- snapshot = _read_file_snapshot(target)+ snapshot = read_file_snapshot(target)mocker.patch.object(Path, "replace", side_effect=OSError("replace failed"))with pytest.raises(OSError, match="replace failed"):- _commit_file_updates([_PendingFileUpdate(snapshot=snapshot, new_content="fixed\n")])+ commit_file_updates([PendingFileUpdate(snapshot=snapshot, new_content="fixed\n")])assert target.read_text(encoding="utf-8") == "original\n"assert list(tmp_path.glob(".*.pipelex-fix-*.tmp")) == []@@ -66,9 +68,9 @@ class TestFixFileTransaction:second_path.write_text("second original\n", encoding="utf-8")third_path.write_text("third original\n", encoding="utf-8")updates = [- _PendingFileUpdate(snapshot=_read_file_snapshot(first_path), new_content="first fixed\n"),- _PendingFileUpdate(snapshot=_read_file_snapshot(second_path), new_content="second fixed\n"),- _PendingFileUpdate(snapshot=_read_file_snapshot(third_path), new_content="third fixed\n"),+ PendingFileUpdate(snapshot=read_file_snapshot(first_path), new_content="first fixed\n"),+ PendingFileUpdate(snapshot=read_file_snapshot(second_path), new_content="second fixed\n"),+ PendingFileUpdate(snapshot=read_file_snapshot(third_path), new_content="third fixed\n"),]original_replace = Path.replacereplacement_count = 0@@ -88,7 +90,7 @@ class TestFixFileTransaction:mocker.patch.object(Path, "replace", new=fail_third_replacement)with pytest.raises(OSError, match="third replace failed"):- _commit_file_updates(updates)+ commit_file_updates(updates)assert first_path.read_text(encoding="utf-8") == "first original\n"assert second_path.read_text(encoding="utf-8") == "second original\n"@@ -104,25 +106,25 @@ class TestFixFileTransaction:second_path.write_text("second original\n", encoding="utf-8")third_path.write_text("third original\n", encoding="utf-8")updates = [- _PendingFileUpdate(snapshot=_read_file_snapshot(first_path), new_content="first fixed\n"),- _PendingFileUpdate(snapshot=_read_file_snapshot(second_path), new_content="second fixed\n"),- _PendingFileUpdate(snapshot=_read_file_snapshot(third_path), new_content="third fixed\n"),+ PendingFileUpdate(snapshot=read_file_snapshot(first_path), new_content="first fixed\n"),+ PendingFileUpdate(snapshot=read_file_snapshot(second_path), new_content="second fixed\n"),+ PendingFileUpdate(snapshot=read_file_snapshot(third_path), new_content="third fixed\n"),]assertion_count = 0- def inject_edit_before_third_commit(snapshot: _FileSnapshot) -> None:+ def inject_edit_before_third_commit(snapshot: FileSnapshot) -> None:nonlocal assertion_countassertion_count += 1if assertion_count == 3:second_path.write_text("concurrent user edit\n", encoding="utf-8")msg = "injected third-file conflict"raise FixWriteConflictError(msg)- _assert_snapshot_unchanged(snapshot)+ assert_snapshot_unchanged(snapshot)- mocker.patch("pipelex.pipeline.fixes.fix_loop._assert_snapshot_unchanged", new=inject_edit_before_third_commit)+ mocker.patch("pipelex.pipeline.fixes.file_transaction.assert_snapshot_unchanged", new=inject_edit_before_third_commit)with pytest.raises(FixTransactionError, match="rollback was incomplete") as exc_info:- _commit_file_updates(updates)+ commit_file_updates(updates)assert first_path.read_text(encoding="utf-8") == "first original\n"assert second_path.read_text(encoding="utf-8") == "concurrent user edit\n"@@ -134,7 +136,7 @@ class TestFixFileTransaction:def test_cleanup_failure_reports_committed_outcome(self, tmp_path: Path, mocker: MockerFixture) -> None:target = tmp_path / "bundle.mthds"target.write_text("original\n", encoding="utf-8")- snapshot = _read_file_snapshot(target)+ snapshot = read_file_snapshot(target)original_unlink = Path.unlinkdef fail_temp_cleanup(path: Path, *, missing_ok: bool = False) -> None:@@ -146,20 +148,20 @@ class TestFixFileTransaction:mocker.patch.object(Path, "unlink", new=fail_temp_cleanup)with pytest.raises(FixTransactionError, match=r"target changes were committed.*cleanup failed"):- _commit_file_updates([_PendingFileUpdate(snapshot=snapshot, new_content="fixed\n")])+ commit_file_updates([PendingFileUpdate(snapshot=snapshot, new_content="fixed\n")])assert target.read_text(encoding="utf-8") == "fixed\n"def test_cleanup_failure_does_not_mask_commit_failure(self, tmp_path: Path, mocker: MockerFixture) -> None:target = tmp_path / "bundle.mthds"target.write_text("original\n", encoding="utf-8")- snapshot = _read_file_snapshot(target)+ snapshot = read_file_snapshot(target)mocker.patch.object(Path, "replace", side_effect=OSError("primary replace failure"))mocker.patch.object(Path, "unlink", side_effect=OSError("secondary cleanup failure"))with pytest.raises(OSError, match="primary replace failure") as exc_info:- _commit_file_updates([_PendingFileUpdate(snapshot=snapshot, new_content="fixed\n")])+ commit_file_updates([PendingFileUpdate(snapshot=snapshot, new_content="fixed\n")])assert target.read_text(encoding="utf-8") == "original\n"assert len(exc_info.value.__notes__) == 1diff --git a/tests/unit/pipelex/test_fix_op_union.py b/tests/unit/pipelex/test_fix_op_union.pyindex d1b8ef61a..02261d076 100644--- a/tests/unit/pipelex/test_fix_op_union.py+++ b/tests/unit/pipelex/test_fix_op_union.py@@ -118,12 +118,17 @@ class TestFixOpUnion:{"op": {"kind": "move_key", "table_path": ["deck", "*"], "key": "k", "new_table_path": ["deck", "*"], "new_key": "k"}})- def test_a_wildcard_source_is_accepted(self) -> None:- """The source side is the whole point of the segment, and stays legal."""- op = _FixOpHolder.model_validate(- {"op": {"kind": "move_key", "table_path": ["deck", "*"], "key": "k", "new_table_path": ["storage"], "new_key": "k"}}- ).op- assert isinstance(op, MoveKeyOp)+ def test_a_wildcard_source_move_is_refused(self) -> None:+ """A move's destination is one fixed key, so every matched entry would move onto the same one and the second conflicts."""+ with pytest.raises(ValidationError, match="wildcard"):+ _FixOpHolder.model_validate(+ {"op": {"kind": "move_key", "table_path": ["deck", "*"], "key": "k", "new_table_path": ["storage"], "new_key": "k"}}+ )++ def test_a_wildcard_source_is_accepted_where_it_acts_in_place(self) -> None:+ """The source side is the whole point of the segment: each matched entry is acted on in place."""+ op = _FixOpHolder.model_validate({"op": {"kind": "rename_table_key", "table_path": ["deck", "*"], "key": "k", "new_key": "n"}}).op+ assert isinstance(op, RenameTableKeyOp)assert op.table_path == ["deck", "*"]def test_a_remap_needs_a_non_empty_mapping(self) -> None:diff --git a/uv.lock b/uv.lockindex 6869fffb7..08660bb13 100644--- a/uv.lock+++ b/uv.lock@@ -1581,6 +1581,90 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/49/79/621a7dbb80c70974f73a597275351ebe03ce5bc65cb5f8f4acb5859252bc/huggingface_hub-1.16.1-py3-none-any.whl", hash = "sha256:64340de934b9ce37857ef85a82de72f5629e8a270f9119eabb12bf495eb53c22", size = 668176, upload-time = "2026-05-21T18:39:58.596Z" },]+[[package]]+name = "hypothesis"+version = "6.165.9"+source = { registry = "https://pypi.org/simple" }+dependencies = [+ { name = "sortedcontainers" },+]+sdist = { url = "https://files.pythonhosted.org/packages/68/a8/f7afb4b55a7f00307f246fb7c3cf2abdc8f8e0ad7eb81bf23f8025480da6/hypothesis-6.165.9.tar.gz", hash = "sha256:a5d1fa312020b499c517e1217ef4440c56a9b52e20b47b420b48a583a81c412b", size = 503594, upload-time = "2026-08-15T02:50:07.518Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/fe/c7/da4298709de649a01c73ce93c4ee5b5651433217e17c494dc91baded86b0/hypothesis-6.165.9-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:01a2a4106fa90396ffa1c402968a8bd25af056e1fdf7974a369c4057cc4035f1", size = 783107, upload-time = "2026-08-15T02:49:32.01Z" },+ { url = "https://files.pythonhosted.org/packages/58/97/d9f265174464d05a4d818d98dcd4e870ad2c2b60e9844033c23b3eb21eda/hypothesis-6.165.9-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:7812f1ce9e2bfc0958f2a96975f16a1e1f2400f9d8b7043723ff5811647fe0f6", size = 778677, upload-time = "2026-08-15T02:48:16.984Z" },+ { url = "https://files.pythonhosted.org/packages/ae/fe/d3364f99dd76f1275b430b2fc2818119df84e71f0657ad7f89b922e90d55/hypothesis-6.165.9-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e547823eca16eab9c9569f292e5ce8f95d8a59f1d67e504e97a841d17eb0b562", size = 1107895, upload-time = "2026-08-15T02:47:42.954Z" },+ { url = "https://files.pythonhosted.org/packages/71/5d/42b524303219351add359a05463df067be126c9ca576b0547218cf8aa54b/hypothesis-6.165.9-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:e50c687c0e86bb5898f36b845ab84439f44ef2369f0f9bbcbf6aa6860640324d", size = 1136455, upload-time = "2026-08-15T02:49:56.069Z" },+ { url = "https://files.pythonhosted.org/packages/af/d5/91948f68b5007a40b2cc356d9365de0f5cb08c200e383b38259ad75130ae/hypothesis-6.165.9-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:f82aed6f0631b5073bd87789aa56609e7c83bd762384fd24c84f3435816b2569", size = 1135112, upload-time = "2026-08-15T02:48:51.211Z" },+ { url = "https://files.pythonhosted.org/packages/55/3c/6e04ac80c5a63da9ecff00199851433437f617da9cd1b5208f11af3337b8/hypothesis-6.165.9-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:488b88b6856e430763a7ac5a5fc6e20a798766dca8df8f94576de2281ee60afd", size = 1157397, upload-time = "2026-08-15T02:49:27.874Z" },+ { url = "https://files.pythonhosted.org/packages/12/40/4a1aeddc254d8d6a8f43bbeea928ef5b576e041eb0c9459a91fce884f4cd/hypothesis-6.165.9-cp310-abi3-manylinux_2_31_riscv64.whl", hash = "sha256:22dd9e264f2a79a1b63a5fae13351e3cf0088f584fb05bb0e23f21b8fba9df7c", size = 1112748, upload-time = "2026-08-15T02:48:35.764Z" },+ { url = "https://files.pythonhosted.org/packages/35/bc/dde8231f1d10ab0c60627c40757b9b832605028ef0780420fcfc3ccba664/hypothesis-6.165.9-cp310-abi3-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:431ab894062666c93157234f9949a4db3549fbf3c220777abae3482e1b43f9bc", size = 1149484, upload-time = "2026-08-15T02:48:53.088Z" },+ { url = "https://files.pythonhosted.org/packages/1b/b2/cb29a983c7f06770a6b348f76cb33f90a70b0594a1e61c4a803aedfb1d1f/hypothesis-6.165.9-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:c7edcc3dcec42b413c900f8e660f3741c21f0055c53bde21d728f72a21713247", size = 1283267, upload-time = "2026-08-15T02:47:55.688Z" },+ { url = "https://files.pythonhosted.org/packages/88/c0/6bfcf3fb379aba87f3d09073b589c7b7e932ad047ed7bc0124300a51f431/hypothesis-6.165.9-cp310-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:0d4a613404ba216b6f698ce9df6f9c22d04d8e8a571c42525e83905cf1b1cd10", size = 1409841, upload-time = "2026-08-15T02:48:23.939Z" },+ { url = "https://files.pythonhosted.org/packages/98/c0/0c73a985144717272d2afec377a45c149688a477af946ae24c16b59c9788/hypothesis-6.165.9-cp310-abi3-musllinux_1_2_ppc64le.whl", hash = "sha256:658c33d8a54cead6081f73fcc4e2c0fbe43d251d3f4170630748d8e5dcbd542f", size = 1264878, upload-time = "2026-08-15T02:50:05.056Z" },+ { url = "https://files.pythonhosted.org/packages/56/91/50d4a6df07a305f5f17578443c93b618a49e8db68c01de8870e539139eb9/hypothesis-6.165.9-cp310-abi3-musllinux_1_2_riscv64.whl", hash = "sha256:7d3297fe95a253cea81b48e581e456105016ebe782d823dfb57ec09c3854301f", size = 1282519, upload-time = "2026-08-15T02:48:22.42Z" },+ { url = "https://files.pythonhosted.org/packages/c9/5f/5a54f4ca30a669a247df7c467aea7bb854d9b4de7e1dc51edb2f91ec603f/hypothesis-6.165.9-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:a9a1e17200a2b369a38c19c2184efbc041a40b784675ba72ac6800d8c1260a54", size = 1324669, upload-time = "2026-08-15T02:48:29.144Z" },+ { url = "https://files.pythonhosted.org/packages/3a/b2/6ef8eac3b30d6c8225fc405be3b99790701a942706ec93a3bd638a0b614a/hypothesis-6.165.9-cp310-abi3-win32.whl", hash = "sha256:5d40a92c6d025725da7827431a71b7657ce2e3923ff8f1773d6b15d717f77ea0", size = 668903, upload-time = "2026-08-15T02:48:34.033Z" },+ { url = "https://files.pythonhosted.org/packages/f0/7f/b9514f65d2cc97277f6442e3b05164177a742489940efa53e794a54b62f6/hypothesis-6.165.9-cp310-abi3-win_amd64.whl", hash = "sha256:59f29ef2965caf1ae367ee0598ac0d293551b95744a5d49888737a3b3d9dd944", size = 675051, upload-time = "2026-08-15T02:47:49.697Z" },+ { url = "https://files.pythonhosted.org/packages/96/01/110123106a3702d2f3ffc47b9bc6a1a29c19cd2d15fcf356430777a4883a/hypothesis-6.165.9-cp310-abi3-win_arm64.whl", hash = "sha256:f52fd4f1b95f614c3fec53840d603f0a3b19e68eaa91c8d609222999c3ee5506", size = 673402, upload-time = "2026-08-15T02:47:48.364Z" },+ { url = "https://files.pythonhosted.org/packages/de/1d/1c2dea64a3b00b9f83ca57bacaf4982255da7797d6c65fc0301f21a0ca94/hypothesis-6.165.9-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:a9995cbf91e0b22cb8d9aaaf6b10994b2ac1ebc3b01b4a91d3e82761a2737043", size = 783582, upload-time = "2026-08-15T02:49:29.881Z" },+ { url = "https://files.pythonhosted.org/packages/aa/8c/d8f51166ef9feaeb08d33737b2874d1effca4ada8bf6598e6f2a634a3ca1/hypothesis-6.165.9-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:f94e4456f7b0dd04f4f80f26a2c499370f6048f393ff61436de4e192e278e2df", size = 779367, upload-time = "2026-08-15T02:48:47.639Z" },+ { url = "https://files.pythonhosted.org/packages/8c/27/749aa12ec03fdcec1e4fa7289145f03ded46e892ae89613e333dd7090a7e/hypothesis-6.165.9-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:bab95b4ef980cbc2b4632e49057588766cda930ffbd727d70edfea289da98dd3", size = 1108251, upload-time = "2026-08-15T02:47:59.983Z" },+ { url = "https://files.pythonhosted.org/packages/2b/cb/a1c417cd39f87ca11e2b5b4a7da0446ae316b43b9baee23e113a59113759/hypothesis-6.165.9-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8795161db0a7c270c4f3e0eb754bd0ed96b17133d6f851c7444f9733c4c9d37f", size = 1157689, upload-time = "2026-08-15T02:49:40.288Z" },+ { url = "https://files.pythonhosted.org/packages/5d/d5/82abf4d3b171f087d323b179a422472e1c1621ed47adf88f0fcdf7be7e35/hypothesis-6.165.9-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:8c7dcb0a21dab077e65bcb082ee321b7e31402cf980da648b4369c71086ad53c", size = 1283592, upload-time = "2026-08-15T02:48:04.698Z" },+ { url = "https://files.pythonhosted.org/packages/0b/ee/f156d374fd7f49220a789a231cd980ffe53bf1eec11a75306ded76265306/hypothesis-6.165.9-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:92b264bef7860bf37b5f22b5949083d26f113351da8b99e93adb0554d3e2c110", size = 1324910, upload-time = "2026-08-15T02:48:11.037Z" },+ { url = "https://files.pythonhosted.org/packages/23/57/d14969c84e59356103c0b5e1a27623a4149d5d371799757552b2ec344da2/hypothesis-6.165.9-cp311-cp311-win_amd64.whl", hash = "sha256:ae34d13d8438ea7f3ee18292290b9f608de6070309f3c5826644a88d197c319d", size = 674776, upload-time = "2026-08-15T02:48:59.008Z" },+ { url = "https://files.pythonhosted.org/packages/dc/7e/03a84262b74300714af486e037403db8f5e19170f01bc32605300896b19d/hypothesis-6.165.9-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:41082aff3e438ac1eeb86002c269b9b5d86bab563e911614c2c8d34f405e02db", size = 784696, upload-time = "2026-08-15T02:48:09.314Z" },+ { url = "https://files.pythonhosted.org/packages/49/50/c77dca38074e38e99373ccd5f361bb5ad251e8f2de92dff4d7035111595d/hypothesis-6.165.9-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:67a0c0346ac0e8e106fad0224269a87c665768cc8d45de6e367e9b260ef36d5a", size = 776265, upload-time = "2026-08-15T02:49:36.232Z" },+ { url = "https://files.pythonhosted.org/packages/1b/80/9de2e558cd966ab1b1d014718c7aaab7d2d8f2d591ccadb27c1ffe639f6d/hypothesis-6.165.9-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:10e826a708456c965e125f15dbf5a6f463e41dcf230a930b7a3e0ce7036f5176", size = 1106706, upload-time = "2026-08-15T02:49:12.466Z" },+ { url = "https://files.pythonhosted.org/packages/04/78/dc89684de1a739347b22beadf3e53a64b8b7eef287dfa34b150432e13d4e/hypothesis-6.165.9-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fba20d9c60d6b8aec76f32cd22109e1b1f19a9985317255e2b43940d27996097", size = 1156753, upload-time = "2026-08-15T02:48:20.512Z" },+ { url = "https://files.pythonhosted.org/packages/b9/bc/495988f0fc788d33a24849e9ce7303a2c646f48baf3f5f010d9850bcc5a7/hypothesis-6.165.9-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:6564fb9bbe1760719070a6fd37335fbf43696fe2911c386ba55fb3cbaaafb82a", size = 1280667, upload-time = "2026-08-15T02:48:30.763Z" },+ { url = "https://files.pythonhosted.org/packages/36/08/9758ba15cc7dcb408d2699081ee0acbd96842f9c5778c84fd145d19953ba/hypothesis-6.165.9-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:126e7bf873892649cc5f697c3b3a6548f077a56ae3601b22fddef2ea1ba6f64d", size = 1324018, upload-time = "2026-08-15T02:48:18.804Z" },+ { url = "https://files.pythonhosted.org/packages/97/09/eaef6ee89a792ea82064cef66726023a5933f8704c0d3dbfc5a451829428/hypothesis-6.165.9-cp312-cp312-win_amd64.whl", hash = "sha256:c908edd69e4308c3cb194f0853aa7dfaa15e905242e81646b9f12881cd2683d3", size = 672207, upload-time = "2026-08-15T02:49:50.121Z" },+ { url = "https://files.pythonhosted.org/packages/d2/1f/99069da612aee3a51519f96f4370c4ba42055b57eae92926353c63e68617/hypothesis-6.165.9-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:11bcdfb14b52d41384cc14ecd3f6d6152735bc2f7af0b9f3fa8685d630a1e98d", size = 784591, upload-time = "2026-08-15T02:48:42.765Z" },+ { url = "https://files.pythonhosted.org/packages/94/10/cabf6a8782253e6b547ea8bdc6a48a9eeab415f1036c749b40ca59df32f8/hypothesis-6.165.9-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:8c3d7d54802641c2f8b49f478f153567770d299d639ee38fc0a0fa3d95ee07cb", size = 776216, upload-time = "2026-08-15T02:49:34.179Z" },+ { url = "https://files.pythonhosted.org/packages/15/5f/da966f98d69a8ee4b4fac509d92f7ab18cb4505b8191acc23bbb52f8bb17/hypothesis-6.165.9-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b8f5b63a8fba74d19a12ad4abaa95afb7466d66d1adc90ba3026e40b7924452c", size = 1106621, upload-time = "2026-08-15T02:49:08.529Z" },+ { url = "https://files.pythonhosted.org/packages/0a/5d/1fffd1031b4281e53fff02ad7adeda31db8218e91b82e7a98c7957541e76/hypothesis-6.165.9-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c01b77160328d8fce22bb5121fe1f5fee36b434a1220603500605b88ab854d7a", size = 1156613, upload-time = "2026-08-15T02:47:45.457Z" },+ { url = "https://files.pythonhosted.org/packages/b1/30/8e93a02016689c0b5cf0c35f8f9199b745601b3db9723296ca6d7366dcd5/hypothesis-6.165.9-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:88bde76aaab7f8abcd9412d135eaa49e8f7d89d068188fae2768a7803c80df89", size = 1280650, upload-time = "2026-08-15T02:49:44.026Z" },+ { url = "https://files.pythonhosted.org/packages/c0/48/2b0987389f141bd7bccc0785af4af86de9dfe9a6bc1b2b5dc612b083ae8a/hypothesis-6.165.9-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:0df8f8736d176d3a5092890b6fb9030f9c726d272b52af0e1ce887d62fc21699", size = 1323749, upload-time = "2026-08-15T02:48:12.519Z" },+ { url = "https://files.pythonhosted.org/packages/bb/a2/0a2380fdedfe0e16a09a09401b86bdc1eb049b3b6256fd837424d652c88e/hypothesis-6.165.9-cp313-cp313-win_amd64.whl", hash = "sha256:7fc9f46d88732fc74539ca7748aedc736c5e111431b06bd8ef2534440379aff2", size = 672195, upload-time = "2026-08-15T02:49:02.806Z" },+ { url = "https://files.pythonhosted.org/packages/e4/d0/95075d5ae3263e1c9143a3e89d9f19b864b1da303fb426964d9f7c08ce69/hypothesis-6.165.9-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:c3432a48e5895cba6b36c3e5abb915163d5169dc755685b4cd4720b710d77e3c", size = 784690, upload-time = "2026-08-15T02:49:23.922Z" },+ { url = "https://files.pythonhosted.org/packages/ed/d9/4a8b58237a36a839abefac22905ce9172b4c5195344708690413c6ef900d/hypothesis-6.165.9-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:70fe20939f63eb661227fd2b05193eba1e5ea6e547a2fbe158839f2acc519f4f", size = 776366, upload-time = "2026-08-15T02:48:27.261Z" },+ { url = "https://files.pythonhosted.org/packages/74/8d/91d1261f09a87299dc0ba89e12d8f9bbfd454e8922560e332b190a5c0a9f/hypothesis-6.165.9-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:20b177ef8aae2a2817a2223d103353d120118e686a152bcd01832a27ddc9388e", size = 1107128, upload-time = "2026-08-15T02:49:10.617Z" },+ { url = "https://files.pythonhosted.org/packages/2b/01/0198547c8b73d73ee5b885d8f73931c4dd3c62055dc93a80b5e18590efeb/hypothesis-6.165.9-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5d755b56098801a28a419172e57beb536b8c32929e7a37bb5ef12f2c00abd284", size = 1156757, upload-time = "2026-08-15T02:48:25.44Z" },+ { url = "https://files.pythonhosted.org/packages/21/3c/a3bde8c9630d361d85570ed5a6248fec0360a3874e26fdb04d62c8ebabf9/hypothesis-6.165.9-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:d49496271282bf424ddd728108f49b92e1d32066a3d0ba18990d5b01c40c3175", size = 1281013, upload-time = "2026-08-15T02:48:44.399Z" },+ { url = "https://files.pythonhosted.org/packages/f7/2d/017895202e6f2d75928b8e66bb8efe28c5c834771f6a3d118fc936d811d0/hypothesis-6.165.9-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:be54673f76785ff954b511745d4a7a18c7d7fe1d893cb81a8917cd28878fbd55", size = 1324110, upload-time = "2026-08-15T02:49:18.05Z" },+ { url = "https://files.pythonhosted.org/packages/40/f8/02f9be17e21b55513da624bac18a21c29011351d4e5d089ca5095555d1b4/hypothesis-6.165.9-cp314-cp314-pyemscripten_2026_0_wasm32.whl", hash = "sha256:af42ad4c24b69d4482e4813e148f458fc090911c0cf9055225c62fbd1f0b102e", size = 616257, upload-time = "2026-08-15T02:49:00.997Z" },+ { url = "https://files.pythonhosted.org/packages/e9/be/c94631e799bb4ee87c8c80de72f2eb44e93e7d0bff45b22b8801c78e5895/hypothesis-6.165.9-cp314-cp314-win_amd64.whl", hash = "sha256:9bc2d40dd2a71ba3971b902d31c8b1b0c40d3d465591fd71bb8ce6f541add530", size = 672006, upload-time = "2026-08-15T02:48:32.418Z" },+ { url = "https://files.pythonhosted.org/packages/43/15/d23548a12e5670513903b73a42fbd4f5ab7b1287bc6fd5a4a890c5746a62/hypothesis-6.165.9-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:d0fb16358e29f3090a9ab529e84df52923ddcd8505785c019bf48cdcd45e094a", size = 783152, upload-time = "2026-08-15T02:50:02.576Z" },+ { url = "https://files.pythonhosted.org/packages/0d/cd/37b5e5c042c557aa50db8c7f554b655b51c83e0b50306fb6ec8b05b03bfe/hypothesis-6.165.9-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:ed61ad45184997f97c3baf255f34aaaf4edff63f1a61aad87919034b4ab9f64e", size = 774782, upload-time = "2026-08-15T02:49:47.988Z" },+ { url = "https://files.pythonhosted.org/packages/56/ed/43b6db3681937ffa191148533d37cdd3c091f707ecb92281424078876d45/hypothesis-6.165.9-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f96f36c3d3677f22ff990d7aab4235576a8429e72dab5382c45bf489630972f5", size = 1105367, upload-time = "2026-08-15T02:49:42.041Z" },+ { url = "https://files.pythonhosted.org/packages/0d/dd/eaab3c678a23ab434c65b4ea5de5b302267ce68aa47b996524d2f34dcd13/hypothesis-6.165.9-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d69ab6500e47d7c21310360d1ee5cb6441660a10408e856d3738f07536732ed8", size = 1155499, upload-time = "2026-08-15T02:48:49.363Z" },+ { url = "https://files.pythonhosted.org/packages/10/fa/27049cbfb3e66cc1b68b32cbe137ac69a91cf09ac8520c4eb8ae9b3e5cbe/hypothesis-6.165.9-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:cc98fa897f48127fdf2e7d1b2428f839656cd52a6344095e501371d222b9c19d", size = 1279038, upload-time = "2026-08-15T02:49:26.001Z" },+ { url = "https://files.pythonhosted.org/packages/7f/d4/ab591a445dfeb7fa59b3c12de3eb9ff09901b4ede4025061b2fe1bc302af/hypothesis-6.165.9-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:a8b2d7415b1f7ceec6e689c992b919af41e339eafb35d6367a3abbb5b6db9753", size = 1322697, upload-time = "2026-08-15T02:48:39.467Z" },+ { url = "https://files.pythonhosted.org/packages/40/fd/4ee3b14d2d73a1d9442ac161b953dea2522fdac8ef7ca265fe5dd205982d/hypothesis-6.165.9-cp314-cp314t-win_amd64.whl", hash = "sha256:1000afaf05be59c88c13b458fd74d123e3361ca3d14fe127814a1231cb9cb7df", size = 672011, upload-time = "2026-08-15T02:48:07.583Z" },+ { url = "https://files.pythonhosted.org/packages/81/a1/20b63e3d17e05e82ef47b93ea3fa9a823297249ed7953e7e38f1b57a1e91/hypothesis-6.165.9-cp315-abi3.abi3t-macosx_10_12_x86_64.whl", hash = "sha256:5f6b4ae650480368e2477e8f875cdbfe8533a7b2e09f049fe206c48b20174622", size = 782729, upload-time = "2026-08-15T02:49:52.008Z" },+ { url = "https://files.pythonhosted.org/packages/7f/13/b43cea9578820a3e41cd70e8395068a0f07ef11c393ab09ff5aeb5d44d7f/hypothesis-6.165.9-cp315-abi3.abi3t-macosx_11_0_arm64.whl", hash = "sha256:1aaba05b943345d2a448b8c1c795bb191adf842b450bfc7207bdd1bf139a9b58", size = 774415, upload-time = "2026-08-15T02:47:57.162Z" },+ { url = "https://files.pythonhosted.org/packages/ea/08/222a8f2aeb8429b54cd53624add076ae5c9b39b0b170afd2435d0e81a8a9/hypothesis-6.165.9-cp315-abi3.abi3t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:306c1efc0f01f3d5c29c45996e055437971337aa24b6c6735b0514cfb5df60ce", size = 1104761, upload-time = "2026-08-15T02:47:46.841Z" },+ { url = "https://files.pythonhosted.org/packages/68/6b/09943bad5c1c5ea689a1be6376bcb7c68c34f3e3db5b792a1c0bb4edd79c/hypothesis-6.165.9-cp315-abi3.abi3t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:1551df11534d192616dc1b4539b6e526c4704d08b22d2ffb6ea3479ce6b87d94", size = 1133212, upload-time = "2026-08-15T02:47:52.814Z" },+ { url = "https://files.pythonhosted.org/packages/c2/ee/8fa90ca7a7d8b2362f3aaa0b2117405abf7b104fa01b52d5fecc4bb3c3fc/hypothesis-6.165.9-cp315-abi3.abi3t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:afd99a069df90c73c15d53e275d11a782fd5f17f5423002868ce9d2438ab567f", size = 1132156, upload-time = "2026-08-15T02:47:58.588Z" },+ { url = "https://files.pythonhosted.org/packages/5f/c3/6dae83eccd7260f5cf3468f145c663ce9b22579a9d9c80e9e7be49bec8b2/hypothesis-6.165.9-cp315-abi3.abi3t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d8c3e8c45ca5e2e4f519021836e5d86fb2229915c3f92fb50db7bb7364dfb2c1", size = 1155040, upload-time = "2026-08-15T02:49:16.085Z" },+ { url = "https://files.pythonhosted.org/packages/50/90/25eaabe2fecb3af3bcb9b4c3261ba9ffe0e092a3199232adce15399c040b/hypothesis-6.165.9-cp315-abi3.abi3t-manylinux_2_31_riscv64.whl", hash = "sha256:676a3bd04261b7bdd35374a3f4dabd82025fcead110a067a76328799f394a4d0", size = 1109751, upload-time = "2026-08-15T02:50:00.293Z" },+ { url = "https://files.pythonhosted.org/packages/1f/81/e8117e7ed3ff1ef7a36f023469efc925458b52c1243c539566a5308f5d77/hypothesis-6.165.9-cp315-abi3.abi3t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:a7f51c2297d116c2c025643762269eed0767589b6fc858d8de69d9f80cf33a23", size = 1144793, upload-time = "2026-08-15T02:49:38.317Z" },+ { url = "https://files.pythonhosted.org/packages/c6/6a/da2ea69d1fed780d86895f1ef7f2811240ed810487536ad22298629cc630/hypothesis-6.165.9-cp315-abi3.abi3t-musllinux_1_2_aarch64.whl", hash = "sha256:e500b97f899e375e47b407a48f3450cb1dd44bca757529c0490cb4533d878b30", size = 1278531, upload-time = "2026-08-15T02:48:37.581Z" },+ { url = "https://files.pythonhosted.org/packages/a2/6e/b3b21f15fe9debb9c448b03aaeb725ee007ba68518e55a6810878d7b9765/hypothesis-6.165.9-cp315-abi3.abi3t-musllinux_1_2_armv7l.whl", hash = "sha256:8401f55aba2d51f20b02c42e9e08b7eff2018639bdb1af16a4c279ba7e129c81", size = 1407073, upload-time = "2026-08-15T02:48:41.256Z" },+ { url = "https://files.pythonhosted.org/packages/da/fe/55f8e9c34f2b0a5ccf895a527a499f8562e9a14f014dd018b2535e8aa1c4/hypothesis-6.165.9-cp315-abi3.abi3t-musllinux_1_2_ppc64le.whl", hash = "sha256:34a4591db1dcf6aa544b4f884520ff0f1bc02ac3d11c76f78c73535b6b672272", size = 1261243, upload-time = "2026-08-15T02:49:58.076Z" },+ { url = "https://files.pythonhosted.org/packages/0b/37/49fb2374bff4e66d3d8264d083ddbb83269ec13903b441ebbc0d3c51502a/hypothesis-6.165.9-cp315-abi3.abi3t-musllinux_1_2_riscv64.whl", hash = "sha256:0bd589a011b3d0c4b98d78f467e5a5a6b96bf5be5d15b39ef47e65c225e8d790", size = 1279060, upload-time = "2026-08-15T02:48:54.843Z" },+ { url = "https://files.pythonhosted.org/packages/ec/b7/c826e67f5d553d20644ac3da5d25de7ff9d50a1f41467981b00da7b2ab9b/hypothesis-6.165.9-cp315-abi3.abi3t-musllinux_1_2_x86_64.whl", hash = "sha256:84a93a152ffaf92fa6b3f94d366f4c866e26f3f1b643fcc6da4ee2ddfa038aa9", size = 1322247, upload-time = "2026-08-15T02:47:54.398Z" },+ { url = "https://files.pythonhosted.org/packages/24/ec/e70e440805820cfd662d04d05aa34b5d29e46d5a49ee871d6937721b2c17/hypothesis-6.165.9-cp315-abi3.abi3t-win32.whl", hash = "sha256:4a2081699e0b16b35c7b34cf0ff01997e8fc6038b520b9f29c106764158c411a", size = 665906, upload-time = "2026-08-15T02:49:53.901Z" },+ { url = "https://files.pythonhosted.org/packages/23/79/968b149b572e1d4284b04280cc181449090c8bd03e55b24d57b64443978e/hypothesis-6.165.9-cp315-abi3.abi3t-win_amd64.whl", hash = "sha256:e1d9492193b52087df4d264edd77618869d447b5c7f6a022c7bb11b88d577cb8", size = 671817, upload-time = "2026-08-15T02:49:46.053Z" },+ { url = "https://files.pythonhosted.org/packages/c2/72/0f951d4f0e07720583e4773638cd556d2b6585605f8a3df54d2fecf2e2ff/hypothesis-6.165.9-cp315-abi3.abi3t-win_arm64.whl", hash = "sha256:f67414675f2263948e5a5977b636b5cb1cc5f33b82a9be34f8a6b9a8293d4707", size = 669795, upload-time = "2026-08-15T02:48:03.176Z" },+ { url = "https://files.pythonhosted.org/packages/43/7d/1b6b48629032b22960a5b010a791467670c08d9413a3c6c15a72f7c1da09/hypothesis-6.165.9-pp311-pypy311_pp73-macosx_10_12_x86_64.whl", hash = "sha256:f6caa4194f37a02232ecb964600d17facc23a6aa5b0295008989f31e329eb837", size = 784507, upload-time = "2026-08-15T02:48:45.961Z" },+ { url = "https://files.pythonhosted.org/packages/95/a6/c29939a55cde73886cee7a3975148e132a9ed6881d39b7c9b8c19e3a854b/hypothesis-6.165.9-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:0a4a32dc8e8728417aa13055903dbf742dd0b5cedda7c2031b4b2a0bcc1d463d", size = 780353, upload-time = "2026-08-15T02:49:14.307Z" },+ { url = "https://files.pythonhosted.org/packages/8c/f9/ac8cd423b1c7506dae01c4a9295c434b53525401a182a116aecaa27332cf/hypothesis-6.165.9-pp311-pypy311_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2f255b389364f71e6e9305ba99bf13733354cb0dc4a97d07db8aef76278a345e", size = 1109224, upload-time = "2026-08-15T02:48:15.557Z" },+ { url = "https://files.pythonhosted.org/packages/f2/26/64c4dfa2836ec421e35a167a3fba92018950a8c7551a268a5fc799329455/hypothesis-6.165.9-pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0c7f0507d8224e5b44c2651d7ef8df15ac6d38eb56c98e7f451c3090804ce109", size = 1158929, upload-time = "2026-08-15T02:49:21.922Z" },+ { url = "https://files.pythonhosted.org/packages/88/8e/fccbea8a0a03532762fdd56d3809fd8398f46af1ec9018f694d4ec5c22cf/hypothesis-6.165.9-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:fd9e242fabb05715ec2d3b9b96dcddf683fc3891a04b6685b79e400e35ccbe18", size = 675885, upload-time = "2026-08-15T02:47:50.986Z" },+]+[[package]]name = "idna"version = "3.18"@@ -3223,6 +3307,7 @@ version = "0.45.0"source = { editable = "." }dependencies = [{ name = "aiofiles" },+ { name = "annotated-types" },{ name = "datamodel-code-generator" },{ name = "filetype" },{ name = "httpx" },@@ -3272,6 +3357,7 @@ bedrock = [dev = [{ name = "boto3-stubs" },{ name = "cryptography" },+ { name = "hypothesis" },{ name = "jsonschema" },{ name = "moto", extra = ["dynamodb", "s3"] },{ name = "mypy" },@@ -3347,6 +3433,7 @@ requires-dist = [{ name = "aioboto3", marker = "extra == 'bedrock'", specifier = ">=13.4.0" },{ name = "aioboto3", marker = "extra == 's3'", specifier = ">=13.4.0" },{ name = "aiofiles", specifier = ">=23.2.1" },+ { name = "annotated-types", specifier = ">=0.7.0" },{ name = "anthropic", marker = "extra == 'anthropic'", specifier = ">=0.78.0" },{ name = "boto3", marker = "extra == 'bedrock'", specifier = ">=1.34.131" },{ name = "boto3", marker = "extra == 'dynamodb'", specifier = ">=1.34.131" },@@ -3365,6 +3452,7 @@ requires-dist = [{ name = "google-genai", marker = "extra == 'google-genai'" },{ name = "httpx", specifier = ">=0.23.0,<1.0.0" },{ name = "huggingface-hub", marker = "extra == 'huggingface'", specifier = ">=1.5.0,<2.0.0" },+ { name = "hypothesis", marker = "extra == 'dev'", specifier = ">=6.140.3" },{ name = "idna", specifier = ">=3.15" },{ name = "instructor", specifier = ">=1.13" },{ name = "instructor", extras = ["google-genai"], marker = "extra == 'google-genai'" },@@ -4903,6 +4991,15 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/e9/44/75a9c9421471a6c4805dbf2356f7c181a29c1879239abab1ea2cc8f38b40/sniffio-1.3.1-py3-none-any.whl", hash = "sha256:2f6da418d1f1e0fddd844478f41680e794e6051915791a034ff65e5f100525a2", size = 10235, upload-time = "2024-02-25T23:20:01.196Z" },]+[[package]]+name = "sortedcontainers"+version = "2.4.0"+source = { registry = "https://pypi.org/simple" }+sdist = { url = "https://files.pythonhosted.org/packages/e8/c4/ba2f8066cceb6f23394729afe52f3bf7adec04bf9ed2c820b39e19299111/sortedcontainers-2.4.0.tar.gz", hash = "sha256:25caa5a06cc30b6b83d11423433f65d1f9d76c4c6a0c90e3379eaa43b9bfdb88", size = 30594, upload-time = "2021-05-16T22:03:42.897Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl", hash = "sha256:a163dcaede0f1c021485e957a39245190e74249897e2ae4b2aa38595db237ee0", size = 29575, upload-time = "2021-05-16T22:03:41.177Z" },+]+[[package]]name = "soupsieve"version = "2.9.2"diff --git a/wip/bugs/enum-completeness-validators-over-user-keys.md b/wip/bugs/enum-completeness-validators-over-user-keys.mdnew file mode 100644index 000000000..14bc38744--- /dev/null+++ b/wip/bugs/enum-completeness-validators-over-user-keys.md@@ -0,0 +1,66 @@+# Completeness validators over user-supplied keys turn an enum addition into a boot failure++**Status:** confirmed defect, not yet fixed. Found 2026-08-15 while re-arguing Migrator ruling R8 (`wip/migrator-3/README.md` at the workspace root); recorded there as a deferred finding the migrator can *name* but not *fix*, because the fix is a model-design change in this repo, not a gate change. Filed here so it rides the migrator worktree and branch and gets its own pull request when someone picks it up. It is not on the Migrator v3 critical path.++## Problem++Two configuration fields are open mappings keyed by the **user's** names, and each carries a validator that demands **every member of an enum** as a key of every entry:++| Field | Type | Keyed by | Validator | Enum |+|---|---|---|---|---|+| `LLMConfig.effort_to_budget_maps` (`pipelex/cogt/config_cogt.py`) | `dict[str, EffortToBudgetMap]` = `dict[str, dict[str, int]]` | reasoning family (`anthropic`, `gemini`, …) | `validate_effort_to_budget_mapping` — raises `ConfigValidationError("Missing reasoning effort levels …")` | `ReasoningEffort` (`none minimal low medium high xhigh max`) |+| `ImgGenConfig.quality_to_steps_maps` (same file) | `dict[str, QualityToStepsMap]` = `dict[str, dict[str, int]]` | image model (`flux`, `sdxl_lightning`, `qwen_image`, …) | `validate_quality_mapping` — raises `ConfigValidationError("Missing quality levels …")` | `Quality` (`low medium high`) |++Both validators run on the **merged** configuration (`PipelexConfig.model_validate` after every tier is deep-merged), so the rule applies to whatever the user ends up with, packaged defaults included.++**The failure.** A user has a hand-authored override — say `~/.pipelex/pipelex_override.toml` or a project `.pipelex/pipelex.toml` — that adds a map for a family or model the packaged defaults do not know:++```toml+[cogt.llm_config.effort_to_budget_maps.my_finetune]+low = 512+medium = 2048+high = 8192+```++That file is valid today only if it names all seven efforts. It is complete on the day it is written. Then a release adds a member to `ReasoningEffort` — which happened on 2026-04-16 when `XHIGH` arrived (`cdabd29a8`), with `pipelex/pipelex.toml` updated in the same commit — and the user's next boot fails with `Missing reasoning effort levels in mapping: {'xhigh'} for target 'my_finetune'`. Nothing in the packaged defaults can rescue them: `deep_update` (`pipelex/tools/misc/json_utils.py`) merges recursively into keys the target *already has*, and the packaged file has no `my_finetune` key to merge beneath.++Users who override a **known** family (`anthropic`) with a sparse map are fine — the deep merge recurses into the packaged `anthropic` dict and the packaged `xhigh` survives. The seeded `~/.pipelex/pipelex.toml` is fine — the kit template (`pipelex/kit/configs/pipelex.toml`) does not carry either map. So the population hit is exactly the users who did the customization the open mapping invites: **their own model, their own family.**++The packaged defaults already pay for the rule in-repo. `pipelex/pipelex.toml:194` reads `none = 0 # Required by validator; unreachable at runtime (level map gates NONE as disabled before budget lookup)` — a dead entry that exists to satisfy the completeness check.++## Why the migration gate cannot catch it, and why that matters++The Migrator's coverage gate (`pipelex/migration/coverage.py`, Phase 1, on `dev` since `052b9df06`) diffs a fingerprint of the schema. It records `effort_to_budget_maps.*.*` as a bare `int` under two wildcard segments — the keys are `str`, so no enum appears on that path. `ReasoningEffort`'s only footprint in the config tree is elsewhere, where an added member is correctly classified as **additive → regenerate the golden, no bump, no entry**. So the gate is green, the changelog says nothing about migration, and the customized user's file breaks. This is the same failure family as value-domain narrowing (R8) — a change the schema calls additive that a validator makes breaking — and it is the case no fingerprint of any shape can see, because the narrowing lives in Python, not in the type.++The asymmetry with the *removal* direction is what makes the "missing" branch the wrong half of the validator. If a member were **removed** from `ReasoningEffort`, the "invalid" branch would fire on old keys — but the migrator can repair that with one wildcard op in the ledger (`delete_key` at `["cogt","llm_config","effort_to_budget_maps","*"]`, key `xhigh`), because removing a key from a user's file is structural. **Adding** a member cannot be repaired by any op in the vocabulary — the migrator will not materialize a value into a user's file, and there is no right value to materialize (a budget for the user's own family is the user's to choose). So the addition direction is exactly the one that must not break the boot.++## Proposed fix++**Drop the "missing" branch of both validators; keep the "invalid" branch.** The invalid branch is a typo guard and stays. Completeness stops being a boot-time invariant and becomes what it already is at the point of use:++- `get_reasoning_budget` already raises `ConfigValidationError("No budget found for reasoning effort '…' and reasoning family '…'")` on a miss.+- `get_num_inference_steps` already raises `ConfigValidationError("No number of inference steps found for quality '…' and model '…'")` on a miss.++That moves the failure from *every boot, for everyone with a custom key* to *the one call that asks for the missing level, for the one user who omitted it* — later, narrower, and with a message that names the exact key. It also retires the dead `none = 0` entry and its apologetic comment in the packaged file.++**Optionally, on top:** a reserved fallback key so a sparse per-family map is a first-class shape rather than a latent runtime error — `effort_to_budget_maps.default` consulted when the family's map lacks the level. That is a small feature, not part of the bug fix, and it should be its own decision (it reserves a name in the user's key space, which the ledger contract would want recorded).++Not proposed: retyping the inner map as `dict[ReasoningEffort, int]`. It reads better but changes nothing here — the migration fingerprint treats any mapping as an open node and records only the value schema, so the enum would still be invisible on that path, and the completeness rule would still be a validator.++## Test shape++`tests/unit/pipelex/cogt/llm/test_llm_config_reasoning.py` already pins the current behaviour and flips:++- `test_validator_missing_effort_level_raises` → becomes `test_validator_accepts_partial_map`: a map for a custom family carrying a subset of the levels validates.+- `test_validator_invalid_effort_level_raises` and `test_validator_missing_and_invalid_effort_levels_raises` → the invalid half stays red; the combined case now reports only the invalid keys.+- New: `get_reasoning_budget` on a family whose map lacks the requested level raises at lookup with the level and family in the message (this is existing behaviour, now the only line of defence, so it deserves its own assertion).+- Same three shapes for `ImgGenConfig` / `Quality` — there is no test file for `validate_quality_mapping` today; add one beside the LLM one.+- One boot-shaped test: layered TOMLs (packaged defaults + a temp override carrying `[cogt.llm_config.effort_to_budget_maps.my_family]` with a subset) → `PipelexConfig` validates. This is the user's actual situation, and it is the case the unit tests above do not cover because they build the model directly.++## Where it touches the Migrator++Nothing in `pipelex/migration/` changes. What the migrator does with this finding is *declare* it: R8 adds one sentence to the fingerprint section of `docs/migration-ledger.md` saying that domain narrowing expressed in a validator is not visible from the schema and stays the author's responsibility, and Phase 3's `config-docs` drift-contract `review` list gains the validator sites so a reviewer sees them on every config-docs run. This doc is the fix on the model side, and it can land in either order.++## Note on the reshape++The configuration reshape (`wip/migrator-3/reshape.md`) moves both fields — `cogt.llm_config.effort_to_budget_maps` → `inference.llm.effort_to_budget_maps`, `cogt.img_gen_config.quality_to_steps_maps` → `inference.img_gen.quality_to_steps_maps` — and does not touch the validators. Whoever fixes this should land it as its own small pull request against `dev`, and rebase across the reshape when it merges; the two changes do not conflict in substance, only in path names.diff --git a/wip/pr-1112-review-notes.md b/wip/pr-1112-review-notes.mdnew file mode 100644index 000000000..0193d3370--- /dev/null+++ b/wip/pr-1112-review-notes.md@@ -0,0 +1,171 @@+# PR #1112 review triage — Migrator Phase 2++PR [#1112](https://github.com/Pipelex/pipelex/pull/1112) → `dev`, branch `feature/Migrator-2`. Three bots left 33 review threads (greptile 1, codex 5, cubic 27). Every one was read in full and verified against the code by hand; the eight that a first pass had left unverified were verified in the round that executed this triage.++**State: arbitrated and executed.** The clusters marked *Fixed* below are fixed on the branch, TDD-first, with `make agent-check`, `make cl`, `make cmig` and the suite green. The clusters marked *Deferred* are recorded here and their threads left open. The ones marked *Won't fix* were answered on their thread and resolved. The arbitration rule was Louis': fix what needs fixing, defer in doubt, no speculative machinery.++## Disposition++| # | Where | Issue | Reporter(s) | Disposition |+|---|---|---|---|---|+| A | `runner.py`, `backup.py` | `FixTransactionError` and pruning failures escaped `migrate_file`, aborting every sibling | greptile #1, codex #4, cubic #16, #22 | **Fixed** |+| B | `backup.py` | Pruning deleted any `<file>.bak.*`, including a user's own hand-made copy | cubic #28 | **Fixed** |+| C | `backup.py` | Staged backup copy leaked when the replace failed | cubic #21 | **Fixed** |+| D | `narrowing.py` | `Literal['a']` → `Literal[1]` suppressed the lost spelling entirely | cubic #15 | **Fixed** |+| E | `transform_check.py` | An unreadable golden crashed the gate with a raw traceback | cubic #29 | **Fixed** |+| E′ | `surfaces.py:117` | An unreadable packaged kit template crashes the gate | cubic #27 | **Won't fix** — a packaged file missing is a broken install, and the traceback names the path |+| F | `narrowing.py` | Float `multiple_of` modulo reported a widening as a narrowing | codex #5, cubic #24 | **Fixed** |+| G | `ledger.py` | `declared_removed_paths = [""]` satisfies the guard vacuously | cubic #17 | **Won't fix** — not exploitable: `""` is never a reserved prefix, so every op of such an entry is refused as live material and the gate is red anyway |+| H | `test_replay_properties.py` | The "older shape" fixture kept the new keys, so renames conflicted instead of applying | cubic #18 | **Fixed** |+| I | `test_engine.py`, `test_telemetry_back_entry.py` | Tests passed without exercising the guarantee they name | cubic #31, #33 | **Fixed** |+| J | `test_fix_applier_rename_dom_consistency.py`, `applier.py` | Docstring contradicted its own pinned assertion | cubic #30 | **Fixed** |+| K | `test_runner.py` | `_write_ledger` duplicated the `write_ledger_file` fixture | cubic #32 | **Fixed** |+| L | `engine.py` | An op-free `unsafe` entry is structurally unreportable | codex #2 | **Deferred — needs a ruling** |+| M | `backup.py`, `runner.py` | Two runs in the same UTC second can lose the only prior backup | codex #6, cubic #12 | **Deferred** |+| N | `runner.py` | A symlinked config file is replaced by a regular file | codex #3, cubic #8 | **Deferred — policy call** |+| O | `surfaces.py` | `telemetry.project.toml` is not a neutrality witness | cubic #11 | **Deferred** |+| P | `fingerprint.py` | Union-member constraints merged with the field's own | cubic #10 | **Deferred** |+| Q | `narrowing.py` | `gt=0` vs `ge=1` on ints; `list[int]` → `list[int \| str]` | cubic #25, #26 | **Deferred** |+| R | `coverage.py:315` | Any remap over a free-form domain suppresses the narrowing | cubic #7 | **Won't fix — as designed** |+| S1 | `documents.py:40` | A quoted key containing `.` flattens into ambiguous segments | cubic #9 | **Deferred** |+| S2 | `walk.py:208` / `suggested_fix.py` | A wildcard-source `move_key` conflicts on any file with two matched entries | cubic #13 | **Fixed** — refused when the ledger is parsed |+| S3 | `ledger_check.py:255` | A pre-history entry cannot chain through a renamed declared path | cubic #23 | **Deferred**, documented as a constraint |+| S4 | `ledger_check.py:518` | Reserved spellings are keyed at the remap's literal path, not the walked one | cubic #14 | **Deferred** |+| S5 | `transform_check.py:192` | A destination beneath an open mapping, absent from the reference document, was a false red | cubic #19 | **Fixed** |+| S6 | `surfaces.py:247` | A project-level `pipelex_service.toml` is claimed by the service surface | cubic #20 | **Won't fix — as designed** |++## What was fixed, and how++- **A.** `_write_migrated_file` is now the per-file boundary in fact: it catches `FixTransactionError` and decides by re-reading the target whether the write landed (the exception cannot say; the file can) — blocked as unwritable when it did not, written with a warning when it did; and `prune_backups_except` sits inside its own `try`, so a backup that will not prune is a warning on a migrated file rather than an escape that aborts the walk. Tests: a transaction error after the write, one before it, a pruning failure — each with the sibling still migrated.+- **B.** `existing_backups_of` matches the whole name we write, stamp included (`\d{8}T\d{6}Z`). Test: `example.toml.bak.notes` survives while a stamped backup is pruned.+- **C.** `write_backup` unlinks the staged copy when the replace fails. Test: a failed backup leaves the directory holding only the source file.+- **D.** `lost_enumerated_spellings` exempts only a destination that admits `str`, which is what its docstring always said. Test: `Literal['a']` → `Literal[1]` loses `'a'`. The contract's fingerprint section now says a non-string literal's domain is invisible to the projection.+- **E.** The two golden reads in `transform_check.py` translate `OSError`/`ValueError` into `MigrationGoldenError` naming the file, mirroring `read_fingerprint_golden`. Test: a non-UTF-8 `[email protected]` is refused by name.+- **F.** `multiple_of` divisibility is decided over `Fraction(str(x))`. Test: `0.3` → `0.1` is a widening.+- **H.** The older-shape generator drops `title` when it puts `heading` back and `section.enabled` when it puts `chatty` back. A second vacuity meta-test finds a document on which the rename *and* the move apply with nothing blocked, which is what makes the idempotence and prefix-coherence properties claims about a rename.+- **I.** The report-never-echoes test asserts the unsafe entry was reported; the telemetry test asserts the whole migrated document.+- **J.** Both docstrings now say what the pinned assertion measures: the old key stays in the raw dict, the new one is never added outside the table branch.+- **K.** `test_runner.py` uses the conftest fixture.+- **S2.** `MoveKeyOp` refuses a wildcard in its `table_path`, symmetric with the existing refusal on `new_table_path`, and the contract's wildcard section says why: with the destination pinned to one fixed key, "move each entry's key" is many-to-one and conflicts on any file where two matched entries carry the key. `test_a_wildcard_source_is_accepted` became two tests — the move is refused, an in-place kind keeps its wildcard.+- **S5.** The transform comparator's first claim reads the fingerprint *with its wildcards*: `deck.claude.new_name` is recorded as `deck.*.new_name`. Test: a rename beneath an open mapping whose destination the new reference document omits under that entry is tolerated.+- **S3, documented.** The pre-history containment rule reads each source at its literal, pre-entry path; the contract now says so and tells the author to address the material first, then rename it.++## The deferrals, and why++### L. An op-free `unsafe` entry can never reach the user — needs a ruling++`MigrationEntry.ops` is documented as legitimately empty when `safety = "unsafe"` (`ledger.py`), and `check_an_op_free_entry_is_unsafe` enforces exactly that shape: it is *the* form for "a change only a human can make". But `_rehearse_unsafe_entry` (`engine.py`) decides whether to report by rehearsing the entry's operations — and an entry with no operations rehearses to nothing, so it returns early and **the entry is never reported to anyone, ever**.++That is precisely the remedy R8 names for a tightened numeric bound, and the PR body says so: *"or `unsafe`, which is the only remedy a tightened numeric bound has, since no structural operation can repair a value."* The accounting accepts such an entry as sufficient; the engine then guarantees the user never hears about it.++**Not reachable today** — no ledger in the tree carries an op-free entry, and the reshape entry is `safe`. It fires the first time someone writes the shape the contract invites.++The three candidate answers are all design decisions:++1. **Report it always.** Restores the guidance, and reintroduces exactly what the rehearsal rule exists to prevent — a warning at every boot, forever, for every user whose file is already fine.+2. **Give the entry a predicate.** The honest one: the entry declares the paths whose domain narrowed, and the engine reports it when the document carries one of them. Costs a new ledger field, a contract change, and a `check-ledger` rule — but it is the only option that reports to the users who need it and stays silent for the rest.+3. **Refuse the shape.** Make an op-free entry illegal and force every narrowing to be expressible. Cheapest, and it contradicts the contract's own sentence plus R8's stated remedy.++A fourth road is tempting and wrong: have the engine validate the document against the model. The engine is deliberately model-free and filesystem-free — that is what lets the gates and a user's migration run the same code — and threading a model into it would cost that.++**Recommendation: (2), in Phase 3**, where `pipelex migrate` and boot tolerance land and the reporting surface exists to carry it. **Louis' call.**++### M. Same-second backup collision++`write_backup` clobbers an existing backup of the same stamp, and `runner.py` unlinks that path on a failed commit as though this run had created it. Two migrations of the same file inside one UTC second, the second having something to apply and failing to commit, lose the only copy of the original. Narrow, but it is the one scenario backups exist for. The fix is small (report whether the destination pre-existed; unlink only what this run created) but it touches the backup naming the contract states, so it goes with N as one backup-semantics pass.++### N. Symlinked configuration files++Confirmed mechanically: `read_file_snapshot` reads *through* the link, `assert_snapshot_unchanged` compares device/inode of the resolved file, and `replacement_snapshot.path.replace(snapshot.path)` then replaces **the link itself** with a regular file, leaving the real target untouched. A user with a dotfile-managed `~/.pipelex/pipelex.toml → ~/dotfiles/pipelex.toml` silently loses the linkage.++Deferred because the answer is a policy choice, not a bug fix — refuse a symlinked target, resolve and transact the target, or keep today's behaviour and document it — and because `file_transaction.py` is **shared with the `.mthds` fix loop**, so whatever is decided changes that path too.++### O. `telemetry.project.toml` is not a witness++`pipelex/kit/configs/telemetry.project.toml` is a different, sparser shape than the global `telemetry.toml` the telemetry surface uses as its kit witness. Measured: replaying the real `telemetry-config` ledger over it returns `changed=False, steps=0, blocked=0` and the very same string, so this is a latent coverage gap, not a live failure — and the property suite already samples sparse documents. Letting a surface carry several kit witnesses is small; it waits for a need.++### P / Q. Refinements to the R8 relation++- **P (#10)** — for a genuine union, taking the widest bound per kind is correct: a union accepts a value if any member does. The narrow hole is mixing the two sources — a binding top-level `Field(le=6)` merged with a union member's `le=100` yields `le=100`. Exotic; recording it means another golden format change.+- **Q (#25)** `gt=0` and `ge=1` accept the same integers, but `_strictest_bound` calls the swap a tightening — a false breaking verdict on an equivalent schema; the fix needs the value type threaded into `_describe_tightenings`.+- **Q (#26)** `_union_members` treats `list[int | str]` as one opaque member, so `list[int]` → `list[int | str]` reads as a narrowing.++Both Q items are crying-wolf failures. Worth doing as one deliberate pass over the relation, with fixtures per shape, rather than folded into a review round.++### S1. A quoted key containing `.` beneath an open mapping++`_flatten_into` joins segments with `.`, so a user's key `"gpt-4.1"` beneath an open mapping flattens to synthetic segments. Consequences today: none on any checked-in document (no surface document carries such a key), and the head-link model validation catches a misspelled destination regardless. The honest fix is one escaping scheme applied everywhere migration paths are split and joined — walk, applier, fingerprint, documents — which is a design pass, not a review fix.++### S3. Pre-history containment is literal++A pre-history entry that renames a declared table and then addresses a child under the new name is refused, because the containment rule reads the child's source literally against the declaration. Every such chain has an unchained equivalent — address the material first, then rename it — so the contract now states the constraint rather than the code growing a second walk.++### S4. Reserved spellings are keyed at the remap's literal path++`derive_reserved_registry` records a remapped-away spelling under the remap operation's own `table_path.key`; the reuse check at `ledger_check.py` looks it up at the current fingerprint path. An entry that remaps `section.mode` and then renames `section` → `area` in that order stores `section.mode=a`, and a later version bringing `a` back at `area.mode` is not caught. Reordering the ops (rename first) makes it work today. The fix keys reservations by the walk's final path; it needs the walk inside `derive_reserved_registry`, which is a moderate change for an exotic ordering.++## Won't fix, and why++- **E′ (#27)** — `surfaces.py` reads a *packaged* file. Missing or unreadable means a broken install; the traceback names the path; a gate-shaped refusal adds nothing an author can act on.+- **G (#17)** — `[""]` is never a reserved prefix, so every operation of such an entry is refused as acting on live material and the gate is red. Tightening the field's shape would be hygiene, not protection.+- **R (#7)** — by design, and it is what R8 ruled: for a free-form domain narrowed into an enumerated one, no remap can be *proved* complete because the old domain is infinite, so the remap *is* the author's account of the spellings that existed. Requiring `unsafe` there would forbid the exact case the coverage docstring names as the fit for a remap.+- **S6 (#20)** — the walk claims by filename, deliberately: a file named `pipelex_service.toml` is a service configuration wherever it sits, and migrating a stray project-level one is harmless and backed up. Restricting a surface by tier is machinery for a case nobody has.++## Round 2 — `/review` pass of 2026-08-16++Run on the branch after round 1, against `dev` at `052b9df06`, with the S5 charter (`wip/migrator-3/sequencing.md`) as the plan. Sources: the checklist pass, four specialist subagents (testing, maintainability, security/data-safety, a Claude adversarial pass) and a Codex adversarial pass. Every finding below was verified by reading or executing the code before it was acted on; the round-1 deferrals were excluded from the brief and are not repeated here.++**Plan completion.** Every S5 "Done when" item that this PR owns is present: `make cl` exists and is in `check` and `agent-check` (both gates run in under a second, which answers Checkpoint B's timing question); the R8 rule is in `_check_head_link` and in accounting, the whitelist projection with `pattern` excluded is in the fingerprint, the goldens were regenerated once, the contract's coverage row and fingerprint section moved with it, including the sentence naming validator-expressed narrowing as outside the gate; the telemetry back-entry and the `pre_history` verification are in the same PR. The join in `_reshape/` is post-merge work and is not part of this PR. The one commit outside the S5 scope, `cc1bc1b92` (`wip/bugs/enum-completeness-validators-over-user-keys.md`), was already recorded in the charter as riding along by choice.++### Fixed in this round++| Where | Issue | Reporter | What was done |+|---|---|---|---|+| `applier.py` | **Renaming or moving a table whose descendant is written in several chunks (`[a.b]` … `[a.c]` … `[a.b.d]`) re-homed only the first chunk and reported `applied`** — the user's file came out split between the old and the new name, and the next run conflicted. Reproduced on tomlkit 0.14; the parent-level proxy case was already handled, the nested case was not. | Claude adversarial | `_first_split_descendant` guards both operations: `CONFLICT` naming the split table, nothing written. A table interleaved at its own level still moves whole. Contract "Limits" bullet, changelog, and a parametrized red/green test. |+| `coverage.py` | **A remap on a path excused every narrowing of that path, a tightened bound included** — `Annotated[int, ge=1] \| Literal["auto"]` → `Annotated[int, ge=8] \| Literal[...]` with a `safe` remap of `auto` was green on all gates; a file saying `4` fails at boot. Real shape: `max_concurrency`, `seed`. | Claude adversarial | For a remapped origin only the bound half (`describe_tightenings`) is asked; a type change is still the remap's to answer for. Contract row sentence; red/green fixture. |+| `coverage.py` | **An over-deletion beneath a renamed table was invisible** — `rename a→b; delete_key [b] x` with `x` live at v2 was green (over-deletion compared by current spelling, and the convergence witness has no value for an optional `None` child). | Claude adversarial | Over-deletion is now decided by origin: each removed origin is carried through its nearest surviving ancestor's rename and that landing is looked up in the new schema. Red/green fixtures. |+| `runner.py` | **A `FixTransactionError` whose file did not carry the new text deleted the only backup** — with a single-file update that error only comes from the post-commit cleanup path, so `_carries()` false means a concurrent edit or an unreadable target: exactly the state a backup exists for. Also, the cleanup `unlink` calls were unguarded, so a permission error escaped the per-file boundary. | security specialist + Codex (multi-source) | The backup is kept whenever the transaction cannot vouch for the state, and the report names it (`backup_path` on a blocked plan). `_discard_backup` swallows an `OSError` into a warning. Runner docstring and contract "Backups" paragraph aligned; the existing test's assertion flipped, plus tests for the concurrent-edit, commit-`OSError`, unguarded-unlink, non-UTF-8, unreadable and removed-before-read branches. |+| `backup.py` | `existing_backups_of` globbed on the raw file name, so a name carrying a glob metacharacter could match — and prune — a sibling's only backup. | security specialist | Prefix match over `iterdir()`; test with `example_?.toml` beside `example_a.toml`. |+| `ledger_check.py` | `RESERVED_VALUE_REUSED` was produced by no test (its path twin was). | testing specialist | Red and green tests added. |+| `fingerprint.py` | The non-numeric-bound drop had no test. | testing specialist | `Decimal` bound fixture. |+| `docs/migration-ledger.md` | The contract said no surface carries an array of tables; `TelemetryConfig.otlp` is one, and the fingerprint records it as a terminal `list[table]` — a change inside an exporter entry is invisible to the coverage gate. | Codex + Claude adversarial | Both sentences corrected: the blind spot is named beside validator-expressed narrowing, with `otlp` as the live instance. The fingerprint format itself is deferred (below). |+| `reserved.py`, `narrowing.py`, `fingerprint.py` | A third hand-rolled `"."` path join; the union separator defined in two modules. | maintainability | `op_source_path` and one `UNION_SEPARATOR`. |+| `coverage.py`, `reserved.py`, `ledger_check.py`, `transform_check.py` | Docstrings said `check-ledger` reads the `before` document (the transform check does); a hardcoded count; a `MIGRATED_DOCUMENT_REJECTED` message that named the wrong starting document for a pre-history entry. | maintainability | Corrected; `_starting_document_label` used. |+| `pipelex/kit/agent_rules/commands.md`, `codex_commands.md` | The `agent-check` target list an agent without `make` follows omitted `check-ledger`. | checklist + maintainability | Sources fixed and `CLAUDE.md` / `AGENTS.md` regenerated with `make rules`. |+| `docs/migration-ledger.md` | "reads checked-in files and nothing else" — the convergence witness for a model-defaults surface is rendered from the live model (rendered, not fingerprinted). | maintainability | Sentence now says what the code does. |++### Deferred from this round++- **`key = "*"` is a literal key to the applier, and a `remap_value` over a container never fires** (Claude adversarial, verified). The walk spells `levels.*` and `check-ledger` accepts `*` at an open node, but `_expand_table_paths` expands only `table_path`, so a remap with `key = "*"` skips with "key '*' not found", and a remap over a `list[enum]` or `dict` value skips as "not a string" — while `_gather_enum_members` descends into containers, so coverage *demands* exactly those remaps for `pipelex.log_config.package_log_levels`, `preferred_agent_targets`, `img_gen_param_defaults.size`. Two coherent answers: teach the applier to expand a `*` key over an open node's entries (the contract's "each of these" reading, and the natural remedy for `dict[str, Enum]`), and have coverage demand `unsafe` for a lost member the remap cannot reach (`list[enum]`); or stop `_gather_enum_members` at the container boundary, mirroring `_collect_constraints` (the maintainability pass's reading), and regenerate the goldens. Either is a design pass with its own fixtures, and it changes the golden format, so it belongs with the R8 relation refinements (P/Q) — before S7 freezes the format.+- **An `unsafe` entry goes silent after a later `safe` entry renames its target** (Claude adversarial, reproduced): `unsafe@2` remaps `a.mode`, `safe@3` renames `a→b`; run 1 reports both, writes `[b] mode = "legacy"`; run 2 is clean while boot fails on `b.mode`. Either forbid it statically in `check-ledger` (a later entry may not rename a subtree an earlier unsafe entry addresses) or rehearse unsafe entries with their paths traced forward. Goes with L (the op-free unsafe entry) as one "what does `unsafe` promise" pass.+- **`list[Model]` is a terminal `list[table]` in the fingerprint** — a renamed field inside `OtlpExporterConfig` changes nothing and demands nothing. Documented now; recording the item model's fields under a synthetic segment is a format change and waits for the same pre-S7 window as P/Q and the `*`-key question above.+- **`up-migration-schemas` overwrites the head golden unconditionally and rides in `make up`** — a habitual `make up` after deleting a field rewrites `fingerprint@N` and erases the removal the coverage gate would have caught. Pre-existing (Phase 1), but this PR is what makes the chain a proof. Candidate: refuse the snapshot when the stored head shows removals or narrowings against the live model, and take `umig` out of `up`.+- **Replay continues after a blocked entry** (Codex): a rehearsed `unsafe` entry or a partially-applied conflicting `safe` entry does not stop later entries, so later entries — authored against a completed prior version — apply over a hybrid shape. This is what the contract says today ("while the rest of the file's entries proceed") and it is a design choice; recorded so the choice is a choice.+- **`min_supported_schema_version` has no reader** (Codex): the contract says the loader fails loudly below the floor, but no code determines a document's version. Held at zero and reserved for a squash, so it costs nothing today; the sentence overclaims until a squash and a version detection exist — Phase 3 or the first squash, whichever comes first.+- **The replay half of `migrate_file` catches only `TOMLKitError`** — a `PipelexUnexpectedError` from the applier, or a non-`TOMLKitError` from tomlkit, would cross the per-file boundary and abort the siblings. No trigger found; the applier's raises are "applier bug" cases where loud is arguably right. Recorded, not changed.+- **`narrowing.py` reports `int → float` and `enum → literal` (same members) as narrowings** — false reds, no real surface has the shape. Goes with Q.+- **Ownership, ACLs, xattrs and hard-link identity are lost on replace** (Codex) — the shared transaction primitive keeps mode only. Pre-existing on the `.mthds` path; a policy call together with N (symlinks) as the one backup/replace-semantics pass.+- **No directory `fsync` after the backup rename and the target replace** (security + Codex, low confidence): crash-durability of the ordering the contract promises. Cheap on POSIX; a decision on whether the guarantee is meant to hold across power loss.+- Maintainability items left as they are: the six-times hand-rolled ancestor-prefix loop (a helper is worth having when the next site appears); `MigrationReport.written_plans` / `blocked_plans` and `MigrationLedger.entry_for_version` have no caller yet (Phase 3's `pipelex migrate` is the reader — remove them there if it does not read them); a head-link narrowing is reported as `REMOVAL_NEEDS_A_BUMP` while `VALUE_DOMAIN_NARROWED` exists (the message says which; a distinct kind is a taste call); no `UNREADABLE` blocked reason (an unreadable file is reported as `UNWRITABLE` with a detail that says "read"); `_print_issues` duplicated across the two gate commands; `_KIT_CONFIGS_DIR` derived by path arithmetic; the temp-file prefix still says `pipelex-fix-`.+- Testing items left: `documents.py` has no direct test (reached through the transform check and the property suite); the two `MigrationReport` properties are untested because they are unused.++## Round 3 — `/code-review high` of the round-2 changes, 2026-08-16++Run over the staged round-2 diff, before it was committed. Every finding was reproduced in the venv (tomlkit 0.14) or against the gate before it was acted on. Working tracker for the round: `wip/pr-1112-round3-progress.md`.++### Fixed in this round++| Where | Issue | What was done |+|---|---|---|+| `applier.py` | **Round 2's split-descendant `CONFLICT` guard was a workaround for a tomlkit rendering-cache bug, and an incomplete one**: it looked at the merged dict facade of a nested out-of-order table, so a split table inside one of its chunks was invisible and the rename still tore the file with an `applied` verdict; and it fired on dotted keys interleaved inside a single table (`b.x = 1`, `c = 2`, `b.y = 3`), refusing a rename tomlkit performs correctly. The real cause: `Table.invalidate_display_name` walks `values()`, one item per key, so the later chunks of a split table keep the stale header. | The guard is gone. `_refresh_table_headers` walks the *body* of a renamed or moved item (Container body, proxy chunks, AoT elements) and clears every Table's `display_name`, so every chunk re-renders under the new path — rename and move now come out whole for every layout the parser accepts. The round-2 "Limits" bullet and changelog line are replaced; the CONFLICT fixtures became APPLIED-and-whole assertions, with the nested-proxy, interleaved-dotted and array-of-tables shapes added. |+| `coverage.py`, `narrowing.py` | **Round 2's split (`describe_tightenings` for a remapped origin) closed the bound half and left the type half open**: `Annotated[int, ge=1] \| Literal["auto","unbounded"]` → `Literal["unbounded"]` under a `safe` remap of `auto` was green on both gates while `retries = 4` fails at boot. | `describe_narrowing(remapped=...)` exempts only the string-typed members of the old type (`str`, `enum`, `literal`) — what a remap can rewrite — and still reports any other lost member and every tightened bound. `describe_tightenings` is removed. Gate fixture and unit test; contract row reworded. |+| `transform_check.py` | A hardcoded count in a docstring this round rewrote. | Reworded. |++### Deferred from this round++- **Renaming a table written in dotted-key form swallows the scalars after it** (pre-existing, reproduced): `[a]\nk.x = 1\nm = 3` + rename `k→kk` yields `[a]\n[a.kk]\nx = 1\n\nm = 3` — `a.m` becomes `a.kk.m`, verdict `applied`. `_replace_at` re-keys the dotted table with a plain `SingleKey`, so tomlkit renders a block header mid-body. The S4 table renames will meet any user file with a dotted section followed by a sibling scalar. Two remedies: refuse when the item's key `is_dotted` (a `CONFLICT` telling the user to write the section as a header), or preserve dotted-ness through `_replace_key_in_container`. Needs a decision, then a red/green fixture in `test_fix_applier_config_surface_shapes.py`.+- **The `UNWRITABLE` branch of the runner mislabels a landed write.** For the single-file commit the runner performs, `_commit_staged_updates` re-raises the original `OSError`/`FixWriteConflictError` when the replace itself fails (rollback of zero committed updates is trivially complete), so the only reachable `FixTransactionError` is the post-commit cleanup one — the write **landed**, and `_carries()` false means the file was changed or made unreadable afterwards. `blocked_detail` says "the file could not be written", `FileBlockedReason.UNWRITABLE` is documented as leaving the directory untouched, and the contract speaks of "a rollback it could not complete" — an unreachable path. Round 2's own table above states the correct reading. Fix is wording plus possibly a distinct reason (`CHANGED_DURING_RUN`); the reason-enum question goes with the `UNREADABLE` item from round 2.+- **The backup kept after an uncertain-state failure is pruned by the next successful run of the same file** — `prune_backups_except(keep=T2)` unlinks the round-1 copy whose provenance was certain, while the report told the user "the original is kept at T1". Either the message says the copy lives until the next successful run, or a backup made by a run that did not write is exempt from pruning. Policy call, with M/N and the replace-semantics items.+- **The over-deletion loop no longer fires on a rename onto a path the new schema keeps** (`label` live at v2, entry renames `label→title`, `title` new at v2): the old spelling-based loop reported `OVER_DELETION`; the origin-based one carries `label` to `title`, finds it in the new schema, and is silent. `check-ledger` refuses the same entry (`OP_ACTS_ON_LIVE_MATERIAL` + `CONVERGENCE_BROKEN`), so the defect cannot ship, but `check-migration-schemas` alone is green and no test pins either behaviour. Add the fixture, and decide whether the surface gate should say it too.+- **`_over_deleted_landings` mirrors `PathState.trace_to_origin`** (origin→landing vs current→origin, same longest-recorded-prefix loop). Round 2 said the ancestor-prefix loop earns a helper "when the next site appears"; this is that site — a `PathState.landing_of_origin` would let the caller collapse to one loop. Do it with the item above.+- Minor, confirmed: `reserved.py` keys a remap's value reservation by its op-time spelling, so a remap-then-rename in one entry never trips `RESERVED_VALUE_REUSED` (goes with S4 above); `str(chunk_key)` renders quoted/whitespace key text in a detail message; the `or {}` unpack repeated in `describe_narrowing`.diff --git a/wip/pr-1112-round3-progress.md b/wip/pr-1112-round3-progress.mdnew file mode 100644index 000000000..ff975df2b--- /dev/null+++ b/wip/pr-1112-round3-progress.md@@ -0,0 +1,30 @@+# PR #1112 — round 3 (`/code-review high` of the staged round-2 changes), progress tracker++Working file so the session can pause and resume. Branch `feature/Migrator-2`, worktree `_migrator/`. The round-2 fixes were STAGED but uncommitted when this round started; round-3 edits go on top of them. Full findings are in the review report; the triage below is what was decided.++## Fix-now set (in order)++| # | Item | Status |+|---|---|---|+| 1 | `applier.py` — replace the split-descendant CONFLICT guard by the tomlkit root-cause fix: clear every Table's stale `display_name` in the renamed/moved subtree so all chunks re-render under the new name. Covers the nested-proxy miss (`:411`), the interleaved-dotted false positive (`:423`), and drops the documented "limit" from CHANGELOG + `docs/migration-ledger.md`. Tests: turn the three CONFLICT fixtures into APPLIED-and-whole assertions, add nested-proxy + interleaved-dotted cases. | DONE — `_first_split_descendant` + `_split_table_detail` removed, `_refresh_table_headers` added and called from `_replace_key_in_container` and the move; tests rewritten (5 shapes × rename+move, root proxy, nested proxy); CHANGELOG entry rewritten, contract "Limits" bullet dropped |+| 2 | `coverage.py:351` — a `safe` remap must exempt only the string members it can rewrite; a lost non-string member of a union still counts as a narrowing (`describe_narrowing` type half must still run for remapped origins). Test: `int \| Literal` → `Literal` under a remap must be red in `check_surface`. | DONE — `describe_narrowing(remapped=...)` exempts only `str`/`enum`/`literal` members; `describe_tightenings` removed; gate test + unit test added; contract table row reworded |+| 3 | `transform_check.py:38` — drop the hardcoded count "Three things". | DONE |+| 4 | Deferrals written into `wip/pr-1112-review-notes.md` § Round 3 (see list below). | DONE — § Round 3 appended to `wip/pr-1112-review-notes.md` |+| 5 | `make agent-check` + targeted tests + `make agent-test`. | DONE — agent-check green, agent-test all passed |++## Deferred (to be written into `wip/pr-1112-review-notes.md`, not fixed now)++- `applier.py:374` pre-existing: renaming a dotted-form table (`k.x = 1` then sibling `m = 3`) swallows the siblings under `[a.kk]` — decision needed: refuse on `is_dotted`, or preserve dotted-ness in `_replace_key_in_container`.+- `runner.py:119` UNWRITABLE wording: for a single-file commit the only reachable `FixTransactionError` is post-commit cleanup, i.e. the write LANDED; the "rollback it could not complete" text in code and `docs/migration-ledger.md` describes an unreachable path.+- `runner.py:127` the backup kept after an uncertain-state failure is pruned by the next successful run of the same file.+- `coverage.py:176` over-deletion loop no longer fires on a rename onto a path the new schema keeps; caught by `check_ledger`, but no test pins it.+- `coverage.py:196` `_over_deleted_landings` mirrors `PathState.trace_to_origin` — a `landing_of_origin` helper would collapse it.+- Minor confirmed items from the reviewer: `reserved.py:85` remap reservation keyed by op-time spelling (remap-then-rename escapes RESERVED_VALUE_REUSED); AoT treated as terminal in the split walk; `str(chunk_key)` rendering of quoted keys in a detail; small duplications in `describe_narrowing` / proxy branch.++## Log++- (start) tracker written; nothing edited yet.+- item 1 done; fixes + migration unit suites green (390 passed).+- items 2 and 3 done; migration unit suite green.+- item 4 done (deferrals written). Next: make agent-check, then make agent-test.+- item 5 done: `make agent-check` green, `make agent-test` all passed. Round 3 fix-now set COMPLETE; everything staged, not committed.