Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Migrator Phase 3, milestones 2-3: what an unsafe entry promises, and catching a narrowing

Louis Choquel0e3e8e11061b
diff --git a/CHANGELOG.md b/CHANGELOG.md
index a50dade2c..a00828ab1 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,8 @@
### Added
+- **An `unsafe` migration entry now says what it is about, and keeps saying it**: `unsafe` is the ledger's form for a change only a human can make, and the coverage gate's only remedy for a tightened numeric bound — but the engine decided whether to report one by rehearsing its operations, so an entry with none was accepted by the accounting and then guaranteed to reach nobody, and any entry went silent the moment a later `safe` entry renamed the table its material sat in (reported on the first run, clean on the second, boot still failing). An entry now declares the paths whose value domain narrowed, in `declared_narrowed_paths`; an entry with no operations must carry one, a `safe` entry may not, `make check-ledger` refuses a path the entry's own version does not record, and the coverage gate no longer takes the bare word `unsafe` as accounting for a narrowing or a lost enumerated spelling — it demands the path be named. The material of an `unsafe` entry is traced through the ledger before a document is questioned: back through the entry's own operations, since it never applied them, and forward through every later `safe` entry, so the same file is reported on every run for as long as it carries it. A file that merely *sets* a narrowed path is reported under a new, weaker `value_domain_narrowed` reason listing the keys to check by hand — the engine is model-free by design and cannot tell a value the new domain refuses from one it accepts — and those keys are named at the spelling the migrated file carries rather than the one that matched, since the entry is questioned where the replay reaches it and the later `safe` entries rename the material before the file is written — and that report, alone among blocked entries, does not fail convergence, since a healthy reference document sets those paths like everyone else's file does. Forward tracing is deliberately confined to `unsafe` entries: a rehearsal may guess, an application may not.
+
- **A migration operation can now remap every value beneath an open mapping**: `remap_value` accepts `key = "*"`, meaning "each key of the addressed table" — the same "each of these" reading the wildcard already had in a `table_path`, applied to keys. A mapping from the user's own keys to an enumerated value (`dict[str, LogLevel]`) keeps its spellings under keys only the document can enumerate, so no fixed key reaches them and this is the only shape in which a member renamed beneath one can be repaired at all. On every other kind a `*` key is now refused when the operation is parsed: deleting every key of a table is `delete_table`, and renaming or moving every key onto one fixed name is the many-to-one that already rules a wildcard out of a `move_key`. Refused rather than tolerated, because an unrefused one is a *dead* operation that skips forever while looking like an accounting.
- **A schema change that narrows what a value may *be* is now caught, not waved through as additive**: A change can keep every path and every enumerated spelling and still break a user's file — `int` becomes `str`, a free string becomes an enum, a bound moves from `ge=1` to `ge=5`. Read as a path diff those look like additions, so the coverage gate used to ask for a golden regeneration, demand no version bump and no entry, and let the next boot reject a valid file with a green gate behind it. It now splits its verdict by *direction*: a widened type or a relaxed bound still costs only a regeneration, while a narrowing demands a bump and an entry carrying a `remap_value` for each narrowed path — or marked `unsafe`, which is the only remedy a tightened numeric bound has, since no structural operation can repair a value. The same rule applies to a stored entry, compared by origin so a narrowing hidden behind a rename is still caught, and to a `pre_history` entry, so the flag is not a way past it. To see a bound at all the fingerprint now records one: a **closed whitelist** over `gt`, `ge`, `lt`, `le`, `min_length`, `max_length` and `multiple_of`, read from `annotated_types` (now a declared dependency) and from the field metadata pydantic folds a top-level `Field(ge=1)` into, with anything unrecognized dropped rather than serialized — `pattern` excluded permanently, because regex containment is not decidable and every pattern edit would read as a tightening. One correction rides along: an enumerated type relaxed into a free string no longer reads as the loss of every spelling it had. The contract states in the same change what the gate structurally cannot see — narrowing expressed in a validator, which no projection of the annotations can reach.
@@ -32,6 +34,13 @@
- **Unknown per-model backend keys must be header-shaped (Breaking)**: A key in a per-model table of a backend TOML that is not a model-spec field is sent to the provider as an HTTP request header — that mechanism is now documented, and it is gated on shape: the key must contain a hyphen (`x-portkey-provider`, `anthropic-beta`). An unknown key without one (`max_tokns`, or a field deleted from the blueprint) used to go out on the wire silently while the real setting stayed unset; it now fails the backend load, in strict and lenient boots alike, with an error naming the key, the model and the file. A hyphenated spelling of a real field (`max-tokens`) is rejected too, naming the field it resembles, and so is a header-shaped key whose value is not a string (`x-foo = 3`) — a header value must be a quoted string, and is never stringified onto the wire. A key that clears the shape gate must also be usable as a header on the wire: the name may contain only letters, digits and RFC 7230 token punctuation, so a quoted `"x-foo bar"` is refused, and the value must be printable ASCII on a single line with no leading or trailing whitespace, so a line break or an invisible trailing space is refused — each of those used to load fine and have the HTTP client reject the request on the first call to that model. In the remotely served Pipelex Gateway config the same rule prunes such keys instead of failing, as version skew. `endpoint_path` — the provider-side route the Gateway image models are called by — is a declared model-spec field rather than a header entry.
- **Remote inference config bumped to `pipelex_remote_config_12.json`**: The Pipelex Gateway model specs are fetched from a new versioned URL whose `defaults` no longer declare the removed field. Earlier releases stay pinned to `_11`, which is frozen. Unknown keys in the *remote* config's `defaults` are now pruned rather than rejected, so a gateway config written by a different release than the client reading it is version skew to tolerate; local backend files stay strict.
+### Fixed
+
+- **Renaming a table written in dotted form no longer swallows the keys after it**: `k.x = 1` and a `[a.k]` header say the same thing to a TOML parser and are two different things to a TOML document, and which one a key is is a flag the parser sets rather than a property of its name. Rebuilding the key from its name alone — what tomlkit's own re-key primitive does — lost that flag, so a renamed dotted key came back out as a block header, and a block header absorbs every scalar that follows it in the same table: a neighbouring `m = 3` silently became a key of the renamed table, with an `applied` verdict and no diagnostic. Renamed at an inner segment the whole chain re-rendered at the document root, taking the subtree out of the table it lived in. The applier now renames the document's body entries where they sit, carrying the flag forward and covering every chunk of a key written on several dotted lines — which also stops a cosmetic blank line being injected after any table it renames. This reaches every caller of the applier, the `.mthds` fix loop included.
+- **A migration no longer replaces a symlinked configuration file with a regular one**: a file symlinked out of a dotfiles directory is now read, backed up and replaced at the end of the link, the way the `.mthds` fix loop already resolves its own targets — the link survives and the real file is the one that gets migrated. Replacing the link path instead put a regular file where the link was and left the real file unmigrated.
+- **A migration run no longer takes away a backup it did not make**: the backup stamp resolves to the second, so two runs of the same file can address the same name. A run now reserves the name atomically and leaves whatever is already there — an older copy of the file, which is to say the original — instead of overwriting it, and a run whose write is refused deletes only the copy it made itself. A copy kept because a write could not be confirmed is renamed out of the `.bak.` rotation into `.rescue.`, so the next successful run cannot prune away the very file the report told the user to go and get — and when that copy cannot be moved, because another run made it or the rescue name is taken, the report says it is still in the rotation and asks the user to take it now instead of promising it will keep; and the backup's directory entry is `fsync`-ed before the target is replaced, so "back up first, replace second" holds across a power loss and not only a process exit.
+- **A blocked configuration file now says which state it is in**: `FileBlockedReason` gained `unreadable` and `state_uncertain`, so an unreadable file is no longer reported as *unwritable* and a write whose outcome the transaction could not describe is no longer reported as one that never happened. The wording of that last case described an unreachable path — for the single-file commit a migration performs, a failed replacement re-raises its own error, so the only transaction failure that reaches a plan is the one raised after the target was replaced.
+
### Removed
- **`prompting_target` (Breaking)**: Gone from `LLMSetting`, `InferenceModelSpec`, `InferenceModelSpecBlueprint` and every backend TOML, along with the `PromptingTarget` enum and the per-target style map (`[pipelex.prompting_config]` and its `prompting_styles` sub-table, replaced by `[pipelex.templating_config]`). **Migration:** delete every `prompting_target` line from your own `.pipelex/inference/backends/*.toml` — backend files are read strictly, so a surviving one fails the boot. Check `portkey.toml` in particular: the key appears in `[defaults]` *and* on individual model entries, and a surviving per-model one is rejected by name as an unknown, non-header key.
diff --git a/docs/migration-ledger.md b/docs/migration-ledger.md
index ea600455d..e268b51d9 100644
--- a/docs/migration-ledger.md
+++ b/docs/migration-ledger.md
@@ -132,6 +132,8 @@ description = "One or two sentences a release note can quote."
guidance = """
Agent-facing Markdown: what a user should understand or decide. Never the mechanism.
"""
+# unsafe entries only — the paths whose value domain this version narrowed:
+# declared_narrowed_paths = ["reporting.retries"]
[[migration.ops]] # rename a nested table in place, inside its parent
kind = "rename_table_key"
@@ -155,7 +157,8 @@ The rules that govern the file:
- **Entries are ordered by `to_schema_version` and replayed in that order.** Versions are contiguous and match their ids. A key renamed at version 2 and moved at version 4 lands correctly for a file coming from version 1, because the entries compose in sequence.
- **`safety` and `guidance` are independent.** `safety` governs whether the applier may act: `safe` means mechanically complete and applied after one confirmation, `unsafe` means reported and never applied. `guidance` is prose any entry may carry, whatever its safety.
-- **`guidance` is the explanation, never the mechanism.** Anything expressible as operations must be operations. An entry may legitimately carry no operations at all with `safety = "unsafe"`, for a change only a human can make.
+- **`guidance` is the explanation, never the mechanism.** Anything expressible as operations must be operations. An entry may legitimately carry no operations at all with `safety = "unsafe"`, for a change only a human can make — and then it must say which paths it is about, in `declared_narrowed_paths`. See [What an `unsafe` entry promises](#what-an-unsafe-entry-promises).
+- **`declared_narrowed_paths` belongs to an `unsafe` entry and nowhere else.** A narrowing a `remap_value` can repair is accounted for by that remap; one it cannot repair is precisely what makes an entry `unsafe`, so the field on a `safe` entry is a contradiction and is refused when the file is parsed.
- **Operations target the narrowest path that expresses the change.** A ledger entry is permanent data that outlives the current field census of its parent table, so deleting a parent because it happens to have one field today is wrong — unless the parent itself is what retires, in which case the entry deletes the parent and the parent's name becomes reserved.
- **The parsed ledger is cached per process.** Replaying a surface over many files must not re-parse the TOML per file.
@@ -175,6 +178,41 @@ This exception is bounded to entries carrying the flag, and exists for the chang
The `before` document lives beside the golden chain as `[email protected]`, and it is the one file there that is neither snapshotted nor regenerated — a regenerated one would describe today's models, which is exactly what it is not about. Write it as a faithful representative of the old shape, carrying every path the entry's operations address, so that the transform check exercises each of them rather than taking them on the entry's word. The link then runs from that document instead of from `defaults@N-1`, and nothing else about the check changes: the same three claims verify it.
+## What an `unsafe` entry promises
+
+`safe` means the applier acts. `unsafe` means it does not — and everything an `unsafe` entry is worth rests on one sentence:
+
+> **An `unsafe` entry is reported on every run, to every file that still carries the material it is about — at whatever spelling that material has reached — and to no other file.**
+
+Both halves are load-bearing, and they pull against each other. Report too little and the entry is a note nobody reads; report too much and every user with a perfectly current file is warned, at every boot, forever, which teaches everyone to ignore the one warning that mattered. So the entry is **questioned** against each document before it is reported, and what makes that question answerable is what the entry declares.
+
+**The material an entry is about** is its operations' sources, plus its `declared_narrowed_paths`.
+
+- **Operations are questioned by rehearsing them** against the document and throwing the result away. This is value-sensitive by construction, which is the point: a `remap_value` is about a stale *spelling*, not about a path, so a user whose value was never stale hears nothing.
+- **Declared narrowed paths are questioned by looking them up.** No operation can express a value the new schema refuses, so there is nothing to rehearse. Presence is the whole predicate — the engine is model-free by design, and cannot tell a value the narrowed domain rejects from one it accepts. What it reports is therefore *check this key*, not *this value is wrong*.
+
+> **An entry with no operations must declare at least one narrowed path, and the ledger refuses one that does not.** That shape is the contract's own form for "a change only a human can make", and the [coverage table](#coverage-every-schema-change-is-accounted-for)'s only remedy for a tightened numeric bound — but with nothing to rehearse and nothing to look up, such an entry answers "nothing to say" for every file there will ever be. The accounting accepts it; the engine then guarantees it reaches nobody.
+
+> **`unsafe` on its own accounts for nothing.** The coverage gate lets an `unsafe` entry leave a narrowing unremapped and an enumerated spelling unmapped — that is what `unsafe` is for — but it demands the path be named in `declared_narrowed_paths`. Otherwise the word satisfies a reader while the user whose value the new schema refuses is left with a failing boot and no message.
+
+> **Declared paths are spelled as the fingerprint at the entry's own version records them**, `*` segments included, and [`make check-ledger`](#the-targets-and-where-the-data-lives) refuses one that version does not have. A narrowing keeps its path and shrinks what it accepts — that is exactly what separates it from a removal, which is accounted for by the operation that removes it. A path no version records is looked for in every file and found in none.
+
+**Whatever spelling the material has reached.** An `unsafe` entry is never applied, so a file it blocks keeps the old shape while later `safe` entries go on renaming the tables around it. Questioning only the entry's own spelling makes an entry that reported on the first run go silent on the second, with the file still broken:
+
+```
+unsafe@2 is about [reporting] mode safe@3 renames reporting → output
+run 1: reports unsafe@2, then applies the rename → the file now says [output] mode
+run 2: no reporting.mode anywhere → silent, while the boot still fails on output.mode
+```
+
+So the material is traced through the ledger before the document is questioned: **back** through the entry's own operations, because the entry never applied them, and **forward** through every later `safe` entry, for a file that has been migrated past this one. A later `unsafe` entry moves nothing, since it is never applied either; material a later entry *deletes* stops being traced, because a file migrated that far no longer carries anything the entry is about. All of it is ledger arithmetic — an operation says which path it moves where — so the engine stays the model-free, filesystem-free function the gates replay.
+
+> **What gets reported is the spelling the file this run *wrote* uses.** Questioning happens where the replay reaches the entry; the later `safe` entries of the same run then rename the material before the file is written. So `narrowed_paths[]` names the end of that forward trace rather than the spelling that matched — otherwise a run that reports `reporting.retries` hands back a file that calls it `output.retries`, and sends the user looking for a key it no longer has.
+
+> **A rehearsal may guess; an application may not.** Forward tracing applies to `unsafe` entries only, because their operations are rehearsed against a copy and discarded: the worst a wrong guess costs is one report too many. A `safe` entry silenced the same way — by its own `CONFLICT`, after which a later entry renames the table around it — is left alone deliberately, because repairing it would mean *writing* at a spelling its author never wrote. That is a different promise and wants its own decision.
+
+> **Convergence exempts a reported narrowing, and nothing else.** The reference documents set the narrowed path exactly as every healthy file does, so an entry reported for that reason must not fail [convergence](#convergence-replay-is-neutral-on-the-witnesses) — otherwise the one remedy for a tightened bound could never be written. An `unsafe` entry whose *operations* fire on a witness is a different matter and still fails: that says the checked-in reference document carries retired material.
+
## The operation vocabulary
The ledger draws on the same operation vocabulary as the `.mthds` fix path, restricted to its structural half. The full union is published as two subsets — **fix operations** (every kind) and **migration operations** (the structural kinds only) — so a ledger containing a non-migration kind fails validation when the ledger is parsed, not later at gate time.
@@ -235,6 +273,7 @@ Enforced by [`make check-ledger`](#the-targets-and-where-the-data-lives), agains
- **Every operation's source must be a path some schema version removed** — recorded as such in the reserved-path registry by the entry that removes it. An operation whose source is still a live path of the current schema would act on a valid file, and replay neutrality would be false. A `remap_value` is exempt by construction: what it retires is an enumerated *spelling*, not the path, which survives into the new schema — the remap legality rule is what governs it instead.
- **No operation may address a concrete key beneath an open node.** Those keys are the user's, they are unbounded, and no schema change can remove one.
- **A `*` segment is legal exactly at an open node**, and nowhere else.
+- **Every `declared_narrowed_path` must be a path the fingerprint at the entry's own version records.** The declaration is what the engine questions a document about, so one no version records reaches nobody — and a path that version has *lost* is a removal, which the operation that removes it accounts for. See [What an `unsafe` entry promises](#what-an-unsafe-entry-promises).
### Sequential path state
@@ -244,8 +283,8 @@ Legality is defined over an entry's **sequential path state**, not operation by
- **A `safe` rename or move must not land on a path the state already has.** The applier refuses to clobber an occupied destination, so a file carrying both keys — a perfectly valid file at the previous schema — would come back `CONFLICT` on every run. Such an entry is refused as **destination occupied**; it can be `unsafe`, or choose a destination the previous schema did not have.
- **Every path surviving the walk must be a path of the new fingerprint.** A survivor that is not is either an unaccounted removal or a misspelled destination; the two are the same defect seen from either end, and one containment check catches both.
- **Every path of the new fingerprint that the walk removed must be a genuine addition** — that is, absent from the previous fingerprint. One that was present is **over-deletion**: an entry that dropped a parent table where it meant to drop one child.
-- **Every enumerated spelling lost between the two fingerprints must be remapped, following the path through the entry's own renames.** Members are compared by origin — an enumerated path of the previous schema is looked up at the path the walk carried it to — so a member lost by a path the same entry renamed still demands its `remap_value`, and the remap that supplies it addresses the key by its new name.
-- **Every path whose value domain narrowed must carry a remap, or the entry must be `unsafe`.** Compared by origin like the spellings, and for the same reason: a narrowing hidden behind a rename would otherwise read as an unrelated addition and removal. The remedies are the two the [coverage table](#coverage-every-schema-change-is-accounted-for) names, and they are fewer than a removal has, because no structural operation can repair a value.
+- **Every enumerated spelling lost between the two fingerprints must be remapped, following the path through the entry's own renames.** Members are compared by origin — an enumerated path of the previous schema is looked up at the path the walk carried it to — so a member lost by a path the same entry renamed still demands its `remap_value`, and the remap that supplies it addresses the key by its new name. An `unsafe` entry may leave one unmapped and answers for it by naming the path in `declared_narrowed_paths`.
+- **Every path whose value domain narrowed must carry a remap, or be named in an `unsafe` entry's `declared_narrowed_paths`.** Compared by origin like the spellings, and for the same reason: a narrowing hidden behind a rename would otherwise read as an unrelated addition and removal. The remedies are the two the [coverage table](#coverage-every-schema-change-is-accounted-for) names, and they are fewer than a removal has, because no structural operation can repair a value. `unsafe` without the declaration is not one of them — see [what an `unsafe` entry promises](#what-an-unsafe-entry-promises).
The end state is deliberately *not* compared as "the new fingerprint minus its additions". That formulation reads a rename's destination as an addition and its source as a removal, so it demands that the destination be absent from the end state — which is exactly where a correct rename puts it. Containment plus the over-deletion check is the same guarantee stated in a form renames satisfy.
@@ -320,8 +359,8 @@ The verdict turns on **direction**, not on whether something moved: a change tha
|---|---|
| Unchanged | Pass. |
| Paths or enum members added, a default changed, or a type or bound **widened** | Regenerate the golden. No version bump and no entry are demanded, because additive changes are absorbed by the defaults layer and a widened domain still accepts every value a file already carries. Widening is defined so the fingerprint can decide it: a union whose members are a superset of the old ones, an enumerated type becoming `str`, and a bound dropped or relaxed. |
-| Any path or enum member removed or renamed | Require a schema version bump, an operation accounting for every removed path (wildcard paths included), and a remap or an `unsafe` entry for every removed enum member. Destinations are cross-checked against added paths. |
-| A type or bound **narrowed** — `int → str`, `str → SomeEnum`, `list[int] → list[str]`, `ge=1 → ge=5` | Require a schema version bump and an entry, exactly as a removal does. Every path survives and every spelling is still enumerated, and a file carrying an out-of-domain value stops validating all the same. No structural operation can repair a value, so the entry must carry a `remap_value` for each narrowed path — where the old spellings can be enumerated and [remap legality](#replay-neutrality) accepts it — or be marked `unsafe`, which is the only remedy a tightened numeric bound has. A remap answers only for the string values it rewrites — a lost `str`, enum or literal member — and for nothing else: an `int \| literal` field that loses its number, or that loses a spelling *and* raises its floor, still demands `unsafe`, because no mapping reaches a number. |
+| Any path or enum member removed or renamed | Require a schema version bump, an operation accounting for every removed path (wildcard paths included), and for every removed enum member either a remap or an `unsafe` entry **naming that path in `declared_narrowed_paths`**. Destinations are cross-checked against added paths. |
+| A type or bound **narrowed** — `int → str`, `str → SomeEnum`, `list[int] → list[str]`, `ge=1 → ge=5` | Require a schema version bump and an entry, exactly as a removal does. Every path survives and every spelling is still enumerated, and a file carrying an out-of-domain value stops validating all the same. No structural operation can repair a value, so the entry must carry a `remap_value` for each narrowed path — where the old spellings can be enumerated and [remap legality](#replay-neutrality) accepts it — or be marked `unsafe` **and name each narrowed path in `declared_narrowed_paths`**, which is the only remedy a tightened numeric bound has. A remap answers only for the string values it rewrites — a lost `str`, enum or literal member — and for nothing else: an `int \| literal` field that loses its number, or that loses a spelling *and* raises its floor, still demands `unsafe`, because no mapping reaches a number. |
Alongside the diff, one standing invariant is checked on the new fingerprint itself, whether or not anything changed:
@@ -335,7 +374,7 @@ Stating it over every required path rather than only over newly added ones costs
### Convergence — replay is neutral on the witnesses
-A full replay of a surface's ledger over each of its reference documents produces zero applied operations, reports nothing about them, and returns the very bytes it was given. The same target checks what else is cheap and unambiguous: the ledger parses into the migration-operation subset, versions are contiguous and match ids, every operation is legal, every `safe` remap satisfies the legality rule, and no entry reintroduces a reserved path or a remapped-away spelling.
+A full replay of a surface's ledger over each of its reference documents produces zero applied operations, reports nothing about them beyond a [declared narrowing](#what-an-unsafe-entry-promises), and returns the very bytes it was given. The same target checks what else is cheap and unambiguous: the ledger parses into the migration-operation subset, versions are contiguous and match ids, every operation is legal, every `safe` remap satisfies the legality rule, and no entry reintroduces a reserved path or a remapped-away spelling.
> **This target reads checked-in files and never fingerprints a live model** — the ledgers, the stored golden chain, the two reference documents (one of which, for a surface whose defaults come from its model, is rendered from that model's default values — rendered, not fingerprinted). That restraint is what earns it a place in `make agent-check`: every failure is a statement about a file the author wrote, and every remedy is to fix one. A check that read the models would go red on an ordinary configuration edit with *regenerate the golden* as its remedy, which is the fail-regenerate-fail cycle that keeps the coverage gate out of the loop agents run constantly. The cost is bounded and visible: between a version bump and the `make up-migration-schemas` that snapshots it, the new link does not exist, so the entry checks that need a fingerprint pair report the missing link by name instead of running.
@@ -415,19 +454,33 @@ MigrationReport
blocked_reason, blocked_detail → set when the file itself could not be processed at all
backup_path, was_written → set when the run wrote the file
steps[] → entry_id, to_schema_version, title, description, breaking, safety, applied_ops[]
- blocked[] → entry_id, to_schema_version, reason, detail, guidance, applied_ops[]
+ blocked[] → entry_id, to_schema_version, reason, detail, guidance, applied_ops[], narrowed_paths[]
unexplained[] → path, note
```
There is no `from_version` and no trusted-version concept: nothing skips, so nothing needs one. Because a replay walks every entry while usually changing little, the report renders what actually changed — a step lists only the operations that fired, not the many that skipped.
-Two different things can be blocked, and the report keeps them apart. An **entry** is blocked when it cannot be applied — it is `unsafe`, or one of its operations came back `CONFLICT` — and it lands in `blocked[]` with the reason and its guidance, while the rest of the file's entries proceed. A **file** is blocked when it cannot be processed at all: it is unparseable, it is unwritable, or it changed on disk between the read and the write. That reason sits on the plan itself, and a blocked file never stops its siblings — every other file in the surface is migrated and reported normally.
+Two different things can be blocked, and the report keeps them apart. An **entry** is blocked when it cannot be applied — it is `unsafe`, or one of its operations came back `CONFLICT` — and it lands in `blocked[]` with the reason and its guidance, while the rest of the file's entries proceed. A **file** is blocked when it cannot be processed at all. That reason sits on the plan itself, and a blocked file never stops its siblings — every other file in the surface is migrated and reported normally.
+
+There is one file-blocked reason per **state the file is in**, rather than one per exception the run happened to catch, because the state is what decides what the user does next:
+
+| Reason | What it says about the file |
+|---|---|
+| `unreadable` | It is there and its bytes would not come. Nothing was written. |
+| `unparseable` | It was read and is not valid UTF-8, or not valid TOML. Nothing was written. |
+| `unwritable` | It needed a change and the run could not make it — the backup would not go down, or the replacement would not. The file is exactly as it was found. |
+| `changed_during_run` | It was removed or edited between the read and the write, so the run refused to write over work it had not seen. |
+| `state_uncertain` | The write could not be confirmed: the transaction could not describe what it left behind, and the file does not hold what the run wrote. |
+
+`state_uncertain` is the only one that cannot promise the file is as it was found, which is why it is not folded into `unwritable`: the next move is to compare the file against the rescue copy the plan names, not to fix a permission and run again. It is also, for the single-file commit a migration performs, the only transaction failure that reaches a plan at all — a replacement that fails re-raises its own error, because rolling back nothing is trivially complete.
Two rules govern how an entry appears, and both exist so that a report is never more optimistic than the file:
> **An entry is reported once.** An entry with a conflicting operation lands in `blocked[]` and not in `steps[]`, carrying in its own `applied_ops` whichever of its operations did land before the conflict was found. Operation-level atomicity is the applier's — a conflicting operation writes nothing — but an entry is not atomic, and saying it arrived whole when part of it could not would be a lie the next run has to correct.
-> **An `unsafe` entry is reported only when it would do something.** It is rehearsed against the document and the result discarded, so an entry with nothing to do on this file stays silent. Reporting every `unsafe` entry regardless would warn every user with a perfectly current file, at every boot, forever — and a warning nobody can act on is a warning everybody learns to ignore.
+> **An `unsafe` entry is reported only when the file still carries the material it is about.** Its operations are rehearsed against the document and the result discarded, and its declared narrowed paths are looked up — at every spelling later entries have given them. An entry with nothing to say about this file stays silent. Reporting every `unsafe` entry regardless would warn every user with a perfectly current file, at every boot, forever — and a warning nobody can act on is a warning everybody learns to ignore. The whole rule is [What an `unsafe` entry promises](#what-an-unsafe-entry-promises).
+
+> **A blocked entry's reason says which claim it is making.** `unsafe` means the file has the old *shape* and the applier will not change it. `conflict` means an operation's destination is already occupied. `value_domain_narrowed` is the weakest and the newest: the file *sets* a path whose accepted values the entry narrowed, and `narrowed_paths[]` lists those paths as the **ledger** spells them — `levels.*`, never the user's own `levels.my_package` — and at the spelling the file this run wrote carries, not the one that matched. It is a list of keys to check by hand rather than a list of errors, because telling one from the other needs the model and the engine has none by design.
> **No value read from a user's file is ever rendered** — not in the command's output, not in the structured plan, not in an error. Paths, operation kinds and ledger-supplied values carry everything a plan needs to say.
@@ -470,6 +523,14 @@ The `.mthds` fix path follows its applier with a canonical reflow of the whole f
Stated more strongly than the library can promise on its own: **a replay in which no operation applies returns the very text it was given** — not a re-serialization of it. Neutrality is therefore a property of the engine rather than of the TOML library it happens to use, and a round-trip can never contribute a change of its own to a file that needed nothing.
+### A rename changes a name, and nothing else about the line
+
+> **A key written as a dotted assignment stays a dotted assignment when it is renamed.**
+
+`package_log_levels.pipelex = "INFO"` and a `[pipelex.log_config.package_log_levels]` header say the same thing to a TOML parser and are two different things to a TOML *document*. Which one a key is is not a property of its name — it is a flag the parser sets and the renderer reads — and rebuilding the key from its name alone, which is what the library's own re-key primitive does, silently loses it. A renamed dotted key then came back out as a block header, and **a block header absorbs every scalar that follows it in the same table**: a neighbouring `m = 3` became a key of the renamed table, with an `applied` verdict and nothing said. Renamed at an inner segment the whole chain re-rendered at the document root, taking the subtree out of the table it lived in.
+
+Refusing the layout with a `CONFLICT` was the alternative and it is the wrong trade: a rename has exactly one correct answer on a dotted key, the layout is ordinary TOML that no formatter rewrites, and refusing would strand a reshape's table renames on any file that happens to be written that way. So the applier renames the document's body entries where they sit — every chunk of a key written on several dotted lines, carrying the flag forward — which also stops the library from injecting a cosmetic blank line after a table it renamed in place.
+
### The document is re-read between operations that applied
> **Operations are applied one at a time, and the document is parsed afresh after each one that changed it.**
@@ -484,11 +545,23 @@ Always back up, before writing. Exactly one backup per file — a successful run
The name is the source file's whole name, extension included, followed by `.bak.` and a compact UTC stamp: `pipelex.toml` backs up to `pipelex.toml.bak.20260815T120000Z`. Extension-included so a backup never shadows a real `.toml`, and separator-free so no filesystem objects to it. Pruning matches that whole shape, stamp included — a copy the user made by hand under a name that merely starts the same way (`pipelex.toml.bak.notes`) is theirs, and is never touched.
-The order of the three steps is itself a guarantee. The backup is written **first**, so there is never a moment with a rewritten file and no copy of the original; the older backups are pruned **last**, so there is never a moment with no backup at all; and a write that fails before touching the file takes its own fresh backup with it, so a failed run leaves the directory exactly as it found it. A write whose outcome the transaction cannot vouch for — a rollback it could not complete — is the one case that keeps its backup: the report names the copy, because that is precisely the state a backup exists for. Nothing that happens after the file is written can un-write it in the report: an older backup that will not prune, or a temp file the transaction could not remove, is a warning on a migrated file, never a blocked one.
+The order of the three steps is itself a guarantee. The backup is written **first**, so there is never a moment with a rewritten file and no copy of the original; the older backups are pruned **last**, so there is never a moment with no backup at all; and a write that fails before touching the file takes its own fresh backup with it, so a failed run leaves the directory exactly as it found it. Nothing that happens after the file is written can un-write it in the report: an older backup that will not prune, or a temp file the transaction could not remove, is a warning on a migrated file, never a blocked one.
+
+Three further rules make that safety net worth the name.
+
+> **A run never overwrites or removes a copy it did not make.** The stamp resolves to the second, so two runs of the same file can address the same name. The second reserves the name atomically, finds it taken, and leaves what is there — because what is there is a copy of an *older* state of the file, which is to say the original if anything is. The same rule governs the other direction: a run whose write is refused deletes the copy it made and never the copy it found.
+
+> **A copy the run cannot vouch for leaves the rotation.** When a write ends `state_uncertain`, the backup is renamed out of the `.bak.` family into `.rescue.` — `pipelex.toml.rescue.20260815T120000Z` — which pruning does not match. Otherwise the next successful run of that file would delete the very copy the report told the user to go and get. A rescue copy is never collected by anything but the user. When the copy cannot be moved — it is another run's, or the rescue name is taken, or the rename will not go — it stays where it is rather than being lost to a tidier name, and the report says the copy is still in the rotation and asks the user to take it now, instead of promising it will be waiting.
+
+> **The copy is durable before the file it copies changes.** Its bytes are `fsync`-ed by the staged write and its name by an `fsync` of the directory it lands in, so "back up first, replace second" survives a power loss and not only a process exit. The replacement of the target is deliberately *not* synced in turn: a migration lost to a crash is replayed by the next run, while a lost backup is lost.
+
+What a backup carries across is the file's **permission bits**, and that is the one deliberate exception to "no value read from a user's file is ever rendered" — a backup contains the user's values by definition, so a `0600` configuration must not acquire a world-readable copy beside it. Ownership, ACLs and extended attributes are **not** carried across the replace, on either the backup or the migrated file: an atomic same-directory replace cannot preserve what the running process has no right to set, and re-attaching an attribute blindly (a quarantine flag, a security label) is a worse guess than leaving it off. The security-relevant bit of a configuration file is its mode.
+
+A configuration file that is a **symlink** is followed: the file the user means is the one at the end of the link, so the run reads, backs up and replaces *that* file, and the link survives. Replacing the link path instead would put a regular file where the link was and leave the real file unmigrated. The plan keeps naming the path the directory walk found; the backup path shows where the bytes actually went. This is what the `.mthds` fix loop already does with its own targets. The two callers still differ on one half: the fix loop pairs its resolution with a write-scope check, so a link pointing outside the directories it was given is refused, and the migration runner has no equivalent — a configuration file symlinked outside every walked directory is migrated where it actually lives. Whether that should be guarded is open; see `wip/migrator-write-scope-and-rename-fidelity.md`.
### Per-file transactions
-Each file is written transactionally and independently: snapshot, stage, atomic replace, restore on failure. A file that is unparseable, unwritable or changed during the run is reported as blocked while its siblings proceed. No run leaves a partially rewritten file behind.
+Each file is written transactionally and independently: snapshot, stage, atomic replace, restore on failure. A file that cannot be processed is reported as blocked, with the reason naming the state it is in, while its siblings proceed. No run leaves a partially rewritten file behind.
## Limits you will meet
@@ -496,7 +569,7 @@ These are measured properties of the TOML library the engine uses, not aspiratio
- **Arrays of tables are unaddressable, and reserved.** An `[[entry]]` node is a list, and no `table_path` segment syntax reaches it or anything under it; the `*` wildcard has no array form. One surface carries one today — the telemetry configuration's `otlp` list of exporters — and the fingerprint records it as a terminal `list[table]`: what happens *inside* an exporter entry (a renamed field, a tightened bound, a new required member) is invisible to the coverage gate, so a change there is the author's to account for by hand, exactly as validator-expressed narrowing is. Addressing what lies beneath one needs a new segment kind and a decision on the day — until then, an operation pointed at one resolves nothing and is reported as a guarded skip rather than raising.
- **A moved table loses its introducing comment.** A comment block written above a table is stored as trailing trivia of the *previous* sibling, not as part of the table it appears to introduce, so it does not travel with a move. Comments *inside* a moved table travel intact, including trailing comments on individual keys. The consequence is real: a file seeded from a heavily-commented template and then migrated ends up with banner comments labelling sections that have moved away. Nothing in the engine may rely on comment fidelity across a move, and the package's own files are corrected by hand rather than by migration.
-- **A migration is not byte-minimal.** A rename adds a small amount of whitespace, and a renamed table that was written out of order across several chunks loses its own bare header while a plain table keeps it. Both forms are semantically identical, both are stable under replay, and neither accumulates — but a migration diff is not the minimal diff a human would have written.
+- **A migration is not byte-minimal.** A rename normalizes the spacing around its own `=`, and a renamed table that was written out of order across several chunks loses its own bare header while a plain table keeps it. Both forms are semantically identical, both are stable under replay, and neither accumulates — but a migration diff is not always the minimal diff a human would have written.
- **A guarded skip is never an error.** Every operation whose target does not resolve reports itself skipped rather than raising. That guard is what makes always-replay possible, and it is why a misdirected operation is caught by the checks rather than by a crash on a user's machine.
- **A renamed key cannot be addressed again in the same in-memory document.** The position-preserving rename updates the body the document renders from but leaves the node's raw `dict` storage stale, for any value that is not a table. Nothing the library renders or looks up is affected — which is why a single rename is correct, and why the `.mthds` fix path, whose renames target tables, never meets it. The engine [re-reads the document between operations that applied](#the-document-is-re-read-between-operations-that-applied), so nothing downstream has to know about this; it is recorded here because it explains a piece of the engine that would otherwise look like superstition.
diff --git a/pipelex/migration/backup.py b/pipelex/migration/backup.py
index d6f94399a..d5b3bd6f0 100644
--- a/pipelex/migration/backup.py
+++ b/pipelex/migration/backup.py
@@ -5,17 +5,36 @@ one, so a directory does not accumulate a decade of copies of a file whose durab
already in git. For the untracked files — and for the moment between two commits — this copy is
the whole safety net.
+Three rules make that safety net worth the name, and each is a policy this module owns:
+
+- **A run never overwrites or removes a copy it did not make.** The stamp resolves to the second,
+ so two runs can address the same name; the second reserves the name atomically, finds it taken,
+ and keeps what is there. It is the older copy that holds the original.
+- **A copy the run cannot vouch for leaves the rotation.** When a write fails in a state the
+ transaction cannot describe, the backup is renamed out of the `.bak.` family into `.rescue.`,
+ which pruning never matches — otherwise the next successful run would delete the very copy the
+ report told the user to go and get. Removing a rescue copy is the user's call, never the tool's.
+ A copy that cannot be moved — another run's, or one whose rescue name is taken — stays where it
+ is and says so, so the report can ask the user to take it now rather than promise it will keep.
+- **The copy is durable before the file it copies changes.** Its bytes are `fsync`-ed by the
+ staged write and its name by an `fsync` of the directory, so "back up first, replace second"
+ survives a power loss and not only a process exit.
+
The mode is inherited from the source rather than left to the process umask, and that is the one
deliberate exception to "no value read from a user's file is ever rendered": a backup contains the
user's values by definition, so a `0600` configuration must not become a world-readable `0644`
-copy sitting beside it.
+copy sitting beside it. Ownership and extended attributes are **not** carried across — an atomic
+replace cannot preserve what the running process has no right to set, and the security-relevant
+bit of a configuration file is its mode.
See `docs/migration-ledger.md` → "Backups".
"""
+import os
import re
from datetime import datetime
from pathlib import Path
+from typing import NamedTuple
from pipelex.pipeline.fixes.file_transaction import FileSnapshot, write_staged_file
@@ -23,6 +42,10 @@ BACKUP_INFIX = ".bak."
"""What marks a file as one of our backups. Appended to the whole file name, extension included,
so `pipelex.toml` backs up to `pipelex.toml.bak.<stamp>` and never shadows a real `.toml`."""
+RESCUE_INFIX = ".rescue."
+"""What marks a copy that has left the rotation. Deliberately outside the `.bak.` family so that
+`existing_backups_of` cannot see it and pruning can never take it."""
+
BACKUP_STAMP_FORMAT = "%Y%m%dT%H%M%SZ"
"""A UTC stamp with no separators a filesystem dislikes — no colons, which Windows refuses."""
@@ -31,6 +54,30 @@ _BACKUP_STAMP_PATTERN = re.compile(r"\d{8}T\d{6}Z")
file the user named `<file>.bak.notes` themselves."""
+class WrittenBackup(NamedTuple):
+ """Where this file's backup is, and whether this run is the one that put it there.
+
+ The second half is what keeps a failed run from deleting somebody else's copy: only a run that
+ created a backup may discard it.
+ """
+
+ path: Path
+ was_created: bool
+
+
+class RescuedBackup(NamedTuple):
+ """Where the pre-migration copy is after a write nobody could vouch for, and whether it is safe there.
+
+ The second half is what the report needs. A copy this run could not move — because another run
+ made it, because the rescue name was taken, or because the rename would not go — is still under
+ a `.bak.<stamp>` name, which the next successful run of the same file prunes. Naming it is
+ right; promising it will be waiting is not.
+ """
+
+ path: Path
+ was_rescued: bool
+
+
def backup_stamp(*, moment: datetime) -> str:
"""Render a moment as the stamp a backup file name carries.
@@ -44,6 +91,10 @@ def backup_path_for(*, path: Path, moment: datetime) -> Path:
return path.with_name(f"{path.name}{BACKUP_INFIX}{backup_stamp(moment=moment)}")
+def rescue_path_for(*, path: Path, moment: datetime) -> Path:
+ return path.with_name(f"{path.name}{RESCUE_INFIX}{backup_stamp(moment=moment)}")
+
+
def existing_backups_of(*, path: Path) -> list[Path]:
"""Every backup of one file currently on disk, oldest name first.
@@ -63,29 +114,72 @@ def existing_backups_of(*, path: Path) -> list[Path]:
)
-def write_backup(*, snapshot: FileSnapshot, moment: datetime) -> Path:
- """Write the pre-migration copy of one file and return where it landed.
+def write_backup(*, snapshot: FileSnapshot, moment: datetime) -> WrittenBackup:
+ """Write the pre-migration copy of one file and return where it landed, and by whose hand.
Staged and atomically renamed like any other write in this codebase, which also gets the mode
right for free: the staged temp file is created `0600` and `fchmod`-ed to the source's mode
before it is ever visible under its final name. A rename that fails takes the staged copy with
it — the user's directory is not where a half-made backup gets left.
+
+ The final name is **reserved** before the rename rather than simply overwritten, so a run that
+ finds the name already taken leaves that copy alone and says so. Two runs inside one UTC second
+ address the same name, and the one already there is a copy of an older state of the file: the
+ original, if anything is. Clobbering it would destroy the only thing a backup is for.
"""
destination = backup_path_for(path=snapshot.path, moment=moment)
staged_path = write_staged_file(snapshot=snapshot, content=snapshot.content, label="backup")
try:
- staged_path.replace(destination)
- except OSError:
+ if not _reserve_name(path=destination):
+ return WrittenBackup(path=destination, was_created=False)
+ try:
+ staged_path.replace(destination)
+ except OSError:
+ destination.unlink(missing_ok=True)
+ raise
+ finally:
+ # A no-op once the rename succeeded, which is the point of `missing_ok`.
staged_path.unlink(missing_ok=True)
- raise
- return destination
+ _make_directory_durable(directory=destination.parent)
+ return WrittenBackup(path=destination, was_created=True)
+
+
+def keep_backup_for_rescue(*, path: Path, backup_path: Path, moment: datetime) -> RescuedBackup:
+ """Move a backup out of the `.bak.` rotation and report where it now lives, and whether it moved.
+
+ Called when a write failed in a state the transaction could not describe. The copy is then the
+ only one whose provenance is certain, and the report names it to the user — so the next
+ successful run of the same file must not prune it away, which is exactly what it would do to
+ anything still named `.bak.<stamp>`.
+
+ Returns the path the copy is actually at, so the report always names a file that exists: a
+ rename that will not go, or a rescue name another run has already taken, leaves the copy where
+ it is rather than losing it to a tidier name. It leaves it inside the rotation too, and
+ `was_rescued` is how the report knows to tell the user to take it now rather than later.
+ """
+ rescue_path = rescue_path_for(path=path, moment=moment)
+ try:
+ if not _reserve_name(path=rescue_path):
+ return RescuedBackup(path=backup_path, was_rescued=False)
+ except OSError:
+ return RescuedBackup(path=backup_path, was_rescued=False)
+ try:
+ backup_path.replace(rescue_path)
+ except OSError:
+ # The reservation is this call's own empty file, and a rename that would not go leaves it
+ # standing in for a copy that is still under its old name.
+ rescue_path.unlink(missing_ok=True)
+ return RescuedBackup(path=backup_path, was_rescued=False)
+ _make_directory_durable(directory=rescue_path.parent)
+ return RescuedBackup(path=rescue_path, was_rescued=True)
def prune_backups_except(*, path: Path, keep: Path) -> list[Path]:
"""Delete every backup of `path` other than `keep`, returning the ones removed.
Called only after the new file is committed: pruning first would leave a window with no
- backup at all, which is the one state this helper exists to prevent.
+ backup at all, which is the one state this helper exists to prevent. A rescue copy is not a
+ backup by this name and is never a candidate.
"""
pruned: list[Path] = []
for candidate in existing_backups_of(path=path):
@@ -94,3 +188,42 @@ def prune_backups_except(*, path: Path, keep: Path) -> list[Path]:
candidate.unlink(missing_ok=True)
pruned.append(candidate)
return pruned
+
+
+def _reserve_name(*, path: Path) -> bool:
+ """Claim a file name atomically, reporting whether this call is the one that got it.
+
+ An exclusive create rather than a `path.exists()` test, because the question only means
+ anything if asking it and answering it are one operation: between a check and a rename another
+ process can place its own copy, and the rename would silently take it away.
+ """
+ try:
+ descriptor = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
+ except FileExistsError:
+ return False
+ os.close(descriptor)
+ return True
+
+
+def _make_directory_durable(*, directory: Path) -> None:
+ """`fsync` a directory, so a copy's *name* survives a crash and not only its bytes.
+
+ `write_staged_file` already `fsync`-s the contents; a rename is durable only once the directory
+ holding the new entry is. Doing it here — after the backup lands, before the target is replaced
+ — is what makes the migration's ordering a guarantee across a power loss rather than only
+ across a process exit. The target's own replacement is deliberately not synced: a migration
+ lost to a crash is replayed by the next run, while a lost backup is lost.
+
+ Directories cannot be opened for reading on Windows, so the call is skipped there; a failure to
+ sync is a weaker guarantee, never a reason to fail a backup that is already written.
+ """
+ try:
+ descriptor = os.open(directory, os.O_RDONLY)
+ except OSError:
+ return
+ try:
+ os.fsync(descriptor)
+ except OSError:
+ return
+ finally:
+ os.close(descriptor)
diff --git a/pipelex/migration/coverage.py b/pipelex/migration/coverage.py
index 41fdca41e..9072adc27 100644
--- a/pipelex/migration/coverage.py
+++ b/pipelex/migration/coverage.py
@@ -278,11 +278,13 @@ def _check_enum_accounting(
Members are compared by *origin*: an enumerated path of the previous schema is looked up at
the path the walk carried it to. A comparison over shared names would never look at a path
the same entry renamed, and would let a lost member ride into the new schema unremapped.
+
+ An `unsafe` entry is allowed to leave a member unremapped — describing what no operation can
+ do is what `unsafe` is for — but it answers for that member by declaring the path, exactly as
+ it answers for a narrowing. The word alone accounts for nothing: the engine questions an
+ unsafe entry before reporting it, and a declaration is the only thing it can question a
+ document about when there is no operation to rehearse (R9).
"""
- if entry.safety is MigrationSafety.UNSAFE:
- # An unsafe entry is reported and never applied, so it is allowed to describe a change no
- # operation can make. That is what `unsafe` is for.
- return []
remapped_by_origin: dict[str, set[str]] = {}
for remap in walk.remaps:
remapped_by_origin.setdefault(remap.origin_path, set()).update(remap.old_values)
@@ -304,11 +306,21 @@ def _check_enum_accounting(
reachable = is_remappable(record=before_record)
unaccounted = sorted(removed_members - remapped_by_origin.get(origin, set())) if reachable else sorted(removed_members)
if unaccounted:
- remedy = (
- "so the entry needs a remap_value for each, or must be marked unsafe"
- if reachable
- else "and no operation can rewrite a value inside a container, so the entry must be marked unsafe"
- )
+ match entry.safety:
+ case MigrationSafety.UNSAFE:
+ if final_path in entry.declared_narrowed_paths:
+ continue
+ remedy = (
+ f"and the entry is unsafe, which is allowed — but `unsafe` is only ever *reported*, and the engine "
+ f"reports it to a file that carries the material it names. Add '{final_path}' to "
+ f"declared_narrowed_paths, or nobody hears about this"
+ )
+ case MigrationSafety.SAFE:
+ remedy = (
+ "so the entry needs a remap_value for each, or must be marked unsafe"
+ if reachable
+ else "and no operation can rewrite a value inside a container, so the entry must be marked unsafe"
+ )
issues.append(
CoverageIssue(
surface_id=surface_id,
@@ -330,7 +342,7 @@ def _check_narrowing_accounting(
after: SurfaceFingerprint,
walk: EntryWalk,
) -> list[CoverageIssue]:
- """Every path whose value domain the entry narrows must carry a remap, or the entry must be unsafe.
+ """Every path whose value domain the entry narrows must carry a remap, or be declared unsafely.
Paths are compared by *origin*, exactly as enumerated members are: a path the same entry
renamed is looked up where the walk carried it, so a narrowing hidden behind a rename still
@@ -342,11 +354,13 @@ def _check_narrowing_accounting(
Everywhere else, a tightened numeric bound above all, the entry has to say `unsafe`: the
migration is then reported to the user and never applied, which is the honest answer when the
tool cannot tell a stale value from a deliberate one.
+
+ **`unsafe` alone is not the accounting; the declaration is.** The engine questions an unsafe
+ entry before reporting it, and the only thing it can question a narrowing about is the path.
+ So an unsafe entry answers for a narrowing by naming that path in `declared_narrowed_paths` —
+ without which the gate would be satisfied by a word while the user whose value the new schema
+ refuses is never told anything (R9).
"""
- if entry.safety is MigrationSafety.UNSAFE:
- # An unsafe entry is reported and never applied, so it is allowed to describe a change no
- # operation can make. That is what `unsafe` is for.
- return []
remapped_origins = {remap.origin_path for remap in walk.remaps}
issues: list[CoverageIssue] = []
@@ -364,14 +378,24 @@ def _check_narrowing_accounting(
reasons = describe_narrowing(before=before_record, after=after_record, remapped=origin in remapped_origins)
if not reasons:
continue
+ match entry.safety:
+ case MigrationSafety.UNSAFE:
+ if final_path in entry.declared_narrowed_paths:
+ continue
+ remedy = (
+ f"the entry is unsafe, which is the right answer, but `unsafe` is only ever *reported* — and the engine "
+ f"reports it to a file that carries the material it names. Add '{final_path}' to declared_narrowed_paths, "
+ f"or nobody hears about this"
+ )
+ case MigrationSafety.SAFE:
+ remedy = "no structural operation can repair a value, so the entry needs a remap_value for that path, or must be marked unsafe"
issues.append(
CoverageIssue(
surface_id=surface_id,
kind=CoverageIssueKind.VALUE_DOMAIN_NARROWED,
message=(
f"entry '{entry.id}': '{final_path}' accepts fewer values than it did — {', '.join(reasons)}. A file "
- f"carrying one of the values it has stopped accepting no longer validates, and no structural operation "
- f"can repair a value, so the entry needs a remap_value for that path, or must be marked unsafe"
+ f"carrying one of the values it has stopped accepting no longer validates, and {remedy}"
),
)
)
diff --git a/pipelex/migration/documents.py b/pipelex/migration/documents.py
index c0ffe08ae..7f434d1d5 100644
--- a/pipelex/migration/documents.py
+++ b/pipelex/migration/documents.py
@@ -12,6 +12,7 @@ Used by the checks that compare a migrated document against the one the new sche
from typing import Any, cast
from pipelex.migration.fingerprint import PATH_SEPARATOR
+from pipelex.suggested_fix import WILDCARD_SEGMENT
def document_paths(*, document: dict[str, Any]) -> set[str]:
@@ -43,3 +44,23 @@ def _flatten_into(*, mapping: dict[str, Any], prefix: tuple[str, ...], flattened
_flatten_into(mapping=cast("dict[str, Any]", value), prefix=path, flattened=flattened)
continue
flattened[dotted] = value
+
+
+def document_carries_path(*, paths: set[str], pattern: str) -> bool:
+ """Whether a document has anything at a schema path, `*` standing for one key of an open node.
+
+ A schema path is not a document path: beneath an open mapping the schema says `levels.*` while
+ the document says `levels.my_package`, and a literal lookup would answer "no" for every file
+ that has ever set one. Each `*` matches exactly one segment, which is what it means in the
+ fingerprint and in an operation alike — never a whole subtree.
+ """
+ if WILDCARD_SEGMENT not in pattern:
+ return pattern in paths
+ expected = pattern.split(PATH_SEPARATOR)
+ for path in paths:
+ segments = path.split(PATH_SEPARATOR)
+ if len(segments) != len(expected):
+ continue
+ if all(wanted in {WILDCARD_SEGMENT, found} for wanted, found in zip(expected, segments, strict=True)):
+ return True
+ return False
diff --git a/pipelex/migration/engine.py b/pipelex/migration/engine.py
index 643593168..1d7ce7e78 100644
--- a/pipelex/migration/engine.py
+++ b/pipelex/migration/engine.py
@@ -25,12 +25,15 @@ Three properties this module is responsible for:
function rather than a property of the TOML library we happen to use. When something does
apply, the output is `tomlkit.dumps` of the mutated document and nothing else: no canonical
reflow, because a one-key rename must not rewrite a user's spacing.
-- **An `unsafe` entry is never written, and is only *reported* when it would do something.** It
- is rehearsed against the text and the result discarded: on a file already at the current schema
- every operation skips, so a correct file stays silent instead of warning at every boot.
+- **An `unsafe` entry is never written, and is only *reported* when the file still carries the
+ material it is about.** Its operations are rehearsed against the text and the result discarded,
+ and the paths it declares narrowed are looked up — at every spelling later entries have given
+ them, and reported at the one this replay leaves behind, because the file the user is sent to
+ check is the file this run wrote. On a file the entry has nothing to say to, it stays silent
+ instead of warning at every boot. Where that material lives is `material.py`'s question.
See `docs/migration-ledger.md` → "Schema versions, and why every run replays everything",
-"Applying", and "What the engine reports".
+"Applying", "What an `unsafe` entry promises", and "What the engine reports".
"""
from collections.abc import Sequence
@@ -38,7 +41,9 @@ from collections.abc import Sequence
import tomlkit
from pydantic import BaseModel, ConfigDict, Field
+from pipelex.migration.documents import document_carries_path, document_paths
from pipelex.migration.ledger import MigrationEntry, MigrationLedger, MigrationSafety
+from pipelex.migration.material import declared_path_spellings, spelling_after_replay, unsafe_op_variants
from pipelex.migration.plan import BlockedEntry, BlockedEntryReason, MigrationStep
from pipelex.pipeline.fixes.applier import FixOpApplication, apply_fix_ops
from pipelex.suggested_fix import MigrationOp
@@ -110,36 +115,91 @@ def replay_ledger_over_text(*, ledger: MigrationLedger, text: str) -> DocumentRe
for entry in ledger.migration:
match entry.safety:
case MigrationSafety.UNSAFE:
- _rehearse_unsafe_entry(entry=entry, text=current_text, blocked=blocked)
+ _rehearse_unsafe_entry(entry=entry, ledger=ledger, text=current_text, blocked=blocked)
case MigrationSafety.SAFE:
current_text = _apply_safe_entry(entry=entry, text=current_text, steps=steps, blocked=blocked)
return DocumentReplay(text=current_text, steps=steps, blocked=blocked)
-def _rehearse_unsafe_entry(*, entry: MigrationEntry, text: str, blocked: list[BlockedEntry]) -> None:
+def _rehearse_unsafe_entry(*, entry: MigrationEntry, ledger: MigrationLedger, text: str, blocked: list[BlockedEntry]) -> None:
"""Find out whether an `unsafe` entry has anything to say about this file, writing nothing.
- The rehearsal's text is discarded, so the file is untouched whatever the outcome. An entry
- with nothing to do stays silent: reporting it regardless would warn every user with a
+ Every rehearsal's text is discarded, so the file is untouched whatever the outcome. An entry
+ with nothing to say stays silent: reporting it regardless would warn every user with a
perfectly current file, at every boot, forever.
"""
- rehearsal = apply_ops_over_text(text=text, ops=entry.ops)
- if not rehearsal.applied_count and not rehearsal.conflicts:
+ operations_would_fire = _operations_would_fire(entry=entry, ledger=ledger, text=text)
+ narrowed_paths = _declared_paths_this_file_carries(entry=entry, ledger=ledger, text=text)
+ if not operations_would_fire and not narrowed_paths:
return
+ if operations_would_fire:
+ reason = BlockedEntryReason.UNSAFE
+ detail = (
+ f"this file needs the changes of '{entry.id}', but the entry is marked unsafe: the applier cannot "
+ f"tell a stale value from a deliberate choice, so it is reported and never applied"
+ )
+ else:
+ reason = BlockedEntryReason.VALUE_DOMAIN_NARROWED
+ detail = (
+ f"'{entry.id}' narrowed what {', '.join(narrowed_paths)} may hold, and this file sets a value there. No "
+ f"operation can repair a value, so the entry is reported and never applied — check those keys by hand"
+ )
blocked.append(
BlockedEntry(
entry_id=entry.id,
to_schema_version=entry.to_schema_version,
- reason=BlockedEntryReason.UNSAFE,
- detail=(
- f"this file needs the changes of '{entry.id}', but the entry is marked unsafe: the applier cannot "
- f"tell a stale value from a deliberate choice, so it is reported and never applied"
- ),
+ reason=reason,
+ detail=detail,
guidance=entry.guidance,
+ narrowed_paths=narrowed_paths,
)
)
+def _operations_would_fire(*, entry: MigrationEntry, ledger: MigrationLedger, text: str) -> bool:
+ """Whether rehearsing the entry's operations against this file would do anything.
+
+ Value-sensitive by construction, because a remap is about a stale *spelling* rather than about
+ a path: reporting on the path alone would warn a user whose value was never stale. Asked at
+ every spelling `material.py` says the material can have, so that an entry which reported on
+ the first run does not go silent on the second because a later `safe` entry renamed the table
+ around it.
+ """
+ for ops in unsafe_op_variants(ledger=ledger, entry=entry):
+ rehearsal = apply_ops_over_text(text=text, ops=ops)
+ if rehearsal.applied_count or rehearsal.conflicts:
+ return True
+ return False
+
+
+def _declared_paths_this_file_carries(*, entry: MigrationEntry, ledger: MigrationLedger, text: str) -> list[str]:
+ """The entry's declared narrowed paths that this file has a value at, as the migrated file spells them.
+
+ Presence is the whole predicate. No operation can express a value the new schema refuses, so
+ there is nothing to rehearse — and the engine is deliberately model-free, so it cannot tell a
+ value the narrowed domain rejects from one it accepts. What comes back is a list of keys for a
+ human to check, and it is reported as the *ledger* spells it, `*` included, so that a key the
+ user chose beneath an open mapping is no more rendered than a value would be.
+
+ Looked up at every spelling the material has ever had, and reported at the one the *end of this
+ replay* leaves: the entry is questioned against the text as it stands here, and the later `safe`
+ entries then rename the material before the file is written. The file the user is sent to check
+ is the one the run wrote, so naming the spelling that matched would name a key it no longer has.
+ """
+ spellings = declared_path_spellings(ledger=ledger, entry=entry)
+ if not spellings:
+ return []
+ present = document_paths(document=tomlkit.loads(text))
+ carried: list[str] = []
+ for spelling in spellings:
+ if not document_carries_path(paths=present, pattern=spelling):
+ continue
+ after_replay = spelling_after_replay(ledger=ledger, entry=entry, spelling=spelling)
+ if after_replay not in carried:
+ carried.append(after_replay)
+ return carried
+
+
def _apply_safe_entry(*, entry: MigrationEntry, text: str, steps: list[MigrationStep], blocked: list[BlockedEntry]) -> str:
"""Apply a `safe` entry and fold its operation outcomes into the plan, returning the new text.
diff --git a/pipelex/migration/ledger.py b/pipelex/migration/ledger.py
index eef8c4c79..8e2a6aa4b 100644
--- a/pipelex/migration/ledger.py
+++ b/pipelex/migration/ledger.py
@@ -94,6 +94,17 @@ class MigrationEntry(BaseModel):
such an entry declares its own removed paths instead."""
declared_removed_paths: list[str] = Field(default_factory=list[str])
+
+ declared_narrowed_paths: list[str] = Field(default_factory=list[str])
+ """The paths whose *value domain* this entry narrowed — a tightened bound, a lost numeric
+ member, anything no operation can repair. Spelled as the fingerprint at this entry's own
+ version records them, `*` included.
+
+ This is what an `unsafe` entry is *about*, and the engine questions the document for it. An
+ entry with operations is questioned by rehearsing them; an entry with none has nothing to
+ rehearse, so without a declaration it would be reported to nobody, ever — which is why one is
+ mandatory there. See `docs/migration-ledger.md` → "What an `unsafe` entry promises"."""
+
ops: list[MigrationOp] = Field(default_factory=list[MigrationOp])
"""May legitimately be empty, with `safety = "unsafe"`, for a change only a human can make."""
@@ -125,6 +136,35 @@ class MigrationEntry(BaseModel):
raise ValueError(msg)
return self
+ @model_validator(mode="after")
+ def check_narrowing_declarations_belong_to_an_unsafe_entry(self) -> Self:
+ if self.declared_narrowed_paths and self.safety is MigrationSafety.SAFE:
+ msg = (
+ f"entry '{self.id}': declared_narrowed_paths is for unsafe entries only — a narrowing a remap_value can "
+ f"repair is accounted for by that remap, and one it cannot repair is exactly what makes an entry unsafe"
+ )
+ raise ValueError(msg)
+ return self
+
+ @model_validator(mode="after")
+ def check_an_op_free_entry_declares_what_it_is_about(self) -> Self:
+ """An entry with neither operations nor a declaration can be reported to nobody, ever.
+
+ The engine questions an `unsafe` entry before reporting it, so that a user whose file is
+ already fine is not warned at every boot forever. Operations are questioned by rehearsing
+ them; a declaration is questioned by looking its paths up in the document. An entry
+ carrying neither answers "nothing to say" for every file there will ever be — the ledger
+ would accept it, and the user it exists for would never hear of it.
+ """
+ if not self.ops and not self.declared_narrowed_paths:
+ msg = (
+ f"entry '{self.id}': an entry with no operations declares the paths whose value domain narrowed — that "
+ f"declaration is the only thing the engine can question a document about, so an entry with neither "
+ f"operations nor a declaration is reported to nobody, ever"
+ )
+ raise ValueError(msg)
+ return self
+
class MigrationLedger(BaseModel):
"""One surface's whole ledger, as parsed from its checked-in TOML file."""
diff --git a/pipelex/migration/ledger_check.py b/pipelex/migration/ledger_check.py
index 5e67e20bf..4d517192e 100644
--- a/pipelex/migration/ledger_check.py
+++ b/pipelex/migration/ledger_check.py
@@ -21,6 +21,10 @@ The questions, each with its own failure kind:
declare material no fingerprint records, and may address nothing outside that declaration —
otherwise the flag, which exempts an entry from being accounted against a diff, would be a way
to opt out of accounting for a change that has one.
+- **Narrowing declarations.** An `unsafe` entry's `declared_narrowed_paths` is what the engine
+ questions a document about, so every declared path must be one the fingerprint at the entry's
+ own version records. A path that version does not have is questioned of every file and found in
+ none — the entry would be, once more, reported to nobody.
- **Convergence.** Replaying the whole ledger over each reference document must apply nothing
and return the very bytes it was given.
@@ -39,6 +43,7 @@ from pipelex.migration.engine import replay_ledger_over_text
from pipelex.migration.fingerprint import PATH_SEPARATOR, SurfaceFingerprint
from pipelex.migration.goldens import read_fingerprint_golden
from pipelex.migration.ledger import INITIAL_SCHEMA_VERSION, MigrationEntry, MigrationLedger, MigrationSafety, load_ledger
+from pipelex.migration.plan import BlockedEntryReason
from pipelex.migration.reserved import ReservedRegistry, derive_reserved_registry
from pipelex.migration.surfaces import Surface, SurfaceRegistry
from pipelex.migration.walk import EntryWalk, WalkedOp, op_source_path, walk_entry
@@ -56,6 +61,7 @@ class LedgerIssueKind(StrEnum):
RESERVED_PATH_REUSED = "reserved_path_reused"
RESERVED_VALUE_REUSED = "reserved_value_reused"
PRE_HISTORY_PATH_IS_RECORDED = "pre_history_path_is_recorded"
+ DECLARED_NARROWED_PATH_IS_ABSENT = "declared_narrowed_path_is_absent"
CONVERGENCE_BROKEN = "convergence_broken"
@@ -147,6 +153,7 @@ def _check_entries(
# The chain check above already named the missing link and its remedy. Checking the
# entry against a shape nobody snapshotted would blame the entry for that gap.
continue
+ issues.extend(_check_narrowing_declaration(surface_id=surface_id, entry=entry, after=after))
if entry.pre_history:
issues.extend(_check_pre_history_entry(surface_id=surface_id, entry=entry, chain=chain, before=before, after=after, reserved=reserved))
continue
@@ -156,6 +163,35 @@ def _check_entries(
return issues
+def _check_narrowing_declaration(*, surface_id: str, entry: MigrationEntry, after: SurfaceFingerprint) -> list[LedgerIssue]:
+ """Every declared narrowed path is one the entry's own version records.
+
+ A narrowing is a path that *survives* with a smaller value domain, so the fingerprint at the
+ entry's own version has it — that is what distinguishes a narrowing from a removal, which the
+ coverage gate accounts against the diff and which no declaration may stand in for. It is also
+ what makes the declaration answerable: the engine looks these paths up in the user's document,
+ and one no version records is looked for in every file and found in none, which puts the entry
+ right back where R9 found it — accepted by the accounting, reported to nobody.
+ """
+ issues: list[LedgerIssue] = []
+ for path in entry.declared_narrowed_paths:
+ if path in after.paths:
+ continue
+ issues.append(
+ LedgerIssue(
+ surface_id=surface_id,
+ kind=LedgerIssueKind.DECLARED_NARROWED_PATH_IS_ABSENT,
+ message=(
+ f"entry '{entry.id}' declares '{path}' as narrowed, but schema version {after.schema_version} has no such "
+ f"path — a narrowing keeps its path and shrinks what it accepts, so this is either a misspelling or a "
+ f"removal, and a removal is accounted for by the operation that removes it. Spell the path as the "
+ f"fingerprint at version {after.schema_version} records it, '{WILDCARD_SEGMENT}' segments included"
+ ),
+ )
+ )
+ return issues
+
+
def _check_pre_history_entry(
*,
surface_id: str,
@@ -543,6 +579,16 @@ def _check_convergence(*, surface: Surface, ledger: MigrationLedger) -> list[Led
back merely equal would mean something applied and was not reported — which is the one failure
a report cannot describe. Where the replay *did* report something, the changed bytes are that
finding's consequence and saying it twice adds nothing.
+
+ **One exemption, and it is the narrowest one that leaves R9's shape usable.** An entry reported
+ as `VALUE_DOMAIN_NARROWED` is reported because the document *sets* a path whose accepted values
+ the entry narrowed — presence, not violation, because the engine is model-free by design and
+ the ruling that put the declaration in the ledger is the same ruling that refused to thread a
+ model into the engine. A witness at the current schema sets the path like any healthy file
+ does, so without the exemption every op-free `unsafe` entry would fail this check and the only
+ remedy the coverage gate offers for a tightened bound could never be written. An `unsafe`
+ entry whose *operations* fire on a witness is a different matter and still fails: that says the
+ checked-in reference document carries retired material.
"""
issues: list[LedgerIssue] = []
for label, text in surface.reference_documents():
@@ -560,7 +606,7 @@ def _check_convergence(*, surface: Surface, ledger: MigrationLedger) -> list[Led
),
)
)
- blocked = [blocked_entry.entry_id for blocked_entry in replay.blocked]
+ blocked = [blocked_entry.entry_id for blocked_entry in replay.blocked if blocked_entry.reason is not BlockedEntryReason.VALUE_DOMAIN_NARROWED]
if blocked:
issues.append(
LedgerIssue(
diff --git a/pipelex/migration/material.py b/pipelex/migration/material.py
new file mode 100644
index 000000000..78c1a4a2b
--- /dev/null
+++ b/pipelex/migration/material.py
@@ -0,0 +1,219 @@
+"""The material an entry is about, and where a document keeps it.
+
+An `unsafe` entry is *questioned* before it is reported: the engine asks whether this file still
+carries the material the entry is about, so that a user whose file is already fine is never warned.
+Two things make that question harder than looking up the entry's own paths.
+
+- **An entry with no operations has no paths to look up.** That shape is the contract's own form
+ for "a change only a human can make" — a tightened numeric bound above all — and it is the only
+ remedy the coverage gate offers for a narrowing no operation can repair. Such an entry declares
+ the paths whose value domain narrowed, and the declaration is what gets looked up.
+- **Later entries move the material.** An `unsafe` entry is never applied, so a file it blocks
+ keeps the old spelling — while a *later* `safe` entry goes on renaming the tables around it, and
+ a file that has been migrated past this entry spells the same material differently. Questioning
+ only the entry's own spelling makes an entry that reported on the first run go silent on the
+ second, with the file still broken. So the material is traced forward through every later `safe`
+ entry, and the document is questioned at every spelling it has ever had — then *reported* at the
+ one the end of the replay leaves, because the file the user is sent to check is the one the run
+ wrote and the later entries have renamed the material by then.
+
+**Everything here is ledger arithmetic.** No fingerprint, no model, no filesystem — an operation
+says which path it moves where, and that is all this module reads. That is what lets the engine
+use it while staying the text-in, text-out function the gates replay.
+
+**A rehearsal may guess; an application may not.** Forward tracing is applied to `unsafe` entries
+only, because their operations are rehearsed against a copy and thrown away: the worst a wrong
+guess costs is one report too many. Extending it to `safe` entries would mean *writing* at a
+spelling the entry's author never wrote, which is a different promise and wants its own decision.
+
+See `docs/migration-ledger.md` → "What an `unsafe` entry promises".
+"""
+
+from collections.abc import Sequence
+
+from pipelex.migration.fingerprint import PATH_SEPARATOR
+from pipelex.migration.ledger import MigrationEntry, MigrationLedger, MigrationSafety
+from pipelex.migration.walk import op_source_path
+from pipelex.suggested_fix import DeleteKeyOp, DeleteTableOp, MigrationOp, MoveKeyOp, RemapValueOp, RenameTableKeyOp
+
+
+def unsafe_op_variants(*, ledger: MigrationLedger, entry: MigrationEntry) -> list[list[MigrationOp]]:
+ """Every spelling of an entry's operations the engine should rehearse against a document.
+
+ The entry's own operations come first and are what nearly every ledger ever needs: a further
+ variant appears only when a later `safe` entry renames or moves an ancestor of something this
+ entry addresses. Operations whose material a later entry *deletes* drop out of that variant —
+ a file migrated far enough for the material to be gone has nothing for the entry to say.
+
+ Only the operation's **source** is respelled. The rehearsal asks whether the entry would do
+ something, and only the source decides that: a move whose destination is occupied is reported
+ as a conflict, which counts as "would do something" exactly like an application does.
+ """
+ variants = [list(entry.ops)]
+ current = list(entry.ops)
+ for later in ledger.migration:
+ if later.to_schema_version <= entry.to_schema_version:
+ continue
+ if later.safety is MigrationSafety.UNSAFE:
+ # An unsafe entry is never applied, so it never gave anything a new spelling.
+ continue
+ respelled = _ops_through_ops(ops=current, later_ops=later.ops)
+ if respelled == current:
+ continue
+ current = respelled
+ if current and current not in variants:
+ variants.append(current)
+ return [variant for variant in variants if variant]
+
+
+def declared_path_spellings(*, ledger: MigrationLedger, entry: MigrationEntry) -> list[str]:
+ """Every spelling a document might use for the paths an entry declares narrowed.
+
+ Three sources, in the order a reader would think of them: the declaration itself, which is
+ spelled as the fingerprint at the entry's own version records it; that path traced *back*
+ through the entry's own operations, because an `unsafe` entry is never applied and a file it
+ blocks therefore keeps the previous version's spelling; and the path traced *forward* through
+ every later `safe` entry, for a file that has been migrated past this one.
+ """
+ spellings: list[str] = []
+ for declared in entry.declared_narrowed_paths:
+ for spelling in _spellings_of_one_declared_path(ledger=ledger, entry=entry, declared=declared):
+ if spelling not in spellings:
+ spellings.append(spelling)
+ return spellings
+
+
+def spelling_after_replay(*, ledger: MigrationLedger, entry: MigrationEntry, spelling: str) -> str:
+ """The last spelling the material has once the rest of the replay has run.
+
+ `declared_path_spellings` answers "which spellings should I look for"; this answers "what will
+ the file the user is sent to check call it". The two differ whenever a later `safe` entry
+ renames the material, because an `unsafe` entry is questioned against the text as it stands
+ when the replay reaches it and those later entries then rename it out from under the report.
+
+ Where a later entry *retires* the material there is no later spelling, and the one the file
+ carries now is the last there was — the same stop `_spellings_of_one_declared_path` makes when
+ it traces forward.
+ """
+ current = spelling
+ for later in ledger.migration:
+ if later.to_schema_version <= entry.to_schema_version or later.safety is MigrationSafety.UNSAFE:
+ continue
+ moved = _path_through_ops(path=current, ops=later.ops)
+ if moved is None:
+ break
+ current = moved
+ return current
+
+
+def _spellings_of_one_declared_path(*, ledger: MigrationLedger, entry: MigrationEntry, declared: str) -> list[str]:
+ spellings = [declared]
+
+ before_entry = declared
+ for op in reversed(entry.ops):
+ before_entry = _path_before_op(path=before_entry, op=op)
+ if before_entry not in spellings:
+ spellings.append(before_entry)
+
+ current = declared
+ for later in ledger.migration:
+ if later.to_schema_version <= entry.to_schema_version or later.safety is MigrationSafety.UNSAFE:
+ continue
+ moved = _path_through_ops(path=current, ops=later.ops)
+ if moved is None:
+ # A later entry retires the material. A file migrated that far no longer carries it,
+ # and every earlier spelling stays in the list for the files that have not.
+ break
+ current = moved
+ if current not in spellings:
+ spellings.append(current)
+ return spellings
+
+
+def _ops_through_ops(*, ops: Sequence[MigrationOp], later_ops: Sequence[MigrationOp]) -> list[MigrationOp]:
+ respelled: list[MigrationOp] = []
+ for op in ops:
+ moved = _op_through_ops(op=op, later_ops=later_ops)
+ if moved is not None:
+ respelled.append(moved)
+ return respelled
+
+
+def _op_through_ops(*, op: MigrationOp, later_ops: Sequence[MigrationOp]) -> MigrationOp | None:
+ source = op_source_path(op=op)
+ moved = _path_through_ops(path=source, ops=later_ops)
+ if moved is None:
+ return None
+ if moved == source:
+ return op
+ return _with_source_path(op=op, source_path=moved)
+
+
+def _path_through_ops(*, path: str, ops: Sequence[MigrationOp]) -> str | None:
+ current = path
+ for op in ops:
+ stepped = _path_through_op(path=current, op=op)
+ if stepped is None:
+ return None
+ current = stepped
+ return current
+
+
+def _path_through_op(*, path: str, op: MigrationOp) -> str | None:
+ """Where `path` is after `op` ran: unchanged, respelled, or `None` when the op retires it."""
+ match op:
+ case RemapValueOp():
+ return path
+ case DeleteKeyOp() | DeleteTableOp():
+ return None if _is_at_or_under(path=path, ancestor=op_source_path(op=op)) else path
+ case RenameTableKeyOp():
+ return _respelled(path=path, source=op_source_path(op=op), destination=_joined(segments=[*op.table_path, op.new_key]))
+ case MoveKeyOp():
+ return _respelled(path=path, source=op_source_path(op=op), destination=_joined(segments=[*op.new_table_path, op.new_key]))
+
+
+def _path_before_op(*, path: str, op: MigrationOp) -> str:
+ """Where `path` was before `op` ran — the inverse of a rename or a move, and nothing else.
+
+ A delete has no inverse and needs none: this walks back from a path the fingerprint at the
+ entry's own version records, so nothing the entry deleted can be on the way.
+ """
+ match op:
+ case RemapValueOp() | DeleteKeyOp() | DeleteTableOp():
+ return path
+ case RenameTableKeyOp():
+ return _respelled(path=path, source=_joined(segments=[*op.table_path, op.new_key]), destination=op_source_path(op=op))
+ case MoveKeyOp():
+ return _respelled(path=path, source=_joined(segments=[*op.new_table_path, op.new_key]), destination=op_source_path(op=op))
+
+
+def _with_source_path(*, op: MigrationOp, source_path: str) -> MigrationOp:
+ """The same operation, addressing `source_path` instead of what it addressed.
+
+ Built with `model_copy` rather than by revalidation on purpose: the result is a rehearsal
+ object derived from an already-legal operation by substituting one path for another, and it is
+ never written to a ledger, never serialized, and never applied to a file.
+ """
+ segments = source_path.split(PATH_SEPARATOR)
+ match op:
+ case DeleteTableOp():
+ return op.model_copy(update={"table_path": segments})
+ case DeleteKeyOp() | RenameTableKeyOp() | MoveKeyOp() | RemapValueOp():
+ return op.model_copy(update={"table_path": segments[:-1], "key": segments[-1]})
+
+
+def _respelled(*, path: str, source: str, destination: str) -> str:
+ if path == source:
+ return destination
+ prefix = source + PATH_SEPARATOR
+ if path.startswith(prefix):
+ return destination + PATH_SEPARATOR + path[len(prefix) :]
+ return path
+
+
+def _is_at_or_under(*, path: str, ancestor: str) -> bool:
+ return path == ancestor or path.startswith(ancestor + PATH_SEPARATOR)
+
+
+def _joined(*, segments: Sequence[str]) -> str:
+ return PATH_SEPARATOR.join(segments)
diff --git a/pipelex/migration/plan.py b/pipelex/migration/plan.py
index 689cea0ac..ab12a8d38 100644
--- a/pipelex/migration/plan.py
+++ b/pipelex/migration/plan.py
@@ -56,6 +56,12 @@ class BlockedEntryReason(StrEnum):
"""An operation's destination is already occupied — typically because the user hand-fixed
part of their file — so applying it would choose on their behalf."""
+ VALUE_DOMAIN_NARROWED = "value_domain_narrowed"
+ """The file sets a path whose accepted values the entry narrowed, and no operation can repair
+ a value. Weaker than `UNSAFE`, and deliberately so: this says *check this key*, not *this file
+ has the old shape*. The engine is model-free, so it knows the file sets the path and not
+ whether the value it holds is one the new schema refuses."""
+
class BlockedEntry(BaseModel):
"""An entry that this file needs and the run would not apply."""
@@ -75,13 +81,47 @@ class BlockedEntry(BaseModel):
"""Operations of this entry that did apply before the conflict was found. Always empty for
an `unsafe` entry, which is rehearsed on a copy and never written."""
+ narrowed_paths: list[str] = Field(default_factory=list[str])
+ """The entry's declared narrowed paths that this file has something at, as the *ledger*
+ spells them — `levels.*` rather than the user's own `levels.my_package`, so that no key a user
+ chose is rendered any more than a value is — and at the spelling the file this run wrote
+ carries, since that is the file being sent back to be checked.
+
+ Keys to check by hand, never a list of errors: telling one from the other needs the model, and
+ the engine has none by design."""
+
class FileBlockedReason(StrEnum):
- """Why a file could not be processed at all."""
+ """Why a file could not be processed at all.
+
+ One member per *state the file is in*, because that is what decides what the user does next —
+ not one per exception the run happened to catch. Four of the five leave the file exactly as it
+ was found; the fifth is the one that does not, and it says so.
+ """
+
+ UNREADABLE = "unreadable"
+ """The file is there and its bytes could not be read. Nothing was written."""
UNPARSEABLE = "unparseable"
+ """The file was read and is not valid UTF-8, or not valid TOML. Nothing was written."""
+
UNWRITABLE = "unwritable"
+ """The file needed a change and the run could not make it — the backup would not go down, or
+ the replacement would not. The file is exactly as it was found."""
+
CHANGED_DURING_RUN = "changed_during_run"
+ """The file was removed or edited between the read and the write, so the run refused to write
+ over work it had not seen. The file is whatever that other writer left."""
+
+ STATE_UNCERTAIN = "state_uncertain"
+ """The write could not be confirmed: the transaction could not describe what it left behind,
+ and the file does not hold what this run wrote. The one reason that cannot promise the file is
+ as it was found, which is why it is not folded into `UNWRITABLE` — the user's next move is to
+ compare the file against the rescue copy the plan names, not to fix a permission and re-run.
+
+ For the single-file commit the runner performs it is also the *only* transaction failure that
+ reaches the plan at all: a replacement that fails re-raises its own `OSError`, because rolling
+ back nothing is trivially complete."""
class UnexplainedPath(BaseModel):
diff --git a/pipelex/migration/runner.py b/pipelex/migration/runner.py
index 819ccb145..da24674df 100644
--- a/pipelex/migration/runner.py
+++ b/pipelex/migration/runner.py
@@ -18,7 +18,7 @@ from pathlib import Path
from tomlkit.exceptions import TOMLKitError
from pipelex import log
-from pipelex.migration.backup import prune_backups_except, write_backup
+from pipelex.migration.backup import RescuedBackup, WrittenBackup, keep_backup_for_rescue, prune_backups_except, write_backup
from pipelex.migration.engine import replay_ledger_over_text
from pipelex.migration.ledger import MigrationLedger, load_ledger_cached
from pipelex.migration.plan import FileBlockedReason, MigrationPlan, MigrationReport
@@ -30,7 +30,12 @@ from pipelex.pipeline.fixes.file_transaction import FileSnapshot, PendingFileUpd
def migrate_file(*, surface: Surface, ledger: MigrationLedger, file_path: Path, dry_run: bool, moment: datetime) -> MigrationPlan:
"""Replay one surface's ledger over one file, writing it only when something applied."""
try:
- snapshot = read_file_snapshot(file_path)
+ # Through a symlink to the file it names, the way the `.mthds` fix loop already resolves its
+ # own targets: a configuration file symlinked out of a dotfiles directory is that directory's
+ # file, and an atomic replace of the link path would delete the link and leave the real file
+ # unmigrated. The plan keeps naming the path the walk found; the backup lands beside the
+ # file that was actually rewritten, which is where a copy of it belongs.
+ snapshot = read_file_snapshot(file_path.resolve())
except FileNotFoundError:
return _blocked_plan(
surface_id=surface.surface_id,
@@ -42,7 +47,7 @@ def migrate_file(*, surface: Surface, ledger: MigrationLedger, file_path: Path,
return _blocked_plan(
surface_id=surface.surface_id,
file_path=file_path,
- reason=FileBlockedReason.UNWRITABLE,
+ reason=FileBlockedReason.UNREADABLE,
detail=f"the file could not be read: {exc.strerror or exc}",
)
@@ -95,7 +100,7 @@ def _write_migrated_file(*, plan: MigrationPlan, snapshot: FileSnapshot, new_con
around it.
"""
try:
- backup_path = write_backup(snapshot=snapshot, moment=moment)
+ backup = write_backup(snapshot=snapshot, moment=moment)
except OSError as exc:
return plan.model_copy(
update={
@@ -109,50 +114,89 @@ def _write_migrated_file(*, plan: MigrationPlan, snapshot: FileSnapshot, new_con
except FixWriteConflictError as exc:
# The primitive refused before touching the target, so the directory is as it was — minus
# the copy this run just made, which has nothing to back up.
- _discard_backup(backup_path=backup_path)
+ _discard_backup(backup=backup)
return plan.model_copy(update={"blocked_reason": FileBlockedReason.CHANGED_DURING_RUN, "blocked_detail": str(exc)})
except OSError as exc:
- _discard_backup(backup_path=backup_path)
+ _discard_backup(backup=backup)
return plan.model_copy(
update={"blocked_reason": FileBlockedReason.UNWRITABLE, "blocked_detail": f"the file could not be written: {exc.strerror or exc}"}
)
except FixTransactionError as exc:
- # The primitive raises this when it cannot vouch for the state it leaves behind — a rollback
- # that did not complete, or a replacement that landed but whose temp files could not be
- # removed. The exception cannot say which; the file can, so ask it. Whatever the answer,
- # the backup stays: it is the one copy of the original whose provenance is certain, and a
- # state the primitive cannot vouch for is exactly the state a backup exists for.
+ # For the single-file commit this runner performs, a replace that fails re-raises its own
+ # `OSError` or `FixWriteConflictError` — a rollback of nothing is trivially complete — so
+ # the only `FixTransactionError` that reaches here is the one raised *after* the target was
+ # replaced, when the temporary files could not be removed. The write landed; whether it is
+ # still what landed is the open question, and the file answers it.
if not _carries(path=snapshot.path, content=new_content):
+ kept = _keep_the_original(backup=backup, path=snapshot.path, moment=moment)
return plan.model_copy(
update={
- "blocked_reason": FileBlockedReason.UNWRITABLE,
- "blocked_detail": f"the file could not be written, and its state is uncertain — the original is kept at '{backup_path}': {exc}",
- "backup_path": backup_path,
+ "blocked_reason": FileBlockedReason.STATE_UNCERTAIN,
+ "blocked_detail": (
+ f"the write could not be confirmed: the file does not hold what this run wrote, and the transaction could not "
+ f"say what it left behind — {_whereabouts_of(kept=kept)}: {exc}"
+ ),
+ "backup_path": kept.path,
}
)
log.warning(f"'{snapshot.path}' was migrated, but the write left something behind: {exc}")
try:
- prune_backups_except(path=snapshot.path, keep=backup_path)
+ prune_backups_except(path=snapshot.path, keep=backup.path)
except OSError as exc:
# An older backup that would not go is a housekeeping failure on a file that is already
# migrated and already backed up. Not the plan's to report as a failure of the file.
log.warning(
- f"'{snapshot.path}' was migrated and backed up to '{backup_path}', but an older backup could not be pruned: {exc.strerror or exc}"
+ f"'{snapshot.path}' was migrated and backed up to '{backup.path}', but an older backup could not be pruned: {exc.strerror or exc}"
)
- return plan.model_copy(update={"backup_path": backup_path, "was_written": True})
+ return plan.model_copy(update={"backup_path": backup.path, "was_written": True})
-def _discard_backup(*, backup_path: Path) -> None:
+def _keep_the_original(*, backup: WrittenBackup, path: Path, moment: datetime) -> RescuedBackup:
+ """Take this run's backup out of the rotation, so the next successful run cannot prune it.
+
+ Only a copy this run made: another run's backup of the same file is that run's to name and to
+ move, and renaming it here would make its report point at a file that no longer exists. That
+ copy holds the original all the same, so it is still what the report names — as a copy still in
+ the rotation, which is what `was_rescued` says.
+ """
+ if not backup.was_created:
+ return RescuedBackup(path=backup.path, was_rescued=False)
+ return keep_backup_for_rescue(path=path, backup_path=backup.path, moment=moment)
+
+
+def _whereabouts_of(*, kept: RescuedBackup) -> str:
+ """How to tell the user where the one certain copy of their file is, and how long it will be there.
+
+ A copy that left the `.bak.` rotation is safe until the user removes it, and the report says so.
+ One that could not be moved is a copy the next successful run of this file will prune, so the
+ report asks for the one thing that saves it — taking it now.
+ """
+ if kept.was_rescued:
+ return f"the copy taken before the migration is kept at '{kept.path}'"
+ return (
+ f"a copy of the file as it was is at '{kept.path}', and this run could not take it out of the way of pruning — "
+ f"copy it aside before the next successful run of this file"
+ )
+
+
+def _discard_backup(*, backup: WrittenBackup) -> None:
"""Remove the copy this run made of a file it then did not rewrite.
+ Only the copy this run made. A backup already sitting at that name belongs to a concurrent run
+ of the same file, and it is a copy of an older state — the original, if anything is; deleting
+ it here would destroy the one thing backups exist for on the way to reporting a write that
+ never happened.
+
A copy that will not go is a stray file beside an untouched original — worth a warning, never
worth an exception crossing the per-file boundary and aborting the siblings.
"""
+ if not backup.was_created:
+ return
try:
- backup_path.unlink(missing_ok=True)
+ backup.path.unlink(missing_ok=True)
except OSError as exc:
- log.warning(f"the backup '{backup_path}' was made for a write that did not happen and could not be removed: {exc.strerror or exc}")
+ log.warning(f"the backup '{backup.path}' was made for a write that did not happen and could not be removed: {exc.strerror or exc}")
def _carries(*, path: Path, content: str) -> bool:
diff --git a/pipelex/pipeline/fixes/applier.py b/pipelex/pipeline/fixes/applier.py
index 928a10395..4614b2377 100644
--- a/pipelex/pipeline/fixes/applier.py
+++ b/pipelex/pipeline/fixes/applier.py
@@ -29,7 +29,7 @@ from pipelex_tools import format_mthds
from pydantic import BaseModel, ConfigDict, Field
from tomlkit import TOMLDocument
from tomlkit.container import Container, OutOfOrderTableProxy
-from tomlkit.items import AbstractTable, AoT, Item, Table
+from tomlkit.items import AbstractTable, AoT, Item, Key, SingleKey, Table
from pipelex.base_exceptions import PipelexUnexpectedError
from pipelex.suggested_fix import (
@@ -115,24 +115,24 @@ def _resolve_table(*, toml_doc: TOMLDocument, table_path: list[str]) -> dict[str
def _rename_key_in_place(*, parent_table: dict[str, Any], key: str, new_key: str) -> None:
"""Rename ``key`` to ``new_key`` in a resolved table node, preserving position and comments.
- tomlkit exposes no public position-preserving rename; ``Container._replace`` is its own
- internal primitive (what ``__setitem__`` uses to re-home an existing key). It swaps the key in
- ``_body`` in place and re-renders the table header, so the renamed table keeps its position
- among siblings and its comments — a ``del`` + re-add would append it to the bottom of the
- parent (the old fixer's reordering bug). The golden byte-compare tests are the CI tripwire if
- a tomlkit bump ever changes this.
+ tomlkit exposes no public position-preserving rename, so ``_replace_key_in_container`` renames
+ the container's body entries in place: the key keeps its slot among its siblings and its
+ comments, where a ``del`` + re-add would append it to the bottom of the parent (the old
+ fixer's reordering bug). The golden byte-compare tests are the CI tripwire if a tomlkit bump
+ ever changes what that body looks like.
⚠ **A rename leaves the node's raw ``dict`` storage stale for a value that is not a Table.**
- ``_replace_at`` leaves the old key in the dict and only writes the new one into it inside its
- table branch, so everything tomlkit renders or looks up stays right (``dumps``, ``in``, ``[]``,
+ A ``Table`` and the ``Container`` inside it are two dict-like objects each holding their own
+ copy of the key set, and a rename reaching the container has no way to reach the parent
+ table's copy. So everything tomlkit renders or looks up stays right (``dumps``, ``in``, ``[]``,
``.get()`` all read the authoritative body) while ``dict.__delitem__`` — which
``Container.remove`` calls — cannot find the new key. Addressing a renamed **scalar or
inline-table** key again in the same DOM therefore raises ``KeyError`` from inside the library.
The ``.mthds`` fix path is unaffected because it renames ``[pipe.*]`` tables, which take the
- branch tomlkit maintains; configuration migration re-reads the document between operations
- that applied, for exactly this reason. Both facts are pinned by
+ branch tomlkit does keep in step; configuration migration re-reads the document between
+ operations that applied, for its own reasons, and is unaffected too. Both facts are pinned by
``tests/unit/pipelex/pipeline/fixes/test_fix_applier_rename_dom_consistency.py``, which is the
- tripwire if a tomlkit bump ever changes it.
+ tripwire if this is ever repaired across every facade.
``_resolve_table`` hands back one of three dict-like shapes, each with its own route to the
``Container`` that owns the key:
@@ -170,9 +170,99 @@ def _rename_key_in_place(*, parent_table: dict[str, Any], key: str, new_key: str
def _replace_key_in_container(*, container: Container, key: str, new_key: str) -> None:
- item = cast("Item", container[key])
- container._replace(key, new_key, item) # pyright: ignore[reportPrivateUsage]
- _refresh_table_headers(item=item)
+ """Rename one key of a container in place — every chunk of it, and nothing beyond the name.
+
+ A rename changes a name. tomlkit's own re-key primitive, ``Container._replace``, does three
+ further things on the way, and each of them is a defect on a **dotted** key — the ordinary way
+ to write a one-line entry of a section (``package_log_levels.pipelex = "INFO"``), and one no
+ formatter rewrites into anything else:
+
+ - **Dotted-ness is dropped.** It is not a property of the name but of how the line was
+ written: the key owns a super-table and carries a flag the parser sets, and the renderer
+ reads that flag to choose between a dotted assignment and a ``[a.k]`` header. Rebuilt as a
+ plain key, ``k.x = 1`` came back out as a header — and a header **absorbs every scalar that
+ follows it in the same table**, so a neighbouring ``m = 3`` silently became ``a.kk.m``, with
+ an ``applied`` verdict and nothing said. Renamed at an inner segment the chain re-rendered
+ at the document *root*, taking the whole subtree out of the table it lived in.
+ - **Chunks are collapsed.** A key written as several dotted lines (``k.x = 1`` then
+ ``k.y = 2``) is several body entries under one key, and the primitive keeps the first, nulls
+ the rest, and rebuilds one merged value from the container's dict facade — which for this
+ shape holds only the last chunk.
+ - **A blank line is injected.** The primitive appends a cosmetic newline to a replaced table.
+ That is right for a block table it is about to re-home, and wrong for an assignment.
+
+ Refusing the layout with a ``CONFLICT`` was the alternative the migration plan named, and it
+ is the wrong trade: a rename has exactly one correct answer on a dotted key (``kk.x = 1``),
+ and refusing would strand a configuration migration's table renames on any file that happens
+ to be written this way. This is a rendering defect with a root cause, like the stale
+ ``display_name`` on a table split across chunks, and it is fixed at the root for the same reason.
+
+ Renaming the body entries directly is also what keeps position and comments — the property
+ ``_replace`` was reached for in the first place — since nothing is removed or re-appended.
+ """
+ renamed_positions = [
+ (position, body_key) for position, (body_key, _) in enumerate(container.body) if body_key is not None and body_key.key == key
+ ]
+ if not renamed_positions:
+ # ``_rename_key_in_place`` checked membership through the dict facade; an empty body means
+ # the facade and the body disagree, which is a bug here rather than a document to migrate.
+ msg = f"key '{key}' is in the container's facade but in none of its body entries during rename — applier bug"
+ raise PipelexUnexpectedError(msg)
+
+ for position, body_key in renamed_positions:
+ _, body_item = container.body[position]
+ container.body[position] = (_renamed_key(previous=body_key, new_key=new_key), body_item)
+ _refresh_table_headers(item=body_item)
+
+ # The side indexes hold one entry per name, and tomlkit keeps the last chunk appended.
+ last_position, last_body_key = renamed_positions[-1]
+ _rehome_key_indexes(
+ container=container,
+ key=key,
+ renamed=_renamed_key(previous=last_body_key, new_key=new_key),
+ item=container.body[last_position][1],
+ )
+
+
+def _renamed_key(*, previous: Key, new_key: str) -> SingleKey:
+ """A fresh key under the new name, carrying forward the one property a rename must not lose.
+
+ Everything else — quoting, separator — is left to tomlkit's own construction, so renaming an
+ ordinary key yields exactly the key the library would have built for it.
+ """
+ renamed = SingleKey(new_key)
+ if previous.is_dotted():
+ # There is no public way to say "dotted": the flag is private, and tomlkit's own parser
+ # sets it exactly this way in ``Container._handle_dotted_key``.
+ renamed._dotted = True # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+ return renamed
+
+
+def _rehome_key_indexes(*, container: Container, key: str, renamed: SingleKey, item: Item) -> None:
+ """Move a renamed key in the two side indexes a ``Container`` carries beside its body.
+
+ The body is what tomlkit renders from; beside it sit a private map from key to body position,
+ which the public accessors read, and the raw ``dict`` storage a ``Container`` inherits, which
+ ``len()``, iteration and ``.get()`` read.
+
+ Both are updated exactly as ``Container._replace_at`` updates them, **including its asymmetry**
+ — the raw storage gets the new name back only for a ``Table``. That asymmetry is the known
+ staleness ``_rename_key_in_place`` documents and ``test_fix_applier_rename_dom_consistency.py``
+ pins, and it is deliberately not repaired here: the storage this function can reach is the
+ container's, while a nested rename's stale facade is the parent ``Table``'s own — a different
+ dict, holding items rather than values. Repairing one of the two would move the inconsistency
+ rather than end it, so the whole set of facades is one deliberate pass, not a rider on this one.
+ """
+ container._map[renamed] = container._map.pop(SingleKey(key)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+ # ``dict`` explicitly on both lines — a ``Container`` is a ``MutableMapping`` *and* a ``dict``,
+ # so its own ``pop`` and ``[] =`` route through ``Container.__getitem__`` / ``__delitem__``,
+ # which read and rewrite the very body this function has just finished renaming. The stubs
+ # type the inherited members as ``dict[Unknown, Unknown]``, hence the narrow ignores.
+ dict.pop(container, key, None) # pyright: ignore[reportUnknownMemberType, reportUnknownArgumentType]
+ if isinstance(item, Table):
+ # The raw storage holds one entry per name, and for a key written as several chunks
+ # tomlkit keeps the last one appended — which is the item this is handed.
+ dict.__setitem__(container, renamed.key, item.value) # noqa: PLC2801 # pyright: ignore[reportUnknownMemberType]
def _as_tomlkit_value(value: TomlValue | None) -> Any:
diff --git a/tests/unit/pipelex/migration/conftest.py b/tests/unit/pipelex/migration/conftest.py
index 4d7a06d26..644b87b77 100644
--- a/tests/unit/pipelex/migration/conftest.py
+++ b/tests/unit/pipelex/migration/conftest.py
@@ -37,6 +37,7 @@ def build_entry() -> EntryBuilder:
breaking: bool = True,
pre_history: bool = False,
declared_removed_paths: list[str] | None = None,
+ declared_narrowed_paths: list[str] | None = None,
) -> MigrationEntry:
return MigrationEntry(
id=f"{surface_id}@{to_schema_version}",
@@ -49,6 +50,7 @@ def build_entry() -> EntryBuilder:
guidance=guidance,
pre_history=pre_history,
declared_removed_paths=declared_removed_paths or [],
+ declared_narrowed_paths=declared_narrowed_paths or [],
ops=ops,
)
diff --git a/tests/unit/pipelex/migration/test_coverage_gate.py b/tests/unit/pipelex/migration/test_coverage_gate.py
index 9504611e8..1bbb0f574 100644
--- a/tests/unit/pipelex/migration/test_coverage_gate.py
+++ b/tests/unit/pipelex/migration/test_coverage_gate.py
@@ -64,6 +64,13 @@ class _SchemaTwoEnumMemberGone(BaseModel):
tier: _TierWithoutBasic = _TierWithoutBasic.PREMIUM
+class _SchemaTwoRenamedWithMemberGone(BaseModel):
+ """`label` renamed *and* a spelling of `tier` retired — an entry that is about the rename only."""
+
+ title: str = "hello"
+ tier: _TierWithoutBasic = _TierWithoutBasic.PREMIUM
+
+
class _SchemaTwoTierRenamedAndMemberGone(BaseModel):
"""`tier` renamed to `level` *and* `basic` dropped, in the same schema version."""
@@ -357,6 +364,7 @@ class TestEntryAccounting:
assert "titel" in issues[0].message
def test_a_removal_with_no_operation_accounting_for_it_is_caught(self, tmp_path: Path) -> None:
+ """An entry that is genuinely about something else still owes the version's removals."""
entry = f"""
[[migration]]
id = "{SURFACE_ID}@2"
@@ -366,6 +374,7 @@ breaking = true
safety = "unsafe"
title = "Drop label"
description = "The key is gone, and this entry forgot to say so."
+declared_narrowed_paths = ["tier"]
"""
_write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=entry)
_snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)
@@ -533,7 +542,7 @@ table_path = ["label"]
class TestEnumAccounting:
- def _entry_with_ops(self, *, safety: str, ops: str) -> str:
+ def _entry_with_ops(self, *, safety: str, ops: str, declared_narrowed_paths: str = "") -> str:
return f"""
[[migration]]
id = "{SURFACE_ID}@2"
@@ -543,6 +552,7 @@ breaking = true
safety = "{safety}"
title = "Retire the basic tier"
description = "The spelling changed."
+{declared_narrowed_paths}
{ops}
"""
@@ -618,13 +628,35 @@ mapping = { basic = "standard" }
_snapshot(migration_dir=tmp_path, config_model=_SchemaTwoEnumMemberGone, schema_version=2)
assert check_surface(surface=_surface(config_model=_SchemaTwoEnumMemberGone), migration_dir=tmp_path) == []
- def test_an_unsafe_entry_may_leave_a_removed_member_unremapped(self, tmp_path: Path) -> None:
+ def test_an_unsafe_entry_that_declares_the_path_may_leave_a_removed_member_unremapped(self, tmp_path: Path) -> None:
"""`unsafe` means reported and never applied, so it is allowed to describe what no operation can do."""
- _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=self._entry_with_ops(safety="unsafe", ops=""))
+ entry = self._entry_with_ops(safety="unsafe", ops="", declared_narrowed_paths='declared_narrowed_paths = ["tier"]')
+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=entry)
_snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)
_snapshot(migration_dir=tmp_path, config_model=_SchemaTwoEnumMemberGone, schema_version=2)
assert check_surface(surface=_surface(config_model=_SchemaTwoEnumMemberGone), migration_dir=tmp_path) == []
+ def test_an_unsafe_entry_that_names_no_path_does_not_account_for_the_lost_member(self, tmp_path: Path) -> None:
+ """R9, on the enum half: the word `unsafe` satisfies a reader and reaches no user.
+
+ The entry has an operation, so it is not the op-free shape the parser refuses — and the
+ operation is about something else entirely, so rehearsing it says nothing to the user
+ whose `tier = "basic"` the new schema rejects.
+ """
+ ops = """
+[[migration.ops]]
+kind = "rename_table_key"
+table_path = []
+key = "label"
+new_key = "title"
+"""
+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=self._entry_with_ops(safety="unsafe", ops=ops))
+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)
+ _snapshot(migration_dir=tmp_path, config_model=_SchemaTwoRenamedWithMemberGone, schema_version=2)
+ issues = check_surface(surface=_surface(config_model=_SchemaTwoRenamedWithMemberGone), migration_dir=tmp_path)
+ assert _kinds(issues) == [CoverageIssueKind.ENUM_MEMBER_NOT_REMAPPED]
+ assert "declared_narrowed_paths" in issues[0].message
+
def test_a_member_lost_inside_a_list_cannot_be_remapped_and_the_gate_says_so(self, tmp_path: Path) -> None:
"""A remap rewrites a string value, and the value here is a list.
@@ -690,7 +722,7 @@ class TestValueDomainNarrowing:
reject a file that was valid the day before with a green gate behind it.
"""
- def _entry_with_ops(self, *, safety: str, ops: str) -> str:
+ def _entry_with_ops(self, *, safety: str, ops: str, declared_narrowed_paths: str = "") -> str:
return f"""
[[migration]]
id = "{SURFACE_ID}@2"
@@ -700,6 +732,7 @@ breaking = true
safety = "{safety}"
title = "Narrow what a value may be"
description = "The domain shrank."
+{declared_narrowed_paths}
{ops}
"""
@@ -807,15 +840,52 @@ mapping = { auto = "unbounded" }
assert _kinds(issues) == [CoverageIssueKind.VALUE_DOMAIN_NARROWED]
assert "its type went from 'int | literal' to 'literal'" in issues[0].message
- def test_an_unsafe_entry_may_leave_a_narrowing_unremapped(self, tmp_path: Path) -> None:
+ def test_an_unsafe_entry_that_declares_the_narrowing_leaves_it_unremapped(self, tmp_path: Path) -> None:
"""For a tightened numeric bound this is the only remedy there is: no mapping can enumerate
the values a bound retires, so the migration is reported to the user and never applied.
"""
- _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=self._entry_with_ops(safety="unsafe", ops=""))
+ entry = self._entry_with_ops(safety="unsafe", ops="", declared_narrowed_paths='declared_narrowed_paths = ["retries"]')
+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=entry)
_snapshot(migration_dir=tmp_path, config_model=_BoundedOne, schema_version=1)
_snapshot(migration_dir=tmp_path, config_model=_BoundedTightened, schema_version=2)
assert check_surface(surface=_surface(config_model=_BoundedTightened), migration_dir=tmp_path) == []
+ def test_an_unsafe_entry_that_declares_nothing_about_its_narrowing_is_refused(self, tmp_path: Path) -> None:
+ """R9 — `unsafe` on its own accounts for nothing, because the engine reports it to nobody.
+
+ The word satisfies a reader; the declaration is what the engine can question a user's file
+ about. An entry that says `unsafe` and names no path passes this gate while every user
+ whose value the new schema refuses is left with a failing boot and no message.
+ """
+ ops = """
+[[migration.ops]]
+kind = "rename_table_key"
+table_path = []
+key = "label"
+new_key = "title"
+"""
+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=self._entry_with_ops(safety="unsafe", ops=ops))
+ _snapshot(migration_dir=tmp_path, config_model=_BoundedOne, schema_version=1)
+ _snapshot(migration_dir=tmp_path, config_model=_BoundedTightenedAndRenamed, schema_version=2)
+ issues = check_surface(surface=_surface(config_model=_BoundedTightenedAndRenamed), migration_dir=tmp_path)
+ assert _kinds(issues) == [CoverageIssueKind.VALUE_DOMAIN_NARROWED]
+ assert "declared_narrowed_paths" in issues[0].message
+
+ def test_an_unsafe_entry_declaring_its_narrowing_beside_a_rename_is_green(self, tmp_path: Path) -> None:
+ """The same entry, with the one line that makes it reach the user it is written for."""
+ ops = """
+[[migration.ops]]
+kind = "rename_table_key"
+table_path = []
+key = "label"
+new_key = "title"
+"""
+ entry = self._entry_with_ops(safety="unsafe", ops=ops, declared_narrowed_paths='declared_narrowed_paths = ["retries"]')
+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=entry)
+ _snapshot(migration_dir=tmp_path, config_model=_BoundedOne, schema_version=1)
+ _snapshot(migration_dir=tmp_path, config_model=_BoundedTightenedAndRenamed, schema_version=2)
+ assert check_surface(surface=_surface(config_model=_BoundedTightenedAndRenamed), migration_dir=tmp_path) == []
+
def test_a_pre_history_entry_cannot_hide_a_narrowing_either(self, tmp_path: Path) -> None:
"""The flag exempts an entry from accounting, so it must not be a way past this one."""
_write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=_pre_history_entry())
diff --git a/tests/unit/pipelex/migration/test_ledger_check.py b/tests/unit/pipelex/migration/test_ledger_check.py
index 8a11282c6..b029c14fb 100644
--- a/tests/unit/pipelex/migration/test_ledger_check.py
+++ b/tests/unit/pipelex/migration/test_ledger_check.py
@@ -185,7 +185,7 @@ def _snapshot(*, migration_dir: Path, config_model: type[BaseModel], schema_vers
)
-def _entry(*, ops: str, to_schema_version: int = 2, safety: str = "safe", pre_history: str = "") -> str:
+def _entry(*, ops: str, to_schema_version: int = 2, safety: str = "safe", pre_history: str = "", declared_narrowed: str = "") -> str:
return f"""
[[migration]]
id = "{SURFACE_ID}@{to_schema_version}"
@@ -196,6 +196,7 @@ safety = "{safety}"
title = "Reshape the synthetic surface"
description = "The surface changed shape."
{pre_history}
+{declared_narrowed}
{ops}
"""
@@ -209,6 +210,14 @@ new_key = "title"
"""
+_DELETE_LABEL = """
+[[migration.ops]]
+kind = "delete_key"
+table_path = []
+key = "label"
+"""
+
+
_DELETE_LEGACY_MODE = """
[[migration.ops]]
kind = "delete_key"
@@ -407,6 +416,46 @@ mapping = { premium = "standard" }
issues = check_ledger(surface=_surface(config_model=_SchemaTwoEnumMemberGone), migration_dir=tmp_path)
assert LedgerIssueKind.ILLEGAL_REMAP not in _kinds(issues)
+ def test_an_op_free_unsafe_entry_declaring_a_live_path_is_green(self, tmp_path: Path) -> None:
+ """R9's shape, end to end through the gate that decides whether a ledger may ship.
+
+ Nothing is removed and nothing is renamed — the schema tightened what a value may be, which
+ only a human can repair. The entry names the path, and the check is satisfied. The
+ convergence half is the interesting one: the reference documents *set* that path, exactly
+ as every healthy file does, and refusing the ledger for that would make the one remedy the
+ coverage gate offers for a tightened bound impossible to write.
+ """
+ entry = _entry(ops="", safety="unsafe", declared_narrowed='declared_narrowed_paths = ["settings.retries"]')
+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=entry)
+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)
+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=2)
+ assert check_ledger(surface=_surface(config_model=_SchemaOne), migration_dir=tmp_path) == []
+
+ def test_a_declared_narrowed_path_the_entrys_own_version_does_not_have_is_refused(self, tmp_path: Path) -> None:
+ """A declaration nothing records is looked for in every file and found in none."""
+ entry = _entry(ops="", safety="unsafe", declared_narrowed='declared_narrowed_paths = ["settings.retriez"]')
+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=entry)
+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)
+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=2)
+ issues = check_ledger(surface=_surface(config_model=_SchemaOne), migration_dir=tmp_path)
+ assert _kinds(issues) == [LedgerIssueKind.DECLARED_NARROWED_PATH_IS_ABSENT]
+ assert "settings.retriez" in issues[0].message
+
+ def test_a_declared_narrowed_path_that_is_really_a_removal_is_refused_as_absent(self, tmp_path: Path) -> None:
+ """The declaration is not a second way to account for a removal.
+
+ `label` is gone at version 2, so the fingerprint there does not record it — and a removal
+ is accounted for by the operation that removes it, which is a thing the applier can
+ actually do for the user.
+ """
+ entry = _entry(ops=_DELETE_LABEL, safety="unsafe", declared_narrowed='declared_narrowed_paths = ["label"]')
+ _write_ledger(migration_dir=tmp_path, current_schema_version=2, entries=entry)
+ _snapshot(migration_dir=tmp_path, config_model=_SchemaOne, schema_version=1)
+ _snapshot(migration_dir=tmp_path, config_model=_SchemaTwoLabelGone, schema_version=2)
+ assert LedgerIssueKind.DECLARED_NARROWED_PATH_IS_ABSENT in _kinds(
+ check_ledger(surface=_surface(config_model=_SchemaTwoLabelGone), migration_dir=tmp_path)
+ )
+
def test_a_retired_name_that_comes_back_is_refused(self, tmp_path: Path) -> None:
"""And the reuse is not academic: the operation that retired the name starts firing again."""
delete_label = """
diff --git a/tests/unit/pipelex/migration/test_ledger_models.py b/tests/unit/pipelex/migration/test_ledger_models.py
index 94d66aa07..1c3397055 100644
--- a/tests/unit/pipelex/migration/test_ledger_models.py
+++ b/tests/unit/pipelex/migration/test_ledger_models.py
@@ -102,11 +102,31 @@ class TestEntryInvariants:
with pytest.raises(ValidationError, match="cannot be 'safe'"):
MigrationLedger.model_validate({"surface": surface, "migration": [_entry(ops=[])]})
- def test_an_op_free_entry_is_legitimate_when_unsafe(self) -> None:
+ def test_an_op_free_entry_is_legitimate_when_unsafe_and_it_declares_what_it_is_about(self) -> None:
"""A change only a human can make is still a change the ledger should record."""
surface = {**_SURFACE_BLOCK, "current_schema_version": 2}
- ledger = MigrationLedger.model_validate({"surface": surface, "migration": [_entry(ops=[], safety="unsafe")]})
+ entry = _entry(ops=[], safety="unsafe", declared_narrowed_paths=["reporting.retries"])
+ ledger = MigrationLedger.model_validate({"surface": surface, "migration": [entry]})
assert ledger.migration[0].ops == []
+ assert ledger.migration[0].declared_narrowed_paths == ["reporting.retries"]
+
+ def test_an_entry_with_neither_operations_nor_a_declaration_is_refused(self) -> None:
+ """R9: the engine questions an entry before reporting it, and this one answers nothing.
+
+ Operations are questioned by rehearsing them and a declaration by looking its paths up in
+ the document. An entry carrying neither says "nothing to say" for every file there will
+ ever be — the accounting accepts it and the user it exists for never hears of it.
+ """
+ surface = {**_SURFACE_BLOCK, "current_schema_version": 2}
+ with pytest.raises(ValidationError, match="reported to nobody, ever"):
+ MigrationLedger.model_validate({"surface": surface, "migration": [_entry(ops=[], safety="unsafe")]})
+
+ def test_a_safe_entry_may_not_declare_a_narrowing(self) -> None:
+ """A narrowing a remap can repair is that remap's business; one it cannot is what unsafe means."""
+ surface = {**_SURFACE_BLOCK, "current_schema_version": 2}
+ entry = _entry(declared_narrowed_paths=["reporting.retries"])
+ with pytest.raises(ValidationError, match="unsafe entries only"):
+ MigrationLedger.model_validate({"surface": surface, "migration": [entry]})
def test_declared_removed_paths_belong_to_a_pre_history_entry(self) -> None:
"""An entry with an observable diff is accounted against that diff, never its own claim."""
diff --git a/tests/unit/pipelex/migration/test_runner.py b/tests/unit/pipelex/migration/test_runner.py
index 76c8f5287..1999b4b4e 100644
--- a/tests/unit/pipelex/migration/test_runner.py
+++ b/tests/unit/pipelex/migration/test_runner.py
@@ -13,11 +13,11 @@ from pathlib import Path
from pytest_mock import MockerFixture
-from pipelex.migration.backup import existing_backups_of, write_backup
+from pipelex.migration.backup import WrittenBackup, existing_backups_of, write_backup
from pipelex.migration.plan import FileBlockedReason
from pipelex.migration.runner import migrate_directories, migrate_file
from pipelex.migration.surfaces import SurfaceRegistry
-from pipelex.pipeline.exceptions import FixTransactionError
+from pipelex.pipeline.exceptions import FixTransactionError, FixWriteConflictError
from pipelex.pipeline.fixes.file_transaction import FileSnapshot, PendingFileUpdate, commit_file_updates
from pipelex.suggested_fix import RenameTableKeyOp
from tests.unit.pipelex.migration.conftest import EXAMPLE_SURFACE_ID, EntryBuilder, LedgerBuilder, SurfaceBuilder
@@ -273,7 +273,7 @@ class TestMigrationRunner:
assert "output = " in first.read_text(encoding="utf-8")
assert existing_backups_of(path=first) == [report.plans[0].backup_path]
- def test_a_transaction_error_before_the_write_landed_blocks_the_file_and_spares_its_siblings(
+ def test_a_transaction_error_the_file_denies_reports_an_uncertain_state_and_spares_its_siblings(
self,
tmp_path: Path,
mocker: MockerFixture,
@@ -303,16 +303,82 @@ class TestMigrationRunner:
)
assert [plan.was_written for plan in report.plans] == [False, True]
- assert report.plans[0].blocked_reason is FileBlockedReason.UNWRITABLE
+ assert report.plans[0].blocked_reason is FileBlockedReason.STATE_UNCERTAIN
assert first.read_text(encoding="utf-8") == OLD_SHAPE
# The transaction could not vouch for the state it left, so the one certain copy of the
- # original stays, and the report says where it is.
- assert existing_backups_of(path=first) == [report.plans[0].backup_path]
+ # original stays — under a name outside the `.bak.` rotation, where the next successful
+ # run of this file cannot prune it away.
+ assert existing_backups_of(path=first) == []
assert report.plans[0].backup_path is not None
+ assert report.plans[0].backup_path.name == "example.toml.rescue.20260815T120000Z"
assert report.plans[0].backup_path.read_text(encoding="utf-8") == OLD_SHAPE
assert str(report.plans[0].backup_path) in (report.plans[0].blocked_detail or "")
assert "output = " in second.read_text(encoding="utf-8")
+ def test_an_uncertain_write_does_not_promise_to_keep_a_copy_another_run_made(
+ self,
+ tmp_path: Path,
+ mocker: MockerFixture,
+ build_entry: EntryBuilder,
+ build_ledger: LedgerBuilder,
+ build_surface: SurfaceBuilder,
+ ) -> None:
+ """A copy this run did not make stays that run's to move, so it also stays prunable.
+
+ The report may still name it — it does hold the original — but it must not tell the user
+ the copy will be waiting, because the next successful run of this file prunes exactly that
+ name. What the user needs to be told is to take it now.
+ """
+ target = tmp_path / "example.toml"
+ target.write_text(OLD_SHAPE, encoding="utf-8")
+ occupied = tmp_path / "example.toml.bak.20260815T120000Z"
+ occupied.write_text(OLD_SHAPE, encoding="utf-8")
+ ledger = build_ledger(
+ entries=[build_entry(to_schema_version=2, ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")])]
+ )
+ mocker.patch("pipelex.migration.runner.commit_file_updates", side_effect=FixTransactionError("commit failed and rollback was incomplete"))
+
+ plan = migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=False, moment=MOMENT)
+
+ assert plan.blocked_reason is FileBlockedReason.STATE_UNCERTAIN
+ assert plan.backup_path == occupied
+ assert existing_backups_of(path=target) == [occupied], "still in the rotation, so a later run would prune it"
+ detail = plan.blocked_detail or ""
+ assert str(occupied) in detail
+ assert "is kept at" not in detail
+ assert "could not take it out of the way of pruning" in detail
+
+ def test_an_uncertain_write_does_not_promise_a_rescue_the_rename_refused(
+ self,
+ tmp_path: Path,
+ mocker: MockerFixture,
+ build_entry: EntryBuilder,
+ build_ledger: LedgerBuilder,
+ build_surface: SurfaceBuilder,
+ ) -> None:
+ """The same promise, broken the other way: the copy is this run's, and the rescue name is taken.
+
+ `keep_backup_for_rescue` leaves the copy where it is rather than losing it to a tidier
+ name, which is right — and leaves it inside the `.bak.` rotation, which the report must say.
+ """
+ target = tmp_path / "example.toml"
+ target.write_text(OLD_SHAPE, encoding="utf-8")
+ occupied_rescue = tmp_path / "example.toml.rescue.20260815T120000Z"
+ occupied_rescue.write_text("another run got here first\n", encoding="utf-8")
+ ledger = build_ledger(
+ entries=[build_entry(to_schema_version=2, ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")])]
+ )
+ mocker.patch("pipelex.migration.runner.commit_file_updates", side_effect=FixTransactionError("commit failed and rollback was incomplete"))
+
+ plan = migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=False, moment=MOMENT)
+
+ assert plan.blocked_reason is FileBlockedReason.STATE_UNCERTAIN
+ assert plan.backup_path is not None
+ assert plan.backup_path.name == "example.toml.bak.20260815T120000Z"
+ assert plan.backup_path.read_text(encoding="utf-8") == OLD_SHAPE
+ assert occupied_rescue.read_text(encoding="utf-8") == "another run got here first\n", "never clobbered"
+ assert "could not take it out of the way of pruning" in (plan.blocked_detail or "")
+
def test_a_file_edited_between_the_read_and_the_write_is_reported_as_changed_and_left_alone(
self,
tmp_path: Path,
@@ -332,10 +398,10 @@ class TestMigrationRunner:
edited_shape = OLD_SHAPE + "extra = 1\n"
real_write_backup = write_backup
- def edit_target_while_backing_up(*, snapshot: FileSnapshot, moment: datetime) -> Path:
- backup_path = real_write_backup(snapshot=snapshot, moment=moment)
+ def edit_target_while_backing_up(*, snapshot: FileSnapshot, moment: datetime) -> WrittenBackup:
+ backup = real_write_backup(snapshot=snapshot, moment=moment)
target.write_text(edited_shape, encoding="utf-8")
- return backup_path
+ return backup
mocker.patch("pipelex.migration.runner.write_backup", side_effect=edit_target_while_backing_up)
@@ -444,7 +510,7 @@ class TestMigrationRunner:
plan = migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=False, moment=MOMENT)
- assert plan.blocked_reason is FileBlockedReason.UNWRITABLE
+ assert plan.blocked_reason is FileBlockedReason.UNREADABLE
assert "could not be read" in (plan.blocked_detail or "")
assert sorted(path.name for path in tmp_path.iterdir()) == ["example.toml"]
@@ -533,6 +599,130 @@ class TestMigrationRunner:
assert "output = " in healthy.read_text(encoding="utf-8")
assert ledger.surface.current_schema_version == 2
+ def test_a_symlinked_configuration_file_migrates_the_file_it_names_and_keeps_the_link(
+ self,
+ tmp_path: Path,
+ build_entry: EntryBuilder,
+ build_ledger: LedgerBuilder,
+ build_surface: SurfaceBuilder,
+ ) -> None:
+ """A dotfiles setup is a link, and the file the user means is at the end of it.
+
+ An atomic replace of the link path would put a regular file where the link was and leave
+ the real file behind, unmigrated — so the runner resolves first, exactly as the `.mthds`
+ fix loop already resolves its own targets. The copy lands beside the file it copies.
+ """
+ dotfiles = tmp_path / "dotfiles"
+ dotfiles.mkdir()
+ real = dotfiles / "example.toml"
+ real.write_text(OLD_SHAPE, encoding="utf-8")
+ config_dir = tmp_path / ".pipelex"
+ config_dir.mkdir()
+ link = config_dir / "example.toml"
+ link.symlink_to(real)
+ ledger = build_ledger(
+ entries=[build_entry(to_schema_version=2, ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")])]
+ )
+
+ plan = migrate_file(surface=build_surface(), ledger=ledger, file_path=link, dry_run=False, moment=MOMENT)
+
+ assert plan.was_written
+ assert link.is_symlink()
+ assert link.resolve() == real.resolve()
+ assert "output = " in real.read_text(encoding="utf-8")
+ # The plan names the path the walk found; the backup shows where the bytes actually went.
+ assert plan.file_path == link
+ assert plan.backup_path == dotfiles / "example.toml.bak.20260815T120000Z"
+ assert not list(existing_backups_of(path=link))
+
+ def test_a_backup_already_at_this_run_s_name_is_kept_rather_than_overwritten(
+ self,
+ tmp_path: Path,
+ build_entry: EntryBuilder,
+ build_ledger: LedgerBuilder,
+ build_surface: SurfaceBuilder,
+ ) -> None:
+ """Two runs inside one UTC second address the same name, and the older copy is the original.
+
+ The stamp resolves to the second, so a concurrent run of the same file can already have put
+ its copy there. Whatever is there is a copy of an *older* state than this run's, so taking
+ it away is the one loss backups exist to prevent.
+ """
+ target = tmp_path / "example.toml"
+ target.write_text(OLD_SHAPE, encoding="utf-8")
+ occupied = tmp_path / "example.toml.bak.20260815T120000Z"
+ occupied.write_text("the copy another run already made\n", encoding="utf-8")
+ ledger = build_ledger(
+ entries=[build_entry(to_schema_version=2, ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")])]
+ )
+
+ plan = migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=False, moment=MOMENT)
+
+ assert plan.was_written
+ assert occupied.read_text(encoding="utf-8") == "the copy another run already made\n"
+ assert plan.backup_path == occupied
+
+ def test_a_refused_write_never_discards_a_backup_this_run_did_not_make(
+ self,
+ tmp_path: Path,
+ mocker: MockerFixture,
+ build_entry: EntryBuilder,
+ build_ledger: LedgerBuilder,
+ build_surface: SurfaceBuilder,
+ ) -> None:
+ """The narrow path the same-second collision actually damages, and the invariant that ends it.
+
+ A run whose commit is refused deletes the copy it made, because a write that did not happen
+ has nothing to back up. Deleting the copy it *found* is a different act entirely: that one
+ belongs to the run that is presently migrating the file.
+ """
+ target = tmp_path / "example.toml"
+ target.write_text(OLD_SHAPE, encoding="utf-8")
+ occupied = tmp_path / "example.toml.bak.20260815T120000Z"
+ occupied.write_text(OLD_SHAPE, encoding="utf-8")
+ ledger = build_ledger(
+ entries=[build_entry(to_schema_version=2, ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")])]
+ )
+ mocker.patch(
+ "pipelex.migration.runner.commit_file_updates",
+ side_effect=FixWriteConflictError("refusing to overwrite: the file changed while changes were being prepared"),
+ )
+
+ plan = migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=False, moment=MOMENT)
+
+ assert plan.blocked_reason is FileBlockedReason.CHANGED_DURING_RUN
+ assert occupied.exists()
+ assert occupied.read_text(encoding="utf-8") == OLD_SHAPE
+
+ def test_a_rescue_copy_survives_the_next_successful_run_of_the_same_file(
+ self,
+ tmp_path: Path,
+ build_entry: EntryBuilder,
+ build_ledger: LedgerBuilder,
+ build_surface: SurfaceBuilder,
+ ) -> None:
+ """The copy a report told the user to go and get must still be there when they go.
+
+ Pruning keeps exactly one `.bak.<stamp>` per file, and a copy kept because a run could not
+ say what it had left is not one of those — it is outside the rotation by name, so nothing
+ collects it but the user.
+ """
+ target = tmp_path / "example.toml"
+ target.write_text(OLD_SHAPE, encoding="utf-8")
+ rescued = tmp_path / "example.toml.rescue.20260101T000000Z"
+ rescued.write_text("the original, from a run that could not vouch for its write\n", encoding="utf-8")
+ stale_backup = tmp_path / "example.toml.bak.20200101T000000Z"
+ stale_backup.write_text("from another era\n", encoding="utf-8")
+ ledger = build_ledger(
+ entries=[build_entry(to_schema_version=2, ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")])]
+ )
+
+ plan = migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=False, moment=MOMENT)
+
+ assert plan.was_written
+ assert not stale_backup.exists()
+ assert rescued.read_text(encoding="utf-8") == "the original, from a run that could not vouch for its write\n"
+
def test_a_missing_configuration_directory_is_skipped_rather_than_refused(
self,
tmp_path: Path,
diff --git a/tests/unit/pipelex/migration/test_transform_check.py b/tests/unit/pipelex/migration/test_transform_check.py
index 090dfa679..44b4b99a4 100644
--- a/tests/unit/pipelex/migration/test_transform_check.py
+++ b/tests/unit/pipelex/migration/test_transform_check.py
@@ -127,7 +127,7 @@ min_supported_schema_version = 0
(directory / f"{SURFACE_ID}.toml").write_text(body, encoding="utf-8")
-def _entry(*, ops: str, to_schema_version: int = 2, safety: str = "safe") -> str:
+def _entry(*, ops: str, to_schema_version: int = 2, safety: str = "safe", declared: str = "") -> str:
return f"""
[[migration]]
id = "{SURFACE_ID}@{to_schema_version}"
@@ -137,6 +137,7 @@ breaking = true
safety = "{safety}"
title = "Reshape the synthetic surface"
description = "The surface changed shape."
+{declared}
{ops}
"""
@@ -389,7 +390,7 @@ mapping = { basic = "standrad" }
legal precisely when it is unsafe — so demanding its operations reach the new shape would
contradict the one thing `unsafe` means.
"""
- _write_ledger(migration_dir=tmp_path, entries=_entry(ops="", safety="unsafe"))
+ _write_ledger(migration_dir=tmp_path, entries=_entry(ops="", safety="unsafe", declared='declared_narrowed_paths = ["label"]'))
_defaults(migration_dir=tmp_path, schema_version=1, document='label = "hello"\n')
_defaults(migration_dir=tmp_path, schema_version=2, document='title = "hello"\n')
assert check_transform_chain(surface=_surface(config_model=_Renamed), migration_dir=tmp_path) == []
diff --git a/tests/unit/pipelex/migration/test_unsafe_promise.py b/tests/unit/pipelex/migration/test_unsafe_promise.py
new file mode 100644
index 000000000..5cffbceb2
--- /dev/null
+++ b/tests/unit/pipelex/migration/test_unsafe_promise.py
@@ -0,0 +1,326 @@
+"""What an `unsafe` entry promises — the one guarantee the engine owes a user it cannot help.
+
+The promise, in one sentence: an `unsafe` entry is reported on every run, to every file that still
+carries the material it is about, and to no other file. Two defects broke it in opposite
+directions, and both are pinned here.
+
+- **An entry with no operations was reported to nobody, ever** (R9). The engine decided whether to
+ report an `unsafe` entry by rehearsing its operations, so the contract's own form for "a change
+ only a human can make" — the only remedy the coverage gate offers for a tightened bound — was
+ accepted by the accounting and then guaranteed never to reach anyone. Such an entry now declares
+ the paths whose value domain narrowed, and the declaration is what gets questioned.
+- **An entry went silent as soon as a later `safe` entry renamed its target.** `unsafe@2` is about
+ `[reporting] mode`; `safe@3` renames `reporting` to `output`. Run 1 reported the entry and then
+ applied the rename; run 2 found no `reporting.mode`, said nothing, and left a file whose boot
+ still fails. The material is now traced forward through every later `safe` entry, so the same
+ file is reported on every run for as long as it carries it.
+
+Silence is tested as carefully as reporting: the reason `unsafe` entries are questioned at all is
+that reporting one unconditionally warns every user with a perfectly current file, at every boot,
+forever.
+"""
+
+from pipelex.migration.engine import replay_ledger_over_text
+from pipelex.migration.ledger import MigrationLedger, MigrationSafety
+from pipelex.migration.material import declared_path_spellings, unsafe_op_variants
+from pipelex.migration.plan import BlockedEntryReason
+from pipelex.suggested_fix import DeleteTableOp, RemapValueOp, RenameTableKeyOp
+from tests.unit.pipelex.migration.conftest import EntryBuilder, LedgerBuilder
+
+FILE_SETTING_RETRIES = """\
+[reporting]
+retries = 40
+"""
+
+FILE_NOT_SETTING_RETRIES = """\
+[reporting]
+directory = "out"
+"""
+
+FILE_WITH_A_STALE_MODE = """\
+[reporting]
+mode = "legacy"
+"""
+
+FILE_WITH_A_STALE_MODE_UNDER_THE_NEW_NAME = """\
+[output]
+mode = "legacy"
+"""
+
+FILE_WITH_USER_KEYS_UNDER_AN_OPEN_MAPPING = """\
+[levels]
+my_package = "verbose"
+"""
+
+
+class TestWhatAnUnsafeEntryPromises:
+ def test_an_op_free_entry_is_reported_to_a_file_that_sets_a_declared_path(
+ self,
+ build_entry: EntryBuilder,
+ build_ledger: LedgerBuilder,
+ ) -> None:
+ """R9 — the declaration is the predicate, because there is nothing to rehearse."""
+ entry = build_entry(
+ to_schema_version=2,
+ ops=[],
+ safety=MigrationSafety.UNSAFE,
+ guidance="`retries` now tops out at 6.",
+ declared_narrowed_paths=["reporting.retries"],
+ )
+ ledger = build_ledger(entries=[entry])
+
+ replay = replay_ledger_over_text(ledger=ledger, text=FILE_SETTING_RETRIES)
+
+ assert [blocked.entry_id for blocked in replay.blocked] == ["example-config@2"]
+ assert replay.blocked[0].reason is BlockedEntryReason.VALUE_DOMAIN_NARROWED
+ assert replay.blocked[0].narrowed_paths == ["reporting.retries"]
+ assert replay.blocked[0].guidance == "`retries` now tops out at 6."
+ assert replay.text is FILE_SETTING_RETRIES
+
+ def test_an_op_free_entry_stays_silent_on_a_file_that_never_sets_the_declared_path(
+ self,
+ build_entry: EntryBuilder,
+ build_ledger: LedgerBuilder,
+ ) -> None:
+ """The whole reason an entry is questioned: a user who never set the key is not warned."""
+ entry = build_entry(
+ to_schema_version=2,
+ ops=[],
+ safety=MigrationSafety.UNSAFE,
+ declared_narrowed_paths=["reporting.retries"],
+ )
+ ledger = build_ledger(entries=[entry])
+
+ replay = replay_ledger_over_text(ledger=ledger, text=FILE_NOT_SETTING_RETRIES)
+
+ assert replay.blocked == []
+ assert replay.steps == []
+ assert replay.text is FILE_NOT_SETTING_RETRIES
+
+ def test_a_declaration_beneath_an_open_mapping_matches_the_users_own_key(
+ self,
+ build_entry: EntryBuilder,
+ build_ledger: LedgerBuilder,
+ ) -> None:
+ """A schema path is not a document path: the schema says `levels.*`, the file says `levels.my_package`."""
+ entry = build_entry(
+ to_schema_version=2,
+ ops=[],
+ safety=MigrationSafety.UNSAFE,
+ declared_narrowed_paths=["levels.*"],
+ )
+ ledger = build_ledger(entries=[entry])
+
+ replay = replay_ledger_over_text(ledger=ledger, text=FILE_WITH_USER_KEYS_UNDER_AN_OPEN_MAPPING)
+
+ assert [blocked.entry_id for blocked in replay.blocked] == ["example-config@2"]
+ assert replay.blocked[0].narrowed_paths == ["levels.*"], "the ledger's spelling is reported, never the user's own key"
+ assert "my_package" not in replay.blocked[0].detail
+
+ def test_a_declaration_beneath_an_open_mapping_stays_silent_on_an_empty_one(
+ self,
+ build_entry: EntryBuilder,
+ build_ledger: LedgerBuilder,
+ ) -> None:
+ """`*` is one key of an open node, never the node itself — a user with no entries set is not warned."""
+ entry = build_entry(
+ to_schema_version=2,
+ ops=[],
+ safety=MigrationSafety.UNSAFE,
+ declared_narrowed_paths=["levels.*"],
+ )
+ ledger = build_ledger(entries=[entry])
+
+ assert replay_ledger_over_text(ledger=ledger, text="[levels]\n").blocked == []
+
+ def test_the_old_shape_is_reported_and_the_later_rename_still_applies(
+ self,
+ build_entry: EntryBuilder,
+ build_ledger: LedgerBuilder,
+ ) -> None:
+ """Run 1, unchanged behaviour: the entry speaks at its own spelling, the later entry works."""
+ ledger = _ledger_with_an_unsafe_remap_then_a_rename(build_entry=build_entry, build_ledger=build_ledger)
+
+ replay = replay_ledger_over_text(ledger=ledger, text=FILE_WITH_A_STALE_MODE)
+
+ assert [blocked.entry_id for blocked in replay.blocked] == ["example-config@2"]
+ assert [step.entry_id for step in replay.steps] == ["example-config@3"]
+ assert "[output]" in replay.text
+
+ def test_the_migrated_shape_is_still_reported_on_every_later_run(
+ self,
+ build_entry: EntryBuilder,
+ build_ledger: LedgerBuilder,
+ ) -> None:
+ """Run 2 and every run after it. Before the forward trace this was clean while boot failed."""
+ ledger = _ledger_with_an_unsafe_remap_then_a_rename(build_entry=build_entry, build_ledger=build_ledger)
+
+ replay = replay_ledger_over_text(ledger=ledger, text=FILE_WITH_A_STALE_MODE_UNDER_THE_NEW_NAME)
+
+ assert [blocked.entry_id for blocked in replay.blocked] == ["example-config@2"]
+ assert replay.text is FILE_WITH_A_STALE_MODE_UNDER_THE_NEW_NAME
+
+ def test_a_current_value_under_the_new_name_says_nothing(
+ self,
+ build_entry: EntryBuilder,
+ build_ledger: LedgerBuilder,
+ ) -> None:
+ """The forward trace must not cost the silence: the remap is still about a stale spelling."""
+ ledger = _ledger_with_an_unsafe_remap_then_a_rename(build_entry=build_entry, build_ledger=build_ledger)
+
+ replay = replay_ledger_over_text(ledger=ledger, text='[output]\nmode = "classic"\n')
+
+ assert replay.blocked == []
+ assert replay.steps == []
+
+ def test_a_declared_path_is_traced_forward_too(
+ self,
+ build_entry: EntryBuilder,
+ build_ledger: LedgerBuilder,
+ ) -> None:
+ """The op-free shape meets the same rename, and the declaration follows the material."""
+ ledger = build_ledger(
+ entries=[
+ build_entry(
+ to_schema_version=2,
+ ops=[],
+ safety=MigrationSafety.UNSAFE,
+ declared_narrowed_paths=["reporting.retries"],
+ ),
+ build_entry(to_schema_version=3, ops=[RenameTableKeyOp(table_path=[], key="reporting", new_key="output")]),
+ ]
+ )
+
+ replay = replay_ledger_over_text(ledger=ledger, text="[output]\nretries = 40\n")
+
+ assert [blocked.entry_id for blocked in replay.blocked] == ["example-config@2"]
+
+ def test_a_declared_path_is_named_as_the_file_this_run_writes_spells_it(
+ self,
+ build_entry: EntryBuilder,
+ build_ledger: LedgerBuilder,
+ ) -> None:
+ """The user is sent to check the file the run *wrote*, so that is the spelling to name.
+
+ The entry is questioned against the text as it stands when the replay reaches it, and the
+ later `safe` entries of the same replay then rename the material out from under it. Naming
+ the spelling that matched sends the user looking for a key the migrated file no longer has.
+ """
+ ledger = build_ledger(
+ entries=[
+ build_entry(
+ to_schema_version=2,
+ ops=[],
+ safety=MigrationSafety.UNSAFE,
+ declared_narrowed_paths=["reporting.retries"],
+ ),
+ build_entry(to_schema_version=3, ops=[RenameTableKeyOp(table_path=[], key="reporting", new_key="output")]),
+ ]
+ )
+
+ replay = replay_ledger_over_text(ledger=ledger, text=FILE_SETTING_RETRIES)
+
+ assert "[output]" in replay.text, "the rename applied, so the file no longer spells it `reporting`"
+ assert replay.blocked[0].narrowed_paths == ["output.retries"]
+ assert "output.retries" in replay.blocked[0].detail
+ assert "reporting.retries" not in replay.blocked[0].detail
+
+ def test_material_a_later_entry_retires_leaves_nothing_to_say_to_a_file_migrated_that_far(
+ self,
+ build_entry: EntryBuilder,
+ build_ledger: LedgerBuilder,
+ ) -> None:
+ """Tracing forward follows material, and stops where the material stops.
+
+ A file migrated past the deletion no longer carries anything the entry is about, and
+ warning it about a section it no longer has would be the crying wolf the questioning
+ exists to prevent. A file that still carries the old shape is still reported.
+ """
+ ledger = build_ledger(
+ entries=[
+ build_entry(
+ to_schema_version=2,
+ ops=[],
+ safety=MigrationSafety.UNSAFE,
+ declared_narrowed_paths=["reporting.retries"],
+ ),
+ build_entry(to_schema_version=3, ops=[DeleteTableOp(table_path=["reporting"])]),
+ ]
+ )
+
+ assert replay_ledger_over_text(ledger=ledger, text='[storage]\ndirectory = "out"\n').blocked == []
+ assert [blocked.entry_id for blocked in replay_ledger_over_text(ledger=ledger, text=FILE_SETTING_RETRIES).blocked] == ["example-config@2"]
+
+ def test_an_ordinary_ledger_has_exactly_one_spelling_to_rehearse(
+ self,
+ build_entry: EntryBuilder,
+ build_ledger: LedgerBuilder,
+ ) -> None:
+ """The common case costs nothing: no later entry moves this entry's material."""
+ entry = build_entry(
+ to_schema_version=2,
+ ops=[RemapValueOp(table_path=["reporting"], key="mode", mapping={"legacy": "classic"})],
+ safety=MigrationSafety.UNSAFE,
+ )
+ ledger = build_ledger(entries=[entry, build_entry(to_schema_version=3, ops=[DeleteTableOp(table_path=["archive"])])])
+
+ assert unsafe_op_variants(ledger=ledger, entry=entry) == [entry.ops]
+
+ def test_a_later_unsafe_entry_moves_nothing_because_it_is_never_applied(
+ self,
+ build_entry: EntryBuilder,
+ build_ledger: LedgerBuilder,
+ ) -> None:
+ """Only an entry that is *applied* can give material a new spelling."""
+ entry = build_entry(
+ to_schema_version=2,
+ ops=[],
+ safety=MigrationSafety.UNSAFE,
+ declared_narrowed_paths=["reporting.retries"],
+ )
+ ledger = build_ledger(
+ entries=[
+ entry,
+ build_entry(
+ to_schema_version=3,
+ ops=[RenameTableKeyOp(table_path=[], key="reporting", new_key="output")],
+ safety=MigrationSafety.UNSAFE,
+ ),
+ ]
+ )
+
+ assert declared_path_spellings(ledger=ledger, entry=entry) == ["reporting.retries"]
+
+ def test_a_declaration_is_traced_back_through_the_entrys_own_operations(
+ self,
+ build_entry: EntryBuilder,
+ build_ledger: LedgerBuilder,
+ ) -> None:
+ """The declaration is spelled at the entry's own version, and the entry never applies.
+
+ So a file the entry blocks keeps the previous version's spelling, and questioning only the
+ post-entry one would find nothing in the very files the entry exists for.
+ """
+ entry = build_entry(
+ to_schema_version=2,
+ ops=[RenameTableKeyOp(table_path=[], key="reporting", new_key="output")],
+ safety=MigrationSafety.UNSAFE,
+ declared_narrowed_paths=["output.retries"],
+ )
+ ledger = build_ledger(entries=[entry])
+
+ assert declared_path_spellings(ledger=ledger, entry=entry) == ["output.retries", "reporting.retries"]
+
+
+def _ledger_with_an_unsafe_remap_then_a_rename(*, build_entry: EntryBuilder, build_ledger: LedgerBuilder) -> MigrationLedger:
+ """`unsafe@2` is about `reporting.mode`; `safe@3` renames the table it sits in."""
+ return build_ledger(
+ entries=[
+ build_entry(
+ to_schema_version=2,
+ ops=[RemapValueOp(table_path=["reporting"], key="mode", mapping={"legacy": "classic"})],
+ safety=MigrationSafety.UNSAFE,
+ ),
+ build_entry(to_schema_version=3, ops=[RenameTableKeyOp(table_path=[], key="reporting", new_key="output")]),
+ ]
+ )
diff --git a/tests/unit/pipelex/pipeline/fixes/test_fix_applier_config_surface_shapes.py b/tests/unit/pipelex/pipeline/fixes/test_fix_applier_config_surface_shapes.py
index ffa88f696..df898d5c1 100644
--- a/tests/unit/pipelex/pipeline/fixes/test_fix_applier_config_surface_shapes.py
+++ b/tests/unit/pipelex/pipeline/fixes/test_fix_applier_config_surface_shapes.py
@@ -166,6 +166,85 @@ class TestFixApplierConfigSurfaceShapes:
assert [application.outcome for application in applications] == [FixOpOutcome.APPLIED]
assert _dumps(toml_doc) == "[section]\nnested.renamed = 1\nother = 2\n"
+ def test_renaming_a_dotted_head_keeps_it_dotted_and_leaves_its_siblings_alone(self) -> None:
+ """Renaming ``k`` in ``k.x = 1`` gives ``kk.x = 1``, not a header that swallows what follows.
+
+ tomlkit stores a dotted assignment as a super-table under a key flagged *dotted*, and its
+ re-key primitive builds a fresh key that carries no such flag. The renderer then emits a
+ block header — and every scalar *after* it in the same table falls under that header, so
+ ``a.m`` silently became ``a.kk.m`` with an ``applied`` verdict. The reshape renames tables,
+ and dotted form is an ordinary way to write one, so this meets a real file on the first
+ machine.
+ """
+ toml_doc = tomlkit.parse("[a]\nk.x = 1\nm = 3\n")
+ applications = apply_fix_ops(toml_doc=toml_doc, ops=[RenameTableKeyOp(table_path=["a"], key="k", new_key="kk")])
+ assert [application.outcome for application in applications] == [FixOpOutcome.APPLIED]
+ rendered = _dumps(toml_doc)
+ assert rendered == "[a]\nkk.x = 1\nm = 3\n"
+ assert tomlkit.parse(rendered).unwrap() == {"a": {"kk": {"x": 1}, "m": 3}}
+
+ def test_renaming_an_inner_segment_of_a_dotted_key_keeps_the_whole_chain_dotted(self) -> None:
+ """``k.x.y = 1`` renamed at ``x`` stays inside ``[a]`` instead of relocating to the root.
+
+ An inner segment is the worse half of the same defect: undotted, it makes tomlkit render
+ the chain as a *top-level* ``[k.xx]`` header, so the data leaves ``[a]`` altogether and
+ ``a`` is left an empty table.
+ """
+ toml_doc = tomlkit.parse("[a]\nk.x.y = 1\nm = 3\n")
+ applications = apply_fix_ops(toml_doc=toml_doc, ops=[RenameTableKeyOp(table_path=["a", "k"], key="x", new_key="xx")])
+ assert [application.outcome for application in applications] == [FixOpOutcome.APPLIED]
+ rendered = _dumps(toml_doc)
+ assert rendered == "[a]\nk.xx.y = 1\nm = 3\n"
+ assert tomlkit.parse(rendered).unwrap() == {"a": {"k": {"xx": {"y": 1}}, "m": 3}}
+
+ def test_renaming_a_dotted_head_at_the_document_root_keeps_it_dotted(self) -> None:
+ """The root container is a ``Container`` rather than a ``Table``, and takes the same branch."""
+ toml_doc = tomlkit.parse("k.x = 1\nm = 3\n")
+ applications = apply_fix_ops(toml_doc=toml_doc, ops=[RenameTableKeyOp(table_path=[], key="k", new_key="kk")])
+ assert [application.outcome for application in applications] == [FixOpOutcome.APPLIED]
+ rendered = _dumps(toml_doc)
+ assert rendered == "kk.x = 1\nm = 3\n"
+ assert tomlkit.parse(rendered).unwrap() == {"kk": {"x": 1}, "m": 3}
+
+ def test_renaming_a_dotted_head_is_replay_neutral_once_applied(self) -> None:
+ """Replayed over its own output the rename skips and changes no bytes, like every other one.
+
+ Worth its own case rather than trusting the general rename: the fix hands tomlkit a key it
+ would not have built itself, and a document that no longer round-trips would break the
+ engine's central guarantee precisely on the layout this fix exists for.
+ """
+ rename_op = RenameTableKeyOp(table_path=["a"], key="k", new_key="kk")
+ toml_doc = tomlkit.parse("[a]\nk.x = 1\nk.y = 2\nm = 3\n")
+ apply_fix_ops(toml_doc=toml_doc, ops=[rename_op])
+ once = _dumps(toml_doc)
+ assert once == "[a]\nkk.x = 1\nkk.y = 2\nm = 3\n"
+
+ replayed_doc = tomlkit.parse(once)
+ applications = apply_fix_ops(toml_doc=replayed_doc, ops=[rename_op])
+ assert [application.outcome for application in applications] == [FixOpOutcome.SKIPPED]
+ assert _dumps(replayed_doc) == once
+
+ def test_renaming_a_block_table_inserts_no_blank_line(self) -> None:
+ """A rename changes a name; it does not reflow the file around it.
+
+ tomlkit's re-key primitive appends a cosmetic newline to the table it replaces, which is
+ right for one it is about to re-home and wrong for one staying exactly where it is. On a
+ configuration surface the output is ``tomlkit.dumps`` and nothing else, so that newline was
+ a line of diff in a user's file for every rename an entry carries.
+ """
+ toml_doc = tomlkit.parse("[a]\nx = 1\n[b]\ny = 2\n")
+ applications = apply_fix_ops(toml_doc=toml_doc, ops=[RenameTableKeyOp(table_path=[], key="a", new_key="aa")])
+ assert [application.outcome for application in applications] == [FixOpOutcome.APPLIED]
+ assert _dumps(toml_doc) == "[aa]\nx = 1\n[b]\ny = 2\n"
+
+ def test_renaming_a_dotted_head_onto_a_name_already_taken_is_a_conflict(self) -> None:
+ """Dotted form does not get its own collision rule: the same name in the same table refuses."""
+ toml_doc = tomlkit.parse("[a]\nk.x = 1\nkk.y = 2\n")
+ before = _dumps(toml_doc)
+ applications = apply_fix_ops(toml_doc=toml_doc, ops=[RenameTableKeyOp(table_path=["a"], key="k", new_key="kk")])
+ assert [application.outcome for application in applications] == [FixOpOutcome.CONFLICT]
+ assert _dumps(toml_doc) == before
+
def test_quoted_key_containing_a_dot_is_one_path_segment(self) -> None:
"""A quoted key with a dot inside is a single segment, and stays quoted when renamed."""
toml_doc = tomlkit.parse('["outer.inner"]\nvalue = 1\n')
diff --git a/tests/unit/pipelex/pipeline/fixes/test_fix_applier_rename_dom_consistency.py b/tests/unit/pipelex/pipeline/fixes/test_fix_applier_rename_dom_consistency.py
index d8d426059..027079f4f 100644
--- a/tests/unit/pipelex/pipeline/fixes/test_fix_applier_rename_dom_consistency.py
+++ b/tests/unit/pipelex/pipeline/fixes/test_fix_applier_rename_dom_consistency.py
@@ -1,22 +1,23 @@
"""Characterization of what a rename leaves behind in the tomlkit DOM.
-`_rename_key_in_place` renames through `Container._replace`, tomlkit's own position-preserving
-primitive — the only one there is, and the reason a renamed key keeps its place among its siblings
-instead of being appended at the bottom of its parent. What it does **not** do, for a value that
-is not a `Table`, is update the node's raw `dict` storage: the old key stays in the dict and the
-new one is only ever written into it inside the table branch of `_replace_at`.
+`_rename_key_in_place` renames a container's body entries where they sit, which is what keeps a
+renamed key in its place among its siblings instead of appended at the bottom of its parent. What
+that cannot reach, for a value that is not a `Table`, is the raw `dict` storage of the **parent
+node**: a `Table` and the `Container` inside it are two dict-like objects each holding their own
+copy of the key set, and only the container's copy is in reach.
Everything tomlkit renders or looks up still works — `dumps`, `in`, `[]`, `.get()` all read the
authoritative body — so a single rename followed by serialization is correct, which is why the
`.mthds` fix path never met this: the keys it renames are `[pipe.*]` tables, and tables take the
-branch that keeps the dict in step.
+branch tomlkit does keep in step.
A migration is the case that meets it, because always-replay runs many operations over one
document. The consequence is pinned below: addressing a renamed non-table key again in the same
-DOM raises `KeyError` from inside the library. `pipelex.migration.engine` works around it by
-re-reading the document between operations that applied, and this module is the tripwire — **if
-these tests go red after a tomlkit bump, the workaround can be reconsidered**, and the engine's
-docstring says so.
+DOM raises `KeyError` from inside the library. `pipelex.migration.engine` re-reads the document
+between operations that applied, for reasons of its own, and so never sees it. Repairing the
+staleness means one deliberate pass over every facade kind — a `Container`, a `Table`, an
+out-of-order proxy — and this module is what tells whoever makes that pass, or a tomlkit bump,
+that the behaviour moved.
"""
from typing import Any, cast
@@ -116,6 +117,29 @@ class TestRenameDomConsistency:
with pytest.raises(KeyError):
apply_fix_ops(toml_doc=toml_doc, ops=[DeleteKeyOp(table_path=table_path, key=new_key)])
+ def test_a_renamed_root_table_moves_in_the_documents_own_storage(self) -> None:
+ """The half of the bookkeeping the rename does get right, pinned so it cannot become a no-op.
+
+ A `Container` is a `MutableMapping` as well as a `dict`, so its own `pop` routes through the
+ accessors that read the body — which, once the body has been renamed, no longer knows the
+ old name and quietly reports nothing to remove. The rename calls `dict.pop` explicitly for
+ that reason, and this is what says so.
+ """
+ toml_doc = tomlkit.loads("[section]\nx = 1\n")
+
+ apply_fix_ops(toml_doc=toml_doc, ops=[RenameTableKeyOp(table_path=[], key="section", new_key="renamed")])
+
+ assert _raw_dict_keys(node=toml_doc) == {"renamed"}
+
+ def test_a_renamed_root_scalar_leaves_neither_name_in_the_documents_own_storage(self) -> None:
+ """The same staleness as above, measured at the root, where the container *is* the node."""
+ toml_doc = tomlkit.loads("retention = 30\n")
+
+ apply_fix_ops(toml_doc=toml_doc, ops=[RenameTableKeyOp(table_path=[], key="retention", new_key="keep_days")])
+
+ assert _raw_dict_keys(node=toml_doc) == set()
+ assert _dumped(toml_doc=toml_doc) == "keep_days = 30\n"
+
def test_renaming_a_table_valued_key_keeps_the_dom_consistent(self) -> None:
"""The branch tomlkit does maintain — and the reason the `.mthds` fix path never met this."""
toml_doc = tomlkit.loads(TABLE_VALUED)
diff --git a/wip/migrator-phase3-progress.md b/wip/migrator-phase3-progress.md
index 1b2775593..5e4d5dafc 100644
--- a/wip/migrator-phase3-progress.md
+++ b/wip/migrator-phase3-progress.md
@@ -2,16 +2,18 @@
The charter is [`wip/migrator-3/sequencing.md` § S6](../../wip/migrator-3/sequencing.md) at the **workspace root** (from this repo: `../wip/migrator-3/sequencing.md`), with the phase content in `plan.md § Phase 3`. This file is the session-crossing record of what S6 has built, what it decided, and what is still open. Open the charter first, then this.
-**Venue.** `_migrator/`, branch `feature/Migrator-3`, cut from `origin/dev` = `0aa8913f2` (the Phase 2 squash merge). Milestone 1 is committed as `077744b3d`; nothing pushed yet, no PR open yet.
+**Venue.** `_migrator/`, branch `feature/Migrator-3`, cut from `origin/dev` (the Phase 2 squash merge). Nothing pushed yet, no PR open yet.
+
+⚠ **Cold-start state.** Milestone 1 is the branch's only commit (*"Migrator Phase 3, milestone 1: settle the golden format before S7 freezes it"*). **Milestones 2 and 3 are green but uncommitted** — staged working-tree state on top of it, across 22 files. Do not `git checkout`/`git stash` anything before reading the two milestone sections below; commit them first if you want a clean tree.
## Build order chosen for this session
The charter lists a great many deliverables without an order. The order below was chosen so that everything touching the **golden format** happens first and the goldens are regenerated exactly once, before any other work can be built on top of a format that then moves:
1. **The golden-format bucket + the narrowing relation** — done, see below.
-2. **R9 + the "what does `unsafe` promise" pass** — not started.
-3. **The applier's dotted-key rename policy, the backup/replace semantics pass, the `UNWRITABLE` wording** — not started.
-4. **The commands** — `pipelex migrate`, `pipelex-agent migrate`, the downgrade diagnosis, the rendering-rule test — not started.
+2. **R9 + the "what does `unsafe` promise" pass** — done, see Milestone 2 below.
+3. **The applier's dotted-key rename policy, the backup/replace semantics pass, the `UNWRITABLE` wording** — done, see Milestone 3 below. That closes the "must be settled before S7" bucket.
+4. **The commands** — `pipelex migrate`, `pipelex-agent migrate`, the downgrade diagnosis, the rendering-rule test — not started. **This is where the next session starts.**
5. **Boot tolerance, `report_validation_error` on the real plan, the telemetry remedies retired, the `doctor` row** — not started.
6. **The skills (`add-migration`, `/release` step 3b), the `config-docs` drift-contract review list, `command-surface-map.md` rows, publishing the contract in the nav** — not started.
@@ -89,6 +91,98 @@ added.
`make agent-check` green. `make cl` green, `make cmig` green. `make umig` is a no-op on the tree as committed. Full `make agent-test` green — every test passed, working tree clean.
+## Milestone 2 — what an `unsafe` entry promises (DONE, uncommitted at the pause)
+
+R9 (ruled option 2 on 2026-08-16) and the round-2 sibling — an `unsafe` entry silenced by a later `safe` rename of its target — were built as the single pass the charter asked for. The promise is now written once, in `docs/migration-ledger.md` § "What an `unsafe` entry promises":
+
+> An `unsafe` entry is reported on every run, to every file that still carries the material it is about — at whatever spelling that material has reached — and to no other file.
+
+### The two halves
+
+**(a) R9 — the entry declares what it is about.** `MigrationEntry.declared_narrowed_paths` is a new ledger field carrying the paths whose *value domain* narrowed, spelled as the fingerprint at the entry's own version records them (`*` included). An op-free entry must carry one (refused at parse otherwise, since it could be reported to nobody, ever); a `safe` entry may not carry one at all (a narrowing a remap repairs is that remap's business, one it cannot repair is what `unsafe` means). `check-ledger` gained `DECLARED_NARROWED_PATH_IS_ABSENT`: a declared path the entry's own version does not record is either a misspelling or a *removal*, and a removal is accounted for by the operation that removes it.
+
+**And the coverage gate no longer accepts the bare word.** `_check_narrowing_accounting` and `_check_enum_accounting` used to `return []` for any `unsafe` entry. They now require the narrowed / member-losing path to appear in `declared_narrowed_paths`. Without that, R9 would have been half-built: the ledger would accept a declaration and the gate would never ask for one.
+
+**(b) The sibling — material moves, and the entry follows it.** New module `pipelex/migration/material.py`: pure ledger arithmetic (no fingerprint, no model, no filesystem, so the engine stays the function the gates replay) that traces an entry's material **back** through the entry's own operations — it is never applied, so a file it blocks keeps the previous spelling — and **forward** through every later `safe` entry. A later `unsafe` entry moves nothing (never applied); material a later entry deletes stops being traced. `unsafe_op_variants` respells the operations' **sources only**, because only the source decides whether the rehearsal would do something.
+
+**Scope boundary, deliberate and recorded in the contract: a rehearsal may guess, an application may not.** Forward tracing is confined to `unsafe` entries, whose operations are rehearsed against a copy and discarded — the worst a wrong guess costs is one report too many. A **`safe`** entry silenced the same way (by its own `CONFLICT`, after which a later entry renames the table around it) has the identical mechanism and is left alone, because repairing it would mean *writing* at a spelling its author never wrote. **That is an open question for a later session** — it is a question about what a `safe` entry promises, and it was not in R9's scope.
+
+### The collision this uncovered, and how it was resolved
+
+The declaration's predicate is **presence**, not violation — the engine is model-free by design, and R9's ruling explicitly refused to thread a model into it. So a *current* reference document sets the narrowed path like every healthy file does, the entry speaks, and `_check_convergence` refused the ledger: **R9's ruled shape was unbuildable as first written** (reproduced before it was fixed).
+
+Resolved as narrowly as it can be: a third `BlockedEntryReason`, `VALUE_DOMAIN_NARROWED`, weaker than `UNSAFE` — *this file sets a key whose accepted values narrowed, check it by hand*, versus *this file has the old shape*. Convergence exempts that reason and nothing else; an `unsafe` entry whose **operations** fire on a witness still fails, because that says the checked-in reference document carries retired material. `BlockedEntry.narrowed_paths` lists the paths **as the ledger spells them** (`levels.*`, never the user's own `levels.my_package`), so the rendering rule holds for keys as well as values.
+
+### Files touched in milestone 2
+
+```
+pipelex/migration/material.py NEW — back/forward tracing of an entry's material through the ledger
+pipelex/migration/ledger.py declared_narrowed_paths + two validators
+pipelex/migration/engine.py the two predicates split; the new reason; ledger passed to the rehearsal
+pipelex/migration/plan.py BlockedEntryReason.VALUE_DOMAIN_NARROWED; BlockedEntry.narrowed_paths
+pipelex/migration/documents.py document_carries_path — `*` matches exactly one segment
+pipelex/migration/ledger_check.py DECLARED_NARROWED_PATH_IS_ABSENT; the convergence exemption
+pipelex/migration/coverage.py unsafe must declare, on both the narrowing and the enum half
+docs/migration-ledger.md § "What an `unsafe` entry promises" + the rules threaded through
+CHANGELOG.md one Added bullet
+tests/unit/pipelex/migration/test_unsafe_promise.py NEW
+tests/unit/pipelex/migration/{conftest,test_ledger_models,test_ledger_check,test_coverage_gate,test_transform_check}.py
+```
+
+The new tests were mutation-tested against the pre-fix engine: five of them go red, which is the guarantee they are meant to hold.
+
+### Verified at the pause
+
+`make agent-check` green, `make cl` green, `make cmig` green, and the full `make agent-test` green — every test passed. Nothing committed yet: milestone 2 is uncommitted working-tree state on top of `9eea2219f`.
+
+## Milestone 3 — the write path: dotted keys, backups, and what a blocked file means (DONE, uncommitted)
+
+The last of the "must be settled before S7" bucket. Three things the charter listed separately turned out to be one pass over the code that actually touches a user's file.
+
+### (a) The dotted-key rename — fixed at the root, not refused
+
+The defect reproduced immediately and is worse than the round-3 note recorded. `[a]` / `k.x = 1` / `m = 3`, rename `k` → `kk`, gives `[a.kk]` with **`m` beneath it** — `a.m` became `a.kk.m`, verdict `applied`, nothing said. Renaming an *inner* segment of `k.x.y = 1` re-rendered the chain as a top-level `[k.xx]`, moving the subtree out of `[a]` entirely and leaving `a` empty. Both are silent data corruption on an ordinary TOML layout.
+
+**Root cause:** dotted-ness is a private flag tomlkit's parser sets on the key, not a property of the name, and `Container._replace_at` builds a fresh `SingleKey` whenever the name changes. The renderer then emits a block header — and a block header absorbs every scalar after it in the same table.
+
+**Decided: preserve, do not refuse.** A rename has exactly one correct answer on a dotted key, the layout is ordinary TOML no formatter rewrites, and refusing would strand the reshape's table renames on any file written that way. Same shape as round 3's stale `display_name`, fixed the same way.
+
+`_replace_key_in_container` no longer calls `Container._replace` at all; it renames the container's body entries where they sit. That drops three things the primitive did that a *rename* has no business doing: losing the dotted flag, collapsing a key written as several dotted lines into one rebuilt value (the dict facade for that shape holds only the last chunk), and injecting a cosmetic blank line after the renamed table. Index bookkeeping stays byte-for-byte what `_replace_at` did, **including its asymmetry** — the raw-storage staleness is deliberately not half-repaired here, because the stale facade for a nested rename is the parent `Table`'s own dict, a different object holding items rather than values. Repairing every facade kind is its own pass; `test_fix_applier_rename_dom_consistency.py` now says so and pins both halves.
+
+### (b) The backup and replace semantics pass
+
+Five questions, five policies, all written into the contract's "Backups" paragraph:
+
+- **A run never overwrites or removes a copy it did not make** (M). The stamp resolves to the second, so two runs can address one name; `write_backup` now *reserves* it with an exclusive create and keeps whatever is there — an older copy is the original, if anything is — and reports `was_created` so `_discard_backup` skips a copy this run merely found. The reachable damage was a concurrent run's refused write unlinking the other run's real backup.
+- **Symlinks are followed** (N). The runner resolves before reading, exactly as the `.mthds` fix loop already resolves its own targets, so the two callers of the shared primitive now agree and `file_transaction.py` needed no change. The plan keeps naming the walked path; the backup lands beside the file that was actually rewritten.
+- **A copy the run cannot vouch for leaves the rotation.** It is renamed `.bak.` → `.rescue.`, which `existing_backups_of` does not match, so the next successful run cannot prune away the copy the report told the user to go and get.
+- **Ownership, ACLs and xattrs are not carried across** — decided, documented, not implemented. An atomic replace cannot preserve what the process has no right to set, and re-attaching an attribute blindly (a quarantine flag, a security label) is a worse guess than leaving it off. Mode is carried, and that is the security-relevant bit.
+- **A directory `fsync` is owed, once** — after the backup lands, before the target is replaced. That is what makes "back up first, replace second" hold across a power loss. The target's own replacement is deliberately not synced: always-replay means a migration lost to a crash is replayed, while a lost backup is lost.
+
+### (c) `UNWRITABLE`, and the reason enum
+
+The round-3 reading is confirmed: for the single-file commit the runner performs, a failed replace re-raises its own `OSError`/`FixWriteConflictError` (a rollback of nothing is trivially complete), so the only `FixTransactionError` reaching a plan is the post-commit cleanup one. `FileBlockedReason` now carries one member per **state the file is in** rather than per exception caught — `UNREADABLE`, `UNPARSEABLE`, `UNWRITABLE`, `CHANGED_DURING_RUN`, `STATE_UNCERTAIN` — each with its own docstring and a row in the contract. `STATE_UNCERTAIN` is the only one that cannot promise the file is as it was found, which is exactly why it is not folded into `UNWRITABLE`: the next move is to compare against the rescue copy, not to fix a permission and re-run. Its wording no longer asserts which way the write went, because the runner genuinely cannot tell — only that the file does not hold what it wrote.
+
+### Files touched in milestone 3
+
+```
+pipelex/pipeline/fixes/applier.py _replace_key_in_container rewritten; _renamed_key; _rehome_key_indexes
+pipelex/migration/backup.py WrittenBackup; name reservation; rescue names; directory fsync
+pipelex/migration/runner.py symlink resolution; the new reasons; rescue on an uncertain write
+pipelex/migration/plan.py FileBlockedReason: UNREADABLE + STATE_UNCERTAIN, all documented
+docs/migration-ledger.md "A rename changes a name"; the blocked-reason table; Backups rewritten
+CHANGELOG.md a Fixed section
+tests/unit/pipelex/pipeline/fixes/test_fix_applier_config_surface_shapes.py the dotted red/green fixtures
+tests/unit/pipelex/pipeline/fixes/test_fix_applier_rename_dom_consistency.py re-aimed at the new implementation
+tests/unit/pipelex/migration/test_runner.py symlink, collision, rescue, the reasons
+```
+
+Every new runner test was mutation-tested: dropping the resolve, clobbering the backup, discarding unconditionally, skipping the rescue rename, and folding `.rescue.` back into `.bak.` each turn exactly the intended test red.
+
+### Verified at the pause
+
+`make agent-check` green, `make cl` green, `make cmig` green, `make docs-check` green, `make drift-check` green **with the changes staged**. Full `make agent-test` green. Milestones 2 and 3 are both uncommitted working-tree state on top of `9eea2219f`.
+
## Already true before this session started (do not rebuild)
- **The directory walk's overlap refusal** — the S6 "Done when" item carried from the #1110 review — **is already built and tested**: `SurfaceRegistry.surface_for_file_name` raises `MigrationRegistryError` naming the file, and `tests/unit/pipelex/migration/test_surface_resolution.py::test_a_file_two_globs_both_claim_stops_by_name` proves it with the `pipelex_*.toml` / `*_local.toml` pair. It landed in Phase 2 ahead of schedule. **What is still owed** is the #1111 note-5 half: use the *real* specimen — `.pipelex/inference/backends/pipelex_gateway.toml`, which matches the `pipelex-config` tier glob and is claimed by nothing — rather than a synthetic name, and pin that the walk is **non-recursive** (it is: `files_by_surface_in_directory` skips anything that is not a file, so the `inference/backends/` subdirectory is never descended). That is a test, not a behaviour change.
@@ -97,13 +191,19 @@ added.
Everything in [§ S6](../../wip/migrator-3/sequencing.md#s6--migrator-phase-3) not listed under Milestone 1 above. The ones that must be settled *here* rather than at S7:
-- **The dotted-key rename policy** — renaming a table written in dotted-key form swallows the scalars after it. Reproduced in round 3; the reshape's table renames will meet it on the first machine. Refuse with a `CONFLICT` naming the section, or preserve dotted-ness through `_replace_key_in_container`; either way a red/green fixture in `test_fix_applier_config_surface_shapes.py`.
-- **The backup and replace semantics pass** — the same-second backup collision (M), a symlinked target replaced by a regular file (N), the backup kept after an uncertain-state failure being pruned by the next successful run, ownership/ACL/xattr loss, and whether a directory `fsync` is owed. One policy decision, written into the contract's "Backups" paragraph. `file_transaction.py` is shared with the `.mthds` fix loop.
-- **The `UNWRITABLE` wording and the reason enum** — for a single-file commit the only reachable `FixTransactionError` is post-commit cleanup, i.e. the write *landed*; the wording describes an unreachable path.
-- **R9** — the op-free `unsafe` entry declares the paths whose domain narrowed; `check-ledger` demands the declaration name at least one path the fingerprint at the entry's own version still has; the rehearsal reads it. Built together with the sibling question (an `unsafe` entry silenced by a later `safe` rename of its target) as one pass over what `unsafe` promises.
+- ~~**The dotted-key rename policy**~~ — done, see Milestone 3(a).
+- ~~**The backup and replace semantics pass**~~ — done, see Milestone 3(b).
+- ~~**The `UNWRITABLE` wording and the reason enum**~~ — done, see Milestone 3(c).
+- ~~**R9**~~ — done, see Milestone 2. What it left open: **a `safe` entry silenced by its own `CONFLICT` plus a later rename** has the same mechanism and was deliberately not fixed, because repairing it means writing at a guessed spelling.
+
+**The "must be settled before S7" bucket is closed.** Two named follow-ups came out of it, neither blocking: the tomlkit raw-storage staleness wants one pass over every facade kind (`Container`, `Table`, `OutOfOrderTableProxy`) rather than the half-repair a rename can reach; and the `safe`-entry sibling above.
Then the phase's own body: the two commands, boot tolerance, `report_validation_error`, the downgrade diagnosis, the telemetry-remedy retirement, `doctor`, the drift-contract review list (plus the four validator sites R8 names), the `add-migration` skill, `/release` step 3b, the `command-surface-map.md` rows, and publishing the contract in the nav.
## Rulings this session took on its own authority
The charter says the format bucket is to be "decided and recorded here" — it does not route it to Louis, unlike R1–R9. The three decisions above were taken on that basis, each with its reasoning written into `docs/migration-ledger.md` rather than only here. **R9 itself is already ruled** (option 2, 2026-08-16) and is a build task, not a decision.
+
+Milestone 3 took the policy calls the charter routed here by name — preserve dotted-ness rather than refuse it; follow a symlink rather than refuse or clobber it; never overwrite or remove a copy this run did not make; take a copy the run cannot vouch for out of the pruning rotation by name; carry the mode and not the ownership or the extended attributes; `fsync` the directory once, after the backup and before the replace. Each is written into `docs/migration-ledger.md` rather than only here.
+
+Milestone 2 took three further rulings on that same basis, each written into `docs/migration-ledger.md` rather than only here: the coverage gate now demands a declaration from an `unsafe` entry rather than accepting the word (without it R9 is half-built); convergence exempts a `VALUE_DOMAIN_NARROWED` report and nothing else (without it R9's ruled shape cannot be written at all); and forward tracing stops at `unsafe` entries (a rehearsal may guess, an application may not). The third is the one that leaves a named open question, listed above.
diff --git a/wip/migrator-write-scope-and-rename-fidelity.md b/wip/migrator-write-scope-and-rename-fidelity.md
new file mode 100644
index 000000000..7b5eee619
--- /dev/null
+++ b/wip/migrator-write-scope-and-rename-fidelity.md
@@ -0,0 +1,49 @@
+# Deferred from the Phase 3 review: write scope, rename fidelity, and a backup name this run did not claim
+
+Three findings from the `/code-review high` pass over the Phase 3 staged changes were deliberately not fixed. Each is a decision rather than a defect, and each is written down here so the decision is taken on purpose rather than by omission. The other findings from that pass — the narrowed-path spelling and the rescue-copy promise — were fixed in the same change.
+
+## 1. The migration runner follows a symlink out of the directories it was told to walk
+
+`migrate_file` in `pipelex/migration/runner.py` calls `read_file_snapshot(file_path.resolve())`, so a configuration file that is a symlink is read, backed up and atomically replaced **at the link's target**. That resolution is deliberate and documented — replacing the link path would put a regular file where the link was and leave the real file unmigrated — and it is what the `.mthds` fix loop does with its own targets.
+
+The difference is what the fix loop does *next*. `pipelex/pipeline/fixes/fix_loop.py` pairs its `.resolve()` with `_partition_by_write_scope` / `is_target_in_write_scope`, so a resolved target that lands outside the directories the loop was given is refused rather than written. The migration runner has no equivalent, so:
+
+> `.pipelex/pipelex.toml` is a symlink to `/somewhere/else/config.toml`. The run rewrites `/somewhere/else/config.toml` and drops a `.bak.<stamp>` beside it — a file and a directory the walk never named.
+
+**Why it is not fixed here.** The behaviour is arguably right: the file the user means *is* the one at the end of the link, and a dotfiles-managed configuration is exactly the case symlink-following exists for. Refusing it would break the setup the resolution was added to support. But "we write outside the walked directories" is a claim worth making on purpose, and today it is made by accident — the docs asserted the two callers agreed, which on this half they did not.
+
+**The decision to take.** One of:
+
+- **Keep it, and say so.** The run's write scope is "the resolved target of any file the walk claims", full stop. Cheapest, and consistent with the reason the resolution exists. The doc sentence in `docs/migration-ledger.md` → "Backups" has already been trimmed to state this honestly.
+- **Guard it like the fix loop.** Add a write-scope check against the walked configuration directories and block a resolved target outside them as `UNWRITABLE`. This makes the two callers genuinely agree, at the cost of refusing the dotfiles case unless the real directory is also walked.
+
+## 2. A rename still rewrites the assignment's spacing and drops a deliberate quoting
+
+`_renamed_key` in `pipelex/pipeline/fixes/applier.py` builds `SingleKey(new_key)` and carries forward only the `_dotted` flag. The separator (`previous.sep`) and the key type (`previous.t`) are left to tomlkit's defaults, so:
+
+```toml
+old = 1 # keep me → new = 1 # keep me
+"old" = 1 → new = 1
+```
+
+The comment and the line's position survive — that is what the body-level rename was written to preserve, and it does preserve it. Column alignment and an intentionally quoted bare-safe key do not.
+
+**Why it is not fixed here.** It is not a regression: the previous `Container._replace` path did the same, and `_renamed_key`'s own docstring states the choice ("Everything else — quoting, separator — is left to tomlkit's own construction, so renaming an ordinary key yields exactly the key the library would have built for it"). But it sits against two claims made elsewhere in the same changeset — `docs/migration-ledger.md` → "A rename changes a name, and nothing else about the line", and the engine's "no canonical reflow, because a one-key rename must not rewrite a user's spacing" — and configuration migration, unlike the `.mthds` path, has no canonical reflow afterwards to normalise the result.
+
+**The decision to take.** Either carry `previous.sep` and `previous.t` forward alongside `_dotted` (a one-line change, and the claims then hold literally), or narrow the claims to what is actually promised: the *line* keeps its position and its comment, the *assignment* is renormalised. Worth checking what a carried-forward `sep` does when the old and new names differ in length — preserving the old padding around a longer name is not obviously better than renormalising it, and that is the real question behind this one.
+
+## 3. A backup name already taken is reported as this run's copy
+
+`write_backup` reserves its name with `O_CREAT | O_EXCL`; when the name is taken it returns `WrittenBackup(path=destination, was_created=False)` — a path it did not write and whose contents it never read. `_write_migrated_file` then commits and reports `backup_path=backup.path, was_written=True`, so the plan names a file this run did not produce.
+
+**Why it is not fixed.** This is the module's documented policy working exactly as intended: the stamp resolves to the second, two runs of the same file can address one name, and the copy already there is a copy of an *older* state — the original, if anything is. Naming it is correct and useful; the user can restore from it.
+
+The scenarios where it is *not* correct both require a file to be sitting at `<file>.bak.<this run's exact UTC second>` from a source other than a concurrent run — a hand-made backup that happens to match the stamp format to the second, or a zero-byte reservation left by a run killed inside the microsecond window between `_reserve_name` and the rename, found by a re-run within the same second. Guarding those would be a guard for a case that does not happen, which this codebase deliberately does not write.
+
+What *was* wrong on this path — the `state_uncertain` report promising to keep a copy it had not moved out of the pruning rotation — is fixed: `keep_backup_for_rescue` now returns `RescuedBackup(path, was_rescued)` and the report tells the user to take the copy now when it could not be secured.
+
+**Residual worth knowing.** If this ever needs revisiting, the honest lever is not a content check but the plan's vocabulary: `MigrationPlan.backup_path` currently means "where a copy of the pre-migration file is", not "where this run wrote one", and nothing in the report distinguishes the two.
+
+## Residual, unrelated to the three above
+
+`_declared_paths_this_file_carries` now reports the spelling the end of the replay leaves. Where a later `safe` entry *deletes* the material, there is no later spelling and the report falls back to the spelling the file carried when the entry was reached — a key the migrated file no longer has. Left alone on purpose: `test_material_a_later_entry_retires_leaves_nothing_to_say_to_a_file_migrated_that_far` pins that such a file is still reported, and changing what happens there is a decision about the reporting *predicate*, not about which spelling to name.