Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Migration keeps a comment banner on the section it introduces (#1118)

Louis Choquele5a189bfdf3d
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 32dac1353..a74f3a967 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,11 @@
# Changelog
+## [Unreleased]
+
+### Fixed
+
+- **A migrated section keeps its comment banner:** `pipelex migrate` moved tables and keys but not the own-line comments written above them, so a file seeded from the commented kit template came out with every section banner labelling the wrong section — the banner stayed behind while its section moved to the end of the file, and the moved section carried the *next* section's banner away with it. The applier now reads a run of own-line comments and blank lines as introducing whatever follows: a moved key or table takes its introduction along, a deleted one drops it, the introduction of whatever came next stays put, and an item appended under an existing table — moved there, or newly written by a fix — lands before that table's trailing banner rather than after it. A key written as a dotted assignment (`pipelex.storage_config = { … }`) is introduced by the comment above it, the same as one written under a header. A file preamble at the top stays where it is. The same rule applies to the `.mthds` fix path, where a comment above a stripped native-concept redeclaration used to dangle onto the next table and now goes with it. Files already migrated by v0.46.0 are not rewritten — the comments in them are a matter of hand-tidying, or of restoring the `.bak` beside each file and running `pipelex migrate` again.
+
## [v0.46.0] - 2026-08-17
### Added
diff --git a/docs/migration-ledger.md b/docs/migration-ledger.md
index 0c3cc14c4..cc417aeb6 100644
--- a/docs/migration-ledger.md
+++ b/docs/migration-ledger.md
@@ -651,6 +651,14 @@ Stated more strongly than the library can promise on its own: **a replay in whic
Refusing the layout with a `CONFLICT` was the alternative and it is the wrong trade: a rename has exactly one correct answer on a dotted key, the layout is ordinary TOML that no formatter rewrites, and refusing would strand a reshape's table renames on any file that happens to be written that way. So the applier renames the document's body entries where they sit — every chunk of a key written on several dotted lines, carrying the flag forward — which also stops the library from injecting a cosmetic blank line after a table it renamed in place.
+### A comment goes with what it introduces
+
+> **A run of own-line comments and blank lines introduces whatever comes next in the file, and a structural operation keeps it there: a moved key or table takes its introduction along, a deleted one takes its introduction away, and the introduction of whatever came *after* stays where it was.**
+
+The TOML library ties neither of them to the item a reader sees them on. A comment written above a key sits before that key in the same table; a banner written above a `[table]` header sits at the *tail of the previous table in document order*, inside its deepest last container, because everything up to the next header belongs to the table being parsed. So a plain delete-and-re-add of a table left its banner behind — now labelling whatever followed — and carried the *next* section's banner away inside the moved body; appending under an existing table landed *after* that table's trailing banner and stole it the same way. A file seeded from the heavily-commented kit template and migrated by the configuration reshape came out with `# Log Config` above the storage section and `# Plugins` above something else entirely.
+
+The applier now applies the rule a reader applies. What travels or is dropped with an item is **the last block of own-line comments before it, with the blank line above that block and anything below it**; an earlier block in the same run — a file preamble above the first section's banner, a closing note under the previous section — stays where it was, and a run holding no comment at all is spacing and moves whole. At the very top of the document, a *lone* comment block that a blank line separates from what follows is the file's preamble and stays too. An item inserted at the end of a table lands before that table's trailing run, never after it, and everything is put back where the parser would have put it — a previous table's deepest tail rather than an implicit parent's own body — so no `[parent]` header that the file never wrote starts rendering because a comment now sits inside it. The cost of the rule is the rare own-line comment written *below* a key to annotate it, which reads as introducing whatever comes next. Comments *inside* a moved table travel with it as before, and inline tables are left alone.
+
### The document is re-read between operations that applied
> **Operations are applied one at a time, and the document is parsed afresh after each one that changed it.**
@@ -688,7 +696,7 @@ Each file is written transactionally and independently: snapshot, stage, atomic
These are measured properties of the TOML library the engine uses, not aspirations. They are stated here because a user or an author will run into them, and being surprised by them is worse than being told.
- **Arrays of tables are unaddressable, and reserved.** An `[[entry]]` node is a list, and no `table_path` segment syntax reaches it or anything under it; the `*` wildcard has no array form. One surface carries one today — the telemetry configuration's `otlp` list of exporters — and the fingerprint records it as a terminal `list[table]`: what happens *inside* an exporter entry (a renamed field, a tightened bound, a new required member) is invisible to the coverage gate, so a change there is the author's to account for by hand, exactly as validator-expressed narrowing is. Addressing what lies beneath one needs a new segment kind and a decision on the day — until then, an operation pointed at one resolves nothing and is reported as a guarded skip rather than raising.
-- **A moved table loses its introducing comment.** A comment block written above a table is stored as trailing trivia of the *previous* sibling, not as part of the table it appears to introduce, so it does not travel with a move. Comments *inside* a moved table travel intact, including trailing comments on individual keys. The consequence is real: a file seeded from a heavily-commented template and then migrated ends up with banner comments labelling sections that have moved away. Nothing in the engine may rely on comment fidelity across a move, and the package's own files are corrected by hand rather than by migration.
+- **Comment ownership is a reading, not a fact.** The TOML library records where a comment *sits*, never what it is *about*, and the applier reads a run of own-line comments as introducing whatever follows it (see [A comment goes with what it introduces](#a-comment-goes-with-what-it-introduces)). That reading is right for a banner above a section, for the kit's per-key comments, and for a preamble at the top of a file; it is wrong for a comment written on its own line *below* a key to annotate it, which will follow the next item instead. A migration keeps a file's comments on the sections and keys they introduce — it does not, and cannot, know what any comment means.
- **A migration is not byte-minimal.** A rename normalizes the spacing around its own `=`, and a renamed table that was written out of order across several chunks loses its own bare header while a plain table keeps it. Both forms are semantically identical, both are stable under replay, and neither accumulates — but a migration diff is not always the minimal diff a human would have written.
- **A guarded skip is never an error.** Every operation whose target does not resolve reports itself skipped rather than raising. That guard is what makes always-replay possible, and it is why a misdirected operation is caught by the checks rather than by a crash on a user's machine.
- **A renamed key cannot be addressed again in the same in-memory document.** The position-preserving rename updates the body the document renders from but leaves the node's raw `dict` storage stale, for any value that is not a table. Nothing the library renders or looks up is affected — which is why a single rename is correct, and why the `.mthds` fix path, whose renames target tables, never meets it. The engine [re-reads the document between operations that applied](#the-document-is-re-read-between-operations-that-applied), so nothing downstream has to know about this; it is recorded here because it explains a piece of the engine that would otherwise look like superstition.
diff --git a/pipelex/migration/ledgers/pipelex-config.toml b/pipelex/migration/ledgers/pipelex-config.toml
index dfe51c301..cc12a4a94 100644
--- a/pipelex/migration/ledgers/pipelex-config.toml
+++ b/pipelex/migration/ledgers/pipelex-config.toml
@@ -95,9 +95,8 @@ moving it somewhere else:
- `boot_orchestrator`, if anyone ever wrote it into a file, is dropped because it is a boot \
argument (`--orchestrator`, or `Pipelex.setup(boot_orchestrator=...)`), not a setting.
-A moved section may appear at the end of the migrated file rather than where it was, and a banner \
-comment introducing a moved section stays behind, now labelling whatever followed it. Both are \
-expected; the file is otherwise untouched.
+A moved section may appear at the end of the migrated file rather than where it was, and it takes \
+the comment banner that introduced it along. That is expected; the file is otherwise untouched.
"""
# --- Kernel-layer infrastructure out of [pipelex], into [runtime] ------------------------------
diff --git a/pipelex/pipeline/fixes/applier.py b/pipelex/pipeline/fixes/applier.py
index bb0b0c9eb..a479e6677 100644
--- a/pipelex/pipeline/fixes/applier.py
+++ b/pipelex/pipeline/fixes/applier.py
@@ -4,7 +4,9 @@ Two clearly-separated steps, because they preserve different things — do not c
- ``apply_fix_ops`` mutates the tomlkit DOM **in place**, never rebuilding containers, so at the
DOM level tomlkit keeps the comments, ordering, and table style of untouched content (and of
- the patched line itself) by construction. A caller that wants the mutated-but-unformatted DOM
+ the patched line itself) by construction. What tomlkit does *not* do on its own is keep a
+ comment on the item it introduces when that item moves or goes — the "Comment fidelity"
+ helpers below do that, see their heading. A caller that wants the mutated-but-unformatted DOM
stops here.
- ``serialize_and_format`` then serializes the whole DOM and hands it to the MTHDS formatter
(``pipelex_tools.format_mthds``) for the one canonical style. This is a **whole-file** reflow:
@@ -29,7 +31,7 @@ from pipelex_tools import format_mthds
from pydantic import BaseModel, ConfigDict, Field
from tomlkit import TOMLDocument
from tomlkit.container import Container, OutOfOrderTableProxy
-from tomlkit.items import AbstractTable, AoT, Item, Key, SingleKey, Table
+from tomlkit.items import AbstractTable, AoT, Comment, Item, Key, Null, SingleKey, Table, Whitespace
from pipelex.base_exceptions import PipelexUnexpectedError
from pipelex.suggested_fix import (
@@ -265,6 +267,500 @@ def _rehome_key_indexes(*, container: Container, key: str, renamed: SingleKey, i
dict.__setitem__(container, renamed.key, item.value) # noqa: PLC2801 # pyright: ignore[reportUnknownMemberType]
+# --- Comment fidelity: the trivia that introduces an item goes where the item goes -----------------
+#
+# tomlkit keeps own-line comments and blank lines as keyless body entries (``Comment``,
+# ``Whitespace``), positioned wherever the parser met them: a comment above a key sits before that
+# key in the same container, and a banner written above a ``[table]`` header sits at the *tail of
+# the previous table in document order* — inside its deepest last container — because everything up
+# to the next header belongs to the table being parsed. Nothing ties either run to the item it
+# visibly introduces, so a plain ``del`` + re-add of a table leaves its banner behind (now labelling
+# whatever followed) and carries the *next* table's banner away inside the moved body; appending
+# under an existing table lands *after* that table's trailing banner, stealing it the same way.
+#
+# The rule the helpers below implement is the one a reader applies: **the last block of own-line
+# comments before an item, with the blank line above it and anything below it, introduces that
+# item.** Earlier blocks in the same run — a file preamble, a note closing the previous section —
+# stay where they are; a run holding no comment at all is spacing and travels whole. A moved item
+# takes its introduction along and leaves its trailing run, the next item's introduction, where it
+# was; a deleted item drops its introduction and keeps its trailing run in place; an item inserted at
+# the end of a container lands before that container's trailing run, never after it. Trivia is
+# always put back where the parser would have put it — a previous table's deepest tail rather than
+# a super-table's own body — so no implicit ``[parent]`` header starts rendering because a comment
+# now sits inside it. Inline tables are left alone: their whitespace entries are layout.
+
+
+class _TriviaRun(NamedTuple):
+ """The positions of a run of comment/blank-line entries in one container's body, ascending."""
+
+ container: Container
+ positions: list[int]
+
+ def items(self) -> list[Item]:
+ return [self.container.body[position][1] for position in self.positions]
+
+
+class _Slot(NamedTuple):
+ """One key's place in the document: the concrete container holding it and its body positions.
+
+ ``owner`` is the table whose ``.value`` the container is — one chunk of it, for a key of an
+ out-of-order table — and ``None`` at the document root.
+ """
+
+ container: Container
+ positions: list[int]
+ owner: Table | None
+
+ @property
+ def first(self) -> int:
+ return self.positions[0]
+
+ @property
+ def last(self) -> int:
+ return self.positions[-1]
+
+
+def _is_trivia(*, item: Item) -> bool:
+ return isinstance(item, (Comment, Whitespace))
+
+
+def _last_container(*, item: Item) -> Container | None:
+ """The container whose tail is the tail of ``item`` in document order, or ``None`` for a leaf.
+
+ Only block tables and arrays of tables extend across lines; an inline table renders on its
+ key's line and never owns any trailing trivia.
+ """
+ if isinstance(item, Table):
+ return item.value
+ if isinstance(item, AoT):
+ return item.body[-1].value if item.body else None
+ if isinstance(item, OutOfOrderTableProxy):
+ return item._tables[-1].value # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+ return None
+
+
+def _renders_header(*, table: Table, key: Key | None) -> bool:
+ """Whether tomlkit writes a ``[header]`` line for ``table``, stored under ``key`` — mirrors ``Container._render_table``.
+
+ An implicit super table has none: ``[a.b]`` written without ``[a]``, or a dotted assignment
+ ``a.b = 1`` whose owning key is dotted, so its scalars render on their own lines. Trivia sitting
+ above such a table's first entry is, to a reader, above the first entry's own header.
+ """
+ if not table.is_super_table():
+ return True
+ if key is not None and key.is_dotted():
+ return False
+ body = table.value.body
+ if any(not isinstance(item, (Table, AoT, Whitespace, Null)) for _, item in body):
+ return True
+ return any(body_key is not None and body_key.is_dotted() for body_key, item in body if isinstance(item, Table))
+
+
+def _key_slot(*, parent_table: dict[str, Any], key: str) -> _Slot:
+ """Where ``key`` is stored beneath ``parent_table``.
+
+ A key written as several dotted chunks has several positions in one container, and a key of an
+ out-of-order table can sit in any of the proxy's chunks: the chunk holding the key's first
+ position is the one taken, since that is where the key's introduction is.
+ """
+ candidates: list[tuple[Container, Table | None]]
+ if isinstance(parent_table, OutOfOrderTableProxy):
+ candidates = [(sub_table.value, sub_table) for sub_table in parent_table._tables if key in sub_table] # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+ elif isinstance(parent_table, Table):
+ candidates = [(parent_table.value, parent_table)]
+ elif isinstance(parent_table, Container):
+ candidates = [(parent_table, None)]
+ else:
+ msg = f"cannot locate key '{key}' in unsupported tomlkit node type '{type(parent_table).__name__}' — applier bug"
+ raise PipelexUnexpectedError(msg)
+ for container, owner in candidates:
+ position = container._map.get(SingleKey(key)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+ if position is None:
+ continue
+ positions = sorted(position) if isinstance(position, tuple) else [position]
+ return _Slot(container=container, positions=positions, owner=owner)
+ msg = f"key '{key}' is in the parent's facade but in none of its body entries — applier bug"
+ raise PipelexUnexpectedError(msg)
+
+
+def _slot_chain(*, toml_doc: TOMLDocument, path: list[str]) -> list[_Slot]:
+ """The slot of every segment of ``path``, root first — the ancestry a climb through implicit parents needs.
+
+ Stops short at an inline table: nothing beneath one carries introductions.
+ """
+ chain: list[_Slot] = []
+ node = cast("dict[str, Any]", toml_doc)
+ for segment in path:
+ if not _carries_trivia(parent_table=node):
+ break
+ chain.append(_key_slot(parent_table=node, key=segment))
+ node = cast("dict[str, Any]", node[segment])
+ return chain
+
+
+def _carries_trivia(*, parent_table: dict[str, Any]) -> bool:
+ """Whether the parent is a container whose comments and blank lines are introductions.
+
+ An inline table's whitespace entries are its layout, so the fidelity rule stays out of it.
+ """
+ return isinstance(parent_table, (Table, Container, OutOfOrderTableProxy))
+
+
+def _run_before(*, container: Container, index: int) -> _TriviaRun:
+ """The trivia run right before the body entry at ``index`` — possibly inside a previous table.
+
+ Walks back over comments and blank lines (``Null`` placeholders left by an earlier removal are
+ transparent). When nothing was collected and the entry before is a table, the run is that
+ table's trailing run in document order, so the walk descends into its deepest last container.
+ Trivia found at this level wins over a deeper tail: it is what the parser put here.
+ """
+ positions: list[int] = []
+ for position in range(index - 1, -1, -1):
+ _, item = container.body[position]
+ if isinstance(item, Null):
+ continue
+ if _is_trivia(item=item):
+ positions.insert(0, position)
+ continue
+ if not positions:
+ deeper = _last_container(item=item)
+ if deeper is not None:
+ return _trailing_run(container=deeper)
+ break
+ return _TriviaRun(container=container, positions=positions)
+
+
+def _run_after(*, container: Container, index: int) -> _TriviaRun:
+ """The trivia run right after the body entry at ``index``, at this level only."""
+ positions: list[int] = []
+ for position in range(index + 1, len(container.body)):
+ _, item = container.body[position]
+ if isinstance(item, Null):
+ continue
+ if not _is_trivia(item=item):
+ break
+ positions.append(position)
+ return _TriviaRun(container=container, positions=positions)
+
+
+def _first_entry_after(*, container: Container, index: int) -> Item | None:
+ """The first body entry after ``index`` that is neither a ``Null`` placeholder nor trivia."""
+ for position in range(index + 1, len(container.body)):
+ _, item = container.body[position]
+ if isinstance(item, Null) or _is_trivia(item=item):
+ continue
+ return item
+ return None
+
+
+def _trailing_run(*, container: Container) -> _TriviaRun:
+ """The trivia run at the tail of ``container`` in document order — its own, or its last table's."""
+ return _run_before(container=container, index=len(container.body))
+
+
+def _introduction(*, run: _TriviaRun, at_document_start: bool = False) -> _TriviaRun:
+ """The part of a run that introduces what follows it: the last comment block, its leading blank line, and what trails it.
+
+ A run with no comment is spacing and is taken whole. Earlier comment blocks stay put — a file
+ preamble above the first section's banner, or a closing note under the previous section. At the
+ very top of the document, a *lone* comment block that a blank line separates from what follows
+ is the file's preamble and stays too: only the blank line after it travels.
+ """
+ items = run.items()
+ last_comment = max((index for index, item in enumerate(items) if isinstance(item, Comment)), default=None)
+ if last_comment is None:
+ return run
+ start = last_comment
+ while start > 0 and isinstance(items[start - 1], Comment):
+ start -= 1
+ is_lone_block = not any(isinstance(item, Comment) for item in items[:start])
+ is_set_apart = last_comment + 1 < len(items) and isinstance(items[last_comment + 1], Whitespace)
+ if at_document_start and is_lone_block and is_set_apart:
+ return _TriviaRun(container=run.container, positions=run.positions[last_comment + 1 :])
+ if start > 0 and isinstance(items[start - 1], Whitespace):
+ start -= 1
+ return _TriviaRun(container=run.container, positions=run.positions[start:])
+
+
+def _at_document_start(*, run: _TriviaRun) -> bool:
+ """Whether the run is the first thing in the file — only a root-level run can be."""
+ if not isinstance(run.container, TOMLDocument):
+ return False
+ return not run.positions or _is_first_entry(container=run.container, index=run.positions[0])
+
+
+def _is_first_entry(*, container: Container, index: int) -> bool:
+ return all(isinstance(item, Null) for _, item in container.body[:index])
+
+
+def _owner_position(*, chain: list[_Slot]) -> int | None:
+ """Where the last link's owner sits in its parent's container — ``None`` at the root, or once it is gone.
+
+ Gone happens: deleting the last key of one chunk of an out-of-order table makes tomlkit drop the
+ emptied chunk from the parent, leaving a ``Null`` where the chunk was.
+ """
+ if len(chain) < 2:
+ return None
+ owner = chain[-1].owner
+ parent_container = chain[-2].container
+ for position in chain[-2].positions:
+ if parent_container.body[position][1] is owner:
+ return position
+ return None
+
+
+def _owner_key(*, chain: list[_Slot]) -> Key | None:
+ """The key the last link's owner is stored under in its parent — ``None`` at the root, or once it is gone."""
+ position = _owner_position(chain=chain)
+ if position is None:
+ return None
+ return chain[-2].container.body[position][0]
+
+
+def _implicit_owner_position(*, chain: list[_Slot]) -> int | None:
+ """Where the last link's owner sits in its own parent's container, when that owner renders no header.
+
+ That is the one situation in which the entries of a table are, to a reader, entries of whatever
+ encloses it: the trivia above the owner introduces the owner's first entry, and trivia meant for
+ the owner's first entry belongs above the owner. ``None`` when the owner writes a header (or is
+ the document root), or when the chain does not reach the parent.
+ """
+ if len(chain) < 2:
+ return None
+ owner = chain[-1].owner
+ if owner is None or _renders_header(table=owner, key=_owner_key(chain=chain)):
+ return None
+ position = _owner_position(chain=chain)
+ if position is None:
+ msg = "an out-of-order chunk is not among its parent's body positions — applier bug"
+ raise PipelexUnexpectedError(msg)
+ return position
+
+
+def _introduction_of_slot(*, chain: list[_Slot]) -> _TriviaRun:
+ """The introduction of the key at the end of ``chain``, climbing through implicit parents.
+
+ A key that is the first entry of a table with no rendered header — ``[runtime.storage]`` at the
+ top of a file that never writes ``[runtime]`` — is introduced by whatever sits above the parent.
+ """
+ slot = chain[-1]
+ run = _run_before(container=slot.container, index=slot.first)
+ if run.positions or not _is_first_entry(container=slot.container, index=slot.first):
+ return _introduction(run=run, at_document_start=_at_document_start(run=run))
+ owner_position = _implicit_owner_position(chain=chain)
+ if owner_position is None:
+ return _introduction(run=run, at_document_start=_at_document_start(run=run))
+ return _introduction_of_slot(chain=[*chain[:-2], chain[-2]._replace(positions=[owner_position])])
+
+
+def _shift_map(*, container: Container, from_index: int, delta: int) -> None:
+ """Keep the container's key→position index in step with a body insert or removal at ``from_index``."""
+ key_map = container._map # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+ for map_key, position in list(key_map.items()):
+ if isinstance(position, tuple):
+ key_map[map_key] = tuple(chunk + delta if chunk >= from_index else chunk for chunk in position)
+ elif position >= from_index:
+ key_map[map_key] = position + delta
+
+
+def _detach_run(*, run: _TriviaRun) -> list[Item]:
+ """Remove the run's entries from their container's body, returning them in document order."""
+ detached: list[Item] = []
+ for position in reversed(run.positions):
+ _, item = run.container.body.pop(position)
+ detached.insert(0, item)
+ _shift_map(container=run.container, from_index=position, delta=-1)
+ return detached
+
+
+def _insert_trivia_at(*, container: Container, index: int, items: Sequence[Item]) -> None:
+ for offset, item in enumerate(items):
+ container.body.insert(index + offset, (None, item))
+ if items:
+ _shift_map(container=container, from_index=index, delta=len(items))
+
+
+def _tail_slot(*, container: Container) -> tuple[Container, int]:
+ """Where an entry appended to ``container`` lands last in document order: after its last table's body."""
+ for position in range(len(container.body) - 1, -1, -1):
+ _, item = container.body[position]
+ if isinstance(item, Null):
+ continue
+ deeper = _last_container(item=item)
+ if deeper is not None:
+ return _tail_slot(container=deeper)
+ break
+ return container, len(container.body)
+
+
+def _insert_before(*, chain: list[_Slot], index: int, items: Sequence[Item]) -> None:
+ """Insert trivia so that it renders right before body entry ``index`` of the last chain link's container.
+
+ Put where the parser would have put it: at the tail of the previous table when there is one,
+ above an implicit parent when the entry is the first of a table with no rendered header, and at
+ this level otherwise.
+ """
+ if not items:
+ return
+ slot = chain[-1]
+ for position in range(index - 1, -1, -1):
+ _, item = slot.container.body[position]
+ if isinstance(item, Null):
+ continue
+ deeper = _last_container(item=item)
+ if deeper is None:
+ _insert_trivia_at(container=slot.container, index=index, items=items)
+ return
+ tail_container, tail_index = _tail_slot(container=deeper)
+ _insert_trivia_at(container=tail_container, index=tail_index, items=items)
+ return
+ owner_position = _implicit_owner_position(chain=chain)
+ if owner_position is not None:
+ _insert_before(chain=chain[:-1], index=owner_position, items=items)
+ return
+ _insert_trivia_at(container=slot.container, index=index, items=items)
+
+
+def _lift_out(*, toml_doc: TOMLDocument, parent_table: dict[str, Any], table_path: list[str], key: str) -> tuple[Item, list[Item]]:
+ """Remove ``key`` from its parent, keeping the document's comments on the items they introduce.
+
+ Returns the removed item and the trivia that introduced it. The removed item's own trailing run
+ — the introduction of whatever came next — is put back where the item was, so the next item
+ keeps its comment; the returned introduction is what a caller re-homes with the item, or drops
+ with it. On a parent that carries no trivia, this is a plain deletion.
+ """
+ moved_value = cast("Item", parent_table[key])
+ if not _carries_trivia(parent_table=parent_table):
+ del parent_table[key]
+ return moved_value, []
+ chain = _slot_chain(toml_doc=toml_doc, path=[*table_path, key])
+ introduction = _detach_run(run=_introduction_of_slot(chain=chain))
+ last_container = _last_container(item=moved_value)
+ trailing = _detach_run(run=_introduction(run=_trailing_run(container=last_container))) if last_container is not None else []
+ # Detaching the introduction may have shifted the key's own positions in the same container.
+ chain = _slot_chain(toml_doc=toml_doc, path=[*table_path, key])
+ resting = chain[-1].last
+ owner_position = _owner_position(chain=chain)
+ opened_the_file = _opens_the_document(chain=chain)
+ del parent_table[key] # ``Container.remove`` leaves a ``Null`` in each slot, so positions hold.
+ if owner_position is not None and _owner_position(chain=chain) is None:
+ # The key was the last one of a chunk of an out-of-order table, and tomlkit dropped the
+ # emptied chunk from the parent: what came after the key now comes right after that slot.
+ _insert_before(chain=chain[:-1], index=owner_position + 1, items=trailing)
+ else:
+ _insert_before(chain=chain, index=resting + 1, items=trailing)
+ if opened_the_file:
+ _drop_leading_blank_line(toml_doc=toml_doc)
+ return moved_value, introduction
+
+
+def _opens_the_document(*, chain: list[_Slot]) -> bool:
+ """Whether the key at the end of ``chain`` is the first thing in the file, in document order.
+
+ It is when it comes first in its container and every parent on the way is itself first in
+ its own, rendering no header — ``[runtime.storage]`` at the top of a file that never writes
+ ``[runtime]``, or a dotted assignment opening the file.
+ """
+ for depth, link in enumerate(chain):
+ if not _is_first_entry(container=link.container, index=link.first):
+ return False
+ if depth > 0 and _implicit_owner_position(chain=chain[: depth + 1]) is None:
+ return False
+ return True
+
+
+def _drop_leading_blank_line(*, toml_doc: TOMLDocument) -> None:
+ """A file does not open on a blank line: the one that separated the removed first entry from the next goes."""
+ _drop_leading_blank_line_in(container=toml_doc)
+
+
+def _drop_leading_blank_line_in(*, container: Container) -> bool:
+ """Walk ``container`` in document order to its first rendered line, dropping it when it is blank.
+
+ A table that renders no header — an implicit parent emptied by the removal, or one whose first
+ entry is next — is walked through, since what opens the file is inside it. Returns whether the
+ walk met anything that renders; ``False`` means the container renders nothing at all.
+ """
+ for position, (body_key, item) in enumerate(container.body):
+ if isinstance(item, Null):
+ continue
+ if isinstance(item, Whitespace):
+ _detach_run(run=_TriviaRun(container=container, positions=[position]))
+ return True
+ if isinstance(item, Table) and not _renders_header(table=item, key=body_key):
+ if _drop_leading_blank_line_in(container=item.value):
+ return True
+ continue
+ return True
+ return False
+
+
+def _settle_inserted(*, toml_doc: TOMLDocument, table_path: list[str], key: str, introduction: Sequence[Item]) -> None:
+ """Give a just-inserted ``key`` its introduction, and the trivia around it back to what it introduces.
+
+ Every insertion goes through here — a moved item, a key ``set_key`` adds, a table ``ensure_table``
+ creates — with an empty introduction when there is nothing to carry. tomlkit appends a table
+ after everything, including the trailing run that introduced whatever came after the container
+ in the file — so that run is moved past the new item, to become the new item's own trailing run.
+ A key it inserts *before the first table* of a container can land in the middle of a run —
+ between a file's preamble and the blank line under it — so the run on both sides is read as
+ one: what introduces the next item goes below the key, what does not (the preamble) goes back
+ above it. The introduction is then inserted right above the item. A carried introduction brings
+ the blank line the source had with it, so the newline tomlkit puts before an appended table's
+ header is dropped in its favour.
+ """
+ chain = _slot_chain(toml_doc=toml_doc, path=[*table_path, key])
+ if len(chain) < len(table_path) + 1:
+ return # beneath an inline table: nothing to settle
+ slot = chain[-1]
+ inserted = slot.container.body[slot.first][1]
+ before = _run_before(container=slot.container, index=slot.first)
+ after = _run_after(container=slot.container, index=slot.first)
+ at_document_start = _at_document_start(run=before)
+ if after.positions and before.container is slot.container:
+ combined = _TriviaRun(container=slot.container, positions=[*before.positions, *after.positions])
+ travelling = _introduction(run=combined, at_document_start=at_document_start).positions
+ moving_down = _TriviaRun(container=slot.container, positions=[pos for pos in before.positions if pos in travelling])
+ moving_up = _TriviaRun(container=slot.container, positions=[pos for pos in after.positions if pos not in travelling])
+ else:
+ moving_down = _introduction(run=before, at_document_start=at_document_start)
+ moving_up = _TriviaRun(container=slot.container, positions=[])
+ # Above-the-item positions come first in the body, so detaching the ones below first leaves them valid.
+ up_items = _detach_run(run=moving_up)
+ down_items = _detach_run(run=moving_down)
+ chain = _slot_chain(toml_doc=toml_doc, path=[*table_path, key])
+ slot = chain[-1]
+ if up_items:
+ _insert_trivia_at(container=slot.container, index=slot.first, items=up_items)
+ chain = _slot_chain(toml_doc=toml_doc, path=[*table_path, key])
+ slot = chain[-1]
+ if down_items:
+ last_container = _last_container(item=inserted)
+ if last_container is None:
+ # tomlkit gives the table that now follows an inserted key a newline of indent; that blank
+ # line belongs above the run going back under the key, not between the run and the header.
+ following = _first_entry_after(container=slot.container, index=slot.last)
+ if isinstance(following, Table) and following.trivia.indent:
+ if not isinstance(down_items[0], Whitespace):
+ down_items = [Whitespace(following.trivia.indent), *down_items]
+ following.trivia.indent = ""
+ _insert_trivia_at(container=slot.container, index=slot.last + 1, items=down_items)
+ else:
+ tail_container, tail_index = _tail_slot(container=last_container)
+ _insert_trivia_at(container=tail_container, index=tail_index, items=down_items)
+ if introduction:
+ # tomlkit sets a table's indent to a newline when it appends it under content that does not
+ # already end on a blank line. That blank line belongs *above* the carried introduction, not
+ # between the introduction and the header, so it moves ahead of it unless the introduction
+ # brought its own.
+ lead: list[Item] = []
+ if isinstance(inserted, Table) and inserted.trivia.indent and not isinstance(introduction[0], Whitespace):
+ lead = [Whitespace(inserted.trivia.indent)]
+ _insert_before(chain=chain, index=slot.first, items=[*lead, *introduction])
+ if isinstance(inserted, Table):
+ inserted.trivia.indent = ""
+
+
def _as_tomlkit_value(value: TomlValue | None) -> Any:
"""Convert a mapping value to an inline table (`inputs = { ... }`, the dominant authoring
form), so a freshly-created mapping stays attached to its pipe rather than being emitted as
@@ -421,7 +917,11 @@ def _apply_at_table_path(*, toml_doc: TOMLDocument, fix_op: FixOp, table_path: l
target_table = _resolve_table(toml_doc=toml_doc, table_path=table_path)
if target_table is None:
return _OpResult(FixOpOutcome.SKIPPED, f"table '{table_path_str}' not found in document")
+ was_absent = fix_op.key not in target_table
target_table[fix_op.key] = _as_tomlkit_value(fix_op.value)
+ if was_absent:
+ # A new key lands before the table's trailing run — the next section's banner — not after it.
+ _settle_inserted(toml_doc=toml_doc, table_path=table_path, key=fix_op.key, introduction=[])
return _OpResult(FixOpOutcome.APPLIED)
case EnsureTableOp():
existing_table = _resolve_table(toml_doc=toml_doc, table_path=table_path)
@@ -438,6 +938,7 @@ def _apply_at_table_path(*, toml_doc: TOMLDocument, fix_op: FixOp, table_path: l
# found" this branch used to report.
return _OpResult(FixOpOutcome.CONFLICT, f"'{table_key}' is already present in '{'.'.join(table_path[:-1])}' and is not a table")
parent_table[table_key] = tomlkit.inline_table()
+ _settle_inserted(toml_doc=toml_doc, table_path=table_path[:-1], key=table_key, introduction=[])
return _OpResult(FixOpOutcome.APPLIED)
case DeleteKeyOp():
target_table = _resolve_table(toml_doc=toml_doc, table_path=table_path)
@@ -445,7 +946,8 @@ def _apply_at_table_path(*, toml_doc: TOMLDocument, fix_op: FixOp, table_path: l
return _OpResult(FixOpOutcome.SKIPPED, f"table '{table_path_str}' not found in document")
if fix_op.key not in target_table:
return _OpResult(FixOpOutcome.SKIPPED, f"key '{fix_op.key}' not found in table '{table_path_str}'")
- del target_table[fix_op.key]
+ # The comment above the key goes with it; the one above the next key stays there.
+ _lift_out(toml_doc=toml_doc, parent_table=target_table, table_path=table_path, key=fix_op.key)
return _OpResult(FixOpOutcome.APPLIED)
case DeleteTableOp():
parent_table = _resolve_table(toml_doc=toml_doc, table_path=table_path[:-1])
@@ -454,7 +956,8 @@ def _apply_at_table_path(*, toml_doc: TOMLDocument, fix_op: FixOp, table_path: l
# (same guarded-skip contract _resolve_table enforces for every other segment).
if parent_table is None or not isinstance(parent_table.get(table_key), dict):
return _OpResult(FixOpOutcome.SKIPPED, f"table '{table_path_str}' not found in document")
- del parent_table[table_key]
+ # The banner introducing the table goes with it; the next section's banner stays put.
+ _lift_out(toml_doc=toml_doc, parent_table=parent_table, table_path=table_path[:-1], key=table_key)
return _OpResult(FixOpOutcome.APPLIED)
case RenameTableKeyOp():
parent_table = _resolve_table(toml_doc=toml_doc, table_path=table_path)
@@ -570,10 +1073,13 @@ def _apply_move_key(*, toml_doc: TOMLDocument, fix_op: MoveKeyOp, table_path: li
if existing_destination is not None and fix_op.new_key in existing_destination:
return _OpResult(FixOpOutcome.CONFLICT, f"cannot move to '{destination_path_str}.{fix_op.new_key}': already present")
- moved_value = cast("Item", source_table[fix_op.key])
- del source_table[fix_op.key]
- destination_table = _create_block_table_path(toml_doc=toml_doc, table_path=fix_op.new_table_path)
- destination_table[fix_op.new_key] = moved_value
+ moved_value, introduction = _lift_out(toml_doc=toml_doc, parent_table=source_table, table_path=table_path, key=fix_op.key)
+ destination = _create_block_table_path(toml_doc=toml_doc, table_path=fix_op.new_table_path)
+ destination.leaf[fix_op.new_key] = moved_value
+ # The introduction goes above whatever this operation put into a pre-existing container: the
+ # moved item itself, or the outermost parent created for it — that is the header a reader sees.
+ head = destination.head or _InsertionHead(table_path=fix_op.new_table_path, key=fix_op.new_key)
+ _settle_inserted(toml_doc=toml_doc, table_path=head.table_path, key=head.key, introduction=introduction)
_refresh_table_headers(item=moved_value)
return _OpResult(FixOpOutcome.APPLIED)
@@ -595,7 +1101,20 @@ def _first_non_table_segment(*, toml_doc: TOMLDocument, table_path: list[str]) -
return None
-def _create_block_table_path(*, toml_doc: TOMLDocument, table_path: list[str]) -> dict[str, Any]:
+class _InsertionHead(NamedTuple):
+ """A key just added to a pre-existing container — the outermost thing an insertion put in the file."""
+
+ table_path: list[str]
+ key: str
+
+
+class _DestinationPath(NamedTuple):
+ leaf: dict[str, Any]
+ head: _InsertionHead | None
+ """The first segment that had to be created, under the deepest node that already existed; ``None`` when every segment existed."""
+
+
+def _create_block_table_path(*, toml_doc: TOMLDocument, table_path: list[str]) -> _DestinationPath:
"""Walk ``table_path``, creating missing segments as block tables, and return the leaf node.
Block tables rather than inline ones, because this creates a *section* of a configuration
@@ -603,8 +1122,11 @@ def _create_block_table_path(*, toml_doc: TOMLDocument, table_path: list[str]) -
ruled out a non-table segment first; this function would silently overwrite one.
"""
node = cast("dict[str, Any]", toml_doc)
- for segment in table_path:
+ head: _InsertionHead | None = None
+ for depth, segment in enumerate(table_path):
if not isinstance(node.get(segment), dict):
node[segment] = tomlkit.table()
+ if head is None:
+ head = _InsertionHead(table_path=table_path[:depth], key=segment)
node = cast("dict[str, Any]", node[segment])
- return node
+ return _DestinationPath(leaf=node, head=head)
diff --git a/tests/unit/pipelex/pipeline/fixes/test_fix_applier_comment_fidelity.py b/tests/unit/pipelex/pipeline/fixes/test_fix_applier_comment_fidelity.py
new file mode 100644
index 000000000..548eeb9bc
--- /dev/null
+++ b/tests/unit/pipelex/pipeline/fixes/test_fix_applier_comment_fidelity.py
@@ -0,0 +1,373 @@
+"""Comment fidelity across the applier's structural operations — the banner that introduces a
+moved or deleted item goes where the item goes, and the banner of whatever came *next* stays put.
+
+tomlkit stores a comment block written above a table as trailing trivia of the *previous* table in
+document order (its deepest last container), and a comment above a key as trivia items sitting
+before that key in the same container. A plain ``del`` + re-add therefore leaves a moved table's
+banner behind — now labelling whatever followed — and carries the *next* table's banner away
+inside the moved body. Both halves are what a migrated configuration file used to show. The
+applier now reads every run of own-line comments and blank lines as introducing whatever comes
+next, and moves or drops that run with the item it introduces.
+
+Everything here is compared as serialized bytes without the MTHDS formatter, exactly as a
+configuration migration runs, so spacing is part of what is pinned.
+"""
+
+from pathlib import Path
+
+import pytest
+import tomlkit
+from tomlkit import TOMLDocument
+
+from pipelex.migration.engine import replay_ledger_over_text
+from pipelex.migration.ledger import load_ledger, packaged_migration_dir
+from pipelex.pipeline.fixes.applier import FixOpOutcome, apply_fix_ops
+from pipelex.suggested_fix import DeleteKeyOp, DeleteTableOp, EnsureTableOp, FixOp, MoveKeyOp, SetKeyOp
+
+_KIT_TEMPLATE = Path("pipelex/kit/configs/pipelex.toml")
+_PRE_RESHAPE_DEFAULTS = Path("pipelex/migration/goldens/pipelex-config/[email protected]")
+
+_THREE_SECTIONS = """\
+[a]
+x = 1
+
+# ---- Section B ----
+[b]
+y = 2
+
+# ---- Section C ----
+[c]
+z = 3
+"""
+
+_NESTED_DESTINATION = """\
+[c]
+z = 3
+
+[c.s]
+q = 1
+
+# ---- Section D ----
+[d]
+w = 4
+
+# ---- Section B ----
+[b]
+y = 2
+"""
+
+_PER_KEY_COMMENTS = """\
+[a]
+# about x
+x = 1
+# about y
+y = 2
+
+[b]
+w = 0
+
+# ---- Section C ----
+[c]
+z = 3
+"""
+
+_TABLE_WITH_SUBTABLE = """\
+[a]
+x = 1
+
+# ---- Section B ----
+[b]
+y = 2
+
+[b.sub]
+q = 1
+
+# ---- Section C ----
+[c]
+z = 3
+"""
+
+
+def _dumps(toml_doc: TOMLDocument) -> str:
+ """One typed funnel over tomlkit's weakly-typed ``dumps``."""
+ return tomlkit.dumps(toml_doc) # pyright: ignore[reportUnknownMemberType]
+
+
+def _apply(*, text: str, ops: list[FixOp]) -> tuple[str, list[FixOpOutcome]]:
+ toml_doc = tomlkit.parse(text)
+ applications = apply_fix_ops(toml_doc=toml_doc, ops=ops)
+ return _dumps(toml_doc), [application.outcome for application in applications]
+
+
+def _lines_above(*, text: str, header: str, count: int) -> list[str]:
+ """The ``count`` lines immediately above the one line equal to ``header``."""
+ lines = text.splitlines()
+ matches = [index for index, line in enumerate(lines) if line == header]
+ assert len(matches) == 1, f"expected exactly one '{header}' line, found {len(matches)}"
+ return lines[matches[0] - count : matches[0]]
+
+
+class TestFixApplierCommentFidelity:
+ def test_a_moved_table_takes_its_banner_and_leaves_the_next_banner_behind(self) -> None:
+ """The banner above ``[b]`` travels to ``[new.b]``; the banner above ``[c]`` stays above ``[c]``."""
+ text, outcomes = _apply(
+ text=_THREE_SECTIONS,
+ ops=[MoveKeyOp(table_path=[], key="b", new_table_path=["new"], new_key="b")],
+ )
+ assert outcomes == [FixOpOutcome.APPLIED]
+ assert text == ("[a]\nx = 1\n\n# ---- Section C ----\n[c]\nz = 3\n\n# ---- Section B ----\n[new.b]\ny = 2\n")
+
+ def test_a_table_moved_into_an_existing_parent_does_not_steal_the_parents_trailing_banner(self) -> None:
+ """Appending under ``[c]`` lands *before* the banner that introduces ``[d]``, not after it."""
+ text, outcomes = _apply(
+ text=_NESTED_DESTINATION,
+ ops=[MoveKeyOp(table_path=[], key="b", new_table_path=["c"], new_key="b")],
+ )
+ assert outcomes == [FixOpOutcome.APPLIED]
+ assert text == ("[c]\nz = 3\n\n[c.s]\nq = 1\n\n# ---- Section B ----\n[c.b]\ny = 2\n\n# ---- Section D ----\n[d]\nw = 4\n")
+
+ def test_a_moved_key_takes_the_comment_above_it_and_lands_before_the_destinations_trailing_banner(self) -> None:
+ """Per-key comments in the kit's style: the one above ``y`` follows ``y``; ``[c]`` keeps its banner."""
+ text, outcomes = _apply(
+ text=_PER_KEY_COMMENTS,
+ ops=[MoveKeyOp(table_path=["a"], key="y", new_table_path=["b"], new_key="y")],
+ )
+ assert outcomes == [FixOpOutcome.APPLIED]
+ assert text == ("[a]\n# about x\nx = 1\n\n[b]\nw = 0\n# about y\ny = 2\n\n# ---- Section C ----\n[c]\nz = 3\n")
+
+ def test_a_moved_table_with_a_sub_table_leaves_the_next_banner_where_it_was(self) -> None:
+ """The next banner lives in the deepest last container of the moved table; it still stays behind."""
+ text, outcomes = _apply(
+ text=_TABLE_WITH_SUBTABLE,
+ ops=[MoveKeyOp(table_path=[], key="b", new_table_path=["new"], new_key="b")],
+ )
+ assert outcomes == [FixOpOutcome.APPLIED]
+ assert text == ("[a]\nx = 1\n\n# ---- Section C ----\n[c]\nz = 3\n\n# ---- Section B ----\n[new.b]\ny = 2\n\n[new.b.sub]\nq = 1\n")
+
+ def test_a_deleted_table_drops_its_own_banner_and_keeps_the_next_one(self) -> None:
+ text, outcomes = _apply(text=_THREE_SECTIONS, ops=[DeleteTableOp(table_path=["b"])])
+ assert outcomes == [FixOpOutcome.APPLIED]
+ assert text == ("[a]\nx = 1\n\n# ---- Section C ----\n[c]\nz = 3\n")
+
+ def test_a_deleted_key_drops_the_comment_above_it_and_keeps_its_neighbours(self) -> None:
+ text, outcomes = _apply(text=_PER_KEY_COMMENTS, ops=[DeleteKeyOp(table_path=["a"], key="y")])
+ assert outcomes == [FixOpOutcome.APPLIED]
+ assert text == ("[a]\n# about x\nx = 1\n\n[b]\nw = 0\n\n# ---- Section C ----\n[c]\nz = 3\n")
+
+ def test_a_move_without_any_comment_around_it_is_unchanged_from_before(self) -> None:
+ """No trivia to carry: the placement and spacing tomlkit gives an appended table are kept."""
+ text, outcomes = _apply(
+ text="[a]\nx = 1\n\n[b]\ny = 2\n\n[c]\nz = 3\n",
+ ops=[MoveKeyOp(table_path=[], key="b", new_table_path=["new"], new_key="b")],
+ )
+ assert outcomes == [FixOpOutcome.APPLIED]
+ assert text == "[a]\nx = 1\n\n[c]\nz = 3\n\n[new.b]\ny = 2\n"
+
+ @pytest.mark.parametrize(
+ ("source", "ops", "expected"),
+ [
+ pytest.param(
+ "[a]\nx = 1\n\n[b]\ny = 2\n",
+ [MoveKeyOp(table_path=[], key="a", new_table_path=["new"], new_key="a")],
+ "[b]\ny = 2\n\n[new.a]\nx = 1\n",
+ id="removing the file's first table leaves no blank line at the top",
+ ),
+ pytest.param(
+ "# my config\n\n[a]\nx = 1\n\n[b]\ny = 2\n",
+ [MoveKeyOp(table_path=[], key="a", new_table_path=["new"], new_key="a")],
+ "# my config\n\n[b]\ny = 2\n\n[new.a]\nx = 1\n",
+ id="a lone comment block at the top, set apart by a blank line, is the file's preamble and stays",
+ ),
+ pytest.param(
+ "# my config\n# more\n\n# A banner\n[a]\nx = 1\n\n[b]\ny = 2\n",
+ [MoveKeyOp(table_path=[], key="a", new_table_path=["new"], new_key="a")],
+ "# my config\n# more\n\n[b]\ny = 2\n\n# A banner\n[new.a]\nx = 1\n",
+ id="a preamble stays while the banner below it travels",
+ ),
+ pytest.param(
+ "# A banner\n[a]\nx = 1\n\n[b]\ny = 2\n",
+ [MoveKeyOp(table_path=[], key="a", new_table_path=["new"], new_key="a")],
+ "[b]\ny = 2\n\n# A banner\n[new.a]\nx = 1\n",
+ id="a banner glued to the first header travels, and gets the blank line tomlkit gives an appended table",
+ ),
+ pytest.param(
+ "# root note\nname = 'x'\n\n[a]\nx = 1\n",
+ [MoveKeyOp(table_path=[], key="name", new_table_path=["a"], new_key="name")],
+ "[a]\nx = 1\n# root note\nname = 'x'\n",
+ id="a comment glued to the file's first key travels with the key",
+ ),
+ pytest.param(
+ "# preamble\n\n[a]\n# about x\nx = 1\n",
+ [MoveKeyOp(table_path=["a"], key="x", new_table_path=[], new_key="x")],
+ "# preamble\n# about x\nx = 1\n\n[a]\n",
+ id="a key moved to the root of a file lands under the preamble, not above it",
+ ),
+ pytest.param(
+ "[t]\nz = 0\n# note on sub\n\n[t.sub]\nq = 1\n\n[a]\n# about x\nx = 1\n",
+ [MoveKeyOp(table_path=["a"], key="x", new_table_path=["t"], new_key="x")],
+ "[t]\nz = 0\n# about x\nx = 1\n# note on sub\n\n[t.sub]\nq = 1\n\n[a]\n",
+ id="a key inserted before a table's first sub-table takes its comment and leaves the sub-table's note on the sub-table",
+ ),
+ pytest.param(
+ "[t]\n# note on sub\n[t.sub]\nq = 1\n\n[a]\nx = 1\n",
+ [MoveKeyOp(table_path=["a"], key="x", new_table_path=["t"], new_key="x")],
+ "[t]\nx = 1\n\n# note on sub\n[t.sub]\nq = 1\n\n[a]\n",
+ id="the blank line tomlkit adds under an inserted key sits above the note that follows it, not between the note and its header",
+ ),
+ pytest.param(
+ "# preamble\n\n# ---- Storage ----\n\n[runtime.storage]\nm = 1\n\n# ---- Secrets ----\n\n[runtime.secrets]\ns = 1\n",
+ [MoveKeyOp(table_path=["runtime"], key="storage", new_table_path=["relocated"], new_key="storage")],
+ "# preamble\n\n# ---- Secrets ----\n\n[runtime.secrets]\ns = 1\n\n# ---- Storage ----\n\n[relocated.storage]\nm = 1\n",
+ id="the first entry of an implicit parent is introduced by what sits above the parent, and no bare [runtime] header appears",
+ ),
+ pytest.param(
+ "[t]\n\n[a]\nx = 1\n\n# B banner\n[b]\ny = 2\n\n[c]\nz = 3\n",
+ [MoveKeyOp(table_path=[], key="b", new_table_path=["t"], new_key="b")],
+ "[t]\n\n# B banner\n[t.b]\ny = 2\n\n[a]\nx = 1\n\n[c]\nz = 3\n",
+ id="moved into an empty explicit table, the banner sits under that table's header",
+ ),
+ pytest.param(
+ "[a]\nx = 1\n\n# B banner\n[b]\ny = 2\n\n# C banner\n[c]\nz = 3\n",
+ [MoveKeyOp(table_path=[], key="b", new_table_path=["a", "new"], new_key="b")],
+ "[a]\nx = 1\n\n# B banner\n[a.new.b]\ny = 2\n\n# C banner\n[c]\nz = 3\n",
+ id="a created intermediate parent under an existing one carries the banner above the outermost created header",
+ ),
+ pytest.param(
+ "[a]\nx = 1\n\n# exporters\n[[otlp]]\nendpoint = 1\n\n[[otlp]]\nendpoint = 2\n\n# next\n[next]\nz = 3\n",
+ [MoveKeyOp(table_path=[], key="otlp", new_table_path=["tel"], new_key="otlp")],
+ "[a]\nx = 1\n\n# next\n[next]\nz = 3\n\n# exporters\n[[tel.otlp]]\nendpoint = 1\n\n[[tel.otlp]]\nendpoint = 2\n",
+ id="an array of tables travels with its banner and leaves the next banner behind",
+ ),
+ pytest.param(
+ "[deck.gpt]\n# tier of gpt\ntier = 1\nprovider = 'o'\n\n"
+ "[deck.claude]\n# tier of claude\ntier = 2\nprovider = 'a'\n\n# after\n[levels]\nx = 1\n",
+ [DeleteKeyOp(table_path=["deck", "*"], key="tier")],
+ "[deck.gpt]\nprovider = 'o'\n\n[deck.claude]\nprovider = 'a'\n\n# after\n[levels]\nx = 1\n",
+ id="a wildcard op applies to every match on one document without losing track of positions",
+ ),
+ pytest.param(
+ "[a]\nx = 1\n\n# B banner\n[b]\ny = 2\n\n# end of file note\n",
+ [MoveKeyOp(table_path=[], key="b", new_table_path=["new"], new_key="b")],
+ "[a]\nx = 1\n\n# B banner\n[new.b]\ny = 2\n\n# end of file note\n",
+ id="a note at the end of the file stays at the end of the file",
+ ),
+ pytest.param(
+ "[a]\nx = 1\n\n# B banner\n[b]\ny = 2\n",
+ [DeleteTableOp(table_path=["b"])],
+ "[a]\nx = 1\n",
+ id="deleting the last table of the file takes its banner and the blank line above it",
+ ),
+ pytest.param(
+ "# ---- Storage ----\n[runtime.storage]\nm = 1\n\n# ---- Secrets ----\n[runtime.secrets]\ns = 1\n",
+ [MoveKeyOp(table_path=["runtime"], key="storage", new_table_path=["relocated"], new_key="storage")],
+ "# ---- Secrets ----\n[runtime.secrets]\ns = 1\n\n# ---- Storage ----\n[relocated.storage]\nm = 1\n",
+ id="removing the file's first table from under an implicit parent leaves no blank line at the top",
+ ),
+ pytest.param(
+ "# preamble\n\n# ---- Storage ----\npipelex.storage_config = { x = 1 }\n\n# ---- Other ----\n[other]\nk = 1\n",
+ [MoveKeyOp(table_path=["pipelex"], key="storage_config", new_table_path=["runtime"], new_key="storage_config")],
+ "# preamble\n\n# ---- Other ----\n[other]\nk = 1\n\n# ---- Storage ----\n[runtime]\nstorage_config = { x = 1 }\n",
+ id="a dotted assignment at the root is introduced by what sits above it, since its implicit parent renders no header",
+ ),
+ pytest.param(
+ "# ---- Log ----\npipelex.log_config.level = 'INFO'\n\n# ---- Other ----\n[other]\nk = 1\n",
+ [MoveKeyOp(table_path=["pipelex"], key="log_config", new_table_path=["runtime"], new_key="log_config")],
+ "# ---- Other ----\n[other]\nk = 1\n\n# ---- Log ----\n[runtime.log_config]\nlevel = 'INFO'\n",
+ id="a deeper dotted assignment climbs through every header-less parent to its banner",
+ ),
+ pytest.param(
+ "# ---- Storage ----\npipelex.storage_config = { x = 1 }\n\n# ---- Other ----\n[other]\nk = 1\n",
+ [DeleteKeyOp(table_path=["pipelex"], key="storage_config")],
+ "# ---- Other ----\n[other]\nk = 1\n",
+ id="deleting a dotted-assignment key drops the banner above it",
+ ),
+ pytest.param(
+ "[a]\nx = 1\n\n# B banner\n[b]\ny = 2\n",
+ [SetKeyOp(table_path=["a"], key="z", value=3)],
+ "[a]\nx = 1\nz = 3\n\n# B banner\n[b]\ny = 2\n",
+ id="a key set into a table lands before the table's trailing banner, not after it",
+ ),
+ pytest.param(
+ "[a]\nx = 1\n\n# B banner\n[b]\ny = 2\n",
+ [EnsureTableOp(table_path=["a", "sub"])],
+ "[a]\nx = 1\nsub = {}\n\n# B banner\n[b]\ny = 2\n",
+ id="a table ensured under a table lands before the table's trailing banner, not after it",
+ ),
+ pytest.param(
+ "[a]\n# about x\nx = 1\n\n# B banner\n[b]\ny = 2\n",
+ [SetKeyOp(table_path=["a"], key="x", value=2)],
+ "[a]\n# about x\nx = 2\n\n# B banner\n[b]\ny = 2\n",
+ id="setting an existing key rewrites it in place and moves no comment",
+ ),
+ pytest.param(
+ "# A banner\n[a]\nx = 1\n",
+ [SetKeyOp(table_path=[], key="main_pipe", value="p")],
+ 'main_pipe = "p"\n\n# A banner\n[a]\nx = 1\n',
+ id="a root key set into a file whose first header carries a banner lands above the banner",
+ ),
+ pytest.param(
+ "[pipe.first]\na = 1\n\n# concepts\n[concept]\nIdea = 'x'\n\n[pipe.first.inputs]\nt = 'Text'\n",
+ [SetKeyOp(table_path=["pipe", "first"], key="output", value="Text")],
+ "[pipe.first]\na = 1\noutput = \"Text\"\n\n# concepts\n[concept]\nIdea = 'x'\n\n[pipe.first.inputs]\nt = 'Text'\n",
+ id="a key set into an out-of-order chunk lands in that chunk, before the next section's banner",
+ ),
+ pytest.param(
+ "[t]\nz = 0\n\n# note on sub\n[t.sub]\nq = 1\n\n[a]\n# about x\nx = 1\n",
+ [MoveKeyOp(table_path=["a"], key="x", new_table_path=["t"], new_key="x")],
+ "[t]\nz = 0\n# about x\nx = 1\n\n# note on sub\n[t.sub]\nq = 1\n\n[a]\n",
+ id="a trailing run that already opens on a blank line does not gain a second one from tomlkit's indent",
+ ),
+ ],
+ )
+ def test_edge_shapes(self, source: str, ops: list[FixOp], expected: str) -> None:
+ text, outcomes = _apply(text=source, ops=ops)
+ assert outcomes == [FixOpOutcome.APPLIED]
+ assert text == expected
+ tomlkit.parse(text)
+
+ def test_the_kit_template_keeps_every_banner_on_its_section_through_a_move(self) -> None:
+ """The real, heavily-commented template: the storage banner follows storage, the next stays."""
+ text, outcomes = _apply(
+ text=_KIT_TEMPLATE.read_text(encoding="utf-8"),
+ ops=[MoveKeyOp(table_path=["runtime"], key="storage", new_table_path=["relocated"], new_key="storage")],
+ )
+ assert outcomes == [FixOpOutcome.APPLIED]
+ rule = "#" * 100
+ assert _lines_above(text=text, header="[relocated.storage]", count=4) == [rule, "# Storage Config", rule, ""]
+ assert _lines_above(text=text, header="[runtime.secrets]", count=4) == [rule, "# Secrets Config", rule, ""]
+
+ def test_the_reshape_entry_carries_every_section_banner_with_its_section(self) -> None:
+ """The shipped `pipelex-config` ledger over the frozen pre-reshape defaults, banner by banner.
+
+ This is the file shape the bug was reported on: a document seeded from a heavily-commented
+ template, migrated by the configuration reshape. Every ``# <Name> config`` banner that
+ introduced a section before must introduce the same section after — wherever it moved to.
+ """
+ ledger = load_ledger(migration_dir=packaged_migration_dir(), surface_id="pipelex-config")
+ replay = replay_ledger_over_text(ledger=ledger, text=_PRE_RESHAPE_DEFAULTS.read_text(encoding="utf-8"))
+ assert replay.blocked == []
+ rule = "#" * 100
+ expected_headers_by_banner = {
+ # moved out of [pipelex] into a created root section
+ "# Log config": "[runtime.log]",
+ "# Tracing config (event log for distributed tracing across workers)": "[runtime.tracing]",
+ "# Templating config": "[inference.templating]",
+ "# Dry run config": "[inference.dry_run]",
+ # moved out of an out-of-order [pipelex] chunk to the document root
+ "# Kit Config": "[kit]",
+ # renamed in place
+ "# Cogt inference config": "[inference]",
+ "# Pipe run config": "[interpreter.pipe_run]",
+ "# Pipeline execution config": "[interpreter.pipeline_execution]",
+ }
+ for banner, header in expected_headers_by_banner.items():
+ assert _lines_above(text=replay.text, header=header, count=4) == [rule, banner, rule, ""], header
+ # The banner of a deleted section goes with it.
+ assert "# Migration config" not in replay.text
+ # And every banner in the migrated file introduces a header, not a stray key.
+ lines = replay.text.splitlines()
+ for index, line in enumerate(lines):
+ if line == rule and index + 2 < len(lines) and lines[index + 2] == rule and lines[index + 1] != "#":
+ # A banner in the last lines of the file introduces nothing: that is a failure, not a skip.
+ following = lines[index + 4] if index + 4 < len(lines) else "<end of file>"
+ assert following.startswith("["), f"banner {lines[index + 1]!r} is followed by {following!r}"
diff --git a/tests/unit/pipelex/pipeline/fixes/test_fix_applier_config_surface_shapes.py b/tests/unit/pipelex/pipeline/fixes/test_fix_applier_config_surface_shapes.py
index a2a3412eb..b12db25ec 100644
--- a/tests/unit/pipelex/pipeline/fixes/test_fix_applier_config_surface_shapes.py
+++ b/tests/unit/pipelex/pipeline/fixes/test_fix_applier_config_surface_shapes.py
@@ -14,9 +14,10 @@ the public ``apply_fix_ops`` — a ``delete_key`` for an absent key reports "tab
path model cannot reach the table and "key not found" when it can — so it characterizes the contract
rather than the private resolver.
-Where a behaviour is a hazard rather than a guarantee, the test says so in its name and docstring:
-``test_moved_table_leaves_its_banner_comment_behind`` pins a *limitation* that the migration contract
-has to state, not something to rely on.
+Where a behaviour is a hazard rather than a guarantee, the test says so in its name and docstring.
+``test_moved_table_takes_its_banner_comment_along`` pins the one comment-fidelity guarantee the
+applier makes on top of tomlkit — a banner travels with the table it introduces — and the byte-level
+cases behind it live in ``test_fix_applier_comment_fidelity.py``.
"""
from pathlib import Path
@@ -98,9 +99,10 @@ def _table_paths(node: object, prefix: list[str]) -> list[list[str]]:
def _move_storage_under_a_fresh_root(toml_doc: TOMLDocument) -> None:
"""Move ``runtime.storage`` to ``relocated.storage``, creating the destination root as it goes.
- The two placement tests need a move whose destination parent does *not* already exist, which is
- the case that makes tomlkit append at the end of the document. ``relocated`` is a name no
- configuration surface uses, so it can never collide with a real section.
+ The placement test needs a move whose destination parent does *not* already exist, which is
+ the case that makes tomlkit append at the end of the document; this is the raw library move,
+ without the applier's comment handling. ``relocated`` is a name no configuration surface uses,
+ so it can never collide with a real section.
"""
runtime_table = _as_mapping(toml_doc["runtime"])
moved = runtime_table["storage"]
@@ -430,13 +432,14 @@ class TestFixApplierConfigSurfaceShapes:
remaining_headers = [line for line in rendered_lines[header_indexes[0] :] if line.startswith("[") and not line.startswith("[relocated")]
assert remaining_headers == []
- def test_moved_table_leaves_its_banner_comment_behind(self) -> None:
- """A comment block *preceding* a table does not travel with it — a documented limitation.
+ def test_moved_table_takes_its_banner_comment_along(self) -> None:
+ """A comment block *preceding* a table travels with it through the applier's ``move_key``.
tomlkit stores such a block as trailing trivia of the *previous* sibling, not as anything
- attached to the table, so a moved table arrives bare and the banner stays put, now labelling
- whatever followed it. The migration contract states this; nothing in the engine relies on
- comment fidelity across a move.
+ attached to the table — so the raw ``del`` + re-add in ``_move_storage_under_a_fresh_root``
+ leaves the banner behind, labelling whatever followed. The applier reads the block as the
+ table's introduction and re-homes it: that is the guarantee the migration contract states,
+ and this test measures it on the real template rather than on a synthetic document.
"""
toml_doc = _parse(_KIT_TEMPLATE)
banner = "# Storage Config"
@@ -444,10 +447,12 @@ class TestFixApplierConfigSurfaceShapes:
banner_index = source_lines.index(banner)
assert source_lines[banner_index + 2 : banner_index + 4] == ["", "[runtime.storage]"]
- _move_storage_under_a_fresh_root(toml_doc)
+ applications = apply_fix_ops(
+ toml_doc=toml_doc,
+ ops=[MoveKeyOp(table_path=["runtime"], key="storage", new_table_path=["relocated"], new_key="storage")],
+ )
+ assert [application.outcome for application in applications] == [FixOpOutcome.APPLIED]
migrated_lines = _dumps(toml_doc).splitlines()
migrated_banner_index = migrated_lines.index(banner)
- assert migrated_lines[migrated_banner_index + 2] != "[relocated.storage]"
- destination_index = migrated_lines.index("[relocated.storage]")
- assert banner not in migrated_lines[destination_index - 3 : destination_index]
+ assert migrated_lines[migrated_banner_index + 2 : migrated_banner_index + 4] == ["", "[relocated.storage]"]
diff --git a/tests/unit/pipelex/pipeline/fixes/test_fix_applier_native_concept.py b/tests/unit/pipelex/pipeline/fixes/test_fix_applier_native_concept.py
index f883faef9..1fd89a5eb 100644
--- a/tests/unit/pipelex/pipeline/fixes/test_fix_applier_native_concept.py
+++ b/tests/unit/pipelex/pipeline/fixes/test_fix_applier_native_concept.py
@@ -65,13 +65,12 @@ class TestFixApplierNativeConcept:
assert [application.outcome for application in applications] == [FixOpOutcome.SKIPPED]
assert _dumps(toml_doc) == once
- def test_leading_comment_on_deleted_table_reflows_onto_successor(self) -> None:
- """Known artifact (pinned): a standalone comment sitting directly above a deleted
- ``[concept.X]`` table reflows onto the next element (tomlkit trivia behavior;
- ``format_mthds`` does not reposition it). The guarantee is that comments on *untouched*
- content survive — a comment sitting on the deleted concept is the only thing that can
- dangle. Pinned so the behavior is explicit rather than a silent surprise; see
- ``wip/autofix/deferred-checkpoint-b-review-items.md``.
+ def test_leading_comment_on_deleted_table_goes_with_it(self) -> None:
+ """A standalone comment sitting directly above a deleted ``[concept.X]`` table is dropped
+ with the table, rather than dangling onto the next element. tomlkit stores such a comment
+ as trailing trivia of the *previous* table, so a plain delete used to leave it there, now
+ annotating whatever followed; the applier reads it as introducing the deleted table and
+ removes it, and the successor keeps its own blank line.
"""
source = (
"[concept.Report]\n"
@@ -86,10 +85,7 @@ class TestFixApplierNativeConcept:
)
toml_doc = tomlkit.loads(source)
apply_fix_ops(toml_doc=toml_doc, ops=[_delete_concept_op(key="Text")])
- dumped = _dumps(toml_doc)
- assert "[concept.Text]" not in dumped
- # The comment reflowed onto the successor table — it no longer annotates the deleted concept.
- assert "# describes the redeclared concept below\n[pipe.make_report]" in dumped
+ assert _dumps(toml_doc) == '[concept.Report]\ndescription = "A report."\n\n[pipe.make_report]\ntype = "PipeLLM"\n'
def test_delete_missing_concept_is_skipped(self) -> None:
"""Stripping a concept absent from the document is reported as skipped, not raised."""
diff --git a/wip/migrator/migrator-comment-fidelity.md b/wip/migrator/migrator-comment-fidelity.md
new file mode 100644
index 000000000..6d3d7df43
--- /dev/null
+++ b/wip/migrator/migrator-comment-fidelity.md
@@ -0,0 +1,36 @@
+# Comment fidelity across a move — the bug report after v0.46.0, and what closed it
+
+**Status: fixed on `fix/Migrator`, unreleased.** The applier now keeps a comment on the item it introduces through `move_key`, `delete_key` and `delete_table`, and settles what `set_key` and `ensure_table` insert before a table's trailing banner rather than after it. This note records why the phase-0 spike wrote the behaviour off as unfixable, why that was wrong, the rule that was chosen and where it is a reading rather than a fact, and what it does not do for files already migrated by v0.46.0.
+
+## The report
+
+> "The migration moves tables but not the comment banners, so banners ended up above the wrong tables."
+
+Exactly the consequence the phase-0 spike measured on the kit template and the S1 contract then stated as a limitation (`docs/migration-ledger.md` → "Limits you will meet", now rewritten). A `~/.pipelex/pipelex.toml` seeded from the kit template is a banner block per section plus a comment per key; the reshape entry `pipelex-config@3` moved most sections, and each banner stayed where it was while its section left, so the migrated file read `# Log Config` above one section and `# Plugins` above another. Valid TOML, lying comments.
+
+## Why the spike thought it could not be fixed, and why it can
+
+The spike's finding was that tomlkit stores a banner "as trailing trivia of the previous sibling, inside that sibling's serialized string", so carrying it "would mean string surgery on a neighbour's tail". The first half is right and the second is not. Comments and blank lines are discrete keyless body entries — `(None, Comment)` and `(None, Whitespace)` — of whichever container the parser was filling when it met them: a comment above a key sits before that key in the same container, and a banner above a `[table]` header sits at the *tail of the previous table in document order*, inside its deepest last container, because everything up to the next header belongs to the table being parsed. Discrete entries can be located, detached and re-inserted; nothing about it is string surgery. What the library does not have is any notion of which item a run of trivia is *about*, and that is the part that needs a rule.
+
+Three symptoms share the root cause, and a fix has to take all three: a moved table left its banner behind; a moved table carried the *next* table's banner away, because that banner sat at the moved table's own tail; and a table appended under an existing parent landed *after* that parent's trailing banner, stealing it the same way. The same held for keys — a moved scalar landed after the destination's trailing banner, and a deleted key left its own comment dangling onto the next one — and for the `.mthds` fix path, where a comment above a stripped native-concept redeclaration used to reflow onto the next table (a test pinned that as a "known artifact"; it now pins the fix).
+
+## The rule
+
+**A run of own-line comments and blank lines introduces whatever comes next.** A moved item takes its introduction along and leaves its trailing run — the next item's introduction — where it was; a deleted item drops its introduction and keeps its trailing run in place; an item inserted at the end of a table — a moved one, a key `set_key` adds, a table `ensure_table` creates — lands before that table's trailing run, never after it. Everything is put back where the parser would have put it — a previous table's deepest tail rather than an implicit parent's own body — because a comment inside a super table's body is exactly what makes tomlkit start rendering a `[parent]` header the file never had.
+
+Two refinements decide the ambiguous cases, and both were forced by real files rather than invented:
+
+- **Only the last comment block travels.** The kit template opens with a thirty-line file preamble, a blank line, the first section's banner, a blank line, then `[runtime.storage]` — one contiguous run of trivia. Taking the whole run would move the preamble to wherever the storage section lands. So what travels is the last block of consecutive comments, with the blank line above it and whatever follows it; earlier blocks stay. A run with no comment at all is spacing and moves whole. At the very top of the document, a *lone* comment block that a blank line separates from what follows is treated as the file's preamble and stays — only the blank line after it travels — so a hand-written config that starts `# My pipelex config` and then a section keeps its header line at the top.
+- **The first entry of an implicit parent is introduced by what sits above the parent.** `[runtime.storage]` opening a file that never writes `[runtime]` is the first entry of a super table whose header is not rendered; its banner is at the root, above the `runtime` item. The applier climbs through such parents in both directions (finding the introduction, and putting the trailing run back), and only through them — a parent that renders a header owns what sits above it. A dotted assignment is the same shape — `pipelex.storage_config = { … }` at the top of a hand-written override is a `pipelex` super table whose header tomlkit never writes because the owning key is dotted, so the predicate that decides whether a parent renders a header takes the owning key into account, exactly as tomlkit's renderer does. The same climb handles a key that is the first entry of one *chunk* of an out-of-order table (`[email protected]`, the frozen pre-reshape defaults, is written that way, and the reshape's `templating_config` move hit exactly this), and the case where deleting the last key of such a chunk makes tomlkit drop the emptied chunk from the parent altogether.
+
+The rule is a reading and the doc says so: it is right for banners above sections, for the kit's per-key comments, and for preambles; it is wrong for a comment written on its own line *below* a key to annotate it, which will follow the next item. Inline tables are left alone — their whitespace entries are layout.
+
+## What is pinned
+
+`tests/unit/pipelex/pipeline/fixes/test_fix_applier_comment_fidelity.py` compares bytes: the three-symptom cases, the kit-style per-key comments, a table with a sub-table, deletes, the top-of-file shapes, an empty explicit destination, a nested created parent, an array of tables, a wildcard op applying several times over one document, and — the case the report was about — the shipped `pipelex-config` ledger replayed over the frozen pre-reshape defaults, asserting banner by banner that each `# <Name> config` still introduces its section wherever it moved, that the deleted section's banner is gone, and that every banner in the migrated file introduces a header. `test_fix_applier_config_surface_shapes.py`'s banner test now goes through the applier and asserts the guarantee instead of the limitation.
+
+## What it does not do
+
+- **Files already migrated by v0.46.0 are not repaired.** Replay is neutral on a current file, so nothing touches them again; the misplaced banners in such a file are a matter of hand-tidying, or of restoring the `.bak` beside it and running `pipelex migrate` again. The changelog says so.
+- **A comment written below a key to annotate it** follows the next item. Stated in the contract as the cost of the rule; no heuristic distinguishes it from a banner without a blank line above the header, and the kit's own per-key style (comment glued above the key) is the far more common shape.
+- **A comment sitting between two dotted chunks of one key** (`k.x = 1`, a comment, `k.y = 2`) is orphaned when `k` moves — tomlkit merges the chunks on the way and the applier does not chase the gap. Rare, and no worse than before.
diff --git a/wip/migrator-phase0-spike.md b/wip/migrator/migrator-phase0-spike.md
similarity index 96%
rename from wip/migrator-phase0-spike.md
rename to wip/migrator/migrator-phase0-spike.md
index b4f33c56a..dfc3eaf91 100644
--- a/wip/migrator-phase0-spike.md
+++ b/wip/migrator/migrator-phase0-spike.md
@@ -61,6 +61,8 @@ All three are stable under replay.
### Comment attachment on moved nodes — the one thing that does not work
+> **Superseded.** This section's conclusion was wrong and the behaviour was fixed after a field bug report following v0.46.0 — see `migrator-comment-fidelity.md`. The observed placement below is accurate, but its "inside that sibling's serialized string" explanation and its "cannot be fixed" reading are not: the trivia are discrete body entries — `(None, Comment)` and `(None, Whitespace)` — not part of a serialized string.
+
**A comment block preceding a table does not travel when the table moves, and this cannot be fixed by being cleverer with the move.** tomlkit does not attach such a block to the table it appears to introduce; it stores it as trailing trivia of the *previous sibling*, inside that sibling's serialized string. Measured directly: in the kit template, the `# Storage Config` banner is part of `tracing_config`'s rendering, not of `storage_config`'s. Carrying it would mean string surgery on a neighbour's tail plus a heuristic for which comments "belong" to what follows.
The consequence is real and user-visible. `ensure_global_config_exists` seeds every `~/.pipelex/pipelex.toml` as a full copy of the kit template, and that template is heavily commented — a banner block per section plus per-key explanations. Running the reshape entry over it leaves the banners in place while their sections move away, so the migrated file ends up with `# Log Config` above `[inference.model_deck]` and `# Plugins` above `[runtime.storage]`. The TOML is correct; the comments now lie.
diff --git a/wip/migrator-phase3-progress.md b/wip/migrator/migrator-phase3-progress.md
similarity index 100%
rename from wip/migrator-phase3-progress.md
rename to wip/migrator/migrator-phase3-progress.md
diff --git a/wip/migrator-pr-1114-review-followups.md b/wip/migrator/migrator-pr-1114-review-followups.md
similarity index 100%
rename from wip/migrator-pr-1114-review-followups.md
rename to wip/migrator/migrator-pr-1114-review-followups.md
diff --git a/wip/migrator-pr-1115-review-followups.md b/wip/migrator/migrator-pr-1115-review-followups.md
similarity index 100%
rename from wip/migrator-pr-1115-review-followups.md
rename to wip/migrator/migrator-pr-1115-review-followups.md
diff --git a/wip/migrator-pr-1116-review-followups.md b/wip/migrator/migrator-pr-1116-review-followups.md
similarity index 100%
rename from wip/migrator-pr-1116-review-followups.md
rename to wip/migrator/migrator-pr-1116-review-followups.md
diff --git a/wip/migrator-pr-1117-review-followups.md b/wip/migrator/migrator-pr-1117-review-followups.md
similarity index 100%
rename from wip/migrator-pr-1117-review-followups.md
rename to wip/migrator/migrator-pr-1117-review-followups.md
diff --git a/TODOS.md b/wip/migrator/migrator-s7b-inference-backend-surface-plan.md
similarity index 99%
rename from TODOS.md
rename to wip/migrator/migrator-s7b-inference-backend-surface-plan.md
index 346afe130..5c6e37ca5 100644
--- a/TODOS.md
+++ b/wip/migrator/migrator-s7b-inference-backend-surface-plan.md
@@ -1,6 +1,8 @@
# S7b — the inference-backend surface: implementation plan
-> **Charter:** [`wip/migrator-3/sequencing.md § S7b`](../wip/migrator-3/sequencing.md) (workspace root). **Substance:** [`wip/migrator-3/backend-config-surface.md`](../wip/migrator-3/backend-config-surface.md). **Ruling:** R10 (2026-08-17) — build the real fourth surface, not a doctor row and not an `update` bolt-on. Both documents were read in full when this plan was written (2026-08-17); this file orders the work, they hold the reasoning. When they disagree with what the code turns out to do, the code was measured later — record the disagreement in the Status block and in the substance document's *Decisions owed*, do not silently pick one.
+> **Archived 2026-08-17 from the repo-root `TODOS.md`, where it was the live tracker while S7b was built.** The work shipped: `feature/Backend-surface` became PR [#1116](https://github.com/Pipelex/pipelex/pull/1116), merged to `dev`, and rode out in release v0.46.0. Kept as the decision record — the Status block below is the account of what was built, what the CP3 review found, and what was deliberately deferred. Paths in this document are relative to the repo root, not to this file's own directory.
+>
+> **Charter:** [`wip/migrator-3/sequencing.md § S7b`](../../../wip/migrator-3/sequencing.md) (workspace root). **Substance:** [`wip/migrator-3/backend-config-surface.md`](../../../wip/migrator-3/backend-config-surface.md). **Ruling:** R10 (2026-08-17) — build the real fourth surface, not a doctor row and not an `update` bolt-on. Both documents were read in full when this plan was written (2026-08-17); this file orders the work, they hold the reasoning. When they disagree with what the code turns out to do, the code was measured later — record the disagreement in the Status block and in the substance document's *Decisions owed*, do not silently pick one.
>
> **Venue:** this worktree, `_backend/`, on `feature/Backend-surface`, cut from `dev` = `61c4786b6` (the reshape merge). Treat this worktree as the repo root; paths below are relative to it. `pipelex/` (the sibling checkout) stays on `dev` and is not touched. Nothing runs alongside S7b.
>
diff --git a/wip/migrator-write-scope-and-rename-fidelity.md b/wip/migrator/migrator-write-scope-and-rename-fidelity.md
similarity index 100%
rename from wip/migrator-write-scope-and-rename-fidelity.md
rename to wip/migrator/migrator-write-scope-and-rename-fidelity.md