Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Migrator Phase 3, part 2: the migrate commands, boot tolerance, and what a stale configuration is told (#1114)
Louis Choquela38f595a7908
diff --git a/.drift/acks/cli-docs.toml b/.drift/acks/cli-docs.tomlindex 4be8d0c7c..8827432e2 100644--- a/.drift/acks/cli-docs.toml+++ b/.drift/acks/cli-docs.toml@@ -1,26 +1,26 @@contract = "cli-docs"-digest = "sha256:922bd9bc5dd5f6ef955ffee4b5843331613c69a8849a730b9f13a4f69f1177e3"+digest = "sha256:7d39d2644991dae40be1f456c4b7ae9ece82178b60399f2836858421e835e47a"reviewed_by = "Louis Choquel"-reviewed_at = "2026-08-15T08:29:38Z"-rationale = "Comment-only change in _run_core.py: reworded the dry-run boot comment so it no longer promises model-handle resolution on a keyless machine (needs_model_specs only keeps real gateway specs; lenient load still skips keyless backends). Reviewed docs/tools/cli/run.md and the agent-CLI CLAUDE.md: neither makes that promise, so no doc change needed."+reviewed_at = "2026-08-16T21:01:37Z"+rationale = "Second bot-review round on PR 1114: the doctor telemetry row reads the verdict off the plan of the file it validated instead of the whole directory; pipelex migrate's cancel path now exits through _exit_on_attention, and the closing panel tests an unconfirmed write before the written count. No flag, field, format, finding value or exit-code rule moved. docs/tools/cli/migrate.md's claim that the command exits non-zero when it leaves something to look at is now true on every path including cancel; the finding semantics are documented in docs/migration-ledger.md, updated in the same change. agent-cli.md only had its stderr single-object sentence scoped to the structured commands."[trigger_files]"pipelex/cli/__init__.py" = "blob:8b137891791fe96927ad78e64b0aad7bded08bdc"-"pipelex/cli/_cli.py" = "blob:6bb9f12c984501a7cc365168c9053e15bb08638d"+"pipelex/cli/_cli.py" = "blob:2fdd24792b8281e1aca3facfef06660ab8b34731""pipelex/cli/agent_cli/__init__.py" = "blob:e69de29bb2d1d6434b8b29ae775ad8c2e48c5391"-"pipelex/cli/agent_cli/_agent_cli.py" = "blob:bbbb33d750a93df1a5afd72b43e0a381d5081279"+"pipelex/cli/agent_cli/_agent_cli.py" = "blob:60b5e959906fc806153ff11332bb9cb56752c5c6""pipelex/cli/agent_cli/commands/__init__.py" = "blob:e69de29bb2d1d6434b8b29ae775ad8c2e48c5391""pipelex/cli/agent_cli/commands/accept_gateway_terms_cmd.py" = "blob:a1fc438119dd10607252c08390ebd6384d999678""pipelex/cli/agent_cli/commands/agent_cli_factory.py" = "blob:caba5ab957e4a2bbb62e329ba6722a441dff781e"-"pipelex/cli/agent_cli/commands/agent_output.py" = "blob:4bc25ac1b0f232dbbb76d4dbe179030b8c74a8e6"+"pipelex/cli/agent_cli/commands/agent_output.py" = "blob:b73cdb4738c7a64a9ae54665bb20d040a81beb41""pipelex/cli/agent_cli/commands/bundle_path_resolver.py" = "blob:30339c5d91048f4e67312d5691ed9a65d57472e0"-"pipelex/cli/agent_cli/commands/check_model_cmd.py" = "blob:1d5a66d03e6fec2d8673959857ace87b78db03ac"+"pipelex/cli/agent_cli/commands/check_model_cmd.py" = "blob:f4bdc5f3732b067e397dde384b511f436f95e526""pipelex/cli/agent_cli/commands/codegen/__init__.py" = "blob:e69de29bb2d1d6434b8b29ae775ad8c2e48c5391""pipelex/cli/agent_cli/commands/codegen/app.py" = "blob:56130b89fe9efec5ad3d014ed6bc16bef1bd1645""pipelex/cli/agent_cli/commands/codegen/check_cmd.py" = "blob:bacfda014d158c9a4e9de5f8ca03ba9628b6b102""pipelex/cli/agent_cli/commands/codegen/types_cmd.py" = "blob:256d428cff7738af33ce69281f8e81b538ba7996""pipelex/cli/agent_cli/commands/concept_cmd.py" = "blob:518d291b5e74a7f5ffb2da1606aae42ca2c57835"-"pipelex/cli/agent_cli/commands/doctor_cmd.py" = "blob:d1ee397d4c62e8768a0ab8a7011468d60581fc12"+"pipelex/cli/agent_cli/commands/doctor_cmd.py" = "blob:8d75a1630eeb94ebb5450eb1fa0273e7d85b18c2""pipelex/cli/agent_cli/commands/fix/__init__.py" = "blob:65662c86f17f610ad55ecfc3e27a022b9aa26e8b""pipelex/cli/agent_cli/commands/fix/app.py" = "blob:e6190d9bde6a05755c04bd21806f699461038ac0""pipelex/cli/agent_cli/commands/fix/bundle_cmd.py" = "blob:aa2328eb97365a37ce3c2fd885847402508c7476"@@ -33,7 +33,8 @@ rationale = "Comment-only change in _run_core.py: reworded the dry-run boot comm"pipelex/cli/agent_cli/commands/inputs/method_cmd.py" = "blob:bc5658f7549e766c5827884d3be939eda6e519fd""pipelex/cli/agent_cli/commands/inputs/pipe_cmd.py" = "blob:eda6ea9a683f01bcc59336b44354c3556b3fcfec""pipelex/cli/agent_cli/commands/lint_cmd.py" = "blob:91ad9cb4d618ba4c0b7da366cde2f74f38d350df"-"pipelex/cli/agent_cli/commands/models_cmd.py" = "blob:90d30128fe924677ee84f926fc733b2016333297"+"pipelex/cli/agent_cli/commands/migrate_cmd.py" = "blob:ce733b78df03e65c3c8a10d46b27b57c05ec054c"+"pipelex/cli/agent_cli/commands/models_cmd.py" = "blob:10ad2916530308ad650de386cf042dc2528b80e8""pipelex/cli/agent_cli/commands/pipe_cmd.py" = "blob:0be9fb13dbec21da0958ed66b45748291b4e5e2d""pipelex/cli/agent_cli/commands/plxt_passthrough.py" = "blob:22a4666dbd179a68b86cb340cdb9dbe207c8c1e1""pipelex/cli/agent_cli/commands/run/__init__.py" = "blob:e69de29bb2d1d6434b8b29ae775ad8c2e48c5391"@@ -81,7 +82,7 @@ rationale = "Comment-only change in _run_core.py: reworded the dry-run boot comm"pipelex/cli/commands/codegen/inputs_cmd.py" = "blob:20738a9e42f0e5ef7ffc489093348af3cb4643a0""pipelex/cli/commands/codegen/types_cmd.py" = "blob:b7c9c7a3664c9fbe66f9b4c3c5ebf9ea1322445c""pipelex/cli/commands/crate_loading.py" = "blob:3d7d651931ecf1a15a3e2d72411b3212b55d74ef"-"pipelex/cli/commands/doctor_cmd.py" = "blob:3cc70c2844c48aa73b91cdf085864589f30bc7f8"+"pipelex/cli/commands/doctor_cmd.py" = "blob:6484b5da59af0f978fe79d675089e05da9d7ac81""pipelex/cli/commands/fix/__init__.py" = "blob:e69de29bb2d1d6434b8b29ae775ad8c2e48c5391""pipelex/cli/commands/fix/_diff_sandbox.py" = "blob:2113f7e7ffd373177a3104ce5b5dfb7f657dd186""pipelex/cli/commands/fix/_fix_core.py" = "blob:8f6a187a0fb42c498629b177753d81620800b0da"@@ -104,7 +105,8 @@ rationale = "Comment-only change in _run_core.py: reworded the dry-run boot comm"pipelex/cli/commands/init/ui/types.py" = "blob:1361f29ca95bacc90bca90ea6c0e39f44b6734ab""pipelex/cli/commands/login/__init__.py" = "blob:e69de29bb2d1d6434b8b29ae775ad8c2e48c5391""pipelex/cli/commands/login/command.py" = "blob:ace1db9879626cc1722adaac12761aadaa6b24ed"-"pipelex/cli/commands/plugins_cmd.py" = "blob:2980268813f1acf67f0f2cc914ccaff6cae6889b"+"pipelex/cli/commands/migrate_cmd.py" = "blob:be405b6310ad28be02f7a17c9abe2a37629845a3"+"pipelex/cli/commands/plugins_cmd.py" = "blob:190a58e16c9e28069fc162555ec0ab24ffd92e9d""pipelex/cli/commands/resolve_cmd.py" = "blob:2928a0e26eab7983367a59a78533f6618817feba""pipelex/cli/commands/run/__init__.py" = "blob:e69de29bb2d1d6434b8b29ae775ad8c2e48c5391""pipelex/cli/commands/run/_inputs_file_loader.py" = "blob:dde781f846968b667a0acfb4b3e3cea01cef7049"@@ -125,7 +127,7 @@ rationale = "Comment-only change in _run_core.py: reworded the dry-run boot comm"pipelex/cli/commands/validate/pipe_cmd.py" = "blob:d67267571559006459628180f2c8b216ba7fee31""pipelex/cli/commands/which_cmd.py" = "blob:68661d4bab22e2c4dab183d59bd9f2c6dd573472""pipelex/cli/deck_notice.py" = "blob:542877125e89e35b02bc969cfee952155b0d7c2c"-"pipelex/cli/error_handlers.py" = "blob:ec4005ac908fac151275db3b28cc1685edd585b0"+"pipelex/cli/error_handlers.py" = "blob:c491c7f72e1d00017dbc9b08f092b788b81b521e""pipelex/cli/exceptions.py" = "blob:1d51c3a53ddd3f6cf8de566fa2ade3fa75ab7a3a""pipelex/cli/installed_methods.py" = "blob:2a1adeda25082bd2bef44887b9af60c29dab8d7b""pipelex/cli/method_resolver.py" = "blob:850ad43ee2c6fb0ffc3f449f62ceea1ab6f17297"diff --git a/.drift/acks/config-docs.toml b/.drift/acks/config-docs.tomlindex de3739eb4..919543457 100644--- a/.drift/acks/config-docs.toml+++ b/.drift/acks/config-docs.toml@@ -1,8 +1,8 @@contract = "config-docs"-digest = "sha256:71968ad3e732b8e92611a165463e2a9c293772fd07ed2d994a26c6f02548ef1f"+digest = "sha256:31b470d034b3bcd4072bb4b286a4e1cf37b4414d5b884c42cd78f421fe72a679"reviewed_by = "Louis Choquel"-reviewed_at = "2026-08-15T12:10:14Z"-rationale = "Reviewed docs/configuration/ against the config-surface read helper, the loader and telemetry wiring, and the PipelexServiceConfig.agreement default. Nothing there went stale: no page documents a reader's tolerance rules, the service-agreement file's location is unchanged, and the telemetry page documents settings rather than the load path. The new reserved meta schema_version key is deliberately NOT added to these pages — nothing writes it, and documenting it where users are told what to put in their files would invite the team-skew breakage the migration contract forbids; it is specified in docs/migration-ledger.md, which is published at Phase 3."+reviewed_at = "2026-08-16T20:18:24Z"+rationale = "Review of the PR 1114 fixes to config_loader.py and config_surface.py: the main-configuration stale warning is now parked on the loader and emitted after logging is configured, and the in-memory retry declines a file declaring a schema version below the ledger floor. Both are boot-tolerance mechanics documented in docs/migration-ledger.md, which was updated in the same change; docs/configuration/ describes files and keys, and telemetry-config.md's out-of-date section still says exactly what a user sees - a boot that warns and names pipelex migrate. No configuration key, default or file moved, so no page under docs/configuration/ is stale."[trigger_files]"pipelex/cogt/config_cogt.py" = "blob:ee1ee2d0c1474dcef5e07690268de37ae3ff0987"@@ -10,12 +10,12 @@ rationale = "Reviewed docs/configuration/ against the config-surface read helper"pipelex/pipelex.toml" = "blob:abaf4f65f3096a4625dad8c6887431e952cc4734""pipelex/system/configuration/__init__.py" = "blob:8b137891791fe96927ad78e64b0aad7bded08bdc""pipelex/system/configuration/config_check.py" = "blob:efa78599c08e5e003d975cc54978ea5d267214a0"-"pipelex/system/configuration/config_loader.py" = "blob:76f9a66ae1ac28df45b9742677218bee63a37121"+"pipelex/system/configuration/config_loader.py" = "blob:f5fe59571d4d6677d9a715bc97269afbc6d636a2""pipelex/system/configuration/config_model.py" = "blob:83b29dc6e93393c9b98c4353636caf0135c80339""pipelex/system/configuration/config_root.py" = "blob:c7d43851270912d17413cfc6a511247d3c7cc160"-"pipelex/system/configuration/config_surface.py" = "blob:18dc1521de776dbffaa27630896c0700c9edf100"-"pipelex/system/configuration/configs.py" = "blob:90b3186c53b431e0c50b73a533685be92ccd94b0"+"pipelex/system/configuration/config_surface.py" = "blob:4144700093b2469d07b15bc0a2845fc2aa2b2900"+"pipelex/system/configuration/configs.py" = "blob:d53cdf2dd2f78efd53b51f1f6c89605f9dd4e5d8""pipelex/system/configuration/pipe_func_config.py" = "blob:df1cbd3698d51c96d8fa8a8c6fdf1c6be57e5771"-"pipelex/system/telemetry/telemetry_config.py" = "blob:db0457ba4214d5b0ecb18db75e1b164054181fed"+"pipelex/system/telemetry/telemetry_config.py" = "blob:7070951a6d3e4c10cb7d59822ce168c6590c53be""pipelex/tools/aws/aws_config.py" = "blob:c1aeafeae08d3921464db33795f11a29ff3892ba""pipelex/tools/log/log_config.py" = "blob:7b6d032ffea29d6ce60e8af7106f2d935bc18348"diff --git a/.drift/acks/hub-layering-convention.toml b/.drift/acks/hub-layering-convention.tomlindex bdd702774..3b22ed043 100644--- a/.drift/acks/hub-layering-convention.toml+++ b/.drift/acks/hub-layering-convention.toml@@ -1,10 +1,10 @@contract = "hub-layering-convention"-digest = "sha256:cd4c268d3557cedc0f5594d73f3db9f9c6eea5781baa184fb29c82cb93c9b5b9"+digest = "sha256:820a8c39f4defbb59e919745a9d040794d65ebe3566027ca1b8fadc9c8780953"reviewed_by = "Louis Choquel"-reviewed_at = "2026-08-13T09:17:09Z"-rationale = "Trigger is interpreter_hub.py arriving at this merge changed on both sides: the branch added the two public entry accessors get_required_entry_pipe and get_optional_entry_pipe, and dev renamed the layer vocabulary from runtime to kernel in two of its comments. Verified the page mechanically rather than by eye, because the staleness this contract exists to catch is an enumeration going quietly out of date: parsed both hub modules with ast for public module-level functions and checked every one against the doc -- 33 on interpreter_hub, 33 on runtime_hub, zero untabulated. The library-lookups row already lists both new accessors and the page already carries the paragraph on choosing between the strict in-body pair and the entry pair, plus the cross-package alias caveat; all of that merged in cleanly from this branch alongside dev's rename, so the tables and the prose agree with the merged modules. Zero occurrences of the stale runtime layer or runtime-layer vocabulary remain on the page. No layer boundary moved: both new symbols are interpreter-layer, live beside their siblings, and add no kernel-layer import, and check-hub-layering passes on the merged tree. Reviewed end to end; no doc change needed."+reviewed_at = "2026-08-16T16:17:00Z"+rationale = "Boot tolerance put a kernel-layer module (system/configuration/config_surface.py) one deferred import away from pipelex.migration, whose engine imports pipeline.fixes.applier on purpose. Reviewing the page against that turned up a real staleness the change did not cause: pipelex.migration is a top-level package in NEITHER column of the accounting sentence, so it read as checked while being unpoliced -- the exact failure the same paragraph warns about. Added it, and explained rather than asserted its side, because it is the interesting case: it never reaches interpreter_hub, so declaring it kernel-layer would PASS the guard and be this page's own first known inversion all over again. Measured the rest instead of eyeballing it: a module-level engine import in config_surface keeps check-hub-layering green and turns nine closure-test entry points red, which is this page's lint-vs-property claim reproduced, so that claim needed no edit. The two hub partition tables are untouched by this milestone -- runtime_hub gained no symbol, only a rerouted call inside setup_config."[trigger_files]"pipelex/cli/dev_cli/commands/hub_layering_guard.py" = "blob:2dbadcbab7d44364a7b5f1d16268b23b26700a00""pipelex/interpreter_hub.py" = "blob:b8c35e37814f0e94c3bf1c364b06a98d0c22ee93"-"pipelex/runtime_hub.py" = "blob:7f538d2acb10f263d7e1b6907b42d37199cbc654"+"pipelex/runtime_hub.py" = "blob:75e5f79bad4104617a32f2fe64d1493b0cfd4cb5"diff --git a/CHANGELOG.md b/CHANGELOG.mdindex a4b24ae91..97909e70d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -4,6 +4,13 @@### Added+- **A configuration that will not load now says whether it is wrong or merely old**: A validation error is raised against the merged configuration and carries no provenance — it names a key, not which file put it there, and certainly not whether that key was correct last month. When a configuration surface's model refuses, the error now also carries a dry-run scan of that surface over the same directories `pipelex migrate` walks: a structured `migration` block naming the remedy, whether anything there is a person's to resolve rather than the tool's, and the plan per file — the same shape `pipelex-agent migrate --dry-run --format json` emits, so an agent that parses one has already parsed the other. `error_domain` deliberately stays `config`: it is a closed cross-repo enum and the agent-hook specification routes anything else to BLOCK, so a stale configuration reported under a new domain would stop an agent instead of telling it what to run. Consumers branch on the block's presence — absent means the failure is not staleness — and the message gains a paragraph naming the files and the command. No value read from a user's file appears in the block, which is the third of the three channels that rule covers. A caller validating something that is not a configuration surface — a `.mthds` bundle, a backend file, a model deck — names no surface and pays for no walk.++- **A configuration file the ledger can explain no longer stops the boot**: A schema change used to leave every machine in the field with a boot that died on `extra="forbid"`, naming a key the user never chose to have. When a configuration surface fails to validate, its ledger is now replayed over the same files **in memory** and the result validated again; a boot that succeeds says so in a warning naming the files and the `pipelex migrate` remedy, and nothing is written — a tolerated boot leaves the file and its directory exactly as it found them, which is why the warning keeps coming back until the command is run. Tolerance widens what starts and never what is accepted: the re-validation is what decides, so material an `unsafe` entry is about still fails the boot because the model still refuses it. Anything that goes wrong inside the retry makes it decline rather than replace the error the user already has — theirs names the key to fix. A healthy configuration pays nothing: it reads no ledger, re-parses no document, and does not even import the migration engine, which is deferred for a second reason too, since the applier lives in the interpreter layer and the configuration loaders sit in the kernel layer that must not load it. The telemetry configuration is the surface this reaches today, because it carries the only entry the package ships.+- **`pipelex migrate` and `pipelex-agent migrate`**: The migration engine now has commands, so a configuration that predates the installed pipelex can be repaired instead of re-initialized. Both walk the global `~/.pipelex/` and the project `.pipelex/` — those two directories only, and not recursively, because a subdirectory such as `inference/backends/` holds a different kind of file and one of them (`pipelex_gateway.toml`) matches the main configuration's tier glob exactly. Neither command boots: a broken configuration is the reason to reach for them, so they use the ledger, the applier and the filesystem and nothing else, and a test hands them a configuration that cannot load to keep it that way. The human command plans, shows the plan and asks; the machine one writes only with `--yes`, since it cannot ask, and passing `--dry-run` and `--yes` together is refused rather than resolved. Its answer is structured, and `needs_attention` is the verdict — *this run left something a person has to decide*, deliberately not *did anything get written* — with the exit code and the Markdown rendering as presentation. Every file rewritten is backed up beside itself first, and no value read from a user's file appears in either command's output or in the structured plan.+- **A migration now names what it cannot explain about a file**: A file *newer* than the running pipelex — an override migrated on one branch, then an older branch checked out — breaks the boot while a replay finds nothing to do, because every operation's source is absent. That silence beside a dead boot is the failure the migration project exists to remove, so every plan now carries an `unexplained[]` listing the paths of the migrated document that the current schema does not know and no ledger entry accounts for, each with the two readings the tool cannot tell apart: a typo, or a file written by a newer pipelex. This is the one part of a migration that needs the configuration model — the question goes to the surface's fingerprint, the same projection the coverage gate diffs, so the replay engine stays model-free. The document it reads is the one the run leaves behind rather than the one it found, which is what keeps every key the ledger is about to repair out of the list, and a path a *blocked* entry is about is left to that entry, which reports it by name with its own guidance. A key the user chose is never rendered: beneath an open mapping a typo is reported at `queues.*.retries`, naming only the segment the schema does not know.+- **A configuration file may now be refused for declaring a schema version its ledger no longer reaches**: `min_supported_schema_version` had no reader, which left the floor a comment. A migration now reads the reserved `[meta] schema_version` — exactly as tolerantly as the configuration loader strips it, so a malformed declaration is no declaration — and refuses a file below the floor by name (`unsupported_schema_version`), writing nothing. Without it, a ledger whose oldest entries had been squashed away would run over a file older than them, change nothing, and report it clean, because the applier skips an operation whose target is absent.+- **An `unsafe` migration entry now says what it is about, and keeps saying it**: `unsafe` is the ledger's form for a change only a human can make, and the coverage gate's only remedy for a tightened numeric bound — but the engine decided whether to report one by rehearsing its operations, so an entry with none was accepted by the accounting and then guaranteed to reach nobody, and any entry went silent the moment a later `safe` entry renamed the table its material sat in (reported on the first run, clean on the second, boot still failing). An entry now declares the paths whose value domain narrowed, in `declared_narrowed_paths`; an entry with no operations must carry one, a `safe` entry may not, `make check-ledger` refuses a path the entry's own version does not record, and the coverage gate no longer takes the bare word `unsafe` as accounting for a narrowing or a lost enumerated spelling — it demands the path be named. The material of an `unsafe` entry is traced through the ledger before a document is questioned: back through the entry's own operations, since it never applied them, and forward through every later `safe` entry, so the same file is reported on every run for as long as it carries it. A file that merely *sets* a narrowed path is reported under a new, weaker `value_domain_narrowed` reason listing the keys to check by hand — the engine is model-free by design and cannot tell a value the new domain refuses from one it accepts — and those keys are named at the spelling the migrated file carries rather than the one that matched, since the entry is questioned where the replay reaches it and the later `safe` entries rename the material before the file is written — and that report, alone among blocked entries, does not fail convergence, since a healthy reference document sets those paths like everyone else's file does. Forward tracing is deliberately confined to `unsafe` entries: a rehearsal may guess, an application may not.- **A migration operation can now remap every value beneath an open mapping**: `remap_value` accepts `key = "*"`, meaning "each key of the addressed table" — the same "each of these" reading the wildcard already had in a `table_path`, applied to keys. A mapping from the user's own keys to an enumerated value (`dict[str, LogLevel]`) keeps its spellings under keys only the document can enumerate, so no fixed key reaches them and this is the only shape in which a member renamed beneath one can be repaired at all. On every other kind a `*` key is now refused when the operation is parsed: deleting every key of a table is `delete_table`, and renaming or moving every key onto one fixed name is the many-to-one that already rules a wildcard out of a `move_key`. Refused rather than tolerated, because an unrefused one is a *dead* operation that skips forever while looking like an accounting.@@ -17,9 +24,11 @@- **The migration engine**: A surface's ledger can now be replayed over a real file. The engine takes a document's text and returns the text it should now have, plus a per-file plan naming the entries that applied, the entries it would not apply, and why. Two reporting rules keep a plan from being more optimistic than the file: an entry is reported once — a conflicting operation routes the whole entry into the blocked list, carrying whichever of its operations did land — and an `unsafe` entry is rehearsed against the document first, so it stays silent on a file that does not need it rather than warning at every boot forever. Writing is transactional per file and always takes a backup: exactly one per file, named `<file>.bak.<UTC stamp>`, inheriting the source file's mode rather than the umask, written before the replacement and pruned only after it commits. A file that is unparseable, unwritable or changed during the run is reported as blocked while every sibling is migrated normally — unlike the `.mthds` fix loop, whose files only make sense together and which still commits a round all-or-nothing; both now share one set of transaction primitives. Nothing calls this from the command line yet.- **The transform goldens**: `make check-migration-schemas` now also *performs* each migration it is asked to believe in. For every ledger entry it applies the entry's operations to the frozen reference document of the version before it and compares the result with the frozen document of the version after: every path the migration creates must be one the new document has, every path the two documents share must survive, and the last link's output — read the way a user's file is really read, beneath the current defaults layer — must be accepted by the current model. That closes the one defect the other two checks pass in silence: a rename whose destination is misspelled is accounted for by coverage and skipped by convergence, and then migrates every file to a key the schema rejects while the tool reports success. The comparison is deliberately one-directional in each claim, so a bump that also adds keys, edits comments, flips defaults or ships one more entry in a packaged deck stays green.- **Replay neutrality is now a property rather than an assertion**: The guarantee a user's machine depends on — replaying a surface's whole ledger over a file already valid at the current schema changes not one byte — is checked over generated documents, not only over the two reference documents. A reference document carries exactly one value per key, so it cannot see an operation that misbehaves on the *other* legal spelling of an enumerated field; a sampler that reaches those spellings can. The sampler proposes each mutation from the surface's fingerprint and lets the models decide whether it lands, because schema membership is settled by validators no type projection can see: a legal enum member can still be rejected for what it requires elsewhere in the document, and a mapping typed with arbitrary string keys can still refuse a key outside a fixed set. Idempotence — replaying twice equals replaying once — and prefix coherence — a file caught halfway lands where a replay from zero lands — are checked alongside it, and each generator carries its own vacuity check, since every one of these properties is trivially true over documents nothing acts on. `hypothesis` joins the development dependencies.+- **`pipelex doctor` reports pending configuration migrations, and `--fix` runs them**: A configuration the migration history explains now boots with a warning rather than an error — and `pipelex-agent` silences logging process-wide as its first act, so a machine consumer never learns from a boot that its configuration is stale. Asking was the only channel left and there was nothing to ask. The health report gains a **Configuration Migrations** row that is `pipelex migrate`'s own dry run: a structured `finding` (`up_to_date`, `pending`, `needs_attention`, `unavailable`), the files a migration would rewrite, and separately the files it will not repair on its own — both sets, because a drifted machine usually has some of each and a reader who heard only the first would stop with a broken file still in place. `pipelex doctor --fix` then offers to run the migration, reaching the command's own write pass rather than a second implementation of it. The row is deliberately not scoped by `--global`: it answers for a command that walks both configuration directories, and a row naming fewer files than the command touches would be a surprise write. A failure inside the scan is reported as a failure to check rather than as health, and never as an exception — an exception there would replace every row of the report with a single line.### Changed+- **A stale telemetry configuration is migrated instead of thrown away**: `telemetry.toml` failing to load used to get one answer everywhere — *the format has changed, run `pipelex init telemetry`* — written for the flat pre-`[custom_posthog]` file and then shown for every other way a file can fail. That command writes a fresh file, so on the very case it named it discarded the PostHog key, the Langfuse credentials and the OTLP exporters that the shipped `telemetry-config@2` entry carries forward. The telemetry loader now raises through the same reporting path as every other configuration surface, so its error carries the structured `migration` block and the human CLI, the agent envelope and `pipelex doctor` all say whether the file is *old* or *wrong* rather than each holding a hardcoded remedy. `TelemetryConfigError` is a `PipelexConfigError`, which is where its `error_domain = config` now comes from — its entry in the agent CLI's domain table is gone as redundant. The `doctor` telemetry row carries a structured `finding` (`healthy`, `not_found`, `unparseable`, `out_of_date`, `invalid`) that every caller branches on; writing a fresh file is offered for a missing file only, and regeneration is described as what it is, a way to start over that discards the file. The telemetry models stayed in `telemetry_config.py` and the loader moved into a new `telemetry_loader.py` beside it, because the loader's failure path reaches the migration registry and the registry imports every configuration model.- **The fingerprint records the bounds a file actually meets, and the enumerated spellings where an operation can reach them**: A bound in a field's own metadata *binds* — pydantic applies it on top of whatever a union member declares — so the two sources are now merged separately and intersected instead of pooled and widened; a field whose own `Field(le=6)` was the real ceiling used to be recorded as `le=100` because a member said so, and a later tightening of that ceiling then read as a change to an already-looser one. Enumerated members beneath an open mapping are now recorded only on the wildcard record, whose own value is the enumerated one: recording them on the container as well made the coverage gate demand a remap of a table value, a demand no operation could satisfy. Every other container is still descended into, since a `list[enum]` has no child record — and there the gate now names `unsafe` as the only remedy rather than crediting a remap that would skip on every run. One golden moved with it: the duplicated member set on `pipelex.log_config.package_log_levels`.- **The narrowing relation stopped crying wolf on four shapes**: A container whose argument widened (`list[int]` → `list[int | str]`) is read structurally instead of as one opaque string; `int` is absorbed by `float`, which accepts every integer in strict validation as well as lax; `enum` and `literal` absorb each other, because they are two renderings of one thing and what moved between two member *sets* is reported by the enumerated-spelling half; and over an integral type `gt=n` and `ge=n+1` are folded to the same bound, so swapping one spelling for the other stops reading as a tightening. Each was a false breaking verdict on a schema no user's file can tell apart, and the remedy the gate named — bump the version, write an `unsafe` entry — was work with no beneficiary.- **`make up-migration-schemas` left `make up`, and refuses to erase a removal**: Every other regenerator in that aggregate rewrites a derived artifact from a live source, so running them out of habit is harmless; the head fingerprint golden is the proof obligation the coverage gate compares against, and rewriting it after deleting a field erases the very removal the gate exists to catch. It is now run on purpose, and it stops before overwriting a head golden recording a path, an enumerated spelling or a value domain the models no longer have. The refusal names both readings — a real removal wants a bump and an entry; a golden that merely predates a change to the fingerprint format wants the new `make umigf` — because the code cannot tell them apart.@@ -36,6 +45,10 @@### Fixed+- **`pipelex doctor --fix` no longer offers to overwrite a telemetry configuration it could migrate**: The fix machinery decided what it could repair by searching the row's *message* for `"format has changed"` or `"invalid configuration"`, and on a match offered to write a fresh `telemetry.toml` — discarding every setting in it. It now branches on a structured finding, so an out-of-date file is answered with `pipelex migrate` and an invalid one is left to a person. Rewording the row can no longer switch the repair path off in silence, which the string search would have done without a test failing.+- **The doctor's telemetry check strips the reserved `[meta]` table the way boot does**: A `telemetry.toml` carrying the migration machinery's reserved `[meta]` table boots perfectly well and was reported invalid by the health check, which validated the raw document.+- **A boot that fails on the main configuration reports it instead of crashing**: `RuntimeBoot` and the doctor both caught `pydantic.ValidationError` around `setup_config`, and the main configuration does not raise one — `ConfigRoot` translates pydantic's error into `ConfigValidationError` before it leaves — so the arm never fired for the one configuration everything depends on, and a machine with a bad `pipelex.toml` got a bare traceback rather than the field-level translation the arm existed to produce. Both sites now catch either shape a refusal takes and share one helper, so the same refusal produces the same message and the same structured block wherever it is caught.+- **The agent CLI no longer prints two error envelopes on one stream**: `agent_error` leaves through `typer.Exit`, which is a `RuntimeError` and not a `SystemExit`, so a command boundary re-raising only `SystemExit` caught the error it had just reported and reported it again. Two JSON documents on stderr is not JSON, and a machine consumer parsing it got "Extra data" instead of the error. `pipelex-agent models` and `pipelex-agent check-model` were the two sites.- **Renaming a table written in dotted form no longer swallows the keys after it**: `k.x = 1` and a `[a.k]` header say the same thing to a TOML parser and are two different things to a TOML document, and which one a key is is a flag the parser sets rather than a property of its name. Rebuilding the key from its name alone — what tomlkit's own re-key primitive does — lost that flag, so a renamed dotted key came back out as a block header, and a block header absorbs every scalar that follows it in the same table: a neighbouring `m = 3` silently became a key of the renamed table, with an `applied` verdict and no diagnostic. Renamed at an inner segment the whole chain re-rendered at the document root, taking the subtree out of the table it lived in. The applier now renames the document's body entries where they sit, carrying the flag forward and covering every chunk of a key written on several dotted lines — which also stops a cosmetic blank line being injected after any table it renames. This reaches every caller of the applier, the `.mthds` fix loop included.- **A migration no longer replaces a symlinked configuration file with a regular one**: a file symlinked out of a dotfiles directory is now read, backed up and replaced at the end of the link, the way the `.mthds` fix loop already resolves its own targets — the link survives and the real file is the one that gets migrated. Replacing the link path instead put a regular file where the link was and left the real file unmigrated.- **A migration run no longer takes away a backup it did not make**: the backup stamp resolves to the second, so two runs of the same file can address the same name. A run now reserves the name atomically and leaves whatever is already there — an older copy of the file, which is to say the original — instead of overwriting it, and a run whose write is refused deletes only the copy it made itself. A copy kept because a write could not be confirmed is renamed out of the `.bak.` rotation into `.rescue.`, so the next successful run cannot prune away the very file the report told the user to go and get — and when that copy cannot be moved, because another run made it or the rescue name is taken, the report says it is still in the rotation and asks the user to take it now instead of promising it will keep; and the backup's directory entry is `fsync`-ed before the target is replaced, so "back up first, replace second" holds across a power loss and not only a process exit.diff --git a/docs/configuration/config-practical/telemetry-config.md b/docs/configuration/config-practical/telemetry-config.mdindex c02c48517..2827ab452 100644--- a/docs/configuration/config-practical/telemetry-config.md+++ b/docs/configuration/config-practical/telemetry-config.md@@ -251,9 +251,23 @@ When set, this disables:---+## When telemetry.toml will not load++Pipelex tells you which of the two situations you are in, because the fix is different.++**Your file is out of date.** The settings are fine, the shape they are written in has moved on. Pipelex boots anyway — it carries the file forward in memory and warns you — and `pipelex doctor` reports the telemetry row as out of date. Make it permanent with:++```bash+pipelex migrate+```++That rewrites the file into the current shape and **keeps every setting that still has a home in it** — your PostHog key, your Langfuse credentials, your OTLP exporters. The few flat-format settings the current shape retired (`respect_dnt`, for one: `DO_NOT_TRACK` is now honoured unconditionally) are dropped, and the ledger entry names each of them and why. It backs the file up first. See [the migration ledger](../../migration-ledger.md) for what it does and does not touch.++**Your file is wrong.** A setting that is not a setting, a value outside what it accepts, or TOML that does not parse. The error names the fields; correct them in `telemetry.toml`. No command fixes this one, because nothing but you knows what you meant.+## Resetting Configuration-To reset your telemetry configuration to defaults:+To start over — **discarding every setting in the file**, which is why it is not the answer to either situation above:```bashrm .pipelex/telemetry.tomldiff --git a/docs/configuration/index.md b/docs/configuration/index.mdindex 6b66126fe..2a20456dd 100644--- a/docs/configuration/index.md+++ b/docs/configuration/index.md@@ -54,7 +54,7 @@ The configuration is organized into four main sections:1. `[pipelex]` - Core Pipelex settings2. `[cogt]` - Cognitive tools and LLM settings3. `[plugins]` - Plugin-specific configurations-4. `[migration]` - Migration helpers for renamed settings+4. `[migration]` - Retired, and empty of meaning: the rename tables it holds are no longer read by anything. Keeping an out-of-date configuration working is now the job of [`pipelex migrate`](../tools/cli/migrate.md), which repairs your files instead of guessing at an error message. The section stays in place for now because removing a table the configuration model still requires is itself a change your files would have to be migrated through.Each section contains multiple subsections for specific features and functionalities.diff --git a/docs/contribute/hub-layering.md b/docs/contribute/hub-layering.mdindex deebc2ec5..c399b42b3 100644--- a/docs/contribute/hub-layering.md+++ b/docs/contribute/hub-layering.md@@ -244,7 +244,7 @@ An AST guard (`pipelex-dev check-hub-layering`, core in `pipelex/cli/dev_cli/com1. **The layer rule.** A module in the declared kernel layer — `pipelex.cogt`, `pipelex.core`, `pipelex.errors`, `pipelex.graph`, `pipelex.kernel`, `pipelex.observer`, `pipelex.plugins`, `pipelex.providers`, `pipelex.reporting`, `pipelex.system`, `pipelex.test_extras`, `pipelex.tools`, `pipelex.tracing`, plus the `pipelex.runtime_hub` and `pipelex.runtime_boot` modules themselves — may not import `pipelex.interpreter_hub`. The declaration is compliant with an empty exception list, so the guard hard-blocks on **any** violation. Entries are matched exact-or-dotted-prefix, which is why a bare *module* is a legal entry beside the packages: without `pipelex.runtime_hub` spelled out, the module at the centre of the rule would be exempt from it, and without `pipelex.runtime_boot` the kernel layer's own composition root would be — see [Where the boot splits](#where-the-boot-splits). `pipelex.core` was six entries until its interpreter tenants moved out — see [Where core splits](#where-core-splits) for what that enumeration was working around.- `graph`, `tracing`, `observer` and `errors` joined the list late, and the delay cost something — see [Known inversions](#known-inversions). They are the run-graph data model with its tracer and renderers, the trace-event assembler feeding it, the run-observation hooks, and the error taxonomy: machinery present at execution time whatever is loaded, which is the kernel layer's own definition. `pipelex.test_extras` is on the list for the same reason: it ships, `runtime_boot.py` imports it at boot, and it measures clean. `pipelex.kernel` is the same rule applied *ahead* of a breach instead of behind one: it was declared in the commit that created the package, because it holds the operator-execution semantics the interpreter's operator classes call into, and a single `interpreter_hub` reach anywhere in it would put the interpreter back into every programmatic caller — the property the package exists to provide. `pipelex.kit` is data files with no module to police; `pipelex.language`, `pipelex.runtime_bridge` and `pipelex.cli` all measure dirty and are interpreter-side by construction. That is every top-level package accounted for — which is the point, because the rule of thumb is that a measured-clean package should be *declared*, not left undeclared: **an undeclared package is not neutral, it is unpoliced.**+ `graph`, `tracing`, `observer` and `errors` joined the list late, and the delay cost something — see [Known inversions](#known-inversions). They are the run-graph data model with its tracer and renderers, the trace-event assembler feeding it, the run-observation hooks, and the error taxonomy: machinery present at execution time whatever is loaded, which is the kernel layer's own definition. `pipelex.test_extras` is on the list for the same reason: it ships, `runtime_boot.py` imports it at boot, and it measures clean. `pipelex.kernel` is the same rule applied *ahead* of a breach instead of behind one: it was declared in the commit that created the package, because it holds the operator-execution semantics the interpreter's operator classes call into, and a single `interpreter_hub` reach anywhere in it would put the interpreter back into every programmatic caller — the property the package exists to provide. `pipelex.kit` is data files with no module to police; `pipelex.language`, `pipelex.runtime_bridge` and `pipelex.cli` all measure dirty and are interpreter-side by construction. `pipelex.migration` measures dirty too, and it is the one entry where that is worth explaining rather than asserting: it never reaches `interpreter_hub`, so **declaring it would pass the guard** — and would be the overreaching declaration this page's first [known inversion](#known-inversions) is about, because its replay engine imports `pipeline.fixes.applier`, deliberately, so that a configuration migration and an `.mthds` fix apply operations through one applier and one set of transaction primitives. It stays undeclared and interpreter-side, and the kernel layer reaches it the only way it may: `system/configuration/config_surface.py` defers the engine import into the boot-tolerance retry, so a healthy boot loads none of it. That deferral is load-bearing and the lint cannot see it — a module-level import there keeps `make check-hub-layering` green and turns the [closure test](#the-property-the-import-closure-test) red at every kernel-layer entry point that reaches the loaders, which is that test's whole reason for existing. That is every top-level package accounted for — which is the point, because the rule of thumb is that a measured-clean package should be *declared*, not left undeclared: **an undeclared package is not neutral, it is unpoliced.**2. **The dead-module rule.** *No* scanned module may reference `pipelex.hub`. It was deleted rather than aliased so a stale import fails loudly; this closes the one hole in that guarantee.3. **The transitive rule.** No kernel-layer module may *reach* `pipelex.interpreter_hub` either, through any chain of module-level imports. Rules 1 and 2 are per-file and see one hop; this one resolves the whole module-level import graph of `pipelex/` and does reachability over it, reporting the shortest chain and the line of its first hop. It exists because the one-hop rules missed a live breach for a whole release: four modules of the declared kernel-layer `pipelex.plugins` package reached the interpreter hub through `runtime_bridge`, `pipeline` and `pipe_operators`, and both gates stayed green (see [Known inversions](#known-inversions)). A *direct* import is left to rule 1, which names the more specific remedy, rather than reported twice.diff --git a/docs/errors/pipelex-config-error.md b/docs/errors/pipelex-config-error.mdindex 3acb7b024..48ab87685 100644--- a/docs/errors/pipelex-config-error.md+++ b/docs/errors/pipelex-config-error.md@@ -7,6 +7,8 @@ description: "Reference for the `PipelexConfigError` Pipelex error class."# Pipelex config+A configuration this process cannot use, optionally with the migration that explains it.+| Field | Value ||---|---|| `error_type` | `PipelexConfigError` |diff --git a/docs/errors/telemetry-config-error.md b/docs/errors/telemetry-config-error.mdindex 67ab14b8b..9e89204ba 100644--- a/docs/errors/telemetry-config-error.md+++ b/docs/errors/telemetry-config-error.md@@ -7,6 +7,8 @@ description: "Reference for the `TelemetryConfigError` Pipelex error class."# Telemetry config+A telemetry configuration this process cannot use.+| Field | Value ||---|---|| `error_type` | `TelemetryConfigError` |@@ -14,6 +16,6 @@ description: "Reference for the `TelemetryConfigError` Pipelex error class."| `type_uri` | `https://docs.pipelex.com/latest/errors/telemetry-config-error/` || `error_domain` | _(inherited from parent)_ || Defined in | `pipelex.system.telemetry.exceptions` |-| Parent class | [`PipelexError`](pipelex-error.md) |+| Parent class | [`PipelexConfigError`](pipelex-config-error.md) |[Back to Error Reference](index.md)diff --git a/docs/migration-ledger.md b/docs/migration-ledger.mdindex 2af4c83c4..3644054be 100644--- a/docs/migration-ledger.md+++ b/docs/migration-ledger.md@@ -24,7 +24,7 @@ A **surface** is one artifact family with one schema version and one ledger. The| `telemetry-config` | `telemetry.toml` | `telemetry_*.toml` | `TelemetryConfig` | Model defaults || `pipelex-service-config` | `pipelex_service.toml` | — | `PipelexServiceConfig` | Model defaults |-File names are relative to a configuration directory, and every surface spans exactly two of them: the global `~/.pipelex/` and the project's `.pipelex/`. `pipelex migrate` walks both, and only those. Two paths that look like configuration are outside the walk by design: the unit-testing tier `./tests/pipelex_{run_mode}.toml`, and any file reached through an explicit `load_config(config_dir=…)`, which is a repository-internal mechanism rather than a user's configuration.+File names are relative to a configuration directory, and every surface spans exactly two of them: the global `~/.pipelex/` and the project's `.pipelex/`. `pipelex migrate` walks both, and only those. Two paths that look like configuration are outside the walk by design: the unit-testing tier `./tests/pipelex_{run_mode}.toml`, and a directory an embedder hands `Pipelex.make(config_dir=…)` that is neither of the two. Boot tolerance and the error-path diagnosis still read such a directory (see [Boot tolerance](#boot-tolerance)), but its files are the embedder's to bring up to date: `pipelex migrate` has no way to be pointed at them, and does not pretend otherwise.The tier set is **open**. Environment and run-mode names are dynamic, so the tier filenames of `pipelex-config` cannot be enumerated in advance — `pipelex_local.toml`, `pipelex_{environment}.toml`, `pipelex_{run_mode}.toml`, `pipelex_override.toml` and `pipelex_temporary_override.toml` are a description of today, not a closed list. That is why the registry matches tiers with a glob, and the glob is what makes the resolution rule necessary:@@ -65,7 +65,7 @@ The reasoning is worth keeping. Any side record of "what has already been applieTwo reservations exist so that later projects stay free:- **`[meta] schema_version`** is a reserved optional in-file key. Every configuration-surface reader tolerates it and strips it before validation; **nothing writes it**. The reason not to write it is not release compatibility, which our principles disclaim, but team skew on tracked files: base configuration files under a project's `.pipelex/` are shared through git right now, so one developer on a newer pipelex writing the key would break a teammate on last week's build the same afternoon.-- **`min_supported_schema_version`** on the surface block, held at zero until a ledger squash ever moves it. Without the floor, a squash silently under-migrates the oldest files in the field, because the applier skips absent targets and reports success. With it, the loader fails loudly instead.+- **`min_supported_schema_version`** on the surface block, held at zero until a ledger squash ever moves it. Without the floor, a squash silently under-migrates the oldest files in the field, because the applier skips absent targets and reports success. With it, a file that *declares* where it stands is refused by name — `unsupported_schema_version`, nothing written — instead of being run over and reported clean. The two reservations are one mechanism: a document's declaration is the only evidence a floor can act on, which is why the migration reads the reserved key exactly as tolerantly as the loader strips it. A file declaring nothing, which is every file in the field, is migrated as always.### The downgrade direction@@ -75,7 +75,14 @@ Under the no-backward-compatibility principle this is not repaired, but it must> When validation fails on an unknown path that no ledger entry removes, the report says so: the path is either a typo or was written by a newer pipelex than this one, and the message asks whether the user is on an older branch or build.-That diagnosis is computable from the ledger alone and needs no stamp.+That diagnosis needs no stamp, and every run produces it — it lands in a plan's `unexplained[]`. Four rules make its answer worth reading:++- **It is the one part of a migration that needs the model.** Whether the current schema knows a path cannot be answered from a ledger, so the question goes to the surface's [fingerprint](#the-fingerprint) — the same projection the coverage gate diffs. The engine stays model-free, and the diagnosis sits beside the runner, which has a surface.+- **The document diagnosed is the one the run leaves behind.** Everything the ledger explains has been carried forward by then, so what is left over is genuinely left over. On a dry run that document exists only in memory, which is why the diagnosis belongs to the run and not to a later pass over the file.+- **A blocked entry answers for its own material.** An `unsafe` entry is never applied, so the old shape it is about is still in the file — already reported, by name, with the entry's guidance. Saying the same key is an unexplained typo would contradict that. The subtraction deliberately over-covers: every path it removes is one the same report names in `blocked[]`.+- **A key the user chose is reported as the schema spells it.** Beneath an open mapping the schema says `levels.*` where the file says `levels.my_package`, and a typo *inside* such an entry is reported at `queues.*.retries` — the same rule that governs a blocked entry's `narrowed_paths[]`. Only the segment the schema does not know is named, because naming it is the whole point.++Two things are deliberately not reported. An unknown *table* is named once instead of once per key inside it, since the shallowest name is the one to fix. And a document that nests below a path the schema says is a scalar is left alone: that is a type error, which the model reports far better, and descending would invent unknown paths beneath a path the schema knows perfectly well.## Replay neutrality@@ -468,6 +475,7 @@ There is one file-blocked reason per **state the file is in**, rather than one p|---|---|| `unreadable` | It is there and its bytes would not come. Nothing was written. || `unparseable` | It was read and is not valid UTF-8, or not valid TOML. Nothing was written. |+| `unsupported_schema_version` | It declares a `[meta] schema_version` below its ledger's floor, so the entries that would carry it forward are no longer there. Nothing was written. || `unwritable` | It needed a change and the run could not make it — the backup would not go down, or the replacement would not. The file is exactly as it was found. || `changed_during_run` | It was removed or edited between the read and the write, so the run refused to write over work it had not seen. || `state_uncertain` | The write could not be confirmed: the transaction could not describe what it left behind, and the file does not hold what the run wrote. |@@ -482,23 +490,47 @@ Two rules govern how an entry appears, and both exist so that a report is never> **A blocked entry's reason says which claim it is making.** `unsafe` means the file has the old *shape* and the applier will not change it. `conflict` means an operation's destination is already occupied. `value_domain_narrowed` is the weakest and the newest: the file *sets* a path whose accepted values the entry narrowed, and `narrowed_paths[]` lists those paths as the **ledger** spells them — `levels.*`, never the user's own `levels.my_package` — and at the spelling the file this run wrote carries, not the one that matched. It is a list of keys to check by hand rather than a list of errors, because telling one from the other needs the model and the engine has none by design.+> **An unexplained path is what neither the schema nor the ledger accounts for.** `unexplained[]` carries the paths of the migrated document that the current schema does not know and no entry — applied or blocked — explains, each with the two readings the tool cannot tell apart: a typo, or a file written by a newer pipelex. The whole rule is [the downgrade direction](#the-downgrade-direction).+> **No value read from a user's file is ever rendered** — not in the command's output, not in the structured plan, not in an error. Paths, operation kinds and ledger-supplied values carry everything a plan needs to say.That is a mechanical rule rather than a list of credential-shaped key names, because such a list is a guess that eventually misses one. The single deliberate exception is the backup file, which contains the user's values by definition and is protected by inheriting the source file's mode rather than the process umask.When migration is reported through a validation error, the error keeps `error_domain: "config"` and gains a structured `migration` block carrying the plan, the remedy and the diagnosis. Consumers branch on the presence of that block, never on wording. Following the workspace convention: the structured fields are the contract, and Markdown, exit codes and HTTP statuses are presentation.+## The commands++Two commands run a migration, and they are the same run with two audiences.++| | `pipelex migrate` | `pipelex-agent migrate` |+|---|---|---|+| Reader | a person | a program |+| Default | plan, show it, ask | plan, and write nothing |+| Writes when | the question is answered, or `--yes` | `--yes` |+| `--dry-run` | plan and stop | plan and stop (the default, said explicitly) |+| Answer | Rich output | JSON, or Markdown with `--format markdown` |++**Both walk the global `~/.pipelex/` and the project `.pipelex/`, and nothing else.** An embedder's `config_dir=` outside those two and this repository's own `tests/pipelex_{run_mode}.toml` are outside the walk: the first is the embedder's to update, the second is not a user's configuration at all, and migrating a directory nobody asked about is how a tool earns a reputation for touching things. A directory that does not exist is skipped, and a project rooted at the home directory is walked once rather than twice.++**The walk is not recursive.** A surface's tier files sit beside its base file; a subdirectory holds a different kind of thing. The specimen is real: `.pipelex/inference/backends/pipelex_gateway.toml` matches the `pipelex-config` tier glob `pipelex_*.toml` exactly and is an inference backend definition, so a recursive walk would replay the main configuration's ledger over it.++> **`--dry-run` and `--yes` together are refused, not resolved.** One asks for no write and the other authorizes one; picking a winner would hide the bug that produced both.++**Neither command boots.** A broken configuration is the reason to reach for `migrate`, so needing a working one would make it useless in exactly the case it exists for. What a migration may use is the ledger, the applier and the filesystem: no configuration load, no model deck, no credentials, no network. That is a property under test rather than an accident, and the test is what keeps a future import from creeping into the list.++**The structured fields are the contract.** `needs_attention` is the verdict — *this run left something a person has to decide* — and it is deliberately not "did anything get written": a run that migrated every file it found has succeeded, and so has a dry run that found nothing blocked. The exit code (`1` when `needs_attention`, `2` on a contradictory pair of flags) and the rendering are presentation, and follow the workspace convention rather than carrying the verdict.+## Boot toleranceA stale configuration should warn, not stop the world — but only when the ledger can explain it.```load user files → deep-merge → validate (extra="forbid")- ok → boot; no ledger touched, no tomlkit loaded+ ok → boot; no ledger read, no document re-parsedfails, surface S- → re-load S's user files with tomlkit- → replay S's `safe` entries in memory, per file (writes nothing)- → re-merge → re-validate+ → re-read S's user files with tomlkit+ → replay S's ledger in memory, per file (writes nothing; `unsafe` entries are rehearsed, never applied)+ → re-merge → re-run the loader's own post-merge step → re-validateok → WARNING naming the file and the `pipelex migrate` remedy → bootfails → unsafe entries, CONFLICT, or unexplained paths → validation errorwith error_domain "config" and the `migration` block@@ -506,11 +538,80 @@ load user files → deep-merge → validate (extra="forbid")The rules this encodes:-- **Boot never writes.** Nothing writes except the explicit `migrate` command; boot, `doctor` and validation detect and report.-- **Boot tolerates only what the ledger explains.** `unsafe` entries, conflicts and unexplained paths still fail the boot. Tolerance widens what starts, never what is silently accepted.-- **The healthy path is untouched.** The replay runs only on the failure path, so a current configuration never loads tomlkit and never reads a ledger. Replay neutrality is what makes the retry free when it does run.+- **Boot never writes.** Nothing writes except the explicit `migrate` command; boot, `doctor` and validation detect and report. A tolerated boot leaves no backup either, which is why the warning keeps coming back until the user runs the command.+- **Boot tolerates only what the ledger explains.** Tolerance widens what starts, never what is silently accepted. **The re-validation is what decides**, and that is stronger than a second gate on the report would be: material an `unsafe` entry is about is still in the file, so the model refuses it and the boot fails, and a `VALUE_DOMAIN_NARROWED` report the model accepts was never a reason to refuse a boot — that report says *check this key*, and the model has now checked it.+- **The retry honours the [schema-version floor](#schema-versions-and-why-every-run-replays-everything).** A file that declares a version below `min_supported_schema_version` is one `pipelex migrate` refuses; the retry declines it through the same predicate rather than carrying it forward under-migrated, and the user's own error — with the `migration` block naming the floor — is what they see.+- **The retry never becomes the failure.** Anything that goes wrong inside it — a ledger that will not load, a file that will not parse — makes it decline, and the error the *configuration* produced is what the user sees. Their error names the key to fix; ours would name our packaging. A file that cannot be re-read abandons the whole retry rather than being skipped, because skipping it would drop a layer from the merge and a re-validation that then succeeded would boot on a configuration the user does not have.+- **The main configuration's warning waits for the logger.** That configuration is what configures logging, so when its retry succeeds there is no logger yet — the loader parks the warning (`config_manager.take_stale_configuration_warning()`) and the boot emits it right after `log.configure`. The telemetry and service loaders run after that point and warn directly. The warning is the same either way; only *when* it is said moves.+- **The healthy path is untouched.** The replay runs only on the failure path, so a current configuration never reads a ledger, never re-parses a document, and never even imports the migration engine. (Not "never loads tomlkit": the ordinary configuration read has always imported it. What a healthy boot avoids is the second, DOM-level read.) Replay neutrality is what makes the retry free when it does run.- **One shared helper**, called by each configuration-surface loader — and that helper is also where `[meta] schema_version` is stripped. The strip belongs there and **not** in the generic TOML reader, which also reads `.mthds` files, backend definitions and the kit index, none of which reserve that key.+**The helper owns the failure path, not the load.** The three loaders do different things between their merge and their validate — the main configuration deep-merges programmatic overrides, telemetry substitutes `${VAR}` placeholders, the service configuration does nothing — so a helper that owned the whole load would have to be told about all three. It is given the surface id and the same ordered path list the loader merged, and it hands back the migrated merge plus a `MigrationPlan` per file; each loader re-runs its own step over that and re-validates. The overrides in particular have to be re-applied by the caller: they are a layer of the *load*, and the replay only ever sees the *files*.++> **The migration engine is imported inside the retry, not at the top of the module.** Its applier lives under `pipelex.pipeline` — an interpreter package — while the configuration loaders sit in `runtime_hub`'s import closure, and the kernel layer's property is that importing it loads zero interpreter modules (see [`hub-layering.md`](contribute/hub-layering.md)). A module-level import would break that silently: the layering guard mechanically checks reachability to `interpreter_hub`, which `pipelex.pipeline.fixes.applier` does not have, so nothing would have gone red. The deferred import is also what makes "the healthy path is untouched" literal rather than approximate.++**Boot tolerance does not run the [downgrade diagnosis](#the-downgrade-direction).** On this path the model has already spoken, and pydantic's own extra-field list is both the same answer and a better one — it knows about validators, which a path walk does not. The diagnosis exists for `pipelex migrate`, where nothing validates the file at all. So "unexplained paths still fail the boot" is satisfied by the re-validation failing, and the `migration` block on the error carries the plans.++## Reporting a stale configuration on a validation error++When the boot's retry declines and the model's refusal stands, the error the user gets says one more thing than pydantic can: whether their configuration is *wrong* or merely *old*.++A validation error cannot answer that by itself. It is raised against the merged configuration and carries no provenance — it names a key, not which of the files that were merged put it there, and certainly not whether that key was correct last month. So `report_validation_error` asks the files instead: a **dry-run scan** of the surface that refused, over the same directories a `pipelex migrate` would walk — or, when the caller loaded one explicit directory (`doctor --global`, an embedder's `config_dir=`), over that directory alone, so the block never names a file the reader did not load.++**The scan is named, not guessed.** The caller passes the surface whose model refused; a caller validating something that is not a configuration surface — a `.mthds` bundle, an inference backend file, a model deck — passes nothing and gets the translation alone. None of those has a ledger, and offering them a `pipelex migrate` remedy would send a user to a command with nothing to do. That is also why the scan does not run at all when no surface is named: it is a directory walk and a ledger replay, and a bundle-validation failure has no business paying for one.++> **Scoping narrows the answer, never the registry.** Which surface owns a file is decided across *all* of them, because an exact base file claims before any glob. A registry built to hold only the surface being asked about removes the other claimants from that arbitration, and `pipelex_service.toml` — another surface's base file, and a match for `pipelex-config`'s `pipelex_*.toml` — is then replayed under the wrong ledger and diagnosed against the wrong model, so its ordinary settings come back reported as paths this build knows nothing about.++The answer rides the error twice, and the two halves are not interchangeable:++- **`error_domain` stays `"config"`.** It does not become a domain of its own. `ErrorDomain` is a closed cross-repo enum and the agent-hook specification routes any domain it does not know to BLOCK, so a stale configuration reported under a new domain would stop an agent rather than tell it what to run.+- **The structured `migration` block is the contract, and consumers branch on its presence.** Absent means the failure is not staleness. Present, it carries `remedy` (the command), `needs_attention` (whether anything here is a person's rather than the tool's), and `plans` — the same `MigrationPlan` shape `pipelex-agent migrate --dry-run --format json` emits under its own `plans` key, so an agent that parses one has already parsed the other. Only the files the scan found something in are listed: unlike the commands' report, which answers *what did the walk visit*, the block answers *what is wrong with this machine*.+- **The message is presentation.** It carries the pydantic analysis followed by a paragraph naming the files, what the command would carry forward, and what it cannot do for anyone. Nothing branches on its wording.++The agent loop this opens is the one the commands were built for: a command fails, the block is present, the agent runs `pipelex-agent migrate --dry-run --format json`, shows the user what would change, and runs `--yes` on confirmation. It never hand-edits a configuration file.++> **No value read from a user's file appears in the block either.** This is the third of the three channels that rule covers, beside the command's own output and its structured plan. The block reports paths, operation kinds and ledger-supplied values — a key holding a live credential is named by its *path* and never by what it holds.++**A failure inside the scan is never the failure the user sees**, for the same reason the boot retry declines quietly: they have an error in front of them that names what to fix, and replacing it with a packaging problem of ours would cost them the only message that helps. A ledger that will not load stays loud where it should be — `make check-ledger`, and `pipelex migrate` itself. The catch is narrow rather than blanket, so a bug in our own applier still surfaces as the bug it is.++### Every surface reports it the same way, and none of them says "start over"++A configuration error reaches a user through several surfaces at once — the human CLI's panel, the agent CLI's JSON envelope, the `doctor` row — and before the ledger each of those carried its own hardcoded remedy. The telemetry ones are the case worth naming, because they were written for a real event and then aged into the wrong advice: *the telemetry.toml format has changed, run `pipelex init telemetry`*, shown for the very flat file that entry `telemetry-config@2` exists to carry forward. That command writes a fresh file. It would have taken the PostHog key, the Langfuse credentials and the OTLP exporters with it.++So no surface holds its own answer any more:++- Every configuration surface's loader raises through `report_validation_error`, so the `migration` block reaches the error wherever it is caught. `TelemetryConfigError` is a `PipelexConfigError` for this reason — it carries `error_domain = CONFIG` from the class rather than from the agent CLI's lookup table, and it can carry a block.+- The **human handler** prints the fields the model refused and then names `pipelex migrate` when the block is there, or an edit when it is not.+- The **agent envelope** carries `migration` as a field and its hint says which of the two readings applies; an agent branches on the field, never on the sentence.+- The **`doctor` row** carries a *finding* — `healthy`, `not_found`, `unparseable`, `out_of_date`, `invalid` — and every caller reads that instead of the row's wording. This one is a correctness fix and not only a tidy-up: `--fix` used to decide what it could repair by searching the message for `"format has changed"`, so rewording the row would have switched the whole repair path off in silence. The finding answers for the one file the row is about: the scan behind `out_of_date` runs over the directory that file lives in — which also holds the `telemetry_*.toml` tier — and the verdict is read off that file's own plan, so a stale tier file beside a wrong base file leaves the base file `invalid`, and the fields the model refused stay on the row.++> **Writing a fresh file repairs exactly one finding: a missing one.** There is nothing in a file that is not there to lose. Every other unhealthy state has the user's own settings in it, so an out-of-date file is offered `pipelex migrate` and a broken one is left to a person. Regeneration is still reachable, and it is described as what it is — a way to start over that discards the file.++## The health report++The boot warns, and the validation error explains — but both of them need something to have gone wrong first, and one of them cannot reach a machine at all. `pipelex-agent` cuts Python's logging off process-wide as its first act, so an agent never sees a boot warning; for a program, **asking is the only channel**, and `pipelex doctor` is the asking.++The doctor's configuration-migration row is a **dry run of `pipelex migrate`** — the command's own run with the writing switched off, not an approximation of it. It reports one of four findings:++| Finding | What it means | What comes next |+|---|---|---|+| `up_to_date` | every file the walk claims is at the current schema | nothing |+| `pending` | at least one file would be rewritten | `pipelex migrate` (and `--fix` offers to run it) |+| `needs_attention` | something is there the command will not do on its own | `pipelex migrate --dry-run`, then a person |+| `unavailable` | the scan itself could not run | check by hand; this is our problem, not the machine's |++Two of those distinctions carry weight. `unavailable` is separate from the rest because **a packaging problem of ours must not be reported as a finding about the user's files** — and, more sharply, must not be reported as health: not knowing is not the same as being up to date. And `pending` is separate from `needs_attention` because only one of them has a command behind it; offering to run a migration over a file it would not change is a prompt whose honest outcome is *nothing was written*.++A run is often both at once, which is the ordinary shape on a machine that has drifted, so the row lists **both sets of files** and names both moves. A reader — or an agent — that heard only the first would stop with a broken file still in place.++> **The row takes no directory, and that is a decision.** Every other check in the doctor reports on a *file* and is scoped to the directory the doctor was pointed at, `--global` included. This one reports on a *command*, and `pipelex migrate` has no `--global`: it walks the global `~/.pipelex/` and the project `.pipelex/` both. A row scoped narrower would name a command that then rewrites a file the row never mentioned — and a tool that writes to a user's files must not spring that. Over-reporting is legible instead: every file is named with its full path, so a reader sees which directory each one is in.++**`--fix` runs the same write pass the command runs.** The row above it *was* the dry run — the same two-pass shape `pipelex migrate` has, with the doctor asking the question in the middle — and fix mode reaches the command's own write half rather than a second implementation that could drift from it. What it does not do is call the command itself: that ends the process when something is left for a person, and the doctor still has rows to render and an exit code of its own to set.++**Nothing inside a file is rendered here either.** The row reports paths and counts. It is the fourth channel the rendering rule covers, beside the command's output, its structured plan, and the block on a validation error.++**A failure inside the scan costs more here than anywhere else**, which is why it is caught rather than raised: an exception escaping this probe reaches the doctor's own outer handler, which prints one line and exits — so a broken packaged ledger would replace *every row the user came for* with "Unexpected error". The catch stays narrow (`MigrationError`, `OSError`), so a bug in our applier still surfaces as itself.+## Applying> **Operations apply to the user's file, and a template is never the remedy.** "Delete your configuration and re-initialize it" is the failure this project exists to remove, not a fallback it may reach for: re-initializing throws away every choice the user made, and it is exactly what a structural change should not cost them.@@ -557,7 +658,7 @@ Three further rules make that safety net worth the name.What a backup carries across is the file's **permission bits**, and that is the one deliberate exception to "no value read from a user's file is ever rendered" — a backup contains the user's values by definition, so a `0600` configuration must not acquire a world-readable copy beside it. Ownership, ACLs and extended attributes are **not** carried across the replace, on either the backup or the migrated file: an atomic same-directory replace cannot preserve what the running process has no right to set, and re-attaching an attribute blindly (a quarantine flag, a security label) is a worse guess than leaving it off. The security-relevant bit of a configuration file is its mode.-A configuration file that is a **symlink** is followed: the file the user means is the one at the end of the link, so the run reads, backs up and replaces *that* file, and the link survives. Replacing the link path instead would put a regular file where the link was and leave the real file unmigrated. The plan keeps naming the path the directory walk found; the backup path shows where the bytes actually went. This is what the `.mthds` fix loop already does with its own targets. The two callers still differ on one half: the fix loop pairs its resolution with a write-scope check, so a link pointing outside the directories it was given is refused, and the migration runner has no equivalent — a configuration file symlinked outside every walked directory is migrated where it actually lives. Whether that should be guarded is open; see `wip/migrator-write-scope-and-rename-fidelity.md`.+A configuration file that is a **symlink** is followed: the file the user means is the one at the end of the link, so the run reads, backs up and replaces *that* file, and the link survives. Replacing the link path instead would put a regular file where the link was and leave the real file unmigrated. The plan keeps naming the path the directory walk found; the backup path shows where the bytes actually went. This is what the `.mthds` fix loop already does with its own targets. The two callers differ on one half, and the difference is deliberate: **a migration's write scope is the resolved target of any file the walk claims**, not the walked directories themselves. The `.mthds` fix loop pairs its resolution with a write-scope check because it is handed a bundle directory and must not write outside it; a configuration directory is a place a user keeps links to files they own, and a dotfiles repository is the ordinary reason one is there. Refusing it would mean the tool declines to migrate exactly the machines whose owner was most deliberate about their configuration. The plan names the walked path and the backup names the resolved one, so a run that followed a link out of the directory says so. The reasoning behind both readings is in `wip/migrator-write-scope-and-rename-fidelity.md`.### Per-file transactionsdiff --git a/docs/tools/cli/agent-cli.md b/docs/tools/cli/agent-cli.mdindex a46899884..d53d27e90 100644--- a/docs/tools/cli/agent-cli.md+++ b/docs/tools/cli/agent-cli.md@@ -187,6 +187,14 @@ Do not confuse this setup-warning shape with the `warnings` array on the `valida}```+For the structured commands, exactly one such object is written to stderr per failure, so the stream parses as a single JSON document.++**`migration`** — a configuration error (`"error_domain": "config"`) carries an extra `migration` object when, and only when, a scan of this machine's configuration directories found something. Its presence is the whole signal: the configuration is *out of date* rather than wrong. It holds `remedy` (the command that fixes what can be fixed), `needs_attention` (whether something there is a person's decision rather than the tool's), and `plans` — one per file, in the same shape [`pipelex-agent migrate`](migrate.md) emits under its own `plans` key. The loop from there is `pipelex-agent migrate --dry-run --format json`, show the user what would change, then `--yes`. Never hand-edit a configuration file, and never answer this by re-initializing one — that discards every setting the migration would have kept.++`TelemetryConfigValidationError` carries it too, on the same terms: present means the `telemetry.toml` is old, absent means it is wrong and the fields the message names are what to correct.++**A failure is not the only way to learn this, and for an agent it is not even the reliable one.** A configuration the migration history explains boots with a warning rather than an error — and `pipelex-agent` silences logging process-wide before anything can emit one, so no error arrives and the machine is still stale. `pipelex-agent doctor` is the channel that always answers: its `checks.pending_migrations` row carries a `finding` (`up_to_date`, `pending`, `needs_attention`, `unavailable`), the `migratable_files` a migration would rewrite, and the `attention_files` it will not repair on its own. Branch on `finding`; `unavailable` means the check could not run and is not a claim that the machine is current. The row answers for both configuration directories even under `--global`, because it describes `pipelex migrate`, which has no such flag.+`fmt` and `lint` are raw passthroughs to the `plxt` binary and bypass the JSON output contract, producing native `plxt` output instead.## Graph Visualizationdiff --git a/docs/tools/cli/index.md b/docs/tools/cli/index.mdindex fbc0870dc..8d98a579a 100644--- a/docs/tools/cli/index.md+++ b/docs/tools/cli/index.md@@ -15,6 +15,7 @@ The Pipelex CLI is organized into several command groups:|---------|-------------|| [**init**](init.md) | Initialize Pipelex configuration || [**update**](update.md) | Refresh the model deck to match the installed pipelex version |+| [**migrate**](migrate.md) | Bring your configuration files up to the schema the installed version expects || [**validate**](validate.md) | Validate configuration and pipelines || [**fix**](fix.md) | Apply deterministic safe fixes to a bundle and re-validate || [**show**](show.md) | Inspect configuration, pipes, and AI models |diff --git a/docs/tools/cli/init.md b/docs/tools/cli/init.mdindex 1429f233d..3b32c1941 100644--- a/docs/tools/cli/init.md+++ b/docs/tools/cli/init.md@@ -66,6 +66,9 @@ This command creates or resets a Pipelex config directory with:- `deck/` - AI model aliases and presets- **telemetry.toml** - Telemetry and observability settings+!!! warning "Init writes a fresh file — it does not update one"+ Every `init` target replaces the file with the template, so whatever was in it is gone. If a configuration file has simply fallen behind the current schema, [`pipelex migrate`](migrate.md) is the command: it rewrites the file in place and keeps every setting — your PostHog key, your Langfuse credentials, your exporters. `pipelex doctor` tells you which of the two you have.+!!! note "Offline cache priming"When Pipelex Gateway is enabled, `pipelex init` also primes an on-disk copy of the gateway's remote config at `~/.pipelex/cache/remote_config.json`. This lets later setup, validation, and dry-runs work even when the remote config service is briefly unreachable. If you run `pipelex init` while offline, it prints a yellow warning and skips priming — re-run it online to enable offline dry-runs.diff --git a/docs/tools/cli/migrate.md b/docs/tools/cli/migrate.mdnew file mode 100644index 000000000..8193f0743--- /dev/null+++ b/docs/tools/cli/migrate.md@@ -0,0 +1,64 @@+---+title: "CLI Migrate"+description: "Bring the Pipelex configuration files on your machine up to the schema the installed version expects."+---++# `pipelex migrate`++Pipelex's configuration files occasionally change shape — a setting moves into a new section, a section is renamed. When that happens, the files already on your machine still hold your choices, but the installed Pipelex no longer recognizes them, and the boot fails.++`pipelex migrate` repairs those files in place. It keeps every value you set; it never replaces a file with a fresh template.++```bash+pipelex migrate # show what would change, then ask+pipelex migrate --dry-run # show what would change and stop+pipelex migrate --yes # apply without asking+```++## What it touches++Two directories, and only those:++- the global `~/.pipelex/`+- the project `.pipelex/`, when the current directory is inside a project that has one++Within each, it looks at the configuration files themselves — `pipelex.toml` and its `pipelex_*.toml` tiers, `telemetry.toml` and its tiers, `pipelex_service.toml`. It does not descend into subdirectories, so your inference backends and model deck are never rewritten by it.++## What it does to a file++Every file it changes is copied first, beside itself, as `<file>.bak.<UTC timestamp>` — with the original file's permissions, not your umask. The copy is on disk before the file is replaced, and the replacement is atomic: there is never a moment when your configuration is half-written.++Running it twice is the same as running it once. Nothing is skipped on the basis of a version record, because there is no version record; every run replays the whole history and leaves alone whatever is already current. A file that is already up to date comes back byte for byte identical.++## When it cannot do the whole job++Some changes cannot be made for you. A setting whose accepted values narrowed, for instance, needs someone who knows what the value was *meant* to be — so Pipelex names the key and leaves it to you rather than guessing. The report says which file, which key, and what to look at. Everything it *can* do is still done, and one file it cannot process never stops the others.++The command exits non-zero when it leaves something for you to look at.++## Running it when nothing else runs++`pipelex migrate` does not boot Pipelex. That is the point: a configuration that cannot load is exactly when you need it, so it uses the migration history, the file editor and the filesystem, and nothing else. It works with no credentials, no model deck and no network.++## The warning that sends you here++You will usually meet this command through a warning rather than a crash. When a configuration file is out of date in a way the migration history explains, Pipelex boots anyway: it carries the file forward **in memory**, tells you which files it did that for, and points at this command. Nothing is written, so the same warning appears at the next boot, and the one after — running `pipelex migrate` is what makes the change to the file and stops it.++A file the history cannot explain is a different case, and it still fails the boot with the configuration error itself: tolerance widens what starts, never what is accepted. That error names this command too, though — beside the key it could not accept, it tells you which of your files a migration would touch, what it would carry forward, and what only you can decide.++`pipelex doctor` is the third way here, and the one to reach for when nothing has gone wrong yet. It has a **Configuration Migrations** row that is this command's own dry run: it names every file a migration would rewrite, with its full path, and separately any file carrying something the command will not do on its own. `pipelex doctor --fix` then offers to run the migration for you — the same run, after showing you what it is about to do.++That row answers for both configuration directories, the global one and the project one, even when the rest of the report is about a single directory. It is describing a command, and the command walks both — a row that named fewer files than the command touches would be the one surprise worth avoiding here.++Nothing that Pipelex tells you about an out-of-date file will ever suggest deleting it and starting over — that is what this command exists to make unnecessary.++## For agents++`pipelex-agent migrate` is the machine-facing counterpart. It writes only when passed `--yes`, since it cannot ask, and answers with a structured plan — `--format json` for the contract, `--format markdown` to read. Branch on the `needs_attention` field, never on the exit code.++An agent usually meets this command through a failure rather than by choosing it. A configuration error carries a `migration` field when — and only when — a scan of the machine found something; its presence is the signal that the configuration is *old* rather than *wrong*. The loop from there is `pipelex-agent migrate --dry-run --format json`, show the user what would change, then `--yes` on confirmation. Never hand-edit a configuration file.++## See also++- [`pipelex doctor`](index.md) — check configuration health+- [`pipelex init`](init.md) — create configuration files from scratchdiff --git a/docs/under-the-hood/init-cli-flows.md b/docs/under-the-hood/init-cli-flows.mdindex 8344237a3..fade06030 100644--- a/docs/under-the-hood/init-cli-flows.md+++ b/docs/under-the-hood/init-cli-flows.md@@ -34,7 +34,7 @@ These two categories are managed by separate steps. `init_config()` copies only| `pipelex init agreement` | `agreement` | Gateway terms acceptance (no reset) || `pipelex init credentials` | `credentials` | Credential setup for enabled backends |-All commands except `agreement` and `credentials` perform a **full reset** (overwrite existing files). Config updates are not yet supported.+All commands except `agreement` and `credentials` perform a **full reset** (overwrite existing files) — every setting in the file is replaced by the template's. That is why init is not the answer to a configuration file that has fallen behind the current schema: [`pipelex migrate`](../tools/cli/migrate.md) rewrites such a file in place and keeps what is in it.### Inputs@@ -454,6 +454,9 @@ flowchart TD| `check_backend_credentials()` | `resolve_config_file()` | Finds `backends.toml` wherever it lives || `check_backend_files()` | `resolve_config_file()` | Finds `inference/backends/` wherever it lives || `check_models()` gateway terms | `global_config_dir` | **Always global** — reads from `~/.pipelex/` |+| `check_pending_migrations()` | none | **Both directories, always** — see below |++`check_pending_migrations()` is the one check that takes no `config_dir` at all, and that is deliberate. Every other row reports on a *file* and is scoped to the directory the doctor was pointed at, `--global` included. That one reports on a *command* — it is `pipelex migrate`'s own dry run — and `pipelex migrate` has no `--global`: it walks the global `~/.pipelex/` and the project `.pipelex/` both. Scoping the row narrower would name a command that then rewrites a file the row never mentioned. Every file it reports is named with its full path, so the wider scope stays legible.### Fix Targeting@@ -468,6 +471,8 @@ replace_backend_file(backend_name, config_dir=resolved_config_dir)This ensures the fix targets the same directory where the issue was found — if the broken file was in the project `.pipelex/`, the replacement goes there too, not to `~/.pipelex/`.+The migration fix is the exception that follows from the row above it: it calls `apply_pending_migrations(config_dirs=config_directories_to_migrate())` and so writes to both directories, because that is what the command it is running does. It calls that helper rather than `migrate_cmd` itself — the command exits the process when a run leaves something for a person, which would cut off the doctor's remaining fixes, its remaining rows and its own exit code.+---## Next Stepsdiff --git a/mkdocs.yml b/mkdocs.ymlindex 94ba23f58..7e9fabb37 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -274,6 +274,7 @@ plugins:- tools/cli/index.md: "CLI Reference Overview"- tools/cli/init.md: "CLI Init"- tools/cli/update.md: "CLI Update"+ - tools/cli/migrate.md: "CLI Migrate"- tools/cli/validate.md: "CLI Validate"- tools/cli/fix.md: "CLI Fix"- tools/cli/run.md: "CLI Run"@@ -508,6 +509,7 @@ nav:- Overview: tools/cli/index.md- Init: tools/cli/init.md- Update: tools/cli/update.md+ - Migrate: tools/cli/migrate.md- Validate: tools/cli/validate.md- Fix: tools/cli/fix.md- Run: tools/cli/run.mddiff --git a/pipelex/base_exceptions.py b/pipelex/base_exceptions.pyindex 6b74cf706..b02c71ec1 100644--- a/pipelex/base_exceptions.py+++ b/pipelex/base_exceptions.py@@ -3,8 +3,10 @@ from enum import StrEnumfrom typing import Any, ClassVarfrom pydantic import BaseModel, ConfigDict, Field+from typing_extensions import overridefrom pipelex.cogt.inference.error_classification import ProviderErrorMetadata, UserAction+from pipelex.migration.plan import MigrationPlanfrom pipelex.suggested_fix import SuggestedFixfrom pipelex.tools.misc.string_utils import pascal_case_to_kebab, pascal_case_to_sentencefrom pipelex.urls import URLs@@ -67,6 +69,12 @@ INTERNAL_ERROR_PLACEHOLDER = "An internal error occurred."# shield — see its docstring). A hosted surface that validates from disk should# therefore use the in-memory path with logical sources rather than rely on STRICT# to scrub the path.+#+# ``migration`` is NOT surfaced, and the contrast with ``validation_errors`` is+# the whole reason: a pending configuration migration describes the *host's* own+# configuration directories — server filesystem paths, and the shape of a+# deployment's settings — never anything the caller submitted. It is diagnostics+# for whoever runs the process, which on a hosted surface is not the caller._STRICT_KEPT_FIELDS: frozenset[str] = frozenset({"error_type", "title", "type_uri", "error_domain", "error_category", "retryable", "validation_errors"})@@ -318,6 +326,45 @@ class ValidationErrorItem(BaseModel):suggested_fix: SuggestedFix | None = None+class MigrationErrorBlock(BaseModel):+ """A pending configuration migration, reported alongside the error it explains.++ The structured half of the answer to *why does my configuration not load* — carried by+ :attr:`ErrorReport.migration`, and present only when a scan of this machine's configuration+ directories found something to say. **Consumers branch on its presence**, never on the+ message text: an absent block means the failure is not staleness.++ ``plans`` is the same shape ``pipelex-agent migrate --dry-run --format json`` emits under its+ own ``plans`` key, deliberately — an agent that reads one has already parsed the other, and a+ projection of its own here would be a second contract to keep in step with the first.++ Declared in this module rather than beside the migration engine for the same reason+ :class:`ValidationErrorItem` is: :class:`ErrorReport` references it as a typed field, and+ ``base_exceptions`` must not import the migration package's error modules, which import back+ into this one. ``pipelex.migration.plan`` holds itself to stdlib + pydantic + low-level+ siblings precisely so this import stays legal.++ > **No value read from a user's file is ever rendered** here either. The plans carry paths,+ > operation kinds and ledger-supplied values, and nothing else — the same mechanical rule the+ > migration report and the CLI output obey, on the third of the three channels it names.+ """++ model_config = ConfigDict(frozen=True, extra="forbid")++ remedy: str+ """The command that applies whatever can be applied without a decision."""++ needs_attention: bool+ """Whether something here is a person's to resolve rather than the tool's — a blocked file, a+ blocked entry, or a path no ledger entry explains. When this is False, running ``remedy`` is+ expected to be enough."""++ plans: list[MigrationPlan]+ """One per configuration file the scan found something in. Files with nothing to say are left+ out: unlike the ``migrate`` commands' report, which answers *what did the walk visit*, this+ block answers *what is wrong with this machine's configuration*."""++class ErrorReport(BaseModel):"""Structured error report — single source of truth for all error serialization.@@ -359,6 +406,13 @@ class ErrorReport(BaseModel):# ``_STRICT_KEPT_FIELDS``): these describe the caller's own submitted bundle,# not server internals.validation_errors: list[ValidationErrorItem] | None = None+ # A pending configuration migration that explains this failure. Populated only by+ # ``PipelexConfigError`` reports whose raiser ran a scan (see+ # ``pipelex.core.validation.report_validation_error``); None on every other report, so a+ # consumer branching on its presence is asking exactly "is my configuration stale?".+ # Deliberately outside ``_STRICT_KEPT_FIELDS`` — it describes the host's own configuration+ # directories, not the caller's submission.+ migration: MigrationErrorBlock | None = Nonedef to_dict(self, *, disclosure_mode: DisclosureMode = DisclosureMode.VERBOSE) -> dict[str, Any]:"""Return a dict with only non-None fields, projected through ``disclosure_mode``.@@ -373,6 +427,13 @@ class ErrorReport(BaseModel):stable identifiers plus the curated ``provider_metadata`` slice."""payload = self.model_dump(exclude_none=True)+ if self.migration is not None:+ # JSON mode for this field alone. ``MigrationPlan`` carries real ``Path`` values, and+ # a payload holding one is not serializable by ``json.dumps`` — which is what the+ # webhook delivery path hands this dict to. Dumping the whole report in JSON mode+ # instead would re-serialize every other field too, so the narrow fix is the safe one.+ # ``from_dict`` still round-trips: pydantic accepts a string for a ``Path`` field.+ payload["migration"] = self.migration.model_dump(mode="json")# ``caller_facing_message`` is redaction plumbing, not consumer-facing# classification: emit it only when set, so the common (non-caller-facing)# report serializes exactly as a report without the field would.@@ -652,8 +713,26 @@ class PipelexUnexpectedError(PipelexError):class PipelexConfigError(PipelexError):+ """A configuration this process cannot use, optionally with the migration that explains it.++ ``error_domain`` stays :attr:`ErrorDomain.CONFIG` whether or not a migration is attached —+ that value is a closed cross-repo enum and the agent-hook spec routes anything else to BLOCK,+ so staleness is reported *inside* the CONFIG domain rather than as a domain of its own.+ """+error_domain = ErrorDomain.CONFIG+ def __init__(self, message: str, *, migration: MigrationErrorBlock | None = None):+ super().__init__(message)+ self.migration = migration++ @override+ def to_error_report(self) -> ErrorReport:+ report = super().to_error_report()+ if self.migration is None:+ return report+ return report.model_copy(update={"migration": self.migration})+class PipelexSetupError(PipelexError):error_domain = ErrorDomain.CONFIGdiff --git a/pipelex/cli/_cli.py b/pipelex/cli/_cli.pyindex 6bb9f12c9..2fdd24792 100644--- a/pipelex/cli/_cli.py+++ b/pipelex/cli/_cli.py@@ -13,6 +13,7 @@ from pipelex.cli.commands.graph_cmd import graph_appfrom pipelex.cli.commands.init.command import init_cmdfrom pipelex.cli.commands.init.ui.types import InitFocusfrom pipelex.cli.commands.login.command import login_cmd+from pipelex.cli.commands.migrate_cmd import migrate_cmdfrom pipelex.cli.commands.plugins_cmd import plugins_appfrom pipelex.cli.commands.resolve_cmd import resolve_cmdfrom pipelex.cli.commands.run.app import run_app@@ -32,6 +33,7 @@ _CORE_COMMAND_ORDER: list[str] = ["init","doctor","update",+ "migrate","build","validate","fix",@@ -150,7 +152,7 @@ def app_callback(""")# Skip checks if no command is being run (e.g., just --help) or if running setup/diagnostic commands- if ctx.invoked_subcommand is None or ctx.invoked_subcommand in {"login", "init", "doctor", "update", "which"}:+ if ctx.invoked_subcommand is None or ctx.invoked_subcommand in {"login", "init", "doctor", "update", "migrate", "which"}:return# Check system readiness (dependencies and venv for dev installs)@@ -225,6 +227,19 @@ def update_command(update_cmd(local=local, yes=yes, dry_run=dry_run, no_backup=no_backup)[email protected](name="migrate", help="Migrate this machine's Pipelex configuration files to the current schema")+def migrate_command(+ dry_run: Annotated[bool, typer.Option("--dry-run", help="Report what would change and write nothing")] = False,+ yes: Annotated[bool, typer.Option("--yes", "-y", help="Apply without the interactive confirmation")] = False,+) -> None:+ """Bring the configuration files in ~/.pipelex/ and the project .pipelex/ up to the current schema.++ Runs on a configuration that cannot load — it needs the migration ledger, the applier and the+ filesystem, and nothing else. Every file it rewrites is backed up beside itself first.+ """+ migrate_cmd(dry_run=dry_run, yes=yes)++app.add_typer(build_app, name="build", help="Generate AI methods from natural language requirements: pipelines in .mthds format and python code to run them")diff --git a/pipelex/cli/agent_cli/CLAUDE.md b/pipelex/cli/agent_cli/CLAUDE.mdindex 8b5eaee35..e258e559e 100644--- a/pipelex/cli/agent_cli/CLAUDE.md+++ b/pipelex/cli/agent_cli/CLAUDE.md@@ -20,6 +20,7 @@ Markdown structure per command:- `run`: `# Pipeline run complete`, a `## Result` section (the rendered `main_stuff` markdown with `--with-memory`, otherwise the concept JSON in a fenced block), and output / graph file paths. Cost is machine-first: on the local runner, when costs are on (`--costs`, the default) and the run did reportable work (any tokens or any cost), a best-effort `cost_report` object (`{total_cost, by_model: [...]}`, real USD) rides the JSON `--with-memory` envelope — there is **no** Rich cost table on stderr (unlike the human `pipelex run` CLI). The markdown renderer ignores the `cost_report` key, so it is JSON-only. A free/zero-price model still reports its token usage (with `total_cost` 0). The `cost_report` is **absent** when the run did no reportable work (dry runs), when `--no-costs` is passed, on the API-runner path (which does not assemble local usage), or if cost-summary aggregation fails (caught and skipped so it never fails an otherwise-successful run) — so consumers must treat it as optional.- `validate`: `# Validation passed`, the bundle path when relevant, and a list of validated pipes with their status. Every entry in `validated_pipes` (markdown and JSON) identifies its pipe by the namespaced `pipe_ref` (`domain.code`) — the same unambiguous identity across `validate all`, `validate bundle`, and `validate pipe`, so the same pipe is never reported under two identifiers. `validate bundle`, `validate method`, and `validate all` carry `pending_signatures` — the library-wide list of pipes still declared as `PipeSignature` (unimplemented forward declarations), namespaced by `pipe_ref` — rendered as a "Pending signatures" section in markdown. They also carry a derived `is_runnable` boolean (`true` ⇔ `pending_signatures` is empty), and the markdown adds an explicit plain-English runnability verdict on those surfaces: a complete bundle/library states it is runnable, while one with pending signatures states it is NOT yet runnable immediately above the verbatim "Pending signatures" heading. These whole-bundle/whole-library surfaces are **strict by default**: they exit non-zero on `not is_runnable` unless `--allow-signatures` is passed (which both tolerates the placeholders for the exit code and mock-runs the signature pipes in the sweep). Single-pipe surfaces make no library-wide runnability claim and never gate the exit code on it: bare `validate pipe <code>` omits the key and emits no verdict, while `validate bundle`/`validate method` with `--pipe` still surface the library-wide `pending_signatures` for information but do not gate on it (the requested slice can be fully implemented even when unrelated placeholders remain elsewhere).- `fix`: `# Fix applied - bundle is valid` after one or more fixes, `# Bundle already valid` when the loop had nothing to apply, or `# Bundle valid but not runnable` when pending `PipeSignature` placeholders remain. The JSON success envelope carries `is_valid`, `bundle_path`, `iterations`, `fixes_applied`, `files_written`, `remaining_errors`, and, on successful validation, `pending_signatures` plus `is_runnable`. Whole-bundle fix runs are strict by default: they emit the success envelope but exit 1 on `not is_runnable` unless `--allow-signatures` is passed. A still-invalid verdict exits 1 via `FixBundleError` on stderr with the same structured result fields plus any `bail_reason`.+- `migrate`: `# Configuration migration`, the mode (applied / dry run) and verdict, the directories walked, a files-walked/changed/written count, then one `## <path>` section per file that is not clean listing the entries applied, the entries blocked with their reason and guidance, and any unexplained paths. Paths, operation kinds and ledger-supplied values only — no value read from the user's file is ever rendered, in either format.- `init`: `# Pipelex initialized` with target directory, enabled backends, and routing profile.- error path: `# Error: <error_type>`, the message, the hint as a `> 💡` callout, and a `## Details` section. `error_source` (internal stack frames) is omitted from markdown — it remains in the JSON envelope (`--error-format json`) for programmatic consumers.@@ -95,12 +96,14 @@ commands/| `pipe` | Converts a JSON pipe spec into raw TOML (stdout). A spec with a `type` (via `--type` or a `type` key) is that concrete pipe; a **typeless** spec (no `type`) is a signature and renders a `[pipe.x]` section with no type line. An explicit `type = "PipeSignature"` is rejected with a migration error — `PipeSignature` is not a type. || `models` | Lists available model presets, aliases, and waterfalls. `--format markdown\|json` (success, default: markdown) + `--error-format markdown\|json` (errors, defaults to `--format`'s value) || `check-model` | Validates a model reference and suggests alternatives if invalid. `--type`/`-t` for model category, `--format markdown\|json` (success, default: markdown) + `--error-format markdown\|json` (errors, defaults to `--format`'s value) |-| `doctor` | Checks config, credentials, models health. `--global`/`-g` for global dir. `--format markdown\|json` (success, default: markdown) + `--error-format markdown\|json` (errors, defaults to `--format`'s value) |+| `migrate` | Migrates this machine's configuration files (global `~/.pipelex/` and project `.pipelex/`, non-recursively) to the current schema by replaying each surface's ledger. Does **not** boot — it is the command for a configuration that cannot load. Writes only with `--yes`; `--dry-run` plans and is the default; the two together are refused with exit 2. `--format markdown\|json` (success, default: markdown) + `--error-format markdown\|json`. The verdict is the structured `needs_attention`, not the exit code. |+| `doctor` | Checks config, credentials, models health. `--global`/`-g` for global dir. `--format markdown\|json` (success, default: markdown) + `--error-format markdown\|json` (errors, defaults to `--format`'s value). The telemetry check carries a `finding` (`healthy`/`not_found`/`unparseable`/`out_of_date`/`invalid`) beside its `healthy` and `message`; branch on that, and note `out_of_date` is `pipelex migrate`'s and never `pipelex init telemetry`'s — a reset would discard the settings the migration keeps. `checks.pending_migrations` is the machine-wide half: a `finding` (`up_to_date`/`pending`/`needs_attention`/`unavailable`) plus `migratable_files` and `attention_files`. It is the only channel a machine has for a pending migration — a tolerated boot warns a person and this CLI silences logging — and it is **not** scoped by `--global`, because it answers for `pipelex migrate`, which walks both directories. |## Key Patterns- **Output contract**: Commands with `--format` emit success via `agent_success_formatted(result, markdown_renderer, output_format)` — JSON or a markdown renderer per the explicit `output_format` argument. `agent_error(message, error_type, cause)` dispatches JSON or markdown by reading the `_agent_cli_error_format` ContextVar, which each `--format`-aware command sets via `set_agent_cli_error_format(error_format or output_format)` at function entry. Exceptions that print directly to stdout: `fmt`/`lint` (plxt passthrough), `concept`/`pipe` (raw TOML). Markdown renderers: `format_run_markdown` (`run/_output_helpers.py`), `format_validate_markdown` (lifted to the non-CLI `pipelex/pipeline/validation_render.py` so `pipelex-api` can import it without CLI/Typer deps), `format_fix_markdown` (non-CLI `pipelex/pipeline/fixes/fix_render.py`), `_format_init_markdown` (`init_cmd.py`).- **Error classification**: Each error type maps to a domain (`input`, `config`, `runtime`), a hint string, and a `retryable` flag. See `AGENT_ERROR_HINTS` dict in `agent_output.py`. The `error_domain` also drives the HTTP-status mapping for downstream APIs — see `error_domain_to_http_status()` in `pipelex/base_exceptions.py`.+- **The `migration` field on a configuration error**: A `PipelexConfigError` carries one when a scan of this machine's configuration directories found something — and only then, so its presence is the whole test. It says the configuration is *old* rather than *wrong*, and the loop it opens is `pipelex-agent migrate --dry-run --format json`, show the user, then `--yes`. Its `plans` are the same shape that command emits under its own `plans` key. `error_domain` stays `config`: a domain the hook specification does not know routes to BLOCK, which would stop an agent instead of telling it what to run. See `docs/migration-ledger.md` → "Reporting a stale configuration on a validation error".- **Init**: All commands that need Pipelex use `make_pipelex_for_agent_cli(library_dirs)`. It catches init errors and routes them through `agent_error()`.- **Async core**: Run and validate are async — commands use `asyncio.run()`.- **File convention**: Generated outputs go to `mthds-wip/` with incremental naming (`pipeline_01/`, `pipeline_02/`).diff --git a/pipelex/cli/agent_cli/_agent_cli.py b/pipelex/cli/agent_cli/_agent_cli.pyindex bbbb33d75..60b5e9599 100644--- a/pipelex/cli/agent_cli/_agent_cli.py+++ b/pipelex/cli/agent_cli/_agent_cli.py@@ -20,6 +20,7 @@ from pipelex.cli.agent_cli.commands.fmt_cmd import fmt_cmdfrom pipelex.cli.agent_cli.commands.init_cmd import agent_init_cmdfrom pipelex.cli.agent_cli.commands.inputs.app import inputs_appfrom pipelex.cli.agent_cli.commands.lint_cmd import lint_cmd+from pipelex.cli.agent_cli.commands.migrate_cmd import agent_migrate_cmdfrom pipelex.cli.agent_cli.commands.models_cmd import agent_models_cmdfrom pipelex.cli.agent_cli.commands.pipe_cmd import pipe_cmdfrom pipelex.cli.agent_cli.commands.run.app import run_app@@ -47,6 +48,7 @@ class PipelexAgentCLI(TyperGroup):"models","check-model","accept-gateway-terms",+ "migrate","doctor",]@@ -147,4 +149,5 @@ app.command(name="pipe", help="Structure a pipe from JSON spec and output TOML")app.command(name="models", help="List available model presets, aliases, and waterfalls")(agent_models_cmd)app.command(name="check-model", help="Check if a model reference is valid and suggest alternatives")(agent_check_model_cmd)app.command(name="accept-gateway-terms", help="Accept Pipelex Gateway terms and mark inference setup complete")(agent_accept_gateway_terms_cmd)+app.command(name="migrate", help="Migrate this machine's Pipelex configuration files to the current schema")(agent_migrate_cmd)app.command(name="doctor", help="Check Pipelex configuration health and auto-fix issues")(agent_doctor_cmd)diff --git a/pipelex/cli/agent_cli/commands/agent_output.py b/pipelex/cli/agent_cli/commands/agent_output.pyindex 4bc25ac1b..b73cdb473 100644--- a/pipelex/cli/agent_cli/commands/agent_output.py+++ b/pipelex/cli/agent_cli/commands/agent_output.py@@ -119,7 +119,14 @@ AGENT_ERROR_HINTS: dict[str, str] = {# Interpreter errors"MthdsParserError": "Check MTHDS file TOML syntax and ensure all referenced concepts and pipes are defined",# Configuration/initialization errors- "TelemetryConfigValidationError": "Run 'pipelex init telemetry' to create a valid telemetry configuration",+ # Two readings, and the `migration` field on this same payload is what separates them: present+ # means the file is out of date, absent means it is wrong. Never `pipelex init telemetry` on+ # the first — that writes a fresh file over settings a migration would have kept.+ "TelemetryConfigValidationError": (+ "If this payload carries a 'migration' field, the configuration is out of date: run "+ "'pipelex-agent migrate --dry-run --format json', then 'pipelex-agent migrate --yes'. "+ "Otherwise correct the telemetry.toml settings named in the message."+ ),"GatewayTermsNotAcceptedError": "Run 'pipelex init config' to accept gateway terms, or disable pipelex_gateway in backends.toml","GatewayApiKeyMissingError": "Set the PIPELEX_GATEWAY_API_KEY environment variable, or disable pipelex_gateway in backends.toml","GatewayDoNotTrackConflictError": "Unset the DO_NOT_TRACK environment variable, or disable pipelex_gateway in backends.toml",@@ -188,7 +195,6 @@ AGENT_ERROR_DOMAINS: dict[str, str] = {"PipeOperatorModelChoiceError": "config","PipeOperatorModelAvailabilityError": "config","ModelDeckPresetValidatonError": "config",- "TelemetryConfigValidationError": "config","BinaryNotFoundError": "config","GatewayUnknownModelError": "config","InitConfigError": "config",@@ -233,6 +239,12 @@ def _assemble_error_payload(message: str, *, error_type: str, cause: BaseExcepti``model`` / ``provider`` from a ``PipelexError`` cause's ``to_error_report()``first, falling back to the lookup dicts for error types that cannotself-describe. ``extra`` is merged last and overrides everything.++ ``migration`` comes from the report too, and it is the one field here that an agent is meant+ to *act* on rather than classify by: its presence says the configuration is stale rather than+ wrong, and the loop it opens is ``pipelex-agent migrate --dry-run --format json`` followed by+ ``--yes``. Emitted only when the report carries one, so branching on its presence is the+ whole test."""error_json: dict[str, Any] = {"error": True,@@ -257,6 +269,8 @@ def _assemble_error_payload(message: str, *, error_type: str, cause: BaseExceptireport_extras["model"] = report.modelif report.provider:report_extras["provider"] = report.provider+ if report.migration is not None:+ report_extras["migration"] = report.migration.model_dump(mode="json")# hint: report-first, fallback to lookup dicthint = report_hint or AGENT_ERROR_HINTS.get(error_type)diff --git a/pipelex/cli/agent_cli/commands/check_model_cmd.py b/pipelex/cli/agent_cli/commands/check_model_cmd.pyindex 1d5a66d03..f4bdc5f37 100644--- a/pipelex/cli/agent_cli/commands/check_model_cmd.py+++ b/pipelex/cli/agent_cli/commands/check_model_cmd.py@@ -106,7 +106,9 @@ def agent_check_model_cmd(agent_success(result)case CliOutputFormat.MARKDOWN:print(_format_check_markdown(result))- except SystemExit:+ except (SystemExit, typer.Exit):+ # `typer.Exit` is a `RuntimeError`, not a `SystemExit`: without it here, an error+ # `agent_error` has already reported is caught again below and reported twice.raiseexcept ModelReferenceParseError as exc:agent_error(f"Invalid model reference: {exc}", error_type="ArgumentError", cause=exc)diff --git a/pipelex/cli/agent_cli/commands/doctor_cmd.py b/pipelex/cli/agent_cli/commands/doctor_cmd.pyindex d1ee397d4..8d75a1630 100644--- a/pipelex/cli/agent_cli/commands/doctor_cmd.py+++ b/pipelex/cli/agent_cli/commands/doctor_cmd.py@@ -14,13 +14,19 @@ from pipelex.cli.agent_cli.commands.agent_output import CliOutputFormat, agent_efrom pipelex.cli.commands.doctor_cmd import (BackendFileReport,ConfigLocationInfo,+ PendingMigrationsCheck,+ PendingMigrationsFinding,+ TelemetryConfigCheck,+ TelemetryConfigFinding,check_backend_credentials,check_config_files,check_models,+ check_pending_migrations,check_telemetry_config,gather_config_location,setup_doctor_runtime,)+from pipelex.core.validation import MIGRATE_COMMANDfrom pipelex.system.configuration.config_loader import config_manager@@ -29,6 +35,61 @@ def _status_icon(*, healthy: bool) -> str:return "\u2705" if healthy else "\u26a0\ufe0f"+def _pending_migrations_actions(*, check: PendingMigrationsCheck) -> list[str]:+ """What to tell an agent about the pending-migrations row — none, one move, or two.++ A machine consumer never learns of a pending migration from a boot: a stale configuration the+ ledger can explain boots with a warning, and the agent CLI silences its own logging before one+ can be emitted. So for an agent this row is not a convenience, it is the only channel.++ Two actions rather than one when a run both migrates some files and leaves others behind,+ which is the ordinary shape on a machine that has drifted: the command is worth running *and*+ something is still owed afterwards, and an agent that only heard the first would stop early.+ """+ actions: list[str] = []+ match check.finding:+ case PendingMigrationsFinding.UP_TO_DATE:+ return actions+ case PendingMigrationsFinding.PENDING:+ migratable = ", ".join(check.migratable_files)+ actions.append(f"Run '{MIGRATE_COMMAND}' to bring these configuration files up to date (it keeps the settings in them): {migratable}")+ case PendingMigrationsFinding.NEEDS_ATTENTION:+ pass+ case PendingMigrationsFinding.UNAVAILABLE:+ actions.append(f"{check.message}. Run '{MIGRATE_COMMAND} --dry-run' to check by hand.")+ if check.attention_files:+ attention = ", ".join(check.attention_files)+ actions.append(+ f"Run '{MIGRATE_COMMAND} --dry-run' to see what these configuration files carry that the migration will not do on its own: {attention}"+ )+ return actions+++def _telemetry_action(*, check: TelemetryConfigCheck, config_location: ConfigLocationInfo, migration_already_recommended: bool) -> str | None:+ """What to tell an agent to do about the telemetry finding, or nothing when it is healthy.++ One remedy per finding, because they are genuinely different moves: a missing file is written,+ an out-of-date one is migrated with every setting kept, and a file this build cannot read is+ a person's to edit. The single `pipelex init telemetry` this replaced said the same thing to+ all four, and on the out-of-date one it was destructive advice.++ `migration_already_recommended` is the one case where a finding produces no action: the+ pending-migrations row names the same command and lists this file among the ones it covers, so+ repeating it here would be the same instruction twice at two different scopes.+ """+ match check.finding:+ case TelemetryConfigFinding.HEALTHY:+ return None+ case TelemetryConfigFinding.NOT_FOUND:+ return "Run 'pipelex init telemetry' to write a telemetry configuration"+ case TelemetryConfigFinding.OUT_OF_DATE:+ if migration_already_recommended:+ return None+ return f"Run '{MIGRATE_COMMAND}' to bring telemetry.toml up to date (it keeps the settings in it)"+ case TelemetryConfigFinding.UNPARSEABLE | TelemetryConfigFinding.INVALID:+ return f"Fix {config_location.config_dir}/telemetry.toml: {check.message}"++def _format_doctor_markdown(result: dict[str, Any]) -> str:"""Format the doctor result dict as markdown with bullet points."""all_healthy: bool = result["all_healthy"]@@ -50,6 +111,15 @@ def _format_doctor_markdown(result: dict[str, Any]) -> str:lines.append(f"\n## Config Files \u2014 {_status_icon(healthy=config_check['healthy'])}\n")lines.append(config_check["message"])+ # Configuration Migrations+ migrations_check = checks["pending_migrations"]+ lines.append(f"\n## Configuration Migrations \u2014 {_status_icon(healthy=migrations_check['healthy'])}\n")+ lines.append(migrations_check["message"])+ for file_path in migrations_check["migratable_files"]:+ lines.append(f"- `{file_path}`: out of date")+ for file_path in migrations_check["attention_files"]:+ lines.append(f"- `{file_path}`: needs a look")+# Telemetrytelemetry_check = checks["telemetry"]lines.append(f"\n## Telemetry \u2014 {_status_icon(healthy=telemetry_check['healthy'])}\n")@@ -153,7 +223,10 @@ def agent_doctor_cmd(# a silent installer on a fresh machine. (The --global path skips materialization# — see load_config.)config_healthy, config_missing_count, config_message = check_config_files(config_dir=config_dir)- telemetry_healthy, telemetry_message = check_telemetry_config(config_dir=config_dir)+ # No config_dir: this row reports on `pipelex migrate`, which has no --global and walks+ # both configuration directories. See `check_pending_migrations`.+ pending_migrations_check = check_pending_migrations()+ telemetry_check = check_telemetry_config(config_dir=config_dir)backends_healthy, backend_credential_reports, backends_message = check_backend_credentials(config_dir=config_dir)# check_models requires the hub + log.configure produced by setup_doctor_runtime.@@ -200,7 +273,7 @@ def agent_doctor_cmd(# installed a hub or configured log — the helper guards both internally).apply_agent_cli_output_discipline()- all_healthy = config_healthy and telemetry_healthy and backends_healthy and models_healthy+ all_healthy = config_healthy and pending_migrations_check.is_healthy and telemetry_check.is_healthy and backends_healthy and models_healthy# Build backend credential detailsbackends_list: list[dict[str, Any]] = []@@ -235,8 +308,14 @@ def agent_doctor_cmd(recommended_actions.append("Run 'pipelex init config' to install missing configuration files")if not config_healthy and config_missing_count == 0:recommended_actions.append(f"Fix validation errors in {config_location.config_dir}/pipelex.toml or run 'pipelex init config'")- if not telemetry_healthy:- recommended_actions.append(f"Run 'pipelex init telemetry' to fix telemetry: {telemetry_message}")+ recommended_actions.extend(_pending_migrations_actions(check=pending_migrations_check))+ recommended_telemetry_action = _telemetry_action(+ check=telemetry_check,+ config_location=config_location,+ migration_already_recommended=pending_migrations_check.finding.is_repaired_by_migrating,+ )+ if recommended_telemetry_action is not None:+ recommended_actions.append(recommended_telemetry_action)if not backends_healthy:for report in backend_credential_reports.values():if report.missing_vars:@@ -264,9 +343,17 @@ def agent_doctor_cmd("message": config_message,"missing_count": config_missing_count,},+ "pending_migrations": {+ "healthy": pending_migrations_check.is_healthy,+ "finding": str(pending_migrations_check.finding),+ "message": pending_migrations_check.message,+ "migratable_files": pending_migrations_check.migratable_files,+ "attention_files": pending_migrations_check.attention_files,+ },"telemetry": {- "healthy": telemetry_healthy,- "message": telemetry_message,+ "healthy": telemetry_check.is_healthy,+ "finding": str(telemetry_check.finding),+ "message": telemetry_check.message,},"backend_credentials": {"healthy": backends_healthy,diff --git a/pipelex/cli/agent_cli/commands/migrate_cmd.py b/pipelex/cli/agent_cli/commands/migrate_cmd.pynew file mode 100644index 000000000..ce733b78d--- /dev/null+++ b/pipelex/cli/agent_cli/commands/migrate_cmd.py@@ -0,0 +1,154 @@+"""``pipelex-agent migrate`` — the machine-facing half of the configuration migration.++Same engine and same walk as ``pipelex migrate``; what differs is who reads the answer and how+the run is authorized.++**Nothing is written unless ``--yes`` is passed.** The human command can ask; this one cannot, so+the default is the safe half of that question. The loop an agent runs is therefore+``migrate --dry-run`` to see the plan, then ``migrate --yes`` to apply it — and passing both is+refused rather than resolved, because an agent that asks for one and the other has a bug that a+silent winner would hide.++**The structured fields are the contract.** ``needs_attention`` is the verdict — *this run left+something a person has to decide* — and it is deliberately not "did anything get written": a run+that migrated every file it found has succeeded, and so has a dry run that found nothing blocked.+The exit code and the Markdown rendering are presentation and may change without the contract+changing.++**This command must run when nothing else does**, exactly as its human sibling: no boot, no model+deck, no credentials, no network. A broken configuration is the reason to reach for it.++See ``docs/migration-ledger.md``.+"""++from typing import Annotated, Any++import typer++from pipelex.cli.agent_cli.commands.agent_cli_factory import silence_logging_for_agent_cli+from pipelex.cli.agent_cli.commands.agent_output import (+ CliOutputFormat,+ agent_error,+ agent_success_formatted,+ set_agent_cli_error_format,+)+from pipelex.cli.commands.migrate_cmd import describe_op+from pipelex.migration.plan import MigrationPlan, MigrationReport+from pipelex.migration.run import config_directories_to_migrate, migrate_config_directories+++def agent_migrate_cmd(+ dry_run: Annotated[+ bool,+ typer.Option("--dry-run", help="Report the plan and write nothing. This is already the default; pass it to say so explicitly."),+ ] = False,+ yes: Annotated[+ bool,+ typer.Option("--yes", "-y", help="Apply the plan. Without it nothing is written."),+ ] = False,+ output_format: Annotated[+ CliOutputFormat,+ typer.Option("--format", help="Success output format: markdown (default) or json (structured)"),+ ] = CliOutputFormat.MARKDOWN,+ error_format: Annotated[+ CliOutputFormat | None,+ typer.Option("--error-format", help="Error output format (defaults to --format value): markdown or json"),+ ] = None,+) -> None:+ """Migrate this machine's Pipelex configuration files to the current schema.++ Walks the global ``~/.pipelex/`` and the project ``.pipelex/``, replays each configuration+ surface's ledger over every file it claims, and — with ``--yes`` — rewrites what changed,+ backing up each original beside itself.+ """+ set_agent_cli_error_format(error_format or output_format)+ silence_logging_for_agent_cli()+ if dry_run and yes:+ agent_error(+ "--dry-run and --yes contradict each other: one refuses to write, the other authorizes it",+ error_type="ArgumentError",+ exit_code=2,+ )++ config_dirs = config_directories_to_migrate()+ report = migrate_config_directories(config_dirs=config_dirs, dry_run=not yes)+ result = _result_payload(report=report, config_dirs=[str(directory) for directory in config_dirs], applied=yes)+ agent_success_formatted(result, markdown_renderer=_render_markdown, output_format=output_format)+ if report.needs_attention:+ raise typer.Exit(1)+++def _result_payload(*, report: MigrationReport, config_dirs: list[str], applied: bool) -> dict[str, Any]:+ """The structured answer: the verdict, the arithmetic, and every plan the run produced.++ Every file the walk visited is here, clean ones included. A report *is* the set of files this+ run looked at, and an agent deciding whether its configuration directory was even reached+ needs the ones that had nothing to say as much as the ones that did.+ """+ return {+ "applied": applied,+ "needs_attention": report.needs_attention,+ "is_clean": report.is_clean,+ "config_dirs": config_dirs,+ "summary": {+ "files_walked": len(report.plans),+ "files_changed": len(report.changed_plans),+ "files_written": len(report.written_plans),+ "files_blocked": len([plan for plan in report.plans if plan.blocked_reason is not None]),+ "entries_blocked": sum(len(plan.blocked) for plan in report.plans),+ "unexplained_paths": sum(len(plan.unexplained) for plan in report.plans),+ },+ "plans": [plan.model_dump(mode="json") for plan in report.plans],+ }+++def _render_markdown(result: dict[str, Any]) -> str:+ """The same answer as prose, for an agent reading it rather than parsing it."""+ applied: bool = result["applied"]+ summary: dict[str, Any] = result["summary"]+ config_dirs: list[str] = result["config_dirs"]++ lines: list[str] = ["# Configuration migration", ""]+ if not config_dirs:+ lines += ["No configuration directory was found on this machine — there is nothing to migrate.", ""]+ return "\n".join(lines)++ verdict = "needs attention" if result["needs_attention"] else "nothing left to decide"+ mode = "applied" if applied else "dry run — nothing was written"+ lines += [f"**Mode:** {mode}", f"**Verdict:** {verdict}", "", "**Directories walked:**"]+ lines += [f"- `{directory}`" for directory in config_dirs]+ lines += ["", f"**Files:** {summary['files_walked']} walked, {summary['files_changed']} changed, {summary['files_written']} written."]++ if result["is_clean"]:+ lines += ["", "Every configuration file walked is at the current schema."]+ return "\n".join(lines)++ for plan_dict in result["plans"]:+ plan = MigrationPlan.model_validate(plan_dict)+ if plan.is_clean:+ continue+ lines += ["", f"## `{plan.file_path}`", "", f"Surface: `{plan.surface_id}`", ""]+ lines += _file_lines(plan=plan, applied=applied)+ return "\n".join(lines)+++def _file_lines(*, plan: MigrationPlan, applied: bool) -> list[str]:+ if plan.blocked_reason is not None:+ return [f"- **This file could not be processed** (`{plan.blocked_reason}`): {plan.blocked_detail}"]+ lines: list[str] = []+ verb = "Applied" if applied else "Would apply"+ for step in plan.steps:+ lines.append(f"- **{verb} `{step.entry_id}` (schema v{step.to_schema_version}):** {step.title} — {step.description}")+ lines += [f" - {describe_op(op=op)}" for op in step.applied_ops]+ for blocked in plan.blocked:+ lines.append(f"- **Blocked `{blocked.entry_id}` (schema v{blocked.to_schema_version}, `{blocked.reason}`):** {blocked.detail}")+ lines += [f" - already applied: {describe_op(op=op)}" for op in blocked.applied_ops]+ if blocked.narrowed_paths:+ lines.append(f" - check by hand: {', '.join(f'`{path}`' for path in blocked.narrowed_paths)}")+ if blocked.guidance:+ lines += [f" - {line}" for line in blocked.guidance.splitlines() if line.strip()]+ for unexplained in plan.unexplained:+ lines.append(f"- **Unexplained path `{unexplained.path}`:** {unexplained.note}")+ if plan.backup_path is not None:+ lines.append(f"- Backup of the original: `{plan.backup_path}`")+ return linesdiff --git a/pipelex/cli/agent_cli/commands/models_cmd.py b/pipelex/cli/agent_cli/commands/models_cmd.pyindex 90d30128f..10ad29165 100644--- a/pipelex/cli/agent_cli/commands/models_cmd.py+++ b/pipelex/cli/agent_cli/commands/models_cmd.py@@ -44,8 +44,11 @@ def agent_models_cmd(agent_success({"success": True, **result})case CliOutputFormat.MARKDOWN:print(format_models_markdown(result))- except SystemExit:- # agent_error already handled and called sys.exit+ except (SystemExit, typer.Exit):+ # `agent_error` has already emitted the envelope and is on its way out. `typer.Exit` is a+ # `RuntimeError`, *not* a `SystemExit`, so leaving it out of this arm dropped it into the+ # broad catch below and printed a second envelope after the first — stderr then held two+ # JSON documents and a machine consumer parsing it got "Extra data".raiseexcept Exception as exc: # noqa: BLE001# Agent CLI command boundary: agent_error() (NoReturn) converts any unexpected failure into the structured error payload.diff --git a/pipelex/cli/commands/doctor_cmd.py b/pipelex/cli/commands/doctor_cmd.pyindex 3cc70c284..6484b5da5 100644--- a/pipelex/cli/commands/doctor_cmd.py+++ b/pipelex/cli/commands/doctor_cmd.py@@ -5,10 +5,12 @@ from __future__ import annotationsimport contextlibimport ioimport sys+from enum import StrEnumfrom pathlib import Pathfrom typing import TYPE_CHECKING, Any-from pydantic import BaseModel, ValidationError+from pydantic import BaseModel, ConfigDict, Field, ValidationError+from rich.markup import escapefrom rich.panel import Panelfrom rich.prompt import Confirmfrom rich.text import Text@@ -18,6 +20,7 @@ from pipelex.base_exceptions import PipelexConfigErrorfrom pipelex.cli.commands.init.command import init_cmdfrom pipelex.cli.commands.init.config_files import init_configfrom pipelex.cli.commands.init.ui.types import InitFocus+from pipelex.cli.commands.migrate_cmd import apply_pending_migrationsfrom pipelex.cli.commands.update_cmd import update_cmdfrom pipelex.cli.exceptions import PipelexCLIErrorfrom pipelex.cogt.exceptions import (@@ -37,13 +40,15 @@ from pipelex.cogt.model_backends.gateway_config import GatewayConfigfrom pipelex.cogt.models.deck_manifest import DeckFileStatus, DeckSyncReport, compute_deck_sync_report, status_rich_labelfrom pipelex.cogt.models.model_manager import ModelManagerfrom pipelex.config import get_config-from pipelex.core.validation import report_validation_error+from pipelex.core.validation import MIGRATE_COMMAND, raise_config_setup_error, report_validation_errorfrom pipelex.kit.paths import get_kit_configs_dir+from pipelex.migration.exceptions import MigrationError+from pipelex.migration.run import config_directories_to_migrate, migrate_config_directories, scan_config_surfacefrom pipelex.runtime_hub import RuntimeHub, get_console, set_runtime_hub-from pipelex.system.configuration.config_loader import config_manager+from pipelex.system.configuration.config_loader import CONFIG_REFUSED, config_manager, pydantic_error_behind+from pipelex.system.configuration.config_surface import PIPELEX_CONFIG_SURFACE_ID, TELEMETRY_CONFIG_SURFACE_ID, strip_reserved_metafrom pipelex.system.configuration.configs import PipelexConfigfrom pipelex.system.environment import get_optional_env-from pipelex.system.exceptions import ConfigValidationErrorfrom pipelex.system.pipelex_service.exceptions import (RemoteConfigUnavailableError,RemoteConfigValidationError,@@ -136,20 +141,20 @@ def check_config_files(*, config_dir: Path | None = None) -> tuple[bool, int, stwith contextlib.redirect_stderr(io.StringIO()), contextlib.redirect_stdout(io.StringIO()):config = config_manager.load_config(config_dir=config_dir)PipelexConfig.model_validate(config)- except ValidationError as validation_error:- validation_error_msg = report_validation_error(category="config", validation_error=validation_error)- msg = f"Configuration validation failed:\n{validation_error_msg}"- return False, 0, msg- except ConfigValidationError as exc:- # ConfigRoot.__init__ wraps pydantic.ValidationError into ConfigValidationError;- # recover the original via __cause__ so we still emit the migration-aware report.- underlying = exc.__cause__- if isinstance(underlying, ValidationError):- validation_error_msg = report_validation_error(category="config", validation_error=underlying)- msg = f"Configuration validation failed:\n{validation_error_msg}"- else:- msg = f"Configuration validation failed: {exc.message}"- return False, 0, msg+ except CONFIG_REFUSED as config_error:+ # Both shapes a refusal takes: `ConfigRoot.__init__` translates pydantic's error into+ # `ConfigValidationError`, and `pydantic_error_behind` reaches back through it for the+ # field-level analysis. A refusal carrying no pydantic error at all keeps its own+ # message, which is then the whole account.+ validation_error = pydantic_error_behind(config_error=config_error)+ if validation_error is None:+ return False, 0, f"Configuration validation failed: {config_error}"+ report = report_validation_error(+ validation_error=validation_error,+ surface_id=PIPELEX_CONFIG_SURFACE_ID,+ config_dirs=[config_dir] if config_dir is not None else None,+ )+ return False, 0, f"Configuration validation failed:\n{report.message}"except (TomlError, OSError) as exc:return False, 0, f"Error loading pipelex.toml: {exc}"@@ -159,33 +164,233 @@ def check_config_files(*, config_dir: Path | None = None) -> tuple[bool, int, streturn False, missing_count, f"{missing_count} configuration file(s) missing"-def check_telemetry_config(*, config_dir: Path | None = None) -> tuple[bool, str]:- """Check if telemetry configuration is valid.+class TelemetryConfigFinding(StrEnum):+ """What the telemetry probe found — the thing every caller branches on.++ A verdict rather than a sentence, and the reason is a bug this replaced: the fix machinery+ used to decide what it could repair by searching the *message* for `"format has changed"`,+ so rewording the row would have switched the whole `--fix` path off in silence.+ """++ HEALTHY = "healthy"+ NOT_FOUND = "not_found"+ UNPARSEABLE = "unparseable"+ OUT_OF_DATE = "out_of_date"+ INVALID = "invalid"++ @property+ def is_healthy(self) -> bool:+ return self is TelemetryConfigFinding.HEALTHY++ @property+ def is_out_of_date(self) -> bool:+ """Whether `pipelex migrate` is the remedy — the ledger explains this file."""+ return self is TelemetryConfigFinding.OUT_OF_DATE++ @property+ def is_repaired_by_initializing(self) -> bool:+ """Whether writing a fresh file is a repair here rather than a loss.++ True for exactly one finding. There is nothing in a file that is not there to preserve,+ while every other unhealthy state has the user's own settings in it — which is why an+ out-of-date file gets `pipelex migrate` and a broken one gets a person, not a reset.+ """+ return self is TelemetryConfigFinding.NOT_FOUND+++class TelemetryConfigCheck(BaseModel):+ """The telemetry row of the health report: what was found, and how to say it."""++ model_config = ConfigDict(frozen=True)++ finding: TelemetryConfigFinding+ message: str++ @property+ def is_healthy(self) -> bool:+ return self.finding.is_healthy+++def check_telemetry_config(*, config_dir: Path | None = None) -> TelemetryConfigCheck:+ """Check if telemetry configuration is valid, and if not, whether it is out of date or wrong.++ The second half is what keeps the remedy honest. A telemetry file the ledger can carry+ forward is migrated by `pipelex migrate`, which keeps every setting in it; telling that user+ to re-initialize would throw away their PostHog keys and their exporters to fix a file that+ was never broken.Args:config_dir: Explicit config directory override (e.g. for --global).If None, uses layered resolution (project .pipelex/ → global ~/.pipelex/).Returns:- Tuple of (is_healthy, message)+ The finding and the message describing it."""telemetry_config_path = config_manager.resolve_config_file(TELEMETRY_CONFIG_FILE_NAME, config_dir=config_dir)try:toml_doc = load_toml_from_path(telemetry_config_path)except FileNotFoundError:- return False, "Telemetry configuration file not found"+ return TelemetryConfigCheck(finding=TelemetryConfigFinding.NOT_FOUND, message="Telemetry configuration file not found")except TomlError as exc:- return False, f"TOML syntax error: {exc}"+ return TelemetryConfigCheck(finding=TelemetryConfigFinding.UNPARSEABLE, message=f"TOML syntax error: {exc}")++ # Exactly as the loader reads it: `[meta]` is reserved for the migration machinery and is+ # stripped before validation, so a file carrying one must not be reported as invalid here+ # while booting perfectly well.+ strip_reserved_meta(config_dict=toml_doc)try:telemetry_config = TelemetryConfig.model_validate(toml_doc)- return True, f"Telemetry configured (mode: {telemetry_config.custom_posthog.mode})"- except ValidationError:- # Check if this looks like the old config format (has telemetry_mode at root level)- if "custom_posthog" not in toml_doc and ("telemetry_mode" in toml_doc or "project_api_key" in toml_doc):- return False, "Config format has changed - run 'pipelex init telemetry' to update"- return False, "Invalid configuration - run 'pipelex init telemetry' to fix"+ except ValidationError as exc:+ return _telemetry_is_out_of_date_or_wrong(validation_error=exc, config_path=telemetry_config_path)+ return TelemetryConfigCheck(+ finding=TelemetryConfigFinding.HEALTHY,+ message=f"Telemetry configured (mode: {telemetry_config.custom_posthog.mode})",+ )+++def _telemetry_is_out_of_date_or_wrong(*, validation_error: ValidationError, config_path: Path) -> TelemetryConfigCheck:+ """Ask the ledger which of the two this is, and say so.++ The scan is aimed at the directory the probe actually read, not at the directories a real+ migration walks, and the answer is read off the plan for the file the probe read: this row+ reports on one file, and answering it with a finding about the other tier — or about the+ `telemetry_*.toml` tier file beside it in the same directory — would name a file the reader+ is not looking at, and send them to a migration that leaves the error on this one behind.++ **A failure inside the scan never takes the health report down with it.** The same rule the+ boot retry and `report_validation_error` follow, and it costs more here than anywhere else: an+ exception escaping this probe reaches the doctor's own outer handler, which prints one line+ and exits — so a packaging problem of ours would replace every row the user came for. Falling+ back to `INVALID` under-reports at worst, and the field-level analysis it carries is what the+ reader needs either way. The catch stays narrow, so a bug in our applier surfaces as itself.+ """+ try:+ report = scan_config_surface(surface_id=TELEMETRY_CONFIG_SURFACE_ID, config_dirs=[config_path.parent])+ except (MigrationError, OSError):+ report = None+ own_plan = next((plan for plan in report.plans if plan.file_path == config_path), None) if report is not None else None+ if own_plan is not None and not own_plan.is_clean:+ return TelemetryConfigCheck(+ finding=TelemetryConfigFinding.OUT_OF_DATE,+ message=f"Configuration is out of date — run '{MIGRATE_COMMAND}' to bring it up to date",+ )+ return TelemetryConfigCheck(+ finding=TelemetryConfigFinding.INVALID,+ message=f"Invalid configuration:\n{report_validation_error(validation_error=validation_error).message}",+ )+++class PendingMigrationsFinding(StrEnum):+ """What a dry run of `pipelex migrate` found on this machine.++ One member per *next move*, which is the only thing they disagree about: nothing to do, run the+ command, read the notes, or we could not look. The last one is deliberately not folded into the+ others — a packaging problem of ours must not be reported as a finding about the user's files.+ """++ UP_TO_DATE = "up_to_date"+ PENDING = "pending"+ NEEDS_ATTENTION = "needs_attention"+ UNAVAILABLE = "unavailable"++ @property+ def is_healthy(self) -> bool:+ return self is PendingMigrationsFinding.UP_TO_DATE++ @property+ def is_repaired_by_migrating(self) -> bool:+ """Whether `pipelex migrate` has something to write here.++ True for `PENDING` alone, and it stays true when that same run also leaves something for a+ person: the command migrates the files it can either way, and the notes are read after.+ """+ return self is PendingMigrationsFinding.PENDING++ @property+ def is_uncheckable(self) -> bool:+ """Whether the question went unanswered, which is neither a yes nor a no."""+ return self is PendingMigrationsFinding.UNAVAILABLE+++class PendingMigrationsCheck(BaseModel):+ """The configuration-migration row: what a dry run found, and which files it was about."""++ model_config = ConfigDict(frozen=True)++ finding: PendingMigrationsFinding+ message: str++ migratable_files: list[str] = Field(default_factory=list[str])+ """The files `pipelex migrate` would rewrite, each backed up first."""++ attention_files: list[str] = Field(default_factory=list[str])+ """The files carrying something the command will not do on its own.++ A file can be on both lists: an entry that conflicts partway through is blocked, and the+ operations of it that applied before the conflict are still written."""++ @property+ def is_healthy(self) -> bool:+ return self.finding.is_healthy+++def check_pending_migrations() -> PendingMigrationsCheck:+ """Whether `pipelex migrate` has anything to do on this machine, and to which files.++ A boot does not tell anyone this. A stale configuration the ledger can explain boots with a+ warning, and the agent CLI silences its own logging before anything can emit one — so asking+ is the only way a machine consumer ever learns that a migration is pending. This row is the+ asking.++ **It takes no directory, and that is the decision rather than an omission.** Every other check+ reports on a *file* and is scoped to the directory the doctor was pointed at, `--global`+ included. This one reports on a *command*, and `pipelex migrate` has no `--global`: it walks+ the global `~/.pipelex/` and the project `.pipelex/` both. A row scoped narrower would name a+ command that then rewrites a file the row never mentioned, which is the one surprise a tool+ that writes to a user's files must not spring. Over-reporting is legible instead — every file+ is named with its full path, so a reader sees which directory each one is in.++ The dry run is the command's own, so this is not an approximation of what `pipelex migrate`+ would do. It is that run, with the writing switched off.++ **A failure inside the scan is reported as a failure to check**, never as a finding about the+ user's files and never as an exception — an exception here reaches `doctor_cmd`'s outer+ handler, which prints one line and exits, so a broken packaged ledger would replace every row+ the user came for. The catch stays narrow, so a bug in our applier still surfaces as itself.+ """+ try:+ report = migrate_config_directories(config_dirs=config_directories_to_migrate(), dry_run=True)+ except (MigrationError, OSError) as exc:+ return PendingMigrationsCheck(+ finding=PendingMigrationsFinding.UNAVAILABLE,+ message=f"Could not check for pending migrations: {exc}",+ )++ if report.is_clean:+ return PendingMigrationsCheck(+ finding=PendingMigrationsFinding.UP_TO_DATE,+ message="Every configuration file is at the current schema",+ )++ migratable_files = [str(plan.file_path) for plan in report.changed_plans]+ # In the order the run visited them, and deduplicated: one file can be both blocked and+ # carrying a path the schema cannot explain.+ attention_paths = {plan.file_path for plan in report.blocked_plans} | {plan.file_path for plan in report.unexplained_plans}+ attention_files = [str(plan.file_path) for plan in report.plans if plan.file_path in attention_paths]++ sentences: list[str] = []+ if migratable_files:+ sentences.append(f"{len(migratable_files)} configuration file(s) can be brought up to date by '{MIGRATE_COMMAND}'")+ if attention_files:+ sentences.append(f"{len(attention_files)} configuration file(s) need a look — run '{MIGRATE_COMMAND} --dry-run' for the detail")+ return PendingMigrationsCheck(+ finding=PendingMigrationsFinding.PENDING if migratable_files else PendingMigrationsFinding.NEEDS_ATTENTION,+ message="; ".join(sentences),+ migratable_files=migratable_files,+ attention_files=attention_files,+ )def check_backend_credentials(*, config_dir: Path | None = None) -> tuple[bool, dict[str, BackendCredentialsReport], str]:@@ -440,8 +645,8 @@ def display_health_report(config_healthy: bool,config_message: str,config_missing_count: int,- telemetry_healthy: bool,- telemetry_message: str,+ pending_migrations_check: PendingMigrationsCheck,+ telemetry_check: TelemetryConfigCheck,backends_healthy: bool,backends_message: str,backend_credential_reports: dict[str, BackendCredentialsReport],@@ -461,8 +666,8 @@ def display_health_report(config_healthy: Whether config files check passedconfig_message: Message about config files statusconfig_missing_count: Number of missing config files- telemetry_healthy: Whether telemetry check passed- telemetry_message: Message about telemetry status+ pending_migrations_check: What a dry run of `pipelex migrate` found, across every surface+ telemetry_check: What the telemetry probe found, and the sentence for itbackends_healthy: Whether backends check passedbackends_message: Message about backends statusbackend_credential_reports: Dict of backend credential reports@@ -478,7 +683,9 @@ def display_health_report(the Models row as a yellow advisory and suppress its standalone Solutions entry,since the Config Files row already steers the user."""- all_healthy = config_healthy and telemetry_healthy and backends_healthy and models_healthy and deck_healthy+ all_healthy = (+ config_healthy and pending_migrations_check.is_healthy and telemetry_check.is_healthy and backends_healthy and models_healthy and deck_healthy+ )# Overall status panelif all_healthy:@@ -516,12 +723,18 @@ def display_health_report(console.print(f" [red]✗[/red] {config_message}")console.print()+ # Configuration Migrations section. It sits next to Configuration Files because it is about+ # the same files — what the installed pipelex would change in them, rather than what it can read.+ console.print("[bold]Configuration Migrations[/bold]")+ _print_pending_migrations(check=pending_migrations_check)+ console.print()+# Telemetry Configuration sectionconsole.print("[bold]Telemetry Configuration[/bold]")- if telemetry_healthy:- console.print(f" [green]✓[/green] {telemetry_message}")+ if telemetry_check.is_healthy:+ console.print(f" [green]✓[/green] {escape(telemetry_check.message)}")else:- console.print(f" [red]✗[/red] {telemetry_message}")+ console.print(f" [red]✗[/red] {escape(telemetry_check.message)}")console.print()# Backend Credentials section@@ -601,12 +814,17 @@ def display_health_report(if not all_healthy:# Check what can be auto-fixedcan_auto_fix_config = not config_healthy and config_missing_count > 0- can_auto_fix_telemetry = not telemetry_healthy and (- "not found" in telemetry_message.lower()- or "format has changed" in telemetry_message.lower()- or "invalid configuration" in telemetry_message.lower()- )- has_telemetry_validation_error = not telemetry_healthy and not can_auto_fix_telemetry+ can_auto_fix_telemetry = telemetry_check.finding.is_repaired_by_initializing+ can_migrate = pending_migrations_check.finding.is_repaired_by_migrating+ telemetry_is_out_of_date = telemetry_check.finding.is_out_of_date+ # The migration row names the same command and covers every surface, so the telemetry-only+ # bullet under it would be the same advice twice. It still appears on its own — a telemetry+ # file whose only pending work is blocked leaves that row out of date and this one silent.+ show_telemetry_migrate_bullet = telemetry_is_out_of_date and not can_migrate+ # Read off the list rather than off the finding: a run can both migrate some files and+ # leave others for a person, and that combination is the ordinary one on a stale machine.+ migrations_need_a_look = bool(pending_migrations_check.attention_files)+ has_telemetry_validation_error = not telemetry_check.is_healthy and not can_auto_fix_telemetry and not telemetry_is_out_of_date# Check for backend file issueshas_backend_file_issues = False@@ -626,6 +844,10 @@ def display_health_report(has_recommendations = (can_auto_fix_configor can_auto_fix_telemetry+ or can_migrate+ or migrations_need_a_look+ or pending_migrations_check.finding.is_uncheckable+ or telemetry_is_out_of_dateor has_telemetry_validation_erroror (not backends_healthy and backend_credential_reports)or has_backend_file_issues@@ -641,9 +863,28 @@ def display_health_report(if can_auto_fix_telemetry:console.print(" • Run [cyan]pipelex init telemetry[/cyan] to configure telemetry preferences")- if has_telemetry_validation_error and "pipelex init telemetry" not in telemetry_message:+ if can_migrate:+ console.print(+ f" • Run [cyan]{MIGRATE_COMMAND}[/cyan] to bring "+ f"{len(pending_migrations_check.migratable_files)} configuration file(s) up to date"+ )++ if migrations_need_a_look:+ console.print(+ f" • Run [cyan]{MIGRATE_COMMAND} --dry-run[/cyan] to see what "+ f"{len(pending_migrations_check.attention_files)} configuration file(s) carry that the migration will not do on its own"+ )++ if pending_migrations_check.finding.is_uncheckable:+ console.print(f" • Run [cyan]{MIGRATE_COMMAND} --dry-run[/cyan] to check for pending migrations — this report could not")++ if show_telemetry_migrate_bullet:+ # Never `init telemetry` here: this file is not broken, it is old, and the+ # migration carries every setting in it forward where a reset would drop them.+ console.print(f" • Run [cyan]{MIGRATE_COMMAND}[/cyan] to bring telemetry.toml up to date")++ if has_telemetry_validation_error:console.print(f" • Fix validation errors in [cyan]{config_location.config_dir}/telemetry.toml[/cyan]")- console.print(" or run [cyan]pipelex init telemetry[/cyan] to regenerate")if has_deck_drift:console.print(" • Run [cyan]pipelex update[/cyan] to refresh the model deck from the current kit")@@ -689,7 +930,7 @@ def display_health_report(console.print()# Only suggest --fix if there are auto-fixable issues AND we're not already in fix mode- if not fix_mode and (can_auto_fix_config or can_auto_fix_telemetry or can_auto_fix_backends):+ if not fix_mode and (can_auto_fix_config or can_auto_fix_telemetry or can_migrate or can_auto_fix_backends):console.print("[dim]Run[/dim] [cyan]pipelex doctor --fix[/cyan] [dim]to interactively fix auto-fixable issues.[/dim]")console.print()@@ -703,6 +944,27 @@ def display_health_report(console.print()+def _print_pending_migrations(*, check: PendingMigrationsCheck) -> None:+ """Render the configuration-migration row: the verdict, then the files it is about.++ Every file is named with its full path — this row is the one place a reader learns that+ `pipelex migrate` would touch a file in a directory they were not asking about. Nothing read+ *inside* a file is rendered, here or anywhere else that reports a migration.+ """+ console = get_console()+ match check.finding:+ case PendingMigrationsFinding.UP_TO_DATE:+ console.print(f" [green]✓[/green] {escape(check.message)}")+ case PendingMigrationsFinding.PENDING | PendingMigrationsFinding.NEEDS_ATTENTION:+ console.print(f" [yellow]⚠[/yellow] {escape(check.message)}")+ for file_path in check.migratable_files:+ console.print(f" [dim]{escape(file_path)}[/dim] — out of date")+ for file_path in check.attention_files:+ console.print(f" [dim]{escape(file_path)}[/dim] — needs a look")+ case PendingMigrationsFinding.UNAVAILABLE:+ console.print(f" [red]✗[/red] {escape(check.message)}")++def check_deck_sync(*, config_dir: Path | None = None) -> tuple[bool, DeckSyncReport, str]:"""Check whether the installed model deck is in sync with the kit shipped by this pipelex version.@@ -775,10 +1037,12 @@ def setup_doctor_runtime(*, log_config_overrides: Mapping[str, Any] | None = Nonset_runtime_hub(runtime_hub)try:runtime_hub.setup_config(config_cls=PipelexConfig, config_dir=config_dir)- except ValidationError as validation_error:- validation_error_msg = report_validation_error(category="config", validation_error=validation_error)- msg = f"Could not setup config because of: {validation_error_msg}"- raise PipelexConfigError(msg) from validation_error+ except CONFIG_REFUSED as config_error:+ raise_config_setup_error(+ config_error=config_error,+ surface_id=PIPELEX_CONFIG_SURFACE_ID,+ config_dirs=[config_dir] if config_dir is not None else None,+ )log_config = get_config().pipelex.log_configif log_config_overrides is not None:@@ -787,6 +1051,8 @@ def setup_doctor_runtime(*, log_config_overrides: Mapping[str, Any] | None = Nonlog_config = LogConfig.model_validate(merged)runtime_hub.set_console_print_target(target=log_config.console_print_target)log.configure_if_unset(log_config=log_config)+ if (stale_warning := config_manager.take_stale_configuration_warning()) is not None:+ log.warning(stale_warning)def check_models(*, config_dir: Path | None = None) -> tuple[bool, str, dict[str, BackendFileReport]]:@@ -929,7 +1195,10 @@ def do_doctor_cmd(# side effect. Running it first would turn check_config_files into a silent installer# on a fresh machine. (The --global path skips materialization — see load_config.)config_healthy, config_missing_count, config_message = check_config_files()- telemetry_healthy, telemetry_message = check_telemetry_config()+ # Runs whether or not the config loads, and that is the point: a configuration that will not+ # load is exactly the machine most likely to be stale, and this row is what names the remedy.+ pending_migrations_check = check_pending_migrations()+ telemetry_check = check_telemetry_config()backends_healthy, backend_credential_reports, backends_message = check_backend_credentials()# check_models requires the hub + log.configure produced by setup_doctor_runtime.@@ -965,8 +1234,8 @@ def do_doctor_cmd(config_healthy=config_healthy,config_message=config_message,config_missing_count=config_missing_count,- telemetry_healthy=telemetry_healthy,- telemetry_message=telemetry_message,+ pending_migrations_check=pending_migrations_check,+ telemetry_check=telemetry_check,backends_healthy=backends_healthy,backends_message=backends_message,backend_credential_reports=backend_credential_reports,@@ -981,7 +1250,9 @@ def do_doctor_cmd(fix_mode=fix,)- all_healthy = config_healthy and telemetry_healthy and backends_healthy and models_healthy and deck_healthy+ all_healthy = (+ config_healthy and pending_migrations_check.is_healthy and telemetry_check.is_healthy and backends_healthy and models_healthy and deck_healthy+ )# Exit code: 0 if healthy, 1 if issues foundif all_healthy:@@ -989,12 +1260,10 @@ def do_doctor_cmd(# Determine what can be auto-fixedcan_fix_config = not config_healthy and config_missing_count > 0- # Telemetry can be fixed if not found, format changed, OR invalid configuration- can_fix_telemetry = not telemetry_healthy and (- "not found" in telemetry_message.lower()- or "format has changed" in telemetry_message.lower()- or "invalid configuration" in telemetry_message.lower()- )+ # Writing a fresh telemetry.toml repairs exactly one finding — a missing file. An out-of-date+ # one is `pipelex migrate`'s (named in the report above), and a broken one is a person's.+ can_fix_telemetry = telemetry_check.finding.is_repaired_by_initializing+ can_fix_migrations = pending_migrations_check.finding.is_repaired_by_migrating# Check for backend file issues that can be auto-fixedcan_fix_backends = False@@ -1006,12 +1275,12 @@ def do_doctor_cmd(can_fix_deck = not deck_healthy and deck_report.kit_version != ""- has_auto_fixable_issues = can_fix_config or can_fix_telemetry or can_fix_backends or can_fix_deck+ has_auto_fixable_issues = can_fix_config or can_fix_migrations or can_fix_telemetry or can_fix_backends or can_fix_deck# Determine what requires manual fixes (excludes auto-fixable issues)has_config_validation_error = not config_healthy and config_missing_count == 0- # Telemetry validation error only if it's not auto-fixable (not "not found" and not "format has changed")- has_telemetry_validation_error = not telemetry_healthy and not can_fix_telemetry+ # A telemetry finding a person has to resolve: neither a fresh file nor a migration gets there.+ has_telemetry_validation_error = not telemetry_check.is_healthy and not can_fix_telemetry and not telemetry_check.finding.is_out_of_datehas_backend_credential_issues = not backends_healthy and backend_credential_reports# If --fix flag is provided, offer to fix auto-fixable issues@@ -1032,17 +1301,28 @@ def do_doctor_cmd(console.print(f"[red]Failed to install configuration files: {exc!s}[/red]")console.print()- # Fix missing or outdated telemetry config+ # Migrate the configuration files the ledger can carry forward. This runs the same write+ # pass `pipelex migrate` runs — the row above was its dry run — rather than a second+ # implementation of it, and it is offered before the rows that report on file *contents*+ # because migrating can be what resolves them.+ if can_fix_migrations:+ migratable_count = len(pending_migrations_check.migratable_files)+ if Confirm.ask(f"[bold]Migrate {migratable_count} configuration file(s) to the current schema?[/bold]", default=True):+ try:+ console.print()+ applied = apply_pending_migrations(config_dirs=config_directories_to_migrate())+ console.print(f"[green]✓[/green] Migrated {len(applied.written_plans)} configuration file(s)")+ # The rows below were measured before this ran, so a file this just repaired can+ # still be reported as broken further down.+ console.print("[dim]Re-run[/dim] [cyan]pipelex doctor[/cyan] [dim]for an updated report.[/dim]")+ except Exception as exc: # noqa: BLE001+ # Doctor --fix handler: wraps the whole migration pass; a fix failure is reported and the doctor run continues.+ console.print(f"[red]Failed to migrate configuration files: {exc!s}[/red]")+ console.print()++ # Fix a missing telemetry configif can_fix_telemetry:- is_format_change = "format has changed" in telemetry_message.lower()- is_invalid_config = "invalid configuration" in telemetry_message.lower()- if is_format_change:- prompt_msg = "[bold]Reset telemetry configuration using the new format?[/bold]"- elif is_invalid_config:- prompt_msg = "[bold]Reset telemetry configuration to fix validation errors?[/bold]"- else:- prompt_msg = "[bold]Configure telemetry preferences?[/bold]"- if Confirm.ask(prompt_msg, default=True):+ if Confirm.ask("[bold]Configure telemetry preferences?[/bold]", default=True):try:console.print()init_cmd(focus=InitFocus.TELEMETRY, skip_confirmation=True)@@ -1109,13 +1389,15 @@ def do_doctor_cmd(console.print("or run [cyan]pipelex init config[/cyan] to regenerate from template.")console.print()- # Telemetry validation errors (skip if message already contains the fix command)- if has_telemetry_validation_error and "pipelex init telemetry" not in telemetry_message:+ # Telemetry validation errors. Regeneration is offered as what it is — a way to start+ # over that discards the file — and only here, where nothing else gets there. An+ # out-of-date file never reaches this branch: it has a migration that keeps its settings.+ if has_telemetry_validation_error:console.print("[bold]Telemetry validation error:[/bold]")- console.print(f" {telemetry_message}")+ console.print(f" {escape(telemetry_check.message)}")console.print()console.print(f"You can fix this manually by editing [cyan]{config_location.config_dir}/telemetry.toml[/cyan]")- console.print("or run [cyan]pipelex init telemetry[/cyan] to regenerate from template.")+ console.print("or run [cyan]pipelex init telemetry[/cyan] to start the file over, discarding what is in it.")console.print()# Backend credentialsdiff --git a/pipelex/cli/commands/migrate_cmd.py b/pipelex/cli/commands/migrate_cmd.pynew file mode 100644index 000000000..be405b631--- /dev/null+++ b/pipelex/cli/commands/migrate_cmd.py@@ -0,0 +1,214 @@+"""``pipelex migrate`` — bring the configuration files on this machine up to the current schema.++The command a user is sent to when their configuration no longer matches the models the installed+pipelex carries. It walks the global `~/.pipelex/` and the project `.pipelex/`, replays each+surface's shipped ledger over every file it claims there, and rewrites the files that changed —+each backed up first.++**This command must run when nothing else does.** A broken configuration is the reason to reach for+it, so it needs the ledger, the applier and the filesystem, and nothing else: no boot, no model+deck, no credentials, no network. That is a property to preserve, not an accident — it is pinned by+a test that hands the command a configuration that cannot load.++Two passes, and they are not the same pass twice for the sake of it. The first is a dry run whose+report is what the user is shown and asked about; the second is the one that writes. Under+always-replay the two agree, and where they do not — because the user edited a file in between, or+another process did — the second is the authoritative one and the transaction refuses rather than+writing over work it never saw.++See `docs/migration-ledger.md`.+"""++from __future__ import annotations++import sys+from typing import TYPE_CHECKING++from rich.markup import escape+from rich.panel import Panel+from rich.prompt import Confirm++from pipelex.migration.run import config_directories_to_migrate, migrate_config_directories+from pipelex.runtime_hub import get_console+from pipelex.suggested_fix import WILDCARD_SEGMENT, DeleteKeyOp, DeleteTableOp, MoveKeyOp, RemapValueOp, RenameTableKeyOp++if TYPE_CHECKING:+ from pathlib import Path++ from pipelex.migration.plan import BlockedEntry, MigrationReport, UnexplainedPath+ from pipelex.suggested_fix import MigrationOp++_NOTHING_TO_DO = "Every configuration file on this machine is at the current schema."+++def migrate_cmd(*, dry_run: bool = False, yes: bool = False) -> None:+ """Migrate the configuration files in the global and project configuration directories.++ Args:+ dry_run: Report what would change and write nothing.+ yes: Apply without the interactive confirmation.+ """+ console = get_console()+ if dry_run and yes:+ console.print("[red]✗[/red] --dry-run and --yes contradict each other: one refuses to write, the other authorizes it.")+ sys.exit(2)++ config_dirs = config_directories_to_migrate()+ if not config_dirs:+ console.print()+ console.print("[yellow]![/yellow] No configuration directory found. Run [cyan]pipelex init[/cyan] to create one.")+ console.print()+ return++ console.print()+ console.print("[bold]Configuration migration[/bold]")+ for directory in config_dirs:+ console.print(f" [dim]{escape(str(directory))}[/dim]")++ rehearsal = migrate_config_directories(config_dirs=config_dirs, dry_run=True)+ _print_report(report=rehearsal, wrote=False)++ if rehearsal.is_clean:+ console.print(_panel(message=_NOTHING_TO_DO, style="green"))+ console.print()+ return++ if dry_run:+ console.print("[dim]Dry run — nothing was written.[/dim]")+ console.print()+ _exit_on_attention(report=rehearsal)+ return++ if not rehearsal.changed_plans:+ # Nothing this tool can do, and something for the user to do. Asking "apply these changes?"+ # when there are none to apply is a prompt with one honest answer.+ console.print(_panel(message="Nothing here can be migrated automatically — see the notes above.", style="yellow"))+ console.print()+ _exit_on_attention(report=rehearsal)+ return++ if not yes and not Confirm.ask(f"[bold]Migrate {len(rehearsal.changed_plans)} file(s)?[/bold]", default=True):+ console.print("[yellow]Migration cancelled — nothing was written.[/yellow]")+ console.print()+ # The "no" answered "shall I write?"; what the rehearsal left for a person is still there.+ _exit_on_attention(report=rehearsal)+ return++ applied = apply_pending_migrations(config_dirs=config_dirs)+ written = len(applied.written_plans)+ unconfirmed = any(plan.blocked_reason is not None and plan.blocked_reason.leaves_the_write_unconfirmed for plan in applied.plans)+ if unconfirmed:+ # Tested first: a file in an unknown state is the verdict of the run, however many others landed.+ lead = f"Migrated {written} file(s), but no write could be confirmed for every file" if written else "No write could be confirmed"+ console.print(_panel(message=f"{lead} — compare each file marked above against the rescue copy named there.", style="red"))+ elif written:+ console.print(_panel(message=f"Migrated {written} file(s); a copy of each original is beside it.", style="green"))+ else:+ console.print(_panel(message="Nothing was written.", style="yellow"))+ console.print()+ _exit_on_attention(report=applied)+++def apply_pending_migrations(*, config_dirs: list[Path]) -> MigrationReport:+ """Write what a migration would write, render what it did, and hand the report back.++ The second of this command's two passes on its own, with no exit code attached to it. A+ caller that has already shown the user a dry run and asked its own question reaches the same+ code rather than reimplementing it — `pipelex doctor --fix` is that caller, and its+ pending-migrations row *is* the dry run. Calling `migrate_cmd` there instead would end the+ doctor's own run: the command exits the process when something is left for a person, and the+ doctor still has rows to render and an exit code of its own to set.+ """+ applied = migrate_config_directories(config_dirs=config_dirs, dry_run=False)+ _print_written(report=applied)+ return applied+++def _exit_on_attention(*, report: MigrationReport) -> None:+ """Leave a non-zero exit code behind when something needs a human.++ The exit code is presentation, not the verdict — `pipelex-agent migrate` carries the same+ verdict as a structured field. What it means here is narrow and worth keeping narrow: *this+ run left something a person has to look at*, never *this run failed*.+ """+ if report.needs_attention:+ sys.exit(1)+++def _print_report(*, report: MigrationReport, wrote: bool) -> None:+ console = get_console()+ for plan in report.plans:+ if plan.is_clean:+ continue+ console.print()+ console.print(f" [cyan]{escape(str(plan.file_path))}[/cyan] [dim]({plan.surface_id})[/dim]")+ if plan.blocked_reason is not None:+ console.print(f" [red]✗[/red] {plan.blocked_reason}: {escape(plan.blocked_detail or '')}")+ continue+ for step in plan.steps:+ marker = "[green]✓[/green]" if wrote else "[green]→[/green]"+ console.print(f" {marker} [dim]v{step.to_schema_version}[/dim] {escape(step.title)}")+ for op in step.applied_ops:+ console.print(f" [dim]{escape(describe_op(op=op))}[/dim]")+ for blocked in plan.blocked:+ _print_blocked_entry(blocked=blocked)+ for unexplained in plan.unexplained:+ _print_unexplained(unexplained=unexplained)+ console.print()+++def _print_written(*, report: MigrationReport) -> None:+ _print_report(report=report, wrote=True)+ console = get_console()+ for plan in report.written_plans:+ if plan.backup_path is not None:+ console.print(f" [dim]backup: {escape(str(plan.backup_path))}[/dim]")+++def _print_blocked_entry(*, blocked: BlockedEntry) -> None:+ console = get_console()+ console.print(f" [yellow]![/yellow] [dim]v{blocked.to_schema_version}[/dim] {blocked.reason}: {escape(blocked.detail)}")+ for op in blocked.applied_ops:+ console.print(f" [dim]applied: {escape(describe_op(op=op))}[/dim]")+ if blocked.guidance:+ for line in blocked.guidance.splitlines():+ console.print(f" {escape(line)}")+++def _print_unexplained(*, unexplained: UnexplainedPath) -> None:+ get_console().print(f" [yellow]?[/yellow] [cyan]{escape(unexplained.path)}[/cyan] — {escape(unexplained.note)}")+++def describe_op(*, op: MigrationOp) -> str:+ """One operation in words, from ledger-supplied material only.++ Every part of this sentence comes from the ledger — the operation's kind, its paths, and the+ spellings a remap names. Nothing read from the user's file appears, which is the same rule the+ plan models hold themselves to and the reason this is a rendering of the *operation* rather+ than a rendering of what the file used to say.+ """+ where = _table_in_words(table_path=op.table_path)+ match op:+ case DeleteKeyOp():+ return f"deleted '{op.key}' from {where}"+ case DeleteTableOp():+ return f"deleted the table {where}"+ case RenameTableKeyOp():+ return f"renamed '{op.key}' to '{op.new_key}' in {where}"+ case MoveKeyOp():+ return f"moved '{op.key}' from {where} to '{op.new_key}' in {_table_in_words(table_path=op.new_table_path)}"+ case RemapValueOp():+ renamings = ", ".join(f"'{before}' -> '{after}'" for before, after in sorted(op.mapping.items()))+ # `*` is the one key a remap may carry that names no key of its own: it means each key+ # of the addressed table, whose spellings only the user's document knows.+ subject = "every value" if op.key == WILDCARD_SEGMENT else f"the value of '{op.key}'"+ return f"rewrote {subject} in {where}: {renamings}"+++def _table_in_words(*, table_path: list[str]) -> str:+ """A table path as a report names it, the empty one included."""+ return f"'{'.'.join(table_path)}'" if table_path else "the document root"+++def _panel(*, message: str, style: str) -> Panel:+ return Panel(message, border_style=style, expand=False)diff --git a/pipelex/cli/commands/plugins_cmd.py b/pipelex/cli/commands/plugins_cmd.pyindex 298026881..190a58e16 100644--- a/pipelex/cli/commands/plugins_cmd.py+++ b/pipelex/cli/commands/plugins_cmd.py@@ -20,7 +20,7 @@ def plugins_list_command() -> None:would be wired in (and what the ``plugins.disabled`` denylist turns off)."""console = get_console()- config = PipelexConfig.model_validate(config_manager.load_config())+ config = config_manager.load_config_validated(config_cls=PipelexConfig)registrar = build_registrar(config=config,builtin_plugins=BUILTIN_PLUGINS,diff --git a/pipelex/cli/dev_cli/commands/check_ledger_cmd.py b/pipelex/cli/dev_cli/commands/check_ledger_cmd.pyindex f1d6fac6a..c61ed2237 100644--- a/pipelex/cli/dev_cli/commands/check_ledger_cmd.py+++ b/pipelex/cli/dev_cli/commands/check_ledger_cmd.py@@ -15,8 +15,9 @@ import sysfrom rich.markup import escapefrom pipelex.migration.exceptions import MigrationError+from pipelex.migration.ledger import packaged_migration_dirfrom pipelex.migration.ledger_check import LedgerIssue, check_ledgers-from pipelex.migration.surfaces import build_config_surface_registry, packaged_migration_dir+from pipelex.migration.surfaces import build_config_surface_registryfrom pipelex.runtime_hub import get_consolediff --git a/pipelex/cli/dev_cli/commands/check_migration_schemas_cmd.py b/pipelex/cli/dev_cli/commands/check_migration_schemas_cmd.pyindex 07c591bf3..fe40317d2 100644--- a/pipelex/cli/dev_cli/commands/check_migration_schemas_cmd.py+++ b/pipelex/cli/dev_cli/commands/check_migration_schemas_cmd.py@@ -19,7 +19,8 @@ from rich.markup import escapefrom pipelex.migration.coverage import CoverageIssue, check_registryfrom pipelex.migration.exceptions import MigrationError-from pipelex.migration.surfaces import build_config_surface_registry, packaged_migration_dir+from pipelex.migration.ledger import packaged_migration_dir+from pipelex.migration.surfaces import build_config_surface_registryfrom pipelex.migration.transform_check import TransformIssue, check_transformsfrom pipelex.runtime_hub import get_consolediff --git a/pipelex/cli/dev_cli/commands/update_migration_schemas_cmd.py b/pipelex/cli/dev_cli/commands/update_migration_schemas_cmd.pyindex afa5c4855..a9ec392e5 100644--- a/pipelex/cli/dev_cli/commands/update_migration_schemas_cmd.py+++ b/pipelex/cli/dev_cli/commands/update_migration_schemas_cmd.py@@ -16,8 +16,9 @@ import sysfrom rich.markup import escapefrom pipelex.migration.exceptions import MigrationError+from pipelex.migration.ledger import packaged_migration_dirfrom pipelex.migration.snapshot import snapshot_registry-from pipelex.migration.surfaces import build_config_surface_registry, packaged_migration_dir+from pipelex.migration.surfaces import build_config_surface_registryfrom pipelex.runtime_hub import get_consolediff --git a/pipelex/cli/error_handlers.py b/pipelex/cli/error_handlers.pyindex ec4005ac9..c491c7f72 100644--- a/pipelex/cli/error_handlers.py+++ b/pipelex/cli/error_handlers.py@@ -405,29 +405,34 @@ def handle_inference_setup_required_error(exc: InferenceSetupRequiredError) -> Ndef handle_telemetry_config_validation_error(exc: TelemetryConfigValidationError) -> NoReturn:- """Handle and display TelemetryConfigValidationError with migration guidance.+ """Handle and display TelemetryConfigValidationError, saying whether the file is old or wrong.- This error typically occurs when users have an old telemetry.toml format- that doesn't match the new nested structure.+ This used to announce a breaking change and send every reader to ``pipelex init telemetry``,+ which was written for one real event — the flat pre-``[custom_posthog]`` format — and then+ kept being shown for every other way a telemetry file can fail. The ledger now carries that+ format forward on its own, so a file reaching here is either one the ledger cannot fully+ explain or one that is genuinely invalid, and telling either reader to regenerate the file+ would throw away the keys and exporters in it.++ What the error itself carries is the answer: the fields the model refused, and a migration+ block when a scan found something a ``pipelex migrate`` would do.Args:exc: The telemetry config validation error exception"""console = get_console()print_traceback_if_requested(console=console)- console.print("\n[bold red]❌ Telemetry configuration format has changed[/bold red]\n")-- console.print(- "[bold yellow]⚠ Breaking Change:[/bold yellow] The telemetry.toml format has been updated.\n"- "Your existing configuration uses the old flat format.\n"- )+ console.print("\n[bold red]❌ Telemetry configuration could not be loaded[/bold red]\n")- console.print("[bold green]💡 To fix:[/bold green] Run [cyan]pipelex init telemetry[/cyan] to create a new config\n")+ console.print(escape(exc.message))+ console.print()- console.print("[dim]This update brings powerful new telemetry options:[/dim]")- console.print("[dim] • Langfuse integration for LLM observability[/dim]")- console.print("[dim] • Support for any OpenTelemetry backend via OTLP exporters[/dim]")- console.print("[dim] • Cleaner separation of PostHog, Langfuse, and OTLP settings[/dim]")+ if exc.migration is not None:+ console.print(f"[bold green]💡 To fix:[/bold green] Run [cyan]{exc.migration.remedy}[/cyan] to bring your configuration up to date\n")+ console.print("[dim]The migration keeps every setting in the file — it does not start it over.[/dim]")+ else:+ console.print("[bold green]💡 To fix:[/bold green] Correct the settings named above in [cyan]telemetry.toml[/cyan]\n")+ console.print("[dim]Or run [cyan]pipelex init telemetry[/cyan] to start the file over — this discards what is in it.[/dim]")console.print()console.print(f"[dim]Join our Discord for help: {URLs.discord}[/dim]\n")diff --git a/pipelex/core/validation.py b/pipelex/core/validation.pyindex 554029f12..a7ebc5f4c 100644--- a/pipelex/core/validation.py+++ b/pipelex/core/validation.py@@ -1,69 +1,174 @@-from pydantic import ValidationError+"""Translating a validation failure into something a person can act on.-from pipelex.runtime_hub import get_optional_config-from pipelex.system.configuration.configs import MigrationConfig, PipelexConfig+Two things can be wrong when a model refuses a document, and they call for opposite next moves.+The document may be *wrong* — a typo, a value out of range, a field that was never a field — and+then the pydantic analysis is the whole answer. Or the document may simply be *old*: written+against a schema this build has since moved past, in which case nothing about it is a mistake and+the remedy is a command rather than an edit.++Only the second case needs anything beyond the translation, and only a configuration surface can+be in it — so a caller that knows which surface refused says so, and this module asks the+migration ledger what a `pipelex migrate` would find. What comes back rides the error twice: as a+sentence in the message a human reads, and as the structured `migration` block a machine consumer+branches on.++> **A scan is a diagnosis, never a repair.** Nothing here writes; the remedy is a command the user+> runs. See `docs/migration-ledger.md` → "Reporting a stale configuration on a validation error".+"""++from pathlib import Path+from typing import NoReturn++from pydantic import BaseModel, ConfigDict, ValidationError++from pipelex.base_exceptions import MigrationErrorBlock, PipelexConfigError+from pipelex.migration.exceptions import MigrationError+from pipelex.migration.plan import MigrationPlan+from pipelex.system.configuration.config_loader import pydantic_error_behindfrom pipelex.tools.typing.pydantic_utils import analyze_pydantic_validation_error+MIGRATE_COMMAND = "pipelex migrate"+++class ValidationErrorReport(BaseModel):+ """A refused document, said twice: once as prose and once as structure.++ The two halves are the same answer for two readers, and the structured one is the contract —+ a consumer branches on `migration` being present, never on the wording of `message`.+ """++ model_config = ConfigDict(frozen=True, extra="forbid")++ message: str+ """The pydantic analysis, plus a paragraph about the pending migration when there is one."""++ migration: MigrationErrorBlock | None = None+ """What a `pipelex migrate` would find for the surface that refused, when a scan found+ anything at all. `None` means the failure is not staleness — or that no surface was named."""+++def report_validation_error(+ *,+ validation_error: ValidationError,+ surface_id: str | None = None,+ config_dirs: list[Path] | None = None,+) -> ValidationErrorReport:+ """Translate a validation failure, and say so when a pending migration would explain it.++ Args:+ validation_error: The pydantic error to translate.+ surface_id: The configuration surface whose model refused, when one did. Naming it is what+ turns on the migration scan; a caller validating something that is not a configuration+ surface — a `.mthds` bundle, an inference backend file, a model deck — passes nothing+ and gets the translation alone.+ config_dirs: The directories the refused configuration was loaded from, when the caller+ bypassed the global/project layering (`doctor --global`, an embedder's `config_dir=`).+ The scan then diagnoses those and only those; `None` is the ordinary load, and the+ scan walks what `pipelex migrate` walks.++ Returns:+ The translated message and, when a scan found something, the structured migration block.+ """+ message = analyze_pydantic_validation_error(validation_error).error_msg+ if surface_id is None:+ return ValidationErrorReport(message=message)+ block = _pending_migration(surface_id=surface_id, config_dirs=config_dirs)+ if block is None:+ return ValidationErrorReport(message=message)+ return ValidationErrorReport(message=f"{message}\n\n{_migration_prose(block=block)}", migration=block)+++def raise_config_setup_error(*, config_error: Exception, surface_id: str, config_dirs: list[Path] | None = None) -> NoReturn:+ """Turn a refused configuration into the `PipelexConfigError` a caller should see, migration and all.++ Shared by every site that loads a configuration surface and cannot continue without it, so+ that the same refusal produces the same message and the same structured block wherever it is+ caught. Doing it by hand per site is how one of them came to catch only pydantic's error and+ stop reporting anything at all for the main configuration.++ A refusal carrying no pydantic error is re-raised untouched: there is no field-level analysis+ to translate, and its own message is already the whole account.++ Args:+ config_error: The refusal, either half of `CONFIG_REFUSED`.+ surface_id: The configuration surface that refused, which is what turns on the scan.+ config_dirs: The directories the configuration was loaded from, when the caller named+ them — see `report_validation_error`.++ Raises:+ PipelexConfigError: Always, unless `config_error` is re-raised as itself.+ """+ validation_error = pydantic_error_behind(config_error=config_error)+ if validation_error is None:+ raise config_error+ report = report_validation_error(validation_error=validation_error, surface_id=surface_id, config_dirs=config_dirs)+ msg = f"Could not setup config because of: {report.message}"+ raise PipelexConfigError(msg, migration=report.migration) from config_error+++def _pending_migration(*, surface_id: str, config_dirs: list[Path] | None) -> MigrationErrorBlock | None:+ """What a `pipelex migrate` would find for this surface, or `None` when it would find nothing.++ Runs on the failure path only, which is what lets it be a filesystem walk and a ledger replay+ at all: a boot whose configuration validates never reaches this module.++ **Nothing that goes wrong inside the scan may become the failure the user sees.** They have a+ configuration error in front of them and it names what to fix; replacing it with a packaging+ problem of ours would cost them the only message that helps. A ledger that will not load is+ loud where it should be loud — `make check-ledger`, and the `migrate` command itself. The+ catch is deliberately narrow rather than a blanket one: an applier bug raises neither of these+ and should keep surfacing as the bug it is.+ """+ # Imported here rather than at module level, and the reason is architectural, not lazy: the+ # migration engine's applier lives under `pipelex.pipeline`, an interpreter package, while+ # this module sits in `runtime_boot`'s import closure — the kernel layer, whose stated+ # property is that importing it loads zero interpreter modules. `make agent-check` would not+ # catch a module-level import here; only the full `make agent-test` would.+ from pipelex.migration.run import scan_config_surface # noqa: PLC0415++ try:+ report = scan_config_surface(surface_id=surface_id, config_dirs=config_dirs)+ except (MigrationError, OSError):+ return None+ plans = [plan for plan in report.plans if not plan.is_clean]+ if not plans:+ return None+ return MigrationErrorBlock(remedy=MIGRATE_COMMAND, needs_attention=report.needs_attention, plans=plans)+++def _migration_prose(*, block: MigrationErrorBlock) -> str:+ """The block as a paragraph, for the reader who gets the message rather than the fields.++ Says the same things as the boot-tolerance warning and in the same order, because the two are+ read in the same places and describe the same machine — the difference is only that this one+ is attached to a failure rather than to a boot that carried on. Everything it names comes from+ the ledger or from a file path: **no value read from a user's file appears here**, which is+ the third of the three channels that rule covers.+ """+ files = ", ".join(f"'{plan.file_path}'" for plan in block.plans)+ carried = sorted({step.title for plan in block.plans for step in plan.steps})+ sentences = [f"Your configuration may be out of date rather than wrong: {files}."]+ if carried:+ sentences.append(f"What `{block.remedy}` would carry forward: {'; '.join(carried)}.")+ unresolved = _what_needs_a_person(plans=block.plans)+ if unresolved:+ sentences.append(f"What it cannot do for you: {'; '.join(unresolved)}.")+ sentences.append(f"Run `{block.remedy}` to bring these files up to date.")+ return " ".join(sentences)+++def _what_needs_a_person(*, plans: list[MigrationPlan]) -> list[str]:+ """Everything in the scan a command will not resolve, each said once and in file order."""+ unresolved: list[str] = []+ for plan in plans:+ where = _file_name(file_path=plan.file_path)+ if plan.blocked_reason is not None:+ unresolved.append(f"{where} could not be read as configuration ({plan.blocked_reason})")+ unresolved.extend(f"{where} needs '{entry.entry_id}' applied by hand ({entry.reason})" for entry in plan.blocked)+ unresolved.extend(f"{where} sets '{unexplained.path}', which this build knows nothing about" for unexplained in plan.unexplained)+ return unresolved+-def report_validation_error(*, category: str, validation_error: ValidationError) -> str:- validation_error_analysis = analyze_pydantic_validation_error(validation_error)-- # Doctor calls this from inside its bootstrap, where setup_config may have raised- # mid-flight and the hub's _config is still None. Reading via the optional getter- # lets the friendly translation work without a try/except RuntimeError catch-all.- config = get_optional_config()- migration_config: MigrationConfig | None- if isinstance(config, PipelexConfig):- migration_config = config.migration- else:- migration_config = None-- migration_reports: list[str] = []-- missing_field_renamings: dict[tuple[tuple[str, str], ...], list[str]] = {}- if migration_config is not None:- for missing_field in validation_error_analysis.missing_fields:- text = missing_field.split(".")[-1]- if renamings := migration_config.text_in_renaming_values(category=category, text=text):- renamings_key = tuple(renamings)- if renamings_key not in missing_field_renamings:- missing_field_renamings[renamings_key] = []- missing_field_renamings[renamings_key].append(missing_field)-- extra_field_renamings: dict[tuple[tuple[str, str], ...], list[str]] = {}- if migration_config is not None:- for extra_field in validation_error_analysis.extra_fields:- # Extract field path before the colon (extra fields include ": value")- field_path = extra_field.split(":")[0].strip()- text = field_path.split(".")[-1]- if renamings := migration_config.text_in_renaming_keys(category=category, text=text):- renamings_key = tuple(renamings)- if renamings_key not in extra_field_renamings:- extra_field_renamings[renamings_key] = []- extra_field_renamings[renamings_key].append(extra_field)-- # Format grouped output for missing fields- for renamings_tuple, fields in missing_field_renamings.items():- renamings_str = "\n".join(f"• '{key}' -> '{value}'" for key, value in renamings_tuple)- if len(fields) == 1:- msg = f"Missing field '{fields[0]}' is possibly a new name related to one of these renamings:\n{renamings_str}"- else:- fields_str = ", ".join(f"'{f}'" for f in fields)- msg = f"Missing fields [{fields_str}] are possibly new names related to one of these renamings:\n{renamings_str}"- migration_reports.append(msg)-- # Format grouped output for extra fields- for renamings_tuple, fields in extra_field_renamings.items():- renamings_str = "\n".join(f"• '{key}' -> '{value}'" for key, value in renamings_tuple)- if len(fields) == 1:- msg = f"Extra field '{fields[0]}' is possibly an old deprecated name related to one of these renamings:\n{renamings_str}"- else:- fields_str = ", ".join(f"'{f}'" for f in fields)- msg = f"Extra fields [{fields_str}] are possibly old deprecated names related to one of these renamings:\n{renamings_str}"- migration_reports.append(msg)-- report_msg = validation_error_analysis.error_msg- if migration_reports:- migration_reports_str = "\n".join(migration_reports)- report_msg += "\n\nNote that some fields have been renamed in the new version of Pipelex.\n\n" + migration_reports_str- return report_msg+def _file_name(*, file_path: Path) -> str:+ """A file named the way a sentence names it — the leaf, since the paths were listed already."""+ return f"'{file_path.name}'"diff --git a/pipelex/libraries/library_manager.py b/pipelex/libraries/library_manager.pyindex 698192c84..9368d238a 100644--- a/pipelex/libraries/library_manager.py+++ b/pipelex/libraries/library_manager.py@@ -773,7 +773,7 @@ class LibraryManager(LibraryManagerAbstract):try:return self.load_from_blueprints(library_id=library_id, blueprints=blueprints)except ValidationError as validation_error:- validation_error_msg = report_validation_error(category="mthds", validation_error=validation_error)+ validation_error_msg = report_validation_error(validation_error=validation_error).messagemsg = f"Could not load blueprints from {[str(pth) for pth in valid_mthds_paths]} because of: {validation_error_msg}"raise LibraryError(message=msg,diff --git a/pipelex/migration/coverage.py b/pipelex/migration/coverage.pyindex ff3c23ff5..858bd7ef9 100644--- a/pipelex/migration/coverage.py+++ b/pipelex/migration/coverage.py@@ -45,8 +45,9 @@ from pydantic import BaseModel, ConfigDict, Fieldfrom pipelex.migration.fingerprint import PATH_SEPARATOR, TABLE_TYPE, SurfaceFingerprint, compute_fingerprintfrom pipelex.migration.goldens import defaults_golden_path, read_fingerprint_golden-from pipelex.migration.ledger import MigrationEntry, MigrationLedger, MigrationSafety, load_ledger+from pipelex.migration.ledger import MigrationEntry, MigrationLedger, load_ledgerfrom pipelex.migration.narrowing import describe_narrowing, is_remappable, lost_enumerated_spellings+from pipelex.migration.safety import MigrationSafetyfrom pipelex.migration.surfaces import Surface, SurfaceRegistryfrom pipelex.migration.walk import EntryWalk, walk_entryfrom pipelex.suggested_fix import WILDCARD_SEGMENTdiff --git a/pipelex/migration/diagnosis.py b/pipelex/migration/diagnosis.pynew file mode 100644index 000000000..c82516caf--- /dev/null+++ b/pipelex/migration/diagnosis.py@@ -0,0 +1,204 @@+"""The downgrade diagnosis — naming what the migration cannot explain.++Always-replay carries files *older* than the running pipelex. The same branch switch that motivates+it also produces files *newer* than it: an override migrated on one branch, then an older branch+checked out or the package downgraded. The older model rejects the newer key and the replay finds+nothing to do, because every operation's source is absent. Under the no-backward-compatibility+principle that is not repaired — but "nothing to do" beside a dead boot is precisely the failure+this project exists to kill, so it is **named**.++**This is the one part of a migration that needs the model.** The engine is deliberately model-free+— text in, text out — which is what lets the gates replay it over documents nobody writes. Asking+whether a path is one the current schema knows cannot be answered from a ledger, so the question is+put to the surface's *fingerprint*: the same projection the coverage gate diffs, computed from the+model the surface names. The diagnosis therefore lives here, beside the runner that has a surface,+rather than inside the engine that has only text.++Three rules make the answer trustworthy rather than merely computable.++- **The document diagnosed is the one the run leaves behind**, not the one it found. Everything the+ ledger explains has been carried forward by then, so what is left over is genuinely left over. On+ a dry run that document exists only in memory, which is why the diagnosis belongs to the run and+ not to a later pass over the file.+- **A blocked entry answers for its own material.** An `unsafe` entry is never applied, so the old+ shape it is about is still in the file — reported already, by name, with the entry's guidance.+ Reporting it a second time as unexplained would contradict the first report and send the user+ looking for a typo in a key the tool has just told them about.+- **A key the user chose is reported as the schema spells it.** Beneath an open mapping the schema+ says `levels.*` where the file says `levels.my_package`, and a typo *inside* such an entry is+ reported at `queues.*.retries`. The unknown segment is named, because naming it is the whole+ point; the user's own key beside it is not.++See `docs/migration-ledger.md` → "The downgrade direction".+"""++import copy+from collections.abc import Sequence+from typing import Any, cast++from pipelex.migration.documents import path_is_at_or_under_pattern+from pipelex.migration.fingerprint import PATH_SEPARATOR, TABLE_TYPE, PathFingerprint, SurfaceFingerprint+from pipelex.migration.ledger import MigrationEntry, MigrationLedger+from pipelex.migration.material import spelling_after_replay+from pipelex.migration.plan import BlockedEntry, UnexplainedPath+from pipelex.migration.walk import op_source_path+from pipelex.suggested_fix import WILDCARD_SEGMENT, RemapValueOp+from pipelex.system.configuration.config_surface import strip_reserved_meta++UNEXPLAINED_NOTE = (+ "the current schema has no setting there, and no ledger entry retires it — either the name is a typo, or this file was "+ "written by a newer pipelex than the one running, so check whether you are on an older branch or an older build"+)+"""The two readings, and nothing else.++They share one note because the remedy differs and the diagnosis genuinely cannot tell them apart:+a misspelling and a key from next month's release look identical to a schema that knows neither.+Guessing would send half the users to the wrong fix, and the two questions are cheap for a human to+tell apart once they are asked.+"""+++def diagnose_unexplained_paths(+ *,+ fingerprint: SurfaceFingerprint,+ document: dict[str, Any],+ ledger: MigrationLedger,+ blocked: Sequence[BlockedEntry],+) -> list[UnexplainedPath]:+ """Every path in a migrated document that neither the current schema nor the ledger explains.++ The document must be the one the run leaves behind. Passing the pre-migration document would+ report every stale key the ledger is about to repair, which is the opposite of the diagnosis.+ """+ # A deep copy, because the strip below edits the nested `[meta]` table in place and this+ # function only ever reads what it was handed.+ diagnosed = copy.deepcopy(document)+ # Exactly what boot does before validating: the reserved key is tolerated there, so a document+ # carrying it must not be reported here either. A `[meta]` holding anything else is not+ # reserved, is left in place, and is diagnosed like any other unknown table.+ strip_reserved_meta(config_dict=diagnosed)+ accounted = _paths_a_blocked_entry_answers_for(ledger=ledger, blocked=blocked)+ unexplained: list[UnexplainedPath] = []+ _diagnose_table(+ table=diagnosed,+ document_prefix=(),+ schema_prefix=(),+ fingerprint=fingerprint,+ accounted=accounted,+ unexplained=unexplained,+ )+ return sorted(unexplained, key=lambda found: found.path)+++def _diagnose_table(+ *,+ table: dict[str, Any],+ document_prefix: tuple[str, ...],+ schema_prefix: tuple[str, ...],+ fingerprint: SurfaceFingerprint,+ accounted: list[str],+ unexplained: list[UnexplainedPath],+) -> None:+ """Walk one table of the document against the schema, reporting the shallowest unknown paths.++ A tree walk rather than a comparison of two flat path sets, and that is what buys the two+ properties a flat diff cannot have: an unknown table is reported once instead of once per key+ inside it, and the schema spelling of every ancestor is known by the time a child is reported.+ """+ for key, value in table.items():+ document_path = (*document_prefix, str(key))+ resolved = _resolve_against_schema(schema_prefix=schema_prefix, key=str(key), fingerprint=fingerprint)+ if resolved is None:+ if not _is_accounted_for(path=document_path, accounted=accounted):+ unexplained.append(UnexplainedPath(path=PATH_SEPARATOR.join((*schema_prefix, str(key))), note=UNEXPLAINED_NOTE))+ # Never descended into: an unknown table's contents are unknown because it is, and+ # listing every key inside it buries the one name the user has to fix.+ continue+ schema_path, record = resolved+ if not isinstance(value, dict):+ continue+ if record.value_type != TABLE_TYPE and not record.open_node:+ # A table where the schema wants a scalar is a *type* error, and the model reports it+ # far better than a path walk could. Descending would invent unknown paths beneath a+ # path the schema knows perfectly well.+ continue+ _diagnose_table(+ table=cast("dict[str, Any]", value),+ document_prefix=document_path,+ schema_prefix=schema_path,+ fingerprint=fingerprint,+ accounted=accounted,+ unexplained=unexplained,+ )+++def _resolve_against_schema(+ *,+ schema_prefix: tuple[str, ...],+ key: str,+ fingerprint: SurfaceFingerprint,+) -> tuple[tuple[str, ...], PathFingerprint] | None:+ """The schema path and record for one document key, or `None` when the schema has neither.++ The exact name is tried first and the `*` child second, in that order, because the `*` child+ exists only beneath an open mapping and an open mapping has no named children — so the two+ lookups can never both succeed and the order costs nothing but reads in the right one.+ """+ named = (*schema_prefix, key)+ record = fingerprint.paths.get(PATH_SEPARATOR.join(named))+ if record is not None:+ return named, record+ under_open_node = (*schema_prefix, WILDCARD_SEGMENT)+ record = fingerprint.paths.get(PATH_SEPARATOR.join(under_open_node))+ if record is not None:+ return under_open_node, record+ return None+++def _is_accounted_for(*, path: tuple[str, ...], accounted: list[str]) -> bool:+ document_path = PATH_SEPARATOR.join(path)+ return any(path_is_at_or_under_pattern(path=document_path, pattern=pattern) for pattern in accounted)+++def _paths_a_blocked_entry_answers_for(*, ledger: MigrationLedger, blocked: Sequence[BlockedEntry]) -> list[str]:+ """The material every blocked entry is about, at every spelling a file could be carrying it under.++ A blocked entry is the *reason* its material is still in the file, and the report already names+ it — with the entry's own guidance, which is worth more than this diagnosis has to offer. So+ the paths it addresses are subtracted here rather than reported twice with two different+ stories.++ Over-covering is the safe direction and is chosen deliberately: every path subtracted is one the+ same report names in `blocked[]`, so the worst a wide answer costs is a second sentence about a+ key the user has already been told about — where a narrow one would contradict the first.+ """+ accounted: list[str] = []+ for blocked_entry in blocked:+ entry = ledger.entry_for_version(schema_version=blocked_entry.to_schema_version)+ if entry is None:+ continue+ for pattern in _paths_one_entry_answers_for(ledger=ledger, entry=entry):+ if pattern not in accounted:+ accounted.append(pattern)+ return accounted+++def _paths_one_entry_answers_for(*, ledger: MigrationLedger, entry: MigrationEntry) -> list[str]:+ """One entry's material: what its operations address and what it declares removed, traced forward.++ A `remap_value` contributes nothing. It changes a value and leaves the path exactly where it+ was, so the path it addresses is one the current schema still has — never an unexplained one,+ and subtracting it would silence a genuine typo that happens to sit at a remapped key.++ The forward trace is there because a later `safe` entry may have renamed the material around a+ blocked one, exactly as it does for the report an `unsafe` entry produces: the file this run+ leaves behind spells the material the way the end of the replay does.+ """+ sources = [op_source_path(op=op) for op in entry.ops if not isinstance(op, RemapValueOp)]+ sources.extend(entry.declared_removed_paths)+ spellings: list[str] = []+ for source in sources:+ for spelling in (source, spelling_after_replay(ledger=ledger, entry=entry, spelling=source)):+ if spelling not in spellings:+ spellings.append(spelling)+ return spellingsdiff --git a/pipelex/migration/documents.py b/pipelex/migration/documents.pyindex 7f434d1d5..e74b33719 100644--- a/pipelex/migration/documents.py+++ b/pipelex/migration/documents.py@@ -51,16 +51,40 @@ def document_carries_path(*, paths: set[str], pattern: str) -> bool:A schema path is not a document path: beneath an open mapping the schema says `levels.*` whilethe document says `levels.my_package`, and a literal lookup would answer "no" for every file- that has ever set one. Each `*` matches exactly one segment, which is what it means in the- fingerprint and in an operation alike — never a whole subtree.+ that has ever set one."""if WILDCARD_SEGMENT not in pattern:return pattern in paths+ return any(path_matches_pattern(path=path, pattern=pattern) for path in paths)+++def path_matches_pattern(*, path: str, pattern: str) -> bool:+ """Whether one document path is the one a schema path names.++ Each `*` matches exactly one segment, which is what it means in the fingerprint and in an+ operation alike — never a whole subtree. This is the single definition of that rule; everything+ that compares a document against a schema path or an operation's target reads it, so the two+ directions of the question cannot come to different answers.+ """expected = pattern.split(PATH_SEPARATOR)- for path in paths:- segments = path.split(PATH_SEPARATOR)- if len(segments) != len(expected):- continue- if all(wanted in {WILDCARD_SEGMENT, found} for wanted, found in zip(expected, segments, strict=True)):- return True- return False+ segments = path.split(PATH_SEPARATOR)+ if len(segments) != len(expected):+ return False+ return _segments_match(segments=segments, expected=expected)+++def path_is_at_or_under_pattern(*, path: str, pattern: str) -> bool:+ """Whether a document path is the one a pattern names, or sits inside it.++ An operation that deletes or renames a table takes the whole subtree with it, so a pattern+ naming that table answers for every path beneath it too.+ """+ expected = pattern.split(PATH_SEPARATOR)+ segments = path.split(PATH_SEPARATOR)+ if len(segments) < len(expected):+ return False+ return _segments_match(segments=segments[: len(expected)], expected=expected)+++def _segments_match(*, segments: list[str], expected: list[str]) -> bool:+ return all(wanted in {WILDCARD_SEGMENT, found} for wanted, found in zip(expected, segments, strict=True))diff --git a/pipelex/migration/engine.py b/pipelex/migration/engine.pyindex 1d7ce7e78..7b84e9648 100644--- a/pipelex/migration/engine.py+++ b/pipelex/migration/engine.py@@ -42,9 +42,10 @@ import tomlkitfrom pydantic import BaseModel, ConfigDict, Fieldfrom pipelex.migration.documents import document_carries_path, document_paths-from pipelex.migration.ledger import MigrationEntry, MigrationLedger, MigrationSafety+from pipelex.migration.ledger import MigrationEntry, MigrationLedgerfrom pipelex.migration.material import declared_path_spellings, spelling_after_replay, unsafe_op_variantsfrom pipelex.migration.plan import BlockedEntry, BlockedEntryReason, MigrationStep+from pipelex.migration.safety import MigrationSafetyfrom pipelex.pipeline.fixes.applier import FixOpApplication, apply_fix_opsfrom pipelex.suggested_fix import MigrationOpdiff --git a/pipelex/migration/ledger.py b/pipelex/migration/ledger.pyindex 8e2a6aa4b..29a62d636 100644--- a/pipelex/migration/ledger.py+++ b/pipelex/migration/ledger.py@@ -11,7 +11,6 @@ gate: the earlier a malformed ledger stops, the fewer places have to cope with oSee `docs/migration-ledger.md` → "The ledger file"."""-from enum import StrEnumfrom functools import cachefrom pathlib import Pathfrom typing import Any@@ -20,6 +19,7 @@ from pydantic import BaseModel, ConfigDict, Field, ValidationError, model_validafrom typing_extensions import Selffrom pipelex.migration.exceptions import MigrationLedgerError+from pipelex.migration.safety import MigrationSafetyfrom pipelex.suggested_fix import MigrationOpfrom pipelex.tools.misc.exceptions import TomlErrorfrom pipelex.tools.misc.toml_utils import load_toml_from_path@@ -29,17 +29,15 @@ INITIAL_SCHEMA_VERSION = 1"""Every surface starts here. There is no retroactive numbering of changes that predate the ledger."""-class MigrationSafety(StrEnum):- """Whether the applier may act on an entry.+def packaged_migration_dir() -> Path:+ """The directory holding the checked-in ledgers and golden chains — this module's own.- Independent of `guidance`, which any entry may carry whatever its safety.+ It lives beside `load_ledger` rather than beside the surface registry because boot tolerance+ needs a ledger and must not need a registry: the registry names the configuration models, and+ pulling those into a loader that is itself part of the configuration package is how an import+ cycle starts. A loader already knows which surface it is."""-- SAFE = "safe"- """Mechanically complete; applied after one confirmation."""-- UNSAFE = "unsafe"- """Reported and never applied — the applier cannot tell a stale value from a deliberate one."""+ return Path(__file__).resolve().parentclass SurfaceBlock(BaseModel):diff --git a/pipelex/migration/ledger_check.py b/pipelex/migration/ledger_check.pyindex 4d517192e..867fe86db 100644--- a/pipelex/migration/ledger_check.py+++ b/pipelex/migration/ledger_check.py@@ -42,9 +42,10 @@ from pydantic import BaseModel, ConfigDict, Fieldfrom pipelex.migration.engine import replay_ledger_over_textfrom pipelex.migration.fingerprint import PATH_SEPARATOR, SurfaceFingerprintfrom pipelex.migration.goldens import read_fingerprint_golden-from pipelex.migration.ledger import INITIAL_SCHEMA_VERSION, MigrationEntry, MigrationLedger, MigrationSafety, load_ledger+from pipelex.migration.ledger import INITIAL_SCHEMA_VERSION, MigrationEntry, MigrationLedger, load_ledgerfrom pipelex.migration.plan import BlockedEntryReasonfrom pipelex.migration.reserved import ReservedRegistry, derive_reserved_registry+from pipelex.migration.safety import MigrationSafetyfrom pipelex.migration.surfaces import Surface, SurfaceRegistryfrom pipelex.migration.walk import EntryWalk, WalkedOp, op_source_path, walk_entryfrom pipelex.suggested_fix import WILDCARD_SEGMENT, MigrationOp, RemapValueOpdiff --git a/pipelex/migration/material.py b/pipelex/migration/material.pyindex 1457fb9b6..c7ba21d0f 100644--- a/pipelex/migration/material.py+++ b/pipelex/migration/material.py@@ -32,7 +32,8 @@ See `docs/migration-ledger.md` → "What an `unsafe` entry promises".from collections.abc import Sequencefrom pipelex.migration.fingerprint import PATH_SEPARATOR-from pipelex.migration.ledger import MigrationEntry, MigrationLedger, MigrationSafety+from pipelex.migration.ledger import MigrationEntry, MigrationLedger+from pipelex.migration.safety import MigrationSafetyfrom pipelex.migration.walk import joined, op_source_pathfrom pipelex.suggested_fix import DeleteKeyOp, DeleteTableOp, MigrationOp, MoveKeyOp, RemapValueOp, RenameTableKeyOpdiff --git a/pipelex/migration/plan.py b/pipelex/migration/plan.pyindex ab12a8d38..87a33702b 100644--- a/pipelex/migration/plan.py+++ b/pipelex/migration/plan.py@@ -16,6 +16,12 @@ and the write — and that reason sits on the plan itself, where it never stops> rather than a list of credential-shaped key names, because such a list is a guess that> eventually misses one.+This module is deliberately low-level — stdlib, pydantic, and the two sibling modules that are+themselves low-level (`safety`, `suggested_fix`) — so that `pipelex.base_exceptions` can import it+without creating a cycle. That is what lets a configuration validation error carry a real+`MigrationPlan` rather than a second projection of one that would drift from this one. Nothing+here may reach `migration.exceptions`, `migration.ledger` or anything that imports them.+See `docs/migration-ledger.md` → "What the engine reports"."""@@ -24,7 +30,7 @@ from pathlib import Pathfrom pydantic import BaseModel, ConfigDict, Field-from pipelex.migration.ledger import MigrationSafety+from pipelex.migration.safety import MigrationSafetyfrom pipelex.suggested_fix import MigrationOp@@ -105,6 +111,15 @@ class FileBlockedReason(StrEnum):UNPARSEABLE = "unparseable""""The file was read and is not valid UTF-8, or not valid TOML. Nothing was written."""+ UNSUPPORTED_SCHEMA_VERSION = "unsupported_schema_version"+ """The file declares a `[meta] schema_version` below its ledger's floor, so the entries that+ would carry it forward are no longer in the ledger. Nothing was written.++ This is the one state a replay cannot detect for itself, and the reason the floor exists: the+ applier skips an operation whose target is absent, so a squashed ledger run over a file older+ than the squash would report success over a file it under-migrated. Refusing by name is the+ alternative to that silence."""+UNWRITABLE = "unwritable""""The file needed a change and the run could not make it — the backup would not go down, orthe replacement would not. The file is exactly as it was found."""@@ -123,6 +138,16 @@ class FileBlockedReason(StrEnum):reaches the plan at all: a replacement that fails re-raises its own `OSError`, because rollingback nothing is trivially complete."""+ @property+ def leaves_the_write_unconfirmed(self) -> bool:+ """Whether this reason means the file may hold something other than what it was found with.++ Every other reason confirms this migration did not write — including `CHANGED_DURING_RUN`,+ under which somebody else did; a summary that says "nothing was written" is true of them+ and false of this one.+ """+ return self is FileBlockedReason.STATE_UNCERTAIN+class UnexplainedPath(BaseModel):"""A path the current schema does not know and no ledger entry removes.@@ -165,6 +190,18 @@ class MigrationPlan(BaseModel):"""Whether the file needs nothing: nothing applied, nothing blocked, nothing unexplained."""return not (self.steps or self.blocked or self.unexplained or self.blocked_reason)+ @property+ def did_change(self) -> bool:+ """Whether anything applied to this file — a whole entry, or the part of a conflicting one+ that landed before the conflict was found.++ The plan-level twin of `DocumentReplay.did_change_document`, which is what the runner writes+ on; the two must agree or a dry run stops predicting the write. A file can therefore be both+ changed and blocked: the entry that conflicted is reported once, under `blocked`, carrying+ the operations of it that did apply.+ """+ return bool(self.steps) or any(entry.applied_ops for entry in self.blocked)+class MigrationReport(BaseModel):"""Every file one run visited."""@@ -175,12 +212,39 @@ class MigrationReport(BaseModel):@propertydef written_plans(self) -> list[MigrationPlan]:+ """The files this run rewrote. Always empty on a dry run, which writes nothing."""return [plan for plan in self.plans if plan.was_written]+ @property+ def changed_plans(self) -> list[MigrationPlan]:+ """The files something applied to — whether or not this run was allowed to write them.++ The difference from `written_plans` is exactly what a dry run is for: these are the files+ a write pass would rewrite — including a file whose only change is the part of a+ conflicting entry that applied before the conflict.+ """+ return [plan for plan in self.plans if plan.did_change]+@propertydef blocked_plans(self) -> list[MigrationPlan]:+ """The files carrying something this run would not do — a blocked file, or a blocked entry."""return [plan for plan in self.plans if plan.blocked_reason is not None or plan.blocked]+ @property+ def unexplained_plans(self) -> list[MigrationPlan]:+ """The files carrying a path the current schema does not know and no entry removes."""+ return [plan for plan in self.plans if plan.unexplained]++ @property+ def needs_attention(self) -> bool:+ """Whether anything here is a human's to resolve rather than the tool's.++ This is the verdict a machine consumer branches on. It is deliberately *not* "did this run+ write anything": a run that migrated every file it found and left nothing blocked has+ succeeded, and a dry run that found nothing blocked has too.+ """+ return bool(self.blocked_plans or self.unexplained_plans)+@propertydef is_clean(self) -> bool:return all(plan.is_clean for plan in self.plans)diff --git a/pipelex/migration/run.py b/pipelex/migration/run.pynew file mode 100644index 000000000..d849db567--- /dev/null+++ b/pipelex/migration/run.py@@ -0,0 +1,101 @@+"""What a migration run is pointed at — the real registry, the packaged ledgers, and which directories.++`runner.py` answers *how* a file is migrated and takes everything it needs as a parameter, which is+what lets the gates replay the same code over documents that are never written anywhere. This+module answers *what a real run is aimed at*, and it is the only place in the migration package+that reads the machine: the surfaces the package owns, the ledgers shipped beside it, and the two+configuration directories a user actually has.++The separation is deliberate. Every consumer that migrates real files — both `migrate` commands,+and the validation error that reports a plan — must aim at exactly the same set of files, or a+user would be told one thing by their boot and another by their tool.++> **The walk is the global `~/.pipelex/` and the project `.pipelex/`, and nothing else.** A+> `config_dir=` load and `./tests/pipelex_{run_mode}.toml` are outside it. Both are deliberate:+> the first is a caller pointing the loader at a directory of its own choosing, and the second is+> this repository's own test fixture. Neither is a user's configuration, and migrating a+> directory nobody asked about is how a tool earns a reputation for touching things.++> **The walk is not recursive.** A surface's tier files sit beside its base file; a subdirectory+> under a configuration directory holds a different kind of thing — `inference/backends/`,+> `deck/` — whose files are not configuration surfaces even when their names would match a tier+> glob. `.pipelex/inference/backends/pipelex_gateway.toml` is the specimen: it matches the+> `pipelex-config` tier glob `pipelex_*.toml` and must never be claimed by it.++See `docs/migration-ledger.md` → "Surfaces" and "Applying".+"""++from pathlib import Path++from pipelex.migration.ledger import packaged_migration_dir+from pipelex.migration.plan import MigrationReport+from pipelex.migration.runner import migrate_directories+from pipelex.migration.surfaces import build_config_surface_registry+from pipelex.system.configuration.config_loader import config_manager+++def config_directories_to_migrate() -> list[Path]:+ """The configuration directories a migration run walks, in tier order, each of which exists.++ The global directory comes first and the project one second, which is the order the loader+ merges them in and therefore the order a report reads most naturally. A machine with only one+ of them is an ordinary machine; a project directory that *is* the global one — a project rooted+ at the home directory — is walked once rather than twice.+ """+ directories: list[Path] = []+ global_dir = config_manager.global_config_dir+ if global_dir.is_dir():+ directories.append(global_dir)+ project_dir = config_manager.project_config_dir+ if project_dir is not None and project_dir.resolve() not in {directory.resolve() for directory in directories}:+ directories.append(project_dir)+ return directories+++def migrate_config_directories(*, config_dirs: list[Path], dry_run: bool) -> MigrationReport:+ """Replay every surface's shipped ledger over every claimed file in the given directories.++ The registry and the ledger directory are the package's own, which is what makes this the+ real run rather than a gate's rehearsal over synthetic models.+ """+ return migrate_directories(+ registry=build_config_surface_registry(),+ migration_dir=packaged_migration_dir(),+ config_dirs=config_dirs,+ dry_run=dry_run,+ )+++def scan_config_surface(*, surface_id: str, config_dirs: list[Path] | None = None) -> MigrationReport:+ """What a `pipelex migrate` would find for one surface, without writing anything.++ The read-only half of the run, and the one a *diagnosis* wants: a configuration validation+ error is raised against the merged configuration and carries no provenance — it says a key is+ wrong, not which of the files that were merged put it there — so the way to answer "is this+ machine's configuration simply stale?" is to look at the files themselves.++ Scoped to the surface whose model refused, and to the same directories a real migration walks.+ Both halves matter: reporting a pending `telemetry.toml` migration underneath a+ `pipelex.toml` error would answer a question nobody asked, and scanning a directory the+ `migrate` command would not touch would offer a remedy that then does nothing.++ The scoping is a filter on the *answer*, and the full registry still decides which surface+ owns which file — see `migrate_directories`. Handing it a one-surface registry instead looks+ equivalent and is not: `pipelex_service.toml` is another surface's base file *and* a match for+ `pipelex-config`'s `pipelex_*.toml`, and with that other surface absent the glob wins.++ Args:+ surface_id: The surface to answer for.+ config_dirs: The directories to look in. The default — the ones a real migration walks —+ is what a boot failure wants, since it has no directory of its own to speak of. A+ caller that was pointed at one directory and is reporting on *that* passes it here+ rather than inheriting the walk: `pipelex doctor --global` inspects a file in+ `~/.pipelex/` and must not answer with a finding about the project directory beside it.+ """+ return migrate_directories(+ registry=build_config_surface_registry(),+ migration_dir=packaged_migration_dir(),+ config_dirs=config_dirs if config_dirs is not None else config_directories_to_migrate(),+ dry_run=True,+ only_surface_id=surface_id,+ )diff --git a/pipelex/migration/runner.py b/pipelex/migration/runner.pyindex 108c89011..1f308a49d 100644--- a/pipelex/migration/runner.py+++ b/pipelex/migration/runner.py@@ -15,16 +15,20 @@ See `docs/migration-ledger.md` → "Applying" and "Per-file transactions".from datetime import UTC, datetimefrom pathlib import Path+import tomlkit+from tomlkit import TOMLDocumentfrom tomlkit.exceptions import TOMLKitErrorfrom pipelex import logfrom pipelex.migration.backup import RescuedBackup, WrittenBackup, keep_backup_for_rescue, prune_backups_except, write_backup-from pipelex.migration.engine import replay_ledger_over_text+from pipelex.migration.diagnosis import diagnose_unexplained_paths+from pipelex.migration.engine import DocumentReplay, replay_ledger_over_textfrom pipelex.migration.ledger import MigrationLedger, load_ledger_cached-from pipelex.migration.plan import FileBlockedReason, MigrationPlan, MigrationReport+from pipelex.migration.plan import FileBlockedReason, MigrationPlan, MigrationReport, UnexplainedPathfrom pipelex.migration.surfaces import Surface, SurfaceRegistryfrom pipelex.pipeline.exceptions import FixTransactionError, FixWriteConflictErrorfrom pipelex.pipeline.fixes.file_transaction import FileSnapshot, PendingFileUpdate, commit_file_updates, read_file_snapshot+from pipelex.system.configuration.config_surface import version_declared_below_the_floordef migrate_file(*, surface: Surface, ledger: MigrationLedger, file_path: Path, dry_run: bool, moment: datetime) -> MigrationPlan:@@ -62,7 +66,7 @@ def migrate_file(*, surface: Surface, ledger: MigrationLedger, file_path: Path,)try:- replay = replay_ledger_over_text(ledger=ledger, text=text)+ document = tomlkit.loads(text)except TOMLKitError as exc:return _blocked_plan(surface_id=surface.surface_id,@@ -72,17 +76,83 @@ def migrate_file(*, surface: Surface, ledger: MigrationLedger, file_path: Path,detail=f"the file is not valid TOML: {exc}",)+ below_the_floor = _refuse_a_file_below_the_floor(surface=surface, ledger=ledger, file_path=file_path, document=document)+ if below_the_floor is not None:+ return below_the_floor++ try:+ replay = replay_ledger_over_text(ledger=ledger, text=text)+ except TOMLKitError as exc:+ # The text parsed a moment ago, so this is an operation failing on a document that is+ # valid TOML — an applier bug rather than a bad file. It is reported loudly against the+ # one file it happened on rather than allowed to abort the walk, which is the per-file+ # scope doing its job on a case nobody planned for.+ return _blocked_plan(+ surface_id=surface.surface_id,+ file_path=file_path,+ reason=FileBlockedReason.UNPARSEABLE,+ detail=f"an operation could not be applied to this file: {exc}",+ )+plan = MigrationPlan(surface_id=surface.surface_id,file_path=file_path,steps=replay.steps,blocked=replay.blocked,+ unexplained=_diagnose(surface=surface, ledger=ledger, document=document, replay=replay, text=text),)if not replay.did_change_document or dry_run:return planreturn _write_migrated_file(plan=plan, snapshot=snapshot, new_content=replay.text, moment=moment)+def _diagnose(*, surface: Surface, ledger: MigrationLedger, document: TOMLDocument, replay: DocumentReplay, text: str) -> list[UnexplainedPath]:+ """Ask what the migration could not explain about the document it is about to leave behind.++ **The document diagnosed is the migrated one**, not the one that was read: everything the+ ledger explains has been carried forward by the time the replay ends, so what is left over is+ genuinely left over. On a dry run that document exists only here, in memory, which is why the+ diagnosis belongs to the run rather than to a later pass over the written file.++ This is where a migration meets the model. The fingerprint is the surface's own projection —+ the same one the coverage gate diffs — and computing it neither loads the user's configuration+ nor touches the network, which is what keeps the command runnable on a machine that cannot+ boot. It costs a walk of the model tree per file, on the order of a couple of milliseconds.+ """+ fingerprint = surface.fingerprint_at(schema_version=ledger.surface.current_schema_version)+ migrated = document if replay.text == text else tomlkit.loads(replay.text)+ return diagnose_unexplained_paths(fingerprint=fingerprint, document=migrated.unwrap(), ledger=ledger, blocked=replay.blocked)+++def _refuse_a_file_below_the_floor(*, surface: Surface, ledger: MigrationLedger, file_path: Path, document: TOMLDocument) -> MigrationPlan | None:+ """Refuse a file that declares a schema version this ledger can no longer migrate from.++ The floor is the one thing a replay cannot work out for itself. The applier skips an operation+ whose target is absent and reports success, so a ledger whose oldest entries were squashed+ away would run over a file older than the squash, change nothing, and say it was fine. A+ document that *declares* where it stands is the only evidence available, which is why the+ reserved `[meta] schema_version` is read here — and, through the same predicate, in the+ boot-tolerance retry, which declines the file this refuses.++ Almost every file declares nothing and this returns `None` — the floor is zero on every+ surface today, and nothing writes the key. It earns its place the day a squash moves the floor.+ """+ declared = version_declared_below_the_floor(ledger=ledger, config_dict=document.unwrap())+ if declared is None:+ return None+ floor = ledger.surface.min_supported_schema_version+ return _blocked_plan(+ surface_id=surface.surface_id,+ file_path=file_path,+ reason=FileBlockedReason.UNSUPPORTED_SCHEMA_VERSION,+ detail=(+ f"this file declares schema version {declared} and the '{surface.surface_id}' ledger only migrates from "+ f"version {floor} onwards, so the entries that would bring it forward are no longer there — migrate it "+ f"with a pipelex release that still carries them, or re-create the file"+ ),+ )++def _write_migrated_file(*, plan: MigrationPlan, snapshot: FileSnapshot, new_content: str, moment: datetime) -> MigrationPlan:"""Back the file up, replace it atomically, then prune the older backups.@@ -229,17 +299,27 @@ def migrate_directories(migration_dir: Path,config_dirs: list[Path],dry_run: bool,+ only_surface_id: str | None = None,moment: datetime | None = None,) -> MigrationReport:"""Migrate every claimed file in every given configuration directory.The walk is over the directories a caller names — in practice the global `~/.pipelex/` and theproject's `.pipelex/`, and only those. A directory that does not exist is skipped.++ `only_surface_id` narrows the answer to one surface, for a caller that is diagnosing a+ particular model's refusal rather than migrating the machine. **It narrows the result, not the+ registry, and that distinction is the whole point**: which surface owns a file is decided+ across *all* of them — an exact base file claims before any glob — so a registry holding one+ surface would hand `pipelex_service.toml` to `pipelex-config`'s `pipelex_*.toml` and replay+ the wrong ledger over it. Arbitration first, then the filter."""stamp = moment if moment is not None else datetime.now(UTC)plans: list[MigrationPlan] = []for directory in config_dirs:for surface, file_path in registry.files_by_surface_in_directory(directory=directory):+ if only_surface_id is not None and surface.surface_id != only_surface_id:+ continueledger = load_ledger_cached(migration_dir=migration_dir, surface_id=surface.surface_id)plans.append(migrate_file(surface=surface, ledger=ledger, file_path=file_path, dry_run=dry_run, moment=stamp))return MigrationReport(plans=plans)diff --git a/pipelex/migration/safety.py b/pipelex/migration/safety.pynew file mode 100644index 000000000..92ac56184--- /dev/null+++ b/pipelex/migration/safety.py@@ -0,0 +1,25 @@+"""What an entry promises the applier — the one word both the ledger and the plan need.++A module of its own, and the reason is the import graph rather than the size. `plan.py` must stay+importable from `pipelex.base_exceptions` — that is what lets a configuration validation error+carry a real `MigrationPlan` instead of a second, drifting projection of one — and `ledger.py`+cannot be in that closure, because it reaches `migration.exceptions` and from there back into+`base_exceptions`. The vocabulary the two share therefore lives below both of them.++See `docs/migration-ledger.md` → "What an entry declares".+"""++from enum import StrEnum+++class MigrationSafety(StrEnum):+ """Whether the applier may act on an entry.++ Independent of `guidance`, which any entry may carry whatever its safety.+ """++ SAFE = "safe"+ """Mechanically complete; applied after one confirmation."""++ UNSAFE = "unsafe"+ """Reported and never applied — the applier cannot tell a stale value from a deliberate one."""diff --git a/pipelex/migration/snapshot.py b/pipelex/migration/snapshot.pyindex 10c78d8f1..acce6a08b 100644--- a/pipelex/migration/snapshot.py+++ b/pipelex/migration/snapshot.py@@ -26,7 +26,7 @@ from pydantic import BaseModel, ConfigDictfrom pipelex.migration.coverage import diff_fingerprintsfrom pipelex.migration.exceptions import MigrationSnapshotRefusedError-from pipelex.migration.fingerprint import SurfaceFingerprint, compute_fingerprint+from pipelex.migration.fingerprint import SurfaceFingerprintfrom pipelex.migration.goldens import read_fingerprint_golden, write_defaults_golden, write_fingerprint_goldenfrom pipelex.migration.ledger import load_ledgerfrom pipelex.migration.surfaces import Surface, SurfaceRegistry@@ -47,7 +47,7 @@ class SurfaceSnapshot(BaseModel):def snapshot_surface(*, surface: Surface, migration_dir: Path, force: bool = False) -> SurfaceSnapshot:ledger = load_ledger(migration_dir=migration_dir, surface_id=surface.surface_id)schema_version = ledger.surface.current_schema_version- fingerprint = compute_surface_fingerprint(surface=surface, schema_version=schema_version)+ fingerprint = surface.fingerprint_at(schema_version=schema_version)if not force:_refuse_a_destructive_head_overwrite(surface=surface, fingerprint=fingerprint, schema_version=schema_version, migration_dir=migration_dir)fingerprint_path = write_fingerprint_golden(migration_dir=migration_dir, fingerprint=fingerprint)@@ -66,15 +66,6 @@ def snapshot_surface(*, surface: Surface, migration_dir: Path, force: bool = Fal)-def compute_surface_fingerprint(*, surface: Surface, schema_version: int) -> SurfaceFingerprint:- return compute_fingerprint(- surface_id=surface.surface_id,- schema_version=schema_version,- config_model=surface.config_model,- defaults_document=surface.read_defaults_document(),- )--def _refuse_a_destructive_head_overwrite(*,surface: Surface,diff --git a/pipelex/migration/surfaces.py b/pipelex/migration/surfaces.pyindex 8cde71cc9..8a15fed85 100644--- a/pipelex/migration/surfaces.py+++ b/pipelex/migration/surfaces.py@@ -21,6 +21,12 @@ from pydantic import BaseModel, ConfigDict, Field, ValidationError, model_validafrom typing_extensions import Selffrom pipelex.migration.exceptions import MigrationRegistryError+from pipelex.migration.fingerprint import SurfaceFingerprint, compute_fingerprint+from pipelex.system.configuration.config_surface import (+ PIPELEX_CONFIG_SURFACE_ID,+ PIPELEX_SERVICE_CONFIG_SURFACE_ID,+ TELEMETRY_CONFIG_SURFACE_ID,+)from pipelex.system.configuration.configs import PipelexConfigfrom pipelex.system.pipelex_service.pipelex_service_agreement import PIPELEX_SERVICE_CONFIG_FILE_NAMEfrom pipelex.system.pipelex_service.pipelex_service_config import PipelexServiceConfig@@ -34,11 +40,6 @@ _PACKAGE_ROOT = Path(__file__).resolve().parent.parent_KIT_CONFIGS_DIR = _PACKAGE_ROOT / "kit" / "configs"-def packaged_migration_dir() -> Path:- """The directory holding the checked-in ledgers and golden chains — this module's own."""- return Path(__file__).resolve().parent--class DefaultsLayerKind(StrEnum):"""Where a surface's current-schema default values come from.@@ -88,6 +89,21 @@ class Surface(BaseModel):raise ValueError(msg)return self+ def fingerprint_at(self, *, schema_version: int) -> SurfaceFingerprint:+ """This surface's live models, projected and labelled with the schema version they are at.++ It lives on the surface because the surface is what holds both halves the projection needs+ — the model tree and the defaults layer it is read against. Both readers want the same+ answer for different reasons: the regenerator writes it into the golden chain, and a+ migration run asks it which paths the current schema knows.+ """+ return compute_fingerprint(+ surface_id=self.surface_id,+ schema_version=schema_version,+ config_model=self.config_model,+ defaults_document=self.read_defaults_document(),+ )+def render_reference_document(self) -> str:"""The surface's complete reference document as TOML text, ready to check in.@@ -267,7 +283,7 @@ def build_config_surface_registry() -> SurfaceRegistry:return SurfaceRegistry(surfaces=[Surface(- surface_id="pipelex-config",+ surface_id=PIPELEX_CONFIG_SURFACE_ID,title="The main Pipelex configuration",base_file=PIPELEX_CONFIG_FILE_NAME,tier_glob="pipelex_*.toml",@@ -277,7 +293,7 @@ def build_config_surface_registry() -> SurfaceRegistry:kit_template_path=_KIT_CONFIGS_DIR / PIPELEX_CONFIG_FILE_NAME,),Surface(- surface_id="telemetry-config",+ surface_id=TELEMETRY_CONFIG_SURFACE_ID,title="Telemetry destinations and capture settings",base_file=TELEMETRY_CONFIG_FILE_NAME,tier_glob="telemetry_*.toml",@@ -286,7 +302,7 @@ def build_config_surface_registry() -> SurfaceRegistry:kit_template_path=_KIT_CONFIGS_DIR / TELEMETRY_CONFIG_FILE_NAME,),Surface(- surface_id="pipelex-service-config",+ surface_id=PIPELEX_SERVICE_CONFIG_SURFACE_ID,title="Pipelex service agreement and onboarding state",base_file=PIPELEX_SERVICE_CONFIG_FILE_NAME,config_model=PipelexServiceConfig,diff --git a/pipelex/migration/transform_check.py b/pipelex/migration/transform_check.pyindex 2bbbc2711..9067b265a 100644--- a/pipelex/migration/transform_check.py+++ b/pipelex/migration/transform_check.py@@ -70,14 +70,14 @@ from typing import Anyfrom pydantic import BaseModel, ConfigDict, Field, ValidationError-from pipelex.migration.documents import document_paths+from pipelex.migration.documents import document_paths, path_matches_patternfrom pipelex.migration.engine import apply_ops_over_textfrom pipelex.migration.exceptions import MigrationGoldenErrorfrom pipelex.migration.fingerprint import PATH_SEPARATOR, SurfaceFingerprintfrom pipelex.migration.goldens import defaults_golden_path, pre_history_document_path, read_fingerprint_golden-from pipelex.migration.ledger import MigrationEntry, MigrationLedger, MigrationSafety, load_ledger+from pipelex.migration.ledger import MigrationEntry, MigrationLedger, load_ledger+from pipelex.migration.safety import MigrationSafetyfrom pipelex.migration.surfaces import Surface, SurfaceRegistry-from pipelex.suggested_fix import WILDCARD_SEGMENTfrom pipelex.system.configuration.config_surface import strip_reserved_metafrom pipelex.tools.misc.json_utils import deep_updatefrom pipelex.tools.misc.toml_utils import load_toml_from_content@@ -325,25 +325,10 @@ def _is_recorded(*, path: str, recorded_paths: set[str]) -> bool:A document names the user's own key where the fingerprint names `*`, so `deck.claude.new_name`is recorded as `deck.*.new_name`. Comparing the two literally would let a rename beneath anopen mapping read as a misspelled destination whenever the reference document happens not to- carry that key under that entry.+ carry that key under that entry. The matching rule is `path_matches_pattern`'s — the one+ definition every comparison of a document against a schema path reads."""- if path in recorded_paths:- return True- segments = path.split(PATH_SEPARATOR)- for recorded in recorded_paths:- recorded_segments = recorded.split(PATH_SEPARATOR)- if len(recorded_segments) != len(segments):- continue- if all(- _segment_matches(recorded=recorded_segment, concrete=segment)- for recorded_segment, segment in zip(recorded_segments, segments, strict=True)- ):- return True- return False---def _segment_matches(*, recorded: str, concrete: str) -> bool:- return recorded in {WILDCARD_SEGMENT, concrete}+ return any(path_matches_pattern(path=path, pattern=recorded) for recorded in recorded_paths)def _ancestors_are_in(*, path: str, paths: set[str]) -> bool:diff --git a/pipelex/pipeline/validate_bundle.py b/pipelex/pipeline/validate_bundle.pyindex 21a8799ab..ee4cdc00c 100644--- a/pipelex/pipeline/validate_bundle.py+++ b/pipelex/pipeline/validate_bundle.py@@ -154,7 +154,7 @@ def translate_to_validate_bundle_error() -> Generator[None, None, None]:) from pipe_errorexcept ValidationError as validation_error:pipe_validation_errors = categorize_pipe_validation_error(validation_error=validation_error)- validation_error_msg = report_validation_error(category="mthds", validation_error=validation_error)+ validation_error_msg = report_validation_error(validation_error=validation_error).messagemsg = f"Could not load blueprints because of: {validation_error_msg}"raise ValidateBundleError(message=msg,diff --git a/pipelex/runtime_boot.py b/pipelex/runtime_boot.pyindex f7fe4ab13..52f54f2e3 100644--- a/pipelex/runtime_boot.py+++ b/pipelex/runtime_boot.py@@ -46,7 +46,7 @@ from kajson.kajson_manager import KajsonManagerfrom pydantic import ValidationErrorfrom pipelex import log-from pipelex.base_exceptions import PipelexConfigError, PipelexSetupError+from pipelex.base_exceptions import PipelexSetupErrorfrom pipelex.cogt.content_generation.content_generator import ContentGeneratorfrom pipelex.cogt.content_generation.content_generator_protocol import (ContentGeneratorProtocol,@@ -71,7 +71,7 @@ from pipelex.cogt.models.model_manager_abstract import ModelManagerAbstractfrom pipelex.config import get_configfrom pipelex.core.registry_models import CoreRegistryModelsfrom pipelex.core.stuffs.stuff_template_set import STUFF_TEMPLATE_SET-from pipelex.core.validation import report_validation_error+from pipelex.core.validation import raise_config_setup_error, report_validation_errorfrom pipelex.graph.mermaidflow.template_set import MERMAID_TEMPLATE_SETfrom pipelex.graph.reactflow.template_set import REACTFLOW_TEMPLATE_SETfrom pipelex.observer.multi_observer import MultiObserver@@ -90,8 +90,9 @@ from pipelex.providers.builtins import KERNEL_BUILTIN_PLUGINS, KERNEL_CORE_UNCONfrom pipelex.reporting.reporting_manager import ReportingManagerfrom pipelex.reporting.reporting_protocol import ReportingProtocolfrom pipelex.runtime_hub import RuntimeHub, set_runtime_hub-from pipelex.system.configuration.config_loader import config_manager+from pipelex.system.configuration.config_loader import CONFIG_REFUSED, config_managerfrom pipelex.system.configuration.config_root import ConfigRoot+from pipelex.system.configuration.config_surface import PIPELEX_CONFIG_SURFACE_IDfrom pipelex.system.configuration.configs import PipelexConfigfrom pipelex.system.pipelex_service.exceptions import (GatewayTermsNotAcceptedError,@@ -191,15 +192,23 @@ class RuntimeBoot(metaclass=MetaSingleton):config_overrides=config_overrides,config_dir=config_dir,)- except ValidationError as validation_error:- validation_error_msg = report_validation_error(category="config", validation_error=validation_error)- msg = f"Could not setup config because of: {validation_error_msg}"- raise PipelexConfigError(msg) from validation_error+ except CONFIG_REFUSED as config_error:+ # Both halves of the pair, and the reason is that this arm used to catch only+ # pydantic's: `PipelexConfig` is a `ConfigRoot`, whose custom `__init__` translates+ # into `ConfigValidationError`, so the arm never fired for the one configuration+ # everything depends on and every boot failure arrived as a bare traceback instead.+ raise_config_setup_error(+ config_error=config_error,+ surface_id=PIPELEX_CONFIG_SURFACE_ID,+ config_dirs=[config_dir] if config_dir is not None else None,+ )log_config = get_config().pipelex.log_configself.runtime_hub.set_console_print_target(target=log_config.console_print_target)log.configure(log_config=log_config)log.verbose("Logs are configured")+ if (stale_warning := config_manager.take_stale_configuration_warning()) is not None:+ log.warning(stale_warning)# toolsself.class_registry: ClassRegistryAbstract | None = None@@ -232,7 +241,11 @@ class RuntimeBoot(metaclass=MetaSingleton):if not isinstance(cause_exc, ValidationError):msg += f"\nUnexpexted cause:{cause_exc}"raise PipelexSetupError(msg) from cause_exc- report = report_validation_error(category="config", validation_error=cause_exc)+ # No surface is named on purpose: the inference backend library and the model deck are not+ # configuration surfaces — they have no ledger and the `migrate` walk does not claim their+ # files — so offering a migration remedy for them would send the user to a command with+ # nothing to do.+ report = report_validation_error(validation_error=cause_exc).messagereturn f"""{msg}{report}diff --git a/pipelex/runtime_hub.py b/pipelex/runtime_hub.pyindex 7f538d2ac..75e5f79ba 100644--- a/pipelex/runtime_hub.py+++ b/pipelex/runtime_hub.py@@ -153,8 +153,11 @@ class RuntimeHub:overrides *are* overrides of). Used by the doctor ``--global`` pathso the hub reflects exactly the directory being reported on."""- config_dict = config_manager.load_config(extra_overrides=config_overrides, config_dir=config_dir)- self.set_config(config=config_cls.model_validate(config_dict))+ # ``load_config_validated`` rather than a load followed by a validate here, and the+ # difference is boot tolerance: a configuration a schema change left behind is replayed+ # through its ledger in memory and re-validated before this raises, so a stale file warns+ # instead of stopping the boot. See ``docs/migration-ledger.md`` → "Boot tolerance".+ self.set_config(config=config_manager.load_config_validated(config_cls=config_cls, extra_overrides=config_overrides, config_dir=config_dir))def set_config(self, config: ConfigRoot):if self._config is not None:diff --git a/pipelex/system/configuration/config_loader.py b/pipelex/system/configuration/config_loader.pyindex 76f9a66ae..f5fe59571 100644--- a/pipelex/system/configuration/config_loader.py+++ b/pipelex/system/configuration/config_loader.py@@ -2,14 +2,46 @@ import shutilfrom pathlib import Pathfrom typing import Any, TypeVar-from pydantic import BaseModel--from pipelex.system.configuration.config_surface import strip_reserved_meta+from pydantic import BaseModel, ValidationError++from pipelex.system.configuration.config_surface import (+ PIPELEX_CONFIG_SURFACE_ID,+ replay_surface_files_in_memory,+ stale_configuration_warning,+ strip_reserved_meta,+)+from pipelex.system.exceptions import ConfigValidationErrorfrom pipelex.system.runtime import runtime_managerfrom pipelex.tools.misc.json_utils import deep_updatefrom pipelex.tools.misc.toml_utils import load_toml_from_path_and_merge_with_overrides_PluginConfigT = TypeVar("_PluginConfigT", bound=BaseModel)+_ConfigT = TypeVar("_ConfigT", bound=BaseModel)++# What a configuration model raises when it refuses a document, and there are two of them.+# `ConfigRoot` gives itself a custom `__init__` that translates pydantic's error into ours, and+# pydantic v2 routes `model_validate` through a custom `__init__` — so the main configuration+# arrives as `ConfigValidationError` while a plain model surface arrives as pydantic's own.+# Catching one of the pair would silently switch boot tolerance off for whichever model it missed.+CONFIG_REFUSED = (ValidationError, ConfigValidationError)+++def pydantic_error_behind(*, config_error: Exception) -> ValidationError | None:+ """The pydantic error a refused configuration actually carries, whichever half of `CONFIG_REFUSED` arrived.++ The companion of the tuple above, and it exists for the same reason: a caller that wants the+ field-level analysis has to reach through `ConfigValidationError` to the `__cause__` the+ translation kept, and doing that by hand at each site is how one of them comes to catch only+ pydantic's and quietly stop reporting anything for the main configuration.++ `None` when the refusal carries no pydantic error at all — a `ConfigValidationError` raised for+ a reason of its own — in which case its own message is the whole account.+ """+ if isinstance(config_error, ValidationError):+ return config_error+ cause = config_error.__cause__+ return cause if isinstance(cause, ValidationError) else None+CONFIG_DIR_NAME = ".pipelex"CONFIG_NAME = "pipelex.toml"@@ -24,6 +56,19 @@ MODEL_DECKS_DIR_NAME = "deck"class ConfigLoader:+ def __init__(self) -> None:+ self._stale_warning: str | None = None++ def take_stale_configuration_warning(self) -> str | None:+ """The warning a tolerated boot owes the user, once — or ``None`` when the load was clean.++ The loader parks it rather than logging it because the main configuration is what+ *configures logging*: at the moment the retry succeeds there is no logger yet, and the+ dispatch raises on any attempt. The boot emits it right after ``log.configure``.+ """+ warning, self._stale_warning = self._stale_warning, None+ return warning+@propertydef pipelex_root_dir(self) -> Path:"""Get the root directory of the installed pipelex package.@@ -311,6 +356,25 @@ class ConfigLoader:Returns:dict[str, Any]: The merged configuration dictionary"""+ if config_dir is None:+ self.ensure_global_config_exists()+ merged = load_toml_from_path_and_merge_with_overrides(paths=self.config_file_paths(config_dir=config_dir))+ strip_reserved_meta(config_dict=merged)+ if extra_overrides:+ deep_update(merged, updates=extra_overrides)+ return merged++ def config_file_paths(self, *, config_dir: Path | None = None) -> list[Path]:+ """The ordered layers ``load_config`` merges, later files winning on a collision.++ Split out of ``load_config`` because boot tolerance has to replay the ledger over exactly+ the files that were merged, in exactly that order — a second list assembled beside this one+ would answer for a different machine the first time the layering changed. Pure: unlike+ ``load_config`` it never creates the global directory, so asking which files *would* be+ read does not bring one of them into existence.++ Missing files stay in the list and are skipped by whoever reads them.+ """is_unit_testing = runtime_manager.is_unit_testinglist_of_configs: list[Path] = [self.pipelex_root_dir / CONFIG_NAME]@@ -321,7 +385,6 @@ class ConfigLoader:self._override_files_for_dir(config_dir, include_run_mode=not is_unit_testing),)else:- self.ensure_global_config_exists()project_dir = self.project_config_dirlist_of_configs.append(self.global_config_dir / CONFIG_NAME)@@ -338,11 +401,66 @@ class ConfigLoader:if is_unit_testing:list_of_configs.append(Path.cwd() / "tests" / f"pipelex_{runtime_manager.run_mode}.toml")- merged = load_toml_from_path_and_merge_with_overrides(paths=list_of_configs)- strip_reserved_meta(config_dict=merged)+ return list_of_configs++ def load_config_validated(+ self,+ *,+ config_cls: type[_ConfigT],+ extra_overrides: dict[str, Any] | None = None,+ config_dir: Path | None = None,+ ) -> _ConfigT:+ """``load_config``, validated — and tolerant of a configuration the ledger can explain.++ This is the boot's entry point, and the tolerance is the whole reason it exists. A file+ left behind by a schema change should not stop the world: when validation fails, the+ surface's ledger is replayed over the same files **in memory**, the result is validated+ again, and a boot that succeeds says so in a warning naming the files and the+ ``pipelex migrate`` remedy — parked on the loader (``take_stale_configuration_warning``)+ for the boot to emit once logging exists. Nothing is written — only the explicit command+ writes.++ A configuration the ledger cannot explain raises exactly what it raised before: the retry+ is the only new behaviour, and it either recovers or gets out of the way.+ """+ config_dict = self.load_config(extra_overrides=extra_overrides, config_dir=config_dir)+ try:+ return config_cls.model_validate(config_dict)+ except CONFIG_REFUSED:+ recovered = self._config_the_ledger_can_explain(config_cls=config_cls, extra_overrides=extra_overrides, config_dir=config_dir)+ if recovered is None:+ raise+ return recovered++ def _config_the_ledger_can_explain(+ self,+ *,+ config_cls: type[_ConfigT],+ extra_overrides: dict[str, Any] | None,+ config_dir: Path | None,+ ) -> _ConfigT | None:+ """The same configuration with the ledger replayed over the user's files, or ``None``.++ ``None`` covers both ways this can decline — the ledger had nothing to say about these+ files, or it did and the result still does not validate. Neither is this method's to+ report: the caller re-raises the error the configuration actually produced, which is a+ truer account than "migration did not help" would be.++ The programmatic overrides are re-applied on top, because they are a layer of the load+ rather than a property of the files, and the replay only ever sees the files.+ """+ replayed = replay_surface_files_in_memory(surface_id=PIPELEX_CONFIG_SURFACE_ID, paths=self.config_file_paths(config_dir=config_dir))+ if replayed is None:+ return None+ config_dict = replayed.config_dictif extra_overrides:- deep_update(merged, updates=extra_overrides)- return merged+ deep_update(config_dict, updates=extra_overrides)+ try:+ config = config_cls.model_validate(config_dict)+ except CONFIG_REFUSED:+ return None+ self._stale_warning = stale_configuration_warning(plans=replayed.plans)+ return configconfig_manager = ConfigLoader()diff --git a/pipelex/system/configuration/config_surface.py b/pipelex/system/configuration/config_surface.pyindex 18dc1521d..414470009 100644--- a/pipelex/system/configuration/config_surface.py+++ b/pipelex/system/configuration/config_surface.py@@ -1,18 +1,37 @@-"""Read-side helper shared by every configuration surface.+"""Read-side helpers shared by every configuration surface.A **configuration surface** is one family of user-owned TOML files with one schema and onemigration ledger — today `pipelex.toml` and its tiers, `telemetry.toml`, and-`pipelex_service.toml`. What they share on the read path is the reserved `[meta]` table, and-this module is the one place that knows about it.+`pipelex_service.toml`. Two things are the same for all three on the read path, and this module+is the one place that knows about either.-The strip lives here and deliberately **not** in the generic TOML reader-(`pipelex.tools.misc.toml_utils`), which also reads `.mthds` files, backend definitions and the-kit index — none of which reserve that key, and all of which should keep rejecting it.+**The reserved `[meta]` table.** The strip lives here and deliberately **not** in the generic TOML+reader (`pipelex.tools.misc.toml_utils`), which also reads `.mthds` files, backend definitions and+the kit index — none of which reserve that key, and all of which should keep rejecting it.-See `docs/migration-ledger.md` → "Schema versions, and why every run replays everything".+**Boot tolerance.** A stale configuration should warn rather than stop the world, but only when+the ledger can explain it. Each surface's loader validates as it always did and, only when that+fails, asks `replay_surface_files_in_memory` for the same files carried forward — writing nothing,+because nothing writes but the explicit `migrate` command. What the loaders do *not* share is the+step between their merge and their validate: one deep-merges programmatic overrides, one+substitutes `${VAR}` placeholders, one does nothing at all. So the shared part is the failure path+alone, and each loader re-runs its own steps over what comes back.++See `docs/migration-ledger.md` → "Schema versions, and why every run replays everything" and+"Boot tolerance"."""-from typing import Any, cast+from collections.abc import Sequence+from pathlib import Path+from typing import Any, NamedTuple, cast++import tomlkit+from tomlkit.exceptions import TOMLKitError++from pipelex.migration.exceptions import MigrationLedgerError+from pipelex.migration.ledger import MigrationLedger, load_ledger_cached, packaged_migration_dir+from pipelex.migration.plan import MigrationPlan+from pipelex.tools.misc.json_utils import deep_update# The reserved in-file table and key. Every configuration-surface reader tolerates them and# strips them before validation; **nothing writes them**. The reason not to write is team skew@@ -39,3 +58,145 @@ def strip_reserved_meta(*, config_dict: dict[str, Any]) -> None:del typed_meta_table[RESERVED_SCHEMA_VERSION_KEY]if not typed_meta_table:del config_dict[RESERVED_META_TABLE]+++def declared_schema_version(*, config_dict: dict[str, Any]) -> int | None:+ """The schema version a configuration document declares, or `None` when it declares none.++ Almost every document returns `None`: **nothing writes the key**, so one carrying it was+ hand-written or came from a tool that is not this one. That is exactly why the read is+ tolerant rather than strict — a value that is not a plain integer is a malformed declaration,+ and the same document boots fine because `strip_reserved_meta` removes the key whatever it+ holds. Reading it as "no declaration" keeps the two halves of the reserved key consistent:+ what boot ignores, migration does not act on either.++ `bool` is excluded on purpose. It is an `int` to Python and never a schema version.+ """+ meta_table = config_dict.get(RESERVED_META_TABLE)+ if not isinstance(meta_table, dict):+ return None+ declared = cast("dict[str, Any]", meta_table).get(RESERVED_SCHEMA_VERSION_KEY)+ if isinstance(declared, bool) or not isinstance(declared, int):+ return None+ return declared+++def version_declared_below_the_floor(*, ledger: MigrationLedger, config_dict: dict[str, Any]) -> int | None:+ """The schema version a document declares when the ledger can no longer migrate from it, else `None`.++ The one comparison against `min_supported_schema_version` in the tree, read by both the+ migration runner (which refuses the file) and the boot-tolerance retry (which declines). The+ applier skips an absent target and reports success, so a ledger whose oldest entries were+ squashed away would run over such a file, change nothing, and call it fine — the declaration is+ the only evidence there is, and both paths must read it the same way or a boot would carry+ forward a file `pipelex migrate` then refuses.+ """+ declared = declared_schema_version(config_dict=config_dict)+ if declared is None or declared >= ledger.surface.min_supported_schema_version:+ return None+ return declared+++# The surface ids, spelled once. The migration registry names the same three, and every ledger+# file's `[surface] id` must agree with them — a boot-tolerance retry that asked for a surface id+# nothing ships would raise rather than tolerate, so the constant is the seam that keeps the+# loader and the registry from drifting apart on a string literal.+PIPELEX_CONFIG_SURFACE_ID = "pipelex-config"+TELEMETRY_CONFIG_SURFACE_ID = "telemetry-config"+PIPELEX_SERVICE_CONFIG_SURFACE_ID = "pipelex-service-config"+++class ReplayedSurface(NamedTuple):+ """A surface's user files as the ledger would leave them, and what it did to get there.++ The dict is a merge of the migrated documents, ready to be validated in place of the one that+ just failed. The plans are what a `pipelex migrate` over the same files would report.+ """++ config_dict: dict[str, Any]+ plans: list[MigrationPlan]+++def replay_surface_files_in_memory(*, surface_id: str, paths: Sequence[Path]) -> ReplayedSurface | None:+ """Re-read a surface's files, replay its ledger over each in memory, and merge the results.++ **The boot-tolerance retry, and it writes nothing.** A stale configuration should warn rather+ than stop the world, but only when the ledger can explain it — so a loader whose validation+ just failed calls this, re-validates what comes back, and boots with a warning if that+ succeeds. Nothing writes but the explicit `migrate` command.++ `None` means the ledger has nothing to say about these files: no operation applied and no+ entry is blocked. The failure is then not staleness, and the caller's own error stands.++ Three details that are the whole reason this is not two lines:++ - **The paths are the ones the loader merged, in the same order**, packaged defaults and all.+ Replaying over our own shipped configuration is not a special case to exclude, it is a+ no-op the gates already enforce — the ledger check replays every ledger over the packaged+ document and the kit template and demands they come back byte-identical.+ - **A file that cannot be read or parsed abandons the whole retry.** Skipping it would drop a+ layer from the merge, and a re-validation that then *succeeded* would boot on a+ configuration the user does not have. That file's own error is the one to show, and the+ first load already raised it.+ - **The reserved `[meta]` strip happens here**, because this replaces the read-and-merge the+ loader did rather than running after it.++ Nothing that goes wrong *inside* the retry is allowed to become the failure the user sees. A+ ledger that will not load is a packaging bug, and `make check-ledger` and `pipelex migrate`+ are where it is loud; on a machine in the field the user has a configuration error in front of+ them, and replacing it with ours would cost them the only message that names what to fix.+ """+ # Imported here rather than at module level, and the reason is architectural: the engine's+ # applier lives under `pipelex.pipeline`, an interpreter package, while this module sits in+ # `runtime_hub`'s import closure — the kernel layer, which loads zero interpreter modules.+ # Deferring it also makes the contract's "the healthy path is untouched" literal: a boot whose+ # configuration validates never even imports the migration engine.+ from pipelex.migration.engine import replay_ledger_over_text # noqa: PLC0415++ try:+ ledger = load_ledger_cached(migration_dir=packaged_migration_dir(), surface_id=surface_id)+ except MigrationLedgerError:+ return None+ merged: dict[str, Any] = {}+ plans: list[MigrationPlan] = []+ for path in paths:+ try:+ text = path.read_text(encoding="utf-8")+ except FileNotFoundError:+ continue+ except (OSError, UnicodeDecodeError):+ return None+ try:+ if version_declared_below_the_floor(ledger=ledger, config_dict=tomlkit.loads(text).unwrap()) is not None:+ # `pipelex migrate` refuses this file; a retry that carried it forward would boot on+ # an under-migrated configuration and then name a command that declines it.+ return None+ replay = replay_ledger_over_text(ledger=ledger, text=text)+ document = tomlkit.loads(replay.text)+ except TOMLKitError:+ return None+ plans.append(MigrationPlan(surface_id=surface_id, file_path=path, steps=replay.steps, blocked=replay.blocked))+ deep_update(merged, updates=document.unwrap())+ if all(plan.is_clean for plan in plans):+ return None+ strip_reserved_meta(config_dict=merged)+ return ReplayedSurface(config_dict=merged, plans=plans)+++def stale_configuration_warning(*, plans: Sequence[MigrationPlan]) -> str:+ """What a boot says when it carried a stale configuration forward in memory rather than dying.++ Names the files and the remedy, and takes everything else it says from the ledger — the same+ rule the migration report obeys, because a boot warning is read in the same places a report is+ and a value read from a user's file has no business in either.+ """+ stale = [plan for plan in plans if not plan.is_clean]+ files = ", ".join(f"'{plan.file_path}'" for plan in stale)+ carried = sorted({step.title for plan in stale for step in plan.steps})+ sentences = [f"Your configuration is out of date, and pipelex read it as if it had been migrated: {files}."]+ if carried:+ sentences.append(f"What the ledger carried forward: {'; '.join(carried)}.")+ if any(plan.blocked for plan in stale):+ sentences.append("Some of what these files need cannot be applied for you — `pipelex migrate` reports it.")+ sentences.append("Nothing was written: run `pipelex migrate` to bring the files up to date.")+ return " ".join(sentences)diff --git a/pipelex/system/configuration/configs.py b/pipelex/system/configuration/configs.pyindex 90b3186c5..d53cdf2dd 100644--- a/pipelex/system/configuration/configs.py+++ b/pipelex/system/configuration/configs.py@@ -213,19 +213,20 @@ class Pipelex(ConfigModel):class MigrationConfig(ConfigModel):- migration_maps: dict[str, dict[str, str]]+ """Dead, and deliberately still here.++ The hand-maintained rename tables that once let a validation error guess "you probably mean+ the new name for this". The migration ledger answers that question properly now — from the+ entries that actually performed the renames, per file rather than per word — so nothing reads+ these any more.- def text_in_renaming_keys(self, text: str, *, category: str) -> list[tuple[str, str]]:- renaming_map = self.migration_maps.get(category)- if not renaming_map:- return []- return [(key, value) for key, value in renaming_map.items() if text in key]-- def text_in_renaming_values(self, text: str, *, category: str) -> list[tuple[str, str]]:- renaming_map = self.migration_maps.get(category)- if not renaming_map:- return []- return [(key, value) for key, value in renaming_map.items() if text in value]+ The field stays because deleting it is a schema change: `pipelex.toml` ships a `[migration]`+ table and `extra="forbid"` would refuse every user file still carrying one. The configuration+ reshape's ledger entry is what removes both halves at once, which is what keeps the reshape a+ single schema bump rather than two.+ """++ migration_maps: dict[str, dict[str, str]]class PluginsConfig(ConfigModel):diff --git a/pipelex/system/pipelex_service/pipelex_service_config.py b/pipelex/system/pipelex_service/pipelex_service_config.pyindex 18dd88fff..be2c5cafc 100644--- a/pipelex/system/pipelex_service/pipelex_service_config.py+++ b/pipelex/system/pipelex_service/pipelex_service_config.py@@ -3,10 +3,16 @@ from typing import Any, castfrom pydantic import Field, ValidationError+from pipelex import logfrom pipelex.cogt.model_backends.backend import PipelexBackendfrom pipelex.system.configuration.config_loader import config_managerfrom pipelex.system.configuration.config_model import ConfigModel-from pipelex.system.configuration.config_surface import strip_reserved_meta+from pipelex.system.configuration.config_surface import (+ PIPELEX_SERVICE_CONFIG_SURFACE_ID,+ replay_surface_files_in_memory,+ stale_configuration_warning,+ strip_reserved_meta,+)from pipelex.system.pipelex_service.exceptions import PipelexServiceConfigValidationErrorfrom pipelex.system.pipelex_service.pipelex_service_agreement import (PIPELEX_SERVICE_CONFIG_FILE_NAME,@@ -42,11 +48,33 @@ def load_pipelex_service_config_if_exists(config_dir: Path) -> PipelexServiceConexcept FileNotFoundError:return Noneexcept ValidationError as exc:+ recovered = _service_config_the_ledger_can_explain(config_path=config_path)+ if recovered is not None:+ return recoveredvalidation_error_msg = format_pydantic_validation_error(exc)msg = f"Invalid Pipelex service configuration: {validation_error_msg}"raise PipelexServiceConfigValidationError(msg) from exc+def _service_config_the_ledger_can_explain(*, config_path: Path) -> PipelexServiceConfig | None:+ """The same service configuration with the ledger replayed over the user's file, or `None`.++ Boot tolerance for the one surface with no tiers: a single file, so the "merge" the shared+ helper performs is a merge of one. The ledger for this surface is empty today, which makes+ this the branch that costs nothing until the first entry lands — exactly the point of wiring+ every surface at once rather than only the one that currently needs it.+ """+ replayed = replay_surface_files_in_memory(surface_id=PIPELEX_SERVICE_CONFIG_SURFACE_ID, paths=[config_path])+ if replayed is None:+ return None+ try:+ service_config = PipelexServiceConfig.model_validate(replayed.config_dict)+ except ValidationError:+ return None+ log.warning(stale_configuration_warning(plans=replayed.plans))+ return service_config++def is_pipelex_gateway_enabled(backends_file_path: Path | None = None) -> bool:"""Check if pipelex_gateway is enabled in the backends configuration.diff --git a/pipelex/system/telemetry/exceptions.py b/pipelex/system/telemetry/exceptions.pyindex cf14c1d52..4da145610 100644--- a/pipelex/system/telemetry/exceptions.py+++ b/pipelex/system/telemetry/exceptions.py@@ -1,8 +1,15 @@-from pipelex.base_exceptions import PipelexError+from pipelex.base_exceptions import PipelexConfigError, PipelexError-class TelemetryConfigError(PipelexError):- pass+class TelemetryConfigError(PipelexConfigError):+ """A telemetry configuration this process cannot use.++ A :class:`PipelexConfigError` rather than a bare :class:`PipelexError`, and both halves of+ that are load-bearing. It carries ``error_domain = CONFIG`` from the class instead of from the+ agent CLI's lookup table, so the class is the single source of truth the drift test asks for;+ and it can carry a :class:`MigrationErrorBlock`, which is what lets a stale ``telemetry.toml``+ say *old* rather than *wrong* on every surface that reports it.+ """class TelemetryConfigValidationError(TelemetryConfigError):diff --git a/pipelex/system/telemetry/telemetry_config.py b/pipelex/system/telemetry/telemetry_config.pyindex db0457ba4..7070951a6 100644--- a/pipelex/system/telemetry/telemetry_config.py+++ b/pipelex/system/telemetry/telemetry_config.py@@ -1,19 +1,17 @@+"""The telemetry configuration models — what a `telemetry.toml` parses into.++Models only. `telemetry_loader.py` beside this reads the files and returns a `TelemetryConfig`,+and it is a module of its own because its failure path reaches the migration registry, which+imports every configuration model including these. See its docstring for the whole account.+"""+from enum import StrEnum-from functools import partialfrom typing import Self-from pydantic import BaseModel, ConfigDict, Field, ValidationError, model_validator+from pydantic import BaseModel, ConfigDict, Field, model_validator-from pipelex.system.configuration.config_loader import config_managerfrom pipelex.system.configuration.config_model import ConfigModel-from pipelex.system.configuration.config_surface import strip_reserved_meta-from pipelex.system.telemetry.exceptions import TelemetryConfigValidationError-from pipelex.tools.misc.dict_utils import apply_to_strings_recursive-from pipelex.tools.misc.toml_utils import load_toml_from_path_and_merge_with_overrides-from pipelex.tools.secrets.exceptions import UnknownVarPrefixError-from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract-from pipelex.tools.secrets.secrets_utils import substitute_vars-from pipelex.tools.typing.pydantic_utils import empty_list_factory_of, format_pydantic_validation_error+from pipelex.tools.typing.pydantic_utils import empty_list_factory_ofTELEMETRY_CONFIG_FILE_NAME = "telemetry.toml"TELEMETRY_CONFIG_OVERRIDE_FILE_NAME = "telemetry_override.toml"@@ -236,64 +234,3 @@ class TelemetryRedactionConfig(BaseModel):redact_output_class_names=True,content_max_length=None,)---def load_telemetry_config(*, secrets_provider: SecretsProviderAbstract) -> TelemetryConfig:- """Load telemetry configuration from a TOML file with variable substitution.-- Files are deep-merged in this order (later wins per leaf key):-- 1. ~/.pipelex/telemetry.toml (global base)- 2. ~/.pipelex/telemetry_override.toml- 3. {project_root}/.pipelex/telemetry.toml (if project dir exists and is- distinct from the global dir)- 4. {project_root}/.pipelex/telemetry_override.toml (same condition)-- This means a project telemetry config layers *on top of* the user's global- one rather than replacing it — secrets and personal observability settings- declared once in ~/.pipelex/ stay in effect across all projects.-- Supports variable placeholders in string values:- - ${VAR_NAME} -> use secrets provider by default- - ${env:ENV_VAR_NAME} -> force use environment variable- - ${secret:SECRET_NAME} -> force use secrets provider- - ${env:ENV_VAR|secret:SECRET} -> try env first, then secret as fallback-- Args:- secrets_provider: Provider for resolving secret/env variable placeholders.-- Returns:- Validated TelemetryConfig instance.-- Raises:- TelemetryConfigValidationError: If configuration is invalid or variable substitution fails.- """- global_config_dir = config_manager.global_config_dir- telemetry_config_paths = [- global_config_dir / TELEMETRY_CONFIG_FILE_NAME,- global_config_dir / TELEMETRY_CONFIG_OVERRIDE_FILE_NAME,- ]- project_config_dir = config_manager.project_config_dir- if project_config_dir is not None and project_config_dir != global_config_dir:- telemetry_config_paths.append(project_config_dir / TELEMETRY_CONFIG_FILE_NAME)- telemetry_config_paths.append(project_config_dir / TELEMETRY_CONFIG_OVERRIDE_FILE_NAME)- telemetry_config_toml_raw = load_toml_from_path_and_merge_with_overrides(paths=telemetry_config_paths)- strip_reserved_meta(config_dict=telemetry_config_toml_raw)-- # Apply variable substitution to all string values (keep placeholders for missing vars)- substitute_vars_with_provider = partial(substitute_vars, secrets_provider=secrets_provider, raise_on_missing_var=False)- try:- telemetry_config_toml = apply_to_strings_recursive(telemetry_config_toml_raw, transform_func=substitute_vars_with_provider)- except UnknownVarPrefixError as exc:- paths_str = "\n".join(str(path) for path in telemetry_config_paths)- msg = f"Variable substitution failed in telemetry configuration based on '{paths_str}': {exc}"- raise TelemetryConfigValidationError(msg) from exc-- try:- telemetry_config = TelemetryConfig.model_validate(telemetry_config_toml)- except ValidationError as exc:- validation_error_msg = format_pydantic_validation_error(exc)- paths_str = "\n".join(str(path) for path in telemetry_config_paths)- msg = f"Invalid telemetry configuration in '{paths_str}':\n{validation_error_msg}"- raise TelemetryConfigValidationError(msg) from exc- return telemetry_configdiff --git a/pipelex/system/telemetry/telemetry_factory.py b/pipelex/system/telemetry/telemetry_factory.pyindex d7183d108..b84e9e65d 100644--- a/pipelex/system/telemetry/telemetry_factory.py+++ b/pipelex/system/telemetry/telemetry_factory.py@@ -9,7 +9,8 @@ from pipelex.system.pipelex_service.pipelex_details import PipelexDetailsfrom pipelex.system.pipelex_service.remote_config import RemoteConfigfrom pipelex.system.runtime import IntegrationModefrom pipelex.system.telemetry.otel_constants import OTelConstants-from pipelex.system.telemetry.telemetry_config import PostHogMode, TelemetryConfig, load_telemetry_config+from pipelex.system.telemetry.telemetry_config import PostHogMode, TelemetryConfig+from pipelex.system.telemetry.telemetry_loader import load_telemetry_configfrom pipelex.system.telemetry.telemetry_manager_abstract import (TelemetryManagerAbstract,TelemetryManagerNoOp,diff --git a/pipelex/system/telemetry/telemetry_loader.py b/pipelex/system/telemetry/telemetry_loader.pynew file mode 100644index 000000000..0a4a1f8e0--- /dev/null+++ b/pipelex/system/telemetry/telemetry_loader.py@@ -0,0 +1,146 @@+"""Reading the telemetry configuration off the machine, and saying what is wrong when it will not load.++Separate from `telemetry_config.py`, which holds the models this returns, and the reason is the+import graph rather than the size. The failure path asks the migration ledger whether the files+are simply *old* — see `pipelex.core.validation` — and reaching that answer means reaching+`migration.surfaces`, the registry, which imports every configuration model there is, this+surface's included. Models and loader in one module close that into a cycle; apart, the edges run+one way: the registry reads the models, the loader reads the registry.++Same shape as `system/configuration/configs.py` beside `config_loader.py`, and the same rule the+migration work arrived at one level down: a loader must be able to reach the migration package+without reaching the registry. Deferring the import is not a way out of it — a cycle check counts+a function-level edge exactly like a module-level one.+"""++from collections.abc import Callable+from functools import partial+from pathlib import Path++from pydantic import ValidationError++from pipelex import log+from pipelex.core.validation import report_validation_error+from pipelex.system.configuration.config_loader import config_manager+from pipelex.system.configuration.config_surface import (+ TELEMETRY_CONFIG_SURFACE_ID,+ replay_surface_files_in_memory,+ stale_configuration_warning,+ strip_reserved_meta,+)+from pipelex.system.telemetry.exceptions import TelemetryConfigValidationError+from pipelex.system.telemetry.telemetry_config import (+ TELEMETRY_CONFIG_FILE_NAME,+ TELEMETRY_CONFIG_OVERRIDE_FILE_NAME,+ TelemetryConfig,+)+from pipelex.tools.misc.dict_utils import apply_to_strings_recursive+from pipelex.tools.misc.toml_utils import load_toml_from_path_and_merge_with_overrides+from pipelex.tools.secrets.exceptions import UnknownVarPrefixError+from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract+from pipelex.tools.secrets.secrets_utils import substitute_vars+++def load_telemetry_config(*, secrets_provider: SecretsProviderAbstract) -> TelemetryConfig:+ """Load telemetry configuration from a TOML file with variable substitution.++ Files are deep-merged in this order (later wins per leaf key):++ 1. ~/.pipelex/telemetry.toml (global base)+ 2. ~/.pipelex/telemetry_override.toml+ 3. {project_root}/.pipelex/telemetry.toml (if project dir exists and is+ distinct from the global dir)+ 4. {project_root}/.pipelex/telemetry_override.toml (same condition)++ This means a project telemetry config layers *on top of* the user's global+ one rather than replacing it — secrets and personal observability settings+ declared once in ~/.pipelex/ stay in effect across all projects.++ Supports variable placeholders in string values:+ - ${VAR_NAME} -> use secrets provider by default+ - ${env:ENV_VAR_NAME} -> force use environment variable+ - ${secret:SECRET_NAME} -> force use secrets provider+ - ${env:ENV_VAR|secret:SECRET} -> try env first, then secret as fallback++ Args:+ secrets_provider: Provider for resolving secret/env variable placeholders.++ Returns:+ Validated TelemetryConfig instance.++ Raises:+ TelemetryConfigValidationError: If configuration is invalid or variable substitution fails.+ """+ global_config_dir = config_manager.global_config_dir+ telemetry_config_paths = [+ global_config_dir / TELEMETRY_CONFIG_FILE_NAME,+ global_config_dir / TELEMETRY_CONFIG_OVERRIDE_FILE_NAME,+ ]+ project_config_dir = config_manager.project_config_dir+ if project_config_dir is not None and project_config_dir != global_config_dir:+ telemetry_config_paths.append(project_config_dir / TELEMETRY_CONFIG_FILE_NAME)+ telemetry_config_paths.append(project_config_dir / TELEMETRY_CONFIG_OVERRIDE_FILE_NAME)+ telemetry_config_toml_raw = load_toml_from_path_and_merge_with_overrides(paths=telemetry_config_paths)+ strip_reserved_meta(config_dict=telemetry_config_toml_raw)++ # Apply variable substitution to all string values (keep placeholders for missing vars)+ substitute_vars_with_provider = partial(substitute_vars, secrets_provider=secrets_provider, raise_on_missing_var=False)+ try:+ telemetry_config_toml = apply_to_strings_recursive(telemetry_config_toml_raw, transform_func=substitute_vars_with_provider)+ except UnknownVarPrefixError as exc:+ paths_str = "\n".join(str(path) for path in telemetry_config_paths)+ msg = f"Variable substitution failed in telemetry configuration based on '{paths_str}': {exc}"+ raise TelemetryConfigValidationError(msg) from exc++ try:+ telemetry_config = TelemetryConfig.model_validate(telemetry_config_toml)+ except ValidationError as exc:+ recovered = _telemetry_config_the_ledger_can_explain(+ paths=telemetry_config_paths,+ substitute_vars_with_provider=substitute_vars_with_provider,+ )+ if recovered is not None:+ return recovered+ # The ledger could not carry these files onto the current shape, so what is left to say is+ # *why*: the fields the model refused, and — when a scan of this surface still finds+ # something a `pipelex migrate` would do — the pending migration that partly explains it.+ # Reaching here is not the same as "the ledger has nothing to say": the retry also+ # declines when it applied real operations and the result still would not load.+ # The scan below replays the ledger a second time, deliberately: it walks the surface's+ # whole claim (`telemetry_*.toml`, not only the files this loader merged) and runs the+ # downgrade diagnosis the retry does not, so its plans are not the retry's plans. An+ # error path over a handful of small files is the right place to pay for that.+ report = report_validation_error(validation_error=exc, surface_id=TELEMETRY_CONFIG_SURFACE_ID)+ paths_str = "\n".join(str(path) for path in telemetry_config_paths)+ msg = f"Invalid telemetry configuration in '{paths_str}':\n{report.message}"+ raise TelemetryConfigValidationError(msg, migration=report.migration) from exc+ return telemetry_config+++def _telemetry_config_the_ledger_can_explain(+ *,+ paths: list[Path],+ substitute_vars_with_provider: Callable[[str], str],+) -> TelemetryConfig | None:+ """The same telemetry configuration with the ledger replayed over the user's files, or `None`.++ Boot tolerance for this surface, and it is the one that earns it today: `telemetry-config@2`+ carries the flat pre-`[custom_posthog]` file that a machine set up before that move still has,+ and such a file fails `extra="forbid"` in the field. Nothing is written — a boot warns and+ carries on, and `pipelex migrate` is what makes the change permanent.++ `None` covers both ways this declines: the ledger had nothing to say, or it did and the result+ still does not load. Substitution is re-run because it is a step of the load rather than a+ property of the files, and a placeholder it cannot read is as much a reason to decline as a+ model that refuses the shape.+ """+ replayed = replay_surface_files_in_memory(surface_id=TELEMETRY_CONFIG_SURFACE_ID, paths=paths)+ if replayed is None:+ return None+ try:+ substituted = apply_to_strings_recursive(replayed.config_dict, transform_func=substitute_vars_with_provider)+ telemetry_config = TelemetryConfig.model_validate(substituted)+ except (UnknownVarPrefixError, ValidationError):+ return None+ log.warning(stale_configuration_warning(plans=replayed.plans))+ return telemetry_configdiff --git a/subject_grants.toml b/subject_grants.tomlindex fbc75a982..094dabf53 100644--- a/subject_grants.toml+++ b/subject_grants.toml@@ -244,6 +244,10 @@ rationale = "Parses the config_arg; single operand."param = "result"rationale = "Verb-object: emits the result; format options stay keyword."+["pipelex/cli/agent_cli/commands/migrate_cmd.py::_render_markdown"]+param = "result"+rationale = "Positional-Callable protocol: passed as markdown_renderer, invoked renderer(result)."+["pipelex/cli/agent_cli/commands/pipe_cmd.py::_pipe_spec_to_toml"]param = "pipe_spec"rationale = "Serializes the pipe_spec to TOML; single operand."@@ -5452,14 +5456,6 @@ rationale = "Verb-object: transforms the input dict; enum class stays keyword."param = "input_list"rationale = "Verb-object: transforms the input list; enum class stays keyword."-["pipelex/system/configuration/configs.py::MigrationConfig.text_in_renaming_keys"]-param = "text"-rationale = "Predicate on the text; single operand."--["pipelex/system/configuration/configs.py::MigrationConfig.text_in_renaming_values"]-param = "text"-rationale = "Predicate on the text; single operand."-["pipelex/system/environment.py::all_env_vars_are_set"]param = "keys"rationale = "Predicate on the given env-var keys; single operand."diff --git a/tests/e2e/pipelex/cli/test_migrate_commands.py b/tests/e2e/pipelex/cli/test_migrate_commands.pynew file mode 100644index 000000000..a6b3f2545--- /dev/null+++ b/tests/e2e/pipelex/cli/test_migrate_commands.py@@ -0,0 +1,648 @@+"""End-to-end tests for ``pipelex migrate`` and ``pipelex-agent migrate``, via subprocess.++Both binaries live in one module because the *scenario* is one: a machine whose configuration+files predate the installed pipelex, a boot that fails because of it, and a command that repairs+it. Splitting the module would duplicate the only hard part — planting that machine — and the two+commands would then be proved against two subtly different ones.++**The old shape is real, and so is the migration.** The fixture is the package's own+``goldens/telemetry-config/[email protected]``, the hand-authored flat document that entry+``telemetry-config@2`` exists to carry forward, read live rather than copied here. The ledger that+migrates it is the one the package ships. Nothing about this test is synthetic except the machine+it runs on, which is what makes the state it starts from a real one: a flat ``telemetry.toml``+fails ``TelemetryConfig``'s ``extra="forbid"`` today, in the field.++**Boot tolerance is why that machine boots at all.** Such a machine used to die at boot; it now+starts, carries the file forward in memory, and warns a person that it did. So the boot probe here+no longer measures "broken, then fixed" — it asserts the machine boots on *both* sides of every+command, which is the tolerance property itself, while what the command changed is measured on the+files. The warning is not asserted here and cannot be: this probe is the agent CLI, which silences+logging process-wide by contract. It is asserted per surface in ``test_boot_tolerance.py``,+alongside the other half — a configuration the ledger cannot explain still fails the boot.++The boot probe is ``pipelex-agent models``: the cheapest command that performs a full Pipelex boot,+including the telemetry load the old shape sends down the tolerance path. ``pipelex show config``+is not a probe — it exits 0 on a machine whose telemetry configuration cannot load, because it+never reads it.+"""++from __future__ import annotations++import json+import subprocess # noqa: S404 - invokes the real pipelex binaries for E2E coverage+from typing import TYPE_CHECKING, Any++from pipelex.migration.backup import existing_backups_of+from pipelex.migration.goldens import pre_history_document_path+from pipelex.migration.ledger import packaged_migration_dir+from pipelex.tools.misc.toml_utils import load_toml_from_path+from tests.e2e.agent_cli.conftest import REPO_ROOT++if TYPE_CHECKING:+ from pathlib import Path++PIPELEX_BIN = REPO_ROOT / ".venv" / "bin" / "pipelex"+PIPELEX_AGENT_BIN = REPO_ROOT / ".venv" / "bin" / "pipelex-agent"++TELEMETRY_SURFACE_ID = "telemetry-config"++# A tier file of the flat era, as a user would have written one: a couple of overrides on top of+# the global file, not the whole census the golden carries.+OLD_SHAPE_PROJECT_OVERRIDE = """\+# My project keeps telemetry quiet.+telemetry_mode = "off"+host = "https://project.example.invalid"+"""+++def _old_shape_telemetry_document() -> str:+ """The flat pre-history document the shipped entry is about, read from the package."""+ path = pre_history_document_path(migration_dir=packaged_migration_dir(), surface_id=TELEMETRY_SURFACE_ID, schema_version=2)+ return path.read_text(encoding="utf-8")+++def _plant_a_stale_machine(*, hermetic_home: Path) -> tuple[Path, Path, Path]:+ """A machine with an old-shape file in the global directory and another in a project.++ Both directories are walked and both files are claimed — the global ``telemetry.toml`` by the+ surface's base file and the project ``telemetry_override.toml`` by its tier glob — so the run+ has to reach two directories and two tiers to be complete. The project override is also the+ tier the telemetry loader actually merges, which is what keeps the boot failure honest.++ Returns the project directory, the global file and the project file.+ """+ global_file = hermetic_home / ".pipelex" / "telemetry.toml"+ global_file.write_text(_old_shape_telemetry_document(), encoding="utf-8")++ project_dir = hermetic_home / "workspace"+ project_config_dir = project_dir / ".pipelex"+ project_config_dir.mkdir(parents=True)+ project_file = project_config_dir / "telemetry_override.toml"+ project_file.write_text(OLD_SHAPE_PROJECT_OVERRIDE, encoding="utf-8")+ return project_dir, global_file, project_file+++def _run(*, args: list[str], env: dict[str, str], cwd: Path, answers: str | None = None) -> subprocess.CompletedProcess[str]:+ # A wide terminal, because the human CLI renders through Rich and Rich wraps at the console+ # width. A temp-directory path is long enough to be broken across two lines at the default 80,+ # which would make these assertions fail for a reason that has nothing to do with migration.+ #+ # `answers` feeds an interactive command's prompts. Every confirmation the doctor asks defaults+ # to yes, so a run of bare newlines accepts each one in turn without this having to know how+ # many there will be or what order they come in.+ return subprocess.run( # noqa: S603+ args,+ env={**env, "COLUMNS": "400"},+ cwd=str(cwd),+ input=answers,+ capture_output=True,+ text=True,+ check=False,+ timeout=180,+ )+++def _boot(*, env: dict[str, str], cwd: Path) -> subprocess.CompletedProcess[str]:+ """Run a command that performs a full Pipelex boot, telemetry load included."""+ return _run(args=[str(PIPELEX_AGENT_BIN), "models", "--format", "json"], env=env, cwd=cwd)+++def _assert_boots(*, env: dict[str, str], cwd: Path) -> None:+ """Boot tolerance, through the real binaries: a machine carrying the old shape starts anyway.++ Asserted on both sides of every command here, which is deliberate — a before/after that only+ checked *after* would pass just as well if the file had never been stale.++ It does **not** assert the warning, and cannot: this probe is ``pipelex-agent``, which cuts+ Python's logging off process-wide as its first act so that nothing pollutes its two structured+ streams. That is the agent CLI's contract working, not a gap — the warning is where a person+ reads it, and it is asserted per surface in ``test_boot_tolerance.py``. What it does mean is+ that a *machine* consumer never learns of a pending migration from a boot, only by asking:+ ``pipelex-agent migrate --dry-run``, or the `doctor` row.+ """+ booted = _boot(env=env, cwd=cwd)+ assert booted.returncode == 0, booted.stderr+++class TestTheHumanMigrateCommand:+ def test_stale_files_in_both_directories_are_migrated_and_the_boot_stops_warning(+ self,+ hermetic_home: Path,+ offline_subprocess_env: dict[str, str],+ ) -> None:+ project_dir, global_file, project_file = _plant_a_stale_machine(hermetic_home=hermetic_home)++ _assert_boots(env=offline_subprocess_env, cwd=project_dir)++ migrated = _run(args=[str(PIPELEX_BIN), "--no-logo", "migrate", "--yes"], env=offline_subprocess_env, cwd=project_dir)++ assert migrated.returncode == 0, migrated.stdout + migrated.stderr+ # The human CLI's console writes to stderr when no runtime hub owns one, which is the case+ # for a command that deliberately does not boot. Read both, and assert on the union.+ report = migrated.stdout + migrated.stderr+ assert str(global_file) in report+ assert str(project_file) in report++ assert load_toml_from_path(path=global_file) == {+ "custom_posthog": {+ "mode": "anonymous",+ "endpoint": "https://eu.i.posthog.com",+ "api_key": "phc_example_project_api_key",+ "user_id": "",+ "geoip": True,+ "debug": False,+ "redact_properties": ["prompt", "system_prompt", "response", "file_path", "url"],+ }+ }+ assert load_toml_from_path(path=project_file) == {"custom_posthog": {"mode": "off", "endpoint": "https://project.example.invalid"}}++ # One copy per file, holding what that file used to say. Exactly one: a run that backed a+ # file up twice, or left an older copy beside the new one, would be leaving the user to+ # work out which is the original.+ for original, path in ((_old_shape_telemetry_document(), global_file), (OLD_SHAPE_PROJECT_OVERRIDE, project_file)):+ backups = existing_backups_of(path=path)+ assert len(backups) == 1, f"expected exactly one backup of {path}, found {backups}"+ assert backups[0].read_text(encoding="utf-8") == original++ _assert_boots(env=offline_subprocess_env, cwd=project_dir)++ def test_a_dry_run_reports_the_same_work_and_writes_nothing(+ self,+ hermetic_home: Path,+ offline_subprocess_env: dict[str, str],+ ) -> None:+ project_dir, global_file, _ = _plant_a_stale_machine(hermetic_home=hermetic_home)+ original = global_file.read_text(encoding="utf-8")++ planned = _run(args=[str(PIPELEX_BIN), "--no-logo", "migrate", "--dry-run"], env=offline_subprocess_env, cwd=project_dir)++ assert planned.returncode == 0, planned.stdout + planned.stderr+ assert "telemetry-config@2" in planned.stdout + planned.stderr or "Nest the flat telemetry" in planned.stdout + planned.stderr+ assert global_file.read_text(encoding="utf-8") == original+ assert existing_backups_of(path=global_file) == []++ def test_the_two_write_flags_contradict_each_other_and_are_refused(+ self,+ hermetic_home: Path, # noqa: ARG002 - the fixture is what makes HOME hermetic for the subprocess+ offline_subprocess_env: dict[str, str],+ ) -> None:+ refused = _run(+ args=[str(PIPELEX_BIN), "--no-logo", "migrate", "--dry-run", "--yes"],+ env=offline_subprocess_env,+ cwd=REPO_ROOT,+ )++ assert refused.returncode == 2+ assert "contradict" in refused.stdout + refused.stderr+++class TestTheAgentMigrateLoop:+ def test_plan_as_json_then_apply_and_the_boot_stops_warning(+ self,+ hermetic_home: Path,+ offline_subprocess_env: dict[str, str],+ ) -> None:+ project_dir, global_file, project_file = _plant_a_stale_machine(hermetic_home=hermetic_home)++ _assert_boots(env=offline_subprocess_env, cwd=project_dir)++ planned = _run(+ args=[str(PIPELEX_AGENT_BIN), "migrate", "--dry-run", "--format", "json"],+ env=offline_subprocess_env,+ cwd=project_dir,+ )+ assert planned.returncode == 0, planned.stderr+ plan: dict[str, Any] = json.loads(planned.stdout)++ assert plan["applied"] is False+ assert plan["needs_attention"] is False+ assert plan["is_clean"] is False+ assert plan["summary"]["files_changed"] == 2+ assert plan["summary"]["files_written"] == 0+ assert existing_backups_of(path=global_file) == [], "a plan is not a write"+ assert {str(hermetic_home / ".pipelex"), str(project_dir / ".pipelex")} == set(plan["config_dirs"])++ applied = _run(args=[str(PIPELEX_AGENT_BIN), "migrate", "--yes", "--format", "json"], env=offline_subprocess_env, cwd=project_dir)+ assert applied.returncode == 0, applied.stderr+ outcome: dict[str, Any] = json.loads(applied.stdout)++ assert outcome["applied"] is True+ assert outcome["summary"]["files_written"] == 2+ written = {plan_dict["file_path"] for plan_dict in outcome["plans"] if plan_dict["was_written"]}+ assert written == {str(global_file), str(project_file)}++ _assert_boots(env=offline_subprocess_env, cwd=project_dir)++ def test_a_second_run_finds_nothing_left_to_do(+ self,+ hermetic_home: Path,+ offline_subprocess_env: dict[str, str],+ ) -> None:+ """Replay neutrality, at the command's own level: migrating twice is migrating once."""+ project_dir, _, _ = _plant_a_stale_machine(hermetic_home=hermetic_home)++ first = _run(args=[str(PIPELEX_AGENT_BIN), "migrate", "--yes", "--format", "json"], env=offline_subprocess_env, cwd=project_dir)+ assert first.returncode == 0, first.stderr++ second = _run(args=[str(PIPELEX_AGENT_BIN), "migrate", "--yes", "--format", "json"], env=offline_subprocess_env, cwd=project_dir)+ assert second.returncode == 0, second.stderr+ outcome: dict[str, Any] = json.loads(second.stdout)+ assert outcome["is_clean"] is True+ assert outcome["summary"]["files_written"] == 0++ def test_the_two_write_flags_contradict_each_other_and_are_refused(+ self,+ hermetic_home: Path, # noqa: ARG002 - the fixture is what makes HOME hermetic for the subprocess+ offline_subprocess_env: dict[str, str],+ ) -> None:+ refused = _run(+ args=[str(PIPELEX_AGENT_BIN), "migrate", "--dry-run", "--yes", "--format", "json"],+ env=offline_subprocess_env,+ cwd=REPO_ROOT,+ )++ assert refused.returncode == 2+ assert json.loads(refused.stderr)["error_type"] == "ArgumentError"+++class TestTheBootstrapPath:+ """The property that makes this command reachable at all: it runs when nothing else does.++ A broken configuration is the reason to reach for `migrate`, so needing a working one would+ make it useless in exactly the case it exists for. What it may use is the migration ledger, the+ applier and the filesystem — and this test is what keeps a future import of the config, the+ model deck or the network from creeping into that list unnoticed.+ """++ def _break_the_configuration(self, *, hermetic_home: Path) -> None:+ config_file = hermetic_home / ".pipelex" / "pipelex.toml"+ # At the top of the document, so the key lands at the root rather than in whichever table+ # happens to be last — a root key no model knows, which no ledger entry removes either.+ config_file.write_text(f"not_a_real_setting = true\n{config_file.read_text(encoding='utf-8')}", encoding="utf-8")++ def test_both_commands_run_against_a_configuration_that_cannot_load(+ self,+ hermetic_home: Path,+ offline_subprocess_env: dict[str, str],+ ) -> None:+ self._break_the_configuration(hermetic_home=hermetic_home)++ boot = _boot(env=offline_subprocess_env, cwd=hermetic_home)+ assert boot.returncode != 0, "the scenario is worthless if this machine boots"+ # `PipelexConfigError` rather than the raw `ConfigValidationError` this used to report:+ # the boot's own arm now catches both shapes a refusal takes and translates. The class is+ # what carries `error_domain: "config"` and the `migration` block to a machine consumer.+ assert "PipelexConfigError" in boot.stderr++ # Both commands answer with a *report* rather than a crash, and that — not the exit code —+ # is what "it runs" means here. Each leaves a 1 behind, because the key planted above is+ # one the downgrade diagnosis has something to say about; the test below is what says so.+ human = _run(args=[str(PIPELEX_BIN), "--no-logo", "migrate", "--dry-run"], env=offline_subprocess_env, cwd=hermetic_home)+ assert human.returncode == 1, human.stdout + human.stderr+ assert str(hermetic_home / ".pipelex") in human.stdout + human.stderr++ agent = _run(+ args=[str(PIPELEX_AGENT_BIN), "migrate", "--dry-run", "--format", "json"],+ env=offline_subprocess_env,+ cwd=hermetic_home,+ )+ assert agent.returncode == 1, agent.stderr+ assert json.loads(agent.stdout)["summary"]["files_walked"] > 0++ def test_the_root_key_no_ledger_removes_is_named_rather_than_passed_over(+ self,+ hermetic_home: Path,+ offline_subprocess_env: dict[str, str],+ ) -> None:+ """The downgrade diagnosis, against the very file that broke the boot.++ A replay finds nothing to do here — no operation's source is present — so without the+ diagnosis the command would report this machine clean beside a boot that will not start,+ which is the exact failure the migration project exists to remove.+ """+ self._break_the_configuration(hermetic_home=hermetic_home)++ planned = _run(+ args=[str(PIPELEX_AGENT_BIN), "migrate", "--dry-run", "--format", "json"],+ env=offline_subprocess_env,+ cwd=hermetic_home,+ )++ assert planned.returncode == 1, "an unexplained path is a person's to resolve, so the run needs attention"+ report: dict[str, Any] = json.loads(planned.stdout)+ assert report["needs_attention"] is True+ assert report["summary"]["unexplained_paths"] == 1+ unexplained = [found for plan in report["plans"] if plan["file_path"].endswith("pipelex.toml") for found in plan["unexplained"]]+ assert [found["path"] for found in unexplained] == ["not_a_real_setting"]+ assert "newer pipelex" in unexplained[0]["note"]++ def test_the_ledger_still_migrates_a_stale_file_beside_a_configuration_that_cannot_load(+ self,+ hermetic_home: Path,+ offline_subprocess_env: dict[str, str],+ ) -> None:+ """The bootstrap path is not merely "does not crash" — it still does the work.++ A command that ran but declined to touch anything while the configuration was broken would+ pass the test above and be useless.+ """+ self._break_the_configuration(hermetic_home=hermetic_home)+ _, global_file, _ = _plant_a_stale_machine(hermetic_home=hermetic_home)++ applied = _run(+ args=[str(PIPELEX_AGENT_BIN), "migrate", "--yes", "--format", "json"],+ env=offline_subprocess_env,+ cwd=hermetic_home,+ )++ # A 1, and the work still done: the broken `pipelex.toml` beside it is a file the diagnosis+ # has something to say about, and one file needing a human never stops another being+ # migrated. That is the per-file scope, read at the level of the whole command.+ assert applied.returncode == 1, applied.stderr+ outcome: dict[str, Any] = json.loads(applied.stdout)+ assert outcome["summary"]["files_written"] == 1+ assert outcome["summary"]["unexplained_paths"] == 1+ assert "custom_posthog" in global_file.read_text(encoding="utf-8")+++class TestNoValueFromAUsersFileIsEverRendered:+ """The contract's mechanical rule, asserted on the channels that exist today.++ > No value read from a user's file is ever rendered — not in the command's output, not in the+ > structured plan, not in an error.++ Mechanical rather than a list of credential-shaped key names, because such a list is a guess+ that eventually misses one. The planted value is carried by a `move_key` the shipped ledger+ performs, so the run demonstrably *handled* it — a rule that held only because the value was+ quietly dropped would be no rule at all.++ Two of the three channels are covered here: the command's own output and the structured plan.+ The third — the `migration` block on a configuration validation error — is+ `TestABootFailureCarriesThePendingMigration` below, which needs a different specimen: boot+ tolerance means a file the ledger *can* explain never reaches an error surface at all.+ """++ PLANTED_SECRET = "phc_L1VE_s3cret_project_key_that_must_never_be_rendered"++ def _plant_the_secret(self, *, hermetic_home: Path) -> tuple[Path, Path]:+ document = _old_shape_telemetry_document().replace("phc_example_project_api_key", self.PLANTED_SECRET)+ assert self.PLANTED_SECRET in document, "the golden no longer carries the api key this test substitutes into"+ project_dir, global_file, _ = _plant_a_stale_machine(hermetic_home=hermetic_home)+ global_file.write_text(document, encoding="utf-8")+ return project_dir, global_file++ def test_neither_command_renders_a_planted_secret_while_migrating_it(+ self,+ hermetic_home: Path,+ offline_subprocess_env: dict[str, str],+ ) -> None:+ project_dir, global_file = self._plant_the_secret(hermetic_home=hermetic_home)++ channels = [+ _run(args=[str(PIPELEX_BIN), "--no-logo", "migrate", "--dry-run"], env=offline_subprocess_env, cwd=project_dir),+ _run(+ args=[str(PIPELEX_AGENT_BIN), "migrate", "--dry-run", "--format", "json"],+ env=offline_subprocess_env,+ cwd=project_dir,+ ),+ _run(+ args=[str(PIPELEX_AGENT_BIN), "migrate", "--dry-run", "--format", "markdown"],+ env=offline_subprocess_env,+ cwd=project_dir,+ ),+ _run(args=[str(PIPELEX_BIN), "--no-logo", "migrate", "--yes"], env=offline_subprocess_env, cwd=project_dir),+ ]+ for channel in channels:+ assert self.PLANTED_SECRET not in channel.stdout+ assert self.PLANTED_SECRET not in channel.stderr++ # And it was moved rather than dropped, which is what makes the assertions above mean+ # something: the run read this value, wrote it under its new key, and never said it.+ assert self.PLANTED_SECRET in global_file.read_text(encoding="utf-8")+ assert load_toml_from_path(path=global_file)["custom_posthog"]["api_key"] == self.PLANTED_SECRET+++class TestABootFailureCarriesThePendingMigration:+ """The third channel: a configuration that will not load says *why*, in fields and in prose.++ Boot tolerance already handles the case the ledger can explain — that machine boots, with a+ warning. What is left is the machine the ledger cannot explain, and until now all it got was+ pydantic's account of a key it had never heard of. It now also gets the scan: which of its+ files the walk would touch, what a `pipelex migrate` would carry forward, and what nobody but+ a person can resolve.++ **The structured block is the contract; the prose is presentation.** A machine consumer+ branches on `migration` being present, never on the wording — which is why both are asserted+ here, on the same run, through the real binary.+ """++ PLANTED_SECRET = "phc_L1VE_boot_error_key_that_must_never_be_rendered"++ def _break_the_configuration_with_a_secret(self, *, hermetic_home: Path) -> Path:+ """A root key no model knows, holding a value that looks exactly like a live credential.++ The unknown key is what breaks the boot *and* what the downgrade diagnosis reports; its+ value is what must never appear. Both halves in one key is deliberate — it means the+ diagnosis demonstrably read the value it declines to render.+ """+ config_file = hermetic_home / ".pipelex" / "pipelex.toml"+ planted = f'posthog_project_key = "{self.PLANTED_SECRET}"\n'+ config_file.write_text(planted + config_file.read_text(encoding="utf-8"), encoding="utf-8")+ return config_file++ def test_the_agent_boot_error_carries_the_config_domain_and_the_block(+ self,+ hermetic_home: Path,+ offline_subprocess_env: dict[str, str],+ ) -> None:+ self._break_the_configuration_with_a_secret(hermetic_home=hermetic_home)++ boot = _boot(env=offline_subprocess_env, cwd=hermetic_home)++ assert boot.returncode != 0, "the scenario is worthless if this machine boots"+ # One envelope, and it has to be said out loud: `agent_error` leaves through `typer.Exit`,+ # which is a `RuntimeError` rather than a `SystemExit`, so a command boundary re-raising+ # only `SystemExit` caught it again and printed a second document after the first. Two+ # JSON documents on one stream is not JSON, and the block below would be unreadable.+ assert boot.stderr.count('"error": true') == 1, boot.stderr+ envelope: dict[str, Any] = json.loads(boot.stderr)+ assert envelope["error_type"] == "PipelexConfigError"+ assert envelope["error_domain"] == "config", "a new domain would route the agent hooks to BLOCK"++ block: dict[str, Any] = envelope["migration"]+ assert block["remedy"] == "pipelex migrate"+ assert block["needs_attention"] is True, "a key no entry explains is a person's to resolve"+ unexplained = [found for plan in block["plans"] for found in plan["unexplained"]]+ assert [found["path"] for found in unexplained] == ["posthog_project_key"]+ assert [plan["file_path"] for plan in block["plans"]] == [str(hermetic_home / ".pipelex" / "pipelex.toml")]++ def test_no_value_from_the_users_file_reaches_either_stream(+ self,+ hermetic_home: Path,+ offline_subprocess_env: dict[str, str],+ ) -> None:+ """The contract's mechanical rule, on the channel that was waiting for this milestone.++ The diagnosis walked this document and reached the very key whose value is planted here —+ it names the key in the block above — so a rule that held only because nothing had read+ the value would be no rule at all.+ """+ config_file = self._break_the_configuration_with_a_secret(hermetic_home=hermetic_home)++ boot = _boot(env=offline_subprocess_env, cwd=hermetic_home)++ assert self.PLANTED_SECRET not in boot.stdout+ assert self.PLANTED_SECRET not in boot.stderr+ assert self.PLANTED_SECRET in config_file.read_text(encoding="utf-8"), "nothing was written, so the value is still there"++ def test_a_machine_the_ledger_can_explain_boots_instead_of_reporting(+ self,+ hermetic_home: Path,+ offline_subprocess_env: dict[str, str],+ ) -> None:+ """The block's absence is a verdict too, and boot tolerance is why it is reachable.++ A stale file the ledger carries forward never reaches this error surface at all — it warns+ and boots. Without this test the block would look like the answer to staleness, when it is+ actually the answer to staleness the ledger *cannot* resolve.+ """+ project_dir, _, _ = _plant_a_stale_machine(hermetic_home=hermetic_home)++ boot = _boot(env=offline_subprocess_env, cwd=project_dir)++ assert boot.returncode == 0, boot.stderr+ assert "migration" not in boot.stderr+++class TestAStaleTelemetryFileIsMigratedNotReset:+ """Every surface that reports a stale `telemetry.toml` names the migration, not a fresh file.++ Until now each of them held its own hardcoded remedy — *config format has changed, run+ `pipelex init telemetry`* — written for exactly the shape the shipped ledger entry exists to+ carry forward. That command writes a new file: the PostHog key, the Langfuse credentials and+ the OTLP exporters a real machine has would all be gone.++ Two different machines are needed here, and the difference is boot tolerance. The `doctor`+ row is a filesystem probe, so it reports a file the ledger *can* explain — that machine boots+ fine, and the doctor is what says the change is not permanent yet. The error envelope needs a+ file that is old *and* wrong, because anything the ledger fully explains never reaches an+ error at all.+ """++ def test_the_agent_doctor_reports_the_finding_and_the_migrate_remedy(+ self,+ hermetic_home: Path,+ offline_subprocess_env: dict[str, str],+ ) -> None:+ project_dir, _, _ = _plant_a_stale_machine(hermetic_home=hermetic_home)++ checked = _run(args=[str(PIPELEX_AGENT_BIN), "doctor", "--format", "json"], env=offline_subprocess_env, cwd=project_dir)++ report: dict[str, Any] = json.loads(checked.stdout)+ telemetry: dict[str, Any] = report["checks"]["telemetry"]+ assert telemetry["healthy"] is False+ assert telemetry["finding"] == "out_of_date"+ actions: list[str] = report["recommended_actions"]+ assert any("pipelex migrate" in action for action in actions)+ assert not any("init telemetry" in action for action in actions), "this file is old, not broken"++ def test_a_telemetry_boot_failure_carries_the_block_in_one_envelope(+ self,+ hermetic_home: Path,+ offline_subprocess_env: dict[str, str],+ ) -> None:+ """The other configuration surface that raises its own error type, on the same contract.++ Boot tolerance means a telemetry file has to be old *and* wrong to get here — flat, and+ carrying a key no telemetry schema ever had — which is exactly the machine the old+ remedy was worst for: it is migratable, and `pipelex init telemetry` would have answered+ it by writing a fresh file over the settings the migration keeps.+ """+ _, global_file, _ = _plant_a_stale_machine(hermetic_home=hermetic_home)+ global_file.write_text(f"{_old_shape_telemetry_document()}\nnot_a_telemetry_setting = true\n", encoding="utf-8")++ boot = _boot(env=offline_subprocess_env, cwd=hermetic_home)++ assert boot.returncode != 0, "a file the ledger cannot fully explain still fails the boot"+ assert boot.stderr.count('"error": true') == 1, boot.stderr+ envelope: dict[str, Any] = json.loads(boot.stderr)+ assert envelope["error_type"] == "TelemetryConfigValidationError"+ assert envelope["error_domain"] == "config", "it comes from the class now, not the lookup table"+ assert envelope["migration"]["remedy"] == "pipelex migrate"+ assert "init telemetry" not in envelope["hint"]++ def test_the_human_doctor_says_migrate_and_never_offers_to_start_the_file_over(+ self,+ hermetic_home: Path,+ offline_subprocess_env: dict[str, str],+ ) -> None:+ project_dir, _, _ = _plant_a_stale_machine(hermetic_home=hermetic_home)++ checked = _run(args=[str(PIPELEX_BIN), "--no-logo", "doctor"], env=offline_subprocess_env, cwd=project_dir)++ # Both streams: the human doctor renders through a Rich console whose target the user's+ # own log configuration decides, so which one it lands on is not this test's business.+ printed = checked.stdout + checked.stderr+ assert "out of date" in printed, printed+ assert "pipelex migrate" in printed+ assert "pipelex init telemetry" not in printed+++class TestTheDoctorReportsAndRepairsAStaleMachine:+ """The row that tells a machine a migration is pending, and the fix mode that runs it.++ A boot never says this: a stale configuration the ledger can explain boots with a warning, and+ `pipelex-agent` cuts logging off process-wide as its first act so nothing can emit one. Asking+ is the only channel a machine has, which is what makes this row load-bearing rather than+ convenient.+ """++ def test_the_agent_doctor_names_every_file_a_migration_would_touch_and_writes_nothing(+ self,+ hermetic_home: Path,+ offline_subprocess_env: dict[str, str],+ ) -> None:+ """Both directories, because both are what the command it names would rewrite.++ The row is scoped to no directory at all, unlike every other check here — `pipelex migrate`+ has no `--global`. And it is a report: the files it names are exactly as it found them.+ """+ project_dir, global_file, project_file = _plant_a_stale_machine(hermetic_home=hermetic_home)+ before = {path: path.read_text(encoding="utf-8") for path in (global_file, project_file)}++ checked = _run(args=[str(PIPELEX_AGENT_BIN), "doctor", "--format", "json"], env=offline_subprocess_env, cwd=project_dir)++ report: dict[str, Any] = json.loads(checked.stdout)+ row: dict[str, Any] = report["checks"]["pending_migrations"]+ assert row["healthy"] is False+ assert row["finding"] == "pending"+ assert sorted(row["migratable_files"]) == sorted([str(global_file), str(project_file)])+ assert any("pipelex migrate" in action for action in report["recommended_actions"])+ for path, text in before.items():+ assert path.read_text(encoding="utf-8") == text, "a health report writes nothing"+ assert existing_backups_of(path=path) == []++ def test_fix_mode_runs_the_migration_and_keeps_what_was_in_the_files(+ self,+ hermetic_home: Path,+ offline_subprocess_env: dict[str, str],+ ) -> None:+ """`--fix` offers the command rather than merely naming it, and it is the same command.++ Every confirmation the doctor asks defaults to yes, so bare newlines accept whatever it+ offers — which is deliberate here: the assertion is about the files, and it holds however+ many other fixes this machine happens to be offered alongside the migration.+ """+ project_dir, global_file, project_file = _plant_a_stale_machine(hermetic_home=hermetic_home)++ _run(args=[str(PIPELEX_BIN), "--no-logo", "doctor", "--fix"], env=offline_subprocess_env, cwd=project_dir, answers="\n" * 8)++ for path in (global_file, project_file):+ migrated = load_toml_from_path(path)+ assert "custom_posthog" in migrated, f"{path} was not migrated"+ assert len(existing_backups_of(path=path)) == 1, "the original is kept beside it"+ # The settings the old destructive remedy would have discarded are still there.+ assert load_toml_from_path(project_file)["custom_posthog"]["endpoint"] == "https://project.example.invalid"++ checked = _run(args=[str(PIPELEX_AGENT_BIN), "doctor", "--format", "json"], env=offline_subprocess_env, cwd=project_dir)+ assert json.loads(checked.stdout)["checks"]["pending_migrations"]["finding"] == "up_to_date"diff --git a/tests/unit/pipelex/cli/test_agent_doctor_cmd.py b/tests/unit/pipelex/cli/test_agent_doctor_cmd.pyindex cd79df817..c2a08abc1 100644--- a/tests/unit/pipelex/cli/test_agent_doctor_cmd.py+++ b/tests/unit/pipelex/cli/test_agent_doctor_cmd.py@@ -14,9 +14,21 @@ if TYPE_CHECKING:from pipelex.cli.agent_cli.commands.agent_cli_factory import AGENT_CLI_STDERR_LOG_FIELDSfrom pipelex.cli.agent_cli.commands.agent_output import CliOutputFormatfrom pipelex.cli.agent_cli.commands.doctor_cmd import agent_doctor_cmd+from pipelex.cli.commands.doctor_cmd import (+ PendingMigrationsCheck,+ PendingMigrationsFinding,+ TelemetryConfigCheck,+ TelemetryConfigFinding,+)from pipelex.cogt.model_backends.backend_library import BackendCredentialsReport+from pipelex.core.validation import MIGRATE_COMMANDfrom pipelex.system.console_target import ConsoleTarget+NO_PENDING_MIGRATIONS = PendingMigrationsCheck(+ finding=PendingMigrationsFinding.UP_TO_DATE,+ message="Every configuration file is at the current schema",+)+class TestAgentDoctorCmd:"""Tests for agent_doctor_cmd JSON output."""@@ -37,6 +49,14 @@ class TestAgentDoctorCmd:mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.setup_doctor_runtime")mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.apply_agent_cli_output_discipline")mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.silence_logging_for_agent_cli")+ # And the migration row, for a fourth reason: unlike every other check it takes no+ # directory, so it reads this machine's own `~/.pipelex/` and project `.pipelex/`. Tests+ # about the output shape must not depend on whose laptop the suite is running on; the+ # ones that are about this row re-patch it with what they need.+ mocker.patch(+ "pipelex.cli.agent_cli.commands.doctor_cmd.check_pending_migrations",+ return_value=NO_PENDING_MIGRATIONS,+ )def test_healthy_output_json(self, mocker: MockerFixture, capsys: pytest.CaptureFixture[str]) -> None:"""Healthy checks should produce JSON with all_healthy=true and no recommended_actions."""@@ -46,7 +66,7 @@ class TestAgentDoctorCmd:)mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.check_telemetry_config",- return_value=(True, "Telemetry configured"),+ return_value=TelemetryConfigCheck(finding=TelemetryConfigFinding.HEALTHY, message="Telemetry configured"),)mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.check_backend_credentials",@@ -64,15 +84,34 @@ class TestAgentDoctorCmd:assert parsed["all_healthy"] is Trueassert "recommended_actions" not in parsed- def test_unhealthy_includes_recommended_actions(self, mocker: MockerFixture, capsys: pytest.CaptureFixture[str]) -> None:- """Unhealthy telemetry should produce recommended_actions in the JSON output."""+ @pytest.mark.parametrize(+ ("finding", "expected_action"),+ [+ (TelemetryConfigFinding.NOT_FOUND, "pipelex init telemetry"),+ (TelemetryConfigFinding.OUT_OF_DATE, MIGRATE_COMMAND),+ (TelemetryConfigFinding.INVALID, "Fix"),+ (TelemetryConfigFinding.UNPARSEABLE, "Fix"),+ ],+ )+ def test_each_telemetry_finding_gets_its_own_recommended_action(+ self,+ finding: TelemetryConfigFinding,+ expected_action: str,+ mocker: MockerFixture,+ capsys: pytest.CaptureFixture[str],+ ) -> None:+ """One remedy per finding, and the finding itself rides the envelope for an agent to branch on.++ All four used to be answered with `pipelex init telemetry`, which on the out-of-date one+ writes a fresh file over the settings a migration would have kept.+ """mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.check_config_files",return_value=(True, 0, "All config files present"),)mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.check_telemetry_config",- return_value=(False, "Config format has changed - run 'pipelex init telemetry' to update"),+ return_value=TelemetryConfigCheck(finding=finding, message="something is up with telemetry"),)mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.check_backend_credentials",@@ -88,11 +127,165 @@ class TestAgentDoctorCmd:parsed = json.loads(capsys.readouterr().out)assert parsed["all_healthy"] is Falseassert parsed["checks"]["telemetry"]["healthy"] is False- assert "recommended_actions" in parsed- assert any("pipelex init telemetry" in action for action in parsed["recommended_actions"])+ assert parsed["checks"]["telemetry"]["finding"] == finding+ actions = parsed["recommended_actions"]+ assert any(expected_action in action for action in actions)# Rich markup should be stripped from the messageassert "[cyan]" not in parsed["checks"]["telemetry"]["message"]+ def test_an_out_of_date_telemetry_file_is_never_told_to_start_over(+ self,+ mocker: MockerFixture,+ capsys: pytest.CaptureFixture[str],+ ) -> None:+ """The destructive half of the old remedy, pinned absent on the finding it was worst for."""+ mocker.patch(+ "pipelex.cli.agent_cli.commands.doctor_cmd.check_config_files",+ return_value=(True, 0, "All config files present"),+ )+ mocker.patch(+ "pipelex.cli.agent_cli.commands.doctor_cmd.check_telemetry_config",+ return_value=TelemetryConfigCheck(finding=TelemetryConfigFinding.OUT_OF_DATE, message="out of date"),+ )+ mocker.patch(+ "pipelex.cli.agent_cli.commands.doctor_cmd.check_backend_credentials",+ return_value=(True, {}, "All backends healthy"),+ )+ mocker.patch(+ "pipelex.cli.agent_cli.commands.doctor_cmd.check_models",+ return_value=(True, "Models valid", {}),+ )++ agent_doctor_cmd(output_format=CliOutputFormat.JSON)++ parsed = json.loads(capsys.readouterr().out)+ assert not any("init telemetry" in action for action in parsed["recommended_actions"])++ def _stub_every_other_check_healthy(self, mocker: MockerFixture) -> None:+ """Everything but the migration row reporting healthy — the row under test speaks alone."""+ mocker.patch(+ "pipelex.cli.agent_cli.commands.doctor_cmd.check_config_files",+ return_value=(True, 0, "All config files present"),+ )+ mocker.patch(+ "pipelex.cli.agent_cli.commands.doctor_cmd.check_telemetry_config",+ return_value=TelemetryConfigCheck(finding=TelemetryConfigFinding.HEALTHY, message="Telemetry configured"),+ )+ mocker.patch(+ "pipelex.cli.agent_cli.commands.doctor_cmd.check_backend_credentials",+ return_value=(True, {}, "All backends healthy"),+ )+ mocker.patch(+ "pipelex.cli.agent_cli.commands.doctor_cmd.check_models",+ return_value=(True, "Models valid", {}),+ )++ def test_a_pending_migration_rides_the_envelope_with_the_files_it_is_about(+ self,+ mocker: MockerFixture,+ capsys: pytest.CaptureFixture[str],+ ) -> None:+ """The only channel a machine has for this: a boot warns a person and says nothing here.++ Both halves are reported — the run is worth starting *and* something is still owed after+ it — because an agent that only heard the first would stop with a broken file in place.+ """+ self._stub_every_other_check_healthy(mocker)+ mocker.patch(+ "pipelex.cli.agent_cli.commands.doctor_cmd.check_pending_migrations",+ return_value=PendingMigrationsCheck(+ finding=PendingMigrationsFinding.PENDING,+ message="1 configuration file(s) can be brought up to date",+ migratable_files=["/home/user/.pipelex/telemetry.toml"],+ attention_files=["/work/project/.pipelex/pipelex.toml"],+ ),+ )++ agent_doctor_cmd(output_format=CliOutputFormat.JSON)++ parsed = json.loads(capsys.readouterr().out)+ assert parsed["all_healthy"] is False+ row = parsed["checks"]["pending_migrations"]+ assert row["healthy"] is False+ assert row["finding"] == "pending"+ assert row["migratable_files"] == ["/home/user/.pipelex/telemetry.toml"]+ assert row["attention_files"] == ["/work/project/.pipelex/pipelex.toml"]+ actions: list[str] = parsed["recommended_actions"]+ assert any(action.startswith(f"Run '{MIGRATE_COMMAND}' ") for action in actions)+ assert any(f"'{MIGRATE_COMMAND} --dry-run'" in action for action in actions)++ def test_an_unreadable_row_is_reported_as_unchecked_rather_than_healthy(+ self,+ mocker: MockerFixture,+ capsys: pytest.CaptureFixture[str],+ ) -> None:+ """A packaging problem of ours is not a verdict about the user's files."""+ self._stub_every_other_check_healthy(mocker)+ mocker.patch(+ "pipelex.cli.agent_cli.commands.doctor_cmd.check_pending_migrations",+ return_value=PendingMigrationsCheck(+ finding=PendingMigrationsFinding.UNAVAILABLE,+ message="Could not check for pending migrations: the packaged ledger will not load",+ ),+ )++ agent_doctor_cmd(output_format=CliOutputFormat.JSON)++ parsed = json.loads(capsys.readouterr().out)+ assert parsed["all_healthy"] is False+ assert parsed["checks"]["pending_migrations"]["finding"] == "unavailable"+ assert any("could not" in action.lower() for action in parsed["recommended_actions"])++ def test_the_same_command_is_not_recommended_twice_for_the_same_file(+ self,+ mocker: MockerFixture,+ capsys: pytest.CaptureFixture[str],+ ) -> None:+ """The machine-wide row supersedes the telemetry-only one: it names the same command."""+ self._stub_every_other_check_healthy(mocker)+ mocker.patch(+ "pipelex.cli.agent_cli.commands.doctor_cmd.check_telemetry_config",+ return_value=TelemetryConfigCheck(finding=TelemetryConfigFinding.OUT_OF_DATE, message="out of date"),+ )+ mocker.patch(+ "pipelex.cli.agent_cli.commands.doctor_cmd.check_pending_migrations",+ return_value=PendingMigrationsCheck(+ finding=PendingMigrationsFinding.PENDING,+ message="1 configuration file(s) can be brought up to date",+ migratable_files=["/home/user/.pipelex/telemetry.toml"],+ ),+ )++ agent_doctor_cmd(output_format=CliOutputFormat.JSON)++ actions: list[str] = json.loads(capsys.readouterr().out)["recommended_actions"]+ assert len(actions) == 1+ assert "telemetry.toml up to date" not in actions[0]++ def test_the_migration_row_is_not_scoped_by_global(+ self,+ mocker: MockerFixture,+ capsys: pytest.CaptureFixture[str],+ ) -> None:+ """`--global` scopes the checks that report on a *file*; this one reports on a *command*.++ `pipelex migrate` has no `--global` and walks both configuration directories, so a row+ narrowed to one of them would name a command that rewrites a file it never mentioned.+ """+ self._stub_every_other_check_healthy(mocker)+ migrations = mocker.patch(+ "pipelex.cli.agent_cli.commands.doctor_cmd.check_pending_migrations",+ return_value=PendingMigrationsCheck(+ finding=PendingMigrationsFinding.UP_TO_DATE,+ message="Every configuration file is at the current schema",+ ),+ )++ agent_doctor_cmd(global_=True, output_format=CliOutputFormat.JSON)++ capsys.readouterr()+ migrations.assert_called_once_with()+def test_backend_details_in_output(self, mocker: MockerFixture, capsys: pytest.CaptureFixture[str]) -> None:"""Backend credential reports should appear as structured data in the JSON output."""credential_reports = {@@ -110,7 +303,7 @@ class TestAgentDoctorCmd:)mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.check_telemetry_config",- return_value=(True, "OK"),+ return_value=TelemetryConfigCheck(finding=TelemetryConfigFinding.HEALTHY, message="OK"),)mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.check_backend_credentials",@@ -167,7 +360,7 @@ class TestAgentDoctorCmd:)mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.check_telemetry_config",- return_value=(True, "OK"),+ return_value=TelemetryConfigCheck(finding=TelemetryConfigFinding.HEALTHY, message="OK"),)mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.check_backend_credentials",@@ -218,7 +411,7 @@ class TestAgentDoctorCmd:)mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.check_telemetry_config",- return_value=(True, "Telemetry configured"),+ return_value=TelemetryConfigCheck(finding=TelemetryConfigFinding.HEALTHY, message="Telemetry configured"),)mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.check_backend_credentials",@@ -263,7 +456,7 @@ class TestAgentDoctorCmd:)mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.check_telemetry_config",- return_value=(True, "OK"),+ return_value=TelemetryConfigCheck(finding=TelemetryConfigFinding.HEALTHY, message="OK"),)mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.check_backend_credentials",@@ -314,7 +507,7 @@ class TestAgentDoctorCmd:)mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.check_telemetry_config",- return_value=(True, "OK"),+ return_value=TelemetryConfigCheck(finding=TelemetryConfigFinding.HEALTHY, message="OK"),)mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.check_backend_credentials",diff --git a/tests/unit/pipelex/cli/test_doctor_cmd.py b/tests/unit/pipelex/cli/test_doctor_cmd.pyindex 529ba0dc7..e3f9a98c6 100644--- a/tests/unit/pipelex/cli/test_doctor_cmd.py+++ b/tests/unit/pipelex/cli/test_doctor_cmd.py@@ -12,6 +12,10 @@ if TYPE_CHECKING:from pytest_mock import MockerFixturefrom pipelex.cli.commands.doctor_cmd import (+ PendingMigrationsCheck,+ PendingMigrationsFinding,+ TelemetryConfigCheck,+ TelemetryConfigFinding,check_backend_credentials,check_telemetry_config,do_doctor_cmd,@@ -24,6 +28,13 @@ from pipelex.system.telemetry.telemetry_config import TELEMETRY_CONFIG_FILE_NAMETELEMETRY_OFF = '[custom_posthog]\nmode = "off"\n'TELEMETRY_ANONYMOUS = '[custom_posthog]\nmode = "anonymous"\n'+# The migration row walks this machine's real configuration directories, so every test that runs+# the whole doctor stubs it — otherwise the result would depend on whose laptop the suite is on.+NO_PENDING_MIGRATIONS = PendingMigrationsCheck(+ finding=PendingMigrationsFinding.UP_TO_DATE,+ message="Every configuration file is at the current schema",+)+class TestDoctorLayeredResolution:"""Verify that doctor checks use layered config resolution:@@ -48,7 +59,7 @@ class TestDoctorLayeredResolution:)mock_check_telemetry = mocker.patch("pipelex.cli.commands.doctor_cmd.check_telemetry_config",- return_value=(True, "OK"),+ return_value=TelemetryConfigCheck(finding=TelemetryConfigFinding.HEALTHY, message="OK"),)mock_check_backends = mocker.patch("pipelex.cli.commands.doctor_cmd.check_backend_credentials",@@ -66,6 +77,10 @@ class TestDoctorLayeredResolution:"OK",),)+ mock_check_migrations = mocker.patch(+ "pipelex.cli.commands.doctor_cmd.check_pending_migrations",+ return_value=NO_PENDING_MIGRATIONS,+ )mocker.patch("pipelex.cli.commands.doctor_cmd.display_health_report")with pytest.raises(SystemExit) as exc_info:@@ -80,6 +95,9 @@ class TestDoctorLayeredResolution:mock_check_backends.assert_called_once_with()mock_check_models.assert_called_once_with()mock_check_deck.assert_called_once_with()+ # The migration row takes no directory at all — it answers for `pipelex migrate`, which+ # walks both configuration directories and has no way to be pointed at one.+ mock_check_migrations.assert_called_once_with()def test_pipelex_config_error_from_bootstrap_preserves_partial_report(self, mocker: MockerFixture) -> None:"""Regression: a config that passes check_config_files's shape check but fails full@@ -105,7 +123,7 @@ class TestDoctorLayeredResolution:)mocker.patch("pipelex.cli.commands.doctor_cmd.check_telemetry_config",- return_value=(True, "OK"),+ return_value=TelemetryConfigCheck(finding=TelemetryConfigFinding.HEALTHY, message="OK"),)mocker.patch("pipelex.cli.commands.doctor_cmd.check_backend_credentials",@@ -119,6 +137,10 @@ class TestDoctorLayeredResolution:"OK",),)+ mocker.patch(+ "pipelex.cli.commands.doctor_cmd.check_pending_migrations",+ return_value=NO_PENDING_MIGRATIONS,+ )mock_display = mocker.patch("pipelex.cli.commands.doctor_cmd.display_health_report")with pytest.raises(SystemExit) as exc_info:@@ -138,7 +160,7 @@ class TestDoctorLayeredResolution:assert "translated validation message" in kwargs["models_message"]# All other partial check tuples survived intact.assert kwargs["config_healthy"] is True- assert kwargs["telemetry_healthy"] is True+ assert kwargs["telemetry_check"].is_healthy is Trueassert kwargs["backends_healthy"] is Truedef test_telemetry_check_falls_back_to_global(self, mocker: MockerFixture, tmp_path: Path) -> None:@@ -154,9 +176,9 @@ class TestDoctorLayeredResolution:mocker.patch.object(ConfigLoader, "global_config_dir", new_callable=mocker.PropertyMock, return_value=global_dir)# No config_dir override → layered resolution should find it in global- healthy, message = check_telemetry_config()- assert healthy is True- assert "off" in message+ check = check_telemetry_config()+ assert check.is_healthy is True+ assert "off" in check.messagedef test_telemetry_check_uses_project_override(self, mocker: MockerFixture, tmp_path: Path) -> None:"""When project .pipelex/ has telemetry.toml, it should be used instead of global."""@@ -170,9 +192,9 @@ class TestDoctorLayeredResolution:mocker.patch.object(ConfigLoader, "project_config_dir", new_callable=mocker.PropertyMock, return_value=project_dir)mocker.patch.object(ConfigLoader, "global_config_dir", new_callable=mocker.PropertyMock, return_value=global_dir)- healthy, message = check_telemetry_config()- assert healthy is True- assert "anonymous" in message+ check = check_telemetry_config()+ assert check.is_healthy is True+ assert "anonymous" in check.messagedef test_telemetry_check_with_explicit_config_dir_skips_layering(self, tmp_path: Path) -> None:"""When config_dir is explicitly provided (e.g. --global), use it directly without layering."""@@ -180,8 +202,7 @@ class TestDoctorLayeredResolution:explicit_dir.mkdir()(explicit_dir / TELEMETRY_CONFIG_FILE_NAME).write_text(TELEMETRY_OFF, encoding="utf-8")- healthy, _message = check_telemetry_config(config_dir=explicit_dir)- assert healthy is True+ assert check_telemetry_config(config_dir=explicit_dir).is_healthy is Truedef test_backend_credentials_falls_back_to_global(self, mocker: MockerFixture, tmp_path: Path) -> None:"""When project .pipelex/ exists but has no backends.toml, fall back to global."""diff --git a/tests/unit/pipelex/cli/test_doctor_config_checks.py b/tests/unit/pipelex/cli/test_doctor_config_checks.pyindex 9fbd1ed29..c11071f94 100644--- a/tests/unit/pipelex/cli/test_doctor_config_checks.py+++ b/tests/unit/pipelex/cli/test_doctor_config_checks.py@@ -2,25 +2,28 @@from __future__ import annotations+from pathlib import Pathfrom typing import TYPE_CHECKING+import pytestfrom pydantic import BaseModel, ValidationErrorfrom pipelex.cli.commands.doctor_cmd import (+ TelemetryConfigFinding,check_config_files,check_telemetry_config,gather_config_location,)from pipelex.cli.exceptions import PipelexCLIError+from pipelex.core.validation import MIGRATE_COMMAND+from pipelex.migration.exceptions import MigrationLedgerErrorfrom pipelex.system.configuration.config_loader import ConfigLoader, config_managerfrom pipelex.system.configuration.configs import PipelexConfigfrom pipelex.system.exceptions import ConfigValidationError-from pipelex.system.telemetry.telemetry_config import TELEMETRY_CONFIG_FILE_NAME+from pipelex.system.telemetry.telemetry_config import TELEMETRY_CONFIG_FILE_NAME, TELEMETRY_CONFIG_OVERRIDE_FILE_NAMEfrom pipelex.tools.misc.exceptions import TomlErrorif TYPE_CHECKING:- from pathlib import Path-from pytest_mock import MockerFixture@@ -111,6 +114,32 @@ class TestDoctorConfigChecks:assert missing_count == 0assert message.startswith("Configuration validation failed:")+ def test_check_config_files_scans_only_the_directory_it_read(self, mocker: MockerFixture, tmp_path: Path) -> None:+ """The telemetry row's rule, on the configuration row: `--global` reads one directory.++ A project `pipelex.toml` carrying a key this build knows nothing about, beside the global+ directory under inspection whose own file refuses to load — the report must not name the+ project file, which a `--global` reader never loaded.+ """+ fake_home = tmp_path / "home"+ global_dir = fake_home / ".pipelex"+ global_dir.mkdir(parents=True)+ (global_dir / "pipelex.toml").write_text("[pipelex]\n", encoding="utf-8")+ project_root = tmp_path / "project"+ (project_root / ".git").mkdir(parents=True)+ (project_root / ".pipelex").mkdir()+ (project_root / ".pipelex" / "pipelex.toml").write_text("not_a_real_setting = true\n", encoding="utf-8")+ mocker.patch.object(Path, "home", return_value=fake_home)+ mocker.patch.object(Path, "cwd", return_value=project_root)+ mocker.patch("pipelex.cli.commands.doctor_cmd.init_config", return_value=0)+ mocker.patch.object(PipelexConfig, "model_validate", side_effect=_make_validation_error())++ healthy, _, message = check_config_files(config_dir=global_dir)++ assert healthy is False+ assert "not_a_real_setting" not in message+ assert MIGRATE_COMMAND not in message+def test_check_config_files_config_validation_error_with_cause(self, mocker: MockerFixture, tmp_path: Path) -> None:"""A ConfigValidationError wrapping a ValidationError recovers the original report."""(tmp_path / "pipelex.toml").write_text("[pipelex]\n", encoding="utf-8")@@ -164,35 +193,122 @@ class TestDoctorConfigChecks:assert "Error loading pipelex.toml: permission denied" in messagedef test_check_telemetry_config_file_not_found(self, tmp_path: Path) -> None:- """A missing telemetry.toml is unhealthy with a clear message."""- healthy, message = check_telemetry_config(config_dir=tmp_path)+ """A missing telemetry.toml is unhealthy, and is the one finding a fresh file repairs."""+ check = check_telemetry_config(config_dir=tmp_path)- assert healthy is False- assert message == "Telemetry configuration file not found"+ assert check.finding is TelemetryConfigFinding.NOT_FOUND+ assert check.finding.is_repaired_by_initializing is True+ assert check.message == "Telemetry configuration file not found"def test_check_telemetry_config_toml_syntax_error(self, tmp_path: Path) -> None:- """Invalid TOML in telemetry.toml is reported as a syntax error."""+ """Invalid TOML in telemetry.toml is reported as a syntax error a person resolves."""(tmp_path / TELEMETRY_CONFIG_FILE_NAME).write_text("not [ valid toml", encoding="utf-8")- healthy, message = check_telemetry_config(config_dir=tmp_path)+ check = check_telemetry_config(config_dir=tmp_path)- assert healthy is False- assert message.startswith("TOML syntax error:")+ assert check.finding is TelemetryConfigFinding.UNPARSEABLE+ assert check.finding.is_repaired_by_initializing is False+ assert check.message.startswith("TOML syntax error:")- def test_check_telemetry_config_old_format_detected(self, tmp_path: Path) -> None:- """The legacy flat format gets the dedicated migration message."""+ def test_check_telemetry_config_old_format_is_out_of_date_not_broken(self, tmp_path: Path) -> None:+ """The legacy flat format is what the shipped ledger entry is about, so it is migratable.++ This used to be answered with `pipelex init telemetry`, which writes a fresh file over+ the very settings — the PostHog key here — that the migration carries forward.+ """(tmp_path / TELEMETRY_CONFIG_FILE_NAME).write_text('telemetry_mode = "off"\nproject_api_key = "key"\n', encoding="utf-8")- healthy, message = check_telemetry_config(config_dir=tmp_path)+ check = check_telemetry_config(config_dir=tmp_path)- assert healthy is False- assert message == "Config format has changed - run 'pipelex init telemetry' to update"+ assert check.finding is TelemetryConfigFinding.OUT_OF_DATE+ assert check.finding.is_repaired_by_initializing is False+ assert MIGRATE_COMMAND in check.message+ assert "init telemetry" not in check.message- def test_check_telemetry_config_invalid_new_format(self, tmp_path: Path) -> None:- """A new-format file that fails validation gets the generic fix message."""+ def test_check_telemetry_config_invalid_new_format_names_the_field(self, tmp_path: Path) -> None:+ """A current-shape file that fails validation is wrong rather than old, and says which field."""(tmp_path / TELEMETRY_CONFIG_FILE_NAME).write_text('[custom_posthog]\nmode = "no-such-mode"\n', encoding="utf-8")- healthy, message = check_telemetry_config(config_dir=tmp_path)+ check = check_telemetry_config(config_dir=tmp_path)- assert healthy is False- assert message == "Invalid configuration - run 'pipelex init telemetry' to fix"+ assert check.finding is TelemetryConfigFinding.INVALID+ assert "custom_posthog.mode" in check.message+ assert MIGRATE_COMMAND not in check.message++ def test_check_telemetry_config_strips_the_reserved_meta_table(self, tmp_path: Path) -> None:+ """`[meta]` belongs to the migration machinery and boot strips it before validating.++ A probe stricter than the loader it reports on would call a perfectly bootable file+ invalid, which is worse than saying nothing.+ """+ (tmp_path / TELEMETRY_CONFIG_FILE_NAME).write_text('[meta]\nschema_version = 2\n\n[custom_posthog]\nmode = "off"\n', encoding="utf-8")++ check = check_telemetry_config(config_dir=tmp_path)++ assert check.finding is TelemetryConfigFinding.HEALTHY++ def test_check_telemetry_config_survives_a_failure_inside_the_scan(self, tmp_path: Path, mocker: MockerFixture) -> None:+ """A packaging problem of ours must not replace every row the user came for.++ An exception escaping this probe reaches the doctor's outer handler, which prints one+ line and exits — so the whole health report would be lost to a broken ledger. The+ fallback under-reports at worst, and still names the fields the model refused.+ """+ (tmp_path / TELEMETRY_CONFIG_FILE_NAME).write_text('[custom_posthog]\nmode = "no-such-mode"\n', encoding="utf-8")+ mocker.patch(+ "pipelex.cli.commands.doctor_cmd.scan_config_surface",+ side_effect=MigrationLedgerError("the packaged ledger will not load"),+ )++ check = check_telemetry_config(config_dir=tmp_path)++ assert check.finding is TelemetryConfigFinding.INVALID+ assert "custom_posthog.mode" in check.message++ def test_check_telemetry_config_lets_an_unexpected_scan_bug_surface(self, tmp_path: Path, mocker: MockerFixture) -> None:+ """The catch is narrow on purpose: an applier bug is not a field condition."""+ (tmp_path / TELEMETRY_CONFIG_FILE_NAME).write_text('[custom_posthog]\nmode = "no-such-mode"\n', encoding="utf-8")+ mocker.patch(+ "pipelex.cli.commands.doctor_cmd.scan_config_surface",+ side_effect=RuntimeError("an applier bug"),+ )++ with pytest.raises(RuntimeError, match="an applier bug"):+ check_telemetry_config(config_dir=tmp_path)++ def test_check_telemetry_config_scans_only_the_directory_it_read(self, tmp_path: Path, mocker: MockerFixture) -> None:+ """The finding answers for the file on the row, not for the other tier beside it.++ A stale file in the global directory and a wrong one in the directory under inspection:+ reporting the first would send a `--global` reader to a migration that does nothing for+ the file they are looking at.+ """+ fake_home = tmp_path / "home"+ global_dir = fake_home / ".pipelex"+ global_dir.mkdir(parents=True)+ (global_dir / TELEMETRY_CONFIG_FILE_NAME).write_text('telemetry_mode = "off"\n', encoding="utf-8")+ mocker.patch.object(Path, "home", return_value=fake_home)++ inspected = tmp_path / "inspected"+ inspected.mkdir()+ (inspected / TELEMETRY_CONFIG_FILE_NAME).write_text('[custom_posthog]\nmode = "no-such-mode"\n', encoding="utf-8")++ check = check_telemetry_config(config_dir=inspected)++ assert check.finding is TelemetryConfigFinding.INVALID++ def test_check_telemetry_config_answers_for_its_own_file_not_the_tier_beside_it(self, tmp_path: Path) -> None:+ """A stale tier file in the same directory does not make the base file old.++ The probe validates one file; the scan walks the directory it lives in, which also holds+ the `telemetry_*.toml` tier. A wrong `telemetry.toml` beside a stale+ `telemetry_override.toml` is wrong, and saying "run migrate" would leave the error behind.+ """+ (tmp_path / TELEMETRY_CONFIG_FILE_NAME).write_text('[custom_posthog]\nmode = "no-such-mode"\n', encoding="utf-8")+ (tmp_path / TELEMETRY_CONFIG_OVERRIDE_FILE_NAME).write_text('telemetry_mode = "off"\n', encoding="utf-8")++ check = check_telemetry_config(config_dir=tmp_path)++ assert check.finding is TelemetryConfigFinding.INVALID+ assert "custom_posthog.mode" in check.message+ assert MIGRATE_COMMAND not in check.messagediff --git a/tests/unit/pipelex/cli/test_doctor_display_report.py b/tests/unit/pipelex/cli/test_doctor_display_report.pyindex 938eba55a..a0ce2397b 100644--- a/tests/unit/pipelex/cli/test_doctor_display_report.py+++ b/tests/unit/pipelex/cli/test_doctor_display_report.py@@ -10,17 +10,27 @@ from rich.console import Consolefrom pipelex.cli.commands.doctor_cmd import (BackendFileReport,ConfigLocationInfo,+ PendingMigrationsCheck,+ PendingMigrationsFinding,+ TelemetryConfigCheck,+ TelemetryConfigFinding,display_health_report,doctor_cmd,)from pipelex.cogt.model_backends.backend_library import BackendCredentialsReportfrom pipelex.cogt.models.deck_manifest import DeckFileStatus, DeckSyncReport+from pipelex.core.validation import MIGRATE_COMMANDif TYPE_CHECKING:from pytest_mock import MockerFixtureCLEAN_DECK = DeckSyncReport(kit_version="1.2.0", installed_kit_version="1.2.0", manifest_present=True, files={})+NO_PENDING_MIGRATIONS = PendingMigrationsCheck(+ finding=PendingMigrationsFinding.UP_TO_DATE,+ message="Every configuration file is at the current schema",+)+PROJECT_LOCATION = ConfigLocationInfo(config_dir="/work/project/.pipelex",is_project_local=True,@@ -41,8 +51,8 @@ def _healthy_report_kwargs() -> dict[str, Any]:"config_healthy": True,"config_message": "All configuration files present and valid","config_missing_count": 0,- "telemetry_healthy": True,- "telemetry_message": "Telemetry configured (mode: off)",+ "pending_migrations_check": NO_PENDING_MIGRATIONS,+ "telemetry_check": TelemetryConfigCheck(finding=TelemetryConfigFinding.HEALTHY, message="Telemetry configured (mode: off)"),"backends_healthy": True,"backends_message": "All 1 enabled backend(s) have valid credentials","backend_credential_reports": {},@@ -229,6 +239,102 @@ class TestDoctorDisplayReport:assert "Interactive fixes for outdated backend configurations will be offered below" in outputassert "Run pipelex doctor --fix" not in output+ def test_pending_migrations_row_names_every_file_and_the_command(self, console: Console) -> None:+ """The row a machine that has drifted actually shows: what runs, and what is still owed."""+ kwargs = _healthy_report_kwargs()+ kwargs["pending_migrations_check"] = PendingMigrationsCheck(+ finding=PendingMigrationsFinding.PENDING,+ message="1 configuration file(s) can be brought up to date by 'pipelex migrate'",+ migratable_files=["/home/user/.pipelex/telemetry.toml"],+ attention_files=["/work/project/.pipelex/pipelex.toml"],+ )++ display_health_report(**kwargs)++ output = console.export_text()+ assert "Configuration Migrations" in output+ assert "/home/user/.pipelex/telemetry.toml" in output, "the row is where a reader learns which directory is touched"+ assert "/work/project/.pipelex/pipelex.toml" in output+ assert f"Run {MIGRATE_COMMAND} to bring 1 configuration file(s) up to date" in output+ assert f"Run {MIGRATE_COMMAND} --dry-run" in output, "a file the command will not repair is still owed a look"++ def test_a_row_that_could_not_be_checked_is_not_reported_as_healthy(self, console: Console) -> None:+ """Not knowing is not the same as being up to date, and it names a way to find out."""+ kwargs = _healthy_report_kwargs()+ kwargs["pending_migrations_check"] = PendingMigrationsCheck(+ finding=PendingMigrationsFinding.UNAVAILABLE,+ message="Could not check for pending migrations: the packaged ledger will not load",+ )++ display_health_report(**kwargs)++ output = console.export_text()+ assert "Issues Found" in output+ assert "the packaged ledger will not load" in output+ assert f"Run {MIGRATE_COMMAND} --dry-run to check for pending migrations" in output++ def test_an_out_of_date_telemetry_file_is_not_sent_to_the_same_command_twice(self, console: Console) -> None:+ """The machine-wide row already names `pipelex migrate` and already lists this file."""+ kwargs = _healthy_report_kwargs()+ kwargs["telemetry_check"] = TelemetryConfigCheck(+ finding=TelemetryConfigFinding.OUT_OF_DATE,+ message="Configuration is out of date",+ )+ kwargs["pending_migrations_check"] = PendingMigrationsCheck(+ finding=PendingMigrationsFinding.PENDING,+ message="1 configuration file(s) can be brought up to date",+ migratable_files=["/home/user/.pipelex/telemetry.toml"],+ )++ display_health_report(**kwargs)++ solutions = console.export_text().split("Possible Solutions", 1)[1]+ assert "telemetry.toml up to date" not in solutions+ assert solutions.count(f"Run {MIGRATE_COMMAND} to bring") == 1++ def test_a_telemetry_file_the_migration_will_not_repair_keeps_its_own_bullet(self, console: Console) -> None:+ """The suppression is scoped, not blanket: with no pending run, the row speaks for itself.++ Reachable when the telemetry file's only pending work is blocked — the surface-scoped+ probe still calls it out of date while the machine-wide row has nothing to write.+ """+ kwargs = _healthy_report_kwargs()+ kwargs["telemetry_check"] = TelemetryConfigCheck(+ finding=TelemetryConfigFinding.OUT_OF_DATE,+ message="Configuration is out of date",+ )+ kwargs["pending_migrations_check"] = PendingMigrationsCheck(+ finding=PendingMigrationsFinding.NEEDS_ATTENTION,+ message="1 configuration file(s) need a look",+ attention_files=["/home/user/.pipelex/telemetry.toml"],+ )++ display_health_report(**kwargs)++ assert "telemetry.toml up to date" in console.export_text()++ def test_a_message_carrying_brackets_renders_literally(self, console: Console) -> None:+ """Both rows can quote a user's file or an OS error, and Rich reads `[...]` as markup.++ A bracketed path segment would be silently dropped and a `[/x]` sequence would raise, so a+ diagnostic that exists to name the file must escape what it prints.+ """+ kwargs = _healthy_report_kwargs()+ kwargs["telemetry_check"] = TelemetryConfigCheck(+ finding=TelemetryConfigFinding.INVALID,+ message="Invalid configuration:\ninvalid enum value '[/x] not-a-mode'",+ )+ kwargs["pending_migrations_check"] = PendingMigrationsCheck(+ finding=PendingMigrationsFinding.UNAVAILABLE,+ message="Could not check for pending migrations: [Errno 13] Permission denied: '/home/[user]/.pipelex'",+ )++ display_health_report(**kwargs)++ output = console.export_text()+ assert "'[/x] not-a-mode'" in output+ assert "'/home/[user]/.pipelex'" in output+def test_doctor_cmd_catches_unexpected_errors(self, console: Console, mocker: MockerFixture) -> None:"""The doctor entry point converts unexpected errors into a friendly exit 1."""mocker.patch("pipelex.cli.commands.doctor_cmd.do_doctor_cmd", side_effect=RuntimeError("totally unexpected"))diff --git a/tests/unit/pipelex/cli/test_doctor_fix_mode.py b/tests/unit/pipelex/cli/test_doctor_fix_mode.pyindex 97a38c1c1..0859ab9dc 100644--- a/tests/unit/pipelex/cli/test_doctor_fix_mode.py+++ b/tests/unit/pipelex/cli/test_doctor_fix_mode.py@@ -2,15 +2,25 @@from __future__ import annotations+from pathlib import Path as PurePathfrom typing import TYPE_CHECKING, Anyimport pytestfrom rich.console import Console-from pipelex.cli.commands.doctor_cmd import BackendFileReport, do_doctor_cmd+from pipelex.cli.commands.doctor_cmd import (+ BackendFileReport,+ PendingMigrationsCheck,+ PendingMigrationsFinding,+ TelemetryConfigCheck,+ TelemetryConfigFinding,+ do_doctor_cmd,+)from pipelex.cli.commands.init.ui.types import InitFocusfrom pipelex.cogt.model_backends.backend_library import BackendCredentialsReportfrom pipelex.cogt.models.deck_manifest import DeckFileStatus, DeckSyncReport+from pipelex.core.validation import MIGRATE_COMMAND+from pipelex.migration.plan import MigrationPlan, MigrationReportif TYPE_CHECKING:from pathlib import Path@@ -18,6 +28,18 @@ if TYPE_CHECKING:from pytest_mock import MockerFixtureCLEAN_DECK = DeckSyncReport(kit_version="1.2.0", installed_kit_version="1.2.0", manifest_present=True, files={})+HEALTHY_TELEMETRY = TelemetryConfigCheck(finding=TelemetryConfigFinding.HEALTHY, message="OK")+NO_PENDING_MIGRATIONS = PendingMigrationsCheck(+ finding=PendingMigrationsFinding.UP_TO_DATE,+ message="Every configuration file is at the current schema",+)+STALE_TELEMETRY_FILE = "/home/user/.pipelex/telemetry.toml"+ONE_MIGRATION_PENDING = PendingMigrationsCheck(+ finding=PendingMigrationsFinding.PENDING,+ message="1 configuration file(s) can be brought up to date",+ migratable_files=[STALE_TELEMETRY_FILE],+)+ONE_FILE_MIGRATED = MigrationReport(plans=[MigrationPlan(surface_id="telemetry-config", file_path=PurePath(STALE_TELEMETRY_FILE), was_written=True)])class TestDoctorFixMode:@@ -27,7 +49,11 @@ class TestDoctorFixMode:mocks: dict[str, Any] = {"setup": mocker.patch("pipelex.cli.commands.doctor_cmd.setup_doctor_runtime"),"config": mocker.patch("pipelex.cli.commands.doctor_cmd.check_config_files", return_value=(True, 0, "OK")),- "telemetry": mocker.patch("pipelex.cli.commands.doctor_cmd.check_telemetry_config", return_value=(True, "OK")),+ "telemetry": mocker.patch("pipelex.cli.commands.doctor_cmd.check_telemetry_config", return_value=HEALTHY_TELEMETRY),+ # Stubbed for every test, not only the ones about it: the real check walks this+ # machine's own configuration directories, and no unit test should depend on those.+ "migrations": mocker.patch("pipelex.cli.commands.doctor_cmd.check_pending_migrations", return_value=NO_PENDING_MIGRATIONS),+ "migrate": mocker.patch("pipelex.cli.commands.doctor_cmd.apply_pending_migrations"),"backends": mocker.patch("pipelex.cli.commands.doctor_cmd.check_backend_credentials", return_value=(True, {}, "OK")),"models": mocker.patch("pipelex.cli.commands.doctor_cmd.check_models", return_value=(True, "OK", {})),"deck": mocker.patch("pipelex.cli.commands.doctor_cmd.check_deck_sync", return_value=(True, CLEAN_DECK, "OK")),@@ -58,15 +84,112 @@ class TestDoctorFixMode:assert "Interactive Fix Mode" in outputassert "Configuration files installed" in output- def test_fix_telemetry_format_change_runs_init_telemetry(self, doctor_mocks: dict[str, Any]) -> None:- """Accepting the telemetry fix runs init focused on telemetry."""- doctor_mocks["telemetry"].return_value = (False, "Config format has changed - run 'pipelex init telemetry' to update")+ def test_fix_missing_telemetry_runs_init_telemetry(self, doctor_mocks: dict[str, Any]) -> None:+ """A telemetry file that is not there is written, and it is the only finding that is."""+ doctor_mocks["telemetry"].return_value = TelemetryConfigCheck(+ finding=TelemetryConfigFinding.NOT_FOUND,+ message="Telemetry configuration file not found",+ )self._run_doctor_expecting_exit_one()doctor_mocks["init_cmd"].assert_called_once_with(focus=InitFocus.TELEMETRY, skip_confirmation=True)assert "Telemetry configured" in doctor_mocks["console"].export_text()+ def test_fix_never_offers_to_reset_an_out_of_date_telemetry_file(self, doctor_mocks: dict[str, Any]) -> None:+ """The defect this milestone retired, pinned as behaviour.++ `--fix` used to read the row's *message* for "format has changed" and, on a match, offer+ to write a fresh telemetry.toml — discarding the PostHog key, the Langfuse credentials+ and the exporters that the migration it should have named would have carried forward.+ """+ doctor_mocks["telemetry"].return_value = TelemetryConfigCheck(+ finding=TelemetryConfigFinding.OUT_OF_DATE,+ message=f"Configuration is out of date — run '{MIGRATE_COMMAND}' to bring it up to date",+ )++ self._run_doctor_expecting_exit_one()++ doctor_mocks["init_cmd"].assert_not_called()++ def test_fix_never_offers_to_reset_an_invalid_telemetry_file(self, doctor_mocks: dict[str, Any]) -> None:+ """A file that is wrong is a person's to edit; a reset is offered as a way to start over.++ The distinction matters on the same file the previous test is about: the old sniff also+ matched "invalid configuration", so a single mistyped enum value was answered by+ rewriting the whole file.+ """+ doctor_mocks["telemetry"].return_value = TelemetryConfigCheck(+ finding=TelemetryConfigFinding.INVALID,+ message="Invalid configuration:\nValidation error(s):",+ )++ self._run_doctor_expecting_exit_one()++ doctor_mocks["init_cmd"].assert_not_called()+ output = doctor_mocks["console"].export_text()+ assert "Telemetry validation error" in output+ assert "discarding what is in it" in output++ def test_fix_migrates_the_files_the_ledger_can_carry_forward(self, doctor_mocks: dict[str, Any]) -> None:+ """The half of this item that `--fix` owed: the command is offered, not merely named.++ It runs the same write pass `pipelex migrate` runs — the row above it was that command's+ own dry run — rather than a second implementation that could drift from it.+ """+ doctor_mocks["migrations"].return_value = ONE_MIGRATION_PENDING+ doctor_mocks["migrate"].return_value = ONE_FILE_MIGRATED++ self._run_doctor_expecting_exit_one()++ doctor_mocks["migrate"].assert_called_once()+ prompts = [call.args[0] for call in doctor_mocks["confirm"].call_args_list]+ assert any("Migrate 1 configuration file(s)" in prompt for prompt in prompts)+ output = doctor_mocks["console"].export_text()+ assert "Migrated 1 configuration file(s)" in output+ assert "Re-run" in output, "the rows below were measured before the migration ran"++ def test_declining_the_migration_writes_nothing(self, doctor_mocks: dict[str, Any]) -> None:+ """The prompt is a real question — a no leaves the machine exactly as it was."""+ doctor_mocks["migrations"].return_value = ONE_MIGRATION_PENDING+ doctor_mocks["confirm"].return_value = False++ self._run_doctor_expecting_exit_one()++ doctor_mocks["migrate"].assert_not_called()++ def test_fix_offers_nothing_for_files_the_migration_would_not_repair(self, doctor_mocks: dict[str, Any]) -> None:+ """A file carrying a key no ledger explains is a person's, and the command would do nothing.++ Offering to run it would be a prompt whose honest outcome is "nothing was written" — the+ same shape as the reset `--fix` used to offer for an out-of-date telemetry file.+ """+ doctor_mocks["migrations"].return_value = PendingMigrationsCheck(+ finding=PendingMigrationsFinding.NEEDS_ATTENTION,+ message="1 configuration file(s) need a look",+ attention_files=["/work/project/.pipelex/pipelex.toml"],+ )++ self._run_doctor_expecting_exit_one()++ doctor_mocks["migrate"].assert_not_called()++ def test_a_failed_migration_is_reported_and_the_doctor_carries_on(self, doctor_mocks: dict[str, Any]) -> None:+ """One fix failing must not take the rest of fix mode down with it."""+ doctor_mocks["migrations"].return_value = ONE_MIGRATION_PENDING+ doctor_mocks["migrate"].side_effect = OSError("the directory is read-only")+ doctor_mocks["deck"].return_value = (+ False,+ DeckSyncReport(kit_version="1.3.0", installed_kit_version="1.2.0", manifest_present=True, files={}),+ "Deck is behind",+ )++ self._run_doctor_expecting_exit_one()++ assert "Failed to migrate configuration files" in doctor_mocks["console"].export_text()+ # The deck fix sits below the migration in fix mode, and it still ran.+ doctor_mocks["update_cmd"].assert_called_once_with(yes=True)+def test_fix_outdated_deck_runs_update(self, doctor_mocks: dict[str, Any]) -> None:"""Accepting the deck fix runs `pipelex update --yes`."""dirty_deck = DeckSyncReport(diff --git a/tests/unit/pipelex/cli/test_doctor_pending_migrations.py b/tests/unit/pipelex/cli/test_doctor_pending_migrations.pynew file mode 100644index 000000000..970460e62--- /dev/null+++ b/tests/unit/pipelex/cli/test_doctor_pending_migrations.py@@ -0,0 +1,223 @@+"""The doctor's configuration-migration row: what a dry run of `pipelex migrate` found.++Nothing else tells a machine this. A stale configuration the ledger can explain boots with a+warning, and the agent CLI cuts logging off process-wide before one can be emitted — so for a+machine consumer, asking is the only channel, and this row is the asking.++The row differs from every other check in the doctor in one way that is a decision rather than an+oversight: **it takes no directory**. The others report on a *file* and are scoped to the directory+the doctor was pointed at, `--global` included. This one reports on a *command*, and `pipelex+migrate` has no `--global` — it walks the global `~/.pipelex/` and the project `.pipelex/` both. A+row scoped narrower would name a command that then rewrites a file the row never mentioned.++The stale document here is the package's own: `telemetry-config@2` is the shipped entry and+`goldens/telemetry-config/[email protected]` is the flat document it exists to carry forward.+"""++from __future__ import annotations++from pathlib import Path+from typing import TYPE_CHECKING, NamedTuple++import pytest++from pipelex.cli.commands.doctor_cmd import PendingMigrationsFinding, check_pending_migrations+from pipelex.core.validation import MIGRATE_COMMAND+from pipelex.migration.backup import existing_backups_of+from pipelex.migration.exceptions import MigrationLedgerError+from pipelex.migration.goldens import pre_history_document_path+from pipelex.migration.ledger import packaged_migration_dir+from pipelex.system.configuration.config_surface import TELEMETRY_CONFIG_SURFACE_ID++if TYPE_CHECKING:+ from pytest_mock import MockerFixture++# A realistic PostHog project key, at the path the shipped entry moves. The old flat format is+# exactly where a real one would be, which makes this the right specimen for the rendering rule.+PLANTED_KEY = "phc_L1VE_telemetry_key_that_must_never_be_rendered"++UNKNOWN_ROOT_KEY = "not_a_real_setting"+++def old_shape_document() -> str:+ """The flat pre-`[custom_posthog]` document the shipped entry is about, read from the package."""+ path = pre_history_document_path(migration_dir=packaged_migration_dir(), surface_id=TELEMETRY_CONFIG_SURFACE_ID, schema_version=2)+ return path.read_text(encoding="utf-8")+++def half_migrated_document() -> str:+ """The flat document with `host` already moved by hand into a `[custom_posthog]` the user created.++ The shipped entry's first move (`telemetry_mode`) lands; its second (`host` → `endpoint`) finds+ the destination taken and blocks the entry with what applied so far.+ """+ return 'telemetry_mode = "anonymous"\nhost = "https://eu.i.posthog.com"\n\n[custom_posthog]\nendpoint = "https://us.i.posthog.com"\n'+++class Machine(NamedTuple):+ """The two configuration directories a migration walks, on a fake home."""++ global_dir: Path+ project_dir: Path+++def machine(tmp_path: Path, mocker: MockerFixture) -> Machine:+ """A fake home and a fake project, so the walk reads this test's files and not the machine's."""+ fake_home = tmp_path / "home"+ global_dir = fake_home / ".pipelex"+ global_dir.mkdir(parents=True)++ project_root = tmp_path / "project"+ (project_root / ".git").mkdir(parents=True)+ project_dir = project_root / ".pipelex"+ project_dir.mkdir()++ mocker.patch.object(Path, "home", return_value=fake_home)+ mocker.patch.object(Path, "cwd", return_value=project_root)++ return Machine(global_dir=global_dir, project_dir=project_dir)+++class TestWhatTheRowReports:+ """The verdict, the files it names, and the two things the row must never do."""++ @pytest.mark.usefixtures("machine")+ def test_a_machine_at_the_current_schema_has_nothing_pending(self) -> None:+ """Two empty configuration directories are two directories with nothing to do."""+ check = check_pending_migrations()++ assert check.finding is PendingMigrationsFinding.UP_TO_DATE+ assert check.is_healthy is True+ assert check.migratable_files == []+ assert check.attention_files == []++ def test_a_stale_file_is_pending_and_the_row_names_it(self, machine: Machine) -> None:+ """The remedy is the command, and the file it would rewrite is named with its full path."""+ stale = machine.global_dir / "telemetry.toml"+ stale.write_text(old_shape_document(), encoding="utf-8")++ check = check_pending_migrations()++ assert check.finding is PendingMigrationsFinding.PENDING+ assert check.finding.is_repaired_by_migrating is True+ assert check.migratable_files == [str(stale)]+ assert MIGRATE_COMMAND in check.message++ def test_a_file_no_ledger_explains_needs_a_look_rather_than_a_command(self, machine: Machine) -> None:+ """A root key this build knows nothing about is a person's, and `--fix` must not offer it.++ The migration would write nothing here, so a row that reported it as pending would send a+ user to a command that runs and changes nothing — and, in fix mode, would offer to do it.+ """+ broken = machine.project_dir / "pipelex.toml"+ broken.write_text(f"{UNKNOWN_ROOT_KEY} = true\n", encoding="utf-8")++ check = check_pending_migrations()++ assert check.finding is PendingMigrationsFinding.NEEDS_ATTENTION+ assert check.finding.is_repaired_by_migrating is False+ assert check.migratable_files == []+ assert check.attention_files == [str(broken)]++ def test_a_run_that_both_migrates_and_leaves_something_reports_both(self, machine: Machine) -> None:+ """The ordinary shape on a machine that has drifted, and the one a single verdict flattens."""+ stale = machine.global_dir / "telemetry.toml"+ stale.write_text(old_shape_document(), encoding="utf-8")+ broken = machine.project_dir / "pipelex.toml"+ broken.write_text(f"{UNKNOWN_ROOT_KEY} = true\n", encoding="utf-8")++ check = check_pending_migrations()++ assert check.finding is PendingMigrationsFinding.PENDING, "there is still something to run"+ assert check.migratable_files == [str(stale)]+ assert check.attention_files == [str(broken)]++ def test_a_file_a_conflict_only_partly_migrates_is_still_pending(self, machine: Machine) -> None:+ """A file the user half-migrated by hand: the entry conflicts, yet part of it applies.++ The write pass rewrites that file — the operations that landed before the conflict are+ real — so the row must name it as migratable *and* as needing a look. A row that reported+ only the conflict would tell `--fix` there is nothing to run on a machine the command+ would change.+ """+ stale = machine.global_dir / "telemetry.toml"+ stale.write_text(half_migrated_document(), encoding="utf-8")++ check = check_pending_migrations()++ assert check.finding is PendingMigrationsFinding.PENDING+ assert check.migratable_files == [str(stale)]+ assert check.attention_files == [str(stale)]++ def test_a_stale_file_in_either_directory_is_reported(self, machine: Machine) -> None:+ """Both directories, because both are what `pipelex migrate` would rewrite.++ A row scoped to the directory the doctor resolved would leave the other one out, and the+ command it names would then touch a file the reader was never shown.+ """+ global_stale = machine.global_dir / "telemetry.toml"+ global_stale.write_text(old_shape_document(), encoding="utf-8")+ project_stale = machine.project_dir / "telemetry_override.toml"+ project_stale.write_text('telemetry_mode = "off"\n', encoding="utf-8")++ check = check_pending_migrations()++ assert sorted(check.migratable_files) == sorted([str(global_stale), str(project_stale)])++ def test_the_row_is_a_dry_run_and_leaves_the_files_exactly_as_it_found_them(self, machine: Machine) -> None:+ """A health report that repaired what it reported would be a command, not a report."""+ stale = machine.global_dir / "telemetry.toml"+ before = old_shape_document()+ stale.write_text(before, encoding="utf-8")++ check = check_pending_migrations()++ assert check.finding is PendingMigrationsFinding.PENDING+ assert stale.read_text(encoding="utf-8") == before+ assert existing_backups_of(path=stale) == [], "a dry run writes nothing, backups included"++ @pytest.mark.usefixtures("machine")+ def test_a_broken_ledger_is_reported_as_a_failure_to_check(self, mocker: MockerFixture) -> None:+ """An exception here would reach the doctor's outer handler and take every row with it.++ That handler prints one line and exits, so a packaging problem of ours would replace the+ whole health report. Saying we could not look is honest and costs the user one row.+ """+ mocker.patch(+ "pipelex.cli.commands.doctor_cmd.migrate_config_directories",+ side_effect=MigrationLedgerError("the packaged ledger will not load"),+ )++ check = check_pending_migrations()++ assert check.finding is PendingMigrationsFinding.UNAVAILABLE+ assert check.finding.is_uncheckable is True+ assert check.is_healthy is False, "not knowing is not the same as being up to date"+ assert "the packaged ledger will not load" in check.message++ @pytest.mark.usefixtures("machine")+ def test_an_unexpected_bug_still_surfaces_as_itself(self, mocker: MockerFixture) -> None:+ """The catch is narrow on purpose: a bug in our applier is not a field condition."""+ mocker.patch(+ "pipelex.cli.commands.doctor_cmd.migrate_config_directories",+ side_effect=RuntimeError("an applier bug"),+ )++ with pytest.raises(RuntimeError, match="an applier bug"):+ check_pending_migrations()+++class TestNoValueFromAUsersFileIsEverRendered:+ """The mechanical rule, on the row that reads every configuration file on the machine."""++ def test_a_planted_secret_reaches_neither_the_message_nor_the_file_lists(self, machine: Machine) -> None:+ stale = machine.global_dir / "telemetry.toml"+ document = old_shape_document().replace('project_api_key = "phc_example_project_api_key"', f'project_api_key = "{PLANTED_KEY}"')+ assert PLANTED_KEY in document, "the specimen must actually carry the secret"+ stale.write_text(document, encoding="utf-8")++ check = check_pending_migrations()++ assert PLANTED_KEY in stale.read_text(encoding="utf-8")+ assert PLANTED_KEY not in str(check.model_dump(mode="json"))diff --git a/tests/unit/pipelex/cli/test_error_handlers_more.py b/tests/unit/pipelex/cli/test_error_handlers_more.pyindex 1224f4bc9..b5a48682f 100644--- a/tests/unit/pipelex/cli/test_error_handlers_more.py+++ b/tests/unit/pipelex/cli/test_error_handlers_more.py@@ -14,6 +14,7 @@ import pytestimport typerfrom rich.console import Console+from pipelex.base_exceptions import MigrationErrorBlockfrom pipelex.cli.error_handlers import (ErrorContext,display_error_panel,@@ -32,6 +33,7 @@ from pipelex.cogt.inference.error_classification import UserAction, UserActionKifrom pipelex.cogt.model_backends.model_type import ModelTypefrom pipelex.core.exceptions import PipelexBundleBlueprintValidationErrorData, PipesAndConceptValidationErrorDatafrom pipelex.core.pipes.exceptions import PipeValidationErrorType+from pipelex.core.validation import MIGRATE_COMMANDfrom pipelex.pipeline.exceptions import ValidateBundleErrorfrom pipelex.system.pipelex_service.exceptions import (GatewayApiKeyMissingError,@@ -212,17 +214,39 @@ class TestErrorHandlersExtended:# signature-specific tip used to leak here from the deleted handle_signatures_not_allowed_error).assert "--allow-signatures" not in output- def test_handle_telemetry_config_validation_error(self, console: Console) -> None:- """The telemetry handler explains the format migration."""- exc = TelemetryConfigValidationError("old flat format")+ def test_handle_telemetry_config_validation_error_on_a_file_that_is_wrong(self, console: Console) -> None:+ """No migration explains this one, so what the reader gets is the fields and an edit."""+ exc = TelemetryConfigValidationError("'custom_posthog.mode': invalid enum value")with pytest.raises(typer.Exit) as exc_info:handle_telemetry_config_validation_error(exc)assert exc_info.value.exit_code == 1output = console.export_text()- assert "Telemetry configuration format has changed" in output- assert "pipelex init telemetry" in output+ assert "custom_posthog.mode" in output+ assert "Correct the settings named above" in output+ assert MIGRATE_COMMAND not in output++ def test_handle_telemetry_config_validation_error_on_a_file_that_is_out_of_date(self, console: Console) -> None:+ """The remedy comes off the error rather than out of the handler.++ This handler used to announce a breaking format change and send every reader to+ `pipelex init telemetry` — which, on exactly this case, writes a fresh file over the+ settings the migration it should have named would have carried forward.+ """+ exc = TelemetryConfigValidationError(+ "extra forbidden field: 'telemetry_mode'",+ migration=MigrationErrorBlock(remedy=MIGRATE_COMMAND, needs_attention=False, plans=[]),+ )++ with pytest.raises(typer.Exit) as exc_info:+ handle_telemetry_config_validation_error(exc)++ assert exc_info.value.exit_code == 1+ output = console.export_text()+ assert MIGRATE_COMMAND in output+ assert "does not start it over" in output+ assert "init telemetry" not in outputdef test_handle_gateway_terms_not_accepted_error(self, console: Console) -> None:"""The terms handler points at init config and the BYOK alternative."""diff --git a/tests/unit/pipelex/cli/test_format_doctor_markdown.py b/tests/unit/pipelex/cli/test_format_doctor_markdown.pyindex f89c390ec..09252450a 100644--- a/tests/unit/pipelex/cli/test_format_doctor_markdown.py+++ b/tests/unit/pipelex/cli/test_format_doctor_markdown.py@@ -23,6 +23,13 @@ class TestFormatDoctorMarkdown:},"checks": {"config_files": {"healthy": True, "message": "All config files present", "missing_count": 0},+ "pending_migrations": {+ "healthy": True,+ "finding": "up_to_date",+ "message": "Every configuration file is at the current schema",+ "migratable_files": [],+ "attention_files": [],+ },"telemetry": {"healthy": True, "message": "Telemetry configured"},"backend_credentials": {"healthy": True, "message": "All backends healthy", "backends": []},"models": {"healthy": True, "message": "Models valid", "backend_files": []},@@ -40,6 +47,13 @@ class TestFormatDoctorMarkdown:},"checks": {"config_files": {"healthy": True, "message": "OK", "missing_count": 0},+ "pending_migrations": {+ "healthy": False,+ "finding": "pending",+ "message": "1 configuration file(s) can be brought up to date by 'pipelex migrate'",+ "migratable_files": ["/home/user/.pipelex/telemetry.toml"],+ "attention_files": [],+ },"telemetry": {"healthy": False, "message": "Config format outdated"},"backend_credentials": {"healthy": False,@@ -77,6 +91,14 @@ class TestFormatDoctorMarkdown:output = _format_doctor_markdown(self._HEALTHY_RESULT)assert "Recommended Actions" not in output+ def test_pending_migrations_section_lists_the_files_a_migration_would_touch(self) -> None:+ """The row an agent reads when nothing has failed yet — a boot would never have said it."""+ rendered = _format_doctor_markdown(self._UNHEALTHY_RESULT)++ assert "## Configuration Migrations" in rendered+ assert "1 configuration file(s) can be brought up to date" in rendered+ assert "`/home/user/.pipelex/telemetry.toml`: out of date" in rendered+def test_unhealthy_output_shows_warning_status(self) -> None:"""Unhealthy result should show 'Issues found' with warning emoji."""output = _format_doctor_markdown(self._UNHEALTHY_RESULT)diff --git a/tests/unit/pipelex/cli/test_migrate_cmd_summary.py b/tests/unit/pipelex/cli/test_migrate_cmd_summary.pynew file mode 100644index 000000000..8c9253185--- /dev/null+++ b/tests/unit/pipelex/cli/test_migrate_cmd_summary.py@@ -0,0 +1,125 @@+"""What the summary panel of `pipelex migrate` says once the write pass has run.++The per-file lines above the panel are the runner's account; the panel is the command's one-line+verdict on the run, and it must not contradict the lines it closes. One state makes that easy to+get wrong: a transaction that left a file's state uncertain has *not* left it as it was found, so+"Nothing was written." would be a false comfort right under a line telling the user to compare the+file against its rescue copy — and "Migrated 1 file(s)" is the same false comfort when the other file+of the run is the uncertain one.++The cancel path is the one place the panel is replaced by a prompt, and it answers to the same rule+as every other exit: a person's "no" to the write does not close what the rehearsal left for a person.+"""++from __future__ import annotations++from typing import TYPE_CHECKING++import pytest+from rich.console import Console++from pipelex.cli.commands import migrate_cmd as migrate_cmd_module+from pipelex.cli.commands.migrate_cmd import migrate_cmd+from pipelex.migration.goldens import pre_history_document_path+from pipelex.migration.ledger import packaged_migration_dir+from pipelex.pipeline.exceptions import FixTransactionError, FixWriteConflictError+from pipelex.pipeline.fixes.file_transaction import PendingFileUpdate, commit_file_updates+from pipelex.system.configuration.config_surface import TELEMETRY_CONFIG_SURFACE_ID+from pipelex.system.telemetry.telemetry_config import TELEMETRY_CONFIG_OVERRIDE_FILE_NAME++if TYPE_CHECKING:+ from pathlib import Path++ from pytest_mock import MockerFixture+++def console(mocker: MockerFixture) -> Console:+ recorded = Console(width=200, record=True, color_system=None)+ mocker.patch.object(migrate_cmd_module, "get_console", return_value=recorded)+ return recorded+++def stale_machine(tmp_path: Path, mocker: MockerFixture) -> Path:+ """One configuration directory holding the flat telemetry document the shipped entry carries forward."""+ config_dir = tmp_path / ".pipelex"+ config_dir.mkdir()+ old_shape = pre_history_document_path(migration_dir=packaged_migration_dir(), surface_id=TELEMETRY_CONFIG_SURFACE_ID, schema_version=2)+ stale = config_dir / "telemetry.toml"+ stale.write_text(old_shape.read_text(encoding="utf-8"), encoding="utf-8")+ mocker.patch.object(migrate_cmd_module, "config_directories_to_migrate", return_value=[config_dir])+ return stale++[email protected]("stale_machine")+class TestTheSummaryPanel:+ def test_an_uncertain_write_is_not_reported_as_nothing_written(self, console: Console, mocker: MockerFixture) -> None:+ """The transaction replaced the file and could not say what it left; the copy is the user's next move."""+ mocker.patch("pipelex.migration.runner.commit_file_updates", side_effect=FixTransactionError("commit failed and rollback was incomplete"))++ with pytest.raises(SystemExit) as leaving:+ migrate_cmd(yes=True)++ assert leaving.value.code == 1, "a file a person has to look at"+ output = console.export_text()+ assert "Nothing was written" not in output+ assert "No write could be confirmed" in output+ assert "state_uncertain" in output++ def test_a_write_the_run_refused_still_says_nothing_was_written(self, console: Console, mocker: MockerFixture) -> None:+ """The other blocked reasons keep the file exactly as it was found, and the panel says so."""+ mocker.patch("pipelex.migration.runner.commit_file_updates", side_effect=FixWriteConflictError("the file changed under the run"))++ with pytest.raises(SystemExit):+ migrate_cmd(yes=True)++ output = console.export_text()+ assert "Nothing was written." in output+ assert "No write could be confirmed" not in output++ def test_a_written_file_does_not_hide_the_uncertain_one_beside_it(self, stale_machine: Path, console: Console, mocker: MockerFixture) -> None:+ """Two stale files, one landed and one in an unknown state: the verdict is about the second."""+ stale_override = stale_machine.parent / TELEMETRY_CONFIG_OVERRIDE_FILE_NAME+ stale_override.write_text(stale_machine.read_text(encoding="utf-8"), encoding="utf-8")++ def commit_only_the_base_file(updates: list[PendingFileUpdate]) -> None:+ if updates[0].snapshot.path == stale_override:+ msg = "commit failed and rollback was incomplete"+ raise FixTransactionError(msg)+ commit_file_updates(updates)++ mocker.patch("pipelex.migration.runner.commit_file_updates", side_effect=commit_only_the_base_file)++ with pytest.raises(SystemExit) as leaving:+ migrate_cmd(yes=True)++ assert leaving.value.code == 1+ output = console.export_text()+ assert "state_uncertain" in output+ assert "Migrated 1 file(s), but no write could be confirmed" in output+ assert "a copy of each original is beside it" not in output++[email protected]("console")+class TestTheCancelPath:+ def test_cancelling_leaves_the_files_alone_and_exits_clean_when_nothing_needs_a_person(self, stale_machine: Path, mocker: MockerFixture) -> None:+ mocker.patch.object(migrate_cmd_module.Confirm, "ask", return_value=False)+ before = stale_machine.read_text(encoding="utf-8")++ migrate_cmd()++ assert stale_machine.read_text(encoding="utf-8") == before++ def test_cancelling_still_exits_non_zero_when_the_rehearsal_left_something_for_a_person(self, stale_machine: Path, mocker: MockerFixture) -> None:+ """The "no" answers "shall I write?", not the unexplained key the rehearsal reported."""+ mocker.patch.object(migrate_cmd_module.Confirm, "ask", return_value=False)+ with stale_machine.open("a", encoding="utf-8") as handle:+ handle.write('no_such_key = "typo"\n')+ before = stale_machine.read_text(encoding="utf-8")++ with pytest.raises(SystemExit) as leaving:+ migrate_cmd()++ assert leaving.value.code == 1+ assert stale_machine.read_text(encoding="utf-8") == beforediff --git a/tests/unit/pipelex/core/test_validation_report.py b/tests/unit/pipelex/core/test_validation_report.pyindex 5f893baf6..c61affecb 100644--- a/tests/unit/pipelex/core/test_validation_report.py+++ b/tests/unit/pipelex/core/test_validation_report.py@@ -1,26 +1,51 @@-"""Unit tests for report_validation_error — runs before/around bootstrap.--Pins the regression where ``log.verbose`` raised ``RuntimeError("LogConfig is not set")``-inside ``report_validation_error`` because the doctor calls this helper from inside its-own bootstrap (setup_config raised, hub._config still None, log not configured). The-fix dropped the now-unnecessary log.verbose calls and replaced the broad-``try/except RuntimeError`` around ``get_config().migration`` with non-raising hub-accessors. This test pins both behaviors.+"""`report_validation_error` — the translation, and the migration that may explain it.++A configuration validation error says a key is wrong. It cannot say *why*, because it is raised+against the merged configuration and carries no provenance: it does not know which of the files+that were merged put the key there, nor whether the key was perfectly correct last week. The+answer to that second question is in the ledger, and this module is where the two meet.++Three properties are what the tests below exist to hold:++- **The scan runs only when a surface is named.** A `.mthds` bundle, a backend file and a model+ deck all reach this helper too, and none of them has a ledger — offering them a `pipelex migrate`+ remedy would send a user to a command with nothing to do.+- **The scan needs nothing but the filesystem.** It runs on a machine whose configuration just+ refused to load, which is a machine with no hub and no config. That is the same bootstrap+ property the `migrate` commands hold, and it is asserted here rather than assumed, because this+ helper used to read the hub and crash when there was not one.+- **Nothing that goes wrong inside the scan becomes the failure the user sees.** They have an+ error in front of them that names what to fix; ours must not replace it."""from __future__ import annotations+import json+from pathlib import Pathfrom typing import TYPE_CHECKINGimport pytestfrom pydantic import BaseModel, ValidationError-from pipelex.core.validation import report_validation_error+from pipelex.base_exceptions import ErrorDomain, ErrorReport, PipelexConfigError+from pipelex.core.validation import MIGRATE_COMMAND, raise_config_setup_error, report_validation_error+from pipelex.migration import run as migration_run_module+from pipelex.migration.exceptions import MigrationLedgerError+from pipelex.migration.goldens import pre_history_document_path+from pipelex.migration.ledger import packaged_migration_dirfrom pipelex.runtime_hub import RuntimeHub+from pipelex.system.configuration.config_loader import CONFIG_REFUSED, pydantic_error_behind+from pipelex.system.configuration.config_surface import PIPELEX_CONFIG_SURFACE_ID, TELEMETRY_CONFIG_SURFACE_ID+from pipelex.system.configuration.configs import PipelexConfig+from pipelex.system.exceptions import ConfigValidationErrorif TYPE_CHECKING:from pytest_mock import MockerFixture+#: The shipped `telemetry-config@2` entry is about a real flat document the package carries, so a+#: stale machine can be built out of our own files rather than out of an invented schema change.+TELEMETRY_ENTRY_ID = "telemetry-config@2"+class _ConfigShape(BaseModel):required_field: str@@ -34,36 +59,296 @@ def _make_validation_error() -> ValidationError:pytest.fail("Expected _ConfigShape.model_validate to raise ValidationError")-class TestReportValidationError:- def test_returns_message_when_hub_singleton_is_none(self, mocker: MockerFixture) -> None:- """report_validation_error must still produce the friendly translation when no hub- is installed on the process singleton.+def _old_shape_telemetry_document() -> str:+ path = pre_history_document_path(migration_dir=packaged_migration_dir(), surface_id=TELEMETRY_CONFIG_SURFACE_ID, schema_version=2)+ return path.read_text(encoding="utf-8")+++def machine(tmp_path: Path, mocker: MockerFixture) -> Path:+ """A global configuration directory this test owns, in place of the one on the machine."""+ fake_home = tmp_path / "home"+ global_dir = fake_home / ".pipelex"+ global_dir.mkdir(parents=True)+ project_root = tmp_path / "project"+ (project_root / ".git").mkdir(parents=True)+ mocker.patch.object(Path, "home", return_value=fake_home)+ mocker.patch.object(Path, "cwd", return_value=project_root)+ return global_dir+++class TestTheScanRunsOnlyWhenASurfaceIsNamed:+ def test_no_surface_means_no_block(self, machine: Path) -> None:+ """The `.mthds` half of this helper's callers, and every other non-surface reader."""+ machine.joinpath("telemetry.toml").write_text(_old_shape_telemetry_document(), encoding="utf-8")++ report = report_validation_error(validation_error=_make_validation_error())++ assert report.migration is None+ assert "required_field" in report.message+ assert MIGRATE_COMMAND not in report.message++ def test_no_surface_means_the_walk_never_happens(self, mocker: MockerFixture) -> None:+ """Not merely an empty answer — the filesystem is not walked at all.++ Without this, a `_pending_migration` that always ran and merely returned nothing useful+ for a `.mthds` error would pass the test above while costing every bundle-validation+ failure a directory walk and a ledger replay.+ """+ scan = mocker.patch.object(migration_run_module, "scan_config_surface")++ report_validation_error(validation_error=_make_validation_error())++ scan.assert_not_called()+++class TestWhatTheScanFinds:+ def test_a_stale_file_becomes_a_block_naming_what_would_be_carried_forward(self, machine: Path) -> None:+ machine.joinpath("telemetry.toml").write_text(_old_shape_telemetry_document(), encoding="utf-8")++ report = report_validation_error(validation_error=_make_validation_error(), surface_id=TELEMETRY_CONFIG_SURFACE_ID)++ assert report.migration is not None+ assert report.migration.remedy == MIGRATE_COMMAND+ assert not report.migration.needs_attention, "a file the ledger carries forward whole needs nobody"+ assert [step.entry_id for plan in report.migration.plans for step in plan.steps] == [TELEMETRY_ENTRY_ID]++ def test_the_message_keeps_the_translation_and_gains_the_remedy(self, machine: Path) -> None:+ """Both halves, in one string: the analysis is what names the actual error."""+ machine.joinpath("telemetry.toml").write_text(_old_shape_telemetry_document(), encoding="utf-8")++ report = report_validation_error(validation_error=_make_validation_error(), surface_id=TELEMETRY_CONFIG_SURFACE_ID)- Reproduces the agent doctor's pre-fix crash path: with the new ordering,- check_config_files calls report_validation_error before setup_doctor_runtime,- so no hub has been installed yet. (Note: tests/conftest.py's autouse Pipelex.make- fixture configures ``log`` for this process — this test pins hub-state isolation,- not log-state isolation.)+ assert "required_field" in report.message+ assert "telemetry.toml" in report.message+ assert MIGRATE_COMMAND in report.message++ def test_a_path_no_entry_explains_needs_a_person(self, machine: Path) -> None:+ """The downgrade diagnosis reaching the error surface — a typo, or a newer pipelex.++ `pipelex-config` ships an entry-free ledger, so nothing here is carried forward and the+ whole finding is the unknown key. That is the shape a stale-looking machine most often+ has, and it is the one where the remedy is *not* the command.+ """+ machine.joinpath("pipelex.toml").write_text("not_a_real_setting = true\n", encoding="utf-8")++ report = report_validation_error(validation_error=_make_validation_error(), surface_id=PIPELEX_CONFIG_SURFACE_ID)++ assert report.migration is not None+ assert report.migration.needs_attention+ assert [found.path for plan in report.migration.plans for found in plan.unexplained] == ["not_a_real_setting"]+ assert "not_a_real_setting" in report.message++ def test_a_current_machine_gets_no_block_at_all(self, machine: Path) -> None:+ """Presence is the contract, so a healthy machine must produce absence."""+ machine.joinpath("telemetry.toml").write_text('[custom_posthog]\nmode = "off"\n', encoding="utf-8")++ report = report_validation_error(validation_error=_make_validation_error(), surface_id=TELEMETRY_CONFIG_SURFACE_ID)++ assert report.migration is None+ assert MIGRATE_COMMAND not in report.message++ def test_only_the_surface_that_refused_is_scanned(self, machine: Path) -> None:+ """A stale `telemetry.toml` is not an answer to a `pipelex.toml` that would not load."""+ machine.joinpath("telemetry.toml").write_text(_old_shape_telemetry_document(), encoding="utf-8")+ machine.joinpath("pipelex.toml").write_text("[pipelex]\n", encoding="utf-8")++ report = report_validation_error(validation_error=_make_validation_error(), surface_id=PIPELEX_CONFIG_SURFACE_ID)++ assert report.migration is None++ def test_another_surfaces_base_file_is_not_swept_up_by_this_ones_glob(self, machine: Path) -> None:+ """Scoping narrows the answer; it must not narrow the registry that decides ownership.++ `pipelex_service.toml` is `pipelex-service-config`'s base file *and* a match for+ `pipelex-config`'s tier glob `pipelex_*.toml`, and the registry resolves that by letting an+ exact base file claim before any glob — **across all surfaces**. Scoping the scan by+ building a registry that holds only the surface asked about removes the other claimant from+ that arbitration, and the glob then wins: the file is replayed under the wrong ledger and+ diagnosed against the wrong model, so its perfectly ordinary settings come back reported as+ paths this build knows nothing about. Measured, not imagined — that is what it did.+ """+ machine.joinpath("pipelex.toml").write_text("not_a_real_setting = true\n", encoding="utf-8")+ machine.joinpath("pipelex_service.toml").write_text("[agreement]\nterms_accepted = true\n", encoding="utf-8")++ report = report_validation_error(validation_error=_make_validation_error(), surface_id=PIPELEX_CONFIG_SURFACE_ID)++ assert report.migration is not None+ assert [plan.file_path.name for plan in report.migration.plans] == ["pipelex.toml"]+++class TestTheScanIsScopedToTheDirectoriesThatWereLoaded:+ """A caller that loaded one directory must be diagnosed against that directory.++ `doctor --global` and an embedder's `config_dir=` bypass the global/project layering and read+ one directory. Answering their refusal with a scan of the default walk would name a file the+ reader never loaded — and stay silent about the one they did.+ """++ def test_a_stale_file_in_a_directory_that_was_not_loaded_is_not_the_answer(self, machine: Path, tmp_path: Path) -> None:+ project_dir = tmp_path / "project" / ".pipelex"+ project_dir.mkdir()+ project_dir.joinpath("pipelex.toml").write_text("not_a_real_setting = true\n", encoding="utf-8")++ report = report_validation_error(validation_error=_make_validation_error(), surface_id=PIPELEX_CONFIG_SURFACE_ID, config_dirs=[machine])++ assert report.migration is None++ def test_a_directory_outside_the_walk_is_diagnosed_when_it_is_the_one_loaded(self, machine: Path, tmp_path: Path) -> None:+ machine.joinpath("pipelex.toml").write_text("another_unknown_setting = true\n", encoding="utf-8")+ elsewhere = tmp_path / "elsewhere"+ elsewhere.mkdir()+ elsewhere.joinpath("pipelex.toml").write_text("not_a_real_setting = true\n", encoding="utf-8")++ report = report_validation_error(validation_error=_make_validation_error(), surface_id=PIPELEX_CONFIG_SURFACE_ID, config_dirs=[elsewhere])++ assert report.migration is not None+ assert [plan.file_path for plan in report.migration.plans] == [elsewhere / "pipelex.toml"]++ def test_the_raised_error_is_scoped_the_same_way(self, machine: Path, tmp_path: Path) -> None:+ project_dir = tmp_path / "project" / ".pipelex"+ project_dir.mkdir()+ project_dir.joinpath("pipelex.toml").write_text("not_a_real_setting = true\n", encoding="utf-8")++ with pytest.raises(PipelexConfigError) as caught:+ raise_config_setup_error(config_error=_make_validation_error(), surface_id=PIPELEX_CONFIG_SURFACE_ID, config_dirs=[machine])++ assert caught.value.migration is None+++class TestTheScanNeedsNothingButTheFilesystem:+ def test_it_runs_with_no_hub_installed(self, machine: Path, mocker: MockerFixture) -> None:+ """The bootstrap property, on the one path that reaches it.++ This helper is called from inside `RuntimeBoot.__init__` and from the doctor's own+ bootstrap, both before any configuration exists — and it used to read the hub, which is+ how it once crashed with `LogConfig is not set` on exactly this path."""- # Ensure the hub singleton is None for this test — mirrors a fresh process- # where Pipelex.make hasn't run yet.mocker.patch.object(RuntimeHub, "_instance", None)+ machine.joinpath("telemetry.toml").write_text(_old_shape_telemetry_document(), encoding="utf-8")++ report = report_validation_error(validation_error=_make_validation_error(), surface_id=TELEMETRY_CONFIG_SURFACE_ID)- validation_error = _make_validation_error()- report = report_validation_error(category="config", validation_error=validation_error)+ assert report.migration is not None+ assert "required_field" in report.message- assert "required_field" in report- def test_returns_message_when_hub_has_no_config(self, mocker: MockerFixture) -> None:- """Hub exists but _config is None (setup_config raised mid-flight).+class TestAFailureInsideTheScanIsNotTheFailureTheUserSees:+ def test_a_ledger_that_will_not_load_leaves_the_translation_standing(self, mocker: MockerFixture) -> None:+ """A packaging bug of ours must not cost the user the message that names their own error.- get_optional_config() returns None; migration hints are omitted; the error- translation still flows.+ It stays loud where it should be: `make check-ledger`, and `pipelex migrate` itself."""- hub_without_config = RuntimeHub()- mocker.patch.object(RuntimeHub, "_instance", hub_without_config)+ mocker.patch.object(migration_run_module, "scan_config_surface", side_effect=MigrationLedgerError("the ledger is not there"))++ report = report_validation_error(validation_error=_make_validation_error(), surface_id=PIPELEX_CONFIG_SURFACE_ID)++ assert report.migration is None+ assert "required_field" in report.message+ assert "the ledger is not there" not in report.message++ def test_an_applier_bug_still_surfaces(self, mocker: MockerFixture) -> None:+ """The catch is narrow on purpose: only what a field machine can legitimately produce.++ A bug in our own code is not that, and swallowing it here would hide it behind every+ configuration error anyone ever hits.+ """+ mocker.patch.object(migration_run_module, "scan_config_surface", side_effect=RuntimeError("an applier bug"))++ with pytest.raises(RuntimeError, match="an applier bug"):+ report_validation_error(validation_error=_make_validation_error(), surface_id=PIPELEX_CONFIG_SURFACE_ID)+++class TestTheErrorCarriesBothTheDomainAndTheBlock:+ """The charter's own pair: `error_domain` stays `config`, and the block rides beside it."""++ def test_the_raised_error_reports_the_config_domain_and_the_migration(self, machine: Path) -> None:+ machine.joinpath("telemetry.toml").write_text(_old_shape_telemetry_document(), encoding="utf-8")+ refusal = ConfigValidationError("the configuration was refused")+ refusal.__cause__ = _make_validation_error()++ with pytest.raises(PipelexConfigError) as raised:+ raise_config_setup_error(config_error=refusal, surface_id=TELEMETRY_CONFIG_SURFACE_ID)++ error_report = raised.value.to_error_report()+ assert error_report.error_domain == ErrorDomain.CONFIG, "a new domain would route the agent hooks to BLOCK"+ assert error_report.migration is not None+ assert error_report.migration.remedy == MIGRATE_COMMAND++ @pytest.mark.usefixtures("machine")+ def test_the_report_the_error_produces_is_json_serializable(self, machine: Path) -> None:+ """`ErrorReport.to_dict()` is a serialization surface, and the block carries real `Path`s.++ The webhook delivery path hands that dict straight to `json.dumps`, which cannot serialize+ a `PosixPath`. Not reachable from a pipeline run today — nothing raises this error inside+ one — but `to_dict` is documented as *the* serialization surface, so a field that quietly+ makes it unserializable is a trap left for whoever wires the next consumer.+ """+ machine.joinpath("telemetry.toml").write_text(_old_shape_telemetry_document(), encoding="utf-8")+ refusal = ConfigValidationError("the configuration was refused")+ refusal.__cause__ = _make_validation_error()++ with pytest.raises(PipelexConfigError) as raised:+ raise_config_setup_error(config_error=refusal, surface_id=TELEMETRY_CONFIG_SURFACE_ID)++ payload = raised.value.to_error_report().to_dict()+ assert json.dumps(payload), "the block must survive a plain json.dumps"+ assert ErrorReport.from_dict(payload).migration is not None, "and it must come back through the round-trip"++ @pytest.mark.usefixtures("machine")+ def test_a_healthy_machine_raises_the_same_error_without_a_block(self) -> None:+ """Absence is what a consumer branches on, so it has to be reachable."""+ refusal = ConfigValidationError("the configuration was refused")+ refusal.__cause__ = _make_validation_error()++ with pytest.raises(PipelexConfigError) as raised:+ raise_config_setup_error(config_error=refusal, surface_id=TELEMETRY_CONFIG_SURFACE_ID)++ assert raised.value.to_error_report().migration is None++ def test_a_refusal_carrying_no_pydantic_error_is_re_raised_as_itself(self) -> None:+ """There is nothing to translate, and its own message is already the whole account."""+ refusal = ConfigValidationError("the configuration directory is not readable")++ with pytest.raises(ConfigValidationError, match="not readable"):+ raise_config_setup_error(config_error=refusal, surface_id=PIPELEX_CONFIG_SURFACE_ID)+++class TestTheRefusalTheMainConfigurationActuallyRaises:+ """The fact that made the boot's own `except` arm dead code for as long as it existed."""++ def test_it_is_not_pydantics_own_error(self) -> None:+ with pytest.raises(CONFIG_REFUSED) as raised:+ PipelexConfig.model_validate({"not_a_real_setting": True})++ assert not isinstance(raised.value, ValidationError), (+ "`ConfigRoot.__init__` translates pydantic's error, so an `except ValidationError` around "+ "`setup_config` never fires — which is exactly what it used to do"+ )+ assert pydantic_error_behind(config_error=raised.value) is not None+++class TestNoValueFromAUsersFileIsEverRendered:+ """The contract's mechanical rule, on the third of its three channels.++ The `migrate` commands' output and their structured plan are covered end to end; this is the+ `migration` block on a configuration validation error. The planted value is carried by a+ `move_key` the shipped entry performs, so the scan demonstrably *handled* it — a rule that+ held only because the value was quietly dropped would be no rule at all.+ """++ PLANTED_SECRET = "phc_L1VE_s3cret_project_key_that_must_never_be_rendered"++ def test_neither_the_message_nor_the_block_carries_a_planted_secret(self, machine: Path) -> None:+ document = _old_shape_telemetry_document().replace("phc_example_project_api_key", self.PLANTED_SECRET)+ assert self.PLANTED_SECRET in document, "the golden no longer carries the api key this test substitutes into"+ machine.joinpath("telemetry.toml").write_text(document, encoding="utf-8")- validation_error = _make_validation_error()- report = report_validation_error(category="config", validation_error=validation_error)+ report = report_validation_error(validation_error=_make_validation_error(), surface_id=TELEMETRY_CONFIG_SURFACE_ID)- assert "required_field" in report+ assert report.migration is not None+ assert [step.entry_id for plan in report.migration.plans for step in plan.steps] == [TELEMETRY_ENTRY_ID], (+ "the entry that moves the planted key has to have fired, or this proves nothing"+ )+ assert self.PLANTED_SECRET not in report.message+ assert self.PLANTED_SECRET not in json.dumps(report.migration.model_dump(mode="json"))diff --git a/tests/unit/pipelex/migration/conftest.py b/tests/unit/pipelex/migration/conftest.pyindex 644b87b77..df2d3212c 100644--- a/tests/unit/pipelex/migration/conftest.py+++ b/tests/unit/pipelex/migration/conftest.py@@ -8,9 +8,10 @@ from collections.abc import Callablefrom pathlib import Pathimport pytest-from pydantic import BaseModel+from pydantic import BaseModel, Field-from pipelex.migration.ledger import MigrationEntry, MigrationLedger, MigrationSafety, SurfaceBlock+from pipelex.migration.ledger import MigrationEntry, MigrationLedger, SurfaceBlock+from pipelex.migration.safety import MigrationSafetyfrom pipelex.migration.surfaces import DefaultsLayerKind, Surface, SurfaceRegistryfrom pipelex.suggested_fix import MigrationOp@@ -65,6 +66,7 @@ def build_ledger() -> LedgerBuilder:surface_id: str = EXAMPLE_SURFACE_ID,base_file: str = EXAMPLE_BASE_FILE,tier_glob: str | None = EXAMPLE_TIER_GLOB,+ min_supported_schema_version: int = 0,) -> MigrationLedger:return MigrationLedger(surface=SurfaceBlock(@@ -73,7 +75,7 @@ def build_ledger() -> LedgerBuilder:base_file=base_file,tier_glob=tier_glob,current_schema_version=1 + len(entries),- min_supported_schema_version=0,+ min_supported_schema_version=min_supported_schema_version,),migration=entries,)@@ -81,10 +83,26 @@ def build_ledger() -> LedgerBuilder:return _build_ledger+class ExampleOutput(BaseModel):+ directory: str = "out"+++class ExampleReporting(BaseModel):+ output: ExampleOutput = Field(default_factory=ExampleOutput)+ destination: str | None = None++class ExampleConfig(BaseModel):- """A stand-in configuration model for surfaces the tests invent."""+ """A stand-in configuration model for surfaces the tests invent.++ It has to name the paths the fixtures' *migrated* documents carry, and not only the ones a test+ happens to assert on. A migration run diagnoses what it leaves behind against this model, so a+ path missing here is reported as unexplained — which is the diagnosis working, and noise in+ every test that was about something else.+ """label: str = "hello"+ reporting: ExampleReporting = Field(default_factory=ExampleReporting)@pytest.fixturediff --git a/tests/unit/pipelex/migration/test_diagnosis.py b/tests/unit/pipelex/migration/test_diagnosis.pynew file mode 100644index 000000000..01a609f61--- /dev/null+++ b/tests/unit/pipelex/migration/test_diagnosis.py@@ -0,0 +1,260 @@+"""Unit tests for the downgrade diagnosis — what a migration cannot explain about a file.++Everything a ledger explains has been carried forward by the time a replay ends, so what is left+over is either a typo or a key from a newer pipelex. The tests are about the two ways that answer+goes wrong: naming something the report has already accounted for, and rendering a key the user+chose rather than the one the schema does not know.+"""++from typing import Any++import pytest+from pydantic import BaseModel, Field++from pipelex.migration.diagnosis import diagnose_unexplained_paths+from pipelex.migration.fingerprint import SurfaceFingerprint, compute_fingerprint+from pipelex.migration.ledger import MigrationLedger+from pipelex.migration.plan import BlockedEntry, BlockedEntryReason+from pipelex.migration.safety import MigrationSafety+from pipelex.suggested_fix import DeleteKeyOp, MigrationOp, RemapValueOp, RenameTableKeyOp+from tests.unit.pipelex.migration.conftest import EXAMPLE_SURFACE_ID, EntryBuilder, LedgerBuilder+++class Output(BaseModel):+ directory: str = "out"+++class Reporting(BaseModel):+ output: Output = Field(default_factory=Output)+ retries: int = 1+++class Queue(BaseModel):+ retries: int = 1+++class Schema(BaseModel):+ label: str = "hello"+ reporting: Reporting = Field(default_factory=Reporting)+ levels: dict[str, str] = Field(default_factory=dict[str, str])+ queues: dict[str, Queue] = Field(default_factory=dict[str, Queue])+++def _fingerprint() -> SurfaceFingerprint:+ return compute_fingerprint(surface_id=EXAMPLE_SURFACE_ID, schema_version=3, config_model=Schema, defaults_document={})+++def _diagnose(*, document: dict[str, Any], ledger: MigrationLedger, blocked: list[BlockedEntry] | None = None) -> list[str]:+ found = diagnose_unexplained_paths(fingerprint=_fingerprint(), document=document, ledger=ledger, blocked=blocked or [])+ return [unexplained.path for unexplained in found]+++def _blocked(*, to_schema_version: int, reason: BlockedEntryReason = BlockedEntryReason.UNSAFE) -> BlockedEntry:+ return BlockedEntry(+ entry_id=f"{EXAMPLE_SURFACE_ID}@{to_schema_version}",+ to_schema_version=to_schema_version,+ reason=reason,+ detail="reported by the engine",+ )+++def empty_ledger(build_ledger: LedgerBuilder) -> MigrationLedger:+ return build_ledger(entries=[])+++class TestWhatTheSchemaDoesNotKnow:+ def test_a_root_key_no_model_knows_is_reported_with_both_readings(self, empty_ledger: MigrationLedger) -> None:+ found = diagnose_unexplained_paths(+ fingerprint=_fingerprint(),+ document={"label": "hi", "not_a_real_setting": True},+ ledger=empty_ledger,+ blocked=[],+ )++ assert [unexplained.path for unexplained in found] == ["not_a_real_setting"]+ note = found[0].note+ assert "typo" in note+ assert "newer pipelex" in note, "the downgrade reading is the one nothing else in the report offers"++ def test_a_document_the_schema_knows_entirely_is_reported_clean(self, empty_ledger: MigrationLedger) -> None:+ document = {"label": "hi", "reporting": {"output": {"directory": "elsewhere"}, "retries": 3}}++ assert _diagnose(document=document, ledger=empty_ledger) == []++ def test_an_unknown_key_inside_a_known_table_is_reported_at_its_full_path(self, empty_ledger: MigrationLedger) -> None:+ document = {"reporting": {"retires": 3}}++ assert _diagnose(document=document, ledger=empty_ledger) == ["reporting.retires"]++ def test_an_unknown_table_is_reported_once_rather_than_once_per_key_inside_it(self, empty_ledger: MigrationLedger) -> None:+ """The shallowest name is the one the user has to fix; listing the subtree buries it."""+ document = {"reportng": {"output": {"directory": "out"}, "retries": 3}}++ assert _diagnose(document=document, ledger=empty_ledger) == ["reportng"]++ def test_a_table_where_the_schema_wants_a_scalar_is_left_to_the_model(self, empty_ledger: MigrationLedger) -> None:+ """A type error, not a downgrade. Descending would invent unknown paths under a known one."""+ document = {"reporting": {"retries": {"how_many": 3}}}++ assert _diagnose(document=document, ledger=empty_ledger) == []+++class TestKeysTheUserChose:+ """A key beneath an open mapping belongs to the user, and is no more rendered than a value."""++ def test_a_users_own_key_beneath_an_open_mapping_is_not_reported(self, empty_ledger: MigrationLedger) -> None:+ document = {"levels": {"my_private_package": "INFO"}}++ assert _diagnose(document=document, ledger=empty_ledger) == []++ def test_a_typo_inside_an_open_mappings_value_is_reported_as_the_schema_spells_it(self, empty_ledger: MigrationLedger) -> None:+ document = {"queues": {"my_private_queue": {"retreis": 2}}}++ found = _diagnose(document=document, ledger=empty_ledger)++ assert found == ["queues.*.retreis"]+ assert "my_private_queue" not in found[0], "the user's own key is not the thing the schema does not know"++ def test_nesting_below_an_open_mappings_scalar_value_is_left_to_the_model(self, empty_ledger: MigrationLedger) -> None:+ document = {"levels": {"my_private_package": {"level": "INFO"}}}++ assert _diagnose(document=document, ledger=empty_ledger) == []+++class TestTheReservedMetaTable:+ def test_the_reserved_schema_version_is_not_reported(self, empty_ledger: MigrationLedger) -> None:+ """Boot tolerates the key, so a migration must not tell the user it is a typo."""+ document = {"meta": {"schema_version": 2}, "label": "hi"}++ assert _diagnose(document=document, ledger=empty_ledger) == []+ assert document == {"meta": {"schema_version": 2}, "label": "hi"}, "a diagnosis reads; it does not edit what it was handed"++ def test_a_meta_table_carrying_anything_else_is_reported(self, empty_ledger: MigrationLedger) -> None:+ """Only that one key is reserved, and what is left over is a table no schema knows.++ Reported as `meta` rather than `meta.author`, which is the shallowest-unknown rule and also+ what the boot says: no configuration model has a `meta` field, so `extra="forbid"` rejects+ the table and not the key inside it.+ """+ document = {"meta": {"schema_version": 2, "author": "someone"}}++ assert _diagnose(document=document, ledger=empty_ledger) == ["meta"]+++class TestWhatABlockedEntryAnswersFor:+ """An entry the run would not apply is the *reason* its material is still in the file."""++ def _ledger_with_a_blocking_entry(self, *, build_entry: EntryBuilder, build_ledger: LedgerBuilder, ops: list[MigrationOp]) -> MigrationLedger:+ return build_ledger(entries=[build_entry(to_schema_version=2, safety=MigrationSafety.UNSAFE, ops=ops)])++ def test_the_old_shape_a_blocked_entry_is_about_is_not_also_called_unexplained(+ self,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ ) -> None:+ ledger = self._ledger_with_a_blocking_entry(+ build_entry=build_entry,+ build_ledger=build_ledger,+ ops=[DeleteKeyOp(table_path=["reporting"], key="legacy_mode")],+ )++ assert _diagnose(document={"reporting": {"legacy_mode": "on"}}, ledger=ledger, blocked=[_blocked(to_schema_version=2)]) == []++ def test_the_same_path_is_reported_when_no_entry_is_blocked(+ self,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ ) -> None:+ """The other half of the pair: the subtraction is the blocked entry's, not the ledger's."""+ ledger = self._ledger_with_a_blocking_entry(+ build_entry=build_entry,+ build_ledger=build_ledger,+ ops=[DeleteKeyOp(table_path=["reporting"], key="legacy_mode")],+ )++ assert _diagnose(document={"reporting": {"legacy_mode": "on"}}, ledger=ledger, blocked=[]) == ["reporting.legacy_mode"]++ def test_a_blocked_entry_answers_for_the_whole_subtree_it_would_have_taken(+ self,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ ) -> None:+ ledger = self._ledger_with_a_blocking_entry(+ build_entry=build_entry,+ build_ledger=build_ledger,+ ops=[DeleteKeyOp(table_path=[], key="legacy")],+ )+ document = {"legacy": {"mode": "on", "nested": {"deeper": 1}}}++ assert _diagnose(document=document, ledger=ledger, blocked=[_blocked(to_schema_version=2)]) == []++ def test_a_blocked_entry_answers_for_the_material_at_the_spelling_later_entries_gave_it(+ self,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ ) -> None:+ """A later `safe` entry renames the table around a blocked one, and the file follows it."""+ ledger = build_ledger(+ entries=[+ build_entry(to_schema_version=2, safety=MigrationSafety.UNSAFE, ops=[DeleteKeyOp(table_path=["old_report"], key="legacy_mode")]),+ build_entry(to_schema_version=3, ops=[RenameTableKeyOp(table_path=[], key="old_report", new_key="reporting")]),+ ]+ )++ assert _diagnose(document={"reporting": {"legacy_mode": "on"}}, ledger=ledger, blocked=[_blocked(to_schema_version=2)]) == []++ def test_a_blocked_entry_answers_for_what_it_declares_removed(+ self,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ ) -> None:+ """A pre-history entry has no fingerprint pair behind it, so its declaration is the record.++ The declared path here is one no operation addresses, which is what makes the declaration+ the only thing that can account for it.+ """+ ledger = build_ledger(+ entries=[+ build_entry(+ to_schema_version=2,+ safety=MigrationSafety.UNSAFE,+ pre_history=True,+ declared_removed_paths=["reporting.legacy_mode", "reporting.other_thing"],+ ops=[DeleteKeyOp(table_path=["reporting"], key="other_thing")],+ )+ ]+ )++ assert _diagnose(document={"reporting": {"legacy_mode": "on"}}, ledger=ledger, blocked=[_blocked(to_schema_version=2)]) == []++ def test_a_blocked_remap_answers_for_nothing(+ self,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ ) -> None:+ """A remap moves a value and leaves the path where it was, so it explains no unknown path.++ Subtracting it would silence a genuine typo that happens to sit at a remapped key.+ """+ ledger = self._ledger_with_a_blocking_entry(+ build_entry=build_entry,+ build_ledger=build_ledger,+ ops=[RemapValueOp(table_path=["reporting"], key="mode", mapping={"old": "new"})],+ )++ assert _diagnose(document={"reporting": {"mode": "old"}}, ledger=ledger, blocked=[_blocked(to_schema_version=2)]) == ["reporting.mode"]++ def test_an_unrelated_typo_survives_a_blocked_entry(+ self,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ ) -> None:+ ledger = self._ledger_with_a_blocking_entry(+ build_entry=build_entry,+ build_ledger=build_ledger,+ ops=[DeleteKeyOp(table_path=["reporting"], key="legacy_mode")],+ )+ document = {"reporting": {"legacy_mode": "on", "retires": 3}}++ assert _diagnose(document=document, ledger=ledger, blocked=[_blocked(to_schema_version=2)]) == ["reporting.retires"]diff --git a/tests/unit/pipelex/migration/test_engine.py b/tests/unit/pipelex/migration/test_engine.pyindex 746bc2697..633446b00 100644--- a/tests/unit/pipelex/migration/test_engine.py+++ b/tests/unit/pipelex/migration/test_engine.py@@ -10,8 +10,8 @@ from typing import Any, castimport tomlkitfrom pipelex.migration.engine import replay_ledger_over_text-from pipelex.migration.ledger import MigrationSafetyfrom pipelex.migration.plan import BlockedEntryReason+from pipelex.migration.safety import MigrationSafetyfrom pipelex.suggested_fix import DeleteKeyOp, MoveKeyOp, RenameTableKeyOpfrom tests.unit.pipelex.migration.conftest import EntryBuilder, LedgerBuilderdiff --git a/tests/unit/pipelex/migration/test_ledger_models.py b/tests/unit/pipelex/migration/test_ledger_models.pyindex 1c3397055..0b0751e40 100644--- a/tests/unit/pipelex/migration/test_ledger_models.py+++ b/tests/unit/pipelex/migration/test_ledger_models.py@@ -12,7 +12,8 @@ import pytestfrom pydantic import ValidationErrorfrom pipelex.migration.exceptions import MigrationLedgerError-from pipelex.migration.ledger import MigrationLedger, MigrationSafety, ledgers_dir, load_ledger+from pipelex.migration.ledger import MigrationLedger, ledgers_dir, load_ledger+from pipelex.migration.safety import MigrationSafetyfrom pipelex.suggested_fix import DeleteKeyOp_SURFACE_BLOCK = {diff --git a/tests/unit/pipelex/migration/test_migration_run.py b/tests/unit/pipelex/migration/test_migration_run.pynew file mode 100644index 000000000..fa613a0f6--- /dev/null+++ b/tests/unit/pipelex/migration/test_migration_run.py@@ -0,0 +1,309 @@+"""What a real migration run is aimed at — the directories, the walk's depth, and the version floor.++`test_runner.py` covers what happens to one file. This module covers the questions that only have+answers once a run is pointed at a real machine: which directories it reaches, how deep it goes,+and which files it refuses before replaying anything over them.+"""++from pathlib import Path++import pytest+from pytest_mock import MockerFixture++from pipelex.migration.ledger import MigrationLedger+from pipelex.migration.plan import FileBlockedReason, MigrationReport+from pipelex.migration.run import config_directories_to_migrate+from pipelex.migration.runner import migrate_file+from pipelex.migration.surfaces import build_config_surface_registry+from pipelex.suggested_fix import RenameTableKeyOp+from pipelex.system.configuration.config_surface import declared_schema_version+from tests.unit.pipelex.migration.conftest import EntryBuilder, LedgerBuilder, SurfaceBuilder+from tests.unit.pipelex.migration.test_runner import MOMENT++CONFIG_LOADER = "pipelex.migration.run.config_manager"+++class TestWhichDirectoriesAreWalked:+ def test_the_global_directory_comes_first_and_the_project_one_second(self, tmp_path: Path, mocker: MockerFixture) -> None:+ global_dir = tmp_path / "home" / ".pipelex"+ project_dir = tmp_path / "project" / ".pipelex"+ global_dir.mkdir(parents=True)+ project_dir.mkdir(parents=True)+ mocker.patch(CONFIG_LOADER).global_config_dir = global_dir+ mocker.patch(f"{CONFIG_LOADER}.project_config_dir", project_dir)++ assert config_directories_to_migrate() == [global_dir, project_dir]++ def test_a_machine_with_no_project_directory_is_an_ordinary_machine(self, tmp_path: Path, mocker: MockerFixture) -> None:+ global_dir = tmp_path / "home" / ".pipelex"+ global_dir.mkdir(parents=True)+ mocker.patch(CONFIG_LOADER).global_config_dir = global_dir+ mocker.patch(f"{CONFIG_LOADER}.project_config_dir", None)++ assert config_directories_to_migrate() == [global_dir]++ def test_a_global_directory_that_does_not_exist_is_skipped(self, tmp_path: Path, mocker: MockerFixture) -> None:+ mocker.patch(CONFIG_LOADER).global_config_dir = tmp_path / "nowhere" / ".pipelex"+ mocker.patch(f"{CONFIG_LOADER}.project_config_dir", None)++ assert config_directories_to_migrate() == []++ def test_a_project_rooted_at_the_home_directory_is_walked_once(self, tmp_path: Path, mocker: MockerFixture) -> None:+ """Every file would otherwise be migrated twice, and the second pass would back up the first.++ A backup of an already-migrated file is not a backup of anything, and it is the copy the+ pruning rotation would keep.+ """+ shared_dir = tmp_path / ".pipelex"+ shared_dir.mkdir(parents=True)+ mocker.patch(CONFIG_LOADER).global_config_dir = shared_dir+ mocker.patch(f"{CONFIG_LOADER}.project_config_dir", shared_dir)++ assert config_directories_to_migrate() == [shared_dir]+++class TestTheWalkIsNotRecursive:+ """A subdirectory of a configuration directory holds a different kind of thing.++ The specimen is real and it is the reason this is pinned rather than assumed:+ `.pipelex/inference/backends/pipelex_gateway.toml` matches the `pipelex-config` tier glob+ `pipelex_*.toml` exactly, and is an inference backend definition rather than a configuration+ surface file. A recursive walk would claim it, replay the main configuration's ledger over it,+ and rewrite it.+ """++ SPECIMEN = Path("inference") / "backends" / "pipelex_gateway.toml"++ def test_the_real_gateway_backend_file_is_not_claimed_where_it_actually_lives(self, tmp_path: Path) -> None:+ specimen = tmp_path / self.SPECIMEN+ specimen.parent.mkdir(parents=True)+ specimen.write_text("", encoding="utf-8")+ (tmp_path / "pipelex.toml").write_text("", encoding="utf-8")++ claimed = build_config_surface_registry().files_by_surface_in_directory(directory=tmp_path)++ assert [path.name for _, path in claimed] == ["pipelex.toml"]++ def test_the_same_file_name_at_the_top_level_is_claimed(self, tmp_path: Path) -> None:+ """The other half of the claim: depth is what saves the specimen, not its name.++ Without this, the test above would still pass if the tier glob stopped matching, and it+ would be proving nothing about recursion at all.+ """+ (tmp_path / self.SPECIMEN.name).write_text("", encoding="utf-8")++ claimed = build_config_surface_registry().files_by_surface_in_directory(directory=tmp_path)++ assert [(surface.surface_id, path.name) for surface, path in claimed] == [("pipelex-config", self.SPECIMEN.name)]+++class TestTheSchemaVersionFloor:+ """A file that declares where it stands, against a ledger that says how far back it reaches.++ This is the one thing a replay cannot work out for itself: the applier skips an operation whose+ target is absent and reports success, so a ledger that no longer carries the oldest entries+ would run over a file older than them, change nothing, and call it clean.+ """++ def _stale_file(self, *, tmp_path: Path, declared: int | None) -> Path:+ target = tmp_path / "example.toml"+ header = f"[meta]\nschema_version = {declared}\n\n" if declared is not None else ""+ target.write_text(f"{header}[reporting]\noutput_config = 'out'\n", encoding="utf-8")+ return target++ def _ledger(self, *, build_entry: EntryBuilder, build_ledger: LedgerBuilder, floor: int) -> MigrationLedger:+ return build_ledger(+ entries=[+ build_entry(to_schema_version=2, ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")]),+ build_entry(to_schema_version=3, ops=[RenameTableKeyOp(table_path=["reporting"], key="output", new_key="destination")]),+ ],+ min_supported_schema_version=floor,+ )++ @pytest.mark.parametrize("declared", [1, 0])+ def test_a_file_declaring_a_version_below_the_floor_is_refused_and_left_alone(+ self,+ declared: int,+ tmp_path: Path,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ build_surface: SurfaceBuilder,+ ) -> None:+ target = self._stale_file(tmp_path=tmp_path, declared=declared)+ original = target.read_text(encoding="utf-8")+ ledger = self._ledger(build_entry=build_entry, build_ledger=build_ledger, floor=2)++ plan = migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=False, moment=MOMENT)++ assert plan.blocked_reason is FileBlockedReason.UNSUPPORTED_SCHEMA_VERSION+ assert plan.blocked_detail is not None+ assert f"declares schema version {declared}" in plan.blocked_detail+ assert not plan.was_written+ assert target.read_text(encoding="utf-8") == original++ def test_a_file_declaring_exactly_the_floor_is_migrated(+ self,+ tmp_path: Path,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ build_surface: SurfaceBuilder,+ ) -> None:+ """The floor is the oldest version the ledger still reaches, not the first one it refuses."""+ target = self._stale_file(tmp_path=tmp_path, declared=2)+ ledger = self._ledger(build_entry=build_entry, build_ledger=build_ledger, floor=2)++ plan = migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=False, moment=MOMENT)++ assert plan.blocked_reason is None+ assert plan.was_written++ def test_a_file_declaring_nothing_is_migrated(+ self,+ tmp_path: Path,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ build_surface: SurfaceBuilder,+ ) -> None:+ """Which is every file in the field: nothing writes the reserved key."""+ target = self._stale_file(tmp_path=tmp_path, declared=None)+ ledger = self._ledger(build_entry=build_entry, build_ledger=build_ledger, floor=2)++ plan = migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=False, moment=MOMENT)++ assert plan.blocked_reason is None+ assert plan.was_written+++class TestReadingTheDeclaredSchemaVersion:+ """The reserved key is read exactly as tolerantly as boot strips it, and no more."""++ def test_a_declared_integer_is_read(self) -> None:+ assert declared_schema_version(config_dict={"meta": {"schema_version": 4}}) == 4++ @pytest.mark.parametrize(+ "config_dict",+ [+ {},+ {"meta": {}},+ {"meta": {"other": 1}},+ {"meta": "not a table"},+ {"meta": {"schema_version": "2"}},+ {"meta": {"schema_version": 2.0}},+ {"meta": {"schema_version": True}},+ ],+ ids=["absent", "empty-meta", "other-key", "meta-is-not-a-table", "string", "float", "bool"],+ )+ def test_anything_that_is_not_a_plain_integer_is_no_declaration_at_all(self, config_dict: dict[str, object]) -> None:+ """A malformed declaration boots fine, because the strip removes the key whatever it holds.++ Acting on it here would make migration stricter than the reader the key exists for — and+ `True` is the trap: it is an `int` to Python and never a schema version.+ """+ assert declared_schema_version(config_dict=config_dict) is None+++class TestTheDocumentTheDiagnosisReadsIsTheOneTheRunLeaves:+ """A run diagnoses what it is about to leave behind, not what it found.++ The distinction is invisible on a file that needed nothing and is the whole answer on a file+ that did: every stale key the ledger repairs would be reported as unexplained if the diagnosis+ read the document as it arrived.+ """++ def _ledger(self, *, build_entry: EntryBuilder, build_ledger: LedgerBuilder) -> MigrationLedger:+ return build_ledger(+ entries=[build_entry(to_schema_version=2, ops=[RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output")])]+ )++ def test_a_key_the_ledger_renames_is_not_reported_as_unexplained(+ self,+ tmp_path: Path,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ build_surface: SurfaceBuilder,+ ) -> None:+ target = tmp_path / "example.toml"+ target.write_text('[reporting]\noutput_config = { directory = "out" }\n', encoding="utf-8")++ plan = migrate_file(+ surface=build_surface(),+ ledger=self._ledger(build_entry=build_entry, build_ledger=build_ledger),+ file_path=target,+ dry_run=False,+ moment=MOMENT,+ )++ assert plan.steps, "the scenario is worthless unless the rename actually fired"+ assert plan.unexplained == []++ def test_a_dry_run_diagnoses_the_document_it_would_have_written(+ self,+ tmp_path: Path,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ build_surface: SurfaceBuilder,+ ) -> None:+ """The document only exists in memory here, which is why the diagnosis belongs to the run."""+ target = tmp_path / "example.toml"+ target.write_text('[reporting]\noutput_config = { directory = "out" }\nretires = 3\n', encoding="utf-8")++ plan = migrate_file(+ surface=build_surface(),+ ledger=self._ledger(build_entry=build_entry, build_ledger=build_ledger),+ file_path=target,+ dry_run=True,+ moment=MOMENT,+ )++ assert not plan.was_written+ assert [unexplained.path for unexplained in plan.unexplained] == ["reporting.retires"]+++class TestADryRunPredictsExactlyWhatAWritePassRewrites:+ """`changed_plans` on a dry run must name the same files `written_plans` names after a write.++ The runner writes a file whenever anything applied to it — a whole entry, or the part of a+ conflicting one that landed before the conflict. A dry run that only counted whole entries+ would tell the interactive command and the doctor there is nothing to run on a file the write+ pass then rewrites.+ """++ def _half_migrated_file(self, *, tmp_path: Path) -> Path:+ """`output_config` moves; `format` finds its destination already taken and blocks the entry."""+ target = tmp_path / "example.toml"+ target.write_text('[reporting]\noutput_config = "out"\nformat = "json"\ndestination_format = "toml"\n', encoding="utf-8")+ return target++ def _ledger(self, *, build_entry: EntryBuilder, build_ledger: LedgerBuilder) -> MigrationLedger:+ return build_ledger(+ entries=[+ build_entry(+ to_schema_version=2,+ ops=[+ RenameTableKeyOp(table_path=["reporting"], key="output_config", new_key="output"),+ RenameTableKeyOp(table_path=["reporting"], key="format", new_key="destination_format"),+ ],+ )+ ]+ )++ def test_a_partly_applied_entry_counts_as_a_change_on_both_passes(+ self,+ tmp_path: Path,+ build_entry: EntryBuilder,+ build_ledger: LedgerBuilder,+ build_surface: SurfaceBuilder,+ ) -> None:+ target = self._half_migrated_file(tmp_path=tmp_path)+ ledger = self._ledger(build_entry=build_entry, build_ledger=build_ledger)++ rehearsal = migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=True, moment=MOMENT)+ assert rehearsal.steps == [], "the scenario is worthless unless the entry actually blocks"+ assert rehearsal.blocked, "…and worthless unless the block is an entry's"+ assert rehearsal.blocked[0].applied_ops, "…and worthless unless part of it applied first"++ dry = MigrationReport(plans=[rehearsal])+ write = MigrationReport(plans=[migrate_file(surface=build_surface(), ledger=ledger, file_path=target, dry_run=False, moment=MOMENT)])++ assert [plan.file_path for plan in write.written_plans] == [target]+ assert [plan.file_path for plan in dry.changed_plans] == [plan.file_path for plan in write.written_plans]diff --git a/tests/unit/pipelex/migration/test_real_surfaces.py b/tests/unit/pipelex/migration/test_real_surfaces.pyindex 15af09e3b..ff6a78d89 100644--- a/tests/unit/pipelex/migration/test_real_surfaces.py+++ b/tests/unit/pipelex/migration/test_real_surfaces.py@@ -9,10 +9,13 @@ specific key, a particular default — and every legitimate configuration changethat is not about configuration, which teaches the reflex the whole design exists to prevent."""+import tomlkit+from pipelex.migration.coverage import check_registry-from pipelex.migration.ledger import INITIAL_SCHEMA_VERSION, load_ledger+from pipelex.migration.diagnosis import diagnose_unexplained_paths+from pipelex.migration.ledger import INITIAL_SCHEMA_VERSION, load_ledger, packaged_migration_dirfrom pipelex.migration.ledger_check import check_ledgers-from pipelex.migration.surfaces import build_config_surface_registry, packaged_migration_dir+from pipelex.migration.surfaces import build_config_surface_registryfrom pipelex.migration.transform_check import check_transforms@@ -30,6 +33,26 @@ class TestTheRealRegistry:for surface in build_config_surface_registry().surfaces:assert surface.read_defaults_document(), f"surface '{surface.surface_id}' supplies no defaults"+ def test_no_surfaces_own_documents_carry_anything_the_diagnosis_cannot_explain(self) -> None:+ """The downgrade diagnosis, over the two documents it must always be silent about.++ Both are at the current schema by construction — one is the defaults layer, the other the+ starter file `pipelex init` copies — so a path it cannot account for in either is the+ diagnosis being wrong about our own schema, not the document being stale.+ """+ migration_dir = packaged_migration_dir()+ for surface in build_config_surface_registry().surfaces:+ ledger = load_ledger(migration_dir=migration_dir, surface_id=surface.surface_id)+ fingerprint = surface.fingerprint_at(schema_version=ledger.surface.current_schema_version)+ for label, text in surface.reference_documents():+ unexplained = diagnose_unexplained_paths(+ fingerprint=fingerprint,+ document=tomlkit.loads(text).unwrap(),+ ledger=ledger,+ blocked=[],+ )+ assert unexplained == [], f"{surface.surface_id} ({label}): {[found.path for found in unexplained]}"+def test_every_surface_has_a_ledger(self) -> None:migration_dir = packaged_migration_dir()for surface in build_config_surface_registry().surfaces:diff --git a/tests/unit/pipelex/migration/test_replay_properties.py b/tests/unit/pipelex/migration/test_replay_properties.pyindex dcb59e84a..8a4b315a1 100644--- a/tests/unit/pipelex/migration/test_replay_properties.py+++ b/tests/unit/pipelex/migration/test_replay_properties.py@@ -27,8 +27,9 @@ from pydantic import BaseModel, ConfigDict, Fieldfrom pipelex.migration.engine import replay_ledger_over_textfrom pipelex.migration.fingerprint import ENUM_TYPE-from pipelex.migration.ledger import MigrationEntry, MigrationLedger, MigrationSafety, SurfaceBlock, load_ledger-from pipelex.migration.surfaces import DefaultsLayerKind, Surface, build_config_surface_registry, packaged_migration_dir+from pipelex.migration.ledger import MigrationEntry, MigrationLedger, SurfaceBlock, load_ledger, packaged_migration_dir+from pipelex.migration.safety import MigrationSafety+from pipelex.migration.surfaces import DefaultsLayerKind, Surface, build_config_surface_registryfrom pipelex.suggested_fix import DeleteKeyOp, DeleteTableOp, MigrationOp, MoveKeyOp, RemapValueOp, RenameTableKeyOpfrom tests.unit.pipelex.migration.document_strategies import (DocumentMutation,diff --git a/tests/unit/pipelex/migration/test_telemetry_back_entry.py b/tests/unit/pipelex/migration/test_telemetry_back_entry.pyindex 5a2debaea..455059cd7 100644--- a/tests/unit/pipelex/migration/test_telemetry_back_entry.py+++ b/tests/unit/pipelex/migration/test_telemetry_back_entry.py@@ -14,8 +14,7 @@ from typing import Anyfrom pipelex.migration.engine import replay_ledger_over_textfrom pipelex.migration.goldens import pre_history_document_path-from pipelex.migration.ledger import load_ledger-from pipelex.migration.surfaces import packaged_migration_dir+from pipelex.migration.ledger import load_ledger, packaged_migration_dirfrom pipelex.system.telemetry.telemetry_config import PostHogMode, TelemetryConfigfrom pipelex.tools.misc.toml_utils import load_toml_from_contentdiff --git a/tests/unit/pipelex/migration/test_unsafe_promise.py b/tests/unit/pipelex/migration/test_unsafe_promise.pyindex 5cffbceb2..356d3ce86 100644--- a/tests/unit/pipelex/migration/test_unsafe_promise.py+++ b/tests/unit/pipelex/migration/test_unsafe_promise.py@@ -21,9 +21,10 @@ forever."""from pipelex.migration.engine import replay_ledger_over_text-from pipelex.migration.ledger import MigrationLedger, MigrationSafety+from pipelex.migration.ledger import MigrationLedgerfrom pipelex.migration.material import declared_path_spellings, unsafe_op_variantsfrom pipelex.migration.plan import BlockedEntryReason+from pipelex.migration.safety import MigrationSafetyfrom pipelex.suggested_fix import DeleteTableOp, RemapValueOp, RenameTableKeyOpfrom tests.unit.pipelex.migration.conftest import EntryBuilder, LedgerBuilderdiff --git a/tests/unit/pipelex/system/configuration/test_boot_tolerance.py b/tests/unit/pipelex/system/configuration/test_boot_tolerance.pynew file mode 100644index 000000000..43523a4cd--- /dev/null+++ b/tests/unit/pipelex/system/configuration/test_boot_tolerance.py@@ -0,0 +1,491 @@+"""Boot tolerance — a stale configuration warns instead of stopping the world.++The failure this is about is ordinary and unpleasant: a schema change lands, the file on the+machine is the old shape, and the next boot dies on `extra="forbid"` with a pydantic error naming+a key the user never chose to have. Tolerance replays the surface's ledger over the same files+**in memory**, validates what comes back, and boots with a warning if that succeeds.++Two properties carry the whole design, and most of the tests below exist to hold one of them:++- **Boot tolerates only what the ledger explains.** Anything else raises exactly what it raised+ before. The retry either recovers or gets out of the way — it never softens an error.+- **Boot never writes.** Only the explicit `migrate` command does. A tolerated boot leaves the+ file exactly as it found it, backups included, which is why the same warning keeps appearing+ until the user runs the command.++The stale documents here are **real**: `telemetry-config@2` is the entry the package ships and+`goldens/telemetry-config/[email protected]` is the flat document it exists to carry forward, read+live rather than copied. The other two surfaces have empty ledgers today, so their tests plant a+synthetic one in a temporary migration directory — the wiring is what is under test there, and a+synthetic ledger tests it without inventing a schema change nobody made.+"""++from __future__ import annotations++from pathlib import Path+from typing import TYPE_CHECKING++import pytest++from pipelex import log+from pipelex.migration.goldens import pre_history_document_path+from pipelex.migration.ledger import packaged_migration_dir+from pipelex.system.configuration import config_surface as config_surface_module+from pipelex.system.configuration.config_loader import ConfigLoader+from pipelex.system.configuration.config_surface import (+ PIPELEX_CONFIG_SURFACE_ID,+ PIPELEX_SERVICE_CONFIG_SURFACE_ID,+ TELEMETRY_CONFIG_SURFACE_ID,+ replay_surface_files_in_memory,+ stale_configuration_warning,+)+from pipelex.system.configuration.configs import PipelexConfig+from pipelex.system.exceptions import ConfigValidationError+from pipelex.system.pipelex_service.exceptions import PipelexServiceConfigValidationError+from pipelex.system.pipelex_service.pipelex_service_config import load_pipelex_service_config_if_exists+from pipelex.system.telemetry import telemetry_loader as telemetry_loader_module+from pipelex.system.telemetry.exceptions import TelemetryConfigValidationError+from pipelex.system.telemetry.telemetry_config import PostHogMode, TelemetryConfig+from pipelex.system.telemetry.telemetry_loader import load_telemetry_config+from pipelex.tools.log.log_dispatch import LogDispatch+from pipelex.tools.log.log_levels import LogLevel+from pipelex.tools.secrets.env_secrets_provider import EnvSecretsProvider++if TYPE_CHECKING:+ from pytest_mock import MockerFixture+++def old_shape_telemetry_document() -> str:+ """The flat pre-`[custom_posthog]` document the shipped entry is about, read from the package."""+ path = pre_history_document_path(migration_dir=packaged_migration_dir(), surface_id=TELEMETRY_CONFIG_SURFACE_ID, schema_version=2)+ return path.read_text(encoding="utf-8")+++def write_synthetic_ledger(*, migration_dir: Path, surface_id: str, base_file: str, ops_body: str, min_supported_schema_version: int = 0) -> None:+ """A one-entry ledger for a surface whose real ledger has nothing in it yet."""+ ledgers_dir = migration_dir / "ledgers"+ ledgers_dir.mkdir(parents=True, exist_ok=True)+ (ledgers_dir / f"{surface_id}.toml").write_text(+ f"""+[surface]+id = "{surface_id}"+title = "A surface under test"+base_file = "{base_file}"+current_schema_version = 2+min_supported_schema_version = {min_supported_schema_version}++[[migration]]+id = "{surface_id}@2"+to_schema_version = 2+introduced_in = "0.46.0"+breaking = true+safety = "safe"+title = "Give the setting its current name"+description = "The setting was renamed and this entry carries a file written before that onto the current shape."++{ops_body}+""",+ encoding="utf-8",+ )+++def fake_dirs(tmp_path: Path, mocker: MockerFixture) -> tuple[Path, Path]:+ """A fake home and project tree, so a loader reads test files rather than the machine's own."""+ fake_home = tmp_path / "home"+ global_dir = fake_home / ".pipelex"+ global_dir.mkdir(parents=True)++ project_root = tmp_path / "project"+ (project_root / ".git").mkdir(parents=True)+ project_dir = project_root / ".pipelex"+ project_dir.mkdir()++ mocker.patch.object(Path, "home", return_value=fake_home)+ mocker.patch.object(Path, "cwd", return_value=project_root)++ return global_dir, project_dir+++def secrets_provider() -> EnvSecretsProvider:+ return EnvSecretsProvider()+++def synthetic_migration_dir(tmp_path: Path, mocker: MockerFixture) -> Path:+ """Point the boot-tolerance helper at a migration directory this test owns."""+ migration_dir = tmp_path / "migration"+ mocker.patch.object(config_surface_module, "packaged_migration_dir", return_value=migration_dir)+ return migration_dir+++class TestReplayingASurfaceInMemory:+ """The shared helper, against the ledger and the old-shape document the package ships."""++ def test_a_current_file_is_nothing_the_ledger_has_to_say_about(self, tmp_path: Path) -> None:+ """`None` is the answer that keeps a healthy machine out of every branch below."""+ current = tmp_path / "telemetry.toml"+ current.write_text('[custom_posthog]\nmode = "off"\n', encoding="utf-8")++ assert replay_surface_files_in_memory(surface_id=TELEMETRY_CONFIG_SURFACE_ID, paths=[current]) is None++ def test_an_old_shape_file_comes_back_carried_forward(self, tmp_path: Path) -> None:+ stale = tmp_path / "telemetry.toml"+ stale.write_text(old_shape_telemetry_document(), encoding="utf-8")++ replayed = replay_surface_files_in_memory(surface_id=TELEMETRY_CONFIG_SURFACE_ID, paths=[stale])++ assert replayed is not None+ assert "telemetry_mode" not in replayed.config_dict, "the flat key is what the entry moves away"+ assert TelemetryConfig.model_validate(replayed.config_dict).custom_posthog is not None+ assert [step.entry_id for step in replayed.plans[0].steps] == ["telemetry-config@2"]++ def test_the_file_on_disk_is_not_touched(self, tmp_path: Path) -> None:+ """Boot never writes. Only `pipelex migrate` does, which is why the warning keeps coming back."""+ stale = tmp_path / "telemetry.toml"+ stale.write_text(old_shape_telemetry_document(), encoding="utf-8")+ before = stale.read_bytes()++ replay_surface_files_in_memory(surface_id=TELEMETRY_CONFIG_SURFACE_ID, paths=[stale])++ assert stale.read_bytes() == before+ assert list(tmp_path.iterdir()) == [stale], "a boot leaves no backup beside the file either"++ def test_a_missing_file_is_skipped_and_the_files_beside_it_still_merge(self, tmp_path: Path) -> None:+ """Every layer of a surface is optional, exactly as it is on the ordinary load path."""+ stale = tmp_path / "telemetry.toml"+ stale.write_text(old_shape_telemetry_document(), encoding="utf-8")++ replayed = replay_surface_files_in_memory(+ surface_id=TELEMETRY_CONFIG_SURFACE_ID,+ paths=[tmp_path / "telemetry_override.toml", stale, tmp_path / "telemetry_local.toml"],+ )++ assert replayed is not None+ assert len(replayed.plans) == 1, "a file that is not there is not a file the report is about"++ def test_a_later_layer_wins_on_a_key_both_carry(self, tmp_path: Path) -> None:+ """The merge order is the loader's, so the tolerated configuration is the same one."""+ base = tmp_path / "telemetry.toml"+ base.write_text('telemetry_mode = "off"\n', encoding="utf-8")+ override = tmp_path / "telemetry_override.toml"+ override.write_text('telemetry_mode = "anonymous"\n', encoding="utf-8")++ replayed = replay_surface_files_in_memory(surface_id=TELEMETRY_CONFIG_SURFACE_ID, paths=[base, override])++ assert replayed is not None+ assert replayed.config_dict["custom_posthog"]["mode"] == PostHogMode.ANONYMOUS++ def test_a_file_that_cannot_be_parsed_abandons_the_whole_retry(self, tmp_path: Path) -> None:+ """Skipping it would drop a layer, and a retry that then validated would boot on a+ configuration the user does not have. The file's own parse error is the one to show.+ """+ stale = tmp_path / "telemetry.toml"+ stale.write_text(old_shape_telemetry_document(), encoding="utf-8")+ broken = tmp_path / "telemetry_override.toml"+ broken.write_text("this is not = = toml\n", encoding="utf-8")++ assert replay_surface_files_in_memory(surface_id=TELEMETRY_CONFIG_SURFACE_ID, paths=[stale, broken]) is None++ def test_the_reserved_meta_table_is_stripped(self, tmp_path: Path) -> None:+ """The helper replaces the loader's read-and-merge, so it owes the strip that came with it."""+ stale = tmp_path / "telemetry.toml"+ # Root keys before the table header, because a header absorbs every scalar that follows it.+ stale.write_text('telemetry_mode = "off"\n\n[meta]\nschema_version = 1\n', encoding="utf-8")++ replayed = replay_surface_files_in_memory(surface_id=TELEMETRY_CONFIG_SURFACE_ID, paths=[stale])++ assert replayed is not None+ assert "meta" not in replayed.config_dict+++class TestTheRetryHonoursTheSchemaVersionFloor:+ """A file older than the ledger reaches is refused by `pipelex migrate`; the retry must not boot it.++ The applier skips an absent target and reports success, so a ledger whose oldest entries were+ squashed away would carry such a file "forward" while leaving it under-migrated — and the boot+ would then tell the user to run a command that declines the same file. The retry gets out of+ the way instead, and the error path's scan reports the floor for what it is.+ """++ OPS = (+ '[[migration.ops]]\nkind = "rename_table_key"\ntable_path = ["pipelex", "log_config"]\n'+ 'key = "old_default_log_level"\nnew_key = "default_log_level"\n'+ )++ def _stale_file(self, *, directory: Path, declared: int) -> Path:+ stale = directory / "pipelex.toml"+ stale.write_text(f'[meta]\nschema_version = {declared}\n\n[pipelex.log_config]\nold_default_log_level = "DEBUG"\n', encoding="utf-8")+ return stale++ def test_a_file_declaring_a_version_below_the_floor_declines_the_retry(self, tmp_path: Path, synthetic_migration_dir: Path) -> None:+ write_synthetic_ledger(+ migration_dir=synthetic_migration_dir,+ surface_id=PIPELEX_CONFIG_SURFACE_ID,+ base_file="pipelex.toml",+ ops_body=self.OPS,+ min_supported_schema_version=2,+ )+ stale = self._stale_file(directory=tmp_path, declared=1)++ assert replay_surface_files_in_memory(surface_id=PIPELEX_CONFIG_SURFACE_ID, paths=[stale]) is None++ def test_a_file_declaring_the_floor_itself_is_still_carried_forward(self, tmp_path: Path, synthetic_migration_dir: Path) -> None:+ """The floor is the oldest version the ledger still reaches, not the first one it refuses."""+ write_synthetic_ledger(+ migration_dir=synthetic_migration_dir,+ surface_id=PIPELEX_CONFIG_SURFACE_ID,+ base_file="pipelex.toml",+ ops_body=self.OPS,+ min_supported_schema_version=2,+ )+ stale = self._stale_file(directory=tmp_path, declared=2)++ replayed = replay_surface_files_in_memory(surface_id=PIPELEX_CONFIG_SURFACE_ID, paths=[stale])++ assert replayed is not None+ assert replayed.config_dict["pipelex"]["log_config"]["default_log_level"] == "DEBUG"++ def test_the_loader_then_raises_the_users_own_error(self, fake_dirs: tuple[Path, Path], synthetic_migration_dir: Path) -> None:+ """Which names the stale key — and whose `migration` block is where the floor gets reported."""+ global_dir, _ = fake_dirs+ write_synthetic_ledger(+ migration_dir=synthetic_migration_dir,+ surface_id=PIPELEX_CONFIG_SURFACE_ID,+ base_file="pipelex.toml",+ ops_body=self.OPS,+ min_supported_schema_version=2,+ )+ self._stale_file(directory=global_dir, declared=1)++ with pytest.raises(ConfigValidationError, match="old_default_log_level"):+ ConfigLoader().load_config_validated(config_cls=PipelexConfig)+++class TestWhatTheWarningSays:+ def test_it_names_the_file_and_the_remedy(self, tmp_path: Path) -> None:+ stale = tmp_path / "telemetry.toml"+ stale.write_text(old_shape_telemetry_document(), encoding="utf-8")+ replayed = replay_surface_files_in_memory(surface_id=TELEMETRY_CONFIG_SURFACE_ID, paths=[stale])+ assert replayed is not None++ warning = stale_configuration_warning(plans=replayed.plans)++ assert str(stale) in warning+ assert "pipelex migrate" in warning+ assert "Nothing was written" in warning++ def test_it_says_what_the_ledger_carried_and_nothing_read_from_the_file(self, tmp_path: Path) -> None:+ """Ledger text only, the same rule the migration report obeys — a boot warning is read in+ the same places a report is, and a user's own values have no business in either.+ """+ stale = tmp_path / "telemetry.toml"+ stale.write_text('telemetry_mode = "off"\nproject_api_key = "phc_a_secret_the_user_owns"\n', encoding="utf-8")+ replayed = replay_surface_files_in_memory(surface_id=TELEMETRY_CONFIG_SURFACE_ID, paths=[stale])+ assert replayed is not None++ warning = stale_configuration_warning(plans=replayed.plans)++ assert "Nest the flat telemetry settings under [custom_posthog]" in warning+ assert "phc_a_secret_the_user_owns" not in warning+++class TestTheTelemetryLoader:+ """The one surface whose shipped ledger already carries a real entry."""++ def test_an_old_shape_file_boots_with_a_warning(+ self,+ fake_dirs: tuple[Path, Path],+ secrets_provider: EnvSecretsProvider,+ mocker: MockerFixture,+ ) -> None:+ global_dir, _ = fake_dirs+ stale = global_dir / "telemetry.toml"+ stale.write_text(old_shape_telemetry_document(), encoding="utf-8")+ before = stale.read_bytes()+ warning = mocker.patch.object(telemetry_loader_module.log, "warning")++ config = load_telemetry_config(secrets_provider=secrets_provider)++ assert config.custom_posthog is not None+ assert stale.read_bytes() == before, "a tolerated boot writes nothing"+ assert "pipelex migrate" in warning.call_args.args[0]++ def test_a_file_the_ledger_cannot_explain_still_raises(+ self,+ fake_dirs: tuple[Path, Path],+ secrets_provider: EnvSecretsProvider,+ ) -> None:+ """Tolerance widens what starts; it never widens what is silently accepted."""+ global_dir, _ = fake_dirs+ (global_dir / "telemetry.toml").write_text("not_a_telemetry_setting = true\n", encoding="utf-8")++ with pytest.raises(TelemetryConfigValidationError):+ load_telemetry_config(secrets_provider=secrets_provider)++ def test_a_file_the_ledger_only_half_explains_raises_the_original_error(+ self,+ fake_dirs: tuple[Path, Path],+ secrets_provider: EnvSecretsProvider,+ ) -> None:+ """The branch where the retry does real work and still does not get there.++ This file is flat *and* carries a key no telemetry schema ever had, so the replay applies+ — the entry's operations fire — and the re-validation fails anyway. Tolerance widens what+ starts, never what is accepted, so what the user sees is the error their file produces.+ """+ global_dir, _ = fake_dirs+ (global_dir / "telemetry.toml").write_text('telemetry_mode = "off"\nnot_a_telemetry_setting = true\n', encoding="utf-8")++ with pytest.raises(TelemetryConfigValidationError, match="not_a_telemetry_setting"):+ load_telemetry_config(secrets_provider=secrets_provider)++ def test_an_old_shape_tier_file_is_carried_forward_too(+ self,+ fake_dirs: tuple[Path, Path],+ secrets_provider: EnvSecretsProvider,+ ) -> None:+ """The retry covers every layer the loader merged, not only the base file."""+ global_dir, project_dir = fake_dirs+ (global_dir / "telemetry.toml").write_text('[custom_posthog]\nmode = "off"\n', encoding="utf-8")+ (project_dir / "telemetry_override.toml").write_text('telemetry_mode = "anonymous"\n', encoding="utf-8")++ config = load_telemetry_config(secrets_provider=secrets_provider)++ assert config.custom_posthog is not None+ assert config.custom_posthog.mode is PostHogMode.ANONYMOUS++ @pytest.mark.usefixtures("fake_dirs")+ def test_a_healthy_configuration_never_reaches_the_ledger(+ self,+ secrets_provider: EnvSecretsProvider,+ mocker: MockerFixture,+ ) -> None:+ """The replay runs on the failure path only, so a current machine pays nothing for it."""+ retry = mocker.spy(telemetry_loader_module, "replay_surface_files_in_memory")++ load_telemetry_config(secrets_provider=secrets_provider)++ assert retry.call_count == 0+++class TestTheMainConfigurationLoader:+ def test_an_old_shape_file_is_validated_after_the_replay(+ self,+ fake_dirs: tuple[Path, Path],+ synthetic_migration_dir: Path,+ mocker: MockerFixture,+ ) -> None:+ """And the *migrated* value is the one that lands, not merely a configuration that parses.++ The main configuration is what *configures logging*, so at this point in a boot no logger+ exists yet: the warning is parked on the loader for the boot to emit once it does. The+ unconfigured dispatch here is a cold boot's, and it raises on any attempt to log through it.+ """+ global_dir, _ = fake_dirs+ write_synthetic_ledger(+ migration_dir=synthetic_migration_dir,+ surface_id=PIPELEX_CONFIG_SURFACE_ID,+ base_file="pipelex.toml",+ ops_body='[[migration.ops]]\nkind = "rename_table_key"\ntable_path = ["pipelex", "log_config"]\n'+ 'key = "old_default_log_level"\nnew_key = "default_log_level"\n',+ )+ (global_dir / "pipelex.toml").write_text('[pipelex.log_config]\nold_default_log_level = "DEBUG"\n', encoding="utf-8")+ mocker.patch.object(log, "log_dispatch", LogDispatch())+ loader = ConfigLoader()++ config = loader.load_config_validated(config_cls=PipelexConfig)++ assert config.pipelex.log_config.default_log_level is LogLevel.DEBUG+ parked = loader.take_stale_configuration_warning()+ assert parked is not None+ assert "pipelex migrate" in parked+ assert loader.take_stale_configuration_warning() is None, "a warning is emitted once"++ @pytest.mark.usefixtures("fake_dirs")+ def test_a_healthy_configuration_parks_no_warning(self) -> None:+ loader = ConfigLoader()++ loader.load_config_validated(config_cls=PipelexConfig)++ assert loader.take_stale_configuration_warning() is None++ def test_programmatic_overrides_are_re_applied_over_the_replay(+ self,+ fake_dirs: tuple[Path, Path],+ synthetic_migration_dir: Path,+ ) -> None:+ """They are a layer of the load rather than a property of the files, and the replay only+ ever sees the files — so a retry that forgot them would boot a different configuration.+ """+ global_dir, _ = fake_dirs+ write_synthetic_ledger(+ migration_dir=synthetic_migration_dir,+ surface_id=PIPELEX_CONFIG_SURFACE_ID,+ base_file="pipelex.toml",+ ops_body='[[migration.ops]]\nkind = "rename_table_key"\ntable_path = ["pipelex", "log_config"]\n'+ 'key = "old_default_log_level"\nnew_key = "default_log_level"\n',+ )+ (global_dir / "pipelex.toml").write_text('[pipelex.log_config]\nold_default_log_level = "DEBUG"\n', encoding="utf-8")++ config = ConfigLoader().load_config_validated(+ config_cls=PipelexConfig,+ extra_overrides={"pipelex": {"log_config": {"default_log_level": "WARNING"}}},+ )++ assert config.pipelex.log_config.default_log_level is LogLevel.WARNING++ @pytest.mark.usefixtures("fake_dirs", "synthetic_migration_dir")+ def test_a_configuration_the_ledger_cannot_explain_still_raises_the_users_own_error(self) -> None:+ """And the message still names the key, which is the only thing that tells them what to fix.++ The migration directory here holds no ledger at all, so the retry cannot even be attempted+ — the case that proves a failure *inside* the retry never replaces the failure outside it.+ """+ with pytest.raises(ConfigValidationError, match="not_a_real_setting"):+ ConfigLoader().load_config_validated(+ config_cls=PipelexConfig,+ extra_overrides={"not_a_real_setting": True},+ )++ @pytest.mark.usefixtures("fake_dirs")+ def test_a_healthy_configuration_never_reaches_the_ledger(self, mocker: MockerFixture) -> None:+ retry = mocker.spy(ConfigLoader, "_config_the_ledger_can_explain")++ ConfigLoader().load_config_validated(config_cls=PipelexConfig)++ assert retry.call_count == 0+++class TestTheServiceConfigLoader:+ """One file, no tiers — the merge the shared helper performs is a merge of one."""++ def test_an_old_shape_file_boots_with_a_warning(self, tmp_path: Path, synthetic_migration_dir: Path, mocker: MockerFixture) -> None:+ write_synthetic_ledger(+ migration_dir=synthetic_migration_dir,+ surface_id=PIPELEX_SERVICE_CONFIG_SURFACE_ID,+ base_file="pipelex_service.toml",+ ops_body='[[migration.ops]]\nkind = "rename_table_key"\ntable_path = []\nkey = "terms"\nnew_key = "agreement"\n',+ )+ config_dir = tmp_path / "config"+ config_dir.mkdir()+ stale = config_dir / "pipelex_service.toml"+ stale.write_text("[terms]\nterms_accepted = true\n", encoding="utf-8")+ warning = mocker.patch("pipelex.system.pipelex_service.pipelex_service_config.log.warning")++ config = load_pipelex_service_config_if_exists(config_dir=config_dir)++ assert config is not None+ assert config.agreement.terms_accepted is True+ assert stale.read_bytes() == b"[terms]\nterms_accepted = true\n", "a tolerated boot writes nothing"+ assert "pipelex migrate" in warning.call_args.args[0]++ @pytest.mark.usefixtures("synthetic_migration_dir")+ def test_a_file_the_ledger_cannot_explain_still_raises(self, tmp_path: Path) -> None:+ config_dir = tmp_path / "config"+ config_dir.mkdir()+ (config_dir / "pipelex_service.toml").write_text("[nonsense]\nwhat = true\n", encoding="utf-8")++ with pytest.raises(PipelexServiceConfigValidationError):+ load_pipelex_service_config_if_exists(config_dir=config_dir)diff --git a/tests/unit/pipelex/system/telemetry/test_load_telemetry_config.py b/tests/unit/pipelex/system/telemetry/test_load_telemetry_config.pyindex db7a04c27..1038a2303 100644--- a/tests/unit/pipelex/system/telemetry/test_load_telemetry_config.py+++ b/tests/unit/pipelex/system/telemetry/test_load_telemetry_config.py@@ -13,8 +13,8 @@ from typing import TYPE_CHECKINGimport pytestfrom pipelex.kit.paths import get_kit_configs_dir-from pipelex.system.telemetry import telemetry_config as telemetry_config_module-from pipelex.system.telemetry.telemetry_config import load_telemetry_config+from pipelex.system.telemetry import telemetry_loader as telemetry_loader_module+from pipelex.system.telemetry.telemetry_loader import load_telemetry_configfrom pipelex.tools.secrets.env_secrets_provider import EnvSecretsProviderif TYPE_CHECKING:@@ -154,7 +154,7 @@ class TestLoadTelemetryConfigLayering:mocker.patch.object(Path, "home", return_value=fake_home)mocker.patch.object(Path, "cwd", return_value=fake_home)- spy = mocker.spy(telemetry_config_module, "load_toml_from_path_and_merge_with_overrides")+ spy = mocker.spy(telemetry_loader_module, "load_toml_from_path_and_merge_with_overrides")config = load_telemetry_config(secrets_provider=secrets_provider)diff --git a/tests/unit/pipelex/system/telemetry/test_telemetry_stale_remedy.py b/tests/unit/pipelex/system/telemetry/test_telemetry_stale_remedy.pynew file mode 100644index 000000000..86348b08d--- /dev/null+++ b/tests/unit/pipelex/system/telemetry/test_telemetry_stale_remedy.py@@ -0,0 +1,142 @@+"""A telemetry configuration that will not load says whether it is old or wrong.++Before the ledger, every telemetry failure got one answer — *the format changed, run+`pipelex init telemetry`* — written for one real event and then shown for every other way a file+can fail. That remedy writes a fresh file, so on the very case it named it discarded the PostHog+key, the Langfuse credentials and the OTLP exporters it was supposed to be helping with.++Boot tolerance now carries an old file forward on its own, so anything reaching this error is+either a file the ledger cannot *fully* explain or one that is genuinely invalid. Telling those+two apart is what the migration block on the error is for, and every surface that reports the+failure — the human CLI, the agent envelope, the doctor — branches on that block rather than on+the wording.++The stale documents here are the package's own: `telemetry-config@2` is the shipped entry and+`goldens/telemetry-config/[email protected]` is the flat document it exists to carry forward.+"""++from __future__ import annotations++from pathlib import Path+from typing import TYPE_CHECKING++import pytest++from pipelex.core.validation import MIGRATE_COMMAND+from pipelex.migration.goldens import pre_history_document_path+from pipelex.migration.ledger import packaged_migration_dir+from pipelex.system.configuration.config_surface import TELEMETRY_CONFIG_SURFACE_ID+from pipelex.system.telemetry.exceptions import TelemetryConfigValidationError+from pipelex.system.telemetry.telemetry_loader import load_telemetry_config+from pipelex.tools.secrets.env_secrets_provider import EnvSecretsProvider++if TYPE_CHECKING:+ from pytest_mock import MockerFixture++# A realistic PostHog project key, planted in the file the error is raised about. The old flat+# format is exactly where a user's real one would be, which is what makes it the right specimen+# for the rendering rule on this channel.+PLANTED_KEY = "phc_L1VE_telemetry_key_that_must_never_be_rendered"++UNKNOWN_KEY = "not_a_telemetry_setting"+++def old_shape_document() -> str:+ """The flat pre-`[custom_posthog]` document the shipped entry is about, read from the package."""+ path = pre_history_document_path(migration_dir=packaged_migration_dir(), surface_id=TELEMETRY_CONFIG_SURFACE_ID, schema_version=2)+ return path.read_text(encoding="utf-8")+++def global_dir(tmp_path: Path, mocker: MockerFixture) -> Path:+ """A fake home, so the loader and the scan both read this test's files rather than the machine's."""+ fake_home = tmp_path / "home"+ config_dir = fake_home / ".pipelex"+ config_dir.mkdir(parents=True)++ project_root = tmp_path / "project"+ (project_root / ".git").mkdir(parents=True)+ (project_root / ".pipelex").mkdir()++ mocker.patch.object(Path, "home", return_value=fake_home)+ mocker.patch.object(Path, "cwd", return_value=project_root)++ return config_dir+++def load_and_capture(*, config_dir: Path, body: str) -> TelemetryConfigValidationError:+ """Write a telemetry file that will not load, and hand back the refusal it produces."""+ (config_dir / "telemetry.toml").write_text(body, encoding="utf-8")+ with pytest.raises(TelemetryConfigValidationError) as caught:+ load_telemetry_config(secrets_provider=EnvSecretsProvider())+ return caught.value+++class TestTheErrorSaysWhetherTheFileIsOldOrWrong:+ """The block is the verdict; a consumer branches on its presence and never on the wording."""++ def test_a_file_the_ledger_only_half_explains_carries_the_pending_migration(self, global_dir: Path) -> None:+ """The reachable specimen, and the one boot tolerance leaves behind.++ A file that is flat *and* carries a key no telemetry schema ever had: the replay does real+ work, the re-validation fails anyway, and what the user is left holding is a file that is+ both old and wrong. Saying only the second half is what sent them to a destructive remedy.+ """+ exc = load_and_capture(config_dir=global_dir, body=f"{old_shape_document()}\n{UNKNOWN_KEY} = true\n")++ assert exc.migration is not None+ assert exc.migration.remedy == MIGRATE_COMMAND+ assert [plan.file_path for plan in exc.migration.plans] == [global_dir / "telemetry.toml"]+ assert MIGRATE_COMMAND in exc.message++ def test_a_file_that_is_simply_wrong_carries_no_block_at_all(self, global_dir: Path) -> None:+ """Absence is a verdict too — this configuration is not stale, and no command repairs it."""+ exc = load_and_capture(config_dir=global_dir, body='[custom_posthog]\nmode = "no-such-mode"\n')++ assert exc.migration is None+ assert MIGRATE_COMMAND not in exc.message+ assert "custom_posthog.mode" in exc.message++ def test_a_key_no_build_knows_is_named_even_with_nothing_to_migrate(self, global_dir: Path) -> None:+ """The diagnosis speaks on its own: an unknown root key is a finding without a single step."""+ exc = load_and_capture(config_dir=global_dir, body=f"{UNKNOWN_KEY} = true\n")++ assert exc.migration is not None+ assert exc.migration.needs_attention is True+ assert [unexplained.path for plan in exc.migration.plans for unexplained in plan.unexplained] == [UNKNOWN_KEY]++ def test_the_domain_comes_from_the_class_and_the_block_rides_the_report(self, global_dir: Path) -> None:+ """What an agent consumer actually reads: one structured envelope, `config` either way."""+ exc = load_and_capture(config_dir=global_dir, body=f"{old_shape_document()}\n{UNKNOWN_KEY} = true\n")++ report = exc.to_error_report()++ assert report.error_domain == "config"+ assert report.migration is not None+ assert report.to_dict()["migration"]["remedy"] == MIGRATE_COMMAND++ def test_no_remedy_ever_tells_a_reader_to_start_the_file_over(self, global_dir: Path) -> None:+ """The whole point of the change: a migratable file is never answered with a reset."""+ exc = load_and_capture(config_dir=global_dir, body=f"{old_shape_document()}\n{UNKNOWN_KEY} = true\n")++ assert "init telemetry" not in exc.message+++class TestNoValueFromAUsersFileIsEverRendered:+ """The mechanical rule, on the telemetry surface's own error.++ The document carries a realistic PostHog key at the exact path the shipped entry moves, and it+ must appear in neither half of what the error hands on — not in the sentence a human reads,+ not in the structured block a machine parses — while the file demonstrably still holds it.+ """++ def test_the_planted_key_is_in_the_file_and_in_neither_half_of_the_error(self, global_dir: Path) -> None:+ stale = old_shape_document().replace('project_api_key = "phc_example_project_api_key"', f'project_api_key = "{PLANTED_KEY}"')+ assert PLANTED_KEY in stale, "the specimen must actually carry the secret"++ exc = load_and_capture(config_dir=global_dir, body=f"{stale}\n{UNKNOWN_KEY} = true\n")++ assert PLANTED_KEY in (global_dir / "telemetry.toml").read_text(encoding="utf-8")+ assert PLANTED_KEY not in exc.message+ assert exc.migration is not None+ assert PLANTED_KEY not in str(exc.migration.model_dump(mode="json"))diff --git a/wip/migrator-phase3-progress.md b/wip/migrator-phase3-progress.mdindex c53e79126..b52247f39 100644--- a/wip/migrator-phase3-progress.md+++ b/wip/migrator-phase3-progress.md@@ -2,16 +2,22 @@The charter is [`wip/migrator-3/sequencing.md` § S6](../../wip/migrator-3/sequencing.md) at the **workspace root** (from this repo: `../wip/migrator-3/sequencing.md`), with the phase content in `plan.md § Phase 3`. This file is the session-crossing record of what S6 has built, what it decided, and what is still open. Open the charter first, then this.-**Venue.** `_migrator/`, branch `feature/Migrator-3`, cut from `origin/dev` (the Phase 2 squash merge). **Pushed to origin, PR open against `dev`** carrying the pre-S7 bucket. S6 therefore lands as **two** PRs rather than the one the charter assumes: this one, and a second for the phase body on a fresh branch once `feature/Migrator-3` is squash-merged and deleted. Record both numbers against S6.+**Venue.** S6 lands as **two** PRs, not the one the charter assumes.-✅ **Cold-start state.** Two commits, both pushed, working tree clean:+- **Part 1 — `feature/Migrator-3`: MERGED.** PR [#1113](https://github.com/Pipelex/pipelex/pull/1113), `dev` = `490d189df`. Milestones 1–4 below. The branch is deleted local and remote.+- **Part 2 — `feature/Migrator-3b`: BUILT, REVIEWED, OPEN.** PR [#1114](https://github.com/Pipelex/pipelex/pull/1114) against `dev`, head `d74b899d4`, every required check green; two bot review rounds executed on the branch, six threads left open as the recorded deferrals in [`migrator-pr-1114-review-followups.md`](./migrator-pr-1114-review-followups.md). Milestones 5–10 below. **Merging it is Louis' call** — squash only, then delete the branch local and remote.+- **Part 3 — `feature/Migrator-3c`**, to be cut from `dev` once #1114 is squash-merged (`git fetch origin dev && git switch -c feature/Migrator-3c origin/dev && git branch --unset-upstream`; first push is `git push -u origin feature/Migrator-3c`). This is the branch a new session works on. Items 6–9 of [What is actually left](#what-is-actually-left): no code — the drift-contract review list, the two skills, the spec rows, publishing the contract.++Record all three numbers against S6.++✅ **Cold-start state (part 1, on `dev`).** Two commits, both pushed, working tree clean:1. *"Migrator Phase 3, milestone 1: settle the golden format before S7 freezes it"*2. *"Migrator Phase 3, milestones 2-3: what an unsafe entry promises, and catching a narrowing"* — milestones 2 and 3 together, plus the fixes from a `/code-review high` pass over them.`make agent-check` and the full `make agent-test` are both green on the branch as pushed.-⚠ **S6 is about a third done.** The charter's "must be settled before S7" bucket is closed; the phase's own body — the two `migrate` commands and everything downstream of them — is **not started**. See [What is actually left](#what-is-actually-left) at the bottom before planning a session.+⚠ **S6 is well past half.** The pre-S7 bucket is closed (part 1, on `dev`), and on `feature/Migrator-3b` the two `migrate` commands (milestone 5), the downgrade diagnosis (milestone 6), boot tolerance (milestone 7), the validation-error report (milestone 8), the telemetry-remedy retirement (milestone 9) and the `doctor` pending-migrations row with `--fix` (milestone 10) are all built. What remains is the rest of the phase body — the drift-contract review list, the skills, the specs rows, and publishing the contract — as **part 3**, on a fresh branch after #1114 merges. See [What is actually left](#what-is-actually-left) at the bottom before planning a session.## Build order chosen for this session@@ -20,9 +26,13 @@ The charter lists a great many deliverables without an order. The order below wa1. **The golden-format bucket + the narrowing relation** — done, see below.2. **R9 + the "what does `unsafe` promise" pass** — done, see Milestone 2 below.3. **The applier's dotted-key rename policy, the backup/replace semantics pass, the `UNWRITABLE` wording** — done, see Milestone 3 below. That closes the "must be settled before S7" bucket.-4. **The commands** — `pipelex migrate`, `pipelex-agent migrate`, the downgrade diagnosis, the rendering-rule test — not started. **This is where the next session starts.**-5. **Boot tolerance, `report_validation_error` on the real plan, the telemetry remedies retired, the `doctor` row** — not started.-6. **The skills (`add-migration`, `/release` step 3b), the `config-docs` drift-contract review list, `command-surface-map.md` rows, publishing the contract in the nav** — not started.+4. **The commands** — `pipelex migrate`, `pipelex-agent migrate`, the rendering-rule test — done, see Milestone 5 below. The **downgrade diagnosis** (`unexplained[]`) was deliberately carried out of this milestone and is where the next session starts.+5. **The downgrade diagnosis** — done, see Milestone 6 below.+6. **Boot tolerance** — done, see Milestone 7 below.+7. **`report_validation_error` on the real plan** — done, see Milestone 8 below.+8. **The telemetry remedies retired** — done, see Milestone 9 below.+9. **The `doctor` pending-migrations row and `--fix` delegating to the migrate command** — done, see Milestone 10 below.+10. **The skills (`add-migration`, `/release` step 3b), the `config-docs` drift-contract review list, `command-surface-map.md` rows, publishing the contract in the nav** — not started. **This is part 3, where the next session starts**, on `feature/Migrator-3c` once #1114 is on `dev`.## Milestone 1 — the golden-format bucket (DONE)@@ -202,6 +212,8 @@ Everything in [§ S6](../../wip/migrator-3/sequencing.md#s6--migrator-phase-3) n- ~~**The backup and replace semantics pass**~~ — done, see Milestone 3(b).- ~~**The `UNWRITABLE` wording and the reason enum**~~ — done, see Milestone 3(c).- ~~**R9**~~ — done, see Milestone 2. What it left open: **a `safe` entry silenced by its own `CONFLICT` plus a later rename** has the same mechanism and was deliberately not fixed, because repairing it means writing at a guessed spelling.+- ~~**The #1111 note-5 half** (the walk is non-recursive, pinned with the real specimen)~~ — done, see Milestone 5.+- ~~**The symlink write scope**~~ — decided in Milestone 5 and written into the contract: the write scope is the resolved target of any file the walk claims.**The "must be settled before S7" bucket is closed.** Two named follow-ups came out of it, neither blocking: the tomlkit raw-storage staleness wants one pass over every facade kind (`Container`, `Table`, `OutOfOrderTableProxy`) rather than the half-repair a rename can reach; and the `safe`-entry sibling above.@@ -217,6 +229,380 @@ A `/code-review high` over the staged changes produced five findings. Two were fThe contract's symlink paragraph was trimmed in the same change: it claimed "the two callers of the shared transaction primitive now agree", which on the write-scope half they do not.+## Milestone 5 — the two `migrate` commands (DONE, on `feature/Migrator-3b`)++The first milestone of part 2, and the one the reshape is gated on: until `pipelex migrate` exists, merging the reshape strands every existing `pipelex.toml` in the field.++### What was built++**`pipelex/migration/run.py` — the aiming.** `runner.py` answers *how* a file is migrated and takes everything as a parameter, which is what lets the gates replay it over documents nobody writes. The new module answers *what a real run is aimed at*: the package's own registry, the ledgers shipped beside it, and `config_directories_to_migrate()` — the global `~/.pipelex/` then the project `.pipelex/`, each only if it exists, and a project rooted at the home directory walked once rather than twice. Both commands and (later) `report_validation_error` must aim at the same files, or a user is told one thing by their boot and another by their tool.++**`pipelex migrate`** (`pipelex/cli/commands/migrate_cmd.py`) plans, shows the plan, and asks; `--yes` skips the question, `--dry-run` stops after the plan. Two passes deliberately: the rehearsal is what the user is shown and asked about, the second is the one that writes, and where they disagree the transaction refuses rather than writing over work it never saw. Registered in `_CORE_COMMAND_ORDER` and added to the app callback's skip set, so no readiness check or deck notice runs ahead of a command whose whole point is a broken machine.++**`pipelex-agent migrate`** (`pipelex/cli/agent_cli/commands/migrate_cmd.py`) is the same run for a machine. **It writes only with `--yes`** — it cannot ask, so the default is the safe half of the question — and `--dry-run` is the explicit spelling of that default, which is the loop the charter names. The two together are **refused, exit 2**, rather than resolved: an agent that asks for both has a bug a silent winner would hide. `needs_attention` is the verdict and is deliberately not "did anything get written"; the exit code and the rendering are presentation.++`describe_op` is shared between them, and renders an operation from ledger-supplied material only.++### The end-to-end tests use the real shipped migration++The one thing that made an honest E2E possible today: **`telemetry-config@2` is a real entry about a real file**. A flat first-generation `telemetry.toml` fails `TelemetryConfig`'s `extra="forbid"` and breaks the boot right now, so the charter's "failing boot → migrate → boot succeeds" runs against the package's own `goldens/telemetry-config/[email protected]` and the ledger the package ships — no fixture invented, no monkeypatching. `tests/e2e/pipelex/cli/test_migrate_commands.py` plants that file in a hermetic HOME **and** a flat `telemetry_override.toml` in a temp project (the tier the telemetry loader actually merges, which is what keeps the boot failure honest), then proves: the boot fails, both files are migrated, exactly one backup each holding what that file used to say, the report names both, and the boot succeeds. The agent loop (`--dry-run --format json` → `--yes`) is the same scenario read as JSON.++**The boot probe is `pipelex-agent models`** — the cheapest command that performs a full boot including the telemetry load. `pipelex show config` is *not* a probe: it exits 0 on a machine whose telemetry cannot load, because it never reads it. Worth knowing before inventing a lighter one.++**The bootstrap proof** plants a root key no model knows at the top of `pipelex.toml`, confirms a booting command exits 1 on it, and shows both `migrate` commands still run. A second test in the same class is the one that matters more: with that same broken `pipelex.toml` in place, a stale `telemetry.toml` beside it is **still migrated**. A command that ran but declined to do anything would pass the first test and be useless.++**The rendering rule**, on the two channels that exist: a realistic secret is planted as the old `project_api_key`, which the shipped entry *moves*, and asserted absent from stdout and stderr of four invocations (human dry-run, human apply, agent JSON, agent Markdown) while the migrated file demonstrably carries it. The third channel — the `migration` block on a configuration validation error — joins when `report_validation_error` lands, and the test's docstring says so.++### `min_supported_schema_version` now has a reader++The charter's "the command reads them or they go", settled for the floor by **giving it a reader** rather than softening the sentence. `declared_schema_version` (in `config_surface.py`, beside the strip that already lived there) reads the reserved `[meta] schema_version` **exactly as tolerantly as boot strips it** — a string, a float, a `bool`, a non-table `[meta]` are all "no declaration", because a malformed declaration boots fine and migration must not be stricter than the reader the key exists for. `migrate_file` refuses a file below its ledger's floor with a new `FileBlockedReason.UNSUPPORTED_SCHEMA_VERSION`, writing nothing. Dormant today (every floor is 0, and nothing writes the key) and pinned with a synthetic ledger; it earns its place the day a squash moves a floor.++Note what this uncovered: **a squash cannot actually be expressed today** — `check_entries_are_contiguous_and_named_for_their_version` requires entries to start at 2 and be contiguous — so the floor is a *declaration* the reader now enforces, not something derivable from the entries. That is worth knowing before anyone tries to squash a ledger.++`MigrationReport` gained `changed_plans`, `unexplained_plans` and `needs_attention`; `written_plans` and `blocked_plans` finally have readers. **`MigrationLedger.entry_for_version` still has none** — see below.++### Decisions taken in this milestone++- **The symlink write scope is settled and written into the contract**, closing §1 of `migrator-write-scope-and-rename-fidelity.md` in favour of keeping the current behaviour and saying so: *a migration's write scope is the resolved target of any file the walk claims*. The `.mthds` fix loop guards its scope because it is handed a bundle directory; a configuration directory is a place a user keeps links to files they own, and a dotfiles repository is the ordinary reason one is there. Refusing it would decline to migrate exactly the machines whose owner was most deliberate.+- **The walk is not recursive, and that is now pinned with the real specimen** (#1111 note 5): `.pipelex/inference/backends/pipelex_gateway.toml` matches the `pipelex-config` tier glob exactly. Both halves are tested — it is not claimed where it lives, *and* the same name at the top level is claimed — because without the second half the first would still pass if the glob stopped matching, and would be proving nothing about recursion.+- **The replay-time `TOMLKitError` catch got an honest message.** The parse now happens up front (for the floor read), so a `TOMLKitError` from the replay is an operation failing on a document that is valid TOML — an applier bug, not a bad file. It is still caught at the per-file boundary rather than aborting the walk, and the detail now says what it is.++### Files touched in milestone 5++```+pipelex/migration/run.py NEW — which directories, which registry, which ledgers+pipelex/migration/runner.py up-front parse; the floor refusal; the honest replay-error detail+pipelex/migration/plan.py UNSUPPORTED_SCHEMA_VERSION; changed_plans/unexplained_plans/needs_attention+pipelex/system/configuration/config_surface.py declared_schema_version+pipelex/cli/commands/migrate_cmd.py NEW — the human command + describe_op+pipelex/cli/agent_cli/commands/migrate_cmd.py NEW — the machine command+pipelex/cli/_cli.py, pipelex/cli/agent_cli/_agent_cli.py registration, ordering, the no-boot skip set+subject_grants.toml _render_markdown (positional-Callable protocol)+docs/migration-ledger.md NEW § "The commands"; the floor bullet; the blocked-reason row; the symlink decision+docs/tools/cli/migrate.md NEW user-facing page (+ index row, + mkdocs nav)+pipelex/cli/agent_cli/CLAUDE.md the migrate row and its markdown structure+CHANGELOG.md two Added bullets+tests/e2e/pipelex/cli/test_migrate_commands.py NEW — both commands, the bootstrap, the rendering rule+tests/unit/pipelex/migration/test_migration_run.py NEW — directories, walk depth, the floor, the reserved-key read+tests/unit/pipelex/migration/conftest.py build_ledger gained min_supported_schema_version+.drift/acks/{cli-docs,config-docs}.toml both contracts reviewed and acked+```++Mutation-tested: dropping the project directory from the walk, making `--yes` not write, removing the floor refusal, and making the walk recursive each turn exactly the intended tests red.++## Milestone 6 — the downgrade diagnosis (DONE, on `feature/Migrator-3b`)++`unexplained[]` had renderers on both commands and nothing ever put anything in it. It does now, and the charter's sentence about the downgrade direction — *"that diagnosis is computable from the ledger alone"* — turned out to be wrong and was corrected in the contract in the same change.++### Where it runs, and why there++**In the runner, on the document the run leaves behind.** `run.py` was the other candidate and cannot do it: on a dry run the migrated document exists only in memory, and re-reading the file afterwards would diagnose the *pre*-migration shape — reporting every stale key the ledger is about to repair. So `migrate_file` diagnoses `replay.text`, which is the only place that document is.++**It is the first thing in the migration package that needs a model, and the engine stayed model-free.** The question goes to the surface's *fingerprint*, not to the model directly, which is what keeps the new module (`diagnosis.py`) as pure as `material.py`: a fingerprint, a document, a ledger and the blocked entries in, a list of `UnexplainedPath` out. `compute_surface_fingerprint` moved off `snapshot.py` (the *regenerator*) and onto `Surface.fingerprint_at()`, where both readers can have it without the runner importing the golden-writing machinery. It costs about 2 ms per file and reads nothing but the package's own files, so the bootstrap property is untouched.++### The four rules that make the answer worth reading++- **A tree walk, not a diff of two flat path sets.** That is what buys the two properties a set diff cannot have: an unknown *table* is named once instead of once per key inside it, and the schema spelling of every ancestor is known by the time a child is reported.+- **A blocked entry answers for its own material.** An `unsafe` entry is never applied, so the old shape it is about is still in the file — already reported, by name, with the entry's guidance. Calling the same key an unexplained typo would contradict the report two lines above it. The subtraction **deliberately over-covers** (op sources plus `declared_removed_paths`, each also traced forward through later `safe` entries), because every path it removes is one the same report names in `blocked[]`. A `remap_value` contributes nothing: it moves a value and leaves the path where it was, so subtracting it would silence a real typo sitting at a remapped key. This is `MigrationLedger.entry_for_version`'s **first caller** — the #1113 review's open "remove it or keep it" question is answered by keeping it.+- **A key the user chose is never rendered.** Beneath an open mapping the schema says `levels.*` where the file says `levels.my_package`, and a typo *inside* such an entry is reported at `queues.*.retreis`. Same rule as `narrowed_paths[]`, and the walk gets it for free by carrying the schema spelling alongside the document path.+- **A document nesting below a path the schema says is a scalar is left alone.** That is a type error, which the model reports far better, and descending would invent unknown paths beneath a path the schema knows perfectly well. Together with the reserved-`[meta]` strip — performed exactly as boot performs it — that is what keeps the diagnosis silent on files it has nothing to say about.++### What the note says++One note for every finding, and deliberately so: *the current schema has no setting there, and no ledger entry retires it — either the name is a typo, or this file was written by a newer pipelex than the one running, so check whether you are on an older branch or an older build.* The two readings are indistinguishable to a schema that knows neither name, and guessing would send half the users to the wrong fix.++**A reserved-path cross-reference was considered and rejected.** `derive_reserved_registry` could say "this path was retired at version N and is still here" — but a reserved path surviving a replay means a *blocked* entry, which the subtraction above already accounts for, so the branch would never fire.++### The witnesses++- **Every real surface's reference document and kit template diagnose clean** (`test_real_surfaces.py`). Both are at the current schema by construction, so a finding in either is the diagnosis being wrong about our own schema rather than a stale document. It was green on the first run, which is what made the design trustworthy before any of the unit tests existed.+- **The e2e bootstrap fixture was already the specimen.** `_break_the_configuration` plants `not_a_real_setting = true` at the top of `pipelex.toml` — a root key no model knows and no entry removes — so the diagnosis is now asserted against the very file that breaks the boot, through the real binary. Both bootstrap tests changed exit expectation from 0 to **1**: `needs_attention` is now true on that machine, which is correct, and the second one is the better test for it — a broken `pipelex.toml` needing a human does not stop the stale `telemetry.toml` beside it from being migrated.++### Files touched in milestone 6++```+pipelex/migration/diagnosis.py NEW — the walk, the blocked-entry subtraction, the note+pipelex/migration/documents.py path_matches_pattern / path_is_at_or_under_pattern extracted; one definition of `*`+pipelex/migration/runner.py _diagnose, on the migrated document+pipelex/migration/surfaces.py Surface.fingerprint_at()+pipelex/migration/snapshot.py compute_surface_fingerprint removed, calls the method+docs/migration-ledger.md "The downgrade direction" rewritten; the unexplained rule in "What the engine reports"+CHANGELOG.md one Added bullet+tests/unit/pipelex/migration/test_diagnosis.py NEW+tests/unit/pipelex/migration/test_real_surfaces.py the clean-witness over our own documents+tests/unit/pipelex/migration/test_migration_run.py the migrated-document-not-the-read-one pair+tests/unit/pipelex/migration/conftest.py ExampleConfig grew the paths its fixtures' migrated documents carry+tests/e2e/pipelex/cli/test_migrate_commands.py the diagnosis through the real binary; two bootstrap exit codes+```++`ExampleConfig` growing a `reporting` block is worth noticing rather than skipping: the synthetic surface's model now has to name the paths the synthetic *migrated* documents carry, or the diagnosis reports them and every runner test gains noise about something it was not testing.++Mutation-tested: dropping the reserved-meta strip, the open-node `*` fallback, the blocked-entry subtraction, the stop-at-an-unknown-table, the scalar-descend guard, the schema spelling, the remap exclusion, the forward trace, the `declared_removed_paths` source, and diagnosing the document as it was read each turn exactly the intended tests red.++## Milestone 7 — boot tolerance (DONE, on `feature/Migrator-3b`)++A stale configuration no longer stops the boot. Each surface's loader validates as it always did and, only when that fails, replays its ledger over the same files **in memory**, re-runs its own post-merge step, and validates again — booting with a warning that names the files and the `pipelex migrate` remedy. Nothing is written; a tolerated boot leaves the directory exactly as it found it, which is why the warning keeps coming back until the command is run.++**The shared helper owns the failure path, not the load** — `replay_surface_files_in_memory` in `config_surface.py`, which was already the home of the reserved-`[meta]` strip. The three loaders differ between their merge and their validate (programmatic overrides / `${VAR}` substitution / nothing), so a helper owning the whole load would have to know about all three. It takes the surface id and the same ordered path list the loader merged, and hands back the migrated merge plus a `MigrationPlan` per file. The overrides in particular are re-applied by the caller, because they are a layer of the *load* while the replay only sees the *files*.++### Three things the groundwork had wrong or had not seen++**The engine cannot be imported at module level, and the mechanical guard would not have caught it.** `pipelex/migration/engine.py` imports `pipelex.pipeline.fixes.applier` — `pipeline` is an *interpreter* package, and the configuration loaders sit in `runtime_hub`'s closure, the kernel layer whose stated property is that importing it loads zero interpreter modules. The previous session's groundwork checked the reverse direction only (that the engine pulls in no configuration model, which is true) and concluded a module-level import was fine. It is not. And `make agent-check` would not have caught it: `check-hub-layering` does reachability to `interpreter_hub`, which `pipeline.fixes.applier` does not reach, so the lint stays green. Measured rather than assumed — a module-level import there was tried, and it turns **nine** entry points of `tests/unit/pipelex/test_kernel_layer_import_closure.py` red while every gate in `agent-check` passes. Only the full `make agent-test` sees it. So the engine import is deferred into the retry — which also makes "the healthy path is untouched" literal rather than approximate. `migration.plan` and `migration.ledger` are clean and are imported normally.++**`PipelexConfig` does not raise `ValidationError`.** `ConfigRoot` gives itself a custom `__init__` that translates pydantic's error into `ConfigValidationError`, and pydantic v2 routes `model_validate` through a custom `__init__` — so the main configuration raises the translated one while a plain-model surface raises pydantic's. An `except ValidationError` would have switched boot tolerance off for the one surface everything depends on, silently and with every test passing. Both are caught, via `CONFIG_REFUSED` in `config_loader.py`.++⚠ **The same fact makes an existing branch dead, and it belongs to item 3.** `runtime_boot.py`'s `except ValidationError as validation_error:` around `setup_config` — the one that calls `report_validation_error(category="config", ...)` and raises `PipelexConfigError` — can never fire, because what `setup_config` raises is `ConfigValidationError`. Pre-existing, unchanged by this milestone, and precisely item 3's business, since that dead branch is one of `report_validation_error`'s two config-side callers. Verified by hand, not inferred.++**A failure *inside* the retry must never become the failure the user sees.** A missing ledger came back as `MigrationLedgerError` and replaced a legible "extra forbidden field: `not_a_real_setting`" with an internal path nobody can act on. The retry now declines on `MigrationLedgerError`; a broken packaged ledger stays loud where it should be loud (`make cl`, `pipelex migrate`). Same reasoning already applied to an unparseable file — and there the retry abandons *entirely* rather than skipping the file, because skipping drops a layer from the merge and a re-validation that then succeeded would boot on a configuration the user does not have.++### Decisions recorded in the contract++**Boot tolerance does not run the downgrade diagnosis.** On this path the model has already spoken, and pydantic's extra-field list is the same answer and a better one — it knows about validators, which a path walk does not. The diagnosis is for `pipelex migrate`, where nothing validates the file.++**The re-validation is what decides, and that is stronger than a second gate.** The contract's "unsafe entries, conflicts and unexplained paths still fail the boot" reads as a *description of why the re-validation fails*, not as an extra refusal on top of it: material an `unsafe` entry is about is still in the file, so the model refuses it; and a `VALUE_DOMAIN_NARROWED` report the model accepts was never a reason to refuse a boot, since that report says *check this key* and the model has now checked it.++### The two e2e tests changed premise, and a fourth finding came out of it++`test_migrate_commands.py` opened both of its end-to-end tests by asserting **the machine does not boot** — that was the scenario. Boot tolerance makes it boot, so the full `make agent-test` caught them (nothing in `agent-check` did). They now assert the machine boots on *both* sides of every command, which is the tolerance property itself, and the migration is still measured on the files.++⚠ **The agent CLI cannot see the warning, by contract.** The boot probe is `pipelex-agent models`, and `pipelex-agent` calls `silence_logging_for_agent_cli()` as its first act so nothing pollutes its two structured streams. So the e2e asserts the boot, not the warning — the warning is asserted per surface in `test_boot_tolerance.py`. The consequence is worth carrying into item 5: **a machine consumer never learns of a pending migration from a boot.** It has to ask — `pipelex-agent migrate --dry-run`, or the `doctor` pending-migrations row — which makes that row more load-bearing than it looked when it was listed.++### Where it landed++```+pipelex/system/configuration/config_surface.py the helper, the warning builder, the three surface-id constants+pipelex/system/configuration/config_loader.py config_file_paths extracted; load_config_validated; CONFIG_REFUSED+pipelex/system/telemetry/telemetry_config.py the retry, re-running the ${VAR} substitution+pipelex/system/pipelex_service/pipelex_service_config.py the retry, one file+pipelex/runtime_hub.py setup_config goes through load_config_validated+pipelex/cli/commands/plugins_cmd.py same, for free+pipelex/migration/ledger.py packaged_migration_dir moved here off the registry+pipelex/migration/surfaces.py imports the surface-id constants instead of spelling them+docs/migration-ledger.md "Boot tolerance" rewritten against what was built+CHANGELOG.md one Added bullet+tests/unit/pipelex/system/configuration/test_boot_tolerance.py NEW+tests/e2e/pipelex/cli/test_migrate_commands.py both e2e tests re-premised on a machine that boots+docs/tools/cli/migrate.md the boot warning is how users will meet the command+docs/contribute/hub-layering.md pipelex.migration accounted for (see the drift ack)+```++`packaged_migration_dir` moved off `surfaces.py` (the registry, which imports every configuration model) and onto `ledger.py`, so a loader can reach a ledger without reaching a registry. The surface ids are now constants in `config_surface.py` that the registry imports, so the loader and the registry cannot drift apart on a string literal.++The stale documents in the tests are **real** where they can be: `telemetry-config@2` and its shipped `[email protected]`. The other two surfaces have empty ledgers, so their tests plant a synthetic one in a temporary migration directory via `packaged_migration_dir` — the wiring is what is under test there.++Mutation-tested: dropping the reserved-meta strip, skipping an unparseable file instead of abandoning the retry, abandoning on a missing file instead of skipping it, reporting a clean surface as stale, letting a ledger failure escape, merging the text as read instead of as replayed, dropping the programmatic overrides from the retry, catching only pydantic's `ValidationError`, re-raising from a failed retry instead of yielding, and building the warning from the file instead of from the ledger — each turned exactly the intended tests red.++⚠ **A process note worth keeping.** During that mutation loop a `git checkout <file>` reverted an *unstaged* edit and silently undid part of the milestone; the next three test runs were measuring a tree that no longer had the code. Mutation loops over uncommitted work must back up with `cp` and restore with `cp` — never with git.++## Milestone 8 — `report_validation_error` on the real plan (DONE, on `feature/Migrator-3b`)++The old `MigrationConfig` / `migration_maps` consumer is gone and a **dry-run scan of the surface that refused** has taken its place, riding the error twice: as a paragraph in the message and as the structured `migration` block a machine consumer branches on. `error_domain` stays `"config"`.++### The signature changed, and `category` did not survive it++`report_validation_error(*, category, validation_error) -> str` became `report_validation_error(*, validation_error, surface_id=None) -> ValidationErrorReport`. `category` selected a renaming map and had no other reader, so with the maps retired it named nothing; `surface_id` expresses the one asymmetry left — **naming a surface is what turns the scan on**. The two `.mthds` callers (`validate_bundle.py`, `library_manager.py`) and `runtime_boot`'s backend/deck helper name none, deliberately: none of those has a ledger, and a `pipelex migrate` remedy for a `.mthds` bundle sends a user to a command with nothing to do. That is pinned twice — no block, *and* the walk never happens.++`MigrationConfig` keeps `migration_maps` and loses both methods, with the reason in its docstring: the field is dead but the `[migration]` table ships and `extra="forbid"` would refuse every user file still carrying one, so the reshape entry is what removes both halves — sequencing rule 3. Two now-dead subject grants came out of `subject_grants.toml` with the methods.++### The block had to live in `base_exceptions.py`, and that cost a module move++`ErrorReport` references the block as a typed field, and `base_exceptions` cannot import the migration package: `plan.py` → `ledger.py` → `migration.exceptions` → `base_exceptions`. The precedent is `ValidationErrorItem`, which lives there for exactly this reason.++Rather than declare a second projection of `MigrationPlan` that would drift from the first, the cycle was **broken**: `MigrationSafety` moved out of `ledger.py` into a new `pipelex/migration/safety.py`, which leaves `plan.py` depending only on stdlib, pydantic, `safety` and `suggested_fix` — all already in `base_exceptions`' closure. `plan.py`'s docstring now states that constraint the way `suggested_fix.py` states its own. So `MigrationErrorBlock.plans` is `list[MigrationPlan]`: **the same shape `pipelex-agent migrate --dry-run --format json` emits under its own `plans` key**, and an agent that parses one has already parsed the other.++`migration` is deliberately **not** in `_STRICT_KEPT_FIELDS`, and the contrast with `validation_errors` is written beside it: validation errors describe the caller's own submitted bundle, a pending migration describes the *host's* configuration directories.++### Three defects found, all real, all fixed++**(a) Scoping the scan by building a one-surface registry silently corrupted the answer.** `SurfaceRegistry.surface_for_file_name` resolves ownership by *"an exact base file claims before any glob, **across all surfaces**"* — and its docstring names `pipelex_service.toml` as the very case it exists for, since that file is one surface's base file and a match for `pipelex-config`'s `pipelex_*.toml`. Handing `migrate_directories` a registry holding only `pipelex-config` removed the other claimant from that arbitration, so the glob won: the file was replayed under the wrong ledger and diagnosed against the wrong model, and its perfectly ordinary `agreement` and `onboarding` settings came back reported as *paths this build knows nothing about*. **Measured in a real run's stderr, not reasoned about.** The fix is `migrate_directories(only_surface_id=...)`: arbitration first with the full registry, filter second. Mutation-tested — reverting it turns the regression test red on exactly `pipelex_service.toml`.++**(b) The boot's `except ValidationError` around `setup_config` was dead**, as Milestone 7 predicted, and so was the doctor's. `PipelexConfig` is a `ConfigRoot`, whose custom `__init__` translates pydantic's error into `ConfigValidationError`, so neither arm ever fired for the one configuration everything depends on — a bad `pipelex.toml` produced a bare traceback rather than the field-level translation the arm existed to produce. Both now catch `CONFIG_REFUSED` and go through one shared `raise_config_setup_error`, with `pydantic_error_behind` (new, beside `CONFIG_REFUSED` in `config_loader.py`) reaching through either shape. A refusal carrying no pydantic error is re-raised as itself.++**(c) The agent CLI printed two error envelopes on one stream.** `agent_error` leaves through `typer.Exit`, which is a `RuntimeError` and **not** a `SystemExit`, so `models_cmd`'s `except SystemExit: raise` never caught it and the broad `except Exception` below reported it a second time. Two JSON documents on stderr is not JSON — `json.loads` gives *"Extra data"*. Pre-existing, and found because the new e2e is the first test to parse that stream instead of substring-matching it. Fixed at both sites (`models_cmd`, `check_model_cmd`) and pinned by an explicit single-envelope assertion.++**(d) And one trap this milestone introduced and closed in the same pass.** `MigrationPlan` carries real `Path` values, so `ErrorReport.to_dict()` — the documented serialization surface, handed straight to `json.dumps` by the webhook delivery path — could no longer be serialized once a block rode on it. Not reachable from a pipeline run today, since nothing raises a `PipelexConfigError` inside one, but leaving it would be a trap for whoever wires the next consumer. `to_dict` now dumps that one field in JSON mode; dumping the whole report in JSON mode would have re-serialized every other field and every round-trip with it. `from_dict` still round-trips, because pydantic accepts a string for a `Path`.++### Decisions taken in this milestone++- **The scan runs only on the failure path and only when a surface is named**, so it is a filesystem walk and a ledger replay that a healthy boot never pays for — and the migration import is deferred inside the function, for the kernel-layer reason Milestone 7 established (`core/validation.py` is in `runtime_boot`'s closure; `make agent-check` would not catch a module-level import, only the full `make agent-test`).+- **A failure inside the scan never becomes the failure the user sees**, same rule as the boot retry — but the catch is `(MigrationError, OSError)` rather than blanket, so an applier bug still surfaces as the bug it is. Both halves are tested.+- **The block is present iff the scan found something**, and only non-clean plans are listed. Absence is the verdict "this is not staleness", which is what "consumers branch on the block's presence" requires to mean anything.+- **The message is presentation** and carries the pydantic analysis plus a paragraph in the same order as `stale_configuration_warning`, because the two are read in the same places.++### Files touched in milestone 8++```+pipelex/migration/safety.py NEW — MigrationSafety, moved out of ledger.py to break the cycle+pipelex/migration/plan.py the low-level constraint stated; imports safety+pipelex/migration/ledger.py + 5 siblings + 7 test modules re-pointed at migration.safety+pipelex/migration/runner.py migrate_directories(only_surface_id=...)+pipelex/migration/run.py scan_config_surface — the dry run, aimed at one surface+pipelex/base_exceptions.py MigrationErrorBlock; ErrorReport.migration; PipelexConfigError carries it+pipelex/core/validation.py REWRITTEN — the scan, the block, the prose, raise_config_setup_error+pipelex/system/configuration/config_loader.py pydantic_error_behind, beside CONFIG_REFUSED+pipelex/system/configuration/configs.py MigrationConfig: methods retired, field and reason kept+pipelex/runtime_boot.py, pipelex/cli/commands/doctor_cmd.py the dead arms fixed, one shared helper+pipelex/cli/agent_cli/commands/agent_output.py the migration block on the error payload+pipelex/cli/agent_cli/commands/{models,check_model}_cmd.py the double-envelope fix+pipelex/pipeline/validate_bundle.py, pipelex/libraries/library_manager.py .message, no surface+subject_grants.toml two dead grants removed+docs/migration-ledger.md NEW § "Reporting a stale configuration on a validation error"+docs/tools/cli/agent-cli.md the error envelope: one document, and the `migration` field+docs/tools/cli/migrate.md the boot-failure path names the command; the agent loop+docs/configuration/index.md the `[migration]` line, which the previous config-docs ack parked for here+pipelex/cli/agent_cli/CLAUDE.md the migration-field bullet+CHANGELOG.md one Added bullet, two Fixed+tests/unit/pipelex/core/test_validation_report.py RE-AIMED — was about hub state, is now about the scan+tests/e2e/pipelex/cli/test_migrate_commands.py NEW class: the third channel, through the real binary+.drift/acks/{config-docs,cli-docs}.toml both reviewed and acked+```++Mutation-tested: building the scan's registry from one surface, dropping the JSON-mode dump of the `migration` field, widening the scan's catch to a blanket `except Exception`, and returning a block when the scan found nothing — each turned exactly the intended tests red.++The re-aiming of `test_validation_report.py` is worth noticing rather than skipping: its two tests existed because the helper *read the hub* and crashed when there was not one. It no longer touches the hub at all, so that concern moved — the bootstrap property is now that the **scan** runs on a machine with no hub and no configuration, which is where it is asserted.++⚠ **The e2e's boot assertion changed, and it is a contract change.** `TestTheBootstrapPath` asserted `"ConfigValidationError" in boot.stderr`; a boot failure now reports `PipelexConfigError`, which is the class that carries `error_domain: "config"` and the block. Anything downstream matching on that error type sees a different name.++## Milestone 9 — the two telemetry remedies retired (DONE, on `feature/Migrator-3b`)++Both of them told a user to re-initialize a file the shipped `telemetry-config@2` entry exists to carry forward. `pipelex init telemetry` **writes a fresh file**, so on the one case they named — the flat pre-`[custom_posthog]` format — the remedy discarded the PostHog key, the Langfuse credentials and the OTLP exporters it was supposed to be helping with. There turned out to be a third one, and a fourth site that was worse than any of them.++### There were four, not two++The tracker named `check_telemetry_config`'s old-shape sniff and `handle_telemetry_config_validation_error`'s banner. Also found and retired:++- **`AGENT_ERROR_HINTS["TelemetryConfigValidationError"]`** — the agent-side twin of the banner, and leaving it would have had the human CLI say `pipelex migrate` while the agent CLI said `pipelex init telemetry` about the same file.+- **`doctor --fix`'s prompt**, which is the one that could actually destroy something. `can_fix_telemetry` was `"format has changed" in telemetry_message.lower()`, and on a match `--fix` offered *"Reset telemetry configuration using the new format?"* → `init_cmd(focus=InitFocus.TELEMETRY)`. Answering yes on a migratable file rewrote it from the template.++That last one is also why the row could not simply be reworded: the message **was** the contract between the probe and the repair machinery, in three places (`display_health_report`, `do_doctor_cmd` twice). Rewording it would have switched the whole `--fix` telemetry path off in silence, with every test still green.++### What was built++**The error carries the answer.** `load_telemetry_config` now raises through `report_validation_error(surface_id=TELEMETRY_CONFIG_SURFACE_ID)`, so a `TelemetryConfigValidationError` carries the same `MigrationErrorBlock` a `PipelexConfigError` does. The message is byte-identical to before plus the migration paragraph — `format_pydantic_validation_error` is literally `analyze_pydantic_validation_error(exc).error_msg`, which is what `report_validation_error` builds on.++**`TelemetryConfigError` is now a `PipelexConfigError`.** It is one — and the reparenting is what gives it both halves for free: `error_domain = CONFIG` from the class (so its `AGENT_ERROR_DOMAINS` entry went, which `test_agent_output_drift.py` — the lookup-dict guard, not a drift *contract* — requires the moment a class declares its own) and the ability to carry a block. Checked every `except PipelexConfigError` site before doing it: `reporting_manager`, `tracing_assembly` and both doctors, none of which load telemetry — `setup_doctor_runtime` only does `setup_config`.++**The doctor row is a verdict, not a sentence.** `check_telemetry_config` returns a `TelemetryConfigCheck(finding, message)` over a new `TelemetryConfigFinding` — `HEALTHY`, `NOT_FOUND`, `UNPARSEABLE`, `OUT_OF_DATE`, `INVALID` — and every caller branches on it. `finding.is_repaired_by_initializing` is the property that answers the destructive question, and it is True for exactly one member: there is nothing in a file that is not there to lose. `OUT_OF_DATE` gets `pipelex migrate`, `UNPARSEABLE`/`INVALID` get a person, and regeneration is still offered on those two but described as what it is — *"start the file over, discarding what is in it"*. The agent doctor emits `finding` on the envelope and derives one recommended action per finding through a match/case.++### Two defects fixed along the way++**(a) The probe was stricter than the loader it reports on.** `check_telemetry_config` validated the raw document without `strip_reserved_meta`, so a `telemetry.toml` carrying the migration machinery's reserved `[meta]` table booted perfectly well and was reported invalid. Dormant today (nothing writes the key) and reachable the moment anything does.++**(b) The scan had to be scoped to the directory the probe read.** `scan_config_surface` gained `config_dirs`, because the doctor validates *one* resolved file — `--global` inspects `~/.pipelex/` — while the default walk is both directories. Without it, a stale global file would make a `--global`-inspected project file report `out_of_date`, sending the reader to a migration that does nothing for the file on their screen. Mutation-tested: reverting it turns two tests red.++### One I nearly shipped++The doctor's scan had no guard around it. Milestone 8's rule — *a failure inside the scan is never the failure the user sees* — costs more in the doctor than anywhere else: an exception escaping the probe reaches `doctor_cmd`'s own outer handler, which prints one line and exits, so a broken packaged ledger would have replaced **every row the user came for** with "Unexpected error". Caught in self-review, not by a test. The catch is `(MigrationError, OSError)` like the others, and both halves are now pinned — the fallback, and an applier bug still surfacing as itself.++### The module split this forced, and why it was the right answer++`make agent-check`'s pyright caught a **real import cycle**: `telemetry_config → core.validation → migration.run → migration.runner → migration.surfaces → telemetry_config`. `surfaces.py` builds the registry and therefore imports every configuration model, this surface's included.++**Deferring the import does not help** — `core.validation`'s reach into `migration.run` is already deferred and pyright counts it as an edge all the same. Tried it; the cycle report was unchanged. Worth knowing before anyone tries the same thing on `pipelex_service_config.py`.++So the module was split the way `system/configuration/` already splits: **`telemetry_config.py` keeps the models, the new `telemetry_loader.py` holds `load_telemetry_config`**. That makes the edges run one way — the registry reads the models, the loader reads the registry — and it is cheap, because everything in the registry's closure (`otel_factory`, `surfaces` itself) imports only models. `telemetry_factory.py` was the single production importer of the loader.++This is the same shape Milestone 7 hit and solved by moving `packaged_migration_dir` off `surfaces.py` onto `ledger.py`, one level up: *a loader must be able to reach the migration package without reaching the registry.*++### Deliberately not done++**`pipelex_service_config.py` is the third surface and was left alone.** It has boot tolerance and no block — but it also has **no hardcoded remedy to retire**, anywhere: nothing in the CLI catches `PipelexServiceConfigValidationError` by name, and it carries agreement/onboarding state rather than settings a user tuned. Closing the gap means the same models/loader split (`surfaces.py` imports `PipelexServiceConfig` from it), which is a change item 4 does not justify. Named here so it reads as a decision rather than an oversight.++**`--fix` does not yet run the migration.** `OUT_OF_DATE` is reported and `pipelex migrate` is named, but fix mode offers nothing for it — deliberately, because "`--fix` delegating to the migrate command" is item 5, whose pending-migrations row covers every surface and would supersede a telemetry-only version built here. What item 4 owed was that fix mode stop offering the *destructive* thing, and it does.++### Files touched in milestone 9++```+pipelex/system/telemetry/exceptions.py TelemetryConfigError is a PipelexConfigError+pipelex/system/telemetry/telemetry_loader.py NEW — the loader, out of the registry's closure+pipelex/system/telemetry/telemetry_config.py models only now+pipelex/system/telemetry/telemetry_factory.py re-pointed at the loader+pipelex/migration/run.py scan_config_surface(config_dirs=...)+pipelex/cli/commands/doctor_cmd.py TelemetryConfigFinding / TelemetryConfigCheck; every text sniff gone+pipelex/cli/agent_cli/commands/doctor_cmd.py _telemetry_action; `finding` on the envelope+pipelex/cli/error_handlers.py the banner rebuilt from the error+pipelex/cli/agent_cli/commands/agent_output.py the hint rewritten; the domain entry removed+docs/migration-ledger.md NEW § "Every surface reports it the same way, and none of them says start over"+docs/configuration/config-practical/telemetry-config.md NEW § "When telemetry.toml will not load"+docs/tools/cli/{migrate,init,agent-cli}.md, docs/under-the-hood/init-cli-flows.md+pipelex/cli/agent_cli/CLAUDE.md the doctor row names the finding+CHANGELOG.md one Changed, two Fixed+tests/unit/pipelex/system/telemetry/test_telemetry_stale_remedy.py NEW+tests/unit/pipelex/cli/{test_doctor_config_checks,test_doctor_cmd,test_doctor_display_report,test_doctor_fix_mode,test_agent_doctor_cmd,test_error_handlers_more}.py+tests/e2e/pipelex/cli/test_migrate_commands.py NEW class: the doctor row and the agent envelope, both through the real binary+.drift/acks/{config-docs,cli-docs}.toml both reviewed and acked+```++Mutation-tested: dropping the block from the raise, un-parenting the error class, dropping the migration prose from the message, making a fresh file "repair" an out-of-date one, dropping the reserved-`[meta]` strip, letting the scan inherit the whole walk, giving every finding one remedy again, removing the doctor's scan guard, and widening that guard to a blanket `except` — each turned exactly the intended tests red.++Also corrected in passing: `TestNoValueFromAUsersFileIsEverRendered`'s docstring still said the third rendering channel would arrive "when `report_validation_error` starts reporting the real plan". Milestone 8 landed it, as its own class, for the reason that class's docstring gives.++## Milestone 10 — the `doctor` pending-migrations row and `--fix` (DONE, on `feature/Migrator-3b`)++The last item of the phase body's first half, and the one Milestone 7 made load-bearing: **a machine consumer never learns of a pending migration from a boot.** A configuration the ledger can explain boots with a warning, and `pipelex-agent` cuts logging off process-wide as its first act, so no warning reaches it. Asking was the only channel left, and until now there was nothing to ask.++### What was built++**`check_pending_migrations()` in `doctor_cmd.py`** — a `PendingMigrationsCheck(finding, message, migratable_files, attention_files)` over a new `PendingMigrationsFinding`: `UP_TO_DATE`, `PENDING`, `NEEDS_ATTENTION`, `UNAVAILABLE`. It is **`pipelex migrate`'s own dry run**, not an approximation of it — `migrate_config_directories(config_dirs=config_directories_to_migrate(), dry_run=True)`, the same call the command makes.++Both renderers carry it: a **Configuration Migrations** section in the human report (placed beside Configuration Files, since it is about the same files), and `checks.pending_migrations` on the agent envelope with the finding and the two file lists. `--fix` offers to run the migration.++### The four decisions, each recorded in the contract++- **The row takes no directory.** Every other check reports on a *file* and is scoped to the directory the doctor was pointed at, `--global` included. This one reports on a *command*, and `pipelex migrate` has no `--global`. A row scoped narrower would name a command that then rewrites a file the row never mentioned — the one surprise a tool that writes to a user's files must not spring. Over-reporting is legible instead: every file is named with its full path. Pinned twice, at unit level (`assert_called_once_with()`, no arguments) and against `--global` on the agent doctor.+- **`UNAVAILABLE` is its own member, and it is not healthy.** A packaging problem of ours must not be reported as a finding about the user's files, and *not knowing* is not the same as *being up to date*.+- **`PENDING` and `NEEDS_ATTENTION` are separate because only one has a command behind it.** Offering to migrate a file the run would not change is a prompt whose honest outcome is "nothing was written" — the same shape as the reset `--fix` used to offer for an out-of-date `telemetry.toml`. So `is_repaired_by_migrating` is true for `PENDING` alone, and that is what gates fix mode.+- **A run is usually both at once**, so the row reports **both file lists** and both channels name both moves. The first draft keyed the second move off the *finding* and therefore said nothing about the attention files whenever anything was also pending; found by running the real binary against a planted machine, not by a test.++### `--fix` reaches the command's write pass, and deliberately not the command++`migrate_cmd.py` grew `apply_pending_migrations(config_dirs=...)` — the second of the command's two passes, with no exit code attached. Both callers use it. Calling `migrate_cmd(yes=True)` from the doctor was the obvious move and is wrong: it exits the process when the run leaves something for a person, which would cut off the doctor's remaining fixes, its remaining rows and its own exit code. The doctor's row *is* the dry run, so the two-pass shape is preserved — dry run, ask, write — with the doctor asking the question in the middle.++A `[dim]Re-run pipelex doctor[/dim]` line follows a successful migration, because every row below it was measured before the migration ran.++### One redundancy retired in the same pass++With a machine-wide row in place, the telemetry row's own `pipelex migrate` advice became the same instruction twice at two scopes. It is suppressed when the machine-wide row is reporting a pending run — **and only then**, which is not decoration: a telemetry file whose only pending work is *blocked* leaves the surface-scoped probe saying `out_of_date` while the machine-wide row has nothing to write, and there the telemetry bullet is the only thing that speaks. Both halves are tested.++### Files touched in milestone 10++```+pipelex/cli/commands/migrate_cmd.py apply_pending_migrations — the write pass, without the exit+pipelex/cli/commands/doctor_cmd.py PendingMigrationsFinding / PendingMigrationsCheck; the row; the fix+pipelex/cli/agent_cli/commands/doctor_cmd.py checks.pending_migrations; _pending_migrations_actions; the markdown section+docs/migration-ledger.md NEW § "The health report"+docs/tools/cli/migrate.md the doctor row as the third way here+docs/tools/cli/agent-cli.md the row is the only channel a machine has+pipelex/cli/agent_cli/CLAUDE.md the doctor row names checks.pending_migrations+CHANGELOG.md one Added bullet+tests/unit/pipelex/cli/test_doctor_pending_migrations.py NEW+tests/unit/pipelex/cli/{test_doctor_cmd,test_doctor_display_report,test_doctor_fix_mode,test_agent_doctor_cmd,test_format_doctor_markdown}.py+tests/e2e/pipelex/cli/test_migrate_commands.py NEW class: the row and `--fix`, through the real binaries+.drift/acks/cli-docs.toml reviewed and acked+```++⚠ **Every doctor test that runs a whole doctor now has to stub this check**, and the reason is worth carrying: unlike every other check it takes no directory, so it walks the *developer's* `~/.pipelex/` and the repo's own `.pipelex/`. Four existing test modules gained the stub. A test that forgets it does not fail — it silently depends on whose laptop the suite is running on.++⚠ **And the agent envelope's shape has a fixture nobody thinks of.** `test_format_doctor_markdown.py` builds the whole result dict by hand, so a new `checks` key makes its renderer raise `KeyError` — nine tests, in a module no part of `agent-check` runs and that a targeted per-area run does not obviously reach. Only the full `make agent-test` caught it. Adding a key to that envelope means editing two `ClassVar` dicts there.++Mutation-tested: scoping the row to one directory, reporting a scan failure as up to date, widening that catch to a blanket `except`, making the row write, dropping the migration from what `--fix` can do, offering it for a file it would not repair, dropping the attention files when something is also pending, repeating the telemetry remedy under the machine-wide row, leaving the row out of either verdict, and making the shared write pass a dry run — each turned exactly the intended tests red.++## Where this session paused++**Paused with PR [#1114](https://github.com/Pipelex/pipelex/pull/1114) open, reviewed and green, awaiting Louis' merge.** Milestones 5–10 were pushed and the PR opened after milestone 10; two bot review rounds (Codex, Cubic) followed and were executed on the branch as *"answer the PR #1114 bot review"* (`3fd7f82cb`) and *"answer the second PR #1114 bot review"* (`d74b899d4`, the head). What each round fixed is in those two commit messages and at the top of [`migrator-pr-1114-review-followups.md`](./migrator-pr-1114-review-followups.md); every thread was answered on the PR (fixed / false positive with the reason / deferred), and the six left open are exactly the deferrals §1–§5 of that note. Every required check is green on the head; `origin/dev` has not moved since the branch base `490d189df`, so the squash is clean. The merge button reads *blocked* only because `dev`'s ruleset requires thread resolution — the same state #1112 merged from, through the team bypass. Working tree clean at the head.++**At the milestone-10 pause, before the PR:** `make agent-check` with everything staged (so `drift-check` is a real green), `make docs-check`, and the full `make agent-test` were all green on the exact tree that was committed.++Only `cli-docs` came open at milestone 10 — no configuration model moved, so `config-docs` stayed shut. Its ack rationale records what was reviewed and what was deliberately left alone (`docs/features/cli.md`, `agent-cli.md`'s command table, `update.md`'s deck claim, and `index.md`, which has no doctor entry at all).++Both drift contracts came open at milestone 9 and were acked with the reviews written into the ack rationales: `config-docs` (which produced the telemetry page's new "When telemetry.toml will not load" section) and `cli-docs` (which produced the finding on the agent doctor's documented envelope, the `agent-cli.md` note that the telemetry error carries the block too, and the honest `init.md` warning that init replaces a file wholesale).++At milestone 8 the same two were acked: `config-docs` produced the `docs/configuration/index.md` fix the *previous* ack had explicitly parked for that milestone, and `cli-docs` produced the agent-CLI error-envelope documentation and the `migrate.md` agent paragraph.++### Milestone 7's own ack record, kept++Three contracts were acked at milestone 7: `hub-layering-convention`, `cli-docs` and `config-docs`.+## What is actually leftMeasured against the charter's own **Done when** list ([§ S6](../../wip/migrator-3/sequencing.md#s6--migrator-phase-3)), verified against the tree rather than against this file:@@ -225,15 +611,46 @@ Measured against the charter's own **Done when** list ([§ S6](../../wip/migrato|---|---|| The golden format is final for S7 (format bucket, dotted-key rename, `umig`, backup/replace in the contract) | ✅ Milestones 1 and 3 || R9 is built | ✅ Milestone 2 |-| The walk stops by name on a file two surfaces' globs both claim | ⚠️ Behaviour and test landed in Phase 2. The #1111 note-5 half is **owed**: re-aim it at the real specimen `.pipelex/inference/backends/pipelex_gateway.toml` and pin that the walk is non-recursive. A test, not a behaviour change. |-| `pipelex migrate --help` / `pipelex-agent migrate --help`, the bootstrap test, both end-to-end tests | ❌ Not started. No `migrate` command exists in `pipelex/cli/commands/`. |-| `release/SKILL.md` step 3b; `.claude/skills/add-migration/` | ❌ Neither exists. |+| The walk stops by name on a file two surfaces' globs both claim | ✅ Behaviour and test landed in Phase 2; the #1111 note-5 half landed in Milestone 5 (`TestTheWalkIsNotRecursive`, both halves, real specimen) |+| `pipelex migrate --help` / `pipelex-agent migrate --help`, the bootstrap test, both end-to-end tests | ✅ Milestone 5 — and the end-to-end pair runs against the **real** `telemetry-config@2` entry rather than current-shape fixtures, which is stronger than the charter asked for |+| `release/SKILL.md` step 3b; `.claude/skills/add-migration/` | ❌ Neither exists |++The phase body the charter lists under **Do**, in the order the next session should take it:++1. ~~**The downgrade diagnosis**~~ — done, see Milestone 6.+2. ~~**Boot tolerance**~~ — done, see Milestone 7.+3. ~~**`report_validation_error` on the real plan**~~ — done, see Milestone 8. The third channel of the rendering-rule test landed with it, as a new e2e class (`TestABootFailureCarriesThePendingMigration`) rather than inside `TestNoValueFromAUsersFileIsEverRendered`, because the reachable specimen is different: boot tolerance means a file the ledger *can* explain never reaches the error surface at all, so the channel is exercised by a key no entry explains, whose value is the planted secret.+4. ~~**The two telemetry remedies retired**~~ — done, see Milestone 9. There were four, and the worst of them was `doctor --fix`'s offer to reset a migratable file. The third surface, `pipelex_service_config.py`, was deliberately left alone — it has no wrong remedy to retire, and closing its gap costs the same models/loader split.+5. ~~**`doctor`: the pending-migrations row and `--fix`**~~ — done, see Milestone 10. Both halves: a row that is `pipelex migrate`'s own dry run across every surface, and fix mode reaching the command's write pass (not the command, which exits). The row deliberately takes no directory — it answers for a command that has no `--global`.+6. **The `config-docs` drift contract's `review` list** plus the four validator sites R8 names (`KitConfig._validate_targets`, `DryRunConfig.validate_image_urls`, `ImgGenConfig.validate_quality_mapping`, `LLMConfig.validate_effort_to_budget_mapping`).+7. **The skills**: `.claude/skills/add-migration/`, and `/release` step 3b.+8. **`docs/specs/command-surface-map.md` rows** — that file is in the **workspace-root** repo, not this one.+9. **Publish the contract**: delete `docs/migration-ledger.md`'s Status banner, remove its `not_in_nav` entry and comment, add a Migration Ledger row to the nav's Project section.++Then the part-3 PR to `dev` from `feature/Migrator-3c`, and Checkpoint C.-And the phase body the charter lists under **Do**, none of it started: boot tolerance in the shared config-surface helper (all three loaders); `report_validation_error` on the real plan — note `core/validation.py`'s current `migration_config` is the old *renaming* config, not the ledger plan; the downgrade diagnosis; the two telemetry remedies retired; the `doctor` row and `--fix`; the `config-docs` drift-contract review list plus the four validator sites R8 names; the `command-surface-map.md` rows; publishing the contract in the nav. Then the PR to `dev`.+### Deferred from the #1114 review-Three "fits here" items are still literally unread by anything, which is the cheapest way to see the commands are missing: `min_supported_schema_version` has no reader beyond its own validator, and `MigrationReport.written_plans` / `blocked_plans` / `MigrationLedger.entry_for_version` have no caller. `pipelex migrate` is the intended reader of all four.+The items the review-bot pass on PR #1114 raised and this branch parked rather than fixed are in [`migrator-pr-1114-review-followups.md`](./migrator-pr-1114-review-followups.md), each verified against the code and each with its trigger, and their owners are recorded in the charter's [S6 Session 4](../../wip/migrator-3/sequencing.md#s6-session-4--milestones-6-to-10-pr-1114-and-the-two-review-rounds):-**The next session starts at the two commands.**+- **§1, the out-of-walk warning text** → S7's final join, because the reshape's `pipelex-config@2` is the first `pipelex-config` entry and is what makes it live (no consumer in the workspace passes `config_dir=` outside tests, so it is an embedder wording fix, not a blocker).+- **§2, what a `migration` block's *presence* means for an unexplained-only plan** (agent hint, `_migration_prose`, the doctor's telemetry row, and the contract's own sentence — one ruling over four surfaces) → **part 3**, as a wording ruling the session may take on its own authority; not blocking S7, but S7's first machine is where a real user first meets it.+- **§3, the service loader warning before logging exists** (`pipelex init`, `pipelex plugins`) → named trigger: the first `pipelex-service-config` entry; the `warnings.warn` shape is on record.+- **§4, the doctor's remaining unescaped Rich fields** → part 3 if there is room, else any session; the log-channel half is deliberately not owed anywhere.+- **§5, a below-floor file that still validates boots silently** → named trigger: the first squash that moves a floor; documented design, and the doctor row already reports such a file.++### Small items still owed, folded in from the #1113 review++- ~~**`MigrationLedger.entry_for_version` still has no caller.**~~ Answered by Milestone 6: the downgrade diagnosis looks a blocked entry's own operations up from the `to_schema_version` its `BlockedEntry` carries, which is the first real caller. It stays.+- **`is_remappable` has no direct unit test** and gates two accountings; and §1 of `pr-1113-review-notes.md` (credit a union by its string branches alone and say so, or ask the non-string branches — about four lines on `_records_enumerated_members`) is still undecided. Cheap, and it belongs with whoever next touches `coverage.py`.+- **The rescue-copy race** (`pr-1113-review-notes.md` §3): a third run's prune can take the `.bak.` a rescue is about to rename. The commands now make concurrent runs producible, so this is decidable — revisit as an age-sparing prune, not a lock.+- Deliberately past S7's freeze, with named triggers, in `pr-1113-review-notes.md` and `migrator-write-scope-and-rename-fidelity.md`: per-member bounds in the golden, the `safe`-entry sibling, the tomlkit raw-storage staleness pass, rename fidelity (§2), the taken-backup-name report, and the residual. §1 of the write-scope note is **closed** by Milestone 5.++**The next session — part 3, on `feature/Migrator-3c` — starts at item 6, the `config-docs` drift contract's `review` list.** Everything that touches *code* in the phase body is now built — items 1 through 5, on `dev` once #1114 merges — and what is left is four items of a different kind: a contract's review list (item 6), two skills (item 7), a spec file in the **workspace-root** repo (item 8), and publishing the ledger contract in the nav (item 9). None of them depends on another, so they can be taken in any order; item 6 is the cheapest and item 9 is the one that closes the phase.++Item 6 concretely: extend `drift.toml`'s `config-docs` contract so its `review` list also covers the ledger files (`pipelex/migration/goldens/`, the ledgers themselves) and the four validator sites R8 names — `KitConfig._validate_targets`, `DryRunConfig.validate_image_urls`, `ImgGenConfig.validate_quality_mapping`, `LLMConfig.validate_effort_to_budget_mapping`. R8's point is that a validator is a schema change the fingerprint cannot see, so a change to one is a review obligation the same way a model field is.++Milestone 7's inherited finding — the dead `except ValidationError` around `setup_config` — **is fixed**, at both of that function's config-side callers, and the same defect turned out to exist in the doctor.## Rulings this session took on its own authority@@ -241,4 +658,12 @@ The charter says the format bucket is to be "decided and recorded here" — it dMilestone 3 took the policy calls the charter routed here by name — preserve dotted-ness rather than refuse it; follow a symlink rather than refuse or clobber it; never overwrite or remove a copy this run did not make; take a copy the run cannot vouch for out of the pruning rotation by name; carry the mode and not the ownership or the extended attributes; `fsync` the directory once, after the backup and before the replace. Each is written into `docs/migration-ledger.md` rather than only here.+Milestone 5 took two on the same basis, both written into `docs/migration-ledger.md`: **a migration's write scope is the resolved target of any file the walk claims** (the `.mthds` fix loop's scope guard is not copied, because a configuration directory is where a user keeps links to files they own); and **`min_supported_schema_version` gets a reader rather than a softened sentence** — a file declaring a version below its ledger's floor is refused by name, reading the reserved key exactly as tolerantly as boot strips it.+Milestone 2 took three further rulings on that same basis, each written into `docs/migration-ledger.md` rather than only here: the coverage gate now demands a declaration from an `unsafe` entry rather than accepting the word (without it R9 is half-built); convergence exempts a `VALUE_DOMAIN_NARROWED` report and nothing else (without it R9's ruled shape cannot be written at all); and forward tracing stops at `unsafe` entries (a rehearsal may guess, an application may not). The third is the one that leaves a named open question, listed above.++Milestone 9 took three on that basis, all written into `docs/migration-ledger.md`: **writing a fresh file repairs exactly one finding, a missing one** — every other unhealthy state holds the user's own settings, so an out-of-date file gets the migration and a broken one gets a person; **a probe's verdict is a field its callers branch on, never a sentence they search** — the doctor's own `--fix` had made the opposite choice and would have lost its repair path to a reword; and **a loader must be able to reach the migration package without reaching the registry**, which is what put the telemetry models and the telemetry loader in two modules. The scope call — leaving `pipelex_service_config.py` alone — is recorded in Milestone 9 above rather than in the contract, because it is a decision about this branch rather than about the ledger.++Milestone 10 took four on that basis, all written into `docs/migration-ledger.md`'s new § "The health report": **a row that reports on a command is scoped by that command, not by the flag the report was invoked with** — `pipelex migrate` has no `--global`, so a narrower row would name a command that rewrites a file it never mentioned; **not knowing is a finding of its own and is never health** — `UNAVAILABLE` sits beside the three verdicts rather than folding into `UP_TO_DATE`; **a fix is offered only where a command would actually write** — a prompt whose honest outcome is "nothing was written" is the same defect as Milestone 9's reset, one step along; and **a caller that has already asked its own question reaches the write pass, never the command** — a command owns the process's exit code, and a caller with rows still to render cannot let it.++Milestone 8 took four, each written into `docs/migration-ledger.md` rather than only here: **the scan is scoped by the surface the caller names**, so a `.mthds` bundle or a backend file pays for no walk and is offered no remedy it has none for; **the block carries `MigrationPlan` itself** rather than a projection of it, which is what forced `MigrationSafety` out of `ledger.py` into `safety.py` — a second shape would have drifted from the commands' own; **scoping narrows the answer and never the registry**, because ownership is decided across all surfaces and a one-surface registry hands `pipelex_service.toml` to the wrong ledger; and **a failure inside the scan declines quietly, but only on `(MigrationError, OSError)`** — an applier bug is not a field condition and must keep surfacing.diff --git a/wip/migrator-pr-1114-review-followups.md b/wip/migrator-pr-1114-review-followups.mdnew file mode 100644index 000000000..eac4c8ad6--- /dev/null+++ b/wip/migrator-pr-1114-review-followups.md@@ -0,0 +1,51 @@+# Migrator Phase 3, part 2 — items deferred from the PR #1114 review++What the review-bot pass on [PR #1114](https://github.com/Pipelex/pipelex/pull/1114) surfaced and this branch did **not** fix, with why. Each was verified against the code before being parked; none is a live user-facing defect today. The confirmed important items of the first round — `changed_plans` blind to a partly-applied entry, the boot-tolerance warning logged before logging exists, the retry ignoring the schema-version floor, the diagnosis scanning the wrong directories under `--global`, and the doctor's unescaped Rich fields — were fixed on the branch. The second round fixed three more: the doctor's telemetry row answering for the whole directory rather than the file it read, the `pipelex migrate` cancel path exiting 0 with something left for a person, and the summary panel reporting a green "Migrated" over a file whose write could not be confirmed.++## 1. The stale warning names `pipelex migrate` for a directory the command never walks++**Reporters:** Codex (`config_loader.py`, `_config_the_ledger_can_explain`), Cubic (`migration/run.py`, `config_directories_to_migrate`) — [thread](https://github.com/Pipelex/pipelex/pull/1114#discussion_r3792653243), [thread](https://github.com/Pipelex/pipelex/pull/1114#discussion_r3792679325).++An embedder's `Pipelex.make(config_dir=…)` outside `~/.pipelex/` and the project `.pipelex/` is read by boot tolerance (the retry replays exactly the files the loader merged) but not by `pipelex migrate`, whose walk is fixed to the two directories by design (`docs/migration-ledger.md` → "Where the ledgers are"). So a stale file there would boot with a warning whose closing sentence — "run `pipelex migrate` to bring the files up to date" — names a command that will not touch it, on every boot.++**Why deferred:** unreachable today. The `pipelex-config` ledger ships no entry, so its retry never fires; the telemetry loader takes no `config_dir`; the service configuration is loaded from the global directory, which is in the walk. It goes live with the first `pipelex-config` entry. The docs now say plainly that such a directory is the embedder's to update; the warning text does not yet.++**Recommendation:** when the first `pipelex-config` entry ships, have `stale_configuration_warning` take the walked directories and, for a stale file outside them, say "this file is outside `pipelex migrate`'s reach — update it where it lives" instead of naming the command. Do **not** add `pipelex migrate --config-dir`: `check_pending_migrations` deliberately takes no directory so the doctor row can never name a command that then rewrites a file it did not mention. This sits beside `migrator-write-scope-and-rename-fidelity.md`, which parks the sibling question of writing outside the walk.++## 2. What the presence of a `migration` block means when the plan is unexplained-only++**Reporter:** Cubic (`agent_output.py`, the `TelemetryConfigValidationError` hint) — [thread](https://github.com/Pipelex/pipelex/pull/1114#discussion_r3792679317).++`_pending_migration` attaches the block whenever any plan is not clean, and a plan carrying only `unexplained` paths is not clean. So a misspelled key in `telemetry.toml` — `extra="forbid"` refuses it, the diagnosis reports it as unexplained, the retry does not repair it — produces a `migration` block with `needs_attention: true` and empty `steps`. The agent hint then says "the configuration is out of date: run `migrate --dry-run`, then `migrate --yes`", and `--yes` writes nothing. The mechanism is correct; the wording over-claims.++**Why deferred:** the hint faithfully implements a documented contract — `docs/tools/cli/agent-cli.md` ("Its presence is the whole signal: the configuration is *out of date* rather than wrong") and `docs/migration-ledger.md` → "Reporting a stale configuration on a validation error" say the same — and `_migration_prose` ends the human message with the same unconditional "run `pipelex migrate`". Rewording the hint alone would leave the CLI and its spec disagreeing. This is one ruling over four surfaces: what "presence" means, or whether the block should carry (or the hint should point at) "is there anything the command would write" as distinct from "is there anything a person must look at". The `--dry-run` the hint sends the agent to does print the unexplained note ("either the name is a typo, or this file was written by a newer pipelex"), so the wasted round trip is the whole harm.++The doctor's telemetry row is a further surface of the same ruling ([thread](https://github.com/Pipelex/pipelex/pull/1114#discussion_r3792785607), which reported the sibling-file bug that was fixed): `_telemetry_is_out_of_date_or_wrong` now reads the plan of the file the row is about, but still branches on `not plan.is_clean`, so an unexplained-only file — a typo under `extra="forbid"` — reads `out_of_date`, names `pipelex migrate` (which writes nothing), and suppresses the manual-fixes section. Whatever "presence" is ruled to mean, that row follows.++## 3. The service-configuration loader still warns directly, and two commands load it before logging exists++**Surfaced by:** the fix for Cubic's `runtime_hub.py` finding — [thread](https://github.com/Pipelex/pipelex/pull/1114#discussion_r3792679304).++The main configuration's warning is now parked on the loader and emitted after `log.configure` (the main configuration *is* the log configuration, so nothing else was possible). The telemetry and service loaders kept their direct `log.warning`, which is safe on every path that exists today for telemetry (loaded inside `setup()`) — but `load_pipelex_service_config_if_exists` is also called from `pipelex init` (`cli/commands/init/command.py`) and its warning would raise `LogConfig is not set` there once the `pipelex-service-config` ledger gains an entry. `pipelex plugins` calls `load_config_validated` without configuring logging and would now silently drop the parked warning rather than crash.++**Recommendation:** when either ledger gains an entry, a `StaleConfigurationWarning(UserWarning)` raised through `warnings.warn` on the `RemoteConfigStaleWarning` pattern (`pipelex_service/exceptions.py`, captured by the agent CLI factory) covers all three loaders and every entry point uniformly, at the cost of a second presentation channel. Not worth building against an empty ledger.++## 4. Pre-existing unescaped Rich fields in the doctor report++**Surfaced by:** Cubic's two escape findings on `doctor_cmd.py` — [thread](https://github.com/Pipelex/pipelex/pull/1114#discussion_r3792679312), [thread](https://github.com/Pipelex/pipelex/pull/1114#discussion_r3792679315).++The telemetry row and the pending-migrations row now escape what they print (this PR is what put user-file content into the telemetry message). `config_message`, `backends_message`, `models_message` and `deck_message` in `display_health_report` (and the Manual-Fixes block) still interpolate unescaped, and `config_message` embeds `report_validation_error(...).message` too. A bracketed path segment is silently dropped by Rich and a `[/x]` sequence raises `MarkupError`.++**Recommendation:** one pass over `display_health_report` escaping every dynamic field, with the recorded-console test pattern from `test_doctor_display_report.py`. Small, mechanical, and outside this PR's scope.++Cubic raised it again in the second round for `config_message` specifically (this PR routes `report_validation_error(...).message` into it) and, in the same run, asked for the paths in `stale_configuration_warning` (`config_surface.py`) to be escaped. The second one is the wrong layer: that function is a plain-string builder whose consumers are `log.warning` — which feeds a `RichHandler(markup=True)` under `is_markup_enabled` *and* a plain `StreamHandler` in `POOR` mode — and two Rich render sites that already `escape()`; a `\[`-escaped string is corrupt in the poor sink, and no builder in the tree escapes (`core/validation.py` builds the identical list unescaped). The residual is a property of the log channel — every logged path or exception text across the codebase is interpreted as markup — and its one correct home, if it is ever wanted, is the rich branch of the log dispatch, not the builders.++## 5. A file declaring a version below the floor still boots when its body validates++**Reporter:** Cubic (`config_loader.py`, `load_config_validated`, P1) — [thread](https://github.com/Pipelex/pipelex/pull/1114#discussion_r3792785605).++The trace is accurate: `load_config` strips `[meta]` before validation, `load_config_validated` returns on the first clean `model_validate` without consulting a ledger, and `version_declared_below_the_floor` runs only inside `replay_surface_files_in_memory` — the failure path. So a file declaring a schema version below `min_supported_schema_version` whose body happens to match the current model boots silently. The same holds for the telemetry and service-configuration loaders; the bot reported one instance of a uniform structure.++**Why deferred:** it is the documented design, not an oversight, and the proposed fix breaks two invariants at once. `docs/migration-ledger.md` → "The healthy path is untouched" says a current configuration never reads a ledger, never re-parses a document, never imports the migration engine — checking the floor *before* validation means tomlkit-parsing every merged path plus a ledger load on every healthy boot, because the floor is per file and the merged dict cannot answer it. The floor's stated purpose is that `pipelex migrate` never reports a squashed-over file clean; the retry declines through the same predicate so boot never carries forward a file the command then refuses. Refusing at boot would turn a working configuration into a hard failure with no in-product remedy, since `migrate` refuses such a file too ("migrate it with a release that still carries the entries, or re-create the file"). And it is dormant: every shipped ledger has `min_supported_schema_version = 0`, and `declared_schema_version` returns `None` for anything that is not a plain non-bool integer. The channel that does report it exists: `check_pending_migrations` runs the dry run unconditionally, so such a file lands as a blocked plan → `attention_files` → `needs_attention` in `pipelex doctor`.++**Recommendation:** the day a squash actually moves a floor, decide whether a below-floor-but-valid file should *warn* at boot. If yes, the shape is an extension of the doctor row plus, at most, a warning through the parked-warning channel (`take_stale_configuration_warning`) — never a refusal, never a pre-validation parse of every file on a machine whose floors are all 0. Tests would follow `tests/unit/pipelex/system/configuration/test_boot_tolerance.py::TestTheRetryHonoursTheSchemaVersionFloor` (it already has the synthetic-ledger and declared-version fixtures) and `test_migration_run.py::test_a_file_declaring_a_version_below_the_floor_is_refused_and_left_alone`.