Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
feat!: storage_scope — the host says where a run's bytes go (#1129) (#1131)
Thomas Hebrardfdcc931499ba
diff --git a/.drift/acks/config-docs.toml b/.drift/acks/config-docs.tomlindex 7fa62aa1f..cc99c874d 100644--- a/.drift/acks/config-docs.toml+++ b/.drift/acks/config-docs.toml@@ -1,13 +1,13 @@contract = "config-docs"-digest = "sha256:bb8a1f8414fe7bf387b68beadaeedf44cededf5f2213f9fbdf94f84968df7056"-reviewed_by = "Louis Choquel"-reviewed_at = "2026-08-17T14:33:11Z"-rationale = "Phase E of S7b, docs and changelog for the fourth surface. The trigger is config_surface.py's module docstring, which had become self-contradictory: it listed backend definitions among the readers that should keep rejecting the reserved [meta] table, while the same paragraph now names them as a configuration surface. Corrected, and the asymmetry behind it recorded honestly - the backend loader still reads through the generic TOML reader rather than through config_surface, so a backend file carrying [meta] would be refused as an unknown model table even though the migration engine reads the key there. Nothing writes that key on any surface, so it is latent rather than live. Reviewed all four targets. docs/configuration/**: inference-backend-config.md gained the section the Phase B ack promised - what the boot says when a backend file is behind, which files pipelex migrate reaches and which it does not, that a key you added yourself is reported rather than dropped, and the backup beside each rewritten file; index.md line 62 already carried the one-clause pointer added in Phase B and its link now resolves to a real section; telemetry-config.md is about a different surface and is unaffected. pipelex/migration/ledgers/: no ledger was edited this session, all four are legal and convergent, and no configuration model moved. configs.py and config_cogt.py are untouched - no key removed, renamed or narrowed, so no validator-level narrowing is owed an unsafe entry."+digest = "sha256:78f92935ef035d0a239c3a731caf5f93549c6243ca1128efa5c078a7599bc8d3"+reviewed_by = "thomashebrard"+reviewed_at = "2026-08-19T21:35:11Z"+rationale = "uri_format now renders the FILENAME only: the prefix (storage_scope plus the new generated/ leaf) moved into GeneratedContentFactory, so {storage_scope} is retired from URI_FORMAT_SUPPORTED_FIELDS and the shipped default in pipelex/pipelex.toml, .pipelex/pipelex.toml and kit/configs/pipelex.toml becomes {hash}.{extension}; a leading / is refused too, since the value is appended after one. Reviewed all four targets. docs/configuration/**: still no page documents uri_format or the storage section (index.md line 54 names [runtime] storage only as a category), so nothing there went stale; the two docs outside this contract's targets that DO name it were updated anyway - storage-provider-plugins.md now says the format renders the filename and lists {hash}/{extension}, and pipelex-kernel.md now names four leaves instead of three, since generated/ joined assets/, results/ and payloads/. pipelex/migration/ledgers/: no entry, and that remains the considered answer - cmig reported the fingerprint move as additive ('nothing breaks and no entry is needed') and per docs/migration-ledger.md a free string's value domain can never be proven stale from the schema, so a remap_value on uri_format could only ever be unsafe (reported, never applied); the failure is loud at boot instead, where _collect_uri_format_error_msgs rejects a retired placeholder by name and lists the supported set. Goldens regenerated with umig; cmig and cl both pass. configs.py and config_cogt.py untouched - no key removed, renamed or narrowed, so no model-level narrowing is owed an entry."[trigger_files]"pipelex/cogt/config_cogt.py" = "blob:02f71cb4a6b01aebda59c2e31f262ef9c865e17b""pipelex/cogt/img_gen/img_gen_job_components.py" = "blob:084985080413fd8d6c3b1b9b0747020cb0161858"-"pipelex/pipelex.toml" = "blob:ce9fd8469393d7c6ee6195447b4e2810653a473e"+"pipelex/pipelex.toml" = "blob:a82dbc5011d2ceba5eb3ef2a463bc720ee0f169c""pipelex/system/configuration/__init__.py" = "blob:8b137891791fe96927ad78e64b0aad7bded08bdc""pipelex/system/configuration/config_check.py" = "blob:efa78599c08e5e003d975cc54978ea5d267214a0""pipelex/system/configuration/config_loader.py" = "blob:fadc4557422f0c93335446a854d9883b3cceaf8c"diff --git a/.drift/acks/pipelex-kernel-docs.toml b/.drift/acks/pipelex-kernel-docs.tomlindex 049ddaef6..b565fb654 100644--- a/.drift/acks/pipelex-kernel-docs.toml+++ b/.drift/acks/pipelex-kernel-docs.toml@@ -1,8 +1,8 @@contract = "pipelex-kernel-docs"-digest = "sha256:1d3cb4f0179a7920d265896a67b35c339b04b96cdb4efdcbc8d409e03e9fbdcb"-reviewed_by = "Louis Choquel"-reviewed_at = "2026-08-15T16:00:59Z"-rationale = "The three kernel modules changed only where they read the config tree: img_gen_ops reads inference.img_gen.img_gen_param_defaults and inference.img_gen.img_gen_job, llm_ops reads inference.llm, templating_style_ops reads inference.templating.default_templating_style. Four lines, all reads, no signature, no entry point, no gate, no new function. Reviewed pipelex-kernel.md against them: the page specifies the kernel's callable surface for programmatic callers, and it names config addresses nowhere, so nothing on it went stale. Checked the module table still lists every public function in pipelex/kernel/ and that the Gates row's deck-reading-helpers wording is untouched by this change. No doc change needed."+digest = "sha256:c87b2054eef43e624731aed0ac71cf54ca4f7dfbc011e4e72f14b54493ae3a23"+reviewed_by = "thomashebrard"+reviewed_at = "2026-08-18T11:19:04Z"+rationale = "PipelexKernel.make gained a required storage_scope beside a user_id that is now also required (both had a fallback path before: user_id coalesced to the tracing placeholder 'anonymous' deep in pipeline_run_setup, and the storage prefix was derived from it). Both code examples on the page called make(user_id=...) and would now be a TypeError, so both are corrected. Added a paragraph on the two fields to the boot section: what storage_scope is (an opaque prefix the caller composes from its own tenancy model, which the kernel composes leaves onto and never parses), that it is validated at construction because it becomes a storage key prefix, and why neither field defaults. Checked the module table at line 69 still lists the kernel's public surface - make/.llm_text/.llm_object/.make_step_metadata are unchanged in name and the only signature that moved is make's. The step_id_source paragraph is untouched by this change."[trigger_files]"pipelex/kernel/__init__.py" = "blob:c4f1b0fe2ccc17db9110733f71b612f4ac1ba4ce"@@ -20,9 +20,9 @@ rationale = "The three kernel modules changed only where they read the config tr"pipelex/kernel/llm_prompt_content.py" = "blob:82c34283dacd2bfd9a2c263915e744736d399875""pipelex/kernel/llm_results.py" = "blob:6e3b8a2878b5bb284ece5fac10262fb59ac2029d""pipelex/kernel/memory_ops.py" = "blob:f6a5745e804ac08f19c70f1beb1f78baf50d51cd"-"pipelex/kernel/pipelex_kernel.py" = "blob:3264150e92940c97346f1fb5dd9313eff8825d50"+"pipelex/kernel/pipelex_kernel.py" = "blob:47009beeb3cef974eb2b738cca0aaa853cccddde""pipelex/kernel/prompt_references.py" = "blob:03ec9ef67ae84f141d072ca2d2c3ef888549ccc0""pipelex/kernel/search_ops.py" = "blob:ed0a8c7953aca1e38de852d0ddef11731b819539""pipelex/kernel/search_results.py" = "blob:101b70dc96cc5ee1721c9f6b1cb725a5ca78ff70""pipelex/kernel/templating_style_ops.py" = "blob:395473806d600dfc4f124c07f2afebb951fc086b"-"tests/unit/pipelex/kernel/test_kernel_boot_contract.py" = "blob:2cbcb3924b1291d741133f8b1e9538622d06a721"+"tests/unit/pipelex/kernel/test_kernel_boot_contract.py" = "blob:0f5f3239738781d20d13c458071382a0fc6931aa"diff --git a/.pipelex/pipelex.toml b/.pipelex/pipelex.tomlindex 3f0a0525d..1a9242f26 100644--- a/.pipelex/pipelex.toml+++ b/.pipelex/pipelex.toml@@ -46,26 +46,26 @@ method = "local"[runtime.storage.local]# Local storage settings-uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"+uri_format = "{hash}.{extension}"local_storage_path = ".pipelex/storage"[runtime.storage.in_memory]# In-memory storage settings-uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"+uri_format = "{hash}.{extension}"[runtime.storage.s3]# AWS S3 storage settings (requires boto3: `uv pip install "pipelex[s3]"`)-uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"+uri_format = "{hash}.{extension}"bucket_name = ""region = ""-signed_urls_lifespan_seconds = 3600 # Set to "disabled" for public URLs+signed_urls_lifespan_seconds = 3600 # Set to "disabled" for public URLs[runtime.storage.gcp]# Google Cloud Storage settings (requires google-cloud-storage: `uv pip install "pipelex[gcp-storage]"`)-uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"+uri_format = "{hash}.{extension}"bucket_name = ""project_id = ""-signed_urls_lifespan_seconds = 3600 # Set to "disabled" for public URLs+signed_urls_lifespan_seconds = 3600 # Set to "disabled" for public URLs##################################################################################################### Secrets Configdiff --git a/CHANGELOG.md b/CHANGELOG.mdindex 2d25feda9..51a698107 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -8,12 +8,72 @@- **Line endings are pinned to LF in the git index and on checkout, on every platform:** this repo carries a lot of generated-but-committed text — `CLAUDE.md` and `AGENTS.md`, `subject_grants.toml`, the drift acks, `.test_durations`, the error-identity snapshot, the generated error pages, the migration goldens, the gateway model references — and each is produced by a writer that passes no explicit `newline`, so on Windows Python translated every `\n` to CRLF and the whole set churned in version control with no change of content. No gate would have caught it: they all compare with universal-newline reads, which fold CRLF back to LF before the comparison. A `.gitattributes` carrying `* text=auto eol=lf` closes that class for tracked files whichever writer produced them. `text=auto` leaves git to tell text from binary, so images and other binaries are untouched, and no tracked file held CRLF when this landed — it introduces no renormalisation diff, it pins the invariant. The complementary writer-by-writer fix is the structural half and is tracked separately.- **Documentation:** `docs/under-the-hood/codegen-projections.md` now covers the strict stamp-header rules, the JSON conformance the `options` payload must meet, the policy for evolving the lock format under `lock_version`, and the cross-platform line-ending guarantee.+- **A configuration that will not load now says whether it is wrong or merely old**: A validation error is raised against the merged configuration and carries no provenance — it names a key, not which file put it there, and certainly not whether that key was correct last month. When a configuration surface's model refuses, the error now also carries a dry-run scan of that surface over the same directories `pipelex migrate` walks: a structured `migration` block naming the remedy, whether that remedy would actually rewrite anything, whether anything there is a person's to resolve rather than the tool's, and the plan per file — the same shape `pipelex-agent migrate --dry-run --format json` emits, so an agent that parses one has already parsed the other. `error_domain` deliberately stays `config`: it is a closed cross-repo enum and the agent-hook specification routes anything else to BLOCK, so a stale configuration reported under a new domain would stop an agent instead of telling it what to run. Consumers branch on the block's presence — absent means the failure is not staleness — and then on `would_write`, because presence means the migration history has something to _say_ about these files and not that a command repairs them: a block whose only finding is a path no entry explains, or an entry blocked before anything applied, has nothing to run, and naming the remedy there would send a reader to a run that writes nothing and leaves the same error standing. Every surface points that reader at `pipelex migrate --dry-run` instead, where the diagnosis is. The message gains a paragraph naming the files and the next move. No value read from a user's file appears in the block, which is the third of the three channels that rule covers. A caller validating something that is not a configuration surface — a `.mthds` bundle, a model deck, a routing profile — names no surface and pays for no walk.++- **Inference backend definitions are a migration surface, and the key the templating change deleted is repaired on every machine that has one**: `#1104` removed `prompting_target` from the model-spec blueprint and from every backend file we ship — but `pipelex init` never overwrites, so the key survives in `inference/backends/*.toml` on every installation that predates it, where it is fatal twice over: in `[defaults]` it is copied wholesale into every model of the file, and on a single model it is rejected by name since unknown per-model keys must be header-shaped. Those files were reachable by no surface, no walk and no gate. They are now the `inference-backend` surface, which required the walk to learn about subdirectories — a file is claimed by the pair (directory, name), one level deep, and only into a directory some configuration family owns — and required the fingerprint to record a document whose _root_ is the open node, since a backend file's root keys are model names rather than a field we could name. Boot tolerance joins at the backend loader too: a stale directory is replayed in memory, boots on the models the current files would produce, and warns once naming every file it carried. **Migration:** run `pipelex migrate` after upgrading — ledger entry `inference-backend@2` deletes `prompting_target` from every root table of every backend file, in `~/.pipelex/` and in each project `.pipelex/`, keeping one timestamped backup per file. Until you run it, each boot repeats the warning; nothing is ever written by a boot. A key _you_ added that we have never heard of is still an error rather than something the entry silently drops, and the served Pipelex Gateway specs are untouched. The one key class the report stays silent about is a per-model request header, legal by shape rather than by name — and only on a model table: `[defaults]` is copied into every model of a file unsplit, so a header put there breaks all of them, and the report names it like any other key the schema cannot explain.++- **A configuration file the ledger can explain no longer stops the boot**: A schema change used to leave every machine in the field with a boot that died on `extra="forbid"`, naming a key the user never chose to have. When a configuration surface fails to validate, its ledger is now replayed over the same files **in memory** and the result validated again; a boot that succeeds says so in a warning naming the files and, for each of them the command would actually reach, the `pipelex migrate` remedy. The retry replays whatever the loader merged, so a file loaded through `Pipelex.make(config_dir=…)` from outside the two directories `migrate` walks is named instead as the caller's to update where it lives, rather than pointed at a command that would then report nothing to do. Nothing is written either way — a tolerated boot leaves the file and its directory exactly as it found them, which is why the warning keeps coming back until the command is run. Tolerance widens what starts and never what is accepted: the re-validation is what decides, so material an `unsafe` entry is about still fails the boot because the model still refuses it. Anything that goes wrong inside the retry makes it decline rather than replace the error the user already has — theirs names the key to fix. A healthy configuration pays nothing: it reads no ledger, re-parses no document, and does not even import the migration engine, which is deferred for a second reason too, since the applier lives in the interpreter layer and the configuration loaders sit in the kernel layer that must not load it. This release is where it matters most: the configuration reshape gives `pipelex.toml` its first ledger entry, so the file every boot reads is the one tolerance carries forward on every machine that upgrades.+- **`pipelex migrate` and `pipelex-agent migrate`**: The migration engine now has commands, so a configuration that predates the installed pipelex can be repaired instead of re-initialized. Both walk the global `~/.pipelex/` and the project `.pipelex/` — those two directories only, one level deep, and into a subdirectory only when a configuration family owns it: `inference/backends/` is such a directory, `inference/deck/` is not and is never entered. A file is claimed by the pair (directory, name), which is what keeps `inference/backends/pipelex_gateway.toml` — a `pipelex_*.toml` match by name — from being rewritten under the main configuration's ledger. Neither command boots: a broken configuration is the reason to reach for them, so they use the ledger, the applier and the filesystem and nothing else, and a test hands them a configuration that cannot load to keep it that way. The human command plans, shows the plan and asks; the machine one writes only with `--yes`, since it cannot ask, and passing `--dry-run` and `--yes` together is refused rather than resolved. Its answer is structured, and `needs_attention` is the verdict — _this run left something a person has to decide_, deliberately not _did anything get written_ — with the exit code and the Markdown rendering as presentation. Every file rewritten is backed up beside itself first, and no value read from a user's file appears in either command's output or in the structured plan.+- **A migration now names what it cannot explain about a file**: A file _newer_ than the running pipelex — an override migrated on one branch, then an older branch checked out — breaks the boot while a replay finds nothing to do, because every operation's source is absent. That silence beside a dead boot is the failure the migration project exists to remove, so every plan now carries an `unexplained[]` listing the paths of the migrated document that the current schema does not know and no ledger entry accounts for, each with the two readings the tool cannot tell apart: a typo, or a file written by a newer pipelex. This is the one part of a migration that needs the configuration model — the question goes to the surface's fingerprint, the same projection the coverage gate diffs, so the replay engine stays model-free. The document it reads is the one the run leaves behind rather than the one it found, which is what keeps every key the ledger is about to repair out of the list, and a path a _blocked_ entry is about is left to that entry, which reports it by name with its own guidance. A key the user chose is never rendered: beneath an open mapping a typo is reported at `queues.*.retries`, naming only the segment the schema does not know.+- **A configuration file may now be refused for declaring a schema version its ledger no longer reaches**: `min_supported_schema_version` had no reader, which left the floor a comment. A migration now reads the reserved `[meta] schema_version` — exactly as tolerantly as the configuration loader strips it, so a malformed declaration is no declaration — and refuses a file below the floor by name (`unsupported_schema_version`), writing nothing. Without it, a ledger whose oldest entries had been squashed away would run over a file older than them, change nothing, and report it clean, because the applier skips an operation whose target is absent.++- **An `unsafe` migration entry now says what it is about, and keeps saying it**: `unsafe` is the ledger's form for a change only a human can make, and the coverage gate's only remedy for a tightened numeric bound — but the engine decided whether to report one by rehearsing its operations, so an entry with none was accepted by the accounting and then guaranteed to reach nobody, and any entry went silent the moment a later `safe` entry renamed the table its material sat in (reported on the first run, clean on the second, boot still failing). An entry now declares the paths whose value domain narrowed, in `declared_narrowed_paths`; an entry with no operations must carry one, a `safe` entry may not, `make check-ledger` refuses a path the entry's own version does not record, and the coverage gate no longer takes the bare word `unsafe` as accounting for a narrowing or a lost enumerated spelling — it demands the path be named. The material of an `unsafe` entry is traced through the ledger before a document is questioned: back through the entry's own operations, since it never applied them, and forward through every later `safe` entry, so the same file is reported on every run for as long as it carries it. A file that merely _sets_ a narrowed path is reported under a new, weaker `value_domain_narrowed` reason listing the keys to check by hand — the engine is model-free by design and cannot tell a value the new domain refuses from one it accepts — and those keys are named at the spelling the migrated file carries rather than the one that matched, since the entry is questioned where the replay reaches it and the later `safe` entries rename the material before the file is written — and that report, alone among blocked entries, does not fail convergence, since a healthy reference document sets those paths like everyone else's file does. Forward tracing is deliberately confined to `unsafe` entries: a rehearsal may guess, an application may not.++- **A migration operation can now remap every value beneath an open mapping**: `remap_value` accepts `key = "*"`, meaning "each key of the addressed table" — the same "each of these" reading the wildcard already had in a `table_path`, applied to keys. A mapping from the user's own keys to an enumerated value (`dict[str, LogLevel]`) keeps its spellings under keys only the document can enumerate, so no fixed key reaches them and this is the only shape in which a member renamed beneath one can be repaired at all. On every other kind a `*` key is now refused when the operation is parsed: deleting every key of a table is `delete_table`, and renaming or moving every key onto one fixed name is the many-to-one that already rules a wildcard out of a `move_key`. Refused rather than tolerated, because an unrefused one is a _dead_ operation that skips forever while looking like an accounting.++- **A schema change that narrows what a value may _be_ is now caught, not waved through as additive**: A change can keep every path and every enumerated spelling and still break a user's file — `int` becomes `str`, a free string becomes an enum, a bound moves from `ge=1` to `ge=5`. Read as a path diff those look like additions, so the coverage gate used to ask for a golden regeneration, demand no version bump and no entry, and let the next boot reject a valid file with a green gate behind it. It now splits its verdict by _direction_: a widened type or a relaxed bound still costs only a regeneration, while a narrowing demands a bump and an entry carrying a `remap_value` for each narrowed path — or marked `unsafe`, which is the only remedy a tightened numeric bound has, since no structural operation can repair a value. The same rule applies to a stored entry, compared by origin so a narrowing hidden behind a rename is still caught, and to a `pre_history` entry, so the flag is not a way past it. To see a bound at all the fingerprint now records one: a **closed whitelist** over `gt`, `ge`, `lt`, `le`, `min_length`, `max_length` and `multiple_of`, read from `annotated_types` (now a declared dependency) and from the field metadata pydantic folds a top-level `Field(ge=1)` into, with anything unrecognized dropped rather than serialized — `pattern` excluded permanently, because regex containment is not decidable and every pattern edit would read as a tightening. One correction rides along: an enumerated type relaxed into a free string no longer reads as the loss of every spelling it had, and that is read structurally, so `list[enum]` becoming `list[str]` is the same benign loosening one container down — while a `list[enum]` flattened to a bare `str`, which is no loosening at all, still loses every spelling it had. The contract states in the same change what the gate structurally cannot see — narrowing expressed in a validator, which no projection of the annotations can reach.+- **The telemetry configuration can now be migrated instead of thrown away, and the escape hatch that would have let it ship unverified is closed**: `telemetry.toml` carries the package's first migration entry, `telemetry-config@2`, which lifts a flat first-generation file — `telemetry_mode`, `host`, `project_api_key` and the rest at the document root — into today's `[custom_posthog]` section, keeping every value the user chose; four settings the current shape has no home for are dropped, and the entry says which and why. Because that change predates the first fingerprint, no diff describes it, and such an entry is marked `pre_history` and exempted from the coverage gate's accounting — so the verification that replaces it lands in the same change and neither ships without the other. A pre-history entry must declare the paths it removes (an empty declaration is refused when the ledger is parsed), none of those paths may appear in any fingerprint at or below its own version, the fingerprint pair it sits between must show no removal at all, and every operation must act inside the declaration. It also ships a hand-authored `[email protected]` beside the golden chain, which the transform check migrates and holds against the current shape and the current model — the same three claims every other entry answers. One comparator refinement came with it: a created path is now checked against `defaults@N` **or** `fingerprint@N`, because an optional key defaulting to `None` is a legal destination that no TOML document can carry.+- **Configuration surfaces now have a migration ledger and a gate that forces one to be written**: Every configuration surface — `pipelex.toml` and its tiers, `telemetry.toml`, `pipelex_service.toml`, and the inference backend definitions — carries a checked-in ledger at `pipelex/migration/ledgers/<surface-id>.toml` recording, as data, every shape change it has ever undergone, plus a golden chain (`fingerprint@N`, `defaults@N`) that pins what the schema looked like at each version. Each is cut at schema version 1 with no entries; the entries this release then ships on top of that baseline are the bullets below. The new `make check-migration-schemas` (alias `cmig`, in `make check`) recomputes each surface's fingerprint and refuses a change that would break a user's file without recording how to repair it — a removed path or a lost enumerated spelling demands a version bump and an operation accounting for it, a destination the new schema does not know is refused by name, and an entry that removes a path the schema still has is refused as over-deletion. Its counterpart `make up-migration-schemas` (alias `umig`, in `make up`) regenerates the goldens; the regeneration diff is the review artifact. One standing invariant is checked whether or not anything changed: every required path must have a value in its surface's defaults layer, because that is what makes an added key absorbable rather than breaking. The contract the whole thing implements is `docs/migration-ledger.md`.+- **The migration operation vocabulary**: Fix operations became a discriminated union on `kind`, published as two subsets — every kind for the `.mthds` fix path, and the structural kinds only for ledgers, so an operation that would write a value into a user's file fails when the ledger is parsed rather than at review time. Two new kinds land with it: `move_key`, which relocates a key across tables (the whole subtree travels when the key is table-valued, and a missing destination parent is created), and `remap_value`, which rewrites a value against an explicit old-to-new mapping. A `*` segment addresses every entry of an open mapping, expanded over the keys the file actually holds. A new `CONFLICT` outcome separates "this cannot be done without choosing on your behalf" — a rename or move onto an occupied destination, or an `ensure_table` over a key holding something that is not a table — from the benign `SKIPPED` it used to hide inside; a conflicting operation writes nothing at all.+- **`[meta] schema_version` is reserved in configuration files**: Every configuration-surface reader now tolerates and strips it. Nothing writes it — the key exists so a later release can stamp a version into a file without that being a breaking change today.+- **`make check-ledger` (alias `cl`)**: The second migration gate, and the first one that joins `make agent-check` as well as `make check`. It asks what a ledger entry is allowed to _say_ and whether replaying it is harmless, where `check-migration-schemas` asks whether a schema change was accounted for. An operation may only act on material some schema version removed — one addressing a path the schema still has would rewrite a perfectly valid file on every run — and it may never name a concrete key beneath an open mapping, whose keys are the user's and unbounded, while the `*` segment that addresses every entry of one is legal exactly there and nowhere else. A `safe` value remap whose old spelling is still legal, or whose target is a free string where staleness cannot be proven, is refused because it would rewrite a deliberate choice; a remap to a spelling the new schema rejects is refused because every migrated file would then fail to load with the tool reporting success. A retired path, or a remapped-away spelling, may never come back. Finally, replaying the whole ledger over each of the surface's two reference documents — the complete packaged configuration and the sparse kit starter template, deliberately different shapes — must apply nothing, report nothing, and hand back the very bytes it was given. It reads checked-in files only and fingerprints no live model, which is what makes it safe to run in the loop agents run constantly: every failure names a file the author wrote.+- **The migration engine**: A surface's ledger can now be replayed over a real file. The engine takes a document's text and returns the text it should now have, plus a per-file plan naming the entries that applied, the entries it would not apply, and why. Two reporting rules keep a plan from being more optimistic than the file: an entry is reported once — a conflicting operation routes the whole entry into the blocked list, carrying whichever of its operations did land — and an `unsafe` entry is rehearsed against the document first, so it stays silent on a file that does not need it rather than warning at every boot forever. Writing is transactional per file and always takes a backup: exactly one per file, named `<file>.bak.<UTC stamp>`, inheriting the source file's mode rather than the umask, written before the replacement and pruned only after it commits. A file that is unparseable, unwritable or changed during the run is reported as blocked while every sibling is migrated normally — unlike the `.mthds` fix loop, whose files only make sense together and which still commits a round all-or-nothing; both now share one set of transaction primitives. Nothing calls this from the command line yet.+- **The transform goldens**: `make check-migration-schemas` now also _performs_ each migration it is asked to believe in. For every ledger entry it applies the entry's operations to the frozen reference document of the version before it and compares the result with the frozen document of the version after: every path the migration creates must be one the new document has, every path the two documents share must survive, and the last link's output — read the way a user's file is really read, beneath the current defaults layer — must be accepted by the current model. That closes the one defect the other two checks pass in silence: a rename whose destination is misspelled is accounted for by coverage and skipped by convergence, and then migrates every file to a key the schema rejects while the tool reports success. The comparison is deliberately one-directional in each claim, so a bump that also adds keys, edits comments, flips defaults or ships one more entry in a packaged deck stays green.+- **Replay neutrality is now a property rather than an assertion**: The guarantee a user's machine depends on — replaying a surface's whole ledger over a file already valid at the current schema changes not one byte — is checked over generated documents, not only over the two reference documents. A reference document carries exactly one value per key, so it cannot see an operation that misbehaves on the _other_ legal spelling of an enumerated field; a sampler that reaches those spellings can. The sampler proposes each mutation from the surface's fingerprint and lets the models decide whether it lands, because schema membership is settled by validators no type projection can see: a legal enum member can still be rejected for what it requires elsewhere in the document, and a mapping typed with arbitrary string keys can still refuse a key outside a fixed set. Idempotence — replaying twice equals replaying once — and prefix coherence — a file caught halfway lands where a replay from zero lands — are checked alongside it, and each generator carries its own vacuity check, since every one of these properties is trivially true over documents nothing acts on. `hypothesis` joins the development dependencies.+- **`pipelex doctor` reports pending configuration migrations, and `--fix` runs them**: A configuration the migration history explains now boots with a warning rather than an error — and `pipelex-agent` silences logging process-wide as its first act, so a machine consumer never learns from a boot that its configuration is stale. Asking was the only channel left and there was nothing to ask. The health report gains a **Configuration Migrations** row that is `pipelex migrate`'s own dry run: a structured `finding` (`up_to_date`, `pending`, `needs_attention`, `unavailable`), the files a migration would rewrite, and separately the files it will not repair on its own — both sets, because a drifted machine usually has some of each and a reader who heard only the first would stop with a broken file still in place. `pipelex doctor --fix` then offers to run the migration, reaching the command's own write pass rather than a second implementation of it. The row is deliberately not scoped by `--global`: it answers for a command that walks both configuration directories, and a row naming fewer files than the command touches would be a surprise write. A failure inside the scan is reported as a failure to check rather than as health, and never as an exception — an exception there would replace every row of the report with a single line.++### Changed++- **Breaking: where a run's bytes go is now told to the runtime, not derived from who is running (`JobMetadata.storage_scope`)**: Storage keys were built from `user_id` — `{user_id}/results/{run_id}/` for delivery, `{primary_id}/{secondary_id}/…` (always `{user_id}/{pipeline_run_id}/…`) for generated content, a flat top-level `normalized/` for inputs pulled off a `data:` URL. That ties _where the bytes live_ to _who uploaded them_, which is fine until two people share the work: the second one asks for a file the first one attached and is refused, because the only thing standing between them is a string comparison on the first path segment. A host that serves teams cannot express "these two people may read the same file" in that layout at all.++ `JobMetadata` gains a required `storage_scope`: one opaque, host-supplied prefix that the runtime composes `assets/`, `results/` and `payloads/` onto and never parses. The runtime does not learn what a tenant, an organization or a method is, and that is deliberate — those are host concepts with no meaning in an MIT-licensed runtime, so threading them through the transport was rejected in favour of one string the host fills however its own tenancy works. `PipelexPipeRunInput` carries it across the boundary alongside a now-required `user_id`.++ **It is validated at construction, on the type.** The value becomes a storage key prefix, so a `..` or a leading slash in it is a traversal out of the tenant's namespace; a `JobMetadata` that exists is one whose scope is safe, which makes every key derived downstream safe by construction rather than by each call site remembering. This replaces a per-segment sanitizer that ran over `user_id` and `pipeline_run_id` separately — collapsing them into one slash-bearing string would otherwise have deleted that control silently.++ **`uri_format` loses both `{primary_id}` and `{secondary_id}`, replaced by one `{storage_scope}`.** Supported placeholders are now `{storage_scope}`, `{hash}` and `{extension}`, and the shipped default becomes `{storage_scope}/{hash}.{extension}`. This is not a rename of `{primary_id}`: keeping that name would have left the configuration saying "primary" with no "secondary" for it to be primary to, and given an operator no way to tell that the placeholder means _the whole tenant prefix_ rather than _some id_. `{hash}` is unchanged — the first 16 hex chars of the SHA-256 of the file's own bytes, so identical output keeps one name instead of accumulating duplicates.++ A configuration naming either retired placeholder is refused **at boot**, by name, with the supported set listed. The free-string value domain is not something the migration ledger can prove stale — per `docs/migration-ledger.md` a `remap_value` there could only be `unsafe`, i.e. reported and never applied — so this is a loud failure rather than a repair. Edit `uri_format` in your `.pipelex/pipelex.toml` if you customized it.++- **Breaking: `"anonymous"` no longer appears anywhere on the identity or storage path, and a test keeps it that way**: The dry-run paths still passed `OTelConstants.DEFAULT_USER_ID` as a `user_id`, which is the last place the telemetry placeholder was still being bound as an identity. They now pass `DRY_RUN_USER_ID` — a dry run has no caller in the identity sense either, so it says so in its own word rather than borrowing tracing's. The constant survives for telemetry, where "a span with no known caller" is a real concept.++ `test_storage_scope.py` now scans every module under `pipelex/` and fails if `DEFAULT_USER_ID` is ever bound to a `user_id` or a `storage_scope` again. The pattern deliberately matches an annotated parameter default (`user_id: str = OTelConstants.DEFAULT_USER_ID`) as well as a plain assignment — the first version of the guard did not, and passed a real injected regression while looking like it worked.++- **Breaking: identity is required, and `"anonymous"` goes back to being a tracing label**: `pipeline_run_setup` coalesced a missing `user_id` to `OTelConstants.DEFAULT_USER_ID` — the string `"anonymous"`, a _tracing_ placeholder for a span with no known caller. It leaked into the identity path and became a storage key prefix, so every run without an authenticated caller wrote into one shared `anonymous/` namespace where each tenant could read the others' outputs. The fallback is what made it silent: a missing identity looked exactly like a present one.++ `user_id` and `storage_scope` are now required parameters of `pipeline_run_setup` and `prepare_pipe_job`, so the same omission is a `TypeError` at the call site instead of a shared prefix in production. The constant survives for telemetry only.++ Two callers pass explicit, greppable values rather than inheriting a default: a dry run — which provably stores nothing — passes `DRY_RUN_STORAGE_SCOPE`, and `PipelexMTHDSProtocol` defaults to `LOCAL_USER_ID` / `LOCAL_STORAGE_SCOPE`, because a run on somebody's own machine genuinely has one user and no tenancy. A constructor default stating a true fact at the boundary where it is true is a different thing from an `or` fallback deep in the call stack, and a multi-tenant host cannot reach these by omission — the seam it calls requires both explicitly.+### Fixed- **The codegen stamp header is now read strictly:** `pipelex codegen check` verified a generated file as pristine even when a line had been injected between the stamp's fence markers, because the content hash covers only the body *below* the fence — so an executable statement could sit inside a header that says `DO NOT EDIT` and the tree would still report as current. Any line inside the header that does not carry the file's comment prefix now makes the stamp unparseable, which the check already reports as `hand-edited`. The `options` value must also be conformant JSON: Python's non-standard `NaN` / `Infinity` / `-Infinity` literals are refused, so a stamp cannot hold something only a Python reader accepts — the header is a cross-language interchange format, and the second implementation of the offline check (in `@pipelex/sdk`) already rejected them. Commented `key: value` lines a reader does not recognise are still ignored, so the header can gain fields without breaking older readers. One consequence to know: a file whose header was tampered with is no longer one `pipelex codegen types` will overwrite — it now refuses a destination it cannot prove it owns rather than clobbering work, so such a file has to be removed before regenerating, which is what the check's advice already tells you to do.- **`pipelex codegen check` now reports drift in one order, whichever half of the report it comes from:** locked artifacts were listed by their full path compared as a string, while stale orphans came out in the order a directory walk found them — a component-by-component order. For a tree holding a `models/` directory beside a `models.py`, the two halves of a single report therefore sorted paths by different rules, and a second implementation could not mirror both with one comparator. Both halves now use the plain full-string sort, and the ordering is written down as a contract: every locked-artifact drift first, then every orphan, each group by path.- **Generated files are written with LF line endings on every platform:** `save_text_to_path` let Python translate each newline to `os.linesep`, so a file emitted on Windows landed as CRLF on disk while its recorded content hash had been taken over the LF text it was built from. The verdict survived, because reads translate back, but the promise that regenerating produces byte-identical output held only within one platform — a team mixing Windows and Linux saw generated trees churn in version control with no change of content. Every artifact this writes (generated code, the codegen lock, JSON outputs, input templates) is now the string verbatim.+- **The scope was validated *after* the first thing that wrote with it.** `prepare_pipe_job` ran the data-url normalizer — which stores bytes at `{storage_scope}/assets/…` — and only then constructed the `JobMetadata` whose validator is supposed to make that key safe. A scope carrying `..` therefore escaped the tenant's namespace, and the guard meant to stop it ran afterwards, on damage already done. The defect was ordering, not absence: this file's own "safe by construction" claim did not hold on the one path that mattered. The scope is now validated at the top of the seam, before anything composes a key from it. Pinned by a test that asserts the normalizer was never called, with real inputs — the empty-inputs harness skips the normalizer on its own, so the same test written that way passes with the guard deleted.++- **Generated content had no leaf: it landed loose at the root of the scope.** Three things write during a run and each built its key separately — the data-url normalizer under `assets/`, storage delivery under `results/`, and generated bytes (an image a pipe produced) directly at `{storage_scope}/<hash>.<ext>`, because `uri_format` rendered the whole key and shipped without one. A file under no leaf is invisible to anything listing a run's content by category, cannot be retained or dropped separately from the rest, and shares the root with every leaf added later. Generated bytes now land under `generated/`.++ **It is its own leaf, not `assets/` or `results/`.** `assets/` is what the caller supplied and this is what the run produced — a provenance split an operator can act on ("drop what we generated, keep what the user gave us"). `results/` is the delivery envelope: written once at the end, under fixed names, only when storage delivery is configured — whereas generated content is written during the run, content-addressed, and usually intermediate, since a generated image is more often an input to a later step than an output of the pipeline.++ **Breaking: `uri_format` renders the FILENAME only**, and `{storage_scope}` is no longer a supported placeholder (`{hash}` and `{extension}` remain; a leading `/` is now refused). The prefix moved into code because a placeholder an operator may omit is an invariant that does not hold — a config dropping it put every run's generated bytes in one namespace. What the format still decides is the filename, including any subdirectory beneath the leaf.++- **Every local run with storage delivery overwrote the previous one.** The run id used to be part of the storage key (`{user_id}/{key_prefix}{pipeline_run_id}`); collapsing the key onto the scope removed it, which is correct for a host — a hosted scope already identifies the run, `<org>/<method>/<run>` — but left the local default naming a tenant and no run. So every local run wrote `local/results/<filename>`, the same key each time, each one silently destroying the last. A local run's scope is now its run id, composed in `pipeline_run_setup` because that is where the run id is minted and the constructor default cannot name a run that does not exist yet. `LOCAL_STORAGE_SCOPE` is a **sentinel** meaning "no host supplied a scope" and never reaches a storage key: a tenancy segment on a laptop separates nothing, since there is exactly one tenant, while the run id is the only part that has to be distinct. Keyed on the sentinel exactly, never a prefix test, so a host-supplied scope is never rewritten.++- **A scope ending in a newline passed validation.** `validate_storage_scope` used `re.match` with a `$` anchor, which admits one trailing newline, so `"org/mt/run\n"` was accepted and the newline travelled into every storage key and log line built from the scope — an unaddressable key and a log-forging primitive. Now `fullmatch`.++- **The bridge payload required the scope but never validated it.** `PipelexPipeRunInput.storage_scope` was a bare `str`, so a payload arriving from another process could carry a traversal that was only refused frames later, wherever a key was first composed. It is validated at construction now, making a malformed payload a decoding error that names the field.+## [v0.46.4] - 2026-08-18### Changed@@ -44,7 +104,7 @@### Fixed-- **Migrator comment fidelity:** Fixed `pipelex migrate` so structural operations no longer leave comment banners behind or attach them to the wrong sections. Moved keys and tables now carry their introducing comment block with them; deleted keys and tables now drop their introducing comment block; items appended under an existing table now land *before* that table's trailing banner; and the `.mthds` fix path now drops comments above stripped native-concept redeclarations. Note: files already migrated by v0.46.0 are not rewritten automatically—tidy the comments by hand or restore the `.bak` files and re-run `pipelex migrate`.+- **Migrator comment fidelity:** Fixed `pipelex migrate` so structural operations no longer leave comment banners behind or attach them to the wrong sections. Moved keys and tables now carry their introducing comment block with them; deleted keys and tables now drop their introducing comment block; items appended under an existing table now land _before_ that table's trailing banner; and the `.mthds` fix path now drops comments above stripped native-concept redeclarations. Note: files already migrated by v0.46.0 are not rewritten automatically—tidy the comments by hand or restore the `.bak` files and re-run `pipelex migrate`.## [v0.46.0] - 2026-08-17@@ -58,7 +118,7 @@- **A migration names what it cannot explain, and refuses a file it cannot reach:** Every plan now carries an `unexplained[]` listing paths of the migrated document that the current schema does not know and no ledger entry accounts for — a typo, or a file written by a newer Pipelex, two readings the tool cannot tell apart. Separately, a file declaring a `[meta] schema_version` below its ledger's floor is refused by name (`unsupported_schema_version`) instead of being run over, left unchanged, and reported clean. No value read from a user's file is ever rendered in either report.- **`[meta] schema_version` is reserved in configuration files:** Every configuration-surface reader now tolerates and strips it. Nothing writes it — the key exists so a later release can stamp a version into a file without that being a breaking change today.- **Value domain narrowing detection:** The migration engine now detects schema changes that narrow allowed values (e.g., `int` to `str`, or tightening a `ge=1` bound to `ge=5`) and forces the author to provide a migration path or an `unsafe` manual warning. Read as a path diff those look like additions, so they used to cost only a golden regeneration and let the next boot reject a valid file with a green gate behind it. An `unsafe` entry must now name the narrowed paths in `declared_narrowed_paths`, and that material is traced through later entries so the same file keeps being reported for as long as it carries it. To see a bound at all the fingerprint records one, over a closed whitelist (`gt`, `ge`, `lt`, `le`, `min_length`, `max_length`, `multiple_of`); `pattern` is excluded permanently, because regex containment is not decidable and every pattern edit would read as a tightening.-- **Replay neutrality is checked as a property, not asserted:** The guarantee a user's machine depends on — replaying a surface's whole ledger over a file already valid at the current schema changes not one byte — is now checked over generated documents rather than only the two reference documents, which carry one value per key and so cannot reach the other legal spelling of an enumerated field. Idempotence and prefix coherence are checked alongside it, each generator carrying its own vacuity check. The coverage gate also *performs* every migration it is asked to believe in, applying each entry to the frozen document of the version before it and holding the result against the version after, which catches a rename whose destination is misspelled — accounted for by coverage, skipped by convergence, and otherwise shipped. `hypothesis` joins the development dependencies.+- **Replay neutrality is checked as a property, not asserted:** The guarantee a user's machine depends on — replaying a surface's whole ledger over a file already valid at the current schema changes not one byte — is now checked over generated documents rather than only the two reference documents, which carry one value per key and so cannot reach the other legal spelling of an enumerated field. Idempotence and prefix coherence are checked alongside it, each generator carrying its own vacuity check. The coverage gate also _performs_ every migration it is asked to believe in, applying each entry to the frozen document of the version before it and holding the result against the version after, which catches a rename whose destination is misspelled — accounted for by coverage, skipped by convergence, and otherwise shipped. `hypothesis` joins the development dependencies.- **Claude skills:** Added the `add-migration` skill to automate ledger entry creation from coverage gate refusals, and updated the `release` skill to enforce ledger completeness before cutting a release.### Changed@@ -69,7 +129,7 @@- **Strict Jinja2 templating filters (Breaking):** The `tag`, `format`, and `with_images` filters no longer fall back to a built-in style when the rendering context lacks one; a missing style now raises `Jinja2ContextError`. Every prompt-rendering entry point now supplies a resolved style, so a missing one is a bug rather than a shape nobody chose. An explicit filter argument (`| format("markdown")`) is unaffected, and an unknown format name is reported as a template error instead of a bare `ValueError`.- **`templating_style` tables reject unknown keys (Breaking):** `TemplatingStyle` and the rich template table of `PipeCompose` are now `extra="forbid"` like every other blueprint. A misspelled optional key — `{ tag_style = "xml", text_formt = "markdown" }`, or `templating_stile` on a template table — used to be dropped silently, yielding a prompt rendered under a shape the author never asked for; it is a validation error now. The MTHDS JSON Schema gains `additionalProperties: false` on both definitions.- **Unknown per-model backend keys must be header-shaped (Breaking):** Any unrecognized key in a per-model table inside `inference/backends/*.toml` is sent to the provider as an HTTP header. To prevent typos from silently becoming headers, these keys must now contain a hyphen (e.g., `x-portkey-provider`); keys without hyphens fail the boot, in strict and lenient boots alike, naming the key, the model and the file. The key must also be usable on the wire — RFC 7230 token characters only, with a printable single-line string value — and a hyphenated spelling of a real field (`max-tokens`) is rejected by name. In the remotely served Pipelex Gateway config the same rule prunes such keys instead of failing, as version skew.-- **Telemetry config is migratable:** `telemetry.toml` is now a migration surface. Upgrading from the legacy flat format to the `[custom_posthog]` format is handled by `pipelex migrate`, preserving user API keys and exporters instead of forcing a destructive `pipelex init telemetry` reset. The loader now raises through the same reporting path as every other configuration surface, so the human CLI, the agent envelope and `pipelex doctor` all say whether the file is *old*, *wrong*, or reported on by a migration that would not repair it, rather than each holding a hardcoded remedy. **Migration:** run `pipelex migrate` — ledger entry `telemetry-config@2` lifts a flat first-generation file into `[custom_posthog]`, keeping every value that has a home in the current shape and saying which settings it drops and why.+- **Telemetry config is migratable:** `telemetry.toml` is now a migration surface. Upgrading from the legacy flat format to the `[custom_posthog]` format is handled by `pipelex migrate`, preserving user API keys and exporters instead of forcing a destructive `pipelex init telemetry` reset. The loader now raises through the same reporting path as every other configuration surface, so the human CLI, the agent envelope and `pipelex doctor` all say whether the file is _old_, _wrong_, or reported on by a migration that would not repair it, rather than each holding a hardcoded remedy. **Migration:** run `pipelex migrate` — ledger entry `telemetry-config@2` lifts a flat first-generation file into `[custom_posthog]`, keeping every value that has a home in the current shape and saying which settings it drops and why.- **The migration ledger contract is published, and a config-model change owes it a review:** `docs/migration-ledger.md` joins the documentation navigation beside the other contributor contracts. The `config-docs` drift contract's review list gains the migration ledgers and the two modules holding the validators that narrow a configuration value domain — a narrowing expressed in a validator is invisible to the fingerprint projection the coverage gate reads, so no derived check can demand the ledger entry it owes, and a review obligation is the only place that can catch it.- **Keyless dry runs:** `pipelex run --dry-run` no longer requires inference credentials, bringing it to parity with `pipelex-agent run --dry-run`. It boots keyless with real model specs, every run forced to DRY. Anyone who relied on `pipelex run --dry-run` as a credentials check should use a live run for that.- **Reasoning budgets:** Thinking budgets are now worker-owned (keyed by family constants like `"anthropic"` or `"gemini"`) rather than read from the removed `prompting_target`. Behavior is unchanged; the budget maps keep their `anthropic` and `gemini` keys.@@ -78,7 +138,7 @@### Fixed- **`pipelex doctor` bracket escaping:** Fixed a crash where `pipelex doctor` failed with a `MarkupError` when a file path or error message contained brackets (e.g., `[dev]`) that Rich interpreted as formatting tags — worst exactly when the report is what the user came for. Every value the report interpolates is now escaped, including configuration paths, backend names, deck filenames and the text of every exception the `--fix` pass reports.-- **`pipelex doctor --fix` no longer offers to overwrite a telemetry config it could migrate:** The fix machinery decided what it could repair by searching the row's *message* for `"format has changed"`, and on a match offered to write a fresh `telemetry.toml`, discarding every setting in it. It now branches on a structured finding, so an out-of-date file is answered with `pipelex migrate` and an invalid one is left to a person. The doctor's telemetry check also strips the reserved `[meta]` table the way boot does, instead of reporting a perfectly bootable file as invalid.+- **`pipelex doctor --fix` no longer offers to overwrite a telemetry config it could migrate:** The fix machinery decided what it could repair by searching the row's _message_ for `"format has changed"`, and on a match offered to write a fresh `telemetry.toml`, discarding every setting in it. It now branches on a structured finding, so an out-of-date file is answered with `pipelex migrate` and an invalid one is left to a person. The doctor's telemetry check also strips the reserved `[meta]` table the way boot does, instead of reporting a perfectly bootable file as invalid.- **Agent CLI double envelopes:** Fixed `pipelex-agent models` and `check-model` printing two JSON error envelopes on a single stream, resulting in invalid JSON output. `agent_error` leaves through `typer.Exit`, which is a `RuntimeError` and not a `SystemExit`, so a command boundary re-raising only `SystemExit` caught the error it had just reported and reported it again.- **Dotted-key renaming:** Fixed a bug in the TOML fix applier where renaming a dotted key (e.g., `a.b = 1`) accidentally converted it into a block header (`[a.b]`), swallowing subsequent scalar values into the wrong table. The applier now also refreshes every chunk of a table written in several places, so renaming `a` in a `[a.b]` … `[a.c]` … `[a.b.d]` layout no longer leaves the later chunks rendering under the old name. Both reach every caller of the applier, the `.mthds` fix loop included.- **Symlink migration:** Migrating a symlinked configuration file now correctly replaces the file at the target of the symlink rather than replacing the symlink itself with a regular file.@@ -86,11 +146,11 @@- **Boot failure reporting:** Fixed a failure to load `pipelex.toml` producing a bare Python traceback instead of the intended user-friendly, field-level error message. `RuntimeBoot` and the doctor both caught `pydantic.ValidationError` around `setup_config`, and the main configuration raises `ConfigValidationError` instead, so the arm never fired for the one configuration everything depends on. Both sites now catch either shape a refusal takes and share one helper.- **Backend errors name the model, the backend and the file:** A refusal cut down to `max_tokens: Input should be a valid integer` sent the reader to grep a directory of files that all have a `max_tokens`; the loader now says which model, which backend and which file before it quotes the pydantic analysis, and the model deck's message keeps its paths the same way. A backend table whose own fields fail the blueprint (`endpoint = 42`) is raised as the library's own validation error rather than escaping as a bare pydantic `ValidationError` that no clause in the boot named. A message carrying a migration block no longer also suggests `pipelex init config`, which would have reset every value the block had just promised to keep.- **Gateway enabled truthiness:** Fixed `enabled = 1` in `backends.toml` being evaluated differently by the loader and the boot process, which silently dropped the gateway. Both readers now read the value the same way.-- **A blocked configuration file says which state it is in:** `FileBlockedReason` gained `unreadable` and `state_uncertain`, so an unreadable file is no longer reported as *unwritable* and a write whose outcome the transaction could not describe is no longer reported as one that never happened.+- **A blocked configuration file says which state it is in:** `FileBlockedReason` gained `unreadable` and `state_uncertain`, so an unreadable file is no longer reported as _unwritable_ and a write whose outcome the transaction could not describe is no longer reported as one that never happened.### Removed-- **`prompting_target` (Breaking):** Removed from `LLMSetting`, `InferenceModelSpec`, `InferenceModelSpecBlueprint`, and all shipped backend TOML files, along with the `PromptingTarget` enum and the per-target style map. Delete `prompting_target` from your local `.pipelex/inference/backends/*.toml` files, or the strict boot loader will reject them — or run `pipelex migrate`, which does it for you. Check `portkey.toml` in particular: the key appears in `[defaults]` *and* on individual model entries, and a surviving per-model one is rejected by name as an unknown, non-header key.+- **`prompting_target` (Breaking):** Removed from `LLMSetting`, `InferenceModelSpec`, `InferenceModelSpecBlueprint`, and all shipped backend TOML files, along with the `PromptingTarget` enum and the per-target style map. Delete `prompting_target` from your local `.pipelex/inference/backends/*.toml` files, or the strict boot loader will reject them — or run `pipelex migrate`, which does it for you. Check `portkey.toml` in particular: the key appears in `[defaults]` _and_ on individual model entries, and a surviving per-model one is rejected by name as an unknown, non-header key.- **Legacy prompt templates:** Removed `LLMPromptTemplate`, `LLMPromptFactoryAbstract`, `LLMPromptTemplateInputs` and `LLMPromptTemplateInputsError`, which relied on hardcoded XML+Markdown styles predating the new configurable templating system and had no production consumer.- **Deprecated config keys:** Removed `[migration.migration_maps]`, `session_id`, and `plugins.boot_orchestrator` from the root configuration. Renames are ledger entries now, so the old-to-new name hints in validation errors go away, `.mthds` hints included; `boot_orchestrator` is strictly a boot argument — pass `--orchestrator` or `Pipelex.setup(boot_orchestrator=...)` and read it back at run time from the `runtime_hub` accessor rather than off the config.diff --git a/docs/under-the-hood/pipelex-kernel.md b/docs/under-the-hood/pipelex-kernel.mdindex 7aed7517c..8dd66eaf4 100644--- a/docs/under-the-hood/pipelex-kernel.md+++ b/docs/under-the-hood/pipelex-kernel.md@@ -25,11 +25,13 @@ from pipelex.runtime_boot import RuntimeBootfrom pipelex.kernel.pipelex_kernel import PipelexKernelRuntimeBoot.make()-kernel = PipelexKernel.make(user_id="my-service")+kernel = PipelexKernel.make(user_id="my-service", storage_scope="my-service")```That boot stands up the model deck, the content generator, the class registry, the reporting delegate and the plugin registries — the machinery inference needs. It does **not** stand up the library manager, the pipe router or the pipeline manager, because a kernel caller has no method to load.+**`user_id` and `storage_scope` are both required, with no default.** `user_id` is who the run is attributed to; `storage_scope` is the opaque prefix every byte the run writes lands under, composed by *you* from whatever your tenancy model is (`"<org>/<run>"`, `"<customer>/<job>"`, or a flat name for a single-tenant service). The kernel composes its own leaves onto it and never parses it, so it never learns what a tenant is: `assets/` for what you supplied, `generated/` for what the run produced, `results/` for the delivered envelope, and `payloads/` for transport spill. It is validated at construction — one to three path-safe segments — because the value becomes a storage key prefix and a `..` in it would escape the namespace. Neither field defaults, deliberately: a missing identity used to fall back to the tracing placeholder `"anonymous"`, which then became the key prefix, pointing every unauthenticated run at one shared namespace.+`needs_inference=False` boots keyless (no credentials, no model-deck validation) and sets the forced-DRY flag: every run the process initiates is coerced to `run_mode=DRY`, so the leaves mock instead of calling a provider. `PipelexKernel.make` applies that rule through the same `runtime_hub.resolve_run_mode_for_boot` the pipe tier uses — a second copy of the rule at a second factory is how the two would drift apart.---@@ -143,7 +145,7 @@ event_log = InMemoryEventLog()trace_context = TraceContext(graph_id=run_id, data_inclusion=data_inclusion, emit_graph_events=False, emit_usage_events=True)get_report_delegate().set_event_log(context_key=trace_context.lookup_key, event_log=event_log, workflow_id="direct", pipeline_run_id=run_id)try:- kernel = PipelexKernel.make(user_id="my-service", trace_context=trace_context)+ kernel = PipelexKernel.make(user_id="my-service", storage_scope="my-service", trace_context=trace_context)...tokens_usages = UsageAggregator.aggregate(event_log.read_events(run_id))finally:diff --git a/docs/under-the-hood/storage-provider-plugins.md b/docs/under-the-hood/storage-provider-plugins.mdindex 4f48d75d2..ca597fcf4 100644--- a/docs/under-the-hood/storage-provider-plugins.md+++ b/docs/under-the-hood/storage-provider-plugins.md@@ -124,7 +124,7 @@ method = "azure" # an out-of-tree provider — no built-in sub-config rWhether that token names an *installed* provider is validated at **registry lookup**, not at parse: an unknown method surfaces as `UnknownStorageMethodError` at boot, which is the right layer — it lists the registered methods so the fix is obvious.!!! note "External-provider config surface is a scoped follow-up"- `StorageProviderConfig` has fixed, typed per-method sub-models, so an out-of-tree `azure` provider has nowhere to read *its* structured config yet. The seam for built-in methods lands first; a generic passthrough sub-config for external providers is a captured follow-up (Decision D3), not built speculatively. Until it lands, an external provider reads its own config from the environment or its own file. **One live gap:** `StorageProviderConfig.uri_format` (read by `GeneratedContentFactory._build_storage_key` on every content store) is only defined for the four built-in methods and raises `StorageConfigError` for any other token — so an external provider selects and boots cleanly but a *generated-content store* through it fails until D3 gives external methods a `uri_format`. External providers are therefore usable today for their own direct storage API, not yet as the backing store for generated content.+ `StorageProviderConfig` has fixed, typed per-method sub-models, so an out-of-tree `azure` provider has nowhere to read *its* structured config yet. The seam for built-in methods lands first; a generic passthrough sub-config for external providers is a captured follow-up (Decision D3), not built speculatively. Until it lands, an external provider reads its own config from the environment or its own file. **One live gap:** `StorageProviderConfig.uri_format` (read by `GeneratedContentFactory._build_storage_key` on every content store, where it renders the FILENAME only — its placeholders are `{hash}` and `{extension}`, while the prefix `{storage_scope}/generated/` is composed in code so a config cannot omit it) is only defined for the four built-in methods and raises `StorageConfigError` for any other token — so an external provider selects and boots cleanly but a *generated-content store* through it fails until D3 gives external methods a `uri_format`. External providers are therefore usable today for their own direct storage API, not yet as the backing store for generated content.---diff --git a/pipelex/cogt/content_generation/extract_generate.py b/pipelex/cogt/content_generation/extract_generate.pyindex a6b974ab0..53dd0d100 100644--- a/pipelex/cogt/content_generation/extract_generate.py+++ b/pipelex/cogt/content_generation/extract_generate.py@@ -33,7 +33,6 @@ async def extract_gen_pages_and_store(return dry_extract_page_contents(extract_assignment)extract_output = await extract_gen_pages(extract_assignment)return await generated_content_factory.make_page_contents(- primary_id=extract_assignment.job_metadata.user_id,- secondary_id=extract_assignment.job_metadata.pipeline_run_id,+ storage_scope=extract_assignment.job_metadata.storage_scope,extract_output=extract_output,)diff --git a/pipelex/cogt/content_generation/generated_content_factory.py b/pipelex/cogt/content_generation/generated_content_factory.pyindex ecd6d8125..f7826757b 100644--- a/pipelex/cogt/content_generation/generated_content_factory.py+++ b/pipelex/cogt/content_generation/generated_content_factory.py@@ -12,6 +12,7 @@ from pipelex.core.stuffs.image_content import ImageContentfrom pipelex.core.stuffs.page_content import PageContentfrom pipelex.core.stuffs.text_and_images_content import TextAndImagesContentfrom pipelex.core.stuffs.text_content import TextContent+from pipelex.system.storage_scope import GENERATED_CONTENT_LEAFfrom pipelex.tools.misc.base64_utils import extract_base64_str_from_base64_url_if_possiblefrom pipelex.tools.misc.file_fetch_utils import fetch_file_from_url_httpxfrom pipelex.tools.misc.image_utils import ImageFormat@@ -24,9 +25,8 @@ class GeneratedContentFactory:def _build_storage_key(self,- primary_id: str,+ storage_scope: str,*,- secondary_id: str,data: bytes,mime_type: str | None,image_format: ImageFormat | None,@@ -34,14 +34,20 @@ class GeneratedContentFactory:"""Build a storage key using a SHA-256 hash of the data.Args:- primary_id: The principal ID- secondary_id: The secondary ID+ storage_scope: The run's opaque storage prefix — see+ `pipelex.system.storage_scope`. It replaced a+ `primary_id` / `secondary_id` pair that was always+ `(user_id, pipeline_run_id)` — WHO ran the pipe, then WHICH run.+ Collapsing them into one host-supplied string is what lets a host+ key generated content by tenant instead of by uploader, without+ the runtime learning what a tenant is.data: The binary data to hashmime_type: Optional MIME type to determine file extensionimage_format: Optional output format to determine file extensionReturns:- A storage key in the format "{primary_id}/{secondary_id}/{hash}.{extension}"+ `{storage_scope}/generated/<filename>`, where the filename is+ rendered from the configured `uri_format` (`{hash}`, `{extension}`)."""hash_digest = hashlib.sha256(data).hexdigest()[:16]@@ -59,17 +65,28 @@ class GeneratedContentFactory:extension = "jpg"else:extension = "jpg"+ # The `generated/` leaf is HARDCODED, and `uri_format` renders only the+ # filename under it. It used to render the whole key, `{storage_scope}/…`,+ # which put generated bytes loose at the root of the scope while the+ # other two writers used a named leaf (`assets/` for what came in,+ # `results/` for what was delivered). Loose files belong to no leaf, so+ # nothing that lists a run's content by category could see them, and a+ # future leaf could collide with a file whose hash spelled its name.+ #+ # Config cannot supply the leaf, deliberately: a placeholder an operator+ # may omit is an invariant that does not hold. What `uri_format` still+ # decides is the filename — including any subdirectory under the leaf.uri_format = get_config().runtime.storage.uri_format- return uri_format.format(primary_id=primary_id, secondary_id=secondary_id, hash=hash_digest, extension=extension)+ filename = uri_format.format(hash=hash_digest, extension=extension)+ return f"{storage_scope}/{GENERATED_CONTENT_LEAF}/{filename}"async def _fetch_remote_content(self, url: str) -> bytes:return await fetch_file_from_url_httpx(url=url)async def make_image_content(self,- primary_id: str,+ storage_scope: str,*,- secondary_id: str,raw_details: GeneratedImageRawDetails,) -> ImageContent:image_format: ImageFormat | None = None@@ -106,8 +123,7 @@ class GeneratedContentFactory:is_remote_url = Trueelif actual_bytes:storage_key = self._build_storage_key(- primary_id=primary_id,- secondary_id=secondary_id,+ storage_scope=storage_scope,data=actual_bytes,mime_type=raw_details.mime_type or base64_extracted_mime_type,image_format=image_format,@@ -137,8 +153,7 @@ class GeneratedContentFactory:public_url = urlelse:storage_key = self._build_storage_key(- primary_id=primary_id,- secondary_id=secondary_id,+ storage_scope=storage_scope,data=actual_bytes,mime_type=mime_type,image_format=image_format,@@ -161,9 +176,8 @@ class GeneratedContentFactory:async def make_page_contents(self,- primary_id: str,+ storage_scope: str,*,- secondary_id: str,extract_output: ExtractOutput,) -> list[PageContent]:page_contents: list[PageContent] = []@@ -172,8 +186,7 @@ class GeneratedContentFactory:page_images: list[ImageContent] = []for extracted_image in page.extracted_images:image_content = await self.make_image_content(- primary_id=primary_id,- secondary_id=secondary_id,+ storage_scope=storage_scope,raw_details=extracted_image,)page_images.append(image_content)diff --git a/pipelex/cogt/content_generation/img_gen_generate.py b/pipelex/cogt/content_generation/img_gen_generate.pyindex 052cd848b..1b625311e 100644--- a/pipelex/cogt/content_generation/img_gen_generate.py+++ b/pipelex/cogt/content_generation/img_gen_generate.py@@ -54,8 +54,7 @@ async def img_gen_single_image_and_store(return dry_img_gen_image_contents(img_gen_assignment.model_copy(update={"nb_images": 1}))[0]generated_image = await img_gen_single_image(img_gen_assignment)image_content = await generated_content_factory.make_image_content(- primary_id=img_gen_assignment.job_metadata.user_id,- secondary_id=img_gen_assignment.job_metadata.pipeline_run_id,+ storage_scope=img_gen_assignment.job_metadata.storage_scope,raw_details=generated_image,)image_content.source_prompt = img_gen_assignment.img_gen_prompt.positive_text@@ -78,8 +77,7 @@ async def img_gen_image_list_and_store(image_contents: list[ImageContent] = []for raw_details in generated_image_list:image_content = await generated_content_factory.make_image_content(- primary_id=img_gen_assignment.job_metadata.user_id,- secondary_id=img_gen_assignment.job_metadata.pipeline_run_id,+ storage_scope=img_gen_assignment.job_metadata.storage_scope,raw_details=raw_details,)image_content.source_prompt = img_gen_assignment.img_gen_prompt.positive_textdiff --git a/pipelex/cogt/content_generation/render_generate.py b/pipelex/cogt/content_generation/render_generate.pyindex b041697d6..4c3f48b6d 100644--- a/pipelex/cogt/content_generation/render_generate.py+++ b/pipelex/cogt/content_generation/render_generate.py@@ -43,8 +43,7 @@ async def render_page_views_and_store(# otherwise holds every page bitmap in memory until the whole store loop finishes.page_view_image.close()image_content = await generated_content_factory.make_image_content(- primary_id=render_assignment.job_metadata.user_id,- secondary_id=render_assignment.job_metadata.pipeline_run_id,+ storage_scope=render_assignment.job_metadata.storage_scope,raw_details=raw_details,)image_contents.append(image_content)diff --git a/pipelex/kernel/pipelex_kernel.py b/pipelex/kernel/pipelex_kernel.pyindex 3264150e9..47009beeb 100644--- a/pipelex/kernel/pipelex_kernel.py+++ b/pipelex/kernel/pipelex_kernel.py@@ -85,6 +85,7 @@ class PipelexKernel:*,run_mode: PipeRunMode = PipeRunMode.LIVE,user_id: str,+ storage_scope: str,is_mock_usage: bool = False,trace_context: TraceContext | None = None,step_id_source: Callable[[], str] | None = None,@@ -118,6 +119,7 @@ class PipelexKernel:return cls(job_metadata=JobMetadata(user_id=user_id,+ storage_scope=storage_scope,pipeline_run_id=trace_context.graph_id if trace_context is not None else str(uuid4()),trace_context=trace_context,),diff --git a/pipelex/kit/configs/pipelex.toml b/pipelex/kit/configs/pipelex.tomlindex 3f0a0525d..1a9242f26 100644--- a/pipelex/kit/configs/pipelex.toml+++ b/pipelex/kit/configs/pipelex.toml@@ -46,26 +46,26 @@ method = "local"[runtime.storage.local]# Local storage settings-uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"+uri_format = "{hash}.{extension}"local_storage_path = ".pipelex/storage"[runtime.storage.in_memory]# In-memory storage settings-uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"+uri_format = "{hash}.{extension}"[runtime.storage.s3]# AWS S3 storage settings (requires boto3: `uv pip install "pipelex[s3]"`)-uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"+uri_format = "{hash}.{extension}"bucket_name = ""region = ""-signed_urls_lifespan_seconds = 3600 # Set to "disabled" for public URLs+signed_urls_lifespan_seconds = 3600 # Set to "disabled" for public URLs[runtime.storage.gcp]# Google Cloud Storage settings (requires google-cloud-storage: `uv pip install "pipelex[gcp-storage]"`)-uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"+uri_format = "{hash}.{extension}"bucket_name = ""project_id = ""-signed_urls_lifespan_seconds = 3600 # Set to "disabled" for public URLs+signed_urls_lifespan_seconds = 3600 # Set to "disabled" for public URLs##################################################################################################### Secrets Configdiff --git a/pipelex/migration/goldens/pipelex-config/[email protected] b/pipelex/migration/goldens/pipelex-config/[email protected]index ce9fd8469..a82dbc501 100644--- a/pipelex/migration/goldens/pipelex-config/[email protected]+++ b/pipelex/migration/goldens/pipelex-config/[email protected]@@ -10,20 +10,20 @@ is_upload_local_content_enabled = truemethod = "local" # local, in_memory, s3, gcp[runtime.storage.local]-uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"+uri_format = "{hash}.{extension}"local_storage_path = ".pipelex/storage"[runtime.storage.in_memory]-uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"+uri_format = "{hash}.{extension}"[runtime.storage.s3]-uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"+uri_format = "{hash}.{extension}"bucket_name = ""region = ""signed_urls_lifespan_seconds = 3600[runtime.storage.gcp]-uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"+uri_format = "{hash}.{extension}"bucket_name = ""project_id = ""signed_urls_lifespan_seconds = 3600diff --git a/pipelex/migration/goldens/pipelex-config/[email protected] b/pipelex/migration/goldens/pipelex-config/[email protected]index dd4485384..cde871cdf 100644--- a/pipelex/migration/goldens/pipelex-config/[email protected]+++ b/pipelex/migration/goldens/pipelex-config/[email protected]@@ -1313,7 +1313,7 @@"runtime.storage.gcp.uri_format": {"value_type": "str","required": true,- "default": "{primary_id}/{secondary_id}/{hash}.{extension}",+ "default": "{hash}.{extension}","open_node": false},"runtime.storage.in_memory": {@@ -1324,7 +1324,7 @@"runtime.storage.in_memory.uri_format": {"value_type": "str","required": true,- "default": "{primary_id}/{secondary_id}/{hash}.{extension}",+ "default": "{hash}.{extension}","open_node": false},"runtime.storage.is_fetch_remote_content_enabled": {@@ -1353,7 +1353,7 @@"runtime.storage.local.uri_format": {"value_type": "str","required": true,- "default": "{primary_id}/{secondary_id}/{hash}.{extension}",+ "default": "{hash}.{extension}","open_node": false},"runtime.storage.method": {@@ -1391,7 +1391,7 @@"runtime.storage.s3.uri_format": {"value_type": "str","required": true,- "default": "{primary_id}/{secondary_id}/{hash}.{extension}",+ "default": "{hash}.{extension}","open_node": false},"runtime.tracing": {diff --git a/pipelex/pipe_run/delivery_executor.py b/pipelex/pipe_run/delivery_executor.pyindex 1a6b9d806..9d86edc53 100644--- a/pipelex/pipe_run/delivery_executor.py+++ b/pipelex/pipe_run/delivery_executor.py@@ -44,7 +44,7 @@ class DeliveryExecutor:self,pipe_output: PipeOutput | None,*,- user_id: str,+ storage_scope: str,pipeline_run_id: str,delivery_assignment: DeliveryAssignment,status: DeliveryStatus,@@ -61,7 +61,11 @@ class DeliveryExecutor:result_url: str | None = Noneif delivery_assignment.storage is not None and pipe_output is not None:result_url = await self._store_results(- pipe_output, user_id=user_id, pipeline_run_id=pipeline_run_id, storage=delivery_assignment.storage, request_id=request_id+ pipe_output,+ storage_scope=storage_scope,+ pipeline_run_id=pipeline_run_id,+ storage=delivery_assignment.storage,+ request_id=request_id,)# Step 2: Notify webhooks with status + result_url (always, even on failure)@@ -315,16 +319,25 @@ class DeliveryExecutor:self,pipe_output: PipeOutput,*,- user_id: str,+ storage_scope: str,pipeline_run_id: str,storage: StorageTarget,request_id: str | None = None,) -> str:- """Generate all result files and store them. Returns the base result URL."""+ """Generate all result files and store them. Returns the base result URL.++ The key is `{storage_scope}/{key_prefix}results/`. It was+ `{user_id}/{key_prefix}{pipeline_run_id}` — keyed by WHO ran the pipe,+ which is what made a run's outputs unreadable by anyone else on the same+ team. `storage_scope` already identifies the run (the host composes it+ from whatever its tenancy model is), so `pipeline_run_id` no longer+ appears in the key; it stays a parameter because the log line below+ correlates on it.+ """try:storage_provider = get_storage_provider()prefix: str = storage.key_prefix or ""- base_key: str = f"{user_id}/{prefix}{pipeline_run_id}"+ base_key: str = f"{storage_scope}/{prefix}results"result_files = await self.generate_result_files(pipe_output)diff --git a/pipelex/pipe_run/dry_run_in_process.py b/pipelex/pipe_run/dry_run_in_process.pyindex 3783cb003..088856f8d 100644--- a/pipelex/pipe_run/dry_run_in_process.py+++ b/pipelex/pipe_run/dry_run_in_process.py@@ -27,7 +27,7 @@ from pipelex.pipeline.execution_seams import prepare_pipe_jobfrom pipelex.pipeline.pipeline_factory import PipelineFactoryfrom pipelex.runtime_hub import scoped_content_generator, scoped_event_logfrom pipelex.system.pipe_run_mode import PipeRunMode-from pipelex.system.telemetry.otel_constants import OTelConstants+from pipelex.system.storage_scope import DRY_RUN_STORAGE_SCOPE, DRY_RUN_USER_IDfrom pipelex.tracing.in_memory_event_log import InMemoryEventLog@@ -142,7 +142,15 @@ async def dry_run_pipe_in_process(pipe: PipeAbstract, *, library_id: str) -> Graexecution_config=execution_config,pipe_run_mode=PipeRunMode.DRY,pipeline_run_id=pipeline_run_id,- user_id=OTelConstants.DEFAULT_USER_ID,+ user_id=DRY_RUN_USER_ID,+ # A dry run provably stores nothing, but `storage_scope` is+ # required — so it says so, loudly and greppably, instead of+ # inheriting a default. A silent default on this field is+ # exactly how the shared `anonymous/` namespace grew: a+ # placeholder never meant to reach storage became the key+ # prefix for every run. If `dry-run-no-storage` ever appears as+ # an S3 prefix, a dry run stored something and that is the bug.+ storage_scope=DRY_RUN_STORAGE_SCOPE,trace_context=trace_context,)# PipeRun.run's finally closes the tracer by pipeline_run_id and assembles thediff --git a/pipelex/pipe_run/pipe_run.py b/pipelex/pipe_run/pipe_run.pyindex c1b0de44a..505321c75 100644--- a/pipelex/pipe_run/pipe_run.py+++ b/pipelex/pipe_run/pipe_run.py@@ -89,7 +89,7 @@ class PipeRun(PipeRunProtocol):try:await self._delivery_executor.execute(pipe_output=pipe_output,- user_id=pipe_job.job_metadata.user_id,+ storage_scope=pipe_job.job_metadata.storage_scope,pipeline_run_id=pipeline_run_id,delivery_assignment=delivery_assignment,status=status,diff --git a/pipelex/pipelex.toml b/pipelex/pipelex.tomlindex ce9fd8469..a82dbc501 100644--- a/pipelex/pipelex.toml+++ b/pipelex/pipelex.toml@@ -10,20 +10,20 @@ is_upload_local_content_enabled = truemethod = "local" # local, in_memory, s3, gcp[runtime.storage.local]-uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"+uri_format = "{hash}.{extension}"local_storage_path = ".pipelex/storage"[runtime.storage.in_memory]-uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"+uri_format = "{hash}.{extension}"[runtime.storage.s3]-uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"+uri_format = "{hash}.{extension}"bucket_name = ""region = ""signed_urls_lifespan_seconds = 3600[runtime.storage.gcp]-uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"+uri_format = "{hash}.{extension}"bucket_name = ""project_id = ""signed_urls_lifespan_seconds = 3600diff --git a/pipelex/pipeline/bundle_validator.py b/pipelex/pipeline/bundle_validator.pyindex d2fe6d78b..b11d3e859 100644--- a/pipelex/pipeline/bundle_validator.py+++ b/pipelex/pipeline/bundle_validator.py@@ -57,8 +57,8 @@ from pipelex.pipeline.pipeline_factory import PipelineFactoryfrom pipelex.runtime_hub import get_telemetry_manager, scoped_content_generatorfrom pipelex.system.configuration.configs import PipelineExecutionConfigfrom pipelex.system.pipe_run_mode import PipeRunMode+from pipelex.system.storage_scope import DRY_RUN_STORAGE_SCOPE, DRY_RUN_USER_IDfrom pipelex.system.telemetry.events import EventName, EventProperty-from pipelex.system.telemetry.otel_constants import OTelConstantsfrom pipelex.tools.typing.pydantic_utils import format_pydantic_validation_errorif TYPE_CHECKING:@@ -279,7 +279,15 @@ class BundleValidator:execution_config=execution_config,pipe_run_mode=PipeRunMode.DRY,pipeline_run_id=dry_run_pipeline_id,- user_id=OTelConstants.DEFAULT_USER_ID,+ user_id=DRY_RUN_USER_ID,+ # A dry run provably stores nothing, but `storage_scope` is+ # required — so it says so, loudly and greppably, instead of+ # inheriting a default. A silent default on this field is+ # exactly how the shared `anonymous/` namespace grew: a+ # placeholder never meant to reach storage became the key+ # prefix for every run. If `dry-run-no-storage` ever appears as+ # an S3 prefix, a dry run stored something and that is the bug.+ storage_scope=DRY_RUN_STORAGE_SCOPE,)await self._pipe_run.run(pipe_job)except (PipelexError, ValidationError, FactoryException) as exc:diff --git a/pipelex/pipeline/execution_seams.py b/pipelex/pipeline/execution_seams.pyindex 6151b2edf..b98915fc1 100644--- a/pipelex/pipeline/execution_seams.py+++ b/pipelex/pipeline/execution_seams.py@@ -48,6 +48,7 @@ from pipelex.pipeline.input_normalizer import normalize_data_urls_to_storagefrom pipelex.system.configuration.configs import PipelineExecutionConfigfrom pipelex.system.job_metadata import JobMetadata, OtelContextfrom pipelex.system.pipe_run_mode import PipeRunMode+from pipelex.system.storage_scope import validate_storage_scopefrom pipelex.tools.misc.file_utils import reject_bare_str_or_pathif TYPE_CHECKING:@@ -162,6 +163,7 @@ async def prepare_pipe_job(pipe_run_mode: PipeRunMode,pipeline_run_id: str,user_id: str,+ storage_scope: str,inputs: PipelineInputs | WorkingMemory | None = None,search_scope: str | None = None,trace_context: "TraceContext | None" = None,@@ -191,6 +193,18 @@ async def prepare_pipe_job(``PipeRunParamsFactory.make_run_params`` (the single writer of ``run_mode``), so it coversevery entry point that builds run params, not just this one."""+ # Validate the scope HERE, before anything composes a storage key from it.+ #+ # `JobMetadata` validates it too, and that is where the invariant is+ # documented — but it is constructed BELOW the data-url normalization, which+ # writes `{storage_scope}/assets/...` to real storage. So an unvalidated+ # scope carrying `..` reached the storage provider and escaped the tenant,+ # and the validator that was supposed to prevent it ran afterwards on a+ # value whose damage was already done. Ordering, not absence, was the defect.+ #+ # This is deliberately not a "second gate": it is the FIRST one on this path.+ storage_scope = validate_storage_scope(value=storage_scope)+working_memory: WorkingMemory | None = None# First, process user-provided inputs. Empty PipelineInputs is falsy and behaves like no inputs.@@ -261,10 +275,11 @@ async def prepare_pipe_job(# Normalize data URLs to pipelex-storage:// URIs if configured.if working_memory and execution_config.is_normalize_data_urls_to_storage and not execution_config.is_mock_inputs:- working_memory = await normalize_data_urls_to_storage(working_memory)+ working_memory = await normalize_data_urls_to_storage(working_memory, storage_scope=storage_scope)job_metadata = JobMetadata(user_id=user_id,+ storage_scope=storage_scope,pipeline_run_id=pipeline_run_id,otel_context=otel_context,trace_context=trace_context,diff --git a/pipelex/pipeline/input_normalizer.py b/pipelex/pipeline/input_normalizer.pyindex 6383f95bd..fe16fc61c 100644--- a/pipelex/pipeline/input_normalizer.py+++ b/pipelex/pipeline/input_normalizer.py@@ -29,7 +29,7 @@ from pipelex.tools.uri.uri_resolver import resolve_uriNormalizableContent = ImageContent | DocumentContent-async def normalize_data_urls_to_storage(working_memory: WorkingMemory) -> WorkingMemory:+async def normalize_data_urls_to_storage(working_memory: WorkingMemory, *, storage_scope: str) -> WorkingMemory:"""Convert all data URLs in ImageContent and DocumentContent to pipelex-storage:// URIs.Scans all stuffs in working memory and for any ImageContent or DocumentContent with@@ -43,6 +43,10 @@ async def normalize_data_urls_to_storage(working_memory: WorkingMemory) -> WorkiArgs:working_memory: The working memory to normalize.+ storage_scope: The run's opaque storage prefix. Normalized bytes land+ under `{storage_scope}/assets/`, inside the run's own namespace —+ they used to go to a flat top-level `normalized/` prefix shared by+ every run of every tenant.Returns:The same WorkingMemory instance with normalized URLs.@@ -51,7 +55,7 @@ async def normalize_data_urls_to_storage(working_memory: WorkingMemory) -> Workifor stuff in working_memory.root.values():content = stuff.content- normalized_content, changed = await _normalize_value(value=content, storage=storage)+ normalized_content, changed = await _normalize_value(value=content, storage=storage, storage_scope=storage_scope)if changed:stuff.content = normalized_content@@ -62,32 +66,35 @@ async def _normalize_value(value: Any,*,storage: StorageProviderAbstract,+ storage_scope: str,) -> tuple[Any, bool]:"""Recursively normalize a value, converting data URLs in ImageContent/DocumentContent to storage URIs.Args:value: The value to normalize (can be ImageContent, DocumentContent, StructuredContent, list, or any other type).storage: The storage provider to use.+ storage_scope: The run's opaque storage prefix; normalized bytes land+ under `{storage_scope}/assets/`.Returns:A tuple of (normalized_value, has_changed)."""# Handle ImageContent and DocumentContentif isinstance(value, (ImageContent, DocumentContent)):- normalized = await _normalize_url_content(content=value, storage=storage)+ normalized = await _normalize_url_content(content=value, storage=storage, storage_scope=storage_scope)return normalized, normalized is not value# Handle StructuredContent (recursively process all fields)if isinstance(value, StructuredContent):- return await _normalize_structured_content(structured_content=value, storage=storage)+ return await _normalize_structured_content(structured_content=value, storage=storage, storage_scope=storage_scope)# Handle ListContentif isinstance(value, ListContent):- return await _normalize_list_content(list_content=value, storage=storage) # pyright: ignore[reportUnknownArgumentType]+ return await _normalize_list_content(list_content=value, storage=storage, storage_scope=storage_scope) # pyright: ignore[reportUnknownArgumentType]# Handle plain lists (might contain ImageContent, DocumentContent, or StructuredContent)if isinstance(value, list):- return await _normalize_list(items=value, storage=storage) # pyright: ignore[reportUnknownArgumentType]+ return await _normalize_list(items=value, storage=storage, storage_scope=storage_scope) # pyright: ignore[reportUnknownArgumentType]# Other types don't need normalizationreturn value, False@@ -97,12 +104,15 @@ async def _normalize_structured_content(structured_content: StructuredContent,*,storage: StorageProviderAbstract,+ storage_scope: str,) -> tuple[StructuredContent, bool]:"""Normalize a StructuredContent by recursively processing all its fields.Args:structured_content: The structured content to normalize.storage: The storage provider to use.+ storage_scope: The run's opaque storage prefix; normalized bytes land+ under `{storage_scope}/assets/`.Returns:A tuple of (normalized_content, has_changed).@@ -111,7 +121,7 @@ async def _normalize_structured_content(has_changes = Falsefor field_name, field_value in structured_content:- normalized_value, changed = await _normalize_value(value=field_value, storage=storage)+ normalized_value, changed = await _normalize_value(value=field_value, storage=storage, storage_scope=storage_scope)if changed:updates[field_name] = normalized_valuehas_changes = True@@ -128,12 +138,15 @@ async def _normalize_list_content(list_content: ListContent[Any],*,storage: StorageProviderAbstract,+ storage_scope: str,) -> tuple[ListContent[Any], bool]:"""Normalize a ListContent by processing all its items.Args:list_content: The list content to normalize.storage: The storage provider to use.+ storage_scope: The run's opaque storage prefix; normalized bytes land+ under `{storage_scope}/assets/`.Returns:A tuple of (normalized_list_content, has_changed).@@ -142,7 +155,7 @@ async def _normalize_list_content(if not raw_items:return list_content, False- normalized_items, has_changes = await _normalize_list(items=raw_items, storage=storage) # pyright: ignore[reportUnknownArgumentType]+ normalized_items, has_changes = await _normalize_list(items=raw_items, storage=storage, storage_scope=storage_scope) # pyright: ignore[reportUnknownArgumentType]if not has_changes:return list_content, False@@ -162,12 +175,15 @@ async def _normalize_list(items: list[Any],*,storage: StorageProviderAbstract,+ storage_scope: str,) -> tuple[list[Any], bool]:"""Normalize a list by processing all its items.Args:items: The list items to normalize.storage: The storage provider to use.+ storage_scope: The run's opaque storage prefix; normalized bytes land+ under `{storage_scope}/assets/`.Returns:A tuple of (normalized_items, has_changed).@@ -176,7 +192,7 @@ async def _normalize_list(has_changes = Falsefor item in items:- normalized_item, changed = await _normalize_value(value=item, storage=storage)+ normalized_item, changed = await _normalize_value(value=item, storage=storage, storage_scope=storage_scope)normalized_items.append(normalized_item)if changed:has_changes = True@@ -188,12 +204,15 @@ async def _normalize_url_content(content: NormalizableContent,*,storage: StorageProviderAbstract,+ storage_scope: str,) -> NormalizableContent:"""Normalize ImageContent or DocumentContent by converting data URLs to storage URIs.Args:content: The image or document content to normalize.storage: The storage provider to use.+ storage_scope: The run's opaque storage prefix; normalized bytes land+ under `{storage_scope}/assets/`.Returns:The original content if no normalization needed, or a new instance@@ -210,7 +229,12 @@ async def _normalize_url_content(raw_bytes = base64.b64decode(resolved_uri.base64_data)file_type = detect_file_type_from_bytes(raw_bytes)mime_type = resolved_uri.mime_type or content.mime_type- key = f"normalized/{shortuuid.uuid()}.{file_type.extension}"+ # `{scope}/assets/`, NOT the old flat `normalized/`. That top-level+ # prefix was shared by every run of every tenant: the bytes of an input+ # someone pasted as a data: URL landed beside everyone else's, keyed+ # only by a random id. Under the run's own scope they are inside the+ # tenant's namespace and go away with the run.+ key = f"{storage_scope}/assets/{shortuuid.uuid()}.{file_type.extension}"storage_uri = await storage.store(data=raw_bytes, key=key, content_type=mime_type)public_url = await storage.public_url(uri=storage_uri)@@ -236,7 +260,7 @@ async def _normalize_url_content(msg = f"Input file cannot be read: '{resolved_uri.path}' ({type(exc).__name__})"raise PipelineInputContentError(msg) from excfile_type = detect_file_type_from_bytes(raw_bytes)- key = f"normalized/{shortuuid.uuid()}.{file_type.extension}"+ key = f"{storage_scope}/assets/{shortuuid.uuid()}.{file_type.extension}"storage_uri = await storage.store(data=raw_bytes, key=key, content_type=file_type.mime)public_url = await storage.public_url(uri=storage_uri)diff --git a/pipelex/pipeline/pipeline_run_setup.py b/pipelex/pipeline/pipeline_run_setup.pyindex 57a9081c5..4b03ab15e 100644--- a/pipelex/pipeline/pipeline_run_setup.py+++ b/pipelex/pipeline/pipeline_run_setup.py@@ -29,6 +29,7 @@ from pipelex.system.configuration.configs import PipelineExecutionConfigfrom pipelex.system.environment import get_optional_envfrom pipelex.system.job_metadata import OtelContextfrom pipelex.system.pipe_run_mode import PipeRunMode+from pipelex.system.storage_scope import LOCAL_STORAGE_SCOPEfrom pipelex.system.telemetry.events import EventName, EventPropertyfrom pipelex.system.telemetry.otel_constants import OTelConstantsfrom pipelex.system.telemetry.otel_factory import OtelFactory@@ -54,7 +55,8 @@ async def pipeline_run_setup(dynamic_output_concept_ref: str | None = None,pipe_run_mode: PipeRunMode | None = None,is_mock_usage: bool = False,- user_id: str | None = None,+ user_id: str,+ storage_scope: str,pipeline_run_id: str | None = None,request_id: str | None = None,inputs_base_dir: Path | None = None,@@ -115,7 +117,12 @@ async def pipeline_run_setup(:attr:`CogtRunParams.is_mock_usage`, which rides every assignment to the leaf in bothdirect and Temporal modes. Requires a DRY run; setting it on a LIVE run fails validation.user_id:- Unique identifier for the user (optional).+ Unique identifier for the caller. REQUIRED — see the note below on why+ it no longer defaults.+ storage_scope:+ Opaque prefix under which every byte this run writes must land. REQUIRED,+ validated at ``JobMetadata`` construction. See+ :mod:`pipelex.system.storage_scope`.pipeline_run_id:Pre-generated pipeline run ID. If provided, this ID is used instead ofgenerating a new one. Use this when the run record has already been created@@ -139,7 +146,19 @@ async def pipeline_run_setup(and the library ID."""- user_id = user_id or OTelConstants.DEFAULT_USER_ID+ # NO `user_id or DEFAULT_USER_ID` HERE, DELIBERATELY.+ #+ # `OTelConstants.DEFAULT_USER_ID` is the string "anonymous", and it is a+ # TRACING placeholder — a label for a span with no known caller. It leaked+ # into the identity path and became a storage key prefix, so every run+ # without an authenticated caller wrote into one shared `anonymous/`+ # namespace where each tenant could read the others' outputs. The failure+ # was silent by construction: the fallback made a missing identity look+ # like a present one.+ #+ # Both fields are now required parameters, so the same mistake is a+ # TypeError at the call site instead of a shared bucket prefix in+ # production. The constant survives for telemetry only.if not mthds_contents and not pipe_code:msg = "Either pipe_code or mthds_contents must be provided to the pipeline API."raise ValueError(msg)@@ -154,6 +173,24 @@ async def pipeline_run_setup(pipeline = get_pipeline_manager().add_new_pipeline(pipe_code=pipe_code, pipeline_run_id=pipeline_run_id)pipeline_run_id = pipeline.pipeline_run_id+ # A local run's scope IS its run id, and this is the only place it can be+ # decided: the runner's constructor default cannot name a run that does not+ # exist yet, and the id is minted on the line above.+ #+ # `LOCAL_STORAGE_SCOPE` is therefore a SENTINEL, not a prefix — it means "no+ # host supplied a scope", and it never reaches a storage key. A tenancy+ # segment would be ceremony on a laptop: there is exactly one tenant, so the+ # only thing the key has to separate is one run from the next.+ #+ # Without this, every local run wrote under the same scope, so anything+ # stored under a fixed name — `results/main_stuff.json`, `results/working_memory.json`+ # — was overwritten by the following run, with nothing failing to say so.+ #+ # Keyed on the sentinel EXACTLY, never a prefix test: a host that serves more+ # than one tenant passes its own scope and is untouched here.+ if storage_scope == LOCAL_STORAGE_SCOPE:+ storage_scope = pipeline_run_id+if not library_id:library_id = pipeline_run_id@@ -283,6 +320,7 @@ async def pipeline_run_setup(is_mock_usage=is_mock_usage,pipeline_run_id=pipeline_run_id,user_id=user_id,+ storage_scope=storage_scope,inputs=inputs,search_scope=search_scope,trace_context=trace_context,diff --git a/pipelex/pipeline/runner.py b/pipelex/pipeline/runner.pyindex 432395a57..906dc81d7 100644--- a/pipelex/pipeline/runner.py+++ b/pipelex/pipeline/runner.py@@ -32,6 +32,7 @@ from pipelex.pipeline.pipeline_response import PipelexRunResultExecute, PipelexRfrom pipelex.pipeline.pipeline_run_setup import pipeline_run_setupfrom pipelex.pipeline.validate_in_process import validate_bundles_in_processfrom pipelex.runtime_hub import get_report_delegate, get_telemetry_manager+from pipelex.system.storage_scope import LOCAL_STORAGE_SCOPE, LOCAL_USER_IDfrom pipelex.system.telemetry.events import EventName, EventProperty, Outcomefrom pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error@@ -112,7 +113,12 @@ class PipelexMTHDSProtocol(MTHDSProtocol["PipeOutput"]):bundle_uris: list[str] | None = None,pipe_run_mode: PipeRunMode | None = None,is_mock_usage: bool = False,- user_id: str | None = None,+ # A local run has exactly one user and no tenancy, so these say "local"+ # rather than being derived from each other or defaulted deep in the+ # call stack. A multi-tenant host passes its own; it cannot reach these+ # by omission, because `pipeline_run_setup` requires both explicitly.+ user_id: str = LOCAL_USER_ID,+ storage_scope: str = LOCAL_STORAGE_SCOPE,execution_config: PipelineExecutionConfig | None = None,pipe_run: PipeRunProtocol | None = None,inputs_base_dir: Path | None = None,@@ -123,6 +129,7 @@ class PipelexMTHDSProtocol(MTHDSProtocol["PipeOutput"]):self.pipe_run_mode = pipe_run_modeself.is_mock_usage = is_mock_usageself.user_id = user_id+ self.storage_scope = storage_scopeself.execution_config = execution_configself._pipe_run = pipe_run# Directory that bare relative local file paths in `inputs` resolve against (Smart Inputs@@ -222,6 +229,7 @@ class PipelexMTHDSProtocol(MTHDSProtocol["PipeOutput"]):pipe_run_mode=self.pipe_run_mode,is_mock_usage=self.is_mock_usage,user_id=self.user_id,+ storage_scope=self.storage_scope,inputs_base_dir=self.inputs_base_dir,)effective_pipe_run = self._pipe_run or get_pipe_run()diff --git a/pipelex/runtime_bridge/payloads.py b/pipelex/runtime_bridge/payloads.pyindex dc9055f47..d6d00eca2 100644--- a/pipelex/runtime_bridge/payloads.py+++ b/pipelex/runtime_bridge/payloads.py@@ -10,11 +10,15 @@ import cycle: pydantic, the orchestration/delivery types, stdlib typing only.from typing import Any-from pydantic import BaseModel, ConfigDict, Field+from pydantic import BaseModel, ConfigDict, Field, field_validatorfrom pipelex.runtime_bridge.delivery_mode import DeliveryModefrom pipelex.runtime_bridge.orchestration_mode import DIRECT_ORCHESTRATION_MODE+# Import-light by design (see module docstring): `storage_scope` pulls in `re`+# and nothing else, so validating here costs the boundary nothing.+from pipelex.system.storage_scope import validate_storage_scope+class PipelexPipeRunInput(BaseModel):"""JSON-safe input crossing the host-runtime boundary."""@@ -25,7 +29,27 @@ class PipelexPipeRunInput(BaseModel):inputs: dict[str, Any] = Field(default_factory=dict)output_name: str | None = Nonepipeline_run_id: str | None = None- user_id: str | None = None+ # Both REQUIRED, and neither nullable — this is the wire boundary where a+ # missing identity used to become the string "anonymous" and a missing+ # scope used to be derived from it. A host that cannot say who is calling+ # and where the bytes go must fail here, at the edge, rather than have the+ # runtime invent an answer that silently shares one namespace across+ # tenants. See `pipelex.system.storage_scope`.+ user_id: str+ storage_scope: str++ @field_validator("storage_scope")+ @classmethod+ def _validate_storage_scope(cls, value: str) -> str:+ """Refuse an unusable scope at the WIRE, not three frames into the run.++ A bridge payload crosses a process boundary, so its scope arrives as+ whatever the other side put there. Validating it at construction means a+ traversal is a payload-decoding error naming the field, rather than a+ failure inside whichever call first pastes the value into a storage key.+ """+ return validate_storage_scope(value=value)+library_crate_dump: dict[str, Any] | None = None# Two orthogonal axes: which orchestrator runs the pipe (open token, defaults to the# core in-process orchestrator) and whether the caller waits (endpoint-set; defaultsdiff --git a/pipelex/system/job_metadata.py b/pipelex/system/job_metadata.pyindex 4a444302c..bbe174fa5 100644--- a/pipelex/system/job_metadata.py+++ b/pipelex/system/job_metadata.py@@ -2,8 +2,9 @@ from datetime import datetimefrom enum import StrEnumfrom typing import Any-from pydantic import BaseModel, Field+from pydantic import BaseModel, Field, field_validator+from pipelex.system.storage_scope import validate_storage_scopefrom pipelex.system.telemetry.otel_context import OtelContextfrom pipelex.system.trace_context import TraceContext@@ -52,6 +53,20 @@ class UnitJobId(StrEnum):class JobMetadata(BaseModel):user_id: strpipeline_run_id: str++ # Where every byte this run writes must land. Opaque to the runtime, which+ # composes `assets/`, `results/` and `payloads/` onto it and never parses it+ # — see `pipelex.system.storage_scope` for why the host's own concepts+ # (tenant, organization, method) deliberately do not cross this boundary.+ #+ # REQUIRED, with no default. It used to be derived from `user_id`, which+ # tied where the bytes live to who uploaded them — fine until two people+ # share the work, at which point the second one cannot read the first one's+ # files. A default here would let that coupling grow back silently, so a+ # caller with nothing to store must say so explicitly with+ # `DRY_RUN_STORAGE_SCOPE`.+ storage_scope: str+pipe_code: str | None = None# Per-process Pipelex session id (``Config.session_id``) captured at the@@ -91,6 +106,21 @@ class JobMetadata(BaseModel):started_at: datetime | None = Field(default_factory=datetime.now)completed_at: datetime | None = None+ @field_validator("storage_scope")+ @classmethod+ def _validate_storage_scope(cls, value: str) -> str:+ """Refuse a scope that would escape its namespace, at construction.++ On the TYPE rather than at the call sites: the value becomes a storage+ key prefix, so a `..` or a leading slash in it is a traversal into+ another tenant's data. Validating here means every key the runtime+ derives is safe by construction. `model_copy` does NOT re-run+ validators, which is fine — `copy_with_update` only ever carries an+ already-validated scope forward, and nothing constructs a scope from+ request data after this point.+ """+ return validate_storage_scope(value=value)+@propertydef duration(self) -> float | None:if self.started_at is not None and self.completed_at is not None:diff --git a/pipelex/system/storage_scope.py b/pipelex/system/storage_scope.pynew file mode 100644index 000000000..2f2232f1e--- /dev/null+++ b/pipelex/system/storage_scope.py@@ -0,0 +1,126 @@+"""`storage_scope` — the one opaque string that decides where a run's bytes go.++**The runtime does not know what an organization or a method is, and must not+learn.** A host that runs Pipelex for many tenants needs every object a run+writes to land inside that tenant's namespace; Pipelex needs to compose leaves+(`assets/`, `results/`, `payloads/`) onto *something*. `storage_scope` is that+something: an opaque, host-supplied prefix that the runtime treats as a unit.++The hosted Pipelex platform fills it with `<org_id>/<method_id>/<run_id>`, but+nothing here knows or checks that — a single-user deployment can pass+`<user_id>/<run_id>` and a local one passes just `<run_id>`. Threading the+host's own concepts through the transport instead was considered and rejected:+`method_id` is a hosted catalog concept with no meaning in an MIT-licensed+runtime, and `uri_format`'s placeholder set is closed by design.++**Why the validation is here rather than at the call sites.** The value reaches+`StorageProviderAbstract.store()` as a key prefix, so a `..` or a leading slash+in it is a path traversal into another tenant's namespace. The Temporal payload+codec used to run a per-segment sanitizer over `user_id` and `pipeline_run_id`+separately; collapsing them into one slash-bearing string makes that sanitizer+unusable, and without an explicit validator here that change would silently+delete an existing traversal control. So the constraint moved to the type: a+`JobMetadata` cannot be constructed with a scope that is not path-safe, which+makes every downstream key safe by construction rather than by remembering.+"""++from __future__ import annotations++import re++# One to three `[A-Za-z0-9_-]` segments joined by single slashes.+#+# Applied with `fullmatch` (see below), it rejects in one rule: the empty string,+# a leading or trailing slash, an empty+# interior segment (`a//b`), `.` and `..` in any position, and any character+# that could re-open a traversal or a query string once the value is pasted into+# a URI. The upper bound of three segments is not cosmetic — it is what keeps a+# scope from swallowing the leaf (`assets/`, `results/`, `payloads/`) that the+# runtime appends to it.+STORAGE_SCOPE_PATTERN = re.compile(r"^[A-Za-z0-9_-]+(?:/[A-Za-z0-9_-]+){0,2}$")++# The scope for a run that provably never stores anything: a dry run.+#+# Dry runs construct a `JobMetadata` without a real run behind them, so they+# need a value for a required field. An explicit, greppable constant is used+# rather than an empty string or a `None` default, because a silent default on+# THIS field is exactly how the `anonymous/` namespace grew the first time — a+# placeholder that was never meant to reach storage became the key prefix for+# every unauthenticated run. If this value ever shows up as an S3 key prefix,+# something stored during a dry run and that is the bug to chase.+DRY_RUN_STORAGE_SCOPE = "dry-run-no-storage"++# The caller a dry run is attributed to.+#+# It used to be `OTelConstants.DEFAULT_USER_ID` — the string "anonymous", which+# is a TELEMETRY placeholder meaning "a span with no known caller". Reusing it+# as an *identity* is how that string reached the storage path in the first+# place, and the whole point of this module is that it must not. A dry run has+# no caller in the identity sense either, so it says so in its own word instead+# of borrowing telemetry's.+DRY_RUN_USER_ID = "dry-run-no-user"++# The identity and scope of a run on somebody's own machine.+#+# These are DEFAULTS ON A CONSTRUCTOR, which is a different thing from the+# `or DEFAULT_USER_ID` fallback they replace, and the difference is the whole+# point. A local run genuinely has one user and no tenancy, so "local" is a true+# statement made once, at the boundary where it is true. The old fallback sat+# deep in `pipeline_run_setup`, where it turned a *missing* identity on a+# multi-tenant server into a present-looking one and pointed every such run at+# one shared namespace.+#+# So: a host that serves more than one tenant must pass its own values. It+# cannot reach these by omission — the seam it calls, `pipeline_run_setup`,+# requires both explicitly.+#+# `LOCAL_STORAGE_SCOPE` is a SENTINEL and never reaches a storage key.+# `pipeline_run_setup` swaps it for the run id the moment that id exists, so a+# local run is scoped `<run_id>`, not `local/<run_id>`. A tenancy segment on a+# laptop separates nothing — there is exactly one tenant — while the run id is+# the only thing that has to be distinct, because anything stored under a fixed+# name (`results/main_stuff.json`) would otherwise be overwritten by the next+# run. `LOCAL_USER_ID` is not a sentinel: it is an identity, it is true, and it+# is reported as-is.+LOCAL_USER_ID = "local"+LOCAL_STORAGE_SCOPE = "local"++# The leaf generated bytes land under, composed onto the scope by+# `GeneratedContentFactory`. It is a constant and not a config value on purpose:+# an operator who can omit the leaf can put generated content back at the root+# of the scope, and an invariant a config may switch off is not one.+#+# It is its own leaf rather than sharing `assets/` because the two differ in+# PROVENANCE — `assets/` is what the caller supplied, this is what the run+# produced — and that distinction is the one an operator needs to act on+# ("drop what we generated, keep what the user gave us"). It is not `results/`:+# that leaf is the delivery envelope, written once at the end under fixed names+# and only when storage delivery is configured, whereas this is written during+# the run, content-addressed, and mostly intermediate — a generated image is+# usually an input to a later step rather than an output of the pipeline.+GENERATED_CONTENT_LEAF = "generated"+++def validate_storage_scope(*, value: str) -> str:+ """Return `value` if it is a usable storage scope, else raise `ValueError`.++ Raises:+ ValueError: the scope is empty, has more than three segments, or+ contains a segment that is not `[A-Za-z0-9_-]+` — which covers+ traversal (`..`), absolute paths, and empty segments.+ """+ # `fullmatch`, NOT `match`. A trailing `$` in a `match` still admits one+ # final newline, so `"org/mt/run\n"` passed this guard and the newline then+ # travelled into every storage key and log line built from the scope — a log+ # forging primitive, and a key nobody can address. `fullmatch` has no such+ # concession, which is why the anchors below are redundant but kept: the+ # pattern is also read on its own by the wire-level validators.+ if not STORAGE_SCOPE_PATTERN.fullmatch(value):+ msg = (+ f"Invalid storage_scope {value!r}: expected one to three path-safe segments "+ "separated by single slashes (e.g. 'tenant/run' or 'tenant/method/run'). "+ "Empty segments, '.', '..' and leading or trailing slashes are refused — "+ "the value becomes a storage key prefix, so a traversal in it escapes the tenant."+ )+ raise ValueError(msg)+ return valuediff --git a/pipelex/tools/storage/storage_config.py b/pipelex/tools/storage/storage_config.pyindex 07f71c1d3..3ba0fee52 100644--- a/pipelex/tools/storage/storage_config.py+++ b/pipelex/tools/storage/storage_config.py@@ -25,7 +25,31 @@ class StorageMethodConfig(ConfigModel):# The exact keyword arguments GeneratedContentFactory._build_storage_key supplies to uri_format.format();# any other placeholder would raise KeyError/IndexError at content-store time, so reject it at config time.- URI_FORMAT_SUPPORTED_FIELDS: ClassVar[frozenset[str]] = frozenset({"primary_id", "secondary_id", "hash", "extension"})+ #+ # `uri_format` renders the FILENAME ONLY. The prefix is not configurable:+ # `GeneratedContentFactory` composes `{storage_scope}/generated/` itself and+ # appends whatever this renders. So `{primary_id}`, `{secondary_id}` and+ # `{storage_scope}` are all gone from the supported set.+ #+ # The first two were always `user_id` then `pipeline_run_id` — "who ran it"+ # then "which run" — and the first half is the bug the storage change exists+ # to remove: keying by the uploader makes a run's output unreadable by+ # anyone else on the same team.+ #+ # `{storage_scope}` was the honest replacement for a while, but leaving the+ # prefix in an operator's hands means the layout is only a default: a config+ # that omits it puts every run's generated bytes in one namespace, and one+ # that omits the leaf puts them loose at the root where nothing listing a+ # run's content by category can find them. An invariant a config may switch+ # off is not an invariant, so the prefix moved into code and this string+ # kept the part that is genuinely a preference — the filename, including any+ # subdirectory the operator wants beneath the leaf.+ #+ # A config naming any retired placeholder is rejected at BOOT rather than+ # KeyError-ing at content-store time — the point of validating the format+ # string here is that the failure lands on the operator who wrote the+ # config, not on the first run that happens to generate an image.+ URI_FORMAT_SUPPORTED_FIELDS: ClassVar[frozenset[str]] = frozenset({"hash", "extension"})provider_label: ClassVar[str]@@ -34,6 +58,11 @@ class StorageMethodConfig(ConfigModel):def _collect_uri_format_error_msgs(self) -> list[str]:if self.uri_format == "":return ["- set a value for uri_format"]+ if self.uri_format.startswith("/"):+ # Appended to `{storage_scope}/generated/`, so a leading slash yields+ # a `//` in the key — a different object from the one the operator+ # meant on every backend that treats the key as an opaque string.+ return ["- uri_format must not start with '/': it renders the filename under the generated/ leaf, not an absolute key"]try:# Parse instead of substring-checking: the escaped literal "{{hash}}" renders as the constant# "{hash}" (same URI for every object), and a format spec like "{hash:.0}" truncates the digest@@ -61,8 +90,8 @@ class StorageMethodConfig(ConfigModel):plain_hash_found = Trueelif field_name != base_name or format_spec or conversion:# Every supported field is a str path component, so a format spec, conversion, or- # attribute/index access renders a constant, colliding, or oversized key ({primary_id:.0}- # truncates to "" → every object shares one URI; {primary_id:1000000000} would allocate a+ # attribute/index access renders a constant, colliding, or oversized key ({storage_scope:.0}+ # truncates to "" → every object shares one URI; {storage_scope:1000000000} would allocate a# ~1GB string at render time) — the same silent-overwrite/OOM hazard we reject on {hash}.nonplain_names.add(field_name)supported_list = ", ".join("{" + name + "}" for name in sorted(self.URI_FORMAT_SUPPORTED_FIELDS))diff --git a/subject_grants.toml b/subject_grants.tomlindex b8f8d5f58..1ae7df851 100644--- a/subject_grants.toml+++ b/subject_grants.toml@@ -1417,20 +1417,20 @@ param = "extract_assignment"rationale = "Extracts from the extract_assignment; single operand."["pipelex/cogt/content_generation/generated_content_factory.py::GeneratedContentFactory._build_storage_key"]-param = "primary_id"-rationale = "Builds the storage key from the primary id; single operand."+param = "storage_scope"+rationale = "Builds the storage key under the run's storage scope; single operand."["pipelex/cogt/content_generation/generated_content_factory.py::GeneratedContentFactory._fetch_remote_content"]param = "url"rationale = "Fetches the remote content at the URL; single operand."["pipelex/cogt/content_generation/generated_content_factory.py::GeneratedContentFactory.make_image_content"]-param = "primary_id"-rationale = "Factory from source: builds the result from the primary_id; single operand."+param = "storage_scope"+rationale = "Factory from source: builds the result under the storage scope; single operand."["pipelex/cogt/content_generation/generated_content_factory.py::GeneratedContentFactory.make_page_contents"]-param = "primary_id"-rationale = "Factory from source: builds the result from the primary_id; single operand."+param = "storage_scope"+rationale = "Factory from source: builds the result under the storage scope; single operand."["pipelex/cogt/content_generation/img_gen_generate.py::img_gen_image_list"]param = "img_gen_assignment"diff --git a/tests/conftest.py b/tests/conftest.pyindex 74e03bb59..a65c77cb3 100644--- a/tests/conftest.py+++ b/tests/conftest.py@@ -199,6 +199,7 @@ def job_metadata(request: pytest.FixtureRequest) -> JobMetadata:pipeline_run_id: str = f"{test_id}-{random_code}"return JobMetadata(+ storage_scope="test/scope",user_id="pytest",pipeline_run_id=pipeline_run_id,)diff --git a/tests/e2e/pipelex/cli/test_migrate_commands.py b/tests/e2e/pipelex/cli/test_migrate_commands.pyindex f12f4caea..ea9bb89bb 100644--- a/tests/e2e/pipelex/cli/test_migrate_commands.py+++ b/tests/e2e/pipelex/cli/test_migrate_commands.py@@ -136,6 +136,13 @@ def _old_shape_telemetry_document() -> str:return path.read_text(encoding="utf-8")+# The storage `uri_format` as schema 1 spelled it, and as the current models require it. The+# placeholder set narrowed without any path moving, so no ledger entry describes it — see the+# note inside the function below.+_STORAGE_URI_FORMAT_AT_SCHEMA_1 = '"{primary_id}/{secondary_id}/{hash}.{extension}"'+_STORAGE_URI_FORMAT_TODAY = '"{hash}.{extension}"'++def _pre_reshape_pipelex_config_document() -> str:"""The whole main configuration in its pre-reshape shape, read from the package.@@ -149,9 +156,28 @@ def _pre_reshape_pipelex_config_document() -> str:higher: the pre-history entry inserted below the reshape changed nothing in the models, so itsreference document is a byte copy of this one. Naming the version where the document's shape wasactually cut is the spelling that says what this is, rather than one inherited from a copy.++ One value is modernized on the way out, and it is not a shape change, which is why the ledger+ is silent about it and right to be. `uri_format`'s PLACEHOLDER SET narrowed —+ `{primary_id}`/`{secondary_id}` gave way to `{storage_scope}` — and a value domain narrowing+ with no path added, removed or moved is a *content* change, out of scope for a structural+ vocabulary (see `docs/migration-ledger.md`). The golden stays a true record of schema 1; what+ this fixture needs is a document that is pre-reshape in SHAPE, since the reshape is what it+ exercises, while still being one the current models will load. Left as-is, the planted machine+ fails `_assert_boots` before `migrate` is ever reached, and the reshape goes untested."""path = defaults_golden_path(migration_dir=packaged_migration_dir(), surface_id=PIPELEX_CONFIG_SURFACE_ID, schema_version=1)- return path.read_text(encoding="utf-8")+ document = path.read_text(encoding="utf-8")+ modernized = document.replace(_STORAGE_URI_FORMAT_AT_SCHEMA_1, _STORAGE_URI_FORMAT_TODAY)+ if modernized == document:+ msg = (+ "the schema-1 storage uri_format is no longer spelled the way this fixture modernizes:\n"+ f"{_STORAGE_URI_FORMAT_AT_SCHEMA_1}\n"+ "If the placeholder set changed again, update the two constants above — a silent no-op here "+ "plants a document that cannot boot, and the reshape stops being tested."+ )+ raise AssertionError(msg)+ return modernizeddef _todays_pipelex_config_document() -> dict[str, Any]:diff --git a/tests/integration/pipelex/cogt/content_generation/test_leaf_dry_object_mocks.py b/tests/integration/pipelex/cogt/content_generation/test_leaf_dry_object_mocks.pyindex 9997d788a..00cb01e79 100644--- a/tests/integration/pipelex/cogt/content_generation/test_leaf_dry_object_mocks.py+++ b/tests/integration/pipelex/cogt/content_generation/test_leaf_dry_object_mocks.py@@ -89,7 +89,7 @@ class SpecWithPipeCode(StructuredContent):def _dry_object_assignment(object_class: type[StructuredContent], nb_items: int | None = None) -> ObjectAssignment:llm_assignment = LLMAssignment(- job_metadata=JobMetadata(user_id="u", pipeline_run_id="run_dry_objects"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_dry_objects"),cogt_run_params=CogtRunParams(run_mode=PipeRunMode.DRY),llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),llm_prompt=LLMPrompt(user_text="make objects"),@@ -130,7 +130,7 @@ class TestLeafDryObjectMocks:async def test_dry_object_list_stamps_regardless_of_mock_usage(self) -> None:"""The stamp is unconditional on the is_mock_usage sub-flag — it only changes reporting (D3)."""llm_assignment = LLMAssignment(- job_metadata=JobMetadata(user_id="u", pipeline_run_id="run_mock_usage_stamp"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_mock_usage_stamp"),cogt_run_params=CogtRunParams(run_mode=PipeRunMode.DRY, is_mock_usage=True),llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),llm_prompt=LLMPrompt(user_text="make specs"),@@ -232,7 +232,7 @@ class TestLeafDryObjectMocks:async def test_boundary_dry_search_structured_dict_validates_against_original_class(self) -> None:"""The structured-search dry boundary leaf returns a dict the original output class accepts."""search_assignment = SearchAssignment(- job_metadata=JobMetadata(user_id="u", pipeline_run_id="run_dry_search_structured"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_dry_search_structured"),cogt_run_params=CogtRunParams(run_mode=PipeRunMode.DRY),query="what is pipelex?",search_setting=SearchSetting(model="mock-search-handle"),diff --git a/tests/integration/pipelex/cogt/content_generation/test_object_class_passthrough.py b/tests/integration/pipelex/cogt/content_generation/test_object_class_passthrough.pyindex 6f3ad83e8..4e0585f61 100644--- a/tests/integration/pipelex/cogt/content_generation/test_object_class_passthrough.py+++ b/tests/integration/pipelex/cogt/content_generation/test_object_class_passthrough.py@@ -80,7 +80,7 @@ class _StubListResponse(BaseModel):def _live_object_assignment() -> ObjectAssignment:llm_assignment = LLMAssignment(- job_metadata=JobMetadata(user_id="u", pipeline_run_id="run_object_class_passthrough"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_object_class_passthrough"),cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),llm_setting=LLMSetting(model="test-model", temperature=0.5),llm_prompt=LLMPrompt(user_text="make a name"),@@ -90,7 +90,7 @@ def _live_object_assignment() -> ObjectAssignment:def _dry_object_assignment() -> ObjectAssignment:llm_assignment = LLMAssignment(- job_metadata=JobMetadata(user_id="u", pipeline_run_id="run_object_class_passthrough_dry"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_object_class_passthrough_dry"),cogt_run_params=CogtRunParams(run_mode=PipeRunMode.DRY),llm_setting=LLMSetting(model="test-model", temperature=0.5),llm_prompt=LLMPrompt(user_text="make a name"),@@ -100,7 +100,7 @@ def _dry_object_assignment() -> ObjectAssignment:def _live_search_assignment() -> SearchAssignment:return SearchAssignment(- job_metadata=JobMetadata(user_id="u", pipeline_run_id="run_search_class_passthrough"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_search_class_passthrough"),cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),query="make a name",search_setting=SearchSetting(model="mock-search-handle"),@@ -236,7 +236,7 @@ class TestObjectClassPassthrough:ObjectAssignment.make_for_class(object_class=NormalizedReference,llm_assignment=LLMAssignment(- job_metadata=JobMetadata(user_id="u", pipeline_run_id="run_wrapped"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_wrapped"),cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),llm_setting=LLMSetting(model="test-model", temperature=0.5),llm_prompt=LLMPrompt(user_text="make a reference"),diff --git a/tests/integration/pipelex/cogt/image/test_data_url_image_flow.py b/tests/integration/pipelex/cogt/image/test_data_url_image_flow.pyindex 4d9a87123..2858deeec 100644--- a/tests/integration/pipelex/cogt/image/test_data_url_image_flow.py+++ b/tests/integration/pipelex/cogt/image/test_data_url_image_flow.py@@ -85,7 +85,7 @@ class TestDataUrlImageFlow:working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)# Normalize- normalized_memory = await normalize_data_urls_to_storage(working_memory)+ normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope")# Verify the URL was converted to pipelex-storage://normalized_stuff = normalized_memory.get_stuff("test_image")@@ -122,7 +122,7 @@ class TestDataUrlImageFlow:working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)# Normalize to storage- normalized_memory = await normalize_data_urls_to_storage(working_memory)+ normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope")normalized_stuff = normalized_memory.get_stuff("test_image")assert isinstance(normalized_stuff.content, ImageContent)storage_uri = normalized_stuff.content.url@@ -163,7 +163,7 @@ class TestDataUrlImageFlow:working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)# Normalize- normalized_memory = await normalize_data_urls_to_storage(working_memory)+ normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope")# Verify all images were convertednormalized_stuff = normalized_memory.get_stuff("image_list")@@ -194,7 +194,7 @@ class TestDataUrlImageFlow:working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)# Normalize- normalized_memory = await normalize_data_urls_to_storage(working_memory)+ normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope")# Verify URL was NOT changednormalized_stuff = normalized_memory.get_stuff("test_image")@@ -224,7 +224,7 @@ class TestDataUrlImageFlow:working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)# Normalize- normalized_memory = await normalize_data_urls_to_storage(working_memory)+ normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope")# Verify the embedded image URL was converted to pipelex-storage://normalized_stuff = normalized_memory.get_stuff("test_article")@@ -264,7 +264,7 @@ class TestDataUrlImageFlow:working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)# Normalize- normalized_memory = await normalize_data_urls_to_storage(working_memory)+ normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope")# Verify all image URLs were convertednormalized_stuff = normalized_memory.get_stuff("nested_article")@@ -314,7 +314,7 @@ class TestDataUrlImageFlow:working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)# Normalize- normalized_memory = await normalize_data_urls_to_storage(working_memory)+ normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope")# Verify all images in the list were convertednormalized_stuff = normalized_memory.get_stuff("gallery")diff --git a/tests/integration/pipelex/cogt/test_extract.py b/tests/integration/pipelex/cogt/test_extract.pyindex eb1974064..0d29d2d8e 100644--- a/tests/integration/pipelex/cogt/test_extract.py+++ b/tests/integration/pipelex/cogt/test_extract.py@@ -41,8 +41,7 @@ class TestExtract:extract_output = await extract_worker.extract_pages(extract_job=extract_job)assert extract_output.pagespage_contents = await generated_content_factory.make_page_contents(- primary_id=job_metadata.user_id,- secondary_id=job_metadata.pipeline_run_id,+ storage_scope=job_metadata.storage_scope,extract_output=extract_output,)assert page_contents@@ -157,8 +156,7 @@ class TestExtract:if page.raw_html:pretty_print(page.raw_html[:500], title="Raw HTML (first 500 chars)")page_contents = await generated_content_factory.make_page_contents(- primary_id=job_metadata.user_id,- secondary_id=job_metadata.pipeline_run_id,+ storage_scope=job_metadata.storage_scope,extract_output=extract_output,)assert page_contentsdiff --git a/tests/integration/pipelex/cogt/test_img2img.py b/tests/integration/pipelex/cogt/test_img2img.pyindex 826aac4c0..70df36039 100644--- a/tests/integration/pipelex/cogt/test_img2img.py+++ b/tests/integration/pipelex/cogt/test_img2img.py@@ -106,8 +106,7 @@ class TestImageToImageGeneration:pretty_print(generated_image_raw_details, title="Generated image raw details")image_content = await generated_content_factory.make_image_content(- primary_id=PRIMARY_ID,- secondary_id=SECONDARY_ID,+ storage_scope="test/scope",raw_details=generated_image_raw_details,)pretty_print(image_content, title="Image content")@@ -163,8 +162,7 @@ class TestImageToImageGeneration:pretty_print(generated_image_raw_details, title="Generated image raw details (style transfer)")image_content = await generated_content_factory.make_image_content(- primary_id=PRIMARY_ID,- secondary_id=SECONDARY_ID,+ storage_scope="test/scope",raw_details=generated_image_raw_details,)pretty_print(image_content, title="Image content (style transfer)")@@ -224,8 +222,7 @@ class TestImageToImageGeneration:pretty_print(generated_image_raw_details, title="Generated image raw details (multi-input)")image_content = await generated_content_factory.make_image_content(- primary_id=PRIMARY_ID,- secondary_id=SECONDARY_ID,+ storage_scope="test/scope",raw_details=generated_image_raw_details,)pretty_print(image_content, title="Image content (multi-input)")@@ -281,8 +278,7 @@ class TestImageToImageGeneration:pretty_print(generated_image_raw_details, title="Generated image raw details (remote URL)")image_content = await generated_content_factory.make_image_content(- primary_id=PRIMARY_ID,- secondary_id=SECONDARY_ID,+ storage_scope="test/scope",raw_details=generated_image_raw_details,)pretty_print(image_content, title="Image content (remote URL)")diff --git a/tests/integration/pipelex/cogt/test_img_gen.py b/tests/integration/pipelex/cogt/test_img_gen.pyindex aea40fa39..dea0e6d1d 100644--- a/tests/integration/pipelex/cogt/test_img_gen.py+++ b/tests/integration/pipelex/cogt/test_img_gen.py@@ -45,8 +45,7 @@ class TestImageGeneration:)pretty_print(generated_image_raw_details, title=f"Generated image raw details for topic '{topic}'")image_content = await generated_content_factory.make_image_content(- primary_id=PRIMARY_ID,- secondary_id=SECONDARY_ID,+ storage_scope="test/scope",raw_details=generated_image_raw_details,)pretty_print(image_content, title=f"Image content for topic '{topic}'")@@ -82,8 +81,7 @@ class TestImageGeneration:)pretty_print(generated_image_raw_details, title=f"Generated image raw details for topic '{topic}'")image_content = await generated_content_factory.make_image_content(- primary_id=PRIMARY_ID,- secondary_id=SECONDARY_ID,+ storage_scope="test/scope",raw_details=generated_image_raw_details,)pretty_print(image_content, title=f"Image content for topic '{topic}'")@@ -117,8 +115,7 @@ class TestImageGeneration:image_content_list = [(await generated_content_factory.make_image_content(- primary_id=PRIMARY_ID,- secondary_id=SECONDARY_ID,+ storage_scope="test/scope",raw_details=generated_image_raw_details,))diff --git a/tests/integration/pipelex/cogt/test_llm_generate_security.py b/tests/integration/pipelex/cogt/test_llm_generate_security.pyindex 61f9f027c..e9dfffc52 100644--- a/tests/integration/pipelex/cogt/test_llm_generate_security.py+++ b/tests/integration/pipelex/cogt/test_llm_generate_security.py@@ -21,7 +21,7 @@ from pipelex.system.pipe_run_mode import PipeRunModedef _make_stub_llm_assignment() -> LLMAssignment:return LLMAssignment(- job_metadata=JobMetadata(user_id="test-user", pipeline_run_id="test-run"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"),cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),llm_setting=LLMSetting(model="test-model", temperature=0.7),llm_prompt=LLMPrompt(user_text="test prompt"),diff --git a/tests/integration/pipelex/fixtures/pipe_job_helpers.py b/tests/integration/pipelex/fixtures/pipe_job_helpers.pyindex c80692585..ee5a805c0 100644--- a/tests/integration/pipelex/fixtures/pipe_job_helpers.py+++ b/tests/integration/pipelex/fixtures/pipe_job_helpers.py@@ -14,7 +14,7 @@ from pipelex.pipe_run.pipe_run_params import PipeRunParamsfrom pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactoryfrom pipelex.system.job_metadata import JobMetadata, SpecialPipelineIdfrom pipelex.system.pipe_run_mode import PipeRunMode-from pipelex.system.telemetry.otel_constants import OTelConstants+from pipelex.system.storage_scope import DRY_RUN_USER_IDPIPE_JOB_HELPERS_PIPELINE_RUN_ID_PREFIX = "pipe_job_helpers_test"@@ -49,7 +49,7 @@ def build_pipe_job(return PipeJobFactory.make_pipe_job(pipe=pipe,pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=pipe_run_mode),- job_metadata=JobMetadata(user_id=OTelConstants.DEFAULT_USER_ID, pipeline_run_id=resolved_pipeline_run_id),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id=DRY_RUN_USER_ID, pipeline_run_id=resolved_pipeline_run_id),working_memory=WorkingMemoryFactory.make_empty(),library_crate=library_crate,)diff --git a/tests/integration/pipelex/kernel/test_usage_parity.py b/tests/integration/pipelex/kernel/test_usage_parity.pyindex b3f43eae0..c3774c716 100644--- a/tests/integration/pipelex/kernel/test_usage_parity.py+++ b/tests/integration/pipelex/kernel/test_usage_parity.py@@ -127,6 +127,7 @@ class TestKernelUsageParity:)try:kernel = PipelexKernel.make(+ storage_scope="test/scope",run_mode=PipeRunMode.DRY,user_id="kernel-usage-parity",is_mock_usage=True,diff --git a/tests/integration/pipelex/pipe_signature/test_pipe_signature.py b/tests/integration/pipelex/pipe_signature/test_pipe_signature.pyindex a001d9560..196ab5585 100644--- a/tests/integration/pipelex/pipe_signature/test_pipe_signature.py+++ b/tests/integration/pipelex/pipe_signature/test_pipe_signature.py@@ -13,7 +13,7 @@ from pipelex.pipe_signature.pipe_signature_blueprint import PipeSignatureBlueprifrom pipelex.pipeline.bundle_validator import BundleValidator, DryRunStatusfrom pipelex.system.job_metadata import JobMetadata, SpecialPipelineIdfrom pipelex.system.pipe_run_mode import PipeRunMode-from pipelex.system.telemetry.otel_constants import OTelConstants+from pipelex.system.storage_scope import DRY_RUN_USER_IDfrom tests.integration.pipelex.pipe_signature.conftest import SIGNATURES_DOMAIN_CODEif TYPE_CHECKING:@@ -101,7 +101,7 @@ class TestPipeSignature:needed = WorkingMemoryFactory.convert_to_working_memory_format(needed_inputs_spec=runtime.needed_inputs())working_memory = WorkingMemoryFactory.make_mock_inputs(needed_inputs=needed)await runtime.run_pipe(- job_metadata=JobMetadata(user_id=OTelConstants.DEFAULT_USER_ID, pipeline_run_id=SpecialPipelineId.DRY_RUN_UNTITLED),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id=DRY_RUN_USER_ID, pipeline_run_id=SpecialPipelineId.DRY_RUN_UNTITLED),working_memory=working_memory,pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.DRY),)@@ -120,7 +120,7 @@ class TestPipeSignature:needed = WorkingMemoryFactory.convert_to_working_memory_format(needed_inputs_spec=runtime.needed_inputs())working_memory = WorkingMemoryFactory.make_mock_inputs(needed_inputs=needed)await runtime.run_pipe(- job_metadata=JobMetadata(user_id=OTelConstants.DEFAULT_USER_ID, pipeline_run_id=SpecialPipelineId.DRY_RUN_UNTITLED),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id=DRY_RUN_USER_ID, pipeline_run_id=SpecialPipelineId.DRY_RUN_UNTITLED),working_memory=working_memory,pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.DRY),)@@ -141,7 +141,7 @@ class TestPipeSignature:working_memory = WorkingMemoryFactory.make_empty()with pytest.raises(PipeSignatureNotExecutableError) as exc_info:await runtime._live_run_pipe( # noqa: SLF001 # pyright: ignore[reportPrivateUsage]- job_metadata=JobMetadata(user_id=OTelConstants.DEFAULT_USER_ID, pipeline_run_id=SpecialPipelineId.UNTITLED),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id=DRY_RUN_USER_ID, pipeline_run_id=SpecialPipelineId.UNTITLED),working_memory=working_memory,pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.LIVE),)@@ -162,7 +162,7 @@ class TestPipeSignature:working_memory = WorkingMemoryFactory.make_empty()with pytest.raises(PipeSignatureNotExecutableError) as exc_info:await runtime.run_pipe(- job_metadata=JobMetadata(user_id=OTelConstants.DEFAULT_USER_ID, pipeline_run_id=SpecialPipelineId.UNTITLED),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id=DRY_RUN_USER_ID, pipeline_run_id=SpecialPipelineId.UNTITLED),working_memory=working_memory,pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.LIVE),)diff --git a/tests/integration/pipelex/pipeline/test_execution_seams.py b/tests/integration/pipelex/pipeline/test_execution_seams.pyindex 6b76e0187..bc373e800 100644--- a/tests/integration/pipelex/pipeline/test_execution_seams.py+++ b/tests/integration/pipelex/pipeline/test_execution_seams.py@@ -16,7 +16,7 @@ from pipelex.pipeline import execution_seams as execution_seams_modulefrom pipelex.pipeline.execution_seams import acquire_library, prepare_pipe_jobfrom pipelex.system.configuration.configs import PipelineExecutionConfigfrom pipelex.system.pipe_run_mode import PipeRunMode-from pipelex.system.telemetry.otel_constants import OTelConstants+from pipelex.system.storage_scope import DRY_RUN_USER_ID_SEAMS_DOMAIN = "seams_test"_SEAMS_MTHDS = f"""@@ -143,12 +143,13 @@ class TestExecutionSeams:assert qualified_main_pipe is not Nonepipe = get_required_entry_pipe(pipe_code=qualified_main_pipe)pipe_job = await prepare_pipe_job(+ storage_scope="test/scope",+ user_id="test-user",pipe=pipe,library_id=library_id,execution_config=_dry_mock_config(),pipe_run_mode=PipeRunMode.DRY,pipeline_run_id="seams-prepare-run-id",- user_id=OTelConstants.DEFAULT_USER_ID,)assert pipe_job.pipe.code == "echo_topic"assert pipe_job.pipe_run_params.run_mode.is_dry@@ -160,6 +161,65 @@ class TestExecutionSeams:library_manager.teardown(library_id=library_id)clear_current_library()+ @pytest.mark.parametrize(+ "bad_scope",+ [+ pytest.param("../other/run", id="traversal-leading"),+ pytest.param("tenant/../other", id="traversal-interior"),+ pytest.param("/tenant/run", id="leading-slash-absolute-key"),+ pytest.param("", id="empty"),+ pytest.param("tenant/run\n", id="trailing-newline"),+ ],+ )+ async def test_an_unsafe_scope_is_refused_before_anything_is_written(self, mocker: MockerFixture, bad_scope: str) -> None:+ """The validator must run BEFORE the normalizer, not after it.++ `JobMetadata` validates the scope, and that is where the invariant is+ documented — but it was constructed *below* the data-url normalization,+ which writes `{storage_scope}/assets/...` to real storage. So a scope+ carrying `..` escaped the tenant's namespace and the guard meant to stop+ it ran afterwards, on damage already done. The defect was ordering, not+ absence, which is why asserting the raise is not enough: the spy is what+ proves nothing was written on the way to it.+ """+ library_manager = get_library_manager()+ library_id = "seams_scope_guard_lib"+ _, qualified_main_pipe = acquire_library(library_id=library_id, mthds_contents=[_SEAMS_MTHDS])+ try:+ assert qualified_main_pipe is not None+ pipe = get_required_entry_pipe(pipe_code=qualified_main_pipe)+ normalize_spy = mocker.spy(execution_seams_module, "normalize_data_urls_to_storage")+ normalize_config = (+ get_config()+ .interpreter.pipeline_execution.with_execution_overrides(+ generate_graph=False,+ mock_inputs=False,+ )+ .model_copy(update={"is_normalize_data_urls_to_storage": True})+ )++ with pytest.raises(ValueError, match="Invalid storage_scope"):+ await prepare_pipe_job(+ storage_scope=bad_scope,+ pipe=pipe,+ library_id=library_id,+ execution_config=normalize_config,+ pipe_run_mode=PipeRunMode.DRY,+ pipeline_run_id="seams-scope-guard-run-id",+ user_id=DRY_RUN_USER_ID,+ # NON-EMPTY on purpose. With `PipelineInputs()` the falsy-inputs+ # gate skips normalize on its own, so the spy assertion below+ # would hold no matter where the validator sits — a test that+ # passes with the guard deleted. Real inputs are what put the+ # normalizer on the path this test is about.+ inputs=PipelineInputs(root={"subject": "a topic"}),+ )++ assert normalize_spy.call_count == 0+ finally:+ library_manager.teardown(library_id=library_id)+ clear_current_library()+async def test_prepare_pipe_job_skips_normalize_for_empty_inputs(self, mocker: MockerFixture) -> None:# Pins the falsy-inputs semantics at the normalize gate — the discriminating check the# pipeline_run_setup characterization test could not make. With empty PipelineInputs (falsy) +@@ -185,12 +245,13 @@ class TestExecutionSeams:)pipe_job = await prepare_pipe_job(+ storage_scope="test/scope",pipe=pipe,library_id=library_id,execution_config=normalize_config,pipe_run_mode=PipeRunMode.DRY,pipeline_run_id="seams-normalize-run-id",- user_id=OTelConstants.DEFAULT_USER_ID,+ user_id=DRY_RUN_USER_ID,inputs=PipelineInputs(),)diff --git a/tests/integration/pipelex/pipeline/test_optional_method_inputs.py b/tests/integration/pipelex/pipeline/test_optional_method_inputs.pyindex 6d8ddabbf..cee841b83 100644--- a/tests/integration/pipelex/pipeline/test_optional_method_inputs.py+++ b/tests/integration/pipelex/pipeline/test_optional_method_inputs.py@@ -64,6 +64,7 @@ class TestOptionalMethodInputs:execution_config = get_config().interpreter.pipeline_executionpipe_job = await prepare_pipe_job(+ storage_scope="test/scope",pipe=pipe,library_id=library_id,execution_config=execution_config,@@ -94,6 +95,7 @@ class TestOptionalMethodInputs:execution_config = get_config().interpreter.pipeline_executionpipe_job = await prepare_pipe_job(+ storage_scope="test/scope",pipe=pipe,library_id=library_id,execution_config=execution_config,@@ -127,6 +129,7 @@ class TestOptionalMethodInputs:chained_memory.record_absence(upstream_record)pipe_job = await prepare_pipe_job(+ storage_scope="test/scope",pipe=pipe,library_id=library_id,execution_config=execution_config,@@ -173,6 +176,7 @@ class TestOptionalMethodInputs:)pipe_job = await prepare_pipe_job(+ storage_scope="test/scope",pipe=pipe,library_id=library_id,execution_config=execution_config,diff --git a/tests/integration/pipelex/pipeline/test_pipeline_run_id_resubmission.py b/tests/integration/pipelex/pipeline/test_pipeline_run_id_resubmission.pyindex eec1156a2..bddcf576e 100644--- a/tests/integration/pipelex/pipeline/test_pipeline_run_id_resubmission.py+++ b/tests/integration/pipelex/pipeline/test_pipeline_run_id_resubmission.py@@ -129,6 +129,8 @@ class TestPipelineRunIdResubmission:# id on this path, so setup must remove its own registration.with pytest.raises(PipeNotFoundError):await pipeline_run_setup(+ storage_scope="test/scope",+ user_id="test-user",execution_config=_dry_mock_config(),mthds_contents=[_RESUBMISSION_MTHDS],pipe_code="absent_pipe",@@ -140,6 +142,8 @@ class TestPipelineRunIdResubmission:# Resubmission of the SAME explicit id now succeeds (before the fix: process-permanent# PipelineManagerAlreadyExistsError).pipe_job, pipeline_run_id, library_id = await pipeline_run_setup(+ storage_scope="test/scope",+ user_id="test-user",execution_config=_dry_mock_config(),mthds_contents=[_RESUBMISSION_MTHDS],pipe_code="echo_topic",@@ -193,6 +197,8 @@ class TestPipelineRunIdResubmission:# The ORIGINAL setup failure propagates — the flush error must not replace it.with pytest.raises(RuntimeError, match=prepare_failure_msg):await pipeline_run_setup(+ storage_scope="test/scope",+ user_id="test-user",execution_config=graph_config,mthds_contents=[_RESUBMISSION_MTHDS],pipe_code="echo_topic",@@ -211,6 +217,8 @@ class TestPipelineRunIdResubmission:mocker.stop(prepare_mock)mocker.stop(teardown_mock)pipe_job, pipeline_run_id, library_id = await pipeline_run_setup(+ storage_scope="test/scope",+ user_id="test-user",execution_config=_dry_mock_config(),mthds_contents=[_RESUBMISSION_MTHDS],pipe_code="echo_topic",diff --git a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_characterization.py b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_characterization.pyindex da5031c58..f6199aabf 100644--- a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_characterization.py+++ b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_characterization.py@@ -113,6 +113,8 @@ class TestPipelineRunSetupCharacterization:teardown_before = teardown_spy.call_countpipe_job, pipeline_run_id, library_id = await pipeline_run_setup(+ storage_scope="test/scope",+ user_id="test-user",execution_config=_dry_mock_config(),mthds_contents=[_CHAR_MTHDS],pipe_code="echo_topic",@@ -151,6 +153,8 @@ class TestPipelineRunSetupCharacterization:wraps=pipeline_run_setup_module.prepare_pipe_job,)_, _, library_id = await pipeline_run_setup(+ storage_scope="test/scope",+ user_id="test-user",execution_config=_dry_mock_config(),mthds_contents=[_CHAR_MTHDS],pipe_code="echo_topic",@@ -165,6 +169,8 @@ class TestPipelineRunSetupCharacterization:async def test_empty_inputs_behave_like_no_inputs(self) -> None:pipe_job, _, library_id = await pipeline_run_setup(+ storage_scope="test/scope",+ user_id="test-user",execution_config=_no_mock_config(),mthds_contents=[_CHAR_MTHDS],pipe_code="echo_topic",@@ -206,6 +212,8 @@ class TestPipelineRunSetupCharacterization:with pytest.raises(PipeNotFoundError):await pipeline_run_setup(+ storage_scope="test/scope",+ user_id="test-user",execution_config=_dry_mock_config(),mthds_contents=[_CHAR_MTHDS],pipe_code="absent_pipe",@@ -257,6 +265,8 @@ class TestPipelineRunSetupCharacterization:try:with pytest.raises(PipeNotFoundError):await pipeline_run_setup(+ storage_scope="test/scope",+ user_id="test-user",execution_config=_dry_mock_config(),mthds_contents=[_CHAR_MTHDS],pipe_code="absent_pipe",@@ -282,6 +292,8 @@ class TestPipelineRunSetupCharacterization:try:with pytest.raises(PipeNotFoundError):await pipeline_run_setup(+ storage_scope="test/scope",+ user_id="test-user",execution_config=_dry_mock_config(),library_id=collide_library_id,mthds_contents=[_CHAR_MTHDS],diff --git a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_emit_gates.py b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_emit_gates.pyindex 55fe9e287..c0c2dd201 100644--- a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_emit_gates.py+++ b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_emit_gates.py@@ -23,6 +23,7 @@ from pipelex.pipeline.pipeline_run_setup import pipeline_run_setupfrom pipelex.runtime_hub import get_report_delegatefrom pipelex.system.configuration.configs import NdjsonTracingConfig, PipelineExecutionConfig, TracingBackendfrom pipelex.system.pipe_run_mode import PipeRunMode+from pipelex.system.storage_scope import LOCAL_STORAGE_SCOPE, LOCAL_USER_ID_GATE_DOMAIN = "prs_emit_gates"_GATE_MTHDS = f"""@@ -79,6 +80,8 @@ class TestPipelineRunSetupEmitGates:set_event_log_spy = mocker.spy(get_report_delegate(), "set_event_log")pipe_job, pipeline_run_id, library_id = await pipeline_run_setup(+ storage_scope="test/scope",+ user_id="test-user",execution_config=_config(generate_graph=False, generate_usage=True),mthds_contents=[_GATE_MTHDS],pipe_code="echo_topic",@@ -101,6 +104,8 @@ class TestPipelineRunSetupEmitGates:set_event_log_spy = mocker.spy(get_report_delegate(), "set_event_log")pipe_job, pipeline_run_id, library_id = await pipeline_run_setup(+ storage_scope="test/scope",+ user_id="test-user",execution_config=_config(generate_graph=True, generate_usage=False),mthds_contents=[_GATE_MTHDS],pipe_code="echo_topic",@@ -116,3 +121,61 @@ class TestPipelineRunSetupEmitGates:assert pipeline_run_id not in registered_keysfinally:_cleanup(pipeline_run_id, library_id)++[email protected](loop_scope="class")+class TestTheLocalScopeIsPerRun:+ """A local run's scope IS its run id, so two local runs cannot share a key.++ The run id used to be IN the storage key+ (`{user_id}/{key_prefix}{pipeline_run_id}`). Collapsing the key onto the+ scope removed it, which is right for a host — a hosted scope already+ identifies the run, `<org>/<method>/<run>` — but left the local default+ naming a tenant and no run. Every local run with storage delivery then wrote+ `local/results/<filename>`: the same key each time, so each run silently+ destroyed the previous one's output, with nothing failing to say so.+ """++ async def test_two_local_runs_get_distinct_scopes(self) -> None:+ seen: list[str] = []+ for _ in range(2):+ pipe_job, pipeline_run_id, library_id = await pipeline_run_setup(+ storage_scope=LOCAL_STORAGE_SCOPE,+ user_id=LOCAL_USER_ID,+ execution_config=_config(generate_graph=False, generate_usage=False),+ mthds_contents=[_GATE_MTHDS],+ pipe_code="echo_topic",+ pipe_run_mode=PipeRunMode.DRY,+ )+ try:+ scope = pipe_job.job_metadata.storage_scope+ # The sentinel must NOT survive onto the job — it is "no host+ # supplied a scope", not a prefix, and a scope shared by every+ # run is the shared-key state this repairs.+ assert scope != LOCAL_STORAGE_SCOPE+ assert scope == pipeline_run_id+ seen.append(scope)+ finally:+ _cleanup(pipeline_run_id, library_id)++ assert seen[0] != seen[1], "two local runs shared a storage scope, so their results overwrite each other"++ async def test_a_host_supplied_scope_is_left_alone(self) -> None:+ """Keyed on the sentinel exactly — a real tenancy is never rewritten.++ `local` is our own constant. A host that serves tenants passes its own+ scope and must come out the other side with precisely that, or the+ per-run suffix would be corrupting the tenant boundary it computed.+ """+ pipe_job, pipeline_run_id, library_id = await pipeline_run_setup(+ storage_scope="org_a/mt_b/run_c",+ user_id="test-user",+ execution_config=_config(generate_graph=False, generate_usage=False),+ mthds_contents=[_GATE_MTHDS],+ pipe_code="echo_topic",+ pipe_run_mode=PipeRunMode.DRY,+ )+ try:+ assert pipe_job.job_metadata.storage_scope == "org_a/mt_b/run_c"+ finally:+ _cleanup(pipeline_run_id, library_id)diff --git a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_request_id.py b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_request_id.pyindex 930005603..bfa6797ec 100644--- a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_request_id.py+++ b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_request_id.py@@ -40,6 +40,8 @@ class TestPipelineRunSetupRequestId:generate_graph=False,)pipe_job, _, _ = await pipeline_run_setup(+ storage_scope="test/scope",+ user_id="test-user",execution_config=execution_config,mthds_contents=[_MINIMAL_MTHDS],pipe_code="echo_topic",@@ -53,6 +55,8 @@ class TestPipelineRunSetupRequestId:generate_graph=False,)pipe_job, _, _ = await pipeline_run_setup(+ storage_scope="test/scope",+ user_id="test-user",execution_config=execution_config,mthds_contents=[_MINIMAL_MTHDS],pipe_code="echo_topic",diff --git a/tests/integration/pipelex/pipes/optionals/test_lift_sequence.py b/tests/integration/pipelex/pipes/optionals/test_lift_sequence.pyindex 60c03cad0..434f0b8af 100644--- a/tests/integration/pipelex/pipes/optionals/test_lift_sequence.py+++ b/tests/integration/pipelex/pipes/optionals/test_lift_sequence.py@@ -249,12 +249,13 @@ class TestLiftSequence:sequence = _build_lift_sequence()pipe_job = await prepare_pipe_job(+ storage_scope="test/scope",+ user_id="test-user",pipe=sequence,library_id=library_id,execution_config=get_config().interpreter.pipeline_execution,pipe_run_mode=PipeRunMode.LIVE,pipeline_run_id="test-optional-boundary-seeding",- user_id="pytest",inputs=PipelineInputs({"topic": "penalties"}),)diff --git a/tests/integration/pipelex/pipes/optionals/test_lift_skip_trichotomy.py b/tests/integration/pipelex/pipes/optionals/test_lift_skip_trichotomy.pyindex ce560f272..bedcdf7ef 100644--- a/tests/integration/pipelex/pipes/optionals/test_lift_skip_trichotomy.py+++ b/tests/integration/pipelex/pipes/optionals/test_lift_skip_trichotomy.py@@ -295,6 +295,7 @@ class TestLiftSkipTrichotomy:return_value=mock_manager,)traced_metadata = JobMetadata(+ storage_scope="test/scope",user_id="pytest",pipeline_run_id=job_metadata.pipeline_run_id,trace_context=make_trace_context(graph_id=job_metadata.pipeline_run_id),diff --git a/tests/integration/pipelex/system/test_keyless_boot_forced_dry.py b/tests/integration/pipelex/system/test_keyless_boot_forced_dry.pyindex 252cee5cc..bfb9a3546 100644--- a/tests/integration/pipelex/system/test_keyless_boot_forced_dry.py+++ b/tests/integration/pipelex/system/test_keyless_boot_forced_dry.py@@ -73,12 +73,13 @@ class TestKeylessBootForcedDry:),)pipe_job = await prepare_pipe_job(+ storage_scope="test/scope",+ user_id="test-user",pipe=pipe,library_id=library_id,execution_config=get_config().interpreter.pipeline_execution,pipe_run_mode=PipeRunMode.LIVE,pipeline_run_id="keyless_forced_dry_run",- user_id="test-user",)assert pipe_job.pipe_run_params.run_mode.is_dryfinally:@@ -88,7 +89,7 @@ class TestKeylessBootForcedDry:"""The kernel tier's run-params factory honours the same flag — and its default is LIVE.Asserted on the default rather than on an explicit ``run_mode=LIVE`` because the default is- the reachable hazard: a programmatic caller writes ``PipelexKernel.make(user_id=...)``,+ the reachable hazard: a programmatic caller writes ``PipelexKernel.make(storage_scope="test/scope", user_id=...)``,names no mode, and without the coercion gets a LIVE ``CogtRunParams`` that walks straightpast the leaf's DRY branch into a real provider call."""@@ -96,7 +97,7 @@ class TestKeylessBootForcedDry:self._boot_keyless()assert is_dry_run_forced()- kernel = PipelexKernel.make(user_id="test-user")+ kernel = PipelexKernel.make(storage_scope="test/scope", user_id="test-user")assert kernel.cogt_run_params.run_mode.is_dryfinally:@@ -114,7 +115,7 @@ class TestKeylessBootForcedDry:assert is_dry_run_forced()with pytest.raises(ValueError, match="is_mock_usage"):- PipelexKernel.make(run_mode=PipeRunMode.LIVE, user_id="test-user", is_mock_usage=True)+ PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.LIVE, user_id="test-user", is_mock_usage=True)finally:Pipelex.teardown_if_needed()diff --git a/tests/integration/pipelex/tools/storage/test_generated_image_storage.py b/tests/integration/pipelex/tools/storage/test_generated_image_storage.pyindex a8a112be7..dd1845c4b 100644--- a/tests/integration/pipelex/tools/storage/test_generated_image_storage.py+++ b/tests/integration/pipelex/tools/storage/test_generated_image_storage.py@@ -59,8 +59,7 @@ class TestGeneratedImageStorage:# Generate ImageContent via factoryimage_content = await generated_content_factory.make_image_content(- primary_id="test_pipeline",- secondary_id="test_step",+ storage_scope="test/scope",raw_details=raw_details,)@@ -102,8 +101,7 @@ class TestGeneratedImageStorage:# Generate ImageContent via factoryimage_content = await generated_content_factory.make_image_content(- primary_id="test_pipeline",- secondary_id="test_step",+ storage_scope="test/scope",raw_details=raw_details,)@@ -138,16 +136,22 @@ class TestGeneratedImageStorage:# Generate ImageContentimage_content = await generated_content_factory.make_image_content(- primary_id="pipeline_abc",- secondary_id="step_xyz",+ storage_scope="test/scope",raw_details=raw_details,)# Extract the key from the URI (remove scheme prefix)storage_key = image_content.url.removeprefix(PIPELEX_STORAGE_SCHEME)- # Verify the key follows the uri_format pattern- # The format is "{primary_id}/{secondary_id}/{hash}.{extension}"- assert "pipeline_abc" in storage_key- assert "step_xyz" in storage_key+ # Verify the key follows the uri_format pattern: "{storage_scope}/{hash}.{extension}".+ # It used to assert a pipeline id and a step id, from the `{primary_id}`/+ # `{secondary_id}` layout — keying generated bytes by WHICH RUN produced+ # them, which is exactly the coupling `storage_scope` replaces.+ # The `generated/` leaf must be THERE. Asserting only the scope prefix+ # would pass just as well with the bytes loose at the root of the scope,+ # which is the state this leaf exists to end: a file under no leaf is+ # invisible to anything listing a run's content by category, and shares+ # the root with every leaf added later.+ assert storage_key.startswith("test/scope/generated/")+ assert storage_key.count("/") == 3 # test / scope / generated / <file>assert storage_key.endswith(".png")diff --git a/tests/integration/pipelex/tools/storage/test_storage_config.py b/tests/integration/pipelex/tools/storage/test_storage_config.pyindex e53cb82a6..8af2abbf5 100644--- a/tests/integration/pipelex/tools/storage/test_storage_config.py+++ b/tests/integration/pipelex/tools/storage/test_storage_config.py@@ -6,24 +6,27 @@ class TestStorageConfigIntegration:"""Integration tests for storage config values."""def test_config_has_local_uri_format(self) -> None:- """Test that local storage config has uri_format defined."""+ """The shipped format renders a FILENAME — it must not try to build the prefix.++ `{storage_scope}` is no longer a supported placeholder: the scope and the+ `generated/` leaf are composed in code, so a config that named them would+ either be rejected at boot or double the prefix.+ """storage_config = get_config().runtime.storageassert storage_config.local is not Noneassert storage_config.local.uri_format is not None- assert "{primary_id}" in storage_config.local.uri_format- assert "{secondary_id}" in storage_config.local.uri_formatassert "{hash}" in storage_config.local.uri_formatassert "{extension}" in storage_config.local.uri_format+ assert "{storage_scope}" not in storage_config.local.uri_formatdef test_config_has_in_memory_uri_format(self) -> None:"""Test that in-memory storage config has uri_format defined."""storage_config = get_config().runtime.storageassert storage_config.in_memory is not Noneassert storage_config.in_memory.uri_format is not None- assert "{primary_id}" in storage_config.in_memory.uri_format- assert "{secondary_id}" in storage_config.in_memory.uri_formatassert "{hash}" in storage_config.in_memory.uri_formatassert "{extension}" in storage_config.in_memory.uri_format+ assert "{storage_scope}" not in storage_config.in_memory.uri_formatdef test_uri_format_property_returns_correct_format_for_local(self) -> None:"""Test that the uri_format property returns the correct format when method is local."""diff --git a/tests/integration/pipelex/tools/storage/test_storage_provider_contract.py b/tests/integration/pipelex/tools/storage/test_storage_provider_contract.pyindex 4a1c5c387..c4cf43b31 100644--- a/tests/integration/pipelex/tools/storage/test_storage_provider_contract.py+++ b/tests/integration/pipelex/tools/storage/test_storage_provider_contract.py@@ -97,7 +97,7 @@ class TestStorageProviderContract:) -> None:"""Contract: store() must handle nested key paths with forward slashes.- The uri_format uses paths like "{primary_id}/{secondary_id}/{hash}.{extension}"+ The uri_format uses paths like "{storage_scope}/{hash}.{extension}"which creates nested directories/paths. Storage providers must support this."""test_data = b"nested path test"diff --git a/tests/integration/pipelex/tools/storage/test_storage_providers.py b/tests/integration/pipelex/tools/storage/test_storage_providers.pyindex b936e7416..131d2b452 100644--- a/tests/integration/pipelex/tools/storage/test_storage_providers.py+++ b/tests/integration/pipelex/tools/storage/test_storage_providers.py@@ -25,26 +25,22 @@ class TestStorageProviders:) -> None:"""Test that the URI format from config works correctly for building storage keys."""test_data = b"config format test"- primary_id = "pipeline_123"- secondary_id = "step_456"hash_value = "abc123def456"extension = "png"- key = uri_format.format(- primary_id=primary_id,- secondary_id=secondary_id,- hash=hash_value,- extension=extension,- )+ # `uri_format` renders the FILENAME only. The prefix — the scope and the+ # `generated/` leaf — is composed in code by `GeneratedContentFactory`,+ # precisely so a config cannot omit it; asserting a scope segment here+ # would be asserting against a string this format no longer produces.+ filename = uri_format.format(hash=hash_value, extension=extension)- returned_uri = await storage_provider.store(data=test_data, key=key)+ returned_uri = await storage_provider.store(data=test_data, key=filename)loaded_data = await storage_provider.load(uri=returned_uri)assert loaded_data == test_data- assert primary_id in key- assert secondary_id in key- assert hash_value in key- assert extension in key+ assert hash_value in filename+ assert extension in filename+ assert not filename.startswith("/")async def test_store_binary_image_data(self, storage_provider: StorageProviderAbstract) -> None:"""Test storing and retrieving binary image data (PNG header)."""diff --git a/tests/integration/pipelex/tools/storage/test_user_provided_image_storage.py b/tests/integration/pipelex/tools/storage/test_user_provided_image_storage.pyindex da87aceff..ca9f4354b 100644--- a/tests/integration/pipelex/tools/storage/test_user_provided_image_storage.py+++ b/tests/integration/pipelex/tools/storage/test_user_provided_image_storage.py@@ -54,7 +54,7 @@ class TestUserProvidedImageStorage:working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)# Normalize to storage- normalized_memory = await normalize_data_urls_to_storage(working_memory)+ normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope")# Verify the URL was converted to pipelex-storage://normalized_stuff = normalized_memory.get_stuff("user_image")@@ -115,7 +115,7 @@ class TestUserProvidedImageStorage:working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)# Normalize (should not change HTTP URLs when fetch is disabled)- normalized_memory = await normalize_data_urls_to_storage(working_memory)+ normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope")# Verify URL was NOT changednormalized_stuff = normalized_memory.get_stuff("remote_image")@@ -139,7 +139,7 @@ class TestUserProvidedImageStorage:working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)# Normalize - should upload the local file to storage- normalized_memory = await normalize_data_urls_to_storage(working_memory)+ normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope")# Verify the URL was converted to pipelex-storage://normalized_stuff = normalized_memory.get_stuff("local_image")@@ -178,7 +178,7 @@ class TestUserProvidedImageStorage:working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)# Normalize (should not change local paths when upload is disabled)- normalized_memory = await normalize_data_urls_to_storage(working_memory)+ normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope")# Verify URL was NOT changednormalized_stuff = normalized_memory.get_stuff("local_image")diff --git a/tests/unit/pipelex/cogt/content_generation/test_assignment_models_schema.py b/tests/unit/pipelex/cogt/content_generation/test_assignment_models_schema.pyindex a1aa73f7d..2eae63c96 100644--- a/tests/unit/pipelex/cogt/content_generation/test_assignment_models_schema.py+++ b/tests/unit/pipelex/cogt/content_generation/test_assignment_models_schema.py@@ -17,6 +17,7 @@ class SampleOutputModel(BaseModel):def _make_stub_job_metadata() -> JobMetadata:return JobMetadata(+ storage_scope="test/scope",user_id="test-user",pipeline_run_id="test-run",)diff --git a/tests/unit/pipelex/cogt/content_generation/test_assignment_models_security.py b/tests/unit/pipelex/cogt/content_generation/test_assignment_models_security.pyindex e0ba4e674..9cd81cf5b 100644--- a/tests/unit/pipelex/cogt/content_generation/test_assignment_models_security.py+++ b/tests/unit/pipelex/cogt/content_generation/test_assignment_models_security.py@@ -19,6 +19,7 @@ from pipelex.system.pipe_run_mode import PipeRunModedef _make_stub_job_metadata() -> JobMetadata:return JobMetadata(+ storage_scope="test/scope",user_id="test-user",pipeline_run_id="test-run",)diff --git a/tests/unit/pipelex/cogt/content_generation/test_dry_mock.py b/tests/unit/pipelex/cogt/content_generation/test_dry_mock.pyindex 9e92731f4..24294610a 100644--- a/tests/unit/pipelex/cogt/content_generation/test_dry_mock.py+++ b/tests/unit/pipelex/cogt/content_generation/test_dry_mock.py@@ -31,7 +31,7 @@ from pipelex.system.pipe_run_mode import PipeRunModedef _dry_object_assignment() -> ObjectAssignment:llm_assignment = LLMAssignment(- job_metadata=JobMetadata(user_id="u", pipeline_run_id="run_dry_mock"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_dry_mock"),cogt_run_params=CogtRunParams(run_mode=PipeRunMode.DRY),llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),llm_prompt=LLMPrompt(),@@ -56,7 +56,7 @@ class TestDryMock:"""report_mock_usage_llm_job emits non-zero usage under the mock_usage sentinel -> reportable."""captured = self._capture_reported_job(mocker)report_mock_usage_llm_job(- job_metadata=JobMetadata(user_id="u", pipeline_run_id="run_mock"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_mock"),llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),llm_prompt=LLMPrompt(),)@@ -73,7 +73,7 @@ class TestDryMock:"""report_dry_llm_job stays zero-token under the dry_run sentinel -> NOT reportable (suppressed)."""captured = self._capture_reported_job(mocker)report_dry_llm_job(- job_metadata=JobMetadata(user_id="u", pipeline_run_id="run_dry"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_dry"),llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),llm_prompt=LLMPrompt(),)diff --git a/tests/unit/pipelex/cogt/content_generation/test_dry_report_tz.py b/tests/unit/pipelex/cogt/content_generation/test_dry_report_tz.pyindex 63497ea7b..7b17455b1 100644--- a/tests/unit/pipelex/cogt/content_generation/test_dry_report_tz.py+++ b/tests/unit/pipelex/cogt/content_generation/test_dry_report_tz.py@@ -47,6 +47,7 @@ class TestDryReportTimezone:)job_metadata = JobMetadata(+ storage_scope="test/scope",user_id="test_user",pipeline_run_id="run_dry_tz",started_at=started_at,diff --git a/tests/unit/pipelex/cogt/content_generation/test_extract_generate_dry_branch.py b/tests/unit/pipelex/cogt/content_generation/test_extract_generate_dry_branch.pyindex e0fba4563..29f0de896 100644--- a/tests/unit/pipelex/cogt/content_generation/test_extract_generate_dry_branch.py+++ b/tests/unit/pipelex/cogt/content_generation/test_extract_generate_dry_branch.py@@ -22,7 +22,7 @@ from pipelex.system.pipe_run_mode import PipeRunModeclass TestExtractGenerateDryBranch:def _assignment(self, *, run_mode: PipeRunMode, extract_input: ExtractInput) -> ExtractAssignment:return ExtractAssignment(- job_metadata=JobMetadata(user_id="u", pipeline_run_id="run_extract_dry"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_extract_dry"),cogt_run_params=CogtRunParams(run_mode=run_mode),extract_handle="mock-extract-handle",extract_input=extract_input,diff --git a/tests/unit/pipelex/cogt/content_generation/test_img_gen_generate_dry_branch.py b/tests/unit/pipelex/cogt/content_generation/test_img_gen_generate_dry_branch.pyindex 859f38ed3..53c698b77 100644--- a/tests/unit/pipelex/cogt/content_generation/test_img_gen_generate_dry_branch.py+++ b/tests/unit/pipelex/cogt/content_generation/test_img_gen_generate_dry_branch.py@@ -21,7 +21,7 @@ from pipelex.system.pipe_run_mode import PipeRunModeclass TestImgGenGenerateDryBranch:def _assignment(self, *, run_mode: PipeRunMode, nb_images: int = 1) -> ImgGenAssignment:return ImgGenAssignment(- job_metadata=JobMetadata(user_id="u", pipeline_run_id="run_img_dry"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_img_dry"),cogt_run_params=CogtRunParams(run_mode=run_mode),img_gen_handle="mock-img-handle",img_gen_prompt=ImgGenPrompt(positive_text="a red apple", negative_text="no worms"),diff --git a/tests/unit/pipelex/cogt/content_generation/test_llm_generate_dry_branch.py b/tests/unit/pipelex/cogt/content_generation/test_llm_generate_dry_branch.pyindex e17240541..2ac244d98 100644--- a/tests/unit/pipelex/cogt/content_generation/test_llm_generate_dry_branch.py+++ b/tests/unit/pipelex/cogt/content_generation/test_llm_generate_dry_branch.py@@ -29,7 +29,7 @@ from pipelex.system.pipe_run_mode import PipeRunModeclass TestLlmGenerateDryBranch:def _assignment(self, *, run_mode: PipeRunMode, is_mock_usage: bool = False) -> LLMAssignment:return LLMAssignment(- job_metadata=JobMetadata(user_id="u", pipeline_run_id="run_dry_branch"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_dry_branch"),cogt_run_params=CogtRunParams(run_mode=run_mode, is_mock_usage=is_mock_usage),llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),llm_prompt=LLMPrompt(),diff --git a/tests/unit/pipelex/cogt/content_generation/test_render_generate_dry_branch.py b/tests/unit/pipelex/cogt/content_generation/test_render_generate_dry_branch.pyindex d1bba83ff..e2f9976aa 100644--- a/tests/unit/pipelex/cogt/content_generation/test_render_generate_dry_branch.py+++ b/tests/unit/pipelex/cogt/content_generation/test_render_generate_dry_branch.py@@ -18,7 +18,7 @@ from pipelex.system.pipe_run_mode import PipeRunModeclass TestRenderGenerateDryBranch:def _assignment(self, *, run_mode: PipeRunMode) -> RenderPageViewsAssignment:return RenderPageViewsAssignment(- job_metadata=JobMetadata(user_id="u", pipeline_run_id="run_render_dry"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_render_dry"),cogt_run_params=CogtRunParams(run_mode=run_mode),document_uri="file:///tmp/doc.pdf",page_views_dpi=72,diff --git a/tests/unit/pipelex/cogt/content_generation/test_search_generate_dry_branch.py b/tests/unit/pipelex/cogt/content_generation/test_search_generate_dry_branch.pyindex a52a54fc0..877e87c8e 100644--- a/tests/unit/pipelex/cogt/content_generation/test_search_generate_dry_branch.py+++ b/tests/unit/pipelex/cogt/content_generation/test_search_generate_dry_branch.py@@ -22,7 +22,7 @@ from pipelex.system.pipe_run_mode import PipeRunModeclass TestSearchGenerateDryBranch:def _assignment(self, *, run_mode: PipeRunMode) -> SearchAssignment:return SearchAssignment(- job_metadata=JobMetadata(user_id="u", pipeline_run_id="run_search_dry"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_search_dry"),cogt_run_params=CogtRunParams(run_mode=run_mode),query="what is pipelex?",search_setting=SearchSetting(model="mock-search-handle"),diff --git a/tests/unit/pipelex/cogt/content_generation/test_templating_generate_dry_branch.py b/tests/unit/pipelex/cogt/content_generation/test_templating_generate_dry_branch.pyindex e55d401f5..d178e0cb2 100644--- a/tests/unit/pipelex/cogt/content_generation/test_templating_generate_dry_branch.py+++ b/tests/unit/pipelex/cogt/content_generation/test_templating_generate_dry_branch.py@@ -20,7 +20,7 @@ from pipelex.tools.jinja2.template_category import TemplateCategoryclass TestTemplatingGenerateDryBranch:def _assignment(self, *, run_mode: PipeRunMode, template: str) -> TemplatingAssignment:return TemplatingAssignment(- job_metadata=JobMetadata(user_id="u", pipeline_run_id="run_templating_dry"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_templating_dry"),cogt_run_params=CogtRunParams(run_mode=run_mode),context={"the_answer": "42"},template=template,diff --git a/tests/unit/pipelex/cogt/extract/test_extract_worker_usage_capture.py b/tests/unit/pipelex/cogt/extract/test_extract_worker_usage_capture.pyindex 882a1deb9..af0e87029 100644--- a/tests/unit/pipelex/cogt/extract/test_extract_worker_usage_capture.py+++ b/tests/unit/pipelex/cogt/extract/test_extract_worker_usage_capture.py@@ -65,7 +65,7 @@ def _make_extract_job() -> ExtractJob:job_params=ExtractJobParams.make_default_extract_job_params(),job_config=ExtractJobConfig(),job_report=ExtractJobReport(),- job_metadata=JobMetadata(user_id="test-user", pipeline_run_id="test-run"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"),)diff --git a/tests/unit/pipelex/cogt/img_gen/conftest.py b/tests/unit/pipelex/cogt/img_gen/conftest.pyindex 56ee9a2e7..23f77f258 100644--- a/tests/unit/pipelex/cogt/img_gen/conftest.py+++ b/tests/unit/pipelex/cogt/img_gen/conftest.py@@ -62,5 +62,5 @@ def make_img_gen_job(),job_config=ImgGenJobConfig(is_sync_mode=False),job_report=ImgGenJobReport(),- job_metadata=JobMetadata(user_id="test-user", pipeline_run_id="test-run"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"),)diff --git a/tests/unit/pipelex/cogt/img_gen/test_img_gen_args_factory.py b/tests/unit/pipelex/cogt/img_gen/test_img_gen_args_factory.pyindex cb3926644..226f9bd21 100644--- a/tests/unit/pipelex/cogt/img_gen/test_img_gen_args_factory.py+++ b/tests/unit/pipelex/cogt/img_gen/test_img_gen_args_factory.py@@ -81,7 +81,7 @@ class TestImgGenArgsFactory:),job_config=ImgGenJobConfig(is_sync_mode=False),job_report=ImgGenJobReport(),- job_metadata=JobMetadata(user_id="test-user", pipeline_run_id="test-run"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"),)@staticmethoddiff --git a/tests/unit/pipelex/cogt/usage/test_data.py b/tests/unit/pipelex/cogt/usage/test_data.pyindex 86515377b..3b7233380 100644--- a/tests/unit/pipelex/cogt/usage/test_data.py+++ b/tests/unit/pipelex/cogt/usage/test_data.py@@ -36,6 +36,7 @@ class UsageFixtures:def full_job_metadata(*, unit_job_id: UnitJobId, job_category: JobCategory) -> JobMetadata:"""A JobMetadata with EVERY optional field populated, so leak tests prove the trim."""return JobMetadata(+ storage_scope="test/scope",user_id="user-42",pipeline_run_id="plr-fixture",pipe_code="analyze_contract",diff --git a/tests/unit/pipelex/kernel/test_kernel_boot_contract.py b/tests/unit/pipelex/kernel/test_kernel_boot_contract.pyindex 2cbcb3924..0f5f32397 100644--- a/tests/unit/pipelex/kernel/test_kernel_boot_contract.py+++ b/tests/unit/pipelex/kernel/test_kernel_boot_contract.py@@ -169,7 +169,7 @@ _KERNEL_CALL_SCRIPT = textwrap.dedent(RuntimeBoot.make(integration_mode=IntegrationMode.PYTEST, needs_inference=False)- kernel = PipelexKernel.make(run_mode=PipeRunMode.DRY, user_id="kernel-boot-contract")+ kernel = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="kernel-boot-contract")model = LLMSetting(model="kernel-boot-contract-model", temperature=0.5)text_concept = ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT)diff --git a/tests/unit/pipelex/kernel/test_llm_object_kernel.py b/tests/unit/pipelex/kernel/test_llm_object_kernel.pyindex c52acbfaa..40cb5d18e 100644--- a/tests/unit/pipelex/kernel/test_llm_object_kernel.py+++ b/tests/unit/pipelex/kernel/test_llm_object_kernel.py@@ -45,7 +45,7 @@ RESULT_NAME = "character"def _make_dry_kernel() -> PipelexKernel:- return PipelexKernel.make(run_mode=PipeRunMode.DRY, user_id="kernel-unit-test")+ return PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="kernel-unit-test")class TestLlmObjectKernel:diff --git a/tests/unit/pipelex/kernel/test_llm_text_kernel.py b/tests/unit/pipelex/kernel/test_llm_text_kernel.pyindex 4f6145d67..14a9d1fd3 100644--- a/tests/unit/pipelex/kernel/test_llm_text_kernel.py+++ b/tests/unit/pipelex/kernel/test_llm_text_kernel.py@@ -40,7 +40,7 @@ class MeetingSummary(TextContent):def _make_dry_kernel() -> PipelexKernel:- return PipelexKernel.make(run_mode=PipeRunMode.DRY, user_id="kernel-unit-test")+ return PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="kernel-unit-test")class TestLlmTextKernel:diff --git a/tests/unit/pipelex/kernel/test_pipelex_kernel_run_state.py b/tests/unit/pipelex/kernel/test_pipelex_kernel_run_state.pyindex 90109a595..e2a40a71c 100644--- a/tests/unit/pipelex/kernel/test_pipelex_kernel_run_state.py+++ b/tests/unit/pipelex/kernel/test_pipelex_kernel_run_state.py@@ -55,22 +55,22 @@ class TestPipelexKernelRunState:"""trace_context = _trace_context()- kernel = PipelexKernel.make(run_mode=PipeRunMode.DRY, user_id="test-user", trace_context=trace_context)+ kernel = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user", trace_context=trace_context)assert kernel.job_metadata.pipeline_run_id == _GRAPH_IDassert kernel.job_metadata.trace_context == trace_contextdef test_without_a_trace_context_the_run_mints_its_own_id_and_traces_nothing(self) -> None:"""The default stays what it was: a fresh run id, and no context for the leaf to emit against."""- first = PipelexKernel.make(run_mode=PipeRunMode.DRY, user_id="test-user")- second = PipelexKernel.make(run_mode=PipeRunMode.DRY, user_id="test-user")+ first = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user")+ second = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user")assert first.job_metadata.trace_context is Noneassert first.job_metadata.pipeline_run_id != second.job_metadata.pipeline_run_iddef test_each_step_inherits_the_trace_context_and_carries_its_own_run_id(self) -> None:"""The per-step copy: same trace context (so every step attributes to it), distinct pipe_run_id."""- kernel = PipelexKernel.make(run_mode=PipeRunMode.DRY, user_id="test-user", trace_context=_trace_context())+ kernel = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user", trace_context=_trace_context())first_step = kernel.make_step_metadata()second_step = kernel.make_step_metadata()@@ -88,7 +88,7 @@ class TestPipelexKernelRunState:Anything that attributes work by `pipe_code` — log correlation, usage accounting, theper-step labelling a distributed backend derives — sees an anonymous step without this."""- kernel = PipelexKernel.make(run_mode=PipeRunMode.DRY, user_id="test-user")+ kernel = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user")assert kernel.make_step_metadata(pipe_code="answer_question").pipe_code == "answer_question"@@ -100,7 +100,7 @@ class TestPipelexKernelRunState:key must be omitted, not passed as None — so this constructs a run that HAS a run-level`pipe_code`, which no caller in the tree does today."""- kernel = PipelexKernel.make(run_mode=PipeRunMode.DRY, user_id="test-user")+ kernel = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user")kernel.job_metadata.pipe_code = "set_at_run_level"assert kernel.make_step_metadata().pipe_code == "set_at_run_level"@@ -120,12 +120,12 @@ class TestPipelexKernelRunState:minted.append(f"step-{len(minted)}")return minted[-1]- kernel = PipelexKernel.make(run_mode=PipeRunMode.DRY, user_id="test-user", step_id_source=_source)+ kernel = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user", step_id_source=_source)assert kernel.make_step_metadata().pipe_run_id == "step-0"assert kernel.make_step_metadata().pipe_run_id == "step-1"- default_kernel = PipelexKernel.make(run_mode=PipeRunMode.DRY, user_id="test-user")+ default_kernel = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user")first_default = default_kernel.make_step_metadata().pipe_run_idassert first_default is not Noneassert UUID(first_default).version == 4@@ -133,7 +133,7 @@ class TestPipelexKernelRunState:def test_mock_usage_rides_the_execution_mode_contract(self) -> None:"""The DRY sub-flag lands on the carrier every cogt leaf reads off its assignment."""- kernel = PipelexKernel.make(run_mode=PipeRunMode.DRY, user_id="test-user", is_mock_usage=True)+ kernel = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user", is_mock_usage=True)assert kernel.cogt_run_params.run_mode.is_dryassert kernel.cogt_run_params.is_mock_usagediff --git a/tests/unit/pipelex/pipe_machinery/test_run_pipe_tracer_metadata.py b/tests/unit/pipelex/pipe_machinery/test_run_pipe_tracer_metadata.pyindex 20a47831e..c1d12e760 100644--- a/tests/unit/pipelex/pipe_machinery/test_run_pipe_tracer_metadata.py+++ b/tests/unit/pipelex/pipe_machinery/test_run_pipe_tracer_metadata.py@@ -66,6 +66,7 @@ class TestRunPipeForwardsTracerMetadata:)job_metadata = JobMetadata(+ storage_scope="test/scope",user_id="pytest",pipeline_run_id="test-run-meta",trace_context=make_trace_context(graph_id="test-run-meta"),diff --git a/tests/unit/pipelex/pipe_operators/pipe_func/test_direct_executor_workdir.py b/tests/unit/pipelex/pipe_operators/pipe_func/test_direct_executor_workdir.pyindex 4d423dc00..678fdb64e 100644--- a/tests/unit/pipelex/pipe_operators/pipe_func/test_direct_executor_workdir.py+++ b/tests/unit/pipelex/pipe_operators/pipe_func/test_direct_executor_workdir.py@@ -63,7 +63,7 @@ class TestDirectExecutorWorkdir:working_memory_raw={},pipe_code="missing_pipe",function_name="missing_func",- job_metadata=JobMetadata(user_id="user", pipeline_run_id="run"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="user", pipeline_run_id="run"),pipe_run_params=PipeRunParams(run_mode=PipeRunMode.DRY, pipe_stack_limit=10, batch_max_concurrency=None),)@@ -88,7 +88,7 @@ class TestDirectExecutorWorkdir:working_memory_raw={},pipe_code="missing_pipe",function_name="missing_func",- job_metadata=JobMetadata(user_id="user", pipeline_run_id="run"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="user", pipeline_run_id="run"),pipe_run_params=PipeRunParams(run_mode=PipeRunMode.DRY, pipe_stack_limit=10, batch_max_concurrency=None),)@@ -140,7 +140,7 @@ class TestDirectExecutorWorkdir:# its lookup is strict, so a bare code would not resolve on the far side.pipe_code="greet_demo.greet",function_name="greet_it",- job_metadata=JobMetadata(user_id="user", pipeline_run_id="run"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="user", pipeline_run_id="run"),pipe_run_params=PipeRunParams(run_mode=PipeRunMode.DRY, pipe_stack_limit=10, batch_max_concurrency=None),)diff --git a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_execution_transport.py b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_execution_transport.pyindex 0c186f59d..201a553a1 100644--- a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_execution_transport.py+++ b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_execution_transport.py@@ -46,7 +46,7 @@ class TestPipeFuncExecutionTransport:working_memory_raw={},pipe_code="my_pipe",function_name="my_func",- job_metadata=JobMetadata(user_id="user", pipeline_run_id="run"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="user", pipeline_run_id="run"),pipe_run_params=PipeRunParams(run_mode=PipeRunMode.DRY, pipe_stack_limit=10, batch_max_concurrency=None),timeout_seconds=float("inf"),)diff --git a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_hosted_mode.py b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_hosted_mode.pyindex b7bf45543..ebb9d556b 100644--- a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_hosted_mode.py+++ b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_hosted_mode.py@@ -88,7 +88,7 @@ class TestPipeFuncHostedMode:)pipe_output = await pipe_func._dry_run_operator_pipe( # noqa: SLF001 # pyright: ignore[reportPrivateUsage]- job_metadata=JobMetadata(user_id="user", pipeline_run_id="run"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="user", pipeline_run_id="run"),working_memory=WorkingMemoryFactory.make_empty(),pipe_run_params=PipeRunParams(run_mode=PipeRunMode.DRY, pipe_stack_limit=10, batch_max_concurrency=None),)diff --git a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_transported_ref.py b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_transported_ref.pyindex 0022236b9..0797a1b71 100644--- a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_transported_ref.py+++ b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_transported_ref.py@@ -56,7 +56,7 @@ class TestPipeFuncTransportedRef:mocker.patch("pipelex.pipe_operators.func.pipe_func.get_pipe_func_executor", return_value=executor)await the_pipe_func._live_run_operator_pipe( # noqa: SLF001 # pyright: ignore[reportPrivateUsage]- job_metadata=JobMetadata(user_id="user", pipeline_run_id="run"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="user", pipeline_run_id="run"),working_memory=WorkingMemoryFactory.make_empty(),pipe_run_params=PipeRunParams(run_mode=PipeRunMode.LIVE, pipe_stack_limit=10, batch_max_concurrency=None),)diff --git a/tests/unit/pipelex/pipe_operators/pipe_llm/test_prompt_rendering_purity.py b/tests/unit/pipelex/pipe_operators/pipe_llm/test_prompt_rendering_purity.pyindex d4eb6c8b4..6026128d1 100644--- a/tests/unit/pipelex/pipe_operators/pipe_llm/test_prompt_rendering_purity.py+++ b/tests/unit/pipelex/pipe_operators/pipe_llm/test_prompt_rendering_purity.py@@ -142,7 +142,7 @@ class TestPromptRenderingPurity:snapshot = pipe_llm.model_dump()await pipe_llm._dry_run_operator_pipe( # noqa: SLF001 # pyright: ignore[reportPrivateUsage]- job_metadata=JobMetadata(user_id="pytest", pipeline_run_id="test_prompt_rendering_purity"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="pytest", pipeline_run_id="test_prompt_rendering_purity"),working_memory=_make_working_memory(),pipe_run_params=PipeRunParams(run_mode=PipeRunMode.DRY, pipe_stack_limit=10, batch_max_concurrency=None),)diff --git a/tests/unit/pipelex/pipe_run/test_delivery_executor.py b/tests/unit/pipelex/pipe_run/test_delivery_executor.pyindex 537472178..089205225 100644--- a/tests/unit/pipelex/pipe_run/test_delivery_executor.py+++ b/tests/unit/pipelex/pipe_run/test_delivery_executor.py@@ -69,7 +69,7 @@ class TestDeliveryExecutor:await executor.execute(pipe_output=mock_output,- user_id="test-user",+ storage_scope="tenant/plr-123",pipeline_run_id="plr-123",delivery_assignment=assignment,status=DeliveryStatus.COMPLETED,@@ -77,12 +77,49 @@ class TestDeliveryExecutor:mock_storage.store.assert_called()stored_keys = [call.kwargs["key"] for call in mock_storage.store.call_args_list]- assert any("test-user/plr-123/working_memory.json" in key for key in stored_keys)+ assert any("tenant/plr-123/results/working_memory.json" in key for key in stored_keys)# A completed run always delivers a main stuff, so the main_stuff artifact files are always written.- assert any("test-user/plr-123/main_stuff.json" in key for key in stored_keys)- assert any("test-user/plr-123/main_stuff.md" in key for key in stored_keys)- assert any("test-user/plr-123/main_stuff.html" in key for key in stored_keys)- assert any("test-user/plr-123/tokens_usages.json" in key for key in stored_keys)+ assert any("tenant/plr-123/results/main_stuff.json" in key for key in stored_keys)+ assert any("tenant/plr-123/results/main_stuff.md" in key for key in stored_keys)+ assert any("tenant/plr-123/results/main_stuff.html" in key for key in stored_keys)+ assert any("tenant/plr-123/results/tokens_usages.json" in key for key in stored_keys)++ async def test_key_prefix_inserts_a_level_and_never_supplies_the_leaf(self, mocker: MockerFixture) -> None:+ """`results/` is the RUNTIME's leaf; `key_prefix` sits before it, never instead of it.++ A caller that passed `key_prefix="results"` — reading it as "the leaf" —+ got `<scope>/results/results/`, because this executor appends its own.+ That is a valid, stable, completely wrong location, and nothing fails:+ the write succeeds, the run reports COMPLETED, and only whoever later+ reads `<scope>/results/` finds an empty prefix. `pipelex-api` shipped+ exactly that, which is what this test pins down.+ """+ mock_storage = mocker.AsyncMock()+ mock_storage.store = mocker.AsyncMock(return_value="pipelex-storage://test-key")+ mock_storage.public_url = mocker.Mock(return_value="file:///tmp/results/plr-123")+ mocker.patch("pipelex.pipe_run.delivery_executor.get_storage_provider", return_value=mock_storage)++ mock_output = _make_output_mock(mocker)+ mock_output.working_memory_raw = None+ mock_output.working_memory.smart_dump.return_value = {"root": {}, "aliases": {}}+ mock_output.working_memory.resolve_main_stuff.return_value = _make_main_stuff()+ mock_output.graph_spec = None++ await DeliveryExecutor().execute(+ pipe_output=mock_output,+ storage_scope="tenant/plr-123",+ pipeline_run_id="plr-123",+ delivery_assignment=DeliveryAssignment(storage=StorageTarget(key_prefix="attempt-2")),+ status=DeliveryStatus.COMPLETED,+ )++ stored_keys = [call.kwargs["key"] for call in mock_storage.store.call_args_list]+ assert stored_keys+ for key in stored_keys:+ # The prefix is an EXTRA level between the scope and the leaf, and the+ # leaf appears exactly once however the caller spells the prefix.+ assert key.startswith("tenant/plr-123/attempt-2/results/")+ assert key.count("/results/") == 1async def test_execute_webhook_only(self, mocker: MockerFixture) -> None:mock_client = mocker.AsyncMock()@@ -100,7 +137,7 @@ class TestDeliveryExecutor:await executor.execute(pipe_output=None,- user_id="test-user",+ storage_scope="test-user",pipeline_run_id="plr-456",delivery_assignment=assignment,status=DeliveryStatus.COMPLETED,@@ -131,7 +168,7 @@ class TestDeliveryExecutor:await executor.execute(pipe_output=None,- user_id="test-user",+ storage_scope="test-user",pipeline_run_id="plr-789",delivery_assignment=assignment,status=DeliveryStatus.FAILED,@@ -151,7 +188,7 @@ class TestDeliveryExecutor:await executor.execute(pipe_output=None,- user_id="test-user",+ storage_scope="test-user",pipeline_run_id="plr-fail",delivery_assignment=assignment,status=DeliveryStatus.FAILED,@@ -166,7 +203,7 @@ class TestDeliveryExecutor:await executor.execute(pipe_output=None,- user_id="test-user",+ storage_scope="test-user",pipeline_run_id="plr-noop",delivery_assignment=assignment,status=DeliveryStatus.COMPLETED,@@ -189,7 +226,7 @@ class TestDeliveryExecutor:with pytest.raises(StorageDeliveryError):await executor.execute(pipe_output=mock_output,- user_id="test-user",+ storage_scope="test-user",pipeline_run_id="plr-err",delivery_assignment=assignment,status=DeliveryStatus.COMPLETED,@@ -288,7 +325,7 @@ class TestDeliveryExecutor:mock_output.tokens_usages = [LLMTokensUsage(model_type="llm",- job_metadata=JobMetadata(user_id="test-user", pipeline_run_id="plr-usage"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="plr-usage"),inference_model_name="test-model",inference_model_id="test-model-id",nb_tokens_by_category={TokenCategory.INPUT: 15, TokenCategory.OUTPUT: 4},@@ -528,7 +565,7 @@ class TestDeliveryExecutor:await executor.execute(pipe_output=None,- user_id="test-user",+ storage_scope="test-user",pipeline_run_id="plr-failed",delivery_assignment=assignment,status=DeliveryStatus.FAILED,@@ -556,7 +593,7 @@ class TestDeliveryExecutor:await executor.execute(pipe_output=None,- user_id="test-user",+ storage_scope="test-user",pipeline_run_id="plr-success",delivery_assignment=assignment,status=DeliveryStatus.COMPLETED,@@ -587,7 +624,7 @@ class TestDeliveryExecutor:await executor.execute(pipe_output=None,- user_id="test-user",+ storage_scope="test-user",pipeline_run_id="plr-failed-no-report",delivery_assignment=assignment,status=DeliveryStatus.FAILED,@@ -619,7 +656,7 @@ class TestDeliveryExecutor:await executor.execute(pipe_output=mock_output,- user_id="test-user",+ storage_scope="test-user",pipeline_run_id="plr-storage-req",delivery_assignment=assignment,status=DeliveryStatus.COMPLETED,@@ -650,7 +687,7 @@ class TestDeliveryExecutor:await executor.execute(pipe_output=None,- user_id="test-user",+ storage_scope="test-user",pipeline_run_id="plr-webhook-req",delivery_assignment=assignment,status=DeliveryStatus.COMPLETED,@@ -707,7 +744,7 @@ class TestDeliveryExecutor:await executor.execute(pipe_output=None,- user_id="test-user",+ storage_scope="test-user",pipeline_run_id="plr-webhook-failed-req",delivery_assignment=assignment,status=DeliveryStatus.FAILED,@@ -739,7 +776,7 @@ class TestDeliveryExecutor:await executor.execute(pipe_output=None,- user_id="test-user",+ storage_scope="test-user",pipeline_run_id="plr-no-req",delivery_assignment=assignment,status=DeliveryStatus.COMPLETED,@@ -769,7 +806,7 @@ class TestDeliveryExecutor:with pytest.raises(SsrfBlockedError):await executor.execute(pipe_output=None,- user_id="test-user",+ storage_scope="test-user",pipeline_run_id="plr-ssrf",delivery_assignment=assignment,status=DeliveryStatus.COMPLETED,@@ -792,7 +829,7 @@ class TestDeliveryExecutor:with pytest.raises(WebhookDeliveryError):await executor.execute(pipe_output=None,- user_id="test-user",+ storage_scope="test-user",pipeline_run_id="plr-err",delivery_assignment=assignment,status=DeliveryStatus.COMPLETED,diff --git a/tests/unit/pipelex/pipe_run/test_pipe_job_hydration.py b/tests/unit/pipelex/pipe_run/test_pipe_job_hydration.pyindex 65035ee79..59541874b 100644--- a/tests/unit/pipelex/pipe_run/test_pipe_job_hydration.py+++ b/tests/unit/pipelex/pipe_run/test_pipe_job_hydration.py@@ -23,7 +23,7 @@ def _make_pipe_job(working_memory=working_memory,working_memory_raw=None,pipe_run_params=PipeRunParamsFactory.make_run_params(),- job_metadata=JobMetadata(user_id="test-user", pipeline_run_id="test-run"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"),output_name=None,library_crate=library_crate,)diff --git a/tests/unit/pipelex/pipe_run/test_pipe_router_run.py b/tests/unit/pipelex/pipe_run/test_pipe_router_run.pyindex d58bd43a3..26d8cbd22 100644--- a/tests/unit/pipelex/pipe_run/test_pipe_router_run.py+++ b/tests/unit/pipelex/pipe_run/test_pipe_router_run.py@@ -44,7 +44,7 @@ def _make_pipe_job() -> PipeJob:working_memory=None,working_memory_raw=None,pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.LIVE),- job_metadata=JobMetadata(user_id="test-user", pipeline_run_id="test-run"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"),output_name=None,library_crate=None,)diff --git a/tests/unit/pipelex/pipe_run/test_run_batch_branch_hook.py b/tests/unit/pipelex/pipe_run/test_run_batch_branch_hook.pyindex 2424a8e55..23a60140c 100644--- a/tests/unit/pipelex/pipe_run/test_run_batch_branch_hook.py+++ b/tests/unit/pipelex/pipe_run/test_run_batch_branch_hook.py@@ -52,7 +52,7 @@ def _make_pipe_job() -> PipeJob:working_memory=None,working_memory_raw=None,pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.LIVE),- job_metadata=JobMetadata(user_id="test-user", pipeline_run_id="test-run"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"),output_name=None,library_crate=None,)diff --git a/tests/unit/pipelex/pipe_run/test_tracing_assembly.py b/tests/unit/pipelex/pipe_run/test_tracing_assembly.pyindex d5323e76d..82d129bfb 100644--- a/tests/unit/pipelex/pipe_run/test_tracing_assembly.py+++ b/tests/unit/pipelex/pipe_run/test_tracing_assembly.py@@ -23,7 +23,7 @@ _MODULE = "pipelex.pipe_run.tracing_assembly"def _make_usage_event(pipeline_run_id: str, node_id: str) -> UsageReportEvent:"""A UsageReportEvent carrying a real LLMTokensUsage (so the aggregator returns it)."""- metadata = JobMetadata(user_id="tracing-assembly-test", pipeline_run_id=pipeline_run_id)+ metadata = JobMetadata(storage_scope="test/scope", user_id="tracing-assembly-test", pipeline_run_id=pipeline_run_id)tokens_usage = LLMTokensUsage(job_metadata=metadata,inference_model_name="fake_model",diff --git a/tests/unit/pipelex/pipeline/test_input_normalizer.py b/tests/unit/pipelex/pipeline/test_input_normalizer.pyindex 7e2e265b0..d41a90b8a 100644--- a/tests/unit/pipelex/pipeline/test_input_normalizer.py+++ b/tests/unit/pipelex/pipeline/test_input_normalizer.py@@ -50,7 +50,7 @@ class TestInputNormalizerUrlGuards:_patch_storage_and_config(mocker)with pytest.raises(PipelineInputContentError, match="blank url"):- await normalize_data_urls_to_storage(_memory_with_document(blank_url))+ await normalize_data_urls_to_storage(_memory_with_document(blank_url), storage_scope="test/scope")async def test_directory_path_raises_input_error(self, mocker: MockerFixture, tmp_path: Path) -> None:"""A local path that IS a directory must be an input error, not an@@ -59,10 +59,10 @@ class TestInputNormalizerUrlGuards:_patch_storage_and_config(mocker)with pytest.raises(PipelineInputContentError, match="cannot be read"):- await normalize_data_urls_to_storage(_memory_with_document(str(tmp_path)))+ await normalize_data_urls_to_storage(_memory_with_document(str(tmp_path)), storage_scope="test/scope")async def test_missing_file_raises_input_error(self, mocker: MockerFixture, tmp_path: Path) -> None:_patch_storage_and_config(mocker)with pytest.raises(PipelineInputContentError, match="cannot be read"):- await normalize_data_urls_to_storage(_memory_with_document(str(tmp_path / "nope.pdf")))+ await normalize_data_urls_to_storage(_memory_with_document(str(tmp_path / "nope.pdf")), storage_scope="test/scope")diff --git a/tests/unit/pipelex/providers/openai/test_openai_completions_usage_capture.py b/tests/unit/pipelex/providers/openai/test_openai_completions_usage_capture.pyindex befd21237..0d2038ed1 100644--- a/tests/unit/pipelex/providers/openai/test_openai_completions_usage_capture.py+++ b/tests/unit/pipelex/providers/openai/test_openai_completions_usage_capture.py@@ -32,7 +32,7 @@ if TYPE_CHECKING:def _make_llm_job_with_empty_usage() -> LLMJob:"""Build an LLMJob whose tokens_usage is seeded empty, mirroring ``llm_job_before_start``."""- job_metadata = JobMetadata(user_id="test-user", pipeline_run_id="test-run")+ job_metadata = JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run")tokens_usage = LLMTokensUsage(job_metadata=job_metadata,inference_model_name="test-model",diff --git a/tests/unit/pipelex/providers/openai/test_openai_img_gen_worker_usage_capture.py b/tests/unit/pipelex/providers/openai/test_openai_img_gen_worker_usage_capture.pyindex fa594f0ca..76debcee7 100644--- a/tests/unit/pipelex/providers/openai/test_openai_img_gen_worker_usage_capture.py+++ b/tests/unit/pipelex/providers/openai/test_openai_img_gen_worker_usage_capture.py@@ -70,7 +70,7 @@ def _make_img_gen_job() -> ImgGenJob:),job_config=ImgGenJobConfig(is_sync_mode=False),job_report=ImgGenJobReport(),- job_metadata=JobMetadata(user_id="test-user", pipeline_run_id="test-run"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"),)diff --git a/tests/unit/pipelex/providers/openai/test_openai_responses_usage_capture.py b/tests/unit/pipelex/providers/openai/test_openai_responses_usage_capture.pyindex b94bd9d9f..8b08d2f28 100644--- a/tests/unit/pipelex/providers/openai/test_openai_responses_usage_capture.py+++ b/tests/unit/pipelex/providers/openai/test_openai_responses_usage_capture.py@@ -30,7 +30,7 @@ if TYPE_CHECKING:def _make_llm_job_with_empty_usage() -> LLMJob:- job_metadata = JobMetadata(user_id="test-user", pipeline_run_id="test-run")+ job_metadata = JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run")tokens_usage = LLMTokensUsage(job_metadata=job_metadata,inference_model_name="test-model",diff --git a/tests/unit/pipelex/reporting/test_emit_runner_fallback.py b/tests/unit/pipelex/reporting/test_emit_runner_fallback.pyindex c92d6b614..c40ba2131 100644--- a/tests/unit/pipelex/reporting/test_emit_runner_fallback.py+++ b/tests/unit/pipelex/reporting/test_emit_runner_fallback.py@@ -55,6 +55,7 @@ def _make_llm_job() -> LLMJob:now = datetime.now(UTC)job_metadata = JobMetadata(+ storage_scope="test/scope",user_id="test_user",pipeline_run_id=pipeline_run_id,trace_context=trace_context,diff --git a/tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py b/tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.pyindex 3ce24415b..886d0a10e 100644--- a/tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py+++ b/tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py@@ -80,6 +80,7 @@ def _trace_context(run_id: str, tracer_key: str) -> TraceContext:def _job_metadata(run_id: str, tracer_key: str) -> JobMetadata:now = datetime.now(UTC)return JobMetadata(+ storage_scope="test/scope",user_id="test-user",pipeline_run_id=run_id,trace_context=_trace_context(run_id, tracer_key),diff --git a/tests/unit/pipelex/reporting/test_emit_usage_event_gating.py b/tests/unit/pipelex/reporting/test_emit_usage_event_gating.pyindex 03ef1a5f4..22adbce98 100644--- a/tests/unit/pipelex/reporting/test_emit_usage_event_gating.py+++ b/tests/unit/pipelex/reporting/test_emit_usage_event_gating.py@@ -40,6 +40,7 @@ DATA_INCLUSION_OFF = DataInclusionConfig(def _make_llm_job(pipeline_run_id: str, trace_context: TraceContext | None) -> LLMJob:now = datetime.now(UTC)job_metadata = JobMetadata(+ storage_scope="test/scope",user_id="test_user",pipeline_run_id=pipeline_run_id,trace_context=trace_context,diff --git a/tests/unit/pipelex/reporting/test_reporting_event_emission.py b/tests/unit/pipelex/reporting/test_reporting_event_emission.pyindex cd2488c34..e56cafb43 100644--- a/tests/unit/pipelex/reporting/test_reporting_event_emission.py+++ b/tests/unit/pipelex/reporting/test_reporting_event_emission.py@@ -41,6 +41,7 @@ def _make_test_llm_job("""Create a minimal LLMJob for testing event emission."""now = datetime.now(UTC)job_metadata = JobMetadata(+ storage_scope="test/scope",user_id="test_user",pipeline_run_id=pipeline_run_id,trace_context=trace_context,diff --git a/tests/unit/pipelex/runtime_bridge/test_direct_orchestrator.py b/tests/unit/pipelex/runtime_bridge/test_direct_orchestrator.pyindex 3e26e0ccb..ee6f386f5 100644--- a/tests/unit/pipelex/runtime_bridge/test_direct_orchestrator.py+++ b/tests/unit/pipelex/runtime_bridge/test_direct_orchestrator.py@@ -25,7 +25,7 @@ def _make_pipe_job() -> PipeJob:working_memory=None,working_memory_raw=None,pipe_run_params=PipeRunParamsFactory.make_run_params(),- job_metadata=JobMetadata(user_id="test-user", pipeline_run_id="test-run"),+ job_metadata=JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"),output_name=None,library_crate=None,)diff --git a/tests/unit/pipelex/runtime_bridge/test_input_models.py b/tests/unit/pipelex/runtime_bridge/test_input_models.pyindex 1f706b553..1c43cfb1d 100644--- a/tests/unit/pipelex/runtime_bridge/test_input_models.py+++ b/tests/unit/pipelex/runtime_bridge/test_input_models.py@@ -7,12 +7,17 @@ from pipelex.runtime_bridge.payloads import PipelexPipeDispatchAck, PipelexPipeRclass TestInputOutputModels:def test_input_defaults_match_design(self):- payload = PipelexPipeRunInput(pipe_code="some_pipe")+ payload = PipelexPipeRunInput(storage_scope="test/scope", user_id="test-user", pipe_code="some_pipe")assert payload.pipe_code == "some_pipe"assert payload.inputs == {}assert payload.output_name is Noneassert payload.pipeline_run_id is None- assert payload.user_id is None+ # `user_id` and `storage_scope` are REQUIRED and non-nullable — this is+ # the wire boundary where a missing identity used to become "anonymous"+ # and a missing scope used to be derived from it, pointing every such run+ # at one shared namespace.+ assert payload.user_id == "test-user"+ assert payload.storage_scope == "test/scope"assert payload.library_crate_dump is Noneassert payload.orchestration_mode == "direct"assert payload.delivery is DeliveryMode.BLOCKING@@ -33,6 +38,7 @@ class TestInputOutputModels:def test_input_round_trip_via_json(self):original = PipelexPipeRunInput(+ storage_scope="test/scope",pipe_code="some_pipe",inputs={"foo": "bar"},orchestration_mode="temporal",@@ -111,3 +117,34 @@ class TestInputOutputModels:"output_dict": {},})+++class TestTheWireRefusesAnUnusableScope:+ """A bridge payload crosses a process boundary, so its scope is whatever the other side sent.++ Requiring the field is not the same as validating it: `storage_scope: str`+ accepted `../other/run` happily, and the value then travelled until+ something pasted it into a storage key. Validating at construction makes a+ traversal a payload-decoding error naming the field, at the edge, before any+ storage call — which is the same reason `user_id` is required here.+ """++ @pytest.mark.parametrize(+ "bad_scope",+ [+ pytest.param("../other/run", id="traversal-leading"),+ pytest.param("tenant/../other", id="traversal-interior"),+ pytest.param("/tenant/run", id="leading-slash-absolute-key"),+ pytest.param("tenant/run/", id="trailing-slash-empty-final-segment"),+ pytest.param("", id="empty"),+ pytest.param("a/b/c/d", id="four-segments-would-swallow-the-leaf"),+ pytest.param("tenant/run\n", id="trailing-newline"),+ ],+ )+ def test_an_unsafe_scope_is_refused_at_construction(self, bad_scope: str):+ with pytest.raises(ValidationError):+ PipelexPipeRunInput(storage_scope=bad_scope, user_id="test-user", pipe_code="some_pipe")++ def test_a_usable_scope_still_passes(self):+ payload = PipelexPipeRunInput(storage_scope="org_a/mt_b/run_c", user_id="test-user", pipe_code="some_pipe")+ assert payload.storage_scope == "org_a/mt_b/run_c"diff --git a/tests/unit/pipelex/system/test_job_metadata_request_id.py b/tests/unit/pipelex/system/test_job_metadata_request_id.pyindex a76264caf..a3cfd6b60 100644--- a/tests/unit/pipelex/system/test_job_metadata_request_id.py+++ b/tests/unit/pipelex/system/test_job_metadata_request_id.py@@ -11,7 +11,7 @@ from pipelex.system.job_metadata import JobMetadataclass TestJobMetadataRequestId:def test_request_id_defaults_to_none(self) -> None:"""``JobMetadata`` constructed without ``request_id`` carries ``None``."""- meta = JobMetadata(user_id="u", pipeline_run_id="r")+ meta = JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="r")assert meta.request_id is Nonedef test_request_id_round_trips_through_json(self) -> None:@@ -20,14 +20,14 @@ class TestJobMetadataRequestId:Temporal's data converter serializes activity args as JSON; this pinsthe contract that ``request_id`` reaches the worker intact."""- meta = JobMetadata(user_id="u", pipeline_run_id="r", request_id="r-abc-123")+ meta = JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="r", request_id="r-abc-123")recovered = JobMetadata.model_validate_json(meta.model_dump_json())assert recovered.request_id == "r-abc-123"assert recovered == metadef test_copy_with_update_preserves_request_id(self) -> None:"""``copy_with_update`` inherits ``request_id`` from the parent metadata."""- parent = JobMetadata(user_id="u", pipeline_run_id="r", request_id="r-1")+ parent = JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="r", request_id="r-1")child = parent.copy_with_update(otel_context=None)assert child.request_id == "r-1"@@ -52,7 +52,7 @@ class TestJobMetadataRequestId:``ErrorReport`` envelopes that quote it."""with pytest.raises(ValidationError):- JobMetadata(user_id="u", pipeline_run_id="r", request_id=bad_request_id)+ JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="r", request_id=bad_request_id)@pytest.mark.parametrize("good_request_id",@@ -65,5 +65,5 @@ class TestJobMetadataRequestId:)def test_request_id_accepts_well_formed_input(self, good_request_id: str) -> None:"""Common ``X-Request-ID`` shapes (UUID, hyphenated, dot-separated) round-trip cleanly."""- meta = JobMetadata(user_id="u", pipeline_run_id="r", request_id=good_request_id)+ meta = JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="r", request_id=good_request_id)assert meta.request_id == good_request_iddiff --git a/tests/unit/pipelex/system/test_storage_scope.py b/tests/unit/pipelex/system/test_storage_scope.pynew file mode 100644index 000000000..cfe5b965b--- /dev/null+++ b/tests/unit/pipelex/system/test_storage_scope.py@@ -0,0 +1,168 @@+"""`storage_scope` — the construction-time guard on where a run's bytes go.++These are the tests that matter most in the storage change. The scope is pasted+straight into a storage key prefix, so a `..` or a leading slash in it is a path+traversal out of the tenant's namespace. Before this field existed, the Temporal+payload codec sanitized `user_id` and `pipeline_run_id` per-segment; collapsing+them into one slash-bearing string made that sanitizer unusable, so without+these checks the change would have silently deleted a traversal control.+"""++from __future__ import annotations++import re+from pathlib import Path++import pytest+from pydantic import ValidationError++from pipelex.system.job_metadata import JobMetadata+from pipelex.system.storage_scope import (+ DRY_RUN_STORAGE_SCOPE,+ DRY_RUN_USER_ID,+ LOCAL_STORAGE_SCOPE,+ LOCAL_USER_ID,+ validate_storage_scope,+)+from pipelex.system.telemetry.otel_constants import OTelConstants+++def _metadata(storage_scope: str) -> JobMetadata:+ return JobMetadata(user_id="u1", pipeline_run_id="run_1", storage_scope=storage_scope)+++class TestAcceptedScopes:+ @pytest.mark.parametrize(+ "scope",+ [+ "tenant",+ "tenant/run_1",+ "org_acme/mt_x/run_1",+ "A-Z_az-09",+ DRY_RUN_STORAGE_SCOPE,+ LOCAL_STORAGE_SCOPE,+ ],+ )+ def test_one_to_three_path_safe_segments_are_accepted(self, scope: str) -> None:+ assert _metadata(scope).storage_scope == scope++ def test_the_two_named_constants_are_valid_scopes(self) -> None:+ """They are used as real field values, so they must satisfy the field.++ A constant that the validator rejects would turn every dry run into a+ `ValidationError` — and dry runs are the validation path, so the failure+ would surface as "your bundle is invalid".+ """+ assert validate_storage_scope(value=DRY_RUN_STORAGE_SCOPE)+ assert validate_storage_scope(value=LOCAL_STORAGE_SCOPE)+++class TestRefusedScopes:+ @pytest.mark.parametrize(+ "scope",+ [+ pytest.param("", id="empty"),+ pytest.param("/tenant/run", id="leading-slash-absolute-key"),+ pytest.param("tenant/run/", id="trailing-slash-empty-final-segment"),+ pytest.param("tenant//run", id="empty-interior-segment"),+ pytest.param("..", id="traversal-alone"),+ pytest.param("tenant/../other", id="traversal-interior"),+ pytest.param("../other/run", id="traversal-leading"),+ pytest.param(".", id="current-dir-segment"),+ pytest.param("tenant/./run", id="current-dir-interior"),+ pytest.param("a/b/c/d", id="four-segments-would-swallow-the-leaf"),+ pytest.param("tenant run", id="space"),+ pytest.param("tenant?x=1", id="query-string"),+ pytest.param("tenant#frag", id="fragment"),+ pytest.param("tenant\\run", id="backslash"),+ pytest.param("tenant%2f..%2f", id="percent-encoded-traversal"),+ # A `$`-anchored `re.match` admits ONE trailing newline, so these+ # passed the guard and the newline travelled into every storage key+ # and log line built from the scope — an unaddressable key and a log+ # forging primitive. `fullmatch` is what closes it; these pin it shut.+ pytest.param("tenant/run\n", id="trailing-newline"),+ pytest.param("tenant\n", id="single-segment-trailing-newline"),+ pytest.param("tenant\nrun", id="interior-newline"),+ pytest.param("tenant/run\r", id="trailing-carriage-return"),+ pytest.param("tenant/run\r\n", id="trailing-crlf"),+ pytest.param("\n", id="newline-alone"),+ ],+ )+ def test_unsafe_scopes_raise_at_construction(self, scope: str) -> None:+ """Refused when the object is BUILT, not when the key is written.++ The distinction is the whole design: a `JobMetadata` that exists is a+ `JobMetadata` whose scope is safe, so every key derived downstream is+ safe by construction rather than by each call site remembering.+ """+ with pytest.raises(ValidationError):+ _metadata(scope)++ def test_a_prefix_of_another_tenant_is_still_its_own_scope(self) -> None:+ """The validator checks SHAPE, never ownership — that is the host's job.++ Recorded so nobody mistakes this for the tenant boundary. `tenant2` is a+ perfectly valid scope even when the caller is `tenant`; refusing it+ requires knowing who is calling, which this layer deliberately does not.+ """+ assert _metadata("tenant2/run_1").storage_scope == "tenant2/run_1"+++class TestScopeIsRequired:+ def test_omitting_it_is_an_error_rather_than_a_default(self) -> None:+ """No default, deliberately.++ A default here is how the shared `anonymous/` namespace grew the first+ time: a placeholder that was never meant to reach storage became the key+ prefix for every run without an authenticated caller, and each tenant+ could read the others' outputs. A missing scope must be a loud failure+ at the call site, not a quiet shared prefix in production.+ """+ with pytest.raises(ValidationError):+ # Both type checkers reject this call, which is the point being+ # tested: the omission is caught statically AND at runtime. The+ # ignores are what let the runtime half be asserted at all.+ JobMetadata(user_id="u1", pipeline_run_id="run_1") # type: ignore[call-arg] # pyright: ignore[reportCallIssue]+++class TestTheTelemetryPlaceholderStaysOutOfIdentity:+ """`OTelConstants.DEFAULT_USER_ID` is `"anonymous"`, and it is TELEMETRY.++ It means "a span with no known caller". It leaked into the identity path,+ became the first segment of every storage key a run without an authenticated+ caller wrote, and put every such tenant into one namespace where each could+ read the others' outputs. The failure was silent by construction: a missing+ identity looked exactly like a present one.++ The constant survives because tracing genuinely has that concept. These tests+ are what keep it there — a future `user_id or DEFAULT_USER_ID` is the exact+ regression they exist to catch, and it would otherwise be invisible until+ someone read an S3 bucket listing.+ """++ def test_no_module_uses_it_as_an_identity_or_a_scope(self) -> None:+ source_root = Path(__file__).resolve().parents[4] / "pipelex"+ assert source_root.is_dir(), source_root++ offenders: list[str] = []+ for path in source_root.rglob("*.py"):+ for line_number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), start=1):+ if line.lstrip().startswith("#"):+ continue+ # The optional `: <annotation>` matters: a parameter default is written+ # `user_id: str = OTelConstants.DEFAULT_USER_ID`, and a pattern+ # without it passes a real regression while looking like a guard.+ if re.search(r"\b(user_id|storage_scope)\b\s*(?::[^=]+)?=\s*[\w.]*DEFAULT_USER_ID", line):+ offenders.append(f"{path.relative_to(source_root)}:{line_number}")++ assert not offenders, (+ "The telemetry placeholder is being bound as an identity or a storage scope at "+ f"{offenders}. It is the string 'anonymous'; using it here is how every "+ "unauthenticated run ended up sharing one storage namespace. Use DRY_RUN_USER_ID "+ "or LOCAL_USER_ID, or require the caller to supply one."+ )++ def test_the_dry_run_identity_is_not_the_telemetry_placeholder(self) -> None:+ """They mean different things and must not converge back onto one string."""+ assert DRY_RUN_USER_ID != OTelConstants.DEFAULT_USER_ID+ assert LOCAL_USER_ID != OTelConstants.DEFAULT_USER_IDdiff --git a/tests/unit/pipelex/tools/storage/test_storage_in_memory_config.py b/tests/unit/pipelex/tools/storage/test_storage_in_memory_config.pyindex 6f9925116..abbb304f3 100644--- a/tests/unit/pipelex/tools/storage/test_storage_in_memory_config.py+++ b/tests/unit/pipelex/tools/storage/test_storage_in_memory_config.py@@ -25,10 +25,8 @@ class TestStorageInMemoryConfig:pytest.param("assets/{hash[0]}", "- the {hash} placeholder must be plain", id="uri-hash-index-access"),# A format spec, conversion, or attribute/index access on any supported field is rejected uniformly# as "must be plain" (a str path component carrying one of these renders a constant/colliding key).- pytest.param("assets/{primary_id!x}/{hash}", "- uri_format placeholder '{primary_id}' must be plain", id="uri-conversion-on-field"),- pytest.param(- "assets/{primary_id.foo}/{hash}", "- uri_format placeholder '{primary_id.foo}' must be plain", id="uri-attr-access-on-field"- ),+ pytest.param("assets/{extension!x}/{hash}", "- uri_format placeholder '{extension}' must be plain", id="uri-conversion-on-field"),+ pytest.param("assets/{extension.foo}/{hash}", "- uri_format placeholder '{extension.foo}' must be plain", id="uri-attr-access-on-field"),pytest.param("assets/{extension[0]}/{hash}", "- uri_format placeholder '{extension[0]}' must be plain", id="uri-index-access-on-field"),],)diff --git a/tests/unit/pipelex/tools/storage/test_storage_s3_config.py b/tests/unit/pipelex/tools/storage/test_storage_s3_config.pyindex e48bd1495..abaeb9fe4 100644--- a/tests/unit/pipelex/tools/storage/test_storage_s3_config.py+++ b/tests/unit/pipelex/tools/storage/test_storage_s3_config.py@@ -38,35 +38,43 @@ class TestStorageS3Config:"assets/{hash}/{tenant}", "my-bucket", "eu-west-1", "- uri_format placeholder '{tenant}' is not supported", id="uri-unsupported"),pytest.param("assets/{hash:.0}", "my-bucket", "eu-west-1", "- the {hash} placeholder must be plain", id="uri-hash-format-spec"),- # A format spec on a non-hash field is just as dangerous: {primary_id:.0} truncates to "" (collisions),- # and a huge width like {primary_id:1000000000} would allocate a ~1GB string if it ever reached a- # test rendering — so it must be rejected at parse time, before any format() call.+ # A format spec on a non-hash field is just as dangerous: {extension:.0} truncates to "" (every+ # object loses its extension), and a huge width like {extension:1000000000} would allocate a ~1GB+ # string if it ever reached a test rendering — so it must be rejected at parse time, before any+ # format() call.pytest.param(- "assets/{hash}/{primary_id:.0}",+ "assets/{hash}/{extension:.0}","my-bucket","eu-west-1",- "- uri_format placeholder '{primary_id}' must be plain",+ "- uri_format placeholder '{extension}' must be plain",id="uri-field-truncate-spec",),pytest.param(- "assets/{hash}/{primary_id:1000000000}",+ "assets/{hash}/{extension:1000000000}","my-bucket","eu-west-1",- "- uri_format placeholder '{primary_id}' must be plain",+ "- uri_format placeholder '{extension}' must be plain",id="uri-field-huge-width",),pytest.param(- "assets/{hash}/{secondary_id!r}",+ # A CONVERSION on a SUPPORTED placeholder. It must use a name+ # the supported set still contains — it was `{secondary_id!r}`,+ # then `{extension!r}`, and BOTH were retired in turn: an+ # unsupported name is refused by a different check first, so the+ # case would go on passing for the wrong reason. Only `{hash}`+ # and `{extension}` remain, and `{hash}` has its own dedicated+ # plainness check, which leaves exactly one valid vehicle here.+ "assets/{hash}/{extension!r}","my-bucket","eu-west-1",- "- uri_format placeholder '{secondary_id}' must be plain",+ "- uri_format placeholder '{extension}' must be plain",id="uri-field-conversion",),pytest.param(- "assets/{hash}/{primary_id.foo}",+ "assets/{hash}/{extension.foo}","my-bucket","eu-west-1",- "- uri_format placeholder '{primary_id.foo}' must be plain",+ "- uri_format placeholder '{extension.foo}' must be plain",id="uri-field-attr-access",),pytest.param(diff --git a/tests/unit/pipelex/tracing/test_assembler_equivalence.py b/tests/unit/pipelex/tracing/test_assembler_equivalence.pyindex acfe51d5d..15c0444df 100644--- a/tests/unit/pipelex/tracing/test_assembler_equivalence.py+++ b/tests/unit/pipelex/tracing/test_assembler_equivalence.py@@ -473,6 +473,7 @@ def _usage_events_for(*, assembled_child_node_id: str, event_log: InMemoryEventLnode_id=assembled_child_node_id,tokens_usage=LLMTokensUsage(job_metadata=JobMetadata(+ storage_scope="test/scope",user_id="user_test",pipeline_run_id=_PIPELINE_RUN_ID,pipe_code="gen_text",diff --git a/tests/unit/pipelex/tracing/test_assembler_usage_attribution.py b/tests/unit/pipelex/tracing/test_assembler_usage_attribution.pyindex 402951445..939d6f663 100644--- a/tests/unit/pipelex/tracing/test_assembler_usage_attribution.py+++ b/tests/unit/pipelex/tracing/test_assembler_usage_attribution.py@@ -45,6 +45,7 @@ def _base(sequence: int) -> dict[str, Any]:def _make_usage(*, unit_costs: CostsByCategoryDict, model_name: str = "test-model") -> AnyTokensUsage:return LLMTokensUsage(job_metadata=JobMetadata(+ storage_scope="test/scope",user_id="user_test",pipeline_run_id=_PIPELINE_RUN_ID,pipe_code="test_pipe",diff --git a/tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py b/tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.pyindex 093b75921..42accdc97 100644--- a/tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py+++ b/tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py@@ -26,7 +26,7 @@ from pipelex.tracing.usage_aggregator import UsageAggregatordef _job_metadata() -> JobMetadata:- return JobMetadata(user_id="test-user", pipeline_run_id="run-non-llm")+ return JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="run-non-llm")def _usage_event(tokens_usage: AnyTokensUsage, sequence: int) -> UsageReportEvent:diff --git a/tests/unit/pipelex/tracing/test_trace_events.py b/tests/unit/pipelex/tracing/test_trace_events.pyindex 0452ae74c..45443c0b9 100644--- a/tests/unit/pipelex/tracing/test_trace_events.py+++ b/tests/unit/pipelex/tracing/test_trace_events.py@@ -69,6 +69,7 @@ class _Shared:@staticmethoddef make_job_metadata() -> JobMetadata:return JobMetadata(+ storage_scope="test/scope",user_id="user_test",pipeline_run_id=_Shared.PIPELINE_RUN_ID,pipe_code="test_pipe",diff --git a/tests/unit/pipelex/tracing/test_usage_aggregator.py b/tests/unit/pipelex/tracing/test_usage_aggregator.pyindex 15d1ff17c..290a4715e 100644--- a/tests/unit/pipelex/tracing/test_usage_aggregator.py+++ b/tests/unit/pipelex/tracing/test_usage_aggregator.py@@ -19,6 +19,7 @@ _WORKFLOW_ID = "wf_abc"def _make_llm_tokens_usage(model_name: str = "claude-sonnet") -> LLMTokensUsage:return LLMTokensUsage(job_metadata=JobMetadata(+ storage_scope="test/scope",user_id="user_test",pipeline_run_id=_PIPELINE_RUN_ID,pipe_code="test_pipe",diff --git a/tests/unit/pipelex/tracing/test_usage_attribution.py b/tests/unit/pipelex/tracing/test_usage_attribution.pyindex 2377dd77f..3f6992bcf 100644--- a/tests/unit/pipelex/tracing/test_usage_attribution.py+++ b/tests/unit/pipelex/tracing/test_usage_attribution.py@@ -29,6 +29,7 @@ def _make_usage() -> AnyTokensUsage:return LLMTokensUsage(job_metadata=JobMetadata(+ storage_scope="test/scope",user_id="user_test",pipeline_run_id="run_001",pipe_code="test_pipe",