Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

fix/Runner-read-scope · L-260929-81bc11 (#1296)

Louis Choquel6ecf16b40699
diff --git a/.drift/acks/pipelex-kernel-docs.toml b/.drift/acks/pipelex-kernel-docs.toml
index cbb4f41a9..bce5e373f 100644
--- a/.drift/acks/pipelex-kernel-docs.toml
+++ b/.drift/acks/pipelex-kernel-docs.toml
@@ -1,8 +1,8 @@
contract = "pipelex-kernel-docs"
-digest = "sha256:e3d7af90de41b84414d57788256a8dd00350049d845262d8d1c427435b785bb9"
+digest = "sha256:3ef90940ed68b1c1c7f772cf97bcda3fc3d3d81bad30805cdc93ed5966350b2a"
reviewed_by = "Louis Choquel"
-reviewed_at = "2026-09-27T18:04:39Z"
-rationale = "Review round 1 fixes: PipelexKernel.make now refuses an empty pipeline_run_id with ValueError, stated in its docstring and the changelog; the kernel doc's log-context section now claims only what the PipeLLM test pins."
+reviewed_at = "2026-09-29T01:36:33Z"
+rationale = "PipelexKernel.make and shape_inputs gained a required read_scope; the kernel doc now passes it in every example, documents the read scope beside storage_scope, and lists it among the run-constant fields of RunMetadata."
[trigger_files]
"pipelex/kernel/__init__.py" = "blob:c4f1b0fe2ccc17db9110733f71b612f4ac1ba4ce"
@@ -19,10 +19,10 @@ rationale = "Review round 1 fixes: PipelexKernel.make now refuses an empty pipel
"pipelex/kernel/llm_ops.py" = "blob:64a6d3d77a165f4f70614ced69876f56d873a8c7"
"pipelex/kernel/llm_prompt_content.py" = "blob:82c34283dacd2bfd9a2c263915e744736d399875"
"pipelex/kernel/llm_results.py" = "blob:6e3b8a2878b5bb284ece5fac10262fb59ac2029d"
-"pipelex/kernel/memory_ops.py" = "blob:f6a5745e804ac08f19c70f1beb1f78baf50d51cd"
-"pipelex/kernel/pipelex_kernel.py" = "blob:355a6e1b118b9b3236fe84ed71887e7c11612c63"
+"pipelex/kernel/memory_ops.py" = "blob:7fbbfaaec99f9712d8e215520cb02e26aad3fb40"
+"pipelex/kernel/pipelex_kernel.py" = "blob:5dec4e68d517d5ab1d7e9cbb91d836f37b10e713"
"pipelex/kernel/prompt_references.py" = "blob:03ec9ef67ae84f141d072ca2d2c3ef888549ccc0"
"pipelex/kernel/search_ops.py" = "blob:f9c1dc09dd3beaf1a54ac8e3774db1d510cbdaa3"
"pipelex/kernel/search_results.py" = "blob:101b70dc96cc5ee1721c9f6b1cb725a5ca78ff70"
"pipelex/kernel/templating_style_ops.py" = "blob:395473806d600dfc4f124c07f2afebb951fc086b"
-"tests/unit/pipelex/kernel/test_kernel_boot_contract.py" = "blob:b9e95adeed43a1eaa5467cb146cda1ba65127273"
+"tests/unit/pipelex/kernel/test_kernel_boot_contract.py" = "blob:5ba8a8315cd5f8ac3b1f57171be46cc8f0598497"
diff --git a/CHANGELOG.md b/CHANGELOG.md
index fe3578177..3cf828f18 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,10 +2,19 @@
## [Unreleased]
+### Changed
+
+- **`read_scope` is a required run parameter (Breaking)**: `RunMetadata`, `pipeline_run_setup`, `prepare_pipe_job`, `PipelexKernel.make`, `shape_inputs`, `InputShaper.shape`, `WorkingMemoryFactory.make_from_pipeline_inputs`, `StuffFactory.make_stuff_from_stuff_content_or_data`, `normalize_data_urls_to_storage` and the runtime bridge's `PipelexPipeRunInput` take a `read_scope` with no default: a host serving several tenants passes the tenant's prefix, which must contain the run's `storage_scope`, and `None` states that the run is unscoped. `PipelexMTHDSProtocol` defaults it to `None`, which is the truth on a laptop.
+- **Dry-run mocks of images and documents carry readable URLs**: a mocked `ImageContent` carries one of `inference.dry_run.image_urls` and a mocked `DocumentContent` a blank one-page PDF as a `data:` URL, where both carried a random string that reads as a local path; and a model nested two levels or more inside a mocked structure now gets the same field providers as a shallower one.
+
### Fixed
- **A generated object's storage key names its own format**: the extension of a key under `generated/` follows the whole MIME type the object is stored under, so a fetched `image/svg+xml` is stored as `.svg`, and a type with no extension of its own gets `.bin`, where both used to get `.jpg`. The MIME type's parameters and case no longer change the extension.
+### Security
+
+- **A scoped run reads only under its read scope, and never the local disk**: on a run with a read scope, a `pipelex-storage://` key is read only when it lies under that scope, compared segment by segment, and a bare path or a `file://` URI is never read, whichever value carries it — an input, a `PipeCompose` construct that assembles a URL from text, a model's structured output. Every content-generation leaf checks the URLs it will read before the dry-run branch, and the input seam checks a CSV input and every file it would upload or sign; a refused read raises `UriReadRefusedError`, answered as a 422, which names where the URL sat without quoting it. A run whose read scope is `None` reads as before.
+
## [v0.69.0] - 2026-09-28
### Highlights
diff --git a/docs/errors/platform-and-tooling.md b/docs/errors/platform-and-tooling.md
index a47154c2a..48249220d 100644
--- a/docs/errors/platform-and-tooling.md
+++ b/docs/errors/platform-and-tooling.md
@@ -52,6 +52,7 @@ own page. Classes are grouped by subsystem.
- [`StorageS3Error`](storage-s3-error.md) — S3 storage error
- [`TomlError`](toml-error.md) — TOML parse error
- [`UnknownVarPrefixError`](unknown-var-prefix-error.md) — Unknown var prefix
+- [`UriReadRefusedError`](uri-read-refused-error.md) — Read outside the run's scope refused
- [`VarFallbackPatternError`](var-fallback-pattern-error.md) — Var fallback pattern
- [`VarNotFoundError`](var-not-found-error.md) — Var not found
diff --git a/docs/errors/uri-read-refused-error.md b/docs/errors/uri-read-refused-error.md
new file mode 100644
index 000000000..6e08430ea
--- /dev/null
+++ b/docs/errors/uri-read-refused-error.md
@@ -0,0 +1,21 @@
+---
+title: "Read outside the run's scope refused"
+description: "Reference for the `UriReadRefusedError` Pipelex error class."
+---
+
+<!-- pipelex:generated -->
+
+# Read outside the run's scope refused
+
+Raised when a run asks to read a URL outside what its host lets it read.
+
+| Field | Value |
+|---|---|
+| `error_type` | `UriReadRefusedError` |
+| `title` | Read outside the run's scope refused |
+| `type_uri` | `https://docs.pipelex.com/latest/errors/uri-read-refused-error/` |
+| `error_domain` | `input` |
+| Defined in | `pipelex.tools.uri.exceptions` |
+| Parent class | [`SecurityError`](security-error.md) |
+
+[Back to Error Reference](index.md)
diff --git a/docs/under-the-hood/distributed-content-generation.md b/docs/under-the-hood/distributed-content-generation.md
index 179f022d7..3b8313e56 100644
--- a/docs/under-the-hood/distributed-content-generation.md
+++ b/docs/under-the-hood/distributed-content-generation.md
@@ -169,6 +169,14 @@ What gets stored vs. what crosses the boundary, by content type:
Each host-runtime plugin dispatches these through its own activities and queue routing; for the Temporal realization (the `act_*` activity set, per-activity task queues), see our Temporal plugin's own docs.
+### What a leaf may read: the read scope
+
+A reference is only as safe as whoever reads it. A value can carry any URL its method chose — a `PipeCompose` construct that assembles one from plain text, a model's structured output, a function's result — and the next leaf reads it: a `pipelex-storage://` key with the process's own storage credentials, or a bare path or `file://` URI from the worker's disk. On a host serving many tenants that is a read of another tenant's file, so `RunMetadata` carries a second host-supplied prefix beside `storage_scope`: the **read scope**.
+
+On a run whose read scope is set, a storage key is read only when it lies under the read scope, compared segment by segment, with no empty, `.` or `..` segment; a local path is never read; `https://` and `data:` URLs are untouched. The storage scope must lie under the read scope, since a run reads its own outputs back, and a mismatch is refused when the metadata is built. A run whose read scope is `None` (a laptop, a single-tenant server) reads exactly as before.
+
+The check runs in the leaves, because both orchestration modes call the same leaves and every read of a value's URL happens while one of them handles an assignment. Each assignment declares the URLs it will read through `referenced_uris()` — the prompt's images and documents given by URI, an image generation's input images, the image or document to extract, the document to render as page views — and the leaf authorizes them as its first statement, before the dry-run branch and before any worker is built. A dry run therefore refuses the same method a live run would, and the check performs no IO. The input seam applies the same rule to what it reads itself: a table given as a CSV input, and a file the normalization would upload or link. A refused read raises `UriReadRefusedError`, a caller-facing input error answered as a 422, which names where the URL sat and never quotes it. A unit test walks every assignment model's fields for URL-shaped ones, so a new assignment or a new field that carries a URL fails until its `referenced_uris()` declares it.
+
---
## Kajson class resolution
diff --git a/docs/under-the-hood/dry-run-mock-generation.md b/docs/under-the-hood/dry-run-mock-generation.md
index 82550ec27..769163dc9 100644
--- a/docs/under-the-hood/dry-run-mock-generation.md
+++ b/docs/under-the-hood/dry-run-mock-generation.md
@@ -118,8 +118,12 @@ class DryRunFactory:
| `IGNORE` | Sets field to `Ignore()` | None/default | `default_value`, `structure` fields |
| `DICT_SNAKE_KEY_PASCAL_VALUE` | `generate_dict_snake_key_pascal_value()` | `{mock_abcd: MockCdef}` | `inputs` dict in PipeSpec |
| `DICT_SINGLE_EXTRACT_INPUT` | `generate_dict_single_extract_input()` | `{mock_abcd: "Image"}` | `inputs` dict in PipeExtract |
+| Image URL | `generate_mock_image_url()` | one of `inference.dry_run.image_urls` | `url` of an `ImageContent`, at any depth |
+| Document URL | `generate_mock_document_url()` | `data:application/pdf;base64,…` (a blank one-page PDF) | `url` of a `DocumentContent`, at any depth |
| Random string | Polyfactory default | `uygNjiAuDMOtZEyibgHw` | All other string fields |
+A mocked image or document never carries a random string as its `url`, because `resolve_uri` takes a random string for a local path. A run with a read scope refuses to read a local path, and it checks before the dry-run branch, so a dry run on such a run would otherwise refuse its own mocks: a mocked input, or an image inside a structure a dry model call returned. An `https://` URL and a `data:` URL are read on nobody's leave, so they are what a mock carries. The providers are keyed on the class (`ImageContent`, `DocumentContent` and their subclasses), not on a field annotation, so they apply wherever such a class is nested. They also recognise a copy of either class rebuilt from a JSON schema, by its name and its `url` field: a worker running a dry model call out of process gets only the output's schema, and the nested file classes it rebuilds are no subclasses of the real ones.
+
---
## Declaring MockFormat on Fields
diff --git a/docs/under-the-hood/pipelex-kernel.md b/docs/under-the-hood/pipelex-kernel.md
index b6c257097..d888b8dd6 100644
--- a/docs/under-the-hood/pipelex-kernel.md
+++ b/docs/under-the-hood/pipelex-kernel.md
@@ -25,13 +25,15 @@ from pipelex.runtime_boot import RuntimeBoot
from pipelex.kernel.pipelex_kernel import PipelexKernel
RuntimeBoot.make()
-kernel = PipelexKernel.make(user_id="my-service", storage_scope="my-service")
+kernel = PipelexKernel.make(user_id="my-service", storage_scope="my-service", read_scope=None)
```
That boot stands up the model deck, the content generator, the class registry, the reporting delegate and the plugin registries — the machinery inference needs. It does **not** stand up the library manager, the pipe router or the pipeline manager, because a kernel caller has no method to load.
**`user_id` and `storage_scope` are both required, with no default.** `user_id` is who the run is attributed to; `storage_scope` is the opaque prefix every byte the run writes lands under, composed by *you* from whatever your tenancy model is (`"<org>/<run>"`, `"<customer>/<job>"`, or a flat name for a single-tenant service). The kernel composes its own leaves onto it and never parses it, so it never learns what a tenant is: `assets/` for what you supplied, `generated/` for what the run produced, `results/` for the delivered envelope, and `payloads/` for transport spill. It is validated at construction — one to three path-safe segments — because the value becomes a storage key prefix and a `..` in it would escape the namespace. Neither field defaults, deliberately: a missing identity used to fall back to the tracing placeholder `"anonymous"`, which then became the key prefix, pointing every unauthenticated run at one shared namespace.
+**`read_scope` is required too, and `None` is a value you pass, not a default.** It bounds what the run may read: every `pipelex-storage://` key a step reads must lie under it, compared segment by segment, and no step reads a bare path or a `file://` URI from the local disk. `https://` and `data:` URLs are untouched. A value can carry any URL its method or its model chose, so the check runs where values are read, in every content-generation leaf before the dry-run branch, and `shape_inputs` applies it to a table given as a CSV input, which is always a local file. The storage scope must lie under the read scope, since a run reads its own outputs back, and a mismatch is refused at construction. A multi-tenant host passes its tenant's prefix (the hosted platform passes the organization id); `None` is the explicit statement that the run is unscoped, which is the truth for a single-tenant service or a laptop. It has no default for the storage scope's reason: a host that forgot it would run unscoped, and an unscoped run reads every key the process's credentials reach. A refused read raises `UriReadRefusedError`, a caller-facing input error that names where the URL sat and never quotes it.
+
`needs_inference=False` boots keyless (no credentials, no model-deck validation) and sets the forced-DRY flag: every run the process initiates is coerced to `run_mode=DRY`, so the leaves mock instead of calling a provider. `PipelexKernel.make` applies that rule through the same `runtime_hub.resolve_run_mode_for_boot` the pipe tier uses — a second copy of the rule at a second factory is how the two would drift apart.
---
@@ -107,7 +109,7 @@ Every kernel function is **fully keyword-only**, with zero entries in `subject_g
- **`extract_main_content_as_list` / `extract_named_content_as_list`** — the same typed read for a call that produced several objects. A multiple-output call stores one `ListContent`, which the single-content reads cannot narrow: the bare item class raises, and `ListContent[item_type]` is rejected by design. These verify every item against `item_type`, so the list comes back typed all the way down.
```python
-memory = shape_inputs(inputs={"topic": "kernels"}, concept_provider=provider, input_specs=specs)
+memory = shape_inputs(inputs={"topic": "kernels"}, concept_provider=provider, input_specs=specs, read_scope=None)
result = await kernel.llm_object(memory=memory, output_class=Summary, concept=summary_concept, model=model, user="Summarize $topic", result="summary")
summary = extract_main_content(memory=result.memory, content_type=Summary)
```
@@ -127,7 +129,7 @@ summaries = extract_main_content_as_list(memory=result.memory, item_type=Summary
`PipelexKernel` holds run-scoped identity, plus the one seam that identity needs:
-- **`job_metadata`** — the run-level metadata. **The two halves are now separate types, and a caller reading a field off this object must know which half it is in.** `JobMetadata.run_metadata` holds what is constant for the whole run — `user_id`, `pipeline_run_id`, `storage_scope`, `request_id`, `extras` — and the rest of `JobMetadata` holds what changes per step (`pipe_code`, `pipe_run_id`, `otel_context`). So it is `kernel.job_metadata.run_metadata.pipeline_run_id`, not `kernel.job_metadata.pipeline_run_id`; there is deliberately no read-through accessor, so one fact has one spelling. `user_id`, `storage_scope`, `request_id` and `extras` are supplied by the host at `PipelexKernel.make(...)` — the last of them opaque, so a multi-tenant host can attribute a kernel-driven run to its own entities without the runtime learning what they are. `request_id` is the inbound request the run serves, validated by `RunMetadata` exactly as on the pipeline entry points, and it is what a hosted deployment filters its logs on. The distinction this bullet already drew informally — run-level metadata, per-step copy — is what the split makes explicit, and it is why the per-step copy can carry the run half through untouched. It is not what a step runs under: every call mints a per-step copy through `make_step_metadata()`, carrying a fresh `pipe_run_id` and inheriting the trace context, so trace and usage attribution stay per-step. This mirrors the interpreter's pass-down-a-modified-copy pattern. `make_step_metadata(pipe_code=…)` names the pipe the step is running — mirroring what the interpreter stamps on its live and dry paths — so log correlation, usage accounting, and the per-step labelling a distributed backend derives see a named step. When no `pipe_code` is supplied the key is omitted from the update rather than passed as `None`, so a run-level `pipe_code` is never silently erased; the direct-call façade (`llm_text`/`llm_object`) stays deliberately anonymous, because the caller ran no pipe.
+- **`job_metadata`** — the run-level metadata. **The two halves are now separate types, and a caller reading a field off this object must know which half it is in.** `JobMetadata.run_metadata` holds what is constant for the whole run — `user_id`, `pipeline_run_id`, `storage_scope`, `read_scope`, `request_id`, `extras` — and the rest of `JobMetadata` holds what changes per step (`pipe_code`, `pipe_run_id`, `otel_context`). So it is `kernel.job_metadata.run_metadata.pipeline_run_id`, not `kernel.job_metadata.pipeline_run_id`; there is deliberately no read-through accessor, so one fact has one spelling. `user_id`, `storage_scope`, `read_scope`, `request_id` and `extras` are supplied by the host at `PipelexKernel.make(...)` — the last of them opaque, so a multi-tenant host can attribute a kernel-driven run to its own entities without the runtime learning what they are. `request_id` is the inbound request the run serves, validated by `RunMetadata` exactly as on the pipeline entry points, and it is what a hosted deployment filters its logs on. The distinction this bullet already drew informally — run-level metadata, per-step copy — is what the split makes explicit, and it is why the per-step copy can carry the run half through untouched. It is not what a step runs under: every call mints a per-step copy through `make_step_metadata()`, carrying a fresh `pipe_run_id` and inheriting the trace context, so trace and usage attribution stay per-step. This mirrors the interpreter's pass-down-a-modified-copy pattern. `make_step_metadata(pipe_code=…)` names the pipe the step is running — mirroring what the interpreter stamps on its live and dry paths — so log correlation, usage accounting, and the per-step labelling a distributed backend derives see a named step. When no `pipe_code` is supplied the key is omitted from the update rather than passed as `None`, so a run-level `pipe_code` is never silently erased; the direct-call façade (`llm_text`/`llm_object`) stays deliberately anonymous, because the caller ran no pipe.
- **`cogt_run_params`** — the execution-mode contract (`run_mode`, and the DRY-only `is_mock_usage` sub-flag) that every cogt leaf reads off the assignment it is handed.
- **`step_id_source`** — where each step's `pipe_run_id` comes from, defaulting to a fresh `uuid4`, which is what every in-process run wants. It exists for a kernel hosted inside a replay-based executor, where ids minted from a non-replay-safe source take different values each re-execution; such a host injects its own replay-safe source via `PipelexKernel.make(step_id_source=…)`. It is a seam, not state — the kernel never inspects what it returns.
- **The run's own id** follows the same rule, once per run rather than once per step. `make` takes it from a `trace_context`'s `graph_id` when one is given, else from `pipeline_run_id=…`, else from a fresh `uuid4`. **A host inside a replay-based executor must pass one of the two**, because the `uuid4` default re-mints on every replay and the replayed run would then name itself differently from the run it replays. Such a host mints the id from its own replay-safe source (a workflow's deterministic random, or its workflow id) and passes the value; a value is enough here where the step id needs a callable, because the run id is minted exactly once.
@@ -148,7 +150,7 @@ event_log = InMemoryEventLog()
trace_context = TraceContext(graph_id=run_id, data_inclusion=data_inclusion, emit_graph_events=False, emit_usage_events=True)
get_report_delegate().set_event_log(context_key=trace_context.lookup_key, event_log=event_log, workflow_id="direct", pipeline_run_id=run_id)
try:
- kernel = PipelexKernel.make(user_id="my-service", storage_scope="my-service", trace_context=trace_context)
+ kernel = PipelexKernel.make(user_id="my-service", storage_scope="my-service", read_scope=None, trace_context=trace_context)
...
tokens_usages = UsageAggregator.aggregate(event_log.read_events(run_id))
finally:
@@ -169,7 +171,7 @@ A record emitted inside a `log.context` block carries the run's identifiers —
- **The host binds the run.** `PipelexKernel.log_context()` binds `request_id` and `pipeline_run_id` and no step, and a host wraps its run in it. The kernel cannot do this for the host, for the reason it cannot own the usage lifecycle: a kernel call has no run boundary to hang the binding on. Inside the host's block, each step's binding merges over the run's for the length of the call, so a line the host emits between two calls names the run, and a line emitted during one names the step as well.
```python
-kernel = PipelexKernel.make(user_id="my-service", storage_scope="my-service", request_id=request_id, pipeline_run_id=run_id)
+kernel = PipelexKernel.make(user_id="my-service", storage_scope="my-service", read_scope=None, request_id=request_id, pipeline_run_id=run_id)
with kernel.log_context():
result = await kernel.llm_text(memory=memory, user="Summarize $topic", result="summary")
```
diff --git a/docs/under-the-hood/storage-provider-plugins.md b/docs/under-the-hood/storage-provider-plugins.md
index c740f5885..d648392e0 100644
--- a/docs/under-the-hood/storage-provider-plugins.md
+++ b/docs/under-the-hood/storage-provider-plugins.md
@@ -161,6 +161,10 @@ Installing the distribution makes the method selectable (`runtime.storage.method
Use `pipelex plugins list` to see every discovered plugin, the entry-point group it was found under, what each contributed, and its denylist state. The **Group** column is the first thing to read when a plugin is missing: a built-in shows `—`, and an external plugin that resolved to the wrong layer shows it there.
+### A provider is not the tenant boundary
+
+A provider reads any key it is handed with the process's own credentials, and it should: it has no notion of whose run asked. On a host serving several tenants, what bounds a run's reads is the **read scope** the host passes with the run, which the content-generation leaves and the input seam check before they call a provider. A key outside that prefix, and any local path, is refused there, so a provider never sees it. A self-hosted deployment serving one tenant passes no read scope and reads as before. See [Distributed Content Generation](distributed-content-generation.md#what-a-leaf-may-read-the-read-scope) for the rule and where it runs.
+
---
## Related
diff --git a/pipelex/cli/dev_cli/commands/generate_projection_corpus_cmd.py b/pipelex/cli/dev_cli/commands/generate_projection_corpus_cmd.py
index 316d0b4d4..4c2f7e714 100644
--- a/pipelex/cli/dev_cli/commands/generate_projection_corpus_cmd.py
+++ b/pipelex/cli/dev_cli/commands/generate_projection_corpus_cmd.py
@@ -491,6 +491,9 @@ class ShapingGate:
concept_provider=get_concept_library(),
input_specs=pipe.inputs,
search_scope=pipe.domain_code,
+ # No run is behind this round trip, and a table reference is never read: the
+ # corpus writes no local file under a template's URLs.
+ read_scope=None,
)
except Exception as exc: # ruff: ignore[blind-except]
# (2) Unbounded surface: the shaper's explicit arm hands values straight to pydantic on
diff --git a/pipelex/cogt/content_generation/assignment_models.py b/pipelex/cogt/content_generation/assignment_models.py
index 40ca1a8f2..8621402a4 100644
--- a/pipelex/cogt/content_generation/assignment_models.py
+++ b/pipelex/cogt/content_generation/assignment_models.py
@@ -15,6 +15,7 @@ from pipelex.cogt.search.search_setting import SearchSetting
from pipelex.system.job_metadata import JobMetadata
from pipelex.tools.jinja2.template_category import TemplateCategory
from pipelex.tools.templating.templating_style import TemplatingStyle
+from pipelex.tools.uri.uri_read_scope import UriReference
class LLMAssignment(BaseModel):
@@ -63,6 +64,10 @@ class LLMAssignment(BaseModel):
def llm_handle(self) -> str:
return self.llm_setting.model
+ def referenced_uris(self) -> list[UriReference]:
+ """The URLs the leaf reads for this assignment, which it authorizes against the run's read scope."""
+ return self.llm_prompt.referenced_uris()
+
@property
def llm_job_params(self) -> LLMJobParams:
return self.llm_setting.make_llm_job_params()
@@ -83,6 +88,10 @@ class ObjectAssignment(BaseModel):
"""Delegates to the nested LLM assignment — single copy, no duplication."""
return self.llm_assignment_for_object.cogt_run_params
+ def referenced_uris(self) -> list[UriReference]:
+ """The URLs the leaf reads for this assignment: those of the nested LLM assignment."""
+ return self.llm_assignment_for_object.referenced_uris()
+
@staticmethod
def make_for_class(
object_class: type[BaseModel],
@@ -107,6 +116,10 @@ class ImgGenAssignment(BaseModel):
img_gen_job_config: ImgGenJobConfig
nb_images: int
+ def referenced_uris(self) -> list[UriReference]:
+ """The URLs the leaf reads for this assignment, which it authorizes against the run's read scope."""
+ return self.img_gen_prompt.referenced_uris()
+
class TemplatingAssignment(BaseModel):
job_metadata: JobMetadata
@@ -116,6 +129,10 @@ class TemplatingAssignment(BaseModel):
templating_style: TemplatingStyle | None = None
category: TemplateCategory
+ def referenced_uris(self) -> list[UriReference]:
+ """None: rendering a template reads no URL, whatever the context carries."""
+ return []
+
class ExtractAssignment(BaseModel):
job_metadata: JobMetadata
@@ -125,6 +142,10 @@ class ExtractAssignment(BaseModel):
extract_job_params: ExtractJobParams
extract_job_config: ExtractJobConfig
+ def referenced_uris(self) -> list[UriReference]:
+ """The URLs the leaf reads for this assignment, which it authorizes against the run's read scope."""
+ return self.extract_input.referenced_uris()
+
class RenderPageViewsAssignment(BaseModel):
job_metadata: JobMetadata
@@ -132,6 +153,10 @@ class RenderPageViewsAssignment(BaseModel):
document_uri: str
page_views_dpi: int
+ def referenced_uris(self) -> list[UriReference]:
+ """The URL the leaf reads for this assignment: the document whose pages it renders."""
+ return [UriReference(uri=self.document_uri, position="the document to render as page views")]
+
class SearchAssignment(BaseModel):
"""Serializable unit for a single web-search leaf call.
@@ -155,6 +180,10 @@ class SearchAssignment(BaseModel):
def search_handle(self) -> str:
return self.search_setting.model
+ def referenced_uris(self) -> list[UriReference]:
+ """None: a web search reads no URL a value carries; the provider fetches what it finds."""
+ return []
+
class SearchObjectAssignment(BaseModel):
"""Structured-search counterpart of ``ObjectAssignment``.
@@ -173,6 +202,10 @@ class SearchObjectAssignment(BaseModel):
"""Delegates to the nested search assignment — single copy, no duplication."""
return self.search_assignment.cogt_run_params
+ def referenced_uris(self) -> list[UriReference]:
+ """The URLs the leaf reads for this assignment: those of the nested search assignment, which are none."""
+ return self.search_assignment.referenced_uris()
+
@staticmethod
def make_for_class(
output_class: type[BaseModel],
diff --git a/pipelex/cogt/content_generation/dry_run_factory.py b/pipelex/cogt/content_generation/dry_run_factory.py
index af42096aa..75a86ddcf 100644
--- a/pipelex/cogt/content_generation/dry_run_factory.py
+++ b/pipelex/cogt/content_generation/dry_run_factory.py
@@ -1,18 +1,42 @@
+import base64
import random
import string
import types
import typing
from collections.abc import Callable
from enum import StrEnum
-from typing import Any, Union, get_args, get_origin
+from typing import TYPE_CHECKING, Any, Union, get_args, get_origin
from polyfactory.factories.pydantic_factory import ModelFactory
from polyfactory.fields import Ignore, PostGenerated, Use
from pydantic import BaseModel
from pydantic.fields import FieldInfo
+from pipelex.config import get_config
+from pipelex.core.stuffs.document_content import DocumentContent
+from pipelex.core.stuffs.image_content import ImageContent
from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar
+if TYPE_CHECKING:
+ from polyfactory.factories.base import BaseFactory
+
+# A blank one-page PDF, the file behind every mocked document. Its xref offsets are exact, so a
+# reader opens it as it is; a test checks that one does.
+_BLANK_PDF_BYTES = (
+ b"%PDF-1.4\n"
+ b"1 0 obj\n<< /Type /Catalog /Pages 2 0 R >>\nendobj\n"
+ b"2 0 obj\n<< /Type /Pages /Kids [3 0 R] /Count 1 >>\nendobj\n"
+ b"3 0 obj\n<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] >>\nendobj\n"
+ b"xref\n0 4\n"
+ b"0000000000 65535 f \n"
+ b"0000000009 00000 n \n"
+ b"0000000058 00000 n \n"
+ b"0000000115 00000 n \n"
+ b"trailer\n<< /Size 4 /Root 1 0 R >>\n"
+ b"startxref\n186\n%%EOF\n"
+)
+MOCK_DOCUMENT_DATA_URL = f"data:application/pdf;base64,{base64.b64encode(_BLANK_PDF_BYTES).decode()}"
+
class MockFormat(StrEnum):
"""Mock format specifications for Field json_schema_extra."""
@@ -93,6 +117,45 @@ class DryRunFactory:
value = random.choice(["Image", "Document"])
return {key: value}
+ @classmethod
+ def generate_mock_image_url(cls) -> str:
+ """Pick one of the configured dry-run image URLs, the ones the dry image generation returns too."""
+ return random.choice(get_config().inference.dry_run.image_urls)
+
+ @classmethod
+ def generate_mock_document_url(cls) -> str:
+ """The data URL of a blank one-page PDF."""
+ return MOCK_DOCUMENT_DATA_URL
+
+ @classmethod
+ def _file_url_providers(cls, *, object_class: type[BaseModel]) -> dict[str, Any]:
+ """Give a mocked image or document a URL that is neither a local path nor a storage key.
+
+ Left to polyfactory, a file's ``url`` is a random string, which ``resolve_uri`` takes for a
+ local path. A run with a read scope refuses to read a local path, and it checks before the
+ dry-run branch, so a dry run on such a run would refuse its own mocks: a mocked input, or an
+ image inside a structure a dry model call returned. An https URL or a data URL is read by
+ nobody's leave, so it is what a mock carries.
+ """
+ if cls._is_file_class(object_class=object_class, file_class=ImageContent):
+ return {"url": Use(cls.generate_mock_image_url)}
+ if cls._is_file_class(object_class=object_class, file_class=DocumentContent):
+ return {"url": Use(cls.generate_mock_document_url)}
+ return {}
+
+ @classmethod
+ def _is_file_class(cls, *, object_class: type[BaseModel], file_class: type[BaseModel]) -> bool:
+ """Whether a class is the file class, a subclass of it, or a copy of it rebuilt from a JSON schema.
+
+ A worker that runs a dry model call out of process gets no class, only the output's JSON
+ schema, and rebuilds the classes from it: the nested file class it builds is no subclass of
+ the real one, but it keeps the real one's name, which the schema carries as the definition's
+ title, and its ``url`` field.
+ """
+ if issubclass(object_class, file_class):
+ return True
+ return object_class.__name__ == file_class.__name__ and "url" in object_class.model_fields
+
@classmethod
def _get_examples_from_field(cls, field_info: FieldInfo) -> list[Any] | None:
"""Extract examples from a field's definition.
@@ -266,6 +329,7 @@ class DryRunFactory:
"__use_examples__": True,
"__allow_none_optionals__": False,
}
+ class_attrs.update(cls._file_url_providers(object_class=nested_class))
# Detect fields with examples and create providers
fields_with_examples = cls._detect_examples_constraints(nested_class) # type: ignore[arg-type]
@@ -373,6 +437,7 @@ class DryRunFactory:
"__use_examples__": True,
"__allow_none_optionals__": False,
}
+ class_attrs.update(cls._file_url_providers(object_class=object_class))
# Add snake_case providers
for field_name in all_snake_case:
@@ -424,15 +489,20 @@ class DryRunFactory:
# Find all nested BaseModel classes and create factories for them
nested_classes = cls._find_nested_base_model_classes(object_class) # type: ignore[arg-type]
- nested_factories: dict[type[BaseModel], type[ModelFactory[Any]]] = {}
+ nested_factories: dict[Any, type[BaseFactory[Any]]] = {}
for nested_class in nested_classes:
nested_factory = cls._create_nested_factory(nested_class)
nested_factories[nested_class] = nested_factory
# Register nested factories using __base_factory_overrides__ attribute
- # This tells polyfactory to use our custom factories for these model types
+ # This tells polyfactory to use our custom factories for these model types.
+ # Every nested factory gets the same map: polyfactory consults the overrides of the factory
+ # building the enclosing model and never passes them down, so without this a model nested
+ # two levels deep was built by a plain factory, with none of the providers above.
if nested_factories:
class_attrs["__base_factory_overrides__"] = nested_factories
+ for registered_factory in nested_factories.values():
+ registered_factory.__base_factory_overrides__ = nested_factories
# Dynamically create the factory class using ModelFactory as the parent
dry_run_factory: type[ModelFactory[BaseModelTypeVar]] = type( # type: ignore[assignment]
diff --git a/pipelex/cogt/content_generation/extract_generate.py b/pipelex/cogt/content_generation/extract_generate.py
index 034c3971c..f00b21e47 100644
--- a/pipelex/cogt/content_generation/extract_generate.py
+++ b/pipelex/cogt/content_generation/extract_generate.py
@@ -1,6 +1,7 @@
from pipelex.cogt.content_generation.assignment_models import ExtractAssignment
from pipelex.cogt.content_generation.dry_mock import dry_extract_page_contents
from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory
+from pipelex.cogt.content_generation.read_authorization import authorize_assignment_reads
from pipelex.cogt.extract.extract_job_factory import ExtractJobFactory
from pipelex.cogt.extract.extract_output import ExtractOutput
from pipelex.core.stuffs.page_content import PageContent
@@ -8,6 +9,7 @@ from pipelex.runtime_hub import get_extract_worker
async def extract_gen_pages(extract_assignment: ExtractAssignment) -> ExtractOutput:
+ authorize_assignment_reads(job_metadata=extract_assignment.job_metadata, uri_references=extract_assignment.referenced_uris())
extract_worker = get_extract_worker(extract_handle=extract_assignment.extract_handle)
extract_job = ExtractJobFactory.make_extract_job(
extract_input=extract_assignment.extract_input,
@@ -27,8 +29,10 @@ async def extract_gen_pages_and_store(
The DRY branch sits at the ``*_and_store`` layer, above the raw provider leaf, so a dry run
performs no storage IO — see the ``dry_mock`` module docstring (eng review D10). Do not
- "unify" it downward into the raw leaf.
+ "unify" it downward into the raw leaf. The read scope is authorized above that branch, so a dry
+ run refuses what a live one would; the raw leaf authorizes again, for a caller that reaches it directly.
"""
+ authorize_assignment_reads(job_metadata=extract_assignment.job_metadata, uri_references=extract_assignment.referenced_uris())
if extract_assignment.cogt_run_params.run_mode.is_dry:
return dry_extract_page_contents(extract_assignment)
extract_output = await extract_gen_pages(extract_assignment)
diff --git a/pipelex/cogt/content_generation/img_gen_generate.py b/pipelex/cogt/content_generation/img_gen_generate.py
index 710937579..3257e56ca 100644
--- a/pipelex/cogt/content_generation/img_gen_generate.py
+++ b/pipelex/cogt/content_generation/img_gen_generate.py
@@ -2,6 +2,7 @@ from pipelex import log
from pipelex.cogt.content_generation.assignment_models import ImgGenAssignment
from pipelex.cogt.content_generation.dry_mock import dry_img_gen_image_contents
from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory
+from pipelex.cogt.content_generation.read_authorization import authorize_assignment_reads
from pipelex.cogt.image.generated_image import GeneratedImageRawDetails
from pipelex.cogt.img_gen.img_gen_job_factory import ImgGenJobFactory
from pipelex.core.stuffs.image_content import ImageContent
@@ -9,6 +10,7 @@ from pipelex.runtime_hub import get_img_gen_worker
async def img_gen_single_image(img_gen_assignment: ImgGenAssignment) -> GeneratedImageRawDetails:
+ authorize_assignment_reads(job_metadata=img_gen_assignment.job_metadata, uri_references=img_gen_assignment.referenced_uris())
img_gen_worker = get_img_gen_worker(img_gen_handle=img_gen_assignment.img_gen_handle)
img_gen_job = ImgGenJobFactory.make_img_gen_job_from_prompt(
img_gen_prompt=img_gen_assignment.img_gen_prompt,
@@ -22,6 +24,7 @@ async def img_gen_single_image(img_gen_assignment: ImgGenAssignment) -> Generate
async def img_gen_image_list(img_gen_assignment: ImgGenAssignment) -> list[GeneratedImageRawDetails]:
+ authorize_assignment_reads(job_metadata=img_gen_assignment.job_metadata, uri_references=img_gen_assignment.referenced_uris())
img_gen_worker = get_img_gen_worker(img_gen_handle=img_gen_assignment.img_gen_handle)
img_gen_job = ImgGenJobFactory.make_img_gen_job_from_prompt(
img_gen_prompt=img_gen_assignment.img_gen_prompt,
@@ -46,8 +49,10 @@ async def img_gen_single_image_and_store(
The DRY branch sits at the ``*_and_store`` layer, above the raw provider leaf, so a dry run
performs no storage IO — see the ``dry_mock`` module docstring (eng review D10). Do not
- "unify" it downward into the raw leaf.
+ "unify" it downward into the raw leaf. The read scope is authorized above that branch, so a dry
+ run refuses what a live one would; the raw leaf authorizes again, for a caller that reaches it directly.
"""
+ authorize_assignment_reads(job_metadata=img_gen_assignment.job_metadata, uri_references=img_gen_assignment.referenced_uris())
if img_gen_assignment.cogt_run_params.run_mode.is_dry:
# Exactly one mock, matching the live single path's one-provider-call semantics
# regardless of the assignment's nb_images.
@@ -69,8 +74,10 @@ async def img_gen_image_list_and_store(
) -> list[ImageContent]:
"""Generate multiple images and store them, returning ImageContent list with URLs (no raw binary data).
- DRY branch at the ``*_and_store`` layer — see ``img_gen_single_image_and_store`` (D10).
+ DRY branch at the ``*_and_store`` layer — see ``img_gen_single_image_and_store`` (D10), and the
+ read scope authorized above it.
"""
+ authorize_assignment_reads(job_metadata=img_gen_assignment.job_metadata, uri_references=img_gen_assignment.referenced_uris())
if img_gen_assignment.cogt_run_params.run_mode.is_dry:
return dry_img_gen_image_contents(img_gen_assignment)
generated_image_list = await img_gen_image_list(img_gen_assignment)
diff --git a/pipelex/cogt/content_generation/llm_generate.py b/pipelex/cogt/content_generation/llm_generate.py
index 060e03326..e0a22a5aa 100644
--- a/pipelex/cogt/content_generation/llm_generate.py
+++ b/pipelex/cogt/content_generation/llm_generate.py
@@ -10,12 +10,14 @@ from pipelex.cogt.content_generation.dry_mock import (
dry_llm_gen_text,
)
from pipelex.cogt.content_generation.object_class_resolution import resolve_object_class
+from pipelex.cogt.content_generation.read_authorization import authorize_assignment_reads
from pipelex.cogt.llm.llm_job_factory import LLMJobFactory
from pipelex.cogt.llm.llm_utils import dump_prompt, dump_response_from_text_gen
from pipelex.runtime_hub import get_llm_worker
async def llm_gen_text(llm_assignment: LLMAssignment) -> str:
+ authorize_assignment_reads(job_metadata=llm_assignment.job_metadata, uri_references=llm_assignment.referenced_uris())
if llm_assignment.cogt_run_params.run_mode.is_dry:
return dry_llm_gen_text(llm_assignment)
llm_worker = get_llm_worker(llm_handle=llm_assignment.llm_handle)
@@ -38,6 +40,7 @@ async def llm_gen_object(object_assignment: ObjectAssignment, *, object_class: t
class from the assignment's JSON schema instead — see :mod:`.object_class_resolution`.
"""
llm_assignment = object_assignment.llm_assignment_for_object
+ authorize_assignment_reads(job_metadata=llm_assignment.job_metadata, uri_references=object_assignment.referenced_uris())
if object_assignment.cogt_run_params.run_mode.is_dry:
return dry_llm_gen_object(object_assignment, object_class=object_class)
llm_worker = get_llm_worker(llm_handle=llm_assignment.llm_handle)
@@ -63,6 +66,7 @@ async def llm_gen_object_list(object_assignment: ObjectAssignment, *, object_cla
item class from the assignment's JSON schema instead — see :mod:`.object_class_resolution`.
"""
llm_assignment = object_assignment.llm_assignment_for_object
+ authorize_assignment_reads(job_metadata=llm_assignment.job_metadata, uri_references=object_assignment.referenced_uris())
if object_assignment.cogt_run_params.run_mode.is_dry:
return dry_llm_gen_object_list(object_assignment, object_class=object_class)
log.verbose(f"llm_gen_object_list to generate a list of '{object_assignment.object_class_name}'")
diff --git a/pipelex/cogt/content_generation/read_authorization.py b/pipelex/cogt/content_generation/read_authorization.py
new file mode 100644
index 000000000..a3a6ae0cf
--- /dev/null
+++ b/pipelex/cogt/content_generation/read_authorization.py
@@ -0,0 +1,29 @@
+"""The content-generation leaves' half of the read scope's check.
+
+Every read of a value's URL in the content-generation layer happens while a leaf handles an
+assignment, and both orchestration modes call the same leaves — the in-process
+``ContentGenerator`` and the Temporal activities — so the leaves are the one place every read and
+every hand-off to another reader share. Each leaf authorizes the URLs its assignment declares
+(``referenced_uris()``) as its first statement, before the dry-run branch, so a dry run refuses the
+method a live run would, and before any worker is built. See :mod:`pipelex.tools.uri.uri_read_scope`
+for the rule.
+"""
+
+from pipelex.system.job_metadata import JobMetadata
+from pipelex.tools.uri.uri_read_scope import UriReference, authorize_uri_reads
+
+
+def authorize_assignment_reads(*, job_metadata: JobMetadata, uri_references: list[UriReference]) -> None:
+ """Refuse the assignment's reads that the run's read scope does not allow.
+
+ The pipe the assignment belongs to is added to each position, so a refusal names the step.
+
+ Raises:
+ UriReadRefusedError: one of the URLs is a storage key outside the read scope, or a local path.
+ """
+ pipe_code = job_metadata.pipe_code
+ if pipe_code is not None:
+ uri_references = [
+ UriReference(uri=uri_reference.uri, position=f"{uri_reference.position} of pipe '{pipe_code}'") for uri_reference in uri_references
+ ]
+ authorize_uri_reads(uri_references=uri_references, read_scope=job_metadata.run_metadata.read_scope)
diff --git a/pipelex/cogt/content_generation/render_generate.py b/pipelex/cogt/content_generation/render_generate.py
index 09eaddb8d..be7829aa3 100644
--- a/pipelex/cogt/content_generation/render_generate.py
+++ b/pipelex/cogt/content_generation/render_generate.py
@@ -9,6 +9,7 @@ Temporal workflow boundary — only URL-bearing ``ImageContent`` references are
from pipelex.cogt.content_generation.assignment_models import RenderPageViewsAssignment
from pipelex.cogt.content_generation.dry_mock import dry_render_page_views
from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory
+from pipelex.cogt.content_generation.read_authorization import authorize_assignment_reads
from pipelex.cogt.image.generated_image import GeneratedImageRawDetails
from pipelex.core.stuffs.image_content import ImageContent
from pipelex.tools.misc.image_utils import ImageFormat
@@ -22,8 +23,10 @@ async def render_page_views_and_store(
"""Render PDF pages as images, store them, and return lightweight ImageContent references.
The DRY branch sits here, above both the pypdfium2 rendering and the store step, so a dry run
- performs no rendering and no storage IO (eng review D10).
+ performs no rendering and no storage IO (eng review D10). The read scope is authorized above it,
+ so a dry run refuses what a live one would.
"""
+ authorize_assignment_reads(job_metadata=render_assignment.job_metadata, uri_references=render_assignment.referenced_uris())
if render_assignment.cogt_run_params.run_mode.is_dry:
return dry_render_page_views(render_assignment)
# Deferred import: avoid pulling the pdf rendering SDK at module-load time
diff --git a/pipelex/cogt/content_generation/search_generate.py b/pipelex/cogt/content_generation/search_generate.py
index c28ffd22f..2109b04c7 100644
--- a/pipelex/cogt/content_generation/search_generate.py
+++ b/pipelex/cogt/content_generation/search_generate.py
@@ -25,6 +25,7 @@ from pipelex.cogt.content_generation.dry_mock import (
dry_search_gen_structured_object,
)
from pipelex.cogt.content_generation.object_revalidation import revalidate_leaf_data
+from pipelex.cogt.content_generation.read_authorization import authorize_assignment_reads
from pipelex.cogt.content_generation.schema_to_model_factory import SchemaToModelFactory
from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.cogt.search.search_job import SearchJob
@@ -58,6 +59,7 @@ def _make_search_job(search_assignment: SearchAssignment) -> SearchJob:
async def search_gen_sourced_answer(search_assignment: SearchAssignment) -> SearchResultContent:
+ authorize_assignment_reads(job_metadata=search_assignment.job_metadata, uri_references=search_assignment.referenced_uris())
if search_assignment.cogt_run_params.run_mode.is_dry:
return dry_search_gen_sourced_answer(search_assignment)
worker = _make_search_worker(search_assignment)
@@ -77,6 +79,7 @@ async def search_gen_structured(search_object_assignment: SearchObjectAssignment
the stack, and handing it down is what keeps its validators and schema hints from being lost.
"""
search_assignment = search_object_assignment.search_assignment
+ authorize_assignment_reads(job_metadata=search_assignment.job_metadata, uri_references=search_object_assignment.referenced_uris())
if search_assignment.cogt_run_params.run_mode.is_dry:
return dry_search_gen_structured(search_object_assignment)
boundary_class = SchemaToModelFactory.make_from_json_schema(
@@ -103,6 +106,7 @@ async def search_gen_structured_object(
submitter. The two arms of this module therefore differ in return type on purpose — a dict is the
wire's shape, an instance is the caller's.
"""
+ authorize_assignment_reads(job_metadata=search_assignment.job_metadata, uri_references=search_assignment.referenced_uris())
if search_assignment.cogt_run_params.run_mode.is_dry:
return dry_search_gen_structured_object(search_assignment, output_class=output_class)
result_dict = await _run_structured_search(search_assignment, schema=output_class)
diff --git a/pipelex/cogt/content_generation/templating_generate.py b/pipelex/cogt/content_generation/templating_generate.py
index 96f4ce7f7..6f7e2c738 100644
--- a/pipelex/cogt/content_generation/templating_generate.py
+++ b/pipelex/cogt/content_generation/templating_generate.py
@@ -1,9 +1,11 @@
from pipelex.cogt.content_generation.assignment_models import TemplatingAssignment
from pipelex.cogt.content_generation.dry_mock import dry_templating_gen_text
+from pipelex.cogt.content_generation.read_authorization import authorize_assignment_reads
from pipelex.cogt.templating.template_rendering import render_template
async def templating_gen_text(templating_assignment: TemplatingAssignment) -> str:
+ authorize_assignment_reads(job_metadata=templating_assignment.job_metadata, uri_references=templating_assignment.referenced_uris())
if templating_assignment.cogt_run_params.run_mode.is_dry:
return dry_templating_gen_text(templating_assignment)
templated_text: str = await render_template(
diff --git a/pipelex/cogt/extract/extract_input.py b/pipelex/cogt/extract/extract_input.py
index 51a9fdccb..ecc6a1299 100644
--- a/pipelex/cogt/extract/extract_input.py
+++ b/pipelex/cogt/extract/extract_input.py
@@ -4,6 +4,7 @@ from pydantic import BaseModel, model_validator
from pipelex.cogt.extract.exceptions import ExtractInputError
from pipelex.tools.typing.validation_utils import has_exactly_one_among_attributes_from_list
+from pipelex.tools.uri.uri_read_scope import UriReference
class ExtractInput(BaseModel):
@@ -16,3 +17,12 @@ class ExtractInput(BaseModel):
msg = "Exactly one of 'image_uri' or 'document_uri' must be provided"
raise ExtractInputError(msg)
return self
+
+ def referenced_uris(self) -> list[UriReference]:
+ """The URL a worker will read to extract: the image or the document."""
+ uri_references: list[UriReference] = []
+ if self.image_uri is not None:
+ uri_references.append(UriReference(uri=self.image_uri, position="the image to extract"))
+ if self.document_uri is not None:
+ uri_references.append(UriReference(uri=self.document_uri, position="the document to extract"))
+ return uri_references
diff --git a/pipelex/cogt/img_gen/img_gen_prompt.py b/pipelex/cogt/img_gen/img_gen_prompt.py
index 9d02f2888..ced41000a 100644
--- a/pipelex/cogt/img_gen/img_gen_prompt.py
+++ b/pipelex/cogt/img_gen/img_gen_prompt.py
@@ -3,9 +3,10 @@ from typing_extensions import override
from pipelex import log
from pipelex.cogt.exceptions import ImgGenPromptError
-from pipelex.cogt.image.prompt_image import PromptImage
+from pipelex.cogt.image.prompt_image import PromptImage, PromptImageUri
from pipelex.system.runtime import ProblemReaction, runtime_manager
from pipelex.tools.misc.json_utils import json_str
+from pipelex.tools.uri.uri_read_scope import UriReference
class ImgGenPrompt(BaseModel):
@@ -13,6 +14,14 @@ class ImgGenPrompt(BaseModel):
negative_text: str | None = None
input_images: list[PromptImage] | None = None
+ def referenced_uris(self) -> list[UriReference]:
+ """The URLs a worker will read to send this prompt: its input images given by URI."""
+ return [
+ UriReference(uri=image.uri, position=f"input image {index} of the image generation")
+ for index, image in enumerate(self.input_images or [], start=1)
+ if isinstance(image, PromptImageUri)
+ ]
+
def validate_before_execution(self):
reaction = runtime_manager.problem_reactions.job
match reaction:
diff --git a/pipelex/cogt/llm/llm_prompt.py b/pipelex/cogt/llm/llm_prompt.py
index 9a4311275..87ea5b095 100644
--- a/pipelex/cogt/llm/llm_prompt.py
+++ b/pipelex/cogt/llm/llm_prompt.py
@@ -2,11 +2,12 @@ from pydantic import BaseModel
from typing_extensions import override
from pipelex import log
-from pipelex.cogt.document.prompt_document import PromptDocument
+from pipelex.cogt.document.prompt_document import PromptDocument, PromptDocumentUri
from pipelex.cogt.exceptions import LLMPromptParameterError
-from pipelex.cogt.image.prompt_image import PromptImage
+from pipelex.cogt.image.prompt_image import PromptImage, PromptImageUri
from pipelex.system.runtime import ProblemReaction, runtime_manager
from pipelex.tools.misc.string_utils import is_none_or_has_text, is_not_none_and_has_text
+from pipelex.tools.uri.uri_read_scope import UriReference
class LLMPrompt(BaseModel):
@@ -15,6 +16,20 @@ class LLMPrompt(BaseModel):
user_images: list[PromptImage] = []
user_documents: list[PromptDocument] = []
+ def referenced_uris(self) -> list[UriReference]:
+ """The URLs a worker will read to send this prompt: its images and documents given by URI."""
+ uri_references = [
+ UriReference(uri=image.uri, position=f"image {index} of the prompt")
+ for index, image in enumerate(self.user_images, start=1)
+ if isinstance(image, PromptImageUri)
+ ]
+ uri_references.extend(
+ UriReference(uri=document.uri, position=f"document {index} of the prompt")
+ for index, document in enumerate(self.user_documents, start=1)
+ if isinstance(document, PromptDocumentUri)
+ )
+ return uri_references
+
def validate_before_execution(self):
reaction = runtime_manager.problem_reactions.job
match reaction:
diff --git a/pipelex/core/memory/input_shaper.py b/pipelex/core/memory/input_shaper.py
index 0e4c17036..9895e35c2 100644
--- a/pipelex/core/memory/input_shaper.py
+++ b/pipelex/core/memory/input_shaper.py
@@ -135,6 +135,7 @@ class InputShaper:
concept_provider: ConceptProviderAbstract,
input_specs: InputStuffSpecs,
search_scope: str | None = None,
+ read_scope: str | None,
inputs_base_dir: Path | None = None,
) -> WorkingMemory:
"""Shape every provided input against its declared ``StuffSpec`` and return a WorkingMemory.
@@ -152,6 +153,9 @@ class InputShaper:
CLI). ``None`` (in-process / inline-JSON callers) leaves relative paths untouched
(the CWD contract). Only bare strings in the file-ish and CSV arms are resolved; the
``{"url": ...}`` dict form is owned by the CLI's url-key walk.
+ read_scope: The run's read scope, which a table read while shaping must satisfy: on a
+ scoped run a CSV input, which is always a local file, is refused. ``None`` for an
+ unscoped run. See :mod:`pipelex.tools.uri.uri_read_scope`.
Raises:
UnknownInputNameError: a provided name is not declared (D8).
@@ -167,6 +171,7 @@ class InputShaper:
stuff_spec=stuff_spec,
variable_name=variable_name,
search_scope=search_scope,
+ read_scope=read_scope,
inputs_base_dir=inputs_base_dir,
)
working_memory.add_new_stuff(name=variable_name, stuff=stuff)
@@ -190,6 +195,7 @@ class InputShaper:
stuff_spec: StuffSpec,
variable_name: str,
search_scope: str | None,
+ read_scope: str | None,
inputs_base_dir: Path | None,
) -> Stuff:
declared_concept = stuff_spec.concept
@@ -204,6 +210,7 @@ class InputShaper:
stuff_spec=stuff_spec,
variable_name=variable_name,
search_scope=search_scope,
+ read_scope=read_scope,
)
# (D9) A top-level null is never a value — absence is expressed by omitting the key.
@@ -226,6 +233,7 @@ class InputShaper:
concept_provider=concept_provider,
name=variable_name,
search_scope=search_scope,
+ read_scope=read_scope,
)
except StuffFactoryError as exc:
# (R8) The factory's refusal names neither the input nor its declared concept, so it
@@ -267,6 +275,7 @@ class InputShaper:
input_kind=input_kind,
variable_name=variable_name,
search_scope=search_scope,
+ read_scope=read_scope,
inputs_base_dir=inputs_base_dir,
is_raw_data=False,
)
@@ -330,6 +339,7 @@ class InputShaper:
input_kind: InputKind,
variable_name: str,
search_scope: str | None,
+ read_scope: str | None,
inputs_base_dir: Path | None,
is_raw_data: bool,
) -> StuffContent:
@@ -346,7 +356,12 @@ class InputShaper:
# concept, tried BEFORE element-wise shaping so the bare string is not misread as one item.
if is_list and input_kind.is_structured:
csv_list_content = cls._try_shape_csv(
- value, concept_provider=concept_provider, stuff_spec=stuff_spec, variable_name=variable_name, inputs_base_dir=inputs_base_dir
+ value,
+ concept_provider=concept_provider,
+ stuff_spec=stuff_spec,
+ variable_name=variable_name,
+ inputs_base_dir=inputs_base_dir,
+ read_scope=read_scope,
)
if csv_list_content is not None:
if fixed_count is not None and len(csv_list_content.items) != fixed_count:
@@ -368,6 +383,7 @@ class InputShaper:
variable_name=variable_name,
fixed_count=fixed_count,
search_scope=search_scope,
+ read_scope=read_scope,
inputs_base_dir=inputs_base_dir,
is_raw_data=is_raw_data,
)
@@ -387,6 +403,7 @@ class InputShaper:
stuff_spec=stuff_spec,
variable_name=variable_name,
search_scope=search_scope,
+ read_scope=read_scope,
inputs_base_dir=inputs_base_dir,
)
@@ -399,6 +416,7 @@ class InputShaper:
stuff_spec: StuffSpec,
variable_name: str,
inputs_base_dir: Path | None,
+ read_scope: str | None,
) -> ListContent[StuffContent] | None:
"""Detect and read a table reference for a declared structured list input (D11).
@@ -420,7 +438,9 @@ class InputShaper:
else:
return None
url = cls._resolve_local_path(url, inputs_base_dir=inputs_base_dir)
- return StuffFactory.try_make_csv_list_content(stuff_spec.concept, concept_provider=concept_provider, content={"url": url}, name=variable_name)
+ return StuffFactory.try_make_csv_list_content(
+ stuff_spec.concept, concept_provider=concept_provider, content={"url": url}, name=variable_name, read_scope=read_scope
+ )
@classmethod
def _resolve_local_path(cls, url: str, *, inputs_base_dir: Path | None) -> str:
@@ -443,6 +463,7 @@ class InputShaper:
variable_name: str,
fixed_count: int | None,
search_scope: str | None,
+ read_scope: str | None,
inputs_base_dir: Path | None,
is_raw_data: bool,
) -> ListContent[StuffContent]:
@@ -481,6 +502,7 @@ class InputShaper:
stuff_spec=stuff_spec,
variable_name=variable_name,
search_scope=search_scope,
+ read_scope=read_scope,
inputs_base_dir=inputs_base_dir,
)
for item_value in item_values
@@ -497,6 +519,7 @@ class InputShaper:
stuff_spec: StuffSpec,
variable_name: str,
search_scope: str | None,
+ read_scope: str | None,
inputs_base_dir: Path | None,
) -> StuffContent:
"""Build one item's ``StuffContent`` from a value, dispatched on the declared kind (D5).
@@ -521,7 +544,11 @@ class InputShaper:
)
if isinstance(value, StuffContent):
built = StuffFactory.make_stuff_from_stuff_content_or_data(
- stuff_content_or_data=value, concept_provider=concept_provider, name=variable_name, search_scope=search_scope
+ stuff_content_or_data=value,
+ concept_provider=concept_provider,
+ name=variable_name,
+ search_scope=search_scope,
+ read_scope=read_scope,
)
if not concept_provider.is_compatible(tested_concept=built.concept, wanted_concept=concept):
raise ExplicitConceptIncompatibleError.make(
@@ -767,6 +794,7 @@ class InputShaper:
stuff_spec: StuffSpec,
variable_name: str,
search_scope: str | None,
+ read_scope: str | None,
) -> Stuff:
"""Build an explicit form bottom-up, then compat-check the built concept against declared (D6).
@@ -786,6 +814,7 @@ class InputShaper:
stuff_spec=stuff_spec,
variable_name=variable_name,
search_scope=search_scope,
+ read_scope=read_scope,
)
if structureless_envelope_stuff is not None:
return structureless_envelope_stuff
@@ -801,6 +830,7 @@ class InputShaper:
stuff_spec=stuff_spec,
variable_name=variable_name,
search_scope=search_scope,
+ read_scope=read_scope,
inputs_base_dir=None,
)
@@ -809,6 +839,7 @@ class InputShaper:
concept_provider=concept_provider,
name=variable_name,
search_scope=search_scope,
+ read_scope=read_scope,
)
if not concept_provider.is_compatible(tested_concept=stuff.concept, wanted_concept=declared_concept):
raise ExplicitConceptIncompatibleError.make(
@@ -833,6 +864,7 @@ class InputShaper:
stuff_spec: StuffSpec,
variable_name: str,
search_scope: str | None,
+ read_scope: str | None,
) -> Stuff | None:
"""Shape an envelope naming a structureless concept, or return ``None`` for every other explicit form (R5).
@@ -886,6 +918,7 @@ class InputShaper:
input_kind=InputKind.ANYTHING,
variable_name=variable_name,
search_scope=search_scope,
+ read_scope=read_scope,
inputs_base_dir=None,
is_raw_data=True,
)
diff --git a/pipelex/core/memory/working_memory_factory.py b/pipelex/core/memory/working_memory_factory.py
index b9463215f..7c534ac69 100644
--- a/pipelex/core/memory/working_memory_factory.py
+++ b/pipelex/core/memory/working_memory_factory.py
@@ -77,6 +77,7 @@ class WorkingMemoryFactory(BaseModel):
input_specs: InputStuffSpecs | None = None,
search_scope: str | None = None,
inputs_base_dir: Path | None = None,
+ read_scope: str | None,
) -> WorkingMemory:
"""Create a WorkingMemory from a pipeline inputs dictionary.
@@ -99,6 +100,8 @@ class WorkingMemoryFactory(BaseModel):
the inputs file's parent when inputs were file-loaded by a CLI. ``None`` for API/SDK
and in-process callers (they pass absolute urls / storage uris). Only consulted by
the shaper's file-ish / CSV arms.
+ read_scope: The run's read scope, which a table read while shaping must satisfy;
+ ``None`` for an unscoped run. See :mod:`pipelex.tools.uri.uri_read_scope`.
Returns:
WorkingMemory object reconstructed from the implicit format
@@ -111,6 +114,7 @@ class WorkingMemoryFactory(BaseModel):
input_specs=input_specs,
search_scope=search_scope,
inputs_base_dir=inputs_base_dir,
+ read_scope=read_scope,
)
working_memory = cls.make_empty()
@@ -121,6 +125,7 @@ class WorkingMemoryFactory(BaseModel):
stuff_content_or_data=stuff_content_or_data,
concept_provider=concept_provider,
search_scope=search_scope,
+ read_scope=read_scope,
)
working_memory.add_new_stuff(name=stuff_key, stuff=stuff)
return working_memory
diff --git a/pipelex/core/stuffs/stuff_factory.py b/pipelex/core/stuffs/stuff_factory.py
index be51e5466..a049efbab 100644
--- a/pipelex/core/stuffs/stuff_factory.py
+++ b/pipelex/core/stuffs/stuff_factory.py
@@ -24,6 +24,7 @@ from pipelex.tools.tabular.csv_codec import is_tabular_path, list_content_from_c
from pipelex.tools.tabular.exceptions import CsvError
from pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error
from pipelex.tools.uri.resolved_uri import ResolvedLocalPath
+from pipelex.tools.uri.uri_read_scope import authorize_uri_read
from pipelex.tools.uri.uri_resolver import resolve_uri
@@ -143,9 +144,10 @@ class StuffFactory:
content: dict[str, Any],
name: str | None,
code: str | None,
+ read_scope: str | None,
) -> Stuff | None:
"""Wrap :meth:`try_make_csv_list_content` into a ``Stuff`` (Case 2.5 envelope path)."""
- list_content = cls.try_make_csv_list_content(concept, concept_provider=concept_provider, content=content, name=name)
+ list_content = cls.try_make_csv_list_content(concept, concept_provider=concept_provider, content=content, name=name, read_scope=read_scope)
if list_content is None:
return None
return cls.make_stuff(concept=concept, content=list_content, name=name, code=code)
@@ -158,6 +160,7 @@ class StuffFactory:
concept_provider: ConceptProviderAbstract,
content: dict[str, Any],
name: str | None,
+ read_scope: str | None,
) -> ListContent[StuffContent] | None:
"""Build a ``ListContent[row-concept]`` from a ``{"url": "...csv"}`` input reference.
@@ -177,6 +180,9 @@ class StuffFactory:
v1 reads LOCAL paths only: a tabular-suffixed remote ``url`` (``http(s)``/``s3``/``gs``/
``pipelex-storage``) is rejected with a clear ``CsvError`` rather than opened as a local path.
+ And a local path is read only when ``read_scope`` allows it: a run with a read scope reads
+ nothing from the local disk (see :mod:`pipelex.tools.uri.uri_read_scope`), so there a table
+ input is refused before it is opened.
(A base64 data URL carries no file suffix, so it is never detected as tabular and simply
falls through to ordinary record handling.)
"""
@@ -242,6 +248,11 @@ class StuffFactory:
# caller-fixable input problem, not a raw ValueError that escapes into core/runner.
msg = f"CSV input for stuff '{name}': concept '{concept.concept_ref}' has no registered structure class to read CSV rows into."
raise CsvError(msg) from exc
+ authorize_uri_read(
+ uri=url,
+ read_scope=read_scope,
+ position=f"the table given for input '{name}'" if name is not None else "a table given as an input",
+ )
return list_content_from_csv(Path(resolved.path), row_model=row_model)
@classmethod
@@ -273,9 +284,13 @@ class StuffFactory:
name: str | None = None,
code: str | None = None,
search_scope: str | None = None,
+ read_scope: str | None,
) -> Stuff:
"""Create a Stuff from StuffContentOrData covering all pipeline inputs cases.
+ ``read_scope`` is the run's read scope, which a table read from a ``{"url": "...csv"}``
+ content (Case 2.5) must satisfy; ``None`` for an unscoped run.
+
Case 1: Direct content (no 'concept' key)
1.1: str → TextContent with Text concept
1.2: list[str] → ListContent[TextContent] with Text concept
@@ -598,7 +613,9 @@ class StuffFactory:
if isinstance(content, dict):
content_dict = cast("dict[str, Any]", content)
# CSV input: a {"url": "...csv"} under a structured row concept loads as ListContent[row-concept].
- csv_stuff = cls._try_make_csv_list_stuff(concept=concept, concept_provider=concept_provider, content=content_dict, name=name, code=code)
+ csv_stuff = cls._try_make_csv_list_stuff(
+ concept=concept, concept_provider=concept_provider, content=content_dict, name=name, code=code, read_scope=read_scope
+ )
if csv_stuff is not None:
return csv_stuff
diff --git a/pipelex/kernel/memory_ops.py b/pipelex/kernel/memory_ops.py
index f6a5745e8..7fbbfaaec 100644
--- a/pipelex/kernel/memory_ops.py
+++ b/pipelex/kernel/memory_ops.py
@@ -36,6 +36,7 @@ def shape_inputs(
input_specs: InputStuffSpecs,
search_scope: str | None = None,
inputs_base_dir: Path | None = None,
+ read_scope: str | None,
) -> WorkingMemory:
"""Interpret a caller's raw inputs against their declared specs and return the memory to run on.
@@ -52,9 +53,14 @@ def shape_inputs(
A malformed input is rejected here with a typed, locator-bearing error rather than coerced — this
is the boundary where a caller's raw values stop being raw, so it is the right place to refuse.
+ `read_scope` is the run's read scope, required for the reason `PipelexKernel.make` requires it:
+ shaping reads a table given as a CSV input, which is a local file, and a run with a read scope
+ reads nothing from the local disk. `None` is the explicit statement that the run is unscoped.
+
Raises:
UnknownInputNameError: a provided name is not declared in `input_specs` (D8).
InputShapingError subclasses: a provided value cannot be shaped to its declared concept (D4).
+ UriReadRefusedError: a table input names a local file on a run with a read scope.
"""
return InputShaper.shape(
inputs,
@@ -62,6 +68,7 @@ def shape_inputs(
input_specs=input_specs,
search_scope=search_scope,
inputs_base_dir=inputs_base_dir,
+ read_scope=read_scope,
)
diff --git a/pipelex/kernel/pipelex_kernel.py b/pipelex/kernel/pipelex_kernel.py
index 355a6e1b1..5dec4e68d 100644
--- a/pipelex/kernel/pipelex_kernel.py
+++ b/pipelex/kernel/pipelex_kernel.py
@@ -93,6 +93,7 @@ class PipelexKernel:
run_mode: PipeRunMode = PipeRunMode.LIVE,
user_id: str,
storage_scope: str,
+ read_scope: str | None,
request_id: str | None = None,
pipeline_run_id: str | None = None,
extras: dict[str, str] | None = None,
@@ -135,6 +136,13 @@ class PipelexKernel:
source and passes the value; unlike a step id, it is minted once per run, so a value is
enough where ``step_id_source`` needs a callable.
+ ``read_scope`` bounds what the run may read, beside ``storage_scope``, which must lie under
+ it: every storage key a step reads lies under it, and no step reads the local disk. It is
+ required, for the storage scope's reason — this tier is the direct programmatic entry point a
+ multi-tenant host embeds, and a default here would open every key to a host that forgot it —
+ and ``None`` is the explicit statement that the run is unscoped. See
+ :mod:`pipelex.tools.uri.uri_read_scope`.
+
``extras`` is the opaque, host-supplied mapping of labels about this run, carried beside
``user_id`` and ``storage_scope``. This tier is the direct programmatic entry
point, so a multi-tenant host embedding the kernel needs the same seam the pipeline entry
@@ -155,6 +163,7 @@ class PipelexKernel:
run_metadata=RunMetadata(
user_id=user_id,
storage_scope=storage_scope,
+ read_scope=read_scope,
pipeline_run_id=cls._resolve_pipeline_run_id(trace_context=trace_context, pipeline_run_id=pipeline_run_id),
request_id=request_id,
extras=extras or {},
diff --git a/pipelex/pipe_machinery/pipe_abstract.py b/pipelex/pipe_machinery/pipe_abstract.py
index e6ab10c3e..d7c2b0889 100644
--- a/pipelex/pipe_machinery/pipe_abstract.py
+++ b/pipelex/pipe_machinery/pipe_abstract.py
@@ -516,7 +516,11 @@ class PipeAbstract(ABC, BaseModel):
# Validate external resources (URLs, file paths) referenced by input contents.
# Skipped in dry-run mode because inputs are mock-generated with fake URLs.
- if not pipe_run_params.run_mode.is_dry:
+ # Skipped on a run with a read scope too: the only thing this checks is that a local path
+ # exists, and such a run reads no local path at all, which the leaves refuse before any IO.
+ # Checking first would stat the host's disk on the method's say-so, and a missing path failing
+ # here while an existing one is refused later would tell the caller which files exist.
+ if not pipe_run_params.run_mode.is_dry and job_metadata.run_metadata.read_scope is None:
for named_stuff_spec in self.needed_inputs().named_stuff_specs:
variable_name = named_stuff_spec.variable_name
stuff = working_memory.get_optional_stuff(variable_name)
diff --git a/pipelex/pipe_run/dry_run_in_process.py b/pipelex/pipe_run/dry_run_in_process.py
index 35381835f..9feb6bf0a 100644
--- a/pipelex/pipe_run/dry_run_in_process.py
+++ b/pipelex/pipe_run/dry_run_in_process.py
@@ -171,6 +171,8 @@ async def dry_run_pipe_in_process(pipe: PipeAbstract, *, library_id: str, caller
# prefix for every run. If `dry-run-no-storage` ever appears as
# an S3 prefix, a dry run stored something and that is the bug.
storage_scope=DRY_RUN_STORAGE_SCOPE,
+ # A dry run on mock inputs reads nothing a caller named, so it is unscoped.
+ read_scope=None,
trace_context=trace_context,
)
# PipeRun.run's finally closes the tracer by pipeline_run_id and assembles the
diff --git a/pipelex/pipeline/bundle_validator.py b/pipelex/pipeline/bundle_validator.py
index 59dbf35cc..5d183e3f4 100644
--- a/pipelex/pipeline/bundle_validator.py
+++ b/pipelex/pipeline/bundle_validator.py
@@ -481,6 +481,8 @@ class BundleValidator:
# prefix for every run. If `dry-run-no-storage` ever appears as
# an S3 prefix, a dry run stored something and that is the bug.
storage_scope=DRY_RUN_STORAGE_SCOPE,
+ # A validation sweep runs on mock inputs and reads nothing a caller named, so it is unscoped.
+ read_scope=None,
)
await self._pipe_run.run(pipe_job)
except (PipelexError, ValidationError, FactoryException) as exc:
diff --git a/pipelex/pipeline/execution_seams.py b/pipelex/pipeline/execution_seams.py
index 508c161af..bbe14b1e2 100644
--- a/pipelex/pipeline/execution_seams.py
+++ b/pipelex/pipeline/execution_seams.py
@@ -52,7 +52,7 @@ from pipelex.system.configuration.configs import PipelineExecutionConfig
from pipelex.system.job_metadata import JobMetadata, OtelContext, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
from pipelex.system.run_extras import validate_run_extras
-from pipelex.system.storage_scope import validate_storage_scope
+from pipelex.system.storage_scope import validate_read_scope, validate_storage_scope, validate_storage_scope_within_read_scope
from pipelex.tools.misc.file_utils import reject_bare_str_or_path
if TYPE_CHECKING:
@@ -197,6 +197,7 @@ async def prepare_pipe_job(
pipeline_run_id: str,
user_id: str,
storage_scope: str,
+ read_scope: str | None,
extras: dict[str, str] | None = None,
inputs: PipelineInputs | WorkingMemory | None = None,
search_scope: str | None = None,
@@ -226,6 +227,11 @@ async def prepare_pipe_job(
submitter exercises the real distribution machinery at zero spend. The flag is resolved by
``PipeRunParamsFactory.make_run_params`` (the single writer of ``run_mode``), so it covers
every entry point that builds run params, not just this one.
+
+ ``read_scope`` bounds what the run may read (see :mod:`pipelex.tools.uri.uri_read_scope`). It is
+ required, ``None`` for an unscoped run, and it rides the job metadata to every leaf. Here it also
+ gates the two reads the input seam makes itself: a CSV input read while shaping, and a local file
+ uploaded by the normalization, which on a scoped run is refused instead, naming the input.
"""
# Validate the scope HERE, before anything composes a storage key from it.
#
@@ -239,6 +245,13 @@ async def prepare_pipe_job(
# This is deliberately not a "second gate": it is the FIRST one on this path.
storage_scope = validate_storage_scope(value=storage_scope)
+ # The read scope too, and its relation to the storage scope, for the same
+ # reason: the shaping and the normalization below read on its authority,
+ # and `RunMetadata` is only built after them.
+ if read_scope is not None:
+ read_scope = validate_read_scope(value=read_scope)
+ validate_storage_scope_within_read_scope(storage_scope=storage_scope, read_scope=read_scope)
+
# And the extras beside it, for the same reason: `RunMetadata` is built at
# the bottom of this function, below the data-url normalization that writes
# to real storage, so validating only there refuses a malformed mapping
@@ -264,6 +277,7 @@ async def prepare_pipe_job(
input_specs=pipe.inputs,
search_scope=search_scope,
inputs_base_dir=inputs_base_dir,
+ read_scope=read_scope,
)
# If mock inputs is enabled, generate mock data for missing required inputs.
@@ -317,12 +331,13 @@ async def prepare_pipe_job(
# Normalize data URLs to pipelex-storage:// URIs, and give every image and document input a public_url, if configured.
if working_memory and execution_config.is_normalize_data_urls_to_storage and not execution_config.is_mock_inputs:
- working_memory = await normalize_data_urls_to_storage(working_memory, storage_scope=storage_scope)
+ working_memory = await normalize_data_urls_to_storage(working_memory, storage_scope=storage_scope, read_scope=read_scope)
job_metadata = JobMetadata(
run_metadata=RunMetadata(
user_id=user_id,
storage_scope=storage_scope,
+ read_scope=read_scope,
pipeline_run_id=pipeline_run_id,
request_id=request_id,
# Already normalized to a mapping and validated at the top of this
diff --git a/pipelex/pipeline/input_normalizer.py b/pipelex/pipeline/input_normalizer.py
index f77ce02a8..15b00c513 100644
--- a/pipelex/pipeline/input_normalizer.py
+++ b/pipelex/pipeline/input_normalizer.py
@@ -26,13 +26,14 @@ from pipelex.tools.misc.http_utils import validate_http_url_syntax
from pipelex.tools.storage.exceptions import StorageInvalidUriError
from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract
from pipelex.tools.uri.resolved_uri import ResolvedBase64DataUrl, ResolvedHttpUrl, ResolvedPipelexStorage
+from pipelex.tools.uri.uri_read_scope import authorize_uri_read
from pipelex.tools.uri.uri_resolver import resolve_uri
# Type alias for content types that can have their URLs normalized
NormalizableContent = ImageContent | DocumentContent
-async def normalize_data_urls_to_storage(working_memory: WorkingMemory, *, storage_scope: str) -> WorkingMemory:
+async def normalize_data_urls_to_storage(working_memory: WorkingMemory, *, storage_scope: str, read_scope: str | None) -> WorkingMemory:
"""Convert data URLs and local files in ImageContent and DocumentContent to pipelex-storage:// URIs, and fill their public_url.
Scans all stuffs in working memory and for any ImageContent or DocumentContent with
@@ -55,15 +56,27 @@ async def normalize_data_urls_to_storage(working_memory: WorkingMemory, *, stora
under `{storage_scope}/assets/`, inside the run's own namespace —
they used to go to a flat top-level `normalized/` prefix shared by
every run of every tenant.
+ read_scope: The run's read scope. On a scoped run every url is authorized
+ before it is read or linked: a local path is refused instead of
+ being uploaded or kept for a later reader, and a storage reference
+ outside the scope is refused instead of being signed, since a signed
+ link is a read. The refusal names the input. ``None`` for an unscoped
+ run. See :mod:`pipelex.tools.uri.uri_read_scope`.
Returns:
The same WorkingMemory instance with normalized URLs.
+
+ Raises:
+ UriReadRefusedError: an input's url is a local path, or a storage reference outside the
+ read scope, on a run with a read scope.
"""
storage = get_storage_provider()
- for stuff in working_memory.root.values():
+ for input_name, stuff in working_memory.root.items():
content = stuff.content
- normalized_content, changed = await _normalize_value(value=content, storage=storage, storage_scope=storage_scope)
+ normalized_content, changed = await _normalize_value(
+ value=content, storage=storage, storage_scope=storage_scope, read_scope=read_scope, input_name=input_name
+ )
if changed:
stuff.content = normalized_content
@@ -75,6 +88,8 @@ async def _normalize_value(
*,
storage: StorageProviderAbstract,
storage_scope: str,
+ read_scope: str | None,
+ input_name: str,
) -> tuple[Any, bool]:
"""Recursively normalize a value, converting data URLs in ImageContent/DocumentContent to storage URIs.
@@ -83,26 +98,44 @@ async def _normalize_value(
storage: The storage provider to use.
storage_scope: The run's opaque storage prefix; normalized bytes land
under `{storage_scope}/assets/`.
+ read_scope: The run's read scope, which every url must satisfy before it is read or linked.
+ input_name: The input the value belongs to, named in a refusal.
Returns:
A tuple of (normalized_value, has_changed).
"""
# Handle ImageContent and DocumentContent
if isinstance(value, (ImageContent, DocumentContent)):
- normalized = await _normalize_url_content(content=value, storage=storage, storage_scope=storage_scope)
+ normalized = await _normalize_url_content(
+ content=value, storage=storage, storage_scope=storage_scope, read_scope=read_scope, input_name=input_name
+ )
return normalized, normalized is not value
# Handle StructuredContent (recursively process all fields)
if isinstance(value, StructuredContent):
- return await _normalize_structured_content(structured_content=value, storage=storage, storage_scope=storage_scope)
+ return await _normalize_structured_content(
+ structured_content=value, storage=storage, storage_scope=storage_scope, read_scope=read_scope, input_name=input_name
+ )
# Handle ListContent
if isinstance(value, ListContent):
- return await _normalize_list_content(list_content=value, storage=storage, storage_scope=storage_scope) # pyright: ignore[reportUnknownArgumentType]
+ return await _normalize_list_content(
+ list_content=value, # pyright: ignore[reportUnknownArgumentType]
+ storage=storage,
+ storage_scope=storage_scope,
+ read_scope=read_scope,
+ input_name=input_name,
+ )
# Handle plain lists (might contain ImageContent, DocumentContent, or StructuredContent)
if isinstance(value, list):
- return await _normalize_list(items=value, storage=storage, storage_scope=storage_scope) # pyright: ignore[reportUnknownArgumentType]
+ return await _normalize_list(
+ items=value, # pyright: ignore[reportUnknownArgumentType]
+ storage=storage,
+ storage_scope=storage_scope,
+ read_scope=read_scope,
+ input_name=input_name,
+ )
# Other types don't need normalization
return value, False
@@ -113,6 +146,8 @@ async def _normalize_structured_content(
*,
storage: StorageProviderAbstract,
storage_scope: str,
+ read_scope: str | None,
+ input_name: str,
) -> tuple[StructuredContent, bool]:
"""Normalize a StructuredContent by recursively processing all its fields.
@@ -121,6 +156,8 @@ async def _normalize_structured_content(
storage: The storage provider to use.
storage_scope: The run's opaque storage prefix; normalized bytes land
under `{storage_scope}/assets/`.
+ read_scope: The run's read scope, which every url must satisfy before it is read or linked.
+ input_name: The input the value belongs to, named in a refusal.
Returns:
A tuple of (normalized_content, has_changed).
@@ -129,7 +166,9 @@ async def _normalize_structured_content(
has_changes = False
for field_name, field_value in structured_content:
- normalized_value, changed = await _normalize_value(value=field_value, storage=storage, storage_scope=storage_scope)
+ normalized_value, changed = await _normalize_value(
+ value=field_value, storage=storage, storage_scope=storage_scope, read_scope=read_scope, input_name=input_name
+ )
if changed:
updates[field_name] = normalized_value
has_changes = True
@@ -147,6 +186,8 @@ async def _normalize_list_content(
*,
storage: StorageProviderAbstract,
storage_scope: str,
+ read_scope: str | None,
+ input_name: str,
) -> tuple[ListContent[Any], bool]:
"""Normalize a ListContent by processing all its items.
@@ -155,6 +196,8 @@ async def _normalize_list_content(
storage: The storage provider to use.
storage_scope: The run's opaque storage prefix; normalized bytes land
under `{storage_scope}/assets/`.
+ read_scope: The run's read scope, which every url must satisfy before it is read or linked.
+ input_name: The input the value belongs to, named in a refusal.
Returns:
A tuple of (normalized_list_content, has_changed).
@@ -163,7 +206,9 @@ async def _normalize_list_content(
if not raw_items:
return list_content, False
- normalized_items, has_changes = await _normalize_list(items=raw_items, storage=storage, storage_scope=storage_scope) # pyright: ignore[reportUnknownArgumentType]
+ normalized_items, has_changes = await _normalize_list(
+ items=raw_items, storage=storage, storage_scope=storage_scope, read_scope=read_scope, input_name=input_name
+ ) # pyright: ignore[reportUnknownArgumentType]
if not has_changes:
return list_content, False
@@ -184,6 +229,8 @@ async def _normalize_list(
*,
storage: StorageProviderAbstract,
storage_scope: str,
+ read_scope: str | None,
+ input_name: str,
) -> tuple[list[Any], bool]:
"""Normalize a list by processing all its items.
@@ -192,6 +239,8 @@ async def _normalize_list(
storage: The storage provider to use.
storage_scope: The run's opaque storage prefix; normalized bytes land
under `{storage_scope}/assets/`.
+ read_scope: The run's read scope, which every url must satisfy before it is read or linked.
+ input_name: The input the value belongs to, named in a refusal.
Returns:
A tuple of (normalized_items, has_changed).
@@ -200,7 +249,9 @@ async def _normalize_list(
has_changes = False
for item in items:
- normalized_item, changed = await _normalize_value(value=item, storage=storage, storage_scope=storage_scope)
+ normalized_item, changed = await _normalize_value(
+ value=item, storage=storage, storage_scope=storage_scope, read_scope=read_scope, input_name=input_name
+ )
normalized_items.append(normalized_item)
if changed:
has_changes = True
@@ -213,6 +264,8 @@ async def _normalize_url_content(
*,
storage: StorageProviderAbstract,
storage_scope: str,
+ read_scope: str | None,
+ input_name: str,
) -> NormalizableContent:
"""Normalize ImageContent or DocumentContent to a storage reference and give it a public_url.
@@ -229,6 +282,8 @@ async def _normalize_url_content(
storage: The storage provider to use.
storage_scope: The run's opaque storage prefix; normalized bytes land
under `{storage_scope}/assets/`.
+ read_scope: The run's read scope, which every url must satisfy before it is read or linked.
+ input_name: The input the value belongs to, named in a refusal.
Returns:
The original content if no normalization needed, or a new instance
@@ -239,11 +294,19 @@ async def _normalize_url_content(
PipelineInputUrlInvalidError: If an http(s) url does not parse as one.
PipelineInputContentError: If a local path cannot be read, or the storage
provider refuses a pipelex-storage:// reference as a key.
+ UriReadRefusedError: If the url is a local path, or a storage reference outside
+ the read scope, on a run with a read scope.
"""
if not content.url.strip():
msg = f"{type(content).__name__} input has a blank url — provide https://, data:, pipelex-storage://, or a local file path."
raise PipelineInputUrlMissingError(msg)
+ # Before anything reads or links the url. A signed link is a read by whoever holds it, so a
+ # foreign storage reference is refused here rather than signed; a local path is refused rather
+ # than uploaded, or kept for the prompt preparation to read later, which is what happens to it
+ # when local uploads are off.
+ authorize_uri_read(uri=content.url, read_scope=read_scope, position=f"the file given for input '{input_name}'")
+
resolved_uri = resolve_uri(content.url)
if isinstance(resolved_uri, ResolvedHttpUrl):
diff --git a/pipelex/pipeline/pipeline_run_setup.py b/pipelex/pipeline/pipeline_run_setup.py
index 44fc45a53..ec4e96179 100644
--- a/pipelex/pipeline/pipeline_run_setup.py
+++ b/pipelex/pipeline/pipeline_run_setup.py
@@ -30,7 +30,12 @@ from pipelex.system.environment import get_optional_env
from pipelex.system.job_metadata import OtelContext
from pipelex.system.pipe_run_mode import PipeRunMode
from pipelex.system.run_extras import validate_run_extras
-from pipelex.system.storage_scope import LOCAL_STORAGE_SCOPE, validate_storage_scope
+from pipelex.system.storage_scope import (
+ LOCAL_STORAGE_SCOPE,
+ validate_read_scope,
+ validate_storage_scope,
+ validate_storage_scope_within_read_scope,
+)
from pipelex.system.telemetry.events import EventName, EventProperty
from pipelex.system.telemetry.otel_constants import OTelConstants
from pipelex.system.telemetry.otel_factory import OtelFactory
@@ -58,6 +63,7 @@ async def pipeline_run_setup(
is_mock_usage: bool = False,
user_id: str,
storage_scope: str,
+ read_scope: str | None,
extras: dict[str, str] | None = None,
pipeline_run_id: str | None = None,
request_id: str | None = None,
@@ -126,6 +132,14 @@ async def pipeline_run_setup(
Opaque prefix under which every byte this run writes must land. REQUIRED,
validated at ``JobMetadata`` construction. See
:mod:`pipelex.system.storage_scope`.
+ read_scope:
+ Opaque prefix every storage key this run reads must lie under, which also
+ forbids the run any read from the local disk. REQUIRED, and ``None`` is
+ the explicit statement that the run is unscoped: a local run, or a server
+ with a single tenant. A set read scope must contain ``storage_scope``, so
+ a host that passes one also passes its own storage scope rather than the
+ local sentinel. See :mod:`pipelex.system.storage_scope` and
+ :mod:`pipelex.tools.uri.uri_read_scope`.
extras:
Opaque, host-supplied mapping of labels about this run — the hosted
platform sends its organization, a single-user deployment sends nothing.
@@ -213,6 +227,22 @@ async def pipeline_run_setup(
# a valid one-segment scope — so the sentinel is not disturbed.
storage_scope = validate_storage_scope(value=storage_scope)
+ # And the read scope beside it, with its relation to the storage scope, still
+ # above everything this function causes. A run with a read scope reads only
+ # under it, and it reads its own outputs back, so its storage scope must lie
+ # under it. The local sentinel is refused outright beside a read scope rather
+ # than checked after the swap below: it becomes the run id, which lies under
+ # no host's prefix, and a host that scopes reads scopes writes too.
+ if read_scope is not None:
+ read_scope = validate_read_scope(value=read_scope)
+ if storage_scope == LOCAL_STORAGE_SCOPE:
+ msg = (
+ "A run with a read_scope must pass its own storage_scope: the local storage scope becomes the run id, "
+ "which lies under no read_scope, and the run could not read what it writes."
+ )
+ raise ValueError(msg)
+ validate_storage_scope_within_read_scope(storage_scope=storage_scope, read_scope=read_scope)
+
# TODO: rethink this, it's not forcing
if pipe_run_mode is None:
if run_mode_from_env := get_optional_env(key=FORCE_DRY_RUN_MODE_ENV_KEY):
@@ -377,6 +407,7 @@ async def pipeline_run_setup(
pipeline_run_id=pipeline_run_id,
user_id=user_id,
storage_scope=storage_scope,
+ read_scope=read_scope,
extras=extras,
inputs=inputs,
search_scope=search_scope,
diff --git a/pipelex/pipeline/runner.py b/pipelex/pipeline/runner.py
index 6aa47dc3a..4db7a63df 100644
--- a/pipelex/pipeline/runner.py
+++ b/pipelex/pipeline/runner.py
@@ -116,10 +116,15 @@ class PipelexMTHDSProtocol(MTHDSProtocol["PipeOutput"]):
is_mock_usage: bool = False,
# A local run has exactly one user and no tenancy, so these say "local"
# rather than being derived from each other or defaulted deep in the
- # call stack. A multi-tenant host passes its own; it cannot reach these
- # by omission, because `pipeline_run_setup` requires both explicitly.
+ # call stack. `pipeline_run_setup` requires both, but this protocol
+ # always forwards its own, so a multi-tenant host building it must pass
+ # its own identity here, and its read scope below.
user_id: str = LOCAL_USER_ID,
storage_scope: str = LOCAL_STORAGE_SCOPE,
+ # Unscoped reads, which is the truth on a laptop: the run may read any
+ # stored key and any local file. A host that serves more than one tenant
+ # passes its own prefix, beside its own storage scope.
+ read_scope: str | None = None,
# Opaque labels the host attaches to every run this protocol starts.
# No local default: a laptop belongs to no organization, and inventing
# a label here would put every such run into one shared entity.
@@ -140,6 +145,7 @@ class PipelexMTHDSProtocol(MTHDSProtocol["PipeOutput"]):
self.is_mock_usage = is_mock_usage
self.user_id = user_id
self.storage_scope = storage_scope
+ self.read_scope = read_scope
self.extras = extras
self.execution_config = execution_config
self._pipe_run = pipe_run
@@ -265,6 +271,7 @@ class PipelexMTHDSProtocol(MTHDSProtocol["PipeOutput"]):
is_mock_usage=self.is_mock_usage,
user_id=self.user_id,
storage_scope=self.storage_scope,
+ read_scope=self.read_scope,
extras=self.extras,
inputs_base_dir=self.inputs_base_dir,
library_dirs_are_callers=self.library_dirs_are_callers,
diff --git a/pipelex/runtime_bridge/payloads.py b/pipelex/runtime_bridge/payloads.py
index a21c3845b..d6a1ad641 100644
--- a/pipelex/runtime_bridge/payloads.py
+++ b/pipelex/runtime_bridge/payloads.py
@@ -8,9 +8,9 @@ any dispatch logic so they can be referenced (e.g. by the orchestrator SPI /
import cycle: pydantic, the orchestration/delivery types, stdlib typing only.
"""
-from typing import Any
+from typing import Any, Self
-from pydantic import BaseModel, ConfigDict, Field, field_validator
+from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
from pipelex.runtime_bridge.delivery_mode import DeliveryMode
from pipelex.runtime_bridge.orchestration_mode import DIRECT_ORCHESTRATION_MODE
@@ -19,7 +19,7 @@ from pipelex.runtime_bridge.orchestration_mode import DIRECT_ORCHESTRATION_MODE
# `extras` each pull in `re` and nothing else, so validating here
# costs the boundary nothing.
from pipelex.system.run_extras import validate_run_extras
-from pipelex.system.storage_scope import validate_storage_scope
+from pipelex.system.storage_scope import validate_read_scope, validate_storage_scope, validate_storage_scope_within_read_scope
class PipelexPipeRunInput(BaseModel):
@@ -52,6 +52,27 @@ class PipelexPipeRunInput(BaseModel):
"""
return validate_storage_scope(value=value)
+ # The prefix every storage key the run reads must lie under, which also
+ # forbids it any read from the local disk; `None` says the run is unscoped.
+ # REQUIRED like the two above, and for the same reason: a host that forgot it
+ # would otherwise run unscoped, which reads every key, without a word. See
+ # `pipelex.tools.uri.uri_read_scope`.
+ read_scope: str | None
+
+ @field_validator("read_scope")
+ @classmethod
+ def _validate_read_scope(cls, value: str | None) -> str | None:
+ """Refuse an unusable read scope at the WIRE, for the storage scope's reason."""
+ if value is None:
+ return None
+ return validate_read_scope(value=value)
+
+ @model_validator(mode="after")
+ def _validate_storage_scope_within_read_scope(self) -> Self:
+ """Refuse, at the WIRE, a run that could not read what it writes."""
+ validate_storage_scope_within_read_scope(storage_scope=self.storage_scope, read_scope=self.read_scope)
+ return self
+
# The opaque labels the host attaches to this run. Unlike the two fields
# above it DEFAULTS: a host with no labels to send invents nothing by
# staying silent, whereas a missing identity or scope used to be invented
diff --git a/pipelex/system/job_metadata.py b/pipelex/system/job_metadata.py
index a10b9f88e..ead1dbf8d 100644
--- a/pipelex/system/job_metadata.py
+++ b/pipelex/system/job_metadata.py
@@ -2,12 +2,12 @@ import re
from contextlib import AbstractContextManager
from datetime import datetime
from enum import StrEnum
-from typing import Any
+from typing import Any, Self
-from pydantic import BaseModel, ConfigDict, Field, field_validator
+from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
from pipelex.system.run_extras import validate_run_extras
-from pipelex.system.storage_scope import validate_storage_scope
+from pipelex.system.storage_scope import validate_read_scope, validate_storage_scope, validate_storage_scope_within_read_scope
from pipelex.system.telemetry.otel_context import OtelContext
from pipelex.system.trace_context import TraceContext
from pipelex.tools.log.log_context import LogContext, bind_log_context
@@ -135,6 +135,20 @@ class RunMetadata(BaseModel):
# `DRY_RUN_STORAGE_SCOPE`.
storage_scope: str
+ # The prefix every storage key this run reads must lie under, and the
+ # statement that it reads nothing from the local disk. Opaque like the
+ # storage scope, which must lie under it, and supplied by the host — see
+ # `pipelex.system.storage_scope` for why it is not derived from the storage
+ # scope, and `pipelex.tools.uri.uri_read_scope` for the check.
+ #
+ # REQUIRED, with no default, for the storage scope's reason: a default on a
+ # tenancy field is how a missing value becomes a present-looking one. Here
+ # the present-looking value would be `None`, which reads everything, so a
+ # host that forgot the field would open every key without a word. `None` is
+ # the explicit statement that the run is unscoped: a local run, or a server
+ # with a single tenant.
+ read_scope: str | None
+
# The API-inbound ``X-Request-ID`` (set by the dispatcher when an external
# HTTP request enters Pipelex). Rides here so it crosses the Temporal
# serialization boundary — every activity / workflow can correlate logs and
@@ -176,6 +190,20 @@ class RunMetadata(BaseModel):
"""
return validate_storage_scope(value=value)
+ @field_validator("read_scope")
+ @classmethod
+ def _validate_read_scope(cls, value: str | None) -> str | None:
+ """Refuse a read scope that is not a path-safe prefix, at construction."""
+ if value is None:
+ return None
+ return validate_read_scope(value=value)
+
+ @model_validator(mode="after")
+ def _validate_storage_scope_within_read_scope(self) -> Self:
+ """Refuse a run that could not read what it writes, at construction rather than at its first read."""
+ validate_storage_scope_within_read_scope(storage_scope=self.storage_scope, read_scope=self.read_scope)
+ return self
+
@field_validator("request_id")
@classmethod
def _validate_request_id(cls, value: str | None) -> str | None:
diff --git a/pipelex/system/storage_scope.py b/pipelex/system/storage_scope.py
index aae543aff..10a234042 100644
--- a/pipelex/system/storage_scope.py
+++ b/pipelex/system/storage_scope.py
@@ -22,6 +22,25 @@ unusable, and without an explicit validator here that change would silently
delete an existing traversal control. So the constraint moved to the type: a
`JobMetadata` cannot be constructed with a scope that is not path-safe, which
makes every downstream key safe by construction rather than by remembering.
+
+**`read_scope` — the second opaque string, deciding what a run may read.** The
+storage scope says where a run's bytes go; it says nothing about what the run
+may read, and a value can carry any URL a method chose: a `pipelex-storage://`
+key the storage provider reads with the process's bucket-wide credentials, or a
+path it reads from the worker's own disk. On a multi-tenant host that is a read
+of another tenant's file. So the host supplies a second prefix, the read scope:
+every storage key the run reads must lie under it, and the run reads nothing
+from the local disk. The hosted platform passes the organization id, which
+covers the organization's uploads and the outputs of its earlier runs, both
+outside the current run's storage scope; a run without a read scope (a laptop, a
+single-tenant server) reads as it always did. The check itself lives in
+`pipelex.tools.uri.uri_read_scope`.
+
+It is a separate value supplied by the host, not a parse of the storage scope.
+Taking the first segment of `<org_id>/<method_id>/<run_id>` would teach the
+runtime the hosted layout this module exists to keep out, and it would be wrong
+everywhere else: a local run's storage scope is its run id, which would confine
+it to its own files.
"""
from __future__ import annotations
@@ -140,3 +159,69 @@ def validate_storage_scope(*, value: str) -> str:
)
raise ValueError(msg)
return value
+
+
+def validate_read_scope(*, value: str) -> str:
+ """Return `value` if it is a usable read scope, else raise `ValueError`.
+
+ The same segment rule as the storage scope, which it must contain: one to
+ three `[A-Za-z0-9_-]+` segments joined by single slashes, applied with
+ `fullmatch` for the reason `validate_storage_scope` gives.
+
+ Raises:
+ ValueError: the scope is empty, has more than three segments, or
+ contains a segment that is not `[A-Za-z0-9_-]+`.
+ """
+ if not STORAGE_SCOPE_PATTERN.fullmatch(value):
+ msg = (
+ f"Invalid read_scope {value!r}: expected one to three path-safe segments "
+ "separated by single slashes (e.g. 'tenant'). Empty segments, '.', '..' and "
+ "leading or trailing slashes are refused — the value bounds every storage key the run may read."
+ )
+ raise ValueError(msg)
+ return value
+
+
+def is_key_within_read_scope(*, key: str, read_scope: str) -> bool:
+ """Whether the storage `key` lies under `read_scope`, compared segment by segment.
+
+ By segment and never by string prefix, because `org_abc` is a string prefix
+ of `org_abcdef`. The key's own segments are checked too: an empty, `.` or
+ `..` segment, or a backslash, is refused wherever it sits, because the local
+ storage provider resolves a key against a directory, where
+ `org_a/../org_b/x.png` passes a lexical prefix test and reads from `org_b`,
+ and a backslash is a separator on Windows.
+ """
+ if "\\" in key:
+ return False
+ key_segments = key.split("/")
+ if any(segment in {"", ".", ".."} for segment in key_segments):
+ return False
+ scope_segments = read_scope.split("/")
+ # Strictly longer than the scope: the scope names a namespace, and a key equal
+ # to it names no object inside that namespace.
+ return len(key_segments) > len(scope_segments) and key_segments[: len(scope_segments)] == scope_segments
+
+
+def validate_storage_scope_within_read_scope(*, storage_scope: str, read_scope: str | None) -> None:
+ """Raise `ValueError` if a run could not read what it writes.
+
+ A run reads its own outputs back — a generated image shown to the next
+ model, a page view of its own extraction — so a storage scope outside the
+ read scope would make every such read a refusal. A host that passes the two
+ mismatched fails here, at construction, rather than at the first read. An
+ unscoped run (`read_scope` is `None`) reads everything and needs no check.
+
+ Raises:
+ ValueError: the storage scope does not lie under the read scope.
+ """
+ if read_scope is None:
+ return
+ storage_segments = storage_scope.split("/")
+ scope_segments = read_scope.split("/")
+ if storage_segments[: len(scope_segments)] != scope_segments:
+ msg = (
+ "The run's storage_scope does not lie under its read_scope: a run must be able to read "
+ "what it writes. Pass a read_scope that is the storage_scope or one of its leading segments."
+ )
+ raise ValueError(msg)
diff --git a/pipelex/tools/uri/exceptions.py b/pipelex/tools/uri/exceptions.py
new file mode 100644
index 000000000..0eda7a796
--- /dev/null
+++ b/pipelex/tools/uri/exceptions.py
@@ -0,0 +1,45 @@
+from enum import StrEnum
+
+from pipelex.base_exceptions import ErrorDomain, SecurityError
+
+
+class UriReadRefusalReason(StrEnum):
+ """Why a run was refused a read, as the caller can act on it."""
+
+ FOREIGN_STORAGE_KEY = "foreign_storage_key"
+ LOCAL_PATH = "local_path"
+
+ @property
+ def desc(self) -> str:
+ match self:
+ case UriReadRefusalReason.FOREIGN_STORAGE_KEY:
+ return "it names a stored file this run may not read"
+ case UriReadRefusalReason.LOCAL_PATH:
+ return "it names a file on the server's own disk, or a URL in a scheme this runtime does not read, and a hosted run reads neither"
+
+
+class UriReadRefusedError(SecurityError):
+ """Raised when a run asks to read a URL outside what its host lets it read.
+
+ On a run whose host supplied a read scope, a ``pipelex-storage://`` key is read only when it
+ lies under that scope, and a bare path or a ``file://`` URI is never read (see
+ :mod:`pipelex.tools.uri.uri_read_scope`). A value can carry any URL its method chose — a
+ construct that assembles one from plain text, a model's structured output, a function's
+ result — so the refusal happens where the value is read, whichever pipe produced it.
+
+ A :class:`SecurityError`, so that a domain-level ``except`` around a read cannot swallow it.
+ ``error_domain = INPUT``, answered as a 422, because the method or its inputs asked for the
+ read and the caller fixes it by passing the file as an input or an upload. The message is
+ caller-facing and keeps its text under STRICT disclosure: it names where the URL sat (an
+ image of a pipe's prompt, an input) and why it was refused, and quotes neither the URL nor the
+ read scope. The refusal happens before any IO, so it says nothing about whether the file
+ exists.
+ """
+
+ error_domain = ErrorDomain.INPUT
+ _declared_title = "Read outside the run's scope refused"
+ _authors_caller_facing_message = True
+
+ def __init__(self, message: str, *, reason: UriReadRefusalReason):
+ super().__init__(message)
+ self.reason = reason
diff --git a/pipelex/tools/uri/uri_read_scope.py b/pipelex/tools/uri/uri_read_scope.py
new file mode 100644
index 000000000..2756fa204
--- /dev/null
+++ b/pipelex/tools/uri/uri_read_scope.py
@@ -0,0 +1,76 @@
+"""The read scope's check: may this run read the URL a value carries?
+
+A run reads whatever URL its values carry, and a value can carry any URL its method chose — a
+construct that assembles one from plain text, a model's structured output, a function's result.
+Policing where values are born is hopeless, so the rule is enforced where they are read: in the
+content-generation leaves, before the dry-run branch and before any worker is built, and at the
+input seam, before an input is uploaded or linked. See `pipelex.system.storage_scope` for where
+the read scope comes from.
+
+On a run that carries a read scope:
+
+- a ``pipelex-storage://`` key is read only when it lies under the read scope, compared segment by
+ segment, and none of its segments is empty, ``.`` or ``..``;
+- a bare path or a ``file://`` URI is never read, and neither is anything ``resolve_uri`` takes
+ for a local path, such as a URL in a scheme it does not know;
+- ``https://``, ``http://`` and ``data:`` are untouched: a data URL carries its own bytes, and an
+ outbound fetch is the SSRF guard's concern.
+
+A run without a read scope reads exactly as it always did.
+
+The check performs no IO, so a refusal says nothing about whether the file exists.
+"""
+
+from typing import NamedTuple
+
+from pipelex.system.storage_scope import is_key_within_read_scope
+from pipelex.tools.storage.storage_provider_abstract import PIPELEX_STORAGE_SCHEME
+from pipelex.tools.uri.exceptions import UriReadRefusalReason, UriReadRefusedError
+from pipelex.tools.uri.resolved_uri import ResolvedBase64DataUrl, ResolvedHttpUrl, ResolvedLocalPath, ResolvedPipelexStorage
+from pipelex.tools.uri.uri_resolver import resolve_uri
+
+
+class UriReference(NamedTuple):
+ """A URL a payload will read, and where it sits, for a refusal to name."""
+
+ uri: str
+ # Where the URL sits, in the caller's words: "image 2 of the prompt", "the document to extract".
+ position: str
+
+
+def authorize_uri_read(*, uri: str, read_scope: str | None, position: str) -> None:
+ """Refuse the read of `uri` if the run's read scope does not allow it.
+
+ Args:
+ uri: The URL about to be read.
+ read_scope: The run's read scope, ``None`` for an unscoped run, which reads everything.
+ position: Where the URL sits, named in the refusal instead of the URL itself.
+
+ Raises:
+ UriReadRefusedError: the URL is a storage key outside the read scope, or a local path.
+ """
+ if read_scope is None:
+ return
+ resolved_uri = resolve_uri(uri)
+ match resolved_uri:
+ case ResolvedHttpUrl() | ResolvedBase64DataUrl():
+ return
+ case ResolvedPipelexStorage():
+ key = resolved_uri.storage_uri.removeprefix(PIPELEX_STORAGE_SCHEME)
+ if is_key_within_read_scope(key=key, read_scope=read_scope):
+ return
+ reason = UriReadRefusalReason.FOREIGN_STORAGE_KEY
+ case ResolvedLocalPath():
+ reason = UriReadRefusalReason.LOCAL_PATH
+ msg = f"Refused to read {position}: {reason.desc}. Pass the file as an input of the run, or upload it first."
+ raise UriReadRefusedError(msg, reason=reason)
+
+
+def authorize_uri_reads(*, uri_references: list[UriReference], read_scope: str | None) -> None:
+ """Refuse the first of `uri_references` the run's read scope does not allow.
+
+ Raises:
+ UriReadRefusedError: one of the URLs is a storage key outside the read scope, or a local path.
+ """
+ for uri_reference in uri_references:
+ authorize_uri_read(uri=uri_reference.uri, read_scope=read_scope, position=uri_reference.position)
diff --git a/tests/conftest.py b/tests/conftest.py
index 2352d8ae1..e544cd786 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -198,4 +198,4 @@ def job_metadata(request: pytest.FixtureRequest) -> JobMetadata:
random_code: str = shortuuid.uuid()[:5]
pipeline_run_id: str = f"{test_id}-{random_code}"
- return JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="pytest", pipeline_run_id=pipeline_run_id))
+ return JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="pytest", pipeline_run_id=pipeline_run_id))
diff --git a/tests/data/errors/error_identity.txt b/tests/data/errors/error_identity.txt
index 9f6e13734..2a4d3b0be 100644
--- a/tests/data/errors/error_identity.txt
+++ b/tests/data/errors/error_identity.txt
@@ -328,6 +328,7 @@ UnknownStorageMethodError | Unknown storage method | https://docs.pipelex.com/la
UnknownVarPrefixError | Unknown var prefix | https://docs.pipelex.com/latest/errors/unknown-var-prefix-error/
UnsafeSchemaError | Unsafe schema | https://docs.pipelex.com/latest/errors/unsafe-schema-error/
UnusedInputError | Unused input | https://docs.pipelex.com/latest/errors/unused-input-error/
+UriReadRefusedError | Read outside the run's scope refused | https://docs.pipelex.com/latest/errors/uri-read-refused-error/
ValidateBundleError | Validate bundle | https://docs.pipelex.com/latest/errors/validate-bundle-error/
VarFallbackPatternError | Var fallback pattern | https://docs.pipelex.com/latest/errors/var-fallback-pattern-error/
VarNotFoundError | Var not found | https://docs.pipelex.com/latest/errors/var-not-found-error/
diff --git a/tests/integration/pipelex/cogt/content_generation/test_leaf_dry_object_mocks.py b/tests/integration/pipelex/cogt/content_generation/test_leaf_dry_object_mocks.py
index 59eabbffd..1f219bd61 100644
--- a/tests/integration/pipelex/cogt/content_generation/test_leaf_dry_object_mocks.py
+++ b/tests/integration/pipelex/cogt/content_generation/test_leaf_dry_object_mocks.py
@@ -89,7 +89,9 @@ class SpecWithPipeCode(StructuredContent):
def _dry_object_assignment(object_class: type[StructuredContent], nb_items: int | None = None) -> ObjectAssignment:
llm_assignment = LLMAssignment(
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_dry_objects")),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="run_dry_objects")
+ ),
cogt_run_params=CogtRunParams(run_mode=PipeRunMode.DRY),
llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),
llm_prompt=LLMPrompt(user_text="make objects"),
@@ -130,7 +132,9 @@ class TestLeafDryObjectMocks:
async def test_dry_object_list_stamps_regardless_of_mock_usage(self) -> None:
"""The stamp is unconditional on the is_mock_usage sub-flag — it only changes reporting (D3)."""
llm_assignment = LLMAssignment(
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_mock_usage_stamp")),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="run_mock_usage_stamp")
+ ),
cogt_run_params=CogtRunParams(run_mode=PipeRunMode.DRY, is_mock_usage=True),
llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),
llm_prompt=LLMPrompt(user_text="make specs"),
@@ -232,7 +236,9 @@ class TestLeafDryObjectMocks:
async def test_boundary_dry_search_structured_dict_validates_against_original_class(self) -> None:
"""The structured-search dry boundary leaf returns a dict the original output class accepts."""
search_assignment = SearchAssignment(
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_dry_search_structured")),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="run_dry_search_structured")
+ ),
cogt_run_params=CogtRunParams(run_mode=PipeRunMode.DRY),
query="what is pipelex?",
search_setting=SearchSetting(model="mock-search-handle"),
diff --git a/tests/integration/pipelex/cogt/content_generation/test_object_class_passthrough.py b/tests/integration/pipelex/cogt/content_generation/test_object_class_passthrough.py
index 1ce453ec2..856c1d2c8 100644
--- a/tests/integration/pipelex/cogt/content_generation/test_object_class_passthrough.py
+++ b/tests/integration/pipelex/cogt/content_generation/test_object_class_passthrough.py
@@ -80,7 +80,9 @@ class _StubListResponse(BaseModel):
def _live_object_assignment() -> ObjectAssignment:
llm_assignment = LLMAssignment(
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_object_class_passthrough")),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="run_object_class_passthrough")
+ ),
cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),
llm_setting=LLMSetting(model="test-model", temperature=0.5),
llm_prompt=LLMPrompt(user_text="make a name"),
@@ -91,7 +93,7 @@ def _live_object_assignment() -> ObjectAssignment:
def _dry_object_assignment() -> ObjectAssignment:
llm_assignment = LLMAssignment(
job_metadata=JobMetadata(
- run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_object_class_passthrough_dry")
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="run_object_class_passthrough_dry")
),
cogt_run_params=CogtRunParams(run_mode=PipeRunMode.DRY),
llm_setting=LLMSetting(model="test-model", temperature=0.5),
@@ -102,7 +104,9 @@ def _dry_object_assignment() -> ObjectAssignment:
def _live_search_assignment() -> SearchAssignment:
return SearchAssignment(
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_search_class_passthrough")),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="run_search_class_passthrough")
+ ),
cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),
query="make a name",
search_setting=SearchSetting(model="mock-search-handle"),
@@ -238,7 +242,9 @@ class TestObjectClassPassthrough:
ObjectAssignment.make_for_class(
object_class=NormalizedReference,
llm_assignment=LLMAssignment(
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_wrapped")),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="run_wrapped")
+ ),
cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),
llm_setting=LLMSetting(model="test-model", temperature=0.5),
llm_prompt=LLMPrompt(user_text="make a reference"),
diff --git a/tests/integration/pipelex/cogt/image/test_data_url_image_flow.py b/tests/integration/pipelex/cogt/image/test_data_url_image_flow.py
index 2858deeec..c36ce7d11 100644
--- a/tests/integration/pipelex/cogt/image/test_data_url_image_flow.py
+++ b/tests/integration/pipelex/cogt/image/test_data_url_image_flow.py
@@ -85,7 +85,7 @@ class TestDataUrlImageFlow:
working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)
# Normalize
- normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope")
+ normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope", read_scope=None)
# Verify the URL was converted to pipelex-storage://
normalized_stuff = normalized_memory.get_stuff("test_image")
@@ -122,7 +122,7 @@ class TestDataUrlImageFlow:
working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)
# Normalize to storage
- normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope")
+ normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope", read_scope=None)
normalized_stuff = normalized_memory.get_stuff("test_image")
assert isinstance(normalized_stuff.content, ImageContent)
storage_uri = normalized_stuff.content.url
@@ -163,7 +163,7 @@ class TestDataUrlImageFlow:
working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)
# Normalize
- normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope")
+ normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope", read_scope=None)
# Verify all images were converted
normalized_stuff = normalized_memory.get_stuff("image_list")
@@ -194,7 +194,7 @@ class TestDataUrlImageFlow:
working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)
# Normalize
- normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope")
+ normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope", read_scope=None)
# Verify URL was NOT changed
normalized_stuff = normalized_memory.get_stuff("test_image")
@@ -224,7 +224,7 @@ class TestDataUrlImageFlow:
working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)
# Normalize
- normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope")
+ normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope", read_scope=None)
# Verify the embedded image URL was converted to pipelex-storage://
normalized_stuff = normalized_memory.get_stuff("test_article")
@@ -264,7 +264,7 @@ class TestDataUrlImageFlow:
working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)
# Normalize
- normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope")
+ normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope", read_scope=None)
# Verify all image URLs were converted
normalized_stuff = normalized_memory.get_stuff("nested_article")
@@ -314,7 +314,7 @@ class TestDataUrlImageFlow:
working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)
# Normalize
- normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope")
+ normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope", read_scope=None)
# Verify all images in the list were converted
normalized_stuff = normalized_memory.get_stuff("gallery")
diff --git a/tests/integration/pipelex/cogt/test_llm_generate_security.py b/tests/integration/pipelex/cogt/test_llm_generate_security.py
index e47854c37..d7ef9aca5 100644
--- a/tests/integration/pipelex/cogt/test_llm_generate_security.py
+++ b/tests/integration/pipelex/cogt/test_llm_generate_security.py
@@ -21,7 +21,9 @@ from pipelex.system.pipe_run_mode import PipeRunMode
def _make_stub_llm_assignment() -> LLMAssignment:
return LLMAssignment(
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run")),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="test-user", pipeline_run_id="test-run")
+ ),
cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),
llm_setting=LLMSetting(model="test-model", temperature=0.7),
llm_prompt=LLMPrompt(user_text="test prompt"),
diff --git a/tests/integration/pipelex/core/memory/test_working_memory_factory.py b/tests/integration/pipelex/core/memory/test_working_memory_factory.py
index dc386e65a..ff0f52637 100644
--- a/tests/integration/pipelex/core/memory/test_working_memory_factory.py
+++ b/tests/integration/pipelex/core/memory/test_working_memory_factory.py
@@ -26,7 +26,9 @@ class TestWorkingMemoryFactory:
},
}
- working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(pipeline_inputs=pipeline_inputs, concept_provider=get_concept_library())
+ working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(
+ pipeline_inputs=pipeline_inputs, concept_provider=get_concept_library(), read_scope=None
+ )
assert working_memory is not None
assert "text_item" in working_memory.root
@@ -62,7 +64,9 @@ class TestWorkingMemoryFactory:
},
}
- working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(pipeline_inputs=pipeline_inputs, concept_provider=get_concept_library())
+ working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(
+ pipeline_inputs=pipeline_inputs, concept_provider=get_concept_library(), read_scope=None
+ )
assert working_memory is not None
assert "complex_page" in working_memory.root
@@ -111,7 +115,9 @@ class TestWorkingMemoryFactory:
"""Test deserialization of empty compact memory."""
pipeline_inputs: PipelineInputs = {}
- working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(pipeline_inputs=pipeline_inputs, concept_provider=get_concept_library())
+ working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(
+ pipeline_inputs=pipeline_inputs, concept_provider=get_concept_library(), read_scope=None
+ )
assert working_memory is not None
assert len(working_memory.root) == 0
@@ -130,7 +136,9 @@ class TestWorkingMemoryFactory:
},
}
- working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(pipeline_inputs=pipeline_inputs, concept_provider=get_concept_library())
+ working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(
+ pipeline_inputs=pipeline_inputs, concept_provider=get_concept_library(), read_scope=None
+ )
assert working_memory is not None
assert len(working_memory.root) == 2
@@ -155,7 +163,7 @@ class TestWorkingMemoryFactory:
pipeline_inputs: dict[str, Any] = {"greeting": "hello"}
working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(
- pipeline_inputs=pipeline_inputs, input_specs=None, concept_provider=get_concept_library()
+ pipeline_inputs=pipeline_inputs, input_specs=None, concept_provider=get_concept_library(), read_scope=None
)
stuff = working_memory.root["greeting"]
@@ -173,7 +181,7 @@ class TestWorkingMemoryFactory:
pipeline_inputs: dict[str, Any] = {"count": 42}
working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(
- pipeline_inputs=pipeline_inputs, input_specs=input_specs, concept_provider=get_concept_library()
+ pipeline_inputs=pipeline_inputs, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None
)
stuff = working_memory.root["count"]
diff --git a/tests/integration/pipelex/csv/test_csv_input_detection.py b/tests/integration/pipelex/csv/test_csv_input_detection.py
index b23f35cf1..67532e418 100644
--- a/tests/integration/pipelex/csv/test_csv_input_detection.py
+++ b/tests/integration/pipelex/csv/test_csv_input_detection.py
@@ -60,7 +60,7 @@ class TestCsvInputDetection:
load_test_library([BUNDLE_DIR])
inputs: PipelineInputs = {"people": {"concept": "csv_demo.Person", "content": {"url": remote_url}}}
with pytest.raises(CsvError) as exc_info:
- WorkingMemoryFactory.make_from_pipeline_inputs(inputs, concept_provider=get_concept_library())
+ WorkingMemoryFactory.make_from_pipeline_inputs(inputs, concept_provider=get_concept_library(), read_scope=None)
message = str(exc_info.value)
assert "local" in message.lower()
# The path is preserved so the user can identify the offending input...
@@ -80,7 +80,7 @@ class TestCsvInputDetection:
assert resolved["people"]["content"]["url"] == "s3://bucket/people.csv" # not rewritten
inputs = cast("PipelineInputs", resolved)
with pytest.raises(CsvError) as exc_info:
- WorkingMemoryFactory.make_from_pipeline_inputs(inputs, concept_provider=get_concept_library())
+ WorkingMemoryFactory.make_from_pipeline_inputs(inputs, concept_provider=get_concept_library(), read_scope=None)
assert "local" in str(exc_info.value).lower()
def test_xlsx_input_routes_to_tabular_seam(self, load_test_library: Callable[[list[Path]], None]) -> None:
@@ -89,7 +89,7 @@ class TestCsvInputDetection:
# raises a clear "needs pipelex[tabular]" CsvError (the Excel backend isn't built in v1).
inputs: PipelineInputs = {"people": {"concept": "csv_demo.Person", "content": {"url": "people.xlsx"}}}
with pytest.raises(CsvError) as exc_info:
- WorkingMemoryFactory.make_from_pipeline_inputs(inputs, concept_provider=get_concept_library())
+ WorkingMemoryFactory.make_from_pipeline_inputs(inputs, concept_provider=get_concept_library(), read_scope=None)
assert "pipelex[tabular]" in str(exc_info.value)
def test_unparseable_url_rejected_without_leak(self, load_test_library: Callable[[list[Path]], None]) -> None:
@@ -99,7 +99,7 @@ class TestCsvInputDetection:
bad_url = "https://[bad/people.csv?token=abc"
inputs: PipelineInputs = {"people": {"concept": "csv_demo.Person", "content": {"url": bad_url}}}
with pytest.raises(CsvError) as exc_info:
- WorkingMemoryFactory.make_from_pipeline_inputs(inputs, concept_provider=get_concept_library())
+ WorkingMemoryFactory.make_from_pipeline_inputs(inputs, concept_provider=get_concept_library(), read_scope=None)
message = str(exc_info.value)
assert "local" in message.lower()
# The unparseable url can't be sanitized, so it is not echoed at all — the token must not leak.
@@ -109,7 +109,7 @@ class TestCsvInputDetection:
load_test_library([URL_FIELD_DIR])
# url field value that is NOT a tabular suffix → ordinary single record, not a CSV table.
inputs: PipelineInputs = {"link": {"concept": "url_field_concept.Link", "content": {"label": "Home", "url": "https://example.com"}}}
- working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(inputs, concept_provider=get_concept_library())
+ working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(inputs, concept_provider=get_concept_library(), read_scope=None)
content = working_memory.get_stuff("link").content
assert not isinstance(content, ListContent)
@@ -122,7 +122,7 @@ class TestCsvInputDetection:
# A record that has sibling keys alongside a .csv-suffixed `url` is NOT a table reference:
# detection is gated to the bare {"url": ...} wrapper, so the siblings are never dropped.
inputs: PipelineInputs = {"link": {"concept": "url_field_concept.Link", "content": {"label": "Home", "url": "report.csv"}}}
- working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(inputs, concept_provider=get_concept_library())
+ working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(inputs, concept_provider=get_concept_library(), read_scope=None)
content = working_memory.get_stuff("link").content
assert not isinstance(content, ListContent)
@@ -134,7 +134,7 @@ class TestCsvInputDetection:
load_test_library([URL_FIELD_DIR])
# The file-level url ends .csv → read as a table; the inner `url` column is plain data.
inputs: PipelineInputs = {"links": {"concept": "url_field_concept.Link", "content": {"url": str(LINKS_CSV)}}}
- working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(inputs, concept_provider=get_concept_library())
+ working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(inputs, concept_provider=get_concept_library(), read_scope=None)
content = working_memory.get_stuff("links").content
assert isinstance(content, ListContent)
@@ -149,7 +149,7 @@ class TestCsvInputDetection:
people_csv.write_text(PEOPLE_CSV.read_text(encoding="utf-8"), encoding="utf-8")
inputs: PipelineInputs = {"people": {"concept": "csv_demo.Person", "content": {"url": str(people_csv)}}}
- first = WorkingMemoryFactory.make_from_pipeline_inputs(inputs, concept_provider=get_concept_library())
+ first = WorkingMemoryFactory.make_from_pipeline_inputs(inputs, concept_provider=get_concept_library(), read_scope=None)
first_people = cast("ListContent[StuffContent]", first.get_stuff("people").content)
assert len(first_people.items) == 3
@@ -158,6 +158,6 @@ class TestCsvInputDetection:
"name,job,country,birth_year,death_year\nAda Lovelace,Mathematician,United Kingdom,1815,1852\n",
encoding="utf-8",
)
- second = WorkingMemoryFactory.make_from_pipeline_inputs(inputs, concept_provider=get_concept_library())
+ second = WorkingMemoryFactory.make_from_pipeline_inputs(inputs, concept_provider=get_concept_library(), read_scope=None)
second_people = cast("ListContent[StuffContent]", second.get_stuff("people").content)
assert len(second_people.items) == 1
diff --git a/tests/integration/pipelex/csv/test_csv_roundtrip.py b/tests/integration/pipelex/csv/test_csv_roundtrip.py
index 4c16474c4..7116a323a 100644
--- a/tests/integration/pipelex/csv/test_csv_roundtrip.py
+++ b/tests/integration/pipelex/csv/test_csv_roundtrip.py
@@ -52,7 +52,7 @@ class TestCsvRoundtrip:
def test_csv_input_builds_typed_list(self, load_test_library: Callable[[list[Path]], None]) -> None:
load_test_library([BUNDLE_DIR])
- working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(people_inputs(), concept_provider=get_concept_library())
+ working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(people_inputs(), concept_provider=get_concept_library(), read_scope=None)
people = working_memory.get_stuff("people").content
assert isinstance(people, ListContent)
diff --git a/tests/integration/pipelex/fixtures/pipe_job_helpers.py b/tests/integration/pipelex/fixtures/pipe_job_helpers.py
index 4e15a10b0..aafc38ff5 100644
--- a/tests/integration/pipelex/fixtures/pipe_job_helpers.py
+++ b/tests/integration/pipelex/fixtures/pipe_job_helpers.py
@@ -50,7 +50,7 @@ def build_pipe_job(
pipe=pipe,
pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=pipe_run_mode),
job_metadata=JobMetadata(
- run_metadata=RunMetadata(storage_scope="test/scope", user_id=DRY_RUN_USER_ID, pipeline_run_id=resolved_pipeline_run_id)
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id=DRY_RUN_USER_ID, pipeline_run_id=resolved_pipeline_run_id)
),
working_memory=WorkingMemoryFactory.make_empty(),
library_crate=library_crate,
diff --git a/tests/integration/pipelex/kernel/test_usage_parity.py b/tests/integration/pipelex/kernel/test_usage_parity.py
index c3774c716..31cfbf7ec 100644
--- a/tests/integration/pipelex/kernel/test_usage_parity.py
+++ b/tests/integration/pipelex/kernel/test_usage_parity.py
@@ -128,6 +128,7 @@ class TestKernelUsageParity:
try:
kernel = PipelexKernel.make(
storage_scope="test/scope",
+ read_scope=None,
run_mode=PipeRunMode.DRY,
user_id="kernel-usage-parity",
is_mock_usage=True,
diff --git a/tests/integration/pipelex/libraries/test_load_prevalidated_crate.py b/tests/integration/pipelex/libraries/test_load_prevalidated_crate.py
index ca60b17c9..09474882a 100644
--- a/tests/integration/pipelex/libraries/test_load_prevalidated_crate.py
+++ b/tests/integration/pipelex/libraries/test_load_prevalidated_crate.py
@@ -72,6 +72,7 @@ async def _dry_run_main_pipe(*, library_id: str) -> str:
execution_config = get_config().interpreter.pipeline_execution.with_execution_overrides(generate_graph=False, mock_inputs=True)
pipe_job = await prepare_pipe_job(
storage_scope="test/scope",
+ read_scope=None,
user_id="pytest",
pipe=pipe,
library_id=library_id,
diff --git a/tests/integration/pipelex/pipe_signature/test_pipe_signature.py b/tests/integration/pipelex/pipe_signature/test_pipe_signature.py
index 57d327c2c..07c7048b4 100644
--- a/tests/integration/pipelex/pipe_signature/test_pipe_signature.py
+++ b/tests/integration/pipelex/pipe_signature/test_pipe_signature.py
@@ -102,7 +102,9 @@ class TestPipeSignature:
working_memory = WorkingMemoryFactory.make_mock_inputs(needed_inputs=needed)
await runtime.run_pipe(
job_metadata=JobMetadata(
- run_metadata=RunMetadata(storage_scope="test/scope", user_id=DRY_RUN_USER_ID, pipeline_run_id=SpecialPipelineId.DRY_RUN_UNTITLED)
+ run_metadata=RunMetadata(
+ storage_scope="test/scope", read_scope=None, user_id=DRY_RUN_USER_ID, pipeline_run_id=SpecialPipelineId.DRY_RUN_UNTITLED
+ )
),
working_memory=working_memory,
pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.DRY),
@@ -123,7 +125,9 @@ class TestPipeSignature:
working_memory = WorkingMemoryFactory.make_mock_inputs(needed_inputs=needed)
await runtime.run_pipe(
job_metadata=JobMetadata(
- run_metadata=RunMetadata(storage_scope="test/scope", user_id=DRY_RUN_USER_ID, pipeline_run_id=SpecialPipelineId.DRY_RUN_UNTITLED)
+ run_metadata=RunMetadata(
+ storage_scope="test/scope", read_scope=None, user_id=DRY_RUN_USER_ID, pipeline_run_id=SpecialPipelineId.DRY_RUN_UNTITLED
+ )
),
working_memory=working_memory,
pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.DRY),
@@ -146,7 +150,9 @@ class TestPipeSignature:
with pytest.raises(PipeSignatureNotExecutableError) as exc_info:
await runtime._live_run_pipe( # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
job_metadata=JobMetadata(
- run_metadata=RunMetadata(storage_scope="test/scope", user_id=DRY_RUN_USER_ID, pipeline_run_id=SpecialPipelineId.UNTITLED)
+ run_metadata=RunMetadata(
+ storage_scope="test/scope", read_scope=None, user_id=DRY_RUN_USER_ID, pipeline_run_id=SpecialPipelineId.UNTITLED
+ )
),
working_memory=working_memory,
pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.LIVE),
@@ -169,7 +175,9 @@ class TestPipeSignature:
with pytest.raises(PipeSignatureNotExecutableError) as exc_info:
await runtime.run_pipe(
job_metadata=JobMetadata(
- run_metadata=RunMetadata(storage_scope="test/scope", user_id=DRY_RUN_USER_ID, pipeline_run_id=SpecialPipelineId.UNTITLED)
+ run_metadata=RunMetadata(
+ storage_scope="test/scope", read_scope=None, user_id=DRY_RUN_USER_ID, pipeline_run_id=SpecialPipelineId.UNTITLED
+ )
),
working_memory=working_memory,
pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.LIVE),
diff --git a/tests/integration/pipelex/pipeline/test_execution_seams.py b/tests/integration/pipelex/pipeline/test_execution_seams.py
index 76ed78840..640b74dd0 100644
--- a/tests/integration/pipelex/pipeline/test_execution_seams.py
+++ b/tests/integration/pipelex/pipeline/test_execution_seams.py
@@ -144,6 +144,7 @@ class TestExecutionSeams:
pipe = get_required_entry_pipe(pipe_code=qualified_main_pipe)
pipe_job = await prepare_pipe_job(
storage_scope="test/scope",
+ read_scope=None,
user_id="test-user",
pipe=pipe,
library_id=library_id,
@@ -201,6 +202,7 @@ class TestExecutionSeams:
with pytest.raises(ValueError, match="Invalid storage_scope"):
await prepare_pipe_job(
storage_scope=bad_scope,
+ read_scope=None,
pipe=pipe,
library_id=library_id,
execution_config=normalize_config,
@@ -246,6 +248,7 @@ class TestExecutionSeams:
pipe_job = await prepare_pipe_job(
storage_scope="test/scope",
+ read_scope=None,
pipe=pipe,
library_id=library_id,
execution_config=normalize_config,
diff --git a/tests/integration/pipelex/pipeline/test_input_form.py b/tests/integration/pipelex/pipeline/test_input_form.py
index bbfe8d3b5..6b71daf29 100644
--- a/tests/integration/pipelex/pipeline/test_input_form.py
+++ b/tests/integration/pipelex/pipeline/test_input_form.py
@@ -666,6 +666,7 @@ class TestKindAssignmentTable:
{"one": {"name": "widget"}},
input_specs=take_one_pipe.inputs,
concept_provider=get_concept_library(),
+ read_scope=None,
)
content = working_memory.root["one"].content
assert not isinstance(content, ListContent), "the runtime frames a `[1]` slot as the item, never a one-item list"
@@ -674,7 +675,7 @@ class TestKindAssignmentTable:
# And it refuses a list there on the same grounds a bare declaration does.
list_payload = cast("PipelineInputs", {"one": [{"name": "widget"}]})
with pytest.raises(ListWhereSingularError):
- InputShaper.shape(list_payload, input_specs=take_one_pipe.inputs, concept_provider=get_concept_library())
+ InputShaper.shape(list_payload, input_specs=take_one_pipe.inputs, concept_provider=get_concept_library(), read_scope=None)
finally:
_teardown_validation_library(outer_library_id)
diff --git a/tests/integration/pipelex/pipeline/test_optional_method_inputs.py b/tests/integration/pipelex/pipeline/test_optional_method_inputs.py
index cee841b83..58b521ad3 100644
--- a/tests/integration/pipelex/pipeline/test_optional_method_inputs.py
+++ b/tests/integration/pipelex/pipeline/test_optional_method_inputs.py
@@ -65,6 +65,7 @@ class TestOptionalMethodInputs:
pipe_job = await prepare_pipe_job(
storage_scope="test/scope",
+ read_scope=None,
pipe=pipe,
library_id=library_id,
execution_config=execution_config,
@@ -96,6 +97,7 @@ class TestOptionalMethodInputs:
pipe_job = await prepare_pipe_job(
storage_scope="test/scope",
+ read_scope=None,
pipe=pipe,
library_id=library_id,
execution_config=execution_config,
@@ -130,6 +132,7 @@ class TestOptionalMethodInputs:
pipe_job = await prepare_pipe_job(
storage_scope="test/scope",
+ read_scope=None,
pipe=pipe,
library_id=library_id,
execution_config=execution_config,
@@ -177,6 +180,7 @@ class TestOptionalMethodInputs:
pipe_job = await prepare_pipe_job(
storage_scope="test/scope",
+ read_scope=None,
pipe=pipe,
library_id=library_id,
execution_config=execution_config,
diff --git a/tests/integration/pipelex/pipeline/test_pipeline_run_id_resubmission.py b/tests/integration/pipelex/pipeline/test_pipeline_run_id_resubmission.py
index bddcf576e..d2a6af67a 100644
--- a/tests/integration/pipelex/pipeline/test_pipeline_run_id_resubmission.py
+++ b/tests/integration/pipelex/pipeline/test_pipeline_run_id_resubmission.py
@@ -130,6 +130,7 @@ class TestPipelineRunIdResubmission:
with pytest.raises(PipeNotFoundError):
await pipeline_run_setup(
storage_scope="test/scope",
+ read_scope=None,
user_id="test-user",
execution_config=_dry_mock_config(),
mthds_contents=[_RESUBMISSION_MTHDS],
@@ -143,6 +144,7 @@ class TestPipelineRunIdResubmission:
# PipelineManagerAlreadyExistsError).
pipe_job, pipeline_run_id, library_id = await pipeline_run_setup(
storage_scope="test/scope",
+ read_scope=None,
user_id="test-user",
execution_config=_dry_mock_config(),
mthds_contents=[_RESUBMISSION_MTHDS],
@@ -198,6 +200,7 @@ class TestPipelineRunIdResubmission:
with pytest.raises(RuntimeError, match=prepare_failure_msg):
await pipeline_run_setup(
storage_scope="test/scope",
+ read_scope=None,
user_id="test-user",
execution_config=graph_config,
mthds_contents=[_RESUBMISSION_MTHDS],
@@ -218,6 +221,7 @@ class TestPipelineRunIdResubmission:
mocker.stop(teardown_mock)
pipe_job, pipeline_run_id, library_id = await pipeline_run_setup(
storage_scope="test/scope",
+ read_scope=None,
user_id="test-user",
execution_config=_dry_mock_config(),
mthds_contents=[_RESUBMISSION_MTHDS],
diff --git a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_characterization.py b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_characterization.py
index 81f598f86..047cecefa 100644
--- a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_characterization.py
+++ b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_characterization.py
@@ -114,6 +114,7 @@ class TestPipelineRunSetupCharacterization:
pipe_job, pipeline_run_id, library_id = await pipeline_run_setup(
storage_scope="test/scope",
+ read_scope=None,
user_id="test-user",
execution_config=_dry_mock_config(),
mthds_contents=[_CHAR_MTHDS],
@@ -154,6 +155,7 @@ class TestPipelineRunSetupCharacterization:
)
_, _, library_id = await pipeline_run_setup(
storage_scope="test/scope",
+ read_scope=None,
user_id="test-user",
execution_config=_dry_mock_config(),
mthds_contents=[_CHAR_MTHDS],
@@ -170,6 +172,7 @@ class TestPipelineRunSetupCharacterization:
async def test_empty_inputs_behave_like_no_inputs(self) -> None:
pipe_job, _, library_id = await pipeline_run_setup(
storage_scope="test/scope",
+ read_scope=None,
user_id="test-user",
execution_config=_no_mock_config(),
mthds_contents=[_CHAR_MTHDS],
@@ -213,6 +216,7 @@ class TestPipelineRunSetupCharacterization:
with pytest.raises(PipeNotFoundError):
await pipeline_run_setup(
storage_scope="test/scope",
+ read_scope=None,
user_id="test-user",
execution_config=_dry_mock_config(),
mthds_contents=[_CHAR_MTHDS],
@@ -266,6 +270,7 @@ class TestPipelineRunSetupCharacterization:
with pytest.raises(PipeNotFoundError):
await pipeline_run_setup(
storage_scope="test/scope",
+ read_scope=None,
user_id="test-user",
execution_config=_dry_mock_config(),
mthds_contents=[_CHAR_MTHDS],
@@ -293,6 +298,7 @@ class TestPipelineRunSetupCharacterization:
with pytest.raises(PipeNotFoundError):
await pipeline_run_setup(
storage_scope="test/scope",
+ read_scope=None,
user_id="test-user",
execution_config=_dry_mock_config(),
library_id=collide_library_id,
diff --git a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_emit_gates.py b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_emit_gates.py
index 9fe2f8830..6a3172a48 100644
--- a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_emit_gates.py
+++ b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_emit_gates.py
@@ -81,6 +81,7 @@ class TestPipelineRunSetupEmitGates:
pipe_job, pipeline_run_id, library_id = await pipeline_run_setup(
storage_scope="test/scope",
+ read_scope=None,
user_id="test-user",
execution_config=_config(generate_graph=False, generate_usage=True),
mthds_contents=[_GATE_MTHDS],
@@ -105,6 +106,7 @@ class TestPipelineRunSetupEmitGates:
pipe_job, pipeline_run_id, library_id = await pipeline_run_setup(
storage_scope="test/scope",
+ read_scope=None,
user_id="test-user",
execution_config=_config(generate_graph=True, generate_usage=False),
mthds_contents=[_GATE_MTHDS],
@@ -141,6 +143,7 @@ class TestTheLocalScopeIsPerRun:
for _ in range(2):
pipe_job, pipeline_run_id, library_id = await pipeline_run_setup(
storage_scope=LOCAL_STORAGE_SCOPE,
+ read_scope=None,
user_id=LOCAL_USER_ID,
execution_config=_config(generate_graph=False, generate_usage=False),
mthds_contents=[_GATE_MTHDS],
@@ -169,6 +172,7 @@ class TestTheLocalScopeIsPerRun:
"""
pipe_job, pipeline_run_id, library_id = await pipeline_run_setup(
storage_scope="org_a/mt_b/run_c",
+ read_scope=None,
user_id="test-user",
execution_config=_config(generate_graph=False, generate_usage=False),
mthds_contents=[_GATE_MTHDS],
diff --git a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_request_id.py b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_request_id.py
index 3815b6106..974c31a5d 100644
--- a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_request_id.py
+++ b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_request_id.py
@@ -41,6 +41,7 @@ class TestPipelineRunSetupRequestId:
)
pipe_job, _, _ = await pipeline_run_setup(
storage_scope="test/scope",
+ read_scope=None,
user_id="test-user",
execution_config=execution_config,
mthds_contents=[_MINIMAL_MTHDS],
@@ -56,6 +57,7 @@ class TestPipelineRunSetupRequestId:
)
pipe_job, _, _ = await pipeline_run_setup(
storage_scope="test/scope",
+ read_scope=None,
user_id="test-user",
execution_config=execution_config,
mthds_contents=[_MINIMAL_MTHDS],
diff --git a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_run_extras.py b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_run_extras.py
index 53f431674..78749fe3f 100644
--- a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_run_extras.py
+++ b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_run_extras.py
@@ -53,6 +53,7 @@ class TestPipelineRunSetupRunExtras:
)
pipe_job, _, _ = await pipeline_run_setup(
storage_scope="test/scope",
+ read_scope=None,
user_id="test-user",
execution_config=execution_config,
mthds_contents=[_MINIMAL_MTHDS],
@@ -68,6 +69,7 @@ class TestPipelineRunSetupRunExtras:
)
pipe_job, _, _ = await pipeline_run_setup(
storage_scope="test/scope",
+ read_scope=None,
user_id="test-user",
execution_config=execution_config,
mthds_contents=[_MINIMAL_MTHDS],
@@ -83,6 +85,7 @@ class TestPipelineRunSetupRunExtras:
with pytest.raises(ValueError, match="extras"):
await pipeline_run_setup(
storage_scope="test/scope",
+ read_scope=None,
user_id="test-user",
execution_config=execution_config,
mthds_contents=[_MINIMAL_MTHDS],
@@ -112,6 +115,7 @@ class TestPipelineRunSetupRunExtras:
with pytest.raises(ValueError, match="extras"):
await pipeline_run_setup(
storage_scope="test/scope",
+ read_scope=None,
user_id="test-user",
execution_config=execution_config,
mthds_contents=[_MINIMAL_MTHDS],
diff --git a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_storage_scope_gate.py b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_storage_scope_gate.py
index 18d28a5eb..a3d6d497e 100644
--- a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_storage_scope_gate.py
+++ b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_storage_scope_gate.py
@@ -58,6 +58,7 @@ class TestPipelineRunSetupStorageScopeGate:
with pytest.raises(ValueError, match="storage_scope"):
await pipeline_run_setup(
storage_scope="../evil",
+ read_scope=None,
user_id="test-user",
execution_config=execution_config,
mthds_contents=[_MINIMAL_MTHDS],
@@ -84,6 +85,7 @@ class TestPipelineRunSetupStorageScopeGate:
with pytest.raises(ValueError, match="storage_scope"):
await pipeline_run_setup(
storage_scope=LOCAL_STORAGE_SCOPE,
+ read_scope=None,
pipeline_run_id="../escape",
user_id="test-user",
execution_config=execution_config,
@@ -104,6 +106,7 @@ class TestPipelineRunSetupStorageScopeGate:
pipe_job, pipeline_run_id, _ = await pipeline_run_setup(
storage_scope=LOCAL_STORAGE_SCOPE,
+ read_scope=None,
user_id="test-user",
execution_config=execution_config,
mthds_contents=[_MINIMAL_MTHDS],
diff --git a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_telemetry_identity.py b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_telemetry_identity.py
index eb10b5b47..0dfb05226 100644
--- a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_telemetry_identity.py
+++ b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_telemetry_identity.py
@@ -52,6 +52,7 @@ async def _run_setup(*, mocker: MockerFixture) -> Any:
execution_config = get_config().interpreter.pipeline_execution.with_execution_overrides(generate_graph=False)
await pipeline_run_setup(
storage_scope="test/scope",
+ read_scope=None,
user_id="user-42",
execution_config=execution_config,
mthds_contents=[_MINIMAL_MTHDS],
diff --git a/tests/integration/pipelex/pipeline/test_read_scope_routes.py b/tests/integration/pipelex/pipeline/test_read_scope_routes.py
new file mode 100644
index 000000000..47094f154
--- /dev/null
+++ b/tests/integration/pipelex/pipeline/test_read_scope_routes.py
@@ -0,0 +1,196 @@
+"""The routes by which a method reads a URL it chose are refused on a scoped run, and still open on an unscoped one.
+
+A `PipeCompose` construct can build an image whose URL is assembled from plain text, with no Python
+structure and no template method call, and the next `PipeLLM` shows that image to a model. On a run
+with a read scope the LLM leaf refuses it before the dry-run branch, whether the URL is another
+organisation's storage key or a path on the worker's disk. A bare-string image input naming a local
+path is refused at the input seam, naming the input. And `pipeline_run_setup` refuses a host that
+passes a read scope its storage scope does not lie under, before anything is registered.
+"""
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.base_exceptions import PipelexError, iter_cause_chain
+from pipelex.config import get_config
+from pipelex.pipeline import pipeline_run_setup as pipeline_run_setup_module
+from pipelex.pipeline.pipeline_run_setup import pipeline_run_setup
+from pipelex.pipeline.runner import PipelexMTHDSProtocol
+from pipelex.system.pipe_run_mode import PipeRunMode
+from pipelex.system.storage_scope import LOCAL_STORAGE_SCOPE
+from pipelex.tools.uri.exceptions import UriReadRefusalReason, UriReadRefusedError
+
+READ_SCOPE = "org_abc"
+STORAGE_SCOPE = "org_abc/mt_1/run_1"
+
+_FORGED_PHOTO_MTHDS = """
+domain = "read_scope_routes"
+description = "Forge an image URL from plain text, then show the image to a model"
+main_pipe = "describe_forged_photo"
+
+[concept.Photo]
+description = "A photo with a caption"
+
+[concept.Photo.structure]
+caption = { type = "text", description = "The caption" }
+picture = { type = "concept", concept_ref = "native.Image", description = "The picture" }
+
+[pipe.describe_forged_photo]
+type = "PipeSequence"
+description = "Forge a photo from a text, then describe it"
+inputs = { where = "Text" }
+output = "Text"
+steps = [
+ { pipe = "forge_photo", result = "photo" },
+ { pipe = "describe_photo", result = "description" },
+]
+
+[pipe.forge_photo]
+type = "PipeCompose"
+description = "Build a photo whose URL is the text it was given"
+inputs = { where = "Text" }
+output = "Photo"
+
+[pipe.forge_photo.construct]
+caption = "forged"
+
+[pipe.forge_photo.construct.picture]
+url = { template = "{{ where }}" }
+
+[pipe.describe_photo]
+type = "PipeLLM"
+description = "Describe the photo"
+inputs = { "photo.picture" = "Image", photo = "Photo" }
+output = "Text"
+prompt = \"\"\"
+Describe this image.
+$photo.picture
+\"\"\"
+"""
+
+_DESCRIBE_IMAGE_MTHDS = """
+domain = "read_scope_input"
+description = "Describe an image input"
+main_pipe = "describe_image"
+
+[pipe.describe_image]
+type = "PipeLLM"
+description = "Describe an image"
+inputs = { photo = "Image" }
+output = "Text"
+prompt = \"\"\"
+Describe this image.
+$photo
+\"\"\"
+"""
+
+
+def _find_refusal(exc: BaseException) -> UriReadRefusedError | None:
+ for node in iter_cause_chain(exc):
+ if isinstance(node, UriReadRefusedError):
+ return node
+ return None
+
+
+def _protocol(*, read_scope: str | None) -> PipelexMTHDSProtocol:
+ return PipelexMTHDSProtocol(
+ pipe_run_mode=PipeRunMode.DRY,
+ user_id="test-user",
+ storage_scope=STORAGE_SCOPE if read_scope is not None else LOCAL_STORAGE_SCOPE,
+ read_scope=read_scope,
+ )
+
+
[email protected](loop_scope="class")
+class TestReadScopeRoutes:
+ @pytest.mark.parametrize(
+ ("forged_url", "reason"),
+ [
+ ("pipelex-storage://org_other/mt_y/run_9/assets/secret.png", UriReadRefusalReason.FOREIGN_STORAGE_KEY),
+ ("/etc/passwd", UriReadRefusalReason.LOCAL_PATH),
+ ("file:///etc/passwd", UriReadRefusalReason.LOCAL_PATH),
+ ],
+ )
+ async def test_a_scoped_run_refuses_the_forged_image(self, forged_url: str, reason: UriReadRefusalReason) -> None:
+ with pytest.raises(PipelexError) as exc_info:
+ await _protocol(read_scope=READ_SCOPE).execute(mthds_contents=[_FORGED_PHOTO_MTHDS], inputs={"where": forged_url})
+
+ refusal = _find_refusal(exc_info.value)
+ assert refusal is not None, f"expected a UriReadRefusedError in the cause chain, got {exc_info.value!r}"
+ assert refusal.reason == reason
+ assert "describe_photo" in refusal.message
+ assert forged_url not in refusal.message
+
+ async def test_a_scoped_run_shows_an_in_scope_image(self) -> None:
+ result = await _protocol(read_scope=READ_SCOPE).execute(
+ mthds_contents=[_FORGED_PHOTO_MTHDS], inputs={"where": "pipelex-storage://org_abc/assets/photo.png"}
+ )
+ assert result.pipe_output.main_stuff_as_str.startswith("DRY RUN:")
+
+ @pytest.mark.parametrize("forged_url", ["pipelex-storage://org_other/mt_y/run_9/assets/secret.png", "/etc/passwd"])
+ async def test_an_unscoped_run_shows_the_forged_image(self, forged_url: str) -> None:
+ result = await _protocol(read_scope=None).execute(mthds_contents=[_FORGED_PHOTO_MTHDS], inputs={"where": forged_url})
+ assert result.pipe_output.main_stuff_as_str.startswith("DRY RUN:")
+
+ async def test_a_scoped_run_refuses_a_local_path_input_naming_it(self) -> None:
+ with pytest.raises(UriReadRefusedError) as exc_info:
+ await _protocol(read_scope=READ_SCOPE).execute(mthds_contents=[_DESCRIBE_IMAGE_MTHDS], inputs={"photo": "/etc/passwd"})
+ assert exc_info.value.reason == UriReadRefusalReason.LOCAL_PATH
+ assert "input 'photo'" in exc_info.value.message
+ assert "/etc/passwd" not in exc_info.value.message
+
+ @pytest.mark.parametrize(
+ ("storage_scope", "match"),
+ [
+ (LOCAL_STORAGE_SCOPE, "must pass its own storage_scope"),
+ ("org_other/mt_1/run_1", "does not lie under its read_scope"),
+ ("org_abcdef/mt_1/run_1", "does not lie under its read_scope"),
+ ],
+ )
+ async def test_a_mismatched_host_registers_no_run(self, mocker: MockerFixture, storage_scope: str, match: str) -> None:
+ execution_config = get_config().interpreter.pipeline_execution.with_execution_overrides(generate_graph=False)
+ pipeline_manager = mocker.spy(pipeline_run_setup_module, "get_pipeline_manager")
+ acquire_library = mocker.spy(pipeline_run_setup_module, "acquire_library")
+
+ with pytest.raises(ValueError, match=match):
+ await pipeline_run_setup(
+ storage_scope=storage_scope,
+ read_scope=READ_SCOPE,
+ user_id="test-user",
+ execution_config=execution_config,
+ mthds_contents=[_DESCRIBE_IMAGE_MTHDS],
+ pipe_code="describe_image",
+ )
+
+ pipeline_manager.assert_not_called()
+ acquire_library.assert_not_called()
+
+ async def test_an_unsafe_read_scope_registers_no_run(self, mocker: MockerFixture) -> None:
+ execution_config = get_config().interpreter.pipeline_execution.with_execution_overrides(generate_graph=False)
+ pipeline_manager = mocker.spy(pipeline_run_setup_module, "get_pipeline_manager")
+
+ with pytest.raises(ValueError, match="Invalid read_scope"):
+ await pipeline_run_setup(
+ storage_scope=STORAGE_SCOPE,
+ read_scope="../org_abc",
+ user_id="test-user",
+ execution_config=execution_config,
+ mthds_contents=[_DESCRIBE_IMAGE_MTHDS],
+ pipe_code="describe_image",
+ )
+
+ pipeline_manager.assert_not_called()
+
+ async def test_the_read_scope_rides_the_run_metadata(self) -> None:
+ execution_config = get_config().interpreter.pipeline_execution.with_execution_overrides(generate_graph=False)
+
+ pipe_job, _, _ = await pipeline_run_setup(
+ storage_scope=STORAGE_SCOPE,
+ read_scope=READ_SCOPE,
+ user_id="test-user",
+ execution_config=execution_config,
+ mthds_contents=[_DESCRIBE_IMAGE_MTHDS],
+ pipe_code="describe_image",
+ )
+
+ assert pipe_job.job_metadata.run_metadata.read_scope == READ_SCOPE
diff --git a/tests/integration/pipelex/pipeline/test_run_path_bundle_verdict.py b/tests/integration/pipelex/pipeline/test_run_path_bundle_verdict.py
index 9b48fdd05..b9f82fc24 100644
--- a/tests/integration/pipelex/pipeline/test_run_path_bundle_verdict.py
+++ b/tests/integration/pipelex/pipeline/test_run_path_bundle_verdict.py
@@ -146,6 +146,7 @@ async def _validate_verdict(*, bundle: str) -> ValidateBundleError:
async def _run_setup(**kwargs: Any) -> None:
await pipeline_run_setup(
storage_scope="test/scope",
+ read_scope=None,
user_id="test-user",
execution_config=_execution_config(),
pipe_run_mode=PipeRunMode.DRY,
diff --git a/tests/integration/pipelex/pipeline/test_validate_bundle_load_refusals.py b/tests/integration/pipelex/pipeline/test_validate_bundle_load_refusals.py
index 75a1e8e49..841d90576 100644
--- a/tests/integration/pipelex/pipeline/test_validate_bundle_load_refusals.py
+++ b/tests/integration/pipelex/pipeline/test_validate_bundle_load_refusals.py
@@ -349,6 +349,7 @@ class TestValidateBundleLoadRefusals:
with pytest.raises(ValidateBundleError) as raised:
await pipeline_run_setup(
storage_scope="test/scope",
+ read_scope=None,
user_id="test-user",
execution_config=execution_config,
mthds_contents=[_llm_bundle(model_line='model = "gpt-5.1"')],
diff --git a/tests/integration/pipelex/pipeline/test_verdict_names_no_host_path.py b/tests/integration/pipelex/pipeline/test_verdict_names_no_host_path.py
index 976a8d5c9..1d06bc5b3 100644
--- a/tests/integration/pipelex/pipeline/test_verdict_names_no_host_path.py
+++ b/tests/integration/pipelex/pipeline/test_verdict_names_no_host_path.py
@@ -170,6 +170,7 @@ def _execution_config() -> Any:
async def _run_setup(**kwargs: Any) -> None:
await pipeline_run_setup(
storage_scope="test/scope",
+ read_scope=None,
user_id="test-user",
execution_config=_execution_config(),
pipe_run_mode=PipeRunMode.DRY,
diff --git a/tests/integration/pipelex/pipes/optionals/test_lift_sequence.py b/tests/integration/pipelex/pipes/optionals/test_lift_sequence.py
index 434f0b8af..d7bb44e5c 100644
--- a/tests/integration/pipelex/pipes/optionals/test_lift_sequence.py
+++ b/tests/integration/pipelex/pipes/optionals/test_lift_sequence.py
@@ -250,6 +250,7 @@ class TestLiftSequence:
pipe_job = await prepare_pipe_job(
storage_scope="test/scope",
+ read_scope=None,
user_id="test-user",
pipe=sequence,
library_id=library_id,
diff --git a/tests/integration/pipelex/pipes/optionals/test_lift_skip_trichotomy.py b/tests/integration/pipelex/pipes/optionals/test_lift_skip_trichotomy.py
index f5c8ce886..825151b35 100644
--- a/tests/integration/pipelex/pipes/optionals/test_lift_skip_trichotomy.py
+++ b/tests/integration/pipelex/pipes/optionals/test_lift_skip_trichotomy.py
@@ -295,7 +295,9 @@ class TestLiftSkipTrichotomy:
return_value=mock_manager,
)
traced_metadata = JobMetadata(
- run_metadata=RunMetadata(storage_scope="test/scope", user_id="pytest", pipeline_run_id=job_metadata.run_metadata.pipeline_run_id),
+ run_metadata=RunMetadata(
+ storage_scope="test/scope", read_scope=None, user_id="pytest", pipeline_run_id=job_metadata.run_metadata.pipeline_run_id
+ ),
trace_context=make_trace_context(graph_id=job_metadata.run_metadata.pipeline_run_id),
)
diff --git a/tests/integration/pipelex/system/telemetry/test_pipe_span_carries_run_identity.py b/tests/integration/pipelex/system/telemetry/test_pipe_span_carries_run_identity.py
index 0d280c14a..0847a676f 100644
--- a/tests/integration/pipelex/system/telemetry/test_pipe_span_carries_run_identity.py
+++ b/tests/integration/pipelex/system/telemetry/test_pipe_span_carries_run_identity.py
@@ -50,6 +50,7 @@ async def _make_pipe_job(*, extras: dict[str, str] | None = None) -> PipeJob:
execution_config = get_config().interpreter.pipeline_execution.with_execution_overrides(generate_graph=False)
pipe_job, _, _ = await pipeline_run_setup(
storage_scope="test/scope",
+ read_scope=None,
user_id="user-42",
execution_config=execution_config,
mthds_contents=[_MINIMAL_MTHDS],
diff --git a/tests/integration/pipelex/system/test_keyless_boot_forced_dry.py b/tests/integration/pipelex/system/test_keyless_boot_forced_dry.py
index bfb9a3546..4fae43514 100644
--- a/tests/integration/pipelex/system/test_keyless_boot_forced_dry.py
+++ b/tests/integration/pipelex/system/test_keyless_boot_forced_dry.py
@@ -74,6 +74,7 @@ class TestKeylessBootForcedDry:
)
pipe_job = await prepare_pipe_job(
storage_scope="test/scope",
+ read_scope=None,
user_id="test-user",
pipe=pipe,
library_id=library_id,
@@ -97,7 +98,7 @@ class TestKeylessBootForcedDry:
self._boot_keyless()
assert is_dry_run_forced()
- kernel = PipelexKernel.make(storage_scope="test/scope", user_id="test-user")
+ kernel = PipelexKernel.make(storage_scope="test/scope", read_scope=None, user_id="test-user")
assert kernel.cogt_run_params.run_mode.is_dry
finally:
@@ -115,7 +116,7 @@ class TestKeylessBootForcedDry:
assert is_dry_run_forced()
with pytest.raises(ValueError, match="is_mock_usage"):
- PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.LIVE, user_id="test-user", is_mock_usage=True)
+ PipelexKernel.make(storage_scope="test/scope", read_scope=None, run_mode=PipeRunMode.LIVE, user_id="test-user", is_mock_usage=True)
finally:
Pipelex.teardown_if_needed()
diff --git a/tests/integration/pipelex/tools/storage/test_user_provided_image_storage.py b/tests/integration/pipelex/tools/storage/test_user_provided_image_storage.py
index dd5d81bc7..cc54a1bd7 100644
--- a/tests/integration/pipelex/tools/storage/test_user_provided_image_storage.py
+++ b/tests/integration/pipelex/tools/storage/test_user_provided_image_storage.py
@@ -54,7 +54,7 @@ class TestUserProvidedImageStorage:
working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)
# Normalize to storage
- normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope")
+ normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope", read_scope=None)
# Verify the URL was converted to pipelex-storage://
normalized_stuff = normalized_memory.get_stuff("user_image")
@@ -116,7 +116,7 @@ class TestUserProvidedImageStorage:
working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)
# Normalize (should not change HTTP URLs when fetch is disabled)
- normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope")
+ normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope", read_scope=None)
# Verify URL was NOT changed
normalized_stuff = normalized_memory.get_stuff("remote_image")
@@ -141,7 +141,7 @@ class TestUserProvidedImageStorage:
working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)
# Normalize - should upload the local file to storage
- normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope")
+ normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope", read_scope=None)
# Verify the URL was converted to pipelex-storage://
normalized_stuff = normalized_memory.get_stuff("local_image")
@@ -180,7 +180,7 @@ class TestUserProvidedImageStorage:
working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)
# Normalize (should not change local paths when upload is disabled)
- normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope")
+ normalized_memory = await normalize_data_urls_to_storage(working_memory, storage_scope="test/scope", read_scope=None)
# Verify URL was NOT changed
normalized_stuff = normalized_memory.get_stuff("local_image")
diff --git a/tests/unit/pipelex/cogt/content_generation/test_assignment_models_schema.py b/tests/unit/pipelex/cogt/content_generation/test_assignment_models_schema.py
index 7bfae4da3..c1ffc6570 100644
--- a/tests/unit/pipelex/cogt/content_generation/test_assignment_models_schema.py
+++ b/tests/unit/pipelex/cogt/content_generation/test_assignment_models_schema.py
@@ -16,7 +16,7 @@ class SampleOutputModel(BaseModel):
def _make_stub_job_metadata() -> JobMetadata:
- return JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"))
+ return JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="test-user", pipeline_run_id="test-run"))
def _make_stub_llm_assignment() -> LLMAssignment:
diff --git a/tests/unit/pipelex/cogt/content_generation/test_assignment_models_security.py b/tests/unit/pipelex/cogt/content_generation/test_assignment_models_security.py
index 5559d8721..43df41ff1 100644
--- a/tests/unit/pipelex/cogt/content_generation/test_assignment_models_security.py
+++ b/tests/unit/pipelex/cogt/content_generation/test_assignment_models_security.py
@@ -18,7 +18,7 @@ from pipelex.system.pipe_run_mode import PipeRunMode
def _make_stub_job_metadata() -> JobMetadata:
- return JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"))
+ return JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="test-user", pipeline_run_id="test-run"))
def _make_stub_llm_assignment() -> LLMAssignment:
diff --git a/tests/unit/pipelex/cogt/content_generation/test_assignment_read_coverage.py b/tests/unit/pipelex/cogt/content_generation/test_assignment_read_coverage.py
new file mode 100644
index 000000000..071b7f199
--- /dev/null
+++ b/tests/unit/pipelex/cogt/content_generation/test_assignment_read_coverage.py
@@ -0,0 +1,193 @@
+"""Every URL an assignment carries is declared by its `referenced_uris()`, which its leaf authorizes.
+
+The leaves authorize what `referenced_uris()` declares, so a URL an assignment carries and does not
+declare would be read unchecked. This test is the tripwire: it walks every assignment model's fields
+for URL-shaped ones (a string field named `uri`, `url` or ending in `_uri` / `_url`), compares them
+with the inventory below, and checks that `referenced_uris()` returns a URL placed in each of them. A
+new assignment, or a new URL field on a payload, fails here until its `referenced_uris()` declares it
+and the inventory names it.
+"""
+
+import inspect
+import re
+import types
+import typing
+from typing import Annotated, Any, Protocol, Union, get_args, get_origin
+
+from pydantic import BaseModel
+
+from pipelex.cogt.content_generation import assignment_models
+from pipelex.cogt.content_generation.assignment_models import (
+ ExtractAssignment,
+ ImgGenAssignment,
+ LLMAssignment,
+ ObjectAssignment,
+ RenderPageViewsAssignment,
+ SearchAssignment,
+ SearchObjectAssignment,
+ TemplatingAssignment,
+)
+from pipelex.cogt.content_generation.cogt_run_params import CogtRunParams
+from pipelex.cogt.document.prompt_document import PromptDocumentUri
+from pipelex.cogt.extract.extract_input import ExtractInput
+from pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams
+from pipelex.cogt.image.prompt_image import PromptImageUri
+from pipelex.cogt.img_gen.img_gen_job_components import AspectRatio, Background, ImgGenJobConfig, ImgGenJobParams
+from pipelex.cogt.img_gen.img_gen_prompt import ImgGenPrompt
+from pipelex.cogt.llm.llm_prompt import LLMPrompt
+from pipelex.cogt.llm.llm_setting import LLMSetting
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
+from pipelex.system.pipe_run_mode import PipeRunMode
+from pipelex.tools.uri.uri_read_scope import UriReference
+
+URL_FIELD_NAME = re.compile(r"(^|_)(uri|url)$")
+
+# Run metadata and run parameters describe the run, not what it reads.
+NON_PAYLOAD_FIELDS = frozenset({"job_metadata", "cogt_run_params"})
+
+# Every URL-shaped field of every assignment, as a dotted path; a list's items are named by the list's field.
+URL_FIELD_INVENTORY: dict[type[BaseModel], set[str]] = {
+ LLMAssignment: {"llm_prompt.user_images.uri", "llm_prompt.user_documents.uri"},
+ ObjectAssignment: {"llm_assignment_for_object.llm_prompt.user_images.uri", "llm_assignment_for_object.llm_prompt.user_documents.uri"},
+ ImgGenAssignment: {"img_gen_prompt.input_images.uri"},
+ TemplatingAssignment: set(),
+ ExtractAssignment: {"extract_input.image_uri", "extract_input.document_uri"},
+ RenderPageViewsAssignment: {"document_uri"},
+ SearchAssignment: set(),
+ SearchObjectAssignment: set(),
+}
+
+
+def _concrete_types(annotation: Any) -> list[Any]:
+ """The concrete types an annotation admits, through Annotated, unions and generic containers."""
+ origin = get_origin(annotation)
+ if origin is Annotated:
+ return _concrete_types(get_args(annotation)[0])
+ if origin is None:
+ return [annotation]
+ if origin in {Union, types.UnionType, typing.Literal}:
+ found: list[Any] = []
+ for arg in get_args(annotation):
+ found.extend(_concrete_types(arg))
+ return found
+ found = []
+ for arg in get_args(annotation):
+ found.extend(_concrete_types(arg))
+ return found
+
+
+def _url_field_paths(model_class: type[BaseModel], *, prefix: str = "") -> set[str]:
+ paths: set[str] = set()
+ for field_name, field_info in model_class.model_fields.items():
+ if field_name in NON_PAYLOAD_FIELDS:
+ continue
+ for concrete_type in _concrete_types(field_info.annotation):
+ if concrete_type is str and URL_FIELD_NAME.search(field_name):
+ paths.add(f"{prefix}{field_name}")
+ elif inspect.isclass(concrete_type) and issubclass(concrete_type, BaseModel):
+ paths |= _url_field_paths(concrete_type, prefix=f"{prefix}{field_name}.")
+ return paths
+
+
+def _assignment_classes() -> list[type[BaseModel]]:
+ return [
+ member
+ for _, member in inspect.getmembers(assignment_models, inspect.isclass)
+ if issubclass(member, BaseModel) and member.__module__ == assignment_models.__name__
+ ]
+
+
+def _job_metadata() -> JobMetadata:
+ return JobMetadata(run_metadata=RunMetadata(user_id="u", pipeline_run_id="run_1", storage_scope="run_1", read_scope=None))
+
+
+def _llm_assignment() -> LLMAssignment:
+ return LLMAssignment(
+ job_metadata=_job_metadata(),
+ cogt_run_params=CogtRunParams(run_mode=PipeRunMode.DRY),
+ llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),
+ llm_prompt=LLMPrompt(
+ user_text="t",
+ user_images=[PromptImageUri(uri="pipelex-storage://s/image_a.png"), PromptImageUri(uri="pipelex-storage://s/image_b.png")],
+ user_documents=[PromptDocumentUri(uri="pipelex-storage://s/document_a.pdf")],
+ ),
+ )
+
+
+class _Out(BaseModel):
+ text: str
+
+
+class _DeclaresReads(Protocol):
+ def referenced_uris(self) -> list[UriReference]: ...
+
+
+# One instance per URL-bearing assignment, with a distinct URL in every URL-shaped field.
+URL_BEARING_SAMPLES: dict[type[BaseModel], tuple[_DeclaresReads, set[str]]] = {
+ LLMAssignment: (
+ _llm_assignment(),
+ {"pipelex-storage://s/image_a.png", "pipelex-storage://s/image_b.png", "pipelex-storage://s/document_a.pdf"},
+ ),
+ ObjectAssignment: (
+ ObjectAssignment.make_for_class(_Out, llm_assignment=_llm_assignment()),
+ {"pipelex-storage://s/image_a.png", "pipelex-storage://s/image_b.png", "pipelex-storage://s/document_a.pdf"},
+ ),
+ ImgGenAssignment: (
+ ImgGenAssignment(
+ job_metadata=_job_metadata(),
+ cogt_run_params=CogtRunParams(run_mode=PipeRunMode.DRY),
+ img_gen_handle="h",
+ img_gen_prompt=ImgGenPrompt(positive_text="p", input_images=[PromptImageUri(uri="pipelex-storage://s/input.png")]),
+ img_gen_job_params=ImgGenJobParams(aspect_ratio=AspectRatio.SQUARE, background=Background.AUTO),
+ img_gen_job_config=ImgGenJobConfig(is_sync_mode=True),
+ nb_images=1,
+ ),
+ {"pipelex-storage://s/input.png"},
+ ),
+ ExtractAssignment: (
+ ExtractAssignment(
+ job_metadata=_job_metadata(),
+ cogt_run_params=CogtRunParams(run_mode=PipeRunMode.DRY),
+ extract_handle="h",
+ extract_input=ExtractInput(document_uri="pipelex-storage://s/doc.pdf"),
+ extract_job_params=ExtractJobParams.make_default_extract_job_params(),
+ extract_job_config=ExtractJobConfig(),
+ ),
+ {"pipelex-storage://s/doc.pdf"},
+ ),
+ RenderPageViewsAssignment: (
+ RenderPageViewsAssignment(
+ job_metadata=_job_metadata(),
+ cogt_run_params=CogtRunParams(run_mode=PipeRunMode.DRY),
+ document_uri="pipelex-storage://s/doc.pdf",
+ page_views_dpi=72,
+ ),
+ {"pipelex-storage://s/doc.pdf"},
+ ),
+}
+
+
+class TestAssignmentReadCoverage:
+ def test_the_inventory_names_every_assignment(self) -> None:
+ assert set(_assignment_classes()) == set(URL_FIELD_INVENTORY)
+
+ def test_every_assignment_declares_what_it_reads(self) -> None:
+ for assignment_class in _assignment_classes():
+ assert "referenced_uris" in assignment_class.__dict__, f"{assignment_class.__name__} does not declare referenced_uris()"
+
+ def test_the_url_fields_of_every_assignment_are_the_inventoried_ones(self) -> None:
+ for assignment_class, inventoried_paths in URL_FIELD_INVENTORY.items():
+ assert _url_field_paths(assignment_class) == inventoried_paths, assignment_class.__name__
+
+ def test_every_url_bearing_assignment_has_a_sample(self) -> None:
+ url_bearing = {assignment_class for assignment_class, paths in URL_FIELD_INVENTORY.items() if paths}
+ assert url_bearing == set(URL_BEARING_SAMPLES)
+
+ def test_referenced_uris_returns_every_url_the_sample_carries(self) -> None:
+ for assignment_class, (sample, expected_uris) in URL_BEARING_SAMPLES.items():
+ referenced_uris = sample.referenced_uris()
+ assert {uri_reference.uri for uri_reference in referenced_uris} == expected_uris, assignment_class.__name__
+
+ def test_the_extract_input_declares_an_image_too(self) -> None:
+ referenced_uris = ExtractInput(image_uri="pipelex-storage://s/scan.png").referenced_uris()
+ assert [uri_reference.uri for uri_reference in referenced_uris] == ["pipelex-storage://s/scan.png"]
diff --git a/tests/unit/pipelex/cogt/content_generation/test_dry_mock.py b/tests/unit/pipelex/cogt/content_generation/test_dry_mock.py
index 9f4c4a8fd..b5ca12bd6 100644
--- a/tests/unit/pipelex/cogt/content_generation/test_dry_mock.py
+++ b/tests/unit/pipelex/cogt/content_generation/test_dry_mock.py
@@ -31,7 +31,7 @@ from pipelex.system.pipe_run_mode import PipeRunMode
def _dry_object_assignment() -> ObjectAssignment:
llm_assignment = LLMAssignment(
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_dry_mock")),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="run_dry_mock")),
cogt_run_params=CogtRunParams(run_mode=PipeRunMode.DRY),
llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),
llm_prompt=LLMPrompt(),
@@ -56,7 +56,7 @@ class TestDryMock:
"""report_mock_usage_llm_job emits non-zero usage under the mock_usage sentinel -> reportable."""
captured = self._capture_reported_job(mocker)
report_mock_usage_llm_job(
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_mock")),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="run_mock")),
llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),
llm_prompt=LLMPrompt(),
)
@@ -73,7 +73,7 @@ class TestDryMock:
"""report_dry_llm_job stays zero-token under the dry_run sentinel -> NOT reportable (suppressed)."""
captured = self._capture_reported_job(mocker)
report_dry_llm_job(
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_dry")),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="run_dry")),
llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),
llm_prompt=LLMPrompt(),
)
diff --git a/tests/unit/pipelex/cogt/content_generation/test_dry_report_tz.py b/tests/unit/pipelex/cogt/content_generation/test_dry_report_tz.py
index a7f0262a9..a064180d0 100644
--- a/tests/unit/pipelex/cogt/content_generation/test_dry_report_tz.py
+++ b/tests/unit/pipelex/cogt/content_generation/test_dry_report_tz.py
@@ -47,7 +47,8 @@ class TestDryReportTimezone:
)
job_metadata = JobMetadata(
- run_metadata=RunMetadata(storage_scope="test/scope", user_id="test_user", pipeline_run_id="run_dry_tz"), started_at=started_at
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="test_user", pipeline_run_id="run_dry_tz"),
+ started_at=started_at,
)
report_dry_llm_job(
diff --git a/tests/unit/pipelex/cogt/content_generation/test_dry_run_factory_file_urls.py b/tests/unit/pipelex/cogt/content_generation/test_dry_run_factory_file_urls.py
new file mode 100644
index 000000000..f74bb5b0c
--- /dev/null
+++ b/tests/unit/pipelex/cogt/content_generation/test_dry_run_factory_file_urls.py
@@ -0,0 +1,75 @@
+"""A mocked image or document carries a URL that a scoped run may read: an https URL or a data URL.
+
+A run with a read scope authorizes URLs before the dry-run branch, so a mock whose URL is a random
+string, which `resolve_uri` takes for a local path, would be refused by the very dry run that made it.
+"""
+
+import base64
+
+import pypdfium2
+from pydantic import BaseModel
+
+from pipelex.cogt.content_generation.dry_mock import build_mock_object
+from pipelex.cogt.content_generation.dry_run_factory import MOCK_DOCUMENT_DATA_URL, DryRunFactory
+from pipelex.cogt.content_generation.schema_to_model_factory import SchemaToModelFactory
+from pipelex.config import get_config
+from pipelex.core.stuffs.document_content import DocumentContent
+from pipelex.core.stuffs.image_content import ImageContent
+from pipelex.core.stuffs.structured_content import StructuredContent
+from pipelex.core.stuffs.text_and_images_content import TextAndImagesContent
+from pipelex.tools.uri.uri_read_scope import authorize_uri_read
+
+
+class Dossier(StructuredContent):
+ cover: ImageContent
+ attachments: list[DocumentContent]
+ page: TextAndImagesContent
+
+
+class _Wrapper(BaseModel):
+ dossier: Dossier
+
+
+class TestDryRunFactoryFileUrls:
+ def test_a_mocked_image_carries_a_configured_dry_run_image_url(self) -> None:
+ image_content = DryRunFactory.make_dry_run_factory(ImageContent).build(factory_use_construct=True)
+ assert image_content.url in get_config().inference.dry_run.image_urls
+
+ def test_a_mocked_document_carries_the_blank_pdf_data_url(self) -> None:
+ document_content = DryRunFactory.make_dry_run_factory(DocumentContent).build(factory_use_construct=True)
+ assert document_content.url == MOCK_DOCUMENT_DATA_URL
+
+ def test_the_blank_pdf_is_a_pdf_a_reader_opens(self) -> None:
+ pdf_bytes = base64.b64decode(MOCK_DOCUMENT_DATA_URL.split(",", 1)[1])
+ pdf_document = pypdfium2.PdfDocument(pdf_bytes)
+ try:
+ assert len(pdf_document) == 1
+ finally:
+ pdf_document.close()
+
+ def test_nested_images_and_documents_are_mocked_the_same_way(self) -> None:
+ wrapper = DryRunFactory.make_dry_run_factory(_Wrapper).build()
+ dossier = wrapper.dossier
+ assert dossier.cover.url in get_config().inference.dry_run.image_urls
+ assert all(attachment.url == MOCK_DOCUMENT_DATA_URL for attachment in dossier.attachments)
+ for image in dossier.page.images or []:
+ assert image.url in get_config().inference.dry_run.image_urls
+
+ def test_classes_rebuilt_from_a_json_schema_are_mocked_the_same_way(self) -> None:
+ # A worker running a dry model call out of process rebuilds the output class from its schema,
+ # and the nested file classes it builds are no subclasses of the real ones.
+ rebuilt_class = SchemaToModelFactory.make_from_json_schema(schema=Dossier.model_json_schema(), class_name="Dossier")
+ rebuilt_image_class = rebuilt_class.model_fields["cover"].annotation
+ assert isinstance(rebuilt_image_class, type)
+ assert not issubclass(rebuilt_image_class, ImageContent)
+
+ dossier = Dossier.model_validate(build_mock_object(rebuilt_class).model_dump(mode="json"))
+ assert dossier.cover.url in get_config().inference.dry_run.image_urls
+ assert all(attachment.url == MOCK_DOCUMENT_DATA_URL for attachment in dossier.attachments)
+ authorize_uri_read(uri=dossier.cover.url, read_scope="org_abc", position="a mocked image")
+
+ def test_a_scoped_run_reads_every_mocked_file_url(self) -> None:
+ image_content = DryRunFactory.make_dry_run_factory(ImageContent).build(factory_use_construct=True)
+ document_content = DryRunFactory.make_dry_run_factory(DocumentContent).build(factory_use_construct=True)
+ authorize_uri_read(uri=image_content.url, read_scope="org_abc", position="a mocked image")
+ authorize_uri_read(uri=document_content.url, read_scope="org_abc", position="a mocked document")
diff --git a/tests/unit/pipelex/cogt/content_generation/test_extract_generate_dry_branch.py b/tests/unit/pipelex/cogt/content_generation/test_extract_generate_dry_branch.py
index 188e8e3b4..8d286c595 100644
--- a/tests/unit/pipelex/cogt/content_generation/test_extract_generate_dry_branch.py
+++ b/tests/unit/pipelex/cogt/content_generation/test_extract_generate_dry_branch.py
@@ -22,7 +22,9 @@ from pipelex.system.pipe_run_mode import PipeRunMode
class TestExtractGenerateDryBranch:
def _assignment(self, *, run_mode: PipeRunMode, extract_input: ExtractInput) -> ExtractAssignment:
return ExtractAssignment(
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_extract_dry")),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="run_extract_dry")
+ ),
cogt_run_params=CogtRunParams(run_mode=run_mode),
extract_handle="mock-extract-handle",
extract_input=extract_input,
diff --git a/tests/unit/pipelex/cogt/content_generation/test_img_gen_generate_dry_branch.py b/tests/unit/pipelex/cogt/content_generation/test_img_gen_generate_dry_branch.py
index fd6b9503c..ff7d21e2a 100644
--- a/tests/unit/pipelex/cogt/content_generation/test_img_gen_generate_dry_branch.py
+++ b/tests/unit/pipelex/cogt/content_generation/test_img_gen_generate_dry_branch.py
@@ -21,7 +21,9 @@ from pipelex.system.pipe_run_mode import PipeRunMode
class TestImgGenGenerateDryBranch:
def _assignment(self, *, run_mode: PipeRunMode, nb_images: int = 1) -> ImgGenAssignment:
return ImgGenAssignment(
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_img_dry")),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="run_img_dry")
+ ),
cogt_run_params=CogtRunParams(run_mode=run_mode),
img_gen_handle="mock-img-handle",
img_gen_prompt=ImgGenPrompt(positive_text="a red apple", negative_text="no worms"),
diff --git a/tests/unit/pipelex/cogt/content_generation/test_leaf_read_scope.py b/tests/unit/pipelex/cogt/content_generation/test_leaf_read_scope.py
new file mode 100644
index 000000000..34bd0a0ca
--- /dev/null
+++ b/tests/unit/pipelex/cogt/content_generation/test_leaf_read_scope.py
@@ -0,0 +1,289 @@
+"""Every content-generation leaf authorizes the URLs its assignment reads, before the dry-run branch.
+
+On a scoped run a leaf refuses a storage key outside the read scope and a local path, and reads an
+in-scope key; it does so in dry mode as well, so a dry run refuses the method a live run would and
+these tests need no inference. The raw provider leaves authorize too, for a caller that reaches them
+directly, and every refusal happens before a worker is built.
+"""
+
+from collections.abc import Awaitable, Callable
+from typing import Any
+
+import pytest
+from pydantic import BaseModel
+from pytest_mock import MockerFixture
+
+from pipelex.cogt.content_generation.assignment_models import (
+ ExtractAssignment,
+ ImgGenAssignment,
+ LLMAssignment,
+ ObjectAssignment,
+ RenderPageViewsAssignment,
+)
+from pipelex.cogt.content_generation.cogt_run_params import CogtRunParams
+from pipelex.cogt.content_generation.extract_generate import extract_gen_pages, extract_gen_pages_and_store
+from pipelex.cogt.content_generation.img_gen_generate import (
+ img_gen_image_list,
+ img_gen_image_list_and_store,
+ img_gen_single_image,
+ img_gen_single_image_and_store,
+)
+from pipelex.cogt.content_generation.llm_generate import llm_gen_object, llm_gen_object_list, llm_gen_text
+from pipelex.cogt.content_generation.render_generate import render_page_views_and_store
+from pipelex.cogt.content_generation.search_generate import search_gen_sourced_answer, search_gen_structured, search_gen_structured_object
+from pipelex.cogt.content_generation.templating_generate import templating_gen_text
+from pipelex.cogt.document.prompt_document import PromptDocumentUri
+from pipelex.cogt.extract.extract_input import ExtractInput
+from pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams
+from pipelex.cogt.image.prompt_image import PromptImageUri
+from pipelex.cogt.img_gen.img_gen_job_components import AspectRatio, Background, ImgGenJobConfig, ImgGenJobParams
+from pipelex.cogt.img_gen.img_gen_prompt import ImgGenPrompt
+from pipelex.cogt.llm.llm_prompt import LLMPrompt
+from pipelex.cogt.llm.llm_setting import LLMSetting
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
+from pipelex.system.pipe_run_mode import PipeRunMode
+from pipelex.tools.uri.exceptions import UriReadRefusalReason, UriReadRefusedError
+
+READ_SCOPE = "org_abc"
+IN_SCOPE_IMAGE = "pipelex-storage://org_abc/assets/photo.png"
+IN_SCOPE_DOCUMENT = "pipelex-storage://org_abc/assets/report.pdf"
+FOREIGN_KEY = "pipelex-storage://org_other/assets/secret.pdf"
+LOCAL_PATH = "/etc/passwd"
+PIPE_CODE = "describe_photo"
+
+
+class Summary(BaseModel):
+ title: str
+
+
+def _job_metadata() -> JobMetadata:
+ return JobMetadata(
+ run_metadata=RunMetadata(user_id="u", pipeline_run_id="run_1", storage_scope="org_abc/mt_1/run_1", read_scope=READ_SCOPE),
+ pipe_code=PIPE_CODE,
+ )
+
+
+def _llm_assignment(*, run_mode: PipeRunMode, image_uri: str | None = None, document_uri: str | None = None) -> LLMAssignment:
+ return LLMAssignment(
+ job_metadata=_job_metadata(),
+ cogt_run_params=CogtRunParams(run_mode=run_mode),
+ llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),
+ llm_prompt=LLMPrompt(
+ user_text="Describe it.",
+ user_images=[PromptImageUri(uri=image_uri)] if image_uri is not None else [],
+ user_documents=[PromptDocumentUri(uri=document_uri)] if document_uri is not None else [],
+ ),
+ )
+
+
+def _img_gen_assignment(*, run_mode: PipeRunMode, image_uri: str) -> ImgGenAssignment:
+ return ImgGenAssignment(
+ job_metadata=_job_metadata(),
+ cogt_run_params=CogtRunParams(run_mode=run_mode),
+ img_gen_handle="mock-img-gen-handle",
+ img_gen_prompt=ImgGenPrompt(positive_text="a red apple", input_images=[PromptImageUri(uri=image_uri)]),
+ img_gen_job_params=ImgGenJobParams(aspect_ratio=AspectRatio.SQUARE, background=Background.AUTO),
+ img_gen_job_config=ImgGenJobConfig(is_sync_mode=True),
+ nb_images=2,
+ )
+
+
+def _extract_assignment(*, run_mode: PipeRunMode, document_uri: str) -> ExtractAssignment:
+ return ExtractAssignment(
+ job_metadata=_job_metadata(),
+ cogt_run_params=CogtRunParams(run_mode=run_mode),
+ extract_handle="mock-extract-handle",
+ extract_input=ExtractInput(document_uri=document_uri),
+ extract_job_params=ExtractJobParams.make_default_extract_job_params(),
+ extract_job_config=ExtractJobConfig(),
+ )
+
+
+def _render_assignment(*, run_mode: PipeRunMode, document_uri: str) -> RenderPageViewsAssignment:
+ return RenderPageViewsAssignment(
+ job_metadata=_job_metadata(),
+ cogt_run_params=CogtRunParams(run_mode=run_mode),
+ document_uri=document_uri,
+ page_views_dpi=72,
+ )
+
+
+# Each case: a leaf called with an assignment reading the given URI in the given mode.
+LeafCall = Callable[[str, PipeRunMode, Any], Awaitable[Any]]
+
+
+async def _call_llm_text_with_image(uri: str, run_mode: PipeRunMode, _factory: Any) -> Any:
+ return await llm_gen_text(llm_assignment=_llm_assignment(run_mode=run_mode, image_uri=uri))
+
+
+async def _call_llm_text_with_document(uri: str, run_mode: PipeRunMode, _factory: Any) -> Any:
+ return await llm_gen_text(llm_assignment=_llm_assignment(run_mode=run_mode, document_uri=uri))
+
+
+async def _call_llm_object(uri: str, run_mode: PipeRunMode, _factory: Any) -> Any:
+ object_assignment = ObjectAssignment.make_for_class(Summary, llm_assignment=_llm_assignment(run_mode=run_mode, document_uri=uri))
+ return await llm_gen_object(object_assignment=object_assignment, object_class=Summary)
+
+
+async def _call_llm_object_list(uri: str, run_mode: PipeRunMode, _factory: Any) -> Any:
+ object_assignment = ObjectAssignment.make_for_class(Summary, llm_assignment=_llm_assignment(run_mode=run_mode, image_uri=uri), nb_items=2)
+ return await llm_gen_object_list(object_assignment=object_assignment, object_class=Summary)
+
+
+async def _call_img_gen_single_and_store(uri: str, run_mode: PipeRunMode, factory: Any) -> Any:
+ return await img_gen_single_image_and_store(
+ img_gen_assignment=_img_gen_assignment(run_mode=run_mode, image_uri=uri), generated_content_factory=factory
+ )
+
+
+async def _call_img_gen_list_and_store(uri: str, run_mode: PipeRunMode, factory: Any) -> Any:
+ return await img_gen_image_list_and_store(
+ img_gen_assignment=_img_gen_assignment(run_mode=run_mode, image_uri=uri), generated_content_factory=factory
+ )
+
+
+async def _call_extract_and_store(uri: str, run_mode: PipeRunMode, factory: Any) -> Any:
+ return await extract_gen_pages_and_store(
+ extract_assignment=_extract_assignment(run_mode=run_mode, document_uri=uri), generated_content_factory=factory
+ )
+
+
+async def _call_render_and_store(uri: str, run_mode: PipeRunMode, factory: Any) -> Any:
+ return await render_page_views_and_store(
+ render_assignment=_render_assignment(run_mode=run_mode, document_uri=uri), generated_content_factory=factory
+ )
+
+
+# The in-scope URI each leaf is given when it should read: an image where the payload is an image.
+DRY_LEAVES: list[tuple[str, LeafCall, str]] = [
+ ("llm_gen_text with an image", _call_llm_text_with_image, IN_SCOPE_IMAGE),
+ ("llm_gen_text with a document", _call_llm_text_with_document, IN_SCOPE_DOCUMENT),
+ ("llm_gen_object", _call_llm_object, IN_SCOPE_DOCUMENT),
+ ("llm_gen_object_list", _call_llm_object_list, IN_SCOPE_IMAGE),
+ ("img_gen_single_image_and_store", _call_img_gen_single_and_store, IN_SCOPE_IMAGE),
+ ("img_gen_image_list_and_store", _call_img_gen_list_and_store, IN_SCOPE_IMAGE),
+ ("extract_gen_pages_and_store", _call_extract_and_store, IN_SCOPE_DOCUMENT),
+ ("render_page_views_and_store", _call_render_and_store, IN_SCOPE_DOCUMENT),
+]
+
+# The leaves whose assignments read no URL today. They call the gate all the same, so a URL field
+# added to their assignment, and declared as the coverage test requires, is authorized too.
+READ_FREE_LEAVES: list[tuple[str, str, Callable[[Any], Awaitable[Any]]]] = [
+ ("templating_gen_text", "pipelex.cogt.content_generation.templating_generate", templating_gen_text),
+ ("search_gen_sourced_answer", "pipelex.cogt.content_generation.search_generate", search_gen_sourced_answer),
+ ("search_gen_structured", "pipelex.cogt.content_generation.search_generate", search_gen_structured),
+ (
+ "search_gen_structured_object",
+ "pipelex.cogt.content_generation.search_generate",
+ lambda assignment: search_gen_structured_object(assignment, output_class=Summary),
+ ),
+]
+
+
+class _GateCalledError(Exception):
+ """Raised by the patched gate, to prove it ran before anything else in the leaf."""
+
+
+def no_workers(mocker: MockerFixture) -> dict[str, Any]:
+ """Spy on every worker lookup and silence the dry leaves' usage reporting."""
+ mocker.patch("pipelex.cogt.content_generation.dry_mock.get_report_delegate", return_value=mocker.MagicMock())
+ return {
+ "llm": mocker.patch("pipelex.cogt.content_generation.llm_generate.get_llm_worker"),
+ "img_gen": mocker.patch("pipelex.cogt.content_generation.img_gen_generate.get_img_gen_worker"),
+ "extract": mocker.patch("pipelex.cogt.content_generation.extract_generate.get_extract_worker"),
+ }
+
+
+class TestLeafReadScope:
+ @pytest.mark.asyncio
+ @pytest.mark.parametrize(("leaf_name", "call_leaf", "in_scope_uri"), DRY_LEAVES)
+ async def test_a_scoped_dry_run_refuses_a_foreign_key(
+ self, mocker: MockerFixture, no_workers: dict[str, Any], leaf_name: str, call_leaf: LeafCall, in_scope_uri: str
+ ) -> None:
+ del in_scope_uri
+ factory = mocker.MagicMock()
+ with pytest.raises(UriReadRefusedError) as exc_info:
+ await call_leaf(FOREIGN_KEY, PipeRunMode.DRY, factory)
+ assert exc_info.value.reason == UriReadRefusalReason.FOREIGN_STORAGE_KEY, leaf_name
+ assert f"of pipe '{PIPE_CODE}'" in str(exc_info.value)
+ assert factory.method_calls == []
+ for worker_spy in no_workers.values():
+ worker_spy.assert_not_called()
+
+ @pytest.mark.asyncio
+ @pytest.mark.usefixtures("no_workers")
+ @pytest.mark.parametrize(("leaf_name", "call_leaf", "in_scope_uri"), DRY_LEAVES)
+ async def test_a_scoped_dry_run_refuses_a_local_path(self, mocker: MockerFixture, leaf_name: str, call_leaf: LeafCall, in_scope_uri: str) -> None:
+ del in_scope_uri
+ with pytest.raises(UriReadRefusedError) as exc_info:
+ await call_leaf(LOCAL_PATH, PipeRunMode.DRY, mocker.MagicMock())
+ assert exc_info.value.reason == UriReadRefusalReason.LOCAL_PATH, leaf_name
+
+ @pytest.mark.asyncio
+ @pytest.mark.usefixtures("no_workers")
+ @pytest.mark.parametrize(("leaf_name", "call_leaf", "in_scope_uri"), DRY_LEAVES)
+ async def test_a_scoped_dry_run_reads_an_in_scope_key(
+ self, mocker: MockerFixture, leaf_name: str, call_leaf: LeafCall, in_scope_uri: str
+ ) -> None:
+ result = await call_leaf(in_scope_uri, PipeRunMode.DRY, mocker.MagicMock())
+ assert result is not None, leaf_name
+
+ @pytest.mark.asyncio
+ @pytest.mark.parametrize(("leaf_name", "call_leaf", "in_scope_uri"), DRY_LEAVES)
+ async def test_a_scoped_live_run_refuses_before_building_a_worker(
+ self, mocker: MockerFixture, no_workers: dict[str, Any], leaf_name: str, call_leaf: LeafCall, in_scope_uri: str
+ ) -> None:
+ del in_scope_uri
+ with pytest.raises(UriReadRefusedError):
+ await call_leaf(FOREIGN_KEY, PipeRunMode.LIVE, mocker.MagicMock())
+ for worker_name, worker_spy in no_workers.items():
+ assert not worker_spy.called, f"{leaf_name} built a {worker_name} worker before refusing"
+
+ @pytest.mark.asyncio
+ async def test_img_gen_single_image_refuses_a_foreign_key(self, no_workers: dict[str, Any]) -> None:
+ with pytest.raises(UriReadRefusedError):
+ await img_gen_single_image(img_gen_assignment=_img_gen_assignment(run_mode=PipeRunMode.LIVE, image_uri=FOREIGN_KEY))
+ no_workers["img_gen"].assert_not_called()
+
+ @pytest.mark.asyncio
+ async def test_img_gen_image_list_refuses_a_local_path(self, no_workers: dict[str, Any]) -> None:
+ with pytest.raises(UriReadRefusedError):
+ await img_gen_image_list(img_gen_assignment=_img_gen_assignment(run_mode=PipeRunMode.LIVE, image_uri=LOCAL_PATH))
+ no_workers["img_gen"].assert_not_called()
+
+ @pytest.mark.asyncio
+ async def test_extract_gen_pages_refuses_a_foreign_key(self, no_workers: dict[str, Any]) -> None:
+ with pytest.raises(UriReadRefusedError):
+ await extract_gen_pages(extract_assignment=_extract_assignment(run_mode=PipeRunMode.LIVE, document_uri=FOREIGN_KEY))
+ no_workers["extract"].assert_not_called()
+
+ @pytest.mark.asyncio
+ @pytest.mark.parametrize(("leaf_name", "module_path", "call_leaf"), READ_FREE_LEAVES)
+ async def test_a_read_free_leaf_calls_the_gate_first(
+ self, mocker: MockerFixture, leaf_name: str, module_path: str, call_leaf: Callable[[Any], Awaitable[Any]]
+ ) -> None:
+ mocker.patch(f"{module_path}.authorize_assignment_reads", side_effect=_GateCalledError)
+ with pytest.raises(_GateCalledError):
+ await call_leaf(mocker.MagicMock())
+ del leaf_name
+
+ @pytest.mark.asyncio
+ async def test_an_unscoped_dry_run_passes_a_foreign_key_and_a_local_path(self, mocker: MockerFixture) -> None:
+ mocker.patch("pipelex.cogt.content_generation.dry_mock.get_report_delegate", return_value=mocker.MagicMock())
+ mocker.patch("pipelex.cogt.content_generation.llm_generate.get_llm_worker")
+ unscoped_metadata = JobMetadata(
+ run_metadata=RunMetadata(user_id="u", pipeline_run_id="run_1", storage_scope="run_1", read_scope=None),
+ )
+ llm_assignment = LLMAssignment(
+ job_metadata=unscoped_metadata,
+ cogt_run_params=CogtRunParams(run_mode=PipeRunMode.DRY),
+ llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),
+ llm_prompt=LLMPrompt(
+ user_text="Describe it.",
+ user_images=[PromptImageUri(uri=FOREIGN_KEY)],
+ user_documents=[PromptDocumentUri(uri=LOCAL_PATH)],
+ ),
+ )
+ result = await llm_gen_text(llm_assignment=llm_assignment)
+ assert result.startswith("DRY RUN:")
diff --git a/tests/unit/pipelex/cogt/content_generation/test_llm_generate_dry_branch.py b/tests/unit/pipelex/cogt/content_generation/test_llm_generate_dry_branch.py
index a26623818..d95b4d381 100644
--- a/tests/unit/pipelex/cogt/content_generation/test_llm_generate_dry_branch.py
+++ b/tests/unit/pipelex/cogt/content_generation/test_llm_generate_dry_branch.py
@@ -29,7 +29,9 @@ from pipelex.system.pipe_run_mode import PipeRunMode
class TestLlmGenerateDryBranch:
def _assignment(self, *, run_mode: PipeRunMode, is_mock_usage: bool = False) -> LLMAssignment:
return LLMAssignment(
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_dry_branch")),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="run_dry_branch")
+ ),
cogt_run_params=CogtRunParams(run_mode=run_mode, is_mock_usage=is_mock_usage),
llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),
llm_prompt=LLMPrompt(),
diff --git a/tests/unit/pipelex/cogt/content_generation/test_render_generate_dry_branch.py b/tests/unit/pipelex/cogt/content_generation/test_render_generate_dry_branch.py
index d572de423..b9fb5398a 100644
--- a/tests/unit/pipelex/cogt/content_generation/test_render_generate_dry_branch.py
+++ b/tests/unit/pipelex/cogt/content_generation/test_render_generate_dry_branch.py
@@ -18,7 +18,9 @@ from pipelex.system.pipe_run_mode import PipeRunMode
class TestRenderGenerateDryBranch:
def _assignment(self, *, run_mode: PipeRunMode) -> RenderPageViewsAssignment:
return RenderPageViewsAssignment(
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_render_dry")),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="run_render_dry")
+ ),
cogt_run_params=CogtRunParams(run_mode=run_mode),
document_uri="file:///tmp/doc.pdf",
page_views_dpi=72,
diff --git a/tests/unit/pipelex/cogt/content_generation/test_search_generate_dry_branch.py b/tests/unit/pipelex/cogt/content_generation/test_search_generate_dry_branch.py
index 6249e3b5a..5f5fda818 100644
--- a/tests/unit/pipelex/cogt/content_generation/test_search_generate_dry_branch.py
+++ b/tests/unit/pipelex/cogt/content_generation/test_search_generate_dry_branch.py
@@ -22,7 +22,9 @@ from pipelex.system.pipe_run_mode import PipeRunMode
class TestSearchGenerateDryBranch:
def _assignment(self, *, run_mode: PipeRunMode) -> SearchAssignment:
return SearchAssignment(
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_search_dry")),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="run_search_dry")
+ ),
cogt_run_params=CogtRunParams(run_mode=run_mode),
query="what is pipelex?",
search_setting=SearchSetting(model="mock-search-handle"),
diff --git a/tests/unit/pipelex/cogt/content_generation/test_templating_generate_dry_branch.py b/tests/unit/pipelex/cogt/content_generation/test_templating_generate_dry_branch.py
index 027b785e7..d92869f3d 100644
--- a/tests/unit/pipelex/cogt/content_generation/test_templating_generate_dry_branch.py
+++ b/tests/unit/pipelex/cogt/content_generation/test_templating_generate_dry_branch.py
@@ -20,7 +20,9 @@ from pipelex.tools.jinja2.template_category import TemplateCategory
class TestTemplatingGenerateDryBranch:
def _assignment(self, *, run_mode: PipeRunMode, template: str) -> TemplatingAssignment:
return TemplatingAssignment(
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_templating_dry")),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="run_templating_dry")
+ ),
cogt_run_params=CogtRunParams(run_mode=run_mode),
context={"the_answer": "42"},
template=template,
diff --git a/tests/unit/pipelex/cogt/extract/test_extract_worker_usage_capture.py b/tests/unit/pipelex/cogt/extract/test_extract_worker_usage_capture.py
index ab58fb7f4..271375f34 100644
--- a/tests/unit/pipelex/cogt/extract/test_extract_worker_usage_capture.py
+++ b/tests/unit/pipelex/cogt/extract/test_extract_worker_usage_capture.py
@@ -65,7 +65,9 @@ def _make_extract_job() -> ExtractJob:
job_params=ExtractJobParams.make_default_extract_job_params(),
job_config=ExtractJobConfig(),
job_report=ExtractJobReport(),
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run")),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="test-user", pipeline_run_id="test-run")
+ ),
)
diff --git a/tests/unit/pipelex/cogt/img_gen/conftest.py b/tests/unit/pipelex/cogt/img_gen/conftest.py
index 9e077719c..53dc783f1 100644
--- a/tests/unit/pipelex/cogt/img_gen/conftest.py
+++ b/tests/unit/pipelex/cogt/img_gen/conftest.py
@@ -62,5 +62,7 @@ def make_img_gen_job(
),
job_config=ImgGenJobConfig(is_sync_mode=False),
job_report=ImgGenJobReport(),
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run")),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="test-user", pipeline_run_id="test-run")
+ ),
)
diff --git a/tests/unit/pipelex/cogt/img_gen/test_img_gen_args_factory.py b/tests/unit/pipelex/cogt/img_gen/test_img_gen_args_factory.py
index 444de1d5b..78071eaa0 100644
--- a/tests/unit/pipelex/cogt/img_gen/test_img_gen_args_factory.py
+++ b/tests/unit/pipelex/cogt/img_gen/test_img_gen_args_factory.py
@@ -81,7 +81,9 @@ class TestImgGenArgsFactory:
),
job_config=ImgGenJobConfig(is_sync_mode=False),
job_report=ImgGenJobReport(),
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run")),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="test-user", pipeline_run_id="test-run")
+ ),
)
@staticmethod
diff --git a/tests/unit/pipelex/cogt/llm/test_llm_worker_current_span.py b/tests/unit/pipelex/cogt/llm/test_llm_worker_current_span.py
index 3662543e8..e6d6b4beb 100644
--- a/tests/unit/pipelex/cogt/llm/test_llm_worker_current_span.py
+++ b/tests/unit/pipelex/cogt/llm/test_llm_worker_current_span.py
@@ -116,7 +116,7 @@ def _make_worker(*, fails: bool = False) -> _RecordingLLMWorker:
def _make_llm_job() -> LLMJob:
"""A job run under a pipe whose span the metadata names as the parent."""
job_metadata = JobMetadata(
- run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-llm", storage_scope="test/scope"),
+ run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-llm", storage_scope="test/scope", read_scope=None),
pipe_code="some_pipe",
otel_context=OtelContext(trace_id=RUN_TRACE_ID, trace_name="some_pipe", trace_name_redacted="some_pipe", span_id=PIPE_SPAN_ID),
)
diff --git a/tests/unit/pipelex/cogt/usage/test_data.py b/tests/unit/pipelex/cogt/usage/test_data.py
index 48165e2f3..7998f17f2 100644
--- a/tests/unit/pipelex/cogt/usage/test_data.py
+++ b/tests/unit/pipelex/cogt/usage/test_data.py
@@ -36,7 +36,9 @@ class UsageFixtures:
def full_job_metadata(*, unit_job_id: UnitJobId, job_category: JobCategory) -> JobMetadata:
"""A JobMetadata with EVERY optional field populated, so leak tests prove the trim."""
return JobMetadata(
- run_metadata=RunMetadata(storage_scope="test/scope", user_id="user-42", pipeline_run_id="plr-fixture", request_id="req-123"),
+ run_metadata=RunMetadata(
+ storage_scope="test/scope", read_scope=None, user_id="user-42", pipeline_run_id="plr-fixture", request_id="req-123"
+ ),
pipe_code="analyze_contract",
session_id="session-abc",
pipe_run_id="a1b2c3d4e5f60718",
diff --git a/tests/unit/pipelex/core/memory/input_shaper/test_anything_envelopes.py b/tests/unit/pipelex/core/memory/input_shaper/test_anything_envelopes.py
index 39ff5463e..9f0f45515 100644
--- a/tests/unit/pipelex/core/memory/input_shaper/test_anything_envelopes.py
+++ b/tests/unit/pipelex/core/memory/input_shaper/test_anything_envelopes.py
@@ -34,7 +34,7 @@ class TestInputShaperAnythingEnvelopes:
input_specs = build_input_specs([("payload", "native.Anything", None)])
provided = {"concept": anything_spelling, "content": content}
- working_memory = InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
stuff = working_memory.root["payload"]
assert stuff.concept.concept_ref == "native.Anything"
@@ -44,7 +44,7 @@ class TestInputShaperAnythingEnvelopes:
input_specs = build_input_specs([("payload", "native.Anything", True)])
provided: dict[str, Any] = {"concept": "native.Anything", "content": [1, "a"]}
- working_memory = InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
stuff = working_memory.root["payload"]
assert stuff.concept.concept_ref == "native.Anything"
@@ -56,7 +56,7 @@ class TestInputShaperAnythingEnvelopes:
provided: dict[str, Any] = {"concept": "native.Anything", "content": [1, "a"]}
with pytest.raises(ListWhereSingularError, match="declares a single"):
- InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library())
+ InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
@pytest.mark.parametrize(
("provided", "expected_concept_ref", "expected_content"),
@@ -73,7 +73,7 @@ class TestInputShaperAnythingEnvelopes:
"""R2 and D6: every concept satisfies `Anything`, and the explicit, more specific concept wins."""
input_specs = build_input_specs([("payload", "native.Anything", None)])
- working_memory = InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
stuff = working_memory.root["payload"]
assert stuff.concept.concept_ref == expected_concept_ref
@@ -83,7 +83,11 @@ class TestInputShaperAnythingEnvelopes:
input_specs = build_input_specs([("payload", "native.Anything", None)])
working_memory = InputShaper.shape(
- {"payload": Question(text="Why?")}, input_specs=input_specs, search_scope="shaper_test", concept_provider=get_concept_library()
+ {"payload": Question(text="Why?")},
+ input_specs=input_specs,
+ search_scope="shaper_test",
+ concept_provider=get_concept_library(),
+ read_scope=None,
)
stuff = working_memory.root["payload"]
@@ -95,7 +99,7 @@ class TestInputShaperAnythingEnvelopes:
input_specs = build_input_specs([("payload", "native.Anything", True)])
provided: list[StuffContent] = [TextContent(text="a"), NumberContent(number=1)]
- working_memory = InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
stuff = working_memory.root["payload"]
assert stuff.concept.concept_ref == "native.Anything"
@@ -106,7 +110,7 @@ class TestInputShaperAnythingEnvelopes:
input_specs = build_input_specs([("payload", "native.Anything", True)])
prebuilt: ListContent[StuffContent] = ListContent(items=[TextContent(text="a"), NumberContent(number=1)])
- working_memory = InputShaper.shape({"payload": prebuilt}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape({"payload": prebuilt}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
stuff = working_memory.root["payload"]
assert stuff.concept.concept_ref == "native.Anything"
@@ -117,14 +121,14 @@ class TestInputShaperAnythingEnvelopes:
prebuilt: ListContent[StuffContent] = ListContent(items=[TextContent(text="a"), NumberContent(number=1)])
with pytest.raises(ListWhereSingularError, match="declares a single"):
- InputShaper.shape({"payload": prebuilt}, input_specs=input_specs, concept_provider=get_concept_library())
+ InputShaper.shape({"payload": prebuilt}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
def test_nested_envelope_escape_holds_an_envelope_shaped_object(self) -> None:
"""An object keyed exactly `concept` and `content` travels as raw data inside an `Anything` envelope."""
input_specs = build_input_specs([("payload", "native.Anything", None)])
provided = {"concept": "native.Anything", "content": {"concept": "x", "content": "y"}}
- working_memory = InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
stuff = working_memory.root["payload"]
assert stuff.concept.concept_ref == "native.Anything"
@@ -136,7 +140,7 @@ class TestInputShaperAnythingEnvelopes:
envelope_shaped_item = {"concept": "Image", "content": {"url": "photo.jpg"}}
provided: dict[str, Any] = {"concept": "native.Anything", "content": [envelope_shaped_item, "caption"]}
- working_memory = InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
stuff = working_memory.root["payload"]
assert stuff.concept.concept_ref == "native.Anything"
@@ -147,7 +151,7 @@ class TestInputShaperAnythingEnvelopes:
input_specs = build_input_specs([("payload", "native.Dynamic", None)])
provided = {"concept": "native.Anything", "content": {"a": 1}}
- working_memory = InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
stuff = working_memory.root["payload"]
assert stuff.concept.concept_ref == "native.Anything"
@@ -160,12 +164,14 @@ class TestInputShaperAnythingEnvelopes:
for multiplicity in (True, 2):
input_specs = build_input_specs([("payload", "native.Anything", multiplicity)])
- working_memory = InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape(
+ {"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None
+ )
assert working_memory.root["payload"].content == prebuilt
single_specs = build_input_specs([("payload", "native.Anything", None)])
with pytest.raises(ListWhereSingularError):
- InputShaper.shape({"payload": provided}, input_specs=single_specs, concept_provider=get_concept_library())
+ InputShaper.shape({"payload": provided}, input_specs=single_specs, concept_provider=get_concept_library(), read_scope=None)
def test_a_prebuilt_list_as_a_list_item_is_refused(self) -> None:
"""An item of a plural slot that is itself a list is refused, whether it is a bare array or a `ListContent`."""
@@ -173,14 +179,14 @@ class TestInputShaperAnythingEnvelopes:
provided: list[ListContent[TextContent]] = [ListContent(items=[TextContent(text="a"), TextContent(text="b")])]
with pytest.raises(WrongScalarKindError, match="a single value, not a list"):
- InputShaper.shape({"items": provided}, input_specs=input_specs, concept_provider=get_concept_library())
+ InputShaper.shape({"items": provided}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
def test_anything_envelope_at_a_dynamic_slot_takes_its_multiplicity_from_its_content(self) -> None:
"""At a `Dynamic` slot the signature cannot say list or single, so the content does, as for every other envelope."""
input_specs = build_input_specs([("payload", "native.Dynamic", None)])
provided = {"concept": "native.Anything", "content": [1, 2]}
- working_memory = InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
stuff = working_memory.root["payload"]
assert stuff.concept.concept_ref == "native.Anything"
diff --git a/tests/unit/pipelex/core/memory/input_shaper/test_anything_schema_agreement.py b/tests/unit/pipelex/core/memory/input_shaper/test_anything_schema_agreement.py
index c9413a385..26b674754 100644
--- a/tests/unit/pipelex/core/memory/input_shaper/test_anything_schema_agreement.py
+++ b/tests/unit/pipelex/core/memory/input_shaper/test_anything_schema_agreement.py
@@ -51,7 +51,7 @@ class TestAnythingSchemaAgreement:
# The `types-jsonschema` overloads of `is_valid` carry an `Unknown` arm.
schema_admits = jsonschema.Draft202012Validator(json_schema).is_valid(provided_value) # pyright: ignore[reportUnknownMemberType]
try:
- InputShaper.shape({"payload": provided_value}, input_specs=input_specs, concept_provider=get_concept_library())
+ InputShaper.shape({"payload": provided_value}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
shaper_takes = True
except InputShapingError:
shaper_takes = False
diff --git a/tests/unit/pipelex/core/memory/input_shaper/test_csv_by_signature.py b/tests/unit/pipelex/core/memory/input_shaper/test_csv_by_signature.py
index 77494c332..35c347cc0 100644
--- a/tests/unit/pipelex/core/memory/input_shaper/test_csv_by_signature.py
+++ b/tests/unit/pipelex/core/memory/input_shaper/test_csv_by_signature.py
@@ -36,7 +36,9 @@ class TestInputShaperCsvBySignature:
def test_bare_csv_path_under_declared_list(self, people_csv: Path):
"""A bare absolute CSV path under `ShaperPerson[]` reads rows into ListContent[ShaperPerson]."""
input_specs = build_input_specs([("people", "shaper_test.ShaperPerson", True)])
- working_memory = InputShaper.shape({"people": str(people_csv)}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape(
+ {"people": str(people_csv)}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None
+ )
people_stuff = working_memory.get_stuff("people")
assert people_stuff.concept.concept_ref == "shaper_test.ShaperPerson"
@@ -46,7 +48,7 @@ class TestInputShaperCsvBySignature:
"""The exact {"url": <csv>} wrapper under `ShaperPerson[]` reads the same table — no envelope."""
value: dict[str, Any] = {"url": str(people_csv)}
input_specs = build_input_specs([("people", "shaper_test.ShaperPerson", True)])
- working_memory = InputShaper.shape({"people": value}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape({"people": value}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
people_stuff = working_memory.get_stuff("people")
assert people_stuff.concept.concept_ref == "shaper_test.ShaperPerson"
@@ -57,7 +59,7 @@ class TestInputShaperCsvBySignature:
"""A bare relative CSV path resolves against inputs_base_dir before the table is read."""
input_specs = build_input_specs([("people", "shaper_test.ShaperPerson", True)])
working_memory = InputShaper.shape(
- {"people": "people.csv"}, input_specs=input_specs, inputs_base_dir=tmp_path, concept_provider=get_concept_library()
+ {"people": "people.csv"}, input_specs=input_specs, inputs_base_dir=tmp_path, concept_provider=get_concept_library(), read_scope=None
)
self._assert_people_list(working_memory.get_stuff("people").content)
@@ -65,7 +67,9 @@ class TestInputShaperCsvBySignature:
def test_fixed_count_matching_rows(self, people_csv: Path):
"""A `ShaperPerson[2]` input accepts a two-row CSV."""
input_specs = build_input_specs([("people", "shaper_test.ShaperPerson", 2)])
- working_memory = InputShaper.shape({"people": str(people_csv)}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape(
+ {"people": str(people_csv)}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None
+ )
self._assert_people_list(working_memory.get_stuff("people").content)
@@ -73,29 +77,31 @@ class TestInputShaperCsvBySignature:
"""A `ShaperPerson[3]` input rejects a two-row CSV with the D2 count-mismatch error."""
input_specs = build_input_specs([("people", "shaper_test.ShaperPerson", 3)])
with pytest.raises(MultiplicityCountMismatchError, match="'people'"):
- InputShaper.shape({"people": str(people_csv)}, input_specs=input_specs, concept_provider=get_concept_library())
+ InputShaper.shape({"people": str(people_csv)}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
def test_csv_under_singular_is_wrong_scalar_kind(self, people_csv: Path):
"""A CSV names a TABLE; under a singular `ShaperPerson` a bare path is a wrong-kind error."""
input_specs = build_input_specs([("people", "shaper_test.ShaperPerson", None)])
with pytest.raises(WrongScalarKindError, match="'people'"):
- InputShaper.shape({"people": str(people_csv)}, input_specs=input_specs, concept_provider=get_concept_library())
+ InputShaper.shape({"people": str(people_csv)}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
def test_non_tabular_string_falls_through(self):
"""A bare non-tabular string under `ShaperPerson[]` is NOT hijacked as a table — normal D5 error."""
input_specs = build_input_specs([("people", "shaper_test.ShaperPerson", True)])
with pytest.raises(WrongScalarKindError, match="'people'"):
- InputShaper.shape({"people": "people.txt"}, input_specs=input_specs, concept_provider=get_concept_library())
+ InputShaper.shape({"people": "people.txt"}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
def test_url_dict_with_sibling_keys_stays_record(self, people_csv: Path):
"""A record dict that merely HAS a url field is not a table ref — validated as a record."""
value: dict[str, Any] = {"url": str(people_csv), "label": "roster"}
input_specs = build_input_specs([("people", "shaper_test.ShaperPerson", True)])
with pytest.raises(StructureValidationError, match="'people'"):
- InputShaper.shape({"people": value}, input_specs=input_specs, concept_provider=get_concept_library())
+ InputShaper.shape({"people": value}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
def test_remote_csv_url_raises_csv_error(self):
"""A remote tabular url is rejected loudly (local paths only in v1), not read or ignored."""
input_specs = build_input_specs([("people", "shaper_test.ShaperPerson", True)])
with pytest.raises(CsvError, match="local file paths only"):
- InputShaper.shape({"people": "https://example.com/people.csv"}, input_specs=input_specs, concept_provider=get_concept_library())
+ InputShaper.shape(
+ {"people": "https://example.com/people.csv"}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None
+ )
diff --git a/tests/unit/pipelex/core/memory/input_shaper/test_csv_read_scope.py b/tests/unit/pipelex/core/memory/input_shaper/test_csv_read_scope.py
new file mode 100644
index 000000000..6b9c850a8
--- /dev/null
+++ b/tests/unit/pipelex/core/memory/input_shaper/test_csv_read_scope.py
@@ -0,0 +1,60 @@
+"""A table given as a CSV input is a local file, so a run with a read scope refuses it before reading.
+
+Every shaping route that reaches the table reader is covered: the bare path and the ``{"url": ...}``
+wrapper read by signature, and the explicit envelope the bottom-up factory reads (Case 2.5).
+"""
+
+from pathlib import Path
+from typing import TYPE_CHECKING, Any, cast
+
+import pytest
+
+from pipelex.core.memory.input_shaper import InputShaper
+from pipelex.core.stuffs.list_content import ListContent
+from pipelex.interpreter_hub import get_concept_library
+from pipelex.kernel.memory_ops import shape_inputs
+from pipelex.tools.uri.exceptions import UriReadRefusalReason, UriReadRefusedError
+from tests.unit.pipelex.core.memory.input_shaper.data import build_input_specs
+
+if TYPE_CHECKING:
+ from pipelex.core.stuffs.stuff_content import StuffContent
+
+PEOPLE_CSV_CONTENT = "name\nAda Lovelace\nGrace Hopper\n"
+
+
+class TestCsvInputReadScope:
+ @pytest.fixture
+ def people_csv(self, tmp_path: Path) -> Path:
+ csv_path = tmp_path / "people.csv"
+ csv_path.write_text(PEOPLE_CSV_CONTENT, encoding="utf-8")
+ return csv_path
+
+ def _values(self, people_csv: Path) -> list[Any]:
+ return [
+ str(people_csv),
+ {"url": str(people_csv)},
+ f"file://{people_csv}",
+ {"concept": "shaper_test.ShaperPerson", "content": {"url": str(people_csv)}},
+ ]
+
+ def test_a_scoped_run_refuses_every_route_to_the_table_reader(self, people_csv: Path) -> None:
+ input_specs = build_input_specs([("people", "shaper_test.ShaperPerson", True)])
+ for value in self._values(people_csv):
+ with pytest.raises(UriReadRefusedError) as exc_info:
+ InputShaper.shape({"people": value}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope="org_abc")
+ assert exc_info.value.reason == UriReadRefusalReason.LOCAL_PATH
+ assert "input 'people'" in str(exc_info.value)
+ assert str(people_csv) not in str(exc_info.value)
+
+ def test_the_kernel_shaping_op_refuses_it_too(self, people_csv: Path) -> None:
+ input_specs = build_input_specs([("people", "shaper_test.ShaperPerson", True)])
+ with pytest.raises(UriReadRefusedError):
+ shape_inputs(inputs={"people": str(people_csv)}, concept_provider=get_concept_library(), input_specs=input_specs, read_scope="org_abc")
+
+ def test_an_unscoped_run_reads_every_route(self, people_csv: Path) -> None:
+ input_specs = build_input_specs([("people", "shaper_test.ShaperPerson", True)])
+ for value in self._values(people_csv):
+ working_memory = InputShaper.shape({"people": value}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
+ people_content = working_memory.get_stuff("people").content
+ assert isinstance(people_content, ListContent)
+ assert len(cast("ListContent[StuffContent]", people_content).items) == 2
diff --git a/tests/unit/pipelex/core/memory/input_shaper/test_errors.py b/tests/unit/pipelex/core/memory/input_shaper/test_errors.py
index 3b88635dc..08a5a7148 100644
--- a/tests/unit/pipelex/core/memory/input_shaper/test_errors.py
+++ b/tests/unit/pipelex/core/memory/input_shaper/test_errors.py
@@ -127,7 +127,7 @@ class TestInputShaperErrors:
input_specs = build_input_specs([("my_input", concept_ref, multiplicity)])
with pytest.raises(expected_exception, match=error_match) as exc_info:
- InputShaper.shape({"my_input": provided_value}, input_specs=input_specs, concept_provider=get_concept_library())
+ InputShaper.shape({"my_input": provided_value}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
# D4 mandates the rendered expected-shape template appears in every shaping-error message.
assert "Expected shape:" in str(exc_info.value), f"Missing rendered shape for {test_name}"
@@ -137,7 +137,7 @@ class TestInputShaperErrors:
input_specs = build_input_specs([("question", "native.Text", None)])
with pytest.raises(UnknownInputNameError, match="not declared") as exc_info:
- InputShaper.shape({"quesion": "typo"}, input_specs=input_specs, concept_provider=get_concept_library())
+ InputShaper.shape({"quesion": "typo"}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
message = str(exc_info.value)
assert "'question'" in message, "Unknown-name error should list the declared inputs"
diff --git a/tests/unit/pipelex/core/memory/input_shaper/test_explicit_forms.py b/tests/unit/pipelex/core/memory/input_shaper/test_explicit_forms.py
index 3a3656b42..9776cb9df 100644
--- a/tests/unit/pipelex/core/memory/input_shaper/test_explicit_forms.py
+++ b/tests/unit/pipelex/core/memory/input_shaper/test_explicit_forms.py
@@ -24,7 +24,7 @@ class TestInputShaperExplicitForms:
input_specs = build_input_specs([("invoice", "shaper_test.ShaperInvoice", None)])
provided = {"concept": "shaper_test.ShaperInvoice", "content": {"invoice_number": "INV-001", "amount": 1250.0}}
- working_memory = InputShaper.shape({"invoice": provided}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape({"invoice": provided}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
stuff = working_memory.root["invoice"]
pretty_print(stuff, title="envelope structured")
@@ -36,7 +36,7 @@ class TestInputShaperExplicitForms:
input_specs = build_input_specs([("answer", "native.Text", None)])
provided = {"concept": "shaper_test.Question", "content": "What are the fees?"}
- working_memory = InputShaper.shape({"answer": provided}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape({"answer": provided}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
stuff = working_memory.root["answer"]
pretty_print(stuff, title="envelope refining wins")
@@ -53,7 +53,7 @@ class TestInputShaperExplicitForms:
input_specs = build_input_specs([("opening", "native.Time", None)])
provided = {"concept": concept_ref, "content": "15:40:00+02:00"}
- working_memory = InputShaper.shape({"opening": provided}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape({"opening": provided}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
stuff = working_memory.root["opening"]
assert stuff.concept.concept_ref == concept_ref
@@ -63,7 +63,7 @@ class TestInputShaperExplicitForms:
input_specs = build_input_specs([("opening", "native.Time", None)])
provided = {"concept": "native.Time", "content": datetime.time(15, 40)}
- working_memory = InputShaper.shape({"opening": provided}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape({"opening": provided}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
assert working_memory.root["opening"].content == TimeContent(time=datetime.time(15, 40))
@@ -73,7 +73,7 @@ class TestInputShaperExplicitForms:
provided = ShaperInvoice(invoice_number="INV-002", amount=99.0)
working_memory = InputShaper.shape(
- {"invoice": provided}, input_specs=input_specs, search_scope="shaper_test", concept_provider=get_concept_library()
+ {"invoice": provided}, input_specs=input_specs, search_scope="shaper_test", concept_provider=get_concept_library(), read_scope=None
)
stuff = working_memory.root["invoice"]
@@ -91,7 +91,7 @@ class TestInputShaperExplicitForms:
provided = [Question(text="a"), Question(text="b")]
working_memory = InputShaper.shape(
- {"questions": provided}, input_specs=input_specs, search_scope="shaper_test", concept_provider=get_concept_library()
+ {"questions": provided}, input_specs=input_specs, search_scope="shaper_test", concept_provider=get_concept_library(), read_scope=None
)
stuff = working_memory.root["questions"]
@@ -105,7 +105,9 @@ class TestInputShaperExplicitForms:
provided = [Question(text="a")]
with pytest.raises(ExplicitConceptIncompatibleError, match="not compatible"):
- InputShaper.shape({"priorities": provided}, input_specs=input_specs, search_scope="shaper_test", concept_provider=get_concept_library())
+ InputShaper.shape(
+ {"priorities": provided}, input_specs=input_specs, search_scope="shaper_test", concept_provider=get_concept_library(), read_scope=None
+ )
def test_explicit_list_into_singular_raises(self) -> None:
"""D2: an explicit ListContent (or envelope-with-list) must not fill a singular-declared slot.
@@ -117,7 +119,9 @@ class TestInputShaperExplicitForms:
provided: ListContent[Question] = ListContent(items=[Question(text="a"), Question(text="b")])
with pytest.raises(ListWhereSingularError, match="declares a single"):
- InputShaper.shape({"question": provided}, input_specs=input_specs, search_scope="shaper_test", concept_provider=get_concept_library())
+ InputShaper.shape(
+ {"question": provided}, input_specs=input_specs, search_scope="shaper_test", concept_provider=get_concept_library(), read_scope=None
+ )
def test_explicit_list_wrong_fixed_count_raises(self) -> None:
"""D2: an explicit ListContent whose length differs from a declared [N] count is a D4 error."""
@@ -125,7 +129,9 @@ class TestInputShaperExplicitForms:
provided: ListContent[Question] = ListContent(items=[Question(text="only-one")])
with pytest.raises(MultiplicityCountMismatchError, match="exactly 2 items"):
- InputShaper.shape({"questions": provided}, input_specs=input_specs, search_scope="shaper_test", concept_provider=get_concept_library())
+ InputShaper.shape(
+ {"questions": provided}, input_specs=input_specs, search_scope="shaper_test", concept_provider=get_concept_library(), read_scope=None
+ )
@pytest.mark.parametrize("declared_concept", ["native.Number", "native.Anything"])
def test_explicit_number_envelope_holding_nan_raises(self, declared_concept: str) -> None:
@@ -134,14 +140,14 @@ class TestInputShaperExplicitForms:
provided: dict[str, Any] = {"concept": "Number", "content": {"number": float("nan")}}
with pytest.raises(ValidationError, match="number must be finite"):
- InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library())
+ InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
def test_explicit_single_into_fixed_count_raises(self) -> None:
"""A single value is one item, never the N a fixed count declares."""
input_specs = build_input_specs([("answers", "native.Text", 2)])
provided = {"concept": "native.Text", "content": "hello"}
with pytest.raises(MultiplicityCountMismatchError, match="but you provided 1"):
- InputShaper.shape({"answers": provided}, input_specs=input_specs, concept_provider=get_concept_library())
+ InputShaper.shape({"answers": provided}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
def test_explicit_list_into_list_slot_ok(self) -> None:
"""D2/D6: an explicit ListContent whose length matches a declared list slot shapes cleanly."""
@@ -149,7 +155,7 @@ class TestInputShaperExplicitForms:
provided: ListContent[Question] = ListContent(items=[Question(text="a"), Question(text="b")])
working_memory = InputShaper.shape(
- {"questions": provided}, input_specs=input_specs, search_scope="shaper_test", concept_provider=get_concept_library()
+ {"questions": provided}, input_specs=input_specs, search_scope="shaper_test", concept_provider=get_concept_library(), read_scope=None
)
stuff = working_memory.root["questions"]
@@ -163,7 +169,7 @@ class TestInputShaperExplicitForms:
provided: ListContent[Question] = ListContent(items=[Question(text="a"), Question(text="b")])
working_memory = InputShaper.shape(
- {"payload": provided}, input_specs=input_specs, search_scope="shaper_test", concept_provider=get_concept_library()
+ {"payload": provided}, input_specs=input_specs, search_scope="shaper_test", concept_provider=get_concept_library(), read_scope=None
)
stuff = working_memory.root["payload"]
@@ -181,7 +187,7 @@ class TestInputShaperExplicitForms:
input_specs = build_input_specs([("weird", "shaper_test.ShaperWeird", None)])
provided = {"concept": "shaper_test.ShaperWeird", "content": {"concept": "x", "content": "y"}}
- working_memory = InputShaper.shape({"weird": provided}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape({"weird": provided}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
stuff = working_memory.root["weird"]
pretty_print(stuff, title="collision rule")
@@ -200,7 +206,7 @@ class TestInputShaperExplicitForms:
input_specs = build_input_specs([("pics", "native.Image", True)])
provided: dict[str, Any] = {"concept": "native.Image", "content": []}
- working_memory = InputShaper.shape({"pics": provided}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape({"pics": provided}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
stuff = working_memory.root["pics"]
pretty_print(stuff, title="envelope empty list")
@@ -212,8 +218,8 @@ class TestInputShaperExplicitForms:
input_specs = build_input_specs([("pics", "native.Image", True)])
envelope: dict[str, Any] = {"concept": "native.Image", "content": []}
- via_envelope = InputShaper.shape({"pics": envelope}, input_specs=input_specs, concept_provider=get_concept_library())
- via_bare = InputShaper.shape({"pics": []}, input_specs=input_specs, concept_provider=get_concept_library())
+ via_envelope = InputShaper.shape({"pics": envelope}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
+ via_bare = InputShaper.shape({"pics": []}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
assert via_envelope.root["pics"].content == via_bare.root["pics"].content
assert via_envelope.root["pics"].concept.concept_ref == via_bare.root["pics"].concept.concept_ref
@@ -223,7 +229,7 @@ class TestInputShaperExplicitForms:
input_specs = build_input_specs([("questions", "shaper_test.Question", True)])
provided: dict[str, Any] = {"concept": "shaper_test.Question", "content": []}
- working_memory = InputShaper.shape({"questions": provided}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape({"questions": provided}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
stuff = working_memory.root["questions"]
assert stuff.concept.concept_ref == "shaper_test.Question"
diff --git a/tests/unit/pipelex/core/memory/input_shaper/test_fallback_refusal.py b/tests/unit/pipelex/core/memory/input_shaper/test_fallback_refusal.py
index 57cfa105a..e4ff57745 100644
--- a/tests/unit/pipelex/core/memory/input_shaper/test_fallback_refusal.py
+++ b/tests/unit/pipelex/core/memory/input_shaper/test_fallback_refusal.py
@@ -34,7 +34,7 @@ class TestInputShaperFallbackRefusal:
records_input: dict[str, Any] = {"records": [{"a": 1}, {"b": 2}]}
with pytest.raises(StructureValidationError) as exc_info:
- InputShaper.shape(records_input, input_specs=input_specs, concept_provider=get_concept_library())
+ InputShaper.shape(records_input, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
error = exc_info.value
message = str(error)
@@ -58,7 +58,7 @@ class TestInputShaperFallbackRefusal:
payload_input: dict[str, Any] = {"payload": 3}
with pytest.raises(StructureValidationError, match=r"Input 'payload' could not be built as 'native\.Dynamic'") as exc_info:
- InputShaper.shape(payload_input, input_specs=input_specs, concept_provider=get_concept_library())
+ InputShaper.shape(payload_input, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
assert "a number (3)" in str(exc_info.value)
assert isinstance(exc_info.value.__cause__, StuffFactoryError)
@@ -79,7 +79,7 @@ class TestInputShaperFallbackRefusal:
payload_input: dict[str, Any] = {"payload": provided_value}
with pytest.raises(StructureValidationError, match="with no reading for this one") as exc_info:
- InputShaper.shape(payload_input, input_specs=input_specs, concept_provider=get_concept_library())
+ InputShaper.shape(payload_input, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
user_action = exc_info.value.user_action
assert user_action is not None
@@ -107,7 +107,7 @@ class TestInputShaperFallbackRefusal:
payload_input: dict[str, Any] = {"payload": provided_value}
with pytest.raises(StructureValidationError, match=r"Input 'payload' could not be built as") as exc_info:
- InputShaper.shape(payload_input, input_specs=input_specs, concept_provider=get_concept_library())
+ InputShaper.shape(payload_input, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
message = str(exc_info.value)
assert factory_reason in message
diff --git a/tests/unit/pipelex/core/memory/input_shaper/test_file_paths.py b/tests/unit/pipelex/core/memory/input_shaper/test_file_paths.py
index 72459d5d9..a652f2021 100644
--- a/tests/unit/pipelex/core/memory/input_shaper/test_file_paths.py
+++ b/tests/unit/pipelex/core/memory/input_shaper/test_file_paths.py
@@ -23,7 +23,7 @@ class TestInputShaperFilePaths:
"""A bare relative path for an Image-refining input becomes the base_dir-resolved path."""
input_specs = build_input_specs([("photo", "shaper_test.Photo", None)])
working_memory = InputShaper.shape(
- {"photo": "photo.jpg"}, input_specs=input_specs, inputs_base_dir=tmp_path, concept_provider=get_concept_library()
+ {"photo": "photo.jpg"}, input_specs=input_specs, inputs_base_dir=tmp_path, concept_provider=get_concept_library(), read_scope=None
)
photo_stuff = working_memory.get_stuff("photo")
@@ -36,7 +36,7 @@ class TestInputShaperFilePaths:
absolute_path = str(tmp_path / "elsewhere" / "photo.jpg")
input_specs = build_input_specs([("photo", "shaper_test.Photo", None)])
working_memory = InputShaper.shape(
- {"photo": absolute_path}, input_specs=input_specs, inputs_base_dir=tmp_path, concept_provider=get_concept_library()
+ {"photo": absolute_path}, input_specs=input_specs, inputs_base_dir=tmp_path, concept_provider=get_concept_library(), read_scope=None
)
photo_stuff = working_memory.get_stuff("photo")
@@ -46,7 +46,7 @@ class TestInputShaperFilePaths:
def test_bare_relative_path_without_base_dir_untouched(self):
"""No base_dir (in-process / inline-JSON callers): a relative path keeps today's CWD contract."""
input_specs = build_input_specs([("photo", "shaper_test.Photo", None)])
- working_memory = InputShaper.shape({"photo": "photo.jpg"}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape({"photo": "photo.jpg"}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
photo_stuff = working_memory.get_stuff("photo")
assert isinstance(photo_stuff.content, Photo)
@@ -65,7 +65,7 @@ class TestInputShaperFilePaths:
"""Remote URLs and scheme-qualified URIs are not local paths — never rewritten."""
input_specs = build_input_specs([("photo", "shaper_test.Photo", None)])
working_memory = InputShaper.shape(
- {"photo": url_value}, input_specs=input_specs, inputs_base_dir=tmp_path, concept_provider=get_concept_library()
+ {"photo": url_value}, input_specs=input_specs, inputs_base_dir=tmp_path, concept_provider=get_concept_library(), read_scope=None
)
photo_stuff = working_memory.get_stuff("photo")
@@ -77,7 +77,11 @@ class TestInputShaperFilePaths:
absolute_path = str(tmp_path / "other" / "b.pdf")
input_specs = build_input_specs([("exhibits", "shaper_test.Exhibit", True)])
working_memory = InputShaper.shape(
- {"exhibits": ["a.pdf", absolute_path]}, input_specs=input_specs, inputs_base_dir=tmp_path, concept_provider=get_concept_library()
+ {"exhibits": ["a.pdf", absolute_path]},
+ input_specs=input_specs,
+ inputs_base_dir=tmp_path,
+ concept_provider=get_concept_library(),
+ read_scope=None,
)
exhibits_stuff = working_memory.get_stuff("exhibits")
@@ -92,7 +96,7 @@ class TestInputShaperFilePaths:
value: dict[str, Any] = {"url": "photo.jpg"}
input_specs = build_input_specs([("photo", "shaper_test.Photo", None)])
working_memory = InputShaper.shape(
- {"photo": value}, input_specs=input_specs, inputs_base_dir=tmp_path, concept_provider=get_concept_library()
+ {"photo": value}, input_specs=input_specs, inputs_base_dir=tmp_path, concept_provider=get_concept_library(), read_scope=None
)
photo_stuff = working_memory.get_stuff("photo")
@@ -103,7 +107,7 @@ class TestInputShaperFilePaths:
"""A ~-prefixed path is home-anchored: it expands to the home dir, never joined onto base_dir."""
input_specs = build_input_specs([("photo", "shaper_test.Photo", None)])
working_memory = InputShaper.shape(
- {"photo": "~/photo.jpg"}, input_specs=input_specs, inputs_base_dir=tmp_path, concept_provider=get_concept_library()
+ {"photo": "~/photo.jpg"}, input_specs=input_specs, inputs_base_dir=tmp_path, concept_provider=get_concept_library(), read_scope=None
)
photo_stuff = working_memory.get_stuff("photo")
@@ -113,7 +117,7 @@ class TestInputShaperFilePaths:
def test_bare_tilde_path_expands_without_base_dir(self):
"""No base_dir: a ~-prefixed path still expands to home (~ is home-anchored, not CWD-relative)."""
input_specs = build_input_specs([("photo", "shaper_test.Photo", None)])
- working_memory = InputShaper.shape({"photo": "~/photo.jpg"}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape({"photo": "~/photo.jpg"}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
photo_stuff = working_memory.get_stuff("photo")
assert isinstance(photo_stuff.content, Photo)
diff --git a/tests/unit/pipelex/core/memory/input_shaper/test_json_envelopes.py b/tests/unit/pipelex/core/memory/input_shaper/test_json_envelopes.py
index 4521b0122..e205abd60 100644
--- a/tests/unit/pipelex/core/memory/input_shaper/test_json_envelopes.py
+++ b/tests/unit/pipelex/core/memory/input_shaper/test_json_envelopes.py
@@ -12,7 +12,7 @@ class TestInputShaperJSONEnvelopes:
input_specs = build_input_specs([("payload", "native.JSON", None)])
provided = {"concept": "native.JSON", "content": {"json_obj": {"a": 1}}}
- working_memory = InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
stuff = working_memory.root["payload"]
assert stuff.concept.concept_ref == "native.JSON"
@@ -24,7 +24,7 @@ class TestInputShaperJSONEnvelopes:
envelope_shaped_object = {"concept": "Image", "content": {"url": "photo.jpg"}}
provided: dict[str, Any] = {"concept": "native.JSON", "content": [{"json_obj": envelope_shaped_object}, {"json_obj": {"b": 2}}]}
- working_memory = InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape({"payload": provided}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
stuff = working_memory.root["payload"]
assert stuff.concept.concept_ref == "native.JSON"
diff --git a/tests/unit/pipelex/core/memory/input_shaper/test_list_of_objects_report.py b/tests/unit/pipelex/core/memory/input_shaper/test_list_of_objects_report.py
index 6e59c2730..6df1dac42 100644
--- a/tests/unit/pipelex/core/memory/input_shaper/test_list_of_objects_report.py
+++ b/tests/unit/pipelex/core/memory/input_shaper/test_list_of_objects_report.py
@@ -60,11 +60,11 @@ class TestListOfObjectsReport:
records_input: dict[str, Any] = {"records": LIST_OF_OBJECTS}
if isinstance(expected, StuffContent):
- working_memory = InputShaper.shape(records_input, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape(records_input, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
stuff = working_memory.root["records"]
assert stuff.concept.concept_ref == concept_ref
assert stuff.content == expected
return
with pytest.raises(expected, match="Input 'records'"):
- InputShaper.shape(records_input, input_specs=input_specs, concept_provider=get_concept_library())
+ InputShaper.shape(records_input, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None)
diff --git a/tests/unit/pipelex/core/memory/input_shaper/test_multiplicity.py b/tests/unit/pipelex/core/memory/input_shaper/test_multiplicity.py
index c2666e438..af0a979bc 100644
--- a/tests/unit/pipelex/core/memory/input_shaper/test_multiplicity.py
+++ b/tests/unit/pipelex/core/memory/input_shaper/test_multiplicity.py
@@ -143,7 +143,9 @@ class TestInputShaperMultiplicity:
log.info(f"Testing multiplicity case: {test_name}")
input_specs = build_input_specs([("my_input", concept_ref, multiplicity)])
- working_memory = InputShaper.shape({"my_input": provided_value}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape(
+ {"my_input": provided_value}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None
+ )
stuff = working_memory.root["my_input"]
pretty_print(stuff, title=f"Result for {test_name}")
diff --git a/tests/unit/pipelex/core/memory/input_shaper/test_provider_injection.py b/tests/unit/pipelex/core/memory/input_shaper/test_provider_injection.py
index 5cf218750..5ab796e09 100644
--- a/tests/unit/pipelex/core/memory/input_shaper/test_provider_injection.py
+++ b/tests/unit/pipelex/core/memory/input_shaper/test_provider_injection.py
@@ -119,6 +119,7 @@ class TestConceptProviderInjection:
{"note": "shaped through the provider"},
concept_provider=stub,
input_specs=InputStuffSpecs(root={"note": StuffSpec(concept=note_concept)}),
+ read_scope=None,
)
stuff = memory.get_stuff(name="note")
diff --git a/tests/unit/pipelex/core/memory/input_shaper/test_scalar_arms.py b/tests/unit/pipelex/core/memory/input_shaper/test_scalar_arms.py
index 9fd34f637..1b1baa474 100644
--- a/tests/unit/pipelex/core/memory/input_shaper/test_scalar_arms.py
+++ b/tests/unit/pipelex/core/memory/input_shaper/test_scalar_arms.py
@@ -120,7 +120,9 @@ class TestInputShaperScalarArms:
log.info(f"Testing scalar arm case: {test_name}")
input_specs = build_input_specs([("my_input", concept_ref, None)])
- working_memory = InputShaper.shape({"my_input": provided_value}, input_specs=input_specs, concept_provider=get_concept_library())
+ working_memory = InputShaper.shape(
+ {"my_input": provided_value}, input_specs=input_specs, concept_provider=get_concept_library(), read_scope=None
+ )
stuff = working_memory.root["my_input"]
pretty_print(stuff, title=f"Result for {test_name}")
diff --git a/tests/unit/pipelex/core/stuffs/test_stuff_factory_empty_list.py b/tests/unit/pipelex/core/stuffs/test_stuff_factory_empty_list.py
index 567657a60..ee1f3e63f 100644
--- a/tests/unit/pipelex/core/stuffs/test_stuff_factory_empty_list.py
+++ b/tests/unit/pipelex/core/stuffs/test_stuff_factory_empty_list.py
@@ -37,6 +37,7 @@ class TestStuffFactoryEmptyList:
stuff_content_or_data={"concept": concept_ref, "content": []},
concept_provider=get_concept_library(),
name="pics",
+ read_scope=None,
)
pretty_print(result, title=f"empty list under {concept_ref}")
@@ -54,6 +55,7 @@ class TestStuffFactoryEmptyList:
stuff_content_or_data={"concept": "native.Image", "content": []},
concept_provider=get_concept_library(),
name="pics",
+ read_scope=None,
)
content = cast("ListContent[StuffContent]", result.content)
@@ -72,6 +74,7 @@ class TestStuffFactoryEmptyList:
stuff_content_or_data=ListContent[StuffContent](items=[]),
concept_provider=get_concept_library(),
name="pics",
+ read_scope=None,
)
def test_non_empty_envelope_list_is_unaffected(self) -> None:
@@ -80,6 +83,7 @@ class TestStuffFactoryEmptyList:
stuff_content_or_data={"concept": "native.Text", "content": ["a", "b"]},
concept_provider=get_concept_library(),
name="notes",
+ read_scope=None,
)
assert result.concept.concept_ref == "native.Text"
diff --git a/tests/unit/pipelex/core/stuffs/test_stuff_factory_implicit_memory.py b/tests/unit/pipelex/core/stuffs/test_stuff_factory_implicit_memory.py
index 8cf9426c5..d24952dd3 100644
--- a/tests/unit/pipelex/core/stuffs/test_stuff_factory_implicit_memory.py
+++ b/tests/unit/pipelex/core/stuffs/test_stuff_factory_implicit_memory.py
@@ -133,7 +133,7 @@ class TestStuffFactoryImplicitMemory:
log.verbose(f"setup_test_concept: {setup_test_concept}")
result = StuffFactory.make_stuff_from_stuff_content_or_data(
- name=stuff_name, code=stuff_code, stuff_content_or_data=stuff_content_or_data, concept_provider=get_concept_library()
+ name=stuff_name, code=stuff_code, stuff_content_or_data=stuff_content_or_data, concept_provider=get_concept_library(), read_scope=None
)
pretty_print(result, title=f"Result for test case: {test_name}")
pretty_print(expected_stuff, title=f"Expected stuff for test case: {test_name}")
@@ -169,6 +169,7 @@ class TestStuffFactoryImplicitMemoryWithSearchDomains:
stuff_content_or_data=stuff_content_or_data,
search_scope=search_scope,
concept_provider=get_concept_library(),
+ read_scope=None,
)
pretty_print(result, title=f"Result for test case: {test_name}")
@@ -208,4 +209,5 @@ class TestStuffFactoryImplicitMemoryErrors:
stuff_content_or_data=stuff_content_or_data,
search_scope=search_scope,
concept_provider=get_concept_library(),
+ read_scope=None,
)
diff --git a/tests/unit/pipelex/kernel/test_kernel_boot_contract.py b/tests/unit/pipelex/kernel/test_kernel_boot_contract.py
index b9e95adee..5ba8a8315 100644
--- a/tests/unit/pipelex/kernel/test_kernel_boot_contract.py
+++ b/tests/unit/pipelex/kernel/test_kernel_boot_contract.py
@@ -176,7 +176,7 @@ _KERNEL_CALL_SCRIPT = textwrap.dedent(
RuntimeBoot.make(integration_mode=IntegrationMode.PYTEST, needs_inference=False)
- kernel = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="kernel-boot-contract")
+ kernel = PipelexKernel.make(storage_scope="test/scope", read_scope=None, run_mode=PipeRunMode.DRY, user_id="kernel-boot-contract")
model = LLMSetting(model="kernel-boot-contract-model", temperature=0.5)
text_concept = ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT)
@@ -188,6 +188,7 @@ _KERNEL_CALL_SCRIPT = textwrap.dedent(
inputs={"topic": SHAPED_TOPIC},
concept_provider=NativeOnlyConceptProvider(),
input_specs=InputStuffSpecs(root={"topic": StuffSpec(concept=text_concept)}),
+ read_scope=None,
)
if extract_named_content(memory=shaped_memory, name="topic", content_type=TextContent).text != SHAPED_TOPIC:
fail("shape_inputs did not land the provided value under its declared name")
diff --git a/tests/unit/pipelex/kernel/test_kernel_log_context.py b/tests/unit/pipelex/kernel/test_kernel_log_context.py
index 5e8e50614..0fd71d81c 100644
--- a/tests/unit/pipelex/kernel/test_kernel_log_context.py
+++ b/tests/unit/pipelex/kernel/test_kernel_log_context.py
@@ -95,7 +95,9 @@ def _install_probe(mocker: MockerFixture, *, ops_module: str) -> ContentGenerato
def _kernel_and_step() -> tuple[PipelexKernel, JobMetadata]:
- kernel = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="kernel-log-context", request_id=REQUEST_ID)
+ kernel = PipelexKernel.make(
+ storage_scope="test/scope", read_scope=None, run_mode=PipeRunMode.DRY, user_id="kernel-log-context", request_id=REQUEST_ID
+ )
return kernel, kernel.make_step_metadata()
diff --git a/tests/unit/pipelex/kernel/test_llm_object_kernel.py b/tests/unit/pipelex/kernel/test_llm_object_kernel.py
index 40cb5d18e..804dc96a0 100644
--- a/tests/unit/pipelex/kernel/test_llm_object_kernel.py
+++ b/tests/unit/pipelex/kernel/test_llm_object_kernel.py
@@ -45,7 +45,7 @@ RESULT_NAME = "character"
def _make_dry_kernel() -> PipelexKernel:
- return PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="kernel-unit-test")
+ return PipelexKernel.make(storage_scope="test/scope", read_scope=None, run_mode=PipeRunMode.DRY, user_id="kernel-unit-test")
class TestLlmObjectKernel:
diff --git a/tests/unit/pipelex/kernel/test_llm_text_kernel.py b/tests/unit/pipelex/kernel/test_llm_text_kernel.py
index 14a9d1fd3..75a1249b2 100644
--- a/tests/unit/pipelex/kernel/test_llm_text_kernel.py
+++ b/tests/unit/pipelex/kernel/test_llm_text_kernel.py
@@ -40,7 +40,7 @@ class MeetingSummary(TextContent):
def _make_dry_kernel() -> PipelexKernel:
- return PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="kernel-unit-test")
+ return PipelexKernel.make(storage_scope="test/scope", read_scope=None, run_mode=PipeRunMode.DRY, user_id="kernel-unit-test")
class TestLlmTextKernel:
diff --git a/tests/unit/pipelex/kernel/test_pipelex_kernel_run_extras.py b/tests/unit/pipelex/kernel/test_pipelex_kernel_run_extras.py
index 8983d3247..439dfe662 100644
--- a/tests/unit/pipelex/kernel/test_pipelex_kernel_run_extras.py
+++ b/tests/unit/pipelex/kernel/test_pipelex_kernel_run_extras.py
@@ -18,6 +18,7 @@ class TestPipelexKernelRunExtras:
def test_supplied_extras_land_on_the_runs_metadata(self) -> None:
kernel = PipelexKernel.make(
storage_scope="test/scope",
+ read_scope=None,
run_mode=PipeRunMode.DRY,
user_id="test-user",
extras={"organization": "org_acme"},
@@ -27,14 +28,14 @@ class TestPipelexKernelRunExtras:
def test_omitting_them_leaves_an_empty_mapping(self) -> None:
"""Every existing caller omits the field; an omission is an empty facet, not an error."""
- kernel = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user")
+ kernel = PipelexKernel.make(storage_scope="test/scope", read_scope=None, run_mode=PipeRunMode.DRY, user_id="test-user")
assert kernel.job_metadata.run_metadata.extras == {}
def test_two_kernels_do_not_share_one_default_mapping(self) -> None:
"""A mutable default shared between runs would leak one host's extras into another's."""
- first = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user")
- second = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user")
+ first = PipelexKernel.make(storage_scope="test/scope", read_scope=None, run_mode=PipeRunMode.DRY, user_id="test-user")
+ second = PipelexKernel.make(storage_scope="test/scope", read_scope=None, run_mode=PipeRunMode.DRY, user_id="test-user")
first.job_metadata.run_metadata.extras["organization"] = "org_acme"
@@ -45,6 +46,7 @@ class TestPipelexKernelRunExtras:
with pytest.raises(ValueError, match="extras"):
PipelexKernel.make(
storage_scope="test/scope",
+ read_scope=None,
run_mode=PipeRunMode.DRY,
user_id="test-user",
extras={"Organization": "org_acme"},
diff --git a/tests/unit/pipelex/kernel/test_pipelex_kernel_run_state.py b/tests/unit/pipelex/kernel/test_pipelex_kernel_run_state.py
index 5e403c84d..27cd2e33b 100644
--- a/tests/unit/pipelex/kernel/test_pipelex_kernel_run_state.py
+++ b/tests/unit/pipelex/kernel/test_pipelex_kernel_run_state.py
@@ -62,22 +62,26 @@ class TestPipelexKernelRunState:
"""
trace_context = _trace_context()
- kernel = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user", trace_context=trace_context)
+ kernel = PipelexKernel.make(
+ storage_scope="test/scope", read_scope=None, run_mode=PipeRunMode.DRY, user_id="test-user", trace_context=trace_context
+ )
assert kernel.job_metadata.run_metadata.pipeline_run_id == _GRAPH_ID
assert kernel.job_metadata.trace_context == trace_context
def test_without_a_trace_context_the_run_mints_its_own_id_and_traces_nothing(self) -> None:
"""The default stays what it was: a fresh run id, and no context for the leaf to emit against."""
- first = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user")
- second = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user")
+ first = PipelexKernel.make(storage_scope="test/scope", read_scope=None, run_mode=PipeRunMode.DRY, user_id="test-user")
+ second = PipelexKernel.make(storage_scope="test/scope", read_scope=None, run_mode=PipeRunMode.DRY, user_id="test-user")
assert first.job_metadata.trace_context is None
assert first.job_metadata.run_metadata.pipeline_run_id != second.job_metadata.run_metadata.pipeline_run_id
def test_a_request_id_rides_the_run_metadata(self) -> None:
"""The field a hosted deployment filters its logs on, carried like `user_id` and `storage_scope`."""
- kernel = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user", request_id="req-kernel")
+ kernel = PipelexKernel.make(
+ storage_scope="test/scope", read_scope=None, run_mode=PipeRunMode.DRY, user_id="test-user", request_id="req-kernel"
+ )
assert kernel.job_metadata.run_metadata.request_id == "req-kernel"
assert kernel.make_step_metadata().run_metadata.request_id == "req-kernel"
@@ -85,11 +89,13 @@ class TestPipelexKernelRunState:
def test_a_malformed_request_id_is_refused_by_the_run_metadata(self) -> None:
"""The validation is `RunMetadata`'s own, so `make` refuses exactly what the pipeline entry points refuse."""
with pytest.raises(ValidationError):
- PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user", request_id="forged\nline")
+ PipelexKernel.make(storage_scope="test/scope", read_scope=None, run_mode=PipeRunMode.DRY, user_id="test-user", request_id="forged\nline")
def test_a_supplied_pipeline_run_id_becomes_the_runs_identity(self) -> None:
"""A host inside a replay-based executor mints the id from its own replay-safe source and hands it over."""
- kernel = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user", pipeline_run_id="plr-host")
+ kernel = PipelexKernel.make(
+ storage_scope="test/scope", read_scope=None, run_mode=PipeRunMode.DRY, user_id="test-user", pipeline_run_id="plr-host"
+ )
assert kernel.job_metadata.run_metadata.pipeline_run_id == "plr-host"
assert kernel.job_metadata.trace_context is None
@@ -97,6 +103,7 @@ class TestPipelexKernelRunState:
def test_a_trace_context_and_an_equal_pipeline_run_id_are_one_identity(self) -> None:
kernel = PipelexKernel.make(
storage_scope="test/scope",
+ read_scope=None,
run_mode=PipeRunMode.DRY,
user_id="test-user",
trace_context=_trace_context(),
@@ -110,6 +117,7 @@ class TestPipelexKernelRunState:
with pytest.raises(ValueError, match="plr-elsewhere"):
PipelexKernel.make(
storage_scope="test/scope",
+ read_scope=None,
run_mode=PipeRunMode.DRY,
user_id="test-user",
trace_context=_trace_context(),
@@ -119,10 +127,10 @@ class TestPipelexKernelRunState:
def test_an_empty_pipeline_run_id_is_refused(self) -> None:
"""An empty id is a caller's bug, and minting one in its place would hide it from a replaying host."""
with pytest.raises(ValueError, match="empty pipeline_run_id"):
- PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user", pipeline_run_id="")
+ PipelexKernel.make(storage_scope="test/scope", read_scope=None, run_mode=PipeRunMode.DRY, user_id="test-user", pipeline_run_id="")
def test_with_neither_the_run_id_is_a_fresh_uuid4(self) -> None:
- kernel = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user")
+ kernel = PipelexKernel.make(storage_scope="test/scope", read_scope=None, run_mode=PipeRunMode.DRY, user_id="test-user")
assert UUID(kernel.job_metadata.run_metadata.pipeline_run_id).version == 4
@@ -130,6 +138,7 @@ class TestPipelexKernelRunState:
"""The host's binding: the run's two identifiers, and no `pipe_run_id`, which each step binds for itself."""
kernel = PipelexKernel.make(
storage_scope="test/scope",
+ read_scope=None,
run_mode=PipeRunMode.DRY,
user_id="test-user",
request_id="req-kernel",
@@ -157,6 +166,7 @@ class TestPipelexKernelRunState:
step_ids = iter(["step-0"])
kernel = PipelexKernel.make(
storage_scope="test/scope",
+ read_scope=None,
run_mode=PipeRunMode.DRY,
user_id="test-user",
request_id="req-kernel",
@@ -177,7 +187,9 @@ class TestPipelexKernelRunState:
def test_each_step_inherits_the_trace_context_and_carries_its_own_run_id(self) -> None:
"""The per-step copy: same trace context (so every step attributes to it), distinct pipe_run_id."""
- kernel = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user", trace_context=_trace_context())
+ kernel = PipelexKernel.make(
+ storage_scope="test/scope", read_scope=None, run_mode=PipeRunMode.DRY, user_id="test-user", trace_context=_trace_context()
+ )
first_step = kernel.make_step_metadata()
second_step = kernel.make_step_metadata()
@@ -195,7 +207,7 @@ class TestPipelexKernelRunState:
Anything that attributes work by `pipe_code` — log correlation, usage accounting, the
per-step labelling a distributed backend derives — sees an anonymous step without this.
"""
- kernel = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user")
+ kernel = PipelexKernel.make(storage_scope="test/scope", read_scope=None, run_mode=PipeRunMode.DRY, user_id="test-user")
assert kernel.make_step_metadata(pipe_code="answer_question").pipe_code == "answer_question"
@@ -207,7 +219,7 @@ class TestPipelexKernelRunState:
key must be omitted, not passed as None — so this constructs a run that HAS a run-level
`pipe_code`, which no caller in the tree does today.
"""
- kernel = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user")
+ kernel = PipelexKernel.make(storage_scope="test/scope", read_scope=None, run_mode=PipeRunMode.DRY, user_id="test-user")
kernel.job_metadata.pipe_code = "set_at_run_level"
assert kernel.make_step_metadata().pipe_code == "set_at_run_level"
@@ -227,12 +239,14 @@ class TestPipelexKernelRunState:
minted.append(f"step-{len(minted)}")
return minted[-1]
- kernel = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user", step_id_source=_source)
+ kernel = PipelexKernel.make(
+ storage_scope="test/scope", read_scope=None, run_mode=PipeRunMode.DRY, user_id="test-user", step_id_source=_source
+ )
assert kernel.make_step_metadata().pipe_run_id == "step-0"
assert kernel.make_step_metadata().pipe_run_id == "step-1"
- default_kernel = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user")
+ default_kernel = PipelexKernel.make(storage_scope="test/scope", read_scope=None, run_mode=PipeRunMode.DRY, user_id="test-user")
first_default = default_kernel.make_step_metadata().pipe_run_id
assert first_default is not None
assert UUID(first_default).version == 4
@@ -240,7 +254,7 @@ class TestPipelexKernelRunState:
def test_mock_usage_rides_the_execution_mode_contract(self) -> None:
"""The DRY sub-flag lands on the carrier every cogt leaf reads off its assignment."""
- kernel = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user", is_mock_usage=True)
+ kernel = PipelexKernel.make(storage_scope="test/scope", read_scope=None, run_mode=PipeRunMode.DRY, user_id="test-user", is_mock_usage=True)
assert kernel.cogt_run_params.run_mode.is_dry
assert kernel.cogt_run_params.is_mock_usage
diff --git a/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_current_span.py b/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_current_span.py
index 6961337f4..00086e452 100644
--- a/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_current_span.py
+++ b/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_current_span.py
@@ -195,7 +195,7 @@ def _make_pipe(*, code: str, nested: CurrentSpanPipe | None = None, fails: bool
def _traced_metadata() -> JobMetadata:
"""What a submission builds when the runtime traces: the run's identifiers and its derived trace."""
return JobMetadata(
- run_metadata=RunMetadata(user_id="pytest", storage_scope="test/scope", pipeline_run_id="plr-span"),
+ run_metadata=RunMetadata(user_id="pytest", storage_scope="test/scope", read_scope=None, pipeline_run_id="plr-span"),
otel_context=RUN_OTEL_CONTEXT,
)
diff --git a/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_log_context.py b/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_log_context.py
index cb0081957..3f1915b31 100644
--- a/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_log_context.py
+++ b/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_log_context.py
@@ -147,7 +147,9 @@ def _live_params() -> PipeRunParams:
def _seam_shaped_metadata() -> JobMetadata:
"""What a submission builds: the run's identifiers, and no pipe_run_id yet."""
- return JobMetadata(run_metadata=RunMetadata(user_id="pytest", storage_scope="test/scope", pipeline_run_id="plr-live", request_id="req-live"))
+ return JobMetadata(
+ run_metadata=RunMetadata(user_id="pytest", storage_scope="test/scope", read_scope=None, pipeline_run_id="plr-live", request_id="req-live")
+ )
def _own_records(caplog: pytest.LogCaptureFixture) -> list[logging.LogRecord]:
diff --git a/tests/unit/pipelex/pipe_machinery/test_pipe_validate_before_run.py b/tests/unit/pipelex/pipe_machinery/test_pipe_validate_before_run.py
index 419a5cd29..b34a2484f 100644
--- a/tests/unit/pipelex/pipe_machinery/test_pipe_validate_before_run.py
+++ b/tests/unit/pipelex/pipe_machinery/test_pipe_validate_before_run.py
@@ -114,7 +114,7 @@ class TestPipeValidateBeforeRun:
# Provide only one input
working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(
- pipeline_inputs={"topic": "Python programming"}, concept_provider=get_concept_library()
+ pipeline_inputs={"topic": "Python programming"}, concept_provider=get_concept_library(), read_scope=None
)
pipe_run_params = PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.LIVE)
diff --git a/tests/unit/pipelex/pipe_machinery/test_pipe_validate_before_run_read_scope.py b/tests/unit/pipelex/pipe_machinery/test_pipe_validate_before_run_read_scope.py
new file mode 100644
index 000000000..e06037ae0
--- /dev/null
+++ b/tests/unit/pipelex/pipe_machinery/test_pipe_validate_before_run_read_scope.py
@@ -0,0 +1,80 @@
+"""On a run with a read scope, the input pre-check does not probe the host's disk for a local path.
+
+The pre-check's resource validation only asks whether a local path exists. A scoped run reads no local
+path at all, and its leaves refuse one before any IO, so probing first would stat the host's disk on
+the method's say-so, and a missing path failing there while an existing one is refused later would
+tell the caller which files exist. An unscoped live run still reports a missing file there.
+"""
+
+from typing import Callable
+
+import pytest
+
+from pipelex.core.concepts.concept_factory import ConceptFactory
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.core.pipes.inputs.exceptions import PipeRunInputsError
+from pipelex.core.stuffs.image_content import ImageContent
+from pipelex.core.stuffs.stuff_factory import StuffFactory
+from pipelex.pipe_machinery.pipe_factory import PipeFactory
+from pipelex.pipe_operators.llm.pipe_llm import PipeLLM
+from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint
+from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
+from pipelex.system.pipe_run_mode import PipeRunMode
+
+MISSING_PATH = "/nonexistent/read-scope-probe/photo.png"
+
+
+def _pipe_llm() -> PipeLLM:
+ return PipeFactory[PipeLLM].make_from_blueprint(
+ domain_code="test_domain",
+ pipe_code="describe_photo",
+ blueprint=PipeLLMBlueprint(
+ description="Describe a photo",
+ inputs={"photo": "native.Image"},
+ output="native.Text",
+ prompt="Describe $photo",
+ ),
+ )
+
+
+def _job_metadata(*, read_scope: str | None) -> JobMetadata:
+ storage_scope = "org_abc/mt_1/run_1" if read_scope is not None else "run_1"
+ return JobMetadata(run_metadata=RunMetadata(user_id="u", pipeline_run_id="run_1", storage_scope=storage_scope, read_scope=read_scope))
+
+
[email protected](loop_scope="class")
+class TestValidateBeforeRunReadScope:
+ async def test_a_scoped_live_run_does_not_probe_a_local_path(self, load_empty_library: Callable[[], None]) -> None:
+ load_empty_library()
+ working_memory = WorkingMemoryFactory.make_from_single_stuff(
+ StuffFactory.make_stuff(
+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.IMAGE),
+ content=ImageContent(url=MISSING_PATH),
+ name="photo",
+ )
+ )
+
+ await _pipe_llm().validate_before_run(
+ job_metadata=_job_metadata(read_scope="org_abc"),
+ working_memory=working_memory,
+ pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.LIVE),
+ )
+
+ async def test_an_unscoped_live_run_still_reports_a_missing_file(self, load_empty_library: Callable[[], None]) -> None:
+ load_empty_library()
+ working_memory = WorkingMemoryFactory.make_from_single_stuff(
+ StuffFactory.make_stuff(
+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.IMAGE),
+ content=ImageContent(url=MISSING_PATH),
+ name="photo",
+ )
+ )
+
+ with pytest.raises(PipeRunInputsError, match="invalid resource"):
+ await _pipe_llm().validate_before_run(
+ job_metadata=_job_metadata(read_scope=None),
+ working_memory=working_memory,
+ pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.LIVE),
+ )
diff --git a/tests/unit/pipelex/pipe_machinery/test_run_pipe_caller_scope.py b/tests/unit/pipelex/pipe_machinery/test_run_pipe_caller_scope.py
index b8ccc14bd..a1ba1008e 100644
--- a/tests/unit/pipelex/pipe_machinery/test_run_pipe_caller_scope.py
+++ b/tests/unit/pipelex/pipe_machinery/test_run_pipe_caller_scope.py
@@ -42,6 +42,7 @@ def _job_metadata() -> JobMetadata:
return JobMetadata(
run_metadata=RunMetadata(
storage_scope="tenant/run-1",
+ read_scope=None,
user_id="caller-7",
pipeline_run_id="run-1",
extras={"organization": "org_caller"},
diff --git a/tests/unit/pipelex/pipe_machinery/test_run_pipe_tracer_metadata.py b/tests/unit/pipelex/pipe_machinery/test_run_pipe_tracer_metadata.py
index 8f0d3a05d..532a074c0 100644
--- a/tests/unit/pipelex/pipe_machinery/test_run_pipe_tracer_metadata.py
+++ b/tests/unit/pipelex/pipe_machinery/test_run_pipe_tracer_metadata.py
@@ -66,7 +66,7 @@ class TestRunPipeForwardsTracerMetadata:
)
job_metadata = JobMetadata(
- run_metadata=RunMetadata(storage_scope="test/scope", user_id="pytest", pipeline_run_id="test-run-meta"),
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="pytest", pipeline_run_id="test-run-meta"),
trace_context=make_trace_context(graph_id="test-run-meta"),
)
diff --git a/tests/unit/pipelex/pipe_operators/pipe_func/test_direct_executor_workdir.py b/tests/unit/pipelex/pipe_operators/pipe_func/test_direct_executor_workdir.py
index 0930a5952..4e7cc1f6e 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_func/test_direct_executor_workdir.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_func/test_direct_executor_workdir.py
@@ -63,7 +63,7 @@ class TestDirectExecutorWorkdir:
working_memory_raw={},
pipe_code="missing_pipe",
function_name="missing_func",
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="user", pipeline_run_id="run")),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="user", pipeline_run_id="run")),
pipe_run_params=PipeRunParams(run_mode=PipeRunMode.DRY, pipe_stack_limit=10, batch_max_concurrency=None),
)
@@ -88,7 +88,7 @@ class TestDirectExecutorWorkdir:
working_memory_raw={},
pipe_code="missing_pipe",
function_name="missing_func",
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="user", pipeline_run_id="run")),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="user", pipeline_run_id="run")),
pipe_run_params=PipeRunParams(run_mode=PipeRunMode.DRY, pipe_stack_limit=10, batch_max_concurrency=None),
)
@@ -140,7 +140,7 @@ class TestDirectExecutorWorkdir:
# its lookup is strict, so a bare code would not resolve on the far side.
pipe_code="greet_demo.greet",
function_name="greet_it",
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="user", pipeline_run_id="run")),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="user", pipeline_run_id="run")),
pipe_run_params=PipeRunParams(run_mode=PipeRunMode.DRY, pipe_stack_limit=10, batch_max_concurrency=None),
)
diff --git a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_execution_transport.py b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_execution_transport.py
index 740fa340d..94e8be542 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_execution_transport.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_execution_transport.py
@@ -46,7 +46,9 @@ class TestPipeFuncExecutionTransport:
working_memory_raw={},
pipe_code="my_pipe",
function_name="my_func",
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="user", pipeline_run_id="run")),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="user", pipeline_run_id="run")
+ ),
pipe_run_params=PipeRunParams(run_mode=PipeRunMode.DRY, pipe_stack_limit=10, batch_max_concurrency=None),
timeout_seconds=float("inf"),
)
diff --git a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_hosted_mode.py b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_hosted_mode.py
index d613b6bad..cb4251e8f 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_hosted_mode.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_hosted_mode.py
@@ -88,7 +88,7 @@ class TestPipeFuncHostedMode:
)
pipe_output = await pipe_func._dry_run_operator_pipe( # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="user", pipeline_run_id="run")),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="user", pipeline_run_id="run")),
working_memory=WorkingMemoryFactory.make_empty(),
pipe_run_params=PipeRunParams(run_mode=PipeRunMode.DRY, pipe_stack_limit=10, batch_max_concurrency=None),
)
diff --git a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_transported_ref.py b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_transported_ref.py
index a81c85cdd..1329c00a5 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_transported_ref.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_transported_ref.py
@@ -56,7 +56,7 @@ class TestPipeFuncTransportedRef:
mocker.patch("pipelex.pipe_operators.func.pipe_func.get_pipe_func_executor", return_value=executor)
await the_pipe_func._live_run_operator_pipe( # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="user", pipeline_run_id="run")),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="user", pipeline_run_id="run")),
working_memory=WorkingMemoryFactory.make_empty(),
pipe_run_params=PipeRunParams(run_mode=PipeRunMode.LIVE, pipe_stack_limit=10, batch_max_concurrency=None),
)
diff --git a/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_log_context.py b/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_log_context.py
index d75e69524..080d87c7d 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_log_context.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_log_context.py
@@ -90,7 +90,9 @@ class TestPipeLLMLogContext:
spy = KernelCallSpy()
mocker.patch("pipelex.pipe_operators.llm.pipe_llm.run_llm_text", new=spy)
job_metadata = JobMetadata(
- run_metadata=RunMetadata(user_id="pytest", storage_scope="test/scope", pipeline_run_id=PIPELINE_RUN_ID, request_id=REQUEST_ID),
+ run_metadata=RunMetadata(
+ user_id="pytest", storage_scope="test/scope", read_scope=None, pipeline_run_id=PIPELINE_RUN_ID, request_id=REQUEST_ID
+ ),
)
expected = LogContext(request_id=REQUEST_ID, pipeline_run_id=PIPELINE_RUN_ID, pipe_run_id=MINTED_PIPE_RUN_ID)
diff --git a/tests/unit/pipelex/pipe_operators/pipe_llm/test_prompt_rendering_purity.py b/tests/unit/pipelex/pipe_operators/pipe_llm/test_prompt_rendering_purity.py
index d01de271f..15731cdd4 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_llm/test_prompt_rendering_purity.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_llm/test_prompt_rendering_purity.py
@@ -143,7 +143,9 @@ class TestPromptRenderingPurity:
await pipe_llm._dry_run_operator_pipe( # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
job_metadata=JobMetadata(
- run_metadata=RunMetadata(storage_scope="test/scope", user_id="pytest", pipeline_run_id="test_prompt_rendering_purity")
+ run_metadata=RunMetadata(
+ storage_scope="test/scope", read_scope=None, user_id="pytest", pipeline_run_id="test_prompt_rendering_purity"
+ )
),
working_memory=_make_working_memory(),
pipe_run_params=PipeRunParams(run_mode=PipeRunMode.DRY, pipe_stack_limit=10, batch_max_concurrency=None),
diff --git a/tests/unit/pipelex/pipe_run/test_delivery_executor.py b/tests/unit/pipelex/pipe_run/test_delivery_executor.py
index 529fbe812..f50890eeb 100644
--- a/tests/unit/pipelex/pipe_run/test_delivery_executor.py
+++ b/tests/unit/pipelex/pipe_run/test_delivery_executor.py
@@ -353,7 +353,9 @@ class TestDeliveryExecutor:
mock_output.tokens_usages = [
LLMTokensUsage(
model_type="llm",
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="plr-usage")),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="test-user", pipeline_run_id="plr-usage")
+ ),
inference_model_name="test-model",
inference_model_id="test-model-id",
nb_tokens_by_category={TokenCategory.INPUT: 15, TokenCategory.OUTPUT: 4},
diff --git a/tests/unit/pipelex/pipe_run/test_pipe_job_hydration.py b/tests/unit/pipelex/pipe_run/test_pipe_job_hydration.py
index fc919efef..4025f27fd 100644
--- a/tests/unit/pipelex/pipe_run/test_pipe_job_hydration.py
+++ b/tests/unit/pipelex/pipe_run/test_pipe_job_hydration.py
@@ -23,7 +23,9 @@ def _make_pipe_job(
working_memory=working_memory,
working_memory_raw=None,
pipe_run_params=PipeRunParamsFactory.make_run_params(),
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run")),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="test-user", pipeline_run_id="test-run")
+ ),
output_name=None,
library_crate=library_crate,
)
diff --git a/tests/unit/pipelex/pipe_run/test_pipe_router_run.py b/tests/unit/pipelex/pipe_run/test_pipe_router_run.py
index 0fe4ed330..9111f2db3 100644
--- a/tests/unit/pipelex/pipe_run/test_pipe_router_run.py
+++ b/tests/unit/pipelex/pipe_run/test_pipe_router_run.py
@@ -95,7 +95,9 @@ def _make_pipe_job() -> PipeJob:
working_memory=None,
working_memory_raw=None,
pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.LIVE),
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run")),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="test-user", pipeline_run_id="test-run")
+ ),
output_name=None,
library_crate=None,
)
diff --git a/tests/unit/pipelex/pipe_run/test_pipe_run.py b/tests/unit/pipelex/pipe_run/test_pipe_run.py
index 89a2b8c7f..defe6234c 100644
--- a/tests/unit/pipelex/pipe_run/test_pipe_run.py
+++ b/tests/unit/pipelex/pipe_run/test_pipe_run.py
@@ -25,7 +25,9 @@ class TestPipeRun:
# A real JobMetadata, not a MagicMock attribute: `PipeRun.run` copies it
# onto the output, and `assemble_tracing_on_output` then feeds its
# `run_metadata` into `TracingAssembly`, which is a typed field.
- mock_job.job_metadata = JobMetadata(run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-123", storage_scope="test/scope"))
+ mock_job.job_metadata = JobMetadata(
+ run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-123", storage_scope="test/scope", read_scope=None)
+ )
pipe_run = PipeRun(pipe_router=mock_router)
assignment = DeliveryAssignment(storage=StorageTarget())
@@ -62,6 +64,7 @@ class TestPipeRun:
user_id="pytest",
pipeline_run_id="plr-req",
storage_scope="test/scope",
+ read_scope=None,
request_id="req-direct-mode",
)
)
@@ -91,7 +94,9 @@ class TestPipeRun:
# A real JobMetadata, not a MagicMock attribute: `PipeRun.run` copies it
# onto the output, and `assemble_tracing_on_output` then feeds its
# `run_metadata` into `TracingAssembly`, which is a typed field.
- mock_job.job_metadata = JobMetadata(run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-no-delivery", storage_scope="test/scope"))
+ mock_job.job_metadata = JobMetadata(
+ run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-no-delivery", storage_scope="test/scope", read_scope=None)
+ )
pipe_run = PipeRun(pipe_router=mock_router)
@@ -123,7 +128,9 @@ class TestPipeRun:
# A real JobMetadata, not a MagicMock attribute: `PipeRun.run` copies it
# onto the output, and `assemble_tracing_on_output` then feeds its
# `run_metadata` into `TracingAssembly`, which is a typed field.
- mock_job.job_metadata = JobMetadata(run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-fail", storage_scope="test/scope"))
+ mock_job.job_metadata = JobMetadata(
+ run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-fail", storage_scope="test/scope", read_scope=None)
+ )
pipe_run = PipeRun(pipe_router=mock_router)
assignment = DeliveryAssignment(storage=StorageTarget())
@@ -157,7 +164,9 @@ class TestPipeRun:
# A real JobMetadata, not a MagicMock attribute: `PipeRun.run` copies it
# onto the output, and `assemble_tracing_on_output` then feeds its
# `run_metadata` into `TracingAssembly`, which is a typed field.
- mock_job.job_metadata = JobMetadata(run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-bare", storage_scope="test/scope"))
+ mock_job.job_metadata = JobMetadata(
+ run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-bare", storage_scope="test/scope", read_scope=None)
+ )
pipe_run = PipeRun(pipe_router=mock_router)
assignment = DeliveryAssignment(storage=StorageTarget())
@@ -202,7 +211,9 @@ class TestPipeRun:
# A real JobMetadata, not a MagicMock attribute: `PipeRun.run` copies it
# onto the output, and `assemble_tracing_on_output` then feeds its
# `run_metadata` into `TracingAssembly`, which is a typed field.
- mock_job.job_metadata = JobMetadata(run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-order", storage_scope="test/scope"))
+ mock_job.job_metadata = JobMetadata(
+ run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-order", storage_scope="test/scope", read_scope=None)
+ )
pipe_run = PipeRun(pipe_router=mock_router)
assignment = DeliveryAssignment(storage=StorageTarget())
@@ -238,7 +249,9 @@ class TestPipeRun:
# A real JobMetadata, not a MagicMock attribute: `PipeRun.run` copies it
# onto the output, and `assemble_tracing_on_output` then feeds its
# `run_metadata` into `TracingAssembly`, which is a typed field.
- mock_job.job_metadata = JobMetadata(run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-mask-tracer", storage_scope="test/scope"))
+ mock_job.job_metadata = JobMetadata(
+ run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-mask-tracer", storage_scope="test/scope", read_scope=None)
+ )
pipe_run = PipeRun(pipe_router=mock_router)
@@ -269,7 +282,9 @@ class TestPipeRun:
# A real JobMetadata, not a MagicMock attribute: `PipeRun.run` copies it
# onto the output, and `assemble_tracing_on_output` then feeds its
# `run_metadata` into `TracingAssembly`, which is a typed field.
- mock_job.job_metadata = JobMetadata(run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-tracer-only", storage_scope="test/scope"))
+ mock_job.job_metadata = JobMetadata(
+ run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-tracer-only", storage_scope="test/scope", read_scope=None)
+ )
pipe_run = PipeRun(pipe_router=mock_router)
@@ -306,7 +321,9 @@ class TestPipeRun:
# A real JobMetadata, not a MagicMock attribute: `PipeRun.run` copies it
# onto the output, and `assemble_tracing_on_output` then feeds its
# `run_metadata` into `TracingAssembly`, which is a typed field.
- mock_job.job_metadata = JobMetadata(run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-triple-fail", storage_scope="test/scope"))
+ mock_job.job_metadata = JobMetadata(
+ run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-triple-fail", storage_scope="test/scope", read_scope=None)
+ )
pipe_run = PipeRun(pipe_router=mock_router)
assignment = DeliveryAssignment(storage=StorageTarget())
diff --git a/tests/unit/pipelex/pipe_run/test_pipe_run_log_context.py b/tests/unit/pipelex/pipe_run/test_pipe_run_log_context.py
index 8615c4993..944418932 100644
--- a/tests/unit/pipelex/pipe_run/test_pipe_run_log_context.py
+++ b/tests/unit/pipelex/pipe_run/test_pipe_run_log_context.py
@@ -45,7 +45,7 @@ class TestPipeRunLogContext:
mock_router.run = mocker.AsyncMock(side_effect=observe)
mock_job = mocker.MagicMock()
mock_job.job_metadata = JobMetadata(
- run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-ctx", storage_scope="test/scope", request_id="req-ctx"),
+ run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-ctx", storage_scope="test/scope", read_scope=None, request_id="req-ctx"),
pipe_run_id="pr-ctx",
)
@@ -72,7 +72,9 @@ class TestPipeRunLogContext:
mock_router = mocker.AsyncMock()
mock_router.run = mocker.AsyncMock(side_effect=observe)
mock_job = mocker.MagicMock()
- mock_job.job_metadata = JobMetadata(run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-only", storage_scope="test/scope"))
+ mock_job.job_metadata = JobMetadata(
+ run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-only", storage_scope="test/scope", read_scope=None)
+ )
await PipeRun(pipe_router=mock_router).run(pipe_job=mock_job)
@@ -83,7 +85,9 @@ class TestPipeRunLogContext:
mock_router = mocker.AsyncMock()
mock_router.run = mocker.AsyncMock(side_effect=RuntimeError("router blew up"))
mock_job = mocker.MagicMock()
- mock_job.job_metadata = JobMetadata(run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-fail", storage_scope="test/scope"))
+ mock_job.job_metadata = JobMetadata(
+ run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-fail", storage_scope="test/scope", read_scope=None)
+ )
with pytest.raises(RuntimeError):
await PipeRun(pipe_router=mock_router).run(pipe_job=mock_job)
diff --git a/tests/unit/pipelex/pipe_run/test_run_batch_branch_hook.py b/tests/unit/pipelex/pipe_run/test_run_batch_branch_hook.py
index d3f091e42..c944e011e 100644
--- a/tests/unit/pipelex/pipe_run/test_run_batch_branch_hook.py
+++ b/tests/unit/pipelex/pipe_run/test_run_batch_branch_hook.py
@@ -52,7 +52,9 @@ def _make_pipe_job() -> PipeJob:
working_memory=None,
working_memory_raw=None,
pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.LIVE),
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run")),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="test-user", pipeline_run_id="test-run")
+ ),
output_name=None,
library_crate=None,
)
diff --git a/tests/unit/pipelex/pipe_run/test_tracing_assembly.py b/tests/unit/pipelex/pipe_run/test_tracing_assembly.py
index dcbf00b3f..4c682e7e5 100644
--- a/tests/unit/pipelex/pipe_run/test_tracing_assembly.py
+++ b/tests/unit/pipelex/pipe_run/test_tracing_assembly.py
@@ -23,7 +23,9 @@ _MODULE = "pipelex.pipe_run.tracing_assembly"
def _make_usage_event(pipeline_run_id: str, node_id: str) -> UsageReportEvent:
"""A UsageReportEvent carrying a real LLMTokensUsage (so the aggregator returns it)."""
- metadata = JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="tracing-assembly-test", pipeline_run_id=pipeline_run_id))
+ metadata = JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="tracing-assembly-test", pipeline_run_id=pipeline_run_id)
+ )
tokens_usage = LLMTokensUsage(
job_metadata=metadata,
inference_model_name="fake_model",
diff --git a/tests/unit/pipelex/pipeline/test_input_normalizer.py b/tests/unit/pipelex/pipeline/test_input_normalizer.py
index 781388c89..f36697b6a 100644
--- a/tests/unit/pipelex/pipeline/test_input_normalizer.py
+++ b/tests/unit/pipelex/pipeline/test_input_normalizer.py
@@ -56,7 +56,7 @@ class TestInputNormalizerUrlGuards:
_patch_storage_and_config(mocker)
with pytest.raises(PipelineInputUrlMissingError, match="blank url"):
- await normalize_data_urls_to_storage(_memory_with_document(blank_url), storage_scope="test/scope")
+ await normalize_data_urls_to_storage(_memory_with_document(blank_url), storage_scope="test/scope", read_scope=None)
@pytest.mark.parametrize("bad_url", ["https://", "https://exa mple.com/file.pdf"])
async def test_malformed_http_url_raises_caller_facing_input_error(self, mocker: MockerFixture, bad_url: str) -> None:
@@ -64,7 +64,7 @@ class TestInputNormalizerUrlGuards:
_patch_storage_and_config(mocker)
with pytest.raises(PipelineInputUrlInvalidError) as exc_info:
- await normalize_data_urls_to_storage(_memory_with_document(bad_url), storage_scope="test/scope")
+ await normalize_data_urls_to_storage(_memory_with_document(bad_url), storage_scope="test/scope", read_scope=None)
strict = exc_info.value.to_error_report().to_dict(disclosure_mode=DisclosureMode.STRICT)
assert "not a valid http(s) URL" in strict["message"]
@@ -74,7 +74,9 @@ class TestInputNormalizerUrlGuards:
_patch_storage_and_config(mocker)
mock_get = mocker.patch("httpx.AsyncClient.get")
- memory = await normalize_data_urls_to_storage(_memory_with_document("https://example.com/file.pdf"), storage_scope="test/scope")
+ memory = await normalize_data_urls_to_storage(
+ _memory_with_document("https://example.com/file.pdf"), storage_scope="test/scope", read_scope=None
+ )
content = memory.get_stuff("document").content
assert isinstance(content, DocumentContent)
@@ -91,7 +93,7 @@ class TestInputNormalizerUrlGuards:
_patch_storage_and_config(mocker)
with pytest.raises(PipelineInputUrlMissingError) as exc_info:
- await normalize_data_urls_to_storage(_memory_with_document(""), storage_scope="test/scope")
+ await normalize_data_urls_to_storage(_memory_with_document(""), storage_scope="test/scope", read_scope=None)
strict = exc_info.value.to_error_report().to_dict(disclosure_mode=DisclosureMode.STRICT)
assert "blank url" in strict["message"]
@@ -109,7 +111,7 @@ class TestInputNormalizerUrlGuards:
missing_path = tmp_path / "nope.pdf"
with pytest.raises(PipelineInputContentError) as exc_info:
- await normalize_data_urls_to_storage(_memory_with_document(str(missing_path)), storage_scope="test/scope")
+ await normalize_data_urls_to_storage(_memory_with_document(str(missing_path)), storage_scope="test/scope", read_scope=None)
report = exc_info.value.to_error_report()
assert str(missing_path) in report.message, "precondition: the raw message carries the path"
@@ -125,17 +127,17 @@ class TestInputNormalizerUrlGuards:
_patch_storage_and_config(mocker)
with pytest.raises(PipelineInputContentError, match="cannot be read"):
- await normalize_data_urls_to_storage(_memory_with_document(str(tmp_path)), storage_scope="test/scope")
+ await normalize_data_urls_to_storage(_memory_with_document(str(tmp_path)), storage_scope="test/scope", read_scope=None)
async def test_path_with_null_byte_raises_input_error(self, mocker: MockerFixture, tmp_path: Path) -> None:
"""A NUL byte makes the path unusable: the read raises ValueError, not OSError, and it must still be an input error."""
_patch_storage_and_config(mocker)
with pytest.raises(PipelineInputContentError, match="cannot be read"):
- await normalize_data_urls_to_storage(_memory_with_document(f"{tmp_path}/a\x00b.pdf"), storage_scope="test/scope")
+ await normalize_data_urls_to_storage(_memory_with_document(f"{tmp_path}/a\x00b.pdf"), storage_scope="test/scope", read_scope=None)
async def test_missing_file_raises_input_error(self, mocker: MockerFixture, tmp_path: Path) -> None:
_patch_storage_and_config(mocker)
with pytest.raises(PipelineInputContentError, match="cannot be read"):
- await normalize_data_urls_to_storage(_memory_with_document(str(tmp_path / "nope.pdf")), storage_scope="test/scope")
+ await normalize_data_urls_to_storage(_memory_with_document(str(tmp_path / "nope.pdf")), storage_scope="test/scope", read_scope=None)
diff --git a/tests/unit/pipelex/pipeline/test_input_normalizer_public_url.py b/tests/unit/pipelex/pipeline/test_input_normalizer_public_url.py
index 3e447e577..7cb531112 100644
--- a/tests/unit/pipelex/pipeline/test_input_normalizer_public_url.py
+++ b/tests/unit/pipelex/pipeline/test_input_normalizer_public_url.py
@@ -35,7 +35,7 @@ def _memory_with_content(content: NormalizableContent) -> WorkingMemory:
async def _normalized_content(content: NormalizableContent) -> Any:
- memory = await normalize_data_urls_to_storage(_memory_with_content(content), storage_scope="test/scope")
+ memory = await normalize_data_urls_to_storage(_memory_with_content(content), storage_scope="test/scope", read_scope=None)
return memory.get_stuff("visual").content
diff --git a/tests/unit/pipelex/pipeline/test_input_normalizer_read_scope.py b/tests/unit/pipelex/pipeline/test_input_normalizer_read_scope.py
new file mode 100644
index 000000000..a1e119bc6
--- /dev/null
+++ b/tests/unit/pipelex/pipeline/test_input_normalizer_read_scope.py
@@ -0,0 +1,129 @@
+"""The input normalization authorizes every url against the run's read scope before reading or linking it.
+
+On a scoped run a local path is refused, naming the input, instead of being uploaded or kept for the
+prompt preparation to read later, and that holds whether local uploads are enabled or not. A storage
+reference outside the scope is refused instead of being signed, since a signed link is a read by
+whoever holds it. An unscoped run normalizes as it always did.
+"""
+
+from pathlib import Path
+from typing import Any
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.core.concepts.concept_factory import ConceptFactory
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.core.stuffs.image_content import ImageContent
+from pipelex.core.stuffs.list_content import ListContent
+from pipelex.core.stuffs.stuff_factory import StuffFactory
+from pipelex.pipeline.input_normalizer import normalize_data_urls_to_storage
+from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract
+from pipelex.tools.uri.exceptions import UriReadRefusalReason, UriReadRefusedError
+
+READ_SCOPE = "org_abc"
+STORAGE_SCOPE = "org_abc/mt_1/run_1"
+
+
+def _memory_with_image(url: str) -> WorkingMemory:
+ stuff = StuffFactory.make_stuff(
+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.IMAGE),
+ content=ImageContent(url=url),
+ name="photo",
+ )
+ return WorkingMemoryFactory.make_from_single_stuff(stuff)
+
+
+def _memory_with_image_list(urls: list[str]) -> WorkingMemory:
+ stuff = StuffFactory.make_stuff(
+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.IMAGE),
+ content=ListContent[ImageContent](items=[ImageContent(url=url) for url in urls]),
+ name="album",
+ )
+ return WorkingMemoryFactory.make_from_single_stuff(stuff)
+
+
+def _patch_storage_and_config(mocker: MockerFixture, *, is_upload_local_content_enabled: bool) -> Any:
+ storage = mocker.Mock(spec=StorageProviderAbstract)
+ storage.public_url = mocker.AsyncMock(return_value="https://signed.example/link")
+ storage.store = mocker.AsyncMock(return_value="pipelex-storage://org_abc/mt_1/run_1/assets/x.png")
+ mocker.patch("pipelex.pipeline.input_normalizer.get_storage_provider", return_value=storage)
+ mocker.patch(
+ "pipelex.pipeline.input_normalizer.get_config",
+ return_value=mocker.Mock(runtime=mocker.Mock(storage=mocker.Mock(is_upload_local_content_enabled=is_upload_local_content_enabled))),
+ )
+ return storage
+
+
[email protected](loop_scope="class")
+class TestInputNormalizerReadScope:
+ @pytest.mark.parametrize("is_upload_local_content_enabled", [True, False])
+ async def test_a_scoped_run_refuses_a_local_path_naming_the_input(
+ self, mocker: MockerFixture, tmp_path: Path, is_upload_local_content_enabled: bool
+ ) -> None:
+ storage = _patch_storage_and_config(mocker, is_upload_local_content_enabled=is_upload_local_content_enabled)
+ local_file = tmp_path / "photo.png"
+ local_file.write_bytes(b"\x89PNG\r\n\x1a\n")
+
+ with pytest.raises(UriReadRefusedError) as exc_info:
+ await normalize_data_urls_to_storage(_memory_with_image(str(local_file)), storage_scope=STORAGE_SCOPE, read_scope=READ_SCOPE)
+
+ assert exc_info.value.reason == UriReadRefusalReason.LOCAL_PATH
+ assert "input 'photo'" in str(exc_info.value)
+ assert str(local_file) not in str(exc_info.value)
+ storage.store.assert_not_called()
+
+ async def test_a_scoped_run_refuses_a_file_uri(self, mocker: MockerFixture) -> None:
+ _patch_storage_and_config(mocker, is_upload_local_content_enabled=True)
+ with pytest.raises(UriReadRefusedError):
+ await normalize_data_urls_to_storage(_memory_with_image("file:///etc/passwd"), storage_scope=STORAGE_SCOPE, read_scope=READ_SCOPE)
+
+ async def test_a_scoped_run_refuses_to_sign_a_foreign_storage_reference(self, mocker: MockerFixture) -> None:
+ storage = _patch_storage_and_config(mocker, is_upload_local_content_enabled=True)
+
+ with pytest.raises(UriReadRefusedError) as exc_info:
+ await normalize_data_urls_to_storage(
+ _memory_with_image("pipelex-storage://org_other/assets/secret.png"), storage_scope=STORAGE_SCOPE, read_scope=READ_SCOPE
+ )
+
+ assert exc_info.value.reason == UriReadRefusalReason.FOREIGN_STORAGE_KEY
+ storage.public_url.assert_not_called()
+
+ async def test_a_foreign_reference_inside_a_list_is_refused_too(self, mocker: MockerFixture) -> None:
+ _patch_storage_and_config(mocker, is_upload_local_content_enabled=True)
+ with pytest.raises(UriReadRefusedError) as exc_info:
+ await normalize_data_urls_to_storage(
+ _memory_with_image_list(["pipelex-storage://org_abc/assets/ok.png", "pipelex-storage://org_other/assets/secret.png"]),
+ storage_scope=STORAGE_SCOPE,
+ read_scope=READ_SCOPE,
+ )
+ assert "input 'album'" in str(exc_info.value)
+
+ async def test_a_scoped_run_signs_an_in_scope_reference(self, mocker: MockerFixture) -> None:
+ storage = _patch_storage_and_config(mocker, is_upload_local_content_enabled=True)
+
+ working_memory = await normalize_data_urls_to_storage(
+ _memory_with_image("pipelex-storage://org_abc/assets/photo.png"), storage_scope=STORAGE_SCOPE, read_scope=READ_SCOPE
+ )
+
+ storage.public_url.assert_awaited_once_with(uri="pipelex-storage://org_abc/assets/photo.png")
+ image_content = working_memory.get_stuff_as_image(name="photo")
+ assert image_content.public_url == "https://signed.example/link"
+
+ async def test_a_scoped_run_keeps_a_data_url_and_an_https_url(self, mocker: MockerFixture) -> None:
+ _patch_storage_and_config(mocker, is_upload_local_content_enabled=True)
+ await normalize_data_urls_to_storage(
+ _memory_with_image("data:image/png;base64,iVBORw0KGgo="), storage_scope=STORAGE_SCOPE, read_scope=READ_SCOPE
+ )
+ await normalize_data_urls_to_storage(_memory_with_image("https://example.com/photo.png"), storage_scope=STORAGE_SCOPE, read_scope=READ_SCOPE)
+
+ async def test_an_unscoped_run_uploads_a_local_path_as_before(self, mocker: MockerFixture, tmp_path: Path) -> None:
+ storage = _patch_storage_and_config(mocker, is_upload_local_content_enabled=True)
+ local_file = tmp_path / "photo.png"
+ local_file.write_bytes(b"\x89PNG\r\n\x1a\n")
+
+ await normalize_data_urls_to_storage(_memory_with_image(str(local_file)), storage_scope="run_1", read_scope=None)
+
+ storage.store.assert_awaited_once()
diff --git a/tests/unit/pipelex/providers/openai/test_openai_completions_usage_capture.py b/tests/unit/pipelex/providers/openai/test_openai_completions_usage_capture.py
index a4ef6601a..cbb5ea978 100644
--- a/tests/unit/pipelex/providers/openai/test_openai_completions_usage_capture.py
+++ b/tests/unit/pipelex/providers/openai/test_openai_completions_usage_capture.py
@@ -32,7 +32,7 @@ if TYPE_CHECKING:
def _make_llm_job_with_empty_usage() -> LLMJob:
"""Build an LLMJob whose tokens_usage is seeded empty, mirroring ``llm_job_before_start``."""
- job_metadata = JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"))
+ job_metadata = JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="test-user", pipeline_run_id="test-run"))
tokens_usage = LLMTokensUsage(
job_metadata=job_metadata,
inference_model_name="test-model",
diff --git a/tests/unit/pipelex/providers/openai/test_openai_img_gen_worker_usage_capture.py b/tests/unit/pipelex/providers/openai/test_openai_img_gen_worker_usage_capture.py
index fa5cb9a5d..41df1783a 100644
--- a/tests/unit/pipelex/providers/openai/test_openai_img_gen_worker_usage_capture.py
+++ b/tests/unit/pipelex/providers/openai/test_openai_img_gen_worker_usage_capture.py
@@ -70,7 +70,9 @@ def _make_img_gen_job() -> ImgGenJob:
),
job_config=ImgGenJobConfig(is_sync_mode=False),
job_report=ImgGenJobReport(),
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run")),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="test-user", pipeline_run_id="test-run")
+ ),
)
diff --git a/tests/unit/pipelex/providers/openai/test_openai_responses_usage_capture.py b/tests/unit/pipelex/providers/openai/test_openai_responses_usage_capture.py
index 0fea7c8d0..0c8e2d7a9 100644
--- a/tests/unit/pipelex/providers/openai/test_openai_responses_usage_capture.py
+++ b/tests/unit/pipelex/providers/openai/test_openai_responses_usage_capture.py
@@ -30,7 +30,7 @@ if TYPE_CHECKING:
def _make_llm_job_with_empty_usage() -> LLMJob:
- job_metadata = JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"))
+ job_metadata = JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="test-user", pipeline_run_id="test-run"))
tokens_usage = LLMTokensUsage(
job_metadata=job_metadata,
inference_model_name="test-model",
diff --git a/tests/unit/pipelex/reporting/test_emit_runner_fallback.py b/tests/unit/pipelex/reporting/test_emit_runner_fallback.py
index 7128ed5b1..1d8aaf38a 100644
--- a/tests/unit/pipelex/reporting/test_emit_runner_fallback.py
+++ b/tests/unit/pipelex/reporting/test_emit_runner_fallback.py
@@ -53,7 +53,7 @@ def _make_llm_job(
) -> LLMJob:
now = datetime.now(UTC)
job_metadata = JobMetadata(
- run_metadata=RunMetadata(storage_scope="test/scope", user_id="test_user", pipeline_run_id=pipeline_run_id),
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="test_user", pipeline_run_id=pipeline_run_id),
trace_context=trace_context,
started_at=now,
completed_at=now + timedelta(seconds=1),
diff --git a/tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py b/tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py
index 0927cd724..4bc5bb018 100644
--- a/tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py
+++ b/tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py
@@ -78,7 +78,7 @@ def _trace_context(run_id: str, tracer_key: str) -> TraceContext:
def _job_metadata(run_id: str, tracer_key: str) -> JobMetadata:
now = datetime.now(UTC)
return JobMetadata(
- run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id=run_id),
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="test-user", pipeline_run_id=run_id),
trace_context=_trace_context(run_id, tracer_key),
started_at=now,
completed_at=now + timedelta(seconds=1),
diff --git a/tests/unit/pipelex/reporting/test_emit_usage_event_gating.py b/tests/unit/pipelex/reporting/test_emit_usage_event_gating.py
index 0e70fe2d9..294339a6d 100644
--- a/tests/unit/pipelex/reporting/test_emit_usage_event_gating.py
+++ b/tests/unit/pipelex/reporting/test_emit_usage_event_gating.py
@@ -38,7 +38,7 @@ DATA_INCLUSION_OFF = DataInclusionConfig(
def _make_llm_job(pipeline_run_id: str, trace_context: TraceContext | None) -> LLMJob:
now = datetime.now(UTC)
job_metadata = JobMetadata(
- run_metadata=RunMetadata(storage_scope="test/scope", user_id="test_user", pipeline_run_id=pipeline_run_id),
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="test_user", pipeline_run_id=pipeline_run_id),
trace_context=trace_context,
started_at=now,
completed_at=now + timedelta(seconds=1),
diff --git a/tests/unit/pipelex/reporting/test_reporting_event_emission.py b/tests/unit/pipelex/reporting/test_reporting_event_emission.py
index 255bbda3f..56fb4b76a 100644
--- a/tests/unit/pipelex/reporting/test_reporting_event_emission.py
+++ b/tests/unit/pipelex/reporting/test_reporting_event_emission.py
@@ -39,7 +39,7 @@ def _make_test_llm_job(
"""Create a minimal LLMJob for testing event emission."""
now = datetime.now(UTC)
job_metadata = JobMetadata(
- run_metadata=RunMetadata(storage_scope="test/scope", user_id="test_user", pipeline_run_id=pipeline_run_id),
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="test_user", pipeline_run_id=pipeline_run_id),
trace_context=trace_context,
started_at=now,
completed_at=now + timedelta(seconds=1),
diff --git a/tests/unit/pipelex/runtime_bridge/test_direct_orchestrator.py b/tests/unit/pipelex/runtime_bridge/test_direct_orchestrator.py
index 62f8bc7e0..b3ccb42ef 100644
--- a/tests/unit/pipelex/runtime_bridge/test_direct_orchestrator.py
+++ b/tests/unit/pipelex/runtime_bridge/test_direct_orchestrator.py
@@ -25,7 +25,9 @@ def _make_pipe_job() -> PipeJob:
working_memory=None,
working_memory_raw=None,
pipe_run_params=PipeRunParamsFactory.make_run_params(),
- job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run")),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="test-user", pipeline_run_id="test-run")
+ ),
output_name=None,
library_crate=None,
)
diff --git a/tests/unit/pipelex/runtime_bridge/test_input_models.py b/tests/unit/pipelex/runtime_bridge/test_input_models.py
index 1c43cfb1d..c29342bd5 100644
--- a/tests/unit/pipelex/runtime_bridge/test_input_models.py
+++ b/tests/unit/pipelex/runtime_bridge/test_input_models.py
@@ -7,7 +7,7 @@ from pipelex.runtime_bridge.payloads import PipelexPipeDispatchAck, PipelexPipeR
class TestInputOutputModels:
def test_input_defaults_match_design(self):
- payload = PipelexPipeRunInput(storage_scope="test/scope", user_id="test-user", pipe_code="some_pipe")
+ payload = PipelexPipeRunInput(storage_scope="test/scope", read_scope=None, user_id="test-user", pipe_code="some_pipe")
assert payload.pipe_code == "some_pipe"
assert payload.inputs == {}
assert payload.output_name is None
@@ -39,6 +39,7 @@ class TestInputOutputModels:
def test_input_round_trip_via_json(self):
original = PipelexPipeRunInput(
storage_scope="test/scope",
+ read_scope=None,
pipe_code="some_pipe",
inputs={"foo": "bar"},
orchestration_mode="temporal",
@@ -143,8 +144,8 @@ class TestTheWireRefusesAnUnusableScope:
)
def test_an_unsafe_scope_is_refused_at_construction(self, bad_scope: str):
with pytest.raises(ValidationError):
- PipelexPipeRunInput(storage_scope=bad_scope, user_id="test-user", pipe_code="some_pipe")
+ PipelexPipeRunInput(storage_scope=bad_scope, read_scope=None, user_id="test-user", pipe_code="some_pipe")
def test_a_usable_scope_still_passes(self):
- payload = PipelexPipeRunInput(storage_scope="org_a/mt_b/run_c", user_id="test-user", pipe_code="some_pipe")
+ payload = PipelexPipeRunInput(storage_scope="org_a/mt_b/run_c", read_scope=None, user_id="test-user", pipe_code="some_pipe")
assert payload.storage_scope == "org_a/mt_b/run_c"
diff --git a/tests/unit/pipelex/runtime_bridge/test_input_read_scope.py b/tests/unit/pipelex/runtime_bridge/test_input_read_scope.py
new file mode 100644
index 000000000..25ab59695
--- /dev/null
+++ b/tests/unit/pipelex/runtime_bridge/test_input_read_scope.py
@@ -0,0 +1,20 @@
+"""`read_scope` on the runtime bridge's run payload: required, path-safe, and above the storage scope, at the wire."""
+
+import pytest
+from pydantic import ValidationError
+
+from pipelex.runtime_bridge.payloads import PipelexPipeRunInput
+
+
+class TestBridgePayloadReadScope:
+ def test_omitting_it_is_an_error_at_the_wire(self) -> None:
+ with pytest.raises(ValidationError, match="read_scope"):
+ PipelexPipeRunInput.model_validate({"pipe_code": "p", "user_id": "u1", "storage_scope": "org_abc/run_1"})
+
+ def test_a_storage_scope_outside_the_read_scope_is_refused_at_the_wire(self) -> None:
+ with pytest.raises(ValidationError, match="does not lie under its read_scope"):
+ PipelexPipeRunInput(pipe_code="p", user_id="u1", storage_scope="org_other/run_1", read_scope="org_abc")
+
+ def test_an_unsafe_read_scope_is_refused_at_the_wire(self) -> None:
+ with pytest.raises(ValidationError, match="Invalid read_scope"):
+ PipelexPipeRunInput(pipe_code="p", user_id="u1", storage_scope="org_abc/run_1", read_scope="../org_abc")
diff --git a/tests/unit/pipelex/runtime_bridge/test_input_run_extras.py b/tests/unit/pipelex/runtime_bridge/test_input_run_extras.py
index e4b07a437..68a91d397 100644
--- a/tests/unit/pipelex/runtime_bridge/test_input_run_extras.py
+++ b/tests/unit/pipelex/runtime_bridge/test_input_run_extras.py
@@ -21,6 +21,7 @@ from pipelex.system.run_extras import RUN_EXTRAS_MAX_ENTRIES
def _payload(extras: dict[str, str]) -> PipelexPipeRunInput:
return PipelexPipeRunInput(
storage_scope="org_a/mt_b/run_c",
+ read_scope=None,
user_id="test-user",
pipe_code="some_pipe",
extras=extras,
@@ -34,7 +35,7 @@ class TestTheWireRefusesMalformedRunExtras:
`user_id` and `storage_scope` are required here because a missing one
used to be invented; a missing label invents nothing, so it defaults.
"""
- payload = PipelexPipeRunInput(storage_scope="org_a/mt_b/run_c", user_id="test-user", pipe_code="some_pipe")
+ payload = PipelexPipeRunInput(storage_scope="org_a/mt_b/run_c", read_scope=None, user_id="test-user", pipe_code="some_pipe")
assert payload.extras == {}
@pytest.mark.parametrize(
diff --git a/tests/unit/pipelex/system/telemetry/test_llm_span_carries_run_identity.py b/tests/unit/pipelex/system/telemetry/test_llm_span_carries_run_identity.py
index 0a8b8634d..46f88dcba 100644
--- a/tests/unit/pipelex/system/telemetry/test_llm_span_carries_run_identity.py
+++ b/tests/unit/pipelex/system/telemetry/test_llm_span_carries_run_identity.py
@@ -64,6 +64,7 @@ def _make_llm_job(*, extras: dict[str, str] | None = None) -> LLMJob:
user_id="user-42",
pipeline_run_id="run-1",
storage_scope="tenant/run-1",
+ read_scope=None,
extras=extras or {},
),
pipe_code="some_pipe",
diff --git a/tests/unit/pipelex/system/telemetry/test_observer_telemetry_identity.py b/tests/unit/pipelex/system/telemetry/test_observer_telemetry_identity.py
index 6b177e2ab..bf156705e 100644
--- a/tests/unit/pipelex/system/telemetry/test_observer_telemetry_identity.py
+++ b/tests/unit/pipelex/system/telemetry/test_observer_telemetry_identity.py
@@ -21,6 +21,7 @@ _RUN_METADATA = RunMetadata(
user_id="user-42",
pipeline_run_id="run-1",
storage_scope="tenant/run-1",
+ read_scope=None,
extras={"organization": "org_acme"},
)
diff --git a/tests/unit/pipelex/system/telemetry/test_telemetry_identity.py b/tests/unit/pipelex/system/telemetry/test_telemetry_identity.py
index 644d17f2f..4b9f8747e 100644
--- a/tests/unit/pipelex/system/telemetry/test_telemetry_identity.py
+++ b/tests/unit/pipelex/system/telemetry/test_telemetry_identity.py
@@ -35,6 +35,7 @@ def _run_metadata(*, user_id: str = "user-42", extras: dict[str, str] | None = N
user_id=user_id,
pipeline_run_id="run-1",
storage_scope="tenant/run-1",
+ read_scope=None,
extras=extras or {},
)
diff --git a/tests/unit/pipelex/system/telemetry/test_telemetry_manager_identity.py b/tests/unit/pipelex/system/telemetry/test_telemetry_manager_identity.py
index 8ab289b40..db1913f4a 100644
--- a/tests/unit/pipelex/system/telemetry/test_telemetry_manager_identity.py
+++ b/tests/unit/pipelex/system/telemetry/test_telemetry_manager_identity.py
@@ -32,6 +32,7 @@ def _run_metadata(*, user_id: str = "user-42", extras: dict[str, str] | None = N
user_id=user_id,
pipeline_run_id="run-1",
storage_scope="tenant/run-1",
+ read_scope=None,
extras=extras or {},
)
diff --git a/tests/unit/pipelex/system/test_caller_identity.py b/tests/unit/pipelex/system/test_caller_identity.py
index 57ef50b81..44054bc27 100644
--- a/tests/unit/pipelex/system/test_caller_identity.py
+++ b/tests/unit/pipelex/system/test_caller_identity.py
@@ -43,6 +43,7 @@ class TestCallerIdentity:
user_id="user-42",
pipeline_run_id="run-1",
storage_scope="tenant/run-1",
+ read_scope=None,
extras={"organization": "org_acme"},
)
diff --git a/tests/unit/pipelex/system/test_job_metadata_log_context.py b/tests/unit/pipelex/system/test_job_metadata_log_context.py
index cc051664d..25bb7f6fb 100644
--- a/tests/unit/pipelex/system/test_job_metadata_log_context.py
+++ b/tests/unit/pipelex/system/test_job_metadata_log_context.py
@@ -15,7 +15,7 @@ from pipelex.tools.log.log_context import LogContext, get_log_context
def _job_metadata(*, request_id: str | None = None, pipe_run_id: str | None = None) -> JobMetadata:
return JobMetadata(
- run_metadata=RunMetadata(user_id="pytest", storage_scope="test/scope", pipeline_run_id="plr-job", request_id=request_id),
+ run_metadata=RunMetadata(user_id="pytest", storage_scope="test/scope", read_scope=None, pipeline_run_id="plr-job", request_id=request_id),
pipe_run_id=pipe_run_id,
)
diff --git a/tests/unit/pipelex/system/test_job_metadata_request_id.py b/tests/unit/pipelex/system/test_job_metadata_request_id.py
index 72c199056..91b0c97ba 100644
--- a/tests/unit/pipelex/system/test_job_metadata_request_id.py
+++ b/tests/unit/pipelex/system/test_job_metadata_request_id.py
@@ -12,7 +12,7 @@ from pipelex.system.job_metadata import JobMetadata, RunMetadata
class TestJobMetadataRequestId:
def test_request_id_defaults_to_none(self) -> None:
"""A ``RunMetadata`` constructed without ``request_id`` carries ``None``."""
- meta = JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="r"))
+ meta = JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="r"))
assert meta.run_metadata.request_id is None
def test_request_id_round_trips_through_json(self) -> None:
@@ -21,14 +21,18 @@ class TestJobMetadataRequestId:
Temporal's data converter serializes activity args as JSON; this pins
the contract that ``request_id`` reaches the worker intact.
"""
- meta = JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="r", request_id="r-abc-123"))
+ meta = JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="r", request_id="r-abc-123")
+ )
recovered = JobMetadata.model_validate_json(meta.model_dump_json())
assert recovered.run_metadata.request_id == "r-abc-123"
assert recovered == meta
def test_copy_with_update_preserves_request_id(self) -> None:
"""``copy_with_update`` inherits ``request_id`` from the parent metadata."""
- parent = JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="r", request_id="r-1"))
+ parent = JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="r", request_id="r-1")
+ )
child = parent.copy_with_update(otel_context=None)
assert child.run_metadata.request_id == "r-1"
@@ -55,7 +59,9 @@ class TestJobMetadataRequestId:
``ErrorReport`` envelopes that quote it.
"""
with pytest.raises(ValidationError):
- JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="r", request_id=bad_request_id))
+ JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="r", request_id=bad_request_id)
+ )
@pytest.mark.parametrize(
"good_request_id",
@@ -68,5 +74,7 @@ class TestJobMetadataRequestId:
)
def test_request_id_accepts_well_formed_input(self, good_request_id: str) -> None:
"""Common ``X-Request-ID`` shapes (UUID, hyphenated, dot-separated) round-trip cleanly."""
- meta = JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="r", request_id=good_request_id))
+ meta = JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="r", request_id=good_request_id)
+ )
assert meta.run_metadata.request_id == good_request_id
diff --git a/tests/unit/pipelex/system/test_read_scope.py b/tests/unit/pipelex/system/test_read_scope.py
new file mode 100644
index 000000000..0671d5d37
--- /dev/null
+++ b/tests/unit/pipelex/system/test_read_scope.py
@@ -0,0 +1,52 @@
+"""`read_scope` on `RunMetadata` — the host-supplied prefix bounding what a run may read.
+
+The check itself is `pipelex.tools.uri.uri_read_scope`; these tests pin the value's own guards: it is
+required, it is a path-safe prefix, and the run's storage scope lies under it, since a run reads its
+own outputs back.
+"""
+
+import pytest
+from pydantic import ValidationError
+
+from pipelex.system.job_metadata import RunMetadata
+
+
+class TestRunMetadataReadScope:
+ def test_omitting_it_is_an_error_rather_than_a_default(self) -> None:
+ with pytest.raises(ValidationError, match="read_scope"):
+ RunMetadata.model_validate({"user_id": "u1", "pipeline_run_id": "run_1", "storage_scope": "org_abc/mt_1/run_1"})
+
+ def test_none_is_the_explicit_unscoped_statement(self) -> None:
+ run_metadata = RunMetadata(user_id="u1", pipeline_run_id="run_1", storage_scope="run_1", read_scope=None)
+ assert run_metadata.read_scope is None
+
+ @pytest.mark.parametrize(
+ ("storage_scope", "read_scope"),
+ [
+ ("org_abc/mt_1/run_1", "org_abc"),
+ ("org_abc/mt_1/run_1", "org_abc/mt_1"),
+ ("org_abc/mt_1/run_1", "org_abc/mt_1/run_1"),
+ ],
+ )
+ def test_a_storage_scope_under_the_read_scope_is_accepted(self, storage_scope: str, read_scope: str) -> None:
+ run_metadata = RunMetadata(user_id="u1", pipeline_run_id="run_1", storage_scope=storage_scope, read_scope=read_scope)
+ assert run_metadata.read_scope == read_scope
+
+ @pytest.mark.parametrize(
+ ("storage_scope", "read_scope"),
+ [
+ ("org_other/mt_1/run_1", "org_abc"),
+ # A string prefix is not a segment prefix.
+ ("org_abcdef/mt_1/run_1", "org_abc"),
+ # A read scope deeper than the storage scope cannot contain it.
+ ("org_abc", "org_abc/mt_1"),
+ ],
+ )
+ def test_a_storage_scope_outside_the_read_scope_is_refused_at_construction(self, storage_scope: str, read_scope: str) -> None:
+ with pytest.raises(ValidationError, match="does not lie under its read_scope"):
+ RunMetadata(user_id="u1", pipeline_run_id="run_1", storage_scope=storage_scope, read_scope=read_scope)
+
+ @pytest.mark.parametrize("read_scope", ["", "..", "org/../other", "/org", "org/", "a/b/c/d", "org\nabc"])
+ def test_an_unsafe_read_scope_is_refused_at_construction(self, read_scope: str) -> None:
+ with pytest.raises(ValidationError, match="Invalid read_scope"):
+ RunMetadata(user_id="u1", pipeline_run_id="run_1", storage_scope="org/run_1", read_scope=read_scope)
diff --git a/tests/unit/pipelex/system/test_read_scope_helpers.py b/tests/unit/pipelex/system/test_read_scope_helpers.py
new file mode 100644
index 000000000..8ac2fba59
--- /dev/null
+++ b/tests/unit/pipelex/system/test_read_scope_helpers.py
@@ -0,0 +1,29 @@
+"""The storage-scope module's read-scope helpers: validation, containment, and the segment-wise key check."""
+
+import pytest
+
+from pipelex.system.storage_scope import is_key_within_read_scope, validate_read_scope, validate_storage_scope_within_read_scope
+
+
+class TestReadScopeHelpers:
+ def test_validate_read_scope_returns_a_valid_value(self) -> None:
+ assert validate_read_scope(value="org_abc") == "org_abc"
+
+ def test_an_unscoped_run_needs_no_containment(self) -> None:
+ validate_storage_scope_within_read_scope(storage_scope="anything/at/all", read_scope=None)
+
+ @pytest.mark.parametrize(
+ ("key", "expected"),
+ [
+ ("org_abc/assets/x.png", True),
+ ("org_abc/x.png", True),
+ ("org_abc", False),
+ ("org_abcdef/x.png", False),
+ ("org_abc/../org_other/x.png", False),
+ ("org_abc/./x.png", False),
+ ("org_abc//x.png", False),
+ ("org_abc\\x.png", False),
+ ],
+ )
+ def test_is_key_within_read_scope_compares_by_segment(self, key: str, expected: bool) -> None:
+ assert is_key_within_read_scope(key=key, read_scope="org_abc") is expected
diff --git a/tests/unit/pipelex/system/test_run_extras.py b/tests/unit/pipelex/system/test_run_extras.py
index 4c2694e55..f25a4d372 100644
--- a/tests/unit/pipelex/system/test_run_extras.py
+++ b/tests/unit/pipelex/system/test_run_extras.py
@@ -29,6 +29,7 @@ def _metadata(extras: dict[str, str]) -> JobMetadata:
user_id="u1",
pipeline_run_id="run_1",
storage_scope="tenant/run_1",
+ read_scope=None,
extras=extras,
)
)
@@ -57,13 +58,13 @@ class TestRunExtras:
it only leaves the group facet empty. The doctrine that made the other
two required is about IDENTITY, and identity is still non-defaulting.
"""
- run_metadata = RunMetadata(user_id="u1", pipeline_run_id="run_1", storage_scope="tenant/run_1")
+ run_metadata = RunMetadata(user_id="u1", pipeline_run_id="run_1", storage_scope="tenant/run_1", read_scope=None)
assert run_metadata.extras == {}
def test_two_run_metadatas_do_not_share_one_default_mapping(self) -> None:
"""A mutable default shared between instances would let one run's extras leak into another."""
- first = RunMetadata(user_id="u1", pipeline_run_id="run_1", storage_scope="tenant/run_1")
- second = RunMetadata(user_id="u2", pipeline_run_id="run_2", storage_scope="tenant/run_2")
+ first = RunMetadata(user_id="u1", pipeline_run_id="run_1", storage_scope="tenant/run_1", read_scope=None)
+ second = RunMetadata(user_id="u2", pipeline_run_id="run_2", storage_scope="tenant/run_2", read_scope=None)
first.extras["organization"] = "org_acme"
assert second.extras == {}
diff --git a/tests/unit/pipelex/system/test_storage_scope.py b/tests/unit/pipelex/system/test_storage_scope.py
index 014249316..676ef951e 100644
--- a/tests/unit/pipelex/system/test_storage_scope.py
+++ b/tests/unit/pipelex/system/test_storage_scope.py
@@ -28,7 +28,7 @@ from pipelex.system.telemetry.otel_constants import OTelConstants
def _metadata(storage_scope: str) -> JobMetadata:
- return JobMetadata(run_metadata=RunMetadata(user_id="u1", pipeline_run_id="run_1", storage_scope=storage_scope))
+ return JobMetadata(run_metadata=RunMetadata(user_id="u1", pipeline_run_id="run_1", storage_scope=storage_scope, read_scope=None))
class TestAcceptedScopes:
@@ -122,7 +122,7 @@ class TestScopeIsRequired:
# Both type checkers reject this call, which is the point being
# tested: the omission is caught statically AND at runtime. The
# ignores are what let the runtime half be asserted at all.
- JobMetadata(run_metadata=RunMetadata(user_id="u1", pipeline_run_id="run_1")) # type: ignore[call-arg] # pyright: ignore[reportCallIssue]
+ JobMetadata(run_metadata=RunMetadata(user_id="u1", pipeline_run_id="run_1", read_scope=None)) # type: ignore[call-arg] # pyright: ignore[reportCallIssue]
class TestTheTelemetryPlaceholderStaysOutOfIdentity:
diff --git a/tests/unit/pipelex/tools/uri/test_uri_read_scope.py b/tests/unit/pipelex/tools/uri/test_uri_read_scope.py
new file mode 100644
index 000000000..0e774c3f4
--- /dev/null
+++ b/tests/unit/pipelex/tools/uri/test_uri_read_scope.py
@@ -0,0 +1,118 @@
+"""The read scope's check: a scoped run reads only its own storage prefix, and never the local disk."""
+
+import pytest
+
+from pipelex.base_exceptions import DisclosureMode, ErrorDomain
+from pipelex.tools.uri.exceptions import UriReadRefusalReason, UriReadRefusedError
+from pipelex.tools.uri.uri_read_scope import UriReference, authorize_uri_read, authorize_uri_reads
+
+READ_SCOPE = "org_abc"
+
+
+class TestAuthorizeUriRead:
+ @pytest.mark.parametrize(
+ "uri",
+ [
+ "pipelex-storage://org_abc/assets/photo.png",
+ "pipelex-storage://org_abc/mt_1/run_9/generated/image.png",
+ "https://example.com/photo.png",
+ "http://example.com/photo.png",
+ "data:image/png;base64,iVBORw0KGgo=",
+ ],
+ )
+ def test_scoped_run_reads_what_the_rule_allows(self, uri: str) -> None:
+ authorize_uri_read(uri=uri, read_scope=READ_SCOPE, position="image 1 of the prompt")
+
+ @pytest.mark.parametrize(
+ "uri",
+ [
+ "pipelex-storage://org_other/assets/secret.png",
+ # A string prefix of the key's first segment is not a segment prefix.
+ "pipelex-storage://org_abcdef/assets/secret.png",
+ "pipelex-storage://org_ab/assets/secret.png",
+ # Dot segments and empty segments, wherever they sit: the local provider resolves a key against a directory.
+ "pipelex-storage://org_abc/../org_other/secret.png",
+ "pipelex-storage://org_abc/./assets/photo.png",
+ "pipelex-storage://org_abc//assets/photo.png",
+ "pipelex-storage:///org_abc/assets/photo.png",
+ "pipelex-storage://org_abc/assets/",
+ # A backslash is a separator on Windows.
+ "pipelex-storage://org_abc/..\\org_other\\secret.png",
+ # The scope itself names no object.
+ "pipelex-storage://org_abc",
+ "pipelex-storage://",
+ ],
+ )
+ def test_scoped_run_refuses_a_key_outside_its_scope(self, uri: str) -> None:
+ with pytest.raises(UriReadRefusedError) as exc_info:
+ authorize_uri_read(uri=uri, read_scope=READ_SCOPE, position="image 1 of the prompt")
+ assert exc_info.value.reason == UriReadRefusalReason.FOREIGN_STORAGE_KEY
+
+ @pytest.mark.parametrize(
+ "uri",
+ [
+ "/etc/passwd",
+ "relative/photo.png",
+ "photo.png",
+ "file:///etc/passwd",
+ "~/photo.png",
+ # A scheme `resolve_uri` does not know is taken for a local path, and refused as one.
+ "s3://bucket/key.png",
+ # A data URL without its base64 marker is not one.
+ "data:image/png,rawbytes",
+ ],
+ )
+ def test_scoped_run_refuses_a_local_path(self, uri: str) -> None:
+ with pytest.raises(UriReadRefusedError) as exc_info:
+ authorize_uri_read(uri=uri, read_scope=READ_SCOPE, position="the input 'photo'")
+ assert exc_info.value.reason == UriReadRefusalReason.LOCAL_PATH
+
+ def test_a_multi_segment_scope_compares_every_segment(self) -> None:
+ authorize_uri_read(uri="pipelex-storage://org_abc/team_1/x.png", read_scope="org_abc/team_1", position="p")
+ with pytest.raises(UriReadRefusedError):
+ authorize_uri_read(uri="pipelex-storage://org_abc/team_2/x.png", read_scope="org_abc/team_1", position="p")
+
+ @pytest.mark.parametrize(
+ "uri",
+ [
+ "pipelex-storage://org_other/assets/secret.png",
+ "pipelex-storage://org_abc/../org_other/secret.png",
+ "/etc/passwd",
+ "file:///etc/passwd",
+ "https://example.com/photo.png",
+ ],
+ )
+ def test_an_unscoped_run_reads_everything(self, uri: str) -> None:
+ authorize_uri_read(uri=uri, read_scope=None, position="image 1 of the prompt")
+
+ def test_the_refusal_names_the_position_and_quotes_neither_the_uri_nor_the_scope(self) -> None:
+ with pytest.raises(UriReadRefusedError) as exc_info:
+ authorize_uri_read(uri="pipelex-storage://org_other/assets/secret.png", read_scope=READ_SCOPE, position="image 2 of the prompt")
+ message = str(exc_info.value)
+ assert "image 2 of the prompt" in message
+ assert "org_other" not in message
+ assert "secret.png" not in message
+ assert READ_SCOPE not in message
+
+ def test_the_refusal_is_a_caller_facing_input_error_answered_as_a_422(self) -> None:
+ with pytest.raises(UriReadRefusedError) as exc_info:
+ authorize_uri_read(uri="/etc/passwd", read_scope=READ_SCOPE, position="the input 'photo'")
+ error_report = exc_info.value.to_error_report()
+ assert error_report.error_domain == ErrorDomain.INPUT
+ assert error_report.http_status == 422
+ strict = error_report.to_dict(disclosure_mode=DisclosureMode.STRICT)
+ assert "the input 'photo'" in strict["message"]
+ assert "/etc/passwd" not in strict["message"]
+
+ def test_the_list_form_refuses_the_first_reference_the_scope_does_not_allow(self) -> None:
+ uri_references = [
+ UriReference(uri="pipelex-storage://org_abc/assets/ok.png", position="image 1 of the prompt"),
+ UriReference(uri="pipelex-storage://org_other/assets/secret.png", position="image 2 of the prompt"),
+ UriReference(uri="/etc/passwd", position="image 3 of the prompt"),
+ ]
+ with pytest.raises(UriReadRefusedError) as exc_info:
+ authorize_uri_reads(uri_references=uri_references, read_scope=READ_SCOPE)
+ assert "image 2 of the prompt" in str(exc_info.value)
+
+ def test_the_list_form_reads_nothing_from_an_empty_list(self) -> None:
+ authorize_uri_reads(uri_references=[], read_scope=READ_SCOPE)
diff --git a/tests/unit/pipelex/tracing/test_assembler_equivalence.py b/tests/unit/pipelex/tracing/test_assembler_equivalence.py
index 406d30a0e..21fab505d 100644
--- a/tests/unit/pipelex/tracing/test_assembler_equivalence.py
+++ b/tests/unit/pipelex/tracing/test_assembler_equivalence.py
@@ -474,7 +474,7 @@ def _usage_events_for(*, assembled_child_node_id: str, event_log: InMemoryEventL
node_id=assembled_child_node_id,
tokens_usage=LLMTokensUsage(
job_metadata=JobMetadata(
- run_metadata=RunMetadata(storage_scope="test/scope", user_id="user_test", pipeline_run_id=_PIPELINE_RUN_ID),
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="user_test", pipeline_run_id=_PIPELINE_RUN_ID),
pipe_code="gen_text",
unit_job_id=UnitJobId.LLM_GEN_TEXT,
job_category=JobCategory.LLM_JOB,
diff --git a/tests/unit/pipelex/tracing/test_assembler_usage_attribution.py b/tests/unit/pipelex/tracing/test_assembler_usage_attribution.py
index 0b64144e2..c5493ac54 100644
--- a/tests/unit/pipelex/tracing/test_assembler_usage_attribution.py
+++ b/tests/unit/pipelex/tracing/test_assembler_usage_attribution.py
@@ -45,7 +45,7 @@ def _base(sequence: int) -> dict[str, Any]:
def _make_usage(*, unit_costs: CostsByCategoryDict, model_name: str = "test-model") -> AnyTokensUsage:
return LLMTokensUsage(
job_metadata=JobMetadata(
- run_metadata=RunMetadata(storage_scope="test/scope", user_id="user_test", pipeline_run_id=_PIPELINE_RUN_ID),
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="user_test", pipeline_run_id=_PIPELINE_RUN_ID),
pipe_code="test_pipe",
unit_job_id=UnitJobId.LLM_GEN_TEXT,
job_category=JobCategory.LLM_JOB,
diff --git a/tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py b/tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py
index e757c657a..32e72c105 100644
--- a/tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py
+++ b/tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py
@@ -26,7 +26,7 @@ from pipelex.tracing.usage_aggregator import UsageAggregator
def _job_metadata() -> JobMetadata:
- return JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="run-non-llm"))
+ return JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="test-user", pipeline_run_id="run-non-llm"))
def _usage_event(tokens_usage: AnyTokensUsage, sequence: int) -> UsageReportEvent:
diff --git a/tests/unit/pipelex/tracing/test_trace_events.py b/tests/unit/pipelex/tracing/test_trace_events.py
index a0196398c..171f71220 100644
--- a/tests/unit/pipelex/tracing/test_trace_events.py
+++ b/tests/unit/pipelex/tracing/test_trace_events.py
@@ -69,7 +69,7 @@ class _Shared:
@staticmethod
def make_job_metadata() -> JobMetadata:
return JobMetadata(
- run_metadata=RunMetadata(storage_scope="test/scope", user_id="user_test", pipeline_run_id=_Shared.PIPELINE_RUN_ID),
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="user_test", pipeline_run_id=_Shared.PIPELINE_RUN_ID),
pipe_code="test_pipe",
unit_job_id=UnitJobId.LLM_GEN_TEXT,
job_category=JobCategory.LLM_JOB,
diff --git a/tests/unit/pipelex/tracing/test_usage_aggregator.py b/tests/unit/pipelex/tracing/test_usage_aggregator.py
index 7764fbe24..13f1a44ac 100644
--- a/tests/unit/pipelex/tracing/test_usage_aggregator.py
+++ b/tests/unit/pipelex/tracing/test_usage_aggregator.py
@@ -19,7 +19,7 @@ _WORKFLOW_ID = "wf_abc"
def _make_llm_tokens_usage(model_name: str = "claude-sonnet") -> LLMTokensUsage:
return LLMTokensUsage(
job_metadata=JobMetadata(
- run_metadata=RunMetadata(storage_scope="test/scope", user_id="user_test", pipeline_run_id=_PIPELINE_RUN_ID),
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="user_test", pipeline_run_id=_PIPELINE_RUN_ID),
pipe_code="test_pipe",
unit_job_id=UnitJobId.LLM_GEN_TEXT,
job_category=JobCategory.LLM_JOB,
diff --git a/tests/unit/pipelex/tracing/test_usage_attribution.py b/tests/unit/pipelex/tracing/test_usage_attribution.py
index 1a7a2d952..1de3fed7c 100644
--- a/tests/unit/pipelex/tracing/test_usage_attribution.py
+++ b/tests/unit/pipelex/tracing/test_usage_attribution.py
@@ -29,7 +29,7 @@ def _make_usage(
) -> AnyTokensUsage:
return LLMTokensUsage(
job_metadata=JobMetadata(
- run_metadata=RunMetadata(storage_scope="test/scope", user_id="user_test", pipeline_run_id="run_001"),
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="user_test", pipeline_run_id="run_001"),
pipe_code="test_pipe",
unit_job_id=UnitJobId.LLM_GEN_TEXT,
job_category=JobCategory.LLM_JOB,