Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feature/Html-template-assets · L-261015-8bf2d8 (#1437)

Louis Choquel3e9874ff1134
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3d6ade41f..a40fb40e9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -11,6 +11,8 @@
- **`pipelex fix function-names` and `pipelex-agent fix function-names`**: rewrite every flat `PipeFunc` `function_name` of a method folder as the function's import path and drop the argument of every `@pipe_func(name=...)` marker, reading the files as syntax trees without importing them or booting Pipelex, and keeping every comment and alignment; a flat name no module of the method's tree registers, or that two register, is left as it is and reported with the markers registering it, which keep their argument, and so is a marker whose name is not a string literal, the agent command's `is_complete` being false and both commands exiting `1`, with `--dry-run` to preview.
- **`pipe_func_function_refused` and the method-tree Python errors**: a `PipeFunc` whose function cannot be found or proven is refused at validation with the `pipe_func_function_refused` validation error type, located on the step's `function_name`, and the new `PipeFuncFunctionError`, `TreePythonError`, `TreePythonHeldError`, `TreeModuleShadowingError` and `FunctionNamesMigrationError` name the failures of a function, of a tree's Python and of the migration.
- **The method file store**: a step reads a file of its method where reading is allowed, at the moment it uses it, through `MethodFileStoreProtocol`, whose `read_tree_listing(tree_digest=..., read_scope=...)` and `read_blob(sha256=..., read_scope=...)` return a tree's listing and a file's bytes under the run's read scope, each read checked against the tree's digest (`pipelex.core.method_trees.method_files`). The local store, `LOCAL_METHOD_FILES`, serves the trees this process's libraries hold, and a plugin whose activities run apart from the process that loaded the method claims `HubSlot.METHOD_FILE_STORE` with `PluginRegistrar.claim_method_file_store`; a file the store does not hold for the run fails its step with the new `MethodFileUnavailableError`. An orchestrator running a job out of process copies every tree the job's crate names into the store its workers read before its `execute` or `start` returns, since the runner may drop the files as soon as it does, and `read_stored_trees(tree_digests=..., read_scope=...)` reads them for that copy, each as a store keeps it, its listing under its digest and its files under their hashes. The local store finds a file by its hash in one lookup and reads a file on disk off the event loop.
+- **Relative URLs in HTML templates read the method's own files**: a relative URL in a `PipeDocGen` HTML template, inline or a template file, such as an image's `src`, a stylesheet's `href`, a font, or a `url()` inside a stylesheet the template links, names a file of the template's method tree, resolved by the MTHDS standard's rules against the template file, or against the bundle file of an inline template, and read through the method file store under the run's read scope wherever the document prints, so a logo, a stylesheet and a font travel with their method. Only images, stylesheets and fonts are read, by suffix, so a URL a run's input supplies cannot make a document carry the method's other files. A URL the template writes as it is is checked when the method loads, and a URL built from the inputs that climbs above the tree's root, names a folder, a missing file, a file the tree's membership rule keeps out or a file that is no asset fails the print with the new `TemplateUrlRefusedError`, an input error naming the URL and the template; a URL with a scheme, such as `https://`, `data:` or `pipelex-storage://`, is read as before.
+- **`RenderJob.base_url` and the method-file arm of `RenderResources.load`**: an HTML job whose template belongs to a method tree carries the URL a document engine joins the document's relative URLs to, under `METHOD_FILE_URL_PREFIX` (`https://method-files.pipelex.invalid/`, which never resolves), and `RenderResources.load` reads a URL joined under it, a stylesheet's own URLs included, from the document's method tree, with no media type; `DocumentComposition.url_base` carries the template's place in its tree to the print stage. They join plugin API version 7, and an engine that never reads `base_url` keeps refusing a relative URL.
### Changed
@@ -32,6 +34,7 @@
- **A `PipeFunc` call names its method tree (Breaking)**: `PipeFuncExecutionRequest.tree_digest` names the tree of the called step, and the process running the call reads its files, every one but the `.mthds` files and a generated `structures` module, through the method file store under the run's read scope (`read_pipe_func_call_files`). The direct executor's transported run writes them into a temporary directory, generates `structures` unless the method shipped one, imports the function by its import path under the process's hold on that directory and unloads its modules before removing it; the transported entrypoint takes as a required third argument the directory a sandbox client unpacked the files into, since a sandbox reaches no method file store, and runs `DirectPipeFuncExecutor.run_pipe_func_transported_at` on it. Only the called step is built in the tree written for the call, which holds none of the tree's bundles: `load_from_crate` and `rehydrate_library_and_memory` take `written_pipe_trees`, keyed by the step's pipe ref, and every other step, a `PipeDocGen` of the same tree included, is built as the crate names it, its template file named by the crate's digest. The transported library's id is public, `TRANSPORTED_LIBRARY_ID`.
- **A `PipeDocGen` template file is read where the document prints (Breaking)**: `DocumentComposition.template_file`, a `ComposedTemplateFile` naming the file by its tree's digest and tree path (`MethodFileRef`), replaces `template_path` and `template_resource`, and the print stage reads it through the method file store under the run's read scope, renders an `.html` one with the step's data, carried as a `TemplateRenderContext` (the stuffs the step's inputs name, the concepts and domains of the run's crate with none of its pipes, the crate's fingerprint, and the run's runtime parameters), through the template file stage the interpreter installs at boot, a dry print included, in the run's own library where this process holds it and otherwise in a library loaded from the concepts, which the process keeps for later prints (`RenderLibraryCache`), and hands an office one's bytes to its engine, whose template check moves from the step's dry run into the dry print. An `.html` template file whose bytes changed after its tree was listed is refused at load, never read from the disk unchecked. `make_render_job` is synchronous again and takes the rendered `html` or the template's bytes; an inline `template` is still rendered where the step runs.
- **Files submitted with content strings make one method tree (Breaking)**: `PipelexMTHDSProtocol`, `validate_bundle`, `validate_bundles_in_process`, `acquire_library`, `pipeline_run_setup`, `resolve_crate_from_contents` and `BundleValidatorProtocol.validate_bundles` take `submitted_files`, a mapping from the path a step names to the file's bytes, in place of `bundle_resources`, and `LibraryManagerAbstract.add_submitted_files` replaces `add_bundle_resources`. The run names their tree by its digest and carries none of the files, so nothing is stored under the run's `assets/bundle/` before it leaves the process, the 512 KiB bound on `.html` template files is lifted, and files that cannot make one tree, such as two paths equal once case-folded, raise `BundleResourcePathError`. A step names a submitted file by any spelling of its path, so `templates//invoice.docx` reads the file submitted as `templates/invoice.docx`.
+- **A relative URL in an HTML template names a file of the method (Breaking)**: a literal relative URL of an HTML template, or of a stylesheet it links, that names no image, stylesheet or font of the method's tree now fails the method's load, `pipelex validate` and dry runs included, where it loaded and failed only at print. A relative path such as `images/logo.png` that an engine printing HTML hands `RenderResources.load` as written now resolves in the template's method tree, and is refused with `TemplateUrlRefusedError` when the template belongs to none, where an unscoped run read it from its working directory; a root-absolute local path, and every URL of a document printed from the auto-layout or an office template, read as before.
### Fixed
diff --git a/docs/building-methods/pipes/pipe-operators/PipeDocGen.md b/docs/building-methods/pipes/pipe-operators/PipeDocGen.md
index 10a364d4c..158e2be4b 100644
--- a/docs/building-methods/pipes/pipe-operators/PipeDocGen.md
+++ b/docs/building-methods/pipes/pipe-operators/PipeDocGen.md
@@ -130,6 +130,44 @@ A `pdf` step's `template`, or its `.html` `template_file`, is rendered with exac
An inline `template` is rendered where the step runs, its text being the bundle's own. A `template_file` is read where the document is printed: the step names the file by its method tree and its path there, and the print stage reads it, renders it with the step's data and prints it, a dry run's print included, so a value the template reads that does not exist fails the dry run as it fails a live one. On a distributed orchestrator such as Temporal, the step runs in a workflow, which reads no file, and the print runs in an activity, which reads the file from the method file store of the deployment. An HTML template file is printed with the inputs the step declares, the only data it may read, so an HTML step's print carries none of the run's other data.
+### Images, stylesheets and fonts beside a template
+
+An HTML template names the files of its method by relative URLs, as a web page names the files beside it. A relative URL resolves against the template file's own place in the method's folder, and an inline `template`'s against its bundle file, so a brand kit travels with its method: the logo beside the template, the stylesheet that lays it out, the font in `fonts/`. Take a method whose folder holds these files:
+
+```text
+METHODS.toml
+invoice.mthds
+templates/invoice.html
+templates/logo.png
+styles/print.css
+fonts/brand.woff2
+```
+
+The template names the logo beside it and the stylesheet one folder up:
+
+```html
+<link rel="stylesheet" href="../styles/print.css">
+<img src="logo.png" alt="Acme">
+<h1>Invoice {{ invoice.number }}</h1>
+```
+
+A URL inside a stylesheet resolves against the stylesheet, as in a browser, so `styles/print.css` names the font from its own folder:
+
+```css
+@font-face { font-family: Brand; src: url("../fonts/brand.woff2"); }
+body { font-family: Brand; }
+```
+
+- **The files are read from the method, never from a disk.** The print reads each file from the method's tree, under the run's read scope, wherever it prints: in the process that loaded the method, and on a distributed orchestrator such as Temporal, where the print activity reads it from the method file store of the deployment.
+- **Only images, stylesheets and fonts are read.** A relative URL naming any other file of the method, such as its Python, a bundle, `METHODS.toml`, `requirements.txt` or a data file, is refused, by its suffix: a URL may come from a run's input, such as `<img src="{{ photo.url }}">`, and a document can carry what it reads, as a `rel="attachment"` link embeds its file in the PDF, so a caller allowed to run a method does not get to read its files that way. The images are `.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`, `.svg`, `.avif`, `.apng`, `.bmp`, `.ico`, `.tif` and `.tiff`, the stylesheets `.css`, and the fonts `.woff`, `.woff2`, `.ttf`, `.otf`, `.ttc` and `.eot`.
+- **`..` climbs anywhere inside the method, never above its root.** A URL that climbs above the folder holding `METHODS.toml`, a root-relative URL such as `/logo.png`, a folder, a missing file, and a file the method leaves out, such as a hidden or a gitignored one, are refused with a message naming the URL, the template and the reason. A file outside the method is never read, even when it exists. A bundle named alone, with no `METHODS.toml` at or above it, is a method of that one file, so a file beside it is refused with the advice to hand over the method's folder.
+- **A URL the template writes is checked when the method loads.** `pipelex validate`, a dry run and every load check each literal relative URL of the template and of the stylesheets it links, such as `<img src="logo.png">`, so a missing or misspelt file fails before any print. A URL the template builds from the run's inputs is checked when the document prints, and a refused one fails the print with a `TemplateUrlRefusedError`.
+- **A URL with a scheme is left as written.** An `https://` URL is fetched through the SSRF guard, a `data:` URL is decoded, and a `pipelex-storage://` URL is read under the run's read scope, as before, so a template's split between the method's files and the caller's is the URL's own: relative for the method, a stored URI for a caller's file or an earlier output. A URL with no path, such as `#top`, names no file.
+- **A file name holding `?`, `#` or `%` is written percent-encoded**, `100%25.png` for `100%.png`, since a URL's query and fragment are dropped before it resolves and a `%` starts an escape.
+- **A bundle loaded from a string** resolves its inline template's relative URLs from the root of the files sent with it, and with no file sent beside it, it belongs to no method tree, so a relative URL its template writes is refused when the method loads, and one built from the run's inputs fails the print. Load such a method from its folder.
+
+The engine that prints HTML joins each relative URL to a base URL Pipelex hands it and reads the file through Pipelex ([Document Engine Plugins](../../../under-the-hood/document-engine-plugins.md#relative-urls-the-files-of-the-templates-method-tree)).
+
### Office template files
An `.xlsx`, `.docx` or `.pptx` template file is filled by its engine from the inputs as plain data, so it sees fields, not the sigils and filters of HTML templates. In plain data a native value is its value, and a verdict is its verdict: a `YesNo` is its boolean, a `Choice` its option key and a `Rating` its level, so the file reads `team` rather than `team.choice`, and a verdict's probability or confidence does not reach it. A `pdf` template reads the verdict whole, `team.choice` and `team.confidence` included, as a `PipeCompose` template does. Its contract is documented with the plugin. When the plugin's engine checks templates, `pipelex validate` and the dry run compare the file with the inputs' concepts: a misspelled name is reported before any run.
@@ -149,7 +187,7 @@ When a bundle is sent as content rather than read from a directory, as it is thr
- **The key is the path the step names**, relative to the bundle and written with forward slashes: a step with `template_file = "templates/invoice.docx"` reads the file sent as `templates/invoice.docx`, and `./templates/invoice.docx` or `templates//invoice.docx`, as a key or as the path a step names, names the same file. Every bundle of the request is read as sitting in one directory. An absolute path, a `..` segment, a backslash and a colon are refused, as a key and as the path a step names: a step whose `template_file` could not be a key is refused when its bundle is parsed, whatever the request sends, so every file a string-loaded step can name can be sent with it.
- **The value is the file's bytes** in standard base64.
-- **Every file sent must be one a step names.** A file no step reads is refused, so a misspelled key fails loudly rather than leaving its step without a template.
+- **Every file sent must be one a step names.** A file no step reads is refused, so a misspelled key fails loudly rather than leaving its step without a template. A file a template names by a relative URL, such as its logo, is named by no step, so it cannot be sent this way: such a method is loaded from its folder.
- **The files are checked like files on disk.** An `.html` template file is checked against the inputs when the method loads, and the engine's template check runs on an office template file at validation and in the dry run.
The files sent with a bundle make one method tree, with the bundles sent as strings at its root: the run names that tree by its digest, as it names the tree of a bundle loaded from a directory, and carries none of its files. The print stage reads a template file from the tree, under the run's read scope, to print it or to check it, wherever it runs: in the process that loaded the method, the tree is the one its library holds; on a remote orchestrator such as Temporal, the deployment's method file store serves it. No template file rides in the run's payloads, so none is bounded by them.
diff --git a/docs/errors/inference-and-providers.md b/docs/errors/inference-and-providers.md
index 6c2714d40..93cb704c1 100644
--- a/docs/errors/inference-and-providers.md
+++ b/docs/errors/inference-and-providers.md
@@ -89,6 +89,7 @@ own page. Classes are grouped by subsystem.
- [`SearchJobFailureError`](search-job-failure-error.md) — Search job failure
- [`SearchModelNotFoundError`](search-model-not-found-error.md) — Search model not found
- [`TemplateSigilSyntaxError`](template-sigil-syntax-error.md) — Template sigil syntax
+- [`TemplateUrlRefusedError`](template-url-refused-error.md) — Relative URL refused
- [`UnsafeSchemaError`](unsafe-schema-error.md) — Unsafe schema
## Provider adapters
diff --git a/docs/errors/template-url-refused-error.md b/docs/errors/template-url-refused-error.md
new file mode 100644
index 000000000..f82398a07
--- /dev/null
+++ b/docs/errors/template-url-refused-error.md
@@ -0,0 +1,32 @@
+---
+title: "Relative URL refused"
+description: "Reference for the `TemplateUrlRefusedError` Pipelex error class."
+---
+
+<!-- pipelex:authored -->
+
+# Relative URL refused
+
+A relative URL of an HTML document names no image, stylesheet or font of its template's method tree, so the print reads nothing.
+
+| Field | Value |
+|---|---|
+| `error_type` | `TemplateUrlRefusedError` |
+| `title` | Relative URL refused |
+| `type_uri` | `https://docs.pipelex.com/latest/errors/template-url-refused-error/` |
+| `error_domain` | `input` |
+| Defined in | `pipelex.cogt.doc_gen.exceptions` |
+| Parent class | [`PipelexError`](pipelex-error.md) |
+
+## When it is raised
+
+A relative URL in a `PipeDocGen` HTML template names a file of the template's method, resolved against the template file, or against the bundle file of an inline template ([Images, stylesheets and fonts beside a template](../building-methods/pipes/pipe-operators/PipeDocGen.md#images-stylesheets-and-fonts-beside-a-template)). The print refuses it, and reads nothing, when:
+
+- **It names no file of the method.** It climbs above the folder holding `METHODS.toml`, it names a folder or a missing file, or it names a file the method leaves out, such as a hidden or a gitignored one.
+- **It names a file that is not an image, a stylesheet or a font.** A document reads only those from its method, whatever the URL's origin, so a URL a run's input supplies cannot make the document carry the method's Python, its bundles, its manifest or its data, as a `rel="attachment"` link would.
+- **The template belongs to no method's folder**, such as an inline template of a bundle loaded from a string with no file sent beside it, where a relative URL names no file at all.
+- **A document with no HTML names a method file.** Only the relative URLs of an HTML template name one.
+
+A URL the template writes as it is, such as `<img src="logo.png">`, is checked when the method loads, so this error is raised at print for a URL the template builds from the run's inputs. `refusal_code` carries the MTHDS standard's refusal code when the standard names the reason, and is `None` otherwise. The message names the method's files by their paths in the method, never by a path on the machine that prints.
+
+[Back to Error Reference](index.md)
diff --git a/docs/under-the-hood/document-engine-plugins.md b/docs/under-the-hood/document-engine-plugins.md
index 5abef0fc7..5108f9646 100644
--- a/docs/under-the-hood/document-engine-plugins.md
+++ b/docs/under-the-hood/document-engine-plugins.md
@@ -118,13 +118,19 @@ An engine's worker subclasses `DocGenWorkerAbstract` from `pipelex.cogt.doc_gen.
def render(self, *, job: RenderJob, resources: RenderResources) -> RenderedDocument: ...
```
-- **`RenderJob`** is plain data, which Pipelex builds in the print stage from the step's composition and hands to the worker in the same process. It has a JSON round trip for an engine that prints in another process, in which the template's bytes are URL-safe base64 and a date or a time is its ISO text. It holds the format and source, the file's name with its suffix, the document's title, and exactly the payload its source names, `layout` (a `LayoutDocument` from `layout_tree.py`), `html`, or `template` with `data`, the inputs as plain data by input name. It carries no Pipelex object, so an ordinary Pipelex release does not break an engine; a change to it, or to the worker, is a change of the plugin contract.
-- **`RenderResources.load(uri=…, position=…)`** is how an engine reads a file its document names, such as an image or a stylesheet. It resolves `pipelex-storage://` keys through the run's storage provider, decodes `data:` URLs, fetches `https://` through the SSRF guard, and refuses what the run's read scope does not allow, a local path included, with `UriReadRefusedError`. An engine reads nothing any other way. It returns a **`LoadedResource`**: the file's bytes as `data`, and as `mime_type` the media type the file's source gives it, described below.
+- **`RenderJob`** is plain data, which Pipelex builds in the print stage from the step's composition and hands to the worker in the same process. It has a JSON round trip for an engine that prints in another process, in which the template's bytes are URL-safe base64 and a date or a time is its ISO text. It holds the format and source, the file's name with its suffix, the document's title, and exactly the payload its source names, `layout` (a `LayoutDocument` from `layout_tree.py`), `html`, or `template` with `data`, the inputs as plain data by input name. An HTML job whose template belongs to a method tree also carries `base_url`, described below. It carries no Pipelex object, so an ordinary Pipelex release does not break an engine; a change to it, or to the worker, is a change of the plugin contract.
+- **`RenderResources.load(uri=…, position=…)`** is how an engine reads a file its document names, such as an image or a stylesheet. It resolves `pipelex-storage://` keys through the run's storage provider, decodes `data:` URLs, fetches `https://` through the SSRF guard, and refuses what the run's read scope does not allow, a local path included, with `UriReadRefusedError`. It reads a file of the document's method tree, named by a relative URL, as the next section describes. An engine reads nothing any other way. It returns a **`LoadedResource`**: the file's bytes as `data`, and as `mime_type` the media type the file's source gives it, described below.
- **`RenderedDocument`** holds the bytes. Their MIME type and suffix are the format's.
-`render` is synchronous and runs on a thread of a print pool of its own, never on the event loop's default executor, which the engine's reads need, while `RenderResources.load` hands each read back to the event loop the print started from. The print stage that calls it ends the print with the event every inference call ends with, `Inference call ends`, carrying no tokens and no cost since an engine reports none (see [Summary events](../tools/logging.md#summary-events)), logged on the coroutine that awaits the thread, so a print cancelled from outside ends `cancelled` while the thread finishes its render; an engine overrides `render` alone and logs nothing for the print itself. An engine keeps no state from one print into the next. It signals a document it cannot print by raising `DocGenRenderError`, which passes through; anything else it raises, beyond a Pipelex error, is reported as a `DocGenRenderError` naming the engine and the file.
+`render` is synchronous and runs on a thread of a print pool of its own, never on the event loop's default executor, which the engine's reads need, while `RenderResources.load` hands each read back to the event loop the print started from. The print stage that calls it ends the print with the event every inference call ends with, `Inference call ends`, carrying no tokens and no cost since an engine reports none (see [Summary events](../tools/logging.md#summary-events)), logged on the coroutine that awaits the thread, so a print cancelled from outside ends `cancelled` while the thread finishes its render; an engine overrides `render` alone and logs nothing for the print itself. An engine keeps no state from one print into the next. It signals a document it cannot print by raising `DocGenRenderError`, which passes through; anything else it raises, beyond a Pipelex error, is reported as a `DocGenRenderError` naming the engine and the file. A Pipelex error raised by `RenderResources.load`, such as `UriReadRefusedError` or `TemplateUrlRefusedError`, is let through as it came, so the caller reads why the file was refused.
-The contract is what an engine uses in `pipelex.cogt.doc_gen`: the worker; the render job, its resources, the `LoadedResource` they return and the rendered document (`render_job.py`); the formats and sources (`doc_gen_format.py`); `DocGenRenderError`, the one error of `exceptions.py` an engine raises, and `MarkdownFormattingBudgetError`, the one the contract raises to an engine that formats Markdown outside a template render; the layout tree (`layout_tree.py`); the template check request and the findings a checker returns (`template_check.py`), with the input shapes the request carries (`InputShape` and `InputShapeKind` in `input_shape.py`); and the modules of the next section. The rest of the package is Pipelex's own side of the print. An engine reaches Pipelex only through the contract and the registration seam every plugin uses, which also hands a worker its model spec, its backend, its SDK clients and its reporting delegate: anything else in Pipelex may move or change in an ordinary release, while a breaking change to the contract moves `PLUGIN_API_VERSION`. A piece an engine is missing is added to the contract, rather than imported from elsewhere in Pipelex.
+The contract is what an engine uses in `pipelex.cogt.doc_gen`: the worker; the render job, its resources, the `LoadedResource` they return and the rendered document (`render_job.py`); the formats and sources (`doc_gen_format.py`); `DocGenRenderError`, the one error of `exceptions.py` an engine raises, `MarkdownFormattingBudgetError`, the one the contract raises to an engine that formats Markdown outside a template render, and `TemplateUrlRefusedError`, the one `RenderResources.load` raises for a relative URL naming no file of the method tree; the layout tree (`layout_tree.py`); the template check request and the findings a checker returns (`template_check.py`), with the input shapes the request carries (`InputShape` and `InputShapeKind` in `input_shape.py`); and the modules of the next section. The rest of the package is Pipelex's own side of the print. An engine reaches Pipelex only through the contract and the registration seam every plugin uses, which also hands a worker its model spec, its backend, its SDK clients and its reporting delegate: anything else in Pipelex may move or change in an ordinary release, while a breaking change to the contract moves `PLUGIN_API_VERSION`. A piece an engine is missing is added to the contract, rather than imported from elsewhere in Pipelex.
+
+### Relative URLs: the files of the template's method tree
+
+A relative URL in an HTML document, such as the `src` of the logo beside its template, a stylesheet's `href`, or a font a stylesheet names with `url()`, names a file of the template's method tree, by the MTHDS standard's rules for a file reference in a URL (the [Method Tree specification](https://mthds.ai/latest/spec/method-tree/), "File References"). An engine resolves it as HTML and CSS resolve one, by joining it to the URL of the file that holds it: `RenderJob.base_url` for the document's own URLs, and a stylesheet's URL for the URLs the stylesheet holds. The base URL stands for the template's place in its tree: `METHOD_FILE_URL_PREFIX` (`https://method-files.pipelex.invalid/`, a host in the `.invalid` domain, which never resolves), the tree's digest, then the template file's tree path, or the bundle's for an inline template. An engine hands a URL it joined under the prefix to `RenderResources.load`, like any other, and never fetches one itself: Pipelex resolves it in the tree's listing and reads the file through the method file store, under the run's read scope, never from a disk. A WeasyPrint engine passes `base_url` to `HTML(string=…, base_url=…)` and routes a URL under the prefix to `load` in its URL fetcher.
+
+A URL that climbs above the tree's root, a root-relative one such as `/logo.png` once joined, a file the tree does not hold, a directory, and a file the tree's membership rule keeps out, such as a hidden or a gitignored one, are refused with `TemplateUrlRefusedError`, an input error naming the URL's position, what it resolved to and the template. So is a file of the tree that is not an image, a stylesheet or a font, told by its suffix, since a URL may come from a run's input and a printer embeds what a `rel="attachment"` link names. A job with no `base_url` has a template that belongs to no method tree, such as an inline template of a bundle loaded from a string with no file sent beside it: a relative URL of its HTML names no file, and `load` refuses it the same way. A URL under the prefix in a document with no HTML is refused too. An engine that never reads `base_url` keeps refusing a relative URL, as it did before the field existed. A URL with a scheme, such as `https://`, `data:` or `pipelex-storage://`, is read as it names, as before, and so is a root-absolute local path an engine hands as written, such as `/tmp/photo.png`, which only an unscoped run reads. A literal relative URL of a template is checked when the method loads, so a refusal at print comes from a URL built from the run's inputs.
### The media type of a file an engine reads
@@ -137,6 +143,7 @@ The contract is what an engine uses in `pipelex.cogt.doc_gen`: the worker; the r
| `pipelex-storage://` on S3 or GCS | The content type recorded when the file was stored. |
| `pipelex-storage://` on the local or in-memory provider | The type identified from the bytes, which covers binary formats only: a stylesheet or an SVG has none. |
| A local path, read only by an unscoped run | `None`: a file system records no type. |
+| A file of the method tree, named by a relative URL | `None`: a method tree records no type. |
A declared type is returned as declared, even a generic one such as `application/octet-stream`, and is not checked against the bytes. An engine that needs a type keeps its own guess, from the URI's extension or from the bytes, for a file whose source gives none or gives one that says nothing. An HTML engine is the case in point, since a print library such as WeasyPrint keeps a linked stylesheet only when its type is `text/css`, and a stylesheet served from a URL with no extension, as a web font service serves one, can be typed only by what its server declared. The built-in PDF engine reads images only and never looks at the type, because Pillow identifies an image from its bytes.
diff --git a/pipelex/cogt/content_generation/doc_gen_generate.py b/pipelex/cogt/content_generation/doc_gen_generate.py
index 944d8b798..598664a6b 100644
--- a/pipelex/cogt/content_generation/doc_gen_generate.py
+++ b/pipelex/cogt/content_generation/doc_gen_generate.py
@@ -14,6 +14,12 @@ step's data by the interpreter's template file stage (``pipelex.cogt.doc_gen.tem
included, so a value the template reads that does not exist fails the dry run as it fails a live one; an office
template file is handed to its engine, and at a dry print to its engine's template checker.
+A relative URL of an HTML document, such as an image beside its template, names a file of the template's method tree
+(`pipelex.cogt.doc_gen.method_file_urls`): the engine joins it to the job's base URL and reads it through the
+`RenderResources` of the print, which resolves it in the tree's listing and reads the file through the method file
+store, under the run's read scope, and never from a disk, when it is an image, a stylesheet or a font. The listing is
+read the first time the engine asks for such a file, so a document that names none reads nothing more than it did.
+
Engines run on a thread pool of their own rather than the event loop's default executor, because an engine's
thread waits while the loop reads a file for it, and that read may itself need a default-executor thread (a
local file read, a DNS lookup, a cloud storage call). Were the engines on the default executor, as many
@@ -27,9 +33,10 @@ import threading
from concurrent.futures import ThreadPoolExecutor
from typing import TYPE_CHECKING, NamedTuple
+from mthds.method_tree.exceptions import MethodTreeRefusedError
from typing_extensions import override
-from pipelex.base_exceptions import PipelexError
+from pipelex.base_exceptions import PipelexError, PipelexUnexpectedError
from pipelex.cogt.content_generation.assignment_models import RenderDocumentAssignment
from pipelex.cogt.content_generation.dry_mock import dry_render_document
from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory
@@ -37,11 +44,12 @@ from pipelex.cogt.content_generation.read_authorization import authorize_assignm
from pipelex.cogt.doc_gen.doc_gen_engine import get_doc_gen_inference_model, require_doc_gen_engine_installed
from pipelex.cogt.doc_gen.doc_gen_worker_factory import DocGenWorkerFactory
from pipelex.cogt.doc_gen.document_composition import DocumentComposition
-from pipelex.cogt.doc_gen.exceptions import DocGenRenderError
-from pipelex.cogt.doc_gen.render_job import LoadedResource, RenderedDocument, RenderJob, RenderResources
+from pipelex.cogt.doc_gen.exceptions import DocGenRenderError, TemplateUrlRefusedError
+from pipelex.cogt.doc_gen.method_file_urls import RelativeUrlBase, is_relative_url, template_asset_refusal
+from pipelex.cogt.doc_gen.render_job import LoadedResource, RenderedDocument, RenderJob, RenderResources, is_method_file_url
from pipelex.cogt.doc_gen.template_check import TemplateCheckRequest
from pipelex.cogt.inference.inference_call_summary import InferenceCallSummary, InferenceOperation
-from pipelex.core.method_trees.method_files import read_method_file
+from pipelex.core.method_trees.method_files import TreeListing, read_listed_file, read_method_file, read_tree_listing
from pipelex.core.stuffs.document_content import DocumentContent
from pipelex.runtime_hub import get_report_delegate, get_template_file_stage
from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract
@@ -63,19 +71,88 @@ class RunRenderResources(RenderResources):
An engine runs in a worker thread and reads synchronously, while storage and fetching are async, so each
read is handed back to the event loop the print was started from, and the engine's thread waits for it.
+
+ In an HTML document, a file of the method tree is read through the method file store: a URL the engine joined
+ under the job's base URL, and a relative reference the engine hands as written. Both resolve in the tree of the
+ document's template (`document.url_base`), are refused when it has none, and are read only when they name an image,
+ a stylesheet or a font, so a relative URL of an HTML document never reaches this host's disk, nor a method file a
+ run's input could otherwise make the document carry. Any other URL is read as it names, the read scope checked
+ first, a root-absolute local path included. A document with no HTML names no file of a method tree.
"""
- def __init__(self, *, storage_provider: StorageProviderAbstract, read_scope: str | None, loop: asyncio.AbstractEventLoop):
+ def __init__(
+ self,
+ *,
+ storage_provider: StorageProviderAbstract,
+ read_scope: str | None,
+ loop: asyncio.AbstractEventLoop,
+ document: DocumentComposition | None = None,
+ ):
self._storage_provider = storage_provider
self._read_scope = read_scope
self._loop = loop
self._loop_thread_id = threading.get_ident()
+ self._source = document.source if document is not None else None
+ self._prints_html = document is not None and document.source.is_html
+ self._url_base = document.url_base if document is not None else None
+ # The listing of the template's method tree, read the first time the engine asks for one of its files.
+ self._tree_listing: TreeListing | None = None
+ self._tree_listing_lock = asyncio.Lock()
async def _load(self, *, uri: str, position: str) -> LoadedResource:
+ if self._prints_html and (is_method_file_url(url=uri) or is_relative_url(url=uri)):
+ return await self._load_method_file(uri=uri, position=position)
+ if is_method_file_url(url=uri):
+ source_desc = f"printed {self._source.desc}" if self._source is not None else "with no HTML"
+ msg = (
+ f"Refused to read {position}. Its URL names a file of a method, which only the relative URLs of an HTML "
+ f"template can name, and this document is {source_desc}."
+ )
+ raise TemplateUrlRefusedError(msg, refusal_code=None)
authorize_uri_read(uri=uri, read_scope=self._read_scope, position=position)
loaded = await load_bytes_and_mime_type_from_any_uri(uri, storage_provider=self._storage_provider)
return LoadedResource(data=loaded.data, mime_type=loaded.mime_type)
+ async def _load_method_file(self, *, uri: str, position: str) -> LoadedResource:
+ """Read the file of the document's method tree `uri` names, through the method file store under the run's read scope.
+
+ Raises:
+ TemplateUrlRefusedError: the document's template belongs to no method tree, `uri` names no file of it, or
+ the file is not an image, a stylesheet or a font.
+ MethodFileUnavailableError: the method file store does not hold the tree or the file for this run.
+ """
+ url_base = self._url_base
+ if url_base is None:
+ msg = (
+ f"Refused to read {position}. A relative URL names a file of the method the document's template belongs to, "
+ "and this template belongs to no method's folder, such as an inline template of a bundle loaded from a string. "
+ "Load the method from its folder, or name the file by an https:// or a data: URL."
+ )
+ raise TemplateUrlRefusedError(msg, refusal_code=None)
+ listing = await self._read_tree_listing(url_base=url_base)
+ try:
+ tree_path = url_base.tree_path_of(url=uri, tree_paths=listing.files.keys())
+ except MethodTreeRefusedError as exc:
+ msg = f"Refused to read {position}. {exc.message}"
+ raise TemplateUrlRefusedError(msg, refusal_code=exc.refusal.code) from exc
+ if tree_path is None:
+ msg = f"The URL of {position} was taken for a relative URL of the document, and it resolves as none."
+ raise PipelexUnexpectedError(msg)
+ asset_refusal = template_asset_refusal(tree_path=tree_path)
+ if asset_refusal is not None:
+ msg = f"Refused to read {position}. {asset_refusal}"
+ raise TemplateUrlRefusedError(msg, refusal_code=None)
+ data = await read_listed_file(listing=listing, tree_path=tree_path, read_scope=self._read_scope)
+ # A method tree records no media type, so the engine types the file itself, as it does a local file's.
+ return LoadedResource(data=data, mime_type=None)
+
+ async def _read_tree_listing(self, *, url_base: RelativeUrlBase) -> TreeListing:
+ """The listing of the template's method tree, read once per print, whatever number of files the engine asks for."""
+ async with self._tree_listing_lock:
+ if self._tree_listing is None:
+ self._tree_listing = await read_tree_listing(tree_digest=url_base.tree_digest, read_scope=self._read_scope)
+ return self._tree_listing
+
@override
def load(self, *, uri: str, position: str) -> LoadedResource:
if threading.get_ident() == self._loop_thread_id:
@@ -104,7 +181,10 @@ async def render_document_and_store(
Raises:
UriReadRefusedError: an image the document names is outside the run's read scope.
- MethodFileUnavailableError: the template file cannot be read through the method file store for this run.
+ TemplateUrlRefusedError: a relative URL of an HTML document names no image, stylesheet or font of its template's
+ method tree, or the template belongs to no method tree.
+ MethodFileUnavailableError: the template file, or a file a relative URL names, cannot be read through the method
+ file store for this run.
ModelNotFoundError: the assignment's engine is not a model served here.
DocGenEngineMissingError: the engine is the built-in one or one of the plugin's and is no longer declared here, or
no installed plugin registers it.
@@ -206,7 +286,10 @@ async def _print_document(*, render_assignment: RenderDocumentAssignment, storag
Raises:
UriReadRefusedError: an image the document names is outside the run's read scope.
- MethodFileUnavailableError: the template file cannot be read through the method file store for this run.
+ TemplateUrlRefusedError: a relative URL of an HTML document names no image, stylesheet or font of its template's
+ method tree, or the template belongs to no method tree.
+ MethodFileUnavailableError: the template file, or a file a relative URL names, cannot be read through the method
+ file store for this run.
ModelNotFoundError: the assignment's engine is not a model served here.
DocGenEngineMissingError: the engine is the built-in one or one of the plugin's and is no longer declared here, or
no installed plugin registers it.
@@ -244,6 +327,7 @@ async def _print_document(*, render_assignment: RenderDocumentAssignment, storag
storage_provider=storage_provider,
read_scope=render_assignment.job_metadata.run_metadata.read_scope,
loop=asyncio.get_running_loop(),
+ document=composition,
)
# The context is carried into the thread, as `asyncio.to_thread` would, so the engine logs under the run.
run_context = contextvars.copy_context()
diff --git a/pipelex/cogt/doc_gen/doc_gen_format.py b/pipelex/cogt/doc_gen/doc_gen_format.py
index 34c7015a9..44c540c9c 100644
--- a/pipelex/cogt/doc_gen/doc_gen_format.py
+++ b/pipelex/cogt/doc_gen/doc_gen_format.py
@@ -108,6 +108,15 @@ class DocGenSource(StrEnum):
case DocGenSource.TEMPLATE_FILE:
return "from a template file"
+ @property
+ def is_html(self) -> bool:
+ """Whether the engine prints composed HTML, whose relative URLs name files of its template's method tree."""
+ match self:
+ case DocGenSource.HTML:
+ return True
+ case DocGenSource.LAYOUT | DocGenSource.TEMPLATE_FILE:
+ return False
+
@classmethod
def for_step(cls, *, doc_gen_format: DocGenFormat, has_template: bool) -> "DocGenSource":
"""The source a step's format and template make: no template is the layout, a template is HTML for a PDF and a file otherwise."""
diff --git a/pipelex/cogt/doc_gen/document_composition.py b/pipelex/cogt/doc_gen/document_composition.py
index 112458b07..a5089b3c6 100644
--- a/pipelex/cogt/doc_gen/document_composition.py
+++ b/pipelex/cogt/doc_gen/document_composition.py
@@ -5,6 +5,7 @@ from pydantic import BaseModel, ConfigDict, Field, model_validator
from pipelex.cogt.doc_gen.doc_gen_format import DocGenFormat, DocGenSource
from pipelex.cogt.doc_gen.input_shape import InputShape
from pipelex.cogt.doc_gen.layout_tree import LayoutDocument
+from pipelex.cogt.doc_gen.method_file_urls import RelativeUrlBase
from pipelex.cogt.doc_gen.render_job import RenderJob
from pipelex.core.method_trees.tree_bundle import MethodFileRef
from pipelex.tools.uri.uri_read_scope import UriReference
@@ -61,6 +62,11 @@ class DocumentComposition(BaseModel):
engine's `RenderJob` by naming the template file rather than carrying its bytes: reading a method file is the print
stage's IO, through the method file store, so the bytes never ride in the composition across a workflow boundary.
An inline HTML template, whose text is the bundle's own, is rendered by the compose stage and travels as `html`.
+
+ An HTML document's relative URLs, such as an image beside the template, name files of the template's method tree:
+ `url_base` says where the template sits there, the template file or the bundle of an inline template, and the print
+ stage hands the engine its base URL and reads each file the engine asks for from the tree. It is None for a document
+ whose template belongs to no method tree, whose relative URLs then name no file, and for a document with no HTML.
"""
model_config = ConfigDict(extra="forbid")
@@ -73,6 +79,9 @@ class DocumentComposition(BaseModel):
html: str | None = None
template_file: ComposedTemplateFile | None = Field(default=None, description="The template file the print stage reads, renders and prints")
data: dict[str, Any] = Field(default_factory=dict)
+ url_base: RelativeUrlBase | None = Field(
+ default=None, description="Where the HTML template sits in its method tree, which every relative URL of the document resolves against"
+ )
@property
def template_name(self) -> str | None:
@@ -107,14 +116,28 @@ class DocumentComposition(BaseModel):
if self.html is not None and self.template_file is not None:
msg = "A document composed from HTML carries its rendered HTML or the template file the print stage renders, not both."
raise ValueError(msg)
+ self._validate_url_base()
return self
+ def _validate_url_base(self) -> None:
+ """A base for relative URLs only for HTML, and for an HTML template file, the template file's own place in its tree."""
+ if self.url_base is None:
+ return
+ if not self.source.is_html:
+ msg = f"A document composed {self.source.desc} has no HTML, so it carries no base for relative URLs."
+ raise ValueError(msg)
+ if self.template_file is not None:
+ file_ref = self.template_file.file
+ if self.url_base.tree_digest != file_ref.tree_digest or self.url_base.holder != file_ref.tree_path:
+ msg = "A document composed from an HTML template file resolves its relative URLs against that file, in its method tree."
+ raise ValueError(msg)
+
def referenced_uris(self) -> list[UriReference]:
"""The files the print stage will read for the document, which it authorizes against the run's read scope first.
The images of the layout tree. A template file is a method file, read through the method file store under the
run's read scope, and composed HTML may name files too, but only an engine laying it out finds them, and it reads
- each through `RenderResources`, which applies the same check.
+ each through `RenderResources`, which applies the same check, and reads a relative URL from the method tree.
"""
if self.layout is None:
return []
@@ -140,4 +163,5 @@ class DocumentComposition(BaseModel):
template=template,
template_name=self.template_name,
data=self.data,
+ base_url=self.url_base.base_url if self.url_base is not None else None,
)
diff --git a/pipelex/cogt/doc_gen/exceptions.py b/pipelex/cogt/doc_gen/exceptions.py
index 059affbe3..b3b43f050 100644
--- a/pipelex/cogt/doc_gen/exceptions.py
+++ b/pipelex/cogt/doc_gen/exceptions.py
@@ -1,5 +1,7 @@
from enum import StrEnum
+from mthds.method_tree.vocabulary import TreeRefusalCode
+
from pipelex.base_exceptions import ErrorDomain, PipelexError
from pipelex.cogt.doc_gen.doc_gen_format import DocGenFormat, DocGenSource, doc_gen_choice_key
@@ -144,3 +146,32 @@ class MarkdownFormattingBudgetError(PipelexError):
error_domain = ErrorDomain.INPUT
_authors_caller_facing_message = True
_declared_title = "Markdown formatting budget exceeded"
+
+
+class TemplateUrlRefusedError(PipelexError):
+ """A relative URL of an HTML document names no image, stylesheet or font of its template's method tree, so the print reads nothing.
+
+ A relative URL in a `PipeDocGen` step's HTML template, or in a stylesheet of the method tree the template links, names
+ a file of the template's method tree, resolved by the MTHDS standard's rules for a relative URL and looked up in the
+ tree's listing (`pipelex.cogt.doc_gen.method_file_urls`), never on a disk. The print refuses it when the standard does:
+ it is malformed, it is not a tree path, it climbs above the tree's root, or the tree holds no file where it resolves,
+ a directory and a file the tree's membership rule keeps out included. It also refuses a file of the tree that is not
+ an image, a stylesheet or a font, such as the method's Python, bundles, manifest or data, since a URL may come from a
+ run's input and a document can carry what it reads; a URL of a method file in a document with no HTML; and any
+ relative URL of a template that belongs to no method tree, such as an inline template of a bundle loaded from a
+ content string with no file submitted beside it. `refusal_code` is the standard's code for the refusal, and None for
+ these last three, which the standard does not name. A URL the template writes as it is is checked when the method
+ loads, so this is raised for one the run's input supplied.
+
+ `RenderResources.load` raises it, and an engine lets it pass, as it lets the read scope's refusal pass. The method's
+ author or the run's caller fixes it, so it is an input error, and its message names only the method's own files, by
+ their tree paths, and where the URL sat in the document, so it is kept for the caller.
+ """
+
+ error_domain = ErrorDomain.INPUT
+ _authors_caller_facing_message = True
+ _declared_title = "Relative URL refused"
+
+ def __init__(self, message: str, *, refusal_code: TreeRefusalCode | None):
+ super().__init__(message)
+ self.refusal_code = refusal_code
diff --git a/pipelex/cogt/doc_gen/method_file_urls.py b/pipelex/cogt/doc_gen/method_file_urls.py
new file mode 100644
index 000000000..2bee4c3cd
--- /dev/null
+++ b/pipelex/cogt/doc_gen/method_file_urls.py
@@ -0,0 +1,214 @@
+"""Where a relative URL of an HTML document resolves: a file of its template's method tree, named by a URL no network serves.
+
+An HTML template names the files beside it in its method tree by relative URLs: an image's `src`, a stylesheet's `href`,
+and the fonts and images a stylesheet names with `url()`. Each is a file reference in URL context, resolved against the
+tree path of the file that holds it (`pipelex.core.method_trees.url_references`). An engine resolves a relative URL as
+HTML and CSS do, by joining it to the URL of the file that holds it, which is how a stylesheet's own URLs resolve against
+the stylesheet rather than the template. So the print hands the engine a base URL standing for the template's place in
+its tree, `RenderJob.base_url`, under `METHOD_FILE_URL_PREFIX`, whose host no network serves. The engine reads a URL it
+joined under the prefix through `RenderResources.load`, and the print maps it back to the tree path it names, looks that
+path up in the tree's listing and reads the file through the method file store. Nothing is ever read from a disk.
+
+The prefix is followed by the tree's digest, as its 64 hexadecimal digits, then by the holder's tree path, each segment
+percent-encoded. The digest's segment is how a URL that climbed above the root is told from one inside the tree: a join
+removes each `..` it can, so a URL that climbed out lands beside that segment, or above it, and is refused as leaving the
+tree, and so is a root-relative URL such as `/logo.png`, which lands there too. Below the digest's segment, the URL's path
+is a path from the root, resolved by the standard's steps from the root: each segment percent-decoded, the result in NFC,
+a `..` the join left, such as a percent-encoded one, resolved lexically, the result checked to be a tree path and looked
+up. A relative URL an engine hands as written, without joining it, is resolved against the holder by the same steps.
+
+An engine's join applies RFC 3986's removal of dot segments, which parts from the standard's lexical resolution in a few
+corner spellings: Python's `urljoin`, which WeasyPrint uses, drops an empty segment, so `a//b` names `a/b` where the
+standard refuses it, and turns a trailing `/.` into `/`, so `logo.png/.` is refused where the standard resolves it. None
+of these reaches a file outside the tree, since whatever the engine joined, the print reads only a file the tree's listing
+names. A URL that climbs out of the tree and back in through the digest's segment resolves inside the tree as well, and
+the template's author cannot write one: the digest is that of a tree holding the template itself.
+
+Only the kinds of file a printed document is made of are read: images, stylesheets and fonts, told by their suffix
+(`TEMPLATE_ASSET_SUFFIXES`). A relative URL may come from a run's input, interpolated into the template, and a document
+can carry what it reads, an attachment for one, so a URL naming any other file of the method, its Python, its bundles or
+its data, is refused: a caller allowed to run a method is not thereby allowed to read its files.
+
+Only a relative reference names a file of the tree. A root-absolute path, such as `/tmp/photo.png`, is no relative
+reference: an engine joining it to the base URL lands outside the tree's segment, and an engine handing it as written has
+it read as the local path it is, under the run's read scope, as any other URI.
+"""
+
+from collections.abc import Set as AbstractSet
+from pathlib import PurePosixPath
+from urllib.parse import quote
+
+from mthds.method_tree.digest import DIGEST_PREFIX
+from mthds.method_tree.exceptions import MethodTreeRefusedError, TreeRefusal
+from mthds.method_tree.references import ROOT_RESOLVED, is_file_reference
+from mthds.method_tree.vocabulary import ReferenceContext, TreeRefusalCode
+from pydantic import BaseModel, ConfigDict, Field
+
+from pipelex.cogt.doc_gen.render_job import METHOD_FILE_URL_PREFIX, is_method_file_url
+from pipelex.core.method_trees.url_references import resolve_url_path, resolve_url_reference, url_path_of
+
+#: The suffixes of the files a document reads from its method tree, compared without regard to case: images, stylesheets
+#: and fonts, the kinds a printed page is made of.
+TEMPLATE_ASSET_SUFFIXES: frozenset[str] = frozenset(
+ {
+ # Images
+ ".apng",
+ ".avif",
+ ".bmp",
+ ".gif",
+ ".ico",
+ ".jpeg",
+ ".jpg",
+ ".png",
+ ".svg",
+ ".tif",
+ ".tiff",
+ ".webp",
+ # Stylesheets
+ ".css",
+ # Fonts
+ ".eot",
+ ".otf",
+ ".ttc",
+ ".ttf",
+ ".woff",
+ ".woff2",
+ }
+)
+
+
+def is_relative_url(*, url: str) -> bool:
+ """Whether `url` is a relative reference with a path, which in an HTML document names a file of its method tree.
+
+ A URL naming a scheme (`https:`, `data:`, `pipelex-storage:`), one holding an authority (`//host/x`), one with no
+ path (`#top`, `?v=2`, the empty URL) and a root-absolute path (`/tmp/photo.png`) are not: each is read as what it
+ names, under the run's read scope, or left as written.
+ """
+ return is_file_reference(url, context=ReferenceContext.URL) and not url.startswith("/")
+
+
+def template_asset_refusal(*, tree_path: str) -> str | None:
+ """Why a document may not read the file of its method tree at `tree_path`, or None for an image, a stylesheet or a font."""
+ if PurePosixPath(tree_path).suffix.lower() in TEMPLATE_ASSET_SUFFIXES:
+ return None
+ return (
+ f"It names '{tree_path}', which is not an image, a stylesheet or a font: a document reads only those from its method, "
+ "so that a URL a run's input supplies cannot read the method's other files."
+ )
+
+
+class RelativeUrlBase(BaseModel):
+ """Where an HTML document's template sits in its method tree, which every relative URL of the document resolves against.
+
+ The composition of a step printing from an HTML template of a method tree carries it, across the workflow boundary
+ on a distributed orchestrator, so it is plain data: the tree's digest, the holder's tree path, and how a refusal names
+ the template.
+ """
+
+ model_config = ConfigDict(frozen=True, extra="forbid")
+
+ tree_digest: str = Field(description="The digest of the template's method tree, 'sha256:' and 64 lowercase hexadecimal digits")
+ holder: str | None = Field(
+ description=(
+ "The tree path of the file holding the template: the template file, or the bundle of an inline template. None "
+ "for an inline template of bundles parsed from content strings, which sit at the root of the tree of the files "
+ "submitted with them without being a file of it, so their relative URLs resolve from the root"
+ )
+ )
+ template_label: str = Field(description="How a refusal names the template: \"the template file 'templates/letter.html'\"")
+ is_one_file_tree: bool = Field(
+ default=False, description="Whether the tree is the one-file tree of a bundle named alone, which holds no file beside the bundle"
+ )
+
+ @property
+ def base_url(self) -> str:
+ """The URL an engine joins a relative URL of the document to: the holder's place in its tree, under the prefix."""
+ root_url = f"{METHOD_FILE_URL_PREFIX}{self._tree_segment}/"
+ if self.holder is None:
+ return root_url
+ return root_url + "/".join(quote(segment, safe="") for segment in self.holder.split("/"))
+
+ @property
+ def _tree_segment(self) -> str:
+ return self.tree_digest.removeprefix(DIGEST_PREFIX)
+
+ def tree_path_of(self, *, url: str, tree_paths: AbstractSet[str]) -> str | None:
+ """The tree path of the file a URL of the document names, by the standard's steps for a relative URL.
+
+ Args:
+ url: A URL an engine joined under `METHOD_FILE_URL_PREFIX`, or a URL of the document as written, which
+ resolves against the holder.
+ tree_paths: The tree paths of the tree's files, as its listing names them.
+
+ Returns:
+ The tree path the URL names, or None for a URL as written that is no relative URL, such as an `https://`
+ or a `data:` URL, which names no file of the tree.
+
+ Raises:
+ MethodTreeRefusedError: The URL names no file of the tree; `refusal.code` is the standard's code for why,
+ and the message says so naming the template.
+ """
+ if is_method_file_url(url=url):
+ return self._tree_path_of_joined(url=url, tree_paths=tree_paths)
+ return resolve_url_reference(
+ reference=url, holder=self.holder, tree_paths=tree_paths, holder_label=self.template_label, is_one_file_tree=self.is_one_file_tree
+ )
+
+ def _tree_path_of_joined(self, *, url: str, tree_paths: AbstractSet[str]) -> str:
+ """The tree path a URL an engine joined under the prefix names, resolved from the root below the tree's segment."""
+ tree_segment, _separator, below_root = url_path_of(reference=url[len(METHOD_FILE_URL_PREFIX) :]).partition("/")
+ if tree_segment.lower() != self._tree_segment:
+ # A join removes each `..` it can, so a URL that climbed above the root, or one written from it, lands
+ # outside the tree's own segment.
+ msg = f"Its relative URL climbs above the root of the method tree of {self.template_label}, or starts with '/'."
+ raise MethodTreeRefusedError(msg, refusal=TreeRefusal(code=TreeRefusalCode.REFERENCE_LEAVES_TREE, names=[below_root]))
+ try:
+ return resolve_url_path(
+ url_path=below_root, reference=below_root, holder=None, tree_paths=tree_paths, is_one_file_tree=self.is_one_file_tree
+ )
+ except MethodTreeRefusedError as exc:
+ msg = self._joined_refusal_message(refusal=exc.refusal)
+ raise MethodTreeRefusedError(msg, refusal=exc.refusal) from exc
+
+ def _joined_refusal_message(self, *, refusal: TreeRefusal) -> str:
+ """Why a URL an engine joined names no file of the tree, naming what it resolved to, since the URL as written is the engine's."""
+ tree_label = f"the method tree of {self.template_label}"
+ match refusal.code:
+ case TreeRefusalCode.REFERENCE_INVALID:
+ return (
+ f"Its relative URL is malformed in {tree_label}: its path holds a backslash, a '%' not followed by two "
+ "hexadecimal digits, or a percent-escape that decodes to a '/', a backslash, a control character or bytes that are not UTF-8."
+ )
+ case TreeRefusalCode.REFERENCE_LEAVES_TREE:
+ return f"Its relative URL climbs above the root of {tree_label}."
+ case TreeRefusalCode.REFERENCE_NOT_A_TREE_PATH:
+ return f"Its relative URL resolves to '{refusal.resolved}', which cannot name a file of {tree_label}."
+ case TreeRefusalCode.REFERENCE_NOT_IN_TREE:
+ if refusal.resolved == ROOT_RESOLVED:
+ return f"Its relative URL resolves to the root of {tree_label}, which is not a file."
+ return f"Its relative URL resolves to '{refusal.resolved}', which is not a file of {tree_label}."
+ case TreeRefusalCode.REFERENCE_IN_ONE_FILE_TREE:
+ return (
+ f"Its relative URL names a file beside the bundle of {self.template_label}, but the bundle was named alone with "
+ "no METHODS.toml at or above it, so its tree holds that one file. Hand over the method's directory instead."
+ )
+ case (
+ TreeRefusalCode.SYMBOLIC_LINK
+ | TreeRefusalCode.SPECIAL_FILE
+ | TreeRefusalCode.INVALID_TREE_PATH
+ | TreeRefusalCode.NORMALIZATION_COLLISION
+ | TreeRefusalCode.PATH_COLLISION
+ | TreeRefusalCode.EMPTY_TREE
+ | TreeRefusalCode.LONE_FILE_EXCLUDED
+ | TreeRefusalCode.GITIGNORE_NOT_UTF8
+ | TreeRefusalCode.DUPLICATE_KEY
+ | TreeRefusalCode.EXCLUDED_NAME
+ | TreeRefusalCode.MALFORMED_ENTRY
+ | TreeRefusalCode.MISSING_BLOB
+ | TreeRefusalCode.FUNCTION_NAME_WITHOUT_MODULE
+ | TreeRefusalCode.FUNCTION_NAME_INVALID_SEGMENT
+ | TreeRefusalCode.MODULE_NOT_IN_TREE
+ | TreeRefusalCode.MODULE_AMBIGUOUS
+ ):
+ # Resolving a URL's path refuses it only by the reference codes above.
+ return f"Its relative URL names no file of {tree_label}."
diff --git a/pipelex/cogt/doc_gen/render_job.py b/pipelex/cogt/doc_gen/render_job.py
index 93cae10a9..7978a47d4 100644
--- a/pipelex/cogt/doc_gen/render_job.py
+++ b/pipelex/cogt/doc_gen/render_job.py
@@ -10,6 +10,13 @@ a file the document names (an image, a stylesheet), it reads through the `Render
applies the run's read scope and returns the file as a `LoadedResource`, its bytes with the media type its source
gives it. The job never carries a Pipelex object, so an ordinary Pipelex release does not break an
engine; changing this module is a change of the plugin contract, versioned by `PLUGIN_API_VERSION`.
+
+A relative URL in an HTML document names a file of its template's method tree, such as the logo beside the template or
+the font a stylesheet beside it names. An engine resolves it as HTML and CSS resolve one, by joining it to the URL of the
+file that holds it: the job's `base_url` for the document's own URLs, and a stylesheet's URL for the URLs the stylesheet
+holds. That base lies under `METHOD_FILE_URL_PREFIX`, a host no network serves, and an engine reads a URL it joined under
+it through `RenderResources.load`, like any other, which finds the file in the method tree and never on a disk
+(`pipelex.cogt.doc_gen.method_file_urls`).
"""
from typing import Any, Protocol, Self
@@ -19,12 +26,31 @@ from pydantic import BaseModel, ConfigDict, Field, model_validator
from pipelex.cogt.doc_gen.doc_gen_format import DocGenFormat, DocGenSource
from pipelex.cogt.doc_gen.layout_tree import LayoutDocument
+#: The start of every URL that names a file of a document's method tree: the job's `base_url`, and every URL an engine
+#: joins to it, such as a stylesheet's and the URLs it holds. Its host is in the `.invalid` domain, which RFC 2606
+#: reserves for names that never resolve, so no URL under it is ever fetched: an engine hands one to
+#: `RenderResources.load`, which reads the file it names from the method tree.
+METHOD_FILE_URL_PREFIX = "https://method-files.pipelex.invalid/"
+
+
+def is_method_file_url(*, url: str) -> bool:
+ """Whether `url` lies under `METHOD_FILE_URL_PREFIX`, where it names a file of a document's method tree.
+
+ A URL's scheme and host are compared without regard to letter case, as URLs compare them. Every check of the prefix
+ goes through here, so an engine, the job's validation and the reads agree on which URLs it covers.
+ """
+ return url[: len(METHOD_FILE_URL_PREFIX)].lower() == METHOD_FILE_URL_PREFIX
+
class RenderJob(BaseModel):
"""Everything an engine needs to print one document, as plain data.
Exactly one payload is set, the one the source names: `layout` for `DocGenSource.LAYOUT`, `html` for
`DocGenSource.HTML`, and `template` with `data` for `DocGenSource.TEMPLATE_FILE`.
+
+ `base_url` is set only for HTML, when its template belongs to a method tree: an engine joins every relative URL of
+ the document to it, and reads what it joined through `RenderResources.load`. Without it, a relative URL in the
+ document names no file.
"""
model_config = ConfigDict(extra="forbid", ser_json_bytes="base64", val_json_bytes="base64")
@@ -41,6 +67,13 @@ class RenderJob(BaseModel):
default_factory=dict,
description="The step's inputs as plain data, by input name: what a template file is filled with",
)
+ base_url: str | None = Field(
+ default=None,
+ description=(
+ "The URL a relative URL of the HTML is joined to, under METHOD_FILE_URL_PREFIX: where the template sits in its "
+ "method tree. None when the document has no template of a method tree"
+ ),
+ )
def _carries_payload_of(self, *, source: DocGenSource) -> bool:
match source:
@@ -61,6 +94,13 @@ class RenderJob(BaseModel):
if extra_payloads:
msg = f"A render job {self.source.desc} carries only its own payload, not the {', '.join(extra_payloads)} one."
raise ValueError(msg)
+ if self.base_url is not None:
+ if not self.source.is_html:
+ msg = f"A render job {self.source.desc} has no HTML, so it carries no base URL for its relative URLs."
+ raise ValueError(msg)
+ if not is_method_file_url(url=self.base_url):
+ msg = f"A render job's base URL names a file of its method tree, under '{METHOD_FILE_URL_PREFIX}'."
+ raise ValueError(msg)
return self
@@ -87,6 +127,7 @@ class LoadedResource(BaseModel):
content type recorded when the file was stored; the local and in-memory providers identify binary formats from
the bytes and have none for a text format, such as a stylesheet or an SVG.
- **A local path**, which only an unscoped run reads: `None`, since a file system records no type.
+ - **A file of the method tree**, named by a URL under `METHOD_FILE_URL_PREFIX`: `None`, since a tree records none.
A declared type is returned as declared, even a generic one such as `application/octet-stream`, and is not checked
against the bytes. An engine that needs a type, such as an HTML engine that keeps a stylesheet only when it is
@@ -103,18 +144,26 @@ class RenderResources(Protocol):
Pipelex hands one to every render. It resolves `pipelex-storage://` keys through the run's storage
provider, decodes `data:` URLs and fetches `https://` through the SSRF guard, and it refuses what the
- run's read scope does not allow, a local path included. It is synchronous, since engines are, and it
- may be called from the thread the engine prints in.
+ run's read scope does not allow, a local path included. A URL under `METHOD_FILE_URL_PREFIX`, which an
+ engine joined to the job's `base_url` or to the URL of a stylesheet read that way, is a file of the
+ document's method tree: it is resolved in the tree by the MTHDS standard's rules for a relative URL and
+ read through the method file store, under the run's read scope, when it is an image, a stylesheet or a
+ font. So is a relative reference of an HTML document an engine hands as written, against the job's
+ `base_url`. Neither ever reaches a disk; a root-absolute local path is read as one, under the read scope. It is synchronous,
+ since engines are, and it may be called from the thread the engine prints in.
"""
def load(self, *, uri: str, position: str) -> LoadedResource:
"""Return the bytes a URI points at, with the media type its source gives them (`LoadedResource`).
Args:
- uri: The URI the document names.
+ uri: The URI the document names, or, for a relative URL, the URL an engine joined it to.
position: Where the URI sits, in words, which a refusal names instead of the URI: "image 2 of the document".
Raises:
UriReadRefusedError: the run's read scope does not allow the URI.
+ TemplateUrlRefusedError: the URI names no image, stylesheet or font of the document's method tree: it
+ climbs above the tree's root, names a file the tree does not hold or one of another kind, or the
+ document has no method tree to resolve it in.
"""
...
diff --git a/pipelex/core/method_trees/method_files.py b/pipelex/core/method_trees/method_files.py
index 1a5660fd7..98639bb50 100644
--- a/pipelex/core/method_trees/method_files.py
+++ b/pipelex/core/method_trees/method_files.py
@@ -74,7 +74,13 @@ async def read_tree_listing(*, tree_digest: str, read_scope: str | None) -> Tree
return TreeListing.parse(tree_digest=tree_digest, listing=listing)
-async def _read_listed_file(*, listing: TreeListing, tree_path: str, read_scope: str | None) -> bytes:
+async def read_listed_file(*, listing: TreeListing, tree_path: str, read_scope: str | None) -> bytes:
+ """The bytes of the file at `tree_path` in a tree whose listing was read, through the method file store and checked.
+
+ Raises:
+ MethodFileUnavailableError: The listing names no file at `tree_path`, the store does not hold its bytes, or it
+ serves bytes the listing does not name.
+ """
sha256 = listing.files.get(tree_path)
if sha256 is None:
msg = f"The tree '{listing.tree_digest}' holds no file '{tree_path}'."
@@ -96,7 +102,7 @@ async def read_method_file(*, file_ref: MethodFileRef, read_scope: str | None) -
MethodFileUnavailableError: The store does not hold the tree or the file, or serves bytes the tree does not name.
"""
listing = await read_tree_listing(tree_digest=file_ref.tree_digest, read_scope=read_scope)
- return await _read_listed_file(listing=listing, tree_path=file_ref.tree_path, read_scope=read_scope)
+ return await read_listed_file(listing=listing, tree_path=file_ref.tree_path, read_scope=read_scope)
async def read_tree_files(*, tree_digest: str, read_scope: str | None, keeps: Callable[[str], bool] | None = None) -> dict[str, bytes]:
@@ -115,7 +121,7 @@ async def read_tree_files(*, tree_digest: str, read_scope: str | None, keeps: Ca
for tree_path in listing.files:
if keeps is not None and not keeps(tree_path):
continue
- files[tree_path] = await _read_listed_file(listing=listing, tree_path=tree_path, read_scope=read_scope)
+ files[tree_path] = await read_listed_file(listing=listing, tree_path=tree_path, read_scope=read_scope)
return files
@@ -151,6 +157,6 @@ async def read_stored_trees(*, tree_digests: Iterable[str], read_scope: str | No
blobs: dict[str, bytes] = {}
for tree_path, sha256 in listing.files.items():
if sha256 not in blobs:
- blobs[sha256] = await _read_listed_file(listing=listing, tree_path=tree_path, read_scope=read_scope)
+ blobs[sha256] = await read_listed_file(listing=listing, tree_path=tree_path, read_scope=read_scope)
stored_trees.append(StoredTree(tree_digest=tree_digest, listing=listing_bytes, blobs=blobs))
return stored_trees
diff --git a/pipelex/core/method_trees/url_references.py b/pipelex/core/method_trees/url_references.py
new file mode 100644
index 000000000..b3f2e7441
--- /dev/null
+++ b/pipelex/core/method_trees/url_references.py
@@ -0,0 +1,185 @@
+"""Resolving a relative URL of a template, a file reference in URL context, in the set of its method tree's paths.
+
+A relative URL in an HTML template, such as an image's `src` or a stylesheet's `href`, names a file of the template's
+own method tree, resolved against the directory of the file that holds it by the MTHDS standard's steps
+(`mthds/docs/spec/method-tree.md`, "File References"): take the URL's path, dropping its fragment and then its query;
+check its form; split it at each `/` and percent-decode each segment; normalize it to NFC; join it to the holder's
+directory and resolve its `.` and `..` segments lexically; check the result is a tree path; look it up. The first step
+that fails decides the refusal. A URL naming a scheme, holding an authority or with no path is no file reference: it is
+left exactly as written, never resolved and never refused.
+
+`mthds-python` implements these steps as `resolve_reference`, for a tree listed from a directory or read from a map of
+file entries, whose holder is a file of the tree. The print of a document knows less than that, and resolves more:
+
+- **A listing.** The print reads the tree through the method file store, whose listing names each file's hash and tree
+ path, and nothing else of where the tree came from, so a reference is looked up in a set of tree paths.
+- **A holder at the root.** Bundles parsed from content strings sit at the root of the tree of the files submitted with
+ them without being a file of it, and the path below the root of a URL an engine joined to the print's base URL is a
+ path from the root (`pipelex.cogt.doc_gen.method_file_urls`): both resolve from the root.
+
+So the steps are run here, over a set of tree paths, with `mthds-python`'s own pieces for each of them, its pattern of a
+malformed escape included, but the URL path's split-then-decode, which it keeps private and which is written again below
+to the same rules. A test holds this module to `resolve_reference`'s answers for every reference a holder of the tree
+writes, in a tree of several files and in the one-file tree of a bundle named alone. A listing records no membership
+rule, so a file the rule keeps out is refused here as a file the tree does not hold, without the rule that kept it out.
+
+Resolving a reference is a lookup in the tree's set of paths, never a file system access: nothing here opens a path.
+"""
+
+import unicodedata
+from collections.abc import Set as AbstractSet
+from urllib.parse import unquote
+
+from mthds.method_tree.exceptions import MethodTreeRefusedError, TreeRefusal
+from mthds.method_tree.references import MALFORMED_PERCENT_ESCAPE, ROOT_RESOLVED, is_file_reference, lexical_join
+from mthds.method_tree.tree_path import CONTROL_CATEGORY, nfc, tree_path_violation
+from mthds.method_tree.vocabulary import ReferenceContext, TreeRefusalCode
+
+#: The holder a path is joined to when it resolves from the root: any lone segment, since a join keeps the holder's
+#: directories and drops its name.
+_ROOT_HOLDER = "."
+
+
+def url_path_of(*, reference: str) -> str:
+ """A URL's path as written: what is left once its fragment, then its query, are dropped.
+
+ `urllib.parse.urlsplit` is not used: it silently drops tabs and line breaks, which the standard keeps so that the
+ result check can refuse them.
+ """
+ return reference.split("#", 1)[0].split("?", 1)[0]
+
+
+def resolve_url_reference(
+ *, reference: str, holder: str | None, tree_paths: AbstractSet[str], holder_label: str | None = None, is_one_file_tree: bool = False
+) -> str | None:
+ """The tree path of the file a relative URL written in `holder` names, by the standard's steps in URL context.
+
+ Args:
+ reference: The URL exactly as written.
+ holder: The tree path of the file holding the URL, or None for a URL that resolves from the root.
+ tree_paths: The tree paths of the tree's files.
+ holder_label: How a refusal names the holder, such as "the template file 'templates/letter.html'"; its tree
+ path when None.
+ is_one_file_tree: Whether the tree is the one-file tree of a bundle named alone, with no `METHODS.toml` at or
+ above it, whose file a reference to any other file cannot reach.
+
+ Returns:
+ The tree path the URL names, or None for a URL that is no file reference, such as `https://…`, `data:…`,
+ `//host/x` or `#top`, which the caller leaves exactly as written.
+
+ Raises:
+ MethodTreeRefusedError: The URL is refused; `refusal.code` says at which step, and `refusal.resolved` holds what
+ it resolved to from the join on.
+ """
+ if not is_file_reference(reference, context=ReferenceContext.URL):
+ return None
+ return resolve_url_path(
+ url_path=url_path_of(reference=reference),
+ reference=reference,
+ holder=holder,
+ tree_paths=tree_paths,
+ holder_label=holder_label,
+ is_one_file_tree=is_one_file_tree,
+ )
+
+
+def resolve_url_path(
+ *,
+ url_path: str,
+ reference: str,
+ holder: str | None,
+ tree_paths: AbstractSet[str],
+ holder_label: str | None = None,
+ is_one_file_tree: bool = False,
+) -> str:
+ """The tree path a URL's path names from `holder`, its query and fragment already dropped: steps 2 to 7.
+
+ A URL's path is always a file reference, whatever it looks like, so it is resolved without first asking whether it
+ names a scheme: an engine that joined a URL to the print's base URL has already told a reference from an absolute
+ URL, and what is left below the root may hold a colon, which the result check refuses.
+
+ Args:
+ url_path: The URL's path, as written or as an engine joined it.
+ reference: How a refusal names the URL.
+ holder: The tree path of the file holding the URL, or None for a path that resolves from the root.
+ tree_paths: The tree paths of the tree's files.
+ holder_label: How a refusal names the holder; its tree path when None.
+ is_one_file_tree: Whether the tree is the one-file tree of a bundle named alone.
+
+ Raises:
+ MethodTreeRefusedError: The path is refused, `refusal.code` saying at which step.
+ """
+ if holder_label is None:
+ holder_label = f"'{holder}'" if holder is not None else "the root of its method tree"
+ if url_path.startswith("/") or "\\" in url_path:
+ raise _refusal(
+ reference=reference,
+ holder_label=holder_label,
+ code=TreeRefusalCode.REFERENCE_INVALID,
+ explanation="starts with '/' or holds a backslash.",
+ )
+ decoded = nfc(_percent_decoded(url_path=url_path, reference=reference, holder_label=holder_label))
+ segments = lexical_join(holder=holder or _ROOT_HOLDER, path=decoded)
+ if segments is None:
+ explanation = "climbs above the root of its method tree."
+ raise _refusal(reference=reference, holder_label=holder_label, code=TreeRefusalCode.REFERENCE_LEAVES_TREE, explanation=explanation)
+ # The root itself, with no segment or with only a trailing '/', passes the result check and is refused at the lookup.
+ is_root = segments in ([], [""])
+ resolved = ROOT_RESOLVED if is_root else "/".join(segments)
+ if not is_root:
+ violation = tree_path_violation(resolved)
+ if violation is not None:
+ explanation = f"resolves to '{resolved}', which is not a tree path: {violation}."
+ raise _refusal(
+ reference=reference,
+ holder_label=holder_label,
+ code=TreeRefusalCode.REFERENCE_NOT_A_TREE_PATH,
+ explanation=explanation,
+ resolved=resolved,
+ )
+ if resolved in tree_paths:
+ return resolved
+ if is_one_file_tree:
+ explanation = (
+ "names a file beside the bundle, but the bundle was named alone with no METHODS.toml at or above it, "
+ "so its tree holds that one file. Hand over the method's directory instead."
+ )
+ raise _refusal(reference=reference, holder_label=holder_label, code=TreeRefusalCode.REFERENCE_IN_ONE_FILE_TREE, explanation=explanation)
+ what = "the root of the tree, not a file" if is_root else f"'{resolved}', which is not a file of the tree"
+ explanation = f"resolves to {what}."
+ raise _refusal(
+ reference=reference, holder_label=holder_label, code=TreeRefusalCode.REFERENCE_NOT_IN_TREE, explanation=explanation, resolved=resolved
+ )
+
+
+def _percent_decoded(*, url_path: str, reference: str, holder_label: str) -> str:
+ """Step 3: the path split at each `/`, each segment percent-decoded, so decoding never creates a separator.
+
+ Raises:
+ MethodTreeRefusedError: A segment holds a `%` not followed by two hexadecimal digits, or decodes to bytes that
+ are not UTF-8, to a `/`, a backslash or a control character.
+ """
+ decoded_segments: list[str] = []
+ for segment in url_path.split("/"):
+ if MALFORMED_PERCENT_ESCAPE.search(segment):
+ explanation = "holds a '%' not followed by two hexadecimal digits; write a literal '%' as '%25'."
+ raise _refusal(reference=reference, holder_label=holder_label, code=TreeRefusalCode.REFERENCE_INVALID, explanation=explanation)
+ try:
+ decoded_segment = unquote(segment, encoding="utf-8", errors="strict")
+ except UnicodeDecodeError as exc:
+ explanation = "percent-decodes to bytes that are not UTF-8."
+ raise _refusal(reference=reference, holder_label=holder_label, code=TreeRefusalCode.REFERENCE_INVALID, explanation=explanation) from exc
+ if "/" in decoded_segment or "\\" in decoded_segment or _control_count(text=decoded_segment) > _control_count(text=segment):
+ explanation = "percent-decodes to a '/', a backslash or a control character, which no segment may hold."
+ raise _refusal(reference=reference, holder_label=holder_label, code=TreeRefusalCode.REFERENCE_INVALID, explanation=explanation)
+ decoded_segments.append(decoded_segment)
+ return "/".join(decoded_segments)
+
+
+def _control_count(*, text: str) -> int:
+ return sum(1 for character in text if unicodedata.category(character) == CONTROL_CATEGORY)
+
+
+def _refusal(*, reference: str, holder_label: str, code: TreeRefusalCode, explanation: str, resolved: str | None = None) -> MethodTreeRefusedError:
+ msg = f"The relative URL '{reference}' in {holder_label} {explanation}"
+ return MethodTreeRefusedError(msg, refusal=TreeRefusal(code=code, names=[reference], resolved=resolved))
diff --git a/pipelex/pipe_operators/doc_gen/pipe_doc_gen.py b/pipelex/pipe_operators/doc_gen/pipe_doc_gen.py
index b593dfb99..2e065d513 100644
--- a/pipelex/pipe_operators/doc_gen/pipe_doc_gen.py
+++ b/pipelex/pipe_operators/doc_gen/pipe_doc_gen.py
@@ -10,11 +10,12 @@ from pipelex.cogt.doc_gen.doc_gen_setting import DocGenModelChoice, DocGenSettin
from pipelex.cogt.doc_gen.document_composition import ComposedTemplateFile, DocumentComposition, TemplateRenderContext
from pipelex.cogt.doc_gen.input_shape import InputShape, shape_of_input
from pipelex.cogt.doc_gen.layout_builder import build_layout_document, humanize
+from pipelex.cogt.doc_gen.method_file_urls import RelativeUrlBase
from pipelex.cogt.doc_gen.plain_data import plain_data
from pipelex.cogt.models.model_deck_check import check_doc_gen_choice_with_deck
from pipelex.cogt.templating.template_preprocessor import rewrite_template_sigils
from pipelex.core.memory.working_memory import WorkingMemory
-from pipelex.core.method_trees.tree_bundle import MethodFileRef
+from pipelex.core.method_trees.tree_bundle import MethodFileRef, TreeBundleRef
from pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecs
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.core.stuffs.document_content import DocumentContent
@@ -58,6 +59,10 @@ class PipeDocGen(PipeOperator[PipeDocGenOutput]):
format and source, checked when the method loads. A dry run composes for real, which checks the templates against
mock inputs; its print reads and renders the template file and runs the engine's template checker on an office
one, and prints and stores nothing.
+
+ A relative URL in an HTML template, such as the `src` of a logo beside it, names a file of the template's method
+ tree, resolved against the template file, or against the bundle of an inline template: the composition says where
+ the template sits in its tree, and the print stage reads each file the engine asks for from the tree.
"""
type: Literal["PipeDocGen"] = "PipeDocGen"
@@ -68,6 +73,17 @@ class PipeDocGen(PipeOperator[PipeDocGenOutput]):
template_file_ref: MethodFileRef | None = Field(
default=None, description="The template file as a file of the step's method tree, which the print stage reads"
)
+ template_bundle_ref: TreeBundleRef | None = Field(
+ default=None,
+ description=(
+ "The bundle of the inline HTML template, as its method tree names it, which the template's relative URLs resolve "
+ "against; None when the bundle belongs to no method tree, or the step has no inline template"
+ ),
+ )
+ template_tree_is_one_file: bool = Field(
+ default=False,
+ description="Whether the inline template's bundle was named alone, so its method tree holds no file beside it",
+ )
filename: str | None = None
# The text of an HTML template file, read when the step was built in a method tree this process holds, for the checks
@@ -266,14 +282,27 @@ class PipeDocGen(PipeOperator[PipeDocGenOutput]):
label="template",
templating_style=resolve_templating_style(authored=None),
)
- return DocumentComposition(format=self.doc_gen_format, source=source, filename=filename, title=title, html=html)
+ return DocumentComposition(
+ format=self.doc_gen_format,
+ source=source,
+ filename=filename,
+ title=title,
+ html=html,
+ url_base=self._inline_template_url_base(),
+ )
+ template_file = self._composed_template_file(
+ render_context=self._render_context(working_memory=working_memory, pipe_run_params=pipe_run_params)
+ )
return DocumentComposition(
format=self.doc_gen_format,
source=source,
filename=filename,
title=title,
- template_file=self._composed_template_file(
- render_context=self._render_context(working_memory=working_memory, pipe_run_params=pipe_run_params)
+ template_file=template_file,
+ url_base=RelativeUrlBase(
+ tree_digest=template_file.file.tree_digest,
+ holder=template_file.file.tree_path,
+ template_label=f"the template file '{template_file.file.tree_path}'",
),
)
case DocGenSource.TEMPLATE_FILE:
@@ -286,6 +315,22 @@ class PipeDocGen(PipeOperator[PipeDocGenOutput]):
data={name: plain_data(content) for name, content in named_contents},
)
+ def _inline_template_url_base(self) -> RelativeUrlBase | None:
+ """Where the inline template's relative URLs resolve: against its bundle, in the bundle's method tree, when it has one."""
+ bundle_ref = self.template_bundle_ref
+ if bundle_ref is None:
+ return None
+ if bundle_ref.tree_path is None:
+ template_label = f"the inline template of PipeDocGen '{self.code}', in bundles loaded from content strings"
+ else:
+ template_label = f"the inline template of PipeDocGen '{self.code}', in '{bundle_ref.tree_path}'"
+ return RelativeUrlBase(
+ tree_digest=bundle_ref.tree_digest,
+ holder=bundle_ref.tree_path,
+ template_label=template_label,
+ is_one_file_tree=self.template_tree_is_one_file,
+ )
+
def _composed_template_file(
self, *, render_context: TemplateRenderContext | None = None, input_shapes: dict[str, InputShape] | None = None
) -> ComposedTemplateFile:
diff --git a/pipelex/pipe_operators/doc_gen/pipe_doc_gen_factory.py b/pipelex/pipe_operators/doc_gen/pipe_doc_gen_factory.py
index 9f3ab24bc..9a18bcd00 100644
--- a/pipelex/pipe_operators/doc_gen/pipe_doc_gen_factory.py
+++ b/pipelex/pipe_operators/doc_gen/pipe_doc_gen_factory.py
@@ -1,6 +1,8 @@
from typing import Any
from mthds.method_tree.exceptions import MethodTreeRefusedError
+from mthds.method_tree.references import is_file_reference
+from mthds.method_tree.vocabulary import ReferenceContext
from typing_extensions import override
from pipelex.core.concepts.native.concept_native import NativeConceptCode
@@ -15,6 +17,7 @@ from pipelex.pipe_machinery.pipe_factory import PipeFactoryProtocol
from pipelex.pipe_operators.doc_gen.exceptions import PipeDocGenFactoryError
from pipelex.pipe_operators.doc_gen.pipe_doc_gen import PipeDocGen
from pipelex.pipe_operators.doc_gen.pipe_doc_gen_blueprint import PipeDocGenBlueprint, template_required_roots
+from pipelex.pipe_operators.doc_gen.template_asset_check import find_template_asset_refusal, literal_template_urls
from pipelex.tools.jinja2.template_category import TemplateCategory
@@ -42,7 +45,11 @@ class PipeDocGenFactory(PipeFactoryProtocol[PipeDocGenBlueprint, PipeDocGen]):
raise PipeDocGenFactoryError(msg)
template_file_ref: MethodFileRef | None = None
+ template_bundle_ref: TreeBundleRef | None = None
+ template_tree_is_one_file = False
template_file_text: str | None = None
+ if blueprint.template is not None:
+ template_bundle_ref, template_tree_is_one_file = cls._inline_template_tree(pipe_code=pipe_code, template=blueprint.template)
if blueprint.template_file is not None:
template_file = blueprint.template_file
match get_building_tree_bundle():
@@ -61,6 +68,13 @@ class PipeDocGenFactory(PipeFactoryProtocol[PipeDocGenBlueprint, PipeDocGen]):
data=cls._read_tree_file(tree_bundle=tree_bundle, tree_path=resolved_tree_path),
)
cls._check_html_template_file(pipe_code=pipe_code, template=template_file_text, inputs=inputs, template_file=template_file)
+ cls._check_template_assets(
+ pipe_code=pipe_code,
+ template=template_file_text,
+ template_label=f"the template file '{resolved_tree_path}'",
+ holder=resolved_tree_path,
+ tree_bundle=tree_bundle,
+ )
case TreeBundleRef() as tree_bundle_ref:
# A step built from a crate reads nothing: the crate names its tree by digest, and the host that
# assembled it resolved the reference in the tree and checked the file. The print stage reads it.
@@ -86,12 +100,68 @@ class PipeDocGenFactory(PipeFactoryProtocol[PipeDocGenBlueprint, PipeDocGen]):
template=blueprint.template,
template_file=blueprint.template_file,
template_file_ref=template_file_ref,
+ template_bundle_ref=template_bundle_ref,
+ template_tree_is_one_file=template_tree_is_one_file,
filename=blueprint.filename,
)
if template_file_text is not None:
pipe_doc_gen.hold_template_file_text(template_file_text=template_file_text)
return pipe_doc_gen
+ @classmethod
+ def _inline_template_tree(cls, *, pipe_code: str, template: str) -> tuple[TreeBundleRef | None, bool]:
+ """The bundle of an inline template, as its method tree names it, which its relative URLs resolve against.
+
+ Returns the bundle's reference, None for bundles parsed from content strings with no file submitted beside them,
+ and whether its tree is the one-file tree of a bundle named alone. The URLs the template writes as they are are
+ checked now in a tree this process holds, and refused in none, where a relative URL names no file; the rest are
+ the rendered HTML's, resolved in the tree when the print's engine asks for each.
+
+ Raises:
+ PipeDocGenFactoryError: A literal relative URL of the template names no image, stylesheet or font of its tree.
+ """
+ match get_building_tree_bundle():
+ case TreeBundle() as tree_bundle:
+ cls._check_template_assets(
+ pipe_code=pipe_code,
+ template=template,
+ template_label=f"the inline template of PipeDocGen '{pipe_code}'",
+ holder=tree_bundle.tree_path,
+ tree_bundle=tree_bundle,
+ )
+ return tree_bundle.make_ref(), tree_bundle.tree.is_one_file
+ case TreeBundleRef() as tree_bundle_ref:
+ return tree_bundle_ref, False
+ case None:
+ for url in literal_template_urls(template=template):
+ if is_file_reference(url, context=ReferenceContext.URL):
+ msg = (
+ f"PipeDocGen '{pipe_code}' names '{url}' by a relative URL in its inline template, but its bundle was "
+ "loaded from a string with no file sent beside it, so it belongs to no method tree and a relative URL "
+ "names no file. Load the method from its folder, or name the file by an https:// or a data: URL."
+ )
+ raise PipeDocGenFactoryError(msg)
+ return None, False
+
+ @classmethod
+ def _check_template_assets(cls, *, pipe_code: str, template: str, template_label: str, holder: str | None, tree_bundle: TreeBundle) -> None:
+ """Check that each literal relative URL of an HTML template names an image, a stylesheet or a font of its tree.
+
+ Raises:
+ PipeDocGenFactoryError: One names no such file; the message names the URL, where it is written and why.
+ """
+ refusal = find_template_asset_refusal(
+ template=template,
+ template_label=template_label,
+ holder=holder,
+ tree=tree_bundle.tree,
+ read_file=lambda tree_path: cls._read_tree_file(tree_bundle=tree_bundle, tree_path=tree_path),
+ )
+ if refusal is None:
+ return
+ msg = f"PipeDocGen '{pipe_code}' names '{refusal.reference}' in {refusal.holder_label}, which its method cannot give it. {refusal.reason}"
+ raise PipeDocGenFactoryError(msg)
+
@classmethod
def _resolve_template_file(cls, *, pipe_code: str, template_file: str, tree_bundle: TreeBundle) -> str:
"""The tree path of the template file, resolved in the bundle's method tree.
diff --git a/pipelex/pipe_operators/doc_gen/template_asset_check.py b/pipelex/pipe_operators/doc_gen/template_asset_check.py
new file mode 100644
index 000000000..220d0b8aa
--- /dev/null
+++ b/pipelex/pipe_operators/doc_gen/template_asset_check.py
@@ -0,0 +1,170 @@
+"""The files an HTML template names by literal relative URLs, checked in its method tree when the method loads.
+
+A relative URL in an HTML template names an image, a stylesheet or a font of the template's method tree, which the print
+reads (`pipelex.cogt.doc_gen.method_file_urls`). A URL the template writes as it is, such as `<img src="logo.png">`, is
+the method's own, so it is checked when the method loads, where a missing or misspelt file is the author's to fix,
+rather than at its first live print: each is resolved in the tree by the MTHDS standard's rules, must name an image, a
+stylesheet or a font, and a stylesheet's own URLs are checked against the stylesheet, from its bytes in the tree.
+
+Only literal URLs are checked. A value holding a template expression or a sigil, such as `src="{{ photo.url }}"` or
+`href="$link"`, is the run's, and is checked when the document prints. What is read is what a printer reads from HTML:
+an image's `src`, an embedded object, a linked stylesheet or attachment, an SVG image, and every `url()` and `@import`
+of a stylesheet, an inline one or a `style` attribute included. A Jinja comment is skipped, and so is a CSS comment.
+"""
+
+import re
+from collections.abc import Callable
+from html.parser import HTMLParser
+from typing import NamedTuple
+
+from mthds.method_tree.exceptions import MethodTreeRefusedError
+from mthds.method_tree.models import MethodTree
+from mthds.method_tree.references import is_file_reference, resolve_reference
+from mthds.method_tree.vocabulary import ReferenceContext
+from typing_extensions import override
+
+from pipelex.cogt.doc_gen.method_file_urls import template_asset_refusal
+from pipelex.core.method_trees.url_references import resolve_url_reference
+
+#: A Jinja comment, which renders as nothing.
+_JINJA_COMMENT = re.compile(r"\{#.*?#\}", re.DOTALL)
+#: A CSS comment, which names nothing.
+_CSS_COMMENT = re.compile(r"/\*.*?\*/", re.DOTALL)
+#: A `url()` of a stylesheet, quoted or not.
+_CSS_URL = re.compile(r"""url\(\s*(?:"([^"]*)"|'([^']*)'|([^)\s'"]*))\s*\)""", re.IGNORECASE)
+#: An `@import` naming its stylesheet by a quoted string; one written with `url()` is a `url()`.
+_CSS_IMPORT = re.compile(r"""@import\s+(?:"([^"]*)"|'([^']*)')""", re.IGNORECASE)
+#: What marks a value as the run's rather than the template's: a Jinja expression or statement, or a sigil.
+_TEMPLATE_VALUE_MARKERS = ("{", "$", "@")
+#: The suffix of a stylesheet, whose own URLs are checked against it.
+_STYLESHEET_SUFFIX = ".css"
+#: The `rel` tokens of a `<link>` or an `<a>` whose `href` a printer reads.
+_READ_LINK_RELS = frozenset({"stylesheet", "attachment"})
+#: The attributes of each element whose URL a printer reads.
+_URL_ATTRIBUTES: dict[str, tuple[str, ...]] = {
+ "img": ("src",),
+ "embed": ("src",),
+ "object": ("data",),
+ "image": ("href", "xlink:href"),
+}
+
+
+class TemplateAssetRefusal(NamedTuple):
+ """A literal URL of a template that names no image, stylesheet or font of its method tree, and why."""
+
+ reference: str
+ holder_label: str
+ reason: str
+
+
+class _HtmlUrls(HTMLParser):
+ """The URLs an HTML template names that a printer reads, as written."""
+
+ def __init__(self) -> None:
+ super().__init__(convert_charrefs=True)
+ self.urls: list[str] = []
+ self._in_style = False
+
+ @override
+ def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
+ attributes = {name: value for name, value in attrs if value is not None}
+ for attribute in _URL_ATTRIBUTES.get(tag, ()):
+ if attribute in attributes:
+ self.urls.append(attributes[attribute])
+ if tag in {"link", "a"} and "href" in attributes:
+ rels = set(attributes.get("rel", "").lower().split())
+ if rels & _READ_LINK_RELS:
+ self.urls.append(attributes["href"])
+ if "style" in attributes:
+ self.urls.extend(stylesheet_urls(stylesheet=attributes["style"]))
+ if tag == "style":
+ self._in_style = True
+
+ @override
+ def handle_endtag(self, tag: str) -> None:
+ if tag == "style":
+ self._in_style = False
+
+ @override
+ def handle_data(self, data: str) -> None:
+ if self._in_style:
+ self.urls.extend(stylesheet_urls(stylesheet=data))
+
+
+def stylesheet_urls(*, stylesheet: str) -> list[str]:
+ """The URLs a stylesheet names, by `url()` or `@import`, as written, its comments skipped."""
+ text = _CSS_COMMENT.sub("", stylesheet)
+ urls: list[str] = []
+ for pattern in (_CSS_URL, _CSS_IMPORT):
+ for match in pattern.finditer(text):
+ urls.append(next(group for group in match.groups() if group is not None))
+ return urls
+
+
+def literal_template_urls(*, template: str) -> list[str]:
+ """The URLs an HTML template writes as they are, which a printer reads: none holding an expression or a sigil."""
+ parser = _HtmlUrls()
+ parser.feed(_JINJA_COMMENT.sub("", template))
+ parser.close()
+ return [url for url in parser.urls if not any(marker in url for marker in _TEMPLATE_VALUE_MARKERS)]
+
+
+def find_template_asset_refusal(
+ *, template: str, template_label: str, holder: str | None, tree: MethodTree, read_file: Callable[[str], bytes | None]
+) -> TemplateAssetRefusal | None:
+ """The first literal relative URL of an HTML template that names no image, stylesheet or font of its method tree.
+
+ Args:
+ template: The template's text, as written.
+ template_label: How a refusal names the template: "the template file 'templates/invoice.html'".
+ holder: The tree path of the file the template belongs to: the template file, or the bundle of an inline
+ template; None for bundles parsed from content strings, at the root of the tree of the files sent with them.
+ tree: The template's method tree, as this process holds it.
+ read_file: Reads a file of the tree by its tree path, for a stylesheet's own URLs; None when it cannot.
+
+ Returns:
+ The refusal, or None when every literal relative URL names an image, a stylesheet or a font of the tree.
+ """
+ checked_stylesheets: set[str] = set()
+ pending: list[tuple[str, str | None, str]] = [(url, holder, template_label) for url in literal_template_urls(template=template)]
+ while pending:
+ reference, url_holder, holder_label = pending.pop(0)
+ try:
+ tree_path = _resolve(reference=reference, holder=url_holder, tree=tree)
+ except MethodTreeRefusedError as exc:
+ return TemplateAssetRefusal(reference=reference, holder_label=holder_label, reason=exc.message)
+ if tree_path is None:
+ continue
+ asset_refusal = template_asset_refusal(tree_path=tree_path)
+ if asset_refusal is not None:
+ return TemplateAssetRefusal(reference=reference, holder_label=holder_label, reason=asset_refusal)
+ if not tree_path.lower().endswith(_STYLESHEET_SUFFIX) or tree_path in checked_stylesheets:
+ continue
+ checked_stylesheets.add(tree_path)
+ stylesheet = _stylesheet_text(data=read_file(tree_path))
+ if stylesheet is not None:
+ stylesheet_label = f"the stylesheet '{tree_path}'"
+ pending.extend((url, tree_path, stylesheet_label) for url in stylesheet_urls(stylesheet=stylesheet))
+ return None
+
+
+def _resolve(*, reference: str, holder: str | None, tree: MethodTree) -> str | None:
+ """The tree path a URL names from its holder, by `mthds-python` when the holder is a file of the tree.
+
+ Its refusals then name the membership rule that keeps a file out, and the one-file tree of a bundle named alone.
+ """
+ if not is_file_reference(reference, context=ReferenceContext.URL):
+ return None
+ if holder is not None:
+ return resolve_reference(reference, holder=holder, context=ReferenceContext.URL, tree=tree)
+ return resolve_url_reference(reference=reference, holder=None, tree_paths=tree.members, is_one_file_tree=tree.is_one_file)
+
+
+def _stylesheet_text(*, data: bytes | None) -> str | None:
+ """A stylesheet's text, or None when it cannot be read as UTF-8, its own URLs then left to the print."""
+ if data is None:
+ return None
+ try:
+ return data.decode("utf-8")
+ except UnicodeError:
+ return None
diff --git a/pipelex/plugins/contract.py b/pipelex/plugins/contract.py
index a86b8f8e2..b56a3b972 100644
--- a/pipelex/plugins/contract.py
+++ b/pipelex/plugins/contract.py
@@ -65,6 +65,13 @@ if TYPE_CHECKING:
# accept, or read a field that is gone, on its first call; the bump turns that into a discovery-time mismatch naming
# the plugin. ``claim_method_file_store``, the hub slot an activity reads method files through by their tree's digest
# (``MethodFileStoreProtocol`` in ``pipelex.core.method_trees.method_file_store``), joined the menu with it.
+#
+# Relative URLs in an HTML document joined the document engine contract under v7 without a bump, as additions: the
+# render job's ``base_url``, under ``METHOD_FILE_URL_PREFIX`` (``render_job``, with ``is_method_file_url`` to test a URL
+# against it), which an engine joins a relative URL of the document to; the method-file arm of
+# ``RenderResources.load``, which reads an image, a stylesheet or a font joined under the prefix from the document's
+# method tree; and the error that arm raises for a URL naming no such file (``TemplateUrlRefusedError``), which an engine
+# lets pass as it came. An engine written before them ignores ``base_url`` and keeps refusing a relative URL, as it did.
PLUGIN_API_VERSION: int = 7
diff --git a/tests/data/errors/error_identity.txt b/tests/data/errors/error_identity.txt
index 72560a479..46e2030b4 100644
--- a/tests/data/errors/error_identity.txt
+++ b/tests/data/errors/error_identity.txt
@@ -347,6 +347,7 @@ StuffSpecFactoryError | Stuff spec factory | https://docs.pipelex.com/latest/err
TelemetryConfigError | Telemetry config | https://docs.pipelex.com/latest/errors/telemetry-config-error/
TelemetryConfigValidationError | Telemetry config validation | https://docs.pipelex.com/latest/errors/telemetry-config-validation-error/
TemplateSigilSyntaxError | Template sigil syntax | https://docs.pipelex.com/latest/errors/template-sigil-syntax-error/
+TemplateUrlRefusedError | Relative URL refused | https://docs.pipelex.com/latest/errors/template-url-refused-error/
TomlError | TOML parse error | https://docs.pipelex.com/latest/errors/toml-error/
ToolError | Tool error | https://docs.pipelex.com/latest/errors/tool-error/
TraceEventModeError | Trace event mode | https://docs.pipelex.com/latest/errors/trace-event-mode-error/
diff --git a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_template_assets.py b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_template_assets.py
new file mode 100644
index 000000000..453bc086d
--- /dev/null
+++ b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_template_assets.py
@@ -0,0 +1,292 @@
+import copy
+import re
+from html.parser import HTMLParser
+from pathlib import Path
+from urllib.parse import urljoin
+
+import pytest
+from typing_extensions import override
+
+from pipelex.base_exceptions import PipelexError
+from pipelex.cogt.doc_gen.exceptions import TemplateUrlRefusedError
+from pipelex.cogt.doc_gen.render_job import METHOD_FILE_URL_PREFIX, RenderedDocument, RenderJob, RenderResources
+from pipelex.pipeline.exceptions import ValidateBundleError
+from pipelex.pipeline.runner import PipelexMTHDSProtocol
+from pipelex.pipeline.validate_bundle import validate_bundle
+from tests.integration.pipelex.pipes.operator.pipe_doc_gen.doc_gen_helpers import (
+ METHOD_MANIFEST,
+ RecordingMethodFileStore,
+ StubEngine,
+ StubEngines,
+ WorkerLikePipeRun,
+ write_bundle,
+)
+from tests.integration.pipelex.pipes.operator.pipe_doc_gen.test_data import PipeDocGenTestData
+
+_CSS_URL = re.compile(r"""url\(\s*['"]?([^'")\s]+)['"]?\s*\)""")
+_DATA_IMAGE = "data:image/png;base64,iVBORw0KGgo="
+_TEMPLATE = f"""<link rel="stylesheet" href="../styles/print.css">
+<style>h1 {{ background: url('../images/band.png'); }}</style>
+<img src="logo.png"><img src="{_DATA_IMAGE}">
+<h1>{{{{ invoice.number }}}}</h1>
+"""
+_STYLESHEET = '@font-face { font-family: Brand; src: url("../fonts/brand.woff2"); }\nbody { font-family: Brand; }\n'
+_ASSETS = {
+ "templates/logo.png": b"\x89PNG logo",
+ "images/band.png": b"\x89PNG band",
+ "styles/print.css": _STYLESHEET.encode(),
+ "fonts/brand.woff2": b"wOF2 brand",
+}
+
+
+class _ReferencesInHtml(HTMLParser):
+ """The URLs an HTML document names that a printer reads: images, stylesheets, attachments, and the `url()` of its inline styles."""
+
+ def __init__(self) -> None:
+ super().__init__()
+ self.references: list[str] = []
+ self._in_style = False
+
+ @override
+ def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
+ attributes = dict(attrs)
+ match tag:
+ case "img":
+ if attributes.get("src"):
+ self.references.append(attributes["src"] or "")
+ case "link" | "a":
+ if attributes.get("rel") in {"stylesheet", "attachment"} and attributes.get("href"):
+ self.references.append(attributes["href"] or "")
+ case "style":
+ self._in_style = True
+ case _:
+ pass
+
+ @override
+ def handle_endtag(self, tag: str) -> None:
+ if tag == "style":
+ self._in_style = False
+
+ @override
+ def handle_data(self, data: str) -> None:
+ if self._in_style:
+ self.references.extend(_CSS_URL.findall(data))
+
+
+class _AssetReadingEngine(StubEngine):
+ """A stub engine that reads the files its HTML names as an HTML printer does, through the resources it is handed.
+
+ Each URL is joined to the job's base URL, and each URL a stylesheet names to the stylesheet's own URL, as HTML and CSS
+ resolve them; a `data:` URL is the printer's own to decode, and with no base URL a relative URL is handed as written.
+ """
+
+ def __init__(self) -> None:
+ super().__init__()
+ self.read: dict[str, bytes] = {}
+
+ @override
+ def render(self, *, job: RenderJob, resources: RenderResources) -> RenderedDocument:
+ self.jobs.append(job)
+ parser = _ReferencesInHtml()
+ parser.feed(job.html or "")
+ for reference in parser.references:
+ self._read(reference=reference, base_url=job.base_url, resources=resources)
+ return RenderedDocument(data=b"%PDF-stub")
+
+ def _read(self, *, reference: str, base_url: str | None, resources: RenderResources) -> None:
+ if reference.startswith("data:"):
+ return
+ url = urljoin(base_url, reference) if base_url is not None else reference
+ loaded = resources.load(uri=url, position=f"the file '{reference}' the document names")
+ self.read[reference] = loaded.data
+ if reference.endswith(".css"):
+ for css_reference in _CSS_URL.findall(loaded.data.decode()):
+ self._read(reference=css_reference, base_url=url, resources=resources)
+
+
+def _write_files_no_template_may_read(*, method_dir: Path, outside_dir: Path) -> None:
+ """Files on disk a template must never read: one outside the method, a hidden and a gitignored one, and the method's Python."""
+ (outside_dir / "outside.png").write_bytes(b"outside")
+ (method_dir / ".private").mkdir()
+ (method_dir / ".private" / "logo.png").write_bytes(b"hidden")
+ (method_dir / "ignored.png").write_bytes(b"ignored")
+ (method_dir / ".gitignore").write_text("ignored.png\n", encoding="utf-8")
+ (method_dir / "funcs").mkdir()
+ (method_dir / "funcs" / "private_logic.py").write_text("SECRET = 42\n", encoding="utf-8")
+
+
+def _write_method(*, method_dir: Path, template: str = _TEMPLATE) -> Path:
+ """A method folder whose step prints from `templates/invoice.html`, with the files its template names beside it."""
+ method_dir.mkdir(parents=True, exist_ok=True)
+ for tree_path, data in _ASSETS.items():
+ (method_dir / tree_path).parent.mkdir(parents=True, exist_ok=True)
+ (method_dir / tree_path).write_bytes(data)
+ (method_dir / "templates" / "invoice.html").write_text(template, encoding="utf-8")
+ return write_bundle(directory=method_dir, step_fields='format = "pdf"\ntemplate_file = "templates/invoice.html"')
+
+
+async def _run_from_its_folder(*, bundle_path: Path) -> None:
+ await PipelexMTHDSProtocol(bundle_uris=[str(bundle_path)]).execute(
+ mthds_contents=[bundle_path.read_text(encoding="utf-8")], inputs=PipeDocGenTestData.INVOICE_INPUTS
+ )
+
+
[email protected](loop_scope="class")
+class TestPipeDocGenTemplateAssets:
+ @pytest.fixture
+ def engine(self, stub_engines: StubEngines) -> _AssetReadingEngine:
+ stub_engines.engine = _AssetReadingEngine()
+ return stub_engines.engine
+
+ async def test_a_template_file_prints_with_the_image_stylesheet_and_font_beside_it_in_its_method(
+ self, engine: _AssetReadingEngine, tmp_path: Path
+ ) -> None:
+ """Relative URLs resolve against the template, and a stylesheet's against the stylesheet, all read from the method tree."""
+ await _run_from_its_folder(bundle_path=_write_method(method_dir=tmp_path / "method"))
+
+ (job,) = engine.jobs
+ assert job.base_url is not None
+ assert job.base_url.startswith(METHOD_FILE_URL_PREFIX)
+ assert job.base_url.endswith("/templates/invoice.html")
+ assert engine.read == {
+ "../styles/print.css": _ASSETS["styles/print.css"],
+ "../fonts/brand.woff2": _ASSETS["fonts/brand.woff2"],
+ "../images/band.png": _ASSETS["images/band.png"],
+ "logo.png": _ASSETS["templates/logo.png"],
+ }
+ assert _DATA_IMAGE in (job.html or "")
+
+ async def test_a_print_on_a_remote_worker_reads_the_files_through_the_method_file_store(
+ self, engine: _AssetReadingEngine, remote_worker: WorkerLikePipeRun, recording_store: RecordingMethodFileStore, tmp_path: Path
+ ) -> None:
+ """The crate names the tree by digest, and every file the template names is read from the store under the run's read scope."""
+ remote_worker.worker_read_scope = "acme/runs"
+
+ await _run_from_its_folder(bundle_path=_write_method(method_dir=tmp_path / "method"))
+
+ assert remote_worker.received_crates
+ assert engine.read["logo.png"] == _ASSETS["templates/logo.png"]
+ assert engine.read["../fonts/brand.woff2"] == _ASSETS["fonts/brand.woff2"]
+ assert set(recording_store.read_scopes) == {"acme/runs"}
+
+ async def test_an_inline_template_resolves_its_relative_urls_against_its_bundle(self, engine: _AssetReadingEngine, tmp_path: Path) -> None:
+ method_dir = tmp_path / "method"
+ (method_dir / "bundles").mkdir(parents=True)
+ (method_dir / "assets").mkdir()
+ (method_dir / "assets" / "logo.png").write_bytes(b"\x89PNG inline")
+ (method_dir / "METHODS.toml").write_text(METHOD_MANIFEST, encoding="utf-8")
+ step_fields = "format = \"pdf\"\ntemplate = '''<img src=\"../assets/logo.png\"><h1>{{ invoice.number }}</h1>'''"
+ bundle_path = write_bundle(directory=method_dir / "bundles", step_fields=step_fields, as_method=False)
+
+ await _run_from_its_folder(bundle_path=bundle_path)
+
+ (job,) = engine.jobs
+ assert job.base_url is not None
+ assert job.base_url.endswith("/bundles/invoice.mthds")
+ assert engine.read == {"../assets/logo.png": b"\x89PNG inline"}
+
+ @pytest.mark.parametrize(
+ ("reference", "expected"),
+ [
+ pytest.param("../../outside.png", "climbs above the root of its method tree", id="outside_the_tree"),
+ pytest.param("../../../../../../outside.png", "climbs above the root of its method tree", id="far_outside_the_tree"),
+ pytest.param("/outside.png", "starts with '/'", id="root_relative"),
+ pytest.param("missing.png", "'templates/missing.png', which is not a file of the tree", id="absent"),
+ pytest.param("../.private/logo.png", "kept out by a path segment starting with a dot", id="hidden"),
+ pytest.param("../ignored.png", "kept out by a .gitignore pattern", id="gitignored"),
+ pytest.param("../funcs/private_logic.py", "which is not an image, a stylesheet or a font", id="not_an_asset"),
+ ],
+ )
+ async def test_a_literal_url_naming_no_asset_of_the_tree_is_refused_when_the_method_loads(
+ self, engine: _AssetReadingEngine, tmp_path: Path, reference: str, expected: str
+ ) -> None:
+ """A URL the template writes as it is is the method's own, so a missing or forbidden file fails the load, not a print."""
+ method_dir = tmp_path / "method"
+ bundle_path = _write_method(method_dir=method_dir, template=f'<img src="{reference}"><h1>{{{{ invoice.number }}}}</h1>')
+ _write_files_no_template_may_read(method_dir=method_dir, outside_dir=tmp_path)
+
+ with pytest.raises(ValidateBundleError) as exc_info:
+ await validate_bundle(mthds_file_path=bundle_path)
+
+ # The report names the caller's own bundle file as its source; the message names files by their tree paths alone.
+ message = exc_info.value.message
+ assert f"names '{reference}' in the template file 'templates/invoice.html'" in message
+ assert expected in message
+ assert str(tmp_path) not in message
+ assert engine.jobs == []
+
+ @pytest.mark.parametrize(
+ ("element", "reference", "expected"),
+ [
+ pytest.param("img", "../../outside.png", "climbs above the root of the method tree", id="outside_the_tree"),
+ pytest.param("img", "missing.png", "'templates/missing.png', which is not a file", id="absent"),
+ pytest.param("img", "../.private/logo.png", "'.private/logo.png', which is not a file", id="hidden"),
+ pytest.param("img", "../METHODS.toml", "'METHODS.toml', which is not an image, a stylesheet or a font", id="manifest"),
+ pytest.param(
+ "attachment", "../funcs/private_logic.py", "'funcs/private_logic.py', which is not an image, a stylesheet or a font", id="attachment"
+ ),
+ ],
+ )
+ async def test_a_url_a_run_s_input_supplies_reads_only_an_asset_of_the_tree(
+ self, engine: _AssetReadingEngine, tmp_path: Path, element: str, reference: str, expected: str
+ ) -> None:
+ """A URL interpolated from the run's inputs is checked at print: it never reads outside the tree, nor a file that is no asset.
+
+ A printer embeds a `rel="attachment"` link's file in the PDF, so a caller who may run a method but not read it would
+ otherwise take its Python home in the document.
+ """
+ method_dir = tmp_path / "method"
+ tag = '<img src="{{ invoice.customer }}">' if element == "img" else '<a rel="attachment" href="{{ invoice.customer }}">terms</a>'
+ bundle_path = _write_method(method_dir=method_dir, template=f"{tag}<h1>{{{{ invoice.number }}}}</h1>")
+ _write_files_no_template_may_read(method_dir=method_dir, outside_dir=tmp_path)
+ inputs = copy.deepcopy(PipeDocGenTestData.INVOICE_INPUTS)
+ inputs["invoice"]["content"]["customer"] = reference
+
+ with pytest.raises(PipelexError) as exc_info:
+ await PipelexMTHDSProtocol(bundle_uris=[str(bundle_path)]).execute(
+ mthds_contents=[bundle_path.read_text(encoding="utf-8")], inputs=inputs
+ )
+
+ report = str(exc_info.value.to_error_report().model_dump())
+ assert TemplateUrlRefusedError.__name__ in report
+ assert expected in report
+ assert str(tmp_path) not in report
+ assert engine.read == {}
+
+ async def test_a_bundle_named_alone_is_told_to_hand_over_its_folder(self, engine: _AssetReadingEngine, tmp_path: Path) -> None:
+ """With no METHODS.toml at or above it, the bundle's tree is that one file, so the logo beside it is out of reach."""
+ (tmp_path / "logo.png").write_bytes(b"\x89PNG logo")
+ step_fields = "format = \"pdf\"\ntemplate = '''<img src=\"logo.png\"><h1>{{ invoice.number }}</h1>'''"
+ bundle_path = write_bundle(directory=tmp_path, step_fields=step_fields, as_method=False)
+
+ with pytest.raises(ValidateBundleError) as exc_info:
+ await validate_bundle(mthds_file_path=bundle_path)
+
+ report = str(exc_info.value.to_error_report().model_dump())
+ assert "Hand over the method's directory instead" in report
+ assert engine.jobs == []
+
+ async def test_an_inline_template_of_a_bundle_with_no_method_tree_names_no_file(self, engine: _AssetReadingEngine) -> None:
+ """Bundles loaded from content strings with no file submitted beside them have no tree: a literal URL fails the load."""
+ step_fields = "format = \"pdf\"\ntemplate = '''<img src=\"logo.png\"><h1>{{ invoice.number }}</h1>'''"
+
+ with pytest.raises(ValidateBundleError) as exc_info:
+ await validate_bundle(mthds_contents=[PipeDocGenTestData.bundle(step_fields=step_fields)])
+
+ assert "belongs to no method tree" in str(exc_info.value.to_error_report().model_dump())
+ assert engine.jobs == []
+
+ async def test_a_url_a_run_s_input_supplies_names_no_file_when_the_template_has_no_method_tree(self, engine: _AssetReadingEngine) -> None:
+ step_fields = "format = \"pdf\"\ntemplate = '''<img src=\"{{ invoice.customer }}\"><h1>{{ invoice.number }}</h1>'''"
+ inputs = copy.deepcopy(PipeDocGenTestData.INVOICE_INPUTS)
+ inputs["invoice"]["content"]["customer"] = "logo.png"
+
+ with pytest.raises(PipelexError) as exc_info:
+ await PipelexMTHDSProtocol().execute(mthds_contents=[PipeDocGenTestData.bundle(step_fields=step_fields)], inputs=inputs)
+
+ report = str(exc_info.value.to_error_report().model_dump())
+ assert TemplateUrlRefusedError.__name__ in report
+ assert "belongs to no method's folder" in report
+ (job,) = engine.jobs
+ assert job.base_url is None
+ assert engine.read == {}
diff --git a/tests/unit/pipelex/cogt/content_generation/test_doc_gen_render_resources.py b/tests/unit/pipelex/cogt/content_generation/test_doc_gen_render_resources.py
index 44137b8c6..27152703d 100644
--- a/tests/unit/pipelex/cogt/content_generation/test_doc_gen_render_resources.py
+++ b/tests/unit/pipelex/cogt/content_generation/test_doc_gen_render_resources.py
@@ -1,19 +1,262 @@
import asyncio
+from pathlib import Path
+from urllib.parse import urljoin
import pytest
+from mthds.method_tree.vocabulary import TreeRefusalCode
from pytest_mock import MockerFixture
from pipelex.cogt.content_generation import doc_gen_generate
from pipelex.cogt.content_generation.doc_gen_generate import RunRenderResources
-from pipelex.cogt.doc_gen.exceptions import DocGenRenderError
-from pipelex.cogt.doc_gen.render_job import LoadedResource
+from pipelex.cogt.doc_gen.doc_gen_format import DocGenFormat, DocGenSource
+from pipelex.cogt.doc_gen.document_composition import DocumentComposition
+from pipelex.cogt.doc_gen.exceptions import DocGenRenderError, TemplateUrlRefusedError
+from pipelex.cogt.doc_gen.layout_tree import LayoutDocument
+from pipelex.cogt.doc_gen.method_file_urls import RelativeUrlBase
+from pipelex.cogt.doc_gen.render_job import METHOD_FILE_URL_PREFIX, LoadedResource
+from pipelex.core.method_trees.method_file_store import HeldFile, LocalMethodFileStore
from pipelex.tools.storage.storage_provider_abstract import PIPELEX_STORAGE_SCHEME, StorageProviderAbstract, StoredData
from pipelex.tools.uri.exceptions import UriReadRefusedError
from pipelex.tools.uri.uri_bytes import UriBytes
+_METHOD_FILES = {
+ "main.mthds": b'domain = "letters"\n',
+ "templates/letter.html": b'<img src="logo.png">',
+ "templates/logo.png": b"\x89PNG logo",
+ "styles/print.css": b"@font-face { src: url(../fonts/brand.woff2) }",
+ "fonts/brand.woff2": b"wOF2 brand",
+}
+
+
+def _html_document(*, url_base: RelativeUrlBase | None) -> DocumentComposition:
+ return DocumentComposition(
+ format=DocGenFormat.PDF, source=DocGenSource.HTML, filename="letter.pdf", title="Letter", html="<p>Letter</p>", url_base=url_base
+ )
+
+
+def _template_url_base(*, tree_digest: str) -> RelativeUrlBase:
+ return RelativeUrlBase(tree_digest=tree_digest, holder="templates/letter.html", template_label="the template file 'templates/letter.html'")
+
@pytest.mark.asyncio(loop_scope="class")
class TestRunRenderResources:
+ @pytest.fixture
+ def method_store(self, mocker: MockerFixture) -> LocalMethodFileStore:
+ """A method file store of the test's own, as the one the print reads method files through."""
+ store = LocalMethodFileStore()
+ mocker.patch("pipelex.core.method_trees.method_files.get_method_file_store", return_value=store)
+ return store
+
+ async def test_a_url_joined_to_the_base_reads_the_method_file_through_the_store(
+ self, mocker: MockerFixture, method_store: LocalMethodFileStore
+ ) -> None:
+ """The image beside the template, a stylesheet and the font it names are read from the tree, the listing once."""
+ tree = method_store.hold_files(holder="library", files={path: HeldFile.of_bytes(data=data) for path, data in _METHOD_FILES.items()})
+ url_base = _template_url_base(tree_digest=tree.digest)
+ listing_spy = mocker.spy(method_store, "read_tree_listing")
+ loader = mocker.patch.object(doc_gen_generate, "load_bytes_and_mime_type_from_any_uri", new_callable=mocker.AsyncMock)
+ resources = RunRenderResources(
+ storage_provider=mocker.MagicMock(spec=StorageProviderAbstract),
+ read_scope="org/acme",
+ loop=asyncio.get_running_loop(),
+ document=_html_document(url_base=url_base),
+ )
+ logo_url = urljoin(url_base.base_url, "logo.png")
+ stylesheet_url = urljoin(url_base.base_url, "../styles/print.css")
+ font_url = urljoin(stylesheet_url, "../fonts/brand.woff2")
+
+ loaded = [await asyncio.to_thread(resources.load, uri=url, position="a file of the document") for url in (logo_url, stylesheet_url, font_url)]
+
+ assert loaded == [
+ LoadedResource(data=_METHOD_FILES["templates/logo.png"], mime_type=None),
+ LoadedResource(data=_METHOD_FILES["styles/print.css"], mime_type=None),
+ LoadedResource(data=_METHOD_FILES["fonts/brand.woff2"], mime_type=None),
+ ]
+ listing_spy.assert_called_once_with(tree_digest=tree.digest, read_scope="org/acme")
+ loader.assert_not_awaited()
+
+ async def test_a_relative_url_handed_as_written_resolves_against_the_template(
+ self, mocker: MockerFixture, method_store: LocalMethodFileStore
+ ) -> None:
+ tree = method_store.hold_files(holder="library", files={path: HeldFile.of_bytes(data=data) for path, data in _METHOD_FILES.items()})
+ resources = RunRenderResources(
+ storage_provider=mocker.MagicMock(spec=StorageProviderAbstract),
+ read_scope=None,
+ loop=asyncio.get_running_loop(),
+ document=_html_document(url_base=_template_url_base(tree_digest=tree.digest)),
+ )
+
+ loaded = await asyncio.to_thread(resources.load, uri="../fonts/brand.woff2", position="font 1 of the document")
+
+ assert loaded == LoadedResource(data=_METHOD_FILES["fonts/brand.woff2"], mime_type=None)
+
+ @pytest.mark.parametrize(
+ ("written", "code", "named"),
+ [
+ pytest.param("../../secret.txt", TreeRefusalCode.REFERENCE_LEAVES_TREE, "climbs above the root", id="climbs_above_the_root"),
+ pytest.param("/etc/passwd", TreeRefusalCode.REFERENCE_LEAVES_TREE, "starts with '/'", id="root_relative"),
+ pytest.param("missing.png", TreeRefusalCode.REFERENCE_NOT_IN_TREE, "'templates/missing.png', which is not a file", id="absent"),
+ ],
+ )
+ async def test_a_url_naming_no_file_of_the_tree_is_refused_and_nothing_else_is_read(
+ self, mocker: MockerFixture, method_store: LocalMethodFileStore, written: str, code: TreeRefusalCode, named: str
+ ) -> None:
+ tree = method_store.hold_files(holder="library", files={path: HeldFile.of_bytes(data=data) for path, data in _METHOD_FILES.items()})
+ url_base = _template_url_base(tree_digest=tree.digest)
+ loader = mocker.patch.object(doc_gen_generate, "load_bytes_and_mime_type_from_any_uri", new_callable=mocker.AsyncMock)
+ blob_spy = mocker.spy(method_store, "read_blob")
+ resources = RunRenderResources(
+ storage_provider=mocker.MagicMock(spec=StorageProviderAbstract),
+ read_scope=None,
+ loop=asyncio.get_running_loop(),
+ document=_html_document(url_base=url_base),
+ )
+
+ with pytest.raises(TemplateUrlRefusedError) as exc_info:
+ await asyncio.to_thread(resources.load, uri=urljoin(url_base.base_url, written), position="image 1 of the document")
+
+ assert exc_info.value.refusal_code == code
+ assert exc_info.value.message.startswith("Refused to read image 1 of the document.")
+ assert named in exc_info.value.message
+ assert "the template file 'templates/letter.html'" in exc_info.value.message
+ loader.assert_not_awaited()
+ blob_spy.assert_not_called()
+
+ @pytest.mark.parametrize(
+ ("written", "tree_path"),
+ [
+ pytest.param("../main.mthds", "main.mthds", id="bundle"),
+ pytest.param("letter.html", "templates/letter.html", id="template"),
+ ],
+ )
+ async def test_a_file_of_the_tree_that_is_no_asset_is_refused_unread(
+ self, mocker: MockerFixture, method_store: LocalMethodFileStore, written: str, tree_path: str
+ ) -> None:
+ """A URL a run's input supplies may name any file of the method: only an image, a stylesheet or a font is read."""
+ tree = method_store.hold_files(holder="library", files={path: HeldFile.of_bytes(data=data) for path, data in _METHOD_FILES.items()})
+ url_base = _template_url_base(tree_digest=tree.digest)
+ blob_spy = mocker.spy(method_store, "read_blob")
+ resources = RunRenderResources(
+ storage_provider=mocker.MagicMock(spec=StorageProviderAbstract),
+ read_scope=None,
+ loop=asyncio.get_running_loop(),
+ document=_html_document(url_base=url_base),
+ )
+
+ for uri in (urljoin(url_base.base_url, written), written):
+ with pytest.raises(TemplateUrlRefusedError) as exc_info:
+ await asyncio.to_thread(resources.load, uri=uri, position="attachment 1 of the document")
+ assert exc_info.value.refusal_code is None
+ assert f"'{tree_path}', which is not an image, a stylesheet or a font" in exc_info.value.message
+ blob_spy.assert_not_called()
+
+ async def test_a_root_absolute_local_path_in_an_html_print_is_read_as_a_local_path(self, mocker: MockerFixture, tmp_path: Path) -> None:
+ """An absolute path is no relative URL: an unscoped run reads it from its disk, as before, and a scoped run refuses it."""
+ photo_path = tmp_path / "photo.png"
+ photo_path.write_bytes(b"\x89PNG local")
+ storage_provider = mocker.MagicMock(spec=StorageProviderAbstract)
+ unscoped = RunRenderResources(
+ storage_provider=storage_provider, read_scope=None, loop=asyncio.get_running_loop(), document=_html_document(url_base=None)
+ )
+ scoped = RunRenderResources(
+ storage_provider=storage_provider, read_scope="org/acme", loop=asyncio.get_running_loop(), document=_html_document(url_base=None)
+ )
+
+ loaded = await asyncio.to_thread(unscoped.load, uri=str(photo_path), position="image 1 of the document")
+
+ assert loaded.data == b"\x89PNG local"
+ with pytest.raises(UriReadRefusedError):
+ await asyncio.to_thread(scoped.load, uri=str(photo_path), position="image 1 of the document")
+
+ async def test_a_method_file_url_in_a_document_with_no_html_is_refused_for_its_source(self, mocker: MockerFixture) -> None:
+ loader = mocker.patch.object(doc_gen_generate, "load_bytes_and_mime_type_from_any_uri", new_callable=mocker.AsyncMock)
+ layout_document = DocumentComposition(
+ format=DocGenFormat.PDF,
+ source=DocGenSource.LAYOUT,
+ filename="letter.pdf",
+ title="Letter",
+ layout=LayoutDocument(title="Letter", blocks=[]),
+ )
+ resources = RunRenderResources(
+ storage_provider=mocker.MagicMock(spec=StorageProviderAbstract),
+ read_scope=None,
+ loop=asyncio.get_running_loop(),
+ document=layout_document,
+ )
+
+ with pytest.raises(TemplateUrlRefusedError) as exc_info:
+ await asyncio.to_thread(resources.load, uri=f"{METHOD_FILE_URL_PREFIX}{'ab' * 32}/logo.png", position="image 1 of the document")
+
+ assert "this document is printed from the auto-layout of its inputs" in exc_info.value.message
+ assert "inline template" not in exc_info.value.message
+ loader.assert_not_awaited()
+
+ async def test_a_relative_url_of_html_with_no_method_tree_is_refused_and_never_read_from_disk(self, mocker: MockerFixture) -> None:
+ """A template that belongs to no method tree names no file by a relative URL, so no local path is ever opened."""
+ loader = mocker.patch.object(doc_gen_generate, "load_bytes_and_mime_type_from_any_uri", new_callable=mocker.AsyncMock)
+ resources = RunRenderResources(
+ storage_provider=mocker.MagicMock(spec=StorageProviderAbstract),
+ read_scope=None,
+ loop=asyncio.get_running_loop(),
+ document=_html_document(url_base=None),
+ )
+
+ for uri in ("logo.png", "../../etc/passwd", f"{METHOD_FILE_URL_PREFIX}{'ab' * 32}/logo.png"):
+ with pytest.raises(TemplateUrlRefusedError) as exc_info:
+ await asyncio.to_thread(resources.load, uri=uri, position="image 1 of the document")
+ assert exc_info.value.refusal_code is None
+ assert "belongs to no method's folder" in exc_info.value.message
+ loader.assert_not_awaited()
+
+ async def test_a_url_with_a_scheme_in_an_html_print_is_read_as_it_names(self, mocker: MockerFixture, method_store: LocalMethodFileStore) -> None:
+ tree = method_store.hold_files(holder="library", files={path: HeldFile.of_bytes(data=data) for path, data in _METHOD_FILES.items()})
+ loader = mocker.patch.object(
+ doc_gen_generate,
+ "load_bytes_and_mime_type_from_any_uri",
+ new_callable=mocker.AsyncMock,
+ return_value=UriBytes(data=b"\x89PNG remote", mime_type="image/png"),
+ )
+ listing_spy = mocker.spy(method_store, "read_tree_listing")
+ resources = RunRenderResources(
+ storage_provider=mocker.MagicMock(spec=StorageProviderAbstract),
+ read_scope=None,
+ loop=asyncio.get_running_loop(),
+ document=_html_document(url_base=_template_url_base(tree_digest=tree.digest)),
+ )
+
+ loaded = await asyncio.to_thread(resources.load, uri="https://cdn.example.com/logo.png", position="image 1 of the document")
+
+ assert loaded == LoadedResource(data=b"\x89PNG remote", mime_type="image/png")
+ loader.assert_awaited_once()
+ listing_spy.assert_not_called()
+
+ async def test_a_print_with_no_html_reads_a_local_path_as_it_did(self, mocker: MockerFixture) -> None:
+ """A layout's image is a URI the step's data gave, never a file of a method tree, so a local path keeps its arm."""
+ loader = mocker.patch.object(
+ doc_gen_generate,
+ "load_bytes_and_mime_type_from_any_uri",
+ new_callable=mocker.AsyncMock,
+ return_value=UriBytes(data=b"\x89PNG local", mime_type=None),
+ )
+ layout_document = DocumentComposition(
+ format=DocGenFormat.PDF,
+ source=DocGenSource.LAYOUT,
+ filename="letter.pdf",
+ title="Letter",
+ layout=LayoutDocument(title="Letter", blocks=[]),
+ )
+ resources = RunRenderResources(
+ storage_provider=mocker.MagicMock(spec=StorageProviderAbstract),
+ read_scope=None,
+ loop=asyncio.get_running_loop(),
+ document=layout_document,
+ )
+
+ loaded = await asyncio.to_thread(resources.load, uri="images/photo.png", position="image 1 of the document")
+
+ assert loaded == LoadedResource(data=b"\x89PNG local", mime_type=None)
+ loader.assert_awaited_once()
+
async def test_a_read_returns_the_bytes_with_the_type_the_source_gives(self, mocker: MockerFixture) -> None:
"""An engine reading from its worker thread gets the file's bytes and the media type its source gave them."""
storage_uri = f"{PIPELEX_STORAGE_SCHEME}scope/assets/theme"
diff --git a/tests/unit/pipelex/cogt/doc_gen/test_document_composition.py b/tests/unit/pipelex/cogt/doc_gen/test_document_composition.py
index 400b40b44..1e679c030 100644
--- a/tests/unit/pipelex/cogt/doc_gen/test_document_composition.py
+++ b/tests/unit/pipelex/cogt/doc_gen/test_document_composition.py
@@ -4,6 +4,8 @@ from pydantic import ValidationError
from pipelex.cogt.doc_gen.doc_gen_format import DocGenFormat, DocGenSource
from pipelex.cogt.doc_gen.document_composition import ComposedTemplateFile, DocumentComposition, TemplateRenderContext
from pipelex.cogt.doc_gen.layout_tree import ImageBlock, LayoutDocument, SectionBlock
+from pipelex.cogt.doc_gen.method_file_urls import RelativeUrlBase
+from pipelex.cogt.doc_gen.render_job import METHOD_FILE_URL_PREFIX
from pipelex.core.method_trees.tree_bundle import MethodFileRef
_TREE_DIGEST = "sha256:" + "a" * 64
@@ -129,3 +131,44 @@ class TestDocumentComposition:
html="<p>Report</p>",
template_file=_OFFICE_TEMPLATE_FILE,
)
+
+ def test_an_html_document_hands_the_engine_the_base_of_its_template_s_place_in_its_tree(self) -> None:
+ url_base = RelativeUrlBase(
+ tree_digest=_TREE_DIGEST, holder="templates/report.html", template_label="the template file 'templates/report.html'"
+ )
+ composition = DocumentComposition(
+ format=DocGenFormat.PDF,
+ source=DocGenSource.HTML,
+ filename="report.pdf",
+ title="Report",
+ template_file=_HTML_TEMPLATE_FILE,
+ url_base=url_base,
+ )
+
+ job = composition.make_render_job(html="<img src='logo.png'>")
+
+ assert job.base_url == f"{METHOD_FILE_URL_PREFIX}{'a' * 64}/templates/report.html"
+ assert DocumentComposition.model_validate_json(composition.model_dump_json()) == composition
+
+ def test_an_html_document_whose_template_has_no_tree_hands_no_base(self) -> None:
+ composition = DocumentComposition(format=DocGenFormat.PDF, source=DocGenSource.HTML, filename="report.pdf", title="Report", html="<p/>")
+
+ assert composition.make_render_job().base_url is None
+
+ @pytest.mark.parametrize(
+ ("source", "template_file", "holder", "message"),
+ [
+ pytest.param(DocGenSource.TEMPLATE_FILE, _OFFICE_TEMPLATE_FILE, "templates/invoice.docx", "has no HTML", id="no_html"),
+ pytest.param(DocGenSource.HTML, _HTML_TEMPLATE_FILE, "templates/other.html", "against that file", id="not_the_template_file"),
+ ],
+ )
+ def test_a_base_for_relative_urls_is_refused_where_it_cannot_be_the_template_s(
+ self, source: DocGenSource, template_file: ComposedTemplateFile, holder: str, message: str
+ ) -> None:
+ url_base = RelativeUrlBase(tree_digest=_TREE_DIGEST, holder=holder, template_label="the template")
+ doc_gen_format = DocGenFormat.PDF if source.is_html else DocGenFormat.DOCX
+
+ with pytest.raises(ValidationError, match=message):
+ DocumentComposition(
+ format=doc_gen_format, source=source, filename="report", title="Report", template_file=template_file, url_base=url_base
+ )
diff --git a/tests/unit/pipelex/cogt/doc_gen/test_method_file_urls.py b/tests/unit/pipelex/cogt/doc_gen/test_method_file_urls.py
new file mode 100644
index 000000000..31b93d8cb
--- /dev/null
+++ b/tests/unit/pipelex/cogt/doc_gen/test_method_file_urls.py
@@ -0,0 +1,168 @@
+import re
+from urllib.parse import urljoin
+
+import pytest
+from mthds.method_tree.exceptions import MethodTreeRefusedError
+from mthds.method_tree.vocabulary import TreeRefusalCode
+
+from pipelex.cogt.doc_gen.method_file_urls import RelativeUrlBase, is_relative_url, template_asset_refusal
+from pipelex.cogt.doc_gen.render_job import METHOD_FILE_URL_PREFIX, is_method_file_url
+
+_HEX = "ab" * 32
+_DIGEST = f"sha256:{_HEX}"
+_TEMPLATE_BASE = RelativeUrlBase(tree_digest=_DIGEST, holder="templates/letter.html", template_label="the template file 'templates/letter.html'")
+_ROOT_BASE = RelativeUrlBase(tree_digest=_DIGEST, holder=None, template_label="the inline template of PipeDocGen 'print_letter'")
+_TREE_PATHS = frozenset(
+ {
+ "main.mthds",
+ "templates/letter.html",
+ "templates/logo.png",
+ "templates/my logo.png",
+ "styles/print.css",
+ "fonts/brand.woff2",
+ "assets/100%.txt",
+ }
+)
+
+
+class TestMethodFileUrls:
+ @pytest.mark.parametrize(
+ ("holder", "expected"),
+ [
+ pytest.param("templates/letter.html", f"{METHOD_FILE_URL_PREFIX}{_HEX}/templates/letter.html", id="template_file"),
+ pytest.param("my templates/100%.html", f"{METHOD_FILE_URL_PREFIX}{_HEX}/my%20templates/100%25.html", id="segments_percent_encoded"),
+ pytest.param("what?#.html", f"{METHOD_FILE_URL_PREFIX}{_HEX}/what%3F%23.html", id="query_and_fragment_characters_encoded"),
+ pytest.param(None, f"{METHOD_FILE_URL_PREFIX}{_HEX}/", id="root"),
+ ],
+ )
+ def test_the_base_url_stands_for_the_holder_s_place_in_its_tree(self, holder: str | None, expected: str) -> None:
+ url_base = RelativeUrlBase(tree_digest=_DIGEST, holder=holder, template_label="the template")
+
+ assert url_base.base_url == expected
+ assert is_method_file_url(url=url_base.base_url)
+
+ @pytest.mark.parametrize(
+ ("written", "expected"),
+ [
+ pytest.param("logo.png", "templates/logo.png", id="beside_the_template"),
+ pytest.param("../styles/print.css", "styles/print.css", id="dot_dot_within_the_tree"),
+ pytest.param("my%20logo.png", "templates/my logo.png", id="percent_encoded_name"),
+ pytest.param("logo.png?v=2#top", "templates/logo.png", id="query_and_fragment_dropped"),
+ pytest.param("../assets/100%25.txt", "assets/100%.txt", id="encoded_percent"),
+ pytest.param("%2E%2E/fonts/brand.woff2", "fonts/brand.woff2", id="encoded_dot_dot_the_join_kept"),
+ ],
+ )
+ def test_a_url_an_engine_joined_to_the_base_names_the_file_of_the_tree(self, written: str, expected: str) -> None:
+ joined = urljoin(_TEMPLATE_BASE.base_url, written)
+
+ assert _TEMPLATE_BASE.tree_path_of(url=joined, tree_paths=_TREE_PATHS) == expected
+
+ def test_a_url_in_a_stylesheet_resolves_against_the_stylesheet(self) -> None:
+ """An engine joins a stylesheet's own URLs to the stylesheet's URL, which lies in the tree too."""
+ stylesheet_url = urljoin(_TEMPLATE_BASE.base_url, "../styles/print.css")
+ font_url = urljoin(stylesheet_url, "../fonts/brand.woff2")
+
+ assert _TEMPLATE_BASE.tree_path_of(url=stylesheet_url, tree_paths=_TREE_PATHS) == "styles/print.css"
+ assert _TEMPLATE_BASE.tree_path_of(url=font_url, tree_paths=_TREE_PATHS) == "fonts/brand.woff2"
+
+ @pytest.mark.parametrize(
+ ("written", "code", "named"),
+ [
+ pytest.param("../../logo.png", TreeRefusalCode.REFERENCE_LEAVES_TREE, "climbs above the root", id="climbs_above_the_root"),
+ pytest.param("../../../../../etc/passwd", TreeRefusalCode.REFERENCE_LEAVES_TREE, "climbs above the root", id="climbs_far_above"),
+ pytest.param("/logo.png", TreeRefusalCode.REFERENCE_LEAVES_TREE, "starts with '/'", id="root_relative"),
+ pytest.param("%2E%2E/%2E%2E/logo.png", TreeRefusalCode.REFERENCE_LEAVES_TREE, "climbs above the root", id="encoded_climb"),
+ pytest.param("missing.png", TreeRefusalCode.REFERENCE_NOT_IN_TREE, "'templates/missing.png', which is not a file", id="absent"),
+ # A join turns '.' into the directory with a trailing '/', which no file's tree path ends with.
+ pytest.param(".", TreeRefusalCode.REFERENCE_NOT_A_TREE_PATH, "'templates/', which cannot name a file", id="a_directory"),
+ pytest.param("..", TreeRefusalCode.REFERENCE_NOT_IN_TREE, "the root of the method tree", id="the_root"),
+ pytest.param("..%2Flogo.png", TreeRefusalCode.REFERENCE_INVALID, "malformed", id="encoded_slash"),
+ pytest.param("100%.png", TreeRefusalCode.REFERENCE_INVALID, "malformed", id="bare_percent"),
+ ],
+ )
+ def test_a_joined_url_naming_no_file_of_the_tree_is_refused_naming_the_template(self, written: str, code: TreeRefusalCode, named: str) -> None:
+ joined = urljoin(_TEMPLATE_BASE.base_url, written)
+
+ with pytest.raises(MethodTreeRefusedError) as exc_info:
+ _TEMPLATE_BASE.tree_path_of(url=joined, tree_paths=_TREE_PATHS)
+
+ assert exc_info.value.refusal.code == code
+ assert named in exc_info.value.message
+ assert "the template file 'templates/letter.html'" in exc_info.value.message
+
+ def test_a_url_under_another_tree_s_segment_is_refused_as_leaving_the_tree(self) -> None:
+ other_tree_url = f"{METHOD_FILE_URL_PREFIX}{'cd' * 32}/templates/logo.png"
+
+ with pytest.raises(MethodTreeRefusedError) as exc_info:
+ _TEMPLATE_BASE.tree_path_of(url=other_tree_url, tree_paths=_TREE_PATHS)
+
+ assert exc_info.value.refusal.code == TreeRefusalCode.REFERENCE_LEAVES_TREE
+
+ def test_the_prefix_and_the_digest_are_read_without_regard_to_case(self) -> None:
+ shouted = f"{METHOD_FILE_URL_PREFIX.upper()}{_HEX.upper()}/templates/logo.png"
+
+ assert is_method_file_url(url=shouted)
+ assert _TEMPLATE_BASE.tree_path_of(url=shouted, tree_paths=_TREE_PATHS) == "templates/logo.png"
+
+ def test_a_joined_url_beside_a_bundle_named_alone_is_told_to_hand_over_its_folder(self) -> None:
+ url_base = RelativeUrlBase(tree_digest=_DIGEST, holder="letters.mthds", template_label="the inline template", is_one_file_tree=True)
+
+ with pytest.raises(MethodTreeRefusedError) as exc_info:
+ url_base.tree_path_of(url=urljoin(url_base.base_url, "logo.png"), tree_paths=frozenset({"letters.mthds"}))
+
+ assert exc_info.value.refusal.code == TreeRefusalCode.REFERENCE_IN_ONE_FILE_TREE
+ assert "Hand over the method's directory instead" in exc_info.value.message
+
+ @pytest.mark.parametrize(
+ ("tree_path", "is_asset"),
+ [
+ pytest.param("templates/logo.PNG", True, id="image_any_case"),
+ pytest.param("images/seal.svg", True, id="svg"),
+ pytest.param("styles/print.css", True, id="stylesheet"),
+ pytest.param("fonts/brand.woff2", True, id="font"),
+ pytest.param("funcs/private_logic.py", False, id="python"),
+ pytest.param("letters.mthds", False, id="bundle"),
+ pytest.param("METHODS.toml", False, id="manifest"),
+ pytest.param("requirements.txt", False, id="requirements"),
+ pytest.param("data/customers.csv", False, id="data"),
+ pytest.param("templates/other.html", False, id="html"),
+ pytest.param("README", False, id="no_suffix"),
+ ],
+ )
+ def test_a_document_reads_only_images_stylesheets_and_fonts_from_its_method(self, tree_path: str, is_asset: bool) -> None:
+ refusal = template_asset_refusal(tree_path=tree_path)
+
+ assert (refusal is None) is is_asset
+ if refusal is not None:
+ assert f"'{tree_path}', which is not an image, a stylesheet or a font" in refusal
+
+ def test_a_relative_url_handed_as_written_resolves_against_the_holder(self) -> None:
+ assert _TEMPLATE_BASE.tree_path_of(url="../fonts/brand.woff2", tree_paths=_TREE_PATHS) == "fonts/brand.woff2"
+ assert _TEMPLATE_BASE.tree_path_of(url="https://cdn.example.com/logo.png", tree_paths=_TREE_PATHS) is None
+ with pytest.raises(MethodTreeRefusedError, match=re.escape("the template file 'templates/letter.html'")):
+ _TEMPLATE_BASE.tree_path_of(url="../../logo.png", tree_paths=_TREE_PATHS)
+
+ def test_a_root_base_resolves_from_the_root_and_cannot_climb_above_it(self) -> None:
+ """Bundles parsed from content strings sit at the root of the tree of the files submitted with them."""
+ assert _ROOT_BASE.tree_path_of(url=urljoin(_ROOT_BASE.base_url, "templates/logo.png"), tree_paths=_TREE_PATHS) == "templates/logo.png"
+ assert _ROOT_BASE.tree_path_of(url="fonts/brand.woff2", tree_paths=_TREE_PATHS) == "fonts/brand.woff2"
+ with pytest.raises(MethodTreeRefusedError) as exc_info:
+ _ROOT_BASE.tree_path_of(url=urljoin(_ROOT_BASE.base_url, "../logo.png"), tree_paths=_TREE_PATHS)
+ assert exc_info.value.refusal.code == TreeRefusalCode.REFERENCE_LEAVES_TREE
+ assert "the inline template of PipeDocGen 'print_letter'" in exc_info.value.message
+
+ @pytest.mark.parametrize(
+ ("url", "expected"),
+ [
+ pytest.param("logo.png", True, id="relative"),
+ pytest.param("../fonts/brand.woff2", True, id="dot_dot"),
+ pytest.param("/srv/photos/photo.png", False, id="root_absolute_path"),
+ pytest.param("https://cdn.example.com/logo.png", False, id="https"),
+ pytest.param("data:image/png;base64,iVBORw0KGgo=", False, id="data"),
+ pytest.param("pipelex-storage://scope/logo.png", False, id="storage"),
+ pytest.param("//cdn.example.com/logo.png", False, id="authority"),
+ pytest.param("#top", False, id="fragment"),
+ ],
+ )
+ def test_only_a_relative_url_with_a_path_names_a_file_of_the_tree(self, url: str, expected: bool) -> None:
+ assert is_relative_url(url=url) is expected
diff --git a/tests/unit/pipelex/cogt/doc_gen/test_render_job.py b/tests/unit/pipelex/cogt/doc_gen/test_render_job.py
index 2f03e323a..ec7ff7634 100644
--- a/tests/unit/pipelex/cogt/doc_gen/test_render_job.py
+++ b/tests/unit/pipelex/cogt/doc_gen/test_render_job.py
@@ -6,7 +6,7 @@ from pydantic import ValidationError
from pipelex.cogt.doc_gen.doc_gen_format import DocGenFormat, DocGenSource
from pipelex.cogt.doc_gen.layout_tree import LayoutDocument, ParagraphsBlock
-from pipelex.cogt.doc_gen.render_job import RenderJob
+from pipelex.cogt.doc_gen.render_job import METHOD_FILE_URL_PREFIX, RenderJob
def _layout() -> LayoutDocument:
@@ -33,6 +33,36 @@ class TestRenderJob:
layout=_layout(),
)
+ def test_an_html_job_carries_the_base_its_relative_urls_join(self) -> None:
+ base_url = f"{METHOD_FILE_URL_PREFIX}{'a' * 64}/templates/letter.html"
+
+ job = RenderJob(format=DocGenFormat.PDF, source=DocGenSource.HTML, filename="notice.pdf", title="Notice", html="<p/>", base_url=base_url)
+
+ assert job.base_url == base_url
+ assert RenderJob.model_validate_json(job.model_dump_json()) == job
+ shouted = base_url.replace(METHOD_FILE_URL_PREFIX, METHOD_FILE_URL_PREFIX.upper())
+ assert (
+ RenderJob(format=DocGenFormat.PDF, source=DocGenSource.HTML, filename="n.pdf", title="N", html="<p/>", base_url=shouted).base_url
+ == shouted
+ )
+
+ @pytest.mark.parametrize(
+ ("source", "payload", "base_url", "message"),
+ [
+ pytest.param(DocGenSource.LAYOUT, {"layout": _layout()}, f"{METHOD_FILE_URL_PREFIX}{'a' * 64}/", "has no HTML", id="no_html"),
+ pytest.param(
+ DocGenSource.HTML, {"html": "<p/>"}, "file:///etc/", "under 'https://method-files.pipelex.invalid/'", id="not_a_method_file"
+ ),
+ ],
+ )
+ def test_a_base_url_is_refused_outside_html_and_outside_the_prefix(
+ self, source: DocGenSource, payload: dict[str, object], base_url: str, message: str
+ ) -> None:
+ with pytest.raises(ValidationError, match=message):
+ RenderJob.model_validate(
+ {"format": "pdf", "source": source, "filename": "notice.pdf", "title": "Notice", "base_url": base_url, **payload}
+ )
+
def test_a_stray_field_is_refused(self) -> None:
with pytest.raises(ValidationError, match="extra"):
RenderJob.model_validate(
diff --git a/tests/unit/pipelex/cogt/test_cogt_dependency_boundaries.py b/tests/unit/pipelex/cogt/test_cogt_dependency_boundaries.py
index 28624fd46..013ed655a 100644
--- a/tests/unit/pipelex/cogt/test_cogt_dependency_boundaries.py
+++ b/tests/unit/pipelex/cogt/test_cogt_dependency_boundaries.py
@@ -68,6 +68,8 @@ EXPECTED_THIRD_PARTY_ROOTS: frozenset[str] = frozenset(
"httpx",
"instructor",
"jinja2",
+ # The MTHDS standard's own library, not a vendor: a document's relative URLs resolve in its method tree by its rules.
+ "mthds",
"opentelemetry",
"polyfactory",
"pydantic",
diff --git a/tests/unit/pipelex/core/method_trees/test_url_references.py b/tests/unit/pipelex/core/method_trees/test_url_references.py
new file mode 100644
index 000000000..a56c354c1
--- /dev/null
+++ b/tests/unit/pipelex/core/method_trees/test_url_references.py
@@ -0,0 +1,153 @@
+import pytest
+from mthds.method_tree.exceptions import MethodTreeRefusedError
+from mthds.method_tree.references import resolve_reference
+from mthds.method_tree.vocabulary import ReferenceContext, TreeRefusalCode
+
+from pipelex.core.method_trees.method_file_store import HeldFile, tree_of_held_files
+from pipelex.core.method_trees.url_references import resolve_url_path, resolve_url_reference
+
+_HOLDER = "templates/letter.html"
+_TREE_PATHS = frozenset(
+ {
+ "main.mthds",
+ "templates/letter.html",
+ "templates/logo.png",
+ "templates/café.png",
+ "shared/logo.png",
+ "assets/100%.txt",
+ "images/logo.png",
+ "fonts/brand.woff2",
+ }
+)
+
+# A relative URL written in `templates/letter.html`, and the tree path it names, as the MTHDS conformance corpus has them.
+_RESOLVED_CASES = [
+ pytest.param("logo.png", "templates/logo.png", id="beside_the_template"),
+ pytest.param("./logo.png", "templates/logo.png", id="dot_segment"),
+ pytest.param("../shared/logo.png", "shared/logo.png", id="dot_dot_within_the_tree"),
+ pytest.param("../fonts/brand.woff2", "fonts/brand.woff2", id="a_font"),
+ pytest.param("logo.png?v=2#top", "templates/logo.png", id="query_and_fragment_dropped"),
+ pytest.param("../assets/100%25.txt", "assets/100%.txt", id="percent_decoded"),
+ pytest.param("%2E%2E/shared/logo.png", "shared/logo.png", id="encoded_dot_dot_resolves"),
+ pytest.param("cafe%CC%81.png", "templates/café.png", id="nfc_normalized"),
+ pytest.param("../images/logo.png/.", "images/logo.png", id="trailing_dot_segment"),
+]
+# A relative URL written in `templates/letter.html`, the code it is refused with, and what it resolved to.
+_REFUSED_CASES = [
+ pytest.param("..%2Fshared%2Flogo.png", TreeRefusalCode.REFERENCE_INVALID, None, id="encoded_slash"),
+ pytest.param("../../logo.png", TreeRefusalCode.REFERENCE_LEAVES_TREE, None, id="climbs_above_the_root"),
+ pytest.param("../..", TreeRefusalCode.REFERENCE_LEAVES_TREE, None, id="climbs_above_the_root_alone"),
+ pytest.param("/logo.png", TreeRefusalCode.REFERENCE_INVALID, None, id="root_relative"),
+ pytest.param("..\\shared\\logo.png", TreeRefusalCode.REFERENCE_INVALID, None, id="backslash"),
+ pytest.param(".", TreeRefusalCode.REFERENCE_NOT_IN_TREE, "templates", id="the_template_s_directory"),
+ pytest.param("..", TreeRefusalCode.REFERENCE_NOT_IN_TREE, ".", id="the_root"),
+ pytest.param("../", TreeRefusalCode.REFERENCE_NOT_IN_TREE, ".", id="the_root_with_a_slash"),
+ pytest.param("../images/.", TreeRefusalCode.REFERENCE_NOT_IN_TREE, "images", id="a_directory"),
+ pytest.param("missing.png", TreeRefusalCode.REFERENCE_NOT_IN_TREE, "templates/missing.png", id="absent"),
+ pytest.param("../assets/what?.txt", TreeRefusalCode.REFERENCE_NOT_IN_TREE, "assets/what", id="unencoded_question_mark"),
+ pytest.param("../images/logo\t.png", TreeRefusalCode.REFERENCE_NOT_A_TREE_PATH, "images/logo\t.png", id="control_character"),
+ pytest.param("%0A", TreeRefusalCode.REFERENCE_INVALID, None, id="encoded_line_feed"),
+ pytest.param("%00", TreeRefusalCode.REFERENCE_INVALID, None, id="encoded_nul"),
+ pytest.param("%FF", TreeRefusalCode.REFERENCE_INVALID, None, id="not_utf8"),
+ pytest.param("caf%E9", TreeRefusalCode.REFERENCE_INVALID, None, id="latin1"),
+ pytest.param("100%.png", TreeRefusalCode.REFERENCE_INVALID, None, id="bare_percent"),
+ pytest.param("%zz", TreeRefusalCode.REFERENCE_INVALID, None, id="non_hex_escape"),
+ pytest.param("%4", TreeRefusalCode.REFERENCE_INVALID, None, id="short_escape"),
+]
+# A URL that is no file reference, left exactly as written.
+_NOT_A_REFERENCE_CASES = [
+ pytest.param("https://cdn.example.com/logo.png", id="https"),
+ pytest.param("data:image/png;base64,iVBORw0KGgo=", id="data"),
+ pytest.param("pipelex-storage://scope/assets/logo.png", id="storage"),
+ pytest.param("//cdn.example.com/logo.png", id="authority"),
+ pytest.param("mailto:[email protected]", id="mailto"),
+ pytest.param("", id="empty"),
+ pytest.param("#top", id="fragment"),
+ pytest.param("?v=2", id="query"),
+]
+
+
+class TestUrlReferences:
+ @pytest.mark.parametrize(("reference", "expected"), _RESOLVED_CASES)
+ def test_a_relative_url_resolves_against_its_holder(self, reference: str, expected: str) -> None:
+ assert resolve_url_reference(reference=reference, holder=_HOLDER, tree_paths=_TREE_PATHS) == expected
+
+ @pytest.mark.parametrize(("reference", "code", "resolved"), _REFUSED_CASES)
+ def test_a_relative_url_the_standard_refuses_is_refused_with_its_code(self, reference: str, code: TreeRefusalCode, resolved: str | None) -> None:
+ with pytest.raises(MethodTreeRefusedError) as exc_info:
+ resolve_url_reference(
+ reference=reference, holder=_HOLDER, tree_paths=_TREE_PATHS, holder_label="the template file 'templates/letter.html'"
+ )
+
+ assert exc_info.value.refusal.code == code
+ assert exc_info.value.refusal.resolved == resolved
+ assert "the template file 'templates/letter.html'" in exc_info.value.message
+
+ @pytest.mark.parametrize("reference", _NOT_A_REFERENCE_CASES)
+ def test_a_url_that_is_no_file_reference_is_left_as_written(self, reference: str) -> None:
+ assert resolve_url_reference(reference=reference, holder=_HOLDER, tree_paths=_TREE_PATHS) is None
+
+ @pytest.mark.parametrize(
+ "written", [pytest.param(case.values[0], id=case.id) for case in (*_RESOLVED_CASES, *_REFUSED_CASES, *_NOT_A_REFERENCE_CASES)]
+ )
+ def test_every_answer_is_the_one_mthds_python_gives(self, written: str) -> None:
+ """Run over a set of tree paths, the steps answer as `resolve_reference` does over the tree itself."""
+ tree = tree_of_held_files(files={tree_path: HeldFile.of_bytes(data=tree_path.encode()) for tree_path in _TREE_PATHS})
+ expected: str | TreeRefusalCode | None
+ try:
+ expected = resolve_reference(written, holder=_HOLDER, context=ReferenceContext.URL, tree=tree)
+ except MethodTreeRefusedError as exc:
+ expected = exc.refusal.code
+ actual: str | TreeRefusalCode | None
+ try:
+ actual = resolve_url_reference(reference=written, holder=_HOLDER, tree_paths=tree.members)
+ except MethodTreeRefusedError as exc:
+ actual = exc.refusal.code
+
+ assert actual == expected
+
+ @pytest.mark.parametrize(
+ ("reference", "expected"),
+ [
+ pytest.param("templates/logo.png", "templates/logo.png", id="from_the_root"),
+ pytest.param("templates/../shared/logo.png", "shared/logo.png", id="dot_dot_within_the_tree"),
+ ],
+ )
+ def test_a_relative_url_with_no_holder_resolves_from_the_root(self, reference: str, expected: str) -> None:
+ """Bundles parsed from content strings sit at the root of the tree of the files submitted with them."""
+ assert resolve_url_reference(reference=reference, holder=None, tree_paths=_TREE_PATHS) == expected
+
+ def test_a_relative_url_with_no_holder_cannot_climb_above_the_root(self) -> None:
+ with pytest.raises(MethodTreeRefusedError) as exc_info:
+ resolve_url_reference(reference="../logo.png", holder=None, tree_paths=_TREE_PATHS)
+
+ assert exc_info.value.refusal.code == TreeRefusalCode.REFERENCE_LEAVES_TREE
+ assert "the root of its method tree" in exc_info.value.message
+
+ def test_a_url_path_is_resolved_even_when_it_reads_as_a_scheme(self) -> None:
+ """What an engine joined below the root is a path whatever it looks like, so a colon is refused, never left as written."""
+ with pytest.raises(MethodTreeRefusedError) as exc_info:
+ resolve_url_path(url_path="templates/a:b.png", reference="templates/a:b.png", holder=None, tree_paths=_TREE_PATHS)
+
+ assert exc_info.value.refusal.code == TreeRefusalCode.REFERENCE_NOT_A_TREE_PATH
+
+ @pytest.mark.parametrize("written", ["logo.png", "./logo.png", "../logo.png", ".", "letters.mthds", "https://cdn.example.com/x.png"])
+ def test_in_the_one_file_tree_of_a_bundle_named_alone_every_answer_is_mthds_python_s(self, written: str) -> None:
+ """A file beside a bundle named alone is refused as out of its one-file tree, with the advice to hand over its folder."""
+ tree = tree_of_held_files(files={"letters.mthds": HeldFile.of_bytes(data=b"x")}).model_copy(update={"is_one_file": True})
+ expected: str | TreeRefusalCode | None
+ try:
+ expected = resolve_reference(written, holder="letters.mthds", context=ReferenceContext.URL, tree=tree)
+ except MethodTreeRefusedError as exc:
+ expected = exc.refusal.code
+ actual: str | TreeRefusalCode | None
+ message = ""
+ try:
+ actual = resolve_url_reference(reference=written, holder="letters.mthds", tree_paths=tree.members, is_one_file_tree=True)
+ except MethodTreeRefusedError as exc:
+ actual = exc.refusal.code
+ message = exc.message
+
+ assert actual == expected
+ if actual == TreeRefusalCode.REFERENCE_IN_ONE_FILE_TREE:
+ assert "Hand over the method's directory instead" in message
diff --git a/tests/unit/pipelex/pipe_operators/pipe_doc_gen/test_template_asset_check.py b/tests/unit/pipelex/pipe_operators/pipe_doc_gen/test_template_asset_check.py
new file mode 100644
index 000000000..4ea8b67a1
--- /dev/null
+++ b/tests/unit/pipelex/pipe_operators/pipe_doc_gen/test_template_asset_check.py
@@ -0,0 +1,120 @@
+import pytest
+
+from pipelex.core.method_trees.method_file_store import HeldFile, tree_of_held_files
+from pipelex.pipe_operators.doc_gen.template_asset_check import find_template_asset_refusal, literal_template_urls, stylesheet_urls
+
+_FILES = {
+ "METHODS.toml": b'[package]\naddress = "github.com/acme/letters"\n',
+ "letters.mthds": b'domain = "letters"\n',
+ "templates/letter.html": b"<p>letter</p>",
+ "templates/logo.png": b"\x89PNG logo",
+ "styles/print.css": b'@import "base.css";\n@font-face { src: url("../fonts/brand.woff2"); }\n/* url(../fonts/commented-out.woff2) */\n',
+ "styles/base.css": b"body { background: url(../images/band.png); }\n",
+ "fonts/brand.woff2": b"wOF2 brand",
+ "images/band.png": b"\x89PNG band",
+ "funcs/private_logic.py": b"SECRET = 42\n",
+}
+_TREE = tree_of_held_files(files={tree_path: HeldFile.of_bytes(data=data) for tree_path, data in _FILES.items()})
+
+
+def _read_file(tree_path: str) -> bytes | None:
+ return _FILES.get(tree_path)
+
+
+class TestTemplateAssetCheck:
+ def test_the_urls_a_printer_reads_are_collected_and_the_run_s_values_skipped(self) -> None:
+ template = """{# <img src="commented-out.png"> #}
+<link rel="stylesheet" href="../styles/print.css"><link rel="icon" href="favicon.ico">
+<img src="logo.png"><img src="{{ photo.url }}"><img src="$logo"><img src="@logo">
+<a href="next-page.html">next</a><a rel="attachment" href="terms.pdf">terms</a>
+<object data="chart.svg"></object><embed src="map.svg"><svg><image href="seal.png"/></svg>
+<div style="background: url('paper.png')"></div>
+<style>h1 { background: url(band.png); } @import 'more.css';</style>
+<!-- <img src="old.png"> -->
+"""
+
+ urls = literal_template_urls(template=template)
+
+ assert sorted(urls) == sorted(
+ ["../styles/print.css", "logo.png", "terms.pdf", "chart.svg", "map.svg", "seal.png", "paper.png", "band.png", "more.css"]
+ )
+
+ def test_a_stylesheet_s_urls_skip_its_comments(self) -> None:
+ stylesheet = _FILES["styles/print.css"].decode()
+
+ assert stylesheet_urls(stylesheet=stylesheet) == ["../fonts/brand.woff2", "base.css"]
+
+ def test_images_stylesheets_and_the_urls_they_name_pass(self) -> None:
+ template = '<link rel="stylesheet" href="../styles/print.css"><img src="logo.png?v=2"><img src="https://cdn.example.com/x.png">'
+
+ refusal = find_template_asset_refusal(
+ template=template, template_label="the template file", holder="templates/letter.html", tree=_TREE, read_file=_read_file
+ )
+
+ assert refusal is None
+
+ @pytest.mark.parametrize(
+ ("template", "reference", "holder_label", "reason"),
+ [
+ pytest.param('<img src="missing.png">', "missing.png", "the template", "'templates/missing.png', which is not a file", id="absent"),
+ pytest.param('<img src="../../logo.png">', "../../logo.png", "the template", "climbs above the root", id="outside_the_tree"),
+ pytest.param('<img src="/logo.png">', "/logo.png", "the template", "starts with '/'", id="root_relative"),
+ pytest.param(
+ '<a rel="attachment" href="../funcs/private_logic.py">x</a>',
+ "../funcs/private_logic.py",
+ "the template",
+ "which is not an image, a stylesheet or a font",
+ id="not_an_asset",
+ ),
+ pytest.param(
+ '<link rel="stylesheet" href="../letters.mthds">', "../letters.mthds", "the template", "which is not an image", id="a_bundle"
+ ),
+ ],
+ )
+ def test_a_literal_url_naming_no_asset_of_the_tree_is_refused(self, template: str, reference: str, holder_label: str, reason: str) -> None:
+ refusal = find_template_asset_refusal(
+ template=template, template_label="the template", holder="templates/letter.html", tree=_TREE, read_file=_read_file
+ )
+
+ assert refusal is not None
+ assert (refusal.reference, refusal.holder_label) == (reference, holder_label)
+ assert reason in refusal.reason
+
+ def test_a_url_of_a_linked_stylesheet_is_checked_against_the_stylesheet(self) -> None:
+ files = {**_FILES, "styles/base.css": b"body { background: url(../images/missing.png); }\n"}
+
+ refusal = find_template_asset_refusal(
+ template='<link rel="stylesheet" href="../styles/print.css">',
+ template_label="the template",
+ holder="templates/letter.html",
+ tree=_TREE,
+ read_file=files.get,
+ )
+
+ assert refusal is not None
+ assert refusal.reference == "../images/missing.png"
+ assert refusal.holder_label == "the stylesheet 'styles/base.css'"
+ assert "'images/missing.png', which is not a file" in refusal.reason
+
+ def test_an_inline_template_of_bundles_from_strings_resolves_from_the_root(self) -> None:
+ refusal = find_template_asset_refusal(
+ template='<img src="templates/logo.png"><img src="../logo.png">',
+ template_label="the template",
+ holder=None,
+ tree=_TREE,
+ read_file=_read_file,
+ )
+
+ assert refusal is not None
+ assert refusal.reference == "../logo.png"
+ assert "climbs above the root" in refusal.reason
+
+ def test_a_bundle_named_alone_is_told_to_hand_over_its_folder(self) -> None:
+ one_file_tree = tree_of_held_files(files={"letters.mthds": HeldFile.of_bytes(data=b"x")}).model_copy(update={"is_one_file": True})
+
+ refusal = find_template_asset_refusal(
+ template='<img src="logo.png">', template_label="the template", holder="letters.mthds", tree=one_file_tree, read_file=_read_file
+ )
+
+ assert refusal is not None
+ assert "Hand over the method's directory instead" in refusal.reason