Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
fix/Set-inside-if-read · L-260930-afc983 (#1319)
Louis Choquel0e9a29067507
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex dba2ae784..af00afee4 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -15,6 +15,7 @@### Fixed+- **A name a template sets is read where Jinja binds it (Breaking)**: the input check of `PipeLLM`, `PipeCompose`, `PipeSearch`, `PipeImgGen` and `PipeDocGen`, `PipeDocGen`'s field check and the optional-input guard now treat a `{% set %}` name as the template's own inside the loop, macro or block that sets it and after an `{% if %}` every branch of which sets it, so a template setting a name in a loop body or in each branch of an `if` and reading it there is no longer refused as reading an undeclared input. A name only some branches set is still read from the input of that name. Newly refused: an input declared only so that such a set would load, which is now never read, a macro called before its `{% macro %}`, an optional input read unguarded in a macro argument's default, and a name a loop or a macro binds read inside a `{% block %}` without `scoped`, which Jinja reads from the input of that name.- **File inputs are checked against the models that read them (Breaking)**: run setup now identifies every image and document input from its first bytes, refuses an `Image` input holding something other than an image with `PipelineInputNotAnImageError`, and refuses a document certain to reach a `PipeExtract` or `PipeLLM` step whose model does not read its format with `PipelineInputFormatUnsupportedError`, both 422 input errors naming the input, the step, the model and the formats it reads; a step reached only through a condition refuses the file itself with `ExtractInputFormatError` or `PromptDocumentFormatError`. A `PipeLLM` document its model does not read now raises `PromptDocumentFormatError` where it raised `LLMCapabilityError`, so a consumer branching on `error_type` must follow. `docling-extract-text` now declares the Word, PowerPoint, Excel, HTML, Markdown, CSV, plain-text, WebVTT and email files it already extracted, which `pipelex update` installs.- **Markdown with unclosed emphasis is converted in memory proportional to its length**: the Markdown parser behind the `markdown` filter, the `Markdown` concept's HTML and the built-in PDF engine held memory growing with the square of the text when emphasis or strikethrough markers were left unmatched, about 1.5 GB for 64,000 characters of `**a `; it now joins the text they leave behind in one pass, and renders the same HTML.- **Routing profiles no longer rewrite their own routes**: matching a model merged the enabled `optional_routes` into the profile's declared `routes` in place, so after the first match a route declared optional had become an unconditional one in the loaded profile. Only the enablement checks each match repeats kept that from changing a routing decision.diff --git a/docs/building-methods/pipes/pipe-operators/PipeCompose.md b/docs/building-methods/pipes/pipe-operators/PipeCompose.mdindex 4ccba455d..84a216407 100644--- a/docs/building-methods/pipes/pipe-operators/PipeCompose.md+++ b/docs/building-methods/pipes/pipe-operators/PipeCompose.md@@ -40,7 +40,7 @@ A template reads data and calls methods of plain values, and nothing else: it ca| `output` | string | The concept for the output | Yes || `template` | string or section | An inline template string, or a `[pipe.name.template]` section (see below) | Yes* |-Every input the pipe declares must be read by the template, and every variable the template reads must be declared in `inputs`. Validation refuses an input the template never reads as `extraneous_input_variable`, naming the input, so you either reference it in the template or remove it from `inputs`; an undeclared variable is refused as `missing_input_variable`.+Every input the pipe declares must be read by the template, and every variable the template reads must be declared in `inputs`. Validation refuses an input the template never reads as `extraneous_input_variable`, naming the input, so you either reference it in the template or remove it from `inputs`; an undeclared variable is refused as `missing_input_variable`. A name the template sets with `{% set %}` is its own, not an input, from that statement on: after an `{% if %}` it stays set only when every branch sets it, the `{% else %}` included, so a name only some branches set is read from the input of that name on the other paths; and a loop, a macro or a block keeps what it sets to its own body.*Template mode requires `template`. When using the rich form (`[pipe.name.template]` section), the following sub-fields are available:diff --git a/docs/building-methods/pipes/pipe-operators/PipeDocGen.md b/docs/building-methods/pipes/pipe-operators/PipeDocGen.mdindex ff90f4397..8bb51ace1 100644--- a/docs/building-methods/pipes/pipe-operators/PipeDocGen.md+++ b/docs/building-methods/pipes/pipe-operators/PipeDocGen.md@@ -112,7 +112,7 @@ A `Text` input prints as plain paragraphs, so a report whose output is `Text` prA `pdf` step's `template`, or its `.html` `template_file`, is rendered with exactly the stack of a `PipeCompose` HTML template: `$x` and `@x`, the `format` and `tag` filters, HTML autoescaping, and the [template sandbox](../../../under-the-hood/template-sandbox.md). A template that is only a fragment is wrapped in the plugin's print stylesheet (A4, running headers, page numbers); a whole HTML document prints as written, with any page setup its CSS gives. Three differences from `PipeCompose` matter:-- **A field a template names must exist.** Each path a template reads, such as `invoice.customer.name`, is checked against the inputs' concepts when the method loads, including a path through a loop variable: in `{% for item in invoice.line_items %}`, `item.amount` is checked against the line item's concept. What the load-time check cannot follow, such as a value set with `{% set %}`, fails when the template renders at the dry run, since a missing value is an error rather than empty text. An absent optional input can still be tested with `{% if notes %}` or `is defined`.+- **A field a template names must exist.** Each path a template reads, such as `invoice.customer.name`, is checked against the inputs' concepts when the method loads, including a path through a loop variable: in `{% for item in invoice.line_items %}`, `item.amount` is checked against the line item's concept. A name the template sets with `{% set %}` is followed no further, so a field read through it is checked when the template renders at the dry run, where a missing value is an error rather than empty text; a name only some branches of an `{% if %}` set is still checked as the input of that name, which the template reads on the paths where no branch ran. An absent optional input can still be tested with `{% if notes %}` or `is defined`.- **Pipelex's `format` filter shadows Jinja's own.** To format a number, write `{{ '%.2f' % invoice.total }}` rather than `{{ '%.2f' | format(invoice.total) }}`.- **The `markdown` filter** turns Markdown held in a plain text field into HTML: `{{ invoice.notes | markdown }}`. A `Markdown` input needs no filter.diff --git a/docs/building-methods/pipes/pipe-operators/PipeLLM.md b/docs/building-methods/pipes/pipe-operators/PipeLLM.mdindex 08e4489b2..adc8b01c3 100644--- a/docs/building-methods/pipes/pipe-operators/PipeLLM.md+++ b/docs/building-methods/pipes/pipe-operators/PipeLLM.md@@ -18,7 +18,7 @@ For structured outputs, you have two options:If you already have text from elsewhere (a PDF extraction, a search result, an upstream pipe), call [`PipeStructure`](./PipeStructure.md) directly — there's no need to wrap a `PipeLLM` around it.-Every input the pipe declares must be read by `prompt` or `system_prompt`, and every variable they read must be declared in `inputs`. Validation refuses an input neither prompt reads as `extraneous_input_variable`, naming the input, so you either reference it in a prompt or remove it from `inputs`; an undeclared variable is refused as `missing_input_variable`.+Every input the pipe declares must be read by `prompt` or `system_prompt`, and every variable they read must be declared in `inputs`. Validation refuses an input neither prompt reads as `extraneous_input_variable`, naming the input, so you either reference it in a prompt or remove it from `inputs`; an undeclared variable is refused as `missing_input_variable`. A name a prompt sets with `{% set %}` is its own, not an input, from that statement on: after an `{% if %}` it stays set only when every branch sets it, the `{% else %}` included, so a name only some branches set is read from the input of that name on the other paths; and a loop, a macro or a block keeps what it sets to its own body.## Working with Images (Vision Language Models)diff --git a/docs/features/validation-dry-run.md b/docs/features/validation-dry-run.mdindex 42fb7cf4b..b61ed5392 100644--- a/docs/features/validation-dry-run.md+++ b/docs/features/validation-dry-run.md@@ -18,7 +18,7 @@ Check pipeline syntax, structure, and compatibility without execution:- **Syntax validation** — Catch MTHDS language errors via plxt linting- **Structure validation** — Resolve pipe and concept references in the loaded library, refuse concepts whose structures form a cycle or whose fields name a concept that cannot be found, and verify each pipe's declared inputs against the concepts that operator accepts — plus a controller's declared output against what it actually produces (for a `PipeSequence`, its last step's output concept and multiplicity)- **Input validation** — Ensure required inputs are provided and correctly typed, and that every variable a step reads is bound by the time that step runs-- **Declared inputs are read** — For `PipeLLM`, `PipeCompose`, `PipeSearch` and `PipeImgGen`, every variable a template reads must be declared in `inputs` (`missing_input_variable`), and every declared input must be read by one of the pipe's templates (`extraneous_input_variable`): the prompt and system prompt of a `PipeLLM`, the template or the construct of a `PipeCompose`, the prompt of a `PipeSearch`, the prompt and negative prompt of a `PipeImgGen`. An unread input is refused on the pipe that declares it, before any controller above it is asked to supply it+- **Declared inputs are read** — For `PipeLLM`, `PipeCompose`, `PipeSearch` and `PipeImgGen`, every variable a template reads must be declared in `inputs` (`missing_input_variable`), and every declared input must be read by one of the pipe's templates (`extraneous_input_variable`): the prompt and system prompt of a `PipeLLM`, the template or the construct of a `PipeCompose`, the prompt of a `PipeSearch`, the prompt and negative prompt of a `PipeImgGen`. An unread input is refused on the pipe that declares it, before any controller above it is asked to supply it. A name the template sets with `{% set %}` is its own, not an input, from that statement on: after an `{% if %}` it stays set only when every branch sets it, the `{% else %}` included, so a name only some branches set is read from the input of that name on the other paths; and a loop, a macro or a block keeps what it sets to its own body.A pipe that references a sub-pipe from a package that isn't loaded is reported as **skipped** rather than validated, so a passing run is not proof that every cross-package dependency resolves.diff --git a/pipelex/pipe_operators/doc_gen/template_field_check.py b/pipelex/pipe_operators/doc_gen/template_field_check.pyindex e57f55819..a527bb9ab 100644--- a/pipelex/pipe_operators/doc_gen/template_field_check.py+++ b/pipelex/pipe_operators/doc_gen/template_field_check.py@@ -22,7 +22,8 @@ from pipelex.core.stuffs.stuff_artefact import StuffArtefactfrom pipelex.core.stuffs.stuff_content import StuffContentfrom pipelex.pipe_operators.doc_gen.exceptions import PipeDocGenFactoryErrorfrom pipelex.runtime_hub import get_class_registry-from pipelex.tools.jinja2.jinja2_field_paths import LIST_ITEM_SEGMENT, TemplateFieldPath, detect_template_field_paths+from pipelex.tools.jinja2.jinja2_field_paths import TemplateFieldPath, detect_template_field_paths+from pipelex.tools.jinja2.jinja2_scopes import LIST_ITEM_SEGMENTfrom pipelex.tools.jinja2.template_category import TemplateCategory_NONE_TYPE = type(None)diff --git a/pipelex/tools/jinja2/jinja2_field_paths.py b/pipelex/tools/jinja2/jinja2_field_paths.pyindex bb3f40242..c2416b95a 100644--- a/pipelex/tools/jinja2/jinja2_field_paths.py+++ b/pipelex/tools/jinja2/jinja2_field_paths.py@@ -5,9 +5,13 @@ reads, so a load-time check can compare each path with the input's concept and rany run. A loop variable is followed back to what it iterates: in `{% for item in invoice.line_items %}`,`item.amount` reads `invoice.line_items[].amount`, the `[]` standing for an item of the list.-It is deliberately conservative, since what it cannot follow is left to the strict undefined at the dry run: a-name set with `{% set %}`, a macro argument, a tuple loop target, and a chain that starts from a subscript, a-call or a filter give no path. Only attribute chains (`a.b.c`) are followed; `a['b']` is not.+Both detectors are handlers on the one read walk of `jinja2_scopes.py`, so they agree on which names the template+binds: a name set in every branch of an `if`, or anywhere before the read in the same loop or block body, is the+template's own and gives no path, while a name set in only some branches is read as the input of that name, and+its path is checked. The walk is deliberately conservative, since what it cannot follow is left to the strict+undefined at the dry run: a name the template sets, a macro argument, a tuple loop target, a called global and+a chain that starts from a subscript, a call or a filter give no path. Only attribute chains (`a.b.c`) are+followed; `a['b']` is not."""from typing import NamedTuple@@ -17,11 +21,9 @@ from jinja2.exceptions import TemplateSyntaxErrorfrom pipelex.tools.jinja2.exceptions import Jinja2DetectVariablesErrorfrom pipelex.tools.jinja2.jinja2_environment import make_jinja2_env_without_loader+from pipelex.tools.jinja2.jinja2_scopes import ScopeBindings, attribute_chain, resolve_bound_path, walk_template_readsfrom pipelex.tools.jinja2.template_category import TemplateCategory-# The segment standing for "an item of the list" in a path reached through a loop variable.-LIST_ITEM_SEGMENT = "[]"-class TemplateFieldPath(NamedTuple):"""One attribute chain a template reads, resolved back to the input it starts from."""@@ -32,104 +34,6 @@ class TemplateFieldPath(NamedTuple):"""The chain as the template writes it, such as `item.amount`, for messages."""-# What a name stands for where it is read: the path it is bound to, or None when it is declared locally and-# cannot be followed (a `set`, a macro argument, `loop`). A name absent from the bindings is an input or a global.-_Bindings = dict[str, tuple[str, ...] | None]---def _chain(node: nodes.Node) -> tuple[str, list[str]] | None:- """The name and attribute names of a pure attribute chain (`a.b.c`), or None for any other shape."""- attributes: list[str] = []- current = node- while isinstance(current, nodes.Getattr):- attributes.append(current.attr)- current = current.node- if isinstance(current, nodes.Name):- return current.name, list(reversed(attributes))- return None---def _resolve(*, name: str, attributes: list[str], bindings: _Bindings) -> tuple[str, ...] | None:- """The full path a chain reads, through the bindings: None when it starts from a name that cannot be followed."""- if name in bindings:- bound = bindings[name]- if bound is None:- return None- return (*bound, *attributes)- return (name, *attributes)---def _target_names(target: nodes.Node) -> list[str]:- if isinstance(target, nodes.Name):- return [target.name]- if isinstance(target, nodes.Tuple):- return [name for item in target.items for name in _target_names(item)]- return []---def _walk(*, node: nodes.Node, bindings: _Bindings, paths: list[TemplateFieldPath]) -> None:- if isinstance(node, (nodes.Template, nodes.Scope)):- _walk_body(body=node.body, bindings=bindings, paths=paths)- return- if isinstance(node, nodes.With):- for value in node.values:- _walk(node=value, bindings=bindings, paths=paths)- with_bindings: _Bindings = dict(bindings)- for target in node.targets:- for target_name in _target_names(target):- with_bindings[target_name] = None- _walk_body(body=node.body, bindings=with_bindings, paths=paths)- return- if isinstance(node, nodes.For):- _walk(node=node.iter, bindings=bindings, paths=paths)- loop_bindings: _Bindings = {**bindings, "loop": None}- iterated = _chain(node.iter)- iterated_path = _resolve(name=iterated[0], attributes=iterated[1], bindings=bindings) if iterated is not None else None- if isinstance(node.target, nodes.Name) and iterated_path is not None:- loop_bindings[node.target.name] = (*iterated_path, LIST_ITEM_SEGMENT)- else:- for target_name in _target_names(node.target):- loop_bindings[target_name] = None- _walk_body(body=node.body, bindings=loop_bindings, paths=paths)- if node.test is not None:- _walk(node=node.test, bindings=loop_bindings, paths=paths)- _walk_body(body=node.else_, bindings=bindings, paths=paths)- return- if isinstance(node, nodes.Macro):- macro_bindings: _Bindings = {**bindings, "caller": None, "varargs": None, "kwargs": None}- for argument in node.args:- macro_bindings[argument.name] = None- for default in node.defaults:- _walk(node=default, bindings=bindings, paths=paths)- _walk_body(body=node.body, bindings=macro_bindings, paths=paths)- return- if isinstance(node, (nodes.Name, nodes.Getattr)):- chain = _chain(node)- if chain is not None:- if isinstance(node, nodes.Name) and node.ctx != "load":- return- name, attributes = chain- path = _resolve(name=name, attributes=attributes, bindings=bindings)- if path is not None:- written = ".".join([name, *attributes])- paths.append(TemplateFieldPath(root=path[0], segments=path[1:], written=written))- return- for child in node.iter_child_nodes():- _walk(node=child, bindings=bindings, paths=paths)---def _walk_body(*, body: list[nodes.Node], bindings: _Bindings, paths: list[TemplateFieldPath]) -> None:- """Walk statements in order: a `{% set %}` declares its name for the statements after it, in this scope."""- scope_bindings: _Bindings = dict(bindings)- for statement in body:- if isinstance(statement, nodes.Macro):- scope_bindings[statement.name] = None- _walk(node=statement, bindings=scope_bindings, paths=paths)- if isinstance(statement, (nodes.Assign, nodes.AssignBlock)):- for target_name in _target_names(statement.target):- scope_bindings[target_name] = None--def detect_template_field_paths(*, template_source: str, template_category: TemplateCategory) -> list[TemplateFieldPath]:"""Every attribute chain the template reads, in source order, resolved back to the name it starts from.@@ -145,5 +49,19 @@ def detect_template_field_paths(*, template_source: str, template_category: Tempmsg = f"Jinja2 detect field paths — syntax error: '{syntax_error}', template_category: {template_category}"raise Jinja2DetectVariablesError(msg) from syntax_errorpaths: list[TemplateFieldPath] = []- _walk(node=parsed_ast, bindings={}, paths=paths)++ def collect_field_path(*, node: nodes.Node, bindings: ScopeBindings) -> bool:+ if not isinstance(node, (nodes.Name, nodes.Getattr)):+ return False+ chain = attribute_chain(node)+ if chain is None:+ # An attribute on a subscript, a call or a filter (`items[0].text`) reads the path its chain starts from+ return False+ name, attributes = chain+ path = resolve_bound_path(name=name, attributes=attributes, bindings=bindings)+ if path is not None:+ paths.append(TemplateFieldPath(root=path[0], segments=path[1:], written=".".join([name, *attributes])))+ return True++ walk_template_reads(template=parsed_ast, global_names=set(jinja2_env.globals), handle_read=collect_field_path)return pathsdiff --git a/pipelex/tools/jinja2/jinja2_optional_guards.py b/pipelex/tools/jinja2/jinja2_optional_guards.pyindex 15afcf7aa..56bf876c7 100644--- a/pipelex/tools/jinja2/jinja2_optional_guards.py+++ b/pipelex/tools/jinja2/jinja2_optional_guards.py@@ -27,6 +27,13 @@ from pydantic.dataclasses import dataclassfrom pipelex.tools.jinja2.exceptions import Jinja2DetectVariablesErrorfrom pipelex.tools.jinja2.jinja2_environment import make_jinja2_env_without_loader+from pipelex.tools.jinja2.jinja2_scopes import (+ definitely_assigned_names,+ dotted_attribute_path,+ frame_bound_names,+ names_assigned_by_statements,+ names_read_before_bound,+)from pipelex.tools.jinja2.template_category import TemplateCategoryfrom pipelex.tools.misc.string_utils import get_root_from_dotted_path@@ -42,20 +49,16 @@ class UnguardedOptionalReference:path: str-def _build_full_path(node: nodes.Node) -> str | None:- """Build the dotted path of a Name / Getattr chain, or None for unsupported shapes."""- if isinstance(node, nodes.Name):- return node.name- if isinstance(node, nodes.Getattr):- parent_path = _build_full_path(node.node)- if parent_path is not None:- return f"{parent_path}.{node.attr}"- return None--class _GuardWalker:"""Recursive AST walk tracking which optional variables are currently guarded and whichnames are locally declared (loop targets, macro params, `{% set %}` assignments).++ Where a name is declared follows Jinja's scopes, from the rules `jinja2_scopes.py` states for+ every template walker: after an `if`, only a name every branch sets is declared, so an optional+ that only some branches rebind may still be read absent; a loop, a macro, a call, filter or set+ block, a `with`, an `autoescape` and a block keep what they set to their own body; a macro sees+ what its whole body binds; and a block without `scoped` reads only the template context, so+ none of the names, nor the guards on them, that the frames around it bind reach it."""def __init__(self, *, optional_variable_names: set[str]) -> None:@@ -87,7 +90,14 @@ class _GuardWalker:return self._guard_vars(test_node.left) | self._guard_vars(test_node.right)return frozenset()- def _walk_test(self, test_node: nodes.Node, *, guarded: frozenset[str], declared: frozenset[str]) -> None:+ def _walk_test(+ self,+ test_node: nodes.Node,+ *,+ guarded: frozenset[str],+ declared: frozenset[str],+ context_declared: frozenset[str] | None,+ ) -> None:"""Walk a test position: bare names and presence tests are safe references there."""if isinstance(test_node, nodes.Name):return@@ -96,83 +106,107 @@ class _GuardWalker:if isinstance(test_node, nodes.And):# `and` short-circuits: the right operand only evaluates when the left is truthy,# so the left operand's guard extends over the right (`{% if var and var.attr %}`).- self._walk_test(test_node.left, guarded=guarded, declared=declared)- self._walk_test(test_node.right, guarded=guarded | self._guard_vars(test_node.left), declared=declared)+ self._walk_test(test_node.left, guarded=guarded, declared=declared, context_declared=context_declared)+ right_guarded = guarded | self._guard_vars(test_node.left)+ self._walk_test(test_node.right, guarded=right_guarded, declared=declared, context_declared=context_declared)returnif isinstance(test_node, nodes.Or):- self._walk_test(test_node.left, guarded=guarded, declared=declared)- self._walk_test(test_node.right, guarded=guarded, declared=declared)+ self._walk_test(test_node.left, guarded=guarded, declared=declared, context_declared=context_declared)+ self._walk_test(test_node.right, guarded=guarded, declared=declared, context_declared=context_declared)returnif isinstance(test_node, nodes.Not):- self._walk_test(test_node.node, guarded=guarded, declared=declared)+ self._walk_test(test_node.node, guarded=guarded, declared=declared, context_declared=context_declared)return- self.walk(test_node, guarded=guarded, declared=declared)-- def _walk_body(self, body_nodes: list[nodes.Node], *, guarded: frozenset[str], declared: frozenset[str]) -> None:- """Walk a statement body sequentially: a `{% set %}` or `{% macro %}` declares its name- for SUBSEQUENT statements only — a read occurring before the assignment still refers to- the (possibly undefined) context value and must be classified against it.+ self.walk(test_node, guarded=guarded, declared=declared, context_declared=context_declared)++ def _walk_body(+ self,+ body_nodes: list[nodes.Node],+ *,+ guarded: frozenset[str],+ declared: frozenset[str],+ context_declared: frozenset[str] | None,+ look_ahead: frozenset[str] | None = None,+ ) -> frozenset[str]:+ """Walk a statement body sequentially, and return the names declared after it.++ What a statement binds for certain (a `{% set %}`, a `{% macro %}`, a name every branch of an+ `if` sets) is declared for SUBSEQUENT statements only — a read occurring before the assignment+ still refers to the (possibly undefined) context value and must be classified against it. A+ macro runs where it is called, so it also sees `look_ahead`, what its frame binds for certain+ wherever it stands, but for the names the frame reads before binding them, whose copy Jinja+ starts from the input; it is computed from the body when the body is a frame of its own, and+ given for the branches of an `if`.+ `context_declared` is what the template context holds where the body runs, and None at the top+ level, whose names the context takes on as they are set."""+ if look_ahead is None:+ look_ahead = frozenset(names_assigned_by_statements(body_nodes) - names_read_before_bound(body_nodes))for body_node in body_nodes:- if isinstance(body_node, nodes.Assign):- # The assignment's right-hand side is evaluated against the current scope.- self.walk(body_node.node, guarded=guarded, declared=declared)- declared |= self._assign_target_names(body_node.target)- continue- if isinstance(body_node, nodes.AssignBlock):- # `{% set x %}...{% endset %}`: the body (and filter) evaluate against the- # current scope; the target declares for subsequent statements only — and the- # target itself is a store, never a read.- self._walk_body(body_node.body, guarded=guarded, declared=declared)+ if isinstance(body_node, nodes.If):+ self._walk_if(body_node, guarded=guarded, declared=declared, context_declared=context_declared, look_ahead=look_ahead)+ elif isinstance(body_node, nodes.Assign):+ # The assignment's right-hand side is evaluated against the current scope; the+ # target is a store, never a read.+ self.walk(body_node.node, guarded=guarded, declared=declared, context_declared=context_declared)+ elif isinstance(body_node, nodes.AssignBlock):+ # `{% set x %}...{% endset %}`: the body runs in a frame of its own, so what it sets+ # stays in it, and the filter is applied in that frame once the body has run; the+ # target is a store, never a read.+ frame_context = declared if context_declared is None else context_declared+ body_declared = self._walk_body(body_node.body, guarded=guarded, declared=declared, context_declared=frame_context)if body_node.filter is not None:- self.walk(body_node.filter, guarded=guarded, declared=declared)- declared |= self._assign_target_names(body_node.target)- continue- if isinstance(body_node, nodes.Macro):- declared |= {body_node.name}- self.walk(body_node, guarded=guarded, declared=declared)-- @classmethod- def _assign_target_names(cls, target: nodes.Node) -> frozenset[str]:- """The names a `{% set %}` target declares — a plain Name or a Tuple of Names."""- if isinstance(target, nodes.Name):- return frozenset({target.name})- if isinstance(target, nodes.Tuple):- return frozenset(item.name for item in target.items if isinstance(item, nodes.Name))- return frozenset()+ self.walk(body_node.filter, guarded=guarded, declared=body_declared, context_declared=frame_context)+ elif isinstance(body_node, nodes.Macro):+ self.walk(body_node, guarded=guarded, declared=declared | look_ahead, context_declared=context_declared)+ else:+ self.walk(body_node, guarded=guarded, declared=declared, context_declared=context_declared)+ declared |= definitely_assigned_names(body_node)+ return declared++ def _walk_if(+ self,+ node: nodes.If,+ *,+ guarded: frozenset[str],+ declared: frozenset[str],+ context_declared: frozenset[str] | None,+ look_ahead: frozenset[str],+ ) -> None:+ """An `if` opens no frame: each branch, and each `elif` test, starts from what is declared before it."""+ for branch in (node, *node.elif_):+ self._walk_test(branch.test, guarded=guarded, declared=declared, context_declared=context_declared)+ branch_guarded = guarded | self._guard_vars(branch.test)+ self._walk_body(branch.body, guarded=branch_guarded, declared=declared, context_declared=context_declared, look_ahead=look_ahead)+ self._walk_body(node.else_, guarded=guarded, declared=declared, context_declared=context_declared, look_ahead=look_ahead)++ def walk(+ self,+ node: nodes.Node,+ *,+ guarded: frozenset[str],+ declared: frozenset[str],+ context_declared: frozenset[str] | None,+ ) -> None:+ # What the template context holds in a frame opened here: from the top level, what is declared where it opens+ frame_context = declared if context_declared is None else context_declared- def walk(self, node: nodes.Node, *, guarded: frozenset[str], declared: frozenset[str]) -> None:if isinstance(node, nodes.Template):- self._walk_body(node.body, guarded=guarded, declared=declared)- return-- if isinstance(node, nodes.If):- body_guarded = guarded | self._guard_vars(node.test)- self._walk_test(node.test, guarded=guarded, declared=declared)- self._walk_body(node.body, guarded=body_guarded, declared=declared)- for elif_node in node.elif_:- self.walk(elif_node, guarded=guarded, declared=declared)- self._walk_body(node.else_, guarded=guarded, declared=declared)+ self._walk_body(node.body, guarded=guarded, declared=declared, context_declared=context_declared)returnif isinstance(node, nodes.For):# The iterable is evaluated before the loop targets bind.- self.walk(node.iter, guarded=guarded, declared=declared)- loop_declared: set[str] = {"loop"}- if isinstance(node.target, nodes.Name):- loop_declared.add(node.target.name)- elif isinstance(node.target, nodes.Tuple):- for item in node.target.items:- if isinstance(item, nodes.Name):- loop_declared.add(item.name)+ self.walk(node.iter, guarded=guarded, declared=declared, context_declared=context_declared)+ loop_declared = declared | frame_bound_names(node)if node.test is not None:# The loop filter evaluates after the target binds per item, so a shadowing# target stays a local while an optional read in the filter gets classified.# Its guard vars deliberately do NOT bless the body (same conservative stance# as else-arms and inverted guards).- self._walk_test(node.test, guarded=guarded, declared=declared | loop_declared)- self._walk_body(node.body, guarded=guarded, declared=declared | loop_declared)- self._walk_body(node.else_, guarded=guarded, declared=declared)+ self._walk_test(node.test, guarded=guarded, declared=loop_declared, context_declared=frame_context)+ self._walk_body(node.body, guarded=guarded, declared=loop_declared, context_declared=frame_context)+ self._walk_body(node.else_, guarded=guarded, declared=declared, context_declared=frame_context)returnif isinstance(node, nodes.With):@@ -180,31 +214,77 @@ class _GuardWalker:# locals, never the shadowed optionals. Walking the targets as children would# miscount the store-context Names as reads.for value in node.values:- self.walk(value, guarded=guarded, declared=declared)- with_declared = frozenset(target.name for target in node.targets if isinstance(target, nodes.Name))- self._walk_body(node.body, guarded=guarded, declared=declared | with_declared)+ self.walk(value, guarded=guarded, declared=declared, context_declared=context_declared)+ with_declared = declared | frame_bound_names(node)+ self._walk_body(node.body, guarded=guarded, declared=with_declared, context_declared=frame_context)return- if isinstance(node, nodes.Macro):- macro_declared = frozenset(arg.name for arg in node.args)- self._walk_body(node.body, guarded=guarded, declared=declared | macro_declared)+ if isinstance(node, (nodes.Macro, nodes.CallBlock)):+ if isinstance(node, nodes.CallBlock):+ # The call is made from the enclosing scope; the block's body is the macro it passes as `caller`.+ self.walk(node.call, guarded=guarded, declared=declared, context_declared=context_declared)+ # The arguments are locals of the macro, and a default is evaluated where they are bound.+ macro_declared = declared | frame_bound_names(node)+ for default in node.defaults:+ self.walk(default, guarded=guarded, declared=macro_declared, context_declared=frame_context)+ self._walk_body(node.body, guarded=guarded, declared=macro_declared, context_declared=frame_context)+ return++ if isinstance(node, nodes.FilterBlock):+ # The body runs in a frame of its own, so what it sets stays in it, and the filter is+ # applied in that frame once the body has run.+ body_declared = self._walk_body(node.body, guarded=guarded, declared=declared, context_declared=frame_context)+ self.walk(node.filter, guarded=guarded, declared=body_declared, context_declared=frame_context)+ return++ if isinstance(node, nodes.Scope):+ # `{% autoescape %}` parses into a scope: a frame of its own.+ self._walk_body(node.body, guarded=guarded, declared=declared, context_declared=frame_context)+ return++ if isinstance(node, nodes.OverlayScope):+ self.walk(node.context, guarded=guarded, declared=declared, context_declared=context_declared)+ self._walk_body(node.body, guarded=guarded, declared=declared, context_declared=frame_context)+ return++ if isinstance(node, nodes.ScopedEvalContextModifier):+ # Its statements run in the frame it stands in.+ for option in node.options:+ self.walk(option, guarded=guarded, declared=declared, context_declared=context_declared)+ self._walk_body(node.body, guarded=guarded, declared=declared, context_declared=context_declared)+ return++ if isinstance(node, nodes.Block):+ if node.scoped:+ # Called with a context derived from the names declared where it stands, which is+ # then the context of the blocks nested in it.+ self._walk_body(node.body, guarded=guarded, declared=declared, context_declared=declared)+ else:+ # Called with the template context alone: a guard on a name a frame around it binds+ # tested that local, not the context's value the block reads.+ block_guarded = guarded - (declared - frame_context)+ self._walk_body(node.body, guarded=block_guarded, declared=frame_context, context_declared=frame_context)returnif isinstance(node, nodes.CondExpr):expr1_guarded = guarded | self._guard_vars(node.test)- self._walk_test(node.test, guarded=guarded, declared=declared)- self.walk(node.expr1, guarded=expr1_guarded, declared=declared)+ self._walk_test(node.test, guarded=guarded, declared=declared, context_declared=context_declared)+ self.walk(node.expr1, guarded=expr1_guarded, declared=declared, context_declared=context_declared)if node.expr2 is not None:- self.walk(node.expr2, guarded=guarded, declared=declared)+ self.walk(node.expr2, guarded=guarded, declared=declared, context_declared=context_declared)+ return++ if isinstance(node, nodes.Name) and node.ctx != "load":+ # The target of a `set` or a `for`, and a macro argument, are written, not read.returnif isinstance(node, (nodes.Name, nodes.Getattr)):- full_path = _build_full_path(node)+ full_path = dotted_attribute_path(node)if full_path is None:# Not a plain Name/Getattr chain (e.g. an attribute on a subscript or call# result): keep walking inward so inner references still get classified.for child in node.iter_child_nodes():- self.walk(child, guarded=guarded, declared=declared)+ self.walk(child, guarded=guarded, declared=declared, context_declared=context_declared)returnif self._is_unguarded_optional(full_path, guarded=guarded, declared=declared):self._record(full_path)@@ -212,7 +292,7 @@ class _GuardWalker:returnfor child in node.iter_child_nodes():- self.walk(child, guarded=guarded, declared=declared)+ self.walk(child, guarded=guarded, declared=declared, context_declared=context_declared)def detect_unguarded_optional_references(@@ -244,5 +324,5 @@ def detect_unguarded_optional_references(raise Jinja2DetectVariablesError(msg) from syntax_errorwalker = _GuardWalker(optional_variable_names=optional_variable_names)- walker.walk(parsed_ast, guarded=frozenset(), declared=frozenset())+ walker.walk(parsed_ast, guarded=frozenset(), declared=frozenset(), context_declared=None)return walker.findingsdiff --git a/pipelex/tools/jinja2/jinja2_required_variables.py b/pipelex/tools/jinja2/jinja2_required_variables.pyindex 8bd4e4755..96497e2cf 100644--- a/pipelex/tools/jinja2/jinja2_required_variables.py+++ b/pipelex/tools/jinja2/jinja2_required_variables.py@@ -1,5 +1,11 @@+"""The variables a template reads from its inputs, and the references through which it attaches images and documents.++Both detectors are handlers on the read walk of `jinja2_scopes.py`, which follows Jinja's own scopes: a name the+template binds (a `{% set %}`, a loop target, a macro argument) is not reported where Jinja binds it, and is+reported as the input of that name everywhere else, such as after an `if` whose branches do not all set it.+"""+from dataclasses import field-from typing import Protocolfrom jinja2 import nodesfrom jinja2.exceptions import (@@ -10,6 +16,7 @@ from pydantic.dataclasses import dataclassfrom pipelex.tools.jinja2.exceptions import Jinja2DetectVariablesError, Jinja2StuffErrorfrom pipelex.tools.jinja2.jinja2_environment import make_jinja2_env_without_loader+from pipelex.tools.jinja2.jinja2_scopes import ScopeBindings, dotted_attribute_path, walk_template_readsfrom pipelex.tools.jinja2.template_category import TemplateCategoryfrom pipelex.tools.misc.string_utils import get_root_from_dotted_path@@ -30,107 +37,6 @@ class VariableReference:filters: list[str] = field(default_factory=list[str])-def _build_full_path(node: nodes.Node) -> str | None:- """Recursively build the full dotted path from a Getattr or Name node.-- Args:- node: A Jinja2 AST node (Name or Getattr)-- Returns:- The full dotted path as a string, or None if the node structure is not supported.- """- if isinstance(node, nodes.Name):- return node.name- if isinstance(node, nodes.Getattr):- parent_path = _build_full_path(node.node)- if parent_path is not None:- return f"{parent_path}.{node.attr}"- return None---def _set_target_name(*, statement: nodes.Node) -> str | None:- """The name a `{% set %}` or a `{% set %}…{% endset %}` block assigns, when it assigns a plain name."""- if isinstance(statement, (nodes.Assign, nodes.AssignBlock)) and isinstance(statement.target, nodes.Name):- return statement.target.name- return None---class _ReadHandler(Protocol):- """Deals with a node that may read a variable, given the names declared where it stands.-- Returns True when it has dealt with the node, which is then not walked into.- """-- def __call__(self, *, node: nodes.Node, declared_names: set[str]) -> bool: ...---def _walk_reads(*, node: nodes.Node, declared_names: set[str], global_names: set[str], handle_read: _ReadHandler) -> None:- """Walk a template's AST in Jinja's scopes, handing every node that may read a variable to ``handle_read``.-- Both detectors below walk through here, so the names they treat as declared cannot drift apart: the required- variables the input check compares with the inputs, and the references that attach images and documents.-- Args:- node: The current AST node- declared_names: The names declared where the node stands (a `set` before it, a loop target, a macro argument)- global_names: The environment's globals (`range`, `namespace`, `dict`...)- handle_read: What to do with a node that may read a variable- """- if isinstance(node, nodes.Template):- # A macro's name is never an input, and a macro body, which runs when the macro is called, sees every- # top-level `set`. Elsewhere a `set` declares its name only from its own statement on:- # `{% set topic = topic|trim %}` reads the incoming `topic`, and so does a read placed before the `set`- top_level_set_names = {name for statement in node.body if (name := _set_target_name(statement=statement)) is not None}- scope_declared = declared_names | {statement.name for statement in node.body if isinstance(statement, nodes.Macro)}- for statement in node.body:- statement_declared = scope_declared | top_level_set_names if isinstance(statement, nodes.Macro) else scope_declared- _walk_reads(node=statement, declared_names=statement_declared, global_names=global_names, handle_read=handle_read)- if (name := _set_target_name(statement=statement)) is not None:- scope_declared.add(name)- return-- if isinstance(node, nodes.For):- # The iterable and the `else` branch are read outside the loop, the body and the loop filter inside it,- # where the loop target and the special `loop` variable are declared- loop_declared = declared_names | {"loop"}- if isinstance(node.target, nodes.Name):- loop_declared.add(node.target.name)- elif isinstance(node.target, nodes.Tuple):- loop_declared.update(item.name for item in node.target.items if isinstance(item, nodes.Name))- # Walked in source order, which is the order the image references are attached in- scoped_nodes = [- (node.iter, declared_names),- *((statement, loop_declared) for statement in node.body),- *((statement, declared_names) for statement in node.else_),- *([(node.test, loop_declared)] if node.test is not None else []),- ]- for scoped_node, scope_declared_names in scoped_nodes:- _walk_reads(node=scoped_node, declared_names=scope_declared_names, global_names=global_names, handle_read=handle_read)- return-- if isinstance(node, nodes.Macro):- # Macro parameters, and the names Jinja provides inside a macro body, are locally declared- declared_names = declared_names | {arg.name for arg in node.args} | {"caller", "varargs", "kwargs"}-- if isinstance(node, nodes.Name) and node.ctx != "load":- # The target of a `set` or a `for` is written, not read- return-- if isinstance(node, nodes.Call) and isinstance(node.node, nodes.Name) and node.node.name in global_names:- # A call to a Jinja global (`range(count)`, `namespace()`) reads only its arguments. Any other read of the- # name (`{{ range }}`, `range.low`) reads the input of that name, which shadows the global when rendered- for child in node.iter_child_nodes():- if child is not node.node:- _walk_reads(node=child, declared_names=declared_names, global_names=global_names, handle_read=handle_read)- return-- if handle_read(node=node, declared_names=declared_names):- return-- for child in node.iter_child_nodes():- _walk_reads(node=child, declared_names=declared_names, global_names=global_names, handle_read=handle_read)--def detect_jinja2_required_variables(*,template_category: TemplateCategory,@@ -170,19 +76,19 @@ def detect_jinja2_required_variables(paths: set[str] = set()- def collect_full_path(*, node: nodes.Node, declared_names: set[str]) -> bool:+ def collect_full_path(*, node: nodes.Node, bindings: ScopeBindings) -> bool:# Only the full path of an access chain is collected: `{{ foo.bar.baz }}` gives `foo.bar.baz`, not `foo.bar` or `foo`if not isinstance(node, (nodes.Name, nodes.Getattr)):return False- full_path = _build_full_path(node)+ full_path = dotted_attribute_path(node)if full_path is None:# An attribute on a subscript, a call or a filter (`items[0].text`) reads the path its chain starts fromreturn False- if get_root_from_dotted_path(full_path) not in declared_names:+ if get_root_from_dotted_path(full_path) not in bindings:paths.add(full_path)return True- _walk_reads(node=parsed_ast, declared_names=set(), global_names=set(jinja2_env.globals), handle_read=collect_full_path)+ walk_template_reads(template=parsed_ast, global_names=set(jinja2_env.globals), handle_read=collect_full_path)return paths@@ -248,7 +154,7 @@ def detect_jinja2_variable_references(references: dict[str, VariableReference] = {}- def collect_reference(*, node: nodes.Node, declared_names: set[str]) -> bool:+ def collect_reference(*, node: nodes.Node, bindings: ScopeBindings) -> bool:# A filter chain is taken whole, with its filter names; a chain or a path that stops at a subscript or a# call (`items[0].image`) gives no reference, since no image or document can be resolved from itif isinstance(node, nodes.Filter):@@ -257,8 +163,8 @@ def detect_jinja2_variable_references(filters, base_node = [], nodeelse:return False- full_path = _build_full_path(base_node) if base_node is not None else None- if full_path is None or get_root_from_dotted_path(full_path) in declared_names:+ full_path = dotted_attribute_path(base_node) if base_node is not None else None+ if full_path is None or get_root_from_dotted_path(full_path) in bindings:return Trueif full_path in references:# The same variable referenced several times combines its filters@@ -269,5 +175,5 @@ def detect_jinja2_variable_references(references[full_path] = VariableReference(path=full_path, filters=filters)return True- _walk_reads(node=parsed_ast, declared_names=set(), global_names=set(jinja2_env.globals), handle_read=collect_reference)+ walk_template_reads(template=parsed_ast, global_names=set(jinja2_env.globals), handle_read=collect_reference)return list(references.values())diff --git a/pipelex/tools/jinja2/jinja2_scopes.py b/pipelex/tools/jinja2/jinja2_scopes.pynew file mode 100644index 000000000..09cc45194--- /dev/null+++ b/pipelex/tools/jinja2/jinja2_scopes.py@@ -0,0 +1,340 @@+"""Where Jinja binds a name a template sets, and the one read walk the template detectors share.++The input check, the image and document references and PipeDocGen's field check all ask which of the names a+template reads come from its inputs. A name the template binds itself (a `{% set %}`, a loop target, a macro+argument) does not, from where Jinja binds it, and the rule is stated here once, as Jinja's compiler applies it+(`jinja2/idtracking.py`):++- A loop body, a loop's `else` branch, a macro body, a call block, a filter block, a set block's body and a+ `with` body each run in a frame of their own, and so does `{% autoescape %}`, which Jinja parses into a scope:+ a name set inside one is not bound after it.+- An `if` opens no frame, and each of its branches starts from the names bound before it. After the `if`, a name+ every branch sets, the `else` included, is bound; a name only some branches set is the input of that name on+ every path where none of them ran, so a read of it after the `if` reads that input.+- Within a frame, a name is bound from the statement that binds it on: a read placed before a `set`, and the+ `set`'s own right-hand side (`{% set topic = topic|trim %}`), read the input.+- A macro body runs where the macro is called, which the walk cannot place among the statements, so it sees the+ names its frame binds for certain, wherever they stand in it, an `if` branch it is defined in opening no frame of+ its own. A name the frame reads before binding it is the exception: Jinja starts the frame's copy of it from the+ input, so a macro called before the binding reads the input.+- A `{% block %}` runs as a function of its own. Without `scoped` it reads only the template context, which holds+ what the top level has set for certain before it, and none of the names a loop, a macro or any other frame around+ it binds; a scoped block reads the names bound where it stands, and the blocks nested in it read those too.++The walk hands every node that may read a variable to a handler, with the bindings where the node stands. A+binding maps a name to the input path it stands for when that can be followed, which is a loop target over an+attribute chain, and to None otherwise.+"""++from typing import Protocol++from jinja2 import nodes++# The segment standing for "an item of the list" in a path reached through a loop variable.+LIST_ITEM_SEGMENT = "[]"++# The names a template binds where a node stands, each mapped to the input path it stands for, or to None when it+# cannot be followed (a `set`, a macro argument, `loop`). A name absent from the bindings is an input, or a global.+ScopeBindings = dict[str, tuple[str, ...] | None]+++def assigned_target_names(target: nodes.Node) -> list[str]:+ """The names a `set`, `with` or loop target binds: a plain name or a tuple of them, nested tuples included.++ A namespace attribute (`{% set ns.found = true %}`) binds no name.+ """+ if isinstance(target, nodes.Name):+ return [target.name]+ if isinstance(target, nodes.Tuple):+ return [name for item in target.items for name in assigned_target_names(item)]+ return []+++def names_assigned_by_statements(statements: list[nodes.Node]) -> set[str]:+ names: set[str] = set()+ for statement in statements:+ names |= definitely_assigned_names(statement)+ return names+++def definitely_assigned_names(statement: nodes.Node) -> set[str]:+ """The names a statement leaves bound, whichever way it runs, in the frame it runs in.++ A set or a set block binds its targets, a macro its name, an import its alias; an `if` binds what every one of+ its branches binds, nothing when it has no `else`; the statements of an `autoescape` run in the frame they stand+ in; and a statement that opens a frame of its own binds nothing outside it.+ """+ if isinstance(statement, (nodes.Assign, nodes.AssignBlock)):+ return set(assigned_target_names(statement.target))+ if isinstance(statement, nodes.Macro):+ return {statement.name}+ if isinstance(statement, nodes.Import):+ return {statement.target}+ if isinstance(statement, nodes.FromImport):+ return {name[1] if isinstance(name, tuple) else name for name in statement.names}+ if isinstance(statement, nodes.If):+ if not statement.else_:+ return set()+ body_names = names_assigned_by_statements(statement.body)+ other_branches = [*(elif_branch.body for elif_branch in statement.elif_), statement.else_]+ return body_names.intersection(*(names_assigned_by_statements(branch) for branch in other_branches))+ if isinstance(statement, nodes.ScopedEvalContextModifier):+ return names_assigned_by_statements(statement.body)+ return set()+++def _frame_level_parts(statement: nodes.Node) -> list[nodes.Node]:+ """The expressions of a statement Jinja reads in the frame the statement stands in."""+ if isinstance(statement, nodes.Output):+ return list(statement.nodes)+ if isinstance(statement, (nodes.Assign, nodes.ExprStmt)):+ return [statement.node]+ if isinstance(statement, nodes.For):+ return [statement.iter]+ if isinstance(statement, nodes.With):+ return list(statement.values)+ if isinstance(statement, nodes.CallBlock):+ return [statement.call]+ if isinstance(statement, nodes.FilterBlock):+ return [statement.filter]+ if isinstance(statement, nodes.OverlayScope):+ return [statement.context]+ if isinstance(statement, (nodes.Import, nodes.FromImport)):+ return [statement.template]+ return []+++def _loaded_names(expression: nodes.Node) -> set[str]:+ names = [expression, *expression.find_all(nodes.Name)]+ return {name.name for name in names if isinstance(name, nodes.Name) and name.ctx == "load"}+++def _collect_reads_before_bound(*, statements: list[nodes.Node], bound: set[str], read: set[str]) -> None:+ bound = set(bound)+ for statement in statements:+ if isinstance(statement, nodes.If):+ for branch in (statement, *statement.elif_):+ read |= _loaded_names(branch.test) - bound+ _collect_reads_before_bound(statements=branch.body, bound=bound, read=read)+ _collect_reads_before_bound(statements=statement.else_, bound=bound, read=read)+ elif isinstance(statement, nodes.ScopedEvalContextModifier):+ _collect_reads_before_bound(statements=statement.body, bound=bound, read=read)+ else:+ for part in _frame_level_parts(statement):+ read |= _loaded_names(part) - bound+ bound |= definitely_assigned_names(statement)+++def names_read_before_bound(statements: list[nodes.Node]) -> set[str]:+ """The names a frame reads at its own level before it binds them for certain.++ Jinja starts the frame's copy of such a name from the input of that name, so a macro the frame defines reads the+ input when it is called before the binding.+ """+ read: set[str] = set()+ _collect_reads_before_bound(statements=statements, bound=set(), read=read)+ return read+++def frame_bound_names(statement: nodes.Node) -> set[str]:+ """The names a statement binds in the frame it opens, before its body runs: a loop's target and `loop`, a+ `with`'s targets, and a macro's or a call block's arguments with the names Jinja provides in a macro.+ """+ if isinstance(statement, nodes.For):+ return {"loop", *assigned_target_names(statement.target)}+ if isinstance(statement, nodes.With):+ return {name for target in statement.targets for name in assigned_target_names(target)}+ if isinstance(statement, (nodes.Macro, nodes.CallBlock)):+ return {*(argument.name for argument in statement.args), "caller", "varargs", "kwargs"}+ return set()+++def attribute_chain(node: nodes.Node) -> tuple[str, list[str]] | None:+ """The name and attribute names of a pure attribute chain (`a.b.c`), or None for any other shape."""+ attributes: list[str] = []+ current = node+ while isinstance(current, nodes.Getattr):+ attributes.append(current.attr)+ current = current.node+ if isinstance(current, nodes.Name):+ return current.name, list(reversed(attributes))+ return None+++def dotted_attribute_path(node: nodes.Node) -> str | None:+ """The dotted path of a pure attribute chain (`user.profile.name`), or None for any other shape."""+ chain = attribute_chain(node)+ if chain is None:+ return None+ name, attributes = chain+ return ".".join([name, *attributes])+++def resolve_bound_path(*, name: str, attributes: list[str], bindings: ScopeBindings) -> tuple[str, ...] | None:+ """The input path a chain reads, through the bindings: None when it starts from a name that cannot be followed."""+ if name not in bindings:+ return (name, *attributes)+ bound_path = bindings[name]+ if bound_path is None:+ return None+ return (*bound_path, *attributes)+++class ReadHandler(Protocol):+ """Deals with a node that may read a variable, given the bindings where it stands.++ Returns True when it has dealt with the node, which is then not walked into.+ """++ def __call__(self, *, node: nodes.Node, bindings: ScopeBindings) -> bool: ...+++def _with_unfollowed(*, bindings: ScopeBindings, names: set[str]) -> ScopeBindings:+ extended = dict(bindings)+ for name in names:+ extended[name] = None+ return extended+++class _ReadWalk:+ def __init__(self, *, global_names: set[str], handle_read: ReadHandler) -> None:+ self.global_names = global_names+ self.handle_read = handle_read++ def walk_statements(+ self,+ *,+ statements: list[nodes.Node],+ bindings: ScopeBindings,+ context: ScopeBindings | None,+ look_ahead: set[str] | None = None,+ ) -> ScopeBindings:+ """Walk a statement list in the frame it runs in, and return the bindings after it.++ Each statement sees what the statements before it bound for certain. A macro also sees `look_ahead`, what its+ frame binds for certain wherever it stands, but for the names the frame reads before binding them; it is+ computed from the statements when they make up a frame of their own, and given for the branches of an `if`.+ `context` is what the template context holds where the frame runs, and None in the top-level frame, whose+ names the context takes on as they are set.+ """+ if look_ahead is None:+ look_ahead = names_assigned_by_statements(statements) - names_read_before_bound(statements)+ scope = dict(bindings)+ for statement in statements:+ if isinstance(statement, nodes.If):+ self._walk_if(node=statement, bindings=scope, context=context, look_ahead=look_ahead)+ elif isinstance(statement, nodes.Macro):+ self.walk(node=statement, bindings=_with_unfollowed(bindings=scope, names=look_ahead), context=context)+ else:+ self.walk(node=statement, bindings=scope, context=context)+ scope = _with_unfollowed(bindings=scope, names=definitely_assigned_names(statement))+ return scope++ def _walk_if(self, *, node: nodes.If, bindings: ScopeBindings, context: ScopeBindings | None, look_ahead: set[str]) -> None:+ # An `if` opens no frame: every branch, and every `elif` test, starts from the bindings before it+ for branch in (node, *node.elif_):+ self.walk(node=branch.test, bindings=bindings, context=context)+ self.walk_statements(statements=branch.body, bindings=bindings, context=context, look_ahead=look_ahead)+ self.walk_statements(statements=node.else_, bindings=bindings, context=context, look_ahead=look_ahead)++ def walk(self, *, node: nodes.Node, bindings: ScopeBindings, context: ScopeBindings | None) -> None:+ # What the template context holds in a frame opened here: from the top level, what is bound where it opens+ frame_context = bindings if context is None else context++ if isinstance(node, nodes.For):+ # The iterable is read outside the loop, the loop filter and the body inside it, and the `else` branch in+ # a frame of its own. A single loop target over an attribute chain reads an item of that chain's list+ self.walk(node=node.iter, bindings=bindings, context=context)+ loop_bindings = _with_unfollowed(bindings=bindings, names=frame_bound_names(node))+ iterated_chain = attribute_chain(node.iter)+ if isinstance(node.target, nodes.Name) and iterated_chain is not None:+ iterated_name, iterated_attributes = iterated_chain+ iterated_path = resolve_bound_path(name=iterated_name, attributes=iterated_attributes, bindings=bindings)+ if iterated_path is not None:+ loop_bindings[node.target.name] = (*iterated_path, LIST_ITEM_SEGMENT)+ if node.test is not None:+ self.walk(node=node.test, bindings=loop_bindings, context=frame_context)+ self.walk_statements(statements=node.body, bindings=loop_bindings, context=frame_context)+ self.walk_statements(statements=node.else_, bindings=bindings, context=frame_context)+ return++ if isinstance(node, nodes.With):+ # The values are read outside the `with`, before its targets are bound+ for value in node.values:+ self.walk(node=value, bindings=bindings, context=context)+ with_bindings = _with_unfollowed(bindings=bindings, names=frame_bound_names(node))+ self.walk_statements(statements=node.body, bindings=with_bindings, context=frame_context)+ return++ if isinstance(node, (nodes.Macro, nodes.CallBlock)):+ if isinstance(node, nodes.CallBlock):+ # The call is made from the enclosing frame; the block's body is the macro it passes as `caller`+ self.walk(node=node.call, bindings=bindings, context=context)+ macro_bindings = _with_unfollowed(bindings=bindings, names=frame_bound_names(node))+ # A default is evaluated in the macro's frame, where the arguments are bound+ for default in node.defaults:+ self.walk(node=default, bindings=macro_bindings, context=frame_context)+ self.walk_statements(statements=node.body, bindings=macro_bindings, context=frame_context)+ return++ if isinstance(node, (nodes.FilterBlock, nodes.AssignBlock)):+ # The body runs in a frame of its own, and the filter is applied in that frame once the body has run+ body_bindings = self.walk_statements(statements=node.body, bindings=bindings, context=frame_context)+ if node.filter is not None:+ self.walk(node=node.filter, bindings=body_bindings, context=frame_context)+ return++ if isinstance(node, nodes.Scope):+ self.walk_statements(statements=node.body, bindings=bindings, context=frame_context)+ return++ if isinstance(node, nodes.Block):+ # A block is called with the template context, or, when scoped, with one derived from the names bound where+ # it stands, which is then the context of the blocks nested in it+ block_bindings = bindings if node.scoped else frame_context+ self.walk_statements(statements=node.body, bindings=block_bindings, context=block_bindings)+ return++ if isinstance(node, nodes.OverlayScope):+ self.walk(node=node.context, bindings=bindings, context=context)+ self.walk_statements(statements=node.body, bindings=bindings, context=frame_context)+ return++ if isinstance(node, nodes.ScopedEvalContextModifier):+ # Its statements run in the frame it stands in, the scope Jinja wraps an `autoescape` in+ for option in node.options:+ self.walk(node=option, bindings=bindings, context=context)+ self.walk_statements(statements=node.body, bindings=bindings, context=context)+ return++ if isinstance(node, nodes.Name) and node.ctx != "load":+ # The target of a `set` or a `for`, and a macro argument, are written, not read+ return++ if isinstance(node, nodes.Call) and isinstance(node.node, nodes.Name) and node.node.name in self.global_names:+ # A call to a Jinja global (`range(count)`, `namespace()`) reads only its arguments. Any other read of the+ # name (`{{ range }}`, `range.low`) reads the input of that name, which shadows the global when rendered+ for child in node.iter_child_nodes():+ if child is not node.node:+ self.walk(node=child, bindings=bindings, context=context)+ return++ if self.handle_read(node=node, bindings=bindings):+ return++ for child in node.iter_child_nodes():+ self.walk(node=child, bindings=bindings, context=context)+++def walk_template_reads(*, template: nodes.Template, global_names: set[str], handle_read: ReadHandler) -> None:+ """Walk a template in Jinja's scopes, handing every node that may read a variable to `handle_read`.++ The detectors that ask which inputs a template reads, and which fields of them, all walk through here, so the+ names they treat as bound cannot drift apart.++ Args:+ template: The parsed template+ global_names: The environment's globals (`range`, `namespace`, `dict`...)+ handle_read: What to do with a node that may read a variable+ """+ _ReadWalk(global_names=global_names, handle_read=handle_read).walk_statements(statements=template.body, bindings={}, context=None)diff --git a/subject_grants.toml b/subject_grants.tomlindex 79a9d5731..7807388f0 100644--- a/subject_grants.toml+++ b/subject_grants.toml@@ -6108,14 +6108,6 @@ rationale = "Factory from source: builds the result from the template_category;param = "template_category"rationale = "Factory from source: builds the result from the template_category; single operand."-["pipelex/tools/jinja2/jinja2_field_paths.py::_chain"]-param = "node"-rationale = "Reads the attribute chain of one Jinja node."--["pipelex/tools/jinja2/jinja2_field_paths.py::_target_names"]-param = "target"-rationale = "Lists the names one assignment target binds."-["pipelex/tools/jinja2/jinja2_filters.py::escape_script_tag"]param = "value"rationale = "Transforms the value; single operand."@@ -6124,10 +6116,6 @@ rationale = "Transforms the value; single operand."param = "value"rationale = "Jinja filter: Jinja passes the filtered value positionally."-["pipelex/tools/jinja2/jinja2_optional_guards.py::_GuardWalker._assign_target_names"]-param = "target"-rationale = "Noun-named derivation: the assigned names of the target node; single operand."-["pipelex/tools/jinja2/jinja2_optional_guards.py::_GuardWalker._guard_vars"]param = "test_node"rationale = "Noun-named derivation: the guarded vars of the test node; single operand."@@ -6144,6 +6132,10 @@ rationale = "Verb-object: records the path; single operand."param = "body_nodes"rationale = "Visitor: walks the body nodes; guard sets stay keyword."+["pipelex/tools/jinja2/jinja2_optional_guards.py::_GuardWalker._walk_if"]+param = "node"+rationale = "Verb-object walk over one if node, like the walker's other walk methods; single operand."+["pipelex/tools/jinja2/jinja2_optional_guards.py::_GuardWalker._walk_test"]param = "test_node"rationale = "Visitor: walks the test node; guard sets stay keyword."@@ -6152,10 +6144,6 @@ rationale = "Visitor: walks the test node; guard sets stay keyword."param = "node"rationale = "Visitor: walks the node; guard sets stay keyword."-["pipelex/tools/jinja2/jinja2_optional_guards.py::_build_full_path"]-param = "node"-rationale = "Noun-named derivation: the full dotted path of the node; single operand."-["pipelex/tools/jinja2/jinja2_parsing.py::check_jinja2_parsing"]param = "template_source"rationale = "Jinja op on the template_source; single operand (context keyword-only)."@@ -6280,13 +6268,45 @@ rationale = "Jinja op on the template_source; single operand (context keyword-onparam = "template_source"rationale = "Jinja op on the template_source; single operand (context keyword-only)."-["pipelex/tools/jinja2/jinja2_required_variables.py::_build_full_path"]+["pipelex/tools/jinja2/jinja2_required_variables.py::_extract_filters_and_variable"]param = "node"rationale = "Extracts/derives from the node; single operand."-["pipelex/tools/jinja2/jinja2_required_variables.py::_extract_filters_and_variable"]+["pipelex/tools/jinja2/jinja2_scopes.py::_frame_level_parts"]+param = "statement"+rationale = "Noun-named derivation over one Jinja AST operand; single operand."++["pipelex/tools/jinja2/jinja2_scopes.py::_loaded_names"]+param = "expression"+rationale = "Noun-named derivation over one Jinja AST operand; single operand."++["pipelex/tools/jinja2/jinja2_scopes.py::assigned_target_names"]+param = "target"+rationale = "Lists the names one assignment target binds."++["pipelex/tools/jinja2/jinja2_scopes.py::attribute_chain"]param = "node"-rationale = "Extracts/derives from the node; single operand."+rationale = "Reads the attribute chain of one Jinja node."++["pipelex/tools/jinja2/jinja2_scopes.py::definitely_assigned_names"]+param = "statement"+rationale = "Noun-named derivation: the names one statement binds for certain; single operand."++["pipelex/tools/jinja2/jinja2_scopes.py::dotted_attribute_path"]+param = "node"+rationale = "Noun-named derivation: the dotted path of the node; single operand."++["pipelex/tools/jinja2/jinja2_scopes.py::frame_bound_names"]+param = "statement"+rationale = "Noun-named derivation: the names one statement binds in the frame it opens; single operand."++["pipelex/tools/jinja2/jinja2_scopes.py::names_assigned_by_statements"]+param = "statements"+rationale = "Noun-named derivation: the names a statement list binds for certain; single operand."++["pipelex/tools/jinja2/jinja2_scopes.py::names_read_before_bound"]+param = "statements"+rationale = "Noun-named derivation over one Jinja AST operand; single operand."["pipelex/tools/jinja2/jinja2_template_loader.py::TemplateLoader.is_loaded"]param = "name"diff --git a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_load_refusals.py b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_load_refusals.pyindex 83d5a3ffd..5677abb1d 100644--- a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_load_refusals.py+++ b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_load_refusals.py@@ -77,8 +77,9 @@ class TestPipeDocGenLoadRefusals:('format = "pdf"\ntemplate = "{% for line in invoice.line_items %}{{ line.amout }}{% endfor %}"', "amout"),('format = "pdf"\nfilename = "invoice-{{ invoice.nmber }}"', "nmber"),('format = "pdf"\ntemplate = "<h1>{{ invoice._stuff }}</h1>"', "_stuff"),+ ('format = "pdf"\ntemplate = "{% if invoice.number %}{% set invoice = \'none\' %}{% endif %}<h1>{{ invoice.nosuch }}</h1>"', "nosuch"),],- ids=["misspelled_field", "misspelled_field_through_a_loop", "misspelled_field_in_the_filename", "private_name"],+ ids=["misspelled_field", "misspelled_field_through_a_loop", "misspelled_field_in_the_filename", "private_name", "set_in_only_some_branches"],)async def test_a_template_mistake_is_refused_at_load(self, step_fields: str, expected: str) -> None:"""A field a template or the filename names is checked against the input's concept, through loop variables too."""diff --git a/tests/unit/pipelex/pipe_operators/pipe_compose/data.py b/tests/unit/pipelex/pipe_operators/pipe_compose/data.pyindex aa2709f1f..65475898f 100644--- a/tests/unit/pipelex/pipe_operators/pipe_compose/data.py+++ b/tests/unit/pipelex/pipe_operators/pipe_compose/data.py@@ -257,6 +257,15 @@ class PipeComposeInputCheckTestCases:"self_referential_set",{"description": "d", "inputs": {"topic": "Text"}, "output": "Text", "template": "{% set topic = topic|trim %}About {{ topic }}"},),+ (+ "set_in_every_branch_of_an_if",+ {+ "description": "d",+ "inputs": {"flag": "Text"},+ "output": "Text",+ "template": "{% if flag %}{% set label = 'on' %}{% else %}{% set label = 'off' %}{% endif %}Status: {{ label }}",+ },+ ),("input_named_like_a_jinja_global",{"description": "d", "inputs": {"namespace": "Text"}, "output": "Text", "template": "List the pods in $namespace"},diff --git a/tests/unit/pipelex/pipe_operators/pipe_doc_gen/test_pipe_doc_gen_blueprint.py b/tests/unit/pipelex/pipe_operators/pipe_doc_gen/test_pipe_doc_gen_blueprint.pyindex a9578113a..d50a5921f 100644--- a/tests/unit/pipelex/pipe_operators/pipe_doc_gen/test_pipe_doc_gen_blueprint.py+++ b/tests/unit/pipelex/pipe_operators/pipe_doc_gen/test_pipe_doc_gen_blueprint.py@@ -43,6 +43,13 @@ class TestPipeDocGenBlueprint:with pytest.raises(ValidationError, match=expected_message):_blueprint(**overrides)+ def test_a_name_every_branch_sets_is_the_template_s_own(self) -> None:+ template = (+ "{% if invoice.paid %}{% set status = 'Paid' %}{% else %}{% set status = 'Due' %}{% endif %}<h1>{{ invoice.number }}: {{ status }}</h1>"+ )+ blueprint = _blueprint(template=template)+ assert blueprint.template == template+def test_a_sigil_in_the_template_reads_its_input(self) -> None:blueprint = _blueprint(template="<h1>$invoice.number</h1>")assert blueprint.template == "<h1>$invoice.number</h1>"diff --git a/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_blueprint.py b/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_blueprint.pyindex f6d4da99f..f76aefb0e 100644--- a/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_blueprint.py+++ b/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_blueprint.py@@ -175,6 +175,44 @@ class TestPipeLLMBlueprint:)assert set(blueprint.input_names) == {"items"}+ def test_validate_inputs_correct_with_a_set_read_in_its_loop_body(self):+ """A name a loop body sets is the template's own where the body reads it, not an input."""+ blueprint = PipeLLMBlueprint(+ description="lorem ipsum",+ inputs={"expenses": "native.Text[]"},+ output="native.Text",+ prompt="Plan these:\n{% for expense in expenses %}{% set day = expense.text %}- {{ day }}\n{% endfor %}",+ )+ assert set(blueprint.input_names) == {"expenses"}++ def test_validate_inputs_incorrect_input_read_only_through_a_loop_set(self):+ """An input declared only for a name the loop body sets is never read: the body reads its own `day`."""+ error = refused_input_error(+ blueprint_class=PipeLLMBlueprint,+ blueprint_kwargs={+ "description": "lorem ipsum",+ "inputs": {"expenses": "native.Text[]", "day": "native.Text"},+ "output": "native.Text",+ "prompt": "{% for expense in expenses %}{% set day = expense.text %}- {{ day }}\n{% endfor %}",+ },+ )+ assert error.error_type == PipeValidationErrorType.EXTRANEOUS_INPUT_VARIABLE+ assert error.variable_names == ["day"]++ def test_validate_inputs_incorrect_set_in_only_some_branches(self):+ """A name only some branches of an `if` set is read from the inputs on the paths where none ran."""+ error = refused_input_error(+ blueprint_class=PipeLLMBlueprint,+ blueprint_kwargs={+ "description": "lorem ipsum",+ "inputs": {"topic": "native.Text"},+ "output": "native.Text",+ "prompt": "{% if topic %}{% set tone = 'formal' %}{% endif %}Write about $topic in a {{ tone }} tone",+ },+ )+ assert error.error_type == PipeValidationErrorType.MISSING_INPUT_VARIABLE+ assert error.variable_names == ["tone"]+def test_validate_inputs_correct_with_input_named_like_a_jinja_global(self):"""An input named like a Jinja global shadows it, so reading the name reads the input."""blueprint = PipeLLMBlueprint(diff --git a/tests/unit/pipelex/tools/test_jinja2_field_paths.py b/tests/unit/pipelex/tools/test_jinja2_field_paths.pyindex 0eeb1ac5b..afb7120de 100644--- a/tests/unit/pipelex/tools/test_jinja2_field_paths.py+++ b/tests/unit/pipelex/tools/test_jinja2_field_paths.py@@ -1,8 +1,12 @@import pytestfrom pipelex.tools.jinja2.exceptions import Jinja2DetectVariablesError-from pipelex.tools.jinja2.jinja2_field_paths import LIST_ITEM_SEGMENT, detect_template_field_paths+from pipelex.tools.jinja2.jinja2_field_paths import detect_template_field_paths+from pipelex.tools.jinja2.jinja2_required_variables import detect_jinja2_required_variables+from pipelex.tools.jinja2.jinja2_scopes import LIST_ITEM_SEGMENTfrom pipelex.tools.jinja2.template_category import TemplateCategory+from pipelex.tools.misc.string_utils import get_root_from_dotted_path+from tests.unit.pipelex.tools.test_jinja2_required_variables import TestData as RequiredVariablesDatadef _paths(template_source: str) -> list[tuple[str, ...]]:@@ -42,6 +46,76 @@ class TestDetectTemplateFieldPaths:paths = _paths(template_source)assert not any("nosuch" in path or "index" in path for path in paths), f"{topic}: {paths}"+ @pytest.mark.parametrize(+ ("topic", "template_source"),+ [+ ("set_in_every_branch_rebinds_an_input", "{% if f %}{% set invoice = a %}{% else %}{% set invoice = b %}{% endif %}{{ invoice.nosuch }}"),+ (+ "loop_target_rebound_in_every_branch",+ (+ "{% for item in invoice.line_items %}{% if f %}{% set item = a %}{% else %}{% set item = b %}{% endif %}"+ "{{ item.nosuch }}{% endfor %}"+ ),+ ),+ ("set_in_a_call_block_body", "{% call m() %}{% set invoice = a %}{{ invoice.nosuch }}{% endcall %}"),+ ("set_in_a_filter_block_body", "{% filter upper %}{% set invoice = a %}{{ invoice.nosuch }}{% endfilter %}"),+ ("macro_default_reads_an_earlier_argument", "{% macro show(invoice, total=invoice.nosuch) %}{{ total }}{% endmacro %}"),+ ],+ )+ def test_a_name_jinja_binds_gives_no_path(self, topic: str, template_source: str) -> None:+ paths = _paths(template_source)+ assert not any("nosuch" in path for path in paths), f"{topic}: {paths}"++ def test_a_called_global_gives_no_path(self) -> None:+ assert _paths("{% for i in range(3) %}{{ dict(a=i).a }}{% endfor %}") == []++ @pytest.mark.parametrize(+ ("topic", "template_source", "expected_path"),+ [+ ("set_in_some_branches", "{% if f %}{% set invoice = other %}{% endif %}{{ invoice.nosuch }}", ("invoice", "nosuch")),+ (+ "loop_target_rebound_in_some_branches",+ "{% for item in invoice.line_items %}{% if f %}{% set item = other %}{% endif %}{{ item.nosuch }}{% endfor %}",+ ("invoice", "line_items", LIST_ITEM_SEGMENT, "nosuch"),+ ),+ ("set_does_not_escape_a_loop", "{% for x in xs %}{% set invoice = x %}{% endfor %}{{ invoice.nosuch }}", ("invoice", "nosuch")),+ (+ "macro_reads_a_name_its_frame_reads_before_setting",+ "{{ invoice }}{% macro m() %}{{ invoice.nosuch }}{% endmacro %}{{ m() }}{% set invoice = other %}",+ ("invoice", "nosuch"),+ ),+ (+ "loop_target_in_a_block",+ "{% for invoice in invoices.all %}{% block b %}{{ invoice.nosuch }}{% endblock %}{% endfor %}",+ ("invoice", "nosuch"),+ ),+ (+ "loop_target_in_a_scoped_block",+ "{% for item in invoice.line_items %}{% block b scoped %}{{ item.nosuch }}{% endblock %}{% endfor %}",+ ("invoice", "line_items", LIST_ITEM_SEGMENT, "nosuch"),+ ),+ ],+ )+ def test_a_name_read_where_jinja_reads_the_input_keeps_its_path(self, topic: str, template_source: str, expected_path: tuple[str, ...]) -> None:+ assert expected_path in _paths(template_source), f"{topic}: {_paths(template_source)}"++ @pytest.mark.parametrize(+ ("topic", "template_source", "expected_variables"),+ RequiredVariablesData.JINJA_SCOPE_RULES,+ )+ def test_roots_match_the_required_variables(+ self,+ topic: str,+ template_source: str,+ expected_variables: set[str], # ruff: ignore[unused-method-argument]+ ) -> None:+ """The field paths and the required variables walk the same scopes, so they start from the same inputs."""+ field_path_roots = {+ field_path.root for field_path in detect_template_field_paths(template_source=template_source, template_category=TemplateCategory.HTML)+ }+ required = detect_jinja2_required_variables(template_category=TemplateCategory.HTML, template_source=template_source)+ assert field_path_roots == {get_root_from_dotted_path(path) for path in required}, f"Failed for topic: {topic}"+def test_a_subscript_is_not_followed(self) -> None:assert _paths("{{ invoice['customer'].name }}") == [("invoice",)]diff --git a/tests/unit/pipelex/tools/test_jinja2_optional_guards.py b/tests/unit/pipelex/tools/test_jinja2_optional_guards.pyindex 55b0df2ba..aebbaae1e 100644--- a/tests/unit/pipelex/tools/test_jinja2_optional_guards.py+++ b/tests/unit/pipelex/tools/test_jinja2_optional_guards.py@@ -35,6 +35,42 @@ class TestDetectUnguardedOptionalReferences:("with_shadow_local", "{% with assessment = 'x' %}{{ assessment }}{% endwith %}"),("for_filter_bare_presence_probe", "{% for item in topic_items if assessment %}{{ item }}{% endfor %}"),("for_filter_target_shadows", "{% for assessment in topic_items if assessment.flag %}{{ assessment }}{% endfor %}"),+ ("set_in_every_branch", "{% if topic %}{% set assessment = 'a' %}{% else %}{% set assessment = 'b' %}{% endif %}{{ assessment.amount }}"),+ (+ "set_in_every_branch_with_elifs",+ (+ "{% if topic %}{% set assessment = 'a' %}{% elif other %}{% set assessment = 'b' %}"+ "{% else %}{% set assessment = 'c' %}{% endif %}{{ assessment }}"+ ),+ ),+ ("set_in_a_call_block_body", "{% call m() %}{% set assessment = 'x' %}{{ assessment.amount }}{% endcall %}"),+ ("set_in_a_filter_block_body", "{% filter upper %}{% set assessment = 'x' %}{{ assessment.amount }}{% endfilter %}"),+ ("call_block_argument_shadows", "{% call(assessment) m() %}{{ assessment.amount }}{% endcall %}"),+ ("set_in_an_autoescape_block", "{% autoescape true %}{% set assessment = 'x' %}{{ assessment.amount }}{% endautoescape %}"),+ ("set_in_a_block_body", "{% block b %}{% set assessment = 'x' %}{{ assessment.amount }}{% endblock %}"),+ ("set_target_in_a_block_is_a_store_not_a_read", "{% block b %}{% set assessment = 'x' %}{% endblock %}"),+ ("macro_reads_a_later_set", "{% macro show() %}{{ assessment.amount }}{% endmacro %}{% set assessment = 'x' %}{{ show() }}"),+ ("top_level_set_reaches_a_block", "{% set assessment = 'x' %}{% block b %}{{ assessment.amount }}{% endblock %}"),+ ("guard_reaches_a_block", "{% if assessment %}{% block b %}{{ assessment.amount }}{% endblock %}{% endif %}"),+ (+ "guard_around_a_loop_reaches_a_block_in_it",+ "{% if assessment %}{% for i in items %}{% block b %}{{ assessment.amount }}{% endblock %}{% endfor %}{% endif %}",+ ),+ (+ "guard_in_a_loop_reaches_a_block",+ "{% for i in items %}{% if assessment %}{% block b %}{{ assessment.amount }}{% endblock %}{% endif %}{% endfor %}",+ ),+ (+ "macro_in_an_if_branch_reads_a_later_set",+ (+ "{% if topic %}{% macro show() %}{{ assessment.amount }}{% endmacro %}{% else %}{% macro show() %}x{% endmacro %}"+ "{% endif %}{% set assessment = 'x' %}{{ show() }}"+ ),+ ),+ (+ "scoped_block_sees_the_loop_target",+ "{% for assessment in items %}{% block b scoped %}{{ assessment.amount }}{% endblock %}{% endfor %}",+ ),],)def test_guarded_references_produce_no_findings(self, topic: str, template_source: str):@@ -64,6 +100,54 @@ class TestDetectUnguardedOptionalReferences:("with_value_reads_optional", "{% with local = assessment %}{{ local }}{% endwith %}", "assessment"),("with_body_reads_other_optional", "{% with other = 'x' %}{{ assessment.amount }}{% endwith %}", "assessment.amount"),("for_filter_deep_access", "{% for item in topic_items if assessment.flag %}{{ item }}{% endfor %}", "assessment.flag"),+ ("set_in_some_branches", "{% if topic %}{% set assessment = 'x' %}{% endif %}{{ assessment.amount }}", "assessment.amount"),+ ("else_does_not_see_the_body_set", "{% if topic %}{% set assessment = 'x' %}{% else %}{{ assessment }}{% endif %}", "assessment"),+ ("set_does_not_escape_a_call_block", "{% call m() %}{% set assessment = 'x' %}{% endcall %}{{ assessment }}", "assessment"),+ ("set_does_not_escape_a_filter_block", "{% filter upper %}{% set assessment = 'x' %}{% endfilter %}{{ assessment }}", "assessment"),+ ("set_does_not_escape_a_set_block", "{% set s %}{% set assessment = 'x' %}{% endset %}{{ assessment }}", "assessment"),+ ("macro_default_reads_optional", "{% macro m(x=assessment.amount) %}{{ x }}{% endmacro %}{{ m() }}", "assessment.amount"),+ (+ "set_does_not_escape_an_autoescape_block",+ "{% autoescape true %}{% set assessment = 'x' %}{% endautoescape %}{{ assessment }}",+ "assessment",+ ),+ ("set_does_not_escape_a_block", "{% block b %}{% set assessment = 'x' %}{% endblock %}{{ assessment }}", "assessment"),+ (+ "macro_reads_a_name_set_in_some_branches",+ "{% macro show() %}{{ assessment.amount }}{% endmacro %}{% if topic %}{% set assessment = 'x' %}{% endif %}{{ show() }}",+ "assessment.amount",+ ),+ (+ "macro_reads_an_optional_its_frame_reads_before_setting",+ (+ "{% if assessment %}{{ assessment }}{% endif %}"+ "{% macro show() %}{{ assessment.amount }}{% endmacro %}{{ show() }}{% set assessment = 'x' %}"+ ),+ "assessment.amount",+ ),+ (+ "loop_target_does_not_reach_a_block",+ "{% for assessment in items %}{% block b %}{{ assessment.amount }}{% endblock %}{% endfor %}",+ "assessment.amount",+ ),+ (+ "guard_on_a_loop_target_does_not_reach_a_block",+ "{% for assessment in items %}{% if assessment %}{% block b %}{{ assessment.amount }}{% endblock %}{% endif %}{% endfor %}",+ "assessment.amount",+ ),+ (+ "autoescape_set_does_not_reach_a_block",+ "{% autoescape true %}{% set assessment = 'x' %}{% block b %}{{ assessment.amount }}{% endblock %}{% endautoescape %}",+ "assessment.amount",+ ),+ (+ "guard_on_a_block_local_does_not_reach_a_nested_block",+ (+ "{% block outer %}{% set assessment = 'x' %}{% if assessment %}"+ "{% block inner %}{{ assessment.amount }}{% endblock %}{% endif %}{% endblock %}"+ ),+ "assessment.amount",+ ),],)def test_unguarded_references_are_reported(self, topic: str, template_source: str, expected_path: str):diff --git a/tests/unit/pipelex/tools/test_jinja2_required_variables.py b/tests/unit/pipelex/tools/test_jinja2_required_variables.pyindex dd9e2eb01..766b6c152 100644--- a/tests/unit/pipelex/tools/test_jinja2_required_variables.py+++ b/tests/unit/pipelex/tools/test_jinja2_required_variables.py@@ -1,13 +1,43 @@-from typing import ClassVar+from collections.abc import Iterator+from typing import Any, ClassVarimport pytest+from jinja2 import Environment, UndefinedError+from typing_extensions import overridefrom pipelex.tools.jinja2.exceptions import Jinja2DetectVariablesErrorfrom pipelex.tools.jinja2.jinja2_required_variables import (detect_jinja2_required_variables,detect_jinja2_variable_references,)+from pipelex.tools.jinja2.jinja2_undefined import PresenceProbingStrictUndefinedfrom pipelex.tools.jinja2.template_category import TemplateCategory+from pipelex.tools.misc.string_utils import get_root_from_dotted_path+++class _PermissiveValue:+ """A context value any template shape can read: every attribute and call gives itself back."""++ def __init__(self, *, is_truthy: bool) -> None:+ self.is_truthy = is_truthy++ def __getattr__(self, name: str) -> "_PermissiveValue":+ if name.startswith("__"):+ raise AttributeError(name)+ return self++ def __call__(self, *_args: Any, **_kwargs: Any) -> "_PermissiveValue":+ return self++ def __iter__(self) -> Iterator["_PermissiveValue"]:+ return iter([self] if self.is_truthy else [])++ def __bool__(self) -> bool:+ return self.is_truthy++ @override+ def __str__(self) -> str:+ return "value"class TestData:@@ -218,6 +248,144 @@ Best regards,("macro_reads_a_later_set", "{{ x }}{% macro m() %}[{{ g }}]{% endmacro %}{% set g = 'y' %}{{ m() }}", {"x"}),]+ # Where Jinja binds a name a template sets. An `if` opens no frame, and after it a name every branch sets, the+ # `else` included, is bound, while a name only some branches set is the input of that name on the paths where+ # none ran. A loop body, a loop's `else`, a macro, a call, filter or set block, a `with` and `autoescape` each+ # open a frame, so a set inside one stays inside it+ JINJA_SCOPE_RULES: ClassVar[list[tuple[str, str, set[str]]]] = [+ ("set_read_later_in_its_if_branch", "{% if x %}{% set y = 1 %}{{ y }}{% endif %}", {"x"}),+ ("set_in_both_branches", "{% if x %}{% set y = 1 %}{% else %}{% set y = 2 %}{% endif %}{{ y }}", {"x"}),+ (+ "set_in_every_branch_with_elifs",+ "{% if a %}{% set y = 1 %}{% elif b %}{% set y = 2 %}{% else %}{% set y = 3 %}{% endif %}{{ y }}",+ {"a", "b"},+ ),+ ("set_in_some_branches", "{% if x %}{% set y = 1 %}{% endif %}{{ y }}", {"x", "y"}),+ ("set_in_every_branch_but_no_else", "{% if a %}{% set y = 1 %}{% elif b %}{% set y = 2 %}{% endif %}{{ y }}", {"a", "b", "y"}),+ ("set_missing_from_an_elif", "{% if a %}{% set y = 1 %}{% elif b %}-{% else %}{% set y = 3 %}{% endif %}{{ y }}", {"a", "b", "y"}),+ ("else_does_not_see_the_body_set", "{% if x %}{% set y = 1 %}{% else %}{{ y }}{% endif %}", {"x", "y"}),+ ("elif_test_does_not_see_the_body_set", "{% if x %}{% set y = 1 %}{% elif y %}-{% endif %}", {"x", "y"}),+ (+ "nested_ifs_set_in_every_branch",+ "{% if a %}{% if b %}{% set y = 1 %}{% else %}{% set y = 2 %}{% endif %}{% else %}{% set y = 3 %}{% endif %}{{ y }}",+ {"a", "b"},+ ),+ ("nested_if_without_else", "{% if a %}{% if b %}{% set y = 1 %}{% endif %}{% else %}{% set y = 3 %}{% endif %}{{ y }}", {"a", "b", "y"}),+ ("set_before_the_if_and_in_a_branch", "{% set y = 1 %}{% if x %}{% set y = 2 %}{% endif %}{{ y }}", {"x"}),+ ("read_before_set_in_a_branch", "{% if x %}{{ y }}{% set y = 1 %}{% endif %}", {"x", "y"}),+ ("set_read_later_in_a_loop_body", "{% for i in items %}{% set day = i.d %}{{ day }}{% endfor %}", {"items"}),+ ("read_before_set_in_a_loop_body", "{% for i in items %}[{{ y }}]{% set y = i %}{% endfor %}", {"items", "y"}),+ ("set_in_some_branches_inside_a_loop", "{% for i in items %}{% if i %}{% set y = i %}{% endif %}{{ y }}{% endfor %}", {"items", "y"}),+ ("set_in_a_loop_else", "{% for i in items %}-{% else %}{% set y = 1 %}{{ y }}{% endfor %}", {"items"}),+ ("set_in_a_with_body", "{% with %}{% set y = 1 %}{{ y }}{% endwith %}", set()),+ ("set_in_a_macro_body", "{% macro m() %}{% set y = 1 %}{{ y }}{% endmacro %}{{ m() }}", set()),+ ("set_in_a_call_block_body", "{% macro m() %}{{ caller() }}{% endmacro %}{% call m() %}{% set y = 1 %}{{ y }}{% endcall %}", set()),+ ("set_in_a_filter_block_body", "{% filter upper %}{% set y = 'a' %}{{ y }}{% endfilter %}", set()),+ ("set_in_a_set_block_body", "{% set z %}{% set y = 'a' %}{{ y }}{% endset %}{{ z }}", set()),+ ("set_in_an_autoescape_block", "{% autoescape true %}{% set y = 1 %}{{ y }}{% endautoescape %}", set()),+ ("set_does_not_escape_a_loop", "{% for i in items %}{% set y = i %}{% endfor %}{{ y }}", {"items", "y"}),+ ("set_does_not_escape_a_loop_else", "{% for i in items %}-{% else %}{% set y = 1 %}{% endfor %}{{ y }}", {"items", "y"}),+ ("set_does_not_escape_a_with", "{% with %}{% set y = 1 %}{% endwith %}{{ y }}", {"y"}),+ ("set_does_not_escape_a_macro", "{% macro m() %}{% set y = 1 %}{% endmacro %}{{ m() }}{{ y }}", {"y"}),+ ("set_does_not_escape_a_call_block", "{% macro m() %}{{ caller() }}{% endmacro %}{% call m() %}{% set y = 1 %}{% endcall %}{{ y }}", {"y"}),+ ("set_does_not_escape_a_filter_block", "{% filter upper %}{% set y = 'a' %}{% endfilter %}{{ y }}", {"y"}),+ ("set_does_not_escape_a_set_block", "{% set z %}{% set y = 'a' %}{% endset %}{{ z }}{{ y }}", {"y"}),+ ("set_does_not_escape_an_autoescape_block", "{% autoescape true %}{% set y = 1 %}{% endautoescape %}{{ y }}", {"y"}),+ ("with_target", "{% with y = 1 %}{{ y }}{% endwith %}", set()),+ ("with_value_reads_outside_the_with", "{% with a = 1, b = a %}{{ b }}{% endwith %}", {"a"}),+ ("tuple_target", "{% set a, b = 1, 2 %}{{ a }}{{ b }}", set()),+ ("nested_tuple_target", "{% set a, (b, c) = 1, (2, 3) %}{{ a }}{{ c }}", set()),+ ("nested_tuple_loop_target", "{% for k, (a, b) in [(1, (2, 3))] %}{{ k }}{{ b }}{% endfor %}", set()),+ (+ "namespace_attribute_set_in_an_if",+ "{% set ns = namespace(found=false) %}{% for i in items %}{% if i %}{% set ns.found = true %}{% endif %}{% endfor %}{{ ns.found }}",+ {"items"},+ ),+ (+ "macro_reads_a_name_every_branch_sets",+ "{% if a %}{% set g = 1 %}{% else %}{% set g = 2 %}{% endif %}{% macro m() %}{{ g }}{% endmacro %}{{ m() }}",+ {"a"},+ ),+ ("macro_reads_a_name_some_branches_set", "{% if a %}{% set g = 1 %}{% endif %}{% macro m() %}{{ g }}{% endmacro %}{{ m() }}", {"a", "g"}),+ (+ "macro_in_a_loop_reads_a_later_set",+ "{% for i in items %}{% macro m() %}{{ y }}{% endmacro %}{% set y = i %}{{ m() }}{% endfor %}",+ {"items"},+ ),+ ("macro_called_before_its_definition", "{{ m() }}{% macro m() %}-{% endmacro %}", {"m"}),+ # A frame that reads a name before setting it starts its own copy from the input, which a macro called before+ # the set then reads+ ("macro_reads_a_name_its_frame_reads_before_setting", "{{ a }}{% macro m() %}{{ a.x }}{% endmacro %}{{ m() }}{% set a = 1 %}", {"a", "a.x"}),+ (+ "macro_in_a_loop_reads_a_name_the_loop_reads_before_setting",+ "{% for i in items %}{% if a is defined %}-{% endif %}{% macro m() %}{{ a.x }}{% endmacro %}{{ m() }}{% set a = i %}{% endfor %}",+ {"items", "a", "a.x"},+ ),+ # An `if` opens no frame, so a macro defined in one of its branches sees what its frame sets+ (+ "macro_in_an_if_branch_reads_a_later_set",+ "{% if c %}{% macro m() %}{{ a }}{% endmacro %}{% else %}{% macro m() %}x{% endmacro %}{% endif %}{% set a = 1 %}{{ m() }}",+ {"c"},+ ),+ (+ "macro_in_an_if_branch_in_a_loop_reads_a_later_set",+ (+ "{% for i in items %}{% if c %}{% macro m() %}{{ a }}{% endmacro %}{% else %}{% macro m() %}x{% endmacro %}{% endif %}"+ "{% set a = i %}{{ m() }}{% endfor %}"+ ),+ {"items", "c"},+ ),+ ("macro_default_reads_an_earlier_argument", "{% macro m(a, b=a) %}{{ b }}{% endmacro %}{{ m(1) }}", set()),+ ("call_block_arguments", "{% macro m() %}{{ caller(1) }}{% endmacro %}{% call(a) m() %}{{ a }}{% endcall %}", set()),+ ("filter_block_filter_reads_its_body_set", "{% filter replace('a', y) %}{% set y = 'b' %}a{% endfilter %}", set()),+ # A block runs as a function of its own. Without `scoped` it reads the template context, which holds what the top+ # level has set for certain before it and nothing of a loop, a macro or any other frame around it; a scoped+ # block reads the names bound where it stands, and so do the blocks nested in it+ ("set_in_a_block_body", "{% block b %}{% set y = 1 %}{{ y }}{% endblock %}", set()),+ ("set_does_not_escape_a_block", "{% block b %}{% set y = 1 %}{% endblock %}{{ y }}", {"y"}),+ ("top_level_set_reaches_a_block", "{% set y = 1 %}{% block b %}{{ y }}{% endblock %}", set()),+ ("top_level_set_after_a_block", "{% block b %}{{ y }}{% endblock %}{% set y = 1 %}", {"y"}),+ (+ "set_in_every_top_level_branch_reaches_a_block",+ "{% if x %}{% set y = 1 %}{% else %}{% set y = 2 %}{% endif %}{% block b %}{{ y }}{% endblock %}",+ {"x"},+ ),+ ("set_in_some_top_level_branches_before_a_block", "{% if x %}{% set y = 1 %}{% endif %}{% block b %}{{ y }}{% endblock %}", {"x", "y"}),+ ("set_earlier_in_a_top_level_branch_reaches_a_block", "{% if x %}{% set y = 1 %}{% block b %}{{ y }}{% endblock %}{% endif %}", {"x"}),+ ("top_level_macro_reaches_a_block", "{% macro m() %}-{% endmacro %}{% block b %}{{ m() }}{% endblock %}", set()),+ (+ "loop_target_does_not_reach_a_block",+ "{% for item in items %}{% block b %}{{ item.name }}{% endblock %}{% endfor %}",+ {"items", "item.name"},+ ),+ ("loop_set_does_not_reach_a_block", "{% for i in items %}{% set y = i %}{% block b %}{{ y }}{% endblock %}{% endfor %}", {"items", "y"}),+ (+ "top_level_set_reaches_a_block_in_a_loop",+ "{% set y = 1 %}{% for i in items %}{% set y = i %}{% block b %}{{ y }}{% endblock %}{% endfor %}",+ {"items"},+ ),+ ("with_target_does_not_reach_a_block", "{% with y = 1 %}{% block b %}{{ y }}{% endblock %}{% endwith %}", {"y"}),+ ("macro_argument_does_not_reach_a_block", "{% macro m(a) %}{% block b %}{{ a }}{% endblock %}{% endmacro %}{{ m(1) }}", {"a"}),+ (+ "call_block_argument_does_not_reach_a_block",+ "{% macro m() %}{{ caller(1) }}{% endmacro %}{% call(y) m() %}{% block b %}{{ y }}{% endblock %}{% endcall %}",+ {"y"},+ ),+ ("autoescape_set_does_not_reach_a_block", "{% autoescape true %}{% set y = 1 %}{% block b %}{{ y }}{% endblock %}{% endautoescape %}", {"y"}),+ ("block_set_does_not_reach_a_nested_block", "{% block outer %}{% set z = 1 %}{% block inner %}{{ z }}{% endblock %}{% endblock %}", {"z"}),+ ("scoped_block_sees_the_loop", "{% for item in items %}{% block b scoped %}{{ item.name }}{% endblock %}{% endfor %}", {"items"}),+ (+ "block_nested_in_a_scoped_block_sees_its_call_site",+ "{% for item in items %}{% block outer scoped %}{% block inner %}{{ item.name }}{% endblock %}{% endblock %}{% endfor %}",+ {"items"},+ ),+ (+ "scoped_block_set_does_not_reach_a_nested_block",+ "{% for i in items %}{% block outer scoped %}{% set z = i %}{% block inner %}{{ z }}{% endblock %}{% endblock %}{% endfor %}",+ {"items", "z"},+ ),+ ]+# Names Jinja provides, and the scope a loop opens, are never inputsSCOPES: ClassVar[list[tuple[str, str, set[str]]]] = [("namespace_idiom", "{% set ns = namespace(found=false) %}{{ ns.found }}", set()),@@ -627,6 +795,51 @@ class TestDetectJinja2Variables:)assert result == expected_variables, f"Failed for topic: {topic}"+ @pytest.mark.parametrize(+ ("topic", "template_source", "expected_variables"),+ TestData.JINJA_SCOPE_RULES,+ )+ def test_jinja_scope_rules(+ self,+ topic: str,+ template_source: str,+ expected_variables: set[str],+ ):+ """A name the template sets is no input where Jinja binds it, and is the input of that name everywhere else."""+ result = detect_jinja2_required_variables(+ template_category=TemplateCategory.LLM_PROMPT,+ template_source=template_source,+ )+ assert result == expected_variables, f"Failed for topic: {topic}"++ @pytest.mark.parametrize(+ ("topic", "template_source", "expected_variables"),+ TestData.JINJA_SCOPE_RULES,+ )+ @pytest.mark.parametrize("is_truthy", [True, False])+ def test_jinja_reads_nothing_beyond_the_required_variables(+ self,+ topic: str,+ template_source: str,+ expected_variables: set[str], # ruff: ignore[unused-method-argument]+ is_truthy: bool,+ ):+ """Jinja itself renders each case with only the required roots in its context, so the walk never takes a read of the context for a local.++ Every root is given a value that answers any attribute or call with itself, and that is truthy and iterates once,+ or falsy and iterates not at all, so that both sides of every `if` and every loop's `else` run.+ """+ required = detect_jinja2_required_variables(+ template_category=TemplateCategory.LLM_PROMPT,+ template_source=template_source,+ )+ context = {get_root_from_dotted_path(path): _PermissiveValue(is_truthy=is_truthy) for path in required}+ template = Environment(undefined=PresenceProbingStrictUndefined).from_string(template_source)+ try:+ template.render(context)+ except UndefinedError as undefined_error:+ pytest.fail(f"{topic}: Jinja read '{undefined_error}' from the context, but the walk reported only {sorted(required)}")+@pytest.mark.parametrize(("topic", "template_source", "expected_variables"),TestData.SCOPES,@@ -807,7 +1020,7 @@ class TestDetectJinja2VariableReferences:@pytest.mark.parametrize(("topic", "template_source", "expected_paths"),- TestData.REFERENCE_SCOPES,+ TestData.REFERENCE_SCOPES + TestData.JINJA_SCOPE_RULES,)def test_scopes_match_required_variables(self,