Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

fix/Set-inside-if-read · L-260930-afc983 (#1319)

Louis Choquel0e9a29067507
diff --git a/CHANGELOG.md b/CHANGELOG.md
index dba2ae784..af00afee4 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -15,6 +15,7 @@
### Fixed
+- **A name a template sets is read where Jinja binds it (Breaking)**: the input check of `PipeLLM`, `PipeCompose`, `PipeSearch`, `PipeImgGen` and `PipeDocGen`, `PipeDocGen`'s field check and the optional-input guard now treat a `{% set %}` name as the template's own inside the loop, macro or block that sets it and after an `{% if %}` every branch of which sets it, so a template setting a name in a loop body or in each branch of an `if` and reading it there is no longer refused as reading an undeclared input. A name only some branches set is still read from the input of that name. Newly refused: an input declared only so that such a set would load, which is now never read, a macro called before its `{% macro %}`, an optional input read unguarded in a macro argument's default, and a name a loop or a macro binds read inside a `{% block %}` without `scoped`, which Jinja reads from the input of that name.
- **File inputs are checked against the models that read them (Breaking)**: run setup now identifies every image and document input from its first bytes, refuses an `Image` input holding something other than an image with `PipelineInputNotAnImageError`, and refuses a document certain to reach a `PipeExtract` or `PipeLLM` step whose model does not read its format with `PipelineInputFormatUnsupportedError`, both 422 input errors naming the input, the step, the model and the formats it reads; a step reached only through a condition refuses the file itself with `ExtractInputFormatError` or `PromptDocumentFormatError`. A `PipeLLM` document its model does not read now raises `PromptDocumentFormatError` where it raised `LLMCapabilityError`, so a consumer branching on `error_type` must follow. `docling-extract-text` now declares the Word, PowerPoint, Excel, HTML, Markdown, CSV, plain-text, WebVTT and email files it already extracted, which `pipelex update` installs.
- **Markdown with unclosed emphasis is converted in memory proportional to its length**: the Markdown parser behind the `markdown` filter, the `Markdown` concept's HTML and the built-in PDF engine held memory growing with the square of the text when emphasis or strikethrough markers were left unmatched, about 1.5 GB for 64,000 characters of `**a `; it now joins the text they leave behind in one pass, and renders the same HTML.
- **Routing profiles no longer rewrite their own routes**: matching a model merged the enabled `optional_routes` into the profile's declared `routes` in place, so after the first match a route declared optional had become an unconditional one in the loaded profile. Only the enablement checks each match repeats kept that from changing a routing decision.
diff --git a/docs/building-methods/pipes/pipe-operators/PipeCompose.md b/docs/building-methods/pipes/pipe-operators/PipeCompose.md
index 4ccba455d..84a216407 100644
--- a/docs/building-methods/pipes/pipe-operators/PipeCompose.md
+++ b/docs/building-methods/pipes/pipe-operators/PipeCompose.md
@@ -40,7 +40,7 @@ A template reads data and calls methods of plain values, and nothing else: it ca
| `output` | string | The concept for the output | Yes |
| `template` | string or section | An inline template string, or a `[pipe.name.template]` section (see below) | Yes* |
-Every input the pipe declares must be read by the template, and every variable the template reads must be declared in `inputs`. Validation refuses an input the template never reads as `extraneous_input_variable`, naming the input, so you either reference it in the template or remove it from `inputs`; an undeclared variable is refused as `missing_input_variable`.
+Every input the pipe declares must be read by the template, and every variable the template reads must be declared in `inputs`. Validation refuses an input the template never reads as `extraneous_input_variable`, naming the input, so you either reference it in the template or remove it from `inputs`; an undeclared variable is refused as `missing_input_variable`. A name the template sets with `{% set %}` is its own, not an input, from that statement on: after an `{% if %}` it stays set only when every branch sets it, the `{% else %}` included, so a name only some branches set is read from the input of that name on the other paths; and a loop, a macro or a block keeps what it sets to its own body.
*Template mode requires `template`. When using the rich form (`[pipe.name.template]` section), the following sub-fields are available:
diff --git a/docs/building-methods/pipes/pipe-operators/PipeDocGen.md b/docs/building-methods/pipes/pipe-operators/PipeDocGen.md
index ff90f4397..8bb51ace1 100644
--- a/docs/building-methods/pipes/pipe-operators/PipeDocGen.md
+++ b/docs/building-methods/pipes/pipe-operators/PipeDocGen.md
@@ -112,7 +112,7 @@ A `Text` input prints as plain paragraphs, so a report whose output is `Text` pr
A `pdf` step's `template`, or its `.html` `template_file`, is rendered with exactly the stack of a `PipeCompose` HTML template: `$x` and `@x`, the `format` and `tag` filters, HTML autoescaping, and the [template sandbox](../../../under-the-hood/template-sandbox.md). A template that is only a fragment is wrapped in the plugin's print stylesheet (A4, running headers, page numbers); a whole HTML document prints as written, with any page setup its CSS gives. Three differences from `PipeCompose` matter:
-- **A field a template names must exist.** Each path a template reads, such as `invoice.customer.name`, is checked against the inputs' concepts when the method loads, including a path through a loop variable: in `{% for item in invoice.line_items %}`, `item.amount` is checked against the line item's concept. What the load-time check cannot follow, such as a value set with `{% set %}`, fails when the template renders at the dry run, since a missing value is an error rather than empty text. An absent optional input can still be tested with `{% if notes %}` or `is defined`.
+- **A field a template names must exist.** Each path a template reads, such as `invoice.customer.name`, is checked against the inputs' concepts when the method loads, including a path through a loop variable: in `{% for item in invoice.line_items %}`, `item.amount` is checked against the line item's concept. A name the template sets with `{% set %}` is followed no further, so a field read through it is checked when the template renders at the dry run, where a missing value is an error rather than empty text; a name only some branches of an `{% if %}` set is still checked as the input of that name, which the template reads on the paths where no branch ran. An absent optional input can still be tested with `{% if notes %}` or `is defined`.
- **Pipelex's `format` filter shadows Jinja's own.** To format a number, write `{{ '%.2f' % invoice.total }}` rather than `{{ '%.2f' | format(invoice.total) }}`.
- **The `markdown` filter** turns Markdown held in a plain text field into HTML: `{{ invoice.notes | markdown }}`. A `Markdown` input needs no filter.
diff --git a/docs/building-methods/pipes/pipe-operators/PipeLLM.md b/docs/building-methods/pipes/pipe-operators/PipeLLM.md
index 08e4489b2..adc8b01c3 100644
--- a/docs/building-methods/pipes/pipe-operators/PipeLLM.md
+++ b/docs/building-methods/pipes/pipe-operators/PipeLLM.md
@@ -18,7 +18,7 @@ For structured outputs, you have two options:
If you already have text from elsewhere (a PDF extraction, a search result, an upstream pipe), call [`PipeStructure`](./PipeStructure.md) directly — there's no need to wrap a `PipeLLM` around it.
-Every input the pipe declares must be read by `prompt` or `system_prompt`, and every variable they read must be declared in `inputs`. Validation refuses an input neither prompt reads as `extraneous_input_variable`, naming the input, so you either reference it in a prompt or remove it from `inputs`; an undeclared variable is refused as `missing_input_variable`.
+Every input the pipe declares must be read by `prompt` or `system_prompt`, and every variable they read must be declared in `inputs`. Validation refuses an input neither prompt reads as `extraneous_input_variable`, naming the input, so you either reference it in a prompt or remove it from `inputs`; an undeclared variable is refused as `missing_input_variable`. A name a prompt sets with `{% set %}` is its own, not an input, from that statement on: after an `{% if %}` it stays set only when every branch sets it, the `{% else %}` included, so a name only some branches set is read from the input of that name on the other paths; and a loop, a macro or a block keeps what it sets to its own body.
## Working with Images (Vision Language Models)
diff --git a/docs/features/validation-dry-run.md b/docs/features/validation-dry-run.md
index 42fb7cf4b..b61ed5392 100644
--- a/docs/features/validation-dry-run.md
+++ b/docs/features/validation-dry-run.md
@@ -18,7 +18,7 @@ Check pipeline syntax, structure, and compatibility without execution:
- **Syntax validation** — Catch MTHDS language errors via plxt linting
- **Structure validation** — Resolve pipe and concept references in the loaded library, refuse concepts whose structures form a cycle or whose fields name a concept that cannot be found, and verify each pipe's declared inputs against the concepts that operator accepts — plus a controller's declared output against what it actually produces (for a `PipeSequence`, its last step's output concept and multiplicity)
- **Input validation** — Ensure required inputs are provided and correctly typed, and that every variable a step reads is bound by the time that step runs
-- **Declared inputs are read** — For `PipeLLM`, `PipeCompose`, `PipeSearch` and `PipeImgGen`, every variable a template reads must be declared in `inputs` (`missing_input_variable`), and every declared input must be read by one of the pipe's templates (`extraneous_input_variable`): the prompt and system prompt of a `PipeLLM`, the template or the construct of a `PipeCompose`, the prompt of a `PipeSearch`, the prompt and negative prompt of a `PipeImgGen`. An unread input is refused on the pipe that declares it, before any controller above it is asked to supply it
+- **Declared inputs are read** — For `PipeLLM`, `PipeCompose`, `PipeSearch` and `PipeImgGen`, every variable a template reads must be declared in `inputs` (`missing_input_variable`), and every declared input must be read by one of the pipe's templates (`extraneous_input_variable`): the prompt and system prompt of a `PipeLLM`, the template or the construct of a `PipeCompose`, the prompt of a `PipeSearch`, the prompt and negative prompt of a `PipeImgGen`. An unread input is refused on the pipe that declares it, before any controller above it is asked to supply it. A name the template sets with `{% set %}` is its own, not an input, from that statement on: after an `{% if %}` it stays set only when every branch sets it, the `{% else %}` included, so a name only some branches set is read from the input of that name on the other paths; and a loop, a macro or a block keeps what it sets to its own body.
A pipe that references a sub-pipe from a package that isn't loaded is reported as **skipped** rather than validated, so a passing run is not proof that every cross-package dependency resolves.
diff --git a/pipelex/pipe_operators/doc_gen/template_field_check.py b/pipelex/pipe_operators/doc_gen/template_field_check.py
index e57f55819..a527bb9ab 100644
--- a/pipelex/pipe_operators/doc_gen/template_field_check.py
+++ b/pipelex/pipe_operators/doc_gen/template_field_check.py
@@ -22,7 +22,8 @@ from pipelex.core.stuffs.stuff_artefact import StuffArtefact
from pipelex.core.stuffs.stuff_content import StuffContent
from pipelex.pipe_operators.doc_gen.exceptions import PipeDocGenFactoryError
from pipelex.runtime_hub import get_class_registry
-from pipelex.tools.jinja2.jinja2_field_paths import LIST_ITEM_SEGMENT, TemplateFieldPath, detect_template_field_paths
+from pipelex.tools.jinja2.jinja2_field_paths import TemplateFieldPath, detect_template_field_paths
+from pipelex.tools.jinja2.jinja2_scopes import LIST_ITEM_SEGMENT
from pipelex.tools.jinja2.template_category import TemplateCategory
_NONE_TYPE = type(None)
diff --git a/pipelex/tools/jinja2/jinja2_field_paths.py b/pipelex/tools/jinja2/jinja2_field_paths.py
index bb3f40242..c2416b95a 100644
--- a/pipelex/tools/jinja2/jinja2_field_paths.py
+++ b/pipelex/tools/jinja2/jinja2_field_paths.py
@@ -5,9 +5,13 @@ reads, so a load-time check can compare each path with the input's concept and r
any run. A loop variable is followed back to what it iterates: in `{% for item in invoice.line_items %}`,
`item.amount` reads `invoice.line_items[].amount`, the `[]` standing for an item of the list.
-It is deliberately conservative, since what it cannot follow is left to the strict undefined at the dry run: a
-name set with `{% set %}`, a macro argument, a tuple loop target, and a chain that starts from a subscript, a
-call or a filter give no path. Only attribute chains (`a.b.c`) are followed; `a['b']` is not.
+Both detectors are handlers on the one read walk of `jinja2_scopes.py`, so they agree on which names the template
+binds: a name set in every branch of an `if`, or anywhere before the read in the same loop or block body, is the
+template's own and gives no path, while a name set in only some branches is read as the input of that name, and
+its path is checked. The walk is deliberately conservative, since what it cannot follow is left to the strict
+undefined at the dry run: a name the template sets, a macro argument, a tuple loop target, a called global and
+a chain that starts from a subscript, a call or a filter give no path. Only attribute chains (`a.b.c`) are
+followed; `a['b']` is not.
"""
from typing import NamedTuple
@@ -17,11 +21,9 @@ from jinja2.exceptions import TemplateSyntaxError
from pipelex.tools.jinja2.exceptions import Jinja2DetectVariablesError
from pipelex.tools.jinja2.jinja2_environment import make_jinja2_env_without_loader
+from pipelex.tools.jinja2.jinja2_scopes import ScopeBindings, attribute_chain, resolve_bound_path, walk_template_reads
from pipelex.tools.jinja2.template_category import TemplateCategory
-# The segment standing for "an item of the list" in a path reached through a loop variable.
-LIST_ITEM_SEGMENT = "[]"
-
class TemplateFieldPath(NamedTuple):
"""One attribute chain a template reads, resolved back to the input it starts from."""
@@ -32,104 +34,6 @@ class TemplateFieldPath(NamedTuple):
"""The chain as the template writes it, such as `item.amount`, for messages."""
-# What a name stands for where it is read: the path it is bound to, or None when it is declared locally and
-# cannot be followed (a `set`, a macro argument, `loop`). A name absent from the bindings is an input or a global.
-_Bindings = dict[str, tuple[str, ...] | None]
-
-
-def _chain(node: nodes.Node) -> tuple[str, list[str]] | None:
- """The name and attribute names of a pure attribute chain (`a.b.c`), or None for any other shape."""
- attributes: list[str] = []
- current = node
- while isinstance(current, nodes.Getattr):
- attributes.append(current.attr)
- current = current.node
- if isinstance(current, nodes.Name):
- return current.name, list(reversed(attributes))
- return None
-
-
-def _resolve(*, name: str, attributes: list[str], bindings: _Bindings) -> tuple[str, ...] | None:
- """The full path a chain reads, through the bindings: None when it starts from a name that cannot be followed."""
- if name in bindings:
- bound = bindings[name]
- if bound is None:
- return None
- return (*bound, *attributes)
- return (name, *attributes)
-
-
-def _target_names(target: nodes.Node) -> list[str]:
- if isinstance(target, nodes.Name):
- return [target.name]
- if isinstance(target, nodes.Tuple):
- return [name for item in target.items for name in _target_names(item)]
- return []
-
-
-def _walk(*, node: nodes.Node, bindings: _Bindings, paths: list[TemplateFieldPath]) -> None:
- if isinstance(node, (nodes.Template, nodes.Scope)):
- _walk_body(body=node.body, bindings=bindings, paths=paths)
- return
- if isinstance(node, nodes.With):
- for value in node.values:
- _walk(node=value, bindings=bindings, paths=paths)
- with_bindings: _Bindings = dict(bindings)
- for target in node.targets:
- for target_name in _target_names(target):
- with_bindings[target_name] = None
- _walk_body(body=node.body, bindings=with_bindings, paths=paths)
- return
- if isinstance(node, nodes.For):
- _walk(node=node.iter, bindings=bindings, paths=paths)
- loop_bindings: _Bindings = {**bindings, "loop": None}
- iterated = _chain(node.iter)
- iterated_path = _resolve(name=iterated[0], attributes=iterated[1], bindings=bindings) if iterated is not None else None
- if isinstance(node.target, nodes.Name) and iterated_path is not None:
- loop_bindings[node.target.name] = (*iterated_path, LIST_ITEM_SEGMENT)
- else:
- for target_name in _target_names(node.target):
- loop_bindings[target_name] = None
- _walk_body(body=node.body, bindings=loop_bindings, paths=paths)
- if node.test is not None:
- _walk(node=node.test, bindings=loop_bindings, paths=paths)
- _walk_body(body=node.else_, bindings=bindings, paths=paths)
- return
- if isinstance(node, nodes.Macro):
- macro_bindings: _Bindings = {**bindings, "caller": None, "varargs": None, "kwargs": None}
- for argument in node.args:
- macro_bindings[argument.name] = None
- for default in node.defaults:
- _walk(node=default, bindings=bindings, paths=paths)
- _walk_body(body=node.body, bindings=macro_bindings, paths=paths)
- return
- if isinstance(node, (nodes.Name, nodes.Getattr)):
- chain = _chain(node)
- if chain is not None:
- if isinstance(node, nodes.Name) and node.ctx != "load":
- return
- name, attributes = chain
- path = _resolve(name=name, attributes=attributes, bindings=bindings)
- if path is not None:
- written = ".".join([name, *attributes])
- paths.append(TemplateFieldPath(root=path[0], segments=path[1:], written=written))
- return
- for child in node.iter_child_nodes():
- _walk(node=child, bindings=bindings, paths=paths)
-
-
-def _walk_body(*, body: list[nodes.Node], bindings: _Bindings, paths: list[TemplateFieldPath]) -> None:
- """Walk statements in order: a `{% set %}` declares its name for the statements after it, in this scope."""
- scope_bindings: _Bindings = dict(bindings)
- for statement in body:
- if isinstance(statement, nodes.Macro):
- scope_bindings[statement.name] = None
- _walk(node=statement, bindings=scope_bindings, paths=paths)
- if isinstance(statement, (nodes.Assign, nodes.AssignBlock)):
- for target_name in _target_names(statement.target):
- scope_bindings[target_name] = None
-
-
def detect_template_field_paths(*, template_source: str, template_category: TemplateCategory) -> list[TemplateFieldPath]:
"""Every attribute chain the template reads, in source order, resolved back to the name it starts from.
@@ -145,5 +49,19 @@ def detect_template_field_paths(*, template_source: str, template_category: Temp
msg = f"Jinja2 detect field paths — syntax error: '{syntax_error}', template_category: {template_category}"
raise Jinja2DetectVariablesError(msg) from syntax_error
paths: list[TemplateFieldPath] = []
- _walk(node=parsed_ast, bindings={}, paths=paths)
+
+ def collect_field_path(*, node: nodes.Node, bindings: ScopeBindings) -> bool:
+ if not isinstance(node, (nodes.Name, nodes.Getattr)):
+ return False
+ chain = attribute_chain(node)
+ if chain is None:
+ # An attribute on a subscript, a call or a filter (`items[0].text`) reads the path its chain starts from
+ return False
+ name, attributes = chain
+ path = resolve_bound_path(name=name, attributes=attributes, bindings=bindings)
+ if path is not None:
+ paths.append(TemplateFieldPath(root=path[0], segments=path[1:], written=".".join([name, *attributes])))
+ return True
+
+ walk_template_reads(template=parsed_ast, global_names=set(jinja2_env.globals), handle_read=collect_field_path)
return paths
diff --git a/pipelex/tools/jinja2/jinja2_optional_guards.py b/pipelex/tools/jinja2/jinja2_optional_guards.py
index 15afcf7aa..56bf876c7 100644
--- a/pipelex/tools/jinja2/jinja2_optional_guards.py
+++ b/pipelex/tools/jinja2/jinja2_optional_guards.py
@@ -27,6 +27,13 @@ from pydantic.dataclasses import dataclass
from pipelex.tools.jinja2.exceptions import Jinja2DetectVariablesError
from pipelex.tools.jinja2.jinja2_environment import make_jinja2_env_without_loader
+from pipelex.tools.jinja2.jinja2_scopes import (
+ definitely_assigned_names,
+ dotted_attribute_path,
+ frame_bound_names,
+ names_assigned_by_statements,
+ names_read_before_bound,
+)
from pipelex.tools.jinja2.template_category import TemplateCategory
from pipelex.tools.misc.string_utils import get_root_from_dotted_path
@@ -42,20 +49,16 @@ class UnguardedOptionalReference:
path: str
-def _build_full_path(node: nodes.Node) -> str | None:
- """Build the dotted path of a Name / Getattr chain, or None for unsupported shapes."""
- if isinstance(node, nodes.Name):
- return node.name
- if isinstance(node, nodes.Getattr):
- parent_path = _build_full_path(node.node)
- if parent_path is not None:
- return f"{parent_path}.{node.attr}"
- return None
-
-
class _GuardWalker:
"""Recursive AST walk tracking which optional variables are currently guarded and which
names are locally declared (loop targets, macro params, `{% set %}` assignments).
+
+ Where a name is declared follows Jinja's scopes, from the rules `jinja2_scopes.py` states for
+ every template walker: after an `if`, only a name every branch sets is declared, so an optional
+ that only some branches rebind may still be read absent; a loop, a macro, a call, filter or set
+ block, a `with`, an `autoescape` and a block keep what they set to their own body; a macro sees
+ what its whole body binds; and a block without `scoped` reads only the template context, so
+ none of the names, nor the guards on them, that the frames around it bind reach it.
"""
def __init__(self, *, optional_variable_names: set[str]) -> None:
@@ -87,7 +90,14 @@ class _GuardWalker:
return self._guard_vars(test_node.left) | self._guard_vars(test_node.right)
return frozenset()
- def _walk_test(self, test_node: nodes.Node, *, guarded: frozenset[str], declared: frozenset[str]) -> None:
+ def _walk_test(
+ self,
+ test_node: nodes.Node,
+ *,
+ guarded: frozenset[str],
+ declared: frozenset[str],
+ context_declared: frozenset[str] | None,
+ ) -> None:
"""Walk a test position: bare names and presence tests are safe references there."""
if isinstance(test_node, nodes.Name):
return
@@ -96,83 +106,107 @@ class _GuardWalker:
if isinstance(test_node, nodes.And):
# `and` short-circuits: the right operand only evaluates when the left is truthy,
# so the left operand's guard extends over the right (`{% if var and var.attr %}`).
- self._walk_test(test_node.left, guarded=guarded, declared=declared)
- self._walk_test(test_node.right, guarded=guarded | self._guard_vars(test_node.left), declared=declared)
+ self._walk_test(test_node.left, guarded=guarded, declared=declared, context_declared=context_declared)
+ right_guarded = guarded | self._guard_vars(test_node.left)
+ self._walk_test(test_node.right, guarded=right_guarded, declared=declared, context_declared=context_declared)
return
if isinstance(test_node, nodes.Or):
- self._walk_test(test_node.left, guarded=guarded, declared=declared)
- self._walk_test(test_node.right, guarded=guarded, declared=declared)
+ self._walk_test(test_node.left, guarded=guarded, declared=declared, context_declared=context_declared)
+ self._walk_test(test_node.right, guarded=guarded, declared=declared, context_declared=context_declared)
return
if isinstance(test_node, nodes.Not):
- self._walk_test(test_node.node, guarded=guarded, declared=declared)
+ self._walk_test(test_node.node, guarded=guarded, declared=declared, context_declared=context_declared)
return
- self.walk(test_node, guarded=guarded, declared=declared)
-
- def _walk_body(self, body_nodes: list[nodes.Node], *, guarded: frozenset[str], declared: frozenset[str]) -> None:
- """Walk a statement body sequentially: a `{% set %}` or `{% macro %}` declares its name
- for SUBSEQUENT statements only — a read occurring before the assignment still refers to
- the (possibly undefined) context value and must be classified against it.
+ self.walk(test_node, guarded=guarded, declared=declared, context_declared=context_declared)
+
+ def _walk_body(
+ self,
+ body_nodes: list[nodes.Node],
+ *,
+ guarded: frozenset[str],
+ declared: frozenset[str],
+ context_declared: frozenset[str] | None,
+ look_ahead: frozenset[str] | None = None,
+ ) -> frozenset[str]:
+ """Walk a statement body sequentially, and return the names declared after it.
+
+ What a statement binds for certain (a `{% set %}`, a `{% macro %}`, a name every branch of an
+ `if` sets) is declared for SUBSEQUENT statements only — a read occurring before the assignment
+ still refers to the (possibly undefined) context value and must be classified against it. A
+ macro runs where it is called, so it also sees `look_ahead`, what its frame binds for certain
+ wherever it stands, but for the names the frame reads before binding them, whose copy Jinja
+ starts from the input; it is computed from the body when the body is a frame of its own, and
+ given for the branches of an `if`.
+ `context_declared` is what the template context holds where the body runs, and None at the top
+ level, whose names the context takes on as they are set.
"""
+ if look_ahead is None:
+ look_ahead = frozenset(names_assigned_by_statements(body_nodes) - names_read_before_bound(body_nodes))
for body_node in body_nodes:
- if isinstance(body_node, nodes.Assign):
- # The assignment's right-hand side is evaluated against the current scope.
- self.walk(body_node.node, guarded=guarded, declared=declared)
- declared |= self._assign_target_names(body_node.target)
- continue
- if isinstance(body_node, nodes.AssignBlock):
- # `{% set x %}...{% endset %}`: the body (and filter) evaluate against the
- # current scope; the target declares for subsequent statements only — and the
- # target itself is a store, never a read.
- self._walk_body(body_node.body, guarded=guarded, declared=declared)
+ if isinstance(body_node, nodes.If):
+ self._walk_if(body_node, guarded=guarded, declared=declared, context_declared=context_declared, look_ahead=look_ahead)
+ elif isinstance(body_node, nodes.Assign):
+ # The assignment's right-hand side is evaluated against the current scope; the
+ # target is a store, never a read.
+ self.walk(body_node.node, guarded=guarded, declared=declared, context_declared=context_declared)
+ elif isinstance(body_node, nodes.AssignBlock):
+ # `{% set x %}...{% endset %}`: the body runs in a frame of its own, so what it sets
+ # stays in it, and the filter is applied in that frame once the body has run; the
+ # target is a store, never a read.
+ frame_context = declared if context_declared is None else context_declared
+ body_declared = self._walk_body(body_node.body, guarded=guarded, declared=declared, context_declared=frame_context)
if body_node.filter is not None:
- self.walk(body_node.filter, guarded=guarded, declared=declared)
- declared |= self._assign_target_names(body_node.target)
- continue
- if isinstance(body_node, nodes.Macro):
- declared |= {body_node.name}
- self.walk(body_node, guarded=guarded, declared=declared)
-
- @classmethod
- def _assign_target_names(cls, target: nodes.Node) -> frozenset[str]:
- """The names a `{% set %}` target declares — a plain Name or a Tuple of Names."""
- if isinstance(target, nodes.Name):
- return frozenset({target.name})
- if isinstance(target, nodes.Tuple):
- return frozenset(item.name for item in target.items if isinstance(item, nodes.Name))
- return frozenset()
+ self.walk(body_node.filter, guarded=guarded, declared=body_declared, context_declared=frame_context)
+ elif isinstance(body_node, nodes.Macro):
+ self.walk(body_node, guarded=guarded, declared=declared | look_ahead, context_declared=context_declared)
+ else:
+ self.walk(body_node, guarded=guarded, declared=declared, context_declared=context_declared)
+ declared |= definitely_assigned_names(body_node)
+ return declared
+
+ def _walk_if(
+ self,
+ node: nodes.If,
+ *,
+ guarded: frozenset[str],
+ declared: frozenset[str],
+ context_declared: frozenset[str] | None,
+ look_ahead: frozenset[str],
+ ) -> None:
+ """An `if` opens no frame: each branch, and each `elif` test, starts from what is declared before it."""
+ for branch in (node, *node.elif_):
+ self._walk_test(branch.test, guarded=guarded, declared=declared, context_declared=context_declared)
+ branch_guarded = guarded | self._guard_vars(branch.test)
+ self._walk_body(branch.body, guarded=branch_guarded, declared=declared, context_declared=context_declared, look_ahead=look_ahead)
+ self._walk_body(node.else_, guarded=guarded, declared=declared, context_declared=context_declared, look_ahead=look_ahead)
+
+ def walk(
+ self,
+ node: nodes.Node,
+ *,
+ guarded: frozenset[str],
+ declared: frozenset[str],
+ context_declared: frozenset[str] | None,
+ ) -> None:
+ # What the template context holds in a frame opened here: from the top level, what is declared where it opens
+ frame_context = declared if context_declared is None else context_declared
- def walk(self, node: nodes.Node, *, guarded: frozenset[str], declared: frozenset[str]) -> None:
if isinstance(node, nodes.Template):
- self._walk_body(node.body, guarded=guarded, declared=declared)
- return
-
- if isinstance(node, nodes.If):
- body_guarded = guarded | self._guard_vars(node.test)
- self._walk_test(node.test, guarded=guarded, declared=declared)
- self._walk_body(node.body, guarded=body_guarded, declared=declared)
- for elif_node in node.elif_:
- self.walk(elif_node, guarded=guarded, declared=declared)
- self._walk_body(node.else_, guarded=guarded, declared=declared)
+ self._walk_body(node.body, guarded=guarded, declared=declared, context_declared=context_declared)
return
if isinstance(node, nodes.For):
# The iterable is evaluated before the loop targets bind.
- self.walk(node.iter, guarded=guarded, declared=declared)
- loop_declared: set[str] = {"loop"}
- if isinstance(node.target, nodes.Name):
- loop_declared.add(node.target.name)
- elif isinstance(node.target, nodes.Tuple):
- for item in node.target.items:
- if isinstance(item, nodes.Name):
- loop_declared.add(item.name)
+ self.walk(node.iter, guarded=guarded, declared=declared, context_declared=context_declared)
+ loop_declared = declared | frame_bound_names(node)
if node.test is not None:
# The loop filter evaluates after the target binds per item, so a shadowing
# target stays a local while an optional read in the filter gets classified.
# Its guard vars deliberately do NOT bless the body (same conservative stance
# as else-arms and inverted guards).
- self._walk_test(node.test, guarded=guarded, declared=declared | loop_declared)
- self._walk_body(node.body, guarded=guarded, declared=declared | loop_declared)
- self._walk_body(node.else_, guarded=guarded, declared=declared)
+ self._walk_test(node.test, guarded=guarded, declared=loop_declared, context_declared=frame_context)
+ self._walk_body(node.body, guarded=guarded, declared=loop_declared, context_declared=frame_context)
+ self._walk_body(node.else_, guarded=guarded, declared=declared, context_declared=frame_context)
return
if isinstance(node, nodes.With):
@@ -180,31 +214,77 @@ class _GuardWalker:
# locals, never the shadowed optionals. Walking the targets as children would
# miscount the store-context Names as reads.
for value in node.values:
- self.walk(value, guarded=guarded, declared=declared)
- with_declared = frozenset(target.name for target in node.targets if isinstance(target, nodes.Name))
- self._walk_body(node.body, guarded=guarded, declared=declared | with_declared)
+ self.walk(value, guarded=guarded, declared=declared, context_declared=context_declared)
+ with_declared = declared | frame_bound_names(node)
+ self._walk_body(node.body, guarded=guarded, declared=with_declared, context_declared=frame_context)
return
- if isinstance(node, nodes.Macro):
- macro_declared = frozenset(arg.name for arg in node.args)
- self._walk_body(node.body, guarded=guarded, declared=declared | macro_declared)
+ if isinstance(node, (nodes.Macro, nodes.CallBlock)):
+ if isinstance(node, nodes.CallBlock):
+ # The call is made from the enclosing scope; the block's body is the macro it passes as `caller`.
+ self.walk(node.call, guarded=guarded, declared=declared, context_declared=context_declared)
+ # The arguments are locals of the macro, and a default is evaluated where they are bound.
+ macro_declared = declared | frame_bound_names(node)
+ for default in node.defaults:
+ self.walk(default, guarded=guarded, declared=macro_declared, context_declared=frame_context)
+ self._walk_body(node.body, guarded=guarded, declared=macro_declared, context_declared=frame_context)
+ return
+
+ if isinstance(node, nodes.FilterBlock):
+ # The body runs in a frame of its own, so what it sets stays in it, and the filter is
+ # applied in that frame once the body has run.
+ body_declared = self._walk_body(node.body, guarded=guarded, declared=declared, context_declared=frame_context)
+ self.walk(node.filter, guarded=guarded, declared=body_declared, context_declared=frame_context)
+ return
+
+ if isinstance(node, nodes.Scope):
+ # `{% autoescape %}` parses into a scope: a frame of its own.
+ self._walk_body(node.body, guarded=guarded, declared=declared, context_declared=frame_context)
+ return
+
+ if isinstance(node, nodes.OverlayScope):
+ self.walk(node.context, guarded=guarded, declared=declared, context_declared=context_declared)
+ self._walk_body(node.body, guarded=guarded, declared=declared, context_declared=frame_context)
+ return
+
+ if isinstance(node, nodes.ScopedEvalContextModifier):
+ # Its statements run in the frame it stands in.
+ for option in node.options:
+ self.walk(option, guarded=guarded, declared=declared, context_declared=context_declared)
+ self._walk_body(node.body, guarded=guarded, declared=declared, context_declared=context_declared)
+ return
+
+ if isinstance(node, nodes.Block):
+ if node.scoped:
+ # Called with a context derived from the names declared where it stands, which is
+ # then the context of the blocks nested in it.
+ self._walk_body(node.body, guarded=guarded, declared=declared, context_declared=declared)
+ else:
+ # Called with the template context alone: a guard on a name a frame around it binds
+ # tested that local, not the context's value the block reads.
+ block_guarded = guarded - (declared - frame_context)
+ self._walk_body(node.body, guarded=block_guarded, declared=frame_context, context_declared=frame_context)
return
if isinstance(node, nodes.CondExpr):
expr1_guarded = guarded | self._guard_vars(node.test)
- self._walk_test(node.test, guarded=guarded, declared=declared)
- self.walk(node.expr1, guarded=expr1_guarded, declared=declared)
+ self._walk_test(node.test, guarded=guarded, declared=declared, context_declared=context_declared)
+ self.walk(node.expr1, guarded=expr1_guarded, declared=declared, context_declared=context_declared)
if node.expr2 is not None:
- self.walk(node.expr2, guarded=guarded, declared=declared)
+ self.walk(node.expr2, guarded=guarded, declared=declared, context_declared=context_declared)
+ return
+
+ if isinstance(node, nodes.Name) and node.ctx != "load":
+ # The target of a `set` or a `for`, and a macro argument, are written, not read.
return
if isinstance(node, (nodes.Name, nodes.Getattr)):
- full_path = _build_full_path(node)
+ full_path = dotted_attribute_path(node)
if full_path is None:
# Not a plain Name/Getattr chain (e.g. an attribute on a subscript or call
# result): keep walking inward so inner references still get classified.
for child in node.iter_child_nodes():
- self.walk(child, guarded=guarded, declared=declared)
+ self.walk(child, guarded=guarded, declared=declared, context_declared=context_declared)
return
if self._is_unguarded_optional(full_path, guarded=guarded, declared=declared):
self._record(full_path)
@@ -212,7 +292,7 @@ class _GuardWalker:
return
for child in node.iter_child_nodes():
- self.walk(child, guarded=guarded, declared=declared)
+ self.walk(child, guarded=guarded, declared=declared, context_declared=context_declared)
def detect_unguarded_optional_references(
@@ -244,5 +324,5 @@ def detect_unguarded_optional_references(
raise Jinja2DetectVariablesError(msg) from syntax_error
walker = _GuardWalker(optional_variable_names=optional_variable_names)
- walker.walk(parsed_ast, guarded=frozenset(), declared=frozenset())
+ walker.walk(parsed_ast, guarded=frozenset(), declared=frozenset(), context_declared=None)
return walker.findings
diff --git a/pipelex/tools/jinja2/jinja2_required_variables.py b/pipelex/tools/jinja2/jinja2_required_variables.py
index 8bd4e4755..96497e2cf 100644
--- a/pipelex/tools/jinja2/jinja2_required_variables.py
+++ b/pipelex/tools/jinja2/jinja2_required_variables.py
@@ -1,5 +1,11 @@
+"""The variables a template reads from its inputs, and the references through which it attaches images and documents.
+
+Both detectors are handlers on the read walk of `jinja2_scopes.py`, which follows Jinja's own scopes: a name the
+template binds (a `{% set %}`, a loop target, a macro argument) is not reported where Jinja binds it, and is
+reported as the input of that name everywhere else, such as after an `if` whose branches do not all set it.
+"""
+
from dataclasses import field
-from typing import Protocol
from jinja2 import nodes
from jinja2.exceptions import (
@@ -10,6 +16,7 @@ from pydantic.dataclasses import dataclass
from pipelex.tools.jinja2.exceptions import Jinja2DetectVariablesError, Jinja2StuffError
from pipelex.tools.jinja2.jinja2_environment import make_jinja2_env_without_loader
+from pipelex.tools.jinja2.jinja2_scopes import ScopeBindings, dotted_attribute_path, walk_template_reads
from pipelex.tools.jinja2.template_category import TemplateCategory
from pipelex.tools.misc.string_utils import get_root_from_dotted_path
@@ -30,107 +37,6 @@ class VariableReference:
filters: list[str] = field(default_factory=list[str])
-def _build_full_path(node: nodes.Node) -> str | None:
- """Recursively build the full dotted path from a Getattr or Name node.
-
- Args:
- node: A Jinja2 AST node (Name or Getattr)
-
- Returns:
- The full dotted path as a string, or None if the node structure is not supported.
- """
- if isinstance(node, nodes.Name):
- return node.name
- if isinstance(node, nodes.Getattr):
- parent_path = _build_full_path(node.node)
- if parent_path is not None:
- return f"{parent_path}.{node.attr}"
- return None
-
-
-def _set_target_name(*, statement: nodes.Node) -> str | None:
- """The name a `{% set %}` or a `{% set %}…{% endset %}` block assigns, when it assigns a plain name."""
- if isinstance(statement, (nodes.Assign, nodes.AssignBlock)) and isinstance(statement.target, nodes.Name):
- return statement.target.name
- return None
-
-
-class _ReadHandler(Protocol):
- """Deals with a node that may read a variable, given the names declared where it stands.
-
- Returns True when it has dealt with the node, which is then not walked into.
- """
-
- def __call__(self, *, node: nodes.Node, declared_names: set[str]) -> bool: ...
-
-
-def _walk_reads(*, node: nodes.Node, declared_names: set[str], global_names: set[str], handle_read: _ReadHandler) -> None:
- """Walk a template's AST in Jinja's scopes, handing every node that may read a variable to ``handle_read``.
-
- Both detectors below walk through here, so the names they treat as declared cannot drift apart: the required
- variables the input check compares with the inputs, and the references that attach images and documents.
-
- Args:
- node: The current AST node
- declared_names: The names declared where the node stands (a `set` before it, a loop target, a macro argument)
- global_names: The environment's globals (`range`, `namespace`, `dict`...)
- handle_read: What to do with a node that may read a variable
- """
- if isinstance(node, nodes.Template):
- # A macro's name is never an input, and a macro body, which runs when the macro is called, sees every
- # top-level `set`. Elsewhere a `set` declares its name only from its own statement on:
- # `{% set topic = topic|trim %}` reads the incoming `topic`, and so does a read placed before the `set`
- top_level_set_names = {name for statement in node.body if (name := _set_target_name(statement=statement)) is not None}
- scope_declared = declared_names | {statement.name for statement in node.body if isinstance(statement, nodes.Macro)}
- for statement in node.body:
- statement_declared = scope_declared | top_level_set_names if isinstance(statement, nodes.Macro) else scope_declared
- _walk_reads(node=statement, declared_names=statement_declared, global_names=global_names, handle_read=handle_read)
- if (name := _set_target_name(statement=statement)) is not None:
- scope_declared.add(name)
- return
-
- if isinstance(node, nodes.For):
- # The iterable and the `else` branch are read outside the loop, the body and the loop filter inside it,
- # where the loop target and the special `loop` variable are declared
- loop_declared = declared_names | {"loop"}
- if isinstance(node.target, nodes.Name):
- loop_declared.add(node.target.name)
- elif isinstance(node.target, nodes.Tuple):
- loop_declared.update(item.name for item in node.target.items if isinstance(item, nodes.Name))
- # Walked in source order, which is the order the image references are attached in
- scoped_nodes = [
- (node.iter, declared_names),
- *((statement, loop_declared) for statement in node.body),
- *((statement, declared_names) for statement in node.else_),
- *([(node.test, loop_declared)] if node.test is not None else []),
- ]
- for scoped_node, scope_declared_names in scoped_nodes:
- _walk_reads(node=scoped_node, declared_names=scope_declared_names, global_names=global_names, handle_read=handle_read)
- return
-
- if isinstance(node, nodes.Macro):
- # Macro parameters, and the names Jinja provides inside a macro body, are locally declared
- declared_names = declared_names | {arg.name for arg in node.args} | {"caller", "varargs", "kwargs"}
-
- if isinstance(node, nodes.Name) and node.ctx != "load":
- # The target of a `set` or a `for` is written, not read
- return
-
- if isinstance(node, nodes.Call) and isinstance(node.node, nodes.Name) and node.node.name in global_names:
- # A call to a Jinja global (`range(count)`, `namespace()`) reads only its arguments. Any other read of the
- # name (`{{ range }}`, `range.low`) reads the input of that name, which shadows the global when rendered
- for child in node.iter_child_nodes():
- if child is not node.node:
- _walk_reads(node=child, declared_names=declared_names, global_names=global_names, handle_read=handle_read)
- return
-
- if handle_read(node=node, declared_names=declared_names):
- return
-
- for child in node.iter_child_nodes():
- _walk_reads(node=child, declared_names=declared_names, global_names=global_names, handle_read=handle_read)
-
-
def detect_jinja2_required_variables(
*,
template_category: TemplateCategory,
@@ -170,19 +76,19 @@ def detect_jinja2_required_variables(
paths: set[str] = set()
- def collect_full_path(*, node: nodes.Node, declared_names: set[str]) -> bool:
+ def collect_full_path(*, node: nodes.Node, bindings: ScopeBindings) -> bool:
# Only the full path of an access chain is collected: `{{ foo.bar.baz }}` gives `foo.bar.baz`, not `foo.bar` or `foo`
if not isinstance(node, (nodes.Name, nodes.Getattr)):
return False
- full_path = _build_full_path(node)
+ full_path = dotted_attribute_path(node)
if full_path is None:
# An attribute on a subscript, a call or a filter (`items[0].text`) reads the path its chain starts from
return False
- if get_root_from_dotted_path(full_path) not in declared_names:
+ if get_root_from_dotted_path(full_path) not in bindings:
paths.add(full_path)
return True
- _walk_reads(node=parsed_ast, declared_names=set(), global_names=set(jinja2_env.globals), handle_read=collect_full_path)
+ walk_template_reads(template=parsed_ast, global_names=set(jinja2_env.globals), handle_read=collect_full_path)
return paths
@@ -248,7 +154,7 @@ def detect_jinja2_variable_references(
references: dict[str, VariableReference] = {}
- def collect_reference(*, node: nodes.Node, declared_names: set[str]) -> bool:
+ def collect_reference(*, node: nodes.Node, bindings: ScopeBindings) -> bool:
# A filter chain is taken whole, with its filter names; a chain or a path that stops at a subscript or a
# call (`items[0].image`) gives no reference, since no image or document can be resolved from it
if isinstance(node, nodes.Filter):
@@ -257,8 +163,8 @@ def detect_jinja2_variable_references(
filters, base_node = [], node
else:
return False
- full_path = _build_full_path(base_node) if base_node is not None else None
- if full_path is None or get_root_from_dotted_path(full_path) in declared_names:
+ full_path = dotted_attribute_path(base_node) if base_node is not None else None
+ if full_path is None or get_root_from_dotted_path(full_path) in bindings:
return True
if full_path in references:
# The same variable referenced several times combines its filters
@@ -269,5 +175,5 @@ def detect_jinja2_variable_references(
references[full_path] = VariableReference(path=full_path, filters=filters)
return True
- _walk_reads(node=parsed_ast, declared_names=set(), global_names=set(jinja2_env.globals), handle_read=collect_reference)
+ walk_template_reads(template=parsed_ast, global_names=set(jinja2_env.globals), handle_read=collect_reference)
return list(references.values())
diff --git a/pipelex/tools/jinja2/jinja2_scopes.py b/pipelex/tools/jinja2/jinja2_scopes.py
new file mode 100644
index 000000000..09cc45194
--- /dev/null
+++ b/pipelex/tools/jinja2/jinja2_scopes.py
@@ -0,0 +1,340 @@
+"""Where Jinja binds a name a template sets, and the one read walk the template detectors share.
+
+The input check, the image and document references and PipeDocGen's field check all ask which of the names a
+template reads come from its inputs. A name the template binds itself (a `{% set %}`, a loop target, a macro
+argument) does not, from where Jinja binds it, and the rule is stated here once, as Jinja's compiler applies it
+(`jinja2/idtracking.py`):
+
+- A loop body, a loop's `else` branch, a macro body, a call block, a filter block, a set block's body and a
+ `with` body each run in a frame of their own, and so does `{% autoescape %}`, which Jinja parses into a scope:
+ a name set inside one is not bound after it.
+- An `if` opens no frame, and each of its branches starts from the names bound before it. After the `if`, a name
+ every branch sets, the `else` included, is bound; a name only some branches set is the input of that name on
+ every path where none of them ran, so a read of it after the `if` reads that input.
+- Within a frame, a name is bound from the statement that binds it on: a read placed before a `set`, and the
+ `set`'s own right-hand side (`{% set topic = topic|trim %}`), read the input.
+- A macro body runs where the macro is called, which the walk cannot place among the statements, so it sees the
+ names its frame binds for certain, wherever they stand in it, an `if` branch it is defined in opening no frame of
+ its own. A name the frame reads before binding it is the exception: Jinja starts the frame's copy of it from the
+ input, so a macro called before the binding reads the input.
+- A `{% block %}` runs as a function of its own. Without `scoped` it reads only the template context, which holds
+ what the top level has set for certain before it, and none of the names a loop, a macro or any other frame around
+ it binds; a scoped block reads the names bound where it stands, and the blocks nested in it read those too.
+
+The walk hands every node that may read a variable to a handler, with the bindings where the node stands. A
+binding maps a name to the input path it stands for when that can be followed, which is a loop target over an
+attribute chain, and to None otherwise.
+"""
+
+from typing import Protocol
+
+from jinja2 import nodes
+
+# The segment standing for "an item of the list" in a path reached through a loop variable.
+LIST_ITEM_SEGMENT = "[]"
+
+# The names a template binds where a node stands, each mapped to the input path it stands for, or to None when it
+# cannot be followed (a `set`, a macro argument, `loop`). A name absent from the bindings is an input, or a global.
+ScopeBindings = dict[str, tuple[str, ...] | None]
+
+
+def assigned_target_names(target: nodes.Node) -> list[str]:
+ """The names a `set`, `with` or loop target binds: a plain name or a tuple of them, nested tuples included.
+
+ A namespace attribute (`{% set ns.found = true %}`) binds no name.
+ """
+ if isinstance(target, nodes.Name):
+ return [target.name]
+ if isinstance(target, nodes.Tuple):
+ return [name for item in target.items for name in assigned_target_names(item)]
+ return []
+
+
+def names_assigned_by_statements(statements: list[nodes.Node]) -> set[str]:
+ names: set[str] = set()
+ for statement in statements:
+ names |= definitely_assigned_names(statement)
+ return names
+
+
+def definitely_assigned_names(statement: nodes.Node) -> set[str]:
+ """The names a statement leaves bound, whichever way it runs, in the frame it runs in.
+
+ A set or a set block binds its targets, a macro its name, an import its alias; an `if` binds what every one of
+ its branches binds, nothing when it has no `else`; the statements of an `autoescape` run in the frame they stand
+ in; and a statement that opens a frame of its own binds nothing outside it.
+ """
+ if isinstance(statement, (nodes.Assign, nodes.AssignBlock)):
+ return set(assigned_target_names(statement.target))
+ if isinstance(statement, nodes.Macro):
+ return {statement.name}
+ if isinstance(statement, nodes.Import):
+ return {statement.target}
+ if isinstance(statement, nodes.FromImport):
+ return {name[1] if isinstance(name, tuple) else name for name in statement.names}
+ if isinstance(statement, nodes.If):
+ if not statement.else_:
+ return set()
+ body_names = names_assigned_by_statements(statement.body)
+ other_branches = [*(elif_branch.body for elif_branch in statement.elif_), statement.else_]
+ return body_names.intersection(*(names_assigned_by_statements(branch) for branch in other_branches))
+ if isinstance(statement, nodes.ScopedEvalContextModifier):
+ return names_assigned_by_statements(statement.body)
+ return set()
+
+
+def _frame_level_parts(statement: nodes.Node) -> list[nodes.Node]:
+ """The expressions of a statement Jinja reads in the frame the statement stands in."""
+ if isinstance(statement, nodes.Output):
+ return list(statement.nodes)
+ if isinstance(statement, (nodes.Assign, nodes.ExprStmt)):
+ return [statement.node]
+ if isinstance(statement, nodes.For):
+ return [statement.iter]
+ if isinstance(statement, nodes.With):
+ return list(statement.values)
+ if isinstance(statement, nodes.CallBlock):
+ return [statement.call]
+ if isinstance(statement, nodes.FilterBlock):
+ return [statement.filter]
+ if isinstance(statement, nodes.OverlayScope):
+ return [statement.context]
+ if isinstance(statement, (nodes.Import, nodes.FromImport)):
+ return [statement.template]
+ return []
+
+
+def _loaded_names(expression: nodes.Node) -> set[str]:
+ names = [expression, *expression.find_all(nodes.Name)]
+ return {name.name for name in names if isinstance(name, nodes.Name) and name.ctx == "load"}
+
+
+def _collect_reads_before_bound(*, statements: list[nodes.Node], bound: set[str], read: set[str]) -> None:
+ bound = set(bound)
+ for statement in statements:
+ if isinstance(statement, nodes.If):
+ for branch in (statement, *statement.elif_):
+ read |= _loaded_names(branch.test) - bound
+ _collect_reads_before_bound(statements=branch.body, bound=bound, read=read)
+ _collect_reads_before_bound(statements=statement.else_, bound=bound, read=read)
+ elif isinstance(statement, nodes.ScopedEvalContextModifier):
+ _collect_reads_before_bound(statements=statement.body, bound=bound, read=read)
+ else:
+ for part in _frame_level_parts(statement):
+ read |= _loaded_names(part) - bound
+ bound |= definitely_assigned_names(statement)
+
+
+def names_read_before_bound(statements: list[nodes.Node]) -> set[str]:
+ """The names a frame reads at its own level before it binds them for certain.
+
+ Jinja starts the frame's copy of such a name from the input of that name, so a macro the frame defines reads the
+ input when it is called before the binding.
+ """
+ read: set[str] = set()
+ _collect_reads_before_bound(statements=statements, bound=set(), read=read)
+ return read
+
+
+def frame_bound_names(statement: nodes.Node) -> set[str]:
+ """The names a statement binds in the frame it opens, before its body runs: a loop's target and `loop`, a
+ `with`'s targets, and a macro's or a call block's arguments with the names Jinja provides in a macro.
+ """
+ if isinstance(statement, nodes.For):
+ return {"loop", *assigned_target_names(statement.target)}
+ if isinstance(statement, nodes.With):
+ return {name for target in statement.targets for name in assigned_target_names(target)}
+ if isinstance(statement, (nodes.Macro, nodes.CallBlock)):
+ return {*(argument.name for argument in statement.args), "caller", "varargs", "kwargs"}
+ return set()
+
+
+def attribute_chain(node: nodes.Node) -> tuple[str, list[str]] | None:
+ """The name and attribute names of a pure attribute chain (`a.b.c`), or None for any other shape."""
+ attributes: list[str] = []
+ current = node
+ while isinstance(current, nodes.Getattr):
+ attributes.append(current.attr)
+ current = current.node
+ if isinstance(current, nodes.Name):
+ return current.name, list(reversed(attributes))
+ return None
+
+
+def dotted_attribute_path(node: nodes.Node) -> str | None:
+ """The dotted path of a pure attribute chain (`user.profile.name`), or None for any other shape."""
+ chain = attribute_chain(node)
+ if chain is None:
+ return None
+ name, attributes = chain
+ return ".".join([name, *attributes])
+
+
+def resolve_bound_path(*, name: str, attributes: list[str], bindings: ScopeBindings) -> tuple[str, ...] | None:
+ """The input path a chain reads, through the bindings: None when it starts from a name that cannot be followed."""
+ if name not in bindings:
+ return (name, *attributes)
+ bound_path = bindings[name]
+ if bound_path is None:
+ return None
+ return (*bound_path, *attributes)
+
+
+class ReadHandler(Protocol):
+ """Deals with a node that may read a variable, given the bindings where it stands.
+
+ Returns True when it has dealt with the node, which is then not walked into.
+ """
+
+ def __call__(self, *, node: nodes.Node, bindings: ScopeBindings) -> bool: ...
+
+
+def _with_unfollowed(*, bindings: ScopeBindings, names: set[str]) -> ScopeBindings:
+ extended = dict(bindings)
+ for name in names:
+ extended[name] = None
+ return extended
+
+
+class _ReadWalk:
+ def __init__(self, *, global_names: set[str], handle_read: ReadHandler) -> None:
+ self.global_names = global_names
+ self.handle_read = handle_read
+
+ def walk_statements(
+ self,
+ *,
+ statements: list[nodes.Node],
+ bindings: ScopeBindings,
+ context: ScopeBindings | None,
+ look_ahead: set[str] | None = None,
+ ) -> ScopeBindings:
+ """Walk a statement list in the frame it runs in, and return the bindings after it.
+
+ Each statement sees what the statements before it bound for certain. A macro also sees `look_ahead`, what its
+ frame binds for certain wherever it stands, but for the names the frame reads before binding them; it is
+ computed from the statements when they make up a frame of their own, and given for the branches of an `if`.
+ `context` is what the template context holds where the frame runs, and None in the top-level frame, whose
+ names the context takes on as they are set.
+ """
+ if look_ahead is None:
+ look_ahead = names_assigned_by_statements(statements) - names_read_before_bound(statements)
+ scope = dict(bindings)
+ for statement in statements:
+ if isinstance(statement, nodes.If):
+ self._walk_if(node=statement, bindings=scope, context=context, look_ahead=look_ahead)
+ elif isinstance(statement, nodes.Macro):
+ self.walk(node=statement, bindings=_with_unfollowed(bindings=scope, names=look_ahead), context=context)
+ else:
+ self.walk(node=statement, bindings=scope, context=context)
+ scope = _with_unfollowed(bindings=scope, names=definitely_assigned_names(statement))
+ return scope
+
+ def _walk_if(self, *, node: nodes.If, bindings: ScopeBindings, context: ScopeBindings | None, look_ahead: set[str]) -> None:
+ # An `if` opens no frame: every branch, and every `elif` test, starts from the bindings before it
+ for branch in (node, *node.elif_):
+ self.walk(node=branch.test, bindings=bindings, context=context)
+ self.walk_statements(statements=branch.body, bindings=bindings, context=context, look_ahead=look_ahead)
+ self.walk_statements(statements=node.else_, bindings=bindings, context=context, look_ahead=look_ahead)
+
+ def walk(self, *, node: nodes.Node, bindings: ScopeBindings, context: ScopeBindings | None) -> None:
+ # What the template context holds in a frame opened here: from the top level, what is bound where it opens
+ frame_context = bindings if context is None else context
+
+ if isinstance(node, nodes.For):
+ # The iterable is read outside the loop, the loop filter and the body inside it, and the `else` branch in
+ # a frame of its own. A single loop target over an attribute chain reads an item of that chain's list
+ self.walk(node=node.iter, bindings=bindings, context=context)
+ loop_bindings = _with_unfollowed(bindings=bindings, names=frame_bound_names(node))
+ iterated_chain = attribute_chain(node.iter)
+ if isinstance(node.target, nodes.Name) and iterated_chain is not None:
+ iterated_name, iterated_attributes = iterated_chain
+ iterated_path = resolve_bound_path(name=iterated_name, attributes=iterated_attributes, bindings=bindings)
+ if iterated_path is not None:
+ loop_bindings[node.target.name] = (*iterated_path, LIST_ITEM_SEGMENT)
+ if node.test is not None:
+ self.walk(node=node.test, bindings=loop_bindings, context=frame_context)
+ self.walk_statements(statements=node.body, bindings=loop_bindings, context=frame_context)
+ self.walk_statements(statements=node.else_, bindings=bindings, context=frame_context)
+ return
+
+ if isinstance(node, nodes.With):
+ # The values are read outside the `with`, before its targets are bound
+ for value in node.values:
+ self.walk(node=value, bindings=bindings, context=context)
+ with_bindings = _with_unfollowed(bindings=bindings, names=frame_bound_names(node))
+ self.walk_statements(statements=node.body, bindings=with_bindings, context=frame_context)
+ return
+
+ if isinstance(node, (nodes.Macro, nodes.CallBlock)):
+ if isinstance(node, nodes.CallBlock):
+ # The call is made from the enclosing frame; the block's body is the macro it passes as `caller`
+ self.walk(node=node.call, bindings=bindings, context=context)
+ macro_bindings = _with_unfollowed(bindings=bindings, names=frame_bound_names(node))
+ # A default is evaluated in the macro's frame, where the arguments are bound
+ for default in node.defaults:
+ self.walk(node=default, bindings=macro_bindings, context=frame_context)
+ self.walk_statements(statements=node.body, bindings=macro_bindings, context=frame_context)
+ return
+
+ if isinstance(node, (nodes.FilterBlock, nodes.AssignBlock)):
+ # The body runs in a frame of its own, and the filter is applied in that frame once the body has run
+ body_bindings = self.walk_statements(statements=node.body, bindings=bindings, context=frame_context)
+ if node.filter is not None:
+ self.walk(node=node.filter, bindings=body_bindings, context=frame_context)
+ return
+
+ if isinstance(node, nodes.Scope):
+ self.walk_statements(statements=node.body, bindings=bindings, context=frame_context)
+ return
+
+ if isinstance(node, nodes.Block):
+ # A block is called with the template context, or, when scoped, with one derived from the names bound where
+ # it stands, which is then the context of the blocks nested in it
+ block_bindings = bindings if node.scoped else frame_context
+ self.walk_statements(statements=node.body, bindings=block_bindings, context=block_bindings)
+ return
+
+ if isinstance(node, nodes.OverlayScope):
+ self.walk(node=node.context, bindings=bindings, context=context)
+ self.walk_statements(statements=node.body, bindings=bindings, context=frame_context)
+ return
+
+ if isinstance(node, nodes.ScopedEvalContextModifier):
+ # Its statements run in the frame it stands in, the scope Jinja wraps an `autoescape` in
+ for option in node.options:
+ self.walk(node=option, bindings=bindings, context=context)
+ self.walk_statements(statements=node.body, bindings=bindings, context=context)
+ return
+
+ if isinstance(node, nodes.Name) and node.ctx != "load":
+ # The target of a `set` or a `for`, and a macro argument, are written, not read
+ return
+
+ if isinstance(node, nodes.Call) and isinstance(node.node, nodes.Name) and node.node.name in self.global_names:
+ # A call to a Jinja global (`range(count)`, `namespace()`) reads only its arguments. Any other read of the
+ # name (`{{ range }}`, `range.low`) reads the input of that name, which shadows the global when rendered
+ for child in node.iter_child_nodes():
+ if child is not node.node:
+ self.walk(node=child, bindings=bindings, context=context)
+ return
+
+ if self.handle_read(node=node, bindings=bindings):
+ return
+
+ for child in node.iter_child_nodes():
+ self.walk(node=child, bindings=bindings, context=context)
+
+
+def walk_template_reads(*, template: nodes.Template, global_names: set[str], handle_read: ReadHandler) -> None:
+ """Walk a template in Jinja's scopes, handing every node that may read a variable to `handle_read`.
+
+ The detectors that ask which inputs a template reads, and which fields of them, all walk through here, so the
+ names they treat as bound cannot drift apart.
+
+ Args:
+ template: The parsed template
+ global_names: The environment's globals (`range`, `namespace`, `dict`...)
+ handle_read: What to do with a node that may read a variable
+ """
+ _ReadWalk(global_names=global_names, handle_read=handle_read).walk_statements(statements=template.body, bindings={}, context=None)
diff --git a/subject_grants.toml b/subject_grants.toml
index 79a9d5731..7807388f0 100644
--- a/subject_grants.toml
+++ b/subject_grants.toml
@@ -6108,14 +6108,6 @@ rationale = "Factory from source: builds the result from the template_category;
param = "template_category"
rationale = "Factory from source: builds the result from the template_category; single operand."
-["pipelex/tools/jinja2/jinja2_field_paths.py::_chain"]
-param = "node"
-rationale = "Reads the attribute chain of one Jinja node."
-
-["pipelex/tools/jinja2/jinja2_field_paths.py::_target_names"]
-param = "target"
-rationale = "Lists the names one assignment target binds."
-
["pipelex/tools/jinja2/jinja2_filters.py::escape_script_tag"]
param = "value"
rationale = "Transforms the value; single operand."
@@ -6124,10 +6116,6 @@ rationale = "Transforms the value; single operand."
param = "value"
rationale = "Jinja filter: Jinja passes the filtered value positionally."
-["pipelex/tools/jinja2/jinja2_optional_guards.py::_GuardWalker._assign_target_names"]
-param = "target"
-rationale = "Noun-named derivation: the assigned names of the target node; single operand."
-
["pipelex/tools/jinja2/jinja2_optional_guards.py::_GuardWalker._guard_vars"]
param = "test_node"
rationale = "Noun-named derivation: the guarded vars of the test node; single operand."
@@ -6144,6 +6132,10 @@ rationale = "Verb-object: records the path; single operand."
param = "body_nodes"
rationale = "Visitor: walks the body nodes; guard sets stay keyword."
+["pipelex/tools/jinja2/jinja2_optional_guards.py::_GuardWalker._walk_if"]
+param = "node"
+rationale = "Verb-object walk over one if node, like the walker's other walk methods; single operand."
+
["pipelex/tools/jinja2/jinja2_optional_guards.py::_GuardWalker._walk_test"]
param = "test_node"
rationale = "Visitor: walks the test node; guard sets stay keyword."
@@ -6152,10 +6144,6 @@ rationale = "Visitor: walks the test node; guard sets stay keyword."
param = "node"
rationale = "Visitor: walks the node; guard sets stay keyword."
-["pipelex/tools/jinja2/jinja2_optional_guards.py::_build_full_path"]
-param = "node"
-rationale = "Noun-named derivation: the full dotted path of the node; single operand."
-
["pipelex/tools/jinja2/jinja2_parsing.py::check_jinja2_parsing"]
param = "template_source"
rationale = "Jinja op on the template_source; single operand (context keyword-only)."
@@ -6280,13 +6268,45 @@ rationale = "Jinja op on the template_source; single operand (context keyword-on
param = "template_source"
rationale = "Jinja op on the template_source; single operand (context keyword-only)."
-["pipelex/tools/jinja2/jinja2_required_variables.py::_build_full_path"]
+["pipelex/tools/jinja2/jinja2_required_variables.py::_extract_filters_and_variable"]
param = "node"
rationale = "Extracts/derives from the node; single operand."
-["pipelex/tools/jinja2/jinja2_required_variables.py::_extract_filters_and_variable"]
+["pipelex/tools/jinja2/jinja2_scopes.py::_frame_level_parts"]
+param = "statement"
+rationale = "Noun-named derivation over one Jinja AST operand; single operand."
+
+["pipelex/tools/jinja2/jinja2_scopes.py::_loaded_names"]
+param = "expression"
+rationale = "Noun-named derivation over one Jinja AST operand; single operand."
+
+["pipelex/tools/jinja2/jinja2_scopes.py::assigned_target_names"]
+param = "target"
+rationale = "Lists the names one assignment target binds."
+
+["pipelex/tools/jinja2/jinja2_scopes.py::attribute_chain"]
param = "node"
-rationale = "Extracts/derives from the node; single operand."
+rationale = "Reads the attribute chain of one Jinja node."
+
+["pipelex/tools/jinja2/jinja2_scopes.py::definitely_assigned_names"]
+param = "statement"
+rationale = "Noun-named derivation: the names one statement binds for certain; single operand."
+
+["pipelex/tools/jinja2/jinja2_scopes.py::dotted_attribute_path"]
+param = "node"
+rationale = "Noun-named derivation: the dotted path of the node; single operand."
+
+["pipelex/tools/jinja2/jinja2_scopes.py::frame_bound_names"]
+param = "statement"
+rationale = "Noun-named derivation: the names one statement binds in the frame it opens; single operand."
+
+["pipelex/tools/jinja2/jinja2_scopes.py::names_assigned_by_statements"]
+param = "statements"
+rationale = "Noun-named derivation: the names a statement list binds for certain; single operand."
+
+["pipelex/tools/jinja2/jinja2_scopes.py::names_read_before_bound"]
+param = "statements"
+rationale = "Noun-named derivation over one Jinja AST operand; single operand."
["pipelex/tools/jinja2/jinja2_template_loader.py::TemplateLoader.is_loaded"]
param = "name"
diff --git a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_load_refusals.py b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_load_refusals.py
index 83d5a3ffd..5677abb1d 100644
--- a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_load_refusals.py
+++ b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_load_refusals.py
@@ -77,8 +77,9 @@ class TestPipeDocGenLoadRefusals:
('format = "pdf"\ntemplate = "{% for line in invoice.line_items %}{{ line.amout }}{% endfor %}"', "amout"),
('format = "pdf"\nfilename = "invoice-{{ invoice.nmber }}"', "nmber"),
('format = "pdf"\ntemplate = "<h1>{{ invoice._stuff }}</h1>"', "_stuff"),
+ ('format = "pdf"\ntemplate = "{% if invoice.number %}{% set invoice = \'none\' %}{% endif %}<h1>{{ invoice.nosuch }}</h1>"', "nosuch"),
],
- ids=["misspelled_field", "misspelled_field_through_a_loop", "misspelled_field_in_the_filename", "private_name"],
+ ids=["misspelled_field", "misspelled_field_through_a_loop", "misspelled_field_in_the_filename", "private_name", "set_in_only_some_branches"],
)
async def test_a_template_mistake_is_refused_at_load(self, step_fields: str, expected: str) -> None:
"""A field a template or the filename names is checked against the input's concept, through loop variables too."""
diff --git a/tests/unit/pipelex/pipe_operators/pipe_compose/data.py b/tests/unit/pipelex/pipe_operators/pipe_compose/data.py
index aa2709f1f..65475898f 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_compose/data.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_compose/data.py
@@ -257,6 +257,15 @@ class PipeComposeInputCheckTestCases:
"self_referential_set",
{"description": "d", "inputs": {"topic": "Text"}, "output": "Text", "template": "{% set topic = topic|trim %}About {{ topic }}"},
),
+ (
+ "set_in_every_branch_of_an_if",
+ {
+ "description": "d",
+ "inputs": {"flag": "Text"},
+ "output": "Text",
+ "template": "{% if flag %}{% set label = 'on' %}{% else %}{% set label = 'off' %}{% endif %}Status: {{ label }}",
+ },
+ ),
(
"input_named_like_a_jinja_global",
{"description": "d", "inputs": {"namespace": "Text"}, "output": "Text", "template": "List the pods in $namespace"},
diff --git a/tests/unit/pipelex/pipe_operators/pipe_doc_gen/test_pipe_doc_gen_blueprint.py b/tests/unit/pipelex/pipe_operators/pipe_doc_gen/test_pipe_doc_gen_blueprint.py
index a9578113a..d50a5921f 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_doc_gen/test_pipe_doc_gen_blueprint.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_doc_gen/test_pipe_doc_gen_blueprint.py
@@ -43,6 +43,13 @@ class TestPipeDocGenBlueprint:
with pytest.raises(ValidationError, match=expected_message):
_blueprint(**overrides)
+ def test_a_name_every_branch_sets_is_the_template_s_own(self) -> None:
+ template = (
+ "{% if invoice.paid %}{% set status = 'Paid' %}{% else %}{% set status = 'Due' %}{% endif %}<h1>{{ invoice.number }}: {{ status }}</h1>"
+ )
+ blueprint = _blueprint(template=template)
+ assert blueprint.template == template
+
def test_a_sigil_in_the_template_reads_its_input(self) -> None:
blueprint = _blueprint(template="<h1>$invoice.number</h1>")
assert blueprint.template == "<h1>$invoice.number</h1>"
diff --git a/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_blueprint.py b/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_blueprint.py
index f6d4da99f..f76aefb0e 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_blueprint.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_blueprint.py
@@ -175,6 +175,44 @@ class TestPipeLLMBlueprint:
)
assert set(blueprint.input_names) == {"items"}
+ def test_validate_inputs_correct_with_a_set_read_in_its_loop_body(self):
+ """A name a loop body sets is the template's own where the body reads it, not an input."""
+ blueprint = PipeLLMBlueprint(
+ description="lorem ipsum",
+ inputs={"expenses": "native.Text[]"},
+ output="native.Text",
+ prompt="Plan these:\n{% for expense in expenses %}{% set day = expense.text %}- {{ day }}\n{% endfor %}",
+ )
+ assert set(blueprint.input_names) == {"expenses"}
+
+ def test_validate_inputs_incorrect_input_read_only_through_a_loop_set(self):
+ """An input declared only for a name the loop body sets is never read: the body reads its own `day`."""
+ error = refused_input_error(
+ blueprint_class=PipeLLMBlueprint,
+ blueprint_kwargs={
+ "description": "lorem ipsum",
+ "inputs": {"expenses": "native.Text[]", "day": "native.Text"},
+ "output": "native.Text",
+ "prompt": "{% for expense in expenses %}{% set day = expense.text %}- {{ day }}\n{% endfor %}",
+ },
+ )
+ assert error.error_type == PipeValidationErrorType.EXTRANEOUS_INPUT_VARIABLE
+ assert error.variable_names == ["day"]
+
+ def test_validate_inputs_incorrect_set_in_only_some_branches(self):
+ """A name only some branches of an `if` set is read from the inputs on the paths where none ran."""
+ error = refused_input_error(
+ blueprint_class=PipeLLMBlueprint,
+ blueprint_kwargs={
+ "description": "lorem ipsum",
+ "inputs": {"topic": "native.Text"},
+ "output": "native.Text",
+ "prompt": "{% if topic %}{% set tone = 'formal' %}{% endif %}Write about $topic in a {{ tone }} tone",
+ },
+ )
+ assert error.error_type == PipeValidationErrorType.MISSING_INPUT_VARIABLE
+ assert error.variable_names == ["tone"]
+
def test_validate_inputs_correct_with_input_named_like_a_jinja_global(self):
"""An input named like a Jinja global shadows it, so reading the name reads the input."""
blueprint = PipeLLMBlueprint(
diff --git a/tests/unit/pipelex/tools/test_jinja2_field_paths.py b/tests/unit/pipelex/tools/test_jinja2_field_paths.py
index 0eeb1ac5b..afb7120de 100644
--- a/tests/unit/pipelex/tools/test_jinja2_field_paths.py
+++ b/tests/unit/pipelex/tools/test_jinja2_field_paths.py
@@ -1,8 +1,12 @@
import pytest
from pipelex.tools.jinja2.exceptions import Jinja2DetectVariablesError
-from pipelex.tools.jinja2.jinja2_field_paths import LIST_ITEM_SEGMENT, detect_template_field_paths
+from pipelex.tools.jinja2.jinja2_field_paths import detect_template_field_paths
+from pipelex.tools.jinja2.jinja2_required_variables import detect_jinja2_required_variables
+from pipelex.tools.jinja2.jinja2_scopes import LIST_ITEM_SEGMENT
from pipelex.tools.jinja2.template_category import TemplateCategory
+from pipelex.tools.misc.string_utils import get_root_from_dotted_path
+from tests.unit.pipelex.tools.test_jinja2_required_variables import TestData as RequiredVariablesData
def _paths(template_source: str) -> list[tuple[str, ...]]:
@@ -42,6 +46,76 @@ class TestDetectTemplateFieldPaths:
paths = _paths(template_source)
assert not any("nosuch" in path or "index" in path for path in paths), f"{topic}: {paths}"
+ @pytest.mark.parametrize(
+ ("topic", "template_source"),
+ [
+ ("set_in_every_branch_rebinds_an_input", "{% if f %}{% set invoice = a %}{% else %}{% set invoice = b %}{% endif %}{{ invoice.nosuch }}"),
+ (
+ "loop_target_rebound_in_every_branch",
+ (
+ "{% for item in invoice.line_items %}{% if f %}{% set item = a %}{% else %}{% set item = b %}{% endif %}"
+ "{{ item.nosuch }}{% endfor %}"
+ ),
+ ),
+ ("set_in_a_call_block_body", "{% call m() %}{% set invoice = a %}{{ invoice.nosuch }}{% endcall %}"),
+ ("set_in_a_filter_block_body", "{% filter upper %}{% set invoice = a %}{{ invoice.nosuch }}{% endfilter %}"),
+ ("macro_default_reads_an_earlier_argument", "{% macro show(invoice, total=invoice.nosuch) %}{{ total }}{% endmacro %}"),
+ ],
+ )
+ def test_a_name_jinja_binds_gives_no_path(self, topic: str, template_source: str) -> None:
+ paths = _paths(template_source)
+ assert not any("nosuch" in path for path in paths), f"{topic}: {paths}"
+
+ def test_a_called_global_gives_no_path(self) -> None:
+ assert _paths("{% for i in range(3) %}{{ dict(a=i).a }}{% endfor %}") == []
+
+ @pytest.mark.parametrize(
+ ("topic", "template_source", "expected_path"),
+ [
+ ("set_in_some_branches", "{% if f %}{% set invoice = other %}{% endif %}{{ invoice.nosuch }}", ("invoice", "nosuch")),
+ (
+ "loop_target_rebound_in_some_branches",
+ "{% for item in invoice.line_items %}{% if f %}{% set item = other %}{% endif %}{{ item.nosuch }}{% endfor %}",
+ ("invoice", "line_items", LIST_ITEM_SEGMENT, "nosuch"),
+ ),
+ ("set_does_not_escape_a_loop", "{% for x in xs %}{% set invoice = x %}{% endfor %}{{ invoice.nosuch }}", ("invoice", "nosuch")),
+ (
+ "macro_reads_a_name_its_frame_reads_before_setting",
+ "{{ invoice }}{% macro m() %}{{ invoice.nosuch }}{% endmacro %}{{ m() }}{% set invoice = other %}",
+ ("invoice", "nosuch"),
+ ),
+ (
+ "loop_target_in_a_block",
+ "{% for invoice in invoices.all %}{% block b %}{{ invoice.nosuch }}{% endblock %}{% endfor %}",
+ ("invoice", "nosuch"),
+ ),
+ (
+ "loop_target_in_a_scoped_block",
+ "{% for item in invoice.line_items %}{% block b scoped %}{{ item.nosuch }}{% endblock %}{% endfor %}",
+ ("invoice", "line_items", LIST_ITEM_SEGMENT, "nosuch"),
+ ),
+ ],
+ )
+ def test_a_name_read_where_jinja_reads_the_input_keeps_its_path(self, topic: str, template_source: str, expected_path: tuple[str, ...]) -> None:
+ assert expected_path in _paths(template_source), f"{topic}: {_paths(template_source)}"
+
+ @pytest.mark.parametrize(
+ ("topic", "template_source", "expected_variables"),
+ RequiredVariablesData.JINJA_SCOPE_RULES,
+ )
+ def test_roots_match_the_required_variables(
+ self,
+ topic: str,
+ template_source: str,
+ expected_variables: set[str], # ruff: ignore[unused-method-argument]
+ ) -> None:
+ """The field paths and the required variables walk the same scopes, so they start from the same inputs."""
+ field_path_roots = {
+ field_path.root for field_path in detect_template_field_paths(template_source=template_source, template_category=TemplateCategory.HTML)
+ }
+ required = detect_jinja2_required_variables(template_category=TemplateCategory.HTML, template_source=template_source)
+ assert field_path_roots == {get_root_from_dotted_path(path) for path in required}, f"Failed for topic: {topic}"
+
def test_a_subscript_is_not_followed(self) -> None:
assert _paths("{{ invoice['customer'].name }}") == [("invoice",)]
diff --git a/tests/unit/pipelex/tools/test_jinja2_optional_guards.py b/tests/unit/pipelex/tools/test_jinja2_optional_guards.py
index 55b0df2ba..aebbaae1e 100644
--- a/tests/unit/pipelex/tools/test_jinja2_optional_guards.py
+++ b/tests/unit/pipelex/tools/test_jinja2_optional_guards.py
@@ -35,6 +35,42 @@ class TestDetectUnguardedOptionalReferences:
("with_shadow_local", "{% with assessment = 'x' %}{{ assessment }}{% endwith %}"),
("for_filter_bare_presence_probe", "{% for item in topic_items if assessment %}{{ item }}{% endfor %}"),
("for_filter_target_shadows", "{% for assessment in topic_items if assessment.flag %}{{ assessment }}{% endfor %}"),
+ ("set_in_every_branch", "{% if topic %}{% set assessment = 'a' %}{% else %}{% set assessment = 'b' %}{% endif %}{{ assessment.amount }}"),
+ (
+ "set_in_every_branch_with_elifs",
+ (
+ "{% if topic %}{% set assessment = 'a' %}{% elif other %}{% set assessment = 'b' %}"
+ "{% else %}{% set assessment = 'c' %}{% endif %}{{ assessment }}"
+ ),
+ ),
+ ("set_in_a_call_block_body", "{% call m() %}{% set assessment = 'x' %}{{ assessment.amount }}{% endcall %}"),
+ ("set_in_a_filter_block_body", "{% filter upper %}{% set assessment = 'x' %}{{ assessment.amount }}{% endfilter %}"),
+ ("call_block_argument_shadows", "{% call(assessment) m() %}{{ assessment.amount }}{% endcall %}"),
+ ("set_in_an_autoescape_block", "{% autoescape true %}{% set assessment = 'x' %}{{ assessment.amount }}{% endautoescape %}"),
+ ("set_in_a_block_body", "{% block b %}{% set assessment = 'x' %}{{ assessment.amount }}{% endblock %}"),
+ ("set_target_in_a_block_is_a_store_not_a_read", "{% block b %}{% set assessment = 'x' %}{% endblock %}"),
+ ("macro_reads_a_later_set", "{% macro show() %}{{ assessment.amount }}{% endmacro %}{% set assessment = 'x' %}{{ show() }}"),
+ ("top_level_set_reaches_a_block", "{% set assessment = 'x' %}{% block b %}{{ assessment.amount }}{% endblock %}"),
+ ("guard_reaches_a_block", "{% if assessment %}{% block b %}{{ assessment.amount }}{% endblock %}{% endif %}"),
+ (
+ "guard_around_a_loop_reaches_a_block_in_it",
+ "{% if assessment %}{% for i in items %}{% block b %}{{ assessment.amount }}{% endblock %}{% endfor %}{% endif %}",
+ ),
+ (
+ "guard_in_a_loop_reaches_a_block",
+ "{% for i in items %}{% if assessment %}{% block b %}{{ assessment.amount }}{% endblock %}{% endif %}{% endfor %}",
+ ),
+ (
+ "macro_in_an_if_branch_reads_a_later_set",
+ (
+ "{% if topic %}{% macro show() %}{{ assessment.amount }}{% endmacro %}{% else %}{% macro show() %}x{% endmacro %}"
+ "{% endif %}{% set assessment = 'x' %}{{ show() }}"
+ ),
+ ),
+ (
+ "scoped_block_sees_the_loop_target",
+ "{% for assessment in items %}{% block b scoped %}{{ assessment.amount }}{% endblock %}{% endfor %}",
+ ),
],
)
def test_guarded_references_produce_no_findings(self, topic: str, template_source: str):
@@ -64,6 +100,54 @@ class TestDetectUnguardedOptionalReferences:
("with_value_reads_optional", "{% with local = assessment %}{{ local }}{% endwith %}", "assessment"),
("with_body_reads_other_optional", "{% with other = 'x' %}{{ assessment.amount }}{% endwith %}", "assessment.amount"),
("for_filter_deep_access", "{% for item in topic_items if assessment.flag %}{{ item }}{% endfor %}", "assessment.flag"),
+ ("set_in_some_branches", "{% if topic %}{% set assessment = 'x' %}{% endif %}{{ assessment.amount }}", "assessment.amount"),
+ ("else_does_not_see_the_body_set", "{% if topic %}{% set assessment = 'x' %}{% else %}{{ assessment }}{% endif %}", "assessment"),
+ ("set_does_not_escape_a_call_block", "{% call m() %}{% set assessment = 'x' %}{% endcall %}{{ assessment }}", "assessment"),
+ ("set_does_not_escape_a_filter_block", "{% filter upper %}{% set assessment = 'x' %}{% endfilter %}{{ assessment }}", "assessment"),
+ ("set_does_not_escape_a_set_block", "{% set s %}{% set assessment = 'x' %}{% endset %}{{ assessment }}", "assessment"),
+ ("macro_default_reads_optional", "{% macro m(x=assessment.amount) %}{{ x }}{% endmacro %}{{ m() }}", "assessment.amount"),
+ (
+ "set_does_not_escape_an_autoescape_block",
+ "{% autoescape true %}{% set assessment = 'x' %}{% endautoescape %}{{ assessment }}",
+ "assessment",
+ ),
+ ("set_does_not_escape_a_block", "{% block b %}{% set assessment = 'x' %}{% endblock %}{{ assessment }}", "assessment"),
+ (
+ "macro_reads_a_name_set_in_some_branches",
+ "{% macro show() %}{{ assessment.amount }}{% endmacro %}{% if topic %}{% set assessment = 'x' %}{% endif %}{{ show() }}",
+ "assessment.amount",
+ ),
+ (
+ "macro_reads_an_optional_its_frame_reads_before_setting",
+ (
+ "{% if assessment %}{{ assessment }}{% endif %}"
+ "{% macro show() %}{{ assessment.amount }}{% endmacro %}{{ show() }}{% set assessment = 'x' %}"
+ ),
+ "assessment.amount",
+ ),
+ (
+ "loop_target_does_not_reach_a_block",
+ "{% for assessment in items %}{% block b %}{{ assessment.amount }}{% endblock %}{% endfor %}",
+ "assessment.amount",
+ ),
+ (
+ "guard_on_a_loop_target_does_not_reach_a_block",
+ "{% for assessment in items %}{% if assessment %}{% block b %}{{ assessment.amount }}{% endblock %}{% endif %}{% endfor %}",
+ "assessment.amount",
+ ),
+ (
+ "autoescape_set_does_not_reach_a_block",
+ "{% autoescape true %}{% set assessment = 'x' %}{% block b %}{{ assessment.amount }}{% endblock %}{% endautoescape %}",
+ "assessment.amount",
+ ),
+ (
+ "guard_on_a_block_local_does_not_reach_a_nested_block",
+ (
+ "{% block outer %}{% set assessment = 'x' %}{% if assessment %}"
+ "{% block inner %}{{ assessment.amount }}{% endblock %}{% endif %}{% endblock %}"
+ ),
+ "assessment.amount",
+ ),
],
)
def test_unguarded_references_are_reported(self, topic: str, template_source: str, expected_path: str):
diff --git a/tests/unit/pipelex/tools/test_jinja2_required_variables.py b/tests/unit/pipelex/tools/test_jinja2_required_variables.py
index dd9e2eb01..766b6c152 100644
--- a/tests/unit/pipelex/tools/test_jinja2_required_variables.py
+++ b/tests/unit/pipelex/tools/test_jinja2_required_variables.py
@@ -1,13 +1,43 @@
-from typing import ClassVar
+from collections.abc import Iterator
+from typing import Any, ClassVar
import pytest
+from jinja2 import Environment, UndefinedError
+from typing_extensions import override
from pipelex.tools.jinja2.exceptions import Jinja2DetectVariablesError
from pipelex.tools.jinja2.jinja2_required_variables import (
detect_jinja2_required_variables,
detect_jinja2_variable_references,
)
+from pipelex.tools.jinja2.jinja2_undefined import PresenceProbingStrictUndefined
from pipelex.tools.jinja2.template_category import TemplateCategory
+from pipelex.tools.misc.string_utils import get_root_from_dotted_path
+
+
+class _PermissiveValue:
+ """A context value any template shape can read: every attribute and call gives itself back."""
+
+ def __init__(self, *, is_truthy: bool) -> None:
+ self.is_truthy = is_truthy
+
+ def __getattr__(self, name: str) -> "_PermissiveValue":
+ if name.startswith("__"):
+ raise AttributeError(name)
+ return self
+
+ def __call__(self, *_args: Any, **_kwargs: Any) -> "_PermissiveValue":
+ return self
+
+ def __iter__(self) -> Iterator["_PermissiveValue"]:
+ return iter([self] if self.is_truthy else [])
+
+ def __bool__(self) -> bool:
+ return self.is_truthy
+
+ @override
+ def __str__(self) -> str:
+ return "value"
class TestData:
@@ -218,6 +248,144 @@ Best regards,
("macro_reads_a_later_set", "{{ x }}{% macro m() %}[{{ g }}]{% endmacro %}{% set g = 'y' %}{{ m() }}", {"x"}),
]
+ # Where Jinja binds a name a template sets. An `if` opens no frame, and after it a name every branch sets, the
+ # `else` included, is bound, while a name only some branches set is the input of that name on the paths where
+ # none ran. A loop body, a loop's `else`, a macro, a call, filter or set block, a `with` and `autoescape` each
+ # open a frame, so a set inside one stays inside it
+ JINJA_SCOPE_RULES: ClassVar[list[tuple[str, str, set[str]]]] = [
+ ("set_read_later_in_its_if_branch", "{% if x %}{% set y = 1 %}{{ y }}{% endif %}", {"x"}),
+ ("set_in_both_branches", "{% if x %}{% set y = 1 %}{% else %}{% set y = 2 %}{% endif %}{{ y }}", {"x"}),
+ (
+ "set_in_every_branch_with_elifs",
+ "{% if a %}{% set y = 1 %}{% elif b %}{% set y = 2 %}{% else %}{% set y = 3 %}{% endif %}{{ y }}",
+ {"a", "b"},
+ ),
+ ("set_in_some_branches", "{% if x %}{% set y = 1 %}{% endif %}{{ y }}", {"x", "y"}),
+ ("set_in_every_branch_but_no_else", "{% if a %}{% set y = 1 %}{% elif b %}{% set y = 2 %}{% endif %}{{ y }}", {"a", "b", "y"}),
+ ("set_missing_from_an_elif", "{% if a %}{% set y = 1 %}{% elif b %}-{% else %}{% set y = 3 %}{% endif %}{{ y }}", {"a", "b", "y"}),
+ ("else_does_not_see_the_body_set", "{% if x %}{% set y = 1 %}{% else %}{{ y }}{% endif %}", {"x", "y"}),
+ ("elif_test_does_not_see_the_body_set", "{% if x %}{% set y = 1 %}{% elif y %}-{% endif %}", {"x", "y"}),
+ (
+ "nested_ifs_set_in_every_branch",
+ "{% if a %}{% if b %}{% set y = 1 %}{% else %}{% set y = 2 %}{% endif %}{% else %}{% set y = 3 %}{% endif %}{{ y }}",
+ {"a", "b"},
+ ),
+ ("nested_if_without_else", "{% if a %}{% if b %}{% set y = 1 %}{% endif %}{% else %}{% set y = 3 %}{% endif %}{{ y }}", {"a", "b", "y"}),
+ ("set_before_the_if_and_in_a_branch", "{% set y = 1 %}{% if x %}{% set y = 2 %}{% endif %}{{ y }}", {"x"}),
+ ("read_before_set_in_a_branch", "{% if x %}{{ y }}{% set y = 1 %}{% endif %}", {"x", "y"}),
+ ("set_read_later_in_a_loop_body", "{% for i in items %}{% set day = i.d %}{{ day }}{% endfor %}", {"items"}),
+ ("read_before_set_in_a_loop_body", "{% for i in items %}[{{ y }}]{% set y = i %}{% endfor %}", {"items", "y"}),
+ ("set_in_some_branches_inside_a_loop", "{% for i in items %}{% if i %}{% set y = i %}{% endif %}{{ y }}{% endfor %}", {"items", "y"}),
+ ("set_in_a_loop_else", "{% for i in items %}-{% else %}{% set y = 1 %}{{ y }}{% endfor %}", {"items"}),
+ ("set_in_a_with_body", "{% with %}{% set y = 1 %}{{ y }}{% endwith %}", set()),
+ ("set_in_a_macro_body", "{% macro m() %}{% set y = 1 %}{{ y }}{% endmacro %}{{ m() }}", set()),
+ ("set_in_a_call_block_body", "{% macro m() %}{{ caller() }}{% endmacro %}{% call m() %}{% set y = 1 %}{{ y }}{% endcall %}", set()),
+ ("set_in_a_filter_block_body", "{% filter upper %}{% set y = 'a' %}{{ y }}{% endfilter %}", set()),
+ ("set_in_a_set_block_body", "{% set z %}{% set y = 'a' %}{{ y }}{% endset %}{{ z }}", set()),
+ ("set_in_an_autoescape_block", "{% autoescape true %}{% set y = 1 %}{{ y }}{% endautoescape %}", set()),
+ ("set_does_not_escape_a_loop", "{% for i in items %}{% set y = i %}{% endfor %}{{ y }}", {"items", "y"}),
+ ("set_does_not_escape_a_loop_else", "{% for i in items %}-{% else %}{% set y = 1 %}{% endfor %}{{ y }}", {"items", "y"}),
+ ("set_does_not_escape_a_with", "{% with %}{% set y = 1 %}{% endwith %}{{ y }}", {"y"}),
+ ("set_does_not_escape_a_macro", "{% macro m() %}{% set y = 1 %}{% endmacro %}{{ m() }}{{ y }}", {"y"}),
+ ("set_does_not_escape_a_call_block", "{% macro m() %}{{ caller() }}{% endmacro %}{% call m() %}{% set y = 1 %}{% endcall %}{{ y }}", {"y"}),
+ ("set_does_not_escape_a_filter_block", "{% filter upper %}{% set y = 'a' %}{% endfilter %}{{ y }}", {"y"}),
+ ("set_does_not_escape_a_set_block", "{% set z %}{% set y = 'a' %}{% endset %}{{ z }}{{ y }}", {"y"}),
+ ("set_does_not_escape_an_autoescape_block", "{% autoescape true %}{% set y = 1 %}{% endautoescape %}{{ y }}", {"y"}),
+ ("with_target", "{% with y = 1 %}{{ y }}{% endwith %}", set()),
+ ("with_value_reads_outside_the_with", "{% with a = 1, b = a %}{{ b }}{% endwith %}", {"a"}),
+ ("tuple_target", "{% set a, b = 1, 2 %}{{ a }}{{ b }}", set()),
+ ("nested_tuple_target", "{% set a, (b, c) = 1, (2, 3) %}{{ a }}{{ c }}", set()),
+ ("nested_tuple_loop_target", "{% for k, (a, b) in [(1, (2, 3))] %}{{ k }}{{ b }}{% endfor %}", set()),
+ (
+ "namespace_attribute_set_in_an_if",
+ "{% set ns = namespace(found=false) %}{% for i in items %}{% if i %}{% set ns.found = true %}{% endif %}{% endfor %}{{ ns.found }}",
+ {"items"},
+ ),
+ (
+ "macro_reads_a_name_every_branch_sets",
+ "{% if a %}{% set g = 1 %}{% else %}{% set g = 2 %}{% endif %}{% macro m() %}{{ g }}{% endmacro %}{{ m() }}",
+ {"a"},
+ ),
+ ("macro_reads_a_name_some_branches_set", "{% if a %}{% set g = 1 %}{% endif %}{% macro m() %}{{ g }}{% endmacro %}{{ m() }}", {"a", "g"}),
+ (
+ "macro_in_a_loop_reads_a_later_set",
+ "{% for i in items %}{% macro m() %}{{ y }}{% endmacro %}{% set y = i %}{{ m() }}{% endfor %}",
+ {"items"},
+ ),
+ ("macro_called_before_its_definition", "{{ m() }}{% macro m() %}-{% endmacro %}", {"m"}),
+ # A frame that reads a name before setting it starts its own copy from the input, which a macro called before
+ # the set then reads
+ ("macro_reads_a_name_its_frame_reads_before_setting", "{{ a }}{% macro m() %}{{ a.x }}{% endmacro %}{{ m() }}{% set a = 1 %}", {"a", "a.x"}),
+ (
+ "macro_in_a_loop_reads_a_name_the_loop_reads_before_setting",
+ "{% for i in items %}{% if a is defined %}-{% endif %}{% macro m() %}{{ a.x }}{% endmacro %}{{ m() }}{% set a = i %}{% endfor %}",
+ {"items", "a", "a.x"},
+ ),
+ # An `if` opens no frame, so a macro defined in one of its branches sees what its frame sets
+ (
+ "macro_in_an_if_branch_reads_a_later_set",
+ "{% if c %}{% macro m() %}{{ a }}{% endmacro %}{% else %}{% macro m() %}x{% endmacro %}{% endif %}{% set a = 1 %}{{ m() }}",
+ {"c"},
+ ),
+ (
+ "macro_in_an_if_branch_in_a_loop_reads_a_later_set",
+ (
+ "{% for i in items %}{% if c %}{% macro m() %}{{ a }}{% endmacro %}{% else %}{% macro m() %}x{% endmacro %}{% endif %}"
+ "{% set a = i %}{{ m() }}{% endfor %}"
+ ),
+ {"items", "c"},
+ ),
+ ("macro_default_reads_an_earlier_argument", "{% macro m(a, b=a) %}{{ b }}{% endmacro %}{{ m(1) }}", set()),
+ ("call_block_arguments", "{% macro m() %}{{ caller(1) }}{% endmacro %}{% call(a) m() %}{{ a }}{% endcall %}", set()),
+ ("filter_block_filter_reads_its_body_set", "{% filter replace('a', y) %}{% set y = 'b' %}a{% endfilter %}", set()),
+ # A block runs as a function of its own. Without `scoped` it reads the template context, which holds what the top
+ # level has set for certain before it and nothing of a loop, a macro or any other frame around it; a scoped
+ # block reads the names bound where it stands, and so do the blocks nested in it
+ ("set_in_a_block_body", "{% block b %}{% set y = 1 %}{{ y }}{% endblock %}", set()),
+ ("set_does_not_escape_a_block", "{% block b %}{% set y = 1 %}{% endblock %}{{ y }}", {"y"}),
+ ("top_level_set_reaches_a_block", "{% set y = 1 %}{% block b %}{{ y }}{% endblock %}", set()),
+ ("top_level_set_after_a_block", "{% block b %}{{ y }}{% endblock %}{% set y = 1 %}", {"y"}),
+ (
+ "set_in_every_top_level_branch_reaches_a_block",
+ "{% if x %}{% set y = 1 %}{% else %}{% set y = 2 %}{% endif %}{% block b %}{{ y }}{% endblock %}",
+ {"x"},
+ ),
+ ("set_in_some_top_level_branches_before_a_block", "{% if x %}{% set y = 1 %}{% endif %}{% block b %}{{ y }}{% endblock %}", {"x", "y"}),
+ ("set_earlier_in_a_top_level_branch_reaches_a_block", "{% if x %}{% set y = 1 %}{% block b %}{{ y }}{% endblock %}{% endif %}", {"x"}),
+ ("top_level_macro_reaches_a_block", "{% macro m() %}-{% endmacro %}{% block b %}{{ m() }}{% endblock %}", set()),
+ (
+ "loop_target_does_not_reach_a_block",
+ "{% for item in items %}{% block b %}{{ item.name }}{% endblock %}{% endfor %}",
+ {"items", "item.name"},
+ ),
+ ("loop_set_does_not_reach_a_block", "{% for i in items %}{% set y = i %}{% block b %}{{ y }}{% endblock %}{% endfor %}", {"items", "y"}),
+ (
+ "top_level_set_reaches_a_block_in_a_loop",
+ "{% set y = 1 %}{% for i in items %}{% set y = i %}{% block b %}{{ y }}{% endblock %}{% endfor %}",
+ {"items"},
+ ),
+ ("with_target_does_not_reach_a_block", "{% with y = 1 %}{% block b %}{{ y }}{% endblock %}{% endwith %}", {"y"}),
+ ("macro_argument_does_not_reach_a_block", "{% macro m(a) %}{% block b %}{{ a }}{% endblock %}{% endmacro %}{{ m(1) }}", {"a"}),
+ (
+ "call_block_argument_does_not_reach_a_block",
+ "{% macro m() %}{{ caller(1) }}{% endmacro %}{% call(y) m() %}{% block b %}{{ y }}{% endblock %}{% endcall %}",
+ {"y"},
+ ),
+ ("autoescape_set_does_not_reach_a_block", "{% autoescape true %}{% set y = 1 %}{% block b %}{{ y }}{% endblock %}{% endautoescape %}", {"y"}),
+ ("block_set_does_not_reach_a_nested_block", "{% block outer %}{% set z = 1 %}{% block inner %}{{ z }}{% endblock %}{% endblock %}", {"z"}),
+ ("scoped_block_sees_the_loop", "{% for item in items %}{% block b scoped %}{{ item.name }}{% endblock %}{% endfor %}", {"items"}),
+ (
+ "block_nested_in_a_scoped_block_sees_its_call_site",
+ "{% for item in items %}{% block outer scoped %}{% block inner %}{{ item.name }}{% endblock %}{% endblock %}{% endfor %}",
+ {"items"},
+ ),
+ (
+ "scoped_block_set_does_not_reach_a_nested_block",
+ "{% for i in items %}{% block outer scoped %}{% set z = i %}{% block inner %}{{ z }}{% endblock %}{% endblock %}{% endfor %}",
+ {"items", "z"},
+ ),
+ ]
+
# Names Jinja provides, and the scope a loop opens, are never inputs
SCOPES: ClassVar[list[tuple[str, str, set[str]]]] = [
("namespace_idiom", "{% set ns = namespace(found=false) %}{{ ns.found }}", set()),
@@ -627,6 +795,51 @@ class TestDetectJinja2Variables:
)
assert result == expected_variables, f"Failed for topic: {topic}"
+ @pytest.mark.parametrize(
+ ("topic", "template_source", "expected_variables"),
+ TestData.JINJA_SCOPE_RULES,
+ )
+ def test_jinja_scope_rules(
+ self,
+ topic: str,
+ template_source: str,
+ expected_variables: set[str],
+ ):
+ """A name the template sets is no input where Jinja binds it, and is the input of that name everywhere else."""
+ result = detect_jinja2_required_variables(
+ template_category=TemplateCategory.LLM_PROMPT,
+ template_source=template_source,
+ )
+ assert result == expected_variables, f"Failed for topic: {topic}"
+
+ @pytest.mark.parametrize(
+ ("topic", "template_source", "expected_variables"),
+ TestData.JINJA_SCOPE_RULES,
+ )
+ @pytest.mark.parametrize("is_truthy", [True, False])
+ def test_jinja_reads_nothing_beyond_the_required_variables(
+ self,
+ topic: str,
+ template_source: str,
+ expected_variables: set[str], # ruff: ignore[unused-method-argument]
+ is_truthy: bool,
+ ):
+ """Jinja itself renders each case with only the required roots in its context, so the walk never takes a read of the context for a local.
+
+ Every root is given a value that answers any attribute or call with itself, and that is truthy and iterates once,
+ or falsy and iterates not at all, so that both sides of every `if` and every loop's `else` run.
+ """
+ required = detect_jinja2_required_variables(
+ template_category=TemplateCategory.LLM_PROMPT,
+ template_source=template_source,
+ )
+ context = {get_root_from_dotted_path(path): _PermissiveValue(is_truthy=is_truthy) for path in required}
+ template = Environment(undefined=PresenceProbingStrictUndefined).from_string(template_source)
+ try:
+ template.render(context)
+ except UndefinedError as undefined_error:
+ pytest.fail(f"{topic}: Jinja read '{undefined_error}' from the context, but the walk reported only {sorted(required)}")
+
@pytest.mark.parametrize(
("topic", "template_source", "expected_variables"),
TestData.SCOPES,
@@ -807,7 +1020,7 @@ class TestDetectJinja2VariableReferences:
@pytest.mark.parametrize(
("topic", "template_source", "expected_paths"),
- TestData.REFERENCE_SCOPES,
+ TestData.REFERENCE_SCOPES + TestData.JINJA_SCOPE_RULES,
)
def test_scopes_match_required_variables(
self,