Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feature/Template-render-bounds · L-260929-7b11f8 (#1318)

Louis Choqueld4f69efc9c1f
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f8640c808..888ce4abc 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,6 +6,14 @@
- **The method-app template and the SDK references link `Pipelex/pipelex-sdk`**: the SDKs and the method-app templates now live in the `Pipelex/pipelex-sdk` repository, so the README and the documentation's **Quick Start** link the method-app template to its `method-apps/` directory, the webapp's by-hand commands clone it and copy `method-apps/webapp-js/`, and the `@pipelex/sdk` and `pipelex-sdk` references point at its `js/docs/` and `python/docs/` directories.
+### Fixed
+
+- **Markdown with unclosed emphasis is converted in memory proportional to its length**: the Markdown parser behind the `markdown` filter, the `Markdown` concept's HTML and the built-in PDF engine held memory growing with the square of the text when emphasis or strikethrough markers were left unmatched, about 1.5 GB for 64,000 characters of `**a `; it now joins the text they leave behind in one pass, and renders the same HTML.
+
+### Security
+
+- **Every template render spends from a budget (Breaking)**: a render is charged for what it reads, allocates and prints, and an operation that would overdraw its budget, such as `'x' * 10 ** 9`, a huge format width, `lipsum(n=10 ** 9)`, nested empty loops or a runaway recursive macro, is refused before it allocates and raises `Jinja2TemplateBudgetError`. Converting Markdown to HTML, with the `markdown` filter or by printing a Markdown value in an HTML template, is charged for every character and table cell of its source, so one render converts at most about 65,000 characters of Markdown. Template arithmetic stops at 64-bit integers, and a template can no longer call `str.encode` or `int.to_bytes`. A filter or test registered on a template environment needs a cost in the render budget's tables, or the first compile raises a `TypeError` naming it, and Pipelex now requires `jinja2` below 3.2.
+
## [v0.71.1] - 2026-10-01
### Fixed
diff --git a/docs/building-methods/pipes/pipe-operators/PipeCompose.md b/docs/building-methods/pipes/pipe-operators/PipeCompose.md
index a5ff723ce..4ccba455d 100644
--- a/docs/building-methods/pipes/pipe-operators/PipeCompose.md
+++ b/docs/building-methods/pipes/pipe-operators/PipeCompose.md
@@ -109,7 +109,7 @@ template = """
"""
```
-**The `markdown` filter.** An HTML template can turn Markdown held in a plain text field into HTML with `{{ order.notes | markdown }}`. Raw HTML inside the Markdown is shown as text rather than passed through. A bare URL becomes a link only when it has a scheme, such as `https://example.com`, so a file name like `README.md` stays text; a link written in Markdown, such as `[the guide](docs/guide.md)`, is a link whatever its target. A [`Markdown`](../../concepts/native-concepts.md) input needs no filter: it renders as HTML by itself in an HTML template.
+**The `markdown` filter.** An HTML template can turn Markdown held in a plain text field into HTML with `{{ order.notes | markdown }}`. Raw HTML inside the Markdown is shown as text rather than passed through. A bare URL becomes a link only when it has a scheme, such as `https://example.com`, so a file name like `README.md` stays text; a link written in Markdown, such as `[the guide](docs/guide.md)`, is a link whatever its target. A [`Markdown`](../../concepts/native-concepts.md) input needs no filter: it renders as HTML by itself in an HTML template. Converting Markdown is charged to the template's [render budget](../../../under-the-hood/template-sandbox.md#the-render-budget), so one render converts at most about 65,000 characters of it, each conversion counting.
## Construct Mode
diff --git a/docs/errors/jinja2-template-budget-error.md b/docs/errors/jinja2-template-budget-error.md
new file mode 100644
index 000000000..9af4506bd
--- /dev/null
+++ b/docs/errors/jinja2-template-budget-error.md
@@ -0,0 +1,19 @@
+---
+title: "Template render budget exceeded"
+description: "Reference for the `Jinja2TemplateBudgetError` Pipelex error class."
+---
+
+<!-- pipelex:generated -->
+
+# Template render budget exceeded
+
+| Field | Value |
+|---|---|
+| `error_type` | `Jinja2TemplateBudgetError` |
+| `title` | Template render budget exceeded |
+| `type_uri` | `https://docs.pipelex.com/latest/errors/jinja2-template-budget-error/` |
+| `error_domain` | `input` |
+| Defined in | `pipelex.tools.jinja2.exceptions` |
+| Parent class | [`ToolError`](tool-error.md) |
+
+[Back to Error Reference](index.md)
diff --git a/docs/errors/platform-and-tooling.md b/docs/errors/platform-and-tooling.md
index 38dff034a..c4f34a74a 100644
--- a/docs/errors/platform-and-tooling.md
+++ b/docs/errors/platform-and-tooling.md
@@ -33,6 +33,7 @@ own page. Classes are grouped by subsystem.
- [`Jinja2ContextError`](jinja2-context-error.md) — Jinja 2 context
- [`Jinja2DetectVariablesError`](jinja2-detect-variables-error.md) — Jinja 2 detect variables
- [`Jinja2StuffError`](jinja2-stuff-error.md) — Jinja 2 stuff
+- [`Jinja2TemplateBudgetError`](jinja2-template-budget-error.md) — Template render budget exceeded
- [`Jinja2TemplateRenderError`](jinja2-template-render-error.md) — Jinja 2 template render
- [`Jinja2TemplateSecurityError`](jinja2-template-security-error.md) — Jinja 2 template security
- [`Jinja2TemplateSyntaxError`](jinja2-template-syntax-error.md) — Jinja 2 template syntax
diff --git a/docs/under-the-hood/template-sandbox.md b/docs/under-the-hood/template-sandbox.md
index b67a4c20f..bddd55a1d 100644
--- a/docs/under-the-hood/template-sandbox.md
+++ b/docs/under-the-hood/template-sandbox.md
@@ -1,13 +1,13 @@
---
title: "Template Sandbox"
-description: "Every Jinja2 template Pipelex renders runs under one sandbox policy: a template reads data and calls methods of plain values, and nothing else."
+description: "Every Jinja2 template Pipelex renders runs under one sandbox policy: a template reads data and calls methods of plain values, and nothing else, within a budget of work per render."
---
# Template Sandbox
A method's templates are code. PipeLLM prompts, PipeCompose templates and construct templates, PipeImgGen and PipeSearch prompts, and PipeCondition expressions are all Jinja2 templates, and they render inside the process that runs the method. When that process is shared, as it is on a hosted runner, a template must not be able to reach anything but the data it was given.
-Pipelex therefore renders every template, its own included, under one policy: **a template reads data and calls methods of plain values, and nothing else.**
+Pipelex therefore renders every template, its own included, under one policy: **a template reads data and calls methods of plain values, and nothing else.** And because a template that only reads data can still ask for a billion-character string, every render also spends from a [budget of work](#the-render-budget), and a render that would overdraw it is refused before it allocates.
## What a template may do
@@ -17,7 +17,7 @@ Pipelex therefore renders every template, its own included, under one policy: **
| Read an input's metadata fields | `{{ invoice._stuff_name }}`, `{{ invoice._content_class }}`, `{{ invoice._concept_code }}`, `{{ invoice._stuff_code }}` |
| Read any key of a plain dict with brackets, underscore keys included | `{{ record['_id'] }}`, `{{ record['__typename'] }}` |
| Call the dict-like accessors of an input | `{{ invoice.get('total') }}`, `{% for key, value in invoice.iter_items() %}` |
-| Call methods of strings, numbers, dates and times, lists, tuples, dicts and sets | `{{ issued_at.isoformat() }}`, `{{ name.upper() }}`, `{{ record.get('a') }}` |
+| Call the methods of strings, numbers, dates and times, lists, tuples, dicts and sets that read them | `{{ issued_at.isoformat() }}`, `{{ name.upper() }}`, `{{ record.get('a') }}` |
| Format a string, including with replacement fields | `{{ '{0} items'.format(count) }}` |
| Use Jinja's own constructs and globals | macros and `caller()`, `loop.cycle()`, `namespace()`, `cycler()`, `joiner()`, `range()`, `dict()` |
| Apply any filter | `{{ doc \| tag }}`, `{{ doc \| format }}`, `{{ page \| with_images }}`, `{{ items \| length }}` |
@@ -31,6 +31,8 @@ Pipelex therefore renders every template, its own included, under one policy: **
| Call a method of anything that is not a plain value | `{{ pages[0].model_dump() }}`, `{{ photos[0].model_copy() }}` |
| Call a class method, even through an instance | `{{ issued_at.now() }}` |
| Change a list, a dict or a set | `{{ items.append(x) }}`, `{{ record.update(other) }}`, `{{ tags.intersection_update(other) }}` |
+| Encode a string, or turn an integer into bytes | `{{ name.encode('utf-8') }}`, `{{ (1).to_bytes(8) }}` |
+| Spend more than its render budget, or compute an integer wider than 64 bits | `{{ 'x' * 10 ** 9 }}`, `{{ 2 ** 70 }}` |
| Store a callable in a `namespace()` | `{% set ns.f = name.upper %}`, `namespace(items=doc.get)` |
| Include or extend another template | `{% include 'header.html' %}` |
@@ -42,6 +44,8 @@ Many refusals are visible in the template itself, and validation reports them be
What validation cannot see, because it depends on the values at run time (a method call, a name computed while rendering, or a bracketed key, which is legitimate on a plain dict), is refused where it happens: the render raises [`Jinja2TemplateSecurityError`](../errors/jinja2-template-security-error.md), whose message names the refused attribute or callable and the type it was reached on. A refusal never renders as an empty string, so a prompt is never sent with a hole in it.
+A render that would overdraw its budget raises [`Jinja2TemplateBudgetError`](../errors/jinja2-template-budget-error.md), in the input domain, whose message names the operation that was refused and what it would have spent, never the template or a value: for example, *the operator '\*' needs about 1,000,000,000, and 134,217,062 are left*. A template whose macros or includes nest deeper than Python allows raises the same error.
+
## How it is built
Every environment comes from one factory, `make_jinja2_env_from_loader` in `pipelex/tools/jinja2/jinja2_environment.py`, and that factory builds a `PipelexTemplateEnvironment` (`pipelex/tools/jinja2/jinja2_sandbox.py`), a subclass of Jinja's `ImmutableSandboxedEnvironment`. Rendering, parsing and compiling all go through it, including the condition pipe's compile at load and the load-time template walkers.
@@ -49,7 +53,7 @@ Every environment comes from one factory, `make_jinja2_env_from_loader` in `pipe
Jinja's stock sandbox refuses internals, but not public methods, and every pydantic model has public constructors. Under the stock sandbox a template could call `model_validate` on a content object and forge an image pointing at a storage key of its choosing. The Pipelex policy closes that by allowing calls by kind rather than by name:
- **Jinja's own runtime**: macros, `caller`, `loop`, `cycler`, `joiner`, block references, and the environment's globals.
-- **Methods a plain value type defines**, bound to an instance of that type. A subclass of a plain type (a `StrEnum` member is a `str`) cannot add or override a callable method, and a class method is refused because it is bound to the class. A method that changes a list, a dict or a set in place is refused by a list Pipelex keeps itself, since Jinja's own misses `set.intersection_update`.
+- **Methods a plain value type defines**, bound to an instance of that type. A subclass of a plain type (a `StrEnum` member is a `str`) cannot add or override a callable method, and a class method is refused because it is bound to the class. The methods allowed are an audited list, `PLAIN_VALUE_METHOD_COSTS` in `pipelex/tools/jinja2/jinja2_render_costs.py`, which also gives each method's cost: a method that changes a list, a dict or a set in place, `str.encode` and `int.to_bytes` are left out, each with its reason in `REFUSED_PLAIN_VALUE_METHODS`, and a method a new Python release adds to a plain type is refused until it is classified.
- **The sandbox's own `str.format` wrapper**, which resolves replacement fields through the same attribute and item checks. The raw `str.format` stays refused.
- **The methods a type declares as its template surface** (`pipelex/tools/jinja2/template_surface.py`).
@@ -61,16 +65,36 @@ Filters are not calls in this sense: they are Pipelex's or Jinja's own code, reg
Jinja's own filters and markupsafe keep that obligation only for values whose attributes the template cannot choose: they look `__html__` up on a value when escaping, and `items` in `dictsort` and `xmlattr`, and call what they find. A `namespace()` answers any name with whatever the template stored under it, so the environment's `namespace` holds data only, and storing a callable in one, at construction or with `{% set ns.x = … %}`, is refused.
+## The render budget
+
+The sandbox limits what a template can reach, not what it can spend, and a template needs no forbidden call to be expensive: `{{ 'x' * 10 ** 9 }}`, a width in `'%1000000000s' % x`, the `center`, `batch` or `format` filters, a string doubled with `~` in a loop, two empty nested loops and a recursive macro each spend what the template asks. Rendering runs in the process that runs the method, on its event loop and sometimes inside a Temporal workflow, so on a shared runner one method's template would stall or exhaust a process that serves others. A timeout cannot help: a render never yields to the event loop, so nothing can interrupt it.
+
+So every render gets a budget of work units, roughly one byte read or allocated each, and every operation spends from it. The default, `DEFAULT_RENDER_BUDGET_UNITS` in `pipelex/tools/jinja2/jinja2_render_budget.py`, is 128 Mi units: it prints about a hundred megabytes of text, and in the measurements behind it the slowest refusal of a hostile template took about half a second. A template included by another spends from its includer's budget, and a macro from its caller's, unless it was imported without `with context` (see below).
+
+Each operation is charged for its inputs before it runs and for its result after. An operation whose result can outgrow its inputs, such as `*`, `**`, `%`, a format width, `center`, `ljust`, `replace`, `join`, `batch`, `wordwrap` or `lipsum`, is estimated first and refused before it allocates anything. An operation that reads a fixed part of its value, such as `length`, `first`, `default` or `dict.get`, is charged a step and its arguments, whatever the size of the value. An object of the run's data, such as a text content, is converted to count its text when an estimate needs it, so repeating it (`([doc] * 20000) | join`) is refused like repeating a string. `str.format` charges each replacement field as it formats it, with its spec resolved, so a width passed as an argument or through `format_map` counts. A filter that orders or deduplicates its input, `sort`, `dictsort`, `groupby`, `min`, `max` or `unique`, is charged for its comparisons, each weighing the heaviest key, read the way the filter reads it, and so are a list's or a tuple's `count` and `index`. Work that grows faster than an operation's result, such as `striptags`, `wordwrap` on long words or `sum` over lists, is spent at every call, not only checked. Integer arithmetic stops at 64 bits, which also keeps a template from building the huge integers whose hashing and printing cost the square of their size. Jinja's own cap on `range` still applies: a range of more than a hundred thousand items fails the render with `Jinja2TemplateRenderError`.
+
+The environment charges through the sandbox's hooks and one rewrite, all in `PipelexTemplateEnvironment`. They rely on internals of Jinja 3.1, which is why Pipelex requires a Jinja below 3.2.
+
+- **Calls and operators** go through the sandbox's `call` and `call_binop` hooks, and every arithmetic operator is intercepted.
+- **Filters and tests** are wrapped before a template compiles. A filter or a test registered on the environment must have a cost in `FILTER_COSTS` or `TEST_COSTS` (`pipelex/tools/jinja2/jinja2_render_costs.py`), or the first compile fails with a `TypeError` naming it.
+- **Printed values** go through the environment's `finalize`, which charges the text it produces. A caller's own `finalize` is passed to the constructor and runs first.
+- **What no hook reaches**, which is loop iteration, `~`, comparisons, slices, list, tuple and dict literals and the template's static text, is routed through internal charging filters by a rewrite of the parsed template (`pipelex/tools/jinja2/jinja2_render_rewrite.py`). The rewrite changes what is charged, never what a template renders. A tuple used as a key, in a dict literal or an item read, is also charged for hashing, since a tuple hashes every element again each time.
+- **Converting Markdown to HTML** is charged where it happens, in `pipelex/tools/markdown/markdown_parser.py`, since markupsafe converts a Markdown value through its `__html__` wherever an HTML template prints, escapes, joins or formats one, and no hook sees that call. The template makes its budget the active one while it renders, and the conversion charges it 2,048 units for every character of its source and every cell of its tables before it parses, then checks that its output, bounded from the parsed tokens, fits what is left before it renders, since a reference link is printed at every use. A conversion costs microseconds a character, so a render converts at most about 65,000 characters of Markdown, a document printed twice counting twice.
+
+The budget is the same for every template category and is not a setting in `pipelex.toml`: it is a safety bound, not a tuning knob. `PipelexTemplateEnvironment` takes it as the `render_budget_units` constructor argument, which the tests use to exercise small budgets.
+
## Paths outside templates
A method also writes dotted paths that are not templates: a PipeCompose construct field's `from` and its `list_to_dict_keyed_by`, a PipeBatch's list, and the image and document references of a prompt. The runtime walks these with `getattr`, so the same rule applies to them: a segment starting with an underscore is refused. A construct's `from` path and `list_to_dict_keyed_by` are refused when the bundle is validated, and every path is refused again when it is walked.
## What the sandbox does not cover
-- **Resource use.** The sandbox limits what a template can reach, not what it can spend. Jinja caps `range`, and a longer range fails the render with `Jinja2TemplateRenderError`, but nothing caps exponentiation, string repetition, nested loops or output size.
+- **What a whole run spends.** The budget is per render. A method that renders many templates, or a batch that renders one per item, spends one budget per render.
+- **Macros imported without context.** A macro imported with `{% import %}` or `{% from … import %}` and no `with context` spends from the budget of the module Jinja builds for the import, which Jinja caches on the imported template. Pipelex builds a new environment for every render, so that budget never outlives one render, and a customer template has no loader to import from.
- **The URLs values carry.** A template cannot forge content under this policy, but a value can still carry a URL, and reading it is the storage layer's decision, not the template's.
## Related
- [StuffArtefact & Image Rendering](stuffartefact-and-image-rendering.md): how inputs reach templates, and how images are extracted from them.
- [`Jinja2TemplateSecurityError`](../errors/jinja2-template-security-error.md): the error a refusal raises at render time.
+- [`Jinja2TemplateBudgetError`](../errors/jinja2-template-budget-error.md): the error a render raises when it would overdraw its budget.
diff --git a/pipelex/tools/jinja2/exceptions.py b/pipelex/tools/jinja2/exceptions.py
index 7e982d579..331a1d2f6 100644
--- a/pipelex/tools/jinja2/exceptions.py
+++ b/pipelex/tools/jinja2/exceptions.py
@@ -1,3 +1,4 @@
+from pipelex.base_exceptions import ErrorDomain
from pipelex.system.exceptions import ToolError
@@ -24,6 +25,17 @@ class Jinja2TemplateSecurityError(ToolError):
pass
+class Jinja2TemplateBudgetError(ToolError):
+ # A render tried to spend more than its budget (`jinja2_render_budget.py`): its template repeated,
+ # padded, joined, looped or recursed past what one render may allocate or compute. The template is
+ # the caller's own, and a Pipelex template can only get there on data too large to display, which
+ # is still the caller's input, so it is the caller's to fix. The message names the operation and
+ # its size, never the template source or a value.
+ error_domain = ErrorDomain.INPUT
+ _authors_caller_facing_message = True
+ _declared_title = "Template render budget exceeded"
+
+
class Jinja2StuffError(ToolError):
pass
diff --git a/pipelex/tools/jinja2/jinja2_environment.py b/pipelex/tools/jinja2/jinja2_environment.py
index 7dbdb1ee8..d4b0f6d77 100644
--- a/pipelex/tools/jinja2/jinja2_environment.py
+++ b/pipelex/tools/jinja2/jinja2_environment.py
@@ -1,6 +1,8 @@
import inspect
+from collections.abc import Callable
+from typing import Any
-from jinja2 import BaseLoader
+from jinja2 import BaseLoader, Undefined
from pipelex.tools.jinja2.jinja2_sandbox import PipelexTemplateEnvironment
from pipelex.tools.jinja2.jinja2_template_registry import TemplateRegistry
@@ -13,10 +15,14 @@ def make_jinja2_env_from_loader(
template_category: TemplateCategory,
loader: BaseLoader,
enable_async: bool = True,
+ finalize: Callable[..., Any] | None = None,
is_undefined_strict: bool = False,
) -> PipelexTemplateEnvironment:
"""Build the environment for one template category. Every template renders sandboxed (`jinja2_sandbox.py`).
+ A `finalize` is handed to the environment here, which wraps it to charge what is printed to the
+ render's budget.
+
With `is_undefined_strict`, a missing value fails the render instead of printing as empty text
(`jinja2_undefined.py`); `PipeDocGen` asks for it, and every other template keeps Jinja's lenient default.
"""
@@ -53,21 +59,15 @@ def make_jinja2_env_from_loader(
trim_blocks = False
lstrip_blocks = False
- if is_undefined_strict:
- return PipelexTemplateEnvironment(
- loader=loader,
- enable_async=enable_async,
- autoescape=autoescape,
- trim_blocks=trim_blocks,
- lstrip_blocks=lstrip_blocks,
- undefined=PresenceProbingStrictUndefined,
- )
+ undefined: type[Undefined] = PresenceProbingStrictUndefined if is_undefined_strict else Undefined
return PipelexTemplateEnvironment(
loader=loader,
enable_async=enable_async,
autoescape=autoescape,
trim_blocks=trim_blocks,
lstrip_blocks=lstrip_blocks,
+ finalize=finalize,
+ undefined=undefined,
)
@@ -99,6 +99,7 @@ def make_jinja2_env_without_loader(
template_category: TemplateCategory,
*,
enable_async: bool = True,
+ finalize: Callable[..., Any] | None = None,
is_undefined_strict: bool = False,
) -> PipelexTemplateEnvironment:
loader = BaseLoader()
@@ -106,6 +107,7 @@ def make_jinja2_env_without_loader(
template_category=template_category,
loader=loader,
enable_async=enable_async,
+ finalize=finalize,
is_undefined_strict=is_undefined_strict,
)
@@ -117,6 +119,7 @@ def make_jinja2_env_from_registry(
template_category: TemplateCategory,
*,
enable_async: bool = True,
+ finalize: Callable[..., Any] | None = None,
is_undefined_strict: bool = False,
) -> PipelexTemplateEnvironment:
"""Create Environment with DictLoader from pre-loaded registry.
@@ -128,6 +131,7 @@ def make_jinja2_env_from_registry(
Args:
template_category: The category of templates being rendered.
enable_async: Whether to enable async mode for the environment.
+ finalize: The caller's own finalize, if any, applied to every printed value.
is_undefined_strict: Whether a missing value fails the render instead of printing as empty text.
Returns:
@@ -138,6 +142,7 @@ def make_jinja2_env_from_registry(
template_category=template_category,
loader=loader,
enable_async=enable_async,
+ finalize=finalize,
is_undefined_strict=is_undefined_strict,
)
diff --git a/pipelex/tools/jinja2/jinja2_render_budget.py b/pipelex/tools/jinja2/jinja2_render_budget.py
new file mode 100644
index 000000000..d18b604de
--- /dev/null
+++ b/pipelex/tools/jinja2/jinja2_render_budget.py
@@ -0,0 +1,523 @@
+"""The budget a template render spends from, and the sizes it is charged in.
+
+A sandbox limits what a template can reach, not what it can spend, and a template renders inside the
+shared runner and worker processes, on their event loop, where no timeout can interrupt it: a render
+never yields. So every render spends from one budget of **work units**, where a unit stands for roughly
+one byte read or allocated, and the render stops with `RenderBudgetExceededError` the moment an
+operation would overdraw it. One number bounds the CPU a render takes, since every step of work is
+charged, the memory it takes, since every allocation is charged, and its output, which is allocated
+like anything else.
+
+An operation is charged for its inputs before it runs, for its result after it runs, and, when its
+result can be much larger than its inputs, for an estimate of that result before it runs:
+
+- Charging the inputs first means an operation never starts on inputs larger than what is left, so an
+ operation whose result is a small multiple of its inputs overshoots by at most that multiple of what
+ was left.
+- Charging the result afterwards makes repeated steps add up: a string doubled in a loop is charged at
+ every doubling.
+- Estimating the result first stops a single step from allocating a gigabyte: `'x' * 10**9` is refused
+ before it runs.
+
+Where the charges are made, and the estimates, are in `jinja2_render_charging.py` and
+`jinja2_render_costs.py`; the environment that applies them is `PipelexTemplateEnvironment`.
+
+A render's budget is also its **active** budget while the render runs (`active_render_budget`), for the
+work a template sets off in code that no hook reaches and that costs far more than the bytes it
+produces: converting a Markdown value to HTML, which markupsafe does through `__html__` wherever a
+value is escaped, is charged where it happens (`markdown_parser.py`).
+"""
+
+from __future__ import annotations
+
+import sys
+from collections.abc import AsyncIterator, Iterator, Sized
+from contextlib import contextmanager
+from contextvars import ContextVar
+from typing import TYPE_CHECKING, Any, Final, Generic, TypeVar, cast
+
+from jinja2.exceptions import TemplateError
+from jinja2.runtime import LoopContext, Undefined
+from jinja2.utils import Namespace
+from markupsafe import Markup
+from pydantic import BaseModel
+
+if TYPE_CHECKING:
+ from collections.abc import AsyncIterable, Callable, Collection, Generator, Iterable
+
+# A render's budget unless the environment is given another: a few hundred megabytes allocated in
+# total at the very worst, and well under a second of CPU.
+DEFAULT_RENDER_BUDGET_UNITS: Final[int] = 128 * 1024 * 1024
+
+# The flat cost of one step: a call, an operator, a filter, a test, a comparison, a printed value. It
+# is what an empty loop body or a macro that renders nothing still costs, and it is set so that a step
+# costs about as many units as the nanoseconds it takes, like the bytes the other charges count.
+STEP_UNITS: Final[int] = 256
+
+# The flat cost of a call a template makes, a method's or a macro's, which takes several steps' time.
+CALL_UNITS: Final[int] = 1024
+
+# The cost of one element a template draws from an iterable or finds in a container it produced: a
+# reference, plus the small object behind it when the element is new, and the time a loop takes to
+# draw it.
+ELEMENT_UNITS: Final[int] = 64
+
+# The cost of converting one character of Markdown to HTML, or one cell of a table, which a render pays
+# wherever a template sets the conversion off (`markdown_parser.py`): the `markdown` filter, or printing,
+# joining or formatting a Markdown value in an HTML template. markdown-it-py takes up to about ten
+# microseconds and five hundred bytes of tokens a character, and five microseconds and a kilobyte a table
+# cell, so the charge stops a render at about 65,000 characters of Markdown under the default budget,
+# which the slowest text measured converts in about six tenths of a second.
+MARKDOWN_UNITS_PER_CHARACTER: Final[int] = 2048
+
+# The factor by which escaping can grow a text: `'` becomes `&#39;` in HTML and `'` in JSON.
+ESCAPE_FACTOR: Final[int] = 6
+
+# The widest integer template arithmetic may produce. Multiplying huge integers costs more than their
+# size suggests, and integers past 2**61 can be chosen to share a hash, which makes a set or a dict of
+# them quadratic to build; 64 bits allows neither.
+MAX_INT_BITS: Final[int] = 64
+
+# The types whose storage `sys.getsizeof` measures in constant time; every other object counts a step.
+_SIZED_TYPES: Final[tuple[type, ...]] = (str, bytes, bytearray, int, float, list, tuple, dict, set, frozenset)
+
+# The types whose own storage is a few bytes whatever their length: a range, and the key, value and item
+# views of a dict. A filter iterating one does as much work as on the list it stands for, so it is
+# sized like that list, a pointer per element.
+_EMPTY_DICT: Final[dict[object, object]] = {}
+_VIEW_TYPES: Final[tuple[type, ...]] = (range, type(_EMPTY_DICT.keys()), type(_EMPTY_DICT.values()), type(_EMPTY_DICT.items()))
+_POINTER_UNITS: Final = 8
+
+# The exact types that are never lazy and never awaitable, checked first on the hot paths.
+EAGER_TYPES: Final[frozenset[type]] = frozenset({str, Markup, int, float, bool, type(None), list, tuple, dict, set, frozenset, range})
+
+# The key, value and item views of a dict, which print every element they show.
+_DICT_VIEW_TYPES: Final[tuple[type, ...]] = (type(_EMPTY_DICT.keys()), type(_EMPTY_DICT.values()), type(_EMPTY_DICT.items()))
+
+# The containers a template can build, fill or read a view of, whose elements a walk visits.
+_CONTAINER_TYPES: Final[tuple[type, ...]] = (list, tuple, dict, set, frozenset, Namespace, *_DICT_VIEW_TYPES)
+
+_ElementT = TypeVar("_ElementT")
+
+
+class RenderBudgetExceededError(TemplateError):
+ """A template render tried to spend more than its budget allows.
+
+ Raised from the hook that caught the overdraft, and turned into `Jinja2TemplateBudgetError` by the
+ render functions. It derives from nothing a filter or Jinja's runtime catches on the way.
+ """
+
+
+class RenderBudget:
+ """What is left of one render's budget. Every render gets its own, created with its context."""
+
+ __slots__ = ("remaining", "total")
+
+ def __init__(self, *, total: int) -> None:
+ self.total = total
+ self.remaining = total
+
+ def charge(self, *, units: int, operation: str | Callable[[], str]) -> None:
+ """Spend `units`, or refuse the operation if they are more than what is left."""
+ if units > self.remaining:
+ self._refuse(units=units, operation=operation)
+ self.remaining -= units
+
+ def afford(self, *, units: int, operation: str | Callable[[], str]) -> None:
+ """Refuse the operation if its estimate is more than what is left, without spending anything."""
+ if units > self.remaining:
+ self._refuse(units=units, operation=operation)
+
+ def _refuse(self, *, units: int, operation: str | Callable[[], str]) -> None:
+ described = operation() if callable(operation) else operation
+ msg = (
+ f"Rendering the template would spend more than its budget of {self.total:,} work units: "
+ f"{described} needs about {units:,}, and {self.remaining:,} are left."
+ )
+ raise RenderBudgetExceededError(msg)
+
+
+_ACTIVE_RENDER_BUDGET: Final[ContextVar[RenderBudget | None]] = ContextVar("pipelex_active_render_budget", default=None)
+
+
+def active_render_budget() -> RenderBudget | None:
+ """The budget of the template render running now, or None outside any render."""
+ return _ACTIVE_RENDER_BUDGET.get()
+
+
+@contextmanager
+def spending_from(*, budget: RenderBudget) -> Generator[None, None, None]:
+ """Make `budget` the active one for the duration of a render, and the previous one again after it."""
+ token = _ACTIVE_RENDER_BUDGET.set(budget)
+ try:
+ yield
+ finally:
+ _ACTIVE_RENDER_BUDGET.reset(token)
+
+
+def refuse_int_result(*, bits: int, operation: str) -> None:
+ """Refuse an integer result wider than `MAX_INT_BITS`."""
+ if bits > MAX_INT_BITS:
+ msg = (
+ f"Rendering the template would produce an integer of about {bits:,} bits by {operation}; "
+ f"template arithmetic stops at {MAX_INT_BITS} bits."
+ )
+ raise RenderBudgetExceededError(msg)
+
+
+def check_int_result(*, value: object, operation: str) -> None:
+ """Refuse `value` if it is an integer wider than `MAX_INT_BITS`."""
+ if isinstance(value, int):
+ refuse_int_result(bits=value.bit_length(), operation=operation)
+
+
+def size_of(value: Any) -> int:
+ """The bytes `value`'s own storage takes, taken in constant time; a flat cost for any other object."""
+ if isinstance(value, _SIZED_TYPES):
+ return sys.getsizeof(cast("object", value))
+ if isinstance(value, _VIEW_TYPES):
+ return _view_size(value=cast("Sized", value))
+ return ELEMENT_UNITS
+
+
+def _view_size(*, value: Sized) -> int:
+ try:
+ length = len(value)
+ except OverflowError:
+ # A range longer than a machine integer can count.
+ return sys.maxsize
+ return sys.getsizeof(value) + _POINTER_UNITS * length
+
+
+def produced_size(value: Any) -> int:
+ """What producing `value` cost: its own storage, plus a flat cost per element of a container."""
+ size = size_of(value)
+ if isinstance(value, (list, tuple, dict, set, frozenset)):
+ size += ELEMENT_UNITS * len(cast("Sized", value))
+ return size
+
+
+def is_lazy(value: object) -> bool:
+ """Whether `value` produces its elements as it is iterated: an iterator, a generator, an async one.
+
+ An undefined value and a loop's `loop` are neither, though Jinja gives the first an `__aiter__` and
+ the second a `__next__`: wrapped, they would stop behaving as what they are.
+ """
+ if type(value) in EAGER_TYPES or isinstance(value, (Undefined, LoopContext)):
+ return False
+ return hasattr(value, "__aiter__") or (hasattr(value, "__next__") and hasattr(value, "__iter__"))
+
+
+class ChargedIterator(Generic[_ElementT]):
+ """An iterable that charges the render's budget for every element drawn from it.
+
+ An element that already existed costs `ELEMENT_UNITS`; an element a lazy iterable produced on the
+ way also costs its size.
+ """
+
+ __slots__ = ("_budget", "_charges_elements", "_iterator")
+
+ def __init__(self, *, iterable: Iterable[_ElementT], budget: RenderBudget) -> None:
+ self._budget = budget
+ self._charges_elements = is_lazy(iterable)
+ self._iterator = iter(iterable)
+
+ def __iter__(self) -> ChargedIterator[_ElementT]:
+ return self
+
+ def __next__(self) -> _ElementT:
+ element = next(self._iterator)
+ budget = self._budget
+ units = ELEMENT_UNITS + produced_size(element) if self._charges_elements else ELEMENT_UNITS
+ if units > budget.remaining:
+ budget.charge(units=units, operation="drawing the next element of an iteration")
+ budget.remaining -= units
+ return element
+
+
+class ChargedSizedIterator(ChargedIterator[_ElementT]):
+ """A `ChargedIterator` over a sized iterable, whose length stays readable, so `loop.length` on a
+ list does not have to exhaust it.
+ """
+
+ __slots__ = ("_length",)
+
+ def __init__(self, *, iterable: Iterable[_ElementT], budget: RenderBudget) -> None:
+ super().__init__(iterable=iterable, budget=budget)
+ self._length = len(cast("Sized", iterable))
+
+ def __len__(self) -> int:
+ return self._length
+
+
+class ChargedAsyncIterator(Generic[_ElementT]):
+ """The async counterpart of `ChargedIterator`, for the async iterables Jinja's async filters return."""
+
+ __slots__ = ("_budget", "_iterator")
+
+ def __init__(self, *, iterable: AsyncIterable[_ElementT], budget: RenderBudget) -> None:
+ self._budget = budget
+ self._iterator: AsyncIterator[_ElementT] = aiter(iterable)
+
+ def __aiter__(self) -> ChargedAsyncIterator[_ElementT]:
+ return self
+
+ async def __anext__(self) -> _ElementT:
+ element = await anext(self._iterator)
+ self._budget.charge(units=ELEMENT_UNITS + produced_size(element), operation="drawing the next element of an iteration")
+ return element
+
+
+def charged_iterable(*, iterable: Any, budget: RenderBudget) -> ChargedIterator[Any] | ChargedAsyncIterator[Any] | Undefined:
+ """Wrap `iterable` so that drawing from it is charged, keeping it async when it is async.
+
+ An undefined value is handed back as it is: it iterates as empty, or fails as Jinja would have it fail.
+ """
+ if isinstance(iterable, Undefined):
+ return iterable
+ if hasattr(iterable, "__aiter__"):
+ return ChargedAsyncIterator(iterable=iterable, budget=budget)
+ if isinstance(iterable, Sized):
+ return ChargedSizedIterator(iterable=cast("Iterable[Any]", iterable), budget=budget)
+ return ChargedIterator(iterable=iterable, budget=budget)
+
+
+_CHARGED_TYPES: Final = frozenset({ChargedIterator, ChargedSizedIterator, ChargedAsyncIterator})
+
+
+def charged_if_lazy(*, value: Any, budget: RenderBudget) -> Any:
+ """Wrap `value` in a charged iterable when it is lazy; hand anything else back as it is."""
+ if type(value) in EAGER_TYPES or type(value) in _CHARGED_TYPES or not is_lazy(value):
+ return value
+ return charged_iterable(iterable=value, budget=budget)
+
+
+def _children(*, value: Any) -> Iterator[object] | None:
+ """The elements a walk visits under `value`, or None when `value` is a leaf."""
+ if isinstance(value, dict):
+ return _dict_children(value=cast("dict[Any, Any]", value))
+ if isinstance(value, (list, tuple, set, frozenset)):
+ return iter(cast("Collection[Any]", value))
+ if isinstance(value, _DICT_VIEW_TYPES):
+ # An item view's pairs are temporaries a walk could not remember by identity: walk its dict.
+ if isinstance(value, type(_EMPTY_DICT.items())):
+ return _dict_children(value=cast("dict[Any, Any]", cast("Any", value).mapping))
+ return iter(cast("Collection[Any]", value))
+ if isinstance(value, Namespace):
+ # A namespace prints as `<Namespace {...}>`, the dict of what the template stored in it.
+ # A namespace answers any name with what was stored under it, except its own store.
+ attrs: dict[str, Any] = getattr(value, "_Namespace__attrs", {})
+ return iter((attrs,))
+ return None
+
+
+def _dict_children(*, value: dict[Any, Any]) -> Iterator[object]:
+ # Keys and values separately, never `items()`, whose tuples are temporaries a walk could not
+ # remember by identity.
+ yield from value.keys()
+ yield from value.values()
+
+
+def _walk(
+ *,
+ root: object,
+ leaf_units: Callable[[object], int],
+ container_units: Callable[[object], int],
+ limit: int,
+ children: Callable[..., Iterator[object] | None] = _children,
+) -> int:
+ """Add up a value's units over the containers under it, without recursing and stopping past `limit`.
+
+ Each container is totalled once and its total counted at every reference to it, so a list holding
+ the same big string a hundred thousand times costs a hundred thousand times the string, while the
+ walk visits the string once. A container reached again while it is being totalled is a cycle, and
+ counts a step.
+ """
+ totals: dict[int, int] = {}
+ in_progress: set[int] = set()
+ root_children = children(value=root)
+ if root_children is None:
+ return leaf_units(root)
+ # Each frame is the container, the elements left to visit, and the running total.
+ frames: list[list[Any]] = [[root, root_children, container_units(root)]]
+ in_progress.add(id(root))
+ while frames:
+ frame = frames[-1]
+ child = next(frame[1], _END)
+ if child is _END:
+ frames.pop()
+ finished: object = frame[0]
+ finished_total: int = frame[2]
+ in_progress.discard(id(finished))
+ totals[id(finished)] = finished_total
+ if not frames:
+ return finished_total
+ parent_total: int = frames[-1][2] + finished_total
+ frames[-1][2] = parent_total
+ if parent_total > limit:
+ return parent_total
+ continue
+ child_id = id(child)
+ known = totals.get(child_id)
+ if known is not None:
+ frame[2] += known
+ elif child_id in in_progress:
+ frame[2] += ELEMENT_UNITS
+ else:
+ grandchildren = children(value=child)
+ if grandchildren is None:
+ units = leaf_units(child)
+ totals[child_id] = units
+ frame[2] += units
+ else:
+ in_progress.add(child_id)
+ frames.append([child, grandchildren, container_units(child)])
+ continue
+ running_total: int = frame[2]
+ if running_total > limit:
+ return running_total
+ return 0
+
+
+_END: Final = object()
+
+
+def _int_text_size(*, value: int) -> int:
+ # A decimal digit carries about 3.32 bits.
+ return value.bit_length() * 10 // 33 + 2
+
+
+def _repr_leaf_size(value: Any) -> int:
+ """The length of `repr(value)` for an element inside a printed container."""
+ if isinstance(value, Markup):
+ return len(value) + 10
+ if isinstance(value, str):
+ return len(value) + 2
+ if isinstance(value, bool) or value is None:
+ return 5
+ if isinstance(value, int):
+ return _int_text_size(value=value)
+ if isinstance(value, float):
+ return 24
+ if isinstance(value, Undefined):
+ return 16
+ return len(repr(value))
+
+
+def _repr_container_size(value: Any) -> int:
+ """What a container's printed form adds to its elements: brackets, separators, a class name."""
+ kind = type(cast("object", value))
+ if isinstance(value, Namespace):
+ return 13
+ if isinstance(value, _DICT_VIEW_TYPES):
+ # `dict_items([('k', 'v')])`: the view's name and brackets, and a pair's parentheses and separators.
+ return 16 + 6 * len(cast("Sized", value))
+ if not isinstance(value, (list, tuple, dict, set, frozenset)):
+ return 0
+ length = len(cast("Sized", value))
+ if isinstance(value, dict):
+ return 2 + 4 * length
+ size = 11 + 2 * length
+ if kind not in {list, tuple, set, frozenset}:
+ # A named tuple prints its class and field names.
+ size += len(kind.__name__) + 16 * length
+ return size
+
+
+def escaped_length(text: str) -> int:
+ """The exact length of `markupsafe.escape(text)`: `&`, `"` and `'` grow by four, `<` and `>` by three."""
+ return len(text) + 4 * (text.count("&") + text.count('"') + text.count("'")) + 3 * (text.count("<") + text.count(">"))
+
+
+_CONTROL_CHARACTERS: Final[dict[int, None]] = dict.fromkeys(range(32))
+
+
+def json_string_length(text: str) -> int:
+ r"""At most the length of `text` dumped as a JSON string by Jinja's `tojson`.
+
+ A control character is escaped as `\u00XX` and `<`, `>`, `&` and `'` as `\u003c` and the like; a
+ quote and a backslash take one more backslash; a character outside ASCII takes one or two `\uXXXX`.
+ """
+ non_ascii = 0 if text.isascii() else len(text) - len(text.encode("ascii", "ignore"))
+ controls = len(text) - len(text.translate(_CONTROL_CHARACTERS))
+ html_sensitive = text.count("<") + text.count(">") + text.count("&") + text.count("'")
+ return 2 + len(text) + text.count('"') + text.count("\\") + 5 * (controls + html_sensitive) + 11 * non_ascii
+
+
+def is_data_object(*, value: object) -> bool:
+ """Whether `value` is an object of the run's data: neither text, a number, nothing, an undefined value
+ nor a container a template can build.
+ """
+ return not isinstance(value, (str, int, float, type(None), Undefined, *_CONTAINER_TYPES))
+
+
+def _data_text_size(*, value: object, memo: dict[int, int] | None) -> int:
+ # An object of the run's data is converted to count its text: that is the conversion the
+ # operation being estimated makes, and the object's text is the run's, not the template's. The
+ # memo converts an object repeated a thousand times once.
+ if memo is None:
+ return len(str(value))
+ known = memo.get(id(value))
+ if known is None:
+ known = len(str(value))
+ memo[id(value)] = known
+ return known
+
+
+def text_size(value: Any, *, limit: int, memo: dict[int, int] | None = None) -> int:
+ """Estimate the length of `str(value)`, stopping once past `limit`.
+
+ A string's length is exact. A container prints the `repr` of its elements, so it is walked (see
+ `_walk`), and an element that is neither a string, a number nor a container counts the length of
+ its own `repr`, taken once. An object of the run's data at the top level is converted, once per
+ `memo` (see `_data_text_size`).
+ """
+ if isinstance(value, str):
+ return len(value)
+ if isinstance(value, Undefined):
+ return 0
+ if isinstance(value, bool) or value is None:
+ return 5
+ if isinstance(value, int):
+ return _int_text_size(value=value)
+ if isinstance(value, float):
+ return 24
+ if not isinstance(value, _CONTAINER_TYPES):
+ return _data_text_size(value=cast("object", value), memo=memo)
+ return _walk(root=cast("object", value), leaf_units=_repr_leaf_size, container_units=_repr_container_size, limit=limit)
+
+
+def _compare_children(*, value: object) -> Iterator[object] | None:
+ # Two pydantic models are equal when their fields are, so a model is walked like a dict of its fields.
+ if isinstance(value, BaseModel):
+ return iter(vars(value).values())
+ return _children(value=value)
+
+
+def _compare_leaf_units(value: Any) -> int:
+ if isinstance(value, (str, bytes, bytearray)):
+ return len(cast("Sized", value))
+ # A number, nothing, an undefined value, or an object whose equality is its own: a step. An
+ # object's `repr` is never taken, since the comparison would not take it.
+ return 8
+
+
+def _compare_container_units(value: Any) -> int:
+ if isinstance(value, Namespace):
+ return 8
+ if isinstance(value, (list, tuple, dict, set, frozenset)):
+ return 8 * len(cast("Sized", value))
+ if isinstance(value, BaseModel):
+ return 8 * len(vars(value))
+ return 0
+
+
+def compare_weight(value: Any, *, limit: int) -> int:
+ """Estimate what comparing `value` with another value can cost, stopping once past `limit`.
+
+ A string costs its length, since two strings of the same length are compared character by
+ character; a container costs a reference per element plus the weight of each element, counted at
+ every reference, since comparing two containers compares their elements in turn. A pydantic model
+ is a container of its fields.
+ """
+ return _walk(root=value, leaf_units=_compare_leaf_units, container_units=_compare_container_units, limit=limit, children=_compare_children)
diff --git a/pipelex/tools/jinja2/jinja2_render_charging.py b/pipelex/tools/jinja2/jinja2_render_charging.py
new file mode 100644
index 000000000..f4c4c2e08
--- /dev/null
+++ b/pipelex/tools/jinja2/jinja2_render_charging.py
@@ -0,0 +1,627 @@
+"""Where a render is charged: the hooks `PipelexTemplateEnvironment` installs, and the budget they share.
+
+Every render gets its own `RenderBudget` with its context (`BudgetedContext`), and every hook below
+reaches it through that context. The template (`BudgetedTemplate`) also makes it the active budget while
+the render runs, for the work a template sets off where no hook reaches (`active_render_budget`):
+
+- **Calls and operators.** The sandbox's `call` and `call_binop` hooks hand over to `charged_call` and
+ `charged_binop`.
+- **Filters and tests.** Every filter and test registered on the environment is wrapped by
+ `charged_filter` and `charged_test` before a template compiles (`charge_registered_functions`),
+ which refuses a function the cost tables do not classify.
+- **Printing.** `{{ value }}` goes through the environment's `finalize`, built by `make_charged_finalize`.
+- **What no hook reaches.** Loops, `~`, comparisons, slices and static text compile to plain Python,
+ so the environment rewrites the parsed template (`jinja2_render_rewrite.py`) to route each through
+ one of the internal filters defined here.
+
+The rule each hook applies is in `jinja2_render_budget.py`, and the estimates in `jinja2_render_costs.py`.
+"""
+
+from __future__ import annotations
+
+import functools
+import inspect
+import operator
+from typing import TYPE_CHECKING, Any, Final, cast
+
+from jinja2 import pass_context
+from jinja2.environment import Template
+from jinja2.runtime import Context, LoopContext, Macro, markup_join, str_join
+from jinja2.sandbox import SandboxedEscapeFormatter, SandboxedFormatter, safe_range
+from jinja2.utils import generate_lorem_ipsum
+from markupsafe import Markup, escape
+from typing_extensions import override
+
+from pipelex.tools.jinja2.jinja2_render_budget import (
+ CALL_UNITS,
+ DEFAULT_RENDER_BUDGET_UNITS,
+ EAGER_TYPES,
+ ESCAPE_FACTOR,
+ STEP_UNITS,
+ RenderBudget,
+ charged_if_lazy,
+ charged_iterable,
+ check_int_result,
+ compare_weight,
+ is_data_object,
+ produced_size,
+ refuse_int_result,
+ size_of,
+ spending_from,
+ text_size,
+)
+from pipelex.tools.jinja2.jinja2_render_costs import (
+ FILTER_COSTS,
+ INTEGER_FILTERS,
+ LINEAR,
+ PLAIN_VALUE_METHOD_COSTS,
+ TEST_COSTS,
+ CostInputs,
+ OperationCost,
+ comparison_units,
+ escaped_text_size,
+ format_field_units,
+ lipsum_units,
+ percent_format_units,
+)
+
+if TYPE_CHECKING:
+ from collections.abc import Awaitable, Callable, Sized
+
+ from jinja2.environment import Environment
+
+# The key a render's budget is kept under among its context's variables. It is no identifier, so no
+# template can name it, and an included template, whose context is built from its includer's
+# variables, shares it.
+RENDER_BUDGET_KEY: Final = "pipelex:render-budget"
+
+
+class BudgetedContext(Context):
+ """A render's context, carrying the budget the render spends from."""
+
+ def __init__(
+ self,
+ environment: Environment,
+ parent: dict[str, Any],
+ name: str | None,
+ blocks: dict[str, Callable[[Context], Any]],
+ **kwargs: Any,
+ ) -> None:
+ super().__init__(environment, parent, name, blocks, **kwargs)
+ budget = parent.get(RENDER_BUDGET_KEY)
+ if not isinstance(budget, RenderBudget):
+ total: int = getattr(environment, "render_budget_units", DEFAULT_RENDER_BUDGET_UNITS)
+ budget = RenderBudget(total=total)
+ self.vars[RENDER_BUDGET_KEY] = budget
+ self.render_budget: RenderBudget = budget
+
+
+def _with_render_budget(*, environment: Environment, variables: dict[str, Any]) -> RenderBudget:
+ """The budget of the render about to start with `variables`, put among them for its context to find."""
+ budget = variables.get(RENDER_BUDGET_KEY)
+ if not isinstance(budget, RenderBudget):
+ total: int = getattr(environment, "render_budget_units", DEFAULT_RENDER_BUDGET_UNITS)
+ budget = RenderBudget(total=total)
+ variables[RENDER_BUDGET_KEY] = budget
+ return budget
+
+
+class BudgetedTemplate(Template):
+ """A template whose render makes its budget the active one (`spending_from`) while it runs.
+
+ The hooks reach the budget through the render's context; the active budget is for code that no hook
+ reaches and that has no context, such as markupsafe calling a Markdown value's `__html__`. A render
+ started with `generate` or `stream` has no active budget, since its output is produced as its caller
+ iterates; Pipelex renders only with `render` and `render_async`.
+ """
+
+ @override
+ def render(self, *args: Any, **kwargs: Any) -> str:
+ variables: dict[str, Any] = dict(*args, **kwargs)
+ with spending_from(budget=_with_render_budget(environment=self.environment, variables=variables)):
+ return super().render(variables)
+
+ @override
+ async def render_async(self, *args: Any, **kwargs: Any) -> str:
+ variables: dict[str, Any] = dict(*args, **kwargs)
+ with spending_from(budget=_with_render_budget(environment=self.environment, variables=variables)):
+ return await super().render_async(variables)
+
+
+def render_budget_of(context: Context) -> RenderBudget:
+ """The budget of the render `context` belongs to."""
+ try:
+ return cast("BudgetedContext", context).render_budget
+ except AttributeError:
+ msg = f"A Pipelex template renders with a '{BudgetedContext.__name__}', not a '{type(context).__name__}'"
+ raise TypeError(msg) from None
+
+
+def _inputs_units(*, values: tuple[Any, ...], keywords: dict[str, Any], step: int = STEP_UNITS) -> int:
+ units = step
+ for value in values:
+ units += size_of(value)
+ for value in keywords.values():
+ units += size_of(value)
+ return units
+
+
+def _charge_result(*, budget: RenderBudget, result: Any, operation: str | Callable[[], str], produces_int: bool) -> Any:
+ if produces_int and type(result) is int:
+ check_int_result(value=result, operation=operation() if callable(operation) else operation)
+ budget.charge(units=produced_size(result), operation=operation)
+ return result
+
+
+async def _charge_awaited(*, budget: RenderBudget, pending: Awaitable[Any], operation: str | Callable[[], str], produces_int: bool) -> Any:
+ result = await pending
+ return _charge_result(budget=budget, result=result, operation=operation, produces_int=produces_int)
+
+
+def _finish(*, budget: RenderBudget, result: Any, operation: str | Callable[[], str], produces_int: bool = False) -> Any:
+ """Charge an operation's result, once awaited when the operation is async."""
+ if type(result) not in EAGER_TYPES and inspect.isawaitable(result):
+ return _charge_awaited(budget=budget, pending=result, operation=operation, produces_int=produces_int)
+ return _charge_result(budget=budget, result=result, operation=operation, produces_int=produces_int)
+
+
+########################################################################################
+# Calls
+########################################################################################
+
+
+_FORMATTING: Final = "formatting a replacement field"
+
+
+class _ChargedFormatter(SandboxedFormatter):
+ """The sandbox's formatter, charging every replacement field it formats.
+
+ A field is formatted with its spec already resolved, so a width taken from an argument (`{:{w}}`)
+ or from a mapping is read as the number it is, and the value is whatever the field's name led to
+ (`{0.text}`), after the sandbox's own checks.
+ """
+
+ def __init__(self, environment: Environment, *, budget: RenderBudget, escaping: bool, **kwargs: Any) -> None:
+ super().__init__(environment, **kwargs)
+ self._budget = budget
+ self._escaping = escaping
+ self._text_sizes: dict[int, int] = {}
+
+ @override
+ def format_field(self, value: Any, format_spec: str) -> Any:
+ budget = self._budget
+ estimate = format_field_units(value=value, spec=format_spec, limit=budget.remaining, escaping=self._escaping, memo=self._text_sizes)
+ budget.afford(units=STEP_UNITS + estimate, operation=_FORMATTING)
+ text = super().format_field(value, format_spec)
+ budget.charge(units=STEP_UNITS + size_of(text), operation=_FORMATTING)
+ return text
+
+
+class _ChargedEscapeFormatter(_ChargedFormatter, SandboxedEscapeFormatter):
+ """The charged formatter of a markup format string, which escapes what it inserts."""
+
+
+class SandboxedStrFormat:
+ """The `str.format` or `str.format_map` of a string, routed through the sandbox's safe formatter.
+
+ Jinja builds its own wrapper when a template reads `format` on a string, so that the replacement
+ fields of the format string (`{0.name}`, `{0[key]}`) go through the environment's own attribute
+ and item checks. Wrapping that in a type of its own lets the call policy recognise it by type,
+ while the raw `str.format` stays refused, and lets a template's call format through a formatter
+ that charges every field it formats (`format_charged`).
+ """
+
+ __slots__ = ("_environment", "_format", "_is_format_map", "template")
+
+ def __init__(self, *, format_function: Callable[..., str], environment: Environment, template: str, is_format_map: bool) -> None:
+ self._format = format_function
+ self._environment = environment
+ self._is_format_map = is_format_map
+ self.template = template
+
+ def __call__(self, *args: Any, **kwargs: Any) -> str:
+ return self._format(*args, **kwargs)
+
+ def format_charged(self, *, args: tuple[Any, ...], kwargs: dict[str, Any], budget: RenderBudget) -> str:
+ """Format as `__call__` does, charging every replacement field to `budget` as it is formatted."""
+ if self._is_format_map:
+ if kwargs or len(args) != 1:
+ # Jinja's own wrapper raises the error Python's `format_map` would.
+ return self._format(*args, **kwargs)
+ mapping: Any = args[0]
+ args, kwargs = (), mapping
+ formatter: _ChargedFormatter
+ if isinstance(self.template, Markup):
+ formatter = _ChargedEscapeFormatter(self._environment, budget=budget, escaping=True, escape=self.template.escape)
+ else:
+ formatter = _ChargedFormatter(self._environment, budget=budget, escaping=False)
+ return type(self.template)(formatter.vformat(self.template, args, kwargs))
+
+
+def _plain_method_cost(*, obj: Any) -> tuple[Any, OperationCost] | None:
+ """The value a plain-value method is bound to and the method's cost, or None for any other callable."""
+ bound_to: Any = getattr(obj, "__self__", None)
+ if bound_to is None or isinstance(bound_to, type):
+ return None
+ name = getattr(obj, "__name__", "")
+ for klass in type(bound_to).__mro__:
+ table = PLAIN_VALUE_METHOD_COSTS.get(klass)
+ if table is not None:
+ cost = table.get(name)
+ return (bound_to, cost) if cost is not None else None
+ return None
+
+
+def _call_inputs_units(*, obj: Any, plain_method: tuple[Any, OperationCost] | None, args: tuple[Any, ...], kwargs: dict[str, Any]) -> int:
+ """What a call is charged for its inputs, before it runs."""
+ if plain_method is not None:
+ bound_value, cost = plain_method
+ return _inputs_units(values=(bound_value, *args) if cost.reads_value else args, keywords=kwargs, step=CALL_UNITS)
+ if isinstance(obj, (Macro, SandboxedStrFormat)):
+ # A macro takes its arguments by reference, and its body is charged step by step; a format
+ # string is charged field by field.
+ return CALL_UNITS
+ return _inputs_units(values=args, keywords=kwargs, step=CALL_UNITS)
+
+
+def charged_call(*, context: Context, obj: Any, args: tuple[Any, ...], kwargs: dict[str, Any], operation: Callable[[], str]) -> Any:
+ """Make a call the sandbox has allowed, charging it to the render's budget."""
+ budget = render_budget_of(context)
+ plain_method = _plain_method_cost(obj=obj)
+ bound_value: Any = plain_method[0] if plain_method is not None else None
+ cost = plain_method[1] if plain_method is not None else LINEAR
+ budget.charge(units=_call_inputs_units(obj=obj, plain_method=plain_method, args=args, kwargs=kwargs), operation=operation)
+ if any(type(argument) not in EAGER_TYPES for argument in args):
+ args = tuple(charged_if_lazy(value=argument, budget=budget) for argument in args)
+ if kwargs:
+ kwargs = {name: charged_if_lazy(value=argument, budget=budget) for name, argument in kwargs.items()}
+ if cost.materializes and args and not isinstance(args[0], (str, list, tuple, dict, set, frozenset)):
+ args = (list(args[0]), *args[1:])
+ estimate = 0
+ if isinstance(obj, SandboxedStrFormat):
+ return _finish(budget=budget, result=obj.format_charged(args=args, kwargs=kwargs, budget=budget), operation=operation)
+ if cost.work is not None:
+ inputs = CostInputs(value=bound_value, args=args, kwargs=kwargs, limit=budget.remaining, escaping=False, environment=context.environment)
+ budget.charge(units=cost.work(inputs=inputs), operation=operation)
+ if cost.estimate is not None:
+ inputs = CostInputs(value=bound_value, args=args, kwargs=kwargs, limit=budget.remaining, escaping=False, environment=context.environment)
+ estimate = cost.estimate(inputs=inputs)
+ elif obj is generate_lorem_ipsum:
+ # Spent rather than only checked: every paragraph is generated, whatever text it ends with.
+ budget.charge(
+ units=lipsum_units(inputs=CostInputs(value=None, args=args, kwargs=kwargs, limit=budget.remaining, escaping=False)), operation=operation
+ )
+ elif obj is safe_range:
+ for bound in (*args, *kwargs.values()):
+ if isinstance(bound, int):
+ refuse_int_result(bits=bound.bit_length(), operation="a range bound")
+ elif isinstance(obj, LoopContext) and args:
+ # A recursive loop's `loop(children)` iterates `children` without passing the loop's own
+ # iterable through the rewrite, so the elements are charged here.
+ args = (charged_iterable(iterable=args[0], budget=budget), *args[1:])
+ budget.afford(units=estimate, operation=operation)
+ result = context.call(obj, *args, **kwargs)
+ return _finish(budget=budget, result=result, operation=operation) if cost.reads_value else result
+
+
+########################################################################################
+# Operators
+########################################################################################
+
+
+_OPERATOR_DESCRIPTIONS: Final[dict[str, str]] = {operator: f"the operator '{operator}'" for operator in ("+", "-", "*", "/", "//", "%", "**")}
+
+
+def _repeat_units(*, sequence: Any, times: Any) -> int | None:
+ """The size of `sequence * times`, or None when that is not a repetition."""
+ if isinstance(times, bool) or not isinstance(times, int) or not isinstance(sequence, (str, bytes, bytearray, list, tuple)):
+ return None
+ count: int = max(times, 0)
+ per_element = 1 if isinstance(sequence, (str, bytes, bytearray)) else 8 + 64
+ return len(cast("Sized", sequence)) * count * per_element
+
+
+_ESTIMATED_OPERATORS: Final = frozenset({"*", "**", "%", "+"})
+
+
+def _estimate_binop(*, operator: str, left: Any, right: Any, limit: int, describe: str) -> int:
+ """Estimate an operator's result before it runs, refusing an integer result that would be too wide."""
+ match operator:
+ case "*":
+ repeated = _repeat_units(sequence=left, times=right)
+ if repeated is None:
+ repeated = _repeat_units(sequence=right, times=left)
+ if repeated is not None:
+ return repeated
+ if isinstance(left, int) and isinstance(right, int):
+ refuse_int_result(bits=left.bit_length() + right.bit_length(), operation=describe)
+ case "**":
+ if isinstance(left, int) and isinstance(right, int) and right > 0 and abs(left) > 1:
+ refuse_int_result(bits=(abs(left).bit_length() - 1) * right + 1, operation=describe)
+ case "%":
+ if isinstance(left, str):
+ return percent_format_units(template=left, arguments=right, limit=limit, escaping=isinstance(left, Markup))
+ case "+":
+ if isinstance(left, (str, list, tuple)) and isinstance(right, (str, list, tuple)):
+ # Adding text to markup escapes the text.
+ left_factor = ESCAPE_FACTOR if isinstance(right, Markup) and not isinstance(left, Markup) else 1
+ right_factor = ESCAPE_FACTOR if isinstance(left, Markup) and not isinstance(right, Markup) else 1
+ return left_factor * size_of(left) + right_factor * size_of(right)
+ case _:
+ pass
+ return 0
+
+
+def charged_binop(*, context: Context, operator: str, left: Any, right: Any, compute: Callable[[Any, Any], Any]) -> Any:
+ """Apply a binary operator the sandbox intercepted, charging it to the render's budget."""
+ budget = render_budget_of(context)
+ describe = _OPERATOR_DESCRIPTIONS[operator]
+ budget.charge(units=STEP_UNITS + size_of(left) + size_of(right), operation=describe)
+ if operator in _ESTIMATED_OPERATORS:
+ budget.afford(units=_estimate_binop(operator=operator, left=left, right=right, limit=budget.remaining, describe=describe), operation=describe)
+ return _charge_result(budget=budget, result=compute(left, right), operation=describe, produces_int=True)
+
+
+########################################################################################
+# Filters and tests
+########################################################################################
+
+
+def _pass_arg_of(*, function: Callable[..., Any]) -> str | None:
+ """What a filter, test or finalize asked Jinja to pass before its value: the name of the mark Jinja's
+ `pass_context`, `pass_eval_context` and `pass_environment` decorators leave on it.
+ """
+ mark = getattr(function, "jinja_pass_arg", None)
+ return None if mark is None else str(mark.name)
+
+
+def _first_argument(*, pass_arg: str | None, context: Context) -> tuple[Any, ...]:
+ """What a filter or test asked Jinja to pass before its value, as Jinja would have passed it."""
+ match pass_arg:
+ case "context":
+ return (context,)
+ case "eval_context":
+ return (context.eval_ctx,)
+ case "environment":
+ return (context.environment,)
+ case _:
+ return ()
+
+
+async def _materialize_then(*, value: Any, then: Callable[[Any], Any]) -> Any:
+ materialized = [element async for element in value]
+ result = then(materialized)
+ return await result if inspect.isawaitable(result) else result
+
+
+CHARGED_MARK: Final = "pipelex_render_charged"
+
+
+def _mark_charged(*, charged: Callable[..., Any], function: Callable[..., Any]) -> None:
+ # `__wrapped__` lets `inspect.unwrap` find the filter or test behind the wrapper. The function's
+ # `__dict__` is left behind (`updated=()`): it holds the function's own `jinja_pass_arg`, and the
+ # wrapper's is its own.
+ functools.update_wrapper(charged, function, updated=())
+ setattr(charged, CHARGED_MARK, True)
+
+
+def charged_filter(*, name: str, function: Callable[..., Any]) -> Callable[..., Any]:
+ """Wrap a filter so that applying it is charged to the render's budget."""
+ cost = FILTER_COSTS.get(function)
+ if cost is None:
+ msg = f"The template filter '{name}' has no cost in the render budget's tables (jinja2_render_costs.py), so it cannot be registered."
+ raise TypeError(msg)
+ produces_int = function in INTEGER_FILTERS
+ pass_arg = _pass_arg_of(function=function)
+ operation = f"the filter '{name}'"
+
+ def apply(*, context: Context, budget: RenderBudget, value: Any, args: tuple[Any, ...], kwargs: dict[str, Any]) -> Any:
+ escaping = bool(context.eval_ctx.autoescape)
+ if cost.work is not None:
+ inputs = CostInputs(value=value, args=args, kwargs=kwargs, limit=budget.remaining, escaping=escaping, environment=context.environment)
+ budget.charge(units=cost.work(inputs=inputs), operation=operation)
+ if cost.estimate is not None:
+ inputs = CostInputs(value=value, args=args, kwargs=kwargs, limit=budget.remaining, escaping=escaping, environment=context.environment)
+ budget.afford(units=cost.estimate(inputs=inputs), operation=operation)
+ result = function(*_first_argument(pass_arg=pass_arg, context=context), value, *args, **kwargs)
+ if not cost.reads_value:
+ return result
+ return _finish(budget=budget, result=result, operation=operation, produces_int=produces_int)
+
+ @pass_context
+ def charged(context: Context, value: Any, *args: Any, **kwargs: Any) -> Any:
+ budget = render_budget_of(context)
+ budget.charge(units=_inputs_units(values=(value, *args) if cost.reads_value else args, keywords=kwargs), operation=operation)
+ value = charged_if_lazy(value=value, budget=budget)
+ if cost.materializes and type(value) not in EAGER_TYPES:
+ if hasattr(value, "__aiter__"):
+ return _materialize_then(
+ value=value, then=lambda materialized: apply(context=context, budget=budget, value=materialized, args=args, kwargs=kwargs)
+ )
+ if hasattr(value, "__next__"):
+ value = list(value)
+ return apply(context=context, budget=budget, value=value, args=args, kwargs=kwargs)
+
+ _mark_charged(charged=charged, function=function)
+ return charged
+
+
+def charged_test(*, name: str, function: Callable[..., Any]) -> Callable[..., Any]:
+ """Wrap a test so that applying it is charged to the render's budget."""
+ cost = TEST_COSTS.get(function)
+ if cost is None:
+ msg = f"The template test '{name}' has no cost in the render budget's tables (jinja2_render_costs.py), so it cannot be registered."
+ raise TypeError(msg)
+
+ pass_arg = _pass_arg_of(function=function)
+ operation = f"the test '{name}'"
+
+ @pass_context
+ def charged(context: Context, value: Any, *args: Any, **kwargs: Any) -> Any:
+ budget = render_budget_of(context)
+ # A test reads its value only when it compares it or scans its text, both charged below.
+ units = _inputs_units(values=args, keywords=kwargs)
+ if cost.comparison is not None and args:
+ units += comparison_units(operator=cost.comparison, left=value, right=args[0], limit=budget.remaining)
+ elif cost.scans_text and isinstance(value, str):
+ units += len(value)
+ budget.charge(units=units, operation=operation)
+ return function(*_first_argument(pass_arg=pass_arg, context=context), value, *args, **kwargs)
+
+ _mark_charged(charged=charged, function=function)
+ return charged
+
+
+def is_charged(*, function: Callable[..., Any]) -> bool:
+ """Whether `function` is a filter or a test already wrapped to charge the render's budget."""
+ return bool(getattr(function, CHARGED_MARK, False))
+
+
+def charge_registered_functions(*, filters: dict[str, Callable[..., Any]], tests: dict[str, Callable[..., Any]]) -> None:
+ """Wrap, in place, every filter and test registered since the last time, the budget's own filters apart.
+
+ The environment calls this before it compiles a template, so a filter registered by assigning into
+ `environment.filters`, as Jinja lets anyone do, is charged like the rest, and one the cost tables
+ do not classify is refused before any template can use it.
+ """
+ for name, function in list(filters.items()):
+ if not is_charged(function=function) and INTERNAL_FILTERS.get(name) is not function:
+ filters[name] = charged_filter(name=name, function=function)
+ for name, function in list(tests.items()):
+ if not is_charged(function=function):
+ tests[name] = charged_test(name=name, function=function)
+
+
+########################################################################################
+# Printing
+########################################################################################
+
+
+_PRINTING: Final = "printing a value"
+
+
+def make_charged_finalize(*, inner: Callable[..., Any] | None) -> Callable[..., Any]:
+ """The environment's `finalize`: convert a printed value to text, charged, after the caller's own `finalize`.
+
+ Jinja converts what `finalize` returns once more (`str()`, or `escape()` under autoescape), which
+ leaves text this has already converted unchanged.
+ """
+ inner_pass_arg = _pass_arg_of(function=inner) if inner is not None else None
+
+ @pass_context
+ def charged_finalize(context: Context, value: Any) -> Any:
+ if inner is not None:
+ value = inner(*_first_argument(pass_arg=inner_pass_arg, context=context), value)
+ budget = render_budget_of(context)
+ autoescape = context.eval_ctx.autoescape
+ # An object of the run's data is not estimated: its estimate would convert it, which printing it
+ # does once anyway, and it is charged once converted.
+ if autoescape and not is_data_object(value=value):
+ budget.afford(units=STEP_UNITS + escaped_text_size(value, limit=budget.remaining), operation=_PRINTING)
+ elif type(value) is not str and not is_data_object(value=value):
+ budget.afford(units=STEP_UNITS + text_size(value, limit=budget.remaining), operation=_PRINTING)
+ text: Any = escape(value) if autoescape else value if isinstance(value, str) else str(value)
+ budget.charge(units=STEP_UNITS + size_of(text), operation=_PRINTING)
+ return text
+
+ return charged_finalize
+
+
+########################################################################################
+# The internal filters the rewritten template calls
+########################################################################################
+
+ITERATE_FILTER: Final = "_pipelex_charged_iteration"
+CONCAT_FILTER: Final = "_pipelex_charged_concat"
+COMPARE_FILTER: Final = "_pipelex_charged_comparison"
+COMPARED_FILTER: Final = "_pipelex_charged_operand"
+SLICED_FILTER: Final = "_pipelex_charged_slice"
+LITERAL_FILTER: Final = "_pipelex_charged_literal"
+
+_COMPARISONS: Final[dict[str, Callable[[Any, Any], Any]]] = {
+ "eq": operator.eq,
+ "ne": operator.ne,
+ "gt": operator.gt,
+ "gteq": operator.ge,
+ "lt": operator.lt,
+ "lteq": operator.le,
+ "in": lambda left, right: left in right,
+ "notin": lambda left, right: left not in right,
+}
+
+
+@pass_context
+def charged_iteration(context: Context, iterable: Any) -> Any:
+ """A loop's iterable, charged for every element the loop draws."""
+ return charged_iterable(iterable=iterable, budget=render_budget_of(context))
+
+
+@pass_context
+def charged_concat(context: Context, parts: list[Any]) -> Any:
+ """`a ~ b ~ c`: join the parts' text, estimated before it is built."""
+ budget = render_budget_of(context)
+ autoescape = bool(context.eval_ctx.autoescape)
+ measure = escaped_text_size if autoescape else text_size
+ estimate = sum(measure(part, limit=budget.remaining) for part in parts)
+ budget.afford(units=STEP_UNITS + estimate, operation="joining text with '~'")
+ joined = markup_join(parts) if autoescape else str_join(parts)
+ budget.charge(units=STEP_UNITS + size_of(joined), operation="joining text with '~'")
+ return joined
+
+
+@pass_context
+def charged_comparison(context: Context, left: Any, operator: str, right: Any) -> Any:
+ """`left <operator> right`, one comparison, charged for what comparing the two can cost."""
+ budget = render_budget_of(context)
+ if operator in {"in", "notin"}:
+ right = charged_if_lazy(value=right, budget=budget)
+ units = STEP_UNITS + comparison_units(operator=operator, left=left, right=right, limit=budget.remaining)
+ budget.charge(units=units, operation="a comparison")
+ return _COMPARISONS[operator](left, right)
+
+
+@pass_context
+def charged_operand(context: Context, value: Any) -> Any:
+ """An operand of a chained comparison (`a < b < c`), charged for everything comparing it can cost."""
+ budget = render_budget_of(context)
+ budget.charge(units=STEP_UNITS + compare_weight(value, limit=budget.remaining), operation="a comparison")
+ return value
+
+
+@pass_context
+def charged_slice(context: Context, value: Any) -> Any:
+ """The result of `value[start:stop]`, which Jinja slices without its `getitem` hook."""
+ budget = render_budget_of(context)
+ budget.charge(units=STEP_UNITS + produced_size(value), operation="a slice")
+ return value
+
+
+@pass_context
+def charged_literal(context: Context, value: Any) -> Any:
+ """A list, tuple or dict a template writes out (`[a, b]`, `(a, b)`, `{k: v}`), charged once built.
+
+ A literal holds as many elements as the template's source writes, so it is charged after it is
+ built, for its storage and its elements, and a dict also for hashing its tuple keys, which Python
+ does again at every build, since a tuple does not keep its hash.
+ """
+ budget = render_budget_of(context)
+ units = STEP_UNITS + produced_size(value)
+ budget.charge(units=units + tuple_keys_weight(literal=value, limit=budget.remaining), operation="building a list, a tuple or a dict")
+ return value
+
+
+def tuple_keys_weight(*, literal: object, limit: int) -> int:
+ """What hashing a dict's tuple keys costs: every element, nested ones included, at every hash."""
+ if not isinstance(literal, dict):
+ return 0
+ return sum(compare_weight(key, limit=limit) for key in cast("dict[object, object]", literal) if isinstance(key, tuple))
+
+
+INTERNAL_FILTERS: Final[dict[str, Callable[..., Any]]] = {
+ ITERATE_FILTER: charged_iteration,
+ CONCAT_FILTER: charged_concat,
+ COMPARE_FILTER: charged_comparison,
+ COMPARED_FILTER: charged_operand,
+ SLICED_FILTER: charged_slice,
+ LITERAL_FILTER: charged_literal,
+}
diff --git a/pipelex/tools/jinja2/jinja2_render_costs.py b/pipelex/tools/jinja2/jinja2_render_costs.py
new file mode 100644
index 000000000..e37f1f13f
--- /dev/null
+++ b/pipelex/tools/jinja2/jinja2_render_costs.py
@@ -0,0 +1,1101 @@
+"""What each operation a template can perform costs: the tables and the estimates of the render budget.
+
+Most operations are charged by their inputs and their result alone (`jinja2_render_budget.py`), which
+is enough when the result is at most a small multiple of the inputs. The operations here can produce
+much more than they are given, from a width, a count, a separator or a nesting, so each has an
+estimate of its result that the render checks before running it. The tables are exhaustive on
+purpose: a method of a plain value, a filter or a test that is not in them is refused, so one that a
+later Python or Jinja adds cannot be reached before someone has decided what it costs, and the tests
+check the tables against everything the running versions offer.
+"""
+
+from __future__ import annotations
+
+import math
+import re
+from collections.abc import Collection, Mapping, Sized
+from datetime import date, datetime, time, timedelta
+from decimal import Decimal
+from typing import TYPE_CHECKING, Any, Final, NamedTuple, Protocol, cast
+
+import jinja2.filters
+import jinja2.tests
+from jinja2.runtime import Undefined
+from markupsafe import Markup
+from pydantic import BaseModel
+
+from pipelex.tools.jinja2.jinja2_filters import escape_script_tag, markdown_to_html, tag, text_format
+from pipelex.tools.jinja2.jinja2_render_budget import (
+ ELEMENT_UNITS,
+ ESCAPE_FACTOR,
+ STEP_UNITS,
+ compare_weight,
+ escaped_length,
+ is_data_object,
+ json_string_length,
+ text_size,
+)
+from pipelex.tools.jinja2.jinja2_with_images_filter import with_images
+
+if TYPE_CHECKING:
+ from collections.abc import Callable, Iterable, Iterator, Sequence
+
+
+class CostInputs(NamedTuple):
+ """What an estimate reads: the operation's inputs and what is left of the budget.
+
+ `value` is the value the operation applies to: a filter's input, or the value a method is bound
+ to. `escaping` says whether the operation escapes what it inserts. `limit` is what is left of the
+ budget, past which a walk may stop counting. `environment` is the render's, when the operation reads
+ an attribute of its items the way a filter does (`sort(attribute=...)`).
+ """
+
+ value: Any
+ args: tuple[Any, ...]
+ kwargs: dict[str, Any]
+ limit: int
+ escaping: bool
+ environment: Any = None
+
+ def arg(self, *, index: int, name: str, default: Any = None) -> Any:
+ """An argument passed either at `index` or by `name`."""
+ if len(self.args) > index:
+ return self.args[index]
+ return self.kwargs.get(name, default)
+
+
+class CostEstimate(Protocol):
+ """The units an operation's result will take, estimated from its inputs before it runs."""
+
+ def __call__(self, *, inputs: CostInputs) -> int: ...
+
+
+class OperationCost(NamedTuple):
+ """How the budget treats one method or filter beyond charging its inputs and its result.
+
+ - `estimate`: the size of the result, checked before the operation runs.
+ - `work`: what the operation spends beyond reading its inputs, such as the comparisons of a sort,
+ charged before it runs.
+ - `materializes`: a lazy input (a filter's value, a method's first argument) is drawn into a list
+ first, so that the estimates can count it.
+ - `reads_value`: whether the operation reads the whole of its value (a filter's input, the value a
+ method is bound to), which is then charged by its size, like its result. One that does not, such
+ as `len` or `dict.get`, reads a fixed part of its value and returns a reference into it or a small
+ value, so it is charged a step and its arguments only.
+ """
+
+ estimate: CostEstimate | None = None
+ work: CostEstimate | None = None
+ materializes: bool = False
+ reads_value: bool = True
+
+
+# Charged by its inputs and its result only.
+LINEAR: Final = OperationCost()
+
+# Charged a step and its arguments: it reads a fixed part of its value, whatever the value's size, and
+# returns a reference into it or a small value.
+CONSTANT: Final = OperationCost(reads_value=False)
+
+
+def _length(*, value: Any) -> int:
+ return len(value) if isinstance(value, Sized) else 0
+
+
+def _digits_value(*, digits: str, limit: int) -> int:
+ """The integer a run of digits in a format spec stands for, saturating past `limit`."""
+ if len(digits) > len(str(limit)):
+ return limit + 1
+ return int(digits)
+
+
+########################################################################################
+# Format strings: `%`, the `format` filter and `str.format`
+########################################################################################
+
+
+class _PercentDirective(NamedTuple):
+ key: str | None
+ width: str | None
+ precision: str | None
+ conversion: str
+
+
+def _digits_at(*, template: str, start: int) -> int:
+ end = start
+ while end < len(template) and template[end].isdigit():
+ end += 1
+ return end
+
+
+def _percent_directives(*, template: str) -> Iterator[_PercentDirective]:
+ """The directives of a `%` format string, read in one pass the way Python reads them.
+
+ A mapping key ends at the parenthesis that closes the one opening it, so `%(a(b))5s` is one directive
+ with the key `a(b)` and a width of 5. A directive left unfinished ends the scan, since formatting it
+ raises.
+ """
+ length = len(template)
+ position = template.find("%")
+ while 0 <= position < length - 1:
+ cursor = position + 1
+ key: str | None = None
+ if template[cursor] == "(":
+ depth = 1
+ key_start = cursor + 1
+ cursor = key_start
+ while cursor < length and depth:
+ if template[cursor] == "(":
+ depth += 1
+ elif template[cursor] == ")":
+ depth -= 1
+ cursor += 1
+ if depth:
+ return
+ key = template[key_start : cursor - 1]
+ while cursor < length and template[cursor] in "#0- +":
+ cursor += 1
+ width: str | None = None
+ if cursor < length and template[cursor] == "*":
+ width = "*"
+ cursor += 1
+ else:
+ end = _digits_at(template=template, start=cursor)
+ width = template[cursor:end] or None
+ cursor = end
+ precision: str | None = None
+ if cursor < length and template[cursor] == ".":
+ cursor += 1
+ if cursor < length and template[cursor] == "*":
+ precision = "*"
+ cursor += 1
+ else:
+ end = _digits_at(template=template, start=cursor)
+ precision = template[cursor:end] or None
+ cursor = end
+ if cursor < length and template[cursor] in "hlL":
+ cursor += 1
+ if cursor >= length:
+ return
+ yield _PercentDirective(key=key, width=width, precision=precision, conversion=template[cursor])
+ position = template.find("%", cursor + 1)
+
+
+def percent_format_units(*, template: str, arguments: Any, limit: int, escaping: bool) -> int:
+ """Estimate the result of `template % arguments`: its text, every width and precision, every argument."""
+ factor = ESCAPE_FACTOR if escaping else 1
+ positional: list[Any] = list(cast("tuple[Any, ...]", arguments)) if isinstance(arguments, tuple) else [arguments]
+ mapping: Mapping[Any, Any] = cast("Mapping[Any, Any]", arguments) if isinstance(arguments, Mapping) else {}
+ total = len(template)
+ position = 0
+ memo: dict[int, int] = {}
+ for spec in _percent_directives(template=template):
+ if spec.conversion == "%":
+ continue
+ for part in (spec.width, spec.precision):
+ if part == "*":
+ star = positional[position] if position < len(positional) else 0
+ position += 1
+ total += abs(star) if isinstance(star, int) else 0
+ elif part:
+ total += _digits_value(digits=part, limit=limit)
+ if spec.key is not None:
+ argument = mapping.get(spec.key)
+ else:
+ argument = positional[position] if position < len(positional) else None
+ position += 1
+ total += factor * text_size(argument, limit=limit, memo=memo)
+ if total > limit:
+ return total
+ return total
+
+
+_SPEC_NUMBER: Final = re.compile(r"\d+")
+
+
+def format_field_units(*, value: Any, spec: str, limit: int, escaping: bool, memo: dict[int, int]) -> int:
+ """Estimate `format(value, spec)`, one replacement field of `str.format` with its spec already resolved.
+
+ It counts the value's text, every number in the spec (a width or a precision), and a multiple of
+ the spec's length, which a date's spec takes as a `strftime` format.
+ """
+ total = text_size(value, limit=limit, memo=memo)
+ if spec:
+ total += sum(_digits_value(digits=number, limit=limit) for number in _SPEC_NUMBER.findall(spec))
+ total += 32 * len(spec)
+ return total * (ESCAPE_FACTOR if escaping else 1)
+
+
+########################################################################################
+# Methods of plain values
+########################################################################################
+
+# The characters `str.split()` splits on, and those `str.splitlines()` ends a line at.
+_ASCII_WHITESPACE: Final = " \t\n\r\x0b\x0c\x1c\x1d\x1e\x1f"
+_WHITESPACE: Final = _ASCII_WHITESPACE + "\x85\xa0            

   "
+_ASCII_LINE_BREAKS: Final = "\n\r\x0b\x0c\x1c\x1d\x1e"
+_LINE_BREAKS: Final = _ASCII_LINE_BREAKS + "\x85

"
+
+
+def _count_any(*, text: str, characters: str) -> int:
+ return sum(text.count(character) for character in characters)
+
+
+def _padded_width(*, inputs: CostInputs) -> int:
+ # `center`, `ljust`, `rjust` and `zfill` pad to their width.
+ width = inputs.arg(index=0, name="width", default=0)
+ return max(_length(value=inputs.value), width if isinstance(width, int) else 0)
+
+
+def _expanded_tabs(*, inputs: CostInputs) -> int:
+ tab_size = inputs.arg(index=0, name="tabsize", default=8)
+ if not isinstance(inputs.value, str) or not isinstance(tab_size, int):
+ return _length(value=inputs.value)
+ return len(inputs.value) + inputs.value.count("\t") * max(tab_size, 0)
+
+
+def _replacement_units(*, text: str, old: str, new: str, count: Any, escaping: bool) -> int:
+ """Estimate `text.replace(old, new, count)`, where escaping can grow the text and the replacement sixfold."""
+ if escaping:
+ text_length = escaped_length(text)
+ # The escaped text can hold `old` where the text does not, so every place it could fit counts.
+ occurrences = text_length // len(old) + 1 if old else text_length + 1
+ new_length = ESCAPE_FACTOR * len(new)
+ else:
+ text_length = len(text)
+ # An empty `old` matches between every two characters.
+ occurrences = text.count(old) if old else text_length + 1
+ new_length = len(new)
+ if isinstance(count, int) and count >= 0:
+ occurrences = min(occurrences, count)
+ return text_length + occurrences * new_length
+
+
+def _replaced(*, inputs: CostInputs) -> int:
+ old = inputs.arg(index=0, name="old")
+ new = inputs.arg(index=1, name="new")
+ if not isinstance(inputs.value, str) or not isinstance(old, str) or not isinstance(new, str):
+ # `str.replace` takes strings only, and fails on anything else before it allocates.
+ return _length(value=inputs.value)
+ return _replacement_units(
+ text=inputs.value,
+ old=old,
+ new=new,
+ count=inputs.arg(index=2, name="count", default=-1),
+ escaping=isinstance(inputs.value, Markup),
+ )
+
+
+def _value_text(*, value: Any, limit: int) -> str | None:
+ """The text a filter converts `value` to, or None when that text would be longer than `limit`.
+
+ A filter that converts its value (`str(value)`) and then works on the text is estimated on the
+ text: converting first costs what the filter's own conversion would, and is refused when a
+ container's text would already be too long.
+ """
+ if isinstance(value, str):
+ return value
+ if not is_data_object(value=value) and text_size(value, limit=limit) > limit:
+ return None
+ return str(value)
+
+
+def _joined_units(*, items: Any, separator: str, limit: int, escaping: bool) -> int:
+ if not isinstance(items, Collection):
+ return len(separator)
+ collection = cast("Collection[Any]", items)
+ factor = ESCAPE_FACTOR if escaping else 1
+ count = len(collection)
+ # Joining draws the items into a sequence first, one reference and often one new object each.
+ total = count * (len(separator) * factor + ELEMENT_UNITS)
+ if isinstance(collection, str):
+ return total + count * factor
+ memo: dict[int, int] = {}
+ for item in collection:
+ total += factor * text_size(item, limit=limit, memo=memo)
+ if total > limit:
+ return total
+ return total
+
+
+def _joined(*, inputs: CostInputs) -> int:
+ # `separator.join(items)`: the method is bound to the separator.
+ separator = inputs.value if isinstance(inputs.value, str) else ""
+ items = inputs.arg(index=0, name="iterable")
+ return _joined_units(items=items, separator=separator, limit=inputs.limit, escaping=inputs.escaping or isinstance(inputs.value, Markup))
+
+
+def _longest_replacement(*, table: Any, limit: int) -> int:
+ if isinstance(table, str):
+ return 1
+ replacements: Iterable[Any]
+ if isinstance(table, Mapping):
+ replacements = cast("Mapping[Any, Any]", table).values()
+ elif isinstance(table, (list, tuple)):
+ replacements = cast("Sequence[Any]", table)
+ else:
+ return ELEMENT_UNITS
+ longest = 1
+ for replacement in replacements:
+ longest = max(longest, text_size(replacement, limit=limit))
+ return longest
+
+
+def _translated(*, inputs: CostInputs) -> int:
+ table = inputs.arg(index=0, name="table")
+ return _length(value=inputs.value) * _longest_replacement(table=table, limit=inputs.limit)
+
+
+def _split_pieces(*, inputs: CostInputs) -> int:
+ # `split` and `rsplit`: every piece is a new string.
+ if not isinstance(inputs.value, str):
+ return _length(value=inputs.value)
+ separator = inputs.arg(index=0, name="sep")
+ max_split = inputs.arg(index=1, name="maxsplit", default=-1)
+ if isinstance(separator, str) and separator:
+ pieces = inputs.value.count(separator) + 1
+ else:
+ pieces = _count_any(text=inputs.value, characters=_ASCII_WHITESPACE if inputs.value.isascii() else _WHITESPACE) + 1
+ if isinstance(max_split, int) and max_split >= 0:
+ pieces = min(pieces, max_split + 1)
+ return len(inputs.value) + ELEMENT_UNITS * pieces
+
+
+def _split_lines(*, inputs: CostInputs) -> int:
+ if not isinstance(inputs.value, str):
+ return _length(value=inputs.value)
+ pieces = _count_any(text=inputs.value, characters=_ASCII_LINE_BREAKS if inputs.value.isascii() else _LINE_BREAKS) + 1
+ return len(inputs.value) + ELEMENT_UNITS * pieces
+
+
+def _stripping_tags_work(*, inputs: CostInputs) -> int:
+ # markupsafe's `striptags` rebuilds the whole string for every comment and every tag it removes, so
+ # this is spent at every call rather than only checked. A value that is not text is converted first,
+ # as the filter converts it.
+ text = _value_text(value=inputs.value, limit=inputs.limit)
+ if text is None:
+ return inputs.limit + 1
+ return len(text) * (1 + text.count("<"))
+
+
+def _formatted_time(*, inputs: CostInputs) -> int:
+ # A `strftime` directive of two characters prints at most a few dozen: `%c` is a whole date.
+ time_format = inputs.arg(index=0, name="format", default="")
+ return STEP_UNITS + 32 * _length(value=time_format)
+
+
+def _set_operands(*, inputs: CostInputs) -> int:
+ # A set operation builds a set from each operand first: a string becomes a set of its characters.
+ return _length(value=inputs.value) * ELEMENT_UNITS + sum(ELEMENT_UNITS * _length(value=operand) for operand in inputs.args)
+
+
+_STR_METHOD_COSTS: Final[dict[str, OperationCost]] = {
+ "capitalize": LINEAR,
+ "casefold": LINEAR,
+ "center": OperationCost(estimate=_padded_width),
+ "count": LINEAR,
+ "endswith": CONSTANT,
+ "expandtabs": OperationCost(estimate=_expanded_tabs),
+ "find": LINEAR,
+ "index": LINEAR,
+ "isalnum": LINEAR,
+ "isalpha": LINEAR,
+ "isascii": CONSTANT,
+ "isdecimal": LINEAR,
+ "isdigit": LINEAR,
+ "isidentifier": LINEAR,
+ "islower": LINEAR,
+ "isnumeric": LINEAR,
+ "isprintable": LINEAR,
+ "isspace": LINEAR,
+ "istitle": LINEAR,
+ "isupper": LINEAR,
+ "join": OperationCost(estimate=_joined, materializes=True),
+ "ljust": OperationCost(estimate=_padded_width),
+ "lower": LINEAR,
+ "lstrip": LINEAR,
+ "partition": LINEAR,
+ "removeprefix": LINEAR,
+ "removesuffix": LINEAR,
+ "replace": OperationCost(estimate=_replaced),
+ "rfind": LINEAR,
+ "rindex": LINEAR,
+ "rjust": OperationCost(estimate=_padded_width),
+ "rpartition": LINEAR,
+ "rsplit": OperationCost(estimate=_split_pieces),
+ "rstrip": LINEAR,
+ "split": OperationCost(estimate=_split_pieces),
+ "splitlines": OperationCost(estimate=_split_lines),
+ "startswith": CONSTANT,
+ "strip": LINEAR,
+ "swapcase": LINEAR,
+ "title": LINEAR,
+ "translate": OperationCost(estimate=_translated),
+ "upper": LINEAR,
+ "zfill": OperationCost(estimate=_padded_width),
+}
+
+_REFUSED_STR_METHODS: Final[dict[str, str]] = {
+ "encode": "it produces bytes, which a template has no use for, and some codecs cost the square of their input",
+ "format": "a string formats through the sandbox's own formatter, never the raw method",
+ "format_map": "a string formats through the sandbox's own formatter, never the raw method",
+}
+
+_INT_METHOD_COSTS: Final[dict[str, OperationCost]] = dict.fromkeys(("as_integer_ratio", "bit_count", "bit_length", "conjugate", "is_integer"), LINEAR)
+
+_DATE_METHOD_COSTS: Final[dict[str, OperationCost]] = {
+ **dict.fromkeys(("ctime", "isocalendar", "isoformat", "isoweekday", "replace", "timetuple", "toordinal", "weekday"), LINEAR),
+ "strftime": OperationCost(estimate=_formatted_time),
+}
+
+_TIME_METHOD_COSTS: Final[dict[str, OperationCost]] = {
+ **dict.fromkeys(("dst", "isoformat", "replace", "tzname", "utcoffset"), LINEAR),
+ "strftime": OperationCost(estimate=_formatted_time),
+}
+
+_SET_OPERATION_COST: Final = OperationCost(estimate=_set_operands)
+
+_FROZENSET_METHOD_COSTS: Final[dict[str, OperationCost]] = {
+ "copy": LINEAR,
+ **dict.fromkeys(("difference", "intersection", "isdisjoint", "issubset", "issuperset", "symmetric_difference", "union"), _SET_OPERATION_COST),
+}
+
+
+def _search_work(*, inputs: CostInputs) -> int:
+ # `count` and `index` compare their argument with every element, as `in` does.
+ needle = inputs.arg(index=0, name="value")
+ return comparison_units(operator="in", left=needle, right=inputs.value, limit=inputs.limit)
+
+
+_SEARCH_COST: Final = OperationCost(work=_search_work)
+
+_MUTATES: Final = "it changes the value in place"
+
+# The methods a template may call on an instance of each plain value type, with their cost. A method
+# absent from its type's table is refused. Class methods and static methods are refused before this
+# table is read, since they are bound to no instance.
+PLAIN_VALUE_METHOD_COSTS: Final[dict[type, dict[str, OperationCost]]] = {
+ str: _STR_METHOD_COSTS,
+ Markup: {**_STR_METHOD_COSTS, "striptags": OperationCost(work=_stripping_tags_work), "unescape": LINEAR},
+ int: _INT_METHOD_COSTS,
+ bool: _INT_METHOD_COSTS,
+ float: dict.fromkeys(("as_integer_ratio", "conjugate", "hex", "is_integer"), LINEAR),
+ # Every `Decimal` operation is bounded by the context's precision.
+ Decimal: dict.fromkeys(
+ (
+ "adjusted",
+ "as_integer_ratio",
+ "as_tuple",
+ "canonical",
+ "compare",
+ "compare_signal",
+ "compare_total",
+ "compare_total_mag",
+ "conjugate",
+ "copy_abs",
+ "copy_negate",
+ "copy_sign",
+ "exp",
+ "fma",
+ "is_canonical",
+ "is_finite",
+ "is_infinite",
+ "is_nan",
+ "is_normal",
+ "is_qnan",
+ "is_signed",
+ "is_snan",
+ "is_subnormal",
+ "is_zero",
+ "ln",
+ "log10",
+ "logb",
+ "logical_and",
+ "logical_invert",
+ "logical_or",
+ "logical_xor",
+ "max",
+ "max_mag",
+ "min",
+ "min_mag",
+ "next_minus",
+ "next_plus",
+ "next_toward",
+ "normalize",
+ "number_class",
+ "quantize",
+ "radix",
+ "remainder_near",
+ "rotate",
+ "same_quantum",
+ "scaleb",
+ "shift",
+ "sqrt",
+ "to_eng_string",
+ "to_integral",
+ "to_integral_exact",
+ "to_integral_value",
+ ),
+ LINEAR,
+ ),
+ date: _DATE_METHOD_COSTS,
+ datetime: {
+ **_DATE_METHOD_COSTS,
+ **dict.fromkeys(("astimezone", "date", "dst", "time", "timestamp", "timetz", "tzname", "utcoffset", "utctimetuple"), LINEAR),
+ },
+ time: _TIME_METHOD_COSTS,
+ timedelta: {"total_seconds": LINEAR},
+ list: {"copy": LINEAR, **dict.fromkeys(("count", "index"), _SEARCH_COST)},
+ tuple: dict.fromkeys(("count", "index"), _SEARCH_COST),
+ dict: {"copy": LINEAR, **dict.fromkeys(("get", "items", "keys", "values"), CONSTANT)},
+ set: _FROZENSET_METHOD_COSTS,
+ frozenset: _FROZENSET_METHOD_COSTS,
+}
+
+# The public instance methods of each plain value type a template may not call, and why. With
+# `PLAIN_VALUE_METHOD_COSTS` they classify every public instance method each type has, which the tests
+# check, so a method a later Python adds is refused until someone decides what it costs.
+REFUSED_PLAIN_VALUE_METHODS: Final[dict[type, dict[str, str]]] = {
+ str: _REFUSED_STR_METHODS,
+ Markup: _REFUSED_STR_METHODS,
+ int: {"to_bytes": "it produces bytes of any length it is asked for"},
+ bool: {"to_bytes": "it produces bytes of any length it is asked for"},
+ list: dict.fromkeys(("append", "clear", "extend", "insert", "pop", "remove", "reverse", "sort"), _MUTATES),
+ dict: dict.fromkeys(("clear", "pop", "popitem", "setdefault", "update"), _MUTATES),
+ set: dict.fromkeys(
+ ("add", "clear", "difference_update", "discard", "intersection_update", "pop", "remove", "symmetric_difference_update", "update"), _MUTATES
+ ),
+}
+
+
+########################################################################################
+# Comparisons
+########################################################################################
+
+
+def comparison_units(*, operator: str, left: Any, right: Any, limit: int) -> int:
+ """Estimate what comparing `left` with `right` costs, for `in`, `not in` and the ordering operators.
+
+ `x in text` scans the text. `x in mapping` hashes `x`. `x in sequence` compares `x` with every
+ element, each comparison costing at most the lighter of the two. An ordering operator or an
+ equality stops at the end of the lighter operand.
+ """
+ if operator in {"in", "notin"}:
+ if isinstance(right, str):
+ return len(right) + (len(left) if isinstance(left, str) else 0)
+ if isinstance(right, (dict, set, frozenset)):
+ return compare_weight(left, limit=limit)
+ if isinstance(right, (list, tuple)):
+ elements = cast("Sequence[Any]", right)
+ left_weight = compare_weight(left, limit=limit)
+ total = 8 * len(elements)
+ weights: dict[int, int] = {}
+ for element in elements:
+ weight = weights.get(id(element))
+ if weight is None:
+ weight = min(compare_weight(element, limit=left_weight), left_weight)
+ weights[id(element)] = weight
+ total += weight
+ if total > limit:
+ return total
+ return total
+ return STEP_UNITS
+ left_weight = compare_weight(left, limit=limit)
+ return min(left_weight, compare_weight(right, limit=left_weight))
+
+
+########################################################################################
+# Filters
+########################################################################################
+
+
+def _text(*, inputs: CostInputs) -> int:
+ return text_size(inputs.value, limit=inputs.limit)
+
+
+def escaped_text_size(value: Any, *, limit: int) -> int:
+ """Estimate the length of `markupsafe.escape(value)`: exact for a string, which is already safe when it is markup."""
+ if isinstance(value, Markup) or hasattr(value, "__html__"):
+ return text_size(value, limit=limit)
+ if isinstance(value, str):
+ return escaped_length(value)
+ return ESCAPE_FACTOR * text_size(value, limit=limit)
+
+
+def _escaped_text(*, inputs: CostInputs) -> int:
+ return escaped_text_size(inputs.value, limit=inputs.limit)
+
+
+def _script_tag_escaped(*, inputs: CostInputs) -> int:
+ # Every `<` becomes `\u003c`; any other value passes through unchanged.
+ return len(inputs.value) + 5 * inputs.value.count("<") if isinstance(inputs.value, str) else 0
+
+
+def _url_encoded_text(*, inputs: CostInputs) -> int:
+ # A character outside ASCII is up to four UTF-8 bytes, each quoted as `%XX`.
+ return 12 * text_size(inputs.value, limit=inputs.limit)
+
+
+def _nesting_depth(*, value: Any, limit: int) -> int:
+ """How deep containers nest under `value`, stopping past `limit`."""
+ deepest = 0
+ pending: list[tuple[Any, int]] = [(value, 1)]
+ seen: set[int] = set()
+ while pending:
+ current, depth = pending.pop()
+ identity = id(current)
+ if identity in seen or not isinstance(current, (list, tuple, dict, set, frozenset)):
+ continue
+ seen.add(identity)
+ deepest = max(deepest, depth)
+ if deepest > limit:
+ return deepest
+ children = cast("dict[Any, Any]", current).values() if isinstance(current, dict) else cast("Collection[Any]", current)
+ pending.extend((child, depth + 1) for child in children)
+ return deepest
+
+
+def _indented_serialisation(*, value: Any, indent: int, limit: int) -> int:
+ # Every line of an indented dump is indented once per level it sits at, and a character outside
+ # ASCII can take twelve.
+ text = text_size(value, limit=limit)
+ return 12 * text + indent * text * _nesting_depth(value=value, limit=limit)
+
+
+def _tojson(*, inputs: CostInputs) -> int:
+ if isinstance(inputs.value, str):
+ return json_string_length(inputs.value)
+ indent = inputs.arg(index=0, name="indent")
+ return _indented_serialisation(value=inputs.value, indent=indent if isinstance(indent, int) and indent > 0 else 0, limit=inputs.limit)
+
+
+def _pprint(*, inputs: CostInputs) -> int:
+ return _indented_serialisation(value=inputs.value, indent=1, limit=inputs.limit)
+
+
+def _materialized(*, inputs: CostInputs) -> int:
+ # A new list with an element per item: a string becomes a list of its characters. A lazy input is
+ # charged element by element as it is drawn.
+ return ELEMENT_UNITS * _length(value=inputs.value)
+
+
+def _reversed(*, inputs: CostInputs) -> int:
+ return _length(value=inputs.value) if isinstance(inputs.value, str) else ELEMENT_UNITS * _length(value=inputs.value)
+
+
+def _centered(*, inputs: CostInputs) -> int:
+ width = inputs.arg(index=0, name="width", default=80)
+ return max(text_size(inputs.value, limit=inputs.limit), width if isinstance(width, int) else 0)
+
+
+def _indented(*, inputs: CostInputs) -> int:
+ width = inputs.arg(index=0, name="width", default=4)
+ indentation = len(width) if isinstance(width, str) else width if isinstance(width, int) else 0
+ text = text_size(inputs.value, limit=inputs.limit)
+ lines = inputs.value.count("\n") + 1 if isinstance(inputs.value, str) else text + 1
+ return text + lines * max(indentation, 0)
+
+
+def _wrap_width(*, inputs: CostInputs) -> int:
+ width = inputs.arg(index=0, name="width", default=79)
+ return max(width, 1) if isinstance(width, int) else 79
+
+
+def _wrapped(*, inputs: CostInputs) -> int:
+ if not isinstance(inputs.value, str):
+ return text_size(inputs.value, limit=inputs.limit)
+ width = _wrap_width(inputs=inputs)
+ wrap_string = inputs.arg(index=2, name="wrapstring")
+ separator_length = len(wrap_string) if isinstance(wrap_string, str) else 1
+ lines = len(inputs.value) // width + inputs.value.count("\n") + 1
+ return len(inputs.value) + lines * separator_length
+
+
+_WORD: Final = re.compile(r"\S+")
+
+
+def _wrapping_work(*, inputs: CostInputs) -> int:
+ # textwrap cuts a word longer than the width one piece at a time, copying the rest each time, so this
+ # is spent at every call rather than only checked.
+ text = inputs.value
+ width = _wrap_width(inputs=inputs)
+ if not isinstance(text, str) or not inputs.arg(index=1, name="break_long_words", default=True) or width >= len(text):
+ return 0
+ total = 0
+ for word in _WORD.finditer(text):
+ word_length = word.end() - word.start()
+ if word_length > width:
+ total += word_length * (word_length // width)
+ if total > inputs.limit:
+ return total
+ return total
+
+
+def _replaced_filter(*, inputs: CostInputs) -> int:
+ # The filter converts its value, `old` and `new` to text before it replaces.
+ text = _value_text(value=inputs.value, limit=inputs.limit)
+ old = _value_text(value=inputs.arg(index=0, name="old"), limit=inputs.limit)
+ new = _value_text(value=inputs.arg(index=1, name="new"), limit=inputs.limit)
+ if text is None or old is None or new is None:
+ return inputs.limit + 1
+ count = inputs.arg(index=2, name="count")
+ return len(text) + _replacement_units(text=text, old=old, new=new, count=count, escaping=inputs.escaping or isinstance(inputs.value, Markup))
+
+
+def _joined_filter(*, inputs: CostInputs) -> int:
+ separator = inputs.arg(index=0, name="d", default="")
+ return _joined_units(items=inputs.value, separator=separator if isinstance(separator, str) else "", limit=inputs.limit, escaping=inputs.escaping)
+
+
+def _formatted_filter(*, inputs: CostInputs) -> int:
+ # The filter converts its value to text, which is the format string.
+ template = _value_text(value=inputs.value, limit=inputs.limit)
+ if template is None:
+ return inputs.limit + 1
+ return percent_format_units(template=template, arguments=inputs.kwargs or inputs.args, limit=inputs.limit, escaping=inputs.escaping)
+
+
+def _batched(*, inputs: CostInputs) -> int:
+ # The last batch is padded up to the batch size when a filler is given.
+ line_count = inputs.arg(index=0, name="linecount", default=0)
+ padding = line_count if isinstance(line_count, int) and inputs.arg(index=1, name="fill_with") is not None else 0
+ return ELEMENT_UNITS * (2 * _length(value=inputs.value) + max(padding, 0))
+
+
+def _sliced(*, inputs: CostInputs) -> int:
+ # One list per slice, each padded by one filler when a filler is given.
+ slices = inputs.arg(index=0, name="slices", default=0)
+ return ELEMENT_UNITS * (_length(value=inputs.value) + 2 * max(slices if isinstance(slices, int) else 0, 0))
+
+
+def _summed_elements(*, inputs: CostInputs) -> int | None:
+ """How many elements summing sequences ends with, or None when the sum is of numbers."""
+ start = inputs.arg(index=1, name="start", default=0)
+ value = inputs.value
+ if isinstance(start, (int, float)) or not isinstance(value, Collection):
+ return None
+ total = _length(value=start)
+ for item in cast("Collection[Any]", value):
+ total += _length(value=item)
+ return total
+
+
+def _summed(*, inputs: CostInputs) -> int:
+ elements = _summed_elements(inputs=inputs)
+ if elements is None:
+ return ELEMENT_UNITS * _length(value=inputs.value)
+ return (8 + ELEMENT_UNITS) * elements
+
+
+def _summing_work(*, inputs: CostInputs) -> int:
+ # Adding sequences copies the running total at every step: the square of their length.
+ elements = _summed_elements(inputs=inputs)
+ if elements is None:
+ return 0
+ return 8 * elements * (_length(value=inputs.value) + 1)
+
+
+def _url_linked(*, inputs: CostInputs) -> int:
+ # Every word may become a link, which repeats the URL and carries the `target` and `rel` attributes.
+ text = text_size(inputs.value, limit=inputs.limit)
+ target = inputs.arg(index=2, name="target")
+ rel = inputs.arg(index=3, name="rel")
+ per_link = ELEMENT_UNITS + ESCAPE_FACTOR * (text_size(target, limit=inputs.limit) + text_size(rel, limit=inputs.limit))
+ return 2 * ESCAPE_FACTOR * text + (text // 2 + 1) * per_link
+
+
+def _counted_words(*, inputs: CostInputs) -> int:
+ # The words are found as a list before they are counted.
+ text = text_size(inputs.value, limit=inputs.limit)
+ return text + ELEMENT_UNITS * (text // 2 + 1)
+
+
+def _attribute_of(*, value: Any, attribute: Any, environment: Any) -> Any:
+ """What `value.attribute` holds, down a dotted path: a dict's item, a list's element or a pydantic
+ model's field, read without running any code, and anything else read the way the filter itself reads
+ it, through the environment's `getitem`, which applies the sandbox's checks. Without an environment, or
+ where nothing is found, the value stands for itself.
+ """
+ current: object = value
+ for part in str(attribute).split("."):
+ key: object = int(part) if part.isdigit() else part
+ found: object = _MISSING
+ if isinstance(current, dict):
+ found = cast("dict[object, object]", current).get(key, _MISSING)
+ elif isinstance(current, (list, tuple)) and isinstance(key, int):
+ sequence = cast("Sequence[object]", current)
+ found = sequence[key] if key < len(sequence) else _MISSING
+ elif isinstance(current, BaseModel):
+ found = vars(current).get(part, _MISSING)
+ elif environment is not None:
+ found = environment.getitem(current, key)
+ if isinstance(found, Undefined):
+ found = _MISSING
+ if found is _MISSING:
+ return cast("object", current)
+ current = found
+ return current
+
+
+_MISSING: Final = object()
+
+
+def _ordering_units(*, items: Any, attribute: Any, case_sensitive: Any, comparisons: Callable[..., int], limit: int, environment: Any = None) -> int:
+ """What ordering `items` costs: `comparisons(n)` comparisons, each weighing the heaviest key.
+
+ Keys that are not compared case-sensitively are lowered first, one copy each.
+ """
+ if not isinstance(items, Collection):
+ return STEP_UNITS
+ collection = cast("Collection[Any]", items)
+ count = len(collection)
+ heaviest = 0
+ lowered = 0
+ weights: dict[int, int] = {}
+ for item in collection:
+ key = item if attribute is None else _attribute_of(value=item, attribute=attribute, environment=environment)
+ weight = weights.get(id(key))
+ if weight is None:
+ weight = compare_weight(key, limit=limit)
+ weights[id(key)] = weight
+ heaviest = max(heaviest, weight)
+ if not case_sensitive and isinstance(key, str):
+ lowered += len(key)
+ if heaviest > limit or lowered > limit:
+ return limit + 1
+ return ELEMENT_UNITS * count + comparisons(count=count) * heaviest + lowered
+
+
+def _sorting_comparisons(*, count: int) -> int:
+ return count * math.ceil(math.log2(count)) if count > 1 else 0
+
+
+def _scanning_comparisons(*, count: int) -> int:
+ return count
+
+
+def _grouping_comparisons(*, count: int) -> int:
+ # A sort by the key, then one comparison per item to find where each group ends.
+ return _sorting_comparisons(count=count) + count
+
+
+def _sorting_work(*, inputs: CostInputs) -> int:
+ # `sort(reverse, case_sensitive, attribute)`
+ return _ordering_units(
+ items=inputs.value,
+ attribute=inputs.arg(index=2, name="attribute"),
+ case_sensitive=inputs.arg(index=1, name="case_sensitive", default=False),
+ comparisons=_sorting_comparisons,
+ limit=inputs.limit,
+ environment=inputs.environment,
+ )
+
+
+def _dict_sorting_work(*, inputs: CostInputs) -> int:
+ # `dictsort(case_sensitive, by, reverse)`: the items are sorted by their key or by their value.
+ value = inputs.value
+ if not isinstance(value, Mapping):
+ return STEP_UNITS
+ mapping = cast("Mapping[Any, Any]", value)
+ by = inputs.arg(index=1, name="by", default="key")
+ return _ordering_units(
+ items=list(mapping.values()) if by == "value" else list(mapping.keys()),
+ attribute=None,
+ case_sensitive=inputs.arg(index=0, name="case_sensitive", default=False),
+ comparisons=_sorting_comparisons,
+ limit=inputs.limit,
+ environment=inputs.environment,
+ )
+
+
+def _grouping_work(*, inputs: CostInputs) -> int:
+ # `groupby(attribute, default, case_sensitive)`: a sort by the attribute, then one comparison per item.
+ return _ordering_units(
+ items=inputs.value,
+ attribute=inputs.arg(index=0, name="attribute"),
+ case_sensitive=inputs.arg(index=2, name="case_sensitive", default=False),
+ comparisons=_grouping_comparisons,
+ limit=inputs.limit,
+ environment=inputs.environment,
+ )
+
+
+def _extremum_work(*, inputs: CostInputs) -> int:
+ # `min(case_sensitive, attribute)` and `max`: one comparison per item.
+ return _ordering_units(
+ items=inputs.value,
+ attribute=inputs.arg(index=1, name="attribute"),
+ case_sensitive=inputs.arg(index=0, name="case_sensitive", default=False),
+ comparisons=_scanning_comparisons,
+ limit=inputs.limit,
+ environment=inputs.environment,
+ )
+
+
+def _deduplicating_work(*, inputs: CostInputs) -> int:
+ # `unique(case_sensitive, attribute)`: every key is lowered unless the case counts, then hashed and
+ # compared with the keys already seen, once each.
+ return _ordering_units(
+ items=inputs.value,
+ attribute=inputs.arg(index=1, name="attribute"),
+ case_sensitive=inputs.arg(index=0, name="case_sensitive", default=False),
+ comparisons=_scanning_comparisons,
+ limit=inputs.limit,
+ environment=inputs.environment,
+ )
+
+
+def _rendered_with_images(*, inputs: CostInputs) -> int:
+ # A list renders each of its items and joins them, so an item repeated a thousand times renders a
+ # thousand times; a single value renders its own data once, and is charged once rendered.
+ value = inputs.value
+ if not isinstance(value, (list, tuple)):
+ return 0
+ memo: dict[int, int] = {}
+ total = 0
+ for item in cast("Sequence[Any]", value):
+ total += ELEMENT_UNITS + text_size(item, limit=inputs.limit, memo=memo)
+ if total > inputs.limit:
+ return total
+ return total
+
+
+_FILTERS: Final = cast("dict[str, Callable[..., Any]]", jinja2.filters.FILTERS)
+
+# A filter that converts its value to text (`str(value)`) before working on it, estimated on that text so
+# that a container is refused before it is converted.
+_CONVERTS_TO_TEXT: Final = OperationCost(estimate=_text)
+
+# The cost of every filter the environment can register, keyed by the filter function itself: Pipelex
+# registers its own `format` under the name Jinja's has.
+FILTER_COSTS: Final[dict[Callable[..., Any], OperationCost]] = {
+ _FILTERS["abs"]: LINEAR,
+ _FILTERS["attr"]: CONSTANT,
+ _FILTERS["batch"]: OperationCost(estimate=_batched),
+ _FILTERS["capitalize"]: _CONVERTS_TO_TEXT,
+ _FILTERS["center"]: OperationCost(estimate=_centered),
+ # `count` is `length`, and `len` reads no element.
+ _FILTERS["count"]: CONSTANT,
+ _FILTERS["default"]: CONSTANT,
+ _FILTERS["dictsort"]: OperationCost(estimate=_materialized, work=_dict_sorting_work),
+ _FILTERS["escape"]: OperationCost(estimate=_escaped_text),
+ _FILTERS["filesizeformat"]: LINEAR,
+ _FILTERS["first"]: CONSTANT,
+ _FILTERS["float"]: LINEAR,
+ _FILTERS["forceescape"]: OperationCost(estimate=_escaped_text),
+ _FILTERS["format"]: OperationCost(estimate=_formatted_filter),
+ _FILTERS["groupby"]: OperationCost(estimate=_materialized, work=_grouping_work, materializes=True),
+ _FILTERS["indent"]: OperationCost(estimate=_indented),
+ _FILTERS["int"]: LINEAR,
+ _FILTERS["items"]: LINEAR,
+ _FILTERS["join"]: OperationCost(estimate=_joined_filter, materializes=True),
+ _FILTERS["last"]: CONSTANT,
+ _FILTERS["list"]: OperationCost(estimate=_materialized),
+ _FILTERS["lower"]: _CONVERTS_TO_TEXT,
+ _FILTERS["map"]: LINEAR,
+ _FILTERS["max"]: OperationCost(work=_extremum_work, materializes=True),
+ _FILTERS["min"]: OperationCost(work=_extremum_work, materializes=True),
+ _FILTERS["pprint"]: OperationCost(estimate=_pprint),
+ _FILTERS["random"]: CONSTANT,
+ _FILTERS["reject"]: LINEAR,
+ _FILTERS["rejectattr"]: LINEAR,
+ _FILTERS["replace"]: OperationCost(estimate=_replaced_filter),
+ _FILTERS["reverse"]: OperationCost(estimate=_reversed),
+ _FILTERS["round"]: LINEAR,
+ _FILTERS["safe"]: OperationCost(estimate=_text),
+ _FILTERS["select"]: LINEAR,
+ _FILTERS["selectattr"]: LINEAR,
+ _FILTERS["slice"]: OperationCost(estimate=_sliced),
+ _FILTERS["sort"]: OperationCost(estimate=_materialized, work=_sorting_work, materializes=True),
+ _FILTERS["string"]: OperationCost(estimate=_text),
+ _FILTERS["striptags"]: OperationCost(estimate=_text, work=_stripping_tags_work),
+ _FILTERS["sum"]: OperationCost(estimate=_summed, work=_summing_work, materializes=True),
+ _FILTERS["title"]: _CONVERTS_TO_TEXT,
+ _FILTERS["tojson"]: OperationCost(estimate=_tojson),
+ _FILTERS["trim"]: _CONVERTS_TO_TEXT,
+ _FILTERS["truncate"]: LINEAR,
+ _FILTERS["unique"]: OperationCost(estimate=_materialized, work=_deduplicating_work, materializes=True),
+ _FILTERS["upper"]: _CONVERTS_TO_TEXT,
+ _FILTERS["urlencode"]: OperationCost(estimate=_url_encoded_text),
+ _FILTERS["urlize"]: OperationCost(estimate=_url_linked),
+ _FILTERS["wordcount"]: OperationCost(estimate=_counted_words),
+ _FILTERS["wordwrap"]: OperationCost(estimate=_wrapped, work=_wrapping_work),
+ _FILTERS["xmlattr"]: OperationCost(estimate=_escaped_text),
+ text_format: LINEAR,
+ tag: _CONVERTS_TO_TEXT,
+ # The conversion charges itself, by its source and its output (`markdown_parser.py`), once the value
+ # is converted to the text it reads.
+ markdown_to_html: _CONVERTS_TO_TEXT,
+ escape_script_tag: OperationCost(estimate=_script_tag_escaped),
+ with_images: OperationCost(estimate=_rendered_with_images),
+}
+
+# The filters whose result is template arithmetic, capped like an operator's.
+INTEGER_FILTERS: Final[frozenset[Callable[..., Any]]] = frozenset({_FILTERS["int"], _FILTERS["sum"]})
+
+
+########################################################################################
+# Tests
+########################################################################################
+
+
+class TestCost(NamedTuple):
+ """How the budget treats a test: a comparison is charged like the operator it spells."""
+
+ comparison: str | None = None
+ scans_text: bool = False
+
+
+_TESTS: Final = cast("dict[str, Callable[..., Any]]", jinja2.tests.TESTS)
+_EQUALITY: Final = TestCost(comparison="eq")
+
+TEST_COSTS: Final[dict[Callable[..., Any], TestCost]] = {
+ **{
+ _TESTS[name]: TestCost()
+ for name in (
+ "boolean",
+ "callable",
+ "defined",
+ "divisibleby",
+ "escaped",
+ "even",
+ "false",
+ "filter",
+ "float",
+ "integer",
+ "iterable",
+ "mapping",
+ "none",
+ "number",
+ "odd",
+ "sameas",
+ "sequence",
+ "string",
+ "test",
+ "true",
+ "undefined",
+ )
+ },
+ **{_TESTS[name]: _EQUALITY for name in ("==", "!=", "<", "<=", ">", ">=")},
+ _TESTS["in"]: TestCost(comparison="in"),
+ _TESTS["lower"]: TestCost(scans_text=True),
+ _TESTS["upper"]: TestCost(scans_text=True),
+}
+
+
+########################################################################################
+# Globals
+########################################################################################
+
+
+def lipsum_units(*, inputs: CostInputs) -> int:
+ """Estimate `lipsum(n, html, min, max)`: `n` paragraphs of up to `max` words of a dozen characters.
+
+ Every paragraph costs a step even when its bounds leave it no word, and the call spends the estimate
+ rather than only checking it, since generating the text is work as well as an allocation.
+ """
+ paragraphs = inputs.arg(index=0, name="n", default=5)
+ fewest = inputs.arg(index=2, name="min", default=20)
+ most = inputs.arg(index=3, name="max", default=100)
+ words = max(fewest if isinstance(fewest, int) else 0, most if isinstance(most, int) else 0, 0)
+ return max(paragraphs if isinstance(paragraphs, int) else 0, 0) * (16 * words + ELEMENT_UNITS)
diff --git a/pipelex/tools/jinja2/jinja2_render_rewrite.py b/pipelex/tools/jinja2/jinja2_render_rewrite.py
new file mode 100644
index 000000000..0b202b408
--- /dev/null
+++ b/pipelex/tools/jinja2/jinja2_render_rewrite.py
@@ -0,0 +1,138 @@
+"""Rewrite a parsed template so that what no sandbox hook reaches is charged to the render's budget.
+
+Six constructs compile to plain Python, with no hook the sandbox could use to charge them:
+
+- iterating a `{% for %}` loop, which an empty loop body makes free;
+- joining with `~`;
+- comparing (`==`, `<`, `in` and the rest);
+- slicing (`value[a:b]`), which bypasses the sandbox's `getitem`;
+- writing out a list, a tuple or a dict (`[a, b]`, `(a, b)`, `{k: v}`), which a loop rebuilds;
+- emitting the template's own static text, which a loop repeats.
+
+`PipelexTemplateEnvironment._generate`, the hook Jinja documents for this, runs the parsed template
+through `rewrite_for_render_budget` before compiling it, so that each goes through one of the charging
+filters of `jinja2_render_charging.py`. Static text becomes a constant marked safe, which the
+environment's `finalize` charges without escaping it. The rewrite changes what is charged, never what
+a template renders.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+from jinja2 import nodes
+from jinja2.visitor import NodeTransformer
+from typing_extensions import override
+
+from pipelex.tools.jinja2.jinja2_render_charging import (
+ COMPARE_FILTER,
+ COMPARED_FILTER,
+ CONCAT_FILTER,
+ ITERATE_FILTER,
+ LITERAL_FILTER,
+ SLICED_FILTER,
+)
+
+if TYPE_CHECKING:
+ from collections.abc import Callable
+
+
+def _charging_filter(*, node: nodes.Node, name: str, args: list[nodes.Expr] | None = None) -> nodes.Filter:
+ return nodes.Filter(node, name, args or [], [], None, None, lineno=node.lineno)
+
+
+def _is_charging_filter(*, node: nodes.Node, name: str) -> bool:
+ return isinstance(node, nodes.Filter) and node.name == name
+
+
+class _RenderBudgetRewriter(NodeTransformer):
+ """Route loops, `~`, comparisons, slices, literals and static text through the budget's charging filters."""
+
+ @override
+ def get_visitor(self, node: nodes.Node) -> Callable[..., Any] | None:
+ match node:
+ case nodes.For():
+ return self._rewrite_loop
+ case nodes.Concat():
+ return self._rewrite_concat
+ case nodes.Compare():
+ return self._rewrite_comparison
+ case nodes.Getitem():
+ return self._rewrite_item
+ case nodes.Filter():
+ return self._rewrite_filter
+ case nodes.Output():
+ return self._rewrite_output
+ case nodes.List() | nodes.Dict():
+ return self._rewrite_literal
+ case nodes.Tuple():
+ # A tuple is also what a `for` loop or a `set` unpacks into, which builds nothing.
+ return self._rewrite_literal if node.ctx == "load" else None
+ case _:
+ return None
+
+ def _rewrite_loop(self, node: nodes.For) -> nodes.Node:
+ self.generic_visit(node)
+ if not _is_charging_filter(node=node.iter, name=ITERATE_FILTER):
+ node.iter = _charging_filter(node=node.iter, name=ITERATE_FILTER)
+ return node
+
+ def _rewrite_concat(self, node: nodes.Concat) -> nodes.Node:
+ self.generic_visit(node)
+ return _charging_filter(node=nodes.List(node.nodes, lineno=node.lineno), name=CONCAT_FILTER)
+
+ def _rewrite_comparison(self, node: nodes.Compare) -> nodes.Node:
+ self.generic_visit(node)
+ if len(node.ops) == 1:
+ operand = node.ops[0]
+ return _charging_filter(node=node.expr, name=COMPARE_FILTER, args=[nodes.Const(operand.op, lineno=node.lineno), operand.expr])
+ # A chained comparison evaluates each operand once and stops at the first false link, so its
+ # operands are charged one by one rather than the comparison as a whole.
+ if not _is_charging_filter(node=node.expr, name=COMPARED_FILTER):
+ node.expr = _charging_filter(node=node.expr, name=COMPARED_FILTER)
+ for operand in node.ops:
+ if not _is_charging_filter(node=operand.expr, name=COMPARED_FILTER):
+ operand.expr = _charging_filter(node=operand.expr, name=COMPARED_FILTER)
+ return node
+
+ def _rewrite_item(self, node: nodes.Getitem) -> nodes.Node:
+ self.generic_visit(node)
+ if isinstance(node.arg, nodes.Slice):
+ return _charging_filter(node=node, name=SLICED_FILTER)
+ return node
+
+ def _rewrite_literal(self, node: nodes.List | nodes.Tuple | nodes.Dict) -> nodes.Node:
+ self.generic_visit(node)
+ return _charging_filter(node=node, name=LITERAL_FILTER)
+
+ def _rewrite_filter(self, node: nodes.Filter) -> nodes.Node:
+ if (node.name == SLICED_FILTER and isinstance(node.node, nodes.Getitem)) or (
+ node.name in {LITERAL_FILTER, CONCAT_FILTER} and isinstance(node.node, (nodes.List, nodes.Tuple, nodes.Dict))
+ ):
+ # A construct already wrapped: visit what it holds, not the construct again.
+ self.generic_visit(node.node)
+ node.args = [self.visit(argument) for argument in node.args]
+ return node
+ self.generic_visit(node)
+ return node
+
+ def _rewrite_output(self, node: nodes.Output) -> nodes.Node:
+ self.generic_visit(node)
+ children: list[Any] = []
+ for child in node.nodes:
+ if isinstance(child, nodes.TemplateData):
+ constant = nodes.Const(child.data, lineno=child.lineno)
+ children.append(nodes.MarkSafeIfAutoescape(constant, lineno=child.lineno))
+ else:
+ children.append(child)
+ node.nodes = children
+ return node
+
+
+def rewrite_for_render_budget(template: nodes.Template) -> nodes.Template:
+ """Rewrite a parsed template in place for the render budget, and return it."""
+ rewritten = _RenderBudgetRewriter().visit(template)
+ if not isinstance(rewritten, nodes.Template):
+ msg = f"Rewriting a template for the render budget returned a '{type(rewritten).__name__}'"
+ raise TypeError(msg)
+ return rewritten
diff --git a/pipelex/tools/jinja2/jinja2_rendering.py b/pipelex/tools/jinja2/jinja2_rendering.py
index 4a3bd8ec2..cff7b9351 100644
--- a/pipelex/tools/jinja2/jinja2_rendering.py
+++ b/pipelex/tools/jinja2/jinja2_rendering.py
@@ -12,6 +12,7 @@ from jinja2.exceptions import (
from pipelex.tools.jinja2.exceptions import (
Jinja2ContextError,
Jinja2StuffError,
+ Jinja2TemplateBudgetError,
Jinja2TemplateRenderError,
Jinja2TemplateSecurityError,
)
@@ -20,6 +21,7 @@ from pipelex.tools.jinja2.jinja2_environment import (
make_jinja2_env_without_loader,
)
from pipelex.tools.jinja2.jinja2_models import Jinja2ContextKey
+from pipelex.tools.jinja2.jinja2_render_budget import RenderBudgetExceededError
from pipelex.tools.jinja2.template_category import TemplateCategory
from pipelex.tools.templating.templating_style import TemplatingStyle
@@ -50,18 +52,17 @@ def _compile_jinja2_template(
jinja2_env = make_jinja2_env_from_registry(
template_category=template_category,
enable_async=enable_async,
+ finalize=finalize,
is_undefined_strict=is_undefined_strict,
)
else:
jinja2_env = make_jinja2_env_without_loader(
template_category=template_category,
enable_async=enable_async,
+ finalize=finalize,
is_undefined_strict=is_undefined_strict,
)
- if finalize is not None:
- jinja2_env.finalize = finalize
-
try:
return jinja2_env.from_string(template_source)
except TemplateAssertionError as exc:
@@ -105,6 +106,16 @@ def _make_security_error_msg(*, security_error: SecurityError) -> str:
return f"Jinja2 render — refused by the template sandbox: {security_error}"
+# A template nesting macro calls or recursive loops deep enough reaches Python's recursion limit before
+# its budget, which is the same overspending by another name.
+_RECURSION_MSG = "Jinja2 render — refused by the render budget: the template nests calls deeper than Python allows."
+
+
+def _make_budget_error_msg(*, budget_error: RenderBudgetExceededError) -> str:
+ # Like a refusal, the message names the operation and its size, never the template source or a value.
+ return f"Jinja2 render — refused by the render budget: {budget_error}"
+
+
def _make_non_type_error_msg(
template_source: str,
*,
@@ -119,6 +130,10 @@ def _render_template_sync(template_source: str, *, template: _Jinja2Template, te
generated_text: str = template.render(**templating_context)
except SecurityError as exc:
raise Jinja2TemplateSecurityError(_make_security_error_msg(security_error=exc)) from exc
+ except RenderBudgetExceededError as exc:
+ raise Jinja2TemplateBudgetError(_make_budget_error_msg(budget_error=exc)) from exc
+ except RecursionError as exc:
+ raise Jinja2TemplateBudgetError(_RECURSION_MSG) from exc
except Jinja2StuffError as exc:
msg = _make_non_type_error_msg(template_source=template_source, error_label="stuff error", error=exc)
raise Jinja2TemplateRenderError(msg) from exc
@@ -150,6 +165,10 @@ async def _render_template_async(template_source: str, *, template: _Jinja2Templ
generated_text: str = await template.render_async(**templating_context)
except SecurityError as exc:
raise Jinja2TemplateSecurityError(_make_security_error_msg(security_error=exc)) from exc
+ except RenderBudgetExceededError as exc:
+ raise Jinja2TemplateBudgetError(_make_budget_error_msg(budget_error=exc)) from exc
+ except RecursionError as exc:
+ raise Jinja2TemplateBudgetError(_RECURSION_MSG) from exc
except Jinja2StuffError as exc:
msg = _make_non_type_error_msg(template_source=template_source, error_label="stuff error", error=exc)
raise Jinja2TemplateRenderError(msg) from exc
diff --git a/pipelex/tools/jinja2/jinja2_sandbox.py b/pipelex/tools/jinja2/jinja2_sandbox.py
index e05391a63..4008047e0 100644
--- a/pipelex/tools/jinja2/jinja2_sandbox.py
+++ b/pipelex/tools/jinja2/jinja2_sandbox.py
@@ -16,14 +16,22 @@ forge content, such as an image pointing at another organisation's storage key.
are data, and a key it does not hold falls back to an attribute read, which the rule above refuses.
Jinja's own refusals (a function's globals, a frame, a class's `mro`) stay in force.
- **Calling.** A template calls Jinja's own runtime (macros, `caller`, `loop`, `cycler`, `joiner`,
- block references and the environment's globals), methods bound to an instance of a plain value type
- (`str`, numbers, dates and times, and the built-in containers), `str.format` through the sandbox's
- safe formatter, and the methods a type declares in its template surface. Everything else is
- refused: pydantic methods, `Stuff` and content methods, classes, free functions and class methods,
- and every method that changes a list, a dict or a set in place.
+ block references and the environment's globals), the methods of a plain value type (`str`, numbers,
+ dates and times, and the built-in containers) that the budget's table lists for that type
+ (`PLAIN_VALUE_METHOD_COSTS` in `jinja2_render_costs.py`), `str.format` through the sandbox's safe
+ formatter, and the methods a type declares in its template surface. Everything else is refused:
+ pydantic methods, `Stuff` and content methods, classes, free functions and class methods, every
+ method that changes a list, a dict or a set in place, and the few that produce bytes.
- **Refusing.** A refusal raises `jinja2.exceptions.SecurityError` at the point of access, which the
render functions turn into `Jinja2TemplateSecurityError`. Jinja's stock sandbox returns an undefined
value that prints as nothing, which would send a prompt with a hole in it.
+- **Spending.** What a template may reach says nothing about what it may spend, so every render also
+ spends from one budget (`jinja2_render_budget.py`), charged by the hooks of this class, by wrappers
+ around every filter and test, and by a rewrite of the parsed template for what no hook reaches
+ (`jinja2_render_charging.py`, `jinja2_render_rewrite.py`). Converting Markdown, which markupsafe
+ does through `__html__` where no hook sees it, charges the budget the template makes active while it
+ renders (`markdown_parser.py`). An overdraft raises `RenderBudgetExceededError`, which the render
+ functions turn into `Jinja2TemplateBudgetError`.
Filters and tests are not calls in this sense: they are Pipelex's or Jinja's own code, registered by
the environment, and Jinja invokes them directly. That makes every filter trusted code with one
@@ -45,7 +53,7 @@ import inspect
import types
from datetime import date, datetime, time, timedelta
from decimal import Decimal
-from typing import TYPE_CHECKING, Any, NoReturn
+from typing import TYPE_CHECKING, Any, NoReturn, cast
from jinja2.exceptions import SecurityError
from jinja2.runtime import BlockReference, Context, LoopContext, Macro, Undefined
@@ -54,11 +62,26 @@ from jinja2.utils import Cycler, Joiner, Namespace, generate_lorem_ipsum
from markupsafe import Markup
from typing_extensions import override
+from pipelex.tools.jinja2.jinja2_render_budget import DEFAULT_RENDER_BUDGET_UNITS, active_render_budget, compare_weight
+from pipelex.tools.jinja2.jinja2_render_charging import (
+ INTERNAL_FILTERS,
+ BudgetedContext,
+ BudgetedTemplate,
+ SandboxedStrFormat,
+ charge_registered_functions,
+ charged_binop,
+ charged_call,
+ make_charged_finalize,
+)
+from pipelex.tools.jinja2.jinja2_render_costs import PLAIN_VALUE_METHOD_COSTS
+from pipelex.tools.jinja2.jinja2_render_rewrite import rewrite_for_render_budget
from pipelex.tools.jinja2.template_surface import get_template_surface
if TYPE_CHECKING:
from collections.abc import Callable
+ from jinja2 import nodes
+
# The types whose own methods a template may call on an instance: formatting, not program logic.
_PLAIN_VALUE_TYPES: tuple[type, ...] = (
str,
@@ -120,36 +143,9 @@ _JINJA_CALLABLE_INSTANCE_TYPES: tuple[type, ...] = (Macro, LoopContext, BlockRef
# `cycler.next()`, `cycler.reset()`.
_JINJA_METHOD_OWNER_TYPES: tuple[type, ...] = (LoopContext, Cycler, BlockReference)
-# The methods that change a mutable plain value in place, listed here rather than taken from Jinja's
-# `modifies_known_mutable`, whose list misses `set.intersection_update`. The test suite checks this list
-# against every public method of each type, so a method a later Python adds cannot slip through.
-_MUTATING_METHOD_NAMES: dict[type, frozenset[str]] = {
- list: frozenset({"append", "clear", "extend", "insert", "pop", "remove", "reverse", "sort"}),
- dict: frozenset({"clear", "pop", "popitem", "setdefault", "update"}),
- set: frozenset({"add", "clear", "difference_update", "discard", "intersection_update", "pop", "remove", "symmetric_difference_update", "update"}),
-}
-
_MISSING = object()
-class _SandboxedStrFormat:
- """The `str.format` or `str.format_map` of a string, routed through the sandbox's safe formatter.
-
- Jinja builds this wrapper when a template reads `format` on a string, so that the replacement
- fields of the format string (`{0.name}`, `{0[key]}`) go through the environment's own attribute
- and item checks. Wrapping it in a type of its own lets the call policy recognise it by type,
- while the raw `str.format` stays refused.
- """
-
- __slots__ = ("_format",)
-
- def __init__(self, format_function: Callable[..., str]) -> None:
- self._format = format_function
-
- def __call__(self, *args: Any, **kwargs: Any) -> str:
- return self._format(*args, **kwargs)
-
-
def _is_private_name(*, name: str) -> bool:
return name.startswith("_")
@@ -175,7 +171,9 @@ def _is_plain_value_method(*, obj: object) -> bool:
The binding has to be to an instance, so a class method reached through an instance
(`created_at.now()`, which reads the clock) is refused. The method has to be the plain type's own,
- so a subclass (a `StrEnum` member is a `str`) cannot add a callable method or override one.
+ so a subclass (a `StrEnum` member is a `str`) cannot add a callable method or override one, and it
+ has to be listed for that type in `PLAIN_VALUE_METHOD_COSTS`, which leaves out the methods that
+ change a value in place and those whose cost the render budget cannot bound.
"""
if not isinstance(obj, (types.BuiltinMethodType, types.MethodType)):
return False
@@ -186,10 +184,9 @@ def _is_plain_value_method(*, obj: object) -> bool:
if plain_type is None:
return False
name = obj.__name__
- if isinstance(bound_to, str) and name in {"format", "format_map"}:
- # Only the sandbox's own wrapper formats a string: the raw method would resolve `{0.__class__}`.
- return False
- if name in _MUTATING_METHOD_NAMES.get(plain_type, frozenset()):
+ # `format` and `format_map` are not listed: only the sandbox's own wrapper formats a string, since
+ # the raw method would resolve `{0.__class__}`.
+ if name not in PLAIN_VALUE_METHOD_COSTS[plain_type]:
return False
plain_member = inspect.getattr_static(plain_type, name, _MISSING)
return plain_member is not _MISSING and inspect.getattr_static(type(bound_to), name, _MISSING) is plain_member
@@ -218,6 +215,8 @@ def _describe_callable(*, obj: object) -> str:
if bound_to is None or isinstance(bound_to, types.ModuleType):
return f"the function '{name}'"
return f"the method '{name}' of a '{type(bound_to).__name__}' value"
+ if isinstance(obj, Macro):
+ return f"the macro '{obj.name}'"
if isinstance(obj, type):
return f"the class '{obj.__name__}'"
if isinstance(obj, types.FunctionType):
@@ -226,11 +225,38 @@ def _describe_callable(*, obj: object) -> str:
class PipelexTemplateEnvironment(ImmutableSandboxedEnvironment):
- """The Jinja2 environment every Pipelex template renders under. The module docstring states its policy."""
+ """The Jinja2 environment every Pipelex template renders under. The module docstring states its policy.
- def __init__(self, *args: Any, **kwargs: Any) -> None:
- super().__init__(*args, **kwargs)
+ `render_budget_units` is the budget each render gets. A caller's `finalize` is passed here, never
+ assigned afterwards, since the environment's own `finalize` wraps it to charge what is printed.
+ """
+
+ context_class = BudgetedContext
+ template_class = BudgetedTemplate
+ intercepted_binops = frozenset({"+", "-", "*", "/", "//", "%", "**"})
+
+ def __init__(
+ self,
+ *args: Any,
+ finalize: Callable[..., Any] | None = None,
+ render_budget_units: int = DEFAULT_RENDER_BUDGET_UNITS,
+ **kwargs: Any,
+ ) -> None:
+ self.render_budget_units = render_budget_units
+ self._charged_finalize = make_charged_finalize(inner=finalize)
+ super().__init__(*args, finalize=self._charged_finalize, **kwargs)
self.globals["namespace"] = DataNamespace
+ self.filters.update(INTERNAL_FILTERS)
+
+ @override
+ def _generate(self, source: nodes.Template, name: str | None, filename: str | None, defer_init: bool = False) -> str:
+ if self.finalize is not self._charged_finalize:
+ msg = "A PipelexTemplateEnvironment takes its finalize at construction: one assigned afterwards would print uncharged."
+ raise RuntimeError(msg)
+ charge_registered_functions(filters=self.filters, tests=cast("dict[str, Callable[..., Any]]", self.tests))
+ rewritten = rewrite_for_render_budget(source)
+ rewritten.set_environment(self)
+ return super()._generate(rewritten, name, filename, defer_init=defer_init)
@override
def is_safe_attribute(self, obj: Any, attr: str, value: Any) -> bool:
@@ -243,7 +269,7 @@ class PipelexTemplateEnvironment(ImmutableSandboxedEnvironment):
if not super().is_safe_callable(obj):
return False
return (
- isinstance(obj, _SandboxedStrFormat)
+ isinstance(obj, SandboxedStrFormat)
or _is_jinja_runtime_callable(obj=obj)
or _is_plain_value_method(obj=obj)
or _is_declared_template_method(obj=obj)
@@ -271,6 +297,12 @@ class PipelexTemplateEnvironment(ImmutableSandboxedEnvironment):
and not _declares_private_name(obj=obj, name=argument)
):
self.unsafe_undefined(obj, argument)
+ key: object = argument
+ if isinstance(key, tuple):
+ # Reading with a tuple key hashes every element of it again, nested ones included.
+ budget = active_render_budget()
+ if budget is not None:
+ budget.charge(units=compare_weight(key, limit=budget.remaining), operation="reading an item by a tuple key")
return super().getitem(obj, argument)
@override
@@ -283,11 +315,17 @@ class PipelexTemplateEnvironment(ImmutableSandboxedEnvironment):
format_function = super().wrap_str_format(value)
if format_function is None:
return None
- return _SandboxedStrFormat(format_function)
+ return SandboxedStrFormat(
+ format_function=format_function, environment=self, template=value.__self__, is_format_map=value.__name__ == "format_map"
+ )
@override
def call(self, context: Context, obj: Any, /, *args: Any, **kwargs: Any) -> Any:
if not self.is_safe_callable(obj):
msg = f"A template may not call {_describe_callable(obj=obj)}: templates read data and call methods of plain values only."
raise SecurityError(msg)
- return context.call(obj, *args, **kwargs)
+ return charged_call(context=context, obj=obj, args=args, kwargs=kwargs, operation=lambda: f"calling {_describe_callable(obj=obj)}")
+
+ @override
+ def call_binop(self, context: Context, operator: str, left: Any, right: Any) -> Any:
+ return charged_binop(context=context, operator=operator, left=left, right=right, compute=self.binop_table[operator])
diff --git a/pipelex/tools/markdown/markdown_parser.py b/pipelex/tools/markdown/markdown_parser.py
index e075c5bce..ed8adfddb 100644
--- a/pipelex/tools/markdown/markdown_parser.py
+++ b/pipelex/tools/markdown/markdown_parser.py
@@ -8,12 +8,50 @@ read Markdown an LLM or a person wrote, so they parse it the same way, here:
- bare URLs linked, but only those with a scheme: markdown-it's default fuzzy matching links `README.md` to
`http://README.md` and `config.py` to `http://config.py`, `.md` and `.py` being country domains, so fuzzy
links and fuzzy emails are off, and `www.` addresses and bare emails stay text.
+
+The parser joins the text fragments emphasis leaves behind in one pass (`_join_text_fragments`), where
+markdown-it's own rule holds a copy of every prefix of a run of fragments at once: 64,000 characters of
+unclosed `**a ` peaked at 1.5 GB with it and at 19 MB with this one, for the same tokens.
+
+A conversion to HTML that a template sets off is charged to that template's render budget
+(`jinja2_render_budget.py`), wherever it happens: the `markdown` filter, and markupsafe's `__html__`, which
+converts a Markdown value whenever an HTML template prints, joins or formats one. It costs far more than the
+bytes it produces, and can produce far more than it reads, so it is charged before it parses, for every
+character of its source and every cell of its tables (`table_cells_bound`), since the table rule pads a short
+row with empty cells; and its output, bounded from the parsed tokens (`html_length_bound`), must fit what is
+left before it renders, since a reference link's destination and title are written once and printed at
+every link that uses the reference.
"""
+import re
+from collections.abc import Sequence
from functools import cache
+from typing import TYPE_CHECKING, Any, Final
from markdown_it import MarkdownIt
+from pipelex.tools.jinja2.jinja2_render_budget import MARKDOWN_UNITS_PER_CHARACTER, active_render_budget
+
+if TYPE_CHECKING:
+ from markdown_it.rules_inline import StateInline
+ from markdown_it.token import Token
+
+_CONVERTING: Final = "converting Markdown to HTML"
+
+# What the renderer writes around one token at most, besides its tag name: `<`, `</`, `>`, ` /`, a line
+# feed, and for a code block `<pre><code class="language-">` and `</code></pre>`.
+_TOKEN_MARKUP_UNITS: Final = 48
+
+# The factor by which escaping can grow a text: `"` becomes `&quot;`.
+_ESCAPE_FACTOR: Final = 6
+
+# markdown-it's line endings (its `normalize` rule), and nothing else: `str.splitlines` would also break at
+# characters markdown-it keeps inside a line.
+_LINE_ENDING: Final = re.compile(r"\r\n?|\n")
+
+# What a table's delimiter row is made of, once its container's indentation and `>` markers are left out.
+_DELIMITER_ROW_CHARACTERS: Final = frozenset("|-: \t")
+
@cache
def get_markdown_parser() -> MarkdownIt:
@@ -24,10 +62,119 @@ def get_markdown_parser() -> MarkdownIt:
msg = "markdown-it-py found no linkify-it-py: Pipelex requires markdown-it-py[linkify]."
raise RuntimeError(msg)
linkify.set({"fuzzy_link": False, "fuzzy_email": False}) # pyright: ignore[reportUnknownMemberType]
+ parser.inline.ruler2.at("fragments_join", _join_text_fragments)
return parser
+def _join_text_fragments(state: "StateInline") -> None:
+ """markdown-it's `fragments_join`, joining each run of adjacent text tokens once instead of pair by pair.
+
+ Emphasis and strikethrough leave each unmatched delimiter as a text token of its own, which this joins
+ with the text around it, and it recalculates every token's level, since emphasis turned some text tokens
+ into tags. The last token of each run keeps the run's text, as with markdown-it's rule, which joins a run
+ pair by pair and leaves every token of it holding the text joined so far until the run is dropped.
+ """
+ tokens = state.tokens
+ count = len(tokens)
+ kept: list[Token] = []
+ run: list[str] = []
+ level = 0
+ for index, token in enumerate(tokens):
+ if token.nesting < 0:
+ level -= 1
+ token.level = level
+ if token.nesting > 0:
+ level += 1
+ if token.type == "text" and index + 1 < count and tokens[index + 1].type == "text":
+ run.append(token.content)
+ continue
+ if run:
+ run.append(token.content)
+ token.content = "".join(run)
+ run.clear()
+ kept.append(token)
+ tokens[:] = kept
+
+
def render_markdown_as_html(markdown_text: str) -> str:
- """Markdown as an HTML fragment, with any raw HTML in the source escaped."""
- html: str = get_markdown_parser().render(markdown_text)
- return html
+ """Markdown as an HTML fragment, with any raw HTML in the source escaped.
+
+ Inside a template render, the conversion is charged to the render's budget, which refuses it with
+ `RenderBudgetExceededError` before it parses a source, or renders an output, larger than what is left.
+ """
+ parser = get_markdown_parser()
+ budget = active_render_budget()
+ if budget is None:
+ html: str = parser.render(markdown_text)
+ return html
+ # The length first: the table scan reads every line, so it runs only once the source is affordable.
+ budget.charge(units=MARKDOWN_UNITS_PER_CHARACTER * len(markdown_text), operation=_CONVERTING)
+ budget.charge(units=MARKDOWN_UNITS_PER_CHARACTER * table_cells_bound(markdown_text=markdown_text), operation=_CONVERTING)
+ env: dict[str, Any] = {}
+ tokens = parser.parse(markdown_text, env)
+ budget.afford(units=html_length_bound(tokens=tokens), operation=_CONVERTING)
+ rendered: str = parser.renderer.render(tokens, parser.options, env)
+ budget.charge(units=len(rendered), operation=_CONVERTING)
+ return rendered
+
+
+def table_cells_bound(*, markdown_text: str) -> int:
+ """At most the number of cells the tables of `markdown_text` parse into, taken from its lines.
+
+ The table rule gives every row as many cells as its header, padding a short row with empty ones, so a
+ table costs the parser its cells rather than its characters: a header of two hundred columns over three
+ hundred one-character rows makes sixty thousand cells, about sixty megabytes of tokens, out of two
+ kilobytes of text. A table needs a delimiter row, a line of `|`, `-`, `:`, spaces and tabs under its
+ header once its container's indentation and `>` markers are left out, with at most one column more than
+ it has `|`; and its rows end at the latest at the next blank line.
+ """
+ total = 0
+ # Read from the bottom: the lines under the current one, down to the next blank line.
+ lines_below = 0
+ for line in reversed(_LINE_ENDING.split(markdown_text)):
+ if not line.strip():
+ lines_below = 0
+ continue
+ content = line.lstrip(" \t>")
+ if content[:1] in {"|", "-", ":"} and "-" in content and _DELIMITER_ROW_CHARACTERS.issuperset(content):
+ # The header row, and every row down to the next blank line.
+ total += (content.count("|") + 1) * (lines_below + 1)
+ lines_below += 1
+ return total
+
+
+def html_length_bound(*, tokens: Sequence["Token"]) -> int:
+ """At most the length of the HTML the renderer makes of `tokens`, taken without rendering them.
+
+ Every token counts its tag, its attributes and its content as if all were escaped, and an image counts
+ once more the text of the tokens it holds, which becomes its `alt`. A token holds a reference link's
+ destination and title as it prints them, so a reference used ten thousand times is counted ten thousand
+ times, while the parsed tokens share one copy of it.
+ """
+ total = 0
+ pending: list[Sequence[Token]] = [tokens]
+ while pending:
+ for token in pending.pop():
+ total += _TOKEN_MARKUP_UNITS + 2 * len(token.tag) + _ESCAPE_FACTOR * (len(token.content) + len(token.info))
+ for key, value in token.attrItems():
+ total += len(key) + 4 + _ESCAPE_FACTOR * len(str(value))
+ if token.children:
+ pending.append(token.children)
+ if token.type == "image":
+ total += _ESCAPE_FACTOR * _inline_text_length(tokens=token.children)
+ return total
+
+
+def _inline_text_length(*, tokens: Sequence["Token"]) -> int:
+ """The length of the text an image's `alt` is made of: its text, and that of the images inside it."""
+ total = 0
+ pending: list[Sequence[Token]] = [tokens]
+ while pending:
+ for token in pending.pop():
+ if token.type == "text":
+ total += len(token.content)
+ elif token.type == "softbreak":
+ total += 1
+ elif token.type == "image" and token.children:
+ pending.append(token.children)
+ return total
diff --git a/pyproject.toml b/pyproject.toml
index 75863d76b..35a8caba9 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -27,7 +27,7 @@ dependencies = [
"httpx>=0.23.0,<1.0.0",
"idna>=3.15",
"instructor>=1.17.0,<2.0.0", # 1.16+ supports mistralai 2.x; 2.0 is a planned major we have not assessed
- "jinja2>=3.1.6",
+ "jinja2>=3.1.6,<3.2",
"json2html>=1.3.0",
"kajson==0.7.1",
"markdown-it-py[linkify]>=3.0.0", # the Markdown concept, the `markdown` filter and the built-in PDF engine
diff --git a/subject_grants.toml b/subject_grants.toml
index 1a09c2150..857b7e3aa 100644
--- a/subject_grants.toml
+++ b/subject_grants.toml
@@ -6100,6 +6100,90 @@ rationale = "Noun-named derivation: the full dotted path of the node; single ope
param = "template_source"
rationale = "Jinja op on the template_source; single operand (context keyword-only)."
+["pipelex/tools/jinja2/jinja2_render_budget.py::_compare_container_units"]
+param = "value"
+rationale = "Positional-Callable walk callback: the walk passes each container positionally."
+
+["pipelex/tools/jinja2/jinja2_render_budget.py::_compare_leaf_units"]
+param = "value"
+rationale = "Positional-Callable walk callback: the walk passes each element positionally."
+
+["pipelex/tools/jinja2/jinja2_render_budget.py::_repr_container_size"]
+param = "value"
+rationale = "Positional-Callable walk callback: the walk passes each container positionally."
+
+["pipelex/tools/jinja2/jinja2_render_budget.py::_repr_leaf_size"]
+param = "value"
+rationale = "Positional-Callable walk callback: the walk passes each element positionally."
+
+["pipelex/tools/jinja2/jinja2_render_budget.py::compare_weight"]
+param = "value"
+rationale = "Estimates what comparing the value costs; the limit stays keyword."
+
+["pipelex/tools/jinja2/jinja2_render_budget.py::escaped_length"]
+param = "text"
+rationale = "Measures the text it is given."
+
+["pipelex/tools/jinja2/jinja2_render_budget.py::is_lazy"]
+param = "value"
+rationale = "Predicate on the value it is given."
+
+["pipelex/tools/jinja2/jinja2_render_budget.py::json_string_length"]
+param = "text"
+rationale = "Measures the text it is given."
+
+["pipelex/tools/jinja2/jinja2_render_budget.py::produced_size"]
+param = "value"
+rationale = "Measures the value an operation produced."
+
+["pipelex/tools/jinja2/jinja2_render_budget.py::size_of"]
+param = "value"
+rationale = "Measures the value it is given; a hot path of every render."
+
+["pipelex/tools/jinja2/jinja2_render_budget.py::text_size"]
+param = "value"
+rationale = "Estimates the printed length of the value; the limit stays keyword."
+
+["pipelex/tools/jinja2/jinja2_render_charging.py::render_budget_of"]
+param = "context"
+rationale = "Reads the budget off the render context it is given."
+
+["pipelex/tools/jinja2/jinja2_render_costs.py::escaped_text_size"]
+param = "value"
+rationale = "Estimates the escaped length of the value, also passed as a positional measure; the limit stays keyword."
+
+["pipelex/tools/jinja2/jinja2_render_rewrite.py::_RenderBudgetRewriter._rewrite_comparison"]
+param = "node"
+rationale = "Jinja NodeVisitor dispatch: visit() calls the visitor get_visitor returns with the node positionally."
+
+["pipelex/tools/jinja2/jinja2_render_rewrite.py::_RenderBudgetRewriter._rewrite_concat"]
+param = "node"
+rationale = "Jinja NodeVisitor dispatch: visit() calls the visitor get_visitor returns with the node positionally."
+
+["pipelex/tools/jinja2/jinja2_render_rewrite.py::_RenderBudgetRewriter._rewrite_filter"]
+param = "node"
+rationale = "Jinja NodeVisitor dispatch: visit() calls the visitor get_visitor returns with the node positionally."
+
+["pipelex/tools/jinja2/jinja2_render_rewrite.py::_RenderBudgetRewriter._rewrite_item"]
+param = "node"
+rationale = "Jinja NodeVisitor dispatch: visit() calls the visitor get_visitor returns with the node positionally."
+
+["pipelex/tools/jinja2/jinja2_render_rewrite.py::_RenderBudgetRewriter._rewrite_literal"]
+param = "node"
+rationale = "Jinja NodeVisitor dispatch: visit() calls the visitor get_visitor returns with the node positionally."
+
+["pipelex/tools/jinja2/jinja2_render_rewrite.py::_RenderBudgetRewriter._rewrite_loop"]
+param = "node"
+rationale = "Jinja NodeVisitor dispatch: visit() calls the visitor get_visitor returns with the node positionally."
+
+["pipelex/tools/jinja2/jinja2_render_rewrite.py::_RenderBudgetRewriter._rewrite_output"]
+param = "node"
+rationale = "Jinja NodeVisitor dispatch: visit() calls the visitor get_visitor returns with the node positionally."
+
+["pipelex/tools/jinja2/jinja2_render_rewrite.py::rewrite_for_render_budget"]
+param = "template"
+rationale = "Rewrites the parsed template it is given."
+
["pipelex/tools/jinja2/jinja2_rendering.py::_add_to_templating_context"]
param = "templating_context"
rationale = "Transforms the templating_context; single operand."
@@ -6248,6 +6332,10 @@ rationale = "Logs the content; single operand (formatting keyword-only)."
param = "channel_name"
rationale = "Renders the channel_name; single operand."
+["pipelex/tools/markdown/markdown_parser.py::_join_text_fragments"]
+param = "state"
+rationale = "markdown-it rule callback: the inline ruler calls each rule with its state positionally."
+
["pipelex/tools/markdown/markdown_parser.py::render_markdown_as_html"]
param = "markdown_text"
rationale = "Renders one Markdown text as HTML."
diff --git a/tests/data/errors/error_identity.txt b/tests/data/errors/error_identity.txt
index 3185abcf0..ee8763104 100644
--- a/tests/data/errors/error_identity.txt
+++ b/tests/data/errors/error_identity.txt
@@ -143,6 +143,7 @@ InputStuffSpecsFactoryError | Input stuff specs factory | https://docs.pipelex.c
Jinja2ContextError | Jinja 2 context | https://docs.pipelex.com/latest/errors/jinja2-context-error/
Jinja2DetectVariablesError | Jinja 2 detect variables | https://docs.pipelex.com/latest/errors/jinja2-detect-variables-error/
Jinja2StuffError | Jinja 2 stuff | https://docs.pipelex.com/latest/errors/jinja2-stuff-error/
+Jinja2TemplateBudgetError | Template render budget exceeded | https://docs.pipelex.com/latest/errors/jinja2-template-budget-error/
Jinja2TemplateRenderError | Jinja 2 template render | https://docs.pipelex.com/latest/errors/jinja2-template-render-error/
Jinja2TemplateSecurityError | Jinja 2 template security | https://docs.pipelex.com/latest/errors/jinja2-template-security-error/
Jinja2TemplateSyntaxError | Jinja 2 template syntax | https://docs.pipelex.com/latest/errors/jinja2-template-syntax-error/
diff --git a/tests/unit/pipelex/tools/markdown/test_markdown_parser.py b/tests/unit/pipelex/tools/markdown/test_markdown_parser.py
index d6632309a..5a3c2a5ab 100644
--- a/tests/unit/pipelex/tools/markdown/test_markdown_parser.py
+++ b/tests/unit/pipelex/tools/markdown/test_markdown_parser.py
@@ -1,4 +1,8 @@
-from pipelex.tools.markdown.markdown_parser import render_markdown_as_html
+import random
+
+from markdown_it import MarkdownIt
+
+from pipelex.tools.markdown.markdown_parser import get_markdown_parser, html_length_bound, render_markdown_as_html, table_cells_bound
class TestMarkdownParser:
@@ -12,3 +16,96 @@ class TestMarkdownParser:
html = render_markdown_as_html("| a | b |\n| - | - |\n| 1 | 2 |\n\n~~gone~~")
assert "<table>" in html
assert "<s>gone</s>" in html
+
+
+# The pieces the generated documents are made of: emphasis and its unmatched delimiters, links, references and
+# images, tables in and out of containers, padded rows, code, escapes, and every line ending markdown-it knows,
+# with characters `str.splitlines` would break at and markdown-it does not.
+_PIECES = [
+ "*",
+ "_",
+ "~~",
+ "**",
+ "a",
+ " ",
+ "\n",
+ "\r",
+ "\r\n",
+ "\n\n",
+ "[",
+ "]",
+ "(",
+ ")",
+ "`",
+ "\\*",
+ "&",
+ "<",
+ '"',
+ 'https://a.co/&"x',
+ "|",
+ "| ",
+ "-",
+ "-|",
+ "|-|",
+ ":-",
+ "# ",
+ "> ",
+ ">",
+ " ",
+ "\t",
+ "1. ",
+ "- ",
+ "![",
+ "](/i.png)",
+ "]: /u",
+ "[r]",
+ "[r][]",
+ '\n[r]: /é&"<x> "t&"\n',
+ "```py&\n",
+ "
",
+ "\x85",
+ "|a|b|c|\n|-|:-:|-:|\n|1\n",
+ "> |a|b|\n> |-|-|\n> |c\n",
+ "- |a|b|\n |-|-|\n |c\n",
+ "![![x](/in)](/out)",
+]
+
+
+def _generated_documents() -> list[str]:
+ generator = random.Random(20261002)
+ return ["".join(generator.choice(_PIECES) for _ in range(generator.randint(1, 120))) for _ in range(1000)]
+
+
+def _stock_parser() -> MarkdownIt:
+ parser = MarkdownIt("commonmark", {"html": False, "linkify": True}).enable(["table", "strikethrough", "linkify"])
+ linkify = parser.linkify
+ assert linkify is not None
+ linkify.set({"fuzzy_link": False, "fuzzy_email": False}) # pyright: ignore[reportUnknownMemberType]
+ return parser
+
+
+class TestMarkdownParserBounds:
+ def test_joining_fragments_renders_as_markdown_it_does(self) -> None:
+ stock = _stock_parser()
+ parser = get_markdown_parser()
+ assert [document for document in _generated_documents() if parser.render(document) != stock.render(document)] == []
+
+ def test_html_length_bound_is_never_below_the_html(self) -> None:
+ parser = get_markdown_parser()
+ for document in _generated_documents():
+ tokens = parser.parse(document)
+ assert html_length_bound(tokens=tokens) >= len(parser.renderer.render(tokens, parser.options, {})), repr(document)
+
+ def test_table_cells_bound_is_never_below_the_cells(self) -> None:
+ parser = get_markdown_parser()
+ for document in _generated_documents():
+ cells = sum(1 for token in parser.parse(document) if token.type in {"th_open", "td_open"})
+ assert table_cells_bound(markdown_text=document) >= cells, repr(document)
+
+ def test_a_reused_reference_is_counted_at_every_use(self) -> None:
+ document = "[a][r] " * 1000 + "\n\n[r]: /" + "y" * 10_000
+ assert html_length_bound(tokens=get_markdown_parser().parse(document)) >= 1000 * 10_000
+
+ def test_padded_cells_are_counted(self) -> None:
+ document = "|" + "a|" * 200 + "\r|" + "-|" * 200 + "\r" + "|a\r" * 300
+ assert table_cells_bound(markdown_text=document) >= 200 * 301
diff --git a/tests/unit/pipelex/tools/test_jinja2_environment.py b/tests/unit/pipelex/tools/test_jinja2_environment.py
index 1afc2cab5..00e4b153a 100644
--- a/tests/unit/pipelex/tools/test_jinja2_environment.py
+++ b/tests/unit/pipelex/tools/test_jinja2_environment.py
@@ -2,8 +2,13 @@
Verifies that async filters are excluded from sync environments to prevent
silent corruption (coroutine objects rendered as strings instead of actual content).
+Every registered filter is wrapped to charge the render budget, so the checks unwrap it.
"""
+import inspect
+from collections.abc import Callable
+from typing import Any
+
import pytest
from pipelex.tools.jinja2.jinja2_environment import make_jinja2_env_without_loader
@@ -12,6 +17,11 @@ from pipelex.tools.jinja2.jinja2_filters import text_format as async_text_format
from pipelex.tools.jinja2.jinja2_models import Jinja2FilterName
from pipelex.tools.jinja2.template_category import TemplateCategory
+
+def _unwrapped(function: Callable[..., Any] | None) -> Callable[..., Any] | None:
+ return None if function is None else inspect.unwrap(function)
+
+
CATEGORIES_WITH_ASYNC_FILTERS = [
TemplateCategory.BASIC,
TemplateCategory.HTML,
@@ -31,8 +41,8 @@ class TestJinja2EnvironmentFilterRegistration:
template_category=template_category,
enable_async=True,
)
- assert jinja2_env.filters[Jinja2FilterName.TAG] is async_tag_filter
- assert jinja2_env.filters[Jinja2FilterName.FORMAT] is async_text_format_filter
+ assert _unwrapped(jinja2_env.filters[Jinja2FilterName.TAG]) is async_tag_filter
+ assert _unwrapped(jinja2_env.filters[Jinja2FilterName.FORMAT]) is async_text_format_filter
@pytest.mark.parametrize("template_category", CATEGORIES_WITH_ASYNC_FILTERS)
def test_async_filters_excluded_when_async_disabled(self, template_category: TemplateCategory) -> None:
@@ -43,7 +53,7 @@ class TestJinja2EnvironmentFilterRegistration:
)
assert Jinja2FilterName.TAG not in jinja2_env.filters
# "format" is a built-in Jinja2 filter, so check it's NOT our async one
- assert jinja2_env.filters.get(Jinja2FilterName.FORMAT) is not async_text_format_filter
+ assert _unwrapped(jinja2_env.filters.get(Jinja2FilterName.FORMAT)) is not async_text_format_filter
def test_sync_filters_registered_when_async_disabled(self) -> None:
"""Sync filters (escape_script_tag) should be registered regardless of enable_async."""
@@ -73,4 +83,4 @@ class TestJinja2EnvironmentFilterRegistration:
enable_async=enable_async,
)
assert Jinja2FilterName.TAG not in jinja2_env.filters
- assert jinja2_env.filters.get(Jinja2FilterName.FORMAT) is not async_text_format_filter
+ assert _unwrapped(jinja2_env.filters.get(Jinja2FilterName.FORMAT)) is not async_text_format_filter
diff --git a/tests/unit/pipelex/tools/test_jinja2_render_budget.py b/tests/unit/pipelex/tools/test_jinja2_render_budget.py
new file mode 100644
index 000000000..8a6fdb809
--- /dev/null
+++ b/tests/unit/pipelex/tools/test_jinja2_render_budget.py
@@ -0,0 +1,698 @@
+"""The render budget: every render spends from one budget, and an operation that would overdraw it is refused.
+
+The refusals are checked at sizes that would take seconds, gigabytes or both without the budget, so a
+hook that stops charging makes its test hang or run out of memory rather than pass: the class's
+timeout turns that into a failure. The renders that must keep working are checked against Jinja's
+stock sandbox, which renders the same text with no budget at all.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import inspect
+import tracemalloc
+from datetime import date, datetime, time, timedelta
+from decimal import Decimal
+from typing import TYPE_CHECKING, Any, cast
+
+import jinja2.filters
+import jinja2.tests
+import pytest
+from jinja2 import DictLoader
+from jinja2.sandbox import ImmutableSandboxedEnvironment
+from markupsafe import Markup
+from typing_extensions import override
+
+from pipelex.base_exceptions import ErrorDomain
+from pipelex.core.stuffs.markdown_content import MarkdownContent
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.tools.jinja2.exceptions import Jinja2TemplateBudgetError
+from pipelex.tools.jinja2.jinja2_environment import make_jinja2_env_without_loader
+from pipelex.tools.jinja2.jinja2_filters import markdown_to_html
+from pipelex.tools.jinja2.jinja2_models import Jinja2ContextKey
+from pipelex.tools.jinja2.jinja2_render_budget import DEFAULT_RENDER_BUDGET_UNITS, RenderBudgetExceededError, active_render_budget
+from pipelex.tools.jinja2.jinja2_render_charging import CHARGED_MARK, INTERNAL_FILTERS
+from pipelex.tools.jinja2.jinja2_render_costs import FILTER_COSTS, PLAIN_VALUE_METHOD_COSTS, REFUSED_PLAIN_VALUE_METHODS, TEST_COSTS
+from pipelex.tools.jinja2.jinja2_rendering import render_jinja2_async, render_jinja2_sync
+from pipelex.tools.jinja2.jinja2_sandbox import PipelexTemplateEnvironment
+from pipelex.tools.jinja2.template_category import TemplateCategory
+from pipelex.tools.markdown.markdown_parser import render_markdown_as_html
+from pipelex.tools.templating.templating_style import TagStyle, TemplatingStyle
+from pipelex.tools.templating.text_format import TextFormat
+
+if TYPE_CHECKING:
+ from collections.abc import Callable
+
+# A budget small enough that a repeated step overdraws it in a few milliseconds.
+_SMALL_BUDGET = 1_000_000
+
+
+def _render(template_source: str, *, budget: int | None = None, autoescape: bool = False, is_async: bool = False, **context: Any) -> str:
+ kwargs: dict[str, Any] = {} if budget is None else {"render_budget_units": budget}
+ env = PipelexTemplateEnvironment(autoescape=autoescape, enable_async=is_async, **kwargs)
+ template = env.from_string(template_source)
+ if is_async:
+ return asyncio.run(template.render_async(**context))
+ return template.render(**context)
+
+
+def _render_stock(template_source: str, *, autoescape: bool = False, is_async: bool = False, **context: Any) -> str:
+ template = ImmutableSandboxedEnvironment(autoescape=autoescape, enable_async=is_async).from_string(template_source)
+ if is_async:
+ return asyncio.run(template.render_async(**context))
+ return template.render(**context)
+
+
+def _refused_render_peak_bytes(render: Callable[[], object]) -> int:
+ """Run a render the budget must refuse, and return the most memory it held on the way."""
+ was_tracing = tracemalloc.is_tracing()
+ if not was_tracing:
+ tracemalloc.start()
+ tracemalloc.reset_peak()
+ try:
+ with pytest.raises(RenderBudgetExceededError):
+ render()
+ _, peak_bytes = tracemalloc.get_traced_memory()
+ finally:
+ if not was_tracing:
+ tracemalloc.stop()
+ return peak_bytes
+
+
+# Single steps whose result would be enormous, each refused before it runs, under the default budget.
+_AMPLIFIERS = [
+ pytest.param("{{ '%(a(b))1000000000s' % {'a(b)': 'x'} }}", id="percent_width_after_a_nested_key"),
+ pytest.param("{{ lipsum(n=1000000000, min=-100, max=-99) }}", id="lipsum_negative_bounds"),
+ pytest.param("{{ 'x' * (10 ** 9) }}", id="repeat_string"),
+ pytest.param("{{ ([0] * (10 ** 9)) | length }}", id="repeat_list"),
+ pytest.param("{{ 2 ** (10 ** 9) }}", id="power"),
+ pytest.param("{{ '%1000000000d' % 1 }}", id="percent_width"),
+ pytest.param("{{ '%.1000000000f' % 1.0 }}", id="percent_precision"),
+ pytest.param("{{ '%*d' % (1000000000, 1) }}", id="percent_star_width"),
+ pytest.param("{{ '%1000000000d' | format(1) }}", id="format_filter_width"),
+ pytest.param("{{ ['%1000000000s'] | format('x') | length }}", id="format_filter_on_a_list"),
+ pytest.param("{{ '{:>1000000000}'.format(1) }}", id="str_format_width"),
+ pytest.param("{{ '{:>{w}}'.format(1, w=1000000000) }}", id="str_format_nested_width"),
+ pytest.param("{{ '{:>{w}}'.format(1, w='1000000000') }}", id="str_format_nested_text_width"),
+ pytest.param("{{ '{a:>{w}}'.format_map({'a': 1, 'w': 1000000000}) }}", id="format_map_width"),
+ pytest.param("{{ '{:.1000000000f}'.format(1.0) }}", id="str_format_precision"),
+ pytest.param("{{ 'x'.ljust(1000000000) }}", id="ljust"),
+ pytest.param("{{ 'x'.rjust(1000000000) }}", id="rjust"),
+ pytest.param("{{ 'x'.center(1000000000) }}", id="center_method"),
+ pytest.param("{{ 'x'.zfill(1000000000) }}", id="zfill"),
+ pytest.param("{{ ('\\t' * 1000).expandtabs(1000000000) }}", id="expandtabs"),
+ pytest.param("{{ ('x' * 1000).replace('x', 'y' * 10000000) }}", id="replace_method"),
+ pytest.param("{{ ('x' * 100000).replace('', 'y' * 100000) }}", id="replace_method_empty_old"),
+ pytest.param("{{ ('y' * 100000).join('x' * 1000000) }}", id="join_method"),
+ pytest.param("{{ ('y' * 1000000).join(range(100000) | reverse) }}", id="join_method_lazy"),
+ pytest.param("{{ ('a' * 100000).translate({97: 'x' * 100000}) }}", id="translate"),
+ pytest.param("{{ ('ab ' * 30000000).split() | length }}", id="split"),
+ pytest.param("{{ ('ab\\n' * 30000000).splitlines() | length }}", id="splitlines"),
+ pytest.param("{{ 'x' | center(1000000000) }}", id="center_filter"),
+ pytest.param("{{ ('x\\n' * 10) | indent(100000000) }}", id="indent_width"),
+ pytest.param("{{ ('x\\n' * 1000000) | indent('y' * 10000) }}", id="indent_string"),
+ pytest.param("{{ ('x ' * 1000000) | wordwrap(1, wrapstring='y' * 10000) }}", id="wordwrap_wrapstring"),
+ pytest.param("{{ ('x' * 1000000) | wordwrap(1) }}", id="wordwrap_long_word"),
+ pytest.param("{{ ('x' * 100000) | replace('', 'y' * 100000) }}", id="replace_filter"),
+ pytest.param("{{ (range(1000) | list) | replace('', 'y' * 200000) | length }}", id="replace_filter_on_a_list"),
+ pytest.param("{{ [1] | batch(1000000000, 'x') | list | length }}", id="batch_padding"),
+ pytest.param("{{ [1] | slice(1000000000) | list | length }}", id="slice_count"),
+ pytest.param("{{ range(100000) | join('x' * 100000) }}", id="join_filter"),
+ pytest.param("{{ range(100000) | map('string') | join('x' * 100000) }}", id="join_filter_lazy"),
+ pytest.param("{{ ([[0] * 1000] * 10000) | sum(start=[]) | length }}", id="sum_of_lists"),
+ pytest.param("{{ ('<!---->' * 1000000) | striptags | length }}", id="striptags_comments"),
+ pytest.param("{{ ('a.com ' * 100000) | urlize(target='t' * 100000) | length }}", id="urlize_target"),
+ pytest.param("{{ lipsum(n=1000000000) }}", id="lipsum_paragraphs"),
+ pytest.param("{{ lipsum(n=1, min=1000000000, max=1000000001) }}", id="lipsum_words"),
+ pytest.param("{% set big = 'x' * 10000 %}{{ ([big] * 100000) | string | length }}", id="string_of_shared_list"),
+ pytest.param("{% set big = 'x' * 10000 %}{{ [big] * 100000 }}", id="print_shared_list"),
+ pytest.param("{% set big = 'x' * 10000 %}{{ ('y' ~ ([big] * 100000)) | length }}", id="concat_shared_list"),
+ pytest.param("{% set big = 'x' * 10000 %}{{ ([big] * 100000) | tojson | length }}", id="tojson_shared_list"),
+ pytest.param("{% set big = 'x' * 10000 %}{{ ([big] * 100000) | pprint | length }}", id="pprint_shared_list"),
+ pytest.param("{% set big = 'x' * 10000 %}{{ '%s' % ([big] * 100000) }}", id="percent_shared_list"),
+ pytest.param("{% set big = 'x' * 10000 %}{{ (big ~ '') in ([big] * 1000000) }}", id="in_shared_list"),
+ pytest.param("{% set a = ['x' * 10000] * 500000 %}{% set b = ['x' * 10000 ~ ''] * 500000 %}{{ a == b }}", id="equal_shared_lists"),
+ pytest.param("{{ range(0, 2305843009213693951 * 100000, 2305843009213693951) | unique | list | length }}", id="colliding_integers"),
+ pytest.param("{{ range(2 ** 70, 2 ** 70 + 3) | list }}", id="range_of_wide_integers"),
+ pytest.param("{{ ('9' * 4000) | int }}", id="int_filter_wide"),
+]
+
+# Steps that are cheap one at a time and overdraw a small budget when repeated. Each is sized so that
+# only the charge it exercises can overdraw: its loop alone, at 64 units an element, stays well inside.
+_REPEATED_STEPS = [
+ pytest.param("{% for i in range(100) %}{% set v = [" + ", ".join(["i"] * 1000) + "] %}{% endfor %}", id="list_literal_in_loop"),
+ pytest.param("{% for i in range(100) %}{% set v = (" + ", ".join(["i"] * 1000) + ") %}{% endfor %}", id="tuple_literal_in_loop"),
+ pytest.param("{% set t = ((1,) * 200,) * 200 %}{% for i in range(30) %}{% set d = {t: i} %}{% endfor %}", id="dict_literal_hashing_a_tuple"),
+ pytest.param("{% set t = ((1,) * 200,) * 200 %}{% set d = {} %}{% for i in range(30) %}{{ d[t] }}{% endfor %}", id="item_read_hashing_a_tuple"),
+ pytest.param("{% set x = '<>' * 200 %}{% for i in range(500) %}{% set s = x | striptags %}{% endfor %}", id="striptags_in_loop"),
+ pytest.param("{% set x = ('<>' * 200) | safe %}{% for i in range(500) %}{% set s = x.striptags() %}{% endfor %}", id="markup_striptags_in_loop"),
+ pytest.param("{% set x = 'a' * 400 %}{% for i in range(100) %}{% set s = x | wordwrap(1) %}{% endfor %}", id="wordwrap_in_loop"),
+ pytest.param("{% set ll = [[0]] * 100 %}{% for i in range(50) %}{% set s = ll | sum(start=[]) %}{% endfor %}", id="sum_in_loop"),
+ pytest.param("{% for i in range(5) %}{% set s = lipsum(n=10000, min=-100, max=-99) %}{% endfor %}", id="lipsum_in_loop"),
+ pytest.param(
+ "{% set ns = namespace(s='x') %}{% for i in range(40) %}{% set ns.s = ns.s ~ ns.s %}{% endfor %}{{ ns.s | length }}", id="doubling_namespace"
+ ),
+ pytest.param(
+ "{% macro d(s, n) %}{% if n %}{{ d(s ~ s, n - 1) }}{% else %}{{ s | length }}{% endif %}{% endmacro %}{{ d('x', 40) }}", id="doubling_macro"
+ ),
+ pytest.param(
+ "{% set ns = namespace(l=[]) %}{% for i in range(1000) %}{% set ns.l = ns.l + [i] %}{% endfor %}{{ ns.l | length }}", id="list_growth"
+ ),
+ pytest.param("{% set ns = namespace(x=3) %}{% for i in range(40) %}{% set ns.x = ns.x * ns.x %}{% endfor %}{{ ns.x }}", id="integer_squaring"),
+ pytest.param(
+ "{% for a in range(1000) %}{% for b in range(1000) %}{% for c in range(1000) %}{% endfor %}{% endfor %}{% endfor %}",
+ id="empty_nested_range_loops",
+ ),
+ pytest.param(
+ "{% set l = ('x' * 1000) | list %}{% for a in l %}{% for b in l %}{% for c in l %}{% endfor %}{% endfor %}{% endfor %}",
+ id="empty_nested_list_loops",
+ ),
+ pytest.param("{% macro f(n) %}{% if n %}{{ f(n - 1) }}{{ f(n - 1) }}{% endif %}{% endmacro %}{{ f(40) }}", id="macro_fan_out"),
+ pytest.param("{% for i in range(1000) %}" + "x" * 10000 + "{% endfor %}", id="static_text_in_loop"),
+ pytest.param("{% set big = 'x' * 10000 %}{% for i in range(1000) %}{{ big }}{% endfor %}", id="printed_value_in_loop"),
+ pytest.param("{% set big = 'x' * 100000 %}{% for i in range(100) %}{% set s = big[1:] %}{% endfor %}", id="slice_in_loop"),
+ pytest.param(
+ "{% set a = 'x' * 100000 %}{% set b = a ~ '' %}{% for i in range(100) %}{% if a == b %}{% endif %}{% endfor %}", id="equality_in_loop"
+ ),
+ pytest.param("{% set a = 'x' * 100000 %}{% for i in range(100) %}{% if 'y' in a %}{% endif %}{% endfor %}", id="substring_in_loop"),
+ # Two lists of equal but distinct strings: small to hold, so charging the test's inputs stays cheap,
+ # and costly to compare, which only the comparison's estimate sees.
+ pytest.param(
+ "{% set a = ['x' * 1000] * 1000 %}{% set b = ['x' * 1000 ~ ''] * 1000 %}{% for i in range(20) %}{% if a is eq b %}{% endif %}{% endfor %}",
+ id="equality_test_in_loop",
+ ),
+ pytest.param(
+ "{% set a = 'x' * 100000 %}{% set b = a ~ '' %}{% for i in range(100) %}{% if a <= b <= a %}{% endif %}{% endfor %}",
+ id="chained_comparison_in_loop",
+ ),
+ # Reading `loop.depth0` costs nothing, and passing a range to `loop(x)` is charged at its size, a few bytes an
+ # element: only charging the elements `loop(x)` draws can overdraw.
+ pytest.param(
+ "{% for x in [range(5000)] * 10 recursive %}{% if not loop.depth0 %}{{ loop(x) }}{% endif %}{% endfor %}",
+ id="recursive_loop",
+ ),
+ pytest.param("{% for i in range(1000) %}{% set s = range(100) | map('string') | join %}{% endfor %}", id="filters_in_loop"),
+ # A lazy input produces its elements as a filter draws them, at no cost any other charge sees: `reject`
+ # draws every element `select` produces, and yields none.
+ pytest.param("{% set s = range(1, 30000) | select | reject | first %}", id="lazy_input_drawn"),
+ # `reject` draws a range at no charge per element, so only sizing the range by its length can overdraw.
+ pytest.param("{% for i in range(100) %}{% set s = range(1, 100000) | reject | first %}{% endfor %}", id="range_drawn_by_a_filter"),
+ pytest.param("{% for i in range(5000) %}{% set s = 'abc'.upper() %}{% endfor %}", id="method_calls_in_loop"),
+ pytest.param("{% for i in range(5000) %}{% set s = [i] | first %}{% endfor %}", id="filter_calls_in_loop"),
+ pytest.param("{% for i in range(10000) %}{% set t = i is odd %}{% endfor %}", id="test_calls_in_loop"),
+]
+
+
+def _text_content(text: str) -> TextContent:
+ return TextContent(text=text)
+
+
+# Single steps that repeat an object of the run's data, whose text only converting it tells: each is
+# refused before it allocates what the repetition asks, a few hundred megabytes.
+_DATA_AMPLIFIERS = [
+ pytest.param("{{ ([doc] * 20000) | join | length }}", id="join"),
+ pytest.param("{{ (('%(a)s' * 20000) % {'a': doc}) | length }}", id="percent"),
+ pytest.param("{{ ('{0.text}' * 20000).format(doc) | length }}", id="str_format_field"),
+ pytest.param("{{ ('{a}' * 20000).format_map({'a': doc}) | length }}", id="format_map"),
+ pytest.param("{{ links | urlize(target='t' * 2000) | length }}", id="urlize"),
+ pytest.param("{{ (blob * 200000000) | length }}", id="bytes_repetition"),
+]
+
+# Ordering filters over long strings that share a prefix: few elements, each comparison scans them all.
+_ORDERINGS = [
+ pytest.param("{{ same | unique | list | length }}", id="unique"),
+ pytest.param("{{ docs | sort(attribute='text') | length }}", id="sort_by_an_object_attribute"),
+ pytest.param("{{ docs | sort(attribute='text', case_sensitive=true) | length }}", id="sort_by_an_object_attribute_case_sensitive"),
+ pytest.param("{{ items | sort | length }}", id="sort"),
+ pytest.param("{{ items | min | length }}", id="min"),
+ pytest.param("{{ items | max(case_sensitive=true) | length }}", id="max"),
+ pytest.param("{{ keyed | dictsort | length }}", id="dictsort"),
+ pytest.param("{{ rows | groupby('k', case_sensitive=true) | list | length }}", id="groupby"),
+]
+
+# Operations that read a fixed part of a large value, repeated: each would overdraw the default budget
+# if it were charged the whole value.
+_FIXED_READS = [
+ pytest.param("{% for i in range(100) %}{{ doc | length }}{{ doc | first }}{{ doc | last }}{% endfor %}", id="length_first_last"),
+ pytest.param("{% for i in range(100) %}{{ lookup.get('k5') }}{{ lookup.keys() | length }}{% endfor %}", id="dict_get_and_keys"),
+ pytest.param("{% for i in range(100) %}{% if doc is defined and doc is string %}y{% endif %}{% endfor %}", id="tests"),
+ pytest.param("{% macro m(x) %}.{% endmacro %}{% for i in range(100) %}{{ m(doc) }}{% endfor %}", id="macro_argument"),
+ pytest.param("{% for i in range(100) %}{{ doc.startswith('x') }}{{ doc | default('d') | length }}{% endfor %}", id="startswith_and_default"),
+]
+
+# Templates that must render exactly as Jinja's stock sandbox renders them.
+_UNCHANGED = [
+ pytest.param("{{ 1 < 2 < 3 }}|{{ 3 > 2 > 2 }}|{{ 1 == 1.0 }}|{{ 'a' != 'b' }}", id="comparisons"),
+ pytest.param("{{ 'a' in 'abc' }}|{{ 'd' not in ['a'] }}|{{ 1 in {1: 2} }}|{{ 2 in range(3) }}|{{ 5 in range(3) | reverse }}", id="membership"),
+ # Parts that are not all constants: Jinja folds a constant `~` at compile time, which drops the markup.
+ pytest.param("{{ x ~ ('<b>' | safe) ~ 1 ~ [2] }}|{{ x ~ x }}", id="concat_with_markup"),
+ pytest.param("<p>{{ x }}</p>{{ '<i>' }}", id="static_text_and_escaping"),
+ pytest.param("{{ 'abcdef'[1:3] }}|{{ [1, 2, 3][::-1] }}|{{ 'abc'[1] }}", id="slices"),
+ pytest.param("{% for i in [1, 2, 3] %}{{ loop.index }}/{{ loop.length }}/{{ loop.revindex }}/{{ loop.last }};{% endfor %}", id="loop_helpers"),
+ pytest.param("{% for x in range(3) | map('string') %}{{ loop.length }}{{ x }}{% endfor %}", id="loop_length_over_lazy_iterable"),
+ pytest.param(
+ "{% for i in range(5) if i is odd %}{{ i }}{% else %}none{% endfor %}|{% for i in [] %}x{% else %}none{% endfor %}", id="loop_test_and_else"
+ ),
+ pytest.param(
+ "{% for x in [[1, [2]], 3] recursive %}{% if x is iterable %}[{{ loop(x) }}]{% else %}{{ x }}{% endif %}{% endfor %}", id="recursive_loop"
+ ),
+ pytest.param("{% macro wrap(x) %}[{{ x }}{{ caller() }}]{% endmacro %}{% call wrap(1) %}C{% endcall %}", id="macro_with_caller"),
+ pytest.param("{% set x %}a{{ 1 }}{% endset %}{{ x }}{% filter upper %}b{{ 'c' }}{% endfilter %}", id="set_and_filter_blocks"),
+ pytest.param("{{ 2 ** 10 }}|{{ 7 // 2 }}|{{ 7 % 3 }}|{{ 1.5 * 2 }}|{{ 'ab' * 3 }}|{{ [1] * 3 }}|{{ -3 + 1 }}|{{ 1 / 4 }}", id="arithmetic"),
+ pytest.param("{{ '%05.1f|%s' % (3.14159, 'x') }}|{{ '%(a)s-%(a)s' % {'a': 1} }}", id="percent_format"),
+ pytest.param("{{ '{:>5}|{}|{x:.2f}'.format('a', 2, x=1.5) }}|{{ '%d items' | format(3) }}", id="str_format"),
+ pytest.param(
+ "{{ 'a,b'.split(',') | join('-') }}|{{ 'x'.ljust(3) }}|{{ ' y '.strip() }}|{{ 'ab'.replace('a', 'c') }}|{{ ','.join(['1', '2']) }}",
+ id="methods",
+ ),
+ pytest.param(
+ "{{ [3, 1, 2] | sort | list }}|{{ [1, 1, 2] | unique | list }}|{{ [1, 2, 3] | batch(2) | list }}|{{ [1, 2, 3] | slice(2) | list }}",
+ id="list_filters",
+ ),
+ pytest.param(
+ "{{ {'b': 1, 'a': 2} | dictsort }}|{{ [1, 2, 3] | sum }}|{{ [1, 2] | min }}|{{ [1, 2] | max }}|{{ [1, 2] | first }}|{{ [1, 2] | last }}",
+ id="aggregates",
+ ),
+ pytest.param(
+ "{{ 'ab' | center(6) }}|{{ 'a\\nb' | indent(2) }}|{{ 'aaa bbb' | wordwrap(3) }}|{{ 'long text' | truncate(5, true, '') }}", id="text_filters"
+ ),
+ pytest.param(
+ "{{ {'a': '<'} | tojson }}|{{ '<x>' | tojson }}|{{ [1, {'b': 2}] | pprint }}|{{ '<b>x</b>' | striptags }}", id="serialisation_filters"
+ ),
+ pytest.param("{{ '<' | escape }}|{{ 'a b' | urlencode }}|{{ 'see a.com' | urlize }}|{{ {'class': 'x'} | xmlattr }}", id="escaping_filters"),
+ pytest.param(
+ "{{ 'abc' | capitalize }}|{{ 'a b' | title }}|{{ ' a ' | trim }}|{{ 'a b c' | wordcount }}|{{ 1000 | filesizeformat }}",
+ id="more_text_filters",
+ ),
+ pytest.param(
+ "{{ '3' | int }}|{{ '1.5' | float }}|{{ 1.55 | round(1) }}|{{ -1 | abs }}|{{ 'abc' | reverse }}|{{ [1, 2] | reverse | list }}",
+ id="number_filters",
+ ),
+ pytest.param("{{ x | default('d') }}|{{ [1, 2] | length }}|{{ {'a': 1} | items | list }}|{{ [1, 2] | string }}", id="misc_filters"),
+ pytest.param(
+ "{{ [{'a': 1}, {'a': 2}] | map(attribute='a') | select('odd') | list }}|{{ [{'a': 1}] | selectattr('a') | list | length }}",
+ id="map_and_select",
+ ),
+ pytest.param("{{ [{'a': 1}, {'a': 1}] | groupby('a') | list | length }}|{{ [1, 2] | reject('odd') | list }}", id="groupby_and_reject"),
+ pytest.param(
+ "{{ 3 is odd }}{{ 'ab' is lower }}{{ nothing is defined }}{{ 2 is in [1, 2] }}{{ 1 is eq 1 }}{{ 2 is gt 1 }}{{ 'a' is string }}", id="tests"
+ ),
+ pytest.param("{% set ns = namespace(n=1) %}{% set ns.n = ns.n + 1 %}{{ ns.n }}|{{ dict(a=1)['a'] }}|{{ range(3) | list }}", id="globals"),
+ pytest.param("{% set c = cycler('a', 'b') %}{{ c.next() }}{{ c.next() }}{% set j = joiner(',') %}{{ j() }}x{{ j() }}y", id="cycler_and_joiner"),
+ pytest.param(
+ "{{ (1).bit_length() }}|{{ 1.5.is_integer() }}|{{ {'a': 1}.get('a') }}|{{ [1, 2].index(2) }}|{{ (1, 2).count(1) }}", id="plain_methods"
+ ),
+ pytest.param(
+ "{{ nope | default('d') }}|{{ nope | length }}|{{ '-'.join(nope) }}|{{ 'a' in nope }}|{% for x in nope %}x{% else %}empty{% endfor %}",
+ id="undefined_values",
+ ),
+ pytest.param("{% macro m(v) %}{{ v is defined }}{% endmacro %}{{ m(nope) }}", id="undefined_macro_argument"),
+ pytest.param(
+ "{% macro show(l) %}[{{ l.index }}/{{ l.length }}]{% endmacro %}{% for i in range(3) %}{{ show(loop) }}{% endfor %}",
+ id="loop_passed_to_a_macro",
+ ),
+ pytest.param("{{ '{:>{w}}|{a}'.format('x', w='3', a=[1]) }}|{{ '{a:>{w}}'.format_map({'a': 'y', 'w': 2}) }}", id="format_fields"),
+ pytest.param(
+ "{{ ['b', 'A', 'a'] | sort | join }}|{{ [{'k': 'b'}, {'k': 'a'}] | sort(attribute='k') | map(attribute='k') | join }}"
+ "|{{ ['b', 'A'] | min }}|{{ {'b': 1, 'A': 2} | dictsort(by='value') }}",
+ id="orderings",
+ ),
+]
+
+
+_STYLE = TemplatingStyle(tag_style=TagStyle.XML, text_format=TextFormat.PLAIN)
+
+
+# A table whose two hundred columns pad three hundred one-character rows: sixty thousand cells, about sixty
+# megabytes of parsed tokens, out of two kilobytes of text.
+_PADDED_TABLE = "'|' ~ ('a|' * 222) ~ '\\n|' ~ ('-|' * 222) ~ '\\n' ~ ('|a\\n' * 300) ~ '\\n'"
+
+# Markdown a template converts with the `markdown` filter, each refused before it parses or renders what it
+# asks: a long source, which costs microseconds a character to parse; padded tables, which cost their cells;
+# and a reference used thousands of times, whose destination is printed at every use.
+_MARKDOWN_AMPLIFIERS = [
+ pytest.param("{{ ('**a ' * 100000) | markdown }}", id="long_source"),
+ pytest.param("{{ ((" + _PADDED_TABLE + ") * 3) | markdown }}", id="padded_tables"),
+ pytest.param("{{ ('[a][r] ' * 2000 ~ '\\n\\n[r]: /' ~ 'y' * 20000) | markdown }}", id="reused_reference"),
+]
+
+# What a refused conversion may hold before the refusal: the source and its table scan, never its tokens
+# or its output.
+_MARKDOWN_REFUSAL_PEAK_BYTES = 32 * 1024 * 1024
+
+# A Markdown value an HTML template converts again at every print, escape, join or format, the last only in
+# async mode, where Pipelex's own `format` filter is registered.
+_MARKDOWN_CONVERSIONS = [
+ *(
+ pytest.param(template_source, is_async, id=f"{name}-{'async' if is_async else 'sync'}")
+ for name, template_source in (
+ ("print", "{% for i in range(100) %}{{ report }}{% endfor %}"),
+ ("escape", "{% for i in range(100) %}{{ report | e }}{% endfor %}"),
+ ("join", "{{ ([report] * 100) | join }}"),
+ )
+ for is_async in (False, True)
+ ),
+ pytest.param("{% for i in range(100) %}{{ report | format }}{% endfor %}", True, id="format-async"),
+]
+
+_MARKDOWN_REPORT = "## Findings\n\nSome *text* with a [link](https://a.co), `code` and more.\n\n- one\n- two\n\n" * 300
+
+_MARKDOWN_SAMPLE = (
+ "# Report\n\n| a | b |\n| :- | -: |\n| 1 | 2 |\n| 3 |\n\n> quoted **bold** and ~~gone~~\n\n"
+ 'See [the docs][docs] and [again][docs], ![a chart](https://a.co/c.png "Chart"), https://a.co & <tags>.\n\n'
+ "```python\nprint('<x>')\n```\n\n[docs]: https://a.co/docs \"The docs\"\n"
+)
+
+
+def _make_html_template(template_source: str, *, is_async: bool) -> Any:
+ return make_jinja2_env_without_loader(TemplateCategory.HTML, enable_async=is_async).from_string(template_source)
+
+
+def _render_html(template_source: str, *, is_async: bool, **context: Any) -> str:
+ template = _make_html_template(template_source, is_async=is_async)
+ context[Jinja2ContextKey.TEXT_FORMAT] = TextFormat.PLAIN
+ if is_async:
+ return cast("str", asyncio.run(template.render_async(**context)))
+ return cast("str", template.render(**context))
+
+
+class _Document:
+ """An object of the run's data whose text is an attribute, read the way a filter reads it."""
+
+ def __init__(self, text: str) -> None:
+ self.text = text
+
+
+class _Opaque:
+ """An object of the run's data whose `repr` fails, as nothing a template does should call it."""
+
+ @override
+ def __repr__(self) -> str:
+ msg = "repr taken"
+ raise RuntimeError(msg)
+
+
+class TestRenderBudget:
+ @pytest.mark.parametrize("template_source", _AMPLIFIERS)
+ @pytest.mark.parametrize("is_async", [False, True])
+ def test_single_step_refused_before_it_runs(self, template_source: str, is_async: bool) -> None:
+ # Refused after it ran, a step would still fail the render, having allocated gigabytes first. What
+ # the render allocates before the refusal is its inputs, which fit in the budget.
+ assert _refused_render_peak_bytes(lambda: _render(template_source, is_async=is_async)) < DEFAULT_RENDER_BUDGET_UNITS
+
+ @pytest.mark.parametrize("template_source", _REPEATED_STEPS)
+ @pytest.mark.parametrize("is_async", [False, True])
+ def test_repeated_steps_overdraw(self, template_source: str, is_async: bool) -> None:
+ with pytest.raises(RenderBudgetExceededError):
+ _render(template_source, budget=_SMALL_BUDGET, is_async=is_async)
+
+ @pytest.mark.parametrize("template_source", _DATA_AMPLIFIERS)
+ def test_repeated_data_is_refused_before_it_is_built(self, template_source: str) -> None:
+ context: dict[str, Any] = {
+ "doc": _text_content("x" * 10_000),
+ "links": _text_content("a.com " * 10_000),
+ "blob": b"x",
+ }
+ template = PipelexTemplateEnvironment(render_budget_units=_SMALL_BUDGET).from_string(template_source)
+ assert _refused_render_peak_bytes(lambda: template.render(**context)) < 4 * _SMALL_BUDGET
+
+ def test_rendering_with_images_is_estimated_item_by_item(self) -> None:
+ template = make_jinja2_env_without_loader(TemplateCategory.LLM_PROMPT, enable_async=False).from_string(
+ "{{ ([doc] * 30000) | with_images | length }}"
+ )
+ context: dict[str, Any] = {"doc": _text_content("x" * 10_000), Jinja2ContextKey.TEXT_FORMAT: TextFormat.PLAIN}
+ assert _refused_render_peak_bytes(lambda: template.render(**context)) < DEFAULT_RENDER_BUDGET_UNITS
+
+ @pytest.mark.parametrize("template_source", _ORDERINGS)
+ def test_ordering_is_charged_for_its_comparisons(self, template_source: str) -> None:
+ items = ["x" * 100_000 + str(index) for index in range(100)]
+ context: dict[str, Any] = {
+ "items": items,
+ "keyed": dict.fromkeys(items, 1),
+ "rows": [{"k": item} for item in items],
+ "docs": [_Document(item) for item in items],
+ "same": ["X" * 100_000] * 100,
+ }
+ with pytest.raises(RenderBudgetExceededError):
+ _render(template_source, budget=_SMALL_BUDGET, **context)
+
+ def test_comparing_models_weighs_their_fields(self) -> None:
+ # Two equal texts of distinct strings, compared character by character; their models' `repr` is short.
+ left = _text_content("x" * 2_000_000)
+ right = _text_content("".join(["x" * 1_000_000, "x" * 1_000_000]))
+ with pytest.raises(RenderBudgetExceededError):
+ _render("{{ left == right }}", budget=_SMALL_BUDGET, left=left, right=right)
+
+ @pytest.mark.parametrize("template_source", ["{{ big }}", "{{ ('' | safe) ~ big }}"])
+ def test_escaped_text_is_estimated_before_it_is_escaped(self, template_source: str) -> None:
+ # Printed under autoescape, or joined to markup, `<` grows fourfold: the text fits the budget, its
+ # escaped form does not.
+ template = PipelexTemplateEnvironment(autoescape=True, render_budget_units=_SMALL_BUDGET).from_string(template_source)
+ big = "<" * 600_000
+ assert _refused_render_peak_bytes(lambda: template.render(big=big)) < _SMALL_BUDGET
+
+ @pytest.mark.parametrize("autoescape", [False, True])
+ def test_escaping_is_estimated(self, autoescape: bool) -> None:
+ # Printing, joining and escaping under autoescape grow `<` fourfold, which the estimate counts.
+ with pytest.raises(RenderBudgetExceededError):
+ _render("{{ big | escape }}{{ big ~ '' }}", budget=_SMALL_BUDGET, autoescape=autoescape, big="<" * 300_000)
+
+ @pytest.mark.parametrize("view", ["keys", "values", "items"])
+ def test_dict_view_is_sized_by_its_length(self, view: str) -> None:
+ # A dict's view is a few bytes, and `reject` iterates all of it.
+ template_source = f"{{% for i in range(100) %}}{{% set s = big.{view}() | reject | first %}}{{% endfor %}}"
+ with pytest.raises(RenderBudgetExceededError):
+ _render(template_source, budget=_SMALL_BUDGET, big={number: number for number in range(1, 100_000)})
+
+ def test_included_template_spends_from_the_includer_budget(self) -> None:
+ env = PipelexTemplateEnvironment(loader=DictLoader({"child": "{{ big }}"}), render_budget_units=_SMALL_BUDGET)
+ template = env.from_string("{% for i in range(1000) %}{% include 'child' %}{% endfor %}")
+ with pytest.raises(RenderBudgetExceededError):
+ template.render(big="x" * 10_000)
+
+ def test_caller_finalize_output_is_charged(self) -> None:
+ def inflate(_value: object) -> str:
+ return "y" * 100_000
+
+ env = PipelexTemplateEnvironment(finalize=inflate, render_budget_units=_SMALL_BUDGET)
+ with pytest.raises(RenderBudgetExceededError):
+ env.from_string("{% for i in range(100) %}{{ i }}{% endfor %}").render()
+
+ def test_refusal_names_the_operation_and_never_the_source(self) -> None:
+ template_source = "{{ 'secret-marker'.ljust(1000000000) }}"
+ with pytest.raises(RenderBudgetExceededError, match=r"calling the method 'ljust' of a 'str' value") as exc_info:
+ _render(template_source)
+ assert "secret-marker" not in str(exc_info.value)
+
+ def test_sync_render_raises_the_budget_error(self) -> None:
+ with pytest.raises(Jinja2TemplateBudgetError, match="refused by the render budget"):
+ render_jinja2_sync(template_source="{{ 'x' * (10 ** 9) }}", template_category=TemplateCategory.BASIC, templating_context={})
+
+ @pytest.mark.asyncio
+ async def test_async_render_raises_the_budget_error(self) -> None:
+ with pytest.raises(Jinja2TemplateBudgetError, match="refused by the render budget"):
+ await render_jinja2_async(
+ template_source="{{ 'x' * (10 ** 9) }}",
+ template_category=TemplateCategory.LLM_PROMPT,
+ templating_context={},
+ templating_style=_STYLE,
+ )
+
+ @pytest.mark.asyncio
+ async def test_deep_recursion_is_a_budget_error(self) -> None:
+ with pytest.raises(Jinja2TemplateBudgetError, match="deeper than Python allows"):
+ await render_jinja2_async(
+ template_source="{% macro f(n) %}{{ f(n + 1) }}{% endmacro %}{{ f(0) }}",
+ template_category=TemplateCategory.LLM_PROMPT,
+ templating_context={},
+ templating_style=_STYLE,
+ )
+
+ @pytest.mark.parametrize("template_source", _MARKDOWN_AMPLIFIERS)
+ @pytest.mark.parametrize("is_async", [False, True])
+ def test_markdown_conversion_refused_before_it_parses_or_renders(self, template_source: str, is_async: bool) -> None:
+ peak_bytes = _refused_render_peak_bytes(lambda: _render_html(template_source, is_async=is_async))
+ assert peak_bytes < _MARKDOWN_REFUSAL_PEAK_BYTES
+
+ @pytest.mark.parametrize(("template_source", "is_async"), _MARKDOWN_CONVERSIONS)
+ def test_markdown_value_is_charged_at_every_conversion(self, template_source: str, is_async: bool) -> None:
+ # Each conversion of this report costs about a third of the default budget, and takes a few
+ # milliseconds whatever it is charged: printed a hundred times, it is refused at the fourth.
+ report = MarkdownContent(text=_MARKDOWN_REPORT)
+ with pytest.raises(RenderBudgetExceededError, match="converting Markdown to HTML"):
+ _render_html(template_source, is_async=is_async, report=report)
+
+ @pytest.mark.parametrize("is_async", [False, True])
+ def test_markdown_renders_as_outside_any_render(self, is_async: bool) -> None:
+ rendered = _render_html(
+ "{{ notes | markdown }}|{{ report }}", is_async=is_async, notes=_MARKDOWN_SAMPLE, report=MarkdownContent(text=_MARKDOWN_SAMPLE)
+ )
+ assert active_render_budget() is None
+ expected = render_markdown_as_html(_MARKDOWN_SAMPLE)
+ assert rendered == f"{expected}|{expected}"
+
+ def test_markdown_the_budget_admits_parses_within_it(self) -> None:
+ # Unclosed emphasis leaves a text token per delimiter, which markdown-it's own join held a copy of
+ # every prefix of: thirty thousand characters peaked at over three hundred megabytes.
+ tracemalloc.start()
+ try:
+ rendered = _render_html("{{ ('**a ' * 7500) | markdown }}", is_async=False)
+ _, peak_bytes = tracemalloc.get_traced_memory()
+ finally:
+ tracemalloc.stop()
+ assert len(rendered) > 30_000
+ assert peak_bytes < _MARKDOWN_REFUSAL_PEAK_BYTES
+
+ def test_the_active_budget_ends_with_its_render(self) -> None:
+ with pytest.raises(RenderBudgetExceededError):
+ _render_html("{{ ('**a ' * 100000) | markdown }}", is_async=False)
+ assert active_render_budget() is None
+
+ @pytest.mark.parametrize("method", ["count", "index"])
+ @pytest.mark.parametrize("container", ["list", "tuple"])
+ def test_searching_a_sequence_is_charged_for_its_comparisons(self, method: str, container: str) -> None:
+ # Two equal strings that are distinct objects compare character by character.
+ needle = "".join(["x" * 1000, "y"])
+ items = ["x" * 1000 + "y" for _ in range(999)] + [needle]
+ sequence: Any = items if container == "list" else tuple(items)
+ with pytest.raises(RenderBudgetExceededError):
+ _render(
+ f"{{% for i in range(20) %}}{{% set c = seq.{method}(needle) %}}{{% endfor %}}", budget=_SMALL_BUDGET, seq=sequence, needle=needle
+ )
+
+ @pytest.mark.parametrize(
+ "template_source", ["{{ big.values() }}", "{{ big.items() }}", "{{ 'a' ~ big.values() }}", "{{ big.values() | string }}"]
+ )
+ def test_a_dict_view_is_estimated_before_it_is_printed(self, template_source: str) -> None:
+ big = {"k": ["x" * 1000] * 5000, **{f"k{index}": "y" * 1000 for index in range(2000)}}
+ template = PipelexTemplateEnvironment(render_budget_units=_SMALL_BUDGET).from_string(template_source)
+ assert _refused_render_peak_bytes(lambda: template.render(big=big)) < 4 * _SMALL_BUDGET
+
+ @pytest.mark.parametrize("filter_name", ["upper", "lower", "capitalize", "title", "trim", "markdown"])
+ def test_a_container_is_estimated_before_a_filter_converts_it(self, filter_name: str) -> None:
+ env = PipelexTemplateEnvironment(autoescape=True, render_budget_units=_SMALL_BUDGET)
+ env.filters["markdown"] = markdown_to_html
+ template = env.from_string(f"{{{{ items | {filter_name} }}}}")
+ assert _refused_render_peak_bytes(lambda: template.render(items=["x" * 1000] * 5000)) < 4 * _SMALL_BUDGET
+
+ def test_a_percent_key_left_open_is_read_in_one_pass(self) -> None:
+ # A quadratic scan of `%(` would take minutes here; Python refuses the format string at once.
+ with pytest.raises(ValueError, match="incomplete format key"):
+ _render("{{ ('%(' * 200000) % {} }}")
+
+ def test_a_long_word_under_its_width_is_read_in_one_pass(self) -> None:
+ # A run of non-space characters one short of the width is rescanned from every start by a pattern.
+ template_source = "{{ (('a' * 199999 ~ ' ') * 2) | wordwrap(199999) | length }}"
+ assert _render(template_source) == _render_stock(template_source)
+
+ def test_stripping_tags_from_a_data_object_reads_its_text(self) -> None:
+ text = "word " * 2400
+ assert _render("{{ doc | striptags }}", doc=_text_content(text)) == _render("{{ text | striptags }}", text=text)
+
+ def test_budget_error_is_the_caller_s_and_caller_facing(self) -> None:
+ report = Jinja2TemplateBudgetError("refused by the render budget").to_error_report()
+ assert report.error_domain == ErrorDomain.INPUT
+ assert report.caller_facing_message
+
+ @pytest.mark.parametrize("template_source", _UNCHANGED)
+ @pytest.mark.parametrize("autoescape", [False, True])
+ @pytest.mark.parametrize("is_async", [False, True])
+ def test_renders_as_the_stock_sandbox(self, template_source: str, autoescape: bool, is_async: bool) -> None:
+ context: dict[str, Any] = {"x": "<&>"}
+ assert _render(template_source, autoescape=autoescape, is_async=is_async, **context) == _render_stock(
+ template_source, autoescape=autoescape, is_async=is_async, **context
+ )
+
+ @pytest.mark.parametrize("template_source", _FIXED_READS)
+ def test_reading_part_of_a_large_value_stays_cheap(self, template_source: str) -> None:
+ context: dict[str, Any] = {"doc": "x" * 2_000_000, "lookup": {f"k{index}": index for index in range(100_000)}}
+ assert _render(template_source, **context) == _render_stock(template_source, **context)
+
+ @pytest.mark.parametrize("is_async", [False, True])
+ def test_comparing_objects_never_takes_their_repr(self, is_async: bool) -> None:
+ template_source = "{{ obj == obj }}|{{ obj in [obj] }}|{{ obj != 1 }}"
+ context: dict[str, Any] = {"obj": _Opaque()}
+ assert _render(template_source, is_async=is_async, **context) == _render_stock(template_source, is_async=is_async, **context)
+
+ def test_each_render_gets_its_own_budget(self) -> None:
+ env = PipelexTemplateEnvironment(render_budget_units=_SMALL_BUDGET)
+ template = env.from_string("{{ big }}")
+ # Each render spends most of its budget, which only works if every render gets a new one.
+ for _ in range(3):
+ assert len(template.render(big="x" * 700_000)) == 700_000
+
+ def test_large_text_in_an_escaping_template(self) -> None:
+ # A 20 MB string of plain text prints under autoescape: its escaped length is counted, not guessed.
+ big = "a" * 20_000_000
+ assert len(_render("{{ big }}{{ big | escape }}", autoescape=True, big=big)) == 40_000_000
+
+ def test_caller_finalize_still_applies(self) -> None:
+ def blank_none(value: object) -> object:
+ return "" if value is None else value
+
+ env = PipelexTemplateEnvironment(finalize=blank_none)
+ assert env.from_string("{{ nothing }}|{{ none }}|{{ 1 }}").render(nothing=None) == "||1"
+
+ def test_finalize_assigned_after_construction_is_refused(self) -> None:
+ env = PipelexTemplateEnvironment()
+ env.finalize = str
+ with pytest.raises(RuntimeError, match="takes its finalize at construction"):
+ env.from_string("{{ 1 }}")
+
+ def test_every_jinja_filter_and_test_is_classified(self) -> None:
+ filters = cast("dict[str, Callable[..., Any]]", jinja2.filters.FILTERS)
+ tests = cast("dict[str, Callable[..., Any]]", jinja2.tests.TESTS)
+ assert [name for name, function in filters.items() if function not in FILTER_COSTS] == []
+ assert [name for name, function in tests.items() if function not in TEST_COSTS] == []
+
+ @pytest.mark.parametrize("category", list(TemplateCategory))
+ def test_every_registered_filter_and_test_is_charged(self, category: TemplateCategory) -> None:
+ env = make_jinja2_env_without_loader(category)
+ env.filters["late"] = jinja2.filters.FILTERS["upper"]
+ env.from_string("{{ 1 }}")
+ for name, function in env.filters.items():
+ assert name in INTERNAL_FILTERS or getattr(function, CHARGED_MARK, False), name
+ for name, function in env.tests.items():
+ assert getattr(function, CHARGED_MARK, False), name
+
+ def test_unclassified_filter_is_refused_before_any_template_compiles(self) -> None:
+ env = PipelexTemplateEnvironment()
+ env.filters["shout"] = str.upper
+ with pytest.raises(TypeError, match="has no cost in the render budget's tables"):
+ env.from_string("{{ 'a' }}")
+
+ @pytest.mark.parametrize("plain_type", list(PLAIN_VALUE_METHOD_COSTS))
+ def test_every_plain_method_is_classified(self, plain_type: type) -> None:
+ """A method a later Python adds fails here until someone decides what it costs."""
+ instance_methods = {
+ name
+ for name in dir(plain_type)
+ if not name.startswith("_")
+ and callable(member := inspect.getattr_static(plain_type, name))
+ and not isinstance(member, (classmethod, staticmethod))
+ and type(member).__name__ not in {"classmethod_descriptor", "builtin_function_or_method"}
+ }
+ allowed = set(PLAIN_VALUE_METHOD_COSTS[plain_type])
+ refused = set(REFUSED_PLAIN_VALUE_METHODS.get(plain_type, {}))
+ assert not allowed & refused
+ assert instance_methods <= allowed | refused, sorted(instance_methods - allowed - refused)
+
+ def test_plain_types_cover_the_sandbox(self) -> None:
+ assert set(PLAIN_VALUE_METHOD_COSTS) == {
+ str,
+ Markup,
+ int,
+ bool,
+ float,
+ Decimal,
+ date,
+ datetime,
+ time,
+ timedelta,
+ list,
+ tuple,
+ dict,
+ set,
+ frozenset,
+ }
diff --git a/tests/unit/pipelex/tools/test_jinja2_sandbox.py b/tests/unit/pipelex/tools/test_jinja2_sandbox.py
index da7d87d4b..8a232be38 100644
--- a/tests/unit/pipelex/tools/test_jinja2_sandbox.py
+++ b/tests/unit/pipelex/tools/test_jinja2_sandbox.py
@@ -23,7 +23,6 @@ from pipelex.tools.jinja2.exceptions import Jinja2TemplateRenderError, Jinja2Tem
from pipelex.tools.jinja2.image_registry import ImageRegistry
from pipelex.tools.jinja2.jinja2_models import Jinja2ContextKey
from pipelex.tools.jinja2.jinja2_rendering import render_jinja2_async, render_jinja2_sync
-from pipelex.tools.jinja2.jinja2_sandbox import _MUTATING_METHOD_NAMES # pyright: ignore[reportPrivateUsage]
from pipelex.tools.jinja2.template_category import TemplateCategory
from pipelex.tools.templating.templating_style import TagStyle, TemplatingStyle
from pipelex.tools.templating.text_format import TextFormat
@@ -115,14 +114,6 @@ async def _render(template_source: str, *, context: dict[str, Any] | None = None
)
-# The methods of each mutable plain type that leave the value unchanged, so that together with the
-# sandbox's own list of mutating methods they classify every public method the type has.
-_NON_MUTATING_METHOD_NAMES: dict[type, frozenset[str]] = {
- list: frozenset({"copy", "count", "index"}),
- dict: frozenset({"copy", "fromkeys", "get", "items", "keys", "values"}),
- set: frozenset({"copy", "difference", "intersection", "isdisjoint", "issubset", "issuperset", "symmetric_difference", "union"}),
-}
-
# Templates handing a filter a value that holds a rendering method as data rather than defining it in its class.
_NAMESPACE_WITH_RENDER_METHODS = (
"{% set ns = namespace(render_with_images=spy, rendered_for_template_async=spy, render_for_tag_async=spy, default_tag_name='ns') %}"
@@ -359,12 +350,3 @@ class TestTemplateSandbox:
templating_context={},
)
assert rendered == "<b>&lt;i&gt;</b>"
-
- @pytest.mark.parametrize("plain_type", [list, dict, set])
- def test_every_mutable_type_method_is_classified(self, plain_type: type) -> None:
- """A method a later Python adds fails here until someone decides whether it mutates."""
- public_names = {name for name in dir(plain_type) if not name.startswith("_")}
- mutating = _MUTATING_METHOD_NAMES[plain_type]
- non_mutating = _NON_MUTATING_METHOD_NAMES[plain_type]
- assert not mutating & non_mutating
- assert public_names == mutating | non_mutating
diff --git a/uv.lock b/uv.lock
index 00190c993..69634b3b5 100644
--- a/uv.lock
+++ b/uv.lock
@@ -3691,7 +3691,7 @@ requires-dist = [
{ name = "idna", specifier = ">=3.15" },
{ name = "instructor", specifier = ">=1.17.0,<2.0.0" },
{ name = "instructor", extras = ["google-genai"], marker = "extra == 'google-genai'" },
- { name = "jinja2", specifier = ">=3.1.6" },
+ { name = "jinja2", specifier = ">=3.1.6,<3.2" },
{ name = "json2html", specifier = ">=1.3.0" },
{ name = "jsonschema", marker = "extra == 'dev'", specifier = ">=4.20.0" },
{ name = "kajson", specifier = "==0.7.1" },