Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
feature/Formatted-markdown-contract · L-261010-bd0748 (#1400)
Louis Choqueld1e3aaca0bce
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 8e03a41dd..04ed87887 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,5 +1,15 @@# Changelog+## [Unreleased]++### Added++- **Formatted Markdown for document engines, and a `markdown` filter for plain-data templates**: the document engine contract gains `format_markdown` in `pipelex.cogt.doc_gen.formatted_markdown`, which reads a Markdown text by the rules the built-in PDF engine prints it by and returns a `FormattedMarkdown`, plain data of paragraphs, headings, list items with their marker and the paragraph or heading they open with, code blocks, rules and tables whose cells carry their column's alignment, every block with the list depth and the quote depth it sits at and holding text spans with their bold, italic, strikethrough, code and link, whose `str()` is its plain text and which is false when the text formats into nothing, so an engine prints Markdown formatted in its own format. `make_plain_data_template_environment()` registers a `markdown` filter that returns one, `{{ invoice.notes | markdown }}`, which is false for an empty or a blank text, so `{% set notes = invoice.notes | markdown %}{% if notes %}…{{ notes }}…{% endif %}` converts the text once to test and print it, and takes an optional `finalize` through which an engine prints it, a value printing its plain text without one. Inside a template render the conversion is charged to the render's budget; called from an engine's own code it spends from a budget of its own, as large as one render's, and an overdraft raises `MarkdownFormattingBudgetError`.++### Fixed++- **A PDF of deeply nested emphasis prints**: the built-in PDF engine no longer fails with a `RecursionError` on a Markdown text whose emphasis nests hundreds deep, such as four hundred `*a ` before a word and four hundred ` c*` after it; inline markup nested past fifty levels is dropped and its text printed.+## [v0.81.0] - 2026-10-10### Addeddiff --git a/docs/errors/inference-and-providers.md b/docs/errors/inference-and-providers.mdindex b457a2a73..46a8176ec 100644--- a/docs/errors/inference-and-providers.md+++ b/docs/errors/inference-and-providers.md@@ -58,6 +58,7 @@ own page. Classes are grouped by subsystem.- [`LLMPromptSpecError`](llm-prompt-spec-error.md) — LLM prompt spec- [`LLMSettingRefusedError`](llm-setting-refused-error.md) — LLM setting refused- [`LLMSettingsValidationError`](llm-settings-validation-error.md) — LLM settings validation+- [`MarkdownFormattingBudgetError`](markdown-formatting-budget-error.md) — Markdown formatting budget exceeded- [`ModelChoiceNotFoundError`](model-choice-not-found-error.md) — Model choice not found- [`ModelDeckNotFoundError`](model-deck-not-found-error.md) — Model deck not found- [`ModelDeckPresetValidatonError`](model-deck-preset-validaton-error.md) — Model deck preset validatondiff --git a/docs/errors/markdown-formatting-budget-error.md b/docs/errors/markdown-formatting-budget-error.mdnew file mode 100644index 000000000..542d182d2--- /dev/null+++ b/docs/errors/markdown-formatting-budget-error.md@@ -0,0 +1,21 @@+---+title: "Markdown formatting budget exceeded"+description: "Reference for the `MarkdownFormattingBudgetError` Pipelex error class."+---++<!-- pipelex:generated -->++# Markdown formatting budget exceeded++Formatting a Markdown text outside any template render would spend more than the budget it gets of its own.++| Field | Value |+|---|---|+| `error_type` | `MarkdownFormattingBudgetError` |+| `title` | Markdown formatting budget exceeded |+| `type_uri` | `https://docs.pipelex.com/latest/errors/markdown-formatting-budget-error/` |+| `error_domain` | `input` |+| Defined in | `pipelex.cogt.doc_gen.exceptions` |+| Parent class | [`PipelexError`](pipelex-error.md) |++[Back to Error Reference](index.md)diff --git a/docs/under-the-hood/document-engine-plugins.md b/docs/under-the-hood/document-engine-plugins.mdindex 629ee3c38..169c7d255 100644--- a/docs/under-the-hood/document-engine-plugins.md+++ b/docs/under-the-hood/document-engine-plugins.md@@ -124,7 +124,7 @@ def render(self, *, job: RenderJob, resources: RenderResources) -> RenderedDocum`render` is synchronous and runs on a thread of a print pool of its own, never on the event loop's default executor, which the engine's reads need, while `RenderResources.load` hands each read back to the event loop the print started from. The print stage that calls it ends the print with the event every inference call ends with, `Inference call ends`, carrying no tokens and no cost since an engine reports none (see [Summary events](../tools/logging.md#summary-events)), logged on the coroutine that awaits the thread, so a print cancelled from outside ends `cancelled` while the thread finishes its render; an engine overrides `render` alone and logs nothing for the print itself. An engine keeps no state from one print into the next. It signals a document it cannot print by raising `DocGenRenderError`, which passes through; anything else it raises, beyond a Pipelex error, is reported as a `DocGenRenderError` naming the engine and the file.-The contract is what an engine uses in `pipelex.cogt.doc_gen`: the worker; the render job, its resources, the `LoadedResource` they return and the rendered document (`render_job.py`); the formats and sources (`doc_gen_format.py`); `DocGenRenderError`, the one error of `exceptions.py` an engine raises; the layout tree (`layout_tree.py`); the template check request and the findings a checker returns (`template_check.py`), with the input shapes the request carries (`InputShape` and `InputShapeKind` in `input_shape.py`); and the two modules of the next section. The rest of the package is Pipelex's own side of the print. An engine reaches Pipelex only through the contract and the registration seam every plugin uses, which also hands a worker its model spec, its backend, its SDK clients and its reporting delegate: anything else in Pipelex may move or change in an ordinary release, while a breaking change to the contract moves `PLUGIN_API_VERSION`. A piece an engine is missing is added to the contract, rather than imported from elsewhere in Pipelex.+The contract is what an engine uses in `pipelex.cogt.doc_gen`: the worker; the render job, its resources, the `LoadedResource` they return and the rendered document (`render_job.py`); the formats and sources (`doc_gen_format.py`); `DocGenRenderError`, the one error of `exceptions.py` an engine raises, and `MarkdownFormattingBudgetError`, the one the contract raises to an engine that formats Markdown outside a template render; the layout tree (`layout_tree.py`); the template check request and the findings a checker returns (`template_check.py`), with the input shapes the request carries (`InputShape` and `InputShapeKind` in `input_shape.py`); and the modules of the next section. The rest of the package is Pipelex's own side of the print. An engine reaches Pipelex only through the contract and the registration seam every plugin uses, which also hands a worker its model spec, its backend, its SDK clients and its reporting delegate: anything else in Pipelex may move or change in an ordinary release, while a breaking change to the contract moves `PLUGIN_API_VERSION`. A piece an engine is missing is added to the contract, rather than imported from elsewhere in Pipelex.### The media type of a file an engine reads@@ -150,9 +150,17 @@ A document should read the same whichever engine printed it, and a template shou- **`display_scalar(value=…)`** writes a scalar as text: blank for nothing, `Yes` or `No` for a boolean, a whole float without its decimals, a date as `2026-09-29`, a datetime as `2026-09-29 14:05`, and a time of day as `09:07`, each followed by the UTC offset it states (` UTC`, ` +02:00`), if any.- **`is_numeric_column(values=…)`** says whether a table's column holds numbers only, blanks aside, and at least one, a boolean not counting as a number: such a column is aligned right.-- **`markdown_as_html(markdown=…)`** converts a `MarkdownBlock` to an HTML fragment for an engine that writes HTML, with the one parser Pipelex reads Markdown with, as the `markdown` filter and the `Markdown` concept's HTML view do: CommonMark with tables and strikethrough, raw HTML shown as text, and only a URL with a scheme turned into a link, so `README.md` stays text. Called from an engine's own code, outside a template render, it is charged to no render budget, so an engine bounds what it converts itself.+- **`markdown_as_html(markdown=…)`** converts a `MarkdownBlock` to an HTML fragment for an engine that writes HTML, with the one parser Pipelex reads Markdown with, as the `markdown` filter and the `Markdown` concept's HTML view do: CommonMark with tables and strikethrough, raw HTML shown as text, and a bare URL turned into a link only when it has a scheme, so a bare `README.md` stays text. An explicit link follows markdown-it's own rule, keeping any target but a `javascript:`, `vbscript:`, `file:` or non-image `data:` one, so `[x](README.md)` is a link, and an image becomes an `<img>`, where `format_markdown` below keeps only `http`, `https` and `mailto` links and prints an image's alt text. Called from an engine's own code, outside a template render, it is charged to no render budget, unlike `format_markdown`, so an engine bounds what it converts itself.-`pipelex.cogt.doc_gen.template_environment` holds **`make_plain_data_template_environment()`**, the Jinja environment an engine fills a template file's tags in when it fills them itself, as a Word template's tags are filled through docxtpl. It is synchronous; sandboxed as every Pipelex template is, each render spending from a budget of its own (see [Template Sandbox](template-sandbox.md)); strict, so a missing value fails the render instead of printing as empty text; and it registers only Jinja's built-in filters, since the plain data it reads holds no stuff for Pipelex's filters to work on.+`pipelex.cogt.doc_gen.formatted_markdown` holds what an engine prints a Markdown text from when it formats it in its own format, as a Word or an Excel template fill does, rather than through HTML. The structure and the conversion live in `pipelex/tools/markdown/markdown_formatting.py`, where the plain-data templates' `markdown` filter reads them too, and the contract module names them in its `__all__`, so an engine imports them from the contract and never from `pipelex.tools`, which may move in an ordinary release:++- **`format_markdown(markdown=…)`** reads a Markdown text with the one parser Pipelex reads Markdown with, by the rules the built-in PDF engine prints it by, which both take from `pipelex/tools/markdown/markdown_rules.py`: CommonMark with tables and strikethrough; raw HTML shown as text; a soft line break read as a space and a hard one, two trailing spaces or a backslash, as a line break; bullets alternating `•` and `–` by depth and an ordered list numbered from its own start; only an `http`, `https` or `mailto` target kept as a link, any other printing its text alone; an image never fetched, printed as `[image: alt]` in italics; a table cell aligned as its column's delimiter row sets; inline markup nested past fifty levels, which markdown-it does not cap, dropped and its text kept, as the built-in engine drops it, so a text cannot overflow the stack; and a node the converter does not know printed as its text rather than failing the document.+- **`FormattedMarkdown`**, what it returns, is plain data, as the layout tree is: its `blocks`, in order, each a `FormattedParagraph`, a `FormattedHeading` with its `level`, a `FormattedListItem` with its `marker` as printed, such as `•` or `3.`, and the paragraph or heading it opens with, its `heading_level` set for a heading, a `FormattedCodeBlock` with its `lines`, a `FormattedRule`, or a `FormattedTable` of rows of cells, each row marked `is_header` or not and each cell carrying its column's `alignment`, `left`, `center`, `right` or `None`; and in the blocks that print text, their `spans`, each a `TextSpan` with its `bold`, `italic`, `strikethrough` and `code` flags and the `link` it opens, if any, or a `LineBreakSpan`. Every block has a `list_depth` and a `quote_depth`, the list items and the quotations it sits in, 0 outside any: a quotation is its blocks one quote deeper, so `> # Title` is a heading at quote depth 1, and the rest of a list item, a later paragraph, a nested list, a code block or a table, follows it at the item's list depth. `str()` of it is its plain text, each block on lines of its own, indented under the list item it sits in with list markers kept, table cells separated by tabs, and every other piece of markup gone, for a place that cannot show formatting; and it is false when it holds no block, as for an empty or a blank text.+- **Its budget.** Inside a template render, as the `markdown` filter below converts, the conversion is charged to the render's budget the way the HTML conversion is, for every character of its source and every cell of its tables before it parses and for its result, bounded from the parsed tokens, before it builds it, a link's address counted with every span that carries it, since writing the structure out, as JSON or as its repr, writes the address once per span, and an overdraft is the render's own `RenderBudgetExceededError`, a Jinja `TemplateError`, which a template fill catches with every other error a tag raises. Called from an engine's own code, outside any render, it spends from a budget of its own, as large as one render's, and an overdraft raises `MarkdownFormattingBudgetError` (from `exceptions.py`), which the engine reports as its own fill error, naming where the value goes.++`pipelex.cogt.doc_gen.template_environment` holds **`make_plain_data_template_environment()`**, the Jinja environment an engine fills a template file's tags in when it fills them itself, as a Word template's tags are filled through docxtpl. It is synchronous; sandboxed as every Pipelex template is, each render spending from a budget of its own (see [Template Sandbox](template-sandbox.md)); strict, so a missing value fails the render instead of printing as empty text; and it registers Jinja's built-in filters and one of Pipelex's, `markdown`, since the plain data it reads holds no stuff for Pipelex's other filters to work on.++`{{ invoice.notes | markdown }}` reads a text as Markdown into a `FormattedMarkdown`, as the HTML templates' `markdown` filter reads it into HTML: None gives an empty one, any other value is read as its string form, and an undefined value fails the render, as it would anywhere in a strict template. An empty one is false, so a template leaves out a section whose notes are None, empty or blank by converting them once into a variable and testing it, `{% set notes = invoice.notes | markdown %}{% if notes %}…{{ notes }}…{% endif %}`: every conversion is charged to the render's budget, so testing `invoice.notes | markdown` and then printing `invoice.notes | markdown` converts the text twice and halves the length a render affords. How a `FormattedMarkdown` prints is the engine's business, so `make_plain_data_template_environment(finalize=…)` takes a function applied to every value a tag prints, before the environment converts it to text: an engine turns a `FormattedMarkdown` into its own form there and hands every other value back as it is. The environment wraps it to charge what is printed, which is why it is passed at construction rather than assigned afterwards. Without one, a `FormattedMarkdown` prints its plain text, so a place that cannot format, such as a document's properties, prints clean text rather than asterisks.---diff --git a/docs/under-the-hood/template-sandbox.md b/docs/under-the-hood/template-sandbox.mdindex 6feb6040c..e38aced55 100644--- a/docs/under-the-hood/template-sandbox.md+++ b/docs/under-the-hood/template-sandbox.md@@ -79,7 +79,7 @@ The environment charges through the sandbox's hooks and one rewrite, all in `Pip- **Filters and tests** are wrapped before a template compiles. A filter or a test registered on the environment must have a cost in `FILTER_COSTS` or `TEST_COSTS` (`pipelex/tools/jinja2/jinja2_render_costs.py`), or the first compile fails with a `TypeError` naming it.- **Printed values** go through the environment's `finalize`, which charges the text it produces. A caller's own `finalize` is passed to the constructor and runs first.- **What no hook reaches**, which is loop iteration, `~`, comparisons, slices, list, tuple and dict literals and the template's static text, is routed through internal charging filters by a rewrite of the parsed template (`pipelex/tools/jinja2/jinja2_render_rewrite.py`). The rewrite changes what is charged, never what a template renders. A tuple used as a key, in a dict literal or an item read, is also charged for hashing, since a tuple hashes every element again each time.-- **Converting Markdown to HTML** is charged where it happens, in `pipelex/tools/markdown/markdown_parser.py`, since markupsafe converts a Markdown value through its `__html__` wherever an HTML template prints, escapes, joins or formats one, and no hook sees that call. The template makes its budget the active one while it renders, and the conversion charges it 2,048 units for every character of its source and every cell of its tables before it parses, then checks that its output, bounded from the parsed tokens, fits what is left before it renders, since a reference link is printed at every use. A conversion costs microseconds a character, so a render converts at most about 65,000 characters of Markdown, a document printed twice counting twice.+- **Converting Markdown to HTML** is charged where it happens, in `pipelex/tools/markdown/markdown_parser.py`, since markupsafe converts a Markdown value through its `__html__` wherever an HTML template prints, escapes, joins or formats one, and no hook sees that call. The template makes its budget the active one while it renders, and the conversion charges it 2,048 units for every character of its source and every cell of its tables before it parses, then checks that its output, bounded from the parsed tokens, fits what is left before it renders, since a reference link is printed at every use. A conversion costs microseconds a character, so a render converts at most about 65,000 characters of Markdown, a document printed twice counting twice. Formatting Markdown for a document engine, which the `markdown` filter of plain-data templates does, is charged the same way, in `pipelex/tools/markdown/markdown_formatting.py`: both conversions parse through `charged_parse`, which holds the order of the charges.The budget is the same for every template category and is not a setting in `pipelex.toml`: it is a safety bound, not a tuning knob. `PipelexTemplateEnvironment` takes it as the `render_budget_units` constructor argument, which the tests use to exercise small budgets.diff --git a/pipelex/cogt/doc_gen/exceptions.py b/pipelex/cogt/doc_gen/exceptions.pyindex 4c6e18448..059affbe3 100644--- a/pipelex/cogt/doc_gen/exceptions.py+++ b/pipelex/cogt/doc_gen/exceptions.py@@ -128,3 +128,19 @@ class DocGenRenderError(PipelexError):"""error_domain = ErrorDomain.RUNTIME+++class MarkdownFormattingBudgetError(PipelexError):+ """Formatting a Markdown text outside any template render would spend more than the budget it gets of its own.++ `format_markdown` (`formatted_markdown.py`) raises it when an engine converts a value from its own code, as an+ Excel fill does, and the text, its tables or what they format into is too large; inside a template render, the+ render's own `RenderBudgetExceededError` is raised instead. The Markdown is the run's input, so the fault is the+ caller's, and the message names only the text's length and the budget, never the text. It is part of the document+ engine contract (`pipelex/plugins/contract.py`): an engine reports it as its own fill error, naming where the value+ goes.+ """++ error_domain = ErrorDomain.INPUT+ _authors_caller_facing_message = True+ _declared_title = "Markdown formatting budget exceeded"diff --git a/pipelex/cogt/doc_gen/formatted_markdown.py b/pipelex/cogt/doc_gen/formatted_markdown.pynew file mode 100644index 000000000..e58dddcd5--- /dev/null+++ b/pipelex/cogt/doc_gen/formatted_markdown.py@@ -0,0 +1,90 @@+"""The document engine contract's formatted Markdown, for an engine that prints Markdown in its own format.++An engine that fills a template of its own format, a Word document's tags or an Excel workbook's cells, prints a text+written in Markdown from `format_markdown`, rather than from markdown-it's syntax tree or a converter of its own, so+it reads Markdown by the rules the built-in PDF engine reads it by: CommonMark with tables and strikethrough, raw HTML+shown as text, only an `http`, `https` or `mailto` target kept as a link, and an image never fetched, printing+`[image: alt]`. The HTML conversion (`layout_display.markdown_as_html`) follows markdown-it's own rules instead, which+keep a link to any target but a `javascript:`, `vbscript:`, `file:` or non-image `data:` one and show an image as an+`<img>`.++The result, a `FormattedMarkdown`, is plain data: an ordered list of blocks, each holding the spans it prints and the+list and quote depths it sits at. The structure and the conversion live in `pipelex/tools/markdown/`+(`markdown_formatting.py`, by the rules of `markdown_rules.py`), where the plain-data templates' `markdown` filter+reads them too, and this module names them for an engine (`__all__`): an engine imports them from here, the contract,+and never from `tools`, which may move in an ordinary release.++Inside a template render, as the `markdown` filter of plain-data templates converts, the conversion is charged to the+render's budget the way the HTML conversion is (`markdown_parser.py`), and an overdraft is the render's own+`RenderBudgetExceededError`. Outside any render, as an engine converts from its own code, it spends from a budget of+its own, as large as one render's, and an overdraft raises `MarkdownFormattingBudgetError`.++This module is part of the document engine contract (`pipelex/plugins/contract.py`), and it imports no engine+library, so a plugin's engine can use it without loading ReportLab.+"""++from pipelex.cogt.doc_gen.exceptions import MarkdownFormattingBudgetError+from pipelex.tools.jinja2.jinja2_render_budget import DEFAULT_RENDER_BUDGET_UNITS, RenderBudget, RenderBudgetExceededError, active_render_budget+from pipelex.tools.markdown.markdown_formatting import (+ FormattedBlock,+ FormattedCodeBlock,+ FormattedHeading,+ FormattedListItem,+ FormattedMarkdown,+ FormattedParagraph,+ FormattedRule,+ FormattedSpan,+ FormattedTable,+ FormattedTableCell,+ FormattedTableRow,+ LineBreakSpan,+ TextSpan,+ format_markdown_within_budget,+ spans_text,+)+from pipelex.tools.markdown.markdown_rules import CellAlignment++__all__ = [+ "CellAlignment",+ "FormattedBlock",+ "FormattedCodeBlock",+ "FormattedHeading",+ "FormattedListItem",+ "FormattedMarkdown",+ "FormattedParagraph",+ "FormattedRule",+ "FormattedSpan",+ "FormattedTable",+ "FormattedTableCell",+ "FormattedTableRow",+ "LineBreakSpan",+ "TextSpan",+ "format_markdown",+ "spans_text",+]+++def format_markdown(*, markdown: str) -> FormattedMarkdown:+ """A Markdown text as the blocks and spans an engine prints, read by the rules the built-in PDF engine reads it by.++ The conversion is charged before it parses, for every character of its source and every cell of its tables, and+ its result, bounded from the parsed tokens, must fit what is left before it is built. Inside a template render it+ spends from the render's budget; outside any, from a budget of its own, as large as one render's.++ Raises:+ RenderBudgetExceededError: inside a template render, the conversion would overdraw the render's budget.+ MarkdownFormattingBudgetError: outside any render, the conversion would spend more than the budget it gets+ of its own.+ """+ budget = active_render_budget()+ if budget is not None:+ return format_markdown_within_budget(markdown_text=markdown, budget=budget)+ own_budget = RenderBudget(total=DEFAULT_RENDER_BUDGET_UNITS)+ try:+ return format_markdown_within_budget(markdown_text=markdown, budget=own_budget)+ except RenderBudgetExceededError as exc:+ msg = (+ f"Formatting a Markdown text of {len(markdown):,} characters would spend more than the {own_budget.total:,} work units "+ "a conversion outside a template render may spend: its text, its tables or what they format into is too large."+ )+ raise MarkdownFormattingBudgetError(msg) from excdiff --git a/pipelex/cogt/doc_gen/layout_display.py b/pipelex/cogt/doc_gen/layout_display.pyindex 6557cbba2..c3cc93eb0 100644--- a/pipelex/cogt/doc_gen/layout_display.py+++ b/pipelex/cogt/doc_gen/layout_display.py@@ -69,8 +69,11 @@ def markdown_as_html(*, markdown: str) -> str:"""Markdown as an HTML fragment, parsed as the `markdown` filter, the `Markdown` concept and the built-in engine parse it.The parser reads CommonMark with tables and strikethrough; raw HTML in the source comes out escaped, so it- shows as text rather than entering the document; and only a URL with a scheme becomes a link, so `README.md`- or `www.example.com` stays text.+ shows as text rather than entering the document; and a bare URL becomes a link only when it has a scheme, so a+ bare `README.md` or `www.example.com` stays text. An explicit link follows markdown-it's own rule, keeping any+ target but a `javascript:`, `vbscript:`, `file:` or non-image `data:` one, so `[x](README.md)` is a link, and an+ image becomes an `<img>`: an engine that formats Markdown itself keeps only `http`, `https` and `mailto` links and+ prints an image's alt text instead (`formatted_markdown.py`).A conversion is charged to the budget of the template render running at the time, so one an engine makes fromits own code, outside any template render, is charged to none: the engine bounds what it converts and prints.diff --git a/pipelex/cogt/doc_gen/template_environment.py b/pipelex/cogt/doc_gen/template_environment.pyindex 586049f7c..dbe4574c3 100644--- a/pipelex/cogt/doc_gen/template_environment.py+++ b/pipelex/cogt/doc_gen/template_environment.py@@ -3,20 +3,26 @@An engine that fills a template file itself, such as a Word template's tags filled through docxtpl, renders them inthis environment rather than in one of its own, so a document template gets the sandbox every other Pipelex templategets. The values it reads are the plain data of the step's inputs (`plain_data.py`), with no stuff in them, so the-environment registers none of Pipelex's filters, which are made for stuff.+environment registers none of Pipelex's filters made for stuff, and one made for plain data: `markdown`, which reads a+text as Markdown for the engine to print formatted (`formatted_markdown.py`).This module is part of the document engine contract (`pipelex/plugins/contract.py`)."""+from collections.abc import Callable+from typing import Any+from jinja2 import BaseLoaderfrom pipelex.tools.jinja2.jinja2_environment import make_jinja2_env_from_loader+from pipelex.tools.jinja2.jinja2_filters import markdown_to_formatted+from pipelex.tools.jinja2.jinja2_models import Jinja2FilterNamefrom pipelex.tools.jinja2.jinja2_sandbox import PipelexTemplateEnvironmentfrom pipelex.tools.jinja2.template_category import TemplateCategory-def make_plain_data_template_environment() -> PipelexTemplateEnvironment:- """A new environment for a template of plain data: synchronous, sandboxed, strict about missing values, filterless.+def make_plain_data_template_environment(*, finalize: Callable[..., Any] | None = None) -> PipelexTemplateEnvironment:+ """A new environment for a template of plain data: synchronous, sandboxed, strict about missing values, with Jinja's filters and `markdown`.- **Synchronous**: a template renders with `render`, as a library like docxtpl calls it.- **Sandboxed** as every Pipelex template is (`jinja2_sandbox.py`): a template reads data and calls the@@ -24,13 +30,23 @@ def make_plain_data_template_environment() -> PipelexTemplateEnvironment:from a budget of its own, an overdraft raising `RenderBudgetExceededError`.- **Strict**: a missing value fails the render with `jinja2.exceptions.UndefinedError` rather than printingas empty text, as in every document template.- - **Without Pipelex's filters**: Jinja's built-in filters only, under the sandbox.+ - **Jinja's built-in filters and `markdown`**, under the sandbox: `{{ invoice.notes | markdown }}` reads a text as+ Markdown into a `FormattedMarkdown` (`formatted_markdown.py`), None into an empty one, its conversion charged to+ the render's budget.++ `finalize`, when given, is applied to every value a tag prints before the environment converts it to text, so an+ engine turns a printed `FormattedMarkdown` into its own form there and hands every other value back as it is.+ Without one, a `FormattedMarkdown` prints its plain text. It is handed to the environment at construction, which a+ Pipelex environment requires, since the environment wraps it to charge what is printed.It loads no template by name, so a template is built from its source with `from_string`."""- return make_jinja2_env_from_loader(+ environment = make_jinja2_env_from_loader(template_category=TemplateCategory.BASIC,loader=BaseLoader(),enable_async=False,+ finalize=finalize,is_undefined_strict=True,)+ environment.filters[Jinja2FilterName.MARKDOWN] = markdown_to_formatted+ return environmentdiff --git a/pipelex/plugins/contract.py b/pipelex/plugins/contract.pyindex dbbcdb7d3..6b6b04a92 100644--- a/pipelex/plugins/contract.py+++ b/pipelex/plugins/contract.py@@ -34,6 +34,12 @@ if TYPE_CHECKING:# ``InputShapeKind``). Two modules joined it with v6, as additions that would have needed no bump of their own: the# rules by which every engine shows the tree's values (``layout_display``: ``display_scalar``, ``is_numeric_column``,# ``markdown_as_html``) and the environment an engine fills a template of plain data in (``template_environment``).+# More joined under v6 on the same reasoning, as additions: the structure an engine prints a Markdown text from in its+# own format (``formatted_markdown``: ``format_markdown``, and ``FormattedMarkdown`` with its blocks and spans, which+# the module names from ``pipelex/tools/markdown/markdown_formatting.py``), the error it raises when a conversion+# outside a template render overdraws the budget it gets of its own (``MarkdownFormattingBudgetError``), and, in the+# plain-data environment, the ``markdown`` filter, which returns a ``FormattedMarkdown``, and the optional+# ``finalize`` keyword an engine prints one through.## ``add_internal_model`` and ``add_doc_gen_default`` — plain data a plugin declares for the model manager to merge at# boot: a model of the internal backend, as the table a backend file would hold, and the model deck's default engine fordiff --git a/pipelex/providers/reportlab/markdown_flowables.py b/pipelex/providers/reportlab/markdown_flowables.pyindex 67ea63e30..a21ce6a57 100644--- a/pipelex/providers/reportlab/markdown_flowables.py+++ b/pipelex/providers/reportlab/markdown_flowables.py@@ -1,8 +1,12 @@"""Markdown as ReportLab flowables, for the built-in PDF engine.The Markdown is parsed by the one parser Pipelex formats Markdown with (`get_markdown_parser`), so a PDF reads it-exactly as the `markdown` filter of HTML templates does, and the converter walks markdown-it's syntax tree rather-than its flat token stream, so a nested list lives inside its item and each list numbers its own items.+as the `markdown` filter of HTML templates parses it, and the converter walks markdown-it's syntax tree rather than+its flat token stream, so a nested list lives inside its item and each list numbers its own items. The tree is built+with its inline nesting capped (`markdown_syntax_tree`), so emphasis nested hundreds deep prints its text rather than+overflowing the stack. The rules it shares with the formatting the engines of the other formats print from, the link+rule, the bullets by depth, a list's start, a heading's level, a code block's text, a table cell's alignment and the+plain text of nodes, are in `pipelex/tools/markdown/markdown_rules.py`.What it prints:@@ -18,8 +22,6 @@ become links: any other target prints its text alone. A Markdown image is neveritalics, as `[image: alt]`. A node the converter does not know prints its text, and never fails the document."""-from urllib.parse import urlsplit-from markdown_it.tree import SyntaxTreeNodefrom reportlab.lib.styles import ParagraphStyle # type: ignore[import-untyped]from reportlab.pdfbase import pdfmetrics # type: ignore[import-untyped]@@ -43,12 +45,18 @@ from pipelex.providers.reportlab.pdf_elements import (heading_break,measure_column,)-from pipelex.tools.markdown.markdown_parser import get_markdown_parser--LINKED_SCHEMES = frozenset({"http", "https", "mailto"})+from pipelex.tools.markdown.markdown_parser import get_markdown_parser, markdown_syntax_tree+from pipelex.tools.markdown.markdown_rules import (+ CellAlignment,+ cell_alignment,+ code_text,+ heading_level,+ is_linked_href,+ list_bullet,+ list_start,+ plain_text,+)-# The bullets of nested bullet lists, by depth, alternating: both are in the bundled sans face.-_BULLETS = ("•", "–")_MINIMUM_LIST_INDENT = 14.0_BULLET_GAP = 6.0_QUOTE_INDENT = 2 * BOX_PADDING@@ -57,7 +65,7 @@ _MINIMUM_CODE_LINE_LENGTH = 20def markdown_to_flowables(*, markdown_text: str, styles: PdfStyles, available_width: float) -> list[Flowable]:"""The flowables that print a Markdown text in a frame `available_width` points wide."""- tree = SyntaxTreeNode(get_markdown_parser().parse(markdown_text))+ tree = markdown_syntax_tree(tokens=get_markdown_parser().parse(markdown_text))return markdown_nodes_to_flowables(nodes=tree.children, styles=styles, available_width=available_width)@@ -83,7 +91,7 @@ class _MarkdownWriter:style = self._styles.list_body if list_depth else self._styles.bodyreturn [Paragraph(inline_markup(nodes=node.children), style)]case "heading":- heading = Paragraph(inline_markup(nodes=node.children), self._styles.heading(level=_heading_level(tag=node.tag)))+ heading = Paragraph(inline_markup(nodes=node.children), self._styles.heading(level=heading_level(tag=node.tag)))if is_contained:return [heading]return [heading_break(), heading]@@ -111,8 +119,9 @@ class _MarkdownWriter:if not items:return []is_ordered = node.type == "ordered_list"- start = _list_start(node=node) if is_ordered else 1- bullet = _BULLETS[list_depth % len(_BULLETS)]+ start = list_start(start=node.attrs.get("start")) if is_ordered else 1+ # Both bullets are in the bundled sans face.+ bullet = list_bullet(depth=list_depth + 1)labels = [f"{start + index}." for index in range(len(items))] if is_ordered else [bullet]font_size = self._styles.body.fontSizelabel_width = max(pdfmetrics.stringWidth(label, SANS_FONT, font_size) for label in labels)@@ -160,24 +169,24 @@ class _MarkdownWriter:for column_index in range(column_count):if column_index < len(row):cell = row[column_index]- style = self._cell_style(alignment=_cell_alignment(node=cell), is_header=is_header)+ style = self._cell_style(alignment=cell_alignment(node=cell), is_header=is_header)cells.append(Paragraph(inline_markup(nodes=cell.children), style))else:cells.append("")table_rows.append(cells)return [data_table(rows=table_rows, col_widths=col_widths, has_header=header_row is not None, right_aligned_columns=[])]- def _cell_style(self, *, alignment: str | None, is_header: bool) -> ParagraphStyle:+ def _cell_style(self, *, alignment: CellAlignment | None, is_header: bool) -> ParagraphStyle:match alignment:case "right":return self._styles.cell_header_right if is_header else self._styles.cell_rightcase "center":return self._styles.cell_header_center if is_header else self._styles.cell_center- case _:+ case "left" | None:return self._styles.cell_header if is_header else self._styles.celldef _code(self, *, text: str, available_width: float) -> list[Flowable]:- code = text.rstrip("\n").expandtabs(4)+ code = code_text(content=text)if not code.strip():return []code_style = self._styles.code@@ -225,56 +234,3 @@ def _link_markup(*, node: SyntaxTreeNode) -> str:if not isinstance(href, str) or not is_linked_href(href=href):return labelreturn f'<a href="{escape_attribute(value=href)}" color="{LINK_COLOR_HEX}">{label}</a>'---def is_linked_href(*, href: str) -> bool:- """Whether a link target becomes a link in the PDF: only `http`, `https` and `mailto` do."""- try:- scheme = urlsplit(href.strip()).scheme- except ValueError:- return False- return scheme.lower() in LINKED_SCHEMES---def plain_text(*, nodes: list[SyntaxTreeNode]) -> str:- """The text of Markdown nodes without their markup, as it reads once printed: what a table column is measured by."""- parts: list[str] = []- for node in nodes:- match node.type:- case "softbreak":- parts.append(" ")- case "hardbreak":- parts.append("\n")- case "image":- parts.append(f"[image: {plain_text(nodes=node.children).strip() or node.content}]")- case _:- if node.children:- parts.append(plain_text(nodes=node.children))- else:- parts.append(node.content)- return "".join(parts)---def _heading_level(*, tag: str) -> int:- if len(tag) == 2 and tag[0] == "h" and tag[1].isdigit():- return int(tag[1])- return 6---def _list_start(*, node: SyntaxTreeNode) -> int:- start = node.attrs.get("start", 1)- try:- return int(start)- except (TypeError, ValueError):- return 1---def _cell_alignment(*, node: SyntaxTreeNode) -> str | None:- style = node.attrs.get("style")- if not isinstance(style, str):- return None- if "text-align:right" in style:- return "right"- if "text-align:center" in style:- return "center"- return Nonediff --git a/pipelex/tools/jinja2/jinja2_filters.py b/pipelex/tools/jinja2/jinja2_filters.pyindex 32607ed12..02227a669 100644--- a/pipelex/tools/jinja2/jinja2_filters.py+++ b/pipelex/tools/jinja2/jinja2_filters.py@@ -9,9 +9,11 @@ from pipelex.tools.jinja2.exceptions import Jinja2ContextErrorfrom pipelex.tools.jinja2.html_renderable import HtmlRenderablefrom pipelex.tools.jinja2.image_registry import ImageRegistryfrom pipelex.tools.jinja2.jinja2_models import Jinja2ContextKey+from pipelex.tools.jinja2.jinja2_render_budget import DEFAULT_RENDER_BUDGET_UNITS, RenderBudget, active_render_budgetfrom pipelex.tools.jinja2.renderable_dispatch import type_implementsfrom pipelex.tools.jinja2.tag_renderable import TagRenderablefrom pipelex.tools.jinja2.text_format_renderable import TextFormatRenderable+from pipelex.tools.markdown.markdown_formatting import FormattedMarkdown, format_markdown_within_budgetfrom pipelex.tools.markdown.markdown_parser import render_markdown_as_htmlfrom pipelex.tools.templating.templating_style import TagStylefrom pipelex.tools.templating.text_format import TextFormat@@ -217,3 +219,27 @@ def markdown_to_html(value: Any) -> Markup:# `str()` of a strict undefined raises, and of a lenient one is empty.source_text = value if isinstance(value, str) else str(value)return Markup(render_markdown_as_html(source_text)) # ruff: ignore[unsafe-markup-use] - raw HTML in the source is escaped by the parser+++def markdown_to_formatted(value: Any) -> FormattedMarkdown:+ """Read a text as Markdown for a document engine to print: the `markdown` filter of plain-data templates.++ It is for Markdown held in a text field, such as an invoice's notes, in a template a document engine fills itself,+ such as a Word template's tags. It reads its value as the HTML templates' filter does (`markdown_to_html`): None+ gives nothing, a text is read as Markdown, anything else as its string form, and an undefined value prints the way+ the template prints one anywhere else, which fails a strict template's render.++ The result is a `FormattedMarkdown` (`markdown_formatting.py`, which the document engine contract names in+ `pipelex/cogt/doc_gen/formatted_markdown.py`), which the engine's `finalize` turns into its own form when it+ prints, which prints as its plain text where nothing does, and which is false when it holds nothing, as for None+ or a blank text. The conversion is charged to the render's budget, or, called outside any render, to a budget of+ its own, as large as one render's.+ """+ if value is None:+ return FormattedMarkdown(blocks=[])+ # `str()` of a strict undefined raises, and of a lenient one is empty.+ source_text = value if isinstance(value, str) else str(value)+ budget = active_render_budget()+ if budget is None:+ budget = RenderBudget(total=DEFAULT_RENDER_BUDGET_UNITS)+ return format_markdown_within_budget(markdown_text=source_text, budget=budget)diff --git a/pipelex/tools/jinja2/jinja2_render_costs.py b/pipelex/tools/jinja2/jinja2_render_costs.pyindex e37f1f13f..ea74eefce 100644--- a/pipelex/tools/jinja2/jinja2_render_costs.py+++ b/pipelex/tools/jinja2/jinja2_render_costs.py@@ -24,7 +24,7 @@ from jinja2.runtime import Undefinedfrom markupsafe import Markupfrom pydantic import BaseModel-from pipelex.tools.jinja2.jinja2_filters import escape_script_tag, markdown_to_html, tag, text_format+from pipelex.tools.jinja2.jinja2_filters import escape_script_tag, markdown_to_formatted, markdown_to_html, tag, text_formatfrom pipelex.tools.jinja2.jinja2_render_budget import (ELEMENT_UNITS,ESCAPE_FACTOR,@@ -1023,9 +1023,10 @@ FILTER_COSTS: Final[dict[Callable[..., Any], OperationCost]] = {_FILTERS["xmlattr"]: OperationCost(estimate=_escaped_text),text_format: LINEAR,tag: _CONVERTS_TO_TEXT,- # The conversion charges itself, by its source and its output (`markdown_parser.py`), once the value- # is converted to the text it reads.+ # Each conversion charges itself, by its source and its output (`markdown_parser.py` for HTML,+ # `markdown_formatting.py` for a document engine), once the value is converted to the text it reads.markdown_to_html: _CONVERTS_TO_TEXT,+ markdown_to_formatted: _CONVERTS_TO_TEXT,escape_script_tag: OperationCost(estimate=_script_tag_escaped),with_images: OperationCost(estimate=_rendered_with_images),}diff --git a/pipelex/tools/markdown/markdown_formatting.py b/pipelex/tools/markdown/markdown_formatting.pynew file mode 100644index 000000000..083761902--- /dev/null+++ b/pipelex/tools/markdown/markdown_formatting.py@@ -0,0 +1,488 @@+"""Markdown formatted into plain data, for a document engine that prints it in its own format rather than through HTML.++An engine that fills a template of its own format, a Word document's tags or an Excel workbook's cells, prints a text+written in Markdown from a `FormattedMarkdown`, rather than from markdown-it's syntax tree or a converter of its own,+so it reads Markdown by the rules the built-in PDF engine reads it by (`markdown_rules.py`). The structure is plain+data: an ordered list of blocks, each holding the spans it prints.++- **Blocks**: a paragraph, a heading with its level, a list item with its marker as printed and its first paragraph+ or heading, a code block's lines, a horizontal rule, and a table of rows of cells, its header row marked and each+ cell aligned as its column is. Every block has a list depth and a quote depth, the list items and the quotations+ it sits in, 0 outside any: a quoted paragraph is a paragraph one quotation deep, and a code block inside a list+ item is a code block one list deep.+- **Spans**: a text with its bold, italic, strikethrough and code flags and the address it links to, if any, or a line+ break.+- **Plain text**: `str()` of the result is its text, each block on lines of its own, indented under the list item it+ sits in, list markers kept and every other piece of markup gone, for a place that cannot show formatting; and the+ result is true when it holds a block, so a template tests an empty or a blank text by converting it once,+ `{% set notes = invoice.notes | markdown %}{% if notes %}…{{ notes }}…{% endif %}`, rather than converting it a+ second time, and paying for it twice, to print it.++It parses with the one parser Pipelex reads Markdown with (`get_markdown_parser`), its inline nesting capped+(`markdown_syntax_tree`), and reads it by the rules the built-in PDF engine follows:++- CommonMark with tables and strikethrough, and raw HTML shown as text;+- a soft line break read as a space, and a hard one, two trailing spaces or a backslash, as a line break;+- bullets alternating a bullet and an en dash by depth, and an ordered list numbered from its own `start`;+- only an `http`, `https` or `mailto` target kept as a link, any other printing its text alone;+- an image never fetched, printing `[image: alt]` in italics;+- a node the converter does not know printing its text, and never failing the document.++The conversion spends from the budget it is given, charged as the HTML conversion is (`charged_parse`), and an+overdraft raises `RenderBudgetExceededError`. The document engine contract's `format_markdown`+(`pipelex/cogt/doc_gen/formatted_markdown.py`) gives it the active render's budget or one of its own.+"""++from collections.abc import Sequence+from typing import TYPE_CHECKING, Annotated, Final, Literal++from markdown_it.tree import SyntaxTreeNode+from pydantic import BaseModel, ConfigDict, Field+from typing_extensions import override++from pipelex.tools.jinja2.jinja2_render_budget import RenderBudget+from pipelex.tools.markdown.markdown_parser import charged_parse, markdown_syntax_tree+from pipelex.tools.markdown.markdown_rules import (+ CellAlignment,+ cell_alignment,+ code_text,+ heading_level,+ is_linked_href,+ list_bullet,+ list_start,+ plain_text,+)++if TYPE_CHECKING:+ from markdown_it.token import Token++_FORMATTING: Final = "formatting Markdown"++# What one part of the result takes at most besides its text, a span, a block, a row or a cell, with the syntax-tree+# node it is read from: a frozen pydantic model and its place in a list measure under six hundred bytes.+_PART_UNITS: Final = 1024++# The indentation of a block's text by list depth, in the plain text.+_PLAIN_INDENT: Final = " "+++########################################################################################+# The structure+########################################################################################+++class TextSpan(BaseModel):+ """A run of text, with the formatting the Markdown gives it."""++ model_config = ConfigDict(frozen=True, extra="forbid")++ kind: Literal["text"] = "text"+ text: str+ bold: bool = False+ italic: bool = False+ strikethrough: bool = False+ code: bool = Field(default=False, description="Inline code, which an engine prints in a monospace font")+ link: str | None = Field(default=None, description="The address the text links to: only an http, https or mailto one is kept")+++class LineBreakSpan(BaseModel):+ """A hard line break inside a block: two trailing spaces or a backslash at the end of a line."""++ model_config = ConfigDict(frozen=True, extra="forbid")++ kind: Literal["line_break"] = "line_break"+++FormattedSpan = Annotated[TextSpan | LineBreakSpan, Field(discriminator="kind")]+++class _PlacedBlock(BaseModel):+ """Where a block sits: inside how many list items, and inside how many quotations."""++ model_config = ConfigDict(frozen=True, extra="forbid")++ list_depth: int = Field(default=0, ge=0, description="How many list items the block sits in: 0 outside any list")+ quote_depth: int = Field(default=0, ge=0, description="How many quotations the block sits in: 0 outside any quotation")+++class FormattedParagraph(_PlacedBlock):+ """A paragraph: inside a list item, one after the paragraph or heading the item's marker prints with."""++ kind: Literal["paragraph"] = "paragraph"+ spans: list[FormattedSpan]+++class FormattedHeading(_PlacedBlock):+ kind: Literal["heading"] = "heading"+ level: int = Field(ge=1, le=6, description="From 1, the largest, to 6")+ spans: list[FormattedSpan]+++class FormattedListItem(_PlacedBlock):+ """A list item's marker, with the paragraph or the heading it opens with; the rest of the item follows as blocks+ one list deeper than the list it sits in, with the item's `list_depth`.+ """++ kind: Literal["list_item"] = "list_item"+ list_depth: int = Field(default=1, ge=1, description="How many lists the item sits in, its own included: 1 for an item of a top-level list")+ marker: str = Field(description="The bullet or the number as printed, such as '•' or '3.'")+ heading_level: int | None = Field(+ default=None,+ ge=1,+ le=6,+ description="The level of the heading the item opens with, or None when it opens with a paragraph or with nothing to print beside its marker",+ )+ spans: list[FormattedSpan] = Field(+ description="The text printed beside the marker: empty when the item opens with a list, a code block or a table"+ )+++class FormattedCodeBlock(_PlacedBlock):+ """A fenced or indented code block, its lines as written, tabs expanded to four spaces."""++ kind: Literal["code_block"] = "code_block"+ lines: list[str]+++class FormattedRule(_PlacedBlock):+ kind: Literal["rule"] = "rule"+++class FormattedTableCell(BaseModel):+ model_config = ConfigDict(frozen=True, extra="forbid")++ spans: list[FormattedSpan]+ alignment: CellAlignment | None = Field(default=None, description="The alignment the column's delimiter row sets, or None when it sets none")+++class FormattedTableRow(BaseModel):+ """A row of a table, every row holding as many cells as the header has columns."""++ model_config = ConfigDict(frozen=True, extra="forbid")++ is_header: bool+ cells: list[FormattedTableCell]+++class FormattedTable(_PlacedBlock):+ kind: Literal["table"] = "table"+ rows: list[FormattedTableRow]+++FormattedBlock = Annotated[+ FormattedParagraph | FormattedHeading | FormattedListItem | FormattedCodeBlock | FormattedRule | FormattedTable,+ Field(discriminator="kind"),+]+++class FormattedMarkdown(BaseModel):+ """A Markdown text as its blocks, in order, for an engine to print in its own format.++ `str()` is its plain text, and it is true when it holds a block.+ """++ model_config = ConfigDict(frozen=True, extra="forbid")++ blocks: list[FormattedBlock]++ @override+ def __str__(self) -> str:+ """The plain text: each block on lines of its own, indented two spaces for every list item it sits in, a list+ item's marker printed before its text, table cells separated by tabs, a code block's lines as written, a rule+ left out, and every other piece of markup, a quotation's included, gone.+ """+ lines: list[str] = []+ for block in self.blocks:+ indent = _PLAIN_INDENT * block.list_depth+ match block:+ case FormattedListItem():+ first_prefix = f"{_PLAIN_INDENT * (block.list_depth - 1)}{block.marker} "+ lines.extend(_indented_lines(text=spans_text(spans=block.spans), first_prefix=first_prefix, prefix=indent))+ case FormattedParagraph() | FormattedHeading():+ lines.extend(_indented_lines(text=spans_text(spans=block.spans), first_prefix=indent, prefix=indent))+ case FormattedCodeBlock():+ lines.extend(_indented_lines(text="\n".join(block.lines), first_prefix=indent, prefix=indent))+ case FormattedRule():+ pass+ case FormattedTable():+ rows = "\n".join("\t".join(spans_text(spans=cell.spans) for cell in row.cells) for row in block.rows)+ lines.extend(_indented_lines(text=rows, first_prefix=indent, prefix=indent))+ return "\n".join(lines)++ def __bool__(self) -> bool:+ """Whether the text formats into anything: an empty or a blank text holds no block."""+ return bool(self.blocks)+++def spans_text(*, spans: Sequence[FormattedSpan]) -> str:+ """The text of spans without their formatting, a line break as a line feed."""+ parts: list[str] = []+ for span in spans:+ match span:+ case TextSpan():+ parts.append(span.text)+ case LineBreakSpan():+ parts.append("\n")+ return "".join(parts)+++def _indented_lines(*, text: str, first_prefix: str, prefix: str) -> list[str]:+ """The lines of a block's text, the first after `first_prefix` and every other after `prefix`, an empty one bare."""+ lines: list[str] = []+ for index, line in enumerate(text.split("\n")):+ line_prefix = first_prefix if index == 0 else prefix+ lines.append(f"{line_prefix}{line}" if line else line_prefix.rstrip())+ return lines+++########################################################################################+# Formatting+########################################################################################+++def format_markdown_within_budget(*, markdown_text: str, budget: RenderBudget) -> FormattedMarkdown:+ """A Markdown text as the blocks and spans an engine prints, spending from `budget`.++ The conversion is charged before it parses, for every character of its source and every cell of its tables, and+ its result, bounded from the parsed tokens, must fit what is left before it is built (`charged_parse`); what it+ built is charged after.++ Raises:+ RenderBudgetExceededError: the conversion would overdraw `budget`.+ """+ parsed = charged_parse(markdown_text=markdown_text, budget=budget, operation=_FORMATTING, output_bound=_formatted_size_bound)+ formatter = _MarkdownFormatter()+ formatter.blocks(nodes=markdown_syntax_tree(tokens=parsed.tokens).children, list_depth=0, quote_depth=0)+ budget.charge(units=formatter.units, operation=_FORMATTING)+ return FormattedMarkdown(blocks=formatter.formatted)+++def _formatted_size_bound(*, tokens: Sequence["Token"]) -> int:+ """At most what the result made of `tokens` takes: a part for every token that opens or stands alone, with its text.++ Every token inside a link counts the link's address too, since each span printed from it carries the address: one+ shared string in memory, but written once per span by whatever writes the structure out, its JSON or its repr,+ so a long address around ten thousand spans counts ten thousand times. A token holds a reference link's+ destination as it prints it, so a reference used ten thousand times counts ten thousand times as well.+ """+ total = 0+ pending: list[Sequence[Token]] = [tokens]+ while pending:+ # The length of the address of the link the tokens sit inside, 0 outside any.+ link_length = 0+ for token in pending.pop():+ if token.nesting < 0:+ if token.type == "link_close":+ link_length = 0+ continue+ total += _PART_UNITS + link_length+ if token.type != "inline":+ total += len(token.content)+ if token.type == "link_open":+ href = token.attrGet("href")+ link_length = len(href) if isinstance(href, str) else 0+ if token.children:+ pending.append(token.children)+ return total+++class _SpanStyle(BaseModel):+ model_config = ConfigDict(frozen=True)++ bold: bool = False+ italic: bool = False+ strikethrough: bool = False+ code: bool = False+ link: str | None = None+++class _SpanBuilder:+ """Spans as an inline walk emits them, adjacent texts of the same style joined into one span."""++ def __init__(self) -> None:+ self.spans: list[FormattedSpan] = []+ self._pending_texts: list[str] = []+ self._pending_style: _SpanStyle | None = None++ def text(self, *, text: str, style: _SpanStyle) -> None:+ if not text:+ return+ if style != self._pending_style:+ self._flush()+ self._pending_style = style+ self._pending_texts.append(text)++ def line_break(self) -> None:+ self._flush()+ self.spans.append(LineBreakSpan())++ def finish(self) -> list[FormattedSpan]:+ self._flush()+ return self.spans++ def _flush(self) -> None:+ style = self._pending_style+ if style is not None and self._pending_texts:+ self.spans.append(+ TextSpan(+ text="".join(self._pending_texts),+ bold=style.bold,+ italic=style.italic,+ strikethrough=style.strikethrough,+ code=style.code,+ link=style.link,+ )+ )+ self._pending_texts = []+ self._pending_style = None+++class _MarkdownFormatter:+ """The walk of markdown-it's syntax tree into blocks, tallying the units of what it builds."""++ def __init__(self) -> None:+ self.formatted: list[FormattedBlock] = []+ self.units = 0++ def blocks(self, *, nodes: list[SyntaxTreeNode], list_depth: int, quote_depth: int) -> None:+ for node in nodes:+ self._block(node=node, list_depth=list_depth, quote_depth=quote_depth)++ def _emit(self, *, block: FormattedBlock, text_length: int) -> None:+ self.formatted.append(block)+ self.units += _PART_UNITS + text_length++ def _block(self, *, node: SyntaxTreeNode, list_depth: int, quote_depth: int) -> None:+ match node.type:+ case "paragraph":+ self._paragraph(spans=self._spans(nodes=node.children), list_depth=list_depth, quote_depth=quote_depth)+ case "heading":+ self._heading(node=node, list_depth=list_depth, quote_depth=quote_depth)+ case "bullet_list" | "ordered_list":+ self._list(node=node, list_depth=list_depth, quote_depth=quote_depth)+ case "table":+ self._table(node=node, list_depth=list_depth, quote_depth=quote_depth)+ case "fence" | "code_block":+ text = code_text(content=node.content)+ if text.strip():+ code_block = FormattedCodeBlock(list_depth=list_depth, quote_depth=quote_depth, lines=text.split("\n"))+ self._emit(block=code_block, text_length=len(text))+ case "blockquote":+ self.blocks(nodes=node.children, list_depth=list_depth, quote_depth=quote_depth + 1)+ case "hr":+ self._emit(block=FormattedRule(list_depth=list_depth, quote_depth=quote_depth), text_length=0)+ case _:+ text = plain_text(nodes=[node]).strip()+ if text:+ self.units += _PART_UNITS+ self._paragraph(spans=[TextSpan(text=text)], list_depth=list_depth, quote_depth=quote_depth)++ def _paragraph(self, *, spans: list[FormattedSpan], list_depth: int, quote_depth: int) -> None:+ paragraph = FormattedParagraph(list_depth=list_depth, quote_depth=quote_depth, spans=spans)+ self._emit(block=paragraph, text_length=_spans_length(spans=spans))++ def _heading(self, *, node: SyntaxTreeNode, list_depth: int, quote_depth: int) -> None:+ spans = self._spans(nodes=node.children)+ heading = FormattedHeading(list_depth=list_depth, quote_depth=quote_depth, level=heading_level(tag=node.tag), spans=spans)+ self._emit(block=heading, text_length=_spans_length(spans=spans))++ def _list(self, *, node: SyntaxTreeNode, list_depth: int, quote_depth: int) -> None:+ items = [child for child in node.children if child.type == "list_item"]+ is_ordered = node.type == "ordered_list"+ start = list_start(start=node.attrs.get("start")) if is_ordered else 1+ depth = list_depth + 1+ for index, item in enumerate(items):+ marker = f"{start + index}." if is_ordered else list_bullet(depth=depth)+ children = item.children+ # The marker prints on the line of the item's first paragraph or heading, or alone when the item opens with+ # anything else, such as a nested list, a code block or nothing.+ opening_spans: list[FormattedSpan] = []+ opening_level: int | None = None+ rest = children+ if children:+ first = children[0]+ match first.type:+ case "paragraph":+ opening_spans = self._spans(nodes=first.children)+ rest = children[1:]+ case "heading":+ opening_spans = self._spans(nodes=first.children)+ opening_level = heading_level(tag=first.tag)+ rest = children[1:]+ case _:+ pass+ list_item = FormattedListItem(list_depth=depth, quote_depth=quote_depth, marker=marker, heading_level=opening_level, spans=opening_spans)+ self._emit(block=list_item, text_length=len(marker) + _spans_length(spans=opening_spans))+ self.blocks(nodes=rest, list_depth=depth, quote_depth=quote_depth)++ def _table(self, *, node: SyntaxTreeNode, list_depth: int, quote_depth: int) -> None:+ rows: list[FormattedTableRow] = []+ text_length = 0+ for section in node.children:+ is_header = section.type == "thead"+ for row in section.children:+ if row.type != "tr":+ continue+ cells: list[FormattedTableCell] = []+ for cell in row.children:+ if cell.type not in {"th", "td"}:+ continue+ spans = self._spans(nodes=cell.children)+ text_length += _spans_length(spans=spans)+ cells.append(FormattedTableCell(spans=spans, alignment=cell_alignment(node=cell)))+ self.units += _PART_UNITS+ if cells:+ rows.append(FormattedTableRow(is_header=is_header, cells=cells))+ self.units += _PART_UNITS+ if rows:+ self._emit(block=FormattedTable(list_depth=list_depth, quote_depth=quote_depth, rows=rows), text_length=text_length)++ def _spans(self, *, nodes: list[SyntaxTreeNode]) -> list[FormattedSpan]:+ builder = _SpanBuilder()+ self._inline(nodes=nodes, style=_SpanStyle(), builder=builder)+ spans = builder.finish()+ self.units += _PART_UNITS * len(spans)+ return spans++ def _inline(self, *, nodes: list[SyntaxTreeNode], style: _SpanStyle, builder: _SpanBuilder) -> None:+ for node in nodes:+ match node.type:+ case "text":+ builder.text(text=node.content, style=style)+ case "softbreak":+ builder.text(text=" ", style=style)+ case "hardbreak":+ builder.line_break()+ case "strong":+ self._inline(nodes=node.children, style=style.model_copy(update={"bold": True}), builder=builder)+ case "em":+ self._inline(nodes=node.children, style=style.model_copy(update={"italic": True}), builder=builder)+ case "s":+ self._inline(nodes=node.children, style=style.model_copy(update={"strikethrough": True}), builder=builder)+ case "code_inline":+ builder.text(text=node.content, style=style.model_copy(update={"code": True}))+ case "link":+ href = node.attrs.get("href")+ link = href if isinstance(href, str) and is_linked_href(href=href) else None+ self._inline(nodes=node.children, style=style.model_copy(update={"link": link}), builder=builder)+ case "image":+ builder.text(text=plain_text(nodes=[node]), style=style.model_copy(update={"italic": True}))+ case _:+ if node.children:+ self._inline(nodes=node.children, style=style, builder=builder)+ else:+ builder.text(text=node.content, style=style)+++def _spans_length(*, spans: Sequence[FormattedSpan]) -> int:+ """The characters spans hold: their texts, and the address of a link once for every span that carries it, which+ whatever writes the structure out writes once per span.+ """+ total = 0+ for span in spans:+ match span:+ case TextSpan():+ total += len(span.text) + len(span.link or "")+ case LineBreakSpan():+ total += 1+ return totaldiff --git a/pipelex/tools/markdown/markdown_parser.py b/pipelex/tools/markdown/markdown_parser.pyindex ed8adfddb..a3d5fb907 100644--- a/pipelex/tools/markdown/markdown_parser.py+++ b/pipelex/tools/markdown/markdown_parser.py@@ -20,17 +20,24 @@ bytes it produces, and can produce far more than it reads, so it is charged befocharacter of its source and every cell of its tables (`table_cells_bound`), since the table rule pads a shortrow with empty cells; and its output, bounded from the parsed tokens (`html_length_bound`), must fit what isleft before it renders, since a reference link's destination and title are written once and printed at-every link that uses the reference.+every link that uses the reference. The formatting a document engine prints from (`markdown_formatting.py`) is+charged the same way, so both parse through `charged_parse`, which holds that order.++A converter that walks the syntax tree rather than the flat tokens, the built-in PDF engine and that formatting,+builds it with `markdown_syntax_tree`, which caps how deep inline markup nests first (`cap_inline_nesting`):+markdown-it caps the nesting of blocks, but not that of emphasis, and the tree and every walk of it recurse once+per level, so four hundred nested `*a ` would overflow Python's stack."""import refrom collections.abc import Sequencefrom functools import cache-from typing import TYPE_CHECKING, Any, Final+from typing import TYPE_CHECKING, Any, Final, NamedTuple, Protocolfrom markdown_it import MarkdownIt+from markdown_it.tree import SyntaxTreeNode-from pipelex.tools.jinja2.jinja2_render_budget import MARKDOWN_UNITS_PER_CHARACTER, active_render_budget+from pipelex.tools.jinja2.jinja2_render_budget import MARKDOWN_UNITS_PER_CHARACTER, RenderBudget, active_render_budgetif TYPE_CHECKING:from markdown_it.rules_inline import StateInline@@ -52,6 +59,10 @@ _LINE_ENDING: Final = re.compile(r"\r\n?|\n")# What a table's delimiter row is made of, once its container's indentation and `>` markers are left out._DELIMITER_ROW_CHARACTERS: Final = frozenset("|-: \t")+# How deep inline markup may nest in a syntax tree, emphasis inside emphasis inside a link, an image's alt text one+# level below the image: far beyond what a written text nests, and far within Python's stack for a walk of the tree.+MAX_INLINE_NESTING: Final = 50+@cachedef get_markdown_parser() -> MarkdownIt:@@ -107,17 +118,85 @@ def render_markdown_as_html(markdown_text: str) -> str:if budget is None:html: str = parser.render(markdown_text)return html- # The length first: the table scan reads every line, so it runs only once the source is affordable.- budget.charge(units=MARKDOWN_UNITS_PER_CHARACTER * len(markdown_text), operation=_CONVERTING)- budget.charge(units=MARKDOWN_UNITS_PER_CHARACTER * table_cells_bound(markdown_text=markdown_text), operation=_CONVERTING)- env: dict[str, Any] = {}- tokens = parser.parse(markdown_text, env)- budget.afford(units=html_length_bound(tokens=tokens), operation=_CONVERTING)- rendered: str = parser.renderer.render(tokens, parser.options, env)+ parsed = charged_parse(markdown_text=markdown_text, budget=budget, operation=_CONVERTING, output_bound=html_length_bound)+ rendered: str = parser.renderer.render(parsed.tokens, parser.options, parsed.env)budget.charge(units=len(rendered), operation=_CONVERTING)return rendered+class TokensBound(Protocol):+ """At most the work units of what a conversion builds out of parsed tokens, taken without building it."""++ def __call__(self, *, tokens: Sequence["Token"]) -> int: ...+++class ParsedMarkdown(NamedTuple):+ """A Markdown text's tokens, and the environment the parser filled, holding its reference definitions."""++ tokens: list["Token"]+ env: dict[str, Any]+++def charged_parse(*, markdown_text: str, budget: RenderBudget, operation: str, output_bound: TokensBound) -> ParsedMarkdown:+ """Parse a Markdown text for a conversion charged to `budget`, refusing it at the first step the budget cannot afford.++ In this order: its length, before anything reads it, since the table scan reads every line; then the cells of its+ tables, before the parser pads them (`table_cells_bound`); then the parse; and then what the conversion builds out+ of the tokens, bounded from them by `output_bound`, must fit what is left. The caller builds its output and+ charges what it built.++ Raises:+ RenderBudgetExceededError: a step would overdraw the budget, `operation` naming the conversion.+ """+ budget.charge(units=MARKDOWN_UNITS_PER_CHARACTER * len(markdown_text), operation=operation)+ budget.charge(units=MARKDOWN_UNITS_PER_CHARACTER * table_cells_bound(markdown_text=markdown_text), operation=operation)+ env: dict[str, Any] = {}+ tokens = get_markdown_parser().parse(markdown_text, env)+ budget.afford(units=output_bound(tokens=tokens), operation=operation)+ return ParsedMarkdown(tokens=tokens, env=env)+++def markdown_syntax_tree(*, tokens: Sequence["Token"]) -> SyntaxTreeNode:+ """The syntax tree of parsed tokens, their inline nesting capped first (`cap_inline_nesting`)."""+ cap_inline_nesting(tokens=tokens)+ return SyntaxTreeNode(tokens)+++def cap_inline_nesting(*, tokens: Sequence["Token"]) -> None:+ """Drop the inline markup of `tokens` that nests deeper than `MAX_INLINE_NESTING`, keeping its text.++ markdown-it caps the nesting of blocks at twenty, but emphasis is paired up after its delimiters are read, so it+ nests as deep as the text asks: four hundred `*a ` before a word and four hundred ` c*` after it nest four hundred+ deep. The cap walks the flat tokens, each inline token's children and each image's, the alt text, one level below+ the image; past it, an opening token is dropped with its closing one, and an image keeps no children, printing its+ alt text as written. The tokens are changed in place.+ """+ pending: list[tuple[Token, int]] = [(token, 0) for token in tokens if token.children]+ while pending:+ parent, depth = pending.pop()+ kept: list[Token] = []+ # For each opening token not yet closed, whether it was dropped, so its closing token is dropped with it.+ open_dropped: list[bool] = []+ for child in parent.children or []:+ if child.nesting > 0:+ is_dropped = depth >= MAX_INLINE_NESTING+ open_dropped.append(is_dropped)+ if is_dropped:+ continue+ depth += 1+ elif child.nesting < 0 and open_dropped:+ if open_dropped.pop():+ continue+ depth -= 1+ if child.children:+ if depth + 1 > MAX_INLINE_NESTING:+ child.children = None+ else:+ pending.append((child, depth + 1))+ kept.append(child)+ parent.children = kept++def table_cells_bound(*, markdown_text: str) -> int:"""At most the number of cells the tables of `markdown_text` parse into, taken from its lines.diff --git a/pipelex/tools/markdown/markdown_rules.py b/pipelex/tools/markdown/markdown_rules.pynew file mode 100644index 000000000..95fe3072b--- /dev/null+++ b/pipelex/tools/markdown/markdown_rules.py@@ -0,0 +1,105 @@+"""The rules by which the built-in PDF engine, and the formatting other document engines print from, read Markdown.++The built-in PDF engine (`pipelex/providers/reportlab/markdown_flowables.py`) and the formatting a document engine+prints from when it formats Markdown in its own format (`markdown_formatting.py`) both walk markdown-it's syntax tree,+and both read it by these rules:++- only an `http`, `https` or `mailto` target becomes a link, any other printing its text alone;+- the bullets of nested bullet lists alternate a bullet and an en dash by depth, and an ordered list is numbered+ from its own `start`;+- a heading has its level from its tag, a code block prints as written with its tabs expanded, and a table cell has+ the alignment its column's delimiter sets;+- an image is never fetched, and prints as `[image: alt]`.++The HTML conversion (`render_markdown_as_html`) is not one of them: it follows markdown-it's own rules, which keep a+link to any target but a `javascript:`, `vbscript:`, `file:` or non-image `data:` one, a relative one included, and+show an image as an `<img>`.++They live here, under `tools/`, rather than in the document engine contract or the PDF engine, because both read+them and the contract (`pipelex/cogt/doc_gen/formatted_markdown.py`) builds on `tools/` rather than the other way+round.+"""++from typing import Final, Literal, TypeAlias+from urllib.parse import urlsplit++from markdown_it.tree import SyntaxTreeNode++LINKED_SCHEMES: Final = frozenset({"http", "https", "mailto"})++# The bullets of nested bullet lists, by depth, alternating: a bullet and an en dash.+_BULLETS: Final = ("•", "–")++CellAlignment: TypeAlias = Literal["left", "center", "right"]+++def is_linked_href(*, href: str) -> bool:+ """Whether a link target becomes a link in a document: only `http`, `https` and `mailto` do."""+ try:+ scheme = urlsplit(href.strip()).scheme+ except ValueError:+ return False+ return scheme.lower() in LINKED_SCHEMES+++def list_bullet(*, depth: int) -> str:+ """The bullet of a bullet list `depth` lists deep, 1 for a top-level list: a bullet and an en dash, alternating."""+ return _BULLETS[(depth - 1) % len(_BULLETS)]+++def list_start(*, start: str | float | None) -> int:+ """The number an ordered list starts at, from the `start` markdown-it reads off its first item: 1 when it has none."""+ if start is None:+ return 1+ try:+ return int(start)+ except (ValueError, OverflowError):+ return 1+++def heading_level(*, tag: str) -> int:+ """A heading's level from its tag, `h1` to `h6`: 6 for any other tag."""+ if len(tag) == 2 and tag[0] == "h" and tag[1].isdigit():+ return int(tag[1])+ return 6+++def code_text(*, content: str) -> str:+ """A code block's text as it prints: as written, without its last line feed, tabs expanded to four spaces."""+ return content.rstrip("\n").expandtabs(4)+++def cell_alignment(*, node: SyntaxTreeNode) -> CellAlignment | None:+ """The alignment a table cell's column sets with the colons of its delimiter row, or None when it sets none.++ markdown-it writes it as the cell's `style`: `| :-- |` is left, `| :-: |` is center and `| --: |` is right.+ """+ style = node.attrs.get("style")+ if not isinstance(style, str):+ return None+ if "text-align:right" in style:+ return "right"+ if "text-align:center" in style:+ return "center"+ if "text-align:left" in style:+ return "left"+ return None+++def plain_text(*, nodes: list[SyntaxTreeNode]) -> str:+ """The text of Markdown nodes without their markup, as it reads once printed, an image as `[image: alt]`."""+ parts: list[str] = []+ for node in nodes:+ match node.type:+ case "softbreak":+ parts.append(" ")+ case "hardbreak":+ parts.append("\n")+ case "image":+ parts.append(f"[image: {plain_text(nodes=node.children).strip() or node.content}]")+ case _:+ if node.children:+ parts.append(plain_text(nodes=node.children))+ else:+ parts.append(node.content)+ return "".join(parts)diff --git a/subject_grants.toml b/subject_grants.tomlindex 7bc38acf7..a5242b243 100644--- a/subject_grants.toml+++ b/subject_grants.toml@@ -5656,6 +5656,10 @@ rationale = "Factory from source: builds the result from the template_category;param = "value"rationale = "Transforms the value; single operand."+["pipelex/tools/jinja2/jinja2_filters.py::markdown_to_formatted"]+param = "value"+rationale = "Jinja filter: Jinja passes the filtered value positionally."+["pipelex/tools/jinja2/jinja2_filters.py::markdown_to_html"]param = "value"rationale = "Jinja filter: Jinja passes the filtered value positionally."diff --git a/tests/data/errors/error_identity.txt b/tests/data/errors/error_identity.txtindex f2c79d69e..e05bbfe28 100644--- a/tests/data/errors/error_identity.txt+++ b/tests/data/errors/error_identity.txt@@ -174,6 +174,7 @@ LinkupSearchResponseError | Linkup search response | https://docs.pipelex.com/laListWhereSingularError | List where singular | https://docs.pipelex.com/latest/errors/list-where-singular-error/LogSinkHeaderValueError | Log sink header value holds a line break | https://docs.pipelex.com/latest/errors/log-sink-header-value-error/LogSinkVariableError | Log sink variable did not resolve | https://docs.pipelex.com/latest/errors/log-sink-variable-error/+MarkdownFormattingBudgetError | Markdown formatting budget exceeded | https://docs.pipelex.com/latest/errors/markdown-formatting-budget-error/MethodDependencyFetchError | Method dependency fetch | https://docs.pipelex.com/latest/errors/method-dependency-fetch-error/MethodFetchDisabledError | Method fetch disabled | https://docs.pipelex.com/latest/errors/method-fetch-disabled-error/MethodFetchError | Method fetch | https://docs.pipelex.com/latest/errors/method-fetch-error/diff --git a/tests/unit/pipelex/cogt/doc_gen/test_contract_import_light.py b/tests/unit/pipelex/cogt/doc_gen/test_contract_import_light.pyindex 8501f827f..122411e4a 100644--- a/tests/unit/pipelex/cogt/doc_gen/test_contract_import_light.py+++ b/tests/unit/pipelex/cogt/doc_gen/test_contract_import_light.py@@ -24,6 +24,7 @@ _CONTRACT_IMPORT_SCRIPT = textwrap.dedent(import pipelex.cogt.doc_gennames = sorted(module.name for module in pkgutil.iter_modules(pipelex.cogt.doc_gen.__path__, "pipelex.cogt.doc_gen."))+ assert "pipelex.cogt.doc_gen.formatted_markdown" in namesassert "pipelex.cogt.doc_gen.layout_display" in namesassert "pipelex.cogt.doc_gen.template_environment" in namesfor name in names:diff --git a/tests/unit/pipelex/cogt/doc_gen/test_formatted_markdown.py b/tests/unit/pipelex/cogt/doc_gen/test_formatted_markdown.pynew file mode 100644index 000000000..1f7a05996--- /dev/null+++ b/tests/unit/pipelex/cogt/doc_gen/test_formatted_markdown.py@@ -0,0 +1,131 @@+import tracemalloc++import pytest+from jinja2.exceptions import TemplateError+from pydantic import BaseModel+from pytest_mock import MockerFixture++from pipelex.cogt.doc_gen import formatted_markdown+from pipelex.cogt.doc_gen.exceptions import MarkdownFormattingBudgetError+from pipelex.cogt.doc_gen.formatted_markdown import FormattedHeading, FormattedParagraph, TextSpan, format_markdown, spans_text+from pipelex.tools.jinja2.jinja2_render_budget import (+ DEFAULT_RENDER_BUDGET_UNITS,+ MARKDOWN_UNITS_PER_CHARACTER,+ RenderBudget,+ RenderBudgetExceededError,+ active_render_budget,+ spending_from,+)+from pipelex.tools.markdown import markdown_formatting, markdown_rules+from pipelex.tools.markdown.markdown_parser import markdown_syntax_tree, table_cells_bound+from tests.unit.pipelex.tools.markdown.test_data import MarkdownFormattingTestData++# What a conversion refused for its tables may hold before the refusal: its source and the scan of its lines, never the+# tokens of its cells, which take tens of megabytes here.+_REFUSED_BEFORE_PARSING_PEAK_BYTES = 4 * 1024 * 1024+++class TestFormattedMarkdown:+ def test_the_contract_names_the_structure_it_formats_into(self) -> None:+ """An engine imports the structure from the contract, which names the very classes the conversion builds, all of them."""+ models = [value for value in vars(markdown_formatting).values() if isinstance(value, type) and issubclass(value, BaseModel)]+ public_models = {+ model.__name__ for model in models if model.__module__ == markdown_formatting.__name__ and not model.__name__.startswith("_")+ }+ assert public_models+ assert public_models <= set(formatted_markdown.__all__)+ for name in formatted_markdown.__all__:+ if name == "format_markdown":+ continue+ source = markdown_formatting if hasattr(markdown_formatting, name) else markdown_rules+ assert getattr(formatted_markdown, name) is getattr(source, name), name+ assert isinstance(format_markdown(markdown="Hello"), markdown_formatting.FormattedMarkdown)++ def test_it_formats_by_the_shared_rules(self) -> None:+ assert format_markdown(markdown="# Report\n\nSee [the site](https://pipelex.com).").blocks == [+ FormattedHeading(level=1, spans=[TextSpan(text="Report")]),+ FormattedParagraph(spans=[TextSpan(text="See "), TextSpan(text="the site", link="https://pipelex.com"), TextSpan(text=".")]),+ ]++ @pytest.mark.parametrize(+ ("markdown", "expected_text"),+ [+ (MarkdownFormattingTestData.DEEP_EMPHASIS, MarkdownFormattingTestData.DEEP_EMPHASIS_TEXT),+ (MarkdownFormattingTestData.DEEP_MIXED_EMPHASIS, MarkdownFormattingTestData.DEEP_MIXED_EMPHASIS_TEXT),+ ],+ )+ def test_emphasis_nested_hundreds_deep_formats_rather_than_overflowing_the_stack(self, markdown: str, expected_text: str) -> None:+ blocks = format_markdown(markdown=markdown).blocks+ assert len(blocks) == 1+ paragraph = blocks[0]+ assert isinstance(paragraph, FormattedParagraph)+ assert spans_text(spans=paragraph.spans) == expected_text++ def test_outside_a_render_its_own_budget_refuses_a_padded_table_before_parsing_it(self) -> None:+ tracemalloc.start()+ try:+ with pytest.raises(MarkdownFormattingBudgetError, match="outside a template render"):+ format_markdown(markdown=MarkdownFormattingTestData.PADDED_TABLE)+ _, peak_bytes = tracemalloc.get_traced_memory()+ finally:+ tracemalloc.stop()+ assert peak_bytes < _REFUSED_BEFORE_PARSING_PEAK_BYTES+ assert active_render_budget() is None++ def test_a_reference_used_thousands_of_times_is_refused_before_it_is_built(self) -> None:+ with pytest.raises(MarkdownFormattingBudgetError):+ format_markdown(markdown=MarkdownFormattingTestData.REUSED_REFERENCE)++ def test_a_long_address_around_many_spans_is_refused_before_the_tree_is_built(self, mocker: MockerFixture) -> None:+ """Outside a render and inside one: each span carries the address, so the bound counts it once per span."""+ tree_builder = mocker.patch("pipelex.tools.markdown.markdown_formatting.markdown_syntax_tree", wraps=markdown_syntax_tree)+ with pytest.raises(MarkdownFormattingBudgetError):+ format_markdown(markdown=MarkdownFormattingTestData.LINK_AROUND_MANY_SPANS)+ budget = RenderBudget(total=DEFAULT_RENDER_BUDGET_UNITS)+ with spending_from(budget=budget), pytest.raises(RenderBudgetExceededError, match="formatting Markdown"):+ format_markdown(markdown=MarkdownFormattingTestData.LINK_AROUND_MANY_SPANS)+ tree_builder.assert_not_called()++ def test_inside_a_render_a_link_s_address_is_charged_with_every_span_that_carries_it(self) -> None:+ markdown = MarkdownFormattingTestData.LINK_AROUND_FEW_SPANS+ budget = RenderBudget(total=DEFAULT_RENDER_BUDGET_UNITS)+ with spending_from(budget=budget):+ blocks = format_markdown(markdown=markdown).blocks+ assert len(blocks) == 1+ paragraph = blocks[0]+ assert isinstance(paragraph, FormattedParagraph)+ linked_spans = [span for span in paragraph.spans if isinstance(span, TextSpan) and span.link]+ assert len(linked_spans) == 3+ spent = DEFAULT_RENDER_BUDGET_UNITS - budget.remaining+ assert spent >= MARKDOWN_UNITS_PER_CHARACTER * len(markdown) + sum(len(span.link or "") for span in linked_spans)++ def test_outside_a_render_an_ordinary_long_text_formats(self) -> None:+ formatted = format_markdown(markdown=MarkdownFormattingTestData.LONG_REPORT)+ assert sum(isinstance(block, FormattedHeading) for block in formatted.blocks) == 300++ def test_the_budget_error_is_the_caller_s(self) -> None:+ with pytest.raises(MarkdownFormattingBudgetError) as caught:+ format_markdown(markdown=MarkdownFormattingTestData.PADDED_TABLE)+ report = caught.value.to_error_report()+ assert report.error_domain == "input"+ assert report.caller_facing_message+ assert f"{DEFAULT_RENDER_BUDGET_UNITS:,} work units" in caught.value.message++ def test_inside_a_render_the_render_s_budget_is_charged_for_the_source_the_tables_and_the_result(self) -> None:+ """A table of padded cells costs more to parse than its result takes: dropping either charge spends less."""+ markdown = MarkdownFormattingTestData.WIDE_SHORT_TABLE+ source_and_tables = MARKDOWN_UNITS_PER_CHARACTER * (len(markdown) + table_cells_bound(markdown_text=markdown))+ budget = RenderBudget(total=DEFAULT_RENDER_BUDGET_UNITS)+ with spending_from(budget=budget):+ format_markdown(markdown=markdown)+ spent = DEFAULT_RENDER_BUDGET_UNITS - budget.remaining+ assert spent > source_and_tables++ def test_inside_a_render_an_overdraft_is_the_render_s_own_template_error(self) -> None:+ # A budget that affords the text alone, not the conversion: the refusal is the render's, which a template fill+ # catches with every other template error.+ budget = RenderBudget(total=len(MarkdownFormattingTestData.NOTES) * 4)+ with spending_from(budget=budget), pytest.raises(RenderBudgetExceededError, match="formatting Markdown") as caught:+ format_markdown(markdown=MarkdownFormattingTestData.NOTES)+ assert isinstance(caught.value, TemplateError)+ assert not isinstance(caught.value, MarkdownFormattingBudgetError)diff --git a/tests/unit/pipelex/cogt/doc_gen/test_template_environment.py b/tests/unit/pipelex/cogt/doc_gen/test_template_environment.pyindex 0ec4c9d75..0305f6b70 100644--- a/tests/unit/pipelex/cogt/doc_gen/test_template_environment.py+++ b/tests/unit/pipelex/cogt/doc_gen/test_template_environment.py@@ -1,9 +1,28 @@import pytest-from jinja2.exceptions import SecurityError, UndefinedError+from jinja2.exceptions import SecurityError, TemplateError, UndefinedErrorfrom jinja2.filters import FILTERS as JINJA_FILTERS+from markupsafe import Markup+from pipelex.cogt.doc_gen.exceptions import MarkdownFormattingBudgetError+from pipelex.cogt.doc_gen.formatted_markdown import FormattedMarkdownfrom pipelex.cogt.doc_gen.template_environment import make_plain_data_template_environment-from pipelex.tools.jinja2.jinja2_render_budget import DEFAULT_RENDER_BUDGET_UNITS, RenderBudgetExceededError+from pipelex.tools.jinja2.jinja2_filters import markdown_to_formatted+from pipelex.tools.jinja2.jinja2_render_budget import DEFAULT_RENDER_BUDGET_UNITS, MARKDOWN_UNITS_PER_CHARACTER, RenderBudgetExceededError+from tests.unit.pipelex.tools.markdown.test_data import MarkdownFormattingTestData+++def _kinds_of_formatted(value: object) -> object:+ """An engine's finalize: a printed `FormattedMarkdown` as the kinds of its blocks, every other value as it is."""+ if isinstance(value, FormattedMarkdown):+ return "<" + ",".join(block.kind for block in value.blocks) + ">"+ return value+++def _marker_for_formatted(value: object) -> object:+ """An engine's finalize printing markup of its own for a `FormattedMarkdown`, every other value as it is."""+ if isinstance(value, FormattedMarkdown):+ return Markup("<?marker?>")+ return valueclass TestPlainDataTemplateEnvironment:@@ -37,8 +56,89 @@ class TestPlainDataTemplateEnvironment:with pytest.raises(RenderBudgetExceededError):template.render(inner=DEFAULT_RENDER_BUDGET_UNITS // 64 // 32)- def test_the_filters_are_jinjas_own(self) -> None:+ def test_the_filters_are_jinjas_own_and_markdown(self) -> None:# The names with a leading underscore are the render budget's own, which its rewrite of a template calls.filters = make_plain_data_template_environment().filtersnamed_filters = {name: function for name, function in filters.items() if not name.startswith("_")}- assert named_filters == JINJA_FILTERS+ assert named_filters == {**JINJA_FILTERS, "markdown": markdown_to_formatted}++ def test_markdown_prints_through_the_engine_s_finalize(self) -> None:+ environment = make_plain_data_template_environment(finalize=_kinds_of_formatted)+ rendered = environment.from_string("{{ notes | markdown }}|{{ count }}|{{ title }}").render(+ notes=MarkdownFormattingTestData.NOTES, count=3, title="Invoice"+ )+ assert rendered == "<paragraph,list_item,list_item>|3|Invoice"++ def test_markdown_prints_its_plain_text_without_a_finalize(self) -> None:+ rendered = (+ make_plain_data_template_environment()+ .from_string("{{ invoice.notes | markdown }}")+ .render(invoice={"notes": MarkdownFormattingTestData.NOTES})+ )+ assert rendered == MarkdownFormattingTestData.NOTES_PLAIN_TEXT++ def test_markdown_reads_any_other_value_as_its_text(self) -> None:+ assert make_plain_data_template_environment().from_string("{{ total | markdown }}").render(total=12.5) == "12.5"++ def test_markdown_of_none_prints_nothing(self) -> None:+ environment = make_plain_data_template_environment(finalize=_kinds_of_formatted)+ assert environment.from_string("[{{ notes | markdown }}]").render(notes=None) == "[<>]"+ assert make_plain_data_template_environment().from_string("[{{ notes | markdown }}]").render(notes=None) == "[]"++ @pytest.mark.parametrize(("notes", "expected"), [(None, "no"), ("", "no"), (" \n\t", "no"), ("Due in *30* days.", "yes")])+ def test_markdown_is_false_when_it_formats_into_nothing(self, notes: str | None, expected: str) -> None:+ template = make_plain_data_template_environment().from_string("{% if notes | markdown %}yes{% else %}no{% endif %}")+ assert template.render(notes=notes) == expected++ @pytest.mark.parametrize(+ ("notes", "expected_text"),+ [+ (MarkdownFormattingTestData.DEEP_EMPHASIS, MarkdownFormattingTestData.DEEP_EMPHASIS_TEXT),+ (MarkdownFormattingTestData.DEEP_MIXED_EMPHASIS, MarkdownFormattingTestData.DEEP_MIXED_EMPHASIS_TEXT),+ ],+ )+ def test_markdown_of_emphasis_nested_hundreds_deep_prints_its_text(self, notes: str, expected_text: str) -> None:+ assert make_plain_data_template_environment().from_string("{{ notes | markdown }}").render(notes=notes) == expected_text++ def test_converting_once_to_test_and_print_affords_what_converting_twice_overdraws(self) -> None:+ """Every conversion is charged, so the idiom the docs give converts a text once into a variable, then tests and prints it."""+ # Each conversion costs about three fifths of a render's budget: one fits, two do not.+ notes = "word " * (DEFAULT_RENDER_BUDGET_UNITS * 3 // 5 // MARKDOWN_UNITS_PER_CHARACTER // 5)+ environment = make_plain_data_template_environment()+ converted_once = environment.from_string("{% set formatted = notes | markdown %}{% if formatted %}[{{ formatted }}]{% endif %}")+ assert converted_once.render(notes=notes) == f"[{notes.strip()}]"+ converted_twice = environment.from_string("{% if notes | markdown %}[{{ notes | markdown }}]{% endif %}")+ with pytest.raises(RenderBudgetExceededError, match="formatting Markdown"):+ converted_twice.render(notes=notes)++ def test_markdown_of_a_long_address_around_many_spans_is_refused(self) -> None:+ template = make_plain_data_template_environment().from_string("{% set formatted = notes | markdown %}{{ formatted.blocks }}")+ with pytest.raises(RenderBudgetExceededError, match="formatting Markdown"):+ template.render(notes=MarkdownFormattingTestData.LINK_AROUND_MANY_SPANS)++ def test_markdown_of_a_missing_value_fails_the_render(self) -> None:+ template = make_plain_data_template_environment().from_string("{{ invoice.notes | markdown }}")+ with pytest.raises(UndefinedError):+ template.render(invoice={})++ def test_markdown_under_autoescape_prints_its_plain_text_escaped(self) -> None:+ # docxtpl turns autoescaping on after the environment is built; the engine's markup passes as it is.+ environment = make_plain_data_template_environment(finalize=_marker_for_formatted)+ environment.autoescape = True+ rendered = environment.from_string("{{ notes | markdown }}|{{ raw }}").render(notes="a < b", raw="<?marker?>")+ assert rendered == "<?marker?>|<?marker?>"+ plain_environment = make_plain_data_template_environment()+ plain_environment.autoescape = True+ assert plain_environment.from_string("{{ notes | markdown }}").render(notes="a < **b** & c") == "a < b & c"++ def test_markdown_is_charged_to_the_render_s_budget(self) -> None:+ # Each conversion costs about two fifths of a render's budget: one renders, and a render making three is+ # refused at the third, with the render's own budget error.+ notes = "word " * (DEFAULT_RENDER_BUDGET_UNITS * 2 // 5 // MARKDOWN_UNITS_PER_CHARACTER // 5)+ environment = make_plain_data_template_environment()+ assert environment.from_string("{{ notes | markdown }}").render(notes=notes) == notes.strip()+ template = environment.from_string("{% for i in range(3) %}{{ notes | markdown }}{% endfor %}")+ with pytest.raises(RenderBudgetExceededError, match="formatting Markdown") as caught:+ template.render(notes=notes)+ assert isinstance(caught.value, TemplateError)+ assert not isinstance(caught.value, MarkdownFormattingBudgetError)diff --git a/tests/unit/pipelex/providers/reportlab/test_markdown_flowables.py b/tests/unit/pipelex/providers/reportlab/test_markdown_flowables.pyindex d2f2a6157..f7c87fde8 100644--- a/tests/unit/pipelex/providers/reportlab/test_markdown_flowables.py+++ b/tests/unit/pipelex/providers/reportlab/test_markdown_flowables.py@@ -3,12 +3,15 @@ import reimport pytestfrom markdown_it.token import Tokenfrom markdown_it.tree import SyntaxTreeNode+from pytest_mock import MockerFixture+from reportlab.lib.enums import TA_CENTER, TA_LEFT, TA_RIGHT # type: ignore[import-untyped]from reportlab.platypus import Paragraph # type: ignore[import-untyped]from pipelex.cogt.doc_gen.layout_tree import MarkdownBlock-from pipelex.providers.reportlab.markdown_flowables import inline_markup, is_linked_href, markdown_nodes_to_flowables, plain_text-from pipelex.providers.reportlab.pdf_elements import MONO_FONT, build_pdf_styles+from pipelex.providers.reportlab.markdown_flowables import inline_markup, markdown_nodes_to_flowables+from pipelex.providers.reportlab.pdf_elements import MONO_FONT, build_pdf_styles, data_tablefrom pipelex.tools.markdown.markdown_parser import get_markdown_parser+from pipelex.tools.markdown.markdown_rules import is_linked_href, plain_textfrom tests.unit.pipelex.providers.reportlab.reportlab_test_helpers import (StubRenderResources,document_text,@@ -19,6 +22,7 @@ from tests.unit.pipelex.providers.reportlab.reportlab_test_helpers import (render_markdown,)from tests.unit.pipelex.providers.reportlab.test_data import MarkdownFlowablesTestData+from tests.unit.pipelex.tools.markdown.test_data import MarkdownFormattingTestDatadef _list_lines(*, text: str) -> list[str]:@@ -53,6 +57,13 @@ class TestMarkdownFlowables:markdown_text = "| Quarter | Revenue |\n| --- | --: |\n| Q1 | **$1.2M** |\n| Q2 |\n"assert "Quarter Revenue\nQ1 $1.2M\nQ2" in document_text(pdf_data=render_markdown(markdown_text=markdown_text))+ def test_a_table_cell_is_aligned_as_its_column_is(self, mocker: MockerFixture) -> None:+ built_table = mocker.patch("pipelex.providers.reportlab.markdown_flowables.data_table", wraps=data_table)+ nodes = SyntaxTreeNode(get_markdown_parser().parse("| L | C | R | N |\n| :-- | :-: | --: | --- |\n| a | b | c | d |\n")).children+ markdown_nodes_to_flowables(nodes=nodes, styles=build_pdf_styles(), available_width=400)+ rows = built_table.call_args.kwargs["rows"]+ assert [[cell.style.alignment for cell in row] for row in rows] == [[TA_LEFT, TA_CENTER, TA_RIGHT, TA_LEFT]] * 2+def test_a_long_table_repeats_its_header_on_the_next_page(self) -> None:rows = "".join(f"| Row {index} | {index * 10} |\n" for index in range(1, 91))pages = page_texts(pdf_data=render_markdown(markdown_text=f"| Label | Value |\n| --- | --- |\n{rows}"))@@ -77,6 +88,17 @@ class TestMarkdownFlowables:expected_fonts = {"OpenSans-Regular", "OpenSans-Bold", "OpenSans-Italic", "OpenSans-BoldItalic", "RobotoMono-Regular"}assert expected_fonts <= embedded_font_names(pdf_data=pdf_data)+ @pytest.mark.parametrize(+ ("markdown_text", "expected_text"),+ [+ (MarkdownFormattingTestData.DEEP_EMPHASIS, MarkdownFormattingTestData.DEEP_EMPHASIS_TEXT),+ (MarkdownFormattingTestData.DEEP_MIXED_EMPHASIS, MarkdownFormattingTestData.DEEP_MIXED_EMPHASIS_TEXT),+ ],+ )+ def test_emphasis_nested_hundreds_deep_prints_its_text(self, markdown_text: str, expected_text: str) -> None:+ text = document_text(pdf_data=render_markdown(markdown_text=markdown_text))+ assert expected_text in " ".join(text.split())+def test_markup_in_markdown_text_prints_as_written(self) -> None:text = document_text(pdf_data=render_markdown(markdown_text="Tom & Jerry say <b>hello</b> & <br/> bye"))assert "Tom & Jerry say <b>hello</b> & <br/> bye" in textdiff --git a/tests/unit/pipelex/tools/markdown/test_data.py b/tests/unit/pipelex/tools/markdown/test_data.pynew file mode 100644index 000000000..f8f5bc1f2--- /dev/null+++ b/tests/unit/pipelex/tools/markdown/test_data.py@@ -0,0 +1,116 @@+from typing import ClassVar++from pipelex.tools.markdown.markdown_formatting import (+ FormattedBlock,+ FormattedCodeBlock,+ FormattedHeading,+ FormattedListItem,+ FormattedParagraph,+ FormattedRule,+ FormattedTable,+ FormattedTableCell,+ FormattedTableRow,+ TextSpan,+)++_ITEM = FormattedListItem(list_depth=1, marker="•", spans=[TextSpan(text="item")])+_ONE_CELL_ROWS = [+ FormattedTableRow(is_header=True, cells=[FormattedTableCell(spans=[TextSpan(text="a")])]),+ FormattedTableRow(is_header=False, cells=[FormattedTableCell(spans=[TextSpan(text="1")])]),+]+++class MarkdownFormattingTestData:+ """Markdown texts the formatting tests read, and what each formats into."""++ # A header of three thousand columns over three hundred one-character rows: nine hundred thousand cells, which the+ # table rule pads every row to, out of about ten kilobytes of text.+ PADDED_TABLE: ClassVar[str] = "|" + "a|" * 3000 + "\n|" + "-|" * 3000 + "\n" + "|a\n" * 300++ # A header of twenty columns over ten one-character rows: its padded cells cost the parser more than the result+ # they format into takes, so charging either alone spends less than charging both.+ WIDE_SHORT_TABLE: ClassVar[str] = "Totals:\n\n|" + "h|" * 20 + "\n|" + "-|" * 20 + "\n" + "|a\n" * 10++ # A reference link used four thousand times, whose twenty-thousand-character address an engine printing a link's+ # address beside its text prints at every use: its text alone fits a budget, what it formats into does not.+ REUSED_REFERENCE: ClassVar[str] = "[a][r] " * 4000 + "\n\n[r]: https://example.com/" + "y" * 20000++ # A twenty-thousand-character address around ten thousand spans, alternating italic and plain: each span carries the+ # address, so the structure written out, as JSON or as its repr, writes it ten thousand times, two hundred megabytes+ # out of forty kilobytes of text.+ LINK_AROUND_MANY_SPANS: ClassVar[str] = "[" + "*a*b" * 5000 + "](https://example.com/" + "y" * 20000 + ")"++ # The same address around three spans, an ordinary text whose result carries the address three times.+ LINK_AROUND_FEW_SPANS: ClassVar[str] = "[*a* b *c*](https://example.com/" + "y" * 20000 + ")"++ # A report section repeated into about thirty thousand characters: an ordinary long text, well within one budget.+ LONG_REPORT: ClassVar[str] = "## Findings\n\nSome *text* with a [link](https://a.co), `code` and more.\n\n- one\n- two\n\n" * 300++ # Emphasis nested four hundred deep, and emphasis and strong emphasis alternating four hundred deep: markdown-it+ # pairs emphasis up without capping its depth, and a syntax tree that deep overflows Python's stack.+ DEEP_EMPHASIS: ClassVar[str] = "*a " * 400 + "b" + " c*" * 400+ DEEP_EMPHASIS_TEXT: ClassVar[str] = "a " * 400 + "b" + " c" * 400+ DEEP_MIXED_EMPHASIS: ClassVar[str] = "*a **b " * 200 + "middle" + " b** a*" * 200+ DEEP_MIXED_EMPHASIS_TEXT: ClassVar[str] = "a b " * 200 + "middle" + " b a" * 200++ # The text an invoice's notes hold, and its plain text.+ NOTES: ClassVar[str] = "Payment by **bank transfer** within 30 days.\n\n- *Late* fees apply\n- IBAN on request"+ NOTES_PLAIN_TEXT: ClassVar[str] = "Payment by bank transfer within 30 days.\n• Late fees apply\n• IBAN on request"++ # Every kind of block, and its plain text: list items indented by depth, a later paragraph of an item indented+ # under its marker, table cells separated by tabs, a code block as written, and the rule left out.+ EVERY_BLOCK: ClassVar[str] = (+ "# Report\n\n"+ "Some **bold** text, a [link](https://pipelex.com) and `code`. \nOn its own line.\n\n"+ "- one\n - two\n\n more about one\n\n"+ "3. three\n4. four\n\n"+ "> Quoted *words*\n\n"+ "---\n\n"+ "| Item | Qty |\n| --- | --: |\n| Tea | 2 |\n\n"+ "```\nprint('x')\n```\n"+ )+ EVERY_BLOCK_PLAIN_TEXT: ClassVar[str] = (+ "Report\n"+ "Some bold text, a link and code.\nOn its own line.\n"+ "• one\n – two\n more about one\n"+ "3. three\n4. four\n"+ "Quoted words\n"+ "Item\tQty\nTea\t2\n"+ "print('x')"+ )++ # Blocks inside a list item, and its plain text: every line of what the item holds indented under its marker, a+ # code block's lines and a table's rows included, and a hard break's next line too.+ INSIDE_A_LIST_ITEM: ClassVar[str] = (+ "- item\n\n ```\n first\n\n third\n ```\n\n | a | b |\n | - | - |\n | 1 | 2 |\n\n > quoted \n > on\n\n ---\n- \n- next\n"+ )+ INSIDE_A_LIST_ITEM_PLAIN_TEXT: ClassVar[str] = "• item\n first\n\n third\n a\tb\n 1\t2\n quoted\n on\n•\n• next"++ # Each block nested in a list item or a quotation, by topic: (topic, the Markdown, the blocks it formats into).+ NESTING_CASES: ClassVar[list[tuple[str, str, list[FormattedBlock]]]] = [+ ("a list in a quotation", "> - item", [FormattedListItem(list_depth=1, quote_depth=1, marker="•", spans=[TextSpan(text="item")])]),+ ("a code block in a list item", "- item\n\n ```\n code\n ```", [_ITEM, FormattedCodeBlock(list_depth=1, lines=["code"])]),+ ("a table in a list item", "- item\n\n | a |\n | - |\n | 1 |", [_ITEM, FormattedTable(list_depth=1, rows=_ONE_CELL_ROWS)]),+ ("a rule in a list item", "- item\n\n ---", [_ITEM, FormattedRule(list_depth=1)]),+ (+ "a quotation in a list item",+ "- item\n\n > quoted",+ [_ITEM, FormattedParagraph(list_depth=1, quote_depth=1, spans=[TextSpan(text="quoted")])],+ ),+ ("a code block in a quotation", "> ```\n> code\n> ```", [FormattedCodeBlock(quote_depth=1, lines=["code"])]),+ ("a table in a quotation", "> | a |\n> | - |\n> | 1 |", [FormattedTable(quote_depth=1, rows=_ONE_CELL_ROWS)]),+ (+ "a rule in a quotation",+ "> before\n>\n> ---",+ [FormattedParagraph(quote_depth=1, spans=[TextSpan(text="before")]), FormattedRule(quote_depth=1)],+ ),+ (+ "a list in a quotation in a quotation",+ "> > - deep\n>\n> > back",+ [+ FormattedListItem(list_depth=1, quote_depth=2, marker="•", spans=[TextSpan(text="deep")]),+ FormattedParagraph(quote_depth=2, spans=[TextSpan(text="back")]),+ ],+ ),+ ("a heading in a quotation", "> # Title", [FormattedHeading(quote_depth=1, level=1, spans=[TextSpan(text="Title")])]),+ ]diff --git a/tests/unit/pipelex/tools/markdown/test_markdown_formatting.py b/tests/unit/pipelex/tools/markdown/test_markdown_formatting.pynew file mode 100644index 000000000..6f897e294--- /dev/null+++ b/tests/unit/pipelex/tools/markdown/test_markdown_formatting.py@@ -0,0 +1,239 @@+import pytest+from markdown_it.token import Token+from pytest_mock import MockerFixture++from pipelex.tools.jinja2.jinja2_render_budget import DEFAULT_RENDER_BUDGET_UNITS, RenderBudget+from pipelex.tools.markdown.markdown_formatting import (+ FormattedBlock,+ FormattedCodeBlock,+ FormattedHeading,+ FormattedListItem,+ FormattedMarkdown,+ FormattedParagraph,+ FormattedRule,+ FormattedSpan,+ FormattedTable,+ LineBreakSpan,+ TextSpan,+ format_markdown_within_budget,+ spans_text,+)+from tests.unit.pipelex.tools.markdown.test_data import MarkdownFormattingTestData+++def _formatted(*, markdown: str) -> FormattedMarkdown:+ return format_markdown_within_budget(markdown_text=markdown, budget=RenderBudget(total=DEFAULT_RENDER_BUDGET_UNITS))+++def _only_block_spans(*, markdown: str) -> list[FormattedSpan]:+ """The spans of a text that formats into one paragraph."""+ blocks = _formatted(markdown=markdown).blocks+ assert len(blocks) == 1+ paragraph = blocks[0]+ assert isinstance(paragraph, FormattedParagraph)+ return paragraph.spans+++def _list_lines(*, markdown: str) -> list[tuple[str, int, str | None, str]]:+ """Each list item of a text, and each paragraph inside one, as its kind, its list depth, its marker and its text."""+ lines: list[tuple[str, int, str | None, str]] = []+ for block in _formatted(markdown=markdown).blocks:+ match block:+ case FormattedListItem():+ lines.append((block.kind, block.list_depth, block.marker, spans_text(spans=block.spans)))+ case FormattedParagraph() if block.list_depth:+ lines.append((block.kind, block.list_depth, None, spans_text(spans=block.spans)))+ case _:+ pass+ return lines+++class _StubParser:+ """A parser handing back tokens a test wrote, standing in for the shared parser."""++ def __init__(self, *, tokens: list[Token]) -> None:+ self._tokens = tokens++ def parse(self, *_arguments: object) -> list[Token]:+ return self._tokens+++class TestMarkdownFormatting:+ def test_bold_italics_strikethrough_and_code_are_flagged(self) -> None:+ spans = _only_block_spans(markdown="Some **bold**, *italic*, ***both***, ~~struck~~ and `**code**`.")+ assert spans == [+ TextSpan(text="Some "),+ TextSpan(text="bold", bold=True),+ TextSpan(text=", "),+ TextSpan(text="italic", italic=True),+ TextSpan(text=", "),+ TextSpan(text="both", bold=True, italic=True),+ TextSpan(text=", "),+ TextSpan(text="struck", strikethrough=True),+ TextSpan(text=" and "),+ TextSpan(text="**code**", code=True),+ TextSpan(text="."),+ ]++ def test_nested_bullets_alternate_their_markers_by_depth(self) -> None:+ assert _list_lines(markdown="- one\n - two\n - three\n- four\n") == [+ ("list_item", 1, "•", "one"),+ ("list_item", 2, "–", "two"),+ ("list_item", 3, "•", "three"),+ ("list_item", 1, "•", "four"),+ ]++ def test_an_ordered_list_numbers_from_its_own_start(self) -> None:+ markdown = "7. seven\n8. eight\n 1. inner one\n 2. inner two\n9. nine\n"+ assert _list_lines(markdown=markdown) == [+ ("list_item", 1, "7.", "seven"),+ ("list_item", 1, "8.", "eight"),+ ("list_item", 2, "1.", "inner one"),+ ("list_item", 2, "2.", "inner two"),+ ("list_item", 1, "9.", "nine"),+ ]++ def test_a_later_paragraph_of_an_item_is_a_paragraph_at_its_list_depth(self) -> None:+ assert _list_lines(markdown="- first\n\n more about it\n- second\n") == [+ ("list_item", 1, "•", "first"),+ ("paragraph", 1, None, "more about it"),+ ("list_item", 1, "•", "second"),+ ]++ def test_an_item_opening_with_a_nested_list_prints_its_marker_alone(self) -> None:+ assert _list_lines(markdown="1. - inner\n") == [("list_item", 1, "1.", ""), ("list_item", 2, "–", "inner")]++ def test_an_item_opening_with_a_heading_carries_it_beside_its_marker(self) -> None:+ assert _formatted(markdown="1. ## Step *one*\n Then more.\n- # Title").blocks == [+ FormattedListItem(list_depth=1, marker="1.", heading_level=2, spans=[TextSpan(text="Step "), TextSpan(text="one", italic=True)]),+ FormattedParagraph(list_depth=1, spans=[TextSpan(text="Then more.")]),+ FormattedListItem(list_depth=1, marker="•", heading_level=1, spans=[TextSpan(text="Title")]),+ ]++ @pytest.mark.parametrize(("topic", "markdown", "expected"), MarkdownFormattingTestData.NESTING_CASES)+ def test_a_nested_block_keeps_its_list_and_quote_depths(self, topic: str, markdown: str, expected: list[FormattedBlock]) -> None:+ assert _formatted(markdown=markdown).blocks == expected, topic++ def test_headings_keep_their_level(self) -> None:+ blocks = _formatted(markdown="# Report\n\n## Findings\n\n###### Small *print*").blocks+ assert blocks == [+ FormattedHeading(level=1, spans=[TextSpan(text="Report")]),+ FormattedHeading(level=2, spans=[TextSpan(text="Findings")]),+ FormattedHeading(level=6, spans=[TextSpan(text="Small "), TextSpan(text="print", italic=True)]),+ ]++ def test_a_code_block_keeps_its_lines_as_written(self) -> None:+ code = 'if a < b && c > d:\n\tprint("<b>**not bold**</b>")'+ blocks = _formatted(markdown=f"```python\n{code}\n```\n\n indented <i>code</i>\n").blocks+ assert blocks == [+ FormattedCodeBlock(lines=["if a < b && c > d:", ' print("<b>**not bold**</b>")']),+ FormattedCodeBlock(lines=["indented <i>code</i>"]),+ ]++ def test_a_quotation_is_its_blocks_at_their_quote_depth(self) -> None:+ blocks = _formatted(markdown="> outer *words*\n>\n> > inner\n").blocks+ assert blocks == [+ FormattedParagraph(quote_depth=1, spans=[TextSpan(text="outer "), TextSpan(text="words", italic=True)]),+ FormattedParagraph(quote_depth=2, spans=[TextSpan(text="inner")]),+ ]++ def test_a_rule_is_a_block_of_its_own(self) -> None:+ blocks = _formatted(markdown="before\n\n---\n\nafter").blocks+ assert blocks == [FormattedParagraph(spans=[TextSpan(text="before")]), FormattedRule(), FormattedParagraph(spans=[TextSpan(text="after")])]++ def test_a_table_marks_its_header_row_and_pads_every_row(self) -> None:+ blocks = _formatted(markdown="| Quarter | Revenue |\n| --- | --- |\n| Q1 | **$1.2M** |\n| Q2 |\n").blocks+ assert len(blocks) == 1+ table = blocks[0]+ assert isinstance(table, FormattedTable)+ assert [row.is_header for row in table.rows] == [True, False, False]+ assert [[cell.spans for cell in row.cells] for row in table.rows] == [+ [[TextSpan(text="Quarter")], [TextSpan(text="Revenue")]],+ [[TextSpan(text="Q1")], [TextSpan(text="$1.2M", bold=True)]],+ [[TextSpan(text="Q2")], []],+ ]++ def test_a_table_cell_is_aligned_as_its_column_is(self) -> None:+ blocks = _formatted(markdown="| L | C | R | N |\n| :-- | :-: | --: | --- |\n| a | b | c | d |\n| e |\n").blocks+ assert len(blocks) == 1+ table = blocks[0]+ assert isinstance(table, FormattedTable)+ assert [[cell.alignment for cell in row.cells] for row in table.rows] == [["left", "center", "right", None]] * 3++ def test_a_soft_break_reads_as_a_space_and_a_hard_break_as_a_break(self) -> None:+ spans = _only_block_spans(markdown="one\ntwo \nthree\\\nfour")+ assert spans == [TextSpan(text="one two"), LineBreakSpan(), TextSpan(text="three"), LineBreakSpan(), TextSpan(text="four")]++ def test_raw_html_prints_as_text(self) -> None:+ formatted = _formatted(markdown='Tom & Jerry say <b>hello</b> & <br/> bye\n\n<div onclick="steal()">block</div>')+ assert str(formatted) == 'Tom & Jerry say <b>hello</b> & <br/> bye\n<div onclick="steal()">block</div>'++ def test_only_web_and_mail_links_keep_their_target(self) -> None:+ markdown = (+ "See [the site](https://pipelex.com), [the old site](http://pipelex.com), [write](mailto:[email protected]), "+ "[evil](javascript:alert(1)), [files](ftp://example.com/file), [the readme](README.md) and https://example.com/auto."+ )+ spans = _only_block_spans(markdown=markdown)+ linked = [(span.text, span.link) for span in spans if isinstance(span, TextSpan) and span.link is not None]+ assert linked == [+ ("the site", "https://pipelex.com"),+ ("the old site", "http://pipelex.com"),+ ("write", "mailto:[email protected]"),+ ("https://example.com/auto", "https://example.com/auto"),+ ]+ assert (+ spans_text(spans=spans)+ == "See the site, the old site, write, [evil](javascript:alert(1)), files, the readme and https://example.com/auto."+ )++ def test_an_image_prints_its_alt_text_in_italics(self) -> None:+ spans = _only_block_spans(markdown="Before  after")+ assert spans == [TextSpan(text="Before "), TextSpan(text="[image: a chart of sales]", italic=True), TextSpan(text=" after")]++ @pytest.mark.parametrize(+ ("markdown", "expected_text"),+ [+ (MarkdownFormattingTestData.DEEP_EMPHASIS, MarkdownFormattingTestData.DEEP_EMPHASIS_TEXT),+ (MarkdownFormattingTestData.DEEP_MIXED_EMPHASIS, MarkdownFormattingTestData.DEEP_MIXED_EMPHASIS_TEXT),+ ],+ )+ def test_emphasis_nested_hundreds_deep_prints_its_text_emphasized(self, markdown: str, expected_text: str) -> None:+ spans = _only_block_spans(markdown=markdown)+ assert spans_text(spans=spans) == expected_text+ assert all(isinstance(span, TextSpan) and span.italic for span in spans)++ def test_str_is_the_plain_text(self) -> None:+ assert str(_formatted(markdown=MarkdownFormattingTestData.EVERY_BLOCK)) == MarkdownFormattingTestData.EVERY_BLOCK_PLAIN_TEXT++ def test_the_plain_text_indents_every_line_of_a_list_item_s_blocks(self) -> None:+ plain_text = str(_formatted(markdown=MarkdownFormattingTestData.INSIDE_A_LIST_ITEM))+ assert plain_text == MarkdownFormattingTestData.INSIDE_A_LIST_ITEM_PLAIN_TEXT++ def test_a_text_without_markdown_is_its_plain_text(self) -> None:+ assert str(_formatted(markdown="Payment within 30 days, by transfer.")) == "Payment within 30 days, by transfer."++ @pytest.mark.parametrize("markdown", ["", " ", "\n\n\t\n"])+ def test_an_empty_or_blank_text_formats_into_nothing_and_is_false(self, markdown: str) -> None:+ formatted = _formatted(markdown=markdown)+ assert formatted == FormattedMarkdown(blocks=[])+ assert not formatted+ assert not str(formatted)++ def test_a_text_that_formats_into_a_block_is_true(self) -> None:+ assert _formatted(markdown="Due in *30* days.")+ assert _formatted(markdown="---")++ def test_unknown_nodes_print_their_text(self, mocker: MockerFixture) -> None:+ tokens = [+ Token(type="custom_block", tag="div", nesting=0, content="Custom <text>"),+ Token(type="weird_open", tag="x", nesting=1),+ Token(+ type="inline", tag="", nesting=0, content="inner", children=[Token(type="mystery_inline", tag="", nesting=0, content="Inner & text")]+ ),+ Token(type="weird_close", tag="x", nesting=-1),+ ]+ mocker.patch("pipelex.tools.markdown.markdown_parser.get_markdown_parser", return_value=_StubParser(tokens=tokens))+ assert _formatted(markdown="anything").blocks == [+ FormattedParagraph(spans=[TextSpan(text="Custom <text>")]),+ FormattedParagraph(spans=[TextSpan(text="Inner & text")]),+ ]diff --git a/tests/unit/pipelex/tools/markdown/test_markdown_parser.py b/tests/unit/pipelex/tools/markdown/test_markdown_parser.pyindex 5a3c2a5ab..ff84b13c3 100644--- a/tests/unit/pipelex/tools/markdown/test_markdown_parser.py+++ b/tests/unit/pipelex/tools/markdown/test_markdown_parser.py@@ -1,22 +1,19 @@import random+import pytestfrom markdown_it import MarkdownIt--from pipelex.tools.markdown.markdown_parser import get_markdown_parser, html_length_bound, render_markdown_as_html, table_cells_bound---class TestMarkdownParser:- def test_only_a_url_with_a_scheme_becomes_a_link(self) -> None:- html = render_markdown_as_html("See https://example.com/docs, README.md, www.example.com and [email protected].")- assert '<a href="https://example.com/docs">' in html- assert "README.md" in html- assert html.count("<a ") == 1-- def test_tables_and_strikethrough_are_formatted(self) -> None:- html = render_markdown_as_html("| a | b |\n| - | - |\n| 1 | 2 |\n\n~~gone~~")- assert "<table>" in html- assert "<s>gone</s>" in html-+from markdown_it.tree import SyntaxTreeNode++from pipelex.tools.markdown.markdown_parser import (+ MAX_INLINE_NESTING,+ get_markdown_parser,+ html_length_bound,+ markdown_syntax_tree,+ render_markdown_as_html,+ table_cells_bound,+)+from pipelex.tools.markdown.markdown_rules import plain_text+from tests.unit.pipelex.tools.markdown.test_data import MarkdownFormattingTestData# The pieces the generated documents are made of: emphasis and its unmatched delimiters, links, references and# images, tables in and out of containers, padded rows, code, escapes, and every line ending markdown-it knows,@@ -84,7 +81,62 @@ def _stock_parser() -> MarkdownIt:return parser-class TestMarkdownParserBounds:+def _deepest_inline_nesting(*, tree: SyntaxTreeNode) -> int:+ """How many emphasis, strong emphasis, strikethrough and link nodes the deepest path of a syntax tree crosses."""+ deepest = 0+ pending: list[tuple[SyntaxTreeNode, int]] = [(tree, 0)]+ while pending:+ node, depth = pending.pop()+ depth += node.type in {"em", "strong", "s", "link"}+ deepest = max(deepest, depth)+ pending.extend((child, depth) for child in node.children)+ return deepest+++class TestMarkdownParser:+ def test_only_a_url_with_a_scheme_becomes_a_link(self) -> None:+ html = render_markdown_as_html("See https://example.com/docs, README.md, www.example.com and [email protected].")+ assert '<a href="https://example.com/docs">' in html+ assert "README.md" in html+ assert html.count("<a ") == 1++ def test_an_explicit_link_keeps_its_target_in_html_whatever_its_scheme(self) -> None:+ html = render_markdown_as_html("[the readme](README.md), [files](ftp://example.com/file) and ")+ assert '<a href="README.md">the readme</a>' in html+ assert '<a href="ftp://example.com/file">files</a>' in html+ assert '<img src="chart.png" alt="a chart" />' in html++ def test_tables_and_strikethrough_are_formatted(self) -> None:+ html = render_markdown_as_html("| a | b |\n| - | - |\n| 1 | 2 |\n\n~~gone~~")+ assert "<table>" in html+ assert "<s>gone</s>" in html++ @pytest.mark.parametrize(+ ("markdown_text", "expected_text"),+ [+ (MarkdownFormattingTestData.DEEP_EMPHASIS, MarkdownFormattingTestData.DEEP_EMPHASIS_TEXT),+ (MarkdownFormattingTestData.DEEP_MIXED_EMPHASIS, MarkdownFormattingTestData.DEEP_MIXED_EMPHASIS_TEXT),+ ],+ )+ def test_a_syntax_tree_caps_inline_nesting_and_keeps_the_text(self, markdown_text: str, expected_text: str) -> None:+ tree = markdown_syntax_tree(tokens=get_markdown_parser().parse(markdown_text))+ assert _deepest_inline_nesting(tree=tree) == MAX_INLINE_NESTING+ assert plain_text(nodes=tree.children) == expected_text++ def test_an_image_past_the_cap_prints_its_alt_text_as_written(self) -> None:+ markdown_text = "*a " * MAX_INLINE_NESTING + "" + " c*" * MAX_INLINE_NESTING+ tree = markdown_syntax_tree(tokens=get_markdown_parser().parse(markdown_text))+ assert "[image: a *chart*]" in plain_text(nodes=tree.children)++ def test_nesting_within_the_cap_is_left_as_parsed(self) -> None:+ parser = get_markdown_parser()+ nested_to_the_cap = "*a " * MAX_INLINE_NESTING + "b" + " c*" * MAX_INLINE_NESTING+ for document in [*_generated_documents(), nested_to_the_cap]:+ tokens = parser.parse(document)+ expected_html = parser.renderer.render(tokens, parser.options, {})+ markdown_syntax_tree(tokens=tokens)+ assert parser.renderer.render(tokens, parser.options, {}) == expected_html, repr(document)+def test_joining_fragments_renders_as_markdown_it_does(self) -> None:stock = _stock_parser()parser = get_markdown_parser()