Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

fix(latex): math-mode parity — a lone dollar must not disable body escaping (#6432)

LearningCircuit83c8d8256eeb
diff --git a/changelog.d/6432.security.md b/changelog.d/6432.security.md
new file mode 100644
index 0000000000..edf38793ed
--- /dev/null
+++ b/changelog.d/6432.security.md
@@ -0,0 +1 @@
+LaTeX export now decides math mode with a left-to-right scanner instead of whole-document dollar parity: an unpaired $ (a price in prose) no longer disables escaping for the rest of the report, and lone dollars are escaped as literal text. Inline $...$ spans are scoped to a single paragraph (never crossing a blank line, like pandoc) and close at the first unescaped $ after the opener; if that $ is preceded by whitespace or followed by a digit, the opener is prose instead (so $5-$10 stays two prices), and a $ rejected for the digit after it cannot open a span either. Prose segments go through the export's single-pass escaper for every LaTeX special, except that a markdown \$ exports as \$ (a literal dollar) rather than a backslash followed by a dollar. Math spans are still emitted verbatim.
diff --git a/src/local_deep_research/text_optimization/citation_formatter.py b/src/local_deep_research/text_optimization/citation_formatter.py
index 486e7f50d5..d4b77db5e9 100644
--- a/src/local_deep_research/text_optimization/citation_formatter.py
+++ b/src/local_deep_research/text_optimization/citation_formatter.py
@@ -1697,6 +1697,206 @@ class RISExporter:
)
+# Body prose is markdown, where "\$" is the escape for a literal
+# dollar. The escaper therefore consumes a "\$" pair as ONE token and
+# emits "\$" (a literal dollar in LaTeX too); every other special,
+# every other backslash included, goes through the _escape_latex_text
+# map. Pair-first alternation in a single regex pass means nothing is
+# ever escaped twice: "\$" never becomes "\textbackslash{}\$" (which
+# would print a stray backslash), and the output never holds "\\"
+# (a line break) or a bare "$" (a live math shift).
+_LATEX_BODY_TEXT_RE = re.compile(
+ r"\\\$|[" + re.escape("".join(_LATEX_TEXT_SPECIALS)) + "]"
+)
+
+
+def _escape_latex_body_text(text: str) -> str:
+ r"""Escape a prose segment of the export body in one pass.
+
+ Identical to :func:`_escape_latex_text` except that a markdown
+ ``\$`` (escaped dollar) becomes ``\$`` rather than
+ ``\textbackslash{}\$``. A backslash run before a dollar pairs only
+ its LAST backslash with the dollar; the others are literal
+ backslashes (``\\$`` -> ``\textbackslash{}\$``).
+ """
+ return _LATEX_BODY_TEXT_RE.sub(
+ lambda m: (
+ "\\$" if m.group() == "\\$" else _LATEX_TEXT_SPECIALS[m.group()]
+ ),
+ text,
+ )
+
+
+# A blank line ends a block: two line breaks separated only by
+# horizontal whitespace and carriage returns. Legacy report_content
+# rows store CRLF newlines and LLM prose often leaves spaces or tabs
+# on the otherwise blank line, so probing for a literal "\n\n" misses
+# real paragraph boundaries; the boundary is matched structurally.
+_BLOCK_END = re.compile(r"\r?\n[ \t\r]*\r?\n")
+
+# Verdicts _split_math_segments assigns to each live "$" as the closer
+# of the opener before it: acceptable; rejected because a digit follows
+# it (a price: "$5-$10"); or rejected because whitespace precedes it.
+_CLOSER_OK = 0
+_CLOSER_DIGIT = 1
+_CLOSER_SPACE = 2
+
+
+def _split_math_segments(text: str) -> list[tuple[str, bool]]:
+ r"""Split *text* into ``(segment, is_math)`` pairs, left to right.
+
+ ``$$...$$`` display spans pair first, then ``$...$`` inline spans.
+ Neither kind ever crosses a blank line (pandoc scopes math to a
+ single block, so a stray ``$$`` or a price in one paragraph can
+ never pair with a delimiter in another; an opener unpaired within
+ its block is literal text). An inline span's opening ``$`` must
+ not be followed by whitespace, and it closes at the FIRST live
+ ``$`` after it in the block, as in pandoc: that closer must not be
+ preceded by whitespace and must not be followed by a digit, and if
+ it is, the OPENER fails and reads as prose (``$5-$10`` is two
+ prices, not ``$5-$`` around ``10``). Rejected closers are never
+ stepped over on the way to a farther one, which would emit a span
+ holding a raw interior dollar. A closer rejected for the digit
+ after it is itself a price and cannot open a span either; pandoc
+ would let it open one, so ``$5-$10 per unit ($n$)`` would make
+ ``$10 per unit ($`` math there, raw prose between two prices
+ (whitespace-rejected closers still open: ``$5 so $x$`` keeps
+ ``$x$``). Mistaking math for prose only escapes it; mistaking
+ prose for math emits its ``%`` and ``&`` raw. (Mirror case: because
+ the rejected closer cannot open, a later ``$`` glued to the text on
+ its left can open instead and pair farther on, so
+ ``$5-$10 per hour (US$), 50% more (US$)`` makes ``$), 50% more (US$``
+ math; this is rarer than the cases the rule prevents.) A ``$``
+ preceded by an odd run of backslashes is a literal dollar
+ (as in ``\$5``), never an opener or a closer; after an even run
+ (``\\$``: the backslashes escape each other) the dollar is as
+ live as a bare one, in both directions. Every other dollar is a
+ literal character inside a text segment. Math segments include their
+ delimiters and are returned verbatim.
+ """
+ segments: list[tuple[str, bool]] = []
+ start = 0 # start of the current text run
+ i = 0
+ n = len(text)
+ # Whether a "$" can close inline math does not depend on the
+ # opener, so every live candidate position and its verdict is
+ # collected once up front; the pairing below advances one pointer
+ # over them, left to right, and never rescans. A "$" ending an odd
+ # backslash run is a literal dollar and never a delimiter in
+ # either direction: excluded here so the walk cannot CLOSE a span
+ # on it, exactly as the "\x" skip below already keeps it from
+ # opening one. A "$" after an even run ("\\$": the backslashes
+ # escape each other) is live and stays a closer candidate; a
+ # single-character lookbehind excluded it too and left "$x\\$"
+ # unpaired. That pattern's lookbehind stops it starting mid-run,
+ # so each match spans the full run and the scan stays linear
+ # even on backslash walls.
+ odd_run_dollars = {
+ m.end() - 1 for m in re.finditer(r"(?<!\\)(?:\\\\)*\\\$", text)
+ }
+ dollars = [
+ m.start()
+ for m in re.finditer(r"\$", text)
+ if m.start() not in odd_run_dollars
+ ]
+ # Whitespace is checked first: a candidate with both flaws ("$fee
+ # $5$") is not a price and may still open its own span.
+ closer_verdict = [
+ _CLOSER_SPACE
+ if p > 0 and text[p - 1].isspace()
+ else _CLOSER_DIGIT
+ if p + 1 < n and text[p + 1] in "0123456789"
+ else _CLOSER_OK
+ for p in dollars
+ ]
+ cand = 0 # index into dollars: first candidate after the opener
+ price_dollar = -1 # a digit-rejected closer: never an opener
+ # First blank line at or after the current position.
+ block_end = n
+ block_end_known = False
+ while i < n:
+ if text[i] == "\\":
+ # A backslash escapes the next character: "\$" is a
+ # literal dollar, never a math delimiter.
+ i += 2
+ continue
+ if text[i] != "$":
+ i += 1
+ continue
+ if i + 1 < n and text[i + 1] == "$":
+ # A display opener pairs only inside its own block, so a
+ # stray "$$" in one paragraph cannot capture a genuine
+ # display span in another (a blank line is never inside
+ # math). The closer scan walks the block: "\x" pairs are
+ # skipped (an interior "\$" is a literal dollar, and the
+ # "$" of "\$$" is not the lead of a closer), the first
+ # raw single "$" means the opener is unpaired (a display
+ # span has no single-dollar interior), and the first raw
+ # "$$" closes.
+ if not block_end_known or i > block_end:
+ boundary = _BLOCK_END.search(text, i)
+ block_end = boundary.start() if boundary is not None else n
+ block_end_known = True
+ close = -1
+ j = i + 2
+ while j < block_end:
+ if text[j] == "\\":
+ j += 2
+ elif text[j] == "$":
+ if j + 1 < n and text[j + 1] == "$":
+ close = j
+ break
+ else:
+ j += 1
+ if close != -1:
+ segments.append((text[start:i], False))
+ segments.append((text[i : close + 2], True))
+ i = close + 2
+ start = i
+ else:
+ # Unpaired $$ within its block: literal text, like
+ # any lone dollar.
+ i += 2
+ continue
+ # Single '$': find a closing '$' the rules accept, without
+ # leaving the paragraph (inline math never crosses a blank
+ # line).
+ if i == price_dollar or (i + 1 < n and text[i + 1].isspace()):
+ i += 1
+ continue
+ # The first blank line at or after i only moves forward as i
+ # does, so the search is cached until i passes its match.
+ if not block_end_known or i > block_end:
+ boundary = _BLOCK_END.search(text, i)
+ block_end = boundary.start() if boundary is not None else n
+ block_end_known = True
+ # The closer is the first live "$" after the opener. Openers
+ # only move right, so the pointer does too.
+ while cand < len(dollars) and dollars[cand] <= i:
+ cand += 1
+ verdict = (
+ closer_verdict[cand]
+ if cand < len(dollars) and dollars[cand] < block_end
+ else None
+ )
+ if verdict == _CLOSER_OK:
+ closer = dollars[cand]
+ segments.append((text[start:i], False))
+ segments.append((text[i : closer + 1], True))
+ i = closer + 1
+ start = i
+ else:
+ # No closer in this block, or the first one is preceded
+ # by whitespace or followed by a digit: the opener fails
+ # (never stepped past to a farther "$") and is a literal
+ # dollar. A digit-rejected closer is a price as well.
+ if verdict == _CLOSER_DIGIT:
+ price_dollar = dollars[cand]
+ i += 1
+ segments.append((text[start:], False))
+ return segments
+
+
class LaTeXExporter:
"""Export markdown documents to LaTeX format."""
@@ -1729,39 +1929,45 @@ class LaTeXExporter:
# Convert markdown to LaTeX
body_content = content
- # Escape special LaTeX characters but preserve math mode
- # Split by $ to preserve math sections
- parts = body_content.split("$")
- for i in range(len(parts)):
- # Even indices are outside math mode
- if i % 2 == 0:
- # Only escape if not inside $$
- if not (
- i > 0
- and parts[i - 1] == ""
- and i < len(parts) - 1
- and parts[i + 1] == ""
- ):
- # Preserve certain patterns that will be processed later
- # like headings (#), emphasis (*), and citations ([n])
- lines = parts[i].split("\n")
- for j, line in enumerate(lines):
- # Escape every LaTeX-special character outside math
- # mode. Markdown markers (*, `, [n]) are not
- # LaTeX-special and survive for their own
- # conversions below.
- heading = _HEADING_PREFIX_RE.match(line)
- if heading:
- # Keep the '#' markers intact for the heading
- # conversion; escape the text after them so
- # \section{...} receives literal text only.
- lines[j] = heading.group(1) + _escape_latex_text(
- heading.group(2)
- )
- else:
- lines[j] = _escape_latex_text(line)
- parts[i] = "\n".join(lines)
- body_content = "$".join(parts)
+ # Escape special LaTeX characters but preserve math mode. A
+ # left-to-right scanner pairs $$...$$ display spans and $...$
+ # inline spans, which pass through verbatim; every other dollar
+ # is prose. The previous whole-document split made ONE unpaired
+ # dollar (a price in prose) invert the parity and silently
+ # disable escaping for the rest of the report.
+ escaped_parts: list[str] = []
+ at_line_start = True
+ for segment, is_math in _split_math_segments(body_content):
+ if is_math:
+ escaped_parts.append(segment)
+ else:
+ lines = segment.split("\n")
+ for j, line in enumerate(lines):
+ # Escape every LaTeX-special character outside math
+ # mode. Markdown markers (*, `, [n]) are not
+ # LaTeX-special and survive for their own
+ # conversions below.
+ heading = (
+ _HEADING_PREFIX_RE.match(line)
+ if j > 0 or at_line_start
+ else None
+ )
+ if heading:
+ # Keep the '#' markers intact for the heading
+ # conversion; escape the text after them so
+ # \section{...} receives literal text only. Only
+ # a TRUE line start qualifies: a text segment
+ # that resumes mid-line after a math span
+ # ("$x$ # y") is prose, and its "#" is escaped.
+ lines[j] = heading.group(1) + _escape_latex_body_text(
+ heading.group(2)
+ )
+ else:
+ lines[j] = _escape_latex_body_text(line)
+ escaped_parts.append("\n".join(lines))
+ if segment:
+ at_line_start = segment.endswith("\n")
+ body_content = "".join(escaped_parts)
# Convert headings
for pattern, replacement in self.heading_patterns:
diff --git a/tests/text_optimization/test_latex_math_parity.py b/tests/text_optimization/test_latex_math_parity.py
new file mode 100644
index 0000000000..e62447657a
--- /dev/null
+++ b/tests/text_optimization/test_latex_math_parity.py
@@ -0,0 +1,604 @@
+r"""Math-mode parity contracts for the LaTeX exporter's body escaper.
+
+The promoted contract in
+``tests/web/services/test_export_generation_contracts.py`` and the math
+preservation tests in ``test_citation_exporter_edge_cases.py`` already
+pin the core scenarios (escaping stays active after a lone ``$``;
+inline and display pairs preserved). This file pins only the edges
+those suites do not: the lone ``$`` itself must be escaped as a
+literal; a well-formed pair AFTER a lone dollar is still recognised as
+math; inline spans stay inside one paragraph and close at the first
+live dollar, failing the opener when that dollar is whitespace-preceded
+or digit-followed; a markdown ``\$`` in prose exports as ``\$`` while
+every other backslash becomes ``\textbackslash{}``; the block-scoping
+and opener guards are pinned against unspaced math openers (a
+space-preceded opener lets the whitespace closer rule mask a scoping
+revert); backslash runs of odd length -- only odd -- neutralise the
+dollar that ends them in the scanner; and a ``#`` in a text segment
+that resumes mid-line after a math span is escaped, not kept as a
+heading marker.
+"""
+
+from __future__ import annotations
+
+import time
+
+from local_deep_research.text_optimization import citation_formatter
+from local_deep_research.text_optimization.citation_formatter import (
+ LaTeXExporter,
+ _escape_latex_body_text,
+ _split_math_segments,
+)
+from hypothesis import given, settings, strategies as st
+
+_META_ALPHABET = "$\\{}[]#%&~^【】,0123456789 \n\t-URL:" + "ab"
+_meta_text = st.text(alphabet=_META_ALPHABET, min_size=0, max_size=400)
+
+
+class TestLoneDollarDoesNotDisableEscaping:
+ """An unpaired ``$`` is prose, not a math-mode opener."""
+
+ def test_lone_dollar_itself_is_escaped_as_a_literal(self):
+ exporter = LaTeXExporter()
+ content = "# Title\n\nThis costs $100 per unit.\n"
+ result = exporter.export_to_latex(content)
+
+ # A bare $ in text mode is a TeX error; it must be escaped.
+ assert "\\$100" in result
+
+ def test_dollar_followed_by_whitespace_never_opens_math(self):
+ r"""A ``$`` followed by whitespace is prose, not an opener.
+
+ Pins the OUTCOME, not the opener guard: the lone ``$`` lands
+ in a text segment (escaped) and the genuine pair behind it
+ still exports as math. These assertions hold with the guard
+ deleted too, because the pair's space-preceded opening ``$``
+ is a whitespace-rejected closer that fails the lone dollar as
+ an opener (pandoc's backtrack rule) before the guard ever
+ runs. The guard itself is pinned by
+ ``test_space_followed_dollar_cannot_steal_a_pairs_opener``,
+ whose pair opener is not space-preceded.
+ """
+ exporter = LaTeXExporter()
+ content = "# Title\n\nTotal: $ and $x$ here.\n"
+ result = exporter.export_to_latex(content)
+
+ assert "\\$ and" in result # the lone dollar: escaped as text
+ assert "$x$" in result # the genuine pair: survives as math
+
+ def test_space_followed_dollar_cannot_steal_a_pairs_opener(self):
+ r"""Pin the opener whitespace guard itself.
+
+ ``Total: $ ($x$) here.`` -- the lone ``$`` is followed by a
+ space, and the pair's opening ``$`` sits right after ``(``,
+ which no closer rule rejects. Without the guard the lone
+ dollar opens a span ending at the pair's opening ``$``
+ (``$ ($`` rides through as math) and destroys the pair; with
+ it, the lone dollar is escaped prose and the pair survives
+ intact. (In the ``...never_opens_math`` test above, the space
+ before the pair's ``$`` masks the guard -- which is why this
+ pin needs an unspaced opener.)
+ """
+ exporter = LaTeXExporter()
+ content = "# Title\n\nTotal: $ ($x$) here.\n"
+ result = exporter.export_to_latex(content)
+
+ assert "Total: \\$ ($x$) here." in result
+
+
+class TestPreEscapedDollarSurvivesVerbatim:
+ r"""A markdown ``\$`` is a literal dollar and exports as ``\$``.
+
+ The app's frontend (marked-katex-extension) renders ``\$`` as a
+ literal dollar and never opens math on it. The body escaper
+ consumes the ``\$`` pair as one token and emits ``\$``; escaping
+ the backslash and the dollar separately would print a stray
+ backslash (``\textbackslash{}\$``), and escaping only the dollar
+ would emit ``\\$`` -- a line-break command followed by a live
+ math-mode dollar.
+ """
+
+ def test_pre_escaped_dollar_in_prose_is_not_double_escaped(self):
+ exporter = LaTeXExporter()
+ content = "# Title\n\nIt costs \\$5 at 100% of R&D.\n"
+ result = exporter.export_to_latex(content)
+
+ assert "It costs \\$5 at 100\\% of R\\&D." in result
+ assert "\\textbackslash{}\\$5" not in result
+ assert "\\\\$5" not in result
+
+ def test_body_escaper_pairs_only_the_backslash_next_to_the_dollar(self):
+ r"""One regex pass; only ``\$`` is special, other backslashes are text.
+
+ In a run of backslashes before a dollar only the last one pairs
+ with it; the rest are literal backslashes, exactly as main's
+ ``_escape_latex_text`` renders them. No output holds ``\\``
+ (a line break) or a dollar not preceded by a single backslash.
+ """
+ assert _escape_latex_body_text("\\$5") == "\\$5"
+ assert _escape_latex_body_text("\\\\$5") == "\\textbackslash{}\\$5"
+ assert (
+ _escape_latex_body_text("\\\\\\$5")
+ == "\\textbackslash{}\\textbackslash{}\\$5"
+ )
+ assert _escape_latex_body_text("a\\b$c") == "a\\textbackslash{}b\\$c"
+ assert _escape_latex_body_text("\\") == "\\textbackslash{}"
+
+ def test_pre_escaped_dollar_never_opens_math_in_the_scanner(self):
+ segments = _split_math_segments(r"It costs \$5 and $x$ today.")
+
+ assert segments == [
+ ("It costs \\$5 and ", False),
+ ("$x$", True),
+ (" today.", False),
+ ]
+
+
+class TestMathSpansSurviveIntact:
+ """Matched math spans reach the output untouched."""
+
+ def test_math_pair_after_a_lone_dollar_is_still_recognized(self):
+ r"""A genuine pair after a lone ``$`` is math in its own right.
+
+ Pins the prose AFTER the span being escaped and ``_``/``^``
+ staying raw inside the math span (main's edge-case tests pin
+ ``^`` only). The recognition assertions are belt-and-braces:
+ they hold with the opener guard deleted, because the pair's
+ space-preceded opening ``$`` fails the lone dollar as a
+ closer before the guard ever runs -- so this test does not
+ pin the guard, and the ``\$ and`` assertion in the
+ ``...never_opens_math`` test above does not either; the
+ guard's regression signal lives in
+ ``test_space_followed_dollar_cannot_steal_a_pairs_opener``.
+ """
+ exporter = LaTeXExporter()
+ content = "# Title\n\nCosts $ 5, and $x_2^2$ holds 100% of cases.\n"
+ result = exporter.export_to_latex(content)
+
+ assert "$x_2^2$" in result # the pair itself, delimiters intact
+ assert "100\\% of cases" in result # prose after it: text, escaped
+
+ def test_display_math_delimiters_survive(self):
+ """The ``$$`` delimiters themselves must reach the output.
+
+ The edge-case suite pins the math CONTENT (``E = mc^2``
+ survives, no ``\\textasciicircum``); nothing else pins that
+ the delimiters are emitted, which the scanner must do by
+ returning matched spans verbatim.
+ """
+ exporter = LaTeXExporter()
+ content = "# Title\n\n$$E = mc^2$$\n\nSome text.\n"
+ result = exporter.export_to_latex(content)
+
+ assert "$$E = mc^2$$" in result
+
+
+class TestInlineMathIsBlockScoped:
+ """Inline ``$...$`` spans never cross a blank line.
+
+ Pandoc scopes inline math to a single block, so a price in one
+ paragraph must not pair with the closer of genuine math in a later
+ paragraph and swallow both paragraphs as one span.
+ """
+
+ def test_price_paragraph_does_not_swallow_a_later_math_paragraph(self):
+ exporter = LaTeXExporter()
+ content = (
+ "# Title\n\n"
+ "Licences cost $5 per seat at 100% of R&D budget.\n\n"
+ "The identity $x^2$ holds.\n"
+ )
+ result = exporter.export_to_latex(content)
+
+ # Paragraph A: prose escaped, the price escaped as a literal.
+ assert "\\$5" in result
+ assert "100\\%" in result
+ assert "R\\&D" in result
+ # Paragraph B: the math pair survives as math.
+ assert "$x^2$" in result
+
+ def test_crlf_blank_line_still_scopes_the_price_paragraph(self):
+ r"""A CRLF blank line is as much a boundary as ``\n\n``.
+
+ Legacy ``report_content`` rows carry CRLF newlines (nothing
+ normalizes them before the exporter), so a price in paragraph
+ A must not pair with paragraph B's math when the blank line
+ between them is ``\r\n\r\n``.
+ """
+ exporter = LaTeXExporter()
+ content = (
+ "# Title\r\n\r\n"
+ "Licences cost $5 per seat at 100% of R&D budget.\r\n\r\n"
+ "The identity $x^2$ holds.\r\n"
+ )
+ result = exporter.export_to_latex(content)
+
+ # Paragraph A: prose escaped, the price escaped as a literal.
+ assert "\\$5" in result
+ assert "100\\%" in result
+ assert "R\\&D" in result
+ # Paragraph B: the math pair survives as math.
+ assert "$x^2$" in result
+
+ def test_blank_line_of_spaces_still_scopes_the_price_paragraph(self):
+ r"""A blank line holding spaces is as much a boundary as ``\n\n``.
+
+ LLM prose often leaves trailing spaces on the otherwise blank
+ line (``\n \n``); that is still one paragraph boundary, so
+ the price paragraph must stay separate from the math
+ paragraph.
+ """
+ exporter = LaTeXExporter()
+ content = (
+ "# Title\n \n"
+ "Licences cost $5 per seat at 100% of R&D budget.\n \n"
+ "The identity $x^2$ holds.\n"
+ )
+ result = exporter.export_to_latex(content)
+
+ # Paragraph A: prose escaped, the price escaped as a literal.
+ assert "\\$5" in result
+ assert "100\\%" in result
+ assert "R\\&D" in result
+ # Paragraph B: the math pair survives as math.
+ assert "$x^2$" in result
+
+ def test_lf_blank_line_scopes_price_against_unspaced_math_opener(self):
+ r"""The block machinery ALONE must block the cross-block pair.
+
+ The three tests above use ``The identity $x^2$ holds.``, whose
+ math opener is space-preceded: the whitespace closer rule
+ independently rejects the cross-block candidate, so those
+ assertions stay green even with the block machinery deleted
+ and cannot see a scoping revert. Here paragraph B's math
+ opener sits directly after a non-space character
+ (``notes($x$)``), so its ``$`` is an acceptable closer
+ candidate and ONLY the blank line stops paragraph A's price
+ from pairing with it across blocks.
+ """
+ exporter = LaTeXExporter()
+ content = "# Title\n\nCosts $5 at 100% of R&D.\n\nnotes($x$) ok.\n"
+ result = exporter.export_to_latex(content)
+
+ # Paragraph A: prose escaped, the price escaped as a literal.
+ assert "\\$5 at 100\\% of R\\&D." in result
+ # Paragraph B: the unspaced-opener math pair survives as math.
+ assert "notes($x$) ok." in result
+
+ def test_crlf_blank_line_scopes_price_against_unspaced_math_opener(self):
+ r"""CRLF form of the unspaced-opener scoping pin.
+
+ With ``_BLOCK_END`` reverted to a literal ``\n\n`` probe, the
+ ``\r\n\r\n`` boundary is invisible and the price pairs with
+ the unspaced ``notes($`` opener across blocks; the structural
+ probe must catch it.
+ """
+ exporter = LaTeXExporter()
+ content = (
+ "# Title\r\n\r\nCosts $5 at 100% of R&D.\r\n\r\nnotes($x$) ok.\r\n"
+ )
+ result = exporter.export_to_latex(content)
+
+ assert "\\$5 at 100\\% of R\\&D." in result
+ assert "notes($x$) ok." in result
+
+ def test_space_padded_blank_line_scopes_price_against_unspaced_opener(
+ self,
+ ):
+ r"""Space-padded blank line form of the unspaced-opener pin.
+
+ A literal ``\n\n`` probe cannot see the ``\n \n`` boundary
+ either; the structural probe must.
+ """
+ exporter = LaTeXExporter()
+ content = "# Title\n \nCosts $5 at 100% of R&D.\n \nnotes($x$) ok.\n"
+ result = exporter.export_to_latex(content)
+
+ assert "\\$5 at 100\\% of R\\&D." in result
+ assert "notes($x$) ok." in result
+
+
+class _CountingPattern:
+ """Stand-in for ``_BLOCK_END`` that counts ``search`` calls."""
+
+ def __init__(self, inner):
+ self.inner = inner
+ self.calls = 0
+
+ def search(self, *args):
+ self.calls += 1
+ return self.inner.search(*args)
+
+
+class TestDollarDenseProseScansInLinearTime:
+ """The scanner must not rescan the block per opener."""
+
+ def test_blank_line_search_runs_once_per_block_not_per_opener(
+ self, monkeypatch
+ ):
+ r"""Every ``$a`` below is an opener that fails on its closer.
+
+ The first blank line after an opener is cached until the scan
+ passes it. Searching again per opener walks to the end of the
+ block every time -- quadratic on a paragraph of prices, on the
+ synchronous export route. Counting the searches pins the cache
+ deterministically; wall-clock bounds alone are too loose to
+ see it at any size a test can afford.
+ """
+ spy = _CountingPattern(citation_formatter._BLOCK_END)
+ monkeypatch.setattr(citation_formatter, "_BLOCK_END", spy)
+ blocks = 20
+ text = ("$a " * 200 + "\n\n") * blocks
+
+ segments = _split_math_segments(text)
+
+ assert not any(is_math for _, is_math in segments)
+ assert spy.calls <= blocks + 1
+
+ def test_price_wall_export_is_not_quadratic(self):
+ r"""A ``$a `` wall of 60 000 failed openers exports in linear time.
+
+ Each opener's closer is the next dollar, rejected because a
+ space precedes it. The scan advances one candidate pointer;
+ restarting it (or the blank-line search) per opener makes this
+ wall take minutes. The bound is deliberately generous: the
+ linear scan takes well under a second here.
+ """
+ exporter = LaTeXExporter()
+ wall = "$a " * 60000
+
+ started = time.perf_counter()
+ result = exporter.export_to_latex(wall)
+ elapsed = time.perf_counter() - started
+
+ assert result.count("\\$a") == 60000 # every dollar: prose
+ assert elapsed < 15.0
+
+
+class TestCloserFollowedByDigitIsRejected:
+ """A closing ``$`` followed by a digit reads as a price.
+
+ Pandoc (and the app's frontend) reject such closers, so ``$5-$10``
+ is two prices in prose -- not ``$5-$`` of math around ``10``. The
+ rejection fails the OPENER: stepping past the ``$10`` to a farther
+ closer would emit one span holding that raw interior dollar. The
+ rejected ``$10`` is a price too and never opens a span of its own.
+ """
+
+ def test_price_range_stays_prose(self):
+ exporter = LaTeXExporter()
+ content = "# Title\n\nIt ranged from $5-$10 per unit.\n"
+ result = exporter.export_to_latex(content)
+
+ assert "from \\$5-\\$10 per unit" in result
+
+ def test_price_range_before_a_genuine_pair_does_not_swallow_it(self):
+ text = "Prices of $5-$10 per unit ($n$) apply."
+
+ assert _split_math_segments(text) == [
+ ("Prices of $5-$10 per unit (", False),
+ ("$n$", True),
+ (") apply.", False),
+ ]
+ result = LaTeXExporter().export_to_latex("# Title\n\n" + text + "\n")
+ assert "Prices of \\$5-\\$10 per unit ($n$) apply." in result
+
+ def test_price_range_cannot_open_a_span_over_prose(self):
+ r"""The rejected ``$10`` must not pair with ``US$``.
+
+ Were it an opener, ``$10 (50% off), paid in US$`` would ride
+ raw as math -- and ``%`` in math is a TeX comment that eats
+ the rest of the line.
+ """
+ text = "Plans cost $5-$10 (50% off), paid in US$ or A$."
+
+ assert _split_math_segments(text) == [(text, False)]
+ result = LaTeXExporter().export_to_latex("# Title\n\n" + text + "\n")
+ assert (
+ "Plans cost \\$5-\\$10 (50\\% off), paid in US\\$ or A\\$."
+ in result
+ )
+
+
+class TestBackslashedDollarNeverClosesMath:
+ r"""A ``\$`` sitting between two genuine dollars is not a closer.
+
+ The scanner promises a backslash-preceded ``$`` is a literal
+ dollar in BOTH directions, but the closer precompute accepted it:
+ ``$x\$y$`` closed at the ``\$``, the genuine closer was then
+ escaped as prose, and the math left the document unbalanced.
+ """
+
+ def test_backslashed_dollar_is_not_a_closer_in_the_scanner(self):
+ segments = _split_math_segments(r"Cost $x\$y$ total")
+
+ assert segments == [
+ ("Cost ", False),
+ ("$x\\$y$", True),
+ (" total", False),
+ ]
+
+ def test_span_closes_at_the_genuine_dollar_not_the_escaped_one(self):
+ exporter = LaTeXExporter()
+ content = "# Title\n\nCost $x\\$y$ and 100% of R&D.\n"
+ result = exporter.export_to_latex(content)
+
+ # One math span, closed by the real "$"; the escaped interior
+ # dollar rides verbatim inside it.
+ assert "$x\\$y$" in result
+ # The prose AFTER the span is still escaped as text.
+ assert "100\\% of R\\&D" in result
+
+
+class TestEvenBackslashRunKeepsTheDollarLive:
+ r"""Only an odd backslash run escapes the ``$`` that ends it.
+
+ ``\\$`` is an escaped backslash followed by a live dollar. In the
+ scanner it is a legitimate closer (excluding it left ``$x\\$``
+ unpaired -- a strict regression: the pre-scanner exporter kept
+ this shape one balanced math span). In prose, the escaper renders
+ the first backslash as ``\textbackslash{}`` and the ``\$`` pair as
+ an escaped dollar: never a live dollar, never ``\\``.
+ """
+
+ def test_even_run_dollar_in_prose_is_escaped(self):
+ r"""``\\$5`` in prose exports as ``\textbackslash{}\$5``.
+
+ No line break (``\\``) and no live dollar reach the ``.tex``;
+ the ``100\%`` and ``R\&D`` around it stay escaped text.
+ """
+ exporter = LaTeXExporter()
+ content = "# Title\n\nIt costs \\\\$5 today at 100% of R&D.\n"
+ result = exporter.export_to_latex(content)
+
+ assert (
+ "It costs \\textbackslash{}\\$5 today at 100\\% of R\\&D." in result
+ )
+
+ def test_even_run_dollar_is_a_closer_in_the_scanner(self):
+ r"""``$x\\$`` pairs as one span, closed by the even-run ``$``."""
+ segments = _split_math_segments(r"a $x\\$ b")
+
+ assert segments == [
+ ("a ", False),
+ ("$x\\\\$", True),
+ (" b", False),
+ ]
+
+ def test_even_run_closer_keeps_the_span_balanced_in_export(self):
+ r"""The balanced-span behavior reaches the exported output.
+
+ The math span ``$x\\$`` rides through verbatim (balanced
+ delimiters, no unclosed math) and the prose after it is
+ escaped as text.
+ """
+ exporter = LaTeXExporter()
+ content = "# Title\n\na $x\\\\$ b and 100% after.\n"
+ result = exporter.export_to_latex(content)
+
+ assert "$x\\\\$ b and 100\\% after." in result
+
+
+class TestWhitespaceRejectedCloserFailsTheOpener:
+ r"""A closer ``$`` preceded by whitespace fails the whole opener.
+
+ Pandoc's rule: when the closing candidate is preceded by
+ whitespace, the opening ``$`` was prose. Skipping past such a
+ candidate to a farther ``$`` emits one math span holding a raw
+ interior dollar -- byte-identical to the pre-PR vulnerable
+ exporter. Unlike a digit-rejected candidate, a whitespace-rejected
+ one may still open its own span.
+ """
+
+ def test_price_then_pair_in_one_block_is_not_one_span(self):
+ r"""A price sentence and a genuine pair share one paragraph.
+
+ The closer candidate after ``so `` is whitespace-preceded, so
+ the price's ``$`` must fail as an opener (escaped prose) and
+ the pair behind it must pair on its own -- not one span
+ ``$5 today, so $x$`` with a raw interior dollar.
+ """
+ exporter = LaTeXExporter()
+ content = "# Title\n\nCosts $5 today, so $x$ holds 100%.\n"
+ result = exporter.export_to_latex(content)
+
+ assert "\\$5 today, so $x$ holds" in result
+ assert "100\\%" in result
+
+ def test_digit_followed_candidate_after_a_space_still_fails_opener(self):
+ r"""``$fee $5$``: the interior candidate has BOTH flaws.
+
+ The ``$5`` candidate is whitespace-preceded AND digit-followed;
+ whitespace wins, so ``$fee`` is prose and ``$5$`` still pairs
+ on its own (a digit-rejected candidate could not open).
+ Stepping over the candidate would emit the unbalanced span
+ ``$fee $5$`` with a raw interior dollar.
+ """
+ segments = _split_math_segments("The $fee $5$ now")
+
+ assert segments == [
+ ("The $fee ", False),
+ ("$5$", True),
+ (" now", False),
+ ]
+
+
+class TestDisplayMathIsBlockScoped:
+ r"""A ``$$`` display span pairs only within its own block.
+
+ The display closer search used to run document-globally, so ONE
+ stray ``$$`` in an earlier paragraph captured the next genuine
+ display opener: the corrupted span swallowed the blank line
+ between the paragraphs (never valid inside math), the prose
+ between them rode raw, and the genuine pair was destroyed.
+ """
+
+ def test_stray_display_dollars_do_not_capture_a_later_span(self):
+ exporter = LaTeXExporter()
+ content = (
+ "# Title\n\n"
+ "Budget rising $$ at 100% & pace.\n\n"
+ "$$E = mc^2$$\n\n"
+ "More R&D prose.\n"
+ )
+ result = exporter.export_to_latex(content)
+
+ # The stray $$ is prose: escaped, and the % and & around it
+ # stay escaped text instead of riding raw inside a span that
+ # runs across the blank line.
+ assert "rising \\$\\$ at 100\\% \\& pace." in result
+ # The genuine display pair in the next block survives intact.
+ assert "$$E = mc^2$$" in result
+ assert "R\\&D prose" in result
+
+
+class TestHashAfterMathIsNotAHeading:
+ r"""Only a true line start can carry a heading marker.
+
+ A text segment that resumes mid-line after a math span begins with
+ whatever follows the closer; ``$x$ # y`` must escape that ``#``
+ rather than keep it raw as if it opened a heading.
+ """
+
+ def test_hash_after_inline_math_is_escaped(self):
+ result = LaTeXExporter().export_to_latex(
+ "# Title\n\nWe have $x$ # 5 and 100% more.\n"
+ )
+
+ assert "We have $x$ \\# 5 and 100\\% more." in result
+
+ def test_heading_with_math_is_escaped_after_the_span(self):
+ result = LaTeXExporter().export_to_latex(
+ "# Energy $E=mc^2$ # 50% & more\n\nBody.\n"
+ )
+
+ assert "\\section{Energy $E=mc^2$ \\# 50\\% \\& more}" in result
+
+
+class TestMathScannerPartitionsExactly:
+ """``_split_math_segments`` must be a lossless partition.
+
+ The scanner this PR introduces is the new input→output boundary for
+ every exported document; these Hypothesis properties pin that it
+ neither drops nor duplicates text, and that anything it calls math
+ is a well-formed span.
+ """
+
+ @settings(max_examples=100, deadline=None)
+ @given(_meta_text)
+ def test_segments_reconstruct_the_input(self, text):
+ segments = _split_math_segments(text)
+
+ assert "".join(segment for segment, _ in segments) == text
+
+ @settings(max_examples=100, deadline=None)
+ @given(_meta_text)
+ def test_math_segments_are_well_formed(self, text):
+ # Lone dollars legitimately live in TEXT segments (the escaper
+ # turns them into \\$ downstream — that is the parity fix); what
+ # must hold is that anything the scanner CALLS math is a
+ # well-formed, dollar-delimited span with a non-empty interior.
+ for segment, is_math in _split_math_segments(text):
+ if is_math:
+ assert len(segment) >= 3
+ assert segment.startswith("$") and segment.endswith("$")
diff --git a/tests/web/services/test_export_generation_contracts.py b/tests/web/services/test_export_generation_contracts.py
index 236e8849e1..91b7d247e9 100644
--- a/tests/web/services/test_export_generation_contracts.py
+++ b/tests/web/services/test_export_generation_contracts.py
@@ -229,26 +229,19 @@ class TestLaTeXBodyEscaping:
assert "\\input" not in body
assert "R\\&D" in body
- @pytest.mark.xfail(
- strict=True,
- reason=(
- "LIVE DEFECT: math mode is preserved by splitting the whole "
- "document on '$' and escaping only even-indexed parts, so ONE "
- "unpaired dollar -- a price in prose -- inverts the parity "
- "and everything after it emerges raw. This breaks compilation "
- "of ordinary benign reports. "
- ),
- )
def test_one_dollar_sign_does_not_switch_escaping_off_for_the_rest(self):
- """Math-mode preservation keys off ``$`` parity across the doc.
-
- ``export_to_latex`` splits the whole document on ``$`` and escapes
- only even-indexed parts. A single unpaired ``$`` -- a price, which
- LLM research prose produces constantly -- inverts the parity, so
- every character after it is treated as math mode and escaped not
- at all. That is a plain correctness break as much as a security
- one: an unescaped ``_`` outside math mode is a hard pdflatex
- error, so the exported document stops compiling.
+ """A lone ``$`` must not disable body escaping anywhere.
+
+ ``export_to_latex`` used to split the whole document on ``$``
+ and escape only even-indexed parts, so a single unpaired ``$``
+ -- a price, which LLM research prose produces constantly --
+ inverted the parity and every character after it was treated as
+ math mode and escaped not at all. That was a plain correctness
+ break as much as a security one: an unescaped ``_`` outside
+ math mode is a hard pdflatex error, so the exported document
+ stopped compiling. Math spans are now paired by a scanner that
+ treats an unpairable ``$`` as prose; this contract pins that
+ the escaping after such a dollar stays on.
"""
report = (
"Licences cost $5 per seat.\n\n"