Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

fix(chat): strip prior-turn citations from follow-up context (format-aware) (#4358)

LearningCircuit5c571d8eeeb7
diff --git a/changelog.d/+chat-followup-citation-stripping.bugfix.md b/changelog.d/+chat-followup-citation-stripping.bugfix.md
new file mode 100644
index 0000000000..691751d8c6
--- /dev/null
+++ b/changelog.d/+chat-followup-citation-stripping.bugfix.md
@@ -0,0 +1 @@
+**Cleaner follow-up context in Chat Mode** — Citation markers from earlier turns (e.g. `[1]`, `[1, 2]`, `[[1]](url)`, `[[arxiv.org-1]](url)`, `[[2024-01-15]](url)`, the plain source-tag form `[local-1]` / `[ai-2]` / `[personal-notes-3]`, `【1】`) are now stripped from the prior-conversation context fed to a follow-up. They reference the previous turn's source list, which the follow-up research renumbers against its own sources, so leaving them in could dangle or collide with the new answer's citations. A single-bracket link is stripped only when its label is the link's own host (`[arxiv.org](https://arxiv.org/abs/…)`); other links, such as `[1998](url)`, `[1](#step-1)`, `[config.yaml](url)` or `[the dataset](url)`, are kept, as are brackets inside a bare URL, code blocks and inline code. In `full` mode only the assistant's earlier answers are cleaned and your own messages are passed on verbatim; `raw` mode carries only the assistant's earlier research answers (cleaned) and never included your messages; in the default `summary` mode your messages reach the summarizer verbatim, but a citation-shaped string you typed is removed if the summary repeats it.
diff --git a/src/local_deep_research/chat/citation_stripping.py b/src/local_deep_research/chat/citation_stripping.py
new file mode 100644
index 0000000000..726df67905
--- /dev/null
+++ b/src/local_deep_research/chat/citation_stripping.py
@@ -0,0 +1,1849 @@
+"""Strip prior-turn citation markers from assistant-authored chat text.
+
+Follow-up research renumbers citations against its own freshly-found sources,
+so a marker carried over from a prior turn's answer dangles or collides. The
+recognized shapes are based on what ``CitationFormatter``
+(text_optimization/citation_formatter.py) writes into a synthesized answer:
+
+- ``number_hyperlinks``: ``[[1]](url)``; comma groups re-emit as
+ adjacent links with no separator
+- ``domain_hyperlinks``: ``[[arxiv.org]](url)``, ``[[slug]](url)``
+- ``domain_id_hyperlinks`` (+always): ``[[arxiv.org-1]](url)``
+- ``source_tagged_hyperlinks``: ``[[tag-N]](url)`` when the source URL is
+ http(s), PLAIN ``[local-1]`` /
+ ``[personal-notes-3]`` otherwise; comma
+ groups concatenate the two with no
+ separator (``[[web-1]](url)[local-2]``).
+ A collection slug can be one character
+ (``AI`` -> ``[ai-1]``, ``X`` ->
+ ``[x-1]``), so slug length proves
+ nothing; ``arr[i-1]`` survives through
+ the word-glue guard below instead
+- ``no_hyperlinks``: ``[1]``, ``[1, 2]``
+- lenticular (some models): ``【1】``, ``【1, 2】``
+
+Every hyperlinked citation the formatter emits uses DOUBLE brackets. One
+single-bracket shape is also stripped, for a model that re-renders a
+citation with one bracket pair: a domain label that names the link's OWN
+host as ``CitationFormatter`` would label it
+(``[arxiv.org](https://arxiv.org/x)``,
+``[arxiv.org-1](https://www.arxiv.org/x)``). Any other single-bracket
+link survives — numeric labels included, since those are years, ranks
+and anchors far more often than citations (``[1998](url)``,
+``[1](#step-1)``, ``[10](url)``) — as do ``[config.yaml](url)``,
+``[README.md](README.md)``, ``[v1.2](url)``,
+``[example.com](https://other.org/)`` and ``[dataset](url)``. Double-bracket
+labels match case-insensitively: ``CitationFormatter _extract_domain``
+preserves URL case, so an uppercase one-label authority
+(``[[LOCALHOST]](http://LOCALHOST/a)``, ``[[LOCALHOST-1]](...)``) strips
+too. Inside ``[[...]](url)`` an all-digit hyphenated label strips too
+(``[[2024-01-15]]``, ``[[1-2-3]]``, ``[[2023-2024-26]]``): the title and
+collection slugs produce them. A destination may hold Unicode spaces
+(only ASCII spaces end one, as in CommonMark), be up to 16384 characters
+long and nest balanced parentheses 16 deep.
+
+Bare numeric ranges such as ``[2020-2024]`` are preserved: without source
+provenance they cannot be distinguished from numeric collection tags.
+Hyperlinked numeric collection tags remain recognizable. A consequence:
+a bare numeric tag (``[2023-2024-27]``) stays, and so does a bare token
+glued right after it (``[9-4][5]``), which reads as an index chain.
+
+The scanner bounds citation lookups and advances over failed bracket runs.
+Text is segmented as:
+
+1. fenced code blocks — an unclosed fence ends with its containing
+ list/blockquote or document — and indented code blocks: emitted
+ verbatim (a fence glyph inside an open HTML code element below is
+ element content, never a fence). "Indented" is measured from the
+ content column of the containers opened on the same line — a list
+ marker's content sits one space past the marker, and tabs advance to
+ the next multiple-of-4 column — so ``- [1]`` and `` \t[1]`` are
+ code while ``- [1]`` is a paragraph continuation that strips. A
+ later line of a list item is measured from column 0, so a paragraph
+ indented four or more columns after a blank line is kept verbatim
+ as indented code even inside the item;
+2. HTML ``<pre>``/``<code>`` elements (nesting-aware, may span lines) and
+ link reference definitions (``[label]: dest``, optionally inside a
+ blockquote): emitted verbatim — the destination is validated per
+ CommonMark (``<...>`` or one whitespace-free token, optionally a
+ well-formed title), so a pseudo definition whose "destination" is
+ prose (``[Note]: See [1] for details.``) has its markers stripped.
+ The destination and the title may each sit on a following line at any
+ indentation (a lazy line included) unless that line opens a fence or
+ ``<pre>``;
+3. prose regions: CommonMark inline code spans (a run of N backticks
+ closes at the next run of exactly N backticks, CommonMark-style, and
+ may span lines within a paragraph — where only a bullet or an
+ ordered marker numbered 1 can interrupt the paragraph, and a ``>``
+ re-entry after a lazy continuation line is still that paragraph) and
+ complete
+ Markdown links/images — relative destinations and labels that merely
+ LOOK citation-shaped (``[see [1]](url)``, ``![alt\n[1]](pic.jpg)``)
+ included — are kept verbatim; the remaining prose is scanned left to
+ right for citation *runs*: a run is one or more adjacent citation
+ tokens (``[local-1][personal-notes-2]``, ``[[web-1]](url)[local-2]``,
+ ``[1][2]``), optionally separated by horizontal whitespace. The run is
+ removed together with ONE of its flanking horizontal-whitespace
+ separators: mid-line, the whitespace before it (one separator is kept
+ when a word or punctuation other than closing punctuation follows; a
+ glued run is replaced by one space only when a word follows it and
+ a word or word-ending punctuation precedes it — see ``_joins_words``
+ — so ``Claim.[[1]](url)Next`` becomes ``Claim. Next`` while
+ ``3.[[1]](url)5``, ``**[[1]](url)Bold**`` and ``/a/[1]/b`` stay
+ joined, and no space is added next to text in a script written
+ without spaces, such as CJK; parentheses left empty mid-line, as in
+ ``Claim ([[1]](url)).``, are removed with the space before them); at the
+ end of a line, the whitespace before it; at the start of a line, the
+ whitespace after it, so indentation before the run (a list item's
+ continuation paragraph) is kept verbatim and the following text is not
+ re-indented into a code block. No other whitespace is touched.
+
+A BARE bracket token (no URL) preceded by a word character, ``]`` or ``{``
+is an indexing expression, not a citation: ``results[1]``, ``arr[i-1]``,
+``x[1][2]`` and the LaTeX subscript ``x_{[i-1]}`` survive. Lenticular
+``【1】`` is never an index and strips even when glued (``2020【1】``). Letters of scripts written without spaces (CJK,
+kana, Thai, ...) are not word glue: ``首都です[1]`` strips. Hyperlinked
+tokens strip regardless of the preceding character — the formatter
+rewrites ``word[1]`` into ``word[[1]](url)`` when the source has a URL,
+and that must not leak. A preceding ``.`` is not a guard: footnote-style
+``holds.[local-1]`` strips.
+
+Two more positions keep a token: inside a bare URL (``https://x.org/a[1]``;
+the URL ends at whitespace, ``<``, ``>``, a quote, or a ``)`` with no
+matching ``(`` in it, such as the one closing a Markdown link
+destination), and a bare token followed by ``:`` that opens a line's
+content after only indentation, blockquote, list or ATX-heading markers
+(``[1]: Smith 2020``, ``- [1]: ...``, ``## [1]: ...``) — a bibliography
+entry. A full reference link ``[text][label]`` is kept only when the
+second label holds no unescaped bracket, so ``[Table 2][[1]](url)`` is
+text followed by a citation.
+"""
+
+import re
+from bisect import bisect_right
+from ipaddress import ip_address
+from urllib.parse import urlsplit
+
+# Bounds that keep every scan linear on adversarial input. Real labels and
+# URLs are far below them; past a bound the text simply is not a citation.
+_LABEL_MAX = 256 # longest label text scanned between brackets
+_LINK_LABEL_MAX = 1000 # longest link label scanned for verbatim links
+_URL_MAX = 16384 # longest URL scanned after "("
+_URL_MAX_DEPTH = 16 # deepest balanced-paren nesting accepted in a URL
+
+# Citation-shaped labels (validated with fullmatch on the bracketed text).
+_NUM_LIST_RE = re.compile(r"\d+(?:\s*,\s*\d+)*") # "1", "1, 2"
+_TOKEN_RE = re.compile(
+ r"[A-Za-z0-9][A-Za-z0-9._-]*"
+) # "arxiv.org", "LOCALHOST"
+# Plain tag form ``<slug>-N``. ``_slugify_collection`` can emit a slug of
+# any length ("AI" -> "ai", "X" -> "x"), so no length floor applies;
+# ``arr[i-1]`` indexing is protected by the word-glue guard instead.
+_TAG_ID_RE = re.compile(r"[a-z0-9][a-z0-9._-]*-\d+") # "x-1", "local-1"
+# All-digit hyphenated label, accepted only inside ``[[...]](url)``:
+# ``_slugify_title``/``_slugify_collection`` turn "2024-01-15", "1.2.3" and
+# "2023/2024" into "2024-01-15", "1-2-3" and "2023-2024", and the id modes
+# append "-N". A BARE ``[2020-2024]`` stays prose (see module docstring).
+_NUMERIC_TAG_ID_RE = re.compile(r"\d+(?:-\d+)+")
+_AUTHORITY_TOKEN_RE = re.compile(
+ r"[A-Za-z0-9](?:[A-Za-z0-9._-]*[A-Za-z0-9])?(?::\d{1,5})?"
+)
+
+# Prose scanning: only these characters can begin (or precede) a citation.
+_INTERESTING_RE = re.compile(r"[\[\u3010 \t]")
+# URL scanning: jump between the only characters that affect paren balance
+# or terminate a URL. CommonMark ends a destination at an ASCII space or
+# control character only; ``_safe_link_destination`` leaves Unicode spaces
+# (U+00A0, U+3000) in place, so they are URL characters here.
+_URL_EVENT_RE = re.compile(r"[()\x00-\x20\x7f]")
+# A bare (autolinked) URL: it runs to whitespace, ``<``, ``>`` or a quote,
+# and a ``)`` with no matching ``(`` inside it ends it (the ``)`` closing
+# a Markdown link destination). Each match consumes its characters, so
+# one finditer over a chunk is linear.
+_BARE_URL_RE = re.compile(
+ r"(?<![A-Za-z0-9+.-])https?://[^\s<>\"']*", re.IGNORECASE
+)
+_PAREN_RE = re.compile(r"[()]")
+# What may precede a ``[1]:`` bibliography-style entry on its own line:
+# indentation, blockquote markers, a list marker or an ATX heading opener.
+# Each marker must be followed by whitespace (``>`` excepted), so a run
+# like ``####`` has exactly one parse and the match stays linear.
+_LINE_LEAD_RE = re.compile(
+ r"[ \t]*(?:(?:>|(?:[-+*]|\d{1,9}[.)]|#{1,6})(?=[ \t]))[ \t]*)*"
+)
+_LINE_LEAD_MAX = 64 # longest such prefix considered
+_CLOSING_TAG_RE = re.compile(r"</[A-Za-z][A-Za-z0-9-]*[ \t]*>")
+_CLOSING_TAG_MAX = 64 # longest closing tag looked back over
+# Everything up to the next unescaped ``[`` or ``]`` (captured), reading
+# backslash escapes pairwise; an unrolled loop, so a failed match is linear.
+_UNESCAPED_BRACKET_RE = re.compile(r"[^\[\]\\]*(?:\\[\s\S][^\[\]\\]*)*([\[\]])")
+_HTML_CODE_TAG_RE = re.compile(r"</?(?:code|pre)(?:\s[^>]*)?>", re.IGNORECASE)
+_HTML_COMMENT_RE = re.compile(r"<!--.*?(?:-->|$)", re.DOTALL)
+# A code span / link label may cross newlines but never a blank line.
+_BLANK_LINE_RE = re.compile(r"\r?\n[ \t]*\r?\n")
+# Link reference definition, optionally inside blockquotes or a list item.
+_REFERENCE_DEF_RE = re.compile(
+ r"^[ \t]{0,3}(?P<quotes>(?:>[ \t]?)*)(?P<list>"
+ r"(?:[-+*]|\d{1,9}[.)])[ \t]+)?\[(?P<label>[^\]]*)\]:(?P<destination>.*)$"
+)
+_CONTAINER_PREFIX_RE = re.compile(
+ r"^(?P<indent>[ \t]{0,3})(?P<quotes>(?:>[ \t]?)*)(?P<body>.*)$"
+)
+_REFERENCE_TITLE_RE = re.compile(
+ r'(?:(?:"(?:[^"\\]|\\.)*")|(?:\'(?:[^\'\\]|\\.)*\')|(?:\([^)]*\)))'
+)
+_ATX_HEADING_RE = re.compile(r"^[ ]{0,3}#{1,6}(?:[ \t]+|$)")
+_THEMATIC_BREAK_RE = re.compile(
+ r"^[ ]{0,3}(?:(?:\*[ \t]*){3,}|(?:_[ \t]*){3,}|(?:-[ \t]*){3,})$"
+)
+_HTML_BLOCK_TAG_RE = re.compile(
+ r"^[ ]{0,3}</?(?:address|article|aside|base|basefont|blockquote|body|caption|"
+ r"center|col|colgroup|dd|details|dialog|dir|div|dl|dt|fieldset|figcaption|"
+ r"figure|footer|form|frame|frameset|h[1-6]|head|header|hr|html|iframe|legend|"
+ r"li|link|main|menu|menuitem|nav|noframes|ol|optgroup|option|p|param|search|"
+ r"section|summary|table|tbody|td|tfoot|th|thead|title|tr|track|ul)(?:[ \t/>]|$)",
+ re.IGNORECASE,
+)
+
+
+def _has_letter(label: str) -> bool:
+ return any("a" <= ch.lower() <= "z" for ch in label)
+
+
+def _is_num_list(label: str) -> bool:
+ return _NUM_LIST_RE.fullmatch(label) is not None
+
+
+def _is_letter_token(label: str) -> bool:
+ """``True`` for slug/domain/tag labels like ``arxiv.org``/``web-1``."""
+ return _TOKEN_RE.fullmatch(label) is not None and _has_letter(label)
+
+
+def _is_ip_token(label: str) -> bool:
+ try:
+ ip_address(label)
+ except ValueError:
+ host, separator, port = label.rpartition(":")
+ if not separator or not port.isdigit():
+ return False
+ try:
+ ip_address(host)
+ except ValueError:
+ return False
+ return True
+
+
+def _is_authority_token(label: str) -> bool:
+ return (
+ (":" in label or "." in label)
+ and _AUTHORITY_TOKEN_RE.fullmatch(label) is not None
+ and (_has_letter(label) or _is_ip_token(label))
+ )
+
+
+def _is_formatter_ipv4_label(label: str, url: str) -> bool:
+ try:
+ parsed = urlsplit(url)
+ host = parsed.hostname
+ if host is None or ip_address(host).version != 4:
+ return False
+ except ValueError:
+ return False
+ return label == ".".join(parsed.netloc.split(".")[-2:])
+
+
+def _is_authority_citation(label: str, url: str) -> bool:
+ try:
+ authority = urlsplit(url).netloc
+ except ValueError:
+ return False
+ if not authority:
+ return False
+ if authority.startswith("www."):
+ authority = authority[4:]
+ display = ".".join(authority.split(".")[-2:])
+ base, separator, suffix = label.rpartition("-")
+ return label == display or bool(
+ separator and suffix.isdigit() and base == display
+ )
+
+
+def _is_dotted_token(label: str) -> bool:
+ """``True`` for domain labels — the only single-bracket link shape."""
+ return _is_letter_token(label) and "." in label
+
+
+def _is_tag_id(label: str) -> bool:
+ """``True`` for the plain source-tag form ``<slug>-N``."""
+ return _TAG_ID_RE.fullmatch(label) is not None and _has_letter(label)
+
+
+def _parse_url(text: str, pos: int) -> int | None:
+ """Parse a balanced-parenthesis URL whose ``(`` is at ``text[pos]``.
+
+ Returns the index just past the closing ``()``, or ``None`` when the
+ parens never balance within ``_URL_MAX`` characters and ``_URL_MAX_DEPTH``
+ nesting levels, or the candidate contains an ASCII space or control
+ character. Jumps between ``(`` / ``)`` / space events so long URL-free stretches cost one
+ C-level scan, never a per-character Python loop.
+ """
+ limit = pos + _URL_MAX
+ depth = 0
+ i = pos
+ while True:
+ m = _URL_EVENT_RE.search(text, i, limit + 1)
+ if m is None or m.start() > limit:
+ return None
+ ch = m.group()
+ if ch == "(":
+ depth += 1
+ if depth > _URL_MAX_DEPTH:
+ return None
+ elif ch == ")":
+ depth -= 1
+ if depth == 0:
+ return m.end()
+ else: # whitespace inside a URL
+ return None
+ i = m.end()
+
+
+def _parse_inline_destination(text: str, pos: int) -> int | None:
+ """Parse a Markdown inline destination and optional title."""
+ limit = min(len(text), pos + _URL_MAX)
+ i = pos + 1
+ if text[i : i + 1] == "<":
+ destination_end = text.find(">", i + 1, limit)
+ if destination_end == -1 or "\n" in text[i:destination_end]:
+ return None
+ i = destination_end + 1
+ else:
+ depth = 0
+ while i < limit:
+ ch = text[i]
+ if ch == "\\" and i + 1 < limit:
+ i += 2
+ continue
+ if ch == "(":
+ depth += 1
+ if depth > _URL_MAX_DEPTH:
+ return None
+ elif ch == ")":
+ if depth == 0:
+ return i + 1
+ depth -= 1
+ elif ch <= " " or ch == "\x7f": # CommonMark: ASCII only
+ break
+ i += 1
+ else:
+ return None
+
+ if i < limit and text[i] == ")":
+ return i + 1
+
+ while i < limit and text[i].isspace():
+ i += 1
+ title = _REFERENCE_TITLE_RE.match(text, i, limit)
+ if title is None:
+ return None
+ i = title.end()
+ while i < limit and text[i].isspace():
+ i += 1
+ return i + 1 if i < limit and text[i] == ")" else None
+
+
+def _match_citation_token(text: str, j: int) -> tuple[int | None, bool, int]:
+ """Match ONE citation token starting at ``text[j]`` (a ``[`` or ``【``).
+
+ Returns ``(end_index_or_None, is_hyperlink, opener_run_end)``. A
+ non-``None`` end means ``text[j:end]`` is a citation-shaped token;
+ ``opener_run_end`` is always the index past the opening bracket run, so
+ a FAILED parse still advances the caller past the whole run — the
+ formatter never emits a citation starting mid-bracket-run, and that rule
+ is what keeps unclosed-bracket floods linear.
+ """
+ n = len(text)
+ if text[j] == "\u3010": # lenticular 【1】 / 【1, 2】 — plain numbers only
+ for closer in ("\u3011", "]"):
+ close = text.find(closer, j + 1, j + 1 + _LABEL_MAX)
+ if close != -1 and _NUM_LIST_RE.fullmatch(text, j + 1, close):
+ return (close + 1, False, j + 1)
+ return (None, False, j + 1)
+ k = j
+ while k < n and text[k] == "[":
+ k += 1
+ # Outer double brackets contain an IPv6 label's own brackets.
+ if k - j >= 2:
+ outer_close = text.find("]](", j + 2, j + 2 + _LABEL_MAX)
+ if outer_close != -1:
+ url_end = _parse_url(text, outer_close + 2)
+ candidate = text[j + 2 : outer_close]
+ if url_end is not None and _is_authority_citation(
+ candidate, text[outer_close + 3 : url_end - 1]
+ ):
+ return (url_end, True, k)
+ close = text.find("]", k, k + _LABEL_MAX)
+ if close == -1:
+ return (None, False, k)
+ q = close
+ while q < n and text[q] == "]":
+ q += 1
+ label = text[k:close]
+ double = k - j >= 2 and q - close >= 2
+ if (
+ k - j >= 3
+ and _is_ip_token(label)
+ and text[close + 1 : close + 2] == ":"
+ ):
+ port_end = close + 2
+ while port_end < n and text[port_end].isdigit():
+ port_end += 1
+ q = port_end
+ while q < n and text[q] == "]":
+ q += 1
+ double = port_end > close + 2 and q - port_end >= 2
+ if q < n and text[q] == "(":
+ url_end = _parse_url(text, q)
+ if url_end is not None:
+ if double:
+ ok = (
+ _is_num_list(label)
+ or _is_tag_id(label)
+ or _NUMERIC_TAG_ID_RE.fullmatch(label) is not None
+ or _is_letter_token(label)
+ or _is_authority_token(label)
+ or _is_ip_token(label)
+ or _is_formatter_ipv4_label(
+ label, text[q + 1 : url_end - 1]
+ )
+ )
+ else:
+ # The formatter only emits double-bracket links; a single
+ # pair is a citation only when it is a domain label naming
+ # the link's own host. A dotted label alone is ordinary
+ # link text (``[config.yaml](url)``), and a numeric one is
+ # a year, rank or anchor (``[1998](url)``, ``[1](#s1)``).
+ ok = _is_dotted_token(label) and _is_authority_citation(
+ label, text[q + 1 : url_end - 1]
+ )
+ return (url_end, True, k) if ok else (None, False, k)
+ if _parse_inline_destination(text, q) is not None:
+ return (None, False, k)
+ # A malformed destination may still follow a bare citation.
+ if double:
+ return (None, False, k) # the formatter never emits a bare [[label]]
+ if _is_num_list(label) or _is_tag_id(label):
+ return (q, False, k)
+ return (None, False, k)
+
+
+def _reference_label(label: str) -> str:
+ return " ".join(label.split()).casefold()
+
+
+def _is_reference_link(
+ text: str, start: int, token_end: int, references: frozenset[str]
+) -> bool:
+ own_label = _reference_label(text[start + 1 : token_end - 1])
+ if text[token_end : token_end + 1] != "[":
+ return (
+ own_label in references and text[token_end : token_end + 1] != "("
+ )
+ close = text.find("]", token_end + 1, token_end + _LINK_LABEL_MAX)
+ if close == -1:
+ return False
+ label = text[token_end + 1 : close] or text[start + 1 : token_end - 1]
+ return _reference_label(label) in references
+
+
+def _match_citation_run(
+ text: str, first_end: int, references: frozenset[str]
+) -> int:
+ """Extend a matched token across adjacent citation tokens.
+
+ ``_replace_comma_citations`` joins formatted parts with no separator,
+ so tokens arrive glued (``[local-1][personal-notes-2]``,
+ ``[[web-1]](url)[local-2]``) or horizontally spaced. Continuation
+ tokens skip the word-glue guard: they follow a citation, not an
+ identifier. Horizontal whitespace is consumed only when another token
+ actually follows it.
+ """
+ n = len(text)
+ end = first_end
+ while end < n:
+ if text[end] in "[\u3010":
+ tok = _match_citation_token(text, end)
+ if tok[0] is None or _is_reference_link(
+ text, end, tok[0], references
+ ):
+ return end
+ end = tok[0]
+ continue
+ k = end
+ while k < n and text[k] in " \t":
+ k += 1
+ if k == end or k >= n or text[k] not in "[\u3010":
+ return end
+ tok = _match_citation_token(text, k)
+ if tok[0] is None or _is_reference_link(text, k, tok[0], references):
+ return end
+ end = tok[0]
+ return end
+
+
+def _bare_url_spans(text: str) -> tuple[list[int], list[int]]:
+ """Return ``(starts, ends)`` of the bare URLs in ``text``, in order."""
+ starts: list[int] = []
+ ends: list[int] = []
+ for match in _BARE_URL_RE.finditer(text):
+ start, end = match.span()
+ depth = 0
+ for paren in _PAREN_RE.finditer(text, start, end):
+ if paren.group() == "(":
+ depth += 1
+ elif depth:
+ depth -= 1
+ else:
+ end = paren.start()
+ break
+ starts.append(start)
+ ends.append(end)
+ return (starts, ends)
+
+
+def _inside_bare_url(spans: tuple[list[int], list[int]], j: int) -> bool:
+ starts, ends = spans
+ idx = bisect_right(starts, j) - 1
+ return idx >= 0 and j < ends[idx]
+
+
+def _is_word_glue(ch: str) -> bool:
+ """Whether ``ch`` glues a following ``[...]`` into an index expression.
+
+ Letters and digits of space-separated scripts do (``results[1]``).
+ Scripts written without spaces (CJK, kana, Thai, ...) do not: a
+ citation sits directly against their text (``首都です[1]``).
+ """
+ return (ch.isalnum() or ch == "_") and not _is_unspaced(ch)
+
+
+def _is_unspaced(ch: str) -> bool:
+ """``True`` for a character of a script written without word spaces,
+ including its punctuation (``。``, ``、``, fullwidth forms)."""
+ code = ord(ch) if len(ch) == 1 else 0
+ return any(low <= code <= high for low, high in _UNSPACED_RANGES)
+
+
+def _line_lead(text: str, j: int, left: str) -> bool:
+ """Whether only container/heading markers precede ``text[j]`` on its
+ line. ``left`` is the character before ``text[0]`` (``""`` at the
+ start of a prose block, which starts a line)."""
+ line_start = max(text.rfind("\n", 0, j), text.rfind("\r", 0, j)) + 1
+ if line_start == 0 and left not in _LINE_START:
+ return False
+ if j - line_start > _LINE_LEAD_MAX:
+ return False
+ return _LINE_LEAD_RE.fullmatch(text, line_start, j) is not None
+
+
+def _try_strip_run(
+ text: str,
+ j: int,
+ guard_prev: bool,
+ references: frozenset[str],
+ url_spans: tuple[list[int], list[int]],
+ prev_char: str = "",
+ left: str = "",
+) -> tuple[int | None, int]:
+ """Try to strip one citation run whose first token starts at ``j``.
+
+ Returns ``(strip_end_or_None, skip_to)``. ``guard_prev`` applies the
+ bare-token word-glue guard (see module docstring); it is skipped when
+ the run starts after horizontal whitespace, where a real citation may
+ legally sit. ``prev_char`` supplies the source character immediately
+ before ``text[0]`` when ``j`` is 0, so a chunk starting right after a
+ protected span still honours the guard. ``left`` is the character
+ before ``text[0]`` (``""`` when ``text`` starts a line). ``url_spans``
+ is ``_bare_url_spans(text)``. On any failure ``skip_to`` is the
+ opener-run end, so the caller never retries mid-run.
+ """
+ tok = _match_citation_token(text, j)
+ if tok[0] is None:
+ return (None, tok[2])
+ end, is_hyperlink = tok[0], tok[1]
+ if _is_escaped(text, j) or _is_reference_link(text, j, end, references):
+ return (None, tok[2])
+ if j > 0 and text[j - 1] == "!":
+ return (None, tok[2])
+ if (
+ not is_hyperlink
+ and text[end : end + 1] == ":"
+ and _line_lead(text, j, left)
+ ):
+ # ``[1]: Smith 2020`` opening a line is a bibliography entry.
+ return (None, tok[2])
+ if _inside_bare_url(url_spans, j):
+ return (None, tok[2])
+ if (
+ guard_prev
+ and not is_hyperlink
+ and text[j] != "\u3010" # nobody indexes with ``x【1】``
+ and (j > 0 or prev_char)
+ ):
+ prev = text[j - 1] if j > 0 else prev_char
+ # ``{`` covers LaTeX subscripts (``x_{[i-1]}``).
+ if _is_word_glue(prev) or prev in "]{":
+ return (None, tok[2])
+ return (_match_citation_run(text, end, references), tok[2])
+
+
+def _match_verbatim_link(text: str, j: int) -> int | None:
+ """Match a complete Markdown link/image whose ``[`` is at ``text[j]``.
+
+ Returns the index past an inline or reference-style destination, or past
+ the label's ``]`` for a destination-less image alt, or ``None``. The caller
+ keeps a matched structure verbatim: its label
+ may contain brackets that look citation-shaped (``[see [1]](url)``)
+ and may span lines, but not a blank line. Scanning is bounded by
+ ``_LINK_LABEL_MAX`` and jumps between brackets with C-level finds, so
+ adversarial input stays linear.
+ """
+ n = len(text)
+ limit = min(n, j + _LINK_LABEL_MAX)
+ is_image = j > 0 and text[j - 1] == "!"
+ if (
+ not is_image
+ and text.find("](", j + 1, limit + 1) == -1
+ and text.find("][", j + 1, limit + 1) == -1
+ ):
+ return None
+ depth = 1
+ i = j + 1
+ while i < limit:
+ # ``i`` always sits just past a bracket, so backslash pairing
+ # from ``i`` is aligned; escaped brackets are skipped at C level
+ # instead of being re-found and re-checked once per nesting step.
+ bracket = _UNESCAPED_BRACKET_RE.match(text, i, limit)
+ if bracket is None:
+ return None
+ close = bracket.start(1)
+ if text[close] == "[":
+ depth += 1
+ i = close + 1
+ continue
+ depth -= 1
+ if depth == 0:
+ if _BLANK_LINE_RE.search(text[j : close + 1]) is not None:
+ return None
+ suffix = close + 1
+ if text[suffix : suffix + 1] == "(":
+ dest_end = _parse_inline_destination(text, suffix)
+ if dest_end is not None:
+ return dest_end
+ if text[suffix : suffix + 1] == "[":
+ # A reference label holds no unescaped bracket, so
+ # ``[Table 2][[1]](url)`` is text followed by a citation.
+ reference = _UNESCAPED_BRACKET_RE.match(text, suffix + 1, limit)
+ if reference is not None and text[reference.start(1)] == "]":
+ return reference.start(1) + 1
+ return close + 1 if is_image else None
+ i = close + 1
+ return None
+
+
+def _is_escaped(text: str, pos: int) -> bool:
+ backslashes = 0
+ pos -= 1
+ while pos >= 0 and text[pos] == "\\":
+ backslashes += 1
+ pos -= 1
+ return backslashes % 2 == 1
+
+
+def _html_tag_spans(text: str) -> list[tuple[int, int]]:
+ """Return spans of HTML tags while respecting quoted attribute values."""
+ spans: list[tuple[int, int]] = []
+ pos = 0
+ while True:
+ start = text.find("<", pos)
+ if start == -1 or start + 1 == len(text):
+ return spans
+ marker = text[start + 1]
+ name_pos = start + 2 if marker == "/" else start + 1
+ if name_pos >= len(text) or not text[name_pos].isalpha():
+ pos = start + 1
+ continue
+ quote = ""
+ i = name_pos + 1
+ while i < len(text):
+ ch = text[i]
+ if quote:
+ if ch == quote:
+ quote = ""
+ elif ch in "\"'":
+ quote = ch
+ elif ch == ">":
+ spans.append((start, i + 1))
+ pos = i + 1
+ break
+ elif ch == "<":
+ pos = i
+ break
+ i += 1
+ else:
+ return spans
+
+
+# Characters after a stripped run that need no separator before them.
+_NO_SEPARATOR_BEFORE = " \t\r\n.,;:!?)]}"
+# Scripts written without word spaces, with their punctuation: Thai and
+# Lao, Tibetan, Myanmar, Khmer, CJK symbols/punctuation, kana, CJK
+# ideographs (with extensions and compatibility forms), Yi, fullwidth and
+# halfwidth forms.
+_UNSPACED_RANGES = (
+ (0x0E00, 0x0EFF),
+ (0x0F00, 0x0FFF),
+ (0x1000, 0x109F),
+ (0x1780, 0x17FF),
+ (0x2E80, 0x2FDF),
+ (0x3000, 0x31FF),
+ (0x3400, 0x4DBF),
+ (0x4E00, 0x9FFF),
+ (0xA000, 0xA4CF),
+ (0xF900, 0xFAFF),
+ (0xFE30, 0xFE4F),
+ (0xFF00, 0xFFEF),
+ (0x20000, 0x3134F),
+)
+
+
+def _ends_closing_tag(text: str, j: int) -> bool:
+ """Whether ``text[:j]`` ends with a closing HTML tag (``</span>``)."""
+ if j == 0 or text[j - 1] != ">":
+ return False
+ start = text.rfind("<", max(0, j - _CLOSING_TAG_MAX), j - 1)
+ return start != -1 and _CLOSING_TAG_RE.fullmatch(text, start, j) is not None
+
+
+def _joins_words(
+ text: str, j: int, end: int, left: str, tag_end: int = -1
+) -> bool:
+ """Whether removing ``text[j:end]`` would join two words, so a space
+ must replace it (``word[[1]](url)next``, ``Claim.[[1]](url)Next``).
+
+ Only a word may follow (not ``_``, and not a script written without
+ spaces). Before the run there must be a word character, or
+ punctuation that ends a word: ``)``, ``]``, ``}``, ``>``, ``!``, ``?``,
+ ``;``, ``:``; ``.`` or ``,`` unless it sits between digits
+ (``3.[[1]](u)5``); ``*``, ``"`` or ``'`` only when it CLOSES, i.e. its
+ run follows a non-space character (``**bold**[[1]](u)Next`` but not
+ ``**[[1]](u)Smith**``). Anything else — ``/page/[1]/x``, an opening
+ bracket — stays joined. ``left`` is the character before ``text[0]``.
+
+ A following ``[`` (a link label) or ``<`` (an inline tag) renders as
+ the start of the next word, so it counts as one when a word character
+ or a closing tag precedes the run: ``word[[1]](u)[dataset](v)`` and
+ ``</span>[[1]](u)<span>`` keep a space. ``tag_end`` is a position
+ of ``text`` known to follow a closing tag outside ``text``.
+ """
+ nxt = text[end]
+ prev = text[j - 1] if j > 0 else left
+ if nxt in "[<":
+ return _is_word_glue(prev) or j == tag_end or _ends_closing_tag(text, j)
+ if nxt == "_" or not _is_word_glue(nxt):
+ return False
+ if _is_word_glue(prev) or prev in ")]}>!?;:":
+ return True
+ if prev in ".,":
+ before = text[j - 2] if j > 1 else (left if j == 1 else "")
+ return not (before.isdigit() and nxt.isdigit())
+ if prev in "*\"'":
+ if j == 0:
+ return False # the run's start lies in another chunk
+ k = j - 1
+ while k > 0 and j - k < 3 and text[k - 1] == prev:
+ k -= 1
+ before = text[k - 1] if k > 0 else left
+ return bool(before) and not before.isspace() and before != prev
+ return False
+
+
+# ``prev`` values meaning "nothing precedes this position on its line"
+# (``""`` is the start of a prose block, which always starts a line).
+_LINE_START = ("", "\n", "\r")
+
+
+def _skip_horizontal(text: str, pos: int) -> int:
+ while pos < len(text) and text[pos] in " \t":
+ pos += 1
+ return pos
+
+
+def _drop_open_paren(out: list[str]) -> None:
+ """Remove the ``(`` that ends ``out`` and the one space before it. The
+ caller has checked that both lie in the current chunk, mid-line."""
+ for _ in range(2):
+ while out and not out[-1]:
+ out.pop()
+ if not out:
+ return
+ out[-1] = out[-1][:-1]
+
+
+def _strip_citations_from_prose(
+ text: str, references: frozenset[str], left: str = "", tag_end: int = -1
+) -> str:
+ """Strip citation runs from one code-span-free prose string.
+
+ ``left`` is the single source character preceding ``text[0]`` (when
+ the chunk starts right after a protected span); it supplies the
+ word-glue guard and separator decision for a run at position 0.
+ ``tag_end`` is the position in ``text`` right after a closing HTML
+ tag that lies outside ``text`` (``-1`` when none).
+ """
+ out: list[str] = []
+ pos = 0
+ n = len(text)
+ url_spans = _bare_url_spans(text)
+ while True:
+ m = _INTERESTING_RE.search(text, pos)
+ if m is None:
+ out.append(text[pos:])
+ return "".join(out)
+ j = m.start()
+ out.append(text[pos:j])
+ if text[j] in "[\u3010":
+ prev = text[j - 1] if j > 0 else left
+ end, skip_to = _try_strip_run(
+ text,
+ j,
+ guard_prev=True,
+ references=references,
+ url_spans=url_spans,
+ prev_char=prev,
+ left=left,
+ )
+ if end is not None:
+ if prev in _LINE_START:
+ # Nothing precedes the run on its line: its separator
+ # is the whitespace AFTER it, which would otherwise
+ # indent the following text.
+ end = _skip_horizontal(text, end)
+ elif (
+ prev == "("
+ and text[end : end + 1] == ")"
+ and j >= 3
+ and text[j - 2] in " \t"
+ and text[j - 3] not in " \t\r\n"
+ and not (end + 1 < n and _is_word_glue(text[end + 1]))
+ ):
+ # ``Claim ([[1]](u)).``: the parentheses held only the
+ # citation; drop them with the space before them.
+ _drop_open_paren(out)
+ end += 1
+ elif end < n and _joins_words(text, j, end, left, tag_end):
+ out.append(" ")
+ pos = end
+ continue
+ link_end = _match_verbatim_link(text, j)
+ if link_end is not None:
+ out.append(text[j:link_end])
+ pos = link_end
+ continue
+ out.append(text[j:skip_to])
+ pos = skip_to
+ else: # horizontal whitespace: a run may start after it
+ k = m.end()
+ while k < n and text[k] in " \t":
+ k += 1
+ if k < n and text[k] in "[\u3010":
+ end, skip_to = _try_strip_run(
+ text,
+ k,
+ guard_prev=False,
+ references=references,
+ url_spans=url_spans,
+ left=left,
+ )
+ if end is not None:
+ after = _skip_horizontal(text, end)
+ if after == n or text[after] in "\r\n":
+ # The run ends its line: drop the whitespace
+ # before it (no trailing whitespace is created);
+ # whitespace after it is kept verbatim.
+ pass
+ elif (text[j - 1] if j > 0 else left) in _LINE_START:
+ # Indentation before a line-leading run is kept
+ # verbatim; the run's separator is the whitespace
+ # after it.
+ out.append(text[j:k])
+ end = after
+ elif text[end] not in _NO_SEPARATOR_BEFORE and not (
+ _is_unspaced(text[end])
+ ):
+ out.append(text[j:k])
+ pos = end
+ continue
+ link_end = _match_verbatim_link(text, k)
+ if link_end is not None:
+ out.append(text[j:link_end])
+ pos = link_end
+ continue
+ out.append(text[j:skip_to])
+ pos = skip_to
+ else:
+ out.append(text[j:k])
+ pos = k
+
+
+def _backtick_runs(text: str) -> list[tuple[int, int, int]]:
+ """Backtick spans grouped by their enclosing paragraph or heading."""
+ runs: list[tuple[int, int, int]] = []
+ offset = 0
+ paragraph = 0
+ paragraph_depth = 0
+ previous_blank = True
+ previous_hard = False
+ for line in text.splitlines(keepends=True):
+ body, depth = _container_body(line)
+ blank = not body.strip()
+ hard = _ends_markdown_block(body)
+ # A setext heading underline only underlines a paragraph line,
+ # but it ends that paragraph — a code span cannot pair across it.
+ hard = hard or (
+ not previous_blank
+ and not previous_hard
+ and _SETEXT_UNDERLINE_RE.fullmatch(body.rstrip("\r\n")) is not None
+ )
+ # CommonMark: a list marker interrupts a paragraph only when it is
+ # a bullet or an ordered marker numbered 1 — "2. text" is a lazy
+ # continuation line, so it must not split an open code-span
+ # paragraph (leaving stray unpaired backticks behind).
+ marker = _LIST_MARKER_RE.match(body)
+ starts_list = marker is not None and (
+ marker.group("bullet") is not None or marker.group("ordered") == "1"
+ )
+ # ``paragraph_depth`` is the container depth the open paragraph
+ # lives at; a lazy continuation line (a paragraph line without
+ # its ``>`` prefix) keeps that paragraph — and its container —
+ # open, so a ``>`` re-entry at the paragraph's own depth is
+ # paragraph continuation text, not a new block. Only a DEEPER
+ # container interrupts the paragraph (a quote start may interrupt
+ # one); a shallower line is itself a lazy continuation.
+ if (
+ previous_blank
+ or blank
+ or hard
+ or previous_hard
+ or starts_list
+ or depth > paragraph_depth
+ ):
+ paragraph += 1
+ paragraph_depth = depth
+ for match in re.finditer(r"`+", line):
+ runs.append(
+ (offset + match.start(), offset + match.end(), paragraph)
+ )
+ offset += len(line)
+ previous_blank = blank
+ previous_hard = hard
+ return runs
+
+
+def _inline_spans(
+ text: str,
+) -> tuple[list[tuple[int, int]], list[tuple[int, int]]]:
+ """Recognize code and comments in order; each protects its own contents.
+
+ A backtick or ``<!--`` inside an HTML tag is attribute text, not an
+ opener: ``_html_tag_spans`` is quote-aware, so every position it
+ covers is skipped before it can start a phantom span reaching past
+ the tag and shielding later prose.
+ """
+ runs = _backtick_runs(text)
+ tags = _html_tag_spans(text)
+ tag_index = 0
+ outside_tags: list[tuple[int, int, int]] = []
+ for run in runs:
+ while tag_index < len(tags) and tags[tag_index][1] <= run[0]:
+ tag_index += 1
+ if tag_index < len(tags) and tags[tag_index][0] <= run[0]:
+ continue
+ outside_tags.append(run)
+ runs = outside_tags
+ next_same: list[int | None] = [None] * len(runs)
+ next_shorter: list[int | None] = [None] * len(runs)
+ next_by_key: dict[tuple[int, int], int] = {}
+ for idx in range(len(runs) - 1, -1, -1):
+ start, end, paragraph = runs[idx]
+ next_same[idx] = next_by_key.get((paragraph, end - start))
+ next_shorter[idx] = next_by_key.get((paragraph, end - start - 1))
+ next_by_key[(paragraph, end - start)] = idx
+ codes: list[tuple[int, int]] = []
+ comments: list[tuple[int, int]] = []
+ comment_index = 0
+
+ def next_comment(from_pos: int) -> int:
+ nonlocal comment_index
+ candidate = text.find("<!--", from_pos)
+ while candidate != -1:
+ if _is_escaped(text, candidate):
+ candidate = text.find("<!--", candidate + 4)
+ continue
+ while (
+ comment_index < len(tags)
+ and tags[comment_index][1] <= candidate
+ ):
+ comment_index += 1
+ if (
+ comment_index < len(tags)
+ and tags[comment_index][0] <= candidate
+ ):
+ candidate = text.find("<!--", tags[comment_index][1])
+ continue
+ break
+ return candidate
+
+ pos = 0
+ i = 0
+ comment = next_comment(0)
+ while pos < len(text):
+ while i < len(runs) and runs[i][0] < pos:
+ i += 1
+ if comment != -1 and comment < pos:
+ comment = next_comment(pos)
+ tick = runs[i][0] if i < len(runs) else len(text)
+ if comment != -1 and comment < tick:
+ close = text.find("-->", comment + 4)
+ pos = len(text) if close == -1 else close + 3
+ comments.append((comment, pos))
+ continue
+ if i == len(runs):
+ break
+ start, end, _ = runs[i]
+ if _is_escaped(text, start):
+ start += 1
+ close_idx = next_shorter[i] if start < end else None
+ else:
+ close_idx = next_same[i]
+ if close_idx is not None:
+ pos = runs[close_idx][1]
+ codes.append((start, pos))
+ i = close_idx + 1
+ else:
+ pos = end
+ i += 1
+ return (codes, comments)
+
+
+def _code_spans(text: str) -> list[tuple[int, int]]:
+ return _inline_spans(text)[0]
+
+
+def _strip_code_and_citations(
+ text: str,
+ references: frozenset[str],
+ left: str = "",
+ left_tag: bool = False,
+) -> str:
+ """Strip citations from prose, keeping inline code and comments verbatim.
+
+ ``left`` is the source character immediately before ``text[0]``; it is
+ threaded into every chunk that starts at ``pos`` so a citation run
+ directly following a protected span keeps one separator instead of
+ joining the neighbouring words. ``left_tag`` says ``text`` starts right
+ after a closing HTML tag outside it.
+ """
+
+ def after_tag(at: int) -> bool:
+ return left_tag if at == 0 else _ends_closing_tag(text, at)
+
+ pieces: list[str] = []
+ pos = 0
+ code_spans, comment_spans = _inline_spans(text)
+ element_ranges = _code_element_ranges(text)
+ protected_spans: list[tuple[int, int]]
+ if element_ranges:
+ # Whole inline code elements swallow every smaller span inside
+ # them (the tags themselves, code spans, comments); emitting an
+ # inner span first would re-expose the element's content.
+ kept: list[tuple[int, int]] = []
+ idx = 0
+ for span in sorted(comment_spans + _html_tag_spans(text) + code_spans):
+ while (
+ idx < len(element_ranges) and element_ranges[idx][1] <= span[0]
+ ):
+ idx += 1
+ if idx < len(element_ranges) and element_ranges[idx][0] <= span[0]:
+ continue
+ kept.append(span)
+ protected_spans = sorted(element_ranges + kept)
+ else:
+ protected_spans = sorted(
+ comment_spans + _html_tag_spans(text) + code_spans
+ )
+ url_spans = _bare_url_spans(text) if protected_spans else ([], [])
+ for start, end in protected_spans:
+ if end <= pos:
+ continue
+ search_pos = pos
+ while True:
+ link_start = text.find("[", search_pos, start)
+ if link_start == -1:
+ break
+ citation_end, skip_to = _try_strip_run(
+ text,
+ link_start,
+ guard_prev=True,
+ references=references,
+ url_spans=url_spans,
+ prev_char=left if link_start == 0 else "",
+ left=left,
+ )
+ link_end = (
+ _match_verbatim_link(text, link_start)
+ if citation_end is None
+ else None
+ )
+ if link_end is not None:
+ pieces.append(
+ _strip_citations_from_prose(
+ text[pos:link_start],
+ references,
+ left=text[pos - 1] if pos else left,
+ tag_end=0 if after_tag(pos) else -1,
+ )
+ )
+ pieces.append(text[link_start:link_end])
+ pos = link_end
+ search_pos = pos
+ continue
+ search_pos = max(link_start + 1, citation_end or skip_to)
+ if end <= pos:
+ continue
+ if start < pos:
+ # Overlapping span: its opener sat inside already-emitted
+ # verbatim content (an HTML tag or a Markdown link), so the
+ # span is phantom — clamping to pos, the remainder is prose
+ # and must never be re-emitted or shield citations.
+ continue
+ context_start = max(0, pos - 1)
+ stripped = _strip_citations_from_prose(
+ text[context_start:start] + text[start],
+ references,
+ left=left if pos == 0 else "",
+ tag_end=pos - context_start if after_tag(pos) else -1,
+ )
+ pieces.append(stripped[pos - context_start : -1])
+ pieces.append(text[start:end])
+ pos = end
+ pieces.append(
+ _strip_citations_from_prose(
+ text[pos:],
+ references,
+ left=text[pos - 1] if pos else left,
+ tag_end=0 if after_tag(pos) else -1,
+ )
+ )
+ return "".join(pieces)
+
+
+def _html_code_tags(text: str) -> list[tuple[int, int, bool]]:
+ tags: list[tuple[int, int, bool]] = []
+ for start, end in _html_tag_spans(text):
+ if re.match(
+ r"</?(?:code|pre)(?=[\s/>])", text[start:end], re.IGNORECASE
+ ):
+ tags.append((start, end, text[start + 1 : start + 2] == "/"))
+ return tags
+
+
+def _code_element_ranges(text: str) -> list[tuple[int, int]]:
+ """Whole ``<code>/<pre>`` element ranges: open-tag start through the
+ matching close-tag end, nesting-aware.
+
+ Used where the element is INLINE HTML inside a paragraph (CommonMark
+ type 7): everything between the tags renders as code and must stay
+ verbatim. Unclosed elements contribute nothing — the raw tag itself
+ is already protected as an HTML tag span.
+ """
+ ranges: list[tuple[int, int]] = []
+ stack: list[int] = []
+ for start, end, closing in _html_code_tags(text):
+ if closing and stack:
+ open_start = stack.pop()
+ if not stack:
+ ranges.append((open_start, end))
+ elif not closing:
+ stack.append(start)
+ return ranges
+
+
+def _code_tags_start_block(
+ tags: list[tuple[int, int, bool]], line: str, prose_open: bool
+) -> bool:
+ """Whether the line's code/pre tags open a CommonMark HTML *block*.
+
+ ``<pre>`` is a type-1 start condition: it may interrupt a paragraph
+ and its content spans blank lines. ``<code>`` is type 7: it may
+ start a block only when no paragraph is open, the tag sits at the
+ container's content-column start (up to three columns in), and only
+ whitespace follows the complete open tag on the line. Any other
+ ``<code>`` occurrence is inline HTML inside a paragraph — the line
+ is prose, and the element's content is protected by
+ ``_code_element_ranges`` instead of element state.
+ """
+ for start, end, closing in tags:
+ if closing:
+ continue
+ tag = line[start : start + 5].lower()
+ if tag.startswith("<pre"):
+ return True
+ if not tag.startswith("<code"):
+ continue
+ if prose_open:
+ return False
+ prefix = _CONTAINER_PREFIX_RE.match(line)
+ content_at = (
+ len(prefix.group("indent")) + len(prefix.group("quotes"))
+ if prefix
+ else 0
+ )
+ if not (
+ content_at <= start <= content_at + 3
+ and not line[content_at:start].strip()
+ ):
+ return False
+ if end <= len(line) and line[end:].strip():
+ return False
+ return True
+ return False
+
+
+def _strip_html_code_line(
+ line: str,
+ tags: list[tuple[int, int, bool]],
+ depth: int,
+ references: frozenset[str],
+) -> tuple[str, int]:
+ """Preserve real code elements, ignoring tag-shaped attribute text.
+
+ ``tags`` are the code/pre tags opening on this line (line-relative
+ offsets, multi-line aware — a tag whose ``>`` sits on a later line
+ ends past this line's length). Each prose segment is stripped with
+ the character before it as left context so a citation directly after
+ a tag keeps one separator.
+ """
+ pieces: list[str] = []
+ pos = 0
+ for start, end, closing in tags:
+ if depth:
+ pieces.append(line[pos:end])
+ depth += -1 if closing else 1
+ elif not closing:
+ pieces.append(
+ _strip_code_and_citations(
+ line[pos:start],
+ references,
+ left=line[pos - 1] if pos else "",
+ left_tag=_ends_closing_tag(line, pos),
+ )
+ )
+ pieces.append(line[start:end])
+ depth = 1
+ else:
+ pieces.append(
+ _strip_code_and_citations(
+ line[pos:end],
+ references,
+ left=line[pos - 1] if pos else "",
+ left_tag=_ends_closing_tag(line, pos),
+ )
+ )
+ pos = end
+ tail = line[pos:]
+ pieces.append(
+ tail
+ if depth
+ else _strip_code_and_citations(
+ tail,
+ references,
+ left=line[pos - 1] if 0 < pos <= len(line) else "",
+ left_tag=0 < pos <= len(line) and _ends_closing_tag(line, pos),
+ )
+ )
+ return ("".join(pieces), depth)
+
+
+def _open_fence(line: str) -> tuple[str, int] | None:
+ """Return ``(fence_char, run_length)`` when ``line`` opens a fence.
+
+ CommonMark: up to three leading spaces, a run of 3+ backticks or
+ tildes; a backtick fence's info string may not contain backticks.
+ """
+ head = line.lstrip(" ")
+ if len(line) - len(head) > 3:
+ return None
+ for ch in ("`", "~"):
+ if head.startswith(ch * 3):
+ run = len(head) - len(head.lstrip(ch))
+ if ch == "`" and "`" in head[run:]:
+ return None
+ return (ch, run)
+ return None
+
+
+def _container_body(line: str) -> tuple[str, int]:
+ pos = 0
+ depth = 0
+ while match := _QUOTE_PREFIX_RE.match(line, pos):
+ pos = match.end()
+ depth += 1
+ return (line[pos:], depth)
+
+
+def _is_close_fence(line: str, ch: str, run: int) -> bool:
+ """``True`` when ``line`` closes a ``(ch, run)`` fence: same character,
+ at least the opening run length, nothing but whitespace after it."""
+ head = line.lstrip(" ")
+ if len(line) - len(head) > 3:
+ return False
+ stripped_run = len(head) - len(head.lstrip(ch))
+ return stripped_run >= run and head[stripped_run:].strip() == ""
+
+
+_QUOTE_PREFIX_RE = re.compile(r"[ ]{0,3}>[ \t]?")
+_LIST_MARKER_RE = re.compile(
+ r"[ ]{0,3}(?:(?P<bullet>[-+*])|(?P<ordered>\d{1,9})[.)])[ \t]+"
+)
+_SETEXT_UNDERLINE_RE = re.compile(r"[ ]{0,3}(?:=+|-+)[ \t]*$")
+
+
+def _mask_comment_line(
+ line: str, spans: list[tuple[int, int]], in_comment: bool
+) -> tuple[str, bool]:
+ """Mask comments outside inline code while preserving line positions.
+
+ ``spans`` are this line's inline code spans (line-relative offsets,
+ computed over the WHOLE text by the caller, exactly as
+ ``_tag_scan_text`` computes them): a span opened on this line may
+ close on a later one, so a per-line ``_code_spans`` would miss it
+ and latch the comment state on an opener that is really span
+ content — masking a genuinely fenced block further down.
+ """
+ span_index = 0
+ scan = list(line)
+ pos = 0
+ while pos < len(line):
+ if in_comment:
+ close = line.find("-->", pos)
+ end = len(line) if close == -1 else close + 3
+ scan[pos:end] = [" " for _ in range(end - pos)]
+ in_comment = close == -1
+ pos = end
+ continue
+ start = line.find("<!--", pos)
+ if start == -1:
+ break
+ while span_index < len(spans) and spans[span_index][1] <= start:
+ span_index += 1
+ if _is_escaped(line, start) or (
+ span_index < len(spans)
+ and spans[span_index][0] <= start < spans[span_index][1]
+ ):
+ pos = start + 4
+ continue
+ pos = start
+ in_comment = True
+ return ("".join(scan), in_comment)
+
+
+def _blank_spans(scan: list[str], spans: list[tuple[int, int]]) -> None:
+ """Blank character spans in place, preserving line positions."""
+ for start, end in spans:
+ scan[start:end] = [
+ "\n" if char == "\n" else " " for char in scan[start:end]
+ ]
+
+
+def _tag_scan_text(text: str) -> str:
+ """Blank inline code spans and comments for HTML code-tag recognition.
+
+ Used before fences are known (fence recognition itself needs code-tag
+ awareness), so tag-shaped text inside an inline code span or comment
+ cannot masquerade as a ``<code>``/``<pre>`` element.
+ """
+ scan = list(text)
+ code_spans, comment_spans = _inline_spans(text)
+ _blank_spans(scan, code_spans + comment_spans)
+ return "".join(scan)
+
+
+def _code_tags_by_line(
+ scan_text: str, lines: list[str]
+) -> list[list[tuple[int, int, bool]]]:
+ """Bucket code/pre tags by the line they open on, multi-line aware.
+
+ ``_html_tag_spans`` already matches tags whose ``>`` sits on a later
+ line when given a whole string; per-line scanning missed them. Tag
+ offsets are relative to their opening line, so a tag ending on a
+ later line yields an end past the opening line's length.
+ """
+ buckets: list[list[tuple[int, int, bool]]] = [[] for _ in lines]
+ line_index = 0
+ offset = 0
+ for start, end, closing in _html_code_tags(scan_text):
+ while line_index + 1 < len(lines) and start > offset + len(
+ lines[line_index]
+ ):
+ offset += len(lines[line_index]) + 1
+ line_index += 1
+ buckets[line_index].append((start - offset, end - offset, closing))
+ return buckets
+
+
+def _code_spans_by_line(
+ spans: list[tuple[int, int]], lines: list[str]
+) -> list[list[tuple[int, int]]]:
+ """Clamp whole-text inline code spans to every line they cover.
+
+ Unlike ``_code_tags_by_line`` (open-line only), a span opened on an
+ earlier line keeps masking comment openers on each line inside it,
+ so each intersected line receives the span clamped to its own
+ bounds. Spans arrive sorted and disjoint, so the walk is linear.
+ """
+ buckets: list[list[tuple[int, int]]] = [[] for _ in lines]
+ line_index = 0
+ offset = 0
+ for start, end in spans:
+ while line_index + 1 < len(lines) and start > offset + len(
+ lines[line_index]
+ ):
+ offset += len(lines[line_index]) + 1
+ line_index += 1
+ i = line_index
+ line_offset = offset
+ while i < len(lines) and line_offset < end:
+ clamp_start = max(start, line_offset) - line_offset
+ clamp_end = min(end, line_offset + len(lines[i])) - line_offset
+ if clamp_start < clamp_end:
+ buckets[i].append((clamp_start, clamp_end))
+ line_offset += len(lines[i]) + 1
+ i += 1
+ return buckets
+
+
+def _fence_ranges(
+ lines: list[str], code_tags: list[list[tuple[int, int, bool]]]
+) -> dict[int, int]:
+ """Map fence openers to exclusive ends, including their container boundary.
+
+ Each visited fence consumes its content once. Lists retain their content
+ indentation so a continuation fence is distinct from indented code.
+ ``code_tags`` (from ``_code_tags_by_line``) tracks open ``<code>``/
+ ``<pre>`` elements: a fence glyph inside one is element content, not a
+ fence, so an unclosed ``` inside a code element cannot swallow — and
+ protect from stripping — the rest of the document. Tags inside a
+ recognized fence are literal fence content and never change the depth.
+ """
+ ranges: dict[int, int] = {}
+ list_indents: list[int] = []
+ previous_depth = 0
+ in_comment = False
+ html_depth = 0
+ # Comment masking needs code spans over the whole text (a span may
+ # straddle lines), computed exactly as ``_tag_scan_text`` does.
+ span_lines = _code_spans_by_line(_code_spans("\n".join(lines)), lines)
+ i = 0
+ while i < len(lines):
+ comment_before = in_comment
+ scan_line, in_comment = _mask_comment_line(
+ lines[i], span_lines[i], in_comment
+ )
+ body, quote_depth = _container_body(scan_line)
+ if quote_depth != previous_depth:
+ list_indents.clear()
+ previous_depth = quote_depth
+ body = body.expandtabs(4)
+ leading = len(body) - len(body.lstrip(" "))
+ if body.strip():
+ while list_indents and leading < list_indents[-1]:
+ list_indents.pop()
+ offset = list_indents[-1] if list_indents else 0
+ while True:
+ marker = _LIST_MARKER_RE.match(body, offset)
+ if marker is not None:
+ offset = marker.end()
+ list_indents.append(offset)
+ continue
+ quote = _QUOTE_PREFIX_RE.match(body, offset)
+ if quote is not None:
+ quote_depth += 1
+ body = body[quote.end() :]
+ offset = 0
+ list_indents.clear()
+ continue
+ break
+ fence = _open_fence(body[offset:])
+ if fence is not None and not html_depth:
+ # Comments in a fence info string are literal fence content.
+ in_comment = comment_before
+ end = i + 1
+ while end < len(lines):
+ continuation, depth = _container_body(lines[end])
+ if depth < quote_depth:
+ break
+ continuation = continuation.expandtabs(4)
+ if depth == quote_depth:
+ if continuation.strip() and not continuation.startswith(
+ " " * offset
+ ):
+ break
+ if _is_close_fence(continuation[offset:], *fence):
+ end += 1
+ break
+ end += 1
+ ranges[i] = end
+ i = end
+ continue
+ for _, _, closing in code_tags[i]:
+ html_depth = max(0, html_depth + (-1 if closing else 1))
+ i += 1
+ return ranges
+
+
+def _indented_past_containers(body: str, quote_depth: int) -> tuple[bool, int]:
+ """Decide indented code against the content column of the containers
+ opened on this line; a list item opened on an earlier line is not
+ tracked, so its later lines are measured from column 0.
+
+ The same container walk ``_fence_ranges`` uses — blockquote body via
+ ``_container_body``, ``expandtabs(4)`` columns, then ``_LIST_MARKER_RE``
+ and inner quote prefixes — but a marker contributes its TRUE content
+ column (one space past the marker itself, not past all its trailing
+ whitespace), so ``- [1]`` is a list item whose content is indented
+ code while ``- [1]`` (three columns past the content column) is a
+ paragraph continuation that still strips. Tabs count toward the next
+ multiple-of-4 column. Returns ``(indented, quote_depth)`` with any
+ quote opened after a list marker added to the depth.
+ """
+ body = body.expandtabs(4)
+ offset = 0
+ while True:
+ marker = _LIST_MARKER_RE.match(body, offset)
+ if marker is not None:
+ text = marker.group()
+ offset = marker.end() - (len(text) - len(text.rstrip())) + 1
+ continue
+ quote = _QUOTE_PREFIX_RE.match(body, offset)
+ if quote is not None:
+ quote_depth += 1
+ body = body[quote.end() :]
+ offset = 0
+ continue
+ break
+ return (body[offset:].startswith(" "), quote_depth)
+
+
+def _html_scan_lines(
+ text: str, lines: list[str], fences: dict[int, int]
+) -> list[str]:
+ """Mask fenced/inline code and comments before recognizing HTML code tags."""
+ scan = list(text)
+ offsets: list[int] = []
+ offset = 0
+ for line in lines:
+ offsets.append(offset)
+ offset += len(line) + 1
+
+ for i, end_line in fences.items():
+ start = offsets[i]
+ end = len(text) if end_line == len(lines) else offsets[end_line] - 1
+ _blank_spans(scan, [(start, end)])
+
+ code_spans, comment_spans = _inline_spans("".join(scan))
+ _blank_spans(scan, code_spans + comment_spans)
+ return "".join(scan).split("\n")
+
+
+def _reference_continuation(reference: re.Match[str], line: str) -> str | None:
+ """Return ``line``'s body when it may continue the definition at
+ ``reference`` (as its destination or title line), else ``None``.
+
+ Such a line continues the definition's paragraph, and indented code
+ cannot interrupt a paragraph, so CommonMark sets no indentation limit
+ on it: indent 0, four spaces, a tab and any indent at, past or short
+ of a list item's content column all qualify, as does a lazy line with
+ fewer ``>`` markers. A blank line ends the paragraph, and a deeper
+ ``>`` opens a nested blockquote, which interrupts it. Lines that open
+ a fence or ``<pre>`` element are rejected by the caller.
+ """
+ continuation = _CONTAINER_PREFIX_RE.match(line)
+ if continuation is None:
+ return None
+ if continuation.group("quotes").count(">") > reference.group(
+ "quotes"
+ ).count(">"):
+ return None
+ body = continuation.group("body")
+ return body if body.strip() else None
+
+
+def _valid_reference_destination(rest: str) -> bool:
+ """``True`` when ``rest`` (the text after ``]:``) parses as a link
+ destination plus an optional title, per CommonMark.
+
+ A destination is ``<...>`` or a single whitespace-free token with
+ balanced parentheses; a well-formed title (``_REFERENCE_TITLE_RE``)
+ may follow, separated by whitespace, and nothing else may remain.
+ A destination with spaces and no valid title ("[Note]: See [1] for
+ details.") is therefore prose, not a reference definition.
+ """
+ s = rest.lstrip()
+ if not s:
+ return False
+ if s.startswith("<"):
+ close = s.find(">")
+ while close != -1 and _is_escaped(s, close):
+ close = s.find(">", close + 1)
+ if close == -1:
+ return False
+ after = s[close + 1 :].strip()
+ else:
+ parts = s.split(None, 1)
+ token = parts[0]
+ depth = 0
+ k = 0
+ while k < len(token):
+ ch = token[k]
+ if ch == "\\":
+ k += 2
+ continue
+ if ch == "(":
+ depth += 1
+ elif ch == ")":
+ depth -= 1
+ if depth < 0:
+ return False
+ k += 1
+ if depth != 0:
+ return False
+ after = parts[1].strip() if len(parts) > 1 else ""
+ return not after or _REFERENCE_TITLE_RE.fullmatch(after) is not None
+
+
+def _reference_definition_extent(
+ reference: re.Match[str],
+ lines: list[str],
+ i: int,
+ block_openers: frozenset[int],
+) -> int | None:
+ """Return the exclusive end of a GENUINE reference definition whose
+ match is at ``lines[i]``, or ``None`` when the line is not one.
+
+ The destination may sit on the definition line or on the next line;
+ one further line may carry the title (``_reference_continuation``
+ decides which lines can continue a definition). ``block_openers``
+ holds the indexes of lines that open a fenced code block or a
+ ``<pre>`` element: either interrupts the definition's paragraph, so
+ it is never a continuation line — consuming one as a destination
+ would expose the block's code to stripping. An invalid destination
+ (see ``_valid_reference_destination``) makes the line prose, so its
+ citation markers strip normally.
+ """
+ next_line = i + 1
+ destination = reference.group("destination")
+ if destination.strip():
+ if not _valid_reference_destination(destination):
+ return None
+ else:
+ if next_line >= len(lines) or next_line in block_openers:
+ return None
+ continuation_body = _reference_continuation(reference, lines[next_line])
+ if continuation_body is None or not _valid_reference_destination(
+ continuation_body.strip()
+ ):
+ return None
+ next_line += 1
+ if next_line < len(lines) and next_line not in block_openers:
+ continuation_body = _reference_continuation(reference, lines[next_line])
+ if (
+ continuation_body is not None
+ and _REFERENCE_TITLE_RE.fullmatch(continuation_body.strip())
+ is not None
+ ):
+ next_line += 1
+ return next_line
+
+
+def _ends_markdown_block(
+ line: str, previous_body: str = "", previous_break: bool = False
+) -> bool:
+ # ``_backtick_runs`` feeds keepends lines; the trailing line ending
+ # must go before a fullmatch can see a bare thematic break.
+ candidate = line.rstrip("\r\n")
+ return (
+ _ATX_HEADING_RE.match(candidate) is not None
+ or _THEMATIC_BREAK_RE.fullmatch(candidate) is not None
+ or _HTML_BLOCK_TAG_RE.match(candidate) is not None
+ # Same condition as ``_backtick_runs``: a setext underline ends
+ # the paragraph above it (so a following indented line is code),
+ # but only when the previous line is a non-blank paragraph line
+ # that did not itself end a block.
+ or (
+ bool(previous_body.strip())
+ and not previous_break
+ and _SETEXT_UNDERLINE_RE.fullmatch(candidate) is not None
+ )
+ )
+
+
+def strip_prior_turn_citations(text: str) -> str:
+ """Remove prior-turn citation markers from ``text`` (see module docstring
+ for the covered shapes). Code blocks/elements, inline code spans,
+ non-citation markdown links and all whitespace other than one separator
+ flanking each stripped run (see the module docstring) pass through
+ verbatim; ``""`` returns ``""``."""
+ if not text:
+ return text
+ lines = text.split("\n")
+ out: list[str] = []
+ prose: list[str] = []
+ # Fence recognition needs code-element awareness (a ``` inside an open
+ # <code>/<pre> is element content), and code-tag recognition wants
+ # inline code and comments blanked — which needs no fence knowledge.
+ fence_tags = _code_tags_by_line(_tag_scan_text(text), lines)
+ fences = _fence_ranges(lines, fence_tags)
+ html_scan_lines = _html_scan_lines(text, lines, fences)
+ code_tags = _code_tags_by_line("\n".join(html_scan_lines), lines)
+ # A fence or <pre> element interrupts a paragraph, so neither opening
+ # line can continue a reference definition. ``prose_open=True`` is the
+ # state the line would meet after an unfinished ``[label]:`` line; a
+ # <code> element (HTML block type 7) cannot interrupt and is excluded.
+ block_openers = frozenset(fences).union(
+ k
+ for k, tags in enumerate(code_tags)
+ if tags and _code_tags_start_block(tags, lines[k], prose_open=True)
+ )
+ # Definitions written as code or comments are examples, not link targets.
+ reference_labels: set[str] = set()
+ reference_code_depth = 0
+ for i, scan_line in enumerate(html_scan_lines):
+ if not reference_code_depth:
+ reference_match = _REFERENCE_DEF_RE.match(scan_line)
+ if (
+ reference_match is not None
+ and _reference_definition_extent(
+ reference_match, html_scan_lines, i, block_openers
+ )
+ is not None
+ ):
+ reference_labels.add(
+ _reference_label(reference_match.group("label"))
+ )
+ for _, _, closing in code_tags[i]:
+ reference_code_depth = max(
+ 0, reference_code_depth + (-1 if closing else 1)
+ )
+ references = frozenset(reference_labels)
+
+ def flush_prose() -> None:
+ if prose:
+ out.append(_strip_code_and_citations("\n".join(prose), references))
+ prose.clear()
+
+ i = 0
+ in_html_code = 0
+ # Outermost open element is <code> (HTML block type 7: its block
+ # ends at the first blank line) vs <pre> (type 1: content spans
+ # blank lines until the closing tag).
+ html_code_type7 = False
+ indented_quote_depth: int | None = None
+ block_boundary = False
+ while i < len(lines):
+ line = lines[i]
+ if in_html_code:
+ if html_code_type7 and not line.strip():
+ # A type-7 block ends at a blank line; the element's
+ # content closed with it and the next paragraph is prose.
+ in_html_code = 0
+ html_code_type7 = False
+ out.append(line)
+ block_boundary = True
+ i += 1
+ continue
+ stripped, in_html_code = _strip_html_code_line(
+ line, code_tags[i], in_html_code, references
+ )
+ out.append(stripped)
+ block_boundary = in_html_code == 0
+ i += 1
+ continue
+ if i in fences:
+ end = fences[i]
+ flush_prose()
+ out.append("\n".join(lines[i:end]))
+ i = end
+ block_boundary = True
+ continue
+ body, quote_depth = _container_body(line)
+ # Indented code is measured from the content column of the
+ # containers opened on this line (list markers, inner quotes, tab
+ # stops), not column 0 of the quote-stripped body; a later line
+ # of a list item carries no marker and is measured from column 0.
+ indented, quote_depth = _indented_past_containers(body, quote_depth)
+ if indented_quote_depth is not None:
+ if quote_depth == indented_quote_depth and (
+ indented or not body.strip()
+ ):
+ flush_prose()
+ out.append(line)
+ i += 1
+ continue
+ indented_quote_depth = None
+ if indented and (not prose or not prose[-1].strip() or block_boundary):
+ flush_prose()
+ out.append(line)
+ indented_quote_depth = quote_depth
+ block_boundary = False
+ i += 1
+ continue
+ reference = _REFERENCE_DEF_RE.match(line)
+ if reference is not None:
+ extent = _reference_definition_extent(
+ reference, lines, i, block_openers
+ )
+ if extent is not None:
+ flush_prose()
+ out.extend(lines[i:extent])
+ i = extent
+ block_boundary = True
+ continue
+ if code_tags[i] and _code_tags_start_block(
+ code_tags[i], line, prose_open=bool(prose and prose[-1].strip())
+ ):
+ flush_prose()
+ stripped, in_html_code = _strip_html_code_line(
+ line, code_tags[i], 0, references
+ )
+ first_open = next((tag for tag in code_tags[i] if not tag[2]), None)
+ html_code_type7 = (
+ in_html_code > 0
+ and first_open is not None
+ and line[first_open[0] : first_open[0] + 5]
+ .lower()
+ .startswith("<code")
+ )
+ out.append(stripped)
+ block_boundary = in_html_code == 0
+ i += 1
+ continue
+ prose.append(line)
+ block_boundary = _ends_markdown_block(
+ body,
+ _container_body(prose[-2])[0] if len(prose) > 1 else "",
+ block_boundary,
+ )
+ i += 1
+ flush_prose()
+ return "\n".join(out)
diff --git a/src/local_deep_research/chat/context.py b/src/local_deep_research/chat/context.py
index 2691a81c5c..eeb2236d51 100644
--- a/src/local_deep_research/chat/context.py
+++ b/src/local_deep_research/chat/context.py
@@ -13,8 +13,33 @@ from typing import Dict, Any, List, Optional
from loguru import logger
+from .citation_stripping import strip_prior_turn_citations
from ..config.thread_settings import get_setting_from_snapshot
+# Recognized prior-turn citation markers are stripped from ASSISTANT-authored text
+# only — see ``_build_conversation_text`` — because they index that turn's
+# source list and dangle once the follow-up renumbers citations against
+# its own sources. The stripping engine — grounded in CitationFormatter's
+# emission shapes, segmenting fenced blocks / inline code spans / prose and
+# scanning prose with bounded citation lookups — lives in
+# ``chat.citation_stripping`` and is documented there.
+
+
+def _strip_or_passthrough(text: str) -> str:
+ """Strip citation markers, failing safe to the unstripped text.
+
+ Stripping is a context-quality cleanup; an unexpected error in it must
+ not fail every later follow-up in the session, so it is logged and the
+ text is carried over as-is.
+ """
+ try:
+ return strip_prior_turn_citations(text)
+ except Exception:
+ logger.opt(exception=True).warning(
+ "Citation stripping failed; carrying prior-turn text unstripped"
+ )
+ return text
+
class ChatContextManager:
"""
@@ -128,6 +153,11 @@ class ChatContextManager:
for m in self.messages
)
+ # Stripping is scoped per-mode inside _select_prior_findings — e.g.
+ # "full" mode strips assistant turns only, as it assembles them, and
+ # must not have a second, role-blind strip pass applied to the
+ # already-mixed-role transcript here (that would delete a markdown
+ # link the USER typed in an earlier turn; see _build_conversation_text).
findings = self._select_prior_findings(current_query, is_multi_turn)
return {
@@ -168,15 +198,30 @@ class ChatContextManager:
if mode == "none":
findings = ""
elif mode == "raw":
+ # _extract_findings_from_history strips each assistant
+ # message itself — BEFORE truncating it, so a citation
+ # crossing the truncation cutoff can't leave half a marker
+ # behind (no pattern matches "[](htt...").
findings = self._extract_findings_from_history()
elif mode == "full":
+ # _build_conversation_text strips citation markers per-turn,
+ # assistant lines only, as it assembles the mixed-role
+ # transcript — a second, role-blind strip pass here would
+ # delete a markdown link the USER typed in an earlier turn.
findings = self._build_conversation_text()
elif current_query:
- # "summary" with a question to focus on.
+ # "summary" with a question to focus on. The transcript the
+ # LLM reads keeps user turns verbatim (assistant turns are
+ # stripped while it is built), but the summary it returns is
+ # stripped as a whole, with no knowledge of which turn a span
+ # came from: a citation-shaped string the user typed — e.g.
+ # ``[1]`` or ``[arxiv.org](https://arxiv.org/abs/...)`` — is
+ # removed if the summary repeats it. Ordinary links survive.
findings = self._summarize_prior_work(current_query)
else:
# "summary" with no question (e.g. a no-arg build_research_context
- # call): fall back to raw recent findings.
+ # call): fall back to raw recent findings (already stripped
+ # during extraction).
findings = self._extract_findings_from_history()
# Observability: the summary path is otherwise silent (no token-counter
@@ -190,21 +235,69 @@ class ChatContextManager:
)
return findings
+ def _strip_citation_markers(self, text: str) -> str:
+ """Remove prior-turn citation markers from carried-over context.
+
+ Handles the shapes CitationFormatter stores — markdown hyperlinks
+ like ``[[1]](url)`` / ``[[arxiv.org-1]](url)`` (plus, for a model
+ that re-renders one with a single bracket pair, a domain label
+ naming the link's own host, ``[arxiv.org](https://arxiv.org/...)``;
+ numeric single-bracket links such as ``[1998](url)`` are kept),
+ the plain source-tagged form ``[arxiv-1]`` / ``[local-1]`` /
+ ``[ai-2]`` / ``[personal-notes-3]``, plain
+ ``[1]`` / ``[1, 2]``, and lenticular ``【1】`` — including runs of
+ adjacent markers (comma groups are concatenated with no separator).
+ See ``chat.citation_stripping`` for the full shape grounding.
+
+ Fenced code blocks (CommonMark ``` and ~~~ fences) and inline code
+ spans are not citation sites and pass through untouched. A stripped
+ run consumes one flanking horizontal-whitespace separator — the one
+ before it mid-line or at the end of a line, the one after it at the
+ start of a line (indentation before it is kept) — so removing
+ ``word [[1]](url) word`` never introduces a doubled space between
+ the flanking words, and it is horizontal-only, so a marker starting
+ the next line never joins two lines. If stripping raises, the error
+ is logged and the text is carried over unstripped. There is
+ deliberately no separate whitespace-collapse pass over the rest of
+ the string:
+ a blanket ``[ \\t]{2,}`` collapse would also flatten unrelated
+ indentation (code blocks, nested markdown lists, table columns)
+ that never had a citation marker anywhere near it.
+ """
+ if not text:
+ return text
+ return _strip_or_passthrough(text)
+
def _build_conversation_text(self) -> str:
"""Render the prior conversation (both roles) as a plain transcript.
Trims oldest-first to ``CONTEXT_INPUT_CHAR_BUDGET`` so the most recent
turns survive and the summarizer's input cap never has to truncate.
+
+ Citation markers are stripped from ASSISTANT lines only, per turn,
+ before assembly — never from the joined transcript. Those markers
+ index the prior turn's own source list and dangle once renumbered
+ against the follow-up's freshly-found sources, but a citation-shaped
+ string a USER typed (e.g. a markdown link they pasted) is verbatim
+ user content and must reach the follow-up prompt untouched.
"""
lines: List[str] = []
used = 0
for msg in reversed(self.messages):
if not isinstance(msg, dict):
continue
- content = (msg.get("content") or "").strip()
- if not content:
+ content = msg.get("content") or ""
+ if not content.strip():
continue
- role = (msg.get("role") or "unknown").capitalize()
+ role_raw = msg.get("role") or "unknown"
+ if role_raw == "assistant":
+ # Strip citation markers BEFORE the outer trim: trimming
+ # first would dedent a leading indented-code line into
+ # prose, and its bracket literals would be deleted as
+ # citations (raw mode never trims, so modes diverged).
+ content = self._strip_citation_markers(content)
+ content = content.strip()
+ role = role_raw.capitalize()
line = f"{role}: {content}"
remaining = self.CONTEXT_INPUT_CHAR_BUDGET - used
if remaining <= 0:
@@ -246,7 +339,17 @@ class ChatContextManager:
)
return ""
- return FocusedSummarizer(
+ class ChatContextSummarizer(FocusedSummarizer):
+ def _truncate(self, summary: str) -> str:
+ # Strip citations before the output cap, so a marker
+ # straddling the cap is removed whole instead of being cut
+ # into residue no pattern matches ("[[1]](https://exa...").
+ # The summary arrives already trimmed (ProcessingLLMWrapper
+ # and BaseSummarizer.summarize both strip it), so a leading
+ # indented-code line in it reads as prose here.
+ return super()._truncate(_strip_or_passthrough(summary).strip())
+
+ return ChatContextSummarizer(
llm,
focus_query=current_query,
max_sentences=self.CONTEXT_SUMMARY_MAX_SENTENCES,
@@ -263,7 +366,10 @@ class ChatContextManager:
for msg in self.messages:
if msg.get("role") == "assistant" and msg.get("research_id"):
- content = msg.get("content") or ""
+ # Strip BEFORE truncating: truncation mid-citation would
+ # cut a marker into a shape no pattern can match (residue
+ # like "[](htt..." would survive into follow-up context).
+ content = self._strip_citation_markers(msg.get("content") or "")
# Summarize long responses - take first part
if len(content) > 500:
# Try to find a natural break point
diff --git a/tests/chat/test_chat_followup_citation_stripping.py b/tests/chat/test_chat_followup_citation_stripping.py
new file mode 100644
index 0000000000..17848672f3
--- /dev/null
+++ b/tests/chat/test_chat_followup_citation_stripping.py
@@ -0,0 +1,1867 @@
+"""Tests for format-aware citation stripping in chat follow-up context.
+
+Prior-turn answers carry citations in whatever shape `report.citation_format`
+produced (markdown hyperlinks `[[1]](url)` / `[[arxiv.org-1]](url)`, the plain
+source-tagged form `[arxiv-1]` / `[local-1]`, plain `[1]`, lenticular `【1】`).
+Those index the prior turn's sources, so they must be stripped from the
+context carried into a follow-up — but ONLY from assistant-authored text;
+anything the user typed themselves must survive untouched.
+"""
+
+import time
+
+import pytest
+from markdown import markdown
+
+from local_deep_research.chat.context import ChatContextManager
+
+
+def _manager(messages=None, mode=None):
+ snapshot = {"chat.followup_context_mode": mode} if mode else None
+ return ChatContextManager(
+ session_id="s1",
+ messages=messages or [],
+ accumulated_context={},
+ settings_snapshot=snapshot,
+ )
+
+
+class TestStripCitationMarkers:
+ @pytest.mark.parametrize(
+ "text,expected",
+ [
+ # number_hyperlinks (default report.citation_format)
+ (
+ "Light is used [[1]](http://a) in cells.",
+ "Light is used in cells.",
+ ),
+ ("Both [[1, 2]](http://a) agree.", "Both agree."),
+ # domain / domain-id hyperlinks
+ # single-bracket domain forms count only when the label names
+ # the link's own host, as CitationFormatter would label it
+ ("Per [arxiv.org-1](https://arxiv.org/abs/1).", "Per."),
+ ("Per [arxiv.org](https://www.arxiv.org/abs/1).", "Per."),
+ # no_hyperlinks plain markers
+ ("Plain [1] cite.", "Plain cite."),
+ ("A group [1, 2] of cites.", "A group of cites."),
+ # lenticular brackets some models emit
+ ("Lenticular 【1】 marker.", "Lenticular marker."),
+ ("Lenticular 【1, 2】 marker.", "Lenticular marker."),
+ # mixed old + new in one string
+ ("Old [1] new [[2]](http://x) mix.", "Old new mix."),
+ # nothing to strip
+ ("No citations at all.", "No citations at all."),
+ ("", ""),
+ ],
+ )
+ def test_strips_all_citation_formats(self, text, expected):
+ assert _manager()._strip_citation_markers(text) == expected
+
+ def test_none_input_passthrough(self):
+ assert _manager()._strip_citation_markers(None) is None
+
+
+class TestBuildResearchContextStripsCitations:
+ """End-to-end: a raw-mode follow-up's past_findings has no leaked markers."""
+
+ def test_raw_mode_strips_hyperlinked_citations(self, mocker):
+ # raw mode makes no LLM call; guard get_llm anyway.
+ mocker.patch("local_deep_research.config.llm_config.get_llm")
+ messages = [
+ {
+ "role": "user",
+ "content": "How do plants make energy?",
+ "message_type": "query",
+ },
+ {
+ "role": "assistant",
+ "content": (
+ "Plants convert light [[1]](http://a) into energy in "
+ "chloroplasts [arxiv.org-2](https://arxiv.org/abs/2)."
+ ),
+ "message_type": "response",
+ "research_id": "r1",
+ },
+ ]
+ result = _manager(messages, mode="raw").build_research_context(
+ current_query="explain more"
+ )
+ past = result["past_findings"]
+ assert "](http" not in past # no markdown link residue
+ assert "[1]" not in past and "[arxiv.org-2]" not in past
+ assert "Plants convert light" in past and "chloroplasts" in past
+
+
+class TestStripScopeAndRobustness:
+ """Adversarial coverage: scope, over/under-strip edges, long input."""
+
+ def test_only_prior_findings_stripped_not_original_query(self, mocker):
+ """Stripping targets carried-over findings, never the user's query.
+
+ ``original_query`` is the session's first user message verbatim; a
+ bracketed token the user themselves typed must survive untouched.
+ """
+ mocker.patch("local_deep_research.config.llm_config.get_llm")
+ messages = [
+ {
+ "role": "user",
+ "content": "What does array[1] mean?",
+ "message_type": "query",
+ },
+ {
+ "role": "assistant",
+ "content": "It indexes [[1]](http://a) the element.",
+ "message_type": "response",
+ "research_id": "r1",
+ },
+ ]
+ result = _manager(messages, mode="raw").build_research_context(
+ current_query="tell me more"
+ )
+ # User's own query preserved verbatim (not routed through the strip).
+ assert result["original_query"] == "What does array[1] mean?"
+ # Prior findings had their carried-over marker removed.
+ assert "](http" not in result["past_findings"]
+
+ def test_long_input_with_many_markers_is_linear(self, monkeypatch):
+ """Many markers plus a glued bracket flood cost O(1) lookups each.
+
+ Work is counted, not timed: every citation-token and verbatim-link
+ lookup is tallied. Each marker needs one lookup, and the whole
+ ``a[[[[...`` flood one, because a failed match advances past the
+ entire opener run; retrying from every ``[`` of the run would make
+ the flood alone cost one lookup (and one run scan) per bracket.
+ """
+ from local_deep_research.chat import citation_stripping as cs
+
+ calls = {"n": 0}
+
+ def counted(real):
+ def wrapper(*args):
+ calls["n"] += 1
+ return real(*args)
+
+ return wrapper
+
+ monkeypatch.setattr(
+ cs, "_match_citation_token", counted(cs._match_citation_token)
+ )
+ monkeypatch.setattr(
+ cs, "_match_verbatim_link", counted(cs._match_verbatim_link)
+ )
+ k = 4000
+ text = (
+ ("word [[1]](http://a) " * k)
+ + ("[1, 2] " * k)
+ + "a"
+ + ("[" * k)
+ + " tail"
+ )
+ out = _manager()._strip_citation_markers(text)
+ assert "](http" not in out
+ assert "[1]" not in out and "[1, 2]" not in out
+ assert out.count("word") == k
+ # 2k markers -> 2k lookups, plus a constant for the flood.
+ assert calls["n"] <= 2 * k + 16
+
+ def test_escaped_bracket_runs_are_not_rescanned(self, monkeypatch):
+ """A label full of escaped ``]`` is skipped once per link lookup.
+
+ The verbatim-link matcher used to re-find and re-check every
+ escaped ``]`` after each nested ``[``, about 6,000 escape checks
+ per input character on this shape (over 1 ms per character).
+ """
+ from local_deep_research.chat import citation_stripping as cs
+
+ calls = {"n": 0}
+ real = cs._is_escaped
+
+ def counted(text, pos):
+ calls["n"] += 1
+ return real(text, pos)
+
+ monkeypatch.setattr(cs, "_is_escaped", counted)
+ text = ("[\\]" * 330 + "](x)") * 3
+ assert _manager()._strip_citation_markers(text) == text
+ assert calls["n"] <= len(text)
+
+ def test_adversarial_unclosed_brackets_terminate(self):
+ """Pathological unclosed-bracket run must not hang and must be inert."""
+ out = _manager()._strip_citation_markers("[" * 20000 + "1")
+ # Nothing forms a complete citation, so content is returned unharmed.
+ assert out.endswith("1")
+
+
+class TestSourceTaggedCitations:
+ """SOURCE_TAGGED_HYPERLINKS' plain (non-hyperlinked) marker shape.
+
+ CitationFormatter._format_source_tagged_hyperlinks emits ``[tag-N]``
+ (no trailing ``(url)``) for any cited source whose URL is not http(s)
+ — local files and RAG/library hits. This used to survive into
+ follow-up context unstripped (the exact dangling-citation problem this
+ module exists to prevent); the tag shape is covered in
+ ``chat.citation_stripping`` (see ``_is_tag_id``).
+ """
+
+ @pytest.mark.parametrize(
+ "text,expected",
+ [
+ # CitationFormatter._extract_source_label's known-tag vocabulary
+ ("From the paper [arxiv-1].", "From the paper."),
+ ("See the docs [local-1] here.", "See the docs here."),
+ ("Per [pubmed-3] and [pmc-4].", "Per and."),
+ ("Study [biorxiv-2] found this.", "Study found this."),
+ ("As in [medrxiv-7] trial.", "As in trial."),
+ ("Per [semantic_scholar-12] study.", "Per study."),
+ ("Ref [doi-9] confirms it.", "Ref confirms it."),
+ # domain-derived tags (_extract_domain always yields a dotted
+ # string) also covered
+ ("See [nytimes.com-9] report.", "See report."),
+ ("Per [arxiv.org-1] source.", "Per source."),
+ # dotless collection slugs — _slugify_collection keeps only
+ # [a-z0-9-], so RAG collection tags never carry the dot the
+ # old pattern required
+ ("Finding [personal-notes-3].", "Finding."),
+ ("From [my-papers-2] here.", "From here."),
+ # digit-leading slug (collection named "2024 notes") still
+ # has letters, so it is a tag — unlike a numeric range
+ ("Cites [2024-notes-3] store.", "Cites store."),
+ # hyperlinked slug label (linkable collection source)
+ ("See [[personal-notes-3]](http://a) cited.", "See cited."),
+ # mixed with the hyperlinked form in one string
+ (
+ "Mix [local-1] and [[arxiv-2]](http://a) both.",
+ "Mix and both.",
+ ),
+ ],
+ )
+ def test_strips_plain_source_tagged_markers(self, text, expected):
+ assert _manager()._strip_citation_markers(text) == expected
+
+ @pytest.mark.parametrize(
+ "text",
+ [
+ # Common code-indexing prose this (developer-facing) tool's
+ # chat regularly carries. The tag charset is lowercase-only
+ # with at least one letter and the (?<![\w.\]]) lookbehind
+ # rejects markers glued to an identifier — so these are NOT
+ # treated as citations (an open `[label-N]` pattern would
+ # catch them — the class of over-widening #5903 fixed for the
+ # sub-index pattern).
+ "x = arr[i-1]",
+ "data[idx-2] = 5",
+ "See [RFC-2119] for definitions.",
+ "Range [1 - 2] applies.",
+ ],
+ )
+ def test_does_not_strip_code_like_or_uppercase_brackets(self, text):
+ assert _manager()._strip_citation_markers(text) == text
+
+ def test_end_to_end_raw_mode_strips_source_tagged_marker(self, mocker):
+ mocker.patch("local_deep_research.config.llm_config.get_llm")
+ messages = [
+ {
+ "role": "user",
+ "content": "What's in my notes?",
+ "message_type": "query",
+ },
+ {
+ "role": "assistant",
+ "content": "Your notes say the deadline is Friday [local-1].",
+ "message_type": "response",
+ "research_id": "r1",
+ },
+ ]
+ result = _manager(messages, mode="raw").build_research_context(
+ current_query="tell me more"
+ )
+ assert "[local-1]" not in result["past_findings"]
+ assert "deadline is Friday" in result["past_findings"]
+
+
+class TestFullModeScopedToAssistantTurns:
+ """`full` mode must strip citations from assistant turns only.
+
+ Regression for the bug where ``build_research_context`` ran the WHOLE
+ joined transcript (both roles) through the citation stripper, deleting
+ any markdown link a USER typed in an earlier turn.
+ """
+
+ def test_user_typed_markdown_link_survives_full_mode(self, mocker):
+ mocker.patch("local_deep_research.config.llm_config.get_llm")
+ link = "[arxiv.org](https://arxiv.org/abs/2401.00001)"
+ user_text = f"See {link} for context."
+ # The label is citation-shaped (it names its own host), so only the
+ # role scoping can keep it: a role-blind pass would delete it.
+ assert link not in _manager()._strip_citation_markers(user_text)
+ messages = [
+ {
+ "role": "user",
+ "content": user_text,
+ "message_type": "query",
+ },
+ {
+ "role": "assistant",
+ "content": "Noted. Light is used [[1]](http://a) in cells.",
+ "message_type": "response",
+ "research_id": "r1",
+ },
+ ]
+ result = _manager(messages, mode="full").build_research_context(
+ current_query="tell me more"
+ )
+ past = result["past_findings"]
+ # The user's own link is preserved verbatim.
+ assert link in past
+ # The assistant's carried-over citation marker is still stripped.
+ assert "[[1]](http://a)" not in past
+ assert "Light is used in cells." in past
+
+ def test_user_typed_source_tagged_shape_survives_full_mode(self, mocker):
+ """A user typing something citation-shaped (e.g. quoting a prior
+ answer) must not be treated as a citation either — scope is by
+ role, not by pattern."""
+ mocker.patch("local_deep_research.config.llm_config.get_llm")
+ messages = [
+ {
+ "role": "user",
+ "content": "You mentioned [arxiv-1] earlier, what did it say?",
+ "message_type": "query",
+ },
+ {
+ "role": "assistant",
+ "content": "It found X [arxiv-1].",
+ "message_type": "response",
+ "research_id": "r1",
+ },
+ ]
+ result = _manager(messages, mode="full").build_research_context(
+ current_query="more"
+ )
+ past = result["past_findings"]
+ assert "You mentioned [arxiv-1] earlier" in past
+ assert "It found X." in past
+
+
+class TestWhitespacePreservedOnStrip:
+ """Stripping must not corrupt unrelated indentation/whitespace.
+
+ Regression for the bug where a blanket ``[ \\t]{2,}`` collapse ran over
+ the ENTIRE findings string on every follow-up turn, flattening code
+ indentation, nested list indentation and table column alignment that
+ never had a citation marker anywhere near them.
+ """
+
+ def test_code_block_indentation_survives(self):
+ text = "def f():\n if x:\n return data [[1]](http://a)"
+ out = _manager()._strip_citation_markers(text)
+ assert out == "def f():\n if x:\n return data"
+
+ def test_nested_list_indentation_survives(self):
+ text = (
+ "- item one\n - nested [[1]](http://a) item\n - deeper item [2]"
+ )
+ out = _manager()._strip_citation_markers(text)
+ assert out == ("- item one\n - nested item\n - deeper item")
+
+ def test_table_column_alignment_survives(self):
+ text = (
+ "| Col A | Col B [[1]](http://a) |\n"
+ "|-------|-------|\n"
+ "| val1 | val2 |"
+ )
+ out = _manager()._strip_citation_markers(text)
+ assert out == (
+ "| Col A | Col B |\n|-------|-------|\n| val1 | val2 |"
+ )
+
+
+class TestStripHardening:
+ """Adversarial-gate cases: non-citation content survives, citations
+ are removed whole.
+
+ These were xfail'd known gaps before the strip gained code-span
+ protection, a balanced-parenthesis URL matcher, and citation-shaped
+ link labels.
+ """
+
+ def test_parenthesized_url_leaves_no_residue(self):
+ # The URL matcher tolerates one level of balanced parentheses, so
+ # a Wikipedia "(disambiguation)" URL is consumed whole instead of
+ # stopping at the first ')' and leaving a stray one behind.
+ out = _manager()._strip_citation_markers(
+ "See [[1]](https://en.wikipedia.org/wiki/Foo_(bar)) now."
+ )
+ assert ")" not in out
+
+ def test_code_block_index_not_stripped(self):
+ # Fenced code is not a citation site.
+ out = _manager()._strip_citation_markers("```\nx = arr[1]\n```")
+ assert "arr[1]" in out
+
+ def test_inline_code_index_not_stripped(self):
+ # Neither is inline code.
+ out = _manager()._strip_citation_markers("Use `results[1]` here.")
+ assert "results[1]" in out
+
+ def test_ordinary_markdown_link_survives(self):
+ # A link whose label is not citation-shaped (here: it contains a
+ # space, which no CitationFormatter label ever does) is verbatim
+ # content, not a citation — it must keep label AND destination.
+ text = (
+ "Download [the dataset](https://example.com/data) before analysis."
+ )
+ assert _manager()._strip_citation_markers(text) == text
+
+ def test_citation_starting_a_line_does_not_join_lines(self):
+ # Whitespace consumption is horizontal-only, so a marker at the
+ # start of the next line can never eat the newline(s); the
+ # indentation before it is kept and the separator after it goes.
+ text = "First line\n [[1]](https://example.com) second line"
+ assert (
+ _manager()._strip_citation_markers(text)
+ == "First line\n second line"
+ )
+
+
+class TestRawModeStripsBeforeTruncating:
+ """Raw mode must strip citations BEFORE the 500-char truncation.
+
+ A citation crossing the truncation cutoff is cut into a shape no
+ pattern can match ("[](htt..."), so stripping AFTER truncating can
+ never clean it up — extraction strips each assistant message first.
+ """
+
+ def test_citation_crossing_cutoff_leaves_no_residue(self, mocker):
+ mocker.patch("local_deep_research.config.llm_config.get_llm")
+ long_answer = "a" * 490 + " [[1]](https://example.com/reference) final"
+ messages = [
+ {
+ "role": "user",
+ "content": "summarize",
+ "message_type": "query",
+ },
+ {
+ "role": "assistant",
+ "content": long_answer,
+ "message_type": "response",
+ "research_id": "r1",
+ },
+ ]
+ result = _manager(messages, mode="raw").build_research_context(
+ current_query="more"
+ )
+ past = result["past_findings"]
+ assert "[[" not in past
+ assert "[]" not in past
+ assert "](http" not in past
+ assert past.endswith("final")
+
+
+class TestAdjacentCitationRuns:
+ """Formatter-produced ADJACENT markers must strip as one run.
+
+ ``CitationFormatter._replace_comma_citations`` joins the formatted
+ parts of a comma group with no separator, so
+ ``[1, 2]`` in source-tagged mode renders as
+ ``[local-1][personal-notes-2]`` (both non-linkable) or
+ ``[[web-1]](url)[local-2]`` (mixed). Round 2 of the adversarial gate
+ showed the round-1 ``]``-lookbehind rejecting the second tag, gluing
+ debris to the preceding word; the run consumer in
+ ``chat.citation_stripping`` removes the whole run. Footnote-style
+ period-glued tags (``holds.[local-1]``) strip too — a period is not
+ an identifier character.
+ """
+
+ @pytest.mark.parametrize(
+ "text,expected",
+ [
+ (
+ "Cites [local-1][personal-notes-2] here.",
+ "Cites here.",
+ ),
+ (
+ "Source [[web-1]](https://example.com)[local-2] says.",
+ "Source says.",
+ ),
+ ("This holds.[local-1]", "This holds."),
+ ("Group [1][2][3] chain.", "Group chain."),
+ (
+ "Mix [[a-1]](http://x)[[b-2]](http://y) done.",
+ "Mix done.",
+ ),
+ ],
+ )
+ def test_adjacent_run_fully_stripped(self, text, expected):
+ assert _manager()._strip_citation_markers(text) == expected
+
+
+class TestSingleWordOrdinaryLinksSurvive:
+ """A one-word ordinary Markdown link is not a citation.
+
+ Grounded in CitationFormatter's actual emissions: every hyperlinked
+ citation is ``[[label]](url)``. The only single-bracket links stripped
+ are domain labels naming the link's own host
+ (``[arxiv.org](https://arxiv.org/x)``) — never a bare dotless word.
+ Round 2 of the gate showed the round-1 charset label deleting
+ ``[dataset](...)`` and ``[release-notes](...)`` from assistant prose.
+ """
+
+ @pytest.mark.parametrize(
+ "text",
+ [
+ "Download [dataset](https://example.com/data) before analysis.",
+ "Read [release-notes](https://example.com/releases) now.",
+ "See [Dataset](https://example.com/data) too.",
+ ],
+ )
+ def test_single_word_link_survives(self, text):
+ assert _manager()._strip_citation_markers(text) == text
+
+ @pytest.mark.parametrize(
+ "text",
+ [
+ "Edit [config.yaml](https://github.com/o/r/blob/main/config.yaml) first.",
+ "the [README.md](README.md) file",
+ "Upgrade to [v1.2](https://github.com/o/r/releases/v1.2) now.",
+ "Open [example.com](https://other.org/) x",
+ "Docs at [docs.python.org](https://docs.python.org/3/) here.",
+ ],
+ )
+ def test_dotted_link_text_survives_unless_it_names_its_host(self, text):
+ """A dot in the label is not enough: ``[config.yaml](url)`` is link
+ text. The formatter never emits a single-bracket hyperlink, so a
+ dotted single-bracket label strips only when it is the link's own
+ host as ``CitationFormatter`` labels it."""
+ assert _manager()._strip_citation_markers(text) == text
+
+ @pytest.mark.parametrize(
+ "text,expected",
+ [
+ ("Per [arxiv.org](https://arxiv.org/abs/1) x", "Per x"),
+ ("Per [arxiv.org-2](https://www.arxiv.org/abs/1) x", "Per x"),
+ ("Per [github.com](https://github.com/o/r) x", "Per x"),
+ ],
+ )
+ def test_dotted_link_naming_its_own_host_strips(self, text, expected):
+ assert _manager()._strip_citation_markers(text) == expected
+
+
+class TestCodeProtectionCommonMark:
+ """Fence/span protection must cover CommonMark variants, and a bare
+ ``identifier[...]`` index in prose is not a citation marker.
+
+ Round 2 of the gate showed tilde fences and double-backtick spans
+ losing their brackets, and plain prose ``results[1]`` being stripped.
+ """
+
+ @pytest.mark.parametrize(
+ "text",
+ [
+ "~~~python\nx = results[1]\n~~~",
+ "````\ny = data[2]\n````",
+ "``code `x[1]` here``",
+ "The second item is results[1].",
+ ],
+ )
+ def test_code_shapes_and_prose_index_survive(self, text):
+ assert _manager()._strip_citation_markers(text) == text
+
+
+class TestNestedParenthesisUrl:
+ """URLs with nested balanced parentheses strip whole (round-2 gate).
+
+ The URL matcher balances parens up to ``_URL_MAX_DEPTH`` (well past
+ any real Wikipedia-style URL), so ``Foo_(bar_(baz))`` leaves no
+ ``[](...)`` residue. Past the bound the text is simply not treated as
+ a URL — nothing is mangled.
+ """
+
+ def test_nested_parens_stripped_whole(self):
+ out = _manager()._strip_citation_markers(
+ "See [[1]](https://example.com/Foo_(bar_(baz))) now."
+ )
+ assert out == "See now."
+
+ def test_depth_beyond_bound_is_inert(self):
+ from local_deep_research.chat.citation_stripping import _URL_MAX_DEPTH
+
+ depth = _URL_MAX_DEPTH + 1
+ deep = "(" * depth + "x" + ")" * depth
+ text = f"See [[1]]({deep}) now."
+ assert _manager()._strip_citation_markers(text) == text
+
+
+class TestBracketRunIsLinear:
+ """An unclosed bracket run must not backtrack quadratically.
+
+ The round-1 ``\\[+``-prefixed link regex took ~10s at 20k brackets and
+ ~90s at 60k; the scanner in ``chat.citation_stripping`` skips the
+ whole bracket run in one pass (measured in milliseconds). 60k is far
+ past any linear cost yet impossible for a quadratic matcher, so a
+ generous bound separates the regimes without wall-clock flake.
+ """
+
+ @pytest.mark.parametrize("suffix", ["1", "`code`", "<span>value</span>"])
+ def test_bracket_run_strips_linearly(self, suffix):
+ """Protected spans must not cause failed prefixes to be parsed repeatedly."""
+ text = "[" * 60_000 + suffix
+ start = time.monotonic()
+ out = _manager()._strip_citation_markers(text)
+ elapsed = time.monotonic() - start
+ assert out == text # inert: nothing forms a complete citation
+ assert elapsed < 10.0
+
+
+class TestRoundFiveFormatterAuthorities:
+ @pytest.mark.parametrize(
+ "text",
+ [
+ "See [[LOCALHOST]](http://LOCALHOST/a) now.",
+ "See [[LOCALHOST-1]](http://LOCALHOST/a) now.",
+ ],
+ )
+ def test_uppercase_dotless_authority_citations_strip(self, text):
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == "See now."
+ assert markdown(out) == "<p>See now.</p>"
+
+
+class TestRoundFiveMarkdownStructures:
+ @pytest.mark.parametrize(
+ "text,rendered_fragment",
+ [
+ (
+ "[dataset](/page/[1]/details)",
+ 'href="/page/[1]/details"',
+ ),
+ (
+ "[1]: /page/[2]/details\n\n[x][1]",
+ 'href="/page/[2]/details"',
+ ),
+ (
+ "> [1]: https://example.com\n\n> [x][1]",
+ '<blockquote>\n<p><a href="https://example.com">x</a></p>',
+ ),
+ (
+ "[see [1]](https://example.com)",
+ ">see [1]</a>",
+ ),
+ (
+ "![alt\n[1]](pic.jpg)",
+ 'alt="alt\n[1]" src="pic.jpg"',
+ ),
+ ],
+ )
+ def test_valid_markdown_structures_survive(self, text, rendered_fragment):
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == text
+ assert rendered_fragment in markdown(out)
+
+
+class TestRoundFiveRenderedSeparation:
+ def test_link_before_removed_citation_keeps_word_boundary(self):
+ text = "[left](x)[[1]](u)right"
+
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == "[left](x) right"
+ assert markdown(out) == '<p><a href="x">left</a> right</p>'
+
+
+class TestRoundFiveCodeProtection:
+ @pytest.mark.parametrize(
+ "text,rendered_fragment",
+ [
+ (
+ "Use `first\n[1] second` now.",
+ "<code>first\n[1] second</code>",
+ ),
+ ("```\n[1]", "[1]"),
+ ("<pre>[1]</pre>", "<pre>[1]</pre>"),
+ ],
+ )
+ def test_multiline_and_block_code_survive(self, text, rendered_fragment):
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == text
+ assert rendered_fragment in markdown(out, extensions=["fenced_code"])
+
+
+class TestRoundSixMarkdownCodeComposition:
+ @pytest.mark.parametrize(
+ "text",
+ [
+ "Click [see `code` [1]](https://example.com).",
+ "![alt `x` [1]](pic.jpg)",
+ ],
+ )
+ def test_link_or_image_with_inline_code_survives(self, text):
+ assert _manager()._strip_citation_markers(text) == text
+
+
+class TestRoundSixHtmlLookingInlineCode:
+ @pytest.mark.parametrize(
+ "text,expected",
+ [
+ (
+ "Run `<code>foo` then see [1].\n\nNext paragraph [2] more.",
+ "Run `<code>foo` then see.\n\nNext paragraph more.",
+ ),
+ (
+ "Run `first\n<code>[1]</code>\nlast` then see [2].",
+ "Run `first\n<code>[1]</code>\nlast` then see.",
+ ),
+ ],
+ )
+ def test_html_looking_tag_in_code_cannot_disable_later_stripping(
+ self, text, expected
+ ):
+ assert _manager()._strip_citation_markers(text) == expected
+
+
+class TestRoundSixRemainingMarkdownBlocks:
+ @pytest.mark.parametrize(
+ "text",
+ [
+ "[ref]:\n /page/[1]/details\n\n[x][ref]",
+ "> [ref]:\n> /page/[1]/details\n>\n> [x][ref]",
+ "- [ref]:\n /page/[1]/details\n\n [x][ref]",
+ "- [ref]:\n\t/page/[1]/details\n\n [x][ref]",
+ ],
+ )
+ def test_multiline_relative_reference_destination_survives(self, text):
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == text
+ assert 'href="/page/[1]/details"' in markdown(out)
+
+ @pytest.mark.parametrize("indent", [" ", "\t"])
+ def test_indented_code_after_fence_survives(self, indent):
+ text = f"```\nx\n```\n{indent}[1]"
+
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == text
+ assert markdown(out, extensions=["fenced_code"]).count("<code>") == 2
+
+
+class TestRoundSixMirroredBacktickScaling:
+ def test_mirrored_run_lengths_scale_linearly(self):
+ elapsed = []
+ for size in (800, 1600):
+ half = "x".join("`" * length for length in range(1, size + 1))
+ text = f"{half}\n\n{half}"
+ start = time.monotonic()
+
+ out = _manager()._strip_citation_markers(text)
+
+ elapsed.append(time.monotonic() - start)
+ assert out == text
+
+ assert elapsed[1] < elapsed[0] * 6
+
+
+class TestRoundSevenFormatterTagsAndLinks:
+ @pytest.mark.parametrize(
+ "text,expected",
+ [
+ ("See [123-1] now.", "See [123-1] now."),
+ ("See [[123-1]](https://example.com) now.", "See now."),
+ ],
+ )
+ def test_numeric_collection_tags_strip(self, text, expected):
+ """Bare numeric tags also represent factual ranges; only links disambiguate."""
+ assert _manager()._strip_citation_markers(text) == expected
+
+ @pytest.mark.parametrize(
+ "text",
+ [
+ "x = arr[123-1]",
+ "[label \\] [1]](u)",
+ "![alt \\] [1]](pic.jpg)",
+ ],
+ )
+ def test_array_and_escaped_label_structures_survive(self, text):
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == text
+ assert markdown(out) == markdown(text)
+
+
+class TestRoundSevenCodeAndReferenceBlocks:
+ def test_crlf_blank_paragraph_ends_inline_code_pairing(self):
+ text = "`before\r\n\r\n[1]` after [2]"
+
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == "`before\r\n\r\n` after"
+
+ @pytest.mark.parametrize(
+ "text",
+ [
+ '[ref]: /page\n "title [1]"\n\n[x][ref]',
+ '> [ref]: /page\n> "title [1]"\n>\n> [x][ref]',
+ '- [ref]: /page\n "title [1]"\n\n [x][ref]',
+ '[ref]:\n /page\n "title [1]"\n\n[x][ref]',
+ ],
+ )
+ def test_multiline_reference_titles_survive(self, text):
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == text
+ assert 'title="title [1]"' in markdown(out)
+
+ @pytest.mark.parametrize(
+ "text",
+ [
+ "# heading\n [1]",
+ "---\n [1]",
+ "heading\n---\n [1]",
+ "<div>block</div>\n [1]",
+ "<div>\nblock\n</div>\n [1]",
+ ],
+ )
+ def test_indented_code_after_completed_block_survives(self, text):
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == text
+ assert "<code>[1]" in markdown(out)
+
+
+class TestRoundSevenHtmlContexts:
+ def test_code_tag_text_inside_html_comment_does_not_open_code(self):
+ text = "<!-- mentions <code> only -->\nProse [1] continues."
+
+ assert _manager()._strip_citation_markers(text) == (
+ "<!-- mentions <code> only -->\nProse continues."
+ )
+
+ def test_backtick_in_fence_cannot_mask_later_html_code(self):
+ text = "~~~\n`\n~~~\n<code>[1]</code> `x`"
+
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == text
+ assert "<code>[1]</code>" in markdown(out, extensions=["fenced_code"])
+
+ def test_html_comment_contents_survive_while_following_prose_strips(self):
+ text = "<!-- hidden [1] -->\nProse [2] continues."
+
+ assert _manager()._strip_citation_markers(text) == (
+ "<!-- hidden [1] -->\nProse continues."
+ )
+
+ def test_backtick_in_comment_cannot_mask_later_html_code(self):
+ text = "<!-- ` -->\n<code>[1]</code> `x`"
+
+ assert _manager()._strip_citation_markers(text) == text
+
+ def test_backtick_inside_comment_cannot_mask_later_html_code(self):
+ text = "<!-- ` -->\n<code>[1]</code> `x`"
+
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == text
+
+ def test_backtick_inside_comment_cannot_mask_later_prose(self):
+ text = "<!-- ` -->\nProse [1] `x`"
+
+ assert (
+ _manager()._strip_citation_markers(text) == "<!-- ` -->\nProse `x`"
+ )
+
+
+class TestRoundEightParserContexts:
+ @pytest.mark.parametrize(
+ "text,expected",
+ [
+ ("See [1-1] now.", "See [1-1] now."),
+ ("See [[1-1]](https://example.com) now.", "See now."),
+ ("See [7-42] now.", "See [7-42] now."),
+ ],
+ )
+ def test_single_digit_numeric_collection_tags_strip(self, text, expected):
+ """Without provenance, preserving numeric ranges takes precedence."""
+ assert _manager()._strip_citation_markers(text) == expected
+
+ @pytest.mark.parametrize(
+ "text",
+ [
+ 'Read [see [1]](/page "a title") now.',
+ "Read [see [1]](/page 'a title') now.",
+ 'Read [see [1]](<page target> "a title") now.',
+ "Read [see [1]][ref] now.\n\n[ref]: /page",
+ "Read [see [1]][] now.",
+ ],
+ )
+ def test_titled_and_reference_markdown_links_survive(self, text):
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == text
+ assert markdown(out) == markdown(text)
+
+ @pytest.mark.parametrize(
+ "text",
+ [
+ '<a href="/page/[1]/details">dataset</a>',
+ '<a data-x="[1]">x</a>',
+ '<a data-x="> [1]">x</a>',
+ '<img alt="[1]" src="x">',
+ ],
+ )
+ def test_citation_shapes_inside_html_attributes_survive(self, text):
+ assert _manager()._strip_citation_markers(text) == text
+
+ @pytest.mark.parametrize(
+ "text",
+ [
+ "> [1]",
+ "> ~~~\n> [1]\n> ~~~",
+ "> ```python\n> [1]\n> ```",
+ ],
+ )
+ def test_blockquote_code_blocks_survive(self, text):
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == text
+ assert "[1]" in markdown(out, extensions=["fenced_code"])
+
+
[email protected]("mode", ["raw", "full"])
+def test_factual_ranges_survive_context_assembly(mode):
+ """Removing bare numeric collection tags also erased unambiguous year facts."""
+ messages = [
+ {"role": "user", "content": "Which years were observed?"},
+ {
+ "role": "assistant",
+ "research_id": "synthetic",
+ "content": "The window is [2020-2024]. Evidence [[1]](https://example.com/a).",
+ },
+ ]
+ past = _manager(messages, mode=mode).build_research_context()[
+ "past_findings"
+ ]
+ assert "The window is [2020-2024]." in past
+ assert "[[1]]" not in past
+
+
+def test_summary_citation_cleanup_precedes_output_cap(mocker):
+ """A citation straddling the summary cap must not become an incomplete URL."""
+ from types import SimpleNamespace
+
+ answer = "x" * 1990 + " [[1]](https://example.org/source) final"
+ model = SimpleNamespace(
+ invoke=lambda prompt: SimpleNamespace(content=answer)
+ )
+ mocker.patch(
+ "local_deep_research.config.llm_config.get_llm", return_value=model
+ )
+ manager = _manager(
+ [
+ {
+ "role": "assistant",
+ "research_id": "r",
+ "content": "Prior fact [1].",
+ }
+ ],
+ mode="summary",
+ )
+ result = manager.build_research_context("Tell me more")["past_findings"]
+ assert result == "x" * 1990 + " final"
+
+
+def test_summary_truncation_does_not_rescan_partial_code(mocker):
+ """Cleanup after truncation would mistake the now-unclosed code for prose."""
+ from types import SimpleNamespace
+
+ answer = "`literal [1] " + "x" * 2100 + "`"
+ model = SimpleNamespace(
+ invoke=lambda prompt: SimpleNamespace(content=answer)
+ )
+ mocker.patch(
+ "local_deep_research.config.llm_config.get_llm", return_value=model
+ )
+ manager = _manager(
+ [
+ {
+ "role": "assistant",
+ "research_id": "r",
+ "content": "Prior fact [1].",
+ }
+ ],
+ mode="summary",
+ )
+ result = manager.build_research_context("Tell me more")["past_findings"]
+ assert result.startswith("`literal [1] ")
+ assert len(result) <= manager.CONTEXT_SUMMARY_MAX_CHARS + 3
+
+
+class TestAdversarialGateBoundaryFixes:
+ """Revert-catching tests for the adversarial-gate round of fixes.
+
+ Each test pins one defect the gate reproduced at the PR head: a
+ multiline HTML code opening tag, a non-1 ordered marker splitting a
+ code-span paragraph, outer trims running before citation cleanup,
+ an unclosed fence inside a code element, and word-joins after a
+ protected span.
+ """
+
+ def test_multiline_code_opening_tag_protects_element_contents(self):
+ """The ``>`` of the opening tag sits on the next line.
+
+ Per-line tag scanning never recognized the element, so the code
+ contents were scanned as prose and ``[1, 2]`` was deleted.
+ """
+ text = 'See <code\nclass="py">values = [1, 2]\n</code> end'
+
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == text
+ assert markdown(out) == markdown(text)
+
+ def test_ordered_marker_other_than_one_is_lazy_continuation(self):
+ """CommonMark: only ``1.`` (or a bullet) can interrupt a paragraph.
+
+ ``2.`` used to split the backtick paragraph, leaving both values
+ deleted and stray unpaired backticks behind.
+ """
+ text = "Set `x = [1]\n2. y = [2]` done"
+
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == text
+
+ def test_context_modes_strip_markers_before_outer_trim(self, mocker):
+ """Full mode trimmed indentation before cleanup.
+
+ The trim dedented a leading indented-code line into prose, so its
+ bracket literals were deleted as citations (raw mode never
+ trimmed, so the modes diverged). Summary mode cannot keep a
+ LEADING indented line: ``ProcessingLLMWrapper`` trims every
+ response before the summarizer sees it. The model below returns
+ what that wrapper delivers, and code later in the summary must
+ still be protected while its markers strip.
+ """
+ content = " data = [1, 2, 3]\n keep = [4]\nDone."
+ messages = [
+ {"role": "user", "content": "show the data"},
+ {
+ "role": "assistant",
+ "research_id": "r",
+ "content": content,
+ },
+ ]
+ full = _manager(messages, mode="full").build_research_context("more")[
+ "past_findings"
+ ]
+ assert "data = [1, 2, 3]" in full
+ assert "keep = [4]" in full
+
+ from types import SimpleNamespace
+
+ # Already trimmed, as ProcessingLLMWrapper._normalize_response
+ # (remove_think_tags + strip) delivers it in production.
+ delivered = (
+ "Summary:\n\n data = [1, 2, 3]\n keep = [4]\n\nDone [1]."
+ )
+ model = SimpleNamespace(
+ invoke=lambda prompt: SimpleNamespace(content=delivered)
+ )
+ mocker.patch(
+ "local_deep_research.config.llm_config.get_llm", return_value=model
+ )
+ summary = _manager(messages, mode="summary").build_research_context(
+ "more"
+ )["past_findings"]
+ assert summary == (
+ "Summary:\n\n data = [1, 2, 3]\n keep = [4]\n\nDone."
+ )
+
+ def test_unclosed_fence_inside_code_element_still_strips_afterwards(self):
+ """A fence glyph inside ``<code>`` is element content, not a fence.
+
+ Recognizing it as an unclosed fence used to protect the rest of
+ the document, so the dangling ``[2]`` after the element survived
+ — the exact residue this stripper exists to remove.
+ """
+ text = "<code>\n```\nfmt = [1]\n</code>\nSee [2] here"
+
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == "<code>\n```\nfmt = [1]\n</code>\nSee here"
+
+ @pytest.mark.parametrize(
+ "text,expected",
+ [
+ ("Run <code>x</code>[1]now", "Run <code>x</code> now"),
+ ("q <b>z</b>[7]w tail", "q <b>z</b> w tail"),
+ (
+ "a <code>1</code>[2]b <code>3</code>",
+ "a <code>1</code> b <code>3</code>",
+ ),
+ ],
+ )
+ def test_citation_after_protected_span_retains_separator(
+ self, text, expected
+ ):
+ """One separator survives when a run directly follows a span.
+
+ The tail and per-segment strip calls had no left context, so
+ removing the citation joined the neighbouring rendered words.
+ """
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == expected
+ assert markdown(out) == markdown(expected)
+
+
+class TestAdversarialGateOverlapAndBlockEnds:
+ """Revert-catching tests for the second adversarial-gate round.
+
+ Each test pins one defect the gate reproduced at the PR head: a
+ ``<!--``/backtick inside an HTML tag's quoted attribute (or inside a
+ Markdown link) starting a phantom span that duplicated text and
+ shielded citations, a thematic break line never ending a backtick
+ paragraph, and a setext underline never ending one either.
+ """
+
+ @pytest.mark.parametrize(
+ "text,expected",
+ [
+ ('<a title="<!--">x [1]', '<a title="<!--">x'),
+ ("[x](y<!--z)q [1]", "[x](y<!--z)q"),
+ ],
+ )
+ def test_overlapping_phantom_span_does_not_duplicate_or_shield(
+ self, text, expected
+ ):
+ """A ``<!--`` inside a tag attribute or link destination is inert.
+
+ The phantom comment span started inside the already-emitted
+ verbatim tag/link and ran past it, so the stitcher re-emitted
+ text[start:end] — duplicating the overlap — and protected the
+ ``[1]`` after it.
+ """
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == expected
+ assert markdown(out) == markdown(expected)
+
+ def test_thematic_break_line_ends_backtick_paragraph(self):
+ """``_backtick_runs`` feeds keepends lines; ``---\\n`` never
+ fullmatched the thematic-break regex, so the code span paired
+ across it and kept both markers verbatim.
+ """
+ text = "`a [1]\n***\nb [2]` tail"
+
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == "`a\n***\nb ` tail"
+ assert markdown(out) == markdown("`a\n***\nb ` tail")
+
+ def test_setext_underline_ends_backtick_paragraph(self):
+ """An ``=====`` underline closes the paragraph above it.
+
+ CommonMark never lets a code span pair across a setext heading;
+ the module kept the span, shielding both markers inside it.
+ """
+ text = "`a [1]\n=====\nb [2]` tail"
+
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == "`a\n=====\nb ` tail"
+ assert markdown(out) == markdown("`a\n=====\nb ` tail")
+
+
+class TestAdversarialGateLazyReentryDefinitionsTagFloor:
+ """Revert-catching tests for the third adversarial-gate round.
+
+ Each test pins one defect the gate reproduced at the PR head: a
+ ``>`` re-entry after a lazy continuation line splitting the
+ blockquoted paragraph (deleting a marker inside a real code span),
+ pseudo reference definitions protecting their whole line, and
+ single-letter slug tags (array indexing) deleted from prose.
+ """
+
+ def test_lazy_quote_reentry_keeps_code_span_paragraph(self):
+ """One blockquoted paragraph spans the lazy line and the re-entry.
+
+ CommonMark keeps ``> one `code``, ``lazy [1]`` and ``> end` tail``
+ in a single blockquoted paragraph, so the code span pairs across
+ all three lines and ``[1]`` renders inside it. The old
+ ``depth > previous_depth`` split ended the run at the re-entry,
+ leaving both backticks unpaired and ``[1]`` deleted.
+ """
+ text = "> one `code\nlazy [1]\n> end` tail"
+
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == text
+ assert "<code>code\nlazy [1]\nend</code>" in markdown(out)
+
+ @pytest.mark.parametrize(
+ "text,expected",
+ [
+ ("[Note]: See [1] for details.", "[Note]: See for details."),
+ (
+ "[TL;DR]: summary [1] of the answer [2] follows.",
+ "[TL;DR]: summary of the answer follows.",
+ ),
+ ],
+ )
+ def test_pseudo_reference_definition_strips_markers(self, text, expected):
+ """A space-bearing destination with no valid title is prose.
+
+ ``[Note]: See [1] for details.`` is not a CommonMark reference
+ definition, so it must not protect its whole line verbatim —
+ only genuine definitions (validated destination, optional
+ well-formed title) pass through untouched.
+ """
+ assert _manager()._strip_citation_markers(text) == expected
+
+ def test_glued_short_slug_indexing_survives(self):
+ """``dp[i-1]`` is indexing: the word-glue guard, not a slug-length
+ floor, protects it (a one-letter collection slug is a real tag;
+ see ``TestShortCollectionSlugTags``)."""
+ text = "compare dp[i-1] with dp[i] and x[1][2]"
+
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == text
+ assert markdown(out) == markdown(text)
+
+
+class TestAdversarialGateIndentColumnsSetextSpans:
+ """Revert-catching tests for the fourth adversarial-gate round.
+
+ Each test pins one defect the gate reproduced at the PR head:
+ indented code decided from column zero of the quote-stripped line
+ instead of the containing block's content column, a setext
+ underline missing from the main block-boundary rule, and comment
+ masking computing inline code spans per line so a multi-line
+ span's comment opener latched the comment state.
+ """
+
+ @pytest.mark.parametrize(
+ "text,expected",
+ [
+ # five spaces past the marker: the item's content is code
+ ("- [1]", "- [1]"),
+ ("1. [1]", "1. [1]"),
+ # tabs advance to column 8, six past the content column
+ ("- \t\t[1]", "- \t\t[1]"),
+ # a tab reaching column 4 is top-level indented code
+ (" \t[1]", " \t[1]"),
+ ("> \t[1]", "> \t[1]"),
+ # controls: four spaces after a marker (or a single tab,
+ # which only reaches two past the content column) stay a
+ # paragraph continuation and strip
+ ("- [1]", "-"),
+ ("- [1]", "-"),
+ ("-\t[1]", "-"),
+ (" [1]", ""),
+ ],
+ )
+ def test_indent_measured_from_content_column(self, text, expected):
+ """Indented code is decided against the containing block's column.
+
+ The old check inspected the quote-stripped body from column
+ zero, so a citation CommonMark renders as indented code inside a
+ list item (or after tab indentation) was deleted — leaving only
+ the list marker, or an empty line.
+ """
+ assert _manager()._strip_citation_markers(text) == expected
+
+ @pytest.mark.parametrize(
+ "text",
+ [
+ "Values\n=====\n [1] xs[2]",
+ "Values\n-----\n [1] xs[2]",
+ ],
+ )
+ def test_setext_underline_ends_block_for_indented_code(self, text):
+ """A setext heading underline completes the paragraph's block.
+
+ The line after ``=====`` is genuinely indented code (the
+ heading ends the paragraph), but the main block-boundary rule
+ had no setext clause — only the backtick-run helper did — so
+ the bracketed index expressions below the underline were
+ stripped. The dashes variant already ended the block via the
+ thematic-break rule.
+ """
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == text
+ assert "<code>[1] xs[2]" in markdown(out)
+
+ @pytest.mark.parametrize(
+ "text",
+ [
+ "`span <!-- continues\ncrossing` text\n~~~\n[1]\n~~~",
+ "`same <!-- line` text\n~~~\n[1]\n~~~",
+ ],
+ )
+ def test_multiline_code_span_comment_cannot_mask_fence(self, text):
+ """Comment masking must compute code spans over the whole text.
+
+ The per-line computation missed a span continuing onto the next
+ line, so a ``<!--`` inside it latched the comment state and
+ masked the fence further down; the fenced ``[1]`` was then
+ stripped as prose. A span opened and closed on one line already
+ masked correctly.
+ """
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == text
+ assert "<code>[1]" in markdown(out, extensions=["fenced_code"])
+
+
+class TestAdversarialGateType7CodeAndSpecIndents:
+ """Revert-catching tests for the fifth adversarial-gate round.
+
+ ``<code>`` is a CommonMark HTML block type 7: it cannot interrupt a
+ paragraph, its block ends at the first blank line, and a mid-line or
+ trailing-content occurrence is inline HTML inside the paragraph.
+ The stripper previously latched element state from any ``<code>``
+ mention, disabling stripping (and fence recognition) for the rest
+ of the answer. ``<pre>`` is type 1 and keeps its blank-spanning
+ behaviour. Separately, reference-definition continuation lines may
+ carry indent 0 (the old "at least one column" floor rejected the
+ spec-legal indent-0 destination line, deleting the rendered link
+ and mangling list-item definitions).
+ """
+
+ @pytest.mark.parametrize(
+ "text,expected",
+ [
+ # prose mentioning the literal tag, then a fence, then prose
+ (
+ "Use the <code> tag\n```\nx[1]\n```\nmore [2] text",
+ "Use the <code> tag\n```\nx[1]\n```\nmore text",
+ ),
+ # an unclosed type-7 element ends at the blank line
+ ("<code>\n\nsome [1] text", "<code>\n\nsome text"),
+ # balanced inline pair: the following indented line is a
+ # lazy paragraph continuation, not indented code
+ ("<code>x</code>\n [1]", "<code>x</code>\n"),
+ ],
+ )
+ def test_type7_code_element_cannot_latch_block_state(self, text, expected):
+ """A ``<code>`` mention no longer freezes the rest of the answer.
+
+ Before the fix, any line containing the tag entered element
+ state that survived blank lines, so a later fence was never
+ recognized and trailing citations leaked verbatim.
+ """
+ assert _manager()._strip_citation_markers(text) == expected
+
+ def test_pre_element_still_spans_blank_lines(self):
+ """``<pre>`` is type 1: a blank line does not end its block."""
+ text = "<pre>\n\nsome [1] text"
+
+ assert _manager()._strip_citation_markers(text) == text
+
+ @pytest.mark.parametrize(
+ "text",
+ [
+ # indent-0 destination line (spec-legal continuation)
+ "[1]:\n/url\nlater [1] ref",
+ # lazy destination line inside a list item
+ "- [1]:\n /url\nlater [1] x",
+ ],
+ )
+ def test_multiline_reference_definitions_at_spec_indents(self, text):
+ """Multi-line definitions keep their rendered reference links.
+
+ The old continuation floor rejected these lines, so the label
+ never registered: the later ``[1]`` was stripped (deleting the
+ rendered link) and the list-item definition line was mangled to
+ its marker and colon.
+ """
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == text
+
+
+class TestReferenceContinuationIndentsAndBlockOpeners:
+ """Revert-catching tests for reference-definition continuation lines.
+
+ A definition's destination or title line continues the definition's
+ paragraph, and indented code cannot interrupt a paragraph, so
+ CommonMark sets no indentation limit on that line; a lazy line with
+ fewer ``>`` markers continues it too. The previous three-column cap
+ was measured from column 0, so a list item's content column
+ (``- [ref]:`` then four spaces or a tab), any deeper indent and lazy
+ quote lines were rejected, and the literal ``[1]`` in the rendered
+ destination or title was stripped. Conversely a line that opens a
+ fence or ``<pre>`` element interrupts the paragraph: consuming it as
+ the destination exposed the block's code to stripping.
+ """
+
+ @pytest.mark.parametrize(
+ "text",
+ [
+ "[ref]:\n /page/[1]/details\n\n[x][ref]",
+ "[ref]:\n /page/[1]/details\n\n[x][ref]",
+ "[ref]:\n\t/page/[1]/details\n\n[x][ref]",
+ "- a\n - [ref]:\n /page/[1]/details\n\n [x][ref]",
+ "> [ref]:\n> /page/[1]/details\n>\n> [x][ref]",
+ # lazy continuation line without the ``>`` prefix
+ "> [ref]:\n /page/[1]/details\n\n> [x][ref]",
+ ],
+ )
+ def test_destination_line_at_any_indent_survives(self, text):
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == text
+ assert 'href="/page/[1]/details"' in markdown(out)
+
+ def test_lazy_title_line_survives(self):
+ text = "> [ref]: /url\n'ti [1] tle'\n\n> [x][ref]"
+
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == text
+ assert 'title="ti [1] tle"' in markdown(out)
+
+ def test_deeper_quote_line_does_not_continue_a_definition(self):
+ """A nested ``>`` interrupts the paragraph, so its text is prose."""
+ text = "> [ref]:\n> > /page/[1]/details\n\n> [x][ref]"
+
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == "> [ref]:\n> > /page//details\n\n> [x][ref]"
+
+ @pytest.mark.parametrize(
+ "text",
+ [
+ "[ref]:\n```\ncode [1]\n```",
+ "[ref]:\n~~~\ncode [1]\n~~~",
+ "[ref]:\n<pre>\ncode [1]\n</pre>",
+ "- [ref]:\n ```\n code [1]\n ```",
+ # opener two columns past the list content column: still a
+ # fence or <pre> block whose content sits at that column
+ "- [ref]:\n ```\n code [1]\n ```",
+ "- [ref]:\n <pre>\n code [1]\n </pre>",
+ ],
+ )
+ def test_fence_or_pre_opener_never_continues_a_definition(self, text):
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == text
+
+ def test_code_element_line_still_continues_a_definition(self):
+ """``<code>`` (HTML block type 7) cannot interrupt a paragraph.
+
+ The line is the definition's destination, so the next line is
+ prose and its marker strips.
+ """
+ text = "[ref]:\n<code>\nx [1]\n</code>"
+
+ out = _manager()._strip_citation_markers(text)
+
+ assert out == "[ref]:\n<code>\nx\n</code>"
+
+
+class TestWhitespaceAroundLineLeadingAndTrailingRuns:
+ """Only one separator flanking a stripped run is removed."""
+
+ @pytest.mark.parametrize(
+ "text,expected",
+ [
+ # Indentation of a list item's continuation paragraph is kept;
+ # the separator after the run goes instead.
+ (
+ "1. First item.\n\n [1] Second paragraph.\n2. Next",
+ "1. First item.\n\n Second paragraph.\n2. Next",
+ ),
+ ("a\n\t[1] b", "a\n\tb"),
+ # A run at the start of a line drops the space after it rather
+ # than leaving the next word indented.
+ ("Line one\n[1] Line two", "Line one\nLine two"),
+ ("[1] Opening line", "Opening line"),
+ ],
+ )
+ def test_line_leading_run_keeps_indentation(self, text, expected):
+ assert _manager()._strip_citation_markers(text) == expected
+
+ @pytest.mark.parametrize(
+ "text,expected",
+ [
+ # Keeping the two spaces would create a Markdown hard break.
+ ("Text [1]\nnext", "Text\nnext"),
+ ("Text [[1]](https://a.b/c)\nnext", "Text\nnext"),
+ ("Text [1]\r\nnext", "Text\r\nnext"),
+ ("x\n [1]\ny", "x\n\ny"),
+ ],
+ )
+ def test_line_ending_run_leaves_no_trailing_whitespace(
+ self, text, expected
+ ):
+ out = _manager()._strip_citation_markers(text)
+ assert out == expected
+ assert markdown(out) == markdown(expected)
+
+
+class TestStrippingFailsSafe:
+ """An unexpected stripping error must not fail every later follow-up."""
+
+ @pytest.mark.parametrize("mode", ["raw", "full"])
+ def test_error_carries_text_unstripped(self, mocker, mode):
+ mocker.patch(
+ "local_deep_research.chat.context.strip_prior_turn_citations",
+ side_effect=RuntimeError("boom"),
+ )
+ messages = [
+ {"role": "user", "content": "q"},
+ {
+ "role": "assistant",
+ "research_id": "r",
+ "content": "Fact [1].",
+ },
+ ]
+ past = _manager(messages, mode=mode).build_research_context("more")[
+ "past_findings"
+ ]
+ assert "Fact [1]." in past
+
+ def test_error_in_summary_cleanup_keeps_the_summary(self, mocker):
+ from types import SimpleNamespace
+
+ model = SimpleNamespace(
+ invoke=lambda prompt: SimpleNamespace(content="Summary [1].")
+ )
+ mocker.patch(
+ "local_deep_research.config.llm_config.get_llm", return_value=model
+ )
+ mocker.patch(
+ "local_deep_research.chat.context.strip_prior_turn_citations",
+ side_effect=RuntimeError("boom"),
+ )
+ messages = [
+ {"role": "user", "content": "q"},
+ {
+ "role": "assistant",
+ "research_id": "r",
+ "content": "Fact [1].",
+ },
+ ]
+ past = _manager(messages, mode="summary").build_research_context(
+ "more"
+ )["past_findings"]
+ assert past == "Summary [1]."
+
+
+class TestBareUrlGuard:
+ """Only a citation INSIDE a bare URL is kept; a ``)`` closing a Markdown
+ link destination, or a line break, ends the URL."""
+
+ @pytest.mark.parametrize(
+ "text,expected",
+ [
+ (
+ "Read [the paper](https://x.org)[1].",
+ "Read [the paper](https://x.org).",
+ ),
+ (
+ "[link](http://a.com)[[1]](http://b.com) next",
+ "[link](http://a.com) next",
+ ),
+ ("(https://a.com)[1] now", "(https://a.com) now"),
+ ("Claim https://x.org/a\n[1] more", "Claim https://x.org/a\nmore"),
+ ("a [[1]](https://x.org/a)\n[[2]](https://x.org/b)", "a\n"),
+ ],
+ )
+ def test_citation_after_a_url_strips(self, text, expected):
+ assert _manager()._strip_citation_markers(text) == expected
+
+ @pytest.mark.parametrize(
+ "text",
+ [
+ "see https://x.org/a[1] here",
+ "see https://en.wikipedia.org/wiki/F_(b)[1] here",
+ ],
+ )
+ def test_brackets_inside_a_bare_url_survive(self, text):
+ assert _manager()._strip_citation_markers(text) == text
+
+ def test_url_guard_scans_each_chunk_once(self, monkeypatch):
+ """The guard used to run a ``$``-anchored search over a 4096-char
+ window per candidate (quadratic per token: 16 KB took ~5 s). Bare
+ URLs are now found in one pass per prose chunk, counted here in
+ characters scanned."""
+ from local_deep_research.chat import citation_stripping as cs
+
+ scanned = {"chars": 0}
+ real = cs._bare_url_spans
+
+ def counted(text):
+ scanned["chars"] += len(text)
+ return real(text)
+
+ monkeypatch.setattr(cs, "_bare_url_spans", counted)
+ text = ("http://" * 500 + " " + "[1]," * 150 + "\n") * 4
+ text += "see https://a.org/x[1] and (https://b.org)[2] `c` [3]\n" * 300
+ out = _manager()._strip_citation_markers(text)
+ assert "[2]" not in out and "[3]" not in out
+ assert out.count("https://a.org/x[1]") == 300
+ assert scanned["chars"] <= 3 * len(text)
+
+
+class TestShortCollectionSlugTags:
+ """``_slugify_collection`` maps "AI" -> "ai" and "X" -> "x", and the
+ formatter emits ``[{slug}-{n}]`` for non-http sources, so a 1- or
+ 2-character slug is a real tag."""
+
+ @pytest.mark.parametrize(
+ "text,expected",
+ [
+ ("Alpha [ai-1] beta.", "Alpha beta."),
+ ("Alpha [x-1] beta.", "Alpha beta."),
+ (
+ "Alpha [ai-1][[arxiv-2]](https://arxiv.org/abs/9) beta",
+ "Alpha beta",
+ ),
+ ("Alpha [[ml-3]](https://a.org/x) beta", "Alpha beta"),
+ ],
+ )
+ def test_short_slug_tags_strip(self, text, expected):
+ assert _manager()._strip_citation_markers(text) == expected
+
+
+class TestSingleBracketNumericLinksSurvive:
+ """The formatter never emits a single-bracket numeric link, and the
+ summarizer reads a transcript whose assistant citations are already
+ stripped; ``[1998](url)`` is a year, ``[1](#step-1)`` an anchor."""
+
+ @pytest.mark.parametrize(
+ "text",
+ [
+ "Founded in [1998](https://en.wikipedia.org/wiki/1998).",
+ "Step [1](#step-1) then",
+ "Top [10](https://x.org/top10) list",
+ "See [1, 2](https://x.org/a) too",
+ ],
+ )
+ def test_numeric_label_link_survives(self, text):
+ assert _manager()._strip_citation_markers(text) == text
+
+
+class TestLineLeadingBibliographyEntries:
+ """``[1]: ...`` at the start of a line's content is a bibliography
+ entry, judged per line through container and heading markers, not by
+ the marker's offset in the prose chunk."""
+
+ @pytest.mark.parametrize(
+ "text",
+ [
+ "Text\n[1]: x",
+ "Sources\n- [1]: Smith 2020",
+ "## [1]: Intro",
+ "> 1. [2]: quoted entry",
+ "Intro\n * [3]: indented entry",
+ ],
+ )
+ def test_line_leading_entry_survives(self, text):
+ assert _manager()._strip_citation_markers(text) == text
+
+ def test_mid_line_colon_marker_still_strips(self):
+ out = _manager()._strip_citation_markers("Note [1]: mid-line")
+ assert out == "Note: mid-line"
+
+
+class TestUnspacedScripts:
+ """CJK text is written without spaces, so a glued marker is a
+ citation, and removing it must not insert a space."""
+
+ @pytest.mark.parametrize(
+ "text,expected",
+ [
+ ("東京は首都です[1]。次", "東京は首都です。次"),
+ ("東京は首都です [1]。次", "東京は首都です。次"),
+ ("東京[1]次", "東京次"),
+ ("東京[[1]](https://a.org/x)次", "東京次"),
+ ],
+ )
+ def test_cjk_glued_citation_strips(self, text, expected):
+ assert _manager()._strip_citation_markers(text) == expected
+
+
+class TestReferenceLabelsHoldNoBrackets:
+ def test_citation_after_bracketed_text_strips(self):
+ """``[Table 2][[1]](url)`` is not a full reference link (a reference
+ label cannot contain ``[``), so the citation strips."""
+ out = _manager()._strip_citation_markers(
+ "Shown in [Table 2][[1]](https://a.org/x) here."
+ )
+ assert out == "Shown in [Table 2] here."
+
+ def test_real_full_reference_link_survives(self):
+ text = "See [text][1] ok\n\n[1]: https://a.org/x"
+ assert _manager()._strip_citation_markers(text) == text
+
+
+class TestSeparatorBetweenWords:
+ @pytest.mark.parametrize(
+ "text,expected",
+ [
+ ("Claim.[[1]](https://a.com)Next", "Claim. Next"),
+ ("word[[1]](https://a.com)next", "word next"),
+ ("x</b>[[1]](https://a.com)y", "x</b> y"),
+ # Path-like and punctuation joins stay joined.
+ ("/page/[[1]](https://a.com)/x", "/page//x"),
+ ("Founded [[1]](https://a.com).", "Founded."),
+ ("(see[[1]](https://a.com))", "(see)"),
+ ],
+ )
+ def test_space_only_between_words(self, text, expected):
+ assert _manager()._strip_citation_markers(text) == expected
+
+
+class TestDoubleBracketLabelsTheFormatterEmits:
+ """Every label shape ``CitationFormatter`` can put inside ``[[...]]``.
+
+ ``_slugify_title``/``_slugify_collection`` turn "2024-01-15", "1.2.3"
+ and "2023/2024" into all-digit hyphenated slugs, and the id modes
+ append "-N"; destinations keep Unicode spaces (only ASCII ones are
+ escaped), may be long, and may nest balanced parentheses.
+ """
+
+ @pytest.mark.parametrize(
+ "text,expected",
+ [
+ ("Claim [[2024-01-15]](/library/document/abc).", "Claim."),
+ ("Claim [[1-2-3]](/library/document/abd).", "Claim."),
+ ("Claim [[2024-01-15-1]](/library/document/abc).", "Claim."),
+ (
+ "Text [[2023-2024-26]](https://shop.example.com/a) more",
+ "Text more",
+ ),
+ (
+ "Claim [[arxiv-1]](https://arxiv.org/abs/1)[local-12]"
+ "[[2023-2024-26]](https://shop.example.com/a)[local-27].",
+ "Claim.",
+ ),
+ ("Claim [[20]](https://example.com/a\xa0b).", "Claim."),
+ ("Claim [[21]](https://example.jp/a\u3000b).", "Claim."),
+ (
+ "Claim [[22]](https://example.com/?q=" + "a" * 4100 + ").",
+ "Claim.",
+ ),
+ (
+ "Claim [[23]](https://example.com/((((((((((x))))))))))).",
+ "Claim.",
+ ),
+ ],
+ )
+ def test_formatter_label_strips(self, text, expected):
+ assert _manager()._strip_citation_markers(text) == expected
+
+ def test_bare_numeric_range_still_survives(self):
+ """Only the hyperlinked form widened: a bare ``[2020-2024]`` has no
+ provenance and stays prose."""
+ text = "The window is [2020-2024] and [1-2-3]."
+ assert _manager()._strip_citation_markers(text) == text
+
+
+class TestLenticularMarkersIgnoreWordGlue:
+ """Nobody indexes with ``x【1】``, so the word-glue guard is skipped."""
+
+ @pytest.mark.parametrize(
+ "text,expected",
+ [
+ ("Founded in 2020【1】.", "Founded in 2020."),
+ ("Paris【1, 2】 is", "Paris is"),
+ ("capital【1】【2】.", "capital."),
+ ],
+ )
+ def test_glued_lenticular_strips(self, text, expected):
+ assert _manager()._strip_citation_markers(text) == expected
+
+
+class TestIndexingGlueAndTradeOffs:
+ def test_latex_subscript_survives(self):
+ text = "$x_{[i-1]}$ and $y_{[1]}$"
+ assert _manager()._strip_citation_markers(text) == text
+
+ def test_spaced_short_slug_is_stripped_documented_trade_off(self):
+ """A one-letter collection slug is a real tag (``X`` -> ``[x-1]``),
+ so an UNGLUED ``[i-1]`` cannot be told apart from one and strips;
+ only the glued ``dp[i-1]`` form is protected."""
+ out = _manager()._strip_citation_markers(
+ "compare dp [i-1] with dp[i-1]"
+ )
+ assert out == "compare dp with dp[i-1]"
+
+
+class TestSeparatorClosingPunctuation:
+ """A space replaces a glued run only between a word (or punctuation
+ that ends one) and a following word."""
+
+ @pytest.mark.parametrize(
+ "text,expected",
+ [
+ # ``*`` / quotes count only when they close a span.
+ ("**[[1]](https://a.com)Smith et al.**", "**Smith et al.**"),
+ ("**bold**[[1]](https://a.com)Next", "**bold** Next"),
+ (
+ 'He said "yes"[[1]](https://a.com)and left',
+ 'He said "yes" and left',
+ ),
+ ('"[[1]](https://a.com)Quote"', '"Quote"'),
+ # ``.`` between digits is a decimal point.
+ ("3.[[1]](https://a.com)5", "3.5"),
+ ("Claim.[[1]](https://a.com)Next", "Claim. Next"),
+ # A non-word, non-closing character before: stays joined.
+ ("/page/[[1]](https://a.com)x", "/page/x"),
+ # ``_`` after the run: an emphasis closer, not a word.
+ ("_word[[1]](https://a.com)_ end", "_word_ end"),
+ # A script written without spaces after the run: no space.
+ ("Claim.[[1]](https://a.com)東京", "Claim.東京"),
+ ("word[[1]](https://a.com)東京", "word東京"),
+ ],
+ )
+ def test_separator(self, text, expected):
+ assert _manager()._strip_citation_markers(text) == expected
+
+
+class TestSeparatorBeforeLinkOrTag:
+ """A following ``[`` (link label) or ``<`` (inline tag) renders as the
+ next word, so it gets a space after a word or a closing tag."""
+
+ @pytest.mark.parametrize(
+ "text,expected",
+ [
+ ("word[[1]](u)[dataset](v)", "word [dataset](v)"),
+ ("<b>x</b>[[1]](u)[y](z)", "<b>x</b> [y](z)"),
+ (
+ "<span>word</span>[[1]](u)<span>next</span>",
+ "<span>word</span> <span>next</span>",
+ ),
+ ("word[[1]](u)<b>next</b>", "word <b>next</b>"),
+ # Paths, an opening tag and unspaced scripts stay joined.
+ ("/a/[[1]](u)[b](c)", "/a/[b](c)"),
+ ("<span>[[1]](u)<b>x</b>", "<span><b>x</b>"),
+ ("東京[[1]](u)[x](y)", "東京[x](y)"),
+ ],
+ )
+ def test_separator_before_link_or_tag(self, text, expected):
+ assert _manager()._strip_citation_markers(text) == expected
+
+
+class TestParenthesesHoldingOnlyACitation:
+ @pytest.mark.parametrize(
+ "text,expected",
+ [
+ ("Claim ([[1]](https://a.com)).", "Claim."),
+ ("Claim ([1][2]) next", "Claim next"),
+ ("Claim ([1]) next", "Claim next"),
+ # Glued to a word, the parentheses may be a call: kept.
+ ("f([[1]](https://a.com)) x", "f() x"),
+ ],
+ )
+ def test_empty_parentheses_are_removed(self, text, expected):
+ assert _manager()._strip_citation_markers(text) == expected
diff --git a/tests/chat/test_citation_stripping_markdown_boundaries.py b/tests/chat/test_citation_stripping_markdown_boundaries.py
new file mode 100644
index 0000000000..abd714dedf
--- /dev/null
+++ b/tests/chat/test_citation_stripping_markdown_boundaries.py
@@ -0,0 +1,168 @@
+"""Preserve literal Markdown while removing citations outside its boundaries."""
+
+import pytest
+
+from local_deep_research.chat.citation_stripping import (
+ strip_prior_turn_citations,
+)
+from local_deep_research.text_optimization.citation_formatter import (
+ CitationFormatter,
+ CitationMode,
+)
+
+
+ "text,expected",
+ [
+ (r"Literal \[1] and fact [2].", r"Literal \[1] and fact."),
+ (
+ r"Escaped \`tick [1] and `code [2]` fact [3].",
+ r"Escaped \`tick and `code [2]` fact.",
+ ),
+ ("[1][manual]\n\n[manual]: /guide", "[1][manual]\n\n[manual]: /guide"),
+ ("[1][2]\n\n[2]: /guide", "[1][2]\n\n[2]: /guide"),
+ ("Fact [1][2].", "Fact."),
+ (
+ "- ```\n [1]\n ```\n\nFact [2].",
+ "- ```\n [1]\n ```\n\nFact.",
+ ),
+ (
+ "- item\n\n ```python\n values = [1]\n ```\n\nFact [2].",
+ "- item\n\n ```python\n values = [1]\n ```\n\nFact.",
+ ),
+ (
+ "10. ```\n code [1]\n ```\n\nFact [2].",
+ "10. ```\n code [1]\n ```\n\nFact.",
+ ),
+ (
+ "> ```\n> code [1]\n\nFact [2].",
+ "> ```\n> code [1]\n\nFact.",
+ ),
+ ],
+)
+def test_literals_and_container_boundaries(text, expected):
+ """Catches stripping before escape/reference protection or across code boundaries."""
+ assert strip_prior_turn_citations(text) == expected
+
+
+ "url",
+ [
+ "http://example.com:8080/a",
+ "http://10.0.0.1/a",
+ "http://10.0.0.1:8080/a",
+ "http://[::1]/a",
+ "http://[::1]:8080/a",
+ ],
+)
+ "mode",
+ [
+ CitationMode.DOMAIN_ID_HYPERLINKS,
+ CitationMode.DOMAIN_ID_ALWAYS_HYPERLINKS,
+ ],
+)
+def test_formatter_authority_ids_are_removed(url, mode):
+ """Two same-authority sources make the conditional formatter emit ID suffixes."""
+ sources = [
+ {"index": 1, "title": "First", "url": url},
+ {"index": 2, "title": "Second", "url": url + "b"},
+ ]
+ formatted = CitationFormatter(mode).apply_inline_hyperlinks(
+ "See [1, 2] now.", sources
+ )
+ assert formatted != "See [1, 2] now."
+ assert strip_prior_turn_citations(formatted) == "See now."
+
+
[email protected]("ticks", [2, 3, 4])
+def test_escaped_tick_leaves_shorter_code_opener(ticks):
+ """An escape consumes one backtick, not the entire run before inline code."""
+ literal = "\\" + "`" * ticks + "code [1]" + "`" * (ticks - 1)
+ assert (
+ strip_prior_turn_citations(literal + " fact [2].") == literal + " fact."
+ )
+
+
+ "example",
+ ["```\n[2]: /url\n```", "<!--\n[2]: /url\n-->", "<pre>\n[2]: /url\n</pre>"],
+)
+def test_code_reference_definitions_do_not_change_prose_citations(example):
+ """Reference-looking code is not a real definition for an adjacent marker run."""
+ assert (
+ strip_prior_turn_citations(example + "\nFact [1][2].")
+ == example + "\nFact."
+ )
+
+
+ "text,expected",
+ [
+ # The heading / list item ends the paragraph, so the first
+ # backtick never closes and ``[1]`` is prose. It ends its line, so
+ # the space before it goes with it (no trailing whitespace is left
+ # behind; see the module docstring's separator rule).
+ (
+ "`literal [1]\n# heading `code [2]` fact [3].",
+ "`literal\n# heading `code [2]` fact.",
+ ),
+ (
+ "`literal [1]\n- item `code [2]` fact [3].",
+ "`literal\n- item `code [2]` fact.",
+ ),
+ (
+ "- > ```\n > values = [1]\n > ```\n\nFact [2].",
+ "- > ```\n > values = [1]\n > ```\n\nFact.",
+ ),
+ (
+ "> - > ```\n> > values = [1]\n> > ```\n\nFact [2].",
+ "> - > ```\n> > values = [1]\n> > ```\n\nFact.",
+ ),
+ ("<!--\n```\n-->\n\nClaim [1].", "<!--\n```\n-->\n\nClaim."),
+ ("```\n<!--\n```\n\nClaim [1].", "```\n<!--\n```\n\nClaim."),
+ ("`<!--`\nClaim [1].", "`<!--`\nClaim."),
+ (
+ '<span title="<pre>">literal</span>\n\nFact [[1]](u).\n\nMore [2].',
+ '<span title="<pre>">literal</span>\n\nFact.\n\nMore.',
+ ),
+ (
+ '<code title="a > b">literal [1]</code> Fact [2].',
+ '<code title="a > b">literal [1]</code> Fact.',
+ ),
+ ("Read [1] now.\n\n[1]: /manual", "Read [1] now.\n\n[1]: /manual"),
+ (
+ 'Read [1](/manual "the guide") now.',
+ 'Read [1](/manual "the guide") now.',
+ ),
+ ("Read [1](<manual page>) now.", "Read [1](<manual page>) now."),
+ (
+ "Fact [local-1][local-2][local-3] and code `x`.",
+ "Fact and code `x`.",
+ ),
+ ],
+)
+def test_composed_markdown_boundaries(text, expected):
+ """Catches cross-block delimiters, nested containers, and literal HTML tags."""
+ assert strip_prior_turn_citations(text) == expected
+
+
[email protected]("suffix", ["`code`", "<span>value</span>"])
+def test_successful_runs_have_bounded_matcher_work(monkeypatch, suffix):
+ """Revisiting each recognized suffix makes the number of lookups quadratic."""
+ from local_deep_research.chat import citation_stripping as scanner
+
+ original = scanner._match_citation_token
+ calls = 0
+
+ def counted(text, position):
+ nonlocal calls
+ calls += 1
+ return original(text, position)
+
+ monkeypatch.setattr(scanner, "_match_citation_token", counted)
+ markers = 600
+ result = scanner.strip_prior_turn_citations("[1] " * markers + suffix)
+ assert "[1]" not in result
+ assert suffix in result
+ assert calls <= 10 * markers
diff --git a/tests/chat/test_citation_stripping_round3.py b/tests/chat/test_citation_stripping_round3.py
new file mode 100644
index 0000000000..d75829c865
--- /dev/null
+++ b/tests/chat/test_citation_stripping_round3.py
@@ -0,0 +1,86 @@
+from markdown import markdown
+import pytest
+
+from local_deep_research.chat.citation_stripping import (
+ strip_prior_turn_citations,
+)
+from local_deep_research.text_optimization.citation_formatter import (
+ CitationFormatter,
+ CitationMode,
+)
+
+
+ "url",
+ [
+ "http://example.com:8080/a",
+ "http://EXAMPLE.COM:8080/a",
+ "http://[::1]:8080/a",
+ ],
+)
+def test_formatter_port_label_is_stripped_when_source_has_authority(
+ url: str,
+) -> None:
+ document = f"See [1] now.\n\n## Sources\n\n[1] Example\nURL: {url}"
+ formatted = CitationFormatter(
+ mode=CitationMode.DOMAIN_HYPERLINKS
+ ).format_document(document)
+ text = formatted.split("\n\n## Sources", maxsplit=1)[0]
+
+ result = strip_prior_turn_citations(text)
+
+ assert result == "See now."
+
+
+ "text",
+ [
+ "![1](image.jpg)",
+ "[1]: https://en.wikipedia.org",
+ "Go to https://example.com/page/[1]/details here",
+ ],
+)
+def test_numeric_markdown_and_url_path_survive_when_not_citations(
+ text: str,
+) -> None:
+ result = strip_prior_turn_citations(text)
+
+ assert result == text
+
+
+ ("text", "expected", "rendered"),
+ [
+ (
+ "<span>word</span> [[1]](u)<span>next</span>",
+ "<span>word</span> <span>next</span>",
+ "<p><span>word</span> <span>next</span></p>",
+ ),
+ (
+ "word [[1]](u)[dataset](v)",
+ "word [dataset](v)",
+ '<p>word <a href="v">dataset</a></p>',
+ ),
+ ("text [1]more", "text more", "<p>text more</p>"),
+ ],
+)
+def test_removed_citation_keeps_rendered_text_separated(
+ text: str, expected: str, rendered: str
+) -> None:
+ result = strip_prior_turn_citations(text)
+
+ assert result == expected
+ assert markdown(result) == rendered
+
+
[email protected]("marker", ["`", "~"])
+def test_fence_longer_than_previous_limit_is_fully_protected(
+ marker: str,
+) -> None:
+ fence = marker * 33
+ shorter = marker * 32
+ text = f"{fence}\n[1] [[2]](u)\n{shorter}\n[3]\n{fence}"
+
+ result = strip_prior_turn_citations(text)
+
+ assert result == text
diff --git a/tests/chat/test_citation_stripping_round4.py b/tests/chat/test_citation_stripping_round4.py
new file mode 100644
index 0000000000..f6ec590e5f
--- /dev/null
+++ b/tests/chat/test_citation_stripping_round4.py
@@ -0,0 +1,96 @@
+from markdown import markdown
+import pytest
+
+from local_deep_research.chat.citation_stripping import (
+ strip_prior_turn_citations,
+)
+from local_deep_research.text_optimization.citation_formatter import (
+ CitationFormatter,
+ CitationMode,
+)
+
+
+ "url",
+ [
+ "http://10.0.0.1/a",
+ "http://10.0.0.1:8080/a",
+ ],
+)
+def test_formatter_ipv4_label_is_stripped(url: str) -> None:
+ document = f"See [1] now.\n\n## Sources\n\n[1] Example\nURL: {url}"
+ formatted = CitationFormatter(
+ mode=CitationMode.DOMAIN_HYPERLINKS
+ ).format_document(document)
+ text = formatted.split("\n\n## Sources", maxsplit=1)[0]
+
+ result = strip_prior_turn_citations(text)
+
+ assert result == "See now."
+ assert markdown(result) == "<p>See now.</p>"
+
+
+ "text",
+ [
+ "[dataset](https://example.com/page/[1]/details)",
+ "<https://example.com/page/[1]/details>",
+ "(https://example.com/page/[1]/details)",
+ '<a href="https://example.com/page/[1]/details">dataset</a>',
+ ],
+)
+def test_numeric_marker_in_punctuation_prefixed_url_survives(text: str) -> None:
+ result = strip_prior_turn_citations(text)
+
+ assert result == text
+ assert markdown(result) == markdown(text)
+
+
+ "text",
+ [
+ "![Figure 1 [1]](pic.jpg)",
+ "![some [1] text](pic.jpg)",
+ ],
+)
+def test_numeric_marker_nested_in_image_alt_text_survives(text: str) -> None:
+ result = strip_prior_turn_citations(text)
+
+ assert result == text
+ assert markdown(result) == markdown(text)
+
+
+ ("text", "expected"),
+ [
+ ("word[[1]](u)[dataset](v)", "word [dataset](v)"),
+ ("word[[1]](u)more", "word more"),
+ (
+ "<span>word</span>[[1]](u)<span>next</span>",
+ "<span>word</span> <span>next</span>",
+ ),
+ ],
+)
+def test_adjacent_citation_removal_separates_rendered_words(
+ text: str, expected: str
+) -> None:
+ result = strip_prior_turn_citations(text)
+
+ assert result == expected
+ assert markdown(result) == markdown(expected)
+
+
+ "text",
+ [
+ " [1]\n [2]",
+ "\t[1]",
+ "<code>[1]</code>",
+ "<code>first [1]\nsecond [2]</code>",
+ ],
+)
+def test_commonmark_code_content_preserves_numeric_brackets(text: str) -> None:
+ result = strip_prior_turn_citations(text)
+
+ assert result == text
+ assert markdown(result) == markdown(text)