~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
fix(chat): strip prior-turn citations from follow-up context (format-aware) (#4358)
LearningCircuit5c571d8eeeb7
diff --git a/changelog.d/+chat-followup-citation-stripping.bugfix.md b/changelog.d/+chat-followup-citation-stripping.bugfix.mdnew file mode 100644index 0000000000..691751d8c6--- /dev/null+++ b/changelog.d/+chat-followup-citation-stripping.bugfix.md@@ -0,0 +1 @@+**Cleaner follow-up context in Chat Mode** — Citation markers from earlier turns (e.g. `[1]`, `[1, 2]`, `[[1]](url)`, `[[arxiv.org-1]](url)`, `[[2024-01-15]](url)`, the plain source-tag form `[local-1]` / `[ai-2]` / `[personal-notes-3]`, `【1】`) are now stripped from the prior-conversation context fed to a follow-up. They reference the previous turn's source list, which the follow-up research renumbers against its own sources, so leaving them in could dangle or collide with the new answer's citations. A single-bracket link is stripped only when its label is the link's own host (`[arxiv.org](https://arxiv.org/abs/…)`); other links, such as `[1998](url)`, `[1](#step-1)`, `[config.yaml](url)` or `[the dataset](url)`, are kept, as are brackets inside a bare URL, code blocks and inline code. In `full` mode only the assistant's earlier answers are cleaned and your own messages are passed on verbatim; `raw` mode carries only the assistant's earlier research answers (cleaned) and never included your messages; in the default `summary` mode your messages reach the summarizer verbatim, but a citation-shaped string you typed is removed if the summary repeats it.diff --git a/src/local_deep_research/chat/citation_stripping.py b/src/local_deep_research/chat/citation_stripping.pynew file mode 100644index 0000000000..726df67905--- /dev/null+++ b/src/local_deep_research/chat/citation_stripping.py@@ -0,0 +1,1849 @@+"""Strip prior-turn citation markers from assistant-authored chat text.++Follow-up research renumbers citations against its own freshly-found sources,+so a marker carried over from a prior turn's answer dangles or collides. The+recognized shapes are based on what ``CitationFormatter``+(text_optimization/citation_formatter.py) writes into a synthesized answer:++- ``number_hyperlinks``: ``[[1]](url)``; comma groups re-emit as+ adjacent links with no separator+- ``domain_hyperlinks``: ``[[arxiv.org]](url)``, ``[[slug]](url)``+- ``domain_id_hyperlinks`` (+always): ``[[arxiv.org-1]](url)``+- ``source_tagged_hyperlinks``: ``[[tag-N]](url)`` when the source URL is+ http(s), PLAIN ``[local-1]`` /+ ``[personal-notes-3]`` otherwise; comma+ groups concatenate the two with no+ separator (``[[web-1]](url)[local-2]``).+ A collection slug can be one character+ (``AI`` -> ``[ai-1]``, ``X`` ->+ ``[x-1]``), so slug length proves+ nothing; ``arr[i-1]`` survives through+ the word-glue guard below instead+- ``no_hyperlinks``: ``[1]``, ``[1, 2]``+- lenticular (some models): ``【1】``, ``【1, 2】``++Every hyperlinked citation the formatter emits uses DOUBLE brackets. One+single-bracket shape is also stripped, for a model that re-renders a+citation with one bracket pair: a domain label that names the link's OWN+host as ``CitationFormatter`` would label it+(``[arxiv.org](https://arxiv.org/x)``,+``[arxiv.org-1](https://www.arxiv.org/x)``). Any other single-bracket+link survives — numeric labels included, since those are years, ranks+and anchors far more often than citations (``[1998](url)``,+``[1](#step-1)``, ``[10](url)``) — as do ``[config.yaml](url)``,+``[README.md](README.md)``, ``[v1.2](url)``,+``[example.com](https://other.org/)`` and ``[dataset](url)``. Double-bracket+labels match case-insensitively: ``CitationFormatter _extract_domain``+preserves URL case, so an uppercase one-label authority+(``[[LOCALHOST]](http://LOCALHOST/a)``, ``[[LOCALHOST-1]](...)``) strips+too. Inside ``[[...]](url)`` an all-digit hyphenated label strips too+(``[[2024-01-15]]``, ``[[1-2-3]]``, ``[[2023-2024-26]]``): the title and+collection slugs produce them. A destination may hold Unicode spaces+(only ASCII spaces end one, as in CommonMark), be up to 16384 characters+long and nest balanced parentheses 16 deep.++Bare numeric ranges such as ``[2020-2024]`` are preserved: without source+provenance they cannot be distinguished from numeric collection tags.+Hyperlinked numeric collection tags remain recognizable. A consequence:+a bare numeric tag (``[2023-2024-27]``) stays, and so does a bare token+glued right after it (``[9-4][5]``), which reads as an index chain.++The scanner bounds citation lookups and advances over failed bracket runs.+Text is segmented as:++1. fenced code blocks — an unclosed fence ends with its containing+ list/blockquote or document — and indented code blocks: emitted+ verbatim (a fence glyph inside an open HTML code element below is+ element content, never a fence). "Indented" is measured from the+ content column of the containers opened on the same line — a list+ marker's content sits one space past the marker, and tabs advance to+ the next multiple-of-4 column — so ``- [1]`` and `` \t[1]`` are+ code while ``- [1]`` is a paragraph continuation that strips. A+ later line of a list item is measured from column 0, so a paragraph+ indented four or more columns after a blank line is kept verbatim+ as indented code even inside the item;+2. HTML ``<pre>``/``<code>`` elements (nesting-aware, may span lines) and+ link reference definitions (``[label]: dest``, optionally inside a+ blockquote): emitted verbatim — the destination is validated per+ CommonMark (``<...>`` or one whitespace-free token, optionally a+ well-formed title), so a pseudo definition whose "destination" is+ prose (``[Note]: See [1] for details.``) has its markers stripped.+ The destination and the title may each sit on a following line at any+ indentation (a lazy line included) unless that line opens a fence or+ ``<pre>``;+3. prose regions: CommonMark inline code spans (a run of N backticks+ closes at the next run of exactly N backticks, CommonMark-style, and+ may span lines within a paragraph — where only a bullet or an+ ordered marker numbered 1 can interrupt the paragraph, and a ``>``+ re-entry after a lazy continuation line is still that paragraph) and+ complete+ Markdown links/images — relative destinations and labels that merely+ LOOK citation-shaped (``[see [1]](url)``, ``![alt\n[1]](pic.jpg)``)+ included — are kept verbatim; the remaining prose is scanned left to+ right for citation *runs*: a run is one or more adjacent citation+ tokens (``[local-1][personal-notes-2]``, ``[[web-1]](url)[local-2]``,+ ``[1][2]``), optionally separated by horizontal whitespace. The run is+ removed together with ONE of its flanking horizontal-whitespace+ separators: mid-line, the whitespace before it (one separator is kept+ when a word or punctuation other than closing punctuation follows; a+ glued run is replaced by one space only when a word follows it and+ a word or word-ending punctuation precedes it — see ``_joins_words``+ — so ``Claim.[[1]](url)Next`` becomes ``Claim. Next`` while+ ``3.[[1]](url)5``, ``**[[1]](url)Bold**`` and ``/a/[1]/b`` stay+ joined, and no space is added next to text in a script written+ without spaces, such as CJK; parentheses left empty mid-line, as in+ ``Claim ([[1]](url)).``, are removed with the space before them); at the+ end of a line, the whitespace before it; at the start of a line, the+ whitespace after it, so indentation before the run (a list item's+ continuation paragraph) is kept verbatim and the following text is not+ re-indented into a code block. No other whitespace is touched.++A BARE bracket token (no URL) preceded by a word character, ``]`` or ``{``+is an indexing expression, not a citation: ``results[1]``, ``arr[i-1]``,+``x[1][2]`` and the LaTeX subscript ``x_{[i-1]}`` survive. Lenticular+``【1】`` is never an index and strips even when glued (``2020【1】``). Letters of scripts written without spaces (CJK,+kana, Thai, ...) are not word glue: ``首都です[1]`` strips. Hyperlinked+tokens strip regardless of the preceding character — the formatter+rewrites ``word[1]`` into ``word[[1]](url)`` when the source has a URL,+and that must not leak. A preceding ``.`` is not a guard: footnote-style+``holds.[local-1]`` strips.++Two more positions keep a token: inside a bare URL (``https://x.org/a[1]``;+the URL ends at whitespace, ``<``, ``>``, a quote, or a ``)`` with no+matching ``(`` in it, such as the one closing a Markdown link+destination), and a bare token followed by ``:`` that opens a line's+content after only indentation, blockquote, list or ATX-heading markers+(``[1]: Smith 2020``, ``- [1]: ...``, ``## [1]: ...``) — a bibliography+entry. A full reference link ``[text][label]`` is kept only when the+second label holds no unescaped bracket, so ``[Table 2][[1]](url)`` is+text followed by a citation.+"""++import re+from bisect import bisect_right+from ipaddress import ip_address+from urllib.parse import urlsplit++# Bounds that keep every scan linear on adversarial input. Real labels and+# URLs are far below them; past a bound the text simply is not a citation.+_LABEL_MAX = 256 # longest label text scanned between brackets+_LINK_LABEL_MAX = 1000 # longest link label scanned for verbatim links+_URL_MAX = 16384 # longest URL scanned after "("+_URL_MAX_DEPTH = 16 # deepest balanced-paren nesting accepted in a URL++# Citation-shaped labels (validated with fullmatch on the bracketed text).+_NUM_LIST_RE = re.compile(r"\d+(?:\s*,\s*\d+)*") # "1", "1, 2"+_TOKEN_RE = re.compile(+ r"[A-Za-z0-9][A-Za-z0-9._-]*"+) # "arxiv.org", "LOCALHOST"+# Plain tag form ``<slug>-N``. ``_slugify_collection`` can emit a slug of+# any length ("AI" -> "ai", "X" -> "x"), so no length floor applies;+# ``arr[i-1]`` indexing is protected by the word-glue guard instead.+_TAG_ID_RE = re.compile(r"[a-z0-9][a-z0-9._-]*-\d+") # "x-1", "local-1"+# All-digit hyphenated label, accepted only inside ``[[...]](url)``:+# ``_slugify_title``/``_slugify_collection`` turn "2024-01-15", "1.2.3" and+# "2023/2024" into "2024-01-15", "1-2-3" and "2023-2024", and the id modes+# append "-N". A BARE ``[2020-2024]`` stays prose (see module docstring).+_NUMERIC_TAG_ID_RE = re.compile(r"\d+(?:-\d+)+")+_AUTHORITY_TOKEN_RE = re.compile(+ r"[A-Za-z0-9](?:[A-Za-z0-9._-]*[A-Za-z0-9])?(?::\d{1,5})?"+)++# Prose scanning: only these characters can begin (or precede) a citation.+_INTERESTING_RE = re.compile(r"[\[\u3010 \t]")+# URL scanning: jump between the only characters that affect paren balance+# or terminate a URL. CommonMark ends a destination at an ASCII space or+# control character only; ``_safe_link_destination`` leaves Unicode spaces+# (U+00A0, U+3000) in place, so they are URL characters here.+_URL_EVENT_RE = re.compile(r"[()\x00-\x20\x7f]")+# A bare (autolinked) URL: it runs to whitespace, ``<``, ``>`` or a quote,+# and a ``)`` with no matching ``(`` inside it ends it (the ``)`` closing+# a Markdown link destination). Each match consumes its characters, so+# one finditer over a chunk is linear.+_BARE_URL_RE = re.compile(+ r"(?<![A-Za-z0-9+.-])https?://[^\s<>\"']*", re.IGNORECASE+)+_PAREN_RE = re.compile(r"[()]")+# What may precede a ``[1]:`` bibliography-style entry on its own line:+# indentation, blockquote markers, a list marker or an ATX heading opener.+# Each marker must be followed by whitespace (``>`` excepted), so a run+# like ``####`` has exactly one parse and the match stays linear.+_LINE_LEAD_RE = re.compile(+ r"[ \t]*(?:(?:>|(?:[-+*]|\d{1,9}[.)]|#{1,6})(?=[ \t]))[ \t]*)*"+)+_LINE_LEAD_MAX = 64 # longest such prefix considered+_CLOSING_TAG_RE = re.compile(r"</[A-Za-z][A-Za-z0-9-]*[ \t]*>")+_CLOSING_TAG_MAX = 64 # longest closing tag looked back over+# Everything up to the next unescaped ``[`` or ``]`` (captured), reading+# backslash escapes pairwise; an unrolled loop, so a failed match is linear.+_UNESCAPED_BRACKET_RE = re.compile(r"[^\[\]\\]*(?:\\[\s\S][^\[\]\\]*)*([\[\]])")+_HTML_CODE_TAG_RE = re.compile(r"</?(?:code|pre)(?:\s[^>]*)?>", re.IGNORECASE)+_HTML_COMMENT_RE = re.compile(r"<!--.*?(?:-->|$)", re.DOTALL)+# A code span / link label may cross newlines but never a blank line.+_BLANK_LINE_RE = re.compile(r"\r?\n[ \t]*\r?\n")+# Link reference definition, optionally inside blockquotes or a list item.+_REFERENCE_DEF_RE = re.compile(+ r"^[ \t]{0,3}(?P<quotes>(?:>[ \t]?)*)(?P<list>"+ r"(?:[-+*]|\d{1,9}[.)])[ \t]+)?\[(?P<label>[^\]]*)\]:(?P<destination>.*)$"+)+_CONTAINER_PREFIX_RE = re.compile(+ r"^(?P<indent>[ \t]{0,3})(?P<quotes>(?:>[ \t]?)*)(?P<body>.*)$"+)+_REFERENCE_TITLE_RE = re.compile(+ r'(?:(?:"(?:[^"\\]|\\.)*")|(?:\'(?:[^\'\\]|\\.)*\')|(?:\([^)]*\)))'+)+_ATX_HEADING_RE = re.compile(r"^[ ]{0,3}#{1,6}(?:[ \t]+|$)")+_THEMATIC_BREAK_RE = re.compile(+ r"^[ ]{0,3}(?:(?:\*[ \t]*){3,}|(?:_[ \t]*){3,}|(?:-[ \t]*){3,})$"+)+_HTML_BLOCK_TAG_RE = re.compile(+ r"^[ ]{0,3}</?(?:address|article|aside|base|basefont|blockquote|body|caption|"+ r"center|col|colgroup|dd|details|dialog|dir|div|dl|dt|fieldset|figcaption|"+ r"figure|footer|form|frame|frameset|h[1-6]|head|header|hr|html|iframe|legend|"+ r"li|link|main|menu|menuitem|nav|noframes|ol|optgroup|option|p|param|search|"+ r"section|summary|table|tbody|td|tfoot|th|thead|title|tr|track|ul)(?:[ \t/>]|$)",+ re.IGNORECASE,+)+++def _has_letter(label: str) -> bool:+ return any("a" <= ch.lower() <= "z" for ch in label)+++def _is_num_list(label: str) -> bool:+ return _NUM_LIST_RE.fullmatch(label) is not None+++def _is_letter_token(label: str) -> bool:+ """``True`` for slug/domain/tag labels like ``arxiv.org``/``web-1``."""+ return _TOKEN_RE.fullmatch(label) is not None and _has_letter(label)+++def _is_ip_token(label: str) -> bool:+ try:+ ip_address(label)+ except ValueError:+ host, separator, port = label.rpartition(":")+ if not separator or not port.isdigit():+ return False+ try:+ ip_address(host)+ except ValueError:+ return False+ return True+++def _is_authority_token(label: str) -> bool:+ return (+ (":" in label or "." in label)+ and _AUTHORITY_TOKEN_RE.fullmatch(label) is not None+ and (_has_letter(label) or _is_ip_token(label))+ )+++def _is_formatter_ipv4_label(label: str, url: str) -> bool:+ try:+ parsed = urlsplit(url)+ host = parsed.hostname+ if host is None or ip_address(host).version != 4:+ return False+ except ValueError:+ return False+ return label == ".".join(parsed.netloc.split(".")[-2:])+++def _is_authority_citation(label: str, url: str) -> bool:+ try:+ authority = urlsplit(url).netloc+ except ValueError:+ return False+ if not authority:+ return False+ if authority.startswith("www."):+ authority = authority[4:]+ display = ".".join(authority.split(".")[-2:])+ base, separator, suffix = label.rpartition("-")+ return label == display or bool(+ separator and suffix.isdigit() and base == display+ )+++def _is_dotted_token(label: str) -> bool:+ """``True`` for domain labels — the only single-bracket link shape."""+ return _is_letter_token(label) and "." in label+++def _is_tag_id(label: str) -> bool:+ """``True`` for the plain source-tag form ``<slug>-N``."""+ return _TAG_ID_RE.fullmatch(label) is not None and _has_letter(label)+++def _parse_url(text: str, pos: int) -> int | None:+ """Parse a balanced-parenthesis URL whose ``(`` is at ``text[pos]``.++ Returns the index just past the closing ``()``, or ``None`` when the+ parens never balance within ``_URL_MAX`` characters and ``_URL_MAX_DEPTH``+ nesting levels, or the candidate contains an ASCII space or control+ character. Jumps between ``(`` / ``)`` / space events so long URL-free stretches cost one+ C-level scan, never a per-character Python loop.+ """+ limit = pos + _URL_MAX+ depth = 0+ i = pos+ while True:+ m = _URL_EVENT_RE.search(text, i, limit + 1)+ if m is None or m.start() > limit:+ return None+ ch = m.group()+ if ch == "(":+ depth += 1+ if depth > _URL_MAX_DEPTH:+ return None+ elif ch == ")":+ depth -= 1+ if depth == 0:+ return m.end()+ else: # whitespace inside a URL+ return None+ i = m.end()+++def _parse_inline_destination(text: str, pos: int) -> int | None:+ """Parse a Markdown inline destination and optional title."""+ limit = min(len(text), pos + _URL_MAX)+ i = pos + 1+ if text[i : i + 1] == "<":+ destination_end = text.find(">", i + 1, limit)+ if destination_end == -1 or "\n" in text[i:destination_end]:+ return None+ i = destination_end + 1+ else:+ depth = 0+ while i < limit:+ ch = text[i]+ if ch == "\\" and i + 1 < limit:+ i += 2+ continue+ if ch == "(":+ depth += 1+ if depth > _URL_MAX_DEPTH:+ return None+ elif ch == ")":+ if depth == 0:+ return i + 1+ depth -= 1+ elif ch <= " " or ch == "\x7f": # CommonMark: ASCII only+ break+ i += 1+ else:+ return None++ if i < limit and text[i] == ")":+ return i + 1++ while i < limit and text[i].isspace():+ i += 1+ title = _REFERENCE_TITLE_RE.match(text, i, limit)+ if title is None:+ return None+ i = title.end()+ while i < limit and text[i].isspace():+ i += 1+ return i + 1 if i < limit and text[i] == ")" else None+++def _match_citation_token(text: str, j: int) -> tuple[int | None, bool, int]:+ """Match ONE citation token starting at ``text[j]`` (a ``[`` or ``【``).++ Returns ``(end_index_or_None, is_hyperlink, opener_run_end)``. A+ non-``None`` end means ``text[j:end]`` is a citation-shaped token;+ ``opener_run_end`` is always the index past the opening bracket run, so+ a FAILED parse still advances the caller past the whole run — the+ formatter never emits a citation starting mid-bracket-run, and that rule+ is what keeps unclosed-bracket floods linear.+ """+ n = len(text)+ if text[j] == "\u3010": # lenticular 【1】 / 【1, 2】 — plain numbers only+ for closer in ("\u3011", "]"):+ close = text.find(closer, j + 1, j + 1 + _LABEL_MAX)+ if close != -1 and _NUM_LIST_RE.fullmatch(text, j + 1, close):+ return (close + 1, False, j + 1)+ return (None, False, j + 1)+ k = j+ while k < n and text[k] == "[":+ k += 1+ # Outer double brackets contain an IPv6 label's own brackets.+ if k - j >= 2:+ outer_close = text.find("]](", j + 2, j + 2 + _LABEL_MAX)+ if outer_close != -1:+ url_end = _parse_url(text, outer_close + 2)+ candidate = text[j + 2 : outer_close]+ if url_end is not None and _is_authority_citation(+ candidate, text[outer_close + 3 : url_end - 1]+ ):+ return (url_end, True, k)+ close = text.find("]", k, k + _LABEL_MAX)+ if close == -1:+ return (None, False, k)+ q = close+ while q < n and text[q] == "]":+ q += 1+ label = text[k:close]+ double = k - j >= 2 and q - close >= 2+ if (+ k - j >= 3+ and _is_ip_token(label)+ and text[close + 1 : close + 2] == ":"+ ):+ port_end = close + 2+ while port_end < n and text[port_end].isdigit():+ port_end += 1+ q = port_end+ while q < n and text[q] == "]":+ q += 1+ double = port_end > close + 2 and q - port_end >= 2+ if q < n and text[q] == "(":+ url_end = _parse_url(text, q)+ if url_end is not None:+ if double:+ ok = (+ _is_num_list(label)+ or _is_tag_id(label)+ or _NUMERIC_TAG_ID_RE.fullmatch(label) is not None+ or _is_letter_token(label)+ or _is_authority_token(label)+ or _is_ip_token(label)+ or _is_formatter_ipv4_label(+ label, text[q + 1 : url_end - 1]+ )+ )+ else:+ # The formatter only emits double-bracket links; a single+ # pair is a citation only when it is a domain label naming+ # the link's own host. A dotted label alone is ordinary+ # link text (``[config.yaml](url)``), and a numeric one is+ # a year, rank or anchor (``[1998](url)``, ``[1](#s1)``).+ ok = _is_dotted_token(label) and _is_authority_citation(+ label, text[q + 1 : url_end - 1]+ )+ return (url_end, True, k) if ok else (None, False, k)+ if _parse_inline_destination(text, q) is not None:+ return (None, False, k)+ # A malformed destination may still follow a bare citation.+ if double:+ return (None, False, k) # the formatter never emits a bare [[label]]+ if _is_num_list(label) or _is_tag_id(label):+ return (q, False, k)+ return (None, False, k)+++def _reference_label(label: str) -> str:+ return " ".join(label.split()).casefold()+++def _is_reference_link(+ text: str, start: int, token_end: int, references: frozenset[str]+) -> bool:+ own_label = _reference_label(text[start + 1 : token_end - 1])+ if text[token_end : token_end + 1] != "[":+ return (+ own_label in references and text[token_end : token_end + 1] != "("+ )+ close = text.find("]", token_end + 1, token_end + _LINK_LABEL_MAX)+ if close == -1:+ return False+ label = text[token_end + 1 : close] or text[start + 1 : token_end - 1]+ return _reference_label(label) in references+++def _match_citation_run(+ text: str, first_end: int, references: frozenset[str]+) -> int:+ """Extend a matched token across adjacent citation tokens.++ ``_replace_comma_citations`` joins formatted parts with no separator,+ so tokens arrive glued (``[local-1][personal-notes-2]``,+ ``[[web-1]](url)[local-2]``) or horizontally spaced. Continuation+ tokens skip the word-glue guard: they follow a citation, not an+ identifier. Horizontal whitespace is consumed only when another token+ actually follows it.+ """+ n = len(text)+ end = first_end+ while end < n:+ if text[end] in "[\u3010":+ tok = _match_citation_token(text, end)+ if tok[0] is None or _is_reference_link(+ text, end, tok[0], references+ ):+ return end+ end = tok[0]+ continue+ k = end+ while k < n and text[k] in " \t":+ k += 1+ if k == end or k >= n or text[k] not in "[\u3010":+ return end+ tok = _match_citation_token(text, k)+ if tok[0] is None or _is_reference_link(text, k, tok[0], references):+ return end+ end = tok[0]+ return end+++def _bare_url_spans(text: str) -> tuple[list[int], list[int]]:+ """Return ``(starts, ends)`` of the bare URLs in ``text``, in order."""+ starts: list[int] = []+ ends: list[int] = []+ for match in _BARE_URL_RE.finditer(text):+ start, end = match.span()+ depth = 0+ for paren in _PAREN_RE.finditer(text, start, end):+ if paren.group() == "(":+ depth += 1+ elif depth:+ depth -= 1+ else:+ end = paren.start()+ break+ starts.append(start)+ ends.append(end)+ return (starts, ends)+++def _inside_bare_url(spans: tuple[list[int], list[int]], j: int) -> bool:+ starts, ends = spans+ idx = bisect_right(starts, j) - 1+ return idx >= 0 and j < ends[idx]+++def _is_word_glue(ch: str) -> bool:+ """Whether ``ch`` glues a following ``[...]`` into an index expression.++ Letters and digits of space-separated scripts do (``results[1]``).+ Scripts written without spaces (CJK, kana, Thai, ...) do not: a+ citation sits directly against their text (``首都です[1]``).+ """+ return (ch.isalnum() or ch == "_") and not _is_unspaced(ch)+++def _is_unspaced(ch: str) -> bool:+ """``True`` for a character of a script written without word spaces,+ including its punctuation (``。``, ``、``, fullwidth forms)."""+ code = ord(ch) if len(ch) == 1 else 0+ return any(low <= code <= high for low, high in _UNSPACED_RANGES)+++def _line_lead(text: str, j: int, left: str) -> bool:+ """Whether only container/heading markers precede ``text[j]`` on its+ line. ``left`` is the character before ``text[0]`` (``""`` at the+ start of a prose block, which starts a line)."""+ line_start = max(text.rfind("\n", 0, j), text.rfind("\r", 0, j)) + 1+ if line_start == 0 and left not in _LINE_START:+ return False+ if j - line_start > _LINE_LEAD_MAX:+ return False+ return _LINE_LEAD_RE.fullmatch(text, line_start, j) is not None+++def _try_strip_run(+ text: str,+ j: int,+ guard_prev: bool,+ references: frozenset[str],+ url_spans: tuple[list[int], list[int]],+ prev_char: str = "",+ left: str = "",+) -> tuple[int | None, int]:+ """Try to strip one citation run whose first token starts at ``j``.++ Returns ``(strip_end_or_None, skip_to)``. ``guard_prev`` applies the+ bare-token word-glue guard (see module docstring); it is skipped when+ the run starts after horizontal whitespace, where a real citation may+ legally sit. ``prev_char`` supplies the source character immediately+ before ``text[0]`` when ``j`` is 0, so a chunk starting right after a+ protected span still honours the guard. ``left`` is the character+ before ``text[0]`` (``""`` when ``text`` starts a line). ``url_spans``+ is ``_bare_url_spans(text)``. On any failure ``skip_to`` is the+ opener-run end, so the caller never retries mid-run.+ """+ tok = _match_citation_token(text, j)+ if tok[0] is None:+ return (None, tok[2])+ end, is_hyperlink = tok[0], tok[1]+ if _is_escaped(text, j) or _is_reference_link(text, j, end, references):+ return (None, tok[2])+ if j > 0 and text[j - 1] == "!":+ return (None, tok[2])+ if (+ not is_hyperlink+ and text[end : end + 1] == ":"+ and _line_lead(text, j, left)+ ):+ # ``[1]: Smith 2020`` opening a line is a bibliography entry.+ return (None, tok[2])+ if _inside_bare_url(url_spans, j):+ return (None, tok[2])+ if (+ guard_prev+ and not is_hyperlink+ and text[j] != "\u3010" # nobody indexes with ``x【1】``+ and (j > 0 or prev_char)+ ):+ prev = text[j - 1] if j > 0 else prev_char+ # ``{`` covers LaTeX subscripts (``x_{[i-1]}``).+ if _is_word_glue(prev) or prev in "]{":+ return (None, tok[2])+ return (_match_citation_run(text, end, references), tok[2])+++def _match_verbatim_link(text: str, j: int) -> int | None:+ """Match a complete Markdown link/image whose ``[`` is at ``text[j]``.++ Returns the index past an inline or reference-style destination, or past+ the label's ``]`` for a destination-less image alt, or ``None``. The caller+ keeps a matched structure verbatim: its label+ may contain brackets that look citation-shaped (``[see [1]](url)``)+ and may span lines, but not a blank line. Scanning is bounded by+ ``_LINK_LABEL_MAX`` and jumps between brackets with C-level finds, so+ adversarial input stays linear.+ """+ n = len(text)+ limit = min(n, j + _LINK_LABEL_MAX)+ is_image = j > 0 and text[j - 1] == "!"+ if (+ not is_image+ and text.find("](", j + 1, limit + 1) == -1+ and text.find("][", j + 1, limit + 1) == -1+ ):+ return None+ depth = 1+ i = j + 1+ while i < limit:+ # ``i`` always sits just past a bracket, so backslash pairing+ # from ``i`` is aligned; escaped brackets are skipped at C level+ # instead of being re-found and re-checked once per nesting step.+ bracket = _UNESCAPED_BRACKET_RE.match(text, i, limit)+ if bracket is None:+ return None+ close = bracket.start(1)+ if text[close] == "[":+ depth += 1+ i = close + 1+ continue+ depth -= 1+ if depth == 0:+ if _BLANK_LINE_RE.search(text[j : close + 1]) is not None:+ return None+ suffix = close + 1+ if text[suffix : suffix + 1] == "(":+ dest_end = _parse_inline_destination(text, suffix)+ if dest_end is not None:+ return dest_end+ if text[suffix : suffix + 1] == "[":+ # A reference label holds no unescaped bracket, so+ # ``[Table 2][[1]](url)`` is text followed by a citation.+ reference = _UNESCAPED_BRACKET_RE.match(text, suffix + 1, limit)+ if reference is not None and text[reference.start(1)] == "]":+ return reference.start(1) + 1+ return close + 1 if is_image else None+ i = close + 1+ return None+++def _is_escaped(text: str, pos: int) -> bool:+ backslashes = 0+ pos -= 1+ while pos >= 0 and text[pos] == "\\":+ backslashes += 1+ pos -= 1+ return backslashes % 2 == 1+++def _html_tag_spans(text: str) -> list[tuple[int, int]]:+ """Return spans of HTML tags while respecting quoted attribute values."""+ spans: list[tuple[int, int]] = []+ pos = 0+ while True:+ start = text.find("<", pos)+ if start == -1 or start + 1 == len(text):+ return spans+ marker = text[start + 1]+ name_pos = start + 2 if marker == "/" else start + 1+ if name_pos >= len(text) or not text[name_pos].isalpha():+ pos = start + 1+ continue+ quote = ""+ i = name_pos + 1+ while i < len(text):+ ch = text[i]+ if quote:+ if ch == quote:+ quote = ""+ elif ch in "\"'":+ quote = ch+ elif ch == ">":+ spans.append((start, i + 1))+ pos = i + 1+ break+ elif ch == "<":+ pos = i+ break+ i += 1+ else:+ return spans+++# Characters after a stripped run that need no separator before them.+_NO_SEPARATOR_BEFORE = " \t\r\n.,;:!?)]}"+# Scripts written without word spaces, with their punctuation: Thai and+# Lao, Tibetan, Myanmar, Khmer, CJK symbols/punctuation, kana, CJK+# ideographs (with extensions and compatibility forms), Yi, fullwidth and+# halfwidth forms.+_UNSPACED_RANGES = (+ (0x0E00, 0x0EFF),+ (0x0F00, 0x0FFF),+ (0x1000, 0x109F),+ (0x1780, 0x17FF),+ (0x2E80, 0x2FDF),+ (0x3000, 0x31FF),+ (0x3400, 0x4DBF),+ (0x4E00, 0x9FFF),+ (0xA000, 0xA4CF),+ (0xF900, 0xFAFF),+ (0xFE30, 0xFE4F),+ (0xFF00, 0xFFEF),+ (0x20000, 0x3134F),+)+++def _ends_closing_tag(text: str, j: int) -> bool:+ """Whether ``text[:j]`` ends with a closing HTML tag (``</span>``)."""+ if j == 0 or text[j - 1] != ">":+ return False+ start = text.rfind("<", max(0, j - _CLOSING_TAG_MAX), j - 1)+ return start != -1 and _CLOSING_TAG_RE.fullmatch(text, start, j) is not None+++def _joins_words(+ text: str, j: int, end: int, left: str, tag_end: int = -1+) -> bool:+ """Whether removing ``text[j:end]`` would join two words, so a space+ must replace it (``word[[1]](url)next``, ``Claim.[[1]](url)Next``).++ Only a word may follow (not ``_``, and not a script written without+ spaces). Before the run there must be a word character, or+ punctuation that ends a word: ``)``, ``]``, ``}``, ``>``, ``!``, ``?``,+ ``;``, ``:``; ``.`` or ``,`` unless it sits between digits+ (``3.[[1]](u)5``); ``*``, ``"`` or ``'`` only when it CLOSES, i.e. its+ run follows a non-space character (``**bold**[[1]](u)Next`` but not+ ``**[[1]](u)Smith**``). Anything else — ``/page/[1]/x``, an opening+ bracket — stays joined. ``left`` is the character before ``text[0]``.++ A following ``[`` (a link label) or ``<`` (an inline tag) renders as+ the start of the next word, so it counts as one when a word character+ or a closing tag precedes the run: ``word[[1]](u)[dataset](v)`` and+ ``</span>[[1]](u)<span>`` keep a space. ``tag_end`` is a position+ of ``text`` known to follow a closing tag outside ``text``.+ """+ nxt = text[end]+ prev = text[j - 1] if j > 0 else left+ if nxt in "[<":+ return _is_word_glue(prev) or j == tag_end or _ends_closing_tag(text, j)+ if nxt == "_" or not _is_word_glue(nxt):+ return False+ if _is_word_glue(prev) or prev in ")]}>!?;:":+ return True+ if prev in ".,":+ before = text[j - 2] if j > 1 else (left if j == 1 else "")+ return not (before.isdigit() and nxt.isdigit())+ if prev in "*\"'":+ if j == 0:+ return False # the run's start lies in another chunk+ k = j - 1+ while k > 0 and j - k < 3 and text[k - 1] == prev:+ k -= 1+ before = text[k - 1] if k > 0 else left+ return bool(before) and not before.isspace() and before != prev+ return False+++# ``prev`` values meaning "nothing precedes this position on its line"+# (``""`` is the start of a prose block, which always starts a line).+_LINE_START = ("", "\n", "\r")+++def _skip_horizontal(text: str, pos: int) -> int:+ while pos < len(text) and text[pos] in " \t":+ pos += 1+ return pos+++def _drop_open_paren(out: list[str]) -> None:+ """Remove the ``(`` that ends ``out`` and the one space before it. The+ caller has checked that both lie in the current chunk, mid-line."""+ for _ in range(2):+ while out and not out[-1]:+ out.pop()+ if not out:+ return+ out[-1] = out[-1][:-1]+++def _strip_citations_from_prose(+ text: str, references: frozenset[str], left: str = "", tag_end: int = -1+) -> str:+ """Strip citation runs from one code-span-free prose string.++ ``left`` is the single source character preceding ``text[0]`` (when+ the chunk starts right after a protected span); it supplies the+ word-glue guard and separator decision for a run at position 0.+ ``tag_end`` is the position in ``text`` right after a closing HTML+ tag that lies outside ``text`` (``-1`` when none).+ """+ out: list[str] = []+ pos = 0+ n = len(text)+ url_spans = _bare_url_spans(text)+ while True:+ m = _INTERESTING_RE.search(text, pos)+ if m is None:+ out.append(text[pos:])+ return "".join(out)+ j = m.start()+ out.append(text[pos:j])+ if text[j] in "[\u3010":+ prev = text[j - 1] if j > 0 else left+ end, skip_to = _try_strip_run(+ text,+ j,+ guard_prev=True,+ references=references,+ url_spans=url_spans,+ prev_char=prev,+ left=left,+ )+ if end is not None:+ if prev in _LINE_START:+ # Nothing precedes the run on its line: its separator+ # is the whitespace AFTER it, which would otherwise+ # indent the following text.+ end = _skip_horizontal(text, end)+ elif (+ prev == "("+ and text[end : end + 1] == ")"+ and j >= 3+ and text[j - 2] in " \t"+ and text[j - 3] not in " \t\r\n"+ and not (end + 1 < n and _is_word_glue(text[end + 1]))+ ):+ # ``Claim ([[1]](u)).``: the parentheses held only the+ # citation; drop them with the space before them.+ _drop_open_paren(out)+ end += 1+ elif end < n and _joins_words(text, j, end, left, tag_end):+ out.append(" ")+ pos = end+ continue+ link_end = _match_verbatim_link(text, j)+ if link_end is not None:+ out.append(text[j:link_end])+ pos = link_end+ continue+ out.append(text[j:skip_to])+ pos = skip_to+ else: # horizontal whitespace: a run may start after it+ k = m.end()+ while k < n and text[k] in " \t":+ k += 1+ if k < n and text[k] in "[\u3010":+ end, skip_to = _try_strip_run(+ text,+ k,+ guard_prev=False,+ references=references,+ url_spans=url_spans,+ left=left,+ )+ if end is not None:+ after = _skip_horizontal(text, end)+ if after == n or text[after] in "\r\n":+ # The run ends its line: drop the whitespace+ # before it (no trailing whitespace is created);+ # whitespace after it is kept verbatim.+ pass+ elif (text[j - 1] if j > 0 else left) in _LINE_START:+ # Indentation before a line-leading run is kept+ # verbatim; the run's separator is the whitespace+ # after it.+ out.append(text[j:k])+ end = after+ elif text[end] not in _NO_SEPARATOR_BEFORE and not (+ _is_unspaced(text[end])+ ):+ out.append(text[j:k])+ pos = end+ continue+ link_end = _match_verbatim_link(text, k)+ if link_end is not None:+ out.append(text[j:link_end])+ pos = link_end+ continue+ out.append(text[j:skip_to])+ pos = skip_to+ else:+ out.append(text[j:k])+ pos = k+++def _backtick_runs(text: str) -> list[tuple[int, int, int]]:+ """Backtick spans grouped by their enclosing paragraph or heading."""+ runs: list[tuple[int, int, int]] = []+ offset = 0+ paragraph = 0+ paragraph_depth = 0+ previous_blank = True+ previous_hard = False+ for line in text.splitlines(keepends=True):+ body, depth = _container_body(line)+ blank = not body.strip()+ hard = _ends_markdown_block(body)+ # A setext heading underline only underlines a paragraph line,+ # but it ends that paragraph — a code span cannot pair across it.+ hard = hard or (+ not previous_blank+ and not previous_hard+ and _SETEXT_UNDERLINE_RE.fullmatch(body.rstrip("\r\n")) is not None+ )+ # CommonMark: a list marker interrupts a paragraph only when it is+ # a bullet or an ordered marker numbered 1 — "2. text" is a lazy+ # continuation line, so it must not split an open code-span+ # paragraph (leaving stray unpaired backticks behind).+ marker = _LIST_MARKER_RE.match(body)+ starts_list = marker is not None and (+ marker.group("bullet") is not None or marker.group("ordered") == "1"+ )+ # ``paragraph_depth`` is the container depth the open paragraph+ # lives at; a lazy continuation line (a paragraph line without+ # its ``>`` prefix) keeps that paragraph — and its container —+ # open, so a ``>`` re-entry at the paragraph's own depth is+ # paragraph continuation text, not a new block. Only a DEEPER+ # container interrupts the paragraph (a quote start may interrupt+ # one); a shallower line is itself a lazy continuation.+ if (+ previous_blank+ or blank+ or hard+ or previous_hard+ or starts_list+ or depth > paragraph_depth+ ):+ paragraph += 1+ paragraph_depth = depth+ for match in re.finditer(r"`+", line):+ runs.append(+ (offset + match.start(), offset + match.end(), paragraph)+ )+ offset += len(line)+ previous_blank = blank+ previous_hard = hard+ return runs+++def _inline_spans(+ text: str,+) -> tuple[list[tuple[int, int]], list[tuple[int, int]]]:+ """Recognize code and comments in order; each protects its own contents.++ A backtick or ``<!--`` inside an HTML tag is attribute text, not an+ opener: ``_html_tag_spans`` is quote-aware, so every position it+ covers is skipped before it can start a phantom span reaching past+ the tag and shielding later prose.+ """+ runs = _backtick_runs(text)+ tags = _html_tag_spans(text)+ tag_index = 0+ outside_tags: list[tuple[int, int, int]] = []+ for run in runs:+ while tag_index < len(tags) and tags[tag_index][1] <= run[0]:+ tag_index += 1+ if tag_index < len(tags) and tags[tag_index][0] <= run[0]:+ continue+ outside_tags.append(run)+ runs = outside_tags+ next_same: list[int | None] = [None] * len(runs)+ next_shorter: list[int | None] = [None] * len(runs)+ next_by_key: dict[tuple[int, int], int] = {}+ for idx in range(len(runs) - 1, -1, -1):+ start, end, paragraph = runs[idx]+ next_same[idx] = next_by_key.get((paragraph, end - start))+ next_shorter[idx] = next_by_key.get((paragraph, end - start - 1))+ next_by_key[(paragraph, end - start)] = idx+ codes: list[tuple[int, int]] = []+ comments: list[tuple[int, int]] = []+ comment_index = 0++ def next_comment(from_pos: int) -> int:+ nonlocal comment_index+ candidate = text.find("<!--", from_pos)+ while candidate != -1:+ if _is_escaped(text, candidate):+ candidate = text.find("<!--", candidate + 4)+ continue+ while (+ comment_index < len(tags)+ and tags[comment_index][1] <= candidate+ ):+ comment_index += 1+ if (+ comment_index < len(tags)+ and tags[comment_index][0] <= candidate+ ):+ candidate = text.find("<!--", tags[comment_index][1])+ continue+ break+ return candidate++ pos = 0+ i = 0+ comment = next_comment(0)+ while pos < len(text):+ while i < len(runs) and runs[i][0] < pos:+ i += 1+ if comment != -1 and comment < pos:+ comment = next_comment(pos)+ tick = runs[i][0] if i < len(runs) else len(text)+ if comment != -1 and comment < tick:+ close = text.find("-->", comment + 4)+ pos = len(text) if close == -1 else close + 3+ comments.append((comment, pos))+ continue+ if i == len(runs):+ break+ start, end, _ = runs[i]+ if _is_escaped(text, start):+ start += 1+ close_idx = next_shorter[i] if start < end else None+ else:+ close_idx = next_same[i]+ if close_idx is not None:+ pos = runs[close_idx][1]+ codes.append((start, pos))+ i = close_idx + 1+ else:+ pos = end+ i += 1+ return (codes, comments)+++def _code_spans(text: str) -> list[tuple[int, int]]:+ return _inline_spans(text)[0]+++def _strip_code_and_citations(+ text: str,+ references: frozenset[str],+ left: str = "",+ left_tag: bool = False,+) -> str:+ """Strip citations from prose, keeping inline code and comments verbatim.++ ``left`` is the source character immediately before ``text[0]``; it is+ threaded into every chunk that starts at ``pos`` so a citation run+ directly following a protected span keeps one separator instead of+ joining the neighbouring words. ``left_tag`` says ``text`` starts right+ after a closing HTML tag outside it.+ """++ def after_tag(at: int) -> bool:+ return left_tag if at == 0 else _ends_closing_tag(text, at)++ pieces: list[str] = []+ pos = 0+ code_spans, comment_spans = _inline_spans(text)+ element_ranges = _code_element_ranges(text)+ protected_spans: list[tuple[int, int]]+ if element_ranges:+ # Whole inline code elements swallow every smaller span inside+ # them (the tags themselves, code spans, comments); emitting an+ # inner span first would re-expose the element's content.+ kept: list[tuple[int, int]] = []+ idx = 0+ for span in sorted(comment_spans + _html_tag_spans(text) + code_spans):+ while (+ idx < len(element_ranges) and element_ranges[idx][1] <= span[0]+ ):+ idx += 1+ if idx < len(element_ranges) and element_ranges[idx][0] <= span[0]:+ continue+ kept.append(span)+ protected_spans = sorted(element_ranges + kept)+ else:+ protected_spans = sorted(+ comment_spans + _html_tag_spans(text) + code_spans+ )+ url_spans = _bare_url_spans(text) if protected_spans else ([], [])+ for start, end in protected_spans:+ if end <= pos:+ continue+ search_pos = pos+ while True:+ link_start = text.find("[", search_pos, start)+ if link_start == -1:+ break+ citation_end, skip_to = _try_strip_run(+ text,+ link_start,+ guard_prev=True,+ references=references,+ url_spans=url_spans,+ prev_char=left if link_start == 0 else "",+ left=left,+ )+ link_end = (+ _match_verbatim_link(text, link_start)+ if citation_end is None+ else None+ )+ if link_end is not None:+ pieces.append(+ _strip_citations_from_prose(+ text[pos:link_start],+ references,+ left=text[pos - 1] if pos else left,+ tag_end=0 if after_tag(pos) else -1,+ )+ )+ pieces.append(text[link_start:link_end])+ pos = link_end+ search_pos = pos+ continue+ search_pos = max(link_start + 1, citation_end or skip_to)+ if end <= pos:+ continue+ if start < pos:+ # Overlapping span: its opener sat inside already-emitted+ # verbatim content (an HTML tag or a Markdown link), so the+ # span is phantom — clamping to pos, the remainder is prose+ # and must never be re-emitted or shield citations.+ continue+ context_start = max(0, pos - 1)+ stripped = _strip_citations_from_prose(+ text[context_start:start] + text[start],+ references,+ left=left if pos == 0 else "",+ tag_end=pos - context_start if after_tag(pos) else -1,+ )+ pieces.append(stripped[pos - context_start : -1])+ pieces.append(text[start:end])+ pos = end+ pieces.append(+ _strip_citations_from_prose(+ text[pos:],+ references,+ left=text[pos - 1] if pos else left,+ tag_end=0 if after_tag(pos) else -1,+ )+ )+ return "".join(pieces)+++def _html_code_tags(text: str) -> list[tuple[int, int, bool]]:+ tags: list[tuple[int, int, bool]] = []+ for start, end in _html_tag_spans(text):+ if re.match(+ r"</?(?:code|pre)(?=[\s/>])", text[start:end], re.IGNORECASE+ ):+ tags.append((start, end, text[start + 1 : start + 2] == "/"))+ return tags+++def _code_element_ranges(text: str) -> list[tuple[int, int]]:+ """Whole ``<code>/<pre>`` element ranges: open-tag start through the+ matching close-tag end, nesting-aware.++ Used where the element is INLINE HTML inside a paragraph (CommonMark+ type 7): everything between the tags renders as code and must stay+ verbatim. Unclosed elements contribute nothing — the raw tag itself+ is already protected as an HTML tag span.+ """+ ranges: list[tuple[int, int]] = []+ stack: list[int] = []+ for start, end, closing in _html_code_tags(text):+ if closing and stack:+ open_start = stack.pop()+ if not stack:+ ranges.append((open_start, end))+ elif not closing:+ stack.append(start)+ return ranges+++def _code_tags_start_block(+ tags: list[tuple[int, int, bool]], line: str, prose_open: bool+) -> bool:+ """Whether the line's code/pre tags open a CommonMark HTML *block*.++ ``<pre>`` is a type-1 start condition: it may interrupt a paragraph+ and its content spans blank lines. ``<code>`` is type 7: it may+ start a block only when no paragraph is open, the tag sits at the+ container's content-column start (up to three columns in), and only+ whitespace follows the complete open tag on the line. Any other+ ``<code>`` occurrence is inline HTML inside a paragraph — the line+ is prose, and the element's content is protected by+ ``_code_element_ranges`` instead of element state.+ """+ for start, end, closing in tags:+ if closing:+ continue+ tag = line[start : start + 5].lower()+ if tag.startswith("<pre"):+ return True+ if not tag.startswith("<code"):+ continue+ if prose_open:+ return False+ prefix = _CONTAINER_PREFIX_RE.match(line)+ content_at = (+ len(prefix.group("indent")) + len(prefix.group("quotes"))+ if prefix+ else 0+ )+ if not (+ content_at <= start <= content_at + 3+ and not line[content_at:start].strip()+ ):+ return False+ if end <= len(line) and line[end:].strip():+ return False+ return True+ return False+++def _strip_html_code_line(+ line: str,+ tags: list[tuple[int, int, bool]],+ depth: int,+ references: frozenset[str],+) -> tuple[str, int]:+ """Preserve real code elements, ignoring tag-shaped attribute text.++ ``tags`` are the code/pre tags opening on this line (line-relative+ offsets, multi-line aware — a tag whose ``>`` sits on a later line+ ends past this line's length). Each prose segment is stripped with+ the character before it as left context so a citation directly after+ a tag keeps one separator.+ """+ pieces: list[str] = []+ pos = 0+ for start, end, closing in tags:+ if depth:+ pieces.append(line[pos:end])+ depth += -1 if closing else 1+ elif not closing:+ pieces.append(+ _strip_code_and_citations(+ line[pos:start],+ references,+ left=line[pos - 1] if pos else "",+ left_tag=_ends_closing_tag(line, pos),+ )+ )+ pieces.append(line[start:end])+ depth = 1+ else:+ pieces.append(+ _strip_code_and_citations(+ line[pos:end],+ references,+ left=line[pos - 1] if pos else "",+ left_tag=_ends_closing_tag(line, pos),+ )+ )+ pos = end+ tail = line[pos:]+ pieces.append(+ tail+ if depth+ else _strip_code_and_citations(+ tail,+ references,+ left=line[pos - 1] if 0 < pos <= len(line) else "",+ left_tag=0 < pos <= len(line) and _ends_closing_tag(line, pos),+ )+ )+ return ("".join(pieces), depth)+++def _open_fence(line: str) -> tuple[str, int] | None:+ """Return ``(fence_char, run_length)`` when ``line`` opens a fence.++ CommonMark: up to three leading spaces, a run of 3+ backticks or+ tildes; a backtick fence's info string may not contain backticks.+ """+ head = line.lstrip(" ")+ if len(line) - len(head) > 3:+ return None+ for ch in ("`", "~"):+ if head.startswith(ch * 3):+ run = len(head) - len(head.lstrip(ch))+ if ch == "`" and "`" in head[run:]:+ return None+ return (ch, run)+ return None+++def _container_body(line: str) -> tuple[str, int]:+ pos = 0+ depth = 0+ while match := _QUOTE_PREFIX_RE.match(line, pos):+ pos = match.end()+ depth += 1+ return (line[pos:], depth)+++def _is_close_fence(line: str, ch: str, run: int) -> bool:+ """``True`` when ``line`` closes a ``(ch, run)`` fence: same character,+ at least the opening run length, nothing but whitespace after it."""+ head = line.lstrip(" ")+ if len(line) - len(head) > 3:+ return False+ stripped_run = len(head) - len(head.lstrip(ch))+ return stripped_run >= run and head[stripped_run:].strip() == ""+++_QUOTE_PREFIX_RE = re.compile(r"[ ]{0,3}>[ \t]?")+_LIST_MARKER_RE = re.compile(+ r"[ ]{0,3}(?:(?P<bullet>[-+*])|(?P<ordered>\d{1,9})[.)])[ \t]+"+)+_SETEXT_UNDERLINE_RE = re.compile(r"[ ]{0,3}(?:=+|-+)[ \t]*$")+++def _mask_comment_line(+ line: str, spans: list[tuple[int, int]], in_comment: bool+) -> tuple[str, bool]:+ """Mask comments outside inline code while preserving line positions.++ ``spans`` are this line's inline code spans (line-relative offsets,+ computed over the WHOLE text by the caller, exactly as+ ``_tag_scan_text`` computes them): a span opened on this line may+ close on a later one, so a per-line ``_code_spans`` would miss it+ and latch the comment state on an opener that is really span+ content — masking a genuinely fenced block further down.+ """+ span_index = 0+ scan = list(line)+ pos = 0+ while pos < len(line):+ if in_comment:+ close = line.find("-->", pos)+ end = len(line) if close == -1 else close + 3+ scan[pos:end] = [" " for _ in range(end - pos)]+ in_comment = close == -1+ pos = end+ continue+ start = line.find("<!--", pos)+ if start == -1:+ break+ while span_index < len(spans) and spans[span_index][1] <= start:+ span_index += 1+ if _is_escaped(line, start) or (+ span_index < len(spans)+ and spans[span_index][0] <= start < spans[span_index][1]+ ):+ pos = start + 4+ continue+ pos = start+ in_comment = True+ return ("".join(scan), in_comment)+++def _blank_spans(scan: list[str], spans: list[tuple[int, int]]) -> None:+ """Blank character spans in place, preserving line positions."""+ for start, end in spans:+ scan[start:end] = [+ "\n" if char == "\n" else " " for char in scan[start:end]+ ]+++def _tag_scan_text(text: str) -> str:+ """Blank inline code spans and comments for HTML code-tag recognition.++ Used before fences are known (fence recognition itself needs code-tag+ awareness), so tag-shaped text inside an inline code span or comment+ cannot masquerade as a ``<code>``/``<pre>`` element.+ """+ scan = list(text)+ code_spans, comment_spans = _inline_spans(text)+ _blank_spans(scan, code_spans + comment_spans)+ return "".join(scan)+++def _code_tags_by_line(+ scan_text: str, lines: list[str]+) -> list[list[tuple[int, int, bool]]]:+ """Bucket code/pre tags by the line they open on, multi-line aware.++ ``_html_tag_spans`` already matches tags whose ``>`` sits on a later+ line when given a whole string; per-line scanning missed them. Tag+ offsets are relative to their opening line, so a tag ending on a+ later line yields an end past the opening line's length.+ """+ buckets: list[list[tuple[int, int, bool]]] = [[] for _ in lines]+ line_index = 0+ offset = 0+ for start, end, closing in _html_code_tags(scan_text):+ while line_index + 1 < len(lines) and start > offset + len(+ lines[line_index]+ ):+ offset += len(lines[line_index]) + 1+ line_index += 1+ buckets[line_index].append((start - offset, end - offset, closing))+ return buckets+++def _code_spans_by_line(+ spans: list[tuple[int, int]], lines: list[str]+) -> list[list[tuple[int, int]]]:+ """Clamp whole-text inline code spans to every line they cover.++ Unlike ``_code_tags_by_line`` (open-line only), a span opened on an+ earlier line keeps masking comment openers on each line inside it,+ so each intersected line receives the span clamped to its own+ bounds. Spans arrive sorted and disjoint, so the walk is linear.+ """+ buckets: list[list[tuple[int, int]]] = [[] for _ in lines]+ line_index = 0+ offset = 0+ for start, end in spans:+ while line_index + 1 < len(lines) and start > offset + len(+ lines[line_index]+ ):+ offset += len(lines[line_index]) + 1+ line_index += 1+ i = line_index+ line_offset = offset+ while i < len(lines) and line_offset < end:+ clamp_start = max(start, line_offset) - line_offset+ clamp_end = min(end, line_offset + len(lines[i])) - line_offset+ if clamp_start < clamp_end:+ buckets[i].append((clamp_start, clamp_end))+ line_offset += len(lines[i]) + 1+ i += 1+ return buckets+++def _fence_ranges(+ lines: list[str], code_tags: list[list[tuple[int, int, bool]]]+) -> dict[int, int]:+ """Map fence openers to exclusive ends, including their container boundary.++ Each visited fence consumes its content once. Lists retain their content+ indentation so a continuation fence is distinct from indented code.+ ``code_tags`` (from ``_code_tags_by_line``) tracks open ``<code>``/+ ``<pre>`` elements: a fence glyph inside one is element content, not a+ fence, so an unclosed ``` inside a code element cannot swallow — and+ protect from stripping — the rest of the document. Tags inside a+ recognized fence are literal fence content and never change the depth.+ """+ ranges: dict[int, int] = {}+ list_indents: list[int] = []+ previous_depth = 0+ in_comment = False+ html_depth = 0+ # Comment masking needs code spans over the whole text (a span may+ # straddle lines), computed exactly as ``_tag_scan_text`` does.+ span_lines = _code_spans_by_line(_code_spans("\n".join(lines)), lines)+ i = 0+ while i < len(lines):+ comment_before = in_comment+ scan_line, in_comment = _mask_comment_line(+ lines[i], span_lines[i], in_comment+ )+ body, quote_depth = _container_body(scan_line)+ if quote_depth != previous_depth:+ list_indents.clear()+ previous_depth = quote_depth+ body = body.expandtabs(4)+ leading = len(body) - len(body.lstrip(" "))+ if body.strip():+ while list_indents and leading < list_indents[-1]:+ list_indents.pop()+ offset = list_indents[-1] if list_indents else 0+ while True:+ marker = _LIST_MARKER_RE.match(body, offset)+ if marker is not None:+ offset = marker.end()+ list_indents.append(offset)+ continue+ quote = _QUOTE_PREFIX_RE.match(body, offset)+ if quote is not None:+ quote_depth += 1+ body = body[quote.end() :]+ offset = 0+ list_indents.clear()+ continue+ break+ fence = _open_fence(body[offset:])+ if fence is not None and not html_depth:+ # Comments in a fence info string are literal fence content.+ in_comment = comment_before+ end = i + 1+ while end < len(lines):+ continuation, depth = _container_body(lines[end])+ if depth < quote_depth:+ break+ continuation = continuation.expandtabs(4)+ if depth == quote_depth:+ if continuation.strip() and not continuation.startswith(+ " " * offset+ ):+ break+ if _is_close_fence(continuation[offset:], *fence):+ end += 1+ break+ end += 1+ ranges[i] = end+ i = end+ continue+ for _, _, closing in code_tags[i]:+ html_depth = max(0, html_depth + (-1 if closing else 1))+ i += 1+ return ranges+++def _indented_past_containers(body: str, quote_depth: int) -> tuple[bool, int]:+ """Decide indented code against the content column of the containers+ opened on this line; a list item opened on an earlier line is not+ tracked, so its later lines are measured from column 0.++ The same container walk ``_fence_ranges`` uses — blockquote body via+ ``_container_body``, ``expandtabs(4)`` columns, then ``_LIST_MARKER_RE``+ and inner quote prefixes — but a marker contributes its TRUE content+ column (one space past the marker itself, not past all its trailing+ whitespace), so ``- [1]`` is a list item whose content is indented+ code while ``- [1]`` (three columns past the content column) is a+ paragraph continuation that still strips. Tabs count toward the next+ multiple-of-4 column. Returns ``(indented, quote_depth)`` with any+ quote opened after a list marker added to the depth.+ """+ body = body.expandtabs(4)+ offset = 0+ while True:+ marker = _LIST_MARKER_RE.match(body, offset)+ if marker is not None:+ text = marker.group()+ offset = marker.end() - (len(text) - len(text.rstrip())) + 1+ continue+ quote = _QUOTE_PREFIX_RE.match(body, offset)+ if quote is not None:+ quote_depth += 1+ body = body[quote.end() :]+ offset = 0+ continue+ break+ return (body[offset:].startswith(" "), quote_depth)+++def _html_scan_lines(+ text: str, lines: list[str], fences: dict[int, int]+) -> list[str]:+ """Mask fenced/inline code and comments before recognizing HTML code tags."""+ scan = list(text)+ offsets: list[int] = []+ offset = 0+ for line in lines:+ offsets.append(offset)+ offset += len(line) + 1++ for i, end_line in fences.items():+ start = offsets[i]+ end = len(text) if end_line == len(lines) else offsets[end_line] - 1+ _blank_spans(scan, [(start, end)])++ code_spans, comment_spans = _inline_spans("".join(scan))+ _blank_spans(scan, code_spans + comment_spans)+ return "".join(scan).split("\n")+++def _reference_continuation(reference: re.Match[str], line: str) -> str | None:+ """Return ``line``'s body when it may continue the definition at+ ``reference`` (as its destination or title line), else ``None``.++ Such a line continues the definition's paragraph, and indented code+ cannot interrupt a paragraph, so CommonMark sets no indentation limit+ on it: indent 0, four spaces, a tab and any indent at, past or short+ of a list item's content column all qualify, as does a lazy line with+ fewer ``>`` markers. A blank line ends the paragraph, and a deeper+ ``>`` opens a nested blockquote, which interrupts it. Lines that open+ a fence or ``<pre>`` element are rejected by the caller.+ """+ continuation = _CONTAINER_PREFIX_RE.match(line)+ if continuation is None:+ return None+ if continuation.group("quotes").count(">") > reference.group(+ "quotes"+ ).count(">"):+ return None+ body = continuation.group("body")+ return body if body.strip() else None+++def _valid_reference_destination(rest: str) -> bool:+ """``True`` when ``rest`` (the text after ``]:``) parses as a link+ destination plus an optional title, per CommonMark.++ A destination is ``<...>`` or a single whitespace-free token with+ balanced parentheses; a well-formed title (``_REFERENCE_TITLE_RE``)+ may follow, separated by whitespace, and nothing else may remain.+ A destination with spaces and no valid title ("[Note]: See [1] for+ details.") is therefore prose, not a reference definition.+ """+ s = rest.lstrip()+ if not s:+ return False+ if s.startswith("<"):+ close = s.find(">")+ while close != -1 and _is_escaped(s, close):+ close = s.find(">", close + 1)+ if close == -1:+ return False+ after = s[close + 1 :].strip()+ else:+ parts = s.split(None, 1)+ token = parts[0]+ depth = 0+ k = 0+ while k < len(token):+ ch = token[k]+ if ch == "\\":+ k += 2+ continue+ if ch == "(":+ depth += 1+ elif ch == ")":+ depth -= 1+ if depth < 0:+ return False+ k += 1+ if depth != 0:+ return False+ after = parts[1].strip() if len(parts) > 1 else ""+ return not after or _REFERENCE_TITLE_RE.fullmatch(after) is not None+++def _reference_definition_extent(+ reference: re.Match[str],+ lines: list[str],+ i: int,+ block_openers: frozenset[int],+) -> int | None:+ """Return the exclusive end of a GENUINE reference definition whose+ match is at ``lines[i]``, or ``None`` when the line is not one.++ The destination may sit on the definition line or on the next line;+ one further line may carry the title (``_reference_continuation``+ decides which lines can continue a definition). ``block_openers``+ holds the indexes of lines that open a fenced code block or a+ ``<pre>`` element: either interrupts the definition's paragraph, so+ it is never a continuation line — consuming one as a destination+ would expose the block's code to stripping. An invalid destination+ (see ``_valid_reference_destination``) makes the line prose, so its+ citation markers strip normally.+ """+ next_line = i + 1+ destination = reference.group("destination")+ if destination.strip():+ if not _valid_reference_destination(destination):+ return None+ else:+ if next_line >= len(lines) or next_line in block_openers:+ return None+ continuation_body = _reference_continuation(reference, lines[next_line])+ if continuation_body is None or not _valid_reference_destination(+ continuation_body.strip()+ ):+ return None+ next_line += 1+ if next_line < len(lines) and next_line not in block_openers:+ continuation_body = _reference_continuation(reference, lines[next_line])+ if (+ continuation_body is not None+ and _REFERENCE_TITLE_RE.fullmatch(continuation_body.strip())+ is not None+ ):+ next_line += 1+ return next_line+++def _ends_markdown_block(+ line: str, previous_body: str = "", previous_break: bool = False+) -> bool:+ # ``_backtick_runs`` feeds keepends lines; the trailing line ending+ # must go before a fullmatch can see a bare thematic break.+ candidate = line.rstrip("\r\n")+ return (+ _ATX_HEADING_RE.match(candidate) is not None+ or _THEMATIC_BREAK_RE.fullmatch(candidate) is not None+ or _HTML_BLOCK_TAG_RE.match(candidate) is not None+ # Same condition as ``_backtick_runs``: a setext underline ends+ # the paragraph above it (so a following indented line is code),+ # but only when the previous line is a non-blank paragraph line+ # that did not itself end a block.+ or (+ bool(previous_body.strip())+ and not previous_break+ and _SETEXT_UNDERLINE_RE.fullmatch(candidate) is not None+ )+ )+++def strip_prior_turn_citations(text: str) -> str:+ """Remove prior-turn citation markers from ``text`` (see module docstring+ for the covered shapes). Code blocks/elements, inline code spans,+ non-citation markdown links and all whitespace other than one separator+ flanking each stripped run (see the module docstring) pass through+ verbatim; ``""`` returns ``""``."""+ if not text:+ return text+ lines = text.split("\n")+ out: list[str] = []+ prose: list[str] = []+ # Fence recognition needs code-element awareness (a ``` inside an open+ # <code>/<pre> is element content), and code-tag recognition wants+ # inline code and comments blanked — which needs no fence knowledge.+ fence_tags = _code_tags_by_line(_tag_scan_text(text), lines)+ fences = _fence_ranges(lines, fence_tags)+ html_scan_lines = _html_scan_lines(text, lines, fences)+ code_tags = _code_tags_by_line("\n".join(html_scan_lines), lines)+ # A fence or <pre> element interrupts a paragraph, so neither opening+ # line can continue a reference definition. ``prose_open=True`` is the+ # state the line would meet after an unfinished ``[label]:`` line; a+ # <code> element (HTML block type 7) cannot interrupt and is excluded.+ block_openers = frozenset(fences).union(+ k+ for k, tags in enumerate(code_tags)+ if tags and _code_tags_start_block(tags, lines[k], prose_open=True)+ )+ # Definitions written as code or comments are examples, not link targets.+ reference_labels: set[str] = set()+ reference_code_depth = 0+ for i, scan_line in enumerate(html_scan_lines):+ if not reference_code_depth:+ reference_match = _REFERENCE_DEF_RE.match(scan_line)+ if (+ reference_match is not None+ and _reference_definition_extent(+ reference_match, html_scan_lines, i, block_openers+ )+ is not None+ ):+ reference_labels.add(+ _reference_label(reference_match.group("label"))+ )+ for _, _, closing in code_tags[i]:+ reference_code_depth = max(+ 0, reference_code_depth + (-1 if closing else 1)+ )+ references = frozenset(reference_labels)++ def flush_prose() -> None:+ if prose:+ out.append(_strip_code_and_citations("\n".join(prose), references))+ prose.clear()++ i = 0+ in_html_code = 0+ # Outermost open element is <code> (HTML block type 7: its block+ # ends at the first blank line) vs <pre> (type 1: content spans+ # blank lines until the closing tag).+ html_code_type7 = False+ indented_quote_depth: int | None = None+ block_boundary = False+ while i < len(lines):+ line = lines[i]+ if in_html_code:+ if html_code_type7 and not line.strip():+ # A type-7 block ends at a blank line; the element's+ # content closed with it and the next paragraph is prose.+ in_html_code = 0+ html_code_type7 = False+ out.append(line)+ block_boundary = True+ i += 1+ continue+ stripped, in_html_code = _strip_html_code_line(+ line, code_tags[i], in_html_code, references+ )+ out.append(stripped)+ block_boundary = in_html_code == 0+ i += 1+ continue+ if i in fences:+ end = fences[i]+ flush_prose()+ out.append("\n".join(lines[i:end]))+ i = end+ block_boundary = True+ continue+ body, quote_depth = _container_body(line)+ # Indented code is measured from the content column of the+ # containers opened on this line (list markers, inner quotes, tab+ # stops), not column 0 of the quote-stripped body; a later line+ # of a list item carries no marker and is measured from column 0.+ indented, quote_depth = _indented_past_containers(body, quote_depth)+ if indented_quote_depth is not None:+ if quote_depth == indented_quote_depth and (+ indented or not body.strip()+ ):+ flush_prose()+ out.append(line)+ i += 1+ continue+ indented_quote_depth = None+ if indented and (not prose or not prose[-1].strip() or block_boundary):+ flush_prose()+ out.append(line)+ indented_quote_depth = quote_depth+ block_boundary = False+ i += 1+ continue+ reference = _REFERENCE_DEF_RE.match(line)+ if reference is not None:+ extent = _reference_definition_extent(+ reference, lines, i, block_openers+ )+ if extent is not None:+ flush_prose()+ out.extend(lines[i:extent])+ i = extent+ block_boundary = True+ continue+ if code_tags[i] and _code_tags_start_block(+ code_tags[i], line, prose_open=bool(prose and prose[-1].strip())+ ):+ flush_prose()+ stripped, in_html_code = _strip_html_code_line(+ line, code_tags[i], 0, references+ )+ first_open = next((tag for tag in code_tags[i] if not tag[2]), None)+ html_code_type7 = (+ in_html_code > 0+ and first_open is not None+ and line[first_open[0] : first_open[0] + 5]+ .lower()+ .startswith("<code")+ )+ out.append(stripped)+ block_boundary = in_html_code == 0+ i += 1+ continue+ prose.append(line)+ block_boundary = _ends_markdown_block(+ body,+ _container_body(prose[-2])[0] if len(prose) > 1 else "",+ block_boundary,+ )+ i += 1+ flush_prose()+ return "\n".join(out)diff --git a/src/local_deep_research/chat/context.py b/src/local_deep_research/chat/context.pyindex 2691a81c5c..eeb2236d51 100644--- a/src/local_deep_research/chat/context.py+++ b/src/local_deep_research/chat/context.py@@ -13,8 +13,33 @@ from typing import Dict, Any, List, Optionalfrom loguru import logger+from .citation_stripping import strip_prior_turn_citationsfrom ..config.thread_settings import get_setting_from_snapshot+# Recognized prior-turn citation markers are stripped from ASSISTANT-authored text+# only — see ``_build_conversation_text`` — because they index that turn's+# source list and dangle once the follow-up renumbers citations against+# its own sources. The stripping engine — grounded in CitationFormatter's+# emission shapes, segmenting fenced blocks / inline code spans / prose and+# scanning prose with bounded citation lookups — lives in+# ``chat.citation_stripping`` and is documented there.+++def _strip_or_passthrough(text: str) -> str:+ """Strip citation markers, failing safe to the unstripped text.++ Stripping is a context-quality cleanup; an unexpected error in it must+ not fail every later follow-up in the session, so it is logged and the+ text is carried over as-is.+ """+ try:+ return strip_prior_turn_citations(text)+ except Exception:+ logger.opt(exception=True).warning(+ "Citation stripping failed; carrying prior-turn text unstripped"+ )+ return text+class ChatContextManager:"""@@ -128,6 +153,11 @@ class ChatContextManager:for m in self.messages)+ # Stripping is scoped per-mode inside _select_prior_findings — e.g.+ # "full" mode strips assistant turns only, as it assembles them, and+ # must not have a second, role-blind strip pass applied to the+ # already-mixed-role transcript here (that would delete a markdown+ # link the USER typed in an earlier turn; see _build_conversation_text).findings = self._select_prior_findings(current_query, is_multi_turn)return {@@ -168,15 +198,30 @@ class ChatContextManager:if mode == "none":findings = ""elif mode == "raw":+ # _extract_findings_from_history strips each assistant+ # message itself — BEFORE truncating it, so a citation+ # crossing the truncation cutoff can't leave half a marker+ # behind (no pattern matches "[](htt...").findings = self._extract_findings_from_history()elif mode == "full":+ # _build_conversation_text strips citation markers per-turn,+ # assistant lines only, as it assembles the mixed-role+ # transcript — a second, role-blind strip pass here would+ # delete a markdown link the USER typed in an earlier turn.findings = self._build_conversation_text()elif current_query:- # "summary" with a question to focus on.+ # "summary" with a question to focus on. The transcript the+ # LLM reads keeps user turns verbatim (assistant turns are+ # stripped while it is built), but the summary it returns is+ # stripped as a whole, with no knowledge of which turn a span+ # came from: a citation-shaped string the user typed — e.g.+ # ``[1]`` or ``[arxiv.org](https://arxiv.org/abs/...)`` — is+ # removed if the summary repeats it. Ordinary links survive.findings = self._summarize_prior_work(current_query)else:# "summary" with no question (e.g. a no-arg build_research_context- # call): fall back to raw recent findings.+ # call): fall back to raw recent findings (already stripped+ # during extraction).findings = self._extract_findings_from_history()# Observability: the summary path is otherwise silent (no token-counter@@ -190,21 +235,69 @@ class ChatContextManager:)return findings+ def _strip_citation_markers(self, text: str) -> str:+ """Remove prior-turn citation markers from carried-over context.++ Handles the shapes CitationFormatter stores — markdown hyperlinks+ like ``[[1]](url)`` / ``[[arxiv.org-1]](url)`` (plus, for a model+ that re-renders one with a single bracket pair, a domain label+ naming the link's own host, ``[arxiv.org](https://arxiv.org/...)``;+ numeric single-bracket links such as ``[1998](url)`` are kept),+ the plain source-tagged form ``[arxiv-1]`` / ``[local-1]`` /+ ``[ai-2]`` / ``[personal-notes-3]``, plain+ ``[1]`` / ``[1, 2]``, and lenticular ``【1】`` — including runs of+ adjacent markers (comma groups are concatenated with no separator).+ See ``chat.citation_stripping`` for the full shape grounding.++ Fenced code blocks (CommonMark ``` and ~~~ fences) and inline code+ spans are not citation sites and pass through untouched. A stripped+ run consumes one flanking horizontal-whitespace separator — the one+ before it mid-line or at the end of a line, the one after it at the+ start of a line (indentation before it is kept) — so removing+ ``word [[1]](url) word`` never introduces a doubled space between+ the flanking words, and it is horizontal-only, so a marker starting+ the next line never joins two lines. If stripping raises, the error+ is logged and the text is carried over unstripped. There is+ deliberately no separate whitespace-collapse pass over the rest of+ the string:+ a blanket ``[ \\t]{2,}`` collapse would also flatten unrelated+ indentation (code blocks, nested markdown lists, table columns)+ that never had a citation marker anywhere near it.+ """+ if not text:+ return text+ return _strip_or_passthrough(text)+def _build_conversation_text(self) -> str:"""Render the prior conversation (both roles) as a plain transcript.Trims oldest-first to ``CONTEXT_INPUT_CHAR_BUDGET`` so the most recentturns survive and the summarizer's input cap never has to truncate.++ Citation markers are stripped from ASSISTANT lines only, per turn,+ before assembly — never from the joined transcript. Those markers+ index the prior turn's own source list and dangle once renumbered+ against the follow-up's freshly-found sources, but a citation-shaped+ string a USER typed (e.g. a markdown link they pasted) is verbatim+ user content and must reach the follow-up prompt untouched."""lines: List[str] = []used = 0for msg in reversed(self.messages):if not isinstance(msg, dict):continue- content = (msg.get("content") or "").strip()- if not content:+ content = msg.get("content") or ""+ if not content.strip():continue- role = (msg.get("role") or "unknown").capitalize()+ role_raw = msg.get("role") or "unknown"+ if role_raw == "assistant":+ # Strip citation markers BEFORE the outer trim: trimming+ # first would dedent a leading indented-code line into+ # prose, and its bracket literals would be deleted as+ # citations (raw mode never trims, so modes diverged).+ content = self._strip_citation_markers(content)+ content = content.strip()+ role = role_raw.capitalize()line = f"{role}: {content}"remaining = self.CONTEXT_INPUT_CHAR_BUDGET - usedif remaining <= 0:@@ -246,7 +339,17 @@ class ChatContextManager:)return ""- return FocusedSummarizer(+ class ChatContextSummarizer(FocusedSummarizer):+ def _truncate(self, summary: str) -> str:+ # Strip citations before the output cap, so a marker+ # straddling the cap is removed whole instead of being cut+ # into residue no pattern matches ("[[1]](https://exa...").+ # The summary arrives already trimmed (ProcessingLLMWrapper+ # and BaseSummarizer.summarize both strip it), so a leading+ # indented-code line in it reads as prose here.+ return super()._truncate(_strip_or_passthrough(summary).strip())++ return ChatContextSummarizer(llm,focus_query=current_query,max_sentences=self.CONTEXT_SUMMARY_MAX_SENTENCES,@@ -263,7 +366,10 @@ class ChatContextManager:for msg in self.messages:if msg.get("role") == "assistant" and msg.get("research_id"):- content = msg.get("content") or ""+ # Strip BEFORE truncating: truncation mid-citation would+ # cut a marker into a shape no pattern can match (residue+ # like "[](htt..." would survive into follow-up context).+ content = self._strip_citation_markers(msg.get("content") or "")# Summarize long responses - take first partif len(content) > 500:# Try to find a natural break pointdiff --git a/tests/chat/test_chat_followup_citation_stripping.py b/tests/chat/test_chat_followup_citation_stripping.pynew file mode 100644index 0000000000..17848672f3--- /dev/null+++ b/tests/chat/test_chat_followup_citation_stripping.py@@ -0,0 +1,1867 @@+"""Tests for format-aware citation stripping in chat follow-up context.++Prior-turn answers carry citations in whatever shape `report.citation_format`+produced (markdown hyperlinks `[[1]](url)` / `[[arxiv.org-1]](url)`, the plain+source-tagged form `[arxiv-1]` / `[local-1]`, plain `[1]`, lenticular `【1】`).+Those index the prior turn's sources, so they must be stripped from the+context carried into a follow-up — but ONLY from assistant-authored text;+anything the user typed themselves must survive untouched.+"""++import time++import pytest+from markdown import markdown++from local_deep_research.chat.context import ChatContextManager+++def _manager(messages=None, mode=None):+ snapshot = {"chat.followup_context_mode": mode} if mode else None+ return ChatContextManager(+ session_id="s1",+ messages=messages or [],+ accumulated_context={},+ settings_snapshot=snapshot,+ )+++class TestStripCitationMarkers:+ @pytest.mark.parametrize(+ "text,expected",+ [+ # number_hyperlinks (default report.citation_format)+ (+ "Light is used [[1]](http://a) in cells.",+ "Light is used in cells.",+ ),+ ("Both [[1, 2]](http://a) agree.", "Both agree."),+ # domain / domain-id hyperlinks+ # single-bracket domain forms count only when the label names+ # the link's own host, as CitationFormatter would label it+ ("Per [arxiv.org-1](https://arxiv.org/abs/1).", "Per."),+ ("Per [arxiv.org](https://www.arxiv.org/abs/1).", "Per."),+ # no_hyperlinks plain markers+ ("Plain [1] cite.", "Plain cite."),+ ("A group [1, 2] of cites.", "A group of cites."),+ # lenticular brackets some models emit+ ("Lenticular 【1】 marker.", "Lenticular marker."),+ ("Lenticular 【1, 2】 marker.", "Lenticular marker."),+ # mixed old + new in one string+ ("Old [1] new [[2]](http://x) mix.", "Old new mix."),+ # nothing to strip+ ("No citations at all.", "No citations at all."),+ ("", ""),+ ],+ )+ def test_strips_all_citation_formats(self, text, expected):+ assert _manager()._strip_citation_markers(text) == expected++ def test_none_input_passthrough(self):+ assert _manager()._strip_citation_markers(None) is None+++class TestBuildResearchContextStripsCitations:+ """End-to-end: a raw-mode follow-up's past_findings has no leaked markers."""++ def test_raw_mode_strips_hyperlinked_citations(self, mocker):+ # raw mode makes no LLM call; guard get_llm anyway.+ mocker.patch("local_deep_research.config.llm_config.get_llm")+ messages = [+ {+ "role": "user",+ "content": "How do plants make energy?",+ "message_type": "query",+ },+ {+ "role": "assistant",+ "content": (+ "Plants convert light [[1]](http://a) into energy in "+ "chloroplasts [arxiv.org-2](https://arxiv.org/abs/2)."+ ),+ "message_type": "response",+ "research_id": "r1",+ },+ ]+ result = _manager(messages, mode="raw").build_research_context(+ current_query="explain more"+ )+ past = result["past_findings"]+ assert "](http" not in past # no markdown link residue+ assert "[1]" not in past and "[arxiv.org-2]" not in past+ assert "Plants convert light" in past and "chloroplasts" in past+++class TestStripScopeAndRobustness:+ """Adversarial coverage: scope, over/under-strip edges, long input."""++ def test_only_prior_findings_stripped_not_original_query(self, mocker):+ """Stripping targets carried-over findings, never the user's query.++ ``original_query`` is the session's first user message verbatim; a+ bracketed token the user themselves typed must survive untouched.+ """+ mocker.patch("local_deep_research.config.llm_config.get_llm")+ messages = [+ {+ "role": "user",+ "content": "What does array[1] mean?",+ "message_type": "query",+ },+ {+ "role": "assistant",+ "content": "It indexes [[1]](http://a) the element.",+ "message_type": "response",+ "research_id": "r1",+ },+ ]+ result = _manager(messages, mode="raw").build_research_context(+ current_query="tell me more"+ )+ # User's own query preserved verbatim (not routed through the strip).+ assert result["original_query"] == "What does array[1] mean?"+ # Prior findings had their carried-over marker removed.+ assert "](http" not in result["past_findings"]++ def test_long_input_with_many_markers_is_linear(self, monkeypatch):+ """Many markers plus a glued bracket flood cost O(1) lookups each.++ Work is counted, not timed: every citation-token and verbatim-link+ lookup is tallied. Each marker needs one lookup, and the whole+ ``a[[[[...`` flood one, because a failed match advances past the+ entire opener run; retrying from every ``[`` of the run would make+ the flood alone cost one lookup (and one run scan) per bracket.+ """+ from local_deep_research.chat import citation_stripping as cs++ calls = {"n": 0}++ def counted(real):+ def wrapper(*args):+ calls["n"] += 1+ return real(*args)++ return wrapper++ monkeypatch.setattr(+ cs, "_match_citation_token", counted(cs._match_citation_token)+ )+ monkeypatch.setattr(+ cs, "_match_verbatim_link", counted(cs._match_verbatim_link)+ )+ k = 4000+ text = (+ ("word [[1]](http://a) " * k)+ + ("[1, 2] " * k)+ + "a"+ + ("[" * k)+ + " tail"+ )+ out = _manager()._strip_citation_markers(text)+ assert "](http" not in out+ assert "[1]" not in out and "[1, 2]" not in out+ assert out.count("word") == k+ # 2k markers -> 2k lookups, plus a constant for the flood.+ assert calls["n"] <= 2 * k + 16++ def test_escaped_bracket_runs_are_not_rescanned(self, monkeypatch):+ """A label full of escaped ``]`` is skipped once per link lookup.++ The verbatim-link matcher used to re-find and re-check every+ escaped ``]`` after each nested ``[``, about 6,000 escape checks+ per input character on this shape (over 1 ms per character).+ """+ from local_deep_research.chat import citation_stripping as cs++ calls = {"n": 0}+ real = cs._is_escaped++ def counted(text, pos):+ calls["n"] += 1+ return real(text, pos)++ monkeypatch.setattr(cs, "_is_escaped", counted)+ text = ("[\\]" * 330 + "](x)") * 3+ assert _manager()._strip_citation_markers(text) == text+ assert calls["n"] <= len(text)++ def test_adversarial_unclosed_brackets_terminate(self):+ """Pathological unclosed-bracket run must not hang and must be inert."""+ out = _manager()._strip_citation_markers("[" * 20000 + "1")+ # Nothing forms a complete citation, so content is returned unharmed.+ assert out.endswith("1")+++class TestSourceTaggedCitations:+ """SOURCE_TAGGED_HYPERLINKS' plain (non-hyperlinked) marker shape.++ CitationFormatter._format_source_tagged_hyperlinks emits ``[tag-N]``+ (no trailing ``(url)``) for any cited source whose URL is not http(s)+ — local files and RAG/library hits. This used to survive into+ follow-up context unstripped (the exact dangling-citation problem this+ module exists to prevent); the tag shape is covered in+ ``chat.citation_stripping`` (see ``_is_tag_id``).+ """++ @pytest.mark.parametrize(+ "text,expected",+ [+ # CitationFormatter._extract_source_label's known-tag vocabulary+ ("From the paper [arxiv-1].", "From the paper."),+ ("See the docs [local-1] here.", "See the docs here."),+ ("Per [pubmed-3] and [pmc-4].", "Per and."),+ ("Study [biorxiv-2] found this.", "Study found this."),+ ("As in [medrxiv-7] trial.", "As in trial."),+ ("Per [semantic_scholar-12] study.", "Per study."),+ ("Ref [doi-9] confirms it.", "Ref confirms it."),+ # domain-derived tags (_extract_domain always yields a dotted+ # string) also covered+ ("See [nytimes.com-9] report.", "See report."),+ ("Per [arxiv.org-1] source.", "Per source."),+ # dotless collection slugs — _slugify_collection keeps only+ # [a-z0-9-], so RAG collection tags never carry the dot the+ # old pattern required+ ("Finding [personal-notes-3].", "Finding."),+ ("From [my-papers-2] here.", "From here."),+ # digit-leading slug (collection named "2024 notes") still+ # has letters, so it is a tag — unlike a numeric range+ ("Cites [2024-notes-3] store.", "Cites store."),+ # hyperlinked slug label (linkable collection source)+ ("See [[personal-notes-3]](http://a) cited.", "See cited."),+ # mixed with the hyperlinked form in one string+ (+ "Mix [local-1] and [[arxiv-2]](http://a) both.",+ "Mix and both.",+ ),+ ],+ )+ def test_strips_plain_source_tagged_markers(self, text, expected):+ assert _manager()._strip_citation_markers(text) == expected++ @pytest.mark.parametrize(+ "text",+ [+ # Common code-indexing prose this (developer-facing) tool's+ # chat regularly carries. The tag charset is lowercase-only+ # with at least one letter and the (?<![\w.\]]) lookbehind+ # rejects markers glued to an identifier — so these are NOT+ # treated as citations (an open `[label-N]` pattern would+ # catch them — the class of over-widening #5903 fixed for the+ # sub-index pattern).+ "x = arr[i-1]",+ "data[idx-2] = 5",+ "See [RFC-2119] for definitions.",+ "Range [1 - 2] applies.",+ ],+ )+ def test_does_not_strip_code_like_or_uppercase_brackets(self, text):+ assert _manager()._strip_citation_markers(text) == text++ def test_end_to_end_raw_mode_strips_source_tagged_marker(self, mocker):+ mocker.patch("local_deep_research.config.llm_config.get_llm")+ messages = [+ {+ "role": "user",+ "content": "What's in my notes?",+ "message_type": "query",+ },+ {+ "role": "assistant",+ "content": "Your notes say the deadline is Friday [local-1].",+ "message_type": "response",+ "research_id": "r1",+ },+ ]+ result = _manager(messages, mode="raw").build_research_context(+ current_query="tell me more"+ )+ assert "[local-1]" not in result["past_findings"]+ assert "deadline is Friday" in result["past_findings"]+++class TestFullModeScopedToAssistantTurns:+ """`full` mode must strip citations from assistant turns only.++ Regression for the bug where ``build_research_context`` ran the WHOLE+ joined transcript (both roles) through the citation stripper, deleting+ any markdown link a USER typed in an earlier turn.+ """++ def test_user_typed_markdown_link_survives_full_mode(self, mocker):+ mocker.patch("local_deep_research.config.llm_config.get_llm")+ link = "[arxiv.org](https://arxiv.org/abs/2401.00001)"+ user_text = f"See {link} for context."+ # The label is citation-shaped (it names its own host), so only the+ # role scoping can keep it: a role-blind pass would delete it.+ assert link not in _manager()._strip_citation_markers(user_text)+ messages = [+ {+ "role": "user",+ "content": user_text,+ "message_type": "query",+ },+ {+ "role": "assistant",+ "content": "Noted. Light is used [[1]](http://a) in cells.",+ "message_type": "response",+ "research_id": "r1",+ },+ ]+ result = _manager(messages, mode="full").build_research_context(+ current_query="tell me more"+ )+ past = result["past_findings"]+ # The user's own link is preserved verbatim.+ assert link in past+ # The assistant's carried-over citation marker is still stripped.+ assert "[[1]](http://a)" not in past+ assert "Light is used in cells." in past++ def test_user_typed_source_tagged_shape_survives_full_mode(self, mocker):+ """A user typing something citation-shaped (e.g. quoting a prior+ answer) must not be treated as a citation either — scope is by+ role, not by pattern."""+ mocker.patch("local_deep_research.config.llm_config.get_llm")+ messages = [+ {+ "role": "user",+ "content": "You mentioned [arxiv-1] earlier, what did it say?",+ "message_type": "query",+ },+ {+ "role": "assistant",+ "content": "It found X [arxiv-1].",+ "message_type": "response",+ "research_id": "r1",+ },+ ]+ result = _manager(messages, mode="full").build_research_context(+ current_query="more"+ )+ past = result["past_findings"]+ assert "You mentioned [arxiv-1] earlier" in past+ assert "It found X." in past+++class TestWhitespacePreservedOnStrip:+ """Stripping must not corrupt unrelated indentation/whitespace.++ Regression for the bug where a blanket ``[ \\t]{2,}`` collapse ran over+ the ENTIRE findings string on every follow-up turn, flattening code+ indentation, nested list indentation and table column alignment that+ never had a citation marker anywhere near them.+ """++ def test_code_block_indentation_survives(self):+ text = "def f():\n if x:\n return data [[1]](http://a)"+ out = _manager()._strip_citation_markers(text)+ assert out == "def f():\n if x:\n return data"++ def test_nested_list_indentation_survives(self):+ text = (+ "- item one\n - nested [[1]](http://a) item\n - deeper item [2]"+ )+ out = _manager()._strip_citation_markers(text)+ assert out == ("- item one\n - nested item\n - deeper item")++ def test_table_column_alignment_survives(self):+ text = (+ "| Col A | Col B [[1]](http://a) |\n"+ "|-------|-------|\n"+ "| val1 | val2 |"+ )+ out = _manager()._strip_citation_markers(text)+ assert out == (+ "| Col A | Col B |\n|-------|-------|\n| val1 | val2 |"+ )+++class TestStripHardening:+ """Adversarial-gate cases: non-citation content survives, citations+ are removed whole.++ These were xfail'd known gaps before the strip gained code-span+ protection, a balanced-parenthesis URL matcher, and citation-shaped+ link labels.+ """++ def test_parenthesized_url_leaves_no_residue(self):+ # The URL matcher tolerates one level of balanced parentheses, so+ # a Wikipedia "(disambiguation)" URL is consumed whole instead of+ # stopping at the first ')' and leaving a stray one behind.+ out = _manager()._strip_citation_markers(+ "See [[1]](https://en.wikipedia.org/wiki/Foo_(bar)) now."+ )+ assert ")" not in out++ def test_code_block_index_not_stripped(self):+ # Fenced code is not a citation site.+ out = _manager()._strip_citation_markers("```\nx = arr[1]\n```")+ assert "arr[1]" in out++ def test_inline_code_index_not_stripped(self):+ # Neither is inline code.+ out = _manager()._strip_citation_markers("Use `results[1]` here.")+ assert "results[1]" in out++ def test_ordinary_markdown_link_survives(self):+ # A link whose label is not citation-shaped (here: it contains a+ # space, which no CitationFormatter label ever does) is verbatim+ # content, not a citation — it must keep label AND destination.+ text = (+ "Download [the dataset](https://example.com/data) before analysis."+ )+ assert _manager()._strip_citation_markers(text) == text++ def test_citation_starting_a_line_does_not_join_lines(self):+ # Whitespace consumption is horizontal-only, so a marker at the+ # start of the next line can never eat the newline(s); the+ # indentation before it is kept and the separator after it goes.+ text = "First line\n [[1]](https://example.com) second line"+ assert (+ _manager()._strip_citation_markers(text)+ == "First line\n second line"+ )+++class TestRawModeStripsBeforeTruncating:+ """Raw mode must strip citations BEFORE the 500-char truncation.++ A citation crossing the truncation cutoff is cut into a shape no+ pattern can match ("[](htt..."), so stripping AFTER truncating can+ never clean it up — extraction strips each assistant message first.+ """++ def test_citation_crossing_cutoff_leaves_no_residue(self, mocker):+ mocker.patch("local_deep_research.config.llm_config.get_llm")+ long_answer = "a" * 490 + " [[1]](https://example.com/reference) final"+ messages = [+ {+ "role": "user",+ "content": "summarize",+ "message_type": "query",+ },+ {+ "role": "assistant",+ "content": long_answer,+ "message_type": "response",+ "research_id": "r1",+ },+ ]+ result = _manager(messages, mode="raw").build_research_context(+ current_query="more"+ )+ past = result["past_findings"]+ assert "[[" not in past+ assert "[]" not in past+ assert "](http" not in past+ assert past.endswith("final")+++class TestAdjacentCitationRuns:+ """Formatter-produced ADJACENT markers must strip as one run.++ ``CitationFormatter._replace_comma_citations`` joins the formatted+ parts of a comma group with no separator, so+ ``[1, 2]`` in source-tagged mode renders as+ ``[local-1][personal-notes-2]`` (both non-linkable) or+ ``[[web-1]](url)[local-2]`` (mixed). Round 2 of the adversarial gate+ showed the round-1 ``]``-lookbehind rejecting the second tag, gluing+ debris to the preceding word; the run consumer in+ ``chat.citation_stripping`` removes the whole run. Footnote-style+ period-glued tags (``holds.[local-1]``) strip too — a period is not+ an identifier character.+ """++ @pytest.mark.parametrize(+ "text,expected",+ [+ (+ "Cites [local-1][personal-notes-2] here.",+ "Cites here.",+ ),+ (+ "Source [[web-1]](https://example.com)[local-2] says.",+ "Source says.",+ ),+ ("This holds.[local-1]", "This holds."),+ ("Group [1][2][3] chain.", "Group chain."),+ (+ "Mix [[a-1]](http://x)[[b-2]](http://y) done.",+ "Mix done.",+ ),+ ],+ )+ def test_adjacent_run_fully_stripped(self, text, expected):+ assert _manager()._strip_citation_markers(text) == expected+++class TestSingleWordOrdinaryLinksSurvive:+ """A one-word ordinary Markdown link is not a citation.++ Grounded in CitationFormatter's actual emissions: every hyperlinked+ citation is ``[[label]](url)``. The only single-bracket links stripped+ are domain labels naming the link's own host+ (``[arxiv.org](https://arxiv.org/x)``) — never a bare dotless word.+ Round 2 of the gate showed the round-1 charset label deleting+ ``[dataset](...)`` and ``[release-notes](...)`` from assistant prose.+ """++ @pytest.mark.parametrize(+ "text",+ [+ "Download [dataset](https://example.com/data) before analysis.",+ "Read [release-notes](https://example.com/releases) now.",+ "See [Dataset](https://example.com/data) too.",+ ],+ )+ def test_single_word_link_survives(self, text):+ assert _manager()._strip_citation_markers(text) == text++ @pytest.mark.parametrize(+ "text",+ [+ "Edit [config.yaml](https://github.com/o/r/blob/main/config.yaml) first.",+ "the [README.md](README.md) file",+ "Upgrade to [v1.2](https://github.com/o/r/releases/v1.2) now.",+ "Open [example.com](https://other.org/) x",+ "Docs at [docs.python.org](https://docs.python.org/3/) here.",+ ],+ )+ def test_dotted_link_text_survives_unless_it_names_its_host(self, text):+ """A dot in the label is not enough: ``[config.yaml](url)`` is link+ text. The formatter never emits a single-bracket hyperlink, so a+ dotted single-bracket label strips only when it is the link's own+ host as ``CitationFormatter`` labels it."""+ assert _manager()._strip_citation_markers(text) == text++ @pytest.mark.parametrize(+ "text,expected",+ [+ ("Per [arxiv.org](https://arxiv.org/abs/1) x", "Per x"),+ ("Per [arxiv.org-2](https://www.arxiv.org/abs/1) x", "Per x"),+ ("Per [github.com](https://github.com/o/r) x", "Per x"),+ ],+ )+ def test_dotted_link_naming_its_own_host_strips(self, text, expected):+ assert _manager()._strip_citation_markers(text) == expected+++class TestCodeProtectionCommonMark:+ """Fence/span protection must cover CommonMark variants, and a bare+ ``identifier[...]`` index in prose is not a citation marker.++ Round 2 of the gate showed tilde fences and double-backtick spans+ losing their brackets, and plain prose ``results[1]`` being stripped.+ """++ @pytest.mark.parametrize(+ "text",+ [+ "~~~python\nx = results[1]\n~~~",+ "````\ny = data[2]\n````",+ "``code `x[1]` here``",+ "The second item is results[1].",+ ],+ )+ def test_code_shapes_and_prose_index_survive(self, text):+ assert _manager()._strip_citation_markers(text) == text+++class TestNestedParenthesisUrl:+ """URLs with nested balanced parentheses strip whole (round-2 gate).++ The URL matcher balances parens up to ``_URL_MAX_DEPTH`` (well past+ any real Wikipedia-style URL), so ``Foo_(bar_(baz))`` leaves no+ ``[](...)`` residue. Past the bound the text is simply not treated as+ a URL — nothing is mangled.+ """++ def test_nested_parens_stripped_whole(self):+ out = _manager()._strip_citation_markers(+ "See [[1]](https://example.com/Foo_(bar_(baz))) now."+ )+ assert out == "See now."++ def test_depth_beyond_bound_is_inert(self):+ from local_deep_research.chat.citation_stripping import _URL_MAX_DEPTH++ depth = _URL_MAX_DEPTH + 1+ deep = "(" * depth + "x" + ")" * depth+ text = f"See [[1]]({deep}) now."+ assert _manager()._strip_citation_markers(text) == text+++class TestBracketRunIsLinear:+ """An unclosed bracket run must not backtrack quadratically.++ The round-1 ``\\[+``-prefixed link regex took ~10s at 20k brackets and+ ~90s at 60k; the scanner in ``chat.citation_stripping`` skips the+ whole bracket run in one pass (measured in milliseconds). 60k is far+ past any linear cost yet impossible for a quadratic matcher, so a+ generous bound separates the regimes without wall-clock flake.+ """++ @pytest.mark.parametrize("suffix", ["1", "`code`", "<span>value</span>"])+ def test_bracket_run_strips_linearly(self, suffix):+ """Protected spans must not cause failed prefixes to be parsed repeatedly."""+ text = "[" * 60_000 + suffix+ start = time.monotonic()+ out = _manager()._strip_citation_markers(text)+ elapsed = time.monotonic() - start+ assert out == text # inert: nothing forms a complete citation+ assert elapsed < 10.0+++class TestRoundFiveFormatterAuthorities:+ @pytest.mark.parametrize(+ "text",+ [+ "See [[LOCALHOST]](http://LOCALHOST/a) now.",+ "See [[LOCALHOST-1]](http://LOCALHOST/a) now.",+ ],+ )+ def test_uppercase_dotless_authority_citations_strip(self, text):+ out = _manager()._strip_citation_markers(text)++ assert out == "See now."+ assert markdown(out) == "<p>See now.</p>"+++class TestRoundFiveMarkdownStructures:+ @pytest.mark.parametrize(+ "text,rendered_fragment",+ [+ (+ "[dataset](/page/[1]/details)",+ 'href="/page/[1]/details"',+ ),+ (+ "[1]: /page/[2]/details\n\n[x][1]",+ 'href="/page/[2]/details"',+ ),+ (+ "> [1]: https://example.com\n\n> [x][1]",+ '<blockquote>\n<p><a href="https://example.com">x</a></p>',+ ),+ (+ "[see [1]](https://example.com)",+ ">see [1]</a>",+ ),+ (+ "![alt\n[1]](pic.jpg)",+ 'alt="alt\n[1]" src="pic.jpg"',+ ),+ ],+ )+ def test_valid_markdown_structures_survive(self, text, rendered_fragment):+ out = _manager()._strip_citation_markers(text)++ assert out == text+ assert rendered_fragment in markdown(out)+++class TestRoundFiveRenderedSeparation:+ def test_link_before_removed_citation_keeps_word_boundary(self):+ text = "[left](x)[[1]](u)right"++ out = _manager()._strip_citation_markers(text)++ assert out == "[left](x) right"+ assert markdown(out) == '<p><a href="x">left</a> right</p>'+++class TestRoundFiveCodeProtection:+ @pytest.mark.parametrize(+ "text,rendered_fragment",+ [+ (+ "Use `first\n[1] second` now.",+ "<code>first\n[1] second</code>",+ ),+ ("```\n[1]", "[1]"),+ ("<pre>[1]</pre>", "<pre>[1]</pre>"),+ ],+ )+ def test_multiline_and_block_code_survive(self, text, rendered_fragment):+ out = _manager()._strip_citation_markers(text)++ assert out == text+ assert rendered_fragment in markdown(out, extensions=["fenced_code"])+++class TestRoundSixMarkdownCodeComposition:+ @pytest.mark.parametrize(+ "text",+ [+ "Click [see `code` [1]](https://example.com).",+ "![alt `x` [1]](pic.jpg)",+ ],+ )+ def test_link_or_image_with_inline_code_survives(self, text):+ assert _manager()._strip_citation_markers(text) == text+++class TestRoundSixHtmlLookingInlineCode:+ @pytest.mark.parametrize(+ "text,expected",+ [+ (+ "Run `<code>foo` then see [1].\n\nNext paragraph [2] more.",+ "Run `<code>foo` then see.\n\nNext paragraph more.",+ ),+ (+ "Run `first\n<code>[1]</code>\nlast` then see [2].",+ "Run `first\n<code>[1]</code>\nlast` then see.",+ ),+ ],+ )+ def test_html_looking_tag_in_code_cannot_disable_later_stripping(+ self, text, expected+ ):+ assert _manager()._strip_citation_markers(text) == expected+++class TestRoundSixRemainingMarkdownBlocks:+ @pytest.mark.parametrize(+ "text",+ [+ "[ref]:\n /page/[1]/details\n\n[x][ref]",+ "> [ref]:\n> /page/[1]/details\n>\n> [x][ref]",+ "- [ref]:\n /page/[1]/details\n\n [x][ref]",+ "- [ref]:\n\t/page/[1]/details\n\n [x][ref]",+ ],+ )+ def test_multiline_relative_reference_destination_survives(self, text):+ out = _manager()._strip_citation_markers(text)++ assert out == text+ assert 'href="/page/[1]/details"' in markdown(out)++ @pytest.mark.parametrize("indent", [" ", "\t"])+ def test_indented_code_after_fence_survives(self, indent):+ text = f"```\nx\n```\n{indent}[1]"++ out = _manager()._strip_citation_markers(text)++ assert out == text+ assert markdown(out, extensions=["fenced_code"]).count("<code>") == 2+++class TestRoundSixMirroredBacktickScaling:+ def test_mirrored_run_lengths_scale_linearly(self):+ elapsed = []+ for size in (800, 1600):+ half = "x".join("`" * length for length in range(1, size + 1))+ text = f"{half}\n\n{half}"+ start = time.monotonic()++ out = _manager()._strip_citation_markers(text)++ elapsed.append(time.monotonic() - start)+ assert out == text++ assert elapsed[1] < elapsed[0] * 6+++class TestRoundSevenFormatterTagsAndLinks:+ @pytest.mark.parametrize(+ "text,expected",+ [+ ("See [123-1] now.", "See [123-1] now."),+ ("See [[123-1]](https://example.com) now.", "See now."),+ ],+ )+ def test_numeric_collection_tags_strip(self, text, expected):+ """Bare numeric tags also represent factual ranges; only links disambiguate."""+ assert _manager()._strip_citation_markers(text) == expected++ @pytest.mark.parametrize(+ "text",+ [+ "x = arr[123-1]",+ "[label \\] [1]](u)",+ "![alt \\] [1]](pic.jpg)",+ ],+ )+ def test_array_and_escaped_label_structures_survive(self, text):+ out = _manager()._strip_citation_markers(text)++ assert out == text+ assert markdown(out) == markdown(text)+++class TestRoundSevenCodeAndReferenceBlocks:+ def test_crlf_blank_paragraph_ends_inline_code_pairing(self):+ text = "`before\r\n\r\n[1]` after [2]"++ out = _manager()._strip_citation_markers(text)++ assert out == "`before\r\n\r\n` after"++ @pytest.mark.parametrize(+ "text",+ [+ '[ref]: /page\n "title [1]"\n\n[x][ref]',+ '> [ref]: /page\n> "title [1]"\n>\n> [x][ref]',+ '- [ref]: /page\n "title [1]"\n\n [x][ref]',+ '[ref]:\n /page\n "title [1]"\n\n[x][ref]',+ ],+ )+ def test_multiline_reference_titles_survive(self, text):+ out = _manager()._strip_citation_markers(text)++ assert out == text+ assert 'title="title [1]"' in markdown(out)++ @pytest.mark.parametrize(+ "text",+ [+ "# heading\n [1]",+ "---\n [1]",+ "heading\n---\n [1]",+ "<div>block</div>\n [1]",+ "<div>\nblock\n</div>\n [1]",+ ],+ )+ def test_indented_code_after_completed_block_survives(self, text):+ out = _manager()._strip_citation_markers(text)++ assert out == text+ assert "<code>[1]" in markdown(out)+++class TestRoundSevenHtmlContexts:+ def test_code_tag_text_inside_html_comment_does_not_open_code(self):+ text = "<!-- mentions <code> only -->\nProse [1] continues."++ assert _manager()._strip_citation_markers(text) == (+ "<!-- mentions <code> only -->\nProse continues."+ )++ def test_backtick_in_fence_cannot_mask_later_html_code(self):+ text = "~~~\n`\n~~~\n<code>[1]</code> `x`"++ out = _manager()._strip_citation_markers(text)++ assert out == text+ assert "<code>[1]</code>" in markdown(out, extensions=["fenced_code"])++ def test_html_comment_contents_survive_while_following_prose_strips(self):+ text = "<!-- hidden [1] -->\nProse [2] continues."++ assert _manager()._strip_citation_markers(text) == (+ "<!-- hidden [1] -->\nProse continues."+ )++ def test_backtick_in_comment_cannot_mask_later_html_code(self):+ text = "<!-- ` -->\n<code>[1]</code> `x`"++ assert _manager()._strip_citation_markers(text) == text++ def test_backtick_inside_comment_cannot_mask_later_html_code(self):+ text = "<!-- ` -->\n<code>[1]</code> `x`"++ out = _manager()._strip_citation_markers(text)++ assert out == text++ def test_backtick_inside_comment_cannot_mask_later_prose(self):+ text = "<!-- ` -->\nProse [1] `x`"++ assert (+ _manager()._strip_citation_markers(text) == "<!-- ` -->\nProse `x`"+ )+++class TestRoundEightParserContexts:+ @pytest.mark.parametrize(+ "text,expected",+ [+ ("See [1-1] now.", "See [1-1] now."),+ ("See [[1-1]](https://example.com) now.", "See now."),+ ("See [7-42] now.", "See [7-42] now."),+ ],+ )+ def test_single_digit_numeric_collection_tags_strip(self, text, expected):+ """Without provenance, preserving numeric ranges takes precedence."""+ assert _manager()._strip_citation_markers(text) == expected++ @pytest.mark.parametrize(+ "text",+ [+ 'Read [see [1]](/page "a title") now.',+ "Read [see [1]](/page 'a title') now.",+ 'Read [see [1]](<page target> "a title") now.',+ "Read [see [1]][ref] now.\n\n[ref]: /page",+ "Read [see [1]][] now.",+ ],+ )+ def test_titled_and_reference_markdown_links_survive(self, text):+ out = _manager()._strip_citation_markers(text)++ assert out == text+ assert markdown(out) == markdown(text)++ @pytest.mark.parametrize(+ "text",+ [+ '<a href="/page/[1]/details">dataset</a>',+ '<a data-x="[1]">x</a>',+ '<a data-x="> [1]">x</a>',+ '<img alt="[1]" src="x">',+ ],+ )+ def test_citation_shapes_inside_html_attributes_survive(self, text):+ assert _manager()._strip_citation_markers(text) == text++ @pytest.mark.parametrize(+ "text",+ [+ "> [1]",+ "> ~~~\n> [1]\n> ~~~",+ "> ```python\n> [1]\n> ```",+ ],+ )+ def test_blockquote_code_blocks_survive(self, text):+ out = _manager()._strip_citation_markers(text)++ assert out == text+ assert "[1]" in markdown(out, extensions=["fenced_code"])++[email protected]("mode", ["raw", "full"])+def test_factual_ranges_survive_context_assembly(mode):+ """Removing bare numeric collection tags also erased unambiguous year facts."""+ messages = [+ {"role": "user", "content": "Which years were observed?"},+ {+ "role": "assistant",+ "research_id": "synthetic",+ "content": "The window is [2020-2024]. Evidence [[1]](https://example.com/a).",+ },+ ]+ past = _manager(messages, mode=mode).build_research_context()[+ "past_findings"+ ]+ assert "The window is [2020-2024]." in past+ assert "[[1]]" not in past+++def test_summary_citation_cleanup_precedes_output_cap(mocker):+ """A citation straddling the summary cap must not become an incomplete URL."""+ from types import SimpleNamespace++ answer = "x" * 1990 + " [[1]](https://example.org/source) final"+ model = SimpleNamespace(+ invoke=lambda prompt: SimpleNamespace(content=answer)+ )+ mocker.patch(+ "local_deep_research.config.llm_config.get_llm", return_value=model+ )+ manager = _manager(+ [+ {+ "role": "assistant",+ "research_id": "r",+ "content": "Prior fact [1].",+ }+ ],+ mode="summary",+ )+ result = manager.build_research_context("Tell me more")["past_findings"]+ assert result == "x" * 1990 + " final"+++def test_summary_truncation_does_not_rescan_partial_code(mocker):+ """Cleanup after truncation would mistake the now-unclosed code for prose."""+ from types import SimpleNamespace++ answer = "`literal [1] " + "x" * 2100 + "`"+ model = SimpleNamespace(+ invoke=lambda prompt: SimpleNamespace(content=answer)+ )+ mocker.patch(+ "local_deep_research.config.llm_config.get_llm", return_value=model+ )+ manager = _manager(+ [+ {+ "role": "assistant",+ "research_id": "r",+ "content": "Prior fact [1].",+ }+ ],+ mode="summary",+ )+ result = manager.build_research_context("Tell me more")["past_findings"]+ assert result.startswith("`literal [1] ")+ assert len(result) <= manager.CONTEXT_SUMMARY_MAX_CHARS + 3+++class TestAdversarialGateBoundaryFixes:+ """Revert-catching tests for the adversarial-gate round of fixes.++ Each test pins one defect the gate reproduced at the PR head: a+ multiline HTML code opening tag, a non-1 ordered marker splitting a+ code-span paragraph, outer trims running before citation cleanup,+ an unclosed fence inside a code element, and word-joins after a+ protected span.+ """++ def test_multiline_code_opening_tag_protects_element_contents(self):+ """The ``>`` of the opening tag sits on the next line.++ Per-line tag scanning never recognized the element, so the code+ contents were scanned as prose and ``[1, 2]`` was deleted.+ """+ text = 'See <code\nclass="py">values = [1, 2]\n</code> end'++ out = _manager()._strip_citation_markers(text)++ assert out == text+ assert markdown(out) == markdown(text)++ def test_ordered_marker_other_than_one_is_lazy_continuation(self):+ """CommonMark: only ``1.`` (or a bullet) can interrupt a paragraph.++ ``2.`` used to split the backtick paragraph, leaving both values+ deleted and stray unpaired backticks behind.+ """+ text = "Set `x = [1]\n2. y = [2]` done"++ out = _manager()._strip_citation_markers(text)++ assert out == text++ def test_context_modes_strip_markers_before_outer_trim(self, mocker):+ """Full mode trimmed indentation before cleanup.++ The trim dedented a leading indented-code line into prose, so its+ bracket literals were deleted as citations (raw mode never+ trimmed, so the modes diverged). Summary mode cannot keep a+ LEADING indented line: ``ProcessingLLMWrapper`` trims every+ response before the summarizer sees it. The model below returns+ what that wrapper delivers, and code later in the summary must+ still be protected while its markers strip.+ """+ content = " data = [1, 2, 3]\n keep = [4]\nDone."+ messages = [+ {"role": "user", "content": "show the data"},+ {+ "role": "assistant",+ "research_id": "r",+ "content": content,+ },+ ]+ full = _manager(messages, mode="full").build_research_context("more")[+ "past_findings"+ ]+ assert "data = [1, 2, 3]" in full+ assert "keep = [4]" in full++ from types import SimpleNamespace++ # Already trimmed, as ProcessingLLMWrapper._normalize_response+ # (remove_think_tags + strip) delivers it in production.+ delivered = (+ "Summary:\n\n data = [1, 2, 3]\n keep = [4]\n\nDone [1]."+ )+ model = SimpleNamespace(+ invoke=lambda prompt: SimpleNamespace(content=delivered)+ )+ mocker.patch(+ "local_deep_research.config.llm_config.get_llm", return_value=model+ )+ summary = _manager(messages, mode="summary").build_research_context(+ "more"+ )["past_findings"]+ assert summary == (+ "Summary:\n\n data = [1, 2, 3]\n keep = [4]\n\nDone."+ )++ def test_unclosed_fence_inside_code_element_still_strips_afterwards(self):+ """A fence glyph inside ``<code>`` is element content, not a fence.++ Recognizing it as an unclosed fence used to protect the rest of+ the document, so the dangling ``[2]`` after the element survived+ — the exact residue this stripper exists to remove.+ """+ text = "<code>\n```\nfmt = [1]\n</code>\nSee [2] here"++ out = _manager()._strip_citation_markers(text)++ assert out == "<code>\n```\nfmt = [1]\n</code>\nSee here"++ @pytest.mark.parametrize(+ "text,expected",+ [+ ("Run <code>x</code>[1]now", "Run <code>x</code> now"),+ ("q <b>z</b>[7]w tail", "q <b>z</b> w tail"),+ (+ "a <code>1</code>[2]b <code>3</code>",+ "a <code>1</code> b <code>3</code>",+ ),+ ],+ )+ def test_citation_after_protected_span_retains_separator(+ self, text, expected+ ):+ """One separator survives when a run directly follows a span.++ The tail and per-segment strip calls had no left context, so+ removing the citation joined the neighbouring rendered words.+ """+ out = _manager()._strip_citation_markers(text)++ assert out == expected+ assert markdown(out) == markdown(expected)+++class TestAdversarialGateOverlapAndBlockEnds:+ """Revert-catching tests for the second adversarial-gate round.++ Each test pins one defect the gate reproduced at the PR head: a+ ``<!--``/backtick inside an HTML tag's quoted attribute (or inside a+ Markdown link) starting a phantom span that duplicated text and+ shielded citations, a thematic break line never ending a backtick+ paragraph, and a setext underline never ending one either.+ """++ @pytest.mark.parametrize(+ "text,expected",+ [+ ('<a title="<!--">x [1]', '<a title="<!--">x'),+ ("[x](y<!--z)q [1]", "[x](y<!--z)q"),+ ],+ )+ def test_overlapping_phantom_span_does_not_duplicate_or_shield(+ self, text, expected+ ):+ """A ``<!--`` inside a tag attribute or link destination is inert.++ The phantom comment span started inside the already-emitted+ verbatim tag/link and ran past it, so the stitcher re-emitted+ text[start:end] — duplicating the overlap — and protected the+ ``[1]`` after it.+ """+ out = _manager()._strip_citation_markers(text)++ assert out == expected+ assert markdown(out) == markdown(expected)++ def test_thematic_break_line_ends_backtick_paragraph(self):+ """``_backtick_runs`` feeds keepends lines; ``---\\n`` never+ fullmatched the thematic-break regex, so the code span paired+ across it and kept both markers verbatim.+ """+ text = "`a [1]\n***\nb [2]` tail"++ out = _manager()._strip_citation_markers(text)++ assert out == "`a\n***\nb ` tail"+ assert markdown(out) == markdown("`a\n***\nb ` tail")++ def test_setext_underline_ends_backtick_paragraph(self):+ """An ``=====`` underline closes the paragraph above it.++ CommonMark never lets a code span pair across a setext heading;+ the module kept the span, shielding both markers inside it.+ """+ text = "`a [1]\n=====\nb [2]` tail"++ out = _manager()._strip_citation_markers(text)++ assert out == "`a\n=====\nb ` tail"+ assert markdown(out) == markdown("`a\n=====\nb ` tail")+++class TestAdversarialGateLazyReentryDefinitionsTagFloor:+ """Revert-catching tests for the third adversarial-gate round.++ Each test pins one defect the gate reproduced at the PR head: a+ ``>`` re-entry after a lazy continuation line splitting the+ blockquoted paragraph (deleting a marker inside a real code span),+ pseudo reference definitions protecting their whole line, and+ single-letter slug tags (array indexing) deleted from prose.+ """++ def test_lazy_quote_reentry_keeps_code_span_paragraph(self):+ """One blockquoted paragraph spans the lazy line and the re-entry.++ CommonMark keeps ``> one `code``, ``lazy [1]`` and ``> end` tail``+ in a single blockquoted paragraph, so the code span pairs across+ all three lines and ``[1]`` renders inside it. The old+ ``depth > previous_depth`` split ended the run at the re-entry,+ leaving both backticks unpaired and ``[1]`` deleted.+ """+ text = "> one `code\nlazy [1]\n> end` tail"++ out = _manager()._strip_citation_markers(text)++ assert out == text+ assert "<code>code\nlazy [1]\nend</code>" in markdown(out)++ @pytest.mark.parametrize(+ "text,expected",+ [+ ("[Note]: See [1] for details.", "[Note]: See for details."),+ (+ "[TL;DR]: summary [1] of the answer [2] follows.",+ "[TL;DR]: summary of the answer follows.",+ ),+ ],+ )+ def test_pseudo_reference_definition_strips_markers(self, text, expected):+ """A space-bearing destination with no valid title is prose.++ ``[Note]: See [1] for details.`` is not a CommonMark reference+ definition, so it must not protect its whole line verbatim —+ only genuine definitions (validated destination, optional+ well-formed title) pass through untouched.+ """+ assert _manager()._strip_citation_markers(text) == expected++ def test_glued_short_slug_indexing_survives(self):+ """``dp[i-1]`` is indexing: the word-glue guard, not a slug-length+ floor, protects it (a one-letter collection slug is a real tag;+ see ``TestShortCollectionSlugTags``)."""+ text = "compare dp[i-1] with dp[i] and x[1][2]"++ out = _manager()._strip_citation_markers(text)++ assert out == text+ assert markdown(out) == markdown(text)+++class TestAdversarialGateIndentColumnsSetextSpans:+ """Revert-catching tests for the fourth adversarial-gate round.++ Each test pins one defect the gate reproduced at the PR head:+ indented code decided from column zero of the quote-stripped line+ instead of the containing block's content column, a setext+ underline missing from the main block-boundary rule, and comment+ masking computing inline code spans per line so a multi-line+ span's comment opener latched the comment state.+ """++ @pytest.mark.parametrize(+ "text,expected",+ [+ # five spaces past the marker: the item's content is code+ ("- [1]", "- [1]"),+ ("1. [1]", "1. [1]"),+ # tabs advance to column 8, six past the content column+ ("- \t\t[1]", "- \t\t[1]"),+ # a tab reaching column 4 is top-level indented code+ (" \t[1]", " \t[1]"),+ ("> \t[1]", "> \t[1]"),+ # controls: four spaces after a marker (or a single tab,+ # which only reaches two past the content column) stay a+ # paragraph continuation and strip+ ("- [1]", "-"),+ ("- [1]", "-"),+ ("-\t[1]", "-"),+ (" [1]", ""),+ ],+ )+ def test_indent_measured_from_content_column(self, text, expected):+ """Indented code is decided against the containing block's column.++ The old check inspected the quote-stripped body from column+ zero, so a citation CommonMark renders as indented code inside a+ list item (or after tab indentation) was deleted — leaving only+ the list marker, or an empty line.+ """+ assert _manager()._strip_citation_markers(text) == expected++ @pytest.mark.parametrize(+ "text",+ [+ "Values\n=====\n [1] xs[2]",+ "Values\n-----\n [1] xs[2]",+ ],+ )+ def test_setext_underline_ends_block_for_indented_code(self, text):+ """A setext heading underline completes the paragraph's block.++ The line after ``=====`` is genuinely indented code (the+ heading ends the paragraph), but the main block-boundary rule+ had no setext clause — only the backtick-run helper did — so+ the bracketed index expressions below the underline were+ stripped. The dashes variant already ended the block via the+ thematic-break rule.+ """+ out = _manager()._strip_citation_markers(text)++ assert out == text+ assert "<code>[1] xs[2]" in markdown(out)++ @pytest.mark.parametrize(+ "text",+ [+ "`span <!-- continues\ncrossing` text\n~~~\n[1]\n~~~",+ "`same <!-- line` text\n~~~\n[1]\n~~~",+ ],+ )+ def test_multiline_code_span_comment_cannot_mask_fence(self, text):+ """Comment masking must compute code spans over the whole text.++ The per-line computation missed a span continuing onto the next+ line, so a ``<!--`` inside it latched the comment state and+ masked the fence further down; the fenced ``[1]`` was then+ stripped as prose. A span opened and closed on one line already+ masked correctly.+ """+ out = _manager()._strip_citation_markers(text)++ assert out == text+ assert "<code>[1]" in markdown(out, extensions=["fenced_code"])+++class TestAdversarialGateType7CodeAndSpecIndents:+ """Revert-catching tests for the fifth adversarial-gate round.++ ``<code>`` is a CommonMark HTML block type 7: it cannot interrupt a+ paragraph, its block ends at the first blank line, and a mid-line or+ trailing-content occurrence is inline HTML inside the paragraph.+ The stripper previously latched element state from any ``<code>``+ mention, disabling stripping (and fence recognition) for the rest+ of the answer. ``<pre>`` is type 1 and keeps its blank-spanning+ behaviour. Separately, reference-definition continuation lines may+ carry indent 0 (the old "at least one column" floor rejected the+ spec-legal indent-0 destination line, deleting the rendered link+ and mangling list-item definitions).+ """++ @pytest.mark.parametrize(+ "text,expected",+ [+ # prose mentioning the literal tag, then a fence, then prose+ (+ "Use the <code> tag\n```\nx[1]\n```\nmore [2] text",+ "Use the <code> tag\n```\nx[1]\n```\nmore text",+ ),+ # an unclosed type-7 element ends at the blank line+ ("<code>\n\nsome [1] text", "<code>\n\nsome text"),+ # balanced inline pair: the following indented line is a+ # lazy paragraph continuation, not indented code+ ("<code>x</code>\n [1]", "<code>x</code>\n"),+ ],+ )+ def test_type7_code_element_cannot_latch_block_state(self, text, expected):+ """A ``<code>`` mention no longer freezes the rest of the answer.++ Before the fix, any line containing the tag entered element+ state that survived blank lines, so a later fence was never+ recognized and trailing citations leaked verbatim.+ """+ assert _manager()._strip_citation_markers(text) == expected++ def test_pre_element_still_spans_blank_lines(self):+ """``<pre>`` is type 1: a blank line does not end its block."""+ text = "<pre>\n\nsome [1] text"++ assert _manager()._strip_citation_markers(text) == text++ @pytest.mark.parametrize(+ "text",+ [+ # indent-0 destination line (spec-legal continuation)+ "[1]:\n/url\nlater [1] ref",+ # lazy destination line inside a list item+ "- [1]:\n /url\nlater [1] x",+ ],+ )+ def test_multiline_reference_definitions_at_spec_indents(self, text):+ """Multi-line definitions keep their rendered reference links.++ The old continuation floor rejected these lines, so the label+ never registered: the later ``[1]`` was stripped (deleting the+ rendered link) and the list-item definition line was mangled to+ its marker and colon.+ """+ out = _manager()._strip_citation_markers(text)++ assert out == text+++class TestReferenceContinuationIndentsAndBlockOpeners:+ """Revert-catching tests for reference-definition continuation lines.++ A definition's destination or title line continues the definition's+ paragraph, and indented code cannot interrupt a paragraph, so+ CommonMark sets no indentation limit on that line; a lazy line with+ fewer ``>`` markers continues it too. The previous three-column cap+ was measured from column 0, so a list item's content column+ (``- [ref]:`` then four spaces or a tab), any deeper indent and lazy+ quote lines were rejected, and the literal ``[1]`` in the rendered+ destination or title was stripped. Conversely a line that opens a+ fence or ``<pre>`` element interrupts the paragraph: consuming it as+ the destination exposed the block's code to stripping.+ """++ @pytest.mark.parametrize(+ "text",+ [+ "[ref]:\n /page/[1]/details\n\n[x][ref]",+ "[ref]:\n /page/[1]/details\n\n[x][ref]",+ "[ref]:\n\t/page/[1]/details\n\n[x][ref]",+ "- a\n - [ref]:\n /page/[1]/details\n\n [x][ref]",+ "> [ref]:\n> /page/[1]/details\n>\n> [x][ref]",+ # lazy continuation line without the ``>`` prefix+ "> [ref]:\n /page/[1]/details\n\n> [x][ref]",+ ],+ )+ def test_destination_line_at_any_indent_survives(self, text):+ out = _manager()._strip_citation_markers(text)++ assert out == text+ assert 'href="/page/[1]/details"' in markdown(out)++ def test_lazy_title_line_survives(self):+ text = "> [ref]: /url\n'ti [1] tle'\n\n> [x][ref]"++ out = _manager()._strip_citation_markers(text)++ assert out == text+ assert 'title="ti [1] tle"' in markdown(out)++ def test_deeper_quote_line_does_not_continue_a_definition(self):+ """A nested ``>`` interrupts the paragraph, so its text is prose."""+ text = "> [ref]:\n> > /page/[1]/details\n\n> [x][ref]"++ out = _manager()._strip_citation_markers(text)++ assert out == "> [ref]:\n> > /page//details\n\n> [x][ref]"++ @pytest.mark.parametrize(+ "text",+ [+ "[ref]:\n```\ncode [1]\n```",+ "[ref]:\n~~~\ncode [1]\n~~~",+ "[ref]:\n<pre>\ncode [1]\n</pre>",+ "- [ref]:\n ```\n code [1]\n ```",+ # opener two columns past the list content column: still a+ # fence or <pre> block whose content sits at that column+ "- [ref]:\n ```\n code [1]\n ```",+ "- [ref]:\n <pre>\n code [1]\n </pre>",+ ],+ )+ def test_fence_or_pre_opener_never_continues_a_definition(self, text):+ out = _manager()._strip_citation_markers(text)++ assert out == text++ def test_code_element_line_still_continues_a_definition(self):+ """``<code>`` (HTML block type 7) cannot interrupt a paragraph.++ The line is the definition's destination, so the next line is+ prose and its marker strips.+ """+ text = "[ref]:\n<code>\nx [1]\n</code>"++ out = _manager()._strip_citation_markers(text)++ assert out == "[ref]:\n<code>\nx\n</code>"+++class TestWhitespaceAroundLineLeadingAndTrailingRuns:+ """Only one separator flanking a stripped run is removed."""++ @pytest.mark.parametrize(+ "text,expected",+ [+ # Indentation of a list item's continuation paragraph is kept;+ # the separator after the run goes instead.+ (+ "1. First item.\n\n [1] Second paragraph.\n2. Next",+ "1. First item.\n\n Second paragraph.\n2. Next",+ ),+ ("a\n\t[1] b", "a\n\tb"),+ # A run at the start of a line drops the space after it rather+ # than leaving the next word indented.+ ("Line one\n[1] Line two", "Line one\nLine two"),+ ("[1] Opening line", "Opening line"),+ ],+ )+ def test_line_leading_run_keeps_indentation(self, text, expected):+ assert _manager()._strip_citation_markers(text) == expected++ @pytest.mark.parametrize(+ "text,expected",+ [+ # Keeping the two spaces would create a Markdown hard break.+ ("Text [1]\nnext", "Text\nnext"),+ ("Text [[1]](https://a.b/c)\nnext", "Text\nnext"),+ ("Text [1]\r\nnext", "Text\r\nnext"),+ ("x\n [1]\ny", "x\n\ny"),+ ],+ )+ def test_line_ending_run_leaves_no_trailing_whitespace(+ self, text, expected+ ):+ out = _manager()._strip_citation_markers(text)+ assert out == expected+ assert markdown(out) == markdown(expected)+++class TestStrippingFailsSafe:+ """An unexpected stripping error must not fail every later follow-up."""++ @pytest.mark.parametrize("mode", ["raw", "full"])+ def test_error_carries_text_unstripped(self, mocker, mode):+ mocker.patch(+ "local_deep_research.chat.context.strip_prior_turn_citations",+ side_effect=RuntimeError("boom"),+ )+ messages = [+ {"role": "user", "content": "q"},+ {+ "role": "assistant",+ "research_id": "r",+ "content": "Fact [1].",+ },+ ]+ past = _manager(messages, mode=mode).build_research_context("more")[+ "past_findings"+ ]+ assert "Fact [1]." in past++ def test_error_in_summary_cleanup_keeps_the_summary(self, mocker):+ from types import SimpleNamespace++ model = SimpleNamespace(+ invoke=lambda prompt: SimpleNamespace(content="Summary [1].")+ )+ mocker.patch(+ "local_deep_research.config.llm_config.get_llm", return_value=model+ )+ mocker.patch(+ "local_deep_research.chat.context.strip_prior_turn_citations",+ side_effect=RuntimeError("boom"),+ )+ messages = [+ {"role": "user", "content": "q"},+ {+ "role": "assistant",+ "research_id": "r",+ "content": "Fact [1].",+ },+ ]+ past = _manager(messages, mode="summary").build_research_context(+ "more"+ )["past_findings"]+ assert past == "Summary [1]."+++class TestBareUrlGuard:+ """Only a citation INSIDE a bare URL is kept; a ``)`` closing a Markdown+ link destination, or a line break, ends the URL."""++ @pytest.mark.parametrize(+ "text,expected",+ [+ (+ "Read [the paper](https://x.org)[1].",+ "Read [the paper](https://x.org).",+ ),+ (+ "[link](http://a.com)[[1]](http://b.com) next",+ "[link](http://a.com) next",+ ),+ ("(https://a.com)[1] now", "(https://a.com) now"),+ ("Claim https://x.org/a\n[1] more", "Claim https://x.org/a\nmore"),+ ("a [[1]](https://x.org/a)\n[[2]](https://x.org/b)", "a\n"),+ ],+ )+ def test_citation_after_a_url_strips(self, text, expected):+ assert _manager()._strip_citation_markers(text) == expected++ @pytest.mark.parametrize(+ "text",+ [+ "see https://x.org/a[1] here",+ "see https://en.wikipedia.org/wiki/F_(b)[1] here",+ ],+ )+ def test_brackets_inside_a_bare_url_survive(self, text):+ assert _manager()._strip_citation_markers(text) == text++ def test_url_guard_scans_each_chunk_once(self, monkeypatch):+ """The guard used to run a ``$``-anchored search over a 4096-char+ window per candidate (quadratic per token: 16 KB took ~5 s). Bare+ URLs are now found in one pass per prose chunk, counted here in+ characters scanned."""+ from local_deep_research.chat import citation_stripping as cs++ scanned = {"chars": 0}+ real = cs._bare_url_spans++ def counted(text):+ scanned["chars"] += len(text)+ return real(text)++ monkeypatch.setattr(cs, "_bare_url_spans", counted)+ text = ("http://" * 500 + " " + "[1]," * 150 + "\n") * 4+ text += "see https://a.org/x[1] and (https://b.org)[2] `c` [3]\n" * 300+ out = _manager()._strip_citation_markers(text)+ assert "[2]" not in out and "[3]" not in out+ assert out.count("https://a.org/x[1]") == 300+ assert scanned["chars"] <= 3 * len(text)+++class TestShortCollectionSlugTags:+ """``_slugify_collection`` maps "AI" -> "ai" and "X" -> "x", and the+ formatter emits ``[{slug}-{n}]`` for non-http sources, so a 1- or+ 2-character slug is a real tag."""++ @pytest.mark.parametrize(+ "text,expected",+ [+ ("Alpha [ai-1] beta.", "Alpha beta."),+ ("Alpha [x-1] beta.", "Alpha beta."),+ (+ "Alpha [ai-1][[arxiv-2]](https://arxiv.org/abs/9) beta",+ "Alpha beta",+ ),+ ("Alpha [[ml-3]](https://a.org/x) beta", "Alpha beta"),+ ],+ )+ def test_short_slug_tags_strip(self, text, expected):+ assert _manager()._strip_citation_markers(text) == expected+++class TestSingleBracketNumericLinksSurvive:+ """The formatter never emits a single-bracket numeric link, and the+ summarizer reads a transcript whose assistant citations are already+ stripped; ``[1998](url)`` is a year, ``[1](#step-1)`` an anchor."""++ @pytest.mark.parametrize(+ "text",+ [+ "Founded in [1998](https://en.wikipedia.org/wiki/1998).",+ "Step [1](#step-1) then",+ "Top [10](https://x.org/top10) list",+ "See [1, 2](https://x.org/a) too",+ ],+ )+ def test_numeric_label_link_survives(self, text):+ assert _manager()._strip_citation_markers(text) == text+++class TestLineLeadingBibliographyEntries:+ """``[1]: ...`` at the start of a line's content is a bibliography+ entry, judged per line through container and heading markers, not by+ the marker's offset in the prose chunk."""++ @pytest.mark.parametrize(+ "text",+ [+ "Text\n[1]: x",+ "Sources\n- [1]: Smith 2020",+ "## [1]: Intro",+ "> 1. [2]: quoted entry",+ "Intro\n * [3]: indented entry",+ ],+ )+ def test_line_leading_entry_survives(self, text):+ assert _manager()._strip_citation_markers(text) == text++ def test_mid_line_colon_marker_still_strips(self):+ out = _manager()._strip_citation_markers("Note [1]: mid-line")+ assert out == "Note: mid-line"+++class TestUnspacedScripts:+ """CJK text is written without spaces, so a glued marker is a+ citation, and removing it must not insert a space."""++ @pytest.mark.parametrize(+ "text,expected",+ [+ ("東京は首都です[1]。次", "東京は首都です。次"),+ ("東京は首都です [1]。次", "東京は首都です。次"),+ ("東京[1]次", "東京次"),+ ("東京[[1]](https://a.org/x)次", "東京次"),+ ],+ )+ def test_cjk_glued_citation_strips(self, text, expected):+ assert _manager()._strip_citation_markers(text) == expected+++class TestReferenceLabelsHoldNoBrackets:+ def test_citation_after_bracketed_text_strips(self):+ """``[Table 2][[1]](url)`` is not a full reference link (a reference+ label cannot contain ``[``), so the citation strips."""+ out = _manager()._strip_citation_markers(+ "Shown in [Table 2][[1]](https://a.org/x) here."+ )+ assert out == "Shown in [Table 2] here."++ def test_real_full_reference_link_survives(self):+ text = "See [text][1] ok\n\n[1]: https://a.org/x"+ assert _manager()._strip_citation_markers(text) == text+++class TestSeparatorBetweenWords:+ @pytest.mark.parametrize(+ "text,expected",+ [+ ("Claim.[[1]](https://a.com)Next", "Claim. Next"),+ ("word[[1]](https://a.com)next", "word next"),+ ("x</b>[[1]](https://a.com)y", "x</b> y"),+ # Path-like and punctuation joins stay joined.+ ("/page/[[1]](https://a.com)/x", "/page//x"),+ ("Founded [[1]](https://a.com).", "Founded."),+ ("(see[[1]](https://a.com))", "(see)"),+ ],+ )+ def test_space_only_between_words(self, text, expected):+ assert _manager()._strip_citation_markers(text) == expected+++class TestDoubleBracketLabelsTheFormatterEmits:+ """Every label shape ``CitationFormatter`` can put inside ``[[...]]``.++ ``_slugify_title``/``_slugify_collection`` turn "2024-01-15", "1.2.3"+ and "2023/2024" into all-digit hyphenated slugs, and the id modes+ append "-N"; destinations keep Unicode spaces (only ASCII ones are+ escaped), may be long, and may nest balanced parentheses.+ """++ @pytest.mark.parametrize(+ "text,expected",+ [+ ("Claim [[2024-01-15]](/library/document/abc).", "Claim."),+ ("Claim [[1-2-3]](/library/document/abd).", "Claim."),+ ("Claim [[2024-01-15-1]](/library/document/abc).", "Claim."),+ (+ "Text [[2023-2024-26]](https://shop.example.com/a) more",+ "Text more",+ ),+ (+ "Claim [[arxiv-1]](https://arxiv.org/abs/1)[local-12]"+ "[[2023-2024-26]](https://shop.example.com/a)[local-27].",+ "Claim.",+ ),+ ("Claim [[20]](https://example.com/a\xa0b).", "Claim."),+ ("Claim [[21]](https://example.jp/a\u3000b).", "Claim."),+ (+ "Claim [[22]](https://example.com/?q=" + "a" * 4100 + ").",+ "Claim.",+ ),+ (+ "Claim [[23]](https://example.com/((((((((((x))))))))))).",+ "Claim.",+ ),+ ],+ )+ def test_formatter_label_strips(self, text, expected):+ assert _manager()._strip_citation_markers(text) == expected++ def test_bare_numeric_range_still_survives(self):+ """Only the hyperlinked form widened: a bare ``[2020-2024]`` has no+ provenance and stays prose."""+ text = "The window is [2020-2024] and [1-2-3]."+ assert _manager()._strip_citation_markers(text) == text+++class TestLenticularMarkersIgnoreWordGlue:+ """Nobody indexes with ``x【1】``, so the word-glue guard is skipped."""++ @pytest.mark.parametrize(+ "text,expected",+ [+ ("Founded in 2020【1】.", "Founded in 2020."),+ ("Paris【1, 2】 is", "Paris is"),+ ("capital【1】【2】.", "capital."),+ ],+ )+ def test_glued_lenticular_strips(self, text, expected):+ assert _manager()._strip_citation_markers(text) == expected+++class TestIndexingGlueAndTradeOffs:+ def test_latex_subscript_survives(self):+ text = "$x_{[i-1]}$ and $y_{[1]}$"+ assert _manager()._strip_citation_markers(text) == text++ def test_spaced_short_slug_is_stripped_documented_trade_off(self):+ """A one-letter collection slug is a real tag (``X`` -> ``[x-1]``),+ so an UNGLUED ``[i-1]`` cannot be told apart from one and strips;+ only the glued ``dp[i-1]`` form is protected."""+ out = _manager()._strip_citation_markers(+ "compare dp [i-1] with dp[i-1]"+ )+ assert out == "compare dp with dp[i-1]"+++class TestSeparatorClosingPunctuation:+ """A space replaces a glued run only between a word (or punctuation+ that ends one) and a following word."""++ @pytest.mark.parametrize(+ "text,expected",+ [+ # ``*`` / quotes count only when they close a span.+ ("**[[1]](https://a.com)Smith et al.**", "**Smith et al.**"),+ ("**bold**[[1]](https://a.com)Next", "**bold** Next"),+ (+ 'He said "yes"[[1]](https://a.com)and left',+ 'He said "yes" and left',+ ),+ ('"[[1]](https://a.com)Quote"', '"Quote"'),+ # ``.`` between digits is a decimal point.+ ("3.[[1]](https://a.com)5", "3.5"),+ ("Claim.[[1]](https://a.com)Next", "Claim. Next"),+ # A non-word, non-closing character before: stays joined.+ ("/page/[[1]](https://a.com)x", "/page/x"),+ # ``_`` after the run: an emphasis closer, not a word.+ ("_word[[1]](https://a.com)_ end", "_word_ end"),+ # A script written without spaces after the run: no space.+ ("Claim.[[1]](https://a.com)東京", "Claim.東京"),+ ("word[[1]](https://a.com)東京", "word東京"),+ ],+ )+ def test_separator(self, text, expected):+ assert _manager()._strip_citation_markers(text) == expected+++class TestSeparatorBeforeLinkOrTag:+ """A following ``[`` (link label) or ``<`` (inline tag) renders as the+ next word, so it gets a space after a word or a closing tag."""++ @pytest.mark.parametrize(+ "text,expected",+ [+ ("word[[1]](u)[dataset](v)", "word [dataset](v)"),+ ("<b>x</b>[[1]](u)[y](z)", "<b>x</b> [y](z)"),+ (+ "<span>word</span>[[1]](u)<span>next</span>",+ "<span>word</span> <span>next</span>",+ ),+ ("word[[1]](u)<b>next</b>", "word <b>next</b>"),+ # Paths, an opening tag and unspaced scripts stay joined.+ ("/a/[[1]](u)[b](c)", "/a/[b](c)"),+ ("<span>[[1]](u)<b>x</b>", "<span><b>x</b>"),+ ("東京[[1]](u)[x](y)", "東京[x](y)"),+ ],+ )+ def test_separator_before_link_or_tag(self, text, expected):+ assert _manager()._strip_citation_markers(text) == expected+++class TestParenthesesHoldingOnlyACitation:+ @pytest.mark.parametrize(+ "text,expected",+ [+ ("Claim ([[1]](https://a.com)).", "Claim."),+ ("Claim ([1][2]) next", "Claim next"),+ ("Claim ([1]) next", "Claim next"),+ # Glued to a word, the parentheses may be a call: kept.+ ("f([[1]](https://a.com)) x", "f() x"),+ ],+ )+ def test_empty_parentheses_are_removed(self, text, expected):+ assert _manager()._strip_citation_markers(text) == expecteddiff --git a/tests/chat/test_citation_stripping_markdown_boundaries.py b/tests/chat/test_citation_stripping_markdown_boundaries.pynew file mode 100644index 0000000000..abd714dedf--- /dev/null+++ b/tests/chat/test_citation_stripping_markdown_boundaries.py@@ -0,0 +1,168 @@+"""Preserve literal Markdown while removing citations outside its boundaries."""++import pytest++from local_deep_research.chat.citation_stripping import (+ strip_prior_turn_citations,+)+from local_deep_research.text_optimization.citation_formatter import (+ CitationFormatter,+ CitationMode,+)+++ "text,expected",+ [+ (r"Literal \[1] and fact [2].", r"Literal \[1] and fact."),+ (+ r"Escaped \`tick [1] and `code [2]` fact [3].",+ r"Escaped \`tick and `code [2]` fact.",+ ),+ ("[1][manual]\n\n[manual]: /guide", "[1][manual]\n\n[manual]: /guide"),+ ("[1][2]\n\n[2]: /guide", "[1][2]\n\n[2]: /guide"),+ ("Fact [1][2].", "Fact."),+ (+ "- ```\n [1]\n ```\n\nFact [2].",+ "- ```\n [1]\n ```\n\nFact.",+ ),+ (+ "- item\n\n ```python\n values = [1]\n ```\n\nFact [2].",+ "- item\n\n ```python\n values = [1]\n ```\n\nFact.",+ ),+ (+ "10. ```\n code [1]\n ```\n\nFact [2].",+ "10. ```\n code [1]\n ```\n\nFact.",+ ),+ (+ "> ```\n> code [1]\n\nFact [2].",+ "> ```\n> code [1]\n\nFact.",+ ),+ ],+)+def test_literals_and_container_boundaries(text, expected):+ """Catches stripping before escape/reference protection or across code boundaries."""+ assert strip_prior_turn_citations(text) == expected+++ "url",+ [+ "http://example.com:8080/a",+ "http://10.0.0.1/a",+ "http://10.0.0.1:8080/a",+ "http://[::1]/a",+ "http://[::1]:8080/a",+ ],+)+ "mode",+ [+ CitationMode.DOMAIN_ID_HYPERLINKS,+ CitationMode.DOMAIN_ID_ALWAYS_HYPERLINKS,+ ],+)+def test_formatter_authority_ids_are_removed(url, mode):+ """Two same-authority sources make the conditional formatter emit ID suffixes."""+ sources = [+ {"index": 1, "title": "First", "url": url},+ {"index": 2, "title": "Second", "url": url + "b"},+ ]+ formatted = CitationFormatter(mode).apply_inline_hyperlinks(+ "See [1, 2] now.", sources+ )+ assert formatted != "See [1, 2] now."+ assert strip_prior_turn_citations(formatted) == "See now."++[email protected]("ticks", [2, 3, 4])+def test_escaped_tick_leaves_shorter_code_opener(ticks):+ """An escape consumes one backtick, not the entire run before inline code."""+ literal = "\\" + "`" * ticks + "code [1]" + "`" * (ticks - 1)+ assert (+ strip_prior_turn_citations(literal + " fact [2].") == literal + " fact."+ )+++ "example",+ ["```\n[2]: /url\n```", "<!--\n[2]: /url\n-->", "<pre>\n[2]: /url\n</pre>"],+)+def test_code_reference_definitions_do_not_change_prose_citations(example):+ """Reference-looking code is not a real definition for an adjacent marker run."""+ assert (+ strip_prior_turn_citations(example + "\nFact [1][2].")+ == example + "\nFact."+ )+++ "text,expected",+ [+ # The heading / list item ends the paragraph, so the first+ # backtick never closes and ``[1]`` is prose. It ends its line, so+ # the space before it goes with it (no trailing whitespace is left+ # behind; see the module docstring's separator rule).+ (+ "`literal [1]\n# heading `code [2]` fact [3].",+ "`literal\n# heading `code [2]` fact.",+ ),+ (+ "`literal [1]\n- item `code [2]` fact [3].",+ "`literal\n- item `code [2]` fact.",+ ),+ (+ "- > ```\n > values = [1]\n > ```\n\nFact [2].",+ "- > ```\n > values = [1]\n > ```\n\nFact.",+ ),+ (+ "> - > ```\n> > values = [1]\n> > ```\n\nFact [2].",+ "> - > ```\n> > values = [1]\n> > ```\n\nFact.",+ ),+ ("<!--\n```\n-->\n\nClaim [1].", "<!--\n```\n-->\n\nClaim."),+ ("```\n<!--\n```\n\nClaim [1].", "```\n<!--\n```\n\nClaim."),+ ("`<!--`\nClaim [1].", "`<!--`\nClaim."),+ (+ '<span title="<pre>">literal</span>\n\nFact [[1]](u).\n\nMore [2].',+ '<span title="<pre>">literal</span>\n\nFact.\n\nMore.',+ ),+ (+ '<code title="a > b">literal [1]</code> Fact [2].',+ '<code title="a > b">literal [1]</code> Fact.',+ ),+ ("Read [1] now.\n\n[1]: /manual", "Read [1] now.\n\n[1]: /manual"),+ (+ 'Read [1](/manual "the guide") now.',+ 'Read [1](/manual "the guide") now.',+ ),+ ("Read [1](<manual page>) now.", "Read [1](<manual page>) now."),+ (+ "Fact [local-1][local-2][local-3] and code `x`.",+ "Fact and code `x`.",+ ),+ ],+)+def test_composed_markdown_boundaries(text, expected):+ """Catches cross-block delimiters, nested containers, and literal HTML tags."""+ assert strip_prior_turn_citations(text) == expected++[email protected]("suffix", ["`code`", "<span>value</span>"])+def test_successful_runs_have_bounded_matcher_work(monkeypatch, suffix):+ """Revisiting each recognized suffix makes the number of lookups quadratic."""+ from local_deep_research.chat import citation_stripping as scanner++ original = scanner._match_citation_token+ calls = 0++ def counted(text, position):+ nonlocal calls+ calls += 1+ return original(text, position)++ monkeypatch.setattr(scanner, "_match_citation_token", counted)+ markers = 600+ result = scanner.strip_prior_turn_citations("[1] " * markers + suffix)+ assert "[1]" not in result+ assert suffix in result+ assert calls <= 10 * markersdiff --git a/tests/chat/test_citation_stripping_round3.py b/tests/chat/test_citation_stripping_round3.pynew file mode 100644index 0000000000..d75829c865--- /dev/null+++ b/tests/chat/test_citation_stripping_round3.py@@ -0,0 +1,86 @@+from markdown import markdown+import pytest++from local_deep_research.chat.citation_stripping import (+ strip_prior_turn_citations,+)+from local_deep_research.text_optimization.citation_formatter import (+ CitationFormatter,+ CitationMode,+)+++ "url",+ [+ "http://example.com:8080/a",+ "http://EXAMPLE.COM:8080/a",+ "http://[::1]:8080/a",+ ],+)+def test_formatter_port_label_is_stripped_when_source_has_authority(+ url: str,+) -> None:+ document = f"See [1] now.\n\n## Sources\n\n[1] Example\nURL: {url}"+ formatted = CitationFormatter(+ mode=CitationMode.DOMAIN_HYPERLINKS+ ).format_document(document)+ text = formatted.split("\n\n## Sources", maxsplit=1)[0]++ result = strip_prior_turn_citations(text)++ assert result == "See now."+++ "text",+ [+ "",+ "[1]: https://en.wikipedia.org",+ "Go to https://example.com/page/[1]/details here",+ ],+)+def test_numeric_markdown_and_url_path_survive_when_not_citations(+ text: str,+) -> None:+ result = strip_prior_turn_citations(text)++ assert result == text+++ ("text", "expected", "rendered"),+ [+ (+ "<span>word</span> [[1]](u)<span>next</span>",+ "<span>word</span> <span>next</span>",+ "<p><span>word</span> <span>next</span></p>",+ ),+ (+ "word [[1]](u)[dataset](v)",+ "word [dataset](v)",+ '<p>word <a href="v">dataset</a></p>',+ ),+ ("text [1]more", "text more", "<p>text more</p>"),+ ],+)+def test_removed_citation_keeps_rendered_text_separated(+ text: str, expected: str, rendered: str+) -> None:+ result = strip_prior_turn_citations(text)++ assert result == expected+ assert markdown(result) == rendered++[email protected]("marker", ["`", "~"])+def test_fence_longer_than_previous_limit_is_fully_protected(+ marker: str,+) -> None:+ fence = marker * 33+ shorter = marker * 32+ text = f"{fence}\n[1] [[2]](u)\n{shorter}\n[3]\n{fence}"++ result = strip_prior_turn_citations(text)++ assert result == textdiff --git a/tests/chat/test_citation_stripping_round4.py b/tests/chat/test_citation_stripping_round4.pynew file mode 100644index 0000000000..f6ec590e5f--- /dev/null+++ b/tests/chat/test_citation_stripping_round4.py@@ -0,0 +1,96 @@+from markdown import markdown+import pytest++from local_deep_research.chat.citation_stripping import (+ strip_prior_turn_citations,+)+from local_deep_research.text_optimization.citation_formatter import (+ CitationFormatter,+ CitationMode,+)+++ "url",+ [+ "http://10.0.0.1/a",+ "http://10.0.0.1:8080/a",+ ],+)+def test_formatter_ipv4_label_is_stripped(url: str) -> None:+ document = f"See [1] now.\n\n## Sources\n\n[1] Example\nURL: {url}"+ formatted = CitationFormatter(+ mode=CitationMode.DOMAIN_HYPERLINKS+ ).format_document(document)+ text = formatted.split("\n\n## Sources", maxsplit=1)[0]++ result = strip_prior_turn_citations(text)++ assert result == "See now."+ assert markdown(result) == "<p>See now.</p>"+++ "text",+ [+ "[dataset](https://example.com/page/[1]/details)",+ "<https://example.com/page/[1]/details>",+ "(https://example.com/page/[1]/details)",+ '<a href="https://example.com/page/[1]/details">dataset</a>',+ ],+)+def test_numeric_marker_in_punctuation_prefixed_url_survives(text: str) -> None:+ result = strip_prior_turn_citations(text)++ assert result == text+ assert markdown(result) == markdown(text)+++ "text",+ [+ "![Figure 1 [1]](pic.jpg)",+ "![some [1] text](pic.jpg)",+ ],+)+def test_numeric_marker_nested_in_image_alt_text_survives(text: str) -> None:+ result = strip_prior_turn_citations(text)++ assert result == text+ assert markdown(result) == markdown(text)+++ ("text", "expected"),+ [+ ("word[[1]](u)[dataset](v)", "word [dataset](v)"),+ ("word[[1]](u)more", "word more"),+ (+ "<span>word</span>[[1]](u)<span>next</span>",+ "<span>word</span> <span>next</span>",+ ),+ ],+)+def test_adjacent_citation_removal_separates_rendered_words(+ text: str, expected: str+) -> None:+ result = strip_prior_turn_citations(text)++ assert result == expected+ assert markdown(result) == markdown(expected)+++ "text",+ [+ " [1]\n [2]",+ "\t[1]",+ "<code>[1]</code>",+ "<code>first [1]\nsecond [2]</code>",+ ],+)+def test_commonmark_code_content_preserves_numeric_brackets(text: str) -> None:+ result = strip_prior_turn_citations(text)++ assert result == text+ assert markdown(result) == markdown(text)