~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
fix(library): resolve chunk-anchor citations in fetch and bibliography (#5381 follow-up) (#5685)
LearningCircuit11d244e90097
diff --git a/changelog.d/5685.bugfix.md b/changelog.d/5685.bugfix.mdnew file mode 100644index 0000000000..1bcf8d14e9--- /dev/null+++ b/changelog.d/5685.bugfix.md@@ -0,0 +1,29 @@+Fixed three downstream effects of the chunk-anchor citation URLs introduced in #5381.++`fetch_content` can read library documents again: the RAG engines now cite `/library/document/<id>/chunks#chunk-<n>`, but the fetch tool's library resolver only matched an optional `/pdf` suffix and never stripped the fragment, so every such citation fell through to the egress gate and was rejected as `unsupported_scheme`. The resolver now accepts the `chunks` route and drops the fragment before matching.++`SearchResultsCollector.sources` no longer loses deduplicated URLs after `reset()`. `reset()` clears `_sources` per report subsection while the dedup maps deliberately persist, so a subsection that re-cited an earlier section's hits reported no sources at all — an empty `sources` list from the MCP server, and `sources_count: 0` in news cards.++Library citations collapse to one bibliography entry per document again. `canonical_url_key` stripped fragments only from absolute URLs, so each cited chunk of a document became a separate `## Sources` line. Library routes now key per document — `/library/document/<id>`, its `/pdf` view and its `/chunks#chunk-<n>` view are one source — which also collapses the two entries a document used to get when some of its chunks carried `chunk_index` metadata and others fell back to `/pdf`. The rendered `URL:` line keeps a `#chunk-<n>` anchor, preferring an anchored view over the anchor-less `/pdf` one regardless of which was registered first, so collapsing the list does not cost the scroll-to-chunk behaviour the anchor exists for. Note the entry carries one URL for the whole group: it anchors at the first cited chunk, so a document cited across several chunks scrolls to the first of them rather than to each citation's own chunk. Per-citation anchors would need a different Sources layout and are not attempted here. Query strings are not part of a library document's identity — `/library/document/<id>?v=2` keys with `/library/document/<id>` — which no current producer emits.++Citations of one library document via the absolute `https://library.document/<id>` alias — the form the agent sometimes emits and the fetch resolver accepts — now share a bibliography entry with that document's other citations instead of rendering as a separate, anchor-less external-looking link. Only the exact `<id>[/pdf|/chunks]` shape normalizes: look-alike hosts, deeper paths, queries and traversal are rejected, so a crafted alias cannot merge into a real document's entry and replace its link.++Library/RAG citations no longer point at the wrong document when a chunk's denormalised metadata disagrees with its `DocumentChunk` foreign key. The authoritative document id is now carried at the top level of each search result and preferred over metadata wherever the citation URL is built or rebuilt.++`format_links_to_markdown` no longer crashes the whole bibliography when a citation index contains a non-ASCII digit character (`"¹".isdigit()` is true while `int("¹")` raises).++Bibliography exports (BibTeX/Quarto and LaTeX) keep library documents that are cited from several chunks. Collapsing those citations onto one Sources line produced a `[1, 2, 3]` index group that the exporters' single-index patterns skipped, silently dropping the source from the exported bibliography.++`sources_count` now agrees with the rendered bibliography. The bibliography groups sources with `canonical_url_key`, but `SearchResultsCollector` still keyed its dedup map and its `sources` list on the raw link, so the three views of one library document (`/library/document/<id>`, `/<id>/pdf`, `/<id>/chunks#chunk-<n>`, plus the absolute `https://library.document/<id>` alias) produced three citation indices and a `sources_count` of 3 for a report whose `## Sources` block renders one line — the count the MCP server returns and the news impact score reads. The collector now dedups on the same canonical key while recording the display URL, anchor included. This also means a search hit and a later `fetch_content` of the same document share one citation index no matter which spelling of the URL the agent typed back.++A subsection that only re-fetches previously cited URLs renders its `## Sources` block again. `find_or_add_result` recorded the source on all three branches but appended to `_results` only when it allocated a new citation, so after `reset()` such a subsection reported N sources, rendered none, and tripped the #4969 "citation pass skipped" warning.++Library-route parsing is now one code path for the relative route, its `/lib/document/<id>` abbreviation and the absolute alias, so all three enforce the same validation. Previously only the alias validated its document id: `/library/document/7/../../../admin/wipe#chunk-1` keyed as document 7, inherited that document's title and citation numbers, and replaced its link. The `/lib/` spelling and percent-encoded ids (`a%2Db` vs `a-b`) now key with the document they denote instead of fanning it across extra entries, and the displayed route is always the registered `/library/document/<id>` one. The bare `[<uuid>]` alias the agent sometimes pastes into `fetch_content` is re-cited as the resolved document's canonical route instead of verbatim, closing the third spelling that used to get its own (dead-linking) entry. A route carrying a control character is now refused outright rather than having its document-id segment truncated: truncating discarded the payload but keyed the crafted URL as the real `/library/document/<id>` it was prefixed with, merging it into that document's bibliography entry and lending it the document's title and citation number. The payload is still neutralised where it is rendered.++An external URL that merely contains `/library/document/` — `https://evil.example/library/document/7/chunks` — is no longer relabelled as a local library document. The eligibility test is anchored at the start of the URL instead of being a substring match.++Chunk indices and document ids are validated against ASCII character classes with a magnitude bound, so non-ASCII digits (`٧`, `𝟝`), whitespace-padded values, absurd magnitudes (`10**30`, `1e30`) and fullwidth document ids can no longer reach a URL fragment or path. The bibliography's "prefer an anchored URL" rule now looks for `#chunk-` specifically, so an empty or unrelated fragment can no longer lock out the real chunk anchor.++`sources` in the strategy result keeps first-seen order. It was passed through `list(set(...))`, which deduped nothing (the collector already guarantees uniqueness) and randomized the ordering differently on every process.++Note: RIS export does not yet handle merged index lines, and mis-attributes rather than merely skipping them — a record can carry one source's title with another source's URL. It needs a different fix from the other exporters and is tracked in #5687.diff --git a/src/local_deep_research/advanced_search_system/strategies/langgraph_agent_strategy.py b/src/local_deep_research/advanced_search_system/strategies/langgraph_agent_strategy.pyindex 7a7d778ffc..60be444b7b 100644--- a/src/local_deep_research/advanced_search_system/strategies/langgraph_agent_strategy.py+++ b/src/local_deep_research/advanced_search_system/strategies/langgraph_agent_strategy.py@@ -30,8 +30,16 @@ from ...utilities.chunk_anchor import (extract_chunk_index,extract_document_id,is_library_chunk_result,+ is_library_document_link,)+from ...utilities.search_utilities import _sanitize_sources_fieldfrom ...utilities.thread_context import get_search_context, search_context+from ...utilities.url_utils import (+ CHUNK_DISPLAY_KEY,+ canonical_url_key,+ library_display_url,+ preferred_chunk_display,+)from ...database.thread_local_session import thread_cleanupfrom ..tools.fetch import FETCH_MODES, build_fetch_tool, make_library_resolverfrom .base_strategy import (@@ -101,6 +109,117 @@ def _scrub_tool_error(message: str) -> str:# ---------------------------------------------------------------------------+def _citation_dedup_key(link: Any) -> str:+ """Canonical key a citation link is deduplicated on.++ This MUST be the same key ``format_links_to_markdown`` groups the+ rendered bibliography by, or the two disagree about how many sources+ a report has. Since #5381 ``canonical_url_key`` collapses+ ``/library/document/<id>``, its ``/pdf`` view and its+ ``/chunks#chunk-<n>`` views onto one key — three views of one+ document render as ONE ``## Sources`` line, so keying the collector+ on the raw link handed back three entries, three citation indices and+ a ``sources_count`` of 3 for a report that displays 1 (MCP's+ ``sources`` payload and the news impact score both read that count).++ Returns ``""`` for anything that cannot be a dict/set key — a+ non-string ``link`` (an engine handing back a list) would otherwise+ raise ``TypeError`` from the membership test. Such a result still gets+ a citation index; it just cannot participate in dedup.+ """+ if not isinstance(link, str) or not link:+ return ""+ try:+ return canonical_url_key(link) or ""+ except Exception: # pragma: no cover - defensive+ logger.debug("citation dedup: falling back to the raw link")+ return link+++def _is_library_citation(value: Any) -> bool:+ """Return ``True`` if *value* addresses a library document.++ Route-shaped OR parseable. The two answer different questions —+ ``is_library_document_link`` asks whether the string ADDRESSES a+ library document, ``library_display_url`` whether it PARSES as a+ citation — and the second says no to things the first says yes to+ (a control character anywhere in the string, an unsafe doc id).+ Taking either as sufficient keeps those out of the "not our business"+ bucket, which is where an unvalidated anchor would survive.++ This sentence previously claimed the pair covered every spelling+ because they failed at different edges. That was false while+ ``is_library_document_link`` was a literal ``startswith`` test: the+ alias with a port or userinfo AND a control character missed BOTH+ arms — the prefix test on the port, the parser on the control+ character — so the fragment rode through all three ingest paths. The+ predicate now matches on host, so the first arm covers those+ spellings; the claim is true as a consequence of that fix, not on its+ own.++ ONE named predicate because both ingest paths must ask it identically.+ Inlining it in the fetch sibling and leaving ``add_results`` on the+ bare ``startswith`` test is exactly how the two drifted: the alias+ spellings were validated on one path and stored verbatim on the other.+ """+ return isinstance(value, str) and (+ is_library_document_link(value)+ or library_display_url(value) is not None+ )+++def _strip_unvalidated_chunk_fragments(result: dict) -> dict:+ """Return *result* with any unusable ``#chunk-`` fragment removed.++ Applied to a LIBRARY citation only. ``preferred_chunk_display`` returns+ ``None`` for every non-library URL, so testing it alone would strip the+ fragment off an ordinary external page that happens to anchor at+ ``#chunk-2`` — a citation truncated by a rule with no business reading+ it.++ ANY fragment, not only ``#chunk-`` ones. That argument for leaving+ other fragments alone is about non-library URLs, which+ ``_is_library_citation`` already excludes. Once a value is known to+ address a library document, a fragment that is not a valid chunk+ anchor names nothing: ``library_display_url`` drops it at render time+ regardless, so dropping it here changes no report — it only stops the+ raw string reaching the sinks the renderer does not guard,+ ``_sources`` (the MCP payload and the news cards) and+ ``select_source_url`` (``research_resources.url``). It also stops a+ crafted spelling keying separately from the document's real citation:+ ``_parse_library_citation`` rejects control characters, so such a+ string falls back to ``url.strip()`` as its dedup key and fans one+ document out into two bibliography entries.++ "Library citation" is :func:`_is_library_citation`, shared with+ ``add_results`` so the two ingest paths cannot disagree about what they+ are allowed to touch.++ Returns *result* itself when nothing changed, so the common path copies+ nothing.+ """+ cleaned: dict | None = None+ for field in ("link", "url"):+ value = result.get(field)+ if not isinstance(value, str) or "#" not in value:+ continue+ if not _is_library_citation(value):+ continue+ if preferred_chunk_display(value) is not None:+ continue+ logger.debug(+ "stripping unvalidated chunk fragment from {}={}",+ field,+ redact_url_for_log(value),+ )+ if cleaned is None:+ cleaned = dict(result)+ # Per field, on its own value: deriving both from one of them+ # overwrote a ``url`` that differed from ``link``.+ cleaned[field] = value.split("#", 1)[0]+ return cleaned if cleaned is not None else result++class SearchResultsCollector:"""Accumulates search results from the lead agent and subagents.@@ -120,19 +239,33 @@ class SearchResultsCollector:the map, so direct external appends remain *visible* but lose O(1)dedup and may reuse indices incorrectly until the next``add_results`` call.++ A seeded entry carrying NO ``index`` is never added to+ ``_index_to_result``, so ``len(_all_links) > len(_index_to_result)``+ stays true for the life of the collector and the O(1) scan-skip in+ ``find_or_add_result`` is disabled from then on. Perf only — the+ linear fallback returns the same answer — but it is why the skip is+ described as exact for the append-only contract rather than+ unconditionally."""def __init__(self, all_links: list | None = None) -> None:self._results: list[dict] = []self._sources: list[str] = []+ # Canonical keys already present in ``_sources`` for the current+ # subsection. Cleared by ``reset()`` alongside ``_sources`` (unlike+ # the dedup maps, which deliberately persist), so a URL first seen+ # in an earlier subsection is recorded again for this one.+ #+ # This holds ``_citation_dedup_key(link)``, while ``_sources``+ # holds the DISPLAY url (anchor and all), so the two are related by+ # ``{_citation_dedup_key(u) for u in _sources} == _sources_seen``+ # rather than by plain equality. ``len(_sources) ==+ # len(_sources_seen)`` still holds, and ``_sources`` still has no+ # duplicates.+ self._sources_seen: set[str] = set()self._lock = threading.Lock()self._all_links = all_links if all_links is not None else []- # O(1) URL → citation-index and index → result lookups. Maintained- # under ``_lock`` alongside ``_all_links`` so the LangGraph path- # (the only path that mutates ``_all_links``) keeps it in sync.- # Seeded once at construction from any pre-existing entries the- # caller passed in; entries without an ``index`` are skipped to- # avoid storing the literal string ``"None"``.self._url_to_index: dict[str, str] = {}self._index_to_result: dict[str, dict] = {}# Highest seeded citation index — used by ``add_results`` and@@ -145,12 +278,34 @@ class SearchResultsCollector:# bare string / None.if not isinstance(r, dict):continue- link = r.get("link") or r.get("url") or ""+ # Seeded entries did not come through add_results, so strip+ # the producer key here too — otherwise a seeded value both+ # wins at the readers AND blocks the collector's own anchor,+ # because _prefer_anchored_link uses setdefault.+ r.pop(CHUNK_DISPLAY_KEY, None)+ # ...and the fragment, for the same reason and by the same+ # helper the other two paths use. Normalising the producer key+ # here while leaving an unusable ``#chunk-``/other fragment was+ # the identical hazard half-handled: a seeded entry rendered+ # its raw fragment into the Sources block, and — because the+ # key below is computed from the link — a fragment the library+ # parser rejects (a control character is enough) keyed off the+ # raw string, so the document fanned out into a second+ # bibliography entry the moment it was also fetched.+ #+ # In-tree every caller seeds an EMPTY list today, so this is+ # not a live leak; but ``all_links_of_system`` is a documented+ # constructor parameter ("List of existing links"), and this+ # loop had already decided seeded entries need normalising.+ cleaned = _strip_unvalidated_chunk_fragments(r)+ if cleaned is not r:+ r.update(cleaned)+ key = _citation_dedup_key(r.get("link") or r.get("url") or "")idx = r.get("index")if idx is not None:idx_str = str(idx)- if link and link not in self._url_to_index:- self._url_to_index[link] = idx_str+ if key and key not in self._url_to_index:+ self._url_to_index[key] = idx_strif idx_str not in self._index_to_result:self._index_to_result[idx_str] = rtry:@@ -208,6 +363,16 @@ class SearchResultsCollector:if not isinstance(raw, dict):continuer = dict(raw) # shallow copy to avoid mutating engine output+ # Drop any producer-supplied anchor. This key is written+ # by the collector and preferred by the renderer and by+ # the code that persists research_resources.url, so an+ # engine setting it would choose both. Stripping it at+ # ingest is the same defence applied to an unvalidated+ # ``#chunk-`` fragment below, applied at the boundary+ # rather than re-litigated at every reader — five rounds+ # of reader-side validation each closed one route and+ # missed the next.+ r.pop(CHUNK_DISPLAY_KEY, None)r["source_engine"] = engine_name# Normalise URL key — citation handler expects "link"if "link" not in r and "url" in r:@@ -225,7 +390,19 @@ class SearchResultsCollector:# URL already carries a fragment).if is_library_chunk_result(r):chunk_idx = extract_chunk_index(metadata)- doc_id = extract_document_id(metadata, r)+ # Authoritative id first, mirroring the RAG engines.+ # ``extract_document_id`` scans its first mapping+ # first, so passing ``metadata`` first would let a+ # chunk's denormalised ``document_id`` override the+ # DocumentChunk FK and rebuild the link pointing at a+ # DIFFERENT document — undoing the engines' own+ # ordering one hop later.+ authoritative = (+ {"source_id": r.get("source_id")}+ if isinstance(r, dict) and r.get("source_id")+ else None+ )+ doc_id = extract_document_id(authoritative, metadata, r)if chunk_idx is not None and doc_id:rebuilt = build_chunk_anchor_url(link, doc_id, chunk_idx@@ -234,24 +411,36 @@ class SearchResultsCollector:link = rebuiltr["link"] = linkr["url"] = link- elif isinstance(link, str) and "#chunk-" in link:- # Producer shipped a chunk fragment we cannot- # validate — strip it so a malformed anchor never- # reaches the bibliography. Leave non-chunk- # fragments alone.- stripped = link.split("#", 1)[0]- logger.debug(- "add_results: stripping unvalidated chunk "- "fragment from link={}",- redact_url_for_log(link),- )- link = stripped- r["link"] = link- r["url"] = link- if link and link in self._url_to_index:+ else:+ # Delegate to the SAME helper the fetch sibling+ # uses, rather than restating its rule a second+ # time here. Every previous divergence between the+ # two ingest paths came from this branch keeping a+ # hand-written copy: first the scope gate, then the+ # alias predicate, then whether a valid anchor+ # survives, then which fields get written. The+ # helper decides all four, so there is nothing left+ # to hand-mirror.+ #+ # (The rebuild branch above still wins when the+ # producer supplied authoritative chunk metadata —+ # an anchor derived from the DocumentChunk FK beats+ # one the producer spelled into the link.)+ cleaned = _strip_unvalidated_chunk_fragments(r)+ if cleaned is not r:+ r.update(cleaned)+ link = r.get("link", link)++ # Dedup on the canonical key, not the raw link: the+ # bibliography collapses every view of one library document+ # onto one line, so keying on the raw spelling would report+ # three sources for a report that renders one.+ key = _citation_dedup_key(link)+ if key and key in self._url_to_index:# Duplicate URL — reuse the existing citation index# so repeated search hits collapse to a single [N].- r["index"] = self._url_to_index[link]+ r["index"] = self._url_to_index[key]+ self._prefer_anchored_link(r["index"], link)logger.debug("add_results: dedup reuse url={} -> [{}]",redact_url_for_log(link),@@ -265,23 +454,91 @@ class SearchResultsCollector:self._max_idx += 1new_idx = str(self._max_idx)r["index"] = new_idx- if link:- self._url_to_index[link] = new_idx- self._index_to_result[new_idx] = r- self._sources.append(link)- else:- # No link — still track by index for find_by_index- self._index_to_result[new_idx] = r+ if key:+ self._url_to_index[key] = new_idx+ # Track by index either way so find_by_index resolves+ # linkless entries too.+ self._index_to_result[new_idx] = rself._all_links.append(r)logger.debug("add_results: new citation url={} -> [{}]",- redact_url_for_log(link) if link else "<no link>",+ # ``key`` is empty for a missing or non-string+ # link; ``redact_url_for_log`` parses its argument+ # as a URL and raises TypeError on anything else.+ redact_url_for_log(link) if key else "<no link>",new_idx,)+ # ``_sources`` records every distinct link cited in the+ # current subsection, whether or not its citation index was+ # deduplicated. Keying off ``_sources_seen`` rather than the+ # new-index branch matters because ``reset()`` clears+ # ``_sources`` per subsection while the dedup maps persist:+ # gating on the new-index branch would silently drop every+ # URL first seen in an earlier subsection, leaving+ # ``sources``/``sources_count`` empty for sections that+ # re-cite earlier hits.+ #+ # ``_sources`` stores the DISPLAY url — the one carrying+ # ``#chunk-<n>`` — while the seen-set holds the canonical+ # key, so a document's chunk anchor survives into the MCP+ # ``sources`` payload without the document being counted+ # once per anchor.+ if key and key not in self._sources_seen:+ # Same flattening as the fetch path — see the note+ # at the other ``_sources.append``.+ self._sources.append(_sanitize_sources_field(link))+ self._sources_seen.add(key)self._results.append(r)indexed.append(r)return start_idx, indexed+ def _prefer_anchored_link(self, index: str, link: str) -> None:+ """Record a chunk-anchored spelling ALONGSIDE the stored citation.++ Deduping on the canonical key leaves one entry per document in+ ``_all_links`` — the first spelling seen — so a document whose+ anchor-less view arrived first would lose its ``#chunk-<n>``. That+ list is what the detailed report appends and what+ ``research_resources.url`` is persisted from, so the loss reaches+ the database rather than one render.++ This writes a SEPARATE key rather than overwriting ``link``/``url``+ deliberately. The overwrite needed a new precondition after every+ review round — is-a-string, is-a-library-route, isn't-already-+ anchored, owns-this-key — each added because the write had been+ found destroying the wrong thing: one source's citation taking+ another document's URL, credentials reaching the database. None of+ those preconditions was about the feature; they existed to make a+ destructive write safe. Writing a field nothing else keys on+ removes the whole class: the worst case becomes an ignored hint on+ an unrelated entry, not a corrupted citation.++ Callers must hold ``_lock``.+ """+ display = preferred_chunk_display(link)+ if display is None:+ return+ stored = self._index_to_result.get(index)+ if stored is None:+ return+ current = stored.get("link") or stored.get("url") or ""+ # The entry's OWN anchored spelling wins. Recording a different+ # chunk of the same document over it points the reader at text the+ # entry's snippet did not come from.+ if isinstance(current, str) and preferred_chunk_display(current):+ return+ # Confirm the entry owns this citation. ``_url_to_index`` and+ # ``_index_to_result`` are filled by independent+ # "if not already present" guards, so they can name different+ # documents — and since both consumers prefer this key over the+ # entry's url, writing blind puts one document's anchor on+ # another's citation, in the report and in the database. Being+ # additive does not make that harmless: a field readers prefer IS+ # the identity as far as they can tell.+ if _citation_dedup_key(current) != _citation_dedup_key(display):+ return+ stored.setdefault(CHUNK_DISPLAY_KEY, display)+def find_or_add_result(self,result: dict,@@ -292,42 +549,173 @@ class SearchResultsCollector:Fetch tools can run concurrently in pooled subagents. Keeping the URLlookup and append under one lock prevents two fetches of the same URLfrom allocating separate citation indices.++ Dedup uses the same canonical key as ``add_results`` and as the+ rendered bibliography, so fetching a document the agent already+ saw in a search hit reuses that citation index no matter which of+ the document's URL spellings the agent typed back."""+ # Strip BEFORE the key is computed and before ``_sources`` records+ # the URL. Doing it at the end of the new-entry branch (as the first+ # version did) left two sinks reading the raw value: the dedup key,+ # which ``canonical_url_key`` leaves fragment-BEARING whenever the+ # library parse fails (a control character in the fragment is+ # enough), so ``_url_to_index`` named an entry whose stored link no+ # longer canonicalised to it — one document, two citation indices,+ # the exact defect the dedup key exists to prevent; and+ # ``_sources``, which fed the raw fragment to the MCP payload and+ # the news cards. ``add_results`` already strips first; this is the+ # same order.+ result = _strip_unvalidated_chunk_fragments(result)url = result.get("link", result.get("url", ""))+ # Computed outside the lock: pure, and ``canonical_url_key`` is+ # LRU-cached.+ key = _citation_dedup_key(url)with self._lock:- if url and url in self._url_to_index:+ # Record the source before branching: all three paths below end+ # with *url* tracked (reused via either dedup path, or freshly+ # registered), and ``_sources`` is per-subsection state that+ # ``reset()`` clears while ``_url_to_index`` deliberately+ # persists. Recording only on the new-entry branch would drop+ # every URL that a previous subsection already cited — a+ # fetch-only subsection would report no sources at all.+ # Whether *url* is new to THIS subsection (``reset()`` clears+ # ``_sources_seen`` but deliberately keeps the dedup maps).+ # Gates the ``_results`` bookkeeping below so the two stay in+ # step: a re-fetch after ``reset()`` repopulates both, while a+ # repeat fetch within one subsection adds to neither.+ first_here = bool(key) and key not in self._sources_seen+ # Flattened with the SAME helper the renderer uses. ``_sources``+ # is the one sink this branch's sanitisation never covered: the+ # bibliography flattens line-breaking characters at render, but+ # this list goes to the MCP ``sources`` payload and the news+ # cards untouched, so a U+2028 in an ORDINARY external URL+ # reached those consumers able to forge a line. Library routes+ # were already refused upstream; this covers everything else.+ # Reusing the renderer's helper rather than restating the rule:+ # every shared rule on this branch that was restated drifted.+ if first_here:+ self._sources.append(_sanitize_sources_field(url))+ self._sources_seen.add(key)++ reused: int | None = None+ # ``reused_key`` keeps the index in its STORED string form.+ # ``str(int(...))`` is not a round trip for a zero-padded seeded+ # index -- ``str(int("007"))`` is ``"7"``, which renumbers the+ # echoed result and misses the ``"007"`` entry in+ # ``_index_to_result``, while the RIS exporter deliberately+ # preserves the padded form. The int is only for the return+ # value and ordering.+ reused_key: str | None = None+ if key and key in self._url_to_index:try:- return int(self._url_to_index[url])+ reused_key = str(self._url_to_index[key])+ reused = int(reused_key)except (ValueError, TypeError):- pass-- for tracked in self._all_links:- if tracked.get("link", tracked.get("url", "")) == url and url:+ reused = None+ reused_key = None++ # Only entries appended OUTSIDE the collector can be missing+ # from ``_url_to_index``. Skipping the scan otherwise keeps a+ # canonicalisation of every element off the hot path: it runs+ # under ``_lock`` on every new fetched URL, and+ # ``canonical_url_key``'s LRU is 1024 entries, so a sequential+ # walk of a longer list evicts exactly what the next walk needs+ # and the hit rate collapses to zero (~170x measured).+ # Every append the collector makes also registers in+ # ``_index_to_result`` — including linkless ones — so a longer+ # ``_all_links`` means entries arrived from outside, which is+ # the only case this scan exists for. Exact for the append-only+ # contract the class documents, and unlike a counter there is+ # no derived state to fall out of step. It cannot detect an+ # entry REPLACED in place; that is stated in the class+ # docstring's aliasing note rather than guessed at here.+ if (+ reused is None+ and key+ and len(self._all_links) > len(self._index_to_result)+ ):+ for tracked in self._all_links:+ # Legacy seeds may hold non-dicts; ``.get`` would raise.+ if not isinstance(tracked, dict):+ continue+ tracked_url = tracked.get("link", tracked.get("url", ""))+ if _citation_dedup_key(tracked_url) != key:+ continueidx = tracked.get("index")- if idx is not None:- idx_str = str(idx)- self._url_to_index[url] = idx_str- if idx_str not in self._index_to_result:- self._index_to_result[idx_str] = tracked- try:- return int(idx)- except (ValueError, TypeError):- pass+ if idx is None:+ continue+ idx_str = str(idx)+ try:+ reused = int(idx_str)+ except (ValueError, TypeError):+ continue+ self._url_to_index[key] = idx_str+ # Externally appended, so it never passed the ingest+ # strip. Unconditional: the entry is already in+ # ``_all_links`` and therefore already reader-visible,+ # so gating this on whether the index happens to be+ # unmapped left the producer key in place for exactly+ # the entries that skip the branch below.+ tracked.pop(CHUNK_DISPLAY_KEY, None)+ if idx_str not in self._index_to_result:+ self._index_to_result[idx_str] = tracked+ reused_key = idx_str+ break++ if reused is not None:+ # The fetch path reuses an index just as ``add_results``+ # does, so it must upgrade the stored spelling too or the+ # anchor loss survives on this branch.+ self._prefer_anchored_link(str(reused), url)+ if first_here:+ # Record the citation in ``_results`` even though no+ # new entry was allocated. ``_results`` is what the+ # caller feeds to ``_format_citations``: leaving a+ # reused URL out of it meant a fetch-only subsection+ # that re-cited sources from an earlier subsection+ # reported N sources while rendering no ``## Sources``+ # block at all, and tripped the #4969 "citation pass+ # skipped" warning. Gated on ``first_here`` so a+ # repeated fetch inside one subsection still does not+ # grow ``_results``.+ echoed = dict(result)+ # The reuse branch builds its own copy, so it needs+ # the same producer-key strip as the allocate branch.+ echoed.pop(CHUNK_DISPLAY_KEY, None)+ echoed["index"] = reused_key or str(reused)+ echoed["source_engine"] = engine_name+ if "link" not in echoed and "url" in echoed:+ echoed["link"] = echoed["url"]+ self._results.append(echoed)+ return reused# Collision-free allocator (same as ``add_results``) so the# fetch fast path can never overwrite a sparse seeded index.self._max_idx += 1index = self._max_idxtracked = dict(result)+ tracked.pop(+ CHUNK_DISPLAY_KEY, None+ ) # producer-supplied: see add_results+ # Same unvalidated-fragment strip ``add_results`` applies. This+ # branch is the fetch path: ``library_resolver`` returns the+ # agent's raw ``fetch_content(url)`` argument as the citation+ # URL (it strips the fragment only to match the route, never to+ # validate it), so an anchor here has been checked by nobody.+ # The REUSE branch above was already covered — it routes through+ # ``_prefer_anchored_link`` → ``preferred_chunk_display`` — which+ # is why the existing regression test, seeding via+ # ``add_results`` first, only ever exercised that sibling.tracked["index"] = str(index)tracked["source_engine"] = engine_nameif "link" not in tracked and "url" in tracked:tracked["link"] = tracked["url"]self._results.append(tracked)- link = tracked.get("link", "")- if link:- self._url_to_index[link] = str(index)- self._sources.append(link)+ if key:+ self._url_to_index[key] = str(index)+ # ``_sources`` was already recorded above, before the+ # dedup branches.self._index_to_result[str(index)] = trackedself._all_links.append(tracked)return index@@ -335,12 +723,19 @@ class SearchResultsCollector:def find_by_url(self, url: str) -> int | None:"""Return the 1-based citation index if *url* is already tracked, else ``None``.+ Matching is on the canonical dedup key, so any spelling of a+ library document's URL finds the citation registered under any+ other spelling of it.+O(1) via ``_url_to_index`` with a linear-scan fallback for entriesthat were appended to ``_all_links`` outside the collector (seealiasing note in ``__init__``) or that carry a non-int-like index."""+ key = _citation_dedup_key(url)+ if not key:+ return Nonewith self._lock:- idx_str = self._url_to_index.get(url)+ idx_str = self._url_to_index.get(key)if idx_str is not None:try:return int(idx_str)@@ -352,7 +747,10 @@ class SearchResultsCollector:)# Fallback scan for externally-mutated or legacy entriesfor r in self._all_links:- if r.get("link", r.get("url", "")) == url:+ # Legacy seeds may hold non-dicts; ``.get`` would raise.+ if not isinstance(r, dict):+ continue+ if _citation_dedup_key(r.get("link", r.get("url", ""))) == key:idx = r.get("index")if idx is None:continue@@ -379,6 +777,11 @@ class SearchResultsCollector:O(1) via ``_index_to_result`` with a linear-scan fallback forexternally-mutated entries.++ Returns a shallow COPY, matching ``results`` and ``sources``: the+ stored dict is the very object ``_url_to_index`` was keyed from, so+ a caller that rewrote its ``link`` in place would silently desync+ the dedup map from ``_all_links``."""if not isinstance(idx, int) or idx < 1:return None@@ -386,14 +789,17 @@ class SearchResultsCollector:with self._lock:result = self._index_to_result.get(idx_str)if result is not None:- return result+ return dict(result)# Fallback scan for externally-mutated or legacy entriesfor r in self._all_links:+ # Legacy seeds may hold non-dicts; ``.get`` would raise.+ if not isinstance(r, dict):+ continuestored = r.get("index")if stored is not None:try:if int(stored) == idx:- return r+ return dict(r)except (ValueError, TypeError):continuereturn None@@ -403,6 +809,7 @@ class SearchResultsCollector:with self._lock:self._results.clear()self._sources.clear()+ self._sources_seen.clear()@propertydef results(self) -> list[dict]:@@ -2361,7 +2768,12 @@ class LangGraphAgentStrategy(BaseSearchStrategy):"questions": {},"formatted_findings": formatted_output,"current_knowledge": synthesized_content,- "sources": list(set(self.collector.sources)),+ # ``_sources`` is already duplicate-free (the collector gates+ # every append on ``_sources_seen``), so ``set()`` deduped+ # nothing here — its only effect was to randomize the order the+ # MCP client and the news impact scorer see, differently on+ # every process because of hash randomization.+ "sources": self.collector.sources,"search_results": all_search_results,"documents": documents,"reasoning_trace": reasoning_trace,diff --git a/src/local_deep_research/advanced_search_system/tools/fetch/__init__.py b/src/local_deep_research/advanced_search_system/tools/fetch/__init__.pyindex 2be5b78ddb..c3271bd64c 100644--- a/src/local_deep_research/advanced_search_system/tools/fetch/__init__.py+++ b/src/local_deep_research/advanced_search_system/tools/fetch/__init__.py@@ -83,6 +83,23 @@ def _register_in_collector(If the URL was already tracked (via a prior search hit) the existingindex is reused so the agent sees a stable citation per URL.++ .. note::+ The ``find_or_add_result``-less branch below is **speculative+ generality**: ``SearchResultsCollector`` is the only collector in+ this repo and it has that method, so nothing in-tree reaches the+ fallback and no test exercises it against a real collector. It+ exists for third-party or pre-#5363 collectors. Treat it as+ unverified against any live caller — do not assume a bug there is+ affecting production, and do not grow it further without a real+ consumer to justify it.++ This branch was left at its pre-existing shape deliberately. An+ earlier revision of this follow-up grew it — duck-typed tuple+ unpacking for a bare-``int`` ``add_results`` contract, a ``getattr``+ guard on ``find_by_url`` — for a collector that does not exist in+ any repo we can see, which is precisely what the paragraph above+ forbids. That growth was reverted rather than shipped."""snippet = snippet_source[:200].strip()if len(snippet_source) > 200:diff --git a/src/local_deep_research/advanced_search_system/tools/fetch/library_resolver.py b/src/local_deep_research/advanced_search_system/tools/fetch/library_resolver.pyindex d170295d31..953f4ef9c5 100644--- a/src/local_deep_research/advanced_search_system/tools/fetch/library_resolver.py+++ b/src/local_deep_research/advanced_search_system/tools/fetch/library_resolver.py@@ -6,8 +6,9 @@ strings that the egress policy correctly rejects asnot know how to fetch even without a policy gate):1. **Local library document references** like- ``/library/document/<uuid>``, ``/lib/document/<uuid>``,- ``https://library.document/<uuid>``, or ``[<uuid>]``.+ ``/library/document/<uuid>``, ``/library/document/<uuid>/chunks#chunk-3``,+ ``/lib/document/<uuid>``, ``https://library.document/<uuid>``, or+ ``[<uuid>]``.The library RAG search engine and the collection search engine emitthese as the citation URL of a search hit (see``web_search_engines.engines.search_engine_library.LibraryRAGSearchEngine.search``@@ -38,12 +39,23 @@ from urllib.parse import unquote, urlsplitfrom loguru import logger+from ....utilities.chunk_anchor import (+ LIBRARY_ROUTE_SUFFIX_ALTERNATION,+ is_safe_document_id,+)+# Match canonical paths and the ``/lib/document/...`` abbreviation observed# in agent tool calls. The same suffix is used by download_service.+#+# ``chunks`` is accepted alongside ``pdf`` because the library/collection+# RAG engines cite chunk-targeted URLs (``/library/document/<id>/chunks``+# plus a ``#chunk-<n>`` fragment) since #5381. Both suffixes resolve to+# the same ``Document.text_content`` read below — the suffix only tells a+# caller which UI route the citation pointed at._LIBRARY_PATH_RE = re.compile(r"^/(?:library|lib)/document/(?P<doc_id>[^/?#]+)"- r"(?:/(?P<suffix>pdf))?/?$"+ rf"(?:/(?P<suffix>{LIBRARY_ROUTE_SUFFIX_ALTERNATION}))?/?$")_DOCUMENT_ID_PATTERN = (r"(?:"@@ -67,6 +79,13 @@ _CITATION_REF_RE = re.compile(r"^\[(\d+)\]$")class _LibraryReference:value: strsuffix: Optional[str] = None+ #: True when the reference arrived as the bare ``[<uuid>]`` alias+ #: rather than as a route. That spelling is not a URL at all, so+ #: registering it verbatim as the citation link gives the document a+ #: THIRD bibliography entry alongside its ``/library/document/<id>``+ #: and ``https://library.document/<id>`` citations — and renders a+ #: dead link. Callers re-emit the canonical route instead.+ bracketed: bool = Falsedef _decode_segment(value: str) -> str | None:@@ -88,13 +107,26 @@ def _parse_library_reference(url: str) -> _LibraryReference | None:if not candidate:return None+ # Drop any ``#fragment`` first. Chunk citations carry ``#chunk-<n>``,+ # which addresses an anchor inside the rendered page and is never part+ # of the document identity. This runs BEFORE the ``://`` test so that+ # (a) the ``https://library.document/<id>/chunks#chunk-3`` alias is not+ # rejected by the absolute branch's no-fragment rule, and (b) a+ # relative path whose fragment happens to contain ``://``+ # (``/library/document/<id>#http://x``) is not misrouted into it.+ candidate = candidate.split("#", 1)[0]+ if not candidate:+ return None+if "://" in candidate:parsed = urlsplit(candidate)+ # No ``parsed.fragment`` check: the fragment was stripped above, so+ # it can never reach here. A query still disqualifies — it can carry+ # server-meaningful state, unlike a client-side anchor.if (parsed.scheme != "https"or parsed.netloc != "library.document"or parsed.query- or parsed.fragment):return Nonecandidate = f"/library/document{parsed.path}"@@ -108,7 +140,7 @@ def _parse_library_reference(url: str) -> _LibraryReference | None:bracket_match = _BRACKETED_DOCUMENT_RE.match(candidate)if bracket_match:- return _LibraryReference(bracket_match.group("doc_id"))+ return _LibraryReference(bracket_match.group("doc_id"), bracketed=True)return None@@ -116,8 +148,11 @@ def _parse_library_reference(url: str) -> _LibraryReference | None:def parse_library_url(url: str) -> tuple[str, Optional[str]] | None:"""Return the local document reference and optional suffix, else ``None``.- ``suffix`` is ``"pdf"`` for the PDF-direct form and ``None`` for the- root document page. In addition to canonical library paths, this accepts+ ``suffix`` is ``"pdf"`` for the PDF-direct form, ``"chunks"`` for the+ chunk-listing route that RAG citations point at, and ``None`` for the+ root document page. Any ``#fragment`` (e.g. ``#chunk-3``) is stripped+ before matching, so a chunk-targeted citation resolves to its+ document. In addition to canonical library paths, this acceptsthe ID-bound aliases emitted by the agent. Filename-only references remainon the normal egress-denial path because filenames are guessable."""@@ -146,7 +181,10 @@ def resolve_library_document(``content``) so callers can pass it straight to ``_register_in_collector``+ the ``[N] Title:\\nURL:\\n\\n<body>`` formatter. The ``url`` field ofthe dict is the *original* library URL, so a downstream re-citation still- reads "library/document/…" instead of something more confusing.+ reads "library/document/…" instead of something more confusing — except+ for the bare ``[<uuid>]`` alias, which is not a URL at all and is+ re-emitted as the resolved document's canonical route so it shares a+ bibliography entry with the document's other citations.Returns ``None`` for:@@ -199,10 +237,30 @@ def resolve_library_document(if document is None:return Nonetext = document.text_content or ""+ citation_url = url+ if reference.bracketed:+ # Built from the RESOLVED row's id, not from the alias+ # value: the lookup above also accepts a document_hash and+ # a dash-less UUID, neither of which is a live route.+ resolved_id = str(document.id or "")+ if is_safe_document_id(resolved_id):+ citation_url = f"/library/document/{resolved_id}"+ else:+ # Practically unreachable — ``Document.id`` is a UUID —+ # but silence here leaves ``citation_url`` as the raw+ # ``[uuid]`` alias, which is not a route: a dead link+ # that also keys separately from the document's other+ # citations. Log it rather than let it look intended.+ logger.warning(+ "library_resolver: resolved document id is not "+ "route-safe, citation left as the bracketed alias "+ "(id={!r})",+ resolved_id,+ )return {"title": document.title or f"Document {lookup_value}","content": text,- "url": url,+ "url": citation_url,# ``snippet`` is what the collector registers as the# citation preview. Use the title fallback when text is# empty so the agent still gets *something* descriptive.diff --git a/src/local_deep_research/security/log_sanitizer.py b/src/local_deep_research/security/log_sanitizer.pyindex b753c38e18..0857ae35cf 100644--- a/src/local_deep_research/security/log_sanitizer.py+++ b/src/local_deep_research/security/log_sanitizer.py@@ -28,6 +28,9 @@ _UNSAFE_CHAR_RE = re.compile(r"\u061c" # Arabic letter markr"\u200b-\u200f" # Zero-width chars + LTR/RTL marksr"\u202a-\u202e" # Embedding/override (incl. RLO)+ r"\u2028\u2029" # Line/paragraph separators — forced breaks in rendered+ # HTML per CSS Text, so a log line carrying one can forge what looks+ # like a separate entry even though re and str.splitlines() ignore themr"\u2060-\u2064" # Word joiner + math invisible operatorsr"\u2066-\u2069" # Isolate charsr"\u206a-\u206f" # Digit shape controlsdiff --git a/src/local_deep_research/text_optimization/citation_formatter.py b/src/local_deep_research/text_optimization/citation_formatter.pyindex 49e91f1b0f..e3f9a71042 100644--- a/src/local_deep_research/text_optimization/citation_formatter.py+++ b/src/local_deep_research/text_optimization/citation_formatter.py@@ -5,6 +5,7 @@ from enum import Enumfrom typing import Any, Dict, List, Optional, Tuplefrom urllib.parse import urlparse+from loguru import loggerfrom slugify import slugifyfrom ..content_fetcher.url_classifier import URLClassifier, URLType@@ -80,6 +81,174 @@ def find_sources_section(return -1, False+_BIB_SOURCES_PATTERN = re.compile(+ r"^\[(\d+(?:,\s*\d+)*)\]\s*(.+?)(?:\n\s*URL:\s*(.+?))?$",+ re.MULTILINE,+)+++# Single-pass character map. A sequential replace() chain would re-escape+# the braces its own replacements introduce — the bug already present in+# _escape_latex, where "\\" becomes "\textbackslash{}" and the later+# brace rules then mangle it.+_BIBTEX_ESCAPES = {+ "\\": r"\textbackslash{}",+ # Balanced pairs, not backslash escapes: BibTeX's field scanner counts+ # brace CHARACTERS and has no escape mechanism, so "\{" is still an open+ # brace to it. A backslashed brace left the value unbalanced, and the+ # scanner ran to EOF swallowing the rest of the .bib.+ "{": r"{\textbraceleft}",+ "}": r"{\textbraceright}",+ "&": r"\&",+ "%": r"\%",+ "$": r"\$",+ "#": r"\#",+ "_": r"\_",+ "~": r"\textasciitilde{}",+ "^": r"\textasciicircum{}",+ '"': "'",+}+++# U+0085 NEL already falls inside the 0x7F-0x9F range below; only these two+# separators sit outside it.+_UNICODE_LINE_SEPARATORS = "\u2028\u2029"+++def is_line_breaking_char(char: str) -> bool:+ """True if *char* can begin a new line for some consumer of our output.++ ONE definition, imported by every producer. The recurring bug in this+ area has been fixing one instance and leaving a sibling — CR handled+ but not NEL, the URL sanitised but not the citation index — so the+ union is deliberately wider than any single consumer needs: C0 and DEL+ for everything, C1 because such a codepoint is never legitimate text,+ and U+2028/U+2029 because CSS Text makes them mandatory breaks in+ rendered HTML even though Python's ``re`` and ``str.splitlines()``+ ignore them.++ Imported by ``search_utilities._sanitize_sources_field``;+ ``test_bibliography_escaping`` asserts the two agree on every BMP+ codepoint, so the claim cannot rot into a comment again.+ """+ code = ord(char)+ return (+ code < 0x20 or 0x7F <= code <= 0x9F or char in _UNICODE_LINE_SEPARATORS+ )+++def _escape_bibtex(value: str) -> str:+ r"""Escape a value for interpolation into a BibTeX field.++ Titles come from search-result metadata — whatever a page calls itself+ — and were previously written into ``title = "{...}"`` raw. A title+ ending the field and opening another (``Benign}", url = "...", note =+ "{x``) injected an attacker-controlled ``url`` that BibTeX preferred,+ and an unbalanced brace ran the parser on into the rest of the file.+ """+ return "".join(+ # Control characters are flattened, not escaped. The .bib is+ # embedded in a ```bibtex markdown fence by the Quarto exporter,+ # and a bare CR — which ``_BIB_SOURCES_PATTERN``'s ``.`` matches,+ # since only ``\n`` is excluded — closes that fence once a reader+ # normalises CRLF, turning the remainder into live markup.+ # ``_safe_bibtex_url`` already rejects these; this matches it.+ " " if is_line_breaking_char(ch) else _BIBTEX_ESCAPES.get(ch, ch)+ for ch in value+ )+++def _safe_bibtex_url(value: str) -> str:+ r"""Return *value* if it is safe to place in ``url = {...}``, else "".++ Escaping is wrong for a URL — a ``\%`` would corrupt a percent-encoded+ octet. Nor may the offending characters simply be DROPPED: removing a+ backslash from ``//trusted.example\@evil.example/p`` turns the trusted+ host into userinfo, so the exported link points at a different host+ than the one the reader vetted in the report. A URL containing a+ breakout character is not a URL to repair — it is omitted, and the+ caller falls through to its existing URL-less branch.+ """+ if any(ch in '{}"\\ ' or is_line_breaking_char(ch) for ch in value):+ # Local-document sources legitimately trip this: a LangChain+ # retriever sets a result's url from its ``source`` metadata, which+ # is often a filesystem path carrying spaces or backslashes.+ # Dropping the link silently would leave the user guessing why a+ # bibliography entry has none, so record it.+ # Truncated: a hostile result can carry a megabyte-long URL, and+ # this fires once per affected entry — and the Quarto exporter+ # builds the bibliography twice per export, so it fires twice per+ # entry in practice. ``!r`` keeps control characters escaped.+ logger.debug(+ "Omitting unsafe bibliography URL: {!r}{}",+ value[:200],+ "…" if len(value) > 200 else "",+ )+ return ""+ return value+++def _iter_citation_group(group: str):+ r"""Yield each index on one Sources line (``"1, 2, 3"`` -> "1", "2", "3").++ ``format_links_to_markdown`` collapses every citation of one source+ onto a single line carrying all of its indices, so a single-index+ pattern drops such a source from the bibliography entirely. Exporters+ emit one entry per index so each ``[N]`` in the body resolves.++ Indices are yielded as STRINGS, exactly as written: the body citations+ are rewritten from the same text, so converting ``007`` to ``7`` here+ would key the entry ``ref7`` while the body still cites ``ref007``.+ No digit filtering either — the pattern's ``\d`` matches only decimal+ digits (category ``Nd``), which ``int()`` always accepts, so filtering+ could only discard valid indices such as Arabic-Indic ones.+ """+ for part in group.split(","):+ part = part.strip()+ if part:+ yield part+++def _sort_key(index: str):+ """Order bibliography entries numerically where possible."""+ try:+ return (0, int(index), "")+ except ValueError:+ return (1, 0, index)+++def _collect_bibliography_entries(content: str) -> dict:+ """Map each citation index to the ``(title, url)`` it names.++ Selection rule: a URL-bearing entry never loses to a URL-less one, and+ among URL-bearing entries the LATER occurrence wins.++ Both halves matter. A report repeats its Sources block once per section+ and then again as a combined block, and the combined block is both last+ and authoritative — so later-wins. And the group-aware pattern can match+ a line of prose that opens with ``[1, 2]`` where a single-index pattern+ could not; such a line carries no ``URL:``, whereas every line+ ``format_links_to_markdown`` renders does, so the URL preference is what+ keeps prose from taking a citation key.++ Deliberately NOT scoped to a Sources section. Bounding the block at+ markers like ``\n---`` looked safer but silently emptied the+ bibliography whenever a thematic break, a setext underline or a table+ delimiter row came first — while the body was still rewritten to cite+ entries that no longer existed.+ """+ entries: dict[str, tuple[str, str]] = {}+ for match in _BIB_SOURCES_PATTERN.finditer(content):+ title = match.group(2).strip()+ url = match.group(3).strip() if match.group(3) else ""+ for index in _iter_citation_group(match.group(1)):+ # Prefer an entry carrying a URL: a real Sources line has one,+ # a prose false positive does not.+ if url or index not in entries:+ entries[index] = (title, url)+ return entries++class CitationMode(Enum):"""Available citation formatting modes."""@@ -910,24 +1079,18 @@ csl: apa.csldef _generate_bibliography(self, content: str) -> str:"""Generate BibTeX bibliography from sources."""- sources_pattern = re.compile(- r"^\[(\d+)\]\s*(.+?)(?:\n\s*URL:\s*(.+?))?$", re.MULTILINE- )+ entries = _collect_bibliography_entries(content)bibliography = ""- matches = list(sources_pattern.finditer(content))-- for match in matches:- citation_num = match.group(1)- title = match.group(2).strip()- url = match.group(3).strip() if match.group(3) else ""-- # Generate BibTeX entry+ for citation_num in sorted(entries, key=_sort_key):+ title, url = entries[citation_num]+ safe_title = _escape_bibtex(title)+ safe_url = _safe_bibtex_url(url)bib_entry = f"@misc{{ref{citation_num},\n"- bib_entry += f' title = "{{{title}}}",\n'- if url:- bib_entry += f" url = {{{url}}},\n"- bib_entry += f' howpublished = "\\url{{{url}}}",\n'+ bib_entry += f' title = "{{{safe_title}}}",\n'+ if safe_url:+ bib_entry += f" url = {{{safe_url}}},\n"+ bib_entry += f' howpublished = "\\url{{{safe_url}}}",\n'bib_entry += f" year = {{{2024}}},\n"bib_entry += ' note = "Accessed: \\today"\n'bib_entry += "}\n"@@ -992,18 +1155,47 @@ class RISExporter:# Split sources into individual entriesimport re- # Pattern to match each source entry. Accept both ASCII "[N]" and- # lenticular "【N】" openers/closers — the inline citation patterns- # in this file already handle lenticular brackets (some LLMs emit- # them), so the source-list parser must stay consistent or it would- # silently drop lenticular-bracketed source entries.+ # Pattern to match each source entry. The opener is GROUP-AWARE+ # (``\d+(?:,\s*\d+)*``, the spelling used by+ # ``comma_citation_pattern`` and ``_BIB_SOURCES_PATTERN`` above):+ # ``format_links_to_markdown`` collapses every citation of one source+ # onto a single merged ``[1, 3] Title`` line, which is routine for+ # library documents since per-document keying (#5685). A single-index+ # ``(\d+)`` capture did not merely drop such a line — it MIS-CITED+ # (#5687), because the lookahead did not terminate at a merged opener+ # either, so the PRECEDING entry swallowed the merged line and the+ # last-wins ``URL:`` scan below then handed that entry the merged+ # line's URL: one source's title against another source's link. Hence+ # the group spelling appears TWICE — once in the capture, once in the+ # lookahead — and the loop emits one record PER INDEX so that every+ # ``[N]`` in the body resolves to a reference of its own.+ #+ # Four properties here are deliberate; keep them when editing:+ # 1. This parser stays bounded to the FIRST Sources section by the+ # ``next_section_markers`` scan above. Do not swap in the+ # module-level ``_collect_bibliography_entries`` — that helper+ # is deliberately not section-scoped, which is why #5687 notes+ # RIS is not a mechanical swap even though BibTeX, Quarto and+ # LaTeX were fixed by adopting it.+ # 2. Lenticular "【N】" openers/closers are accepted alongside+ # ASCII "[N]": the inline citation patterns in this file already+ # handle them (some LLMs emit them), so the source-list parser+ # must stay consistent or it would silently drop those entries.+ # 3. Entries are split by LOOKAHEAD under ``DOTALL`` so continuation+ # lines (``URL:``, ``DOI:``, ``Published in``) stay attached to+ # the entry they belong to; a line-anchored ``$`` split would+ # strip them.+ # 4. ``self.sources_pattern`` above is group-aware but UNUSED, and+ # is not this fix — it is single-line and section-unaware, so+ # wiring it in would lose properties 1 and 3.source_entry_pattern = re.compile(- r"^[\[【](\d+)[\]】]\s*(.+?)(?=^[\[【]\d+[\]】]|\Z)",+ r"^[\[【](\d+(?:,\s*\d+)*)[\]】]\s*(.+?)"+ r"(?=^[\[【]\d+(?:,\s*\d+)*[\]】]|\Z)",re.MULTILINE | re.DOTALL,)for match in source_entry_pattern.finditer(sources_content):- citation_num = match.group(1)+ citation_group = match.group(1)entry_text = match.group(2).strip()# Extract the title (first line)@@ -1033,15 +1225,21 @@ class RISExporter:# Combine title with additional metadata lines for full contextfull_text = entry_text- # Create a unique key to avoid duplicates- ref_key = (citation_num, title, url)- if ref_key not in seen_refs:- seen_refs.add(ref_key)- # Create RIS entry with full text for metadata extraction- ris_entry = self._create_ris_entry(- citation_num, full_text, url, metadata- )- ris_entries.append(ris_entry)+ # One record per index, keyed on the INDIVIDUAL index rather than+ # the group text: a report repeats its Sources block, and the same+ # source may be written ``[1]`` in one block and ``[1, 3]`` in+ # another, so keying on the raw group would let ``ref1`` be emitted+ # twice. ``metadata`` is shared across the indices of one line;+ # ``_create_ris_entry`` only reads it.+ for citation_num in _iter_citation_group(citation_group):+ ref_key = (citation_num, title, url)+ if ref_key not in seen_refs:+ seen_refs.add(ref_key)+ # Create RIS entry with full text for metadata extraction+ ris_entry = self._create_ris_entry(+ citation_num, full_text, url, metadata+ )+ ris_entries.append(ris_entry)return "\n".join(ris_entries)@@ -1352,26 +1550,26 @@ class LaTeXExporter:return ""sources_content = content[sources_start:]- pattern = re.compile(- r"^\[(\d+)\]\s*(.+?)(?:\n\s*URL:\s*(.+?))?$", re.MULTILINE- )+ entries = _collect_bibliography_entries(sources_content)bibliography = "\n\\begin{thebibliography}{99}\n"- for match in pattern.finditer(sources_content):- citation_num = match.group(1)- title = match.group(2).strip()- url = match.group(3).strip() if match.group(3) else ""-- # Escape special LaTeX characters in title+ for citation_num in sorted(entries, key=_sort_key):+ title, url = entries[citation_num]escaped_title = self._escape_latex(title)- if url:- bibliography += f"\\bibitem{{{citation_num}}} {escaped_title}. \\url{{{url}}}\n"+ # NOTE: thebibliography numbers entries by POSITION, so a gap+ # in the index sequence makes \\cite{5} print [3]. That is+ # pre-existing and unchanged here — an explicit+ # ``\\bibitem[label]{key}`` would fix it but changes an output+ # format existing tests pin, so it belongs in its own PR.+ # Sorting at least keeps the common contiguous case correct.+ item = f"\\bibitem{{{citation_num}}}"+ safe_url = _safe_bibtex_url(url)+ if safe_url:+ bibliography += f"{item} {escaped_title}. \\url{{{safe_url}}}\n"else:- bibliography += (- f"\\bibitem{{{citation_num}}} {escaped_title}.\n"- )+ bibliography += f"{item} {escaped_title}.\n"bibliography += "\\end{thebibliography}\n"diff --git a/src/local_deep_research/utilities/chunk_anchor.py b/src/local_deep_research/utilities/chunk_anchor.pyindex dd138a4840..6497570b97 100644--- a/src/local_deep_research/utilities/chunk_anchor.py+++ b/src/local_deep_research/utilities/chunk_anchor.py@@ -12,9 +12,48 @@ valid chunk index looks like.from __future__ import annotations+import refrom typing import Any, Mapping, Optional+# The document views a library route may address. Shared by the two+# route-matching regexes — ``library_resolver._LIBRARY_PATH_RE`` and+# ``url_utils._LIBRARY_ROUTE_PATH_RE`` — so adding a view teaches both at+# once instead of relying on one to be hand-mirrored from the other.+#+# It does NOT unify every route predicate in the codebase:+# ``url_utils._is_library_route`` is a prefix test that answers a+# different question (is this string shaped like a library route at all)+# and is deliberately looser. Do not read this constant as making all+# three agree.+LIBRARY_ROUTE_SUFFIXES = ("pdf", "chunks")+# Escaped: these are interpolated into regexes, so a future view containing+# a metacharacter (``v1.2``, ``raw+text``) would otherwise silently widen+# what counts as a library route.+LIBRARY_ROUTE_SUFFIX_ALTERNATION = "|".join(+ re.escape(suffix) for suffix in LIBRARY_ROUTE_SUFFIXES+)++# Root-relative routes the library / collection RAG engines emit as citation+# URLs, plus the absolute alias the agent sometimes types instead. Shared+# with ``utilities.url_utils`` (which imports from here — this module stays+# dependency-free so the import can only go one way).+LIBRARY_ROUTE_PREFIXES = ("/library/document/", "/lib/document/")+LIBRARY_ALIAS_HOST = "library.document"+_LIBRARY_ALIAS_SCHEME_PREFIX = "https://"++# Chunk indices are small ordinals within one document. The bound is far+# above any realistic chunk count and exists so an absurd value (a+# timestamp, a float that rounded to 10**30, a 64-bit id mistaken for an+# index) cannot be interpolated into a URL fragment.+MAX_CHUNK_INDEX = 1_000_000++# Document ids are UUIDs / hex hashes / slugs. ASCII only: ``str.isalnum()``+# accepts every Unicode letter and digit, so a fullwidth "\uff17" would pass+# and then key separately from its percent-encoded form.+_SAFE_ID_RE = re.compile(r"^[0-9A-Za-z_-]+$")++def extract_chunk_index(metadata: Mapping[str, Any] | None) -> Optional[int]:"""Return a validated non-negative chunk index from a result's metadata,or ``None`` if the metadata does not carry a usable value.@@ -30,7 +69,7 @@ def extract_chunk_index(metadata: Mapping[str, Any] | None) -> Optional[int]:Accepts:* ``int`` (excluding ``bool``) — used as-is.- * ``str`` whose trimmed value parses as a non-negative integer.+ * ``str`` composed solely of ASCII digits.* ``float`` whose value is a whole non-negative integer (catchese.g. ``0.0`` from numpy conversion).@@ -38,7 +77,15 @@ def extract_chunk_index(metadata: Mapping[str, Any] | None) -> Optional[int]:* ``bool`` (avoids ``True`` being printed as ``#chunk-True``).* Negative integers — zero is allowed because chunks are 0-indexed.- * Strings that are not pure digits (``"550e8400-..."`` UUIDs, etc).+ * Values above :data:`MAX_CHUNK_INDEX`. No document has a million+ chunks, so a value that large is a timestamp, a 64-bit id, or the+ silent binary rounding of a float literal such as ``1e30`` — none of+ which addresses a real anchor.+ * Strings that are not pure ASCII digits: ``"550e8400-..."`` UUIDs,+ signed forms (``"+5"``/``"-1"``), surrounding whitespace (``" 5 "``),+ and non-ASCII digit characters that ``str.isdigit()`` accepts but+ that no anchor id can contain (Arabic-Indic ``"\u0667"``,+ mathematical ``"\U0001d7dd"``, ...).* Floats with a fractional part.* Anything else (``None``, dicts, lists, ...)."""@@ -60,12 +107,15 @@ def extract_chunk_index(metadata: Mapping[str, Any] | None) -> Optional[int]:if isinstance(chunk_idx_raw, int):value = chunk_idx_rawelif isinstance(chunk_idx_raw, str):- stripped = chunk_idx_raw.strip()- # Reject leading/trailing whitespace, signs, plus signs,- # anything that isn't a pure non-negative integer string.- if not stripped or not stripped.isdigit():+ # Reject surrounding whitespace, signs, and every non-ASCII+ # digit: the value is interpolated verbatim into an anchor, and+ # ``str.isdigit()`` alone is Unicode-wide (it accepts "\u0667"+ # and "\U0001d7dd", which ``int()`` then happily converts to 7+ # and 5). Deliberately no ``.strip()`` — the docstring has+ # always promised whitespace is rejected.+ if not chunk_idx_raw.isascii() or not chunk_idx_raw.isdigit():return None- value = int(stripped)+ value = int(chunk_idx_raw)elif isinstance(chunk_idx_raw, float):if not chunk_idx_raw.is_integer():return None@@ -79,7 +129,7 @@ def extract_chunk_index(metadata: Mapping[str, Any] | None) -> Optional[int]:# allowed: the document_chunks.html template renders ``chunk.index``# directly as the HTML anchor id, so a 0-indexed chunk anchors at# ``#chunk-0``.- if value < 0:+ if value < 0 or value > MAX_CHUNK_INDEX:return Nonereturn value@@ -126,19 +176,36 @@ def extract_document_id(stripped = raw.strip()if not stripped:continue- if not all(c.isalnum() or c in "-_" for c in stripped):+ if not is_safe_document_id(stripped):continuereturn strippedreturn None+def is_safe_document_id(value: str) -> bool:+ """Return ``True`` if *value* is safe to interpolate into a URL path.++ **ASCII** alphanumerics, dashes and underscores only — which excludes+ ``/``, ``?``, ``#``, ``%``, whitespace, control characters and ``..``+ traversal. ASCII matters: ``str.isalnum()`` is true for every Unicode+ letter and digit, so a fullwidth ``"\uff17"`` would pass, be emitted+ unencoded into a URL path, and then key separately from its+ ``%EF%BC%97`` form — one document, two bibliography entries.++ Shared by :func:`extract_document_id`, :func:`build_chunk_anchor_url`+ and ``url_utils``'s library-route parser so none of them can drift+ apart.+ """+ return isinstance(value, str) and _SAFE_ID_RE.match(value) is not None++def is_library_chunk_result(result: Mapping[str, Any] | None) -> bool:"""Return ``True`` if *result* is a library/RAG hit that should receivea chunk anchor.Heuristic: the result's ``source`` / ``source_type`` field is- ``"library"`` or its link points at ``/library/document/``. Mirrors- the producer-side checks in+ ``"library"`` or its link *is* a library-document route (see+ :func:`is_library_document_link`). Mirrors the producer-side checks in:class:`langgraph_agent_strategy.SearchResultsCollector` so the twoagree on the eligibility rule."""@@ -148,8 +215,61 @@ def is_library_chunk_result(result: Mapping[str, Any] | None) -> bool:return Trueif result.get("source_type") == "library":return True- link = result.get("link") or result.get("url") or ""- return isinstance(link, str) and "/library/document/" in link+ return is_library_document_link(result.get("link") or result.get("url"))+++def _alias_host(stripped: str) -> str | None:+ """Return the lowercased host of an ``https://<host>/...`` URL, or ``None``.++ Userinfo and port are removed so ``https://[email protected]:443/x``+ and ``https://library.document/x`` give the same host. A trailing+ ``/`` is required, so a bare authority with no path is not a document+ link — matching what the literal-prefix test accepted before.+ """+ if not stripped.lower().startswith(_LIBRARY_ALIAS_SCHEME_PREFIX):+ return None+ rest = stripped[len(_LIBRARY_ALIAS_SCHEME_PREFIX) :]+ authority, sep, _ = rest.partition("/")+ if not sep:+ return None+ # Userinfo may itself contain ``@``; the host is after the LAST one.+ authority = authority.rpartition("@")[2]+ if authority.startswith("["):+ # IPv6 literal: the port, if any, follows the closing bracket.+ host = authority.partition("]")[0] + "]"+ else:+ host = authority.partition(":")[0]+ return host.lower()+++def is_library_document_link(link: Any) -> bool:+ """Return ``True`` if *link* addresses a local library document.++ Anchored at the start of the URL on purpose. A substring test+ (``"/library/document/" in link``) also matches+ ``https://evil.example/library/document/7/chunks``, and the caller+ reacts by REPLACING the link with a local route — silently relabelling+ an external result as a document in the user's own library.+ """+ if not isinstance(link, str):+ return False+ stripped = link.strip()+ if stripped.startswith(LIBRARY_ROUTE_PREFIXES):+ return True+ # The absolute alias the agent sometimes emits. Matched on the HOST,+ # not on a literal ``https://library.document/`` prefix: the prefix+ # form misses the ``:443`` and userinfo spellings that+ # ``url_utils._normalize_library_alias`` deliberately accepts as the+ # same document, so a caller asking "is this a library citation" got+ # False for a string the renderer would happily normalise. Callers+ # that strip an unusable fragment then skipped those spellings+ # entirely, and the raw value reached the DB and the MCP payload.+ #+ # Parsed by hand rather than with ``urlsplit``, which silently DELETES+ # embedded tab/newline/CR — the very characters that make a crafted+ # alias worth catching here, and which would otherwise let+ # ``library.doc\tument`` answer for ``library.document``.+ return _alias_host(stripped) == LIBRARY_ALIAS_HOSTdef build_chunk_anchor_url(@@ -164,7 +284,9 @@ def build_chunk_anchor_url(* the chunk index failed validation (UUID, bool, negative, string,float with a fractional part, or any other non-int-like value), or- * the document id is missing/rejected.+ * the document id is missing, or is not composed solely of+ alphanumerics, dashes and underscores (see+ :func:`is_safe_document_id`).Callers should leave the original *link* unchanged on ``None`` —appending ``#chunk-...`` to whatever route was already in the link@@ -177,13 +299,31 @@ def build_chunk_anchor_url(# forgot to validate) must not be able to inject a malformed# fragment. ``int`` excludes ``bool`` (which is technically an# ``int`` subclass) and ``None``.- if doc_id is None:+ # Enforce the doc-id contract here rather than trusting the caller.+ # Every in-tree caller runs ``extract_document_id`` first, but this+ # function BUILDS the URL, so an unvalidated id would be interpolated+ # straight into the path — a ``../..`` id yields a traversal route,+ # and an id containing ``#`` yields two fragments. The+ # docstring above has always promised a rejected id returns ``None``;+ # this makes that true.+ # Accept exactly what ``extract_document_id`` accepts — ``int``+ # (excluding ``bool``) or ``str`` — so the two genuinely cannot drift,+ # and build the URL from the STRIPPED value that was validated rather+ # than from the raw argument, or leading/trailing control characters+ # survive into the returned URL.+ if doc_id is None or isinstance(doc_id, bool):+ return None+ if not isinstance(doc_id, (int, str)):+ return None+ safe_doc_id = str(doc_id).strip()+ if not is_safe_document_id(safe_doc_id):return Noneif (chunk_index is Noneor isinstance(chunk_index, bool)or not isinstance(chunk_index, int)or chunk_index < 0+ or chunk_index > MAX_CHUNK_INDEX):return None- return f"/library/document/{doc_id}/chunks#chunk-{chunk_index}"+ return f"/library/document/{safe_doc_id}/chunks#chunk-{chunk_index}"diff --git a/src/local_deep_research/utilities/search_utilities.py b/src/local_deep_research/utilities/search_utilities.pyindex c2a95fcc52..c7162a664e 100644--- a/src/local_deep_research/utilities/search_utilities.py+++ b/src/local_deep_research/utilities/search_utilities.py@@ -4,9 +4,15 @@ from typing import Dict, Listfrom loguru import loggerfrom local_deep_research.text_optimization.citation_formatter import (+ is_line_breaking_char,LDR_APPENDED_SOURCES_SENTINEL,)-from .url_utils import canonical_url_key+from .url_utils import (+ CHUNK_DISPLAY_KEY,+ canonical_url_key,+ library_display_url,+ preferred_chunk_display,+)LANGUAGE_CODE_MAP = {@@ -217,6 +223,60 @@ def extract_links_from_search_results(search_results: List[Dict]) -> List[Dict]:return links+def _sanitize_sources_field(value: str) -> str:+ """Flatten a value being rendered into the Sources block.++ Titles come from search-result metadata — i.e. from whatever a page+ calls itself — and were previously rendered verbatim, so a crafted+ title could forge a whole extra numbered citation pointing anywhere.+ URLs get the same treatment because the non-library canonical-key+ fallback returns arbitrary scheme-less paths unchanged.++ Control characters are replaced with a space rather than dropped, so a+ forged ``\n[9] Fake`` degrades to visible text on the same line instead+ of silently vanishing.+ """+ if not value:+ return value+ return "".join(+ " " if is_line_breaking_char(ch) else ch for ch in value+ ).strip()+++def _owned_chunk_display(recorded: object, canon: str) -> str | None:+ """Return a recorded chunk anchor only if it names *this* citation.++ .. note::+ This check is the PRIMARY control, not defence in depth. Only+ ``SearchResultsCollector`` strips a producer-supplied key at+ ingest, and it exists solely in the LangGraph strategy —+ ``source_based``, ``focused_iteration`` and+ ``topic_organization`` all extend ``all_links_of_system`` with+ raw engine dicts. So on those paths a key present here is+ producer-supplied by construction, and the residual is bounded to+ what this function permits: a wrong chunk or view segment of the+ correctly-identified document, never a foreign document or an+ arbitrary URL.++ Shape validation alone is not enough. The value is preferred over the+ entry's own url, so a well-formed anchor for a DIFFERENT document —+ which arrives for free, since ``add_results`` copies the engine's dict+ — would render and persist under this citation. The writer refuses to+ record a foreign anchor; the reader has to refuse to read one, or the+ two disagree about which spelling is authoritative.++ Comparing canonical keys is exact here: ``canonical_url_key`` collapses+ every view of a library document onto one key, so an anchor for the+ same document matches and an anchor for any other does not.+ """+ if not isinstance(recorded, str):+ return None+ display = preferred_chunk_display(recorded)+ if display is None:+ return None+ return display if canonical_url_key(display) == canon else None++def format_links_to_markdown(all_links: List[Dict]) -> str:parts: list[str] = []logger.info(f"Formatting {len(all_links)} links to markdown...")@@ -236,11 +296,66 @@ def format_links_to_markdown(all_links: List[Dict]) -> str:# citation tag (e.g. `[mypapers-7]`) instead of falling back to# the generic `local` label.canon_to_collection: dict[str, str] = {}+ # Library routes key per-document (so a document's /pdf and+ # /chunks#chunk-N views share one entry) but must DISPLAY the first+ # URL seen, anchor included — the canonical key drops #chunk-<n>,+ # and that anchor is what makes a citation scroll to the cited+ # chunk. Non-library sources keep displaying the canonical form so+ # utm_*/fbclid clutter and credentials stay out of the report.+ canon_to_display: dict[str, str] = {}for link in all_links:raw = link.get("url") or link.get("link") or ""+ # Skipped, not coerced. These dicts reach here straight from+ # engine output on the non-LangGraph strategies, and+ # canonical_url_key raises on a non-str. Stringifying instead+ # renders a Python repr as a clickable URL and can merge two+ # distinct sources onto one citation — and it would disagree+ # with ``_citation_dedup_key``, which refuses a non-str link+ # outright and whose docstring requires the two to group+ # identically.+ if not isinstance(raw, str):+ continuecanon = canonical_url_key(raw)if not canon:continue+ # An anchored spelling recorded by the collector wins: after+ # canonical-key dedup only one entry per document survives, so+ # the anchor may live on that entry rather than in its ``url``.+ # Validate the recorded anchor rather than trusting it. It is+ # preferred over the entry's own url, so an unvalidated read+ # would let whatever set the key choose the rendered AND the+ # persisted URL — and the key rides in a dict built with+ # ``dict(raw)`` from engine output, so "only the collector+ # writes it" is a coincidence, not a control.+ # The entry's OWN anchored spelling wins, ahead of anything+ # recorded alongside it. ``_prefer_anchored_link`` refuses to+ # record over a collector-built anchor for exactly this reason+ # — a different chunk of the same document points the reader at+ # text the snippet did not come from — and the reader has to+ # apply the same rule or the two disagree. The recorded key+ # only fills the gap where the entry has no anchor of its own.+ display = (+ preferred_chunk_display(raw)+ or _owned_chunk_display(link.get(CHUNK_DISPLAY_KEY), canon)+ or library_display_url(raw)+ )+ if display:+ # Prefer a URL that carries a #chunk-<n> anchor over one+ # that doesn't. Views of a document key together, so+ # first-seen alone would show `/library/document/<id>/pdf`+ # — losing the anchor — whenever a chunk-less hit happened+ # to be registered first.+ #+ # The test is for ``#chunk-`` specifically, not for any+ # ``#``: an empty fragment (``/library/document/9#``) or an+ # unrelated one (``#section-intro``) would otherwise count+ # as "anchored", win the preference, and permanently lock+ # out the real chunk anchor that arrives later.+ current = canon_to_display.get(canon)+ if current is None or (+ "#chunk-" not in current and "#chunk-" in display+ ):+ canon_to_display[canon] = displayurl_to_indices.setdefault(canon, []).append(link.get("index", ""))canon_to_title.setdefault(canon, link.get("title", "Untitled"))# Track journal quality per canonical URL (first non-None wins)@@ -249,6 +364,8 @@ def format_links_to_markdown(all_links: List[Dict]) -> str:# First non-empty collection name wins (mirrors title/quality).if canon not in canon_to_collection:metadata = link.get("metadata") or {}+ if not isinstance(metadata, dict):+ metadata = {}collection = metadata.get("collection_name")if collection:canon_to_collection[canon] = str(collection)@@ -257,25 +374,62 @@ def format_links_to_markdown(all_links: List[Dict]) -> str:seen: set[str] = set()for link in all_links:raw = link.get("url") or link.get("link") or ""+ # Skipped, not coerced. These dicts reach here straight from+ # engine output on the non-LangGraph strategies, and+ # canonical_url_key raises on a non-str. Stringifying instead+ # renders a Python repr as a clickable URL and can merge two+ # distinct sources onto one citation — and it would disagree+ # with ``_citation_dedup_key``, which refuses a non-str link+ # outright and whose docstring requires the two to group+ # identically.+ if not isinstance(raw, str):+ continuecanon = canonical_url_key(raw)if not canon or canon in seen:continuetitle = canon_to_title[canon]+ # Coerced for the same reason the url is skipped: it comes+ # from the same engine dict, and ``.replace`` below raises on+ # a non-str, taking the whole Sources block with it.+ if title is not None and not isinstance(title, str):+ title = str(title)if title:- title = title.replace(LDR_APPENDED_SOURCES_SENTINEL, "")+ title = _sanitize_sources_field(+ title.replace(LDR_APPENDED_SOURCES_SENTINEL, "")+ )# Indices arrive as int (from strategy enumeration) or str (from# _build_sources_markdown's fallback). Coerce so dedup collapses# 1 and "1", and sorted() doesn't TypeError on mixed types.indices = sorted({str(i) for i in url_to_indices[canon]},- key=lambda s: (0, int(s)) if s.isdigit() else (1, s),+ # ``isdigit()`` is True for non-ASCII digit characters that+ # ``int()`` rejects — e.g. the superscript "\u00b9" — so it+ # alone would raise ValueError here and crash the whole+ # bibliography. Require ASCII before converting.+ key=lambda s: (+ (0, int(s)) if s.isascii() and s.isdigit() else (1, s)+ ),)+ # Sanitised like the title, collection and URL. Indices are+ # LDR-internal enumerations today, but ``_create_documents``+ # PRESERVES a pre-existing ``index`` on a raw engine dict and+ # ``_build_sources_markdown`` reads it back out of persisted+ # ``original_data`` — so "no engine emits this key" is a+ # coincidence, not a control.+ #+ # ``quality_tag`` below is the one field still interpolated+ # raw. It reaches the renderer by the same persisted route,+ # and is safe only incidentally: the fall-through branch that+ # builds it uses ``repr()``, which escapes newlines and+ # non-printables. Incidental, not designed — do not remove+ # that ``repr()`` without sanitising here instead.+ indices = [_sanitize_sources_field(i) for i in indices]indices_str = f"[{', '.join(indices)}]"quality_tag = _format_quality_tag(canon_to_quality.get(canon))collection = canon_to_collection.get(canon, "")if collection:- collection = collection.replace(- LDR_APPENDED_SOURCES_SENTINEL, ""+ collection = _sanitize_sources_field(+ collection.replace(LDR_APPENDED_SOURCES_SENTINEL, ""))collection_line = (f" Collection: {collection}\n" if collection else ""@@ -283,7 +437,7 @@ def format_links_to_markdown(all_links: List[Dict]) -> str:parts.append(f"{indices_str} {title}{quality_tag} "f"(source nr: {', '.join(map(str, indices))})\n"- f" URL: {canon}\n"+ f" URL: {_sanitize_sources_field(canon_to_display.get(canon, canon))}\n"f"{collection_line}"f"\n")diff --git a/src/local_deep_research/utilities/url_utils.py b/src/local_deep_research/utilities/url_utils.pyindex feedd02aab..5d8fe2b319 100644--- a/src/local_deep_research/utilities/url_utils.py+++ b/src/local_deep_research/utilities/url_utils.py@@ -2,10 +2,18 @@from functools import lru_cachefrom typing import Optional-from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit+import re+from urllib.parse import parse_qsl, unquote, urlencode, urlsplit, urlunsplitfrom loguru import logger+from .chunk_anchor import (+ LIBRARY_ALIAS_HOST,+ LIBRARY_ROUTE_PREFIXES,+ LIBRARY_ROUTE_SUFFIX_ALTERNATION,+ MAX_CHUNK_INDEX,+ is_safe_document_id,+)from ..security import redact_url_for_log, validate_urlfrom ..security.network_utils import is_private_ip@@ -14,6 +22,7 @@ __all__ = ["normalize_url","is_private_ip","canonical_url_key",+ "library_display_url","is_safe_custom_llm_endpoint",]@@ -94,6 +103,316 @@ def normalize_url(raw_url: str) -> str:return f"{scheme}://{raw_url}"+# Internal library-document routes emitted as citation URLs by the RAG+# search engines (see ``LibraryRAGSearchEngine._get_document_url``). These+# are the only scheme-less URLs whose shape ``canonical_url_key`` knows+# well enough to canonicalize; everything else falls back to the raw+# string. Imported from ``chunk_anchor`` so the producer side (which+# decides what counts as a library hit) and this consumer side cannot+# disagree about the prefix set.+_LIBRARY_ROUTE_PREFIXES = LIBRARY_ROUTE_PREFIXES+++# The absolute alias the agent sometimes emits for a library document; the+# fetch resolver accepts it (see ``library_resolver._parse_library_reference``)+# and hands the ORIGINAL string back, which is then registered as a citation+# link. It must key and display as the relative route it denotes, or one+# document occupies two bibliography entries.+_LIBRARY_ALIAS_SCHEME = "https"+_LIBRARY_ALIAS_HOST = LIBRARY_ALIAS_HOST+# Mirrors ``library_resolver._LIBRARY_PATH_RE``: a document id plus at most+# one known view. Anything deeper is not a library route and must not be+# rewritten into one.+_LIBRARY_ALIAS_PATH_RE = re.compile(+ rf"^/(?P<doc_id>[^/?#]+)(?:/(?P<suffix>{LIBRARY_ROUTE_SUFFIX_ALTERNATION}))?/?$"+)+# The same shape for the RELATIVE routes, including the ``/lib/`` spelling.+# Both spellings denote one document (``library_resolver._LIBRARY_PATH_RE``+# accepts either), so both normalize to the ``/library/`` form: keying them+# separately fans a document across two bibliography entries, and only+# ``/library/document/<id>`` is a registered Flask route — a ``/lib/...``+# link renders dead.+_LIBRARY_ROUTE_PATH_RE = re.compile(+ r"^/(?:library|lib)/document/(?P<doc_id>[^/?#]+)"+ rf"(?:/(?P<suffix>{LIBRARY_ROUTE_SUFFIX_ALTERNATION}))?/?$"+)+++def _is_library_route(path: str) -> bool:+ return path.startswith(_LIBRARY_ROUTE_PREFIXES)+++def _has_control_chars(value: str) -> bool:+ """True if *value* carries a C0/DEL character.++ These are rendered verbatim into the ``## Sources`` block, where an+ embedded newline forges extra lines. The library-route branches build+ their result by string-splitting rather than via ``urlunsplit``, so —+ unlike the absolute-URL branch — they cannot drop such a character on+ their own and must reject it explicitly.+ """+ return any(ord(ch) < 0x20 or ord(ch) == 0x7F for ch in value)+++def _normalize_library_alias(url: str) -> str | None:+ """Map ``https://library.document/<id>[/pdf|/chunks]`` to its relative route.++ Returns ``None`` for anything else. The accepted shape is deliberately+ narrow — a document id followed by at most the ``pdf`` or ``chunks``+ view, mirroring ``library_resolver._LIBRARY_PATH_RE`` — because the+ result is keyed AND displayed as an internal route. A permissive+ rewrite lets a crafted alias merge into a real document's bibliography+ entry, inherit its title, and replace its link: pasting an unvalidated+ path after ``/library/document`` turns+ ``https://library.document/<id>/../../../admin/x#chunk-1`` into a+ same-origin link to ``/admin/x`` rendered under the real document's+ name.++ Two deliberate differences from the resolver, both narrowing or+ identity-preserving rather than widening:++ * A query disqualifies (as in the resolver) — it can carry+ server-meaningful state, unlike a client-side anchor.+ * Host comparison uses ``hostname``/``port``, so ``LIBRARY.DOCUMENT``,+ ``:443`` and userinfo forms name the same document rather than+ fanning out into separate bibliography entries. The resolver rejects+ those for fetching; they are still the same source for citation+ purposes.++ The fragment is carried over — it is the chunk anchor.+ """+ # Cheap reject first: this runs once per citation inside+ # ``format_links_to_markdown``, and almost every real citation is a+ # plain external URL that can never match. Skipping ``urlsplit`` for+ # those keeps the bibliography pass off a per-link parse.+ if "://" not in url or _LIBRARY_ALIAS_HOST not in url.lower():+ return None+ # Reject tab/newline/CR before ``urlsplit`` sees them: it deletes them+ # silently, which would collide ``/a<TAB>b`` with a real ``/ab`` and+ # rewrite the citation to a DIFFERENT document. The neighbouring+ # manual splits avoid ``urlsplit`` for exactly this reason.+ if any(ch in url for ch in "\t\n\r"):+ return None+ try:+ parsed = urlsplit(url)+ except ValueError:+ return None+ if (+ parsed.scheme != _LIBRARY_ALIAS_SCHEME+ or parsed.query+ or (parsed.hostname or "") != _LIBRARY_ALIAS_HOST+ ):+ return None+ try:+ if parsed.port not in (None, 443):+ return None+ except ValueError: # malformed port+ return None+ path_match = _LIBRARY_ALIAS_PATH_RE.match(parsed.path)+ if not path_match:+ return None+ doc_id = unquote(path_match.group("doc_id"))+ # Same predicate ``extract_document_id`` applies, imported rather than+ # restated: the doc id is decoded before BOTH validating and+ # re-emitting, so ``/%2D`` and ``/-`` forms of one id give one key.+ if not is_safe_document_id(doc_id):+ return None+ suffix = path_match.group("suffix")+ rel = f"/library/document/{doc_id}"+ if suffix:+ rel = f"{rel}/{suffix}"+ return f"{rel}#{parsed.fragment}" if parsed.fragment else rel+++def _parse_library_citation(raw: str) -> tuple[str, str] | None:+ """Return ``(dedup_key, display_url)`` for a library citation, else ``None``.++ One parser for every accepted spelling — the relative+ ``/library/document/<id>`` route, its ``/lib/`` abbreviation, and the+ absolute ``https://library.document/<id>`` alias — so the two paths+ cannot enforce different rules. They used to: only the alias validated+ its document id, which meant the exact traversal this function's+ sibling docstring promises to block sailed straight through when the+ same URL arrived relative. ``/library/document/7/../../../admin/wipe``+ keyed as document ``7`` and *displayed* as ``/admin/wipe``, so a+ crafted citation merged into a real document's entry, inherited its+ title and citation numbers, and replaced its link.++ * ``dedup_key`` is ``/library/document/<id>`` — the ``/pdf``,+ ``/chunks`` and ``#chunk-<n>`` views of one document are one source+ and belong on one bibliography line.+ * ``display_url`` keeps the view suffix and the ``#chunk-<n>`` anchor,+ which is what makes a citation scroll to the cited text.++ Normalisations, all identity-preserving: ``/lib/`` becomes+ ``/library/``, the document id is percent-decoded (so ``a%2Db`` and+ ``a-b`` are one key, matching what the resolver looks up), and a query+ string is dropped (the route ignores it).+ """+ if not raw:+ return None+ stripped = raw.strip()+ # Cheap reject before any parsing: the overwhelming majority of+ # citations are ordinary external URLs.+ if not (+ stripped.startswith(_LIBRARY_ROUTE_PREFIXES)+ or ("://" in stripped and _LIBRARY_ALIAS_HOST in stripped.lower())+ ):+ return None+ # Refuse anything carrying a control character rather than truncating+ # the doc-id segment at it. Truncating keys the crafted URL as the+ # real ``/library/document/<id>`` it was prefixed with, merging it+ # into that document's bibliography entry and lending it the+ # document's title and citation number. Returning ``None`` keeps the+ # two sources apart; the payload still cannot forge a Sources line,+ # because every field rendered into that block goes through+ # ``search_utilities._sanitize_sources_field``.+ if _has_control_chars(stripped):+ return None+ alias = _normalize_library_alias(stripped)+ if alias is not None:+ stripped = alias+ if not _is_library_route(stripped):+ return None+ # Split fragment and query manually rather than via ``urlsplit``,+ # which silently deletes embedded tab/newline and would collide+ # ``/a<TAB>b`` with a real ``/ab``.+ head, sep, fragment = stripped.partition("#")+ path = head.split("?", 1)[0]+ path_match = _LIBRARY_ROUTE_PATH_RE.match(path)+ if not path_match:+ return None+ doc_id = unquote(path_match.group("doc_id"))+ # Same predicate ``extract_document_id`` applies, imported rather than+ # restated: the id is decoded before BOTH validating and re-emitting,+ # so ``/%2D`` and ``/-`` forms of one id give one key.+ if not is_safe_document_id(doc_id):+ return None+ key = f"/library/document/{doc_id}"+ suffix = path_match.group("suffix")+ display = f"{key}/{suffix}" if suffix else key+ # Re-attach the fragment ONLY when it is a real chunk anchor. A+ # library route carries no other meaningful fragment, so anything else+ # is either a producer bug or an unvalidated string that reached us+ # from the agent (``fetch_content`` passes its argument through+ # ``library_resolver`` verbatim). Re-emitting it built a citation URL+ # whose anchor cannot name a chunk — a dead link that still claims to+ # cite one — and smuggled arbitrary text into the rendered Sources+ # block, bypassing the ``MAX_CHUNK_INDEX``/format contract that+ # ``chunk_anchor`` calls itself the single source of truth for.+ # Dropping it keeps the citation (the route still resolves to the+ # document) and loses only an anchor that never pointed anywhere.+ if sep and fragment and is_valid_chunk_fragment(fragment):+ display = f"{display}#{fragment}"+ return key, display+++def library_display_url(raw: str) -> str | None:+ """Return the display URL when *raw* is a library route, else ``None``.++ ``canonical_url_key`` collapses library routes to a per-document key,+ which is right for grouping and wrong for display: the ``#chunk-<n>``+ anchor is the entire point of a chunk-targeted citation. Renderers+ group on the key and display this.++ The returned URL is the *normalized* route (``/library/`` spelling,+ percent-decoded id, no query), never the caller's raw string — a+ string that does not parse as a library route returns ``None`` rather+ than being echoed back into the report.+ """+ parsed = _parse_library_citation(raw)+ return parsed[1] if parsed is not None else None+++# A chunk anchor is ``#chunk-<n>`` with the exact index shape+# ``build_chunk_anchor_url`` emits. ``\d`` is Unicode-wide, so it is spelled+# out here: ``chunk-\u0667`` and ``chunk-12`` would otherwise be accepted+# while ``chunk_anchor.extract_chunk_index`` — which calls itself the+# single source of truth for this — rejects them.+# ``\Z``, not ``$``: Python's ``$`` also matches just before a single+# trailing newline even without ``re.MULTILINE``, so ``chunk-1\n``+# passed. Both current callers strip and reject control characters+# upstream, so nothing reaches here with one today — but this is the+# shared predicate, and a future caller that skips those gates would+# inherit a newline-smuggling hole in the one rule both paths trust.+_CHUNK_FRAGMENT_RE = re.compile(r"^chunk-(0|[1-9][0-9]*)\Z", re.ASCII)+++def is_valid_chunk_fragment(fragment: str) -> bool:+ """Return ``True`` if *fragment* (the part after ``#``) is a chunk+ anchor that can actually name a chunk.++ Shared by :func:`library_display_url` (which drops a fragment failing+ this test) and :func:`preferred_chunk_display` (which rejects the whole+ URL), so the render path and the store path cannot disagree about what+ a valid anchor is.+ """+ if not isinstance(fragment, str):+ return False+ match = _CHUNK_FRAGMENT_RE.match(fragment)+ if match is None:+ return False+ # Bound the digit run BEFORE converting: the pattern admits an+ # unbounded one, and int() raises above CPython's 4300-digit limit —+ # out of a pure formatter with no caller catching it, which would take+ # down the whole Sources block.+ if len(match.group(1)) > len(str(MAX_CHUNK_INDEX)):+ return False+ # Bound it the way the producer does, so a fragment that cannot name a+ # real chunk is not treated as though it could.+ return int(match.group(1)) <= MAX_CHUNK_INDEX+++# Result-dict key holding a chunk-anchored spelling recorded alongside a+# citation whose stored link has no anchor. A separate key, never an+# overwrite of ``link``/``url``: consumers opt in, and a stray write can+# only add an ignored hint rather than corrupt a citation URL.+CHUNK_DISPLAY_KEY = "chunk_display_url"+++def preferred_chunk_display(raw: str) -> str | None:+ """Return the display URL when *raw* is a library route WITH a valid+ chunk anchor, else ``None``.++ ``library_display_url`` validates the route and keeps a VALID chunk+ anchor, dropping any other fragment — a citation without an anchor is+ still a citation. This is the stricter question a caller asks when it+ wants to *store* an anchored spelling in preference to one it already+ has: the anchor has to be present, not merely permitted.++ (This paragraph said "re-emits the fragment verbatim" until that+ behaviour was removed — the wording was corrected in the twin function+ and left stale here, which is the same one-instance-at-a-time slip the+ functions themselves keep exhibiting.)++ One helper for both callers on purpose. The collector previously+ re-derived this rule from a comment saying it matched the renderer's,+ and the copy drifted twice — once accepting any ``#chunk-`` substring,+ once validating the route while ignoring the fragment entirely.+ """+ display = library_display_url(raw)+ if display is None:+ return None+ _, sep, fragment = display.partition("#")+ # ``library_display_url`` already drops a fragment that fails this+ # test, so the ``is_valid_chunk_fragment`` call here is INERT today:+ # mutating it away kills no test, because nothing can reach it with a+ # fragment that would fail. It is defence in depth, not a live check —+ # said plainly so nobody reads a passing test as evidence it fires.+ #+ # Kept anyway, deliberately: it makes this function correct+ # independently of what ``library_display_url`` does, and the two+ # answer different questions ("what do I render" vs "is this worth+ # storing"), so a future change to either must not silently make this+ # one the looser of the pair — which is exactly how the collector's+ # hand-rolled copy of this rule drifted twice before it was+ # centralised here. The ``not sep`` half IS load-bearing: it is what+ # distinguishes "has an anchor" from "has none".+ if not sep or not is_valid_chunk_fragment(fragment):+ return None+ return display++@lru_cache(maxsize=1024)def canonical_url_key(url: str) -> str:"""Return a canonical form of ``url`` suitable for deduplication and@@ -114,12 +433,28 @@ def canonical_url_key(url: str) -> str:Percent-encoding is not normalized; query param order is preservedas-is.- Falls back to ``url.strip()`` when the input is not a recognizable- absolute URL (e.g. ``mailto:``, ``data:``, or protocol-relative- ``//host/p``), since canonicalization would be ambiguous.+ Internal library-document routes (``/library/document/<id>``, plus its+ ``/pdf`` and ``/chunks#chunk-<n>`` views) collapse to a per-document+ key, because they are views of one source and belong on one+ bibliography line. That key drops the ``#chunk-<n>`` anchor, so+ anything that RENDERS a link must display the original URL instead —+ see :func:`library_display_url`.++ Falls back to ``url.strip()`` for everything else that is not a+ recognizable absolute URL (``mailto:``, ``data:``, protocol-relative+ ``//host/p``, bare filesystem paths, SPA ``/app#/route`` links), since+ canonicalization would be ambiguous — and merging distinct sources is+ worse than leaving them separate."""if not url:return ""+ # Library-document routes — relative, ``/lib/``-abbreviated, or the+ # absolute ``https://library.document/<id>`` alias the agent emits —+ # all key to one per-document route, so every view of one document+ # shares a single bibliography entry.+ library = _parse_library_citation(url)+ if library is not None:+ return library[0]try:parsed = urlsplit(url)except Exception:@@ -127,6 +462,28 @@ def canonical_url_key(url: str) -> str:# Require both a scheme and a netloc; otherwise canonicalization is# ambiguous (mailto:, data:, protocol-relative, etc.).if not parsed.scheme or not parsed.netloc:+ # Library routes were already handled above. Everything else that+ # is not a recognizable absolute URL falls back to the raw string:+ # canonicalization is deliberately NOT generalized to every+ # root-relative path, because a LangChain retriever sets a+ # result's url from its ``source`` metadata, which is commonly an+ # absolute filesystem path. Treating those as routes merges+ # genuinely distinct sources — ``/docs/C#/a.md`` and+ # ``/docs/C#/b.md`` would both key to ``/docs/C``, a path that+ # does not exist — which is worse than the fan-out being fixed.+ # The same applies to SPA-style ``/app#/route`` links.+ #+ # A library route with a control character does NOT reach here as+ # a library key: ``_parse_library_citation`` refuses it, so it+ # falls through to this verbatim fallback. That is deliberate.+ # Truncating the doc-id segment instead would key the crafted URL+ # as the real ``/library/document/<id>`` it was prefixed with,+ # merging it into that document's bibliography entry and handing+ # it the document's title and citation number. The payload cannot+ # forge a line from here either: every field rendered into the+ # ``## Sources`` block — this URL included — goes through+ # ``search_utilities._sanitize_sources_field``, which flattens+ # control characters to spaces.return url.strip()scheme = parsed.scheme.lower()diff --git a/src/local_deep_research/web/services/research_sources_service.py b/src/local_deep_research/web/services/research_sources_service.pyindex 4358ad5315..db9f3d00ef 100644--- a/src/local_deep_research/web/services/research_sources_service.py+++ b/src/local_deep_research/web/services/research_sources_service.py@@ -9,6 +9,8 @@ from typing import List, Dict, Any, Optionalfrom datetime import datetime, UTCfrom loguru import loggerfrom sqlalchemy import or_+import numbers+from sqlalchemy.exc import IntegrityErrorfrom sqlalchemy.orm import Session, load_only@@ -21,6 +23,375 @@ from ...database.models import ()from ...database.session_context import get_user_db_sessionfrom ...utilities.citation_normalizer import normalize_citation+from ...utilities.url_utils import (+ CHUNK_DISPLAY_KEY,+ canonical_url_key,+ library_display_url,+ preferred_chunk_display,+)+++def _as_text(value: object) -> str:+ """Coerce a stored-dict value to ``str`` for slicing and ``len()``.++ Every read in the per-source loop comes from a JSON blob that+ round-tripped through the database, so an ``int`` or ``list`` in a+ text field is possible. Slicing one raises inside the broad+ ``except`` below, which rolls back and drops the whole citation with+ no message — the same silent-loss failure the url guards prevent.+ """+ if value is None:+ return ""+ if isinstance(value, str):+ return value+ try:+ return str(value)+ except Exception:+ # A coercion helper on a silent-drop path must not be able to+ # raise: an object whose ``__str__`` fails would otherwise take+ # out the exception handler that reports the failure, and with it+ # every sibling source in the batch.+ return ""+++# Every field in this loop that is read as text — by normalize_citation,+# by a slice, as a dict key, or as a Text column. Coercing them once at+# the top of the loop is what stops the "guard the read a review named,+# miss its sibling" cycle: nine commits guarded url, then snippet, then+# ct_matched, then title, while pmid, arxiv_id and normalize_citation's+# own re-read of url stayed raw and kept dropping the citation.+_TEXT_FIELDS = (+ "url",+ "link",+ "title",+ "name",+ "snippet",+ "content_preview",+ "description",+ "source_type",+ "source_engine",+ "source",+ "container_title",+ "container-title",+ "journal",+ "venue",+ "journal_ref",+ "journal_name_matched",+ "volume",+ "issue",+ "pages",+ "publisher",+)+++# Identifier fields. These are NOT text-coerced: they feed ``unique``+# columns on ``Paper`` and the dedup SELECT that decides whether two+# engines found the same paper. ``_extract_doi`` documents and tests a+# LIST doi (it takes ``doi[0]``), so stringifying one writes+# ``"['10.1/x', '10.2/y']"`` into a unique key — worse than the drop it+# replaced, because it is permanent and defeats dedup. Unwrap instead.+_IDENTIFIER_FIELDS = ("doi", "pmid", "pmcid", "arxiv_id")+++# The two fields whose contents ``_parse_authors_list`` turns into author+# records. Only inside these does an empty name key mean "junk author".+_AUTHOR_LIST_KEYS = ("authors", "authors_csl")++# Author fields whose emptiness produces a junk author record.+# ``name``/``display_name`` are the two ``citation_normalizer._parse_name``+# strips — ``display_name`` is the OpenAlex shape, missing from an earlier+# spelling of this tuple. ``family``/``given``/``suffix`` are copied+# verbatim by ``_parse_authors_list``'s CSL branch, which tests only+# ``"family" in author``, so an empty one of those is embedded in the+# stored csl_json rather than ignored. ``literal`` is deliberately absent:+# ``_parse_name`` only ever EMITS it, never reads it from input, and an+# earlier spelling carried it (plus two more) as keys that did nothing.+_AUTHOR_NAME_KEYS = ("name", "display_name", "family", "given", "suffix")+++# Where the walk currently is, relative to an author record. A bool was+# not enough: it could say "inside authors" but not "inside an author+# record's OWN keys", so once set it stayed set for every descendant and+# deleted an empty ``given`` from ``authors[0].affiliation`` or an+# OpenAlex ``institutions[].display_name`` — neither of which+# ``_parse_authors_list`` ever reads.+_AUTHORS_OUTSIDE = 0+_AUTHORS_LIST = 1 # this value IS an authors/authors_csl list+_AUTHORS_RECORD = 2 # this value IS one author record+++def _json_text_safe(+ value: object, _depth: int = 0, _authors: int = _AUTHORS_OUTSIDE+) -> object:+ """Recursively make *value* safe to serialize and to read as text.++ The ingest boundary is the TREE, not the top-level dict. ``metadata``+ and ``authors`` come from the same untrusted producer, are read by the+ same ``normalize_citation`` call, and are serialized into the same JSON+ column — so a ``set`` under ``metadata["journal"]`` or an ``int`` under+ ``authors[0]["name"]`` raises exactly where a top-level one used to.++ ``_authors`` scopes the empty-name drop to an author record's OWN+ keys, which are the only ones ``_parse_authors_list`` reads. Anything+ nested below them is inert data the producer may want preserved.+ """+ if _depth > 6:+ return str(value)+ if value is None or isinstance(value, (str, int, float, bool)):+ return value+ if isinstance(value, dict):+ in_record = _authors == _AUTHORS_RECORD+ # Author name fields are coerced ONCE and reused. Calling+ # ``_as_text`` separately in the filter and the value expression+ # meant a non-deterministic ``__str__`` could pass the filter on+ # one call and write "" from the next.+ coerced = (+ {+ str(k): _as_text(v)+ for k, v in value.items()+ if k in _AUTHOR_NAME_KEYS+ }+ if in_record+ else {}+ )+ return {+ str(k): (+ coerced[str(k)]+ if in_record and k in _AUTHOR_NAME_KEYS+ # Descending out of the record: re-arm only on another+ # authors key, never inherit.+ else _json_text_safe(+ v,+ _depth + 1,+ _AUTHORS_LIST+ if k in _AUTHOR_LIST_KEYS+ else _AUTHORS_OUTSIDE,+ )+ )+ for k, v in value.items()+ # An author name that coerces to nothing is DROPPED, not kept.+ # ``_parse_name(None)`` raises and ``_parse_name("")``+ # synthesises ``{"literal": ""}``; the CSL branch of+ # ``_parse_authors_list`` copies ``family``/``given``/``suffix``+ # verbatim, so an empty one of those lands as a junk author in+ # the stored csl_json just the same. ``.strip()`` because+ # ``_parse_name`` strips before its own emptiness test, so a+ # whitespace-only name produces the same junk.+ if not (+ in_record+ and k in _AUTHOR_NAME_KEYS+ and not (coerced.get(str(k)) or "").strip()+ )+ }+ if isinstance(value, (list, tuple)):+ # The ELEMENTS of an authors list are the author records.+ child = (+ _AUTHORS_RECORD if _authors == _AUTHORS_LIST else _AUTHORS_OUTSIDE+ )+ return [_json_text_safe(v, _depth + 1, child) for v in value]+ return str(value)+++# SQLite's INTEGER column is 64-bit. Python's int is not, so a value can+# pass ``int()`` inside citation_normalizer and still raise OverflowError+# at flush() — and because that is not an IntegrityError, the per-source+# SAVEPOINT retry does not apply and the Session is left rolled back, so+# every LATER source in the batch is lost too and nothing commits.+_SQLITE_INT_MIN = -(2**63)+_SQLITE_INT_MAX = 2**63 - 1+++def _coerce_year(value: object) -> object:+ """Drop a year that cannot be stored, rather than losing the batch."""+ if value is None or isinstance(value, bool):+ return value+ # Duck-typed on the operation, not on a type list. ``_parse_date`` does+ # ``int(raw_year)``, which accepts anything with ``__int__`` — numpy+ # scalars, Decimal, Fraction, even UUID — and this file's own comments+ # name numpy as an expected producer. Gating on ``(int, float, str)``+ # let every one of those past the bound and into flush().+ try:+ as_int = int(float(value) if isinstance(value, float) else value)+ except (ValueError, OverflowError, TypeError, ArithmeticError):+ # Not int()-able, so ``_parse_date`` cannot overflow on it either:+ # leave strings for its regex, drop anything else.+ return value if isinstance(value, str) else None+ return value if _SQLITE_INT_MIN <= as_int <= _SQLITE_INT_MAX else None+++def _coerce_identifier(value: object) -> object:+ """Return *value* fit for a ``unique`` identifier column, else ``None``.++ Unwrap-or-DROP, never unwrap-or-stringify. These feed ``Paper.doi``,+ ``.pmid`` and ``.arxiv_id`` and the dedup SELECT, where a repr is worse+ than absence twice over: it is permanent, and two DIFFERENT papers+ carrying the same unparseable value match each other and collapse into+ one row. Absence just means "this engine gave us no identifier", which+ ``_extract_doi`` already handles by falling through to its next+ channel.++ A list is unwrapped once and its element re-checked, so ``[["10.1/x"]]``+ drops rather than stringifying the inner list.+ """+ if isinstance(value, (list, tuple)):+ value = value[0] if value else None+ if value is None or isinstance(value, str):+ return value or None+ # Integral-ness, not ``isinstance(int)``: a PMID legitimately arrives+ # as an int, and numpy.int64 / Decimal("12345") are integers that are+ # NOT int subclasses. Rejecting them dropped real identifiers and+ # broke dedup — the same harm as stringifying, in the other direction.+ # ``str()`` on a large int is exact, so there is nothing to protect+ # against here.+ # A numpy boolean is neither a Python ``bool`` nor a+ # ``numbers.Real``/``Integral``, so it slips both gates while+ # ``int()``-ing to 1 and comparing equal to it — a True in a pmid+ # field was stored as the identifier "1".+ #+ # Matched on the dtype kind rather than the type name: numpy 1.x calls+ # the scalar ``bool_`` and numpy 2.x calls it ``bool``, so a name check+ # written against either version silently misses the other.+ if isinstance(value, bool):+ return None+ try:+ # ``getattr`` only swallows a MISSING attribute; a ``.kind``+ # property that raises would propagate out of a coercion helper+ # that every other step in this function keeps total.+ if getattr(getattr(value, "dtype", None), "kind", None) == "b":+ return None+ except Exception:+ return None+ # ``numbers.Real`` rather than ``isinstance(value, float)``: numpy's+ # float32 is not a float subclass, so an exactly-integral one slipped+ # through as an identifier. Engines send ints and strings, never reals.+ if isinstance(value, numbers.Real) and not isinstance(+ value, numbers.Integral+ ):+ return None+ try:+ as_int = int(value)+ except (ValueError, OverflowError, TypeError, ArithmeticError):+ return None+ # Exactness. ``numbers.Integral`` first, because ``as_int != value``+ # relies on a symmetric ``__eq__``: numpy ints and Decimal have one,+ # so both are accepted. An object that defines ``__int__`` but neither+ # registers as Integral nor compares equal to its own integer is+ # DROPPED — deliberately. This feeds a unique column, and a value that+ # cannot be shown equal to the integer it claims to be is not one to+ # guess at. So this is int()-ability plus provable exactness, not+ # int()-ability alone.+ if not isinstance(value, numbers.Integral) and as_int != value:+ return None+ return str(as_int) if as_int else None+++def normalize_source_fields(source: dict) -> dict:+ """Return a copy of *source* with its text fields coerced to ``str``.++ Engine dicts reach this service from arbitrary producers — a LangChain+ retriever's ``Document.metadata`` is whatever the user put there — and+ a non-str reaching a slice, a dict key, a regex or a Text column raises+ inside the per-source ``except``, which rolls back and drops the whole+ citation with only a counter to show for it.++ A falsy non-str becomes ``""``, not its repr: ``{"link": []}`` should+ keep skipping the source, as it did before any coercion existed, rather+ than persisting the string ``"[]"`` and rendering it as a link.++ Typed fields are left alone — ``authors`` stays a list, ``year`` an+ int, ``score`` a float — and only a non-dict ``metadata`` is replaced,+ since callers index into it.+ """+ normalized = dict(source)+ for field in _TEXT_FIELDS:+ value = normalized.get(field)+ if value is None or isinstance(value, str):+ continue+ normalized[field] = _as_text(value) if value else ""+ for field in ("year", "publication_year"):+ if field in normalized:+ normalized[field] = _coerce_year(normalized[field])++ for field in _IDENTIFIER_FIELDS:+ # ``in``, not ``.get``: materialising ``doi: None`` on every plain+ # web result changes what ``original_data`` stores for rows that+ # never had an academic identity.+ if field not in normalized:+ continue+ normalized[field] = _coerce_identifier(normalized[field])++ if "metadata" in normalized:+ metadata = normalized.get("metadata")+ normalized["metadata"] = (+ _json_text_safe(metadata) if isinstance(metadata, dict) else {}+ )++ # D2: ``authors_csl`` is read BEFORE ``authors`` by+ # ``normalize_citation``, in the same expression, and is NASA ADS's+ # primary author channel — normalizing only one of the two left the+ # other reaching the same json.dumps.+ for field in _AUTHOR_LIST_KEYS:+ value = normalized.get(field)+ if isinstance(value, (list, tuple)):+ normalized[field] = _json_text_safe(+ list(value), _authors=_AUTHORS_LIST+ )++ # D1: ``external_ids`` is the second DOI source ``_extract_doi``+ # documents. Leaving it raw put a list DOI into the unique Paper.doi+ # by the very channel the identifier unwrap above exists to protect.+ for field in ("external_ids", "externalIds"):+ value = normalized.get(field)+ if value is not None:+ normalized[field] = (+ _json_text_safe(value) if isinstance(value, dict) else {}+ )+ for key in ("DOI", "doi"):+ if key in normalized[field]:+ normalized[field][key] = _coerce_identifier(+ normalized[field][key]+ )+ return normalized+++def select_source_url(source: dict) -> object:+ """Choose the URL to persist for *source*.++ Extracted so the test suite can exercise THIS function. It previously+ lived inline and was pinned by a test that reimplemented it; the mirror+ stayed faithful right up until it faithfully mirrored a bug, and went+ green while the hole was open.++ Fails CLOSED on a non-str url: an earlier version made the isinstance+ check one conjunct of the rejection condition, so a non-str url made+ the whole ``and``-chain false, the ownership check never ran, and a+ recorded anchor naming a DIFFERENT document was persisted unchecked.+ """+ own_url = source.get("url", "") or source.get("link", "")+ own_key = canonical_url_key(own_url) if isinstance(own_url, str) else None++ raw_key = source.get(CHUNK_DISPLAY_KEY)+ recorded = (+ preferred_chunk_display(raw_key) if isinstance(raw_key, str) else None+ )+ # No usable key for this entry means no way to prove ownership, so the+ # recorded anchor is refused rather than trusted.+ if recorded and (own_key is None or canonical_url_key(recorded) != own_key):+ recorded = None++ own_anchor = (+ preferred_chunk_display(own_url) if isinstance(own_url, str) else None+ )+ # Normalise before falling back to the raw string. Without this the+ # absolute alias is persisted as typed, and the ``:443``/userinfo+ # spellings — which the alias parser accepts as the same document but+ # ``library_resolver`` refuses, since it compares netloc exactly —+ # become dead links in ``research_resources.url``. Returns None for a+ # non-library URL, so external sources keep their own spelling.+ normalised = (+ library_display_url(own_url) if isinstance(own_url, str) else None+ )+ return own_anchor or recorded or normalised or own_urlclass ResearchSourcesService:@@ -81,17 +452,40 @@ class ResearchSourcesService:for source in sources:sp = Nonetry:+ # Once, before anything reads it — see+ # normalize_source_fields.+ source = normalize_source_fields(source)# Extract fields from various possible formats- url = source.get("url", "") or source.get("link", "")- title = source.get("title", "") or source.get(- "name", ""+ # A chunk-anchored spelling recorded by the+ # collector wins. After canonical-key dedup only+ # one entry per library document survives, and it+ # may be the anchor-less view — persisting that+ # loses the #chunk-<n> permanently, since this row+ # is what later renders the source.+ # Validated, not trusted: this value is written+ # to the database and later rendered as a link, so+ # an unchecked read would let whatever set the key+ # choose the stored URL.+ # ``_as_text`` here, not just at the slices: these+ # three reach ``Text`` columns, and a list or dict+ # raises at ``flush()`` rather than at the read —+ # so guarding the read alone only moved the+ # exception, and the citation was dropped just the+ # same.+ url = _as_text(select_source_url(source))+ title = _as_text(+ source.get("title", "") or source.get("name", ""))- snippet = (+ # Coerced, not assumed: these come from the same+ # stored dict as the url, and a non-str raises+ # inside the per-source ``except`` below — which+ # drops the whole citation silently.+ snippet = _as_text(source.get("snippet", "")or source.get("content_preview", "")or source.get("description", ""))- source_type = source.get("source_type", "web")+ source_type = _as_text(source.get("source_type", "web"))# Skip if no URLif not url:@@ -139,6 +533,11 @@ class ResearchSourcesService:# export). Memoized per batch to avoid repeat# lookups for the same venue.ct = citation_fields.get("container_title")+ # Coerced before use as a dict key: CSL+ # ``container-title`` is legitimately an array,+ # and an unhashable value raises here — 34+ # lines before the guard on the same value.+ ct = _as_text(ct) or Noneif ct in journal_id_cache:journal_id = journal_id_cache[ct]else:@@ -173,6 +572,12 @@ class ResearchSourcesService:ct_matched = (source.get("journal_name_matched") or ct_raw)+ # ``or None`` like its sibling above: an empty+ # container_title must stay NULL, or every+ # venue-less paper forms an empty-string group+ # in the journal metrics instead of being+ # excluded by ``isnot(None)``.+ ct_matched = _as_text(ct_matched) or Noneif ct_matched and len(ct_matched) > 500:logger.debug(f"Truncating container_title to 500 "@@ -274,11 +679,33 @@ class ResearchSourcesService:# Roll back just this source's savepoint; earlier# sources in the batch stay committed at the# outer transaction level.- if sp is not None and sp.is_active:- sp.rollback()+ #+ # Unconditional, NOT gated on ``sp.is_active``: a+ # failure inside flush() deactivates the savepoint,+ # so the guard skipped the rollback in exactly the+ # case that needs it. The Session then stayed in+ # its failed state, every later source raised+ # PendingRollbackError, and the final commit()+ # raised out of the function — one bad source lost+ # the whole batch and the caller got an exception+ # instead of a reduced count.+ if sp is not None:+ try:+ sp.rollback()+ except Exception:+ # Already unwound; nothing further to undo+ # for this source.+ logger.debug(+ "savepoint rollback was already done"+ )failed_count += 1+ # ``_as_text``: the only unguarded read of url+ # left, and it runs when something has ALREADY+ # failed — an exception raised here escapes the+ # handler and loses the sibling sources too.logger.exception(- f"Failed to save source {source.get('url', 'unknown')}"+ "Failed to save source "+ + _as_text(source.get("url", "unknown")))continuediff --git a/src/local_deep_research/web_search_engines/engines/search_engine_collection.py b/src/local_deep_research/web_search_engines/engines/search_engine_collection.pyindex e0bae65140..0b403d1db7 100644--- a/src/local_deep_research/web_search_engines/engines/search_engine_collection.py+++ b/src/local_deep_research/web_search_engines/engines/search_engine_collection.py@@ -259,9 +259,16 @@ class CollectionSearchEngine(LibraryRAGSearchEngine):# malformed chunk index (UUID, boolean, negative) or# path-traversal doc_id cannot leak into the citation.chunk_idx = extract_chunk_index(metadata)+ # Authoritative id FIRST. ``extract_document_id`` scans its+ # first mapping before any later one, so passing ``metadata``+ # first would let a chunk's denormalised metadata override+ # ``r.source_id`` — the DocumentChunk FK that decides which+ # document the chunk actually belongs to. Pre-#5381 the+ # column always won; a divergent metadata id would otherwise+ # point the citation at a different document.sanitised_doc_id = extract_document_id(+ {"source_id": doc_id} if doc_id else None,metadata,- {"source_id": doc_id} if doc_id else {},)document_url = self._get_document_url(sanitised_doc_id, chunk_index=chunk_idx@@ -300,6 +307,11 @@ class CollectionSearchEngine(LibraryRAGSearchEngine):"link": document_url,"source": "library","source_type": "library",+ # Authoritative document id at the TOP level, not only+ # nested in ``metadata``: SearchResultsCollector rebuilds+ # this citation URL and would otherwise have nothing but+ # the denormalised metadata to go on.+ "source_id": doc_id,"relevance_score": float(relevance),"metadata": metadata,}diff --git a/src/local_deep_research/web_search_engines/engines/search_engine_library.py b/src/local_deep_research/web_search_engines/engines/search_engine_library.pyindex 73a59abacc..2294155a38 100644--- a/src/local_deep_research/web_search_engines/engines/search_engine_library.py+++ b/src/local_deep_research/web_search_engines/engines/search_engine_library.py@@ -14,8 +14,10 @@ from ...research_library.services.library_rag_service import LibraryRAGServicefrom ...research_library.services.library_service import LibraryServicefrom ...config.thread_settings import get_setting_from_snapshotfrom ...utilities.chunk_anchor import (+ build_chunk_anchor_url,extract_chunk_index,extract_document_id,+ is_safe_document_id,)from ...utilities.llm_utils import get_server_urlfrom ...utilities.type_utils import to_bool@@ -373,9 +375,16 @@ class LibraryRAGSearchEngine(BaseSearchEngine):# primary ``r.source_id`` path is sanitised the same way# as metadata keys.chunk_idx = extract_chunk_index(metadata)+ # Authoritative id FIRST. ``extract_document_id`` scans its+ # first mapping before any later one, so passing ``metadata``+ # first would let a chunk's denormalised metadata override+ # ``r.source_id`` — the DocumentChunk FK that decides which+ # document the chunk actually belongs to. Pre-#5381 the+ # column always won; a divergent metadata id would otherwise+ # point the citation at a different document.sanitised_doc_id = extract_document_id(+ {"source_id": doc_id} if doc_id else None,metadata,- {"source_id": doc_id} if doc_id else {},)document_url = self._get_document_url(sanitised_doc_id, chunk_index=chunk_idx@@ -388,6 +397,11 @@ class LibraryRAGSearchEngine(BaseSearchEngine):"link": document_url, # Add "link" for source extraction"source": "library","source_type": "library",+ # Authoritative document id at the TOP level, not only+ # nested in ``metadata``: SearchResultsCollector rebuilds+ # this citation URL and would otherwise have nothing but+ # the denormalised metadata to go on.+ "source_id": doc_id,"relevance_score": float(relevance),"metadata": metadata,}@@ -486,14 +500,25 @@ class LibraryRAGSearchEngine(BaseSearchEngine):"""Get the URL for viewing a document, targeting a specific chunk if provided."""if not doc_id:return "#"+ # Validate here rather than trusting the caller. Both in-tree call+ # sites pre-sanitise via ``extract_document_id``, but this method+ # BUILDS a URL path, so an unvalidated id would be interpolated+ # straight into it — a ``../..`` id yields a traversal route on all+ # three spellings below, not just the chunk one.+ if not is_safe_document_id(str(doc_id)):+ return "#"- if (- chunk_index is not None- and isinstance(chunk_index, int)- and not isinstance(chunk_index, bool)- and chunk_index >= 0- ):- return f"/library/document/{doc_id}/chunks#chunk-{chunk_index}"+ if chunk_index is not None:+ # Route through the shared builder instead of re-deriving the+ # rule. The hand-rolled test this replaces accepted any+ # non-negative int, so it had no ``MAX_CHUNK_INDEX`` bound —+ # a third, weaker copy of a contract ``chunk_anchor`` calls+ # itself the single source of truth for. On rejection fall+ # through to the plain document route rather than emitting an+ # anchor that cannot name a chunk.+ anchored = build_chunk_anchor_url("", doc_id, chunk_index)+ if anchored is not None:+ return anchoreddocument_url = f"/library/document/{doc_id}"try:diff --git a/tests/advanced_search_system/tools/test_fetch_modes.py b/tests/advanced_search_system/tools/test_fetch_modes.pyindex a8366e0c4a..2a057aace8 100644--- a/tests/advanced_search_system/tools/test_fetch_modes.py+++ b/tests/advanced_search_system/tools/test_fetch_modes.py@@ -1009,6 +1009,15 @@ def test_non_string_url_returns_none_in_try_resolve_url():assert _try_resolve_url(12345, None, None) is None+# NOTE: two tests were removed here alongside the code they covered — one+# for a ``getattr`` guard on ``find_by_url`` and one for a pre-#5381 bare-int+# ``add_results`` contract. Both exercised speculative generality in+# ``_register_in_collector``'s fallback that this branch had added and then+# reverted: no collector in any repo we can see has that shape, and the+# fallback's own docstring forbids growing it without a real consumer.+# Keeping tests for deliberately-removed code would just re-document it.++def test_register_in_collector_assigns_new_index():from local_deep_research.advanced_search_system.tools.fetch import (_register_in_collector,@@ -1039,27 +1048,86 @@ def test_register_in_collector_fast_path_reuses_without_reappend():assert len(collector.results) == before-def test_register_in_collector_slow_path_returns_deduped_index_not_start_plus_one():- """Regression for Observation A: when ``find_by_url`` misses (TOCTOU /- race) but the URL is already in the collector, the slow path must return- the index from the indexed copy — not ``start + 1``, which would be one- past the real citation after dedup.+class _ListReturningCustomCollector:+ """A custom collector returning a 2-LIST rather than a tuple.++ Unpacking is duck-typed, so any 2-item iterable must keep working —+ narrowing the fallback to ``isinstance(outcome, tuple)`` would trade the+ bare-int compatibility hole for a list/generator one."""++ def __init__(self, index_value="1"):+ self._index_value = index_value++ def add_results(self, results, engine_name="web"):+ return [0, [dict(r, index=self._index_value) for r in results]]++ def find_by_url(self, url):+ return None+++def test_register_in_collector_accepts_two_item_list_return():from local_deep_research.advanced_search_system.tools.fetch import (_register_in_collector,)- collector = SearchResultsCollector([])- collector.add_results(- [{"title": "A", "link": "http://a.com", "snippet": "a"}],- engine_name="web",+ collector = _ListReturningCustomCollector(index_value="7")+ assert _register_in_collector(collector, "http://a.com", "A", "body") == 7+++class _TupleReturningCustomCollector:+ """A custom collector on the post-#5381 tuple contract but without the+ fetch fast path — the other population the fallback must serve."""++ def __init__(self, index_value="1", url_lookup=None):+ self._index_value = index_value+ self._url_lookup = url_lookup+ self.added = []++ def add_results(self, results, engine_name="web"):+ self.added.extend(results)+ return 0, [dict(r, index=self._index_value) for r in results]++ def find_by_url(self, url):+ return self._url_lookup+++def test_register_in_collector_tuple_contract_uses_indexed_copy():+ """A custom collector on the tuple contract yields its assigned index."""+ from local_deep_research.advanced_search_system.tools.fetch import (+ _register_in_collector,)- # Simulate find_by_url racing / missing while the URL is already tracked.- with patch.object(collector, "find_by_url", return_value=None):- idx = _register_in_collector(- collector, "http://a.com", "A again", "fetched body"- )- assert idx == 1- # start would have been 1 (len(_all_links)); start+1 == 2 is the bug.- assert idx != 2+ collector = _TupleReturningCustomCollector(index_value="7")+ assert not hasattr(collector, "find_or_add_result")++ idx = _register_in_collector(collector, "http://a.com", "A", "body")++ assert idx == 7+++def test_register_in_collector_falls_back_on_bad_index():+ """A non-int index falls through to ``find_by_url`` instead of raising."""+ from local_deep_research.advanced_search_system.tools.fetch import (+ _register_in_collector,+ )++ collector = _TupleReturningCustomCollector(+ index_value="not-an-int", url_lookup=4+ )+ idx = _register_in_collector(collector, "http://a.com", "A", "body")++ assert idx == 4+++def test_register_in_collector_raises_when_unresolvable():+ """No usable index anywhere is a hard error, not a silent bad citation."""+ from local_deep_research.advanced_search_system.tools.fetch import (+ _register_in_collector,+ )++ collector = _TupleReturningCustomCollector(+ index_value=None, url_lookup=None+ )+ with pytest.raises(RuntimeError, match="Failed to register fetched URL"):+ _register_in_collector(collector, "http://a.com", "A", "body")diff --git a/tests/advanced_search_system/tools/test_library_resolver.py b/tests/advanced_search_system/tools/test_library_resolver.pyindex 1e0a14b678..b3f1306ee5 100644--- a/tests/advanced_search_system/tools/test_library_resolver.py+++ b/tests/advanced_search_system/tools/test_library_resolver.py@@ -38,6 +38,133 @@ def test_parse_library_url_pdf_suffix():)+def test_parse_library_url_chunks_suffix_with_anchor():+ """Regression for #5381: RAG citations are emitted as+ ``/library/document/<id>/chunks#chunk-<n>``. Before the ``chunks``+ suffix and fragment stripping existed, these failed to match and the+ fetch tool fell through to the egress gate, which rejects a relative+ path as ``unsupported_scheme`` — i.e. the agent could not read the+ library documents it had just cited.+ """+ from local_deep_research.advanced_search_system.tools.fetch.library_resolver import (+ parse_library_url,+ )++ assert parse_library_url("/library/document/abc-123/chunks#chunk-5") == (+ "abc-123",+ "chunks",+ )+ # Chunk 0 is a real chunk — the anchor must not be treated as falsy.+ assert parse_library_url("/library/document/abc-123/chunks#chunk-0") == (+ "abc-123",+ "chunks",+ )+ # Bare chunks route, and the ``/lib/`` abbreviation.+ assert parse_library_url("/library/document/abc-123/chunks") == (+ "abc-123",+ "chunks",+ )+ assert parse_library_url("/lib/document/abc-123/chunks#chunk-2") == (+ "abc-123",+ "chunks",+ )+++def test_parse_library_url_strips_fragment_on_root_form():+ """A fragment addresses a position inside the page, never a different+ document, so it is dropped before matching."""+ from local_deep_research.advanced_search_system.tools.fetch.library_resolver import (+ parse_library_url,+ )++ assert parse_library_url("/library/document/abc-123#chunk-2") == (+ "abc-123",+ None,+ )+ assert parse_library_url("/library/document/abc-123/pdf#page=4") == (+ "abc-123",+ "pdf",+ )+++def test_parse_library_url_absolute_alias_with_chunk_anchor():+ """The fragment is stripped BEFORE the ``://`` branch, so the+ ``https://library.document/...`` alias the agent sometimes emits also+ resolves when it carries a chunk anchor. Stripping after that branch+ left this shape unresolvable."""+ from local_deep_research.advanced_search_system.tools.fetch.library_resolver import (+ parse_library_url,+ )++ assert parse_library_url(+ "https://library.document/abc-123/chunks#chunk-3"+ ) == ("abc-123", "chunks")+++def test_parse_library_url_fragment_containing_scheme_not_misrouted():+ """A relative path whose fragment contains ``://`` must not be pushed+ into the absolute branch by the ``"://" in candidate`` test."""+ from local_deep_research.advanced_search_system.tools.fetch.library_resolver import (+ parse_library_url,+ )++ assert parse_library_url("/library/document/abc-123#http://anything") == (+ "abc-123",+ None,+ )+++def test_parse_library_url_chunk_anchor_still_rejects_bad_doc_ids():+ """Fragment stripping must not widen what counts as a document id."""+ from local_deep_research.advanced_search_system.tools.fetch.library_resolver import (+ parse_library_url,+ )++ for bad in (+ "/library/document//chunks#chunk-1",+ "/library/document/abc%00def/chunks#chunk-1",+ "/library/document/abc/chunks/extra#chunk-1",+ "/library/document/abc/txt#chunk-1",+ "#chunk-1",+ ):+ assert parse_library_url(bad) is None, bad+++def test_resolve_library_document_reads_document_for_chunk_url():+ """End-to-end on the resolver: a chunk-anchored citation resolves to the+ document's text content rather than falling through to egress."""+ from local_deep_research.advanced_search_system.tools.fetch.library_resolver import (+ resolve_library_document,+ )++ document = MagicMock()+ document.title = "My Paper"+ document.text_content = "full document body"++ session = MagicMock()+ session.query.return_value.filter_by.return_value.first.return_value = (+ document+ )+ session_cm = MagicMock()+ session_cm.__enter__.return_value = session+ session_cm.__exit__.return_value = False++ with patch(+ "local_deep_research.database.session_context.get_user_db_session",+ return_value=session_cm,+ ):+ result = resolve_library_document(+ "/library/document/abc-123/chunks#chunk-3", "alice"+ )++ assert result is not None+ assert result["title"] == "My Paper"+ assert result["content"] == "full document body"+ # The original URL (fragment included) is preserved so a downstream+ # re-citation still points at the cited chunk.+ assert result["url"] == "/library/document/abc-123/chunks#chunk-3"++def test_parse_library_url_trailing_slash():from local_deep_research.advanced_search_system.tools.fetch.library_resolver import (parse_library_url,@@ -476,3 +603,45 @@ def test_make_library_resolver_returns_none_for_no_username():resolver = make_library_resolver(None)assert resolver("/library/document/abc-123") is None+++def test_bracket_alias_is_recited_as_the_canonical_route():+ """The bare ``[<uuid>]`` alias is not a URL, so registering it verbatim+ gave the document a THIRD bibliography entry (alongside its+ ``/library/document/<id>`` and ``https://library.document/<id>``+ citations) that also rendered as a dead link. It is re-emitted as the+ RESOLVED row's canonical route — the lookup also accepts a+ document_hash and a dash-less UUID, neither of which is a live route.+ """+ from local_deep_research.advanced_search_system.tools.fetch.library_resolver import (+ resolve_library_document,+ )+ from local_deep_research.utilities.url_utils import canonical_url_key++ doc_id = "550e8400-e29b-41d4-a716-446655440000"+ document = MagicMock()+ document.id = doc_id+ document.title = "My Paper"+ document.text_content = "full document body"++ session = MagicMock()+ session.query.return_value.filter_by.return_value.first.return_value = (+ document+ )+ session_cm = MagicMock()+ session_cm.__enter__.return_value = session+ session_cm.__exit__.return_value = False++ with patch(+ "local_deep_research.database.session_context.get_user_db_session",+ return_value=session_cm,+ ):+ result = resolve_library_document(f"[{doc_id}]", "alice")++ assert result is not None+ assert result["url"] == f"/library/document/{doc_id}"+ # ... which shares a bibliography entry with the document's other+ # citation spellings.+ assert canonical_url_key(result["url"]) == canonical_url_key(+ f"/library/document/{doc_id}/chunks#chunk-3"+ )diff --git a/tests/exporters/test_ris_exporter.py b/tests/exporters/test_ris_exporter.pyindex 97c0b22cd6..937409ad6b 100644--- a/tests/exporters/test_ris_exporter.py+++ b/tests/exporters/test_ris_exporter.py@@ -4,6 +4,234 @@ import pytestfrom unittest.mock import patch+def _ris_records(ris_output: str) -> dict[str, str]:+ """Map ``refN`` -> the text of the RIS record that declares that ID.++ Records are delimited by the mandatory leading ``TY - `` tag. A+ duplicate ID would collapse here, so tests that care about duplication+ count ``ID - refN`` occurrences in the raw output instead.+ """+ records: dict[str, str] = {}+ for chunk in ris_output.split("TY - "):+ chunk = chunk.strip()+ if not chunk:+ continue+ for line in chunk.split("\n"):+ if line.startswith("ID - "):+ records[line[len("ID - ") :].strip()] = chunk+ break+ return records+++class TestRISExporterMergedSourceLines:+ """Regression tests for #5687 — merged ``[N, M]`` Sources lines.++ ``format_links_to_markdown`` collapses every citation of one source onto+ a single line carrying all of its indices (``[2, 4] Title``), which is+ routine for library documents since per-document keying. The RIS parser+ used to accept only a single-index opener, which both dropped those+ sources and — because the entry-splitting lookahead did not terminate at+ a merged opener either — let the PRECEDING entry swallow the merged line+ and inherit its URL.+ """++ @pytest.fixture+ def exporter(self):+ """The legacy exporter that owns the Sources-block parser."""+ from local_deep_research.text_optimization.citation_formatter import (+ RISExporter,+ )++ return RISExporter()++ @pytest.fixture+ def merged_after_plain(self):+ """A plain entry followed by a merged one — the mis-citation case."""+ return """# Research Report++Some text citing [1] and then [2] and [4].++## Sources++[1] First Source Title+URL: https://first.example/one++[2, 4] Merged Source Title+URL: https://merged.example/doc+"""++ def test_every_index_on_merged_line_gets_a_record(+ self, exporter, merged_after_plain+ ):+ """``[2, 4]`` must emit a record for 2 AND for 4, not neither."""+ records = _ris_records(exporter.export_to_ris(merged_after_plain))++ assert set(records) == {"ref1", "ref2", "ref4"}+ for ref in ("ref2", "ref4"):+ assert "TI - Merged Source Title" in records[ref]+ assert "UR - https://merged.example/doc" in records[ref]++ def test_preceding_entry_keeps_its_own_url(+ self, exporter, merged_after_plain+ ):+ """The entry before a merged line must not inherit the merged URL.++ This is the mis-attribution half of #5687: a record carrying one+ source's title against a different source's link.+ """+ records = _ris_records(exporter.export_to_ris(merged_after_plain))++ assert "TI - First Source Title" in records["ref1"]+ assert "UR - https://first.example/one" in records["ref1"]+ assert "merged.example" not in records["ref1"]++ def test_merged_line_first_in_block_is_not_dropped(self, exporter):+ """A merged line with no entry before it used to vanish entirely."""+ content = """# Research Report++## Sources++[1, 3] Alpha Source Title+URL: https://alpha.example/a++[2] Beta Source Title+URL: https://beta.example/b+"""++ records = _ris_records(exporter.export_to_ris(content))++ assert set(records) == {"ref1", "ref2", "ref3"}+ for ref in ("ref1", "ref3"):+ assert "TI - Alpha Source Title" in records[ref]+ assert "UR - https://alpha.example/a" in records[ref]+ assert "UR - https://beta.example/b" in records["ref2"]++ def test_lenticular_merged_opener_is_parsed(self, exporter):+ """Lenticular 【1, 3】 openers must work exactly like ASCII ones."""+ content = """# Research Report++## Sources++【1, 3】 Lenticular Merged Source+URL: https://lenticular.example/x++【2】 Lenticular Plain Source+URL: https://lenticular.example/y+"""++ records = _ris_records(exporter.export_to_ris(content))++ assert set(records) == {"ref1", "ref2", "ref3"}+ for ref in ("ref1", "ref3"):+ assert "TI - Lenticular Merged Source" in records[ref]+ assert "UR - https://lenticular.example/x" in records[ref]+ assert "UR - https://lenticular.example/y" in records["ref2"]++ def test_merged_multi_line_entry_keeps_url_and_doi(self, exporter):+ """Continuation lines stay attached to every index of a merged line."""+ content = """# Research Report++## Sources++[1] First Source Title+URL: https://first.example/one++[2, 4] Merged Journal Article+URL: https://merged.example/doc+DOI: 10.1000/merged.5687+Published in Journal of Testing+"""++ records = _ris_records(exporter.export_to_ris(content))++ assert set(records) == {"ref1", "ref2", "ref4"}+ for ref in ("ref2", "ref4"):+ assert "TI - Merged Journal Article" in records[ref]+ assert "UR - https://merged.example/doc" in records[ref]+ assert "DO - 10.1000/merged.5687" in records[ref]+ assert "UR - https://first.example/one" in records["ref1"]+ assert "10.1000/merged.5687" not in records["ref1"]++ def test_zero_padded_index_is_not_renumbered(self, exporter):+ """``007`` must stay ``ref007`` — the body cites it as written."""+ content = """# Research Report++## Sources++[007, 8] Zero Padded Source+URL: https://padded.example/z+"""++ records = _ris_records(exporter.export_to_ris(content))++ assert set(records) == {"ref007", "ref8"}+ assert "ref7" not in records++ def test_repeated_block_is_deduplicated(self, exporter):+ """A report repeats its Sources block; records must not double."""+ block = """[1] First Source Title+URL: https://first.example/one++[2, 4] Merged Source Title+URL: https://merged.example/doc+"""+ content = f"# Research Report\n\n## Sources\n\n{block}\n{block}"++ ris_output = exporter.export_to_ris(content)++ assert ris_output.count("TY - ELEC") == 3+ for ref in ("ref1", "ref2", "ref4"):+ assert ris_output.count(f"ID - {ref}\n") == 1++ def test_same_source_written_merged_and_single_dedups(self, exporter):+ """Dedup keys on the individual index, not on the group text.++ One block writes the source ``[1]``, a later one writes it+ ``[1, 3]``. Keying dedup on the raw group would treat those as+ different references and emit ``ref1`` twice.+ """+ content = """# Research Report++## Sources++[1] Shared Source Title+URL: https://shared.example/s++## Sources++[1, 3] Shared Source Title+URL: https://shared.example/s+"""++ ris_output = exporter.export_to_ris(content)++ assert ris_output.count("ID - ref1\n") == 1+ assert ris_output.count("ID - ref3\n") == 1+ assert ris_output.count("TY - ELEC") == 2++ def test_still_bounded_to_the_first_sources_section(self, exporter):+ """The ``---`` / ALL SOURCES bound survives the group-aware opener."""+ content = """# Research Report++## Sources++[1, 3] Bounded Source Title+URL: https://bounded.example/b++---++## ALL SOURCES++[9] Out Of Scope Source+URL: https://outofscope.example/o+"""++ records = _ris_records(exporter.export_to_ris(content))++ assert set(records) == {"ref1", "ref3"}+ assert "outofscope.example" not in exporter.export_to_ris(content)++class TestRISExporterProperties:"""Tests for RISExporter properties."""diff --git a/tests/strategies/test_langgraph_agent_strategy.py b/tests/strategies/test_langgraph_agent_strategy.pyindex b0b47666d8..c5705a91f5 100644--- a/tests/strategies/test_langgraph_agent_strategy.py+++ b/tests/strategies/test_langgraph_agent_strategy.py@@ -18,6 +18,14 @@ from unittest.mock import ANY, MagicMock, patchimport pytest+from local_deep_research.utilities.search_utilities import (+ format_links_to_markdown,+)+from local_deep_research.utilities.url_utils import (+ CHUNK_DISPLAY_KEY,+ canonical_url_key,+)+from local_deep_research.advanced_search_system.strategies.primary_search_metadata import (PrimarySourceClassification,PrimarySourceScope,@@ -191,12 +199,24 @@ class TestSearchResultsCollector:collector.add_results(results_chunk1)collector.add_results(results_chunk0_again)- assert len(all_links) == 2+ # Every chunk of doc1 is ONE source and gets ONE citation index:+ # the rendered bibliography collapses the document's chunk anchors+ # onto a single ``## Sources`` line, so allocating a second index+ # here would make ``sources_count`` (and MCP's ``sources`` payload,+ # and the news impact score) report two sources for a report that+ # displays one.+ assert len(all_links) == 1assert all_links[0]["link"] == "/library/document/doc1/chunks#chunk-0"assert all_links[0]["index"] == "1"- assert all_links[1]["link"] == "/library/document/doc1/chunks#chunk-1"- assert all_links[1]["index"] == "2"+ # ...but each hit still carries its OWN anchor for the agent, so a+ # per-chunk snippet is still attributable to the chunk it came from.+ assert (+ collector.results[1]["link"]+ == "/library/document/doc1/chunks#chunk-1"+ )+ assert collector.results[1]["index"] == "1"assert collector.results[2]["index"] == "1"+ assert collector.sources == ["/library/document/doc1/chunks#chunk-0"]@pytest.mark.parametrize("doc_id_structure",@@ -272,6 +292,34 @@ class TestSearchResultsCollector:assert len(all_links) == 1assert all_links[0]["index"] == "1"+ def test_zero_padded_seeded_index_survives_reuse(self):+ """A zero-padded seeded index must not be renumbered on reuse.++ The reuse branch parses the stored index with ``int()`` to return it,+ but ``str(int("007"))`` is ``"7"`` -- echoing that both renumbered the+ result and missed the ``"007"`` key in ``_index_to_result``. The RIS+ exporter deliberately preserves padded forms, so the two would have+ disagreed about which index a source carries.+ """+ all_links = [+ {"title": "Padded", "link": "http://pad.com", "index": "007"},+ ]+ collector, _ = self._make_collector(all_links)++ returned = collector.find_or_add_result(+ {"title": "Padded again", "link": "http://pad.com", "snippet": "s"},+ engine_name="test",+ )++ # Numeric identity unchanged...+ assert returned == 7+ # ...but the echoed result keeps the stored, padded form.+ echoed = [+ r for r in collector._results if r.get("link") == "http://pad.com"+ ]+ assert echoed, "reused source was not echoed into _results"+ assert echoed[-1]["index"] == "007"+def test_preseeded_all_links_seed_url_index(self):"""A pre-seeded ``_all_links`` entry seeds ``_url_to_index`` so alater ``add_results`` with the same URL reuses its citation index.@@ -324,1069 +372,1468 @@ class TestSearchResultsCollector:# all_links must NOT be clearedassert len(all_links) == 1- def test_sources_tracks_links(self):+ def test_sources_survive_dedup_across_reset(self):+ """Regression for the #5381 follow-up: ``reset()`` clears+ ``_sources`` per subsection while the dedup maps deliberately+ persist. If ``_sources`` were only appended on the new-index+ branch, a subsection that re-cites an earlier section's URLs would+ report zero sources — ``_finalize`` returns+ ``list(set(collector.sources))``, which feeds ``sources_count`` in+ the MCP server and news cards.+ """collector, _ = self._make_collector()+ collector.add_results(+ [{"title": "A", "link": "http://a.com", "snippet": "a"}]+ )+ assert collector.sources == ["http://a.com"]++ # New subsection.+ collector.reset()+ assert collector.sources == []+collector.add_results([- {"title": "A", "link": "http://a.com", "snippet": "a"},+ {"title": "A", "link": "http://a.com", "snippet": "a2"},+ # Repeated within this batch too, so the list comparison+ # below distinguishes the dedup guard from an unconditional+ # append (with one hit each, both produce the same list).+ {"title": "A", "link": "http://a.com", "snippet": "a3"},{"title": "B", "link": "http://b.com", "snippet": "b"},])- assert set(collector.sources) == {"http://a.com", "http://b.com"}-- def test_add_results_does_not_mutate_input(self):+ # The repeat hit still reuses citation [1] ...+ assert collector.find_by_url("http://a.com") == 1+ assert collector.find_by_url("http://b.com") == 2+ # ... but must not vanish from this section's sources. Compared as+ # a list so a dropped dedup guard (unconditional append) shows up as+ # a repeated entry rather than being flattened away by set().+ assert collector.sources == ["http://a.com", "http://b.com"]++ def test_find_or_add_result_sources_survive_reset(self):+ """The fetch fast path must record sources per subsection too.++ ``find_or_add_result`` returns early from both dedup paths, and+ ``_url_to_index`` survives ``reset()`` by design — so the fast path+ always hits for a URL cited in an earlier subsection. Recording only+ on the new-entry branch would make a fetch-only subsection report no+ sources at all.+ """collector, _ = self._make_collector()- original = {"title": "A", "link": "http://a.com", "snippet": "a"}- collector.add_results([original])+ assert collector.find_or_add_result({"link": "http://a.com"}) == 1+ assert collector.sources == ["http://a.com"]- # Original dict should NOT have index/source_engine added- assert "index" not in original+ collector.reset()+ assert collector.sources == []- def test_empty_results_returns_current_length(self):+ # Same URL again: dedup fast path, but this subsection must still+ # report it as one of its sources.+ assert collector.find_or_add_result({"link": "http://a.com"}) == 1+ assert collector.sources == ["http://a.com"]+ # ...and only once.+ assert collector.find_or_add_result({"link": "http://a.com"}) == 1+ assert collector.sources == ["http://a.com"]++ def test_add_results_prefers_authoritative_source_id_over_metadata(self):+ """The collector REBUILDS every library citation URL, so it must+ prefer the result's top-level ``source_id`` (the DocumentChunk FK)+ over a chunk's denormalised metadata. Consulting metadata first+ rebuilds the link pointing at a DIFFERENT document, undoing the+ RAG engines' own ordering one hop later.+ """collector, _ = self._make_collector()- collector.add_results(- [{"title": "A", "link": "http://a.com", "snippet": "a"}]+ _, indexed = collector.add_results(+ [+ {+ "title": "Doc",+ # Anchor-less on input, so the assertion below can only+ # pass if the rebuild actually ran — and ran with the+ # authoritative id.+ "link": "/library/document/DOC-REAL",+ "url": "/library/document/DOC-REAL",+ "source": "library",+ "source_id": "DOC-REAL",+ "metadata": {+ "document_id": "DOC-OTHER",+ "chunk_index": 3,+ },+ }+ ])- start, indexed = collector.add_results([])- assert start == 1- assert indexed == []- def test_thread_safety_no_duplicate_indices(self):- """Multiple threads adding results should never produce duplicate indices."""+ assert indexed[0]["link"] == (+ "/library/document/DOC-REAL/chunks#chunk-3"+ )++ def test_add_results_falls_back_to_metadata_without_source_id(self):+ """Producers that don't set a top-level ``source_id`` keep the+ previous metadata-driven behaviour."""collector, _ = self._make_collector()- errors = []+ _, indexed = collector.add_results(+ [+ {+ "title": "Doc",+ # Anchor-less on input for the same reason: asserting+ # the output equals the input would pass even with the+ # rebuild removed entirely.+ "link": "/library/document/DOC-OTHER",+ "url": "/library/document/DOC-OTHER",+ "source": "library",+ "metadata": {+ "document_id": "DOC-OTHER",+ "chunk_index": 3,+ },+ }+ ]+ )- def add_batch(thread_id):- try:- results_per_thread = [- {- "title": f"T{thread_id}-{i}",- "link": f"http://t{thread_id}-{i}.com",- "snippet": f"s{i}",- }- for i in range(5)- ]- collector.add_results(- results_per_thread,- engine_name=f"thread-{thread_id}",- )- except Exception as exc:- errors.append(exc)+ assert indexed[0]["link"] == (+ "/library/document/DOC-OTHER/chunks#chunk-3"+ )- threads = [- threading.Thread(target=add_batch, args=(i,)) for i in range(4)- ]- for t in threads:- t.start()- for t in threads:- t.join()+ def test_all_links_keeps_the_chunk_anchor_whichever_view_arrives_first(+ self,+ ):+ """Deduping on the canonical key leaves ONE entry per document in+ ``_all_links`` — the first spelling seen. The render-time anchor+ preference only fires when two entries share a key, so on that list+ it cannot fire at all, and an anchor-less view arriving first would+ drop ``#chunk-<n>`` permanently.++ This list is what the detailed report appends and what+ ``research_resources.url`` is persisted from, so the loss would+ reach the database, not just one render.+ """+ plain = {+ "title": "Doc",+ "link": "/library/document/doc1",+ "source": "library",+ }+ anchored = {+ "title": "Doc",+ "link": "/library/document/doc1/chunks#chunk-5",+ "source": "library",+ "metadata": {"chunk_index": 5, "document_id": "doc1"},+ }- assert not errors- all_results = collector.results- assert len(all_results) == 20 # 4 threads × 5 results- indices = [r["index"] for r in all_results]- assert len(indices) == len(set(indices)), "Duplicate indices found!"+ for order in ([plain, anchored], [anchored, plain]):+ all_links = []+ collector, _ = self._make_collector(all_links)+ collector.add_results([dict(entry) for entry in order])++ assert len(all_links) == 1+ entry = all_links[0]+ # Recorded alongside, never overwriting the citation URL.+ anchored = "/library/document/doc1/chunks#chunk-5"+ assert (+ entry.get(CHUNK_DISPLAY_KEY) == anchored+ or entry["link"] == anchored+ )+ assert anchored in format_links_to_markdown(all_links)++ def test_find_or_add_result_still_finds_externally_appended_entries(self):+ """The fallback scan is skipped when ``_url_to_index`` accounts for+ every entry — it canonicalises the whole list under the lock, which+ is ~170x the old string compare once the LRU thrashes. It must+ still run for entries appended outside the collector, which are the+ only ones the map can be missing.+ """+ all_links = []+ collector, _ = self._make_collector(all_links)+ # Appended AFTER construction on purpose: seeding it through the+ # constructor populates ``_url_to_index``, so the answer would come+ # from the O(1) map and the scan would never run — the first+ # version of this test passed with the scan loop deleted.+ all_links.append(+ {"title": "outside", "link": "https://outside.test/p", "index": "9"}+ )- def test_thread_safety_same_url_dedup(self):- """Concurrent ``add_results`` with the SAME URL must dedup to one index.+ assert (+ collector.find_or_add_result({"link": "https://outside.test/p"})+ == 9+ )- Regression for F3: the original thread-safety test used unique URLs- per thread, so it never exercised the dedup path under concurrency.+ def test_linkless_result_does_not_disable_the_scan_skip(self):+ """A linkless result is appended to ``_all_links`` but records no+ dedup key, so comparing map size against list size stays unequal+ for the rest of the run and the scan would never be skipped again.+ The guard counts what the collector itself appended instead."""- collector, all_links = self._make_collector()- errors: list[Exception] = []-- def add_same_url():- try:- collector.add_results(- [- {- "title": "Same",- "link": "http://same.com",- "snippet": "s",- }- ],- engine_name="test",- )- except Exception as exc: # pragma: no cover- errors.append(exc)+ collector, _ = self._make_collector()+ collector.add_results(+ [{"title": "no link"}]+ + [+ {"title": f"t{i}", "link": f"https://e{i}.test/p"}+ for i in range(5)+ ]+ )- threads = [threading.Thread(target=add_same_url) for _ in range(8)]- for t in threads:- t.start()- for t in threads:- t.join()+ canonical_url_key.cache_clear()+ collector.find_or_add_result({"link": "https://brand-new.test/p"})- assert not errors- # All 8 calls appended to _results, but only one unique URL in _all_links- assert len(collector.results) == 8- assert len(all_links) == 1- indices = [r["index"] for r in collector.results]- assert len(set(indices)) == 1- assert indices[0] == "1"- assert collector.find_by_url("http://same.com") == 1- # find_by_index O(1) path- assert collector.find_by_index(1)["link"] == "http://same.com"+ assert canonical_url_key.cache_info().misses == 1- def test_chunk_index_boolean_rejected(self):- """Boolean chunk_index (e.g. True) must not format as '#chunk-True'."""- collector, _ = self._make_collector()+ def test_anchor_upgrade_stores_the_normalized_library_route(self):+ """The upgrade applies the renderer's rule: library routes only,+ normalized — never the caller's spelling. ``_all_links`` is+ persisted to ``research_resources.url``, so echoing the raw string+ would put credentials and tracking params in the database.+ """+ for hostile in (+ "https://alice:[email protected]/p#chunk-1",+ "https://ex.test/p?utm_source=spam&gclid=xyz#chunk-2",+ "https://ex.test/p#chunk-not-a-number",+ ):+ all_links = []+ collector, _ = self._make_collector(all_links)+ collector.add_results([{"title": "T", "link": "https://ex.test/p"}])+ collector.add_results([{"title": "T", "link": hostile}])++ assert all_links[0]["link"] == "https://ex.test/p", hostile+ assert all_links[0].get(CHUNK_DISPLAY_KEY) is None, hostile++ def test_anchor_upgrade_does_not_touch_a_colliding_citation(self):+ """``_url_to_index`` and ``_index_to_result`` are filled by+ independent "if not already present" guards, so they can name+ different documents. Both consumers PREFER the recorded anchor over+ the entry's own url, so writing blind puts one document's anchor on+ another's citation — in the report and in the database.++ Driven through ``find_or_add_result`` so the fallback scan really+ registers the collision: the previous version of this test used+ ``add_results`` without chunk metadata, so the fragment was+ stripped, a fresh index was allocated, and+ ``_prefer_anchored_link`` was never called at all.+ """+ all_links = []+ collector, _ = self._make_collector(all_links)collector.add_results([{- "title": "A",- "link": "http://a.com/doc",+ "title": "alpha",+ "link": "https://alpha.test/a","source": "library",- "metadata": {"chunk_index": True, "doc_id": "doc123"},}])- assert collector.results[0]["link"] == "http://a.com/doc"+ # External append claiming the index the collector already used.+ all_links.append(+ {"title": "beta", "link": "/library/document/doc9", "index": "1"}+ )- def test_doc_id_integer_coerced(self):- """Integer doc_id (e.g. 123) is coerced to string and properly formatted."""- collector, _ = self._make_collector()+ collector.find_or_add_result(+ {+ "title": "beta",+ "link": "/library/document/doc9/chunks#chunk-3",+ }+ )++ assert all_links[0]["link"] == "https://alpha.test/a"+ assert all_links[0].get(CHUNK_DISPLAY_KEY) is None+ # beta is a legitimate separate entry and renders its own line; what+ # must not happen is beta's ANCHOR appearing anywhere, since the+ # only entry it could have been attached to is alpha's.+ assert "#chunk-3" not in format_links_to_markdown(all_links)++ def test_entry_keeps_its_own_anchor_against_a_different_chunk(self):+ """The entry's snippet came from the chunk its own link names, so a+ later view of a DIFFERENT chunk of the same document must not+ replace the displayed anchor — the reader would be sent to text the+ snippet did not come from.+ """+ all_links = []+ collector, _ = self._make_collector(all_links)collector.add_results([{- "title": "A",- "link": "http://a.com/doc",+ "title": "D",+ "link": "/library/document/doc1",+ "snippet": "text of chunk 5","source": "library",- "metadata": {"chunk_index": 2, "doc_id": 123},+ "metadata": {"chunk_index": 5, "document_id": "doc1"},}])- assert (- collector.results[0]["link"]- == "/library/document/123/chunks#chunk-2"- )-- def test_find_or_add_result_is_atomic_for_same_url(self):- """Concurrent fetch registration reuses one citation index."""- collector, all_links = self._make_collector()- barrier = threading.Barrier(8)- indices = []- errors = []-- def register():- try:- barrier.wait()- index = collector.find_or_add_result(- {- "title": "Fetched page",- "link": "https://example.com/shared",- "snippet": "shared",- },- engine_name="fetch",- )- indices.append(index)- except Exception as exc:- errors.append(exc)- threads = [threading.Thread(target=register) for _ in range(8)]- for thread in threads:- thread.start()- for thread in threads:- thread.join()+ collector.add_results(+ [+ {+ "title": "D",+ "link": "/library/document/doc1",+ "source": "library",+ "metadata": {"chunk_index": 9, "document_id": "doc1"},+ }+ ]+ )- assert not errors- assert indices == [1] * 8- assert len(collector.results) == 1- assert len(all_links) == 1- assert collector.sources == ["https://example.com/shared"]+ assert "#chunk-5" in format_links_to_markdown(all_links)+ assert "#chunk-9" not in format_links_to_markdown(all_links)- def test_find_by_url_fallback_scans_past_unindexed_entries(self):- """find_by_url fallback scan continues past unindexed matching entries."""- all_links = [- {"link": "http://a.com"}, # missing "index"- {"link": "http://a.com", "index": "2"},- ]+ def test_producer_set_anchor_key_is_not_trusted(self):+ """``add_results`` copies the engine's dict, so a result can arrive+ already carrying the key. Both consumers prefer it over the url, so+ an unvalidated read would let an engine choose the rendered and the+ persisted link.+ """+ all_links = []collector, _ = self._make_collector(all_links)- assert collector.find_by_url("http://a.com") == 2-- def test_find_by_index_returns_result_dict_when_present(self):- """``find_by_index(N)`` returns the dict stored at citation N so the- fetch tool can resolve a bare ``[N]`` marker to its source URL- (A3 follow-up)."""- collector, _ = self._make_collector()collector.add_results([- {"title": "A", "link": "http://a.com", "snippet": "a"},- {"title": "B", "link": "http://b.com", "snippet": "b"},+ {+ "title": "Doc",+ "source": "library",+ "link": "/library/document/doc1/chunks#chunk-abc",+ CHUNK_DISPLAY_KEY: "javascript:alert(document.domain)",+ }])- result = collector.find_by_index(1)- assert result is not None- assert result["title"] == "A"- assert result["link"] == "http://a.com"-- result = collector.find_by_index(2)- assert result["title"] == "B"+ rendered = format_links_to_markdown(all_links)+ assert "javascript:" not in rendered- def test_find_by_index_returns_none_when_absent(self):- collector, _ = self._make_collector()+ def test_find_or_add_result_upgrades_the_anchor_too(self):+ """The fetch path reuses an index just as ``add_results`` does, so+ the anchor loss has to be fixed on both branches."""+ all_links = []+ collector, _ = self._make_collector(all_links)collector.add_results(- [{"title": "A", "link": "http://a.com", "snippet": "a"}]+ [+ {+ "title": "D",+ "link": "/library/document/doc1",+ "source": "library",+ }+ ])- assert collector.find_by_index(9999) is None- assert collector.find_by_index(0) is None- assert collector.find_by_index(-1) is None- def test_find_by_index_uses_all_links_across_resets(self):- """Citation indices survive a ``reset()`` because they live on- ``_all_links``, not on the per-call ``_results`` list. The fetch- tool's citation resolution depends on this — ``reset()`` runs- before every subsection in detailed-report mode."""- collector, _ = self._make_collector()- collector.add_results(- [{"title": "A", "link": "http://a.com", "snippet": "a"}]+ collector.find_or_add_result(+ {"title": "D", "link": "/library/document/doc1/chunks#chunk-5"})- collector.reset()- # Citation 1 still resolvable after reset.- result = collector.find_by_index(1)- assert result is not None- assert result["link"] == "http://a.com"+ anchored = "/library/document/doc1/chunks#chunk-5"+ assert all_links[0].get(CHUNK_DISPLAY_KEY) == anchored- def test_find_or_add_result_updates_dedup_maps_and_prevents_add_results_duplicates(- self,- ):- """Registering a URL via find_or_add_result (fetch path) updates- _url_to_index and _index_to_result so a subsequent add_results call- (search path) reuses the citation index without duplicating _all_links.+ def test_seeded_collector_still_skips_the_scan(self):+ """A collector constructed with existing entries must not lose the+ skip: seeded entries are in ``_all_links`` from the start, so a+ counter of "appends we made" is permanently short and the scan runs+ forever. ``_index_to_result`` is seeded too, so comparing against+ it stays exact."""- collector, all_links = self._make_collector()- index = collector.find_or_add_result(+ seed = [{- "title": "Fetch first",- "link": "http://fetch.com",- "snippet": "f",- },- engine_name="fetch",- )- assert index == 1- assert collector._url_to_index.get("http://fetch.com") == "1"- assert collector._index_to_result.get("1")["link"] == "http://fetch.com"+ "title": f"s{i}",+ "link": f"https://s{i}.test/p",+ "index": str(i + 1),+ }+ for i in range(5)+ ]+ collector, _ = self._make_collector(seed)- # Subsequent search returns the same URL- start, indexed = collector.add_results(+ canonical_url_key.cache_clear()+ collector.find_or_add_result({"link": "https://brand-new.test/p"})++ assert canonical_url_key.cache_info().misses == 1++ def test_malformed_chunk_fragments_are_not_recorded(self):+ """``library_display_url`` validates the ROUTE and re-emits the+ fragment verbatim, so gating on it alone let ``#chunk-abc`` through+ on the fetch path. ``preferred_chunk_display`` validates both.+ """+ for bad in ("#chunk-abc", "#chunk--1", "#chunk-1'\"><img src=x>"):+ all_links = []+ collector, _ = self._make_collector(all_links)+ collector.add_results(+ [+ {+ "title": "D",+ "link": "/library/document/doc1",+ "source": "library",+ }+ ]+ )++ collector.find_or_add_result(+ {"title": "D", "link": f"/library/document/doc1/chunks{bad}"}+ )++ assert all_links[0]["link"] == "/library/document/doc1", bad+ assert all_links[0].get(CHUNK_DISPLAY_KEY) is None, bad++ def test_producer_anchor_for_another_document_is_ignored(self):+ """Shape validation is not enough. The recorded anchor is preferred+ over the entry's own url, so a WELL-FORMED anchor naming a+ different document would render and persist under this citation —+ and it arrives for free, because ``add_results`` copies the+ engine's dict. The writer refuses to record a foreign anchor; the+ reader has to refuse to read one.+ """+ all_links = []+ collector, _ = self._make_collector(all_links)+ collector.add_results([{- "title": "Search second",- "link": "http://fetch.com",- "snippet": "s",+ "title": "Trusted Looking Paper",+ "source": "library",+ "link": "/library/document/doc1",+ "metadata": {"chunk_index": 2, "document_id": "doc1"},+ CHUNK_DISPLAY_KEY: (+ "/library/document/OTHERDOC/chunks#chunk-42"+ ),}],- engine_name="web",+ engine_name="library",)- assert len(all_links) == 1- assert indexed[0]["index"] == "1"- assert collector.find_by_url("http://fetch.com") == 1- assert collector.find_by_index(1)["title"] == "Fetch first"- def test_add_results_rejects_malicious_doc_id_path_traversal(self):- """Malicious doc_id containing path traversal (e.g. ../../etc/passwd)- is rejected — and the original link is left unchanged rather than- receiving an anchor against the wrong route."""- collector, all_links = self._make_collector()+ rendered = format_links_to_markdown(all_links)+ assert "OTHERDOC" not in rendered+ assert "/library/document/doc1/chunks#chunk-2" in rendered++ def test_producer_anchor_cannot_relabel_an_external_source(self):+ """An external result must not render as a local library route."""+ all_links = []+ collector, _ = self._make_collector(all_links)collector.add_results([{- "title": "Malicious Doc",- "link": "/library/document/doc1",- "source": "library",- "metadata": {- "doc_id": "../../etc/passwd",- "chunk_index": 0,- },+ "title": "External Blog",+ "link": "https://blog.example/post",+ CHUNK_DISPLAY_KEY: (+ "/library/document/otherdoc/chunks#chunk-9"+ ),}])- # No sanitizable doc_id → link stays exactly as the producer- # supplied it, no ``#chunk-...`` fragment appended.- assert collector.results[0]["link"] == "/library/document/doc1"- def test_add_results_rejects_uuid_or_non_int_chunk_id(self):- """UUID string chunk_id is rejected and does not append #chunk-<uuid>."""- collector, _ = self._make_collector()+ rendered = format_links_to_markdown(all_links)+ assert "https://blog.example/post" in rendered+ assert "/library/document/" not in rendered++ def test_absurdly_long_chunk_index_does_not_raise(self):+ """The fragment pattern admits an unbounded digit run, and ``int()``+ raises above CPython's 4300-digit limit — from a pure formatter+ with no caller catching it, which would take down the whole Sources+ block.+ """+ all_links = []+ collector, _ = self._make_collector(all_links)collector.add_results([{- "title": "UUID chunk",- "link": "http://a.com/doc",- "source": "library",- "metadata": {- "chunk_id": "550e8400-e29b-41d4-a716-446655440000",- "doc_id": "doc123",- },+ "title": "D",+ "link": "/library/document/d1",+ CHUNK_DISPLAY_KEY: (+ "/library/document/d1/chunks#chunk-" + "1" * 5000+ ),}])- assert collector.results[0]["link"] == "http://a.com/doc"- def test_add_results_idempotent_when_chunk_already_in_link(self):- """Link already containing a *validated* #chunk- marker is rebuilt- to the same canonical URL (idempotent under valid metadata)."""- collector, _ = self._make_collector()+ assert "/library/document/d1" in format_links_to_markdown(all_links)++ def test_producer_supplied_anchor_key_is_stripped_at_ingest(self):+ """The key is written by the collector and preferred by both the+ renderer and the code that persists ``research_resources.url``, so+ an engine setting it would choose both.++ Reader-side validation alone was not enough: it closed the+ cross-document route while leaving the producer able to supply an+ anchor wherever the collector had DECLINED to build one — after a+ failed metadata validation, or an invalid chunk index. Stripping at+ ingest closes every route at once.+ """+ cases = [+ # collector builds its own anchor; producer's must not win+ {+ "source": "library",+ "title": "T",+ "link": "/library/document/doc1",+ "metadata": {"chunk_index": 2, "document_id": "doc1"},+ CHUNK_DISPLAY_KEY: "/library/document/doc1/chunks#chunk-777",+ },+ # fragment stripped as unvalidatable; key must not reinstate one+ {+ "source": "library",+ "title": "T",+ "link": "/library/document/doc1/chunks#chunk-abc",+ "metadata": {},+ CHUNK_DISPLAY_KEY: "/library/document/doc1/chunks#chunk-5",+ },+ # invalid index; key must not choose the view segment either+ {+ "source": "library",+ "title": "T",+ "link": "/library/document/doc1",+ "metadata": {"chunk_index": -3},+ CHUNK_DISPLAY_KEY: "/library/document/doc1/pdf#chunk-999999",+ },+ ]+ for case in cases:+ all_links = []+ collector, _ = self._make_collector(all_links)+ collector.add_results([dict(case)], engine_name="library")++ assert CHUNK_DISPLAY_KEY not in all_links[0], case["link"]+ rendered = format_links_to_markdown(all_links)+ assert "chunk-777" not in rendered+ assert "chunk-999999" not in rendered+ assert "chunk-5" not in rendered++ def test_find_or_add_result_also_strips_the_producer_key(self):+ collector, all_links = self._make_collector()++ collector.find_or_add_result(+ {+ "title": "T",+ "link": "/library/document/doc1",+ CHUNK_DISPLAY_KEY: "/library/document/doc1/chunks#chunk-9",+ }+ )++ assert CHUNK_DISPLAY_KEY not in all_links[0]++ def test_seeded_producer_key_is_stripped(self):+ """Seeded entries never passed the ingest strip, and+ ``_prefer_anchored_link`` uses ``setdefault`` — so a seeded+ producer key both won at the readers AND blocked the collector's+ own validated anchor from ever being recorded.+ """+ seed = [+ {+ "title": "Doc One",+ "link": "/library/document/doc1",+ "index": "1",+ CHUNK_DISPLAY_KEY: "/library/document/doc1/chunks#chunk-777",+ }+ ]+ collector, _ = self._make_collector(seed)+collector.add_results([{- "title": "Already chunked",- "link": "/library/document/doc1/chunks#chunk-0","source": "library",- "metadata": {"doc_id": "doc1", "chunk_index": 0},+ "title": "Doc One",+ "link": "/library/document/doc1/chunks",+ "metadata": {"chunk_index": 2, "document_id": "doc1"},}- ]- )- assert (- collector.results[0]["link"]- == "/library/document/doc1/chunks#chunk-0"+ ],+ engine_name="library",)- def test_add_results_strips_unvalidated_producer_chunk_fragment(self):- """A producer-supplied ``#chunk-<uuid>`` (or other malformed fragment)- with no valid metadata must be stripped — never trusted as-is."""+ rendered = format_links_to_markdown(seed)+ assert "chunk-777" not in rendered+ assert "#chunk-2" in rendered++ def test_reuse_branch_strips_the_producer_key(self):+ """``find_or_add_result`` has two dict-producing branches; the+ reuse one builds ``echoed`` and appends it to ``_results``."""collector, _ = self._make_collector()collector.add_results(- [- {- "title": "Bad producer fragment",- "link": (- "/library/document/doc1/chunks"- "#chunk-550e8400-e29b-41d4-a716-446655440000"- ),- "source": "library",- "metadata": {- "doc_id": "doc1",- "chunk_id": "550e8400-e29b-41d4-a716-446655440000",- },- }- ]+ [{"title": "D", "link": "/library/document/doc1"}])- link = collector.results[0]["link"]- assert "#chunk-" not in link- assert link == "/library/document/doc1/chunks"- def test_constructor_skips_non_dict_seeded_entries(self):- """Non-dict legacy entries (``None``, bare strings) in ``all_links``- must not crash ``__init__``."""- all_links = [- None, # legacy malformed- "raw string", # legacy malformed- {"title": "OK", "link": "http://ok.com", "index": "1"},- 42, # legacy malformed- ]- # Should not raise.- collector, _ = self._make_collector(all_links)- assert "http://ok.com" in collector._url_to_index- assert collector._url_to_index["http://ok.com"] == "1"+ collector.find_or_add_result(+ {+ "title": "D",+ "link": "/library/document/doc1",+ CHUNK_DISPLAY_KEY: "/library/document/doc1/chunks#chunk-999",+ }+ )- def test_add_results_skips_non_dict_inputs(self):- """Non-dict entries in the input list are skipped silently (they- cannot carry an index anyway)."""+ assert CHUNK_DISPLAY_KEY not in collector.results[-1]++ def test_sources_tracks_links(self):collector, _ = self._make_collector()- start, indexed = collector.add_results(+ collector.add_results([- None,- "raw string",- {"title": "OK", "link": "http://ok.com", "snippet": "ok"},- 42,+ {"title": "A", "link": "http://a.com", "snippet": "a"},+ {"title": "B", "link": "http://b.com", "snippet": "b"},])- assert start == 0- assert len(indexed) == 1- assert indexed[0]["link"] == "http://ok.com"- def test_seed_collision_uses_max_plus_one(self):- """Appending to a pre-seeded list whose highest index is far above- ``len(_all_links)`` must allocate one past the max, never collide- with a sparse seeded index."""- all_links = [- {"title": "Sparse", "link": "http://sparse.com", "index": "42"},- {"title": "Sparse 2", "link": "http://sparse2.com", "index": "99"},- ]- collector, _ = self._make_collector(all_links)- _, indexed = collector.add_results(- [{"title": "Fresh", "link": "http://fresh.com", "snippet": "f"}]- )- # The sparse seed has max index 99; the fresh entry must NOT- # collide with 3 (``len+1``) or any other seeded index.- assert indexed[0]["index"] == "100"- # The seed entries are still resolvable via their own indices.- assert collector.find_by_index(42)["link"] == "http://sparse.com"- assert collector.find_by_index(99)["link"] == "http://sparse2.com"- assert collector.find_by_index(100)["link"] == "http://fresh.com"+ assert set(collector.sources) == {"http://a.com", "http://b.com"}- def test_find_or_add_result_seed_collision_uses_max_plus_one(self):- """``find_or_add_result`` must use the same collision-free allocator- as ``add_results``."""- all_links = [- {"title": "Sparse", "link": "http://sparse.com", "index": "42"},- ]- collector, _ = self._make_collector(all_links)- index = collector.find_or_add_result(- {"title": "Fetch", "link": "http://fetch.com", "snippet": "f"},- engine_name="fetch",+ def test_add_results_does_not_mutate_input(self):+ collector, _ = self._make_collector()+ original = {"title": "A", "link": "http://a.com", "snippet": "a"}+ collector.add_results([original])++ # Original dict should NOT have index/source_engine added+ assert "index" not in original++ def test_empty_results_returns_current_length(self):+ collector, _ = self._make_collector()+ collector.add_results(+ [{"title": "A", "link": "http://a.com", "snippet": "a"}])- assert index == 43- # The seeded index 42 is still resolvable.- assert collector.find_by_index(42)["link"] == "http://sparse.com"- assert collector.find_by_index(43)["link"] == "http://fetch.com"+ start, indexed = collector.add_results([])+ assert start == 1+ assert indexed == []- def test_find_by_url_fallback_after_external_append(self):- """When a URL is appended to ``_all_links`` outside the collector- (legacy direct-append code paths), ``find_by_url`` must still- resolve it via the linear fallback."""- all_links = [- {"title": "External", "link": "http://external.com", "index": "5"},- ]- collector, _ = self._make_collector(all_links)- # Externally append WITHOUT going through add_results /- # find_or_add_result — the maps must not know about this URL.- collector._all_links.append(- {"title": "Also external", "link": "http://also.com", "index": "6"}- )- assert "http://external.com" in collector._url_to_index- assert "http://also.com" not in collector._url_to_index- # The fallback linear scan still resolves both.- assert collector.find_by_url("http://external.com") == 5- assert collector.find_by_url("http://also.com") == 6+ def test_thread_safety_no_duplicate_indices(self):+ """Multiple threads adding results should never produce duplicate indices."""+ collector, _ = self._make_collector()+ errors = []- def test_find_by_index_fallback_after_external_append(self):- """Linear fallback for ``find_by_index`` must work when an entry is- appended to ``_all_links`` outside the collector."""- all_links = [- {"title": "Seeded", "link": "http://seeded.com", "index": "5"},- ]- collector, _ = self._make_collector(all_links)- # Externally append — map should be unaware.- collector._all_links.append(- {"title": "Ghost", "link": "http://ghost.com", "index": "9"}- )- assert "9" not in collector._index_to_result- assert collector.find_by_index(5)["link"] == "http://seeded.com"- assert collector.find_by_index(9)["link"] == "http://ghost.com"+ def add_batch(thread_id):+ try:+ results_per_thread = [+ {+ "title": f"T{thread_id}-{i}",+ "link": f"http://t{thread_id}-{i}.com",+ "snippet": f"s{i}",+ }+ for i in range(5)+ ]+ collector.add_results(+ results_per_thread,+ engine_name=f"thread-{thread_id}",+ )+ except Exception as exc:+ errors.append(exc)- def test_find_by_url_continues_past_unindexed_collision(self):- """If a legacy entry with a matching link but no ``index`` precedes- a real indexed entry, ``find_by_url`` must continue scanning to- find the real index."""- all_links = [- {"title": "Ghost", "link": "http://a.com"}, # no index- {"title": "Real", "link": "http://a.com", "index": "11"},+ threads = [+ threading.Thread(target=add_batch, args=(i,)) for i in range(4)]- collector, _ = self._make_collector(all_links)- # The first matching entry has no index; the fallback must- # continue and return the real one (11), not ``None``.- assert collector.find_by_url("http://a.com") == 11+ for t in threads:+ t.start()+ for t in threads:+ t.join()- def test_add_results_negative_chunk_index_rejected(self):- """Negative chunk_index is not a valid anchor target — the link is- left unchanged."""+ assert not errors+ all_results = collector.results+ assert len(all_results) == 20 # 4 threads × 5 results+ indices = [r["index"] for r in all_results]+ assert len(indices) == len(set(indices)), "Duplicate indices found!"++ def test_thread_safety_same_url_dedup(self):+ """Concurrent ``add_results`` with the SAME URL must dedup to one index.++ Regression for F3: the original thread-safety test used unique URLs+ per thread, so it never exercised the dedup path under concurrency.+ """+ collector, all_links = self._make_collector()+ errors: list[Exception] = []++ def add_same_url():+ try:+ collector.add_results(+ [+ {+ "title": "Same",+ "link": "http://same.com",+ "snippet": "s",+ }+ ],+ engine_name="test",+ )+ except Exception as exc: # pragma: no cover+ errors.append(exc)++ threads = [threading.Thread(target=add_same_url) for _ in range(8)]+ for t in threads:+ t.start()+ for t in threads:+ t.join()++ assert not errors+ # All 8 calls appended to _results, but only one unique URL in _all_links+ assert len(collector.results) == 8+ assert len(all_links) == 1+ indices = [r["index"] for r in collector.results]+ assert len(set(indices)) == 1+ assert indices[0] == "1"+ assert collector.find_by_url("http://same.com") == 1+ # find_by_index O(1) path+ assert collector.find_by_index(1)["link"] == "http://same.com"++ def test_chunk_index_boolean_rejected(self):+ """Boolean chunk_index (e.g. True) must not format as '#chunk-True'."""collector, _ = self._make_collector()collector.add_results([{- "title": "Neg",- "link": "/library/document/doc1",+ "title": "A",+ "link": "http://a.com/doc","source": "library",- "metadata": {"doc_id": "doc1", "chunk_index": -1},+ "metadata": {"chunk_index": True, "doc_id": "doc123"},}])- assert collector.results[0]["link"] == "/library/document/doc1"+ assert collector.results[0]["link"] == "http://a.com/doc"- def test_add_results_missing_doc_id_leaves_link_unchanged(self):- """chunk_index present but no sanitisable doc_id (metadata empty,- top-level empty) leaves the link unchanged — must NOT append a- ``#chunk-...`` fragment to whatever the producer happened to set- the link to."""+ def test_doc_id_integer_coerced(self):+ """Integer doc_id (e.g. 123) is coerced to string and properly formatted."""collector, _ = self._make_collector()collector.add_results([{- "title": "No doc id",- "link": "/some/other/route",+ "title": "A",+ "link": "http://a.com/doc","source": "library",- "metadata": {"chunk_index": 5}, # no doc_id/source_id+ "metadata": {"chunk_index": 2, "doc_id": 123},}])- # Link unchanged — no fragment appended to a non-library route.- assert collector.results[0]["link"] == "/some/other/route"---class TestSearchToolMakers:- """Direct tests for _make_web_search_tool and _make_specialized_search_tool."""-- def test_make_web_search_tool_executes_and_unpacks_add_results(self):- from unittest.mock import MagicMock, patch- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- SearchResultsCollector,- _make_web_search_tool,+ assert (+ collector.results[0]["link"]+ == "/library/document/123/chunks#chunk-2")- mock_engine = MagicMock()- mock_engine.run.return_value = [- {"title": "Result 1", "link": "http://a.com", "snippet": "s1"}- ]+ def test_find_or_add_result_is_atomic_for_same_url(self):+ """Concurrent fetch registration reuses one citation index."""+ collector, all_links = self._make_collector()+ barrier = threading.Barrier(8)+ indices = []+ errors = []- collector = SearchResultsCollector()- with patch(- "local_deep_research.web_search_engines.search_engine_factory.create_search_engine",- return_value=mock_engine,- ):- tool_fn = _make_web_search_tool(- search_engine_name="duckduckgo",- model=MagicMock(),- settings_snapshot={},- collector=collector,- )- result_str = tool_fn.invoke({"query": "test query"})+ def register():+ try:+ barrier.wait()+ index = collector.find_or_add_result(+ {+ "title": "Fetched page",+ "link": "https://example.com/shared",+ "snippet": "shared",+ },+ engine_name="fetch",+ )+ indices.append(index)+ except Exception as exc:+ errors.append(exc)- assert "[1] Result 1 (http://a.com)" in result_str- assert len(collector.results) == 1- assert collector.results[0]["index"] == "1"+ threads = [threading.Thread(target=register) for _ in range(8)]+ for thread in threads:+ thread.start()+ for thread in threads:+ thread.join()- def test_make_specialized_search_tool_executes_and_unpacks_add_results(- self,- ):- from unittest.mock import MagicMock, patch- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- SearchResultsCollector,- _make_specialized_search_tool,- )+ assert not errors+ assert indices == [1] * 8+ assert len(collector.results) == 1+ assert len(all_links) == 1+ assert collector.sources == ["https://example.com/shared"]- mock_engine = MagicMock()- mock_engine.run.return_value = [- {"title": "Arxiv 1", "link": "http://arxiv.org/1", "snippet": "ax1"}+ def test_find_by_url_fallback_scans_past_unindexed_entries(self):+ """find_by_url fallback scan continues past unindexed matching entries."""+ all_links = [+ {"link": "http://a.com"}, # missing "index"+ {"link": "http://a.com", "index": "2"},]+ collector, _ = self._make_collector(all_links)+ assert collector.find_by_url("http://a.com") == 2- collector = SearchResultsCollector()- with patch(- "local_deep_research.web_search_engines.search_engine_factory.create_search_engine",- return_value=mock_engine,- ):- tool_fn = _make_specialized_search_tool(- engine_name="arxiv",- description="Arxiv search",- model=MagicMock(),- settings_snapshot={},- collector=collector,- )- result_str = tool_fn.invoke({"query": "physics"})+ def test_find_by_index_returns_result_dict_when_present(self):+ """``find_by_index(N)`` returns the dict stored at citation N so the+ fetch tool can resolve a bare ``[N]`` marker to its source URL+ (A3 follow-up)."""+ collector, _ = self._make_collector()+ collector.add_results(+ [+ {"title": "A", "link": "http://a.com", "snippet": "a"},+ {"title": "B", "link": "http://b.com", "snippet": "b"},+ ]+ )- assert "[1] Arxiv 1 (http://arxiv.org/1)" in result_str- assert len(collector.results) == 1- assert collector.results[0]["index"] == "1"- assert collector.results[0]["source_engine"] == "arxiv"+ result = collector.find_by_index(1)+ assert result is not None+ assert result["title"] == "A"+ assert result["link"] == "http://a.com"+ result = collector.find_by_index(2)+ assert result["title"] == "B"-# ----------------------------------------------------------------------------# Format results helper-# ------------------------------------------------------------------------------class TestFormatResults:- def test_format_results_basic(self):- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- _format_results,- )-- results = [- {- "title": "Test",- "link": "http://test.com",- "snippet": "A snippet",- },- ]- output = _format_results(results, start_idx=0)- assert "[1]" in output- assert "Test" in output- assert "http://test.com" in output- assert "A snippet" in output-- def test_format_results_offset(self):- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- _format_results,+ def test_find_by_index_returns_none_when_absent(self):+ collector, _ = self._make_collector()+ collector.add_results(+ [{"title": "A", "link": "http://a.com", "snippet": "a"}])+ assert collector.find_by_index(9999) is None+ assert collector.find_by_index(0) is None+ assert collector.find_by_index(-1) is None- results = [- {"title": "Test", "link": "http://test.com", "snippet": "snip"},- ]- output = _format_results(results, start_idx=5)- assert "[6]" in output-- def test_format_empty_returns_no_results(self):- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- _format_results,+ def test_find_by_index_uses_all_links_across_resets(self):+ """Citation indices survive a ``reset()`` because they live on+ ``_all_links``, not on the per-call ``_results`` list. The fetch+ tool's citation resolution depends on this — ``reset()`` runs+ before every subsection in detailed-report mode."""+ collector, _ = self._make_collector()+ collector.add_results(+ [{"title": "A", "link": "http://a.com", "snippet": "a"}])+ collector.reset()- assert _format_results([], 0) == "No results."+ # Citation 1 still resolvable after reset.+ result = collector.find_by_index(1)+ assert result is not None+ assert result["link"] == "http://a.com"- def test_format_results_honours_assigned_index_from_collector(self):- """Regression for the dedup integration: ``_format_results`` must- use the ``index`` the collector assigned, not fall through to the- ``start_idx + i + 1`` fallback. Otherwise deduped citations render- as dangling ``[N]`` markers the agent can't fetch.+ def test_find_or_add_result_updates_dedup_maps_and_prevents_add_results_duplicates(+ self,+ ):+ """Registering a URL via find_or_add_result (fetch path) updates+ _url_to_index and _index_to_result so a subsequent add_results call+ (search path) reuses the citation index without duplicating _all_links."""- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- SearchResultsCollector,- _format_results,+ collector, all_links = self._make_collector()+ index = collector.find_or_add_result(+ {+ "title": "Fetch first",+ "link": "http://fetch.com",+ "snippet": "f",+ },+ engine_name="fetch",)+ assert index == 1+ assert collector._url_to_index.get("http://fetch.com") == "1"+ assert collector._index_to_result.get("1")["link"] == "http://fetch.com"- collector = SearchResultsCollector([])- collector.add_results(- [{"title": "A", "link": "http://a.com", "snippet": "a"}],- engine_name="test",- )+ # Subsequent search returns the same URLstart, indexed = collector.add_results([{- "title": "A again",- "link": "http://a.com",- "snippet": "a2",- },- {"title": "B", "link": "http://b.com", "snippet": "b"},+ "title": "Search second",+ "link": "http://fetch.com",+ "snippet": "s",+ }],- engine_name="test",+ engine_name="web",)+ assert len(all_links) == 1+ assert indexed[0]["index"] == "1"+ assert collector.find_by_url("http://fetch.com") == 1+ assert collector.find_by_index(1)["title"] == "Fetch first"- output = _format_results(indexed, start)-- # The duplicate collapses to [1]; only the new URL is [2].- assert "[1]" in output- assert "[2]" in output- assert "[3]" not in output # start + i + 1 fallback would emit this- # The dedup entry kept the original URL so the [1] in this batch- # is a real, fetchable link (not a dangling marker).- assert "(http://a.com)" in output- assert "(http://b.com)" in output-- def test_format_results_missing_index_logs_warning(self):- """When a result dict lacks the 'index' key, _format_results falls back- to start_idx + i + 1 and logs a warning.- """- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- _format_results,+ def test_add_results_rejects_malicious_doc_id_path_traversal(self):+ """Malicious doc_id containing path traversal (e.g. ../../etc/passwd)+ is rejected — and the original link is left unchanged rather than+ receiving an anchor against the wrong route."""+ collector, all_links = self._make_collector()+ collector.add_results(+ [+ {+ "title": "Malicious Doc",+ "link": "/library/document/doc1",+ "source": "library",+ "metadata": {+ "doc_id": "../../etc/passwd",+ "chunk_index": 0,+ },+ }+ ])- from loguru import logger+ # No sanitizable doc_id → link stays exactly as the producer+ # supplied it, no ``#chunk-...`` fragment appended.+ assert collector.results[0]["link"] == "/library/document/doc1"- logs = []- handler_id = logger.add(- lambda msg: logs.append(str(msg)), level="WARNING", diagnose=False+ def test_add_results_rejects_uuid_or_non_int_chunk_id(self):+ """UUID string chunk_id is rejected and does not append #chunk-<uuid>."""+ collector, _ = self._make_collector()+ collector.add_results(+ [+ {+ "title": "UUID chunk",+ "link": "http://a.com/doc",+ "source": "library",+ "metadata": {+ "chunk_id": "550e8400-e29b-41d4-a716-446655440000",+ "doc_id": "doc123",+ },+ }+ ])- logger.enable("local_deep_research")- try:- results = [{"title": "No Index", "link": "http://no-index.com"}]- output = _format_results(results, start_idx=0)- assert "[1]" in output- assert any("result missing 'index' key" in log for log in logs)- finally:- logger.remove(handler_id)+ assert collector.results[0]["link"] == "http://a.com/doc"- def test_format_results_warning_redacts_url(self):- """The fallback warning must route the URL through- ``redact_url_for_log`` so the path / query / fragment never appear- raw."""- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- _format_results,+ def test_add_results_idempotent_when_chunk_already_in_link(self):+ """Link already containing a *validated* #chunk- marker is rebuilt+ to the same canonical URL (idempotent under valid metadata)."""+ collector, _ = self._make_collector()+ collector.add_results(+ [+ {+ "title": "Already chunked",+ "link": "/library/document/doc1/chunks#chunk-0",+ "source": "library",+ "metadata": {"doc_id": "doc1", "chunk_index": 0},+ }+ ])- from loguru import logger-- logs = []- # Use the default sink interface (``message`` is a string after- # the format string is applied) — matches the existing- # ``test_format_results_missing_index_logs_warning`` style.- handler_id = logger.add(- lambda msg: logs.append(str(msg)), level="WARNING", diagnose=False+ assert (+ collector.results[0]["link"]+ == "/library/document/doc1/chunks#chunk-0")- logger.enable("local_deep_research")- try:- # URL carries sensitive path/query that must not appear raw.- results = [++ def test_add_results_strips_unvalidated_producer_chunk_fragment(self):+ """A producer-supplied ``#chunk-<uuid>`` (or other malformed fragment)+ with no valid metadata must be stripped — never trusted as-is."""+ collector, _ = self._make_collector()+ collector.add_results(+ [{- "title": "Token Leak Risk",- "link": "https://api.example.com/secret?token=abc123",+ "title": "Bad producer fragment",+ "link": (+ "/library/document/doc1/chunks"+ "#chunk-550e8400-e29b-41d4-a716-446655440000"+ ),+ "source": "library",+ "metadata": {+ "doc_id": "doc1",+ "chunk_id": "550e8400-e29b-41d4-a716-446655440000",+ },}]- _format_results(results, start_idx=0)- warning_lines = [line for line in logs if "result missing" in line]- assert warning_lines, "expected a fallback warning"- for line in warning_lines:- assert "token=abc123" not in line, (- f"raw query leaked into warning: {line!r}"- )- assert "/secret" not in line, (- f"raw path leaked into warning: {line!r}"- )- finally:- logger.remove(handler_id)---# ----------------------------------------------------------------------------# Strategy instantiation and configuration-# ------------------------------------------------------------------------------class TestLangGraphAgentStrategy:- """Test strategy construction and configuration."""-- def _make_strategy(self, **overrides):- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- LangGraphAgentStrategy,)+ link = collector.results[0]["link"]+ assert "#chunk-" not in link+ assert link == "/library/document/doc1/chunks"- defaults = {- "model": MagicMock(),- "search": MagicMock(),- "all_links_of_system": [],- "settings_snapshot": {"search.tool": {"value": "duckduckgo"}},- }- defaults.update(overrides)- return LangGraphAgentStrategy(**defaults)-- def test_basic_instantiation(self):- strategy = self._make_strategy()- assert strategy is not None- assert hasattr(strategy, "analyze_topic")- assert hasattr(strategy, "collector")+ def test_constructor_skips_non_dict_seeded_entries(self):+ """Non-dict legacy entries (``None``, bare strings) in ``all_links``+ must not crash ``__init__``."""+ all_links = [+ None, # legacy malformed+ "raw string", # legacy malformed+ {"title": "OK", "link": "http://ok.com", "index": "1"},+ 42, # legacy malformed+ ]+ # Should not raise.+ collector, _ = self._make_collector(all_links)+ assert "http://ok.com" in collector._url_to_index+ assert collector._url_to_index["http://ok.com"] == "1"- def test_format_agent_error_scrubs_credentials(self):- """_format_agent_error is rendered to the user, so it must scrub- credentials from the exception text — while keeping the- 'Agent error: <Type>:' prefix the ErrorReportGenerator pattern map- matches on (credential-leak follow-up to #4625)."""- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- LangGraphAgentStrategy,+ def test_add_results_skips_non_dict_inputs(self):+ """Non-dict entries in the input list are skipped silently (they+ cannot carry an index anyway)."""+ collector, _ = self._make_collector()+ start, indexed = collector.add_results(+ [+ None,+ "raw string",+ {"title": "OK", "link": "http://ok.com", "snippet": "ok"},+ 42,+ ])+ assert start == 0+ assert len(indexed) == 1+ assert indexed[0]["link"] == "http://ok.com"- exc = RuntimeError(- "LLM call failed: https://api.example.com/v1?api_key=SECRETKEY123"+ def test_seed_collision_uses_max_plus_one(self):+ """Appending to a pre-seeded list whose highest index is far above+ ``len(_all_links)`` must allocate one past the max, never collide+ with a sparse seeded index."""+ all_links = [+ {"title": "Sparse", "link": "http://sparse.com", "index": "42"},+ {"title": "Sparse 2", "link": "http://sparse2.com", "index": "99"},+ ]+ collector, _ = self._make_collector(all_links)+ _, indexed = collector.add_results(+ [{"title": "Fresh", "link": "http://fresh.com", "snippet": "f"}])- out = LangGraphAgentStrategy._format_agent_error(exc)+ # The sparse seed has max index 99; the fresh entry must NOT+ # collide with 3 (``len+1``) or any other seeded index.+ assert indexed[0]["index"] == "100"+ # The seed entries are still resolvable via their own indices.+ assert collector.find_by_index(42)["link"] == "http://sparse.com"+ assert collector.find_by_index(99)["link"] == "http://sparse2.com"+ assert collector.find_by_index(100)["link"] == "http://fresh.com"- assert "SECRETKEY123" not in out # credential scrubbed- assert out.startswith("Agent error: RuntimeError:") # type prefix kept+ def test_find_or_add_result_seed_collision_uses_max_plus_one(self):+ """``find_or_add_result`` must use the same collision-free allocator+ as ``add_results``."""+ all_links = [+ {"title": "Sparse", "link": "http://sparse.com", "index": "42"},+ ]+ collector, _ = self._make_collector(all_links)+ index = collector.find_or_add_result(+ {"title": "Fetch", "link": "http://fetch.com", "snippet": "f"},+ engine_name="fetch",+ )+ assert index == 43+ # The seeded index 42 is still resolvable.+ assert collector.find_by_index(42)["link"] == "http://sparse.com"+ assert collector.find_by_index(43)["link"] == "http://fetch.com"- def test_format_agent_error_keeps_categorizable_token_past_200_chars(self):- """The larger (500) cap for tool/agent errors keeps the categorizable- signal that can sit deep in a long exception message — the 200-char- HTTP-client default would truncate it and degrade ErrorReporter- categorization to 'unknown' (#4633)."""- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- LangGraphAgentStrategy,+ def test_find_by_url_fallback_after_external_append(self):+ """When a URL is appended to ``_all_links`` outside the collector+ (legacy direct-append code paths), ``find_by_url`` must still+ resolve it via the linear fallback."""+ all_links = [+ {"title": "External", "link": "http://external.com", "index": "5"},+ ]+ collector, _ = self._make_collector(all_links)+ # Externally append WITHOUT going through add_results /+ # find_or_add_result — the maps must not know about this URL.+ collector._all_links.append(+ {"title": "Also external", "link": "http://also.com", "index": "6"})+ assert "http://external.com" in collector._url_to_index+ assert "http://also.com" not in collector._url_to_index+ # The fallback linear scan still resolves both.+ assert collector.find_by_url("http://external.com") == 5+ assert collector.find_by_url("http://also.com") == 6- exc = RuntimeError(("x" * 230) + " Connection refused")- out = LangGraphAgentStrategy._format_agent_error(exc)+ def test_find_by_index_fallback_after_external_append(self):+ """Linear fallback for ``find_by_index`` must work when an entry is+ appended to ``_all_links`` outside the collector."""+ all_links = [+ {"title": "Seeded", "link": "http://seeded.com", "index": "5"},+ ]+ collector, _ = self._make_collector(all_links)+ # Externally append — map should be unaware.+ collector._all_links.append(+ {"title": "Ghost", "link": "http://ghost.com", "index": "9"}+ )+ assert "9" not in collector._index_to_result+ assert collector.find_by_index(5)["link"] == "http://seeded.com"+ assert collector.find_by_index(9)["link"] == "http://ghost.com"- # The token sits past char 200; it survives the 500 cap (a 200 cap- # would drop it).- assert "Connection refused" in out+ def test_find_by_url_continues_past_unindexed_collision(self):+ """If a legacy entry with a matching link but no ``index`` precedes+ a real indexed entry, ``find_by_url`` must continue scanning to+ find the real index."""+ all_links = [+ {"title": "Ghost", "link": "http://a.com"}, # no index+ {"title": "Real", "link": "http://a.com", "index": "11"},+ ]+ collector, _ = self._make_collector(all_links)+ # The first matching entry has no index; the fallback must+ # continue and return the real one (11), not ``None``.+ assert collector.find_by_url("http://a.com") == 11- def test_default_params(self):- strategy = self._make_strategy()- assert strategy.max_iterations == 50- assert strategy.max_sub_iterations == 8- assert strategy.include_sub_research is True+ def test_add_results_negative_chunk_index_rejected(self):+ """Negative chunk_index is not a valid anchor target — the link is+ left unchanged."""+ collector, _ = self._make_collector()+ collector.add_results(+ [+ {+ "title": "Neg",+ "link": "/library/document/doc1",+ "source": "library",+ "metadata": {"doc_id": "doc1", "chunk_index": -1},+ }+ ]+ )+ assert collector.results[0]["link"] == "/library/document/doc1"- def test_custom_params(self):- strategy = self._make_strategy(- max_iterations=50, max_sub_iterations=3, include_sub_research=False+ def test_add_results_missing_doc_id_leaves_link_unchanged(self):+ """chunk_index present but no sanitisable doc_id (metadata empty,+ top-level empty) leaves the link unchanged — must NOT append a+ ``#chunk-...`` fragment to whatever the producer happened to set+ the link to."""+ collector, _ = self._make_collector()+ collector.add_results(+ [+ {+ "title": "No doc id",+ "link": "/some/other/route",+ "source": "library",+ "metadata": {"chunk_index": 5}, # no doc_id/source_id+ }+ ])- assert strategy.max_iterations == 50- assert strategy.max_sub_iterations == 3- assert strategy.include_sub_research is False+ # Link unchanged — no fragment appended to a non-library route.+ assert collector.results[0]["link"] == "/some/other/route"- def test_low_max_iterations_uses_default(self):- """Pipeline-style low values (e.g. search.iterations=3) should not- constrain the agent — it needs many more ReAct cycles."""- strategy = self._make_strategy(max_iterations=3)- assert strategy.max_iterations == 50 # DEFAULT_MAX_ITERATIONS- def test_super_init_called_with_kwargs(self):- """Verify base class attributes are set correctly."""- all_links = [{"existing": True}]- strategy = self._make_strategy(all_links_of_system=all_links)- assert strategy.all_links_of_system is all_links+class TestSearchToolMakers:+ """Direct tests for _make_web_search_tool and _make_specialized_search_tool."""- def test_collector_shares_all_links_reference(self):- all_links = []- strategy = self._make_strategy(all_links_of_system=all_links)- strategy.collector.add_results(- [{"title": "T", "link": "http://t.com", "snippet": "s"}]+ def test_make_web_search_tool_executes_and_unpacks_add_results(self):+ from unittest.mock import MagicMock, patch+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ SearchResultsCollector,+ _make_web_search_tool,)- assert len(all_links) == 1- def test_engine_name_from_settings(self):- strategy = self._make_strategy(- settings_snapshot={"search.tool": {"value": "brave"}}- )- assert strategy._search_engine_name == "brave"+ mock_engine = MagicMock()+ mock_engine.run.return_value = [+ {"title": "Result 1", "link": "http://a.com", "snippet": "s1"}+ ]- def test_engine_name_from_settings_string(self):- strategy = self._make_strategy(- settings_snapshot={"search.tool": "searxng"}+ collector = SearchResultsCollector()+ with patch(+ "local_deep_research.web_search_engines.search_engine_factory.create_search_engine",+ return_value=mock_engine,+ ):+ tool_fn = _make_web_search_tool(+ search_engine_name="duckduckgo",+ model=MagicMock(),+ settings_snapshot={},+ collector=collector,+ )+ result_str = tool_fn.invoke({"query": "test query"})++ assert "[1] Result 1 (http://a.com)" in result_str+ assert len(collector.results) == 1+ assert collector.results[0]["index"] == "1"++ def test_make_specialized_search_tool_executes_and_unpacks_add_results(+ self,+ ):+ from unittest.mock import MagicMock, patch+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ SearchResultsCollector,+ _make_specialized_search_tool,)- assert strategy._search_engine_name == "searxng"- def test_engine_name_fallback_to_class(self):- mock_search = MagicMock()- mock_search.__class__.__name__ = "DuckDuckGoSearchEngine"- strategy = self._make_strategy(search=mock_search, settings_snapshot={})- # Registry reverse-lookup yields the canonical id (``ddg``),- # NOT the class-derived ``duckduckgo`` the previous heuristic- # produced — that mismatch let ``search_duckduckgo`` slip into- # the agent's specialized tool list alongside ``web_search`` when- # DuckDuckGo was the configured primary.- assert strategy._search_engine_name == "ddg"+ mock_engine = MagicMock()+ mock_engine.run.return_value = [+ {"title": "Arxiv 1", "link": "http://arxiv.org/1", "snippet": "ax1"}+ ]- def test_engine_name_semantic_scholar_resolves_to_canonical(self):- """``SemanticScholarSearchEngine`` -> ``semantic_scholar`` (canonical),- not ``semanticscholar`` (class-derived). Without this lookup the- helper's primary-skip misses by one underscore and the user ends- up with both ``web_search`` and ``search_semantic_scholar`` (#5015- follow-up after the original review)."""- mock_search = MagicMock()- mock_search.__class__.__name__ = "SemanticScholarSearchEngine"- strategy = self._make_strategy(search=mock_search, settings_snapshot={})- assert strategy._search_engine_name == "semantic_scholar"-- def test_display_tool_name_web_search_uses_curated_engine_name(self):- """``web_search`` renders the configured engine through the curated- display-name map, with brand-correct casing rather than the raw- lowercase id."""- strategy = self._make_strategy(- settings_snapshot={"search.tool": {"value": "duckduckgo"}}- )- assert strategy._display_tool_name("web_search") == "DuckDuckGo"-- def test_display_tool_name_web_search_searxng(self):- strategy = self._make_strategy(- settings_snapshot={"search.tool": {"value": "searxng"}}- )- assert strategy._display_tool_name("web_search") == "the web (SearXNG)"+ collector = SearchResultsCollector()+ with patch(+ "local_deep_research.web_search_engines.search_engine_factory.create_search_engine",+ return_value=mock_engine,+ ):+ tool_fn = _make_specialized_search_tool(+ engine_name="arxiv",+ description="Arxiv search",+ model=MagicMock(),+ settings_snapshot={},+ collector=collector,+ )+ result_str = tool_fn.invoke({"query": "physics"})- def test_display_tool_name_web_search_multiword_engine(self):- """Multi-word engine ids resolve to their curated display name."""- strategy = self._make_strategy(- settings_snapshot={"search.tool": {"value": "semantic_scholar"}}- )- assert strategy._display_tool_name("web_search") == "Semantic Scholar"+ assert "[1] Arxiv 1 (http://arxiv.org/1)" in result_str+ assert len(collector.results) == 1+ assert collector.results[0]["index"] == "1"+ assert collector.results[0]["source_engine"] == "arxiv"- def test_display_tool_name_web_search_unknown_engine_titlecased(self):- """Engines absent from the curated map fall back to a cleaned,- title-cased name — never the raw lowercase id."""- strategy = self._make_strategy(- settings_snapshot={"search.tool": {"value": "tavily"}}- )- assert strategy._display_tool_name("web_search") == "Tavily"- def test_display_tool_name_specialized_tool_uses_map(self):- """Non-web_search tools keep their curated display name."""- strategy = self._make_strategy()- assert strategy._display_tool_name("search_pubmed") == "PubMed"+# ---------------------------------------------------------------------------+# Format results helper+# ---------------------------------------------------------------------------- @pytest.mark.parametrize(- "tool_name",- ("web_search", "search_collection_abc123"),- )- def test_display_tool_name_collection_uses_configured_label(- self, tool_name: str- ):- from local_deep_research.web_search_engines import search_engines_config- collection_engine = "collection_abc123"- strategy = self._make_strategy(- settings_snapshot={"search.tool": collection_engine}+class TestFormatResults:+ def test_format_results_basic(self):+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ _format_results,)- with patch.object(- search_engines_config,- "search_config",- return_value={- collection_engine: {"display_name": "Library (Collection)"}+ results = [+ {+ "title": "Test",+ "link": "http://test.com",+ "snippet": "A snippet",},- ):- display_name = strategy._display_tool_name(tool_name)+ ]+ output = _format_results(results, start_idx=0)+ assert "[1]" in output+ assert "Test" in output+ assert "http://test.com" in output+ assert "A snippet" in output- assert display_name == "Library (Collection)"+ def test_format_results_offset(self):+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ _format_results,+ )- def test_display_tool_names_collection_loads_config_once(self):- from local_deep_research.web_search_engines import search_engines_config+ results = [+ {"title": "Test", "link": "http://test.com", "snippet": "snip"},+ ]+ output = _format_results(results, start_idx=5)+ assert "[6]" in output- strategy = self._make_strategy()+ def test_format_empty_returns_no_results(self):+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ _format_results,+ )- with patch.object(- search_engines_config,- "search_config",- return_value={- "collection_abc123": {"display_name": "Library (Collection)"},- "collection_def456": {"display_name": "History (Collection)"},- },- ) as search_config:- display_names = (- strategy._display_tool_name("search_collection_abc123"),- strategy._display_tool_name("search_collection_def456"),- )+ assert _format_results([], 0) == "No results."- assert display_names == ("Library (Collection)", "History (Collection)")- search_config.assert_called_once_with(- settings_snapshot=strategy.settings_snapshot+ def test_format_results_honours_assigned_index_from_collector(self):+ """Regression for the dedup integration: ``_format_results`` must+ use the ``index`` the collector assigned, not fall through to the+ ``start_idx + i + 1`` fallback. Otherwise deduped citations render+ as dangling ``[N]`` markers the agent can't fetch.+ """+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ SearchResultsCollector,+ _format_results,)- def test_display_tool_name_collection_load_failure_uses_generic_name(self):- from local_deep_research.web_search_engines import search_engines_config+ collector = SearchResultsCollector([])+ collector.add_results(+ [{"title": "A", "link": "http://a.com", "snippet": "a"}],+ engine_name="test",+ )+ start, indexed = collector.add_results(+ [+ {+ "title": "A again",+ "link": "http://a.com",+ "snippet": "a2",+ },+ {"title": "B", "link": "http://b.com", "snippet": "b"},+ ],+ engine_name="test",+ )- strategy = self._make_strategy()+ output = _format_results(indexed, start)- with patch.object(- search_engines_config,- "search_config",- side_effect=RuntimeError("configuration unavailable"),- ) as search_config:- display_names = (- strategy._display_tool_name("search_collection_abc123"),- strategy._display_tool_name("search_collection_abc123"),- )+ # The duplicate collapses to [1]; only the new URL is [2].+ assert "[1]" in output+ assert "[2]" in output+ assert "[3]" not in output # start + i + 1 fallback would emit this+ # The dedup entry kept the original URL so the [1] in this batch+ # is a real, fetchable link (not a dangling marker).+ assert "(http://a.com)" in output+ assert "(http://b.com)" in output- assert display_names == ("Collection", "Collection")- assert all(- "abc123" not in display_name for display_name in display_names+ def test_format_results_missing_index_logs_warning(self):+ """When a result dict lacks the 'index' key, _format_results falls back+ to start_idx + i + 1 and logs a warning.+ """+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ _format_results,)- search_config.assert_called_once_with(- settings_snapshot=strategy.settings_snapshot+ from loguru import logger++ logs = []+ handler_id = logger.add(+ lambda msg: logs.append(str(msg)), level="WARNING", diagnose=False)+ logger.enable("local_deep_research")+ try:+ results = [{"title": "No Index", "link": "http://no-index.com"}]+ output = _format_results(results, start_idx=0)+ assert "[1]" in output+ assert any("result missing 'index' key" in log for log in logs)+ finally:+ logger.remove(handler_id)- def test_display_tool_name_collection_without_label_uses_generic_name(self):- from local_deep_research.web_search_engines import search_engines_config+ def test_format_results_warning_redacts_url(self):+ """The fallback warning must route the URL through+ ``redact_url_for_log`` so the path / query / fragment never appear+ raw."""+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ _format_results,+ )+ from loguru import logger- collection_engine = "collection_abc123"- strategy = self._make_strategy(- settings_snapshot={"search.tool": collection_engine}+ logs = []+ # Use the default sink interface (``message`` is a string after+ # the format string is applied) — matches the existing+ # ``test_format_results_missing_index_logs_warning`` style.+ handler_id = logger.add(+ lambda msg: logs.append(str(msg)), level="WARNING", diagnose=False)+ logger.enable("local_deep_research")+ try:+ # URL carries sensitive path/query that must not appear raw.+ results = [+ {+ "title": "Token Leak Risk",+ "link": "https://api.example.com/secret?token=abc123",+ }+ ]+ _format_results(results, start_idx=0)+ warning_lines = [line for line in logs if "result missing" in line]+ assert warning_lines, "expected a fallback warning"+ for line in warning_lines:+ assert "token=abc123" not in line, (+ f"raw query leaked into warning: {line!r}"+ )+ assert "/secret" not in line, (+ f"raw path leaked into warning: {line!r}"+ )+ finally:+ logger.remove(handler_id)- with patch.object(- search_engines_config,- "search_config",- return_value={collection_engine: {"display_name": ""}},- ):- display_name = strategy._display_tool_name(- "search_collection_abc123"- )- assert display_name == "Collection"- assert "abc123" not in display_name+# ---------------------------------------------------------------------------+# Strategy instantiation and configuration+# ---------------------------------------------------------------------------- @pytest.mark.parametrize(- "malformed_key, malformed_value",- [- pytest.param(- "collection_bad",- {},- id="missing-display-name",- ),- pytest.param(- "collection_bad",- {"display_name": 42},- id="nonstring-display-name",- ),- pytest.param(- "collection_bad",- None,- id="none-value",- ),- pytest.param(- None,- {"display_name": "Bad"},- id="nonstring-key",- ),- ],- )- def test_display_tool_name_collection_malformed_entries_degrade(- self, malformed_key, malformed_value- ):- """Malformed collection entries never crash ``_display_tool_name``:- a missing/non-string ``display_name`` is skipped outright, while an- entry that breaks parsing itself (None value, non-string key) aborts- the rest of the load — either way the malformed tool renders the- generic ``Collection``, labels cached before the failure survive- (the valid sibling is listed first), and ``search_config`` loads- exactly once (#5332 follow-up)."""- from local_deep_research.web_search_engines import search_engines_config- strategy = self._make_strategy()- config = {- "collection_abc123": {"display_name": "Library (Collection)"},- malformed_key: malformed_value,+class TestLangGraphAgentStrategy:+ """Test strategy construction and configuration."""++ def _make_strategy(self, **overrides):+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ LangGraphAgentStrategy,+ )++ defaults = {+ "model": MagicMock(),+ "search": MagicMock(),+ "all_links_of_system": [],+ "settings_snapshot": {"search.tool": {"value": "duckduckgo"}},}+ defaults.update(overrides)+ return LangGraphAgentStrategy(**defaults)- with patch.object(- search_engines_config, "search_config", return_value=config- ) as search_config:- valid = strategy._display_tool_name("search_collection_abc123")- bad = strategy._display_tool_name("search_collection_bad")+ def test_basic_instantiation(self):+ strategy = self._make_strategy()+ assert strategy is not None+ assert hasattr(strategy, "analyze_topic")+ assert hasattr(strategy, "collector")- assert valid == "Library (Collection)"- assert bad == "Collection"- search_config.assert_called_once_with(- settings_snapshot=strategy.settings_snapshot+ def test_format_agent_error_scrubs_credentials(self):+ """_format_agent_error is rendered to the user, so it must scrub+ credentials from the exception text — while keeping the+ 'Agent error: <Type>:' prefix the ErrorReportGenerator pattern map+ matches on (credential-leak follow-up to #4625)."""+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ LangGraphAgentStrategy,)- def test_display_tool_name_reuses_prefetched_config(self):- """``_build_tools`` seeds the label cache with the- ``search_config()`` result it already fetched; after that,- ``_display_tool_name`` must not trigger a second fetch (#5332- follow-up: avoid a duplicate per-user DB round-trip)."""- from local_deep_research.web_search_engines import search_engines_config+ exc = RuntimeError(+ "LLM call failed: https://api.example.com/v1?api_key=SECRETKEY123"+ )+ out = LangGraphAgentStrategy._format_agent_error(exc)++ assert "SECRETKEY123" not in out # credential scrubbed+ assert out.startswith("Agent error: RuntimeError:") # type prefix kept++ def test_format_agent_error_keeps_categorizable_token_past_200_chars(self):+ """The larger (500) cap for tool/agent errors keeps the categorizable+ signal that can sit deep in a long exception message — the 200-char+ HTTP-client default would truncate it and degrade ErrorReporter+ categorization to 'unknown' (#4633)."""+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ LangGraphAgentStrategy,+ )+ exc = RuntimeError(("x" * 230) + " Connection refused")+ out = LangGraphAgentStrategy._format_agent_error(exc)++ # The token sits past char 200; it survives the 500 cap (a 200 cap+ # would drop it).+ assert "Connection refused" in out++ def test_default_params(self):strategy = self._make_strategy()- strategy._load_collection_display_names(- {"collection_abc123": {"display_name": "Library (Collection)"}}+ assert strategy.max_iterations == 50+ assert strategy.max_sub_iterations == 8+ assert strategy.include_sub_research is True++ def test_custom_params(self):+ strategy = self._make_strategy(+ max_iterations=50, max_sub_iterations=3, include_sub_research=False)+ assert strategy.max_iterations == 50+ assert strategy.max_sub_iterations == 3+ assert strategy.include_sub_research is False- with patch.object(- search_engines_config, "search_config"- ) as search_config:- display_name = strategy._display_tool_name(- "search_collection_abc123"- )+ def test_low_max_iterations_uses_default(self):+ """Pipeline-style low values (e.g. search.iterations=3) should not+ constrain the agent — it needs many more ReAct cycles."""+ strategy = self._make_strategy(max_iterations=3)+ assert strategy.max_iterations == 50 # DEFAULT_MAX_ITERATIONS- assert display_name == "Library (Collection)"- search_config.assert_not_called()+ def test_super_init_called_with_kwargs(self):+ """Verify base class attributes are set correctly."""+ all_links = [{"existing": True}]+ strategy = self._make_strategy(all_links_of_system=all_links)+ assert strategy.all_links_of_system is all_links- def test_display_tool_name_collection_whitespace_and_padding_normalized(- self,+ def test_collector_shares_all_links_reference(self):+ all_links = []+ strategy = self._make_strategy(all_links_of_system=all_links)+ strategy.collector.add_results(+ [{"title": "T", "link": "http://t.com", "snippet": "s"}]+ )+ assert len(all_links) == 1++ def test_engine_name_from_settings(self):+ strategy = self._make_strategy(+ settings_snapshot={"search.tool": {"value": "brave"}}+ )+ assert strategy._search_engine_name == "brave"++ def test_engine_name_from_settings_string(self):+ strategy = self._make_strategy(+ settings_snapshot={"search.tool": "searxng"}+ )+ assert strategy._search_engine_name == "searxng"++ def test_engine_name_fallback_to_class(self):+ mock_search = MagicMock()+ mock_search.__class__.__name__ = "DuckDuckGoSearchEngine"+ strategy = self._make_strategy(search=mock_search, settings_snapshot={})+ # Registry reverse-lookup yields the canonical id (``ddg``),+ # NOT the class-derived ``duckduckgo`` the previous heuristic+ # produced — that mismatch let ``search_duckduckgo`` slip into+ # the agent's specialized tool list alongside ``web_search`` when+ # DuckDuckGo was the configured primary.+ assert strategy._search_engine_name == "ddg"++ def test_engine_name_semantic_scholar_resolves_to_canonical(self):+ """``SemanticScholarSearchEngine`` -> ``semantic_scholar`` (canonical),+ not ``semanticscholar`` (class-derived). Without this lookup the+ helper's primary-skip misses by one underscore and the user ends+ up with both ``web_search`` and ``search_semantic_scholar`` (#5015+ follow-up after the original review)."""+ mock_search = MagicMock()+ mock_search.__class__.__name__ = "SemanticScholarSearchEngine"+ strategy = self._make_strategy(search=mock_search, settings_snapshot={})+ assert strategy._search_engine_name == "semantic_scholar"++ def test_display_tool_name_web_search_uses_curated_engine_name(self):+ """``web_search`` renders the configured engine through the curated+ display-name map, with brand-correct casing rather than the raw+ lowercase id."""+ strategy = self._make_strategy(+ settings_snapshot={"search.tool": {"value": "duckduckgo"}}+ )+ assert strategy._display_tool_name("web_search") == "DuckDuckGo"++ def test_display_tool_name_web_search_searxng(self):+ strategy = self._make_strategy(+ settings_snapshot={"search.tool": {"value": "searxng"}}+ )+ assert strategy._display_tool_name("web_search") == "the web (SearXNG)"++ def test_display_tool_name_web_search_multiword_engine(self):+ """Multi-word engine ids resolve to their curated display name."""+ strategy = self._make_strategy(+ settings_snapshot={"search.tool": {"value": "semantic_scholar"}}+ )+ assert strategy._display_tool_name("web_search") == "Semantic Scholar"++ def test_display_tool_name_web_search_unknown_engine_titlecased(self):+ """Engines absent from the curated map fall back to a cleaned,+ title-cased name — never the raw lowercase id."""+ strategy = self._make_strategy(+ settings_snapshot={"search.tool": {"value": "tavily"}}+ )+ assert strategy._display_tool_name("web_search") == "Tavily"++ def test_display_tool_name_specialized_tool_uses_map(self):+ """Non-web_search tools keep their curated display name."""+ strategy = self._make_strategy()+ assert strategy._display_tool_name("search_pubmed") == "PubMed"++ @pytest.mark.parametrize(+ "tool_name",+ ("web_search", "search_collection_abc123"),+ )+ def test_display_tool_name_collection_uses_configured_label(+ self, tool_name: str):- """A whitespace-only ``display_name`` falls back to the generic- ``Collection``; a padded label is stripped before caching (#5332- follow-up)."""from local_deep_research.web_search_engines import search_engines_config- strategy = self._make_strategy()- config = {- "collection_blank": {"display_name": " "},- "collection_padded": {"display_name": " Library (Collection) "},- }+ collection_engine = "collection_abc123"+ strategy = self._make_strategy(+ settings_snapshot={"search.tool": collection_engine}+ )with patch.object(- search_engines_config, "search_config", return_value=config+ search_engines_config,+ "search_config",+ return_value={+ collection_engine: {"display_name": "Library (Collection)"}+ },):- blank = strategy._display_tool_name("search_collection_blank")- padded = strategy._display_tool_name("search_collection_padded")+ display_name = strategy._display_tool_name(tool_name)- assert blank == "Collection"- assert padded == "Library (Collection)"+ assert display_name == "Library (Collection)"- def test_display_tool_name_collection_non_dict_return_uses_generic_name(- self,- ):- """If ``search_config()`` returns a non-dict (e.g. ``None``), the- fallback ``Collection`` label is used instead of crashing with- ``AttributeError`` on ``.items()`` (#5332 follow-up, AI-reviewer)."""+ def test_display_tool_names_collection_loads_config_once(self):from local_deep_research.web_search_engines import search_engines_configstrategy = self._make_strategy()@@ -1394,19 +1841,190 @@ class TestLangGraphAgentStrategy:with patch.object(search_engines_config,"search_config",- return_value=None,+ return_value={+ "collection_abc123": {"display_name": "Library (Collection)"},+ "collection_def456": {"display_name": "History (Collection)"},+ },) as search_config:- display_name = strategy._display_tool_name(- "search_collection_abc123"+ display_names = (+ strategy._display_tool_name("search_collection_abc123"),+ strategy._display_tool_name("search_collection_def456"),)- assert display_name == "Collection"- assert "abc123" not in display_name+ assert display_names == ("Library (Collection)", "History (Collection)")search_config.assert_called_once_with(settings_snapshot=strategy.settings_snapshot)- def test_display_tool_name_fetch_content(self):+ def test_display_tool_name_collection_load_failure_uses_generic_name(self):+ from local_deep_research.web_search_engines import search_engines_config++ strategy = self._make_strategy()++ with patch.object(+ search_engines_config,+ "search_config",+ side_effect=RuntimeError("configuration unavailable"),+ ) as search_config:+ display_names = (+ strategy._display_tool_name("search_collection_abc123"),+ strategy._display_tool_name("search_collection_abc123"),+ )++ assert display_names == ("Collection", "Collection")+ assert all(+ "abc123" not in display_name for display_name in display_names+ )+ search_config.assert_called_once_with(+ settings_snapshot=strategy.settings_snapshot+ )++ def test_display_tool_name_collection_without_label_uses_generic_name(self):+ from local_deep_research.web_search_engines import search_engines_config++ collection_engine = "collection_abc123"+ strategy = self._make_strategy(+ settings_snapshot={"search.tool": collection_engine}+ )++ with patch.object(+ search_engines_config,+ "search_config",+ return_value={collection_engine: {"display_name": ""}},+ ):+ display_name = strategy._display_tool_name(+ "search_collection_abc123"+ )++ assert display_name == "Collection"+ assert "abc123" not in display_name++ @pytest.mark.parametrize(+ "malformed_key, malformed_value",+ [+ pytest.param(+ "collection_bad",+ {},+ id="missing-display-name",+ ),+ pytest.param(+ "collection_bad",+ {"display_name": 42},+ id="nonstring-display-name",+ ),+ pytest.param(+ "collection_bad",+ None,+ id="none-value",+ ),+ pytest.param(+ None,+ {"display_name": "Bad"},+ id="nonstring-key",+ ),+ ],+ )+ def test_display_tool_name_collection_malformed_entries_degrade(+ self, malformed_key, malformed_value+ ):+ """Malformed collection entries never crash ``_display_tool_name``:+ a missing/non-string ``display_name`` is skipped outright, while an+ entry that breaks parsing itself (None value, non-string key) aborts+ the rest of the load — either way the malformed tool renders the+ generic ``Collection``, labels cached before the failure survive+ (the valid sibling is listed first), and ``search_config`` loads+ exactly once (#5332 follow-up)."""+ from local_deep_research.web_search_engines import search_engines_config++ strategy = self._make_strategy()+ config = {+ "collection_abc123": {"display_name": "Library (Collection)"},+ malformed_key: malformed_value,+ }++ with patch.object(+ search_engines_config, "search_config", return_value=config+ ) as search_config:+ valid = strategy._display_tool_name("search_collection_abc123")+ bad = strategy._display_tool_name("search_collection_bad")++ assert valid == "Library (Collection)"+ assert bad == "Collection"+ search_config.assert_called_once_with(+ settings_snapshot=strategy.settings_snapshot+ )++ def test_display_tool_name_reuses_prefetched_config(self):+ """``_build_tools`` seeds the label cache with the+ ``search_config()`` result it already fetched; after that,+ ``_display_tool_name`` must not trigger a second fetch (#5332+ follow-up: avoid a duplicate per-user DB round-trip)."""+ from local_deep_research.web_search_engines import search_engines_config++ strategy = self._make_strategy()+ strategy._load_collection_display_names(+ {"collection_abc123": {"display_name": "Library (Collection)"}}+ )++ with patch.object(+ search_engines_config, "search_config"+ ) as search_config:+ display_name = strategy._display_tool_name(+ "search_collection_abc123"+ )++ assert display_name == "Library (Collection)"+ search_config.assert_not_called()++ def test_display_tool_name_collection_whitespace_and_padding_normalized(+ self,+ ):+ """A whitespace-only ``display_name`` falls back to the generic+ ``Collection``; a padded label is stripped before caching (#5332+ follow-up)."""+ from local_deep_research.web_search_engines import search_engines_config++ strategy = self._make_strategy()+ config = {+ "collection_blank": {"display_name": " "},+ "collection_padded": {"display_name": " Library (Collection) "},+ }++ with patch.object(+ search_engines_config, "search_config", return_value=config+ ):+ blank = strategy._display_tool_name("search_collection_blank")+ padded = strategy._display_tool_name("search_collection_padded")++ assert blank == "Collection"+ assert padded == "Library (Collection)"++ def test_display_tool_name_collection_non_dict_return_uses_generic_name(+ self,+ ):+ """If ``search_config()`` returns a non-dict (e.g. ``None``), the+ fallback ``Collection`` label is used instead of crashing with+ ``AttributeError`` on ``.items()`` (#5332 follow-up, AI-reviewer)."""+ from local_deep_research.web_search_engines import search_engines_config++ strategy = self._make_strategy()++ with patch.object(+ search_engines_config,+ "search_config",+ return_value=None,+ ) as search_config:+ display_name = strategy._display_tool_name(+ "search_collection_abc123"+ )++ assert display_name == "Collection"+ assert "abc123" not in display_name+ search_config.assert_called_once_with(+ settings_snapshot=strategy.settings_snapshot+ )++ def test_display_tool_name_fetch_content(self):"""``fetch_content`` resolves through the curated map to "the page"(regression for the ``fetch_url`` → ``fetch_content`` rename — thestrategy used to key the dict entry on the legacy ``fetch_url`` name@@ -1680,581 +2298,1190 @@ class TestPrimaryWebSearchClassification:] == expected_primary_metadata_calls-class TestPrimarySearchDescriptionText:- """Pin the literal LLM-facing description strings — UNMOCKED.+class TestPrimarySearchDescriptionText:+ """Pin the literal LLM-facing description strings — UNMOCKED.++ The routing test above patches ``format_primary_search_description``+ out, so on its own it could not catch a regression that reintroduced a+ raw engine key, collection UUID, or user-supplied config prose into the+ model-visible string. These tests call the real formatter and assert the+ exact fixed prose, then drive the real ``_build_tools`` path with an+ identifier-laden primary and prove none of it leaks.+ """++ def test_neutral_description_is_fixed_prose(self):+ from local_deep_research.advanced_search_system.strategies.primary_search_metadata import (+ NEUTRAL_PRIMARY_SEARCH_DESCRIPTION,+ )++ assert NEUTRAL_PRIMARY_SEARCH_DESCRIPTION == (+ "Search the primary source selected by the user. "+ "Source classification: unavailable. "+ "Returns search result snippets with source indices."+ )++ @pytest.mark.parametrize(+ ("source_type", "scope", "expected"),+ [+ (+ PrimarySourceType.SEARCH,+ PrimarySourceScope.PUBLIC,+ "Search the primary source selected by the user. "+ "Source type: configured search source. "+ "Source scope: public. "+ "Returns search result snippets with source indices.",+ ),+ (+ PrimarySourceType.LIBRARY,+ PrimarySourceScope.LOCAL,+ "Search the primary source selected by the user. "+ "Source type: document library. "+ "Source scope: local. "+ "Returns search result snippets with source indices.",+ ),+ (+ PrimarySourceType.COLLECTION,+ PrimarySourceScope.PUBLIC_AND_LOCAL,+ "Search the primary source selected by the user. "+ "Source type: selected document collection. "+ "Source scope: public and local. "+ "Returns search result snippets with source indices.",+ ),+ (+ PrimarySourceType.RETRIEVER,+ PrimarySourceScope.UNSPECIFIED,+ "Search the primary source selected by the user. "+ "Source type: registered retriever. "+ "Source scope: unspecified. "+ "Returns search result snippets with source indices.",+ ),+ ],+ ids=(+ "search_public",+ "library_local",+ "collection_both",+ "retriever_unspecified",+ ),+ )+ def test_format_produces_exact_fixed_prose(+ self, source_type, scope, expected+ ):+ from local_deep_research.advanced_search_system.strategies.primary_search_metadata import (+ format_primary_search_description,+ )++ description = format_primary_search_description(+ PrimarySourceClassification(source_type=source_type, scope=scope)+ )+ assert description == expected++ def test_built_description_leaks_no_engine_identifiers(self):+ """End-to-end: an identifier-laden collection primary yields ONLY+ the fixed prose — no engine key, UUID, display name, or config+ description reaches the model-visible tool description."""+ import local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy as mod+ from local_deep_research.web_search_engines import search_engines_config++ engine_key = "collection_8f3a9b2c-4e1d-4a7b-9c6f-1d2e3f4a5b6c"+ source_config = {+ "is_public": True,+ "display_name": "Internal Compliance Corpus",+ "description": "Scanned internal compliance PDFs",+ }+ strategy = mod.LangGraphAgentStrategy(+ model=MagicMock(),+ search=SimpleNamespace(),+ all_links_of_system=[],+ settings_snapshot={"search.tool": engine_key},+ )+ with (+ patch.object(+ search_engines_config,+ "search_config",+ return_value={engine_key: source_config},+ ),+ patch.object(+ mod,+ "classify_engine",+ return_value=EngineClassification(+ is_public=True, is_local=True+ ),+ ),+ patch.object(+ search_engines_config,+ "list_eligible_engine_configs",+ return_value={},+ ),+ patch.object(mod, "build_fetch_tool", return_value=None),+ ):+ tools = strategy._build_tools()++ description = next(+ tool for tool in tools if tool.name == "web_search"+ ).description+ assert description == (+ "Search the primary source selected by the user. "+ "Source type: selected document collection. "+ "Source scope: public and local. "+ "Returns search result snippets with source indices."+ )+ for identifier in (+ engine_key,+ "8f3a9b2c",+ "Internal Compliance Corpus",+ "Scanned internal compliance PDFs",+ ):+ assert identifier not in description+++# ---------------------------------------------------------------------------+# Library resolver wiring (A3)+#+# The strategy threads a library_resolver into both the lead-agent's fetch+# tool and the subagent's fetch tool so a /library/document/<uuid> URL or+# a [N] citation marker doesn't get rejected by the egress policy. Without+# the resolver, every fetch in a library-only run returns+# ``unsupported_scheme`` (the f3045c5b run produced zero usable pages).+# ---------------------------------------------------------------------------+++class TestBuildLibraryResolver:+ """``_build_library_resolver`` returns a callable for the fetch tool.++ Returns ``None`` for callers without a username (programmatic mode,+ benchmarks, news) — those callers preserve the pre-A3 behaviour so+ the policy gate still rejects library URLs as ``unsupported_scheme``.+ """++ def _make_strategy(self, **overrides):+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ LangGraphAgentStrategy,+ )++ defaults = {+ "model": MagicMock(),+ "search": MagicMock(),+ "all_links_of_system": [],+ "settings_snapshot": {"search.tool": {"value": "duckduckgo"}},+ }+ defaults.update(overrides)+ return LangGraphAgentStrategy(**defaults)++ def test_returns_resolver_when_username_present_in_snapshot(self):+ """The username injected by ``ensure_snapshot_username`` (via the+ ``_username`` snapshot key) drives the resolver build. The web+ run calls this; programmatic mode without a username does not."""+ from local_deep_research.advanced_search_system.tools.fetch import (+ build_fetch_tool,+ )++ strategy = self._make_strategy(+ settings_snapshot={+ "search.tool": {"value": "duckduckgo"},+ "_username": "alice",+ }+ )+ resolver = strategy._build_library_resolver()+ assert resolver is not None+ # Round-trip: the returned callable is wired into the fetch tool.+ tool = build_fetch_tool("full", MagicMock(), library_resolver=resolver)+ assert tool is not None++ def test_returns_none_when_snapshot_has_no_username(self):+ """No ``_username`` key (programmatic mode, benchmarks) — preserve+ the pre-A3 behaviour so the egress policy rejects library URLs+ unchanged."""+ strategy = self._make_strategy(+ settings_snapshot={"search.tool": {"value": "duckduckgo"}}+ )+ assert strategy._build_library_resolver() is None++ def test_returns_none_when_snapshot_is_empty(self):+ strategy = self._make_strategy(settings_snapshot={})+ assert strategy._build_library_resolver() is None++ def test_returns_resolver_when_username_attr_set_without_snapshot(self):+ """When settings_snapshot is empty or None, but _username is set on the strategy,+ _build_library_resolver returns a resolver bound to that username."""+ strategy = self._make_strategy(settings_snapshot={})+ strategy._username = "bob"+ resolver = strategy._build_library_resolver()+ assert resolver is not None+++#+# Regression coverage for the ``fetch_url`` → ``fetch_content`` rename:+# prior to the fix, the display-renderer branch in ``analyze_topic``+# keyed on ``raw_name == "fetch_url"``, which never matched because the+# tool the model actually invokes is ``fetch_content``. Every fetch fell+# through to the generic search-style renderer and emitted+# "🔍 Searching Fetch Content: …" instead of "📖 Reading the page: …".+# ---------------------------------------------------------------------------+++class TestToolCallProgressFormatting:+ """Pin the per-tool emoji + argument extraction in+ ``LangGraphAgentStrategy._format_tool_call_progress``.+ """++ def _make_strategy(self):+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ LangGraphAgentStrategy,+ )++ return LangGraphAgentStrategy(+ model=MagicMock(),+ search=MagicMock(),+ all_links_of_system=[],+ settings_snapshot={"search.tool": {"value": "duckduckgo"}},+ )++ def _tc(self, name, **args):+ return {"name": name, "args": args, "id": "tc_test"}++ # ---- fetch_content ------------------------------------------------------++ def test_fetch_content_renders_reading_page_with_url(self):+ """fetch_content (the actual tool name) must take the+ ``📖 Reading the page`` branch — previously keyed on the legacy+ ``fetch_url`` name and never matched."""+ strategy = self._make_strategy()+ out = strategy._format_tool_call_progress(+ self._tc("fetch_content", url="https://example.org/a"),+ "the page",+ )+ assert out == '📖 Reading the page: "https://example.org/a"'++ def test_fetch_content_missing_url_renders_empty_quotes(self):+ """Missing URL arg → empty quoted target, not a crash and not a+ fall-through to the search-style renderer."""+ strategy = self._make_strategy()+ out = strategy._format_tool_call_progress(+ self._tc("fetch_content"), "the page"+ )+ assert out == '📖 Reading the page: ""'++ def test_fetch_content_url_is_truncated_to_80_chars(self):+ strategy = self._make_strategy()+ long_url = "https://example.org/" + ("a" * 200)+ out = strategy._format_tool_call_progress(+ self._tc("fetch_content", url=long_url), "the page"+ )+ # 80 chars of URL + ellipsis marker inside the quoted target — the+ # cut must be visible, not read as a complete URL.+ quoted = out.split(chr(34))[1]+ assert quoted == long_url[:80] + "…"++ def test_short_args_get_no_ellipsis(self):+ strategy = self._make_strategy()+ out = strategy._format_tool_call_progress(+ self._tc("search_pubmed", query="short query"), "PubMed"+ )+ assert out == '🔍 Searching PubMed: "short query"'++ def test_long_query_is_truncated_with_ellipsis(self):+ strategy = self._make_strategy()+ long_query = "q" * 120+ out = strategy._format_tool_call_progress(+ self._tc("search_pubmed", query=long_query), "PubMed"+ )+ assert out == f'🔍 Searching PubMed: "{"q" * 80}…"'++ def test_subtopics_list_is_capped_per_item_not_globally(self):+ """A realistic 3-subtopic call easily exceeds 80 chars joined; every+ subtopic must stay visible (the collapsed step row ellipsizes via+ CSS and expands on click) — only an individual overlong item gets+ cut, with an ellipsis."""+ strategy = self._make_strategy()+ subtopics = [+ "history of the transformer architecture in NLP",+ "current benchmark results for long-context models",+ "z" * 100,+ ]+ out = strategy._format_tool_call_progress(+ self._tc("research_subtopic", subtopics=subtopics),+ "subtopic researcher",+ )+ assert subtopics[0] in out+ assert subtopics[1] in out+ assert "z" * 80 + "…" in out+ assert "z" * 81 not in out++ def test_legacy_fetch_url_name_now_falls_through_to_search(self):+ """The legacy ``fetch_url`` name no longer matches the curated+ fetch branch — it falls through to the search-style renderer.+ Pins that the rename is complete and one-sided."""+ strategy = self._make_strategy()+ out = strategy._format_tool_call_progress(+ self._tc("fetch_url", url="https://example.org"), "fetch_url"+ )+ # Falls through to the else branch — search-style prefix.+ assert out.startswith("🔍 Searching ")+ assert "https://example.org" in out++ # ---- research_subtopic --------------------------------------------------++ def test_research_subtopic_with_subtopics_list(self):+ strategy = self._make_strategy()+ out = strategy._format_tool_call_progress(+ self._tc("research_subtopic", subtopics=["alpha", "beta"]),+ "subtopic researcher",+ )+ assert out == '🔬 Investigating subtopic: "alpha, beta"'++ def test_research_subtopic_with_query_fallback(self):+ """Forward-compat: an older ``query`` arg is accepted."""+ strategy = self._make_strategy()+ out = strategy._format_tool_call_progress(+ self._tc("research_subtopic", query="legacy topic"),+ "subtopic researcher",+ )+ assert out == '🔬 Investigating subtopic: "legacy topic"'++ # ---- search / specialized engines --------------------------------------++ def test_search_tool_uses_query_arg(self):+ strategy = self._make_strategy()+ out = strategy._format_tool_call_progress(+ self._tc("search_pubmed", query="covid"),+ "PubMed",+ )+ assert out == '🔍 Searching PubMed: "covid"'++ def test_web_search_falls_back_to_url_when_query_missing(self):+ """Generic web_search — should pick the URL if query is absent+ (preserves the legacy fallback behaviour)."""+ strategy = self._make_strategy()+ out = strategy._format_tool_call_progress(+ self._tc("web_search", url="https://example.org"),+ "DuckDuckGo",+ )+ assert out == '🔍 Searching DuckDuckGo: "https://example.org"'++ def test_unknown_tool_uses_search_prefix(self):+ """A tool that isn't fetch_content or research_subtopic gets the+ generic search prefix (default render path)."""+ strategy = self._make_strategy()+ out = strategy._format_tool_call_progress(+ self._tc("search_arxiv", query="transformers"),+ "arXiv",+ )+ assert out == '🔍 Searching arXiv: "transformers"'+++# ---------------------------------------------------------------------------+# Observation progress events (message + expandable detail)+# ---------------------------------------------------------------------------+++class TestObservationEvent:+ """Pin ``LangGraphAgentStrategy._observation_event``: the one-line+ message stays bounded for the log panel / current-task line, while+ ``metadata["content"]`` carries the (capped) full tool output for the+ click-to-expand chat step and the agent-thinking panel."""++ def _make_strategy(self):+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ LangGraphAgentStrategy,+ )++ return LangGraphAgentStrategy(+ model=MagicMock(),+ search=MagicMock(),+ all_links_of_system=[],+ settings_snapshot={"search.tool": {"value": "searxng"}},+ )++ def _msg(self, name="web_search", content=""):+ from types import SimpleNamespace++ return SimpleNamespace(name=name, content=content)++ def test_message_is_flattened_150_char_preview(self):+ strategy = self._make_strategy()+ content = "line one\nline two " + "x" * 200+ message, _ = strategy._observation_event(self._msg(content=content))++ assert message.startswith("📄 From the web (SearXNG): ")+ preview = message.split("📄 From the web (SearXNG): ", 1)[1]+ assert len(preview) == 150+ assert "\n" not in message+ assert preview.startswith("line one line two ")++ def test_metadata_carries_full_detail_with_newlines(self):+ strategy = self._make_strategy()+ content = "\n\n".join(+ f"[{i}] Title {i} (http://a{i}.com)\nSnippet text for result {i}"+ for i in range(1, 6)+ )+ assert len(content) > 150 # long enough that the preview truncates+ message, metadata = strategy._observation_event(+ self._msg(content=content)+ )++ assert metadata["phase"] == "observation"+ assert metadata["tool"] == "web_search"+ # Detail preserves the full formatted result including newlines —+ # the expanded chat step renders it pre-wrap.+ assert metadata["content"] == content++ def test_short_output_attaches_no_detail(self):+ """Output the preview already shows verbatim must not attach a+ detail — the expanded step would just repeat the line+ ("No results." twice)."""+ strategy = self._make_strategy()+ _, metadata = strategy._observation_event(+ self._msg(content="No results.")+ )++ assert "content" not in metadata++ def test_short_multiline_output_keeps_formatted_detail(self):+ """A short output WITH newlines differs from the flattened+ preview, so the detail (preserving the formatting) must still be+ attached — length alone must not gate it."""+ strategy = self._make_strategy()+ content = "Title: Foo Bar\nURL: http://example.com\nSnippet: short"+ assert len(content) <= 150+ _, metadata = strategy._observation_event(self._msg(content=content))++ assert metadata["content"] == content++ def test_detail_attached_only_beyond_preview_length(self):+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ _OBSERVATION_PREVIEW_MAX_CHARS,+ )++ strategy = self._make_strategy()+ at_limit = "y" * _OBSERVATION_PREVIEW_MAX_CHARS+ over_limit = "y" * (_OBSERVATION_PREVIEW_MAX_CHARS + 1)+ _, meta_at = strategy._observation_event(self._msg(content=at_limit))+ _, meta_over = strategy._observation_event(+ self._msg(content=over_limit)+ )++ assert "content" not in meta_at+ assert meta_over["content"] == over_limit++ def test_detail_is_capped_with_ellipsis(self):+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ _OBSERVATION_DETAIL_MAX_CHARS,+ )++ strategy = self._make_strategy()+ content = "y" * (_OBSERVATION_DETAIL_MAX_CHARS + 500)+ _, metadata = strategy._observation_event(self._msg(content=content))++ assert len(metadata["content"]) == _OBSERVATION_DETAIL_MAX_CHARS + 2+ assert metadata["content"].endswith(" …")++ def test_detail_at_cap_is_not_marked_truncated(self):+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ _OBSERVATION_DETAIL_MAX_CHARS,+ )++ strategy = self._make_strategy()+ content = "y" * _OBSERVATION_DETAIL_MAX_CHARS+ _, metadata = strategy._observation_event(self._msg(content=content))++ assert metadata["content"] == content++ def test_fetch_content_denial_returns_none(self):+ """``_observation_event`` returns ``None`` when the tool result is a+ ``fetch_content`` denial or error string. The caller skips the+ MILESTONE in that case (the WARNING in ``policy.py:_record_denial``+ is the audit signal). Returning a tuple here would render+ ``📄 From the page: Cannot fetch …`` in the chat panel — a framing+ that reads as if the page was read.+ """+ strategy = self._make_strategy()+ denial = (+ "Cannot fetch https://example.com/page: blocked by egress "+ "policy (scope_mismatch_private_only). In this run only …"+ )+ assert (+ strategy._observation_event(+ self._msg(name="fetch_content", content=denial)+ )+ is None+ )++ error = (+ "Error fetching https://example.com/page: ConnectionError('boom')"+ )+ assert (+ strategy._observation_event(+ self._msg(name="fetch_content", content=error)+ )+ is None+ )++ def test_successful_fetch_still_emits_milestone(self):+ """A successful ``fetch_content`` observation (the tool returns a+ ``[N] Title: …\\nURL: …`` payload) must still produce a milestone —+ the suppression above is denial-only.+ """+ strategy = self._make_strategy()+ content = (+ "[1] Title: Foo\nURL: https://example.com/page\n\nSummary text"+ )+ message, metadata = strategy._observation_event(+ self._msg(name="fetch_content", content=content)+ )++ assert message.startswith("📄 From the page: ")+ assert metadata["phase"] == "observation"+ assert metadata["tool"] == "fetch_content"+ # The URL is part of the flattened preview — covered by the existing+ # test_message_is_flattened_150_char_preview contract for general+ # observations, so just assert presence here.+ assert "https://example.com/page" in message++ @pytest.mark.parametrize(+ "tool_name",+ ["web_search", "research_subtopic", "arxiv", "synthetic_tool"],+ )+ def test_non_fetch_tool_denial_prefix_is_not_suppressed(self, tool_name):+ """Suppression is gated on ``tool_name == "fetch_content"`` — a+ non-fetch tool whose result happens to start with ``Cannot fetch``+ or ``Error fetching`` (e.g. an engine returning a denial string)+ must still surface as a MILESTONE. The earlier string-prefix-only+ match would silently drop legitimate observations from other+ tools whose content happens to begin with those words.+ """+ strategy = self._make_strategy()+ content = "Cannot fetch results: upstream returned 503 after retries"+ message, metadata = strategy._observation_event(+ self._msg(name=tool_name, content=content)+ )++ assert message is not None+ assert message.startswith("📄 From ")+ assert "Cannot fetch results" in message+ assert metadata["phase"] == "observation"+ assert metadata["tool"] == tool_name+++# ---------------------------------------------------------------------------+# Step heartbeat (full tool listing)+# ---------------------------------------------------------------------------+++class TestHeartbeatMessage:+ """Pin ``LangGraphAgentStrategy._heartbeat_message``: once sources are+ gathered the heartbeat lists EVERY enabled tool by friendly name — the+ old 3-name sample with "+N more" hid most engines."""++ def _make_strategy(self, links=None):+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ LangGraphAgentStrategy,+ )++ return LangGraphAgentStrategy(+ model=MagicMock(),+ search=MagicMock(),+ all_links_of_system=links if links is not None else [],+ settings_snapshot={"search.tool": {"value": "searxng"}},+ )++ def test_zero_sources_reports_planning_with_tool_count(self):+ strategy = self._make_strategy()+ strategy._tool_names = ["web_search", "search_arxiv"]++ out = strategy._heartbeat_message(1)++ assert (+ out == "Step 1 · planning approach with 2 research tools available…"+ )++ def test_lists_all_tools_without_more_suffix(self):+ strategy = self._make_strategy(links=[{"link": "http://a.com"}] * 5)+ strategy._tool_names = [+ "web_search",+ "search_arxiv",+ "search_pubmed",+ "search_wikipedia",+ "search_github",+ "search_semantic_scholar",+ ]++ out = strategy._heartbeat_message(3)++ assert out.startswith(+ "Step 3 · 5 sources gathered · selecting next action from "+ )+ for name in (+ "the web (SearXNG)",+ "arXiv",+ "PubMed",+ "Wikipedia",+ "GitHub",+ "Semantic Scholar",+ ):+ assert name in out+ assert "more" not in out+ assert not out.endswith("…")++ def test_uses_configured_collection_label(self):+ from local_deep_research.web_search_engines import search_engines_config++ strategy = self._make_strategy(links=[{"link": "http://a.com"}])+ strategy._tool_names = ["search_collection_abc123"]++ with patch.object(+ search_engines_config,+ "search_config",+ return_value={+ "collection_abc123": {"display_name": "History (Collection)"}+ },+ ):+ out = strategy._heartbeat_message(2)++ assert "History (Collection)" in out+ assert "abc123" not in out++ def test_non_search_tools_use_list_friendly_labels(self):+ """`fetch_content` ("the page") and `research_subtopic`+ ("subtopic researcher") read wrong in a comma list — the heartbeat+ must use the list-friendly overrides."""+ strategy = self._make_strategy(links=[{"link": "http://a.com"}])+ strategy._tool_names = [+ "web_search",+ "fetch_content",+ "research_subtopic",+ ]++ out = strategy._heartbeat_message(2)++ assert "page fetching" in out+ assert "subtopic research" in out+ assert "the page" not in out+ assert "subtopic researcher" not in out++ def test_single_source_uses_singular(self):+ strategy = self._make_strategy(links=[{"link": "http://a.com"}])+ strategy._tool_names = ["web_search"]++ out = strategy._heartbeat_message(2)++ assert "1 source gathered" in out+++# ---------------------------------------------------------------------------+# Citation offset for detailed report mode+# ---------------------------------------------------------------------------+++class TestCitationOffset:+ """Test that nr_of_links is handled correctly across multiple calls."""++ def _make_strategy(self):+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ LangGraphAgentStrategy,+ )++ model = MagicMock()+ model.invoke = MagicMock(+ return_value=MagicMock(content="Synthesized answer")+ )+ return LangGraphAgentStrategy(+ model=model,+ search=MagicMock(),+ all_links_of_system=[],+ settings_snapshot={"search.tool": {"value": "mock"}},+ )++ def test_collector_reset_on_analyze_topic(self):+ """Collector should be reset at the start of each analyze_topic call."""+ strategy = self._make_strategy()++ # Pre-populate collector+ strategy.collector.add_results(+ [{"title": "Old", "link": "http://old.com", "snippet": "old"}]+ )+ assert len(strategy.collector.results) == 1++ # analyze_topic should reset the collector+ with patch(+ "local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy.LangGraphAgentStrategy._build_tools",+ return_value=[],+ ):+ result = strategy.analyze_topic("test query")- The routing test above patches ``format_primary_search_description``- out, so on its own it could not catch a regression that reintroduced a- raw engine key, collection UUID, or user-supplied config prose into the- model-visible string. These tests call the real formatter and assert the- exact fixed prose, then drive the real ``_build_tools`` path with an- identifier-laden primary and prove none of it leaks.- """+ # Collector should have been reset (even though _build_tools returned empty)+ # reset() happens before _build_tools, so the error path still resets+ assert result["error"] is not None # error because no tools+ assert len(strategy.collector.results) == 0 # verify reset happened- def test_neutral_description_is_fixed_prose(self):- from local_deep_research.advanced_search_system.strategies.primary_search_metadata import (- NEUTRAL_PRIMARY_SEARCH_DESCRIPTION,- )+ def test_all_links_accumulates_across_calls(self):+ """all_links_of_system should grow across calls, not reset."""+ strategy = self._make_strategy()+ all_links = strategy.all_links_of_system- assert NEUTRAL_PRIMARY_SEARCH_DESCRIPTION == (- "Search the primary source selected by the user. "- "Source classification: unavailable. "- "Returns search result snippets with source indices."+ strategy.collector.add_results(+ [{"title": "A", "link": "http://a.com", "snippet": "a"}])+ assert len(all_links) == 1- @pytest.mark.parametrize(- ("source_type", "scope", "expected"),- [- (- PrimarySourceType.SEARCH,- PrimarySourceScope.PUBLIC,- "Search the primary source selected by the user. "- "Source type: configured search source. "- "Source scope: public. "- "Returns search result snippets with source indices.",- ),- (- PrimarySourceType.LIBRARY,- PrimarySourceScope.LOCAL,- "Search the primary source selected by the user. "- "Source type: document library. "- "Source scope: local. "- "Returns search result snippets with source indices.",- ),- (- PrimarySourceType.COLLECTION,- PrimarySourceScope.PUBLIC_AND_LOCAL,- "Search the primary source selected by the user. "- "Source type: selected document collection. "- "Source scope: public and local. "- "Returns search result snippets with source indices.",- ),- (- PrimarySourceType.RETRIEVER,- PrimarySourceScope.UNSPECIFIED,- "Search the primary source selected by the user. "- "Source type: registered retriever. "- "Source scope: unspecified. "- "Returns search result snippets with source indices.",- ),- ],- ids=(- "search_public",- "library_local",- "collection_both",- "retriever_unspecified",- ),- )- def test_format_produces_exact_fixed_prose(- self, source_type, scope, expected- ):- from local_deep_research.advanced_search_system.strategies.primary_search_metadata import (- format_primary_search_description,- )+ strategy.collector.reset()- description = format_primary_search_description(- PrimarySourceClassification(source_type=source_type, scope=scope)+ strategy.collector.add_results(+ [{"title": "B", "link": "http://b.com", "snippet": "b"}])- assert description == expected+ assert len(all_links) == 2- def test_built_description_leaks_no_engine_identifiers(self):- """End-to-end: an identifier-laden collection primary yields ONLY- the fixed prose — no engine key, UUID, display name, or config- description reaches the model-visible tool description."""- import local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy as mod- from local_deep_research.web_search_engines import search_engines_config+ def test_citation_indices_unique_across_sections(self):+ """After reset, new results should get globally unique indices+ (not restart from 1) so detailed report citations don't collide."""+ strategy = self._make_strategy()- engine_key = "collection_8f3a9b2c-4e1d-4a7b-9c6f-1d2e3f4a5b6c"- source_config = {- "is_public": True,- "display_name": "Internal Compliance Corpus",- "description": "Scanned internal compliance PDFs",- }- strategy = mod.LangGraphAgentStrategy(- model=MagicMock(),- search=SimpleNamespace(),- all_links_of_system=[],- settings_snapshot={"search.tool": engine_key},+ # Section 1: adds 2 results → indices "1", "2"+ strategy.collector.add_results(+ [+ {"title": "A", "link": "http://a.com", "snippet": "a"},+ {"title": "B", "link": "http://b.com", "snippet": "b"},+ ])- with (- patch.object(- search_engines_config,- "search_config",- return_value={engine_key: source_config},- ),- patch.object(- mod,- "classify_engine",- return_value=EngineClassification(- is_public=True, is_local=True- ),- ),- patch.object(- search_engines_config,- "list_eligible_engine_configs",- return_value={},- ),- patch.object(mod, "build_fetch_tool", return_value=None),- ):- tools = strategy._build_tools()+ assert strategy.all_links_of_system[0]["index"] == "1"+ assert strategy.all_links_of_system[1]["index"] == "2"- description = next(- tool for tool in tools if tool.name == "web_search"- ).description- assert description == (- "Search the primary source selected by the user. "- "Source type: selected document collection. "- "Source scope: public and local. "- "Returns search result snippets with source indices."+ # Simulate new section: reset per-call state+ strategy.collector.reset()++ # Section 2: should continue from "3", not restart at "1"+ strategy.collector.add_results(+ [+ {"title": "C", "link": "http://c.com", "snippet": "c"},+ {"title": "D", "link": "http://d.com", "snippet": "d"},+ ])- for identifier in (- engine_key,- "8f3a9b2c",- "Internal Compliance Corpus",- "Scanned internal compliance PDFs",- ):- assert identifier not in description+ assert strategy.all_links_of_system[2]["index"] == "3"+ assert strategy.all_links_of_system[3]["index"] == "4"# ----------------------------------------------------------------------------# Library resolver wiring (A3)-#-# The strategy threads a library_resolver into both the lead-agent's fetch-# tool and the subagent's fetch tool so a /library/document/<uuid> URL or-# a [N] citation marker doesn't get rejected by the egress policy. Without-# the resolver, every fetch in a library-only run returns-# ``unsupported_scheme`` (the f3045c5b run produced zero usable pages).+# Error handling# ----------------------------------------------------------------------------class TestBuildLibraryResolver:- """``_build_library_resolver`` returns a callable for the fetch tool.-- Returns ``None`` for callers without a username (programmatic mode,- benchmarks, news) — those callers preserve the pre-A3 behaviour so- the policy gate still rejects library URLs as ``unsupported_scheme``.- """+class TestErrorHandling:+ """Test error paths return proper error dicts."""- def _make_strategy(self, **overrides):+ def _make_strategy(self):from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (LangGraphAgentStrategy,)- defaults = {- "model": MagicMock(),- "search": MagicMock(),- "all_links_of_system": [],- "settings_snapshot": {"search.tool": {"value": "duckduckgo"}},- }- defaults.update(overrides)- return LangGraphAgentStrategy(**defaults)-- def test_returns_resolver_when_username_present_in_snapshot(self):- """The username injected by ``ensure_snapshot_username`` (via the- ``_username`` snapshot key) drives the resolver build. The web- run calls this; programmatic mode without a username does not."""- from local_deep_research.advanced_search_system.tools.fetch import (- build_fetch_tool,+ return LangGraphAgentStrategy(+ model=MagicMock(),+ search=MagicMock(),+ all_links_of_system=[],+ settings_snapshot={"search.tool": {"value": "mock"}},)- strategy = self._make_strategy(- settings_snapshot={- "search.tool": {"value": "duckduckgo"},- "_username": "alice",- }- )- resolver = strategy._build_library_resolver()- assert resolver is not None- # Round-trip: the returned callable is wired into the fetch tool.- tool = build_fetch_tool("full", MagicMock(), library_resolver=resolver)- assert tool is not None+ def test_error_result_structure(self):+ strategy = self._make_strategy()+ result = strategy._error_result("something broke")- def test_returns_none_when_snapshot_has_no_username(self):- """No ``_username`` key (programmatic mode, benchmarks) — preserve- the pre-A3 behaviour so the egress policy rejects library URLs- unchanged."""- strategy = self._make_strategy(- settings_snapshot={"search.tool": {"value": "duckduckgo"}}+ assert result["error"] == "something broke"+ assert result["findings"] == []+ assert result["iterations"] == 0+ assert result["current_knowledge"] == ""+ assert isinstance(result["reasoning_trace"], list)++ def test_no_tools_returns_error(self):+ strategy = self._make_strategy()+ with patch(+ "local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy.LangGraphAgentStrategy._build_tools",+ return_value=[],+ ):+ result = strategy.analyze_topic("test")++ assert result["error"] is not None+ assert "No tools" in result["error"]++ def test_agent_creation_failure_returns_error(self):+ strategy = self._make_strategy()+ with (+ patch(+ "local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy.LangGraphAgentStrategy._build_tools",+ return_value=[MagicMock()],+ ),+ patch(+ "langchain.agents.create_agent",+ side_effect=ValueError("Model doesn't support tools"),+ ),+ ):+ result = strategy.analyze_topic("test")++ assert result["error"] is not None+ assert "tool calling" in result["error"]++ def test_format_agent_error_includes_exception_type(self):+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ LangGraphAgentStrategy,)- assert strategy._build_library_resolver() is None- def test_returns_none_when_snapshot_is_empty(self):- strategy = self._make_strategy(settings_snapshot={})- assert strategy._build_library_resolver() is None+ msg = LangGraphAgentStrategy._format_agent_error(ValueError("boom"))- def test_returns_resolver_when_username_attr_set_without_snapshot(self):- """When settings_snapshot is empty or None, but _username is set on the strategy,- _build_library_resolver returns a resolver bound to that username."""- strategy = self._make_strategy(settings_snapshot={})- strategy._username = "bob"- resolver = strategy._build_library_resolver()- assert resolver is not None+ assert "ValueError" in msg+ assert "boom" in msg-#-# Regression coverage for the ``fetch_url`` → ``fetch_content`` rename:-# prior to the fix, the display-renderer branch in ``analyze_topic``-# keyed on ``raw_name == "fetch_url"``, which never matched because the-# tool the model actually invokes is ``fetch_content``. Every fetch fell-# through to the generic search-style renderer and emitted-# "🔍 Searching Fetch Content: …" instead of "📖 Reading the page: …".+# ---------------------------------------------------------------------------+# Factory integration# ----------------------------------------------------------------------------class TestToolCallProgressFormatting:- """Pin the per-tool emoji + argument extraction in- ``LangGraphAgentStrategy._format_tool_call_progress``.- """+class TestFactoryIntegration:+ """Test that the strategy integrates with the factory correctly."""- def _make_strategy(self):+ def test_factory_creates_langgraph_agent(self):from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (LangGraphAgentStrategy,)+ from local_deep_research.search_system_factory import create_strategy- return LangGraphAgentStrategy(+ strategy = create_strategy(+ strategy_name="langgraph-agent",model=MagicMock(),search=MagicMock(),- all_links_of_system=[],- settings_snapshot={"search.tool": {"value": "duckduckgo"}},- )-- def _tc(self, name, **args):- return {"name": name, "args": args, "id": "tc_test"}-- # ---- fetch_content -------------------------------------------------------- def test_fetch_content_renders_reading_page_with_url(self):- """fetch_content (the actual tool name) must take the- ``📖 Reading the page`` branch — previously keyed on the legacy- ``fetch_url`` name and never matched."""- strategy = self._make_strategy()- out = strategy._format_tool_call_progress(- self._tc("fetch_content", url="https://example.org/a"),- "the page",+ settings_snapshot={},)- assert out == '📖 Reading the page: "https://example.org/a"'+ assert isinstance(strategy, LangGraphAgentStrategy)- def test_fetch_content_missing_url_renders_empty_quotes(self):- """Missing URL arg → empty quoted target, not a crash and not a- fall-through to the search-style renderer."""- strategy = self._make_strategy()- out = strategy._format_tool_call_progress(- self._tc("fetch_content"), "the page"+ def test_factory_underscore_alias(self):+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ LangGraphAgentStrategy,)- assert out == '📖 Reading the page: ""'+ from local_deep_research.search_system_factory import create_strategy- def test_fetch_content_url_is_truncated_to_80_chars(self):- strategy = self._make_strategy()- long_url = "https://example.org/" + ("a" * 200)- out = strategy._format_tool_call_progress(- self._tc("fetch_content", url=long_url), "the page"+ strategy = create_strategy(+ strategy_name="langgraph_agent",+ model=MagicMock(),+ search=MagicMock(),+ settings_snapshot={},)- # 80 chars of URL + ellipsis marker inside the quoted target — the- # cut must be visible, not read as a complete URL.- quoted = out.split(chr(34))[1]- assert quoted == long_url[:80] + "…"+ assert isinstance(strategy, LangGraphAgentStrategy)- def test_short_args_get_no_ellipsis(self):- strategy = self._make_strategy()- out = strategy._format_tool_call_progress(- self._tc("search_pubmed", query="short query"), "PubMed"+ def test_strategy_in_available_list(self):+ from local_deep_research.search_system_factory import (+ get_available_strategies,)- assert out == '🔍 Searching PubMed: "short query"'- def test_long_query_is_truncated_with_ellipsis(self):- strategy = self._make_strategy()- long_query = "q" * 120- out = strategy._format_tool_call_progress(- self._tc("search_pubmed", query=long_query), "PubMed"- )- assert out == f'🔍 Searching PubMed: "{"q" * 80}…"'+ names = [s["name"] for s in get_available_strategies()]+ assert "langgraph-agent" in names- def test_subtopics_list_is_capped_per_item_not_globally(self):- """A realistic 3-subtopic call easily exceeds 80 chars joined; every- subtopic must stay visible (the collapsed step row ellipsizes via- CSS and expands on click) — only an individual overlong item gets- cut, with an ellipsis."""- strategy = self._make_strategy()- subtopics = [- "history of the transformer architecture in NLP",- "current benchmark results for long-context models",- "z" * 100,- ]- out = strategy._format_tool_call_progress(- self._tc("research_subtopic", subtopics=subtopics),- "subtopic researcher",- )- assert subtopics[0] in out- assert subtopics[1] in out- assert "z" * 80 + "…" in out- assert "z" * 81 not in out+ def test_factory_passes_custom_params(self):+ from local_deep_research.search_system_factory import create_strategy- def test_legacy_fetch_url_name_now_falls_through_to_search(self):- """The legacy ``fetch_url`` name no longer matches the curated- fetch branch — it falls through to the search-style renderer.- Pins that the rename is complete and one-sided."""- strategy = self._make_strategy()- out = strategy._format_tool_call_progress(- self._tc("fetch_url", url="https://example.org"), "fetch_url"+ strategy = create_strategy(+ strategy_name="langgraph-agent",+ model=MagicMock(),+ search=MagicMock(),+ settings_snapshot={},+ max_iterations=20,+ max_sub_iterations=3,+ include_sub_research=False,)- # Falls through to the else branch — search-style prefix.- assert out.startswith("🔍 Searching ")- assert "https://example.org" in out+ assert strategy.max_iterations == 20+ assert strategy.max_sub_iterations == 3+ assert strategy.include_sub_research is False- # ---- research_subtopic --------------------------------------------------- def test_research_subtopic_with_subtopics_list(self):- strategy = self._make_strategy()- out = strategy._format_tool_call_progress(- self._tc("research_subtopic", subtopics=["alpha", "beta"]),- "subtopic researcher",- )- assert out == '🔬 Investigating subtopic: "alpha, beta"'+# ---------------------------------------------------------------------------+# fetch_content collector registration (regression for PR #3457)+# ---------------------------------------------------------------------------- def test_research_subtopic_with_query_fallback(self):- """Forward-compat: an older ``query`` arg is accepted."""- strategy = self._make_strategy()- out = strategy._format_tool_call_progress(- self._tc("research_subtopic", query="legacy topic"),- "subtopic researcher",- )- assert out == '🔬 Investigating subtopic: "legacy topic"'- # ---- search / specialized engines --------------------------------------+class TestFetchContentCollectorRegistration:+ """Regression coverage for PR #3457.- def test_search_tool_uses_query_arg(self):- strategy = self._make_strategy()- out = strategy._format_tool_call_progress(- self._tc("search_pubmed", query="covid"),- "PubMed",- )- assert out == '🔍 Searching PubMed: "covid"'+ Prior to the fix, ``_make_fetch_content_tool`` accepted ``collector`` but+ never used it, so every URL opened via the LLM's ``fetch_content`` tool+ was silently dropped from the final Sources section and citation system.+ These tests pin the fix: a successful fetch must register the URL, a+ duplicate fetch must reuse the existing citation index, and a failed+ fetch must not register anything.+ """- def test_web_search_falls_back_to_url_when_query_missing(self):- """Generic web_search — should pick the URL if query is absent- (preserves the legacy fallback behaviour)."""- strategy = self._make_strategy()- out = strategy._format_tool_call_progress(- self._tc("web_search", url="https://example.org"),- "DuckDuckGo",+ def _make_collector(self):+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ SearchResultsCollector,)- assert out == '🔍 Searching DuckDuckGo: "https://example.org"'- def test_unknown_tool_uses_search_prefix(self):- """A tool that isn't fetch_content or research_subtopic gets the- generic search prefix (default render path)."""- strategy = self._make_strategy()- out = strategy._format_tool_call_progress(- self._tc("search_arxiv", query="transformers"),- "arXiv",- )- assert out == '🔍 Searching arXiv: "transformers"'+ return SearchResultsCollector([])+ def _fetcher_cm(+ self, *, status="success", title="Page", content="Body", error=None+ ):+ """Return a MagicMock that behaves like ``ContentFetcher(...)``."""+ result = {"status": status, "title": title, "content": content}+ if error is not None:+ result["error"] = error+ fetcher = MagicMock()+ fetcher.fetch.return_value = result+ cm = MagicMock()+ cm.__enter__.return_value = fetcher+ cm.__exit__.return_value = False+ return cm-# ----------------------------------------------------------------------------# Observation progress events (message + expandable detail)-# ---------------------------------------------------------------------------+ def _make_tool(self, collector):+ from local_deep_research.advanced_search_system.tools.fetch import (+ build_fetch_tool,+ )+ return build_fetch_tool("full", collector)-class TestObservationEvent:- """Pin ``LangGraphAgentStrategy._observation_event``: the one-line- message stays bounded for the log panel / current-task line, while- ``metadata["content"]`` carries the (capped) full tool output for the- click-to-expand chat step and the agent-thinking panel."""+ def test_successful_fetch_registers_url_in_collector(self):+ collector = self._make_collector()+ tool = self._make_tool(collector)+ cm = self._fetcher_cm(title="Hello", content="some body text")- def _make_strategy(self):- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- LangGraphAgentStrategy,- )+ with patch(+ "local_deep_research.content_fetcher.ContentFetcher",+ return_value=cm,+ ):+ output = tool.invoke({"url": "http://example.com/page"})- return LangGraphAgentStrategy(- model=MagicMock(),- search=MagicMock(),- all_links_of_system=[],- settings_snapshot={"search.tool": {"value": "searxng"}},- )+ assert "http://example.com/page" in collector.sources+ assert len(collector.results) == 1+ entry = collector.results[0]+ assert entry["link"] == "http://example.com/page"+ assert entry["title"] == "Hello"+ assert entry["source_engine"] == "fetch"+ # Tool return is prefixed with the 1-based citation index so the+ # agent can cite fetched pages the same way it cites web_search hits.+ assert output.startswith("[1] ")- def _msg(self, name="web_search", content=""):- from types import SimpleNamespace+ def test_repeated_fetch_of_same_url_reuses_citation_index(self):+ collector = self._make_collector()+ # Simulate web_search having already captured this URL.+ collector.add_results(+ [+ {+ "title": "From search",+ "link": "http://example.com/page",+ "snippet": "snip",+ }+ ],+ engine_name="web",+ )+ assert len(collector.results) == 1- return SimpleNamespace(name=name, content=content)+ tool = self._make_tool(collector)+ cm = self._fetcher_cm(title="From fetch", content="full body")- def test_message_is_flattened_150_char_preview(self):- strategy = self._make_strategy()- content = "line one\nline two " + "x" * 200- message, _ = strategy._observation_event(self._msg(content=content))+ with patch(+ "local_deep_research.content_fetcher.ContentFetcher",+ return_value=cm,+ ):+ output = tool.invoke({"url": "http://example.com/page"})- assert message.startswith("📄 From the web (SearXNG): ")- preview = message.split("📄 From the web (SearXNG): ", 1)[1]- assert len(preview) == 150- assert "\n" not in message- assert preview.startswith("line one line two ")+ # No duplicate entry; the fetch reuses the existing citation slot.+ assert len(collector.results) == 1+ assert output.startswith("[1] ")- def test_metadata_carries_full_detail_with_newlines(self):- strategy = self._make_strategy()- content = "\n\n".join(- f"[{i}] Title {i} (http://a{i}.com)\nSnippet text for result {i}"- for i in range(1, 6)- )- assert len(content) > 150 # long enough that the preview truncates- message, metadata = strategy._observation_event(- self._msg(content=content)+ def test_failed_fetch_does_not_register_url(self):+ collector = self._make_collector()+ tool = self._make_tool(collector)+ cm = self._fetcher_cm(+ status="error", title="", content="", error="timeout")- assert metadata["phase"] == "observation"- assert metadata["tool"] == "web_search"- # Detail preserves the full formatted result including newlines —- # the expanded chat step renders it pre-wrap.- assert metadata["content"] == content+ with patch(+ "local_deep_research.content_fetcher.ContentFetcher",+ return_value=cm,+ ):+ output = tool.invoke({"url": "http://broken.example/page"})- def test_short_output_attaches_no_detail(self):- """Output the preview already shows verbatim must not attach a- detail — the expanded step would just repeat the line- ("No results." twice)."""- strategy = self._make_strategy()- _, metadata = strategy._observation_event(- self._msg(content="No results.")- )+ assert collector.results == []+ assert collector.sources == []+ assert "Failed to fetch" in output- assert "content" not in metadata+ def test_long_content_snippet_is_truncated_with_ellipsis(self):+ collector = self._make_collector()+ tool = self._make_tool(collector)+ cm = self._fetcher_cm(title="Long", content="A" * 500)- def test_short_multiline_output_keeps_formatted_detail(self):- """A short output WITH newlines differs from the flattened- preview, so the detail (preserving the formatting) must still be- attached — length alone must not gate it."""- strategy = self._make_strategy()- content = "Title: Foo Bar\nURL: http://example.com\nSnippet: short"- assert len(content) <= 150- _, metadata = strategy._observation_event(self._msg(content=content))+ with patch(+ "local_deep_research.content_fetcher.ContentFetcher",+ return_value=cm,+ ):+ tool.invoke({"url": "http://example.com/long"})- assert metadata["content"] == content+ snippet = collector.results[0]["snippet"]+ assert snippet.endswith("...")+ assert len(snippet) == 203 # 200 chars + "..."- def test_detail_attached_only_beyond_preview_length(self):- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- _OBSERVATION_PREVIEW_MAX_CHARS,+ def test_find_by_url_returns_index_when_present(self):+ collector = self._make_collector()+ collector.add_results(+ [{"title": "A", "link": "http://a.com", "snippet": "a"}],+ engine_name="web",)+ assert collector.find_by_url("http://a.com") == 1- strategy = self._make_strategy()- at_limit = "y" * _OBSERVATION_PREVIEW_MAX_CHARS- over_limit = "y" * (_OBSERVATION_PREVIEW_MAX_CHARS + 1)- _, meta_at = strategy._observation_event(self._msg(content=at_limit))- _, meta_over = strategy._observation_event(- self._msg(content=over_limit)+ def test_find_by_url_returns_none_when_absent(self):+ collector = self._make_collector()+ collector.add_results(+ [{"title": "A", "link": "http://a.com", "snippet": "a"}],+ engine_name="web",)+ assert collector.find_by_url("http://missing.com") is None- assert "content" not in meta_at- assert meta_over["content"] == over_limit- def test_detail_is_capped_with_ellipsis(self):- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- _OBSERVATION_DETAIL_MAX_CHARS,- )+class TestFetchModeSettingResolution:+ """``LangGraphAgentStrategy.__init__`` reads the ``search.fetch.mode``+ setting (added in #3680; default changed to ``summary_focus_query``+ in #3793) and feeds it to ``build_fetch_tool``. The constructor must:- strategy = self._make_strategy()- content = "y" * (_OBSERVATION_DETAIL_MAX_CHARS + 500)- _, metadata = strategy._observation_event(self._msg(content=content))+ - Accept any value in ``FETCH_MODES`` verbatim.+ - Reject any other value, log a warning, and fall back to+ ``summary_focus_query`` rather than crashing or letting an unknown+ mode reach ``build_fetch_tool``.- assert len(metadata["content"]) == _OBSERVATION_DETAIL_MAX_CHARS + 2- assert metadata["content"].endswith(" …")+ The existing tests covered the constructor and tool-building paths+ but not this guard.+ """- def test_detail_at_cap_is_not_marked_truncated(self):+ def _make_strategy(self, **overrides):from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- _OBSERVATION_DETAIL_MAX_CHARS,+ LangGraphAgentStrategy,)- strategy = self._make_strategy()- content = "y" * _OBSERVATION_DETAIL_MAX_CHARS- _, metadata = strategy._observation_event(self._msg(content=content))-- assert metadata["content"] == content+ defaults = {+ "model": MagicMock(),+ "search": MagicMock(),+ "all_links_of_system": [],+ "settings_snapshot": {"search.tool": "duckduckgo"},+ }+ defaults.update(overrides)+ return LangGraphAgentStrategy(**defaults)- def test_fetch_content_denial_returns_none(self):- """``_observation_event`` returns ``None`` when the tool result is a- ``fetch_content`` denial or error string. The caller skips the- MILESTONE in that case (the WARNING in ``policy.py:_record_denial``- is the audit signal). Returning a tuple here would render- ``📄 From the page: Cannot fetch …`` in the chat panel — a framing- that reads as if the page was read.+ def test_known_fetch_mode_accepted_verbatim(self):+ """``summary_focus`` (one of the ``FETCH_MODES``) must round-trip+ through the constructor unchanged."""- strategy = self._make_strategy()- denial = (- "Cannot fetch https://example.com/page: blocked by egress "- "policy (scope_mismatch_private_only). In this run only …"+ strategy = self._make_strategy(+ settings_snapshot={+ "search.tool": "duckduckgo",+ "search.fetch.mode": "summary_focus",+ })- assert (- strategy._observation_event(- self._msg(name="fetch_content", content=denial)+ assert strategy.fetch_mode == "summary_focus"++ def test_unknown_fetch_mode_falls_back_to_default_with_warning(+ self, loguru_caplog+ ):+ """A misconfigured setting must not crash the constructor or+ propagate an unknown mode into ``build_fetch_tool``. The guard+ at the top of ``__init__`` logs a warning and substitutes the+ default. Anyone removing the guard would surface as the mode+ leaking through unchanged AND the warning going missing.+ """+ with loguru_caplog.at_level("WARNING"):+ strategy = self._make_strategy(+ settings_snapshot={+ "search.tool": "duckduckgo",+ "search.fetch.mode": "definitely-not-a-real-mode",+ })- is None- )- error = (- "Error fetching https://example.com/page: ConnectionError('boom')"+ assert strategy.fetch_mode == "summary_focus_query"+ assert "Unknown search.fetch.mode" in loguru_caplog.text+ assert "definitely-not-a-real-mode" in loguru_caplog.text++ def test_disabled_fetch_mode_omits_fetch_tool(self):+ """``fetch_mode='disabled'`` must produce a tool list with NO+ fetch tool — ``build_fetch_tool`` returns ``None`` and the+ ``if fetch is not None`` guard skips the append. A regression+ that always-appended would surface here as an extra tool.+ """+ strategy = self._make_strategy(+ settings_snapshot={+ "search.tool": "duckduckgo",+ "search.fetch.mode": "disabled",+ })- assert (- strategy._observation_event(- self._msg(name="fetch_content", content=error)- )- is None++ tools = strategy._build_tools(overall_query="anything")++ tool_names = {+ getattr(t, "name", None) or getattr(t, "__name__", None)+ for t in tools+ }+ # No tool whose name contains 'fetch'.+ assert all(+ "fetch" not in (name or "").lower() for name in tool_names+ ), (+ f"Expected no fetch tool with fetch_mode='disabled' but got "+ f"tools: {tool_names}")- def test_successful_fetch_still_emits_milestone(self):- """A successful ``fetch_content`` observation (the tool returns a- ``[N] Title: …\\nURL: …`` payload) must still produce a milestone —- the suppression above is denial-only.- """- strategy = self._make_strategy()- content = (- "[1] Title: Foo\nURL: https://example.com/page\n\nSummary text"++class TestResolveEngineNameIgnoresNonString:+ """``_resolve_engine_name`` short-circuits to the settings value only+ when it is a string (``isinstance(tool_setting, str)``); anything+ else — a list, a dict without a ``value`` key, an int — falls+ through to the class-name heuristic. The existing tests covered+ the success path and the bare-class fallback but didn't pin the+ non-string guard against realistic misconfiguration shapes.+ """++ def _make_strategy_with_search_tool_value(self, search_tool_value):+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ LangGraphAgentStrategy,)- message, metadata = strategy._observation_event(- self._msg(name="fetch_content", content=content)++ mock_search = MagicMock()+ mock_search.__class__.__name__ = "BraveSearchEngine"+ return LangGraphAgentStrategy(+ model=MagicMock(),+ search=mock_search,+ all_links_of_system=[],+ settings_snapshot={"search.tool": search_tool_value},)- assert message.startswith("📄 From the page: ")- assert metadata["phase"] == "observation"- assert metadata["tool"] == "fetch_content"- # The URL is part of the flattened preview — covered by the existing- # test_message_is_flattened_150_char_preview contract for general- # observations, so just assert presence here.- assert "https://example.com/page" in message-- @pytest.mark.parametrize(- "tool_name",- ["web_search", "research_subtopic", "arxiv", "synthetic_tool"],- )- def test_non_fetch_tool_denial_prefix_is_not_suppressed(self, tool_name):- """Suppression is gated on ``tool_name == "fetch_content"`` — a- non-fetch tool whose result happens to start with ``Cannot fetch``- or ``Error fetching`` (e.g. an engine returning a denial string)- must still surface as a MILESTONE. The earlier string-prefix-only- match would silently drop legitimate observations from other- tools whose content happens to begin with those words.+ def test_list_settings_value_falls_through_to_class_heuristic(self):+ """A list at ``search.tool`` is not a valid engine name — the+ ``isinstance(..., str)`` guard rejects it and the class-name+ heuristic kicks in."""- strategy = self._make_strategy()- content = "Cannot fetch results: upstream returned 503 after retries"- message, metadata = strategy._observation_event(- self._msg(name=tool_name, content=content)+ strategy = self._make_strategy_with_search_tool_value(+ ["this is not a string"])+ assert strategy._search_engine_name == "brave"- assert message is not None- assert message.startswith("📄 From ")- assert "Cannot fetch results" in message- assert metadata["phase"] == "observation"- assert metadata["tool"] == tool_name+ def test_int_settings_value_falls_through_to_class_heuristic(self):+ """Numeric values likewise fall through — pins that the guard+ rejects any non-string type, not just dicts.+ """+ strategy = self._make_strategy_with_search_tool_value(42)+ assert strategy._search_engine_name == "brave"# ----------------------------------------------------------------------------# Step heartbeat (full tool listing)+# Original research question must survive the tool-call display loop# ----------------------------------------------------------------------------class TestHeartbeatMessage:- """Pin ``LangGraphAgentStrategy._heartbeat_message``: once sources are- gathered the heartbeat lists EVERY enabled tool by friendly name — the- old 3-name sample with "+N more" hid most engines."""+class TestQueryParameterNotClobbered:+ """Regression for the ``query`` parameter clobber in ``analyze_topic``.- def _make_strategy(self, links=None):+ The tool-call display loop builds a short label from each search tool's+ argument. A prior version assigned that label to ``query`` — the method+ parameter holding the *user's original research question* — so after the+ first ``web_search`` call, the original question was silently replaced by+ a truncated (<=80 char) search arg. That clobbered value then flowed into+ ``_finalize`` (the citation re-synthesis and the recorded+ ``findings[0]["question"]``) and the fallback ``_synthesize_from_collector``+ prompt, steering the final answer at the *wrong* question on the default+ research strategy. This test pins that the original question reaches+ ``_finalize`` unchanged after a run that issues a search tool call.+ """++ def _make_strategy(self):from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (LangGraphAgentStrategy,)@@ -2262,1751 +3489,1929 @@ class TestHeartbeatMessage:return LangGraphAgentStrategy(model=MagicMock(),search=MagicMock(),- all_links_of_system=links if links is not None else [],- settings_snapshot={"search.tool": {"value": "searxng"}},+ all_links_of_system=[],+ settings_snapshot={"search.tool": {"value": "mock"}},)- def test_zero_sources_reports_planning_with_tool_count(self):- strategy = self._make_strategy()- strategy._tool_names = ["web_search", "search_arxiv"]+ def test_original_query_reaches_finalize_after_search_tool_call(self):+ from langchain_core.messages import AIMessage- out = strategy._heartbeat_message(1)+ strategy = self._make_strategy()- assert (- out == "Step 1 · planning approach with 2 research tools available…"+ original_query = (+ "What are the long-term cardiovascular effects of chronic sleep "+ "deprivation in adults over the age of fifty?")- def test_lists_all_tools_without_more_suffix(self):- strategy = self._make_strategy(links=[{"link": "http://a.com"}] * 5)- strategy._tool_names = [- "web_search",- "search_arxiv",- "search_pubmed",- "search_wikipedia",- "search_github",- "search_semantic_scholar",- ]-- out = strategy._heartbeat_message(3)+ # Agent emits a web_search tool call (whose arg differs from and is+ # shorter-after-truncation than the original question), then a final+ # answer message with no tool calls.+ tool_call_msg = AIMessage(+ content="",+ tool_calls=[+ {+ "name": "web_search",+ "args": {+ "query": "sleep deprivation heart disease older adults"+ },+ "id": "call_1",+ "type": "tool_call",+ }+ ],+ )+ answer_msg = AIMessage(content="Final synthesized answer with [1].")- assert out.startswith(- "Step 3 · 5 sources gathered · selecting next action from "+ mock_agent = MagicMock()+ mock_agent.stream.return_value = iter(+ [+ {"agent": {"messages": [tool_call_msg]}},+ {"agent": {"messages": [answer_msg]}},+ ])- for name in (- "the web (SearXNG)",- "arXiv",- "PubMed",- "Wikipedia",- "GitHub",- "Semantic Scholar",- ):- assert name in out- assert "more" not in out- assert not out.endswith("…")- def test_uses_configured_collection_label(self):- from local_deep_research.web_search_engines import search_engines_config+ captured = {}- strategy = self._make_strategy(links=[{"link": "http://a.com"}])- strategy._tool_names = ["search_collection_abc123"]+ def fake_finalize(query, final_answer, *args, **kwargs):+ captured["query"] = query+ return {+ "findings": [{"question": query, "content": final_answer}],+ "current_knowledge": final_answer,+ "iterations": 1,+ "error": None,+ }- with patch.object(- search_engines_config,- "search_config",- return_value={- "collection_abc123": {"display_name": "History (Collection)"}- },+ with (+ patch.object(strategy, "_build_tools", return_value=[MagicMock()]),+ patch("langchain.agents.create_agent", return_value=mock_agent),+ patch.object(strategy, "_update_progress"),+ patch.object(strategy, "_finalize", side_effect=fake_finalize),):- out = strategy._heartbeat_message(2)-- assert "History (Collection)" in out- assert "abc123" not in out-- def test_non_search_tools_use_list_friendly_labels(self):- """`fetch_content` ("the page") and `research_subtopic`- ("subtopic researcher") read wrong in a comma list — the heartbeat- must use the list-friendly overrides."""- strategy = self._make_strategy(links=[{"link": "http://a.com"}])- strategy._tool_names = [- "web_search",- "fetch_content",- "research_subtopic",- ]-- out = strategy._heartbeat_message(2)-- assert "page fetching" in out- assert "subtopic research" in out- assert "the page" not in out- assert "subtopic researcher" not in out-- def test_single_source_uses_singular(self):- strategy = self._make_strategy(links=[{"link": "http://a.com"}])- strategy._tool_names = ["web_search"]-- out = strategy._heartbeat_message(2)-- assert "1 source gathered" in out+ result = strategy.analyze_topic(original_query)+ # The user's original question — not the truncated search arg — must+ # reach _finalize and be recorded as the question.+ assert captured["query"] == original_query+ assert result["findings"][0]["question"] == original_query-# ----------------------------------------------------------------------------# Citation offset for detailed report mode-# ---------------------------------------------------------------------------+class TestProgressMetadataKeepsStableId:+ """Progress metadata ``tool`` must carry the STABLE tool id while the+ human-readable engine label appears only in the message text.-class TestCitationOffset:- """Test that nr_of_links is handled correctly across multiple calls."""+ A prior revision of this PR overwrote ``metadata["tool"]`` with the+ friendly label; that discards the only machine-readable id reaching+ progress consumers. This pins the id-in-metadata / label-in-message+ split so a regression can't silently re-introduce the overwrite.+ """def _make_strategy(self):from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (LangGraphAgentStrategy,)- model = MagicMock()- model.invoke = MagicMock(- return_value=MagicMock(content="Synthesized answer")- )return LangGraphAgentStrategy(- model=model,+ model=MagicMock(),search=MagicMock(),all_links_of_system=[],- settings_snapshot={"search.tool": {"value": "mock"}},- )-- def test_collector_reset_on_analyze_topic(self):- """Collector should be reset at the start of each analyze_topic call."""- strategy = self._make_strategy()-- # Pre-populate collector- strategy.collector.add_results(- [{"title": "Old", "link": "http://old.com", "snippet": "old"}]- )- assert len(strategy.collector.results) == 1-- # analyze_topic should reset the collector- with patch(- "local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy.LangGraphAgentStrategy._build_tools",- return_value=[],- ):- result = strategy.analyze_topic("test query")-- # Collector should have been reset (even though _build_tools returned empty)- # reset() happens before _build_tools, so the error path still resets- assert result["error"] is not None # error because no tools- assert len(strategy.collector.results) == 0 # verify reset happened-- def test_all_links_accumulates_across_calls(self):- """all_links_of_system should grow across calls, not reset."""- strategy = self._make_strategy()- all_links = strategy.all_links_of_system-- strategy.collector.add_results(- [{"title": "A", "link": "http://a.com", "snippet": "a"}]+ settings_snapshot={"search.tool": {"value": "duckduckgo"}},)- assert len(all_links) == 1-- strategy.collector.reset()- strategy.collector.add_results(- [{"title": "B", "link": "http://b.com", "snippet": "b"}]- )- assert len(all_links) == 2+ def test_tool_call_metadata_keeps_id_label_in_message(self):+ from langchain_core.messages import AIMessage- def test_citation_indices_unique_across_sections(self):- """After reset, new results should get globally unique indices- (not restart from 1) so detailed report citations don't collide."""strategy = self._make_strategy()- # Section 1: adds 2 results → indices "1", "2"- strategy.collector.add_results(- [- {"title": "A", "link": "http://a.com", "snippet": "a"},- {"title": "B", "link": "http://b.com", "snippet": "b"},- ]+ tool_call_msg = AIMessage(+ content="",+ tool_calls=[+ {+ "name": "web_search",+ "args": {"query": "anything"},+ "id": "call_1",+ "type": "tool_call",+ }+ ],)- assert strategy.all_links_of_system[0]["index"] == "1"- assert strategy.all_links_of_system[1]["index"] == "2"-- # Simulate new section: reset per-call state- strategy.collector.reset()+ answer_msg = AIMessage(content="Final answer with [1].")- # Section 2: should continue from "3", not restart at "1"- strategy.collector.add_results(+ mock_agent = MagicMock()+ mock_agent.stream.return_value = iter([- {"title": "C", "link": "http://c.com", "snippet": "c"},- {"title": "D", "link": "http://d.com", "snippet": "d"},+ {"agent": {"messages": [tool_call_msg]}},+ {"agent": {"messages": [answer_msg]}},])- assert strategy.all_links_of_system[2]["index"] == "3"- assert strategy.all_links_of_system[3]["index"] == "4"++ progress_calls = []++ def capture(*args, **kwargs):+ message = args[0] if args else kwargs.get("message", "")+ metadata = (+ args[2] if len(args) > 2 else kwargs.get("metadata", {})+ ) or {}+ progress_calls.append((message, metadata))++ with (+ patch.object(strategy, "_build_tools", return_value=[MagicMock()]),+ patch("langchain.agents.create_agent", return_value=mock_agent),+ patch.object(strategy, "_update_progress", side_effect=capture),+ patch.object(+ strategy,+ "_finalize",+ return_value={+ "findings": [],+ "current_knowledge": "",+ "iterations": 1,+ "error": None,+ },+ ),+ ):+ strategy.analyze_topic("test query")++ tool_calls = [+ (msg, md)+ for msg, md in progress_calls+ if md.get("phase") == "tool_call"+ ]+ assert tool_calls, "expected a tool_call progress event"+ message, metadata = tool_calls[0]+ # metadata keeps the stable id ...+ assert metadata["tool"] == "web_search"+ # ... while the user sees the brand label in the message text.+ assert "DuckDuckGo" in message# ----------------------------------------------------------------------------# Error handling+# Egress-scope tool filtering# ---------------------------------------------------------------------------+#+# The strategy's ``_build_tools`` filters the specialized-engine tool list+# against the user's ``policy.egress_scope`` BEFORE the tools reach+# ``create_agent`` (see langgraph_agent_strategy.py line 591-655). That+# pre-filter is the "core fix for the original LangGraph silent-expansion+# complaint": the factory PEP would already refuse to instantiate a+# forbidden engine at runtime, but a runtime refusal still leaks policy+# state through the LLM's tool schema and through differential denial+# latency. Filtering the *list* means the forbidden tool names never+# enter the prompt at all.+#+# These tests pin that filter at the boundary that matters — the+# LangGraph tool list — using the real ``evaluate_engine`` /+# ``evaluate_retriever`` PDPs against a controlled engine fixture. A+# regression in either the strategy's filter loop OR the PDP itself+# shows up here.-class TestErrorHandling:- """Test error paths return proper error dicts."""+class TestEgressScopeFiltering:+ """LangGraph tool list must honour ``policy.egress_scope`` so the LLM+ never even sees engines outside the active scope.+ """- def _make_strategy(self):+ # Available-engines fixture. ``arxiv`` and ``pubmed`` are registered+ # public engines (``is_public = True`` on their classes); ``library``+ # is hardcoded local in ``evaluate_engine`` (line 322-326).+ # ``duckduckgo`` is the current primary — already added as+ # ``web_search`` and explicitly skipped at line 618.+ _FIXTURE_AVAILABLE = {+ "arxiv": {+ "is_local": False,+ "description": "arXiv preprints",+ "strengths": ["physics", "math"],+ },+ "pubmed": {+ "is_local": False,+ "description": "PubMed biomedical literature",+ "strengths": ["medicine"],+ },+ "library": {+ "is_local": True,+ "is_retriever": False,+ "description": "Local library",+ "strengths": ["personal documents"],+ },+ # A per-collection engine. evaluate_engine hardcodes the+ # ``collection_*`` name prefix as local (egress_policy.py ~322),+ # a DISTINCT code path from the ``library`` all-collections engine.+ "collection_abc123": {+ "is_local": True,+ "is_retriever": False,+ "description": "My research papers (Collection)",+ "strengths": ["curated documents"],+ },+ "duckduckgo": {+ "is_local": False,+ "description": "DuckDuckGo",+ },+ }++ def _make_strategy(self, scope, primary_engine="duckduckgo"):from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (LangGraphAgentStrategy,)+ mock_search = MagicMock()+ mock_search.__class__.__name__ = "DuckDuckGoSearchEngine"return LangGraphAgentStrategy(model=MagicMock(),- search=MagicMock(),+ search=mock_search,all_links_of_system=[],- settings_snapshot={"search.tool": {"value": "mock"}},+ settings_snapshot={+ "search.tool": primary_engine,+ "policy.egress_scope": scope,+ },)- def test_error_result_structure(self):- strategy = self._make_strategy()- result = strategy._error_result("something broke")+ @staticmethod+ def _tool_names(tools):+ names = set()+ for t in tools:+ name = getattr(t, "name", None) or getattr(t, "__name__", None)+ if name:+ names.add(name)+ return names- assert result["error"] == "something broke"- assert result["findings"] == []- assert result["iterations"] == 0- assert result["current_knowledge"] == ""- assert isinstance(result["reasoning_trace"], list)+ # ------------------------------------------------------------------+ # STRICT — only the primary web_search; NO specialized engines at all+ # ------------------------------------------------------------------- def test_no_tools_returns_error(self):- strategy = self._make_strategy()+ def test_collection_engine_treated_as_local(self):+ """A per-collection ``collection_<id>`` engine hits a DISTINCT+ classifier branch from ``library`` (the name-prefix rule in+ evaluate_engine, not a config flag). Pin that it behaves as local:+ present under PRIVATE_ONLY, filtered under PUBLIC_ONLY.+ """+ # PRIVATE_ONLY: the collection survives (it's local).+ strat_priv = self._make_strategy(scope="private_only")with patch(- "local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy.LangGraphAgentStrategy._build_tools",- return_value=[],+ "local_deep_research.web_search_engines.search_engines_config.list_eligible_engine_configs",+ return_value=self._FIXTURE_AVAILABLE,):- result = strategy.analyze_topic("test")+ priv_names = self._tool_names(+ strat_priv._build_tools(overall_query="q")+ )+ assert "search_collection_abc123" in priv_names, (+ "collection_<id> is local — must pass PRIVATE_ONLY"+ )- assert result["error"] is not None- assert "No tools" in result["error"]+ # PUBLIC_ONLY: the collection is filtered (local data stays local).+ strat_pub = self._make_strategy(scope="public_only")+ with patch(+ "local_deep_research.web_search_engines.search_engines_config.list_eligible_engine_configs",+ return_value=self._FIXTURE_AVAILABLE,+ ):+ pub_names = self._tool_names(+ strat_pub._build_tools(overall_query="q")+ )+ assert "search_collection_abc123" not in pub_names, (+ "collection_<id> is local — must be filtered under PUBLIC_ONLY"+ )- def test_agent_creation_failure_returns_error(self):- strategy = self._make_strategy()- with (- patch(- "local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy.LangGraphAgentStrategy._build_tools",- return_value=[MagicMock()],- ),- patch(- "langchain.agents.create_agent",- side_effect=ValueError("Model doesn't support tools"),- ),+ def test_strict_registers_no_specialized_search_tools(self):+ """STRICT means the agent gets only the primary ``web_search``+ (plus generic helpers like fetch_content / research_subtopic).+ Every ``search_*`` tool — public OR local — must be filtered+ out by the ``continue`` at line 623-627.+ """+ strategy = self._make_strategy(scope="strict")+ with patch(+ "local_deep_research.web_search_engines.search_engines_config.list_eligible_engine_configs",+ return_value=self._FIXTURE_AVAILABLE,):- result = strategy.analyze_topic("test")+ tools = strategy._build_tools(overall_query="q")- assert result["error"] is not None- assert "tool calling" in result["error"]+ names = self._tool_names(tools)+ # The primary web_search is unaffected.+ assert "web_search" in names+ # No specialized search_* — not arxiv, not pubmed, not library.+ specialized = {n for n in names if n.startswith("search_")}+ assert specialized == set(), (+ f"STRICT must register zero specialized search_* tools, "+ f"got: {specialized}"+ )- def test_format_agent_error_includes_exception_type(self):- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- LangGraphAgentStrategy,+ # ------------------------------------------------------------------+ # PRIVATE_ONLY — public engines filtered, local engines kept+ # ------------------------------------------------------------------++ def test_private_only_filters_out_public_specialized_engines(self):+ """Under PRIVATE_ONLY the agent must NOT see arxiv or pubmed —+ ``scope_mismatch_private_only`` from ``evaluate_engine`` — but+ library (``is_local=True``) passes through.+ """+ strategy = self._make_strategy(scope="private_only")+ with patch(+ "local_deep_research.web_search_engines.search_engines_config.list_eligible_engine_configs",+ return_value=self._FIXTURE_AVAILABLE,+ ):+ tools = strategy._build_tools(overall_query="q")++ names = self._tool_names(tools)+ assert "search_arxiv" not in names, (+ "arXiv is public — must be filtered under PRIVATE_ONLY"+ )+ assert "search_pubmed" not in names, (+ "PubMed is public — must be filtered under PRIVATE_ONLY"+ )+ assert "search_library" in names, (+ "library is local — must pass PRIVATE_ONLY filter")- msg = LangGraphAgentStrategy._format_agent_error(ValueError("boom"))-- assert "ValueError" in msg- assert "boom" in msg+ # ------------------------------------------------------------------+ # PUBLIC_ONLY — local engines filtered, public engines kept+ # ------------------------------------------------------------------+ def test_public_only_filters_out_local_specialized_engines(self):+ """Under PUBLIC_ONLY the agent must NOT see ``search_library`` —+ ``scope_mismatch_public_only`` — but arxiv and pubmed remain.+ This is the user-data-stays-on-the-box property: a PUBLIC_ONLY+ run must never load local indexes into the agent's tool surface.+ """+ strategy = self._make_strategy(scope="public_only")+ with patch(+ "local_deep_research.web_search_engines.search_engines_config.list_eligible_engine_configs",+ return_value=self._FIXTURE_AVAILABLE,+ ):+ tools = strategy._build_tools(overall_query="q")-# ----------------------------------------------------------------------------# Factory integration-# ---------------------------------------------------------------------------+ names = self._tool_names(tools)+ assert "search_library" not in names, (+ "library is local — must be filtered under PUBLIC_ONLY"+ )+ assert "search_arxiv" in names, (+ "arXiv is public — must pass PUBLIC_ONLY filter"+ )+ assert "search_pubmed" in names, (+ "PubMed is public — must pass PUBLIC_ONLY filter"+ )+ # ------------------------------------------------------------------+ # BOTH (default) — every classified engine is registered+ # -------------------------------------------------------------------class TestFactoryIntegration:- """Test that the strategy integrates with the factory correctly."""+ def test_both_scope_registers_every_classified_engine(self):+ """The default scope BOTH must register every classified engine+ in the available dict. The current primary is excluded by the+ explicit ``continue`` at line 618 — NOT by the scope filter — so+ a regression that moved it into the scope-mismatch path would+ still be caught by the assertion that it's absent.+ """+ strategy = self._make_strategy(scope="both")+ with patch(+ "local_deep_research.web_search_engines.search_engines_config.list_eligible_engine_configs",+ return_value=self._FIXTURE_AVAILABLE,+ ):+ tools = strategy._build_tools(overall_query="q")- def test_factory_creates_langgraph_agent(self):- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- LangGraphAgentStrategy,- )- from local_deep_research.search_system_factory import create_strategy+ names = self._tool_names(tools)+ for expected in ("search_arxiv", "search_pubmed", "search_library"):+ assert expected in names, (+ f"Expected {expected} under BOTH but got: {sorted(names)}"+ )+ # The current engine is NEVER added as a specialized tool+ # regardless of scope.+ assert "search_duckduckgo" not in names- strategy = create_strategy(- strategy_name="langgraph-agent",- model=MagicMock(),- search=MagicMock(),- settings_snapshot={},- )- assert isinstance(strategy, LangGraphAgentStrategy)+ # ------------------------------------------------------------------+ # Fail-closed: corrupted scope value+ # ------------------------------------------------------------------- def test_factory_underscore_alias(self):- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- LangGraphAgentStrategy,+ def test_corrupted_scope_value_propagates_policy_denied(self):+ """A junk ``policy.egress_scope`` value must NOT silently fall+ through to BOTH (the most permissive scope). ``context_from_snapshot``+ raises ``PolicyDeniedError(unknown_egress_scope)``; the strategy's+ ``_build_egress_context`` re-raises it (only ValueError / KeyError /+ TypeError get swallowed). The run aborts instead of running+ unfiltered.+ """+ from local_deep_research.security.egress.policy import (+ PolicyDeniedError,)- from local_deep_research.search_system_factory import create_strategy- strategy = create_strategy(- strategy_name="langgraph_agent",- model=MagicMock(),- search=MagicMock(),- settings_snapshot={},- )- assert isinstance(strategy, LangGraphAgentStrategy)+ strategy = self._make_strategy(scope="not-a-real-scope")+ with pytest.raises(PolicyDeniedError):+ strategy._build_tools(overall_query="q")- def test_strategy_in_available_list(self):- from local_deep_research.search_system_factory import (- get_available_strategies,- )+ # ------------------------------------------------------------------+ # Audit log — every block must leave an audit-bound trail+ # ------------------------------------------------------------------- names = [s["name"] for s in get_available_strategies()]- assert "langgraph-agent" in names+ def test_blocked_engine_emits_policy_audit_log(self, loguru_caplog):+ """When the filter drops an engine, the strategy emits the+ ``specialized tool filtered by egress policy`` info line. Under+ PUBLIC_ONLY with this fixture exactly one engine (``library``)+ is local, so the line must fire exactly once — a regression+ that bypassed the filter would fire zero times, and a regression+ that over-filtered (e.g. also dropped public engines under+ PUBLIC_ONLY) would fire more than once.- def test_factory_passes_custom_params(self):- from local_deep_research.search_system_factory import create_strategy+ Note: ``logger.bind(policy_audit=True).info("...", engine=..., ...)``+ attaches the engine name and the ``policy_audit`` flag as loguru+ record extras, NOT to the rendered message text. Asserting the+ bound flag itself would require a custom loguru sink; we settle+ for the rendered-line invariant here.+ """+ strategy = self._make_strategy(scope="public_only")+ with (+ loguru_caplog.at_level("INFO"),+ patch(+ "local_deep_research.web_search_engines.search_engines_config.list_eligible_engine_configs",+ return_value=self._FIXTURE_AVAILABLE,+ ),+ ):+ strategy._build_tools(overall_query="q")- strategy = create_strategy(- strategy_name="langgraph-agent",- model=MagicMock(),- search=MagicMock(),- settings_snapshot={},- max_iterations=20,- max_sub_iterations=3,- include_sub_research=False,+ marker = "specialized tool filtered by egress policy"+ occurrences = loguru_caplog.text.count(marker)+ # Under PUBLIC_ONLY every LOCAL engine in the fixture is dropped:+ # ``library`` and ``collection_abc123``. One audit line per drop.+ local_engine_count = sum(+ 1+ for name, cfg in self._FIXTURE_AVAILABLE.items()+ if cfg.get("is_local") is True+ )+ assert occurrences == local_engine_count, (+ f"Expected one audit-log line per dropped local engine "+ f"({local_engine_count}), got {occurrences}. Captured text:\n"+ f"{loguru_caplog.text}")- assert strategy.max_iterations == 20- assert strategy.max_sub_iterations == 3- assert strategy.include_sub_research is False# ----------------------------------------------------------------------------# fetch_content collector registration (regression for PR #3457)+# Policy addendum — the LLM-facing scope signal# ---------------------------------------------------------------------------+#+# Filtering the tool LIST closes the latency-leak half of the timing+# attack. The other half is the prompt addendum: the LLM is *told* which+# tools exist so it doesn't waste tokens probing for forbidden engines.+# These tests pin that the addendum text varies by scope and is empty+# under BOTH (we don't want to bleed policy state into the LLM for the+# default scope).-class TestFetchContentCollectorRegistration:- """Regression coverage for PR #3457.-- Prior to the fix, ``_make_fetch_content_tool`` accepted ``collector`` but- never used it, so every URL opened via the LLM's ``fetch_content`` tool- was silently dropped from the final Sources section and citation system.- These tests pin the fix: a successful fetch must register the URL, a- duplicate fetch must reuse the existing citation index, and a failed- fetch must not register anything.+class TestEgressScopePolicyAddendum:+ """``analyze_topic`` injects a policy addendum into the system prompt+ that gets passed to ``create_agent``. The addendum's presence and+ wording must reflect the active scope."""- def _make_collector(self):+ def _make_strategy(self, scope, primary_engine="duckduckgo"):from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- SearchResultsCollector,+ LangGraphAgentStrategy,)- return SearchResultsCollector([])-- def _fetcher_cm(- self, *, status="success", title="Page", content="Body", error=None- ):- """Return a MagicMock that behaves like ``ContentFetcher(...)``."""- result = {"status": status, "title": title, "content": content}- if error is not None:- result["error"] = error- fetcher = MagicMock()- fetcher.fetch.return_value = result- cm = MagicMock()- cm.__enter__.return_value = fetcher- cm.__exit__.return_value = False- return cm-- def _make_tool(self, collector):- from local_deep_research.advanced_search_system.tools.fetch import (- build_fetch_tool,+ mock_search = MagicMock()+ mock_search.__class__.__name__ = "DuckDuckGoSearchEngine"+ return LangGraphAgentStrategy(+ model=MagicMock(),+ search=mock_search,+ all_links_of_system=[],+ settings_snapshot={+ "search.tool": primary_engine,+ "policy.egress_scope": scope,+ },)- return build_fetch_tool("full", collector)-- def test_successful_fetch_registers_url_in_collector(self):- collector = self._make_collector()- tool = self._make_tool(collector)- cm = self._fetcher_cm(title="Hello", content="some body text")-- with patch(- "local_deep_research.content_fetcher.ContentFetcher",- return_value=cm,- ):- output = tool.invoke({"url": "http://example.com/page"})+ def _capture_prompt(self, scope, primary="duckduckgo"):+ """Run analyze_topic in a heavily-mocked harness and return the+ ``system_prompt`` string passed to ``create_agent``. There's no+ smaller public hook for the addendum — the prompt-string is+ the surface the LLM actually receives.+ """+ from langchain_core.messages import AIMessage- assert "http://example.com/page" in collector.sources- assert len(collector.results) == 1- entry = collector.results[0]- assert entry["link"] == "http://example.com/page"- assert entry["title"] == "Hello"- assert entry["source_engine"] == "fetch"- # Tool return is prefixed with the 1-based citation index so the- # agent can cite fetched pages the same way it cites web_search hits.- assert output.startswith("[1] ")+ strategy = self._make_strategy(scope=scope, primary_engine=primary)+ captured = {}- def test_repeated_fetch_of_same_url_reuses_citation_index(self):- collector = self._make_collector()- # Simulate web_search having already captured this URL.- collector.add_results(- [- {- "title": "From search",- "link": "http://example.com/page",- "snippet": "snip",- }- ],- engine_name="web",+ mock_agent = MagicMock()+ mock_agent.stream.return_value = iter(+ [{"agent": {"messages": [AIMessage(content="done")]}}])- assert len(collector.results) == 1- tool = self._make_tool(collector)- cm = self._fetcher_cm(title="From fetch", content="full body")+ def fake_create_agent(model=None, tools=None, system_prompt=None, **kw):+ captured["system_prompt"] = system_prompt+ return mock_agent- with patch(- "local_deep_research.content_fetcher.ContentFetcher",- return_value=cm,+ with (+ patch.object(strategy, "_build_tools", return_value=[MagicMock()]),+ patch(+ "langchain.agents.create_agent",+ side_effect=fake_create_agent,+ ),+ patch.object(strategy, "_update_progress"),+ patch.object(+ strategy,+ "_finalize",+ return_value={+ "findings": [],+ "current_knowledge": "",+ "iterations": 0,+ "error": None,+ },+ ),):- output = tool.invoke({"url": "http://example.com/page"})+ strategy.analyze_topic("q")+ return captured.get("system_prompt", "") or ""- # No duplicate entry; the fetch reuses the existing citation slot.- assert len(collector.results) == 1- assert output.startswith("[1] ")+ def test_strict_addendum_locks_llm_to_primary_engine(self):+ """STRICT must tell the LLM that ``search_*`` tools don't exist+ and name the primary engine — otherwise the LLM may probe for+ a denied tool, and the denial latency leaks policy state.+ """+ prompt = self._capture_prompt("strict")+ assert "RESTRICTED MODE" in prompt+ # The primary engine name must be cited.+ assert "duckduckgo" in prompt.lower()- def test_failed_fetch_does_not_register_url(self):- collector = self._make_collector()- tool = self._make_tool(collector)- cm = self._fetcher_cm(- status="error", title="", content="", error="timeout"- )+ def test_private_only_addendum_names_public_engines_as_unavailable(self):+ """PRIVATE-ONLY addendum must explicitly warn the LLM that+ public engines are out of scope so it doesn't waste turns+ calling search_arxiv etc.+ """+ prompt = self._capture_prompt("private_only")+ assert "PRIVATE-ONLY MODE" in prompt+ # Names at least one canonical public engine so the LLM+ # generalises correctly.+ assert "arxiv" in prompt.lower()- with patch(- "local_deep_research.content_fetcher.ContentFetcher",- return_value=cm,- ):- output = tool.invoke({"url": "http://broken.example/page"})+ def test_public_only_addendum_names_local_engines_as_unavailable(self):+ """PUBLIC-ONLY addendum must mark local tools as unavailable —+ and it must NOT be the STRICT addendum (different scope, different+ rules).+ """+ prompt = self._capture_prompt("public_only")+ assert "PUBLIC-ONLY MODE" in prompt+ assert "RESTRICTED MODE" not in prompt+ # Names at least one canonical local tool.+ assert "library" in prompt.lower()- assert collector.results == []- assert collector.sources == []- assert "Failed to fetch" in output+ def test_both_scope_injects_no_policy_addendum(self):+ """Under BOTH (default), the strategy MUST NOT inject any of the+ three scope-specific marker phrases. Bleeding scope state into+ every prompt would (a) bloat the default-case prompt for no+ reason and (b) leak which scope the user picked even when they+ didn't restrict anything.+ """+ prompt = self._capture_prompt("both")+ assert "RESTRICTED MODE" not in prompt+ assert "PRIVATE-ONLY MODE" not in prompt+ assert "PUBLIC-ONLY MODE" not in prompt- def test_long_content_snippet_is_truncated_with_ellipsis(self):- collector = self._make_collector()- tool = self._make_tool(collector)- cm = self._fetcher_cm(title="Long", content="A" * 500)- with patch(- "local_deep_research.content_fetcher.ContentFetcher",- return_value=cm,- ):- tool.invoke({"url": "http://example.com/long"})+# ---------------------------------------------------------------------------+# research_subtopic overflow handling (#5012, #5281)+# ---------------------------------------------------------------------------- snippet = collector.results[0]["snippet"]- assert snippet.endswith("...")- assert len(snippet) == 203 # 200 chars + "..."- def test_find_by_url_returns_index_when_present(self):- collector = self._make_collector()- collector.add_results(- [{"title": "A", "link": "http://a.com", "snippet": "a"}],- engine_name="web",- )- assert collector.find_by_url("http://a.com") == 1+class TestResearchSubtopicToolOverflow:+ """MAX_SUBTOPICS stays the prompt contract while bounded overflow queues.- def test_find_by_url_returns_none_when_absent(self):- collector = self._make_collector()- collector.add_results(- [{"title": "A", "link": "http://a.com", "snippet": "a"}],- engine_name="web",+ Calls beyond the hard limit reject the whole batch so partial first-N+ execution cannot silently discard the tail.+ """++ MODULE = (+ "local_deep_research.advanced_search_system.strategies."+ "langgraph_agent_strategy"+ )++ def _make_tool(self, progress_callback=None, max_subagent_workers=None):+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ MAX_SUBTOPICS,+ SearchResultsCollector,+ _make_research_subtopic_tool,)- assert collector.find_by_url("http://missing.com") is None+ collector = SearchResultsCollector([])+ worker_kwargs = (+ {"max_subagent_workers": max_subagent_workers}+ if max_subagent_workers is not None+ else {}+ )+ tool = _make_research_subtopic_tool(+ search_engine_name="duckduckgo",+ model=MagicMock(),+ settings_snapshot={"search.tool": {"value": "duckduckgo"}},+ collector=collector,+ max_sub_iterations=8,+ progress_callback=progress_callback,+ **worker_kwargs,+ )+ return tool, MAX_SUBTOPICS-class TestFetchModeSettingResolution:- """``LangGraphAgentStrategy.__init__`` reads the ``search.fetch.mode``- setting (added in #3680; default changed to ``summary_focus_query``- in #3793) and feeds it to ``build_fetch_tool``. The constructor must:+ def _patched_run(+ self,+ subtopics,+ progress_callback=None,+ max_subagent_workers=None,+ invoke_side_effect=None,+ ):+ import local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy as mod- - Accept any value in ``FETCH_MODES`` verbatim.- - Reject any other value, log a warning, and fall back to- ``summary_focus_query`` rather than crashing or letting an unknown- mode reach ``build_fetch_tool``.+ tool, max_sub = self._make_tool(+ progress_callback, max_subagent_workers=max_subagent_workers+ )- The existing tests covered the constructor and tool-building paths- but not this guard.- """+ agent_mock = MagicMock()+ if invoke_side_effect is not None:+ agent_mock.invoke.side_effect = invoke_side_effect+ else:+ agent_mock.invoke.return_value = {+ "messages": [MagicMock(content="finding for topic")]+ }+ with patch.object(+ mod, "_make_web_search_tool", return_value=MagicMock()+ ):+ with patch.object(mod, "build_fetch_tool", return_value=None):+ with patch(+ "langchain.agents.create_agent", return_value=agent_mock+ ):+ result = tool.invoke({"subtopics": subtopics})+ return result, max_sub- def _make_strategy(self, **overrides):+ def test_constant_matches_prompt_and_docstring_contract(self):from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ MAX_SUBTOPICS,+ MAX_SUBTOPICS_HARD_LIMIT,LangGraphAgentStrategy,)- defaults = {- "model": MagicMock(),- "search": MagicMock(),- "all_links_of_system": [],- "settings_snapshot": {"search.tool": "duckduckgo"},- }- defaults.update(overrides)- return LangGraphAgentStrategy(**defaults)+ # The constant must agree with the 'pass 2-5' docstring + lead prompt.+ assert MAX_SUBTOPICS == 5+ assert MAX_SUBTOPICS_HARD_LIMIT == 10+ assert MAX_SUBTOPICS_HARD_LIMIT > MAX_SUBTOPICS- def test_known_fetch_mode_accepted_verbatim(self):- """``summary_focus`` (one of the ``FETCH_MODES``) must round-trip- through the constructor unchanged.- """- strategy = self._make_strategy(- settings_snapshot={- "search.tool": "duckduckgo",- "search.fetch.mode": "summary_focus",- }- )- assert strategy.fetch_mode == "summary_focus"+ # The research_subtopic tool docstring is what the lead LLM sees in+ # its tool schema, so it must render the same constant — a magic+ # number here could silently diverge if MAX_SUBTOPICS ever changes+ # (reviewer note on PR #5013 follow-up).+ tool, _ = self._make_tool()+ assert f"2-{MAX_SUBTOPICS}" in tool.description+ assert f"up to {MAX_SUBTOPICS_HARD_LIMIT}" in tool.description+ assert "rejected without starting any subagents" in tool.description- def test_unknown_fetch_mode_falls_back_to_default_with_warning(- self, loguru_caplog- ):- """A misconfigured setting must not crash the constructor or- propagate an unknown mode into ``build_fetch_tool``. The guard- at the top of ``__init__`` logs a warning and substitutes the- default. Anyone removing the guard would surface as the mode- leaking through unchanged AND the warning going missing.- """- with loguru_caplog.at_level("WARNING"):- strategy = self._make_strategy(- settings_snapshot={- "search.tool": "duckduckgo",- "search.fetch.mode": "definitely-not-a-real-mode",- }- )+ # And the lead prompt must render the *same* constant, so the two+ # can't silently drift apart — a magic number in the prompt text+ # could otherwise diverge from MAX_SUBTOPICS.+ captured = {}++ def _fake_create_agent(model=None, tools=None, system_prompt=None):+ captured["system_prompt"] = system_prompt+ return MagicMock()++ strategy = LangGraphAgentStrategy(+ model=MagicMock(),+ search=MagicMock(),+ settings_snapshot={},+ max_sub_iterations=8,+ )+ with patch(+ "langchain.agents.create_agent", side_effect=_fake_create_agent+ ):+ with patch.object(+ strategy,+ "_build_tools",+ return_value=[MagicMock(name="web_search")],+ ):+ strategy.analyze_topic("does the prompt honor the limit?")- assert strategy.fetch_mode == "summary_focus_query"- assert "Unknown search.fetch.mode" in loguru_caplog.text- assert "definitely-not-a-real-mode" in loguru_caplog.text+ prompt = captured["system_prompt"]+ assert f"pass 2-{MAX_SUBTOPICS}" in prompt+ assert f"{MAX_SUBTOPICS + 1}-{MAX_SUBTOPICS_HARD_LIMIT}" in prompt+ assert "rejected without doing work" in prompt- def test_disabled_fetch_mode_omits_fetch_tool(self):- """``fetch_mode='disabled'`` must produce a tool list with NO- fetch tool — ``build_fetch_tool`` returns ``None`` and the- ``if fetch is not None`` guard skips the append. A regression- that always-appended would surface here as an extra tool.- """- strategy = self._make_strategy(- settings_snapshot={- "search.tool": "duckduckgo",- "search.fetch.mode": "disabled",- }+ def test_below_preferred_limit_has_no_overflow_metadata(self):+ captured = {}+ subtopics = [f"topic {i}" for i in range(3)]++ result, _ = self._patched_run(+ subtopics,+ progress_callback=lambda *a: captured.update({"meta": a[2]}),)- tools = strategy._build_tools(overall_query="anything")+ assert "## topic 0" in result+ assert "## topic 2" in result+ assert "overflow_strategy" not in captured["meta"]+ assert "overflow_queued_count" not in captured["meta"]+ assert "truncated_from" not in captured["meta"]- tool_names = {- getattr(t, "name", None) or getattr(t, "__name__", None)- for t in tools- }- # No tool whose name contains 'fetch'.- assert all(- "fetch" not in (name or "").lower() for name in tool_names- ), (- f"Expected no fetch tool with fetch_mode='disabled' but got "- f"tools: {tool_names}"- )+ def test_bounded_overflow_queues_extra_and_warns(self):+ captured = {}+ subtopics = [f"topic {i}" for i in range(8)]+ with patch(+ "local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy.logger"+ ) as log:+ result, max_sub = self._patched_run(+ subtopics,+ progress_callback=lambda *a: captured.update(+ {"message": a[0], "meta": a[2]}+ ),+ )-class TestResolveEngineNameIgnoresNonString:- """``_resolve_engine_name`` short-circuits to the settings value only- when it is a string (``isinstance(tool_setting, str)``); anything- else — a list, a dict without a ``value`` key, an int — falls- through to the class-name heuristic. The existing tests covered- the success path and the bare-class fallback but didn't pin the- non-string guard against realistic misconfiguration shapes.- """+ for topic in subtopics:+ assert f"## {topic}" in result+ assert max_sub == 5+ assert captured["meta"]["overflow_strategy"] == "queued"+ assert captured["meta"]["overflow_queued_count"] == 3+ assert "truncated_from" not in captured["meta"]+ assert "up to 4 in parallel" in captured["message"]+ assert "3 above the preferred limit queued" in captured["message"]+ log.warning.assert_called_once()+ warning_args = log.warning.call_args.args+ assert len(warning_args) == 4+ assert warning_args[1] == 8+ assert warning_args[2] == 3+ assert warning_args[3] == 5+ assert warning_args[0].count("{}") == 3- def _make_strategy_with_search_tool_value(self, search_tool_value):- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- LangGraphAgentStrategy,- )+ def test_queued_overflow_topic_failure_still_appends_note(self):+ captured = {}+ subtopics = [f"topic {i}" for i in range(8)]- mock_search = MagicMock()- mock_search.__class__.__name__ = "BraveSearchEngine"- return LangGraphAgentStrategy(- model=MagicMock(),- search=mock_search,- all_links_of_system=[],- settings_snapshot={"search.tool": search_tool_value},- )+ def invoke(payload, _config):+ topic = payload["messages"][0]["content"]+ if topic == "topic 6":+ raise RuntimeError("queued worker failed")+ return {"messages": [MagicMock(content=f"finding for {topic}")]}- def test_list_settings_value_falls_through_to_class_heuristic(self):- """A list at ``search.tool`` is not a valid engine name — the- ``isinstance(..., str)`` guard rejects it and the class-name- heuristic kicks in.- """- strategy = self._make_strategy_with_search_tool_value(- ["this is not a string"]+ result, _ = self._patched_run(+ subtopics,+ progress_callback=lambda *a: captured.update({"meta": a[2]}),+ invoke_side_effect=invoke,)- assert strategy._search_engine_name == "brave"-- def test_int_settings_value_falls_through_to_class_heuristic(self):- """Numeric values likewise fall through — pins that the guard- rejects any non-string type, not just dicts.- """- strategy = self._make_strategy_with_search_tool_value(42)- assert strategy._search_engine_name == "brave"+ assert "## topic 6" in result+ assert "Research on 'topic 6' failed: queued worker failed" in result+ assert "Overflow handling: 3 subtopic(s)" in result+ assert captured["meta"]["overflow_strategy"] == "queued"+ assert captured["meta"]["overflow_queued_count"] == 3-# ----------------------------------------------------------------------------# Original research question must survive the tool-call display loop-# ---------------------------------------------------------------------------+ @pytest.mark.parametrize(+ ("topic_count", "configured_workers", "expected_workers"),+ [+ (3, 0, 1),+ (3, 1, 1),+ (4, 3, 3),+ (8, 10, 5),+ ],+ )+ def test_worker_pool_clamps_all_boundaries(+ self, topic_count, configured_workers, expected_workers+ ):+ import local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy as mod+ real_executor = mod.ThreadPoolExecutor+ observed = {}-class TestQueryParameterNotClobbered:- """Regression for the ``query`` parameter clobber in ``analyze_topic``.+ def recording_executor(max_workers):+ observed["max_workers"] = max_workers+ return real_executor(max_workers=max_workers)- The tool-call display loop builds a short label from each search tool's- argument. A prior version assigned that label to ``query`` — the method- parameter holding the *user's original research question* — so after the- first ``web_search`` call, the original question was silently replaced by- a truncated (<=80 char) search arg. That clobbered value then flowed into- ``_finalize`` (the citation re-synthesis and the recorded- ``findings[0]["question"]``) and the fallback ``_synthesize_from_collector``- prompt, steering the final answer at the *wrong* question on the default- research strategy. This test pins that the original question reaches- ``_finalize`` unchanged after a run that issues a search tool call.- """+ with patch.object(mod, "ThreadPoolExecutor", recording_executor):+ self._patched_run(+ [f"topic {i}" for i in range(topic_count)],+ max_subagent_workers=configured_workers,+ )- def _make_strategy(self):- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- LangGraphAgentStrategy,- )+ assert observed["max_workers"] == expected_workers- return LangGraphAgentStrategy(- model=MagicMock(),- search=MagicMock(),- all_links_of_system=[],- settings_snapshot={"search.tool": {"value": "mock"}},- )+ def test_bounded_overflow_is_explained_to_lead_agent(self):+ subtopics = [f"topic {i}" for i in range(8)]- def test_original_query_reaches_finalize_after_search_tool_call(self):- from langchain_core.messages import AIMessage+ result, _ = self._patched_run(subtopics)- strategy = self._make_strategy()+ assert "Overflow handling:" in result+ assert "3 subtopic(s)" in result+ assert "were queued for processing instead of being dropped" in result+ assert "not investigated" not in result- original_query = (- "What are the long-term cardiovascular effects of chronic sleep "- "deprivation in adults over the age of fifty?"+ def test_exactly_at_preferred_limit_has_no_overflow_signal(self):+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ MAX_SUBTOPICS,)- # Agent emits a web_search tool call (whose arg differs from and is- # shorter-after-truncation than the original question), then a final- # answer message with no tool calls.- tool_call_msg = AIMessage(- content="",- tool_calls=[- {- "name": "web_search",- "args": {- "query": "sleep deprivation heart disease older adults"- },- "id": "call_1",- "type": "tool_call",- }- ],+ captured = {}+ subtopics = [f"topic {i}" for i in range(MAX_SUBTOPICS)]++ result, _ = self._patched_run(+ subtopics,+ progress_callback=lambda *a: captured.update({"meta": a[2]}),)- answer_msg = AIMessage(content="Final synthesized answer with [1].")- mock_agent = MagicMock()- mock_agent.stream.return_value = iter(- [- {"agent": {"messages": [tool_call_msg]}},- {"agent": {"messages": [answer_msg]}},- ]+ for i in range(MAX_SUBTOPICS):+ assert f"## topic {i}" in result+ assert "overflow_strategy" not in captured["meta"]+ assert "truncated_from" not in captured["meta"]+ assert "Overflow handling:" not in result++ def test_one_over_preferred_limit_queues_exactly_one(self):+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ MAX_SUBTOPICS,)captured = {}+ subtopics = [f"topic {i}" for i in range(MAX_SUBTOPICS + 1)]- def fake_finalize(query, final_answer, *args, **kwargs):- captured["query"] = query- return {- "findings": [{"question": query, "content": final_answer}],- "current_knowledge": final_answer,- "iterations": 1,- "error": None,- }+ result, _ = self._patched_run(+ subtopics,+ progress_callback=lambda *a: captured.update({"meta": a[2]}),+ )- with (- patch.object(strategy, "_build_tools", return_value=[MagicMock()]),- patch("langchain.agents.create_agent", return_value=mock_agent),- patch.object(strategy, "_update_progress"),- patch.object(strategy, "_finalize", side_effect=fake_finalize),- ):- result = strategy.analyze_topic(original_query)+ for i in range(MAX_SUBTOPICS + 1):+ assert f"## topic {i}" in result+ assert captured["meta"]["overflow_queued_count"] == 1+ assert "Overflow handling: 1 subtopic(s)" in result- # The user's original question — not the truncated search arg — must- # reach _finalize and be recorded as the question.- assert captured["query"] == original_query- assert result["findings"][0]["question"] == original_query+ def test_exactly_at_hard_limit_is_processed(self):+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ MAX_SUBTOPICS_HARD_LIMIT,+ )+ subtopics = [f"topic {i}" for i in range(MAX_SUBTOPICS_HARD_LIMIT)]-class TestProgressMetadataKeepsStableId:- """Progress metadata ``tool`` must carry the STABLE tool id while the- human-readable engine label appears only in the message text.+ result, _ = self._patched_run(subtopics)- A prior revision of this PR overwrote ``metadata["tool"]`` with the- friendly label; that discards the only machine-readable id reaching- progress consumers. This pins the id-in-metadata / label-in-message- split so a regression can't silently re-introduce the overwrite.- """+ for topic in subtopics:+ assert f"## {topic}" in result+ assert "Overflow handling: 5 subtopic(s)" in result- def _make_strategy(self):+ def test_above_hard_limit_rejects_whole_batch_before_agent_creation(self):from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- LangGraphAgentStrategy,+ MAX_SUBTOPICS_HARD_LIMIT,)- return LangGraphAgentStrategy(- model=MagicMock(),- search=MagicMock(),- all_links_of_system=[],- settings_snapshot={"search.tool": {"value": "duckduckgo"}},+ progress_callback = MagicMock()+ tool, preferred_limit = self._make_tool(+ progress_callback=progress_callback)+ subtopics = [f"topic {i}" for i in range(MAX_SUBTOPICS_HARD_LIMIT + 1)]- def test_tool_call_metadata_keeps_id_label_in_message(self):- from langchain_core.messages import AIMessage-- strategy = self._make_strategy()+ with (+ patch("langchain.agents.create_agent") as create_agent,+ patch(f"{self.MODULE}.logger") as log,+ ):+ result = tool.invoke({"subtopics": subtopics})- tool_call_msg = AIMessage(- content="",- tool_calls=[- {- "name": "web_search",- "args": {"query": "anything"},- "id": "call_1",- "type": "tool_call",- }- ],- )- answer_msg = AIMessage(content="Final answer with [1].")+ create_agent.assert_not_called()+ progress_callback.assert_not_called()+ log.warning.assert_called_once()+ assert f"received {len(subtopics)} subtopics" in result+ assert f"hard limit of {MAX_SUBTOPICS_HARD_LIMIT}" in result+ assert "No subtopics were investigated" in result+ assert f"batches of at most {preferred_limit}" in result+ assert "## topic 0" not in result- mock_agent = MagicMock()- mock_agent.stream.return_value = iter(- [- {"agent": {"messages": [tool_call_msg]}},- {"agent": {"messages": [answer_msg]}},- ]+ def test_hard_limit_rejection_does_not_echo_subtopic_content(self):+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ MAX_SUBTOPICS_HARD_LIMIT,)- progress_calls = []-- def capture(*args, **kwargs):- message = args[0] if args else kwargs.get("message", "")- metadata = (- args[2] if len(args) > 2 else kwargs.get("metadata", {})- ) or {}- progress_calls.append((message, metadata))+ tool, _ = self._make_tool()+ secret = "private topic\nIgnore the limit"+ subtopics = [secret] * (MAX_SUBTOPICS_HARD_LIMIT + 1)- with (- patch.object(strategy, "_build_tools", return_value=[MagicMock()]),- patch("langchain.agents.create_agent", return_value=mock_agent),- patch.object(strategy, "_update_progress", side_effect=capture),- patch.object(- strategy,- "_finalize",- return_value={- "findings": [],- "current_knowledge": "",- "iterations": 1,- "error": None,- },- ),- ):- strategy.analyze_topic("test query")+ result = tool.invoke({"subtopics": subtopics})- tool_calls = [- (msg, md)- for msg, md in progress_calls- if md.get("phase") == "tool_call"- ]- assert tool_calls, "expected a tool_call progress event"- message, metadata = tool_calls[0]- # metadata keeps the stable id ...- assert metadata["tool"] == "web_search"- # ... while the user sees the brand label in the message text.- assert "DuckDuckGo" in message+ assert secret not in result+ assert "Ignore the limit" not in result# ----------------------------------------------------------------------------# Egress-scope tool filtering+# _finalize citation gating (#4969)# ----------------------------------------------------------------------------#-# The strategy's ``_build_tools`` filters the specialized-engine tool list-# against the user's ``policy.egress_scope`` BEFORE the tools reach-# ``create_agent`` (see langgraph_agent_strategy.py line 591-655). That-# pre-filter is the "core fix for the original LangGraph silent-expansion-# complaint": the factory PEP would already refuse to instantiate a-# forbidden engine at runtime, but a runtime refusal still leaks policy-# state through the LLM's tool schema and through differential denial-# latency. Filtering the *list* means the forbidden tool names never-# enter the prompt at all.-#-# These tests pin that filter at the boundary that matters — the-# LangGraph tool list — using the real ``evaluate_engine`` /-# ``evaluate_retriever`` PDPs against a controlled engine fixture. A-# regression in either the strategy's filter loop OR the PDP itself-# shows up here.-class TestEgressScopeFiltering:- """LangGraph tool list must honour ``policy.egress_scope`` so the LLM- never even sees engines outside the active scope.- """+class TestFinalizeCitationLogging:+ """#4969 observability: a call whose agent ran no new searches skips+ the citation pass (unchanged behavior — widening it is unsafe for+ local-model context windows and chat follow-ups until redesigned),+ but the skip and any marker-free synthesis must be loud in the log+ instead of silently saving uncited prose."""- # Available-engines fixture. ``arxiv`` and ``pubmed`` are registered- # public engines (``is_public = True`` on their classes); ``library``- # is hardcoded local in ``evaluate_engine`` (line 322-326).- # ``duckduckgo`` is the current primary — already added as- # ``web_search`` and explicitly skipped at line 618.- _FIXTURE_AVAILABLE = {- "arxiv": {- "is_local": False,- "description": "arXiv preprints",- "strengths": ["physics", "math"],- },- "pubmed": {- "is_local": False,- "description": "PubMed biomedical literature",- "strengths": ["medicine"],- },- "library": {- "is_local": True,- "is_retriever": False,- "description": "Local library",- "strengths": ["personal documents"],- },- # A per-collection engine. evaluate_engine hardcodes the- # ``collection_*`` name prefix as local (egress_policy.py ~322),- # a DISTINCT code path from the ``library`` all-collections engine.- "collection_abc123": {- "is_local": True,- "is_retriever": False,- "description": "My research papers (Collection)",- "strengths": ["curated documents"],- },- "duckduckgo": {- "is_local": False,- "description": "DuckDuckGo",- },- }+ _LOGGER_PATH = (+ "local_deep_research.advanced_search_system.strategies."+ "langgraph_agent_strategy.logger"+ )- def _make_strategy(self, scope, primary_engine="duckduckgo"):+ def _make_strategy(self, all_links=None, citation_handler=None):from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (LangGraphAgentStrategy,)- mock_search = MagicMock()- mock_search.__class__.__name__ = "DuckDuckGoSearchEngine"return LangGraphAgentStrategy(model=MagicMock(),- search=mock_search,- all_links_of_system=[],- settings_snapshot={- "search.tool": primary_engine,- "policy.egress_scope": scope,- },+ search=MagicMock(),+ all_links_of_system=all_links if all_links is not None else [],+ settings_snapshot={"search.tool": {"value": "duckduckgo"}},+ citation_handler=citation_handler,)@staticmethod- def _tool_names(tools):- names = set()- for t in tools:- name = getattr(t, "name", None) or getattr(t, "__name__", None)- if name:- names.add(name)- return names-- # ------------------------------------------------------------------- # STRICT — only the primary web_search; NO specialized engines at all- # -------------------------------------------------------------------- def test_collection_engine_treated_as_local(self):- """A per-collection ``collection_<id>`` engine hits a DISTINCT- classifier branch from ``library`` (the name-prefix rule in- evaluate_engine, not a config flag). Pin that it behaves as local:- present under PRIVATE_ONLY, filtered under PUBLIC_ONLY.- """- # PRIVATE_ONLY: the collection survives (it's local).- strat_priv = self._make_strategy(scope="private_only")- with patch(- "local_deep_research.web_search_engines.search_engines_config.list_eligible_engine_configs",- return_value=self._FIXTURE_AVAILABLE,- ):- priv_names = self._tool_names(- strat_priv._build_tools(overall_query="q")- )- assert "search_collection_abc123" in priv_names, (- "collection_<id> is local — must pass PRIVATE_ONLY"- )+ def _link(idx, url):+ return {+ "index": str(idx),+ "title": f"Source {idx}",+ "link": url,+ "snippet": "snippet",+ }- # PUBLIC_ONLY: the collection is filtered (local data stays local).- strat_pub = self._make_strategy(scope="public_only")- with patch(- "local_deep_research.web_search_engines.search_engines_config.list_eligible_engine_configs",- return_value=self._FIXTURE_AVAILABLE,- ):- pub_names = self._tool_names(- strat_pub._build_tools(overall_query="q")- )- assert "search_collection_abc123" not in pub_names, (- "collection_<id> is local — must be filtered under PUBLIC_ONLY"- )+ def _warnings(self, mock_logger):+ return [str(c.args[0]) for c in mock_logger.warning.call_args_list]- def test_strict_registers_no_specialized_search_tools(self):- """STRICT means the agent gets only the primary ``web_search``- (plus generic helpers like fetch_content / research_subtopic).- Every ``search_*`` tool — public OR local — must be filtered- out by the ``continue`` at line 623-627.- """- strategy = self._make_strategy(scope="strict")- with patch(- "local_deep_research.web_search_engines.search_engines_config.list_eligible_engine_configs",- return_value=self._FIXTURE_AVAILABLE,- ):- tools = strategy._build_tools(overall_query="q")+ def test_empty_collector_skips_pass_and_warns(self):+ """Empty per-call collector + accumulated sources → the pass is+ skipped (raw answer kept, handler untouched) and the skip is+ logged as a warning naming the accumulated count."""+ handler = MagicMock()+ links = [self._link(1, "https://a.example/x")]+ strategy = self._make_strategy(+ all_links=links, citation_handler=handler+ )+ assert strategy.collector.results == []- names = self._tool_names(tools)- # The primary web_search is unaffected.- assert "web_search" in names- # No specialized search_* — not arxiv, not pubmed, not library.- specialized = {n for n in names if n.startswith("search_")}- assert specialized == set(), (- f"STRICT must register zero specialized search_* tools, "- f"got: {specialized}"+ with patch(self._LOGGER_PATH) as mock_logger:+ result = strategy._finalize("q", "Uncited raw answer.", 1, 0, [])++ handler.analyze_followup.assert_not_called()+ assert result["current_knowledge"] == "Uncited raw answer."+ assert any(+ "raw answer contains no inline [N] markers" in warning+ for warning in self._warnings(mock_logger))- # ------------------------------------------------------------------- # PRIVATE_ONLY — public engines filtered, local engines kept- # ------------------------------------------------------------------+ def test_empty_collector_reports_existing_markers(self):+ """A raw answer that already cites prior context must not be+ diagnosed as having no inline citations."""+ handler = MagicMock()+ links = [self._link(1, "https://a.example/x")]+ strategy = self._make_strategy(+ all_links=links, citation_handler=handler+ )- def test_private_only_filters_out_public_specialized_engines(self):- """Under PRIVATE_ONLY the agent must NOT see arxiv or pubmed —- ``scope_mismatch_private_only`` from ``evaluate_engine`` — but- library (``is_local=True``) passes through.- """- strategy = self._make_strategy(scope="private_only")- with patch(- "local_deep_research.web_search_engines.search_engines_config.list_eligible_engine_configs",- return_value=self._FIXTURE_AVAILABLE,- ):- tools = strategy._build_tools(overall_query="q")+ with patch(self._LOGGER_PATH) as mock_logger:+ result = strategy._finalize(+ "q", "Prior evidence [1] remains relevant [2, 3].", 1, 0, []+ )- names = self._tool_names(tools)- assert "search_arxiv" not in names, (- "arXiv is public — must be filtered under PRIVATE_ONLY"+ handler.analyze_followup.assert_not_called()+ assert result["current_knowledge"] == (+ "Prior evidence [1] remains relevant [2, 3].")- assert "search_pubmed" not in names, (- "PubMed is public — must be filtered under PRIVATE_ONLY"+ warnings = self._warnings(mock_logger)+ assert any(+ "already contains 2 inline [N] marker(s)" in warning+ for warning in warnings)- assert "search_library" in names, (- "library is local — must pass PRIVATE_ONLY filter"+ assert not any(+ "will have no inline [N]" in warning for warning in warnings)- # ------------------------------------------------------------------- # PUBLIC_ONLY — local engines filtered, public engines kept- # -------------------------------------------------------------------- def test_public_only_filters_out_local_specialized_engines(self):- """Under PUBLIC_ONLY the agent must NOT see ``search_library`` —- ``scope_mismatch_public_only`` — but arxiv and pubmed remain.- This is the user-data-stays-on-the-box property: a PUBLIC_ONLY- run must never load local indexes into the agent's tool surface.- """- strategy = self._make_strategy(scope="public_only")- with patch(- "local_deep_research.web_search_engines.search_engines_config.list_eligible_engine_configs",- return_value=self._FIXTURE_AVAILABLE,- ):- tools = strategy._build_tools(overall_query="q")-- names = self._tool_names(tools)- assert "search_library" not in names, (- "library is local — must be filtered under PUBLIC_ONLY"- )- assert "search_arxiv" in names, (- "arXiv is public — must pass PUBLIC_ONLY filter"+ def test_no_results_sentinel_does_not_warn_about_skip(self):+ """An agent that produced nothing returns NO_RESULTS_MESSAGE —+ that is an agent failure, not a missing-citations case, so the+ skip warning must stay quiet."""+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ NO_RESULTS_MESSAGE,)- assert "search_pubmed" in names, (- "PubMed is public — must pass PUBLIC_ONLY filter"++ handler = MagicMock()+ links = [self._link(1, "https://a.example/x")]+ strategy = self._make_strategy(+ all_links=links, citation_handler=handler)- # ------------------------------------------------------------------- # BOTH (default) — every classified engine is registered- # ------------------------------------------------------------------+ with patch(self._LOGGER_PATH) as mock_logger:+ result = strategy._finalize("q", NO_RESULTS_MESSAGE, 1, 0, [])- def test_both_scope_registers_every_classified_engine(self):- """The default scope BOTH must register every classified engine- in the available dict. The current primary is excluded by the- explicit ``continue`` at line 618 — NOT by the scope filter — so- a regression that moved it into the scope-mismatch path would- still be caught by the assertion that it's absent.- """- strategy = self._make_strategy(scope="both")- with patch(- "local_deep_research.web_search_engines.search_engines_config.list_eligible_engine_configs",- return_value=self._FIXTURE_AVAILABLE,- ):- tools = strategy._build_tools(overall_query="q")+ handler.analyze_followup.assert_not_called()+ assert result["current_knowledge"] == NO_RESULTS_MESSAGE+ assert not any(+ "Citation pass skipped" in w for w in self._warnings(mock_logger)+ )- names = self._tool_names(tools)- for expected in ("search_arxiv", "search_pubmed", "search_library"):- assert expected in names, (- f"Expected {expected} under BOTH but got: {sorted(names)}"- )- # The current engine is NEVER added as a specialized tool- # regardless of scope.- assert "search_duckduckgo" not in names+ def test_both_empty_does_not_warn(self):+ """No sources anywhere → nothing to cite, no warning noise."""+ handler = MagicMock()+ strategy = self._make_strategy(all_links=[], citation_handler=handler)- # ------------------------------------------------------------------- # Fail-closed: corrupted scope value- # ------------------------------------------------------------------+ with patch(self._LOGGER_PATH) as mock_logger:+ result = strategy._finalize("q", "Raw answer.", 1, 0, [])- def test_corrupted_scope_value_propagates_policy_denied(self):- """A junk ``policy.egress_scope`` value must NOT silently fall- through to BOTH (the most permissive scope). ``context_from_snapshot``- raises ``PolicyDeniedError(unknown_egress_scope)``; the strategy's- ``_build_egress_context`` re-raises it (only ValueError / KeyError /- TypeError get swallowed). The run aborts instead of running- unfiltered.- """- from local_deep_research.security.egress.policy import (- PolicyDeniedError,+ handler.analyze_followup.assert_not_called()+ assert result["current_knowledge"] == "Raw answer."+ assert not any(+ "Citation pass skipped" in w for w in self._warnings(mock_logger))- strategy = self._make_strategy(scope="not-a-real-scope")- with pytest.raises(PolicyDeniedError):- strategy._build_tools(overall_query="q")-- # ------------------------------------------------------------------- # Audit log — every block must leave an audit-bound trail- # ------------------------------------------------------------------+ def test_populated_collector_runs_pass_unchanged(self):+ """Per-call results present → citation pass runs exactly as+ before, with the per-call list."""+ handler = MagicMock()+ handler.analyze_followup.return_value = {+ "content": "Cited [2].",+ "documents": [],+ }+ links = [self._link(1, "https://a.example/x")]+ strategy = self._make_strategy(+ all_links=links, citation_handler=handler+ )+ strategy.collector.add_results(+ [{"title": "New", "link": "https://b.example/y", "snippet": "s"}],+ engine_name="web",+ )- def test_blocked_engine_emits_policy_audit_log(self, loguru_caplog):- """When the filter drops an engine, the strategy emits the- ``specialized tool filtered by egress policy`` info line. Under- PUBLIC_ONLY with this fixture exactly one engine (``library``)- is local, so the line must fire exactly once — a regression- that bypassed the filter would fire zero times, and a regression- that over-filtered (e.g. also dropped public engines under- PUBLIC_ONLY) would fire more than once.+ result = strategy._finalize("q", "raw", 1, 1, [])- Note: ``logger.bind(policy_audit=True).info("...", engine=..., ...)``- attaches the engine name and the ``policy_audit`` flag as loguru- record extras, NOT to the rendered message text. Asserting the- bound flag itself would require a custom loguru sink; we settle- for the rendered-line invariant here.- """- strategy = self._make_strategy(scope="public_only")- with (- loguru_caplog.at_level("INFO"),- patch(- "local_deep_research.web_search_engines.search_engines_config.list_eligible_engine_configs",- return_value=self._FIXTURE_AVAILABLE,- ),- ):- strategy._build_tools(overall_query="q")+ passed_sources = handler.analyze_followup.call_args.args[1]+ assert [r["link"] for r in passed_sources] == ["https://b.example/y"]+ assert result["current_knowledge"] == "Cited [2]."- marker = "specialized tool filtered by egress policy"- occurrences = loguru_caplog.text.count(marker)- # Under PUBLIC_ONLY every LOCAL engine in the fixture is dropped:- # ``library`` and ``collection_abc123``. One audit line per drop.- local_engine_count = sum(- 1- for name, cfg in self._FIXTURE_AVAILABLE.items()- if cfg.get("is_local") is True- )- assert occurrences == local_engine_count, (- f"Expected one audit-log line per dropped local engine "- f"({local_engine_count}), got {occurrences}. Captured text:\n"- f"{loguru_caplog.text}"+ def test_zero_marker_synthesis_logs_warning(self):+ """If the citation pass ran but its output carries no [N]+ markers, that must be visible in the server log."""+ handler = MagicMock()+ handler.analyze_followup.return_value = {+ "content": "Still no markers at all.",+ "documents": [],+ }+ strategy = self._make_strategy(all_links=[], citation_handler=handler)+ strategy.collector.add_results(+ [{"title": "New", "link": "https://b.example/y", "snippet": "s"}],+ engine_name="web",)+ with patch(self._LOGGER_PATH) as mock_logger:+ strategy._finalize("q", "raw", 1, 0, [])-# ----------------------------------------------------------------------------# Policy addendum — the LLM-facing scope signal-# ----------------------------------------------------------------------------#-# Filtering the tool LIST closes the latency-leak half of the timing-# attack. The other half is the prompt addendum: the LLM is *told* which-# tools exist so it doesn't waste tokens probing for forbidden engines.-# These tests pin that the addendum text varies by scope and is empty-# under BOTH (we don't want to bleed policy state into the LLM for the-# default scope).+ assert any(+ "no inline [N] citation markers" in w+ for w in self._warnings(mock_logger)+ )++ def test_marker_bearing_synthesis_does_not_warn(self):+ handler = MagicMock()+ handler.analyze_followup.return_value = {+ "content": "Cited [1] properly.",+ "documents": [],+ }+ strategy = self._make_strategy(all_links=[], citation_handler=handler)+ strategy.collector.add_results(+ [{"title": "New", "link": "https://b.example/y", "snippet": "s"}],+ engine_name="web",+ )+ with patch(self._LOGGER_PATH) as mock_logger:+ strategy._finalize("q", "raw", 1, 0, [])-class TestEgressScopePolicyAddendum:- """``analyze_topic`` injects a policy addendum into the system prompt- that gets passed to ``create_agent``. The addendum's presence and- wording must reflect the active scope.- """+ assert not any(+ "no inline [N] citation markers" in w+ for w in self._warnings(mock_logger)+ )- def _make_strategy(self, scope, primary_engine="duckduckgo"):+ def test_milestone_skipped_for_no_results_sentinel(self):+ """NO_RESULTS_MESSAGE sentinel must suppress the progress milestone completely."""from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- LangGraphAgentStrategy,+ NO_RESULTS_MESSAGE,)- mock_search = MagicMock()- mock_search.__class__.__name__ = "DuckDuckGoSearchEngine"- return LangGraphAgentStrategy(- model=MagicMock(),- search=mock_search,- all_links_of_system=[],- settings_snapshot={- "search.tool": primary_engine,- "policy.egress_scope": scope,- },+ strategy = self._make_strategy(all_links=[])+ progress_updates = []+ strategy.set_progress_callback(+ lambda msg, pct, meta: progress_updates.append((msg, pct, meta)))- def _capture_prompt(self, scope, primary="duckduckgo"):- """Run analyze_topic in a heavily-mocked harness and return the- ``system_prompt`` string passed to ``create_agent``. There's no- smaller public hook for the addendum — the prompt-string is- the surface the LLM actually receives.- """- from langchain_core.messages import AIMessage+ strategy._finalize("q", NO_RESULTS_MESSAGE, 1, 0, [])- strategy = self._make_strategy(scope=scope, primary_engine=primary)- captured = {}+ # The synthesis milestone (progress 90) must not be emitted+ synthesis_updates = [+ u for u in progress_updates if u[2].get("phase") == "synthesis"+ ]+ assert len(synthesis_updates) == 0- mock_agent = MagicMock()- mock_agent.stream.return_value = iter(- [{"agent": {"messages": [AIMessage(content="done")]}}]+ def test_milestone_describes_accumulated_sources_when_new_empty(self):+ """Empty per-call collector but accumulated sources present -> show accumulated sources."""+ links = [self._link(1, "https://a.example/x")]+ strategy = self._make_strategy(all_links=links)+ progress_updates = []+ strategy.set_progress_callback(+ lambda msg, pct, meta: progress_updates.append((msg, pct, meta)))- def fake_create_agent(model=None, tools=None, system_prompt=None, **kw):- captured["system_prompt"] = system_prompt- return mock_agent-- with (- patch.object(strategy, "_build_tools", return_value=[MagicMock()]),- patch(- "langchain.agents.create_agent",- side_effect=fake_create_agent,- ),- patch.object(strategy, "_update_progress"),- patch.object(- strategy,- "_finalize",- return_value={- "findings": [],- "current_knowledge": "",- "iterations": 0,- "error": None,- },- ),- ):- strategy.analyze_topic("q")- return captured.get("system_prompt", "") or ""+ strategy._finalize("q", "prose", 1, 0, [])- def test_strict_addendum_locks_llm_to_primary_engine(self):- """STRICT must tell the LLM that ``search_*`` tools don't exist- and name the primary engine — otherwise the LLM may probe for- a denied tool, and the denial latency leaks policy state.- """- prompt = self._capture_prompt("strict")- assert "RESTRICTED MODE" in prompt- # The primary engine name must be cited.- assert "duckduckgo" in prompt.lower()+ synthesis_updates = [+ u for u in progress_updates if u[2].get("phase") == "synthesis"+ ]+ assert len(synthesis_updates) == 1+ msg, pct, meta = synthesis_updates[0]+ assert (+ "Skipping citation synthesis (reusing 1 accumulated sources)" in msg+ )+ assert meta.get("citation_pass_skipped") is True+ assert meta.get("accumulated_sources") == 1- def test_private_only_addendum_names_public_engines_as_unavailable(self):- """PRIVATE-ONLY addendum must explicitly warn the LLM that- public engines are out of scope so it doesn't waste turns- calling search_arxiv etc.- """- prompt = self._capture_prompt("private_only")- assert "PRIVATE-ONLY MODE" in prompt- # Names at least one canonical public engine so the LLM- # generalises correctly.- assert "arxiv" in prompt.lower()+ def test_milestone_both_empty_emits_followup(self):+ """Both collectors empty -> emit only the fallback explanation milestone."""+ strategy = self._make_strategy(all_links=[])+ progress_updates = []+ strategy.set_progress_callback(+ lambda msg, pct, meta: progress_updates.append((msg, pct, meta))+ )- def test_public_only_addendum_names_local_engines_as_unavailable(self):- """PUBLIC-ONLY addendum must mark local tools as unavailable —- and it must NOT be the STRICT addendum (different scope, different- rules).- """- prompt = self._capture_prompt("public_only")- assert "PUBLIC-ONLY MODE" in prompt- assert "RESTRICTED MODE" not in prompt- # Names at least one canonical local tool.- assert "library" in prompt.lower()+ strategy._finalize("q", "prose", 1, 0, [])- def test_both_scope_injects_no_policy_addendum(self):- """Under BOTH (default), the strategy MUST NOT inject any of the- three scope-specific marker phrases. Bleeding scope state into- every prompt would (a) bloat the default-case prompt for no- reason and (b) leak which scope the user picked even when they- didn't restrict anything.- """- prompt = self._capture_prompt("both")- assert "RESTRICTED MODE" not in prompt- assert "PRIVATE-ONLY MODE" not in prompt- assert "PUBLIC-ONLY MODE" not in prompt+ synthesis_updates = [+ u for u in progress_updates if u[2].get("phase") == "synthesis"+ ]+ assert len(synthesis_updates) == 1+ assert (+ "No sources available for citation synthesis"+ in synthesis_updates[0][0]+ )# ----------------------------------------------------------------------------# research_subtopic overflow handling (#5012, #5281)+# Round-2 follow-up: collector/bibliography agreement (#5685)# ----------------------------------------------------------------------------class TestResearchSubtopicToolOverflow:- """MAX_SUBTOPICS stays the prompt contract while bounded overflow queues.+class TestCollectorCanonicalDedup:+ """The collector must dedup on exactly the key the rendered+ bibliography groups by.- Calls beyond the hard limit reject the whole batch so partial first-N- execution cannot silently discard the tail.+ #5381 taught ``canonical_url_key`` that ``/library/document/<id>``,+ ``/<id>/pdf``, ``/<id>/chunks#chunk-N`` and+ ``https://library.document/<id>/chunks#chunk-N`` are ONE source, but+ the collector still keyed on the raw link — so ``sources_count`` (read+ by the MCP server's ``sources`` payload and by the news impact score)+ reported three sources for a report whose ``## Sources`` block renders+ one line."""- MODULE = (- "local_deep_research.advanced_search_system.strategies."- "langgraph_agent_strategy"- )-- def _make_tool(self, progress_callback=None, max_subagent_workers=None):+ def _make_collector(self, all_links=None):from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- MAX_SUBTOPICS,SearchResultsCollector,- _make_research_subtopic_tool,)- collector = SearchResultsCollector([])- worker_kwargs = (- {"max_subagent_workers": max_subagent_workers}- if max_subagent_workers is not None- else {}- )- tool = _make_research_subtopic_tool(- search_engine_name="duckduckgo",- model=MagicMock(),- settings_snapshot={"search.tool": {"value": "duckduckgo"}},- collector=collector,- max_sub_iterations=8,- progress_callback=progress_callback,- **worker_kwargs,+ links = all_links if all_links is not None else []+ return SearchResultsCollector(links), links++ @staticmethod+ def _rendered_source_count(all_links):+ from local_deep_research.utilities.search_utilities import (+ format_links_to_markdown,)- return tool, MAX_SUBTOPICS- def _patched_run(- self,- subtopics,- progress_callback=None,- max_subagent_workers=None,- invoke_side_effect=None,- ):- import local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy as mod+ return format_links_to_markdown(all_links).count(" URL: ")- tool, max_sub = self._make_tool(- progress_callback, max_subagent_workers=max_subagent_workers+ def test_library_views_collapse_to_one_citation_and_one_source(self):+ collector, all_links = self._make_collector()+ collector.add_results(+ [+ {+ "title": "Doc",+ "link": "/library/document/abc/chunks",+ "source": "library",+ "metadata": {"doc_id": "abc", "chunk_index": 3},+ },+ {"title": "Doc", "link": "/library/document/abc/pdf"},+ {"title": "Doc", "link": "/library/document/abc"},+ ],+ engine_name="library",+ )++ assert [r["index"] for r in collector.results] == ["1", "1", "1"]+ # The recorded source keeps the anchor — that is what makes the+ # citation scroll to the cited chunk.+ assert collector.sources == ["/library/document/abc/chunks#chunk-3"]+ # ... and matches what the report actually renders.+ assert self._rendered_source_count(all_links) == len(collector.sources)+ assert self._rendered_source_count(all_links) == 1++ def test_fetch_after_search_reuses_the_search_citation_index(self):+ """The agent is shown ``[1] Doc (/library/document/abc/chunks#chunk-3)``+ and pastes a different spelling of that URL into ``fetch_content``.+ ``add_results`` ran the chunk-anchor rebuild and+ ``find_or_add_result`` does not, so keying on the raw string handed+ the same document a SECOND citation index."""+ collector, all_links = self._make_collector()+ collector.add_results(+ [+ {+ "title": "Doc",+ "link": "/library/document/abc",+ "source": "library",+ "metadata": {"doc_id": "abc", "chunk_index": 3},+ }+ ],+ engine_name="library",)+ assert all_links[0]["link"] == "/library/document/abc/chunks#chunk-3"- agent_mock = MagicMock()- if invoke_side_effect is not None:- agent_mock.invoke.side_effect = invoke_side_effect- else:- agent_mock.invoke.return_value = {- "messages": [MagicMock(content="finding for topic")]- }- with patch.object(- mod, "_make_web_search_tool", return_value=MagicMock()+ for spelling in (+ "https://library.document/abc/chunks#chunk-3",+ "/library/document/abc/chunks#chunk-3",+ "/library/document/abc/pdf",+ "/lib/document/abc",):- with patch.object(mod, "build_fetch_tool", return_value=None):- with patch(- "langchain.agents.create_agent", return_value=agent_mock- ):- result = tool.invoke({"subtopics": subtopics})- return result, max_sub+ assert (+ collector.find_or_add_result(+ {"title": "Doc", "link": spelling}, engine_name="fetch"+ )+ == 1+ ), spelling+ assert collector.find_by_url(spelling) == 1, spelling- def test_constant_matches_prompt_and_docstring_contract(self):- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- MAX_SUBTOPICS,- MAX_SUBTOPICS_HARD_LIMIT,- LangGraphAgentStrategy,+ assert len(all_links) == 1+ assert len(collector.sources) == 1+ assert self._rendered_source_count(all_links) == 1++ def test_refetch_subsection_records_results_alongside_sources(self):+ """``_sources`` was appended before all three ``find_or_add_result``+ branches while ``_results`` was appended only by the allocator, so a+ fetch-only subsection re-citing earlier URLs reported N sources and+ rendered no ``## Sources`` block at all (and tripped the #4969+ "citation pass skipped" warning)."""+ collector, _ = self._make_collector()+ collector.add_results(+ [{"title": "A", "link": "http://a.com", "snippet": "a"}])- # The constant must agree with the 'pass 2-5' docstring + lead prompt.- assert MAX_SUBTOPICS == 5- assert MAX_SUBTOPICS_HARD_LIMIT == 10- assert MAX_SUBTOPICS_HARD_LIMIT > MAX_SUBTOPICS-- # The research_subtopic tool docstring is what the lead LLM sees in- # its tool schema, so it must render the same constant — a magic- # number here could silently diverge if MAX_SUBTOPICS ever changes- # (reviewer note on PR #5013 follow-up).- tool, _ = self._make_tool()- assert f"2-{MAX_SUBTOPICS}" in tool.description- assert f"up to {MAX_SUBTOPICS_HARD_LIMIT}" in tool.description- assert "rejected without starting any subagents" in tool.description-- # And the lead prompt must render the *same* constant, so the two- # can't silently drift apart — a magic number in the prompt text- # could otherwise diverge from MAX_SUBTOPICS.- captured = {}+ collector.reset()+ assert (+ collector.find_or_add_result({"title": "A", "link": "http://a.com"})+ == 1+ )+ assert (+ collector.find_or_add_result({"title": "B", "link": "http://b.com"})+ == 2+ )- def _fake_create_agent(model=None, tools=None, system_prompt=None):- captured["system_prompt"] = system_prompt- return MagicMock()+ assert collector.sources == ["http://a.com", "http://b.com"]+ assert [r["link"] for r in collector.results] == [+ "http://a.com",+ "http://b.com",+ ]+ assert [r["index"] for r in collector.results] == ["1", "2"]- strategy = LangGraphAgentStrategy(- model=MagicMock(),- search=MagicMock(),- settings_snapshot={},- max_sub_iterations=8,+ # A repeat within the SAME subsection still must not grow either.+ collector.find_or_add_result({"title": "A", "link": "http://a.com"})+ assert len(collector.results) == 2+ assert len(collector.sources) == 2++ def test_sources_and_seen_set_stay_in_step(self):+ """``_sources`` holds display URLs and ``_sources_seen`` canonical+ keys, so they are related by canonicalization rather than equality —+ but ``_sources`` must still be duplicate-free and the same length."""+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ _citation_dedup_key,)- with patch(- "langchain.agents.create_agent", side_effect=_fake_create_agent- ):- with patch.object(- strategy,- "_build_tools",- return_value=[MagicMock(name="web_search")],- ):- strategy.analyze_topic("does the prompt honor the limit?")- prompt = captured["system_prompt"]- assert f"pass 2-{MAX_SUBTOPICS}" in prompt- assert f"{MAX_SUBTOPICS + 1}-{MAX_SUBTOPICS_HARD_LIMIT}" in prompt- assert "rejected without doing work" in prompt+ collector, _ = self._make_collector()+ collector.add_results(+ [+ {"title": "A", "link": "/library/document/abc/pdf"},+ {"title": "A", "link": "https://library.document/abc"},+ {"title": "B", "link": "http://b.com/x?utm_source=news"},+ {"title": "B", "link": "http://b.com/x"},+ ]+ )+ collector.find_or_add_result(+ {"title": "A", "link": "/lib/document/abc"}+ )- def test_below_preferred_limit_has_no_overflow_metadata(self):- captured = {}- subtopics = [f"topic {i}" for i in range(3)]+ assert len(collector.sources) == len(collector._sources_seen)+ assert len(collector.sources) == len(set(collector.sources))+ assert {+ _citation_dedup_key(u) for u in collector.sources+ } == collector._sources_seen- result, _ = self._patched_run(- subtopics,- progress_callback=lambda *a: captured.update({"meta": a[2]}),+ def test_non_string_link_does_not_raise(self):+ """An unhashable ``link`` used to raise ``TypeError`` from the+ ``in self._url_to_index`` membership test."""+ collector, all_links = self._make_collector()+ collector.add_results([{"title": "Weird", "link": ["a", "b"]}])+ assert (+ collector.find_or_add_result({"title": "Weird", "link": ["a", "b"]})+ == 2)+ assert collector.find_by_url(["a", "b"]) is None+ assert len(all_links) == 2- assert "## topic 0" in result- assert "## topic 2" in result- assert "overflow_strategy" not in captured["meta"]- assert "overflow_queued_count" not in captured["meta"]- assert "truncated_from" not in captured["meta"]+ def test_scan_loops_tolerate_legacy_non_dict_seed_entries(self):+ """``__init__`` skips non-dict seed entries; the three scan loops it+ protects used to raise ``AttributeError`` on the same seed."""+ seed = ["a legacy string", None, {"index": "3", "link": "http://c.com"}]+ collector, _ = self._make_collector(seed)- def test_bounded_overflow_queues_extra_and_warns(self):- captured = {}- subtopics = [f"topic {i}" for i in range(8)]+ assert collector.find_by_url("http://c.com") == 3+ assert collector.find_by_url("http://missing.com") is None+ assert collector.find_by_index(3)["link"] == "http://c.com"+ assert collector.find_or_add_result({"link": "http://c.com"}) == 3+ assert collector.find_or_add_result({"link": "http://d.com"}) == 4++ def test_find_by_index_returns_a_copy(self):+ """``results``/``sources`` both return copies; ``find_by_index``+ returned the live dict, so a caller rewriting ``link`` silently+ desynced ``_url_to_index``."""+ collector, all_links = self._make_collector()+ collector.add_results([{"title": "A", "link": "http://a.com"}])- with patch(- "local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy.logger"- ) as log:- result, max_sub = self._patched_run(- subtopics,- progress_callback=lambda *a: captured.update(- {"message": a[0], "meta": a[2]}- ),- )+ got = collector.find_by_index(1)+ got["link"] = "http://hijacked.example"- for topic in subtopics:- assert f"## {topic}" in result- assert max_sub == 5- assert captured["meta"]["overflow_strategy"] == "queued"- assert captured["meta"]["overflow_queued_count"] == 3- assert "truncated_from" not in captured["meta"]- assert "up to 4 in parallel" in captured["message"]- assert "3 above the preferred limit queued" in captured["message"]- log.warning.assert_called_once()- warning_args = log.warning.call_args.args- assert len(warning_args) == 4- assert warning_args[1] == 8- assert warning_args[2] == 3- assert warning_args[3] == 5- assert warning_args[0].count("{}") == 3+ assert all_links[0]["link"] == "http://a.com"+ assert collector.find_by_url("http://a.com") == 1- def test_queued_overflow_topic_failure_still_appends_note(self):- captured = {}- subtopics = [f"topic {i}" for i in range(8)]- def invoke(payload, _config):- topic = payload["messages"][0]["content"]- if topic == "topic 6":- raise RuntimeError("queued worker failed")- return {"messages": [MagicMock(content=f"finding for {topic}")]}+class TestFinalizeSourcesPayload:+ """``_finalize``'s ``sources`` payload feeds MCP clients and the news+ impact score."""- result, _ = self._patched_run(- subtopics,- progress_callback=lambda *a: captured.update({"meta": a[2]}),- invoke_side_effect=invoke,+ def _make_strategy(self, all_links=None):+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ LangGraphAgentStrategy,)- assert "## topic 6" in result- assert "Research on 'topic 6' failed: queued worker failed" in result- assert "Overflow handling: 3 subtopic(s)" in result- assert captured["meta"]["overflow_strategy"] == "queued"- assert captured["meta"]["overflow_queued_count"] == 3-- @pytest.mark.parametrize(- ("topic_count", "configured_workers", "expected_workers"),- [- (3, 0, 1),- (3, 1, 1),- (4, 3, 3),- (8, 10, 5),- ],- )- def test_worker_pool_clamps_all_boundaries(- self, topic_count, configured_workers, expected_workers- ):- import local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy as mod+ return LangGraphAgentStrategy(+ model=MagicMock(),+ search=MagicMock(),+ all_links_of_system=all_links if all_links is not None else [],+ settings_snapshot={"search.tool": {"value": "duckduckgo"}},+ citation_handler=MagicMock(),+ )- real_executor = mod.ThreadPoolExecutor- observed = {}+ def test_sources_preserve_first_seen_order(self):+ """``list(set(...))`` deduped nothing — ``_sources_seen`` already+ guarantees uniqueness — and only randomized the order, differently+ per process because of hash randomization."""+ strategy = self._make_strategy()+ urls = [f"https://example{i}.test/page" for i in range(12)]+ strategy.collector.add_results(+ [{"title": f"S{i}", "link": u} for i, u in enumerate(urls)]+ )- def recording_executor(max_workers):- observed["max_workers"] = max_workers- return real_executor(max_workers=max_workers)+ result = strategy._finalize("q", "Answer [1].", 1, 0, [])- with patch.object(mod, "ThreadPoolExecutor", recording_executor):- self._patched_run(- [f"topic {i}" for i in range(topic_count)],- max_subagent_workers=configured_workers,- )+ assert result["sources"] == urls- assert observed["max_workers"] == expected_workers+ def test_refetch_only_subsection_renders_its_sources(self):+ """End-to-end for the ``_results``/``_sources`` desync: a subsection+ that only re-fetches URLs cited earlier must render a ``## Sources``+ block instead of reporting sources it never renders."""+ strategy = self._make_strategy()+ strategy.collector.add_results(+ [{"title": "A", "link": "https://a.example/x", "snippet": "a"}]+ )+ strategy.collector.reset()+ strategy.collector.find_or_add_result(+ {"title": "A", "link": "https://a.example/x", "snippet": "a"},+ engine_name="fetch",+ )- def test_bounded_overflow_is_explained_to_lead_agent(self):- subtopics = [f"topic {i}" for i in range(8)]+ result = strategy._finalize("q", "Answer [1].", 1, 0, [])- result, _ = self._patched_run(subtopics)+ assert result["sources"] == ["https://a.example/x"]+ assert "## Sources" in result["formatted_findings"]+ assert "https://a.example/x" in result["formatted_findings"]- assert "Overflow handling:" in result- assert "3 subtopic(s)" in result- assert "were queued for processing instead of being dropped" in result- assert "not investigated" not in result- def test_exactly_at_preferred_limit_has_no_overflow_signal(self):- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- MAX_SUBTOPICS,- )+def test_fetch_path_strips_unvalidated_chunk_fragment_on_FIRST_registration():+ """``find_or_add_result``'s NEW-ENTRY branch validates the anchor.- captured = {}- subtopics = [f"topic {i}" for i in range(MAX_SUBTOPICS)]+ ``test_malformed_chunk_fragments_are_not_recorded`` uses these same+ payloads but seeds the document through ``add_results`` first, so it+ only ever drives the REUSE branch (which validates via+ ``_prefer_anchored_link``). The new-entry branch — the one the fetch+ tool actually hits on first registration, carrying the agent's raw+ ``fetch_content`` argument — was unguarded. Registering FIRST here,+ with no seeding, is the whole point of this test.+ """+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ SearchResultsCollector,+ )+ from local_deep_research.utilities.search_utilities import (+ format_links_to_markdown,+ )- result, _ = self._patched_run(- subtopics,- progress_callback=lambda *a: captured.update({"meta": a[2]}),+ route = "/library/document/doc1/chunks"+ for bad in (+ "#chunk-1'\"><img src=x>",+ "#chunk-" + "9" * 40,+ "#chunk-1000001",+ "#chunk-nope",+ ):+ collector = SearchResultsCollector()+ collector.find_or_add_result(+ {+ "title": "D",+ "link": route + bad,+ "url": route + bad,+ "source": "library",+ })+ stored = collector._all_links[0]+ assert stored["link"] == route, bad+ assert stored["url"] == route, bad+ assert bad not in format_links_to_markdown(collector._all_links), bad- for i in range(MAX_SUBTOPICS):- assert f"## topic {i}" in result- assert "overflow_strategy" not in captured["meta"]- assert "truncated_from" not in captured["meta"]- assert "Overflow handling:" not in result- def test_one_over_preferred_limit_queues_exactly_one(self):- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- MAX_SUBTOPICS,- )+def test_fetch_path_keeps_a_valid_chunk_anchor_on_first_registration():+ """The strip must not eat a legitimate anchor arriving by the same path."""+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ SearchResultsCollector,+ )- captured = {}- subtopics = [f"topic {i}" for i in range(MAX_SUBTOPICS + 1)]+ good = "/library/document/doc1/chunks#chunk-4"+ collector = SearchResultsCollector()+ collector.find_or_add_result(+ {"title": "D", "link": good, "url": good, "source": "library"}+ )+ assert collector._all_links[0]["link"] == good- result, _ = self._patched_run(- subtopics,- progress_callback=lambda *a: captured.update({"meta": a[2]}),- )- for i in range(MAX_SUBTOPICS + 1):- assert f"## topic {i}" in result- assert captured["meta"]["overflow_queued_count"] == 1- assert "Overflow handling: 1 subtopic(s)" in result+def test_non_chunk_fragments_are_left_alone_by_the_fetch_path():+ """Only ``#chunk-`` fragments are the collector's business — mirrors the+ ``add_results`` rule so the two siblings stay identical."""+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ SearchResultsCollector,+ )- def test_exactly_at_hard_limit_is_processed(self):- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- MAX_SUBTOPICS_HARD_LIMIT,- )+ url = "https://example.com/paper#section-3"+ collector = SearchResultsCollector()+ collector.find_or_add_result({"title": "P", "link": url, "url": url})+ assert collector._all_links[0]["link"] == url- subtopics = [f"topic {i}" for i in range(MAX_SUBTOPICS_HARD_LIMIT)]- result, _ = self._patched_run(subtopics)+def test_external_url_with_a_chunk_fragment_is_NOT_truncated():+ """The strip is scoped to LIBRARY routes, like its ``add_results`` sibling.- for topic in subtopics:- assert f"## {topic}" in result- assert "Overflow handling: 5 subtopic(s)" in result+ The first version of this guard fired on ``"#chunk-" in link`` alone.+ ``preferred_chunk_display`` returns None for every non-library URL, so+ an ordinary page anchored at ``#chunk-2`` had its fragment stripped —+ a citation truncated by a rule with no business reading it. The+ previous test used ``#section-3``, which cannot catch this: the bug+ needed a fragment that starts with ``chunk-`` on a NON-library URL.+ """+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ SearchResultsCollector,+ )- def test_above_hard_limit_rejects_whole_batch_before_agent_creation(self):- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- MAX_SUBTOPICS_HARD_LIMIT,+ for url in (+ "https://example.com/docs/page#chunk-2",+ "https://example.com/docs/page#chunk-99999999999",+ "https://evil.example/library/document/7/chunks#chunk-1",+ ):+ fetched = SearchResultsCollector()+ fetched.find_or_add_result({"title": "P", "link": url, "url": url})+ seeded = SearchResultsCollector()+ seeded.add_results([{"title": "P", "link": url, "url": url}])+ assert fetched._all_links[0]["link"] == url, url+ # The two ingest siblings must agree on scope, not merely on+ # outcome for library payloads.+ assert fetched._all_links[0]["link"] == seeded._all_links[0]["link"], (+ url)- progress_callback = MagicMock()- tool, preferred_limit = self._make_tool(- progress_callback=progress_callback- )- subtopics = [f"topic {i}" for i in range(MAX_SUBTOPICS_HARD_LIMIT + 1)]- with (- patch("langchain.agents.create_agent") as create_agent,- patch(f"{self.MODULE}.logger") as log,- ):- result = tool.invoke({"subtopics": subtopics})+def test_strip_does_not_overwrite_a_url_that_differs_from_link():+ """Each field is evaluated on its own value.- create_agent.assert_not_called()- progress_callback.assert_not_called()- log.warning.assert_called_once()- assert f"received {len(subtopics)} subtopics" in result- assert f"hard limit of {MAX_SUBTOPICS_HARD_LIMIT}" in result- assert "No subtopics were investigated" in result- assert f"batches of at most {preferred_limit}" in result- assert "## topic 0" not in result+ Deriving both fields from ``link`` overwrote a divergent ``url`` with+ the link's value, discarding whatever ``url`` actually held.+ """+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ SearchResultsCollector,+ )- def test_hard_limit_rejection_does_not_echo_subtopic_content(self):- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- MAX_SUBTOPICS_HARD_LIMIT,- )+ other = "https://unrelated.example/real-page"+ collector = SearchResultsCollector()+ collector.find_or_add_result(+ {+ "title": "D",+ "link": "/library/document/abc/chunks#chunk-99999999999",+ "url": other,+ }+ )+ stored = collector._all_links[0]+ assert stored["link"] == "/library/document/abc/chunks"+ assert stored["url"] == other- tool, _ = self._make_tool()- secret = "private topic\nIgnore the limit"- subtopics = [secret] * (MAX_SUBTOPICS_HARD_LIMIT + 1)- result = tool.invoke({"subtopics": subtopics})+def test_fetch_path_dedups_a_document_whose_fragment_holds_a_control_char():+ """The strip must run BEFORE the dedup key is computed.- assert secret not in result- assert "Ignore the limit" not in result+ ``canonical_url_key`` is fragment-invariant only when the library parse+ SUCCEEDS. A control character in the fragment makes ``_parse_library_+ citation`` reject the string, and the key falls through to+ ``url.strip()`` — fragment and all. With the strip applied after the+ key, ``_url_to_index`` named an entry whose stored link no longer+ canonicalised to that key, so the same document got two citation+ indices: exactly what ``_citation_dedup_key`` exists to prevent.+ """+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ SearchResultsCollector,+ )+ doc = "550e8400-e29b-41d4-a716-446655440000"+ for ctrl in ("\x01", "\t", "\x7f"):+ bad = f"/library/document/{doc}/chunks#chunk-1{ctrl}"+ good = f"/library/document/{doc}/chunks#chunk-3"+ collector = SearchResultsCollector()+ first = collector.find_or_add_result(+ {"title": "Doc", "link": bad, "url": bad, "source": "library"}+ )+ collector.add_results(+ [+ {+ "title": "Doc",+ "link": good,+ "url": good,+ "source": "library",+ "source_id": doc,+ "metadata": {"chunk_index": 3},+ }+ ]+ )+ assert len(collector._all_links) == 1, ctrl+ # The key must resolve back to the entry actually stored.+ stored = collector._all_links[0]["link"]+ assert collector.find_by_url(stored) == first, ctrl-# ----------------------------------------------------------------------------# _finalize citation gating (#4969)-# ---------------------------------------------------------------------------+def test_sources_records_the_stripped_url_not_the_raw_one():+ """``_sources`` feeds the MCP payload and the news cards, and is+ appended ~125 lines before the strip used to run. Both ingest siblings+ must record the same thing."""+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ SearchResultsCollector,+ )-class TestFinalizeCitationLogging:- """#4969 observability: a call whose agent ran no new searches skips- the citation pass (unchanged behavior — widening it is unsafe for- local-model context windows and chat follow-ups until redesigned),- but the skip and any marker-free synthesis must be loud in the log- instead of silently saving uncited prose."""+ route = "/library/document/abc/chunks"+ for bad in ("#chunk-<script>", "#chunk-99999999", "#chunk-nope"):+ fetched = SearchResultsCollector()+ fetched.find_or_add_result(+ {+ "title": "D",+ "link": route + bad,+ "url": route + bad,+ "source": "library",+ }+ )+ seeded = SearchResultsCollector()+ seeded.add_results(+ [+ {+ "title": "D",+ "link": route + bad,+ "url": route + bad,+ "source": "library",+ }+ ]+ )+ assert fetched.sources == [route], bad+ assert fetched.sources == seeded.sources, bad- _LOGGER_PATH = (- "local_deep_research.advanced_search_system.strategies."- "langgraph_agent_strategy.logger"++def test_a_library_FLAGGED_result_with_an_external_link_keeps_its_fragment():+ """``add_results`` gates on ``is_library_chunk_result``, which is also+ true for a result merely flagged ``source: "library"``. Without a+ library-LINK test on the strip itself, an external URL carrying+ ``#chunk-2`` on such a result lost its fragment — the over-broad strip+ that was fixed in the fetch sibling, still live in this one."""+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ SearchResultsCollector,)- def _make_strategy(self, all_links=None, citation_handler=None):- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- LangGraphAgentStrategy,+ ext = "https://example.com/p#chunk-2"+ for flag in ("source", "source_type"):+ fetched = SearchResultsCollector()+ fetched.find_or_add_result(+ {"title": "P", "link": ext, "url": ext, flag: "library"})-- return LangGraphAgentStrategy(- model=MagicMock(),- search=MagicMock(),- all_links_of_system=all_links if all_links is not None else [],- settings_snapshot={"search.tool": {"value": "duckduckgo"}},- citation_handler=citation_handler,+ seeded = SearchResultsCollector()+ seeded.add_results(+ [{"title": "P", "link": ext, "url": ext, flag: "library"}])+ assert fetched._all_links[0]["link"] == ext, flag+ assert seeded._all_links[0]["link"] == ext, flag- @staticmethod- def _link(idx, url):- return {- "index": str(idx),- "title": f"Source {idx}",- "link": url,- "snippet": "snippet",- }- def _warnings(self, mock_logger):- return [str(c.args[0]) for c in mock_logger.warning.call_args_list]+def test_alias_with_port_or_userinfo_still_has_its_anchor_validated():+ """``is_library_document_link`` is a ``startswith`` test that rejects the+ alias with a port or userinfo, while ``_normalize_library_alias``+ accepts both — so gating on the prefix test alone let those spellings+ carry an unvalidated anchor into ``select_source_url`` and the DB."""+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ SearchResultsCollector,+ )- def test_empty_collector_skips_pass_and_warns(self):- """Empty per-call collector + accumulated sources → the pass is- skipped (raw answer kept, handler untouched) and the skip is- logged as a warning naming the accumulated count."""- handler = MagicMock()- links = [self._link(1, "https://a.example/x")]- strategy = self._make_strategy(- all_links=links, citation_handler=handler+ for raw, expected in (+ (+ "https://library.document:443/abc/chunks#chunk-99999999",+ "https://library.document:443/abc/chunks",+ ),+ (+ "https://[email protected]/abc/chunks#chunk-<script>",+ "https://[email protected]/abc/chunks",+ ),+ ):+ collector = SearchResultsCollector()+ collector.find_or_add_result(+ {"title": "D", "link": raw, "url": raw, "source": "library"})- assert strategy.collector.results == []+ assert collector._all_links[0]["link"] == expected, raw- with patch(self._LOGGER_PATH) as mock_logger:- result = strategy._finalize("q", "Uncited raw answer.", 1, 0, [])- handler.analyze_followup.assert_not_called()- assert result["current_knowledge"] == "Uncited raw answer."- assert any(- "raw answer contains no inline [N] markers" in warning- for warning in self._warnings(mock_logger)- )+def test_both_ingest_siblings_apply_ONE_library_citation_rule():+ """The two ingest paths must ask the same question about the same value.- def test_empty_collector_reports_existing_markers(self):- """A raw answer that already cites prior context must not be- diagnosed as having no inline citations."""- handler = MagicMock()- links = [self._link(1, "https://a.example/x")]- strategy = self._make_strategy(- all_links=links, citation_handler=handler- )+ The alias gate was widened in the shared helper (used only by+ ``find_or_add_result``) while ``add_results`` kept the bare+ ``is_library_document_link`` ``startswith`` test — which rejects the+ alias with a port or userinfo that ``_normalize_library_alias``+ accepts. So those spellings were validated on one path and stored+ verbatim on the other, reaching ``_sources`` and the DB with the+ fragment intact. Both now call ``_is_library_citation``.+ """+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ SearchResultsCollector,+ )- with patch(self._LOGGER_PATH) as mock_logger:- result = strategy._finalize(- "q", "Prior evidence [1] remains relevant [2, 3].", 1, 0, []- )+ cases = [+ # (raw, expected stored link)+ (+ "https://[email protected]/abc/chunks#chunk-<script>",+ "https://[email protected]/abc/chunks",+ ),+ (+ "https://library.document:443/abc/chunks#chunk-99999999",+ "https://library.document:443/abc/chunks",+ ),+ (+ "/library/document/abc/chunks#chunk-99999999",+ "/library/document/abc/chunks",+ ),+ # A valid anchor survives on BOTH paths: the rebuild branch prefers+ # metadata, but with none supplied a well-formed anchor is better+ # than none, and stripping it on one path only made the siblings+ # disagree where a real citation had something to lose.+ (+ "/library/document/abc/chunks#chunk-4",+ "/library/document/abc/chunks#chunk-4",+ ),+ # Not a library citation on either path: untouched.+ ("https://example.com/p#chunk-2", "https://example.com/p#chunk-2"),+ ]+ for raw, expected in cases:+ fetched = SearchResultsCollector()+ fetched.find_or_add_result(+ {"title": "D", "link": raw, "url": raw, "source": "library"}+ )+ seeded = SearchResultsCollector()+ seeded.add_results(+ [{"title": "D", "link": raw, "url": raw, "source": "library"}]+ )+ assert fetched._all_links[0]["link"] == expected, raw+ assert seeded._all_links[0]["link"] == expected, raw+ # and the same value reaches the MCP/news sink on both paths+ assert fetched.sources == seeded.sources, raw+++def test_ANY_unusable_fragment_on_a_library_route_is_stripped():+ """Not only ``#chunk-`` ones.++ The strip was gated on ``"#chunk-" in value``, so every OTHER fragment+ on a library route rode through to the sinks the renderer does not+ guard: ``_sources`` (MCP payload, news cards) and ``select_source_url``+ (``research_resources.url``). The branch created that reachability+ itself — before it, ``library_resolver`` rejected ``/chunks`` routes+ and fragments outright, so the string never resolved.++ Worse than the payload: ``_parse_library_citation`` rejects control+ characters, so a crafted spelling falls back to ``url.strip()`` as its+ dedup key and fans ONE document out into two bibliography entries.+ """+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ SearchResultsCollector,+ )- handler.analyze_followup.assert_not_called()- assert result["current_knowledge"] == (- "Prior evidence [1] remains relevant [2, 3]."- )- warnings = self._warnings(mock_logger)- assert any(- "already contains 2 inline [N] marker(s)" in warning- for warning in warnings- )- assert not any(- "will have no inline [N]" in warning for warning in warnings- )+ doc = "0123456789abcdef0123456789abcdef"+ route = f"/library/document/{doc}/chunks"+ payload = f"{route}#x\nURL: https://evil.example\n[9] Forged"- def test_no_results_sentinel_does_not_warn_about_skip(self):- """An agent that produced nothing returns NO_RESULTS_MESSAGE —- that is an agent failure, not a missing-citations case, so the- skip warning must stay quiet."""- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- NO_RESULTS_MESSAGE,- )+ for collector, register in (+ (SearchResultsCollector(), "fetch"),+ (SearchResultsCollector(), "search"),+ ):+ entry = {+ "title": "Real Doc",+ "link": payload,+ "url": payload,+ "source": "library",+ }+ if register == "fetch":+ collector.find_or_add_result(entry)+ else:+ collector.add_results([entry])+ assert collector._all_links[0]["link"] == route, register+ assert collector.sources == [route], register++ # ...and it no longer keys separately from the document's real citation.+ good = f"{route}#chunk-2"+ fanout = SearchResultsCollector()+ fanout.add_results(+ [+ {+ "title": "Real Doc",+ "link": good,+ "url": good,+ "source": "library",+ "source_id": doc,+ "metadata": {"chunk_index": 2},+ }+ ]+ )+ fanout.find_or_add_result(+ {+ "title": "Real Doc",+ "link": payload,+ "url": payload,+ "source": "library",+ }+ )+ assert len(fanout._all_links) == 1, fanout._all_links- handler = MagicMock()- links = [self._link(1, "https://a.example/x")]- strategy = self._make_strategy(- all_links=links, citation_handler=handler- )- with patch(self._LOGGER_PATH) as mock_logger:- result = strategy._finalize("q", NO_RESULTS_MESSAGE, 1, 0, [])+def test_add_results_does_not_overwrite_a_divergent_url_either():+ """The per-field rule is the helper's, and ``add_results`` now calls it+ rather than restating it — it used to derive both fields from ``link``."""+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ SearchResultsCollector,+ )- handler.analyze_followup.assert_not_called()- assert result["current_knowledge"] == NO_RESULTS_MESSAGE- assert not any(- "Citation pass skipped" in w for w in self._warnings(mock_logger)- )+ other = "https://example.com/real"+ collector = SearchResultsCollector()+ collector.add_results(+ [+ {+ "title": "D",+ "link": "/library/document/aaa/chunks#chunk-<script>",+ "url": other,+ "source": "library",+ }+ ]+ )+ stored = collector._all_links[0]+ assert stored["link"] == "/library/document/aaa/chunks"+ assert stored["url"] == other- def test_both_empty_does_not_warn(self):- """No sources anywhere → nothing to cite, no warning noise."""- handler = MagicMock()- strategy = self._make_strategy(all_links=[], citation_handler=handler)- with patch(self._LOGGER_PATH) as mock_logger:- result = strategy._finalize("q", "Raw answer.", 1, 0, [])+def test_seeded_entries_are_normalised_like_the_other_two_paths():+ """Constructor seeding is a THIRD ingest path.- handler.analyze_followup.assert_not_called()- assert result["current_knowledge"] == "Raw answer."- assert not any(- "Citation pass skipped" in w for w in self._warnings(mock_logger)- )+ ``__init__`` already normalised seeded entries for one hazard (it pops+ the producer-supplied ``CHUNK_DISPLAY_KEY``) while leaving the+ fragment, which is the same hazard half-handled. A seeded entry+ rendered its raw fragment into the Sources block, and because the+ dedup key is computed from the link, a fragment the library parser+ rejects keyed off the raw string — so the document fanned out into a+ second bibliography entry as soon as it was also fetched.- def test_populated_collector_runs_pass_unchanged(self):- """Per-call results present → citation pass runs exactly as- before, with the per-call list."""- handler = MagicMock()- handler.analyze_followup.return_value = {- "content": "Cited [2].",- "documents": [],- }- links = [self._link(1, "https://a.example/x")]- strategy = self._make_strategy(- all_links=links, citation_handler=handler- )- strategy.collector.add_results(- [{"title": "New", "link": "https://b.example/y", "snippet": "s"}],- engine_name="web",- )+ Dead in-tree today (every caller seeds an empty list), but+ ``all_links_of_system`` is a documented constructor parameter.+ """+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ SearchResultsCollector,+ )+ from local_deep_research.utilities.search_utilities import (+ format_links_to_markdown,+ )- result = strategy._finalize("q", "raw", 1, 1, [])+ doc = "0123456789abcdef0123456789abcdef"+ route = f"/library/document/{doc}/chunks"+ payload = f"{route}#x\nURL: https://evil.example\n[9] Forged"- passed_sources = handler.analyze_followup.call_args.args[1]- assert [r["link"] for r in passed_sources] == ["https://b.example/y"]- assert result["current_knowledge"] == "Cited [2]."+ collector = SearchResultsCollector(+ all_links=[+ {+ "title": "D",+ "link": payload,+ "url": payload,+ "index": "1",+ "source": "library",+ }+ ]+ )+ assert collector._all_links[0]["link"] == route- def test_zero_marker_synthesis_logs_warning(self):- """If the citation pass ran but its output carries no [N]- markers, that must be visible in the server log."""- handler = MagicMock()- handler.analyze_followup.return_value = {- "content": "Still no markers at all.",- "documents": [],- }- strategy = self._make_strategy(all_links=[], citation_handler=handler)- strategy.collector.add_results(- [{"title": "New", "link": "https://b.example/y", "snippet": "s"}],- engine_name="web",- )+ # ...and it dedups against the same document fetched afterwards.+ good = f"{route}#chunk-2"+ collector.find_or_add_result(+ {"title": "D", "link": good, "url": good, "source": "library"}+ )+ assert len(collector._all_links) == 1, collector._all_links+ assert "evil.example" not in format_links_to_markdown(collector._all_links)- with patch(self._LOGGER_PATH) as mock_logger:- strategy._finalize("q", "raw", 1, 0, [])+ # A VALID seeded anchor is preserved, as on the other two paths.+ valid = f"{route}#chunk-4"+ kept = SearchResultsCollector(+ all_links=[{"title": "D", "link": valid, "url": valid, "index": "1"}]+ )+ assert kept._all_links[0]["link"] == valid- assert any(- "no inline [N] citation markers" in w- for w in self._warnings(mock_logger)- )- def test_marker_bearing_synthesis_does_not_warn(self):- handler = MagicMock()- handler.analyze_followup.return_value = {- "content": "Cited [1] properly.",- "documents": [],- }- strategy = self._make_strategy(all_links=[], citation_handler=handler)- strategy.collector.add_results(- [{"title": "New", "link": "https://b.example/y", "snippet": "s"}],- engine_name="web",- )+def test_alias_with_port_AND_a_control_char_is_stripped_on_all_three_paths():+ """The spelling that missed BOTH arms of ``_is_library_citation``.- with patch(self._LOGGER_PATH) as mock_logger:- strategy._finalize("q", "raw", 1, 0, [])+ ``is_library_document_link`` was a literal ``startswith`` test, so the+ ``:443``/userinfo forms missed it; ``library_display_url`` refuses any+ string containing a control character BEFORE it normalises the alias,+ so those missed it too. A URL with both therefore looked like "not a+ library citation" and kept its fragment on every ingest path — while+ ``canonical_url_key`` still keyed it as a library route, fanning one+ document into two bibliography entries.+ """+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ SearchResultsCollector,+ )- assert not any(- "no inline [N] citation markers" in w- for w in self._warnings(mock_logger)+ doc = "0123456789abcdef0123456789abcdef"+ for raw, expected in (+ (+ f"https://library.document:443/{doc}/chunks#\x01evil",+ f"https://library.document:443/{doc}/chunks",+ ),+ (+ f"https://[email protected]/{doc}/chunks#\x01evil",+ f"https://[email protected]/{doc}/chunks",+ ),+ ):+ seeded = SearchResultsCollector(+ all_links=[{"title": "D", "link": raw, "url": raw, "index": "1"}])+ assert seeded._all_links[0]["link"] == expected, raw- def test_milestone_skipped_for_no_results_sentinel(self):- """NO_RESULTS_MESSAGE sentinel must suppress the progress milestone completely."""- from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (- NO_RESULTS_MESSAGE,+ fetched = SearchResultsCollector()+ fetched.find_or_add_result(+ {"title": "D", "link": raw, "url": raw, "source": "library"})+ assert fetched._all_links[0]["link"] == expected, raw- strategy = self._make_strategy(all_links=[])- progress_updates = []- strategy.set_progress_callback(- lambda msg, pct, meta: progress_updates.append((msg, pct, meta))+ searched = SearchResultsCollector()+ searched.add_results(+ [{"title": "D", "link": raw, "url": raw, "source": "library"}])+ assert searched._all_links[0]["link"] == expected, raw- strategy._finalize("q", NO_RESULTS_MESSAGE, 1, 0, [])- # The synthesis milestone (progress 90) must not be emitted- synthesis_updates = [- u for u in progress_updates if u[2].get("phase") == "synthesis"- ]- assert len(synthesis_updates) == 0+def test_sources_payload_flattens_line_breaking_chars_in_ANY_url():+ """``_sources`` feeds the MCP payload and the news cards, neither of+ which re-normalises.- def test_milestone_describes_accumulated_sources_when_new_empty(self):- """Empty per-call collector but accumulated sources present -> show accumulated sources."""- links = [self._link(1, "https://a.example/x")]- strategy = self._make_strategy(all_links=links)- progress_updates = []- strategy.set_progress_callback(- lambda msg, pct, meta: progress_updates.append((msg, pct, meta))- )+ The bibliography flattens line-breaking characters at render, and+ library routes carrying them are refused upstream — but an ORDINARY+ external URL was recorded verbatim, so a U+2028 reached those+ consumers able to forge a line. This was the one sink the branch's+ sanitisation theme never covered.+ """+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ SearchResultsCollector,+ )- strategy._finalize("q", "prose", 1, 0, [])+ forged = "https://example.com/p URL: https://evil [9] Forged"+ flat = "https://example.com/p URL: https://evil [9] Forged"- synthesis_updates = [- u for u in progress_updates if u[2].get("phase") == "synthesis"- ]- assert len(synthesis_updates) == 1- msg, pct, meta = synthesis_updates[0]- assert (- "Skipping citation synthesis (reusing 1 accumulated sources)" in msg- )- assert meta.get("citation_pass_skipped") is True- assert meta.get("accumulated_sources") == 1+ fetched = SearchResultsCollector()+ fetched.find_or_add_result({"title": "P", "link": forged, "url": forged})+ assert fetched.sources == [flat]- def test_milestone_both_empty_emits_followup(self):- """Both collectors empty -> emit only the fallback explanation milestone."""- strategy = self._make_strategy(all_links=[])- progress_updates = []- strategy.set_progress_callback(- lambda msg, pct, meta: progress_updates.append((msg, pct, meta))- )+ seeded = SearchResultsCollector()+ seeded.add_results([{"title": "P", "link": forged, "url": forged}])+ assert seeded.sources == [flat]- strategy._finalize("q", "prose", 1, 0, [])+ # Every line-breaking character the renderer flattens, not just .+ for ch in ("\n", "\r", " ", "\x85"):+ c = SearchResultsCollector()+ u = f"https://example.com/a{ch}b"+ c.find_or_add_result({"title": "P", "link": u, "url": u})+ assert c.sources == ["https://example.com/a b"], repr(ch)- synthesis_updates = [- u for u in progress_updates if u[2].get("phase") == "synthesis"- ]- assert len(synthesis_updates) == 1- assert (- "No sources available for citation synthesis"- in synthesis_updates[0][0]++def test_sources_payload_does_not_mangle_ordinary_urls():+ """Flattening must be a no-op for anything legitimate — query strings,+ fragments and valid chunk anchors all survive intact."""+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ SearchResultsCollector,+ )++ for url in (+ "https://example.com/p?a=1&b=2#frag",+ "https://example.com/path/with%20escapes",+ "/library/document/abc/chunks#chunk-3",+ ):+ collector = SearchResultsCollector()+ collector.find_or_add_result(+ {"title": "P", "link": url, "url": url, "source": "library"})+ assert collector.sources == [url], urldiff --git a/tests/text_optimization/test_bibliography_escaping.py b/tests/text_optimization/test_bibliography_escaping.pynew file mode 100644index 0000000000..9940bfb808--- /dev/null+++ b/tests/text_optimization/test_bibliography_escaping.py@@ -0,0 +1,144 @@+"""Escaping contracts for the bibliography exporters.++These pin behaviour that regressed once already: an earlier revision escaped+BibTeX braces as ``\\{``/``\\}``, which is a LaTeX escape. BibTeX's field+scanner counts brace CHARACTERS and has no escape mechanism, so the value+stayed unbalanced and the scanner ran to EOF, swallowing the rest of the+file. Nothing asserted the contract, so it shipped.+"""++from __future__ import annotations++import itertools++import pytest++from local_deep_research.text_optimization.citation_formatter import (+ _escape_bibtex,+ _safe_bibtex_url,+ is_line_breaking_char,+)+from local_deep_research.utilities.search_utilities import (+ _sanitize_sources_field,+)+++def _scan_quoted_value(text: str) -> str:+ """Model BibTeX's scanner over ``title = "<text>"``.++ The value ends at the first ``"`` seen at brace depth 0. Returns the+ consumed value, or a marker for the two failure modes.+ """+ depth = 0+ for position, char in enumerate(text):+ if char == "{":+ depth += 1+ elif char == "}":+ depth -= 1+ if depth < 0:+ return "UNBALANCED"+ elif char == '"' and depth == 0:+ return text[:position]+ return "RUNAWAY"+++ "hostile",+ [+ 'Benign}", url = "http://evil.example", note = "{x',+ "Benign{X",+ "Benign}X",+ "{{{{",+ "}}}}",+ "\\textbraceleft",+ "a\\b",+ "100% of $5 & #1 ~ ^2 _x",+ ],+)+def test_escaped_title_terminates_exactly_at_the_closing_quote(hostile):+ """No hostile title may end the field early or run to EOF."""+ field = f'{_escape_bibtex(hostile)}", year = {{2024}}'+ assert _scan_quoted_value(field) == _escape_bibtex(hostile)+++def test_no_short_string_of_brace_and_quote_chars_breaks_the_field():+ """Exhaustive over the characters that could break out."""+ for length in range(1, 4):+ for combo in itertools.product('{}"\\', repeat=length):+ hostile = "".join(combo)+ escaped = _escape_bibtex(hostile)+ field = f'{escaped}", year = {{2024}}'+ assert _scan_quoted_value(field) == escaped, hostile+++def test_escaped_title_is_brace_balanced():+ for hostile in ("{", "}", "{}", "}{", "a{b}c", '"{'):+ escaped = _escape_bibtex(hostile)+ assert escaped.count("{") == escaped.count("}"), hostile+++ "code",+ [+ 0x00,+ 0x0D, # CR — closes the ```bibtex fence in the Quarto export+ 0x1F,+ 0x7F,+ 0x85, # NEL+ 0x9B, # C1+ 0x2028, # LINE SEPARATOR+ 0x2029, # PARAGRAPH SEPARATOR+ ],+)+def test_escape_bibtex_flattens_line_breaking_characters(code):+ """The set must match ``_sanitize_sources_field``.++ Fixing CR alone was not enough: the .bib path is reachable without+ going through the Sources renderer, so every character either function+ treats as a break has to be handled by both.+ """+ char = chr(code)+ assert char not in _escape_bibtex(f"Pwn{char}```bibtex{char}junk")+ assert _safe_bibtex_url(f"http://a.test/{char}") == ""+++def test_unsafe_urls_are_omitted_not_repaired():+ """Dropping characters can rewrite a URL's host.++ Removing the backslash from the first case turns the trusted host into+ userinfo, so the export would link somewhere the reader never saw.+ """+ assert _safe_bibtex_url("//trusted.example.com\\@evil.example/pwn") == ""+ assert _safe_bibtex_url("/docs/a b/c") == ""+ assert _safe_bibtex_url("https://real.example/p?a=1#frag") == (+ "https://real.example/p?a=1#frag"+ )+++def test_both_producers_agree_on_every_bmp_codepoint():+ """The Sources renderer and the .bib escaper must not drift.++ They were previously two hand-rolled expressions differing on 31+ codepoints while a comment claimed they matched "exactly". Comparing+ behaviour rather than reading the source is the only way that claim+ stays true.+ """+ disagreements = [+ code+ for code in range(0x10000)+ if (_sanitize_sources_field(f"a{chr(code)}b") != f"a{chr(code)}b")+ != is_line_breaking_char(chr(code))+ ]+ assert disagreements == []+++def test_legitimate_non_ascii_titles_are_untouched():+ """Widening the control set must not reject real text."""+ for title in (+ "Étude sur les systèmes",+ "汉字标题 — 测试",+ "Ελληνικά",+ "naïve café",+ ):+ assert _escape_bibtex(title) == title+ assert _sanitize_sources_field(title) == titlediff --git a/tests/text_optimization/test_citation_formatter_bibliography.py b/tests/text_optimization/test_citation_formatter_bibliography.pynew file mode 100644index 0000000000..968fac9268--- /dev/null+++ b/tests/text_optimization/test_citation_formatter_bibliography.py@@ -0,0 +1,60 @@+"""Bibliography export of MERGED Sources lines.++``format_links_to_markdown`` collapses every citation of one source onto a+single line carrying all of its indices. Library documents cited across+several chunks hit this routinely (#5685), so an exporter that matches only+single-index lines drops those sources from the bibliography entirely.+"""++from __future__ import annotations++_CONTENT = """# Report++### SOURCES USED IN THIS SECTION:+[1, 3] Doc A (source nr: 1, 3)+ URL: /library/document/a/chunks#chunk-0++[2] Doc B+ URL: https://e.com/b++## ALL SOURCES:+[1, 3] Doc A (source nr: 1, 3)+ URL: /library/document/a/chunks#chunk-0++[2] Doc B+ URL: https://e.com/b+"""+++def test_bibtex_keeps_merged_sources_without_duplicate_keys():+ from local_deep_research.text_optimization.citation_formatter import (+ QuartoExporter,+ )++ bib = QuartoExporter()._generate_bibliography(_CONTENT)+ keys = [ln for ln in bib.splitlines() if ln.startswith("@misc")]++ # The merged source survives...+ assert "Doc A" in bib+ assert keys == ["@misc{ref1,", "@misc{ref2,", "@misc{ref3,"]+ # ...and the Sources block appearing twice (per-section plus the+ # combined block) must not yield duplicate keys, which bibtex rejects.+ assert len(keys) == len(set(keys))+++def test_latex_bibitems_are_ordered_by_index():+ from local_deep_research.text_optimization.citation_formatter import (+ LaTeXExporter,+ )++ tex = LaTeXExporter()._create_bibliography(_CONTENT)+ items = [+ ln.split(" ")[0]+ for ln in tex.splitlines()+ if ln.startswith("\\bibitem")+ ]++ assert "Doc A" in tex+ # thebibliography numbers by POSITION, so out-of-order keys make+ # \\cite{3} print [2]. A merged [1, 3] group must not disturb that.+ assert items == ["\\bibitem{1}", "\\bibitem{2}", "\\bibitem{3}"]diff --git a/tests/utilities/test_chunk_anchor.py b/tests/utilities/test_chunk_anchor.pyindex 6a6c017715..f9d3539668 100644--- a/tests/utilities/test_chunk_anchor.py+++ b/tests/utilities/test_chunk_anchor.py@@ -325,3 +325,196 @@ class TestProducerCollectorTemplateContract:link = collector.results[0]["link"]assert "#chunk-" not in linkassert link == "/library/document/doc1/chunks"+++def test_build_chunk_anchor_url_rejects_unsafe_document_ids():+ """The function BUILDS the URL, so it must enforce the doc-id contract+ itself rather than trusting the caller. Every in-tree caller happens to+ pre-sanitise, which is why this went uncovered — a future one may not."""+ from local_deep_research.utilities.chunk_anchor import (+ build_chunk_anchor_url,+ )++ for bad in ("../../secrets", "doc#1", "a/b", "a b", "", "a?q=1"):+ assert build_chunk_anchor_url("/x", bad, 5) is None, bad+ # Leading/trailing whitespace is stripped, not smuggled into the URL.+ assert (+ build_chunk_anchor_url("/x", "\r\nok-id_9 ", 0)+ == "/library/document/ok-id_9/chunks#chunk-0"+ )+++def test_build_chunk_anchor_url_accepts_the_same_types_as_extract():+ """The two share ``is_safe_document_id`` precisely so they cannot+ disagree; bool and non-str/int objects must be rejected by both."""+ import uuid++ from local_deep_research.utilities.chunk_anchor import (+ build_chunk_anchor_url,+ extract_document_id,+ )++ for value in (True, False, uuid.uuid4(), 1.5, ["x"]):+ assert build_chunk_anchor_url("/x", value, 0) is None, value+ assert extract_document_id({"doc_id": value}) is None, value+ # int (not bool) is accepted by both.+ assert extract_document_id({"doc_id": 7}) == "7"+ assert (+ build_chunk_anchor_url("/x", 7, 0)+ == "/library/document/7/chunks#chunk-0"+ )+++def test_extract_chunk_index_requires_ascii_digits_and_a_sane_bound():+ """``str.isdigit()`` is Unicode-wide and ``str.strip()`` undid the+ "whitespace is rejected" promise, so " 5 ", the Arabic-Indic "٧"+ and the mathematical "\U0001d7dd" all produced anchors. Unbounded+ magnitudes did too — including ``1e30``, which ``int()`` silently+ rounds to 1000000000000000019884624838656."""+ from local_deep_research.utilities.chunk_anchor import (+ MAX_CHUNK_INDEX,+ extract_chunk_index,+ )++ for bad in (+ " 5 ",+ "5 ",+ "\t5",+ "٧",+ "\U0001d7dd",+ "5",+ 10**30,+ 1e30,+ MAX_CHUNK_INDEX + 1,+ float(MAX_CHUNK_INDEX + 1),+ ):+ assert extract_chunk_index({"chunk_index": bad}) is None, repr(bad)++ # Everything previously accepted still is — including the 0 boundary,+ # which must stay distinct from "no anchor".+ assert extract_chunk_index({"chunk_index": 0}) == 0+ assert extract_chunk_index({"chunk_index": "0"}) == 0+ assert extract_chunk_index({"chunk_index": 0.0}) == 0+ assert extract_chunk_index({"chunk_index": 5}) == 5+ assert extract_chunk_index({"chunk_index": "5"}) == 5+ assert extract_chunk_index({"chunk_index": MAX_CHUNK_INDEX})+ assert extract_chunk_index({"chunk_id": "12"}) == 12+++def test_is_safe_document_id_is_ascii_only():+ """``str.isalnum()`` accepts every Unicode letter/digit, so a fullwidth+ id was emitted unencoded into a URL path and then keyed separately from+ its ``%EF%BC%97`` form — one document, two bibliography entries."""+ from local_deep_research.utilities.chunk_anchor import (+ build_chunk_anchor_url,+ is_safe_document_id,+ )++ for bad in ("7", "١", "café", "²", "a b", "a/b", ""):+ assert is_safe_document_id(bad) is False, repr(bad)+ assert build_chunk_anchor_url("/x", bad, 0) is None, repr(bad)++ for good in ("7", "abc-123", "a_b", "0123456789abcdef" * 2):+ assert is_safe_document_id(good) is True, good+++def test_is_library_chunk_result_is_anchored_at_the_url_start():+ """An unanchored ``"/library/document/" in link`` also matched+ ``https://evil.example/library/document/7/chunks`` — and the collector+ reacts by REPLACING the link with a local route, silently relabelling+ an external result as a document in the user's own library."""+ from local_deep_research.utilities.chunk_anchor import (+ is_library_chunk_result,+ )++ for external in (+ "https://evil.example/library/document/7/chunks",+ "https://evil.example/?next=/library/document/7",+ "http://library.document.evil.example/7/chunks",+ ):+ assert (+ is_library_chunk_result({"link": external, "chunk_index": 3})+ is False+ ), external++ for internal in (+ "/library/document/7/chunks",+ "/lib/document/7",+ "https://library.document/7/chunks#chunk-3",+ ):+ assert is_library_chunk_result({"link": internal}) is True, internal+ # The explicit source markers still win regardless of the link.+ assert is_library_chunk_result({"source": "library", "link": ""}) is True+++def test_external_library_lookalike_is_not_rewritten_by_the_collector():+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ SearchResultsCollector,+ )++ all_links = []+ collector = SearchResultsCollector(all_links)+ collector.add_results(+ [+ {+ "title": "Evil",+ "link": "https://evil.example/library/document/7/chunks",+ "metadata": {"doc_id": "7", "chunk_index": 3},+ }+ ],+ engine_name="web",+ )++ assert (+ all_links[0]["link"] == "https://evil.example/library/document/7/chunks"+ )+++class TestLibraryDocumentLinkAliasHost:+ """``is_library_document_link`` matches the alias on HOST, not on a+ literal ``https://library.document/`` prefix.++ The prefix form missed the ``:443`` and userinfo spellings that+ ``_normalize_library_alias`` accepts as the same document. Combined+ with a control character — which makes the PARSER refuse the string+ too — such a URL missed both arms of ``_is_library_citation``, so its+ unvalidated fragment rode through all three collector ingest paths+ into the DB and the MCP payload.+ """++ def test_alias_spellings_that_name_the_same_document_are_accepted(self):+ from local_deep_research.utilities.chunk_anchor import (+ is_library_document_link,+ )++ for url in (+ "https://library.document/abc/chunks",+ "https://library.document:443/abc/chunks",+ "https://[email protected]/abc/chunks",+ "https://u:[email protected]:443/abc/chunks",+ "HTTPS://LIBRARY.DOCUMENT/abc",+ " https://library.document/abc ",+ ):+ assert is_library_document_link(url), url++ def test_look_alike_hosts_are_still_refused(self):+ """The caller REPLACES a matching link with a local route, so a+ false positive relabels an external page as the user's own+ document."""+ from local_deep_research.utilities.chunk_anchor import (+ is_library_document_link,+ )++ for url in (+ "https://library.document.evil.test/abc/chunks",+ "https://xlibrary.document/abc",+ "https://evil.example/library/document/7/chunks",+ "http://library.document/abc", # scheme matters+ "https://library.document", # authority with no path+ # urlsplit DELETES these, which would let a smuggled host+ # answer for the real one; the manual parse must not.+ "https://library.doc\tument/abc",+ "https://library.doc\nument/abc",+ "https://[email protected]/abc",+ ):+ assert not is_library_document_link(url), urldiff --git a/tests/utilities/test_search_utilities_formatting.py b/tests/utilities/test_search_utilities_formatting.pyindex 8b6371cbbc..4b5b0791c3 100644--- a/tests/utilities/test_search_utilities_formatting.py+++ b/tests/utilities/test_search_utilities_formatting.py@@ -862,8 +862,401 @@ class TestFormatLinksToMarkdownCollections:)assert LDR_APPENDED_SOURCES_SENTINEL not in sources- assert "Collection: My Papers\n\nArchive" in sources+ # The name is also flattened onto one line: newlines in a rendered+ # Sources field are what let a crafted value forge an extra entry,+ # so they are replaced with spaces rather than preserved. That is+ # strictly stronger than this test's own premise — the name can no+ # longer introduce line structure at all.+ assert "Collection: My Papers Archive" in sources+ assert "\n" not in sources.split("Collection: ", 1)[1].split("\n")[0]assert find_sources_section(content) == (content.index(LDR_APPENDED_SOURCES_SENTINEL),True,)+++def test_format_links_collapses_library_chunk_anchors_to_one_entry():+ """Regression for the #5381 follow-up: eight cited chunks of one library+ document must render as ONE Sources line carrying all eight indices, not+ eight near-identical lines. ``canonical_url_key`` only stripped fragments+ from absolute URLs, so root-relative chunk anchors each became their own+ group."""+ from local_deep_research.utilities.search_utilities import (+ format_links_to_markdown,+ )+ from local_deep_research.utilities.url_utils import canonical_url_key++ canonical_url_key.cache_clear()+ # Distinct per-chunk titles and a SECOND document: without both, the+ # test would pass even if grouping were keyed on title rather than on+ # the canonical URL, which is the thing actually under test.+ links = [+ {+ "title": f"My Paper (chunk {i})",+ "url": f"/library/document/doc1/chunks#chunk-{i}",+ "index": i + 1,+ }+ for i in range(8)+ ]+ links.append(+ {+ "title": "Other Paper",+ "url": "/library/document/doc2/chunks#chunk-0",+ "index": 9,+ }+ )++ out = format_links_to_markdown(links)++ # doc1's eight chunks collapse to one entry carrying all eight indices,+ # keyed by document despite the eight different titles.+ assert "[1, 2, 3, 4, 5, 6, 7, 8]" in out+ assert out.count("/library/document/doc1") == 1+ # ...and the entry keeps a usable anchor: collapsing must not strip the+ # #chunk-<n> that makes the citation scroll to the cited chunk.+ assert "/library/document/doc1/chunks#chunk-0" in out+ # doc2 is a genuinely different document and must NOT be merged into it.+ assert "/library/document/doc2/chunks#chunk-0" in out+ assert "[9]" in out+++def test_format_links_collapses_pdf_and_chunks_views_of_one_document():+ """A document whose chunks have mixed metadata yields both a+ ``/chunks#chunk-N`` and a ``/pdf`` citation URL (``_get_document_url``+ falls back to ``/pdf`` when ``chunk_index`` is missing). Those are one+ source and must not occupy two bibliography entries."""+ from local_deep_research.utilities.search_utilities import (+ format_links_to_markdown,+ )+ from local_deep_research.utilities.url_utils import canonical_url_key++ canonical_url_key.cache_clear()+ out = format_links_to_markdown(+ [+ {+ "title": "My Paper",+ "url": "/library/document/doc1/chunks#chunk-0",+ "index": 1,+ },+ {+ "title": "My Paper",+ "url": "/library/document/doc1/pdf",+ "index": 2,+ },+ ]+ )++ assert "[1, 2]" in out+ assert out.count("My Paper") == 1+ # The merged entry must still carry the anchor, not degrade to the+ # anchor-less /pdf view.+ assert "/library/document/doc1/chunks#chunk-0" in out+++def test_format_links_merges_absolute_library_alias_with_relative_route():+ """A document cited once by the RAG engine (relative route) and once via+ fetch_content on the absolute alias is ONE source. The fetch resolver+ registers the agent's original alias string verbatim, so without alias+ normalization this fans out into two entries and the alias one loses its+ anchor."""+ from local_deep_research.utilities.search_utilities import (+ format_links_to_markdown,+ )+ from local_deep_research.utilities.url_utils import canonical_url_key++ canonical_url_key.cache_clear()+ # The ANCHOR-LESS view is registered first on purpose. With the+ # anchored relative hit first, the anchor-preference rule would supply+ # the anchor regardless, and a broken alias path in+ # ``library_display_url`` would go undetected.+ out = format_links_to_markdown(+ [+ {+ "title": "My Paper",+ "url": "/library/document/abc123/pdf",+ "index": 1,+ },+ {+ "title": "My Paper",+ "url": "https://library.document/abc123/chunks#chunk-7",+ "index": 2,+ },+ ]+ )++ assert "[1, 2]" in out+ assert out.count("My Paper") == 1+ # The anchor can only have come from the alias, via normalization.+ assert "/library/document/abc123/chunks#chunk-7" in out+ assert "https://library.document" not in out+++def test_format_links_prefers_anchored_url_regardless_of_order():+ """The anchor must survive even when a chunk-less view of the document+ is registered FIRST — views key together, so plain first-seen ordering+ would display `/pdf` and silently drop the scroll-to-chunk target."""+ from local_deep_research.utilities.search_utilities import (+ format_links_to_markdown,+ )+ from local_deep_research.utilities.url_utils import canonical_url_key++ canonical_url_key.cache_clear()+ out = format_links_to_markdown(+ [+ {+ "title": "My Paper",+ "url": "/library/document/doc1/pdf",+ "index": 1,+ },+ {+ "title": "My Paper",+ "url": "/library/document/doc1/chunks#chunk-4",+ "index": 2,+ },+ ]+ )++ assert "[1, 2]" in out+ assert "/library/document/doc1/chunks#chunk-4" in out+++def test_format_links_survives_non_ascii_digit_index():+ """``"²".isdigit()`` is True while ``int("²")`` raises, which took down+ the whole bibliography. The repo's Hypothesis fuzzer only finds this+ from a cached example database, so pin it deterministically."""+ from local_deep_research.utilities.search_utilities import (+ format_links_to_markdown,+ )++ # Only superscripts qualify: int("١٢٣") == 123 (Arabic-Indic digits are+ # decimal), so that input never hit the old int() failure.+ for index in ("²", "¹"):+ out = format_links_to_markdown(+ [{"title": "t", "url": "https://x.test/p", "index": index}]+ )+ assert index in out+++def test_format_links_prefers_the_chunk_anchor_over_any_other_fragment():+ """The display preference tested for ``"#" in display`` rather than+ ``"#chunk-" in display``, so an empty or unrelated fragment counted as+ "anchored", won, and permanently locked out the real chunk anchor."""+ from local_deep_research.utilities.search_utilities import (+ format_links_to_markdown,+ )+ from local_deep_research.utilities.url_utils import canonical_url_key++ for decoy in ("/library/document/9#", "/library/document/9#section-intro"):+ canonical_url_key.cache_clear()+ out = format_links_to_markdown(+ [+ {"title": "Doc", "url": decoy, "index": 1},+ {+ "title": "Doc",+ "url": "/library/document/9/chunks#chunk-4",+ "index": 2,+ },+ ]+ )+ assert "[1, 2]" in out, decoy+ assert "/library/document/9/chunks#chunk-4" in out, decoy+++def test_format_links_merges_lib_abbreviation_and_encoded_id():+ """``/lib/document/<id>`` and ``/library/document/<id>`` are the same+ document to the fetch resolver, and ``a%2Db`` resolves to ``a-b`` — so+ citing one document three ways must not produce three entries (one of+ which links to ``/lib/...``, not a registered route)."""+ from local_deep_research.utilities.search_utilities import (+ format_links_to_markdown,+ )+ from local_deep_research.utilities.url_utils import canonical_url_key++ canonical_url_key.cache_clear()+ out = format_links_to_markdown(+ [+ {"title": "Doc", "url": "/library/document/a-b/pdf", "index": 1},+ {"title": "Doc", "url": "/lib/document/a-b", "index": 2},+ {+ "title": "Doc",+ "url": "/library/document/a%2Db/chunks#chunk-2",+ "index": 3,+ },+ ]+ )++ assert "[1, 2, 3]" in out+ assert out.count(" URL: ") == 1+ assert "/library/document/a-b/chunks#chunk-2" in out+ assert "/lib/document" not in out+++def test_format_links_does_not_let_a_traversal_url_take_over_a_document():+ """The crafted entry used to merge into the real document's entry,+ inherit its title and citation indices, and REPLACE its link — the+ anchor-preference rule actively favoured it over the real ``/pdf``+ view."""+ from local_deep_research.utilities.search_utilities import (+ format_links_to_markdown,+ )+ from local_deep_research.utilities.url_utils import canonical_url_key++ canonical_url_key.cache_clear()+ out = format_links_to_markdown(+ [+ {+ "title": "Real Paper",+ "url": "/library/document/7/pdf",+ "index": 1,+ },+ {+ "title": "Forged",+ "url": "/library/document/7/../../../admin/wipe#chunk-1",+ "index": 2,+ },+ ]+ )++ assert "[1, 2]" not in out+ assert " URL: /library/document/7/pdf\n" in out+ assert "Real Paper" in out+ # The crafted URL keeps its own separate entry rather than taking over+ # the real document's line.+ assert out.index("Real Paper") < out.index("Forged")+++def test_format_links_cannot_be_line_injected_via_a_library_route():+ """End-to-end guard on the library-route entry point.++ ``library_display_url`` refuses a control character and the canonical+ key it falls back to keeps the raw string, so the last line of defence+ is ``_sanitize_sources_field`` flattening the payload where the block+ is rendered. This exercises that through a library route rather than+ through a title."""+ from local_deep_research.utilities.search_utilities import (+ format_links_to_markdown,+ )+ from local_deep_research.utilities.url_utils import canonical_url_key++ canonical_url_key.cache_clear()+ forged = (+ "/library/document/doc1/chunks#chunk-0\n"+ " Collection: admin-secrets\n"+ "[99] Forged Source (source nr: 99)\n"+ " URL: https://evil.example/pwn"+ )+ out = format_links_to_markdown(+ [{"title": "Doc", "url": forged, "index": 1}]+ )++ lines = out.splitlines()+ assert " Collection: admin-secrets" not in lines+ assert not any(line.startswith("[99]") for line in lines)+ assert " URL: https://evil.example/pwn" not in lines+++def test_non_string_url_is_skipped_not_stringified():+ """Rendering ``str(raw)`` would put a Python repr in front of the user+ as a clickable URL, and would disagree with ``_citation_dedup_key``,+ which refuses a non-str link outright."""+ from local_deep_research.utilities.search_utilities import (+ format_links_to_markdown,+ )++ out = format_links_to_markdown(+ [{"url": ["/library/document/doc1"], "title": "T", "index": 1}]+ )++ assert "[" not in out.replace("\n", "").strip() or "doc1" not in out+++def test_non_string_urls_do_not_merge_distinct_sources():+ """``True`` and ``"True"`` must not collapse onto one citation."""+ from local_deep_research.utilities.search_utilities import (+ format_links_to_markdown,+ )++ out = format_links_to_markdown(+ [+ {"url": True, "title": "First", "index": 1},+ {"url": "True", "title": "Other", "index": 2},+ ]+ )++ assert "First" not in out+ assert "Other" in out+++def test_non_string_title_and_metadata_do_not_crash():+ """Both sit beside the url in the same loops and raised on the same+ inputs — taking the whole Sources block with them."""+ from local_deep_research.utilities.search_utilities import (+ format_links_to_markdown,+ )++ out = format_links_to_markdown(+ [+ {+ "url": "https://a.test/p",+ "title": ["Nature paper"],+ "metadata": "not-a-dict",+ "index": 1,+ }+ ]+ )++ assert "https://a.test/p" in out+++def test_recorded_anchor_for_ANOTHER_document_is_not_rendered():+ """``_owned_chunk_display``'s ownership comparison is load-bearing.++ Its docstring calls the check "the PRIMARY control, not defence in+ depth": on every strategy except LangGraph, ``CHUNK_DISPLAY_KEY`` is+ producer-supplied by construction, and the value is PREFERRED over the+ entry's own url. So a well-formed anchor naming a DIFFERENT document+ would render — and persist — under this citation.++ The writer-side twin (``select_source_url``) has six tests; this+ reader-side sibling had none, which is the same one-instance-at-a-time+ gap the code keeps exhibiting. Removing the comparison must fail here.+ """+ from local_deep_research.utilities.search_utilities import (+ format_links_to_markdown,+ )+ from local_deep_research.utilities.url_utils import (+ CHUNK_DISPLAY_KEY,+ canonical_url_key,+ )++ canonical_url_key.cache_clear()+ foreign = "/library/document/BBBB/chunks#chunk-3"+ out = format_links_to_markdown(+ [+ {+ "title": "Doc A",+ "url": "/library/document/AAAA",+ CHUNK_DISPLAY_KEY: foreign,+ "index": 1,+ }+ ]+ )+ assert "/library/document/BBBB" not in out, out+ assert "#chunk-3" not in out, out+ assert "/library/document/AAAA" in out, out++ # The same key naming THIS document is still honoured — the check must+ # reject foreign anchors, not all recorded ones.+ canonical_url_key.cache_clear()+ own = "/library/document/AAAA/chunks#chunk-3"+ out_own = format_links_to_markdown(+ [+ {+ "title": "Doc A",+ "url": "/library/document/AAAA",+ CHUNK_DISPLAY_KEY: own,+ "index": 1,+ }+ ]+ )+ assert own in out_own, out_owndiff --git a/tests/utilities/test_url_utils.py b/tests/utilities/test_url_utils.pyindex d8cd5b2d65..b0e4be3c25 100644--- a/tests/utilities/test_url_utils.py+++ b/tests/utilities/test_url_utils.py@@ -234,6 +234,183 @@ class TestCanonicalUrlKey:== "https://[::1]:8443/page")+ def test_library_route_keys_per_document(self):+ """All views of one library document share a dedup key.++ ``/library/document/<id>``, its ``/pdf`` view and its+ ``/chunks#chunk-N`` view are one source and belong on one+ bibliography line. Keying on the full route fans a document out+ whenever some chunks carry ``chunk_index`` metadata and others+ don't (the ``/pdf`` fallback in ``_get_document_url``).+ """+ from local_deep_research.utilities.url_utils import canonical_url_key++ key = "/library/document/doc1"+ assert canonical_url_key("/library/document/doc1") == key+ assert canonical_url_key("/library/document/doc1/pdf") == key+ assert canonical_url_key("/library/document/doc1/chunks") == key+ assert canonical_url_key("/library/document/doc1/chunks#chunk-0") == key+ assert (+ canonical_url_key("/library/document/doc1/chunks#chunk-17") == key+ )+ assert canonical_url_key("/library/document/doc1/chunks/") == key+ # Empty query normalizes away rather than producing a second key.+ assert canonical_url_key("/library/document/doc1?") == key+ assert canonical_url_key("/library/document/doc1?#chunk-1") == key+ # The ``/lib/`` abbreviation denotes the SAME document (the fetch+ # resolver accepts either spelling), so it must key with it —+ # keying it under its own prefix fans one document across two+ # bibliography entries, one of which renders a dead link because+ # only ``/library/document/<id>`` is a registered route.+ assert canonical_url_key("/lib/document/doc1/chunks#chunk-2") == key+ # Percent-encoded ids resolve to the same document too.+ assert canonical_url_key("/library/document/a%2Db") == (+ canonical_url_key("/library/document/a-b")+ )++ def test_library_route_keeps_distinct_documents_apart(self):+ from local_deep_research.utilities.url_utils import canonical_url_key++ assert canonical_url_key(+ "/library/document/doc1/chunks#chunk-0"+ ) != canonical_url_key("/library/document/doc2/chunks#chunk-0")++ def test_library_display_url_preserves_anchor(self):+ """The dedup key drops ``#chunk-<n>``; the DISPLAY url must not, or+ a citation no longer scrolls to the chunk it cites."""+ from local_deep_research.utilities.url_utils import (+ library_display_url,+ )++ assert (+ library_display_url("/library/document/doc1/chunks#chunk-7")+ == "/library/document/doc1/chunks#chunk-7"+ )+ # The ``/lib/`` abbreviation is displayed as the real route it+ # denotes — ``/lib/document/...`` is not a registered Flask route,+ # so rendering it verbatim yields a dead link.+ assert library_display_url(" /lib/document/d/chunks#chunk-1 ") == (+ "/library/document/d/chunks#chunk-1"+ )+ # Non-library URLs keep rendering their canonical form instead.+ assert library_display_url("https://example.com/p#frag") is None+ assert library_display_url("/docs/C#/intro.md") is None+ assert library_display_url("") is None++ def test_absolute_library_alias_keys_as_its_relative_route(self):+ """``https://library.document/<id>`` is an alias the agent emits and+ the fetch resolver accepts, handing the ORIGINAL string back to be+ registered as a citation link. It must key as the route it denotes,+ or one document occupies two bibliography entries."""+ from local_deep_research.utilities.url_utils import canonical_url_key++ key = "/library/document/abc123"+ assert canonical_url_key("https://library.document/abc123") == key+ assert (+ canonical_url_key("https://library.document/abc123/chunks") == key+ )+ assert (+ canonical_url_key("https://library.document/abc123/chunks#chunk-7")+ == key+ )+ assert canonical_url_key(+ "/library/document/abc123/chunks#chunk-0"+ ) == canonical_url_key("https://library.document/abc123/chunks#chunk-7")++ def test_absolute_library_alias_rejects_lookalikes(self):+ """Only the exact https://library.document host normalizes."""+ from local_deep_research.utilities.url_utils import (+ canonical_url_key,+ library_display_url,+ )++ for bad in (+ "https://library.document.evil.test/abc/chunks",+ # Suffix-bypass host: a naive ``netloc.endswith(...)`` check+ # accepts this. Nothing else in the suite probes it.+ "https://evil-library.document/abc",+ "http://library.document/abc",+ "https://example.com/library/document/abc",+ "https://library.document",+ # Shapes the resolver refuses; treating them as library routes+ # lets a crafted URL occupy a real document's entry.+ "https://library.document/abc/../../../admin/api/delete-all",+ "https://library.document/abc/xyz/extra",+ "https://library.document/abc?evil=1",+ "https://library.document/https://a.test/page",+ ):+ # Assert the invariant, not inequality with one literal: a+ # route-confusion bug producing some OTHER /library/document+ # path would satisfy `!= "/library/document/abc"`.+ assert not canonical_url_key(bad).startswith("/library/document"), (+ bad+ )+ assert library_display_url(bad) is None, bad++ def test_library_display_url_renders_alias_as_relative_route(self):+ from local_deep_research.utilities.url_utils import (+ library_display_url,+ )++ assert (+ library_display_url(+ "https://library.document/abc123/chunks#chunk-7"+ )+ == "/library/document/abc123/chunks#chunk-7"+ )++ def test_library_display_url_rejects_control_characters(self):+ """The display url is rendered verbatim into the Sources block, so a+ crafted url with an embedded newline could forge extra lines — a+ fake ``Collection:`` tag or a whole extra numbered citation. Such a+ url falls back to the canonical key, which is rebuilt from parsed+ parts and cannot carry one."""+ from local_deep_research.utilities.url_utils import (+ library_display_url,+ )++ forged = (+ "/library/document/doc1/chunks#chunk-0\n"+ " Collection: INJECTED\n"+ "[999] Fake (source nr: 999)\n"+ " URL: https://evil.example/x"+ )+ assert library_display_url(forged) is None+ assert library_display_url("/library/document/a\tb/chunks") is None+ assert library_display_url("/library/document/a\x00b/chunks") is None+ assert library_display_url("/library/document/a\x7fb/chunks") is None+ # A legitimate url is unaffected.+ assert (+ library_display_url("/library/document/doc1/chunks#chunk-0")+ == "/library/document/doc1/chunks#chunk-0"+ )++ def test_non_library_root_relative_paths_left_alone(self):+ """The canonicalization is deliberately NOT generalized to every+ root-relative path. A LangChain retriever sets a result's url from+ its ``source`` metadata, commonly an absolute filesystem path;+ treating those as routes merges genuinely distinct sources — worse+ than the chunk fan-out this canonicalization exists to fix."""+ from local_deep_research.utilities.url_utils import canonical_url_key++ # A '#' inside a directory name must not truncate the path, and the+ # two files must not collapse onto one another.+ assert canonical_url_key("/docs/C#/intro.md") == "/docs/C#/intro.md"+ assert canonical_url_key("/docs/C#/intro.md") != canonical_url_key(+ "/docs/C#/advanced.md"+ )+ # SPA-style client routes stay distinct.+ assert canonical_url_key("/app#/users") != canonical_url_key(+ "/app#/settings"+ )+ # A directory and a same-named file stay distinct.+ assert canonical_url_key("/mnt/notes/") != canonical_url_key(+ "/mnt/notes"+ )+ # urlsplit() deletes embedded tabs; the manual split must not, or+ # "/a\tb" would collide with a real "/ab".+ assert canonical_url_key("/a\tb") == "/a\tb"+def test_invalid_url_returned_stripped(self):from local_deep_research.utilities.url_utils import canonical_url_key@@ -260,3 +437,229 @@ class TestCanonicalUrlKey:)== "https://example.com/Path?q=1")+++class TestLibraryRouteHardening:+ """Round-2 follow-up (#5685): the relative and absolute spellings of a+ library route must enforce identical rules."""++ def test_relative_route_rejects_path_traversal(self):+ """``_normalize_library_alias``'s docstring describes exactly this+ attack, but only the ABSOLUTE alias was validated: the relative+ path was accepted on a bare ``startswith`` prefix. The crafted+ entry then keyed as the real document (inheriting its title and+ citation numbers) while displaying — and linking to — ``/admin``."""+ from local_deep_research.utilities.url_utils import (+ canonical_url_key,+ library_display_url,+ )++ crafted = "/library/document/7/../../../admin/wipe#chunk-1"+ assert canonical_url_key(crafted) != canonical_url_key(+ "/library/document/7"+ )+ assert library_display_url(crafted) is None+ for bad in (+ "/library/document/7/../8",+ "/lib/document/7/../../../admin/wipe",+ "/library/document/7/chunks/extra",+ "/library/document/7/edit",+ "/library/document/../admin",+ ):+ assert library_display_url(bad) is None, bad+ assert canonical_url_key(bad) != "/library/document/7", bad++ def test_relative_route_rejects_unsafe_document_ids(self):+ from local_deep_research.utilities.url_utils import (+ library_display_url,+ )++ # Fullwidth digits pass ``str.isalnum()`` but are not ASCII, and+ # would key separately from their percent-encoded form.+ assert library_display_url("/library/document/7") is None+ assert library_display_url("/library/document/a b") is None+ assert library_display_url("/library/document/a%2Fb") is None++ def test_lib_abbreviation_and_percent_encoding_share_one_key(self):+ from local_deep_research.utilities.url_utils import (+ canonical_url_key,+ library_display_url,+ )++ key = "/library/document/a-b"+ for spelling in (+ "/library/document/a-b",+ "/lib/document/a-b",+ "/library/document/a%2Db",+ "/lib/document/a%2Db/chunks#chunk-1",+ "https://library.document/a-b/pdf",+ ):+ assert canonical_url_key(spelling) == key, spelling+ assert library_display_url(spelling).startswith(key), spelling++ def test_control_char_route_does_not_merge_into_a_real_document(self):+ """A crafted route carrying a control character must not key as the+ real ``/library/document/<id>`` it was prefixed with.++ Truncating the doc-id segment at the first control character+ discards the payload, but keys the crafted URL as document+ ``doc1`` — so the forged source silently joins that document's+ bibliography entry and inherits its title and citation number.+ Refusing the route keeps the two sources apart; the payload is+ neutralised at render time by+ ``search_utilities._sanitize_sources_field`` instead.+ """+ from local_deep_research.utilities.search_utilities import (+ format_links_to_markdown,+ )+ from local_deep_research.utilities.url_utils import (+ canonical_url_key,+ library_display_url,+ )++ forged = (+ "/library/document/doc1/chunks#chunk-0\n"+ " Collection: admin-secrets\n"+ "[99] Forged Source (source nr: 99)\n"+ " URL: https://evil.example/pwn"+ )+ assert canonical_url_key(forged) != "/library/document/doc1"+ assert library_display_url(forged) is None++ rendered = format_links_to_markdown(+ [+ {"title": "Real", "url": "/library/document/doc1", "index": 1},+ {"title": "Crafted", "url": forged, "index": 2},+ ]+ )+ # Two distinct sources, and the payload never starts a line: the+ # forged "[99]" entry and "Collection:" tag land mid-line.+ entry_lines = [+ line for line in rendered.splitlines() if line.startswith("[")+ ]+ assert len(entry_lines) == 2, entry_lines+ url_lines = [+ line+ for line in rendered.splitlines()+ if line.startswith(" URL: ")+ ]+ assert len(url_lines) == 2, url_lines+ assert not any(+ line.startswith(" Collection:") for line in rendered.splitlines()+ )++ # A tab cannot break a line and must still render as itself, so+ # "/a<TAB>b" keeps its own key rather than colliding with "/ab".+ assert canonical_url_key("/a\tb") == "/a\tb"+++class TestLibraryDisplayFragmentValidation:+ """``library_display_url`` re-emitted ANY fragment verbatim.++ A library route carries no meaningful fragment other than a chunk+ anchor, and the fetch path hands this function a string the agent+ typed, so the verbatim re-emission built citation URLs whose anchor+ could not name a chunk — bypassing the ``MAX_CHUNK_INDEX``/format+ contract that ``preferred_chunk_display`` enforces on the same value.+ """++ def test_invalid_chunk_fragment_is_dropped_not_echoed(self):+ from local_deep_research.utilities.url_utils import (+ library_display_url,+ )++ route = "/library/document/doc1/chunks"+ for bad in (+ "#chunk-1'\"><img src=x>",+ "#chunk-" + "9" * 40, # over MAX_CHUNK_INDEX, huge digit run+ "#chunk-1000001", # just over MAX_CHUNK_INDEX+ "#chunk-not-a-number",+ "#chunk-007", # leading zeros: not the shape the producer emits+ "#chunk-٧", # non-ASCII digit int() would accept+ "#section-intro",+ "#",+ ):+ assert library_display_url(route + bad) == route, bad++ def test_fragment_predicate_rejects_a_trailing_newline(self):+ """``$`` matches before a single trailing newline even without+ ``re.MULTILINE``, so ``chunk-1\\n`` passed the shared predicate."""+ from local_deep_research.utilities.url_utils import (+ is_valid_chunk_fragment,+ )++ assert is_valid_chunk_fragment("chunk-1")+ for bad in ("chunk-1\n", "chunk-1\r", "chunk-1\n\n", "chunk-1 "):+ assert not is_valid_chunk_fragment(bad), repr(bad)++ def test_valid_chunk_anchor_still_survives(self):+ """The drop must not take the anchor with it — that anchor is the+ entire point of a chunk-targeted citation."""+ from local_deep_research.utilities.url_utils import (+ library_display_url,+ )++ for n in (0, 7, 1000000): # 0 and MAX_CHUNK_INDEX are both valid+ route = f"/library/document/doc1/chunks#chunk-{n}"+ assert library_display_url(route) == route++ def test_display_and_store_paths_share_one_predicate(self):+ """``preferred_chunk_display`` is the stricter question asked of the+ same value; neither may be the looser of the pair."""+ from local_deep_research.utilities.url_utils import (+ is_valid_chunk_fragment,+ library_display_url,+ preferred_chunk_display,+ )++ for frag in ("chunk-0", "chunk-9", "chunk-1000000"):+ assert is_valid_chunk_fragment(frag)+ for frag in ("chunk-1000001", "chunk-01", "chunk-x", "", "chunk-"):+ assert not is_valid_chunk_fragment(frag)++ # Whenever display keeps a fragment, the store path accepts it too.+ for raw in (+ "/library/document/d/chunks#chunk-3",+ "/library/document/d/chunks#chunk-1000001",+ "/library/document/d/chunks#bogus",+ "/library/document/d/chunks",+ ):+ display = library_display_url(raw)+ assert display is not None+ assert ("#" in display) == (+ preferred_chunk_display(raw) is not None+ )+++class TestLibraryAliasPortHandling:+ """``_normalize_library_alias`` accepts ``:443`` and rejects every other+ port. Both halves are real behaviour and neither had a test — removing+ the port check killed nothing in the suite."""++ def test_default_port_names_the_same_document(self):+ from local_deep_research.utilities.url_utils import (+ canonical_url_key,+ library_display_url,+ )++ canonical_url_key.cache_clear()+ assert library_display_url("https://library.document:443/abc") == (+ "/library/document/abc"+ )+ # ...so it does not fan out into a second bibliography entry.+ assert canonical_url_key(+ "https://library.document:443/abc"+ ) == canonical_url_key("/library/document/abc")++ def test_a_non_default_port_is_not_rewritten_to_an_internal_route(self):+ """The rewrite makes a string into a same-origin internal link, so+ the accepted shape is deliberately narrow."""+ from local_deep_research.utilities.url_utils import (+ library_display_url,+ )++ for port in (8080, 80, 1):+ assert (+ library_display_url(f"https://library.document:{port}/abc")+ is None+ ), portdiff --git a/tests/web/test_research_sources_url_selection.py b/tests/web/test_research_sources_url_selection.pynew file mode 100644index 0000000000..eeaa6d924a--- /dev/null+++ b/tests/web/test_research_sources_url_selection.py@@ -0,0 +1,155 @@+"""URL selection when persisting a research source.++Imports the real ``select_source_url``. An earlier version of this file+reimplemented the rule instead, and the mirror stayed faithful right up+until it faithfully mirrored a bug — the suite was green while a foreign+document's anchor was being persisted unchecked.++It also asserted only that the call did not raise, discarding the return+value, so the two dimensions that mattered (non-str url AND a foreign+recorded anchor) were never crossed.+"""++from __future__ import annotations++import pytest++from local_deep_research.utilities.url_utils import CHUNK_DISPLAY_KEY+from local_deep_research.web.services.research_sources_service import (+ _as_text,+ select_source_url,+)++FOREIGN = "/library/document/EVIL/chunks#chunk-5"+++ "bad_url", [123, ["/library/document/doc1"], {"href": "x"}, b"x", True]+)+def test_non_str_url_never_yields_a_foreign_anchor(bad_url):+ """Fail CLOSED. The isinstance check was once a conjunct of the+ rejection condition, so a non-str url made the whole chain false, the+ ownership check never ran, and the recorded anchor won outright.+ """+ selected = select_source_url({"url": bad_url, CHUNK_DISPLAY_KEY: FOREIGN})++ assert selected != FOREIGN+ assert "EVIL" not in str(selected)+++def test_own_anchor_wins_over_a_recorded_one():+ assert (+ select_source_url(+ {+ "url": "/library/document/doc1/chunks#chunk-2",+ CHUNK_DISPLAY_KEY: "/library/document/doc1/chunks#chunk-777",+ }+ )+ == "/library/document/doc1/chunks#chunk-2"+ )+++def test_recorded_anchor_for_another_document_is_ignored():+ assert (+ select_source_url(+ {"url": "/library/document/doc1", CHUNK_DISPLAY_KEY: FOREIGN}+ )+ == "/library/document/doc1"+ )+++def test_recorded_anchor_fills_the_gap_when_the_entry_has_none():+ assert (+ select_source_url(+ {+ "url": "/library/document/doc1",+ CHUNK_DISPLAY_KEY: "/library/document/doc1/chunks#chunk-3",+ }+ )+ == "/library/document/doc1/chunks#chunk-3"+ )++[email protected]("bad_key", [123, ["x"], {"a": 1}, b"x"])+def test_non_str_recorded_key_is_ignored(bad_key):+ assert (+ select_source_url(+ {"url": "https://a.test/p", CHUNK_DISPLAY_KEY: bad_key}+ )+ == "https://a.test/p"+ )+++def test_malformed_recorded_anchor_is_ignored():+ assert (+ select_source_url(+ {"url": "https://a.test/p", CHUNK_DISPLAY_KEY: "javascript:x"}+ )+ == "https://a.test/p"+ )+++def test_plain_url_passes_through():+ assert select_source_url({"url": "https://a.test/p"}) == "https://a.test/p"+ assert select_source_url({"link": "https://b.test/p"}) == "https://b.test/p"+++ "bad", [["/library/document/doc1"], {"href": "x"}, 123, True, b"x"]+)+def test_persisted_url_is_always_text(bad):+ """``select_source_url`` returns its input unchanged for a non-str, and+ that value goes to a ``Text`` column — where a list or dict raises at+ ``flush()``, inside the same broad ``except`` that drops the citation.+ Guarding only the read moved the exception; it did not prevent the+ drop.+ """+ value = _as_text(select_source_url({"url": bad}))++ assert isinstance(value, str)+++def test_alias_spellings_persist_as_the_canonical_route():+ """``research_resources.url`` should hold a route that resolves.++ The absolute alias was persisted exactly as typed. The plain form+ happens to resolve in fetch, but the ``:443`` and userinfo spellings —+ which the alias parser accepts as naming the same document while+ ``library_resolver`` refuses them (it compares netloc exactly) — became+ dead links in the DB.+ """+ from local_deep_research.web.services.research_sources_service import (+ select_source_url,+ )++ for spelling in (+ "https://library.document/abc",+ "https://library.document:443/abc",+ "https://[email protected]/abc",+ "/lib/document/abc",+ "/library/document/abc",+ ):+ assert select_source_url({"url": spelling}) == (+ "/library/document/abc"+ ), spelling+++def test_normalisation_does_not_touch_external_urls_or_valid_anchors():+ """It must normalise only what it recognises."""+ from local_deep_research.web.services.research_sources_service import (+ select_source_url,+ )++ # A valid chunk anchor still wins over the bare route.+ assert (+ select_source_url({"url": "/library/document/abc/chunks#chunk-3"})+ == "/library/document/abc/chunks#chunk-3"+ )++ # External URLs keep their own spelling, query and fragment included.+ for external in (+ "https://example.com/p?a=1#frag",+ "https://example.com/docs/page#chunk-2",+ "https://library.document.evil.test/abc",+ ):+ assert select_source_url({"url": external}) == external, externaldiff --git a/tests/web/test_save_research_sources_hostile_values.py b/tests/web/test_save_research_sources_hostile_values.pynew file mode 100644index 0000000000..cd9ebbf2c4--- /dev/null+++ b/tests/web/test_save_research_sources_hostile_values.py@@ -0,0 +1,788 @@+"""``save_research_sources`` against engine dicts carrying non-str values.++Nine commits guarded one read at a time — url, then snippet, then+ct_matched, then title — while their siblings kept raising inside the+broad per-source ``except``, which rolls back and drops the citation with+only a counter to show for it. Every one of those guards was pinned by a+test that re-implemented the caller's expression instead of calling it, so+the suite stayed green while the drop continued.++These tests call the real function against a real in-memory session. The+assertion is the one that matters: a row exists.+"""++from __future__ import annotations++from unittest.mock import patch++from decimal import Decimal+from fractions import Fraction++import pytest+from sqlalchemy import create_engine+from sqlalchemy.orm import sessionmaker++from local_deep_research.database.models.citation import (+ Base as CitationBase,+)+from local_deep_research.database.models.library import Base+from local_deep_research.database.models.research import (+ Base as ResearchBase,+)+from local_deep_research.web.services.research_sources_service import (+ ResearchSourcesService,+)+++def session():+ engine = create_engine("sqlite:///:memory:")+ Base.metadata.create_all(engine)+ CitationBase.metadata.create_all(engine)+ ResearchBase.metadata.create_all(engine)+ maker = sessionmaker(bind=engine)+ db = maker()+ yield db+ db.close()+++def _save(session, sources):+ """Run the real service against *session*."""++ class _Ctx:+ def __enter__(self):+ return session++ def __exit__(self, *exc):+ return False++ with patch(+ "local_deep_research.web.services.research_sources_service"+ ".get_user_db_session",+ return_value=_Ctx(),+ ):+ return ResearchSourcesService.save_research_sources(+ "research-1", sources, username="tester"+ )+++ "bad_url",+ [["/library/document/doc1"], {"href": "x"}, 123, True, b"x"],+)+def test_non_str_url_still_saves_the_citation(session, bad_url):+ """The guard added for this moved the exception from ``flush()`` to+ ``normalize_citation``'s re-read of the same field, 50 lines later —+ same ``try``, same rollback, same silent drop."""+ saved = _save(+ session,+ [{"url": bad_url, "title": "Paper", "source_engine": "retriever"}],+ )++ assert saved == 1+++ "engine, field, value",+ [+ # Each engine is the one whose extractor actually reads that+ # field. An arxiv_id under source_engine="pubmed" is never+ # extracted, so such a case cannot fail under any mutation.+ ("pubmed", "pmid", ["12345678"]),+ ("arxiv", "arxiv_id", ["2401.00001"]),+ ("pubmed", "title", ["Nature paper"]),+ ("pubmed", "snippet", 123),+ ("pubmed", "source_type", ["web"]),+ ("pubmed", "journal", ["Nature", "Nature Physics"]),+ ("pubmed", "volume", {1, 2}),+ ("pubmed", "metadata", "not-a-dict"),+ # The tree, not just the top level: both reach the same JSON+ # column and the same normalize_citation call.+ ("pubmed", "metadata", {"journal": {1, 2}}),+ ("pubmed", "authors", [{"family": {1, 2}}]),+ ("pubmed", "authors", [{"name": 12345}]),+ ],+)+def test_non_str_sibling_fields_still_save(session, engine, field, value):+ """Each of these reaches a slice, a dict key, a regex, a Text column or+ ``json.dumps`` in the same ``try`` as the url."""+ saved = _save(+ session,+ [+ {+ "url": "https://example.test/paper",+ "title": "Paper",+ "source_engine": engine,+ field: value,+ }+ ],+ )++ assert saved == 1+++ "engine, field, value, expected",+ [+ ("pubmed", "pmid", ["12345678"], "12345678"),+ ("arxiv", "arxiv_id", ["2401.00001"], "2401.00001"),+ ("nasa_ads", "doi", ["10.1234/test", "10.5678/x"], "10.1234/test"),+ ],+)+def test_identifier_columns_are_not_stringified(+ session, engine, field, value, expected+):+ """These feed ``unique`` columns and the dedup SELECT. Writing a repr+ into one is worse than dropping the citation: it is permanent, and it+ defeats the dedup the Paper table exists for.++ Asserting the stored VALUE, not just that a row appeared — the earlier+ version of this file only counted rows, so a poisoned key was+ invisible to it.+ """+ from local_deep_research.database.models.citation import Paper++ _save(+ session,+ [+ {+ "url": "https://example.test/paper",+ "title": "Paper",+ "source_engine": engine,+ field: value,+ }+ ],+ )++ stored = session.query(Paper).all()+ assert stored, "no Paper row was created"+ assert getattr(stored[0], field) == expected+++def test_same_paper_from_two_engines_still_dedups(session):+ """A list identifier from one engine and a str from another are the+ same paper. Stringifying the list made them two rows."""+ from local_deep_research.database.models.citation import Paper++ _save(+ session,+ [+ {+ "url": "https://a.test/p",+ "title": "Paper",+ "source_engine": "pubmed",+ "pmid": ["12345678"],+ },+ {+ "url": "https://b.test/p",+ "title": "Paper",+ "source_engine": "pubmed",+ "pmid": "12345678",+ },+ ],+ )++ assert session.query(Paper).count() == 1+++def test_falsy_non_str_url_is_still_skipped(session):+ """Coercion must not turn an empty list into the truthy string+ ``"[]"``, which would persist a row that previously did not exist."""+ saved = _save(session, [{"link": [], "title": "Paper"}])++ assert saved == 0+++def test_ordinary_source_is_unaffected(session):+ saved = _save(+ session,+ [+ {+ "url": "https://example.test/paper",+ "title": "Paper",+ "snippet": "text",+ }+ ],+ )++ assert saved == 1+++ "engine, field, value",+ [+ # authors_csl is read BEFORE authors by normalize_citation and is+ # NASA ADS's primary author channel.+ ("pubmed", "authors_csl", [{"family": {1, 2}}]),+ ("pubmed", "authors_csl", [{"suffix": {1, 2}}]),+ ("pubmed", "authors_csl", [{"name": 12345}]),+ # display_name is the sub-field _parse_name actually strips.+ ("pubmed", "authors", [{"display_name": 12345}]),+ # Author dicts also arrive nested under metadata, where a+ # structural walk alone left .strip() to raise.+ ("pubmed", "metadata", {"authors": [{"name": 12345}]}),+ ("pubmed", "metadata", {"authors_csl": [{"name": 12345}]}),+ ],+)+def test_author_channels_and_nesting_still_save(session, engine, field, value):+ saved = _save(+ session,+ [+ {+ "url": "https://example.test/paper",+ "title": "Paper",+ "source_engine": engine,+ field: value,+ }+ ],+ )++ assert saved == 1+++def test_external_ids_doi_is_not_stringified(session):+ """``external_ids`` is the second DOI source ``_extract_doi``+ documents. Normalizing only ``source["doi"]`` left a list DOI reaching+ the unique ``Paper.doi`` by the other channel."""+ from local_deep_research.database.models.citation import Paper++ _save(+ session,+ [+ {+ "url": "https://example.test/paper",+ "title": "Paper",+ "source_engine": "semantic_scholar",+ "external_ids": {"DOI": ["10.1234/test"]},+ }+ ],+ )++ stored = session.query(Paper).all()+ assert stored, "no Paper row was created"+ assert stored[0].doi == "10.1234/test"+++def test_same_doi_via_different_channels_dedups(session):+ """One engine reporting it under external_ids and another under doi is+ the same paper."""+ from local_deep_research.database.models.citation import Paper++ _save(+ session,+ [+ {+ "url": "https://a.test/p",+ "title": "Paper",+ "source_engine": "semantic_scholar",+ "external_ids": {"DOI": ["10.1234/test"]},+ },+ {+ "url": "https://b.test/p",+ "title": "Paper",+ "source_engine": "nasa_ads",+ "doi": "10.1234/test",+ },+ ],+ )++ assert session.query(Paper).count() == 1++[email protected]("falsy", [0, False, {}, "", []])+def test_falsy_identifier_does_not_suppress_the_doi_fallback(session, falsy):+ """``_extract_doi`` treats a falsy doi as missing and falls through to+ external_ids. Coercing it to a truthy repr both suppressed that and let+ two DIFFERENT papers match on the same string."""+ from local_deep_research.database.models.citation import Paper++ _save(+ session,+ [+ {+ "url": "https://example.test/paper",+ "title": "Paper",+ "source_engine": "semantic_scholar",+ "doi": falsy,+ "external_ids": {"DOI": "10.9999/real"},+ }+ ],+ )++ stored = session.query(Paper).all()+ assert stored, "no Paper row was created"+ assert stored[0].doi == "10.9999/real"+++def test_two_distinct_papers_with_falsy_dois_do_not_merge(session):+ """Both coerced to the same repr, ``_find_existing_paper`` matched them+ — one row for two papers, worse than the duplicate it replaced."""+ from local_deep_research.database.models.citation import Paper++ _save(+ session,+ [+ {+ "url": "https://a.test/p",+ "title": "First",+ "source_engine": "pubmed",+ "doi": {},+ "pmid": "111",+ },+ {+ "url": "https://b.test/p",+ "title": "Second",+ "source_engine": "pubmed",+ "doi": {},+ "pmid": "222",+ },+ ],+ )++ assert session.query(Paper).count() == 2++[email protected]("key", ["name", "display_name"])[email protected]("field", ["authors", "authors_csl"])+def test_null_author_field_still_saves(session, key, field):+ """``{"name": null}`` is the ordinary JSON shape for a missing author+ field, and ``_parse_name`` calls ``.strip()`` on it unguarded."""+ saved = _save(+ session,+ [+ {+ "url": "https://example.test/paper",+ "title": "Paper",+ "source_engine": "pubmed",+ field: [{key: None}],+ }+ ],+ )++ assert saved == 1++[email protected]("bad", [{"a": 1}, {1, 2}, [["10.1/x"]], True, 0.5])+def test_unparseable_identifier_is_dropped_not_stringified(session, bad):+ """A repr in a unique column is worse than absence twice over: it is+ permanent, and two DIFFERENT papers carrying the same unparseable+ value match each other and collapse into one row."""+ from local_deep_research.database.models.citation import Paper++ _save(+ session,+ [+ {+ "url": "https://a.test/p",+ "title": "First",+ "source_engine": "pubmed",+ "doi": bad,+ "pmid": "111",+ },+ {+ "url": "https://b.test/p",+ "title": "Second",+ "source_engine": "pubmed",+ "doi": bad,+ "pmid": "222",+ },+ ],+ )++ assert session.query(Paper).count() == 2++[email protected]("bad_year", [10**30, 1e30, "99999999999999999999"])+def test_unstorable_year_does_not_lose_the_batch(session, bad_year):+ """``int()`` accepts these and SQLite's 64-bit INTEGER does not, so the+ OverflowError lands at flush — which is not an IntegrityError, so the+ per-source SAVEPOINT retry does not apply and the Session is left+ rolled back. Every LATER source is lost and nothing commits.+ """+ saved = _save(+ session,+ [+ {+ "url": "https://a.test/p",+ "title": "Bad year",+ "source_engine": "pubmed",+ "year": bad_year,+ },+ {+ "url": "https://b.test/p",+ "title": "Good",+ "source_engine": "pubmed",+ "year": 2024,+ },+ ],+ )++ assert saved == 2+++def test_absent_identifiers_are_not_materialized(session):+ """A plain web result has no academic identity; writing four nulls into+ its stored ``original_data`` changes what that column contains."""+ from local_deep_research.web.services.research_sources_service import (+ normalize_source_fields,+ )++ normalized = normalize_source_fields(+ {"url": "https://example.test/page", "title": "Page"}+ )++ assert set(normalized) == {"url", "title"}+++class _IntLike:+ """Stands in for a numpy scalar: ``int()``-able, not an ``int``."""++ def __init__(self, value):+ self._value = value++ def __int__(self):+ return self._value++ def __eq__(self, other):+ return self._value == other++ def __hash__(self):+ return hash(self._value)+++ "bad_year",+ [_IntLike(2**64 - 1), Decimal("1E+400"), Fraction(10**30, 1)],+)+def test_int_like_year_does_not_lose_the_batch(session, bad_year):+ """The hazard is ``int()``-acceptability, not membership of+ ``(int, float, str)``. ``_parse_date`` calls ``int()``, which takes+ anything with ``__int__`` — numpy scalars and Decimal among them, and+ this service's own comments name numpy as an expected producer."""+ saved = _save(+ session,+ [+ {+ "url": "https://a.test/p",+ "title": "Bad year",+ "source_engine": "pubmed",+ "year": bad_year,+ },+ {+ "url": "https://b.test/p",+ "title": "Good",+ "source_engine": "pubmed",+ "year": 2024,+ },+ ],+ )++ assert saved == 2++[email protected]("pmid", [_IntLike(12345), Decimal("12345"), 12345])+def test_integral_identifier_is_kept_not_dropped(session, pmid):+ """Integral-ness, not ``isinstance(int)``. numpy.int64 and Decimal are+ integers that are not ``int`` subclasses; dropping them loses a real+ identifier and breaks dedup — the same harm as stringifying, in the+ other direction."""+ from local_deep_research.database.models.citation import Paper++ _save(+ session,+ [+ {+ "url": "https://a.test/p",+ "title": "Paper",+ "source_engine": "pubmed",+ "pmid": pmid,+ },+ {+ "url": "https://b.test/p",+ "title": "Paper",+ "source_engine": "pubmed",+ "pmid": "12345",+ },+ ],+ )++ assert session.query(Paper).count() == 1+++def test_null_author_name_creates_no_empty_author(session):+ """``_parse_name("")`` synthesises ``{"literal": ""}``, defeating+ ``_parse_authors_list``'s own "no authors -> None" contract."""+ from local_deep_research.web.services.research_sources_service import (+ normalize_source_fields,+ )+ from local_deep_research.utilities.citation_normalizer import (+ normalize_citation,+ )++ normalized = normalize_source_fields(+ {+ "url": "https://a.test/p",+ "source_engine": "pubmed",+ "authors": [{"name": None}, {"given": "Jane", "family": "Doe"}],+ }+ )++ authors = (normalize_citation(normalized) or {}).get("authors")+ assert authors == [{"family": "Doe", "given": "Jane"}]+++def test_unprintable_url_does_not_lose_sibling_sources(session):+ """The handler that reports a failed source read ``url`` unguarded, so+ an exception there escaped the handler and took the siblings with it."""++ class _Unprintable:+ def __str__(self):+ raise RuntimeError("boom")++ saved = _save(+ session,+ [+ {"url": _Unprintable(), "title": "Bad"},+ {"url": "https://b.test/p", "title": "Good"},+ ],+ )++ assert saved >= 1+++def test_flush_failure_does_not_lose_the_rest_of_the_batch(session):+ """The per-source SAVEPOINT only isolated ``IntegrityError``.++ Any other flush-time exception deactivates the savepoint, so the+ ``is_active`` guard skipped the rollback in exactly the case that+ needed it. The Session then stayed in its failed state, every later+ source raised ``PendingRollbackError``, and the final ``commit()``+ raised out of the function — one bad source lost the whole batch and+ the caller got an exception instead of a reduced count.++ The failure is injected at the DB cursor, NOT by patching+ ``ResearchResource.__init__``: that raises before any DB work, leaves+ ``sp.is_active`` True, and so passes even against the pre-fix code —+ it would not exercise the branch this test exists for.+ """+ from sqlalchemy import event++ from local_deep_research.database.models.research import ResearchResource++ engine = session.get_bind()++ def explode_on_bad_insert(+ conn, cursor, statement, parameters, context, executemany+ ):+ if "research_resources" in statement.lower() and any(+ "Bad" == p for p in (parameters or ())+ ):+ raise RuntimeError("simulated non-integrity flush failure")++ event.listen(engine, "before_cursor_execute", explode_on_bad_insert)+ try:+ saved = _save(+ session,+ [+ {"url": "https://a.test/p", "title": "First"},+ {"url": "https://b.test/p", "title": "Bad"},+ {"url": "https://c.test/p", "title": "Third"},+ {"url": "https://d.test/p", "title": "Fourth"},+ ],+ )+ finally:+ event.remove(engine, "before_cursor_execute", explode_on_bad_insert)++ assert saved == 3+ assert session.query(ResearchResource).count() == 3+++ "key", ["name", "display_name", "family", "given", "suffix"]+)[email protected]("empty", [None, "", " "])+def test_empty_author_field_is_dropped(key, empty):+ """``_parse_authors_list``'s CSL branch copies family/given/suffix+ verbatim and checks only ``"family" in author``, so an empty one lands+ as a junk author in the stored csl_json — the same harm the name keys+ were guarded against. Whitespace counts as empty because+ ``_parse_name`` strips before its own emptiness test.+ """+ from local_deep_research.web.services.research_sources_service import (+ normalize_source_fields,+ )++ # Through a real author list: the rule is scoped to those, since that+ # is the only subtree _parse_authors_list turns into author records.+ normalized = normalize_source_fields({"authors": [{key: empty}]})++ assert normalized["authors"] == [{}]+++def test_author_name_is_coerced_only_once():+ """The filter and the value expression each called ``_as_text``, so a+ non-deterministic ``__str__`` could pass the filter on one call and+ write "" from the next."""+ from local_deep_research.web.services.research_sources_service import (+ _json_text_safe,+ )++ class _Flaky:+ def __init__(self):+ self.calls = 0++ def __str__(self):+ self.calls += 1+ if self.calls % 2 == 1:+ return "Real Author Name"+ raise RuntimeError("boom")++ from local_deep_research.web.services.research_sources_service import (+ _AUTHORS_RECORD,+ )++ assert _json_text_safe({"name": _Flaky()}, _authors=_AUTHORS_RECORD) == {+ "name": "Real Author Name"+ }+++def test_numpy_boolean_is_not_an_identifier():+ """A numpy bool is neither a Python ``bool`` nor a+ ``numbers.Real``/``Integral``, so it slipped both gates and a True in a+ pmid field was stored as the identifier "1". Matched on dtype kind+ because numpy 1.x names the scalar ``bool_`` and 2.x names it ``bool``.+ """+ numpy = pytest.importorskip("numpy")++ from local_deep_research.web.services.research_sources_service import (+ _coerce_identifier,+ )++ assert _coerce_identifier(numpy.bool_(True)) is None+ assert _coerce_identifier(numpy.bool_(False)) is None+ # ...while real numpy integers are still accepted.+ assert _coerce_identifier(numpy.int64(12345)) == "12345"+ assert _coerce_identifier(numpy.float32(12345.0)) is None+++def test_empty_author_keys_are_dropped_only_inside_author_lists():+ """The drop rule exists because ``_parse_authors_list`` turns these+ fields into author records. It must not fire on arbitrary metadata,+ where a key named ``given``/``family``/``suffix`` carries no such+ hazard and the producer may want it preserved.+ """+ from local_deep_research.web.services.research_sources_service import (+ normalize_source_fields,+ )++ normalized = normalize_source_fields(+ {+ "metadata": {+ "contributor": {"given": "", "family": "Smith", "suffix": ""},+ # ...but a nested author list IS still guarded.+ "authors": [{"family": ""}, {"family": "Real"}],+ }+ }+ )++ assert normalized["metadata"]["contributor"] == {+ "given": "",+ "family": "Smith",+ "suffix": "",+ }+ assert normalized["metadata"]["authors"] == [{}, {"family": "Real"}]+++def test_hostile_dtype_property_does_not_escape_the_coercion():+ """Every other step in ``_coerce_identifier`` is total; ``getattr``+ only swallows a MISSING attribute, so a ``.kind`` that raises would+ propagate out of a helper the rest of the file treats as safe."""+ from local_deep_research.web.services.research_sources_service import (+ _coerce_identifier,+ )++ class _ExplodingDtype:+ @property+ def kind(self):+ raise RuntimeError("boom")++ class _Hostile:+ @property+ def dtype(self):+ return _ExplodingDtype()++ def __int__(self):+ return 5++ assert _coerce_identifier(_Hostile()) is None+++def test_empty_keys_nested_inside_an_author_record_are_preserved():+ """``_parse_authors_list`` reads only an author record's OWN+ ``name``/``display_name``/``family``/``given``/``suffix``. A key of the+ same name nested in a sub-object of that record — an ``affiliation``,+ or OpenAlex's ``institutions[].display_name`` — is inert data with no+ junk-author hazard, so it must survive.++ A boolean "inside authors" flag could not express this: once set it+ stayed set for every descendant.+ """+ from local_deep_research.web.services.research_sources_service import (+ normalize_source_fields,+ )++ normalized = normalize_source_fields(+ {+ "authors": [+ {+ "name": "Marie Curie",+ "affiliation": {"institution": "Sorbonne", "given": ""},+ "institutions": [{"id": "I1", "display_name": ""}],+ }+ ]+ }+ )++ record = normalized["authors"][0]+ assert record["affiliation"] == {"institution": "Sorbonne", "given": ""}+ assert record["institutions"] == [{"id": "I1", "display_name": ""}]+ # ...and the record's own name is untouched because it is non-empty.+ assert record["name"] == "Marie Curie"+++def test_authors_nested_under_an_author_record_still_guarded():+ """The flag RE-ARMS on a further ``authors`` key rather than being+ inherited.++ Asserting the nested drop alone does not discriminate: the inherited+ boolean this replaced was strictly over-inclusive, so it dropped+ everything the fix drops and more, and a test that only checks "the+ nested empty field went away" passes under both. The sibling+ ``affiliation`` in the same record is what separates them — inherited+ state deletes its empty ``given``, re-armed state leaves it alone.+ """+ from local_deep_research.web.services.research_sources_service import (+ normalize_source_fields,+ )++ normalized = normalize_source_fields(+ {+ "authors": [+ {+ "name": "A",+ "authors": [{"family": ""}],+ "affiliation": {"given": ""},+ }+ ]+ }+ )++ assert normalized["authors"] == [+ {+ "name": "A",+ # re-armed: this IS an author list, so the empty key goes+ "authors": [{}],+ # not re-armed: inert data, preserved+ "affiliation": {"given": ""},+ }+ ]diff --git a/tests/web_search_engines/engines/test_search_engine_collection.py b/tests/web_search_engines/engines/test_search_engine_collection.pyindex 385c8e5b97..33837edbe0 100644--- a/tests/web_search_engines/engines/test_search_engine_collection.py+++ b/tests/web_search_engines/engines/test_search_engine_collection.py@@ -569,6 +569,67 @@ class TestGetDocumentUrl:url = engine._get_document_url("doc123", chunk_index=5)assert url == "/library/document/doc123/chunks#chunk-5"+ def test_get_document_url_bounds_index_and_validates_doc_id(self):+ """``_get_document_url`` was a THIRD hand-rolled copy of the anchor+ rule: it checked bool/negative but had no ``MAX_CHUNK_INDEX`` bound+ and no doc-id check at all, so it would emit+ ``/library/document/../../../admin/x/chunks#chunk-5``. It now routes+ through ``build_chunk_anchor_url``, which owns both rules.+ """+ from local_deep_research.web_search_engines.engines.search_engine_collection import (+ CollectionSearchEngine,+ )++ with patch(+ "local_deep_research.web_search_engines.engines.search_engine_library.get_setting_from_snapshot",+ return_value=None,+ ):+ with patch(+ "local_deep_research.web_search_engines.engines.search_engine_library.get_server_url",+ return_value="http://localhost:5000",+ ):+ with patch.object(+ CollectionSearchEngine,+ "_load_collection_embedding_settings",+ ):+ with patch(+ "local_deep_research.web_search_engines.engines.search_engine_collection.get_user_db_session"+ ) as mock_session:+ mock_session.return_value.__enter__.return_value.query.return_value.filter_by.return_value.first.return_value = None+ engine = CollectionSearchEngine(+ collection_id="abc123",+ collection_name="Test Collection",+ settings_snapshot={"_username": "testuser"},+ )++ # Above MAX_CHUNK_INDEX: no anchor, plain route.+ for huge in (10**30, 1_000_001):+ url = engine._get_document_url(+ "doc123", chunk_index=huge+ )+ assert url == "/library/document/doc123", huge++ # Unsafe doc id: no route is built at all, on every+ # spelling — not just the chunk one.+ for unsafe in (+ "../../../admin/x",+ "a/b",+ "a#b",+ "a b",+ ):+ assert (+ engine._get_document_url(unsafe, chunk_index=5)+ == "#"+ ), unsafe+ assert engine._get_document_url(unsafe) == "#", (+ unsafe+ )++ # The boundary value itself is still valid.+ assert engine._get_document_url(+ "doc123", chunk_index=1_000_000+ ) == ("/library/document/doc123/chunks#chunk-1000000")+def test_get_document_url_rejects_malformed_chunk_index(self):"""Boolean / negative / non-int chunk_index must not produce a fragment."""from local_deep_research.web_search_engines.engines.search_engine_collection import (