~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
fix(tests): check query-split fetch URLs and cover safeFetchWithAuth call sites (#6190)
LearningCircuit4be9475becf9
diff --git a/tests/infrastructure_tests/test_hardcoded_fetch_urls.py b/tests/infrastructure_tests/test_hardcoded_fetch_urls.pyindex a344b6cf16..1418db6c65 100644--- a/tests/infrastructure_tests/test_hardcoded_fetch_urls.py+++ b/tests/infrastructure_tests/test_hardcoded_fetch_urls.py@@ -7,13 +7,14 @@ Why this guard exists``test_urls_js.py`` validates the central ``URLS`` table in``static/js/config/urls.js`` against the live app — but the frontend alsocontains ~75 hardcoded ``fetch('/...')`` calls across templates and component-scripts (news-subscription-form.html alone has a dozen). Those literals have-no guard at all today, and the browser-level Puppeteer suites run only in-the release pipeline (``strict-mode``), so a renamed or removed route breaks-the UI silently and is discovered weeks later at release time. PR #3299's-own changelog shows the failure class is real: five legacy ``/api/news/*``-endpoints were removed and every hardcoded client reference had to be found-by hand.+scripts (news-subscription-form.html alone has a dozen), plus a comparable+number of calls through the app's hardened fetch wrappers (below). Those+literals have no guard at all today, and the browser-level Puppeteer suites+run only in the release pipeline (``strict-mode``), so a renamed or removed+route breaks the UI silently and is discovered weeks later at release time.+PR #3299's own changelog shows the failure class is real: five legacy+``/api/news/*`` endpoints were removed and every hardcoded client reference+had to be found by hand.Scope, deliberately narrow--------------------------@@ -26,13 +27,42 @@ String concatenation is skipped only when it extends the *path* itself(``fetch('/api/research/' + id + '/status')``) — the captured``/api/research/`` fragment is not itself a URL and matches no route byconstruction, so flagging it would be a false positive, not a caught bug.-Query-string concatenation (``fetch('/news/api/feed?limit=' + n)``) is-*not* skipped: the path portion before the ``?`` is already complete (the-check ignores everything from ``?`` on regardless), so it is still-resolvable and still checked.+Query-string concatenation is *not* skipped, in either of its two idioms:+appended to the path literal itself (``fetch('/news/api/feed?limit=' + n)``+— the path portion before the ``?`` is already complete, since the check+ignores everything from ``?`` on regardless) or concatenated as its own+separate literal (``fetch('/news/api/feed' + '?limit=' + n)`` — the+captured ``/news/api/feed`` is already the complete path, even though it+contains no ``?`` itself; see ``_is_concatenated`` for the precise rule).+Both are still resolvable and still checked. Fragment concatenation+behaves identically, since the check also ignores everything from ``#``+on: an embedded fragment (``fetch('/news/api/feed#row-' + n)``) and a+separate-literal fragment (``fetch('/news/api/feed' + '#row')``) are+both still checked too.Generated bundles (``dist/``, ``*.min.js``) are excluded — they are buildoutput, not sources.+Beyond bare ``fetch()``/``XMLHttpRequest.open()``, the guard also recognises+the codebase's hardened fetch wrappers — ``safeFetch``/``safeFetchWithAuth``/+``safeFetchJson`` (static/js/security/safe-fetch.js; ``safeFetchWithAuth`` is+the wrapper the codebase now prefers for new code), ``fetchWithErrorHandling``/+``window.api.fetchWithErrorHandling`` (static/js/services/api.js), and the+smaller per-page helpers ``getJSON``/``postJSON``/``apiGet``/``apiPost``. All+of them take the URL as their first argument, exactly like ``fetch()``, so+the same literal/concatenation/interpolation rules apply unchanged — only+the set of recognised call names is wider. Skipping these wrappers would+leave a large and growing share of the app's actual HTTP surface unguarded,+since the guard's job is to catch every *resolvable* literal URL, not just+those reached via one specific call spelling.++This is the *covered* set, not a complete inventory of every URL-first+wrapper shape in the tree: it currently misses the lowercase ``postJson``+(``notes_shared.js``, case-sensitive regex), ``deleteRequest``/+``getRequest``/``postRequest`` (``delete_manager.js``), and+``_callSynthesize`` (``notes.js``) with its own literal endpoints. None of+these is unresolved today, so nothing is broken, but the guard does not see+them. #6227 tracks widening the recognised set to close this gap.+The route table is derived by AST/regex from the mounted routers ratherthan by importing the app, mirroring ``test_route_table_parity.py``'sapproach: no import side effects, no dependency on a bootable environment,@@ -51,6 +81,7 @@ ignored, as are external and non-root-relative URLs.# this guard deliberately avoids (see module docstring).import re+import sysfrom functools import lru_cachefrom pathlib import Path@@ -76,6 +107,35 @@ _XHR_OPEN_RE = re.compile(r"\.open\(\s*[\"'][A-Za-z]+[\"']\s*,\s*([\"'])([^\"']+)\1")+# The codebase's hardened fetch wrappers — all of which take the URL as+# their first argument, exactly like ``fetch()`` itself:+# - ``safeFetch`` / ``safeFetchWithAuth`` / ``safeFetchJson``, defined in+# static/js/security/safe-fetch.js and exposed as globals. This is the+# *preferred* wrapper going forward (it validates the URL and, for the+# auth-aware variant, handles session-expiry redirects), but bare+# ``fetch()`` is still the more common call spelling in the tree today+# (192 bare ``fetch(`` call sites vs. 135 across all these wrappers+# combined) — these wrappers are worth guarding because they are a+# large and growing share of the app's HTTP surface, not because they+# already dominate it.+# - ``window.api.fetchWithErrorHandling``, the older wrapper defined in+# static/js/services/api.js (also callable unqualified as+# ``fetchWithErrorHandling`` from within that same file/closure).+# - ``getJSON`` / ``postJSON`` / ``apiGet`` / ``apiPost`` — smaller local+# fetch helpers with the same "URL first" shape, defined per-page/+# component (e.g. templates/pages/zotero.html, static/js/components/+# chat.js). ``postJSON`` is the exception: it is *also* a shared helper+# in static/js/services/api.js (``api.js:142``), not only per-page.+# Without this, every hardcoded URL routed through one of these — the+# common case, not the exception — is invisible to the guard.+_FETCH_WRAPPER_RE = re.compile(+ r"\b(?:"+ r"safeFetchWithAuth|safeFetchJson|safeFetch"+ r"|window\.api\.fetchWithErrorHandling|fetchWithErrorHandling"+ r"|getJSON|postJSON|apiGet|apiPost"+ r")\(\s*([\"'`])([^\"'`]*)\1"+)+#: Backtick URL containing interpolation — dynamic, skipped by design._INTERPOLATION_MARKERS = ("${", "{{", "{%")@@ -84,6 +144,11 @@ def _line_no(text: str, offset: int) -> int:return text.count("\n", 0, offset) + 1+#: A quoted literal immediately following a ``+`` — used to look one token+#: past the ``+`` to decide what the concatenation is actually building.+_NEXT_LITERAL_RE = re.compile(r"""(['"`])([^'"`]*)\1""")++def _is_concatenated(text: str, match: re.Match[str]) -> bool:"""True if the matched literal is a truncated *path* fragment continuedby string concatenation, e.g. the ``'/api/research/'`` in@@ -96,16 +161,40 @@ def _is_concatenated(text: str, match: re.Match[str]) -> bool:``+``. A leading-``+`` check would therefore be dead code; it isomitted rather than kept as a check that can never fire.- A trailing ``+`` alone is not enough, though: if the literal contains a- ``?``, everything before that ``?`` is already a *complete* path —- ``_normalise`` strips the query string, so ``'<path>?param=' + value``- (the standard query-string-concatenation idiom) still yields a fully- resolvable path and must still be checked, not skipped. Only when the- ``+`` extends the path itself — no ``?`` in the literal — is the- fragment genuinely unresolvable.+ A trailing ``+`` alone is not enough to call the fragment incomplete,+ though. Two independent ways the rest of the path can already be+ complete before the ``+``:++ 1. The captured literal itself contains a ``?`` or a ``#``:+ everything before that marker is already a complete path+ (``_normalise`` strips the query string / fragment), e.g.+ ``'<path>?param=' + value`` or ``'<path>#row-' + value``.+ 2. The captured literal has *no* ``?`` or ``#``, but the very next+ token is itself a quoted literal that *starts* with ``?`` or ``#``,+ e.g. ``'/api/news/feed' + '?limit=' + n``. Here the query string is+ concatenated as its own separate literal rather than appended to+ the path literal, so the captured ``/api/news/feed`` is already the+ complete, resolvable path — even though it contains no ``?`` and is+ followed by a ``+``. Checking only the captured literal's own+ contents (case 1) misses this idiom entirely, which is exactly what+ made the guard skip (and thus never check) fragments like this one.++ Only when neither holds — no ``?``/``#`` in the captured literal,+ and the next token after the ``+`` is not a literal beginning with+ ``?``/``#`` (typically an identifier, e.g. ``... + id + '/status'``) — does the+ ``+`` genuinely extend the path itself, making the fragment+ unresolvable on its own and the case to skip."""after = text[match.end() :].lstrip()- return after.startswith("+") and "?" not in match.group(2)+ if not after.startswith("+"):+ return False+ if "?" in match.group(2) or "#" in match.group(2):+ return False+ next_token = after[1:].lstrip()+ next_literal = _NEXT_LITERAL_RE.match(next_token)+ if next_literal and next_literal.group(2).startswith(("?", "#")):+ return False+ return True@lru_cache(maxsize=None)@@ -236,7 +325,7 @@ def _collect_literals() -> dict[str, list[str]]:found: dict[str, list[str]] = {}for path in _iter_frontend_files():text = path.read_text(encoding="utf-8", errors="replace")- for regex in (_FETCH_RE, _XHR_OPEN_RE):+ for regex in (_FETCH_RE, _XHR_OPEN_RE, _FETCH_WRAPPER_RE):for match in regex.finditer(text):raw = match.group(2)if any(marker in raw for marker in _INTERPOLATION_MARKERS):@@ -257,7 +346,20 @@ def test_frontend_literal_urls_resolve_to_routes():literals = _collect_literals()# Anti-vacuity: the extractor must find a substantial body of literals —# a regex regression that matches nothing would otherwise pass silently.- assert len(literals) >= 40, (+ # Bare fetch()/XHR alone currently yield 47; adding the fetch-wrapper+ # coverage (safeFetchWithAuth & co., see module docstring) brings the+ # real count to 69. The floor is set well below that observed count —+ # room for routine churn — but well above the old 40, so it is *more*+ # likely to catch a regression that drops wrapper coverage than the old+ # bare-fetch-only floor was. It is not a guarantee for every wrapper+ # name on its own: losing `safeFetchWithAuth` alone (the largest single+ # contributor) does trip this floor, but losing a smaller wrapper name+ # — e.g. `safeFetchJson`, which contributes no distinct literal today —+ # would not move this integer at all. `_FETCH_WRAPPER_RE` being wired+ # into `_collect_literals` in the first place, and each individual name+ # staying matched, is instead asserted by name in+ # `test_each_wrapper_name_is_wired_into_the_collector` below.+ assert len(literals) >= 60, (f"only {len(literals)} distinct literal URLs found across ""templates/static-js — the extractor is likely broken")@@ -325,6 +427,194 @@ def test_query_string_concatenation_is_not_skipped():assert not _url_resolves(normalised)+def test_query_string_as_separate_literal_is_not_skipped():+ """The other query-string-concatenation idiom: the ``?...`` portion is+ its own separate literal rather than appended to the path literal, e.g.+ ``fetch('/api/news/feed' + '?limit=' + n)``. The captured+ ``/api/news/feed`` fragment contains no ``?`` and is followed by a+ ``+``, which superficially looks like the genuine path-continuation+ idiom (``'/api/research/' + id + '/status'``) — but it is not: the path+ is already complete, and the ``+`` is only concatenating on the query+ string. It must still be checked, not skipped."""+ text = "fetch('/api/news/feed' + '?limit=' + n)"+ match = next(_FETCH_RE.finditer(text))+ assert match.group(2) == "/api/news/feed"+ assert not _is_concatenated(text, match), (+ "a captured literal followed by '+ <literal starting with ?>' must "+ "still be checked — the path itself is already complete"+ )+ # A bogus path hidden behind the same idiom must still be caught.+ bogus_text = "fetch('/api/does-not-exist' + '?limit=' + n)"+ bogus_match = next(_FETCH_RE.finditer(bogus_text))+ assert not _is_concatenated(bogus_text, bogus_match)+ assert not _url_resolves(bogus_match.group(2))+++def test_fragment_concatenated_as_separate_literal_is_not_skipped():+ """Same idiom as above, with a ``#fragment`` instead of a ``?query`` —+ the rule also checks for a leading ``#`` on the following literal."""+ text = "fetch('/api/research/history' + '#section')"+ match = next(_FETCH_RE.finditer(text))+ assert not _is_concatenated(text, match)+++def test_fragment_embedded_in_concatenated_literal_is_not_skipped():+ """The embedded-fragment idiom: the ``#...`` portion sits inside the+ captured literal itself, e.g. ``fetch('/api/x#row-' + id)``. Like an+ embedded ``?`` (see ``test_query_string_concatenation_is_not_skipped``),+ the path portion before the ``#`` is already complete — ``_normalise``+ strips the fragment — so the call must be checked, not skipped."""+ text = "fetch('/api/research/history#row-' + id)"+ match = next(_FETCH_RE.finditer(text))+ assert not _is_concatenated(text, match), (+ "fragment-bearing concatenation must still be checked, not skipped"+ )+ # A bogus path hidden behind the same idiom must still be caught.+ bogus_text = "fetch('/api/does-not-exist#row-' + id)"+ bogus_match = next(_FETCH_RE.finditer(bogus_text))+ assert not _is_concatenated(bogus_text, bogus_match)+ normalised = _normalise(bogus_match.group(2))+ assert normalised == "/api/does-not-exist"+ assert not _url_resolves(normalised)+++def test_genuine_path_concatenation_still_skipped():+ """The fix for the query-as-separate-literal idiom must not regress the+ genuine path-continuation case: when the token after ``+`` is an+ identifier (not a quoted literal), the captured fragment is still an+ unresolvable partial path and must still be skipped."""+ text = "fetch('/api/research/' + id + '/status')"+ match = next(_FETCH_RE.finditer(text))+ assert _is_concatenated(text, match)+++def test_fetch_wrapper_call_sites_are_recognised():+ """The guard must not be blind to the codebase's hardened fetch+ wrappers: a literal URL passed to ``safeFetchWithAuth`` (the wrapper the+ codebase now prefers over bare ``fetch()``) must be extracted and+ checked exactly like a bare ``fetch()`` call.++ This only exercises ``_FETCH_WRAPPER_RE`` in isolation, not whether it+ is actually wired into ``_collect_literals`` (see+ ``test_each_wrapper_name_is_wired_into_the_collector`` for that). A+ ``_is_concatenated`` check is deliberately not repeated here: the+ character right after this capture is always ``,`` (the call's second+ argument), never ``+``, so that assertion would always trivially pass+ regardless of ``_is_concatenated``'s behaviour — see+ ``test_fetch_wrapper_genuine_concatenation_still_skipped`` for the real+ concatenation coverage on wrapper calls."""+ text = "const r = await safeFetchWithAuth('/api/does-not-exist', {method: 'GET'});"+ match = next(_FETCH_WRAPPER_RE.finditer(text))+ assert match.group(2) == "/api/does-not-exist"+ assert not _url_resolves(match.group(2))+++ "text",+ [+ "safeFetchWithAuth('/settings/api/' + key, {method: 'GET'})",+ "safeFetch('/settings/api/' + key)",+ "getJSON('/api/research/' + id)",+ "postJSON('/api/research/' + id + '/status')",+ "apiGet('/api/research/' + id)",+ "apiPost('/api/research/' + id, body)",+ "window.api.fetchWithErrorHandling('/api/research/' + id)",+ ],+)+def test_fetch_wrapper_genuine_concatenation_still_skipped(text):+ """The wrapper call shapes reuse ``_is_concatenated`` unchanged, so the+ genuine path-continuation skip must still apply to them, not just to+ bare ``fetch()``. The ``window.api.fetchWithErrorHandling`` parameter+ cannot fail under any single-name mutation of ``_FETCH_WRAPPER_RE``,+ for the same ``\\b`` fallback reason disclosed on+ ``test_each_wrapper_name_is_wired_into_the_collector``."""+ match = next(_FETCH_WRAPPER_RE.finditer(text))+ assert _is_concatenated(text, match)+++#: One bogus, non-resolving literal call per alternative in+#: ``_FETCH_WRAPPER_RE`` — every name the regex recognises, not a subset.+_WRAPPER_FIXTURE_CALLS = {+ "safeFetchWithAuth": "safeFetchWithAuth('/bogus/safeFetchWithAuth')",+ "safeFetchJson": "safeFetchJson('/bogus/safeFetchJson')",+ "safeFetch": "safeFetch('/bogus/safeFetch')",+ "window.api.fetchWithErrorHandling": (+ "window.api.fetchWithErrorHandling('/bogus/windowApiFetchWithErrorHandling')"+ ),+ "fetchWithErrorHandling": "fetchWithErrorHandling('/bogus/fetchWithErrorHandling')",+ "getJSON": "getJSON('/bogus/getJSON')",+ "postJSON": "postJSON('/bogus/postJSON')",+ "apiGet": "apiGet('/bogus/apiGet')",+ "apiPost": "apiPost('/bogus/apiPost')",+}++[email protected]("wrapper_name", sorted(_WRAPPER_FIXTURE_CALLS))+def test_each_wrapper_name_is_wired_into_the_collector(+ wrapper_name, tmp_path, monkeypatch+):+ """Named, per-wrapper proof that a literal URL passed to *this specific*+ wrapper name reaches ``_collect_literals()`` — not just that+ ``_FETCH_WRAPPER_RE`` can match it in isolation+ (``test_fetch_wrapper_call_sites_are_recognised`` doesn't touch the+ collector at all), and not just that some aggregate integer stays above+ a floor (the anti-vacuity floor in+ ``test_frontend_literal_urls_resolve_to_routes`` only reacts to losing+ ``safeFetchWithAuth`` — the largest single contributor — or to the+ regex being unwired from ``_collect_literals`` at its call site+ entirely, both of which drop the count far enough to trip it; every+ other wrapper name can silently fall out of ``_FETCH_WRAPPER_RE``+ without moving that floor at all).++ Dropping any one name from ``_FETCH_WRAPPER_RE`` fails that name's case+ here, with one caveat: dropping only the ``window.api.`` qualified+ alternative still passes, because ``\\b`` also matches immediately after+ the ``.`` in ``window.api.fetchWithErrorHandling(``, so the plain+ ``fetchWithErrorHandling`` alternative independently matches the same+ call site at a later position in the alternation. Only dropping *both*+ fails both cases. Unwiring ``_FETCH_WRAPPER_RE`` from+ ``_collect_literals`` entirely fails every case.+ """+ fixture_dir = tmp_path / "static" / "js"+ fixture_dir.mkdir(parents=True)+ fixture_file = fixture_dir / "wrapper_fixture.js"+ fixture_file.write_text(+ "\n".join(_WRAPPER_FIXTURE_CALLS.values()), encoding="utf-8"+ )+ this_module = sys.modules[__name__]+ monkeypatch.setattr(+ this_module, "_iter_frontend_files", lambda: [fixture_file]+ )+ # `_collect_literals` also does `path.relative_to(SRC_ROOT.parent.parent)`+ # to build its location strings — `tmp_path` lives outside the repo, so+ # without this it raises ValueError rather than exercising the wrapper+ # logic. Repoint `SRC_ROOT` so `tmp_path` (the real ancestor of+ # `fixture_file`) is where that walk-up lands; nothing else reads+ # `SRC_ROOT` through this monkeypatched call, since `_iter_frontend_files`+ # is replaced above rather than driven by `WEB_DIR`.+ monkeypatch.setattr(+ this_module, "SRC_ROOT", tmp_path / "src" / "local_deep_research"+ )++ literals = _collect_literals()++ expected_url = re.search(+ r"""\(\s*(['"`])([^'"`]*)\1""", _WRAPPER_FIXTURE_CALLS[wrapper_name]+ ).group(2)+ assert expected_url.startswith("/bogus/"), (+ f"fixture call text is malformed — extracted {expected_url!r}"+ )+ assert expected_url in literals, (+ f"{wrapper_name!r} call site not surfaced by _collect_literals() — "+ "either this name no longer matches in _FETCH_WRAPPER_RE, or the "+ "regex is unwired from the collector"+ )+ assert not _url_resolves(expected_url), (+ f"fixture URL {expected_url!r} unexpectedly resolves to a real "+ "route — this test relies on it being unresolvable"+ )++def test_route_extractor_finds_the_app_level_and_router_routes():"""The static route table must contain representatives of every shape itis matched against — app-level, prefixed-router, and parameterized."""