Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

fix(tests): check query-split fetch URLs and cover safeFetchWithAuth call sites (#6190)

LearningCircuit4be9475becf9
diff --git a/tests/infrastructure_tests/test_hardcoded_fetch_urls.py b/tests/infrastructure_tests/test_hardcoded_fetch_urls.py
index a344b6cf16..1418db6c65 100644
--- a/tests/infrastructure_tests/test_hardcoded_fetch_urls.py
+++ b/tests/infrastructure_tests/test_hardcoded_fetch_urls.py
@@ -7,13 +7,14 @@ Why this guard exists
``test_urls_js.py`` validates the central ``URLS`` table in
``static/js/config/urls.js`` against the live app — but the frontend also
contains ~75 hardcoded ``fetch('/...')`` calls across templates and component
-scripts (news-subscription-form.html alone has a dozen). Those literals have
-no guard at all today, and the browser-level Puppeteer suites run only in
-the release pipeline (``strict-mode``), so a renamed or removed route breaks
-the UI silently and is discovered weeks later at release time. PR #3299's
-own changelog shows the failure class is real: five legacy ``/api/news/*``
-endpoints were removed and every hardcoded client reference had to be found
-by hand.
+scripts (news-subscription-form.html alone has a dozen), plus a comparable
+number of calls through the app's hardened fetch wrappers (below). Those
+literals have no guard at all today, and the browser-level Puppeteer suites
+run only in the release pipeline (``strict-mode``), so a renamed or removed
+route breaks the UI silently and is discovered weeks later at release time.
+PR #3299's own changelog shows the failure class is real: five legacy
+``/api/news/*`` endpoints were removed and every hardcoded client reference
+had to be found by hand.
Scope, deliberately narrow
--------------------------
@@ -26,13 +27,42 @@ String concatenation is skipped only when it extends the *path* itself
(``fetch('/api/research/' + id + '/status')``) — the captured
``/api/research/`` fragment is not itself a URL and matches no route by
construction, so flagging it would be a false positive, not a caught bug.
-Query-string concatenation (``fetch('/news/api/feed?limit=' + n)``) is
-*not* skipped: the path portion before the ``?`` is already complete (the
-check ignores everything from ``?`` on regardless), so it is still
-resolvable and still checked.
+Query-string concatenation is *not* skipped, in either of its two idioms:
+appended to the path literal itself (``fetch('/news/api/feed?limit=' + n)``
+— the path portion before the ``?`` is already complete, since the check
+ignores everything from ``?`` on regardless) or concatenated as its own
+separate literal (``fetch('/news/api/feed' + '?limit=' + n)`` — the
+captured ``/news/api/feed`` is already the complete path, even though it
+contains no ``?`` itself; see ``_is_concatenated`` for the precise rule).
+Both are still resolvable and still checked. Fragment concatenation
+behaves identically, since the check also ignores everything from ``#``
+on: an embedded fragment (``fetch('/news/api/feed#row-' + n)``) and a
+separate-literal fragment (``fetch('/news/api/feed' + '#row')``) are
+both still checked too.
Generated bundles (``dist/``, ``*.min.js``) are excluded — they are build
output, not sources.
+Beyond bare ``fetch()``/``XMLHttpRequest.open()``, the guard also recognises
+the codebase's hardened fetch wrappers — ``safeFetch``/``safeFetchWithAuth``/
+``safeFetchJson`` (static/js/security/safe-fetch.js; ``safeFetchWithAuth`` is
+the wrapper the codebase now prefers for new code), ``fetchWithErrorHandling``/
+``window.api.fetchWithErrorHandling`` (static/js/services/api.js), and the
+smaller per-page helpers ``getJSON``/``postJSON``/``apiGet``/``apiPost``. All
+of them take the URL as their first argument, exactly like ``fetch()``, so
+the same literal/concatenation/interpolation rules apply unchanged — only
+the set of recognised call names is wider. Skipping these wrappers would
+leave a large and growing share of the app's actual HTTP surface unguarded,
+since the guard's job is to catch every *resolvable* literal URL, not just
+those reached via one specific call spelling.
+
+This is the *covered* set, not a complete inventory of every URL-first
+wrapper shape in the tree: it currently misses the lowercase ``postJson``
+(``notes_shared.js``, case-sensitive regex), ``deleteRequest``/
+``getRequest``/``postRequest`` (``delete_manager.js``), and
+``_callSynthesize`` (``notes.js``) with its own literal endpoints. None of
+these is unresolved today, so nothing is broken, but the guard does not see
+them. #6227 tracks widening the recognised set to close this gap.
+
The route table is derived by AST/regex from the mounted routers rather
than by importing the app, mirroring ``test_route_table_parity.py``'s
approach: no import side effects, no dependency on a bootable environment,
@@ -51,6 +81,7 @@ ignored, as are external and non-root-relative URLs.
# this guard deliberately avoids (see module docstring).
import re
+import sys
from functools import lru_cache
from pathlib import Path
@@ -76,6 +107,35 @@ _XHR_OPEN_RE = re.compile(
r"\.open\(\s*[\"'][A-Za-z]+[\"']\s*,\s*([\"'])([^\"']+)\1"
)
+# The codebase's hardened fetch wrappers — all of which take the URL as
+# their first argument, exactly like ``fetch()`` itself:
+# - ``safeFetch`` / ``safeFetchWithAuth`` / ``safeFetchJson``, defined in
+# static/js/security/safe-fetch.js and exposed as globals. This is the
+# *preferred* wrapper going forward (it validates the URL and, for the
+# auth-aware variant, handles session-expiry redirects), but bare
+# ``fetch()`` is still the more common call spelling in the tree today
+# (192 bare ``fetch(`` call sites vs. 135 across all these wrappers
+# combined) — these wrappers are worth guarding because they are a
+# large and growing share of the app's HTTP surface, not because they
+# already dominate it.
+# - ``window.api.fetchWithErrorHandling``, the older wrapper defined in
+# static/js/services/api.js (also callable unqualified as
+# ``fetchWithErrorHandling`` from within that same file/closure).
+# - ``getJSON`` / ``postJSON`` / ``apiGet`` / ``apiPost`` — smaller local
+# fetch helpers with the same "URL first" shape, defined per-page/
+# component (e.g. templates/pages/zotero.html, static/js/components/
+# chat.js). ``postJSON`` is the exception: it is *also* a shared helper
+# in static/js/services/api.js (``api.js:142``), not only per-page.
+# Without this, every hardcoded URL routed through one of these — the
+# common case, not the exception — is invisible to the guard.
+_FETCH_WRAPPER_RE = re.compile(
+ r"\b(?:"
+ r"safeFetchWithAuth|safeFetchJson|safeFetch"
+ r"|window\.api\.fetchWithErrorHandling|fetchWithErrorHandling"
+ r"|getJSON|postJSON|apiGet|apiPost"
+ r")\(\s*([\"'`])([^\"'`]*)\1"
+)
+
#: Backtick URL containing interpolation — dynamic, skipped by design.
_INTERPOLATION_MARKERS = ("${", "{{", "{%")
@@ -84,6 +144,11 @@ def _line_no(text: str, offset: int) -> int:
return text.count("\n", 0, offset) + 1
+#: A quoted literal immediately following a ``+`` — used to look one token
+#: past the ``+`` to decide what the concatenation is actually building.
+_NEXT_LITERAL_RE = re.compile(r"""(['"`])([^'"`]*)\1""")
+
+
def _is_concatenated(text: str, match: re.Match[str]) -> bool:
"""True if the matched literal is a truncated *path* fragment continued
by string concatenation, e.g. the ``'/api/research/'`` in
@@ -96,16 +161,40 @@ def _is_concatenated(text: str, match: re.Match[str]) -> bool:
``+``. A leading-``+`` check would therefore be dead code; it is
omitted rather than kept as a check that can never fire.
- A trailing ``+`` alone is not enough, though: if the literal contains a
- ``?``, everything before that ``?`` is already a *complete* path —
- ``_normalise`` strips the query string, so ``'<path>?param=' + value``
- (the standard query-string-concatenation idiom) still yields a fully
- resolvable path and must still be checked, not skipped. Only when the
- ``+`` extends the path itself — no ``?`` in the literal — is the
- fragment genuinely unresolvable.
+ A trailing ``+`` alone is not enough to call the fragment incomplete,
+ though. Two independent ways the rest of the path can already be
+ complete before the ``+``:
+
+ 1. The captured literal itself contains a ``?`` or a ``#``:
+ everything before that marker is already a complete path
+ (``_normalise`` strips the query string / fragment), e.g.
+ ``'<path>?param=' + value`` or ``'<path>#row-' + value``.
+ 2. The captured literal has *no* ``?`` or ``#``, but the very next
+ token is itself a quoted literal that *starts* with ``?`` or ``#``,
+ e.g. ``'/api/news/feed' + '?limit=' + n``. Here the query string is
+ concatenated as its own separate literal rather than appended to
+ the path literal, so the captured ``/api/news/feed`` is already the
+ complete, resolvable path — even though it contains no ``?`` and is
+ followed by a ``+``. Checking only the captured literal's own
+ contents (case 1) misses this idiom entirely, which is exactly what
+ made the guard skip (and thus never check) fragments like this one.
+
+ Only when neither holds — no ``?``/``#`` in the captured literal,
+ and the next token after the ``+`` is not a literal beginning with
+ ``?``/``#`` (typically an identifier, e.g. ``... + id + '/status'``) — does the
+ ``+`` genuinely extend the path itself, making the fragment
+ unresolvable on its own and the case to skip.
"""
after = text[match.end() :].lstrip()
- return after.startswith("+") and "?" not in match.group(2)
+ if not after.startswith("+"):
+ return False
+ if "?" in match.group(2) or "#" in match.group(2):
+ return False
+ next_token = after[1:].lstrip()
+ next_literal = _NEXT_LITERAL_RE.match(next_token)
+ if next_literal and next_literal.group(2).startswith(("?", "#")):
+ return False
+ return True
@lru_cache(maxsize=None)
@@ -236,7 +325,7 @@ def _collect_literals() -> dict[str, list[str]]:
found: dict[str, list[str]] = {}
for path in _iter_frontend_files():
text = path.read_text(encoding="utf-8", errors="replace")
- for regex in (_FETCH_RE, _XHR_OPEN_RE):
+ for regex in (_FETCH_RE, _XHR_OPEN_RE, _FETCH_WRAPPER_RE):
for match in regex.finditer(text):
raw = match.group(2)
if any(marker in raw for marker in _INTERPOLATION_MARKERS):
@@ -257,7 +346,20 @@ def test_frontend_literal_urls_resolve_to_routes():
literals = _collect_literals()
# Anti-vacuity: the extractor must find a substantial body of literals —
# a regex regression that matches nothing would otherwise pass silently.
- assert len(literals) >= 40, (
+ # Bare fetch()/XHR alone currently yield 47; adding the fetch-wrapper
+ # coverage (safeFetchWithAuth & co., see module docstring) brings the
+ # real count to 69. The floor is set well below that observed count —
+ # room for routine churn — but well above the old 40, so it is *more*
+ # likely to catch a regression that drops wrapper coverage than the old
+ # bare-fetch-only floor was. It is not a guarantee for every wrapper
+ # name on its own: losing `safeFetchWithAuth` alone (the largest single
+ # contributor) does trip this floor, but losing a smaller wrapper name
+ # — e.g. `safeFetchJson`, which contributes no distinct literal today —
+ # would not move this integer at all. `_FETCH_WRAPPER_RE` being wired
+ # into `_collect_literals` in the first place, and each individual name
+ # staying matched, is instead asserted by name in
+ # `test_each_wrapper_name_is_wired_into_the_collector` below.
+ assert len(literals) >= 60, (
f"only {len(literals)} distinct literal URLs found across "
"templates/static-js — the extractor is likely broken"
)
@@ -325,6 +427,194 @@ def test_query_string_concatenation_is_not_skipped():
assert not _url_resolves(normalised)
+def test_query_string_as_separate_literal_is_not_skipped():
+ """The other query-string-concatenation idiom: the ``?...`` portion is
+ its own separate literal rather than appended to the path literal, e.g.
+ ``fetch('/api/news/feed' + '?limit=' + n)``. The captured
+ ``/api/news/feed`` fragment contains no ``?`` and is followed by a
+ ``+``, which superficially looks like the genuine path-continuation
+ idiom (``'/api/research/' + id + '/status'``) — but it is not: the path
+ is already complete, and the ``+`` is only concatenating on the query
+ string. It must still be checked, not skipped."""
+ text = "fetch('/api/news/feed' + '?limit=' + n)"
+ match = next(_FETCH_RE.finditer(text))
+ assert match.group(2) == "/api/news/feed"
+ assert not _is_concatenated(text, match), (
+ "a captured literal followed by '+ <literal starting with ?>' must "
+ "still be checked — the path itself is already complete"
+ )
+ # A bogus path hidden behind the same idiom must still be caught.
+ bogus_text = "fetch('/api/does-not-exist' + '?limit=' + n)"
+ bogus_match = next(_FETCH_RE.finditer(bogus_text))
+ assert not _is_concatenated(bogus_text, bogus_match)
+ assert not _url_resolves(bogus_match.group(2))
+
+
+def test_fragment_concatenated_as_separate_literal_is_not_skipped():
+ """Same idiom as above, with a ``#fragment`` instead of a ``?query`` —
+ the rule also checks for a leading ``#`` on the following literal."""
+ text = "fetch('/api/research/history' + '#section')"
+ match = next(_FETCH_RE.finditer(text))
+ assert not _is_concatenated(text, match)
+
+
+def test_fragment_embedded_in_concatenated_literal_is_not_skipped():
+ """The embedded-fragment idiom: the ``#...`` portion sits inside the
+ captured literal itself, e.g. ``fetch('/api/x#row-' + id)``. Like an
+ embedded ``?`` (see ``test_query_string_concatenation_is_not_skipped``),
+ the path portion before the ``#`` is already complete — ``_normalise``
+ strips the fragment — so the call must be checked, not skipped."""
+ text = "fetch('/api/research/history#row-' + id)"
+ match = next(_FETCH_RE.finditer(text))
+ assert not _is_concatenated(text, match), (
+ "fragment-bearing concatenation must still be checked, not skipped"
+ )
+ # A bogus path hidden behind the same idiom must still be caught.
+ bogus_text = "fetch('/api/does-not-exist#row-' + id)"
+ bogus_match = next(_FETCH_RE.finditer(bogus_text))
+ assert not _is_concatenated(bogus_text, bogus_match)
+ normalised = _normalise(bogus_match.group(2))
+ assert normalised == "/api/does-not-exist"
+ assert not _url_resolves(normalised)
+
+
+def test_genuine_path_concatenation_still_skipped():
+ """The fix for the query-as-separate-literal idiom must not regress the
+ genuine path-continuation case: when the token after ``+`` is an
+ identifier (not a quoted literal), the captured fragment is still an
+ unresolvable partial path and must still be skipped."""
+ text = "fetch('/api/research/' + id + '/status')"
+ match = next(_FETCH_RE.finditer(text))
+ assert _is_concatenated(text, match)
+
+
+def test_fetch_wrapper_call_sites_are_recognised():
+ """The guard must not be blind to the codebase's hardened fetch
+ wrappers: a literal URL passed to ``safeFetchWithAuth`` (the wrapper the
+ codebase now prefers over bare ``fetch()``) must be extracted and
+ checked exactly like a bare ``fetch()`` call.
+
+ This only exercises ``_FETCH_WRAPPER_RE`` in isolation, not whether it
+ is actually wired into ``_collect_literals`` (see
+ ``test_each_wrapper_name_is_wired_into_the_collector`` for that). A
+ ``_is_concatenated`` check is deliberately not repeated here: the
+ character right after this capture is always ``,`` (the call's second
+ argument), never ``+``, so that assertion would always trivially pass
+ regardless of ``_is_concatenated``'s behaviour — see
+ ``test_fetch_wrapper_genuine_concatenation_still_skipped`` for the real
+ concatenation coverage on wrapper calls."""
+ text = "const r = await safeFetchWithAuth('/api/does-not-exist', {method: 'GET'});"
+ match = next(_FETCH_WRAPPER_RE.finditer(text))
+ assert match.group(2) == "/api/does-not-exist"
+ assert not _url_resolves(match.group(2))
+
+
+ "text",
+ [
+ "safeFetchWithAuth('/settings/api/' + key, {method: 'GET'})",
+ "safeFetch('/settings/api/' + key)",
+ "getJSON('/api/research/' + id)",
+ "postJSON('/api/research/' + id + '/status')",
+ "apiGet('/api/research/' + id)",
+ "apiPost('/api/research/' + id, body)",
+ "window.api.fetchWithErrorHandling('/api/research/' + id)",
+ ],
+)
+def test_fetch_wrapper_genuine_concatenation_still_skipped(text):
+ """The wrapper call shapes reuse ``_is_concatenated`` unchanged, so the
+ genuine path-continuation skip must still apply to them, not just to
+ bare ``fetch()``. The ``window.api.fetchWithErrorHandling`` parameter
+ cannot fail under any single-name mutation of ``_FETCH_WRAPPER_RE``,
+ for the same ``\\b`` fallback reason disclosed on
+ ``test_each_wrapper_name_is_wired_into_the_collector``."""
+ match = next(_FETCH_WRAPPER_RE.finditer(text))
+ assert _is_concatenated(text, match)
+
+
+#: One bogus, non-resolving literal call per alternative in
+#: ``_FETCH_WRAPPER_RE`` — every name the regex recognises, not a subset.
+_WRAPPER_FIXTURE_CALLS = {
+ "safeFetchWithAuth": "safeFetchWithAuth('/bogus/safeFetchWithAuth')",
+ "safeFetchJson": "safeFetchJson('/bogus/safeFetchJson')",
+ "safeFetch": "safeFetch('/bogus/safeFetch')",
+ "window.api.fetchWithErrorHandling": (
+ "window.api.fetchWithErrorHandling('/bogus/windowApiFetchWithErrorHandling')"
+ ),
+ "fetchWithErrorHandling": "fetchWithErrorHandling('/bogus/fetchWithErrorHandling')",
+ "getJSON": "getJSON('/bogus/getJSON')",
+ "postJSON": "postJSON('/bogus/postJSON')",
+ "apiGet": "apiGet('/bogus/apiGet')",
+ "apiPost": "apiPost('/bogus/apiPost')",
+}
+
+
[email protected]("wrapper_name", sorted(_WRAPPER_FIXTURE_CALLS))
+def test_each_wrapper_name_is_wired_into_the_collector(
+ wrapper_name, tmp_path, monkeypatch
+):
+ """Named, per-wrapper proof that a literal URL passed to *this specific*
+ wrapper name reaches ``_collect_literals()`` — not just that
+ ``_FETCH_WRAPPER_RE`` can match it in isolation
+ (``test_fetch_wrapper_call_sites_are_recognised`` doesn't touch the
+ collector at all), and not just that some aggregate integer stays above
+ a floor (the anti-vacuity floor in
+ ``test_frontend_literal_urls_resolve_to_routes`` only reacts to losing
+ ``safeFetchWithAuth`` — the largest single contributor — or to the
+ regex being unwired from ``_collect_literals`` at its call site
+ entirely, both of which drop the count far enough to trip it; every
+ other wrapper name can silently fall out of ``_FETCH_WRAPPER_RE``
+ without moving that floor at all).
+
+ Dropping any one name from ``_FETCH_WRAPPER_RE`` fails that name's case
+ here, with one caveat: dropping only the ``window.api.`` qualified
+ alternative still passes, because ``\\b`` also matches immediately after
+ the ``.`` in ``window.api.fetchWithErrorHandling(``, so the plain
+ ``fetchWithErrorHandling`` alternative independently matches the same
+ call site at a later position in the alternation. Only dropping *both*
+ fails both cases. Unwiring ``_FETCH_WRAPPER_RE`` from
+ ``_collect_literals`` entirely fails every case.
+ """
+ fixture_dir = tmp_path / "static" / "js"
+ fixture_dir.mkdir(parents=True)
+ fixture_file = fixture_dir / "wrapper_fixture.js"
+ fixture_file.write_text(
+ "\n".join(_WRAPPER_FIXTURE_CALLS.values()), encoding="utf-8"
+ )
+ this_module = sys.modules[__name__]
+ monkeypatch.setattr(
+ this_module, "_iter_frontend_files", lambda: [fixture_file]
+ )
+ # `_collect_literals` also does `path.relative_to(SRC_ROOT.parent.parent)`
+ # to build its location strings — `tmp_path` lives outside the repo, so
+ # without this it raises ValueError rather than exercising the wrapper
+ # logic. Repoint `SRC_ROOT` so `tmp_path` (the real ancestor of
+ # `fixture_file`) is where that walk-up lands; nothing else reads
+ # `SRC_ROOT` through this monkeypatched call, since `_iter_frontend_files`
+ # is replaced above rather than driven by `WEB_DIR`.
+ monkeypatch.setattr(
+ this_module, "SRC_ROOT", tmp_path / "src" / "local_deep_research"
+ )
+
+ literals = _collect_literals()
+
+ expected_url = re.search(
+ r"""\(\s*(['"`])([^'"`]*)\1""", _WRAPPER_FIXTURE_CALLS[wrapper_name]
+ ).group(2)
+ assert expected_url.startswith("/bogus/"), (
+ f"fixture call text is malformed — extracted {expected_url!r}"
+ )
+ assert expected_url in literals, (
+ f"{wrapper_name!r} call site not surfaced by _collect_literals() — "
+ "either this name no longer matches in _FETCH_WRAPPER_RE, or the "
+ "regex is unwired from the collector"
+ )
+ assert not _url_resolves(expected_url), (
+ f"fixture URL {expected_url!r} unexpectedly resolves to a real "
+ "route — this test relies on it being unresolvable"
+ )
+
+
def test_route_extractor_finds_the_app_level_and_router_routes():
"""The static route table must contain representatives of every shape it
is matched against — app-level, prefixed-router, and parameterized."""