Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

43.9 KB
Raw Blame
"""Does every frontend ``fetch()`` use the right VERB and the right FIELD NAMES?

``tests/test_advertised_but_dead_sweep.py`` already settled the *existence*
question: no URL literal in ``static/js`` points at a path nothing serves.
That sweep is method-blind and payload-blind, and those are exactly the two
things a Flask(WSGI) -> FastAPI(ASGI) port breaks silently:

* ``fetch(url, {method: 'POST'})`` against a path that came back as
  ``@router.get`` is a 405. The URL exists, so an existence sweep is green,
  and the button is dead.
* A body key renamed or re-cased while a handler was rewritten from
  ``request.get_json()`` to a Pydantic model is a 422 the user experiences
  as "the button does nothing".

Two sections, in that order:

A. **Verb parity, static.** Every ``fetch``/``safeFetch``/``safeFetchJson``/
   ``safeFetchWithAuth``/``fetchWithErrorHandling``/``postJSON`` call site in
   ``web/static/js/**`` is extracted with its (method, url). URLs are
   resolved through the real ``config/urls.js`` registry and the
   ``URLBuilder`` shortcuts. The result is matched against the **imported**
   ``app.routes`` table -- never a re-derivation of FastAPI's routing.

B. **Field-name parity, live.** For the calls whose JSON body shape is
   statically parseable, the *real* endpoint is driven over ``TestClient``
   with that exact key set and must not answer 422.

ANTI-VACUITY. A static sweep whose extractor quietly stops matching reports
a perfect score, and a live 422 check against an endpoint that does not
validate bodies at all passes for every payload. So:

* **Floors** -- ``>= 180`` call sites extracted, ``>= 300`` routes,
  ``>= 150`` (method, path) pairs actually compared, ``>= 40`` parseable
  request bodies. A regex that stops matching fails instead of passing.
* **Positive control** -- synthetic JS with a known verb mismatch, which the
  extractor + comparator must flag.
* **Negative control** -- real, known-good call sites they must not flag.
* **Live negative control** -- every endpoint driven in section B is *first*
  driven with a deliberately mis-named payload and must answer 422. Only
  endpoints that demonstrably validate their body count toward the "correct
  shape is accepted" assertion, and a floor is asserted on how many did.

Every request here is shape-only: ids are non-existent, so handlers answer
404/400 rather than mutating anything. 404 still proves the body passed
validation, because FastAPI validates the body before the handler runs.
"""

from __future__ import annotations

import dataclasses
import functools
import re
import uuid
from dataclasses import dataclass
from pathlib import Path

import pytest
from fastapi.routing import APIRoute
from fastapi.testclient import TestClient

from local_deep_research.web.fastapi_app import app as fastapi_app

REPO_ROOT = Path(__file__).resolve().parents[2]
WEB = REPO_ROOT / "src" / "local_deep_research" / "web"
JS_ROOT = WEB / "static" / "js"

#: Stand-in for a runtime-interpolated URL segment (``${id}``, ``' + key``).
WILD = "\x00"

# Floors. Chosen from what the extractor actually finds today, minus a
# margin, so ordinary churn does not trip them but a silently broken
# extractor does.
MIN_FETCH_CALLS = 180
MIN_ROUTES = 300
MIN_COMPARED = 150
MIN_PARSEABLE_BODIES = 40
MIN_VALIDATING_ENDPOINTS = 6


# ---------------------------------------------------------------------------
# Section A1 -- JavaScript source surgery
# ---------------------------------------------------------------------------


def strip_js_comments(src: str) -> str:
    """Blank ``//`` and ``/* */`` comments, preserving offsets and newlines.

    Offsets are preserved so reported line numbers stay usable, and string
    literals are walked so a ``'http://x'`` inside a string is not mistaken
    for the start of a comment.
    """
    out: list[str] = []
    i, n = 0, len(src)
    while i < n:
        c = src[i]
        if c in "'\"`":
            quote = c
            out.append(c)
            i += 1
            while i < n:
                if src[i] == "\\":
                    out.append(src[i : i + 2])
                    i += 2
                    continue
                out.append(src[i])
                if src[i] == quote:
                    i += 1
                    break
                i += 1
            continue
        if c == "/" and i + 1 < n and src[i + 1] == "/":
            while i < n and src[i] != "\n":
                out.append(" ")
                i += 1
            continue
        if c == "/" and i + 1 < n and src[i + 1] == "*":
            end = src.find("*/", i + 2)
            end = n if end < 0 else end + 2
            out.append("".join(ch if ch == "\n" else " " for ch in src[i:end]))
            i = end
            continue
        out.append(c)
        i += 1
    return "".join(out)


def split_call_args(text: str, open_paren: int) -> tuple[list[str], int]:
    """Split the argument list of a call whose ``(`` is at ``open_paren``.

    Bracket- and string-aware, so an object literal containing commas stays
    one argument. Returns ``(args, index_of_closing_paren)``; the index is
    ``-1`` when the call is unterminated.
    """
    assert text[open_paren] == "("
    i, n = open_paren + 1, len(text)
    depth = 0
    args: list[str] = []
    cur: list[str] = []
    while i < n:
        c = text[i]
        if c in "'\"`":
            quote = c
            cur.append(c)
            i += 1
            while i < n:
                if text[i] == "\\":
                    cur.append(text[i : i + 2])
                    i += 2
                    continue
                cur.append(text[i])
                if text[i] == quote:
                    i += 1
                    break
                i += 1
            continue
        if c in "([{":
            depth += 1
            cur.append(c)
            i += 1
            continue
        if c in ")]}":
            if c == ")" and depth == 0:
                args.append("".join(cur))
                return args, i
            depth -= 1
            cur.append(c)
            i += 1
            continue
        if c == "," and depth == 0:
            args.append("".join(cur))
            cur = []
            i += 1
            continue
        cur.append(c)
        i += 1
    return args, -1


# ---------------------------------------------------------------------------
# Section A2 -- URL expression resolution
# ---------------------------------------------------------------------------


@functools.cache
def urls_registry() -> dict[str, str]:
    """``GROUP.KEY -> '/path'`` from ``static/js/config/urls.js``."""
    src = strip_js_comments((JS_ROOT / "config" / "urls.js").read_text())
    out: dict[str, str] = {}
    for group_match in re.finditer(r"(\b[A-Z][A-Z0-9_]*)\s*:\s*\{", src):
        group = group_match.group(1)
        i = group_match.end() - 1
        depth = 0
        while i < len(src):
            if src[i] == "{":
                depth += 1
            elif src[i] == "}":
                depth -= 1
                if depth == 0:
                    break
            i += 1
        body = src[group_match.end() : i]
        for key_match in re.finditer(
            r"(\b[A-Z][A-Z0-9_]*)\s*:\s*'([^']*)'", body
        ):
            out[f"{group}.{key_match.group(1)}"] = key_match.group(2)
    return out


@functools.cache
def url_builder_shortcuts() -> dict[str, str]:
    """``URLBuilder.<name>`` -> resolved path pattern.

    ``urls.js`` defines a couple of dozen one-line convenience methods
    (``researchStatus(id)`` -> ``this.build(URLS.API.RESEARCH_STATUS, id)``).
    Without resolving them ~25 real call sites drop out of the comparison.
    """
    src = strip_js_comments((JS_ROOT / "config" / "urls.js").read_text())
    registry = urls_registry()
    out: dict[str, str] = {}
    for match in re.finditer(
        r"\n    ([A-Za-z_$][\w$]*)\s*\([^)]*\)\s*\{(.*?)\n    \}", src, re.S
    ):
        name, body = match.group(1), match.group(2)
        ref = re.search(r"URLS\.([A-Z0-9_]+\.[A-Z0-9_]+)", body)
        if ref is None:
            continue
        path = registry.get(ref.group(1))
        if path is None:
            continue
        out[name] = re.sub(r"\{\w+\}", WILD, path)
    return out


_INTERP_RE = re.compile(r"\$\{[^{}]*(?:\{[^{}]*\}[^{}]*)*\}")


def _template_literal_to_path(
    body: str, consts: dict[str, str] | None = None
) -> str:
    """``/api/x/${id}/y`` -> ``/api/x/<WILD>/y``.

    A ``${NAME}`` whose NAME is a known path constant is substituted rather
    than wildcarded, so ``` `${SETTINGS_API_BASE}${key}` ``` keeps its
    ``/settings/api/`` prefix instead of collapsing to a single WILD.
    """

    def sub(match: re.Match[str]) -> str:
        inner = match.group(0)[2:-1].strip()
        if consts and inner in consts:
            return consts[inner]
        return WILD

    return _INTERP_RE.sub(sub, body)


_STR_RE = re.compile(r"'(?:[^'\\]|\\.)*'|\"(?:[^\"\\]|\\.)*\"|`[^`]*`", re.S)


def _literal_value(
    expr: str, consts: dict[str, str] | None = None
) -> str | None:
    expr = expr.strip()
    if not _STR_RE.fullmatch(expr):
        return None
    inner = expr[1:-1]
    return _template_literal_to_path(inner, consts) if expr[0] == "`" else inner


def resolve_url_expression(
    expr: str, local_consts: dict[str, str]
) -> str | None:
    """Best-effort static resolution of a JS URL expression.

    ``None`` means "could not be resolved", which callers must treat as
    *not examined* -- never as clean.
    """
    expr = expr.strip()
    registry = urls_registry()

    match = re.fullmatch(
        r"URLBuilder\.(?:build|buildWithReplacements)\("
        r"\s*URLS\.([A-Z0-9_.]+)\s*(?:,.*)?\)",
        expr,
        re.S,
    )
    if match:
        path = registry.get(match.group(1))
        return re.sub(r"\{\w+\}", WILD, path) if path else None

    match = re.fullmatch(
        r"URLBuilder\.([A-Za-z_$][\w$]*)\((?:.*)?\)", expr, re.S
    )
    if match and match.group(1) in url_builder_shortcuts():
        return url_builder_shortcuts()[match.group(1)]

    match = re.fullmatch(r"URLS\.([A-Z0-9_]+\.[A-Z0-9_]+)", expr)
    if match:
        path = registry.get(match.group(1))
        return re.sub(r"\{\w+\}", WILD, path) if path else None

    literal = _literal_value(expr, local_consts)
    if literal is not None:
        return literal

    if "+" in expr:
        pieces: list[str] = []
        for part in expr.split("+"):
            part = part.strip()
            value = _literal_value(part, local_consts)
            if value is not None:
                pieces.append(value)
            elif part in local_consts:
                pieces.append(local_consts[part])
            elif re.fullmatch(r"[\w$.\[\]()]+", part):
                pieces.append(WILD)  # runtime value
            else:
                return None
        if pieces and pieces[0].startswith("/"):
            return "".join(pieces)
        return None

    if re.fullmatch(r"[A-Za-z_$][\w$]*", expr):
        return local_consts.get(expr)
    return None


#: A declaration whose initialiser contains any of these is a function, an
#: object or a conditional -- not a URL constant. Wildcarding it would invent
#: a match, so it is left unresolved instead.
_NOT_A_URL_EXPR = ("=>", "function", "{", "?", "\n\n")

_DECL_RE = re.compile(r"(?:const|let|var)\s+([A-Za-z_$][\w$]*)\s*=\s*([^;]*?);")


def local_url_constants(text: str) -> dict[str, str]:
    """``const NAME = <url expression>`` declarations in one file.

    Resolved to a fixed point (three passes) so a constant defined in terms
    of another constant -- ``const API = BASE + '/x'`` -- comes out whole.
    A name declared twice with different values is dropped rather than
    guessed at.
    """
    raw: dict[str, list[str]] = {}
    for match in _DECL_RE.finditer(text):
        expr = match.group(2).strip()
        if not expr or any(bad in expr for bad in _NOT_A_URL_EXPR):
            continue
        raw.setdefault(match.group(1), []).append(expr)

    resolved: dict[str, str] = {}
    for _ in range(3):
        changed = False
        for name, exprs in raw.items():
            if name in resolved:
                continue
            values = {resolve_url_expression(e, resolved) for e in exprs}
            values.discard(None)
            if len(values) == 1:
                value = values.pop()
                if value.startswith("/"):
                    resolved[name] = value
                    changed = True
        if not changed:
            break
    return resolved


# ---------------------------------------------------------------------------
# Section A3 -- fetch call extraction
# ---------------------------------------------------------------------------

#: Every function the codebase uses to reach the backend. ``safeFetch*`` are
#: defined in ``security/safe-fetch.js``; ``fetchWithErrorHandling`` and
#: ``postJSON`` in ``services/api.js``. ``postChat`` in research.js uses the
#: auth-context-fenced CSRF helper. Both post helpers hard-code POST + JSON.
FETCH_FNS = (
    "safeFetchWithAuth",
    "safeFetchJson",
    "fetchWithErrorHandling",
    "safeFetch",
    "postJSON",
    "postChat",
    "fetch",
)

_FETCH_RE = re.compile(
    r"(?<![\w.$])(?:window\.)?(" + "|".join(FETCH_FNS) + r")\s*\("
)

#: Files that *define* the wrappers. Inside them ``fetch(url, options)`` is
#: the plumbing itself, with a caller-supplied url -- not a call site.
_WRAPPER_DEFINITION_FILES = frozenset(
    {"config/urls.js", "security/safe-fetch.js", "services/api.js"}
)


@dataclass(frozen=True)
class FetchCall:
    rel_path: str
    lineno: int
    fn: str
    url_expr: str
    options_src: str
    method: str  # verb, or "?" when it cannot be known statically
    path: str | None  # resolved pattern (WILD for interpolated segments)

    @property
    def where(self) -> str:
        return f"{self.rel_path}:{self.lineno}"


def method_of(fn: str, options_src: str) -> str:
    """The HTTP verb a call site sends, or ``"?"`` when not statically known."""
    if fn in {"postJSON", "postChat"}:
        return "POST"  # Both helpers hard-code method: 'POST'.
    match = re.search(r"(?<![\w$])method\s*:\s*['\"](\w+)['\"]", options_src)
    if match:
        return match.group(1).upper()
    if re.search(r"(?<![\w$])method\s*:", options_src):
        return "?"  # method: someVariable
    if re.search(r"\.\.\.\s*[A-Za-z_$]", options_src):
        return "?"  # spread options object may carry a method
    return "GET"  # fetch()'s documented default


def extract_fetch_calls(src: str, rel_path: str) -> list[FetchCall]:
    text = strip_js_comments(src)
    consts = local_url_constants(text)
    calls: list[FetchCall] = []
    for match in _FETCH_RE.finditer(text):
        fn = match.group(1)
        # `async function fetchWithErrorHandling(url, options)` is a
        # definition, not a call.
        if re.search(r"(?:function|async)\s+$", text[: match.start()]):
            continue
        args, end = split_call_args(text, match.end() - 1)
        if end < 0 or not args:
            continue
        url_expr = args[0].strip()
        options = args[1] if len(args) > 1 else ""
        calls.append(
            FetchCall(
                rel_path=rel_path,
                lineno=text[: match.start()].count("\n") + 1,
                fn=fn,
                url_expr=url_expr,
                options_src=options,
                method=method_of(fn, options),
                path=resolve_url_expression(url_expr, consts),
            )
        )
    return calls


@functools.cache
def all_fetch_calls() -> tuple[FetchCall, ...]:
    calls: list[FetchCall] = []
    for path in sorted(JS_ROOT.rglob("*.js")):
        rel = path.relative_to(JS_ROOT).as_posix()
        if rel in _WRAPPER_DEFINITION_FILES:
            continue
        calls.extend(
            extract_fetch_calls(
                path.read_text(encoding="utf-8", errors="ignore"), rel
            )
        )
    return tuple(calls)


def same_origin_api_path(call_path: str | None) -> str | None:
    """Normalise a resolved path, or ``None`` if it is not our own API."""
    if (
        not call_path
        or not call_path.startswith("/")
        or call_path.startswith("//")
    ):
        return None
    path = call_path.split("?")[0].split("#")[0]
    if path.startswith("/static/"):
        return None
    return path


# ---------------------------------------------------------------------------
# Section A4 -- the real route table, imported (never re-derived)
# ---------------------------------------------------------------------------


@functools.cache
def route_table() -> tuple[tuple[str, frozenset[str]], ...]:
    return tuple(
        (route.path, frozenset(route.methods or ()))
        for route in fastapi_app.routes
        if isinstance(route, APIRoute)
    )


def _is_param(segment: str) -> bool:
    return segment.startswith("{") and segment.endswith("}")


def path_matches_route(call_path: str, route_path: str) -> bool:
    """Would a URL of shape ``call_path`` be routed to ``route_path``?

    A WILD segment is a runtime value, so it matches only a ``{param}``
    segment: assuming an interpolation happens to produce some literal
    segment would invent matches that do not exist.
    """
    if "{path:path}" in route_path:
        return call_path.startswith(route_path.split("{path:path}")[0])
    left = call_path.strip("/").split("/")
    right = route_path.strip("/").split("/")
    if len(left) != len(right):
        return False
    for a, b in zip(left, right):
        if WILD in a:
            if not _is_param(b):
                return False
            continue
        if _is_param(b):
            continue
        if a != b:
            return False
    return True


def routes_matching(call_path: str) -> list[tuple[str, frozenset[str]]]:
    return [r for r in route_table() if path_matches_route(call_path, r[0])]


def find_verb_mismatches(calls) -> list[str]:
    """Call sites whose METHOD no route serving that path accepts.

    Starlette records a *partial* match when the path matches but the method
    does not, and keeps scanning; only if no full match exists anywhere does
    it answer 405. So the set a call site may legally use is the union over
    every route whose path matches -- not just the first one.
    """
    problems: list[str] = []
    for call in calls:
        path = same_origin_api_path(call.path)
        if path is None or call.method == "?":
            continue
        matched = routes_matching(path)
        if not matched:
            continue  # path existence is test_advertised_but_dead_sweep's job
        allowed: set[str] = set()
        for _, methods in matched:
            allowed |= methods
        if call.method not in allowed:
            problems.append(
                f"{call.where}: {call.fn}({call.url_expr.splitlines()[0]!r}) "
                f"sends {call.method} to {path.replace(WILD, '*')!r}, but the "
                f"only route(s) serving it accept {sorted(allowed)} "
                f"({[m[0] for m in matched]})"
            )
    return problems


# ---------------------------------------------------------------------------
# Section A5 -- floors and controls for the static extractor
# ---------------------------------------------------------------------------


def test_extractor_floor_fetch_calls():
    """A regex that stops matching must fail here, not report a clean sweep."""
    calls = all_fetch_calls()
    assert len(calls) >= MIN_FETCH_CALLS, (
        f"only {len(calls)} fetch call sites extracted from "
        f"{JS_ROOT} (floor {MIN_FETCH_CALLS}) -- the extractor has "
        f"stopped matching, so every downstream 'clean' result is vacuous"
    )
    # The wrapper must actually be reached, not just bare `fetch` -- if the
    # `safeFetchWithAuth` arm of the extractor died, ~40% of the frontend's
    # traffic would silently drop out of every check below.
    by_fn = {fn: sum(1 for c in calls if c.fn == fn) for fn in FETCH_FNS}
    for fn in ("fetch", "safeFetchWithAuth"):
        assert by_fn[fn] > 0, f"no {fn}() call sites found: {by_fn}"
    # (safeFetch / safeFetchJson / fetchWithErrorHandling / postJSON are
    # defined in the wrapper files but have no remaining JS call sites; the
    # extractor still recognises them so a future migration back is covered.)


def test_route_table_floor():
    routes = route_table()
    assert len(routes) >= MIN_ROUTES, (
        f"only {len(routes)} APIRoutes on the assembled app "
        f"(floor {MIN_ROUTES}) -- routers failed to mount"
    )


def test_url_resolution_floor():
    """Most call sites must resolve, or the comparison examines nothing."""
    calls = all_fetch_calls()
    resolved = [c for c in calls if same_origin_api_path(c.path) is not None]
    assert len(resolved) >= MIN_COMPARED, (
        f"only {len(resolved)}/{len(calls)} call sites resolved to a "
        f"same-origin path (floor {MIN_COMPARED}); urls.js registry has "
        f"{len(urls_registry())} entries and "
        f"{len(url_builder_shortcuts())} URLBuilder shortcuts"
    )


# --- positive control: synthetic source with a known verb mismatch ---------

_POSITIVE_CONTROL_JS = """
// A button that POSTs to a GET-only endpoint: dead, but URL-existence-clean.
async function loadSettings() {
    const response = await fetch('/settings/api/categories', {
        method: 'POST',
        headers: {'Content-Type': 'application/json'},
        body: JSON.stringify({category: 'llm'})
    });
    return response.json();
}
"""

# --- negative control: verbatim from a real, working call site -------------

_NEGATIVE_CONTROL_JS = """
async function loadCategories() {
    const response = await fetch('/settings/api/categories');
    return response.json();
}
async function saveSetting(key, value) {
    return await fetch(URLBuilder.updateSetting(key), {
        method: 'PUT',
        headers: {'Content-Type': 'application/json', 'X-CSRFToken': token},
        body: JSON.stringify({value: value})
    });
}
"""


def test_positive_control_verb_mismatch_is_flagged():
    """Feed the detector a dead POST. It must say so."""
    calls = extract_fetch_calls(_POSITIVE_CONTROL_JS, "<synthetic>")
    assert len(calls) == 1, calls
    assert calls[0].method == "POST"
    assert calls[0].path == "/settings/api/categories"

    problems = find_verb_mismatches(calls)
    assert len(problems) == 1, (
        "the verb-mismatch detector did not flag a POST at a GET-only "
        f"route; it is inert and every clean result below is meaningless. "
        f"Got: {problems}"
    )
    assert "sends POST" in problems[0], problems[0]


def test_negative_control_good_calls_are_not_flagged():
    """Real, working call shapes must not be flagged."""
    calls = extract_fetch_calls(_NEGATIVE_CONTROL_JS, "<synthetic>")
    assert len(calls) == 2, calls
    assert [c.method for c in calls] == ["GET", "PUT"]
    assert calls[1].path == "/settings/api/" + WILD, calls[1]
    assert find_verb_mismatches(calls) == []


def test_method_extraction_control():
    """``method:`` from a variable must be reported as unknown, not GET."""
    calls = extract_fetch_calls(
        "fetch('/api/x', {method: verb});"
        "fetch('/api/y', {...opts});"
        "fetch('/api/z');",
        "<synthetic>",
    )
    assert [c.method for c in calls] == ["?", "?", "GET"]


def test_mutation_control_over_the_real_corpus():
    """Every real call site must be *reachable* by the comparator.

    The synthetic positive control above proves the comparator can flag one
    call. This proves it actually looks at all of them: each real,
    resolvable call is re-issued with a verb no route anywhere accepts, and
    every single one must come back flagged. A path pattern that silently
    fails to match any route, or a call whose method was mis-read, drops out
    of ``find_verb_mismatches`` unnoticed -- here it fails loudly.
    """
    real = [
        c
        for c in all_fetch_calls()
        if c.method != "?"
        and same_origin_api_path(c.path) is not None
        and routes_matching(same_origin_api_path(c.path))
    ]
    assert len(real) >= MIN_COMPARED, len(real)

    # "TRACE" is registered by no router in this app.
    assert not any("TRACE" in methods for _, methods in route_table())
    mutated = [dataclasses.replace(c, method="TRACE") for c in real]
    flagged = find_verb_mismatches(mutated)
    assert len(flagged) == len(mutated), (
        f"the comparator flagged only {len(flagged)} of {len(mutated)} call "
        f"sites when every one of them was given an unserved verb -- it is "
        f"skipping call sites it appears to check"
    )


#: Verb mismatches that exist on this branch. Each is pinned by a strict
#: xfail below, so fixing one turns that xfail into a failure and forces the
#: entry to be removed. New mismatches are NOT suppressed by this map.
#: EMPTY, and it should stay that way. The one entry this map was written
#: for -- the logpanel HEAD pre-flight -- is FIXED in the same commit that
#: added this file, so the sweep below is a plain "no mismatches" assertion
#: again. The map is kept because the stale-entry check makes it safe: a
#: future suppression cannot silently outlive the defect it describes.
KNOWN_VERB_DEFECTS: dict[str, str] = {}


# --- the actual sweep ------------------------------------------------------


def test_every_frontend_call_uses_a_verb_its_route_accepts():
    calls = all_fetch_calls()
    compared = [
        c
        for c in calls
        if c.method != "?"
        and same_origin_api_path(c.path) is not None
        and routes_matching(same_origin_api_path(c.path))
    ]
    assert len(compared) >= MIN_COMPARED, (
        f"only {len(compared)} (method, path) pairs were actually compared "
        f"against the route table (floor {MIN_COMPARED})"
    )
    problems = find_verb_mismatches(calls)
    unknown = [
        p
        for p in problems
        if not any(p.startswith(where + ":") for where in KNOWN_VERB_DEFECTS)
    ]
    assert not unknown, (
        f"{len(unknown)} frontend call(s) send a verb no route serving "
        "that path accepts (405 == a control that does nothing):\n  "
        + "\n  ".join(unknown)
    )
    # A suppression that no longer describes a real mismatch is a lie about
    # coverage, so stale entries fail too.
    stale = [
        where
        for where in KNOWN_VERB_DEFECTS
        if not any(p.startswith(where + ":") for p in problems)
    ]
    assert not stale, (
        f"KNOWN_VERB_DEFECTS lists {stale}, which no longer mismatch. "
        f"Delete the entry (and the xfail that pins it)."
    )


# ---------------------------------------------------------------------------
# Section B1 -- static extraction of JSON request-body shapes
# ---------------------------------------------------------------------------


def _skip_string(src: str, i: int) -> int:
    """Index just past the string literal starting at ``src[i]``."""
    quote = src[i]
    i += 1
    n = len(src)
    while i < n:
        if src[i] == "\\":
            i += 2
            continue
        if src[i] == quote:
            return i + 1
        i += 1
    return i


def _skip_value(src: str, i: int) -> int:
    """Index of the top-level ``,`` or ``}`` that ends the value at ``i``."""
    depth = 0
    n = len(src)
    while i < n:
        c = src[i]
        if c in "'\"`":
            i = _skip_string(src, i)
            continue
        if c in "{[(":
            depth += 1
        elif c in "}])":
            if depth == 0:
                return i
            depth -= 1
        elif c == "," and depth == 0:
            return i
        i += 1
    return i


#: Emitted for a computed key (``{[name]: v}``) -- the name is a runtime
#: value, so the shape is known to be incomplete.
COMPUTED_KEY = "<computed>"
#: Emitted for a spread element (``{...base, v: 1}``) -- likewise incomplete.
SPREAD_KEY = "..."

_IDENT_RE = re.compile(r"[A-Za-z_$][\w$]*")


def json_body_keys(options_src: str) -> list[str] | None:
    """Top-level keys of ``body: JSON.stringify({...})``.

    Handles quoted keys, ES6 shorthand (``{query, mode}``), computed keys and
    spread elements. ``None`` when the body is absent or not a literal object
    (a variable, a ``FormData``): those are *not examined*, never "clean".
    """
    match = re.search(r"(?<![\w$])body\s*:\s*JSON\.stringify\s*\(", options_src)
    if match is None:
        return None
    args, end = split_call_args(options_src, match.end() - 1)
    if end < 0 or not args:
        return None
    obj = args[0].strip()
    if not obj.startswith("{"):
        return None

    keys: list[str] = []
    i, n = 1, len(obj)
    while i < n:
        while i < n and obj[i] in " \t\r\n,":
            i += 1
        if i >= n or obj[i] == "}":
            break

        if obj.startswith("...", i):
            keys.append(SPREAD_KEY)
            i = _skip_value(obj, i + 3)
            continue

        if obj[i] in "'\"`":
            key_end = _skip_string(obj, i)
            key = obj[i + 1 : key_end - 1]
        elif obj[i] == "[":
            key_end = _skip_value(obj, i)
            key = COMPUTED_KEY
        else:
            ident = _IDENT_RE.match(obj, i)
            if ident is None:  # unparseable element; give up on this object
                return None
            key_end, key = ident.end(), ident.group(0)

        rest = key_end
        while rest < n and obj[rest] in " \t\r\n":
            rest += 1
        keys.append(key)
        if rest < n and obj[rest] == ":":
            i = _skip_value(obj, rest + 1)  # explicit key: value
        elif rest < n and obj[rest] == "(":
            i = _skip_value(obj, rest)  # method shorthand -- still a key
        else:
            i = rest  # ES6 shorthand: {query, mode}
    return keys


@functools.cache
def parseable_bodies() -> tuple[tuple[FetchCall, tuple[str, ...]], ...]:
    """Every mutating call site whose JSON body is a static object literal."""
    out = []
    for call in all_fetch_calls():
        if call.method not in {"POST", "PUT", "PATCH", "DELETE"}:
            continue
        if same_origin_api_path(call.path) is None:
            continue
        keys = json_body_keys(call.options_src)
        if keys:  # non-empty; [] means computed/empty and is not examined
            out.append((call, tuple(keys)))
    return tuple(out)


def test_body_shape_extractor_floor():
    bodies = parseable_bodies()
    assert len(bodies) >= MIN_PARSEABLE_BODIES, (
        f"only {len(bodies)} mutating call sites with a statically parseable "
        f"JSON body (floor {MIN_PARSEABLE_BODIES}) -- the body extractor has "
        f"stopped matching"
    )


def test_body_shape_extractor_controls():
    """Positive and negative controls for the key scanner itself."""
    # Positive: keys in order -- unquoted, quoted, and nested objects and
    # arrays must not leak their inner keys to the top level.
    assert json_body_keys(
        "{method: 'POST', body: JSON.stringify({"
        "  query: q, 'mode': m, metadata: {nested: 1, deeper: [{x: 2}]}"
        "})}"
    ) == ["query", "mode", "metadata"]
    # ES6 shorthand. Missing this silently under-reports the shape of every
    # `JSON.stringify({query, mode, ...})` in the codebase, which is the
    # exact way this sweep would go quietly vacuous.
    assert json_body_keys("{body: JSON.stringify({query, mode, x: 1})}") == [
        "query",
        "mode",
        "x",
    ]
    # A value that happens to be a bare identifier must NOT become a key.
    assert json_body_keys("{body: JSON.stringify({a: b, c: d})}") == ["a", "c"]
    # Spread and computed keys are reported, so a partially-known shape is
    # never mistaken for a complete one.
    assert json_body_keys("{body: JSON.stringify({...base, value: v})}") == [
        SPREAD_KEY,
        "value",
    ]
    assert json_body_keys("{body: JSON.stringify({[k]: v, n: 1})}") == [
        COMPUTED_KEY,
        "n",
    ]
    # Strings containing braces or colons must not confuse the scanner.
    assert json_body_keys(
        "{body: JSON.stringify({q: 'a{b}:c,d', mode: 'x'})}"
    ) == ["q", "mode"]
    # Negative: a non-literal body is "not examined", not "clean".
    assert json_body_keys("{method: 'POST', body: payload}") is None
    assert json_body_keys("{method: 'POST', body: formData}") is None
    assert json_body_keys("{method: 'GET'}") is None


# ---------------------------------------------------------------------------
# Section B2 -- drive the real endpoints with the frontend's own key sets
#
# Each case names the JS call site it was copied from. ``good`` must equal
# the key set that call site actually sends (asserted below), so the payload
# cannot drift away from the frontend it claims to model. ``bad`` is the same
# payload with the field renamed the way a careless port renames it.
#
# Every id is deliberately non-existent, so handlers answer 404/400 instead
# of mutating anything. That is still a valid result: FastAPI validates the
# request body *before* the handler runs, so any status that is not 422
# proves the key set was accepted.
# ---------------------------------------------------------------------------

NOPE_UUID = "00000000-0000-0000-0000-000000000000"


@dataclass(frozen=True)
class LiveCase:
    js_ref: str  # "<rel_path>:<lineno>" of the call site modelled
    method: str
    url: str
    good: dict
    bad: dict

    @property
    def label(self) -> str:
        return f"{self.method} {self.url.split('?')[0]}"


def _unique_js_ref(rel_path: str, url_expr: str) -> str:
    """Locate a unique fetch by its source and URL expression, not its line."""
    matches = [
        call.where
        for call in all_fetch_calls()
        if call.rel_path == rel_path and call.url_expr == url_expr
    ]
    if len(matches) != 1:
        raise AssertionError(
            f"expected one {url_expr} fetch in {rel_path}, found {matches}"
        )
    return matches[0]


LIVE_CASES = (
    LiveCase(
        js_ref="components/settings_sync.js:16",
        method="PUT",
        url="/settings/api/llm.temperature",
        good={"value": 0.5},
        bad={"val": 0.5},
    ),
    LiveCase(
        js_ref=_unique_js_ref(
            "components/research.js",
            "URLS.SETTINGS_API.SEARCH_FAVORITES_TOGGLE",
        ),
        method="POST",
        url="/settings/api/search-favorites/toggle",
        good={"engine_id": "zzz_no_such_engine"},
        bad={"engineId": "zzz_no_such_engine"},
    ),
    LiveCase(
        js_ref="followup.js:311",
        method="POST",
        url="/api/followup/prepare",
        good={"parent_research_id": NOPE_UUID, "question": "q"},
        bad={"research_id": NOPE_UUID, "q": "q"},
    ),
    LiveCase(
        # NB: with a non-existent card id the handler answers 500, not 404.
        # That is a separate defect (unguarded lookup) and out of scope here;
        # what matters for *this* test is that 500 means the body passed
        # validation and the handler ran.
        js_ref="pages/news.js:1507",
        method="POST",
        url=f"/news/api/feedback/{NOPE_UUID}",
        good={"vote": "up"},
        bad={"rating": "up"},
    ),
    LiveCase(
        # Located by its URL expression, like the research.js case above:
        # chat.js keeps growing above this call (the ?q= pre-fill rules
        # moved it from line 2631), and the line is not what is modelled.
        js_ref=_unique_js_ref(
            "components/chat.js",
            "`/api/chat/sessions/${encodeURIComponent(sessionIdAtRequest)}`",
        ),
        method="PATCH",
        url=f"/api/chat/sessions/{NOPE_UUID}",
        good={"title": "t"},
        bad={"name": "t"},
    ),
    LiveCase(
        js_ref="pages/note-detail.js:2909",
        method="POST",
        url=f"/notes/api/notes/{NOPE_UUID}/accept-link",
        good={"target_note_id": NOPE_UUID},
        bad={"targetNoteId": NOPE_UUID},
    ),
    LiveCase(
        js_ref="components/save_to_collection.js:197",
        method="POST",
        url=f"/library/api/research/{NOPE_UUID}/add-to-collection",
        good={"collection_id": 999999},
        bad={"collectionId": 999999},
    ),
    LiveCase(
        js_ref="pages/note-detail.js:1743",
        method="POST",
        url=f"/notes/api/notes/{NOPE_UUID}/research",
        good={"research_id": NOPE_UUID},
        bad={"researchId": NOPE_UUID},
    ),
)


def concrete_url_matches_js_pattern(url: str, pattern: str) -> bool:
    """Is ``url`` an instance of the JS-resolved path ``pattern``?"""
    left = url.strip("/").split("/")
    right = pattern.strip("/").split("/")
    if len(left) != len(right):
        return False
    return all(WILD in b or a == b for a, b in zip(left, right))


def test_live_cases_match_the_javascript_they_model():
    """Anti-drift: each case's payload is the frontend's real key set.

    Without this the live section would test a payload shape invented here,
    which is worth nothing.
    """
    by_ref = {call.where: keys for call, keys in parseable_bodies()}
    missing = [c.js_ref for c in LIVE_CASES if c.js_ref not in by_ref]
    assert not missing, (
        f"live cases reference call sites the extractor no longer sees: "
        f"{missing}. Either the JS moved (update the line numbers) or the "
        f"body extractor regressed."
    )
    by_where = {c.where: c for c in all_fetch_calls()}
    for case in LIVE_CASES:
        assert set(by_ref[case.js_ref]) == set(case.good), (
            f"{case.js_ref} sends {sorted(by_ref[case.js_ref])} but this "
            f"test drives the endpoint with {sorted(case.good)}"
        )
        call = by_where[case.js_ref]
        assert call.method == case.method, (
            f"{case.js_ref} uses {call.method}, this test uses {case.method}"
        )
        assert concrete_url_matches_js_pattern(case.url, call.path), (
            f"{case.js_ref} calls {call.path.replace(WILD, '*')!r}, but this "
            f"test drives {case.url!r}"
        )
        # The bad payload must differ from the good one in *name* only, or
        # a rejection would not be evidence about field names.
        assert set(case.bad) != set(case.good), case.js_ref
        assert len(case.bad) == len(case.good), case.js_ref


def test_live_case_count_floor():
    assert len(LIVE_CASES) >= MIN_VALIDATING_ENDPOINTS, (
        f"only {len(LIVE_CASES)} endpoints driven live "
        f"(floor {MIN_VALIDATING_ENDPOINTS})"
    )


# ---------------------------------------------------------------------------
# Section B3 -- the live run
# ---------------------------------------------------------------------------

#: This app never actually emits 422 for a bad body: every handler exercised
#: below reads ``await request.json()`` and validates by hand, answering 400
#: with a message naming the missing field. Both statuses count as "rejected
#: for shape" so the check survives a later move to Pydantic models.
REJECTED_FOR_SHAPE = frozenset({400, 422})


@pytest.fixture(scope="module")
def auth_client():
    """One registered, logged-in, CSRF-armed client for the whole module.

    Module-scoped on purpose: a fresh user per case would bootstrap an
    encrypted database each time and exhaust the connection pool.
    """
    client = TestClient(fastapi_app, raise_server_exceptions=False)
    username = f"js_parity_{uuid.uuid4().hex[:8]}"
    password = "TestPassword123!"  # noqa: S105

    def csrf() -> str:
        client.get("/auth/login")
        resp = client.get("/auth/csrf-token")
        return (
            resp.json().get("csrf_token", "") if resp.status_code == 200 else ""
        )

    client.post(
        "/auth/register",
        data={
            "username": username,
            "password": password,
            "confirm_password": password,
            "acknowledge": "true",
            "csrf_token": csrf(),
        },
        follow_redirects=False,
    )
    login = client.post(
        "/auth/login",
        data={"username": username, "password": password, "csrf_token": csrf()},
        follow_redirects=False,
    )
    if login.status_code != 302:
        pytest.fail(
            f"login bootstrap failed: {login.status_code} {login.text[:300]}"
        )
    token = client.get("/auth/csrf-token").json().get("csrf_token")
    assert token, "no CSRF token issued; every mutating request below would 403"
    client.headers.update({"X-CSRFToken": token})

    yield client

    client.post("/auth/logout", follow_redirects=False)


@pytest.mark.parametrize("case", LIVE_CASES, ids=lambda c: c.js_ref)
def test_frontend_field_names_are_the_ones_the_endpoint_reads(
    auth_client, case
):
    """The frontend's own key set must not be rejected for shape.

    The mis-named payload goes first and is *required* to be rejected. That
    is the anti-vacuity control: without it, an endpoint that ignores its
    body entirely would make the second assertion pass for free.
    """
    bad = auth_client.request(case.method, case.url, json=case.bad)
    assert bad.status_code in REJECTED_FOR_SHAPE, (
        f"CONTROL FAILED for {case.label}: a payload with the fields renamed "
        f"to {sorted(case.bad)} was answered {bad.status_code}, not "
        f"{sorted(REJECTED_FOR_SHAPE)}. This endpoint does not validate its "
        f"body, so the real-payload assertion below would prove nothing. "
        f"Body: {bad.text[:300]!r}"
    )

    good = auth_client.request(case.method, case.url, json=case.good)
    assert good.status_code not in REJECTED_FOR_SHAPE, (
        f"{case.js_ref} sends {sorted(case.good)} to {case.label}, and the "
        f"endpoint rejects it for shape ({good.status_code}). The button is "
        f"dead: the field names were renamed on one side of the port only. "
        f"Body: {good.text[:300]!r}"
    )


# ---------------------------------------------------------------------------
# Section C -- the defect this sweep found, pinned
# ---------------------------------------------------------------------------

_LOG_EXPORT_FIX = """
mechanism: components/logpanel.js does a HEAD pre-flight before creating the
download anchor:

    const res = await fetch(exportUrl, { method: 'HEAD' });
    if (!res.ok) { ...showAlert...; return; }

Under Flask this worked, because werkzeug adds HEAD to every rule that has
GET (werkzeug/routing/rules.py: `if "HEAD" not in methods and "GET" in
methods: methods.add("HEAD")`). The port declares the endpoint as
`@router.get("/api/research/{research_id}/logs/export")`, and FastAPI's
APIRoute takes `methods` literally (fastapi/routing.py: `self.methods =
{method.upper() for method in methods}`) -- it never adds HEAD. So the
pre-flight is answered 405 (Allow: GET), `res.ok` is false, the handler
shows "Failed to export logs (HTTP 405)" and RETURNS. The anchor is never
created and the export never starts: the button is dead for every user.

FIXED in this commit by declaring the route for both verbs --

    @router.api_route(
        "/api/research/{research_id}/logs/export", methods=["GET", "HEAD"]
    )

(or add a `@router.head` twin). Dropping the JS pre-flight also works but
loses the 404/429 guard it exists for.
"""


def test_log_export_head_preflight_is_not_405(auth_client):
    """The log-export pre-flight must reach the endpoint, not bounce off 405.

    The research id need not exist: method dispatch happens before the
    handler runs, so a working endpoint answers 404 here and a verb it does
    not serve answers 405.
    """
    resp = auth_client.head("/api/research/1/logs/export")
    assert resp.status_code != 405, (
        f"HEAD /api/research/1/logs/export -> {resp.status_code} "
        f"(Allow: {resp.headers.get('allow')!r}); the export button's "
        f"pre-flight fails and it returns before starting the download"
    )


def test_log_export_get_still_works():
    """Companion to the xfail: the endpoint itself is fine, only the verb.

    Asserted against the route table so this costs no request; the live GET
    is exercised by the router's own tests.
    """
    matches = routes_matching("/api/research/" + WILD + "/logs/export")
    assert matches, "the log-export route vanished entirely"
    allowed = set().union(*(m[1] for m in matches))
    assert "GET" in allowed, allowed
    # Both verbs, deliberately: see _LOG_EXPORT_FIX. Dropping HEAD again
    # would silently kill the download button, so it is pinned here rather
    # than left to the live probe alone.
    assert "HEAD" in allowed, (
        f"HEAD is no longer served ({sorted(allowed)}); the log-export "
        "pre-flight in components/logpanel.js:1854 will 405 and the "
        "download button goes dead again"
    )