Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

45.2 KB
Raw Blame
"""The full browser-vs-API matrix for every registered exception handler.

``_register_exception_handlers`` (``web/fastapi_app.py``) installs nine
error paths -- ``HTTPException``, status-code ``404``, the catch-all
``Exception``, ``json.JSONDecodeError``, ``WebAPIException``,
``NewsAPIException``, ``PolicyDeniedError``, plus the two hand-rolled
misses in ``favicon`` / ``serve_static``. Whether each one answers a
browser navigation with HTML or a programmatic caller with JSON has been
got wrong three separate times on this branch:

1. **The 404 handler did not differentiate at all.** Every miss returned
   ``{"error": "Not found"}``, so a user who mistyped a URL or followed a
   stale bookmark landed in the browser's raw JSON viewer. Flask's
   ``@app.errorhandler(404)`` branched and returned ``make_response("Not
   found", 404)`` as ``text/html``; the sibling 401 handler already
   branched too. Fixed -- ``Test404HandlerMatrix`` pins both branches.
2. **Four library page routes ``return``ed a ``JSONResponse`` instead of
   raising**, so they never reached the fixed handler and kept showing
   browsers JSON. ``return`` hands Starlette a finished response;
   only ``raise`` enters the exception-handler stack at all. Fixed --
   ``test_returning_a_404_response_bypasses_the_handler_entirely``
   demonstrates the mechanism on a throwaway app so the distinction
   cannot quietly stop being true.
3. **The ``JSONDecodeError`` handler interpolated the exception.**
   ``json.JSONDecodeError`` carries the ENTIRE offending document on
   ``.doc``, so one ``__str__`` change away it would have written a
   request body -- or a downstream provider's response -- into the log.
   Fixed to log bounded fields (``.msg``/``.lineno``/``.colno``) --
   ``test_the_offending_document_never_reaches_the_log`` pins it with a
   positive control proving the log line still says something useful.

Beyond differentiation this file pins, per handler: that no internal
detail (exception text, traceback, source path, SQL) escapes into a
response body; that the status code each handler claims is the status it
returns; that the catch-all re-stamps the security headers, because
Starlette's ``ServerErrorMiddleware`` is installed OUTSIDE every
``add_middleware`` layer and its response never passes through
``SecurityHeadersMiddleware``; and that the 401 browser redirect
preserves the ``next`` target.

Relationship to ``test_exception_handler_contract.py``: that file pins
the per-handler *body shapes* (404 JSON/HTML, 405 + ``Allow``, 401
redirect encoding, the scrubbed 500, ``WebAPIException`` /
``AuthenticationRequiredError`` / ``PolicyDeniedError`` envelopes, the
malformed-JSON 400) and unit-tests ``_is_api_request`` directly. This
file is the orthogonal axis: the same *condition* driven down both the
browser and the API branch of every handler, so a future edit that fixes
one branch and forgets the other fails here. ``NewsAPIException``,
``favicon`` / ``serve_static``, the security-header re-stamping, the
``HTTPException(404, detail=...)`` swallowing and the ``.doc`` logging
leak are covered nowhere else.

Two behaviours pinned here are known defects, flagged as such at their
assertions rather than blessed: the 401 redirect drops the query string
from ``next=``, and ``favicon`` / ``serve_static`` answer a browser with
JSON for the same ``return``-instead-of-``raise`` reason as bug 2.
"""

import json

import pytest
from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import JSONResponse
from fastapi.testclient import TestClient
from starlette.middleware.sessions import SessionMiddleware

from local_deep_research.web.exceptions import WebAPIException
from local_deep_research.web.fastapi_app import (
    SecurityHeadersMiddleware,
    _register_exception_handlers,
)

# Distinct markers so a leak can be attributed to the thing that leaked.
RAISE_SITE_SECRET = "raise-site-secret-xyzzy"
BODY_SECRET = "request-body-secret-xyzzy"
TARGET_SECRET = "egress-target-secret-xyzzy"
SQL_FRAGMENT = "SELECT api_key FROM user_settings"

# Headers every error response must carry, with the substring of each
# value that proves it is the real policy and not an empty string. The
# catch-all 500 stamps these itself; the other handlers get them from
# SecurityHeadersMiddleware on the way out.
#
# Written out literally rather than read back from
# SecurityHeadersMiddleware.unconditional_headers(): asserting a handler's
# output equals the very helper that handler calls proves nothing.
EXPECTED_ERROR_HEADERS = {
    "content-security-policy": "default-src 'self'",
    "x-frame-options": "SAMEORIGIN",
    "x-content-type-options": "nosniff",
    "referrer-policy": "strict-origin-when-cross-origin",
    "cross-origin-opener-policy": "same-origin",
    "cache-control": "no-store",
    "pragma": "no-cache",
    "expires": "0",
}


# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------


def _matrix_app() -> FastAPI:
    """A throwaway app wired through the REAL handler registration.

    Every failure mode is exposed twice -- once under an ``/api/`` path
    and once under a browser-shaped page path -- so the two branches of
    each handler can be driven from identical conditions. Real endpoints
    cannot do this: no production route can be made to raise
    ``NewsAPIException`` on demand under both path shapes.
    """
    app = FastAPI()
    _register_exception_handlers(app)

    def _policy_error():
        from local_deep_research.security.egress.policy import (
            Decision,
            PolicyDeniedError,
        )

        return PolicyDeniedError(
            Decision(allowed=False, reason="scope_mismatch"),
            target=f"https://internal.example/{TARGET_SECRET}",
        )

    # --- catch-all Exception -------------------------------------------
    @app.get("/api/boom")
    @app.get("/page/boom")
    async def boom():
        raise ValueError(
            f"{RAISE_SITE_SECRET}: {SQL_FRAGMENT} "
            f"(/srv/ldr/private/settings_service.py)"
        )

    # --- HTTPException, 401 and other codes -----------------------------
    @app.get("/api/needs-auth")
    @app.get("/page/needs-auth")
    @app.get("/page/needs-auth/deep")
    async def needs_auth():
        raise HTTPException(status_code=401, detail="Authentication required")

    @app.get("/api/forbidden")
    @app.get("/page/forbidden")
    async def forbidden():
        raise HTTPException(status_code=403, detail="Forbidden")

    @app.get("/api/challenge")
    @app.get("/page/challenge")
    @app.get("/api/v1")
    @app.get("/api/v1/challenge")
    async def challenge():
        raise HTTPException(
            status_code=401,
            detail="Authentication required",
            headers={"WWW-Authenticate": 'Bearer realm="ldr"'},
        )

    @app.get("/api/retry")
    async def retry_later():
        raise HTTPException(
            status_code=429,
            detail="Slow down",
            headers={"Retry-After": "17"},
        )

    @app.get("/api/teapot")
    @app.get("/api/v11/teapot")
    @app.get("/api/v1evil/teapot")
    async def teapot():
        raise HTTPException(status_code=418, detail="I am a teapot")

    # --- 404 -------------------------------------------------------------
    @app.get("/api/raises-404")
    @app.get("/page/raises-404")
    async def raises_404():
        raise HTTPException(
            status_code=404, detail=f"Document {RAISE_SITE_SECRET} not found"
        )

    @app.get("/page/returns-404")
    async def returns_404():
        # The bug-2 shape, preserved deliberately on this throwaway app as
        # the executable counter-example. Never copy this into a router.
        return JSONResponse({"error": "Not found"}, status_code=404)

    # --- JSONDecodeError --------------------------------------------------
    @app.post("/api/echo")
    @app.post("/page/echo")
    async def echo(request: Request):
        return {"ok": await request.json()}

    @app.get("/api/downstream-json")
    async def downstream_json():
        # A JSONDecodeError raised deep inside a handler, nowhere near the
        # request body: parsing a malformed response from an upstream
        # provider. Same exception class, entirely different fault owner.
        json.loads(f'{{"upstream": {RAISE_SITE_SECRET}')

    # --- WebAPIException --------------------------------------------------
    @app.get("/api/webapi")
    @app.get("/page/webapi")
    async def webapi():
        raise WebAPIException(
            "Upstream provider rejected the request",
            status_code=429,
            error_code="RATE_LIMITED",
        )

    # --- NewsAPIException -------------------------------------------------
    @app.get("/api/news")
    @app.get("/page/news")
    async def news():
        from local_deep_research.news.exceptions import (
            NewsFeatureDisabledException,
        )

        raise NewsFeatureDisabledException()

    # --- PolicyDeniedError ------------------------------------------------
    @app.get("/api/policy")
    @app.get("/page/policy")
    async def policy():
        raise _policy_error()

    # --- success control --------------------------------------------------
    @app.get("/api/fine")
    @app.get("/page/fine")
    async def fine():
        return {"ok": True}

    return app


@pytest.fixture(scope="module")
def matrix_client():
    """Handlers only -- no middleware, so header assertions are unambiguous."""
    return TestClient(_matrix_app(), raise_server_exceptions=False)


@pytest.fixture(scope="module")
def stamped_client():
    """Real handlers PLUS ``SecurityHeadersMiddleware``, as production runs."""
    app = _matrix_app()
    app.add_middleware(SecurityHeadersMiddleware)
    return TestClient(app, raise_server_exceptions=False)


@pytest.fixture(scope="module")
def unstamped_client():
    """Negative control for the header re-stamping.

    Identical stack -- ``SecurityHeadersMiddleware`` installed the same
    way -- but the catch-all does NOT stamp the headers itself. If the
    middleware covered 500s, this app's 500 would carry them too and
    ``stamped_client``'s passing assertions would prove nothing about the
    handler.
    """
    app = FastAPI()

    @app.exception_handler(Exception)
    async def naive_handler(request: Request, exc):
        return JSONResponse({"error": "Server error"}, status_code=500)

    @app.get("/boom")
    async def boom():
        raise ValueError("x")

    @app.get("/fine")
    async def fine():
        return {"ok": True}

    app.add_middleware(SecurityHeadersMiddleware)
    return TestClient(app, raise_server_exceptions=False)


@pytest.fixture(scope="module")
def real_client():
    """The real app: whole middleware + handler + routing stack."""
    from local_deep_research.web.fastapi_app import app

    return TestClient(app, raise_server_exceptions=False)


@pytest.fixture(scope="module")
def sessioned_client():
    """Real handlers, WITH ``SessionMiddleware`` installed -- unlike
    ``matrix_client``, which deliberately omits it so
    ``_render_branded_error_page`` falls back to plain text (see the
    fallback tests above). Two test-only routes drive session state
    directly, the same "real handler registration on a throwaway app"
    technique ``_matrix_app()`` uses, since no anonymous real route both
    queues a flash and is reachable without authentication.
    """
    from local_deep_research.web.dependencies.flash import (
        flash,
        get_flashed_messages,
    )

    app = _matrix_app()
    app.add_middleware(
        SessionMiddleware, secret_key="test-only-secret-key-not-for-prod"
    )

    @app.get("/__set_flash__")
    async def _set_flash(request: Request):
        flash(request, "Settings saved.", "success")
        return {"ok": True}

    @app.get("/__peek_flash__")
    async def _peek_flash(request: Request):
        return {"flashes": get_flashed_messages(request, with_categories=True)}

    return TestClient(app, raise_server_exceptions=False)


BROWSER = {"Accept": "text/html,application/xhtml+xml,*/*;q=0.8"}
API = {"Accept": "application/json"}


def _assert_no_internal_detail(response):
    """Nothing from the raise site may appear in the body.

    Always call alongside a positive assertion on the body -- on its own
    this passes for an empty response.
    """
    body = response.text
    assert RAISE_SITE_SECRET not in body, "exception message reached the client"
    assert SQL_FRAGMENT not in body, "SQL text reached the client"
    assert "Traceback" not in body, "traceback reached the client"
    assert "/home/" not in body, "server filesystem path reached the client"
    assert ".py" not in body, "source filename reached the client"
    assert "ValueError" not in body, "exception class name reached the client"


# ---------------------------------------------------------------------------
# HTTPException handler
# ---------------------------------------------------------------------------


class TestHTTPExceptionHandlerMatrix:
    """``handle_http_exception`` -- the only handler that redirects."""

    def test_browser_401_redirects_to_login_preserving_next(
        self, matrix_client
    ):
        resp = matrix_client.get(
            "/page/needs-auth/deep", headers=BROWSER, follow_redirects=False
        )
        assert resp.status_code == 302
        assert (
            resp.headers["location"] == "/auth/login?next=/page/needs-auth/deep"
        )

    def test_api_path_401_is_json_even_for_a_browser_accept(
        self, matrix_client
    ):
        # Mechanism 1 of _is_api_request: an "/api/" path segment, which
        # wins outright over whatever Accept says.
        resp = matrix_client.get(
            "/api/needs-auth", headers=BROWSER, follow_redirects=False
        )
        assert resp.status_code == 401
        assert resp.json() == {"detail": "Authentication required"}
        assert "location" not in resp.headers

    def test_json_accept_401_on_a_page_path_is_json(self, matrix_client):
        # Mechanism 2: Accept: application/json with no text/html, which
        # is how the frontend's fetch() calls a non-/api/ route. Same
        # path as the redirect test above -- only the header differs, so
        # this isolates the header mechanism.
        resp = matrix_client.get(
            "/page/needs-auth", headers=API, follow_redirects=False
        )
        assert resp.status_code == 401
        assert resp.json() == {"detail": "Authentication required"}
        assert "location" not in resp.headers

    def test_mixed_accept_stays_on_the_browser_branch(self, matrix_client):
        # A fetch() that lists both types must NOT be mistaken for an API
        # caller: text/html present means a browser is involved.
        resp = matrix_client.get(
            "/page/needs-auth",
            headers={"Accept": "application/json, text/html"},
            follow_redirects=False,
        )
        assert resp.status_code == 302

    @pytest.mark.parametrize(
        ("path", "headers", "expected_status"),
        [
            ("/api/challenge", API, 401),
            ("/api/v1", BROWSER, 401),
            ("/api/v1/challenge", API, 401),
            ("/page/challenge", BROWSER, 302),
        ],
    )
    def test_authenticate_header_survives_every_401_branch(
        self, matrix_client, path, headers, expected_status
    ):
        resp = matrix_client.get(path, headers=headers, follow_redirects=False)
        assert resp.status_code == expected_status
        assert resp.headers["www-authenticate"] == 'Bearer realm="ldr"'

    def test_exact_external_api_root_uses_the_v1_error_envelope(
        self, matrix_client
    ):
        resp = matrix_client.get("/api/v1", headers=BROWSER)
        assert resp.status_code == 401
        assert resp.json() == {
            "error": "Authentication required",
            "detail": "Authentication required",
        }
        assert "location" not in resp.headers

    def test_retry_after_header_survives_json_handler(self, matrix_client):
        resp = matrix_client.get("/api/retry", headers=API)
        assert resp.status_code == 429
        assert resp.json() == {"detail": "Slow down"}
        assert resp.headers["retry-after"] == "17"

    @pytest.mark.parametrize("path", ["/api/v11/teapot", "/api/v1evil/teapot"])
    def test_external_api_envelope_matches_on_a_path_segment_boundary(
        self, matrix_client, path
    ):
        resp = matrix_client.get(path, headers=API)
        assert resp.status_code == 418
        assert resp.json() == {"detail": "I am a teapot"}

    def test_next_target_keeps_the_query_string(self, matrix_client):
        # Fixed (was a KNOWN GAP): the handler now builds next= from
        # request.url.path + "?" + request.url.query, matching Flask's
        # @login_required (which used request.url, the full URL). A user
        # bounced to login from /library/?collection=42&page=3 is
        # returned there, not to a bare /library/. The query is
        # URL-encoded as a single opaque value so it survives as one
        # well-formed ?next= parameter.
        resp = matrix_client.get(
            "/page/needs-auth?collection=42&page=3",
            headers=BROWSER,
            follow_redirects=False,
        )
        assert resp.status_code == 302
        assert (
            resp.headers["location"]
            == "/auth/login?next=/page/needs-auth%3Fcollection%3D42%26page%3D3"
        )

    def test_real_app_401_redirect_also_keeps_the_query_string(
        self, real_client
    ):
        # The same fix on the real stack, so nobody assumes it is an
        # artefact of the throwaway app.
        resp = real_client.get(
            "/settings/?tab=llm", headers=BROWSER, follow_redirects=False
        )
        assert resp.status_code == 302
        assert (
            resp.headers["location"]
            == "/auth/login?next=/settings/%3Ftab%3Dllm"
        )

    @pytest.mark.parametrize("path", ["/api/forbidden", "/page/forbidden"])
    def test_non_401_codes_are_json_on_both_branches(self, matrix_client, path):
        # Differentiation is scoped to 401 by design -- only 401 has a
        # meaningful HTML destination (the login page). Everything else
        # gets the same JSON envelope whatever the caller looks like.
        resp = matrix_client.get(path, headers=BROWSER)
        assert resp.status_code == 403
        assert resp.headers["content-type"].startswith("application/json")
        assert resp.json() == {"detail": "Forbidden"}

    def test_status_code_is_carried_through_verbatim(self, matrix_client):
        resp = matrix_client.get("/api/teapot")
        assert resp.status_code == 418
        assert resp.json() == {"detail": "I am a teapot"}


# ---------------------------------------------------------------------------
# 404 handler -- bug 1 and bug 2
# ---------------------------------------------------------------------------


class Test404HandlerMatrix:
    def test_browser_miss_gets_html(self, matrix_client):
        resp = matrix_client.get("/page/no-such-thing", headers=BROWSER)
        assert resp.status_code == 404
        assert resp.headers["content-type"].startswith("text/html")
        # Positive control: the branch produces the real Flask-parity
        # body, not an empty 404 that would satisfy a "no JSON" check.
        assert resp.text == "Not found"

    def test_api_miss_gets_json(self, matrix_client):
        resp = matrix_client.get("/api/no-such-thing", headers=API)
        assert resp.status_code == 404
        assert resp.headers["content-type"].startswith("application/json")
        assert resp.json() == {"error": "Not found"}

    def test_api_path_forces_json_despite_a_browser_accept(self, matrix_client):
        # Path mechanism, isolated: browser Accept, /api/ path.
        resp = matrix_client.get("/api/no-such-thing", headers=BROWSER)
        assert resp.status_code == 404
        assert resp.headers["content-type"].startswith("application/json")
        assert resp.json() == {"error": "Not found"}

    def test_json_accept_forces_json_on_a_page_path(self, matrix_client):
        # Header mechanism, isolated: same path as the HTML test above.
        resp = matrix_client.get("/page/no-such-thing", headers=API)
        assert resp.status_code == 404
        assert resp.headers["content-type"].startswith("application/json")
        assert resp.json() == {"error": "Not found"}

    def test_path_ending_in_api_counts_as_api(self, matrix_client):
        # _is_api_request also matches a path that *ends* with "/api"
        # (no trailing segment), e.g. /settings/api. Pinned over HTTP
        # because the endswith() branch is easy to drop when the "/api/"
        # substring check is refactored.
        resp = matrix_client.get("/settings/api", headers=BROWSER)
        assert resp.status_code == 404
        assert resp.headers["content-type"].startswith("application/json")

    @pytest.mark.parametrize(
        ("path", "headers", "content_type"),
        [
            ("/api/raises-404", API, "application/json"),
            ("/page/raises-404", BROWSER, "text/html"),
        ],
    )
    def test_custom_404_detail_is_swallowed(
        self, matrix_client, path, headers, content_type
    ):
        # PINNED BEHAVIOUR, NOT AN ENDORSEMENT. A router that writes
        #     raise HTTPException(404, detail="Document not found")
        # never sees that detail reach the client. Starlette resolves
        # handlers by status code BEFORE walking the exception's class
        # MRO, so the @app.exception_handler(404) registration wins over
        # @app.exception_handler(HTTPException) -- and the 404 handler
        # takes `exc` and ignores it, emitting the fixed body.
        #
        # Consequence: a 404 detail is a silent no-op everywhere in this
        # app. A route that needs to tell the user WHICH thing was
        # missing must return its own response (as the library page
        # routes now do with HTMLResponse) rather than raise.
        resp = matrix_client.get(path, headers=headers)
        assert resp.status_code == 404
        assert resp.headers["content-type"].startswith(content_type)
        assert RAISE_SITE_SECRET not in resp.text, (
            "the 404 handler started echoing exc.detail -- if that is "
            "intended, the reflected-path XSS guards in "
            "test_exception_handler_contract.py must be re-examined first"
        )
        assert resp.text in ("Not found", '{"error":"Not found"}')

    def test_returning_a_404_response_bypasses_the_handler_entirely(
        self, matrix_client
    ):
        # Executable demonstration of bug 2's mechanism. /page/raises-404
        # and /page/returns-404 differ ONLY in `raise` vs `return`, and
        # only the raising one reaches the handler that branches on
        # Accept. This is why fixing the 404 handler in bug 1 did not fix
        # the four library page routes: a returned response is already
        # final and never enters the exception-handler stack.
        raised = matrix_client.get("/page/raises-404", headers=BROWSER)
        returned = matrix_client.get("/page/returns-404", headers=BROWSER)

        assert raised.status_code == returned.status_code == 404
        assert raised.headers["content-type"].startswith("text/html")
        assert returned.headers["content-type"].startswith(
            "application/json"
        ), (
            "a returned JSONResponse must stay JSON -- if this ever "
            "becomes HTML, Starlette started routing returned responses "
            "through exception handlers and the library-route fix's "
            "premise has changed"
        )

    def test_real_app_unrouted_paths_differentiate(self, real_client):
        browser = real_client.get("/no-such-page-xyzzy", headers=BROWSER)
        api = real_client.get("/api/no-such-route-xyzzy", headers=API)

        assert browser.status_code == api.status_code == 404
        assert browser.headers["content-type"].startswith("text/html")
        # Real app -> the standalone `pages/error.html` page (PR #5424)
        # renders successfully here, not the bare "Not found" fallback text
        # used when rendering isn't possible (see the throwaway-app tests
        # above, which pin that fallback body because their bare `FastAPI()`
        # app has no SessionMiddleware, so `request.session` raises inside
        # `_LDRTemplates.TemplateResponse` and the render is swallowed).
        assert "data-error-page" in browser.text
        assert api.json() == {"error": "Not found"}

    def test_anonymous_non_api_404_sets_no_session_cookie(
        self, sessioned_client
    ):
        """Regression guard for PR #5424's tidy-round fix.

        ``_LDRTemplates.TemplateResponse`` (template_config.py) mints and
        stores a CSRF token in the session -- and merely *reading*
        ``request.session`` marks it accessed, which makes
        SessionMiddleware add ``Vary: Cookie`` -- unless the render
        context already supplies ``csrf_token``/the other injected keys.
        Before the fix, a visitor who mistyped a URL or hit a stale
        bookmark got a session established (Set-Cookie) and a
        cache-fragmenting ``Vary: Cookie`` on what should be a stateless
        miss. On head (9afa3c45f) this failed: anonymous GET /page/... got
        both ``Set-Cookie: session=...`` and ``Vary: Cookie``.
        """
        c = sessioned_client
        c.cookies.clear()
        resp = c.get("/page/no-such-thing", headers=BROWSER)
        assert resp.status_code == 404
        assert "data-error-page" in resp.text  # positive control
        assert "set-cookie" not in resp.headers
        assert "cookie" not in resp.headers.get("vary", "").lower()

    def test_pending_flash_survives_a_404(self, sessioned_client):
        """Regression guard: a 404 must not consume unrelated flashes.

        A flash queued before an unrelated navigation miss (e.g. "Settings
        saved" followed by a mistyped/stale link) must still reach the
        NEXT real page. Before the fix, rendering the branded error page
        called ``get_flashed_messages()``, which POPS ``_flashes`` off the
        session -- on head (9afa3c45f) the flash queued here did not
        survive the intervening 404: ``/__peek_flash__`` came back empty.
        """
        c = sessioned_client
        c.cookies.clear()
        assert c.get("/__set_flash__").status_code == 200

        miss = c.get("/page/no-such-thing", headers=BROWSER)
        assert miss.status_code == 404
        assert "data-error-page" in miss.text  # positive control

        peeked = c.get("/__peek_flash__")
        assert peeked.json() == {"flashes": [["success", "Settings saved."]]}


# ---------------------------------------------------------------------------
# Catch-all Exception handler
# ---------------------------------------------------------------------------


class TestCatchAllHandlerMatrix:
    @pytest.mark.parametrize("path", ["/api/boom", "/page/boom"])
    def test_fixed_body_on_both_branches(self, matrix_client, path):
        # The ONE handler that deliberately does not differentiate: the
        # source comment argues a raw JSON 500 is far less user-visible
        # than a raw JSON 404 (users hit 404s by mistyping URLs) and that
        # two suites pin the JSON body. Pinned here as implemented, with
        # the divergence from Flask -- which served "Server error" as
        # text/html to non-API paths -- recorded rather than hidden.
        headers = BROWSER if path.startswith("/page") else API
        resp = matrix_client.get(path, headers=headers)

        assert resp.status_code == 500
        assert resp.headers["content-type"].startswith("application/json")
        # Positive control first: the body is a real, complete envelope.
        assert resp.json() == {"error": "Server error"}
        _assert_no_internal_detail(resp)

    def test_the_scrub_is_a_boundary_not_a_swallow(
        self, matrix_client, loguru_caplog_full
    ):
        # The client-facing body says nothing; the operator's log must
        # still carry the full exception and traceback, or the handler
        # has traded a leak for an outage nobody can debug.
        resp = matrix_client.get("/api/boom", headers=API)
        assert resp.json() == {"error": "Server error"}

        assert "Unhandled exception: GET '/api/boom'" in loguru_caplog_full.text
        assert RAISE_SITE_SECRET in loguru_caplog_full.text, (
            "the catch-all must log the exception it hides from the client"
        )
        assert "Traceback" in loguru_caplog_full.text

    def test_success_responses_are_untouched(self, matrix_client):
        # Negative control for the whole handler set: registering a
        # handler for the bare Exception class must not intercept
        # ordinary traffic. Without this, a handler that 500'd everything
        # would satisfy most assertions in this file.
        for path in ("/api/fine", "/page/fine"):
            resp = matrix_client.get(path, headers=BROWSER)
            assert resp.status_code == 200
            assert resp.json() == {"ok": True}


# ---------------------------------------------------------------------------
# json.JSONDecodeError handler -- bug 3
# ---------------------------------------------------------------------------


class TestJSONDecodeErrorHandlerMatrix:
    @pytest.mark.parametrize("path", ["/api/echo", "/page/echo"])
    def test_malformed_body_is_400_on_both_branches(self, matrix_client, path):
        resp = matrix_client.post(
            path,
            content=b'{"q": "' + BODY_SECRET.encode() + b'", broken',
            headers={
                "Content-Type": "application/json",
                **(BROWSER if path.startswith("/page") else API),
            },
        )
        # Status fidelity: this is the Flask request.get_json() parity
        # fix; before the handler existed it surfaced as a 500.
        assert resp.status_code == 400
        assert resp.headers["content-type"].startswith("application/json")
        assert resp.json() == {"error": "Invalid JSON body"}
        # Non-differentiating by construction, like the catch-all: a
        # malformed body only ever comes from a programmatic caller, so
        # there is no browser navigation to render HTML for.
        assert BODY_SECRET not in resp.text, "request body echoed back"

    def test_the_offending_document_never_reaches_the_log(
        self, matrix_client, loguru_caplog
    ):
        # Bug 3. json.JSONDecodeError.doc is the WHOLE document that
        # failed to parse -- here a request body. The handler must log
        # only bounded fields (.msg, .lineno, .colno), never `exc` and
        # never `exc.doc`; interpolating the exception was one __str__
        # change away from writing request bodies (or a provider's
        # response, including its credentials) into the log file, the DB
        # log sink and every connected Socket.IO client.
        secret_body = b'{"password": "' + BODY_SECRET.encode() + b'", not-valid'
        resp = matrix_client.post(
            "/api/echo",
            content=secret_body,
            headers={"Content-Type": "application/json"},
        )
        assert resp.status_code == 400

        text = loguru_caplog.text
        # Positive control: the log line exists and is genuinely useful,
        # so the absence assertion below is not passing on an empty
        # capture or a handler that logs nothing at all.
        assert "JSON decode error handling POST '/api/echo'" in text
        assert "Expecting" in text, (
            "the parser's own .msg must be logged -- without it the "
            "operator cannot tell a truncated body from a wrong "
            "content type"
        )
        assert "line 1 column" in text

        assert BODY_SECRET not in text, (
            "the offending document reached the log: the handler is "
            "interpolating exc or exc.doc again (bug 3)"
        )

    def test_valid_json_is_not_intercepted(self, matrix_client):
        # Negative control for this handler: it must fire only on a real
        # parse failure.
        resp = matrix_client.post("/api/echo", json={"a": 1})
        assert resp.status_code == 200
        assert resp.json() == {"ok": {"a": 1}}

    def test_a_server_side_parse_failure_also_becomes_400(self, matrix_client):
        # KNOWN WART, pinned deliberately. The handler is registered on
        # the exception CLASS, so a JSONDecodeError raised while parsing
        # an upstream provider's response -- a server fault the client
        # cannot fix -- is reported to that client as 400 "Invalid JSON
        # body", blaming the caller for someone else's malformed
        # payload. The source comment acknowledges this ("a server-side
        # fault wearing a 400") and compensates by logging every one it
        # takes -- the log assertion in
        # test_the_offending_document_never_reaches_the_log is what keeps
        # that compensation honest. Distinguishing the two cases means
        # catching at the downstream call site, not widening this handler.
        resp = matrix_client.get("/api/downstream-json", headers=API)
        assert resp.status_code == 400
        assert resp.json() == {"error": "Invalid JSON body"}
        assert RAISE_SITE_SECRET not in resp.text


# ---------------------------------------------------------------------------
# WebAPIException handler
# ---------------------------------------------------------------------------


class TestWebAPIExceptionHandlerMatrix:
    @pytest.mark.parametrize("path", ["/api/webapi", "/page/webapi"])
    def test_same_envelope_and_status_on_both_branches(
        self, matrix_client, path
    ):
        headers = BROWSER if path.startswith("/page") else API
        resp = matrix_client.get(path, headers=headers)

        # Status fidelity: the exception's own status_code, not a
        # flattened 500.
        assert resp.status_code == 429
        assert resp.headers["content-type"].startswith("application/json")
        body = resp.json()
        assert body["status"] == "error"
        assert body["error_code"] == "RATE_LIMITED"
        assert body["message"] == "Upstream provider rejected the request"

    def test_browser_and_api_bodies_are_byte_identical(self, matrix_client):
        # Explicit differentiation check rather than two separate
        # assertions that happen to agree: this handler is documented as
        # non-differentiating, and the pin is that it stays that way.
        browser = matrix_client.get("/page/webapi", headers=BROWSER)
        api = matrix_client.get("/api/webapi", headers=API)
        assert browser.text == api.text
        assert browser.status_code == api.status_code


# ---------------------------------------------------------------------------
# NewsAPIException handler
# ---------------------------------------------------------------------------


class TestNewsAPIExceptionHandlerMatrix:
    """Registered conditionally (inside a ``try: ... except ImportError``).

    Covered nowhere else in the suite, so an accidental removal of the
    registration -- or an ImportError inside ``news.exceptions`` quietly
    swallowing it, which would turn every news failure into a 500 --
    would otherwise go unnoticed.
    """

    @pytest.mark.parametrize("path", ["/api/news", "/page/news"])
    def test_status_and_envelope_on_both_branches(self, matrix_client, path):
        headers = BROWSER if path.startswith("/page") else API
        resp = matrix_client.get(path, headers=headers)

        assert resp.status_code == 503, (
            "NewsFeatureDisabledException's own 503 must survive; a 500 "
            "here means the handler registration was skipped and the "
            "catch-all took the exception instead"
        )
        assert resp.headers["content-type"].startswith("application/json")
        body = resp.json()
        assert body["error"] == "News system is disabled"
        assert body["error_code"] == "NEWS_DISABLED"
        assert body["status_code"] == 503

    def test_news_envelope_differs_from_the_webapi_envelope(
        self, matrix_client
    ):
        # Pinned divergence, not a bug: NewsAPIException.to_dict() emits
        # {"error", "error_code", "status_code"} while
        # WebAPIException.to_dict() emits {"status", "message",
        # "error_code"}. Both are ported verbatim from main's two
        # separate Flask errorhandlers. A client reading `message` off a
        # news error gets None, so the two shapes are worth stating out
        # loud before someone "unifies" them and breaks news/*.js.
        news = matrix_client.get("/api/news", headers=API).json()
        webapi = matrix_client.get("/api/webapi", headers=API).json()

        assert set(news) == {"error", "error_code", "status_code"}
        assert set(webapi) == {"status", "message", "error_code"}


# ---------------------------------------------------------------------------
# PolicyDeniedError handler
# ---------------------------------------------------------------------------


class TestPolicyDeniedHandlerMatrix:
    @pytest.mark.parametrize("path", ["/api/policy", "/page/policy"])
    def test_400_with_reason_only_on_both_branches(self, matrix_client, path):
        headers = BROWSER if path.startswith("/page") else API
        resp = matrix_client.get(path, headers=headers)

        # Status fidelity: an escaped egress denial is a clean 400, not
        # the 500 it would be without the handler.
        assert resp.status_code == 400
        assert resp.headers["content-type"].startswith("application/json")
        body = resp.json()
        assert body["status"] == "error"
        assert "scope_mismatch" in body["message"]
        # The denied target can embed a user's query or an internal
        # hostname; only the decision reason may ship.
        assert TARGET_SECRET not in resp.text
        assert "internal.example" not in resp.text

    def test_browser_and_api_bodies_are_byte_identical(self, matrix_client):
        browser = matrix_client.get("/page/policy", headers=BROWSER)
        api = matrix_client.get("/api/policy", headers=API)
        assert browser.text == api.text


# ---------------------------------------------------------------------------
# favicon / serve_static -- the two hand-rolled 404s
# ---------------------------------------------------------------------------


class TestStaticMissHandlers:
    """``favicon`` and ``serve_static`` build their own 404 responses.

    Both ``return JSONResponse({"error": "Not found"}, 404)`` -- the exact
    ``return``-instead-of-``raise`` shape of bug 2, so neither reaches the
    404 handler and neither differentiates. Pinned as-is because the
    impact is genuinely different from the library routes': these paths
    are fetched as subresources (``<img>``, ``<script>``, the tab icon),
    where the browser never renders the body, so a JSON miss is invisible
    to the user. Worth revisiting only if a static path ever becomes a
    navigation target.
    """

    def test_missing_static_asset_is_json_even_for_a_browser(self, real_client):
        resp = real_client.get(
            "/static/no-such-asset-xyzzy.js", headers=BROWSER
        )
        assert resp.status_code == 404
        assert resp.headers["content-type"].startswith("application/json")
        assert resp.json() == {"error": "Not found"}

    def test_static_miss_does_not_reflect_the_requested_path(self, real_client):
        # The miss body is a fixed constant: no echo of the caller's path
        # (this route is unauthenticated and rate-limit exempt, so it is
        # the cheapest reflection surface in the app) and no filesystem
        # detail about where the static root actually lives.
        resp = real_client.get(
            "/static/deep/nested/no-such-asset-xyzzy.js", headers=BROWSER
        )
        assert resp.status_code == 404
        assert "xyzzy" not in resp.text
        assert "/home/" not in resp.text

    def test_existing_static_asset_still_serves(self, real_client):
        # Positive control: the miss assertions above would also pass on
        # a route that had stopped serving anything at all.
        resp = real_client.get("/static/favicon.png", headers=BROWSER)
        assert resp.status_code == 200
        assert (
            resp.headers["cache-control"]
            == "public, max-age=0, must-revalidate"
        )
        assert len(resp.content) > 0

    def test_missing_favicon_is_a_json_404(self, real_client):
        # static/favicon.ico is absent from the tree (only favicon.png
        # ships), so this exercises the route's own miss branch. If an
        # .ico is ever added this becomes a 200 and should be re-pinned
        # against a deliberately absent path instead.
        resp = real_client.get("/favicon.ico", headers=BROWSER)
        assert resp.status_code == 404
        assert resp.json() == {"error": "Not found"}


# ---------------------------------------------------------------------------
# Security headers on error responses
# ---------------------------------------------------------------------------


class TestSecurityHeadersOnErrorResponses:
    """``ServerErrorMiddleware`` sits outside every ``add_middleware`` layer.

    Registering a handler for the bare ``Exception`` class makes Starlette
    wire it in as ``ServerErrorMiddleware``'s handler, and that middleware
    is installed by ``build_middleware_stack`` itself -- outside
    ``SecurityHeadersMiddleware``. Its response is written to the raw ASGI
    ``send``, so unless the handler stamps the headers itself a 500 ships
    with no CSP, no nosniff, no frame-options and no cache directives.
    """

    @pytest.mark.parametrize(
        ("header", "expected_substring"), sorted(EXPECTED_ERROR_HEADERS.items())
    )
    def test_catch_all_500_carries_the_header(
        self, stamped_client, header, expected_substring
    ):
        resp = stamped_client.get("/api/boom", headers=API)
        assert resp.status_code == 500
        assert expected_substring in resp.headers.get(header, "")

    def test_the_middleware_alone_does_not_cover_500s(self, unstamped_client):
        # NEGATIVE CONTROL for the test above. Same middleware, same
        # installation order, but a catch-all that does not stamp: its
        # 500 has no security headers at all, while its 200 has them.
        # Together these show the middleware is present and working AND
        # that it genuinely cannot reach the 500 -- so the passing
        # assertions above are attributable to the handler's own
        # stamping, not to middleware doing the job anyway.
        ok = unstamped_client.get("/fine")
        assert ok.status_code == 200
        assert "default-src 'self'" in ok.headers.get(
            "content-security-policy", ""
        ), "the control app's middleware is not working; test is void"

        boom = unstamped_client.get("/boom")
        assert boom.status_code == 500
        assert "content-security-policy" not in boom.headers, (
            "SecurityHeadersMiddleware started covering ServerError"
            "Middleware's responses -- if Starlette changed the stack "
            "order, the hand-stamping in the catch-all can be removed"
        )
        assert "x-content-type-options" not in boom.headers

    def test_html_404_also_carries_the_headers(self, real_client):
        # The 404 handler does NOT stamp anything itself, and does not
        # need to: it is a normal exception handler, so its response
        # travels back out through the whole middleware stack. Pinned
        # because an HTML error body without CSP/nosniff is exactly where
        # a future reflected-content mistake would become exploitable.
        resp = real_client.get("/no-such-page-xyzzy", headers=BROWSER)
        assert resp.status_code == 404
        assert "data-error-page" in resp.text
        for header, expected in EXPECTED_ERROR_HEADERS.items():
            assert expected in resp.headers.get(header, ""), (
                f"{header} missing from the browser 404"
            )

    def test_401_redirect_also_carries_the_headers(self, real_client):
        resp = real_client.get(
            "/settings/", headers=BROWSER, follow_redirects=False
        )
        assert resp.status_code == 302
        assert "default-src 'self'" in resp.headers.get(
            "content-security-policy", ""
        )
        assert resp.headers.get("x-content-type-options") == "nosniff"


# ---------------------------------------------------------------------------
# Status-code fidelity sweep
# ---------------------------------------------------------------------------


@pytest.mark.parametrize(
    ("path", "expected_status"),
    [
        ("/api/needs-auth", 401),  # HTTPException handler
        ("/api/forbidden", 403),  # HTTPException handler, non-401
        ("/api/teapot", 418),  # HTTPException handler, exotic code
        ("/api/no-such-thing", 404),  # 404 handler
        ("/api/raises-404", 404),  # 404 handler via raise
        ("/api/boom", 500),  # catch-all
        ("/api/webapi", 429),  # WebAPIException
        ("/api/news", 503),  # NewsAPIException
        ("/api/policy", 400),  # PolicyDeniedError
        ("/api/downstream-json", 400),  # JSONDecodeError
        ("/api/fine", 200),  # control
    ],
)
def test_every_handler_returns_the_status_it_claims(
    matrix_client, path, expected_status
):
    """One sweep over the whole matrix.

    Catches the failure mode where a handler is registered but shadowed
    -- an exception class that another handler already claims via MRO, or
    a status-code registration that outranks a class registration -- which
    shows up as the wrong status long before the body shape looks wrong.
    """
    resp = matrix_client.get(path, headers=API)
    assert resp.status_code == expected_status