~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
"""ASGI middleware ORDER and SECURITY-HEADER invariants (Flask -> FastAPI).
Flask had no equivalent of this: Flask's ``before_request``/``after_request``
hooks and its ``app.wsgi_app`` wrapping have no "outer vs inner" ordering
puzzle — every ``after_request`` handler always sees every response.
Starlette/FastAPI middleware is fundamentally different: ``app.add_middleware``
is **LIFO** — the *last* call wins the outermost slot — so the plain,
top-to-bottom reading order of the ``app.add_middleware(...)`` calls in
``web/fastapi_app.py`` is the exact *reverse* of the order requests/responses
actually pass through them. Swap two calls, or insert a new one on the wrong
side of an existing one, and the code still imports and runs — it just
quietly stops enforcing what its authors intended (e.g. CSRF checked before
the session that CSRF reads from is populated, or a whole class of error
response silently losing its security headers). Nothing but a test that
reads the *live* middleware chain and asserts specific behavioural
consequences of the order can catch that.
This file has four parts:
1. ``TestRegistrationOrderPinned`` — pins the exact ``app.user_middleware``
sequence (Starlette's own bookkeeping list) AND cross-checks it against
the real, built ASGI call chain. Each pinned entry documents what a
reorder would silently break.
2. ``TestEffectiveExecutionOrderBehavioral`` — proves two of those ordering
claims through real HTTP requests rather than just list inspection: a
CSRF-rejected response still carries security headers (SecurityHeaders is
outside CSRF), and CSRF validates against a session token that
SessionMiddleware — not CSRF itself — populated from the request cookie
(Session is outside CSRF).
3. ``TestSecurityHeadersOnEveryResponseClass`` — the known-gap surface: a
200 can carry every security header while an error path quietly drops
them. Six response classes are driven through the REAL app (full
middleware stack, no mocks): 200, 404, 405, 422, CSRF-403, and an
unhandled/unregistered exception -> 500.
*** REAL FINDING (see ``TestSecurityHeadersOnEveryResponseClass`` /
``test_KNOWN_GAP_...`` below): the 500 produced by an exception type
with no registered handler carries **NONE** of the security headers.
Starlette's ``ServerErrorMiddleware`` is the true outermost layer
(added automatically, outside every ``app.add_middleware`` call) and,
on an uncaught exception, sends its fallback response directly on the
*raw* ASGI ``send`` it was itself given — bypassing every layer this
app registered, ``SecurityHeadersMiddleware`` included. This is
PINNED, not swept under a weaker assertion — see that test's docstring
for the exact mechanism and what fixing it would require.
4. ``TestServerHeaderNotLeaked`` — confirms (having read the code first)
that ``SecurityHeadersMiddleware`` unconditionally strips any inbound
``Server`` header before a response leaves the app, and proves it
through the full real stack (not just the isolated middleware class).
"""
import pytest
from fastapi.testclient import TestClient
from slowapi.middleware import SlowAPIMiddleware
from starlette.middleware.sessions import SessionMiddleware
from starlette.responses import PlainTextResponse
from local_deep_research.web.dependencies.csrf import CSRFMiddleware
from local_deep_research.web.fastapi_app import (
BodySizeLimitMiddleware,
DatabaseMiddleware,
RememberMeMiddleware,
SecureCookieMiddleware,
SecurityHeadersMiddleware,
app,
)
# ---------------------------------------------------------------------------
# Test-only routes, registered ONCE on the live singleton `app` at collection
# time (mirrors tests/web/routers/test_route_ordering.py's bare
# `from ...fastapi_app import app`). They run through the exact same,
# already-built middleware stack as every production route — nothing about
# CSRF/Session/SecurityHeaders/etc. is re-created or mocked for these tests.
# Distinctive `/__mw_order_probe__/...` prefix: no production route starts
# with it, and (verified against test_route_ordering.py's own check) it
# cannot be shadowed by / cannot shadow any real route — the only
# parameterized route in the app is `/static/{path:path}`.
# ---------------------------------------------------------------------------
class _ProbeUnregisteredError(RuntimeError):
"""Deliberately NOT one of the types `_register_exception_handlers`
binds: not HTTPException, WebAPIException, NewsAPIException,
PolicyDeniedError, or json.JSONDecodeError. It only matches the bare
``Exception`` handler — which Starlette's `Starlette.build_middleware_stack`
pulls out of `exception_handlers` and hands to `ServerErrorMiddleware`
as its `handler`, NOT to `ExceptionMiddleware`. That distinction is the
whole mechanism `test_KNOWN_GAP_headers_missing_on_unregistered_exception`
documents below.
"""
@app.get("/__mw_order_probe__/boom", include_in_schema=False)
async def _mw_order_probe_boom():
raise _ProbeUnregisteredError("probe: unregistered exception type")
@app.get("/__mw_order_probe__/int-check", include_in_schema=False)
async def _mw_order_probe_int_check(n: int):
return {"n": n}
@app.get("/__mw_order_probe__/leaky-server-header", include_in_schema=False)
async def _mw_order_probe_leaky_server_header():
# A handler that (accidentally or otherwise) sets its own Server
# header, the way an unpinned/legacy dependency might.
return PlainTextResponse(
"ok", headers={"Server": "LDR-internal/9.9.9-do-not-leak"}
)
# ---------------------------------------------------------------------------
# Shared helpers
# ---------------------------------------------------------------------------
# The 8 headers SecurityHeadersMiddleware stamps on EVERY http response
# unconditionally (HSTS is scheme-conditional and cache-control/pragma are
# path-conditional — see the middleware source — so they're deliberately
# excluded from this "must always be present" set; exact values for all of
# them are pinned separately in tests/web/test_security_headers.py).
UNCONDITIONAL_SECURITY_HEADERS = (
"content-security-policy",
"x-frame-options",
"x-content-type-options",
"cross-origin-opener-policy",
"cross-origin-embedder-policy",
"cross-origin-resource-policy",
"permissions-policy",
"referrer-policy",
)
@pytest.fixture
def client(app):
"""Plain TestClient on the `app` fixture (per-test data dir/env).
Parameter name intentionally matches the module-level `app` import
above (used only to register the probe routes at collection time) --
both are the same singleton object; this just goes through
conftest's fixture for per-test isolation like the rest of the suite.
"""
return TestClient(app, raise_server_exceptions=False)
def _assert_all_security_headers_present(resp, label):
missing = [
h for h in UNCONDITIONAL_SECURITY_HEADERS if h not in resp.headers
]
assert not missing, f"{label}: missing security headers {missing!r}"
# ---------------------------------------------------------------------------
# 1. Registration order is pinned
# ---------------------------------------------------------------------------
# Outer -> inner. `app.add_middleware` PREPENDS to `app.user_middleware`
# (Starlette: `self.user_middleware.insert(0, Middleware(cls, ...))`), so the
# LAST-registered middleware is index 0 = OUTERMOST = sees the request
# first / the response last. `fastapi_app.py` registers, top-to-bottom:
# Database, CSRF, Session, RememberMe, BodySizeLimit, SecurityHeaders,
# SecureCookie (all in the main setup block), then SlowAPIMiddleware last of
# all, from `_setup_rate_limiting()` called after that block. So the actual
# outer->inner order is that list reversed, with SlowAPI prepended in front
# of everything else. Each entry below documents what breaks if it moves.
EXPECTED_ORDER_OUTER_TO_INNER = [
(
SlowAPIMiddleware,
"Global rate limiting (slowapi). Registered LAST of all "
"(`_setup_rate_limiting`, called after every other "
"`app.add_middleware`), so it is the outermost layer. If it moved "
"inward of DatabaseMiddleware, a request that will be rejected for "
"being rate-limited would open a user DB connection first — wasted "
"work on the exact path meant to shed load.",
),
(
SecureCookieMiddleware,
"Stamps `; Secure` onto outbound Set-Cookie headers when the "
"connection is HTTPS. Must stay OUTSIDE SessionMiddleware and "
"RememberMeMiddleware so it can see (and rewrite) the Set-Cookie "
"headers THEY just wrote on this response. If it moved inside "
"Session, the session cookie would ship over HTTPS without the "
"Secure flag.",
),
(
SecurityHeadersMiddleware,
"Stamps CSP / X-Frame-Options / etc. Must stay outside CSRF, "
"Session, RememberMe, BodySizeLimit and Database so EVERY response "
"produced by any of them — including a CSRF-rejected 403 and a "
"body-too-large 413 — still carries the headers on its way out. "
"If it moved inside CSRFMiddleware, a CSRF rejection (which "
"returns straight from CSRFMiddleware without reaching the "
"router) would carry no security headers at all — see "
"`TestEffectiveExecutionOrderBehavioral` below for the passing "
"proof that today it does not regress this way.",
),
(
BodySizeLimitMiddleware,
"Global request-body size cap. Must stay outside Session/CSRF "
"(so an oversized body is rejected before CSRFMiddleware buffers "
"a urlencoded form body into memory to read the token out of it) "
"but inside SecurityHeaders (so its own 413 responses still carry "
"them).",
),
(
RememberMeMiddleware,
"Strips Max-Age/Expires off the session cookie when 'remember "
"me' was unchecked. Must stay just OUTSIDE SessionMiddleware so "
"it observes the Set-Cookie header Session just wrote on this "
"very response. If it moved inside Session, it would never see a "
"session Set-Cookie to rewrite and every login would get the "
"full 30-day persistent cookie regardless of the checkbox.",
),
(
SessionMiddleware,
"Decodes the signed cookie into `scope['session']`. Must stay "
"OUTSIDE CSRFMiddleware and DatabaseMiddleware, both of which "
"read `scope['session']`. If it moved inside CSRF, CSRF would "
"always observe an empty session and reject every state-changing "
"request with 'CSRF token missing', valid cookie or not — see "
"`test_csrf_validates_against_a_session_populated_by_session_"
"middleware` below for the passing proof that today it does not.",
),
(
CSRFMiddleware,
"Rejects state-changing requests with a missing/invalid "
"X-CSRFToken before they reach the router. Must stay OUTSIDE "
"DatabaseMiddleware so a forged/CSRF-less request is rejected "
"before a per-user encrypted DB connection is opened for it.",
),
(
DatabaseMiddleware,
"Opens the per-user encrypted DB connection ('Runs INSIDE "
"SessionMiddleware so request.session is available' — its own "
"docstring). Innermost of the app's own middleware by design: "
"every other layer gets to reject the request first without "
"paying for a DB open.",
),
]
class TestRegistrationOrderPinned:
def test_user_middleware_exact_sequence(self, app):
"""`app.user_middleware`, outer -> inner, must match EXACTLY.
This is Starlette's own bookkeeping list, populated purely by the
`add_middleware` calls in `fastapi_app.py`. A reorder, an
insertion on the wrong side, or a removal anywhere in that file's
middleware block changes this list and fails here. See the
per-entry reasons above for what each specific move would
silently break — this test only pins the observable *shape*.
"""
actual = [m.cls for m in app.user_middleware]
expected = [cls for cls, _reason in EXPECTED_ORDER_OUTER_TO_INNER]
assert actual == expected, (
"Middleware registration order changed.\n"
f" expected (outer->inner): {[c.__name__ for c in expected]}\n"
f" actual (outer->inner): {[c.__name__ for c in actual]}"
)
def test_built_asgi_chain_walks_through_them_in_the_same_order(self, app):
"""Cross-check against the REAL, constructed ASGI call chain.
`app.user_middleware` is a bookkeeping list `add_middleware`
maintains; this walks `.app` on the actual object graph
`build_middleware_stack()` wires up, to prove the two agree (and
to catch a future Starlette/FastAPI internal change that decouples
them). Starlette inserts its own `ServerErrorMiddleware` /
`ExceptionMiddleware` / etc. around and between the app's own
middleware, so this asserts our classes appear as a
strictly-increasing subsequence, not full list equality.
"""
layer = app.middleware_stack
if layer is None:
layer = app.build_middleware_stack()
walked = []
seen_ids = set()
while layer is not None and id(layer) not in seen_ids:
seen_ids.add(id(layer))
walked.append(type(layer))
layer = getattr(layer, "app", None)
expected_classes = [
cls for cls, _reason in EXPECTED_ORDER_OUTER_TO_INNER
]
try:
positions = [walked.index(cls) for cls in expected_classes]
except ValueError as exc:
pytest.fail(
f"expected middleware class missing from the built ASGI "
f"chain: {exc}\nchain was: {[c.__name__ for c in walked]}"
)
assert positions == sorted(positions), (
"Built ASGI chain does not visit our middleware in the "
f"expected outer->inner order.\n"
f" expected order: {[c.__name__ for c in expected_classes]}\n"
f" full chain: {[c.__name__ for c in walked]}"
)
# ---------------------------------------------------------------------------
# 2. Effective execution order matches intent (behavioural proof)
# ---------------------------------------------------------------------------
class TestEffectiveExecutionOrderBehavioral:
"""Prove two of the ordering claims above through real requests, not
just structural list inspection. `/auth/logout` is a real, unmodified
production route (POST-only, CSRF-guarded, safe to call anonymously —
it no-ops when there is no `username` in the session) so this exercises
the actual production wiring end to end."""
def test_csrf_rejection_still_carries_security_headers(self, client):
"""SecurityHeadersMiddleware is OUTSIDE CSRFMiddleware.
A CSRF-rejected request is answered straight from
`CSRFMiddleware.__call__` (it calls `response(scope, receive,
send)` and returns without ever reaching the router) — but that
`send` it calls is the *wrapped* send handed down from every
middleware registered outside it, so the response still climbs
back out through BodySizeLimit -> SecurityHeaders -> SecureCookie
on its way to the client. If SecurityHeadersMiddleware were
registered on the wrong side of CSRF (inside it), this response
would carry none of them.
"""
resp = client.post("/auth/logout") # no session, no X-CSRFToken
assert resp.status_code == 403
assert resp.json()["error"].startswith("CSRF token missing")
_assert_all_security_headers_present(resp, "CSRF-rejected 403")
def test_csrf_validates_against_a_session_populated_by_session_middleware(
self, client
):
"""SessionMiddleware is OUTSIDE CSRFMiddleware.
`CSRFMiddleware` reads `scope['session']` — a key only
`SessionMiddleware` ever sets, by decoding the incoming cookie
BEFORE calling further into the stack. If CSRF were registered
outside Session, `scope['session']` would never be populated by
the time CSRF runs, so a request carrying a perfectly valid,
previously-issued session cookie AND the matching CSRF token
would still be rejected with "CSRF token missing" — CSRF would
always see an empty session, request after request.
This drives that exact path: mint a real session + token via the
real `/auth/csrf-token` route (GET, CSRF-exempt), then present
both back on a real CSRF-guarded POST and confirm it is NOT
rejected for a missing session token.
"""
no_cookie_yet = client.post("/auth/logout")
assert no_cookie_yet.status_code == 403
assert "missing" in no_cookie_yet.json()["error"]
token = client.get("/auth/csrf-token").json()["csrf_token"]
assert token
authorized = client.post(
"/auth/logout",
headers={"X-CSRFToken": token},
follow_redirects=False,
)
# Not the CSRF 403 anymore — the real route ran (redirect to
# login is /auth/logout's normal, no-op-when-anonymous response).
assert authorized.status_code != 403, (
"CSRF rejected a request carrying the session's own, "
"just-issued token — SessionMiddleware may not be running "
"before CSRFMiddleware anymore"
)
assert authorized.status_code == 302
assert authorized.headers["location"] == "/auth/login"
# ---------------------------------------------------------------------------
# 3. Security headers on EVERY response class
# ---------------------------------------------------------------------------
class TestSecurityHeadersOnEveryResponseClass:
"""Six response classes, all driven through the real app / real
middleware stack (`raise_server_exceptions=False` so a 500 comes back
as a Response instead of re-raising into the test)."""
def test_200_has_security_headers(self, client):
resp = client.get("/api/v1/health")
assert resp.status_code == 200
_assert_all_security_headers_present(resp, "200 (health)")
def test_404_has_security_headers(self, client):
resp = client.get("/this-route-does-not-exist-mw-order-test")
assert resp.status_code == 404
_assert_all_security_headers_present(resp, "404 (unrouted path)")
def test_405_has_security_headers(self, client):
# /auth/logout is POST-only; GET is CSRF-safe so this reaches
# routing and gets Starlette's routing-raised 405.
resp = client.get("/auth/logout")
assert resp.status_code == 405
_assert_all_security_headers_present(resp, "405 (method not allowed)")
def test_422_has_security_headers(self, client):
resp = client.get("/__mw_order_probe__/int-check?n=not-an-int")
assert resp.status_code == 422
_assert_all_security_headers_present(resp, "422 (validation error)")
def test_csrf_403_has_security_headers(self, client):
resp = client.post("/auth/logout")
assert resp.status_code == 403
_assert_all_security_headers_present(resp, "403 (CSRF rejection)")
def test_headers_present_on_unregistered_exception_500(self, client):
"""Security headers survive the ServerErrorMiddleware bypass.
This response class reaches the client WITHOUT passing through
SecurityHeadersMiddleware, so it is the one path where the headers
have to be stamped by hand. The bypass is structural, not
incidental:
`_register_exception_handlers` registers `@app.exception_handler
(Exception)` as a catch-all so unhandled exceptions get a clean,
scrubbed JSON 500 instead of a bare traceback (see
`tests/web/test_exception_handler_contract.py::Test500Contract`,
which pins that *body* contract on a minimal app with no
middleware). But Starlette's `Starlette.build_middleware_stack`
special-cases exactly that registration:
for key, value in self.exception_handlers.items():
if key in (500, Exception):
error_handler = value # -> ServerErrorMiddleware
else:
exception_handlers[key] = value # -> ExceptionMiddleware
A handler registered for the literal `Exception` class is pulled
OUT of the dict `ExceptionMiddleware` uses and wired instead as
`ServerErrorMiddleware`'s own `handler`. `ServerErrorMiddleware`
is added automatically by Starlette OUTSIDE every
`app.add_middleware(...)` call this app makes — it is the true
outermost layer, ahead of even `SlowAPIMiddleware`.
When an exception type with no specific handler (not
HTTPException, WebAPIException, NewsAPIException,
PolicyDeniedError, or json.JSONDecodeError — this probe route
raises a locally-defined `_ProbeUnregisteredError(RuntimeError)`
that matches none of them) propagates out of the router, it
skips `ExceptionMiddleware` (not in its dict) and unwinds as a
live Python exception through every one of this app's own
middleware — none of which wraps its call to the next layer in a
`try/except`, so none of their response-rewriting `send`
wrappers ever run. It is only caught at `ServerErrorMiddleware`,
which builds the response and sends it on the RAW `send` IT was
given — the one from the ASGI server, never wrapped by
SecurityHeadersMiddleware, SecureCookieMiddleware,
BodySizeLimitMiddleware, RememberMeMiddleware, SessionMiddleware,
CSRFMiddleware, or DatabaseMiddleware. All of it is bypassed.
This survey originally found the 500 shipping with NO security
headers at all. Nothing can be registered outside Starlette's own
outermost wrapper via `app.add_middleware`, but the fix does not
need that: the handler Starlette wires INTO
`ServerErrorMiddleware` is the app's own catch-all, so it stamps
the headers on its own response. It reuses
`SecurityHeadersMiddleware.unconditional_headers()` so the
middleware and the fallback path cannot drift apart.
Only the unconditional set is asserted. HSTS is scheme-dependent
and cache-control is skipped for /static/, so neither belongs on
this path.
"""
resp = client.get("/__mw_order_probe__/boom")
assert resp.status_code == 500
assert resp.json() == {"error": "Server error"}
missing = [
h for h in UNCONDITIONAL_SECURITY_HEADERS if h not in resp.headers
]
assert missing == [], (
"Security headers missing on the unregistered-exception 500 "
f"path: {missing!r}. This response bypasses "
"SecurityHeadersMiddleware entirely (see above), so it relies "
"on the catch-all exception handler in fastapi_app.py stamping "
"them itself via "
"SecurityHeadersMiddleware.unconditional_headers()."
)
assert "server" not in resp.headers
# ---------------------------------------------------------------------------
# 4. Server header does not leak version info
# ---------------------------------------------------------------------------
class TestServerHeaderNotLeaked:
"""Verified first, by reading `SecurityHeadersMiddleware.__call__`
(web/fastapi_app.py): it unconditionally drops any inbound `server`
header --
headers = [(k, v) for k, v in headers if k.lower() != b"server"]
-- and never adds a replacement, on every response, unconditionally.
tests/web/test_security_headers.py::TestServerHeaderStripped already
proves this against the isolated middleware class wrapping a
synthetic ASGI app; these prove it end-to-end through the real,
fully-wired app on live response classes.
"""
def test_no_server_header_on_normal_response(self, client):
resp = client.get("/api/v1/health")
assert "server" not in resp.headers
def test_no_server_header_on_error_response(self, client):
resp = client.get("/this-route-does-not-exist-mw-order-test")
assert "server" not in resp.headers
def test_handler_set_server_header_is_stripped_end_to_end(self, client):
"""A route handler that sets its OWN `Server` header (e.g. an
unpinned dependency doing it for you) must still not leak past
SecurityHeadersMiddleware -- through the real stack, not a
synthetic inner app."""
resp = client.get("/__mw_order_probe__/leaky-server-header")
assert resp.status_code == 200
assert resp.text == "ok"
assert "server" not in resp.headers
for value in resp.headers.values():
assert "9.9.9" not in value
assert "LDR-internal" not in value