~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
"""What the FastAPI port decides at IMPORT time, and what it decides late.
``tests/web/test_lifespan_startup_shutdown.py`` covers the lifespan as a
whole: that every documented step ran, that shutdown released the threads
startup acquired, that an unguarded failure aborts the boot. This file takes
the two questions that survey does not ask.
**Import time versus lifespan time.** ``web/fastapi_app.py`` is not an app
factory. Importing it builds the ``FastAPI`` object, installs the middleware
stack, constructs the rate limiter, registers the template globals, imports
and mounts twenty routers and validates the link table -- all at module
scope, before any lifespan runs and before any caller can influence a thing.
Whatever configuration those statements read is therefore frozen for the life
of the process. For example, ``LDR_TEST_MODE`` determines cookie test mode
once at import; pytest's per-test marker does not affect that policy. The
inventory below pins these reads across five modules, so adding another
restart-only setting is a visible diff.
**Which startup failures still serve traffic.** Twelve of the lifespan's
steps sit inside a ``try/except Exception`` that logs and continues (ten
before #5982 moved the log-queue daemon across the line; eleven before
#6547 added the guarded WebSocket origin-policy log line). That is a
deliberate policy -- a stale stylesheet is not worth refusing to boot over --
but it means the guarded list *is* the list of ways this app can come up
half-initialised and answer requests anyway. It is pinned here as a census,
together with the three steps that are deliberately NOT guarded.
**Ordering.** ``_validate_url_for_bindings`` must run after ``_mount_all``;
its own docstring says validating earlier would "trivially pass" against an
empty route table. That claim is checked here by running the real validator
against a bare app. Likewise the SlowAPI middleware must be registered after
``SecurityHeadersMiddleware`` (that is why ``_rate_limit_exceeded`` stamps
security headers by hand) and the CORS wrapper after both.
**Re-entrancy.** The suite already runs at most one lifespan per process
(see the ``lifespan`` marker in pyproject.toml). This file establishes what
the second cycle would actually do -- which of the four startup singletons
refuse to double-start, and which two do not.
Everything here is static analysis or a cheap in-process call. Nothing in
this file enters a lifespan, so the one-per-process budget that
``tests/web/test_lifespan_boot.py`` spends is untouched.
"""
from __future__ import annotations
import ast
import asyncio
import os
from pathlib import Path
import pytest
from local_deep_research.web import fastapi_app
from local_deep_research.web.services import socketio_asgi
WEB_ROOT = Path(fastapi_app.__file__).resolve().parent
# ---------------------------------------------------------------------------
# Static-analysis helpers.
#
# Both scanners are exercised against synthetic source in the negative
# controls immediately below them, because a scanner that silently matches
# nothing turns every inventory built on it into a test that passes by
# finding zero of everything.
# ---------------------------------------------------------------------------
#: Callables whose result depends on the process environment / on-disk
#: config. A call to one of these from module scope freezes its answer for
#: the lifetime of the process.
_CONFIG_READERS = frozenset(
{
"getenv",
"get_env_setting",
"get_security_default",
"check_env_setting",
"is_rate_limiting_enabled",
"load_server_config",
"get_data_directory",
"get_data_dir",
}
)
def _module_scope_statements(tree: ast.Module) -> list[ast.stmt]:
"""Every statement that executes as a side effect of importing.
Descends into ``if`` / ``try`` / ``for`` / ``with`` and into class
bodies (which also run at import), but stops at any function
boundary: a read inside a ``def`` happens at call time and is
therefore still live.
"""
found: list[ast.stmt] = []
def walk(body: list[ast.stmt]) -> None:
for node in body:
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
continue
found.append(node)
if isinstance(node, ast.ClassDef):
walk(node.body)
continue
for field in ("body", "orelse", "finalbody"):
inner = getattr(node, field, None)
if isinstance(inner, list):
walk(inner)
if isinstance(node, ast.Try):
for handler in node.handlers:
walk(handler.body)
walk(tree.body)
return found
_NESTED_SCOPES = (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)
def _walk_at_import(node: ast.AST):
"""Walk ``node``, never descending into a nested function or class.
``ast.walk`` alone is wrong here: it happily walks from a module-scope
``class`` statement straight into its methods, and a read inside a
method resolves at CALL time -- the opposite of what this file is
inventorying. Class *bodies* do run at import, so they are visited via
``_module_scope_statements``, which lists their statements separately.
"""
if isinstance(node, _NESTED_SCOPES):
return
stack = [node]
while stack:
current = stack.pop()
yield current
for child in ast.iter_child_nodes(current):
if isinstance(child, _NESTED_SCOPES):
continue
stack.append(child)
def _call_name(call: ast.Call) -> str:
"""Dotted source text of a call's callee, e.g. ``db.close_all``."""
return ast.unparse(call.func)
def _config_reads_at_import(source: str) -> set[str]:
"""Names of the settings/env vars a module resolves at import time.
Returns the string argument where there is one (``LDR_TEST_MODE``,
``security.session_timeout_hours``) so the inventory survives the
line numbers moving, and ``name()`` for a no-argument reader.
"""
tree = ast.parse(source)
reads: set[str] = set()
for stmt in _module_scope_statements(tree):
for node in _walk_at_import(stmt):
if not isinstance(node, ast.Call):
continue
callee = _call_name(node)
tail = callee.rsplit(".", 1)[-1]
is_environ_get = callee.endswith("environ.get")
if tail not in _CONFIG_READERS and not is_environ_get:
continue
if node.args and isinstance(node.args[0], ast.Constant):
if isinstance(node.args[0].value, str):
reads.add(node.args[0].value)
continue
reads.add(f"{tail}()")
return reads
def _swallowing_guards(func: ast.AST) -> list[dict]:
"""Top-level ``try`` blocks in ``func`` whose handlers never re-raise.
Each entry carries the dotted names called in the guarded body and the
``logger.*`` calls the handlers make, which is what distinguishes a
guard that records a traceback from one that drops it.
"""
guards: list[dict] = []
for stmt in func.body:
if not isinstance(stmt, ast.Try):
continue
handlers = stmt.handlers
if not handlers:
continue
if any(
isinstance(node, ast.Raise)
for handler in handlers
for node in ast.walk(handler)
):
continue
calls = {
_call_name(node)
for node in ast.walk(stmt)
if isinstance(node, ast.Call)
}
logged = {
_call_name(node)
for handler in handlers
for node in ast.walk(handler)
if isinstance(node, ast.Call)
and _call_name(node).startswith("logger.")
}
guards.append({"lineno": stmt.lineno, "calls": calls, "logged": logged})
return guards
def _lifespan_function() -> ast.AsyncFunctionDef:
tree = ast.parse(Path(fastapi_app.__file__).read_text(encoding="utf-8"))
for node in tree.body:
if isinstance(node, ast.AsyncFunctionDef) and node.name == "lifespan":
return node
raise AssertionError(
"web/fastapi_app.py no longer defines an async `lifespan` function; "
"this whole file is scoped to it"
)
def _split_at_yield(
func: ast.AsyncFunctionDef,
) -> tuple[list[ast.stmt], list[ast.stmt]]:
"""Split the lifespan body into (startup, shutdown) around ``yield``."""
for index, stmt in enumerate(func.body):
if isinstance(stmt, ast.Expr) and isinstance(stmt.value, ast.Yield):
return func.body[:index], func.body[index + 1 :]
raise AssertionError(
"the lifespan body has no top-level `yield`; startup and shutdown "
"can no longer be told apart"
)
# ---------------------------------------------------------------------------
# 0. Negative controls for both scanners.
# ---------------------------------------------------------------------------
_SCANNER_PROBE = """
import os
from x import get_env_setting
MODULE_LEVEL = os.environ.get("PROBE_MODULE")
if True:
NESTED_IN_IF = os.getenv("PROBE_IF")
try:
NESTED_IN_TRY = get_env_setting("probe.try")
except Exception:
FROM_HANDLER = os.getenv("PROBE_HANDLER")
class C:
IN_CLASS_BODY = os.getenv("PROBE_CLASS")
def method(self):
return os.getenv("PROBE_METHOD")
def helper():
return os.environ.get("PROBE_FUNCTION")
"""
def test_the_import_time_scanner_sees_import_time_reads_and_only_those():
"""Positive and negative control for ``_config_reads_at_import``.
Without this, an inventory that quietly matched nothing would report a
clean bill of health for a module full of frozen configuration.
"""
found = _config_reads_at_import(_SCANNER_PROBE)
expected_found = {
"PROBE_MODULE",
"PROBE_IF",
"probe.try",
"PROBE_HANDLER",
"PROBE_CLASS",
}
assert expected_found <= found, (
"the import-time scanner missed reads that really do run at import: "
f"{sorted(expected_found - found)}. Every inventory below is built "
"on it, so a blind scanner makes them all pass vacuously."
)
assert "PROBE_METHOD" not in found and "PROBE_FUNCTION" not in found, (
"the scanner reported reads that live inside a function body; those "
"resolve at CALL time and are exactly the ones that are NOT frozen. "
f"Found: {sorted(found)}"
)
_GUARD_PROBE = """
async def lifespan(app):
try:
swallowed()
except Exception:
logger.warning("dropped")
try:
reraised()
except Exception:
logger.exception("kept")
raise
if cond:
try:
not_top_level()
except Exception:
pass
yield
try:
on_shutdown()
except Exception:
logger.debug("quiet")
"""
def test_the_guard_scanner_separates_swallowing_from_reraising():
"""Positive and negative control for ``_swallowing_guards``."""
func = ast.parse(_GUARD_PROBE).body[0]
startup, shutdown = _split_at_yield(func)
startup_calls = {
name for g in _swallowing_guards(func) for name in g["calls"]
}
assert "swallowed" in startup_calls, (
"the guard scanner missed a plain swallowing try/except; the "
"startup census below would then report an empty guarded list"
)
assert "reraised" not in startup_calls, (
"the guard scanner treated a re-raising handler as swallowing -- "
"a step that aborts the boot would be miscounted as one that "
"leaves the app serving degraded"
)
assert "not_top_level" not in startup_calls, (
"the scanner descended into a nested try; the census is defined "
"over the lifespan's own top-level steps"
)
assert len(startup) == 3 and len(shutdown) == 1, (
"the yield split is wrong: got "
f"{len(startup)} startup / {len(shutdown)} shutdown statements "
"from a probe with 3 and 1"
)
# ---------------------------------------------------------------------------
# 1. What the web package resolves at import, and can never revisit.
# ---------------------------------------------------------------------------
#: Every module-scope configuration read in the modules that
#: ``import local_deep_research.web.fastapi_app`` pulls in. Each entry is a
#: value the process resolves once, before the lifespan and before any
#: caller exists, and then reuses until restart. All of them read the
#: environment, so an operator who sets the variable and restarts is fine;
#: what none of them can do is respond to a change made afterwards --
#: notably to a setting edited in the UI, or to an env var a test or an
#: embedding process sets after the first import.
_IMPORT_TIME_CONFIG = {
"web/fastapi_app.py": {
"LDR_TEST_MODE",
# Gates /api/docs AND /openapi.json.
"LDR_EXPOSE_DOCS",
# SessionMiddleware's itsdangerous max_age.
"security.session_remember_me_days",
# _NON_REMEMBER_ME_SESSION_SECONDS.
"security.session_timeout_hours",
},
"web/dependencies/rate_limit.py": {
# Decides whether X-Forwarded-For is honoured when picking the
# rate-limit bucket key.
"TRUST_PROXY_HEADERS",
# The master on/off switch for brute-force protection.
"is_rate_limiting_enabled()",
"RATE_LIMIT_STORAGE_URI",
# Read at module scope, then resolved (with slowapi's ./.env
# fallback) by ``_resolve_legacy_storage_uri()`` -- also at import.
"RATELIMIT_STORAGE_URL",
# Which direct peers uvicorn's ProxyHeadersMiddleware trusts
# forwarded headers from when TRUST_PROXY_HEADERS=true; decides
# whether the startup guard raises or only warns, and gates
# whether _get_client_ip trusts request.client outright.
"LDR_SERVER_FORWARDED_ALLOW_IPS",
# How many trusted proxy hops append to X-Forwarded-For. Frozen
# for the same reason as the two above: it is the other half of
# the import-time startup guard's input, and _get_client_ip's
# hop-count branch reads the module global rather than the
# environment. The module reads it through ``os.environ.get`` as
# well as through the typed ``TRUSTED_PROXY_COUNT`` setting
# precisely so this scanner (which matches ``environ.get`` /
# ``getenv`` call sites, not a settings object's
# ``.get_value()``) can see it.
"LDR_SERVER_TRUSTED_PROXY_COUNT",
# Supplies DEFAULT_RATE_LIMIT and the per-route login/registration
# limits, which are then baked into the Limiter and into every
# @limiter.limit decorator at router-import time.
"load_server_config()",
},
"web/services/socketio_asgi.py": {
# The WebSocket origin allowlist handed to python-socketio.
"security.websocket.allowed_origins",
},
"web/services/research_service.py": {
# Sizes _global_research_semaphore, a threading.Semaphore built at
# import; the count cannot be changed afterwards at all.
"server.max_concurrent_research",
},
"web/models/database.py": {
"get_data_directory()",
},
}
def test_the_import_time_configuration_inventory_is_pinned():
"""Pin every setting the web package freezes at import.
A floor is asserted first: if the scanner or the import graph moved and
this found (say) three reads, an equality check alone could be satisfied
by shrinking the expectation, so the count and the named security-
relevant entries are checked independently.
"""
actual = {
rel: _config_reads_at_import(
(WEB_ROOT / rel.removeprefix("web/")).read_text(encoding="utf-8")
)
for rel in _IMPORT_TIME_CONFIG
}
total = sum(len(v) for v in actual.values())
assert total >= 12, (
"the import-time sweep found only "
f"{total} module-scope configuration reads across "
f"{len(_IMPORT_TIME_CONFIG)} modules. It should find at least 12; a "
"near-empty result means the scanner stopped matching, not that the "
f"port stopped freezing configuration. Found: {actual}"
)
must_include = {
"web/fastapi_app.py": {"LDR_TEST_MODE", "LDR_EXPOSE_DOCS"},
"web/dependencies/rate_limit.py": {
"TRUST_PROXY_HEADERS",
"is_rate_limiting_enabled()",
},
"web/services/socketio_asgi.py": {"security.websocket.allowed_origins"},
}
for rel, required in must_include.items():
assert required <= actual[rel], (
f"{rel} no longer resolves {sorted(required - actual[rel])} at "
"import time. That is very likely an improvement -- but it "
"changes when the value is decided, so update "
"_IMPORT_TIME_CONFIG deliberately rather than by accident."
)
for rel, expected in _IMPORT_TIME_CONFIG.items():
assert actual[rel] == expected, (
f"{rel}: the set of configuration values frozen at import "
f"changed.\n added: {sorted(actual[rel] - expected)}\n"
f" removed: {sorted(expected - actual[rel])}\n"
"Anything ADDED here is a value that can no longer be changed "
"without restarting the process, and that no lifespan step or "
"request handler can revisit. Confirm that is intended, then "
"update this inventory."
)
def test_the_freeze_is_the_call_sites_doing_not_the_settings_layers():
"""``get_env_setting`` reads ``os.environ`` on every call.
Load-bearing for the inventory's claim. If the registry itself
snapshotted the environment, the module-scope call sites above would be
incidental and the inventory would be blaming the wrong layer. It does
not: the freeze is created by assigning the result to a module-level
name, and is therefore fixable per call site.
"""
from local_deep_research.settings import env_registry
key = "security.websocket.allowed_origins"
env_var = env_registry.registry.get_env_var(key)
assert env_var, (
f"{key!r} is not a registered env-only setting any more; the "
"premise of this test (and of the socket.io entry in the "
"inventory above) is gone"
)
original = os.environ.get(env_var)
try:
os.environ[env_var] = "https://probe.example"
assert env_registry.get_env_setting(key) == "https://probe.example", (
"get_env_setting did not observe an environment change made "
"after import, so the settings registry snapshots os.environ "
"and the module-scope reads are not the only freeze"
)
finally:
if original is None:
os.environ.pop(env_var, None)
else:
os.environ[env_var] = original
# ...and yet the value socket.io actually enforces was decided once.
assert socketio_asgi._socketio_cors is None or isinstance(
socketio_asgi._socketio_cors, (str, list)
), "premise: _socketio_cors is the resolved module-level allowlist"
module_scope_names = {
target.id
for stmt in _module_scope_statements(
ast.parse(Path(socketio_asgi.__file__).read_text(encoding="utf-8"))
)
if isinstance(stmt, ast.Assign)
for target in stmt.targets
if isinstance(target, ast.Name)
}
assert "_socketio_cors" in module_scope_names, (
"SECURITY: the socket.io origin allowlist is expected to be "
"computed at module scope (and therefore frozen). If it moved into "
"a function this test should be replaced by one that checks it "
"tracks the environment."
)
def _directory_creating_calls(calls: set[str]) -> set[str]:
"""The subset of ``calls`` that materialises a directory on disk.
Matched as a category -- the audited chokepoint by name, raw stdlib
creation by method tail (``mkdir`` / ``makedirs``, whatever the
receiver expression happens to be) -- so a rename inside the category
is not a failure while a step OUT of it is.
"""
return {
name
for name in calls
if name == "create_directory"
or name.rsplit(".", 1)[-1] in {"mkdir", "makedirs"}
}
def test_importing_the_web_app_has_the_filesystem_side_effects_it_has():
"""Import is not side-effect free, and two of those effects write.
``SECRET_KEY = _load_secret_key()`` creates ``<data dir>/.secret_key``
(mode 0600) and ``web/models/database.py`` creates the data directory
itself. So merely importing the app -- a doc build, a linter that
imports, an editor's autocomplete, ``pytest --collect-only`` -- both
resolves ``LDR_DATA_DIR`` and materialises a signing key under it.
Pinned so that a third write, or a move of these into the lifespan,
is a deliberate change.
The directory half is pinned by CATEGORY, not by one call name: #5161
routed every production directory creation through
``security/directory_creation.create_directory`` (an audited chokepoint
that rejects null bytes and ``..``, resolves symlinks and audit-logs),
so the old ``os.makedirs`` pin here failed on a rename while the
side effect it cared about was unchanged. Asserting on the set below
keeps both halves of the contract: the import still creates the data
directory, AND it still does so through the chokepoint rather than a
raw ``mkdir`` that would slip past the audit.
"""
app_src = Path(fastapi_app.__file__).read_text(encoding="utf-8")
module_calls = {
_call_name(node)
for stmt in _module_scope_statements(ast.parse(app_src))
for node in _walk_at_import(stmt)
if isinstance(node, ast.Call)
}
assert "_load_secret_key" in module_calls, (
"SECRET_KEY is no longer loaded at module scope. That changes when "
"a missing/unreadable key aborts the process -- "
"tests/web/test_lifespan_startup_shutdown.py pins that it aborts "
"the IMPORT, i.e. the boot."
)
db_src = (WEB_ROOT / "models" / "database.py").read_text(encoding="utf-8")
db_calls = {
_call_name(node)
for stmt in _module_scope_statements(ast.parse(db_src))
for node in _walk_at_import(stmt)
if isinstance(node, ast.Call)
}
creators = _directory_creating_calls(db_calls)
assert creators == {"create_directory"}, (
"web/models/database.py's import-time data-directory creation "
f"changed: found {sorted(creators)}.\n"
" * empty -- the module no longer creates the data directory at "
"import. That is an improvement (it is the reason importing the "
"web package is not safe against an unwritable or unset "
"LDR_DATA_DIR), so update this pin deliberately.\n"
" * a raw mkdir/makedirs -- SECURITY: the creation bypasses "
"security/directory_creation.create_directory, the audited "
"chokepoint every production directory creation is supposed to go "
"through (#5161). Route it back through create_directory."
)
# ---------------------------------------------------------------------------
# 2. Startup failure: which steps leave the app serving anyway.
# ---------------------------------------------------------------------------
#: One distinctive call per guarded startup step. These are the ways the
#: app can finish `lifespan` startup, report `lifespan.startup.complete`,
#: and begin answering requests with that subsystem dead.
_GUARDED_STARTUP_STEPS = {
# ADDED BY #6547, deliberately. Logging-only: the WebSocket origin
# policy is resolved and applied at `socketio_asgi` import time, so this
# step only reports the precomputed value (moved here from import time,
# where package logging was still disabled and the allow-all
# INFO/WARNING was silently dropped). Degraded mode is "the startup log
# lacks the origin-policy line", while the enforced policy is unchanged.
# Not worth refusing to boot over, and the handler uses
# logger.exception, so the failure still carries a traceback.
"log_socketio_cors_policy": "WebSocket origin-policy startup log line",
"warn_if_threadpool_exceeds_db_pool": "AnyIO worker pool sizing",
"theme_registry.get_combined_css": "themes.css regeneration",
"warn_if_weak_kdf_with_existing_databases": "weak-SQLCipher-KDF check",
# ADDED BY #7169, deliberately. Sets research_outputs/ (optional plaintext
# report backups) to 0700. Degraded mode is "an older install's directory
# keeps its old mode", and FileReportStorage independently refuses a
# directory that is not private, so no new backup lands there. Not worth
# refusing to boot over; the guard uses logger.exception.
"restrict_research_outputs_directory": "research_outputs/ privacy",
"get_background_job_scheduler": "news scheduler",
"start_connection_cleanup_scheduler": "connection-cleanup scheduler",
# MOVED HERE FROM _UNGUARDED_STARTUP_STEPS BY #5982, deliberately.
# `start_log_queue_processor` ends in `threading.Thread(...).start()`,
# which raises `RuntimeError: can't start new thread` when a container
# pids / RLIMIT_NPROC ceiling is reached. Unguarded, that aborted the
# whole boot: the server never answered a single request because a
# best-effort logging daemon could not be spawned. main guards the same
# call in `web/app.py::main()` (`try: start_log_queue_processor(app)` /
# `except Exception: logger.exception(...)`, PR #3488) and the guard was
# lost in the FastAPI port; #5982 restores it. The degradation is
# bounded: `stop_log_queue_processor()` on the shutdown path is itself
# guarded and is a no-op when no daemon ever started, and queued entries
# still reach the DB via the shutdown flush.
"start_log_queue_processor": "log-queue drain daemon",
}
#: Steps deliberately left unguarded: if they raise, the lifespan raises,
#: uvicorn gets `lifespan.startup.failed` and refuses to serve.
_UNGUARDED_STARTUP_STEPS = (
"enforce_process_dump_policy",
"set_main_loop",
"init_lock",
"queue_processor.start",
)
def test_the_degraded_startup_census_is_pinned():
"""Enumerate every startup step that can fail and still serve.
The guarded/unguarded split is a policy decision, not an accident, and
it is invisible in review: wrapping one more step in ``try/except`` is
a three-line diff that converts "the container restarts" into "the
container comes up with that subsystem silently off".
"""
startup, _ = _split_at_yield(_lifespan_function())
guards = [
guard
for guard in _swallowing_guards(_lifespan_function())
if guard["lineno"] <= max(stmt.lineno for stmt in startup)
]
assert len(guards) >= 6, (
f"expected at least 6 swallowing guards in lifespan startup, found "
f"{len(guards)}. A near-zero count means the scanner broke, not "
"that the boot became fail-fast."
)
guarded_calls = {name for guard in guards for name in guard["calls"]}
missing = [
call for call in _GUARDED_STARTUP_STEPS if call not in guarded_calls
]
assert not missing, (
"these startup steps are no longer inside a swallowing guard: "
f"{[_GUARDED_STARTUP_STEPS[c] for c in missing]}. If that is "
"intentional they now abort the boot on failure, which is a "
"deployment-visible behaviour change; update this census."
)
assert len(guards) == len(_GUARDED_STARTUP_STEPS), (
f"lifespan startup now has {len(guards)} swallowing guards, not "
f"{len(_GUARDED_STARTUP_STEPS)}. Each one is a distinct way for the "
"app to serve traffic half-initialised, so a new one has to be "
"added to this census on purpose. Guard bodies: "
f"{[sorted(g['calls'])[:4] for g in guards]}"
)
def test_the_load_bearing_startup_steps_still_abort_the_boot():
"""The required steps whose failure must stop the server from serving.
``enforce_process_dump_policy`` makes the process non-dumpable and
zeroes its core-file limit; serving after it failed would leave
session passwords and decrypted data exposed to core dumps and
same-user debuggers, so a failure refuses the boot instead.
``set_main_loop`` and ``init_lock`` decide whether background threads
can reach the browser at all; ``queue_processor.start`` is the only
drain path for queued research. Wrapping any of them would turn a hard
failure into an app that boots, serves, and silently does nothing in
the background.
Was four until #5982: ``start_log_queue_processor`` was listed here as
load-bearing, and it is not. It drains DB *log* entries -- losing it
costs steady-state log persistence (the shutdown flush still runs), not
a user's research -- while `threading.Thread(...).start()` inside it
raises `RuntimeError: can't start new thread` under a container pids
ceiling, which unguarded took the whole server down. See the comment on
its entry in ``_GUARDED_STARTUP_STEPS`` above.
"""
func = _lifespan_function()
startup, _ = _split_at_yield(func)
guarded_calls = {
name
for guard in _swallowing_guards(func)
if guard["lineno"] <= max(stmt.lineno for stmt in startup)
for name in guard["calls"]
}
startup_calls = {
_call_name(node)
for stmt in startup
for node in ast.walk(stmt)
if isinstance(node, ast.Call)
}
absent = [c for c in _UNGUARDED_STARTUP_STEPS if c not in startup_calls]
assert not absent, (
f"lifespan startup no longer calls {absent} at all -- the "
"corresponding subsystem is simply never started. "
f"Observed calls: {sorted(startup_calls)}"
)
wrapped = [c for c in _UNGUARDED_STARTUP_STEPS if c in guarded_calls]
assert not wrapped, (
f"{wrapped} moved inside a swallowing try/except. A failure there "
"will now be logged and the app will report a successful startup "
"and begin serving requests, with socket.io emits or research "
"dispatch dead. Moving a step across this line is a real decision "
"(see #5982, which did it for the log-queue daemon on purpose): "
"either revert the wrap, or move the entry into "
"_GUARDED_STARTUP_STEPS with a note saying why the degraded mode "
"is preferable to refusing to boot."
)
@pytest.mark.xfail(
strict=True,
reason=(
"SECURITY: the connection-cleanup scheduler's startup guard logs "
"with logger.warning, so a failure to start it carries no "
"traceback. That scheduler is the only thing that purges expired "
"sessions and calls session_password_store.clear_all_for_user for "
"users who closed the tab instead of logging out -- i.e. it is the "
"teardown path for the MAJORITY of users. If it fails to start, "
"the app serves normally while cached plaintext SQLCipher keys "
"stay in a process-global dict indefinitely, and the single log "
"line naming the problem has no stack to diagnose it with."
),
)
def test_a_failed_cleanup_scheduler_is_reported_with_a_traceback():
"""Every startup guard that hides a security control must log a trace."""
func = _lifespan_function()
startup, _ = _split_at_yield(func)
guards = [
guard
for guard in _swallowing_guards(func)
if guard["lineno"] <= max(stmt.lineno for stmt in startup)
]
matching = [
guard
for guard in guards
if "start_connection_cleanup_scheduler" in guard["calls"]
]
assert matching, (
"premise broken: no swallowing guard around "
"start_connection_cleanup_scheduler; the census test above should "
"have caught this first"
)
assert matching[0]["logged"] == {"logger.exception"}, (
"the connection-cleanup scheduler's failure is logged with "
f"{sorted(matching[0]['logged'])} instead of logger.exception, so "
"the operator gets one sentence and no stack. Contrast the news "
"scheduler and the weak-KDF check, whose guards both use "
"logger.exception."
)
# ---------------------------------------------------------------------------
# 3. Ordering at module scope.
# ---------------------------------------------------------------------------
def _module_scope_call_order() -> list[tuple[int, str]]:
"""(index, dotted callee) for every top-level call in fastapi_app."""
tree = ast.parse(Path(fastapi_app.__file__).read_text(encoding="utf-8"))
order: list[tuple[int, str]] = []
for index, stmt in enumerate(tree.body):
if isinstance(stmt, ast.Expr) and isinstance(stmt.value, ast.Call):
order.append((index, _call_name(stmt.value)))
return order
def test_the_module_scope_wiring_runs_in_its_documented_order():
"""Four ordering dependencies that are silent when broken.
None of these produce an error when reversed. Registering CORS before
the rate limiter puts SlowAPI outside it, so preflights start getting
rate limited; adding any middleware after the routers changes the
stack's nesting without a word; running the link validator before the
routers are mounted makes it pass against an empty route table.
"""
order = _module_scope_call_order()
positions = {name: index for index, name in order}
required = [
"_setup_rate_limiting",
"_configure_cors",
"_setup_template_globals",
"_mount_all",
"_validate_url_for_bindings",
]
missing = [name for name in required if name not in positions]
assert not missing, (
f"web/fastapi_app.py no longer calls {missing} at module scope. "
f"Top-level calls found: {[name for _, name in order]}"
)
assert positions["_setup_rate_limiting"] < positions["_configure_cors"], (
"CORS must be registered AFTER the rate limiter so it ends up the "
"outermost middleware and answers a preflight OPTIONS before auth, "
"CSRF or rate limiting run (see _configure_cors's call site "
"comment). Reversed, cross-origin preflights consume the caller's "
"rate-limit budget."
)
assert positions["_mount_all"] < positions["_validate_url_for_bindings"], (
"_validate_url_for_bindings reads app.routes; run before "
"_mount_all it validates an empty route table and passes "
"trivially, which is exactly what its docstring warns about."
)
assert (
positions["_setup_template_globals"]
< positions["_validate_url_for_bindings"]
), (
"the link validator resolves names through "
"templates.env.globals['url_for'], which _setup_template_globals "
"installs; before it, the global does not exist"
)
body = ast.parse(
Path(fastapi_app.__file__).read_text(encoding="utf-8")
).body
last_middleware = max(
(
index
for index, stmt in enumerate(body)
# Only statements that RUN at import; an add_middleware call
# inside a `def` (there are two) says nothing about ordering.
if not isinstance(
stmt,
(ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef),
)
for node in ast.walk(stmt)
if isinstance(node, ast.Call)
and _call_name(node).endswith("add_middleware")
),
default=-1,
)
assert last_middleware != -1, (
"premise broken: no add_middleware call found at module scope"
)
assert last_middleware < positions["_mount_all"], (
"middleware is being added after the routers are mounted. "
"Starlette builds the stack lazily, so this does not raise -- it "
"just silently changes which middleware wraps which, and "
"tests/web/test_middleware_order_and_headers.py pins that order."
)
def test_the_link_validator_is_vacuous_against_an_unmounted_app(monkeypatch):
"""Prove the ordering above is load-bearing, not decorative.
Same function, same strict flag, two route tables: the real app passes,
a bare FastAPI fails. That is the difference `_mount_all` makes, and it
is the reason `_validate_url_for_bindings(app)` is the last statement
in the module rather than a step inside `_setup_template_globals`.
"""
from fastapi import FastAPI
monkeypatch.setenv("LDR_STRICT_TEMPLATE_LINKS", "true")
# Premise: the scan finds template usages at all. If TEMPLATE_DIR is
# unreadable (a zip/pex install) the validator returns early and both
# halves below would "pass" for the wrong reason.
template_dir = Path(fastapi_app.TEMPLATE_DIR)
names: set[str] = set()
for tpl in template_dir.rglob("*.html"):
names.update(
fastapi_app._URL_FOR_NAME_RE.findall(
tpl.read_text(encoding="utf-8", errors="replace")
)
)
assert len(names) > 10, (
f"premise broken: only {len(names)} url_for() names found under "
f"{template_dir}, so the validator short-circuits and this test "
"cannot distinguish a mounted app from an empty one"
)
fastapi_app._validate_url_for_bindings(fastapi_app.app)
with pytest.raises(RuntimeError, match="url_for") as excinfo:
fastapi_app._validate_url_for_bindings(FastAPI())
assert "auth.login" in str(excinfo.value), (
"the validator ran against an empty route table and did not report "
"the login link as dead, so it is not actually checking resolved "
f"paths against app.routes. Message: {excinfo.value}"
)
# ---------------------------------------------------------------------------
# 4. Re-entrancy: what a second startup would do.
# ---------------------------------------------------------------------------
class _NoopManager:
"""Stand-in for session_manager / db_manager.
The cleanup job is scheduled at a 99999s interval and the scheduler is
shut down immediately, so nothing on this object is ever called; it
exists only so ``add_job`` has arguments to bind.
"""
def test_the_startup_singletons_that_do_refuse_to_double_start():
"""Positive control: three of the five startup objects are idempotent.
Establishes that "starting twice is a no-op" is both the intent and
achievable here, which is what makes the failure below a defect rather
than a design choice.
"""
from local_deep_research.scheduler.background import (
BackgroundJobScheduler,
get_background_job_scheduler,
)
from local_deep_research.web.queue.processor_v2 import QueueProcessorV2
assert get_background_job_scheduler() is get_background_job_scheduler(), (
"the news scheduler is no longer a process singleton; a second "
"lifespan cycle would build a second APScheduler and duplicate "
"cleanup_inactive_users / reload_config"
)
for cls, flag in (
(BackgroundJobScheduler, "is_running"),
(QueueProcessorV2, "running"),
):
source = ast.parse(
Path( # the class's own module
__import__(cls.__module__, fromlist=["_"]).__file__
).read_text(encoding="utf-8")
)
starts = [
node
for node in ast.walk(source)
if isinstance(node, ast.FunctionDef) and node.name == "start"
]
assert starts, f"{cls.__name__} has no start() to inspect"
guarded = any(
any(
isinstance(node, ast.Attribute) and node.attr == flag
for stmt in start.body
if isinstance(stmt, ast.If)
for node in ast.walk(stmt.test)
)
and any(
isinstance(node, ast.Return)
for start_stmt in start.body
if isinstance(start_stmt, ast.If)
for node in ast.walk(start_stmt)
)
for start in starts
)
assert guarded, (
f"{cls.__name__}.start() lost its `if self.{flag}: return` "
"re-entrancy guard, so a second lifespan cycle would start a "
"second thread / add its jobs twice"
)
@pytest.mark.xfail(
strict=True,
reason=(
"start_connection_cleanup_scheduler() constructs a fresh "
"BackgroundScheduler on every call instead of returning a "
"singleton, so a second lifespan startup in one process runs "
"cleanup_idle_connections on TWO schedulers concurrently and the "
"first scheduler is unreachable -- the lifespan's shutdown only "
"holds the second, so its thread leaks with references to the "
"stale session_manager/db_manager. Not reachable in production "
"today only because socketio_asgi.init_lock() already caps the "
"process at one lifespan (see the `lifespan` marker in "
"pyproject.toml); fixing that cap without fixing this would make "
"it reachable on any in-process restart."
),
)
def test_a_second_startup_reuses_the_connection_cleanup_scheduler():
"""Startup must be re-runnable without duplicating a scheduler job."""
from local_deep_research.web.auth.connection_cleanup import (
start_connection_cleanup_scheduler,
)
first = start_connection_cleanup_scheduler(
_NoopManager(), _NoopManager(), interval_seconds=99999
)
second = None
try:
second = start_connection_cleanup_scheduler(
_NoopManager(), _NoopManager(), interval_seconds=99999
)
assert first.running and second.running, (
"premise broken: start_connection_cleanup_scheduler returned a "
f"scheduler that is not running ({first.running}, "
f"{second.running})"
)
job_ids = [job.id for job in first.get_jobs()]
assert job_ids == ["cleanup_idle_connections"], (
f"premise broken: expected one cleanup job, got {job_ids}"
)
assert second is first, (
"a second startup created a SECOND connection-cleanup "
f"scheduler ({id(first):x} vs {id(second):x}), both running "
f"{[j.id for j in second.get_jobs()]}. The idle-connection "
"sweep -- which closes per-user SQLCipher databases and clears "
"cached master keys -- would then run twice over the same "
"state, and only one of the two schedulers can be shut down."
)
finally:
for scheduler in (first, second):
if scheduler is not None and scheduler.running:
scheduler.shutdown(wait=False)
@pytest.mark.xfail(
strict=True,
reason=(
"socketio_asgi.init_lock() assigns only when _lock is None, so a "
"second lifespan hands out the lock created for the FIRST event "
"loop. asyncio.Lock binds to a loop on its first CONTENDED "
"acquire, so this is latent: an uncontended second boot looks "
"fine, and only a concurrent socket.io connect/subscribe on the "
"new loop raises 'is bound to a different event loop' -- leaving "
"the lock permanently held. This is the constraint the `lifespan` "
"pytest marker exists to work around."
),
)
def test_a_second_startup_gets_a_lock_usable_on_the_new_loop():
"""The socket.io subscription lock must follow the running loop."""
saved_lock = socketio_asgi._lock
saved_loop = socketio_asgi._main_loop
try:
socketio_asgi._lock = None
socketio_asgi._main_loop = None
async def cycle():
socketio_asgi.set_main_loop(asyncio.get_running_loop())
socketio_asgi.init_lock()
lock = socketio_asgi._lock
async def hold():
async with lock:
await asyncio.sleep(0.005)
# Two holders, so the second one actually waits -- an
# uncontended acquire never touches the loop and would hide
# the binding entirely.
await asyncio.gather(hold(), hold())
return lock
first_lock = asyncio.run(cycle())
assert first_lock is not None, (
"premise broken: init_lock() did not create a lock"
)
second_lock = asyncio.run(cycle())
assert second_lock is not first_lock or not second_lock.locked(), (
"the second startup reused the first loop's lock and left it "
"LOCKED after the cross-loop error, so every later "
"subscribe/connect on this process deadlocks"
)
finally:
socketio_asgi._lock = saved_lock
socketio_asgi._main_loop = saved_loop
# ---------------------------------------------------------------------------
# 5. Shutdown: what startup acquired that shutdown does not release.
#
# Thread and DB teardown ordering is covered by
# tests/web/test_lifespan_startup_shutdown.py. What follows is the part it
# does not reach: the two module-level globals.
# ---------------------------------------------------------------------------
@pytest.mark.xfail(
strict=True,
reason=(
"lifespan startup sets socketio_asgi._main_loop and ._lock; "
"shutdown clears neither. The process is therefore left holding a "
"reference to a CLOSED event loop and a lock built for it. "
"_get_main_loop() defends itself with an is_closed() check, but "
"init_lock() does not, which is precisely why the process is "
"capped at one lifespan cycle. Releasing both here is the smaller "
"half of fixing that cap."
),
)
def test_shutdown_releases_the_socket_io_globals_startup_acquired():
"""Shutdown must undo the two module-global assignments startup made."""
func = _lifespan_function()
startup, shutdown = _split_at_yield(func)
startup_calls = {
_call_name(node)
for stmt in startup
for node in ast.walk(stmt)
if isinstance(node, ast.Call)
}
assert {"set_main_loop", "init_lock"} <= startup_calls, (
"premise broken: lifespan startup no longer registers the loop "
f"and the lock. Calls: {sorted(startup_calls)}"
)
shutdown_calls = {
_call_name(node)
for stmt in shutdown
for node in ast.walk(stmt)
if isinstance(node, ast.Call)
}
releasers = {
name
for name in shutdown_calls
if "main_loop" in name or "lock" in name.lower()
}
assert releasers, (
"lifespan shutdown stops every thread it started but never "
"releases socketio_asgi._main_loop or ._lock. After shutdown the "
"module still points at a closed loop, and init_lock() will hand "
"the next cycle a lock bound to it. Shutdown calls: "
f"{sorted(shutdown_calls)}"
)