Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

41.8 KB
Raw Blame
"""Hostile-input and data-leak coverage for the metrics / history routers
(FastAPI port).

``web/routes/metrics_routes.py`` became ``web/routers/metrics.py`` almost
verbatim in the Flask->FastAPI migration, but ~200 tests across two metrics
files were deleted with the Flask blueprint. At the historical ADR-0010
snapshot, the behaviors below remained in the branch code after their direct
assertions were removed. This file is their committed regression evidence.

Scope note: bare authentication ("does this route reject anonymous
callers") is covered structurally by the route-table sweep in
``tests/security/test_auth_dependencies_fastapi.py`` and is deliberately
NOT re-tested here. This file covers what that sweep cannot see — what a
route *leaks* and how it handles *hostile* input:

1. ORDER BY column injection on ``GET /metrics/api/journals?sort=``.
   The allow-list is ``journal_quality/db.py::_SORT_COLUMNS``; the route
   forwards the raw query param into ``get_journals_page(sort=...)``,
   which does ``getattr(Source, sort)``. Without the allow-list a payload
   that is not a Source attribute raises ``AttributeError`` into the
   route's outer ``except`` (500), and a payload that *is* an unexported
   attribute (``id``, ``issn``, ``name_lower``) silently re-orders the
   dashboard. The observed behaviour is FALL BACK TO ``quality``, not
   reject — pinned as such below.

2. The ``score_source`` allow-list (``metrics.py::_ALLOWED_SCORE_SOURCES``)
   -> 400 on anything outside it. 3 deleted tests.

3. IDOR on ``GET /metrics/api/journals/research/{id}`` — the ownership
   check is the ``research_history`` existence probe inside the *caller's*
   encrypted DB.

4. Egress-scope enforcement on ``POST /metrics/api/journal-data/download``.
   This route reaches out to OpenAlex/DOAJ, so it sits on
   ``test_full_surface_smoke.py``'s ``MUTATING_DENY_LIST`` and no sweep
   touches it at all. 6 + 3 deleted tests.

5. Per-user DB scoping of the strategy lookup in
   ``GET /history/details/{id}`` — ``routers/history.py`` must pass
   ``username=`` to ``get_research_strategy``; a revert to the implicit
   session fallback crosses user boundaries.

VACUITY (the trap this file exists to avoid): with an empty reference DB
every journals query returns ``[]``, and with an empty per-user DB every
isolation assertion holds even with the ownership predicate deleted. So:

* every ``sort``/``score_source`` case runs against a SEEDED reference DB
  whose three rows produce a DIFFERENT name order under each allow-listed
  column, and the allow-list tests assert the *positive* control (each
  legitimate value is accepted AND actually changes the ordering /
  actually filters) before asserting that hostile values are refused;
* every isolation test asserts the caller CAN read their own row first,
  then that they cannot read the other user's.

NETWORK: ``_get_ref_db_or_none()`` -> ``JournalQualityDB._ensure_engine``
-> ``ensure_journal_data()`` will try to DOWNLOAD several hundred MB from
upstream when no reference DB exists on disk. Every test that can reach
that code path uses the ``reference_db`` fixture, which points the
singleton at a small seeded SQLite file instead. Do not remove it from a
test "because it doesn't query journals" — the enrichment step runs
whenever the per-research query returns any row.
"""

import ast
import inspect
import itertools
import os
import uuid

import pytest
from fastapi.testclient import TestClient
from sqlalchemy import create_engine, func, select
from sqlalchemy.orm import Session as SASession

# The app's docs_url toggle reads PYTEST_CURRENT_TEST at app-import time;
# pytest only sets that per-test, not at collection (mirrors
# test_metrics_benchmark_hostile_input.py / test_full_surface_smoke.py).
os.environ.setdefault("TESTING", "1")

# Imported so this file fails loudly if a route is renamed or removed
# rather than silently asserting against a 404.
from local_deep_research.web.routers import history as history_router_mod
from local_deep_research.web.routers.history import router as history_router
from local_deep_research.web.routers.metrics import (
    _ALLOWED_SCORE_SOURCES,
    router as metrics_router,
)
from local_deep_research.journal_quality.db import _SORT_COLUMNS

_METRICS_PATHS = {r.path for r in metrics_router.routes if hasattr(r, "path")}
_HISTORY_PATHS = {r.path for r in history_router.routes if hasattr(r, "path")}

JOURNALS = "/metrics/api/journals"
RESEARCH_JOURNALS = "/metrics/api/journals/research/{}"
DOWNLOAD = "/metrics/api/journal-data/download"
HISTORY_DETAILS = "/history/details/{}"

PASSWORD = "MetricsHostile!Pass123"  # noqa: S105 — test-only credential


# ---------------------------------------------------------------------------
# Harness
# ---------------------------------------------------------------------------

# MONOTONIC, not random. Rate limiting is keyed per client IP
# (``dependencies/rate_limit._get_client_ip`` trusts X-Forwarded-For from
# the TestClient sentinel peer). Random octets collide inside a file this
# size and produce 429s from /auth/register's per-IP bucket that
# have nothing to do with the guard under test.
_IP_COUNTER = itertools.count(1)


def _next_forwarded_for() -> str:
    n = next(_IP_COUNTER)
    return f"10.211.{(n // 250) % 250}.{(n % 250) + 1}"


def _fresh_client(app):
    client = TestClient(app, raise_server_exceptions=False)
    client.headers.update({"X-Forwarded-For": _next_forwarded_for()})
    return client


def _csrf(client) -> str:
    """CSRF is enforced by ASGI middleware — fetch a real token."""
    client.get("/auth/login")
    return client.get("/auth/csrf-token").json()["csrf_token"]


def _register_and_login(app, username: str):
    client = _fresh_client(app)

    resp = client.post(
        "/auth/register",
        data={
            "username": username,
            "password": PASSWORD,
            "confirm_password": PASSWORD,
            "acknowledge": "true",
            "csrf_token": _csrf(client),
        },
        follow_redirects=False,
    )
    assert resp.status_code in (200, 302), (
        f"registration of {username!r} failed: "
        f"{resp.status_code} / {resp.text[:400]}"
    )

    resp = client.post(
        "/auth/login",
        data={
            "username": username,
            "password": PASSWORD,
            "csrf_token": _csrf(client),
        },
        follow_redirects=False,
    )
    assert resp.status_code in (200, 302), (
        f"login of {username!r} failed: {resp.status_code} / {resp.text[:400]}"
    )

    client.headers.update(
        {"X-CSRFToken": client.get("/auth/csrf-token").json()["csrf_token"]}
    )

    whoami = client.get("/auth/check")
    assert whoami.status_code == 200 and whoami.json().get("username") == (
        username
    ), f"session did not bind to {username!r}: {whoami.text[:300]}"
    return client


def _user(prefix: str) -> str:
    return f"{prefix}_{uuid.uuid4().hex[:10]}"


# ---------------------------------------------------------------------------
# Seeded reference DB
# ---------------------------------------------------------------------------

ALPHA = "Alpha Journal"
BETA = "Beta Journal"
GAMMA = "Gamma Journal"

# Three rows chosen so that EVERY allow-listed sort column has at least two
# distinct values, and so that several of them produce a name order that
# differs from the default (quality desc). Without that, "the guard always
# falls back to quality" and "the guard honours the requested column" would
# be indistinguishable.
#
# score_source is constrained at the DB level to ('openalex', 'doaj')
# (models.py::ck_sources_score_source), so 'llm' — which IS in the API
# allow-list for cache rows — cannot be seeded here; it is covered
# as an accepted-but-empty filter instead.
SEED_ROWS = [
    dict(
        name=ALPHA,
        issn="1111-1111",
        quality=9,
        h_index=10,
        impact_factor=1.0,
        quartile="Q3",
        publisher="Zeta Press",
        source_type="journal",
        is_predatory=False,
        is_in_doaj=False,
        score_source="openalex",
        cited_by_count=100,
    ),
    dict(
        name=BETA,
        issn="3333-3333",
        quality=5,
        h_index=30,
        impact_factor=3.0,
        quartile="Q1",
        publisher="Mid Press",
        source_type="conference",
        is_predatory=True,
        is_in_doaj=True,
        score_source="doaj",
        cited_by_count=300,
    ),
    dict(
        name=GAMMA,
        issn="2222-2222",
        quality=1,
        h_index=20,
        impact_factor=2.0,
        quartile="Q2",
        publisher="Alpha Press",
        source_type="repository",
        is_predatory=False,
        is_in_doaj=False,
        score_source="openalex",
        cited_by_count=200,
    ),
]

# sort=quality, order=desc — the route's documented defaults.
DEFAULT_ORDER = [ALPHA, BETA, GAMMA]


@pytest.fixture
def reference_db(tmp_path, monkeypatch):
    """A real ``JournalQualityDB`` over a small seeded file.

    Built with the production models + the production accessor class, so
    ``get_journals_page`` (and therefore ``_SORT_COLUMNS``) is the real
    code under test — only the *file* is substituted.

    ``PRAGMA user_version`` is left at 0, which ``_validate_existing_db``
    grandfathers in rather than rebuilding, so no download is triggered.
    """
    from local_deep_research.journal_quality import db as jq_db
    from local_deep_research.journal_quality.models import (
        JournalQualityBase,
        Source,
    )
    from local_deep_research.journal_quality.scoring import normalize_name

    path = tmp_path / "journal_quality.db"
    engine = create_engine(f"sqlite:///{path}")
    JournalQualityBase.metadata.create_all(engine)
    with SASession(engine) as session:
        for row in SEED_ROWS:
            session.add(Source(name_lower=normalize_name(row["name"]), **row))
        session.commit()
    engine.dispose()

    ref = jq_db.JournalQualityDB()
    monkeypatch.setattr(ref, "_resolve_db_path", lambda: path)
    monkeypatch.setattr(jq_db, "get_journal_reference_db", lambda: ref)
    assert ref.available, "seeded reference DB did not open"
    return path


@pytest.fixture
def journals_client(app, reference_db):
    """One logged-in user plus the seeded reference DB."""
    return _register_and_login(app, _user("jq"))


def _names(response) -> list:
    payload = response.json()
    assert payload.get("status") == "success", payload
    return [j["name"] for j in payload["journals"]]


def _reference_row_count(path) -> int:
    """Re-open the seeded file directly and count ``sources`` rows."""
    from local_deep_research.journal_quality.models import Source

    engine = create_engine(f"sqlite:///{path}")
    try:
        with engine.connect() as conn:
            return conn.execute(
                select(func.count()).select_from(Source)
            ).scalar_one()
    finally:
        engine.dispose()


# ---------------------------------------------------------------------------
# Premise guard
# ---------------------------------------------------------------------------


def test_routes_under_test_still_exist():
    """If any of these move, every case below would pass against a 404."""
    assert "/metrics/api/journals" in _METRICS_PATHS
    assert "/metrics/api/journals/research/{research_id}" in _METRICS_PATHS
    assert "/metrics/api/journal-data/download" in _METRICS_PATHS
    assert "/history/details/{research_id}" in _HISTORY_PATHS


def test_sort_allowlist_is_a_closed_set():
    """The allow-list must stay an explicit closed set of column names.

    A regression that widens it to "any Source attribute" (or drops it)
    re-opens the ORDER BY to ``getattr(Source, <user input>)``.
    """
    assert isinstance(_SORT_COLUMNS, frozenset)
    assert _SORT_COLUMNS == {
        "name",
        "quality",
        "quartile",
        "h_index",
        "impact_factor",
        "score_source",
        "source_type",
        "publisher",
        "is_predatory",
    }


# ===========================================================================
# COVERAGE AREA 1 — sort-field restriction on GET /metrics/api/journals
# ===========================================================================


def test_default_sort_is_quality_desc(journals_client):
    """Baseline for every fallback assertion below.

    If this order ever changes, the injection tests would be comparing
    against the wrong expectation and could pass vacuously.
    """
    resp = journals_client.get(JOURNALS)
    assert resp.status_code == 200, resp.text[:400]
    assert _names(resp) == DEFAULT_ORDER


# NB on structure: the sweeps below loop inside a single test rather than
# using @parametrize. Each case needs an authenticated caller, and the only
# app fixture available is function-scoped (fresh temp data dir per test),
# so one registered user per parameter turned this file into a 5-minute
# run. The loops carry the case in every assertion message, so a failure
# still names the exact payload.


# POSITIVE CONTROL. Each allow-listed column, in both directions. Without
# these, "reject/ignore everything" would satisfy the injection tests.
def test_every_allowlisted_sort_column_is_accepted_and_orders_by_it(
    journals_client,
):
    for column in sorted(_SORT_COLUMNS):
        for order in ("asc", "desc"):
            case = f"sort={column!r} order={order!r}"
            resp = journals_client.get(
                JOURNALS, params={"sort": column, "order": order}
            )
            assert resp.status_code == 200, f"{case}: {resp.text[:300]}"
            payload = resp.json()
            assert payload["status"] == "success", case
            rows = payload["journals"]
            assert len(rows) == len(SEED_ROWS), (case, rows)

            # bool -> int so JSON booleans compare like the SQL column does.
            values = [
                int(r[column]) if isinstance(r[column], bool) else r[column]
                for r in rows
            ]
            assert len(set(values)) >= 2, (
                f"{case}: seed data gives fewer than 2 distinct values — "
                f"the monotonicity assertion below would be vacuous"
            )
            pairs = list(zip(values, values[1:]))
            if order == "desc":
                assert all(a >= b for a, b in pairs), (case, values)
            else:
                assert all(a <= b for a, b in pairs), (case, values)


# POSITIVE CONTROL, exact orders. Proves a legitimate sort value really
# reaches the ORDER BY and produces a DIFFERENT row order — several of
# these differ from DEFAULT_ORDER, so a "silently always sort by quality"
# regression fails here.
EXACT_SORT_ORDERS = [
    ("name", "asc", [ALPHA, BETA, GAMMA]),
    ("name", "desc", [GAMMA, BETA, ALPHA]),
    ("h_index", "desc", [BETA, GAMMA, ALPHA]),
    ("h_index", "asc", [ALPHA, GAMMA, BETA]),
    ("impact_factor", "desc", [BETA, GAMMA, ALPHA]),
    ("quartile", "asc", [BETA, GAMMA, ALPHA]),
    ("quartile", "desc", [ALPHA, GAMMA, BETA]),
    ("source_type", "asc", [BETA, ALPHA, GAMMA]),
    ("publisher", "asc", [GAMMA, BETA, ALPHA]),
    ("quality", "asc", [GAMMA, BETA, ALPHA]),
]


def test_allowlisted_sort_changes_the_actual_row_order(journals_client):
    differs_from_default = 0
    for column, order, expected in EXACT_SORT_ORDERS:
        case = f"sort={column!r} order={order!r}"
        resp = journals_client.get(
            JOURNALS, params={"sort": column, "order": order}
        )
        assert resp.status_code == 200, f"{case}: {resp.text[:300]}"
        assert _names(resp) == expected, case
        if expected != DEFAULT_ORDER:
            differs_from_default += 1
    assert differs_from_default >= 5, (
        "the expectation table no longer distinguishes 'honours the "
        "requested column' from 'always falls back to quality desc'"
    )


# Payloads that are not Source attributes at all. Without the allow-list
# ``getattr(Source, sort)`` raises AttributeError into the route's outer
# ``except Exception`` -> hardcoded 500.
SORT_INJECTION_PAYLOADS = [
    "1; DROP TABLE sources--",
    "name); DELETE FROM sources--",
    "quality; DROP TABLE sources; --",
    "quality DESC, (SELECT 1)",
    "name' OR '1'='1",
    "quality UNION SELECT issn FROM sources",
    "QUALITY",
    "Quality",
    "H_Index",
    " quality",
    "quality ",
    "qualіty",  # Cyrillic i
    "quality\u200b",  # trailing zero-width space
    "quality",  # fullwidth q
    "quality\x00",
    "",
    "../../etc/passwd",
]


def test_sort_injection_payloads_fall_back_to_the_safe_default(
    journals_client, reference_db
):
    """A non-allow-listed ``sort`` never reaches the ORDER BY.

    The observed guard is FALL BACK (``db.py`` sets ``sort = "quality"``),
    not reject — so the contract pinned here is: 200, and the rows come
    back in the default quality-desc order, identical to a request with no
    ``sort`` at all.
    """
    for payload in SORT_INJECTION_PAYLOADS:
        resp = journals_client.get(JOURNALS, params={"sort": payload})
        assert resp.status_code == 200, (
            f"sort={payload!r} produced {resp.status_code}: {resp.text[:300]}"
        )
        assert _names(resp) == DEFAULT_ORDER, f"sort={payload!r}"

        body = resp.text.lower()
        for leak in (
            "traceback",
            "sqlalchemy",
            "no such column",
            "operationalerror",
        ):
            assert leak not in body, (
                f"sort={payload!r} leaked {leak!r}: {body[:300]}"
            )

    # The reference DB is opened mode=ro/immutable, but assert the rows
    # survived anyway: this is the assertion that would notice if a future
    # refactor swapped in a writable engine.
    assert _reference_row_count(reference_db) == len(SEED_ROWS)


# Real Source columns that are deliberately NOT in the allow-list. These
# are the dangerous half: they do NOT raise, so without the allow-list the
# request silently succeeds with a different ordering and nothing 500s.
NON_ALLOWLISTED_REAL_COLUMNS = [
    ("issn", [BETA, GAMMA, ALPHA]),
    ("name_lower", [GAMMA, BETA, ALPHA]),
    ("id", [GAMMA, BETA, ALPHA]),
    ("cited_by_count", [BETA, GAMMA, ALPHA]),
    ("is_in_doaj", [BETA, ALPHA, GAMMA]),
]


def test_real_but_non_allowlisted_columns_are_refused_not_honoured(
    journals_client,
):
    for column, unguarded_order in NON_ALLOWLISTED_REAL_COLUMNS:
        # Sanity: the seed really does order differently under this column,
        # so the assertion below has teeth.
        assert unguarded_order != DEFAULT_ORDER, column
        resp = journals_client.get(JOURNALS, params={"sort": column})
        assert resp.status_code == 200, f"sort={column!r}: {resp.text[:300]}"
        assert _names(resp) == DEFAULT_ORDER, (
            f"sort={column!r} was honoured — the ORDER BY accepted a column "
            f"outside _SORT_COLUMNS"
        )


# Python/SQLAlchemy attributes that exist on the mapped class but are not
# columns. ``getattr(Source, "metadata").desc()`` raises AttributeError.
def test_sort_python_attribute_names_do_not_reach_getattr(journals_client):
    for payload in (
        "metadata",
        "registry",
        "__class__",
        "__init__",
        "__dict__",
    ):
        resp = journals_client.get(JOURNALS, params={"sort": payload})
        assert resp.status_code == 200, (
            f"sort={payload!r} produced {resp.status_code} (a 500 here means "
            f"the value reached getattr(Source, ...)): {resp.text[:300]}"
        )
        assert _names(resp) == DEFAULT_ORDER, f"sort={payload!r}"


def test_order_direction_outside_allowlist_falls_back_to_desc(journals_client):
    """``order`` feeds the same ORDER BY clause and has its own two-value
    allow-list in ``get_journals_page``."""
    for payload in ("asc; DROP TABLE sources--", "DESC", "ASC", "sideways", ""):
        resp = journals_client.get(
            JOURNALS, params={"sort": "name", "order": payload}
        )
        assert resp.status_code == 200, f"order={payload!r}: {resp.text[:300]}"
        assert _names(resp) == [GAMMA, BETA, ALPHA], (
            f"order={payload!r} did not fall back to desc"
        )


# ===========================================================================
# COVERAGE AREA 2 — score_source allow-list
# ===========================================================================


def test_score_source_allowlist_is_a_closed_set():
    assert _ALLOWED_SCORE_SOURCES == frozenset({"openalex", "doaj", "llm"})


# POSITIVE CONTROL: every allowed value is accepted AND actually filters.
# 'llm' is allow-listed for cache rows; the reference DB's CHECK
# constraint forbids seeding one, so its assertion is "accepted with an
# empty result", not "rejected".
ALLOWED_SCORE_SOURCE_RESULTS = [
    ("openalex", [ALPHA, GAMMA]),
    ("doaj", [BETA]),
    ("llm", []),
]


def test_allowed_score_source_values_are_accepted_and_filter(journals_client):
    assert {v for v, _ in ALLOWED_SCORE_SOURCE_RESULTS} == set(
        _ALLOWED_SCORE_SOURCES
    ), "every allow-listed value must have a positive control"
    for value, expected in ALLOWED_SCORE_SOURCE_RESULTS:
        resp = journals_client.get(JOURNALS, params={"score_source": value})
        assert resp.status_code == 200, (
            f"score_source={value!r}: {resp.text[:300]}"
        )
        assert _names(resp) == expected, f"score_source={value!r}"
        assert resp.json()["pagination"]["total_count"] == len(expected)


def test_empty_score_source_means_no_filter(journals_client):
    for params in ({}, {"score_source": ""}):
        resp = journals_client.get(JOURNALS, params=params)
        assert resp.status_code == 200, f"{params}: {resp.text[:300]}"
        assert sorted(_names(resp)) == sorted([ALPHA, BETA, GAMMA]), params


SCORE_SOURCE_REJECTED = [
    "bogus",
    "OPENALEX",
    "OpenAlex",
    " openalex",
    "openalex ",
    "openalex'--",
    "openalex' OR '1'='1",
    "openalex; DROP TABLE sources--",
    "opеnalex",  # Cyrillic e
    "openalex\u200b",  # trailing zero-width space
    "%",
    "openalex,doaj",
    "['openalex']",
]


def test_score_source_outside_allowlist_returns_400(journals_client):
    expected_message = (
        f"Invalid score_source; must be one of {sorted(_ALLOWED_SCORE_SOURCES)}"
    )
    for value in SCORE_SOURCE_REJECTED:
        resp = journals_client.get(JOURNALS, params={"score_source": value})
        assert resp.status_code == 400, (
            f"score_source={value!r} returned {resp.status_code} instead of "
            f"400: {resp.text[:300]}"
        )
        payload = resp.json()
        assert payload["status"] == "error", value
        # Fixed text built from the allow-list — the rejected value is never
        # echoed back, so a payload cannot ride the error message out.
        assert payload["message"] == expected_message, value
        body = resp.text.lower()
        for leak in ("traceback", "sqlalchemy", "operationalerror"):
            assert leak not in body, (value, leak)


def test_rejected_score_source_does_not_leak_rows(journals_client):
    """A refused filter must not degrade into "no filter"."""
    resp = journals_client.get(JOURNALS, params={"score_source": "bogus"})
    assert resp.status_code == 400
    assert "journals" not in resp.json()


# ===========================================================================
# COVERAGE AREA 3 — ownership on GET /metrics/api/journals/research/{id}
# ===========================================================================


def _seed_research_with_paper(username, research_id, query, journal_name):
    """Seed one research + one resource + one paper into a user's own
    encrypted DB, wired through the Paper -> PaperAppearance ->
    ResearchResource join chain the route walks."""
    from local_deep_research.database.models import (
        Paper,
        PaperAppearance,
        ResearchHistory,
        ResearchResource,
    )
    from local_deep_research.database.session_context import (
        get_user_db_session,
    )

    with get_user_db_session(username, PASSWORD) as session:
        session.add(
            ResearchHistory(
                id=research_id,
                query=query,
                mode="quick_summary",
                status="completed",
                created_at="2024-01-01T00:00:00",
            )
        )
        session.flush()
        resource = ResearchResource(
            research_id=research_id,
            title=query,
            url="https://example.invalid/paper",
            created_at="2024-01-01T00:00:00",
        )
        session.add(resource)
        paper = Paper(
            doi=f"10.9999/{research_id}",
            container_title=journal_name,
            year=2021,
        )
        session.add(paper)
        session.flush()
        session.add(PaperAppearance(paper_id=paper.id, resource_id=resource.id))
        session.commit()


RID_A = "aaaaaaaa-1111-4111-8111-aaaaaaaaaaaa"
RID_B = "bbbbbbbb-2222-4222-8222-bbbbbbbbbbbb"


@pytest.fixture
def two_seeded_users(app, reference_db):
    """Two real users, each with their own research + paper row.

    Both sides are seeded on purpose: if only user A had data, "user B
    sees nothing" would also hold for a completely broken B database, and
    the isolation assertion would be vacuous.
    """
    alice, bob = _user("alice"), _user("bob")
    alice_client = _register_and_login(app, alice)
    bob_client = _register_and_login(app, bob)
    _seed_research_with_paper(alice, RID_A, "alice query", ALPHA)
    _seed_research_with_paper(bob, RID_B, "bob query", BETA)
    return {
        "alice": alice,
        "bob": bob,
        "alice_client": alice_client,
        "bob_client": bob_client,
    }


def test_research_journals_owner_can_read_own_metrics(two_seeded_users):
    """POSITIVE CONTROL for the isolation test below.

    Without this, an empty/broken metrics DB would make "user B sees no
    rows of user A" pass with the ownership predicate entirely removed.
    """
    resp = two_seeded_users["alice_client"].get(RESEARCH_JOURNALS.format(RID_A))
    assert resp.status_code == 200, resp.text[:400]
    payload = resp.json()
    assert payload["status"] == "success"
    assert [j["name"] for j in payload["journals"]] == [ALPHA]
    assert payload["summary"]["total_journals"] == 1
    assert payload["summary"]["total_papers"] == 1

    resp = two_seeded_users["bob_client"].get(RESEARCH_JOURNALS.format(RID_B))
    assert resp.status_code == 200, resp.text[:400]
    assert [j["name"] for j in resp.json()["journals"]] == [BETA]


def test_research_journals_are_not_readable_across_users(two_seeded_users):
    """IDOR: user B must not read user A's per-research journal metrics."""
    resp = two_seeded_users["bob_client"].get(RESEARCH_JOURNALS.format(RID_A))
    assert resp.status_code == 404, (
        f"bob read alice's research metrics: {resp.status_code} "
        f"{resp.text[:400]}"
    )
    assert ALPHA not in resp.text
    assert "alice query" not in resp.text

    resp = two_seeded_users["alice_client"].get(RESEARCH_JOURNALS.format(RID_B))
    assert resp.status_code == 404, (
        f"alice read bob's research metrics: {resp.status_code} "
        f"{resp.text[:400]}"
    )
    assert BETA not in resp.text


def test_research_journals_unknown_id_is_404_without_leaking(two_seeded_users):
    for research_id in (
        "cccccccc-3333-4333-8333-cccccccccccc",
        "not-a-uuid",
        "1 OR 1=1",
        "%27%20OR%201%3D1",
    ):
        resp = two_seeded_users["alice_client"].get(
            RESEARCH_JOURNALS.format(research_id)
        )
        assert resp.status_code == 404, (
            f"research_id={research_id!r}: {resp.text[:300]}"
        )
        body = resp.text.lower()
        for leak in ("traceback", "sqlalchemy", "no such", "sqlite"):
            assert leak not in body, (research_id, resp.text[:300])


# ===========================================================================
# COVERAGE AREA 4 — journal-download egress-scope enforcement
# ===========================================================================

# The route is rate-limited to "2 per hour" per authenticated user
# (_JOURNAL_DATA_LIMIT), so each test below registers its own user and
# issues at most two POSTs. Sharing a user across cases would 429.


def _set_scope(username, raw):
    from local_deep_research.utilities.db_utils import get_settings_manager

    manager = get_settings_manager(username=username)
    manager.set_setting("policy.egress_scope", raw)
    assert manager.get_setting("policy.egress_scope", "adaptive") == raw, (
        "the scope under test was not persisted"
    )


def _stub_downloader(monkeypatch, *, success, message, counts=None):
    """Replace the outbound download so no test ever hits the network."""
    from local_deep_research.journal_quality import downloader

    monkeypatch.setattr(
        downloader,
        "download_journal_data",
        lambda force=False: (success, message),
    )
    monkeypatch.setattr(
        downloader, "get_download_state", lambda: {"counts": counts}
    )


REFUSED_SCOPES = [
    pytest.param("private_only", id="private_only"),
    pytest.param("strict", id="strict"),
    pytest.param("  PRIVATE_ONLY  ", id="whitespace-and-case-padded"),
    pytest.param("STRICT", id="upper-case"),
    pytest.param("totally-bogus", id="corrupt-value-fails-closed"),
    pytest.param("", id="empty-value-fails-closed"),
    pytest.param("private_only; DROP TABLE settings--", id="injection-ish"),
]


@pytest.mark.parametrize("scope", REFUSED_SCOPES)
def test_journal_download_refused_under_offline_or_corrupt_scope(
    app, monkeypatch, scope
):
    """The manual "Download Data" button must fail closed.

    An unresolvable scope (``parse_user_egress_scope`` raises
    ``PolicyDeniedError`` -> ``scope = None``) is refused exactly like an
    explicit offline scope. The downloader is stubbed so that a regression
    which *does* proceed fails on the status code instead of spending
    minutes on real egress.
    """
    username = _user("egress")
    client = _register_and_login(app, username)
    _set_scope(username, scope)
    _stub_downloader(
        monkeypatch, success=True, message="SHOULD-NOT-RUN", counts=None
    )

    resp = client.post(DOWNLOAD, json={})
    assert resp.status_code == 403, (
        f"scope={scope!r} was not refused: {resp.status_code} {resp.text[:300]}"
    )
    payload = resp.json()
    assert payload["success"] is False
    assert "egress policy" in payload["message"]
    # The refusal must not echo the stored scope value or downloader state.
    if scope.strip():
        assert scope.strip() not in payload["message"]
    assert "SHOULD-NOT-RUN" not in resp.text
    body = resp.text.lower()
    for leak in ("traceback", "policydeniederror", "sqlalchemy"):
        assert leak not in body, resp.text[:300]


# POSITIVE CONTROL for the refusal above: scopes that resolve to the
# adaptive default must NOT be refused, or "always 403" would pass every
# test in this section.
@pytest.mark.parametrize(
    "scope,why",
    [
        ("adaptive", "the registry default"),
        ("public_only", "explicit public egress"),
        ("both", "retired scope, read as adaptive"),
    ],
)
def test_journal_download_allowed_under_resolvable_public_scope(
    app, monkeypatch, scope, why
):
    username = _user("egressok")
    client = _register_and_login(app, username)
    _set_scope(username, scope)
    _stub_downloader(monkeypatch, success=True, message="internal", counts=None)

    resp = client.post(DOWNLOAD, json={})
    assert resp.status_code == 200, (
        f"scope={scope!r} ({why}) was refused: {resp.status_code} "
        f"{resp.text[:300]}"
    )
    assert resp.json()["success"] is True


def test_journal_download_does_not_read_the_default_scope_during_a_sidecar_fault(
    app, monkeypatch
):
    """A stored ``strict`` must never be replaced by the default scope.

    The route reads ``policy.egress_scope`` through
    ``get_settings_manager(username=...)``, which reaches the user's cached
    engine through ``db_manager.get_session``. A rekey sidecar fault there
    (an fd-exhaustion spike that starts after ``require_auth``'s probe
    passed) used to come back as that method's None, which
    ``get_settings_manager`` answers as "this user has no database" from
    the defaults file: the scope read as the default ``adaptive`` and the
    public download went ahead against the stored ``strict``. The request
    must fail closed with the retryable 503 instead, and the stored scope
    must still refuse the download once the fault clears.

    The fault is injected at the real ``os.open`` of this user's sidecar,
    only while the route reads the scope.
    """
    import errno

    from local_deep_research.database import encrypted_db, rekey_lock
    from local_deep_research.journal_quality import downloader
    from local_deep_research.web.routers import metrics as metrics_routes

    username = _user("egressfault")
    client = _register_and_login(app, username)
    _set_scope(username, "strict")
    downloads = []

    def record_download(force=False):
        downloads.append(force)
        return True, "SHOULD-NOT-RUN"

    monkeypatch.setattr(downloader, "download_journal_data", record_download)
    monkeypatch.setattr(
        downloader, "get_download_state", lambda: {"counts": None}
    )

    user_db = encrypted_db.db_manager._get_user_db_path(username)
    sidecar = os.fspath(user_db.with_name(f"{user_db.name}.rekey.lock"))
    real_open = rekey_lock.os.open
    spike = {"on": False, "hits": 0}

    def emfile_during_spike(path, flags, *args, **kwargs):
        if spike["on"] and os.fspath(path) == sidecar:
            spike["hits"] += 1
            raise OSError(errno.EMFILE, "Too many open files")
        return real_open(path, flags, *args, **kwargs)

    real_resolve = metrics_routes._resolve_persisted_egress_scope

    def resolve_during_spike(name):
        # require_auth (and DatabaseMiddleware's probe before it) passed:
        # the route body is running.
        spike["on"] = True
        try:
            return real_resolve(name)
        finally:
            spike["on"] = False

    monkeypatch.setattr(rekey_lock.os, "open", emfile_during_spike)
    monkeypatch.setattr(
        metrics_routes, "_resolve_persisted_egress_scope", resolve_during_spike
    )
    resp = client.post(DOWNLOAD, json={})

    assert spike["hits"] >= 1, (
        "the scope read never reached the sidecar: measuring the wrong branch"
    )
    assert downloads == [], (
        "the public download ran during a sidecar fault, i.e. with the "
        f"default scope instead of the stored 'strict': {resp.text[:300]}"
    )
    assert resp.status_code == 503, (
        f"expected the retryable 503, got {resp.status_code}: {resp.text[:300]}"
    )
    assert resp.headers.get("retry-after") == "5"
    assert resp.json() == {
        "detail": "Your database is temporarily unavailable. "
        "Please try again in a moment."
    }
    assert "errno" not in resp.text

    # Once the fault clears, the stored scope is read again and refuses.
    monkeypatch.setattr(rekey_lock.os, "open", real_open)
    resp = client.post(DOWNLOAD, json={})
    assert resp.status_code == 403, resp.text[:300]
    assert downloads == []


def test_stale_unprotected_scope_with_gate_off_is_treated_as_adaptive(
    app, monkeypatch
):
    """A stored ``unprotected`` with the operator gate OFF must resolve to
    adaptive (``disabled_unprotected="adaptive"``), not fail closed —
    matching every other runtime reader."""
    from local_deep_research.security.egress.policy import (
        unprotected_egress_allowed,
    )

    assert unprotected_egress_allowed() is False, (
        "precondition: the operator escape hatch must be off by default"
    )

    username = _user("egressunp")
    client = _register_and_login(app, username)
    _set_scope(username, "unprotected")
    _stub_downloader(monkeypatch, success=True, message="internal", counts=None)

    resp = client.post(DOWNLOAD, json={})
    assert resp.status_code == 200, resp.text[:300]
    assert resp.json()["success"] is True


INTERNAL_SECRET = (
    "OpenAlex fetch failed at /home/ldr/.local/share/ldr/journal_quality.db "
    "(api_key=sk-live-abcdef)"
)


def test_download_failure_and_success_do_not_echo_internal_messages(
    app, monkeypatch
):
    """CWE-209 on both terminal paths of the route.

    ``download_journal_data`` returns an internal message on failure and
    on success; neither may reach the client. The success message is
    rebuilt locally from structured state instead.
    """
    username = _user("egressleak")
    client = _register_and_login(app, username)
    _set_scope(username, "adaptive")

    _stub_downloader(monkeypatch, success=False, message=INTERNAL_SECRET)
    resp = client.post(DOWNLOAD, json={})
    assert resp.status_code == 200, resp.text[:300]
    assert resp.json() == {"success": False, "message": "Download failed"}
    assert "sk-live" not in resp.text
    assert "/home/ldr" not in resp.text

    _stub_downloader(monkeypatch, success=True, message=INTERNAL_SECRET)
    resp = client.post(DOWNLOAD, json={})
    assert resp.status_code == 200, resp.text[:300]
    payload = resp.json()
    assert payload["success"] is True
    assert "sk-live" not in resp.text
    assert "/home/ldr" not in resp.text
    assert INTERNAL_SECRET not in resp.text


def test_download_exception_path_does_not_leak_a_traceback(app, monkeypatch):
    from local_deep_research.journal_quality import downloader

    username = _user("egressboom")
    client = _register_and_login(app, username)
    _set_scope(username, "adaptive")

    def _boom(force=False):
        raise RuntimeError(INTERNAL_SECRET)

    monkeypatch.setattr(downloader, "download_journal_data", _boom)

    resp = client.post(DOWNLOAD, json={})
    assert resp.status_code == 500, resp.text[:300]
    assert resp.json() == {"success": False, "message": "Download failed"}
    body = resp.text.lower()
    for leak in ("traceback", "runtimeerror", "sk-live", "/home/ldr"):
        assert leak not in body, resp.text[:300]


# ===========================================================================
# COVERAGE AREA 5 — per-user history strategy lookup
# ===========================================================================

RID_SHARED = "dddddddd-4444-4444-8444-dddddddddddd"
STRATEGY_A = "strategy-belonging-to-alice"
STRATEGY_B = "strategy-belonging-to-bob"


def _seed_research_with_strategy(username, research_id, query, strategy_name):
    from local_deep_research.database.models import (
        ResearchHistory,
        ResearchStrategy,
    )
    from local_deep_research.database.session_context import (
        get_user_db_session,
    )

    with get_user_db_session(username, PASSWORD) as session:
        session.add(
            ResearchHistory(
                id=research_id,
                query=query,
                mode="quick_summary",
                status="completed",
                created_at="2024-01-01T00:00:00",
            )
        )
        session.flush()
        session.add(
            ResearchStrategy(
                research_id=research_id, strategy_name=strategy_name
            )
        )
        session.commit()


@pytest.fixture
def strategy_users(app):
    """Two users holding the SAME research_id with DIFFERENT strategies.

    Colliding ids is the whole point: a strategy lookup that reads from
    anything other than the caller's own database returns the other
    user's ``strategy_name`` and this fixture makes that visible. A
    lookup that returns ``None`` (the shape of a revert to the implicit
    session fallback) is caught by the same assertion.
    """
    alice, bob = _user("halice"), _user("hbob")
    alice_client = _register_and_login(app, alice)
    bob_client = _register_and_login(app, bob)
    _seed_research_with_strategy(
        alice, RID_SHARED, "alice history query", STRATEGY_A
    )
    _seed_research_with_strategy(
        bob, RID_SHARED, "bob history query", STRATEGY_B
    )
    return {
        "alice": alice,
        "bob": bob,
        "alice_client": alice_client,
        "bob_client": bob_client,
    }


def test_history_details_strategy_comes_from_the_callers_own_db(
    strategy_users,
):
    resp = strategy_users["alice_client"].get(
        HISTORY_DETAILS.format(RID_SHARED)
    )
    assert resp.status_code == 200, resp.text[:400]
    alice_payload = resp.json()
    assert alice_payload["strategy"] == STRATEGY_A, alice_payload
    assert alice_payload["query"] == "alice history query"
    assert STRATEGY_B not in resp.text

    resp = strategy_users["bob_client"].get(HISTORY_DETAILS.format(RID_SHARED))
    assert resp.status_code == 200, resp.text[:400]
    bob_payload = resp.json()
    assert bob_payload["strategy"] == STRATEGY_B, bob_payload
    assert bob_payload["query"] == "bob history query"
    assert STRATEGY_A not in resp.text


def test_get_research_strategy_selects_the_db_by_username(strategy_users):
    """The service itself, not just the route: ``username`` must pick the
    database, so the route's kwarg is load-bearing rather than decorative.
    """
    from local_deep_research.web.services.research_service import (
        get_research_strategy,
    )

    assert (
        get_research_strategy(RID_SHARED, username=strategy_users["alice"])
        == STRATEGY_A
    )
    assert (
        get_research_strategy(RID_SHARED, username=strategy_users["bob"])
        == STRATEGY_B
    )


def test_history_details_passes_username_to_the_strategy_lookup():
    """Static guard on the call site.

    ``get_research_strategy``'s ``username`` is keyword-only; a revert to
    the implicit session fallback would drop the kwarg. Pinned here so the
    regression is caught even if the two-user runtime test above is ever
    weakened.
    """
    tree = ast.parse(inspect.getsource(history_router_mod.get_research_details))
    calls = [
        node
        for node in ast.walk(tree)
        if isinstance(node, ast.Call)
        and isinstance(node.func, ast.Name)
        and node.func.id == "get_research_strategy"
    ]
    assert len(calls) == 1, "expected exactly one strategy lookup in the route"
    kwargs = {kw.arg for kw in calls[0].keywords}
    assert "username" in kwargs, (
        "get_research_strategy must be called with an explicit username= "
        "so the strategy is read from the caller's own database"
    )