Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

85.7 KB
Raw Blame
"""Route-layer authorization / input-bound coverage for the library and notes
surfaces lost in the Flask -> FastAPI migration (PR #3299).

Companion to ``tests/security/test_library_rag_security_fastapi.py``, which
provides evidence for the single-document note-protection 403 and
``_format_test_embedding_error``. This file covers four additional areas
identified by the historical ADR-0010 review. Nothing here duplicates it.

COVERAGE AREA 1 -- note protection on bulk paths and the protected-collection
    flag. ``DELETE /library/api/documents/bulk`` is already covered next door;
    its two mutating siblings are not:
    ``DELETE /library/api/documents/blobs`` (loops ``delete_blob_only``) and
    ``DELETE /library/api/collection/{cid}/documents/bulk`` (loops
    ``remove_from_collection``). Neither route has the single-document route's
    403 mapping -- ``BulkDeletionService`` returns 200 with a per-item failure --
    so the SERVICE guard is the only thing standing between a "select all ->
    delete" in the library UI and mass note damage. Also here:
    ``rag.py::_is_protected_collection``, whose result the collection-details
    route serializes so the UI can hide destructive affordances on the three
    system collection types.

COVERAGE AREA 2 -- ``POST /library/api/open-folder`` must HARD-403 (the branch's smoke
    sweeps only assert ``< 500``, which a re-enabled 200 satisfies); the upload
    EXTENSION allowlist at the route (size and count caps are already covered by
    ``tests/web/routers/test_rag_upload_limits_source_of_truth.py``, extension is
    not); ``library.py::get_authenticated_user_password`` fail-closed; and the
    SSE auth-failure fail-closed events in ``download_all_text`` /
    ``download_bulk``.

COVERAGE AREA 3 -- notes: version-read cross-note scoping (the branch covers
    the analogous guard for RESTORE only), the fact-check grade route's
    cross-resource authorization ("research is not linked to this note") and its
    claim sanitisation, the ``MAX_SEARCH_LEN`` / ``MAX_CLAIM_LEN`` /
    ``MAX_PASSAGE_LEN`` input-size clamps, the ``MAX_LINK_TEXT_LEN`` clamp on
    ``POST /notes/api/notes/resolve-link``, and the annotation-delete
    anchor/ownership check on ``DELETE .../research/{id}/annotations/{note_id}``
    and ``.../documents/{id}/annotations/{note_id}``.

COVERAGE AREA 4 -- the collection-type allowlist (``rag.py:1761``) and the
    system-collection rename guard (``rag.py:1885``) AT THE ROUTE, plus
    ``library_delete.py:288``'s protected-type -> 409 mapping. That last one is
    helper-covered (``tests/deletion/test_collection_deletion.py`` proves the
    service refuses), but the route mapping lacked direct coverage at the
    review snapshot. This file now pins that call site and status contract.

Harness (matching the sibling security files):
  * ``TestClient(app, raise_server_exceptions=False)`` against the live FastAPI
    app -- no dependency overrides, because every route here opens the caller's
    ENCRYPTED per-user database and that only works after a real login.
  * CSRF is ASGI-middleware-enforced, so each client mints a real token. A bare
    mutating request 403s at CSRF before any dependency runs, which would test
    CSRF rather than the guard under test.
  * Rate limiting is keyed per client IP (registration is limited per IP) AND, for the
    notes AI buckets, per authenticated user (``_notes_factcheck_limit`` is
    3/minute). Every test therefore gets a FRESH user on a FRESH IP drawn from a
    MONOTONIC counter -- random addresses collide in a long session and surface
    as unrelated 429s.
  * Rows are seeded into the caller's own encrypted DB through
    ``get_user_db_session(username, password)`` so the guards run against real
    data, not mocks.

VACUITY: every "the destructive thing did not happen" assertion is paired with a
positive control asserted in the same test or an adjacent one -- an empty
library, a 500, or a blanket refusal would otherwise satisfy the negative half
on its own. Each protected case is also paired with its 404 / 200 discriminator
so a route that collapsed every outcome into one status could not pass.
"""

import itertools
import json
import os
import uuid
from contextlib import contextmanager, suppress
from datetime import datetime, timezone
from unittest.mock import patch

import pytest
from fastapi.testclient import TestClient

# The app's docs_url toggle reads PYTEST_CURRENT_TEST at app-import time;
# pytest only sets that per-test, not at collection.
os.environ.setdefault("TESTING", "1")

PASSWORD = "TestPassword123!"  # noqa: S105 -- test-only credential

# Monotonic, never-reused client IPs. Rate limiting buckets per IP, so a random
# address can collide with an earlier client in a large session and produce a
# 429 that has nothing to do with the assertion under test. 172.31/16 also keeps
# these clear of the 10/8 addresses the sibling security files draw from.
_IP_COUNTER = itertools.count(1)


def _next_forwarded_for():
    n = next(_IP_COUNTER)
    return f"172.31.{(n // 250) % 250}.{(n % 250) + 1}"


# ---------------------------------------------------------------------------
# Harness
# ---------------------------------------------------------------------------


def _csrf(client):
    """CSRF is enforced by ASGI middleware -- fetch a real token."""
    client.get("/auth/login")
    resp = client.get("/auth/csrf-token")
    return resp.json().get("csrf_token", "") if resp.status_code == 200 else ""


def _cleanup_client(client, username):
    try:
        with suppress(Exception):
            client.post("/auth/logout", follow_redirects=False)
        with suppress(Exception):
            from local_deep_research.web.auth.session_manager import (
                session_manager,
            )

            session_manager.destroy_all_user_sessions(username)
        with suppress(Exception):
            from local_deep_research.database.session_passwords import (
                session_password_store,
            )

            session_password_store.clear_all_for_user(username)
        with suppress(Exception):
            from local_deep_research.database.thread_local_session import (
                clear_user_credentials,
            )

            clear_user_credentials(username)
    finally:
        client.close()


@pytest.fixture
def authed(temp_data_dir, monkeypatch, request):
    """One real registered and auto-authenticated user per test.

    Returns ``(client, username, password)``. A per-test user (not a
    module-scoped one) is required because the notes AI rate limits key on the
    authenticated user: ``_notes_factcheck_limit`` is 3/minute, which a shared
    user would exhaust part-way through this file.
    """
    monkeypatch.setenv("LDR_DATA_DIR", str(temp_data_dir))
    # Production PBKDF2 iterations would dominate wall-clock in a per-test
    # fixture (same reason tests/conftest.py's `app` fixture lowers it).
    monkeypatch.setenv("LDR_DB_CONFIG_KDF_ITERATIONS", "1000")

    from local_deep_research.web.fastapi_app import app

    client = TestClient(app, raise_server_exceptions=False)
    username = f"lna_{uuid.uuid4().hex[:10]}"
    request.addfinalizer(lambda: _cleanup_client(client, username))
    client.headers.update({"X-Forwarded-For": _next_forwarded_for()})

    reg = client.post(
        "/auth/register",
        data={
            "username": username,
            "password": PASSWORD,
            "confirm_password": PASSWORD,
            "acknowledge": "true",
            "csrf_token": _csrf(client),
        },
        follow_redirects=False,
    )
    if reg.status_code != 302:
        pytest.fail(
            f"Auth bootstrap broken: registration returned {reg.status_code} "
            f"(expected 302): {reg.text[:300]}"
        )

    who = client.get("/auth/check")
    if (
        who.status_code != 200
        or who.json().get("authenticated") is not True
        or who.json().get("username") != username
    ):
        pytest.fail(
            "Auth bootstrap broken: registration did not establish the "
            f"expected session: {who.status_code} {who.text[:300]}"
        )

    token = client.get("/auth/csrf-token")
    if token.status_code == 200 and token.json().get("csrf_token"):
        client.headers.update({"X-CSRFToken": token.json()["csrf_token"]})

    return client, username, PASSWORD


# ---------------------------------------------------------------------------
# Direct-DB seeding helpers (the caller's own encrypted database)
# ---------------------------------------------------------------------------


def _session(username, password):
    from local_deep_research.database.session_context import get_user_db_session

    return get_user_db_session(username, password)


def _seed_document(
    username, password, *, source_type_name, title, storage_mode="database"
):
    """Insert one real ``Document`` row and return its id.

    ``source_type_name`` selects a ``SourceType`` seeded by
    ``initialize_library_for_user`` on register/login: "note" produces a
    document ``_is_note_document`` recognises, "user_upload" an ordinary one.
    """
    from local_deep_research.database.models.library import Document, SourceType

    doc_id = str(uuid.uuid4())
    with _session(username, password) as session:
        source_type = (
            session.query(SourceType).filter_by(name=source_type_name).first()
        )
        assert source_type is not None, (
            f"{source_type_name!r} SourceType should have been seeded by "
            "initialize_library_for_user on register/login"
        )
        session.add(
            Document(
                id=doc_id,
                source_type_id=source_type.id,
                document_hash=uuid.uuid4().hex,
                filename=f"{source_type_name}-fixture.md",
                file_size=11,
                file_type="md",
                text_content="hello world",
                title=title,
                storage_mode=storage_mode,
            )
        )
        session.commit()
    return doc_id


def _seed_collection(username, password, *, collection_type, name=None):
    """Insert one real ``Collection`` row and return its id."""
    from local_deep_research.database.models.library import Collection

    collection_id = str(uuid.uuid4())
    with _session(username, password) as session:
        session.add(
            Collection(
                id=collection_id,
                name=name or f"{collection_type}-{uuid.uuid4().hex[:6]}",
                collection_type=collection_type,
            )
        )
        session.commit()
    return collection_id


def _link(username, password, document_id, collection_id):
    from local_deep_research.database.models.library import DocumentCollection

    with _session(username, password) as session:
        session.add(
            DocumentCollection(
                document_id=document_id, collection_id=collection_id
            )
        )
        session.commit()


def _document_row(username, password, document_id):
    """Return ``(exists, storage_mode, file_path)`` for a document."""
    from local_deep_research.database.models.library import Document

    with _session(username, password) as session:
        doc = session.get(Document, document_id)
        if doc is None:
            return (False, None, None)
        return (True, doc.storage_mode, doc.file_path)


def _link_exists(username, password, document_id, collection_id):
    from local_deep_research.database.models.library import DocumentCollection

    with _session(username, password) as session:
        return (
            session.query(DocumentCollection)
            .filter_by(document_id=document_id, collection_id=collection_id)
            .first()
            is not None
        )


def _collection_exists(username, password, collection_id):
    from local_deep_research.database.models.library import Collection

    with _session(username, password) as session:
        return session.get(Collection, collection_id) is not None


def _seed_research(username, password, *, status="completed"):
    """Insert one ``ResearchHistory`` row and return its id."""
    from local_deep_research.database.models import ResearchHistory

    research_id = str(uuid.uuid4())
    with _session(username, password) as session:
        session.add(
            ResearchHistory(
                id=research_id,
                query="does the sky exist",
                mode="quick_summary",
                status=status,
                created_at=datetime.now(timezone.utc).isoformat(),
                title="fixture research",
            )
        )
        session.commit()
    return research_id


def _link_research_to_note(username, password, note_id, research_id):
    from local_deep_research.database.models import NoteResearch

    with _session(username, password) as session:
        session.add(NoteResearch(document_id=note_id, research_id=research_id))
        session.commit()


# ---------------------------------------------------------------------------
# HTTP helpers
# ---------------------------------------------------------------------------


@contextmanager
def _second_authed_client(prefix="lna2"):
    """A second, independent registered and auto-authenticated user.

    Same registration bootstrap as the ``authed`` fixture, callable a second
    time from inside a single test that already holds an ``authed`` client
    (which -- via ``monkeypatch``/``temp_data_dir`` -- has already pointed
    ``LDR_DATA_DIR`` at the shared per-test data directory both users need to
    land in). Used only for the cross-user (ownership) case; every other test
    in this file needs exactly one user and uses ``authed``.
    """
    from local_deep_research.web.fastapi_app import app

    client = TestClient(app, raise_server_exceptions=False)
    username = f"{prefix}_{uuid.uuid4().hex[:10]}"
    try:
        client.headers.update({"X-Forwarded-For": _next_forwarded_for()})
        reg = client.post(
            "/auth/register",
            data={
                "username": username,
                "password": PASSWORD,
                "confirm_password": PASSWORD,
                "acknowledge": "true",
                "csrf_token": _csrf(client),
            },
            follow_redirects=False,
        )
        if reg.status_code != 302:
            pytest.fail(
                "Auth bootstrap broken for second user: registration returned "
                f"{reg.status_code} (expected 302): {reg.text[:300]}"
            )

        who = client.get("/auth/check")
        if (
            who.status_code != 200
            or who.json().get("authenticated") is not True
            or who.json().get("username") != username
        ):
            pytest.fail(
                "Auth bootstrap broken for second user: registration did not "
                f"establish the expected session: {who.status_code} "
                f"{who.text[:300]}"
            )

        token = client.get("/auth/csrf-token")
        if token.status_code == 200 and token.json().get("csrf_token"):
            client.headers.update({"X-CSRFToken": token.json()["csrf_token"]})

        yield client, username, PASSWORD
    finally:
        _cleanup_client(client, username)


def _create_note(client, *, title=None, content="hello world"):
    resp = client.post(
        "/notes/api/notes",
        json={
            "title": title or f"Note {uuid.uuid4().hex[:8]}",
            "content": content,
        },
    )
    assert resp.status_code == 201, f"note create failed: {resp.text[:400]}"
    return resp.json()["id"]


def _create_collection_via_api(client, *, collection_type="user_collection"):
    resp = client.post(
        "/library/api/collections",
        json={
            "name": f"coll-{uuid.uuid4().hex[:8]}",
            "type": collection_type,
        },
    )
    assert resp.status_code == 200, resp.text
    return resp.json()["collection"]["id"]


def _sse_events(text):
    """Parse an SSE body into the list of decoded ``data:`` payloads."""
    events = []
    for line in text.splitlines():
        if line.startswith("data: "):
            events.append(json.loads(line[len("data: ") :]))
    return events


def _no_db_password():
    """Context manager making ``get_authenticated_user_password`` fail.

    Patches the password STORE rather than the resolver, so the real
    ``library.get_authenticated_user_password`` executes and its real
    ``AuthenticationRequiredError`` is what the routes have to cope with.
    ``require_auth`` does not consult the store (it checks the session plus
    ``db_manager.is_user_connected``), so the request still reaches the handler
    -- which is precisely the state these guards exist for: an authenticated
    session whose DB password is no longer resolvable.
    """
    from local_deep_research.database.session_passwords import (
        session_password_store,
    )

    class _Both:
        def __enter__(self):
            self._a = patch.object(
                session_password_store,
                "get_session_password",
                return_value=None,
            )
            self._b = patch.object(
                session_password_store,
                "get_any_session_password",
                return_value=None,
            )
            self._a.start()
            self._b.start()
            return self

        def __exit__(self, *exc):
            self._b.stop()
            self._a.stop()
            return False

    return _Both()


# ===========================================================================
# COVERAGE AREA 1 -- note protection on bulk delete paths
#
# The single-document routes map the service's refusal to a 403; the bulk
# routes do NOT (they return 200 with a per-item failure), so the route layer
# offers these paths no protection at all. Only the service guard
# (`_is_note_document`) stands between the library UI's "select all -> delete"
# and mass note damage. `DELETE /library/api/documents/bulk` is covered in
# test_library_rag_security_fastapi.py; these are its two mutating siblings.
# ===========================================================================


class TestBulkBlobDeleteNoteProtection:
    """``DELETE /library/api/documents/blobs`` loops ``delete_blob_only``.

    Without the guard a note's ``storage_mode`` / ``file_path`` are overwritten
    with the blob-deleted sentinel, which is how the library serves and
    re-indexes it.
    """

    def test_bulk_blob_delete_refuses_a_note_and_leaves_it_untouched(
        self, authed
    ):
        client, username, password = authed
        note_id = _seed_document(
            username,
            password,
            source_type_name="note",
            title="Protected note",
        )

        resp = client.request(
            "DELETE",
            "/library/api/documents/blobs",
            json={"document_ids": [note_id]},
        )

        assert resp.status_code == 200, resp.text
        body = resp.json()
        assert body["deleted"] == 0, (
            f"the note's blob must not be deleted: {body!r}"
        )
        # "skipped" is the no-stored-PDF bucket; a refused note is a FAILURE,
        # so a guard that degraded into the skip branch would fail here.
        assert body["skipped"] == 0, body
        assert body["failed"] == 1, body
        assert body["errors"][0]["document_id"] == note_id, body
        assert "notes api" in body["errors"][0]["error"].lower(), (
            f"the refusal must route the caller to the notes API: {body!r}"
        )

        exists, storage_mode, file_path = _document_row(
            username, password, note_id
        )
        assert exists, "the note row must survive"
        assert storage_mode == "database", (
            "the note's storage_mode must not be rewritten to the "
            f"blob-deleted sentinel: {storage_mode!r}"
        )
        assert file_path is None, (
            f"the note's file_path must not be stamped: {file_path!r}"
        )

    def test_bulk_blob_delete_still_works_on_an_ordinary_document(self, authed):
        """Positive control (non-vacuity): same route, same request shape, a
        non-note document -- the blob really is deleted and the row really is
        rewritten. Proves the refusal above is the note guard firing rather
        than a broken harness, an empty library, or a blanket refusal.
        """
        client, username, password = authed
        doc_id = _seed_document(
            username,
            password,
            source_type_name="user_upload",
            title="Ordinary document",
        )

        resp = client.request(
            "DELETE",
            "/library/api/documents/blobs",
            json={"document_ids": [doc_id]},
        )

        assert resp.status_code == 200, resp.text
        body = resp.json()
        assert body["deleted"] == 1, body
        assert body["failed"] == 0, body

        exists, storage_mode, file_path = _document_row(
            username, password, doc_id
        )
        assert exists, "blob-only delete must keep the document row"
        assert storage_mode == "none", (
            f"an ordinary document's blob delete must land: {storage_mode!r}"
        )
        assert file_path is not None, (
            "an ordinary document must get the blob-deleted sentinel path"
        )

    def test_bulk_blob_delete_of_a_mixed_batch_protects_only_the_note(
        self, authed
    ):
        """The amplification shape that matters: one "select all" request
        carrying both kinds. The ordinary document must be processed AND the
        note must be refused, in the same response.
        """
        client, username, password = authed
        note_id = _seed_document(
            username, password, source_type_name="note", title="Note in batch"
        )
        doc_id = _seed_document(
            username,
            password,
            source_type_name="user_upload",
            title="Document in batch",
        )

        resp = client.request(
            "DELETE",
            "/library/api/documents/blobs",
            json={"document_ids": [note_id, doc_id]},
        )

        assert resp.status_code == 200, resp.text
        body = resp.json()
        assert body["total"] == 2, body
        assert body["deleted"] == 1, f"only the ordinary document: {body!r}"
        assert body["failed"] == 1, f"only the note refused: {body!r}"
        assert [e["document_id"] for e in body["errors"]] == [note_id], body

        assert _document_row(username, password, note_id)[1] == "database"
        assert _document_row(username, password, doc_id)[1] == "none"


class TestBulkCollectionRemovalNoteProtection:
    """``DELETE /library/api/collection/{cid}/documents/bulk`` loops
    ``remove_from_collection``.

    The Notes collection is every note's permanent home
    (``_get_or_create_notes_collection``) and nothing re-links a note after an
    unlink, so removal there silently drops the note out of semantic search,
    the collection view and the auto-reindex worker while ``list_notes`` still
    shows it.
    """

    def test_bulk_removal_refuses_a_note_from_its_notes_collection(
        self, authed
    ):
        client, username, password = authed
        notes_collection = _seed_collection(
            username, password, collection_type="notes", name="Notes"
        )
        note_id = _seed_document(
            username, password, source_type_name="note", title="Homed note"
        )
        _link(username, password, note_id, notes_collection)

        resp = client.request(
            "DELETE",
            f"/library/api/collection/{notes_collection}/documents/bulk",
            json={"document_ids": [note_id]},
        )

        assert resp.status_code == 200, resp.text
        body = resp.json()
        assert body["unlinked"] == 0, body
        assert body["deleted"] == 0, body
        assert body["failed"] == 1, body
        assert "permanent home" in body["errors"][0]["error"].lower(), body

        assert _link_exists(username, password, note_id, notes_collection), (
            "the note must still be linked to its notes collection"
        )
        assert _document_row(username, password, note_id)[0], (
            "the note document itself must survive"
        )

    def test_bulk_removal_still_unlinks_an_ordinary_document_from_a_notes_collection(
        self, authed
    ):
        """Positive control #1 (discriminates on the DOCUMENT half of the
        guard): the guard is ``notes collection AND note document``. A plain
        document sitting in a notes collection is not protected.
        """
        client, username, password = authed
        notes_collection = _seed_collection(
            username, password, collection_type="notes", name="Notes"
        )
        doc_id = _seed_document(
            username,
            password,
            source_type_name="user_upload",
            title="Ordinary in notes collection",
        )
        _link(username, password, doc_id, notes_collection)

        resp = client.request(
            "DELETE",
            f"/library/api/collection/{notes_collection}/documents/bulk",
            json={"document_ids": [doc_id]},
        )

        assert resp.status_code == 200, resp.text
        body = resp.json()
        assert body["failed"] == 0, body
        assert body["unlinked"] == 1, body
        assert not _link_exists(username, password, doc_id, notes_collection)

    def test_bulk_removal_still_unlinks_a_note_from_an_ordinary_collection(
        self, authed
    ):
        """Positive control #2 (discriminates on the COLLECTION half): a note
        may be removed from an ordinary collection -- only its notes-collection
        home is protected -- and the note itself survives that removal because
        it is still linked to its home.
        """
        client, username, password = authed
        notes_collection = _seed_collection(
            username, password, collection_type="notes", name="Notes"
        )
        user_collection = _seed_collection(
            username, password, collection_type="user_collection"
        )
        note_id = _seed_document(
            username, password, source_type_name="note", title="Shared note"
        )
        _link(username, password, note_id, notes_collection)
        _link(username, password, note_id, user_collection)

        resp = client.request(
            "DELETE",
            f"/library/api/collection/{user_collection}/documents/bulk",
            json={"document_ids": [note_id]},
        )

        assert resp.status_code == 200, resp.text
        body = resp.json()
        assert body["failed"] == 0, body
        assert body["unlinked"] == 1, body
        assert not _link_exists(username, password, note_id, user_collection)
        assert _link_exists(username, password, note_id, notes_collection), (
            "removing a note from an ordinary collection must not touch its home"
        )
        assert _document_row(username, password, note_id)[0]


class TestProtectedCollectionFlagSerialization:
    """``rag.py::_is_protected_collection`` reaches the browser as
    ``collection.is_protected`` on
    ``GET /library/api/collections/{id}/documents``.

    The UI hides destructive affordances on the strength of that flag; the
    deletion service's refusal is covered elsewhere, this serialization is not.
    (Audit note: the register calls the field "protected"; the shipped spelling
    is ``is_protected`` -- these tests pin the shipped one.)
    """

    @pytest.mark.parametrize(
        "collection_type", ["notes", "research_history", "default_library"]
    )
    def test_system_collection_types_are_flagged_protected(
        self, authed, collection_type
    ):
        client, username, password = authed
        collection_id = _seed_collection(
            username, password, collection_type=collection_type
        )

        resp = client.get(f"/library/api/collections/{collection_id}/documents")

        assert resp.status_code == 200, resp.text
        collection = resp.json()["collection"]
        # Positive control on the payload itself: the response really is about
        # THIS collection, so `is_protected` cannot be passing by accident on
        # an error body or a different row.
        assert collection["id"] == collection_id
        assert collection["collection_type"] == collection_type
        assert collection["is_protected"] is True, (
            f"{collection_type!r} must serialize as protected: {collection!r}"
        )

    @pytest.mark.parametrize(
        "collection_type", ["user_collection", "user_uploads", "zotero"]
    )
    def test_user_collection_types_are_not_flagged_protected(
        self, authed, collection_type
    ):
        """Non-vacuity for the test above: if ``is_protected`` were hardcoded
        True (or the key merely present), this would fail. ``zotero`` is
        included deliberately -- it is NOT in ``PROTECTED_COLLECTION_TYPES``
        (users create and remove Zotero-synced collections), so a "protect
        every system-ish type" regression is caught here.
        """
        client, username, password = authed
        collection_id = _seed_collection(
            username, password, collection_type=collection_type
        )

        resp = client.get(f"/library/api/collections/{collection_id}/documents")

        assert resp.status_code == 200, resp.text
        collection = resp.json()["collection"]
        assert collection["id"] == collection_id
        assert collection["collection_type"] == collection_type
        assert collection["is_protected"] is False, (
            f"{collection_type!r} must NOT serialize as protected: "
            f"{collection!r}"
        )


# ===========================================================================
# COVERAGE AREA 2 -- library route validation and authorization
# ===========================================================================


class TestOpenFolderIsHardDisabled:
    """``POST /library/api/open-folder`` launches a file manager ON THE SERVER
    HOST. It is disabled unconditionally.

    The branch's only coverage is the generic mutating-route smoke sweep, which
    asserts ``< 500`` -- a re-enabled handler returning 200 passes that sweep.
    """

    @pytest.mark.parametrize(
        "body",
        [
            {},
            {"document_id": "some-document"},
            {"path": "/etc"},
            {"path": "../../.."},
        ],
        ids=["empty", "document_id", "absolute-path", "traversal-path"],
    )
    def test_open_folder_always_403s(self, authed, body):
        client, _, _ = authed

        resp = client.post("/library/api/open-folder", json=body)

        assert resp.status_code == 403, (
            "open-folder must hard-403 for an authenticated caller; a 200 "
            f"means the server-side file manager is reachable: "
            f"{resp.status_code} {resp.text[:300]}"
        )
        payload = resp.json()
        assert payload["status"] == "error", payload
        assert "disabled" in payload["message"].lower(), payload
        assert "desktop" in payload["message"].lower(), payload

    def test_the_403_is_not_a_blanket_refusal_of_this_client(self, authed):
        """Non-vacuity: the same authenticated, CSRF-carrying client can drive
        a sibling library mutation successfully, so the 403 above is the
        open-folder guard and not a dead session or a missing token.
        """
        client, _, _ = authed

        ok = client.post(
            "/library/api/collections",
            json={
                "name": f"probe-{uuid.uuid4().hex[:6]}",
                "type": "user_collection",
            },
        )

        assert ok.status_code == 200, (
            f"the harness itself must be able to mutate: {ok.text[:300]}"
        )


class TestUploadExtensionAllowlistAtTheRoute:
    """``POST /library/api/collections/{id}/upload`` must reject a
    non-allowlisted extension (``is_extension_supported``).

    The per-file size cap and the per-request file count cap are already
    covered by ``tests/web/routers/test_rag_upload_limits_source_of_truth.py``;
    the extension allowlist is covered only at the helper
    (``tests/document_loaders/test_loader_registry.py``), never at the route
    that actually stores the bytes.
    """

    @staticmethod
    def _upload(client, collection_id, filename, content=b"payload bytes"):
        return client.post(
            f"/library/api/collections/{collection_id}/upload",
            files={"files": (filename, content, "application/octet-stream")},
            data={"storage_mode": "database"},
        )

    def test_a_supported_extension_is_accepted(self, authed):
        """Positive control, asserted first: with an empty collection every
        "the file was not stored" assertion below would pass trivially.
        """
        client, _, _ = authed
        collection_id = _create_collection_via_api(client)

        resp = self._upload(
            client, collection_id, "notes.txt", b"real extractable text\n"
        )

        assert resp.status_code == 200, resp.text
        body = resp.json()
        assert body["success"] is True, body
        assert body["summary"]["successful"] == 1, body
        assert body["summary"]["failed"] == 0, body
        assert body["errors"] == [], body

    @pytest.mark.parametrize(
        "filename",
        [
            "payload.exe",
            "payload.sh",
            "payload.dll",
            "payload.so",
            "archive.zip",
            # Double extension: only the LAST suffix decides, so a name that
            # "looks like" an allowlisted type must still be rejected.
            "payload.txt.exe",
        ],
    )
    def test_an_unsupported_extension_is_rejected_and_nothing_is_stored(
        self, authed, filename
    ):
        client, username, password = authed
        collection_id = _create_collection_via_api(client)

        resp = self._upload(client, collection_id, filename)

        assert resp.status_code == 200, resp.text
        body = resp.json()
        assert body["summary"]["successful"] == 0, (
            f"{filename!r} must not be stored: {body!r}"
        )
        assert body["summary"]["failed"] == 1, body
        assert body["uploaded"] == [], body
        assert len(body["errors"]) == 1, body
        assert body["errors"][0]["error"].startswith("Unsupported format:"), (
            f"the rejection must be the extension allowlist, not extraction "
            f"failure or a sanitiser: {body!r}"
        )

        from local_deep_research.database.models.library import Document

        with _session(username, password) as session:
            assert (
                session.query(Document).filter_by(filename=filename).first()
                is None
            ), f"no Document row may exist for {filename!r}"


class TestGetAuthenticatedUserPasswordFailsClosed:
    """``library.py::get_authenticated_user_password`` must RAISE
    ``AuthenticationRequiredError`` when no DB password can be resolved.

    Every download route calls it before constructing a ``DownloadService``. A
    regression that returned ``None`` (or ``""``) instead of raising would hand
    that falsy value to the SQLCipher key derivation. At the review snapshot,
    other tests only *patched* this function; this class now pins its contract.
    """

    def test_it_returns_the_real_password_when_a_session_has_one(self, authed):
        """Positive control, asserted first -- otherwise "it raised" would be
        satisfied by a function that always raises.
        """
        from local_deep_research.web.routers.library import (
            get_authenticated_user_password,
        )

        _, username, password = authed

        assert get_authenticated_user_password(username) == password

    def test_it_raises_when_no_session_password_exists(self, authed):
        from local_deep_research.web.exceptions import (
            AuthenticationRequiredError,
        )
        from local_deep_research.web.routers.library import (
            get_authenticated_user_password,
        )

        _, username, _ = authed

        with _no_db_password():
            with pytest.raises(AuthenticationRequiredError) as excinfo:
                get_authenticated_user_password(username)

        assert "Authentication required" in str(excinfo.value)

    def test_it_raises_rather_than_falling_back_when_the_store_errors(
        self, authed
    ):
        """The session-scoped lookup is wrapped in ``except Exception``. That
        swallow must fall through to the "no password" RAISE, never to a
        silent ``None`` return.
        """
        from local_deep_research.database.session_passwords import (
            session_password_store,
        )
        from local_deep_research.web.exceptions import (
            AuthenticationRequiredError,
        )
        from local_deep_research.web.routers.library import (
            get_authenticated_user_password,
        )

        _, username, _ = authed

        with (
            patch.object(
                session_password_store,
                "get_session_password",
                side_effect=RuntimeError("store down"),
            ),
            patch.object(
                session_password_store,
                "get_any_session_password",
                return_value=None,
            ),
        ):
            with pytest.raises(AuthenticationRequiredError):
                get_authenticated_user_password(username, session_id="sid")


class TestSseStreamsFailClosedOnAuthFailure:
    """``download_all_text`` and ``download_bulk`` build their SSE generator
    AFTER the response has already been committed as 200 ``text/event-stream``,
    so an unresolvable DB password cannot be signalled with a status code. Both
    must emit a terminal error EVENT and stop -- never fall through and
    construct a ``DownloadService`` with a missing password.

    The branch's ``test_sse_response_headers.py`` asserts the headers only; it
    patches the password resolver rather than exercising its failure.
    """

    ALL_TEXT = "/library/api/download-all-text"
    BULK = "/library/api/download-bulk"

    def test_download_all_text_streams_normally_when_authenticated(
        self, authed
    ):
        """Positive control, asserted first: an authenticated caller gets a
        real stream that never mentions authentication.
        """
        client, _, _ = authed

        resp = client.post(self.ALL_TEXT)

        assert resp.status_code == 200, resp.text
        assert resp.headers["content-type"].startswith("text/event-stream")
        events = _sse_events(resp.text)
        assert events, f"the stream must carry events: {resp.text[:300]!r}"
        assert events[-1]["complete"] is True, events
        assert "error" not in events[-1], events
        assert "Authentication required" not in resp.text

    def test_download_all_text_emits_an_error_event_and_stops(self, authed):
        client, _, _ = authed

        with _no_db_password():
            resp = client.post(self.ALL_TEXT)

        assert resp.status_code == 200, resp.text
        events = _sse_events(resp.text)
        assert len(events) == 1, (
            f"the stream must stop after the refusal, not keep going: {events!r}"
        )
        assert events[0]["error"] == "Authentication required", events
        assert events[0]["complete"] is True, events
        assert events[0]["total"] == 0, events

    def test_download_bulk_streams_normally_when_authenticated(self, authed):
        """Positive control for the bulk stream. An unknown research id queues
        nothing, so the stream reports "no new papers" -- the point is that it
        gets PAST the credential gate and produces the normal two-event shape.
        """
        client, _, _ = authed

        resp = client.post(
            self.BULK, json={"research_ids": [str(uuid.uuid4())], "mode": "pdf"}
        )

        assert resp.status_code == 200, resp.text
        assert resp.headers["content-type"].startswith("text/event-stream")
        events = _sse_events(resp.text)
        assert len(events) == 2, events
        assert events[0] == {"progress": 0, "current": 0, "total": 0}, events
        assert events[1]["complete"] is True, events
        assert "Authentication required" not in resp.text

    def test_download_bulk_emits_an_error_event_and_stops(self, authed):
        client, _, _ = authed

        with _no_db_password():
            resp = client.post(
                self.BULK,
                json={"research_ids": [str(uuid.uuid4())], "mode": "pdf"},
            )

        assert resp.status_code == 200, resp.text
        events = _sse_events(resp.text)
        assert len(events) == 1, (
            f"the bulk stream must stop at the refusal: {events!r}"
        )
        assert events[0]["error"] == "Authentication required", events
        assert events[0]["complete"] is True, events
        assert events[0]["total"] == 0, events


# ===========================================================================
# COVERAGE AREA 3 -- notes router
# ===========================================================================


def _versions(client, note_id):
    resp = client.get(f"/notes/api/notes/{note_id}/versions")
    assert resp.status_code == 200, resp.text
    return resp.json()["versions"]


class TestNoteVersionReadRoutesAreScopedToTheirOwnNote:
    """``GET .../notes/{note_id}/versions/{version_id}`` and
    ``.../versions/semantic-diff`` filter by ``document_id=note_id``.

    The branch covers the analogous guard for RESTORE only
    (``test_note_integration.py::
    test_restore_with_bookends_rejects_version_from_another_note_same_user``).
    At the review snapshot, the READ paths -- which return the version's full
    content -- had no direct test. The cases below now pin them. Cross-note
    here, not cross-user: both notes belong to the same caller, so per-user DB
    isolation cannot mask a missing filter.
    """

    def test_reading_another_notes_version_is_404(self, authed):
        client, _, _ = authed
        note_a = _create_note(client, content="alpha secret content")
        note_b = _create_note(client, content="beta content")
        version_a = _versions(client, note_a)[0]["id"]

        # Positive control FIRST: the version is readable through its OWN note.
        own = client.get(f"/notes/api/notes/{note_a}/versions/{version_a}")
        assert own.status_code == 200, own.text
        assert own.json()["version"]["id"] == version_a
        assert "alpha secret content" in own.json()["version"]["content"]

        cross = client.get(f"/notes/api/notes/{note_b}/versions/{version_a}")

        assert cross.status_code == 404, (
            "a version must not be readable through a sibling note's id: "
            f"{cross.status_code} {cross.text[:300]}"
        )
        assert "alpha secret content" not in cross.text

    def test_semantic_diff_rejects_a_version_from_another_note(
        self, authed, monkeypatch
    ):
        from local_deep_research.web.routers import notes as notes_router

        class _FakeAI:
            def __init__(self, *args, **kwargs):
                pass

            async def semantic_diff(self, a, b):
                return {"summary": "fake diff", "a": a, "b": b}

        monkeypatch.setattr(notes_router, "NoteAIService", _FakeAI)

        client, _, _ = authed
        note_a = _create_note(client, content="alpha secret content")
        note_b = _create_note(client, content="beta content")
        version_a = _versions(client, note_a)[0]["id"]

        # Positive control FIRST.
        own = client.get(
            f"/notes/api/notes/{note_a}/versions/semantic-diff",
            params={"version1": version_a, "version2": "current"},
        )
        assert own.status_code == 200, own.text
        assert own.json()["diff"]["summary"] == "fake diff"

        cross = client.get(
            f"/notes/api/notes/{note_b}/versions/semantic-diff",
            params={"version1": version_a, "version2": "current"},
        )

        assert cross.status_code == 404, (
            "semantic-diff must not accept a version belonging to another "
            f"note: {cross.status_code} {cross.text[:300]}"
        )
        assert "alpha secret content" not in cross.text

    def test_semantic_diff_rejects_a_foreign_version2(
        self, authed, monkeypatch
    ):
        """``version2`` goes through the same ``document_id=note_id`` filter as
        ``version1``; a filter dropped from only the second lookup would still
        pass the test above.
        """
        from local_deep_research.web.routers import notes as notes_router

        class _FakeAI:
            def __init__(self, *args, **kwargs):
                pass

            async def semantic_diff(self, a, b):
                return {"summary": "fake diff"}

        monkeypatch.setattr(notes_router, "NoteAIService", _FakeAI)

        client, _, _ = authed
        note_a = _create_note(client, content="alpha secret content")
        note_b = _create_note(client, content="beta content")
        version_a = _versions(client, note_a)[0]["id"]
        version_b = _versions(client, note_b)[0]["id"]

        # Positive control FIRST: two versions of the SAME note diff fine.
        own = client.get(
            f"/notes/api/notes/{note_b}/versions/semantic-diff",
            params={"version1": version_b, "version2": "current"},
        )
        assert own.status_code == 200, own.text

        cross = client.get(
            f"/notes/api/notes/{note_b}/versions/semantic-diff",
            params={"version1": version_b, "version2": version_a},
        )

        assert cross.status_code == 404, (
            f"version2 must be scoped too: {cross.status_code} {cross.text[:300]}"
        )
        assert "alpha secret content" not in cross.text

    @pytest.mark.parametrize("route", ["versions", "version", "semantic-diff"])
    def test_version_routes_404_for_a_non_note_document(
        self, authed, route, monkeypatch
    ):
        """``/versions*`` must not become a generic reader for ANY library
        document's history. Each route re-checks ``_is_note`` on the parent, so
        the notes API cannot be pointed at an arbitrary uploaded document.
        """
        from local_deep_research.web.routers import notes as notes_router

        class _FakeAI:
            def __init__(self, *args, **kwargs):
                pass

            async def semantic_diff(self, a, b):
                return {"summary": "fake diff"}

        monkeypatch.setattr(notes_router, "NoteAIService", _FakeAI)

        client, username, password = authed
        note_id = _create_note(client, content="a real note")
        version_id = _versions(client, note_id)[0]["id"]
        document_id = _seed_document(
            username,
            password,
            source_type_name="user_upload",
            title="Not a note",
        )

        if route == "versions":
            good = client.get(f"/notes/api/notes/{note_id}/versions")
            bad = client.get(f"/notes/api/notes/{document_id}/versions")
        elif route == "version":
            good = client.get(
                f"/notes/api/notes/{note_id}/versions/{version_id}"
            )
            bad = client.get(
                f"/notes/api/notes/{document_id}/versions/{version_id}"
            )
        else:
            # Both version params are required; supplying only one 400s at the
            # arity check BEFORE the _is_note guard, which would make this
            # test pass for the wrong reason.
            params = {"version1": version_id, "version2": "current"}
            good = client.get(
                f"/notes/api/notes/{note_id}/versions/semantic-diff",
                params=params,
            )
            bad = client.get(
                f"/notes/api/notes/{document_id}/versions/semantic-diff",
                params=params,
            )

        # Positive control: the same shape against a real note is NOT a 404.
        assert good.status_code != 404, (
            f"harness check failed for {route!r}: {good.status_code} "
            f"{good.text[:300]}"
        )
        assert bad.status_code == 404, (
            f"{route!r} must 404 for a non-note document id: "
            f"{bad.status_code} {bad.text[:300]}"
        )
        assert bad.json()["error"] == "Note not found", bad.text


class TestFactCheckGradeCrossResourceAuthz:
    """``POST /notes/api/notes/{note_id}/fact-check/{research_id}/grade`` must
    refuse a research run that is not linked to the note in the path.

    Without it a client can grade an arbitrary completed research against an
    arbitrary note and stamp a foreign ``note_id`` into that research's
    ``research_meta['fact_check']`` -- an IDOR-shaped cross-resource write.
    """

    @staticmethod
    def _grade(client, note_id, research_id, claims=None):
        return client.post(
            f"/notes/api/notes/{note_id}/fact-check/{research_id}/grade",
            json={"claims": claims or ["the sky is blue"]},
        )

    def test_grading_a_research_linked_to_another_note_is_refused(self, authed):
        client, username, password = authed
        note_owner = _create_note(client, content="linked note")
        note_other = _create_note(client, content="unlinked note")
        research_id = _seed_research(username, password, status="completed")
        _link_research_to_note(username, password, note_owner, research_id)

        # Positive control FIRST: the LINKED note gets past the authz check.
        # It stops later, at the report gate (502) -- proving the 404 below is
        # the link check and not "this research is unusable".
        linked = self._grade(client, note_owner, research_id)
        assert linked.status_code == 502, (
            "the linked note must pass the authz check and reach the report "
            f"gate: {linked.status_code} {linked.text[:300]}"
        )

        cross = self._grade(client, note_other, research_id)

        assert cross.status_code == 404, (
            f"cross-note grading must be refused: {cross.status_code} "
            f"{cross.text[:300]}"
        )
        assert cross.json()["error"] == "Research is not linked to this note"

    def test_an_unknown_research_id_reports_not_found_not_not_linked(
        self, authed
    ):
        """Discriminator for the test above: the two 404s carry DIFFERENT
        errors, so a handler that collapsed "no such research" and "not linked"
        into one branch could not pass both.
        """
        client, _, _ = authed
        note_id = _create_note(client)

        resp = self._grade(client, note_id, str(uuid.uuid4()))

        assert resp.status_code == 404, resp.text
        assert resp.json()["error"] == "Research not found", resp.text

    def test_an_incomplete_linked_research_is_409_not_graded(self, authed):
        """The status ladder below the authz check: a linked-but-unfinished
        research must not be graded either.
        """
        client, username, password = authed
        note_id = _create_note(client)
        research_id = _seed_research(username, password, status="in_progress")
        _link_research_to_note(username, password, note_id, research_id)

        resp = self._grade(client, note_id, research_id)

        assert resp.status_code == 409, resp.text
        assert resp.json()["status"] == "in_progress", resp.text


class TestFactCheckClaimSanitisation:
    """``claims`` is free client text that gets framed into an LLM prompt and
    fanned out into research runs -- an input-bound AND cost-amplification
    surface. ``notes.py:2319-2325`` strips, truncates to ``MAX_CLAIM_LEN`` and
    caps at ``NoteAIService.MAX_CLAIMS_PER_NOTE``.

    The sanitisation is observed by stubbing ``_grade_note_claims`` -- the seam
    the route hands the sanitised list to -- so the assertion is on what the
    ROUTE produced, not on downstream behaviour.
    """

    @pytest.fixture
    def captured(self, monkeypatch):
        from local_deep_research.web.routers import notes as notes_router

        box = {}

        async def _fake(username, note_id, research_id, claims):
            box["claims"] = claims
            return {"success": True, "verdicts": []}, 200

        monkeypatch.setattr(notes_router, "_grade_note_claims", _fake)
        return box

    @staticmethod
    def _grade(client, note_id, claims):
        return client.post(
            f"/notes/api/notes/{note_id}/fact-check/{uuid.uuid4()}/grade",
            json={"claims": claims},
        )

    def test_ordinary_claims_pass_through_unchanged(self, authed, captured):
        """Positive control, asserted first: without it every "the payload was
        trimmed" assertion below would also pass on a route that dropped
        claims entirely.
        """
        client, _, _ = authed
        note_id = _create_note(client)

        resp = self._grade(client, note_id, ["one claim", "  two claim  "])

        assert resp.status_code == 200, resp.text
        assert captured["claims"] == ["one claim", "two claim"], captured

    def test_each_claim_is_truncated_to_max_claim_len(self, authed, captured):
        from local_deep_research.web.routers.notes import MAX_CLAIM_LEN

        client, _, _ = authed
        note_id = _create_note(client)

        resp = self._grade(client, note_id, ["A" * (MAX_CLAIM_LEN * 3)])

        assert resp.status_code == 200, resp.text
        assert len(captured["claims"]) == 1
        assert len(captured["claims"][0]) == MAX_CLAIM_LEN, (
            f"claim must be truncated to {MAX_CLAIM_LEN}: "
            f"{len(captured['claims'][0])}"
        )

    def test_the_claim_count_is_capped(self, authed, captured):
        from local_deep_research.research_library.notes.services.note_ai_service import (
            NoteAIService,
        )

        client, _, _ = authed
        note_id = _create_note(client)
        cap = NoteAIService.MAX_CLAIMS_PER_NOTE

        resp = self._grade(
            client, note_id, [f"claim {i}" for i in range(cap * 5)]
        )

        assert resp.status_code == 200, resp.text
        assert len(captured["claims"]) == cap, (
            f"claim count must be capped at {cap}: {len(captured['claims'])}"
        )

    @pytest.mark.parametrize(
        "claims",
        [
            "a bare string",
            {"claim": "an object"},
            42,
            None,
        ],
        ids=["string", "object", "int", "null"],
    )
    def test_a_non_list_claims_field_is_400(self, authed, captured, claims):
        client, _, _ = authed
        note_id = _create_note(client)

        resp = self._grade(client, note_id, claims)

        assert resp.status_code == 400, resp.text
        assert resp.json()["error"] == "claims required", resp.text
        assert "claims" not in captured, (
            "a rejected body must never reach the grader"
        )

    @pytest.mark.parametrize(
        "claims",
        [[], ["", "   "], [None, 7, {"a": 1}]],
        ids=["empty", "blank-strings", "non-strings"],
    )
    def test_a_list_with_no_usable_claims_is_400(
        self, authed, captured, claims
    ):
        """Non-string / blank entries are filtered out, and an empty result
        must 400 rather than starting a grade with zero claims.
        """
        client, _, _ = authed
        note_id = _create_note(client)

        resp = self._grade(client, note_id, claims)

        assert resp.status_code == 400, resp.text
        assert resp.json()["error"] == "claims required", resp.text
        assert "claims" not in captured


class TestNotesInputSizeClamps:
    """``_clamp_text_query`` / ``MAX_SEARCH_LEN`` and ``MAX_PASSAGE_LEN``.

    Unbounded free text here flows into ``ILIKE %...%`` over unbounded TEXT
    columns and into embedding calls. At the review snapshot both guards
    survived in ``src/`` without direct tests; this class now pins them.
    """

    def test_a_search_at_the_cap_is_accepted(self, authed):
        """Positive control, asserted first: the 400 below is the length cap,
        not "long queries break the route".
        """
        from local_deep_research.web.routers.notes import MAX_SEARCH_LEN

        client, _, _ = authed

        resp = client.get(
            "/notes/api/notes", params={"search": "q" * MAX_SEARCH_LEN}
        )

        assert resp.status_code == 200, resp.text
        assert resp.json()["success"] is True

    def test_a_search_over_the_cap_is_400(self, authed):
        from local_deep_research.web.routers.notes import MAX_SEARCH_LEN

        client, _, _ = authed

        resp = client.get(
            "/notes/api/notes", params={"search": "q" * (MAX_SEARCH_LEN + 1)}
        )

        assert resp.status_code == 400, (
            f"a search one char over MAX_SEARCH_LEN must 400: "
            f"{resp.status_code} {resp.text[:300]}"
        )
        assert str(MAX_SEARCH_LEN) in resp.json()["error"], resp.text

    def test_semantic_search_query_is_capped_too(self, authed, monkeypatch):
        """The same clamp guards the embedding-backed search, where an
        unbounded query is also an embedding-cost amplifier.
        """
        from local_deep_research.web.routers import notes as notes_router
        from local_deep_research.web.routers.notes import MAX_SEARCH_LEN

        class _FakeAI:
            def __init__(self, *args, **kwargs):
                pass

            def semantic_search(self, query, **kwargs):
                return []

        monkeypatch.setattr(notes_router, "NoteAIService", _FakeAI)

        client, _, _ = authed

        # Positive control FIRST: a query at exactly the cap is served, so the
        # 400 below is the length clamp and not "this route always 400s".
        at_cap = client.get(
            "/notes/api/notes/semantic-search",
            params={"q": "q" * MAX_SEARCH_LEN},
        )
        assert at_cap.status_code == 200, at_cap.text
        assert at_cap.json()["success"] is True

        resp = client.get(
            "/notes/api/notes/semantic-search",
            params={"q": "q" * (MAX_SEARCH_LEN + 1)},
        )

        assert resp.status_code == 400, resp.text
        assert str(MAX_SEARCH_LEN) in resp.json()["error"], resp.text

    def test_similar_passages_truncates_to_max_passage_len(
        self, authed, monkeypatch
    ):
        from local_deep_research.web.routers import notes as notes_router
        from local_deep_research.web.routers.notes import MAX_PASSAGE_LEN

        seen = {}

        class _RecordingAI:
            def __init__(self, *args, **kwargs):
                pass

            def find_similar_passages(self, text, exclude_note_id=None):
                seen["text"] = text
                return []

        monkeypatch.setattr(notes_router, "NoteAIService", _RecordingAI)

        client, _, _ = authed
        note_id = _create_note(client)

        # Positive control FIRST: a short passage reaches the service verbatim,
        # so the recorder is really wired and truncation is not "it never ran".
        short = client.post(
            f"/notes/api/notes/{note_id}/similar-passages",
            json={"text": "  a short selection  "},
        )
        assert short.status_code == 200, short.text
        assert seen["text"] == "a short selection", seen

        resp = client.post(
            f"/notes/api/notes/{note_id}/similar-passages",
            json={"text": "P" * (MAX_PASSAGE_LEN * 3)},
        )

        assert resp.status_code == 200, resp.text
        assert len(seen["text"]) == MAX_PASSAGE_LEN, (
            f"the passage must be truncated to {MAX_PASSAGE_LEN} before it "
            f"reaches the embedding call: {len(seen['text'])}"
        )


class TestResolveLinkInputSizeClamp:
    """``POST /notes/api/notes/resolve-link`` clamps ``link_text`` at
    ``MAX_LINK_TEXT_LEN`` (``notes.py:56``) via ``_clamp_text_query`` before it
    ever reaches ``NoteService.resolve_link``.

    Note the route guard is not the only line of defence -- the service also
    silently truncates at its own ``MAX_LINK_TEXT_LENGTH`` (same value, 500,
    but a separate constant in ``note_service.py``) as a belt-and-braces
    measure. That means the OBSERVABLE difference the route guard buys is the
    status code: with the route guard, an oversized ``link_text`` 400s before
    the service is even called; without it, the request would fall through to
    the service's silent truncate-and-continue and come back 404 (or 200) --
    never 400. The assertions below pin the 400 specifically so a disabled
    route guard cannot hide behind the service's silent fallback.
    """

    def test_link_text_at_the_cap_is_not_rejected_for_its_length(self, authed):
        """Positive control, asserted first: a link_text AT the cap must not
        400. It will still 404 ("Note not found") because nothing matches
        1000 copies of the letter q -- the point is that the failure is NOT
        the length clamp.
        """
        from local_deep_research.web.routers.notes import MAX_LINK_TEXT_LEN

        client, _, _ = authed

        resp = client.post(
            "/notes/api/notes/resolve-link",
            json={"link_text": "q" * MAX_LINK_TEXT_LEN},
        )

        assert resp.status_code != 400, (
            f"a link_text exactly at the cap must not be rejected for its "
            f"length: {resp.status_code} {resp.text[:300]}"
        )

    def test_link_text_over_the_cap_is_400(self, authed):
        from local_deep_research.web.routers.notes import MAX_LINK_TEXT_LEN

        client, _, _ = authed

        resp = client.post(
            "/notes/api/notes/resolve-link",
            json={"link_text": "q" * (MAX_LINK_TEXT_LEN + 1)},
        )

        assert resp.status_code == 400, (
            f"a link_text one char over MAX_LINK_TEXT_LEN must 400: "
            f"{resp.status_code} {resp.text[:300]}"
        )
        assert str(MAX_LINK_TEXT_LEN) in resp.json()["error"], resp.text

    def test_a_real_link_still_resolves_under_the_cap(self, authed):
        """Non-vacuity: the clamp does not interfere with an ordinary
        resolvable link -- proves the 400 above is the length guard, not a
        route that always refuses this endpoint.
        """
        client, _, _ = authed
        title = f"Linkable {uuid.uuid4().hex[:8]}"
        _create_note(client, title=title)

        resp = client.post(
            "/notes/api/notes/resolve-link", json={"link_text": title}
        )

        assert resp.status_code == 200, resp.text
        assert resp.json()["success"] is True


# ===========================================================================
# COVERAGE AREA 3 (continued) -- annotation-delete anchor/ownership check
#
# DELETE .../research/{id}/annotations/{note_id} and
# .../documents/{id}/annotations/{note_id} must refuse unless
# NoteService.has_annotation() finds a NoteReference row anchoring note_id to
# the EXACT resource named in the path (and that lookup runs exclusively
# inside the caller's own per-user encrypted database, which is the ownership
# half). The deleted predecessors were
# test_research_notes_routes.py::test_delete_requires_matching_anchor and
# ::test_delete_document_annotation_requires_anchor -- mocked-service unit
# tests with no FastAPI successor. These exercise the real route, the real
# NoteService and a real per-user database.
# ===========================================================================


class TestAnnotationDeleteAnchorCheck:
    @staticmethod
    def _create_research_annotation(client, research_id, quote="quoted text"):
        resp = client.post(
            f"/notes/api/research/{research_id}/annotations",
            json={"comment": "a comment", "quote": quote},
        )
        assert resp.status_code == 201, resp.text
        return resp.json()["annotation"]["note_id"]

    @staticmethod
    def _create_document_annotation(client, document_id, quote="quoted text"):
        resp = client.post(
            f"/notes/api/documents/{document_id}/annotations",
            json={"comment": "a comment", "quote": quote},
        )
        assert resp.status_code == 201, resp.text
        return resp.json()["annotation"]["note_id"]

    @staticmethod
    def _research_annotation_note_ids(client, research_id):
        resp = client.get(f"/notes/api/research/{research_id}/annotations")
        assert resp.status_code == 200, resp.text
        return {a["note_id"] for a in resp.json()["annotations"]}

    @staticmethod
    def _document_annotation_note_ids(client, document_id):
        resp = client.get(f"/notes/api/documents/{document_id}/annotations")
        assert resp.status_code == 200, resp.text
        return {a["note_id"] for a in resp.json()["annotations"]}

    # -- research target -----------------------------------------------

    def test_deleting_a_research_annotation_through_its_own_id_succeeds(
        self, authed
    ):
        """Positive control, asserted first: otherwise every "the annotation
        must still exist" assertion below would pass on a route that refused
        every delete.
        """
        client, username, password = authed
        research_id = _seed_research(username, password)
        note_id = self._create_research_annotation(client, research_id)
        assert note_id in self._research_annotation_note_ids(
            client, research_id
        )

        resp = client.delete(
            f"/notes/api/research/{research_id}/annotations/{note_id}"
        )

        assert resp.status_code == 200, resp.text
        assert note_id not in self._research_annotation_note_ids(
            client, research_id
        )

    def test_deleting_a_research_annotation_via_a_different_research_id_is_404(
        self, authed
    ):
        client, username, password = authed
        research_a = _seed_research(username, password)
        research_b = _seed_research(username, password)
        note_id = self._create_research_annotation(client, research_a)

        resp = client.delete(
            f"/notes/api/research/{research_b}/annotations/{note_id}"
        )

        assert resp.status_code == 404, (
            "an annotation anchored to research A must not be deletable "
            f"through research B's path: {resp.status_code} "
            f"{resp.text[:300]}"
        )
        assert resp.json()["error"] == "Annotation not found", resp.text
        assert note_id in self._research_annotation_note_ids(
            client, research_a
        ), "the annotation must survive under its real research id"

    def test_deleting_a_plain_note_via_the_research_annotation_route_is_404(
        self, authed
    ):
        """A note with no NoteReference anchor at all (created through the
        ordinary POST /notes/api/notes route) must not be deletable through
        the annotation route -- ``has_annotation`` requires a real anchored
        reference, not merely a note that exists.
        """
        client, username, password = authed
        research_id = _seed_research(username, password)
        note_id = _create_note(client, content="not an annotation")

        resp = client.delete(
            f"/notes/api/research/{research_id}/annotations/{note_id}"
        )

        assert resp.status_code == 404, resp.text
        # the plain note itself must survive
        still_there = client.get(f"/notes/api/notes/{note_id}/versions")
        assert still_there.status_code == 200, (
            "the unrelated note must not have been deleted: "
            f"{still_there.status_code}"
        )

    # -- document target -------------------------------------------------

    def test_deleting_a_document_annotation_through_its_own_id_succeeds(
        self, authed
    ):
        """Positive control for the document twin, asserted first."""
        client, username, password = authed
        document_id = _seed_document(
            username, password, source_type_name="user_upload", title="Doc A"
        )
        note_id = self._create_document_annotation(client, document_id)
        assert note_id in self._document_annotation_note_ids(
            client, document_id
        )

        resp = client.delete(
            f"/notes/api/documents/{document_id}/annotations/{note_id}"
        )

        assert resp.status_code == 200, resp.text
        assert note_id not in self._document_annotation_note_ids(
            client, document_id
        )

    def test_deleting_a_document_annotation_via_a_different_document_id_is_404(
        self, authed
    ):
        client, username, password = authed
        document_a = _seed_document(
            username, password, source_type_name="user_upload", title="Doc A"
        )
        document_b = _seed_document(
            username, password, source_type_name="user_upload", title="Doc B"
        )
        note_id = self._create_document_annotation(client, document_a)

        resp = client.delete(
            f"/notes/api/documents/{document_b}/annotations/{note_id}"
        )

        assert resp.status_code == 404, (
            "an annotation anchored to document A must not be deletable "
            f"through document B's path: {resp.status_code} "
            f"{resp.text[:300]}"
        )
        assert resp.json()["error"] == "Annotation not found", resp.text
        assert note_id in self._document_annotation_note_ids(
            client, document_a
        ), "the annotation must survive under its real document id"

    # -- ownership (cross-user) ------------------------------------------

    def test_a_second_user_cannot_delete_another_users_research_annotation(
        self, authed
    ):
        """Ownership half of S13: ``has_annotation`` runs entirely inside
        ``get_user_db_session(self.username)`` -- the caller's OWN encrypted
        database. Even a second authenticated user who somehow obtained the
        exact ``note_id`` (e.g. it leaked through another channel) cannot
        delete it, because that id simply does not exist in their database.
        """
        client, username, password = authed
        research_id = _seed_research(username, password)
        note_id = self._create_research_annotation(client, research_id)

        with _second_authed_client() as (
            other_client,
            other_username,
            other_password,
        ):
            other_research_id = _seed_research(other_username, other_password)
            resp = other_client.delete(
                f"/notes/api/research/{other_research_id}/annotations/{note_id}"
            )

            assert resp.status_code == 404, (
                "a foreign user's note_id must not be deletable: "
                f"{resp.status_code} {resp.text[:300]}"
            )

        assert note_id in self._research_annotation_note_ids(
            client, research_id
        ), "the victim's annotation must survive the cross-user attempt"


# ===========================================================================
# COVERAGE AREA 4 -- collection type, rename, and protected-delete rules
# ===========================================================================


class TestCreateCollectionTypeAllowlist:
    """``POST /library/api/collections`` allowlists user-creatable types.

    A user-crafted ``type:"notes"`` impostor is undeletable under
    ``PROTECTED_COLLECTION_TYPES`` and can nondeterministically win
    ``_get_or_create_notes_collection``'s unordered ``.first()`` lookup,
    splitting the notes corpus into an attacker-named collection.
    ``tests/api_tests/test_collections_api.py`` covers name validation only.
    """

    @pytest.mark.parametrize(
        "collection_type", ["user_uploads", "user_collection"]
    )
    def test_allowlisted_types_are_accepted_and_persisted(
        self, authed, collection_type
    ):
        """Positive control, asserted first -- otherwise "creation was
        refused" would pass on a route that refused every type.
        """
        client, username, password = authed

        resp = client.post(
            "/library/api/collections",
            json={
                "name": f"ok-{uuid.uuid4().hex[:6]}",
                "type": collection_type,
            },
        )

        assert resp.status_code == 200, resp.text
        collection_id = resp.json()["collection"]["id"]

        from local_deep_research.database.models.library import Collection

        with _session(username, password) as session:
            stored = session.get(Collection, collection_id)
            assert stored is not None
            assert stored.collection_type == collection_type

    @pytest.mark.parametrize(
        "collection_type",
        [
            "notes",
            "default_library",
            "research_history",
            "zotero",
            "anything_else",
            "",
        ],
    )
    def test_non_allowlisted_types_are_refused_with_400(
        self, authed, collection_type
    ):
        client, username, password = authed
        name = f"impostor-{uuid.uuid4().hex[:6]}"

        resp = client.post(
            "/library/api/collections",
            json={"name": name, "type": collection_type},
        )

        assert resp.status_code == 400, (
            f"type={collection_type!r} must be refused: {resp.status_code} "
            f"{resp.text[:300]}"
        )
        body = resp.json()
        assert body["success"] is False
        assert "Invalid collection type" in body["error"], body

        from local_deep_research.database.models.library import Collection

        with _session(username, password) as session:
            assert (
                session.query(Collection).filter_by(name=name).first() is None
            ), "a refused collection must not be written to the database"

    @pytest.mark.parametrize(
        "collection_type",
        [123, ["notes"], {"type": "notes"}, True],
        ids=["int", "list", "dict", "bool"],
    )
    def test_a_non_string_type_is_refused_with_400(
        self, authed, collection_type
    ):
        """An unhashable (or simply wrong-typed) value must be rejected by its
        own branch rather than crashing the ``in`` test or the ``.strip()``.
        """
        client, _, _ = authed

        resp = client.post(
            "/library/api/collections",
            json={
                "name": f"badtype-{uuid.uuid4().hex[:6]}",
                "type": collection_type,
            },
        )

        assert resp.status_code == 400, (
            f"type={collection_type!r} must 400, not 500: {resp.status_code} "
            f"{resp.text[:300]}"
        )
        assert resp.json()["error"] == "Collection type must be a string"

    def test_the_default_type_is_an_allowlisted_one(self, authed):
        """Omitting ``type`` must not open a hole around the allowlist."""
        client, username, password = authed

        resp = client.post(
            "/library/api/collections",
            json={"name": f"default-{uuid.uuid4().hex[:6]}"},
        )

        assert resp.status_code == 200, resp.text

        from local_deep_research.database.models.library import Collection
        from local_deep_research.research_library.deletion.services.collection_deletion import (
            PROTECTED_COLLECTION_TYPES,
        )

        with _session(username, password) as session:
            stored = session.get(Collection, resp.json()["collection"]["id"])
            assert stored.collection_type == "user_uploads"
            assert stored.collection_type not in PROTECTED_COLLECTION_TYPES


class TestSystemCollectionRenameGuardAtTheRoute:
    """``PUT /library/api/collections/{id}`` refuses to rename or redescribe a
    system collection (409), while deliberately leaving the per-collection
    ``is_public`` / ``agent_enabled`` toggles working.
    """

    @pytest.mark.parametrize(
        "collection_type", ["notes", "research_history", "default_library"]
    )
    @pytest.mark.parametrize(
        "payload",
        [
            {"name": "hijacked"},
            {"description": "hijacked"},
            {"name": "hijacked", "description": "hijacked"},
        ],
        ids=["name", "description", "both"],
    )
    def test_renaming_a_system_collection_is_409(
        self, authed, collection_type, payload
    ):
        client, username, password = authed
        original = f"System {uuid.uuid4().hex[:6]}"
        collection_id = _seed_collection(
            username,
            password,
            collection_type=collection_type,
            name=original,
        )

        resp = client.put(
            f"/library/api/collections/{collection_id}", json=payload
        )

        assert resp.status_code == 409, (
            f"{collection_type!r} must refuse {payload!r} with 409: "
            f"{resp.status_code} {resp.text[:300]}"
        )
        assert "system collection" in resp.json()["error"].lower(), resp.text

        from local_deep_research.database.models.library import Collection

        with _session(username, password) as session:
            stored = session.get(Collection, collection_id)
            assert stored.name == original, "the name must be unchanged"
            assert stored.description is None, (
                "the description must be unchanged"
            )

    def test_renaming_a_user_collection_still_works(self, authed):
        """Positive control: the 409s above are the system-collection guard,
        not a broken PUT route.
        """
        client, username, password = authed
        collection_id = _create_collection_via_api(client)

        resp = client.put(
            f"/library/api/collections/{collection_id}",
            json={"name": "renamed-fine", "description": "also fine"},
        )

        assert resp.status_code == 200, resp.text

        from local_deep_research.database.models.library import Collection

        with _session(username, password) as session:
            stored = session.get(Collection, collection_id)
            assert stored.name == "renamed-fine"
            assert stored.description == "also fine"

    def test_non_identity_toggles_still_work_on_a_system_collection(
        self, authed
    ):
        """The guard locks IDENTITY fields only. A "harden everything" change
        that also froze ``is_public`` / ``agent_enabled`` would silently break
        egress classification for the Notes collection -- fail loudly instead.
        """
        client, username, password = authed
        collection_id = _seed_collection(
            username, password, collection_type="notes", name="Notes"
        )

        resp = client.put(
            f"/library/api/collections/{collection_id}",
            json={"is_public": True, "agent_enabled": False},
        )

        assert resp.status_code == 200, resp.text

        from local_deep_research.database.models.library import Collection

        with _session(username, password) as session:
            stored = session.get(Collection, collection_id)
            assert bool(stored.is_public) is True
            assert stored.agent_enabled is False


class TestProtectedCollectionDeleteMapsTo409AtTheRoute:
    """``library_delete.py:288`` -- ``result["collection_type"] in
    PROTECTED_COLLECTION_TYPES`` -> 409.

    The service refusal was already covered
    (``tests/deletion/test_collection_deletion.py::
    TestProtectedCollectionTypes``); the route's status mapping was not. This
    class pins the helper-covered/route-uncovered shape identified by the
    review. The mapping is load-bearing,
    because the frontend distinguishes 409 (refused by policy) from 404
    (gone) and 400 (generic failure).
    """

    @pytest.mark.parametrize(
        "collection_type", ["notes", "research_history", "default_library"]
    )
    def test_deleting_a_protected_collection_is_409(
        self, authed, collection_type
    ):
        client, username, password = authed
        collection_id = _seed_collection(
            username, password, collection_type=collection_type
        )
        document_id = _seed_document(
            username,
            password,
            source_type_name="user_upload",
            title="Inside the system collection",
        )
        _link(username, password, document_id, collection_id)

        resp = client.delete(f"/library/api/collections/{collection_id}")

        assert resp.status_code == 409, (
            f"a protected collection must map to 409 (not 400, not 404): "
            f"{resp.status_code} {resp.text[:300]}"
        )
        body = resp.json()
        assert body["success"] is False
        assert body["collection_type"] == collection_type, body
        assert "system collection" in body["error"].lower(), body

        assert _collection_exists(username, password, collection_id), (
            "the protected collection must still exist"
        )
        assert _document_row(username, password, document_id)[0], (
            "its documents must not have been cascade-orphan-deleted"
        )

    def test_deleting_an_unknown_collection_is_404(self, authed):
        """Discriminator: the route's other refusal branch must stay 404, so
        "everything is 409" could not pass.
        """
        client, _, _ = authed

        resp = client.delete(f"/library/api/collections/{uuid.uuid4()}")

        assert resp.status_code == 404, resp.text
        assert "not found" in resp.json()["error"].lower()

    def test_deleting_an_ordinary_collection_still_returns_200(self, authed):
        """Positive control: the 409 above is the protected-type mapping, not
        a DELETE route that refuses everything.
        """
        client, username, password = authed
        collection_id = _create_collection_via_api(client)

        resp = client.delete(f"/library/api/collections/{collection_id}")

        assert resp.status_code == 200, resp.text
        assert resp.json()["success"] is True
        assert not _collection_exists(username, password, collection_id), (
            "an ordinary collection must really be gone"
        )


class TestNotesListPayloadElementTypes:
    """The list-shaped notes payloads must reject non-string ELEMENTS.

    Ported from ``tests/notes/test_notes_routes_review_fixes.py`` (deleted in
    this migration), which asserted each of these 400s and their exact
    messages. At the review snapshot nothing had replaced them; this class now
    pins all five messages in ``web/routers/notes.py``.

    These are not cosmetic validation. The routes' own comments say what the
    guards are standing in front of -- ``synthesize_notes`` does
    ``dict.fromkeys(note_ids)`` and the reorder path dedups -- so an
    unhashable element (a dict or a nested list) raises ``TypeError`` and
    FastAPI turns that into an opaque 500. The guard is the difference between
    a clean 400 telling the caller what is wrong and a request that looks like
    a backend outage in logs and monitoring.

    A shape that is merely *deep* rather than wrong is the interesting case:
    ``["a", ["b"]]`` passes ``isinstance(x, list)`` and only fails on the
    element check, which is exactly the assertion a naive rewrite drops.
    """

    #: (route, payload key, extra payload) for the three element-checked
    #: bodies. ``reorder`` is parametrised separately: it needs a note id.
    _AI_ROUTES = [
        ("/notes/api/notes/suggest-tags", "existing_tags"),
        ("/notes/api/notes/synthesize/preview", "note_ids"),
        ("/notes/api/notes/synthesize", "note_ids"),
    ]

    @pytest.mark.parametrize("route,key", _AI_ROUTES)
    @pytest.mark.parametrize(
        "bad",
        [
            pytest.param([{"a": 1}], id="dict-element"),
            pytest.param(["ok", ["nested"]], id="nested-list-element"),
            pytest.param([1, 2], id="int-elements"),
        ],
    )
    def test_non_string_elements_are_400_not_500(self, authed, route, key, bad):
        client, _, _ = authed

        resp = client.post(route, json={key: bad, "content": "x"})

        assert resp.status_code == 400, (
            f"{route} must reject a non-string element in {key!r} with 400, "
            f"not let it reach dict.fromkeys / dedup and 500: "
            f"{resp.status_code} {resp.text[:300]}"
        )
        assert resp.json()["error"] == f"{key} must be a list of strings"
        assert resp.json()["success"] is False

    @pytest.mark.parametrize("route,key", _AI_ROUTES)
    def test_a_well_formed_list_gets_past_the_element_guard(
        self, authed, route, key
    ):
        """Positive control. Without this, a route that 400s unconditionally
        would satisfy every assertion above.

        The request may still fail for its own reasons (no such note, AI not
        configured); what must NOT come back is the element-type error.
        """
        client, _, _ = authed

        resp = client.post(
            route,
            json={key: ["plain-string-id"], "content": "x"},
        )

        body = (
            resp.json()
            if resp.headers.get("content-type", "").startswith(
                "application/json"
            )
            else {}
        )
        assert body.get("error") != f"{key} must be a list of strings", (
            f"a list of plain strings must clear the element guard on {route}"
        )

    def test_reorder_rejects_a_non_list_and_an_empty_list(self, authed):
        client, _, _ = authed
        note_id = _create_note(client)

        for payload in ({"research_ids": "not-a-list"}, {"research_ids": []}):
            resp = client.post(
                f"/notes/api/notes/{note_id}/research/reorder",
                json=payload,
            )
            assert resp.status_code == 400, resp.text
            assert (
                resp.json()["error"] == "research_ids must be a non-empty list"
            ), resp.text

    def test_reorder_rejects_non_string_elements(self, authed):
        client, _, _ = authed
        note_id = _create_note(client)

        resp = client.post(
            f"/notes/api/notes/{note_id}/research/reorder",
            json={"research_ids": ["ok", {"nested": "object"}]},
        )

        assert resp.status_code == 400, resp.text
        assert (
            resp.json()["error"] == "research_ids must be a list of strings"
        ), resp.text

    def test_a_well_formed_reorder_reaches_the_ownership_check(self, authed):
        """Positive control for the two reorder tests above, and coverage of
        the third message in one shot.

        A syntactically valid list of strings must clear BOTH shape guards and
        fail later, on whether those ids are actually this note's linked
        research. Getting that third error proves the shape guards passed
        rather than the route rejecting everything.
        """
        client, _, _ = authed
        note_id = _create_note(client)

        resp = client.post(
            f"/notes/api/notes/{note_id}/research/reorder",
            json={"research_ids": ["not-linked-to-this-note"]},
        )

        assert resp.status_code == 400, resp.text
        assert resp.json()["error"] == (
            "research_ids do not match the note's linked research"
        ), resp.text