~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
docs(adr): record ADR-0012 wontfix for legacy documents.resource_id CASCADE FK (#6402)
LearningCircuit7204e8c8010d
diff --git a/docs/decisions/0012-wontfix-legacy-documents-resource-cascade.md b/docs/decisions/0012-wontfix-legacy-documents-resource-cascade.mdnew file mode 100644index 0000000000..6fe82f578c--- /dev/null+++ b/docs/decisions/0012-wontfix-legacy-documents-resource-cascade.md@@ -0,0 +1,308 @@+# ADR-0012: Wontfix — legacy `documents.resource_id` CASCADE FK (v1.3.0–v1.3.58 DBs)++**Date:** 2026-09-12+**Status:** Accepted (wontfix — residual risk tracked in [#6300](https://github.com/LearningCircuit/local-deep-research/issues/6300))++## Context++Databases whose `documents` table was created by an installed version from+**v1.3.0** (2025-12-07 — the first release containing the `Document` model's+introduction, commit `590e0112c`) through **v1.3.58** inclusive (the last+release before the fix) declare the `documents.resource_id` foreign key with+`ON DELETE CASCADE` at the **schema level**. **v1.3.59** is the first release+containing the model fix, PR+[#2587](https://github.com/LearningCircuit/local-deep-research/pull/2587) /+commit `2033f977e`. PR #2587 changed the model to `SET NULL` — deleting a+`ResearchResource` was supposed to orphan the `Document` instead of deleting+it — but no migration reconciled existing databases. Exposure is a property+of the *installed version* that first created the database, not of the+calendar date it was created: a `v1.3.0`–`v1.3.58` install first run months+after v1.3.59 shipped still creates a CASCADE-shaped `documents` table.++PR [#3772](https://github.com/LearningCircuit/local-deep-research/pull/3772)+(a follow-up to the investigation of user issue+[#3747](https://github.com/LearningCircuit/local-deep-research/issues/3747))+proposed exactly that migration: detect the FK's actual `on_delete` via+`PRAGMA foreign_key_list(documents)`, conditionally rebuild the table with+`SET NULL` through Alembic `batch_alter_table`, no-op on already-fixed DBs,+verify the result. That core design is sound, confirmed empirically by+replaying the branch's upgrade under the current migration runner (FK+enforcement off during the window, evidence 5; 2026-09-12): aligned+legacy databases converted correctly (rows, FKs, and indexes preserved),+the already-fixed schema was a no-op, and repeated application was+idempotent. The literal file is not reusable verbatim: its revision+(`0011`) now conflicts with the migration line (head `0030`), so reuse+requires renumbering/re-parenting onto the current head, and it must be+hardened with both alignment-aware settlement before the rebuild and a+failing `PRAGMA foreign_key_check` after it (evidence 5 details both;+the file as written does neither: the replay showed it converts the+FK while leaving missing-resource orphan rows and the resulting+`foreign_key_check` violations in place). The branch nevertheless went+stale (draft, conflicting, migration numbering superseded twice by a main+that advanced to `0030`), which forced a cost/benefit re-examination rather+than a rote rebase — the disposition complement to the long-lived-branch+guidance in [ADR-0008](0008-reviewing-long-lived-migration-branches.md),+which governs branches that are pursued.++## Evidence++1. **The FK could not fire for most of its existence.** SQLite enforces+ foreign keys only when `PRAGMA foreign_keys = ON` is set per connection.+ This application enabled that pragma in **v1.6.0** (published 2026-04-25;+ the enabling change, commit `3b1d6c6b2`, landed 2026-04-20 via PR+ [#3081](https://github.com/LearningCircuit/local-deep-research/pull/3081)).+ Migration 0007's notes — written about the `download_tracker` cascades,+ but stating the app-wide timing — say *"cascade was inert before+ v1.6.0"*. For the entire creation window the DB-level actions were+ inert, but not every resource delete left a dangling `resource_id`:+ the ORM's individual-resource delete still nulled `Document.resource_id`+ before deleting the resource (the shield of evidence 2, independent of+ pragma state), and pre-v1.6.0 bulk history deletion did not cascade+ into `research_resources`; only direct, raw, or manual deletion of a+ resource row could leave a document FK dangling.++2. **The reachable blast radius is narrow, and one of its two paths is+ shielded by the ORM.** Verified empirically (2026-09-12, in-memory+ SQLite with the pragma on, against the imported production models —+ all relationships live — and the legacy schema confirmed via+ `PRAGMA foreign_key_list(documents)`):+ - *Individual-resource delete* — the sole ORM path is+ `web/services/resource_service.py::delete_resource`. The model has+ carried `Document.resource = relationship("ResearchResource",+ backref="documents")` since the `Document` model's introduction+ (commit `590e0112c`); PR+ [#2582](https://github.com/LearningCircuit/local-deep-research/pull/2582)+ (2026-03-07) only *modified* that line, adding+ `foreign_keys="[Document.resource_id]"` to disambiguate it — forced by+ the reverse FK `ResearchResource.document_id` that the same PR+ introduced. With SQLAlchemy's default `passive_deletes=False`, on flush+ the ORM nulls `resource_id` **before** deleting the resource, so the+ DB-level CASCADE never fires — the document survives with+ `resource_id = NULL`, the same outcome as the fixed schema. This shield+ has held since the model's introduction; there was never a window in+ which the individual-delete path cascade-deleted the document. One+ mechanism nuance: the governing knob is the *collection-side* backref+ (`ResearchResource.documents`) — adding `passive_deletes=True` there+ (a documented SQLAlchemy optimization for CASCADE schemas) silently+ disarms the shield on legacy databases, while the same flag on+ `Document.resource` alone does not. The shield is a load-bearing+ invariant, not an incidental default; pinning it with a regression test+ is tracked in #6300.+ - *Research-session delete* — not one path but seven, all bulk+ `Query.delete()` calls on `ResearchHistory` that bypass ORM cascade+ handling; wherever the deleted session holds data, DB-level actions+ fire the identical chain: resources die (CASCADE, all DBs) and the+ session's documents die (`Document.research_id` is CASCADE on+ **all** databases — uniform behavior, out of scope for this record).+ Three of the seven are potentially data-bearing:+ `web/routers/research.py::delete_research` (single session);+ `web/routers/research.py::clear_history` (`:1696-1705`, all deletable+ sessions at once); and `chat/service.py::delete_attempt`+ (`:1085-1087`). The remaining four are rollback-only:+ `web/routers/chat.py::_cleanup_chat_send_rows`,+ `web/routers/research.py::_start_research_sync` (`:1336-1338`), and+ `web/routers/followup.py::_start_followup_sync` (`:463-465` and+ `:572-574`, two distinct deletes within that one function). Those+ four remove a newly created placeholder session before research+ starts, so resources and documents cannot exist there and their+ current blast radius is zero. None of the seven+ deletes `research_resources` directly, and the blast radius is+ unchanged across the three potentially data-bearing sites — the+ legacy FK adds marginal damage only for documents whose `resource_id`+ points into the deleted session while their `research_id` does not; current+ code sets `resource_id` once at creation and never re-points it, so+ such divergence is not manufactured today.+ - No other deletion route to `research_resources` exists under+ `src/local_deep_research/` — beyond the ORM `delete_resource` path+ above, no direct, bulk, or raw-SQL deletion of `ResearchResource`+ rows exists, and no ORM `session.delete()` of a `ResearchHistory`+ exists either (which would ORM-cascade through `resources`), so the+ ORM-cascade route to resource deletion is unused (verified September+ 2026 by sweeping `session.delete(`, `Query.delete(`, and raw-SQL+ patterns; code paths in this record are package-relative to that+ directory).++3. **Zero reports.** GitHub issue search (documents+ disappeared/lost/gone/empty, cascade, resource delete) and a web search+ (Reddit, HN, release notes) surface no user reports of this behavior —+ including ~4.5 months in which the FK has actually been enforced. The+ original defect was found by code review in PR+ [#2582](https://github.com/LearningCircuit/local-deep-research/pull/2582),+ not by a user. This null result carries limited weight: silent loss with+ no error or log is structurally hard for victims to report, discovery+ lags the trigger by months, and Discord/Reddit were not searchable.++4. **The exposed cohort pre-dates most of the user base.** Wayback Machine+ snapshots put the project at ~3.5k stars (Oct 2025) → ~4.1k (Mar 2026),+ versus ~9k today. The vulnerable schema shape shipped in releases for ~3+ months (v1.3.0, 2025-12-07 → v1.3.58, 2026-03-03); the majority of current+ and surviving legacy databases keep shrinking through reinstall and+ container volume recreation.++5. **The bounded benefit does not justify a migration for this cohort.**+ A batch rebuild of `documents` per se has shipped precedent — migration+ `0015` (PR+ [#4643](https://github.com/LearningCircuit/local-deep-research/pull/4643))+ rebuilt the table to drop the `notes` column on every production database+ without incident — so the operation is not unprecedented. The concrete+ differential is a bounded migration precondition, narrower than "any+ rebuild": legacy databases can carry orphaned `resource_id` values —+ ORM-path deletes were shielded even in the inert era, but raw-SQL or+ manual resource deletes left them unreconciled while FK enforcement+ was off. The standard migration runner disables FK enforcement for+ the migration window (`_disable_fk_for_migration`,+ `database/alembic_runner.py:267-307`) and re-arms it afterwards+ without validating existing rows (`_restore_fk_after_migration`,+ `:310-353` — no `PRAGMA foreign_key_check`), so a safe rebuild needs+ both controls. Before the rebuild, settle alignment by nulling+ `resource_id` whenever the referenced resource is missing or its+ `research_id` does not equal the document's:++ ```sql+ UPDATE documents SET resource_id = NULL+ WHERE resource_id IS NOT NULL AND NOT EXISTS (+ SELECT 1 FROM research_resources+ WHERE research_resources.id = documents.resource_id+ AND research_resources.research_id = documents.research_id);+ ```++ The `=` treats a document with NULL `research_id` as misaligned+ (`NULL = x` is never true, and `ResearchResource.research_id` is+ `nullable=False`, so only the document side can be NULL). After the+ rebuild, `PRAGMA foreign_key_check` must return no rows or the+ migration fails. Neither control suffices alone: an existence-only+ cleanup (`resource_id NOT IN (SELECT id FROM research_resources)`)+ misses divergent links, which remain FK-valid because the resource+ row exists; a check alone fails the migration on missing-resource+ orphans instead of fixing them, and it cannot detect divergence at+ all for that same reason.+ A nameable precondition, not a categorical barrier. Against a defect+ whose only ORM path is shielded (evidence 2), whose bulk-path+ exposure requires link divergence current code does not create, and+ which has zero recorded victims, even that bounded cost does not+ clear the bar.++6. **Precedent.** The codebase already tolerates legacy schema drift:+ migration 0005 deliberately did not enforce+ `research_resources.document_id` on existing databases (SQLite+ batch-alter limitation; see the model comment in+ `database/models/research.py`). Fresh installs and legacy installs+ already differ in FK shape.++## Decision++Close PR #3772 as **wontfix** — the migration will not be ported, and no+schema-reconciliation migration will be authored for databases whose+`documents` table was created by a `v1.3.0`–`v1.3.58` install. Track the+residual risk in issue+[#6300](https://github.com/LearningCircuit/local-deep-research/issues/6300),+which records: the affected version range; the guard condition, which names+the invariants this decision depends on — (1) **no new deletion route+or call site beyond the analyzed baseline (the seven existing bulk+`ResearchHistory` delete call sites documented in evidence 2), and no+semantic expansion or modification of an existing route, may reach+`research_resources` rows, directly or via cascade from ancestor+tables, while skipping the ORM null-out shield** (a semantic+invariant, not a syntax blacklist): any such deletion fires the armed+CASCADE on legacy DBs, whether legacy `Query.delete()`, SQLAlchemy 2+`session.execute(delete(...))` or Core DML, raw SQL, a database+trigger, or an ancestor cascade; (2) **no code+path that creates or updates a `Document` with `resource_id` set while+`research_id` is NULL or points to a different research session than+the one the `resource_id` belongs to**; (3) **the ORM must keep nulling+`Document.resource_id` before any ORM delete of a `ResearchResource`**,+which requires the `Document.resource` ↔ `ResearchResource.documents`+relationship pair/backref to remain in place (removal or rename that+drops the collection relationship triggers review as well) and+prohibits on that pair any `passive_deletes` value that disables the+null-out, `viewonly`, `lazy="noload"`, and `delete`/`delete-orphan`+cascade behavior, each of which silently disarms the ORM shield; and+(4) **`Document.research_id` remains `ON DELETE CASCADE` uniformly+across model-created and migrated schemas** (the marginal-damage+comparison depends on that uniformity, so any model or migration change+to it also triggers re-review) — any invariant breaking must be reviewed+against this ADR;+user guidance (individual-resource deletes through the application are+shielded by the ORM today; before recreating a database for any reason,+preserve an encrypted full-database backup as described in+[Database Backup](../security/database-backup.md); report export is not a+substitute, since it omits library documents and notes; and the default+Max Backups setting of 1 means the pre-recreation backup should be copied+outside the managed backup directory before a later automatic backup+replaces it); and the retained core migration design, to be+renumbered/re-parented to the then-current head and hardened with both+alignment-aware settlement before the rebuild and a failing+`PRAGMA foreign_key_check` after it, rather than reused verbatim,+should a real victim ever surface.++## Consequences++- Through current application paths, users on legacy databases who delete an+ individual research resource do **not** lose the linked document — it+ survives with `resource_id = NULL`, identical to the fixed schema's+ behavior. The model/schema divergence is real but has no unshielded+ trigger under `src/local_deep_research/` as of 2026-09-12.+- The armed CASCADE remains latent on those databases: a bulk deletion of+ research history (or any future bulk/raw-SQL resource deletion) fires it.+ For documents whose `resource_id` and `research_id` point at the same+ session — the case current code always produces — the fixed schema+ deletes those same `Document` rows anyway, via `Document.research_id`'s+ own CASCADE (present on all databases); the legacy FK changes nothing+ there. The legacy-only marginal damage — deleting a document the fixed+ schema would instead orphan with `resource_id = NULL` — is confined to+ the divergence case: `resource_id` pointing into the deleted session while+ `research_id` does not. Issue #6300 records this as the review trigger for+ new deletion paths.+- Second-order blast radius, gated on that same divergence condition:+ migrations `0021` and `0022` (notes-v2, applied on every database) attach+ `ON DELETE CASCADE` foreign keys from several notes tables to+ `documents.id` — `0021_add_note_tables.py:65,125,131,172,285` and+ `0022_add_note_references.py:62,69`, including `note_links` and+ `note_references`' `target_document_id` columns, which can point at a+ research-downloaded document. Any document the legacy CASCADE removes in+ the divergence case also removes its note links via those FKs.+- These guards bind application code only. Ad-hoc SQL executed against the+ database file by users or external tools (some of which enforce FKs by+ default) remains unsafe on legacy schemas and is accepted as residual+ risk — the one residual only the declined migration would fully close.+- One verified secondary hazard sits inside that same accepted residual: on+ a legacy database where manual or raw deletion left a dangling+ `resource_id`, SQLite can later reuse the deleted integer+ `ResearchResource.id` for a new resource, silently reattaching the document+ to it and misattributing its title, URL, and content. No current+ application path creates that dangling state; tracked in #6300.+- The affected cohort is expected to shrink overall (database recreation,+ attrition of pre-fix installs), but the decline is not strictly monotonic:+ a stale `v1.3.0`–`v1.3.58` installation can still create a new affected+ database. Current-version installs are correct from `create_all()`.+- No reliable automatic post-hoc detector exists: a fired CASCADE leaves no+ violating row, so lost documents surface only through a corroborated victim+ report or a backup comparison. Reopen this decision on either of those, or+ on a proposed code change that breaks a guard invariant; the response+ remains issue #6300 plus the retained #3772 core approach, renumbered+ and re-parented to the then-current migration head and hardened with+ both alignment-aware settlement before the rebuild and a failing+ `PRAGMA foreign_key_check` after it; not a verbatim reuse of the #3772+ file, and not a new investigation.+- Reviewers should not treat this specific divergence ("model says SET NULL+ but legacy schema says CASCADE", for `documents.resource_id` on databases+ whose `documents` table was created by a `v1.3.0`–`v1.3.58` install,+ regardless of when that install first ran) as an open defect; this record+ is the decision that it is accepted drift. New model/schema FK divergences+ remain open defects.++## References++- PR [#3772](https://github.com/LearningCircuit/local-deep-research/pull/3772) — the wontfixed migration (core approach retained: reuse requires renumbering/re-parenting to the current head plus both alignment-aware settlement before the rebuild and a failing `PRAGMA foreign_key_check` after it, not the literal file)+- Issue [#3747](https://github.com/LearningCircuit/local-deep-research/issues/3747)+ — user report (login-blocker; fixed by #3770) whose investigation prompted+ the #3772 follow-up; the defect itself was found by code review (#2582)+- PR [#2587](https://github.com/LearningCircuit/local-deep-research/pull/2587) — model-level CASCADE → SET NULL fix (commit `2033f977e`)+- PR [#2582](https://github.com/LearningCircuit/local-deep-research/pull/2582) — disambiguated the pre-existing `Document.resource` relationship (`foreign_keys="[Document.resource_id]"`, forced by the reverse FK `ResearchResource.document_id` it introduced) whose ORM null-out shields the delete path (commit `2b17c92d5`)+- PR [#3081](https://github.com/LearningCircuit/local-deep-research/pull/3081) — enabled `PRAGMA foreign_keys = ON` (commit `3b1d6c6b2`; shipped in v1.6.0)+- PR [#3708](https://github.com/LearningCircuit/local-deep-research/pull/3708) — FK-target fallout repairs; its migration 0007 notes document the arming timing (shipped in v1.6.4)+- PR [#4643](https://github.com/LearningCircuit/local-deep-research/pull/4643) / migration `0015` — precedent for a shipped `documents` batch rebuild+- Issue [#6300](https://github.com/LearningCircuit/local-deep-research/issues/6300) — residual-risk tracker+- `web/services/resource_service.py::delete_resource` — the sole ORM single-resource delete path+- `web/routers/research.py::delete_research`, `web/routers/research.py::clear_history` (`:1696-1705`), `chat/service.py::delete_attempt` (`:1085-1087`) — the three potentially data-bearing bulk session-delete paths that fire DB-level cascades; four rollback-only call sites — `web/routers/chat.py::_cleanup_chat_send_rows`, `web/routers/research.py::_start_research_sync` (`:1336-1338`), and `web/routers/followup.py::_start_followup_sync` (`:463-465`, `:572-574`) — delete newly created placeholder sessions (zero blast radius)+- [ADR-0008: Reviewing long-lived migration branches](0008-reviewing-long-lived-migration-branches.md) — the process record for branches that are pursued; this ADR records the complementary disposition