~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
"""The schema-change rule the migration gates enforce, with no git in it.
``test_migration_chain_integrity.py`` and
``test_upgrade_from_pre_migration_install.py`` both compare two trees --
one from the schema's live metadata, one from a directory listing -- and
then ask the same question of the result. That question lives here, as
pure functions over ``{path: blob}`` maps, so that:
* both gates enforce ONE rule rather than two that can drift apart, and
* the rule can be pinned against hand-written inputs
(``test_the_schema_change_rule_is_pinned_without_touching_git``) rather
than only against whatever the current checkout happens to contain. A
git-backed assertion on a clean tree sees no changes at all, so it
would pass just as happily if the rule were "anything goes".
The rule, in one sentence: **a shipped revision is immutable, adding a
revision is ordinary, and a guarded schema source file may only move in a
change that also adds a revision, unless isolated schema snapshots prove
that an edit emits exactly the same SQLite schema.**
"Guarded schema source file" is a path set, not a semantic judgement.
Each caller builds its own -- one from the live ``Base.metadata``
registry, one from the ``database/models`` directory listing -- and both
add :data:`DECLARATIVE_BASE_MODULE`, which no mapper walk and no models
directory listing can reach (see its comment below). The rule is only
ever as good as the set it is handed; it does not go looking for schema
sources of its own.
"Revision" means a revision *file*, not any file the callers happen to
hand over. Both callers build their second map from the whole
``database/migrations`` subtree, which also holds ``env.py`` and two
``__init__.py`` package markers. Those three are not revisions: no
user's ``alembic_version`` row names them, so they are ordinary source
files that may be edited, and adding one does not migrate anything.
:func:`is_revision` is what tells the two apart.
Finally, an edit is only an edit if it changes code. A blob hash moves
when a comment, a docstring or a line wrap moves, and neither Alembic nor
``create_all()`` can see any of those. :func:`drop_comment_only_edits`
lets the callers filter those out before asking :func:`violations`, so
the gate fires on changes that can actually reach a user's database.
The callers also exclude one byte-exact, schema-neutral JSON reader edit
from #7012. Its SQLite column DDL is checked separately; any later edit
changes the blob and again requires a migration.
"""
import ast
from pathlib import PurePosixPath
# The two subtrees the callers diff. Shared here for the same reason the
# rule is: a gate that watches a directory the other gate does not is a
# gap nobody notices, and a rename must be made in one place.
MIGRATIONS_SUBTREE = "src/local_deep_research/database/migrations"
MODELS_SUBTREE = "src/local_deep_research/database/models"
# The canonical ``Base = declarative_base()``. Guarded explicitly because
# neither caller's usual discovery reaches it: it defines no mapper, so it
# is invisible to the ``Base.registry.mappers`` walk in
# ``test_migration_chain_integrity.py``, and it sits one directory ABOVE
# MODELS_SUBTREE, so the directory listing in
# ``test_upgrade_from_pre_migration_install.py`` misses it too
# (``database/models/base.py`` only re-exports this object). It declares
# no ``__tablename__`` either, so the source scan in
# ``test_every_table_declared_under_src_is_either_migrated_or_a_separate_db``
# skips it as well. Yet giving ``declarative_base()`` a
# ``metadata=MetaData(naming_convention=...)`` here changes the name of
# every index and constraint ``0001``'s ``create_all()`` emits for a fresh
# install, while existing users keep the old names -- exactly the
# fresh-vs-upgraded divergence this rule exists to catch, and it would
# have passed with zero violations.
DECLARATIVE_BASE_MODULE = "src/local_deep_research/database/base.py"
# The directory Alembic loads revisions from. Nothing outside a directory
# with this name can be a revision, whatever it is called.
VERSIONS_DIR = "versions"
# Anti-vacuity floors, shared for the same reason the rule is: a "for each
# file under X" assertion proves nothing when X resolved to nothing, and
# two gates asking the same question must not disagree about how empty is
# too empty. Set at or below the real numbers, not below them: today the
# migrations subtree holds 33 files of which 30 are revisions, the mapper
# walk resolves 23 schema source files (24 once DECLARATIVE_BASE_MODULE is
# added), and the database/models listing holds 25 (26 with it).
# MIN_REVISIONS is therefore exactly today's revision count rather than
# under it, which is safe in the one direction that matters: a revision
# that shipped is immutable, so the count only ever grows, and a floor
# that equals it can only be tripped by a tree that failed to resolve --
# which is the whole job of these floors. Every floor is still well above
# the "at least one file" that a collapsed tree can still satisfy.
MIN_REVISIONS = 30
MIN_MODEL_FILES = 20
MIN_METADATA_TABLES = 60
# #7012 replaces only the read processor for these two JSON columns. The
# decorator still emits SQL JSON and does not change the bind path. These
# exact before/after blobs are reviewed as schema-neutral; any further edit
# to research.py changes its blob and again requires a migration. The live
# SQLite DDL is pinned in test_research_models.py.
_SCHEMA_NEUTRAL_MODEL_EDIT = {
"src/local_deep_research/database/models/research.py": (
"17688ead0b4210d9e5e82db4e851f3de516aad7b",
"ca9a78dcfc5bc631ce74b4cca60a025edf652171",
),
}
def drop_proven_schema_neutral_edits(
changes: dict, before_blobs: dict, after_blobs: dict
) -> dict:
"""Exempt only byte-exact model edits proven not to alter emitted DDL."""
kept = [
path
for path in changes["edited"]
if (before_blobs.get(path), after_blobs.get(path))
!= _SCHEMA_NEUTRAL_MODEL_EDIT.get(path)
]
return {**changes, "edited": kept}
def is_revision(path: str) -> bool:
"""Is ``path`` an Alembic revision file rather than migration plumbing?
Deliberately the same test ``_revision_files()`` in
``test_migration_chain_integrity.py`` uses to enumerate the chain: a
``.py`` module directly inside a ``versions/`` directory that is not
a dunder package marker. Alembic can load nothing else as a revision.
Not narrowed further to this repository's ``NNNN_name.py`` numbering.
The maps here carry blob hashes, not file contents, so "contains
``revision = ``" is not available; and the numbering itself is
already pinned by ``test_alembic_migrations.py::
test_migration_revision_ids_match_filenames``. Requiring it here
would re-enforce naming as a side effect and -- much worse -- would
quietly exempt an unconventionally named revision from the
immutability rule below, which is the direction that loses user
data. This predicate errs wide, so a real revision is always
immutable.
Erring wide is safe only because two other tests fence the wide side,
and a future simplification must keep BOTH: ``test_upgrade_from_pre_
migration_install.py::test_every_revision_file_is_on_the_single_chain_
to_head`` makes Alembic itself reject a non-revision ``.py`` dropped
into ``versions/`` (a ``helpers.py`` there is not on the chain), and
``test_only_files_alembic_can_load_as_revisions_count_as_revisions``
cross-checks this predicate over an ``rglob`` of the real directory
against the ``glob`` the chain walk uses, so the two cannot drift.
"""
parts = PurePosixPath(path).parts
if len(parts) < 2 or parts[-2] != VERSIONS_DIR:
return False
name = parts[-1]
return name.endswith(".py") and not name.startswith("__")
def classify(before: dict, after: dict) -> dict:
"""Split two ``{path: blob-hash}`` maps into added / removed / edited.
Blob hashes rather than names, so an *edited* file is caught as well
as an added or deleted one.
"""
return {
"added": sorted(set(after) - set(before)),
"removed": sorted(set(before) - set(after)),
"edited": sorted(
path
for path in set(before) & set(after)
if before[path] != after[path]
),
}
def _strip_docstrings(tree):
"""Remove every docstring statement from ``tree``, in place.
The ``ast.get_docstring`` definition of a docstring and nothing
wider: the first statement in a module, class or function body, when
it is a bare string constant. A string constant anywhere else stays,
because anywhere else it is a value the code can read.
"""
for node in ast.walk(tree):
if not isinstance(
node,
(ast.Module, ast.ClassDef, ast.FunctionDef, ast.AsyncFunctionDef),
):
continue
body = node.body
if (
body
and isinstance(body[0], ast.Expr)
and isinstance(body[0].value, ast.Constant)
and isinstance(body[0].value.value, str)
):
del body[0]
return tree
def code_is_unchanged(before_source: str, after_source: str) -> bool:
"""Do two versions of a ``.py`` file differ only in things Python drops?
True when the two parse to the same AST once docstrings are stripped
-- so comments, docstrings, blank lines, line wrapping and quote
style are all invisible here, and nothing else is. ``ast.dump`` is
called without ``include_attributes``, so line and column numbers do
not enter the comparison either.
This is the whole reason the gate can be strict without being
ignored: a revision file whose comments were reworded ships the same
``upgrade()`` to every user, and a model file whose class docstring
grew emits the same DDL from ``create_all()``. Reporting those as
schema changes trains people to work around the gate.
Conservative on failure: anything that does not parse (a syntax
error, a file that is not Python at all) is reported as changed, so
the filter can only ever be too quiet, never too permissive. A
pathologically deep expression can blow the interpreter's recursion
limit inside ``ast.parse``/``ast.dump`` rather than raising a normal
parse error; that is caught the same way, for the same reason.
"""
try:
before = _strip_docstrings(ast.parse(before_source))
after = _strip_docstrings(ast.parse(after_source))
return ast.dump(before) == ast.dump(after)
except (SyntaxError, ValueError, RecursionError):
return False
def drop_comment_only_edits(changes: dict, read_pair) -> dict:
"""A :func:`classify` result with cosmetic ``edited`` entries removed.
``read_pair(path)`` returns ``(before_source, after_source)`` for one
path, or ``None`` if either side cannot be read; a path whose sources
are unavailable stays reported. Only ``edited`` is filtered: an added
or removed file is a real change no matter what is in it, and a
removed revision is the worst case of all.
Only ``.py`` paths are considered. Nothing else in the guarded sets
has an AST, and a non-Python file that changed changed.
"""
kept = []
for path in changes["edited"]:
if path.endswith(".py"):
pair = read_pair(path)
if pair is not None and code_is_unchanged(*pair):
continue
kept.append(path)
return {**changes, "edited": kept}
def violations(
model_changes: dict,
revision_changes: dict,
*,
schema_unchanged: bool = False,
) -> list:
"""Every way a change breaks the rule, as actionable sentences.
Both arguments are :func:`classify` results -- one for the files that
feed the shipped schema, one for the Alembic ``migrations`` subtree.
An empty list means the change is allowed.
``schema_unchanged`` is evidence supplied by the isolated snapshot
comparison in ``schema_snapshot.model_edits_preserve_schema``. It
exempts model edits only; additions/removals and shipped revisions
remain guarded. Without that evidence the conservative rule applies.
Only the revision files in ``revision_changes`` are considered, on
both halves of the rule (see :func:`is_revision`): editing
``migrations/env.py`` is not editing a shipped revision, and adding
``migrations/README.md`` is not shipping a migration.
Both callers run their inputs through :func:`drop_comment_only_edits`
first, so an ``edited`` path that reaches here changed executable
code and not only comments, docstrings or formatting. The messages
below say so; they are written for that caller contract, since the
hand-written pins are the only other caller.
Deliberately NOT a violation: an added revision with no model change.
Data migrations, backfills, index-only revisions and constraint
repairs are ordinary and touch no ORM class.
Deliberately NOT checked, because a path set cannot see it: whether
an added revision has anything to do with the model change it is
paired with. See the callers' docstrings -- that is a reviewer's job,
and no test in this repository does it.
"""
problems = []
for kind in ("edited", "removed"):
for path in revision_changes[kind]:
if not is_revision(path):
continue
problems.append(
f"migration {kind}: {path}. A revision that shipped is "
"immutable -- an existing user's alembic_version row "
"already names it, so the new body never executes and "
"their database is permanently behind the code with "
"nothing to indicate it. Add a NEW revision instead. "
"(Comment-, docstring- and formatting-only edits are "
"filtered out before this check, so this one changes "
"the code Alembic runs.)"
)
added_revisions = [
path for path in revision_changes["added"] if is_revision(path)
]
moved = sorted(
set(model_changes["added"])
| set(model_changes["removed"])
| set(model_changes["edited"])
)
verified_refactor = (
schema_unchanged
and not model_changes["added"]
and not model_changes["removed"]
)
if moved and not added_revisions and not verified_refactor:
problems.append(
f"schema source file(s) changed with no migration added: {moved}. "
"Revision 0001 builds the baseline by calling "
"Base.metadata.create_all(), so a model edit lands in every "
"FRESH install immediately while existing users -- who only "
"replay the numbered revisions -- never receive it. The two "
"populations then run different schemas, and the first query "
"against the drifted table raises OperationalError on real "
"user data after the upgrade has already happened. Add the "
"Alembic revision, revert the model change, or prove that "
"the old and new models create identical SQLite schemas. (Comment-, "
"docstring- and formatting-only edits are filtered out "
"before this check, so the file(s) above changed code.)"
)
return problems