~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
fix(database): make SQLite migration schema changes atomic (#7099)
LearningCircuitf2139126d251
diff --git a/changelog.d/+sqlite-migration-ddl-rollback.bugfix.md b/changelog.d/+sqlite-migration-ddl-rollback.bugfix.mdnew file mode 100644index 0000000000..6f2b9beab9--- /dev/null+++ b/changelog.d/+sqlite-migration-ddl-rollback.bugfix.md@@ -0,0 +1,10 @@+Start SQLite and SQLCipher migrations with a physical `BEGIN IMMEDIATE`+transaction before schema changes, so failed upgrades roll back tables,+indexes and columns as well as data and the migration revision. The database+write lock is taken as that transaction begins, so a write from another+connection waits for the upgrade (up to the busy timeout) instead of landing+between its schema changes. When the initial migration creates a new+database's tables, as it does for an unencrypted database, it no longer+commits each statement separately. A new encrypted database still creates its+tables and indexes one committed statement at a time before its migrations+run.diff --git a/src/local_deep_research/database/alembic_runner.py b/src/local_deep_research/database/alembic_runner.pyindex a554ed3e4c..de995edae5 100644--- a/src/local_deep_research/database/alembic_runner.py+++ b/src/local_deep_research/database/alembic_runner.py@@ -272,14 +272,14 @@ def _disable_fk_for_migration(conn: Connection) -> None:``apply_performance_pragmas`` set ``PRAGMA foreign_keys = ON`` atconnect. SQLite then *silently ignores* further toggles of``foreign_keys`` once any transaction (explicit or driver-implicit)- is active. The sqlite3/sqlcipher3 driver auto-begins on the first- DML; PRAGMA itself isn't DML, so issuing the PRAGMA before any DML- is the only window where it actually takes effect.-- With multi-migration upgrades (revision 0001 → 0009), the first- migration to issue DML auto-begins the driver transaction and- freezes FK in the connect-time ON state for the rest of the upgrade.- That defeats migration 0007's defensive PRAGMA OFF and makes its+ is active. ``run_migrations`` starts the physical transaction at its+ ``conn.begin()`` boundary, before the first revision runs, so this+ helper must run before that boundary for the toggle to take effect.++ Before this helper, the first DML of a multi-migration upgrade+ (revision 0001 → 0009) auto-began the driver transaction and froze+ FK in the connect-time ON state for the rest of the upgrade.+ That defeated migration 0007's defensive PRAGMA OFF and made itsorphan-scrub DELETE fail with ``foreign key mismatch`` on tableswhose FK target lacks a UNIQUE backing — exactly the broken-schemastate migration 0007 is meant to repair.@@ -390,11 +390,14 @@ def run_migrations(engine: Engine, target: str = "head") -> None:databases. This ensures any missing tables are created.When ``target == "head"`` and the database is already at head, the call- short-circuits without opening a write transaction — calling- ``command.upgrade()`` unconditionally would still open a write transaction- via ``engine.begin()`` (taking a RESERVED lock on the SQLite file) just to- discover there's nothing to apply, serialising concurrent readers behind- a no-op on every cold engine reopen.+ short-circuits before the migration transaction opens. On SQLite that+ transaction starts with ``BEGIN IMMEDIATE`` (unless a begin-event hook+ already started one), so running it anyway would take the write lock+ just to discover there's nothing to apply, making other writers wait+ behind a no-op on every cold engine reopen. Any other ``target``, even+ the head revision's own ID, is not resolved up front, so the+ transaction opens and ``command.upgrade()`` runs even when the database+ is already at that revision.Security validations performed before running migrations:- Migration directory path is within expected package boundary@@ -407,10 +410,17 @@ def run_migrations(engine: Engine, target: str = "head") -> None:(issue #3990). Re-enabled after a successful upgrade, before theconnection returns to the pool.+ Inside the migration transaction, before Alembic runs, a database at+ revision 0001, 0002 or 0003 gains ``settings.env_var`` if its settings+ table lacks it (see ``_add_settings_env_var_before_0004``), so a failed+ upgrade rolls that column back too.+On failure inside the migration transaction, the inner``conn.begin()`` rolls back automatically — the database stays at- its previous revision. The original exception is re-raised so- callers can decide how to handle it.+ its previous revision. On SQLite the physical transaction starts+ before the first revision runs, so the rollback covers schema changes+ as well as data. The original exception is re-raised so callers can+ decide how to handle it.Args:engine: SQLAlchemy engine to migrate@@ -488,13 +498,12 @@ def run_migrations(engine: Engine, target: str = "head") -> None:"proceeding with upgrade to head")- # Short-circuit when the database is already at head. Calling- # command.upgrade() unconditionally opens a write transaction via- # engine.begin() even when there is nothing to apply — SQLite takes- # a RESERVED lock on the file as soon as the first DML lands inside- # that transaction, serialising concurrent readers behind a no-op on- # every cold engine reopen. The fresh-DB path (current is None) still- # runs the upgrade so tables and the alembic_version row get created.+ # Short-circuit when the database is already at head. The migration+ # transaction below starts with BEGIN IMMEDIATE, which takes SQLite's+ # write lock even when there is nothing to apply, making other writers+ # wait behind a no-op on every cold engine reopen. The fresh-DB path+ # (current is None) still runs the upgrade so tables and the+ # alembic_version row get created.if current is not None and current == head and target == "head":logger.info(f"Database already at revision {head}; skipping upgrade")return@@ -520,6 +529,23 @@ def run_migrations(engine: Engine, target: str = "head") -> None:# raise after SQLite has already accepted PRAGMA ... = OFF._disable_fk_for_migration(conn)with conn.begin():+ # SQLite's legacy driver begins only on DML. Start its+ # physical transaction before DDL so schema changes roll+ # back too. IMMEDIATE takes the write lock now, so+ # busy_timeout covers waiting for another writer. With a+ # DEFERRED BEGIN, Alembic's first reads would pin a WAL+ # snapshot, and a commit from another connection before+ # the first write would fail that write at once with+ # SQLITE_BUSY_SNAPSHOT, which busy_timeout never retries.+ # A begin-event hook may already have started the+ # transaction; its mode is kept.+ if (+ conn.dialect.name == "sqlite"+ and not conn.connection.driver_connection.in_transaction+ ):+ conn.exec_driver_sql("BEGIN IMMEDIATE")+ # After the BEGIN, so a failed upgrade also rolls back+ # this ADD COLUMN instead of leaving it committed._add_settings_env_var_before_0004(conn, current)config.attributes["connection"] = conncommand.upgrade(config, target)@@ -542,7 +568,7 @@ def run_migrations(engine: Engine, target: str = "head") -> None:except Exception:logger.exception("Database migration failed — transaction rolled back where "- "supported; SQLite DDL from a failed revision may require repair"+ "supported")raisediff --git a/src/local_deep_research/database/migrations/env.py b/src/local_deep_research/database/migrations/env.pyindex cb6c1e2cd9..1b2e53e1cd 100644--- a/src/local_deep_research/database/migrations/env.py+++ b/src/local_deep_research/database/migrations/env.py@@ -30,7 +30,11 @@ def run_migrations_online():Run migrations in 'online' mode using the provided connection.The connection is passed via config.attributes["connection"] by alembic_runner.py,- which opens it inside engine.begin() so that failures auto-rollback.+ which runs the upgrade inside conn.begin() with SQLite's physical+ transaction already open (see run_migrations), so a failure rolls back+ schema changes as well as data. Because conn.begin() is already active,+ context.begin_transaction() below is a no-op and every revision runs in+ that one transaction."""connection = config.attributes.get("connection")diff --git a/src/local_deep_research/database/migrations/versions/0006_journal_quality_system.py b/src/local_deep_research/database/migrations/versions/0006_journal_quality_system.pyindex 3e7048db15..ccb92d5ea9 100644--- a/src/local_deep_research/database/migrations/versions/0006_journal_quality_system.py+++ b/src/local_deep_research/database/migrations/versions/0006_journal_quality_system.py@@ -256,10 +256,11 @@ def upgrade() -> None:## Concurrency note: BEGIN EXCLUSIVE was considered here and# rejected — it raises "cannot start a transaction within a- # transaction" inside the outer engine.begin() that- # alembic_runner.py:268 opens. In practice, SQLite WAL +- # _connections_lock (encrypted_db.py) + the current==head- # short-circuit in alembic_runner serialize access sufficiently.+ # transaction" inside the migration transaction that+ # alembic_runner.run_migrations opens before the first revision.+ # The runner starts that transaction with BEGIN IMMEDIATE, so+ # SQLite's write lock is already held and no other connection can+ # write between this read and the delete below.journals_pre = sa.Table("journals", sa.MetaData(), autoload_with=bind)all_rows = bind.execute(sa.select(diff --git a/src/local_deep_research/database/migrations/versions/0007_backfill_missing_indexes.py b/src/local_deep_research/database/migrations/versions/0007_backfill_missing_indexes.pyindex 78c0b6814e..e617316702 100644--- a/src/local_deep_research/database/migrations/versions/0007_backfill_missing_indexes.py+++ b/src/local_deep_research/database/migrations/versions/0007_backfill_missing_indexes.py@@ -157,14 +157,13 @@ def upgrade() -> None:# The actual FK toggle that takes effect is in# ``alembic_runner._disable_fk_for_migration``, run BEFORE the migration# transaction opens — SQLite silently ignores ``PRAGMA foreign_keys``- # once any transaction is active, and earlier migrations in this same- # upgrade chain may have already issued DML, auto-beginning the driver- # transaction. Issuing the PRAGMA again here is a defensive belt-and-- # suspenders no-op when the runner has already disabled FK; it preserves- # the migration's contract for any caller that runs it outside the- # standard ``run_migrations`` path. The runner re-enables FK on the- # same connection after the upgrade commits, before returning the- # connection to the pool.+ # once any transaction is active, and the runner starts the migration+ # transaction before the first revision runs. Under the runner this+ # PRAGMA is therefore always ignored, and FK is already off; it+ # preserves the migration's contract for a caller that runs it outside+ # ``run_migrations`` on a connection with no transaction open yet. The+ # runner re-enables FK on the same connection after the upgrade+ # commits, before returning the connection to the pool.bind.execute(text("PRAGMA foreign_keys = OFF"))_scrub_download_tracker_duplicates(bind)diff --git a/src/local_deep_research/database/migrations/versions/0010_add_chat_tables.py b/src/local_deep_research/database/migrations/versions/0010_add_chat_tables.pyindex d9da47fc09..f8e0abb520 100644--- a/src/local_deep_research/database/migrations/versions/0010_add_chat_tables.py+++ b/src/local_deep_research/database/migrations/versions/0010_add_chat_tables.py@@ -255,9 +255,10 @@ def upgrade():inspector = inspect(bind)# Validate any pre-existing FK before creating chat tables or indexes.- # SQLite DDL is not transactionally rolled back by Alembic, so discovering- # an incompatible constraint later would otherwise leave a partially- # upgraded schema even though the revision remains at 0009.+ # Under run_migrations a failure also rolls back this revision's DDL, but+ # a caller that runs it on a connection with no transaction open would+ # otherwise keep any DDL issued before a late validation failure, leaving+ # a partially upgraded schema even though the revision remains at 0009.research_history_exists = inspector.has_table("research_history")chat_session_fk_exists = Falseif research_history_exists:diff --git a/src/local_deep_research/database/migrations/versions/0031_reconcile_corrupted_settings.py b/src/local_deep_research/database/migrations/versions/0031_reconcile_corrupted_settings.pyindex db3422da61..5e31fdb71e 100644--- a/src/local_deep_research/database/migrations/versions/0031_reconcile_corrupted_settings.py+++ b/src/local_deep_research/database/migrations/versions/0031_reconcile_corrupted_settings.py@@ -406,8 +406,9 @@ def _repair_corrupted_values(if sensitive_keys is None:sensitive_keys = _sensitive_setting_keys(conn)# SQLite's legacy driver starts a transaction for DML, not SELECT or- # index DDL. Start it even when every repair target is sensitive and- # the loop below skips all updates. The false predicate changes no row.+ # index DDL. run_migrations has already started one; for a caller that+ # has not, start it even when every repair target is sensitive and the+ # loop below skips all updates. The false predicate changes no row.conn.execute(sa.update(_settings).where(sa.false()).values(value=None))repaired_count = 0# Normalize equivalent SQLite TEXT/BLOB storage classes for exact matches.@@ -444,10 +445,13 @@ def upgrade() -> None:# Retain that classification so repair cannot overwrite a newer clear.sensitive_keys = _sensitive_setting_keys(conn)duplicate_count = _deduplicate_settings(conn)- # Keep index DDL after a guaranteed DML statement. SQLite/SQLCipher legacy- # transaction control does not emit a driver-level BEGIN for SELECT or DDL,- # while repair explicitly issues a zero-row UPDATE even if every target- # is skipped. This makes index replacement participate in Alembic's rollback.+ # Keep index DDL after a guaranteed DML statement. run_migrations has+ # already started the driver transaction. For a caller that has not, on a+ # connection in SQLite/SQLCipher's default deferred mode, legacy+ # transaction control emits no driver-level BEGIN for SELECT or DDL, but+ # the zero-row UPDATE in repair begins one even if every target is+ # skipped, so index replacement still rolls back with the enclosing+ # transaction.repaired_count = _repair_corrupted_values(conn, sensitive_keys)_ensure_unique_settings_key_index(conn)diff --git a/src/local_deep_research/database/migrations/versions/0032_repair_pre_alembic_databases.py b/src/local_deep_research/database/migrations/versions/0032_repair_pre_alembic_databases.pyindex 8e287a69ea..58af487e38 100644--- a/src/local_deep_research/database/migrations/versions/0032_repair_pre_alembic_databases.py+++ b/src/local_deep_research/database/migrations/versions/0032_repair_pre_alembic_databases.py@@ -47,9 +47,11 @@ without a NULL fallback:``none``.Each backfill only fills NULL rows and runs whenever the column exists, not-only when this step adds it. On SQLite and SQLCipher an ADD COLUMN issued-before any DML commits on its own, so a failed upgrade can leave the column-added but not backfilled; the retry then fills it.+only when this step adds it. The migration runner now rolls back a failed+upgrade's ADD COLUMN with the rest of the upgrade, but before #7099 an ADD+COLUMN issued before any DML committed on its own on SQLite and SQLCipher.+An upgrade that failed under that runner can have left the column added but+not backfilled; the retry then fills it.3. Rewrite ``settings.type`` values to SettingType member names.@@ -212,9 +214,9 @@ def upgrade() -> None:op.add_column(table, sa.Column(name, column_type, nullable=True))- # Also when the column already existed: an earlier attempt may- # have added it (the DDL commits on its own) and then failed- # before its backfill committed.+ # Also when the column already existed: an attempt before+ # #7099, whose runner committed this DDL on its own, may have+ # added it and then failed before its backfill committed.backfill = _BACKFILLS.get((table, name))if backfill is not None:backfill(conn, existing)diff --git a/tests/database/test_alembic_migrations.py b/tests/database/test_alembic_migrations.pyindex 0b37afa4f1..dd083406d8 100644--- a/tests/database/test_alembic_migrations.py+++ b/tests/database/test_alembic_migrations.py@@ -223,9 +223,9 @@ class TestAlembicRunner:self, fresh_engine, loguru_caplog):"""When the database is already at head, run_migrations must not- call command.upgrade() — that would open a no-op write transaction- and hold a RESERVED lock under isolation_level="IMMEDIATE", blocking- concurrent readers for no benefit.+ call command.upgrade() — the migration transaction starts with+ BEGIN IMMEDIATE, so a no-op upgrade would still take SQLite's write+ lock and make other writers wait for no benefit."""# Migrate to head firstrun_migrations(fresh_engine)@@ -3840,17 +3840,18 @@ class TestCryptographicConsistency:def _insert_witness_then_fail(config, target):"""side_effect for patched ``command.upgrade``.- Inserts a witness row through the transaction's connection, then- raises. If ``engine.begin()``'s rollback works, the row must not- be present after the raise — this is what makes the rollback- tests non-tautological. A plain ``side_effect=RuntimeError(...)``- fires before any DB write, so the post-failure "revision- preserved" assertion would be trivially true.-- Uses DML (INSERT) rather than DDL (CREATE TABLE) because pysqlite- does not reliably roll back DDL statements on default SQLite. The- caller must pre-create a ``rollback_witness`` table before- patching ``command.upgrade`` with this side_effect.+ Inserts a witness row through the migration transaction's+ connection, then raises. If the runner's rollback works, the row+ must not be present after the raise — this is what makes the+ rollback tests non-tautological. A plain+ ``side_effect=RuntimeError(...)`` fires before any DB write, so the+ post-failure "revision preserved" assertion would be trivially true.++ The witness is a data row. The runner starts SQLite's physical+ transaction before ``command.upgrade`` runs, so DDL rolls back as+ well; ``test_migration_ddl_rollback.py`` covers that. The caller+ must pre-create a ``rollback_witness`` table before patching+ ``command.upgrade`` with this side_effect."""conn = config.attributes["connection"]conn.execute(text("INSERT INTO rollback_witness (x) VALUES (1)"))@@ -3962,8 +3963,8 @@ class TestMigrationErrorSanitization:try:# Pre-create the witness table outside the transaction the- # patched call will run in. DML (INSERT) rollback is- # reliable on default SQLite; DDL (CREATE TABLE) is not.+ # patched call will run in, so only the witness row is inside+ # it.with engine.begin() as conn:conn.execute(text("CREATE TABLE rollback_witness (x INTEGER)"))@@ -4558,8 +4559,10 @@ class TestPreAlembicDatabaseUpgrade:):"""A retry must backfill columns that already exist but hold NULL.- On SQLite an ADD COLUMN issued before any DML commits on its own, so- a failed 0032 can leave the column added but not backfilled.+ Before #7099 the runner committed an ADD COLUMN issued before any+ DML on its own on SQLite, so a failed 0032 could leave the column+ added but not backfilled. The runner now rolls that column back+ with the failed upgrade, so this test builds the state by hand."""engine = create_engine(f"sqlite:///{tmp_path / 'interrupted.db'}")try:@@ -5328,23 +5331,25 @@ class TestUpgradeFromBuggyV16xUserDbProductionEngine:"""Regression for #3990 — multi-migration upgrade fails with FK mismatch.The existing ``TestUpgradeFromBuggyV16xUserDatabase`` covers the same- schema corruption but stamps at revision 0006, so migration 0007 is the- first to run any DML in the upgrade transaction — which lets 0007's- own ``PRAGMA foreign_keys = OFF`` actually take effect (no auto-begun+ schema corruption but stamps at revision 0006. Before the runner fix,+ migration 0007 was then the first to run any DML in the upgrade, which+ let 0007's own ``PRAGMA foreign_keys = OFF`` take effect (no auto-beguntransaction yet).Real production users (issue #3990) upgrade from 0001 to head. Migrations- 0002–0006 run DML before 0007, auto-beginning the sqlite3 driver+ 0002–0006 ran DML before 0007, auto-beginning the sqlite3 drivertransaction. ``PRAGMA foreign_keys`` is silently a no-op once atransaction is active (per sqlite.org/pragma.html#pragma_foreign_keys),- so 0007's defensive PRAGMA never lands and the orphan-scrub DELETE- fails with ``foreign key mismatch``.+ so 0007's defensive PRAGMA never landed and the orphan-scrub DELETE+ failed with ``foreign key mismatch``.The fix (in ``alembic_runner.run_migrations``) issues PRAGMA OFF- *before* opening the migration transaction. This test reproduces the- production failure exactly: ``isolation_level=""`` (matching the- sqlcipher3 engine in ``encrypted_db.py``) + FK ON at connect via the- same event handler ``apply_performance_pragmas`` installs.+ *before* opening the migration transaction. The runner now also starts+ that transaction before the first revision runs, so 0007's own PRAGMA+ is ignored in both classes. This test reproduces the production setup+ exactly: ``isolation_level=""`` (matching the sqlcipher3 engine in+ ``encrypted_db.py``) + FK ON at connect via the same event handler+ ``apply_performance_pragmas`` installs."""@pytest.fixture@@ -5353,9 +5358,10 @@ class TestUpgradeFromBuggyV16xUserDbProductionEngine:connect, with the buggy v1.6.x schema stamped at revision 0005.Stamping at 0005 (not 0006) means migration 0006's data backfill- (``UPDATE journals SET name_lower = ...``) runs DML before 0007,- auto-beginning the driver transaction and freezing FK in the- connect-time ON state for the rest of the upgrade.+ (``UPDATE journals SET name_lower = ...``) runs DML before 0007, as+ in the #3990 reports, where that DML auto-began the driver+ transaction and froze FK in the connect-time ON state for the rest+ of the upgrade."""import sqlite3diff --git a/tests/database/test_migration_ddl_rollback.py b/tests/database/test_migration_ddl_rollback.pynew file mode 100644index 0000000000..e47db8fc6e--- /dev/null+++ b/tests/database/test_migration_ddl_rollback.py@@ -0,0 +1,438 @@+"""Migrations must hold SQLite's write lock, let readers read, and roll back+schema changes."""++import sqlite3++import pytest+from sqlalchemy import create_engine, event, inspect++from local_deep_research.database import alembic_runner as runner+from local_deep_research.database.models import Base+from local_deep_research.database.sqlcipher_utils import (+ create_sqlcipher_connection,+)++[email protected](params=["sqlite", "sqlcipher"])+def migration_database(request, tmp_path):+ """Yield an engine and a function that opens an independent connection.++ The SQLCipher engine pools a single connection per thread, so a second+ connection must come from ``connect`` rather than from the engine.+ """+ database = tmp_path / "migration.db"+ if request.param == "sqlcipher":+ pytest.importorskip("sqlcipher3", reason="SQLCipher driver unavailable")++ def connect():+ return create_sqlcipher_connection(+ database,+ hex_key="ab" * 32,+ creation_mode=not database.exists()+ or database.stat().st_size == 0,+ )++ engine = create_engine("sqlite://", creator=connect)+ else:++ def connect():+ return sqlite3.connect(database)++ engine = create_engine(f"sqlite:///{database}")++ @event.listens_for(engine, "connect")+ def enable_foreign_keys(dbapi_connection, connection_record):+ cursor = dbapi_connection.cursor()+ cursor.execute("PRAGMA foreign_keys = ON")+ cursor.close()++ yield engine, connect+ engine.dispose()+++def migration_engine(migration_database):+ engine, _connect = migration_database+ return engine+++def open_connection_that_never_waits(connect):+ """Open another connection whose statements fail at once on a held lock."""+ connection = connect()+ connection.execute("PRAGMA busy_timeout = 0")+ return connection+++def try_write(connection):+ """Insert and commit a row; return SQLite's lock error if refused."""+ try:+ connection.execute(+ "INSERT INTO stored_values (value) VALUES (?)", ("other writer",)+ )+ connection.commit()+ except connection.OperationalError as error:+ connection.rollback()+ return str(error)+ return None+++def try_read(connection):+ """Return the committed values; return SQLite's lock error if refused."""+ try:+ rows = connection.execute("SELECT value FROM stored_values").fetchall()+ except connection.OperationalError as error:+ return str(error)+ return [value for (value,) in rows]+++def assert_foreign_keys_enabled(engine):+ with engine.connect() as connection:+ assert (+ connection.exec_driver_sql("PRAGMA foreign_keys").scalar_one() == 1+ )+++def seed_previous_revision(engine):+ with engine.begin() as connection:+ connection.exec_driver_sql(+ "CREATE TABLE stored_values (id INTEGER PRIMARY KEY, value TEXT)"+ )+ connection.exec_driver_sql(+ "INSERT INTO stored_values (id, value) VALUES (?, ?)",+ (1, "original"),+ )+ connection.exec_driver_sql(+ "CREATE TABLE alembic_version (version_num VARCHAR(32) PRIMARY KEY)"+ )+ connection.exec_driver_sql(+ "INSERT INTO alembic_version (version_num) VALUES (?)", ("0001",)+ )+++def change_schema_and_data(config, target):+ assert target == "0002"+ connection = config.attributes["connection"]+ # DDL precedes DML, as it does in real schema migrations. A driver that+ # waits until the UPDATE to begin cannot roll back these schema changes.+ connection.exec_driver_sql(+ "ALTER TABLE stored_values ADD COLUMN extra TEXT"+ )+ connection.exec_driver_sql(+ "CREATE INDEX stored_value_index ON stored_values (value)"+ )+ connection.exec_driver_sql(+ "CREATE TABLE added_table (id INTEGER PRIMARY KEY)"+ )+ connection.exec_driver_sql(+ "UPDATE stored_values SET value = ?", ("changed",)+ )+ connection.exec_driver_sql(+ "UPDATE alembic_version SET version_num = ?", ("0002",)+ )+++def test_failed_fresh_migration_leaves_no_schema(migration_engine, monkeypatch):+ def fail_after_schema(config, target):+ connection = config.attributes["connection"]+ connection.exec_driver_sql(+ "CREATE TABLE added_table (id INTEGER PRIMARY KEY)"+ )+ connection.exec_driver_sql(+ "CREATE INDEX added_index ON added_table (id)"+ )+ raise RuntimeError("injected migration failure")++ monkeypatch.setattr(runner.command, "upgrade", fail_after_schema)+ with pytest.raises(RuntimeError, match="injected migration failure"):+ runner.run_migrations(migration_engine)++ migration_engine.dispose()+ assert inspect(migration_engine).get_table_names() == []+ assert runner.get_current_revision(migration_engine) is None+ assert_foreign_keys_enabled(migration_engine)++[email protected]("failure", [RuntimeError, KeyboardInterrupt])+def test_failed_upgrade_preserves_schema_data_and_revision(+ migration_engine, monkeypatch, failure+):+ seed_previous_revision(migration_engine)++ def fail_after_changes(config, target):+ change_schema_and_data(config, target)+ raise failure("injected migration failure")++ monkeypatch.setattr(runner.command, "upgrade", fail_after_changes)+ with pytest.raises(failure, match="injected migration failure"):+ runner.run_migrations(migration_engine, target="0002")++ migration_engine.dispose()+ inspector = inspect(migration_engine)+ assert set(inspector.get_table_names()) == {+ "stored_values",+ "alembic_version",+ }+ assert {+ column["name"] for column in inspector.get_columns("stored_values")+ } == {+ "id",+ "value",+ }+ assert inspector.get_indexes("stored_values") == []+ with migration_engine.connect() as connection:+ assert (+ connection.exec_driver_sql(+ "SELECT value FROM stored_values"+ ).scalar_one()+ == "original"+ )+ assert runner.get_current_revision(migration_engine) == "0001"+ assert_foreign_keys_enabled(migration_engine)+++def test_successful_upgrade_commits_schema_data_and_revision(+ migration_engine, monkeypatch+):+ seed_previous_revision(migration_engine)+ monkeypatch.setattr(runner.command, "upgrade", change_schema_and_data)+ runner.run_migrations(migration_engine, target="0002")++ migration_engine.dispose()+ inspector = inspect(migration_engine)+ assert "added_table" in inspector.get_table_names()+ assert "extra" in {+ column["name"] for column in inspector.get_columns("stored_values")+ }+ assert {+ index["name"] for index in inspector.get_indexes("stored_values")+ } == {"stored_value_index"}+ with migration_engine.connect() as connection:+ assert (+ connection.exec_driver_sql(+ "SELECT value FROM stored_values"+ ).scalar_one()+ == "changed"+ )+ assert runner.get_current_revision(migration_engine) == "0002"+ assert_foreign_keys_enabled(migration_engine)+++def test_real_initial_migration_rolls_back_created_models(+ migration_engine, monkeypatch+):+ create_all = Base.metadata.create_all++ def fail_after_models(*args, **kwargs):+ create_all(*args, **kwargs)+ raise RuntimeError("injected migration failure")++ monkeypatch.setattr(Base.metadata, "create_all", fail_after_models)+ with pytest.raises(RuntimeError, match="injected migration failure"):+ runner.run_migrations(migration_engine)++ migration_engine.dispose()+ assert inspect(migration_engine).get_table_names() == []+ assert runner.get_current_revision(migration_engine) is None+ assert_foreign_keys_enabled(migration_engine)+++def settings_columns(bind):+ return {column["name"] for column in inspect(bind).get_columns("settings")}+++def test_failed_upgrade_rolls_back_env_var_added_before_0004(+ migration_engine, monkeypatch+):+ """The settings.env_var column added for revision 0004 rolls back too.++ The runner adds it to a database below 0004 before Alembic runs. Added+ before the BEGIN, that ADD COLUMN would commit on its own and survive+ a failure in a later revision.+ """+ runner.run_migrations(migration_engine, target="0003")+ with migration_engine.begin() as connection:+ connection.exec_driver_sql("ALTER TABLE settings DROP COLUMN env_var")+ # 0004 names env_var when it re-scopes this legacy key.+ connection.exec_driver_sql(+ "INSERT INTO settings (key, value, type, name, ui_element, "+ "visible, editable) VALUES ('app.iterations', '3', 'APP', "+ "'Legacy', 'text', 1, 1)"+ )+ upgrade = runner.command.upgrade+ during_upgrade = []++ def fail_after_later_revisions(config, target):+ upgrade(config, target)+ connection = config.attributes["connection"]+ during_upgrade.append(+ (+ "env_var" in settings_columns(connection),+ connection.exec_driver_sql(+ "SELECT version_num FROM alembic_version"+ ).scalar_one(),+ )+ )+ raise RuntimeError("injected migration failure")++ monkeypatch.setattr(runner.command, "upgrade", fail_after_later_revisions)+ with pytest.raises(RuntimeError, match="injected migration failure"):+ runner.run_migrations(migration_engine)++ # 0004 and every later revision ran on the added column, then failed.+ assert during_upgrade == [(True, runner.get_head_revision())]+ migration_engine.dispose()+ assert "env_var" not in settings_columns(migration_engine)+ assert runner.get_current_revision(migration_engine) == "0003"+ with migration_engine.connect() as connection:+ keys = set(+ connection.exec_driver_sql("SELECT key FROM settings").scalars()+ )+ assert "app.iterations" in keys+ assert "search.iterations" not in keys+ assert_foreign_keys_enabled(migration_engine)+++def test_migration_supports_transaction_started_by_begin_event(+ migration_engine, monkeypatch+):+ disable_fk = runner._disable_fk_for_migration+ starts = []++ def prepare_connection(connection):+ disable_fk(connection)++ def begin_transaction(active_connection):+ active_connection.exec_driver_sql("BEGIN")+ starts.append(True)++ # Start a physical transaction at the migration's existing begin()+ # boundary, after the foreign-key pragma has been changed.+ event.listen(connection, "begin", begin_transaction, once=True)++ def create_schema(config, target):+ config.attributes["connection"].exec_driver_sql(+ "CREATE TABLE added_table (id INTEGER PRIMARY KEY)"+ )++ monkeypatch.setattr(runner, "_disable_fk_for_migration", prepare_connection)+ monkeypatch.setattr(runner.command, "upgrade", create_schema)+ runner.run_migrations(migration_engine)++ assert starts == [True]+ migration_engine.dispose()+ assert inspect(migration_engine).get_table_names() == ["added_table"]+ assert_foreign_keys_enabled(migration_engine)+++def assert_only_migrated_row(engine):+ with engine.connect() as connection:+ assert connection.exec_driver_sql(+ "SELECT value FROM stored_values"+ ).scalars().all() == ["changed"]+++def test_migration_holds_write_lock_before_upgrade_runs(+ migration_database, monkeypatch+):+ engine, connect = migration_database+ seed_previous_revision(engine)+ other = open_connection_that_never_waits(connect)+ lock_errors = []++ def write_from_other_connection_then_change(config, target):+ # BEGIN IMMEDIATE took the write lock before any migration statement.+ lock_errors.append(try_write(other))+ change_schema_and_data(config, target)++ monkeypatch.setattr(+ runner.command, "upgrade", write_from_other_connection_then_change+ )+ try:+ runner.run_migrations(engine, target="0002")+ finally:+ other.close()++ assert lock_errors == ["database is locked"]+ engine.dispose()+ assert_only_migrated_row(engine)+ assert runner.get_current_revision(engine) == "0002"+ assert_foreign_keys_enabled(engine)+++def test_reader_can_read_while_migration_holds_write_lock(+ migration_database, monkeypatch+):+ """BEGIN IMMEDIATE locks out other writers but not readers.++ In WAL mode readers do not wait for a writer, and BEGIN EXCLUSIVE acts+ like IMMEDIATE. With a rollback journal, BEGIN EXCLUSIVE would also lock+ readers out until the upgrade commits, so this test uses one.+ """+ monkeypatch.setenv("LDR_DB_CONFIG_JOURNAL_MODE", "DELETE")+ engine, connect = migration_database+ seed_previous_revision(engine)+ other = open_connection_that_never_waits(connect)+ lock_errors = []+ reads = []++ def read_from_other_connection_during_upgrade(config, target):+ lock_errors.append(try_write(other))+ change_schema_and_data(config, target)+ # Nothing is committed yet, so the reader sees the old row.+ reads.append(try_read(other))++ monkeypatch.setattr(+ runner.command, "upgrade", read_from_other_connection_during_upgrade+ )+ try:+ journal_mode = other.execute("PRAGMA journal_mode").fetchone()+ assert journal_mode[0].lower() == "delete"+ runner.run_migrations(engine, target="0002")+ finally:+ other.close()++ assert lock_errors == ["database is locked"]+ assert reads == [["original"]]+ engine.dispose()+ assert_only_migrated_row(engine)+ assert runner.get_current_revision(engine) == "0002"+ assert_foreign_keys_enabled(engine)+++def test_commit_after_version_read_does_not_fail_migration(+ migration_database, monkeypatch+):+ engine, connect = migration_database+ seed_previous_revision(engine)+ engine.dispose()+ other = open_connection_that_never_waits(connect)+ lock_errors = []++ def read_version_then_change(config, target):+ connection = config.attributes["connection"]+ # Alembic reads the version table before the first schema change. A+ # DEFERRED BEGIN would pin a WAL snapshot here, and a commit from+ # another connection would then fail the first write at once with+ # SQLITE_BUSY_SNAPSHOT ("database is locked"), with no busy wait.+ connection.exec_driver_sql(+ "SELECT version_num FROM alembic_version"+ ).all()+ lock_errors.append(try_write(other))+ change_schema_and_data(config, target)++ monkeypatch.setattr(runner.command, "upgrade", read_version_then_change)+ try:+ # Only WAL lets another connection commit while a reader holds an+ # older snapshot, so the race needs this mode.+ journal_mode = other.execute("PRAGMA journal_mode = WAL").fetchone()+ assert journal_mode[0].lower() == "wal"+ runner.run_migrations(engine, target="0002")+ finally:+ other.close()++ assert lock_errors == ["database is locked"]+ engine.dispose()+ assert "added_table" in inspect(engine).get_table_names()+ assert_only_migrated_row(engine)+ assert runner.get_current_revision(engine) == "0002"+ assert_foreign_keys_enabled(engine)