Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

fix(database): make SQLite migration schema changes atomic (#7099)

LearningCircuitf2139126d251
diff --git a/changelog.d/+sqlite-migration-ddl-rollback.bugfix.md b/changelog.d/+sqlite-migration-ddl-rollback.bugfix.md
new file mode 100644
index 0000000000..6f2b9beab9
--- /dev/null
+++ b/changelog.d/+sqlite-migration-ddl-rollback.bugfix.md
@@ -0,0 +1,10 @@
+Start SQLite and SQLCipher migrations with a physical `BEGIN IMMEDIATE`
+transaction before schema changes, so failed upgrades roll back tables,
+indexes and columns as well as data and the migration revision. The database
+write lock is taken as that transaction begins, so a write from another
+connection waits for the upgrade (up to the busy timeout) instead of landing
+between its schema changes. When the initial migration creates a new
+database's tables, as it does for an unencrypted database, it no longer
+commits each statement separately. A new encrypted database still creates its
+tables and indexes one committed statement at a time before its migrations
+run.
diff --git a/src/local_deep_research/database/alembic_runner.py b/src/local_deep_research/database/alembic_runner.py
index a554ed3e4c..de995edae5 100644
--- a/src/local_deep_research/database/alembic_runner.py
+++ b/src/local_deep_research/database/alembic_runner.py
@@ -272,14 +272,14 @@ def _disable_fk_for_migration(conn: Connection) -> None:
``apply_performance_pragmas`` set ``PRAGMA foreign_keys = ON`` at
connect. SQLite then *silently ignores* further toggles of
``foreign_keys`` once any transaction (explicit or driver-implicit)
- is active. The sqlite3/sqlcipher3 driver auto-begins on the first
- DML; PRAGMA itself isn't DML, so issuing the PRAGMA before any DML
- is the only window where it actually takes effect.
-
- With multi-migration upgrades (revision 0001 → 0009), the first
- migration to issue DML auto-begins the driver transaction and
- freezes FK in the connect-time ON state for the rest of the upgrade.
- That defeats migration 0007's defensive PRAGMA OFF and makes its
+ is active. ``run_migrations`` starts the physical transaction at its
+ ``conn.begin()`` boundary, before the first revision runs, so this
+ helper must run before that boundary for the toggle to take effect.
+
+ Before this helper, the first DML of a multi-migration upgrade
+ (revision 0001 → 0009) auto-began the driver transaction and froze
+ FK in the connect-time ON state for the rest of the upgrade.
+ That defeated migration 0007's defensive PRAGMA OFF and made its
orphan-scrub DELETE fail with ``foreign key mismatch`` on tables
whose FK target lacks a UNIQUE backing — exactly the broken-schema
state migration 0007 is meant to repair.
@@ -390,11 +390,14 @@ def run_migrations(engine: Engine, target: str = "head") -> None:
databases. This ensures any missing tables are created.
When ``target == "head"`` and the database is already at head, the call
- short-circuits without opening a write transaction — calling
- ``command.upgrade()`` unconditionally would still open a write transaction
- via ``engine.begin()`` (taking a RESERVED lock on the SQLite file) just to
- discover there's nothing to apply, serialising concurrent readers behind
- a no-op on every cold engine reopen.
+ short-circuits before the migration transaction opens. On SQLite that
+ transaction starts with ``BEGIN IMMEDIATE`` (unless a begin-event hook
+ already started one), so running it anyway would take the write lock
+ just to discover there's nothing to apply, making other writers wait
+ behind a no-op on every cold engine reopen. Any other ``target``, even
+ the head revision's own ID, is not resolved up front, so the
+ transaction opens and ``command.upgrade()`` runs even when the database
+ is already at that revision.
Security validations performed before running migrations:
- Migration directory path is within expected package boundary
@@ -407,10 +410,17 @@ def run_migrations(engine: Engine, target: str = "head") -> None:
(issue #3990). Re-enabled after a successful upgrade, before the
connection returns to the pool.
+ Inside the migration transaction, before Alembic runs, a database at
+ revision 0001, 0002 or 0003 gains ``settings.env_var`` if its settings
+ table lacks it (see ``_add_settings_env_var_before_0004``), so a failed
+ upgrade rolls that column back too.
+
On failure inside the migration transaction, the inner
``conn.begin()`` rolls back automatically — the database stays at
- its previous revision. The original exception is re-raised so
- callers can decide how to handle it.
+ its previous revision. On SQLite the physical transaction starts
+ before the first revision runs, so the rollback covers schema changes
+ as well as data. The original exception is re-raised so callers can
+ decide how to handle it.
Args:
engine: SQLAlchemy engine to migrate
@@ -488,13 +498,12 @@ def run_migrations(engine: Engine, target: str = "head") -> None:
"proceeding with upgrade to head"
)
- # Short-circuit when the database is already at head. Calling
- # command.upgrade() unconditionally opens a write transaction via
- # engine.begin() even when there is nothing to apply — SQLite takes
- # a RESERVED lock on the file as soon as the first DML lands inside
- # that transaction, serialising concurrent readers behind a no-op on
- # every cold engine reopen. The fresh-DB path (current is None) still
- # runs the upgrade so tables and the alembic_version row get created.
+ # Short-circuit when the database is already at head. The migration
+ # transaction below starts with BEGIN IMMEDIATE, which takes SQLite's
+ # write lock even when there is nothing to apply, making other writers
+ # wait behind a no-op on every cold engine reopen. The fresh-DB path
+ # (current is None) still runs the upgrade so tables and the
+ # alembic_version row get created.
if current is not None and current == head and target == "head":
logger.info(f"Database already at revision {head}; skipping upgrade")
return
@@ -520,6 +529,23 @@ def run_migrations(engine: Engine, target: str = "head") -> None:
# raise after SQLite has already accepted PRAGMA ... = OFF.
_disable_fk_for_migration(conn)
with conn.begin():
+ # SQLite's legacy driver begins only on DML. Start its
+ # physical transaction before DDL so schema changes roll
+ # back too. IMMEDIATE takes the write lock now, so
+ # busy_timeout covers waiting for another writer. With a
+ # DEFERRED BEGIN, Alembic's first reads would pin a WAL
+ # snapshot, and a commit from another connection before
+ # the first write would fail that write at once with
+ # SQLITE_BUSY_SNAPSHOT, which busy_timeout never retries.
+ # A begin-event hook may already have started the
+ # transaction; its mode is kept.
+ if (
+ conn.dialect.name == "sqlite"
+ and not conn.connection.driver_connection.in_transaction
+ ):
+ conn.exec_driver_sql("BEGIN IMMEDIATE")
+ # After the BEGIN, so a failed upgrade also rolls back
+ # this ADD COLUMN instead of leaving it committed.
_add_settings_env_var_before_0004(conn, current)
config.attributes["connection"] = conn
command.upgrade(config, target)
@@ -542,7 +568,7 @@ def run_migrations(engine: Engine, target: str = "head") -> None:
except Exception:
logger.exception(
"Database migration failed — transaction rolled back where "
- "supported; SQLite DDL from a failed revision may require repair"
+ "supported"
)
raise
diff --git a/src/local_deep_research/database/migrations/env.py b/src/local_deep_research/database/migrations/env.py
index cb6c1e2cd9..1b2e53e1cd 100644
--- a/src/local_deep_research/database/migrations/env.py
+++ b/src/local_deep_research/database/migrations/env.py
@@ -30,7 +30,11 @@ def run_migrations_online():
Run migrations in 'online' mode using the provided connection.
The connection is passed via config.attributes["connection"] by alembic_runner.py,
- which opens it inside engine.begin() so that failures auto-rollback.
+ which runs the upgrade inside conn.begin() with SQLite's physical
+ transaction already open (see run_migrations), so a failure rolls back
+ schema changes as well as data. Because conn.begin() is already active,
+ context.begin_transaction() below is a no-op and every revision runs in
+ that one transaction.
"""
connection = config.attributes.get("connection")
diff --git a/src/local_deep_research/database/migrations/versions/0006_journal_quality_system.py b/src/local_deep_research/database/migrations/versions/0006_journal_quality_system.py
index 3e7048db15..ccb92d5ea9 100644
--- a/src/local_deep_research/database/migrations/versions/0006_journal_quality_system.py
+++ b/src/local_deep_research/database/migrations/versions/0006_journal_quality_system.py
@@ -256,10 +256,11 @@ def upgrade() -> None:
#
# Concurrency note: BEGIN EXCLUSIVE was considered here and
# rejected — it raises "cannot start a transaction within a
- # transaction" inside the outer engine.begin() that
- # alembic_runner.py:268 opens. In practice, SQLite WAL +
- # _connections_lock (encrypted_db.py) + the current==head
- # short-circuit in alembic_runner serialize access sufficiently.
+ # transaction" inside the migration transaction that
+ # alembic_runner.run_migrations opens before the first revision.
+ # The runner starts that transaction with BEGIN IMMEDIATE, so
+ # SQLite's write lock is already held and no other connection can
+ # write between this read and the delete below.
journals_pre = sa.Table("journals", sa.MetaData(), autoload_with=bind)
all_rows = bind.execute(
sa.select(
diff --git a/src/local_deep_research/database/migrations/versions/0007_backfill_missing_indexes.py b/src/local_deep_research/database/migrations/versions/0007_backfill_missing_indexes.py
index 78c0b6814e..e617316702 100644
--- a/src/local_deep_research/database/migrations/versions/0007_backfill_missing_indexes.py
+++ b/src/local_deep_research/database/migrations/versions/0007_backfill_missing_indexes.py
@@ -157,14 +157,13 @@ def upgrade() -> None:
# The actual FK toggle that takes effect is in
# ``alembic_runner._disable_fk_for_migration``, run BEFORE the migration
# transaction opens — SQLite silently ignores ``PRAGMA foreign_keys``
- # once any transaction is active, and earlier migrations in this same
- # upgrade chain may have already issued DML, auto-beginning the driver
- # transaction. Issuing the PRAGMA again here is a defensive belt-and-
- # suspenders no-op when the runner has already disabled FK; it preserves
- # the migration's contract for any caller that runs it outside the
- # standard ``run_migrations`` path. The runner re-enables FK on the
- # same connection after the upgrade commits, before returning the
- # connection to the pool.
+ # once any transaction is active, and the runner starts the migration
+ # transaction before the first revision runs. Under the runner this
+ # PRAGMA is therefore always ignored, and FK is already off; it
+ # preserves the migration's contract for a caller that runs it outside
+ # ``run_migrations`` on a connection with no transaction open yet. The
+ # runner re-enables FK on the same connection after the upgrade
+ # commits, before returning the connection to the pool.
bind.execute(text("PRAGMA foreign_keys = OFF"))
_scrub_download_tracker_duplicates(bind)
diff --git a/src/local_deep_research/database/migrations/versions/0010_add_chat_tables.py b/src/local_deep_research/database/migrations/versions/0010_add_chat_tables.py
index d9da47fc09..f8e0abb520 100644
--- a/src/local_deep_research/database/migrations/versions/0010_add_chat_tables.py
+++ b/src/local_deep_research/database/migrations/versions/0010_add_chat_tables.py
@@ -255,9 +255,10 @@ def upgrade():
inspector = inspect(bind)
# Validate any pre-existing FK before creating chat tables or indexes.
- # SQLite DDL is not transactionally rolled back by Alembic, so discovering
- # an incompatible constraint later would otherwise leave a partially
- # upgraded schema even though the revision remains at 0009.
+ # Under run_migrations a failure also rolls back this revision's DDL, but
+ # a caller that runs it on a connection with no transaction open would
+ # otherwise keep any DDL issued before a late validation failure, leaving
+ # a partially upgraded schema even though the revision remains at 0009.
research_history_exists = inspector.has_table("research_history")
chat_session_fk_exists = False
if research_history_exists:
diff --git a/src/local_deep_research/database/migrations/versions/0031_reconcile_corrupted_settings.py b/src/local_deep_research/database/migrations/versions/0031_reconcile_corrupted_settings.py
index db3422da61..5e31fdb71e 100644
--- a/src/local_deep_research/database/migrations/versions/0031_reconcile_corrupted_settings.py
+++ b/src/local_deep_research/database/migrations/versions/0031_reconcile_corrupted_settings.py
@@ -406,8 +406,9 @@ def _repair_corrupted_values(
if sensitive_keys is None:
sensitive_keys = _sensitive_setting_keys(conn)
# SQLite's legacy driver starts a transaction for DML, not SELECT or
- # index DDL. Start it even when every repair target is sensitive and
- # the loop below skips all updates. The false predicate changes no row.
+ # index DDL. run_migrations has already started one; for a caller that
+ # has not, start it even when every repair target is sensitive and the
+ # loop below skips all updates. The false predicate changes no row.
conn.execute(sa.update(_settings).where(sa.false()).values(value=None))
repaired_count = 0
# Normalize equivalent SQLite TEXT/BLOB storage classes for exact matches.
@@ -444,10 +445,13 @@ def upgrade() -> None:
# Retain that classification so repair cannot overwrite a newer clear.
sensitive_keys = _sensitive_setting_keys(conn)
duplicate_count = _deduplicate_settings(conn)
- # Keep index DDL after a guaranteed DML statement. SQLite/SQLCipher legacy
- # transaction control does not emit a driver-level BEGIN for SELECT or DDL,
- # while repair explicitly issues a zero-row UPDATE even if every target
- # is skipped. This makes index replacement participate in Alembic's rollback.
+ # Keep index DDL after a guaranteed DML statement. run_migrations has
+ # already started the driver transaction. For a caller that has not, on a
+ # connection in SQLite/SQLCipher's default deferred mode, legacy
+ # transaction control emits no driver-level BEGIN for SELECT or DDL, but
+ # the zero-row UPDATE in repair begins one even if every target is
+ # skipped, so index replacement still rolls back with the enclosing
+ # transaction.
repaired_count = _repair_corrupted_values(conn, sensitive_keys)
_ensure_unique_settings_key_index(conn)
diff --git a/src/local_deep_research/database/migrations/versions/0032_repair_pre_alembic_databases.py b/src/local_deep_research/database/migrations/versions/0032_repair_pre_alembic_databases.py
index 8e287a69ea..58af487e38 100644
--- a/src/local_deep_research/database/migrations/versions/0032_repair_pre_alembic_databases.py
+++ b/src/local_deep_research/database/migrations/versions/0032_repair_pre_alembic_databases.py
@@ -47,9 +47,11 @@ without a NULL fallback:
``none``.
Each backfill only fills NULL rows and runs whenever the column exists, not
-only when this step adds it. On SQLite and SQLCipher an ADD COLUMN issued
-before any DML commits on its own, so a failed upgrade can leave the column
-added but not backfilled; the retry then fills it.
+only when this step adds it. The migration runner now rolls back a failed
+upgrade's ADD COLUMN with the rest of the upgrade, but before #7099 an ADD
+COLUMN issued before any DML committed on its own on SQLite and SQLCipher.
+An upgrade that failed under that runner can have left the column added but
+not backfilled; the retry then fills it.
3. Rewrite ``settings.type`` values to SettingType member names.
@@ -212,9 +214,9 @@ def upgrade() -> None:
op.add_column(
table, sa.Column(name, column_type, nullable=True)
)
- # Also when the column already existed: an earlier attempt may
- # have added it (the DDL commits on its own) and then failed
- # before its backfill committed.
+ # Also when the column already existed: an attempt before
+ # #7099, whose runner committed this DDL on its own, may have
+ # added it and then failed before its backfill committed.
backfill = _BACKFILLS.get((table, name))
if backfill is not None:
backfill(conn, existing)
diff --git a/tests/database/test_alembic_migrations.py b/tests/database/test_alembic_migrations.py
index 0b37afa4f1..dd083406d8 100644
--- a/tests/database/test_alembic_migrations.py
+++ b/tests/database/test_alembic_migrations.py
@@ -223,9 +223,9 @@ class TestAlembicRunner:
self, fresh_engine, loguru_caplog
):
"""When the database is already at head, run_migrations must not
- call command.upgrade() — that would open a no-op write transaction
- and hold a RESERVED lock under isolation_level="IMMEDIATE", blocking
- concurrent readers for no benefit.
+ call command.upgrade() — the migration transaction starts with
+ BEGIN IMMEDIATE, so a no-op upgrade would still take SQLite's write
+ lock and make other writers wait for no benefit.
"""
# Migrate to head first
run_migrations(fresh_engine)
@@ -3840,17 +3840,18 @@ class TestCryptographicConsistency:
def _insert_witness_then_fail(config, target):
"""side_effect for patched ``command.upgrade``.
- Inserts a witness row through the transaction's connection, then
- raises. If ``engine.begin()``'s rollback works, the row must not
- be present after the raise — this is what makes the rollback
- tests non-tautological. A plain ``side_effect=RuntimeError(...)``
- fires before any DB write, so the post-failure "revision
- preserved" assertion would be trivially true.
-
- Uses DML (INSERT) rather than DDL (CREATE TABLE) because pysqlite
- does not reliably roll back DDL statements on default SQLite. The
- caller must pre-create a ``rollback_witness`` table before
- patching ``command.upgrade`` with this side_effect.
+ Inserts a witness row through the migration transaction's
+ connection, then raises. If the runner's rollback works, the row
+ must not be present after the raise — this is what makes the
+ rollback tests non-tautological. A plain
+ ``side_effect=RuntimeError(...)`` fires before any DB write, so the
+ post-failure "revision preserved" assertion would be trivially true.
+
+ The witness is a data row. The runner starts SQLite's physical
+ transaction before ``command.upgrade`` runs, so DDL rolls back as
+ well; ``test_migration_ddl_rollback.py`` covers that. The caller
+ must pre-create a ``rollback_witness`` table before patching
+ ``command.upgrade`` with this side_effect.
"""
conn = config.attributes["connection"]
conn.execute(text("INSERT INTO rollback_witness (x) VALUES (1)"))
@@ -3962,8 +3963,8 @@ class TestMigrationErrorSanitization:
try:
# Pre-create the witness table outside the transaction the
- # patched call will run in. DML (INSERT) rollback is
- # reliable on default SQLite; DDL (CREATE TABLE) is not.
+ # patched call will run in, so only the witness row is inside
+ # it.
with engine.begin() as conn:
conn.execute(text("CREATE TABLE rollback_witness (x INTEGER)"))
@@ -4558,8 +4559,10 @@ class TestPreAlembicDatabaseUpgrade:
):
"""A retry must backfill columns that already exist but hold NULL.
- On SQLite an ADD COLUMN issued before any DML commits on its own, so
- a failed 0032 can leave the column added but not backfilled.
+ Before #7099 the runner committed an ADD COLUMN issued before any
+ DML on its own on SQLite, so a failed 0032 could leave the column
+ added but not backfilled. The runner now rolls that column back
+ with the failed upgrade, so this test builds the state by hand.
"""
engine = create_engine(f"sqlite:///{tmp_path / 'interrupted.db'}")
try:
@@ -5328,23 +5331,25 @@ class TestUpgradeFromBuggyV16xUserDbProductionEngine:
"""Regression for #3990 — multi-migration upgrade fails with FK mismatch.
The existing ``TestUpgradeFromBuggyV16xUserDatabase`` covers the same
- schema corruption but stamps at revision 0006, so migration 0007 is the
- first to run any DML in the upgrade transaction — which lets 0007's
- own ``PRAGMA foreign_keys = OFF`` actually take effect (no auto-begun
+ schema corruption but stamps at revision 0006. Before the runner fix,
+ migration 0007 was then the first to run any DML in the upgrade, which
+ let 0007's own ``PRAGMA foreign_keys = OFF`` take effect (no auto-begun
transaction yet).
Real production users (issue #3990) upgrade from 0001 to head. Migrations
- 0002–0006 run DML before 0007, auto-beginning the sqlite3 driver
+ 0002–0006 ran DML before 0007, auto-beginning the sqlite3 driver
transaction. ``PRAGMA foreign_keys`` is silently a no-op once a
transaction is active (per sqlite.org/pragma.html#pragma_foreign_keys),
- so 0007's defensive PRAGMA never lands and the orphan-scrub DELETE
- fails with ``foreign key mismatch``.
+ so 0007's defensive PRAGMA never landed and the orphan-scrub DELETE
+ failed with ``foreign key mismatch``.
The fix (in ``alembic_runner.run_migrations``) issues PRAGMA OFF
- *before* opening the migration transaction. This test reproduces the
- production failure exactly: ``isolation_level=""`` (matching the
- sqlcipher3 engine in ``encrypted_db.py``) + FK ON at connect via the
- same event handler ``apply_performance_pragmas`` installs.
+ *before* opening the migration transaction. The runner now also starts
+ that transaction before the first revision runs, so 0007's own PRAGMA
+ is ignored in both classes. This test reproduces the production setup
+ exactly: ``isolation_level=""`` (matching the sqlcipher3 engine in
+ ``encrypted_db.py``) + FK ON at connect via the same event handler
+ ``apply_performance_pragmas`` installs.
"""
@pytest.fixture
@@ -5353,9 +5358,10 @@ class TestUpgradeFromBuggyV16xUserDbProductionEngine:
connect, with the buggy v1.6.x schema stamped at revision 0005.
Stamping at 0005 (not 0006) means migration 0006's data backfill
- (``UPDATE journals SET name_lower = ...``) runs DML before 0007,
- auto-beginning the driver transaction and freezing FK in the
- connect-time ON state for the rest of the upgrade.
+ (``UPDATE journals SET name_lower = ...``) runs DML before 0007, as
+ in the #3990 reports, where that DML auto-began the driver
+ transaction and froze FK in the connect-time ON state for the rest
+ of the upgrade.
"""
import sqlite3
diff --git a/tests/database/test_migration_ddl_rollback.py b/tests/database/test_migration_ddl_rollback.py
new file mode 100644
index 0000000000..e47db8fc6e
--- /dev/null
+++ b/tests/database/test_migration_ddl_rollback.py
@@ -0,0 +1,438 @@
+"""Migrations must hold SQLite's write lock, let readers read, and roll back
+schema changes."""
+
+import sqlite3
+
+import pytest
+from sqlalchemy import create_engine, event, inspect
+
+from local_deep_research.database import alembic_runner as runner
+from local_deep_research.database.models import Base
+from local_deep_research.database.sqlcipher_utils import (
+ create_sqlcipher_connection,
+)
+
+
[email protected](params=["sqlite", "sqlcipher"])
+def migration_database(request, tmp_path):
+ """Yield an engine and a function that opens an independent connection.
+
+ The SQLCipher engine pools a single connection per thread, so a second
+ connection must come from ``connect`` rather than from the engine.
+ """
+ database = tmp_path / "migration.db"
+ if request.param == "sqlcipher":
+ pytest.importorskip("sqlcipher3", reason="SQLCipher driver unavailable")
+
+ def connect():
+ return create_sqlcipher_connection(
+ database,
+ hex_key="ab" * 32,
+ creation_mode=not database.exists()
+ or database.stat().st_size == 0,
+ )
+
+ engine = create_engine("sqlite://", creator=connect)
+ else:
+
+ def connect():
+ return sqlite3.connect(database)
+
+ engine = create_engine(f"sqlite:///{database}")
+
+ @event.listens_for(engine, "connect")
+ def enable_foreign_keys(dbapi_connection, connection_record):
+ cursor = dbapi_connection.cursor()
+ cursor.execute("PRAGMA foreign_keys = ON")
+ cursor.close()
+
+ yield engine, connect
+ engine.dispose()
+
+
+def migration_engine(migration_database):
+ engine, _connect = migration_database
+ return engine
+
+
+def open_connection_that_never_waits(connect):
+ """Open another connection whose statements fail at once on a held lock."""
+ connection = connect()
+ connection.execute("PRAGMA busy_timeout = 0")
+ return connection
+
+
+def try_write(connection):
+ """Insert and commit a row; return SQLite's lock error if refused."""
+ try:
+ connection.execute(
+ "INSERT INTO stored_values (value) VALUES (?)", ("other writer",)
+ )
+ connection.commit()
+ except connection.OperationalError as error:
+ connection.rollback()
+ return str(error)
+ return None
+
+
+def try_read(connection):
+ """Return the committed values; return SQLite's lock error if refused."""
+ try:
+ rows = connection.execute("SELECT value FROM stored_values").fetchall()
+ except connection.OperationalError as error:
+ return str(error)
+ return [value for (value,) in rows]
+
+
+def assert_foreign_keys_enabled(engine):
+ with engine.connect() as connection:
+ assert (
+ connection.exec_driver_sql("PRAGMA foreign_keys").scalar_one() == 1
+ )
+
+
+def seed_previous_revision(engine):
+ with engine.begin() as connection:
+ connection.exec_driver_sql(
+ "CREATE TABLE stored_values (id INTEGER PRIMARY KEY, value TEXT)"
+ )
+ connection.exec_driver_sql(
+ "INSERT INTO stored_values (id, value) VALUES (?, ?)",
+ (1, "original"),
+ )
+ connection.exec_driver_sql(
+ "CREATE TABLE alembic_version (version_num VARCHAR(32) PRIMARY KEY)"
+ )
+ connection.exec_driver_sql(
+ "INSERT INTO alembic_version (version_num) VALUES (?)", ("0001",)
+ )
+
+
+def change_schema_and_data(config, target):
+ assert target == "0002"
+ connection = config.attributes["connection"]
+ # DDL precedes DML, as it does in real schema migrations. A driver that
+ # waits until the UPDATE to begin cannot roll back these schema changes.
+ connection.exec_driver_sql(
+ "ALTER TABLE stored_values ADD COLUMN extra TEXT"
+ )
+ connection.exec_driver_sql(
+ "CREATE INDEX stored_value_index ON stored_values (value)"
+ )
+ connection.exec_driver_sql(
+ "CREATE TABLE added_table (id INTEGER PRIMARY KEY)"
+ )
+ connection.exec_driver_sql(
+ "UPDATE stored_values SET value = ?", ("changed",)
+ )
+ connection.exec_driver_sql(
+ "UPDATE alembic_version SET version_num = ?", ("0002",)
+ )
+
+
+def test_failed_fresh_migration_leaves_no_schema(migration_engine, monkeypatch):
+ def fail_after_schema(config, target):
+ connection = config.attributes["connection"]
+ connection.exec_driver_sql(
+ "CREATE TABLE added_table (id INTEGER PRIMARY KEY)"
+ )
+ connection.exec_driver_sql(
+ "CREATE INDEX added_index ON added_table (id)"
+ )
+ raise RuntimeError("injected migration failure")
+
+ monkeypatch.setattr(runner.command, "upgrade", fail_after_schema)
+ with pytest.raises(RuntimeError, match="injected migration failure"):
+ runner.run_migrations(migration_engine)
+
+ migration_engine.dispose()
+ assert inspect(migration_engine).get_table_names() == []
+ assert runner.get_current_revision(migration_engine) is None
+ assert_foreign_keys_enabled(migration_engine)
+
+
[email protected]("failure", [RuntimeError, KeyboardInterrupt])
+def test_failed_upgrade_preserves_schema_data_and_revision(
+ migration_engine, monkeypatch, failure
+):
+ seed_previous_revision(migration_engine)
+
+ def fail_after_changes(config, target):
+ change_schema_and_data(config, target)
+ raise failure("injected migration failure")
+
+ monkeypatch.setattr(runner.command, "upgrade", fail_after_changes)
+ with pytest.raises(failure, match="injected migration failure"):
+ runner.run_migrations(migration_engine, target="0002")
+
+ migration_engine.dispose()
+ inspector = inspect(migration_engine)
+ assert set(inspector.get_table_names()) == {
+ "stored_values",
+ "alembic_version",
+ }
+ assert {
+ column["name"] for column in inspector.get_columns("stored_values")
+ } == {
+ "id",
+ "value",
+ }
+ assert inspector.get_indexes("stored_values") == []
+ with migration_engine.connect() as connection:
+ assert (
+ connection.exec_driver_sql(
+ "SELECT value FROM stored_values"
+ ).scalar_one()
+ == "original"
+ )
+ assert runner.get_current_revision(migration_engine) == "0001"
+ assert_foreign_keys_enabled(migration_engine)
+
+
+def test_successful_upgrade_commits_schema_data_and_revision(
+ migration_engine, monkeypatch
+):
+ seed_previous_revision(migration_engine)
+ monkeypatch.setattr(runner.command, "upgrade", change_schema_and_data)
+ runner.run_migrations(migration_engine, target="0002")
+
+ migration_engine.dispose()
+ inspector = inspect(migration_engine)
+ assert "added_table" in inspector.get_table_names()
+ assert "extra" in {
+ column["name"] for column in inspector.get_columns("stored_values")
+ }
+ assert {
+ index["name"] for index in inspector.get_indexes("stored_values")
+ } == {"stored_value_index"}
+ with migration_engine.connect() as connection:
+ assert (
+ connection.exec_driver_sql(
+ "SELECT value FROM stored_values"
+ ).scalar_one()
+ == "changed"
+ )
+ assert runner.get_current_revision(migration_engine) == "0002"
+ assert_foreign_keys_enabled(migration_engine)
+
+
+def test_real_initial_migration_rolls_back_created_models(
+ migration_engine, monkeypatch
+):
+ create_all = Base.metadata.create_all
+
+ def fail_after_models(*args, **kwargs):
+ create_all(*args, **kwargs)
+ raise RuntimeError("injected migration failure")
+
+ monkeypatch.setattr(Base.metadata, "create_all", fail_after_models)
+ with pytest.raises(RuntimeError, match="injected migration failure"):
+ runner.run_migrations(migration_engine)
+
+ migration_engine.dispose()
+ assert inspect(migration_engine).get_table_names() == []
+ assert runner.get_current_revision(migration_engine) is None
+ assert_foreign_keys_enabled(migration_engine)
+
+
+def settings_columns(bind):
+ return {column["name"] for column in inspect(bind).get_columns("settings")}
+
+
+def test_failed_upgrade_rolls_back_env_var_added_before_0004(
+ migration_engine, monkeypatch
+):
+ """The settings.env_var column added for revision 0004 rolls back too.
+
+ The runner adds it to a database below 0004 before Alembic runs. Added
+ before the BEGIN, that ADD COLUMN would commit on its own and survive
+ a failure in a later revision.
+ """
+ runner.run_migrations(migration_engine, target="0003")
+ with migration_engine.begin() as connection:
+ connection.exec_driver_sql("ALTER TABLE settings DROP COLUMN env_var")
+ # 0004 names env_var when it re-scopes this legacy key.
+ connection.exec_driver_sql(
+ "INSERT INTO settings (key, value, type, name, ui_element, "
+ "visible, editable) VALUES ('app.iterations', '3', 'APP', "
+ "'Legacy', 'text', 1, 1)"
+ )
+ upgrade = runner.command.upgrade
+ during_upgrade = []
+
+ def fail_after_later_revisions(config, target):
+ upgrade(config, target)
+ connection = config.attributes["connection"]
+ during_upgrade.append(
+ (
+ "env_var" in settings_columns(connection),
+ connection.exec_driver_sql(
+ "SELECT version_num FROM alembic_version"
+ ).scalar_one(),
+ )
+ )
+ raise RuntimeError("injected migration failure")
+
+ monkeypatch.setattr(runner.command, "upgrade", fail_after_later_revisions)
+ with pytest.raises(RuntimeError, match="injected migration failure"):
+ runner.run_migrations(migration_engine)
+
+ # 0004 and every later revision ran on the added column, then failed.
+ assert during_upgrade == [(True, runner.get_head_revision())]
+ migration_engine.dispose()
+ assert "env_var" not in settings_columns(migration_engine)
+ assert runner.get_current_revision(migration_engine) == "0003"
+ with migration_engine.connect() as connection:
+ keys = set(
+ connection.exec_driver_sql("SELECT key FROM settings").scalars()
+ )
+ assert "app.iterations" in keys
+ assert "search.iterations" not in keys
+ assert_foreign_keys_enabled(migration_engine)
+
+
+def test_migration_supports_transaction_started_by_begin_event(
+ migration_engine, monkeypatch
+):
+ disable_fk = runner._disable_fk_for_migration
+ starts = []
+
+ def prepare_connection(connection):
+ disable_fk(connection)
+
+ def begin_transaction(active_connection):
+ active_connection.exec_driver_sql("BEGIN")
+ starts.append(True)
+
+ # Start a physical transaction at the migration's existing begin()
+ # boundary, after the foreign-key pragma has been changed.
+ event.listen(connection, "begin", begin_transaction, once=True)
+
+ def create_schema(config, target):
+ config.attributes["connection"].exec_driver_sql(
+ "CREATE TABLE added_table (id INTEGER PRIMARY KEY)"
+ )
+
+ monkeypatch.setattr(runner, "_disable_fk_for_migration", prepare_connection)
+ monkeypatch.setattr(runner.command, "upgrade", create_schema)
+ runner.run_migrations(migration_engine)
+
+ assert starts == [True]
+ migration_engine.dispose()
+ assert inspect(migration_engine).get_table_names() == ["added_table"]
+ assert_foreign_keys_enabled(migration_engine)
+
+
+def assert_only_migrated_row(engine):
+ with engine.connect() as connection:
+ assert connection.exec_driver_sql(
+ "SELECT value FROM stored_values"
+ ).scalars().all() == ["changed"]
+
+
+def test_migration_holds_write_lock_before_upgrade_runs(
+ migration_database, monkeypatch
+):
+ engine, connect = migration_database
+ seed_previous_revision(engine)
+ other = open_connection_that_never_waits(connect)
+ lock_errors = []
+
+ def write_from_other_connection_then_change(config, target):
+ # BEGIN IMMEDIATE took the write lock before any migration statement.
+ lock_errors.append(try_write(other))
+ change_schema_and_data(config, target)
+
+ monkeypatch.setattr(
+ runner.command, "upgrade", write_from_other_connection_then_change
+ )
+ try:
+ runner.run_migrations(engine, target="0002")
+ finally:
+ other.close()
+
+ assert lock_errors == ["database is locked"]
+ engine.dispose()
+ assert_only_migrated_row(engine)
+ assert runner.get_current_revision(engine) == "0002"
+ assert_foreign_keys_enabled(engine)
+
+
+def test_reader_can_read_while_migration_holds_write_lock(
+ migration_database, monkeypatch
+):
+ """BEGIN IMMEDIATE locks out other writers but not readers.
+
+ In WAL mode readers do not wait for a writer, and BEGIN EXCLUSIVE acts
+ like IMMEDIATE. With a rollback journal, BEGIN EXCLUSIVE would also lock
+ readers out until the upgrade commits, so this test uses one.
+ """
+ monkeypatch.setenv("LDR_DB_CONFIG_JOURNAL_MODE", "DELETE")
+ engine, connect = migration_database
+ seed_previous_revision(engine)
+ other = open_connection_that_never_waits(connect)
+ lock_errors = []
+ reads = []
+
+ def read_from_other_connection_during_upgrade(config, target):
+ lock_errors.append(try_write(other))
+ change_schema_and_data(config, target)
+ # Nothing is committed yet, so the reader sees the old row.
+ reads.append(try_read(other))
+
+ monkeypatch.setattr(
+ runner.command, "upgrade", read_from_other_connection_during_upgrade
+ )
+ try:
+ journal_mode = other.execute("PRAGMA journal_mode").fetchone()
+ assert journal_mode[0].lower() == "delete"
+ runner.run_migrations(engine, target="0002")
+ finally:
+ other.close()
+
+ assert lock_errors == ["database is locked"]
+ assert reads == [["original"]]
+ engine.dispose()
+ assert_only_migrated_row(engine)
+ assert runner.get_current_revision(engine) == "0002"
+ assert_foreign_keys_enabled(engine)
+
+
+def test_commit_after_version_read_does_not_fail_migration(
+ migration_database, monkeypatch
+):
+ engine, connect = migration_database
+ seed_previous_revision(engine)
+ engine.dispose()
+ other = open_connection_that_never_waits(connect)
+ lock_errors = []
+
+ def read_version_then_change(config, target):
+ connection = config.attributes["connection"]
+ # Alembic reads the version table before the first schema change. A
+ # DEFERRED BEGIN would pin a WAL snapshot here, and a commit from
+ # another connection would then fail the first write at once with
+ # SQLITE_BUSY_SNAPSHOT ("database is locked"), with no busy wait.
+ connection.exec_driver_sql(
+ "SELECT version_num FROM alembic_version"
+ ).all()
+ lock_errors.append(try_write(other))
+ change_schema_and_data(config, target)
+
+ monkeypatch.setattr(runner.command, "upgrade", read_version_then_change)
+ try:
+ # Only WAL lets another connection commit while a reader holds an
+ # older snapshot, so the race needs this mode.
+ journal_mode = other.execute("PRAGMA journal_mode = WAL").fetchone()
+ assert journal_mode[0].lower() == "wal"
+ runner.run_migrations(engine, target="0002")
+ finally:
+ other.close()
+
+ assert lock_errors == ["database is locked"]
+ engine.dispose()
+ assert "added_table" in inspect(engine).get_table_names()
+ assert_only_migrated_row(engine)
+ assert runner.get_current_revision(engine) == "0002"
+ assert_foreign_keys_enabled(engine)