Speakr is a personal, self-hosted web application designed for transcribing audio recordings
Python
fix: serialize startup migrations behind a cross-process lock, add a one-shot migration ledger, and make migration failures loud instead of a single warning line
Murtaza Nasire3c0bc4e66c7
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex 6d82d0c7..b67f95d6 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -82,8 +82,22 @@ instead, which is what the helpers do.**Make it idempotent.** Migrations run several times per container start, some of themconcurrently: from the entrypoint, from the admin user script, and once per worker-process. A migration must recognise that its work is already done and return without-acting.+process. A `migration_lock()` keeps them from interleaving, but it fails open after a+timeout rather than holding the container down, so every migration must still recognise+that its work is already done and return without acting.++**Register genuinely one-shot work instead of re-detecting it.** A data fix that can+only ever be needed once, or whose "has this run?" check would scan a whole table on+every boot, belongs in the ledger:++```python+run_once(engine, '0002_short_description_of_the_change', my_migration, logger=app.logger)+```++It runs the callable the first time only and records the id in `schema_migrations`. A+failure is left unrecorded, so it is retried on the next startup. Number ids in+sequence and never reuse or renumber one, since existing installations have the old id+recorded.**Make failure recoverable.** pysqlite does not open a transaction for DDL, so a bare`CREATE TABLE` or `ALTER TABLE` commits immediately even inside a@@ -95,19 +109,38 @@ yourself; `drop_not_null()` shows the pattern.function, so referencing a column that is added further down fails on precisely theupgrade path the migration exists for.+**Test the upgrade, not just the result.** `tests/test_upgrade_path.py` takes real+schemas from older releases, seeds them, runs the current migrations over them, and+asserts that no column, index or value was lost and that the schema ends up matching+today's models. Every other test in the suite runs against a database built fresh from+current models, which is the one shape no upgrading user ever has.++The fixtures live in `tests/fixtures/schemas/` as plain SQL, dumped from each tag's own+models, so the tests need no old dependencies and no network. Add a version with:++```bash+python scripts/generate_schema_fixtures.py v0.9.0-alpha+```+Verify with:```bash-python -m pytest tests/test_migration_compatibility.py tests/test_migration_drop_not_null.py -q+python -m pytest -q tests/test_migration_compatibility.py tests/test_upgrade_path.py \+ tests/test_migration_drop_not_null.py tests/test_migration_infrastructure.py-# and against a real PostgreSQL, which the tests skip without a URL+# and against a real PostgreSQL, which the tests skip without onedocker compose -f docker-compose.postgres.yml up -d postgres-SPEAKR_TEST_POSTGRES_URL=postgresql://speakr:speakr@localhost:5432/speakr \- python -m pytest tests/test_migration_drop_not_null.py -q+TEST_DATABASE_URI=postgresql://speakr:speakr@localhost:5432/speakr python -m pytest -q tests/```+These run in CI, and the pre-commit hook runs them whenever you touch `src/init_db.py`,+`src/utils/database.py` or `src/models/`.+A migration needs a test asserting on what a bad one would take away, meaning the othercolumns, the indexes and the row values, not merely that the intended change happened.+Values matter as much as schema: when the #379 migration destroyed 27 columns, the+`add_column_if_not_exists()` calls further down the same function immediately put the+names back, so only the emptied values revealed the damage.### Commit Message Guidelinesdiff --git a/scripts/pre-commit b/scripts/pre-commitindex 4a60749c..1e5bc740 100755--- a/scripts/pre-commit+++ b/scripts/pre-commit@@ -1,8 +1,9 @@#!/bin/sh## Pre-commit hook:-# 1. Runs PostgreSQL migration compatibility tests when migration-related-# files are staged.+# 1. Runs the migration checks when migration-related files are staged:+# the static PostgreSQL compatibility scan, and the behavioural tests+# that upgrade real databases from older releases.# 2. Keeps the VERSION file in sync with the README version badge so the# About page in the app never lags behind a release commit.#@@ -11,7 +12,7 @@STAGED_FILES=$(git diff --cached --name-only --diff-filter=ACM)# ----------------------------------------------------------------------------# 1. Migration compatibility check+# 1. Migration checks# ---------------------------------------------------------------------------if echo "$STAGED_FILES" | grep -qE '^(src/init_db\.py|src/utils/database\.py|src/models/)'; thenecho "Migration files changed — running PostgreSQL compatibility checks..."@@ -23,6 +24,28 @@ if echo "$STAGED_FILES" | grep -qE '^(src/init_db\.py|src/utils/database\.py|srcexit 1fiecho "All migration compatibility checks passed."++ # The static scan only reads the source. These actually upgrade databases+ # created by older releases and assert nothing was lost on the way, which+ # is the check that would have caught issue #379.+ if python -c "import pytest" >/dev/null 2>&1; then+ echo "Running migration behaviour tests (upgrade path, ~20s)..."+ python -m pytest -q \+ tests/test_upgrade_path.py \+ tests/test_migration_drop_not_null.py \+ tests/test_migration_infrastructure.py+ if [ $? -ne 0 ]; then+ echo ""+ echo "Pre-commit hook FAILED: a migration test did not pass."+ echo "Upgrading from an older release loses data or does not complete."+ echo "Set TEST_DATABASE_URI to a PostgreSQL URL to include that backend."+ exit 1+ fi+ echo "All migration behaviour tests passed."+ else+ echo "pytest not installed — skipping migration behaviour tests."+ echo "CI still runs them, but installing pytest catches problems earlier."+ fifi# ---------------------------------------------------------------------------diff --git a/src/init_db.py b/src/init_db.pyindex 5932932e..2e68ee0a 100644--- a/src/init_db.py+++ b/src/init_db.py@@ -16,7 +16,15 @@ from sqlalchemy import text, inspectfrom src.database import dbfrom src.models import Recording, TranscriptChunk, SystemSetting, Userfrom src.services.embeddings import process_recording_chunks-from src.utils import add_column_if_not_exists, migrate_column_type, create_index_if_not_exists, drop_not_null+from src.utils import (+ add_column_if_not_exists,+ migrate_column_type,+ create_index_if_not_exists,+ drop_not_null,+ migration_lock,+ run_once,+)+from src.utils.database import ensure_migration_ledger# ConfigurationENABLE_INQUIRE_MODE = os.environ.get('ENABLE_INQUIRE_MODE', 'false').lower() == 'true'@@ -116,6 +124,20 @@ def initialize_database(app):except Exception as e:app.logger.warning(f"Could not enable WAL mode: {e}")+ # Only one process at a time, so the several startup processes cannot+ # interleave and see each other's half-applied schema. Failing to take the+ # lock is not fatal; every migration below is independently idempotent.+ with migration_lock(engine, logger=app.logger):+ try:+ ensure_migration_ledger(engine)+ except Exception as e:+ app.logger.warning(f"Could not create the schema_migrations ledger: {e}")++ _run_migrations(app, engine)+++def _run_migrations(app, engine):+ """Apply every schema migration. Assumes the migration lock is held."""try:# Add is_inbox column with default value of 1 (True)if add_column_if_not_exists(engine, 'recording', 'is_inbox', 'BOOLEAN DEFAULT 1'):@@ -885,11 +907,12 @@ def initialize_database(app):# One-shot migration: clean up legacy User.transcription_language values# that were stored as display names ("Français", "English") before the# account-settings input was a dropdown. Issue #256.- try:+ #+ # Recorded in the ledger rather than re-detected, because detecting it+ # means loading every user on every startup for a fix that can only ever+ # be needed once.+ def _normalize_transcription_languages(_engine):from src.utils.language import normalize_language_code- from sqlalchemy import or_- # Touch only rows where the value isn't already a valid 2-letter code,- # to keep this idempotent across restarts.stale_users = User.query.filter(User.transcription_language.isnot(None)).all()cleaned = 0for u in stale_users:@@ -903,6 +926,10 @@ def initialize_database(app):if cleaned:db.session.commit()app.logger.info(f"Normalized transcription_language for {cleaned} user(s)")++ try:+ run_once(engine, '0001_normalize_transcription_language',+ _normalize_transcription_languages, logger=app.logger)except Exception as e:db.session.rollback()app.logger.warning(f"transcription_language normalization migration skipped: {e}")@@ -964,7 +991,18 @@ def initialize_database(app):app.logger.info("Existing recordings can be migrated later using the admin API or migration script.")except Exception as e:- app.logger.error(f"Error during database migration: {e}")+ # Everything above shares one try block, so this aborts each remaining+ # migration and leaves the schema part-upgraded. That must not scroll+ # past as a single line among the ordinary startup chatter.+ app.logger.error("=" * 72)+ app.logger.error("DATABASE MIGRATION FAILED: %s", e)+ app.logger.error("The schema may be partly upgraded and later migrations were skipped.")+ app.logger.error("The application will keep starting, but expect errors until this is fixed.")+ app.logger.error("Please report this with the traceback below: "+ "https://github.com/murtaza-nasir/speakr/issues")+ app.logger.error("=" * 72)+ app.logger.exception("Migration traceback")+ app.config['MIGRATION_FAILED'] = str(e)if __name__ == '__main__':diff --git a/src/utils/__init__.py b/src/utils/__init__.pyindex 27ed5121..48dd8f1c 100644--- a/src/utils/__init__.py+++ b/src/utils/__init__.py@@ -29,7 +29,9 @@ from .database import (add_column_if_not_exists,migrate_column_type,create_index_if_not_exists,- drop_not_null+ drop_not_null,+ migration_lock,+ run_once)from .token_auth import (@@ -63,6 +65,8 @@ __all__ = ['migrate_column_type','create_index_if_not_exists','drop_not_null',+ 'migration_lock',+ 'run_once',# Token authentication'extract_token_from_request','hash_token',diff --git a/src/utils/database.py b/src/utils/database.pyindex d70f8076..c9864d71 100644--- a/src/utils/database.py+++ b/src/utils/database.py@@ -9,9 +9,149 @@ IMPORTANT: All migrations must be compatible with both SQLite and PostgreSQL.- Use create_index_if_not_exists() for index creation with proper quoting"""+import fcntl+import hashlib+import osimport re+import tempfile+import time+from contextlib import contextmanager+from sqlalchemy import inspect, text+# Arbitrary but fixed key for PostgreSQL advisory locking, derived from a name so+# it cannot collide with an application lock chosen the same way.+_PG_MIGRATION_LOCK_KEY = int.from_bytes(+ hashlib.sha256(b'speakr.startup_migrations').digest()[:4], 'big'+)+++@contextmanager+def migration_lock(engine, logger=None, timeout=120):+ """Serialise startup migrations across every process touching this database.++ `initialize_database()` runs about five times per container start: from the+ entrypoint, from the admin-user script, and once per gunicorn worker, several+ of them simultaneously. Without a lock they interleave, and a migration can+ see a half-applied schema produced by a sibling process.++ Yields True when the lock was taken and False when it timed out. Timing out+ is not fatal and the caller should proceed: migrations are individually+ idempotent, so the lock is protection against wasted work and interleaving+ rather than a correctness requirement. Failing open keeps a stuck lock from+ holding the container down.+ """+ acquired = False+ handle = None+ connection = None++ try:+ if engine.name == 'postgresql':+ connection = engine.connect()+ deadline = time.monotonic() + timeout+ while True:+ acquired = bool(connection.execute(+ text('SELECT pg_try_advisory_lock(:key)'),+ {'key': _PG_MIGRATION_LOCK_KEY},+ ).scalar())+ if acquired or time.monotonic() >= deadline:+ break+ time.sleep(0.5)+ else:+ # One lock file per database URL so unrelated databases on the same+ # host do not serialise against each other.+ digest = hashlib.sha256(str(engine.url).encode()).hexdigest()[:16]+ lock_path = os.path.join(tempfile.gettempdir(), f'speakr_migration_{digest}.lock')+ handle = open(lock_path, 'w')+ deadline = time.monotonic() + timeout+ while True:+ try:+ fcntl.flock(handle.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)+ acquired = True+ except BlockingIOError:+ acquired = False+ if acquired or time.monotonic() >= deadline:+ break+ time.sleep(0.5)++ if not acquired and logger:+ logger.warning(+ "Could not acquire the migration lock within %ss; proceeding anyway. "+ "Migrations are idempotent, so this is safe but may duplicate work.",+ timeout,+ )++ yield acquired++ finally:+ try:+ if connection is not None:+ if acquired:+ connection.execute(+ text('SELECT pg_advisory_unlock(:key)'),+ {'key': _PG_MIGRATION_LOCK_KEY},+ )+ connection.close()+ if handle is not None:+ if acquired:+ fcntl.flock(handle.fileno(), fcntl.LOCK_UN)+ handle.close()+ except Exception:+ # Both lock types are released by the OS or the connection closing,+ # so a failure here must not mask whatever the caller was doing.+ pass+++def ensure_migration_ledger(engine):+ """Create the table recording which one-shot migrations have already run."""+ with engine.connect() as conn:+ conn.execute(text(+ 'CREATE TABLE IF NOT EXISTS schema_migrations ('+ ' migration_id VARCHAR(255) NOT NULL PRIMARY KEY,'+ ' applied_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP'+ ')'+ ))+ conn.commit()+++def run_once(engine, migration_id, migration, logger=None):+ """Run a one-shot migration the first time only, then never again.++ Most migrations here detect their own state, which works but means every+ startup pays to re-check, and the detection is itself a source of bugs when+ the schema drifts underneath it. A migration that cannot cheaply tell+ whether it has run, or that would otherwise scan a whole table on every+ boot, should be registered here instead.++ The callable is passed the engine and runs before the id is recorded, so a+ failure leaves the migration unrecorded and it is retried next startup.++ Returns True if the migration ran on this call, False if it had already run.+ """+ ensure_migration_ledger(engine)++ with engine.connect() as conn:+ already_applied = conn.execute(+ text('SELECT 1 FROM schema_migrations WHERE migration_id = :id'),+ {'id': migration_id},+ ).scalar()++ if already_applied:+ return False++ migration(engine)++ with engine.connect() as conn:+ conn.execute(+ text('INSERT INTO schema_migrations (migration_id) VALUES (:id)'),+ {'id': migration_id},+ )+ conn.commit()++ if logger:+ logger.info("Applied one-shot migration '%s'", migration_id)+ return True+def add_column_if_not_exists(engine, table_name, column_name, column_type):"""diff --git a/tests/test_migration_drop_not_null.py b/tests/test_migration_drop_not_null.pyindex 8ddaf839..ecd5967b 100644--- a/tests/test_migration_drop_not_null.py+++ b/tests/test_migration_drop_not_null.py@@ -9,8 +9,9 @@ the frozen DDL had fallen behind on.The replacement swaps a single column, so the tests below assert on what therebuild destroyed: the other columns, the indexes and the row data.-Set SPEAKR_TEST_POSTGRES_URL to also exercise the PostgreSQL path, e.g.- SPEAKR_TEST_POSTGRES_URL=postgresql://speakr:speakr@localhost:5433/speakr+Set TEST_DATABASE_URI to a PostgreSQL URL to also exercise that path, e.g.+ TEST_DATABASE_URI=postgresql://speakr:speakr@localhost:5432/speakr+CI sets it already, so these run there without further configuration."""import os@@ -225,10 +226,11 @@ class TestSQLiteIssue379:assert "user_new" not in inspect(sqlite_engine).get_table_names()-POSTGRES_URL = os.environ.get("SPEAKR_TEST_POSTGRES_URL")+_TEST_DB_URI = os.environ.get("TEST_DATABASE_URI", "")+POSTGRES_URL = _TEST_DB_URI if _TEST_DB_URI.startswith("postgresql") else None[email protected](not POSTGRES_URL, reason="SPEAKR_TEST_POSTGRES_URL is not set")[email protected](not POSTGRES_URL, reason="TEST_DATABASE_URI is not a PostgreSQL URL")class TestPostgreSQL:"""The production deployment runs PostgreSQL, so the native path needs cover too."""diff --git a/tests/test_migration_infrastructure.py b/tests/test_migration_infrastructure.pynew file mode 100644index 00000000..b25deb2b--- /dev/null+++ b/tests/test_migration_infrastructure.py@@ -0,0 +1,216 @@+"""Tests for the migration lock and the one-shot migration ledger.++Both exist because `initialize_database()` runs about five times per container+start, several of them concurrently: from the entrypoint, from the admin-user+script, and once per gunicorn worker.++Set TEST_DATABASE_URI to a PostgreSQL URL to also exercise those paths. CI sets+it already, so these run there without further configuration.+"""++import os+import subprocess+import sys+import textwrap++import pytest+from sqlalchemy import create_engine, inspect, text++from src.utils.database import ensure_migration_ledger, migration_lock, run_once+++REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))+_TEST_DB_URI = os.environ.get("TEST_DATABASE_URI", "")+POSTGRES_URL = _TEST_DB_URI if _TEST_DB_URI.startswith("postgresql") else None+++def sqlite_engine(tmp_path):+ return create_engine(f"sqlite:///{tmp_path / 'infra.db'}")+++class TestRunOnce:+ def test_runs_the_first_time_only(self, sqlite_engine):+ calls = []++ assert run_once(sqlite_engine, "m1", lambda engine: calls.append(1)) is True+ assert run_once(sqlite_engine, "m1", lambda engine: calls.append(1)) is False+ assert run_once(sqlite_engine, "m1", lambda engine: calls.append(1)) is False++ assert calls == [1]++ def test_different_migrations_are_independent(self, sqlite_engine):+ calls = []++ run_once(sqlite_engine, "m1", lambda engine: calls.append("a"))+ run_once(sqlite_engine, "m2", lambda engine: calls.append("b"))++ assert calls == ["a", "b"]++ def test_a_failure_is_not_recorded_and_retries(self, sqlite_engine):+ """An unrecorded failure must come back on the next startup."""+ attempts = []++ def failing(engine):+ attempts.append(1)+ raise RuntimeError("boom")++ with pytest.raises(RuntimeError):+ run_once(sqlite_engine, "m1", failing)++ with sqlite_engine.connect() as conn:+ recorded = conn.execute(+ text("SELECT COUNT(*) FROM schema_migrations WHERE migration_id = 'm1'")+ ).scalar()+ assert recorded == 0++ assert run_once(sqlite_engine, "m1", lambda engine: attempts.append(2)) is True+ assert attempts == [1, 2]++ def test_ledger_is_created_on_demand(self, sqlite_engine):+ assert "schema_migrations" not in inspect(sqlite_engine).get_table_names()++ ensure_migration_ledger(sqlite_engine)++ assert "schema_migrations" in inspect(sqlite_engine).get_table_names()++ def test_ledger_creation_is_idempotent(self, sqlite_engine):+ ensure_migration_ledger(sqlite_engine)+ ensure_migration_ledger(sqlite_engine)++ assert "schema_migrations" in inspect(sqlite_engine).get_table_names()++ def test_records_when_it_was_applied(self, sqlite_engine):+ run_once(sqlite_engine, "m1", lambda engine: None)++ with sqlite_engine.connect() as conn:+ applied_at = conn.execute(+ text("SELECT applied_at FROM schema_migrations WHERE migration_id = 'm1'")+ ).scalar()++ assert applied_at is not None+++class TestMigrationLock:+ def test_acquires_when_uncontended(self, sqlite_engine):+ with migration_lock(sqlite_engine) as acquired:+ assert acquired is True++ def test_is_reusable_after_release(self, sqlite_engine):+ with migration_lock(sqlite_engine) as first:+ assert first is True+ with migration_lock(sqlite_engine) as second:+ assert second is True++ def test_releases_even_when_the_body_raises(self, sqlite_engine):+ with pytest.raises(RuntimeError):+ with migration_lock(sqlite_engine):+ raise RuntimeError("boom")++ with migration_lock(sqlite_engine) as acquired:+ assert acquired is True++ def test_times_out_and_fails_open_while_another_process_holds_it(self, tmp_path):+ """A held lock must not hang startup: the caller proceeds with False."""+ db_path = tmp_path / "contended.db"+ engine = create_engine(f"sqlite:///{db_path}")++ holder = subprocess.Popen(+ [sys.executable, "-c", textwrap.dedent(f"""+ import sys, time+ sys.path.insert(0, {REPO_ROOT!r})+ from sqlalchemy import create_engine+ from src.utils.database import migration_lock+ engine = create_engine("sqlite:///{db_path}")+ with migration_lock(engine) as acquired:+ print("HELD" if acquired else "FAILED", flush=True)+ time.sleep(30)+ """)],+ stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True,+ env={**os.environ, "PYTHONPATH": REPO_ROOT},+ )+ try:+ assert holder.stdout.readline().strip() == "HELD"++ with migration_lock(engine, timeout=1) as acquired:+ assert acquired is False, "should fail open rather than block forever"+ finally:+ holder.kill()+ holder.wait(timeout=10)++ def test_lock_is_released_when_the_holder_dies(self, tmp_path):+ """A killed worker must not leave the lock held for the next start."""+ db_path = tmp_path / "orphan.db"+ engine = create_engine(f"sqlite:///{db_path}")++ holder = subprocess.Popen(+ [sys.executable, "-c", textwrap.dedent(f"""+ import sys, time+ sys.path.insert(0, {REPO_ROOT!r})+ from sqlalchemy import create_engine+ from src.utils.database import migration_lock+ engine = create_engine("sqlite:///{db_path}")+ with migration_lock(engine):+ print("HELD", flush=True)+ time.sleep(30)+ """)],+ stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True,+ env={**os.environ, "PYTHONPATH": REPO_ROOT},+ )+ assert holder.stdout.readline().strip() == "HELD"+ holder.kill()+ holder.wait(timeout=10)++ with migration_lock(engine, timeout=5) as acquired:+ assert acquired is True++ def test_separate_databases_do_not_block_each_other(self, tmp_path):+ one = create_engine(f"sqlite:///{tmp_path / 'one.db'}")+ two = create_engine(f"sqlite:///{tmp_path / 'two.db'}")++ with migration_lock(one) as first:+ with migration_lock(two, timeout=2) as second:+ assert first is True+ assert second is True++[email protected](not POSTGRES_URL, reason="TEST_DATABASE_URI is not a PostgreSQL URL")+class TestPostgreSQL:+ @pytest.fixture+ def pg_engine(self):+ engine = create_engine(POSTGRES_URL)+ with engine.connect() as conn:+ conn.execute(text("DROP TABLE IF EXISTS schema_migrations"))+ conn.commit()+ yield engine+ with engine.connect() as conn:+ conn.execute(text("DROP TABLE IF EXISTS schema_migrations"))+ conn.commit()+ engine.dispose()++ def test_run_once_runs_the_first_time_only(self, pg_engine):+ calls = []++ assert run_once(pg_engine, "m1", lambda engine: calls.append(1)) is True+ assert run_once(pg_engine, "m1", lambda engine: calls.append(1)) is False++ assert calls == [1]++ def test_ledger_is_created_on_demand(self, pg_engine):+ ensure_migration_ledger(pg_engine)++ assert "schema_migrations" in inspect(pg_engine).get_table_names()++ def test_advisory_lock_is_exclusive_across_sessions(self, pg_engine):+ """A second connection is a second session, which is what a worker is."""+ other = create_engine(POSTGRES_URL)+ try:+ with migration_lock(pg_engine) as first:+ assert first is True+ with migration_lock(other, timeout=1) as second:+ assert second is False+ # Released, so the other session can take it now.+ with migration_lock(other, timeout=5) as third:+ assert third is True+ finally:+ other.dispose()