Speakr is a personal, self-hosted web application designed for transcribing audio recordings
Python
fix: elect a single owner for the file monitor, schedulers and webhook dispatcher so background work stops running once per gunicorn worker (#384)
Murtaza Nasir15f102470aee
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex 842072c4..2a1a9194 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -147,6 +147,66 @@ Values matter as much as schema: when the #379 migration destroyed 27 columns, t`add_column_if_not_exists()` calls further down the same function immediately put thenames back, so only the emptied values revealed the damage.+### Background Work at Startup++`src/app.py` calls `run_startup_tasks()` at module scope, so **every process that+imports the app runs it**: the entrypoint's schema check, `docker_create_admin.py`, and+each gunicorn worker. The shipped image runs three workers, so anything started there+runs about five times per container, several of them concurrently.++A `global _x_started` flag does not help. Each process has its own copy, so it guards a+second start inside one process and nothing else.++This is not a hypothetical risk. In #384 the job queue's orphan recovery ran in every+booting process, and because it resets every row in `processing` back to `queued` with+no way to tell a job abandoned by a crash from one a sibling is transcribing right now,+a worker starting up un-claimed live work and the same audio went to the ASR service+once per worker. It shipped for nine months.++**Anything that starts a thread, watches a directory, or sweeps a table goes through+`start_single_instance()` in `src/config/startup.py`,** with a lock name registered in+`SINGLE_INSTANCE_LOCKS`:++```python+start_single_instance(+ app, MY_TASK_OWNER_LOCK,+ lambda: initialize_my_task(app), 'my task')+```++The election uses a file lock held for the life of the process, so leadership ends on+process death, including a crash, and the replacement worker wins the next one. It fails+open: if the election cannot run, the process takes ownership rather than leaving the+work undone.++Two things stay outside the election. Work every process genuinely needs, such as+binding the job queue so any worker can accept an upload and enqueue it. And work with+nothing to duplicate, such as creating a directory.++`tests/test_startup_single_instance.py` enforces this, and the enforcing test is+structural rather than a list: it neutralises `start_single_instance` and asserts that+`run_startup_tasks` then starts no threads at all, so a new task that skips the election+fails it without anyone remembering to add a case.++### Testing Concurrency++Test cross-process behaviour **with processes**. The suite already had+`tests/test_job_queue_race_condition.py` when #384 was reported, written specifically to+prove that two workers cannot claim the same job. It passes, and it is correct. It also+could not have caught the bug: it uses a `ThreadPoolExecutor` inside one process, and+its own docstring notes that it never calls `start()`. Claiming was never broken. Several+*processes* running the queue was.++A test near a bug that structurally cannot observe it is worse than no test, because it+makes the area feel covered.++Prefer real subprocesses over `fork` for these. A gunicorn worker without `--preload`+imports the app fresh, so a new interpreter is the honest simulation, and forking a+process that already has background threads holding locks can deadlock the child.++Assert on the duplication itself, not on the lock's return value, and keep the workers+alive at the same time. Started one after another, each wins every election and the test+passes while proving nothing.+### Commit Message GuidelinesFollow the format used in the project:diff --git a/src/config/startup.py b/src/config/startup.pyindex 2f9c6c9b..255b8e5b 100644--- a/src/config/startup.py+++ b/src/config/startup.py@@ -11,8 +11,66 @@ from flask import current_appENABLE_AUTO_DELETION = os.environ.get('ENABLE_AUTO_DELETION', 'false').lower() == 'true'GLOBAL_RETENTION_DAYS = int(os.environ.get('GLOBAL_RETENTION_DAYS', '0'))-# Cross-process election that decides which process runs the job queue+# Cross-process elections for work that must run in exactly one process.+#+# src/app.py calls run_startup_tasks() at module scope, so EVERY process that+# imports it runs this file: the entrypoint's schema check, the admin-user+# script, and each gunicorn worker (three by default). Anything here that+# starts a thread, watches a directory or sweeps a table therefore ran that+# many times over.+#+# That is not a hypothetical. It is #384: orphan recovery in every booting+# process un-claimed jobs a sibling was mid-way through transcribing, and the+# same audio went to the ASR service once per worker. The process-local+# `global _x_started` guards several of these used are worthless here, because+# each process has its own copy of the global.+#+# The fix is an election per responsibility. Route background work through+# start_single_instance() below rather than calling it directly, so that+# adding a new one forces the question "which process should run this?" to be+# answered rather than skipped.JOB_QUEUE_OWNER_LOCK = 'speakr.job_queue_owner'+FILE_MONITOR_OWNER_LOCK = 'speakr.file_monitor_owner'+AUTO_DELETION_OWNER_LOCK = 'speakr.auto_deletion_owner'+SESSION_CLEANUP_OWNER_LOCK = 'speakr.session_cleanup_owner'+WEBHOOK_DISPATCHER_OWNER_LOCK = 'speakr.webhook_dispatcher_owner'++# Every lock name this module elects on. The test suite asserts that each+# background starter in run_startup_tasks() goes through one of these.+SINGLE_INSTANCE_LOCKS = (+ JOB_QUEUE_OWNER_LOCK,+ FILE_MONITOR_OWNER_LOCK,+ AUTO_DELETION_OWNER_LOCK,+ SESSION_CLEANUP_OWNER_LOCK,+ WEBHOOK_DISPATCHER_OWNER_LOCK,+)+++def start_single_instance(app, lock_name, start, description):+ """Run `start` only in the process that wins the election for `lock_name`.++ Losing is the normal outcome for most processes and is logged at debug;+ it is not a failure. The election fails open, so the worst case is the+ pre-election behaviour rather than work that never runs at all.+ """+ from src.database import db+ from src.utils.database import acquire_singleton_lock++ try:+ if not acquire_singleton_lock(db.engine, lock_name, app.logger):+ app.logger.debug("%s runs in another process; skipping here", description)+ return False+ except Exception as e:+ # Never let the election itself cost us the work it guards.+ app.logger.error(+ "Could not run the election for %s, starting it here anyway: %s", description, e)++ try:+ start()+ return True+ except Exception as e:+ app.logger.warning("%s failed to start: %s", description, e)+ return Falsedef initialize_file_monitor(app):@@ -111,48 +169,49 @@ def initialize_file_exporter(app):app.logger.warning(f"File exporter initialization failed: {e}")-def initialize_job_queue(app):- """Initialize the background job queue; only the elected owner process recovers orphans and runs workers."""- try:- from src.database import db- from src.services.job_queue import job_queue- from src.utils.database import acquire_singleton_lock+def bind_job_queue(app):+ """Make the queue usable in this process. Every process needs this.- # Initialize job queue with app context- job_queue.init_app(app)+ Binding is not the same as owning: a process that lost the election still+ accepts uploads and enqueues them, it just runs no workers.+ """+ from src.services.job_queue import job_queue+ job_queue.init_app(app)- # Only the elected owner recovers orphans and runs workers; the rest just enqueue- if not acquire_singleton_lock(db.engine, JOB_QUEUE_OWNER_LOCK, app.logger):- app.logger.info(- "Job queue owned by another process; this one will enqueue only"- )- return- job_queue.mark_as_owner()+def start_job_queue(app):+ """Recover orphaned jobs and run the workers. Owner process only.- # Recover any jobs that were processing when the app crashed- job_queue.recover_orphaned_jobs()+ Recovery is the reason this must not run everywhere: it resets every row+ in `processing` back to `queued` with no way to tell a job abandoned by a+ crash from one a sibling process is transcribing right now (#384).+ """+ from src.services.job_queue import job_queue- # Start worker threads- job_queue.start()+ job_queue.mark_as_owner()+ job_queue.recover_orphaned_jobs()+ job_queue.start()++ status = job_queue.get_queue_status()+ t_queue = status['transcription_queue']+ s_queue = status['summary_queue']+ app.logger.info(+ f"Job queues started: "+ f"transcription ({t_queue['workers']} workers, {t_queue['queued']} queued), "+ f"summary ({s_queue['workers']} workers, {s_queue['queued']} queued)"+ )- # Get queue status- status = job_queue.get_queue_status()- t_queue = status['transcription_queue']- s_queue = status['summary_queue']- app.logger.info(- f"Job queues started: "- f"transcription ({t_queue['workers']} workers, {t_queue['queued']} queued), "- f"summary ({s_queue['workers']} workers, {s_queue['queued']} queued)"- )- except Exception as e:- app.logger.error(f"Failed to start job queue: {e}", exc_info=True)++def initialize_job_queue(app):+ """Backwards-compatible wrapper: bind, elect, then start if we won."""+ bind_job_queue(app)+ start_single_instance(app, JOB_QUEUE_OWNER_LOCK,+ lambda: start_job_queue(app), 'job queue')-# Module-level guard so reloads (Flask --reload, gunicorn --reload,-# container restart-on-change) don't spawn a second cleanup thread on-# every reinit. The webhook dispatcher in src/services/webhook_dispatch.py-# uses the same pattern.+# Guards a SECOND START INSIDE ONE PROCESS (Flask --reload, gunicorn --reload,+# container restart-on-change). It cannot guard against other processes,+# because each has its own copy of it; start_single_instance() does that._cleanup_thread_started = False@@ -228,21 +287,44 @@ def run_startup_tasks(app):f"max_audio_only_video_size_mb={max_audio_only_video_mb})")- # Initialize job queue for background processing- initialize_job_queue(app)-- # Initialize file monitor after app setup- initialize_file_monitor(app)+ # Bind the queue in EVERY process, so any of them can accept an+ # upload and enqueue it. Only the owner elected below runs workers.+ bind_job_queue(app)++ # Everything below that runs in the background goes through+ # start_single_instance(). Adding one here without an election means+ # it runs once per gunicorn worker plus twice more from the+ # entrypoint scripts; see the comment on SINGLE_INSTANCE_LOCKS.+ start_single_instance(+ app, JOB_QUEUE_OWNER_LOCK,+ lambda: start_job_queue(app), 'job queue')++ # Watches one directory. Several watchers racing over the same+ # dropped file is how one upload becomes several recordings.+ start_single_instance(+ app, FILE_MONITOR_OWNER_LOCK,+ lambda: initialize_file_monitor(app), 'automated file processing')++ # Deletes recordings on a retention policy. Concurrent sweeps race+ # each other over the same rows and files.+ start_single_instance(+ app, AUTO_DELETION_OWNER_LOCK,+ lambda: initialize_auto_deletion_scheduler(app), 'auto-deletion scheduler')++ # Reaps abandoned recording sessions (#287 c/d).+ start_single_instance(+ app, SESSION_CLEANUP_OWNER_LOCK,+ lambda: initialize_recording_session_cleanup(app), 'recording-session cleanup')++ # Posts webhook deliveries (#275). Its own guard is a module global,+ # which is per-process, so without this every worker ran a dispatcher+ # and each POSTed the same due rows.+ from src.services.webhook_dispatch import start_dispatcher_thread+ start_single_instance(+ app, WEBHOOK_DISPATCHER_OWNER_LOCK,+ lambda: start_dispatcher_thread(app), 'webhook dispatcher')- # Initialize file exporter+ # Not elected: this only creates a directory and logs. No thread, no+ # sweep, nothing to duplicate. If it ever grows a background task it+ # needs to move up into the block above.initialize_file_exporter(app)-- # Initialize auto-deletion scheduler- initialize_auto_deletion_scheduler(app)-- # Initialize recording-session cleanup scheduler (#287 c/d)- initialize_recording_session_cleanup(app)-- # Initialize webhook dispatcher (#275)- from src.services.webhook_dispatch import start_dispatcher_thread- start_dispatcher_thread(app)diff --git a/tests/test_startup_single_instance.py b/tests/test_startup_single_instance.pynew file mode 100644index 00000000..5cfbc81d--- /dev/null+++ b/tests/test_startup_single_instance.py@@ -0,0 +1,301 @@+"""Background work must start in exactly one process (#384).++src/app.py runs run_startup_tasks() at module scope, so every process that+imports it runs this: the entrypoint's schema check, docker_create_admin.py,+and each gunicorn worker. The image ships three workers, so anything that+starts a thread or sweeps a table here ran five or so times per container.++The tests that were already in the tree when #384 was reported+(tests/test_job_queue_race_condition.py) missed it, and the way they missed it+is worth stating, because it shapes what is written below. They exercise job+CLAIMING with a ThreadPoolExecutor and never call start(). Claiming was never+broken. The bug was that several PROCESSES ran the queue at all. A test using+threads in one process cannot see that, no matter how thoroughly it tests the+thing it does test.++So: real processes, assertions on the duplication itself rather than on the+lock's return value, and one structural test that fails for a background task+nobody has written yet.+"""++import json+import os+import subprocess+import sys+import tempfile+import time+import threading+import uuid+from unittest.mock import patch++import pytest++from src.app import app+from src.config import startup+++def _reset_process_local_guards():+ """Clear the `global _x_started` flags several starters keep.++ Importing src.app has already set them in this process, so without this a+ starter that BYPASSED the election would return early on its own guard and+ look innocent. That is exactly the masking that let #384 survive: a guard+ that is useless across processes still hides the bypass from an+ in-process test.+ """+ from src.services import webhook_dispatch+ import src.file_monitor as file_monitor+ webhook_dispatch._dispatcher_thread_started = False+ startup._cleanup_thread_started = False+ file_monitor.file_monitor = None+++def unguarded():+ """Run with the process-local guards down, and put them back after."""+ from src.services import webhook_dispatch+ import src.file_monitor as file_monitor+ saved = (webhook_dispatch._dispatcher_thread_started,+ startup._cleanup_thread_started,+ file_monitor.file_monitor)+ _reset_process_local_guards()+ yield+ (webhook_dispatch._dispatcher_thread_started,+ startup._cleanup_thread_started,+ file_monitor.file_monitor) = saved+++# ---------------------------------------------------------------------------+# the structural test: does anything bypass the gate?+# ---------------------------------------------------------------------------++def test_run_startup_tasks_starts_no_background_work_outside_the_election(unguarded):+ """The test that should fail for a background task nobody has written yet.++ Neutralise start_single_instance, then run the whole of+ run_startup_tasks and watch for threads. Anything that appears started+ itself directly, which means it will run once per gunicorn worker.+ """+ started_threads = []+ real_start = threading.Thread.start++ def record_and_skip(self):+ started_threads.append(self.name)+ # Do not actually run it; we only care that it was asked to start.++ with patch.object(startup, 'start_single_instance', return_value=False), \+ patch.object(threading.Thread, 'start', record_and_skip):+ startup.run_startup_tasks(app)++ assert started_threads == [], (+ "run_startup_tasks started these threads without going through "+ f"start_single_instance(), so each gunicorn worker will start its own: {started_threads}"+ )+ threading.Thread.start = real_start+++def test_every_background_starter_is_registered_for_an_election(unguarded):+ """Each gated call must use a name from SINGLE_INSTANCE_LOCKS.++ Catches a new task gated with an ad-hoc string, which would elect+ correctly but silently drop out of this module's inventory.+ """+ used = []++ def capture(app_, lock_name, start, description):+ used.append(lock_name)+ return False++ with patch.object(startup, 'start_single_instance', capture):+ startup.run_startup_tasks(app)++ assert used, "run_startup_tasks elected nothing at all"+ unknown = [n for n in used if n not in startup.SINGLE_INSTANCE_LOCKS]+ assert unknown == [], f"lock names missing from SINGLE_INSTANCE_LOCKS: {unknown}"+ # Every registered lock should actually be used; a stale name means a task+ # was removed and the inventory not updated.+ unused = [n for n in startup.SINGLE_INSTANCE_LOCKS if n not in used]+ assert unused == [], f"declared but never elected on: {unused}"+++def test_the_job_queue_is_bound_in_every_process_even_when_it_loses(unguarded):+ """Losing the election must not stop a process accepting uploads.++ A web worker that cannot enqueue is worse than a duplicated transcription.+ """+ with patch.object(startup, 'start_single_instance', return_value=False), \+ patch.object(threading.Thread, 'start', lambda self: None):+ startup.run_startup_tasks(app)++ from src.services.job_queue import job_queue+ assert job_queue._app is not None, "queue was not bound in a non-owner process"+++# ---------------------------------------------------------------------------+# the cross-process test: what actually happens with several workers+# ---------------------------------------------------------------------------+#+# Real subprocesses, not fork. Two reasons. A gunicorn worker without+# --preload imports the app fresh, so a fresh interpreter is the honest+# simulation. And forking a process that already has background threads+# holding locks can deadlock the child, which is exactly what happened when+# this was first written with multiprocessing: the file passed alone and hung+# the whole suite.+#+# Lock names are namespaced per run because this test process, and the dev+# server if one is running, already hold the production names; children would+# otherwise just measure the parent winning.++# Booting the app prints a page of startup logging to stdout, so the result+# is handed over in a file rather than parsed out of that noise.+def _run_workers(count, timeout=120):+ """Boot `count` app processes at once, collect what each one started.++ They are held alive together on purpose. Run one after another, each+ would win every election and the test would pass while proving nothing.+ """+ run_id = f'speakr_test_{os.getpid()}_{uuid.uuid4().hex[:8]}'+ repo_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))+ procs, out_paths = [], []+ with tempfile.TemporaryDirectory() as tmp:+ for i in range(count):+ out = os.path.join(tmp, f'worker-{i}.json')+ out_paths.append(out)+ procs.append(subprocess.Popen(+ [sys.executable, os.path.abspath(__file__), run_id, out],+ stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, text=True,+ cwd=repo_root))+ try:+ deadline = time.time() + timeout+ while time.time() < deadline:+ if all(os.path.exists(p) and os.path.getsize(p) > 0 for p in out_paths):+ break+ dead = [p for p in procs if p.poll() is not None]+ if dead:+ raise AssertionError(+ f'a worker exited before reporting: {dead[0].stderr.read()[-2000:]}')+ time.sleep(0.2)+ else:+ raise AssertionError(f'workers did not all report within {timeout}s')+ return [json.loads(open(p).read()) for p in out_paths]+ finally:+ for p in procs:+ p.terminate()+ for p in procs:+ try:+ p.wait(timeout=15)+ except subprocess.TimeoutExpired: # pragma: no cover+ p.kill()+++def test_three_workers_booting_together_start_each_task_exactly_once():+ """The shipped configuration: the Dockerfile CMD runs --workers 3."""+ per_process = _run_workers(3)++ started = [name for r in per_process for name in r['started']]+ assert started, f"no process started anything; election failed closed: {per_process}"++ duplicated = sorted({n for n in started if started.count(n) > 1})+ assert not duplicated, (+ f"these ran in more than one of 3 processes: {duplicated}. "+ f"per-process breakdown: {per_process}"+ )++ assert len(set(started)) == len(startup.SINGLE_INSTANCE_LOCKS), (+ f"expected all {len(startup.SINGLE_INSTANCE_LOCKS)} tasks to run once across "+ f"3 processes, got {sorted(set(started))}"+ )+++def test_eight_workers_do_not_start_eight_dispatchers():+ """More workers must not mean more duplication. #384 scaled with the+ worker count: 3 workers produced 2 ASR submissions of one file, 8 produced 7.+ """+ per_process = _run_workers(8)+ started = [name for r in per_process for name in r['started']]+ for task in ('webhook dispatcher', 'job queue', 'auto-deletion scheduler',+ 'automated file processing', 'recording-session cleanup'):+ assert started.count(task) == 1, (+ f"{task!r} started {started.count(task)} times across 8 processes: {per_process}")+++def test_every_worker_can_still_enqueue_even_though_only_one_owns_the_queue():+ """Seven of eight processes lose, and all eight must still take uploads."""+ per_process = _run_workers(8)+ assert all(r['queue_bound'] for r in per_process), per_process+ owners = [r for r in per_process if 'job queue' in r['started']]+ assert len(owners) == 1, per_process+++# ---------------------------------------------------------------------------+# failure behaviour+# ---------------------------------------------------------------------------++def test_a_broken_election_still_starts_the_work():+ """Fail open. An election that cannot run must not silently disable every+ background task, which would leave an app that accepts uploads and+ transcribes nothing."""+ ran = []+ with app.app_context(), \+ patch('src.utils.database.acquire_singleton_lock', side_effect=OSError('no locks')):+ result = startup.start_single_instance(+ app, 'speakr.test_broken', lambda: ran.append('started'), 'test task')++ assert ran == ['started']+ assert result is True+++def test_a_task_that_throws_does_not_stop_the_others():+ """One failing starter must not take out the rest of startup."""+ with app.app_context():+ result = startup.start_single_instance(+ app, 'speakr.test_throws', lambda: (_ for _ in ()).throw(RuntimeError('boom')),+ 'exploding task')+ assert result is False+++def test_losing_is_not_reported_as_a_failure():+ """A loser returns False, same as a crashed starter, so the two must be+ distinguishable by their logging rather than only by the return value."""+ with app.app_context(), \+ patch('src.utils.database.acquire_singleton_lock', return_value=False), \+ patch.object(app.logger, 'debug') as debug, \+ patch.object(app.logger, 'warning') as warn:+ result = startup.start_single_instance(+ app, 'speakr.test_loser', lambda: None, 'test task')+ assert result is False+ assert debug.called, "a lost election should log at debug, it is the normal case"+ assert not warn.called, "a lost election is not a warning"+++# ---------------------------------------------------------------------------+# worker entry point (not collected by pytest; see _run_workers above)+# ---------------------------------------------------------------------------++def _main():+ """Boot like a gunicorn worker and report what this process started."""+ run_id, out_path = sys.argv[1], sys.argv[2]+ _reset_process_local_guards()++ started = []+ real = startup.start_single_instance++ def gated(app_, lock_name, start, description):+ return real(app_, f'{run_id}.{lock_name}',+ lambda: started.append(description), description)++ from src.services.job_queue import job_queue+ with patch.object(startup, 'start_single_instance', gated), \+ patch.object(threading.Thread, 'start', lambda self: None):+ startup.run_startup_tasks(app)++ with open(out_path, 'w') as f:+ json.dump({'started': sorted(started),+ 'queue_bound': job_queue._app is not None}, f)+ # Hold the locks so siblings genuinely contend rather than inheriting a+ # free field from a process that already exited.+ time.sleep(60)+++if __name__ == '__main__':+ _main()