Skip to content

Speakr is a personal, self-hosted web application designed for transcribing audio recordings

Python

fix: elect a single owner for the file monitor, schedulers and webhook dispatcher so background work stops running once per gunicorn worker (#384)

Murtaza Nasir15f102470aee
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 842072c4..2a1a9194 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -147,6 +147,66 @@ Values matter as much as schema: when the #379 migration destroyed 27 columns, t
`add_column_if_not_exists()` calls further down the same function immediately put the
names back, so only the emptied values revealed the damage.
+### Background Work at Startup
+
+`src/app.py` calls `run_startup_tasks()` at module scope, so **every process that
+imports the app runs it**: the entrypoint's schema check, `docker_create_admin.py`, and
+each gunicorn worker. The shipped image runs three workers, so anything started there
+runs about five times per container, several of them concurrently.
+
+A `global _x_started` flag does not help. Each process has its own copy, so it guards a
+second start inside one process and nothing else.
+
+This is not a hypothetical risk. In #384 the job queue's orphan recovery ran in every
+booting process, and because it resets every row in `processing` back to `queued` with
+no way to tell a job abandoned by a crash from one a sibling is transcribing right now,
+a worker starting up un-claimed live work and the same audio went to the ASR service
+once per worker. It shipped for nine months.
+
+**Anything that starts a thread, watches a directory, or sweeps a table goes through
+`start_single_instance()` in `src/config/startup.py`,** with a lock name registered in
+`SINGLE_INSTANCE_LOCKS`:
+
+```python
+start_single_instance(
+ app, MY_TASK_OWNER_LOCK,
+ lambda: initialize_my_task(app), 'my task')
+```
+
+The election uses a file lock held for the life of the process, so leadership ends on
+process death, including a crash, and the replacement worker wins the next one. It fails
+open: if the election cannot run, the process takes ownership rather than leaving the
+work undone.
+
+Two things stay outside the election. Work every process genuinely needs, such as
+binding the job queue so any worker can accept an upload and enqueue it. And work with
+nothing to duplicate, such as creating a directory.
+
+`tests/test_startup_single_instance.py` enforces this, and the enforcing test is
+structural rather than a list: it neutralises `start_single_instance` and asserts that
+`run_startup_tasks` then starts no threads at all, so a new task that skips the election
+fails it without anyone remembering to add a case.
+
+### Testing Concurrency
+
+Test cross-process behaviour **with processes**. The suite already had
+`tests/test_job_queue_race_condition.py` when #384 was reported, written specifically to
+prove that two workers cannot claim the same job. It passes, and it is correct. It also
+could not have caught the bug: it uses a `ThreadPoolExecutor` inside one process, and
+its own docstring notes that it never calls `start()`. Claiming was never broken. Several
+*processes* running the queue was.
+
+A test near a bug that structurally cannot observe it is worse than no test, because it
+makes the area feel covered.
+
+Prefer real subprocesses over `fork` for these. A gunicorn worker without `--preload`
+imports the app fresh, so a new interpreter is the honest simulation, and forking a
+process that already has background threads holding locks can deadlock the child.
+
+Assert on the duplication itself, not on the lock's return value, and keep the workers
+alive at the same time. Started one after another, each wins every election and the test
+passes while proving nothing.
+
### Commit Message Guidelines
Follow the format used in the project:
diff --git a/src/config/startup.py b/src/config/startup.py
index 2f9c6c9b..255b8e5b 100644
--- a/src/config/startup.py
+++ b/src/config/startup.py
@@ -11,8 +11,66 @@ from flask import current_app
ENABLE_AUTO_DELETION = os.environ.get('ENABLE_AUTO_DELETION', 'false').lower() == 'true'
GLOBAL_RETENTION_DAYS = int(os.environ.get('GLOBAL_RETENTION_DAYS', '0'))
-# Cross-process election that decides which process runs the job queue
+# Cross-process elections for work that must run in exactly one process.
+#
+# src/app.py calls run_startup_tasks() at module scope, so EVERY process that
+# imports it runs this file: the entrypoint's schema check, the admin-user
+# script, and each gunicorn worker (three by default). Anything here that
+# starts a thread, watches a directory or sweeps a table therefore ran that
+# many times over.
+#
+# That is not a hypothetical. It is #384: orphan recovery in every booting
+# process un-claimed jobs a sibling was mid-way through transcribing, and the
+# same audio went to the ASR service once per worker. The process-local
+# `global _x_started` guards several of these used are worthless here, because
+# each process has its own copy of the global.
+#
+# The fix is an election per responsibility. Route background work through
+# start_single_instance() below rather than calling it directly, so that
+# adding a new one forces the question "which process should run this?" to be
+# answered rather than skipped.
JOB_QUEUE_OWNER_LOCK = 'speakr.job_queue_owner'
+FILE_MONITOR_OWNER_LOCK = 'speakr.file_monitor_owner'
+AUTO_DELETION_OWNER_LOCK = 'speakr.auto_deletion_owner'
+SESSION_CLEANUP_OWNER_LOCK = 'speakr.session_cleanup_owner'
+WEBHOOK_DISPATCHER_OWNER_LOCK = 'speakr.webhook_dispatcher_owner'
+
+# Every lock name this module elects on. The test suite asserts that each
+# background starter in run_startup_tasks() goes through one of these.
+SINGLE_INSTANCE_LOCKS = (
+ JOB_QUEUE_OWNER_LOCK,
+ FILE_MONITOR_OWNER_LOCK,
+ AUTO_DELETION_OWNER_LOCK,
+ SESSION_CLEANUP_OWNER_LOCK,
+ WEBHOOK_DISPATCHER_OWNER_LOCK,
+)
+
+
+def start_single_instance(app, lock_name, start, description):
+ """Run `start` only in the process that wins the election for `lock_name`.
+
+ Losing is the normal outcome for most processes and is logged at debug;
+ it is not a failure. The election fails open, so the worst case is the
+ pre-election behaviour rather than work that never runs at all.
+ """
+ from src.database import db
+ from src.utils.database import acquire_singleton_lock
+
+ try:
+ if not acquire_singleton_lock(db.engine, lock_name, app.logger):
+ app.logger.debug("%s runs in another process; skipping here", description)
+ return False
+ except Exception as e:
+ # Never let the election itself cost us the work it guards.
+ app.logger.error(
+ "Could not run the election for %s, starting it here anyway: %s", description, e)
+
+ try:
+ start()
+ return True
+ except Exception as e:
+ app.logger.warning("%s failed to start: %s", description, e)
+ return False
def initialize_file_monitor(app):
@@ -111,48 +169,49 @@ def initialize_file_exporter(app):
app.logger.warning(f"File exporter initialization failed: {e}")
-def initialize_job_queue(app):
- """Initialize the background job queue; only the elected owner process recovers orphans and runs workers."""
- try:
- from src.database import db
- from src.services.job_queue import job_queue
- from src.utils.database import acquire_singleton_lock
+def bind_job_queue(app):
+ """Make the queue usable in this process. Every process needs this.
- # Initialize job queue with app context
- job_queue.init_app(app)
+ Binding is not the same as owning: a process that lost the election still
+ accepts uploads and enqueues them, it just runs no workers.
+ """
+ from src.services.job_queue import job_queue
+ job_queue.init_app(app)
- # Only the elected owner recovers orphans and runs workers; the rest just enqueue
- if not acquire_singleton_lock(db.engine, JOB_QUEUE_OWNER_LOCK, app.logger):
- app.logger.info(
- "Job queue owned by another process; this one will enqueue only"
- )
- return
- job_queue.mark_as_owner()
+def start_job_queue(app):
+ """Recover orphaned jobs and run the workers. Owner process only.
- # Recover any jobs that were processing when the app crashed
- job_queue.recover_orphaned_jobs()
+ Recovery is the reason this must not run everywhere: it resets every row
+ in `processing` back to `queued` with no way to tell a job abandoned by a
+ crash from one a sibling process is transcribing right now (#384).
+ """
+ from src.services.job_queue import job_queue
- # Start worker threads
- job_queue.start()
+ job_queue.mark_as_owner()
+ job_queue.recover_orphaned_jobs()
+ job_queue.start()
+
+ status = job_queue.get_queue_status()
+ t_queue = status['transcription_queue']
+ s_queue = status['summary_queue']
+ app.logger.info(
+ f"Job queues started: "
+ f"transcription ({t_queue['workers']} workers, {t_queue['queued']} queued), "
+ f"summary ({s_queue['workers']} workers, {s_queue['queued']} queued)"
+ )
- # Get queue status
- status = job_queue.get_queue_status()
- t_queue = status['transcription_queue']
- s_queue = status['summary_queue']
- app.logger.info(
- f"Job queues started: "
- f"transcription ({t_queue['workers']} workers, {t_queue['queued']} queued), "
- f"summary ({s_queue['workers']} workers, {s_queue['queued']} queued)"
- )
- except Exception as e:
- app.logger.error(f"Failed to start job queue: {e}", exc_info=True)
+
+def initialize_job_queue(app):
+ """Backwards-compatible wrapper: bind, elect, then start if we won."""
+ bind_job_queue(app)
+ start_single_instance(app, JOB_QUEUE_OWNER_LOCK,
+ lambda: start_job_queue(app), 'job queue')
-# Module-level guard so reloads (Flask --reload, gunicorn --reload,
-# container restart-on-change) don't spawn a second cleanup thread on
-# every reinit. The webhook dispatcher in src/services/webhook_dispatch.py
-# uses the same pattern.
+# Guards a SECOND START INSIDE ONE PROCESS (Flask --reload, gunicorn --reload,
+# container restart-on-change). It cannot guard against other processes,
+# because each has its own copy of it; start_single_instance() does that.
_cleanup_thread_started = False
@@ -228,21 +287,44 @@ def run_startup_tasks(app):
f"max_audio_only_video_size_mb={max_audio_only_video_mb})"
)
- # Initialize job queue for background processing
- initialize_job_queue(app)
-
- # Initialize file monitor after app setup
- initialize_file_monitor(app)
+ # Bind the queue in EVERY process, so any of them can accept an
+ # upload and enqueue it. Only the owner elected below runs workers.
+ bind_job_queue(app)
+
+ # Everything below that runs in the background goes through
+ # start_single_instance(). Adding one here without an election means
+ # it runs once per gunicorn worker plus twice more from the
+ # entrypoint scripts; see the comment on SINGLE_INSTANCE_LOCKS.
+ start_single_instance(
+ app, JOB_QUEUE_OWNER_LOCK,
+ lambda: start_job_queue(app), 'job queue')
+
+ # Watches one directory. Several watchers racing over the same
+ # dropped file is how one upload becomes several recordings.
+ start_single_instance(
+ app, FILE_MONITOR_OWNER_LOCK,
+ lambda: initialize_file_monitor(app), 'automated file processing')
+
+ # Deletes recordings on a retention policy. Concurrent sweeps race
+ # each other over the same rows and files.
+ start_single_instance(
+ app, AUTO_DELETION_OWNER_LOCK,
+ lambda: initialize_auto_deletion_scheduler(app), 'auto-deletion scheduler')
+
+ # Reaps abandoned recording sessions (#287 c/d).
+ start_single_instance(
+ app, SESSION_CLEANUP_OWNER_LOCK,
+ lambda: initialize_recording_session_cleanup(app), 'recording-session cleanup')
+
+ # Posts webhook deliveries (#275). Its own guard is a module global,
+ # which is per-process, so without this every worker ran a dispatcher
+ # and each POSTed the same due rows.
+ from src.services.webhook_dispatch import start_dispatcher_thread
+ start_single_instance(
+ app, WEBHOOK_DISPATCHER_OWNER_LOCK,
+ lambda: start_dispatcher_thread(app), 'webhook dispatcher')
- # Initialize file exporter
+ # Not elected: this only creates a directory and logs. No thread, no
+ # sweep, nothing to duplicate. If it ever grows a background task it
+ # needs to move up into the block above.
initialize_file_exporter(app)
-
- # Initialize auto-deletion scheduler
- initialize_auto_deletion_scheduler(app)
-
- # Initialize recording-session cleanup scheduler (#287 c/d)
- initialize_recording_session_cleanup(app)
-
- # Initialize webhook dispatcher (#275)
- from src.services.webhook_dispatch import start_dispatcher_thread
- start_dispatcher_thread(app)
diff --git a/tests/test_startup_single_instance.py b/tests/test_startup_single_instance.py
new file mode 100644
index 00000000..5cfbc81d
--- /dev/null
+++ b/tests/test_startup_single_instance.py
@@ -0,0 +1,301 @@
+"""Background work must start in exactly one process (#384).
+
+src/app.py runs run_startup_tasks() at module scope, so every process that
+imports it runs this: the entrypoint's schema check, docker_create_admin.py,
+and each gunicorn worker. The image ships three workers, so anything that
+starts a thread or sweeps a table here ran five or so times per container.
+
+The tests that were already in the tree when #384 was reported
+(tests/test_job_queue_race_condition.py) missed it, and the way they missed it
+is worth stating, because it shapes what is written below. They exercise job
+CLAIMING with a ThreadPoolExecutor and never call start(). Claiming was never
+broken. The bug was that several PROCESSES ran the queue at all. A test using
+threads in one process cannot see that, no matter how thoroughly it tests the
+thing it does test.
+
+So: real processes, assertions on the duplication itself rather than on the
+lock's return value, and one structural test that fails for a background task
+nobody has written yet.
+"""
+
+import json
+import os
+import subprocess
+import sys
+import tempfile
+import time
+import threading
+import uuid
+from unittest.mock import patch
+
+import pytest
+
+from src.app import app
+from src.config import startup
+
+
+def _reset_process_local_guards():
+ """Clear the `global _x_started` flags several starters keep.
+
+ Importing src.app has already set them in this process, so without this a
+ starter that BYPASSED the election would return early on its own guard and
+ look innocent. That is exactly the masking that let #384 survive: a guard
+ that is useless across processes still hides the bypass from an
+ in-process test.
+ """
+ from src.services import webhook_dispatch
+ import src.file_monitor as file_monitor
+ webhook_dispatch._dispatcher_thread_started = False
+ startup._cleanup_thread_started = False
+ file_monitor.file_monitor = None
+
+
+def unguarded():
+ """Run with the process-local guards down, and put them back after."""
+ from src.services import webhook_dispatch
+ import src.file_monitor as file_monitor
+ saved = (webhook_dispatch._dispatcher_thread_started,
+ startup._cleanup_thread_started,
+ file_monitor.file_monitor)
+ _reset_process_local_guards()
+ yield
+ (webhook_dispatch._dispatcher_thread_started,
+ startup._cleanup_thread_started,
+ file_monitor.file_monitor) = saved
+
+
+# ---------------------------------------------------------------------------
+# the structural test: does anything bypass the gate?
+# ---------------------------------------------------------------------------
+
+def test_run_startup_tasks_starts_no_background_work_outside_the_election(unguarded):
+ """The test that should fail for a background task nobody has written yet.
+
+ Neutralise start_single_instance, then run the whole of
+ run_startup_tasks and watch for threads. Anything that appears started
+ itself directly, which means it will run once per gunicorn worker.
+ """
+ started_threads = []
+ real_start = threading.Thread.start
+
+ def record_and_skip(self):
+ started_threads.append(self.name)
+ # Do not actually run it; we only care that it was asked to start.
+
+ with patch.object(startup, 'start_single_instance', return_value=False), \
+ patch.object(threading.Thread, 'start', record_and_skip):
+ startup.run_startup_tasks(app)
+
+ assert started_threads == [], (
+ "run_startup_tasks started these threads without going through "
+ f"start_single_instance(), so each gunicorn worker will start its own: {started_threads}"
+ )
+ threading.Thread.start = real_start
+
+
+def test_every_background_starter_is_registered_for_an_election(unguarded):
+ """Each gated call must use a name from SINGLE_INSTANCE_LOCKS.
+
+ Catches a new task gated with an ad-hoc string, which would elect
+ correctly but silently drop out of this module's inventory.
+ """
+ used = []
+
+ def capture(app_, lock_name, start, description):
+ used.append(lock_name)
+ return False
+
+ with patch.object(startup, 'start_single_instance', capture):
+ startup.run_startup_tasks(app)
+
+ assert used, "run_startup_tasks elected nothing at all"
+ unknown = [n for n in used if n not in startup.SINGLE_INSTANCE_LOCKS]
+ assert unknown == [], f"lock names missing from SINGLE_INSTANCE_LOCKS: {unknown}"
+ # Every registered lock should actually be used; a stale name means a task
+ # was removed and the inventory not updated.
+ unused = [n for n in startup.SINGLE_INSTANCE_LOCKS if n not in used]
+ assert unused == [], f"declared but never elected on: {unused}"
+
+
+def test_the_job_queue_is_bound_in_every_process_even_when_it_loses(unguarded):
+ """Losing the election must not stop a process accepting uploads.
+
+ A web worker that cannot enqueue is worse than a duplicated transcription.
+ """
+ with patch.object(startup, 'start_single_instance', return_value=False), \
+ patch.object(threading.Thread, 'start', lambda self: None):
+ startup.run_startup_tasks(app)
+
+ from src.services.job_queue import job_queue
+ assert job_queue._app is not None, "queue was not bound in a non-owner process"
+
+
+# ---------------------------------------------------------------------------
+# the cross-process test: what actually happens with several workers
+# ---------------------------------------------------------------------------
+#
+# Real subprocesses, not fork. Two reasons. A gunicorn worker without
+# --preload imports the app fresh, so a fresh interpreter is the honest
+# simulation. And forking a process that already has background threads
+# holding locks can deadlock the child, which is exactly what happened when
+# this was first written with multiprocessing: the file passed alone and hung
+# the whole suite.
+#
+# Lock names are namespaced per run because this test process, and the dev
+# server if one is running, already hold the production names; children would
+# otherwise just measure the parent winning.
+
+# Booting the app prints a page of startup logging to stdout, so the result
+# is handed over in a file rather than parsed out of that noise.
+def _run_workers(count, timeout=120):
+ """Boot `count` app processes at once, collect what each one started.
+
+ They are held alive together on purpose. Run one after another, each
+ would win every election and the test would pass while proving nothing.
+ """
+ run_id = f'speakr_test_{os.getpid()}_{uuid.uuid4().hex[:8]}'
+ repo_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+ procs, out_paths = [], []
+ with tempfile.TemporaryDirectory() as tmp:
+ for i in range(count):
+ out = os.path.join(tmp, f'worker-{i}.json')
+ out_paths.append(out)
+ procs.append(subprocess.Popen(
+ [sys.executable, os.path.abspath(__file__), run_id, out],
+ stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, text=True,
+ cwd=repo_root))
+ try:
+ deadline = time.time() + timeout
+ while time.time() < deadline:
+ if all(os.path.exists(p) and os.path.getsize(p) > 0 for p in out_paths):
+ break
+ dead = [p for p in procs if p.poll() is not None]
+ if dead:
+ raise AssertionError(
+ f'a worker exited before reporting: {dead[0].stderr.read()[-2000:]}')
+ time.sleep(0.2)
+ else:
+ raise AssertionError(f'workers did not all report within {timeout}s')
+ return [json.loads(open(p).read()) for p in out_paths]
+ finally:
+ for p in procs:
+ p.terminate()
+ for p in procs:
+ try:
+ p.wait(timeout=15)
+ except subprocess.TimeoutExpired: # pragma: no cover
+ p.kill()
+
+
+def test_three_workers_booting_together_start_each_task_exactly_once():
+ """The shipped configuration: the Dockerfile CMD runs --workers 3."""
+ per_process = _run_workers(3)
+
+ started = [name for r in per_process for name in r['started']]
+ assert started, f"no process started anything; election failed closed: {per_process}"
+
+ duplicated = sorted({n for n in started if started.count(n) > 1})
+ assert not duplicated, (
+ f"these ran in more than one of 3 processes: {duplicated}. "
+ f"per-process breakdown: {per_process}"
+ )
+
+ assert len(set(started)) == len(startup.SINGLE_INSTANCE_LOCKS), (
+ f"expected all {len(startup.SINGLE_INSTANCE_LOCKS)} tasks to run once across "
+ f"3 processes, got {sorted(set(started))}"
+ )
+
+
+def test_eight_workers_do_not_start_eight_dispatchers():
+ """More workers must not mean more duplication. #384 scaled with the
+ worker count: 3 workers produced 2 ASR submissions of one file, 8 produced 7.
+ """
+ per_process = _run_workers(8)
+ started = [name for r in per_process for name in r['started']]
+ for task in ('webhook dispatcher', 'job queue', 'auto-deletion scheduler',
+ 'automated file processing', 'recording-session cleanup'):
+ assert started.count(task) == 1, (
+ f"{task!r} started {started.count(task)} times across 8 processes: {per_process}")
+
+
+def test_every_worker_can_still_enqueue_even_though_only_one_owns_the_queue():
+ """Seven of eight processes lose, and all eight must still take uploads."""
+ per_process = _run_workers(8)
+ assert all(r['queue_bound'] for r in per_process), per_process
+ owners = [r for r in per_process if 'job queue' in r['started']]
+ assert len(owners) == 1, per_process
+
+
+# ---------------------------------------------------------------------------
+# failure behaviour
+# ---------------------------------------------------------------------------
+
+def test_a_broken_election_still_starts_the_work():
+ """Fail open. An election that cannot run must not silently disable every
+ background task, which would leave an app that accepts uploads and
+ transcribes nothing."""
+ ran = []
+ with app.app_context(), \
+ patch('src.utils.database.acquire_singleton_lock', side_effect=OSError('no locks')):
+ result = startup.start_single_instance(
+ app, 'speakr.test_broken', lambda: ran.append('started'), 'test task')
+
+ assert ran == ['started']
+ assert result is True
+
+
+def test_a_task_that_throws_does_not_stop_the_others():
+ """One failing starter must not take out the rest of startup."""
+ with app.app_context():
+ result = startup.start_single_instance(
+ app, 'speakr.test_throws', lambda: (_ for _ in ()).throw(RuntimeError('boom')),
+ 'exploding task')
+ assert result is False
+
+
+def test_losing_is_not_reported_as_a_failure():
+ """A loser returns False, same as a crashed starter, so the two must be
+ distinguishable by their logging rather than only by the return value."""
+ with app.app_context(), \
+ patch('src.utils.database.acquire_singleton_lock', return_value=False), \
+ patch.object(app.logger, 'debug') as debug, \
+ patch.object(app.logger, 'warning') as warn:
+ result = startup.start_single_instance(
+ app, 'speakr.test_loser', lambda: None, 'test task')
+ assert result is False
+ assert debug.called, "a lost election should log at debug, it is the normal case"
+ assert not warn.called, "a lost election is not a warning"
+
+
+# ---------------------------------------------------------------------------
+# worker entry point (not collected by pytest; see _run_workers above)
+# ---------------------------------------------------------------------------
+
+def _main():
+ """Boot like a gunicorn worker and report what this process started."""
+ run_id, out_path = sys.argv[1], sys.argv[2]
+ _reset_process_local_guards()
+
+ started = []
+ real = startup.start_single_instance
+
+ def gated(app_, lock_name, start, description):
+ return real(app_, f'{run_id}.{lock_name}',
+ lambda: started.append(description), description)
+
+ from src.services.job_queue import job_queue
+ with patch.object(startup, 'start_single_instance', gated), \
+ patch.object(threading.Thread, 'start', lambda self: None):
+ startup.run_startup_tasks(app)
+
+ with open(out_path, 'w') as f:
+ json.dump({'started': sorted(started),
+ 'queue_bound': job_queue._app is not None}, f)
+ # Hold the locks so siblings genuinely contend rather than inheriting a
+ # free field from a process that already exited.
+ time.sleep(60)
+
+
+if __name__ == '__main__':
+ _main()