Speakr is a personal, self-hosted web application designed for transcribing audio recordings
Python
Add unified progress tracking with smart polling and fix queue management
Murtaza Nasirddf664faaead
diff --git a/docs/advanced/job-queue-system.md b/docs/advanced/job-queue-system.mdnew file mode 100644index 00000000..9f78a073--- /dev/null+++ b/docs/advanced/job-queue-system.md@@ -0,0 +1,393 @@+# Job Queue & Progress Tracking System++This document describes the backend job queue system and frontend unified progress tracking implementation. This architecture handles audio processing jobs (transcription, summarization) with fair scheduling across users and provides real-time progress feedback.++## Architecture Overview++```+┌─────────────────────────────────────────────────────────────────────────┐+│ FRONTEND │+│ ┌─────────────────┐ ┌──────────────────┐ ┌───────────────────┐ │+│ │ Upload Queue │───▶│ Unified Progress │◀───│ Job Queue API │ │+│ │ (client-side) │ │ Items │ │ (polling) │ │+│ └─────────────────┘ └──────────────────┘ └───────────────────┘ │+└─────────────────────────────────────────────────────────────────────────┘+ │+ ▼+┌─────────────────────────────────────────────────────────────────────────┐+│ BACKEND │+│ ┌─────────────────┐ ┌──────────────────┐ ┌───────────────────┐ │+│ │ Upload API │───▶│ ProcessingJob │◀───│ Job Queue │ │+│ │ /upload │ │ Model │ │ Service │ │+│ └─────────────────┘ └──────────────────┘ └───────────────────┘ │+│ │ │ │+│ ▼ ▼ │+│ ┌──────────────────┐ ┌───────────────────┐ │+│ │ SQLite DB │ │ Worker Thread │ │+│ │ processing_job │ │ (background) │ │+│ └──────────────────┘ └───────────────────┘ │+└─────────────────────────────────────────────────────────────────────────┘+```++## Backend Components++### 1. ProcessingJob Model (`src/models/processing_job.py`)++Database model for persistent job tracking:++```python+class ProcessingJob(db.Model):+ __tablename__ = 'processing_job'++ id = db.Column(db.Integer, primary_key=True)+ user_id = db.Column(db.Integer, db.ForeignKey('user.id'), nullable=False)+ recording_id = db.Column(db.Integer, db.ForeignKey('recording.id'), nullable=False)++ # Job type: transcribe, summarize, reprocess_transcription, reprocess_summary+ job_type = db.Column(db.String(50), nullable=False)++ # Status: queued, processing, completed, failed+ status = db.Column(db.String(20), default='queued', nullable=False)++ # JSON blob for job-specific parameters+ params = db.Column(db.Text, nullable=True)++ # Error tracking+ error_message = db.Column(db.Text, nullable=True)+ retry_count = db.Column(db.Integer, default=0, nullable=False)++ # Track if this is a new upload (vs reprocessing) - for cleanup on failure+ is_new_upload = db.Column(db.Boolean, default=False, nullable=False)++ # Timestamps+ created_at = db.Column(db.DateTime, default=datetime.utcnow)+ started_at = db.Column(db.DateTime, nullable=True)+ completed_at = db.Column(db.DateTime, nullable=True)+```++**Key Fields:**+- `is_new_upload`: When `True` and job fails permanently, the associated recording and audio file are deleted. When `False` (reprocessing), only the recording status is set to FAILED.+- `status`: Job lifecycle state+- `retry_count`: Jobs retry up to 3 times before permanent failure++### 2. Job Queue Service (`src/services/job_queue.py`)++Background worker that processes jobs fairly across users.++**Key Features:**++#### Fair Scheduling (Round-Robin per User)+```python+def _claim_next_job(self, job_types, queue_name):+ # Find the next user who has waiting jobs+ # Round-robin through users to ensure fairness+ users_with_jobs = db.session.query(ProcessingJob.user_id).filter(+ ProcessingJob.status == 'queued',+ ProcessingJob.job_type.in_(job_types)+ ).distinct().all()++ # Pick next user in rotation+ user_id = self._get_next_user_in_rotation(users_with_jobs)++ # Claim oldest job for that user+ job = ProcessingJob.query.filter(+ ProcessingJob.user_id == user_id,+ ProcessingJob.status == 'queued'+ ).order_by(ProcessingJob.created_at.asc()).first()+```++#### Race Condition Prevention+SQLite doesn't support `FOR UPDATE SKIP LOCKED`, so we use a threading lock:++```python+def __init__(self):+ self._claim_lock = threading.Lock()++def _claim_next_job(self, job_types, queue_name):+ with self._claim_lock:+ # ... claim logic+```++#### Session Management+Jobs are claimed in one database session context, then processed in another to avoid detached object issues:++```python+def _process_job(self, job):+ # Save job attributes before context ends+ job_id = job.id+ job_type = job.job_type+ recording_id = job.recording_id+ is_new_upload = job.is_new_upload++ with self._app_context():+ # Re-fetch job in new session+ job = db.session.get(ProcessingJob, job_id)+ # ... process job+```++#### Failed Upload Cleanup+When a new upload fails permanently:++```python+if is_new_upload and recording:+ # Delete audio file+ if recording.audio_path and os.path.exists(recording.audio_path):+ os.remove(recording.audio_path)+ # Delete all processing jobs for this recording+ ProcessingJob.query.filter_by(recording_id=recording_id).delete()+ # Delete the recording+ db.session.delete(recording)+```++### 3. Job Queue API (`src/api/recordings.py`)++#### Get Job Queue Status+`GET /api/recordings/job-queue-status`++Returns all jobs for the current user (active + completed/failed from last hour):++```json+{+ "jobs": [+ {+ "id": 123,+ "recording_id": 456,+ "recording_title": "Meeting Notes",+ "job_status": "processing",+ "job_type": "transcribe",+ "queue_type": "transcription",+ "position": null,+ "is_new_upload": true,+ "error_message": null,+ "created_at": "2025-11-27T01:00:00",+ "started_at": "2025-11-27T01:00:05",+ "completed_at": null+ }+ ]+}+```++**Queue Types:**+- `transcription`: transcribe, reprocess_transcription+- `summary`: summarize, reprocess_summary++#### Retry Failed Job+`POST /api/recordings/jobs/<id>/retry`++Resets job status to `queued` for another attempt.++#### Delete Job+`DELETE /api/recordings/jobs/<id>`++Deletes a job. If it's a failed new upload, also deletes the recording and audio file.++#### Clear Completed Jobs+`POST /api/recordings/jobs/clear-completed`++Removes all completed jobs for the current user.++### 4. Recording Deletion Cascade++When deleting a recording, associated processing jobs must be deleted first due to the NOT NULL constraint on `recording_id`:++```python+# In delete_recording(), delete_job(), job_queue._process_job(), retention.py+ProcessingJob.query.filter_by(recording_id=recording_id).delete()+db.session.delete(recording)+```++## Frontend Components++### 1. Unified Progress Tracking System (`static/js/app.modular.js`)++The frontend merges multiple data sources into a single unified list to prevent duplicate entries.++#### Data Sources+1. **Backend Jobs** (`allJobs`) - Polled from `/api/recordings/job-queue-status`+2. **Upload Queue** (`uploadQueue`) - Client-side tracking of uploads+3. **Global Progress Refs** - `currentlyProcessingFile`, `processingProgress`, `processingMessage`++#### Unified Progress Items+```javascript+const unifiedProgressItems = computed(() => {+ const items = new Map(); // Key by recordingId or clientId++ // 1. Add backend jobs (most accurate status)+ for (const job of allJobs.value) {+ const key = `rec_${job.recording_id}`;+ items.set(key, {+ id: key,+ recordingId: job.recording_id,+ jobId: job.id,+ title: job.recording_title,+ status: mapJobStatus(job),+ progress: getProgress(job),+ progressMessage: getMessage(job),+ // ...+ });+ }++ // 2. Add/merge upload queue items+ for (const upload of uploadQueue.value) {+ // If recordingId exists and matches a job, merge+ // Otherwise create new entry+ // Use global progress refs for currently uploading file+ }++ // Sort by status priority+ return Array.from(items.values()).sort(byStatusPriority);+});+```++#### Unified Status States+| Status | Description | Icon | Color |+|--------|-------------|------|-------|+| `uploading` | File uploading to server | cloud-upload-alt | blue |+| `transcribing` | Server transcribing audio | microphone-alt | purple |+| `summarizing` | Server generating summary | file-alt | green |+| `queued` | Waiting in server queue | clock | yellow |+| `ready` | Waiting to upload | clock | gray |+| `completed` | Processing finished | check-circle | green |+| `failed` | Server processing failed | exclamation-circle | red |+| `upload_failed` | Upload failed | exclamation-circle | red |++#### Filtered Views+```javascript+const activeProgressItems = computed(() =>+ unifiedProgressItems.value.filter(item =>+ ['uploading', 'transcribing', 'summarizing', 'queued', 'ready'].includes(item.status)+ )+);++const completedProgressItems = computed(() =>+ unifiedProgressItems.value.filter(item => item.status === 'completed')+);++const failedProgressItems = computed(() =>+ unifiedProgressItems.value.filter(item =>+ ['failed', 'upload_failed'].includes(item.status)+ )+);+```++#### Key Deduplication Logic+- Items keyed by `rec_${recordingId}` when recordingId is known+- Items keyed by `client_${clientId}` for uploads without recordingId yet+- When upload completes and gets recordingId, it merges with any existing job entry+- Backend job data takes precedence for status (more accurate)+- Upload progress refs used for live progress during upload phase++### 2. Progress Popup Template (`templates/components/progress-popup.html`)++Displays unified progress items in a single list:++```html+<!-- Active Items -->+<div v-for="item in activeProgressItems" :key="item.id">+ <i :class="getStatusDisplay(item.status).icon"></i>+ <span>${item.title}</span>+ <div v-if="item.status === 'uploading'" class="progress-bar">+ <div :style="{width: item.progress + '%'}"></div>+ </div>+ <p>${item.progressMessage}</p>+</div>++<!-- Failed Items (with retry/delete buttons) -->+<div v-for="item in failedProgressItems" :key="item.id">+ <span>${item.title}</span>+ <span v-if="item.errorMessage">${item.errorMessage}</span>+ <button @click="retryProgressItem(item)">Retry</button>+ <button @click="removeProgressItem(item)">Delete</button>+</div>++<!-- Completed Items -->+<div v-for="item in completedProgressItems" :key="item.id">+ <span>${item.title}</span>+ <span>Done</span>+</div>+```++## Job Lifecycle Flow++### New Upload Flow+```+1. User drops file+ └─▶ uploadQueue.push({status: 'queued', clientId: 'client-xxx'})++2. User clicks Upload+ └─▶ item.status = 'ready'++3. Upload starts+ └─▶ item.status = 'uploading'+ └─▶ processingProgress updates (10%, 20%, ...)++4. Upload completes (HTTP 202)+ └─▶ item.recordingId = response.id+ └─▶ item.status = 'pending'+ └─▶ Recording created in DB with status='PENDING'+ └─▶ ProcessingJob created with status='queued', is_new_upload=true++5. Job Queue Worker claims job+ └─▶ job.status = 'processing'+ └─▶ Recording.status = 'PROCESSING'++6. Transcription completes+ └─▶ job.status = 'completed' (if no auto-summary)+ └─▶ OR Recording.status = 'SUMMARIZING' (if auto-summary)++7. Summary completes (if applicable)+ └─▶ Summary job.status = 'completed'+ └─▶ Recording.status = 'COMPLETED'+```++### Reprocessing Flow+```+1. User clicks Reprocess+ └─▶ ProcessingJob created with is_new_upload=false++2. Job Queue Worker processes+ └─▶ On failure: Recording.status = 'FAILED' (recording NOT deleted)++3. User can retry or manually fix+```++## Database Migrations++The `is_new_upload` column was added via auto-migration in `src/init_db.py`:++```python+if add_column_if_not_exists(engine, 'processing_job', 'is_new_upload', 'BOOLEAN DEFAULT 0'):+ app.logger.info("Added is_new_upload column to processing_job table")+```++## Error Handling++### Upload Failures+- Stored in IndexedDB for background sync retry+- Shown in progress popup with retry button++### Processing Failures+- `is_new_upload=true`: Recording and audio file deleted+- `is_new_upload=false`: Recording marked as FAILED, can be retried++### Race Conditions+- Threading lock prevents multiple workers claiming same job+- SQLite WAL mode enabled for better concurrency++## API Endpoints Summary++| Endpoint | Method | Description |+|----------|--------|-------------|+| `/api/recordings/job-queue-status` | GET | Get all jobs for current user |+| `/api/recordings/jobs/<id>/retry` | POST | Retry a failed job |+| `/api/recordings/jobs/<id>` | DELETE | Delete a job (and recording if failed new upload) |+| `/api/recordings/jobs/clear-completed` | POST | Clear all completed jobs |++## Future Improvements++1. **WebSocket Support**: Replace polling with WebSocket for real-time updates+2. **Job Priority**: Add priority levels for urgent jobs+3. **Batch Operations**: Support batch retry/delete operations+4. **Progress Estimation**: More accurate progress based on file size/duration+5. **Distributed Workers**: Support for multiple worker processes/servers+6. **Job Cancellation**: Allow canceling in-progress jobsdiff --git a/src/api/recordings.py b/src/api/recordings.pyindex a1f032f1..e066effd 100644--- a/src/api/recordings.py+++ b/src/api/recordings.py@@ -2083,7 +2083,8 @@ def upload_file():user_id=current_user.id,recording_id=recording.id,job_type='transcribe',- params=job_params+ params=job_params,+ is_new_upload=True)current_app.logger.info(f"Transcription job queued for recording ID: {recording.id}")@@ -2135,6 +2136,12 @@ def delete_recording(recording_id):if chunk_count > 0:current_app.logger.info(f"Deleting {chunk_count} transcript chunks with embeddings for recording {recording_id}")+ # Delete associated processing jobs (required because recording_id is NOT NULL)+ from src.models.processing_job import ProcessingJob+ deleted_jobs = ProcessingJob.query.filter_by(recording_id=recording_id).delete()+ if deleted_jobs > 0:+ current_app.logger.info(f"Deleted {deleted_jobs} processing jobs for recording {recording_id}")+# Delete the database record (cascade will handle chunks/embeddings)db.session.delete(recording)db.session.commit()@@ -2355,6 +2362,175 @@ def get_batch_recording_status():return jsonify({'error': 'An unexpected error occurred.'}), 500+@recordings_bp.route('/api/recordings/job-queue-status', methods=['GET'])+@login_required+def get_job_queue_status():+ """+ Get detailed job queue status for all jobs (active, completed, and failed).+ Returns status for the user's jobs within the last hour.+ """+ try:+ from src.models import ProcessingJob+ from src.services.job_queue import TRANSCRIPTION_JOBS, SUMMARY_JOBS+ from datetime import timedelta++ # Expire all cached objects to ensure we see latest data from worker threads+ db.session.expire_all()++ # Get all jobs for the user (active + recent completed/failed within last hour)+ cutoff_time = datetime.utcnow() - timedelta(hours=1)+ all_jobs = ProcessingJob.query.filter(+ ProcessingJob.user_id == current_user.id,+ db.or_(+ ProcessingJob.status.in_(['queued', 'processing']),+ db.and_(+ ProcessingJob.status.in_(['completed', 'failed']),+ ProcessingJob.completed_at >= cutoff_time+ )+ )+ ).order_by(ProcessingJob.created_at.desc()).all()++ job_details = []+ for job in all_jobs:+ recording = db.session.get(Recording, job.recording_id)+ recording_title = None+ if recording:+ recording_title = recording.title or recording.original_filename or 'Untitled'++ # Determine queue type+ queue_type = 'summary' if job.job_type in SUMMARY_JOBS else 'transcription'++ # Calculate position if queued+ position = None+ if job.status == 'queued':+ job_types = SUMMARY_JOBS if job.job_type in SUMMARY_JOBS else TRANSCRIPTION_JOBS+ ahead_in_queue = ProcessingJob.query.filter(+ ProcessingJob.status == 'queued',+ ProcessingJob.job_type.in_(job_types),+ ProcessingJob.created_at < job.created_at+ ).count()+ currently_processing = ProcessingJob.query.filter(+ ProcessingJob.status == 'processing',+ ProcessingJob.job_type.in_(job_types)+ ).count()+ position = ahead_in_queue + currently_processing + 1++ job_details.append({+ 'id': job.id,+ 'recording_id': job.recording_id,+ 'recording_title': recording_title,+ 'job_status': job.status,+ 'job_type': job.job_type,+ 'queue_type': queue_type,+ 'position': position,+ 'is_new_upload': job.is_new_upload,+ 'error_message': job.error_message,+ 'created_at': job.created_at.isoformat() if job.created_at else None,+ 'started_at': job.started_at.isoformat() if job.started_at else None,+ 'completed_at': job.completed_at.isoformat() if job.completed_at else None+ })++ return jsonify({'jobs': job_details})+ except Exception as e:+ current_app.logger.error(f"Error fetching job queue status: {e}", exc_info=True)+ return jsonify({'error': 'An unexpected error occurred.'}), 500+++@recordings_bp.route('/api/recordings/jobs/<int:job_id>/retry', methods=['POST'])+@login_required+def retry_failed_job(job_id):+ """Retry a failed job."""+ try:+ from src.models import ProcessingJob++ job = db.session.get(ProcessingJob, job_id)+ if not job:+ return jsonify({'error': 'Job not found'}), 404++ if job.user_id != current_user.id:+ return jsonify({'error': 'Access denied'}), 403++ if job.status != 'failed':+ return jsonify({'error': 'Only failed jobs can be retried'}), 400++ # Reset job for retry+ job.status = 'queued'+ job.error_message = None+ job.retry_count = 0+ job.started_at = None+ job.completed_at = None+ db.session.commit()++ current_app.logger.info(f"Job {job_id} queued for retry by user {current_user.id}")+ return jsonify({'success': True, 'message': 'Job queued for retry'})++ except Exception as e:+ current_app.logger.error(f"Error retrying job {job_id}: {e}", exc_info=True)+ return jsonify({'error': 'An unexpected error occurred.'}), 500+++@recordings_bp.route('/api/recordings/jobs/<int:job_id>', methods=['DELETE'])+@login_required+def delete_job(job_id):+ """Delete a job (clear from queue or history)."""+ try:+ from src.models import ProcessingJob+ import os++ job = db.session.get(ProcessingJob, job_id)+ if not job:+ return jsonify({'error': 'Job not found'}), 404++ if job.user_id != current_user.id:+ return jsonify({'error': 'Access denied'}), 403++ # If it's a failed new upload, also delete the recording+ if job.status == 'failed' and job.is_new_upload:+ recording = db.session.get(Recording, job.recording_id)+ if recording:+ # Delete audio file+ if recording.audio_path and os.path.exists(recording.audio_path):+ try:+ os.remove(recording.audio_path)+ except Exception as e:+ current_app.logger.error(f"Error deleting audio file: {e}")+ # Delete ALL processing jobs for this recording first+ ProcessingJob.query.filter_by(recording_id=recording.id).delete()+ db.session.delete(recording)+ else:+ # Just delete this job+ db.session.delete(job)+ db.session.commit()++ current_app.logger.info(f"Job {job_id} deleted by user {current_user.id}")+ return jsonify({'success': True, 'message': 'Job deleted'})++ except Exception as e:+ current_app.logger.error(f"Error deleting job {job_id}: {e}", exc_info=True)+ return jsonify({'error': 'An unexpected error occurred.'}), 500+++@recordings_bp.route('/api/recordings/jobs/clear-completed', methods=['POST'])+@login_required+def clear_completed_jobs():+ """Clear all completed jobs for the current user."""+ try:+ from src.models import ProcessingJob++ deleted = ProcessingJob.query.filter(+ ProcessingJob.user_id == current_user.id,+ ProcessingJob.status == 'completed'+ ).delete(synchronize_session=False)++ db.session.commit()+ current_app.logger.info(f"Cleared {deleted} completed jobs for user {current_user.id}")+ return jsonify({'success': True, 'deleted': deleted})++ except Exception as e:+ current_app.logger.error(f"Error clearing completed jobs: {e}", exc_info=True)+ return jsonify({'error': 'An unexpected error occurred.'}), 500++@recordings_bp.route('/status/<int:recording_id>', methods=['GET'])@login_requireddef get_status(recording_id):diff --git a/src/app.py b/src/app.pyindex d90eaefe..7f25e21f 100644--- a/src/app.py+++ b/src/app.py@@ -4,10 +4,7 @@ import sysfrom flask import Flask, render_template, request, jsonify, send_file, redirect, url_for, flash, Response, make_responsefrom urllib.parse import urlparse, urljoin, quotefrom email.utils import encode_rfc2231-try:- from flask import Markup-except ImportError:- from markupsafe import Markup+from markupsafe import Markupfrom flask_sqlalchemy import SQLAlchemyfrom datetime import datetime, timedeltafrom openai import OpenAI # Keep using the OpenAI library@@ -434,6 +431,9 @@ def exempt_status_endpoints():return Trueif request.path.endswith('/batch-status') and request.method == 'POST':return True+ # Exempt job queue status polling (polled every 5-30 seconds during processing)+ if request.path == '/api/recordings/job-queue-status' and request.method == 'GET':+ return Truereturn Falsecsrf = CSRFProtect(app)diff --git a/src/file_monitor.py b/src/file_monitor.pyindex acadc299..03a08354 100644--- a/src/file_monitor.py+++ b/src/file_monitor.py@@ -342,7 +342,8 @@ class FileMonitor:user_id=user.id,recording_id=recording.id,job_type='transcribe',- params={}+ params={},+ is_new_upload=True)self.logger.info(f"Queued background processing for recording ID: {recording.id}")diff --git a/src/init_db.py b/src/init_db.pyindex 7c00397c..4ba41a81 100644--- a/src/init_db.py+++ b/src/init_db.py@@ -116,6 +116,10 @@ def initialize_database(app):if add_column_if_not_exists(engine, 'speaker', 'confidence_score', 'REAL'):app.logger.info("Added confidence_score column to speaker table")+ # Add is_new_upload column to processing_job table for tracking upload vs reprocessing jobs+ if add_column_if_not_exists(engine, 'processing_job', 'is_new_upload', 'BOOLEAN DEFAULT 0'):+ app.logger.info("Added is_new_upload column to processing_job table")+if add_column_if_not_exists(engine, 'tag', 'group_id', 'INTEGER'):app.logger.info("Added group_id column to tag table")diff --git a/src/models/processing_job.py b/src/models/processing_job.pyindex 8c487e8f..c8bdc09a 100644--- a/src/models/processing_job.py+++ b/src/models/processing_job.py@@ -31,6 +31,9 @@ class ProcessingJob(db.Model):error_message = db.Column(db.Text, nullable=True)retry_count = db.Column(db.Integer, default=0, nullable=False)+ # Track if this is a new upload (vs reprocessing) - for cleanup on failure+ is_new_upload = db.Column(db.Boolean, default=False, nullable=False)+# Timestampscreated_at = db.Column(db.DateTime, default=datetime.utcnow, nullable=False, index=True)started_at = db.Column(db.DateTime, nullable=True)@@ -52,6 +55,7 @@ class ProcessingJob(db.Model):'job_type': self.job_type,'status': self.status,'retry_count': self.retry_count,+ 'is_new_upload': self.is_new_upload,'created_at': self.created_at.isoformat() if self.created_at else None,'started_at': self.started_at.isoformat() if self.started_at else None,'completed_at': self.completed_at.isoformat() if self.completed_at else None,diff --git a/src/services/job_queue.py b/src/services/job_queue.pyindex 6f884fd1..30548634 100644--- a/src/services/job_queue.py+++ b/src/services/job_queue.py@@ -63,6 +63,8 @@ class FairJobQueue:# Separate round-robin tracking for each queueself._last_user_id_transcription = Noneself._last_user_id_summary = None+ # Lock for claiming jobs (SQLite doesn't support row-level locking)+ self._claim_lock = threading.Lock()self._initialized = Truelogger.info(f"FairJobQueue initialized: {TRANSCRIPTION_WORKERS} transcription workers, {SUMMARY_WORKERS} summary workers")@@ -144,75 +146,90 @@ class FairJobQueue:Returns the claimed job or None if no jobs available."""- with self._app_context():- from src.database import db- from src.models import ProcessingJob-- try:- # Get list of users with queued jobs of our types- users_with_jobs = db.session.query(- ProcessingJob.user_id- ).filter(- ProcessingJob.status == 'queued',- ProcessingJob.job_type.in_(job_types)- ).group_by(- ProcessingJob.user_id- ).order_by(- db.func.min(ProcessingJob.created_at)- ).all()-- if not users_with_jobs:- return None-- user_ids = [u[0] for u in users_with_jobs]-- # Get last user ID for this queue type- last_user_id = (self._last_user_id_transcription- if queue_name == 'transcription'- else self._last_user_id_summary)-- # Round-robin: pick next user after last processed- next_user_id = None- if last_user_id is not None and last_user_id in user_ids:- idx = user_ids.index(last_user_id)- next_user_id = user_ids[(idx + 1) % len(user_ids)]- else:- next_user_id = user_ids[0]-- # Get oldest queued job of our types for this user- job = ProcessingJob.query.filter(- ProcessingJob.user_id == next_user_id,- ProcessingJob.status == 'queued',- ProcessingJob.job_type.in_(job_types)- ).order_by(- ProcessingJob.created_at- ).with_for_update(skip_locked=True).first()-- if job:- # Claim the job- job.status = 'processing'- job.started_at = datetime.utcnow()- db.session.commit()-- # Update last user ID for this queue- if queue_name == 'transcription':- self._last_user_id_transcription = next_user_id+ # Use lock to prevent race conditions (SQLite doesn't support row-level locking)+ with self._claim_lock:+ with self._app_context():+ from src.database import db+ from src.models import ProcessingJob++ try:+ # Get list of users with queued jobs of our types+ users_with_jobs = db.session.query(+ ProcessingJob.user_id+ ).filter(+ ProcessingJob.status == 'queued',+ ProcessingJob.job_type.in_(job_types)+ ).group_by(+ ProcessingJob.user_id+ ).order_by(+ db.func.min(ProcessingJob.created_at)+ ).all()++ if not users_with_jobs:+ return None++ user_ids = [u[0] for u in users_with_jobs]++ # Get last user ID for this queue type+ last_user_id = (self._last_user_id_transcription+ if queue_name == 'transcription'+ else self._last_user_id_summary)++ # Round-robin: pick next user after last processed+ next_user_id = None+ if last_user_id is not None and last_user_id in user_ids:+ idx = user_ids.index(last_user_id)+ next_user_id = user_ids[(idx + 1) % len(user_ids)]else:- self._last_user_id_summary = next_user_id-- wait_time = (datetime.utcnow() - job.created_at).total_seconds()- logger.info(f"[{queue_name.upper()}] Claimed job {job.id} (type={job.job_type}) for user {job.user_id}, recording {job.recording_id} (waited {wait_time:.1f}s)")- return job+ next_user_id = user_ids[0]++ # Get oldest queued job of our types for this user+ job = ProcessingJob.query.filter(+ ProcessingJob.user_id == next_user_id,+ ProcessingJob.status == 'queued',+ ProcessingJob.job_type.in_(job_types)+ ).order_by(+ ProcessingJob.created_at+ ).first()++ if job:+ # Claim the job+ job.status = 'processing'+ job.started_at = datetime.utcnow()++ # Also update Recording.status to reflect active processing+ from src.models import Recording+ recording = db.session.get(Recording, job.recording_id)+ if recording and recording.status == 'QUEUED':+ recording.status = 'PROCESSING'++ db.session.commit()++ # Update last user ID for this queue+ if queue_name == 'transcription':+ self._last_user_id_transcription = next_user_id+ else:+ self._last_user_id_summary = next_user_id++ wait_time = (datetime.utcnow() - job.created_at).total_seconds()+ logger.info(f"[{queue_name.upper()}] Claimed job {job.id} (type={job.job_type}) for user {job.user_id}, recording {job.recording_id} (waited {wait_time:.1f}s)")+ return job- return None+ return None- except Exception as e:- logger.error(f"Error claiming {queue_name} job: {e}", exc_info=True)- db.session.rollback()- return None+ except Exception as e:+ logger.error(f"Error claiming {queue_name} job: {e}", exc_info=True)+ db.session.rollback()+ return Nonedef _process_job(self, job):"""Process a single job by dispatching to the appropriate task function."""+ job_id = job.id+ job_type = job.job_type+ recording_id = job.recording_id+ params_str = job.params+ is_new_upload = job.is_new_upload+with self._app_context():from src.database import dbfrom src.models import ProcessingJob, Recording@@ -220,55 +237,80 @@ class FairJobQueue:try:# Parse job parameters- params = json.loads(job.params) if job.params else {}+ params = json.loads(params_str) if params_str else {}++ # Re-fetch the job in this session context to ensure it's attached+ job = db.session.get(ProcessingJob, job_id)+ if not job:+ logger.error(f"Job {job_id} not found when trying to process")+ return# Get recording- recording = db.session.get(Recording, job.recording_id)+ recording = db.session.get(Recording, recording_id)if not recording:- raise ValueError(f"Recording {job.recording_id} not found")+ raise ValueError(f"Recording {recording_id} not found")# Dispatch based on job type- if job.job_type == 'transcribe':+ if job_type == 'transcribe':self._run_transcription(job, recording, params)- elif job.job_type == 'summarize':+ elif job_type == 'summarize':self._run_summarization(job, recording, params)- elif job.job_type == 'reprocess_transcription':+ elif job_type == 'reprocess_transcription':self._run_reprocess_transcription(job, recording, params)- elif job.job_type == 'reprocess_summary':+ elif job_type == 'reprocess_summary':self._run_reprocess_summary(job, recording, params)else:- raise ValueError(f"Unknown job type: {job.job_type}")+ raise ValueError(f"Unknown job type: {job_type}")- # Mark as completed- job.status = 'completed'- job.completed_at = datetime.utcnow()- db.session.commit()-- logger.info(f"Job {job.id} completed successfully")+ # Mark as completed - re-fetch to ensure we have latest state+ job = db.session.get(ProcessingJob, job_id)+ if job:+ job.status = 'completed'+ job.completed_at = datetime.utcnow()+ db.session.commit()+ logger.info(f"Job {job_id} completed successfully")except Exception as e:- logger.error(f"Job {job.id} failed: {e}", exc_info=True)-- # Update job with error- job.error_message = str(e)- job.retry_count += 1+ logger.error(f"Job {job_id} failed: {e}", exc_info=True)- if job.retry_count < MAX_RETRIES:- # Re-queue for retry- job.status = 'queued'- job.started_at = None- logger.info(f"Job {job.id} re-queued for retry ({job.retry_count}/{MAX_RETRIES})")- else:- job.status = 'failed'- job.completed_at = datetime.utcnow()- # Update recording status to FAILED- recording = db.session.get(Recording, job.recording_id)- if recording:- recording.status = 'FAILED'- recording.error_message = str(e)- logger.error(f"Job {job.id} failed permanently after {MAX_RETRIES} retries")+ # Re-fetch job to update it+ job = db.session.get(ProcessingJob, job_id)+ if job:+ job.error_message = str(e)+ job.retry_count += 1++ if job.retry_count < MAX_RETRIES:+ # Re-queue for retry+ job.status = 'queued'+ job.started_at = None+ logger.info(f"Job {job_id} re-queued for retry ({job.retry_count}/{MAX_RETRIES})")+ else:+ job.status = 'failed'+ job.completed_at = datetime.utcnow()+ recording = db.session.get(Recording, recording_id)++ if is_new_upload and recording:+ # For failed new uploads, delete the recording and file+ logger.info(f"Deleting failed new upload: recording {recording_id}")+ try:+ # Delete the audio file+ if recording.audio_path and os.path.exists(recording.audio_path):+ os.remove(recording.audio_path)+ logger.info(f"Deleted audio file: {recording.audio_path}")+ # Delete ALL processing jobs for this recording (required due to NOT NULL constraint)+ ProcessingJob.query.filter_by(recording_id=recording_id).delete()+ # Delete the recording from database+ db.session.delete(recording)+ except Exception as delete_err:+ logger.error(f"Error deleting failed upload recording: {delete_err}")+ elif recording:+ # For reprocessing failures, just mark as failed+ recording.status = 'FAILED'+ recording.error_message = str(e)++ logger.error(f"Job {job_id} failed permanently after {MAX_RETRIES} retries")- db.session.commit()+ db.session.commit()def _run_transcription(self, job, recording, params):"""Run transcription task. Status updates handled by task function."""@@ -339,7 +381,8 @@ class FairJobQueue:user_id: int,recording_id: int,job_type: str,- params: Dict[str, Any] = None+ params: Dict[str, Any] = None,+ is_new_upload: bool = False) -> int:"""Add a job to the database queue.@@ -349,6 +392,7 @@ class FairJobQueue:recording_id: ID of the recording to processjob_type: Type of job (transcribe, summarize, reprocess_transcription, reprocess_summary)params: Optional parameters for the job+ is_new_upload: True if this is a new file upload (for cleanup on failure)Returns:The created job ID@@ -374,7 +418,8 @@ class FairJobQueue:user_id=user_id,recording_id=recording_id,job_type=job_type,- params=json.dumps(params) if params else None+ params=json.dumps(params) if params else None,+ is_new_upload=is_new_upload)db.session.add(job)diff --git a/src/services/retention.py b/src/services/retention.pyindex a85b29b4..35af4d71 100644--- a/src/services/retention.py+++ b/src/services/retention.py@@ -176,6 +176,10 @@ def process_auto_deletion():if recording.audio_path and os.path.exists(recording.audio_path):os.remove(recording.audio_path)+ # Delete associated processing jobs (required due to NOT NULL constraint)+ from src.models.processing_job import ProcessingJob+ ProcessingJob.query.filter_by(recording_id=recording.id).delete()+# Delete the database record (cascades to chunks, shares, etc.)db.session.delete(recording)db.session.commit()diff --git a/static/js/app.modular.js b/static/js/app.modular.jsindex 20ee2e13..349a29bc 100644--- a/static/js/app.modular.js+++ b/static/js/app.modular.js@@ -1087,7 +1087,14 @@ document.addEventListener('DOMContentLoaded', async () => {// Upload queue computed propertiesconst totalInQueue = computed(() => uploadQueue.value.length);const completedInQueue = computed(() => uploadQueue.value.filter(item => item.status === 'completed' || item.status === 'failed').length);- const finishedFilesInQueue = computed(() => uploadQueue.value.filter(item => ['completed', 'failed'].includes(item.status)));+ // Filter out upload completions that already have a backend job (to avoid duplicates)+ const finishedFilesInQueue = computed(() => {+ const backendRecordingIds = new Set(allJobs.value.map(j => j.recording_id));+ return uploadQueue.value.filter(item =>+ ['completed', 'failed'].includes(item.status) &&+ !backendRecordingIds.has(item.recordingId)+ );+ });const waitingFilesInQueue = computed(() => uploadQueue.value.filter(item => item.status === 'ready'));const pendingQueueFiles = computed(() => uploadQueue.value.filter(item => item.status === 'queued'));@@ -1096,19 +1103,405 @@ document.addEventListener('DOMContentLoaded', async () => {return recordings.value.filter(r => ['PENDING', 'PROCESSING', 'SUMMARIZING', 'QUEUED'].includes(r.status));});- // Combined processing queue count (uploads + backend processing)+ // All jobs from backend (queued, processing, completed, failed)+ const allJobs = ref([]);+ let jobQueuePollInterval = null;+ let lastJobQueueFetch = 0; // Timestamp of last fetch+ const JOB_QUEUE_POLL_INTERVAL = 5000; // Poll every 5 seconds when active+ const JOB_QUEUE_FETCH_DEBOUNCE = 2000; // Minimum 2 seconds between fetches++ // Computed properties for different job states+ const activeJobs = computed(() => allJobs.value.filter(j => ['queued', 'processing'].includes(j.job_status)));+ const completedJobs = computed(() => allJobs.value.filter(j => j.job_status === 'completed'));+ const failedJobs = computed(() => allJobs.value.filter(j => j.job_status === 'failed'));++ // Job queue details map (for backward compatibility with progress popup)+ const jobQueueDetails = computed(() => {+ const detailsMap = {};+ for (const job of allJobs.value) {+ // Use recording_id as key, store the most relevant job (prefer active over completed)+ if (!detailsMap[job.recording_id] || ['queued', 'processing'].includes(job.job_status)) {+ detailsMap[job.recording_id] = job;+ }+ }+ return detailsMap;+ });++ // Fetch job queue status from backend (with debounce protection)+ const fetchJobQueueStatus = async (force = false) => {+ const now = Date.now();+ // Debounce: skip if fetched recently (unless forced)+ if (!force && (now - lastJobQueueFetch) < JOB_QUEUE_FETCH_DEBOUNCE) {+ return;+ }+ lastJobQueueFetch = now;++ try {+ const response = await fetch('/api/recordings/job-queue-status');+ if (response.ok) {+ const data = await response.json();+ allJobs.value = data.jobs || [];+ } else if (response.status === 429) {+ console.warn('Job queue polling rate limited');+ }+ } catch (error) {+ console.error('Error fetching job queue status:', error);+ }+ };++ // Start polling job queue status+ const startJobQueuePolling = () => {+ if (jobQueuePollInterval) return;+ fetchJobQueueStatus(true); // Fetch immediately (forced)+ jobQueuePollInterval = setInterval(() => fetchJobQueueStatus(true), JOB_QUEUE_POLL_INTERVAL);+ };++ const stopJobQueuePolling = () => {+ if (jobQueuePollInterval) {+ clearInterval(jobQueuePollInterval);+ jobQueuePollInterval = null;+ }+ };++ // Check if we have active items that need polling+ const hasActiveProcessing = computed(() => {+ const completedStatuses = ['completed', 'failed', 'COMPLETED', 'FAILED'];+ const hasActiveUploads = uploadQueue.value.some(item =>+ !completedStatuses.includes(item.status)+ );+ const hasActiveJobs = activeJobs.value.length > 0;+ const hasProcessingRecordings = backendProcessingRecordings.value.length > 0;+ return hasActiveUploads || hasActiveJobs || hasProcessingRecordings;+ });++ // Start/stop polling based on whether we have active items+ watch(hasActiveProcessing, (hasActive) => {+ if (hasActive) {+ startJobQueuePolling();+ } else {+ // Stop polling after a delay (to catch final status updates)+ setTimeout(() => {+ if (!hasActiveProcessing.value) {+ stopJobQueuePolling();+ }+ }, 10000);+ }+ }, { immediate: true });++ // When popup opens, do a one-time fetch to populate it+ watch(() => progressPopupClosed.value, (closed) => {+ if (!closed) {+ // Popup just opened - fetch current status+ fetchJobQueueStatus();+ }+ });++ // Get job details for a recording+ const getJobDetails = (recordingId) => {+ return jobQueueDetails.value[recordingId] || null;+ };++ // Retry a failed job+ const retryJob = async (jobId) => {+ try {+ const response = await fetch(`/api/recordings/jobs/${jobId}/retry`, {+ method: 'POST',+ headers: { 'Content-Type': 'application/json' }+ });+ if (response.ok) {+ fetchJobQueueStatus();+ showToast('Job queued for retry', 'success');+ } else {+ const data = await response.json();+ showToast(data.error || 'Failed to retry job', 'error');+ }+ } catch (error) {+ console.error('Error retrying job:', error);+ showToast('Failed to retry job', 'error');+ }+ };++ // Delete/clear a job+ const deleteJob = async (jobId) => {+ try {+ const response = await fetch(`/api/recordings/jobs/${jobId}`, {+ method: 'DELETE'+ });+ if (response.ok) {+ fetchJobQueueStatus();+ } else {+ const data = await response.json();+ showToast(data.error || 'Failed to delete job', 'error');+ }+ } catch (error) {+ console.error('Error deleting job:', error);+ }+ };++ // Clear all completed jobs+ const clearCompletedJobs = async () => {+ try {+ const response = await fetch('/api/recordings/jobs/clear-completed', {+ method: 'POST',+ headers: { 'Content-Type': 'application/json' }+ });+ if (response.ok) {+ // Clear upload queue completed/failed items+ uploadQueue.value = uploadQueue.value.filter(item =>+ !['completed', 'failed', 'COMPLETED', 'FAILED'].includes(item.status)+ );+ // Force fetch to update the job list (bypass debounce)+ await fetchJobQueueStatus(true);+ }+ } catch (error) {+ console.error('Error clearing completed jobs:', error);+ }+ };++ // Combined clear function for backward compatibility+ const clearAllCompleted = () => {+ clearCompletedJobs();+ };++ // ============================================+ // UNIFIED PROGRESS TRACKING SYSTEM+ // Merges upload queue, backend recordings, and job queue into single list+ // Each recording appears ONCE with its current status+ // ============================================+ const unifiedProgressItems = computed(() => {+ const items = new Map(); // Key by recordingId or clientId++ // Get the currently uploading file's clientId for special handling+ const currentUploadClientId = currentlyProcessingFile.value?.clientId;++ // 1. First, add all backend jobs (these have the most accurate status)+ for (const job of allJobs.value) {+ const key = `rec_${job.recording_id}`;+ const existing = items.get(key);++ // Determine unified status from job+ let unifiedStatus = 'queued';+ if (job.job_status === 'processing') {+ unifiedStatus = job.queue_type === 'summary' ? 'summarizing' : 'transcribing';+ } else if (job.job_status === 'completed') {+ unifiedStatus = 'completed';+ } else if (job.job_status === 'failed') {+ unifiedStatus = 'failed';+ }++ // Prefer active jobs over completed/failed+ if (!existing || ['queued', 'transcribing', 'summarizing'].includes(unifiedStatus)) {+ items.set(key, {+ id: key,+ recordingId: job.recording_id,+ jobId: job.id,+ clientId: null,+ title: job.recording_title || 'Untitled',+ status: unifiedStatus,+ progress: unifiedStatus === 'transcribing' ? 50 : (unifiedStatus === 'summarizing' ? 80 : null),+ progressMessage: unifiedStatus === 'queued' ? `#${job.position || '?'} in queue` :+ unifiedStatus === 'transcribing' ? 'Transcribing audio...' :+ unifiedStatus === 'summarizing' ? 'Generating summary...' :+ unifiedStatus === 'completed' ? 'Done' : 'Failed',+ queuePosition: job.position,+ errorMessage: job.error_message,+ completedAt: job.completed_at,+ source: 'job'+ });+ }+ }++ // 2. Add upload queue items (client-side tracking)+ for (const upload of uploadQueue.value) {+ // Check if this is the currently uploading file+ const isCurrentUpload = upload.clientId === currentUploadClientId;++ // If we have a recordingId and it's already tracked from jobs, check if we should merge+ if (upload.recordingId) {+ const key = `rec_${upload.recordingId}`;+ const existing = items.get(key);++ if (existing) {+ // Merge upload info into existing item+ existing.clientId = upload.clientId;+ existing.file = upload.file;+ // If this is the current upload, use the global progress refs+ if (isCurrentUpload && ['uploading', 'pending', 'processing', 'summarizing'].includes(upload.status)) {+ // Map upload status to unified status+ if (upload.status === 'uploading') {+ existing.status = 'uploading';+ } else if (upload.status === 'summarizing') {+ existing.status = 'summarizing';+ } else if (upload.status === 'processing' || upload.status === 'pending') {+ existing.status = 'transcribing';+ }+ existing.progress = processingProgress.value;+ existing.progressMessage = processingMessage.value || 'Processing...';+ existing.title = upload.displayName || upload.file?.name || existing.title;+ }+ continue;+ }+ }++ // Determine unified status from upload status+ let unifiedStatus = 'ready';+ let progressVal = 0;+ let progressMsg = 'Waiting to upload...';++ // If this is the currently processing file, use global progress refs+ if (isCurrentUpload) {+ progressVal = processingProgress.value;+ progressMsg = processingMessage.value || 'Processing...';++ if (upload.status === 'uploading') {+ unifiedStatus = 'uploading';+ } else if (upload.status === 'pending' || upload.status === 'processing' || upload.status === 'PROCESSING') {+ unifiedStatus = 'transcribing';+ } else if (upload.status === 'summarizing' || upload.status === 'SUMMARIZING') {+ unifiedStatus = 'summarizing';+ } else if (upload.status === 'completed' || upload.status === 'COMPLETED') {+ unifiedStatus = 'completed';+ progressMsg = 'Done';+ } else if (upload.status === 'failed' || upload.status === 'FAILED') {+ unifiedStatus = 'failed';+ progressMsg = upload.error || 'Processing failed';+ } else if (upload.status === 'ready') {+ unifiedStatus = 'ready';+ progressMsg = 'Waiting to upload...';+ }+ } else {+ // Not the current upload - use item's own status+ if (upload.status === 'uploading') {+ unifiedStatus = 'uploading';+ progressMsg = 'Uploading...';+ } else if (upload.status === 'completed' || upload.status === 'COMPLETED') {+ unifiedStatus = 'completed';+ progressMsg = 'Done';+ } else if (upload.status === 'failed' || upload.status === 'FAILED') {+ unifiedStatus = 'upload_failed';+ progressMsg = upload.error || 'Upload failed';+ } else if (upload.status === 'ready') {+ unifiedStatus = 'ready';+ progressMsg = 'Waiting to upload...';+ } else if (upload.status === 'queued') {+ unifiedStatus = 'ready';+ progressMsg = 'Waiting to upload...';+ }+ }++ const key = upload.recordingId ? `rec_${upload.recordingId}` : `client_${upload.clientId}`;++ // Skip if we already have an entry with the same recordingId (from jobs)+ if (upload.recordingId && items.has(key)) {+ continue;+ }++ items.set(key, {+ id: key,+ recordingId: upload.recordingId,+ jobId: null,+ clientId: upload.clientId,+ title: upload.displayName || upload.file?.name || 'Unknown file',+ status: unifiedStatus,+ progress: progressVal,+ progressMessage: progressMsg,+ queuePosition: null,+ errorMessage: upload.status === 'failed' ? upload.error : null,+ file: upload.file,+ source: 'upload'+ });+ }++ // Convert to array and sort: active first, then by status priority+ const statusOrder = {+ 'uploading': 1,+ 'transcribing': 2,+ 'summarizing': 3,+ 'queued': 4,+ 'ready': 5,+ 'completed': 6,+ 'failed': 7,+ 'upload_failed': 8+ };++ return Array.from(items.values()).sort((a, b) => {+ return (statusOrder[a.status] || 99) - (statusOrder[b.status] || 99);+ });+ });++ // Filtered views of unified items+ const activeProgressItems = computed(() =>+ unifiedProgressItems.value.filter(item =>+ ['uploading', 'transcribing', 'summarizing', 'queued', 'ready'].includes(item.status)+ )+ );++ const completedProgressItems = computed(() =>+ unifiedProgressItems.value.filter(item => item.status === 'completed')+ );++ const failedProgressItems = computed(() =>+ unifiedProgressItems.value.filter(item =>+ ['failed', 'upload_failed'].includes(item.status)+ )+ );++ // Helper to get status display info+ const getStatusDisplay = (status) => {+ const displays = {+ 'ready': { label: 'Waiting', color: 'gray', icon: 'fa-clock' },+ 'uploading': { label: 'Uploading', color: 'blue', icon: 'fa-cloud-upload-alt', animate: true },+ 'queued': { label: 'Queued', color: 'yellow', icon: 'fa-clock' },+ 'transcribing': { label: 'Transcribing', color: 'purple', icon: 'fa-microphone-alt', animate: true },+ 'summarizing': { label: 'Summarizing', color: 'green', icon: 'fa-file-alt', animate: true },+ 'completed': { label: 'Done', color: 'green', icon: 'fa-check-circle' },+ 'failed': { label: 'Failed', color: 'red', icon: 'fa-exclamation-circle' },+ 'upload_failed': { label: 'Upload Failed', color: 'red', icon: 'fa-exclamation-circle' }+ };+ return displays[status] || displays['ready'];+ };++ // Cancel/remove an item from the queue+ const removeProgressItem = async (item) => {+ if (item.jobId && ['failed', 'completed'].includes(item.status)) {+ // Delete backend job+ await deleteJob(item.jobId);+ } else if (item.clientId && !item.jobId) {+ // Remove from upload queue+ uploadQueue.value = uploadQueue.value.filter(u => u.clientId !== item.clientId);+ }+ };++ // Retry a failed item+ const retryProgressItem = async (item) => {+ if (item.jobId) {+ await retryJob(item.jobId);+ }+ };++ // Track recently completed for backward compat (now using allJobs)+ const recentlyCompletedBackend = computed(() => {+ return completedJobs.value.map(j => ({+ id: j.recording_id,+ title: j.recording_title || 'Untitled',+ status: 'completed',+ completedAt: j.completed_at+ }));+ });++ // Combined processing queue countconst totalProcessingCount = computed(() => {- // Count active uploads (not completed/failed)- const activeUploads = uploadQueue.value.filter(item => !['completed', 'failed'].includes(item.status)).length;- // Count backend processing (excluding items that are in uploadQueue to avoid double counting)- const uploadRecordingIds = new Set(uploadQueue.value.map(item => item.recordingId).filter(id => id));- const backendOnly = backendProcessingRecordings.value.filter(r => !uploadRecordingIds.has(r.id)).length;- return activeUploads + backendOnly;+ return activeProgressItems.value.length;});// Should show the processing popupconst showProcessingPopup = computed(() => {- return uploadQueue.value.length > 0 || backendProcessingRecordings.value.length > 0;+ return unifiedProgressItems.value.length > 0;+ });++ // All completed items count+ const allCompletedCount = computed(() => {+ return completedProgressItems.value.length + failedProgressItems.value.length;});// Speaker computed properties@@ -1422,6 +1815,26 @@ document.addEventListener('DOMContentLoaded', async () => {backendProcessingRecordings,totalProcessingCount,showProcessingPopup,+ jobQueueDetails,+ getJobDetails,+ allJobs,+ activeJobs,+ completedJobs,+ failedJobs,+ retryJob,+ deleteJob,+ clearCompletedJobs,+ recentlyCompletedBackend,+ clearAllCompleted,+ allCompletedCount,+ // Unified progress tracking+ unifiedProgressItems,+ activeProgressItems,+ completedProgressItems,+ failedProgressItems,+ getStatusDisplay,+ removeProgressItem,+ retryProgressItem,hasSpeakerNames,tagsWithCustomPrompts,getTagPromptPreview,diff --git a/static/js/modules/composables/modals.js b/static/js/modules/composables/modals.jsindex 9f8b0d4f..0400385f 100644--- a/static/js/modules/composables/modals.js+++ b/static/js/modules/composables/modals.js@@ -12,7 +12,7 @@ export function useModals(state, utils) {showShareDeleteModal, showUnifiedShareModal, showColorSchemeModal,showSystemAudioHelpModal, editingRecording, recordingToDelete, recordingToReset,selectedRecording, recordings, selectedNewTagId, tagSearchFilter,- availableTags, currentView, totalRecordings, toasts+ availableTags, currentView, totalRecordings, toasts, uploadQueue} = state;const { showToast, setGlobalError } = utils;@@ -84,19 +84,25 @@ export function useModals(state, utils) {const deleteRecording = async () => {if (!recordingToDelete.value) return;+ const deletedId = recordingToDelete.value.id;try {const csrfToken = document.querySelector('meta[name="csrf-token"]')?.getAttribute('content');- const response = await fetch(`/recording/${recordingToDelete.value.id}`, {+ const response = await fetch(`/recording/${deletedId}`, {method: 'DELETE',headers: { 'X-CSRFToken': csrfToken }});const data = await response.json();if (!response.ok) throw new Error(data.error || 'Failed to delete recording');- recordings.value = recordings.value.filter(r => r.id !== recordingToDelete.value.id);+ recordings.value = recordings.value.filter(r => r.id !== deletedId);totalRecordings.value--;- if (selectedRecording.value?.id === recordingToDelete.value.id) {+ // Also remove from upload queue if present (clears processing queue entry)+ if (uploadQueue?.value) {+ uploadQueue.value = uploadQueue.value.filter(item => item.recordingId !== deletedId);+ }++ if (selectedRecording.value?.id === deletedId) {selectedRecording.value = null;currentView.value = 'upload';}diff --git a/static/js/modules/composables/recordings.js b/static/js/modules/composables/recordings.jsindex 75efc197..504323da 100644--- a/static/js/modules/composables/recordings.js+++ b/static/js/modules/composables/recordings.js@@ -81,25 +81,10 @@ export function useRecordings(state, utils, reprocessComposable) {}}- // Handle incomplete recordings- const incompleteRecordings = recordingsList.filter(r =>- ['PENDING', 'PROCESSING', 'SUMMARIZING'].includes(r.status)- );- if (incompleteRecordings.length > 0 && !isProcessingActive.value) {- for (const recording of incompleteRecordings) {- let queueItem = uploadQueue.value.find(item => item.recordingId === recording.id);- if (!queueItem) {- queueItem = {- file: { name: recording.title || `Recording ${recording.id}`, size: recording.file_size },- status: 'queued',- recordingId: recording.id,- clientId: `reload-${recording.id}`,- error: null- };- uploadQueue.value.unshift(queueItem);- }- }- }+ // NOTE: Removed auto-queueing of incomplete recordings.+ // Backend processing recordings are now shown via backendProcessingRecordings+ // computed property, which filters recordings by status (PENDING, PROCESSING, etc.)+ // The job queue system (ProcessingJob) handles background processing.} catch (error) {console.error('Load Recordings Error:', error);diff --git a/templates/components/progress-popup.html b/templates/components/progress-popup.htmlindex 549c819c..89358b5d 100644--- a/templates/components/progress-popup.html+++ b/templates/components/progress-popup.html@@ -1,157 +1,152 @@-<!-- Processing Queue Popup -->+<!-- Processing Queue Popup - Compact Unified Progress Tracking --><div v-if="showProcessingPopup && !progressPopupClosed":class="[- 'fixed bottom-4 left-4 z-[100] w-96 transition-all duration-300',+ 'fixed bottom-4 left-4 z-[100] w-80 transition-all duration-300',progressPopupMinimized ? 'minimized' : '']">- <div class="bg-[var(--bg-secondary)] border-2 border-[var(--bg-accent)] rounded-lg shadow-2xl overflow-hidden backdrop-blur-sm bg-opacity-95- ring-4 ring-[var(--bg-accent)] ring-opacity-20"- style="box-shadow: 0 20px 25px -5px rgba(0, 0, 0, 0.1), 0 10px 10px -5px rgba(0, 0, 0, 0.04), 0 0 0 1px var(--bg-accent), 0 0 20px rgba(var(--bg-accent-rgb), 0.3)">- <!-- Progress Header -->- <div class="bg-gradient-to-r from-[var(--bg-accent)] to-[var(--bg-accent-hover)] px-4 py-3">- <!-- Top row: Title and buttons -->- <div class="flex items-center justify-between mb-2">+ <div class="bg-[var(--bg-secondary)] border border-[var(--bg-accent)] rounded-lg shadow-xl overflow-hidden backdrop-blur-sm bg-opacity-95">+ <!-- Compact Header -->+ <div class="bg-gradient-to-r from-[var(--bg-accent)] to-[var(--bg-accent-hover)] px-3 py-2">+ <div class="flex items-center justify-between"><div class="flex items-center gap-2 min-w-0">- <i class="fas fa-tasks text-gray-800 dark:text-white"></i>- <span class="font-semibold text-gray-800 dark:text-white truncate">Processing Queue</span>- <span class="text-xs bg-black dark:bg-white bg-opacity-20 dark:bg-opacity-20 px-2 py-0.5 rounded-full text-gray-800 dark:text-white font-medium">+ <i class="fas fa-tasks text-sm text-gray-800 dark:text-white"></i>+ <span class="font-medium text-sm text-gray-800 dark:text-white">Processing Queue</span>+ <span class="text-xs bg-black dark:bg-white bg-opacity-20 dark:bg-opacity-20 px-1.5 py-0.5 rounded-full text-gray-800 dark:text-white font-medium">${totalProcessingCount}</span></div>- <div class="flex items-center gap-2 flex-shrink-0">- <button v-if="completedInQueue > 0" @click="clearCompletedUploads()"- class="px-2 py-0.5 text-xs bg-black dark:bg-white bg-opacity-10 dark:bg-opacity-20 hover:bg-opacity-20 dark:hover:bg-opacity-30 rounded transition-all duration-200 text-gray-800 dark:text-white font-medium"- :title="t('buttons.clearCompleted')">+ <div class="flex items-center gap-1 flex-shrink-0">+ <button v-if="allCompletedCount > 0" @click="clearAllCompleted()"+ class="px-1.5 py-0.5 text-xs bg-black dark:bg-white bg-opacity-10 dark:bg-opacity-20 hover:bg-opacity-20 dark:hover:bg-opacity-30 rounded transition-all text-gray-800 dark:text-white">Clear</button><button @click="progressPopupMinimized = !progressPopupMinimized"class="p-1 rounded hover:bg-black dark:hover:bg-white hover:bg-opacity-10 dark:hover:bg-opacity-20 transition-colors text-gray-800 dark:text-white">- <i :class="progressPopupMinimized ? 'fas fa-chevron-up' : 'fas fa-chevron-down'"></i>+ <i :class="progressPopupMinimized ? 'fas fa-chevron-up text-xs' : 'fas fa-chevron-down text-xs'"></i></button><button @click="progressPopupClosed = true"- class="p-1 rounded hover:bg-red-600 dark:hover:bg-red-500 hover:bg-opacity-20 dark:hover:bg-opacity-30 text-gray-800 dark:text-white hover:text-white transition-colors">- <i class="fas fa-times"></i>+ class="p-1 rounded hover:bg-red-500 hover:bg-opacity-30 text-gray-800 dark:text-white hover:text-red-600 dark:hover:text-red-400 transition-colors">+ <i class="fas fa-times text-xs"></i></button></div></div>- <!-- Overall Progress Summary -->- <div class="text-xs text-gray-800 dark:text-white opacity-80">- <span v-if="backendProcessingRecordings.length > 0">- ${backendProcessingRecordings.filter(r => r.status === 'PROCESSING').length} transcribing,- ${backendProcessingRecordings.filter(r => r.status === 'SUMMARIZING').length} summarizing+ <!-- Compact Summary Line -->+ <div v-if="activeProgressItems.length > 0" class="text-xs text-gray-800 dark:text-white opacity-70 mt-1">+ <span v-if="activeProgressItems.filter(i => i.status === 'uploading').length > 0">+ ${activeProgressItems.filter(i => i.status === 'uploading').length} uploading<span v-if="activeProgressItems.filter(i => i.status !== 'uploading').length > 0">, </span></span>- <span v-if="waitingFilesInQueue.length > 0">- <span v-if="backendProcessingRecordings.length > 0">, </span>- ${waitingFilesInQueue.length} uploading+ <span v-if="activeProgressItems.filter(i => i.status === 'transcribing').length > 0">+ ${activeProgressItems.filter(i => i.status === 'transcribing').length} transcribing<span v-if="activeProgressItems.filter(i => ['summarizing', 'queued'].includes(i.status)).length > 0">, </span>+ </span>+ <span v-if="activeProgressItems.filter(i => i.status === 'summarizing').length > 0">+ ${activeProgressItems.filter(i => i.status === 'summarizing').length} summarizing<span v-if="activeProgressItems.filter(i => i.status === 'queued').length > 0">, </span>+ </span>+ <span v-if="activeProgressItems.filter(i => i.status === 'queued').length > 0">+ ${activeProgressItems.filter(i => i.status === 'queued').length} queued</span></div></div>- <!-- Progress Content -->- <div v-if="!progressPopupMinimized" class="p-3 max-h-80 overflow-y-auto">- <!-- Currently Uploading File -->- <div v-if="currentlyProcessingFile" class="mb-3 p-3 bg-blue-500 bg-opacity-10 border border-blue-400 border-opacity-30 rounded-lg">- <div class="flex items-center gap-2 mb-2">- <i class="fas fa-cloud-upload-alt text-blue-500 animate-pulse"></i>- <span class="text-xs font-semibold text-blue-600 dark:text-blue-400 uppercase tracking-wider">Uploading</span>- </div>- <div class="flex items-center justify-between mb-2">- <span class="font-medium truncate min-w-0 text-sm text-[var(--text-primary)]">${currentlyProcessingFile.file.name}</span>- <span class="text-sm flex-shrink-0 ml-2 font-bold text-blue-600 dark:text-blue-400">${processingProgress}%</span>- </div>- <div class="w-full bg-black dark:bg-[var(--bg-tertiary)] bg-opacity-20 rounded-full h-2 mb-2 overflow-hidden">- <div class="bg-gradient-to-r from-blue-500 to-blue-400 h-full rounded-full transition-all duration-300 relative"- :style="{width: processingProgress + '%'}">- <div class="absolute inset-0 bg-white opacity-30 animate-pulse"></div>+ <!-- Compact Content -->+ <div v-if="!progressPopupMinimized" class="p-2 max-h-64 overflow-y-auto space-y-1.5">+ <!-- Active Items -->+ <template v-for="item in activeProgressItems" :key="item.id">+ <div :class="[+ 'px-2 py-1.5 rounded-md transition-all',+ item.status === 'uploading' ? 'bg-blue-500/10 border-l-2 border-blue-500' :+ item.status === 'transcribing' ? 'bg-purple-500/10 border-l-2 border-purple-500' :+ item.status === 'summarizing' ? 'bg-green-500/10 border-l-2 border-green-500' :+ 'bg-[var(--bg-tertiary)]/50 border-l-2 border-yellow-500'+ ]">+ <!-- Title row with status icon -->+ <div class="flex items-center gap-2">+ <i :class="[+ 'fas text-xs',+ getStatusDisplay(item.status).icon,+ item.status === 'uploading' ? 'text-blue-500' :+ item.status === 'transcribing' ? 'text-purple-500' :+ item.status === 'summarizing' ? 'text-green-500' :+ 'text-yellow-500',+ getStatusDisplay(item.status).animate ? 'animate-pulse' : ''+ ]"></i>+ <span class="text-xs font-medium truncate flex-1 text-[var(--text-primary)]">${item.title}</span>+ <span v-if="item.status === 'uploading' && item.progress !== null"+ class="text-xs font-bold text-blue-500">${item.progress}%</span>+ <span v-else :class="[+ 'text-xs font-medium px-1.5 py-0.5 rounded',+ item.status === 'uploading' ? 'bg-blue-500/20 text-blue-500' :+ item.status === 'transcribing' ? 'bg-purple-500/20 text-purple-500' :+ item.status === 'summarizing' ? 'bg-green-500/20 text-green-500' :+ 'bg-yellow-500/20 text-yellow-500'+ ]">${getStatusDisplay(item.status).label}</span></div>- </div>- <p class="text-xs text-[var(--text-secondary)]">${processingMessage}</p>- </div>-- <!-- Backend Processing Recordings -->- <div v-if="backendProcessingRecordings.length > 0" class="space-y-2 mb-3">- <h4 class="text-xs font-semibold text-[var(--text-secondary)] uppercase tracking-wider opacity-70 flex items-center gap-2">- <i class="fas fa-server"></i>- Server Processing (${backendProcessingRecordings.length})- </h4>- <div v-for="recording in backendProcessingRecordings"- :key="'backend-' + recording.id"- @click="selectRecording(recording)"- class="p-2 bg-[var(--bg-tertiary)] bg-opacity-50 rounded-lg flex items-center gap-2 hover:bg-opacity-70 transition-all duration-200 cursor-pointer">- <!-- Status Icon -->- <div class="flex-shrink-0">- <i v-if="recording.status === 'PROCESSING'"- class="fas fa-microphone-alt text-purple-500 animate-pulse"></i>- <i v-else-if="recording.status === 'SUMMARIZING'"- class="fas fa-file-alt text-green-500 animate-pulse"></i>- <i v-else-if="recording.status === 'QUEUED' || recording.status === 'PENDING'"- class="fas fa-clock text-yellow-500"></i>+ <!-- Progress bar for uploads -->+ <div v-if="item.status === 'uploading' && item.progress !== null"+ class="mt-1 h-1 bg-black/10 dark:bg-white/10 rounded-full overflow-hidden">+ <div class="h-full bg-blue-500 rounded-full transition-all duration-300"+ :style="{width: item.progress + '%'}"></div></div>- <!-- Recording Info -->- <div class="flex-1 min-w-0">- <span class="truncate text-sm block text-[var(--text-primary)]">${recording.title || recording.original_filename || 'Untitled'}</span>+ <!-- Compact message -->+ <div class="flex items-center justify-between mt-0.5">+ <span class="text-[10px] text-[var(--text-muted)]">${item.progressMessage}</span>+ <button v-if="item.status === 'ready' && item.clientId"+ @click="removeProgressItem(item)"+ class="text-[10px] text-red-500 hover:text-red-600">+ <i class="fas fa-times"></i> Cancel+ </button></div>- <!-- Status Badge -->- <span :class="[- 'text-xs px-2 py-1 rounded-full font-medium flex-shrink-0',- recording.status === 'PROCESSING' ? 'bg-purple-500 bg-opacity-20 text-purple-600 dark:text-purple-400' :- recording.status === 'SUMMARIZING' ? 'bg-green-500 bg-opacity-20 text-green-600 dark:text-green-400' :- 'bg-yellow-500 bg-opacity-20 text-yellow-600 dark:text-yellow-400'- ]">- <span v-if="recording.status === 'PROCESSING'">Transcribing</span>- <span v-else-if="recording.status === 'SUMMARIZING'">Summarizing</span>- <span v-else>Queued</span>- </span></div>- </div>+ </template>- <!-- Waiting Upload Files -->- <div v-if="waitingFilesInQueue.length > 0" class="space-y-2 mb-3">- <h4 class="text-xs font-semibold text-[var(--text-secondary)] uppercase tracking-wider opacity-70 flex items-center gap-2">- <i class="fas fa-upload"></i>- Upload Queue (${waitingFilesInQueue.length})- </h4>- <div v-for="item in waitingFilesInQueue"- :key="item.clientId"- class="p-2 bg-[var(--bg-tertiary)] bg-opacity-50 rounded-lg flex items-center gap-2 hover:bg-opacity-70 transition-all duration-200">- <i class="fas fa-file-audio text-[var(--text-muted)] text-sm"></i>- <span class="truncate text-sm flex-1 text-[var(--text-primary)]">${item.file.name}</span>- <span class="text-xs text-[var(--text-muted)]">${formatFileSize(item.file.size)}</span>- <button @click="cancelWaitingFile(item.clientId)"- class="p-1 rounded hover:bg-red-500 hover:bg-opacity-20 transition-colors flex-shrink-0"- :title="'Remove from queue'">- <i class="fas fa-times text-[var(--text-muted)] hover:text-red-500 text-xs"></i>- </button>+ <!-- Failed Items -->+ <div v-if="failedProgressItems.length > 0" class="mt-2">+ <div class="text-[10px] font-semibold text-red-500 uppercase tracking-wider mb-1 flex items-center gap-1">+ <i class="fas fa-exclamation-triangle"></i>+ Failed (${failedProgressItems.length})+ </div>+ <div v-for="item in failedProgressItems"+ :key="item.id"+ class="px-2 py-1.5 bg-red-500/10 border-l-2 border-red-500 rounded-md mb-1">+ <div class="flex items-center gap-2">+ <i class="fas fa-exclamation-circle text-xs text-red-500"></i>+ <span class="text-xs truncate flex-1 text-[var(--text-primary)]">${item.title}</span>+ <div class="flex items-center gap-1">+ <button v-if="item.jobId" @click.stop="retryProgressItem(item)"+ class="p-1 rounded bg-blue-500/20 hover:bg-blue-500/40 transition-colors"+ title="Retry">+ <i class="fas fa-redo text-blue-500 text-[10px]"></i>+ </button>+ <button @click.stop="removeProgressItem(item)"+ class="p-1 rounded bg-red-500/20 hover:bg-red-500/40 transition-colors"+ title="Delete">+ <i class="fas fa-trash text-red-500 text-[10px]"></i>+ </button>+ </div>+ </div>+ <span v-if="item.errorMessage" class="text-[10px] text-red-400 block truncate mt-0.5">${item.errorMessage}</span></div></div>- <!-- Finished Files -->- <div v-if="finishedFilesInQueue.length > 0" class="space-y-2">- <h4 class="text-xs font-semibold text-[var(--text-secondary)] uppercase tracking-wider opacity-70 flex items-center gap-2">+ <!-- Completed Items -->+ <div v-if="completedProgressItems.length > 0" class="mt-2">+ <div class="text-[10px] font-semibold text-[var(--text-muted)] uppercase tracking-wider mb-1 flex items-center gap-1"><i class="fas fa-check-double"></i>- Recently Completed (${finishedFilesInQueue.length})- </h4>- <div v-for="item in finishedFilesInQueue"- :key="item.clientId"- class="p-2 bg-[var(--bg-tertiary)] bg-opacity-50 rounded-lg flex items-center gap-2 hover:bg-opacity-70 transition-all duration-200">- <i :class="[- 'fas',- item.status === 'completed' ? 'fa-check-circle text-green-500' : 'fa-exclamation-circle text-red-500'- ]"></i>- <span class="truncate text-sm flex-1 text-[var(--text-primary)]">${item.file.name}</span>- <span :class="[- 'text-xs px-2 py-1 rounded-full font-semibold',- item.status === 'completed' ? 'bg-green-500 bg-opacity-20 text-green-600 dark:text-green-400' : 'bg-red-500 bg-opacity-20 text-red-600 dark:text-red-400'- ]">- ${item.status === 'completed' ? t('tagsModal.done') : t('status.failed')}- </span>+ Completed (${completedProgressItems.length})+ </div>+ <div v-for="item in completedProgressItems"+ :key="item.id"+ @click="item.recordingId && selectRecording({id: item.recordingId})"+ class="px-2 py-1 bg-[var(--bg-tertiary)]/30 rounded-md flex items-center gap-2 hover:bg-[var(--bg-tertiary)]/50 cursor-pointer mb-1">+ <i class="fas fa-check-circle text-xs text-green-500"></i>+ <span class="text-xs truncate flex-1 text-[var(--text-primary)]">${item.title}</span>+ <span class="text-[10px] px-1.5 py-0.5 rounded bg-green-500/20 text-green-500 font-medium">Done</span></div></div><!-- Empty State -->- <div v-if="!currentlyProcessingFile && backendProcessingRecordings.length === 0 && waitingFilesInQueue.length === 0 && finishedFilesInQueue.length === 0"- class="text-center py-4 text-[var(--text-muted)] text-sm">- <i class="fas fa-inbox text-2xl mb-2 opacity-50"></i>+ <div v-if="unifiedProgressItems.length === 0"+ class="text-center py-3 text-[var(--text-muted)] text-xs">+ <i class="fas fa-inbox text-lg mb-1 opacity-50"></i><p>No items in queue</p></div></div>