Skip to content

Speakr is a personal, self-hosted web application designed for transcribing audio recordings

Python

Add dual local/S3 storage layer (ported from #268)

Dmitry Abramov0bc720791bfe
diff --git a/config/env.transcription.example b/config/env.transcription.example
index e836b57a..cd3749cc 100644
--- a/config/env.transcription.example
+++ b/config/env.transcription.example
@@ -409,7 +409,33 @@ SQLALCHEMY_DATABASE_URI=sqlite:////data/instance/transcriptions.db
UPLOAD_FOLDER=/data/uploads
# =============================================================================
-# ADDITIONAL PROVIDER NOTES
+# FILE STORAGE (Local / S3-Compatible: AWS S3, MinIO)
+# =============================================================================
+
+# Storage backend for recording audio files: local | s3
+FILE_STORAGE_BACKEND=local
+
+# Key prefix inside the selected backend
+FILE_STORAGE_KEY_PREFIX=recordings
+
+# Optional local staging directory for uploads/conversion before final store
+# FILE_STORAGE_STAGING_DIR=/data/uploads/_staging
+
+# S3-compatible settings (used when FILE_STORAGE_BACKEND=s3)
+# S3_BUCKET_NAME=speakr-audio
+# S3_REGION=us-east-1
+# S3_ENDPOINT_URL=http://minio:9000
+# S3_ACCESS_KEY_ID=minioadmin
+# S3_SECRET_ACCESS_KEY=minioadmin
+# S3_SESSION_TOKEN=
+# S3_USE_PATH_STYLE=true
+# S3_VERIFY_SSL=false
+# S3_PRESIGN_TTL_SECONDS=900
+# S3_PRESIGN_PUBLIC_TTL_SECONDS=300
+# Audio delivery for S3 uses redirect to presigned URLs by default.
+
+# =============================================================================
+# FUTURE: Additional Provider Notes
# =============================================================================
# The connector architecture is designed to support additional providers.
# Currently available: openai_whisper, openai_transcribe, asr_endpoint,
diff --git a/docs/admin-guide/migration-guide.md b/docs/admin-guide/migration-guide.md
index 50af4c6a..fc09ffea 100644
--- a/docs/admin-guide/migration-guide.md
+++ b/docs/admin-guide/migration-guide.md
@@ -1,4 +1,4 @@
-# Migration Guide: Connector Architecture
+# Migration Guide
This guide helps you migrate from the legacy transcription configuration to the new connector-based architecture introduced in Speakr v0.8.
@@ -243,3 +243,184 @@ If you encounter issues during migration:
1. Check the [troubleshooting guide](../troubleshooting.md)
2. Review the [installation guide](../getting-started/installation.md) for complete configuration examples
3. Open an issue on [GitHub](https://github.com/murtaza-nasir/speakr/issues)
+
+---
+
+# Migrating Audio Files to S3
+
+Speakr supports storing recording audio files in S3-compatible object storage (AWS S3, MinIO, etc.) alongside or instead of the local filesystem. This section covers how to transition an existing instance from local storage to S3.
+
+## Prerequisites
+
+- Speakr updated to a version that includes the storage abstraction layer
+- S3 bucket created and accessible from your Speakr server
+- S3 credentials configured in `.env` (see [installation guide](../getting-started/installation.md#file-storage-backend-local-s3-compatible))
+- `boto3>=1.34.0` installed (included in the default Docker image)
+- **S3 bucket CORS configured** for browser playback/download via presigned URLs (required when your app domain differs from the S3/MinIO endpoint)
+
+## Migration Phases
+
+The migration is designed to be **gradual and zero-downtime**. Each phase is independent and can be performed separately.
+
+### Phase 1: Deploy Storage Abstraction (No Behavior Change)
+
+Update Speakr to the version with the storage layer while keeping `FILE_STORAGE_BACKEND=local` (the default). The application continues to work exactly as before — all reads, writes, and deletions now go through the unified storage service but still operate on local files.
+
+```bash
+FILE_STORAGE_BACKEND=local
+```
+
+### Phase 2: Normalize Legacy Paths
+
+Existing recordings may have inconsistent `audio_path` values (absolute paths, relative paths). The normalization script converts them all to the `local://` locator format without moving any files:
+
+```bash
+# Preview changes without writing
+docker compose exec app python scripts/migrate_local_paths_to_local_locator.py --dry-run
+
+# Run the actual normalization
+docker compose exec app python scripts/migrate_local_paths_to_local_locator.py
+```
+
+**Available options:**
+
+| Flag | Description |
+|------|-------------|
+| `--dry-run` | Preview changes without writing to DB |
+| `--limit N` | Process only the first N records |
+| `--recording-id ID` | Process a single recording |
+| `--only-user ID` | Process recordings for a specific user |
+| `--allow-missing-file` | Normalize even if the local file is missing |
+| `--report-jsonl <path>` | Write a JSONL report of all actions |
+
+The script is **idempotent** — running it multiple times is safe.
+
+### Phase 3: Switch New Uploads to S3
+
+Configure S3 credentials and switch the storage backend:
+
+```bash
+FILE_STORAGE_BACKEND=s3
+S3_BUCKET_NAME=speakr-audio
+S3_ENDPOINT_URL=http://minio:9000 # For MinIO
+S3_ACCESS_KEY_ID=minioadmin
+S3_SECRET_ACCESS_KEY=minioadmin
+S3_USE_PATH_STYLE=true # Required for MinIO
+```
+
+After restarting, **new uploads** go to S3, while existing `local://` recordings continue to be served from the local filesystem. The application supports both backends simultaneously.
+
+> **Important (CORS):** Because Speakr serves S3 audio using browser redirects to presigned URLs, your S3/MinIO bucket must allow cross-origin requests from your Speakr web origin.
+>
+> At minimum, allow:
+> - methods: `GET`, `HEAD`
+> - headers: `Range` (recommended for audio seeking/streaming)
+> - your Speakr origin (for example `https://speakr.example.com`)
+>
+> Example AWS S3 CORS (adjust origin):
+>
+> ```json
+> [
+> {
+> "AllowedHeaders": ["*"],
+> "AllowedMethods": ["GET", "HEAD"],
+> "AllowedOrigins": ["https://speakr.example.com"],
+> "ExposeHeaders": ["Accept-Ranges", "Content-Length", "Content-Range", "Content-Type", "ETag"],
+> "MaxAgeSeconds": 3000
+> }
+> ]
+> ```
+>
+> For MinIO, configure equivalent CORS rules for the bucket (via Console, `mc`, or your provisioning tool).
+
+### Phase 4: Migrate Historical Files to S3
+
+Move existing local files to S3 using the migration script:
+
+```bash
+# Preview what would be migrated
+docker compose exec app python scripts/migrate_local_recordings_to_s3.py --dry-run
+
+# Run the migration (verifies upload size by default)
+docker compose exec app python scripts/migrate_local_recordings_to_s3.py --limit 100
+```
+
+**Available options:**
+
+| Flag | Description |
+|------|-------------|
+| `--dry-run` | Preview without uploading or modifying DB |
+| `--limit N` | Migrate only N records per run |
+| `--recording-id ID` | Migrate a specific recording |
+| `--only-user ID` | Migrate recordings for a specific user |
+| `--verify-size` | Verify uploaded size matches local (enabled by default) |
+
+The script is **idempotent**:
+
+- Recordings already on `s3://` are skipped
+- If the S3 object already exists with matching size, only the DB is updated
+- Recordings in `PROCESSING` or `QUEUED` status are skipped to avoid race conditions
+
+Run in batches and monitor progress. Repeat until all local files are migrated.
+
+### Optional: Backfill Cached Audio Durations (`audio_duration_seconds`)
+
+If your instance already had recordings before the `audio_duration_seconds` cache field was introduced, older rows may still have `NULL` values. You can backfill them with the dedicated script:
+
+```bash
+# Preview only (no DB writes)
+docker compose exec app python scripts/backfill_audio_duration_seconds.py --dry-run
+
+# Run backfill for all eligible recordings
+docker compose exec app python scripts/backfill_audio_duration_seconds.py
+```
+
+The script:
+
+- updates **only** recordings where `audio_duration_seconds IS NULL`
+- skips recordings with `audio_deleted_at` set
+- skips active jobs (`PROCESSING`, `QUEUED`)
+- calculates duration using the storage abstraction + `ffprobe`, so it works for both `local://` and `s3://` recordings
+
+**Useful options:**
+
+| Flag | Description |
+|------|-------------|
+| `--dry-run` | Preview without updating DB |
+| `--limit N` | Process only the first N recordings |
+| `--recording-id ID` | Backfill one recording |
+| `--only-user ID` | Backfill recordings for one user |
+| `--report-jsonl <path>` | Write a JSONL report |
+| `--ffprobe-timeout SECONDS` | Override ffprobe timeout (default: 30) |
+
+Example with report:
+
+```bash
+docker compose exec app python scripts/backfill_audio_duration_seconds.py --report-jsonl /tmp/audio-duration-backfill.jsonl
+```
+
+### Phase 5: Cleanup Local Files
+
+After confirming all recordings are served from S3, you can reclaim local disk space. The migration script does not delete local source files after successful upload by default. You can set --delete-local-after-success parameter to the migration script to auto-delete migrated local files.
+
+## Verifying the Migration
+
+After migration, verify that:
+
+1. **Audio playback** works for migrated recordings (they should redirect to presigned S3 URLs)
+2. **New uploads** are stored in S3 (check `recording.audio_path` starts with `s3://`)
+3. **Reprocessing** works — the worker materializes audio from S3 to a temporary local file for transcription
+4. **Deletion** and **retention** properly remove S3 objects
+5. **Shared links** generate working presigned URLs with appropriate TTL
+6. **Cached durations** are backfilled for older rows (check `recording.audio_duration_seconds` is no longer `NULL` for historical recordings)
+
+## Rollback
+
+If you need to revert to local storage:
+
+1. Set `FILE_STORAGE_BACKEND=local` in `.env`
+2. Restart the container
+3. New uploads will go to local storage again
+4. Existing `s3://` recordings continue to work as long as S3 credentials remain configured
+
+The system is designed so that both backends can coexist indefinitely.
diff --git a/docs/admin-guide/retention.md b/docs/admin-guide/retention.md
index ddf7116c..b658a602 100644
--- a/docs/admin-guide/retention.md
+++ b/docs/admin-guide/retention.md
@@ -355,12 +355,15 @@ Example group tag retention configuration:
- Skip if exempt
↓
4. Delete based on mode:
- - audio_only: Remove file, keep DB record, set audio_deleted_at
- - full_recording: Remove file and DB record
+ - audio_only: Remove file via storage service, keep DB record, set audio_deleted_at
+ - full_recording: Remove file via storage service and DB record
↓
5. Return statistics
```
+!!! info "S3 Storage Support"
+ Audio deletion works transparently with both local and S3 storage backends. The storage service resolves each recording's storage locator (`local://...` or `s3://...`) and deletes the file from the correct backend. If the file is already missing, the deletion is treated as successful and processing continues.
+
## Migration Guide
### Enabling Auto-Deletion on Existing System
diff --git a/docs/advanced/job-queue-system.md b/docs/advanced/job-queue-system.md
index 440f9ac9..4adc9133 100644
--- a/docs/advanced/job-queue-system.md
+++ b/docs/advanced/job-queue-system.md
@@ -150,9 +150,10 @@ When a new upload fails permanently:
```python
if is_new_upload and recording:
- # Delete audio file
- if recording.audio_path and os.path.exists(recording.audio_path):
- os.remove(recording.audio_path)
+ # Delete audio file via storage service (works for both local and S3)
+ if recording.audio_path:
+ from src.services.storage import get_storage_service
+ get_storage_service().delete(recording.audio_path, missing_ok=True)
# Delete all processing jobs for this recording
ProcessingJob.query.filter_by(recording_id=recording_id).delete()
# Delete the recording
diff --git a/docs/faq.md b/docs/faq.md
index f917cf54..f31fc8d7 100644
--- a/docs/faq.md
+++ b/docs/faq.md
@@ -69,7 +69,7 @@ Run Speakr once with internet access to cache the embedding model, then disconne
### How do I backup my Speakr data?
-Your Speakr data consists of three essential components: the SQLite database in the `instance/` directory, audio files and transcriptions in the `uploads/` directory, and your configuration in the `.env` file. To create a complete backup, stop the container first to ensure database consistency, then backup all three directories:
+Your Speakr data consists of three essential components: the SQLite database in the `instance/` directory, audio files in the `uploads/` directory (when using local storage), and your configuration in the `.env` file. To create a complete backup, stop the container first to ensure database consistency, then backup all three directories:
```bash
docker compose down
@@ -77,6 +77,8 @@ tar czf speakr_backup_$(date +%Y%m%d).tar.gz uploads/ instance/ .env
docker compose up -d
```
+If you use S3 storage for audio files (`FILE_STORAGE_BACKEND=s3`), audio data is stored externally and managed by your S3 provider's durability guarantees. In that case you only need to back up `instance/` and `.env`.
+
Regular automated backups are highly recommended for production use.
## Transcription and AI Features
@@ -271,6 +273,14 @@ AUDIO_UNSUPPORTED_CODECS=opus,vorbis
Files using these codecs will be automatically converted to your target format before transcription. See the [troubleshooting guide](troubleshooting.md#format-not-recognised-errors) for more details.
+### Can I store audio files in S3 or MinIO instead of local disk?
+
+Yes. Set `FILE_STORAGE_BACKEND=s3` in your `.env` and configure the S3 connection variables (`S3_BUCKET_NAME`, `S3_ENDPOINT_URL`, etc.). Speakr will store new uploads in S3 and serve audio via short-lived presigned URLs instead of streaming through the backend. Existing local recordings continue to work — the system reads from both backends based on each recording's storage locator. See the [installation guide](getting-started/installation.md#file-storage-backend-local-s3-compatible) for full configuration and the [migration guide](admin-guide/migration-guide.md#migrating-audio-files-to-s3) for moving existing files to S3.
+
+### What happens to existing recordings when I switch to S3 storage?
+
+Nothing breaks. Existing local recordings keep working as before — Speakr reads from both local and S3 storage simultaneously. Only **new** uploads go to S3 after you change `FILE_STORAGE_BACKEND=s3`. To migrate historical files, use the provided migration scripts: first normalize legacy paths with `scripts/migrate_local_paths_to_local_locator.py`, then move files to S3 with `scripts/migrate_local_recordings_to_s3.py`. Both scripts support `--dry-run` and are idempotent.
+
---
Return to [Home](index.md) →
\ No newline at end of file
diff --git a/docs/features.md b/docs/features.md
index 6ea58efb..4589debc 100644
--- a/docs/features.md
+++ b/docs/features.md
@@ -210,6 +210,10 @@ The events that can be subscribed to are `recording.created`, `recording.transcr
Handle large audio files that exceed API limits through intelligent chunking. See the [troubleshooting guide](troubleshooting.md#files-over-25mb-fail-with-openai) for configuration details. Learn about [chunking strategies](faq.md#whats-the-difference-between-chunking-by-size-vs-duration) in the FAQ. The system automatically splits long recordings into manageable segments, processes them separately, then seamlessly reassembles the results. Configure chunk size by duration or file size to match your API provider's requirements.
+### S3-Compatible Object Storage
+
+Store recording audio files in S3-compatible object storage (AWS S3, MinIO, or any S3 API provider) instead of, or in addition to, the local filesystem. Switch storage backends with a single environment variable (`FILE_STORAGE_BACKEND=s3`). Audio playback and downloads use short-lived presigned URLs, keeping large file traffic off your application server. Existing local recordings continue to work alongside S3 — Speakr reads from both backends transparently based on each recording's storage locator. Migration scripts are provided to normalize legacy paths and move historical files to S3. See the [installation guide](getting-started/installation.md#file-storage-backend-local-s3-compatible) for configuration details and the [migration guide](admin-guide/migration-guide.md#migrating-audio-files-to-s3) for transitioning existing data.
+
### Black Hole Processing
Set up a watched directory where dropped audio files are automatically processed. Configure this in [system settings](admin-guide/system-settings.md) for automated workflows. Perfect for automation workflows or batch processing, the black hole directory monitors for new files and queues them for transcription without manual intervention.
diff --git a/docs/getting-started/installation.md b/docs/getting-started/installation.md
index 7f3384ca..8299654e 100644
--- a/docs/getting-started/installation.md
+++ b/docs/getting-started/installation.md
@@ -384,6 +384,50 @@ When enabled, Speakr checks the watch directory every 30 seconds for new audio f
To use this feature, you'll need to mount an additional volume in your Docker Compose configuration, which we'll cover in the next steps.
+#### File Storage Backend (Local / S3-Compatible)
+
+Speakr supports two storage backends for recording audio files: **local filesystem** (default) and **S3-compatible object storage** (AWS S3, MinIO, and other S3 API providers). You can switch between them using a single environment variable:
+
+```bash
+# Storage backend: local (default) or s3
+FILE_STORAGE_BACKEND=local
+
+# Key prefix inside the selected backend (default: recordings)
+FILE_STORAGE_KEY_PREFIX=recordings
+
+# Optional staging directory for uploads/conversion before final store
+# FILE_STORAGE_STAGING_DIR=/data/uploads/_staging
+```
+
+When set to `local`, audio files are stored under `UPLOAD_FOLDER` as before — no additional configuration is needed.
+
+To use **S3 or MinIO**, set `FILE_STORAGE_BACKEND=s3` and configure the S3 connection:
+
+```bash
+FILE_STORAGE_BACKEND=s3
+
+S3_BUCKET_NAME=speakr-audio
+S3_REGION=us-east-1
+S3_ENDPOINT_URL=http://minio:9000 # For MinIO / custom S3-compatible endpoints
+S3_ACCESS_KEY_ID=minioadmin
+S3_SECRET_ACCESS_KEY=minioadmin
+S3_USE_PATH_STYLE=true # Required for MinIO
+S3_VERIFY_SSL=false # Set false for local MinIO without TLS
+
+# Presigned URL TTL (seconds) for audio playback/download
+S3_PRESIGN_TTL_SECONDS=900 # Private access (default 15 min)
+S3_PRESIGN_PUBLIC_TTL_SECONDS=300 # Shared links (default 5 min)
+```
+
+!!! info "How S3 Audio Delivery Works"
+ When using S3 storage, audio endpoints return a `302` redirect to a short-lived presigned URL instead of streaming the file through the backend. This keeps bandwidth off your application server while preserving compatibility with `<audio>` tags and direct downloads. The presigned URLs expire according to the TTL settings above.
+
+!!! tip "Mixed Storage Support"
+ Speakr can read from **both** local and S3 storage simultaneously. Existing local recordings continue to work even after switching `FILE_STORAGE_BACKEND=s3` — only new uploads go to S3. This allows a gradual transition. See the [Migration Guide](../admin-guide/migration-guide.md#migrating-audio-files-to-s3) for details on migrating historical files.
+
+!!! warning "boto3 Required"
+ S3 support requires the `boto3` Python package, which is included in the default Docker image. If you build from source, ensure `boto3>=1.34.0` is installed.
+
### Step 6: Set Up Data Directories
Speakr needs local directories to store your data persistently. These directories are mounted as Docker volumes, ensuring your recordings and database survive container updates and restarts:
@@ -846,7 +890,7 @@ Apache requires these modules: `sudo a2enmod ssl proxy proxy_http proxy_wstunnel
### Backup Strategy
-Regular backups are essential for production deployments. Your Speakr data consists of three critical components that need backing up: the SQLite database in the `instance` directory, the audio files and transcriptions in the `uploads` directory, and your configuration in the `.env` file.
+Regular backups are essential for production deployments. Your Speakr data consists of three critical components that need backing up: the SQLite database in the `instance` directory, the audio files in the `uploads` directory (when using local storage), and your configuration in the `.env` file. If you use S3 storage, audio files are managed by your S3 provider's durability guarantees and do not need to be included in local backups.
Create a backup script that captures all three components:
diff --git a/requirements.txt b/requirements.txt
index 6d7f8c4d..81d9c0f8 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -28,3 +28,4 @@ numpy==1.24.3
scikit-learn==1.3.0
scipy<1.15
psycopg2-binary>=2.9.0
+boto3>=1.34.0
diff --git a/scripts/backfill_audio_duration_seconds.py b/scripts/backfill_audio_duration_seconds.py
new file mode 100644
index 00000000..e0398515
--- /dev/null
+++ b/scripts/backfill_audio_duration_seconds.py
@@ -0,0 +1,137 @@
+#!/usr/bin/env python3
+"""Backfill `recording.audio_duration_seconds` for existing recordings.
+
+Updates only rows where `audio_duration_seconds IS NULL`, skips deleted audio and
+active processing states, and reads audio via the storage facade (local/S3).
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from datetime import datetime
+
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
+
+from src.app import app # noqa: E402
+from src.database import db # noqa: E402
+from src.models import Recording # noqa: E402
+from src.services.storage import get_storage_service # noqa: E402
+from src.utils.ffprobe import get_duration # noqa: E402
+
+SKIP_STATUSES = {'PROCESSING', 'QUEUED'}
+
+
+def parse_args():
+ parser = argparse.ArgumentParser(description='Backfill recording.audio_duration_seconds using storage layer + ffprobe')
+ parser.add_argument('--limit', type=int, default=None)
+ parser.add_argument('--recording-id', type=int, default=None)
+ parser.add_argument('--only-user', type=int, default=None)
+ parser.add_argument('--dry-run', action='store_true')
+ parser.add_argument('--report-jsonl', type=str, default=None)
+ parser.add_argument('--ffprobe-timeout', type=int, default=30)
+ return parser.parse_args()
+
+
+def _report(fp, *, recording_id, action, duration=None, audio_path=None, error=None):
+ if not fp:
+ return
+ row = {
+ 'ts': datetime.utcnow().isoformat(),
+ 'recording_id': recording_id,
+ 'action': action,
+ 'duration': duration,
+ 'audio_path': audio_path,
+ 'error': error,
+ }
+ fp.write(json.dumps(row, ensure_ascii=False) + '\n')
+ fp.flush()
+
+
+def main() -> int:
+ args = parse_args()
+ storage = get_storage_service()
+ report_fp = open(args.report_jsonl, 'a', encoding='utf-8') if args.report_jsonl else None
+
+ stats = {
+ 'scanned': 0,
+ 'updated': 0,
+ 'skipped_status': 0,
+ 'skipped_audio_deleted': 0,
+ 'skipped_no_audio_path': 0,
+ 'skipped_already_set': 0,
+ 'skipped_missing_object': 0,
+ 'skipped_no_duration': 0,
+ 'errors': 0,
+ }
+
+ try:
+ with app.app_context():
+ query = Recording.query
+ if args.recording_id:
+ query = query.filter(Recording.id == args.recording_id)
+ if args.only_user:
+ query = query.filter(Recording.user_id == args.only_user)
+
+ query = query.order_by(Recording.id.asc())
+ if args.limit:
+ query = query.limit(args.limit)
+
+ for recording in query.all():
+ stats['scanned'] += 1
+
+ if recording.audio_duration_seconds is not None:
+ stats['skipped_already_set'] += 1
+ continue
+
+ if recording.status in SKIP_STATUSES:
+ stats['skipped_status'] += 1
+ _report(report_fp, recording_id=recording.id, action='skip_status', audio_path=recording.audio_path)
+ continue
+
+ if recording.audio_deleted_at is not None:
+ stats['skipped_audio_deleted'] += 1
+ _report(report_fp, recording_id=recording.id, action='skip_audio_deleted', audio_path=recording.audio_path)
+ continue
+
+ if not recording.audio_path:
+ stats['skipped_no_audio_path'] += 1
+ _report(report_fp, recording_id=recording.id, action='skip_no_audio_path')
+ continue
+
+ try:
+ if not storage.exists(recording.audio_path):
+ stats['skipped_missing_object'] += 1
+ _report(report_fp, recording_id=recording.id, action='skip_missing_object', audio_path=recording.audio_path)
+ continue
+
+ with storage.materialize(recording.audio_path) as materialized:
+ duration = get_duration(materialized.local_path, timeout=args.ffprobe_timeout)
+
+ if duration is None:
+ stats['skipped_no_duration'] += 1
+ _report(report_fp, recording_id=recording.id, action='skip_no_duration', audio_path=recording.audio_path)
+ continue
+
+ _report(report_fp, recording_id=recording.id, action='update', duration=float(duration), audio_path=recording.audio_path)
+ if not args.dry_run:
+ recording.audio_duration_seconds = float(duration)
+ db.session.add(recording)
+ db.session.commit()
+ stats['updated'] += 1
+ except Exception as exc:
+ db.session.rollback()
+ stats['errors'] += 1
+ _report(report_fp, recording_id=recording.id, action='error', audio_path=recording.audio_path, error=str(exc))
+ finally:
+ if report_fp:
+ report_fp.close()
+
+ print(json.dumps({'timestamp': datetime.utcnow().isoformat(), **stats}, ensure_ascii=False, indent=2))
+ return 0 if stats['errors'] == 0 else 1
+
+
+if __name__ == '__main__':
+ raise SystemExit(main())
diff --git a/scripts/migrate_local_paths_to_local_locator.py b/scripts/migrate_local_paths_to_local_locator.py
new file mode 100644
index 00000000..363d0952
--- /dev/null
+++ b/scripts/migrate_local_paths_to_local_locator.py
@@ -0,0 +1,117 @@
+#!/usr/bin/env python3
+"""Normalize legacy absolute local recording paths into local:// locators."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from datetime import datetime
+from pathlib import Path
+
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
+
+from src.app import app # noqa: E402
+from src.database import db # noqa: E402
+from src.models import Recording # noqa: E402
+from src.services.storage import get_storage_service # noqa: E402
+
+
+def _is_absolute_path(value: str) -> bool:
+ if not value:
+ return False
+ if len(value) >= 3 and value[1] == ':' and value[2] in ('\\', '/'):
+ return True
+ return os.path.isabs(value)
+
+
+def parse_args():
+ p = argparse.ArgumentParser(description='Normalize legacy absolute recording paths to local:// locators')
+ p.add_argument('--dry-run', action='store_true', help='Preview changes without writing to DB')
+ p.add_argument('--limit', type=int, default=None)
+ p.add_argument('--recording-id', type=int, default=None)
+ p.add_argument('--only-user', type=int, default=None)
+ p.add_argument('--allow-missing-file', action='store_true')
+ p.add_argument('--report-jsonl', type=str, default=None)
+ return p.parse_args()
+
+
+def main():
+ args = parse_args()
+ storage = get_storage_service()
+
+ stats = {
+ 'scanned': 0,
+ 'normalized': 0,
+ 'skipped_non_legacy': 0,
+ 'skipped_missing_file': 0,
+ 'errors': 0,
+ }
+
+ report_fp = open(args.report_jsonl, 'a', encoding='utf-8') if args.report_jsonl else None
+
+ with app.app_context():
+ query = Recording.query
+ if args.recording_id:
+ query = query.filter(Recording.id == args.recording_id)
+ if args.only_user:
+ query = query.filter(Recording.user_id == args.only_user)
+ query = query.order_by(Recording.id.asc())
+ if args.limit:
+ query = query.limit(args.limit)
+
+ for recording in query.all():
+ stats['scanned'] += 1
+ old_value = recording.audio_path or ''
+
+ if not old_value or not _is_absolute_path(old_value):
+ stats['skipped_non_legacy'] += 1
+ continue
+
+ try:
+ if (not args.allow_missing_file) and (not os.path.exists(old_value)):
+ stats['skipped_missing_file'] += 1
+ _write_report(report_fp, recording.id, 'skip_missing_file', old_value, None)
+ continue
+
+ new_locator = storage.build_local_locator_from_path(old_value)
+ if old_value == new_locator:
+ stats['skipped_non_legacy'] += 1
+ continue
+
+ _write_report(report_fp, recording.id, 'normalize', old_value, new_locator)
+ if not args.dry_run:
+ recording.audio_path = new_locator
+ db.session.add(recording)
+ db.session.commit()
+ stats['normalized'] += 1
+ except Exception as exc:
+ db.session.rollback()
+ stats['errors'] += 1
+ _write_report(report_fp, recording.id, 'error', old_value, None, str(exc))
+
+ if report_fp:
+ report_fp.close()
+
+ print(json.dumps({'timestamp': datetime.utcnow().isoformat(), **stats}, ensure_ascii=False, indent=2))
+ return 0 if stats['errors'] == 0 else 1
+
+
+def _write_report(fp, recording_id, action, old_value, new_value=None, error=None):
+ if not fp:
+ return
+ row = {
+ 'ts': datetime.utcnow().isoformat(),
+ 'recording_id': recording_id,
+ 'action': action,
+ 'old_audio_path': old_value,
+ 'new_audio_path': new_value,
+ 'error': error,
+ }
+ fp.write(json.dumps(row, ensure_ascii=False) + '\n')
+ fp.flush()
+
+
+if __name__ == '__main__':
+ raise SystemExit(main())
diff --git a/scripts/migrate_local_recordings_to_s3.py b/scripts/migrate_local_recordings_to_s3.py
new file mode 100644
index 00000000..1959ac81
--- /dev/null
+++ b/scripts/migrate_local_recordings_to_s3.py
@@ -0,0 +1,142 @@
+#!/usr/bin/env python3
+"""Migrate recordings from local:// locators to s3:// locators."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from datetime import datetime
+
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
+
+from src.app import app # noqa: E402
+from src.database import db # noqa: E402
+from src.models import Recording # noqa: E402
+from src.services.storage import get_storage_service # noqa: E402
+
+
+def parse_args():
+ p = argparse.ArgumentParser(description='Migrate recording audio from local:// to s3://')
+ p.add_argument('--dry-run', action='store_true')
+ p.add_argument('--limit', type=int, default=None)
+ p.add_argument('--recording-id', type=int, default=None)
+ p.add_argument('--only-user', type=int, default=None)
+ p.add_argument('--verify-size', action='store_true', default=True)
+ p.add_argument('--delete-local-after-success', action='store_true')
+ p.add_argument('--report-jsonl', type=str, default=None)
+ return p.parse_args()
+
+
+def main():
+ args = parse_args()
+ storage = get_storage_service()
+
+ if storage.default_backend_kind() != 's3':
+ print('ERROR: FILE_STORAGE_BACKEND must be s3 to run this migration script', file=sys.stderr)
+ return 2
+
+ stats = {
+ 'scanned': 0,
+ 'migrated': 0,
+ 'skipped_already_s3': 0,
+ 'skipped_not_local': 0,
+ 'skipped_missing_local': 0,
+ 'skipped_in_progress': 0,
+ 'errors': 0,
+ }
+ report_fp = open(args.report_jsonl, 'a', encoding='utf-8') if args.report_jsonl else None
+
+ try:
+ with app.app_context():
+ query = Recording.query.filter(Recording.audio_path.isnot(None))
+ if args.recording_id:
+ query = query.filter(Recording.id == args.recording_id)
+ if args.only_user:
+ query = query.filter(Recording.user_id == args.only_user)
+ query = query.order_by(Recording.id.asc())
+ if args.limit:
+ query = query.limit(args.limit)
+
+ for recording in query.all():
+ stats['scanned'] += 1
+ old_locator = (recording.audio_path or '').strip()
+
+ if old_locator.startswith('s3://'):
+ stats['skipped_already_s3'] += 1
+ _report(report_fp, recording.id, 'skip_already_s3', old_locator)
+ continue
+
+ if not old_locator.startswith('local://'):
+ stats['skipped_not_local'] += 1
+ _report(report_fp, recording.id, 'skip_not_local_locator', old_locator)
+ continue
+
+ if recording.status in ('PROCESSING', 'QUEUED'):
+ stats['skipped_in_progress'] += 1
+ _report(report_fp, recording.id, 'skip_in_progress', old_locator)
+ continue
+
+ try:
+ local_path = storage.resolve_local_filesystem_path(old_locator)
+ if not os.path.exists(local_path):
+ stats['skipped_missing_local'] += 1
+ _report(report_fp, recording.id, 'skip_missing_local_file', old_locator, error=local_path)
+ continue
+
+ key = storage.build_recording_key(recording.original_filename or os.path.basename(local_path), recording.id)
+
+ if args.dry_run:
+ preview_locator = storage.build_default_locator(key)
+ _report(report_fp, recording.id, 'dry_run', old_locator, preview_locator)
+ continue
+
+ old_size = os.path.getsize(local_path) if args.verify_size else None
+ stored = storage.upload_local_file(local_path, key, content_type=recording.mime_type, delete_source=False)
+
+ if args.verify_size and old_size is not None and stored.size is not None and int(stored.size) != int(old_size):
+ raise ValueError(f'size mismatch local={old_size} s3={stored.size}')
+
+ prev_locator = recording.audio_path
+ recording.audio_path = stored.locator
+ db.session.add(recording)
+ db.session.commit()
+
+ if args.delete_local_after_success:
+ try:
+ get_storage_service().delete(prev_locator, missing_ok=True)
+ except Exception as cleanup_exc:
+ _report(report_fp, recording.id, 'warning_delete_local_failed', prev_locator, stored.locator, str(cleanup_exc))
+
+ stats['migrated'] += 1
+ _report(report_fp, recording.id, 'migrated', prev_locator, stored.locator)
+ except Exception as exc:
+ db.session.rollback()
+ stats['errors'] += 1
+ _report(report_fp, recording.id, 'error', old_locator, error=str(exc))
+ finally:
+ if report_fp:
+ report_fp.close()
+
+ print(json.dumps({'timestamp': datetime.utcnow().isoformat(), **stats}, ensure_ascii=False, indent=2))
+ return 0 if stats['errors'] == 0 else 1
+
+
+def _report(fp, recording_id, action, old_locator, new_locator=None, error=None):
+ if not fp:
+ return
+ row = {
+ 'ts': datetime.utcnow().isoformat(),
+ 'recording_id': recording_id,
+ 'action': action,
+ 'old_audio_path': old_locator,
+ 'new_audio_path': new_locator,
+ 'error': error,
+ }
+ fp.write(json.dumps(row, ensure_ascii=False) + '\n')
+ fp.flush()
+
+
+if __name__ == '__main__':
+ raise SystemExit(main())
diff --git a/src/api/admin.py b/src/api/admin.py
index e573884f..6299caca 100644
--- a/src/api/admin.py
+++ b/src/api/admin.py
@@ -336,11 +336,12 @@ def admin_delete_user(user_id):
total_chunks = TranscriptChunk.query.filter_by(user_id=user_id).count()
if total_chunks > 0:
current_app.logger.info(f"Deleting {total_chunks} transcript chunks with embeddings for user {user_id}")
-
+ from src.services.storage import get_storage_service
+ storage = get_storage_service()
for recording in user.recordings:
try:
- if recording.audio_path and os.path.exists(recording.audio_path):
- os.remove(recording.audio_path)
+ if recording.audio_path:
+ storage.delete(recording.audio_path, missing_ok=True)
except Exception as e:
current_app.logger.error(f"Error deleting audio file {recording.audio_path}: {e}")
@@ -1530,4 +1531,3 @@ def admin_inquire_status():
# --- Group Management API (Admin Only) ---
-
diff --git a/src/api/api_v1.py b/src/api/api_v1.py
index 05cb1c49..a0d39224 100644
--- a/src/api/api_v1.py
+++ b/src/api/api_v1.py
@@ -19,7 +19,7 @@ import json
from datetime import datetime, date, timedelta
from typing import Optional
-from flask import Blueprint, jsonify, request, current_app, send_file
+from flask import Blueprint, jsonify, request, current_app, send_file, redirect
from flask_login import login_required, current_user
from sqlalchemy import func, extract, or_, and_
@@ -32,6 +32,7 @@ from src.services.token_tracking import TokenTracker
from src.services.transcription_tracking import transcription_tracker
from src.file_exporter import format_transcription_with_template
from src.api.recordings import upload_file as _upload_file_ui
+from src.services.storage import get_storage_service
# Create blueprint with /api/v1 prefix
api_v1_bp = Blueprint('api_v1', __name__, url_prefix='/api/v1')
@@ -2508,22 +2509,33 @@ def download_audio(recording_id):
if not recording.audio_path:
return jsonify({'error': 'No audio file'}), 404
- audio_path = os.path.join(current_app.config.get('UPLOAD_FOLDER', 'uploads'), recording.audio_path)
- if not os.path.exists(audio_path):
- return jsonify({'error': 'Audio file not found'}), 404
-
download = request.args.get('download', 'false').lower() == 'true'
# original_filename is user-controlled at upload time. Sanitize it
- # via secure_filename before passing to send_file so null bytes or
- # path-traversal segments can't reach a confused download dialog
- # in old clients.
+ # via secure_filename before using it as a download name so null bytes or
+ # path-traversal segments can't reach a confused download dialog (this also
+ # feeds the S3 presigned Content-Disposition below).
from werkzeug.utils import secure_filename
raw_name = recording.original_filename or f'recording-{recording_id}.mp3'
safe_download_name = secure_filename(raw_name) or f'recording-{recording_id}'
+ storage = get_storage_service()
+ delivery = storage.get_audio_delivery(
+ recording.audio_path,
+ download=download,
+ mime_type=recording.mime_type or 'audio/mpeg',
+ download_name=safe_download_name,
+ is_public=False,
+ )
+
+ if delivery.mode == 'redirect_url':
+ return redirect(delivery.url, code=302)
+
+ if not delivery.local_path or not os.path.exists(delivery.local_path):
+ return jsonify({'error': 'Audio file not found'}), 404
+
return send_file(
- audio_path,
+ delivery.local_path,
mimetype=recording.mime_type or 'audio/mpeg',
as_attachment=download,
download_name=safe_download_name,
diff --git a/src/api/recordings.py b/src/api/recordings.py
index 4ff5826b..93456e01 100644
--- a/src/api/recordings.py
+++ b/src/api/recordings.py
@@ -85,8 +85,9 @@ from src.services.document import process_markdown_to_docx
from src.services.llm import client, chat_client, call_llm_completion, call_chat_completion, process_streaming_with_thinking, TokenBudgetExceeded
from src.services.embeddings import process_recording_chunks
from src.file_exporter import export_recording, mark_export_as_deleted
-from src.utils.ffprobe import get_codec_info, get_creation_date, FFProbeError
+from src.utils.ffprobe import get_codec_info, get_creation_date, get_duration, FFProbeError
from src.utils.audio_conversion import convert_if_needed
+from src.services.storage import get_storage_service
from src.utils.file_hash import compute_file_sha256
# Create blueprint
@@ -1040,7 +1041,7 @@ def reprocess_transcription(recording_id):
if not has_recording_access(recording, current_user, require_edit=True):
return jsonify({'error': 'You do not have permission to reprocess this recording'}), 403
- if not recording.audio_path or not os.path.exists(recording.audio_path):
+ if not recording.audio_path or not get_storage_service().exists(recording.audio_path):
return jsonify({'error': 'Audio file not found for reprocessing'}), 404
if recording.status in ['QUEUED', 'PROCESSING', 'SUMMARIZING']:
@@ -2245,7 +2246,9 @@ def upload_file():
original_filename = file.filename
safe_filename = secure_filename(original_filename)
- filepath = os.path.join(current_app.config['UPLOAD_FOLDER'], f"{datetime.now().strftime('%Y%m%d%H%M%S')}_{safe_filename}")
+ storage = get_storage_service()
+ staging_dir = storage.get_staging_dir()
+ filepath = os.path.join(staging_dir, f"{datetime.now().strftime('%Y%m%d%H%M%S')}_{safe_filename}")
# Per-upload override for VIDEO_RETENTION. When True, the
# processing pipeline extracts audio and discards the video stream
@@ -2491,6 +2494,16 @@ def upload_file():
current_app.logger.info(f"Final MIME type: {mime_type} for file {filepath}")
+ # Cache media duration while the upload is still available locally.
+ # This avoids later S3 materialization just to serialize API responses.
+ audio_duration_seconds = None
+ try:
+ detected_duration = get_duration(filepath, timeout=30)
+ if detected_duration is not None and detected_duration > 0:
+ audio_duration_seconds = float(detected_duration)
+ except Exception as e:
+ current_app.logger.warning(f"Could not determine duration for upload {original_filename}: {e}")
+
# Get notes from the form
notes = request.form.get('notes')
@@ -2678,7 +2691,7 @@ def upload_file():
cached_duration = None
recording = Recording(
- audio_path=filepath,
+ audio_path=None,
original_filename=original_filename,
title=resolve_upload_title(user_title, original_filename),
file_size=final_file_size,
@@ -2686,6 +2699,7 @@ def upload_file():
meeting_date=meeting_date,
user_id=current_user.id,
mime_type=mime_type,
+ audio_duration_seconds=audio_duration_seconds,
notes=notes,
folder_id=selected_folder.id if selected_folder else None,
processing_source='upload', # Track that this was manually uploaded
@@ -2699,7 +2713,19 @@ def upload_file():
audio_duration_seconds=cached_duration,
)
db.session.add(recording)
- db.session.commit()
+ db.session.flush() # Assign recording.id without committing transaction yet
+
+ storage_key = storage.build_recording_key(original_filename, recording.id, now=now)
+ stored_object = storage.upload_local_file(
+ filepath,
+ storage_key,
+ content_type=mime_type,
+ delete_source=True,
+ )
+ # Intentionally do not reuse `filepath` after upload: with delete_source=True
+ # the staging file may already be gone (notably on S3 backend). Workers should
+ # always read from recording.audio_path via the storage facade.
+ recording.audio_path = stored_object.locator
# Add tags to recording if selected (preserve order)
for order, tag in enumerate(selected_tags, 1):
@@ -2711,8 +2737,9 @@ def upload_file():
)
db.session.add(new_association)
+ db.session.commit()
+
if selected_tags:
- db.session.commit()
tag_names = [tag.name for tag in selected_tags]
current_app.logger.info(f"Added {len(selected_tags)} tags to recording {recording.id}: {', '.join(tag_names)}")
@@ -3144,9 +3171,10 @@ def delete_recording(recording_id):
# Delete the audio file first
try:
- if recording.audio_path and os.path.exists(recording.audio_path):
- os.remove(recording.audio_path)
- current_app.logger.info(f"Deleted audio file: {recording.audio_path}")
+ if recording.audio_path:
+ storage = get_storage_service()
+ storage.delete(recording.audio_path, missing_ok=True)
+ current_app.logger.info(f"Deleted audio file via storage backend: {recording.audio_path}")
except Exception as e:
current_app.logger.error(f"Error deleting audio file {recording.audio_path}: {e}")
@@ -3533,9 +3561,9 @@ def delete_job(job_id):
recording = db.session.get(Recording, job.recording_id)
if recording:
# Delete audio file
- if recording.audio_path and os.path.exists(recording.audio_path):
+ if recording.audio_path:
try:
- os.remove(recording.audio_path)
+ get_storage_service().delete(recording.audio_path, missing_ok=True)
except Exception as e:
current_app.logger.error(f"Error deleting audio file: {e}")
# Delete ALL processing jobs for this recording first
@@ -3626,22 +3654,44 @@ def get_audio(recording_id):
if not has_access:
return jsonify({'error': 'You do not have permission to access this audio file'}), 403
- if not os.path.exists(recording.audio_path):
- current_app.logger.error(f"Audio file missing from server: {recording.audio_path}")
- return jsonify({'error': 'Audio file missing from server'}), 404
+ storage = get_storage_service()
# Check if download is requested
download = request.args.get('download', 'false').lower() == 'true'
+ download_name = None
if download:
# Generate filename from recording title or use default
filename = recording.title or f'recording_{recording_id}'
# Sanitize filename and add extension
filename = "".join(c for c in filename if c.isalnum() or c in (' ', '-', '_')).strip()
- ext = os.path.splitext(recording.audio_path)[1] or '.mp3'
- filename = f"{filename}{ext}"
- return send_file(recording.audio_path, as_attachment=True, download_name=filename, mimetype=recording.mime_type, conditional=True)
+ ext = os.path.splitext(recording.original_filename or '')[1] or '.mp3'
+ download_name = f"{filename}{ext}"
+
+ delivery = storage.get_audio_delivery(
+ recording.audio_path,
+ download=download,
+ mime_type=recording.mime_type,
+ download_name=download_name,
+ is_public=False,
+ )
+
+ if delivery.mode == 'redirect_url':
+ return redirect(delivery.url, code=302)
+
+ if not delivery.local_path or not os.path.exists(delivery.local_path):
+ current_app.logger.error(f"Audio file missing from server: {recording.audio_path}")
+ return jsonify({'error': 'Audio file missing from server'}), 404
+
+ if download:
+ return send_file(
+ delivery.local_path,
+ as_attachment=True,
+ download_name=download_name,
+ mimetype=recording.mime_type,
+ conditional=True,
+ )
- return send_file(recording.audio_path, mimetype=recording.mime_type, conditional=True)
+ return send_file(delivery.local_path, mimetype=recording.mime_type, conditional=True)
except Exception as e:
current_app.logger.error(f"Error serving audio for recording {recording_id}: {e}", exc_info=True)
return jsonify({'error': 'An unexpected error occurred.'}), 500
@@ -4104,9 +4154,9 @@ def bulk_delete_recordings():
continue
# Delete audio file
- if recording.audio_path and os.path.exists(recording.audio_path):
+ if recording.audio_path:
try:
- os.remove(recording.audio_path)
+ get_storage_service().delete(recording.audio_path, missing_ok=True)
except Exception as e:
current_app.logger.error(f"Error deleting audio file {recording.audio_path}: {e}")
@@ -4268,7 +4318,7 @@ def bulk_reprocess():
# For transcription reprocess, need audio file
if reprocess_type == 'transcription':
- if not recording.audio_path or not os.path.exists(recording.audio_path):
+ if not recording.audio_path or not get_storage_service().exists(recording.audio_path):
continue
job_type = 'reprocess_transcription'
else:
diff --git a/src/api/shares.py b/src/api/shares.py
index d875058d..d16dcd59 100644
--- a/src/api/shares.py
+++ b/src/api/shares.py
@@ -17,6 +17,7 @@ from werkzeug.exceptions import HTTPException
from src.database import db
from src.models import Recording, Share, InternalShare, SharedRecordingState, User, TranscriptChunk, ShareAuditLog
from src.utils import md_to_html
+from src.services.storage import get_storage_service
# Configuration from environment
ENABLE_PUBLIC_SHARING = os.environ.get('ENABLE_PUBLIC_SHARING', 'true').lower() == 'true'
@@ -157,10 +158,19 @@ def get_shared_audio(public_id):
recording = share.recording
if not recording or not recording.audio_path:
return jsonify({'error': 'Recording or audio file not found'}), 404
- if not os.path.exists(recording.audio_path):
+ delivery = get_storage_service().get_audio_delivery(
+ recording.audio_path,
+ download=False,
+ mime_type=recording.mime_type,
+ download_name=recording.original_filename,
+ is_public=True,
+ )
+ if delivery.mode == 'redirect_url':
+ return redirect(delivery.url, code=302)
+ if not delivery.local_path or not os.path.exists(delivery.local_path):
current_app.logger.error(f"Audio file missing from server: {recording.audio_path}")
return jsonify({'error': 'Audio file missing from server'}), 404
- return send_file(recording.audio_path, conditional=True)
+ return send_file(delivery.local_path, mimetype=(recording.mime_type or delivery.mimetype), conditional=True)
except HTTPException:
# Let first_or_404() (and any other HTTP error) keep its real status —
# an unknown public_id must stay a 404, not be masked as a 500.
diff --git a/src/api/speakers.py b/src/api/speakers.py
index acad8158..878ee5e0 100644
--- a/src/api/speakers.py
+++ b/src/api/speakers.py
@@ -459,43 +459,46 @@ def get_snippet_audio(recording_id):
if recording.audio_deleted_at:
return jsonify({'error': 'Audio file has been deleted'}), 410
- if not recording.audio_path or not os.path.exists(recording.audio_path):
+ from src.services.storage import get_storage_service
+ storage = get_storage_service()
+ if not recording.audio_path or not storage.exists(recording.audio_path):
return jsonify({'error': 'Audio file not found'}), 404
- # Detect audio codec to pick the right output container for stream copy
- codec_to_container = {
- 'mp3': ('.mp3', 'audio/mpeg'),
- 'aac': ('.m4a', 'audio/mp4'),
- 'opus': ('.ogg', 'audio/ogg'),
- 'vorbis': ('.ogg', 'audio/ogg'),
- 'flac': ('.flac', 'audio/flac'),
- 'pcm_s16le': ('.wav', 'audio/wav'),
- 'pcm_s24le': ('.wav', 'audio/wav'),
- 'pcm_s32le': ('.wav', 'audio/wav'),
- 'pcm_f32le': ('.wav', 'audio/wav'),
- }
- snippet_ext = '.mp3'
- snippet_mime = 'audio/mpeg'
- try:
- codec_info = get_codec_info(recording.audio_path, timeout=10)
- audio_codec = codec_info.get('audio_codec')
- if audio_codec and audio_codec in codec_to_container:
- snippet_ext, snippet_mime = codec_to_container[audio_codec]
- except FFProbeError:
- pass # Fall back to mp3
-
- # Create temporary file for the snippet
- with tempfile.NamedTemporaryFile(delete=False, suffix=snippet_ext) as tmp_file:
- output_path = tmp_file.name
-
try:
- # Use centralized FFmpeg utility to extract the audio segment
- extract_audio_segment(
- recording.audio_path,
- output_path,
- start_time,
- duration
- )
+ with storage.materialize(recording.audio_path) as materialized:
+ # Detect audio codec to pick the right output container for stream copy
+ codec_to_container = {
+ 'mp3': ('.mp3', 'audio/mpeg'),
+ 'aac': ('.m4a', 'audio/mp4'),
+ 'opus': ('.ogg', 'audio/ogg'),
+ 'vorbis': ('.ogg', 'audio/ogg'),
+ 'flac': ('.flac', 'audio/flac'),
+ 'pcm_s16le': ('.wav', 'audio/wav'),
+ 'pcm_s24le': ('.wav', 'audio/wav'),
+ 'pcm_s32le': ('.wav', 'audio/wav'),
+ 'pcm_f32le': ('.wav', 'audio/wav'),
+ }
+ snippet_ext = '.mp3'
+ snippet_mime = 'audio/mpeg'
+ try:
+ codec_info = get_codec_info(materialized.local_path, timeout=10)
+ audio_codec = codec_info.get('audio_codec')
+ if audio_codec and audio_codec in codec_to_container:
+ snippet_ext, snippet_mime = codec_to_container[audio_codec]
+ except FFProbeError:
+ pass # Fall back to mp3
+
+ # Create temporary file for the snippet
+ with tempfile.NamedTemporaryFile(delete=False, suffix=snippet_ext) as tmp_file:
+ output_path = tmp_file.name
+
+ # Use centralized FFmpeg utility to extract the audio segment
+ extract_audio_segment(
+ materialized.local_path,
+ output_path,
+ start_time,
+ duration
+ )
# Send the file
response = send_file(
@@ -593,4 +596,3 @@ def merge_speaker_profiles():
return jsonify({'error': str(e)}), 500
-
diff --git a/src/config/app_config.py b/src/config/app_config.py
index b4111b16..99f66238 100644
--- a/src/config/app_config.py
+++ b/src/config/app_config.py
@@ -87,6 +87,22 @@ AUDIO_BITRATE = os.environ.get('AUDIO_BITRATE', '128k') # For lossy codecs
# Useful for custom ASR backends that handle video/multi-track audio internally
VIDEO_PASSTHROUGH_ASR = os.environ.get('VIDEO_PASSTHROUGH_ASR', 'false').lower() == 'true'
+# File storage backend configuration (dual local / S3-compatible)
+UPLOAD_FOLDER = os.environ.get('UPLOAD_FOLDER', '/data/uploads')
+FILE_STORAGE_BACKEND = os.environ.get('FILE_STORAGE_BACKEND', 'local').lower().strip() # local | s3
+FILE_STORAGE_KEY_PREFIX = (os.environ.get('FILE_STORAGE_KEY_PREFIX', 'recordings') or 'recordings').strip().strip('/')
+FILE_STORAGE_STAGING_DIR = os.environ.get('FILE_STORAGE_STAGING_DIR', '').strip() or os.path.join(UPLOAD_FOLDER, '_staging')
+S3_BUCKET_NAME = os.environ.get('S3_BUCKET_NAME', '').strip() or None
+S3_REGION = os.environ.get('S3_REGION', '').strip() or None
+S3_ENDPOINT_URL = os.environ.get('S3_ENDPOINT_URL', '').strip() or None
+S3_ACCESS_KEY_ID = os.environ.get('S3_ACCESS_KEY_ID', '').strip() or None
+S3_SECRET_ACCESS_KEY = os.environ.get('S3_SECRET_ACCESS_KEY', '').strip() or None
+S3_SESSION_TOKEN = os.environ.get('S3_SESSION_TOKEN', '').strip() or None
+S3_USE_PATH_STYLE = os.environ.get('S3_USE_PATH_STYLE', 'false').lower() == 'true'
+S3_VERIFY_SSL = os.environ.get('S3_VERIFY_SSL', 'true').lower() == 'true'
+S3_PRESIGN_TTL_SECONDS = int(os.environ.get('S3_PRESIGN_TTL_SECONDS', '900'))
+S3_PRESIGN_PUBLIC_TTL_SECONDS = int(os.environ.get('S3_PRESIGN_PUBLIC_TTL_SECONDS', '300'))
+
# Unsupported codecs - comma-separated list of codecs to exclude from the default supported list
# Useful when your transcription service doesn't support certain codecs (e.g., vllm doesn't support opus)
# Example: AUDIO_UNSUPPORTED_CODECS=opus,vorbis
diff --git a/src/file_monitor.py b/src/file_monitor.py
index 2ca5b8be..a9e881a7 100644
--- a/src/file_monitor.py
+++ b/src/file_monitor.py
@@ -16,7 +16,7 @@ from datetime import datetime
from pathlib import Path
import mimetypes
from werkzeug.utils import secure_filename
-from src.utils.ffprobe import get_codec_info, get_creation_date, FFProbeError
+from src.utils.ffprobe import get_codec_info, get_creation_date, get_duration, FFProbeError
from src.utils.ffmpeg_utils import FFmpegError, FFmpegNotFoundError
from src.utils.audio_conversion import convert_if_needed
@@ -336,9 +336,12 @@ class FileMonitor:
timestamp = datetime.now().strftime('%Y%m%d%H%M%S')
new_filename = f"auto_{timestamp}_{safe_filename}"
- uploads_dir = Path(app.config['UPLOAD_FOLDER'])
- uploads_dir.mkdir(parents=True, exist_ok=True)
- destination_path = uploads_dir / new_filename
+ from src.services.storage import get_storage_service
+
+ storage = get_storage_service()
+ staging_dir = Path(storage.get_staging_dir())
+ staging_dir.mkdir(parents=True, exist_ok=True)
+ destination_path = staging_dir / new_filename
# Copy locked file to uploads directory
import shutil
@@ -443,8 +446,16 @@ class FileMonitor:
f"({existing.title}). Processing anyway."
)
+ audio_duration_seconds = None
+ try:
+ detected_duration = get_duration(str(final_path), timeout=30)
+ if detected_duration is not None and detected_duration > 0:
+ audio_duration_seconds = float(detected_duration)
+ except Exception as e:
+ self.logger.warning(f"Could not determine duration for auto-processed file {original_filename}: {e}")
+
recording = Recording(
- audio_path=str(final_path),
+ audio_path=None,
original_filename=original_filename,
title=f"Auto-processed - {original_filename}",
file_size=file_size,
@@ -452,13 +463,27 @@ class FileMonitor:
meeting_date=meeting_date,
user_id=user_id,
mime_type=mime_type,
+ audio_duration_seconds=audio_duration_seconds,
is_inbox=True, # Auto-processed files go to inbox
processing_source='auto_process', # Track that this was auto-processed
file_hash=file_hash
)
db.session.add(recording)
- db.session.commit()
+ db.session.flush()
+
+ storage_key = storage.build_recording_key(original_filename, recording.id, now=now)
+ stored_object = storage.upload_local_file(
+ str(final_path),
+ storage_key,
+ content_type=mime_type,
+ delete_source=True,
+ )
+ recording.audio_path = stored_object.locator
+
+ user_hotwords = (user.transcription_hotwords or '').strip() if getattr(user, 'transcription_hotwords', None) else None
+ user_initial_prompt = (user.transcription_initial_prompt or '').strip() if getattr(user, 'transcription_initial_prompt', None) else None
+ user_language = (user.transcription_language or '').strip() if getattr(user, 'transcription_language', None) else None
self.logger.info(f"Created recording record with ID: {recording.id} for user: {user.username}")
@@ -492,6 +517,15 @@ class FileMonitor:
job_params['custom_prompt'] = tag.custom_prompt
job_params['tag_id'] = tag_id
+ if 'language' not in job_params and user_language:
+ job_params['language'] = user_language
+ if 'hotwords' not in job_params and user_hotwords:
+ job_params['hotwords'] = user_hotwords
+ if 'initial_prompt' not in job_params and user_initial_prompt:
+ job_params['initial_prompt'] = user_initial_prompt
+
+ db.session.commit()
+
# Queue for background processing
from src.services.job_queue import job_queue
job_queue.enqueue(
diff --git a/src/init_db.py b/src/init_db.py
index 5e7d016c..61593431 100644
--- a/src/init_db.py
+++ b/src/init_db.py
@@ -181,6 +181,8 @@ def initialize_database(app):
if add_column_if_not_exists(engine, 'recording', 'mime_type', 'VARCHAR(100)'):
app.logger.info("Added mime_type column to recording table")
+ if add_column_if_not_exists(engine, 'recording', 'audio_duration_seconds', 'FLOAT'):
+ app.logger.info("Added audio_duration_seconds column to recording table")
if add_column_if_not_exists(engine, 'recording', 'completed_at', 'DATETIME'):
app.logger.info("Added completed_at column to recording table")
if add_column_if_not_exists(engine, 'recording', 'processing_time_seconds', 'INTEGER'):
diff --git a/src/models/recording.py b/src/models/recording.py
index f7ecfcc3..2ab2b9e2 100644
--- a/src/models/recording.py
+++ b/src/models/recording.py
@@ -35,6 +35,7 @@ class Recording(db.Model):
is_inbox = db.Column(db.Boolean, default=True) # New recordings are marked as inbox by default
is_highlighted = db.Column(db.Boolean, default=False) # Recordings can be highlighted by the user
mime_type = db.Column(db.String(100), nullable=True)
+ audio_duration_seconds = db.Column(db.Float, nullable=True) # Cached audio duration to avoid materializing remote storage during serialization
completed_at = db.Column(db.DateTime, nullable=True)
processing_time_seconds = db.Column(db.Integer, nullable=True)
transcription_duration_seconds = db.Column(db.Integer, nullable=True) # Time taken for transcription
@@ -149,21 +150,18 @@ class Recording(db.Model):
).first()
return state.personal_notes if state else None
- def get_audio_duration(self, allow_probe=False):
+ def get_audio_duration(self, allow_probe_fallback=False):
"""
- Get the audio duration in seconds.
+ Get cached audio duration in seconds.
- Reads the cached ``audio_duration_seconds`` column when present
- (populated at transcription completion in src/tasks/processing.py).
- Falls back to a live ffprobe probe only when ``allow_probe=True``
- AND the cached value is missing — list/detail API serializers
- should leave this False to avoid an ffprobe subprocess per row.
+ By default this avoids materializing audio from storage (especially S3)
+ during normal API serialization (which would otherwise turn a list
+ serialization into an ffprobe subprocess per row). Optional probe
+ fallback is kept for explicit callers that need best-effort backfill.
Args:
- allow_probe: If True, ffprobe the file when no cached value
- exists. Defaults to False so callers can't accidentally
- turn a serialization step into a multi-row subprocess
- blast.
+ allow_probe_fallback: When True, probe the stored file if cached
+ duration is missing.
Returns:
Float duration in seconds, or None if unavailable.
@@ -171,15 +169,29 @@ class Recording(db.Model):
if self.audio_deleted_at is not None:
return None
if self.audio_duration_seconds is not None:
- return float(self.audio_duration_seconds)
- if not allow_probe:
+ try:
+ return float(self.audio_duration_seconds)
+ except (TypeError, ValueError):
+ return None
+
+ if not allow_probe_fallback:
return None
- if not self.audio_path or not os.path.exists(self.audio_path):
+
+ if not self.audio_path:
return None
try:
+ from src.services.storage import get_storage_service
from src.utils.ffprobe import get_duration
- duration = get_duration(self.audio_path, timeout=30)
- return duration
+ storage = get_storage_service()
+ if not storage.exists(self.audio_path):
+ return None
+
+ with storage.materialize(self.audio_path) as materialized:
+ # Allow longer timeout for packet scanning fallback on files without duration metadata
+ duration = get_duration(materialized.local_path, timeout=30)
+ if duration is not None:
+ self.audio_duration_seconds = float(duration)
+ return duration
except Exception as e:
logger.warning(f"Failed to get duration for recording {self.id}: {e}")
return None
@@ -367,7 +379,7 @@ class Recording(db.Model):
'mime_type': self.mime_type,
'audio_deleted_at': self.audio_deleted_at.isoformat() if self.audio_deleted_at else None,
'audio_available': self.audio_deleted_at is None,
- 'audio_duration': self.get_audio_duration(),
+ 'audio_duration': self.get_audio_duration(allow_probe_fallback=False),
'deletion_exempt': self.deletion_exempt,
'folder_id': self.folder_id,
'folder': self.folder.to_dict() if self.folder else None,
diff --git a/src/services/job_queue.py b/src/services/job_queue.py
index 7710e015..6fd30f6e 100644
--- a/src/services/job_queue.py
+++ b/src/services/job_queue.py
@@ -420,34 +420,42 @@ class FairJobQueue:
"""Run transcription task. Status updates handled by task function."""
from src.tasks.processing import transcribe_audio_task
from flask import current_app
-
- filepath = recording.audio_path
- filename_for_asr = recording.original_filename or os.path.basename(filepath)
-
- # Emit recording.transcription.started AFTER we know the file
- # is on disk so subscribers don't see misleading started→failed
- # events for jobs that abort immediately. Best-effort; emit
- # errors don't block the actual work.
- if filepath and os.path.exists(filepath):
- try:
- self._emit_started_webhook('transcribe', recording.id)
- except Exception as e:
- logger.warning(f"Webhook emit (transcribe started) failed: {e}")
-
- transcribe_audio_task(
- current_app._get_current_object().app_context(),
- recording.id,
- filepath,
- filename_for_asr,
- datetime.utcnow(),
- language=params.get('language'),
- min_speakers=params.get('min_speakers'),
- max_speakers=params.get('max_speakers'),
- tag_id=params.get('tag_id'),
- hotwords=params.get('hotwords'),
- initial_prompt=params.get('initial_prompt'),
- transcription_model=params.get('transcription_model'),
- )
+ from src.services.storage import get_storage_service
+
+ storage = get_storage_service()
+ audio_path = (recording.audio_path or '').strip()
+ if not audio_path:
+ logger.error(f"Cannot run transcription job for recording {recording.id}: missing audio_path")
+ raise ValueError(f"Missing audio_path for recording {recording.id}")
+ with storage.materialize(audio_path) as materialized:
+ filepath = materialized.local_path
+ filename_for_asr = recording.original_filename or os.path.basename(filepath)
+
+ # Emit recording.transcription.started AFTER the file is on disk
+ # (materialized from storage) so subscribers don't see misleading
+ # started→failed events for jobs that abort immediately. Best-effort;
+ # emit errors don't block the actual work. Kept INSIDE the materialize
+ # block so the temp file still exists when transcription runs.
+ if filepath and os.path.exists(filepath):
+ try:
+ self._emit_started_webhook('transcribe', recording.id)
+ except Exception as e:
+ logger.warning(f"Webhook emit (transcribe started) failed: {e}")
+
+ transcribe_audio_task(
+ current_app._get_current_object().app_context(),
+ recording.id,
+ filepath,
+ filename_for_asr,
+ datetime.utcnow(),
+ language=params.get('language'),
+ min_speakers=params.get('min_speakers'),
+ max_speakers=params.get('max_speakers'),
+ tag_id=params.get('tag_id'),
+ hotwords=params.get('hotwords'),
+ initial_prompt=params.get('initial_prompt'),
+ transcription_model=params.get('transcription_model'),
+ )
def _run_summarization(self, job, recording, params):
"""Run summarization-only task. Status updates handled by task function."""
@@ -466,31 +474,39 @@ class FairJobQueue:
"""Run transcription reprocessing task. Status updates handled by task function."""
from src.tasks.processing import transcribe_audio_task
from flask import current_app
-
- filepath = recording.audio_path
- filename_for_asr = recording.original_filename or os.path.basename(filepath)
-
- # Emit started AFTER file existence check (see _run_transcription).
- if filepath and os.path.exists(filepath):
- try:
- self._emit_started_webhook('reprocess_transcription', recording.id)
- except Exception as e:
- logger.warning(f"Webhook emit (reprocess_transcription started) failed: {e}")
-
- transcribe_audio_task(
- current_app._get_current_object().app_context(),
- recording.id,
- filepath,
- filename_for_asr,
- datetime.utcnow(),
- language=params.get('language'),
- min_speakers=params.get('min_speakers'),
- max_speakers=params.get('max_speakers'),
- tag_id=params.get('tag_id'),
- hotwords=params.get('hotwords'),
- initial_prompt=params.get('initial_prompt'),
- transcription_model=params.get('transcription_model'),
- )
+ from src.services.storage import get_storage_service
+
+ storage = get_storage_service()
+ audio_path = (recording.audio_path or '').strip()
+ if not audio_path:
+ logger.error(f"Cannot run reprocess transcription job for recording {recording.id}: missing audio_path")
+ raise ValueError(f"Missing audio_path for recording {recording.id}")
+ with storage.materialize(audio_path) as materialized:
+ filepath = materialized.local_path
+ filename_for_asr = recording.original_filename or os.path.basename(filepath)
+
+ # Emit started AFTER file existence check (see _run_transcription).
+ # Kept INSIDE the materialize block so the temp file still exists.
+ if filepath and os.path.exists(filepath):
+ try:
+ self._emit_started_webhook('reprocess_transcription', recording.id)
+ except Exception as e:
+ logger.warning(f"Webhook emit (reprocess_transcription started) failed: {e}")
+
+ transcribe_audio_task(
+ current_app._get_current_object().app_context(),
+ recording.id,
+ filepath,
+ filename_for_asr,
+ datetime.utcnow(),
+ language=params.get('language'),
+ min_speakers=params.get('min_speakers'),
+ max_speakers=params.get('max_speakers'),
+ tag_id=params.get('tag_id'),
+ hotwords=params.get('hotwords'),
+ initial_prompt=params.get('initial_prompt'),
+ transcription_model=params.get('transcription_model'),
+ )
def _run_reprocess_summary(self, job, recording, params):
"""Run summary reprocessing task. Status updates handled by task function."""
diff --git a/src/services/retention.py b/src/services/retention.py
index 35af4d71..61d17dfd 100644
--- a/src/services/retention.py
+++ b/src/services/retention.py
@@ -8,6 +8,7 @@ from flask import current_app
from src.database import db
from src.models import Recording, RecordingTag, Tag
+from src.services.storage import get_storage_service
ENABLE_AUTO_DELETION = os.environ.get('ENABLE_AUTO_DELETION', 'false').lower() == 'true'
GLOBAL_RETENTION_DAYS = int(os.environ.get('GLOBAL_RETENTION_DAYS', '0'))
@@ -151,9 +152,10 @@ def process_auto_deletion():
# Determine deletion mode
if DELETION_MODE == 'audio_only':
# Delete only the audio file, keep transcription
- if recording.audio_path and os.path.exists(recording.audio_path):
+ storage = get_storage_service()
+ if recording.audio_path and storage.exists(recording.audio_path):
current_app.logger.info(f"Recording {recording.id} is past retention ({retention_days} days), deleting audio")
- os.remove(recording.audio_path)
+ storage.delete(recording.audio_path, missing_ok=True)
current_app.logger.info(f"Auto-deleted audio file: {recording.audio_path}")
recording.audio_deleted_at = datetime.utcnow()
db.session.commit()
@@ -173,8 +175,8 @@ def process_auto_deletion():
current_app.logger.info(f"Recording {recording.id} is past retention ({retention_days} days), deleting fully")
# Delete audio file if it exists
- if recording.audio_path and os.path.exists(recording.audio_path):
- os.remove(recording.audio_path)
+ if recording.audio_path:
+ get_storage_service().delete(recording.audio_path, missing_ok=True)
# Delete associated processing jobs (required due to NOT NULL constraint)
from src.models.processing_job import ProcessingJob
diff --git a/src/services/storage/__init__.py b/src/services/storage/__init__.py
new file mode 100644
index 00000000..edd01ec8
--- /dev/null
+++ b/src/services/storage/__init__.py
@@ -0,0 +1,20 @@
+"""Unified file storage service supporting local and S3 backends."""
+
+from .interfaces import AudioDeliveryResult, MaterializedFile, ObjectStat, StorageLocator, StoredObject
+from .locator import build_local_locator, build_s3_locator, parse_locator, relative_key_from_local_path
+from .service import StorageService, get_storage_service, reset_storage_service_singleton
+
+__all__ = [
+ 'AudioDeliveryResult',
+ 'MaterializedFile',
+ 'ObjectStat',
+ 'StorageLocator',
+ 'StoredObject',
+ 'build_local_locator',
+ 'build_s3_locator',
+ 'parse_locator',
+ 'relative_key_from_local_path',
+ 'StorageService',
+ 'get_storage_service',
+ 'reset_storage_service_singleton',
+]
diff --git a/src/services/storage/factory.py b/src/services/storage/factory.py
new file mode 100644
index 00000000..4f45de47
--- /dev/null
+++ b/src/services/storage/factory.py
@@ -0,0 +1,69 @@
+"""Factory for configuring file storage backends from environment variables."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Optional
+
+from .local import LocalStorageBackend
+from .s3 import S3StorageBackend
+
+
+@dataclass
+class StorageSettings:
+ backend: str
+ local_root: str
+ key_prefix: str
+ staging_dir: str
+ presign_ttl_seconds: int
+ presign_public_ttl_seconds: int
+ s3_bucket_name: Optional[str] = None
+ s3_region: Optional[str] = None
+ s3_endpoint_url: Optional[str] = None
+ s3_access_key_id: Optional[str] = None
+ s3_secret_access_key: Optional[str] = None
+ s3_session_token: Optional[str] = None
+ s3_use_path_style: bool = False
+ s3_verify_ssl: bool = True
+
+
+def load_storage_settings_from_env() -> StorageSettings:
+ # Kept for backward compatibility of the factory API, but values now come
+ # from app_config to avoid duplicated env parsing / drift.
+ from src.config import app_config
+
+ return StorageSettings(
+ backend=(app_config.FILE_STORAGE_BACKEND or 'local').strip().lower() or 'local',
+ local_root=app_config.UPLOAD_FOLDER,
+ key_prefix=(app_config.FILE_STORAGE_KEY_PREFIX or 'recordings').strip().strip('/').replace('\\', '/'),
+ staging_dir=app_config.FILE_STORAGE_STAGING_DIR,
+ presign_ttl_seconds=int(app_config.S3_PRESIGN_TTL_SECONDS),
+ presign_public_ttl_seconds=int(app_config.S3_PRESIGN_PUBLIC_TTL_SECONDS),
+ s3_bucket_name=app_config.S3_BUCKET_NAME,
+ s3_region=app_config.S3_REGION,
+ s3_endpoint_url=app_config.S3_ENDPOINT_URL,
+ s3_access_key_id=app_config.S3_ACCESS_KEY_ID,
+ s3_secret_access_key=app_config.S3_SECRET_ACCESS_KEY,
+ s3_session_token=app_config.S3_SESSION_TOKEN,
+ s3_use_path_style=bool(app_config.S3_USE_PATH_STYLE),
+ s3_verify_ssl=bool(app_config.S3_VERIFY_SSL),
+ )
+
+
+def build_local_backend(settings: StorageSettings) -> LocalStorageBackend:
+ return LocalStorageBackend(settings.local_root)
+
+
+def build_s3_backend(settings: StorageSettings) -> Optional[S3StorageBackend]:
+ if not settings.s3_bucket_name:
+ return None
+ return S3StorageBackend(
+ bucket=settings.s3_bucket_name,
+ region=settings.s3_region,
+ endpoint_url=settings.s3_endpoint_url,
+ access_key_id=settings.s3_access_key_id,
+ secret_access_key=settings.s3_secret_access_key,
+ session_token=settings.s3_session_token,
+ use_path_style=settings.s3_use_path_style,
+ verify_ssl=settings.s3_verify_ssl,
+ )
diff --git a/src/services/storage/interfaces.py b/src/services/storage/interfaces.py
new file mode 100644
index 00000000..f068ed45
--- /dev/null
+++ b/src/services/storage/interfaces.py
@@ -0,0 +1,69 @@
+"""Storage interfaces and shared dataclasses for file storage backends."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from datetime import datetime
+from typing import Optional
+
+
+@dataclass
+class StorageLocator:
+ """Parsed locator for local/s3/legacy paths."""
+
+ scheme: str # local | s3 | legacy_local_abs | legacy_local_rel
+ raw: str
+ key: Optional[str] = None
+ bucket: Optional[str] = None
+ path: Optional[str] = None # absolute local path for legacy abs
+
+ @property
+ def is_legacy(self) -> bool:
+ return self.scheme.startswith('legacy_')
+
+ @property
+ def is_local(self) -> bool:
+ return self.scheme in ('local', 'legacy_local_abs', 'legacy_local_rel')
+
+ @property
+ def is_s3(self) -> bool:
+ return self.scheme == 's3'
+
+
+@dataclass
+class StoredObject:
+ """Result of storing an object."""
+
+ locator: str
+ key: str
+ size: Optional[int] = None
+ content_type: Optional[str] = None
+ etag: Optional[str] = None
+
+
+@dataclass
+class ObjectStat:
+ """Storage object metadata."""
+
+ size: Optional[int] = None
+ last_modified: Optional[datetime] = None
+ etag: Optional[str] = None
+ content_type: Optional[str] = None
+
+
+@dataclass
+class MaterializedFile:
+ """Local filesystem path prepared for processing/read."""
+
+ local_path: str
+ cleanup_required: bool = False
+
+
+@dataclass
+class AudioDeliveryResult:
+ """How API should deliver audio to client."""
+
+ mode: str # local_file | redirect_url
+ mimetype: Optional[str] = None
+ local_path: Optional[str] = None
+ url: Optional[str] = None
diff --git a/src/services/storage/local.py b/src/services/storage/local.py
new file mode 100644
index 00000000..d2b687f7
--- /dev/null
+++ b/src/services/storage/local.py
@@ -0,0 +1,81 @@
+"""Local filesystem storage backend."""
+
+from __future__ import annotations
+
+import os
+import shutil
+from pathlib import Path
+from typing import BinaryIO, Optional
+
+from .interfaces import MaterializedFile, ObjectStat, StorageLocator, StoredObject
+from .locator import build_local_locator, local_path_from_key
+
+
+class LocalStorageBackend:
+ """Local filesystem implementation for the storage contract."""
+
+ def __init__(self, root: str):
+ self.root = str(Path(root))
+ Path(self.root).mkdir(parents=True, exist_ok=True)
+
+ def build_locator(self, key: str) -> str:
+ return build_local_locator(key)
+
+ def resolve_path(self, locator: StorageLocator) -> str:
+ if locator.scheme == 'local':
+ if not locator.key:
+ raise ValueError('local locator missing key')
+ return local_path_from_key(self.root, locator.key)
+ if locator.scheme == 'legacy_local_abs':
+ if not locator.path:
+ raise ValueError('legacy absolute local path is empty')
+ return locator.path
+ if locator.scheme == 'legacy_local_rel':
+ if not locator.key:
+ raise ValueError('legacy relative local path is empty')
+ return local_path_from_key(self.root, locator.key)
+ raise ValueError(f"Unsupported locator for local backend: {locator.scheme}")
+
+ def save_fileobj(self, fileobj: BinaryIO, key: str, content_type: Optional[str] = None, metadata: Optional[dict] = None) -> StoredObject:
+ dst = local_path_from_key(self.root, key)
+ Path(dst).parent.mkdir(parents=True, exist_ok=True)
+ with open(dst, 'wb') as out_f:
+ shutil.copyfileobj(fileobj, out_f)
+ size = os.path.getsize(dst)
+ return StoredObject(locator=self.build_locator(key), key=key, size=size, content_type=content_type)
+
+ def upload_local_file(self, local_path: str, key: str, content_type: Optional[str] = None, metadata: Optional[dict] = None, delete_source: bool = False) -> StoredObject:
+ dst = local_path_from_key(self.root, key)
+ Path(dst).parent.mkdir(parents=True, exist_ok=True)
+
+ src_resolved = str(Path(local_path).resolve())
+ dst_resolved = str(Path(dst).resolve())
+ if src_resolved != dst_resolved:
+ if delete_source:
+ shutil.move(local_path, dst)
+ else:
+ shutil.copy2(local_path, dst)
+ size = os.path.getsize(dst)
+ return StoredObject(locator=self.build_locator(key), key=key, size=size, content_type=content_type)
+
+ def exists(self, locator: StorageLocator) -> bool:
+ return os.path.exists(self.resolve_path(locator))
+
+ def delete(self, locator: StorageLocator, missing_ok: bool = True) -> bool:
+ path = self.resolve_path(locator)
+ if not os.path.exists(path):
+ return bool(missing_ok)
+ os.remove(path)
+ return True
+
+ def stat(self, locator: StorageLocator) -> ObjectStat:
+ path = self.resolve_path(locator)
+ st = os.stat(path)
+ return ObjectStat(size=st.st_size)
+
+ def materialize(self, locator: StorageLocator) -> MaterializedFile:
+ path = self.resolve_path(locator)
+ return MaterializedFile(local_path=path, cleanup_required=False)
+
+ def presign_get_url(self, locator: StorageLocator, *args, **kwargs) -> str:
+ raise NotImplementedError('Local backend does not support presigned URLs')
diff --git a/src/services/storage/locator.py b/src/services/storage/locator.py
new file mode 100644
index 00000000..f342c9ce
--- /dev/null
+++ b/src/services/storage/locator.py
@@ -0,0 +1,97 @@
+"""Locator parsing/serialization helpers for local and S3 storage."""
+
+from __future__ import annotations
+
+import os
+from pathlib import Path
+from typing import Optional
+
+from .interfaces import StorageLocator
+
+LOCAL_SCHEME = 'local://'
+S3_SCHEME = 's3://'
+
+
+def _normalize_key(key: str) -> str:
+ key = (key or '').replace('\\', '/').strip()
+ while '//' in key:
+ key = key.replace('//', '/')
+ return key.lstrip('/')
+
+
+def build_local_locator(key: str) -> str:
+ return f"{LOCAL_SCHEME}{_normalize_key(key)}"
+
+
+def build_s3_locator(bucket: str, key: str) -> str:
+ return f"{S3_SCHEME}{bucket}/{_normalize_key(key)}"
+
+
+def is_probably_windows_abs(path: str) -> bool:
+ if not path or len(path) < 3:
+ return False
+ return path[1] == ':' and path[2] in ('\\', '/')
+
+
+def is_absolute_local_path(value: str) -> bool:
+ if not value:
+ return False
+ if is_probably_windows_abs(value):
+ return True
+ return os.path.isabs(value)
+
+
+def parse_locator(value: Optional[str]) -> Optional[StorageLocator]:
+ """Parse locator string into a typed structure."""
+ if value is None:
+ return None
+
+ raw = str(value).strip()
+ if not raw:
+ return None
+
+ if raw.startswith(LOCAL_SCHEME):
+ key = _normalize_key(raw[len(LOCAL_SCHEME):])
+ return StorageLocator(scheme='local', raw=raw, key=key)
+
+ if raw.startswith(S3_SCHEME):
+ tail = raw[len(S3_SCHEME):]
+ if '/' not in tail:
+ raise ValueError(f"Invalid s3 locator (missing key): {raw}")
+ bucket, key = tail.split('/', 1)
+ bucket = bucket.strip()
+ key = _normalize_key(key)
+ if not bucket or not key:
+ raise ValueError(f"Invalid s3 locator: {raw}")
+ return StorageLocator(scheme='s3', raw=raw, bucket=bucket, key=key)
+
+ if is_absolute_local_path(raw):
+ return StorageLocator(scheme='legacy_local_abs', raw=raw, path=raw)
+
+ return StorageLocator(scheme='legacy_local_rel', raw=raw, key=_normalize_key(raw))
+
+
+def local_path_from_key(local_root: str, key: str) -> str:
+ """Resolve a storage key under local_root and prevent path traversal."""
+ safe_key = _normalize_key(key)
+ root = Path(local_root).resolve()
+ candidate = (root / Path(*safe_key.split('/'))).resolve()
+ try:
+ candidate.relative_to(root)
+ except Exception as exc:
+ raise ValueError(f"Local storage key resolves outside root: {key}") from exc
+ return str(candidate)
+
+
+def relative_key_from_local_path(abs_path: str, local_root: str) -> str:
+ """Convert absolute local path to a storage key relative to local root."""
+ root = Path(local_root).resolve()
+ path = Path(abs_path).resolve()
+ try:
+ rel = path.relative_to(root)
+ except Exception as exc:
+ raise ValueError(f"Path '{abs_path}' is outside local storage root '{local_root}'") from exc
+ rel_key = _normalize_key(rel.as_posix())
+ if not rel_key:
+ raise ValueError(f"Cannot build key from path '{abs_path}'")
+ return rel_key
diff --git a/src/services/storage/s3.py b/src/services/storage/s3.py
new file mode 100644
index 00000000..3d0bf498
--- /dev/null
+++ b/src/services/storage/s3.py
@@ -0,0 +1,160 @@
+"""S3-compatible storage backend (AWS S3 / MinIO)."""
+
+from __future__ import annotations
+
+import os
+import tempfile
+from pathlib import Path
+from typing import BinaryIO, Optional
+
+from .interfaces import MaterializedFile, ObjectStat, StorageLocator, StoredObject
+from .locator import build_s3_locator
+
+
+class S3StorageBackend:
+ """S3 storage backend with lazy boto3 initialization."""
+
+ def __init__(self, *, bucket: str, region: Optional[str] = None, endpoint_url: Optional[str] = None,
+ access_key_id: Optional[str] = None, secret_access_key: Optional[str] = None,
+ session_token: Optional[str] = None, use_path_style: bool = False,
+ verify_ssl: bool = True):
+ self.bucket = bucket
+ self.region = region
+ self.endpoint_url = endpoint_url
+ self.access_key_id = access_key_id
+ self.secret_access_key = secret_access_key
+ self.session_token = session_token
+ self.use_path_style = use_path_style
+ self.verify_ssl = verify_ssl
+ self._client = None
+
+ def _get_client(self):
+ if self._client is not None:
+ return self._client
+
+ try:
+ import boto3
+ from botocore.config import Config
+ except Exception as exc:
+ raise RuntimeError('S3 backend requires boto3 and botocore installed') from exc
+
+ client_kwargs = {
+ 'service_name': 's3',
+ 'verify': self.verify_ssl,
+ }
+ if self.region:
+ client_kwargs['region_name'] = self.region
+ if self.endpoint_url:
+ client_kwargs['endpoint_url'] = self.endpoint_url
+ if self.access_key_id:
+ client_kwargs['aws_access_key_id'] = self.access_key_id
+ if self.secret_access_key:
+ client_kwargs['aws_secret_access_key'] = self.secret_access_key
+ if self.session_token:
+ client_kwargs['aws_session_token'] = self.session_token
+
+ addressing_style = 'path' if self.use_path_style else 'auto'
+ client_kwargs['config'] = Config(signature_version='s3v4', s3={'addressing_style': addressing_style})
+
+ self._client = boto3.client(**client_kwargs)
+ return self._client
+
+ def _bucket_key(self, locator: StorageLocator):
+ if locator.scheme != 's3':
+ raise ValueError(f"Unsupported locator for S3 backend: {locator.scheme}")
+ bucket = locator.bucket or self.bucket
+ key = locator.key
+ if not bucket or not key:
+ raise ValueError('S3 locator missing bucket or key')
+ return bucket, key
+
+ def build_locator(self, key: str) -> str:
+ return build_s3_locator(self.bucket, key)
+
+ def save_fileobj(self, fileobj: BinaryIO, key: str, content_type: Optional[str] = None, metadata: Optional[dict] = None) -> StoredObject:
+ client = self._get_client()
+ extra = {}
+ if content_type:
+ extra['ContentType'] = content_type
+ if metadata:
+ extra['Metadata'] = metadata
+ if extra:
+ client.upload_fileobj(fileobj, self.bucket, key, ExtraArgs=extra)
+ else:
+ client.upload_fileobj(fileobj, self.bucket, key)
+ stat = self.stat(StorageLocator(scheme='s3', raw=self.build_locator(key), bucket=self.bucket, key=key))
+ return StoredObject(locator=self.build_locator(key), key=key, size=stat.size, content_type=stat.content_type, etag=stat.etag)
+
+ def upload_local_file(self, local_path: str, key: str, content_type: Optional[str] = None, metadata: Optional[dict] = None, delete_source: bool = False) -> StoredObject:
+ client = self._get_client()
+ extra = {}
+ if content_type:
+ extra['ContentType'] = content_type
+ if metadata:
+ extra['Metadata'] = metadata
+ if extra:
+ client.upload_file(local_path, self.bucket, key, ExtraArgs=extra)
+ else:
+ client.upload_file(local_path, self.bucket, key)
+ if delete_source:
+ try:
+ os.remove(local_path)
+ except FileNotFoundError:
+ pass
+ stat = self.stat(StorageLocator(scheme='s3', raw=self.build_locator(key), bucket=self.bucket, key=key))
+ return StoredObject(locator=self.build_locator(key), key=key, size=stat.size, content_type=stat.content_type, etag=stat.etag)
+
+ def exists(self, locator: StorageLocator) -> bool:
+ from botocore.exceptions import ClientError
+
+ try:
+ self.stat(locator)
+ return True
+ except ClientError as exc:
+ response = getattr(exc, 'response', {}) or {}
+ status_code = (response.get('ResponseMetadata') or {}).get('HTTPStatusCode')
+ error_code = str((response.get('Error') or {}).get('Code') or '')
+ if status_code == 404 or error_code in ('404', 'NoSuchKey', 'NotFound'):
+ return False
+ raise
+
+ def delete(self, locator: StorageLocator, missing_ok: bool = True) -> bool:
+ client = self._get_client()
+ bucket, key = self._bucket_key(locator)
+ client.delete_object(Bucket=bucket, Key=key)
+ return True
+
+ def stat(self, locator: StorageLocator) -> ObjectStat:
+ client = self._get_client()
+ bucket, key = self._bucket_key(locator)
+ data = client.head_object(Bucket=bucket, Key=key)
+ return ObjectStat(
+ size=data.get('ContentLength'),
+ last_modified=data.get('LastModified'),
+ etag=(data.get('ETag') or '').strip('"') or None,
+ content_type=data.get('ContentType'),
+ )
+
+ def materialize(self, locator: StorageLocator) -> MaterializedFile:
+ client = self._get_client()
+ bucket, key = self._bucket_key(locator)
+ suffix = Path(key).suffix
+ fd, tmp_path = tempfile.mkstemp(prefix='speakr_s3_', suffix=suffix)
+ os.close(fd)
+ client.download_file(bucket, key, tmp_path)
+ return MaterializedFile(local_path=tmp_path, cleanup_required=True)
+
+ def presign_get_url(self, locator: StorageLocator, expires_seconds: int, response_content_type: Optional[str] = None,
+ response_content_disposition: Optional[str] = None) -> str:
+ client = self._get_client()
+ bucket, key = self._bucket_key(locator)
+ params = {'Bucket': bucket, 'Key': key}
+ if response_content_type:
+ params['ResponseContentType'] = response_content_type
+ if response_content_disposition:
+ params['ResponseContentDisposition'] = response_content_disposition
+ return client.generate_presigned_url(
+ 'get_object',
+ Params=params,
+ ExpiresIn=int(expires_seconds),
+ )
diff --git a/src/services/storage/service.py b/src/services/storage/service.py
new file mode 100644
index 00000000..f49243e1
--- /dev/null
+++ b/src/services/storage/service.py
@@ -0,0 +1,164 @@
+"""Storage service facade supporting local, S3, and temporary legacy path compatibility."""
+
+from __future__ import annotations
+
+import os
+from contextlib import contextmanager
+from dataclasses import dataclass
+from datetime import datetime
+from pathlib import Path
+from typing import Iterator, Optional
+from uuid import uuid4
+import threading
+
+from .factory import StorageSettings, build_local_backend, build_s3_backend, load_storage_settings_from_env
+from .interfaces import AudioDeliveryResult, MaterializedFile, StorageLocator, StoredObject
+from .locator import build_local_locator, parse_locator, relative_key_from_local_path
+
+
+@dataclass
+class _ResolvedBackend:
+ kind: str
+ backend: object
+ locator: StorageLocator
+
+
+class StorageService:
+ """Facade to hide storage backend details from business logic."""
+
+ def __init__(self, settings: Optional[StorageSettings] = None):
+ self.settings = settings or load_storage_settings_from_env()
+ self.local = build_local_backend(self.settings)
+ self.s3 = build_s3_backend(self.settings)
+
+ def _parse(self, locator_value: str) -> StorageLocator:
+ locator = parse_locator(locator_value)
+ if not locator:
+ raise ValueError('Empty storage locator')
+ return locator
+
+ def parse_locator(self, locator_value: str) -> Optional[StorageLocator]:
+ return parse_locator(locator_value)
+
+ def _resolve_backend_for_locator(self, locator_value: str) -> _ResolvedBackend:
+ locator = self._parse(locator_value)
+ if locator.scheme == 's3':
+ if not self.s3:
+ raise RuntimeError('S3 locator encountered but S3 backend is not configured')
+ return _ResolvedBackend(kind='s3', backend=self.s3, locator=locator)
+ return _ResolvedBackend(kind='local', backend=self.local, locator=locator)
+
+ def default_backend_kind(self) -> str:
+ return self.settings.backend
+
+ def get_staging_dir(self) -> str:
+ Path(self.settings.staging_dir).mkdir(parents=True, exist_ok=True)
+ return self.settings.staging_dir
+
+ def build_recording_key(self, original_filename: Optional[str], recording_id: Optional[int] = None, *, now: Optional[datetime] = None) -> str:
+ now = now or datetime.utcnow()
+ base_name = os.path.basename(original_filename or 'recording.bin').replace('\\', '_').replace('/', '_')
+ safe_name = ''.join(c for c in base_name if c.isalnum() or c in (' ', '.', '-', '_')).strip() or 'recording.bin'
+ safe_name = safe_name.replace(' ', '_')
+ ts = now.strftime('%Y%m%d%H%M%S')
+ rec_part = str(recording_id) if recording_id is not None else f"tmp-{uuid4().hex[:8]}"
+ prefix = (self.settings.key_prefix or 'recordings').strip('/').replace('\\', '/')
+ return f"{prefix}/{now.strftime('%Y/%m')}/{rec_part}/{ts}_{safe_name}"
+
+ def build_default_locator(self, key: str) -> str:
+ if self.settings.backend == 's3':
+ if not self.s3:
+ raise RuntimeError('FILE_STORAGE_BACKEND=s3 but S3 backend is not configured')
+ return self.s3.build_locator(key)
+ return self.local.build_locator(key)
+
+ def build_local_locator_from_path(self, abs_path: str) -> str:
+ key = relative_key_from_local_path(abs_path, self.settings.local_root)
+ return build_local_locator(key)
+
+ def maybe_normalize_local_legacy_locator(self, locator_value: Optional[str]) -> Optional[str]:
+ if not locator_value:
+ return locator_value
+ locator = parse_locator(locator_value)
+ if not locator:
+ return locator_value
+ if locator.scheme == 'legacy_local_abs':
+ return self.build_local_locator_from_path(locator.path)
+ if locator.scheme == 'legacy_local_rel':
+ return build_local_locator(locator.key or '')
+ return locator_value
+
+ def resolve_local_filesystem_path(self, locator_value: str) -> str:
+ resolved = self._resolve_backend_for_locator(locator_value)
+ if resolved.kind != 'local':
+ raise ValueError('Locator is not local')
+ return self.local.resolve_path(resolved.locator)
+
+ def exists(self, locator_value: str) -> bool:
+ resolved = self._resolve_backend_for_locator(locator_value)
+ return resolved.backend.exists(resolved.locator)
+
+ def delete(self, locator_value: Optional[str], missing_ok: bool = True) -> bool:
+ if not locator_value:
+ return bool(missing_ok)
+ resolved = self._resolve_backend_for_locator(locator_value)
+ return resolved.backend.delete(resolved.locator, missing_ok=missing_ok)
+
+ def upload_local_file(self, local_path: str, key: str, *, content_type: Optional[str] = None, delete_source: bool = False) -> StoredObject:
+ if self.settings.backend == 's3':
+ if not self.s3:
+ raise RuntimeError('FILE_STORAGE_BACKEND=s3 but S3 backend is not configured')
+ return self.s3.upload_local_file(local_path, key, content_type=content_type, delete_source=delete_source)
+ return self.local.upload_local_file(local_path, key, content_type=content_type, delete_source=delete_source)
+
+ @contextmanager
+ def materialize(self, locator_value: str) -> Iterator[MaterializedFile]:
+ resolved = self._resolve_backend_for_locator(locator_value)
+ materialized = resolved.backend.materialize(resolved.locator)
+ try:
+ yield materialized
+ finally:
+ if materialized.cleanup_required:
+ try:
+ os.remove(materialized.local_path)
+ except FileNotFoundError:
+ pass
+
+ def get_audio_delivery(self, locator_value: str, *, download: bool = False, mime_type: Optional[str] = None,
+ download_name: Optional[str] = None, is_public: bool = False) -> AudioDeliveryResult:
+ resolved = self._resolve_backend_for_locator(locator_value)
+ if resolved.kind == 'local':
+ local_path = self.local.resolve_path(resolved.locator)
+ return AudioDeliveryResult(mode='local_file', local_path=local_path, mimetype=mime_type)
+
+ ttl = self.settings.presign_public_ttl_seconds if is_public else self.settings.presign_ttl_seconds
+ disposition = None
+ if download:
+ safe_name = (download_name or 'recording').replace('"', '')
+ disposition = f'attachment; filename="{safe_name}"'
+ url = self.s3.presign_get_url(
+ resolved.locator,
+ expires_seconds=ttl,
+ response_content_type=mime_type,
+ response_content_disposition=disposition,
+ )
+ return AudioDeliveryResult(mode='redirect_url', url=url, mimetype=mime_type)
+
+
+_storage_service_singleton: Optional[StorageService] = None
+_storage_service_singleton_lock = threading.Lock()
+
+
+def get_storage_service() -> StorageService:
+ global _storage_service_singleton
+ if _storage_service_singleton is None:
+ with _storage_service_singleton_lock:
+ if _storage_service_singleton is None:
+ _storage_service_singleton = StorageService()
+ return _storage_service_singleton
+
+
+def reset_storage_service_singleton() -> None:
+ global _storage_service_singleton
+ with _storage_service_singleton_lock:
+ _storage_service_singleton = None
diff --git a/src/tasks/processing.py b/src/tasks/processing.py
index 3f007c48..22401513 100644
--- a/src/tasks/processing.py
+++ b/src/tasks/processing.py
@@ -1681,11 +1681,12 @@ def transcribe_with_connector(app_context, recording_id, filepath, original_file
current_app.logger.info(f"Video passthrough: sending original video to ASR (no audio extraction)")
actual_filepath = filepath # Send video as-is to connector
if effective_video_retention:
- # Also keep the video for playback. Derive the MIME from
- # the file's actual container (we're in the is_video
- # branch, so has_video=True) rather than guessing.
+ # Also keep the video for playback, but preserve the existing
+ # persistent storage locator: recording.audio_path is a local://
+ # or s3:// locator set at upload time, not a raw temp path, so
+ # don't overwrite it. Derive the MIME from the file's actual
+ # container (we're in the is_video branch, so has_video=True).
from src.utils.mime import resolve_media_mime
- recording.audio_path = filepath
recording.mime_type = resolve_media_mime(filepath, has_video=True)
db.session.commit()
elif effective_video_retention:
@@ -1698,14 +1699,19 @@ def transcribe_with_connector(app_context, recording_id, filepath, original_file
actual_content_type = audio_mime_type
actual_filename = os.path.basename(audio_filepath)
- # Keep original video file as the stored media. Derive
- # the MIME from the actual container (is_video branch →
- # has_video=True) instead of guessing from extension.
+ # Keep the original video as the stored media: recording.audio_path
+ # is a storage locator set at upload time, so don't overwrite it.
+ # Derive the MIME from the actual container (is_video → has_video=True).
from src.utils.mime import resolve_media_mime
- recording.audio_path = filepath
recording.mime_type = resolve_media_mime(filepath, has_video=True)
+ try:
+ extracted_duration = chunking_service.get_audio_duration(audio_filepath) if chunking_service else None
+ if extracted_duration and extracted_duration > 0:
+ recording.audio_duration_seconds = float(extracted_duration)
+ except Exception as duration_err:
+ current_app.logger.warning(f"Could not determine extracted audio duration for recording {recording.id}: {duration_err}")
db.session.commit()
- current_app.logger.info(f"Video retained at: {filepath}, temp audio extracted: {audio_filepath}")
+ current_app.logger.info(f"Video retained (media path unchanged), temp audio extracted: {audio_filepath}")
except Exception as e:
current_app.logger.error(f"Failed to extract audio from video: {str(e)}")
recording.status = 'FAILED'
@@ -1713,7 +1719,12 @@ def transcribe_with_connector(app_context, recording_id, filepath, original_file
db.session.commit()
raise
else:
- # Default: extract audio, delete original video
+ # Legacy/repair fallback path: in the normal upload flow, video containers should
+ # already be converted/extracted before the Recording is stored. We still handle
+ # video here for historical records / reprocess jobs where a video object may
+ # remain in recording.audio_path.
+ # TODO(storage-cleanup): after migrating legacy video-backed recordings to audio
+ # objects and normalizing cached durations, this branch should be removable.
current_app.logger.info(f"Video container detected, extracting audio...")
try:
audio_filepath, audio_mime_type = extract_audio_from_video(filepath)
@@ -1721,9 +1732,43 @@ def transcribe_with_connector(app_context, recording_id, filepath, original_file
actual_content_type = audio_mime_type
actual_filename = os.path.basename(audio_filepath)
- recording.audio_path = audio_filepath
- recording.mime_type = audio_mime_type
- db.session.commit()
+ try:
+ from src.services.storage import get_storage_service
+ storage = get_storage_service()
+ old_audio_locator = recording.audio_path
+ extracted_key = storage.build_recording_key(actual_filename, recording.id)
+ stored_audio = storage.upload_local_file(
+ audio_filepath,
+ extracted_key,
+ content_type=audio_mime_type,
+ delete_source=False,
+ )
+ recording.audio_path = stored_audio.locator
+ recording.mime_type = audio_mime_type
+ try:
+ extracted_duration = chunking_service.get_audio_duration(audio_filepath) if chunking_service else None
+ if extracted_duration and extracted_duration > 0:
+ recording.audio_duration_seconds = float(extracted_duration)
+ except Exception as duration_err:
+ current_app.logger.warning(f"Could not determine extracted audio duration for recording {recording.id}: {duration_err}")
+ db.session.commit()
+ current_app.logger.info(f"Audio extracted and stored: {stored_audio.locator}")
+
+ # Best-effort cleanup of previous stored source (e.g. original video)
+ if old_audio_locator and old_audio_locator != stored_audio.locator:
+ try:
+ storage.delete(old_audio_locator, missing_ok=True)
+ except Exception as cleanup_err:
+ current_app.logger.warning(f"Failed to delete original source after audio extraction: {cleanup_err}")
+ except Exception:
+ # Do NOT persist temporary extracted path in DB. Keep previous locator and continue
+ # processing with the local extracted file for this job only.
+ recording.mime_type = audio_mime_type
+ db.session.commit()
+ current_app.logger.warning(
+ "Failed to persist extracted audio via storage layer; keeping previous recording.audio_path and continuing with temporary local file",
+ exc_info=True,
+ )
current_app.logger.info(f"Audio extracted: {audio_filepath}")
except Exception as e:
current_app.logger.error(f"Failed to extract audio from video: {str(e)}")
@@ -1944,15 +1989,50 @@ def transcribe_with_connector(app_context, recording_id, filepath, original_file
# Calculate and save transcription duration
transcription_end_time = time.time()
recording.transcription_duration_seconds = int(transcription_end_time - transcription_start_time)
+
+ # Persist/correct audio duration cache when we have local access during processing.
+ # Prefer the original job input path for billing consistency, but fall back to the
+ # actual processed file (e.g., extracted audio from video) when needed.
+ try:
+ cached_audio_duration = None
+ if chunking_service:
+ # Legacy cache-repair fallback: new uploads should already populate
+ # recording.audio_duration_seconds, and extracted-video paths above also try to do
+ # that. This probe exists to repair historical rows and edge cases during
+ # processing/reprocessing while we still have local access to media.
+ # TODO(storage-cleanup): after running backfill/migrations for missing
+ # audio_duration_seconds (and cleaning legacy video-backed objects), remove this
+ # multi-path probing block and rely on DB-stored duration.
+ duration_probe_candidates = []
+ if filepath:
+ duration_probe_candidates.append(filepath)
+ if actual_filepath and actual_filepath not in duration_probe_candidates:
+ duration_probe_candidates.append(actual_filepath)
+ for candidate_path in duration_probe_candidates:
+ cached_audio_duration = chunking_service.get_audio_duration(candidate_path)
+ if cached_audio_duration and cached_audio_duration > 0:
+ recording.audio_duration_seconds = float(cached_audio_duration)
+ break
+ except Exception as duration_err:
+ current_app.logger.warning(f"Failed to update cached audio duration for recording {recording_id}: {duration_err}")
+
db.session.commit()
current_app.logger.info(f"Transcription completed in {recording.transcription_duration_seconds}s")
# Record transcription usage for billing/budgeting
try:
# Get actual audio duration (not processing time)
- audio_duration = None
- if chunking_service:
- audio_duration = chunking_service.get_audio_duration(recording.audio_path)
+ audio_duration = recording.audio_duration_seconds
+ if (audio_duration is None or audio_duration <= 0) and chunking_service:
+ duration_probe_candidates = []
+ if actual_filepath:
+ duration_probe_candidates.append(actual_filepath)
+ if filepath and filepath not in duration_probe_candidates:
+ duration_probe_candidates.append(filepath)
+ for candidate_path in duration_probe_candidates:
+ audio_duration = chunking_service.get_audio_duration(candidate_path)
+ if audio_duration and audio_duration > 0:
+ break
if audio_duration and audio_duration > 0:
# Get model name from connector if available
diff --git a/tests/test_postgres_migrations.py b/tests/test_postgres_migrations.py
index 7ea39d27..f7b36ab6 100644
--- a/tests/test_postgres_migrations.py
+++ b/tests/test_postgres_migrations.py
@@ -118,7 +118,7 @@ class TestDatabaseMigrations(unittest.TestCase):
columns = self._get_column_names('recording')
expected = [
'id', 'is_inbox', 'is_highlighted', 'mime_type',
- 'completed_at', 'processing_time_seconds', 'error_message',
+ 'audio_duration_seconds', 'completed_at', 'processing_time_seconds', 'error_message',
'folder_id', 'audio_deleted_at', 'deletion_exempt',
'speaker_embeddings',
]
diff --git a/tests/test_video_passthrough.py b/tests/test_video_passthrough.py
index d28481ba..6f5fcf02 100644
--- a/tests/test_video_passthrough.py
+++ b/tests/test_video_passthrough.py
@@ -159,15 +159,17 @@ class TestProcessingMainPath(unittest.TestCase):
passthrough_block = video_block[pt_start:pt_end]
self.assertIn('actual_filepath = filepath', passthrough_block)
- def test_passthrough_with_retention_sets_recording_path(self):
- """When both passthrough and retention are on, recording.audio_path is set."""
+ def test_passthrough_with_retention_preserves_locator(self):
+ """When both passthrough and retention are on, persistent media locator is preserved."""
video_block = PROCESSING_MAIN[PROCESSING_MAIN.find('if is_video:'):]
pt_start = video_block.find('if VIDEO_PASSTHROUGH_ASR:')
pt_end = video_block.find('elif effective_video_retention:')
passthrough_block = video_block[pt_start:pt_end]
self.assertIn('if effective_video_retention:', passthrough_block,
"Passthrough branch should conditionally handle retention")
- self.assertIn('recording.audio_path = filepath', passthrough_block)
+ # Storage model: the persistent locator set at upload time is preserved,
+ # so the passthrough branch must NOT overwrite recording.audio_path.
+ self.assertNotIn('recording.audio_path = filepath', passthrough_block)
# mime_type is derived from the actual container via the shared
# probe-driven resolver (corrects audio/webm-style mislabels).
self.assertIn('recording.mime_type = resolve_media_mime(filepath, has_video=True)', passthrough_block)
diff --git a/tests/test_video_retention.py b/tests/test_video_retention.py
index e597519f..7f9b33e0 100644
--- a/tests/test_video_retention.py
+++ b/tests/test_video_retention.py
@@ -65,10 +65,10 @@ class TestProcessingPipelineVideoRetention(unittest.TestCase):
with open(os.path.join(PROJECT_ROOT, 'src/tasks/processing.py'), 'r') as f:
cls.content = f.read()
- def test_video_retention_true_keeps_original(self):
- """When VIDEO_RETENTION=True, recording.audio_path is set to original filepath."""
- # The VIDEO_RETENTION=True branch should set recording.audio_path = filepath
- self.assertIn('recording.audio_path = filepath', self.content)
+ def test_video_retention_true_preserves_stored_media_reference(self):
+ """When VIDEO_RETENTION=True, the persistent media reference is left untouched."""
+ self.assertIn('Video retained (media path unchanged)', self.content)
+ self.assertNotIn('recording.audio_path = filepath', self.content)
def test_video_retention_true_extracts_without_cleanup(self):
"""When VIDEO_RETENTION=True, extract_audio_from_video is called with cleanup_original=False."""
@@ -112,10 +112,10 @@ class TestProcessingPipelineVideoRetention(unittest.TestCase):
self.assertNotIn("mimetypes.guess_type(filepath)[0] or 'video/mp4'", self.content)
def test_duration_uses_recording_audio_path(self):
- """Duration lookup uses recording.audio_path (always valid), not filepath."""
- self.assertIn('chunking_service.get_audio_duration(recording.audio_path)', self.content)
- # Should NOT use bare filepath for duration (pre-existing bug was fixed)
- self.assertNotIn('chunking_service.get_audio_duration(filepath)', self.content)
+ """Duration logic prefers cached DB values and local probe candidates."""
+ self.assertIn('recording.audio_duration_seconds = float(cached_audio_duration)', self.content)
+ self.assertIn('duration_probe_candidates = []', self.content)
+ self.assertIn('chunking_service.get_audio_duration(candidate_path)', self.content)
class TestUploadHandlerVideoRetention(unittest.TestCase):
@@ -259,18 +259,20 @@ class TestSendFileConditional(unittest.TestCase):
def test_recordings_streaming_has_conditional(self):
"""Streaming send_file in recordings.py has conditional=True."""
content = self._read_file('src/api/recordings.py')
- # Find the non-download send_file call
- self.assertIn('send_file(recording.audio_path, mimetype=recording.mime_type, conditional=True)', content)
+ self.assertIn('return send_file(delivery.local_path, mimetype=recording.mime_type, conditional=True)', content)
def test_recordings_download_has_conditional(self):
"""Download send_file in recordings.py has conditional=True."""
content = self._read_file('src/api/recordings.py')
- self.assertIn('as_attachment=True, download_name=filename, mimetype=recording.mime_type, conditional=True', content)
+ self.assertIn('as_attachment=True,', content)
+ self.assertIn('download_name=download_name,', content)
+ self.assertIn('mimetype=recording.mime_type,', content)
+ self.assertIn('conditional=True,', content)
def test_shares_has_conditional(self):
"""send_file in shares.py has conditional=True."""
content = self._read_file('src/api/shares.py')
- self.assertIn('send_file(recording.audio_path, conditional=True)', content)
+ self.assertIn('send_file(delivery.local_path, mimetype=(recording.mime_type or delivery.mimetype), conditional=True)', content)
class TestFrontendTemplates(unittest.TestCase):