Skip to content

Speakr is a personal, self-hosted web application designed for transcribing audio recordings

Python

feat: upload files larger than the proxy body limit in slices

François Catuhebc03e9996ca5
diff --git a/docs/admin-guide/recording-sessions.md b/docs/admin-guide/recording-sessions.md
index 6b252184..e29fd314 100644
--- a/docs/admin-guide/recording-sessions.md
+++ b/docs/admin-guide/recording-sessions.md
@@ -25,6 +25,45 @@ so the legacy 200 MB cap and its warning still apply to them. If a recording
did stream to the server and the user switches it to incognito in the review
pane afterwards, the server-side chunks are deleted before processing.
+## Sliced uploads of files on disk
+
+The same session machinery also carries uploads of files the user already
+has on disk. A normal upload is one multipart `POST /upload` of the whole
+file, so a reverse proxy whose body limit is below the file size rejects
+it before Speakr sees it (Cloudflare's proxy caps request bodies at
+100 MB on most plans). Any file larger than one slice is instead sent as
+fixed-size byte slices through `POST /upload/session/{id}/chunks/{N}` and
+finalized with `POST /upload/session/{id}/finalize-upload`, which
+byte-joins the slices and runs the result through the same ingestion
+pipeline as a single-shot upload.
+
+This needs no configuration and is not covered by
+`ENABLE_SERVER_RECORDING_CHUNKS`, which governs the recorder only. The
+slice size is `RECORDING_SESSION_MAX_CHUNK_BYTES` (16 MB), advertised in
+the create response, and files at or below that size still go up in one
+request.
+
+A few differences from a recorder session are worth knowing when reading
+logs:
+
+- Its `recording_session` row has `upload_filename` set. That is what
+ distinguishes the two, and each finalize route rejects the other's
+ sessions with a 409.
+- The slices are byte ranges of one complete container, not
+ MediaRecorder timeslices, so finalize does a plain byte-join with no
+ ffmpeg concat, no remux, and no container sniffing. The stored file is
+ byte-identical to what the user picked.
+- `finalize-upload` is synchronous: hashing, probing and any conversion
+ happen in-request exactly as they do for `POST /upload`, so it holds
+ the connection for as long as that takes and needs the same generous
+ proxy read timeout.
+- An abandoned one is expired by the cleanup sweep rather than
+ auto-finalized. The user still has the original file, and its slices
+ are a truncated container rather than a playable partial recording.
+- Losing the finalize response loses the upload: unlike the recorder's
+ `/finalize`, it is not replayable. The client re-uploads, and
+ duplicate detection flags the second copy if the first landed.
+
## Configuration
Environment variables, all optional:
@@ -50,7 +89,8 @@ so neither is killed in flight.
```nginx
location /upload/session/ {
- # Chunk POSTs are small; keep timeouts modest.
+ # Chunk POSTs are capped at RECORDING_SESSION_MAX_CHUNK_BYTES (16 MB),
+ # whether they carry a recorder timeslice or a file slice.
proxy_pass http://speakr_upstream;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
@@ -60,9 +100,10 @@ location /upload/session/ {
proxy_read_timeout 30s;
}
-location ~* ^/upload/session/.+/finalize$ {
- # Finalize triggers ffmpeg concat which can take tens of seconds
- # for long recordings. Give it room.
+location ~* ^/upload/session/.+/finalize(-upload)?$ {
+ # Finalize triggers ffmpeg concat which can take tens of seconds for
+ # long recordings, and finalize-upload also hashes, probes and may
+ # convert the file in-request. Give it room.
proxy_pass http://speakr_upstream;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
@@ -80,7 +121,7 @@ the larger timeout; finalize will get the headroom it needs.
### Caddy
```
-@finalize path_regexp ^/upload/session/[^/]+/finalize$
+@finalize path_regexp ^/upload/session/[^/]+/finalize(-upload)?$
handle @finalize {
reverse_proxy speakr:8899 {
transport http {
@@ -103,6 +144,7 @@ LimitRequestBody 33554432
UPLOAD_FOLDER/_sessions/
<uuid-1>/
session.json
+ joined.bin ← sliced uploads only, during finalize
chunk-000001.bin
chunk-000002.bin
...
@@ -153,10 +195,11 @@ on the server, or discard them.
| Method | Path | Purpose |
|---|---|---|
-| POST | `/upload/session` | Create a new session. Body: `{mime_type}`. |
+| POST | `/upload/session` | Create a new session. Body: `{mime_type}`, plus `{filename}` for a sliced file upload. |
| POST | `/upload/session/{id}/chunks/{N}` | Append chunk N (must be `chunk_count + 1`). Body is raw bytes. |
| GET | `/upload/session/{id}` | Status of an existing session. |
| POST | `/upload/session/{id}/finalize` | Request asynchronous stitch + transcribe kickoff. |
+| POST | `/upload/session/{id}/finalize-upload` | Reassemble a sliced file upload and ingest it. Body is the `POST /upload` form without the file. |
| DELETE | `/upload/session/{id}` | Abort and remove the on-disk chunks. |
All endpoints require an authenticated session. The CSRF token from the
diff --git a/docs/features.md b/docs/features.md
index f53d3265..9afc406d 100644
--- a/docs/features.md
+++ b/docs/features.md
@@ -20,6 +20,10 @@ A per-recording Stats tab shows total length, speaker count, conversation turns,
An opt-in mode (`ENABLE_SERVER_RECORDING_CHUNKS`) streams in-app recording chunks to the server as you record instead of buffering them in browser memory, raising the practical recording ceiling to hours-long sessions and letting you resume a recording after a page reload. See the [Recording guide](user-guide/recording.md#server-side-recording-sessions-long-recordings) and [admin setup](admin-guide/recording-sessions.md).
+### Sliced File Uploads
+
+Uploads of files larger than 16 MB are sent as fixed-size slices through the same session endpoints and reassembled server-side, so a reverse proxy with a request-body limit below the file size (Cloudflare caps bodies at 100 MB on most plans) no longer blocks large uploads. This needs no configuration; see [Recording Sessions](admin-guide/recording-sessions.md#sliced-uploads-of-files-on-disk).
+
## Core Transcription Features
### Multi-Engine Support
diff --git a/src/api/recording_sessions.py b/src/api/recording_sessions.py
index 0c6f6326..13de1a91 100644
--- a/src/api/recording_sessions.py
+++ b/src/api/recording_sessions.py
@@ -13,11 +13,22 @@ Endpoints
POST /upload/session/<sid>/chunks/<n> append chunk (n=1..)
GET /upload/session/<sid> status
POST /upload/session/<sid>/finalize request stitch
+ POST /upload/session/<sid>/finalize-upload reassemble a sliced file
DELETE /upload/session/<sid> abort + cleanup
All endpoints require login. They write the on-disk layout described in
``src/models/recording_session.py``.
+Creating a session with a ``filename`` makes it a sliced upload of a file
+the user already has on disk (used to cross a reverse proxy whose body
+limit is below the file size) rather than a recorder session. Those
+sessions share create / chunk / abort with the recorder but finalize
+through ``/finalize-upload``, which byte-joins the slices and hands the
+result to the same ingestion pipeline ``POST /upload`` uses. The declared
+mime type is not checked against ``RECORDING_SESSION_ALLOWED_MIME_TYPES``
+for them: the user picks the container, and ffprobe gates it at ingest.
+The two finalize routes reject each other's sessions.
+
Finalize is async: the endpoint creates a placeholder Recording row in
``STITCHING`` status, enqueues a ``stitch`` job via the existing
``job_queue``, and returns the recording id immediately. The worker
@@ -49,6 +60,7 @@ from flask_login import login_required, current_user
from src.database import db
from src.models import RecordingSession, Recording, RECORDING_SESSION_STATUSES
from src.services.job_queue import job_queue
+from src.services.recording_stitch import byte_join, session_chunk_paths
recording_sessions_bp = Blueprint('recording_sessions', __name__)
@@ -263,17 +275,27 @@ def _replay_finalize_response(session):
def create_session():
"""Create a new in-progress recording session.
- Body (JSON, all optional): {"mime_type": "audio/webm"}.
+ Body (JSON, all optional): ``{"mime_type": "audio/webm"}`` for a
+ recorder session, plus ``{"filename": "talk.mp4"}`` to make it a
+ sliced upload of a file the user already has on disk.
Returns ``{session_id, expires_at, max_chunk_bytes}`` on success.
"""
data = request.get_json(silent=True) or {}
- mime_type = (data.get('mime_type') or 'audio/webm').strip().lower()
- if mime_type not in _allowed_mime_types():
- return jsonify({
- 'error': f'Unsupported mime_type {mime_type!r}',
- 'allowed': sorted(_allowed_mime_types()),
- }), 400
+ declared_mime = (data.get('mime_type') or '').strip().lower()
+ upload_filename = (data.get('filename') or '').strip()
+
+ if upload_filename:
+ if len(upload_filename) > 255:
+ return jsonify({'error': 'filename must be 255 characters or fewer'}), 400
+ mime_type = declared_mime or 'application/octet-stream'
+ else:
+ mime_type = declared_mime or 'audio/webm'
+ if mime_type not in _allowed_mime_types():
+ return jsonify({
+ 'error': f'Unsupported mime_type {mime_type!r}',
+ 'allowed': sorted(_allowed_mime_types()),
+ }), 400
# Per-user quota check before we let them open another session.
if _user_bytes_in_progress(current_user.id) >= _max_bytes_per_user():
@@ -285,6 +307,7 @@ def create_session():
session = RecordingSession(
user_id=current_user.id,
mime_type=mime_type,
+ upload_filename=upload_filename or None,
status='recording',
)
db.session.add(session)
@@ -304,6 +327,7 @@ def create_session():
return jsonify({
'session_id': session.id,
'mime_type': session.mime_type,
+ 'upload_filename': session.upload_filename,
'status': session.status,
'expires_at': expires_at.isoformat(),
'max_chunk_bytes': _max_chunk_bytes(),
@@ -438,6 +462,11 @@ def finalize_session(session_id):
if err is not None:
return err
+ if session.upload_filename:
+ return jsonify({
+ 'error': 'Session is a sliced file upload; finalize it with /finalize-upload',
+ }), 409
+
if session.status not in ('recording', 'finalizing'):
return jsonify({
'error': f'Session is in status {session.status!r}; cannot finalize',
@@ -523,6 +552,127 @@ def finalize_session(session_id):
}), 202
+class ReassembledUpload:
+ """A Werkzeug ``FileStorage``-shaped view of reassembled slices.
+
+ Exposes only what :func:`ingest_uploaded_recording` asks of an upload
+ (``filename``, ``seek``, ``tell``, ``save``) so a sliced upload runs
+ through the same ingestion pipeline as a single-shot one. ``save``
+ moves the joined file rather than copying it, so a large upload is
+ written once.
+ """
+
+ def __init__(self, path, filename):
+ self._path = path
+ self.filename = filename
+ self._size = os.path.getsize(path)
+ self._pos = 0
+
+ def seek(self, offset, whence=os.SEEK_SET):
+ if whence == os.SEEK_END:
+ self._pos = self._size + offset
+ elif whence == os.SEEK_CUR:
+ self._pos += offset
+ else:
+ self._pos = offset
+ return self._pos
+
+ def tell(self):
+ return self._pos
+
+ def save(self, dst):
+ shutil.move(self._path, dst)
+
+
+@recording_sessions_bp.route('/upload/session/<string:session_id>/finalize-upload', methods=['POST'])
+@login_required
+def finalize_sliced_upload(session_id):
+ """Reassemble a sliced file upload and ingest it.
+
+ The body is the same multipart form ``POST /upload`` takes, minus the
+ file part: the bytes come from the session's slices instead. The
+ response is that endpoint's response, so a client can treat the two
+ transports interchangeably.
+
+ Unlike the recorder's ``/finalize``, this is synchronous and not
+ replayable: ingestion (hash, probe, convert) happens in-request
+ exactly as it does for a single-shot upload. A client that loses the
+ response has to re-upload, and Speakr's duplicate detection flags the
+ second copy. Concurrent calls are claimed atomically so only one of
+ them ingests.
+ """
+ session = db.session.get(RecordingSession, session_id)
+ err = _ensure_owned(session)
+ if err is not None:
+ return err
+
+ if not session.upload_filename:
+ return jsonify({
+ 'error': 'Session is a recorder session; finalize it with /finalize',
+ }), 409
+
+ if session.status != 'recording':
+ return jsonify({
+ 'error': f'Session is in status {session.status!r}; cannot finalize',
+ }), 409
+
+ dir_path = _session_dir(session.id)
+ slice_paths = session_chunk_paths(dir_path)
+ if not slice_paths or len(slice_paths) != session.chunk_count:
+ return jsonify({
+ 'error': 'Slices on disk do not match the session',
+ 'expected_chunk_index': len(slice_paths) + 1,
+ 'chunk_count': session.chunk_count,
+ }), 409
+
+ claimed = db.session.query(RecordingSession).filter(
+ RecordingSession.id == session.id,
+ RecordingSession.status == 'recording',
+ ).update({'status': 'finalizing'}, synchronize_session=False)
+ if not claimed:
+ db.session.rollback()
+ return jsonify({'error': 'Session is already being finalized'}), 409
+ session.last_seen_at = datetime.utcnow()
+ db.session.commit()
+
+ joined_path = os.path.join(dir_path, 'joined.bin')
+ try:
+ byte_join(slice_paths, joined_path)
+ except OSError as e:
+ current_app.logger.error(f"Could not join slices for session {session_id}: {e}")
+ _fail_sliced_upload(session, f'Could not reassemble slices: {e}')
+ return jsonify({'error': 'Server could not reassemble the uploaded slices'}), 500
+
+ from src.api.recordings import ingest_uploaded_recording
+ result = ingest_uploaded_recording(
+ owner=current_user,
+ uploaded_file=ReassembledUpload(joined_path, session.upload_filename),
+ form=request.form,
+ original_filename=session.upload_filename,
+ )
+
+ response, status = result if isinstance(result, tuple) else (result, 200)
+ if 200 <= status < 300:
+ body = response.get_json(silent=True) or {}
+ session.status = 'finalized'
+ session.finalized_at = datetime.utcnow()
+ session.finalized_recording_id = body.get('id')
+ session.last_seen_at = datetime.utcnow()
+ db.session.commit()
+ else:
+ _fail_sliced_upload(session, f'Ingestion rejected the upload (HTTP {status})')
+
+ _remove_session_dir(session_id)
+ return result
+
+
+def _fail_sliced_upload(session, message):
+ session.status = 'failed'
+ session.error_message = message
+ session.last_seen_at = datetime.utcnow()
+ db.session.commit()
+
+
@recording_sessions_bp.route('/upload/session/<string:session_id>', methods=['DELETE'])
@login_required
def abort_session(session_id):
@@ -563,6 +713,9 @@ def cleanup_expired_sessions(app=None):
what was uploaded and land it in the user's library (same path as a
manual finalize), rather than discarding it.
- ``recording`` with no chunks → expire and delete the empty directory.
+ - a sliced file upload (``upload_filename`` set) → expire, whatever it
+ holds: the user still has the original file, and its slices are a
+ truncated container rather than a playable partial recording.
- ``finalizing`` (already has a placeholder recording) but never reached
a terminal stitch state → **re-enqueue** the stitch to unstick it
(enqueue dedupes an already-active job). Bumping ``last_seen_at`` here
@@ -584,7 +737,7 @@ def cleanup_expired_sessions(app=None):
finalized = rekicked = expired = 0
for s in candidates:
try:
- if s.status == 'recording' and (s.chunk_count or 0) > 0:
+ if s.status == 'recording' and (s.chunk_count or 0) > 0 and not s.upload_filename:
when = s.created_at.strftime('%Y-%m-%d %H:%M') if s.created_at else ''
title = f"Recovered recording {when}".strip()
recovered_rec, _enq_err = _finalize_session_into_stitch(
diff --git a/src/init_db.py b/src/init_db.py
index be0c678f..d9676c3f 100644
--- a/src/init_db.py
+++ b/src/init_db.py
@@ -652,6 +652,10 @@ def _run_migrations(app, engine):
app.logger.warning(f"Error during meeting_date migration: {e}")
app.logger.warning("New recordings will work correctly, but existing dates may need manual migration")
+ with _migration_section(app, failures, "sliced file upload sessions"):
+ if add_column_if_not_exists(engine, 'recording_session', 'upload_filename', 'VARCHAR(255)'):
+ app.logger.info("Added upload_filename column to recording_session table")
+
with _migration_section(app, failures, "performance and uniqueness indexes"):
# Add index on TranscriptChunk.speaker_name for performance
# This improves speaker rename operations which update all chunks
diff --git a/src/models/recording_session.py b/src/models/recording_session.py
index 4434faf9..73136d88 100644
--- a/src/models/recording_session.py
+++ b/src/models/recording_session.py
@@ -20,6 +20,16 @@ On finalize, the chunks are stitched into a single file via ffmpeg concat
demux (handles pause/resume across MediaRecorder restarts, which a raw
cat would corrupt). The resulting file becomes the new Recording's
audio_path and the session directory is removed.
+
+The same session machinery also carries sliced uploads of files the user
+already has on disk, so those can cross a reverse proxy with a body-size
+limit below the file size. Such a session has ``upload_filename`` set,
+which is what distinguishes it from a recorder session. Its chunks are
+fixed-size byte slices of one complete container rather than
+MediaRecorder timeslices, so it finalizes through the normal upload
+ingestion path (plain byte-join, no stitch, no remux) and is never
+auto-finalized when abandoned: the user's original file still exists, so
+there is nothing to recover.
"""
import uuid
@@ -63,6 +73,8 @@ class RecordingSession(db.Model):
# the declared type) and pick the right ffmpeg input args at stitch time.
mime_type = db.Column(db.String(100), nullable=False, default='audio/webm')
+ upload_filename = db.Column(db.String(255), nullable=True)
+
# Status transitions are guarded by the API; see RECORDING_SESSION_STATUSES.
status = db.Column(db.String(20), nullable=False, default='recording', index=True)
@@ -112,6 +124,7 @@ class RecordingSession(db.Model):
'session_id': self.id,
'status': self.status,
'mime_type': self.mime_type,
+ 'upload_filename': self.upload_filename,
'chunk_count': self.chunk_count,
'bytes_received': self.bytes_received,
'created_at': self.created_at.isoformat() if self.created_at else None,
diff --git a/static/js/modules/composables/upload.js b/static/js/modules/composables/upload.js
index 581e7dbd..2293eecf 100644
--- a/static/js/modules/composables/upload.js
+++ b/static/js/modules/composables/upload.js
@@ -7,6 +7,7 @@ import * as FailedUploads from '../db/failed-uploads.js';
import * as IncognitoStorage from '../db/incognito-storage.js';
import * as RecordingDB from '../db/recording-persistence.js';
import { getUploadCsrfToken, isCsrfRejection } from '../csrf.js';
+import { shouldSliceUpload, uploadFileInSlices } from '../db/sliced-file-upload.js';
import { computeUploadTimeout } from '../utils/upload-timeout.js';
// Parse error message and return friendly error info
@@ -467,16 +468,23 @@ export function useUpload(state, utils) {
* Upload a single file to the server.
* Acquires a concurrency slot, uploads, then releases.
* Status updates are per-item (no global processingProgress).
+ *
+ * Files past the slice threshold go up through the sliced-upload
+ * transport instead of one multipart POST, so a reverse proxy whose
+ * body limit is below the file size cannot reject them. Both
+ * transports send the same form and return the same response, so
+ * everything below the transport call is shared.
*/
const uploadSingleFile = async (fileItem) => {
await acquireUploadSlot();
fileItem.status = 'uploading';
fileItem.progress = 5;
+ const sliced = shouldSliceUpload(fileItem.file);
try {
const formData = new FormData();
- formData.append('file', fileItem.file);
+ if (!sliced) formData.append('file', fileItem.file);
// Send file's lastModified timestamp for meeting_date
if (fileItem.file.lastModified) {
@@ -650,18 +658,27 @@ export function useUpload(state, utils) {
xhr.send(formData);
});
- // Refresh the token right before sending, and retry exactly once
- // with another fresh token if the server still rejects it —
- // mirroring what the fetch interceptor in csrf-refresh.js does
- // for non-XHR requests.
let data;
- try {
- data = await sendUpload(await getUploadCsrfToken());
- } catch (uploadError) {
- if (!uploadError.isCsrfRejection) throw uploadError;
- console.warn(`[Upload] CSRF rejection for ${fileItem.file.name}; refreshing token and retrying once.`);
- fileItem.progress = 5;
- data = await sendUpload(await getUploadCsrfToken());
+ if (sliced) {
+ data = await uploadFileInSlices(fileItem.file, formData, {
+ onProgress: (fraction) => {
+ fileItem.progress = Math.round(5 + fraction * 85);
+ },
+ onXhr: (xhr) => { fileItem._xhr = xhr; },
+ });
+ } else {
+ // Refresh the token right before sending, and retry exactly once
+ // with another fresh token if the server still rejects it —
+ // mirroring what the fetch interceptor in csrf-refresh.js does
+ // for non-XHR requests.
+ try {
+ data = await sendUpload(await getUploadCsrfToken());
+ } catch (uploadError) {
+ if (!uploadError.isCsrfRejection) throw uploadError;
+ console.warn(`[Upload] CSRF rejection for ${fileItem.file.name}; refreshing token and retrying once.`);
+ fileItem.progress = 5;
+ data = await sendUpload(await getUploadCsrfToken());
+ }
}
// Upload succeeded - recording is now on the server
diff --git a/static/js/modules/db/sliced-file-upload.js b/static/js/modules/db/sliced-file-upload.js
new file mode 100644
index 00000000..cb7c447b
--- /dev/null
+++ b/static/js/modules/db/sliced-file-upload.js
@@ -0,0 +1,219 @@
+/**
+ * Sliced upload of a file the user already has on disk.
+ *
+ * Speakr's normal upload is one multipart POST of the whole file, so a
+ * reverse proxy with a body-size limit below the file size (Cloudflare's
+ * 100 MB, for instance) rejects it before Speakr sees it. This module
+ * sends the same file as fixed-size byte slices through the
+ * recording-session endpoints instead, then POSTs the upload form to
+ * /finalize-upload, which byte-joins the slices server-side and ingests
+ * the result exactly as POST /upload would.
+ *
+ * SLICE_BYTES mirrors the server's RECORDING_SESSION_MAX_CHUNK_BYTES
+ * default; the create response is authoritative, so a deployment that
+ * lowered that env var lowers the slice size with it.
+ *
+ * It deliberately does NOT reuse server-recording-sessions.js's
+ * createSession / abortSession: those maintain the recorder's
+ * localStorage crash-recovery marker, which a file upload must not
+ * touch. The slice POSTs go through XMLHttpRequest rather than fetch so
+ * the progress bar moves within a slice and not just between slices.
+ *
+ * File slices are Blob views, so nothing here holds the file in memory:
+ * a multi-gigabyte upload costs one slice's worth of buffering in the
+ * browser's network stack.
+ */
+
+import { getUploadCsrfToken, isCsrfRejection } from '../csrf.js';
+import { computeUploadTimeout } from '../utils/upload-timeout.js';
+
+const SESSION_BASE = '/upload/session';
+
+export const SLICE_BYTES = 16 * 1024 * 1024;
+
+const MAX_SLICE_ATTEMPTS = 6;
+const BASE_RETRY_MS = 1000;
+const MAX_RETRY_MS = 30000;
+
+const RETRYABLE_STATUSES = new Set([408, 425, 429, 500, 502, 503, 504]);
+
+/** True when the file is large enough that slicing it is worth a session. */
+export function shouldSliceUpload(file) {
+ return !!file && file.size > SLICE_BYTES;
+}
+
+const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
+
+const retryDelay = (attempts) =>
+ Math.min(MAX_RETRY_MS, BASE_RETRY_MS * Math.pow(2, attempts - 1));
+
+function parseJson(text) {
+ try { return JSON.parse(text); } catch (_) { return null; }
+}
+
+/**
+ * POST via XHR, resolving for every HTTP status so callers can decide
+ * what is retryable. Rejects only on a network error or timeout, where
+ * there is no status and a retry is the right answer.
+ */
+function xhrPost(url, body, { token, timeoutMs, onProgress, contentType, onXhr }) {
+ return new Promise((resolve, reject) => {
+ const xhr = new XMLHttpRequest();
+ xhr.open('POST', url);
+ if (token) xhr.setRequestHeader('X-CSRFToken', token);
+ if (contentType) xhr.setRequestHeader('Content-Type', contentType);
+ xhr.timeout = timeoutMs;
+ if (onProgress) {
+ xhr.upload.onprogress = (e) => {
+ if (e.lengthComputable) onProgress(e.loaded, e.total);
+ };
+ }
+ xhr.onload = () => resolve({ status: xhr.status, text: xhr.responseText });
+ xhr.onerror = () => reject(new Error('Network error during upload'));
+ xhr.ontimeout = () => reject(new Error('Upload timed out'));
+ xhr.onabort = () => reject(new Error('Upload cancelled'));
+ if (onXhr) onXhr(xhr);
+ xhr.send(body);
+ });
+}
+
+async function createFileSession(file, token) {
+ const response = await fetch(SESSION_BASE, {
+ method: 'POST',
+ headers: { 'Content-Type': 'application/json', 'X-CSRFToken': token },
+ credentials: 'same-origin',
+ body: JSON.stringify({ filename: file.name, mime_type: file.type || '' }),
+ });
+ const body = parseJson(await response.text());
+ if (response.status !== 201 || !body?.session_id) {
+ throw new Error(body?.error || `Could not open upload session (HTTP ${response.status})`);
+ }
+ if (!body.upload_filename) {
+ throw new Error('Server does not support sliced uploads');
+ }
+ return body;
+}
+
+async function deleteFileSession(sessionId, token) {
+ try {
+ await fetch(`${SESSION_BASE}/${encodeURIComponent(sessionId)}`, {
+ method: 'DELETE',
+ credentials: 'same-origin',
+ headers: { 'X-CSRFToken': token },
+ });
+ } catch (error) {
+ console.warn('[SlicedUpload] Could not abort session:', error);
+ }
+}
+
+/**
+ * Send one slice, retrying transport failures and refreshing the CSRF
+ * token when the server rejects it. Returns the index the server expects
+ * next, which is normally `index + 1` but can jump when a 409 reveals
+ * the server already holds slices whose responses we lost.
+ */
+async function sendSlice(sessionId, index, blob, tokenRef, options) {
+ const url = `${SESSION_BASE}/${encodeURIComponent(sessionId)}/chunks/${index}`;
+ for (let attempt = 1; ; attempt++) {
+ let result;
+ try {
+ result = await xhrPost(url, blob, {
+ token: tokenRef.token,
+ timeoutMs: computeUploadTimeout(blob.size),
+ contentType: 'application/octet-stream',
+ onProgress: options.onSliceProgress,
+ onXhr: options.onXhr,
+ });
+ } catch (transportError) {
+ if (attempt >= MAX_SLICE_ATTEMPTS) throw transportError;
+ await sleep(retryDelay(attempt));
+ continue;
+ }
+
+ if (result.status === 204) return index + 1;
+
+ const body = parseJson(result.text);
+
+ if (result.status === 409 && Number.isInteger(body?.expected_chunk_index)) {
+ return body.expected_chunk_index;
+ }
+
+ if (isCsrfRejection(result.status, result.text) && attempt < MAX_SLICE_ATTEMPTS) {
+ tokenRef.token = await getUploadCsrfToken();
+ continue;
+ }
+
+ const error = new Error(sliceRejectionMessage(index, blob.size, result, body));
+ error.status = result.status;
+ if (!RETRYABLE_STATUSES.has(result.status) || attempt >= MAX_SLICE_ATTEMPTS) throw error;
+ await sleep(retryDelay(attempt));
+ }
+}
+
+/**
+ * A 413 on a slice is the reverse proxy, not Speakr: its body limit is
+ * below the slice size. Naming it the way a proxy does lets the upload
+ * composable's error mapping give the user the client_max_body_size
+ * guidance instead of a generic processing error.
+ */
+function sliceRejectionMessage(index, sliceSize, result, body) {
+ if (result.status === 413 && !body?.error) {
+ const megabytes = Math.round(sliceSize / (1024 * 1024));
+ return `Request entity too large: the reverse proxy rejected a ${megabytes} MB upload slice`;
+ }
+ return body?.error || `Slice ${index} rejected (HTTP ${result.status})`;
+}
+
+async function finalize(sessionId, formData, tokenRef, options) {
+ const url = `${SESSION_BASE}/${encodeURIComponent(sessionId)}/finalize-upload`;
+ for (let attempt = 1; ; attempt++) {
+ const result = await xhrPost(url, formData, {
+ token: tokenRef.token,
+ timeoutMs: options.finalizeTimeoutMs,
+ onXhr: options.onXhr,
+ });
+ const body = parseJson(result.text);
+ if (result.status === 202 && body?.id) return body;
+ if (isCsrfRejection(result.status, result.text) && attempt === 1) {
+ tokenRef.token = await getUploadCsrfToken();
+ continue;
+ }
+ throw new Error(body?.error
+ || `Upload could not be finalized (HTTP ${result.status})`);
+ }
+}
+
+/**
+ * Upload `file` in slices and finalize with `formData` (the same form the
+ * single-shot path builds, minus the file part). Resolves to the created
+ * recording, matching what POST /upload returns.
+ *
+ * `onProgress` is called with a 0..1 fraction of bytes accepted by the
+ * server. `onXhr` receives each in-flight XHR so a caller can cancel.
+ */
+export async function uploadFileInSlices(file, formData, options = {}) {
+ const tokenRef = { token: await getUploadCsrfToken() };
+ const session = await createFileSession(file, tokenRef.token);
+ const sliceBytes = Math.min(SLICE_BYTES, session.max_chunk_bytes || SLICE_BYTES);
+ const onProgress = options.onProgress || (() => {});
+
+ try {
+ let index = 1;
+ while ((index - 1) * sliceBytes < file.size) {
+ const start = (index - 1) * sliceBytes;
+ const end = Math.min(start + sliceBytes, file.size);
+ index = await sendSlice(session.session_id, index, file.slice(start, end), tokenRef, {
+ onXhr: options.onXhr,
+ onSliceProgress: (loaded) => onProgress(Math.min(1, (start + loaded) / file.size)),
+ });
+ onProgress(Math.min(1, ((index - 1) * sliceBytes) / file.size));
+ }
+ return await finalize(session.session_id, formData, tokenRef, {
+ finalizeTimeoutMs: computeUploadTimeout(file.size),
+ onXhr: options.onXhr,
+ });
+ } catch (error) {
+ await deleteFileSession(session.session_id, tokenRef.token);
+ throw error;
+ }
+}
diff --git a/static/js/modules/db/sliced-file-upload.test.js b/static/js/modules/db/sliced-file-upload.test.js
new file mode 100644
index 00000000..3cd0d3ed
--- /dev/null
+++ b/static/js/modules/db/sliced-file-upload.test.js
@@ -0,0 +1,225 @@
+/**
+ * Vitest tests for the sliced file-upload transport.
+ *
+ * The interesting code is the slice loop: fixed-size offsets, in-order
+ * indexes, retry, 409 resync, and cleaning up the session when the upload
+ * cannot continue. HTTP is exercised through fetch and XMLHttpRequest
+ * mocks.
+ */
+
+import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
+import { SLICE_BYTES, shouldSliceUpload, uploadFileInSlices } from './sliced-file-upload.js';
+
+const MB = 1024 * 1024;
+
+function fakeFile(size, name = 'talk.mp4', type = 'video/mp4') {
+ return {
+ name,
+ type,
+ size,
+ slice: (start, end) => ({ size: end - start, start, end }),
+ };
+}
+
+let sent;
+let sliceResponses;
+let finalizeResponse;
+let fetchCalls;
+
+function _installMocks() {
+ sent = [];
+ fetchCalls = [];
+ sliceResponses = [];
+ finalizeResponse = { status: 202, text: JSON.stringify({ id: 42, title: 'talk' }) };
+
+ global.window = { csrfManager: { refreshToken: async () => 'fresh-token' } };
+ global.document = { querySelector: () => null };
+
+ global.fetch = vi.fn(async (url, options) => {
+ fetchCalls.push({ url, options });
+ if (options.method === 'POST') {
+ return {
+ status: 201,
+ text: async () => JSON.stringify({
+ session_id: 'sess-1',
+ upload_filename: JSON.parse(options.body).filename,
+ max_chunk_bytes: SLICE_BYTES,
+ }),
+ };
+ }
+ return { status: 204, text: async () => '' };
+ });
+
+ global.XMLHttpRequest = class {
+ constructor() {
+ this.upload = {};
+ this.timeout = 0;
+ this.headers = {};
+ }
+
+ open(method, url) { this.url = url; }
+
+ setRequestHeader(key, value) { this.headers[key] = value; }
+
+ send(body) {
+ const call = { url: this.url, body, headers: this.headers };
+ sent.push(call);
+ const response = this.url.endsWith('/finalize-upload')
+ ? finalizeResponse
+ : (sliceResponses.shift() || { status: 204, text: '' });
+ queueMicrotask(() => {
+ if (response.networkError) {
+ this.onerror();
+ return;
+ }
+ if (this.upload.onprogress && body?.size) {
+ this.upload.onprogress({ lengthComputable: true, loaded: body.size, total: body.size });
+ }
+ this.status = response.status;
+ this.responseText = response.text ?? '';
+ this.onload();
+ });
+ }
+ };
+}
+
+const slicePosts = () => sent.filter((call) => !call.url.endsWith('/finalize-upload'));
+
+describe('shouldSliceUpload', () => {
+ it('leaves files that fit one request on the single-shot path', () => {
+ expect(shouldSliceUpload(fakeFile(SLICE_BYTES))).toBe(false);
+ expect(shouldSliceUpload(fakeFile(SLICE_BYTES + 1))).toBe(true);
+ expect(shouldSliceUpload(null)).toBe(false);
+ });
+});
+
+describe('uploadFileInSlices', () => {
+ beforeEach(() => { _installMocks(); });
+ afterEach(() => {
+ vi.restoreAllMocks();
+ vi.useRealTimers();
+ delete global.fetch;
+ delete global.XMLHttpRequest;
+ delete global.window;
+ delete global.document;
+ });
+
+ it('opens a session for the filename, sends whole slices in order, and finalizes', async () => {
+ const file = fakeFile(40 * MB);
+ const form = new Map([['title', 'Sliced']]);
+ const progress = [];
+
+ const result = await uploadFileInSlices(file, form, {
+ onProgress: (fraction) => progress.push(fraction),
+ });
+
+ expect(JSON.parse(fetchCalls[0].options.body)).toEqual({
+ filename: 'talk.mp4',
+ mime_type: 'video/mp4',
+ });
+ expect(slicePosts().map((call) => call.url)).toEqual([
+ '/upload/session/sess-1/chunks/1',
+ '/upload/session/sess-1/chunks/2',
+ '/upload/session/sess-1/chunks/3',
+ ]);
+ expect(slicePosts().map((call) => call.body.size)).toEqual([SLICE_BYTES, SLICE_BYTES, 8 * MB]);
+ expect(slicePosts()[0].headers['X-CSRFToken']).toBe('fresh-token');
+
+ const finalizeCall = sent[sent.length - 1];
+ expect(finalizeCall.url).toBe('/upload/session/sess-1/finalize-upload');
+ expect(finalizeCall.body).toBe(form);
+ expect(result).toEqual({ id: 42, title: 'talk' });
+ expect(progress[progress.length - 1]).toBe(1);
+ });
+
+ it('honours a lower max_chunk_bytes from the server', async () => {
+ global.fetch = vi.fn(async (url, options) => ({
+ status: 201,
+ text: async () => JSON.stringify({
+ session_id: 'sess-1',
+ upload_filename: 'talk.mp4',
+ max_chunk_bytes: 4 * MB,
+ }),
+ }));
+
+ await uploadFileInSlices(fakeFile(10 * MB), new Map(), {});
+
+ expect(slicePosts().map((call) => call.body.size)).toEqual([4 * MB, 4 * MB, 2 * MB]);
+ });
+
+ it('resends the same offset after a network error', async () => {
+ vi.useFakeTimers();
+ sliceResponses = [{ networkError: true }];
+
+ const upload = uploadFileInSlices(fakeFile(20 * MB), new Map(), {});
+ await vi.advanceTimersByTimeAsync(2000);
+ await upload;
+
+ expect(slicePosts().map((call) => [call.url, call.body.start])).toEqual([
+ ['/upload/session/sess-1/chunks/1', 0],
+ ['/upload/session/sess-1/chunks/1', 0],
+ ['/upload/session/sess-1/chunks/2', SLICE_BYTES],
+ ]);
+ });
+
+ it('skips ahead when a 409 says the server already holds the slice', async () => {
+ sliceResponses = [{ status: 409, text: JSON.stringify({ expected_chunk_index: 3 }) }];
+
+ await uploadFileInSlices(fakeFile(40 * MB), new Map(), {});
+
+ expect(slicePosts().map((call) => [call.url, call.body.start])).toEqual([
+ ['/upload/session/sess-1/chunks/1', 0],
+ ['/upload/session/sess-1/chunks/3', 2 * SLICE_BYTES],
+ ]);
+ });
+
+ it('gives up on a rejected slice and deletes the half-uploaded session', async () => {
+ sliceResponses = [
+ { status: 204, text: '' },
+ { status: 413, text: JSON.stringify({ error: 'Chunk exceeds max_chunk_bytes' }) },
+ ];
+
+ await expect(uploadFileInSlices(fakeFile(40 * MB), new Map(), {}))
+ .rejects.toThrow('Chunk exceeds max_chunk_bytes');
+
+ expect(slicePosts()).toHaveLength(2);
+ expect(fetchCalls[fetchCalls.length - 1]).toMatchObject({
+ url: '/upload/session/sess-1',
+ options: { method: 'DELETE' },
+ });
+ });
+
+ it('blames the reverse proxy for a 413 with no Speakr error body', async () => {
+ sliceResponses = [{ status: 413, text: '<html><head><title>413 Request Entity Too Large</title></head></html>' }];
+
+ await expect(uploadFileInSlices(fakeFile(40 * MB), new Map(), {}))
+ .rejects.toThrow('Request entity too large: the reverse proxy rejected a 16 MB upload slice');
+ });
+
+ it('retries a slice with a fresh token when the server rejects the CSRF one', async () => {
+ sliceResponses = [{ status: 400, text: 'The CSRF token has expired.' }];
+
+ await uploadFileInSlices(fakeFile(20 * MB), new Map(), {});
+
+ expect(slicePosts().map((call) => call.body.start)).toEqual([0, 0, SLICE_BYTES]);
+ });
+
+ it('refuses to upload when the server does not understand sliced uploads', async () => {
+ global.fetch = vi.fn(async () => ({
+ status: 201,
+ text: async () => JSON.stringify({ session_id: 'sess-1', mime_type: 'audio/webm' }),
+ }));
+
+ await expect(uploadFileInSlices(fakeFile(20 * MB), new Map(), {}))
+ .rejects.toThrow('does not support sliced uploads');
+ expect(sent).toHaveLength(0);
+ });
+
+ it('surfaces the finalize error and deletes the session', async () => {
+ finalizeResponse = { status: 409, text: JSON.stringify({ error: 'Slices on disk do not match the session' }) };
+
+ await expect(uploadFileInSlices(fakeFile(20 * MB), new Map(), {}))
+ .rejects.toThrow('Slices on disk do not match the session');
+ expect(fetchCalls[fetchCalls.length - 1].options.method).toBe('DELETE');
+ });
+});
diff --git a/tests/test_sliced_file_upload.py b/tests/test_sliced_file_upload.py
new file mode 100644
index 00000000..f3ebe163
--- /dev/null
+++ b/tests/test_sliced_file_upload.py
@@ -0,0 +1,305 @@
+"""End-to-end tests for sliced uploads of files already on disk.
+
+A sliced upload reuses the recording-session endpoints to carry a file in
+fixed-size byte slices past a reverse proxy whose body limit is below the
+file size, then finalizes through /finalize-upload, which byte-joins the
+slices and runs them through the same ingestion pipeline POST /upload
+uses.
+
+The external effects of that pipeline (codec probe, conversion, storage,
+job queue) are mocked with the harness the /upload tests already use, so
+these tests assert the parts that are specific to slicing: the session is
+recognisable as a file upload, the reassembled bytes are the original
+bytes, the recorder's stitch path is kept away from them, and an
+abandoned one is expired rather than assembled into a truncated
+recording.
+"""
+
+import hashlib
+import os
+import shutil
+import sys
+import tempfile
+import uuid
+from datetime import datetime, timedelta
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from src.app import app, db
+from src.models import Recording, RecordingSession
+from src.api.recording_sessions import cleanup_expired_sessions
+
+from tests.test_cov_recordings_write import _cleanup, _mk_user, _upload_mocks
+from tests.test_recording_sessions import _login
+
+app.config["WTF_CSRF_ENABLED"] = False
+
+SLICE_BYTES = 4096
+
+
+def _tmp_upload_folder():
+ return tempfile.mkdtemp(prefix="speakr-test-sliced-")
+
+
+def _payload(size):
+ return bytes(range(256)) * (size // 256) + b"\xab" * (size % 256)
+
+
+def _open_session(client, filename="talk.mp4", mime_type="video/mp4"):
+ response = client.post("/upload/session", json={"filename": filename, "mime_type": mime_type})
+ assert response.status_code == 201, response.data
+ return response.get_json()
+
+
+def _send_slices(client, session_id, payload, slice_bytes=SLICE_BYTES):
+ for index, start in enumerate(range(0, len(payload), slice_bytes), start=1):
+ response = client.post(
+ f"/upload/session/{session_id}/chunks/{index}",
+ data=payload[start:start + slice_bytes],
+ content_type="application/octet-stream",
+ )
+ assert response.status_code == 204, (index, response.data)
+
+
+def test_create_session_with_filename_marks_it_a_file_upload():
+ upload_folder = _tmp_upload_folder()
+ with app.app_context():
+ app.config["UPLOAD_FOLDER"] = upload_folder
+ user = _mk_user("slice_create")
+ client = app.test_client()
+ _login(client, user)
+
+ body = _open_session(client)
+ assert body["upload_filename"] == "talk.mp4"
+ assert db.session.get(RecordingSession, body["session_id"]).upload_filename == "talk.mp4"
+
+ client.delete(f"/upload/session/{body['session_id']}")
+ _cleanup(user)
+ shutil.rmtree(upload_folder, ignore_errors=True)
+
+
+def test_create_session_with_filename_skips_the_recorder_mime_whitelist():
+ """A user's own file can be any container ffmpeg reads, so the whitelist
+ that guards the recorder's stitch path must not gate these sessions."""
+ upload_folder = _tmp_upload_folder()
+ with app.app_context():
+ app.config["UPLOAD_FOLDER"] = upload_folder
+ user = _mk_user("slice_mime")
+ client = app.test_client()
+ _login(client, user)
+
+ rejected = client.post("/upload/session", json={"mime_type": "video/x-matroska"})
+ assert rejected.status_code == 400
+
+ accepted = _open_session(client, filename="lecture.mkv", mime_type="video/x-matroska")
+ assert accepted["mime_type"] == "video/x-matroska"
+
+ client.delete(f"/upload/session/{accepted['session_id']}")
+ _cleanup(user)
+ shutil.rmtree(upload_folder, ignore_errors=True)
+
+
+def test_finalize_upload_ingests_the_reassembled_original_bytes():
+ upload_folder = _tmp_upload_folder()
+ with app.app_context():
+ app.config["UPLOAD_FOLDER"] = upload_folder
+ user = _mk_user("slice_ingest")
+ client = app.test_client()
+ _login(client, user)
+
+ payload = _payload(SLICE_BYTES * 3 + 17)
+ session_id = _open_session(client, filename="sample.mp3", mime_type="audio/mpeg")["session_id"]
+ _send_slices(client, session_id, payload)
+
+ staging = os.path.join(upload_folder, f"stg_{uuid.uuid4().hex[:6]}")
+ with _upload_mocks(staging) as (storage, enqueue):
+ response = client.post(
+ f"/upload/session/{session_id}/finalize-upload",
+ data={"title": "Sliced", "notes": "n"},
+ content_type="multipart/form-data",
+ )
+
+ assert response.status_code == 202, response.data
+ recording = db.session.get(Recording, response.get_json()["id"])
+ assert recording.title == "Sliced"
+ assert recording.original_filename == "sample.mp3"
+ assert recording.file_hash == hashlib.sha256(payload).hexdigest()
+ assert enqueue.called
+
+ session = db.session.get(RecordingSession, session_id)
+ assert session.status == "finalized"
+ assert session.finalized_recording_id == recording.id
+ assert not os.path.isdir(os.path.join(upload_folder, "_sessions", session_id))
+
+ _cleanup(recording, user)
+ shutil.rmtree(upload_folder, ignore_errors=True)
+
+
+def test_finalize_upload_carries_the_upload_form_through_ingestion():
+ upload_folder = _tmp_upload_folder()
+ with app.app_context():
+ app.config["UPLOAD_FOLDER"] = upload_folder
+ user = _mk_user("slice_form")
+ client = app.test_client()
+ _login(client, user)
+
+ session_id = _open_session(client, filename="sample.mp3", mime_type="audio/mpeg")["session_id"]
+ _send_slices(client, session_id, _payload(SLICE_BYTES * 2))
+
+ staging = os.path.join(upload_folder, f"stg_{uuid.uuid4().hex[:6]}")
+ with _upload_mocks(staging) as (storage, enqueue):
+ response = client.post(
+ f"/upload/session/{session_id}/finalize-upload",
+ data={"language": "fr", "min_speakers": "2", "max_speakers": "4"},
+ content_type="multipart/form-data",
+ )
+
+ assert response.status_code == 202, response.data
+ params = enqueue.call_args.kwargs["params"]
+ assert params["language"] == "fr"
+ assert params["min_speakers"] == 2
+ assert params["max_speakers"] == 4
+
+ _cleanup(db.session.get(Recording, response.get_json()["id"]), user)
+ shutil.rmtree(upload_folder, ignore_errors=True)
+
+
+def test_finalize_upload_rejects_a_slice_count_that_disagrees_with_disk():
+ """A slice lost between the write and the bookkeeping commit would
+ otherwise be silently ingested as a truncated file."""
+ upload_folder = _tmp_upload_folder()
+ with app.app_context():
+ app.config["UPLOAD_FOLDER"] = upload_folder
+ user = _mk_user("slice_short")
+ client = app.test_client()
+ _login(client, user)
+
+ session_id = _open_session(client, filename="sample.mp3", mime_type="audio/mpeg")["session_id"]
+ _send_slices(client, session_id, _payload(SLICE_BYTES * 2))
+ os.remove(os.path.join(upload_folder, "_sessions", session_id, "chunk-000002.bin"))
+
+ response = client.post(
+ f"/upload/session/{session_id}/finalize-upload",
+ data={},
+ content_type="multipart/form-data",
+ )
+ assert response.status_code == 409
+ assert response.get_json()["expected_chunk_index"] == 2
+
+ client.delete(f"/upload/session/{session_id}")
+ _cleanup(user)
+ shutil.rmtree(upload_folder, ignore_errors=True)
+
+
+def test_the_two_finalize_routes_reject_each_others_sessions():
+ """The recorder's finalize remuxes and sniffs container headers per
+ chunk, which would rewrite or mis-split a byte-sliced file; the sliced
+ finalize skips the stitch a paused recording needs."""
+ upload_folder = _tmp_upload_folder()
+ with app.app_context():
+ app.config["UPLOAD_FOLDER"] = upload_folder
+ user = _mk_user("slice_routes")
+ client = app.test_client()
+ _login(client, user)
+
+ file_session = _open_session(client, filename="sample.mp3", mime_type="audio/mpeg")["session_id"]
+ _send_slices(client, file_session, _payload(SLICE_BYTES))
+ stitch_refusal = client.post(f"/upload/session/{file_session}/finalize", json={})
+ assert stitch_refusal.status_code == 409
+ assert "finalize-upload" in stitch_refusal.get_json()["error"]
+
+ recorder_session = client.post(
+ "/upload/session", json={"mime_type": "audio/webm"}).get_json()["session_id"]
+ client.post(f"/upload/session/{recorder_session}/chunks/1",
+ data=b"\x00" * 64, content_type="application/octet-stream")
+ upload_refusal = client.post(
+ f"/upload/session/{recorder_session}/finalize-upload",
+ data={}, content_type="multipart/form-data")
+ assert upload_refusal.status_code == 409
+ assert "/finalize" in upload_refusal.get_json()["error"]
+
+ client.delete(f"/upload/session/{file_session}")
+ client.delete(f"/upload/session/{recorder_session}")
+ _cleanup(user)
+ shutil.rmtree(upload_folder, ignore_errors=True)
+
+
+def test_cleanup_expires_an_abandoned_sliced_upload_instead_of_assembling_it():
+ """Abandoned recorder sessions are auto-finalized because the browser
+ holds the only copy of that audio. A sliced upload's original is still
+ on the user's disk, and its slices are a truncated container, so
+ assembling one would land a broken recording in the library."""
+ upload_folder = _tmp_upload_folder()
+ with app.app_context():
+ app.config["UPLOAD_FOLDER"] = upload_folder
+ user = _mk_user("slice_expire")
+ client = app.test_client()
+ _login(client, user)
+
+ session_id = _open_session(client, filename="sample.mp3", mime_type="audio/mpeg")["session_id"]
+ _send_slices(client, session_id, _payload(SLICE_BYTES))
+
+ session = db.session.get(RecordingSession, session_id)
+ session.last_seen_at = datetime.utcnow() - timedelta(hours=48)
+ db.session.commit()
+
+ cleanup_expired_sessions(app)
+
+ db.session.expire_all()
+ assert db.session.get(RecordingSession, session_id).status == "expired"
+ assert not os.path.isdir(os.path.join(upload_folder, "_sessions", session_id))
+ assert Recording.query.filter_by(user_id=user.id).count() == 0
+
+ _cleanup(user)
+ shutil.rmtree(upload_folder, ignore_errors=True)
+
+
+def test_a_replayed_finalize_upload_does_not_ingest_the_file_twice():
+ """Finalize is synchronous, so a client that retries it after a lost
+ response would otherwise land the same file in the library twice."""
+ upload_folder = _tmp_upload_folder()
+ with app.app_context():
+ app.config["UPLOAD_FOLDER"] = upload_folder
+ user = _mk_user("slice_replay")
+ client = app.test_client()
+ _login(client, user)
+
+ session_id = _open_session(client, filename="sample.mp3", mime_type="audio/mpeg")["session_id"]
+ _send_slices(client, session_id, _payload(SLICE_BYTES))
+
+ staging = os.path.join(upload_folder, f"stg_{uuid.uuid4().hex[:6]}")
+ with _upload_mocks(staging):
+ first = client.post(f"/upload/session/{session_id}/finalize-upload",
+ data={}, content_type="multipart/form-data")
+ second = client.post(f"/upload/session/{session_id}/finalize-upload",
+ data={}, content_type="multipart/form-data")
+
+ assert first.status_code == 202, first.data
+ assert second.status_code == 409
+ assert Recording.query.filter_by(user_id=user.id).count() == 1
+
+ _cleanup(db.session.get(Recording, first.get_json()["id"]), user)
+ shutil.rmtree(upload_folder, ignore_errors=True)
+
+
+def test_another_user_cannot_finalize_someone_elses_sliced_upload():
+ upload_folder = _tmp_upload_folder()
+ with app.app_context():
+ app.config["UPLOAD_FOLDER"] = upload_folder
+ owner = _mk_user("slice_owner")
+ intruder = _mk_user("slice_intruder")
+ client = app.test_client()
+ _login(client, owner)
+ session_id = _open_session(client, filename="sample.mp3", mime_type="audio/mpeg")["session_id"]
+ _send_slices(client, session_id, _payload(SLICE_BYTES))
+
+ other = app.test_client()
+ _login(other, intruder)
+ response = other.post(
+ f"/upload/session/{session_id}/finalize-upload",
+ data={}, content_type="multipart/form-data")
+ assert response.status_code == 404
+
+ client.delete(f"/upload/session/{session_id}")
+ _cleanup(owner, intruder)
+ shutil.rmtree(upload_folder, ignore_errors=True)