Speakr is a personal, self-hosted web application designed for transcribing audio recordings
Python
feat: upload files larger than the proxy body limit in slices
François Catuhebc03e9996ca5
diff --git a/docs/admin-guide/recording-sessions.md b/docs/admin-guide/recording-sessions.mdindex 6b252184..e29fd314 100644--- a/docs/admin-guide/recording-sessions.md+++ b/docs/admin-guide/recording-sessions.md@@ -25,6 +25,45 @@ so the legacy 200 MB cap and its warning still apply to them. If a recordingdid stream to the server and the user switches it to incognito in the reviewpane afterwards, the server-side chunks are deleted before processing.+## Sliced uploads of files on disk++The same session machinery also carries uploads of files the user already+has on disk. A normal upload is one multipart `POST /upload` of the whole+file, so a reverse proxy whose body limit is below the file size rejects+it before Speakr sees it (Cloudflare's proxy caps request bodies at+100 MB on most plans). Any file larger than one slice is instead sent as+fixed-size byte slices through `POST /upload/session/{id}/chunks/{N}` and+finalized with `POST /upload/session/{id}/finalize-upload`, which+byte-joins the slices and runs the result through the same ingestion+pipeline as a single-shot upload.++This needs no configuration and is not covered by+`ENABLE_SERVER_RECORDING_CHUNKS`, which governs the recorder only. The+slice size is `RECORDING_SESSION_MAX_CHUNK_BYTES` (16 MB), advertised in+the create response, and files at or below that size still go up in one+request.++A few differences from a recorder session are worth knowing when reading+logs:++- Its `recording_session` row has `upload_filename` set. That is what+ distinguishes the two, and each finalize route rejects the other's+ sessions with a 409.+- The slices are byte ranges of one complete container, not+ MediaRecorder timeslices, so finalize does a plain byte-join with no+ ffmpeg concat, no remux, and no container sniffing. The stored file is+ byte-identical to what the user picked.+- `finalize-upload` is synchronous: hashing, probing and any conversion+ happen in-request exactly as they do for `POST /upload`, so it holds+ the connection for as long as that takes and needs the same generous+ proxy read timeout.+- An abandoned one is expired by the cleanup sweep rather than+ auto-finalized. The user still has the original file, and its slices+ are a truncated container rather than a playable partial recording.+- Losing the finalize response loses the upload: unlike the recorder's+ `/finalize`, it is not replayable. The client re-uploads, and+ duplicate detection flags the second copy if the first landed.+## ConfigurationEnvironment variables, all optional:@@ -50,7 +89,8 @@ so neither is killed in flight.```nginxlocation /upload/session/ {- # Chunk POSTs are small; keep timeouts modest.+ # Chunk POSTs are capped at RECORDING_SESSION_MAX_CHUNK_BYTES (16 MB),+ # whether they carry a recorder timeslice or a file slice.proxy_pass http://speakr_upstream;proxy_set_header Host $host;proxy_set_header X-Real-IP $remote_addr;@@ -60,9 +100,10 @@ location /upload/session/ {proxy_read_timeout 30s;}-location ~* ^/upload/session/.+/finalize$ {- # Finalize triggers ffmpeg concat which can take tens of seconds- # for long recordings. Give it room.+location ~* ^/upload/session/.+/finalize(-upload)?$ {+ # Finalize triggers ffmpeg concat which can take tens of seconds for+ # long recordings, and finalize-upload also hashes, probes and may+ # convert the file in-request. Give it room.proxy_pass http://speakr_upstream;proxy_set_header Host $host;proxy_set_header X-Real-IP $remote_addr;@@ -80,7 +121,7 @@ the larger timeout; finalize will get the headroom it needs.### Caddy```-@finalize path_regexp ^/upload/session/[^/]+/finalize$+@finalize path_regexp ^/upload/session/[^/]+/finalize(-upload)?$handle @finalize {reverse_proxy speakr:8899 {transport http {@@ -103,6 +144,7 @@ LimitRequestBody 33554432UPLOAD_FOLDER/_sessions/<uuid-1>/session.json+ joined.bin ← sliced uploads only, during finalizechunk-000001.binchunk-000002.bin...@@ -153,10 +195,11 @@ on the server, or discard them.| Method | Path | Purpose ||---|---|---|-| POST | `/upload/session` | Create a new session. Body: `{mime_type}`. |+| POST | `/upload/session` | Create a new session. Body: `{mime_type}`, plus `{filename}` for a sliced file upload. || POST | `/upload/session/{id}/chunks/{N}` | Append chunk N (must be `chunk_count + 1`). Body is raw bytes. || GET | `/upload/session/{id}` | Status of an existing session. || POST | `/upload/session/{id}/finalize` | Request asynchronous stitch + transcribe kickoff. |+| POST | `/upload/session/{id}/finalize-upload` | Reassemble a sliced file upload and ingest it. Body is the `POST /upload` form without the file. || DELETE | `/upload/session/{id}` | Abort and remove the on-disk chunks. |All endpoints require an authenticated session. The CSRF token from thediff --git a/docs/features.md b/docs/features.mdindex f53d3265..9afc406d 100644--- a/docs/features.md+++ b/docs/features.md@@ -20,6 +20,10 @@ A per-recording Stats tab shows total length, speaker count, conversation turns,An opt-in mode (`ENABLE_SERVER_RECORDING_CHUNKS`) streams in-app recording chunks to the server as you record instead of buffering them in browser memory, raising the practical recording ceiling to hours-long sessions and letting you resume a recording after a page reload. See the [Recording guide](user-guide/recording.md#server-side-recording-sessions-long-recordings) and [admin setup](admin-guide/recording-sessions.md).+### Sliced File Uploads++Uploads of files larger than 16 MB are sent as fixed-size slices through the same session endpoints and reassembled server-side, so a reverse proxy with a request-body limit below the file size (Cloudflare caps bodies at 100 MB on most plans) no longer blocks large uploads. This needs no configuration; see [Recording Sessions](admin-guide/recording-sessions.md#sliced-uploads-of-files-on-disk).+## Core Transcription Features### Multi-Engine Supportdiff --git a/src/api/recording_sessions.py b/src/api/recording_sessions.pyindex 0c6f6326..13de1a91 100644--- a/src/api/recording_sessions.py+++ b/src/api/recording_sessions.py@@ -13,11 +13,22 @@ EndpointsPOST /upload/session/<sid>/chunks/<n> append chunk (n=1..)GET /upload/session/<sid> statusPOST /upload/session/<sid>/finalize request stitch+ POST /upload/session/<sid>/finalize-upload reassemble a sliced fileDELETE /upload/session/<sid> abort + cleanupAll endpoints require login. They write the on-disk layout described in``src/models/recording_session.py``.+Creating a session with a ``filename`` makes it a sliced upload of a file+the user already has on disk (used to cross a reverse proxy whose body+limit is below the file size) rather than a recorder session. Those+sessions share create / chunk / abort with the recorder but finalize+through ``/finalize-upload``, which byte-joins the slices and hands the+result to the same ingestion pipeline ``POST /upload`` uses. The declared+mime type is not checked against ``RECORDING_SESSION_ALLOWED_MIME_TYPES``+for them: the user picks the container, and ffprobe gates it at ingest.+The two finalize routes reject each other's sessions.+Finalize is async: the endpoint creates a placeholder Recording row in``STITCHING`` status, enqueues a ``stitch`` job via the existing``job_queue``, and returns the recording id immediately. The worker@@ -49,6 +60,7 @@ from flask_login import login_required, current_userfrom src.database import dbfrom src.models import RecordingSession, Recording, RECORDING_SESSION_STATUSESfrom src.services.job_queue import job_queue+from src.services.recording_stitch import byte_join, session_chunk_pathsrecording_sessions_bp = Blueprint('recording_sessions', __name__)@@ -263,17 +275,27 @@ def _replay_finalize_response(session):def create_session():"""Create a new in-progress recording session.- Body (JSON, all optional): {"mime_type": "audio/webm"}.+ Body (JSON, all optional): ``{"mime_type": "audio/webm"}`` for a+ recorder session, plus ``{"filename": "talk.mp4"}`` to make it a+ sliced upload of a file the user already has on disk.Returns ``{session_id, expires_at, max_chunk_bytes}`` on success."""data = request.get_json(silent=True) or {}- mime_type = (data.get('mime_type') or 'audio/webm').strip().lower()- if mime_type not in _allowed_mime_types():- return jsonify({- 'error': f'Unsupported mime_type {mime_type!r}',- 'allowed': sorted(_allowed_mime_types()),- }), 400+ declared_mime = (data.get('mime_type') or '').strip().lower()+ upload_filename = (data.get('filename') or '').strip()++ if upload_filename:+ if len(upload_filename) > 255:+ return jsonify({'error': 'filename must be 255 characters or fewer'}), 400+ mime_type = declared_mime or 'application/octet-stream'+ else:+ mime_type = declared_mime or 'audio/webm'+ if mime_type not in _allowed_mime_types():+ return jsonify({+ 'error': f'Unsupported mime_type {mime_type!r}',+ 'allowed': sorted(_allowed_mime_types()),+ }), 400# Per-user quota check before we let them open another session.if _user_bytes_in_progress(current_user.id) >= _max_bytes_per_user():@@ -285,6 +307,7 @@ def create_session():session = RecordingSession(user_id=current_user.id,mime_type=mime_type,+ upload_filename=upload_filename or None,status='recording',)db.session.add(session)@@ -304,6 +327,7 @@ def create_session():return jsonify({'session_id': session.id,'mime_type': session.mime_type,+ 'upload_filename': session.upload_filename,'status': session.status,'expires_at': expires_at.isoformat(),'max_chunk_bytes': _max_chunk_bytes(),@@ -438,6 +462,11 @@ def finalize_session(session_id):if err is not None:return err+ if session.upload_filename:+ return jsonify({+ 'error': 'Session is a sliced file upload; finalize it with /finalize-upload',+ }), 409+if session.status not in ('recording', 'finalizing'):return jsonify({'error': f'Session is in status {session.status!r}; cannot finalize',@@ -523,6 +552,127 @@ def finalize_session(session_id):}), 202+class ReassembledUpload:+ """A Werkzeug ``FileStorage``-shaped view of reassembled slices.++ Exposes only what :func:`ingest_uploaded_recording` asks of an upload+ (``filename``, ``seek``, ``tell``, ``save``) so a sliced upload runs+ through the same ingestion pipeline as a single-shot one. ``save``+ moves the joined file rather than copying it, so a large upload is+ written once.+ """++ def __init__(self, path, filename):+ self._path = path+ self.filename = filename+ self._size = os.path.getsize(path)+ self._pos = 0++ def seek(self, offset, whence=os.SEEK_SET):+ if whence == os.SEEK_END:+ self._pos = self._size + offset+ elif whence == os.SEEK_CUR:+ self._pos += offset+ else:+ self._pos = offset+ return self._pos++ def tell(self):+ return self._pos++ def save(self, dst):+ shutil.move(self._path, dst)+++@recording_sessions_bp.route('/upload/session/<string:session_id>/finalize-upload', methods=['POST'])+@login_required+def finalize_sliced_upload(session_id):+ """Reassemble a sliced file upload and ingest it.++ The body is the same multipart form ``POST /upload`` takes, minus the+ file part: the bytes come from the session's slices instead. The+ response is that endpoint's response, so a client can treat the two+ transports interchangeably.++ Unlike the recorder's ``/finalize``, this is synchronous and not+ replayable: ingestion (hash, probe, convert) happens in-request+ exactly as it does for a single-shot upload. A client that loses the+ response has to re-upload, and Speakr's duplicate detection flags the+ second copy. Concurrent calls are claimed atomically so only one of+ them ingests.+ """+ session = db.session.get(RecordingSession, session_id)+ err = _ensure_owned(session)+ if err is not None:+ return err++ if not session.upload_filename:+ return jsonify({+ 'error': 'Session is a recorder session; finalize it with /finalize',+ }), 409++ if session.status != 'recording':+ return jsonify({+ 'error': f'Session is in status {session.status!r}; cannot finalize',+ }), 409++ dir_path = _session_dir(session.id)+ slice_paths = session_chunk_paths(dir_path)+ if not slice_paths or len(slice_paths) != session.chunk_count:+ return jsonify({+ 'error': 'Slices on disk do not match the session',+ 'expected_chunk_index': len(slice_paths) + 1,+ 'chunk_count': session.chunk_count,+ }), 409++ claimed = db.session.query(RecordingSession).filter(+ RecordingSession.id == session.id,+ RecordingSession.status == 'recording',+ ).update({'status': 'finalizing'}, synchronize_session=False)+ if not claimed:+ db.session.rollback()+ return jsonify({'error': 'Session is already being finalized'}), 409+ session.last_seen_at = datetime.utcnow()+ db.session.commit()++ joined_path = os.path.join(dir_path, 'joined.bin')+ try:+ byte_join(slice_paths, joined_path)+ except OSError as e:+ current_app.logger.error(f"Could not join slices for session {session_id}: {e}")+ _fail_sliced_upload(session, f'Could not reassemble slices: {e}')+ return jsonify({'error': 'Server could not reassemble the uploaded slices'}), 500++ from src.api.recordings import ingest_uploaded_recording+ result = ingest_uploaded_recording(+ owner=current_user,+ uploaded_file=ReassembledUpload(joined_path, session.upload_filename),+ form=request.form,+ original_filename=session.upload_filename,+ )++ response, status = result if isinstance(result, tuple) else (result, 200)+ if 200 <= status < 300:+ body = response.get_json(silent=True) or {}+ session.status = 'finalized'+ session.finalized_at = datetime.utcnow()+ session.finalized_recording_id = body.get('id')+ session.last_seen_at = datetime.utcnow()+ db.session.commit()+ else:+ _fail_sliced_upload(session, f'Ingestion rejected the upload (HTTP {status})')++ _remove_session_dir(session_id)+ return result+++def _fail_sliced_upload(session, message):+ session.status = 'failed'+ session.error_message = message+ session.last_seen_at = datetime.utcnow()+ db.session.commit()++@recording_sessions_bp.route('/upload/session/<string:session_id>', methods=['DELETE'])@login_requireddef abort_session(session_id):@@ -563,6 +713,9 @@ def cleanup_expired_sessions(app=None):what was uploaded and land it in the user's library (same path as amanual finalize), rather than discarding it.- ``recording`` with no chunks → expire and delete the empty directory.+ - a sliced file upload (``upload_filename`` set) → expire, whatever it+ holds: the user still has the original file, and its slices are a+ truncated container rather than a playable partial recording.- ``finalizing`` (already has a placeholder recording) but never reacheda terminal stitch state → **re-enqueue** the stitch to unstick it(enqueue dedupes an already-active job). Bumping ``last_seen_at`` here@@ -584,7 +737,7 @@ def cleanup_expired_sessions(app=None):finalized = rekicked = expired = 0for s in candidates:try:- if s.status == 'recording' and (s.chunk_count or 0) > 0:+ if s.status == 'recording' and (s.chunk_count or 0) > 0 and not s.upload_filename:when = s.created_at.strftime('%Y-%m-%d %H:%M') if s.created_at else ''title = f"Recovered recording {when}".strip()recovered_rec, _enq_err = _finalize_session_into_stitch(diff --git a/src/init_db.py b/src/init_db.pyindex be0c678f..d9676c3f 100644--- a/src/init_db.py+++ b/src/init_db.py@@ -652,6 +652,10 @@ def _run_migrations(app, engine):app.logger.warning(f"Error during meeting_date migration: {e}")app.logger.warning("New recordings will work correctly, but existing dates may need manual migration")+ with _migration_section(app, failures, "sliced file upload sessions"):+ if add_column_if_not_exists(engine, 'recording_session', 'upload_filename', 'VARCHAR(255)'):+ app.logger.info("Added upload_filename column to recording_session table")+with _migration_section(app, failures, "performance and uniqueness indexes"):# Add index on TranscriptChunk.speaker_name for performance# This improves speaker rename operations which update all chunksdiff --git a/src/models/recording_session.py b/src/models/recording_session.pyindex 4434faf9..73136d88 100644--- a/src/models/recording_session.py+++ b/src/models/recording_session.py@@ -20,6 +20,16 @@ On finalize, the chunks are stitched into a single file via ffmpeg concatdemux (handles pause/resume across MediaRecorder restarts, which a rawcat would corrupt). The resulting file becomes the new Recording'saudio_path and the session directory is removed.++The same session machinery also carries sliced uploads of files the user+already has on disk, so those can cross a reverse proxy with a body-size+limit below the file size. Such a session has ``upload_filename`` set,+which is what distinguishes it from a recorder session. Its chunks are+fixed-size byte slices of one complete container rather than+MediaRecorder timeslices, so it finalizes through the normal upload+ingestion path (plain byte-join, no stitch, no remux) and is never+auto-finalized when abandoned: the user's original file still exists, so+there is nothing to recover."""import uuid@@ -63,6 +73,8 @@ class RecordingSession(db.Model):# the declared type) and pick the right ffmpeg input args at stitch time.mime_type = db.Column(db.String(100), nullable=False, default='audio/webm')+ upload_filename = db.Column(db.String(255), nullable=True)+# Status transitions are guarded by the API; see RECORDING_SESSION_STATUSES.status = db.Column(db.String(20), nullable=False, default='recording', index=True)@@ -112,6 +124,7 @@ class RecordingSession(db.Model):'session_id': self.id,'status': self.status,'mime_type': self.mime_type,+ 'upload_filename': self.upload_filename,'chunk_count': self.chunk_count,'bytes_received': self.bytes_received,'created_at': self.created_at.isoformat() if self.created_at else None,diff --git a/static/js/modules/composables/upload.js b/static/js/modules/composables/upload.jsindex 581e7dbd..2293eecf 100644--- a/static/js/modules/composables/upload.js+++ b/static/js/modules/composables/upload.js@@ -7,6 +7,7 @@ import * as FailedUploads from '../db/failed-uploads.js';import * as IncognitoStorage from '../db/incognito-storage.js';import * as RecordingDB from '../db/recording-persistence.js';import { getUploadCsrfToken, isCsrfRejection } from '../csrf.js';+import { shouldSliceUpload, uploadFileInSlices } from '../db/sliced-file-upload.js';import { computeUploadTimeout } from '../utils/upload-timeout.js';// Parse error message and return friendly error info@@ -467,16 +468,23 @@ export function useUpload(state, utils) {* Upload a single file to the server.* Acquires a concurrency slot, uploads, then releases.* Status updates are per-item (no global processingProgress).+ *+ * Files past the slice threshold go up through the sliced-upload+ * transport instead of one multipart POST, so a reverse proxy whose+ * body limit is below the file size cannot reject them. Both+ * transports send the same form and return the same response, so+ * everything below the transport call is shared.*/const uploadSingleFile = async (fileItem) => {await acquireUploadSlot();fileItem.status = 'uploading';fileItem.progress = 5;+ const sliced = shouldSliceUpload(fileItem.file);try {const formData = new FormData();- formData.append('file', fileItem.file);+ if (!sliced) formData.append('file', fileItem.file);// Send file's lastModified timestamp for meeting_dateif (fileItem.file.lastModified) {@@ -650,18 +658,27 @@ export function useUpload(state, utils) {xhr.send(formData);});- // Refresh the token right before sending, and retry exactly once- // with another fresh token if the server still rejects it —- // mirroring what the fetch interceptor in csrf-refresh.js does- // for non-XHR requests.let data;- try {- data = await sendUpload(await getUploadCsrfToken());- } catch (uploadError) {- if (!uploadError.isCsrfRejection) throw uploadError;- console.warn(`[Upload] CSRF rejection for ${fileItem.file.name}; refreshing token and retrying once.`);- fileItem.progress = 5;- data = await sendUpload(await getUploadCsrfToken());+ if (sliced) {+ data = await uploadFileInSlices(fileItem.file, formData, {+ onProgress: (fraction) => {+ fileItem.progress = Math.round(5 + fraction * 85);+ },+ onXhr: (xhr) => { fileItem._xhr = xhr; },+ });+ } else {+ // Refresh the token right before sending, and retry exactly once+ // with another fresh token if the server still rejects it —+ // mirroring what the fetch interceptor in csrf-refresh.js does+ // for non-XHR requests.+ try {+ data = await sendUpload(await getUploadCsrfToken());+ } catch (uploadError) {+ if (!uploadError.isCsrfRejection) throw uploadError;+ console.warn(`[Upload] CSRF rejection for ${fileItem.file.name}; refreshing token and retrying once.`);+ fileItem.progress = 5;+ data = await sendUpload(await getUploadCsrfToken());+ }}// Upload succeeded - recording is now on the serverdiff --git a/static/js/modules/db/sliced-file-upload.js b/static/js/modules/db/sliced-file-upload.jsnew file mode 100644index 00000000..cb7c447b--- /dev/null+++ b/static/js/modules/db/sliced-file-upload.js@@ -0,0 +1,219 @@+/**+ * Sliced upload of a file the user already has on disk.+ *+ * Speakr's normal upload is one multipart POST of the whole file, so a+ * reverse proxy with a body-size limit below the file size (Cloudflare's+ * 100 MB, for instance) rejects it before Speakr sees it. This module+ * sends the same file as fixed-size byte slices through the+ * recording-session endpoints instead, then POSTs the upload form to+ * /finalize-upload, which byte-joins the slices server-side and ingests+ * the result exactly as POST /upload would.+ *+ * SLICE_BYTES mirrors the server's RECORDING_SESSION_MAX_CHUNK_BYTES+ * default; the create response is authoritative, so a deployment that+ * lowered that env var lowers the slice size with it.+ *+ * It deliberately does NOT reuse server-recording-sessions.js's+ * createSession / abortSession: those maintain the recorder's+ * localStorage crash-recovery marker, which a file upload must not+ * touch. The slice POSTs go through XMLHttpRequest rather than fetch so+ * the progress bar moves within a slice and not just between slices.+ *+ * File slices are Blob views, so nothing here holds the file in memory:+ * a multi-gigabyte upload costs one slice's worth of buffering in the+ * browser's network stack.+ */++import { getUploadCsrfToken, isCsrfRejection } from '../csrf.js';+import { computeUploadTimeout } from '../utils/upload-timeout.js';++const SESSION_BASE = '/upload/session';++export const SLICE_BYTES = 16 * 1024 * 1024;++const MAX_SLICE_ATTEMPTS = 6;+const BASE_RETRY_MS = 1000;+const MAX_RETRY_MS = 30000;++const RETRYABLE_STATUSES = new Set([408, 425, 429, 500, 502, 503, 504]);++/** True when the file is large enough that slicing it is worth a session. */+export function shouldSliceUpload(file) {+ return !!file && file.size > SLICE_BYTES;+}++const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));++const retryDelay = (attempts) =>+ Math.min(MAX_RETRY_MS, BASE_RETRY_MS * Math.pow(2, attempts - 1));++function parseJson(text) {+ try { return JSON.parse(text); } catch (_) { return null; }+}++/**+ * POST via XHR, resolving for every HTTP status so callers can decide+ * what is retryable. Rejects only on a network error or timeout, where+ * there is no status and a retry is the right answer.+ */+function xhrPost(url, body, { token, timeoutMs, onProgress, contentType, onXhr }) {+ return new Promise((resolve, reject) => {+ const xhr = new XMLHttpRequest();+ xhr.open('POST', url);+ if (token) xhr.setRequestHeader('X-CSRFToken', token);+ if (contentType) xhr.setRequestHeader('Content-Type', contentType);+ xhr.timeout = timeoutMs;+ if (onProgress) {+ xhr.upload.onprogress = (e) => {+ if (e.lengthComputable) onProgress(e.loaded, e.total);+ };+ }+ xhr.onload = () => resolve({ status: xhr.status, text: xhr.responseText });+ xhr.onerror = () => reject(new Error('Network error during upload'));+ xhr.ontimeout = () => reject(new Error('Upload timed out'));+ xhr.onabort = () => reject(new Error('Upload cancelled'));+ if (onXhr) onXhr(xhr);+ xhr.send(body);+ });+}++async function createFileSession(file, token) {+ const response = await fetch(SESSION_BASE, {+ method: 'POST',+ headers: { 'Content-Type': 'application/json', 'X-CSRFToken': token },+ credentials: 'same-origin',+ body: JSON.stringify({ filename: file.name, mime_type: file.type || '' }),+ });+ const body = parseJson(await response.text());+ if (response.status !== 201 || !body?.session_id) {+ throw new Error(body?.error || `Could not open upload session (HTTP ${response.status})`);+ }+ if (!body.upload_filename) {+ throw new Error('Server does not support sliced uploads');+ }+ return body;+}++async function deleteFileSession(sessionId, token) {+ try {+ await fetch(`${SESSION_BASE}/${encodeURIComponent(sessionId)}`, {+ method: 'DELETE',+ credentials: 'same-origin',+ headers: { 'X-CSRFToken': token },+ });+ } catch (error) {+ console.warn('[SlicedUpload] Could not abort session:', error);+ }+}++/**+ * Send one slice, retrying transport failures and refreshing the CSRF+ * token when the server rejects it. Returns the index the server expects+ * next, which is normally `index + 1` but can jump when a 409 reveals+ * the server already holds slices whose responses we lost.+ */+async function sendSlice(sessionId, index, blob, tokenRef, options) {+ const url = `${SESSION_BASE}/${encodeURIComponent(sessionId)}/chunks/${index}`;+ for (let attempt = 1; ; attempt++) {+ let result;+ try {+ result = await xhrPost(url, blob, {+ token: tokenRef.token,+ timeoutMs: computeUploadTimeout(blob.size),+ contentType: 'application/octet-stream',+ onProgress: options.onSliceProgress,+ onXhr: options.onXhr,+ });+ } catch (transportError) {+ if (attempt >= MAX_SLICE_ATTEMPTS) throw transportError;+ await sleep(retryDelay(attempt));+ continue;+ }++ if (result.status === 204) return index + 1;++ const body = parseJson(result.text);++ if (result.status === 409 && Number.isInteger(body?.expected_chunk_index)) {+ return body.expected_chunk_index;+ }++ if (isCsrfRejection(result.status, result.text) && attempt < MAX_SLICE_ATTEMPTS) {+ tokenRef.token = await getUploadCsrfToken();+ continue;+ }++ const error = new Error(sliceRejectionMessage(index, blob.size, result, body));+ error.status = result.status;+ if (!RETRYABLE_STATUSES.has(result.status) || attempt >= MAX_SLICE_ATTEMPTS) throw error;+ await sleep(retryDelay(attempt));+ }+}++/**+ * A 413 on a slice is the reverse proxy, not Speakr: its body limit is+ * below the slice size. Naming it the way a proxy does lets the upload+ * composable's error mapping give the user the client_max_body_size+ * guidance instead of a generic processing error.+ */+function sliceRejectionMessage(index, sliceSize, result, body) {+ if (result.status === 413 && !body?.error) {+ const megabytes = Math.round(sliceSize / (1024 * 1024));+ return `Request entity too large: the reverse proxy rejected a ${megabytes} MB upload slice`;+ }+ return body?.error || `Slice ${index} rejected (HTTP ${result.status})`;+}++async function finalize(sessionId, formData, tokenRef, options) {+ const url = `${SESSION_BASE}/${encodeURIComponent(sessionId)}/finalize-upload`;+ for (let attempt = 1; ; attempt++) {+ const result = await xhrPost(url, formData, {+ token: tokenRef.token,+ timeoutMs: options.finalizeTimeoutMs,+ onXhr: options.onXhr,+ });+ const body = parseJson(result.text);+ if (result.status === 202 && body?.id) return body;+ if (isCsrfRejection(result.status, result.text) && attempt === 1) {+ tokenRef.token = await getUploadCsrfToken();+ continue;+ }+ throw new Error(body?.error+ || `Upload could not be finalized (HTTP ${result.status})`);+ }+}++/**+ * Upload `file` in slices and finalize with `formData` (the same form the+ * single-shot path builds, minus the file part). Resolves to the created+ * recording, matching what POST /upload returns.+ *+ * `onProgress` is called with a 0..1 fraction of bytes accepted by the+ * server. `onXhr` receives each in-flight XHR so a caller can cancel.+ */+export async function uploadFileInSlices(file, formData, options = {}) {+ const tokenRef = { token: await getUploadCsrfToken() };+ const session = await createFileSession(file, tokenRef.token);+ const sliceBytes = Math.min(SLICE_BYTES, session.max_chunk_bytes || SLICE_BYTES);+ const onProgress = options.onProgress || (() => {});++ try {+ let index = 1;+ while ((index - 1) * sliceBytes < file.size) {+ const start = (index - 1) * sliceBytes;+ const end = Math.min(start + sliceBytes, file.size);+ index = await sendSlice(session.session_id, index, file.slice(start, end), tokenRef, {+ onXhr: options.onXhr,+ onSliceProgress: (loaded) => onProgress(Math.min(1, (start + loaded) / file.size)),+ });+ onProgress(Math.min(1, ((index - 1) * sliceBytes) / file.size));+ }+ return await finalize(session.session_id, formData, tokenRef, {+ finalizeTimeoutMs: computeUploadTimeout(file.size),+ onXhr: options.onXhr,+ });+ } catch (error) {+ await deleteFileSession(session.session_id, tokenRef.token);+ throw error;+ }+}diff --git a/static/js/modules/db/sliced-file-upload.test.js b/static/js/modules/db/sliced-file-upload.test.jsnew file mode 100644index 00000000..3cd0d3ed--- /dev/null+++ b/static/js/modules/db/sliced-file-upload.test.js@@ -0,0 +1,225 @@+/**+ * Vitest tests for the sliced file-upload transport.+ *+ * The interesting code is the slice loop: fixed-size offsets, in-order+ * indexes, retry, 409 resync, and cleaning up the session when the upload+ * cannot continue. HTTP is exercised through fetch and XMLHttpRequest+ * mocks.+ */++import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';+import { SLICE_BYTES, shouldSliceUpload, uploadFileInSlices } from './sliced-file-upload.js';++const MB = 1024 * 1024;++function fakeFile(size, name = 'talk.mp4', type = 'video/mp4') {+ return {+ name,+ type,+ size,+ slice: (start, end) => ({ size: end - start, start, end }),+ };+}++let sent;+let sliceResponses;+let finalizeResponse;+let fetchCalls;++function _installMocks() {+ sent = [];+ fetchCalls = [];+ sliceResponses = [];+ finalizeResponse = { status: 202, text: JSON.stringify({ id: 42, title: 'talk' }) };++ global.window = { csrfManager: { refreshToken: async () => 'fresh-token' } };+ global.document = { querySelector: () => null };++ global.fetch = vi.fn(async (url, options) => {+ fetchCalls.push({ url, options });+ if (options.method === 'POST') {+ return {+ status: 201,+ text: async () => JSON.stringify({+ session_id: 'sess-1',+ upload_filename: JSON.parse(options.body).filename,+ max_chunk_bytes: SLICE_BYTES,+ }),+ };+ }+ return { status: 204, text: async () => '' };+ });++ global.XMLHttpRequest = class {+ constructor() {+ this.upload = {};+ this.timeout = 0;+ this.headers = {};+ }++ open(method, url) { this.url = url; }++ setRequestHeader(key, value) { this.headers[key] = value; }++ send(body) {+ const call = { url: this.url, body, headers: this.headers };+ sent.push(call);+ const response = this.url.endsWith('/finalize-upload')+ ? finalizeResponse+ : (sliceResponses.shift() || { status: 204, text: '' });+ queueMicrotask(() => {+ if (response.networkError) {+ this.onerror();+ return;+ }+ if (this.upload.onprogress && body?.size) {+ this.upload.onprogress({ lengthComputable: true, loaded: body.size, total: body.size });+ }+ this.status = response.status;+ this.responseText = response.text ?? '';+ this.onload();+ });+ }+ };+}++const slicePosts = () => sent.filter((call) => !call.url.endsWith('/finalize-upload'));++describe('shouldSliceUpload', () => {+ it('leaves files that fit one request on the single-shot path', () => {+ expect(shouldSliceUpload(fakeFile(SLICE_BYTES))).toBe(false);+ expect(shouldSliceUpload(fakeFile(SLICE_BYTES + 1))).toBe(true);+ expect(shouldSliceUpload(null)).toBe(false);+ });+});++describe('uploadFileInSlices', () => {+ beforeEach(() => { _installMocks(); });+ afterEach(() => {+ vi.restoreAllMocks();+ vi.useRealTimers();+ delete global.fetch;+ delete global.XMLHttpRequest;+ delete global.window;+ delete global.document;+ });++ it('opens a session for the filename, sends whole slices in order, and finalizes', async () => {+ const file = fakeFile(40 * MB);+ const form = new Map([['title', 'Sliced']]);+ const progress = [];++ const result = await uploadFileInSlices(file, form, {+ onProgress: (fraction) => progress.push(fraction),+ });++ expect(JSON.parse(fetchCalls[0].options.body)).toEqual({+ filename: 'talk.mp4',+ mime_type: 'video/mp4',+ });+ expect(slicePosts().map((call) => call.url)).toEqual([+ '/upload/session/sess-1/chunks/1',+ '/upload/session/sess-1/chunks/2',+ '/upload/session/sess-1/chunks/3',+ ]);+ expect(slicePosts().map((call) => call.body.size)).toEqual([SLICE_BYTES, SLICE_BYTES, 8 * MB]);+ expect(slicePosts()[0].headers['X-CSRFToken']).toBe('fresh-token');++ const finalizeCall = sent[sent.length - 1];+ expect(finalizeCall.url).toBe('/upload/session/sess-1/finalize-upload');+ expect(finalizeCall.body).toBe(form);+ expect(result).toEqual({ id: 42, title: 'talk' });+ expect(progress[progress.length - 1]).toBe(1);+ });++ it('honours a lower max_chunk_bytes from the server', async () => {+ global.fetch = vi.fn(async (url, options) => ({+ status: 201,+ text: async () => JSON.stringify({+ session_id: 'sess-1',+ upload_filename: 'talk.mp4',+ max_chunk_bytes: 4 * MB,+ }),+ }));++ await uploadFileInSlices(fakeFile(10 * MB), new Map(), {});++ expect(slicePosts().map((call) => call.body.size)).toEqual([4 * MB, 4 * MB, 2 * MB]);+ });++ it('resends the same offset after a network error', async () => {+ vi.useFakeTimers();+ sliceResponses = [{ networkError: true }];++ const upload = uploadFileInSlices(fakeFile(20 * MB), new Map(), {});+ await vi.advanceTimersByTimeAsync(2000);+ await upload;++ expect(slicePosts().map((call) => [call.url, call.body.start])).toEqual([+ ['/upload/session/sess-1/chunks/1', 0],+ ['/upload/session/sess-1/chunks/1', 0],+ ['/upload/session/sess-1/chunks/2', SLICE_BYTES],+ ]);+ });++ it('skips ahead when a 409 says the server already holds the slice', async () => {+ sliceResponses = [{ status: 409, text: JSON.stringify({ expected_chunk_index: 3 }) }];++ await uploadFileInSlices(fakeFile(40 * MB), new Map(), {});++ expect(slicePosts().map((call) => [call.url, call.body.start])).toEqual([+ ['/upload/session/sess-1/chunks/1', 0],+ ['/upload/session/sess-1/chunks/3', 2 * SLICE_BYTES],+ ]);+ });++ it('gives up on a rejected slice and deletes the half-uploaded session', async () => {+ sliceResponses = [+ { status: 204, text: '' },+ { status: 413, text: JSON.stringify({ error: 'Chunk exceeds max_chunk_bytes' }) },+ ];++ await expect(uploadFileInSlices(fakeFile(40 * MB), new Map(), {}))+ .rejects.toThrow('Chunk exceeds max_chunk_bytes');++ expect(slicePosts()).toHaveLength(2);+ expect(fetchCalls[fetchCalls.length - 1]).toMatchObject({+ url: '/upload/session/sess-1',+ options: { method: 'DELETE' },+ });+ });++ it('blames the reverse proxy for a 413 with no Speakr error body', async () => {+ sliceResponses = [{ status: 413, text: '<html><head><title>413 Request Entity Too Large</title></head></html>' }];++ await expect(uploadFileInSlices(fakeFile(40 * MB), new Map(), {}))+ .rejects.toThrow('Request entity too large: the reverse proxy rejected a 16 MB upload slice');+ });++ it('retries a slice with a fresh token when the server rejects the CSRF one', async () => {+ sliceResponses = [{ status: 400, text: 'The CSRF token has expired.' }];++ await uploadFileInSlices(fakeFile(20 * MB), new Map(), {});++ expect(slicePosts().map((call) => call.body.start)).toEqual([0, 0, SLICE_BYTES]);+ });++ it('refuses to upload when the server does not understand sliced uploads', async () => {+ global.fetch = vi.fn(async () => ({+ status: 201,+ text: async () => JSON.stringify({ session_id: 'sess-1', mime_type: 'audio/webm' }),+ }));++ await expect(uploadFileInSlices(fakeFile(20 * MB), new Map(), {}))+ .rejects.toThrow('does not support sliced uploads');+ expect(sent).toHaveLength(0);+ });++ it('surfaces the finalize error and deletes the session', async () => {+ finalizeResponse = { status: 409, text: JSON.stringify({ error: 'Slices on disk do not match the session' }) };++ await expect(uploadFileInSlices(fakeFile(20 * MB), new Map(), {}))+ .rejects.toThrow('Slices on disk do not match the session');+ expect(fetchCalls[fetchCalls.length - 1].options.method).toBe('DELETE');+ });+});diff --git a/tests/test_sliced_file_upload.py b/tests/test_sliced_file_upload.pynew file mode 100644index 00000000..f3ebe163--- /dev/null+++ b/tests/test_sliced_file_upload.py@@ -0,0 +1,305 @@+"""End-to-end tests for sliced uploads of files already on disk.++A sliced upload reuses the recording-session endpoints to carry a file in+fixed-size byte slices past a reverse proxy whose body limit is below the+file size, then finalizes through /finalize-upload, which byte-joins the+slices and runs them through the same ingestion pipeline POST /upload+uses.++The external effects of that pipeline (codec probe, conversion, storage,+job queue) are mocked with the harness the /upload tests already use, so+these tests assert the parts that are specific to slicing: the session is+recognisable as a file upload, the reassembled bytes are the original+bytes, the recorder's stitch path is kept away from them, and an+abandoned one is expired rather than assembled into a truncated+recording.+"""++import hashlib+import os+import shutil+import sys+import tempfile+import uuid+from datetime import datetime, timedelta++sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))++from src.app import app, db+from src.models import Recording, RecordingSession+from src.api.recording_sessions import cleanup_expired_sessions++from tests.test_cov_recordings_write import _cleanup, _mk_user, _upload_mocks+from tests.test_recording_sessions import _login++app.config["WTF_CSRF_ENABLED"] = False++SLICE_BYTES = 4096+++def _tmp_upload_folder():+ return tempfile.mkdtemp(prefix="speakr-test-sliced-")+++def _payload(size):+ return bytes(range(256)) * (size // 256) + b"\xab" * (size % 256)+++def _open_session(client, filename="talk.mp4", mime_type="video/mp4"):+ response = client.post("/upload/session", json={"filename": filename, "mime_type": mime_type})+ assert response.status_code == 201, response.data+ return response.get_json()+++def _send_slices(client, session_id, payload, slice_bytes=SLICE_BYTES):+ for index, start in enumerate(range(0, len(payload), slice_bytes), start=1):+ response = client.post(+ f"/upload/session/{session_id}/chunks/{index}",+ data=payload[start:start + slice_bytes],+ content_type="application/octet-stream",+ )+ assert response.status_code == 204, (index, response.data)+++def test_create_session_with_filename_marks_it_a_file_upload():+ upload_folder = _tmp_upload_folder()+ with app.app_context():+ app.config["UPLOAD_FOLDER"] = upload_folder+ user = _mk_user("slice_create")+ client = app.test_client()+ _login(client, user)++ body = _open_session(client)+ assert body["upload_filename"] == "talk.mp4"+ assert db.session.get(RecordingSession, body["session_id"]).upload_filename == "talk.mp4"++ client.delete(f"/upload/session/{body['session_id']}")+ _cleanup(user)+ shutil.rmtree(upload_folder, ignore_errors=True)+++def test_create_session_with_filename_skips_the_recorder_mime_whitelist():+ """A user's own file can be any container ffmpeg reads, so the whitelist+ that guards the recorder's stitch path must not gate these sessions."""+ upload_folder = _tmp_upload_folder()+ with app.app_context():+ app.config["UPLOAD_FOLDER"] = upload_folder+ user = _mk_user("slice_mime")+ client = app.test_client()+ _login(client, user)++ rejected = client.post("/upload/session", json={"mime_type": "video/x-matroska"})+ assert rejected.status_code == 400++ accepted = _open_session(client, filename="lecture.mkv", mime_type="video/x-matroska")+ assert accepted["mime_type"] == "video/x-matroska"++ client.delete(f"/upload/session/{accepted['session_id']}")+ _cleanup(user)+ shutil.rmtree(upload_folder, ignore_errors=True)+++def test_finalize_upload_ingests_the_reassembled_original_bytes():+ upload_folder = _tmp_upload_folder()+ with app.app_context():+ app.config["UPLOAD_FOLDER"] = upload_folder+ user = _mk_user("slice_ingest")+ client = app.test_client()+ _login(client, user)++ payload = _payload(SLICE_BYTES * 3 + 17)+ session_id = _open_session(client, filename="sample.mp3", mime_type="audio/mpeg")["session_id"]+ _send_slices(client, session_id, payload)++ staging = os.path.join(upload_folder, f"stg_{uuid.uuid4().hex[:6]}")+ with _upload_mocks(staging) as (storage, enqueue):+ response = client.post(+ f"/upload/session/{session_id}/finalize-upload",+ data={"title": "Sliced", "notes": "n"},+ content_type="multipart/form-data",+ )++ assert response.status_code == 202, response.data+ recording = db.session.get(Recording, response.get_json()["id"])+ assert recording.title == "Sliced"+ assert recording.original_filename == "sample.mp3"+ assert recording.file_hash == hashlib.sha256(payload).hexdigest()+ assert enqueue.called++ session = db.session.get(RecordingSession, session_id)+ assert session.status == "finalized"+ assert session.finalized_recording_id == recording.id+ assert not os.path.isdir(os.path.join(upload_folder, "_sessions", session_id))++ _cleanup(recording, user)+ shutil.rmtree(upload_folder, ignore_errors=True)+++def test_finalize_upload_carries_the_upload_form_through_ingestion():+ upload_folder = _tmp_upload_folder()+ with app.app_context():+ app.config["UPLOAD_FOLDER"] = upload_folder+ user = _mk_user("slice_form")+ client = app.test_client()+ _login(client, user)++ session_id = _open_session(client, filename="sample.mp3", mime_type="audio/mpeg")["session_id"]+ _send_slices(client, session_id, _payload(SLICE_BYTES * 2))++ staging = os.path.join(upload_folder, f"stg_{uuid.uuid4().hex[:6]}")+ with _upload_mocks(staging) as (storage, enqueue):+ response = client.post(+ f"/upload/session/{session_id}/finalize-upload",+ data={"language": "fr", "min_speakers": "2", "max_speakers": "4"},+ content_type="multipart/form-data",+ )++ assert response.status_code == 202, response.data+ params = enqueue.call_args.kwargs["params"]+ assert params["language"] == "fr"+ assert params["min_speakers"] == 2+ assert params["max_speakers"] == 4++ _cleanup(db.session.get(Recording, response.get_json()["id"]), user)+ shutil.rmtree(upload_folder, ignore_errors=True)+++def test_finalize_upload_rejects_a_slice_count_that_disagrees_with_disk():+ """A slice lost between the write and the bookkeeping commit would+ otherwise be silently ingested as a truncated file."""+ upload_folder = _tmp_upload_folder()+ with app.app_context():+ app.config["UPLOAD_FOLDER"] = upload_folder+ user = _mk_user("slice_short")+ client = app.test_client()+ _login(client, user)++ session_id = _open_session(client, filename="sample.mp3", mime_type="audio/mpeg")["session_id"]+ _send_slices(client, session_id, _payload(SLICE_BYTES * 2))+ os.remove(os.path.join(upload_folder, "_sessions", session_id, "chunk-000002.bin"))++ response = client.post(+ f"/upload/session/{session_id}/finalize-upload",+ data={},+ content_type="multipart/form-data",+ )+ assert response.status_code == 409+ assert response.get_json()["expected_chunk_index"] == 2++ client.delete(f"/upload/session/{session_id}")+ _cleanup(user)+ shutil.rmtree(upload_folder, ignore_errors=True)+++def test_the_two_finalize_routes_reject_each_others_sessions():+ """The recorder's finalize remuxes and sniffs container headers per+ chunk, which would rewrite or mis-split a byte-sliced file; the sliced+ finalize skips the stitch a paused recording needs."""+ upload_folder = _tmp_upload_folder()+ with app.app_context():+ app.config["UPLOAD_FOLDER"] = upload_folder+ user = _mk_user("slice_routes")+ client = app.test_client()+ _login(client, user)++ file_session = _open_session(client, filename="sample.mp3", mime_type="audio/mpeg")["session_id"]+ _send_slices(client, file_session, _payload(SLICE_BYTES))+ stitch_refusal = client.post(f"/upload/session/{file_session}/finalize", json={})+ assert stitch_refusal.status_code == 409+ assert "finalize-upload" in stitch_refusal.get_json()["error"]++ recorder_session = client.post(+ "/upload/session", json={"mime_type": "audio/webm"}).get_json()["session_id"]+ client.post(f"/upload/session/{recorder_session}/chunks/1",+ data=b"\x00" * 64, content_type="application/octet-stream")+ upload_refusal = client.post(+ f"/upload/session/{recorder_session}/finalize-upload",+ data={}, content_type="multipart/form-data")+ assert upload_refusal.status_code == 409+ assert "/finalize" in upload_refusal.get_json()["error"]++ client.delete(f"/upload/session/{file_session}")+ client.delete(f"/upload/session/{recorder_session}")+ _cleanup(user)+ shutil.rmtree(upload_folder, ignore_errors=True)+++def test_cleanup_expires_an_abandoned_sliced_upload_instead_of_assembling_it():+ """Abandoned recorder sessions are auto-finalized because the browser+ holds the only copy of that audio. A sliced upload's original is still+ on the user's disk, and its slices are a truncated container, so+ assembling one would land a broken recording in the library."""+ upload_folder = _tmp_upload_folder()+ with app.app_context():+ app.config["UPLOAD_FOLDER"] = upload_folder+ user = _mk_user("slice_expire")+ client = app.test_client()+ _login(client, user)++ session_id = _open_session(client, filename="sample.mp3", mime_type="audio/mpeg")["session_id"]+ _send_slices(client, session_id, _payload(SLICE_BYTES))++ session = db.session.get(RecordingSession, session_id)+ session.last_seen_at = datetime.utcnow() - timedelta(hours=48)+ db.session.commit()++ cleanup_expired_sessions(app)++ db.session.expire_all()+ assert db.session.get(RecordingSession, session_id).status == "expired"+ assert not os.path.isdir(os.path.join(upload_folder, "_sessions", session_id))+ assert Recording.query.filter_by(user_id=user.id).count() == 0++ _cleanup(user)+ shutil.rmtree(upload_folder, ignore_errors=True)+++def test_a_replayed_finalize_upload_does_not_ingest_the_file_twice():+ """Finalize is synchronous, so a client that retries it after a lost+ response would otherwise land the same file in the library twice."""+ upload_folder = _tmp_upload_folder()+ with app.app_context():+ app.config["UPLOAD_FOLDER"] = upload_folder+ user = _mk_user("slice_replay")+ client = app.test_client()+ _login(client, user)++ session_id = _open_session(client, filename="sample.mp3", mime_type="audio/mpeg")["session_id"]+ _send_slices(client, session_id, _payload(SLICE_BYTES))++ staging = os.path.join(upload_folder, f"stg_{uuid.uuid4().hex[:6]}")+ with _upload_mocks(staging):+ first = client.post(f"/upload/session/{session_id}/finalize-upload",+ data={}, content_type="multipart/form-data")+ second = client.post(f"/upload/session/{session_id}/finalize-upload",+ data={}, content_type="multipart/form-data")++ assert first.status_code == 202, first.data+ assert second.status_code == 409+ assert Recording.query.filter_by(user_id=user.id).count() == 1++ _cleanup(db.session.get(Recording, first.get_json()["id"]), user)+ shutil.rmtree(upload_folder, ignore_errors=True)+++def test_another_user_cannot_finalize_someone_elses_sliced_upload():+ upload_folder = _tmp_upload_folder()+ with app.app_context():+ app.config["UPLOAD_FOLDER"] = upload_folder+ owner = _mk_user("slice_owner")+ intruder = _mk_user("slice_intruder")+ client = app.test_client()+ _login(client, owner)+ session_id = _open_session(client, filename="sample.mp3", mime_type="audio/mpeg")["session_id"]+ _send_slices(client, session_id, _payload(SLICE_BYTES))++ other = app.test_client()+ _login(other, intruder)+ response = other.post(+ f"/upload/session/{session_id}/finalize-upload",+ data={}, content_type="multipart/form-data")+ assert response.status_code == 404++ client.delete(f"/upload/session/{session_id}")+ _cleanup(owner, intruder)+ shutil.rmtree(upload_folder, ignore_errors=True)