Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

fix([issue-11659]): preserve voice connections for supported media sizes (#11692)

[._.]/68462350bc6f
diff --git a/client/src/lib/README.md b/client/src/lib/README.md
index d55c568cbc..2be60f2837 100644
--- a/client/src/lib/README.md
+++ b/client/src/lib/README.md
@@ -153,7 +153,7 @@ grep -i "what you want to do" client/src/lib/README.md
| `avatarStyles.js` | Re-export of `server/lib/avatarStyles.js` — the CoS avatar-style vocabulary (#6253): `AVATAR_STYLES`, `AVATAR_STYLE_IDS`, `AVATAR_STYLE_LABELS`, `WEBGL_AVATAR_STYLE_IDS`. The picker labels, the lazy-load map / WebGL-stage set, and the server's `avatarStyle` zod enum all read the one list. |
| `assessmentTuningNotice.js` | `tuningNoticeChip(entry)` — the short form of how a measured local-model assessment says its launch configuration did NOT take effect. `tuningApplied === false` covers two opposite cases and `tuningKey` separates them: a TUNED run whose knobs never reached the daemon (`tuning not applied`), versus an UNTUNED run PortOS could not put back on backend defaults (`not at defaults`) — #4759. Shared by `ModelThroughputReport.jsx`'s table cell and `LocalModelAssessments.jsx`'s result toast; the long form is the server's exclusion reason in `getAssessmentReport`. |
| `callAudioBridge.js` | Pure device/format helpers for the call-host page (`pages/VoiceCallHost.jsx`, #5308 + #5311) — shared by its Call mode (FaceTime Audio) and its Capture system audio mode (meeting transcription). `missingCallHostApis(scope)` lists every required browser API this browser lacks (MediaStreamTrackProcessor/Generator, AudioWorklet, `enumerateDevices`, `setSinkId`, Web Locks) so the page names all the gaps at once instead of one per reload; `findAudioDevice(devices, label, kind)` matches a device label EXACTLY after trim+case-fold (a substring match would resolve "BlackHole 1" to the 16ch device, indistinguishable from a dead call); `describeDeviceProblem(devices, {inputLabel, outputLabel})` returns the one blocking message or `null`, and deliberately blames MICROPHONE PERMISSION rather than a missing driver when every label comes back blank — an ungranted tab enumerates unlabeled devices, and "reinstall BlackHole" would send the user to fix the wrong thing; `outputLabel` is OPTIONAL — capture mode only listens, so omitting it skips the output-device check entirely rather than reporting a device named "undefined" as missing. Format side: `downmixToMono(channels)` (BlackHole 16ch carries the call on all channels; truncates to the shortest rather than reading past one), `resampleTo16k(samples, fromRate)` (linear, 48k→16k for whisper), `floatToInt16` (CLAMPED — wrapping would flip a hot sample's sign into an audible click), `rmsLevel` for the input meter, and `buildTestTone` (faded in/out so it cannot click through the call). Plus `CALL_HOST_LOCK` (the Web Lock name that stops a second tab double-answering — shared by both modes, since they read the same physical device) and `CALL_FRAME_SAMPLES`. |
-| `audioRecorder.js` | `startMemoRecording()` → `{ stream, stop, cancel }` handle whose `stop()` resolves to `{ audioBase64, mimeType, peak, durationMs }` (16 kHz mono WAV, base64); the live `stream` lets a tuner tap the same mic. `createStreamAnalyser(stream, { fftSize })` → `{ analyser, context, close }` taps an `AnalyserNode` off an existing stream (no second getUserMedia). Plus `blobToWav16k` (decode a recorded blob → `{ wav, peak }`), `float32ToWav16k(samples, sourceRate)` (resample a raw Float32 PCM buffer, e.g. an AudioWorklet capture, → `{ wav, peak }`; `{ wav: null, peak: 0 }` for empty input), `encodePcmToWav`, `TARGET_SAMPLE_RATE` (16000), `pickRecordingMimeType`, `arrayBufferToBase64`. Shared WAV-encode/resample path for both this standalone memo-capture flow AND the live voice-agent recorder in `services/voiceClient.js`. **`openAnalysisMic({ getUserMedia })`** → `{ stream, processing }` is the ONE way a pitch-analysis surface opens a mic: it requests `ANALYSIS_AUDIO_CONSTRAINTS` (echo cancellation / noise suppression / AGC off — plain booleans, never `{ exact }`, so a browser that can't honor one still opens the mic) and reads `track.getSettings()` back so the UI can say what actually stuck. `readAppliedProcessing(stream)` reports each stage as `true` / `false` / `null` (browser didn't say — never collapse unknown into off) and `hasUnwantedProcessing(p)` gates the warning. Memo capture deliberately keeps the speech-tuned defaults; Whisper wants them. |
+| `audioRecorder.js` | `startMemoRecording()` → `{ stream, stop, cancel }` handle whose `stop()` resolves to `{ audioBase64, mimeType, peak, durationMs }` (16 kHz mono WAV, base64); the live `stream` lets a tuner tap the same mic. `createStreamAnalyser(stream, { fftSize })` → `{ analyser, context, close }` taps an `AnalyserNode` off an existing stream (no second getUserMedia). Plus `blobToWav16k` (decode a recorded blob → `{ wav, peak }`), `float32ToWav16k(samples, sourceRate)` (resample a raw Float32 PCM buffer, e.g. an AudioWorklet capture, → `{ wav, peak }`; `{ wav: null, peak: 0 }` for empty input), `encodePcmToWav`, `MAX_VOICE_AUDIO_BYTES` (8 MiB), `RECORDING_TOO_LONG_MESSAGE`, `TARGET_SAMPLE_RATE` (16000), `pickRecordingMimeType`, `arrayBufferToBase64`. Shared WAV-encode/resample path for both this standalone memo-capture flow AND the live voice-agent recorder in `services/voiceClient.js`. **`openAnalysisMic({ getUserMedia })`** → `{ stream, processing }` is the ONE way a pitch-analysis surface opens a mic: it requests `ANALYSIS_AUDIO_CONSTRAINTS` (echo cancellation / noise suppression / AGC off — plain booleans, never `{ exact }`, so a browser that can't honor one still opens the mic) and reads `track.getSettings()` back so the UI can say what actually stuck. `readAppliedProcessing(stream)` reports each stage as `true` / `false` / `null` (browser didn't say — never collapse unknown into off) and `hasUnwantedProcessing(p)` gates the warning. Memo capture deliberately keeps the speech-tuned defaults; Whisper wants them. |
| `bareUrl.js` | `parseBareUrl(text)` — the normalized URL when a capture's entire text is a URL, else null. Re-exported from `server/lib/bareUrl.js`, the rule the brain-capture route applies. |
| `youtubeUrl.js` | `isYoutubeVideoUrl(text)` / `youtubeVideoId(url)` re-exported from `server/lib/youtubeUrl.js`, so Quick Capture can only offer the ingest path for a URL the server accepts. Also the client-only `INGEST_OPTIONS` — the single table of the three ingest artifacts (`key` / `settingKey` / `fallback` / `label` / `hint`) — plus `defaultIngestOptions()` and `ingestOptionsFromSettings(settings)`. Everything that enumerates the switches derives from that table; add an artifact there, not at each call site. |
| `boundedMap.js` | `evictOldest(map, max, onEvict?)` — evict oldest-first (`map.keys().next().value`) until a Map is within `max`, with an optional per-evicted-key teardown callback for a companion timer map; `ORPHAN_BUFFER_MAX` (64), the shared cap for the orphan-event race-buffers. The one home for the "stash an unmatched event in a `jobId → …` Map, evict the oldest on overflow" idiom shared by `useSceneRenderLifecycle`, `useImageGenQueue`, and `clientErrorReporter`. |
diff --git a/client/src/lib/audioRecorder.js b/client/src/lib/audioRecorder.js
index d0b03c174a..76cdacd0eb 100644
--- a/client/src/lib/audioRecorder.js
+++ b/client/src/lib/audioRecorder.js
@@ -9,8 +9,12 @@
// `blobToWav16k`, `float32ToWav16k`) ARE shared with that module — it imports
// them rather than keeping its own copy.
+import { SOCKET_MEDIA_MAX_BYTES } from '../../../server/lib/socketPayloadLimits.js';
import { resumeAudioContext, acquireAudioSession } from './audioContext.js';
+export const MAX_VOICE_AUDIO_BYTES = SOCKET_MEDIA_MAX_BYTES;
+export const RECORDING_TOO_LONG_MESSAGE = 'Recording too long. Please record a shorter message.';
+
export const TARGET_SAMPLE_RATE = 16000;
// Pick a MediaRecorder mime the browser supports; Safari lands on mp4, others
diff --git a/client/src/pages/FableLoomHostedJoin.jsx b/client/src/pages/FableLoomHostedJoin.jsx
index 6af6458dfc..ee48390a31 100644
--- a/client/src/pages/FableLoomHostedJoin.jsx
+++ b/client/src/pages/FableLoomHostedJoin.jsx
@@ -8,6 +8,7 @@
import { useEffect, useRef, useState } from 'react';
import { io } from 'socket.io-client';
+import { MAX_VOICE_AUDIO_BYTES, RECORDING_TOO_LONG_MESSAGE } from '../lib/audioRecorder.js';
import useMounted from '../hooks/useMounted.js';
import {
AlertCircle,
@@ -28,10 +29,13 @@ export default function FableLoomHostedJoin() {
const [transcript, setTranscript] = useState([]);
const [textInput, setTextInput] = useState('');
const [isRecording, setIsRecording] = useState(false);
+ const [recordingError, setRecordingError] = useState(null);
const mediaRecorderRef = useRef(null);
const pressedRef = useRef(false);
const startGenRef = useRef(0);
+ // Release preserves final recorder callbacks; a newer press retires them.
+ const recordingGenRef = useRef(0);
const mountedRef = useMounted();
const transcriptScrollRef = useRef(null);
const audioPlayerRef = useRef(null);
@@ -128,8 +132,10 @@ export default function FableLoomHostedJoin() {
// Microphone recording controls
const startRecording = async () => {
if (pressedRef.current || mediaRecorderRef.current || turnPhase === 'thinking' || turnPhase === 'speaking' || turnPhase === 'ended') return;
+ setRecordingError(null);
pressedRef.current = true;
const generation = ++startGenRef.current;
+ const recordingGeneration = ++recordingGenRef.current;
let stream;
try {
stream = await navigator.mediaDevices.getUserMedia({ audio: true });
@@ -155,13 +161,21 @@ export default function FableLoomHostedJoin() {
};
recorder.onstop = () => {
+ stopTracks();
+ if (!mountedRef.current || recordingGeneration !== recordingGenRef.current) return;
const audioBlob = new Blob(chunks, { type: recorder.mimeType || 'audio/webm' });
+ if (audioBlob.size > MAX_VOICE_AUDIO_BYTES) {
+ if (mountedRef.current) {
+ setRecordingError(RECORDING_TOO_LONG_MESSAGE);
+ socket?.emit('hosted:turn:abort');
+ }
+ return;
+ }
audioBlob.arrayBuffer().then((buf) => {
- if (mountedRef.current && socket) {
+ if (mountedRef.current && recordingGeneration === recordingGenRef.current && socket) {
socket.emit('hosted:mic:stop', new Uint8Array(buf));
}
});
- stopTracks();
};
recorder.start(100);
@@ -190,6 +204,7 @@ export default function FableLoomHostedJoin() {
};
useEffect(() => () => {
+ ++recordingGenRef.current;
pressedRef.current = false;
++startGenRef.current;
const active = mediaRecorderRef.current;
@@ -222,6 +237,7 @@ export default function FableLoomHostedJoin() {
return (
<main className="h-dvh-screen bg-slate-950 text-white flex flex-col justify-between max-w-md mx-auto relative overflow-y-auto shadow-2xl">
+ {recordingError && <p role="alert" className="px-4 text-sm text-red-400">{recordingError}</p>}
<audio ref={audioPlayerRef} className="hidden" />
{/* Header */}
diff --git a/client/src/pages/FableLoomHostedJoin.test.jsx b/client/src/pages/FableLoomHostedJoin.test.jsx
index 3a30943132..55709f7d04 100644
--- a/client/src/pages/FableLoomHostedJoin.test.jsx
+++ b/client/src/pages/FableLoomHostedJoin.test.jsx
@@ -158,6 +158,73 @@ describe('FableLoomHostedJoin', () => {
vi.unstubAllGlobals();
});
+ it('shows an over-cap recording refusal, aborts the turn, and keeps the guest connected', async () => {
+ const { MAX_VOICE_AUDIO_BYTES, RECORDING_TOO_LONG_MESSAGE } = await import('../lib/audioRecorder.js');
+ window.location.hash = '#session=sess-123&token=tok-abc';
+ const track = { stop: vi.fn() };
+ Object.defineProperty(navigator, 'mediaDevices', {
+ configurable: true,
+ value: { getUserMedia: vi.fn().mockResolvedValue({ getTracks: () => [track] }) },
+ });
+ let recorder;
+ vi.stubGlobal('MediaRecorder', class {
+ constructor() { this.state = 'inactive'; this.mimeType = 'audio/webm'; recorder = this; }
+ start() { this.state = 'recording'; }
+ stop() { this.state = 'inactive'; this.onstop?.(); }
+ });
+ try {
+ render(<FableLoomHostedJoin />);
+ const button = screen.getByRole('button', { name: /hold talk/i });
+ fireEvent.pointerDown(button);
+ await waitFor(() => expect(recorder?.state).toBe('recording'));
+ recorder.ondataavailable({ data: new Blob([new Uint8Array(MAX_VOICE_AUDIO_BYTES + 1)]) });
+ fireEvent.pointerUp(button);
+ expect(screen.getByRole('alert')).toHaveTextContent(RECORDING_TOO_LONG_MESSAGE);
+ expect(mockSocket.emit).not.toHaveBeenCalledWith('hosted:mic:stop', expect.anything());
+ expect(mockSocket.emit).toHaveBeenCalledWith('hosted:turn:abort');
+ expect(mockSocket.disconnect).not.toHaveBeenCalled();
+ expect(track.stop).toHaveBeenCalledOnce();
+ } finally {
+ vi.unstubAllGlobals();
+ }
+ });
+
+ it('ignores an old oversized stop callback after a newer recording starts', async () => {
+ const { MAX_VOICE_AUDIO_BYTES } = await import('../lib/audioRecorder.js');
+ window.location.hash = '#session=sess-123&token=tok-abc';
+ Object.defineProperty(navigator, 'mediaDevices', {
+ configurable: true,
+ value: { getUserMedia: vi.fn(async () => ({ getTracks: () => [{ stop: vi.fn() }] })) },
+ });
+ const recorders = [];
+ vi.stubGlobal('MediaRecorder', class {
+ constructor() { this.state = 'inactive'; this.mimeType = 'audio/webm'; recorders.push(this); }
+ start() { this.state = 'recording'; }
+ stop() { this.state = 'inactive'; }
+ });
+ try {
+ render(<FableLoomHostedJoin />);
+ const button = screen.getByRole('button', { name: /hold talk/i });
+ fireEvent.pointerDown(button);
+ await waitFor(() => expect(recorders).toHaveLength(1));
+ recorders[0].ondataavailable({ data: new Blob([new Uint8Array(MAX_VOICE_AUDIO_BYTES + 1)]) });
+ fireEvent.pointerUp(button);
+ fireEvent.pointerDown(button);
+ await waitFor(() => expect(recorders).toHaveLength(2));
+ act(() => recorders[0].onstop());
+ expect(mockSocket.emit).not.toHaveBeenCalledWith('hosted:turn:abort');
+ expect(screen.queryByRole('alert')).not.toBeInTheDocument();
+ recorders[1].ondataavailable({ data: new Blob(['new recording']) });
+ fireEvent.pointerUp(button);
+ act(() => recorders[1].onstop());
+ await waitFor(() => expect(mockSocket.emit).toHaveBeenCalledWith('hosted:mic:stop', expect.any(Uint8Array)));
+ const bytes = mockSocket.emit.mock.calls.find(([event]) => event === 'hosted:mic:stop')[1];
+ expect(new TextDecoder().decode(bytes)).toBe('new recording');
+ } finally {
+ vi.unstubAllGlobals();
+ }
+ });
+
it('keeps audio chunks separate across two press and release cycles', async () => {
window.location.hash = '#session=sess-123&token=tok-abc';
const tracks = [{ stop: vi.fn() }, { stop: vi.fn() }];
diff --git a/client/src/services/voiceClient.js b/client/src/services/voiceClient.js
index d93fa4159a..f394068708 100644
--- a/client/src/services/voiceClient.js
+++ b/client/src/services/voiceClient.js
@@ -9,7 +9,7 @@ import socket from './socket';
import { subscribeVisibility } from '../hooks/useVisibilityEvent.js';
import { sleep } from '../utils/sleep.js';
import { resumeAudioContext, acquireAudioSession } from '../lib/audioContext.js';
-import { blobToWav16k, float32ToWav16k } from '../lib/audioRecorder.js';
+import { blobToWav16k, float32ToWav16k, MAX_VOICE_AUDIO_BYTES, RECORDING_TOO_LONG_MESSAGE } from '../lib/audioRecorder.js';
import { ECHO_WINDOW_MS, MIN_TOKENS_FOR_ECHO_CHECK, MIN_SHARED_TRIGRAMS, tokenize, trigramsOf } from '../../../server/lib/voiceEcho.js';
// iOS audio-session claims held while a mic stream is open — one per capture
@@ -530,6 +530,10 @@ export const stopCapture = async ({ submit = true } = {}) => {
const { wav, peak } = await blobToWav16k(blob);
if (!isOwnerCurrent(ownerGeneration) || recorderGeneration !== stoppedGeneration) return null;
+ if (wav.byteLength > MAX_VOICE_AUDIO_BYTES) {
+ refuseLongRecording();
+ return null;
+ }
socket.emit('voice:turn', { audio: wav, mimeType: 'audio/wav' });
return { mimeType: 'audio/wav', size: wav.byteLength, sourceSize: blob.size, peak };
};
@@ -587,13 +591,22 @@ export const resetConversation = () => {
export const isCapturing = () => recorder !== null;
+const localVoiceErrorHandlers = new Set();
+const refuseLongRecording = () => {
+ for (const handler of localVoiceErrorHandlers) handler({ stage: 'turn', message: RECORDING_TOO_LONG_MESSAGE });
+};
+
/**
* Subscribe to voice events. Returns an unsubscribe function.
* Events: voice:transcript, voice:llm:delta, voice:llm:done, voice:error, voice:idle
*/
export const onVoiceEvent = (event, handler) => {
socket.on(event, handler);
- return () => socket.off(event, handler);
+ if (event === 'voice:error') localVoiceErrorHandlers.add(handler);
+ return () => {
+ socket.off(event, handler);
+ if (event === 'voice:error') localVoiceErrorHandlers.delete(handler);
+ };
};
// Persistent, user-authorized screen-capture stream for the voice agent's
@@ -684,7 +697,7 @@ export const captureScreenForVision = async () => {
video.pause();
if (!cctx) return null;
cctx.drawImage(video, 0, 0, w, h);
- // JPEG @ 0.8 keeps the payload well under the server's 16 MB cap even at 4K.
+ // JPEG @ 0.8 keeps normal tab captures below the encoded 8 MiB cap.
return canvas.toDataURL('image/jpeg', 0.8);
};
@@ -693,7 +706,7 @@ export const captureScreenForVision = async () => {
// requestId so the server resolves the matching waiter and a late reply can't
// satisfy a newer capture.
export const sendScreenshotResult = (requestId, dataUrl) => {
- socket.emit('voice:screenshot:result', { requestId, dataUrl: dataUrl || null });
+ socket.emit('voice:screenshot:result', { requestId, dataUrl: typeof dataUrl === 'string' && dataUrl.length <= MAX_VOICE_AUDIO_BYTES ? dataUrl : null });
};
const playPreviewAudio = (arrayBuffer) => enqueuePlay(arrayBuffer);
@@ -869,6 +882,11 @@ const submitUtterance = async (capture) => {
capture.callbacks?.onSubmit?.({ submitted: false, peak });
return;
}
+ if (wav.byteLength > MAX_VOICE_AUDIO_BYTES) {
+ refuseLongRecording();
+ capture.callbacks?.onSubmit?.({ submitted: false, peak });
+ return;
+ }
socket.emit('voice:turn', { audio: wav, mimeType: 'audio/wav' });
capture.callbacks?.onSubmit?.({ submitted: true, peak, size: wav.byteLength });
};
diff --git a/client/src/services/voiceClient.test.js b/client/src/services/voiceClient.test.js
index 63e875f90b..015afd4a12 100644
--- a/client/src/services/voiceClient.test.js
+++ b/client/src/services/voiceClient.test.js
@@ -66,12 +66,15 @@ const { socket, listeners, audio, FakeAudioContext } = vi.hoisted(() => {
return { socket, listeners, audio, FakeAudioContext };
});
-const audioRecorderStub = vi.hoisted(() => ({ blobToWav16k: null }));
+const audioRecorderStub = vi.hoisted(() => ({ blobToWav16k: null, float32ToWav16k: null }));
vi.mock('../lib/audioRecorder.js', async (importOriginal) => {
const actual = await importOriginal();
return {
...actual,
+ float32ToWav16k: (...args) => audioRecorderStub.float32ToWav16k
+ ? audioRecorderStub.float32ToWav16k(...args)
+ : actual.float32ToWav16k(...args),
blobToWav16k: (...args) => audioRecorderStub.blobToWav16k
? audioRecorderStub.blobToWav16k(...args)
: actual.blobToWav16k(...args),
@@ -164,6 +167,7 @@ describe('voice playback cancellation', () => {
audio.sources.length = 0;
socket.emit.mockClear();
audioRecorderStub.blobToWav16k = null;
+ audioRecorderStub.float32ToWav16k = null;
voiceClient.disposeCaptureOwner();
});
@@ -403,6 +407,56 @@ describe('voice playback cancellation', () => {
});
});
+ it('sends capped screenshots and resolves oversized captures as unavailable', async () => {
+ const { MAX_VOICE_AUDIO_BYTES } = await import('../lib/audioRecorder.js');
+ const prefix = 'data:image/jpeg;base64,';
+ const capped = prefix + 'A'.repeat(MAX_VOICE_AUDIO_BYTES - prefix.length);
+ voiceClient.sendScreenshotResult('example-capture', capped);
+ expect(socket.emit).toHaveBeenLastCalledWith('voice:screenshot:result', { requestId: 'example-capture', dataUrl: capped });
+ voiceClient.sendScreenshotResult('example-capture', capped + 'A');
+ expect(socket.emit).toHaveBeenLastCalledWith('voice:screenshot:result', { requestId: 'example-capture', dataUrl: null });
+ });
+
+ it('refuses an over-cap WAV locally through the voice error subscription and accepts the cap', async () => {
+ const previousMediaDevices = Object.getOwnPropertyDescriptor(navigator, 'mediaDevices');
+ const track = { stop: vi.fn() };
+ Object.defineProperty(navigator, 'mediaDevices', {
+ configurable: true,
+ value: { getUserMedia: vi.fn().mockResolvedValue({ getTracks: () => [track] }) },
+ });
+ const previousGlobalRecorder = globalThis.MediaRecorder;
+ globalThis.MediaRecorder = FakeMediaRecorder;
+ const previousRecorder = window.MediaRecorder;
+ window.MediaRecorder = FakeMediaRecorder;
+ const onError = vi.fn();
+ const unsubscribe = voiceClient.onVoiceEvent('voice:error', onError);
+ try {
+ const { MAX_VOICE_AUDIO_BYTES, RECORDING_TOO_LONG_MESSAGE } = await import('../lib/audioRecorder.js');
+ for (const size of [MAX_VOICE_AUDIO_BYTES + 1, MAX_VOICE_AUDIO_BYTES]) {
+ audioRecorderStub.blobToWav16k = vi.fn(async () => ({ wav: new ArrayBuffer(size), peak: 0.5 }));
+ await voiceClient.startCapture();
+ FakeMediaRecorder.last.dispatch('dataavailable', { data: new Blob(['x'.repeat(1000)]) });
+ const result = await voiceClient.stopCapture();
+ if (size > MAX_VOICE_AUDIO_BYTES) {
+ expect(result).toBeNull();
+ expect(socket.emit).not.toHaveBeenCalledWith('voice:turn', expect.anything());
+ expect(onError).toHaveBeenCalledWith({ stage: 'turn', message: RECORDING_TOO_LONG_MESSAGE });
+ } else {
+ expect(result.size).toBe(size);
+ expect(socket.emit).toHaveBeenCalledWith('voice:turn', expect.objectContaining({ audio: expect.any(ArrayBuffer) }));
+ }
+ }
+ expect(track.stop).toHaveBeenCalledTimes(2);
+ } finally {
+ unsubscribe();
+ window.MediaRecorder = previousRecorder;
+ if (previousGlobalRecorder === undefined) delete globalThis.MediaRecorder;
+ else globalThis.MediaRecorder = previousGlobalRecorder;
+ if (previousMediaDevices) Object.defineProperty(navigator, 'mediaDevices', previousMediaDevices);
+ else delete navigator.mediaDevices;
+ }
+ });
+
it('does not emit captured audio when the owner is disposed during WAV conversion', async () => {
const previousMediaDevices = Object.getOwnPropertyDescriptor(navigator, 'mediaDevices');
const track = { stop: vi.fn() };
@@ -489,6 +543,37 @@ describe('voice playback cancellation', () => {
}
});
+ it('settles a continuous turn locally when its converted WAV exceeds the cap', async () => {
+ const { MAX_VOICE_AUDIO_BYTES, RECORDING_TOO_LONG_MESSAGE } = await import('../lib/audioRecorder.js');
+ const previousMediaDevices = Object.getOwnPropertyDescriptor(navigator, 'mediaDevices');
+ Object.defineProperty(navigator, 'mediaDevices', {
+ configurable: true,
+ value: { getUserMedia: vi.fn().mockResolvedValue({ getTracks: () => [{ stop: vi.fn() }] }) },
+ });
+ let now = 1000;
+ vi.spyOn(performance, 'now').mockImplementation(() => now);
+ audioRecorderStub.float32ToWav16k = vi.fn(async () => ({ wav: new ArrayBuffer(MAX_VOICE_AUDIO_BYTES + 1), peak: 0.5 }));
+ const onError = vi.fn();
+ const onSubmit = vi.fn();
+ const unsubscribe = voiceClient.onVoiceEvent('voice:error', onError);
+ try {
+ await voiceClient.startContinuous({ onSubmit });
+ const frame = FakeAudioWorkletNode.last.port.onmessage;
+ now = 1700;
+ frame({ data: new Float32Array(128) });
+ for (let i = 0; i < 40; i++) frame({ data: new Float32Array(128).fill(0.5) });
+ now = 18000;
+ frame({ data: new Float32Array(128).fill(0.5) });
+ await vi.waitFor(() => expect(onSubmit).toHaveBeenCalledWith({ submitted: false, peak: 0.5 }));
+ expect(onError).toHaveBeenCalledWith({ stage: 'turn', message: RECORDING_TOO_LONG_MESSAGE });
+ expect(socket.emit).not.toHaveBeenCalledWith('voice:turn', expect.anything());
+ } finally {
+ unsubscribe();
+ if (previousMediaDevices) Object.defineProperty(navigator, 'mediaDevices', previousMediaDevices);
+ else delete navigator.mediaDevices;
+ }
+ });
+
it('ignores VAD worklet frames and async submissions after owner disposal', async () => {
const previousMediaDevices = navigator.mediaDevices;
const track = { stop: vi.fn() };
diff --git a/server/index.js b/server/index.js
index aee03b4154..77bcbdcbfa 100644
--- a/server/index.js
+++ b/server/index.js
@@ -3,6 +3,7 @@ import providerFallbackRoutes from './routes/providerFallbacks.js';
import './services/databaseBootFence.js';
import express from 'express';
import { Server } from 'socket.io';
+import { SOCKET_MAX_PAYLOAD_BYTES } from './lib/socketPayloadLimits.js';
import { fileURLToPath } from 'url';
import { dirname, join } from 'path';
import { PATHS } from './lib/fileUtils.js';
@@ -214,6 +215,7 @@ setHttpsEnabledAtBoot(httpsEnabled);
// Socket.IO with relative path support for Tailscale
const io = new Server(httpServer, {
+ maxHttpBufferSize: SOCKET_MAX_PAYLOAD_BYTES,
// CORS headers are reflected only for handshakes allowRequest admits: the
// same browser-relay guard as the HTTP gate refuses a foreign Origin or a
// rebindable Host on both the polling and websocket transports, in both
diff --git a/server/lib/README.md b/server/lib/README.md
index 6765f9a282..72ff3796d1 100644
--- a/server/lib/README.md
+++ b/server/lib/README.md
@@ -150,6 +150,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub
| `postValidation.js` | MeatSpace POST (Power On Self Test) schemas — drill config (incl. adaptive toggle), drill generation/scoring, sessions, memory builder, training log. |
| `privacyValidation.js` | Privacy Center schemas — PII Vault (issue #2140): vault record create/update (partial PUT), list query, UUID params; the vault type/status vocabularies + the sensitive-type (`ssn`/`passport`/`drivers_license`/`financial_account`) `useForScans` hard-false rule and per-type scan defaults. Trusted Organizations registry (issue #2141): org create/update (partial PUT), list query, UUID params, and the replace-set holdings schema. Data-broker database + case ledger (issue #2144): broker list/case-list query filters, the scan-start + refresh action bodies, and the `PRIVACY_BROKER_CASE_STATES` vocabulary. |
| `roundsValidation.js` | Rounds workbench shape bounds + pure record sanitizers (sections, layers, recordings, pitch analysis, references, progress) — `sanitizeRound(raw, builtinIds)` and the length/count limits shared with routes/rounds.js. |
+| `socketPayloadLimits.js` | Shared 8 MiB media cap and transport ceiling accounting for polling base64 encoding plus 64 KiB framing headroom. |
| `socketValidation.js` | Socket event payload schemas. |
| `spriteValidation.js` | Sprite Manager Zod schemas — record import/create/update, reference generate/lock/unlock + fork, walk-cycle generate/approve/reopen/trim, per-track generate/approve schemas built from the animation-track registry, publish binding + runtime contract, atlas publish/compile, and user-defined animation-track create/update. Split out of `validation.js` (#3873), which re-exports it. |
| `styleSourcePrompt.js` | `universeStyleLines(universe)` / `moodBoardSection(board)` — a resolved universe style guide or mood board (`services/creativeStyleSources.js`) as prompt text; shared by Code Animation and Creative Commissions. |
diff --git a/server/lib/index.js b/server/lib/index.js
index 2835d2e0ba..5776f2aa1b 100644
--- a/server/lib/index.js
+++ b/server/lib/index.js
@@ -75,6 +75,7 @@ export * as postRhetoric from './postRhetoric.js';
export * as postValidation from './postValidation.js';
export * as privacyValidation from './privacyValidation.js';
export * as roundsValidation from './roundsValidation.js';
+export * from './socketPayloadLimits.js';
export * as socketValidation from './socketValidation.js';
export * from './socketEventContracts.js';
export * from './socketEventInventory.js';
diff --git a/server/lib/socketPayloadLimits.js b/server/lib/socketPayloadLimits.js
new file mode 100644
index 0000000000..d36eeaa910
--- /dev/null
+++ b/server/lib/socketPayloadLimits.js
@@ -0,0 +1,5 @@
+// Keep the transport above the handler caps even on Engine.IO polling, which
+// base64-encodes binary attachments. Reserve 64 KiB for packet framing and
+// metadata, including a small over-cap payload reaching its handler refusal.
+export const SOCKET_MEDIA_MAX_BYTES = 8 * 1024 * 1024;
+export const SOCKET_MAX_PAYLOAD_BYTES = 4 * Math.ceil(SOCKET_MEDIA_MAX_BYTES / 3) + 64 * 1024;
diff --git a/server/sockets/fableLoomHosted.js b/server/sockets/fableLoomHosted.js
index 744074e438..946147b5a2 100644
--- a/server/sockets/fableLoomHosted.js
+++ b/server/sockets/fableLoomHosted.js
@@ -6,6 +6,7 @@
* acquires general PortOS auth privileges or access to other socket events.
*/
+import { SOCKET_MEDIA_MAX_BYTES } from '../lib/socketPayloadLimits.js';
import {
_getInternalSession,
abortHostedTurn,
@@ -27,7 +28,7 @@ import {
// two seconds of headroom over the ~20 ms bursts the audience page ships, and
// 8 MB is ~4 minutes of 16 kHz mono audio for a single turn.
const MAX_MIC_FRAME_BYTES = 64 * 1024;
-const MAX_UTTERANCE_BYTES = 8 * 1024 * 1024;
+const MAX_UTTERANCE_BYTES = SOCKET_MEDIA_MAX_BYTES;
const AUDIO_TOO_LARGE = Symbol('audio-too-large');
// Socket payloads are untrusted: never let Buffer.from coerce arbitrary objects
diff --git a/server/sockets/socketPayloadLimits.test.js b/server/sockets/socketPayloadLimits.test.js
new file mode 100644
index 0000000000..ebe3652282
--- /dev/null
+++ b/server/sockets/socketPayloadLimits.test.js
@@ -0,0 +1,119 @@
+import { createServer } from 'node:http';
+import { once } from 'node:events';
+import { readFileSync } from 'node:fs';
+import { Server } from 'socket.io';
+import { io as connect } from 'socket.io-client';
+import { describe, it, expect, vi } from 'vitest';
+import { SOCKET_MAX_PAYLOAD_BYTES } from '../lib/socketPayloadLimits.js';
+import { registerVoiceHandlers, MAX_AUDIO_BYTES, MAX_SCREENSHOT_BYTES } from './voice.js';
+import { registerFableLoomHostedNamespace, MAX_UTTERANCE_BYTES } from './fableLoomHosted.js';
+
+vi.mock('../services/voice/config.js', () => ({ getVoiceConfig: vi.fn(async () => ({ enabled: true })) }));
+vi.mock('../services/voice/pipeline.js', () => ({ runTurn: vi.fn(async ({ emit, state, text }) => {
+ if (text === 'capture') {
+ const screenshot = new Promise((resolve) => state.screenshotWaiters.set('example-capture', resolve));
+ emit('voice:screenshot:request', { requestId: 'example-capture' });
+ const dataUrl = await screenshot;
+ emit('test:screenshot', { length: dataUrl?.length ?? null });
+ }
+ emit('voice:idle', {});
+ return {};
+}) }));
+vi.mock('../services/voice/stt.js', () => ({ transcribe: vi.fn() }));
+vi.mock('../services/voice/tts.js', () => ({ synthesize: vi.fn() }));
+vi.mock('../services/fableLoom/hostedSession.js', () => {
+ const session = { id: 'example-session', status: 'active', turnPhase: 'listening' };
+ return {
+ _getInternalSession: () => session,
+ abortHostedTurn: vi.fn(),
+ endHostedSession: vi.fn(),
+ getHostedSession: () => session,
+ processHostedUtterance: vi.fn(async (_id, { io }) => {
+ io.of('/fableloom-hosted').emit('hosted:accepted');
+ }),
+ sanitizeHostedSession: () => session,
+ startHostedListening: vi.fn(),
+ switchHostedEpisode: vi.fn(),
+ updateHostedSession: vi.fn(),
+ verifyHostedHostToken: () => false,
+ verifyHostedToken: (_id, token) => token === 'example-token',
+ };
+});
+
+const nextEvent = (socket, event) => once(socket, event, { signal: AbortSignal.timeout(5000) });
+
+describe('Socket.IO media transport contract', () => {
+ it('boots with the shared ceiling above encoded media caps and bounded framing', () => {
+ const bootstrap = readFileSync(new URL('../index.js', import.meta.url), 'utf8');
+ expect(bootstrap).toContain('maxHttpBufferSize: SOCKET_MAX_PAYLOAD_BYTES');
+ for (const cap of [MAX_AUDIO_BYTES, MAX_SCREENSHOT_BYTES, MAX_UTTERANCE_BYTES]) {
+ expect(SOCKET_MAX_PAYLOAD_BYTES - 4 * Math.ceil(cap / 3)).toBeGreaterThanOrEqual(64 * 1024);
+ }
+ });
+
+ // Polling expands binary to base64; websocket sends raw bytes. The maximum
+ // permitted recording and handler refusal must work on both shipped paths.
+ it.each(['websocket', 'polling'])('accepts capped media and refuses over-cap media without losing %s subscriptions', async (transport) => {
+ const http = createServer();
+ const server = new Server(http, { maxHttpBufferSize: SOCKET_MAX_PAYLOAD_BYTES });
+ server.on('connection', (socket) => {
+ registerVoiceHandlers(socket);
+ socket.on('test:ping', () => socket.emit('test:pong'));
+ });
+ registerFableLoomHostedNamespace(server);
+ const clients = [];
+ try {
+ http.listen(0, '127.0.0.1');
+ await once(http, 'listening');
+ const origin = `http://127.0.0.1:${http.address().port}`;
+ const voice = connect(origin, { transports: [transport], reconnection: false });
+ clients.push(voice);
+ await nextEvent(voice, 'connect');
+ expect(voice.io.engine.transport.name).toBe(transport);
+ for (const size of [2 * 1024 * 1024, MAX_AUDIO_BYTES]) {
+ const accepted = nextEvent(voice, 'voice:idle');
+ voice.emit('voice:turn', { audio: Buffer.alloc(size), mimeType: 'audio/wav' });
+ await accepted;
+ }
+ const refused = nextEvent(voice, 'voice:error');
+ voice.emit('voice:turn', { audio: Buffer.alloc(MAX_AUDIO_BYTES + 1) });
+ expect((await refused)[0]).toMatchObject({ stage: 'turn', message: expect.stringContaining('audio too large') });
+ const pong = nextEvent(voice, 'test:pong');
+ voice.emit('test:ping');
+ await pong;
+ expect(voice.connected).toBe(true);
+
+ for (const size of [MAX_SCREENSHOT_BYTES, MAX_SCREENSHOT_BYTES + 1]) {
+ const request = nextEvent(voice, 'voice:screenshot:request');
+ voice.emit('voice:text', { text: 'capture' });
+ const [{ requestId }] = await request;
+ const result = nextEvent(voice, 'test:screenshot');
+ const prefix = 'data:image/jpeg;base64,';
+ voice.emit('voice:screenshot:result', { requestId, dataUrl: prefix + 'A'.repeat(size - prefix.length) });
+ expect((await result)[0]).toEqual({ length: size <= MAX_SCREENSHOT_BYTES ? size : null });
+ }
+ expect(voice.connected).toBe(true);
+
+ const guest = connect(`${origin}/fableloom-hosted`, {
+ transports: [transport], reconnection: false,
+ auth: { sessionId: 'example-session', token: 'example-token', role: 'audience' },
+ forceNew: true,
+ });
+ clients.push(guest);
+ await nextEvent(guest, 'connect');
+ const accepted = nextEvent(guest, 'hosted:accepted');
+ guest.emit('hosted:mic:stop', Buffer.alloc(MAX_UTTERANCE_BYTES));
+ await accepted;
+ const guestRefused = nextEvent(guest, 'hosted:error');
+ guest.emit('hosted:mic:stop', Buffer.alloc(MAX_UTTERANCE_BYTES + 1));
+ expect((await guestRefused)[0]).toMatchObject({ code: 'UTTERANCE_TOO_LARGE' });
+ const recovered = nextEvent(guest, 'hosted:accepted');
+ guest.emit('hosted:mic:stop', Buffer.alloc(1024));
+ await recovered;
+ expect(guest.connected).toBe(true);
+ } finally {
+ clients.forEach((client) => client.disconnect());
+ await new Promise((resolve) => server.close(resolve));
+ }
+ });
+});
diff --git a/server/sockets/voice.js b/server/sockets/voice.js
index 8045465084..922c53c0ea 100644
--- a/server/sockets/voice.js
+++ b/server/sockets/voice.js
@@ -16,6 +16,7 @@
// Inbound: voice:capture:start | voice:call:audio | voice:capture:stop
// Outbound: voice:capture:state
+import { SOCKET_MEDIA_MAX_BYTES } from '../lib/socketPayloadLimits.js';
import { socketHasCurrentHostControl } from '../services/authGate.js';
import { runTurn } from '../services/voice/pipeline.js';
import { getVoiceConfig } from '../services/voice/config.js';
@@ -68,7 +69,9 @@ const HISTORY_MESSAGES = 24;
// Payload size caps. Voice audio is typically 16 kHz mono PCM/WebM (~32 KB/s),
// so 8 MB leaves headroom for ~4 min of audio even in WAV. Text utterances are
// short; 4 KB covers any realistic spoken turn and rejects prompt-stuffing.
-const MAX_AUDIO_BYTES = 8 * 1024 * 1024;
+export const MAX_AUDIO_BYTES = SOCKET_MEDIA_MAX_BYTES;
+// Encoded data-URL string length, including the prefix; not decoded image bytes.
+export const MAX_SCREENSHOT_BYTES = SOCKET_MEDIA_MAX_BYTES;
const MAX_TEXT_LEN = 4000;
// Cap on the visible-text snapshot the client ships alongside each UI index.
// Matches `MAX_TEXT_CHARS` in `client/src/services/domIndex.js` so the ~8 KB
@@ -363,8 +366,8 @@ export const registerVoiceHandlers = (socket) => {
// captured tab (or null if capture failed / the user denied permission). The
// requestId echoes the request so a late result can't resolve a newer waiter.
// Cap the payload to bound memory from a runaway/malicious client — a
- // full-screen PNG data URL is typically a few MB, so 16 MB is generous.
- const MAX_SCREENSHOT_BYTES = 16 * 1024 * 1024;
+ // full-screen JPEG data URL is typically a few MB; cap the encoded string
+ // at 8 MiB so it fits the transport alongside packet metadata.
socket.on('voice:screenshot:result', (payload) => {
if (!payload || typeof payload !== 'object') return;
const { requestId, dataUrl: raw } = payload;