Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
Voice Mode
PortOS includes an optional voice assistant with support for fully local operation. Supports push-to-talk and hands-free continuous mode with barge-in (start speaking over a reply to interrupt it).
Privacy note: LLM, TTS, and pipeline orchestration always run locally. For speech-to-text, the default Web Speech API uses your browser's built-in recognition service — in Chrome/Chromium this forwards audio to a Google cloud endpoint. Switch stt.engine to whisper (whisper.cpp, local HTTP server) under Settings → Voice for fully-offline STT with no audio leaving the machine.
Stack
| Stage | Default engine | Alternatives | Local? |
|---|---|---|---|
| Speech-to-text | Browser Web Speech API (default — note: Chromium browsers forward audio to a vendor cloud speech service) or whisper.cpp via whisper-server (HTTP :5562, fully local) |
— | ✅ (whisper) / ⚠️ (web-speech) |
| LLM | LM Studio (/v1/chat/completions) |
OpenAI-compatible local server | ✅ |
| Text-to-speech | Piper (CLI) | Qwen3-TTS (CPU/CUDA/Apple Silicon MLX) | ✅ |
| Voice activity | AudioWorklet + RMS VAD (hands-free) or MediaRecorder (push-to-talk) — Web Speech mode bypasses server audio and posts final text via voice:text |
— | ✅ |
The TTS engine is selectable in Settings → Voice → TTS engine.
Qwen3-TTS status: the bundled runner calls the official
qwen-tts inference API
for voice design and reference cloning on CPU/CUDA. Install qwen-tts in the
isolated Qwen Python environment; the package supplies Torch, NumPy and
SoundFile. Apple Silicon uses MLX Audio
against the same verified official snapshots. In the isolated Qwen Python
environment, install mlx-audio at commit
784b29e2691a93ca7483147d86f61859dfaa6296 (the adapter's reference API) plus
soundfile and huggingface_hub. The runtime probe checks MLX/Metal availability
without loading weights. MLX cloning requires a reference transcript and a
loaded speech-tokenizer encoder; otherwise it fails rather than generating an
unconditioned voice. Speech-rate changes and instruction-controlled Base cloning are
refused rather than reported as applied controls. Fine-tuning is described
under Voice Studio below. Explicit model
downloads in Settings fetch the official Qwen snapshot at an immutable Hub
revision and verify the size and digest of every required file, including the
speech tokenizer weights, before publishing readiness. Downloads require
huggingface_hub in the isolated Qwen Python environment. A missing dependency
or failed verification returns an error; it never marks the model downloaded.
Earlier versions produced test tones, placeholder checkpoints and download
metadata; those artifacts are not evidence of model readiness. Runtime probes
report dependency/hardware availability without loading weights. Model download readiness is separate: it
requires a verified snapshot marker and all required files. Unchanged verified
files use their size, change timestamps, and filesystem identity; changed files
are rehashed before they can report ready. Windows probes rehash all files because
Python does not expose a reliable change timestamp there; old metadata-only model folders remain unavailable.
Inference loads only these verified snapshots with Hub/Transformers offline
mode enabled, never a repository ID that might trigger an implicit download.
The returned model revision includes the immutable commit hash. No model
download runs automatically. Audio is buffered, not streamed.
Apple Silicon real-model generation and an independent transcript check passed on 2026-09-27 (evidence below). The operator confirmed clear, natural design speech and clear cloned speech with the same speaker identity. Fixture tests prove the adapter contract, not speech quality. On a supported host with the isolated environment and verified weights, run the following manually from the repository root (substitute the isolated Python executable and model root):
python scripts/qwen3_tts_runner.py --models-dir <model-root> \
--mode design --model-id Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \
--text 'The example garden gate is open.' --instructions 'A clear warm alto.' \
--output-wav <scratch-dir>/design.wav
python scripts/qwen3_tts_runner.py --models-dir <model-root> \
--mode clone --model-id Qwen/Qwen3-TTS-12Hz-1.7B-Base \
--reference-audio <scratch-dir>/design.wav \
--reference-transcript 'The example garden gate is open.' \
--text 'Please close the gate after you enter.' --output-wav <scratch-dir>/clone.wav
Listen to both files and record intelligibility and voice continuity with the reported revisions before treating this as a validated speech runtime. The reference is generated, so no personal recording or consent is needed for this smoke. A successful command or non-silent WAV alone does not establish quality.
Apple Silicon smoke evidence — 2026-09-27 (#8857). The two commands above
ran successfully on macOS arm64 through the production runner at 4a8dafc3a,
with seed 42 and rate 1.0. No runtime code changes were needed. The reference
was the generated design WAV, not a personal recording. Model acquisition used
the runner's explicit --download operation for each model; all required files
passed upstream size/digest verification before inference loaded local snapshots
with Hub/Transformers offline mode enabled.
| Model | Verified revision |
|---|---|
Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign |
5ecdb67327fd37bb2e042aab12ff7391903235d3 |
Qwen/Qwen3-TTS-12Hz-1.7B-Base |
fd4b254389122332181a7c3db7f27e918eec64e3 |
The isolated environment used Python 3.11.11, MLX 0.32.2, MLX Audio 0.4.8
at the pinned commit above, Transformers 5.17.0, NumPy 2.4.6, SoundFile 0.14.0,
and huggingface-hub 1.33.0. PyTorch was absent; the probe reported device: mlx,
both 1.7B snapshots downloaded, and training_adapter: null.
| Output | Format | Duration | RMS | Peak | Independent local transcript |
|---|---|---|---|---|---|
design.wav |
Mono PCM16, 24 kHz | 1.76 s | 0.115736 | 0.622681 | The example Garden Gate is open. |
clone.wav |
Mono PCM16, 24 kHz | 1.84 s | 0.098126 | 0.410736 | Please close the gate after you enter. |
Both decoded WAVs were finite and non-silent. Local MLX Whisper 0.4.3
(mlx-community/whisper-small.en-mlx revision
52a88bf6e98b114a210c21bb83e22d6e1505cb73, English, no supplied transcript prompt)
recovered every requested word, ignoring case and punctuation. This supports
intelligibility for these two short samples; it does not establish naturalness,
speaker identity, general quality, or browser playback latency. WAV SHA-256:
design.wav c05344914d4c81b7099c427483f44d620edfe743f7b17af1f08cdb93044ace29
clone.wav 7ff72e5745adb753b36f318f567cddceaf13e42c37f5e7d21e6a9969492dee0e
After listening to both files, the operator confirmed that the design sample was clear and natural, and that the clone clearly spoke the requested sentence and sounded like the same speaker. These are listening observations for the two samples, not a numerical similarity score or general quality benchmark.
A real --mode fine-tune invocation on this host exited 1 with
QWEN3_RUNTIME_UNAVAILABLE (requires CUDA with bf16), creating no checkpoint
directory. A real CUDA training run and an intelligible checkpoint audition
remain open in #8857; these MLX results do not close that issue.
Interactive profile qualification plays a fresh buffered WAV in the browser.
The first click renders and transfers the probe. A second explicit playback
click preserves browser autoplay permission. The latency gate sums measured
render-request time and click-to-playing startup time, including response
transfer and decoding, while excluding the operator's wait between clicks.
It is a render-plus-playback-start benchmark, not continuous request-to-playback
or first-chunk streaming latency. Rendering
alone never enables the route. A one-use receipt expires after two minutes, binds
the measurement to the rendered profile revision, and saves its boundary, model
revision and route decision together. Restarting the server or changing the
profile requires a fresh benchmark. The probe must reach ended after playing
before submitting its startup measurement. Playback rejection or timeout cannot
qualify.
The browser reports its own playback event; this is a trusted operator-reported
measurement, not an attestation of another machine's audio output. The receipt
binds the report to a current probe but cannot prove that a remote client played
it. Qualification changes profile routing, not execution authority. No similarity score is
invented, and passing this latency gate does not establish intelligibility or
speaker identity.
FaceTime Audio control plane
FaceTime Audio controls are off by default and remain machine-local. On macOS, run npm run setup:facetime, grant the installed helper Accessibility permission in System Settings, choose the configured BlackHole devices in FaceTime, then enable FaceTime Audio in Settings → Features. Set a target name and E.164 phone number or email in Settings → Voice, save, and use Probe, Test call, or Hang up. The helper refuses ambiguous FaceTime surfaces and never uses coordinate clicks.
FaceTime Audio call bridge
The control plane can dial and hang up. Carrying the conversation takes two virtual audio devices and a browser tab.
Devices. npm run setup:facetime offers to run brew install blackhole-2ch blackhole-16ch. BlackHole is GPLv3 and is never bundled — PortOS asks, you
install it, and declining is fine (dial and hang up keep working without it).
In FaceTime, set the output to BlackHole 16ch and the microphone to
BlackHole 2ch. Both must run at 48 kHz; Settings → Voice → Check setup
verifies the label, rate, and channel count of each and names the exact problem
when one is wrong. A device list that cannot be read at all reports that
plainly rather than claiming the device is missing.
Call host. Open /voice/call-host in a browser tab on the Mac running
PortOS and press Attach call host. Device permissions and setSinkId need
a real browser profile, so this cannot live on the server. The page reads
BlackHole 16ch (what FaceTime plays into), streams it to PortOS as 16 kHz mono
PCM, and plays each reply back through BlackHole 2ch (what FaceTime hears as
its microphone). An input-level meter and a one-second test tone confirm both
directions before you rely on them.
The page fails closed and says why: a browser missing any required API names all of them at once, a missing or misconfigured device is named specifically, an unlabeled device list is reported as a missing microphone permission rather than a missing driver, and a second tab is refused — it holds a Web Lock, and the server refuses a second host independently, so two tabs can never double-answer one call.
Turns. A phone call has no push-to-talk, so the server decides where a turn ends: energy-based voice activity detection with 700 ms of trailing silence and a 20-second ceiling per utterance. Each utterance runs the existing voice pipeline — same persona, tools, confirm gate, and TTS settings as the widget. Speaking over a reply interrupts it, exactly as the widget's barge-in does.
Ending. The call ends when the caller hangs up, after 60 seconds of caller silence, at the configured maximum call length (Settings → Voice, default 15 minutes), or when the call-host tab goes away — a call nobody can hear is ended rather than left running. The helper's own view of the FaceTime window is the source of truth throughout: a probe that fails is treated as unknown, never as a hangup.
What is kept. A text transcript is appended to the daily journal, labelled
Caller and PortOS. The call audio is never persisted, and the configured
handle never appears in the transcript or its metadata.
Calls PortOS places on its own. Two opt-in paths can dial without you
asking: the Persistent Mind's voice.call-user grant, and the
critical-notification escalation in Settings → Voice. Both are off by
default and share one gate and one budget — never while a browser tab can speak
the message, never inside voice quiet hours, at most 3 calls per rolling 24
hours and at least 30 minutes apart, counted in durable state so a restart
cannot reset them. Escalation additionally fires only for a critical
notification that is still unread after escalateAfterMinutes. See
Chief of Staff enhancement → Persistent Mind phone calls.
Calling PortOS back. Turn on Automatically answer incoming calls in
Settings → Voice → FaceTime Audio (facetime.autoAnswer, off by default) to
call the Mac from your phone or watch and talk to your Chief of Staff. Fail
closed by construction: the helper's answer command presses only the
Notification Center action naming your own configured identity — a call from
any other handle is left ringing, and PortOS never logs anything that would
name who it was. Answering also needs the call host tab open and attached
on this Mac (same tab the outbound bridge uses); without it, an authorized
call rings unanswered and a medium notification records the miss instead of
silently dropping it. Quiet hours change only the greeting's tone, never
whether the call is picked up — you placed it, so PortOS answers at any hour.
Once answered, the call runs exactly like an outbound one: whisper STT →
voice LLM/persona → Piper/Qwen3 TTS, the same silence/max-duration/hangup
rules, and a transcript in the daily journal. If the Persistent Mind is
running, the call carries its persona and context and the transcript is
handed back to it as a message on hangup, continuing the same conversation on
its next wake; if the mind isn't running, it's the plain voice persona, same
as the widget. The Mind tab shows an active-call chip with a Hang up
button (voice:call:hangup) for either direction, from any tab — not just
the one carrying the audio.
Meeting capture
The call-host page's second mode: /voice/call-host?mode=capture (or the Capture system audio tab on the same page) turns live audio from a Zoom/Meet/FaceTime meeting into a timestamped transcript in the daily journal and the Brain inbox — no dialing, no reply, no LLM call.
Setup. Reads the same BlackHole 16ch device as the call bridge, so it needs no separate driver — but only the input half: capture never plays anything back, so BlackHole 2ch is not required. Route the meeting app's output to BlackHole 16ch (a macOS Multi-Output Device lets you also hear it through your speakers at the same time).
Capturing. Press Start capture; PortOS transcribes continuously with the same whisper.cpp STT and energy-based endpointing (700 ms trailing silence) the call bridge uses, but stops there — it never runs the LLM/tools pipeline, so no AI provider is called while a meeting is being captured. Press Stop capture (or close the tab) to finalize: the transcript is appended to the daily journal under a "Meeting capture" heading with start/stop timestamps, and filed as a Brain inbox item with auto-classify off — exactly like a manually-typed thought with classification skipped. The usual summarize/tag flow applies only once you ask for it from the inbox.
Mutually exclusive with a call. Both modes read the same BlackHole device and the same host tab, so starting one while the other is active on this tab is refused with a specific reason rather than fighting over the device.
Default TTS engine
Piper is the default local TTS engine. Kokoro is retired: existing settings migrate to Piper while preserving custom Piper voices. Upgraded installs wait for Save & Reconcile before installing Piper. Existing Kokoro character profiles remain available as historical records; select or promote a Piper replacement to synthesize new audio.
First-time setup
- Open PortOS → Settings → Voice.
- Pick your TTS engine (default: Piper), Whisper model size, and CoreML toggle (macOS).
- Toggle Enable voice mode and click Save & Reconcile.
PortOS will:
- Install
whisper-cppvia Homebrew if missing (the whisper.cpp binary + its base models) - Download the selected Whisper ggml
.binmodel into~/.portos/voice/models/ - On macOS with CoreML enabled, download the matching
<model>-encoder.mlmodelc(2–3× faster STT on Apple Silicon, requires a custom whisper.cpp build) - If Piper is selected, download the pre-built Piper binary + phonemize libs from rhasspy/piper and rhasspy/piper-phonemize GitHub Releases (Piper is not on Homebrew), then fetch the selected voice
.onnxinto~/.portos/voice/voices/ - Start
portos-whisperunder PM2
Piper voice models live under ~/.portos/voice/voices/.
Model download completion and repair
A model file that merely exists is not proof its download finished — a dropped
connection leaves a truncated file at the final path. Both setup scripts
(scripts/setup-voice.sh, scripts/setup-voice.ps1) therefore:
- download each managed asset (Whisper
.bin, Piper.onnxand.onnx.json) to a unique temp sibling, check that the transfer completed with a non-empty payload, and parse the Piper config (audio.sample_rate) before promoting anything — a failed replacement leaves the previous files in place, and temp files are removed on failure; - write a
<asset>.portos-complete.jsonreceipt (size + sha256 per file) only after the whole asset is promoted — for Piper, only once the.onnxand its.onnx.jsonare both in place; - on every run, re-verify the receipt (including sha256) and repair anything that no longer matches.
The server applies the same contract through server/lib/voiceModelAssets.js. A model is ready
when its receipt matches the file sizes on disk, or — for installs that predate receipts and for
files at a path you chose yourself — when it is non-empty and, for Piper, the .onnx.json next to
it parses. A Piper .onnx without a valid .onnx.json, an empty file, or a file that disagrees
with its receipt is incomplete: /api/voice/status and the Piper health badge stop reporting it
ready (models.ttsVoiceState / sttModelState carry verified, unverified, incomplete or
missing).
Repair is always user-authorized. Save & Reconcile and the Settings voice-picker download re-run the setup script for an incomplete model instead of skipping it. Server boot only reads state: an incomplete model is logged and left for the next Save & Reconcile, and boot starts no model download.
Models that predate receipts keep working. The next Save & Reconcile has the script adopt them: a
Piper pair with a parseable config is recorded as complete with no network access; a Whisper file is
compared with the remote size (adopted on a match, re-downloaded on a mismatch, left untouched when
the remote size cannot be read). Only files directly in ~/.portos/voice/models/ and
~/.portos/voice/voices/ are managed this way — a model at any other path is validated but never
replaced, so give a hand-installed model its own filename or directory.
You can also run the bootstrap script directly:
TTS_ENGINE=piper INSTALL_COREML=1 bash scripts/setup-voice.sh
TTS_ENGINE=piper VOICE_NAME=en_US-ryan-high bash scripts/setup-voice.sh
MODEL_NAME=ggml-small.en.bin bash scripts/setup-voice.sh
Configuration options
All options live in data/settings.json under voice (Settings UI patches this file).
Replacing the entire voice root through PUT /api/settings requires operator
authority, including empty or null replacements that reset voice policy. The
dedicated PUT /api/voice/config endpoint continues to check changes to protected
instructions, tools, and the STT endpoint while allowing ordinary voice preferences.
| Option | Default | Notes |
|---|---|---|
enabled |
false |
Master toggle. Triggers reconcile on change. |
hotkey |
Space |
Held to talk. Ignored while typing in inputs. |
stt.model |
base.en |
tiny.en · base.en · small.en · medium.en · large-v3 |
stt.coreml |
false |
Optional on macOS. Enable to use the CoreML encoder companion (requires a custom whisper.cpp build with -DWHISPER_COREML=1). |
stt.endpoint |
http://127.0.0.1:5562 |
whisper-server listen address (whisper engine only). |
tts.engine |
piper |
piper or qwen3-tts |
tts.rate |
1.0 |
Speech rate, 0.5–2.0 |
tts.piper.voice |
en_GB-jenny_dioco-medium |
Piper voice id (path-encoded) |
tts.piper.voicePath |
~/.portos/voice/voices/<voice>.onnx |
ONNX file location |
llm.model |
auto |
auto picks first loaded LM Studio model |
llm.systemPrompt |
(concise voice prompt) | Edit to change personality |
llm.fastPath.enabled |
false |
Fast-resolution cascade (see below). Off = every turn runs on the server LLM. |
llm.fastPath.triggers |
true |
Tier 1: resolve navigation commands ("go to tasks") in the browser, no LLM. |
llm.fastPath.browserLlm |
true |
Tier 2: answer simple/conversational turns with Chrome's on-device Gemini Nano. |
llm.fastPath.browser.temperature |
0.7 |
Nano sampling temperature (0–2). |
llm.fastPath.browser.topK |
3 |
Nano sampling top-K (1–128). |
Using voice mode
Two modes, toggle with the headset icon in the voice widget:
- Push-to-talk: click/hold the mic or hold the configured hotkey. Release to send. The hotkey ignores keypresses while an input/textarea is focused.
- Hands-free: the mic stays live; an AudioWorklet computes an RMS envelope and auto-submits once you've been silent for
vad.endOfSpeechMs(default 700 ms). Ambient noise is calibrated at session start.
Either mode: start speaking while the assistant is replying to interrupt it (barge-in). Click the square button to stop current TTS playback without sending a new turn.
With stt.engine = 'web-speech' the browser's SpeechRecognition handles STT entirely client-side; PortOS sends the final transcript as voice:text instead of shipping audio, which skips whisper.cpp and avoids a 250–800 ms server-side round-trip.
Fast-resolution cascade (lower latency)
The server LLM is the slowest part of a turn — a local model with tools attached can take several seconds. When llm.fastPath.enabled is on, the client triages each turn through faster tiers first and only falls through to the server LLM when it has to:
- Trigger (
fastPath.triggers) — deterministic, offline. A navigation command ("go to tasks", "open the daily log") is matched against the ⌘K palette nav manifest and navigates immediately. No LLM. - Browser LLM (
fastPath.browserLlm) — Chrome's on-device Gemini Nano (the Prompt API:window.LanguageModel/ legacyself.ai.languageModel) answers simple/conversational turns entirely in the browser (fast, private, offline). Nano is also asked to replyESCALATEfor anything that needs a real action. - Server — the configured provider/model (recommend Ollama with a small model) via the existing pipeline. Handles every tool/action turn, personal-data retrieval, dictation, confirmations, and anything the fast tiers decline or can't run.
Notes:
- The cascade only applies to client-produced transcripts — Web Speech STT or typed input. Whisper / hands-free audio turns have no client transcript to triage, so they stay fully server-driven. Set
stt.engine = 'web-speech'to get the benefit. - Trigger/Nano replies are spoken through the server's configured TTS (
POST /api/voice/public/synthesize), reusing the normal playback queue and echo-suppression, so barge-in keeps working. - Fast-tier turns are handled without a server round-trip, so they are not added to the server-side conversation history; a later server turn won't have that chit-chat in its context. Action turns (which always hit the server) are unaffected.
- Nano availability is surfaced in Settings → Voice → Fast resolution. When it isn't downloaded/enabled (
chrome://flags/#prompt-api-for-gemini-nano+#optimization-guide-on-device-model), tier 2 transparently falls through to the server. - Nothing here runs a model the user hasn't triggered — Nano only executes on a real spoken/typed turn, consistent with the no-cold-bootstrap AI policy.
Client modules: client/src/services/browserLlm.js (Nano client), client/src/services/voiceFastPath.js (the routing decision), wired into VoiceWidget.jsx.
Architecture
browser mic → MediaRecorder (PTT) OR AudioWorklet + RMS VAD (hands-free)
→ Socket.IO 'voice:turn' (audio) OR 'voice:text' (Web Speech final)
→ whisper.cpp /inference (STT for audio path)
→ LM Studio /v1/chat (streaming) (LLM)
→ sentence-boundary TTS dispatch (Piper CLI | Qwen3-TTS runtime)
→ Socket.IO 'voice:tts:audio' → Web Audio playback queue
Pipeline orchestration: server/services/voice/pipeline.js. The pipeline emits events as it runs:
voice:transcript— STT resultvoice:llm:delta— each token delta from LM Studiovoice:llm:done— full assistant replyvoice:tts:audio— one WAV per sentence as soon as TTS finishes itvoice:idle— turn complete (or interrupted)
Barge-in works by aborting the shared AbortController tied to the current turn — the LLM stream is torn down and any queued TTS is discarded.
Endpoints
| Method | Path | Purpose |
|---|---|---|
| GET | /api/voice/status |
Health probes + active engines + binary/model presence |
| GET | /api/voice/config |
Current merged voice config |
| PUT | /api/voice/config |
Deep-merge patch; triggers PM2 + setup reconcile |
| GET | /api/voice/voices |
Voices for the active TTS engine |
| POST | /api/voice/test |
Body { text }, returns WAV bytes — verifies TTS |
| GET | /api/voice/facetime/status | BlackHole device + helper + identity preflight |
| POST | /api/voice/facetime/{probe,call,answer,hangup} | Machine-local call control |
Socket events are documented in server/sockets/voice.js — including the
call-host bridge (voice:call:attach / voice:call:audio / voice:call:detach
/ voice:call:hangup inbound, voice:call:state / voice:call:tts outbound —
voice:call:state broadcasts to every connected tab, not just the call host,
so the Mind tab's active-call chip stays in sync) and meeting capture, which
reuses the same voice:call:audio PCM frames (voice:capture:start /
voice:capture:stop inbound, voice:capture:state outbound).
Troubleshooting
- Whisper badge red —
brew install whisper-cpp, thenwhich whisper-server. - CoreML missing — re-run
INSTALL_COREML=1 bash scripts/setup-voice.sh(or toggle voice off/on after enabling CoreML). - Piper spawn fails —
which piperand check voice file at~/.portos/voice/voices/<name>.onnx. - LM Studio red — start LM Studio and load a chat model; the voice pipeline uses
/v1/chat/completions. - No audio playback — browsers require a user gesture before AudioContext can play. Click the page once or press the mic button.
Performance notes
| Engine | Cold start | Warm latency (per sentence) | Quality |
|---|---|---|---|
| Piper | ~100 ms (CLI spawn) | ~100 ms | Mid |
| Whisper base.en (no CoreML) | 0 (server resident) | 400–800 ms / 2 s of audio | Good |
| Whisper base.en + CoreML | 0 | 150–300 ms / 2 s of audio | Good |
| Whisper small.en + CoreML | 0 | 300–600 ms / 2 s of audio | Better |
Character Voice Studio
Create → Voice Studio (/voices) is the reusable voice library. A character's
sheet links here with its universe and character selected. /voices/new
creates a voice; /voices/:profileId auditions and assigns it. Series dialogue
and FableLoom live conversations reuse the universe-character voice resolver.
Assigning copies an approved reference into a separate character profile, so
experimentation in the library does not silently replace a cast voice. Profiles
and reference audio remain machine-local, following the existing voice policy.
AuK-Flash uses the official feat/mlx-apple-silicon backend with 8-bit inference.
On Apple Silicon, Set up AuK locally provisions an isolated Python runtime
under ~/.portos/auk, downloads weights, and converts them to MLX. It requires
uv, network access, and at least 40 GB free disk space. Setup is explicit;
startup and page reads never download weights or generate speech. Other hosts
can browse their stored library but cannot run this MLX engine.
Timbre/accent/delivery use natural-language direction. Pitch is a separate AuK acoustic edit in semitones. Pace changes target duration; the operator must listen for intelligibility and clipping before assigning. Previews are 2–6 base seconds, scaled by pace. Longer dialogue is split without dropping text into model segments of at most 12 seconds, all conditioned on the same voice reference, then joined into one WAV (up to 32 segments per request). Controls take effect on generation, not continuously during playback. Each generation creates a new candidate, with a preserved WAV reference, transcript, seed, and settings. The resident model is reused for previews and unloaded after ten idle minutes or through Unload model. Inference is serialized and times out after three minutes. Assignment offers an explicit Use this voice in FableLoom live conversations too option. It accepts buffered playback (the whole reply must finish before playback, up to the three-minute runtime timeout), not low-latency streaming qualification. Without that opt-in, AuK remains studio-only and the existing interactive fallback is retained. The production planner checks the interactive route rather than treating every approved studio voice as live-ready.
The separate Qwen runtime can fine-tune one speaker with the official
Qwen3-TTS single-speaker recipe
(adapter qwen-tts-sft-12hz). The runner ports that recipe rather than
calling the upstream scripts. The probe names the adapter only on CUDA with
bf16 support and qwen-tts, librosa and safetensors installed. CPU and Apple Silicon hosts report
training_adapter: null, and starts return 503 QWEN3_TRAINING_UNAVAILABLE before any job record or process
exists. Training uses a verified, downloaded Base snapshot offline (409 QWEN3_MODEL_NOT_INSTALLED
otherwise). It uses the profile's transcribed source recordings, with the first recording as the speaker
reference. Full-parameter training needs substantial GPU memory. Each checkpoint is a full model copy
(about the base snapshot's size), so the checkpoint interval (optimizer steps) bounds disk use.
A checkpoint is published only after the runner records the SHA-256 of every file it loads. It must pass
the same verification and loader used for synthesis and render a non-silent audition (audition.wav).
Checkpoints that fail are deleted. The published revision is
<base-model>@<base-revision>+sha256.<weights-digest>; promotion records it, and fine-tuned synthesis
refuses a checkpoint whose files changed or whose revision differs. Checkpoints load only through
qwen-tts on CPU/CUDA; MLX refuses them. Checkpoints recorded by earlier placeholder runners have no
producing adapter or sealed revision and are refused with 409 CHECKPOINT_UNVERIFIED rather than promoted.
Fixture tests prove this publication contract, not training quality. A real training run on supported
hardware is still pending in #8857. Operators should listen to each audition before promoting. The runtime is excluded from
Voice Studio assignment until repaired (issue #8857). Voice Studio does not
claim a Qwen runtime is working merely because its metadata exists.
The character Voice Lab's Fine-Tuning tab drives the whole run (#10400). On open it reads the profile's
runs from their job.json sidecars (GET /api/voice/profiles/:id/fine-tune, newest first), so a reload
recovers them. After that it applies voice:fine-tune:updated frames, which carry the whole job, on each
status change, each sealed checkpoint, and at most once a second during training. It does not poll. Each
checkpoint plays its audition from /data/voice-profiles/…. Promote is disabled, with the refusal
reason beside it, on any checkpoint the server would refuse. Cancel appears while a run is live. A voice
trains one run at a time (409 FINE_TUNE_ALREADY_RUNNING), and a cancelled run still counts until its
trainer process has exited (processActive). A
sidecar left running by a server restart is reported as interrupted, because its process is gone. Its
sealed checkpoints remain promotable.