Skip to content

Open-Source Frontier Voice AI

Python

Serve streaming ASR through the vLLM plugin

AlexTYJca6529ae5982
diff --git a/docs/vibevoice-vllm-asr-streaming.md b/docs/vibevoice-vllm-asr-streaming.md
new file mode 100644
index 0000000..e2a763e
--- /dev/null
+++ b/docs/vibevoice-vllm-asr-streaming.md
@@ -0,0 +1,213 @@
+# VibeVoice vLLM Streaming ASR Deployment
+
+<a href="https://huggingface.co/microsoft/VibeVoice-ASR-Streaming"><img alt="Huggingface" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-VibeVoice--ASR--Streaming-blue"></a>
+
+Deploy VibeVoice ASR Streaming model as a high-performance API service using [vLLM](https://github.com/vllm-project/vllm). A session pushes audio while it is still being spoken and gets one transcript back per chunk, so text appears before the audio ends.
+
+**Non-streaming:** [vLLM-asr](./vibevoice-vllm-asr.md)<br>
+
+## πŸ”₯ Key Features
+
+- **🎀 True Streaming**: Transcribe from a live microphone or a growing stream, one chunk at a time
+- **πŸš€ Prefix-Cached Sessions**: Each chunk reuses the KV cache of every chunk before it, so cost stays flat as a session grows
+- **πŸ“‘ WebSocket + REST**: `/v1/stream` for live audio, `/v1/transcribe` and `/v1/chat/completions` for whole files
+- **πŸ”Œ Plugin Architecture**: No vLLM source code modification required - just install and run
+- **🎚️ Checkpoint-Defined Chunking**: Chunk and lookahead are read from the checkpoint, so a model always runs at the geometry it was trained on
+
+## πŸ› οΈ Installation
+
+Using Official vLLM Docker Image (Recommended)
+
+1. Clone the repository
+```bash
+git clone https://github.com/microsoft/VibeVoice.git
+cd VibeVoice
+```
+
+2. Launch the server (background mode)
+```bash
+docker run -d --gpus all --name vibevoice-vllm-streaming \
+ --ipc=host \
+ -p 8000:8000 \
+ -e VIBEVOICE_FFMPEG_MAX_CONCURRENCY=64 \
+ -e PYTORCH_ALLOC_CONF=expandable_segments:True \
+ -v $(pwd):/app \
+ -w /app \
+ --entrypoint bash \
+ vllm/vllm-openai:v0.14.1 \
+ -c "python3 /app/vllm_plugin/scripts/start_streaming_server.py"
+```
+
+## ⚑ Multi-GPU Deployment
+
+### Tensor Parallel
+
+Split a single model across 2 GPUs (useful if GPU memory is limited):
+
+```bash
+docker run -d --gpus '"device=0,1"' --name vibevoice-vllm-streaming \
+ --ipc=host \
+ -p 8000:8000 \
+ -e VIBEVOICE_FFMPEG_MAX_CONCURRENCY=64 \
+ -e PYTORCH_ALLOC_CONF=expandable_segments:True \
+ -v $(pwd):/app \
+ -w /app \
+ --entrypoint bash \
+ vllm/vllm-openai:v0.14.1 \
+ -c "python3 /app/vllm_plugin/scripts/start_streaming_server.py --tp 2"
+```
+
+### Data Parallel
+
+`--dp` is not available here. A session's audio lives in one replica's prefix cache, and round-robin would send its later chunks to a replica that never saw the earlier ones. To use N GPUs for throughput, run one server per GPU on its own port and route **whole sessions**, not individual chunks:
+
+```bash
+for i in 0 1 2 3; do
+ docker run -d --gpus "\"device=$i\"" --name vibevoice-vllm-streaming-$i \
+ --ipc=host \
+ -p $((8000 + i)):8000 \
+ -e PYTORCH_ALLOC_CONF=expandable_segments:True \
+ -v $(pwd):/app \
+ -w /app \
+ --entrypoint bash \
+ vllm/vllm-openai:v0.14.1 \
+ -c "python3 /app/vllm_plugin/scripts/start_streaming_server.py"
+done
+```
+
+3. View logs
+```bash
+docker logs -f vibevoice-vllm-streaming
+```
+
+> **Note**:
+> - The `-d` flag runs the container in background (detached mode)
+> - Use `docker stop vibevoice-vllm-streaming` to stop the service
+> - The model will be downloaded to HuggingFace cache (`~/.cache/huggingface`) inside the container
+
+## πŸš€ Usages
+
+### Usage 1: Test the API
+
+Once the streaming server is running, test it with the provided script:
+
+```bash
+# Basic streaming transcription
+docker exec -it vibevoice-vllm-streaming python3 vllm_plugin/tests/test_api_streaming.py /app/audio.wav
+
+# With hotwords for better recognition of specific terms
+docker exec -it vibevoice-vllm-streaming python3 vllm_plugin/tests/test_api_streaming.py /app/audio.wav --hotwords "Microsoft,VibeVoice"
+```
+
+Whole-file endpoints are available too, for callers that do not stream:
+
+```bash
+docker exec -it vibevoice-vllm-streaming python3 vllm_plugin/tests/test_api.py /app/audio.wav
+```
+
+> **Note**:
+> - The audio/video file must be inside the mounted directory (`/app` in the container). Copy your files to the VibeVoice folder before testing.
+> - Hotwords help improve recognition of domain-specific terms like proper nouns, technical terms, and speaker names.
+
+### Usage 2: Launch the demo page
+
+Start the demo in a second container on the host network, so that `--api_url` reaches the server already running there:
+
+```bash
+docker run -d --network host --name vibevoice-vllm-streaming-demo \
+ -v $(pwd):/app \
+ -w /app \
+ --entrypoint bash \
+ vllm/vllm-openai:v0.14.1 \
+ -c "python3 /app/vllm_plugin/scripts/fastapi_asr_streaming_demo_api.py --api_url http://localhost:8000"
+```
+
+Open `http://localhost:7860`, then record from the microphone or pick a file. The page keeps a WebSocket open for the whole recording, so text appears while you are still speaking.
+
+Nothing extra to install: the demo needs only `fastapi`, `uvicorn`, `httpx` and `websockets`, all of which the vLLM image already ships.
+
+### Usage 3: Share the demo publicly
+
+Add `--cloudflared` to get a public link. The flag downloads the `cloudflared` binary to `~/.local/bin` on first use, so the container needs outbound network:
+
+```bash
+docker run -d --network host --name vibevoice-vllm-streaming-demo \
+ -v $(pwd):/app \
+ -w /app \
+ --entrypoint bash \
+ vllm/vllm-openai:v0.14.1 \
+ -c "python3 /app/vllm_plugin/scripts/fastapi_asr_streaming_demo_api.py --api_url http://localhost:8000 --cloudflared"
+```
+
+The link is printed once the tunnel is up:
+
+```bash
+docker logs vibevoice-vllm-streaming-demo
+```
+
+```
+relaying to http://localhost:8000, open http://localhost:7860
+
+ public URL: https://xxxx-xxxx-xxxx.trycloudflare.com
+```
+
+> **Note**: use the `https` link rather than a plain `http` address when the page is not on `localhost` β€” browsers only grant microphone access in a secure context.
+
+### Demo Options
+
+| Flag | Description | Default |
+|------|-------------|---------|
+| `--api_url URL` | Streaming server URL | `http://localhost:8000` |
+| `--port PORT` | Local demo port | `7860` |
+| `--host HOST` | Address to bind | `0.0.0.0` |
+| `--cloudflared` | Create a public link using cloudflared tunnel | off |
+| `--max_new_tokens N` | Default max new tokens per chunk | `256` |
+
+### Endpoints
+
+| Endpoint | Description |
+|----------|-------------|
+| `WS /v1/stream` | Live session: JSON config, then float32 PCM frames, then `"end"` |
+| `POST /v1/transcribe` | One whole file, chunked internally, full transcript in the reply |
+| `POST /v1/transcribe_batch` | Several files in one request |
+| `POST /v1/chat/completions` | OpenAI-compatible, for existing clients |
+| `GET /v1/config` | Sample rate and chunk geometry of the loaded checkpoint |
+
+### Environment Variables
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `VIBEVOICE_FFMPEG_MAX_CONCURRENCY` | Cap on simultaneous FFmpeg decodes | unset β€” uncapped; the `docker run` commands above set `64` |
+| `PYTORCH_ALLOC_CONF` | PyTorch memory allocator config | unset; the `docker run` commands above set `expandable_segments:True` |
+
+Unlike `start_server.py`, the streaming launcher imposes no FFmpeg limit of its own, so an unset `VIBEVOICE_FFMPEG_MAX_CONCURRENCY` really does mean unlimited.
+
+## πŸ“Š Performance Tips
+
+1. **GPU Memory**: Use `--gpu-memory-utilization 0.9` for maximum throughput if you have dedicated GPU
+2. **Session Length**: `--max-model-len` is what caps how long one session can run; raise it for hour-long streams
+3. **Concurrent Sessions**: Raise `--mm-processor-cache-gb` before raising concurrency β€” it must hold every in-flight session's audio windows
+
+## 🚨 Troubleshooting
+
+### Common Issues
+
+1. **"CUDA out of memory"**
+ - Reduce `--gpu-memory-utilization`
+ - Reduce `--mm-processor-cache-gb`
+ - Use smaller `--max-model-len`
+
+2. **"is not a streaming checkpoint"**
+ - The checkpoint has no `chunk_frames` in `preprocessor_config.json`
+ - Serve it with `start_server.py` instead
+
+3. **"tokenizer layout mismatch"**
+ - `<|text_chunk_end|>` is missing or sits at another id, so nothing would ever end a chunk. The server refuses to start rather than serve a first chunk and then empty ones
+ - Regenerate: `python3 -m vllm_plugin.tools.generate_tokenizer_files --output MODEL_PATH --streaming`
+
+4. **"Cache miss" or a session stalling after many chunks**
+ - Raise `--mm-processor-cache-gb`, or lower the number of concurrent sessions
+
+5. **"Plugin not loaded"**
+ - Verify installation: `pip show vibevoice`
+ - Check entry point: `pip show -f vibevoice | grep entry`
diff --git a/vllm_plugin/__init__.py b/vllm_plugin/__init__.py
index 989acb4..df70e86 100644
--- a/vllm_plugin/__init__.py
+++ b/vllm_plugin/__init__.py
@@ -56,6 +56,10 @@ def register_vibevoice():
# This name must match the "architectures" list in config.json
ModelRegistry.register_model("VibeVoice", VibeVoiceForCausalLM)
ModelRegistry.register_model("VibeVoiceForASRTraining", VibeVoiceForCausalLM)
+ # Streaming checkpoints declare their own architecture name. The decoder is
+ # identical -- streaming only changes how audio and text interleave in the
+ # prompt, which is handled in _get_prompt_updates.
+ ModelRegistry.register_model("VibeVoiceForASRStreamingTraining", VibeVoiceForCausalLM)
# Note: This function is called via vllm.general_plugins entry point
diff --git a/vllm_plugin/asr_streaming.py b/vllm_plugin/asr_streaming.py
new file mode 100644
index 0000000..c97412f
--- /dev/null
+++ b/vllm_plugin/asr_streaming.py
@@ -0,0 +1,332 @@
+"""The streaming ASR sequence format, apart from the server that drives it.
+
+The model was trained on a single interleaved sequence
+
+ Prompt + [<sp_start> AudioWindow <sp_end> Text <|text_chunk_end|>] * N
+
+so a client's job is to cut audio into the windows training used and rebuild
+that sequence chunk by chunk. This module does that, plus the inverse: turning
+the per-chunk text back into readable segments. Three processes need one or
+both -- the vLLM engine, the streaming server, and the browser demo -- and a
+copy of the chunk arithmetic in each would be free to drift.
+
+Nothing here imports vLLM or torch at module level, so the demo can reach it by
+bare import (``vllm_plugin/`` on sys.path, ``import asr_streaming``) without
+executing ``vllm_plugin/__init__.py`` and its model registration.
+"""
+import json
+import os
+import re
+import tempfile
+from dataclasses import dataclass
+from typing import Dict, List, Optional
+
+import numpy as np
+
+# The model signals end-of-chunk with this token rather than EOS. It is the
+# first free slot after Qwen2.5's vocabulary and was trained, not synthesised.
+# The tokenizer reads it from the checkpoint rather than defining it, so a
+# checkpoint whose files disagree with this literal resolves to something else
+# -- or to nothing, in which case every chunk runs to max_tokens.
+TEXT_CHUNK_END_ID = 151665
+
+# Plain text, no chat template, no BOS. Not the processor's SYSTEM_PROMPT +
+# <|im_start|> template, which serves the non-streaming single-shot path only.
+_PROMPT_HEAD = (
+ "You are a helpful assistant that transcribes audio input into text output. "
+ "Please transcribe the following audios streamingly with these keys: "
+ "speaker, content"
+)
+
+
+@dataclass(frozen=True)
+class ChunkGeometry:
+ """How a checkpoint cuts audio: read from it, never guessed.
+
+ A window carries ``chunk_frames`` of new audio plus ``lookahead_frames``
+ that the next window repeats. The released checkpoints differ here -- 15
+ frames (2.0s) and 22 frames (2.9333s), 4 frames of lookahead either way --
+ and serving one at the other's geometry does not raise, it just transcribes
+ worse. So it is read from ``preprocessor_config.json``, which every
+ streaming checkpoint records it in.
+ """
+
+ sample_rate: int
+ frame_samples: int
+ chunk_frames: int
+ lookahead_frames: int
+
+ @property
+ def window_frames(self) -> int:
+ return self.chunk_frames + self.lookahead_frames
+
+ @property
+ def chunk_samples(self) -> int:
+ return self.chunk_frames * self.frame_samples
+
+ @property
+ def window_samples(self) -> int:
+ return self.window_frames * self.frame_samples
+
+ @property
+ def chunk_seconds(self) -> float:
+ """The advance, which segment times and the demo's capture use.
+
+ Chunk k is fed audio ``[k*advance, k*advance + window)`` and the text it
+ returns covers ``[k*advance, (k+1)*advance)`` -- the trailing lookahead
+ is audio whose transcript arrives with the next chunk.
+ """
+ return self.chunk_samples / self.sample_rate
+
+ @classmethod
+ def from_pretrained(cls, model_path: str) -> "ChunkGeometry":
+ """Load the geometry from a local checkpoint directory or a HF repo id."""
+ name = "preprocessor_config.json"
+ if os.path.isdir(model_path):
+ config_path = os.path.join(model_path, name)
+ else:
+ from huggingface_hub import hf_hub_download
+
+ config_path = hf_hub_download(model_path, name)
+
+ with open(config_path) as f:
+ cfg = json.load(f)
+
+ missing = [k for k in ("chunk_frames", "lookahead_frames") if k not in cfg]
+ if missing:
+ raise ValueError(
+ f"{config_path} has no {', '.join(missing)}, so {model_path} is "
+ "not a streaming checkpoint. Serve it with start_server.py instead."
+ )
+ return cls(
+ sample_rate=cfg["target_sample_rate"],
+ frame_samples=cfg["speech_tok_compress_ratio"],
+ chunk_frames=cfg["chunk_frames"],
+ lookahead_frames=cfg["lookahead_frames"],
+ )
+
+ def describe(self) -> str:
+ return (f"chunk {self.chunk_frames} frames ({self.chunk_seconds:.4f}s), "
+ f"lookahead {self.lookahead_frames} frames, "
+ f"window {self.window_samples} samples @ {self.sample_rate} Hz")
+
+
+def build_prompt(context_info: Optional[str] = None) -> str:
+ """Training-format prompt, with the optional hotword/context suffix."""
+ if context_info:
+ return f"{_PROMPT_HEAD} and extra info: {context_info}\n"
+ return f"{_PROMPT_HEAD}\n"
+
+
+def split_windows(audio, geometry: ChunkGeometry, pad_last: bool = True) -> List[np.ndarray]:
+ """Cut audio into overlapping windows, matching ``split_then_encode``.
+
+ ``pad_last`` zero-pads the final window to a full window. That is not
+ cosmetic: training split already-encoded features, so its last chunk had
+ ``floor(remainder/frame)`` frames, while encoding a short segment rounds
+ *up*. Without the pad the final window would carry one extra frame and no
+ lookahead -- off-distribution exactly where the sequence ends.
+ """
+ audio = np.asarray(audio, dtype=np.float32)
+ total = audio.shape[0]
+ window_samples = geometry.window_samples
+
+ windows = []
+ start = 0
+ while start < total:
+ seg = audio[start:min(start + window_samples, total)]
+ if seg.shape[0]:
+ if pad_last and seg.shape[0] < window_samples:
+ seg = np.pad(seg, (0, window_samples - seg.shape[0]))
+ windows.append(seg)
+ # Advance by the text boundary, not by the window: a window that already
+ # reaches the end of the audio still only transcribes its first chunk,
+ # so the trailing lookahead needs a window of its own.
+ start = min(start + geometry.chunk_samples, total)
+ return windows
+
+
+def load_audio_bytes(audio_bytes: bytes, target_sr: int) -> np.ndarray:
+ """Decode in-memory audio to the mono float32 the windows are cut from.
+
+ The suffix is a placeholder: ffmpeg sniffs the container from the bytes, so
+ an mp3 or a webm opus blob written to a ``.wav`` name decodes fine.
+
+ Decodes at the native rate and resamples with librosa rather than letting
+ ffmpeg do both, which is what the streaming checkpoints were evaluated with.
+ The two resamplers differ by ~1% RMS, and near-tied greedy argmaxes turn
+ that into different words.
+ """
+ from vibevoice.processor.audio_utils import load_audio_use_ffmpeg
+
+ with tempfile.NamedTemporaryFile(suffix=".wav", delete=True) as tmp:
+ tmp.write(audio_bytes)
+ tmp.flush()
+ audio, sr = load_audio_use_ffmpeg(tmp.name, resample=False)
+
+ if sr != target_sr:
+ import librosa
+
+ audio = librosa.resample(audio, orig_sr=sr, target_sr=target_sr)
+ return audio
+
+
+# --- Chunks back into readable segments -------------------------------------
+# The model emits one block per chunk, each re-stating the speaker, each free to
+# end mid-sentence. Shown verbatim that is thirty fragments a minute. The job
+# below merges them into paragraphs that end where sentences do, identically for
+# every caller -- a rule that differs between the demo and the server is a bug
+# nobody notices until the two are compared side by side.
+
+_CHUNK_SPEAKER_RE = re.compile(r"Speaker\s+(\d+)\s*:")
+
+# Chunks merge until the segment is at least this long, then close at the next
+# sentence-ending punctuation. The cap stops a stretch with no punctuation from
+# growing one card until it swallows the whole clip; kept at twice the minimum,
+# so punctuation and not the cap is what usually ends a card.
+SEGMENT_MIN_S = 10.0
+SEGMENT_MAX_S = 20.0
+
+# Two dialects, because the rule differs. A CJK stop mark is unambiguous. An
+# ASCII period is not -- requiring whitespace or end-of-text after it keeps
+# "3.5" and "Mr. Smith" from splitting a segment in half.
+_SENTENCE_END_RE = re.compile(
+ r"[γ€‚οΌοΌŸβ€¦]+[”’」』)】》]*"
+ r"|[.!?]+[\"'’”)\]]*(?=\s|$)"
+)
+
+# Backreference, so only a repeat of the *same* tag collapses: [Noise][Noise]
+# folds, [Noise][Laughter] does not. Tags become adjacent only when blocks are
+# merged into one card, which is why this runs here and not per block.
+_REPEAT_TAG_RE = re.compile(r"(\[[^\[\]]*\])(?:\s*\1)+")
+
+# A card is one paragraph, so the line breaks the model emits inside a block
+# fold to single spaces -- except between two CJK characters, where a space is
+# not a word separator and just leaves a visible gap.
+_WS_RUN_RE = re.compile(r"\s+")
+_CJK = r"γ€€-〿぀-γƒΏδΈ€-ιΏΏοΌ€-ο½₯"
+_CJK_SPACE_RE = re.compile(rf"(?<=[{_CJK}]) (?=[{_CJK}])")
+
+
+def tidy_body(text: str) -> str:
+ """Fold a merged card's text back into one paragraph."""
+ return _CJK_SPACE_RE.sub("", _WS_RUN_RE.sub(" ", text)).strip()
+
+
+def chunk_segments(chunk_texts: List[str],
+ geometry: ChunkGeometry,
+ duration: Optional[float] = None) -> List[Dict]:
+ """Merge the per-chunk stream into readable, sentence-aligned segments.
+
+ Times stay arithmetic rather than inferred: chunk k *is* the k-th window, so
+ every boundary is a multiple of ``geometry.chunk_seconds``. Two things split
+ a chunk internally -- a speaker change, and a sentence end past the
+ SEGMENT_MIN_S mark. In both cases the words go to the right card while the
+ timestamp stays on the chunk edge.
+
+ The speaker carries forward because the model only re-states it when it
+ changes, so a chunk with no label belongs to the previous speaker.
+
+ ``duration`` only ever clamps the tail. While a clip is still streaming its
+ real duration is unknown, and the chunk count is what defines the timeline.
+ """
+ advance = geometry.chunk_seconds
+ segments: List[Dict] = []
+ cur_start = 0.0
+ cur_text = ""
+ speaker = None
+
+ def flush(end: float) -> None:
+ nonlocal cur_start, cur_text
+ body = tidy_body(_REPEAT_TAG_RE.sub(r"\1", cur_text))
+ # Silence really does come back empty; those chunks fold into whichever
+ # segment is open instead of becoming a card of their own.
+ if body:
+ if end > cur_start:
+ segments.append({
+ "Start": round(cur_start, 2),
+ "End": round(end, 2),
+ "Speaker": speaker if speaker is not None else "?",
+ "Content": body,
+ })
+ elif segments:
+ # No timeline left to give this text -- the remainder after a
+ # sentence split in the final chunk. Dropping it is how words go
+ # missing off the end of a transcript, so it joins the card it
+ # was split from instead.
+ prev = segments[-1]
+ prev["Content"] = tidy_body(
+ _REPEAT_TAG_RE.sub(r"\1", prev["Content"] + " " + body))
+ cur_start = end
+ cur_text = ""
+
+ for k, text in enumerate(chunk_texts):
+ start = k * advance
+ end = start + advance
+ if duration and duration > start:
+ end = min(end, duration)
+
+ # split() on a single-group pattern yields [pre, id, text, id, text...],
+ # which is what lets a mid-chunk speaker change break the segment.
+ pieces = _CHUNK_SPEAKER_RE.split(text)
+ added = pieces[0]
+ # Two people can easily both talk inside one chunk. Closing the card at
+ # ``start`` would put the boundary where the previous card already
+ # ended, and a zero-width card carries no text -- so the cut is placed
+ # proportionally through the chunk, by how much of its text is consumed.
+ chunk_chars = max(1, sum(len(p) for p in pieces))
+ consumed = len(pieces[0])
+ for i in range(1, len(pieces), 2):
+ if speaker is not None and pieces[i] != speaker:
+ cur_text += added
+ flush(max(start + (end - start) * (consumed / chunk_chars), cur_start))
+ added = ""
+ speaker = pieces[i]
+ added += pieces[i + 1]
+ consumed += len(pieces[i]) + len(pieces[i + 1])
+ cur_text += added
+
+ span = end - cur_start
+ if span >= SEGMENT_MIN_S:
+ # Only a stop mark inside *this* chunk is usable: one further back
+ # would strand the text between it and here in the wrong card.
+ last = None
+ for last in _SENTENCE_END_RE.finditer(added):
+ pass
+ if last is not None:
+ carry = added[last.end():]
+ cur_text = cur_text[:len(cur_text) - len(carry)] if carry else cur_text
+ flush(end)
+ cur_text = carry
+ continue
+ if span >= SEGMENT_MAX_S:
+ flush(end)
+
+ total = len(chunk_texts) * advance
+ flush(min(total, duration) if duration and duration > cur_start else total)
+ return segments
+
+
+def segments_to_text(segments: List[Dict]) -> str:
+ """Render segments as the plain transcript, one line per segment."""
+ return "\n".join(
+ f"Speaker {seg.get('Speaker', '?')}: {seg.get('Content', '')}"
+ for seg in segments
+ )
+
+
+def _srt_time(seconds: float) -> str:
+ ms = int(round(seconds * 1000))
+ h, ms = divmod(ms, 3_600_000)
+ m, ms = divmod(ms, 60_000)
+ s, ms = divmod(ms, 1000)
+ return f"{h:02d}:{m:02d}:{s:02d},{ms:03d}"
+
+
+def segments_to_srt(segments: List[Dict]) -> str:
+ """SRT, driven by the same segments as everything else so they agree."""
+ return "\n".join(
+ f"{i}\n{_srt_time(seg['Start'])} --> {_srt_time(seg['End'])}\n"
+ f"Speaker {seg.get('Speaker', '?')}: {seg.get('Content', '')}\n"
+ for i, seg in enumerate(segments, 1)
+ )
diff --git a/vllm_plugin/asr_streaming_server.py b/vllm_plugin/asr_streaming_server.py
new file mode 100644
index 0000000..90d315b
--- /dev/null
+++ b/vllm_plugin/asr_streaming_server.py
@@ -0,0 +1,851 @@
+"""FastAPI server for VibeVoice streaming ASR on vLLM V1.
+
+The model transcribes one chunk at a time off a growing interleaved sequence,
+so a request here is a *session*, not a single generate call. Three layers,
+bottom-up: the engine holder, the per-chunk session that drives it, and the
+transport that exposes it. The sequence format itself lives next door in
+``asr_streaming.py``, where it reads as a record of the training run rather
+than as this server's internals.
+
+This serves the same model as ``start_server.py``, and the choice between them
+is about who holds the session. A stock ``vllm serve`` speaks request/response
+only, so a client there has to re-send every audio window on every chunk.
+Holding the engine in-process lets a session keep its windows engine-side
+behind stable uuids -- flat payload per turn -- and lets the server offer a
+WebSocket, which is what a live microphone needs.
+
+Endpoints:
+ GET /health, /healthz, /v1/models, /v1/config
+ POST /v1/transcribe whole clip, one response, chunk texts included
+ POST /v1/transcribe_batch several clips concurrently
+ POST /v1/chat/completions OpenAI dialect, the one both backends share
+ WS /v1/stream true streaming: send audio frames, get text back
+
+Run:
+ python3 -m vllm_plugin.asr_streaming_server --model /path/to/checkpoint
+"""
+import argparse
+import asyncio
+import base64
+import json
+import logging
+import re
+import time
+import uuid
+from contextlib import asynccontextmanager
+from dataclasses import dataclass
+from typing import Any, Dict, List, Optional, Sequence
+from urllib.parse import urlparse
+
+import numpy as np
+import requests
+import uvicorn
+from fastapi import (FastAPI, HTTPException, Request, WebSocket,
+ WebSocketDisconnect)
+from fastapi.responses import StreamingResponse
+from pydantic import BaseModel, Field
+
+from vllm_plugin.asr_streaming import (ChunkGeometry, TEXT_CHUNK_END_ID,
+ build_prompt, chunk_segments,
+ load_audio_bytes, segments_to_srt,
+ split_windows)
+
+logger = logging.getLogger("vibevoice.streaming")
+
+MAX_AUDIO_BYTES = 256 * 1024 * 1024
+MAX_BATCH_ITEMS = 64
+
+
+@dataclass
+class ServerConfig:
+ """Everything the server needs that is not read from the checkpoint."""
+
+ model: str
+ served_model_name: str = "vibevoice"
+ dtype: str = "bfloat16"
+ gpu_memory_utilization: float = 0.85
+ max_model_len: int = 16384
+ tensor_parallel_size: int = 1
+ # One window per streaming chunk, so this caps session length, not audio
+ # size: 512 windows is about 17 minutes at a 2.0s chunk, proportionally
+ # longer for a checkpoint with a larger one.
+ max_audio_windows: int = 512
+ # Sized to hold every window of every in-flight session. An LRU eviction
+ # here is not a slowdown but an error: a request whose history was evicted
+ # arrives with None for that window and the engine raises "Cache miss".
+ mm_processor_cache_gb: float = 16.0
+ enforce_eager: bool = False
+ enable_prefix_caching: bool = True
+ resend_all_audio: bool = False
+
+
+# --- engine ---------------------------------------------------------------
+# The audio never leaves the request: it rides along as multimodal data and the
+# plugin's registered processor VAE-encodes it inside the engine. Three things
+# follow, and a streaming session depends on all three:
+#
+# * automatic prefix caching, which is where all of streaming's speedup lives
+# -- chunk k re-reads chunks 0..k-1 as a prefix instead of recomputing them;
+# * the scheduler's encoder-skip, which drops the VAE call for any window
+# already covered by the KV cache;
+# * continuous batching across sessions, rather than one session at a time.
+
+class VibeVoiceAsyncEngine:
+ """Owns a single ``AsyncLLM``; all sessions share it."""
+
+ def __init__(self, config: ServerConfig, geometry: ChunkGeometry):
+ from vllm.engine.arg_utils import AsyncEngineArgs
+ from vllm.v1.engine.async_llm import AsyncLLM
+
+ self.config = config
+ self.geometry = geometry
+
+ engine_args = AsyncEngineArgs(
+ model=config.model,
+ # The checkpoint's root config says float32; loading it as such
+ # would more than double the weight footprint for no benefit.
+ dtype=config.dtype,
+ gpu_memory_utilization=config.gpu_memory_utilization,
+ max_model_len=config.max_model_len,
+ tensor_parallel_size=config.tensor_parallel_size,
+ trust_remote_code=True,
+ # Non-negotiable for streaming: without it every chunk re-prefills
+ # the whole session and the cost goes quadratic. Exposed only so the
+ # APC-on/APC-off token equality check has something to turn off.
+ enable_prefix_caching=config.enable_prefix_caching,
+ limit_mm_per_prompt={"audio": config.max_audio_windows},
+ mm_processor_cache_gb=config.mm_processor_cache_gb,
+ enforce_eager=config.enforce_eager,
+ )
+ self.engine = AsyncLLM.from_engine_args(engine_args)
+
+ # Loaded by concrete class, not AutoTokenizer: the checkpoint's
+ # tokenizer_config.json names VibeVoiceASRTextTokenizerFast, which is
+ # not importable from transformers. The session layer needs this class
+ # specifically anyway -- text_chunk_end_id lives on it.
+ from vibevoice.modular.modular_vibevoice_text_tokenizer import (
+ VibeVoiceASRTextTokenizerFast)
+
+ self.tokenizer = VibeVoiceASRTextTokenizerFast.from_pretrained(config.model)
+
+ # Nothing downstream notices this on its own: the weights still load,
+ # start-up is clean, the first chunk transcribes correctly -- and then
+ # stop_token_ids never matches what the model emits and every later
+ # chunk comes back empty with the request still returning 200.
+ if self.tokenizer.text_chunk_end_id != TEXT_CHUNK_END_ID:
+ raise RuntimeError(
+ f"tokenizer layout mismatch: <|text_chunk_end|> resolved to "
+ f"{self.tokenizer.text_chunk_end_id}, expected {TEXT_CHUNK_END_ID}.\n"
+ f" model path: {config.model}\n"
+ f" id {TEXT_CHUNK_END_ID} there is "
+ f"{self.tokenizer.convert_ids_to_tokens(TEXT_CHUNK_END_ID)!r}\n"
+ "Regenerate the tokenizer with "
+ "vllm_plugin/tools/generate_tokenizer_files.py.")
+
+ async def generate(
+ self,
+ prompt_token_ids: Sequence[int],
+ audio_items: Sequence[Optional[Any]],
+ audio_uuids: Sequence[str],
+ sampling_params: Any,
+ request_id: str,
+ ):
+ """Run one chunk turn and return the final ``RequestOutput``.
+
+ ``audio_items`` carries ``None`` for every window the engine has already
+ processed; ``audio_uuids`` must stay byte-identical across turns for
+ those, since that identifier is what keeps the prefix-cache block hashes
+ stable.
+ """
+ from vllm.inputs import TokensPrompt
+
+ if len(audio_items) != len(audio_uuids):
+ raise ValueError(
+ f"audio items ({len(audio_items)}) and uuids "
+ f"({len(audio_uuids)}) must line up one-to-one")
+
+ prompt = TokensPrompt(
+ prompt_token_ids=list(prompt_token_ids),
+ multi_modal_data={"audio": list(audio_items)},
+ multi_modal_uuids={"audio": list(audio_uuids)},
+ )
+
+ final = None
+ async for out in self.engine.generate(prompt, sampling_params, request_id):
+ final = out
+ if final is None:
+ raise RuntimeError(f"engine produced no output for {request_id}")
+ return final
+
+ async def shutdown(self):
+ self.engine.shutdown()
+
+
+# --- session --------------------------------------------------------------
+# The reference implementation (``modeling_vibevoice_asr.py::streaming_generate``)
+# walks the interleaved sequence with a hand-held KV cache. vLLM has no API for
+# "keep my KV cache and append", so the equivalent here is one request per chunk
+# carrying the whole sequence so far, with prefix caching making the repeated
+# prefix free. That inverts the usual advice -- resending the prompt is normally
+# waste -- but it is what lets the engine batch many sessions continuously.
+#
+# Two invariants this layer exists to protect:
+#
+# * The token sequence must be byte-identical to the HF path. No newline after
+# <sp_end>; <|text_chunk_end|> appended after *every* chunk including one cut
+# short by EOS.
+# * The accumulated prefix is the *pre-expansion* one, holding a single
+# <|AUDIO|> per window. The plugin's processor expands each into
+# [sp_start] + N*[sp_pad] + [sp_end]; feeding an expanded sequence back in
+# would leave no placeholder to match.
+
+_RESIDUAL_SPECIALS = (
+ "<|text_chunk_end|>",
+ "<|object_ref_start|>",
+ "<|object_ref_end|>",
+ "<|box_start|>",
+ "<|speech_start|>",
+ "<|speech_end|>",
+ "<|speech_pad|>",
+)
+
+
+def _strip_specials(text: str) -> str:
+ for tok in _RESIDUAL_SPECIALS:
+ text = text.replace(tok, "")
+ return text
+
+
+class StreamingSession:
+ """One ASR conversation. Not thread-safe; drive it from a single task."""
+
+ def __init__(
+ self,
+ engine: VibeVoiceAsyncEngine,
+ session_id: str,
+ *,
+ context_info: Optional[str] = None,
+ max_new_tokens: int = 256,
+ temperature: float = 0.0,
+ top_p: float = 1.0,
+ repetition_penalty: float = 1.0,
+ ):
+ from vllm import SamplingParams
+
+ self.engine = engine
+ self.config = engine.config
+ self.geometry = engine.geometry
+ self.session_id = session_id
+
+ tokenizer = engine.tokenizer
+ self.audio_token_id = tokenizer.convert_tokens_to_ids("<|AUDIO|>")
+ self.text_chunk_end_id = tokenizer.text_chunk_end_id
+ if self.audio_token_id is None or self.audio_token_id < 0:
+ raise RuntimeError("tokenizer has no <|AUDIO|> placeholder")
+
+ self._tokens = list(
+ tokenizer.encode(build_prompt(context_info), add_special_tokens=False))
+ self._windows: List[np.ndarray] = []
+ self._turn = 0
+ self.max_new_tokens = max_new_tokens
+ # Per-chunk generated token ids, kept for the HF token-level comparison.
+ # Text equality is not enough: two token sequences can detokenize to the
+ # same string, and a divergence that starts invisible compounds.
+ self.chunk_token_ids: List[List[int]] = []
+
+ self.sampling_params = SamplingParams(
+ temperature=temperature,
+ top_p=top_p,
+ max_tokens=max_new_tokens,
+ # Caveat above 1.0: the reference penalizes only tokens generated
+ # within the current chunk, while vLLM penalizes prompt tokens too
+ # -- here, the entire session history.
+ repetition_penalty=repetition_penalty,
+ # The model signals end-of-chunk with this token rather than EOS. It
+ # is excluded from the output and re-appended to the prefix by hand,
+ # so it lands in the sequence even on an EOS stop.
+ stop_token_ids=[self.text_chunk_end_id],
+ detokenize=True,
+ )
+
+ @property
+ def num_windows(self) -> int:
+ return len(self._windows)
+
+ async def push(self, window) -> str:
+ """Transcribe one window of audio and advance the session."""
+ window = np.asarray(window, dtype=np.float32).reshape(-1)
+ self._windows.append(window)
+
+ tokens = self._tokens + [self.audio_token_id]
+ k = len(self._windows) - 1
+
+ # Two separate ceilings, checked here rather than left to the engine,
+ # which reports either as an opaque mid-session request failure. The
+ # window cap is fixed per session while the context fills at a rate that
+ # depends on how much text each chunk produces, so which one is reached
+ # first varies with the audio.
+ if len(self._windows) > self.config.max_audio_windows:
+ minutes = (self.config.max_audio_windows
+ * self.geometry.chunk_seconds / 60)
+ raise ValueError(
+ f"session exceeded {self.config.max_audio_windows} audio "
+ f"windows (~{minutes:.0f} minutes at this checkpoint's chunk). "
+ f"Raise --max-audio-windows, or start a new session.")
+ # Against what the engine sees, not what is accumulated here: each
+ # <|AUDIO|> is one token in the pre-expansion prefix and
+ # window_frames + 2 once the processor expands it. Counting the prefix
+ # as-is is short by window_frames + 1 per window -- 20 at the released
+ # geometry, enough that the engine hits its own limit hundreds of
+ # windows earlier and reports the opaque failure this check replaces.
+ expanded = len(tokens) + len(self._windows) * (
+ self.geometry.window_frames + 1)
+ if expanded + self.max_new_tokens > self.config.max_model_len:
+ raise ValueError(
+ f"session outgrew the {self.config.max_model_len}-token context "
+ f"after {k} windows ({expanded} tokens once the audio windows "
+ f"are expanded). Raise --max-model-len, or start a new session.")
+
+ # Only the new window carries data. The rest are None + a stable uuid,
+ # which keeps the per-turn payload flat instead of growing with the
+ # session, and keeps the prefix-cache block hashes stable across turns.
+ uuids = [f"{self.session_id}-{i}" for i in range(k + 1)]
+ if self.config.resend_all_audio:
+ audio: List[Optional[np.ndarray]] = list(self._windows)
+ else:
+ audio = [None] * k + [window]
+
+ self._turn += 1
+ req_id = f"{self.session_id}-t{self._turn}"
+
+ try:
+ out = await self.engine.generate(
+ tokens, audio, uuids, self.sampling_params, req_id)
+ except ValueError as exc:
+ if "data is not provided" not in str(exc):
+ raise
+ # The multimodal processor cache evicted a window we claimed was
+ # already there. Recoverable only by re-sending every window; if
+ # this fires at all, --mm-processor-cache-gb is undersized.
+ logger.warning("session %s lost a cached window, resending all %d",
+ self.session_id, len(self._windows))
+ out = await self.engine.generate(
+ tokens, list(self._windows), uuids, self.sampling_params,
+ req_id + "-full")
+
+ gen = list(out.outputs[0].token_ids)
+ # Defensive: vLLM drops the stop token, but a stop on EOS or on the
+ # length cap leaves whatever the model actually emitted.
+ while gen and gen[-1] == self.text_chunk_end_id:
+ gen.pop()
+
+ self._tokens = tokens + gen + [self.text_chunk_end_id]
+ self.chunk_token_ids.append(gen)
+ return _strip_specials(out.outputs[0].text)
+
+ async def transcribe(self, audio):
+ """Run a whole clip through the session, yielding text per chunk."""
+ for window in split_windows(audio, self.geometry):
+ yield await self.push(window)
+
+
+# --- transport ------------------------------------------------------------
+
+def _fetch_url(url: str, timeout: int) -> bytes:
+ """Fetch audio over http(s), refusing other schemes and oversized bodies."""
+ if urlparse(url).scheme not in ("http", "https"):
+ raise HTTPException(status_code=400,
+ detail="audio_url must be http or https")
+ try:
+ resp = requests.get(url, timeout=timeout, stream=True)
+ resp.raise_for_status()
+ body = resp.raw.read(MAX_AUDIO_BYTES + 1, decode_content=True)
+ except requests.RequestException as exc:
+ raise HTTPException(status_code=400,
+ detail=f"Failed to fetch audio_url: {exc}") from None
+ if len(body) > MAX_AUDIO_BYTES:
+ raise HTTPException(status_code=413,
+ detail=f"audio_url body exceeds {MAX_AUDIO_BYTES} bytes")
+ return body
+
+
+class TranscribeRequest(BaseModel):
+ audio_base64: Optional[str] = Field(None, description="Base64 audio bytes.")
+ audio_url: Optional[str] = Field(None, description="Audio URL if no base64.")
+ context_info: Optional[str] = Field(None, description="Hotwords / context.")
+ max_tokens: int = Field(256, ge=1, le=2048)
+ # Greedy is the reference decoding config; 2.0 is where the demo's slider
+ # ends. Rejecting the top of the slider with a 422 kills the session
+ # mid-transcription, and clamping silently would make it do nothing.
+ temperature: float = Field(0.0, ge=0.0, le=2.0)
+ top_p: float = Field(1.0, ge=0.0, le=1.0)
+ repetition_penalty: float = Field(1.0, ge=0.5, le=2.0)
+
+ def audio_bytes(self) -> bytes:
+ if self.audio_base64:
+ return base64.b64decode(self.audio_base64)
+ if self.audio_url:
+ return _fetch_url(self.audio_url, timeout=10)
+ raise HTTPException(status_code=400,
+ detail="audio_base64 or audio_url is required")
+
+
+class TranscribeResponse(BaseModel):
+ text: str
+ chunk_texts: List[str]
+ total_chunks: int
+ audio_duration: float
+ generate_time: float
+ # Elapsed over duration, the orientation the tests and demos print: below 1
+ # is faster than real time. The inverse reads the same but ranks backwards.
+ rtf: float
+
+
+class BatchAudioItem(BaseModel):
+ audio_base64: Optional[str] = None
+ audio_url: Optional[str] = None
+
+
+class BatchTranscribeRequest(BaseModel):
+ audios: List[BatchAudioItem] = Field(..., min_length=1,
+ max_length=MAX_BATCH_ITEMS)
+ context_info: Optional[str] = None
+ max_tokens: int = Field(256, ge=1, le=2048)
+ temperature: float = Field(0.0, ge=0.0, le=2.0)
+ top_p: float = Field(1.0, ge=0.0, le=1.0)
+ repetition_penalty: float = Field(1.0, ge=0.5, le=2.0)
+
+
+class BatchTranscribeResponse(BaseModel):
+ results: List[TranscribeResponse]
+ total_time: float
+
+
+def _new_session(engine: VibeVoiceAsyncEngine,
+ req: TranscribeRequest) -> StreamingSession:
+ return StreamingSession(
+ engine,
+ f"s-{uuid.uuid4().hex[:16]}",
+ context_info=req.context_info,
+ max_new_tokens=req.max_tokens,
+ temperature=req.temperature,
+ top_p=req.top_p,
+ repetition_penalty=req.repetition_penalty,
+ )
+
+
+async def _run_clip(engine: VibeVoiceAsyncEngine,
+ req: TranscribeRequest) -> TranscribeResponse:
+ # Both of these block, and both go to a thread for the same reason: a URL
+ # fetch waits on the network and ffmpeg shells out, either of which would
+ # otherwise stall the event loop for every other in-flight session.
+ raw = await asyncio.to_thread(req.audio_bytes)
+ sample_rate = engine.geometry.sample_rate
+ audio = await asyncio.to_thread(load_audio_bytes, raw, sample_rate)
+
+ session = _new_session(engine, req)
+ t0 = time.time()
+ chunk_texts = [text async for text in session.transcribe(audio)]
+ gen_t = time.time() - t0
+
+ duration = len(audio) / sample_rate
+ return TranscribeResponse(
+ text="".join(chunk_texts),
+ chunk_texts=chunk_texts,
+ total_chunks=len(chunk_texts),
+ audio_duration=duration,
+ generate_time=gen_t,
+ rtf=(gen_t / duration) if duration > 0 else 0.0,
+ )
+
+
+# --- OpenAI dialect -------------------------------------------------------
+# The session endpoints above are this server's own. Chat-completions is the one
+# dialect it shares with ``start_server.py``, and sharing it is the point: same
+# served model name, so an existing client can be moved between the two backends
+# without being rewritten.
+#
+# The transcript leaves as a JSON array of segments because that is what the
+# dialect already returned -- the non-streaming checkpoint emits segments with
+# timestamps directly. A streaming checkpoint emits plain per-chunk blocks, so
+# the array is assembled on this side rather than the reply format silently
+# changing with the checkpoint.
+
+# Two callers write hotwords into the text part of the user message, and they
+# word it differently: the demo puts them below a "context information" heading,
+# tests/test_api.py inline after "with extra info:". Everything outside the
+# match is the caller's own boilerplate, and feeding that to build_prompt()
+# would replace the training prompt with something the model never saw.
+_CONTEXT_RE = re.compile(
+ r"context information[^\n:]*:[ \t]*\n(.+)|with extra info:[ \t]*([^\n]+)",
+ re.IGNORECASE | re.DOTALL)
+
+
+def _audio_from_messages(messages: list) -> bytes:
+ for msg in messages:
+ content = msg.get("content")
+ if not isinstance(content, list):
+ continue
+ for part in content:
+ if part.get("type") != "audio_url":
+ continue
+ url = (part.get("audio_url") or {}).get("url", "")
+ if url.startswith("data:"):
+ return base64.b64decode(url.split(",", 1)[1])
+ return _fetch_url(url, timeout=30)
+ raise HTTPException(status_code=400, detail="No audio_url part in messages")
+
+
+def _context_from_messages(messages: list) -> Optional[str]:
+ """Pull the hotword block out of the text parts, if there is one.
+
+ Text without the heading is ignored rather than passed through -- an
+ arbitrary instruction is far more likely to derail the prompt than to act
+ as context.
+ """
+ for msg in messages:
+ content = msg.get("content")
+ if not isinstance(content, list):
+ continue
+ for part in content:
+ if part.get("type") != "text":
+ continue
+ m = _CONTEXT_RE.search(part.get("text") or "")
+ if m:
+ found = (m.group(1) or m.group(2) or "").strip()
+ if found:
+ return found
+ return None
+
+
+def _sse(cid: str, model: str, delta: dict, finish: Optional[str] = None) -> str:
+ return "data: " + json.dumps({
+ "id": cid, "object": "chat.completion.chunk", "model": model,
+ "choices": [{"index": 0, "delta": delta, "finish_reason": finish}],
+ }) + "\n\n"
+
+
+def _segment_delta(seg: dict, index: int) -> str:
+ """One element of the JSON array, with the comma that joins it to the last.
+
+ Sent per segment rather than in one piece at the end so cards appear while
+ the clip is still being transcribed; a partial array is a valid intermediate
+ state for a client that recovers whole objects as they arrive.
+ """
+ return ("," if index else "") + json.dumps(seg, ensure_ascii=False)
+
+
+def _field(body: dict, name: str, default):
+ """Read an optional body field without treating 0 or 0.0 as absent."""
+ value = body.get(name)
+ return default if value is None else type(default)(value)
+
+
+def create_app(config: ServerConfig) -> FastAPI:
+ """Build the app around one engine. The engine starts with the app."""
+ geometry = ChunkGeometry.from_pretrained(config.model)
+
+ @asynccontextmanager
+ async def lifespan(app: FastAPI):
+ logger.info("loading %s -- %s", config.model, geometry.describe())
+ app.state.engine = VibeVoiceAsyncEngine(config, geometry)
+ app.state.geometry = geometry
+ logger.info("ready: %s served as %r", config.model,
+ config.served_model_name)
+ yield
+ await app.state.engine.shutdown()
+
+ app = FastAPI(title="VibeVoice vLLM streaming ASR", version="2.0",
+ lifespan=lifespan)
+
+ @app.get("/healthz")
+ @app.get("/health")
+ async def health():
+ return {"status": "ok"}
+
+ @app.get("/v1/models")
+ async def models():
+ return {"object": "list",
+ "data": [{"id": config.served_model_name, "object": "model",
+ "owned_by": "vibevoice"}]}
+
+ @app.get("/v1/config")
+ async def get_config():
+ """The geometry a client has to cut audio at, read from the checkpoint.
+
+ The demo captures at ``sample_rate`` and lays its cards out on
+ ``chunk_seconds``, so both are served rather than hard-coded there --
+ that is what lets one page drive either released checkpoint.
+ """
+ return {
+ "model": config.served_model_name,
+ "sample_rate": geometry.sample_rate,
+ "chunk_seconds": geometry.chunk_seconds,
+ "chunk_frames": geometry.chunk_frames,
+ "lookahead_frames": geometry.lookahead_frames,
+ "window_samples": geometry.window_samples,
+ "max_audio_windows": config.max_audio_windows,
+ }
+
+ @app.post("/v1/transcribe", response_model=TranscribeResponse)
+ async def transcribe(req: TranscribeRequest):
+ try:
+ return await _run_clip(app.state.engine, req)
+ except HTTPException:
+ raise
+ except Exception as e:
+ logger.exception("transcribe failed")
+ raise HTTPException(status_code=500, detail=str(e))
+
+ @app.post("/v1/transcribe_batch", response_model=BatchTranscribeResponse)
+ async def transcribe_batch(req: BatchTranscribeRequest):
+ """Transcribe several clips. Concurrent -- the engine does the batching."""
+ subs = [
+ TranscribeRequest(
+ audio_base64=item.audio_base64,
+ audio_url=item.audio_url,
+ context_info=req.context_info,
+ max_tokens=req.max_tokens,
+ temperature=req.temperature,
+ top_p=req.top_p,
+ repetition_penalty=req.repetition_penalty,
+ )
+ for item in req.audios
+ ]
+ try:
+ t0 = time.time()
+ results = await asyncio.gather(
+ *(_run_clip(app.state.engine, s) for s in subs))
+ return BatchTranscribeResponse(results=results,
+ total_time=time.time() - t0)
+ except HTTPException:
+ raise
+ except Exception as e:
+ logger.exception("batch transcribe failed")
+ raise HTTPException(status_code=500, detail=str(e))
+
+ @app.post("/v1/chat/completions")
+ async def chat_completions(request: Request):
+ try:
+ body = await request.json()
+ messages = body.get("messages") or []
+ req = TranscribeRequest(
+ max_tokens=min(_field(body, "max_tokens", 256), 2048),
+ temperature=_field(body, "temperature", 0.0),
+ top_p=_field(body, "top_p", 1.0),
+ repetition_penalty=_field(body, "repetition_penalty", 1.0),
+ context_info=_context_from_messages(messages),
+ )
+ except (AttributeError, TypeError, ValueError) as exc:
+ # This dialect reads the raw body, so none of the automatic 422s
+ # FastAPI gives the typed endpoints apply here. Without this, a
+ # malformed body or an out-of-range temperature returns 500, which
+ # reads as a server fault for what is a client mistake.
+ raise HTTPException(status_code=400, detail=str(exc))
+ # To a thread for the same reason as _run_clip: an audio_url part is
+ # fetched over the network, and a stall here stalls every other session.
+ audio_bytes = await asyncio.to_thread(_audio_from_messages, messages)
+ cid = f"chatcmpl-{uuid.uuid4().hex[:24]}"
+ sample_rate = geometry.sample_rate
+
+ if not body.get("stream"):
+ audio = await asyncio.to_thread(load_audio_bytes, audio_bytes,
+ sample_rate)
+ session = _new_session(app.state.engine, req)
+ chunks = [t async for t in session.transcribe(audio)]
+ segments = chunk_segments(chunks, geometry, len(audio) / sample_rate)
+ return {
+ "id": cid, "object": "chat.completion",
+ "model": config.served_model_name,
+ "choices": [{"index": 0, "finish_reason": "stop",
+ "message": {"role": "assistant",
+ "content": json.dumps(
+ segments, ensure_ascii=False)}}],
+ }
+
+ async def events():
+ model_name = config.served_model_name
+ try:
+ audio = await asyncio.to_thread(load_audio_bytes, audio_bytes,
+ sample_rate)
+ session = _new_session(app.state.engine, req)
+ yield _sse(cid, model_name, {"role": "assistant", "content": "["})
+ chunks, sent = [], 0
+ async for chunk_text in session.transcribe(audio):
+ chunks.append(chunk_text)
+ # Every segment but the last: the trailing one is still open
+ # and a later chunk can extend it or move where it ends.
+ for seg in chunk_segments(chunks, geometry)[sent:-1]:
+ yield _sse(cid, model_name,
+ {"content": _segment_delta(seg, sent)})
+ sent += 1
+ # The real duration is known now, so the tail can be clamped.
+ for seg in chunk_segments(chunks, geometry,
+ len(audio) / sample_rate)[sent:]:
+ yield _sse(cid, model_name,
+ {"content": _segment_delta(seg, sent)})
+ sent += 1
+ yield _sse(cid, model_name, {"content": "]"})
+ yield _sse(cid, model_name, {}, finish="stop")
+ except Exception as e:
+ logger.exception("chat stream failed")
+ # The stream already carries a 200, so an error can only be
+ # reported inside it. It lands outside the array, which leaves
+ # whatever segments already arrived renderable.
+ yield _sse(cid, model_name, {"content": f"\n[error] {e}"},
+ finish="stop")
+ yield "data: [DONE]\n\n"
+
+ return StreamingResponse(events(), media_type="text/event-stream",
+ headers={"Cache-Control": "no-cache",
+ "X-Accel-Buffering": "no"})
+
+ @app.websocket("/v1/stream")
+ async def stream(ws: WebSocket):
+ """Live session: push PCM, receive one transcript per chunk of audio.
+
+ Protocol -- first message is JSON config (all keys optional), then binary
+ frames of little-endian float32 mono PCM at the checkpoint's sample rate.
+ A text message ``"end"`` flushes the tail. Each chunk comes back as
+ ``{"chunk": idx, "text": ..., "segments": [...]}``, and the final message
+ adds ``done``, ``total_chunks`` and ``srt``.
+
+ The buffer keeps the lookahead overlap rather than draining fully: the
+ window is longer than the advance.
+ """
+ await ws.accept()
+ try:
+ cfg = await ws.receive_json()
+ req = TranscribeRequest(**{
+ k: v for k, v in cfg.items()
+ if k in ("context_info", "max_tokens", "temperature", "top_p",
+ "repetition_penalty")
+ })
+ session = _new_session(app.state.engine, req)
+
+ window_samples = geometry.window_samples
+ chunk_samples = geometry.chunk_samples
+ buf = np.zeros(0, dtype=np.float32)
+ texts: List[str] = []
+
+ async def emit(window) -> None:
+ texts.append(await session.push(window))
+ payload: Dict[str, Any] = {
+ "chunk": len(texts) - 1,
+ "text": texts[-1],
+ "segments": chunk_segments(texts, geometry),
+ }
+ await ws.send_json(payload)
+
+ while True:
+ msg = await ws.receive()
+ if msg.get("type") == "websocket.disconnect":
+ return
+ if msg.get("bytes") is not None:
+ # "<f4" not np.float32: the browser sends little-endian and
+ # the native dtype would reinterpret it on a big-endian host.
+ buf = np.concatenate(
+ [buf, np.frombuffer(msg["bytes"], dtype="<f4")])
+ while len(buf) >= window_samples:
+ await emit(buf[:window_samples])
+ buf = buf[chunk_samples:]
+ elif msg.get("text") == "end":
+ # Same tail rule as the offline path: pad the remainder out
+ # to a full window instead of encoding a short one, which
+ # would emit one extra VAE frame and no lookahead.
+ for window in split_windows(buf, geometry):
+ await emit(window)
+ segments = chunk_segments(texts, geometry)
+ await ws.send_json({"done": True,
+ "total_chunks": len(texts),
+ "text": "".join(texts),
+ "segments": segments,
+ "srt": segments_to_srt(segments)})
+ return
+ except WebSocketDisconnect:
+ return
+ except Exception as e:
+ logger.exception("stream session failed")
+ try:
+ await ws.send_json({"error": str(e)})
+ except Exception:
+ pass
+
+ return app
+
+
+def parse_args(argv=None) -> argparse.Namespace:
+ parser = argparse.ArgumentParser(
+ description="VibeVoice streaming ASR server (in-process vLLM engine)",
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ epilog="""
+Examples:
+ # Serve a streaming checkpoint
+ python3 -m vllm_plugin.asr_streaming_server --model /path/to/checkpoint
+
+ # Two GPUs, longer sessions
+ python3 -m vllm_plugin.asr_streaming_server --model /path/to/checkpoint \\
+ --tensor-parallel-size 2 --max-model-len 32768
+
+The chunk and lookahead are read from the checkpoint's preprocessor_config.json,
+so a checkpoint always runs at the geometry it was trained on.
+ """)
+ parser.add_argument("--model", "-m", required=True,
+ help="Path or HF repo id of a streaming checkpoint")
+ parser.add_argument("--served-model-name", default="vibevoice",
+ help="Name reported by /v1/models (default: vibevoice)")
+ parser.add_argument("--host", default="0.0.0.0",
+ help="Address to bind (default: 0.0.0.0)")
+ parser.add_argument("--port", "-p", type=int, default=8000,
+ help="Port to serve on (default: 8000)")
+ parser.add_argument("--dtype", default="bfloat16",
+ help="Weight dtype (default: bfloat16)")
+ parser.add_argument("--tensor-parallel-size", "--tp", dest="tensor_parallel_size",
+ type=int, default=1,
+ help="Number of GPUs per replica (default: 1)")
+ parser.add_argument("--gpu-memory-utilization", type=float, default=0.85,
+ help="Fraction of GPU memory to use (default: 0.85)")
+ parser.add_argument("--max-model-len", type=int, default=16384,
+ help="Context length per session (default: 16384)")
+ parser.add_argument("--max-audio-windows", type=int, default=512,
+ help="Audio windows per session (default: 512)")
+ parser.add_argument("--mm-processor-cache-gb", type=float, default=16.0,
+ help="Multimodal cache, must hold every in-flight "
+ "session's windows (default: 16)")
+ parser.add_argument("--enforce-eager", action="store_true",
+ help="Skip CUDA graph capture (default: False)")
+ parser.add_argument("--no-enable-prefix-caching", dest="enable_prefix_caching",
+ action="store_false",
+ help="Disable prefix caching; diagnostic only, streaming "
+ "goes quadratic without it (default: enabled)")
+ parser.add_argument("--resend-all-audio", action="store_true",
+ help="Send every window as data each turn instead of "
+ "uuid placeholders; diagnostic only (default: False)")
+ return parser.parse_args(argv)
+
+
+def main():
+ args = parse_args()
+ logging.basicConfig(
+ level=logging.INFO,
+ format="%(asctime)s %(levelname)s %(name)s: %(message)s")
+
+ config = ServerConfig(
+ model=args.model,
+ served_model_name=args.served_model_name,
+ dtype=args.dtype,
+ gpu_memory_utilization=args.gpu_memory_utilization,
+ max_model_len=args.max_model_len,
+ tensor_parallel_size=args.tensor_parallel_size,
+ max_audio_windows=args.max_audio_windows,
+ mm_processor_cache_gb=args.mm_processor_cache_gb,
+ enforce_eager=args.enforce_eager,
+ enable_prefix_caching=args.enable_prefix_caching,
+ resend_all_audio=args.resend_all_audio,
+ )
+ uvicorn.run(create_app(config), host=args.host, port=args.port,
+ log_level="info", workers=1)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/vllm_plugin/model.py b/vllm_plugin/model.py
index bd9060f..76dad86 100644
--- a/vllm_plugin/model.py
+++ b/vllm_plugin/model.py
@@ -90,7 +90,7 @@ from transformers import BatchFeature
from transformers.models.whisper import WhisperFeatureExtractor
from vllm.config import VllmConfig
from vllm.multimodal import MULTIMODAL_REGISTRY
-from vllm.multimodal.parse import MultiModalDataParser
+from vllm.multimodal.parse import AudioProcessorItems, MultiModalDataParser
from vllm.sequence import IntermediateTensors
from vllm.model_executor.models.interfaces import SupportsMultiModal, SupportsPP, MultiModalEmbeddings
from vllm.model_executor.models.utils import (
@@ -474,9 +474,27 @@ class VibeVoiceAudioEncoder(nn.Module):
# vLLM Multimodal Processing Infrastructure
# ============================================================================
+def _cfg_get(hf_config, key: str, default):
+ """Read a key from an HF config that may be a dict or an object."""
+ if isinstance(hf_config, dict):
+ return hf_config.get(key, default)
+ return getattr(hf_config, key, default)
+
+
+def _is_streaming_asr(hf_config) -> bool:
+ """Whether the checkpoint interleaves audio and text (streaming ASR).
+
+ Derived from the architecture name rather than a separate config flag so
+ that a checkpoint cannot be served under the wrong protocol by mistake:
+ streaming checkpoints declare "VibeVoiceForASRStreamingTraining".
+ """
+ archs = _cfg_get(hf_config, "architectures", None) or []
+ return any("Streaming" in str(a) for a in archs)
+
+
class VibeVoiceProcessingInfo(BaseProcessingInfo):
"""Processing info for VibeVoice multimodal model."""
-
+
def get_hf_config(self):
return self.ctx.get_hf_config()
@@ -553,7 +571,28 @@ class VibeVoiceProcessingInfo(BaseProcessingInfo):
return tokens
+ def get_streaming_geometry(self):
+ """The window a streaming checkpoint was trained on, or None.
+
+ Read off the checkpoint rather than configured, so the engine sizes its
+ multimodal budget for exactly the window a client cuts. The budget is a
+ hard limit: a client cutting larger windows than the engine sized for is
+ rejected, not merely mismatched.
+ """
+ if not _is_streaming_asr(self.get_hf_config()):
+ return None
+ if not hasattr(self, "_streaming_geometry"):
+ from .asr_streaming import ChunkGeometry
+ self._streaming_geometry = ChunkGeometry.from_pretrained(
+ self.ctx.model_config.model)
+ return self._streaming_geometry
+
def get_supported_mm_limits(self) -> Mapping[str, int | None]:
+ # Streaming prompts interleave one audio window per chunk, so a single
+ # prompt carries N audio items rather than one. None means no fixed cap
+ # here; the real bound comes from --limit-mm-per-prompt.
+ if _is_streaming_asr(self.get_hf_config()):
+ return {"audio": None}
return {"audio": 1}
def get_mm_max_tokens_per_item(
@@ -583,6 +622,14 @@ class VibeVoiceProcessingInfo(BaseProcessingInfo):
compress_ratio = int(_cfg("speech_tok_compress_ratio", 3200))
sample_rate = int(_cfg("target_sample_rate", 24000))
+ # Streaming items are one fixed-size window each, never a whole
+ # utterance, so the 61-minute bound below would overstate the encoder
+ # budget by ~4 orders of magnitude.
+ geometry = self.get_streaming_geometry()
+ if geometry is not None:
+ # speech_start + window frames + speech_end
+ return {"audio": geometry.window_frames + 2}
+
# Upper bound: 61-minute audio at 24 kHz
max_audio_samples = 61 * 60 * sample_rate # 88,464,000
max_audio_tokens = int(np.ceil(max_audio_samples / compress_ratio)) + 3
@@ -621,6 +668,13 @@ class VibeVoiceDummyInputsBuilder(BaseDummyInputsBuilder[VibeVoiceProcessingInfo
compress_ratio = int(_cfg("speech_tok_compress_ratio", 3200))
sample_rate = int(_cfg("target_sample_rate", 24000))
+ # Must mirror get_mm_max_tokens_per_item: profiling a 61-minute clip
+ # per item would try to VAE-encode ~88M samples N times over and OOM at
+ # startup, long before serving a single request.
+ geometry = self.info.get_streaming_geometry()
+ if geometry is not None:
+ return geometry.window_samples
+
# Upper bound: 61-minute audio at 24 kHz
max_hour_samples = 61 * 60 * sample_rate # 88,464,000
max_tokens_from_audio = int(np.ceil(max_hour_samples / compress_ratio)) + 3
@@ -699,6 +753,45 @@ def _vibevoice_field_config(hf_inputs: Mapping[str, torch.Tensor]):
return config
+class _NoneTolerantAudioParser(MultiModalDataParser):
+ """Let a per-item ``None`` survive audio parsing.
+
+ A streaming session re-sends its whole audio list every turn, with the
+ windows vLLM has already processed replaced by ``None`` and pinned by a
+ stable ``multi_modal_uuids`` entry. That keeps the per-turn payload flat
+ instead of growing O(N) with the number of chunks.
+
+ vLLM's cache layer is built for exactly this -- ``_get_cache_missing_items``
+ raises "data is not provided" only for a ``None`` it genuinely needs to
+ re-encode -- and ``_parse_image_data`` passes list elements through
+ untouched. ``_parse_audio_data`` is the odd one out: it feeds every element
+ to ``_get_audio_with_sr``, which ends in ``assert_never`` on ``None``. So
+ resample only the real items and stitch the ``None`` holes back afterwards.
+ """
+
+ def _parse_audio_data(self, data):
+ # A list specifically, not a tuple: upstream reads a bare
+ # ``(audio, sr)`` tuple as one item, and ``sr=None`` is how a caller
+ # says "already at the target rate" -- which for this model, at 24 kHz,
+ # is the ordinary case. Matching tuples here would read that lone
+ # sample rate as a second audio item and hand the engine a hole that was
+ # never there. Holes only ever arrive in the list a session sends.
+ if not isinstance(data, list) or not any(x is None for x in data):
+ return super()._parse_audio_data(data)
+
+ present = [x for x in data if x is not None]
+ parsed = super()._parse_audio_data(present) if present else None
+ if parsed is not None and not isinstance(parsed, AudioProcessorItems):
+ # Embeddings take a different branch upstream and have no per-item
+ # holes to fill; don't try to interleave into them.
+ return parsed
+
+ processed = iter(parsed.data if parsed is not None else ())
+ return AudioProcessorItems(
+ [None if x is None else next(processed) for x in data]
+ )
+
+
class VibeVoiceMultiModalProcessor(BaseMultiModalProcessor[VibeVoiceProcessingInfo]):
"""
Multimodal processor for VibeVoice.
@@ -711,7 +804,7 @@ class VibeVoiceMultiModalProcessor(BaseMultiModalProcessor[VibeVoiceProcessingIn
"""Create a data parser with the correct target sample rate (24kHz)."""
# VibeVoice requires 24kHz, not 16kHz (Whisper default)
target_sr = 24000
- return MultiModalDataParser(target_sr=target_sr)
+ return _NoneTolerantAudioParser(target_sr=target_sr)
def _call_hf_processor(
self,
@@ -777,11 +870,18 @@ class VibeVoiceMultiModalProcessor(BaseMultiModalProcessor[VibeVoiceProcessingIn
result["raw_audio_lengths"] = torch.tensor(audio_lengths, dtype=torch.long)
# Add a random salt to ensure unique hash and bypass cache
- import uuid
- # Use a random integer for salt
- salt_val = hash(str(uuid.uuid4())) % 100000
- result["salt"] = torch.tensor([salt_val], dtype=torch.long).expand(len(raw_audio_list))
-
+ #
+ # Streaming is the exception, and needs the opposite. A session re-sends
+ # its whole audio list every turn and pins the already-encoded windows
+ # by ``multi_modal_uuids``; a per-call random salt would change their
+ # hash every turn, so vLLM would re-encode every window it was just told
+ # it could reuse -- turning a flat per-turn cost into an O(N) one.
+ if not _is_streaming_asr(self.info.get_hf_config()):
+ import uuid
+ # Use a random integer for salt
+ salt_val = hash(str(uuid.uuid4())) % 100000
+ result["salt"] = torch.tensor([salt_val], dtype=torch.long).expand(len(raw_audio_list))
+
return result
def _hf_processor_applies_updates(
@@ -873,7 +973,9 @@ class VibeVoiceMultiModalProcessor(BaseMultiModalProcessor[VibeVoiceProcessingIn
# If a full tensor is passed accidentally, fall back to its length
return int(x.shape[0])
return int(x)
-
+
+ is_streaming = _is_streaming_asr(hf_config)
+
def get_replacement(item_idx: int):
if raw_audio_lengths and item_idx < len(raw_audio_lengths):
audio_len = _to_int_len(raw_audio_lengths[item_idx])
@@ -890,10 +992,16 @@ class VibeVoiceMultiModalProcessor(BaseMultiModalProcessor[VibeVoiceProcessingIn
# Build replacement token sequence:
# <|speech_start|> + N * <|speech_pad|> + <|speech_end|> + \n
# The newline is important for correct prompt structure.
+ #
+ # Streaming is the exception: there the audio window is followed
+ # directly by the transcript of that chunk, with no separator (see
+ # modeling_vibevoice_asr.py building [sp_start, feat, sp_end]).
+ # Emitting a stray '\n' would put every chunk off-distribution.
newline_id = 198 # '\n' token
+ trailer = [] if is_streaming else [newline_id]
if speech_start_id is not None and speech_pad_id is not None and speech_end_id is not None:
embed_id = int(speech_pad_id)
- replacement_ids = [int(speech_start_id)] + [embed_id] * num_features + [int(speech_end_id), newline_id]
+ replacement_ids = [int(speech_start_id)] + [embed_id] * num_features + [int(speech_end_id)] + trailer
# Fallback: add audio BOS/EOS boundaries around repeated <|AUDIO|>.
elif audio_bos_id is not None and audio_eos_id is not None:
embed_id = int(audio_token_id)
diff --git a/vllm_plugin/scripts/fastapi_asr_streaming_demo_api.py b/vllm_plugin/scripts/fastapi_asr_streaming_demo_api.py
new file mode 100644
index 0000000..a49398d
--- /dev/null
+++ b/vllm_plugin/scripts/fastapi_asr_streaming_demo_api.py
@@ -0,0 +1,646 @@
+#!/usr/bin/env python
+"""
+VibeVoice ASR Streaming FastAPI Demo
+
+This demo uses the streaming ASR server instead of loading the model directly.
+The browser holds one WebSocket and pushes raw PCM down it; between the
+microphone and the model there is a socket and a relay, nothing else. Upload
+uses the same socket -- a file is just PCM that already exists -- so the two
+paths cannot drift apart.
+
+ GET / the UI, one self-contained page
+ GET /config the chunk geometry, read from the upstream server
+ GET /healthz readiness, includes upstream status
+ WS /ws/asr audio up, transcript down
+
+Usage:
+ python fastapi_asr_streaming_demo_api.py --api_url http://localhost:8000
+"""
+import argparse
+import asyncio
+import json
+import logging
+import os
+import re
+import subprocess
+import threading
+import time
+import urllib.request
+from urllib.parse import urlparse
+
+import httpx
+import uvicorn
+import websockets
+from fastapi import FastAPI, WebSocket, WebSocketDisconnect
+from fastapi.responses import HTMLResponse, JSONResponse
+from websockets.exceptions import ConnectionClosed
+
+logger = logging.getLogger("vibevoice.demo")
+
+
+def _ws_url(api_url: str) -> str:
+ """http://host:port -> ws://host:port/v1/stream."""
+ u = urlparse(api_url)
+ scheme = "wss" if u.scheme == "https" else "ws"
+ return f"{scheme}://{u.netloc}/v1/stream"
+
+
+async def _pump_up(browser: WebSocket, upstream) -> None:
+ """Browser -> server: PCM frames, then the "end" marker."""
+ while True:
+ msg = await browser.receive()
+ if msg.get("type") == "websocket.disconnect":
+ await upstream.close()
+ return
+ if msg.get("bytes") is not None:
+ await upstream.send(msg["bytes"])
+ elif msg.get("text") is not None:
+ await upstream.send(msg["text"])
+ if msg["text"] == "end":
+ return
+
+
+async def _pump_down(browser: WebSocket, upstream) -> None:
+ """Server -> browser: one message per chunk, verbatim.
+
+ The server hangs up as soon as it has sent the final message, often without
+ a closing handshake. That is the end of the relay, not a failure -- a
+ session that really was cut short is missing its "done" message, which the
+ page already treats as unfinished.
+ """
+ try:
+ async for raw in upstream:
+ await browser.send_text(raw if isinstance(raw, str) else raw.decode())
+ except ConnectionClosed:
+ return
+
+
+def create_app(api_url: str, max_new_tokens: int) -> FastAPI:
+ """Build the demo around one upstream server."""
+ app = FastAPI(title="VibeVoice streaming ASR demo")
+ upstream_ws = _ws_url(api_url)
+
+ async def upstream_config() -> dict:
+ """The geometry the server reads off its checkpoint.
+
+ Fetched rather than hard-coded: the page captures at ``sample_rate``,
+ and the two released checkpoints differ in chunk length. Cached after
+ the first success so the page does not re-probe on every reload.
+ """
+ if not hasattr(app.state, "config"):
+ async with httpx.AsyncClient(timeout=10) as client:
+ resp = await client.get(f"{api_url}/v1/config")
+ resp.raise_for_status()
+ app.state.config = resp.json()
+ return app.state.config
+
+ @app.get("/healthz")
+ async def healthz():
+ try:
+ cfg = await upstream_config()
+ except Exception as e:
+ return JSONResponse({"status": "upstream unreachable",
+ "api_url": api_url, "error": str(e)},
+ status_code=503)
+ return {"status": "ok", "api_url": api_url, "model": cfg.get("model")}
+
+ @app.get("/config")
+ async def config():
+ cfg = dict(await upstream_config())
+ cfg["max_new_tokens"] = max_new_tokens
+ return JSONResponse(cfg)
+
+ @app.get("/", response_class=HTMLResponse)
+ async def index():
+ return PAGE
+
+ @app.websocket("/ws/asr")
+ async def ws_asr(browser: WebSocket):
+ """Relay one browser session to one upstream session.
+
+ Both directions run concurrently: the browser keeps pushing audio while
+ transcripts for earlier chunks are still coming back, which is the whole
+ point of streaming.
+ """
+ await browser.accept()
+ try:
+ opts = json.loads(await browser.receive_text())
+ except Exception:
+ await browser.close()
+ return
+
+ opts.setdefault("max_tokens", max_new_tokens)
+ try:
+ async with websockets.connect(upstream_ws, max_size=None) as upstream:
+ await upstream.send(json.dumps(opts))
+ up = asyncio.create_task(_pump_up(browser, upstream))
+ down = asyncio.create_task(_pump_down(browser, upstream))
+ done, _ = await asyncio.wait(
+ {up, down}, return_when=asyncio.FIRST_COMPLETED)
+ if down in done:
+ # Upstream is finished or gone; nothing left to relay.
+ up.cancel()
+ else:
+ # The browser sent "end", but the tail chunks and the final
+ # "done" message are still upstream. Cancelling here would
+ # drop exactly the part the user waits for.
+ await down
+ for task in (up, down):
+ if task.done() and not task.cancelled():
+ task.result()
+ except WebSocketDisconnect:
+ return
+ except Exception as e:
+ logger.exception("relay failed")
+ try:
+ await browser.send_text(json.dumps({"error": str(e)}))
+ except Exception:
+ pass
+
+ return app
+
+
+PAGE = """<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>VibeVoice Streaming ASR</title>
+<style>
+ :root {
+ color-scheme: dark;
+ --bg:#0f1117; --panel:#171a23; --line:#262b38; --fg:#e6e8ef;
+ --muted:#8b93a7; --faint:#5c6478; --accent:#5b8cff; --rec:#ff4d5e;
+ --ok:#31c48d; --ghost:#232838; --field:#10131c;
+ --hover:#1d2230; --sel:#1d2436;
+ }
+ :root[data-theme="light"] {
+ color-scheme: light;
+ --bg:#f4f6fa; --panel:#ffffff; --line:#dde2ec; --fg:#171a22;
+ --muted:#5b6377; --faint:#8b93a7; --accent:#2f5fd0; --rec:#d92d3f;
+ --ok:#0e9250; --ghost:#eaeef6; --field:#f7f9fc;
+ --hover:#eef2f9; --sel:#e5edfb;
+ }
+ * { box-sizing: border-box; }
+ body { margin:0; background:var(--bg); color:var(--fg); font-size:15px;
+ font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,
+ "Helvetica Neue","PingFang SC","Microsoft YaHei",sans-serif; }
+ header { padding:18px 24px; border-bottom:1px solid var(--line);
+ display:flex; align-items:center; gap:12px; flex-wrap:wrap; }
+ h1 { font-size:17px; margin:0; font-weight:600; }
+ .dot { width:8px; height:8px; border-radius:50%; background:var(--muted); }
+ .dot.ok { background:var(--ok); }
+ .dot.rec { background:var(--rec); animation:pulse 1.2s infinite; }
+ @keyframes pulse { 50% { opacity:.25; } }
+ main { display:grid; grid-template-columns:320px 1fr; gap:20px; padding:20px 24px;
+ min-height:calc(100vh - 118px); align-content:stretch; }
+ main > * { min-width:0; }
+ .panel { background:var(--panel); border:1px solid var(--line);
+ border-radius:10px; padding:16px; }
+ .panel h2 { font-size:13px; text-transform:uppercase; letter-spacing:.06em;
+ color:var(--muted); margin:0 0 12px; font-weight:600; }
+ button { font:inherit; border:0; border-radius:8px; padding:10px 16px;
+ background:var(--accent); color:#fff; cursor:pointer; min-height:40px; }
+ button.ghost { background:var(--ghost); color:var(--fg); }
+ button:disabled { opacity:.45; cursor:not-allowed; }
+ #theme { margin-left:auto; background:transparent; color:var(--muted);
+ border:1px solid var(--line); padding:0 11px; min-height:32px;
+ font-size:15px; line-height:1; }
+ label { display:block; font-size:13px; color:var(--muted); margin:14px 0 6px; }
+ input[type=text], input[type=number] { width:100%; padding:9px 11px;
+ border-radius:8px; border:1px solid var(--line); background:var(--field);
+ color:var(--fg); font:inherit; }
+ input[type=file] { width:100%; font-size:13px; color:var(--muted); }
+ .row { display:flex; gap:10px; flex-wrap:wrap; }
+ .seg { border-left:2px solid var(--accent); padding:2px 0 2px 12px;
+ margin:0 0 16px; border-radius:0 6px 6px 0; }
+ .seg.play { cursor:pointer; }
+ @media (hover:hover) { .seg.play:hover { background:var(--hover); } }
+ .seg.on { background:var(--sel); }
+ .seg .meta { font-size:12px; color:var(--muted); margin-bottom:4px; }
+ .seg.on .meta { color:var(--accent); }
+ .seg .body { line-height:1.75; white-space:pre-wrap; word-break:break-word; }
+ #out:empty::before { content:"Transcript appears here as you speak.";
+ color:var(--muted); }
+ #out { min-height:320px; }
+ .status { font-size:13px; color:var(--muted); margin-top:12px; }
+ .bar { display:flex; align-items:center; justify-content:space-between;
+ gap:10px; flex-wrap:wrap; margin-bottom:12px; }
+ .bar button { min-height:32px; padding:6px 12px; font-size:13px; }
+ .mut { color:var(--faint); font-weight:400; }
+</style>
+<script>
+// Runs before first paint so the page does not flash the wrong theme. Follows
+// the OS until the header button is pressed, after which the choice sticks.
+(function () {
+ var t = null;
+ try { t = localStorage.getItem('vv-theme'); } catch (e) {}
+ if (t !== 'light' && t !== 'dark')
+ t = matchMedia('(prefers-color-scheme: light)').matches ? 'light' : 'dark';
+ document.documentElement.dataset.theme = t;
+})();
+</script>
+</head>
+<body>
+<header>
+ <h1>VibeVoice Streaming ASR</h1>
+ <span class="dot" id="dot"></span>
+ <span class="status" id="conn" style="margin:0">idle</span>
+ <button id="theme" type="button"></button>
+</header>
+<main>
+ <div class="panel">
+ <h2>Input</h2>
+ <div class="row">
+ <button id="rec">Record</button>
+ <button id="stop" class="ghost" disabled>Stop</button>
+ </div>
+ <label>Or transcribe a file</label>
+ <input type="file" id="file" accept="audio/*,video/*">
+ <div class="row" style="margin-top:10px">
+ <button id="send" disabled>Transcribe</button>
+ <button id="cancel" class="ghost" disabled>Cancel</button>
+ </div>
+ <label>Hotwords (context_info)</label>
+ <input type="text" id="ctx" placeholder="Microsoft,VibeVoice">
+ <label>Temperature <span class="mut">0 = greedy</span></label>
+ <input type="number" id="temp" value="0" min="0" max="2" step="0.1">
+ <label>Top-p <span class="mut" id="toppNote">needs temperature &gt; 0</span></label>
+ <input type="number" id="topp" value="1" min="0.05" max="1" step="0.05" disabled>
+ <div class="status" id="status">ready</div>
+ </div>
+ <div class="panel">
+ <div class="bar">
+ <h2 style="margin:0">Transcript</h2>
+ <div class="row">
+ <button class="ghost" id="copy" disabled>Copy</button>
+ <button class="ghost" id="srt" disabled>SRT</button>
+ <button class="ghost" id="clear" disabled>Clear</button>
+ </div>
+ </div>
+ <div id="out"></div>
+ </div>
+</main>
+<script>
+const $ = id => document.getElementById(id);
+let SR = 24000, MAXTOK = 256, sock = null, ctxAudio = null, node = null,
+ media = null, lastSrt = '', segments = [], pcm = null;
+
+// Kept so a card can be played back. The upload is already one Float32Array;
+// the mic is stored as Int16 in arrival order, which halves the memory for
+// something whose only destination is a speaker. Past the cap audio is dropped
+// and those cards stop being clickable, rather than the tab dying.
+const PLAYBACK_MAX_S = 1800;
+let playPcm = null, micChunks = [], micSamples = 0;
+let playCtx = null, player = null, playing = null;
+
+function playable() { return !!(playPcm || micSamples); }
+
+function keepMic(frame) {
+ if (micSamples >= PLAYBACK_MAX_S * SR) return;
+ const q = new Int16Array(frame.length);
+ for (let i = 0; i < frame.length; i++)
+ q[i] = Math.max(-1, Math.min(1, frame[i])) * 32767;
+ micChunks.push(q);
+ micSamples += q.length;
+}
+
+// Samples [a, b) of this run, whichever source it came from.
+function sliceAudio(a, b) {
+ if (playPcm)
+ return a >= playPcm.length ? null
+ : playPcm.subarray(a, Math.min(b, playPcm.length));
+ b = Math.min(b, micSamples);
+ if (a >= b) return null;
+ const out = new Float32Array(b - a);
+ let pos = 0;
+ for (const c of micChunks) {
+ const s = Math.max(a, pos), e = Math.min(b, pos + c.length);
+ for (let i = s; i < e; i++) out[i - a] = c[i - pos] / 32768;
+ pos += c.length;
+ if (pos >= b) break;
+ }
+ return out;
+}
+
+function stopPlay() {
+ if (player) {
+ player.onended = null;
+ try { player.stop(); } catch (e) {}
+ player.disconnect();
+ player = null;
+ }
+ playing = null;
+}
+
+function playSegment(i) {
+ const s = segments[i];
+ if (!s || !playable()) return;
+ if (playing === i) { stopPlay(); render(); return; }
+ const clip = sliceAudio(Math.floor(s.Start * SR), Math.ceil(s.End * SR));
+ if (!clip || !clip.length) { $('status').textContent = 'no audio kept for this card'; return; }
+ stopPlay();
+ playCtx = playCtx || new AudioContext();
+ const buf = playCtx.createBuffer(1, clip.length, SR);
+ buf.getChannelData(0).set(clip);
+ player = playCtx.createBufferSource();
+ player.buffer = buf;
+ player.connect(playCtx.destination);
+ player.onended = () => { if (playing === i) { playing = null; render(); } };
+ player.start();
+ playing = i;
+ render();
+}
+
+$('out').onclick = ev => {
+ const el = ev.target.closest('.seg');
+ if (el) playSegment(+el.dataset.i);
+};
+
+fetch('/config').then(r => r.json()).then(c => {
+ SR = c.sample_rate; MAXTOK = c.max_new_tokens;
+ $('status').textContent = 'ready -- ' + c.chunk_seconds.toFixed(2) + 's chunks';
+});
+
+function setConn(text, cls) {
+ $('conn').textContent = text;
+ $('dot').className = 'dot' + (cls ? ' ' + cls : '');
+}
+function options() {
+ return { context_info: $('ctx').value.trim() || null,
+ temperature: parseFloat($('temp').value) || 0,
+ top_p: parseFloat($('topp').value) || 1,
+ max_tokens: MAXTOK };
+}
+// Top-p only bites once sampling is on, so it follows the temperature box.
+$('temp').oninput = () => {
+ const on = (parseFloat($('temp').value) || 0) > 0;
+ $('topp').disabled = !on;
+ $('toppNote').textContent = on ? 'nucleus sampling' : 'needs temperature > 0';
+};
+// textContent throughout: a transcript is model output, and building this with
+// innerHTML would let a spoken "<script>" run in the page.
+function render() {
+ const out = $('out');
+ out.textContent = '';
+ const canPlay = playable();
+ segments.forEach((seg, i) => {
+ const card = document.createElement('div');
+ card.className = 'seg' + (canPlay ? ' play' : '') + (playing === i ? ' on' : '');
+ card.dataset.i = i;
+ if (canPlay) card.title = 'Click to play this segment';
+ const meta = document.createElement('div');
+ meta.className = 'meta';
+ meta.textContent = (playing === i ? 'β–Ά ' : '')
+ + 'Speaker ' + seg.Speaker + ' Β· '
+ + seg.Start.toFixed(2) + 's - ' + seg.End.toFixed(2) + 's';
+ const body = document.createElement('div');
+ body.className = 'body';
+ body.textContent = seg.Content;
+ card.appendChild(meta); card.appendChild(body); out.appendChild(card);
+ });
+ const empty = segments.length === 0;
+ $('copy').disabled = empty;
+ $('clear').disabled = empty;
+ $('srt').disabled = !lastSrt;
+}
+function connect() {
+ return new Promise((resolve, reject) => {
+ const proto = location.protocol === 'https:' ? 'wss' : 'ws';
+ const ws = new WebSocket(proto + '://' + location.host + '/ws/asr');
+ ws.onopen = () => { ws.send(JSON.stringify(options())); setConn('connected', 'ok'); resolve(ws); };
+ ws.onerror = () => reject(new Error('connection failed'));
+ ws.onclose = () => setConn('idle', '');
+ ws.onmessage = ev => {
+ const msg = JSON.parse(ev.data);
+ if (msg.error) { $('status').textContent = 'error: ' + msg.error; finish(); return; }
+ if (msg.segments) { segments = msg.segments; render(); }
+ $('status').textContent = msg.done
+ ? 'done -- ' + msg.total_chunks + ' chunks'
+ : 'receiving -- ' + (msg.chunk + 1) + ' chunks';
+ if (msg.done) { lastSrt = msg.srt || ''; render(); finish(); }
+ };
+ });
+}
+function finish() {
+ $('rec').disabled = false; $('stop').disabled = true; $('file').disabled = false;
+ $('send').disabled = !$('file').files.length; $('cancel').disabled = true;
+ setConn('idle', '');
+ if (node) { node.disconnect(); node = null; }
+ if (media) { media.getTracks().forEach(t => t.stop()); media = null; }
+ if (ctxAudio) { ctxAudio.close(); ctxAudio = null; }
+}
+
+function reset(msg) {
+ segments = []; lastSrt = '';
+ stopPlay();
+ playPcm = null; micChunks = []; micSamples = 0;
+ render();
+ $('status').textContent = msg;
+}
+
+$('rec').onclick = async () => {
+ reset('starting...');
+ try {
+ media = await navigator.mediaDevices.getUserMedia({ audio: true });
+ } catch (e) { $('status').textContent = 'microphone denied'; return; }
+ sock = await connect();
+ $('rec').disabled = true; $('stop').disabled = false; $('file').disabled = true;
+ $('send').disabled = true; $('cancel').disabled = false;
+ setConn('recording', 'rec');
+ ctxAudio = new AudioContext({ sampleRate: SR });
+ const src = ctxAudio.createMediaStreamSource(media);
+ node = ctxAudio.createScriptProcessor(4096, 1, 1);
+ node.onaudioprocess = e => {
+ if (!sock || sock.readyState !== WebSocket.OPEN) return;
+ const frame = new Float32Array(e.inputBuffer.getChannelData(0));
+ // Kept before the send, so a card still plays what the microphone heard
+ // even if the network dropped that frame on the way out.
+ keepMic(frame);
+ sock.send(frame.buffer);
+ };
+ src.connect(node); node.connect(ctxAudio.destination);
+};
+
+$('stop').onclick = () => {
+ $('status').textContent = 'flushing...';
+ if (sock && sock.readyState === WebSocket.OPEN) sock.send('end');
+ $('stop').disabled = true;
+};
+
+// Decoding happens on pick, sending on the button: options set after choosing a
+// file still apply, and a mis-picked file costs nothing.
+$('file').onchange = async ev => {
+ pcm = null; $('send').disabled = true;
+ const f = ev.target.files[0];
+ if (!f) { $('status').textContent = 'ready'; return; }
+ $('status').textContent = 'decoding...';
+ try {
+ const raw = await f.arrayBuffer();
+ const tmp = new AudioContext();
+ const decoded = await tmp.decodeAudioData(raw);
+ tmp.close();
+ const off = new OfflineAudioContext(1, Math.ceil(decoded.duration * SR), SR);
+ const s = off.createBufferSource();
+ s.buffer = decoded; s.connect(off.destination); s.start();
+ pcm = (await off.startRendering()).getChannelData(0);
+ } catch (e) { $('status').textContent = 'cannot decode: ' + e.message; return; }
+ $('send').disabled = false;
+ $('status').textContent = 'ready -- ' + (pcm.length / SR).toFixed(1) + 's decoded';
+};
+
+$('send').onclick = async () => {
+ if (!pcm) return;
+ reset('streaming...');
+ playPcm = pcm;
+ sock = await connect();
+ $('rec').disabled = true; $('file').disabled = true;
+ $('send').disabled = true; $('cancel').disabled = false;
+ // Sent in the cadence a microphone would produce, so text arrives gradually.
+ for (let i = 0; i < pcm.length; i += SR / 2) {
+ if (!sock || sock.readyState !== WebSocket.OPEN) return;
+ sock.send(new Float32Array(pcm.slice(i, i + SR / 2)).buffer);
+ await new Promise(r => setTimeout(r, 10));
+ }
+ if (sock && sock.readyState === WebSocket.OPEN) sock.send('end');
+ $('status').textContent = 'sent, waiting for the tail...';
+};
+
+$('cancel').onclick = () => {
+ if (sock) sock.close();
+ sock = null;
+ finish();
+ $('status').textContent = 'cancelled';
+};
+
+$('clear').onclick = () => { reset('ready'); };
+
+$('copy').onclick = () => {
+ navigator.clipboard.writeText(
+ segments.map(s => 'Speaker ' + s.Speaker + ': ' + s.Content).join('\\n'));
+ $('status').textContent = 'copied ' + segments.length + ' segments';
+};
+
+$('srt').onclick = () => {
+ const url = URL.createObjectURL(new Blob([lastSrt], { type: 'text/plain' }));
+ const a = document.createElement('a');
+ a.href = url; a.download = 'transcript.srt'; a.click();
+ URL.revokeObjectURL(url);
+};
+
+// The glyph shows what pressing it gets you, not what you are looking at.
+const root = document.documentElement;
+function paintTheme() {
+ const light = root.dataset.theme === 'light';
+ $('theme').textContent = light ? '☾' : 'β˜€';
+ $('theme').title = light ? 'Switch to dark' : 'Switch to light';
+ $('theme').setAttribute('aria-label', $('theme').title);
+}
+$('theme').onclick = () => {
+ root.dataset.theme = root.dataset.theme === 'light' ? 'dark' : 'light';
+ try { localStorage.setItem('vv-theme', root.dataset.theme); } catch (e) {}
+ paintTheme();
+};
+paintTheme();
+render();
+</script>
+</body>
+</html>
+"""
+
+
+CLOUDFLARED_PATH = os.path.expanduser("~/.local/bin/cloudflared")
+CLOUDFLARED_URL = ("https://github.com/cloudflare/cloudflared/releases/latest"
+ "/download/cloudflared-linux-amd64")
+TUNNEL_RE = re.compile(r"https://[-a-z0-9]+\.trycloudflare\.com")
+
+
+def download_cloudflared() -> bool:
+ """Fetch the cloudflared binary unless it is already there."""
+ if os.path.exists(CLOUDFLARED_PATH):
+ return True
+ print("Downloading cloudflared...")
+ os.makedirs(os.path.dirname(CLOUDFLARED_PATH), exist_ok=True)
+ try:
+ urllib.request.urlretrieve(CLOUDFLARED_URL, CLOUDFLARED_PATH)
+ os.chmod(CLOUDFLARED_PATH, 0o755)
+ return True
+ except Exception as e:
+ print(f"Failed to download cloudflared: {e}")
+ return False
+
+
+def start_cloudflared_tunnel(port: int):
+ """Publish the demo on a public https URL, printed once it is assigned.
+
+ https matters beyond convenience: getUserMedia only runs in a secure
+ context, so a plain http address on anything but localhost cannot reach
+ the microphone at all.
+ """
+ if not download_cloudflared():
+ return None
+ process = subprocess.Popen(
+ [CLOUDFLARED_PATH, "tunnel", "--url", f"http://localhost:{port}",
+ "--no-autoupdate"],
+ stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
+
+ def watch():
+ for line in process.stdout:
+ found = TUNNEL_RE.search(line)
+ if found:
+ print(f"\n public URL: {found.group(0)}\n", flush=True)
+
+ threading.Thread(target=watch, daemon=True).start()
+ time.sleep(3)
+ return process
+
+
+def main():
+ parser = argparse.ArgumentParser(description="VibeVoice ASR Streaming FastAPI Demo")
+ parser.add_argument(
+ "--api_url",
+ type=str,
+ default="http://localhost:8000",
+ help="URL of the streaming ASR server"
+ )
+ parser.add_argument(
+ "--max_new_tokens",
+ type=int,
+ default=256,
+ help="Default max new tokens per chunk"
+ )
+ parser.add_argument(
+ "--host",
+ type=str,
+ default="0.0.0.0",
+ help="Host to bind the server to"
+ )
+ parser.add_argument(
+ "--port",
+ type=int,
+ default=7860,
+ help="Port to bind the server to"
+ )
+ parser.add_argument(
+ "--cloudflared",
+ action="store_true",
+ help="Create a public link using cloudflared tunnel"
+ )
+ args = parser.parse_args()
+
+ logging.basicConfig(
+ level=logging.INFO,
+ format="%(asctime)s %(levelname)s %(name)s: %(message)s")
+
+ tunnel = start_cloudflared_tunnel(args.port) if args.cloudflared else None
+ app = create_app(args.api_url, args.max_new_tokens)
+ print(f"relaying to {args.api_url}, open http://localhost:{args.port}")
+ try:
+ uvicorn.run(app, host=args.host, port=args.port)
+ finally:
+ if tunnel:
+ tunnel.terminate()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/vllm_plugin/scripts/start_streaming_server.py b/vllm_plugin/scripts/start_streaming_server.py
new file mode 100644
index 0000000..07d270f
--- /dev/null
+++ b/vllm_plugin/scripts/start_streaming_server.py
@@ -0,0 +1,256 @@
+#!/usr/bin/env python3
+"""
+VibeVoice vLLM Streaming ASR Server Launcher
+
+One-click deployment script that handles:
+1. Installing system dependencies (FFmpeg, etc.)
+2. Installing VibeVoice Python package
+3. Downloading model from HuggingFace
+4. Refusing a checkpoint that cannot be streamed
+5. Generating tokenizer files
+6. Starting the streaming ASR server
+
+Steps 1-3 are shared with start_server.py; the rest differ. A streaming
+checkpoint transcribes one chunk at a time off a growing sequence, so a request
+is a session rather than a single generate call and stock `vllm serve` has
+nowhere to keep it. This launches vllm_plugin.asr_streaming_server instead,
+which holds the same engine in-process and adds a WebSocket for live audio.
+
+Usage:
+ python3 start_streaming_server.py [--model MODEL_ID] [--port PORT]
+"""
+
+import argparse
+import json
+import os
+import sys
+
+# Same directory, run as a script rather than a package -- make the sibling
+# importable so the four bootstrap steps stay in one place instead of drifting.
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+# And vllm_plugin/ itself, for a bare import of asr_streaming: the package
+# __init__ imports vLLM, while the checks below exist to run before the engine
+# does -- including on a box where vLLM is not importable yet.
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from asr_streaming import ChunkGeometry, TEXT_CHUNK_END_ID # noqa: E402
+from start_server import (download_model, install_system_deps, # noqa: E402
+ install_vibevoice, run_command)
+
+
+def resolve_model(model_id: str) -> str:
+ """A local checkpoint directory is served as-is; anything else is downloaded."""
+ if os.path.isdir(model_id):
+ print(f"\n{'='*60}")
+ print(f" Using local model")
+ print(f" πŸ“ Path: {model_id}")
+ print(f"{'='*60}\n")
+ return model_id
+ return download_model(model_id)
+
+
+def generate_streaming_tokenizer(model_path: str) -> None:
+ """Generate tokenizer files, unless the checkpoint already ships them.
+
+ A streaming checkpoint holds <|text_chunk_end|> at 151665 and its audio
+ tokens one id higher. Writing the non-streaming layout over that would take
+ away the token every chunk ends on, so the layout is asked for explicitly
+ and a checkpoint that already has it is left alone.
+ """
+ added_tokens = os.path.join(model_path, "added_tokens.json")
+ if os.path.isfile(added_tokens):
+ with open(added_tokens) as f:
+ if json.load(f).get("<|text_chunk_end|>") == TEXT_CHUNK_END_ID:
+ print("\nTokenizer files already have <|text_chunk_end|>, skipping\n")
+ return
+ run_command(
+ [sys.executable, "-m", "vllm_plugin.tools.generate_tokenizer_files",
+ "--output", model_path, "--streaming"],
+ "Generating tokenizer files"
+ )
+
+
+def check_streaming_checkpoint(model_path: str) -> None:
+ """Refuse a checkpoint that cannot be streamed, before the engine starts.
+
+ Serving a non-streaming checkpoint here does not raise on its own: the
+ weights load, the first chunk transcribes, and every later chunk comes back
+ empty because nothing ever emits <|text_chunk_end|>.
+ """
+ geometry = ChunkGeometry.from_pretrained(model_path)
+ print(f"\n{'='*60}")
+ print(f" βœ… Streaming checkpoint")
+ print(f" πŸ“ Path: {model_path}")
+ print(f" 🎚️ {geometry.describe()}")
+ print(f"{'='*60}\n")
+
+
+def start_streaming_server(model_path: str, port: int,
+ tensor_parallel_size: int = 1,
+ max_model_len: int = 16384,
+ max_audio_windows: int = 512,
+ mm_processor_cache_gb: float = 16.0,
+ gpu_memory_utilization: float = 0.85) -> None:
+ """Start the streaming ASR server (replaces current process)."""
+ print(f"\n{'='*60}")
+ print(f" Starting streaming ASR server on port {port}")
+ print(f" Tensor Parallel (TP): {tensor_parallel_size}")
+ print(f" Max Model Len: {max_model_len}")
+ print(f" Max Audio Windows: {max_audio_windows}")
+ print(f" MM Cache (GB): {mm_processor_cache_gb}")
+ print(f" GPU Mem Utilization: {gpu_memory_utilization}")
+ print(f"{'='*60}\n")
+
+ cmd = [
+ sys.executable, "-m", "vllm_plugin.asr_streaming_server",
+ "--model", model_path,
+ "--served-model-name", "vibevoice",
+ "--port", str(port),
+ "--dtype", "bfloat16",
+ "--tensor-parallel-size", str(tensor_parallel_size),
+ "--max-model-len", str(max_model_len),
+ "--max-audio-windows", str(max_audio_windows),
+ "--mm-processor-cache-gb", str(mm_processor_cache_gb),
+ "--gpu-memory-utilization", str(gpu_memory_utilization),
+ ]
+ os.execvp(sys.executable, cmd)
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="VibeVoice vLLM Streaming ASR Server - One-Click Deployment",
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ epilog="""
+Examples:
+ # Start with default settings (single GPU)
+ python3 start_streaming_server.py
+
+ # Use custom port
+ python3 start_streaming_server.py --port 8080
+
+ # Tensor parallel: split model across 2 GPUs
+ python3 start_streaming_server.py --tp 2
+
+ # Longer sessions (the context is what caps how long a session can run)
+ python3 start_streaming_server.py --max-model-len 32768
+
+ # Skip dependency installation (if already installed)
+ python3 start_streaming_server.py --skip-deps
+
+The chunk and lookahead are read from the checkpoint's preprocessor_config.json,
+so a checkpoint always runs at the geometry it was trained on.
+ """
+ )
+ parser.add_argument(
+ "--model", "-m",
+ default="microsoft/VibeVoice-ASR-Streaming",
+ help="HuggingFace model ID or local path (default: microsoft/VibeVoice-ASR-Streaming)"
+ )
+ parser.add_argument(
+ "--port", "-p",
+ type=int,
+ default=8000,
+ help="Server port (default: 8000)"
+ )
+ parser.add_argument(
+ "--skip-deps",
+ action="store_true",
+ help="Skip installing system dependencies"
+ )
+ parser.add_argument(
+ "--skip-tokenizer",
+ action="store_true",
+ help="Skip generating tokenizer files"
+ )
+ parser.add_argument(
+ "--tp", "--tensor-parallel-size",
+ type=int,
+ default=1,
+ dest="tensor_parallel_size",
+ help="Tensor parallel size: split one model across N GPUs (default: 1)"
+ )
+ parser.add_argument(
+ "--dp", "--data-parallel-size",
+ type=int,
+ default=1,
+ dest="data_parallel_size",
+ help="Not supported for streaming: a session's KV cache lives on one "
+ "replica, and round-robin would send its later chunks to replicas "
+ "that never saw its audio (default: 1)"
+ )
+ parser.add_argument(
+ "--max-model-len",
+ type=int,
+ default=16384,
+ dest="max_model_len",
+ help="Context per session, which is what caps session length (default: 16384)"
+ )
+ parser.add_argument(
+ "--max-audio-windows",
+ type=int,
+ default=512,
+ dest="max_audio_windows",
+ help="Audio windows per session, ~17 min at a 2s chunk (default: 512)"
+ )
+ parser.add_argument(
+ "--mm-processor-cache-gb",
+ type=float,
+ default=16.0,
+ dest="mm_processor_cache_gb",
+ help="Multimodal cache; must hold every in-flight session's windows "
+ "(default: 16)"
+ )
+ parser.add_argument(
+ "--gpu-memory-utilization",
+ type=float,
+ default=0.85,
+ dest="gpu_memory_utilization",
+ help="GPU memory utilization fraction (default: 0.85)"
+ )
+ args = parser.parse_args()
+
+ if args.data_parallel_size > 1:
+ parser.error(
+ "--dp is not supported for streaming. Each session keeps its audio "
+ "windows in one replica's prefix cache, and an nginx round-robin "
+ "would send its later chunks to a replica that never saw them. Run "
+ "one server per GPU on separate ports and route whole sessions, or "
+ "use --tp to split the model across GPUs."
+ )
+
+ print("\n" + "="*60)
+ print(" VibeVoice vLLM Streaming ASR Server - One-Click Deployment")
+ print("="*60)
+
+ # Step 1: Install system dependencies
+ if not args.skip_deps:
+ install_system_deps()
+
+ # Step 2: Install VibeVoice
+ install_vibevoice()
+
+ # Step 3: Download model
+ model_path = resolve_model(args.model)
+
+ # Step 4: Refuse a checkpoint that cannot be streamed -- before step 5,
+ # which rewrites the tokenizer files in place and would leave a
+ # non-streaming checkpoint unusable by start_server.py too.
+ check_streaming_checkpoint(model_path)
+
+ # Step 5: Generate tokenizer files
+ if not args.skip_tokenizer:
+ generate_streaming_tokenizer(model_path)
+
+ # Step 6: Start server
+ start_streaming_server(
+ model_path, args.port,
+ tensor_parallel_size=args.tensor_parallel_size,
+ max_model_len=args.max_model_len,
+ max_audio_windows=args.max_audio_windows,
+ mm_processor_cache_gb=args.mm_processor_cache_gb,
+ gpu_memory_utilization=args.gpu_memory_utilization,
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/vllm_plugin/tests/test_api_streaming.py b/vllm_plugin/tests/test_api_streaming.py
new file mode 100644
index 0000000..f6cb9a5
--- /dev/null
+++ b/vllm_plugin/tests/test_api_streaming.py
@@ -0,0 +1,185 @@
+#!/usr/bin/env python3
+"""
+Test VibeVoice vLLM Streaming ASR API over WebSocket, with Optional Hotwords.
+
+This script tests the streaming ASR server: the audio is pushed as PCM frames
+and a transcript comes back once per chunk, while the rest of the file is still
+being sent. Text appears as it is recognized rather than at the end.
+
+Optionally, you can provide hotwords (context_info) to improve recognition
+of domain-specific content like proper nouns, technical terms, and speaker names.
+Hotwords are embedded in the prompt as "with extra info: {hotwords}".
+
+Usage:
+ python test_api_streaming.py [audio_path] [--url URL] [--hotwords "word1,word2"]
+
+Examples:
+ # Standard streaming transcription (no hotwords)
+ python3 test_api_streaming.py audio.wav
+
+ # With hotwords for better recognition of specific terms
+ python3 test_api_streaming.py audio.wav --hotwords "Microsoft,Azure,VibeVoice"
+"""
+import asyncio
+import json
+import os
+import sys
+import time
+import argparse
+from urllib.parse import urlparse
+
+import requests
+import websockets
+
+# Bare import off vllm_plugin/: the package __init__ imports vLLM, which this
+# client does not need. Decoding here rather than in ffmpeg alone keeps the
+# resampler identical to the server's, so the transcript matches.
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+from asr_streaming import load_audio_bytes # noqa: E402
+
+
+def _ws_url(base_url: str) -> str:
+ """http://host:port -> ws://host:port/v1/stream."""
+ u = urlparse(base_url)
+ scheme = "wss" if u.scheme == "https" else "ws"
+ return f"{scheme}://{u.netloc}/v1/stream"
+
+
+async def test_streaming_with_hotwords(
+ audio_path: str,
+ context_info: str = None,
+ base_url: str = "http://localhost:8000",
+):
+ """
+ Test streaming ASR transcription with customized hotwords.
+
+ Args:
+ audio_path: Path to the audio file
+ context_info: Hotwords string (e.g., "Microsoft,Azure,VibeVoice")
+ base_url: Streaming ASR server URL
+ """
+
+ print(f"=" * 70)
+ print(f"Testing Streaming Transcription")
+ print(f"=" * 70)
+ print(f"Input file: {audio_path}")
+ print(f"Hotwords: {context_info or '(none)'}")
+ print()
+
+ # The chunk length is a property of the checkpoint, not of this client.
+ try:
+ config = requests.get(f"{base_url}/v1/config", timeout=10).json()
+ except Exception as e:
+ print(f"❌ Cannot reach {base_url}: {e}")
+ return
+ sample_rate = config["sample_rate"]
+ print(f"Model: {config['model']}")
+ print(f"Chunk: {config['chunk_seconds']:.4f}s "
+ f"({config['chunk_frames']} frames + {config['lookahead_frames']} lookahead)")
+
+ # Load audio (video files work too: ffmpeg takes the audio track)
+ try:
+ with open(audio_path, "rb") as f:
+ audio = load_audio_bytes(f.read(), sample_rate)
+ except Exception as e:
+ print(f"Error preparing audio: {e}")
+ return
+ duration = len(audio) / sample_rate
+ print(f"Audio duration: {duration:.2f} seconds")
+
+ if context_info and context_info.strip():
+ print(f"\nπŸ“ Hotwords embedded in prompt: '{context_info}'")
+ else:
+ print(f"\nπŸ“ No hotwords provided")
+
+ url = _ws_url(base_url)
+ print(f"\n{'=' * 70}")
+ print(f"Streaming to {url}")
+ print(f"{'=' * 70}")
+
+ t0 = time.time()
+ result = None
+ try:
+ async with websockets.connect(url, max_size=None) as ws:
+ await ws.send(json.dumps({
+ "context_info": context_info,
+ "max_tokens": 256,
+ "temperature": 0.0,
+ "top_p": 1.0,
+ }))
+
+ async def send_audio():
+ # Half a second per frame, as a microphone would deliver it.
+ step = sample_rate // 2
+ for i in range(0, len(audio), step):
+ await ws.send(audio[i:i + step].astype("<f4").tobytes())
+ await ws.send("end")
+
+ sender = asyncio.create_task(send_audio())
+
+ print("\nβœ… Connected. Streaming content:\n")
+ print("-" * 50)
+ async for raw in ws:
+ msg = json.loads(raw)
+ if "error" in msg:
+ print(f"\n❌ Error: {msg['error']}")
+ break
+ if msg.get("done"):
+ result = msg
+ print("\n" + "-" * 50)
+ print(f"βœ… [Finished] {msg['total_chunks']} chunks")
+ break
+ # Each chunk carries only its own text; the timestamp shows how
+ # far behind the audio the transcript is running.
+ print(f"[{time.time() - t0:6.2f}s] {msg['text']}", flush=True)
+ await sender
+ except Exception as e:
+ print(f"❌ Error: {e}")
+
+ elapsed = time.time() - t0
+ if result:
+ print(f"\n{'=' * 70}")
+ print("Segments")
+ print(f"{'=' * 70}")
+ for seg in result["segments"]:
+ print(f"[{seg['Start']:7.2f} - {seg['End']:7.2f}] "
+ f"Speaker {seg['Speaker']}: {seg['Content']}")
+
+ print(f"\n{'=' * 70}")
+ print(f"⏱️ Total time elapsed: {elapsed:.2f}s")
+ print(f"πŸ“Š RTF (Real-Time Factor): {elapsed / duration:.2f}x")
+ print(f"{'=' * 70}")
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="Test VibeVoice vLLM Streaming ASR API"
+ )
+ parser.add_argument(
+ "audio_path",
+ help="Path to audio file (wav, mp3, flac, etc.) or video file"
+ )
+ parser.add_argument(
+ "--url",
+ default="http://localhost:8000",
+ help="Streaming ASR server URL (default: http://localhost:8000)"
+ )
+ parser.add_argument(
+ "--hotwords",
+ type=str,
+ default=None,
+ help="Hotwords to improve recognition (e.g., 'Microsoft,Azure,VibeVoice')"
+ )
+
+ args = parser.parse_args()
+
+ # Run test
+ asyncio.run(test_streaming_with_hotwords(
+ audio_path=args.audio_path,
+ context_info=args.hotwords,
+ base_url=args.url,
+ ))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/vllm_plugin/tools/generate_tokenizer_files.py b/vllm_plugin/tools/generate_tokenizer_files.py
index 62de55f..ede98a9 100644
--- a/vllm_plugin/tools/generate_tokenizer_files.py
+++ b/vllm_plugin/tools/generate_tokenizer_files.py
@@ -52,6 +52,22 @@ VIBEVOICE_AUDIO_TOKENS = {
# All extended tokens (Qwen2.5 + VibeVoice)
ALL_EXTENDED_TOKENS = {**QWEN25_EXTENDED_TOKENS, **VIBEVOICE_AUDIO_TOKENS}
+# Streaming checkpoints train <|text_chunk_end|> into the first free slot after
+# Qwen2.5, so their audio tokens all sit one id higher.
+VIBEVOICE_STREAMING_AUDIO_TOKENS = {
+ "<|text_chunk_end|>": 151665,
+ "<|AUDIO|>": 151666,
+ "<|audio_bos|>": 151667,
+ "<|audio_eos|>": 151668,
+}
+
+
+def _token_maps(streaming: bool):
+ """(audio tokens, every extended token) for the requested layout."""
+ audio = VIBEVOICE_STREAMING_AUDIO_TOKENS if streaming else VIBEVOICE_AUDIO_TOKENS
+ return audio, {**QWEN25_EXTENDED_TOKENS, **audio}
+
+
# Chat template with audio support
# Key modification: handles part['type'] == 'audio' or 'audio_url' -> '<|AUDIO|>'
VIBEVOICE_CHAT_TEMPLATE = """{%- if tools %}
@@ -174,20 +190,21 @@ def download_qwen_tokenizer_files(output_dir: str, qwen_model: str = DEFAULT_QWE
)
-def patch_tokenizer_config(output_dir: str) -> None:
+def patch_tokenizer_config(output_dir: str, streaming: bool = False) -> None:
"""
Patch tokenizer_config.json with VibeVoice audio tokens and chat template.
"""
+ audio_tokens, all_tokens = _token_maps(streaming)
config_path = os.path.join(output_dir, "tokenizer_config.json")
-
+
with open(config_path, "r", encoding="utf-8") as f:
config = json.load(f)
-
+
# 1. Add ALL extended tokens to added_tokens_decoder (Qwen2.5 + VibeVoice audio)
if "added_tokens_decoder" not in config:
config["added_tokens_decoder"] = {}
-
- for token, token_id in ALL_EXTENDED_TOKENS.items():
+
+ for token, token_id in all_tokens.items():
if str(token_id) not in config["added_tokens_decoder"]:
# Determine if token should be marked as "special"
# tool_call tokens are NOT special in Qwen2.5
@@ -207,7 +224,7 @@ def patch_tokenizer_config(output_dir: str) -> None:
if "additional_special_tokens" not in config:
config["additional_special_tokens"] = []
- for token in VIBEVOICE_AUDIO_TOKENS.keys():
+ for token in audio_tokens.keys():
if token not in config["additional_special_tokens"]:
config["additional_special_tokens"].append(token)
@@ -251,10 +268,11 @@ def patch_tokenizer_config(output_dir: str) -> None:
print(f"Patched {config_path}")
-def patch_tokenizer_json(output_dir: str) -> None:
+def patch_tokenizer_json(output_dir: str, streaming: bool = False) -> None:
"""
Patch tokenizer.json with VibeVoice audio tokens.
"""
+ _, all_tokens = _token_maps(streaming)
tokenizer_path = os.path.join(output_dir, "tokenizer.json")
with open(tokenizer_path, "r", encoding="utf-8") as f:
@@ -267,7 +285,7 @@ def patch_tokenizer_json(output_dir: str) -> None:
existing_ids.add(token_entry.get("id"))
# Add ALL extended tokens (Qwen2.5 + VibeVoice audio)
- for token, token_id in ALL_EXTENDED_TOKENS.items():
+ for token, token_id in all_tokens.items():
if token_id not in existing_ids:
# Determine if token should be marked as "special"
is_special = token not in ("<tool_call>", "</tool_call>", "<|fim_prefix|>",
@@ -312,10 +330,11 @@ def generate_added_tokens_json(output_dir: str) -> None:
print(f"Generated {output_path}")
-def generate_special_tokens_map_json(output_dir: str) -> None:
+def generate_special_tokens_map_json(output_dir: str, streaming: bool = False) -> None:
"""
Generate special_tokens_map.json with VibeVoice special tokens.
"""
+ audio_tokens, _ = _token_maps(streaming)
# Build the special tokens map
special_tokens_map = {
"additional_special_tokens": [],
@@ -325,7 +344,7 @@ def generate_special_tokens_map_json(output_dir: str) -> None:
}
# Add audio tokens as additional_special_tokens
- for token in VIBEVOICE_AUDIO_TOKENS.keys():
+ for token in audio_tokens.keys():
special_tokens_map["additional_special_tokens"].append({
"content": token,
"lstrip": False,
@@ -352,10 +371,11 @@ def generate_special_tokens_map_json(output_dir: str) -> None:
print(f"Generated {output_path}")
-def generate_vibevoice_tokenizer_files(output_dir: str, qwen_model: str = DEFAULT_QWEN_MODEL) -> None:
+def generate_vibevoice_tokenizer_files(output_dir: str, qwen_model: str = DEFAULT_QWEN_MODEL,
+ streaming: bool = False) -> None:
"""
Generate all 6 VibeVoice tokenizer files.
-
+
Files generated:
1. vocab.json - from Qwen2.5 (unchanged)
2. merges.txt - from Qwen2.5 (unchanged)
@@ -365,21 +385,21 @@ def generate_vibevoice_tokenizer_files(output_dir: str, qwen_model: str = DEFAUL
6. special_tokens_map.json - generated with VibeVoice tokens
"""
print(f"=== Generating VibeVoice tokenizer files to {output_dir} ===\n")
-
+
# Step 1: Download base files from Qwen2
download_qwen_tokenizer_files(output_dir, qwen_model)
-
+
# Step 2: Patch tokenizer_config.json
- patch_tokenizer_config(output_dir)
-
+ patch_tokenizer_config(output_dir, streaming)
+
# Step 3: Patch tokenizer.json
- patch_tokenizer_json(output_dir)
-
+ patch_tokenizer_json(output_dir, streaming)
+
# Step 4: Generate added_tokens.json
generate_added_tokens_json(output_dir)
-
+
# Step 5: Generate special_tokens_map.json
- generate_special_tokens_map_json(output_dir)
+ generate_special_tokens_map_json(output_dir, streaming)
print(f"\nβœ… All 6 tokenizer files generated in {output_dir}")
@@ -544,6 +564,12 @@ def main():
help=f"Qwen model to download base tokenizer from (default: {DEFAULT_QWEN_MODEL})"
)
+ parser.add_argument(
+ "--streaming",
+ action="store_true",
+ help="Use the streaming token layout (<|text_chunk_end|> at 151665)"
+ )
+
args = parser.parse_args()
# Determine output directory
@@ -556,7 +582,7 @@ def main():
try:
# Generate files
- generate_vibevoice_tokenizer_files(output_dir, args.qwen_model)
+ generate_vibevoice_tokenizer_files(output_dir, args.qwen_model, args.streaming)
# Compare if requested
if args.compare: