Open-Source Frontier Voice AI
Python
Serve streaming ASR through the vLLM plugin
AlexTYJca6529ae5982
diff --git a/docs/vibevoice-vllm-asr-streaming.md b/docs/vibevoice-vllm-asr-streaming.mdnew file mode 100644index 0000000..e2a763e--- /dev/null+++ b/docs/vibevoice-vllm-asr-streaming.md@@ -0,0 +1,213 @@+# VibeVoice vLLM Streaming ASR Deployment++<a href="https://huggingface.co/microsoft/VibeVoice-ASR-Streaming"><img alt="Huggingface" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-VibeVoice--ASR--Streaming-blue"></a>++Deploy VibeVoice ASR Streaming model as a high-performance API service using [vLLM](https://github.com/vllm-project/vllm). A session pushes audio while it is still being spoken and gets one transcript back per chunk, so text appears before the audio ends.++**Non-streaming:** [vLLM-asr](./vibevoice-vllm-asr.md)<br>++## π₯ Key Features++- **π€ True Streaming**: Transcribe from a live microphone or a growing stream, one chunk at a time+- **π Prefix-Cached Sessions**: Each chunk reuses the KV cache of every chunk before it, so cost stays flat as a session grows+- **π‘ WebSocket + REST**: `/v1/stream` for live audio, `/v1/transcribe` and `/v1/chat/completions` for whole files+- **π Plugin Architecture**: No vLLM source code modification required - just install and run+- **ποΈ Checkpoint-Defined Chunking**: Chunk and lookahead are read from the checkpoint, so a model always runs at the geometry it was trained on++## π οΈ Installation++Using Official vLLM Docker Image (Recommended)++1. Clone the repository+```bash+git clone https://github.com/microsoft/VibeVoice.git+cd VibeVoice+```++2. Launch the server (background mode)+```bash+docker run -d --gpus all --name vibevoice-vllm-streaming \+ --ipc=host \+ -p 8000:8000 \+ -e VIBEVOICE_FFMPEG_MAX_CONCURRENCY=64 \+ -e PYTORCH_ALLOC_CONF=expandable_segments:True \+ -v $(pwd):/app \+ -w /app \+ --entrypoint bash \+ vllm/vllm-openai:v0.14.1 \+ -c "python3 /app/vllm_plugin/scripts/start_streaming_server.py"+```++## β‘ Multi-GPU Deployment++### Tensor Parallel++Split a single model across 2 GPUs (useful if GPU memory is limited):++```bash+docker run -d --gpus '"device=0,1"' --name vibevoice-vllm-streaming \+ --ipc=host \+ -p 8000:8000 \+ -e VIBEVOICE_FFMPEG_MAX_CONCURRENCY=64 \+ -e PYTORCH_ALLOC_CONF=expandable_segments:True \+ -v $(pwd):/app \+ -w /app \+ --entrypoint bash \+ vllm/vllm-openai:v0.14.1 \+ -c "python3 /app/vllm_plugin/scripts/start_streaming_server.py --tp 2"+```++### Data Parallel++`--dp` is not available here. A session's audio lives in one replica's prefix cache, and round-robin would send its later chunks to a replica that never saw the earlier ones. To use N GPUs for throughput, run one server per GPU on its own port and route **whole sessions**, not individual chunks:++```bash+for i in 0 1 2 3; do+ docker run -d --gpus "\"device=$i\"" --name vibevoice-vllm-streaming-$i \+ --ipc=host \+ -p $((8000 + i)):8000 \+ -e PYTORCH_ALLOC_CONF=expandable_segments:True \+ -v $(pwd):/app \+ -w /app \+ --entrypoint bash \+ vllm/vllm-openai:v0.14.1 \+ -c "python3 /app/vllm_plugin/scripts/start_streaming_server.py"+done+```++3. View logs+```bash+docker logs -f vibevoice-vllm-streaming+```++> **Note**:+> - The `-d` flag runs the container in background (detached mode)+> - Use `docker stop vibevoice-vllm-streaming` to stop the service+> - The model will be downloaded to HuggingFace cache (`~/.cache/huggingface`) inside the container++## π Usages++### Usage 1: Test the API++Once the streaming server is running, test it with the provided script:++```bash+# Basic streaming transcription+docker exec -it vibevoice-vllm-streaming python3 vllm_plugin/tests/test_api_streaming.py /app/audio.wav++# With hotwords for better recognition of specific terms+docker exec -it vibevoice-vllm-streaming python3 vllm_plugin/tests/test_api_streaming.py /app/audio.wav --hotwords "Microsoft,VibeVoice"+```++Whole-file endpoints are available too, for callers that do not stream:++```bash+docker exec -it vibevoice-vllm-streaming python3 vllm_plugin/tests/test_api.py /app/audio.wav+```++> **Note**:+> - The audio/video file must be inside the mounted directory (`/app` in the container). Copy your files to the VibeVoice folder before testing.+> - Hotwords help improve recognition of domain-specific terms like proper nouns, technical terms, and speaker names.++### Usage 2: Launch the demo page++Start the demo in a second container on the host network, so that `--api_url` reaches the server already running there:++```bash+docker run -d --network host --name vibevoice-vllm-streaming-demo \+ -v $(pwd):/app \+ -w /app \+ --entrypoint bash \+ vllm/vllm-openai:v0.14.1 \+ -c "python3 /app/vllm_plugin/scripts/fastapi_asr_streaming_demo_api.py --api_url http://localhost:8000"+```++Open `http://localhost:7860`, then record from the microphone or pick a file. The page keeps a WebSocket open for the whole recording, so text appears while you are still speaking.++Nothing extra to install: the demo needs only `fastapi`, `uvicorn`, `httpx` and `websockets`, all of which the vLLM image already ships.++### Usage 3: Share the demo publicly++Add `--cloudflared` to get a public link. The flag downloads the `cloudflared` binary to `~/.local/bin` on first use, so the container needs outbound network:++```bash+docker run -d --network host --name vibevoice-vllm-streaming-demo \+ -v $(pwd):/app \+ -w /app \+ --entrypoint bash \+ vllm/vllm-openai:v0.14.1 \+ -c "python3 /app/vllm_plugin/scripts/fastapi_asr_streaming_demo_api.py --api_url http://localhost:8000 --cloudflared"+```++The link is printed once the tunnel is up:++```bash+docker logs vibevoice-vllm-streaming-demo+```++```+relaying to http://localhost:8000, open http://localhost:7860++ public URL: https://xxxx-xxxx-xxxx.trycloudflare.com+```++> **Note**: use the `https` link rather than a plain `http` address when the page is not on `localhost` β browsers only grant microphone access in a secure context.++### Demo Options++| Flag | Description | Default |+|------|-------------|---------|+| `--api_url URL` | Streaming server URL | `http://localhost:8000` |+| `--port PORT` | Local demo port | `7860` |+| `--host HOST` | Address to bind | `0.0.0.0` |+| `--cloudflared` | Create a public link using cloudflared tunnel | off |+| `--max_new_tokens N` | Default max new tokens per chunk | `256` |++### Endpoints++| Endpoint | Description |+|----------|-------------|+| `WS /v1/stream` | Live session: JSON config, then float32 PCM frames, then `"end"` |+| `POST /v1/transcribe` | One whole file, chunked internally, full transcript in the reply |+| `POST /v1/transcribe_batch` | Several files in one request |+| `POST /v1/chat/completions` | OpenAI-compatible, for existing clients |+| `GET /v1/config` | Sample rate and chunk geometry of the loaded checkpoint |++### Environment Variables++| Variable | Description | Default |+|----------|-------------|---------|+| `VIBEVOICE_FFMPEG_MAX_CONCURRENCY` | Cap on simultaneous FFmpeg decodes | unset β uncapped; the `docker run` commands above set `64` |+| `PYTORCH_ALLOC_CONF` | PyTorch memory allocator config | unset; the `docker run` commands above set `expandable_segments:True` |++Unlike `start_server.py`, the streaming launcher imposes no FFmpeg limit of its own, so an unset `VIBEVOICE_FFMPEG_MAX_CONCURRENCY` really does mean unlimited.++## π Performance Tips++1. **GPU Memory**: Use `--gpu-memory-utilization 0.9` for maximum throughput if you have dedicated GPU+2. **Session Length**: `--max-model-len` is what caps how long one session can run; raise it for hour-long streams+3. **Concurrent Sessions**: Raise `--mm-processor-cache-gb` before raising concurrency β it must hold every in-flight session's audio windows++## π¨ Troubleshooting++### Common Issues++1. **"CUDA out of memory"**+ - Reduce `--gpu-memory-utilization`+ - Reduce `--mm-processor-cache-gb`+ - Use smaller `--max-model-len`++2. **"is not a streaming checkpoint"**+ - The checkpoint has no `chunk_frames` in `preprocessor_config.json`+ - Serve it with `start_server.py` instead++3. **"tokenizer layout mismatch"**+ - `<|text_chunk_end|>` is missing or sits at another id, so nothing would ever end a chunk. The server refuses to start rather than serve a first chunk and then empty ones+ - Regenerate: `python3 -m vllm_plugin.tools.generate_tokenizer_files --output MODEL_PATH --streaming`++4. **"Cache miss" or a session stalling after many chunks**+ - Raise `--mm-processor-cache-gb`, or lower the number of concurrent sessions++5. **"Plugin not loaded"**+ - Verify installation: `pip show vibevoice`+ - Check entry point: `pip show -f vibevoice | grep entry`diff --git a/vllm_plugin/__init__.py b/vllm_plugin/__init__.pyindex 989acb4..df70e86 100644--- a/vllm_plugin/__init__.py+++ b/vllm_plugin/__init__.py@@ -56,6 +56,10 @@ def register_vibevoice():# This name must match the "architectures" list in config.jsonModelRegistry.register_model("VibeVoice", VibeVoiceForCausalLM)ModelRegistry.register_model("VibeVoiceForASRTraining", VibeVoiceForCausalLM)+ # Streaming checkpoints declare their own architecture name. The decoder is+ # identical -- streaming only changes how audio and text interleave in the+ # prompt, which is handled in _get_prompt_updates.+ ModelRegistry.register_model("VibeVoiceForASRStreamingTraining", VibeVoiceForCausalLM)# Note: This function is called via vllm.general_plugins entry pointdiff --git a/vllm_plugin/asr_streaming.py b/vllm_plugin/asr_streaming.pynew file mode 100644index 0000000..c97412f--- /dev/null+++ b/vllm_plugin/asr_streaming.py@@ -0,0 +1,332 @@+"""The streaming ASR sequence format, apart from the server that drives it.++The model was trained on a single interleaved sequence++ Prompt + [<sp_start> AudioWindow <sp_end> Text <|text_chunk_end|>] * N++so a client's job is to cut audio into the windows training used and rebuild+that sequence chunk by chunk. This module does that, plus the inverse: turning+the per-chunk text back into readable segments. Three processes need one or+both -- the vLLM engine, the streaming server, and the browser demo -- and a+copy of the chunk arithmetic in each would be free to drift.++Nothing here imports vLLM or torch at module level, so the demo can reach it by+bare import (``vllm_plugin/`` on sys.path, ``import asr_streaming``) without+executing ``vllm_plugin/__init__.py`` and its model registration.+"""+import json+import os+import re+import tempfile+from dataclasses import dataclass+from typing import Dict, List, Optional++import numpy as np++# The model signals end-of-chunk with this token rather than EOS. It is the+# first free slot after Qwen2.5's vocabulary and was trained, not synthesised.+# The tokenizer reads it from the checkpoint rather than defining it, so a+# checkpoint whose files disagree with this literal resolves to something else+# -- or to nothing, in which case every chunk runs to max_tokens.+TEXT_CHUNK_END_ID = 151665++# Plain text, no chat template, no BOS. Not the processor's SYSTEM_PROMPT ++# <|im_start|> template, which serves the non-streaming single-shot path only.+_PROMPT_HEAD = (+ "You are a helpful assistant that transcribes audio input into text output. "+ "Please transcribe the following audios streamingly with these keys: "+ "speaker, content"+)+++@dataclass(frozen=True)+class ChunkGeometry:+ """How a checkpoint cuts audio: read from it, never guessed.++ A window carries ``chunk_frames`` of new audio plus ``lookahead_frames``+ that the next window repeats. The released checkpoints differ here -- 15+ frames (2.0s) and 22 frames (2.9333s), 4 frames of lookahead either way --+ and serving one at the other's geometry does not raise, it just transcribes+ worse. So it is read from ``preprocessor_config.json``, which every+ streaming checkpoint records it in.+ """++ sample_rate: int+ frame_samples: int+ chunk_frames: int+ lookahead_frames: int++ @property+ def window_frames(self) -> int:+ return self.chunk_frames + self.lookahead_frames++ @property+ def chunk_samples(self) -> int:+ return self.chunk_frames * self.frame_samples++ @property+ def window_samples(self) -> int:+ return self.window_frames * self.frame_samples++ @property+ def chunk_seconds(self) -> float:+ """The advance, which segment times and the demo's capture use.++ Chunk k is fed audio ``[k*advance, k*advance + window)`` and the text it+ returns covers ``[k*advance, (k+1)*advance)`` -- the trailing lookahead+ is audio whose transcript arrives with the next chunk.+ """+ return self.chunk_samples / self.sample_rate++ @classmethod+ def from_pretrained(cls, model_path: str) -> "ChunkGeometry":+ """Load the geometry from a local checkpoint directory or a HF repo id."""+ name = "preprocessor_config.json"+ if os.path.isdir(model_path):+ config_path = os.path.join(model_path, name)+ else:+ from huggingface_hub import hf_hub_download++ config_path = hf_hub_download(model_path, name)++ with open(config_path) as f:+ cfg = json.load(f)++ missing = [k for k in ("chunk_frames", "lookahead_frames") if k not in cfg]+ if missing:+ raise ValueError(+ f"{config_path} has no {', '.join(missing)}, so {model_path} is "+ "not a streaming checkpoint. Serve it with start_server.py instead."+ )+ return cls(+ sample_rate=cfg["target_sample_rate"],+ frame_samples=cfg["speech_tok_compress_ratio"],+ chunk_frames=cfg["chunk_frames"],+ lookahead_frames=cfg["lookahead_frames"],+ )++ def describe(self) -> str:+ return (f"chunk {self.chunk_frames} frames ({self.chunk_seconds:.4f}s), "+ f"lookahead {self.lookahead_frames} frames, "+ f"window {self.window_samples} samples @ {self.sample_rate} Hz")+++def build_prompt(context_info: Optional[str] = None) -> str:+ """Training-format prompt, with the optional hotword/context suffix."""+ if context_info:+ return f"{_PROMPT_HEAD} and extra info: {context_info}\n"+ return f"{_PROMPT_HEAD}\n"+++def split_windows(audio, geometry: ChunkGeometry, pad_last: bool = True) -> List[np.ndarray]:+ """Cut audio into overlapping windows, matching ``split_then_encode``.++ ``pad_last`` zero-pads the final window to a full window. That is not+ cosmetic: training split already-encoded features, so its last chunk had+ ``floor(remainder/frame)`` frames, while encoding a short segment rounds+ *up*. Without the pad the final window would carry one extra frame and no+ lookahead -- off-distribution exactly where the sequence ends.+ """+ audio = np.asarray(audio, dtype=np.float32)+ total = audio.shape[0]+ window_samples = geometry.window_samples++ windows = []+ start = 0+ while start < total:+ seg = audio[start:min(start + window_samples, total)]+ if seg.shape[0]:+ if pad_last and seg.shape[0] < window_samples:+ seg = np.pad(seg, (0, window_samples - seg.shape[0]))+ windows.append(seg)+ # Advance by the text boundary, not by the window: a window that already+ # reaches the end of the audio still only transcribes its first chunk,+ # so the trailing lookahead needs a window of its own.+ start = min(start + geometry.chunk_samples, total)+ return windows+++def load_audio_bytes(audio_bytes: bytes, target_sr: int) -> np.ndarray:+ """Decode in-memory audio to the mono float32 the windows are cut from.++ The suffix is a placeholder: ffmpeg sniffs the container from the bytes, so+ an mp3 or a webm opus blob written to a ``.wav`` name decodes fine.++ Decodes at the native rate and resamples with librosa rather than letting+ ffmpeg do both, which is what the streaming checkpoints were evaluated with.+ The two resamplers differ by ~1% RMS, and near-tied greedy argmaxes turn+ that into different words.+ """+ from vibevoice.processor.audio_utils import load_audio_use_ffmpeg++ with tempfile.NamedTemporaryFile(suffix=".wav", delete=True) as tmp:+ tmp.write(audio_bytes)+ tmp.flush()+ audio, sr = load_audio_use_ffmpeg(tmp.name, resample=False)++ if sr != target_sr:+ import librosa++ audio = librosa.resample(audio, orig_sr=sr, target_sr=target_sr)+ return audio+++# --- Chunks back into readable segments -------------------------------------+# The model emits one block per chunk, each re-stating the speaker, each free to+# end mid-sentence. Shown verbatim that is thirty fragments a minute. The job+# below merges them into paragraphs that end where sentences do, identically for+# every caller -- a rule that differs between the demo and the server is a bug+# nobody notices until the two are compared side by side.++_CHUNK_SPEAKER_RE = re.compile(r"Speaker\s+(\d+)\s*:")++# Chunks merge until the segment is at least this long, then close at the next+# sentence-ending punctuation. The cap stops a stretch with no punctuation from+# growing one card until it swallows the whole clip; kept at twice the minimum,+# so punctuation and not the cap is what usually ends a card.+SEGMENT_MIN_S = 10.0+SEGMENT_MAX_S = 20.0++# Two dialects, because the rule differs. A CJK stop mark is unambiguous. An+# ASCII period is not -- requiring whitespace or end-of-text after it keeps+# "3.5" and "Mr. Smith" from splitting a segment in half.+_SENTENCE_END_RE = re.compile(+ r"[γοΌοΌβ¦]+[ββγγοΌγγ]*"+ r"|[.!?]+[\"'ββ)\]]*(?=\s|$)"+)++# Backreference, so only a repeat of the *same* tag collapses: [Noise][Noise]+# folds, [Noise][Laughter] does not. Tags become adjacent only when blocks are+# merged into one card, which is why this runs here and not per block.+_REPEAT_TAG_RE = re.compile(r"(\[[^\[\]]*\])(?:\s*\1)+")++# A card is one paragraph, so the line breaks the model emits inside a block+# fold to single spaces -- except between two CJK characters, where a space is+# not a word separator and just leaves a visible gap.+_WS_RUN_RE = re.compile(r"\s+")+_CJK = r"γ-γΏγ-γΏδΈ-ιΏΏοΌ-ο½₯"+_CJK_SPACE_RE = re.compile(rf"(?<=[{_CJK}]) (?=[{_CJK}])")+++def tidy_body(text: str) -> str:+ """Fold a merged card's text back into one paragraph."""+ return _CJK_SPACE_RE.sub("", _WS_RUN_RE.sub(" ", text)).strip()+++def chunk_segments(chunk_texts: List[str],+ geometry: ChunkGeometry,+ duration: Optional[float] = None) -> List[Dict]:+ """Merge the per-chunk stream into readable, sentence-aligned segments.++ Times stay arithmetic rather than inferred: chunk k *is* the k-th window, so+ every boundary is a multiple of ``geometry.chunk_seconds``. Two things split+ a chunk internally -- a speaker change, and a sentence end past the+ SEGMENT_MIN_S mark. In both cases the words go to the right card while the+ timestamp stays on the chunk edge.++ The speaker carries forward because the model only re-states it when it+ changes, so a chunk with no label belongs to the previous speaker.++ ``duration`` only ever clamps the tail. While a clip is still streaming its+ real duration is unknown, and the chunk count is what defines the timeline.+ """+ advance = geometry.chunk_seconds+ segments: List[Dict] = []+ cur_start = 0.0+ cur_text = ""+ speaker = None++ def flush(end: float) -> None:+ nonlocal cur_start, cur_text+ body = tidy_body(_REPEAT_TAG_RE.sub(r"\1", cur_text))+ # Silence really does come back empty; those chunks fold into whichever+ # segment is open instead of becoming a card of their own.+ if body:+ if end > cur_start:+ segments.append({+ "Start": round(cur_start, 2),+ "End": round(end, 2),+ "Speaker": speaker if speaker is not None else "?",+ "Content": body,+ })+ elif segments:+ # No timeline left to give this text -- the remainder after a+ # sentence split in the final chunk. Dropping it is how words go+ # missing off the end of a transcript, so it joins the card it+ # was split from instead.+ prev = segments[-1]+ prev["Content"] = tidy_body(+ _REPEAT_TAG_RE.sub(r"\1", prev["Content"] + " " + body))+ cur_start = end+ cur_text = ""++ for k, text in enumerate(chunk_texts):+ start = k * advance+ end = start + advance+ if duration and duration > start:+ end = min(end, duration)++ # split() on a single-group pattern yields [pre, id, text, id, text...],+ # which is what lets a mid-chunk speaker change break the segment.+ pieces = _CHUNK_SPEAKER_RE.split(text)+ added = pieces[0]+ # Two people can easily both talk inside one chunk. Closing the card at+ # ``start`` would put the boundary where the previous card already+ # ended, and a zero-width card carries no text -- so the cut is placed+ # proportionally through the chunk, by how much of its text is consumed.+ chunk_chars = max(1, sum(len(p) for p in pieces))+ consumed = len(pieces[0])+ for i in range(1, len(pieces), 2):+ if speaker is not None and pieces[i] != speaker:+ cur_text += added+ flush(max(start + (end - start) * (consumed / chunk_chars), cur_start))+ added = ""+ speaker = pieces[i]+ added += pieces[i + 1]+ consumed += len(pieces[i]) + len(pieces[i + 1])+ cur_text += added++ span = end - cur_start+ if span >= SEGMENT_MIN_S:+ # Only a stop mark inside *this* chunk is usable: one further back+ # would strand the text between it and here in the wrong card.+ last = None+ for last in _SENTENCE_END_RE.finditer(added):+ pass+ if last is not None:+ carry = added[last.end():]+ cur_text = cur_text[:len(cur_text) - len(carry)] if carry else cur_text+ flush(end)+ cur_text = carry+ continue+ if span >= SEGMENT_MAX_S:+ flush(end)++ total = len(chunk_texts) * advance+ flush(min(total, duration) if duration and duration > cur_start else total)+ return segments+++def segments_to_text(segments: List[Dict]) -> str:+ """Render segments as the plain transcript, one line per segment."""+ return "\n".join(+ f"Speaker {seg.get('Speaker', '?')}: {seg.get('Content', '')}"+ for seg in segments+ )+++def _srt_time(seconds: float) -> str:+ ms = int(round(seconds * 1000))+ h, ms = divmod(ms, 3_600_000)+ m, ms = divmod(ms, 60_000)+ s, ms = divmod(ms, 1000)+ return f"{h:02d}:{m:02d}:{s:02d},{ms:03d}"+++def segments_to_srt(segments: List[Dict]) -> str:+ """SRT, driven by the same segments as everything else so they agree."""+ return "\n".join(+ f"{i}\n{_srt_time(seg['Start'])} --> {_srt_time(seg['End'])}\n"+ f"Speaker {seg.get('Speaker', '?')}: {seg.get('Content', '')}\n"+ for i, seg in enumerate(segments, 1)+ )diff --git a/vllm_plugin/asr_streaming_server.py b/vllm_plugin/asr_streaming_server.pynew file mode 100644index 0000000..90d315b--- /dev/null+++ b/vllm_plugin/asr_streaming_server.py@@ -0,0 +1,851 @@+"""FastAPI server for VibeVoice streaming ASR on vLLM V1.++The model transcribes one chunk at a time off a growing interleaved sequence,+so a request here is a *session*, not a single generate call. Three layers,+bottom-up: the engine holder, the per-chunk session that drives it, and the+transport that exposes it. The sequence format itself lives next door in+``asr_streaming.py``, where it reads as a record of the training run rather+than as this server's internals.++This serves the same model as ``start_server.py``, and the choice between them+is about who holds the session. A stock ``vllm serve`` speaks request/response+only, so a client there has to re-send every audio window on every chunk.+Holding the engine in-process lets a session keep its windows engine-side+behind stable uuids -- flat payload per turn -- and lets the server offer a+WebSocket, which is what a live microphone needs.++Endpoints:+ GET /health, /healthz, /v1/models, /v1/config+ POST /v1/transcribe whole clip, one response, chunk texts included+ POST /v1/transcribe_batch several clips concurrently+ POST /v1/chat/completions OpenAI dialect, the one both backends share+ WS /v1/stream true streaming: send audio frames, get text back++Run:+ python3 -m vllm_plugin.asr_streaming_server --model /path/to/checkpoint+"""+import argparse+import asyncio+import base64+import json+import logging+import re+import time+import uuid+from contextlib import asynccontextmanager+from dataclasses import dataclass+from typing import Any, Dict, List, Optional, Sequence+from urllib.parse import urlparse++import numpy as np+import requests+import uvicorn+from fastapi import (FastAPI, HTTPException, Request, WebSocket,+ WebSocketDisconnect)+from fastapi.responses import StreamingResponse+from pydantic import BaseModel, Field++from vllm_plugin.asr_streaming import (ChunkGeometry, TEXT_CHUNK_END_ID,+ build_prompt, chunk_segments,+ load_audio_bytes, segments_to_srt,+ split_windows)++logger = logging.getLogger("vibevoice.streaming")++MAX_AUDIO_BYTES = 256 * 1024 * 1024+MAX_BATCH_ITEMS = 64+++@dataclass+class ServerConfig:+ """Everything the server needs that is not read from the checkpoint."""++ model: str+ served_model_name: str = "vibevoice"+ dtype: str = "bfloat16"+ gpu_memory_utilization: float = 0.85+ max_model_len: int = 16384+ tensor_parallel_size: int = 1+ # One window per streaming chunk, so this caps session length, not audio+ # size: 512 windows is about 17 minutes at a 2.0s chunk, proportionally+ # longer for a checkpoint with a larger one.+ max_audio_windows: int = 512+ # Sized to hold every window of every in-flight session. An LRU eviction+ # here is not a slowdown but an error: a request whose history was evicted+ # arrives with None for that window and the engine raises "Cache miss".+ mm_processor_cache_gb: float = 16.0+ enforce_eager: bool = False+ enable_prefix_caching: bool = True+ resend_all_audio: bool = False+++# --- engine ---------------------------------------------------------------+# The audio never leaves the request: it rides along as multimodal data and the+# plugin's registered processor VAE-encodes it inside the engine. Three things+# follow, and a streaming session depends on all three:+#+# * automatic prefix caching, which is where all of streaming's speedup lives+# -- chunk k re-reads chunks 0..k-1 as a prefix instead of recomputing them;+# * the scheduler's encoder-skip, which drops the VAE call for any window+# already covered by the KV cache;+# * continuous batching across sessions, rather than one session at a time.++class VibeVoiceAsyncEngine:+ """Owns a single ``AsyncLLM``; all sessions share it."""++ def __init__(self, config: ServerConfig, geometry: ChunkGeometry):+ from vllm.engine.arg_utils import AsyncEngineArgs+ from vllm.v1.engine.async_llm import AsyncLLM++ self.config = config+ self.geometry = geometry++ engine_args = AsyncEngineArgs(+ model=config.model,+ # The checkpoint's root config says float32; loading it as such+ # would more than double the weight footprint for no benefit.+ dtype=config.dtype,+ gpu_memory_utilization=config.gpu_memory_utilization,+ max_model_len=config.max_model_len,+ tensor_parallel_size=config.tensor_parallel_size,+ trust_remote_code=True,+ # Non-negotiable for streaming: without it every chunk re-prefills+ # the whole session and the cost goes quadratic. Exposed only so the+ # APC-on/APC-off token equality check has something to turn off.+ enable_prefix_caching=config.enable_prefix_caching,+ limit_mm_per_prompt={"audio": config.max_audio_windows},+ mm_processor_cache_gb=config.mm_processor_cache_gb,+ enforce_eager=config.enforce_eager,+ )+ self.engine = AsyncLLM.from_engine_args(engine_args)++ # Loaded by concrete class, not AutoTokenizer: the checkpoint's+ # tokenizer_config.json names VibeVoiceASRTextTokenizerFast, which is+ # not importable from transformers. The session layer needs this class+ # specifically anyway -- text_chunk_end_id lives on it.+ from vibevoice.modular.modular_vibevoice_text_tokenizer import (+ VibeVoiceASRTextTokenizerFast)++ self.tokenizer = VibeVoiceASRTextTokenizerFast.from_pretrained(config.model)++ # Nothing downstream notices this on its own: the weights still load,+ # start-up is clean, the first chunk transcribes correctly -- and then+ # stop_token_ids never matches what the model emits and every later+ # chunk comes back empty with the request still returning 200.+ if self.tokenizer.text_chunk_end_id != TEXT_CHUNK_END_ID:+ raise RuntimeError(+ f"tokenizer layout mismatch: <|text_chunk_end|> resolved to "+ f"{self.tokenizer.text_chunk_end_id}, expected {TEXT_CHUNK_END_ID}.\n"+ f" model path: {config.model}\n"+ f" id {TEXT_CHUNK_END_ID} there is "+ f"{self.tokenizer.convert_ids_to_tokens(TEXT_CHUNK_END_ID)!r}\n"+ "Regenerate the tokenizer with "+ "vllm_plugin/tools/generate_tokenizer_files.py.")++ async def generate(+ self,+ prompt_token_ids: Sequence[int],+ audio_items: Sequence[Optional[Any]],+ audio_uuids: Sequence[str],+ sampling_params: Any,+ request_id: str,+ ):+ """Run one chunk turn and return the final ``RequestOutput``.++ ``audio_items`` carries ``None`` for every window the engine has already+ processed; ``audio_uuids`` must stay byte-identical across turns for+ those, since that identifier is what keeps the prefix-cache block hashes+ stable.+ """+ from vllm.inputs import TokensPrompt++ if len(audio_items) != len(audio_uuids):+ raise ValueError(+ f"audio items ({len(audio_items)}) and uuids "+ f"({len(audio_uuids)}) must line up one-to-one")++ prompt = TokensPrompt(+ prompt_token_ids=list(prompt_token_ids),+ multi_modal_data={"audio": list(audio_items)},+ multi_modal_uuids={"audio": list(audio_uuids)},+ )++ final = None+ async for out in self.engine.generate(prompt, sampling_params, request_id):+ final = out+ if final is None:+ raise RuntimeError(f"engine produced no output for {request_id}")+ return final++ async def shutdown(self):+ self.engine.shutdown()+++# --- session --------------------------------------------------------------+# The reference implementation (``modeling_vibevoice_asr.py::streaming_generate``)+# walks the interleaved sequence with a hand-held KV cache. vLLM has no API for+# "keep my KV cache and append", so the equivalent here is one request per chunk+# carrying the whole sequence so far, with prefix caching making the repeated+# prefix free. That inverts the usual advice -- resending the prompt is normally+# waste -- but it is what lets the engine batch many sessions continuously.+#+# Two invariants this layer exists to protect:+#+# * The token sequence must be byte-identical to the HF path. No newline after+# <sp_end>; <|text_chunk_end|> appended after *every* chunk including one cut+# short by EOS.+# * The accumulated prefix is the *pre-expansion* one, holding a single+# <|AUDIO|> per window. The plugin's processor expands each into+# [sp_start] + N*[sp_pad] + [sp_end]; feeding an expanded sequence back in+# would leave no placeholder to match.++_RESIDUAL_SPECIALS = (+ "<|text_chunk_end|>",+ "<|object_ref_start|>",+ "<|object_ref_end|>",+ "<|box_start|>",+ "<|speech_start|>",+ "<|speech_end|>",+ "<|speech_pad|>",+)+++def _strip_specials(text: str) -> str:+ for tok in _RESIDUAL_SPECIALS:+ text = text.replace(tok, "")+ return text+++class StreamingSession:+ """One ASR conversation. Not thread-safe; drive it from a single task."""++ def __init__(+ self,+ engine: VibeVoiceAsyncEngine,+ session_id: str,+ *,+ context_info: Optional[str] = None,+ max_new_tokens: int = 256,+ temperature: float = 0.0,+ top_p: float = 1.0,+ repetition_penalty: float = 1.0,+ ):+ from vllm import SamplingParams++ self.engine = engine+ self.config = engine.config+ self.geometry = engine.geometry+ self.session_id = session_id++ tokenizer = engine.tokenizer+ self.audio_token_id = tokenizer.convert_tokens_to_ids("<|AUDIO|>")+ self.text_chunk_end_id = tokenizer.text_chunk_end_id+ if self.audio_token_id is None or self.audio_token_id < 0:+ raise RuntimeError("tokenizer has no <|AUDIO|> placeholder")++ self._tokens = list(+ tokenizer.encode(build_prompt(context_info), add_special_tokens=False))+ self._windows: List[np.ndarray] = []+ self._turn = 0+ self.max_new_tokens = max_new_tokens+ # Per-chunk generated token ids, kept for the HF token-level comparison.+ # Text equality is not enough: two token sequences can detokenize to the+ # same string, and a divergence that starts invisible compounds.+ self.chunk_token_ids: List[List[int]] = []++ self.sampling_params = SamplingParams(+ temperature=temperature,+ top_p=top_p,+ max_tokens=max_new_tokens,+ # Caveat above 1.0: the reference penalizes only tokens generated+ # within the current chunk, while vLLM penalizes prompt tokens too+ # -- here, the entire session history.+ repetition_penalty=repetition_penalty,+ # The model signals end-of-chunk with this token rather than EOS. It+ # is excluded from the output and re-appended to the prefix by hand,+ # so it lands in the sequence even on an EOS stop.+ stop_token_ids=[self.text_chunk_end_id],+ detokenize=True,+ )++ @property+ def num_windows(self) -> int:+ return len(self._windows)++ async def push(self, window) -> str:+ """Transcribe one window of audio and advance the session."""+ window = np.asarray(window, dtype=np.float32).reshape(-1)+ self._windows.append(window)++ tokens = self._tokens + [self.audio_token_id]+ k = len(self._windows) - 1++ # Two separate ceilings, checked here rather than left to the engine,+ # which reports either as an opaque mid-session request failure. The+ # window cap is fixed per session while the context fills at a rate that+ # depends on how much text each chunk produces, so which one is reached+ # first varies with the audio.+ if len(self._windows) > self.config.max_audio_windows:+ minutes = (self.config.max_audio_windows+ * self.geometry.chunk_seconds / 60)+ raise ValueError(+ f"session exceeded {self.config.max_audio_windows} audio "+ f"windows (~{minutes:.0f} minutes at this checkpoint's chunk). "+ f"Raise --max-audio-windows, or start a new session.")+ # Against what the engine sees, not what is accumulated here: each+ # <|AUDIO|> is one token in the pre-expansion prefix and+ # window_frames + 2 once the processor expands it. Counting the prefix+ # as-is is short by window_frames + 1 per window -- 20 at the released+ # geometry, enough that the engine hits its own limit hundreds of+ # windows earlier and reports the opaque failure this check replaces.+ expanded = len(tokens) + len(self._windows) * (+ self.geometry.window_frames + 1)+ if expanded + self.max_new_tokens > self.config.max_model_len:+ raise ValueError(+ f"session outgrew the {self.config.max_model_len}-token context "+ f"after {k} windows ({expanded} tokens once the audio windows "+ f"are expanded). Raise --max-model-len, or start a new session.")++ # Only the new window carries data. The rest are None + a stable uuid,+ # which keeps the per-turn payload flat instead of growing with the+ # session, and keeps the prefix-cache block hashes stable across turns.+ uuids = [f"{self.session_id}-{i}" for i in range(k + 1)]+ if self.config.resend_all_audio:+ audio: List[Optional[np.ndarray]] = list(self._windows)+ else:+ audio = [None] * k + [window]++ self._turn += 1+ req_id = f"{self.session_id}-t{self._turn}"++ try:+ out = await self.engine.generate(+ tokens, audio, uuids, self.sampling_params, req_id)+ except ValueError as exc:+ if "data is not provided" not in str(exc):+ raise+ # The multimodal processor cache evicted a window we claimed was+ # already there. Recoverable only by re-sending every window; if+ # this fires at all, --mm-processor-cache-gb is undersized.+ logger.warning("session %s lost a cached window, resending all %d",+ self.session_id, len(self._windows))+ out = await self.engine.generate(+ tokens, list(self._windows), uuids, self.sampling_params,+ req_id + "-full")++ gen = list(out.outputs[0].token_ids)+ # Defensive: vLLM drops the stop token, but a stop on EOS or on the+ # length cap leaves whatever the model actually emitted.+ while gen and gen[-1] == self.text_chunk_end_id:+ gen.pop()++ self._tokens = tokens + gen + [self.text_chunk_end_id]+ self.chunk_token_ids.append(gen)+ return _strip_specials(out.outputs[0].text)++ async def transcribe(self, audio):+ """Run a whole clip through the session, yielding text per chunk."""+ for window in split_windows(audio, self.geometry):+ yield await self.push(window)+++# --- transport ------------------------------------------------------------++def _fetch_url(url: str, timeout: int) -> bytes:+ """Fetch audio over http(s), refusing other schemes and oversized bodies."""+ if urlparse(url).scheme not in ("http", "https"):+ raise HTTPException(status_code=400,+ detail="audio_url must be http or https")+ try:+ resp = requests.get(url, timeout=timeout, stream=True)+ resp.raise_for_status()+ body = resp.raw.read(MAX_AUDIO_BYTES + 1, decode_content=True)+ except requests.RequestException as exc:+ raise HTTPException(status_code=400,+ detail=f"Failed to fetch audio_url: {exc}") from None+ if len(body) > MAX_AUDIO_BYTES:+ raise HTTPException(status_code=413,+ detail=f"audio_url body exceeds {MAX_AUDIO_BYTES} bytes")+ return body+++class TranscribeRequest(BaseModel):+ audio_base64: Optional[str] = Field(None, description="Base64 audio bytes.")+ audio_url: Optional[str] = Field(None, description="Audio URL if no base64.")+ context_info: Optional[str] = Field(None, description="Hotwords / context.")+ max_tokens: int = Field(256, ge=1, le=2048)+ # Greedy is the reference decoding config; 2.0 is where the demo's slider+ # ends. Rejecting the top of the slider with a 422 kills the session+ # mid-transcription, and clamping silently would make it do nothing.+ temperature: float = Field(0.0, ge=0.0, le=2.0)+ top_p: float = Field(1.0, ge=0.0, le=1.0)+ repetition_penalty: float = Field(1.0, ge=0.5, le=2.0)++ def audio_bytes(self) -> bytes:+ if self.audio_base64:+ return base64.b64decode(self.audio_base64)+ if self.audio_url:+ return _fetch_url(self.audio_url, timeout=10)+ raise HTTPException(status_code=400,+ detail="audio_base64 or audio_url is required")+++class TranscribeResponse(BaseModel):+ text: str+ chunk_texts: List[str]+ total_chunks: int+ audio_duration: float+ generate_time: float+ # Elapsed over duration, the orientation the tests and demos print: below 1+ # is faster than real time. The inverse reads the same but ranks backwards.+ rtf: float+++class BatchAudioItem(BaseModel):+ audio_base64: Optional[str] = None+ audio_url: Optional[str] = None+++class BatchTranscribeRequest(BaseModel):+ audios: List[BatchAudioItem] = Field(..., min_length=1,+ max_length=MAX_BATCH_ITEMS)+ context_info: Optional[str] = None+ max_tokens: int = Field(256, ge=1, le=2048)+ temperature: float = Field(0.0, ge=0.0, le=2.0)+ top_p: float = Field(1.0, ge=0.0, le=1.0)+ repetition_penalty: float = Field(1.0, ge=0.5, le=2.0)+++class BatchTranscribeResponse(BaseModel):+ results: List[TranscribeResponse]+ total_time: float+++def _new_session(engine: VibeVoiceAsyncEngine,+ req: TranscribeRequest) -> StreamingSession:+ return StreamingSession(+ engine,+ f"s-{uuid.uuid4().hex[:16]}",+ context_info=req.context_info,+ max_new_tokens=req.max_tokens,+ temperature=req.temperature,+ top_p=req.top_p,+ repetition_penalty=req.repetition_penalty,+ )+++async def _run_clip(engine: VibeVoiceAsyncEngine,+ req: TranscribeRequest) -> TranscribeResponse:+ # Both of these block, and both go to a thread for the same reason: a URL+ # fetch waits on the network and ffmpeg shells out, either of which would+ # otherwise stall the event loop for every other in-flight session.+ raw = await asyncio.to_thread(req.audio_bytes)+ sample_rate = engine.geometry.sample_rate+ audio = await asyncio.to_thread(load_audio_bytes, raw, sample_rate)++ session = _new_session(engine, req)+ t0 = time.time()+ chunk_texts = [text async for text in session.transcribe(audio)]+ gen_t = time.time() - t0++ duration = len(audio) / sample_rate+ return TranscribeResponse(+ text="".join(chunk_texts),+ chunk_texts=chunk_texts,+ total_chunks=len(chunk_texts),+ audio_duration=duration,+ generate_time=gen_t,+ rtf=(gen_t / duration) if duration > 0 else 0.0,+ )+++# --- OpenAI dialect -------------------------------------------------------+# The session endpoints above are this server's own. Chat-completions is the one+# dialect it shares with ``start_server.py``, and sharing it is the point: same+# served model name, so an existing client can be moved between the two backends+# without being rewritten.+#+# The transcript leaves as a JSON array of segments because that is what the+# dialect already returned -- the non-streaming checkpoint emits segments with+# timestamps directly. A streaming checkpoint emits plain per-chunk blocks, so+# the array is assembled on this side rather than the reply format silently+# changing with the checkpoint.++# Two callers write hotwords into the text part of the user message, and they+# word it differently: the demo puts them below a "context information" heading,+# tests/test_api.py inline after "with extra info:". Everything outside the+# match is the caller's own boilerplate, and feeding that to build_prompt()+# would replace the training prompt with something the model never saw.+_CONTEXT_RE = re.compile(+ r"context information[^\n:]*:[ \t]*\n(.+)|with extra info:[ \t]*([^\n]+)",+ re.IGNORECASE | re.DOTALL)+++def _audio_from_messages(messages: list) -> bytes:+ for msg in messages:+ content = msg.get("content")+ if not isinstance(content, list):+ continue+ for part in content:+ if part.get("type") != "audio_url":+ continue+ url = (part.get("audio_url") or {}).get("url", "")+ if url.startswith("data:"):+ return base64.b64decode(url.split(",", 1)[1])+ return _fetch_url(url, timeout=30)+ raise HTTPException(status_code=400, detail="No audio_url part in messages")+++def _context_from_messages(messages: list) -> Optional[str]:+ """Pull the hotword block out of the text parts, if there is one.++ Text without the heading is ignored rather than passed through -- an+ arbitrary instruction is far more likely to derail the prompt than to act+ as context.+ """+ for msg in messages:+ content = msg.get("content")+ if not isinstance(content, list):+ continue+ for part in content:+ if part.get("type") != "text":+ continue+ m = _CONTEXT_RE.search(part.get("text") or "")+ if m:+ found = (m.group(1) or m.group(2) or "").strip()+ if found:+ return found+ return None+++def _sse(cid: str, model: str, delta: dict, finish: Optional[str] = None) -> str:+ return "data: " + json.dumps({+ "id": cid, "object": "chat.completion.chunk", "model": model,+ "choices": [{"index": 0, "delta": delta, "finish_reason": finish}],+ }) + "\n\n"+++def _segment_delta(seg: dict, index: int) -> str:+ """One element of the JSON array, with the comma that joins it to the last.++ Sent per segment rather than in one piece at the end so cards appear while+ the clip is still being transcribed; a partial array is a valid intermediate+ state for a client that recovers whole objects as they arrive.+ """+ return ("," if index else "") + json.dumps(seg, ensure_ascii=False)+++def _field(body: dict, name: str, default):+ """Read an optional body field without treating 0 or 0.0 as absent."""+ value = body.get(name)+ return default if value is None else type(default)(value)+++def create_app(config: ServerConfig) -> FastAPI:+ """Build the app around one engine. The engine starts with the app."""+ geometry = ChunkGeometry.from_pretrained(config.model)++ @asynccontextmanager+ async def lifespan(app: FastAPI):+ logger.info("loading %s -- %s", config.model, geometry.describe())+ app.state.engine = VibeVoiceAsyncEngine(config, geometry)+ app.state.geometry = geometry+ logger.info("ready: %s served as %r", config.model,+ config.served_model_name)+ yield+ await app.state.engine.shutdown()++ app = FastAPI(title="VibeVoice vLLM streaming ASR", version="2.0",+ lifespan=lifespan)++ @app.get("/healthz")+ @app.get("/health")+ async def health():+ return {"status": "ok"}++ @app.get("/v1/models")+ async def models():+ return {"object": "list",+ "data": [{"id": config.served_model_name, "object": "model",+ "owned_by": "vibevoice"}]}++ @app.get("/v1/config")+ async def get_config():+ """The geometry a client has to cut audio at, read from the checkpoint.++ The demo captures at ``sample_rate`` and lays its cards out on+ ``chunk_seconds``, so both are served rather than hard-coded there --+ that is what lets one page drive either released checkpoint.+ """+ return {+ "model": config.served_model_name,+ "sample_rate": geometry.sample_rate,+ "chunk_seconds": geometry.chunk_seconds,+ "chunk_frames": geometry.chunk_frames,+ "lookahead_frames": geometry.lookahead_frames,+ "window_samples": geometry.window_samples,+ "max_audio_windows": config.max_audio_windows,+ }++ @app.post("/v1/transcribe", response_model=TranscribeResponse)+ async def transcribe(req: TranscribeRequest):+ try:+ return await _run_clip(app.state.engine, req)+ except HTTPException:+ raise+ except Exception as e:+ logger.exception("transcribe failed")+ raise HTTPException(status_code=500, detail=str(e))++ @app.post("/v1/transcribe_batch", response_model=BatchTranscribeResponse)+ async def transcribe_batch(req: BatchTranscribeRequest):+ """Transcribe several clips. Concurrent -- the engine does the batching."""+ subs = [+ TranscribeRequest(+ audio_base64=item.audio_base64,+ audio_url=item.audio_url,+ context_info=req.context_info,+ max_tokens=req.max_tokens,+ temperature=req.temperature,+ top_p=req.top_p,+ repetition_penalty=req.repetition_penalty,+ )+ for item in req.audios+ ]+ try:+ t0 = time.time()+ results = await asyncio.gather(+ *(_run_clip(app.state.engine, s) for s in subs))+ return BatchTranscribeResponse(results=results,+ total_time=time.time() - t0)+ except HTTPException:+ raise+ except Exception as e:+ logger.exception("batch transcribe failed")+ raise HTTPException(status_code=500, detail=str(e))++ @app.post("/v1/chat/completions")+ async def chat_completions(request: Request):+ try:+ body = await request.json()+ messages = body.get("messages") or []+ req = TranscribeRequest(+ max_tokens=min(_field(body, "max_tokens", 256), 2048),+ temperature=_field(body, "temperature", 0.0),+ top_p=_field(body, "top_p", 1.0),+ repetition_penalty=_field(body, "repetition_penalty", 1.0),+ context_info=_context_from_messages(messages),+ )+ except (AttributeError, TypeError, ValueError) as exc:+ # This dialect reads the raw body, so none of the automatic 422s+ # FastAPI gives the typed endpoints apply here. Without this, a+ # malformed body or an out-of-range temperature returns 500, which+ # reads as a server fault for what is a client mistake.+ raise HTTPException(status_code=400, detail=str(exc))+ # To a thread for the same reason as _run_clip: an audio_url part is+ # fetched over the network, and a stall here stalls every other session.+ audio_bytes = await asyncio.to_thread(_audio_from_messages, messages)+ cid = f"chatcmpl-{uuid.uuid4().hex[:24]}"+ sample_rate = geometry.sample_rate++ if not body.get("stream"):+ audio = await asyncio.to_thread(load_audio_bytes, audio_bytes,+ sample_rate)+ session = _new_session(app.state.engine, req)+ chunks = [t async for t in session.transcribe(audio)]+ segments = chunk_segments(chunks, geometry, len(audio) / sample_rate)+ return {+ "id": cid, "object": "chat.completion",+ "model": config.served_model_name,+ "choices": [{"index": 0, "finish_reason": "stop",+ "message": {"role": "assistant",+ "content": json.dumps(+ segments, ensure_ascii=False)}}],+ }++ async def events():+ model_name = config.served_model_name+ try:+ audio = await asyncio.to_thread(load_audio_bytes, audio_bytes,+ sample_rate)+ session = _new_session(app.state.engine, req)+ yield _sse(cid, model_name, {"role": "assistant", "content": "["})+ chunks, sent = [], 0+ async for chunk_text in session.transcribe(audio):+ chunks.append(chunk_text)+ # Every segment but the last: the trailing one is still open+ # and a later chunk can extend it or move where it ends.+ for seg in chunk_segments(chunks, geometry)[sent:-1]:+ yield _sse(cid, model_name,+ {"content": _segment_delta(seg, sent)})+ sent += 1+ # The real duration is known now, so the tail can be clamped.+ for seg in chunk_segments(chunks, geometry,+ len(audio) / sample_rate)[sent:]:+ yield _sse(cid, model_name,+ {"content": _segment_delta(seg, sent)})+ sent += 1+ yield _sse(cid, model_name, {"content": "]"})+ yield _sse(cid, model_name, {}, finish="stop")+ except Exception as e:+ logger.exception("chat stream failed")+ # The stream already carries a 200, so an error can only be+ # reported inside it. It lands outside the array, which leaves+ # whatever segments already arrived renderable.+ yield _sse(cid, model_name, {"content": f"\n[error] {e}"},+ finish="stop")+ yield "data: [DONE]\n\n"++ return StreamingResponse(events(), media_type="text/event-stream",+ headers={"Cache-Control": "no-cache",+ "X-Accel-Buffering": "no"})++ @app.websocket("/v1/stream")+ async def stream(ws: WebSocket):+ """Live session: push PCM, receive one transcript per chunk of audio.++ Protocol -- first message is JSON config (all keys optional), then binary+ frames of little-endian float32 mono PCM at the checkpoint's sample rate.+ A text message ``"end"`` flushes the tail. Each chunk comes back as+ ``{"chunk": idx, "text": ..., "segments": [...]}``, and the final message+ adds ``done``, ``total_chunks`` and ``srt``.++ The buffer keeps the lookahead overlap rather than draining fully: the+ window is longer than the advance.+ """+ await ws.accept()+ try:+ cfg = await ws.receive_json()+ req = TranscribeRequest(**{+ k: v for k, v in cfg.items()+ if k in ("context_info", "max_tokens", "temperature", "top_p",+ "repetition_penalty")+ })+ session = _new_session(app.state.engine, req)++ window_samples = geometry.window_samples+ chunk_samples = geometry.chunk_samples+ buf = np.zeros(0, dtype=np.float32)+ texts: List[str] = []++ async def emit(window) -> None:+ texts.append(await session.push(window))+ payload: Dict[str, Any] = {+ "chunk": len(texts) - 1,+ "text": texts[-1],+ "segments": chunk_segments(texts, geometry),+ }+ await ws.send_json(payload)++ while True:+ msg = await ws.receive()+ if msg.get("type") == "websocket.disconnect":+ return+ if msg.get("bytes") is not None:+ # "<f4" not np.float32: the browser sends little-endian and+ # the native dtype would reinterpret it on a big-endian host.+ buf = np.concatenate(+ [buf, np.frombuffer(msg["bytes"], dtype="<f4")])+ while len(buf) >= window_samples:+ await emit(buf[:window_samples])+ buf = buf[chunk_samples:]+ elif msg.get("text") == "end":+ # Same tail rule as the offline path: pad the remainder out+ # to a full window instead of encoding a short one, which+ # would emit one extra VAE frame and no lookahead.+ for window in split_windows(buf, geometry):+ await emit(window)+ segments = chunk_segments(texts, geometry)+ await ws.send_json({"done": True,+ "total_chunks": len(texts),+ "text": "".join(texts),+ "segments": segments,+ "srt": segments_to_srt(segments)})+ return+ except WebSocketDisconnect:+ return+ except Exception as e:+ logger.exception("stream session failed")+ try:+ await ws.send_json({"error": str(e)})+ except Exception:+ pass++ return app+++def parse_args(argv=None) -> argparse.Namespace:+ parser = argparse.ArgumentParser(+ description="VibeVoice streaming ASR server (in-process vLLM engine)",+ formatter_class=argparse.RawDescriptionHelpFormatter,+ epilog="""+Examples:+ # Serve a streaming checkpoint+ python3 -m vllm_plugin.asr_streaming_server --model /path/to/checkpoint++ # Two GPUs, longer sessions+ python3 -m vllm_plugin.asr_streaming_server --model /path/to/checkpoint \\+ --tensor-parallel-size 2 --max-model-len 32768++The chunk and lookahead are read from the checkpoint's preprocessor_config.json,+so a checkpoint always runs at the geometry it was trained on.+ """)+ parser.add_argument("--model", "-m", required=True,+ help="Path or HF repo id of a streaming checkpoint")+ parser.add_argument("--served-model-name", default="vibevoice",+ help="Name reported by /v1/models (default: vibevoice)")+ parser.add_argument("--host", default="0.0.0.0",+ help="Address to bind (default: 0.0.0.0)")+ parser.add_argument("--port", "-p", type=int, default=8000,+ help="Port to serve on (default: 8000)")+ parser.add_argument("--dtype", default="bfloat16",+ help="Weight dtype (default: bfloat16)")+ parser.add_argument("--tensor-parallel-size", "--tp", dest="tensor_parallel_size",+ type=int, default=1,+ help="Number of GPUs per replica (default: 1)")+ parser.add_argument("--gpu-memory-utilization", type=float, default=0.85,+ help="Fraction of GPU memory to use (default: 0.85)")+ parser.add_argument("--max-model-len", type=int, default=16384,+ help="Context length per session (default: 16384)")+ parser.add_argument("--max-audio-windows", type=int, default=512,+ help="Audio windows per session (default: 512)")+ parser.add_argument("--mm-processor-cache-gb", type=float, default=16.0,+ help="Multimodal cache, must hold every in-flight "+ "session's windows (default: 16)")+ parser.add_argument("--enforce-eager", action="store_true",+ help="Skip CUDA graph capture (default: False)")+ parser.add_argument("--no-enable-prefix-caching", dest="enable_prefix_caching",+ action="store_false",+ help="Disable prefix caching; diagnostic only, streaming "+ "goes quadratic without it (default: enabled)")+ parser.add_argument("--resend-all-audio", action="store_true",+ help="Send every window as data each turn instead of "+ "uuid placeholders; diagnostic only (default: False)")+ return parser.parse_args(argv)+++def main():+ args = parse_args()+ logging.basicConfig(+ level=logging.INFO,+ format="%(asctime)s %(levelname)s %(name)s: %(message)s")++ config = ServerConfig(+ model=args.model,+ served_model_name=args.served_model_name,+ dtype=args.dtype,+ gpu_memory_utilization=args.gpu_memory_utilization,+ max_model_len=args.max_model_len,+ tensor_parallel_size=args.tensor_parallel_size,+ max_audio_windows=args.max_audio_windows,+ mm_processor_cache_gb=args.mm_processor_cache_gb,+ enforce_eager=args.enforce_eager,+ enable_prefix_caching=args.enable_prefix_caching,+ resend_all_audio=args.resend_all_audio,+ )+ uvicorn.run(create_app(config), host=args.host, port=args.port,+ log_level="info", workers=1)+++if __name__ == "__main__":+ main()diff --git a/vllm_plugin/model.py b/vllm_plugin/model.pyindex bd9060f..76dad86 100644--- a/vllm_plugin/model.py+++ b/vllm_plugin/model.py@@ -90,7 +90,7 @@ from transformers import BatchFeaturefrom transformers.models.whisper import WhisperFeatureExtractorfrom vllm.config import VllmConfigfrom vllm.multimodal import MULTIMODAL_REGISTRY-from vllm.multimodal.parse import MultiModalDataParser+from vllm.multimodal.parse import AudioProcessorItems, MultiModalDataParserfrom vllm.sequence import IntermediateTensorsfrom vllm.model_executor.models.interfaces import SupportsMultiModal, SupportsPP, MultiModalEmbeddingsfrom vllm.model_executor.models.utils import (@@ -474,9 +474,27 @@ class VibeVoiceAudioEncoder(nn.Module):# vLLM Multimodal Processing Infrastructure# ============================================================================+def _cfg_get(hf_config, key: str, default):+ """Read a key from an HF config that may be a dict or an object."""+ if isinstance(hf_config, dict):+ return hf_config.get(key, default)+ return getattr(hf_config, key, default)+++def _is_streaming_asr(hf_config) -> bool:+ """Whether the checkpoint interleaves audio and text (streaming ASR).++ Derived from the architecture name rather than a separate config flag so+ that a checkpoint cannot be served under the wrong protocol by mistake:+ streaming checkpoints declare "VibeVoiceForASRStreamingTraining".+ """+ archs = _cfg_get(hf_config, "architectures", None) or []+ return any("Streaming" in str(a) for a in archs)++class VibeVoiceProcessingInfo(BaseProcessingInfo):"""Processing info for VibeVoice multimodal model."""-+def get_hf_config(self):return self.ctx.get_hf_config()@@ -553,7 +571,28 @@ class VibeVoiceProcessingInfo(BaseProcessingInfo):return tokens+ def get_streaming_geometry(self):+ """The window a streaming checkpoint was trained on, or None.++ Read off the checkpoint rather than configured, so the engine sizes its+ multimodal budget for exactly the window a client cuts. The budget is a+ hard limit: a client cutting larger windows than the engine sized for is+ rejected, not merely mismatched.+ """+ if not _is_streaming_asr(self.get_hf_config()):+ return None+ if not hasattr(self, "_streaming_geometry"):+ from .asr_streaming import ChunkGeometry+ self._streaming_geometry = ChunkGeometry.from_pretrained(+ self.ctx.model_config.model)+ return self._streaming_geometry+def get_supported_mm_limits(self) -> Mapping[str, int | None]:+ # Streaming prompts interleave one audio window per chunk, so a single+ # prompt carries N audio items rather than one. None means no fixed cap+ # here; the real bound comes from --limit-mm-per-prompt.+ if _is_streaming_asr(self.get_hf_config()):+ return {"audio": None}return {"audio": 1}def get_mm_max_tokens_per_item(@@ -583,6 +622,14 @@ class VibeVoiceProcessingInfo(BaseProcessingInfo):compress_ratio = int(_cfg("speech_tok_compress_ratio", 3200))sample_rate = int(_cfg("target_sample_rate", 24000))+ # Streaming items are one fixed-size window each, never a whole+ # utterance, so the 61-minute bound below would overstate the encoder+ # budget by ~4 orders of magnitude.+ geometry = self.get_streaming_geometry()+ if geometry is not None:+ # speech_start + window frames + speech_end+ return {"audio": geometry.window_frames + 2}+# Upper bound: 61-minute audio at 24 kHzmax_audio_samples = 61 * 60 * sample_rate # 88,464,000max_audio_tokens = int(np.ceil(max_audio_samples / compress_ratio)) + 3@@ -621,6 +668,13 @@ class VibeVoiceDummyInputsBuilder(BaseDummyInputsBuilder[VibeVoiceProcessingInfocompress_ratio = int(_cfg("speech_tok_compress_ratio", 3200))sample_rate = int(_cfg("target_sample_rate", 24000))+ # Must mirror get_mm_max_tokens_per_item: profiling a 61-minute clip+ # per item would try to VAE-encode ~88M samples N times over and OOM at+ # startup, long before serving a single request.+ geometry = self.info.get_streaming_geometry()+ if geometry is not None:+ return geometry.window_samples+# Upper bound: 61-minute audio at 24 kHzmax_hour_samples = 61 * 60 * sample_rate # 88,464,000max_tokens_from_audio = int(np.ceil(max_hour_samples / compress_ratio)) + 3@@ -699,6 +753,45 @@ def _vibevoice_field_config(hf_inputs: Mapping[str, torch.Tensor]):return config+class _NoneTolerantAudioParser(MultiModalDataParser):+ """Let a per-item ``None`` survive audio parsing.++ A streaming session re-sends its whole audio list every turn, with the+ windows vLLM has already processed replaced by ``None`` and pinned by a+ stable ``multi_modal_uuids`` entry. That keeps the per-turn payload flat+ instead of growing O(N) with the number of chunks.++ vLLM's cache layer is built for exactly this -- ``_get_cache_missing_items``+ raises "data is not provided" only for a ``None`` it genuinely needs to+ re-encode -- and ``_parse_image_data`` passes list elements through+ untouched. ``_parse_audio_data`` is the odd one out: it feeds every element+ to ``_get_audio_with_sr``, which ends in ``assert_never`` on ``None``. So+ resample only the real items and stitch the ``None`` holes back afterwards.+ """++ def _parse_audio_data(self, data):+ # A list specifically, not a tuple: upstream reads a bare+ # ``(audio, sr)`` tuple as one item, and ``sr=None`` is how a caller+ # says "already at the target rate" -- which for this model, at 24 kHz,+ # is the ordinary case. Matching tuples here would read that lone+ # sample rate as a second audio item and hand the engine a hole that was+ # never there. Holes only ever arrive in the list a session sends.+ if not isinstance(data, list) or not any(x is None for x in data):+ return super()._parse_audio_data(data)++ present = [x for x in data if x is not None]+ parsed = super()._parse_audio_data(present) if present else None+ if parsed is not None and not isinstance(parsed, AudioProcessorItems):+ # Embeddings take a different branch upstream and have no per-item+ # holes to fill; don't try to interleave into them.+ return parsed++ processed = iter(parsed.data if parsed is not None else ())+ return AudioProcessorItems(+ [None if x is None else next(processed) for x in data]+ )++class VibeVoiceMultiModalProcessor(BaseMultiModalProcessor[VibeVoiceProcessingInfo]):"""Multimodal processor for VibeVoice.@@ -711,7 +804,7 @@ class VibeVoiceMultiModalProcessor(BaseMultiModalProcessor[VibeVoiceProcessingIn"""Create a data parser with the correct target sample rate (24kHz)."""# VibeVoice requires 24kHz, not 16kHz (Whisper default)target_sr = 24000- return MultiModalDataParser(target_sr=target_sr)+ return _NoneTolerantAudioParser(target_sr=target_sr)def _call_hf_processor(self,@@ -777,11 +870,18 @@ class VibeVoiceMultiModalProcessor(BaseMultiModalProcessor[VibeVoiceProcessingInresult["raw_audio_lengths"] = torch.tensor(audio_lengths, dtype=torch.long)# Add a random salt to ensure unique hash and bypass cache- import uuid- # Use a random integer for salt- salt_val = hash(str(uuid.uuid4())) % 100000- result["salt"] = torch.tensor([salt_val], dtype=torch.long).expand(len(raw_audio_list))-+ #+ # Streaming is the exception, and needs the opposite. A session re-sends+ # its whole audio list every turn and pins the already-encoded windows+ # by ``multi_modal_uuids``; a per-call random salt would change their+ # hash every turn, so vLLM would re-encode every window it was just told+ # it could reuse -- turning a flat per-turn cost into an O(N) one.+ if not _is_streaming_asr(self.info.get_hf_config()):+ import uuid+ # Use a random integer for salt+ salt_val = hash(str(uuid.uuid4())) % 100000+ result["salt"] = torch.tensor([salt_val], dtype=torch.long).expand(len(raw_audio_list))+return resultdef _hf_processor_applies_updates(@@ -873,7 +973,9 @@ class VibeVoiceMultiModalProcessor(BaseMultiModalProcessor[VibeVoiceProcessingIn# If a full tensor is passed accidentally, fall back to its lengthreturn int(x.shape[0])return int(x)-++ is_streaming = _is_streaming_asr(hf_config)+def get_replacement(item_idx: int):if raw_audio_lengths and item_idx < len(raw_audio_lengths):audio_len = _to_int_len(raw_audio_lengths[item_idx])@@ -890,10 +992,16 @@ class VibeVoiceMultiModalProcessor(BaseMultiModalProcessor[VibeVoiceProcessingIn# Build replacement token sequence:# <|speech_start|> + N * <|speech_pad|> + <|speech_end|> + \n# The newline is important for correct prompt structure.+ #+ # Streaming is the exception: there the audio window is followed+ # directly by the transcript of that chunk, with no separator (see+ # modeling_vibevoice_asr.py building [sp_start, feat, sp_end]).+ # Emitting a stray '\n' would put every chunk off-distribution.newline_id = 198 # '\n' token+ trailer = [] if is_streaming else [newline_id]if speech_start_id is not None and speech_pad_id is not None and speech_end_id is not None:embed_id = int(speech_pad_id)- replacement_ids = [int(speech_start_id)] + [embed_id] * num_features + [int(speech_end_id), newline_id]+ replacement_ids = [int(speech_start_id)] + [embed_id] * num_features + [int(speech_end_id)] + trailer# Fallback: add audio BOS/EOS boundaries around repeated <|AUDIO|>.elif audio_bos_id is not None and audio_eos_id is not None:embed_id = int(audio_token_id)diff --git a/vllm_plugin/scripts/fastapi_asr_streaming_demo_api.py b/vllm_plugin/scripts/fastapi_asr_streaming_demo_api.pynew file mode 100644index 0000000..a49398d--- /dev/null+++ b/vllm_plugin/scripts/fastapi_asr_streaming_demo_api.py@@ -0,0 +1,646 @@+#!/usr/bin/env python+"""+VibeVoice ASR Streaming FastAPI Demo++This demo uses the streaming ASR server instead of loading the model directly.+The browser holds one WebSocket and pushes raw PCM down it; between the+microphone and the model there is a socket and a relay, nothing else. Upload+uses the same socket -- a file is just PCM that already exists -- so the two+paths cannot drift apart.++ GET / the UI, one self-contained page+ GET /config the chunk geometry, read from the upstream server+ GET /healthz readiness, includes upstream status+ WS /ws/asr audio up, transcript down++Usage:+ python fastapi_asr_streaming_demo_api.py --api_url http://localhost:8000+"""+import argparse+import asyncio+import json+import logging+import os+import re+import subprocess+import threading+import time+import urllib.request+from urllib.parse import urlparse++import httpx+import uvicorn+import websockets+from fastapi import FastAPI, WebSocket, WebSocketDisconnect+from fastapi.responses import HTMLResponse, JSONResponse+from websockets.exceptions import ConnectionClosed++logger = logging.getLogger("vibevoice.demo")+++def _ws_url(api_url: str) -> str:+ """http://host:port -> ws://host:port/v1/stream."""+ u = urlparse(api_url)+ scheme = "wss" if u.scheme == "https" else "ws"+ return f"{scheme}://{u.netloc}/v1/stream"+++async def _pump_up(browser: WebSocket, upstream) -> None:+ """Browser -> server: PCM frames, then the "end" marker."""+ while True:+ msg = await browser.receive()+ if msg.get("type") == "websocket.disconnect":+ await upstream.close()+ return+ if msg.get("bytes") is not None:+ await upstream.send(msg["bytes"])+ elif msg.get("text") is not None:+ await upstream.send(msg["text"])+ if msg["text"] == "end":+ return+++async def _pump_down(browser: WebSocket, upstream) -> None:+ """Server -> browser: one message per chunk, verbatim.++ The server hangs up as soon as it has sent the final message, often without+ a closing handshake. That is the end of the relay, not a failure -- a+ session that really was cut short is missing its "done" message, which the+ page already treats as unfinished.+ """+ try:+ async for raw in upstream:+ await browser.send_text(raw if isinstance(raw, str) else raw.decode())+ except ConnectionClosed:+ return+++def create_app(api_url: str, max_new_tokens: int) -> FastAPI:+ """Build the demo around one upstream server."""+ app = FastAPI(title="VibeVoice streaming ASR demo")+ upstream_ws = _ws_url(api_url)++ async def upstream_config() -> dict:+ """The geometry the server reads off its checkpoint.++ Fetched rather than hard-coded: the page captures at ``sample_rate``,+ and the two released checkpoints differ in chunk length. Cached after+ the first success so the page does not re-probe on every reload.+ """+ if not hasattr(app.state, "config"):+ async with httpx.AsyncClient(timeout=10) as client:+ resp = await client.get(f"{api_url}/v1/config")+ resp.raise_for_status()+ app.state.config = resp.json()+ return app.state.config++ @app.get("/healthz")+ async def healthz():+ try:+ cfg = await upstream_config()+ except Exception as e:+ return JSONResponse({"status": "upstream unreachable",+ "api_url": api_url, "error": str(e)},+ status_code=503)+ return {"status": "ok", "api_url": api_url, "model": cfg.get("model")}++ @app.get("/config")+ async def config():+ cfg = dict(await upstream_config())+ cfg["max_new_tokens"] = max_new_tokens+ return JSONResponse(cfg)++ @app.get("/", response_class=HTMLResponse)+ async def index():+ return PAGE++ @app.websocket("/ws/asr")+ async def ws_asr(browser: WebSocket):+ """Relay one browser session to one upstream session.++ Both directions run concurrently: the browser keeps pushing audio while+ transcripts for earlier chunks are still coming back, which is the whole+ point of streaming.+ """+ await browser.accept()+ try:+ opts = json.loads(await browser.receive_text())+ except Exception:+ await browser.close()+ return++ opts.setdefault("max_tokens", max_new_tokens)+ try:+ async with websockets.connect(upstream_ws, max_size=None) as upstream:+ await upstream.send(json.dumps(opts))+ up = asyncio.create_task(_pump_up(browser, upstream))+ down = asyncio.create_task(_pump_down(browser, upstream))+ done, _ = await asyncio.wait(+ {up, down}, return_when=asyncio.FIRST_COMPLETED)+ if down in done:+ # Upstream is finished or gone; nothing left to relay.+ up.cancel()+ else:+ # The browser sent "end", but the tail chunks and the final+ # "done" message are still upstream. Cancelling here would+ # drop exactly the part the user waits for.+ await down+ for task in (up, down):+ if task.done() and not task.cancelled():+ task.result()+ except WebSocketDisconnect:+ return+ except Exception as e:+ logger.exception("relay failed")+ try:+ await browser.send_text(json.dumps({"error": str(e)}))+ except Exception:+ pass++ return app+++PAGE = """<!doctype html>+<html lang="en">+<head>+<meta charset="utf-8">+<meta name="viewport" content="width=device-width, initial-scale=1">+<title>VibeVoice Streaming ASR</title>+<style>+ :root {+ color-scheme: dark;+ --bg:#0f1117; --panel:#171a23; --line:#262b38; --fg:#e6e8ef;+ --muted:#8b93a7; --faint:#5c6478; --accent:#5b8cff; --rec:#ff4d5e;+ --ok:#31c48d; --ghost:#232838; --field:#10131c;+ --hover:#1d2230; --sel:#1d2436;+ }+ :root[data-theme="light"] {+ color-scheme: light;+ --bg:#f4f6fa; --panel:#ffffff; --line:#dde2ec; --fg:#171a22;+ --muted:#5b6377; --faint:#8b93a7; --accent:#2f5fd0; --rec:#d92d3f;+ --ok:#0e9250; --ghost:#eaeef6; --field:#f7f9fc;+ --hover:#eef2f9; --sel:#e5edfb;+ }+ * { box-sizing: border-box; }+ body { margin:0; background:var(--bg); color:var(--fg); font-size:15px;+ font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,+ "Helvetica Neue","PingFang SC","Microsoft YaHei",sans-serif; }+ header { padding:18px 24px; border-bottom:1px solid var(--line);+ display:flex; align-items:center; gap:12px; flex-wrap:wrap; }+ h1 { font-size:17px; margin:0; font-weight:600; }+ .dot { width:8px; height:8px; border-radius:50%; background:var(--muted); }+ .dot.ok { background:var(--ok); }+ .dot.rec { background:var(--rec); animation:pulse 1.2s infinite; }+ @keyframes pulse { 50% { opacity:.25; } }+ main { display:grid; grid-template-columns:320px 1fr; gap:20px; padding:20px 24px;+ min-height:calc(100vh - 118px); align-content:stretch; }+ main > * { min-width:0; }+ .panel { background:var(--panel); border:1px solid var(--line);+ border-radius:10px; padding:16px; }+ .panel h2 { font-size:13px; text-transform:uppercase; letter-spacing:.06em;+ color:var(--muted); margin:0 0 12px; font-weight:600; }+ button { font:inherit; border:0; border-radius:8px; padding:10px 16px;+ background:var(--accent); color:#fff; cursor:pointer; min-height:40px; }+ button.ghost { background:var(--ghost); color:var(--fg); }+ button:disabled { opacity:.45; cursor:not-allowed; }+ #theme { margin-left:auto; background:transparent; color:var(--muted);+ border:1px solid var(--line); padding:0 11px; min-height:32px;+ font-size:15px; line-height:1; }+ label { display:block; font-size:13px; color:var(--muted); margin:14px 0 6px; }+ input[type=text], input[type=number] { width:100%; padding:9px 11px;+ border-radius:8px; border:1px solid var(--line); background:var(--field);+ color:var(--fg); font:inherit; }+ input[type=file] { width:100%; font-size:13px; color:var(--muted); }+ .row { display:flex; gap:10px; flex-wrap:wrap; }+ .seg { border-left:2px solid var(--accent); padding:2px 0 2px 12px;+ margin:0 0 16px; border-radius:0 6px 6px 0; }+ .seg.play { cursor:pointer; }+ @media (hover:hover) { .seg.play:hover { background:var(--hover); } }+ .seg.on { background:var(--sel); }+ .seg .meta { font-size:12px; color:var(--muted); margin-bottom:4px; }+ .seg.on .meta { color:var(--accent); }+ .seg .body { line-height:1.75; white-space:pre-wrap; word-break:break-word; }+ #out:empty::before { content:"Transcript appears here as you speak.";+ color:var(--muted); }+ #out { min-height:320px; }+ .status { font-size:13px; color:var(--muted); margin-top:12px; }+ .bar { display:flex; align-items:center; justify-content:space-between;+ gap:10px; flex-wrap:wrap; margin-bottom:12px; }+ .bar button { min-height:32px; padding:6px 12px; font-size:13px; }+ .mut { color:var(--faint); font-weight:400; }+</style>+<script>+// Runs before first paint so the page does not flash the wrong theme. Follows+// the OS until the header button is pressed, after which the choice sticks.+(function () {+ var t = null;+ try { t = localStorage.getItem('vv-theme'); } catch (e) {}+ if (t !== 'light' && t !== 'dark')+ t = matchMedia('(prefers-color-scheme: light)').matches ? 'light' : 'dark';+ document.documentElement.dataset.theme = t;+})();+</script>+</head>+<body>+<header>+ <h1>VibeVoice Streaming ASR</h1>+ <span class="dot" id="dot"></span>+ <span class="status" id="conn" style="margin:0">idle</span>+ <button id="theme" type="button"></button>+</header>+<main>+ <div class="panel">+ <h2>Input</h2>+ <div class="row">+ <button id="rec">Record</button>+ <button id="stop" class="ghost" disabled>Stop</button>+ </div>+ <label>Or transcribe a file</label>+ <input type="file" id="file" accept="audio/*,video/*">+ <div class="row" style="margin-top:10px">+ <button id="send" disabled>Transcribe</button>+ <button id="cancel" class="ghost" disabled>Cancel</button>+ </div>+ <label>Hotwords (context_info)</label>+ <input type="text" id="ctx" placeholder="Microsoft,VibeVoice">+ <label>Temperature <span class="mut">0 = greedy</span></label>+ <input type="number" id="temp" value="0" min="0" max="2" step="0.1">+ <label>Top-p <span class="mut" id="toppNote">needs temperature > 0</span></label>+ <input type="number" id="topp" value="1" min="0.05" max="1" step="0.05" disabled>+ <div class="status" id="status">ready</div>+ </div>+ <div class="panel">+ <div class="bar">+ <h2 style="margin:0">Transcript</h2>+ <div class="row">+ <button class="ghost" id="copy" disabled>Copy</button>+ <button class="ghost" id="srt" disabled>SRT</button>+ <button class="ghost" id="clear" disabled>Clear</button>+ </div>+ </div>+ <div id="out"></div>+ </div>+</main>+<script>+const $ = id => document.getElementById(id);+let SR = 24000, MAXTOK = 256, sock = null, ctxAudio = null, node = null,+ media = null, lastSrt = '', segments = [], pcm = null;++// Kept so a card can be played back. The upload is already one Float32Array;+// the mic is stored as Int16 in arrival order, which halves the memory for+// something whose only destination is a speaker. Past the cap audio is dropped+// and those cards stop being clickable, rather than the tab dying.+const PLAYBACK_MAX_S = 1800;+let playPcm = null, micChunks = [], micSamples = 0;+let playCtx = null, player = null, playing = null;++function playable() { return !!(playPcm || micSamples); }++function keepMic(frame) {+ if (micSamples >= PLAYBACK_MAX_S * SR) return;+ const q = new Int16Array(frame.length);+ for (let i = 0; i < frame.length; i++)+ q[i] = Math.max(-1, Math.min(1, frame[i])) * 32767;+ micChunks.push(q);+ micSamples += q.length;+}++// Samples [a, b) of this run, whichever source it came from.+function sliceAudio(a, b) {+ if (playPcm)+ return a >= playPcm.length ? null+ : playPcm.subarray(a, Math.min(b, playPcm.length));+ b = Math.min(b, micSamples);+ if (a >= b) return null;+ const out = new Float32Array(b - a);+ let pos = 0;+ for (const c of micChunks) {+ const s = Math.max(a, pos), e = Math.min(b, pos + c.length);+ for (let i = s; i < e; i++) out[i - a] = c[i - pos] / 32768;+ pos += c.length;+ if (pos >= b) break;+ }+ return out;+}++function stopPlay() {+ if (player) {+ player.onended = null;+ try { player.stop(); } catch (e) {}+ player.disconnect();+ player = null;+ }+ playing = null;+}++function playSegment(i) {+ const s = segments[i];+ if (!s || !playable()) return;+ if (playing === i) { stopPlay(); render(); return; }+ const clip = sliceAudio(Math.floor(s.Start * SR), Math.ceil(s.End * SR));+ if (!clip || !clip.length) { $('status').textContent = 'no audio kept for this card'; return; }+ stopPlay();+ playCtx = playCtx || new AudioContext();+ const buf = playCtx.createBuffer(1, clip.length, SR);+ buf.getChannelData(0).set(clip);+ player = playCtx.createBufferSource();+ player.buffer = buf;+ player.connect(playCtx.destination);+ player.onended = () => { if (playing === i) { playing = null; render(); } };+ player.start();+ playing = i;+ render();+}++$('out').onclick = ev => {+ const el = ev.target.closest('.seg');+ if (el) playSegment(+el.dataset.i);+};++fetch('/config').then(r => r.json()).then(c => {+ SR = c.sample_rate; MAXTOK = c.max_new_tokens;+ $('status').textContent = 'ready -- ' + c.chunk_seconds.toFixed(2) + 's chunks';+});++function setConn(text, cls) {+ $('conn').textContent = text;+ $('dot').className = 'dot' + (cls ? ' ' + cls : '');+}+function options() {+ return { context_info: $('ctx').value.trim() || null,+ temperature: parseFloat($('temp').value) || 0,+ top_p: parseFloat($('topp').value) || 1,+ max_tokens: MAXTOK };+}+// Top-p only bites once sampling is on, so it follows the temperature box.+$('temp').oninput = () => {+ const on = (parseFloat($('temp').value) || 0) > 0;+ $('topp').disabled = !on;+ $('toppNote').textContent = on ? 'nucleus sampling' : 'needs temperature > 0';+};+// textContent throughout: a transcript is model output, and building this with+// innerHTML would let a spoken "<script>" run in the page.+function render() {+ const out = $('out');+ out.textContent = '';+ const canPlay = playable();+ segments.forEach((seg, i) => {+ const card = document.createElement('div');+ card.className = 'seg' + (canPlay ? ' play' : '') + (playing === i ? ' on' : '');+ card.dataset.i = i;+ if (canPlay) card.title = 'Click to play this segment';+ const meta = document.createElement('div');+ meta.className = 'meta';+ meta.textContent = (playing === i ? 'βΆ ' : '')+ + 'Speaker ' + seg.Speaker + ' Β· '+ + seg.Start.toFixed(2) + 's - ' + seg.End.toFixed(2) + 's';+ const body = document.createElement('div');+ body.className = 'body';+ body.textContent = seg.Content;+ card.appendChild(meta); card.appendChild(body); out.appendChild(card);+ });+ const empty = segments.length === 0;+ $('copy').disabled = empty;+ $('clear').disabled = empty;+ $('srt').disabled = !lastSrt;+}+function connect() {+ return new Promise((resolve, reject) => {+ const proto = location.protocol === 'https:' ? 'wss' : 'ws';+ const ws = new WebSocket(proto + '://' + location.host + '/ws/asr');+ ws.onopen = () => { ws.send(JSON.stringify(options())); setConn('connected', 'ok'); resolve(ws); };+ ws.onerror = () => reject(new Error('connection failed'));+ ws.onclose = () => setConn('idle', '');+ ws.onmessage = ev => {+ const msg = JSON.parse(ev.data);+ if (msg.error) { $('status').textContent = 'error: ' + msg.error; finish(); return; }+ if (msg.segments) { segments = msg.segments; render(); }+ $('status').textContent = msg.done+ ? 'done -- ' + msg.total_chunks + ' chunks'+ : 'receiving -- ' + (msg.chunk + 1) + ' chunks';+ if (msg.done) { lastSrt = msg.srt || ''; render(); finish(); }+ };+ });+}+function finish() {+ $('rec').disabled = false; $('stop').disabled = true; $('file').disabled = false;+ $('send').disabled = !$('file').files.length; $('cancel').disabled = true;+ setConn('idle', '');+ if (node) { node.disconnect(); node = null; }+ if (media) { media.getTracks().forEach(t => t.stop()); media = null; }+ if (ctxAudio) { ctxAudio.close(); ctxAudio = null; }+}++function reset(msg) {+ segments = []; lastSrt = '';+ stopPlay();+ playPcm = null; micChunks = []; micSamples = 0;+ render();+ $('status').textContent = msg;+}++$('rec').onclick = async () => {+ reset('starting...');+ try {+ media = await navigator.mediaDevices.getUserMedia({ audio: true });+ } catch (e) { $('status').textContent = 'microphone denied'; return; }+ sock = await connect();+ $('rec').disabled = true; $('stop').disabled = false; $('file').disabled = true;+ $('send').disabled = true; $('cancel').disabled = false;+ setConn('recording', 'rec');+ ctxAudio = new AudioContext({ sampleRate: SR });+ const src = ctxAudio.createMediaStreamSource(media);+ node = ctxAudio.createScriptProcessor(4096, 1, 1);+ node.onaudioprocess = e => {+ if (!sock || sock.readyState !== WebSocket.OPEN) return;+ const frame = new Float32Array(e.inputBuffer.getChannelData(0));+ // Kept before the send, so a card still plays what the microphone heard+ // even if the network dropped that frame on the way out.+ keepMic(frame);+ sock.send(frame.buffer);+ };+ src.connect(node); node.connect(ctxAudio.destination);+};++$('stop').onclick = () => {+ $('status').textContent = 'flushing...';+ if (sock && sock.readyState === WebSocket.OPEN) sock.send('end');+ $('stop').disabled = true;+};++// Decoding happens on pick, sending on the button: options set after choosing a+// file still apply, and a mis-picked file costs nothing.+$('file').onchange = async ev => {+ pcm = null; $('send').disabled = true;+ const f = ev.target.files[0];+ if (!f) { $('status').textContent = 'ready'; return; }+ $('status').textContent = 'decoding...';+ try {+ const raw = await f.arrayBuffer();+ const tmp = new AudioContext();+ const decoded = await tmp.decodeAudioData(raw);+ tmp.close();+ const off = new OfflineAudioContext(1, Math.ceil(decoded.duration * SR), SR);+ const s = off.createBufferSource();+ s.buffer = decoded; s.connect(off.destination); s.start();+ pcm = (await off.startRendering()).getChannelData(0);+ } catch (e) { $('status').textContent = 'cannot decode: ' + e.message; return; }+ $('send').disabled = false;+ $('status').textContent = 'ready -- ' + (pcm.length / SR).toFixed(1) + 's decoded';+};++$('send').onclick = async () => {+ if (!pcm) return;+ reset('streaming...');+ playPcm = pcm;+ sock = await connect();+ $('rec').disabled = true; $('file').disabled = true;+ $('send').disabled = true; $('cancel').disabled = false;+ // Sent in the cadence a microphone would produce, so text arrives gradually.+ for (let i = 0; i < pcm.length; i += SR / 2) {+ if (!sock || sock.readyState !== WebSocket.OPEN) return;+ sock.send(new Float32Array(pcm.slice(i, i + SR / 2)).buffer);+ await new Promise(r => setTimeout(r, 10));+ }+ if (sock && sock.readyState === WebSocket.OPEN) sock.send('end');+ $('status').textContent = 'sent, waiting for the tail...';+};++$('cancel').onclick = () => {+ if (sock) sock.close();+ sock = null;+ finish();+ $('status').textContent = 'cancelled';+};++$('clear').onclick = () => { reset('ready'); };++$('copy').onclick = () => {+ navigator.clipboard.writeText(+ segments.map(s => 'Speaker ' + s.Speaker + ': ' + s.Content).join('\\n'));+ $('status').textContent = 'copied ' + segments.length + ' segments';+};++$('srt').onclick = () => {+ const url = URL.createObjectURL(new Blob([lastSrt], { type: 'text/plain' }));+ const a = document.createElement('a');+ a.href = url; a.download = 'transcript.srt'; a.click();+ URL.revokeObjectURL(url);+};++// The glyph shows what pressing it gets you, not what you are looking at.+const root = document.documentElement;+function paintTheme() {+ const light = root.dataset.theme === 'light';+ $('theme').textContent = light ? 'βΎ' : 'β';+ $('theme').title = light ? 'Switch to dark' : 'Switch to light';+ $('theme').setAttribute('aria-label', $('theme').title);+}+$('theme').onclick = () => {+ root.dataset.theme = root.dataset.theme === 'light' ? 'dark' : 'light';+ try { localStorage.setItem('vv-theme', root.dataset.theme); } catch (e) {}+ paintTheme();+};+paintTheme();+render();+</script>+</body>+</html>+"""+++CLOUDFLARED_PATH = os.path.expanduser("~/.local/bin/cloudflared")+CLOUDFLARED_URL = ("https://github.com/cloudflare/cloudflared/releases/latest"+ "/download/cloudflared-linux-amd64")+TUNNEL_RE = re.compile(r"https://[-a-z0-9]+\.trycloudflare\.com")+++def download_cloudflared() -> bool:+ """Fetch the cloudflared binary unless it is already there."""+ if os.path.exists(CLOUDFLARED_PATH):+ return True+ print("Downloading cloudflared...")+ os.makedirs(os.path.dirname(CLOUDFLARED_PATH), exist_ok=True)+ try:+ urllib.request.urlretrieve(CLOUDFLARED_URL, CLOUDFLARED_PATH)+ os.chmod(CLOUDFLARED_PATH, 0o755)+ return True+ except Exception as e:+ print(f"Failed to download cloudflared: {e}")+ return False+++def start_cloudflared_tunnel(port: int):+ """Publish the demo on a public https URL, printed once it is assigned.++ https matters beyond convenience: getUserMedia only runs in a secure+ context, so a plain http address on anything but localhost cannot reach+ the microphone at all.+ """+ if not download_cloudflared():+ return None+ process = subprocess.Popen(+ [CLOUDFLARED_PATH, "tunnel", "--url", f"http://localhost:{port}",+ "--no-autoupdate"],+ stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)++ def watch():+ for line in process.stdout:+ found = TUNNEL_RE.search(line)+ if found:+ print(f"\n public URL: {found.group(0)}\n", flush=True)++ threading.Thread(target=watch, daemon=True).start()+ time.sleep(3)+ return process+++def main():+ parser = argparse.ArgumentParser(description="VibeVoice ASR Streaming FastAPI Demo")+ parser.add_argument(+ "--api_url",+ type=str,+ default="http://localhost:8000",+ help="URL of the streaming ASR server"+ )+ parser.add_argument(+ "--max_new_tokens",+ type=int,+ default=256,+ help="Default max new tokens per chunk"+ )+ parser.add_argument(+ "--host",+ type=str,+ default="0.0.0.0",+ help="Host to bind the server to"+ )+ parser.add_argument(+ "--port",+ type=int,+ default=7860,+ help="Port to bind the server to"+ )+ parser.add_argument(+ "--cloudflared",+ action="store_true",+ help="Create a public link using cloudflared tunnel"+ )+ args = parser.parse_args()++ logging.basicConfig(+ level=logging.INFO,+ format="%(asctime)s %(levelname)s %(name)s: %(message)s")++ tunnel = start_cloudflared_tunnel(args.port) if args.cloudflared else None+ app = create_app(args.api_url, args.max_new_tokens)+ print(f"relaying to {args.api_url}, open http://localhost:{args.port}")+ try:+ uvicorn.run(app, host=args.host, port=args.port)+ finally:+ if tunnel:+ tunnel.terminate()+++if __name__ == "__main__":+ main()diff --git a/vllm_plugin/scripts/start_streaming_server.py b/vllm_plugin/scripts/start_streaming_server.pynew file mode 100644index 0000000..07d270f--- /dev/null+++ b/vllm_plugin/scripts/start_streaming_server.py@@ -0,0 +1,256 @@+#!/usr/bin/env python3+"""+VibeVoice vLLM Streaming ASR Server Launcher++One-click deployment script that handles:+1. Installing system dependencies (FFmpeg, etc.)+2. Installing VibeVoice Python package+3. Downloading model from HuggingFace+4. Refusing a checkpoint that cannot be streamed+5. Generating tokenizer files+6. Starting the streaming ASR server++Steps 1-3 are shared with start_server.py; the rest differ. A streaming+checkpoint transcribes one chunk at a time off a growing sequence, so a request+is a session rather than a single generate call and stock `vllm serve` has+nowhere to keep it. This launches vllm_plugin.asr_streaming_server instead,+which holds the same engine in-process and adds a WebSocket for live audio.++Usage:+ python3 start_streaming_server.py [--model MODEL_ID] [--port PORT]+"""++import argparse+import json+import os+import sys++# Same directory, run as a script rather than a package -- make the sibling+# importable so the four bootstrap steps stay in one place instead of drifting.+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))+# And vllm_plugin/ itself, for a bare import of asr_streaming: the package+# __init__ imports vLLM, while the checks below exist to run before the engine+# does -- including on a box where vLLM is not importable yet.+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))++from asr_streaming import ChunkGeometry, TEXT_CHUNK_END_ID # noqa: E402+from start_server import (download_model, install_system_deps, # noqa: E402+ install_vibevoice, run_command)+++def resolve_model(model_id: str) -> str:+ """A local checkpoint directory is served as-is; anything else is downloaded."""+ if os.path.isdir(model_id):+ print(f"\n{'='*60}")+ print(f" Using local model")+ print(f" π Path: {model_id}")+ print(f"{'='*60}\n")+ return model_id+ return download_model(model_id)+++def generate_streaming_tokenizer(model_path: str) -> None:+ """Generate tokenizer files, unless the checkpoint already ships them.++ A streaming checkpoint holds <|text_chunk_end|> at 151665 and its audio+ tokens one id higher. Writing the non-streaming layout over that would take+ away the token every chunk ends on, so the layout is asked for explicitly+ and a checkpoint that already has it is left alone.+ """+ added_tokens = os.path.join(model_path, "added_tokens.json")+ if os.path.isfile(added_tokens):+ with open(added_tokens) as f:+ if json.load(f).get("<|text_chunk_end|>") == TEXT_CHUNK_END_ID:+ print("\nTokenizer files already have <|text_chunk_end|>, skipping\n")+ return+ run_command(+ [sys.executable, "-m", "vllm_plugin.tools.generate_tokenizer_files",+ "--output", model_path, "--streaming"],+ "Generating tokenizer files"+ )+++def check_streaming_checkpoint(model_path: str) -> None:+ """Refuse a checkpoint that cannot be streamed, before the engine starts.++ Serving a non-streaming checkpoint here does not raise on its own: the+ weights load, the first chunk transcribes, and every later chunk comes back+ empty because nothing ever emits <|text_chunk_end|>.+ """+ geometry = ChunkGeometry.from_pretrained(model_path)+ print(f"\n{'='*60}")+ print(f" β Streaming checkpoint")+ print(f" π Path: {model_path}")+ print(f" ποΈ {geometry.describe()}")+ print(f"{'='*60}\n")+++def start_streaming_server(model_path: str, port: int,+ tensor_parallel_size: int = 1,+ max_model_len: int = 16384,+ max_audio_windows: int = 512,+ mm_processor_cache_gb: float = 16.0,+ gpu_memory_utilization: float = 0.85) -> None:+ """Start the streaming ASR server (replaces current process)."""+ print(f"\n{'='*60}")+ print(f" Starting streaming ASR server on port {port}")+ print(f" Tensor Parallel (TP): {tensor_parallel_size}")+ print(f" Max Model Len: {max_model_len}")+ print(f" Max Audio Windows: {max_audio_windows}")+ print(f" MM Cache (GB): {mm_processor_cache_gb}")+ print(f" GPU Mem Utilization: {gpu_memory_utilization}")+ print(f"{'='*60}\n")++ cmd = [+ sys.executable, "-m", "vllm_plugin.asr_streaming_server",+ "--model", model_path,+ "--served-model-name", "vibevoice",+ "--port", str(port),+ "--dtype", "bfloat16",+ "--tensor-parallel-size", str(tensor_parallel_size),+ "--max-model-len", str(max_model_len),+ "--max-audio-windows", str(max_audio_windows),+ "--mm-processor-cache-gb", str(mm_processor_cache_gb),+ "--gpu-memory-utilization", str(gpu_memory_utilization),+ ]+ os.execvp(sys.executable, cmd)+++def main():+ parser = argparse.ArgumentParser(+ description="VibeVoice vLLM Streaming ASR Server - One-Click Deployment",+ formatter_class=argparse.RawDescriptionHelpFormatter,+ epilog="""+Examples:+ # Start with default settings (single GPU)+ python3 start_streaming_server.py++ # Use custom port+ python3 start_streaming_server.py --port 8080++ # Tensor parallel: split model across 2 GPUs+ python3 start_streaming_server.py --tp 2++ # Longer sessions (the context is what caps how long a session can run)+ python3 start_streaming_server.py --max-model-len 32768++ # Skip dependency installation (if already installed)+ python3 start_streaming_server.py --skip-deps++The chunk and lookahead are read from the checkpoint's preprocessor_config.json,+so a checkpoint always runs at the geometry it was trained on.+ """+ )+ parser.add_argument(+ "--model", "-m",+ default="microsoft/VibeVoice-ASR-Streaming",+ help="HuggingFace model ID or local path (default: microsoft/VibeVoice-ASR-Streaming)"+ )+ parser.add_argument(+ "--port", "-p",+ type=int,+ default=8000,+ help="Server port (default: 8000)"+ )+ parser.add_argument(+ "--skip-deps",+ action="store_true",+ help="Skip installing system dependencies"+ )+ parser.add_argument(+ "--skip-tokenizer",+ action="store_true",+ help="Skip generating tokenizer files"+ )+ parser.add_argument(+ "--tp", "--tensor-parallel-size",+ type=int,+ default=1,+ dest="tensor_parallel_size",+ help="Tensor parallel size: split one model across N GPUs (default: 1)"+ )+ parser.add_argument(+ "--dp", "--data-parallel-size",+ type=int,+ default=1,+ dest="data_parallel_size",+ help="Not supported for streaming: a session's KV cache lives on one "+ "replica, and round-robin would send its later chunks to replicas "+ "that never saw its audio (default: 1)"+ )+ parser.add_argument(+ "--max-model-len",+ type=int,+ default=16384,+ dest="max_model_len",+ help="Context per session, which is what caps session length (default: 16384)"+ )+ parser.add_argument(+ "--max-audio-windows",+ type=int,+ default=512,+ dest="max_audio_windows",+ help="Audio windows per session, ~17 min at a 2s chunk (default: 512)"+ )+ parser.add_argument(+ "--mm-processor-cache-gb",+ type=float,+ default=16.0,+ dest="mm_processor_cache_gb",+ help="Multimodal cache; must hold every in-flight session's windows "+ "(default: 16)"+ )+ parser.add_argument(+ "--gpu-memory-utilization",+ type=float,+ default=0.85,+ dest="gpu_memory_utilization",+ help="GPU memory utilization fraction (default: 0.85)"+ )+ args = parser.parse_args()++ if args.data_parallel_size > 1:+ parser.error(+ "--dp is not supported for streaming. Each session keeps its audio "+ "windows in one replica's prefix cache, and an nginx round-robin "+ "would send its later chunks to a replica that never saw them. Run "+ "one server per GPU on separate ports and route whole sessions, or "+ "use --tp to split the model across GPUs."+ )++ print("\n" + "="*60)+ print(" VibeVoice vLLM Streaming ASR Server - One-Click Deployment")+ print("="*60)++ # Step 1: Install system dependencies+ if not args.skip_deps:+ install_system_deps()++ # Step 2: Install VibeVoice+ install_vibevoice()++ # Step 3: Download model+ model_path = resolve_model(args.model)++ # Step 4: Refuse a checkpoint that cannot be streamed -- before step 5,+ # which rewrites the tokenizer files in place and would leave a+ # non-streaming checkpoint unusable by start_server.py too.+ check_streaming_checkpoint(model_path)++ # Step 5: Generate tokenizer files+ if not args.skip_tokenizer:+ generate_streaming_tokenizer(model_path)++ # Step 6: Start server+ start_streaming_server(+ model_path, args.port,+ tensor_parallel_size=args.tensor_parallel_size,+ max_model_len=args.max_model_len,+ max_audio_windows=args.max_audio_windows,+ mm_processor_cache_gb=args.mm_processor_cache_gb,+ gpu_memory_utilization=args.gpu_memory_utilization,+ )+++if __name__ == "__main__":+ main()diff --git a/vllm_plugin/tests/test_api_streaming.py b/vllm_plugin/tests/test_api_streaming.pynew file mode 100644index 0000000..f6cb9a5--- /dev/null+++ b/vllm_plugin/tests/test_api_streaming.py@@ -0,0 +1,185 @@+#!/usr/bin/env python3+"""+Test VibeVoice vLLM Streaming ASR API over WebSocket, with Optional Hotwords.++This script tests the streaming ASR server: the audio is pushed as PCM frames+and a transcript comes back once per chunk, while the rest of the file is still+being sent. Text appears as it is recognized rather than at the end.++Optionally, you can provide hotwords (context_info) to improve recognition+of domain-specific content like proper nouns, technical terms, and speaker names.+Hotwords are embedded in the prompt as "with extra info: {hotwords}".++Usage:+ python test_api_streaming.py [audio_path] [--url URL] [--hotwords "word1,word2"]++Examples:+ # Standard streaming transcription (no hotwords)+ python3 test_api_streaming.py audio.wav++ # With hotwords for better recognition of specific terms+ python3 test_api_streaming.py audio.wav --hotwords "Microsoft,Azure,VibeVoice"+"""+import asyncio+import json+import os+import sys+import time+import argparse+from urllib.parse import urlparse++import requests+import websockets++# Bare import off vllm_plugin/: the package __init__ imports vLLM, which this+# client does not need. Decoding here rather than in ffmpeg alone keeps the+# resampler identical to the server's, so the transcript matches.+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))+from asr_streaming import load_audio_bytes # noqa: E402+++def _ws_url(base_url: str) -> str:+ """http://host:port -> ws://host:port/v1/stream."""+ u = urlparse(base_url)+ scheme = "wss" if u.scheme == "https" else "ws"+ return f"{scheme}://{u.netloc}/v1/stream"+++async def test_streaming_with_hotwords(+ audio_path: str,+ context_info: str = None,+ base_url: str = "http://localhost:8000",+):+ """+ Test streaming ASR transcription with customized hotwords.++ Args:+ audio_path: Path to the audio file+ context_info: Hotwords string (e.g., "Microsoft,Azure,VibeVoice")+ base_url: Streaming ASR server URL+ """++ print(f"=" * 70)+ print(f"Testing Streaming Transcription")+ print(f"=" * 70)+ print(f"Input file: {audio_path}")+ print(f"Hotwords: {context_info or '(none)'}")+ print()++ # The chunk length is a property of the checkpoint, not of this client.+ try:+ config = requests.get(f"{base_url}/v1/config", timeout=10).json()+ except Exception as e:+ print(f"β Cannot reach {base_url}: {e}")+ return+ sample_rate = config["sample_rate"]+ print(f"Model: {config['model']}")+ print(f"Chunk: {config['chunk_seconds']:.4f}s "+ f"({config['chunk_frames']} frames + {config['lookahead_frames']} lookahead)")++ # Load audio (video files work too: ffmpeg takes the audio track)+ try:+ with open(audio_path, "rb") as f:+ audio = load_audio_bytes(f.read(), sample_rate)+ except Exception as e:+ print(f"Error preparing audio: {e}")+ return+ duration = len(audio) / sample_rate+ print(f"Audio duration: {duration:.2f} seconds")++ if context_info and context_info.strip():+ print(f"\nπ Hotwords embedded in prompt: '{context_info}'")+ else:+ print(f"\nπ No hotwords provided")++ url = _ws_url(base_url)+ print(f"\n{'=' * 70}")+ print(f"Streaming to {url}")+ print(f"{'=' * 70}")++ t0 = time.time()+ result = None+ try:+ async with websockets.connect(url, max_size=None) as ws:+ await ws.send(json.dumps({+ "context_info": context_info,+ "max_tokens": 256,+ "temperature": 0.0,+ "top_p": 1.0,+ }))++ async def send_audio():+ # Half a second per frame, as a microphone would deliver it.+ step = sample_rate // 2+ for i in range(0, len(audio), step):+ await ws.send(audio[i:i + step].astype("<f4").tobytes())+ await ws.send("end")++ sender = asyncio.create_task(send_audio())++ print("\nβ Connected. Streaming content:\n")+ print("-" * 50)+ async for raw in ws:+ msg = json.loads(raw)+ if "error" in msg:+ print(f"\nβ Error: {msg['error']}")+ break+ if msg.get("done"):+ result = msg+ print("\n" + "-" * 50)+ print(f"β [Finished] {msg['total_chunks']} chunks")+ break+ # Each chunk carries only its own text; the timestamp shows how+ # far behind the audio the transcript is running.+ print(f"[{time.time() - t0:6.2f}s] {msg['text']}", flush=True)+ await sender+ except Exception as e:+ print(f"β Error: {e}")++ elapsed = time.time() - t0+ if result:+ print(f"\n{'=' * 70}")+ print("Segments")+ print(f"{'=' * 70}")+ for seg in result["segments"]:+ print(f"[{seg['Start']:7.2f} - {seg['End']:7.2f}] "+ f"Speaker {seg['Speaker']}: {seg['Content']}")++ print(f"\n{'=' * 70}")+ print(f"β±οΈ Total time elapsed: {elapsed:.2f}s")+ print(f"π RTF (Real-Time Factor): {elapsed / duration:.2f}x")+ print(f"{'=' * 70}")+++def main():+ parser = argparse.ArgumentParser(+ description="Test VibeVoice vLLM Streaming ASR API"+ )+ parser.add_argument(+ "audio_path",+ help="Path to audio file (wav, mp3, flac, etc.) or video file"+ )+ parser.add_argument(+ "--url",+ default="http://localhost:8000",+ help="Streaming ASR server URL (default: http://localhost:8000)"+ )+ parser.add_argument(+ "--hotwords",+ type=str,+ default=None,+ help="Hotwords to improve recognition (e.g., 'Microsoft,Azure,VibeVoice')"+ )++ args = parser.parse_args()++ # Run test+ asyncio.run(test_streaming_with_hotwords(+ audio_path=args.audio_path,+ context_info=args.hotwords,+ base_url=args.url,+ ))+++if __name__ == "__main__":+ main()diff --git a/vllm_plugin/tools/generate_tokenizer_files.py b/vllm_plugin/tools/generate_tokenizer_files.pyindex 62de55f..ede98a9 100644--- a/vllm_plugin/tools/generate_tokenizer_files.py+++ b/vllm_plugin/tools/generate_tokenizer_files.py@@ -52,6 +52,22 @@ VIBEVOICE_AUDIO_TOKENS = {# All extended tokens (Qwen2.5 + VibeVoice)ALL_EXTENDED_TOKENS = {**QWEN25_EXTENDED_TOKENS, **VIBEVOICE_AUDIO_TOKENS}+# Streaming checkpoints train <|text_chunk_end|> into the first free slot after+# Qwen2.5, so their audio tokens all sit one id higher.+VIBEVOICE_STREAMING_AUDIO_TOKENS = {+ "<|text_chunk_end|>": 151665,+ "<|AUDIO|>": 151666,+ "<|audio_bos|>": 151667,+ "<|audio_eos|>": 151668,+}+++def _token_maps(streaming: bool):+ """(audio tokens, every extended token) for the requested layout."""+ audio = VIBEVOICE_STREAMING_AUDIO_TOKENS if streaming else VIBEVOICE_AUDIO_TOKENS+ return audio, {**QWEN25_EXTENDED_TOKENS, **audio}++# Chat template with audio support# Key modification: handles part['type'] == 'audio' or 'audio_url' -> '<|AUDIO|>'VIBEVOICE_CHAT_TEMPLATE = """{%- if tools %}@@ -174,20 +190,21 @@ def download_qwen_tokenizer_files(output_dir: str, qwen_model: str = DEFAULT_QWE)-def patch_tokenizer_config(output_dir: str) -> None:+def patch_tokenizer_config(output_dir: str, streaming: bool = False) -> None:"""Patch tokenizer_config.json with VibeVoice audio tokens and chat template."""+ audio_tokens, all_tokens = _token_maps(streaming)config_path = os.path.join(output_dir, "tokenizer_config.json")-+with open(config_path, "r", encoding="utf-8") as f:config = json.load(f)-+# 1. Add ALL extended tokens to added_tokens_decoder (Qwen2.5 + VibeVoice audio)if "added_tokens_decoder" not in config:config["added_tokens_decoder"] = {}-- for token, token_id in ALL_EXTENDED_TOKENS.items():++ for token, token_id in all_tokens.items():if str(token_id) not in config["added_tokens_decoder"]:# Determine if token should be marked as "special"# tool_call tokens are NOT special in Qwen2.5@@ -207,7 +224,7 @@ def patch_tokenizer_config(output_dir: str) -> None:if "additional_special_tokens" not in config:config["additional_special_tokens"] = []- for token in VIBEVOICE_AUDIO_TOKENS.keys():+ for token in audio_tokens.keys():if token not in config["additional_special_tokens"]:config["additional_special_tokens"].append(token)@@ -251,10 +268,11 @@ def patch_tokenizer_config(output_dir: str) -> None:print(f"Patched {config_path}")-def patch_tokenizer_json(output_dir: str) -> None:+def patch_tokenizer_json(output_dir: str, streaming: bool = False) -> None:"""Patch tokenizer.json with VibeVoice audio tokens."""+ _, all_tokens = _token_maps(streaming)tokenizer_path = os.path.join(output_dir, "tokenizer.json")with open(tokenizer_path, "r", encoding="utf-8") as f:@@ -267,7 +285,7 @@ def patch_tokenizer_json(output_dir: str) -> None:existing_ids.add(token_entry.get("id"))# Add ALL extended tokens (Qwen2.5 + VibeVoice audio)- for token, token_id in ALL_EXTENDED_TOKENS.items():+ for token, token_id in all_tokens.items():if token_id not in existing_ids:# Determine if token should be marked as "special"is_special = token not in ("<tool_call>", "</tool_call>", "<|fim_prefix|>",@@ -312,10 +330,11 @@ def generate_added_tokens_json(output_dir: str) -> None:print(f"Generated {output_path}")-def generate_special_tokens_map_json(output_dir: str) -> None:+def generate_special_tokens_map_json(output_dir: str, streaming: bool = False) -> None:"""Generate special_tokens_map.json with VibeVoice special tokens."""+ audio_tokens, _ = _token_maps(streaming)# Build the special tokens mapspecial_tokens_map = {"additional_special_tokens": [],@@ -325,7 +344,7 @@ def generate_special_tokens_map_json(output_dir: str) -> None:}# Add audio tokens as additional_special_tokens- for token in VIBEVOICE_AUDIO_TOKENS.keys():+ for token in audio_tokens.keys():special_tokens_map["additional_special_tokens"].append({"content": token,"lstrip": False,@@ -352,10 +371,11 @@ def generate_special_tokens_map_json(output_dir: str) -> None:print(f"Generated {output_path}")-def generate_vibevoice_tokenizer_files(output_dir: str, qwen_model: str = DEFAULT_QWEN_MODEL) -> None:+def generate_vibevoice_tokenizer_files(output_dir: str, qwen_model: str = DEFAULT_QWEN_MODEL,+ streaming: bool = False) -> None:"""Generate all 6 VibeVoice tokenizer files.-+Files generated:1. vocab.json - from Qwen2.5 (unchanged)2. merges.txt - from Qwen2.5 (unchanged)@@ -365,21 +385,21 @@ def generate_vibevoice_tokenizer_files(output_dir: str, qwen_model: str = DEFAUL6. special_tokens_map.json - generated with VibeVoice tokens"""print(f"=== Generating VibeVoice tokenizer files to {output_dir} ===\n")-+# Step 1: Download base files from Qwen2download_qwen_tokenizer_files(output_dir, qwen_model)-+# Step 2: Patch tokenizer_config.json- patch_tokenizer_config(output_dir)-+ patch_tokenizer_config(output_dir, streaming)+# Step 3: Patch tokenizer.json- patch_tokenizer_json(output_dir)-+ patch_tokenizer_json(output_dir, streaming)+# Step 4: Generate added_tokens.jsongenerate_added_tokens_json(output_dir)-+# Step 5: Generate special_tokens_map.json- generate_special_tokens_map_json(output_dir)+ generate_special_tokens_map_json(output_dir, streaming)print(f"\nβ All 6 tokenizer files generated in {output_dir}")@@ -544,6 +564,12 @@ def main():help=f"Qwen model to download base tokenizer from (default: {DEFAULT_QWEN_MODEL})")+ parser.add_argument(+ "--streaming",+ action="store_true",+ help="Use the streaming token layout (<|text_chunk_end|> at 151665)"+ )+args = parser.parse_args()# Determine output directory@@ -556,7 +582,7 @@ def main():try:# Generate files- generate_vibevoice_tokenizer_files(output_dir, args.qwen_model)+ generate_vibevoice_tokenizer_files(output_dir, args.qwen_model, args.streaming)# Compare if requestedif args.compare: