Skip to content

Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.

docs: enhance needle.transcribe description with additional details on audio processing and voice activity detection

jakmro21aab0d878ef
diff --git a/README.md b/README.md
index 950e951..6329a26 100644
--- a/README.md
+++ b/README.md
@@ -67,7 +67,7 @@ print(needle.transcribe("clip.wav")["text"])
# turn off the kitchen lights
```
-Every call returns the text, the language, the milliseconds to the first token and the decoder's tokens per second after it. `word_timestamps=True` adds each word with its start, end and probability, `keywords=[...]` favours the names, product words and jargon your users say, and `language="de"` forces the language instead of detecting it. `needle.Whistle()` is the same model as an object, for `embed(audio)` or to hold one tuned `.cact`. Silence and steady noise return an empty transcript rather than an invented sentence. `needle.stream(chunks)` transcribes live for as long as the microphone runs, committing the words two passes agree on. `needle whistle playground` transcribes from the microphone, and `needle whistle compare` puts Whistle next to Whisper and Moonshine on the same clip.
+Every call returns the text, the language, the milliseconds to the first token and the decoder's tokens per second after it. `word_timestamps=True` adds each word with its start, end and probability, `keywords=[...]` favours the names, product words and jargon your users say, and `language="de"` forces the language instead of detecting it. `needle.Whistle()` is the same model as an object, for `embed(audio)` or to hold one tuned `.cact`. Silence and steady noise return an empty transcript rather than an invented sentence; other sounds, music especially, can still come out as text, so put a voice activity detector or a wake word in front of an always-on microphone. `needle.stream(chunks)` transcribes live for as long as the microphone runs, committing the words two passes agree on. `needle whistle playground` transcribes from the microphone, and `needle whistle compare` puts Whistle next to Whisper and Moonshine on the same clip.
At 16.9 MB Whistle is 8.6x smaller than Whisper base and 6.6x quicker to the first token. It is ahead on LibriSpeech test-clean and test-other, on SPGISpeech, on Earnings-22 and on the FLEURS average. Whisper base is ahead on TED-LIUM, on AMI and on the MLS average.
diff --git a/llms.txt b/llms.txt
index 29bac81..f20ce73 100644
--- a/llms.txt
+++ b/llms.txt
@@ -26,7 +26,7 @@ pip install "cactus-needle[train]" # adds JAX + training deps for finetune
- `needle.tool` - decorator that turns a function into a tool schema (attached as `fn._needle_tool`); `@needle.tool(triggers=[r"\b(turn|switch)\b.*\b(on|off)\b"])` adds case-insensitive regexes that route a matching request to this tool and require a call; a triggered call ships even below the confidence floor (raw schemas take the same `"triggers"` list beside `description`).
- `needle.Field(default=..., *, description, enum, const, ge, le, gt, lt, multiple_of, min_length, max_length, pattern, format, min_items, max_items, unique_items)` - per-argument constraints; attach inline with `typing.Annotated`.
- `needle.extract(text, schema, system=None, max_new_tokens=512, weights=None)` - one-shot extraction; returns a Pydantic instance if `schema` is a model, else a dict (or `None` if nothing matched). `weights` selects a tuned `.cact` and defaults to whatever the engine already has loaded, so extraction inherits the active tuned model unless you pass another path.
-- `needle.transcribe(audio, language=None, keywords=None, word_timestamps=False, weights=None) -> dict` - speech to text in English, German, French, Spanish, Italian, Dutch and Polish, on the same engine and the same install as Needle. `audio` is a WAV path or 16 kHz mono float samples in [-1, 1] of any length, one pass up to 30 s and windowed beyond it. Returns `{"text", "language", "ttft_ms", "decode_tps"}`, plus `"words"` with `start`, `end` and `probability` when `word_timestamps` is set. `language` is one of `en de fr es it nl pl`, or `None` to detect it; `keywords` is a list or a newline-separated string of words and phrases to favour. Silence and steady noise return an empty text and language. 16 kHz WAV files and raw samples need nothing extra; other sample rates need the `[mic]` extra for soxr. `whistle.cact` is fetched from `Cactus-Compute/whistle` on first use.
+- `needle.transcribe(audio, language=None, keywords=None, word_timestamps=False, weights=None) -> dict` - speech to text in English, German, French, Spanish, Italian, Dutch and Polish, on the same engine and the same install as Needle. `audio` is a WAV path or 16 kHz mono float samples in [-1, 1] of any length, one pass up to 30 s and windowed beyond it. Returns `{"text", "language", "ttft_ms", "decode_tps"}`, plus `"words"` with `start`, `end` and `probability` when `word_timestamps` is set. `language` is one of `en de fr es it nl pl`, or `None` to detect it; `keywords` is a list or a newline-separated string of words and phrases to favour. Silence and steady noise return an empty text and language; other sounds, music especially, can still come out as text, so put a voice activity detector or a wake word in front of an always-on microphone. 16 kHz WAV files and raw samples need nothing extra; other sample rates need the `[mic]` extra for soxr. `whistle.cact` is fetched from `Cactus-Compute/whistle` on first use.
- `needle.stream(chunks, language=None, keywords=None, weights=None)` - live transcription with no limit on length. `chunks` is an iterable of 16 kHz mono float sample buffers, about a second each; the generator yields one dict per chunk with the `"text"` and `"words"` committed by it (times from the start of the stream), the unconfirmed `"pending"` tail, the `"language"`, the seconds `"received"` and `"pass_ms"`, then one more for the tail when the chunks end. Join the texts with a space. Every chunk re-transcribes the last seconds and commits what two passes agree on.
- `needle.Whistle(weights=None, buffer_size=262144)` - the same model as an object, for `whistle.transcribe(...)`, `whistle.stream(...)`, `whistle.embed(audio)` (the encoder output, one row of floats per 80 ms frame, flattened) or to hold one tuned `.cact`. One model per process, not thread-safe.