Skip to content

Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.

feat: enhance Whistle fine-tuning with local LoRA support and update documentation

jakmrobf69a0ad55a2
diff --git a/README.md b/README.md
index ef0d600..950e951 100644
--- a/README.md
+++ b/README.md
@@ -83,6 +83,14 @@ One engine holds both models: `needle_load` reads whichever one a `.cact` carrie
The benchmarks against Whisper and Moonshine, the architecture and the interactive demo are at [cactuscompute.com/whistle](https://cactuscompute.com/whistle); the weights and every platform engine are on [Hugging Face](https://huggingface.co/Cactus-Compute/whistle).
+Whistle fine-tunes locally: LoRA adapters on the attention projections, base frozen, merged at export, 4-bit. Each JSONL line is `{"audio": "clip.wav", "text": "...", "language": "pt"}`; a language code outside the seven adds that language, and `--tokens` takes a file of words or characters to add to the vocabulary, one per line.
+
+```sh
+pip install "cactus-needle[train]"
+needle whistle finetune clips.jsonl --tokens tokens.txt --out adapter.safetensors
+needle whistle build --lora adapter.safetensors --out tuned.cact
+```
+
## Guides
- [How to design tools for Needle 3](https://cactuscompute.com/blog/designing-tools-for-needle): one tool per action, names users would say, formats in descriptions, constraints in the grammar, triggers.
diff --git a/llms.txt b/llms.txt
index cb95889..29bac81 100644
--- a/llms.txt
+++ b/llms.txt
@@ -190,6 +190,25 @@ agent.run("...")
Options: `finetune --lora-rank 16 --lora-alpha 32 --lr 1e-4 --batch-size 16 --max-len 1024 --val-split 0.1 --checkpoint <base.safetensors> --checkpoint-dir <dir> --out <adapter.safetensors> --generate <n> --model <id> --workers <n>`; `--model` defaults to `deepseek/deepseek-flash-latest`, `--workers` defaults to `8`, and `--checkpoint-dir` defaults to `checkpoints`. `--generate` synthesizes extra examples through the configured OpenRouter endpoint before training. `build --bits 2|4 --upload` (with `NEEDLE_HF_REPO=<you>/<model>`).
+Whistle fine-tunes the same way: LoRA on the frozen base, merged at export, 4-bit. Data is JSONL, one clip per line, the audio path relative to the file; a `language` outside `en de fr es it nl pl` adds that language:
+
+```json
+{"audio": "clips/0001.wav", "text": "Quero reservar uma mesa para dois.", "language": "pt"}
+```
+
+```sh
+needle whistle finetune clips.jsonl --tokens tokens.txt # LoRA; auto-downloads base
+needle whistle build checkpoints/whistle.safetensors --lora checkpoints/whistle_lora.safetensors --out my.cact
+```
+
+Run a tuned model:
+
+```python
+needle.transcribe("clip.wav", weights="my.cact")
+```
+
+Options: `whistle finetune --language <code> --tokens <file> --epochs 3 --lora-rank 16 --lora-alpha 32 --lr 1e-4 --batch-size 16 --val-split 0.1 --checkpoint <base.safetensors> --checkpoint-dir <dir> --out <adapter.safetensors>`; `--language` fills lines without one, `--tokens` is a file of words or characters to add to the vocabulary, one per line, and `--checkpoint-dir` defaults to `checkpoints`. Clips over 30 s are skipped, and the run prints the held-out word error rate (character error rate for Chinese, Japanese, Korean and Thai) on the 4-bit weights. `whistle build --layers N --upload` (with `NEEDLE_HF_REPO=<you>/<model>`).
+
## Platform (hosted fine-tuning)
The hosted platform at https://cactuscompute.com/v1 trains every size from 2 layers up on GPUs, scores each on your test file, and serves the `.cact` files; the API reference is https://cactuscompute.com/llms/api. The package drives it with a key from https://cactuscompute.com/dashboard/api-keys in `NEEDLE_API_KEY` (never print it). Files are chat-format JSONL: `messages` (OpenAI chat turns, tool calls on the assistant turn as `tool_calls` with `arguments` as a JSON string) and OpenAI-form `tools`.
@@ -239,6 +258,7 @@ Adapt by swapping the Literal values for your own and keeping the shapes: closed
- `needle playground` - the browser UI locally (hosted at https://cactuscompute.com/needle).
- `needle whistle playground [clip.wav] [--language en] [--keywords "Siobhan, Krzysztof"] [--word-timestamps] [--weights whistle.cact]` - Whistle speech to text in the terminal: press Enter to speak, Enter again to stop, or pass a WAV file; `/language`, `/keywords`, `/timestamps` and `/file` change the settings between recordings, and `/stream` switches to live transcription, the words printing as they are committed. Recording from the microphone needs the `[mic]` extra. `whistle.cact` is fetched from `Cactus-Compute/whistle` on first use and runs on the same engine as Needle; `NEEDLE_WHISTLE_WEIGHTS` points at a local copy and `NEEDLE3_LIB_PATH` at a local engine.
- `needle whistle compare [clip.wav] [--weights whistle.cact]` - the same English speech through Whistle, Whisper tiny and base (openai-whisper) and Moonshine tiny v2 (moonshine-voice), one line per model with its weight size, time to first token, decode tokens per second, total time and transcript; `/file` runs a WAV between recordings. Whistle's timing comes from the engine's own JSON. Needs the `[mic,compare]` extras.
+- `needle whistle finetune` / `needle whistle build [checkpoint] [--lora adapter] [--layers N] [--platform <folder>] [--out <path>]` - the Whistle fine-tuning pipeline (section above); `build` merges the adapter, slices the `--layers` rung (default 8), exports at 4 bits, and fetches the published `whistle.cact` on every run (without an adapter at full depth it ships that published archive as is). `--platform` also downloads that platform's engine and header and makes `--out` a folder holding them plus `whistle.cact`.
- `needle download whistle [--out <dir>]` - pre-download the `whistle.cact` speech weights. There is one engine for both models, so `needle fetch` and `needle download <platform>` cover the library and the platform folders.
- `needle fetch [--platform-tag <tag>] [--out <dir>]` - pre-download the inference engine for this machine (or another platform) into the cache; prints the path. For air-gapped devices, also see `NEEDLE3_LIB_PATH` and `HF_HUB_OFFLINE` at https://cactuscompute.com/blog/needle-python-docs.
- `needle platform finetune | generate | jobs | models | files | billing` - the hosted platform from the CLI (section above).
diff --git a/needle/agent/whistle.py b/needle/agent/whistle.py
index ea23173..4f7ec0f 100644
--- a/needle/agent/whistle.py
+++ b/needle/agent/whistle.py
@@ -14,6 +14,7 @@ import wave
SAMPLE_RATE = 16000
LANGUAGES = ("en", "de", "fr", "es", "it", "nl", "pl")
+OUTPUT_BYTES_PER_SECOND = 4096
_handle = None
_loaded = None
@@ -51,21 +52,52 @@ def _lib():
return _handle
+_WAV_TYPES = {(1, 1): "B", (1, 2): "h", (1, 3): None, (1, 4): "i", (3, 4): "f", (3, 8): "d"}
+
+
+def _wav_chunks(path):
+ with open(path, "rb") as source:
+ data = source.read()
+ chunks, at = {}, 12
+ while at + 8 <= len(data):
+ size = struct.unpack_from("<I", data, at + 4)[0]
+ chunks.setdefault(data[at:at + 4], data[at + 8:at + 8 + size])
+ at += 8 + size + (size & 1)
+ if data[:4] != b"RIFF" or data[8:12] != b"WAVE" or b"fmt " not in chunks or b"data" not in chunks:
+ raise wave.Error(f"{path} is not a WAV file")
+ tag, channels, rate, _, align, _ = struct.unpack_from("<HHIIHH", chunks[b"fmt "])
+ if tag == 0xFFFE:
+ tag = struct.unpack_from("<H", chunks[b"fmt "], 24)[0]
+ width = align // channels
+ if (tag, width) not in _WAV_TYPES:
+ raise wave.Error(f"{path} is not integer PCM or float WAV")
+ return tag, channels, width, rate, chunks[b"data"]
+
+
def _read_wav(path):
- with wave.open(path, "rb") as source:
- channels, width, rate = source.getnchannels(), source.getsampwidth(), source.getframerate()
- raw = source.readframes(source.getnframes())
- if width == 3:
- values = [int.from_bytes(raw[i:i + 3], "little", signed=True) for i in range(0, len(raw), 3)]
+ tag, channels, width, rate, raw = _wav_chunks(path)
+ offset, scale = (0, 1.0) if tag == 3 else (128, 128.0) if width == 1 else (0, float(1 << (8 * width - 1)))
+ try:
+ import numpy
+ except ImportError:
+ numpy = None
+ if numpy is not None:
+ if width == 3:
+ bytes3 = numpy.frombuffer(raw, numpy.uint8).reshape(-1, 3).astype(numpy.int32)
+ values = (bytes3[:, 0] | bytes3[:, 1] << 8 | bytes3[:, 2] << 16) << 8 >> 8
+ else:
+ values = numpy.frombuffer(raw, "<" + _WAV_TYPES[tag, width])
+ mono = (values.reshape(-1, channels).mean(axis=1, dtype=numpy.float32) - offset) / numpy.float32(scale)
else:
- values = array.array({1: "B", 2: "h", 4: "i"}[width])
- values.frombytes(raw)
- if sys.byteorder == "big":
- values.byteswap()
- offset, scale = (128, 128.0) if width == 1 else (0, float(1 << (8 * width - 1)))
- scale *= channels
- mono = [(sum(frame) - offset * channels) / scale for frame in zip(*(values[c::channels] for c in range(channels)))]
- if rate == SAMPLE_RATE or not mono:
+ if width == 3:
+ values = [int.from_bytes(raw[i:i + 3], "little", signed=True) for i in range(0, len(raw), 3)]
+ else:
+ values = array.array(_WAV_TYPES[tag, width])
+ values.frombytes(raw)
+ if sys.byteorder == "big":
+ values.byteswap()
+ mono = [(sum(frame) / channels - offset) / scale for frame in zip(*(values[c::channels] for c in range(channels)))]
+ if rate == SAMPLE_RATE or not len(mono):
return mono
try:
import numpy
@@ -151,6 +183,10 @@ class Whistle:
raise RuntimeError(_lib().needle_last_error().decode("utf-8", "replace"))
return json.loads(self._buffer.value.decode("utf-8", "replace"))
+ def _reserve(self, size):
+ if size > len(self._buffer):
+ self._buffer = ctypes.create_string_buffer(size)
+
def transcribe(self, audio, language=None, keywords=None, word_timestamps=False) -> dict:
"""Returns {"text", "language", "ttft_ms", "decode_tps"}, plus "words" with start, end and probability when word_timestamps is set.
@@ -159,6 +195,7 @@ class Whistle:
lib = self._bind()
samples, count = _samples(audio)
language, keywords = _options(language, keywords)
+ self._reserve(count // SAMPLE_RATE * OUTPUT_BYTES_PER_SECOND)
return self._result(lib.needle_transcribe(samples, count, language, keywords, int(bool(word_timestamps)),
self._buffer, len(self._buffer)))
@@ -174,6 +211,7 @@ class Whistle:
try:
for chunk in chunks:
samples, count = _samples(chunk)
+ self._reserve(count // SAMPLE_RATE * OUTPUT_BYTES_PER_SECOND)
yield self._result(lib.needle_stream_transcribe_process(samples, count, language, keywords, self._buffer,
len(self._buffer)))
except BaseException:
diff --git a/needle/model/whistle_finetune.py b/needle/model/whistle_finetune.py
index e154110..0e8c887 100644
--- a/needle/model/whistle_finetune.py
+++ b/needle/model/whistle_finetune.py
@@ -62,7 +62,12 @@ def read_tokens(path):
if not os.path.isfile(path):
raise SystemExit(f"--tokens takes a file with one token per line; {path} is not a file")
with open(path, encoding="utf-8-sig") as handle:
- return [line.strip() for line in handle if line.strip()]
+ tokens = [line.strip() for line in handle if line.strip()]
+ for token in tokens:
+ if re.fullmatch(r"<\|.*\|>", token):
+ raise SystemExit(f"{path}: {token} looks like a language tag; a new language gets its tag from the clips' "
+ "\"language\" code, so leave it out of --tokens")
+ return tokens
def token_pieces(sp, tokens):
@@ -81,19 +86,14 @@ def token_pieces(sp, tokens):
def read_audio(path):
import wave
+ from ..agent.whistle import _read_wav
try:
- with wave.open(path, "rb") as source:
- channels, width, rate = source.getnchannels(), source.getsampwidth(), source.getframerate()
- raw = source.readframes(source.getnframes())
- if width != 2:
- from ..agent.whistle import _read_wav
- return np.asarray(_read_wav(path), np.float32)
- samples = np.frombuffer(raw, "<i2").reshape(-1, channels).mean(1).astype(np.float32) / 32768.0
- except (wave.Error, EOFError):
+ return np.asarray(_read_wav(path), np.float32)
+ except wave.Error:
try:
import soundfile
except ImportError:
- raise RuntimeError(f"{path} is not a PCM WAV file; reading it needs soundfile: "
+ raise RuntimeError(f"{path} is not a PCM or float WAV file; reading it needs soundfile: "
"pip install soundfile") from None
data, rate = soundfile.read(path, dtype="float32", always_2d=True)
samples = data.mean(1)
diff --git a/tests/test_whistle.py b/tests/test_whistle.py
index 78bb467..efcdd7f 100644
--- a/tests/test_whistle.py
+++ b/tests/test_whistle.py
@@ -17,7 +17,7 @@ def _whistle_weights():
return os.environ.get("NEEDLE_WHISTLE_WEIGHTS") or os.path.join(fetch.cache_dir(fetch.WHISTLE), fetch.base_weights(fetch.WHISTLE))
-requires_whistle = pytest.mark.skipif(not (_engine_available("whistle") and os.path.exists(_whistle_weights())),
+requires_whistle = pytest.mark.skipif(not (_engine_available(3) and os.path.exists(_whistle_weights())),
reason="Whistle engine or whistle.cact not installed (auto-fetched from HF on first real use)")
@@ -165,6 +165,50 @@ def test_wav_files_become_16k_mono_floats(tmp_path, rate, channels, width):
assert 108 <= crossings <= 111
+def _riff(tag, channels, rate, width, payload, extensible=False, before=b"", after=b""):
+ fmt = struct.pack("<HHIIHH", 0xFFFE if extensible else tag, channels, rate, rate * channels * width, channels * width, 8 * width)
+ if extensible:
+ fmt += struct.pack("<HHIH", 22, 8 * width, 0, tag) + bytes(14)
+
+ def chunk(name, body):
+ return name + struct.pack("<I", len(body)) + body + b"\0" * (len(body) & 1)
+
+ body = b"WAVE" + before + chunk(b"fmt ", fmt) + after + chunk(b"data", payload)
+ return b"RIFF" + struct.pack("<I", len(body)) + body
+
+
[email protected]("numpy_available", [True, False])
+def test_float_and_extensible_wav_files_read_like_pcm(tmp_path, monkeypatch, numpy_available):
+ from needle.agent.whistle import _read_wav
+
+ if not numpy_available:
+ monkeypatch.setitem(sys.modules, "numpy", None)
+ values = [0.0, 0.5, -0.25, 0.75]
+ cases = {
+ "float32": _riff(3, 1, 16000, 4, struct.pack("<4f", *values)),
+ "float64 stereo": _riff(3, 2, 16000, 8, struct.pack("<8d", *[v for v in values for _ in range(2)])),
+ "extensible pcm16": _riff(1, 1, 16000, 2, struct.pack("<4h", *[int(v * 32768) for v in values]), extensible=True),
+ "extensible float32": _riff(3, 1, 16000, 4, struct.pack("<4f", *values), extensible=True),
+ "odd chunk first": _riff(3, 1, 16000, 4, struct.pack("<4f", *values), before=b"LIST\x05\0\0\0abcde\0"),
+ }
+ for name, data in cases.items():
+ path = tmp_path / f"{name}.wav"
+ path.write_bytes(data)
+ assert [round(float(v), 4) for v in _read_wav(str(path))] == values, name
+
+
+def test_files_that_are_not_pcm_or_float_wav_fail(tmp_path):
+ from needle.agent.whistle import _read_wav
+
+ path = tmp_path / "clip.wav"
+ for data in (b"", b"fLaC" + bytes(40), b"RIFF\x04\0\0\0WAVE", _riff(7, 1, 8000, 1, bytes(8))):
+ path.write_bytes(data)
+ with pytest.raises(wave.Error):
+ _read_wav(str(path))
+ path.write_bytes(_riff(1, 1, 16000, 2, b""))
+ assert len(_read_wav(str(path))) == 0
+
+
def test_samples_accept_lists_arrays_and_float32_bytes():
from needle.agent.whistle import _samples
@@ -318,6 +362,35 @@ class _StreamEngine:
return self._write(out, "tail", len(self.calls))
+class _TranscribeEngine:
+ def __init__(self):
+ self.capacities = []
+
+ def needle_load(self, data, size):
+ return 0
+
+ def needle_transcribe(self, samples, count, language, keywords, word_timestamps, out, capacity):
+ self.capacities.append(capacity)
+ out.value = json.dumps({"text": "", "language": "", "ttft_ms": 0.0, "decode_tps": 0.0}).encode()
+ return 0
+
+
+def test_output_buffers_grow_with_the_audio(monkeypatch):
+ from needle.agent import whistle
+
+ monkeypatch.setattr(whistle, "_loaded", None)
+ engine = _TranscribeEngine()
+ monkeypatch.setattr(whistle, "_lib", lambda: engine)
+ model = whistle.Whistle(weights=__file__)
+ model.transcribe([0.0] * 16000)
+ model.transcribe([0.0] * (3600 * 16000))
+ assert engine.capacities == [1 << 18, 3600 * whistle.OUTPUT_BYTES_PER_SECOND]
+ monkeypatch.setattr(whistle, "_lib", lambda: _StreamEngine())
+ streamed = whistle.Whistle(weights=__file__)
+ list(streamed.stream([[0.0] * (120 * 16000)]))
+ assert len(streamed._buffer) == 120 * whistle.OUTPUT_BYTES_PER_SECOND
+
+
def test_stream_feeds_each_chunk_and_flushes_the_tail(monkeypatch):
from needle.agent import whistle
diff --git a/tests/test_whistle_finetune.py b/tests/test_whistle_finetune.py
index 36656ea..fceb719 100644
--- a/tests/test_whistle_finetune.py
+++ b/tests/test_whistle_finetune.py
@@ -4,6 +4,8 @@ import wave
import pytest
+from conftest import _engine_available
+
TINY = dict(dim=64, kv_heads=2, qk_dim=16, v_dim=16, enc_layers=2, dec_layers=3, conv_kernel=3, lanes=2,
enc_lanes=2, sites=(2,), slots=64, hashes=1, stem=8, stem_stages=2, max_tokens=64)
@@ -190,6 +192,9 @@ def test_tokens_come_from_a_file(tmp_path):
listed = tmp_path / "tokens.txt"
listed.write_bytes("ação \r\n\n coração\n".encode("utf-8"))
assert read_tokens(str(listed)) == ["ação", "coração"]
+ listed.write_text("ação\n<|pt|>\n", encoding="utf-8")
+ with pytest.raises(SystemExit, match="looks like a language tag"):
+ read_tokens(str(listed))
@pytest.mark.slow
@@ -242,3 +247,28 @@ def test_cli_routes_finetune_and_build_and_takes_a_tuned_language(monkeypatch):
"clips.jsonl", "pt", "tokens.txt", 16, 1e-4)
assert (built.lora, built.layers, built.out, built.platform, built.upload) == ("a.safetensors", 4, "t.cact", None, False)
assert played.language == "pt"
+
+
[email protected](not _engine_available(3), reason="Whistle engine not installed")
+def test_the_python_encoder_matches_the_engine_embedding(tiny_whistle, published_whistle):
+ import numpy as np
+ from needle.agent.whistle import Whistle as Engine
+ from needle.model.quantize import WEIGHT_BITS, cq_ste_params
+ from needle.model.whistle import Whistle
+ from needle.model.whistle_finetune import load_base
+
+ params, cfg = load_base(tiny_whistle)
+ quantised = cq_ste_params(params, WEIGHT_BITS)
+ engine = Engine(weights=published_whistle[0])
+ t = np.arange(32000)
+ sound = 0.2 * np.sin(0.07 * t) * (1 + np.sin(0.002 * t)) + 0.01 * np.random.default_rng(0).standard_normal(32000)
+ for level in (1.0, 0.01):
+ audio = (sound * level).astype(np.float32)
+ theirs = np.array(engine.embed(audio)).reshape(-1, cfg.dim)
+ spans = np.sqrt((audio[:len(audio) // 320 * 320].reshape(-1, 320) ** 2).mean(1))
+ levelled = audio * (0.1 / np.sort(spans)[int(0.99 * (len(spans) - 1))])
+ ours, frames = Whistle(cfg).apply({"params": quantised}, levelled[None], np.array([len(audio)]), True, method=Whistle.encode)
+ ours = np.asarray(ours[0, :int(frames[0])])
+ assert ours.shape == theirs.shape
+ assert np.linalg.norm(ours - theirs) < 0.05 * np.linalg.norm(theirs)