Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.
feat: enhance Whistle fine-tuning with local LoRA support and update documentation
jakmrobf69a0ad55a2
diff --git a/README.md b/README.mdindex ef0d600..950e951 100644--- a/README.md+++ b/README.md@@ -83,6 +83,14 @@ One engine holds both models: `needle_load` reads whichever one a `.cact` carrieThe benchmarks against Whisper and Moonshine, the architecture and the interactive demo are at [cactuscompute.com/whistle](https://cactuscompute.com/whistle); the weights and every platform engine are on [Hugging Face](https://huggingface.co/Cactus-Compute/whistle).+Whistle fine-tunes locally: LoRA adapters on the attention projections, base frozen, merged at export, 4-bit. Each JSONL line is `{"audio": "clip.wav", "text": "...", "language": "pt"}`; a language code outside the seven adds that language, and `--tokens` takes a file of words or characters to add to the vocabulary, one per line.++```sh+pip install "cactus-needle[train]"+needle whistle finetune clips.jsonl --tokens tokens.txt --out adapter.safetensors+needle whistle build --lora adapter.safetensors --out tuned.cact+```+## Guides- [How to design tools for Needle 3](https://cactuscompute.com/blog/designing-tools-for-needle): one tool per action, names users would say, formats in descriptions, constraints in the grammar, triggers.diff --git a/llms.txt b/llms.txtindex cb95889..29bac81 100644--- a/llms.txt+++ b/llms.txt@@ -190,6 +190,25 @@ agent.run("...")Options: `finetune --lora-rank 16 --lora-alpha 32 --lr 1e-4 --batch-size 16 --max-len 1024 --val-split 0.1 --checkpoint <base.safetensors> --checkpoint-dir <dir> --out <adapter.safetensors> --generate <n> --model <id> --workers <n>`; `--model` defaults to `deepseek/deepseek-flash-latest`, `--workers` defaults to `8`, and `--checkpoint-dir` defaults to `checkpoints`. `--generate` synthesizes extra examples through the configured OpenRouter endpoint before training. `build --bits 2|4 --upload` (with `NEEDLE_HF_REPO=<you>/<model>`).+Whistle fine-tunes the same way: LoRA on the frozen base, merged at export, 4-bit. Data is JSONL, one clip per line, the audio path relative to the file; a `language` outside `en de fr es it nl pl` adds that language:++```json+{"audio": "clips/0001.wav", "text": "Quero reservar uma mesa para dois.", "language": "pt"}+```++```sh+needle whistle finetune clips.jsonl --tokens tokens.txt # LoRA; auto-downloads base+needle whistle build checkpoints/whistle.safetensors --lora checkpoints/whistle_lora.safetensors --out my.cact+```++Run a tuned model:++```python+needle.transcribe("clip.wav", weights="my.cact")+```++Options: `whistle finetune --language <code> --tokens <file> --epochs 3 --lora-rank 16 --lora-alpha 32 --lr 1e-4 --batch-size 16 --val-split 0.1 --checkpoint <base.safetensors> --checkpoint-dir <dir> --out <adapter.safetensors>`; `--language` fills lines without one, `--tokens` is a file of words or characters to add to the vocabulary, one per line, and `--checkpoint-dir` defaults to `checkpoints`. Clips over 30 s are skipped, and the run prints the held-out word error rate (character error rate for Chinese, Japanese, Korean and Thai) on the 4-bit weights. `whistle build --layers N --upload` (with `NEEDLE_HF_REPO=<you>/<model>`).+## Platform (hosted fine-tuning)The hosted platform at https://cactuscompute.com/v1 trains every size from 2 layers up on GPUs, scores each on your test file, and serves the `.cact` files; the API reference is https://cactuscompute.com/llms/api. The package drives it with a key from https://cactuscompute.com/dashboard/api-keys in `NEEDLE_API_KEY` (never print it). Files are chat-format JSONL: `messages` (OpenAI chat turns, tool calls on the assistant turn as `tool_calls` with `arguments` as a JSON string) and OpenAI-form `tools`.@@ -239,6 +258,7 @@ Adapt by swapping the Literal values for your own and keeping the shapes: closed- `needle playground` - the browser UI locally (hosted at https://cactuscompute.com/needle).- `needle whistle playground [clip.wav] [--language en] [--keywords "Siobhan, Krzysztof"] [--word-timestamps] [--weights whistle.cact]` - Whistle speech to text in the terminal: press Enter to speak, Enter again to stop, or pass a WAV file; `/language`, `/keywords`, `/timestamps` and `/file` change the settings between recordings, and `/stream` switches to live transcription, the words printing as they are committed. Recording from the microphone needs the `[mic]` extra. `whistle.cact` is fetched from `Cactus-Compute/whistle` on first use and runs on the same engine as Needle; `NEEDLE_WHISTLE_WEIGHTS` points at a local copy and `NEEDLE3_LIB_PATH` at a local engine.- `needle whistle compare [clip.wav] [--weights whistle.cact]` - the same English speech through Whistle, Whisper tiny and base (openai-whisper) and Moonshine tiny v2 (moonshine-voice), one line per model with its weight size, time to first token, decode tokens per second, total time and transcript; `/file` runs a WAV between recordings. Whistle's timing comes from the engine's own JSON. Needs the `[mic,compare]` extras.+- `needle whistle finetune` / `needle whistle build [checkpoint] [--lora adapter] [--layers N] [--platform <folder>] [--out <path>]` - the Whistle fine-tuning pipeline (section above); `build` merges the adapter, slices the `--layers` rung (default 8), exports at 4 bits, and fetches the published `whistle.cact` on every run (without an adapter at full depth it ships that published archive as is). `--platform` also downloads that platform's engine and header and makes `--out` a folder holding them plus `whistle.cact`.- `needle download whistle [--out <dir>]` - pre-download the `whistle.cact` speech weights. There is one engine for both models, so `needle fetch` and `needle download <platform>` cover the library and the platform folders.- `needle fetch [--platform-tag <tag>] [--out <dir>]` - pre-download the inference engine for this machine (or another platform) into the cache; prints the path. For air-gapped devices, also see `NEEDLE3_LIB_PATH` and `HF_HUB_OFFLINE` at https://cactuscompute.com/blog/needle-python-docs.- `needle platform finetune | generate | jobs | models | files | billing` - the hosted platform from the CLI (section above).diff --git a/needle/agent/whistle.py b/needle/agent/whistle.pyindex ea23173..4f7ec0f 100644--- a/needle/agent/whistle.py+++ b/needle/agent/whistle.py@@ -14,6 +14,7 @@ import waveSAMPLE_RATE = 16000LANGUAGES = ("en", "de", "fr", "es", "it", "nl", "pl")+OUTPUT_BYTES_PER_SECOND = 4096_handle = None_loaded = None@@ -51,21 +52,52 @@ def _lib():return _handle+_WAV_TYPES = {(1, 1): "B", (1, 2): "h", (1, 3): None, (1, 4): "i", (3, 4): "f", (3, 8): "d"}+++def _wav_chunks(path):+ with open(path, "rb") as source:+ data = source.read()+ chunks, at = {}, 12+ while at + 8 <= len(data):+ size = struct.unpack_from("<I", data, at + 4)[0]+ chunks.setdefault(data[at:at + 4], data[at + 8:at + 8 + size])+ at += 8 + size + (size & 1)+ if data[:4] != b"RIFF" or data[8:12] != b"WAVE" or b"fmt " not in chunks or b"data" not in chunks:+ raise wave.Error(f"{path} is not a WAV file")+ tag, channels, rate, _, align, _ = struct.unpack_from("<HHIIHH", chunks[b"fmt "])+ if tag == 0xFFFE:+ tag = struct.unpack_from("<H", chunks[b"fmt "], 24)[0]+ width = align // channels+ if (tag, width) not in _WAV_TYPES:+ raise wave.Error(f"{path} is not integer PCM or float WAV")+ return tag, channels, width, rate, chunks[b"data"]++def _read_wav(path):- with wave.open(path, "rb") as source:- channels, width, rate = source.getnchannels(), source.getsampwidth(), source.getframerate()- raw = source.readframes(source.getnframes())- if width == 3:- values = [int.from_bytes(raw[i:i + 3], "little", signed=True) for i in range(0, len(raw), 3)]+ tag, channels, width, rate, raw = _wav_chunks(path)+ offset, scale = (0, 1.0) if tag == 3 else (128, 128.0) if width == 1 else (0, float(1 << (8 * width - 1)))+ try:+ import numpy+ except ImportError:+ numpy = None+ if numpy is not None:+ if width == 3:+ bytes3 = numpy.frombuffer(raw, numpy.uint8).reshape(-1, 3).astype(numpy.int32)+ values = (bytes3[:, 0] | bytes3[:, 1] << 8 | bytes3[:, 2] << 16) << 8 >> 8+ else:+ values = numpy.frombuffer(raw, "<" + _WAV_TYPES[tag, width])+ mono = (values.reshape(-1, channels).mean(axis=1, dtype=numpy.float32) - offset) / numpy.float32(scale)else:- values = array.array({1: "B", 2: "h", 4: "i"}[width])- values.frombytes(raw)- if sys.byteorder == "big":- values.byteswap()- offset, scale = (128, 128.0) if width == 1 else (0, float(1 << (8 * width - 1)))- scale *= channels- mono = [(sum(frame) - offset * channels) / scale for frame in zip(*(values[c::channels] for c in range(channels)))]- if rate == SAMPLE_RATE or not mono:+ if width == 3:+ values = [int.from_bytes(raw[i:i + 3], "little", signed=True) for i in range(0, len(raw), 3)]+ else:+ values = array.array(_WAV_TYPES[tag, width])+ values.frombytes(raw)+ if sys.byteorder == "big":+ values.byteswap()+ mono = [(sum(frame) / channels - offset) / scale for frame in zip(*(values[c::channels] for c in range(channels)))]+ if rate == SAMPLE_RATE or not len(mono):return monotry:import numpy@@ -151,6 +183,10 @@ class Whistle:raise RuntimeError(_lib().needle_last_error().decode("utf-8", "replace"))return json.loads(self._buffer.value.decode("utf-8", "replace"))+ def _reserve(self, size):+ if size > len(self._buffer):+ self._buffer = ctypes.create_string_buffer(size)+def transcribe(self, audio, language=None, keywords=None, word_timestamps=False) -> dict:"""Returns {"text", "language", "ttft_ms", "decode_tps"}, plus "words" with start, end and probability when word_timestamps is set.@@ -159,6 +195,7 @@ class Whistle:lib = self._bind()samples, count = _samples(audio)language, keywords = _options(language, keywords)+ self._reserve(count // SAMPLE_RATE * OUTPUT_BYTES_PER_SECOND)return self._result(lib.needle_transcribe(samples, count, language, keywords, int(bool(word_timestamps)),self._buffer, len(self._buffer)))@@ -174,6 +211,7 @@ class Whistle:try:for chunk in chunks:samples, count = _samples(chunk)+ self._reserve(count // SAMPLE_RATE * OUTPUT_BYTES_PER_SECOND)yield self._result(lib.needle_stream_transcribe_process(samples, count, language, keywords, self._buffer,len(self._buffer)))except BaseException:diff --git a/needle/model/whistle_finetune.py b/needle/model/whistle_finetune.pyindex e154110..0e8c887 100644--- a/needle/model/whistle_finetune.py+++ b/needle/model/whistle_finetune.py@@ -62,7 +62,12 @@ def read_tokens(path):if not os.path.isfile(path):raise SystemExit(f"--tokens takes a file with one token per line; {path} is not a file")with open(path, encoding="utf-8-sig") as handle:- return [line.strip() for line in handle if line.strip()]+ tokens = [line.strip() for line in handle if line.strip()]+ for token in tokens:+ if re.fullmatch(r"<\|.*\|>", token):+ raise SystemExit(f"{path}: {token} looks like a language tag; a new language gets its tag from the clips' "+ "\"language\" code, so leave it out of --tokens")+ return tokensdef token_pieces(sp, tokens):@@ -81,19 +86,14 @@ def token_pieces(sp, tokens):def read_audio(path):import wave+ from ..agent.whistle import _read_wavtry:- with wave.open(path, "rb") as source:- channels, width, rate = source.getnchannels(), source.getsampwidth(), source.getframerate()- raw = source.readframes(source.getnframes())- if width != 2:- from ..agent.whistle import _read_wav- return np.asarray(_read_wav(path), np.float32)- samples = np.frombuffer(raw, "<i2").reshape(-1, channels).mean(1).astype(np.float32) / 32768.0- except (wave.Error, EOFError):+ return np.asarray(_read_wav(path), np.float32)+ except wave.Error:try:import soundfileexcept ImportError:- raise RuntimeError(f"{path} is not a PCM WAV file; reading it needs soundfile: "+ raise RuntimeError(f"{path} is not a PCM or float WAV file; reading it needs soundfile: ""pip install soundfile") from Nonedata, rate = soundfile.read(path, dtype="float32", always_2d=True)samples = data.mean(1)diff --git a/tests/test_whistle.py b/tests/test_whistle.pyindex 78bb467..efcdd7f 100644--- a/tests/test_whistle.py+++ b/tests/test_whistle.py@@ -17,7 +17,7 @@ def _whistle_weights():return os.environ.get("NEEDLE_WHISTLE_WEIGHTS") or os.path.join(fetch.cache_dir(fetch.WHISTLE), fetch.base_weights(fetch.WHISTLE))-requires_whistle = pytest.mark.skipif(not (_engine_available("whistle") and os.path.exists(_whistle_weights())),+requires_whistle = pytest.mark.skipif(not (_engine_available(3) and os.path.exists(_whistle_weights())),reason="Whistle engine or whistle.cact not installed (auto-fetched from HF on first real use)")@@ -165,6 +165,50 @@ def test_wav_files_become_16k_mono_floats(tmp_path, rate, channels, width):assert 108 <= crossings <= 111+def _riff(tag, channels, rate, width, payload, extensible=False, before=b"", after=b""):+ fmt = struct.pack("<HHIIHH", 0xFFFE if extensible else tag, channels, rate, rate * channels * width, channels * width, 8 * width)+ if extensible:+ fmt += struct.pack("<HHIH", 22, 8 * width, 0, tag) + bytes(14)++ def chunk(name, body):+ return name + struct.pack("<I", len(body)) + body + b"\0" * (len(body) & 1)++ body = b"WAVE" + before + chunk(b"fmt ", fmt) + after + chunk(b"data", payload)+ return b"RIFF" + struct.pack("<I", len(body)) + body++[email protected]("numpy_available", [True, False])+def test_float_and_extensible_wav_files_read_like_pcm(tmp_path, monkeypatch, numpy_available):+ from needle.agent.whistle import _read_wav++ if not numpy_available:+ monkeypatch.setitem(sys.modules, "numpy", None)+ values = [0.0, 0.5, -0.25, 0.75]+ cases = {+ "float32": _riff(3, 1, 16000, 4, struct.pack("<4f", *values)),+ "float64 stereo": _riff(3, 2, 16000, 8, struct.pack("<8d", *[v for v in values for _ in range(2)])),+ "extensible pcm16": _riff(1, 1, 16000, 2, struct.pack("<4h", *[int(v * 32768) for v in values]), extensible=True),+ "extensible float32": _riff(3, 1, 16000, 4, struct.pack("<4f", *values), extensible=True),+ "odd chunk first": _riff(3, 1, 16000, 4, struct.pack("<4f", *values), before=b"LIST\x05\0\0\0abcde\0"),+ }+ for name, data in cases.items():+ path = tmp_path / f"{name}.wav"+ path.write_bytes(data)+ assert [round(float(v), 4) for v in _read_wav(str(path))] == values, name+++def test_files_that_are_not_pcm_or_float_wav_fail(tmp_path):+ from needle.agent.whistle import _read_wav++ path = tmp_path / "clip.wav"+ for data in (b"", b"fLaC" + bytes(40), b"RIFF\x04\0\0\0WAVE", _riff(7, 1, 8000, 1, bytes(8))):+ path.write_bytes(data)+ with pytest.raises(wave.Error):+ _read_wav(str(path))+ path.write_bytes(_riff(1, 1, 16000, 2, b""))+ assert len(_read_wav(str(path))) == 0++def test_samples_accept_lists_arrays_and_float32_bytes():from needle.agent.whistle import _samples@@ -318,6 +362,35 @@ class _StreamEngine:return self._write(out, "tail", len(self.calls))+class _TranscribeEngine:+ def __init__(self):+ self.capacities = []++ def needle_load(self, data, size):+ return 0++ def needle_transcribe(self, samples, count, language, keywords, word_timestamps, out, capacity):+ self.capacities.append(capacity)+ out.value = json.dumps({"text": "", "language": "", "ttft_ms": 0.0, "decode_tps": 0.0}).encode()+ return 0+++def test_output_buffers_grow_with_the_audio(monkeypatch):+ from needle.agent import whistle++ monkeypatch.setattr(whistle, "_loaded", None)+ engine = _TranscribeEngine()+ monkeypatch.setattr(whistle, "_lib", lambda: engine)+ model = whistle.Whistle(weights=__file__)+ model.transcribe([0.0] * 16000)+ model.transcribe([0.0] * (3600 * 16000))+ assert engine.capacities == [1 << 18, 3600 * whistle.OUTPUT_BYTES_PER_SECOND]+ monkeypatch.setattr(whistle, "_lib", lambda: _StreamEngine())+ streamed = whistle.Whistle(weights=__file__)+ list(streamed.stream([[0.0] * (120 * 16000)]))+ assert len(streamed._buffer) == 120 * whistle.OUTPUT_BYTES_PER_SECOND++def test_stream_feeds_each_chunk_and_flushes_the_tail(monkeypatch):from needle.agent import whistlediff --git a/tests/test_whistle_finetune.py b/tests/test_whistle_finetune.pyindex 36656ea..fceb719 100644--- a/tests/test_whistle_finetune.py+++ b/tests/test_whistle_finetune.py@@ -4,6 +4,8 @@ import waveimport pytest+from conftest import _engine_available+TINY = dict(dim=64, kv_heads=2, qk_dim=16, v_dim=16, enc_layers=2, dec_layers=3, conv_kernel=3, lanes=2,enc_lanes=2, sites=(2,), slots=64, hashes=1, stem=8, stem_stages=2, max_tokens=64)@@ -190,6 +192,9 @@ def test_tokens_come_from_a_file(tmp_path):listed = tmp_path / "tokens.txt"listed.write_bytes("ação \r\n\n coração\n".encode("utf-8"))assert read_tokens(str(listed)) == ["ação", "coração"]+ listed.write_text("ação\n<|pt|>\n", encoding="utf-8")+ with pytest.raises(SystemExit, match="looks like a language tag"):+ read_tokens(str(listed))@pytest.mark.slow@@ -242,3 +247,28 @@ def test_cli_routes_finetune_and_build_and_takes_a_tuned_language(monkeypatch):"clips.jsonl", "pt", "tokens.txt", 16, 1e-4)assert (built.lora, built.layers, built.out, built.platform, built.upload) == ("a.safetensors", 4, "t.cact", None, False)assert played.language == "pt"++[email protected](not _engine_available(3), reason="Whistle engine not installed")+def test_the_python_encoder_matches_the_engine_embedding(tiny_whistle, published_whistle):+ import numpy as np+ from needle.agent.whistle import Whistle as Engine+ from needle.model.quantize import WEIGHT_BITS, cq_ste_params+ from needle.model.whistle import Whistle+ from needle.model.whistle_finetune import load_base++ params, cfg = load_base(tiny_whistle)+ quantised = cq_ste_params(params, WEIGHT_BITS)+ engine = Engine(weights=published_whistle[0])+ t = np.arange(32000)+ sound = 0.2 * np.sin(0.07 * t) * (1 + np.sin(0.002 * t)) + 0.01 * np.random.default_rng(0).standard_normal(32000)+ for level in (1.0, 0.01):+ audio = (sound * level).astype(np.float32)+ theirs = np.array(engine.embed(audio)).reshape(-1, cfg.dim)+ spans = np.sqrt((audio[:len(audio) // 320 * 320].reshape(-1, 320) ** 2).mean(1))+ levelled = audio * (0.1 / np.sort(spans)[int(0.99 * (len(spans) - 1))])+ ours, frames = Whistle(cfg).apply({"params": quantised}, levelled[None], np.array([len(audio)]), True, method=Whistle.encode)+ ours = np.asarray(ours[0, :int(frames[0])])+ assert ours.shape == theirs.shape+ assert np.linalg.norm(ours - theirs) < 0.05 * np.linalg.norm(theirs)