Skip to content

Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.

feat: update Whistle fine-tuning process to support separate training, validation, and test JSONL files; enhance documentation and CLI arguments

jakmrob7c53453e91a
diff --git a/README.md b/README.md
index 6329a26..354da0a 100644
--- a/README.md
+++ b/README.md
@@ -83,11 +83,11 @@ One engine holds both models: `needle_load` reads whichever one a `.cact` carrie
The benchmarks against Whisper and Moonshine, the architecture and the interactive demo are at [cactuscompute.com/whistle](https://cactuscompute.com/whistle); the weights and every platform engine are on [Hugging Face](https://huggingface.co/Cactus-Compute/whistle).
-Whistle fine-tunes locally: LoRA adapters on the attention projections, base frozen, merged at export, 4-bit. Each JSONL line is `{"audio": "clip.wav", "text": "...", "language": "pt"}`; a language code outside the seven adds that language, and `--tokens` takes a file of words or characters to add to the vocabulary, one per line.
+Whistle fine-tunes locally: LoRA adapters on the attention projections, base frozen, merged at export, 4-bit. It takes training, validation and test JSONL files, each line `{"audio": "clip.wav", "text": "...", "language": "pt"}`; a language code outside the seven adds that language, and `--tokens` takes a file of words or characters to add to the vocabulary, one per line.
```sh
pip install "cactus-needle[train]"
-needle whistle finetune clips.jsonl --tokens tokens.txt --out adapter.safetensors
+needle whistle finetune train.jsonl validation.jsonl test.jsonl --tokens tokens.txt --out adapter.safetensors
needle whistle build --lora adapter.safetensors --out tuned.cact
```
@@ -97,6 +97,8 @@ needle whistle build --lora adapter.safetensors --out tuned.cact
- [Leveraging Needle's confidence](https://cactuscompute.com/blog/needle-confidence): what the score measures, what the engine withholds, and routing on act, confirm or refuse.
- [Structured JSON extraction with Needle](https://cactuscompute.com/blog/structured-extraction-with-needle): the record as the only tool, typed results, classification with enums.
- [Fine-tuning Needle](https://cactuscompute.com/blog/finetuning-needle): the data format, the commands, reading the loss, sizing the dataset.
+- [Fine-tuning Whistle](https://cactuscompute.com/blog/finetuning-whistle): the data format, new characters, the commands, how much data a new language needs.
+- [Getting the most out of Whistle](https://cactuscompute.com/blog/whistle-best-practices): input, word timestamps, live transcription, language, keywords, depth, Whistle with Needle.
- [Needle Python docs](https://cactuscompute.com/blog/needle-python-docs): the API, the response shape, the behaviour contract, system facts, tool retrieval, offline devices, environments, the CLI.
- [What devices are supported on Needle](https://cactuscompute.com/blog/needle-supported-devices): every platform folder, the CLI runner, the C API, the browser, WASI, air-gapped setup.
- [The .cact format](https://cactuscompute.com/blog/cact-format): the file the engine maps and reads in place, Cactus Quants at 2.125 bits per weight, and how to parse it yourself.
diff --git a/llms.txt b/llms.txt
index f20ce73..d5e69a3 100644
--- a/llms.txt
+++ b/llms.txt
@@ -25,7 +25,7 @@ pip install "cactus-needle[train]" # adds JAX + training deps for finetune
- `agent.reset()` - rewind the conversation, keep the tools loaded.
- `needle.tool` - decorator that turns a function into a tool schema (attached as `fn._needle_tool`); `@needle.tool(triggers=[r"\b(turn|switch)\b.*\b(on|off)\b"])` adds case-insensitive regexes that route a matching request to this tool and require a call; a triggered call ships even below the confidence floor (raw schemas take the same `"triggers"` list beside `description`).
- `needle.Field(default=..., *, description, enum, const, ge, le, gt, lt, multiple_of, min_length, max_length, pattern, format, min_items, max_items, unique_items)` - per-argument constraints; attach inline with `typing.Annotated`.
-- `needle.extract(text, schema, system=None, max_new_tokens=512, weights=None)` - one-shot extraction; returns a Pydantic instance if `schema` is a model, else a dict (or `None` if nothing matched). `weights` selects a tuned `.cact` and defaults to whatever the engine already has loaded, so extraction inherits the active tuned model unless you pass another path.
+- `needle.extract(text, schema, system=None, max_new_tokens=512, weights=None, strict=True, generation=None)` - one-shot extraction; returns a Pydantic instance if `schema` is a model, else a dict (or `None` if nothing matched). `weights` loads a tuned `.cact`; without it the base Needle 3 model runs (`generation=2` for Needle 2). `strict=True` raises `needle.ExtractionValidationError` instead of returning a date whose year contradicts the text or a value the engine flags as invented or negated.
- `needle.transcribe(audio, language=None, keywords=None, word_timestamps=False, weights=None) -> dict` - speech to text in English, German, French, Spanish, Italian, Dutch and Polish, on the same engine and the same install as Needle. `audio` is a WAV path or 16 kHz mono float samples in [-1, 1] of any length, one pass up to 30 s and windowed beyond it. Returns `{"text", "language", "ttft_ms", "decode_tps"}`, plus `"words"` with `start`, `end` and `probability` when `word_timestamps` is set. `language` is one of `en de fr es it nl pl`, or `None` to detect it; `keywords` is a list or a newline-separated string of words and phrases to favour. Silence and steady noise return an empty text and language; other sounds, music especially, can still come out as text, so put a voice activity detector or a wake word in front of an always-on microphone. 16 kHz WAV files and raw samples need nothing extra; other sample rates need the `[mic]` extra for soxr. `whistle.cact` is fetched from `Cactus-Compute/whistle` on first use.
- `needle.stream(chunks, language=None, keywords=None, weights=None)` - live transcription with no limit on length. `chunks` is an iterable of 16 kHz mono float sample buffers, about a second each; the generator yields one dict per chunk with the `"text"` and `"words"` committed by it (times from the start of the stream), the unconfirmed `"pending"` tail, the `"language"`, the seconds `"received"` and `"pass_ms"`, then one more for the tail when the chunks end. Join the texts with a space. Every chunk re-transcribes the last seconds and commits what two passes agree on.
- `needle.Whistle(weights=None, buffer_size=262144)` - the same model as an object, for `whistle.transcribe(...)`, `whistle.stream(...)`, `whistle.embed(audio)` (the encoder output, one row of floats per 80 ms frame, flattened) or to hold one tuned `.cact`. One model per process, not thread-safe.
@@ -190,14 +190,14 @@ agent.run("...")
Options: `finetune --lora-rank 16 --lora-alpha 32 --lr 1e-4 --batch-size 16 --max-len 1024 --val-split 0.1 --checkpoint <base.safetensors> --checkpoint-dir <dir> --out <adapter.safetensors> --generate <n> --model <id> --workers <n>`; `--model` defaults to `deepseek/deepseek-flash-latest`, `--workers` defaults to `8`, and `--checkpoint-dir` defaults to `checkpoints`. `--generate` synthesizes extra examples through the configured OpenRouter endpoint before training. `build --bits 2|4 --upload` (with `NEEDLE_HF_REPO=<you>/<model>`).
-Whistle fine-tunes the same way: LoRA on the frozen base, merged at export, 4-bit. Data is JSONL, one clip per line, the audio path relative to the file; a `language` outside `en de fr es it nl pl` adds that language:
+Whistle fine-tunes the same way: LoRA on the frozen base, merged at export, 4-bit. Data is three JSONL files, training, validation and test, one clip per line, the audio path relative to the file; a `language` outside `en de fr es it nl pl` adds that language:
```json
{"audio": "clips/0001.wav", "text": "Quero reservar uma mesa para dois.", "language": "pt"}
```
```sh
-needle whistle finetune clips.jsonl --tokens tokens.txt # LoRA; auto-downloads base
+needle whistle finetune train.jsonl validation.jsonl test.jsonl --tokens tokens.txt # LoRA; auto-downloads base
needle whistle build checkpoints/whistle.safetensors --lora checkpoints/whistle_lora.safetensors --out my.cact
```
@@ -207,7 +207,7 @@ Run a tuned model:
needle.transcribe("clip.wav", weights="my.cact")
```
-Options: `whistle finetune --language <code> --tokens <file> --epochs 3 --lora-rank 16 --lora-alpha 32 --lr 1e-4 --batch-size 16 --val-split 0.1 --checkpoint <base.safetensors> --checkpoint-dir <dir> --out <adapter.safetensors>`; `--language` fills lines without one, `--tokens` is a file of words or characters to add to the vocabulary, one per line, and `--checkpoint-dir` defaults to `checkpoints`. Clips over 30 s are skipped, and the run prints the held-out word error rate (character error rate for Chinese, Japanese, Korean and Thai) on the 4-bit weights. `whistle build --layers N --upload` (with `NEEDLE_HF_REPO=<you>/<model>`).
+Options: `whistle finetune --language <code> --tokens <file> --epochs 3 --lora-rank 16 --lora-alpha 32 --lr 1e-4 --batch-size 16 --checkpoint <base.safetensors> --checkpoint-dir <dir> --out <adapter.safetensors>`; `--language` fills lines without one, `--tokens` is a file of words or characters to add to the vocabulary, one per line, and `--checkpoint-dir` defaults to `checkpoints`. Clips over 30 s are skipped. The run prints the validation loss after every epoch and ends with the test set's word error rate (character error rate for Chinese, Japanese, Korean and Thai), a quick greedy decode on the 4-bit weights; for real-world numbers, build the `.cact` and transcribe the test set with it. `whistle build --layers N --upload` (with `NEEDLE_HF_REPO=<you>/<model>`).
## Platform (hosted fine-tuning)
@@ -260,9 +260,9 @@ Adapt by swapping the Literal values for your own and keeping the shapes: closed
- `needle whistle compare [clip.wav] [--weights whistle.cact]` - the same English speech through Whistle, Whisper tiny and base (openai-whisper) and Moonshine tiny v2 (moonshine-voice), one line per model with its weight size, time to first token, decode tokens per second, total time and transcript; `/file` runs a WAV between recordings. Whistle's timing comes from the engine's own JSON. Needs the `[mic,compare]` extras.
- `needle whistle finetune` / `needle whistle build [checkpoint] [--lora adapter] [--layers N] [--platform <folder>] [--out <path>]` - the Whistle fine-tuning pipeline (section above); `build` merges the adapter, slices the `--layers` rung (default 8), exports at 4 bits, and fetches the published `whistle.cact` on every run (without an adapter at full depth it ships that published archive as is). `--platform` also downloads that platform's engine and header and makes `--out` a folder holding them plus `whistle.cact`.
- `needle download whistle [--out <dir>]` - pre-download the `whistle.cact` speech weights. There is one engine for both models, so `needle fetch` and `needle download <platform>` cover the library and the platform folders.
-- `needle fetch [--platform-tag <tag>] [--out <dir>]` - pre-download the inference engine for this machine (or another platform) into the cache; prints the path. For air-gapped devices, also see `NEEDLE3_LIB_PATH` and `HF_HUB_OFFLINE` at https://cactuscompute.com/blog/needle-python-docs.
+- `needle fetch [--platform-tag <tag>] [--out <dir>] [--generation 2|3]` - pre-download the inference engine for this machine (or another platform) into the cache; prints the path. `--generation 2` fetches the Needle 2 engine. For air-gapped devices, also see `NEEDLE3_LIB_PATH` and `HF_HUB_OFFLINE` at https://cactuscompute.com/blog/needle-python-docs.
- `needle platform finetune | generate | jobs | models | files | billing` - the hosted platform from the CLI (section above).
-- `needle download needle3 | needle3.safetensors | model-<id> | <org>/<repo>[/<file>.cact] | <platform> [--out <dir>]` - pull the base Needle 3 archive, a checkpoint to fine-tune, a platform-trained model by id (`--depth N` for one size), a published `.cact` (single-archive repos need only `<org>/<repo>`), or pass a platform folder name (`macos-arm64`, `linux-x86_64`, `wasm`, `wasm-component`, ...) to fetch that platform's engine files into `<out>/<platform>/`. Native runners are marked executable; `wasm-component` contains the WASI Preview 2 component and WIT contract.
+- `needle download needle3 | needle3.safetensors | model-<id> | <org>/<repo>[/<file>.cact] | <platform> [--out <dir>] [--generation 2|3]` - pull the base Needle 3 archive, a checkpoint to fine-tune, a platform-trained model by id (`--depth N` for one size), a published `.cact` (single-archive repos need only `<org>/<repo>`), or pass a platform folder name (`macos-arm64`, `linux-x86_64`, `wasm`, `wasm-component`, ...) to fetch that platform's engine files into `<out>/<platform>/` (`--generation 2` for the Needle 2 engine). Native runners are marked executable; `wasm-component` contains the WASI Preview 2 component and WIT contract.
## Common mistakes to avoid
@@ -272,7 +272,6 @@ Adapt by swapping the Literal values for your own and keeping the shapes: closed
- `import needle` does not import JAX; only `needle finetune`/`build` do.
- `weights=` expects a `.cact` (from `needle build`), not a `.safetensors` checkpoint. `Needle(tools=...)` runs the base Needle 3 archive (fetched once, cached next to the engine); pass `generation=2` for the embedded Needle 2 engine.
- Checkpoints and adapters must be `.safetensors`; pickle (`.pkl`) is not supported and will raise a `ValueError`. The base is `checkpoints/needle3.safetensors` (20 layers); `needle build --layers N` exports any rung from 2 to 20 of it.
-- The engine cannot unload weights: once a tuned `.cact` is bound, constructing or calling a base-model agent raises instead of silently answering with those weights. Construct agents that want the base model before any tuned one, or run them in separate processes.
## Telemetry
diff --git a/needle/cli.py b/needle/cli.py
index e9f5c64..f91dddb 100644
--- a/needle/cli.py
+++ b/needle/cli.py
@@ -286,8 +286,10 @@ def main():
v.add_argument("--weights", type=str, default=None,
help="whistle.cact to load (default: the published weights)")
v = verbs.add_parser("finetune")
- v.add_argument("jsonl_path", type=str,
- help='JSONL of {"audio": WAV path, "text": transcript, "language": code}, one clip per line')
+ v.add_argument("train", type=str,
+ help='Training JSONL of {"audio": WAV path, "text": transcript, "language": code}, one clip per line')
+ v.add_argument("validation", type=str, help="Validation JSONL, scored after every epoch")
+ v.add_argument("test", type=str, help="Test JSONL, scored once after training")
v.add_argument("--language", type=str, default=None,
help="Language of lines that do not name one; a new code adds that language")
v.add_argument("--tokens", type=str, default=None,
@@ -299,10 +301,8 @@ def main():
v.add_argument("--lr", type=float, default=1e-4)
v.add_argument("--lora-rank", type=int, default=16, help="LoRA adapter rank (default: 16)")
v.add_argument("--lora-alpha", type=float, default=32.0, help="LoRA scaling alpha (default: 32)")
- v.add_argument("--val-split", type=float, default=0.1,
- help="Fraction of clips held out for validation (0 disables)")
v.add_argument("--seed", type=int, default=0,
- help="Random seed for LoRA init, validation split, and epoch shuffling")
+ help="Random seed for LoRA init and epoch shuffling")
v.add_argument("--checkpoint-dir", type=str, default="checkpoints")
v.add_argument("--out", type=str, default=None,
help="Output adapter path (.safetensors)")
diff --git a/needle/model/whistle_finetune.py b/needle/model/whistle_finetune.py
index da409e1..3e2a8bf 100644
--- a/needle/model/whistle_finetune.py
+++ b/needle/model/whistle_finetune.py
@@ -101,7 +101,7 @@ def read_audio(path):
import soxr
except ImportError:
raise RuntimeError(f"{path} is {rate} Hz; resampling it to {SAMPLE_RATE} Hz needs soxr: "
- "pip install cactus-needle[mic]") from None
+ "pip install cactus-needle[train]") from None
samples = soxr.resample(samples, rate, SAMPLE_RATE, quality="HQ")
return samples.astype(np.float32)
@@ -199,7 +199,7 @@ def errors(reference, hypothesis, language):
return row[-1], len(ref)
-def _score_quantised(model, params, cfg, tokenizer, lora, scale, rows, held, batch):
+def _score_quantised(model, params, cfg, tokenizer, lora, scale, rows, clips, batch):
import jax
from .quantize import cq_ste_params, WEIGHT_BITS
@@ -210,8 +210,8 @@ def _score_quantised(model, params, cfg, tokenizer, lora, scale, rows, held, bat
method=Whistle.logits))
tags = np.arange(tuned.vocab) >= tuned.text
edits = units = 0
- for start in range(0, len(held), batch):
- part = held[start:start + batch]
+ for start in range(0, len(clips), batch):
+ part = clips[start:start + batch]
enc, frames = encode(merged, *_arrays(part, range(len(part)), tuned)[:2])
tokens = np.full((len(part), tuned.max_tokens), tuned.pad, np.int32)
tokens[:, 0] = tuned.bos
@@ -269,16 +269,23 @@ def finetune_main(args):
params = jax.device_put(params)
print(f" {'backend':<9} {backend} float32")
- clips, skipped = read_clips(args.jsonl_path, args.language)
- if not clips:
- raise SystemExit("no usable clips in " + args.jsonl_path)
- languages = tuple(cfg.languages) + tuple(sorted({lang for *_, lang in clips} - set(cfg.languages)))
+ splits, skipped = [], 0
+ for path in (args.train, args.validation, args.test):
+ clips, dropped = read_clips(path, args.language)
+ if not clips:
+ raise SystemExit("no usable clips in " + path)
+ splits.append(clips)
+ skipped += dropped
+ codes = {lang for clips in splits for *_, lang in clips}
+ languages = tuple(cfg.languages) + tuple(sorted(codes - set(cfg.languages)))
tokens = read_tokens(args.tokens) if args.tokens else []
pieces = token_pieces(WhistleTokenizer().sp, tokens)
tokenizer = WhistleTokenizer(pieces, languages)
tuned = replace(cfg, text=tokenizer.text, languages=languages)
note = f" {skipped} skipped (empty, or over {SECONDS} s)" if skipped else ""
- print(f" {'data':<9} {len(clips)} clips {sum(len(c[0]) for c in clips) / SAMPLE_RATE / 60:.1f} min{note}")
+ minutes = sum(len(c[0]) for c in splits[0]) / SAMPLE_RATE / 60
+ print(f" {'data':<9} {len(splits[0])} train, {len(splits[1])} validation, {len(splits[2])} test clips"
+ f" {minutes:.1f} min to train on{note}")
added = [lang for lang in languages if lang not in cfg.languages]
print(f" {'vocab':<9} {len(tokens)} tokens {len(pieces)} new pieces"
+ (f" new languages {', '.join(added)}" if added else ""))
@@ -286,10 +293,13 @@ def finetune_main(args):
if split:
print(f" {'warning':<9} not one piece: {', '.join(split)}")
- items = [(samples, [cfg.bos, tuned.tag(lang), *tokenizer.encode(text), cfg.eos], text)
- for samples, text, lang in clips]
- long = sum(len(ids) > cfg.max_tokens + 1 for _, ids, _ in items)
- items = [item for item in items if len(item[1]) <= cfg.max_tokens + 1]
+ def encode(clips):
+ return [(samples, [cfg.bos, tuned.tag(lang), *tokenizer.encode(text), cfg.eos], text)
+ for samples, text, lang in clips]
+
+ encoded = [encode(clips) for clips in splits]
+ long = sum(len(ids) > cfg.max_tokens + 1 for clips in encoded for _, ids, _ in clips)
+ items, validation, test = ([item for item in clips if len(item[1]) <= cfg.max_tokens + 1] for clips in encoded)
if long:
print(f" {'skipped':<9} {long} clips with transcripts over {cfg.max_tokens} tokens")
if not items:
@@ -310,13 +320,6 @@ def finetune_main(args):
print(f" {'lora':<9} rank {args.lora_rank} alpha {args.lora_alpha:g} {len(paths)} weight groups"
f" +{len(train['rows'])} vocabulary rows")
- n_val = min(int(len(items) * getattr(args, "val_split", 0.1)), len(items) - 1)
- held = []
- if n_val > 0:
- order = rng.permutation(len(items))
- held, items = [items[i] for i in order[:n_val]], [items[i] for i in order[n_val:]]
- print(f" {'holdout':<9} {n_val} clips for validation")
-
def loss_fn(train, wav, lengths, tokens):
merged = cq_ste_params(tuned_params(params, cfg, tokenizer, train["lora"], scale, train["rows"]), WEIGHT_BITS)
logits = model.apply({"params": merged}, wav, lengths, tokens[:, :-1], quant=True)
@@ -356,20 +359,16 @@ def finetune_main(args):
step_i += 1
if step_i % every == 0:
print(f" {'step':<9} {step_i}/{total_steps} loss {last:.4f}", flush=True)
- if held:
- val = np.mean([float(eval_step(train, *_arrays(held, range(i, min(i + batch, n_val)), cfg)))
- for i in range(0, n_val, batch)])
- print(f" {'epoch':<9} {epoch + 1}/{args.epochs} loss {last:.4f} val {val:.4f}", flush=True)
- else:
- print(f" {'epoch':<9} {epoch + 1}/{args.epochs} loss {last:.4f}", flush=True)
-
- if held and getattr(args, "score", True):
- edits, units = _score_quantised(model, params, cfg, tokenizer, train["lora"], scale, train["rows"],
- held, batch)
- kind = "cer" if all(by_character(text) for *_, text in held) else "wer"
- if units:
- print(f" {kind:<9} {edits / units:.1%} on {len(held)} held-out clips, scored on the "
- f"W{WEIGHT_BITS} weights the archive ships")
+ val = np.mean([float(eval_step(train, *_arrays(validation, range(i, min(i + batch, len(validation))), cfg)))
+ for i in range(0, len(validation), batch)])
+ print(f" {'epoch':<9} {epoch + 1}/{args.epochs} loss {last:.4f} val {val:.4f}", flush=True)
+
+ edits, units = _score_quantised(model, params, cfg, tokenizer, train["lora"], scale, train["rows"], test, batch)
+ kind = "cer" if all(by_character(text) for *_, text in test) else "wer"
+ if units:
+ print(f" {kind:<9} {edits / units:.1%} on {len(test)} test clips, "
+ f"greedy decoding on the W{WEIGHT_BITS} weights")
+ print(f" {'note':<9} for real-world numbers, build the .cact and transcribe the test set with it")
os.makedirs(args.checkpoint_dir, exist_ok=True)
out = args.out or os.path.join(args.checkpoint_dir, "whistle_lora.safetensors")
diff --git a/pyproject.toml b/pyproject.toml
index ea259e7..1df9418 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -18,7 +18,8 @@ train = [
"optax",
"safetensors",
"sentencepiece",
- "whisper-normalizer"
+ "whisper-normalizer",
+ "soxr"
]
gpu = ["jax[cuda12]"]
metal = [
diff --git a/requirements-train.txt b/requirements-train.txt
index cb2e295..16f183a 100644
--- a/requirements-train.txt
+++ b/requirements-train.txt
@@ -9,3 +9,4 @@ optax
safetensors
sentencepiece
whisper-normalizer
+soxr
diff --git a/tests/test_whistle_finetune.py b/tests/test_whistle_finetune.py
index fceb719..e319efe 100644
--- a/tests/test_whistle_finetune.py
+++ b/tests/test_whistle_finetune.py
@@ -96,15 +96,20 @@ def test_finetune_adds_a_language_and_builds_an_archive_with_it(tiny_whistle, pu
from needle.model.export import parse_tokenizer_blob, read_export
from needle.model.whistle_finetune import WhistleTokenizer, build_main, finetune_main, token_pieces
- data = _clips(tmp_path, ["não é coração", "ação não", "não não", "coração"] * 2)
+ files = {}
+ for name, lines in (("train", ["não é coração", "ação não", "não não", "coração"] * 2),
+ ("validation", ["não é coração", "ação"]), ("test", ["coração não", "não"])):
+ (tmp_path / name).mkdir()
+ files[name] = _clips(tmp_path / name, lines)
tokens = tmp_path / "tokens.txt"
tokens.write_text("ação\ncoração\n", encoding="utf-8")
adapter = str(tmp_path / "adapter.safetensors")
finetune_main(types.SimpleNamespace(
- jsonl_path=data, language="pt", tokens=str(tokens), checkpoint=tiny_whistle, epochs=1, batch_size=2, lr=1e-3,
- lora_rank=4, lora_alpha=8.0, val_split=0.25, seed=0, checkpoint_dir=str(tmp_path), out=adapter))
+ **files, language="pt", tokens=str(tokens), checkpoint=tiny_whistle, epochs=1, batch_size=2, lr=1e-3,
+ lora_rank=4, lora_alpha=8.0, seed=0, checkpoint_dir=str(tmp_path), out=adapter))
printed = capsys.readouterr().out
- assert "CQ W4 STE + A8 (matches export)" in printed and "held-out clips, scored on the W4 weights" in printed
+ assert "8 train, 2 validation, 2 test clips" in printed and "val " in printed
+ assert "CQ W4 STE + A8 (matches export)" in printed and "on 2 test clips, greedy decoding on the W4 weights" in printed
saved = read_adapter(adapter)
pieces = token_pieces(WhistleTokenizer().sp, ["ação", "coração"])
assert saved["languages"][-1] == "pt" and saved["pieces"] == pieces
@@ -237,14 +242,14 @@ def test_cli_routes_finetune_and_build_and_takes_a_tuned_language(monkeypatch):
monkeypatch.setattr(tune, "finetune_main", lambda args: seen.append(("finetune", args)))
monkeypatch.setattr(tune, "build_main", lambda args: seen.append(("build", args)))
monkeypatch.setattr(needle.agent.whistle, "playground", lambda args: seen.append(("playground", args)))
- for argv in (["finetune", "clips.jsonl", "--language", "pt", "--tokens", "tokens.txt"],
+ for argv in (["finetune", "train.jsonl", "dev.jsonl", "test.jsonl", "--language", "pt", "--tokens", "tokens.txt"],
["build", "--lora", "a.safetensors", "--layers", "4", "--out", "t.cact"],
["playground", "--language", "pt", "--weights", "t.cact"]):
monkeypatch.setattr(sys, "argv", ["needle", "whistle", *argv])
needle.cli.main()
(_, tuned), (_, built), (_, played) = seen
- assert (tuned.jsonl_path, tuned.language, tuned.tokens, tuned.batch_size, tuned.lr) == (
- "clips.jsonl", "pt", "tokens.txt", 16, 1e-4)
+ assert (tuned.train, tuned.validation, tuned.test, tuned.language, tuned.tokens, tuned.batch_size, tuned.lr) == (
+ "train.jsonl", "dev.jsonl", "test.jsonl", "pt", "tokens.txt", 16, 1e-4)
assert (built.lora, built.layers, built.out, built.platform, built.upload) == ("a.safetensors", 4, "t.cact", None, False)
assert played.language == "pt"