Skip to content

Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.

Refactor quantization and model loading logic; update tests for new checkpoint handling

HenryNdubuaku26b69669c393
diff --git a/README.md b/README.md
index 58ee290..e93064f 100644
--- a/README.md
+++ b/README.md
@@ -1,24 +1,23 @@
![Needle](assets/banner.png)
-# Needle 2
+# Needle 3
-Needle 2 is an open 45M-parameter model for tool calling, device use and structured extraction. The whole model is a single 14MB binary that runs a full session in about 28MB of RAM. It is built on our Simple Attention Network findings, compressed to CQ2-bit with Cactus Quants, and baked into its own engine. On the benchmarks below, Needle 2 trades wins with other small models like FunctionGemma 270M, LFM2.5 230M and Apple FM, at 5x to 70x smaller, and 2 bits against their f16.
+Needle 3 is a 25-121M parameter foundation model for tool calling and structured extraction on tiny devices. It is a Laddered Simple Attention Network: any depth from 2 to 20 layers is a complete model, compressed to CQ2-bit with Cactus Quants into one 9-35 MB weights file that a sub-1 MB engine loads at start.
-This repository is the Python package: inference, LoRA fine-tuning, and export. `pip install cactus-needle`, describe your tools, and call them from Python. The inference engine is fetched once from Hugging Face and cached; there is nothing else to build, and offline setup for air gapped devices is covered in [doc/apis.md](doc/apis.md).
+This repository is the Python package: inference, LoRA fine-tuning, and export. `pip install cactus-needle`, describe your tools, and call them from Python. The engine and the weights are fetched once from Hugging Face and cached; there is nothing else to build.
-- **Self-contained**: weights baked into a single 14MB engine; no separate model files to manage, and inference does no network.
+- **Self-contained**: a sub-1 MB engine per platform plus one `needle3.cact` weights file; inference does no network.
- **Simple contract**: tool calls come back as structured data, text in, JSON out; a byte-level grammar compiled from your schemas constrains every token.
- **Confidence-gated**: every response carries a calibrated confidence score from a learned head; set a threshold, act above it, escalate below it.
-- **Tool retrieval**: declare a large catalogue and a built-in retrieval head renders only the top five tools per turn, with the grammar constrained to that subset.
-- **Bounded memory**: a 256-token sliding window with the tools pinned as KV sinks, so total memory stays near 28MB no matter how long the conversation runs.
+- **Bounded memory**: a 256-token sliding window with the tools pinned as KV sinks, so memory stays flat no matter how long the conversation runs.
-Weights: [huggingface.co/Cactus-Compute/needle2](https://huggingface.co/Cactus-Compute/needle2) · source: [github.com/cactus-compute/needle](https://github.com/cactus-compute/needle).
+Weights: [huggingface.co/Cactus-Compute/needle3](https://huggingface.co/Cactus-Compute/needle3) · source: [github.com/cactus-compute/needle](https://github.com/cactus-compute/needle).
![Size-quality frontier: mobile-class and below](assets/frontier.png)
-## Simple Attention Network
+## Laddered Simple Attention Network
-Needle 2 is a Simple Attention Network, our dense small-model recipe: a Hadamard MLP in place of the FFN, GQA attention, engram key-value memory, and multi-lane hyper-connections. See the paper for the design and ablations: [arXiv:2607.18363](https://arxiv.org/abs/2607.18363).
+Needle 3 is a Laddered Simple Attention Network, our small-model recipe: a Monarch Hadamard MLP in place of the FFN, GQA attention with causal conv taps, engram n-gram memory read by gather, and multi-lane hyper-connections, trained so that every depth from 2 to 20 layers is a deployable model. See the paper for the design and ablations: [arXiv:2607.18363](https://arxiv.org/abs/2607.18363).
![Simple Attention Network architecture](assets/architecture.png)
@@ -121,7 +120,7 @@ needle finetune data.jsonl --epochs 10
needle finetune data.jsonl --epochs 10 --generate 300 --lora-rank 16 --lora-alpha 32
```
-Key options: `--epochs` (default 3), `--layers <n>` (fine-tune the n-layer rung of the base, see below), `--lora-rank` (16), `--lora-alpha` (32), `--lr` (1e-4), `--batch-size` (16), `--max-len` (1024), `--val-split` (0.1), `--checkpoint <base.safetensors or .pkl>`, `--checkpoint-dir <dir>` (default `checkpoints`), `--out <adapter.safetensors or .pkl>`, `--generate <n>`, `--model <id>` (default `deepseek/deepseek-v4-flash`), and `--workers <n>` (default 8). `--generate` uses the configured OpenRouter endpoint to synthesize extra examples before training. The adapter is written to `checkpoints/needle_lora.pkl` by default. A validation loss prints each epoch from the held out split.
+Key options: `--epochs` (default 3), `--lora-rank` (16), `--lora-alpha` (32), `--lr` (1e-4), `--batch-size` (16), `--max-len` (1024), `--val-split` (0.1), `--checkpoint <base.safetensors or .pkl>`, `--checkpoint-dir <dir>` (default `checkpoints`), `--out <adapter.safetensors or .pkl>`, `--generate <n>`, `--model <id>` (default `deepseek/deepseek-v4-flash`), and `--workers <n>` (default 8). `--generate` uses the configured OpenRouter endpoint to synthesize extra examples before training. The adapter is written to `checkpoints/needle_lora.pkl` by default. A validation loss prints each epoch from the held out split.
Training is plain JAX and runs on any accelerator jax supports. On an NVIDIA machine install the CUDA build and the same command trains on the GPU:
@@ -135,30 +134,26 @@ On Apple Silicon the `metal` extra trains on the GPU:
pip install "cactus-needle[train,metal]"
```
-The base is Needle 3, a depth ladder: every rung from 2 layers up to the full stack is a trained model whose blocks are a nested subset of the full one. `--layers n` slices the base to its n-layer rung before training, trains that rung at full depth, and records the depth in the adapter, so `needle build` exports the same rung. A smaller rung trains and runs faster and fits a smaller device at some accuracy cost; without `--layers` the full base is fine-tuned.
+Fine-tuning trains LoRA through 4-bit quantisation-aware numerics at the full 20 layers, so the adapter matches the archive `needle build` writes.
-```sh
-needle finetune data.jsonl --epochs 10 --layers 8
-```
-
-**3. Build a tuned `.cact`.** Merge the adapter into the base and quantize. The checkpoint is optional: `needle build` uses the base the adapter was trained on, and auto-downloads the Needle 3 base if absent.
+**3. Build a tuned `.cact`.** Merge the adapter into the base and quantise to 4 bits. The checkpoint is optional: `needle build` uses the base the adapter was trained on, and auto-downloads the Needle 3 base if absent. Every build fetches the published `needle3.cact` and takes its tokenizer from it.
```sh
needle build --lora checkpoints/needle_lora.safetensors --out my_needle.cact
-needle build checkpoints/needle3_enterprise.safetensors --lora adapter.safetensors --out my_needle.cact # 20L base
-needle build --layers 4 --out needle3_4l.cact # untuned 4-layer rung
+needle build --lora checkpoints/needle_lora.safetensors --layers 8 --out my_needle_8l.cact # 8-layer rung of the tuned base
```
-The engine is weights-agnostic and never rebuilt: one engine library per platform (under 1MB) runs any archive, and the archive shrinks with the rung (about 9MB at 2 layers, 29MB at 16). `needle download needle3` fetches the base 16-layer archive by itself, `needle download needle3.safetensors` (or `needle3_enterprise.safetensors`) the checkpoint to fine-tune.
+The engine is weights-agnostic and never rebuilt: one engine library per platform (under 1MB) runs any archive. `needle download needle3` fetches the base 20-layer archive by itself, `needle download needle3.safetensors` the checkpoint to fine-tune. `needle build --layers n` exports any rung from 2 to 20 layers of the tuned base.
-Add `--bits 2` for a smaller model (by default the export follows the checkpoint's declared per-layer bit map, falling back to 4 when the checkpoint declares none), or set `NEEDLE_HF_REPO=<you>/<model>` and pass `--upload` to publish the `.cact`. The counterpart `needle download <you>/<model>/my_needle.cact` pulls a published archive on any machine, and `needle download <platform>` (e.g. `macos-arm64`) fetches that platform's engine runner.
+Set `NEEDLE_HF_REPO=<you>/<model>` and pass `--upload` to publish the `.cact`. The counterpart `needle download <you>/<model>/my_needle.cact` pulls a published archive on any machine, and `needle download <platform>` (e.g. `macos-arm64`) fetches that platform's engine runner.
**4. Run it.** The engine is weights-agnostic, so a tuned `.cact` runs on it directly - no recompilation:
```python
import needle
agent = needle.Needle(weights="my_needle.cact", tools=[...])
-agent = needle.Needle(tools=[...], generation=3) # the base Needle 3 archive, fetched once and cached
+agent = needle.Needle(tools=[...]) # the base Needle 3 archive, fetched once and cached
+agent = needle.Needle(tools=[...], generation=2) # Needle 2, for existing deployments
agent.run("...")
```
@@ -168,11 +163,11 @@ Cactus Compute collects strictly anonymous usage telemetry (function name, packa
## Citation
-Needle 2 is built by the Cactus Compute team. If you use it in your work, please cite:
+Needle 3 is built by the Cactus Compute team. If you use it in your work, please cite:
```bibtex
-@misc{needle2_2026,
- title = {Needle 2: A 45M-Parameter Foundation Tool-Calling Model for Tiny Devices},
+@misc{needle3_2026,
+ title = {Needle 3: A Foundation Tool-Calling Model for Tiny Devices},
author = {Ndubuaku, Henry and Mosoyan, Karen and Mroz, Jakub and Cylich, Noah and
Kumar, Satyajit and Sandhu, Parkirat and Shemet, Roman and Lee, Justin H.},
year = {2026},
diff --git a/doc/apis.md b/doc/apis.md
index ad45130..dfe2a07 100644
--- a/doc/apis.md
+++ b/doc/apis.md
@@ -3,28 +3,14 @@
| Call | What it does |
| --- | --- |
| `needle.Needle(tools=None, system=None, weights=None, tool_index_path=None, buffer_size=65536)` | Create an agent bound to one toolset. `tools` takes decorated functions, Pydantic models, raw JSON schema dicts, or a JSON string. `system` carries environment facts. `weights` loads a tuned `.cact`. `tool_index_path` persists tool embeddings for large catalogues. |
-| `agent.complete(text="", max_new_tokens=512, audio=None, audio_format="wav", sample_rate=0, channels=1)` | Complete a text, speech, or mixed turn. `audio` accepts WAV bytes or a path; PCM16 and float32 buffers use the accompanying format, rate, and channel fields. |
-| `agent.embed(text="", audio=None, audio_format="wav", sample_rate=0, channels=1)` | Embed text, serialized tool schemas, audio, or a mixed input with the Needle 3 retrieval head. |
-| `agent.run(query="", max_steps=8, max_new_tokens=512, audio=None, audio_format="wav", sample_rate=0, channels=1, strict=True)` | Full agentic loop: the model picks calls, Needle executes your Python functions and feeds results back, and the final response carries the executed tool results as `results`. |
+| `agent.complete(text="", max_new_tokens=512)` | One turn. You execute the call and feed the result back yourself via the next `complete(...)`. |
+| `agent.embed(text="")` | Embed text or serialized tool schemas with the Needle 3 retrieval head. |
+| `agent.run(query="", max_steps=8, max_new_tokens=512, strict=True)` | Full agentic loop: the model picks calls, Needle executes your Python functions and feeds results back, and the final response carries the executed tool results as `results`. |
| `agent.reset()` | Rewind the conversation, keep the tools loaded. |
| `needle.tool` | Decorator that turns a function into a tool schema (attached as `fn._needle_tool`). |
| `needle.Field(...)` | Per argument constraints, attached inline with `typing.Annotated` or passed as a default. |
| `needle.extract(text, schema, system=None, max_new_tokens=512, weights=None, strict=True)` | One shot extraction. Returns a Pydantic instance if `schema` is a model, else a dict, or `None` if nothing matched. Strict mode rejects temporal values that contradict literal source years and engine-reported ungrounded values. |
-Audio tokenization is internal. Pass a WAV path or WAV bytes directly:
-
-```python
-agent.complete(audio="command.wav")
-agent.embed(audio=wav_bytes)
-```
-
-Microphone PCM can be passed without wrapping it in a WAV container:
-
-```python
-agent.complete(audio=pcm16_bytes, audio_format="pcm16",
- sample_rate=48_000, channels=2)
-```
-
When one tuned archive is active, `extract(..., weights=None)` inherits it for backwards compatibility. If both a Needle 2 and Needle 3 tune are active, pass `weights=` explicitly; the client refuses to guess between generations.
## Declaring tools
@@ -119,10 +105,9 @@ Needle solves every problem as a function call. The context declares what may be
- A request no declared tool can serve is refused with the empty call `[]`. That is the whole contract for off-topic input; there is no free-text fallback.
- Arguments contain only values evidenced by the input. An optional field with no evidence is omitted, not guessed; omission is the field-level `[]`.
-- An escalated call is withheld by the engine: `function_calls` is empty and `suppressed_calls` carries it. The agent prefixes the local `date:` fact to `system` automatically (`auto_date=False` opts out) so relative dates resolve.
+- A call below the confidence threshold, or contradicted by the request, is withheld by the engine: `function_calls` is empty and `suppressed_calls` carries it. The agent prefixes the local `date:` fact to `system` automatically (`auto_date=False` opts out) so relative dates resolve.
- A date argument whose year matches none of the years written in the conversation so far or in the `system` facts is reported in `validation.ungrounded` as `tool.field`, alongside anything the engine itself flags as ungrounded. `run()` does not execute such a call; its result is `{"error": "ungrounded field"}` and the model continues from it. Pass `strict=False` to execute anyway.
-- Speech: `complete(text, audio=clip)` transcribes the clip in a clean context, then answers a text turn that carries the transcript wrapped in `<context>` followed by `text` (or "Fulfill the spoken request." when `text` is empty); the speech codes are not part of that turn and the transcript is returned as `transcript` with `transcript_confidence` (geometric mean token probability of the transcription decode).
-- `extract()` declares the record as the only tool; a turn with a single declared tool is a record extraction, so an escalated record is returned with the `escalate` flag rather than withheld, and `strict` handles grounding.
+- `extract()` declares the record as the only tool and takes the record from `function_calls` or, when the engine withheld it, from `suppressed_calls`; `strict` handles grounding.
- Arguments arrive after the engine's deterministic repair step, which grounds them in the request text (first/last name split, phone completion, verbatim quoted titles, place queries with their street address, weekday and explicit dates resolved against the `date:` fact, empty optionals and duplicate calls dropped). Enumerated parameters are never altered.
- `reasoning` is the model's short derivation of each argument from its source span (`'ten minutes' -> minutes 10`). It is generated unconstrained; only the call itself is grammar-constrained, so the JSON cannot be malformed while the derivation stays legible.
- After you execute a call, pass the result back as the next `complete()`. The model continues from it, and later arguments may depend on earlier results: `search_for_contact` first, then `send_instant_message` with the returned `contact_id`. A final `"type": "respond"` with empty `function_calls` signals the loop is done; the answer is the tool results themselves, which `run()` collects on the final response as `results`. No free text is generated.
@@ -172,7 +157,7 @@ Five or fewer declared tools render directly. Above that, retrieval engages: at
## Confidence
-The `confidence` field is the minimum of two signals: a calibrated post-hoc head that scores the full prompt plus the call the model just produced, and the decoding probability of the call tokens. A call is accepted only when both agree, so the failure mode is escalation, not wrong execution. The contract: pick a threshold for your product, act at or above it, re-ask or route to a bigger model below it. Off-topic requests return the empty call `[]`. Every response also carries `escalate`, true when `confidence` is below the engine's default threshold of 0.1.
+The `confidence` field is the minimum of two signals: a calibrated post-hoc head that scores the full prompt plus the call the model just produced, and the decoding probability of the call tokens. A call is accepted only when both agree, so the failure mode is escalation, not wrong execution. The contract: pick a threshold for your product, act at or above it, re-ask or route to a bigger model below it. Off-topic requests return the empty call `[]`. A call below the engine's default threshold of 0.1 is withheld into `suppressed_calls`.
Calibration holds for the base model only. Fine-tuning does not update the head, so an agent running tuned weights reports `confidence` as `None` and warns once at construction.
diff --git a/doc/finetuning.md b/doc/finetuning.md
index d18e70d..423a495 100644
--- a/doc/finetuning.md
+++ b/doc/finetuning.md
@@ -33,26 +33,27 @@ needle finetune data.jsonl --epochs 10 --out adapter.safetensors
needle build checkpoints/needle3.safetensors --lora adapter.safetensors --out tuned.cact
```
-Checkpoints and adapters are `.safetensors`; pickle files still load. The base is `checkpoints/needle3.safetensors` (the 16-layer Needle 3); `needle3_enterprise.safetensors` is the 20-layer parent it was sliced from and fine-tunes the same way.
+Checkpoints and adapters are `.safetensors`; pickle files still load. The base is `checkpoints/needle3.safetensors`, the 20-layer Needle 3.
+
+```python
+agent = needle.Needle(tools=[...], weights="tuned.cact")
+```
+
+Fine-tuning is quantisation-aware: LoRA trains through the 4-bit CQ numerics
+`needle build` exports, at the full 20 layers of the base. Every build fetches the
+published `needle3.cact` and packages its tokenizer.
## Depth
-Needle 3 is a depth ladder: every rung from 2 layers to the full stack was trained as a model, and a rung's blocks are a nested subset of the full stack. `--layers n` slices the base to its n-layer rung before training and fine-tunes that rung at its full depth, the adapter records the depth, and `needle build` exports the same rung (passing a different `--layers` to build is refused). Pick a rung by device budget: fewer layers is proportionally faster and smaller, at some accuracy cost. `needle build --layers n` without an adapter exports an untuned rung.
+Needle 3 is a depth ladder: every rung from 2 to 20 layers is a trained model whose
+blocks are a nested subset of the full stack. `needle build --layers n` merges the
+adapter into the full base and exports the n-layer rung, so one fine-tune serves
+every device budget.
```sh
-needle finetune data.jsonl --epochs 10 --layers 8 --out adapter.safetensors
-needle build checkpoints/needle3.safetensors --lora adapter.safetensors --out tuned_8l.cact
+needle build checkpoints/needle3.safetensors --lora adapter.safetensors --layers 8 --out tuned_8l.cact
```
-```python
-agent = needle.Needle(tools=[...], weights="tuned.cact")
-```
-
-Fine-tuning is quantization-aware by default. `--qat-bits auto` trains through
-the checkpoint's declared CQ export scheme and records that scheme in the
-adapter; `needle build` rejects a conflicting bit-width override. Use
-`--qat-bits none` only when intentionally targeting float evaluation.
-
To share a tuned model, set `NEEDLE_HF_REPO=<you>/<model>` and pass `--upload` to `needle build`; on any other machine `needle download <you>/<model>/tuned.cact` pulls it back down (or pass just `<you>/<model>` when the repo holds a single archive).
Defaults: batch size 16, learning rate 0.0001 with warmup and cosine decay, gradient clipping at norm 1, rank 16, alpha 32, max length 1024, validation split 0.1, random seed 0. Use `--seed` to reproduce or intentionally vary LoRA initialization, validation selection, and epoch shuffling. The base checkpoint downloads from Hugging Face on first run.
@@ -99,8 +100,6 @@ The confidence head. Scores are calibrated for the base model on its training mi
The tokenizer. Non English text fragments into roughly 1.7 times more tokens (measured on Spanish), which taxes both quality and the 256 token window.
-Audio input. A tuned `.cact` built by this package ships the text vocabulary and no speech tokenizer, so it answers text only; the base archive keeps audio.
-
## Troubleshooting
1. Loss goes NaN within the first steps on CPU: fixed, run `pip install --upgrade cactus-needle`.
diff --git a/llms.txt b/llms.txt
index c6ff3f1..4a7ac69 100644
--- a/llms.txt
+++ b/llms.txt
@@ -1,6 +1,6 @@
-# Needle 2 (cactus-needle)
+# Needle 3 (cactus-needle)
-> Needle 2 is an open 45M-parameter on-device model for tool calling, device control, and structured extraction. This Python package (`cactus-needle`) is inference + LoRA fine-tuning + export. Text goes in, a JSON tool call comes back; a byte-level grammar compiled from your schemas constrains every token, so the call is always well-formed. The inference engine is a 14MB binary fetched once from Hugging Face and cached; a full session runs in ~28MB RAM.
+> Needle 3 is a 25-121M parameter on-device model for tool calling, device control, and structured extraction. This Python package (`cactus-needle`) is inference + LoRA fine-tuning + export. Text goes in, a JSON tool call comes back; a byte-level grammar compiled from your schemas constrains every token, so the call is always well-formed. A sub-1 MB engine and the 35 MB `needle3.cact` weights are fetched once from Hugging Face and cached; nothing is compiled locally.
This file is written for AI coding assistants. It is enough to write correct Needle code without reading the source. Copy the patterns; do not invent API that is not listed here.
@@ -103,11 +103,10 @@ Every turn returns one dict:
- `function_calls` is a list of `{"name", "arguments"}`. Read `arguments` directly; it is grammar-guaranteed to match your schema.
- `reasoning` is a short unconstrained derivation of each argument from its source span.
- `confidence` is a calibrated score in [0,1] (see Confidence gating).
-- `suppressed_calls` holds a call the engine withheld because `escalate` is true (confidence below 0.1 or a grounding gate fired); `function_calls` is then empty. Show it to the user to confirm, or treat the turn as a refusal.
+- `suppressed_calls` holds a call the engine withheld (confidence below 0.1 or a grounding gate fired); `function_calls` is then empty. Show it to the user to confirm, or treat the turn as a refusal.
- `Needle(system=...)` prefixes the local `date:` fact automatically unless the text already carries a date; pass `auto_date=False` to opt out.
- `validation.ungrounded`, when present, lists `tool.field` names whose value is not grounded in the input, including date arguments whose year matches nothing in the conversation or the `system` facts. `run()` refuses those calls with `{"error": "ungrounded field"}` unless `strict=False`.
-- Speech: `complete(text, audio=clip)` transcribes the clip in a clean context, then answers a text turn that carries the transcript wrapped in `<context>` followed by `text` (or "Fulfill the spoken request." when `text` is empty); the speech codes are not part of that turn and the transcript is returned as `transcript` with `transcript_confidence` (geometric mean token probability of the transcription decode).
-- `extract()` declares the record as the only tool; a turn with a single declared tool is a record extraction, so an escalated record is returned with the `escalate` flag rather than withheld, and `strict` handles grounding.
+- `extract()` declares the record as the only tool and takes the record from `function_calls` or, when the engine withheld it, from `suppressed_calls`; `strict` handles grounding.
- Arguments arrive after the engine's deterministic repair step, which grounds them in the request text: a full name in one of a first/last pair is split, a truncated phone number is completed, quoted titles are restored verbatim, place queries recover dropped words and a following street address, weekday and `March 4th` style dates resolve against the `date:` fact, empty optional strings and duplicate calls are dropped. Enumerated parameters are never altered.
## Behaviour contract (important for correct code)
@@ -129,7 +128,7 @@ if r["type"] == "call":
## Confidence gating
-`confidence` is the min of a post-hoc calibration head and the decode probability of the call. Calibration holds for the base model only: an agent constructed with `weights=` reports `confidence` as None (fine-tuning does not update the head). Pick a threshold per product; act at/above it, escalate below it (`escalate` in the response applies the engine default of 0.1):
+`confidence` is the min of a post-hoc calibration head and the decode probability of the call. Calibration holds for the base model only: an agent constructed with `weights=` reports `confidence` as None (fine-tuning does not update the head). Pick a threshold per product; act at/above it, escalate below it:
```python
r = agent.complete(user_text)
@@ -137,7 +136,7 @@ calls = r.get("function_calls") or []
if calls and r["confidence"] >= 0.8:
execute(calls[0])
else:
- escalate_or_reask()
+ confirm_or_reask()
```
## System facts (optional)
@@ -181,7 +180,7 @@ agent = needle.Needle(weights="my.cact", tools=[...])
agent.run("...")
```
-Options: `finetune --layers <n> (fine-tune the n-layer rung of the depth ladder; build exports the same rung) --lora-rank 16 --lora-alpha 32 --lr 1e-4 --batch-size 16 --max-len 1024 --val-split 0.1 --checkpoint <base.safetensors or .pkl> --checkpoint-dir <dir> --out <adapter.safetensors or .pkl> --generate <n> --model <id> --workers <n>`; `--model` defaults to `deepseek/deepseek-v4-flash`, `--workers` defaults to `8`, and `--checkpoint-dir` defaults to `checkpoints`. `--generate` synthesizes extra examples through the configured OpenRouter endpoint before training. `build --bits 2|4 --upload` (with `NEEDLE_HF_REPO=<you>/<model>`).
+Options: `finetune --lora-rank 16 --lora-alpha 32 --lr 1e-4 --batch-size 16 --max-len 1024 --val-split 0.1 --checkpoint <base.safetensors or .pkl> --checkpoint-dir <dir> --out <adapter.safetensors or .pkl> --generate <n> --model <id> --workers <n>`; `--model` defaults to `deepseek/deepseek-v4-flash`, `--workers` defaults to `8`, and `--checkpoint-dir` defaults to `checkpoints`. `--generate` synthesizes extra examples through the configured OpenRouter endpoint before training. `build --bits 2|4 --upload` (with `NEEDLE_HF_REPO=<you>/<model>`).
## Playground (browser)
@@ -206,10 +205,10 @@ Adapt by swapping the Literal values for your own and keeping the shapes: closed
## CLI summary
- `needle run --checkpoint <base.safetensors or .pkl> --query "..." --tools tools.json` - JAX reference inference from a checkpoint (dev path; normal inference is the Python `Needle` API above).
-- `needle generate-data` / `needle finetune [--layers N]` / `needle build [checkpoint] [--lora adapter] [--layers N]` - the fine-tuning pipeline; `build` defaults to the adapter's base (auto-downloaded Needle 3 base otherwise) and exports only the chosen rung.
+- `needle generate-data` / `needle finetune` / `needle build [checkpoint] [--lora adapter] [--layers N]` - the fine-tuning pipeline; fine-tuning runs at the full 20 layers, `build` merges the adapter, slices the `--layers` rung (default 20), exports at 4 bits, and fetches the published `needle3.cact` for its tokenizer on every run.
- `needle playground` - browser UI.
- `needle fetch [--platform-tag <tag>] [--out <dir>]` - pre-download the inference engine for this machine (or another platform) into the cache; prints the path. For air-gapped devices, also see `NEEDLE_LIB_PATH` and `HF_HUB_OFFLINE` in doc/apis.md.
-- `needle download needle3 | needle3.safetensors | needle3_enterprise.safetensors | <org>/<repo>[/<file>.cact] | <platform> [--out <dir>]` - pull the base Needle 3 archive, a checkpoint to fine-tune, a published `.cact` (single-archive repos need only `<org>/<repo>`), or pass a platform folder name (`macos-arm64`, `linux-x86_64`, `wasm`, `wasm-component`, ...) to fetch that platform's engine files into `<out>/<platform>/`. Native runners are marked executable; `wasm-component` contains the WASI Preview 2 component and WIT contract.
+- `needle download needle3 | needle3.safetensors | <org>/<repo>[/<file>.cact] | <platform> [--out <dir>]` - pull the base Needle 3 archive, a checkpoint to fine-tune, a published `.cact` (single-archive repos need only `<org>/<repo>`), or pass a platform folder name (`macos-arm64`, `linux-x86_64`, `wasm`, `wasm-component`, ...) to fetch that platform's engine files into `<out>/<platform>/`. Native runners are marked executable; `wasm-component` contains the WASI Preview 2 component and WIT contract.
## Common mistakes to avoid
@@ -217,8 +216,8 @@ Adapt by swapping the Literal values for your own and keeping the shapes: closed
- Do not read `arguments` keys that were not evidenced in the input; optional fields may be absent.
- Do not create one `Needle` per turn for a conversation; reuse the instance so context carries. New tools = new instance.
- `import needle` does not import JAX; only `needle finetune`/`build` do.
-- `weights=` expects a `.cact` (from `needle build`), not a `.safetensors` or `.pkl` checkpoint. `Needle(tools=..., generation=3)` runs the base Needle 3 archive (fetched once, cached next to the engine); the default without arguments stays the embedded Needle 2.
-- Checkpoints and adapters are `.safetensors`; pickle checkpoints still load. The base is `checkpoints/needle3.safetensors` (16 layers); `needle3_enterprise.safetensors` is the 20-layer parent.
+- `weights=` expects a `.cact` (from `needle build`), not a `.safetensors` or `.pkl` checkpoint. `Needle(tools=...)` runs the base Needle 3 archive (fetched once, cached next to the engine); pass `generation=2` for the embedded Needle 2 engine.
+- Checkpoints and adapters are `.safetensors`; pickle checkpoints still load. The base is `checkpoints/needle3.safetensors` (20 layers); `needle build --layers N` exports any rung from 2 to 20 of it.
- The engine cannot unload weights: once a tuned `.cact` is bound, constructing or calling a base-model agent raises instead of silently answering with those weights. Construct agents that want the base model before any tuned one, or run them in separate processes.
## Telemetry
diff --git a/needle/__init__.py b/needle/__init__.py
index ac16e03..2b8555c 100644
--- a/needle/__init__.py
+++ b/needle/__init__.py
@@ -6,7 +6,6 @@ import datetime
import json
import os
import re
-import sys
import warnings
from .agent.tools import Field, build_schema, pydantic_schema, tool, _is_pydantic_model
@@ -85,54 +84,19 @@ def _base_weights_path(generation):
_lib_handles = {}
_active = {}
+_loaded_base = {}
-class _NeedleAudio(ctypes.Structure):
- _fields_ = [
- ("data", ctypes.c_void_p),
- ("size", ctypes.c_uint64),
- ("sample_rate", ctypes.c_int),
- ("channels", ctypes.c_int),
- ("format", ctypes.c_int),
- ]
-
-
-_AUDIO_FORMATS = {"wav": 1, "pcm16": 2, "float32": 3}
-
-
-def _prepare_audio(audio, audio_format="wav", sample_rate=0, channels=1):
- if audio is None:
- return None
- if isinstance(audio, (str, os.PathLike)):
- with open(os.fspath(audio), "rb") as handle:
- data = handle.read()
- else:
- try:
- data = bytes(audio)
- except (TypeError, ValueError) as exc:
- raise TypeError("audio must be a path or bytes-like object") from exc
- if not data:
- raise ValueError("audio is empty")
- try:
- format_code = _AUDIO_FORMATS[str(audio_format).lower()]
- except KeyError as exc:
- raise ValueError("audio_format must be 'wav', 'pcm16', or 'float32'") from exc
- if format_code != _AUDIO_FORMATS["wav"] and (
- int(sample_rate) <= 0 or int(channels) <= 0):
- raise ValueError("PCM audio requires positive sample_rate and channels")
- return {"data": data, "format": format_code,
- "sample_rate": int(sample_rate), "channels": int(channels)}
-
-
-def _native_audio(payload):
- if payload is None:
- return None, None
- data = payload["data"]
- storage = ctypes.create_string_buffer(data, len(data))
- descriptor = _NeedleAudio(
- ctypes.cast(storage, ctypes.c_void_p), len(data),
- payload["sample_rate"], payload["channels"], payload["format"])
- return ctypes.byref(descriptor), (storage, descriptor)
+def _load_base(lib, generation):
+ if generation < 3 or _loaded_base.get(generation):
+ return
+ path = _base_weights_path(generation)
+ with open(path, "rb") as handle:
+ data = handle.read()
+ if lib.needle_load(data, len(data)) < 0:
+ raise RuntimeError(f"needle_load failed for {path}")
+ _loaded_base[generation] = path
+ _active.pop(generation, None)
def _lib(generation=2):
@@ -141,18 +105,13 @@ def _lib(generation=2):
lib = ctypes.CDLL(_library_path(generation))
lib.needle_init.argtypes = [ctypes.c_char_p, ctypes.c_char_p, ctypes.c_char_p]
lib.needle_init.restype = ctypes.c_int
+ lib.needle_complete.argtypes = [
+ ctypes.c_char_p, ctypes.c_int, ctypes.c_char_p, ctypes.c_int]
+ lib.needle_complete.restype = ctypes.c_int
if generation >= 3:
- lib.needle_complete.argtypes = [
- ctypes.c_char_p, ctypes.POINTER(_NeedleAudio), ctypes.c_int,
- ctypes.c_char_p, ctypes.c_int]
lib.needle_embed.argtypes = [
- ctypes.c_char_p, ctypes.POINTER(_NeedleAudio),
- ctypes.POINTER(ctypes.c_float), ctypes.c_int]
+ ctypes.c_char_p, ctypes.POINTER(ctypes.c_float), ctypes.c_int]
lib.needle_embed.restype = ctypes.c_int
- else:
- lib.needle_complete.argtypes = [
- ctypes.c_char_p, ctypes.c_int, ctypes.c_char_p, ctypes.c_int]
- lib.needle_complete.restype = ctypes.c_int
lib.needle_reset.argtypes = []
lib.needle_reset.restype = None
lib.needle_load.argtypes = [ctypes.c_char_p, ctypes.c_uint64]
@@ -170,9 +129,7 @@ class Needle:
if self._tuned:
self._generation = _weight_generation(self._weights)
else:
- self._generation = int(generation or 2)
- if self._generation >= 3:
- self._weights = _base_weights_path(self._generation)
+ self._generation = int(generation or 3)
self._worker = None
self._closed = False
if self._tuned:
@@ -209,6 +166,7 @@ class Needle:
return
generation = self._generation
lib = _lib(generation)
+ _load_base(lib, generation)
if _active.get(generation) is self:
return
if lib.needle_init(self._system, self._tools_json, self._tool_index_path) < 0:
@@ -236,32 +194,21 @@ class Needle:
return {"n_tools": self._n_tools, "tuned": self._tuned,
"generation": self._generation}
- def complete(self, text: str = "", max_new_tokens: int = 512,
- audio=None, audio_format="wav", sample_rate=0,
- channels=1) -> dict:
+ def complete(self, text: str = "", max_new_tokens: int = 512) -> dict:
_track("complete", self._track_props())
- payload = _prepare_audio(audio, audio_format, sample_rate, channels)
- return self._complete(text, max_new_tokens, payload)
+ return self._complete(text, max_new_tokens)
def _complete(self, text: str, max_new_tokens: int = 512,
- audio=None, ground: bool = True) -> dict:
- if audio is not None and self._generation < 3:
- raise ValueError("audio requires a Needle 3 model")
+ ground: bool = True) -> dict:
self._bind()
self._seen_years |= _source_years(text or "")
if self._worker is not None:
- raw = self._worker.complete(text, max_new_tokens, audio)
+ raw = self._worker.complete(text, max_new_tokens)
else:
lib = _lib(self._generation)
- if self._generation >= 3:
- native_audio, keepalive = _native_audio(audio)
- rc = lib.needle_complete(
- text.encode("utf-8"), native_audio, int(max_new_tokens),
- self._buffer, len(self._buffer))
- else:
- rc = lib.needle_complete(
- text.encode("utf-8"), int(max_new_tokens), self._buffer,
- len(self._buffer))
+ rc = lib.needle_complete(
+ text.encode("utf-8"), int(max_new_tokens), self._buffer,
+ len(self._buffer))
if rc < 0:
detail = self._buffer.value.decode("utf-8", "replace")
raise RuntimeError(detail or f"needle_complete failed (code {rc})")
@@ -279,31 +226,26 @@ class Needle:
self._system_text, text)
return response
- def embed(self, text: str = "", audio=None, audio_format="wav",
- sample_rate=0, channels=1) -> list[float]:
+ def embed(self, text: str = "") -> list[float]:
if self._generation < 3:
raise ValueError("embeddings require a Needle 3 model")
- payload = _prepare_audio(audio, audio_format, sample_rate, channels)
self._bind()
if self._worker is not None:
- return self._worker.embed(text, payload)
+ return self._worker.embed(text)
lib = _lib(self._generation)
- native_audio, keepalive = _native_audio(payload)
- dim = lib.needle_embed(text.encode("utf-8"), native_audio, None, 0)
+ dim = lib.needle_embed(text.encode("utf-8"), None, 0)
if dim <= 0:
raise RuntimeError(f"needle_embed failed (code {dim})")
output = (ctypes.c_float * dim)()
- rc = lib.needle_embed(text.encode("utf-8"), native_audio, output, dim)
+ rc = lib.needle_embed(text.encode("utf-8"), output, dim)
if rc != dim:
raise RuntimeError(f"needle_embed failed (code {rc})")
return list(output)
def run(self, query: str = "", max_steps: int = 8,
- max_new_tokens: int = 512, audio=None, audio_format="wav",
- sample_rate=0, channels=1, strict: bool = True) -> dict:
+ max_new_tokens: int = 512, strict: bool = True) -> dict:
_track("run", self._track_props())
- payload = _prepare_audio(audio, audio_format, sample_rate, channels)
- response = self._complete(query, max_new_tokens, payload)
+ response = self._complete(query, max_new_tokens)
executed = []
for _ in range(max_steps):
calls = response.get("function_calls") or []
@@ -595,7 +537,7 @@ def extract(text: str, schema: type | dict, system: str | None = None,
:class:`ExtractionValidationError` instead of being returned silently.
"""
selected = weights
- generation = _weight_generation(selected) if selected else int(generation or 2)
+ generation = _weight_generation(selected) if selected else int(generation or 3)
_track("extract", {"n_tools": 1, "tuned": bool(selected),
"generation": generation})
agent = Needle(tools=[schema], system=system, weights=selected, generation=generation)
@@ -603,7 +545,7 @@ def extract(text: str, schema: type | dict, system: str | None = None,
response = agent._complete(text, max_new_tokens)
finally:
agent.close()
- calls = response.get("function_calls") or []
+ calls = response.get("function_calls") or response.get("suppressed_calls") or []
if not calls:
return None
arguments = calls[0].get("arguments") or {}
diff --git a/needle/_worker.py b/needle/_worker.py
index d950ecd..df76cb7 100644
--- a/needle/_worker.py
+++ b/needle/_worker.py
@@ -1,6 +1,5 @@
from __future__ import annotations
-import base64
import ctypes
import json
import os
@@ -15,36 +14,6 @@ _HEADER = struct.Struct("!Q")
_EOF = object()
-class _NeedleAudio(ctypes.Structure):
- _fields_ = [
- ("data", ctypes.c_void_p),
- ("size", ctypes.c_uint64),
- ("sample_rate", ctypes.c_int),
- ("channels", ctypes.c_int),
- ("format", ctypes.c_int),
- ]
-
-
-def _native_audio(payload):
- if not payload:
- return None, None
- data = base64.b64decode(payload["data"], validate=True)
- storage = ctypes.create_string_buffer(data, len(data))
- descriptor = _NeedleAudio(
- ctypes.cast(storage, ctypes.c_void_p), len(data),
- int(payload["sample_rate"]), int(payload["channels"]),
- int(payload["format"]))
- return ctypes.byref(descriptor), (storage, descriptor)
-
-
-def _wire_audio(payload):
- if payload is None:
- return None
- return {"data": base64.b64encode(payload["data"]).decode("ascii"),
- "sample_rate": payload["sample_rate"],
- "channels": payload["channels"], "format": payload["format"]}
-
-
def _read_exact(stream, size):
chunks = []
remaining = size
@@ -83,18 +52,13 @@ def _load_library(path, generation=2):
lib.needle_init.argtypes = [ctypes.c_char_p, ctypes.c_char_p,
ctypes.c_char_p]
lib.needle_init.restype = ctypes.c_int
+ lib.needle_complete.argtypes = [ctypes.c_char_p, ctypes.c_int,
+ ctypes.c_char_p, ctypes.c_int]
+ lib.needle_complete.restype = ctypes.c_int
if int(generation) >= 3:
- lib.needle_complete.argtypes = [
- ctypes.c_char_p, ctypes.POINTER(_NeedleAudio), ctypes.c_int,
- ctypes.c_char_p, ctypes.c_int]
lib.needle_embed.argtypes = [
- ctypes.c_char_p, ctypes.POINTER(_NeedleAudio),
- ctypes.POINTER(ctypes.c_float), ctypes.c_int]
+ ctypes.c_char_p, ctypes.POINTER(ctypes.c_float), ctypes.c_int]
lib.needle_embed.restype = ctypes.c_int
- else:
- lib.needle_complete.argtypes = [ctypes.c_char_p, ctypes.c_int,
- ctypes.c_char_p, ctypes.c_int]
- lib.needle_complete.restype = ctypes.c_int
lib.needle_reset.argtypes = []
lib.needle_reset.restype = None
lib.needle_load.argtypes = [ctypes.c_char_p, ctypes.c_uint64]
@@ -131,15 +95,9 @@ def _child():
break
operation = request.get("operation")
if operation == "complete":
- if generation >= 3:
- audio, keepalive = _native_audio(request.get("audio"))
- code = lib.needle_complete(
- request["text"].encode("utf-8"), audio,
- int(request["max_new_tokens"]), output, len(output))
- else:
- code = lib.needle_complete(
- request["text"].encode("utf-8"),
- int(request["max_new_tokens"]), output, len(output))
+ code = lib.needle_complete(
+ request["text"].encode("utf-8"),
+ int(request["max_new_tokens"]), output, len(output))
if code < 0:
_write_message(protocol, {
"status": "error",
@@ -157,9 +115,8 @@ def _child():
"message": "embeddings require a Needle 3 model",
})
continue
- audio, keepalive = _native_audio(request.get("audio"))
text = request["text"].encode("utf-8")
- dim = lib.needle_embed(text, audio, None, 0)
+ dim = lib.needle_embed(text, None, 0)
if dim <= 0:
_write_message(protocol, {
"status": "error",
@@ -167,7 +124,7 @@ def _child():
})
continue
embedding = (ctypes.c_float * dim)()
- code = lib.needle_embed(text, audio, embedding, dim)
+ code = lib.needle_embed(text, embedding, dim)
if code != dim:
_write_message(protocol, {
"status": "error",
@@ -281,19 +238,17 @@ class FineTuneWorker:
raise RuntimeError(response.get("message", "Needle worker failed"))
return response
- def complete(self, text, max_new_tokens, audio=None):
+ def complete(self, text, max_new_tokens):
return self._request({
"operation": "complete",
"text": text,
"max_new_tokens": int(max_new_tokens),
- "audio": _wire_audio(audio),
})["response"]
- def embed(self, text, audio=None):
+ def embed(self, text):
return self._request({
"operation": "embed",
"text": text,
- "audio": _wire_audio(audio),
})["embedding"]
def reset(self):
diff --git a/needle/agent/fetch.py b/needle/agent/fetch.py
index 0733b08..5985c1b 100644
--- a/needle/agent/fetch.py
+++ b/needle/agent/fetch.py
@@ -20,7 +20,6 @@ CHECKPOINT_PREFIX = "checkpoints"
# Backwards-compatible aliases for callers that explicitly fetch Needle 2.
HF_REPO = ENGINE_REPOS[2]
-ENGINE_VERSION = ENGINE_VERSIONS[2]
PLATFORMS = ("macos-arm64", "linux-x86_64", "linux-arm64", "linux-armv7",
"linux-riscv64", "linux-mipsel", "windows-x86_64", "windows-arm64",
@@ -128,24 +127,29 @@ def cache_dir(generation=2):
f"v{int(generation)}", engine_version(generation))
-def fetch_weights(generation=2, dest_dir=None):
- """Download the base .cact archive of a generation (cached next to its engine)."""
+def fetch_weights(generation=2, dest_dir=None, force=False):
+ """Download the base .cact archive of a generation (cached next to its engine).
+
+ ``force`` fetches it again even when the cache holds a copy, so every build
+ is a download of the published model.
+ """
import shutil
from huggingface_hub import hf_hub_download
name = base_weights(generation)
dest_dir = dest_dir or cache_dir(generation)
out = os.path.join(dest_dir, name)
- if os.path.exists(out):
+ if os.path.exists(out) and not force:
return out
_register_download(generation)
- cached = hf_hub_download(repo_id=engine_repo(generation), filename=name, repo_type="model")
+ cached = hf_hub_download(repo_id=engine_repo(generation), filename=name, repo_type="model",
+ force_download=force)
os.makedirs(dest_dir, exist_ok=True)
shutil.copyfile(cached, out)
return out
-def fetch_checkpoint(name, dest_dir, generation=3):
+def fetch_checkpoint(name, dest_dir, generation=3, force=False):
"""Download a training checkpoint (needle3.safetensors, ...) for finetune/build."""
import shutil
from huggingface_hub import hf_hub_download
@@ -157,7 +161,8 @@ def fetch_checkpoint(name, dest_dir, generation=3):
cached = None
for candidate in (f"{CHECKPOINT_PREFIX}/{base}", base):
try:
- cached = hf_hub_download(repo_id=repo, filename=candidate, repo_type="model")
+ cached = hf_hub_download(repo_id=repo, filename=candidate, repo_type="model",
+ force_download=force)
break
except EntryNotFoundError:
continue
@@ -181,9 +186,11 @@ def fetch_library(version=None, dest_dir=None, tag=None, generation=2):
_register_download(generation)
path = hf_hub_download(repo_id=repo, filename="python/" + wheel, repo_type="model")
lib = _lib_name_for(tag)
+ stem, suffix = os.path.splitext(lib)
+ member = f"{stem}{generation}{suffix}" if int(generation) >= 3 else lib
os.makedirs(dest_dir, exist_ok=True)
with zipfile.ZipFile(path) as archive:
- data = archive.read("needle/" + lib)
+ data = archive.read("needle/" + member)
out = os.path.join(dest_dir, lib)
with open(out, "wb") as handle:
handle.write(data)
diff --git a/needle/cli.py b/needle/cli.py
index 5279596..9b59d17 100644
--- a/needle/cli.py
+++ b/needle/cli.py
@@ -39,8 +39,8 @@ def _download_target(spec):
if spec.endswith((".safetensors", ".pkl")):
return "checkpoint", spec
raise SystemExit(
- f"unknown download {spec!r}: pass needle3 (base weights), needle3.safetensors or "
- f"needle3_enterprise.safetensors (a checkpoint to fine-tune), a platform ("
+ f"unknown download {spec!r}: pass needle3 (base weights), needle3.safetensors "
+ f"(the checkpoint to fine-tune), a platform ("
+ ", ".join(fetch.PLATFORMS) + "), or <org>/<repo>[/<file>.cact]")
@@ -170,13 +170,6 @@ def main():
p.add_argument("--checkpoint-dir", type=str, default="checkpoints")
p.add_argument("--out", type=str, default=None,
help="Output adapter path (.safetensors, or .pkl)")
- p.add_argument(
- "--qat-bits", choices=["auto", "none", "2", "4"], default="auto",
- help="LoRA training numerics: auto matches the checkpoint export scheme "
- "(default), or force none/2/4")
- p.add_argument("--layers", type=int, default=None,
- help="Fine-tune the N-layer rung of the base (2..L); the adapter "
- "and the built .cact keep that depth")
p = sub.add_parser("generate-data")
p.add_argument("--tools", type=str, default=None, help="Tool schemas JSON to seed generation")
@@ -195,15 +188,13 @@ def main():
p.add_argument("--lora", type=str, default=None, help="LoRA adapter to merge before export")
p.add_argument("--out", type=str, default=None, help="Output .cact path")
p.add_argument("--upload", action="store_true", help="Push the .cact to $NEEDLE_HF_REPO")
- p.add_argument("--bits", type=str, default=None, choices=["2", "4"])
p.add_argument("--layers", type=int, default=None,
- help="Export the N-layer rung of the checkpoint (an adapter "
- "trained with --layers sets this itself)")
+ help="Export the N-layer rung of the base (2..20); default the full 20")
p = sub.add_parser("download")
p.add_argument("spec", type=str,
- help="needle3 (base weights), needle3.safetensors or "
- "needle3_enterprise.safetensors (checkpoints to fine-tune), a platform "
+ help="needle3 (base weights), needle3.safetensors (the checkpoint to "
+ "fine-tune), a platform "
"folder (e.g. macos-arm64), or a Hugging Face spec: "
"<org>/<repo>/<file>.cact, or <org>/<repo> if it holds one archive")
p.add_argument("--out", type=str, default=".", help="Directory to place the files")
@@ -254,12 +245,15 @@ def main():
if kind == "platform":
paths = fetch.download_platform(target, args.out,
generation=args.generation)
+ if args.generation >= 3:
+ paths.append(fetch.fetch_weights(args.generation, os.path.join(args.out, target)))
for path in paths:
print(f" {'file':<9} {path} {os.path.getsize(path) / 1e6:.2f} MB")
runner = next((p for p in paths
if os.path.basename(p) in ("needle", "needle.exe")), None)
if runner:
- print(f" {'next':<9} {runner} --tools tools.json --serve")
+ weights = f" --model {fetch.base_weights(args.generation)}" if args.generation >= 3 else ""
+ print(f" {'next':<9} {runner}{weights} --tools tools.json --serve")
elif kind == "base":
os.makedirs(args.out, exist_ok=True)
path = fetch.fetch_weights(target, args.out)
diff --git a/needle/model/architecture.py b/needle/model/architecture.py
index d7024f1..368cc5a 100644
--- a/needle/model/architecture.py
+++ b/needle/model/architecture.py
@@ -1,5 +1,5 @@
import math
-from dataclasses import dataclass, replace
+from dataclasses import dataclass
import numpy as np
import jax
@@ -72,13 +72,6 @@ class TransformerConfig:
mhc_lanes: int = 4
qkv_conv_taps: int = 3
out_vocab: int = 0
- image_code_offset: int = 0
- image_sem_codes: int = 0
- image_sem_dim: int = 0
- image_tex_books: int = 0
- image_tex_codes: int = 0
- image_tex_stages: int = 0
- image_tex_dim: int = 0
kv_window: int = 0
kv_bits: int = 8
act_bits: int = 8
@@ -119,11 +112,7 @@ class TransformerConfig:
("ladder_depths", ()), ("ladder_sample", False),
("ladder_widths", ()), ("ladder_order", ()),
("engram_seed_heads", 0),
- ("qkv_conv_taps", 0), ("out_vocab", 0),
- ("image_code_offset", 0), ("image_sem_codes", 0),
- ("image_sem_dim", 0), ("image_tex_books", 0),
- ("image_tex_codes", 0), ("image_tex_stages", 0),
- ("image_tex_dim", 0)):
+ ("qkv_conv_taps", 0), ("out_vocab", 0)):
saved.setdefault(key, off)
return cls(**saved)
@@ -174,6 +163,7 @@ def ladder_order(spec):
return _ladder_layer_order(spec.num_layers)
+
def ladder_layer_indices(spec, depth):
"""Return the stable, nested original-layer indices for a depth rung.
@@ -189,15 +179,6 @@ def ladder_layer_indices(spec, depth):
return tuple(sorted(ladder_order(spec)[:depth]))
-def ladder_layer_ranks(spec):
- """Map each original block to its selection rank in the nested ladder."""
- order = ladder_order(spec)
- ranks = [0] * len(order)
- for rank, layer in enumerate(order):
- ranks[layer] = rank
- return tuple(ranks)
-
-
def ladder_config(config, depth):
assert depth < config.num_layers
assert tuple(config.engram_layers) == tuple(sorted(config.engram_layers))
@@ -215,14 +196,6 @@ def ladder_config(config, depth):
return TransformerConfig(**fields)
-def ladder_row_keep(config, exit_depth=None):
- if exit_depth is None:
- return None
- ranks = jnp.asarray(ladder_layer_ranks(config), jnp.int32)
- return jnp.concatenate([jnp.ones((1,), jnp.float32),
- (ranks < exit_depth).astype(jnp.float32)])
-
-
def ladder_slice(params, config, depth):
selected = ladder_layer_indices(config, depth)
selected_set = set(selected)
@@ -251,15 +224,23 @@ def ladder_slice(params, config, depth):
return out
-def parse_ladder_depths(raw):
- if ".." in raw:
- lo, hi = raw.split("..")
- return tuple(range(int(lo), int(hi) + 1)), True
- return tuple(int(v) for v in raw.split(",") if v), False
+def ladder_layer_ranks(spec):
+ """Map each original block to its selection rank in the nested ladder."""
+ order = ladder_order(spec)
+ ranks = [0] * len(order)
+ for rank, layer in enumerate(order):
+ ranks[layer] = rank
+ return tuple(ranks)
-def parse_ladder_widths(raw):
- return tuple(int(v) for v in raw.split(",") if v)
+
+def ladder_row_keep(config, exit_depth=None):
+ if exit_depth is None:
+ return None
+ ranks = jnp.asarray(ladder_layer_ranks(config), jnp.int32)
+ return jnp.concatenate([jnp.ones((1,), jnp.float32),
+ (ranks < exit_depth).astype(jnp.float32)])
+
def width_config(config, w):
@@ -361,34 +342,11 @@ def width_slice(params, config, w):
return a[..., :w]
if p == "embedding/embedding":
return a[..., :w]
- if p.endswith("image_projection/kernel"):
- return a[..., :w]
return a
return jax.tree_util.tree_map_with_path(cut, params)
-def ladder_pool(config, weight):
- pool = []
- for depth in (d for d in config.ladder_depths if d < config.num_layers):
- dcfg = ladder_config(config, depth)
- pool.append((depth, config.d_model,
- SimpleAttentionNetwork(dcfg).apply,
- (lambda p, _d=depth: ladder_slice(p, config, _d)),
- weight * depth / config.num_layers))
- for w in config.ladder_widths:
- pool.append((depth, w,
- SimpleAttentionNetwork(width_config(dcfg, w)).apply,
- (lambda p, _d=depth, _dc=dcfg, _w=w:
- width_slice(ladder_slice(p, config, _d), _dc, _w)),
- weight * (depth / config.num_layers) * (w / config.d_model)))
- for w in config.ladder_widths:
- pool.append((config.num_layers, w,
- SimpleAttentionNetwork(width_config(config, w)).apply,
- (lambda p, _w=w: width_slice(p, config, _w)),
- weight * w / config.d_model))
- return pool
-
def precompute_rope_freqs(head_dim, seq_len, theta=10000.0):
freqs = 1.0 / (theta ** (jnp.arange(0, head_dim, 2).astype(jnp.float32) / head_dim))
@@ -919,10 +877,6 @@ def probe_pool(cells, keep, probes, gain, query, bias, row_keep=None,
).reshape(b, q * d).astype(dtype)
-def head_config(config, key, probes, queries):
- name = key.removesuffix("_head")
- return replace(config, **{f"{name}_probes": probes, f"{name}_queries": queries})
-
class HeadProjection(nn.Module):
features: int
@@ -1015,14 +969,6 @@ HEADS = (EmbeddingHead, ConfidenceHead, RouterHead)
HEAD_KEYS = tuple(head.key for head in HEADS)
-def init_head(params, model, head, seed):
- cfg = model.config
- f32 = lambda tree: jax.tree.map(lambda x: jnp.asarray(x, jnp.float32), tree)
- dummy = jnp.zeros((1, 8, cfg.num_layers + 1, cfg.d_model))
- init = head.init({"params": jax.random.PRNGKey(seed)}, dummy,
- keep=jnp.ones((1, 8)))["params"]
- return f32({k: v for k, v in params.items() if k != head.key}), {head.key: f32(init)}
-
class SimpleAttentionNetwork(nn.Module):
config: TransformerConfig
@@ -1030,20 +976,6 @@ class SimpleAttentionNetwork(nn.Module):
def setup(self):
cfg = self.config
self.embedding = nn.Embed(cfg.vocab_size, cfg.d_model, embedding_init=jinit.normal(stddev=0.02))
- if cfg.image_sem_codes:
- if (cfg.image_sem_dim <= 0 or cfg.image_tex_books <= 0
- or cfg.image_tex_codes <= 0 or cfg.image_tex_stages <= 0
- or cfg.image_tex_dim <= 0):
- raise ValueError("incomplete composite-image embedding geometry")
- self.image_sem_embedding = nn.Embed(
- cfg.image_sem_codes, cfg.image_sem_dim,
- embedding_init=jinit.normal(stddev=0.02))
- self.image_texture_embedding = nn.Embed(
- cfg.image_tex_books * cfg.image_tex_codes * cfg.image_tex_stages,
- cfg.image_tex_dim, embedding_init=jinit.normal(stddev=0.02))
- self.image_projection = nn.Dense(
- cfg.d_model, use_bias=False, kernel_init=default_init())
- self.image_input_norm = ZCRMSNorm(dtype=cfg.jax_dtype)
self.embed_scale = math.sqrt(cfg.d_model)
self.stack = Stack(cfg)
self.embedding_head = EmbeddingHead(cfg)
@@ -1074,48 +1006,14 @@ class SimpleAttentionNetwork(nn.Module):
pairs = [e(indices, ngram_ok, tap_ok, quant=quant) for e in self.engrams]
return jnp.stack([k for k, _ in pairs]), jnp.stack([v for _, v in pairs])
- def _input_embeddings(self, tokens, image_texture=None):
- cfg = self.config
- x = self.embedding(tokens) * self.embed_scale
- if image_texture is None:
- if not (self.is_initializing() and cfg.image_sem_codes):
- return x
- image_texture = jnp.full(
- tokens.shape + (cfg.image_tex_books * cfg.image_tex_stages,),
- 255, dtype=jnp.uint8)
- if not cfg.image_sem_codes:
- raise ValueError("image components were provided to a text-only model")
- expected = cfg.image_tex_books * cfg.image_tex_stages
- if image_texture.shape != tokens.shape + (expected,):
- raise ValueError(
- f"image components must have shape tokens+({expected},)")
- active = image_texture[..., 0] != 255
- texture = jnp.where(active[..., None], image_texture, 0).astype(jnp.int32)
- semantic = jnp.clip(
- tokens.astype(jnp.int32) - cfg.image_code_offset,
- 0, cfg.image_sem_codes - 1)
- sem = self.image_sem_embedding(semantic)
- offsets = (jnp.arange(expected, dtype=jnp.int32)
- * cfg.image_tex_codes)
- tex = self.image_texture_embedding(texture + offsets)
- tex = tex.reshape(
- *tokens.shape, cfg.image_tex_stages, cfg.image_tex_books,
- cfg.image_tex_dim).sum(axis=-3)
- composite = jnp.concatenate(
- [sem, tex.reshape(*tokens.shape,
- cfg.image_tex_books * cfg.image_tex_dim)],
- axis=-1)
- image_x = self.image_projection(composite)
- image_x = (self.image_input_norm(image_x)
- * (0.02 * self.embed_scale)).astype(x.dtype)
- return jnp.where(active[..., None], image_x, x)
+ def _input_embeddings(self, tokens):
+ return self.embedding(tokens) * self.embed_scale
def __call__(self, tokens, mask=None, quant=False,
- image_texture=None, exit_depth=None, exit_only=False,
- subnetwork_only=False):
+ exit_depth=None, exit_only=False, subnetwork_only=False):
if mask is None:
mask = make_causal_mask(tokens.shape[1])
- x = self._input_embeddings(tokens, image_texture)
+ x = self._input_embeddings(tokens)
rope = self._rope(tokens.shape[1])
engram_kv = self._engram_kv(tokens, mask, quant)
stack_out = self.stack(
@@ -1141,8 +1039,7 @@ class SimpleAttentionNetwork(nn.Module):
logits = _aq(x, quant).astype(jnp.float32) @ head.T
return (logits, exit_logits) if exit_x is not None else logits
- def hidden_cells(self, tokens, quant=False, window=0, sink=None,
- image_texture=None, exit_depth=None):
+ def hidden_cells(self, tokens, quant=False, window=0, sink=None, exit_depth=None):
cfg = self.config
mask = (make_causal_mask(tokens.shape[1])
& make_padding_mask(tokens, cfg.pad_token_id))
@@ -1151,7 +1048,7 @@ class SimpleAttentionNetwork(nn.Module):
recent = ((pos[:, None] - pos[None, :]) < window)[None, None, :, :]
keep = recent if sink is None else (recent | sink[:, None, None, :])
mask = mask & keep
- x0 = self._input_embeddings(tokens, image_texture)
+ x0 = self._input_embeddings(tokens)
rope = self._rope(tokens.shape[1])
engram_kv = self._engram_kv(tokens, mask, quant)
hidden = self.stack(
@@ -1182,10 +1079,10 @@ class SimpleAttentionNetwork(nn.Module):
return self.router_head(
*self._head_cells(tokens, quant, window, sink, exit_depth), quant=quant)
- def hidden_states(self, tokens, mask=None, image_texture=None):
+ def hidden_states(self, tokens, mask=None):
if mask is None:
mask = make_causal_mask(tokens.shape[1])
- x = self._input_embeddings(tokens, image_texture)
+ x = self._input_embeddings(tokens)
rope = self._rope(tokens.shape[1])
engram_kv = self._engram_kv(tokens, mask, False)
_, hidden = self.stack(x, mask=mask, rope=rope, engram_kv=engram_kv,
@@ -1237,17 +1134,3 @@ def effective_kv_window(config):
return min(budget, config.kv_window) if config.kv_window else budget
-def make_causal_packing_mask(seg_ids, prefix=None, window=0):
- T = seg_ids.shape[1]
- causal = jnp.tril(jnp.ones((T, T), dtype=jnp.bool_))
- block = (seg_ids[:, :, None] == seg_ids[:, None, :]) & (seg_ids[:, :, None] > 0)
- mask = block & causal[None, :, :]
- static_zero = isinstance(window, (int, np.integer)) and int(window) == 0
- if window is not None and not static_zero:
- pos = jnp.arange(T)
- limit = jnp.where(jnp.asarray(window) > 0, window, T + 1)
- recent = (pos[:, None] - pos[None, :]) < limit
- sink = (jnp.zeros_like(seg_ids, dtype=jnp.bool_) if prefix is None
- else prefix > 0)
- mask = mask & (recent[None, :, :] | sink[:, None, :])
- return mask[:, None, :, :]
diff --git a/needle/model/checkpoints.py b/needle/model/checkpoints.py
index e8f8b1b..2f4791b 100644
--- a/needle/model/checkpoints.py
+++ b/needle/model/checkpoints.py
@@ -92,7 +92,7 @@ def write_checkpoint(path, checkpoint):
pickle.dump(checkpoint, handle)
-_ADAPTER_FIELDS = ("scale", "base", "rank", "qat_bits", "qat_bits_map", "seed", "layers")
+_ADAPTER_FIELDS = ("scale", "base", "rank", "seed")
def write_adapter(path, adapter):
diff --git a/needle/model/export.py b/needle/model/export.py
index d647878..cb0c99f 100644
--- a/needle/model/export.py
+++ b/needle/model/export.py
@@ -29,9 +29,8 @@ HEADER (49 u32-sized fields, rope_theta an f32, everything else u32):
global_mask bits (layer i = bit i, lo then hi u32) mark full-attention
layers. qkv_conv_taps is the causal depthwise conv width on Q,K,V (0 = none);
then codebook_len * f32 - the shared Lloyd-Max unit-sphere codebooks,
-cb2[4] | cb3[8] | cb4[16] concatenated, so mixed-precision blobs need one
-header. Binary CQ1 uses its analytic {-c, +c} codebook and does not enlarge the
-header. kv_window is the sliding-window width the model was trained with (0 =
+cb2[4] | cb3[8] | cb4[16] concatenated, the layout the runtime indexes by
+width. kv_window is the sliding-window width the model was trained with (0 =
size it from the KV budget alone); kv_bits is the KV-cache width it was
post-trained for (8 = int8, else 2/3/4). Both ride in the blob because a model
quantized for one must be RUN at it - leaving either to a runtime flag silently
@@ -40,23 +39,15 @@ serves the wrong numerics.
DIRECTORY (num_tensors records, {REC_SIZE} bytes each, no names):
u8 dtype, u8 ndim, u16 _pad, u32 shape[4], u64 offset, u64 nbytes,
u32 group_size, u32 bits. dtype: 1=FP16, 2=FP32, 3=CQ (bits gives the
- width: 1, 2, 3 or 4; 5 denotes ternary crumbs), 4=RAW.
+ width: 1, 2, 3 or 4; 5 denotes ternary crumbs; this package writes 4), 4=RAW.
TENSOR ORDER (what the runtime indexes by position):
embedding; per layer [norm_in, q_proj, k_proj, v_proj,
(q_taps, k_taps, v_taps when qkv_conv_taps > 0), q_norm, k_norm,
gate_proj, out_proj, post_norm, attn_gate, pre_hada, d1, d2, b2, d3, d4,
w1a, w1b, w2a, w2b, w3a, w3b, cond_v, cond_u]; the 9 mHC blocks; per engram
- site [tables, key_proj, value_proj, taps]; final_norm; optional composite
- image tensors [manifest, semantic table, texture table, projection,
- input_norm]; the optional probe heads (see below); then the RAW tokenizer.
-
-The image manifest is FP32 with fields [magic, code_offset, positions,
-semantic_codes, semantic_dim, texture_books, texture_codes, texture_stages,
-texture_dim]. The vision encoder transports semantic IDs first, followed by
-positions * texture_books * texture_stages texture IDs. The runtime consumes
-that buffer as `positions` LM inputs rather than flattening every code into a
-separate token.
+ site [tables, key_proj, value_proj, taps]; final_norm; the optional probe
+ heads (see below); then the RAW tokenizer.
PROBE HEADS (appended so pre-head blobs stay loadable). Let
extra = num_tensors - (through final_norm) - (tokenizer). extra == 0 means no
@@ -91,10 +82,9 @@ c = 1.2240064, are analytic.
BINARY (record bits=1, 2 levels) stores eight indices per byte. Its codebook is
also analytic: {-sqrt(2/pi), +sqrt(2/pi)} / sqrt(group).
-RAW attachments follow the model tensors and heads. The first is the
-self-contained SentencePiece BPE dump; the optional second is the speech
-encoder's uncompressed NPZ archive. The SentencePiece dump format (see RefTokenizer
-below for the reference encoder/decoder this specifies):
+One RAW attachment follows the model tensors and heads: the self-contained
+SentencePiece BPE dump. Its format (see RefTokenizer below for the reference
+encoder/decoder this specifies):
header u32 n_pieces, u32 pad/eos/bos/unk id, u8 add_dummy_prefix,
u8 byte_fallback, u16 _pad;
then n_pieces records, id order: f32 score, u8 type
@@ -102,22 +92,17 @@ below for the reference encoder/decoder this specifies):
u16 surface_len, surface_len UTF-8 bytes.
=============================================================================
"""
-import os
import struct
import numpy as np
-from .quantize import (_bits_for, _cq_codebook_np, _cq_hadamard_np,
- parse_bits_map, TERNARY_BITS)
+from .quantize import _cq_codebook_np, _cq_hadamard_np, TERNARY_BITS, WEIGHT_BITS
TAG = 0x05E12A84
ALIGN = 64
FP16, FP32, CQ, RAW = 1, 2, 3, 4
CB_BITS = (2, 3, 4)
-PACK_BITS = (1, *CB_BITS)
TERNARY_RECORD_BITS = 5
-IMAGE_MANIFEST_MAGIC = 7391.0
-IMAGE_TENSOR_COUNT = 5
TK_NORMAL, TK_UNKNOWN, TK_CONTROL, TK_USER_DEFINED, TK_BYTE = 0, 1, 2, 3, 4
_TK_HDR = "<IIIIIBBH"
@@ -191,20 +176,14 @@ def _packed_row_bytes(in_pad, bits, group):
return in_pad * bits // 8
-def _pack_ternary_crumbs(idx):
- crumbs = np.where(idx == 0, 3, idx - 1).astype(np.uint8)
- return _pack_lsb(crumbs, 2)
-
-
def _unpack_ternary_crumbs(packed, in_pad):
crumbs = _unpack_lsb(packed, 2, in_pad)
return np.where(crumbs == 3, 0, crumbs + 1).astype(np.uint8)
def _cq_pack(w, bits, group):
- if bits not in PACK_BITS and bits != TERNARY_BITS:
- raise ValueError(
- f"CQ packing supports bits in {PACK_BITS} or {TERNARY_BITS}; got bits={bits}")
+ if bits != WEIGHT_BITS:
+ raise ValueError(f"CQ packing supports bits={WEIGHT_BITS}; got bits={bits}")
cb = _cq_codebook_np(bits, group)
H = _cq_hadamard_np(group)
out, D = w.shape
@@ -216,8 +195,7 @@ def _cq_pack(w, bits, group):
norm = np.sqrt((rot ** 2).sum(-1, keepdims=True))
unit = rot / np.maximum(norm, 1e-12)
idx = _nearest_idx(unit, cb).reshape(out, in_pad)
- packed = _pack_ternary_crumbs(idx) if bits == TERNARY_BITS else _pack_lsb(idx, bits)
- return packed, norm[:, :, 0].astype(np.float16)
+ return _pack_lsb(idx, bits), norm[:, :, 0].astype(np.float16)
def _cq_unpack(packed, norms, out, in_dim, bits, group):
@@ -251,19 +229,14 @@ def _fp16(name, arr):
def _q(name, mat, bits, group):
- if isinstance(bits, tuple):
- bits = _bits_for(name, *bits)
packed, norms = _cq_pack(np.asarray(mat, np.float32), bits, group)
- rec_bits = TERNARY_RECORD_BITS if bits == TERNARY_BITS else bits
- return _Tensor(name, CQ, mat.shape, packed.tobytes() + norms.tobytes(), group, rec_bits)
+ return _Tensor(name, CQ, mat.shape, packed.tobytes() + norms.tobytes(), group, bits)
-def _tensors(params, config, bits, group, vocab_rows=None):
+def _tensors(params, config, bits, group):
_, _, orders, heads, sub_dim, sites = _geometry(config)
num_tables = len(orders) * heads
embedding = np.asarray(_get(params, ("embedding", "embedding")))
- if vocab_rows:
- embedding = np.concatenate([embedding[start:end] for start, end in vocab_rows])
ts = [_q("embedding", embedding, bits, group)]
taps_n = int(getattr(config, "qkv_conv_taps", 0))
@@ -334,57 +307,10 @@ def _tensors(params, config, bits, group, vocab_rows=None):
]
ts.append(_fp16("final_norm", params["stack"]["final_norm"]["scale"]))
- ts += _image_tensors(params, config, bits, group)
ts += _head_tensors(params, group)
return ts
-def _image_tensors(params, config, bits, group):
- sem_codes = int(getattr(config, "image_sem_codes", 0) or 0)
- if not sem_codes:
- return []
- code_offset = int(getattr(config, "image_code_offset", 0) or 0)
- positions = int(getattr(config, "image_positions", 256) or 256)
- sem_dim = int(getattr(config, "image_sem_dim", 0) or 0)
- tex_books = int(getattr(config, "image_tex_books", 0) or 0)
- tex_codes = int(getattr(config, "image_tex_codes", 0) or 0)
- tex_stages = int(getattr(config, "image_tex_stages", 0) or 0)
- tex_dim = int(getattr(config, "image_tex_dim", 0) or 0)
- if (code_offset < 0 or positions <= 0 or sem_dim <= 0 or tex_books <= 0
- or tex_codes <= 0 or tex_stages <= 0 or tex_dim <= 0
- or code_offset + sem_codes > int(config.vocab_size)):
- raise ValueError("invalid composite-image export geometry")
-
- semantic = np.asarray(
- params["image_sem_embedding"]["embedding"], np.float32)
- texture = np.asarray(
- params["image_texture_embedding"]["embedding"], np.float32)
- projection = np.asarray(params["image_projection"]["kernel"], np.float32)
- input_norm = np.asarray(params["image_input_norm"]["scale"], np.float32)
- composite = sem_dim + tex_books * tex_dim
- expected_texture = tex_books * tex_codes * tex_stages
- if semantic.shape != (sem_codes, sem_dim):
- raise ValueError("semantic image table disagrees with model config")
- if texture.shape != (expected_texture, tex_dim):
- raise ValueError("texture image table disagrees with model config")
- if projection.shape != (composite, int(config.d_model)):
- raise ValueError("image projection disagrees with composite geometry")
- if input_norm.shape != (int(config.d_model),):
- raise ValueError("image input norm disagrees with model dimension")
-
- manifest = np.asarray([
- IMAGE_MANIFEST_MAGIC, code_offset, positions, sem_codes, sem_dim,
- tex_books, tex_codes, tex_stages, tex_dim,
- ], np.float32)
- return [
- _Tensor("image.manifest", FP32, manifest.shape, manifest.tobytes()),
- _fp16("image.semantic", semantic),
- _fp16("image.texture", texture),
- _q("image_projection", projection.T, bits, group),
- _fp16("image.input_norm", input_norm),
- ]
-
-
HEAD_MATRICES = ("probes", "query", "proj")
@@ -436,28 +362,38 @@ def _tokenizer_blob(tok):
return bytes(out)
-def vocab_rows_for(config):
- text = int(getattr(config, "out_vocab", 0) or 0)
- if not text or text >= int(config.vocab_size):
- return None
- return [(0, text)]
+def _tokenizer_pieces(blob):
+ return struct.unpack_from(_TK_HDR, blob, 0)[0]
-def _pack_cact(params, config, bits, group, tokenizer, kv_window=0,
- speech_tokenizer=None, vocab_rows=None):
+def read_tokenizer_blob(path):
+ """The packaged tokenizer of a .cact archive, read without unpacking its weights."""
+ with open(path, "rb") as f:
+ raw = f.read()
+ tag, num_tensors, cb_n = struct.unpack_from(_HDR_FMT, raw, 0)[:3]
+ if tag != TAG:
+ raise ValueError(f"{path} is not a Needle 3 .cact archive")
+ off = struct.calcsize(_HDR_FMT) + cb_n * 4
+ for _ in range(num_tensors):
+ rec = struct.unpack(_REC_FMT, raw[off:off + REC_SIZE]); off += REC_SIZE
+ if rec[0] == RAW:
+ return raw[rec[7]:rec[7] + rec[8]]
+ raise ValueError(f"{path} carries no tokenizer")
+
+
+def _pack_cact(params, config, bits, group, tokenizer, kv_window=0):
params = {k: v for k, v in params.items()}
- ts = _tensors(params, config, bits, group, vocab_rows)
- vocab = (sum(end - start for start, end in vocab_rows) if vocab_rows
- else int(config.vocab_size))
+ ts = _tensors(params, config, bits, group)
+ vocab = int(config.vocab_size)
out_vocab = int(getattr(config, "out_vocab", 0) or 0)
if out_vocab > vocab:
raise ValueError(f"out_vocab {out_vocab} exceeds the exported vocab {vocab}")
if tokenizer is not None:
- if tokenizer.vocab_size > vocab:
- raise ValueError(f"tokenizer vocab {tokenizer.vocab_size} > exported vocab {vocab}")
- ts.append(_Tensor("tokenizer", RAW, (), _tokenizer_blob(tokenizer)))
- if speech_tokenizer is not None:
- ts.append(_Tensor("speech_tokenizer", RAW, (), speech_tokenizer))
+ blob = tokenizer if isinstance(tokenizer, (bytes, bytearray)) else _tokenizer_blob(tokenizer)
+ pieces = _tokenizer_pieces(blob)
+ if pieces > vocab:
+ raise ValueError(f"tokenizer vocab {pieces} > exported vocab {vocab}")
+ ts.append(_Tensor("tokenizer", RAW, (), bytes(blob)))
cb = np.concatenate([_cq_codebook_np(b, group) for b in CB_BITS]).astype(np.float32)
kv_bits = int(getattr(config, "kv_bits", 8) or 8)
qk_hd, v_hd, orders, heads, sub_dim, sites = _geometry(config)
@@ -501,19 +437,10 @@ def _pack_cact(params, config, bits, group, tokenizer, kv_window=0,
return bytes(buf), len(ts)
-def build_export(params, config, bits=4, group=128, tokenizer=None, bits_map=None,
- kv_window=0, speech_tokenizer=None, vocab_rows=None):
- b = parse_bits_map(bits_map) if bits_map else bits
- return _pack_cact(params, config, b, group, tokenizer, kv_window,
- speech_tokenizer, vocab_rows)[0]
-
-def write_export(params, config, path, bits=4, group=128, tokenizer=None,
- bits_map=None, kv_window=0, speech_tokenizer=None,
- vocab_rows=None):
- b = parse_bits_map(bits_map) if bits_map else bits
- buf, n = _pack_cact(params, config, b, group, tokenizer, kv_window,
- speech_tokenizer, vocab_rows)
+def write_export(params, config, path, bits=WEIGHT_BITS, group=128, tokenizer=None,
+ kv_window=0):
+ buf, n = _pack_cact(params, config, bits, group, tokenizer, kv_window)
with open(path, "wb") as f:
f.write(buf)
return {"path": path, "bytes": len(buf), "tensors": n}
diff --git a/needle/model/finetune.py b/needle/model/finetune.py
index 2b8769f..eddab9c 100644
--- a/needle/model/finetune.py
+++ b/needle/model/finetune.py
@@ -258,6 +258,7 @@ def load_jsonl(path, tokenizer, max_len):
return np.array(seqs, np.int32), np.array(masks, np.float32)
+
def rung(params, config, layers):
from .architecture import ladder_config, ladder_slice
@@ -317,8 +318,7 @@ def finetune_local(args, progress=None):
import optax
from .run import load_checkpoint
from .architecture import SimpleAttentionNetwork
- from .quantize import (configure_deploy, cq_ste_params,
- cq_ste_mixed_params, parse_bits_map)
+ from .quantize import configure_deploy, cq_ste_params, WEIGHT_BITS
def emit(msg):
print(msg, flush=True)
@@ -332,8 +332,6 @@ def finetune_local(args, progress=None):
workers=getattr(args, "workers", 8))
params, config = load_checkpoint(base_path)
- layers = getattr(args, "layers", None)
- params, config = rung(params, config, layers)
config.dtype = "float32"
params = jax.tree.map(lambda a: np.asarray(a).astype(np.float32), params)
backend = jax.default_backend().lower()
@@ -352,29 +350,9 @@ def finetune_local(args, progress=None):
emit(f" {'data':<9} {len(seqs)} examples seq_len {max_len} cap {args.max_len}")
model = SimpleAttentionNetwork(config)
- qat_mode = getattr(args, "qat_bits", "auto")
- qat_mode = "none" if qat_mode is None else str(qat_mode).lower()
- qat_bits = None
- qat_bits_map = None
- parsed_bits_map = None
- if qat_mode == "auto":
- qat_bits_map = getattr(config, "weight_bits", "") or None
- if qat_bits_map:
- parsed_bits_map = parse_bits_map(qat_bits_map)
- else:
- qat_bits = 4
- elif qat_mode in ("2", "4"):
- qat_bits = int(qat_mode)
- elif qat_mode != "none":
- raise ValueError("--qat-bits must be auto, none, 2, or 4")
- qat_enabled = qat_bits is not None or qat_bits_map is not None
- if qat_enabled:
- configure_deploy(act_bits=getattr(config, "act_bits", 8),
- kv_bits=getattr(config, "kv_bits", 8))
- scheme = f"mixed[{qat_bits_map}]" if qat_bits_map else f"W{qat_bits}"
- emit(f" {'numerics':<9} CQ {scheme} STE + A8 (matches export)")
- else:
- emit(f" {'numerics':<9} full precision")
+ configure_deploy(act_bits=getattr(config, "act_bits", 8),
+ kv_bits=getattr(config, "kv_bits", 8))
+ emit(f" {'numerics':<9} CQ W{WEIGHT_BITS} STE + A8 (matches export)")
paths = lora_target_paths(params)
scale = args.lora_alpha / args.lora_rank
seed = int(getattr(args, "seed", 0))
@@ -402,13 +380,8 @@ def finetune_local(args, progress=None):
emit(f" {'schedule':<9} {total_steps} steps warmup {warmup} cosine decay clip 1.0 (compiling...)")
def loss_fn(lora, ids, mask):
- merged = merge_lora(params, lora, scale)
- if qat_bits_map is not None:
- bits_map, default_bits = parsed_bits_map
- merged = cq_ste_mixed_params(merged, bits_map, default_bits)
- elif qat_bits is not None:
- merged = cq_ste_params(merged, qat_bits)
- logits = model.apply({"params": merged}, ids, quant=qat_enabled)
+ merged = cq_ste_params(merge_lora(params, lora, scale), WEIGHT_BITS)
+ logits = model.apply({"params": merged}, ids, quant=True)
logits, targets, mask = logits[:, :-1], ids[:, 1:], mask[:, 1:]
ce = optax.softmax_cross_entropy_with_integer_labels(logits, targets)
return (ce * mask).sum() / jnp.maximum(mask.sum(), 1.0)
@@ -450,10 +423,7 @@ def finetune_local(args, progress=None):
"scale": float(scale),
"base": base_path,
"rank": args.lora_rank,
- "qat_bits": qat_bits,
- "qat_bits_map": qat_bits_map,
"seed": seed,
- "layers": int(config.num_layers),
})
print(f" {'adapter':<9} {out}")
print(f" {'next':<9} needle build {base_path} --lora {out}")
@@ -462,66 +432,33 @@ def finetune_local(args, progress=None):
def build_main(args):
import jax.numpy as jnp
+ from ..agent import fetch
from .run import load_checkpoint
from .architecture import effective_kv_window
- from .export import vocab_rows_for, write_export
+ from .export import read_tokenizer_blob, write_export
+ from .quantize import WEIGHT_BITS
+ base_archive = fetch.fetch_weights(3, force=True)
+ print(f" {'base':<9} {base_archive} {os.path.getsize(base_archive) / 1e6:.2f} MB")
adapter = read_adapter(args.lora) if args.lora else None
checkpoint = args.checkpoint
if not checkpoint and adapter and adapter.get("base") and os.path.exists(adapter["base"]):
checkpoint = adapter["base"]
checkpoint = checkpoint or DEFAULT_BASE
params, config, _ = load_checkpoint(checkpoint, return_run=True)
- layers = getattr(args, "layers", None)
-
- adapter_qat_bits = None
- adapter_qat_bits_map = None
- if args.lora:
- adapter_layers = adapter.get("layers")
- if adapter_layers and layers and int(layers) != int(adapter_layers):
- raise ValueError(
- f"adapter was trained on the {adapter_layers}-layer rung, but --layers "
- f"{layers} would export a different depth")
- layers = adapter_layers or layers
- params, config = rung(params, config, layers)
if args.lora:
lora = {tuple(key.split("/")): {"A": jnp.asarray(v["A"]), "B": jnp.asarray(v["B"])}
for key, v in adapter["lora"].items()}
params = merge_lora(params, lora, adapter["scale"])
print(f" {'merged':<9} {len(lora)} weight groups {args.lora}")
- adapter_qat_bits = adapter.get("qat_bits")
- adapter_qat_bits_map = adapter.get("qat_bits_map")
-
- bits = args.bits
- if adapter_qat_bits_map is not None:
- if bits is not None:
- raise ValueError(
- "adapter was trained for the checkpoint's mixed CQ bit map, but "
- f"--bits {bits} would deploy different numerics")
- bits_map = adapter_qat_bits_map
- print(f" {'scheme':<9} CQ mixed[{bits_map}] from QAT adapter metadata")
- elif adapter_qat_bits is not None:
- if bits is not None and int(bits) != int(adapter_qat_bits):
- raise ValueError(
- f"adapter was trained for CQ W{adapter_qat_bits}, but --bits {bits} "
- "would deploy different numerics; rebuild at the adapter bit width")
- bits = str(adapter_qat_bits)
- bits_map = None
- print(f" {'scheme':<9} CQ W{bits} from QAT adapter metadata")
- else:
- bits_map = None if bits else (getattr(config, "weight_bits", "") or None)
- if not bits and not bits_map:
- bits = "4"
+ params, config = rung(params, config, getattr(args, "layers", None))
+ print(f" {'depth':<9} {config.num_layers} layers")
out = args.out or (os.path.splitext(os.path.basename(checkpoint))[0] + ".cact")
- info = write_export(params, config, out,
- bits=int(bits) if bits else 4,
- bits_map=bits_map,
- tokenizer=get_tokenizer(config.vocab_size),
- kv_window=effective_kv_window(config),
- vocab_rows=vocab_rows_for(config))
- scheme = f"mixed[{bits_map}]" if bits_map else f"W{bits}"
- print(f" {'wrote':<9} {info['path']} {info['bytes'] / 1e6:.2f} MB {info['tensors']} tensors {scheme}A8")
+ info = write_export(params, config, out, bits=WEIGHT_BITS,
+ tokenizer=read_tokenizer_blob(base_archive),
+ kv_window=effective_kv_window(config))
+ print(f" {'wrote':<9} {info['path']} {info['bytes'] / 1e6:.2f} MB {info['tensors']} tensors W{WEIGHT_BITS}A8")
print(f" {'next':<9} needle.Needle(weights={out!r}, tools=[...])")
if args.upload:
diff --git a/needle/model/quantize.py b/needle/model/quantize.py
index 9013c46..cc36937 100644
--- a/needle/model/quantize.py
+++ b/needle/model/quantize.py
@@ -1,6 +1,5 @@
import functools
import math
-import re
import numpy as np
import jax
@@ -59,18 +58,12 @@ def maybe_quant_kv(x, quant):
quant, quantize, lambda t: t, x)
-QAT_EVERY = 0
-_WEIGHT_GROUP = 128
-_WEIGHT_BITS = 4
+WEIGHT_BITS = 4
ACT_BITS = 8
KV_BITS = 0
_KV_GROUP = 64
-def configure_qat(every, weight_group=128, weight_bits=4):
- global QAT_EVERY, _WEIGHT_GROUP, _WEIGHT_BITS
- QAT_EVERY, _WEIGHT_GROUP, _WEIGHT_BITS = int(every), int(weight_group), int(weight_bits)
-
def configure_deploy(act_bits=8, kv_bits=8, kv_group=64):
global ACT_BITS, KV_BITS, _KV_GROUP
@@ -81,29 +74,6 @@ def configure_deploy(act_bits=8, kv_bits=8, kv_group=64):
jax.clear_caches()
-def quantize_params_configured(params):
- return cq_ste_params(params, _WEIGHT_BITS, _WEIGHT_GROUP)
-
-
-def deploy_quantize(params, config):
- spec = getattr(config, "weight_bits", "") or ""
- if spec:
- bits_map, default_bits = parse_bits_map(spec)
- return cq_mixed_params(params, bits_map, default_bits), spec
- return (cq_quantize_params(params, _WEIGHT_BITS, _WEIGHT_GROUP),
- f"CQ W{_WEIGHT_BITS}")
-
-
-def weight_bits():
- return _WEIGHT_BITS
-
-
-def maybe_quant_weights(params, do_quantize):
- if not QAT_EVERY:
- return params
- return jax.lax.cond(
- do_quantize, quantize_params_configured, lambda p: p, params)
-
@functools.lru_cache(maxsize=None)
def _lloyd_max_gaussian(bits, iters=200, samples=400000, seed=0):
@@ -122,6 +92,7 @@ def _lloyd_max_gaussian(bits, iters=200, samples=400000, seed=0):
c[k] = x[m].mean()
return np.sort(c)
+
TERNARY_BITS = 1.58
_TERNARY_CB = np.array([-1.2240064, 0.0, 1.2240064])
@@ -187,126 +158,10 @@ def _ab_of(params):
return params.get(AB_KEY) if hasattr(params, "get") else None
-def ab_init_params(params, seed=0, signs=False):
- """a = b = 1 (exact no-op at step 0; the PTQ default) or, with signs=True,
- a = b = random +-1 (randomized-Hadamard init for from-scratch/SFT runs).
- a*b = 1 either way, so effective FP weights are unchanged. A tree that
- already carries AB (a resumed QAPT checkpoint) is returned unchanged."""
- if _ab_of(params):
- return dict(params)
- rng = np.random.RandomState(seed)
- ab = {}
-
- def visit(path, leaf):
- if _is_quant_leaf(path, leaf):
- r = leaf.shape[-2] if _reduces_second_last(path) else leaf.shape[-1]
- v = ((rng.randint(0, 2, r) * 2 - 1) if signs else np.ones(r))
- ab[leaf_name(path)] = {"a": jnp.asarray(v, jnp.float32),
- "b": jnp.asarray(v, jnp.float32)}
- return leaf
-
- jax.tree_util.tree_map_with_path(visit, params)
- out = dict(params)
- out[AB_KEY] = ab
- return out
-
-
-def freeze_quant_grads(grads):
- """Zero the gradients of every CQ-quantized leaf (signs stay frozen);
- norms, gates, AB scales and other smooth params keep theirs."""
- def z(path, leaf):
- return jnp.zeros_like(leaf) if _is_quant_leaf(path, leaf) else leaf
- return jax.tree_util.tree_map_with_path(z, grads)
-
-
-def ab_fold_params(params):
- """FP-effective tree of an AB-carrying model (w <- a*b*w, AB_KEY dropped):
- the clean-forward reference of the function the quant transforms deploy."""
- if not _ab_of(params):
- return params
- folded = dict(_map_quant_leaves(params, lambda w, i: w))
- folded.pop(AB_KEY)
- return folded
-
-
-def cq_model_bytes(params, bits, group_size=128):
- acc = {"q": 0, "groups": 0, "other": 0}
-
- def visit(path, leaf):
- n = int(np.prod(leaf.shape))
- if _is_quant_leaf(path, leaf):
- acc["q"] += n
- per = leaf.shape[-2] if _reduces_second_last(path) else leaf.shape[-1]
- acc["groups"] += (n // per) * (-(-per // group_size))
- else:
- acc["other"] += n
- return leaf
-
- jax.tree_util.tree_map_with_path(visit, params)
- return acc["q"] * bits / 8 + acc["groups"] * 2 + acc["other"] * 2
-
-
-def model_bytes_fp16(params):
- return 2 * sum(int(np.prod(l.shape)) for l in jax.tree_util.tree_leaves(params))
-
-
-
CQ_GROUP_SIZE = 128
-CQ_BITS = (1, 2, 3, 4)
-MIN_BITS, MAX_BITS = 1, 8
-
-
-def canonical_tensor_name(name):
- n = re.sub(r"^layer\d+\.", "attn.", name)
- n = re.sub(r"^engrams?_?(\d+)[./]?", r"engram\1.", n)
- if n.startswith("stack/"):
- n = n[len("stack/"):]
- n = n.replace("layers/block/self_attn/", "attn.")
- n = n.replace("mtp_block/self_attn/", "mtp.")
- if n.endswith("/kernel"):
- n = n[:-len("/kernel")]
- if n == "embedding/embedding":
- n = "embedding"
- n = re.sub(r"^(engram\d+\.)embedding$", r"\1tables", n)
- return n.replace("/", ".")
-
-
-def _bits_for(name, bits_map, default_bits):
- canon = canonical_tensor_name(name)
- hits = [k for k in bits_map if canon.startswith(k)]
- return bits_map[max(hits, key=len)] if hits else default_bits
[email protected]_cache(maxsize=None)
-def cq_distortion(bits, group_size=CQ_GROUP_SIZE, rows=2048, seed=0):
- w = np.random.RandomState(seed).randn(rows, 4 * group_size).astype(np.float32)
- deq = np.asarray(cq_quantize(jnp.asarray(w), bits, group_size))
- return float(np.mean((deq - w) ** 2) / np.mean(w ** 2))
-
-
-def noise_scale(bits, group_size=CQ_GROUP_SIZE):
- b = min(max(float(bits), MIN_BITS), MAX_BITS)
- lo, hi = int(np.floor(b)), int(np.ceil(b))
- d_lo = cq_distortion(lo, group_size)
- if hi == lo:
- return float(np.sqrt(d_lo))
- d_hi = cq_distortion(hi, group_size)
- t = b - lo
- return float(np.sqrt(np.exp((1.0 - t) * np.log(d_lo) + t * np.log(d_hi))))
-
-
-def add_cq_noise(w, key, scale, group_size=CQ_GROUP_SIZE):
- D, g = w.shape[-1], group_size
- pad = (-D) % g
- wp = jnp.pad(w, [(0, 0)] * (w.ndim - 1) + [(0, pad)]) if pad else w
- groups = wp.reshape(*wp.shape[:-1], -1, g).astype(jnp.float32)
- rms = jnp.sqrt(jnp.mean(jnp.square(groups), axis=-1, keepdims=True))
- sigma = jax.lax.stop_gradient(rms) * scale
- eps = jax.random.normal(key, groups.shape, dtype=jnp.float32)
- noisy = (groups + sigma * eps).reshape(wp.shape).astype(w.dtype)
- return noisy[..., :D] if pad else noisy
-
def _map_quant_leaves(params, fn):
"""fn(leaf_with_reduction_axis_last, index) over the CQ-quantized leaves;
@@ -332,21 +187,6 @@ def _map_quant_leaves(params, fn):
return jax.tree_util.tree_map_with_path(q, params)
-def noise_params(params, key, scale, group_size=CQ_GROUP_SIZE):
- return _map_quant_leaves(
- params,
- lambda w, i: add_cq_noise(w, jax.random.fold_in(key, i), scale, group_size),
- )
-
-
-def noise_params_only(params, key, scale, name, group_size=CQ_GROUP_SIZE):
- names = [n for n, _ in quant_leaf_names(params)]
- return _map_quant_leaves(
- params,
- lambda w, i: (add_cq_noise(w, key, scale, group_size)
- if names[i].startswith(name) else w),
- )
-
def leaf_name(path):
# Partitioned/AxisMetadata leaves add a trailing GetAttrKey("value").
@@ -370,58 +210,6 @@ def quant_leaf_names(params):
return out
-def parse_bits_map(spec):
- m, default = {}, None
- for part in str(spec or "").split(","):
- if not part.strip():
- continue
- k, v = part.split("=", 1)
- b = TERNARY_BITS if float(v) == TERNARY_BITS else int(v)
- if k.strip() == "default":
- default = b
- else:
- m[canonical_tensor_name(k.strip())] = b
- if default is None:
- raise ValueError(f"bits map {spec!r} needs a default=<b> entry")
- bad = {b for b in list(m.values()) + [default]
- if b not in CQ_BITS and b != TERNARY_BITS}
- if bad:
- raise ValueError(f"bits map {spec!r} has unsupported widths {sorted(bad)}")
- return m, default
-
-
-def cq_mixed_params(params, bits_map, default_bits, group_size=CQ_GROUP_SIZE):
- names = [n for n, _ in quant_leaf_names(params)]
-
- def fn(w, i):
- b = _bits_for(names[i], bits_map, default_bits)
- return cq_quantize(w, b, group_size)
-
- return _map_quant_leaves(params, fn)
-
-
-def cq_mixed_stats(params, bits_map, default_bits, group_size=CQ_GROUP_SIZE):
- qbits = qn = 0
- other = 0
- groups = 0
-
- def visit(path, leaf):
- nonlocal qbits, qn, other, groups
- n = int(np.prod(leaf.shape))
- if _is_quant_leaf(path, leaf):
- b = _bits_for(leaf_name(path), bits_map, default_bits)
- qbits += b * n
- qn += n
- per = leaf.shape[-2] if _reduces_second_last(path) else leaf.shape[-1]
- groups += (n // per) * (-(-per // group_size))
- else:
- other += n
- return leaf
-
- jax.tree_util.tree_map_with_path(visit, params)
- mb = (qbits / 8 + groups * 2 + other * 2) / 1e6
- return qbits / max(qn, 1), mb
-
def cq_ste(w, bits, group_size=CQ_GROUP_SIZE):
return w + jax.lax.stop_gradient(cq_quantize(w, bits, group_size) - w)
@@ -445,53 +233,3 @@ def cq_ste_params(params, bits, group_size=CQ_GROUP_SIZE):
return _map_quant_leaves(params, lambda w, i: cq_ste(w, bits, group_size))
-def cq_ste_mixed_params(params, bits_map, default_bits, group_size=CQ_GROUP_SIZE):
- names = [n for n, _ in quant_leaf_names(params)]
- return _map_quant_leaves(
- params,
- lambda w, i: cq_ste(w, _bits_for(names[i], bits_map, default_bits), group_size))
-
-
-def kurtosis_penalty(params, group_size=None):
- g = _WEIGHT_GROUP if group_size is None else group_size
- ab = _ab_of(params)
- H = jnp.asarray(_cq_hadamard_np(g))
- terms = []
-
- def visit(path, leaf):
- if not _is_quant_leaf(path, leaf):
- return leaf
- s = ab.get(leaf_name(path)) if ab else None
- w = jnp.swapaxes(leaf, -1, -2) if _reduces_second_last(path) else leaf
- if s is not None:
- w = w * s["b"]
- pad = (-w.shape[-1]) % g
- wp = jnp.pad(w, [(0, 0)] * (w.ndim - 1) + [(0, pad)]) if pad else w
- rot = wp.reshape(*wp.shape[:-1], -1, g).astype(jnp.float32) @ H
- m2 = jnp.mean(rot ** 2, axis=-1)
- m4 = jnp.mean(rot ** 4, axis=-1)
- kurt = m4 / jnp.maximum(m2 ** 2, 1e-12)
- dev = (kurt - 3.0) ** 2
- terms.append((jnp.sum(dev), dev.size))
- return leaf
-
- jax.tree_util.tree_map_with_path(visit, params)
- return sum(t for t, _ in terms) / max(sum(n for _, n in terms), 1)
-
-
-def attractor_penalty(params, bits=None, group_size=None):
- b = _WEIGHT_BITS if bits is None else bits
- g = _WEIGHT_GROUP if group_size is None else group_size
- terms = []
-
- def fn(w, i):
- q = jax.lax.stop_gradient(cq_quantize(w, b, g))
- d = (w - q).astype(jnp.float32)
- ref = jax.lax.stop_gradient(w.astype(jnp.float32))
- terms.append((jnp.sum(d * d), jnp.sum(ref * ref)))
- return w
-
- _map_quant_leaves(params, fn)
- num = sum(t for t, _ in terms)
- den = sum(r for _, r in terms)
- return num / jnp.maximum(den, 1e-12)
diff --git a/needle/model/run.py b/needle/model/run.py
index 9fd447c..487fcd8 100644
--- a/needle/model/run.py
+++ b/needle/model/run.py
@@ -1,6 +1,5 @@
import os
import json
-import re
import sys
import jax
@@ -9,79 +8,20 @@ import numpy as np
from .tokenizer import get_tokenizer, BOS_ID, EOS_ID, PAD_ID
from .architecture import SimpleAttentionNetwork, TransformerConfig
-from .checkpoints import read_checkpoint, write_checkpoint
+from .checkpoints import read_checkpoint
CHECKPOINT_FORMAT_VERSION = 2
BUF_BUCKET = 128
_decode_fn_cache = {}
-def average_checkpoints(paths, out):
- acc, meta, steps = None, None, []
-
- def add(dst, src, scale):
- for k, v in src.items():
- if isinstance(v, dict):
- add(dst.setdefault(k, {}), v, scale)
- else:
- arr = np.asarray(v, np.float32) * scale
- dst[k] = dst[k] + arr if k in dst else arr
-
- scale = 1.0 / len(paths)
- for p in paths:
- ckpt = read_checkpoint(p)
- if ckpt.get("format_version") != CHECKPOINT_FORMAT_VERSION:
- raise ValueError(f"{p} is not a format-v{CHECKPOINT_FORMAT_VERSION} checkpoint")
- acc = acc if acc is not None else {}
- add(acc, ckpt["params"], scale)
- meta = ckpt
- steps.append(ckpt.get("step"))
-
- def to_fp16(t):
- return {k: to_fp16(v) if isinstance(v, dict) else v.astype(np.float16)
- for k, v in t.items()}
-
- write_checkpoint(out, {
- "format_version": CHECKPOINT_FORMAT_VERSION,
- "params": to_fp16(acc),
- "config": meta["config"],
- "step": meta.get("step"),
- "run": {**(meta.get("run") or {}), "averaged_from": steps},
- })
- print(f"Averaged {len(paths)} checkpoints (steps {steps}) -> {out}")
-
-
-def head_output_name(checkpoint, suffix):
- stem = os.path.splitext(os.path.basename(checkpoint))[0]
- stem = re.sub(r"_step_\d+$", "", stem)
- return f"{stem}_{suffix}"
-
def load_checkpoint(path, return_run=False):
if not os.path.exists(path):
- from huggingface_hub import hf_hub_download
- from huggingface_hub.errors import EntryNotFoundError
- from .tokenizer import HF_REPO
+ from ..agent import fetch
print(f" {'fetch':<9} {path} downloading from Hugging Face", flush=True)
- rel = os.path.normpath(path).replace(os.sep, "/")
- names = []
- if not os.path.isabs(path):
- names.append(rel if rel.startswith("checkpoints/") else "checkpoints/" + rel)
- flat = "checkpoints/" + os.path.basename(path)
- if flat not in names:
- names.append(flat)
- root = os.path.basename(path)
- if root not in names:
- names.append(root)
- err = None
- for name in names:
- try:
- path = hf_hub_download(HF_REPO, name, repo_type="model", local_dir=".")
- break
- except EntryNotFoundError as e:
- err = e
- else:
- raise err
+ dest_dir = os.path.dirname(path) or fetch.CHECKPOINT_PREFIX
+ path = fetch.fetch_checkpoint(os.path.basename(path), dest_dir, generation=3)
ckpt = read_checkpoint(path)
version = ckpt.get("format_version") if isinstance(ckpt, dict) else None
if version != CHECKPOINT_FORMAT_VERSION:
@@ -120,63 +60,6 @@ def _step_stats(step_logits):
return np.asarray(chosen), np.asarray(entropy)
-def batch_generate(model, params, tokenizer, prompts, max_new_tokens=96,
- return_signals=False):
- enc = [([BOS_ID] + tokenizer.encode(p))[:model.config.max_seq_len - 1] for p in prompts]
- plens = [len(e) for e in enc]
- buf_len = min(model.config.max_seq_len,
- -(-(max(plens) + max_new_tokens) // BUF_BUCKET) * BUF_BUCKET)
- B = len(prompts)
- buffer = np.full((B, buf_len), PAD_ID, dtype=np.int32)
- for i, e in enumerate(enc):
- buffer[i, :plens[i]] = e[:buf_len]
- decode_fn = _get_decode_fn(model, buf_len)
- params = jax.device_put(params)
-
- done = [plens[i] >= buf_len for i in range(B)]
- gen = [[] for _ in range(B)]
- probs = [[] for _ in range(B)]
- ents = [[] for _ in range(B)]
- for pos in range(min(plens) - 1, buf_len - 1):
- logits = decode_fn(params, jnp.asarray(buffer))
- step = logits[:, pos]
- nxt = np.asarray(jnp.argmax(step, axis=-1))
- chosen_p, chosen_h = _step_stats(step) if return_signals else (None, None)
- for i in range(B):
- if done[i] or pos < plens[i] - 1:
- continue
- t = int(nxt[i])
- if t == EOS_ID:
- done[i] = True
- continue
- gen[i].append(t)
- if return_signals:
- probs[i].append(float(chosen_p[i]))
- ents[i].append(float(chosen_h[i]))
- buffer[i, pos + 1] = t
- if len(gen[i]) >= max_new_tokens:
- done[i] = True
- if all(done):
- break
-
- texts = [tokenizer.decode(g) for g in gen]
- if not return_signals:
- return texts
-
- out = []
- for i in range(B):
- pr = probs[i]
- out.append({
- "text": texts[i],
- "tokens": gen[i],
- "token_probs": pr,
- "token_entropies": ents[i],
- "min_prob": min(pr) if pr else 0.0,
- "mean_logprob": float(np.mean(np.log(np.clip(pr, 1e-12, 1.0)))) if pr else float("-inf"),
- "max_entropy": max(ents[i]) if ents[i] else 0.0,
- })
- return out
-
def generate(model, params, tokenizer, prompt, max_new_tokens=256, temperature=0.0,
seed=0, stream=True):
diff --git a/needle/model/tokenizer.py b/needle/model/tokenizer.py
index 9200768..284ab5b 100644
--- a/needle/model/tokenizer.py
+++ b/needle/model/tokenizer.py
@@ -21,14 +21,6 @@ TOOL_RESULT_START = "<tool_result>"
TOOL_RESULT_END = "</tool_result>"
CONTEXT_START = "<context>"
CONTEXT_END = "</context>"
-IMAGE_START = "<image>"
-IMAGE_END = "</image>"
-SPEECH_START = "<speech>"
-SPEECH_END = "</speech>"
-TTS_START = "<tts>"
-TTS_END = "</tts>"
-IMAGEN_START = "<imagen>"
-IMAGEN_END = "</imagen>"
EXTRACT_START = "<extract>"
EXTRACT_END = "</extract>"
SCHEMA_START = "<schema>"
@@ -37,9 +29,7 @@ CHAT_MARKERS = [
IM_START, IM_END, THINK_START, THINK_END,
TOOLS_START, TOOLS_END, TOOL_CALL_START, TOOL_CALL_END,
TOOL_RESULT_START, TOOL_RESULT_END,
- CONTEXT_START, CONTEXT_END, IMAGE_START, IMAGE_END,
- SPEECH_START, SPEECH_END,
- TTS_START, TTS_END, IMAGEN_START, IMAGEN_END,
+ CONTEXT_START, CONTEXT_END,
EXTRACT_START, EXTRACT_END, SCHEMA_START, SCHEMA_END,
]
IM_START_ID, IM_END_ID, THINK_START_ID, THINK_END_ID = 4, 5, 6, 7
diff --git a/tests/conftest.py b/tests/conftest.py
index ca183dc..d3d7c19 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -4,7 +4,7 @@ import pickle
import pytest
-def _engine_available(generation=2):
+def _engine_available(generation=3):
"""Whether ``needle._library_path`` would find an engine without downloading.
Every location this looks in has to be one that function looks in, or the
@@ -72,3 +72,29 @@ def tiny_checkpoint_safetensors(tiny_checkpoint, tmp_path_factory):
path = tmp_path_factory.mktemp("ckpt") / "tiny.safetensors"
write_checkpoint(path, read_checkpoint(tiny_checkpoint))
return str(path)
+
+
[email protected](scope="session")
+def tiny_base_archive(tiny_checkpoint, tmp_path_factory):
+ from needle.model.export import write_export
+ from needle.model.run import load_checkpoint
+ from needle.model.tokenizer import get_tokenizer
+
+ params, config = load_checkpoint(tiny_checkpoint)
+ path = tmp_path_factory.mktemp("base") / "needle3.cact"
+ write_export(params, config, str(path), bits=4, tokenizer=get_tokenizer(config.vocab_size))
+ return str(path)
+
+
+def published_base(tiny_base_archive, monkeypatch):
+ from needle.agent import fetch
+
+ calls = []
+
+ def fake_fetch(generation=2, dest_dir=None, force=False):
+ calls.append((generation, force))
+ return tiny_base_archive
+
+ monkeypatch.setattr(fetch, "fetch_weights", fake_fetch)
+ return calls
diff --git a/tests/test_build.py b/tests/test_build.py
index ba18288..338b2c0 100644
--- a/tests/test_build.py
+++ b/tests/test_build.py
@@ -6,17 +6,16 @@ import pytest
pytestmark = pytest.mark.slow
-def _build_args(checkpoint, out, bits="4", lora=None):
- return types.SimpleNamespace(checkpoint=checkpoint, lora=lora, out=out,
- upload=False, bits=bits)
+def _build_args(checkpoint, out, lora=None):
+ return types.SimpleNamespace(checkpoint=checkpoint, lora=lora, out=out, upload=False)
-def test_build_exports_loadable_cact(tiny_checkpoint, tmp_path):
+def test_build_exports_loadable_cact(tiny_checkpoint, tmp_path, published_base):
from needle.model.finetune import build_main
from needle.model.export import read_export
out = str(tmp_path / "tiny.cact")
- build_main(_build_args(tiny_checkpoint, out, bits="4"))
+ build_main(_build_args(tiny_checkpoint, out))
assert os.path.exists(out)
assert os.path.getsize(out) > 0
@@ -24,33 +23,7 @@ def test_build_exports_loadable_cact(tiny_checkpoint, tmp_path):
assert header["num_tensors"] > 0
assert len(tensors) == header["num_tensors"]
assert any(isinstance(t, (bytes, bytearray)) for t in tensors)
-
-
-def test_build_layers_exports_the_rung(tiny_checkpoint, tmp_path):
- from needle.model.finetune import build_main
- from needle.model.export import read_export
-
- out = str(tmp_path / "tiny_3l.cact")
- args = _build_args(tiny_checkpoint, out, bits="4")
- args.layers = 3
- build_main(args)
- header, _ = read_export(out)
- assert header["num_layers"] == 3
-
-
-def test_rung_of_a_rung_is_the_trained_rung(tiny_checkpoint):
- import jax
- import numpy as np
- from needle.model.finetune import rung
- from needle.model.run import load_checkpoint
-
- params, config = load_checkpoint(tiny_checkpoint)
- p3, c3 = rung(params, config, 3)
- via_3, c_via = rung(p3, c3, 2)
- direct, c_direct = rung(params, config, 2)
- assert c3.ladder_order and vars(c_via) == vars(c_direct)
- for a, b in zip(jax.tree_util.tree_leaves(via_3), jax.tree_util.tree_leaves(direct)):
- assert np.array_equal(np.asarray(a), np.asarray(b))
+ assert published_base == [(3, True)]
def test_load_checkpoint_drops_the_mtp_block(tiny_checkpoint, tmp_path):
@@ -68,16 +41,6 @@ def test_load_checkpoint_drops_the_mtp_block(tiny_checkpoint, tmp_path):
assert "mtp_combine" not in params
-def test_build_at_two_bits(tiny_checkpoint, tmp_path):
- from needle.model.finetune import build_main
- from needle.model.export import read_export
-
- out = str(tmp_path / "tiny_w2.cact")
- build_main(_build_args(tiny_checkpoint, out, bits="2"))
- header, _ = read_export(out)
- assert header["num_tensors"] > 0
-
-
def test_export_round_trips_a_projection(tiny_checkpoint, tmp_path):
import pickle
import numpy as np
@@ -101,7 +64,7 @@ def test_export_round_trips_a_projection(tiny_checkpoint, tmp_path):
assert np.corrcoef(dequant.ravel(), original.ravel())[0, 1] > 0.9
-def test_build_from_safetensors_checkpoint(tiny_checkpoint, tiny_checkpoint_safetensors, tmp_path):
+def test_build_from_safetensors_checkpoint(tiny_checkpoint, tiny_checkpoint_safetensors, tmp_path, published_base):
from needle.model.finetune import build_main
from needle.model.export import read_export
from needle.model.run import load_checkpoint
@@ -114,6 +77,18 @@ def test_build_from_safetensors_checkpoint(tiny_checkpoint, tiny_checkpoint_safe
assert all(np.array_equal(np.asarray(x), np.asarray(y))
for x, y in zip(jax.tree_util.tree_leaves(a), jax.tree_util.tree_leaves(b)))
out = str(tmp_path / "from_safetensors.cact")
- build_main(_build_args(tiny_checkpoint_safetensors, out, bits="4"))
+ build_main(_build_args(tiny_checkpoint_safetensors, out))
header, _ = read_export(out)
assert header["num_tensors"] > 0
+
+
+def test_build_layers_exports_the_rung(tiny_checkpoint, tmp_path, published_base):
+ from needle.model.finetune import build_main
+ from needle.model.export import read_export
+
+ out = str(tmp_path / "rung.cact")
+ args = _build_args(tiny_checkpoint, out)
+ args.layers = 3
+ build_main(args)
+ header, _ = read_export(out)
+ assert header["num_layers"] == 3
diff --git a/tests/test_fetch.py b/tests/test_fetch.py
index d1f9041..eeaf876 100644
--- a/tests/test_fetch.py
+++ b/tests/test_fetch.py
@@ -21,6 +21,8 @@ def test_legacy_lib_override_cannot_capture_v3(tmp_path, monkeypatch):
assert needle._library_path(3) == str(v3)
monkeypatch.delenv("NEEDLE3_LIB_PATH")
+ monkeypatch.setenv("HOME", str(tmp_path))
+ monkeypatch.setenv("USERPROFILE", str(tmp_path))
monkeypatch.setattr(fetch, "fetch_library", lambda *args, **kwargs: str(v3))
assert needle._library_path(3) == str(v3)
@@ -33,7 +35,6 @@ def test_download_target_kinds():
assert _download_target("needle3") == ("base", 3)
assert _download_target("needle2.cact") == ("base", 2)
assert _download_target("needle3.safetensors") == ("checkpoint", "needle3.safetensors")
- assert _download_target("needle3_enterprise.safetensors") == ("checkpoint", "needle3_enterprise.safetensors")
assert _download_target("acme/tuned/model.cact") == ("hub", "acme/tuned/model.cact")
with pytest.raises(SystemExit, match="unknown download"):
_download_target("needle4")
@@ -127,11 +128,11 @@ def test_engine_gate_finds_the_cache_the_runtime_loads_from(tmp_path, monkeypatc
monkeypatch.setenv("HOME", str(tmp_path))
monkeypatch.setenv("USERPROFILE", str(tmp_path))
monkeypatch.delenv("NEEDLE_LIB_PATH", raising=False)
- monkeypatch.delenv("NEEDLE2_LIB_PATH", raising=False)
+ monkeypatch.delenv("NEEDLE3_LIB_PATH", raising=False)
assert not _engine_available()
- cache = tmp_path / ".cache" / "cactus-needle" / "v2" / fetch.engine_version(2)
+ cache = tmp_path / ".cache" / "cactus-needle" / "v3" / fetch.engine_version(3)
cache.mkdir(parents=True)
(cache / fetch._lib_name()).write_bytes(b"")
@@ -151,8 +152,8 @@ def test_engine_gate_honours_the_library_override(tmp_path, monkeypatch):
engine = tmp_path / fetch._lib_name()
engine.write_bytes(b"")
- monkeypatch.setenv("NEEDLE_LIB_PATH", str(engine))
+ monkeypatch.setenv("NEEDLE3_LIB_PATH", str(engine))
assert _engine_available()
- monkeypatch.setenv("NEEDLE_LIB_PATH", str(tmp_path / "gone"))
+ monkeypatch.setenv("NEEDLE3_LIB_PATH", str(tmp_path / "gone"))
assert not _engine_available()
diff --git a/tests/test_finetune.py b/tests/test_finetune.py
index 4bf0c91..8d73d38 100644
--- a/tests/test_finetune.py
+++ b/tests/test_finetune.py
@@ -24,11 +24,11 @@ def _write_data(path):
handle.write(json.dumps(row) + "\n")
-def _finetune_args(data, checkpoint, out, ckpt_dir, qat_bits="auto"):
+def _finetune_args(data, checkpoint, out, ckpt_dir):
return types.SimpleNamespace(
jsonl_path=str(data), checkpoint=checkpoint, epochs=1, batch_size=2,
lr=1e-3, lora_rank=4, lora_alpha=8.0, max_len=64, generate=0,
- model=None, checkpoint_dir=str(ckpt_dir), out=str(out), qat_bits=qat_bits)
+ model=None, checkpoint_dir=str(ckpt_dir), out=str(out))
def test_finetune_writes_adapter(tiny_checkpoint, tmp_path):
@@ -38,7 +38,7 @@ def test_finetune_writes_adapter(tiny_checkpoint, tmp_path):
_write_data(data)
out = tmp_path / "adapter.pkl"
progress = []
- finetune_local(_finetune_args(data, tiny_checkpoint, out, tmp_path / "ck", qat_bits=4),
+ finetune_local(_finetune_args(data, tiny_checkpoint, out, tmp_path / "ck"),
progress=progress.append)
assert any("loss" in m for m in progress)
@@ -49,19 +49,13 @@ def test_finetune_writes_adapter(tiny_checkpoint, tmp_path):
assert adapter["rank"] == 4
assert abs(adapter["scale"] - 2.0) < 1e-6
assert adapter["base"] == tiny_checkpoint
- assert adapter["qat_bits"] == 4
+ assert "qat_bits" not in adapter
assert adapter["lora"]
for value in adapter["lora"].values():
assert "A" in value and "B" in value
- from needle.model.finetune import build_main
- with pytest.raises(ValueError, match="trained for CQ W4"):
- build_main(types.SimpleNamespace(checkpoint=tiny_checkpoint, lora=str(out),
- out=str(tmp_path / "wrong.cact"),
- upload=False, bits="2"))
-
-def test_finetune_then_build_merges(tiny_checkpoint, tmp_path):
+def test_finetune_then_build_merges(tiny_checkpoint, tmp_path, published_base):
from needle.model.finetune import finetune_local, build_main
from needle.model.export import read_export
@@ -72,80 +66,13 @@ def test_finetune_then_build_merges(tiny_checkpoint, tmp_path):
out = str(tmp_path / "merged.cact")
build_main(types.SimpleNamespace(checkpoint=tiny_checkpoint, lora=str(adapter),
- out=out, upload=False, bits="4"))
+ out=out, upload=False))
assert os.path.exists(out)
header, _ = read_export(out)
assert header["num_tensors"] > 0
-def test_auto_qat_preserves_checkpoint_mixed_bit_map(tiny_checkpoint, tmp_path):
- from needle.model.finetune import finetune_local, build_main
- from needle.model.export import read_export
-
- with open(tiny_checkpoint, "rb") as handle:
- checkpoint = pickle.load(handle)
- bit_map = "embedding=4,mhc=4,default=2"
- checkpoint["config"]["weight_bits"] = bit_map
- mixed_checkpoint = tmp_path / "mixed.pkl"
- with open(mixed_checkpoint, "wb") as handle:
- pickle.dump(checkpoint, handle)
-
- data = tmp_path / "data.jsonl"
- _write_data(data)
- adapter_path = tmp_path / "mixed-adapter.pkl"
- progress = []
- finetune_local(_finetune_args(data, str(mixed_checkpoint), adapter_path,
- tmp_path / "ck"), progress=progress.append)
- assert any(f"mixed[{bit_map}]" in message for message in progress)
- with open(adapter_path, "rb") as handle:
- adapter = pickle.load(handle)
- assert adapter["qat_bits"] is None
- assert adapter["qat_bits_map"] == bit_map
-
- out = tmp_path / "mixed.cact"
- build_main(types.SimpleNamespace(checkpoint=str(mixed_checkpoint),
- lora=str(adapter_path), out=str(out),
- upload=False, bits=None))
- header, tensors = read_export(out)
- assert header["num_tensors"] == len(tensors)
-
- with pytest.raises(ValueError, match="mixed CQ bit map"):
- build_main(types.SimpleNamespace(checkpoint=str(mixed_checkpoint),
- lora=str(adapter_path),
- out=str(tmp_path / "wrong.cact"),
- upload=False, bits="4"))
-
-
-def test_finetune_on_a_rung_builds_that_depth(tiny_checkpoint, tmp_path):
- from needle.model.finetune import finetune_local, build_main
- from needle.model.checkpoints import read_adapter
- from needle.model.export import read_export
-
- data = tmp_path / "data.jsonl"
- _write_data(data)
- adapter_path = tmp_path / "rung.safetensors"
- args = _finetune_args(data, tiny_checkpoint, adapter_path, tmp_path / "ck", qat_bits="none")
- args.layers = 2
- progress = []
- finetune_local(args, progress=progress.append)
- assert any("2 layers" in message for message in progress)
- adapter = read_adapter(adapter_path)
- assert adapter["layers"] == 2
- assert all(v["A"].shape[0] == 2 for v in adapter["lora"].values())
-
- out = str(tmp_path / "rung.cact")
- build_main(types.SimpleNamespace(checkpoint=tiny_checkpoint, lora=str(adapter_path),
- out=out, upload=False, bits="4"))
- header, _ = read_export(out)
- assert header["num_layers"] == 2
-
- with pytest.raises(ValueError, match="2-layer rung"):
- build_main(types.SimpleNamespace(checkpoint=tiny_checkpoint, lora=str(adapter_path),
- out=str(tmp_path / "wrong.cact"), upload=False,
- bits="4", layers=3))
-
-
-def test_build_without_a_checkpoint_uses_the_adapter_base(tiny_checkpoint, tmp_path):
+def test_build_without_a_checkpoint_uses_the_adapter_base(tiny_checkpoint, tmp_path, published_base):
from needle.model.finetune import finetune_local, build_main
from needle.model.export import read_export
@@ -155,7 +82,7 @@ def test_build_without_a_checkpoint_uses_the_adapter_base(tiny_checkpoint, tmp_p
finetune_local(_finetune_args(data, tiny_checkpoint, adapter, tmp_path / "ck"))
out = str(tmp_path / "from_adapter.cact")
build_main(types.SimpleNamespace(checkpoint=None, lora=str(adapter), out=out,
- upload=False, bits="4"))
+ upload=False))
header, _ = read_export(out)
assert header["num_layers"] == 4
@@ -189,7 +116,7 @@ def test_finetune_adapter_records_realized_seed(tiny_checkpoint, tmp_path):
handle.write(json.dumps(row) + "\n")
out = tmp_path / "seeded-adapter.pkl"
- args = _finetune_args(data, tiny_checkpoint, out, tmp_path / "ck", qat_bits="none")
+ args = _finetune_args(data, tiny_checkpoint, out, tmp_path / "ck")
args.seed = 17
args.val_split = 0.0
finetune_local(args)
@@ -199,7 +126,7 @@ def test_finetune_adapter_records_realized_seed(tiny_checkpoint, tmp_path):
assert adapter["seed"] == 17
-def test_finetune_adapter_defaults_to_safetensors_and_builds(tiny_checkpoint_safetensors, tmp_path):
+def test_finetune_adapter_defaults_to_safetensors_and_builds(tiny_checkpoint_safetensors, tmp_path, published_base):
from needle.model.finetune import finetune_local, build_main
from needle.model.checkpoints import read_adapter
from needle.model.export import read_export
@@ -218,6 +145,6 @@ def test_finetune_adapter_defaults_to_safetensors_and_builds(tiny_checkpoint_saf
out = str(tmp_path / "merged_st.cact")
build_main(types.SimpleNamespace(checkpoint=tiny_checkpoint_safetensors,
- lora=str(adapter_path), out=out, upload=False, bits="4"))
+ lora=str(adapter_path), out=out, upload=False))
header, _ = read_export(out)
assert header["num_tensors"] > 0
diff --git a/tests/test_grounding.py b/tests/test_grounding.py
index df62771..a65d010 100644
--- a/tests/test_grounding.py
+++ b/tests/test_grounding.py
@@ -30,13 +30,17 @@ class _Stub:
@pytest.fixture
-def stub(monkeypatch):
+def stub(monkeypatch, tmp_path):
import needle
engine = _Stub()
- monkeypatch.setattr(needle, "_lib", lambda generation=2: engine)
- monkeypatch.setattr(needle, "_library_path", lambda generation=2: "/tmp/libneedle2")
+ base = tmp_path / "needle3.cact"
+ base.write_bytes((0x05E12A84).to_bytes(4, "little") + b"base weights")
+ monkeypatch.setattr(needle, "_lib", lambda generation=3: engine)
+ monkeypatch.setattr(needle, "_library_path", lambda generation=3: "/tmp/libneedle3")
+ monkeypatch.setattr(needle, "_base_weights_path", lambda generation: str(base))
monkeypatch.setattr(needle, "_active", {})
+ monkeypatch.setattr(needle, "_loaded_base", {})
return engine
diff --git a/tests/test_run.py b/tests/test_run.py
index 1f1911d..110cc05 100644
--- a/tests/test_run.py
+++ b/tests/test_run.py
@@ -58,16 +58,20 @@ def test_main_loads_tools_file_and_runs(tiny_checkpoint, tmp_path, capsys):
def test_missing_checkpoint_is_looked_up_under_checkpoints_then_at_the_repo_root(monkeypatch, tmp_path):
from huggingface_hub.errors import EntryNotFoundError
import huggingface_hub
+ from needle.agent import fetch
from needle.model import run
attempted = []
+ registered = []
- def fake_download(repo, name, repo_type, local_dir):
- attempted.append(name)
+ def fake_download(**kwargs):
+ attempted.append(kwargs["filename"])
raise EntryNotFoundError("missing")
monkeypatch.setattr(huggingface_hub, "hf_hub_download", fake_download)
+ monkeypatch.setattr(fetch, "_register_download", lambda generation: registered.append(generation))
monkeypatch.chdir(tmp_path)
- with pytest.raises(EntryNotFoundError):
+ with pytest.raises(FileNotFoundError):
run.load_checkpoint("needle3.safetensors")
assert attempted == ["checkpoints/needle3.safetensors", "needle3.safetensors"]
+ assert registered == [3]
diff --git a/tests/test_weights.py b/tests/test_weights.py
index f6f25fe..1805f23 100644
--- a/tests/test_weights.py
+++ b/tests/test_weights.py
@@ -40,17 +40,12 @@ class _WorkerStub:
type(self).next_pid += 1
self.calls.append(("worker_start", self.pid, library, weights))
- def complete(self, text, max_new_tokens, audio=None):
- if audio:
- self.calls.append(("worker_complete_audio", self.pid, text,
- audio))
- else:
- self.calls.append(("worker_complete", self.pid, text))
+ def complete(self, text, max_new_tokens):
+ self.calls.append(("worker_complete", self.pid, text))
return ENVELOPE.decode("utf-8")
- def embed(self, text, audio=None):
- self.calls.append(("worker_embed", self.pid, text,
- audio))
+ def embed(self, text):
+ self.calls.append(("worker_embed", self.pid, text))
return [0.6, 0.8]
def reset(self):
@@ -61,7 +56,7 @@ class _WorkerStub:
@pytest.fixture
-def engine(monkeypatch):
+def engine(monkeypatch, tmp_path):
import needle
calls = []
@@ -76,6 +71,10 @@ def engine(monkeypatch):
_WorkerStub(calls, library, weights, system, tools, tool_index,
buffer_size, generation))
monkeypatch.setattr(needle, "_active", {})
+ monkeypatch.setattr(needle, "_loaded_base", {})
+ base = tmp_path / "needle3.cact"
+ base.write_bytes((0x05E12A84).to_bytes(4, "little") + b"base weights")
+ monkeypatch.setattr(needle, "_base_weights_path", lambda generation: str(base))
return calls
@@ -101,35 +100,45 @@ def _tuned_agent(path):
return needle.Needle(tools="[]", weights=path)
-def test_generation_3_without_weights_runs_the_base_archive(engine, tuned_v3, monkeypatch):
+def test_generation_3_without_weights_loads_the_base_archive_in_process(engine):
import needle
- monkeypatch.setattr(needle, "_base_weights_path", lambda generation: tuned_v3)
with warnings.catch_warnings():
warnings.simplefilter("error")
agent = needle.Needle(tools="[]", generation=3)
- assert agent._generation == 3 and agent._tuned is False
+ assert agent._generation == 3 and agent._tuned is False and agent._weights is None
response = agent.complete("hello")
assert response.get("confidence", "kept") is not None
- assert any(call[0] == "worker_start" and call[3] == tuned_v3
- for call in engine if isinstance(call, tuple))
+ assert engine.count("load") == 1 and "complete" in engine
+ assert not any(call[0] == "worker_start" for call in engine if isinstance(call, tuple))
+ needle.Needle(tools="[]", generation=3).complete("again")
+ assert engine.count("load") == 1
assert agent._track_props()["tuned"] is False
-def test_default_agent_is_still_the_embedded_needle_2(engine):
+def test_default_agent_is_needle_3(engine):
import needle
agent = needle.Needle(tools="[]")
+ assert agent._generation == 3 and agent._weights is None and agent._tuned is False
+ agent.complete("hello")
+ assert engine.count("load") == 1 and "complete" in engine
+
+
+def test_generation_2_runs_its_embedded_engine(engine):
+ import needle
+
+ agent = needle.Needle(tools="[]", generation=2)
assert agent._generation == 2 and agent._weights is None
agent.complete("hello")
- assert "complete" in engine
+ assert engine.count("load") == 0 and "complete" in engine
def test_base_agent_and_tuned_worker_coexist(engine, tuned):
import needle
tuned_agent = _tuned_agent(tuned)
- base = needle.Needle(tools="[]")
+ base = needle.Needle(tools="[]", generation=2)
tuned_agent.complete("tuned")
base.complete("base")
assert any(call[0] == "worker_complete" for call in engine if isinstance(call, tuple))
@@ -139,7 +148,7 @@ def test_base_agent_and_tuned_worker_coexist(engine, tuned):
def test_tuned_worker_does_not_rebind_existing_base_agent(engine, tuned):
import needle
- base = needle.Needle(tools="[]")
+ base = needle.Needle(tools="[]", generation=2)
base.complete("hello")
tuned_agent = _tuned_agent(tuned)
tuned_agent.complete("hello")
@@ -188,14 +197,10 @@ def test_v2_and_v3_weights_route_to_independent_engines(engine, tuned, tuned_v3,
assert routed == [2, 3]
-def test_v3_routes_audio_and_embeddings_to_its_worker(engine, tuned_v3):
+def test_v3_routes_embeddings_to_its_worker(engine, tuned_v3):
agent = _tuned_agent(tuned_v3)
- wav = b"RIFFordinary wav bytes"
- agent.complete("transcribe", audio=wav)
- assert agent.embed("schema", audio=wav) == [0.6, 0.8]
- payload = {"data": wav, "format": 1, "sample_rate": 0, "channels": 1}
- assert ("worker_complete_audio", agent._worker.pid, "transcribe", payload) in engine
- assert ("worker_embed", agent._worker.pid, "schema", payload) in engine
+ assert agent.embed("schema") == [0.6, 0.8]
+ assert ("worker_embed", agent._worker.pid, "schema") in engine
def test_unknown_archive_tag_is_rejected_before_loading(engine, tmp_path):