Skip to content

Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.

fix: update documentation for API and finetuning; enhance compatibility for Apple Silicon and adjust dependencies

HenryNdubuakufb381b75ccdb
diff --git a/README.md b/README.md
index 5b97d6a..76e15ac 100644
--- a/README.md
+++ b/README.md
@@ -30,7 +30,7 @@ Each block carries its update rule. Here x̂ is the RMS-normalised flattening of
pip install cactus-needle
```
-Needle reads your tool descriptions to decide what to call and how to fill arguments, so describing them well is the whole game. You can do it three ways, from least to most control.
+Needle reads your tool descriptions to decide what to call and how to fill arguments, so describing them well is the whole game.
**Simple**: decorate a function. The signature gives the argument types, the docstring is the tool description, and `run()` completes the loop: model picks the call, Needle executes your function, feeds the result back, and returns the final response with the executed tool results attached as `results`.
@@ -47,42 +47,6 @@ print(agent.run("what's it like in Lagos right now?")["results"])
# [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]
```
-**Medium**: describe each argument and offer choices. Needle reads a Google-style `Args:` block for per-parameter descriptions; a default makes an argument optional; a `Literal` becomes a fixed set the model must choose from (it cannot emit anything else).
-
-```python
-from typing import Literal
-
-def set_thermostat(temperature: int, mode: Literal["heat", "cool", "auto"] = "auto"):
- """Set the thermostat.
-
- Args:
- temperature: target temperature in Celsius
- mode: heating strategy to use
- """
- return {"temperature": temperature, "mode": mode}
-
-agent = needle.Needle(tools=[set_thermostat])
-agent.run("make it 21 and cool the room")
-```
-
-**Advanced**: constrain the values with `needle.Field`, attached inline via `Annotated`. Ranges, patterns, lengths, and item counts are compiled into the decode grammar, so the model can only ever emit values that satisfy them.
-
-```python
-from typing import Annotated
-
-def send_money(
- amount: Annotated[float, needle.Field(gt=0, le=10000, description="USD, up to 10,000")],
- to: Annotated[str, needle.Field(pattern=r"^@[a-z0-9_]+$", description="recipient handle")],
- memo: Annotated[str, needle.Field(max_length=80)] = "",
-):
- "Send money to a handle."
- return {"sent": amount, "to": to}
-```
-
-`Field` supports `description`, `enum`, `const`, `ge`/`le`/`gt`/`lt`, `multiple_of`, `min_length`/`max_length`, `pattern`, `format`, `min_items`/`max_items`, and `unique_items`.
-
**Extraction**: to pull structured data out of text, declare the shape and call `extract()`. Pass a Pydantic model and you get a typed object back.
```python
@@ -97,51 +61,7 @@ invoice = needle.extract("Invoice from Acme Corp, $1,200.00, due 2026-09-01", In
print(invoice.vendor, invoice.total) # -> Acme Corp 1200.0
```
-**By hand** - the decorator just builds a JSON schema; you can pass that schema directly, which is exactly what Needle consumes. This is how you set descriptions and constraints without the decorator:
-
-```python
-tools = [{
- "name": "set_lights",
- "description": "Turn a room's lights on or off and set brightness",
- "parameters": {
- "type": "object",
- "properties": {
- "room": {"type": "string", "description": "which room to control"},
- "on": {"type": "boolean"},
- "brightness": {"type": "integer", "minimum": 0, "maximum": 100},
- },
- "required": ["room", "on"],
- },
-}]
-agent = needle.Needle(tools=tools)
-```
-
-Prefer to drive the loop yourself instead of `run()`? `complete()` returns the raw call and you execute it:
-
-```python
-import json
-response = agent.complete("dim the living room to 30")
-if response["type"] == "call":
- result = set_lights(**response["function_calls"][0]["arguments"])
- response = agent.complete(json.dumps(result)) # feed the result back
-```
-
-With a large catalogue, persist tool embeddings across runs with `needle.Needle(tools=..., tool_index_path="tools.idx")`. Every turn returns one JSON object:
-
-```json
-{
- "type": "call",
- "success": true,
- "error": null,
- "error_code": null,
- "function_calls": [ { "name": "set_lights", "arguments": { "room": "living room", "on": true, "brightness": 30 } } ],
- "reasoning": "'living room' -> room; 'dim' -> on true, brightness 30",
- "confidence": 0.94,
- "prefill_tps": 4300.0,
- "decode_tps": 850.0,
- "peak_ram_mb": 28.5
-}
-```
+Per argument descriptions and choices, value constraints compiled into the decode grammar, raw JSON schemas, driving the loop with `complete()`, the response contract, system facts, tool retrieval, and confidence gating are all covered in [doc/apis.md](doc/apis.md).
## Playground
@@ -154,62 +74,6 @@ needle playground --weights my.cact # a tuned model
The server downloads and initializes the model before serving, so the first query is instant. The **Finetune on these tools** button runs the fine-tuning pipeline below from the UI and hands back a downloadable `.cact`.
-## Behaviour
-
-Needle solves every problem as a function call. The context declares what may be called; the model answers with calls. Performing an action and extracting structured data are the same operation, the only difference is what you declare.
-
-- A request no declared tool can serve is refused with the empty call `[]`. That is the whole contract for off-topic input; there is no free-text fallback.
-- Arguments contain only values evidenced by the input. An optional field with no evidence is omitted, not guessed; omission is the field-level `[]`.
-- `reasoning` is the model's short derivation of each argument from its source span (`'ten minutes' -> minutes 10`). It is generated unconstrained; only the call itself is grammar-constrained, so the JSON cannot be malformed while the derivation stays legible.
-- After you execute a call, pass the result back as the next `complete()`. The model continues from it, and later arguments may depend on earlier results: `search_for_contact` first, then `send_instant_message` with the returned `contact_id`. A final `"type": "respond"` with empty `function_calls` signals the loop is done; the answer is the tool results themselves, which `run()` collects on the final response as `results`. No free text is generated.
-- A session shares one toolset. Later turns are bare queries against the same tools; `reset()` rewinds the conversation and keeps the tools loaded.
-
-## Extraction
-
-Extraction is not a separate mode - it is tool calling with one tool. Declare the record as the only schema and pass the content where the query goes; the returned call's `arguments` are the extracted fields. With one declared tool the grammar admits exactly one call of that name, so schema conformance is guaranteed rather than requested. Use the `extract()` helper for a typed result (shown in Quickstart), or pass a plain schema and read the call:
-
-```python
-receipt = [{
- "name": "receipt",
- "description": "A purchase receipt shared as text",
- "parameters": {
- "type": "object",
- "properties": {
- "merchant": {"type": "string"},
- "total": {"type": "number"},
- "currency": {"type": "string"},
- "line_items": {"type": "array", "items": {"type": "object"}},
- },
- "required": ["merchant", "total"],
- },
-}]
-agent = needle.Needle(tools=receipt)
-print(agent.complete("GreenMart receipt: oat milk 3.50, total 7.75 paid by visa")["function_calls"])
-# -> [{'name': 'receipt', 'arguments': {'merchant': 'GreenMart', 'total': 7.75}}]
-```
-
-Because it is the same operation, everything else applies unchanged: `confidence` gates the extraction, unsupported input returns the empty call `[]`, and fine-tuning uses the same data format (the record as the tool, the passage as the query).
-
-## System facts
-
-An optional system turn carries environment state as facts, never instructions:
-
-```python
-agent = needle.Needle(tools=tools, system="date: 2026-07-21 Tue 14:30; locale: en-US; device: phone; battery: 62%")
-```
-
-Recognized keys are `date`, `locale`, `device`, `battery`, `network`, `location`, `user`, and `assistant`. The model resolves relative language against them: "tomorrow at 7" becomes an absolute time only when a `date:` fact licenses it, otherwise the human phrase passes through verbatim. `assistant:` declares the identity the model binds to. Needle trains with and without the turn, so omitting it is safe; instructions placed there do not steer the model.
-
-## Tool retrieval
-
-Five or fewer declared tools render directly. Above that, retrieval engages: at init every tool schema is embedded once by a built-in contrastive head, each turn embeds the query, and only the five highest-scoring tools enter the context, with the grammar rebuilt over just that subset. An unselected tool is unreachable, not merely unlikely. `tool_index_path` persists the embeddings on disk, keyed by a fingerprint over the schemas and the model; a matching fingerprint loads instantly, a changed schema re-embeds only what changed.
-
-## Confidence
-
-The `confidence` field is the minimum of two signals: a calibrated post-hoc head that scores the full prompt plus the call the model just produced, and the decoding probability of the call tokens. A call is accepted only when both agree, so the failure mode is escalation, not wrong execution. The contract: pick a threshold for your product, act at or above it, re-ask or route to a bigger model below it. Off-topic requests return the empty call `[]`.
-
-Calibration holds for the base model only. Fine-tuning does not update the head, so an agent running tuned weights reports `confidence` as `None` and warns once at construction.
-
## Fine-tuning
Needle fine-tunes with LoRA on the frozen base and merges the adapter at export, so a run is cheap and the tuned model is still a single `.cact` that runs on the same engine. The workflow is: (optionally) synthesize data, LoRA fine-tune, then build a tuned `.cact`. See [doc/finetuning.md](doc/finetuning.md) for dataset sizing, reading the loss curve, and troubleshooting.
@@ -245,6 +109,12 @@ Training is plain JAX and runs on any accelerator jax supports. On an NVIDIA mac
pip install "cactus-needle[gpu]"
```
+On Apple Silicon the `metal` extra trains on the GPU:
+
+```sh
+pip install "cactus-needle[metal]"
+```
+
**3. Build a tuned `.cact`.** Merge the adapter into the base and quantize. The base auto-downloads if absent.
```sh
diff --git a/doc/apis.md b/doc/apis.md
new file mode 100644
index 0000000..54a9424
--- /dev/null
+++ b/doc/apis.md
@@ -0,0 +1,153 @@
+# API
+
+| Call | What it does |
+| --- | --- |
+| `needle.Needle(tools=None, system=None, weights=None, tool_index_path=None, buffer_size=65536)` | Create an agent bound to one toolset. `tools` takes decorated functions, Pydantic models, raw JSON schema dicts, or a JSON string. `system` carries environment facts. `weights` loads a tuned `.cact`. `tool_index_path` persists tool embeddings for large catalogues. |
+| `agent.run(query, max_steps=8, max_new_tokens=256)` | Full agentic loop: the model picks calls, Needle executes your Python functions and feeds results back, and the final response carries the executed tool results as `results`. |
+| `agent.complete(text, max_new_tokens=256)` | One turn. You execute the call and feed the result back yourself via the next `complete(...)`. |
+| `agent.reset()` | Rewind the conversation, keep the tools loaded. |
+| `needle.tool` | Decorator that turns a function into a tool schema (attached as `fn._needle_tool`). |
+| `needle.Field(...)` | Per argument constraints, attached inline with `typing.Annotated` or passed as a default. |
+| `needle.extract(text, schema, system=None, max_new_tokens=256)` | One shot extraction. Returns a Pydantic instance if `schema` is a model, else a dict, or `None` if nothing matched. |
+
+## Declaring tools
+
+**Describe each argument and offer choices.** Needle reads a Google-style `Args:` block for per-parameter descriptions; a default makes an argument optional; a `Literal` becomes a fixed set the model must choose from (it cannot emit anything else).
+
+```python
+from typing import Literal
+
+def set_thermostat(temperature: int, mode: Literal["heat", "cool", "auto"] = "auto"):
+ """Set the thermostat.
+
+ Args:
+ temperature: target temperature in Celsius
+ mode: heating strategy to use
+ """
+ return {"temperature": temperature, "mode": mode}
+
+agent = needle.Needle(tools=[set_thermostat])
+agent.run("make it 21 and cool the room")
+```
+
+**Constrain the values** with `needle.Field`, attached inline via `Annotated`. Ranges, patterns, lengths, and item counts are compiled into the decode grammar, so the model can only ever emit values that satisfy them.
+
+```python
+from typing import Annotated
+
+def send_money(
+ amount: Annotated[float, needle.Field(gt=0, le=10000, description="USD, up to 10,000")],
+ to: Annotated[str, needle.Field(pattern=r"^@[a-z0-9_]+$", description="recipient handle")],
+ memo: Annotated[str, needle.Field(max_length=80)] = "",
+):
+ "Send money to a handle."
+ return {"sent": amount, "to": to}
+```
+
+`Field` supports `description`, `enum`, `const`, `ge`/`le`/`gt`/`lt`, `multiple_of`, `min_length`/`max_length`, `pattern`, `format`, `min_items`/`max_items`, and `unique_items`.
+
+**By hand.** The decorator just builds a JSON schema; you can pass that schema directly, which is exactly what Needle consumes. This is how you set descriptions and constraints without the decorator:
+
+```python
+tools = [{
+ "name": "set_lights",
+ "description": "Turn a room's lights on or off and set brightness",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "room": {"type": "string", "description": "which room to control"},
+ "on": {"type": "boolean"},
+ "brightness": {"type": "integer", "minimum": 0, "maximum": 100},
+ },
+ "required": ["room", "on"],
+ },
+}]
+agent = needle.Needle(tools=tools)
+```
+
+## Driving the loop
+
+Prefer to drive the loop yourself instead of `run()`? `complete()` returns the raw call and you execute it:
+
+```python
+import json
+response = agent.complete("dim the living room to 30")
+if response["type"] == "call":
+ result = set_lights(**response["function_calls"][0]["arguments"])
+ response = agent.complete(json.dumps(result)) # feed the result back
+```
+
+Every turn returns one JSON object:
+
+```json
+{
+ "type": "call",
+ "success": true,
+ "error": null,
+ "error_code": null,
+ "function_calls": [ { "name": "set_lights", "arguments": { "room": "living room", "on": true, "brightness": 30 } } ],
+ "reasoning": "'living room' -> room; 'dim' -> on true, brightness 30",
+ "confidence": 0.94,
+ "prefill_tps": 4300.0,
+ "decode_tps": 850.0,
+ "peak_ram_mb": 28.5
+}
+```
+
+## Behaviour
+
+Needle solves every problem as a function call. The context declares what may be called; the model answers with calls. Performing an action and extracting structured data are the same operation, the only difference is what you declare.
+
+- A request no declared tool can serve is refused with the empty call `[]`. That is the whole contract for off-topic input; there is no free-text fallback.
+- Arguments contain only values evidenced by the input. An optional field with no evidence is omitted, not guessed; omission is the field-level `[]`.
+- `reasoning` is the model's short derivation of each argument from its source span (`'ten minutes' -> minutes 10`). It is generated unconstrained; only the call itself is grammar-constrained, so the JSON cannot be malformed while the derivation stays legible.
+- After you execute a call, pass the result back as the next `complete()`. The model continues from it, and later arguments may depend on earlier results: `search_for_contact` first, then `send_instant_message` with the returned `contact_id`. A final `"type": "respond"` with empty `function_calls` signals the loop is done; the answer is the tool results themselves, which `run()` collects on the final response as `results`. No free text is generated.
+- A session shares one toolset. Later turns are bare queries against the same tools; `reset()` rewinds the conversation and keeps the tools loaded.
+
+## Extraction
+
+Extraction is not a separate mode - it is tool calling with one tool. Declare the record as the only schema and pass the content where the query goes; the returned call's `arguments` are the extracted fields. With one declared tool the grammar admits exactly one call of that name, so schema conformance is guaranteed rather than requested. Use the `extract()` helper for a typed result, or pass a plain schema and read the call:
+
+```python
+receipt = [{
+ "name": "receipt",
+ "description": "A purchase receipt shared as text",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "merchant": {"type": "string"},
+ "total": {"type": "number"},
+ "currency": {"type": "string"},
+ "line_items": {"type": "array", "items": {"type": "object"}},
+ },
+ "required": ["merchant", "total"],
+ },
+}]
+agent = needle.Needle(tools=receipt)
+print(agent.complete("GreenMart receipt: oat milk 3.50, total 7.75 paid by visa")["function_calls"])
+# -> [{'name': 'receipt', 'arguments': {'merchant': 'GreenMart', 'total': 7.75}}]
+```
+
+Because it is the same operation, everything else applies unchanged: `confidence` gates the extraction, unsupported input returns the empty call `[]`, and fine-tuning uses the same data format (the record as the tool, the passage as the query).
+
+## System facts
+
+An optional system turn carries environment state as facts, never instructions:
+
+```python
+agent = needle.Needle(tools=tools, system="date: 2026-07-21 Tue 14:30; locale: en-US; device: phone; battery: 62%")
+```
+
+Recognized keys are `date`, `locale`, `device`, `battery`, `network`, `location`, `user`, and `assistant`. The model resolves relative language against them: "tomorrow at 7" becomes an absolute time only when a `date:` fact licenses it, otherwise the human phrase passes through verbatim. `assistant:` declares the identity the model binds to. Needle trains with and without the turn, so omitting it is safe; instructions placed there do not steer the model.
+
+## Tool retrieval
+
+Five or fewer declared tools render directly. Above that, retrieval engages: at init every tool schema is embedded once by a built-in contrastive head, each turn embeds the query, and only the five highest-scoring tools enter the context, with the grammar rebuilt over just that subset. An unselected tool is unreachable, not merely unlikely. `tool_index_path` persists the embeddings on disk, keyed by a fingerprint over the schemas and the model; a matching fingerprint loads instantly, a changed schema re-embeds only what changed.
+
+## Confidence
+
+The `confidence` field is the minimum of two signals: a calibrated post-hoc head that scores the full prompt plus the call the model just produced, and the decoding probability of the call tokens. A call is accepted only when both agree, so the failure mode is escalation, not wrong execution. The contract: pick a threshold for your product, act at or above it, re-ask or route to a bigger model below it. Off-topic requests return the empty call `[]`.
+
+Calibration holds for the base model only. Fine-tuning does not update the head, so an agent running tuned weights reports `confidence` as `None` and warns once at construction.
diff --git a/doc/finetuning.md b/doc/finetuning.md
index 75575b1..c098369 100644
--- a/doc/finetuning.md
+++ b/doc/finetuning.md
@@ -47,7 +47,13 @@ Training is plain JAX, so it runs on any accelerator jax supports. On an NVIDIA
pip install "cactus-needle[gpu]"
```
-Apple GPUs are not currently supported: the jax metal plugin (0.1.1) fails on basic operations with current jax. Apple Silicon trains on CPU, which handles small LoRA runs comfortably now that padding fits the data.
+Apple GPUs train through the jax metal plugin, which does not work past jax 0.4.38, so the `metal` extra pins an older stack:
+
+```sh
+pip install "cactus-needle[metal]"
+```
+
+Needle detects the Metal backend and adapts automatically: manual attention, no rematerialisation, unrolled layer stack, and the `ENABLE_PJRT_COMPATIBILITY` variable the plugin requires is set for you. Measured on an M5 Max: 0.71 seconds per step against 2.90 on CPU at the same shape, about 4 times faster, with a one time compile of about 23 seconds. Training runs in float32 on every backend.
## Reading the loss
diff --git a/needle/model/architecture.py b/needle/model/architecture.py
index 16371d5..8d35d51 100644
--- a/needle/model/architecture.py
+++ b/needle/model/architecture.py
@@ -78,6 +78,8 @@ class TransformerConfig:
kv_bits: int = 8
act_bits: int = 8
weight_bits: str = ""
+ remat: bool = True
+ scan_unroll: int = 1
def __init__(self, **kwargs):
valid = {f.name for f in self.__dataclass_fields__.values()}
@@ -406,11 +408,12 @@ class Stack(nn.Module):
x = jnp.broadcast_to(x[:, :, None, :], (*x.shape[:2], n, x.shape[-1]))
ScanBlock = nn.scan(
- nn.remat(_ScanBody),
+ nn.remat(_ScanBody) if cfg.remat else _ScanBody,
variable_axes={"params": 0},
split_rngs={"params": True},
length=cfg.num_layers,
in_axes=(0, nn.broadcast, nn.broadcast, nn.broadcast, nn.broadcast),
+ unroll=cfg.scan_unroll,
)
x, hidden = ScanBlock(
cfg.num_heads, cfg.num_kv_heads, cfg.d_model, cfg.num_layers, dt,
diff --git a/needle/model/finetune.py b/needle/model/finetune.py
index 977e8bd..e76d986 100644
--- a/needle/model/finetune.py
+++ b/needle/model/finetune.py
@@ -2,9 +2,16 @@ import concurrent.futures
import json
import os
import pickle
+import sys
import urllib.request
from concurrent.futures import ThreadPoolExecutor
+# The jax metal plugin refuses to load against a newer PJRT API without this;
+# it must be in the environment before jax initialises its backend, and this
+# module is imported before any jax import on every training path.
+if sys.platform == "darwin":
+ os.environ.setdefault("ENABLE_PJRT_COMPATIBILITY", "1")
+
import numpy as np
from .tokenizer import (
@@ -303,7 +310,15 @@ def finetune_local(args, progress=None):
workers=getattr(args, "workers", 8))
params, config = load_checkpoint(base_path)
+ config.dtype = "float32"
+ params = jax.tree.map(lambda a: np.asarray(a).astype(np.float32), params)
+ backend = jax.default_backend().lower()
+ if backend == "metal":
+ config.flash = False
+ config.remat = False
+ config.scan_unroll = config.num_layers
params = jax.device_put(params)
+ emit(f" {'backend':<9} {backend} float32")
tokenizer = get_tokenizer(config.vocab_size)
max_len = fit_max_len(data_path, tokenizer, args.max_len)
seqs, masks = load_jsonl(data_path, tokenizer, max_len)
diff --git a/pyproject.toml b/pyproject.toml
index 5595fe0..12e6af4 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -10,13 +10,14 @@ dependencies = [
"numpy",
"jax",
"jaxlib",
- "flax>=0.12.8",
+ "flax>=0.10.2",
"optax",
"sentencepiece",
]
[project.optional-dependencies]
gpu = ["jax[cuda12]"]
+metal = ["jax==0.4.38", "jaxlib==0.4.38", "jax-metal", "flax==0.10.2", "optax==0.2.4"]
test = ["pytest", "pydantic"]
[project.scripts]