Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.
Add architecture and frontier assets; remove outdated configuration and documentation files
HenryNdubuaku3dd2e5a84ecf
diff --git a/README.md b/README.mdindex f4075cc..ec1dcfe 100644--- a/README.md+++ b/README.md@@ -1,192 +1,296 @@-# Needle+-<img src="assets/banner.png" alt="Logo" style="border-radius: 30px; width: 100%;">+# Needle 2-A 26m parameter "[Simple Attention Network](https://arxiv.org/abs/2607.18363)" for function calling that you can even finetune locally on your Mac/PC.-In production, Needle runs on [Cactus](https://github.com/cactus-compute/cactus) at 6000 toks/sec prefill and 1200 decode speed.-Weights are fully open on [Cactus-Compute/needle](https://huggingface.co/Cactus-Compute/needle), as well as the dataset generation.+Needle 2 is an open 45M-parameter model for tool calling, device use and structured extraction. The whole model is a single 14MB binary that runs a full session in 28MB of RAM. It is built on our Simple Attention Network findings, compressed to CQ2-bit with Cactus Quants, and baked into its own engine. On the benchmarks below, Needle 2 trades wins with other small models like FunctionGemma 270M, LFM2.5 230M and Apple FM, at 5x to 70x smaller, and 2 bits against their f16. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges 300-700 on sub-$200 phones such as the Samsung A-Series. With a peak session RAM around 28MB, Needle reaches microcontrollers like the ESP32-P4; others have reported running it on an ESP32-S3 in about 11MB.-```-d=512, 8H/4KV, BPE=8192- ┌──────────────┐- │ Tool Call │- └──────┬───────┘- ┌┴──────────┐- │ Softmax │- └─────┬─────┘- ┌─────┴─────┐- │ Linear (T)│ ← tied- └─────┬─────┘- ┌─────┴─────┐- │ ZCRMSNorm │- └─────┬─────┘- ┌────────┴────────┐- │ Decoder x 8 │- │┌───────────────┐│- ││ ZCRMSNorm ││- ││ Masked Self ││- ││ Attn + RoPE ││- ││ Gated Residual││- │├───────────────┤│- ┌──────────────┐ ││ ZCRMSNorm ││- │ Encoder x 12 │──────────────────────▶Cross Attn ││- │ │ ││ Gated Residual││- │ ┌──────────┐ │ │└───────────────┘│- │ │ZCRMSNorm │ │ └────────┬────────┘- │ │Self Attn │ │ ┌─────┴─────┐- │ │ GQA+RoPE │ │ │ Embedding │ ← shared- │ │Gated Res │ │ └─────┬─────┘- │ │ │ │ ┌───────┴───────-┐- │ │ (no FFN) │ │ │[EOS]<tool_call>│- │ └──────────┘ │ │ + answer │- │ │ └───────────────-┘- └──────┬───────┘- │- ┌────┴──────┐- │ Embedding │- └────┬──────┘- │- ┌────┴──────┐- │ Text │- │ query │- └───────────┘-```+- **Self-contained**: model baked into the binary, no runtime, no downloads, no network.+- **Runs everywhere**: ARM64, x86-64, ARMv7, RISC-V, and WebAssembly, on Apple, Windows, Linux, Android, Raspberry Pi.+- **Simple contract**: tool calls come back as structured data, text in, JSON out; a byte-level grammar compiled from your schemas constrains every token.+- **Confidence-gated**: every response carries a calibrated confidence score from a learned head; set a threshold, act above it, escalate below it.+- **Tool retrieval**: declare a large catalogue and a built-in retrieval head renders only the top five tools per turn, with the grammar constrained to that subset.+- **Bounded memory**: a 256-token sliding window with the tools pinned as KV sinks, so total memory stays near 28MB no matter how long the conversation runs.-- Pretrained on 16 TPU v6e for 200B tokens (27hrs).-- Post-trained on 2B tokens of single-shot function call dataset (45mins).+Weights on: [github.com/cactus-compute/needle](Cactus-Compute/needle2).-Needle is an experimental run for Simple Attention Networks, geared at redefining tiny AI for consumer devices (phones, watches, glasses...).-So while it beats FunctionGemma-270m, Qwen-0.6B, Graninte-350m, LFM2.5-350m on single-shot function call for personal AI,-Those model are have more scope/capacity and excel in conversational settings. Also, small models can be finicky.-Please use the UI in the next section to test on your own tools, and finetune accordingly, at the click of a button.+-## Quickstart+## Simple Attention Network-### macOS / Linux+Needle 2 is a Simple Attention Network, our dense small-model recipe: a Hadamard MLP in place of the FFN, GQA attention, engram key-value memory, and multi-lane hyper-connections. See the paper for the design and ablations: [arXiv:2607.18363](https://arxiv.org/abs/2607.18363).-```bash-git clone https://github.com/cactus-compute/needle.git-cd needle && source ./setup-needle playground-```+++Each block carries its update rule. Here x̂ is the RMS-normalised flattening of the four residual streams, H the orthonormal Walsh-Hadamard transform (a fixed matrix, applied in n log n time with no weights to read), (kₜ, vₜ) rows gathered from hashed n-gram tables, and P the doubly-stochastic normalisation of the routing logits A, computed by Sinkhorn iteration; a, b, g and all σ-gates are learned and input-dependent. Both attention and MLP residuals are sandwich-normed and gated, the engram sites fire at two layers, and decoding is constrained by a byte-level grammar compiled from the declared schemas.-### Windows+## Quickstart with Python-```bat-git clone https://github.com/cactus-compute/needle.git-cd needle-.\setup.bat+```sh+pip install cactus-needle```-> **Note:** `setup.bat` creates a `.venv`, installs all dependencies (including JAX in CPU mode), and prompts for an optional W&B API key.+Needle reads your tool descriptions to decide what to call and how to fill arguments, so describing them well is the whole game. You can do it three ways, from least to most control.++**Simple**: decorate a function. The signature gives the argument types, the docstring is the tool description, and `run()` completes the loop: model picks the call, Needle executes your function, feeds the result back, and returns the model's final answer.++```python+import needle-After setup completes, activate the virtual environment in every new terminal session before running needle commands:+def get_weather(city: str):+ "Get the current weather for a city."+ return {"city": city, "temp_c": 27, "sky": "clear"}-```powershell-.\.venv\Scripts\Activate.ps1+agent = needle.Needle(tools=[get_weather])+print(agent.run("what's it like in Lagos right now?")["reasoning"])```-> **Tip:** If PowerShell blocks the script, run `Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser` once and retry.+**Medium**: describe each argument and offer choices. Needle reads a Google-style `Args:` block for per-parameter descriptions; a default makes an argument optional; a `Literal` becomes a fixed set the model must choose from (it cannot emit anything else).++```python+from typing import Literal++def set_thermostat(temperature: int, mode: Literal["heat", "cool", "auto"] = "auto"):+ """Set the thermostat.++ Args:+ temperature: target temperature in Celsius+ mode: heating strategy to use+ """+ return {"temperature": temperature, "mode": mode}++agent = needle.Needle(tools=[set_thermostat])+agent.run("make it 21 and cool the room")+```-Then launch the playground:+**Advanced**: constrain the values with `needle.Field`, attached inline via `Annotated`. Ranges, patterns, lengths, and item counts are compiled into the decode grammar, so the model can only ever emit values that satisfy them.-```powershell-needle playground+```python+from typing import Annotated++def send_money(+ amount: Annotated[float, needle.Field(gt=0, le=10000, description="USD, up to 10,000")],+ to: Annotated[str, needle.Field(pattern=r"^@[a-z0-9_]+$", description="recipient handle")],+ memo: Annotated[str, needle.Field(max_length=80)] = "",+):+ "Send money to a handle."+ return {"sent": amount, "to": to}```-Opens a web UI at http://127.0.0.1:7860 where you can test and finetune on your own tools. Weights are auto-downloaded.+`Field` supports `description`, `enum`, `const`, `ge`/`le`/`gt`/`lt`, `multiple_of`, `min_length`/`max_length`, `pattern`, `format`, `min_items`/`max_items`, and `unique_items`.-## Usage (Python)+**Extraction**: to pull structured data out of text, declare the shape and call `extract()`. Pass a Pydantic model and you get a typed object back.```python-from needle import SimpleAttentionNetwork, load_checkpoint, generate, get_tokenizer+from pydantic import BaseModel-params, config = load_checkpoint("checkpoints/needle.pkl")-model = SimpleAttentionNetwork(config)-tokenizer = get_tokenizer()+class Invoice(BaseModel):+ vendor: str+ total: float+ due_date: str-result = generate(- model, params, tokenizer,- query="What's the weather in San Francisco?",- tools='[{"name":"get_weather","description":"Get current weather for a city.","parameters":{"location":{"type":"string","description":"City name.","required":true}}}]',- stream=False,-)-print(result)-# [{"name":"get_weather","arguments":{"location":"San Francisco"}}]+invoice = needle.extract("Invoice from Acme Corp, $1,200.00, due 2026-09-01", Invoice)+print(invoice.vendor, invoice.total) # -> Acme Corp 1200.0```-## Finetuning--```bash-# Playground (generates data via Gemini, trains, evaluates, bundles result)-needle playground+**By hand** - the decorator just builds a JSON schema; you can pass that schema directly, which is exactly what Needle consumes. This is how you set descriptions and constraints without the decorator (and `tools.json` for the CLI is the same shape):-# CLI (auto-downloads weights if not local)-needle finetune data.jsonl+```python+tools = [{+ "name": "set_lights",+ "description": "Turn a room's lights on or off and set brightness",+ "parameters": {+ "type": "object",+ "properties": {+ "room": {"type": "string", "description": "which room to control"},+ "on": {"type": "boolean"},+ "brightness": {"type": "integer", "minimum": 0, "maximum": 100},+ },+ "required": ["room", "on"],+ },+}]+agent = needle.Needle(tools=tools)```-### Data format+Prefer to drive the loop yourself instead of `run()`? `complete()` returns the raw call and you execute it:-Each line in the JSONL file has three fields: `query`, `tools`, and `answers`.+```python+import json+response = agent.complete("dim the living room to 30")+if response["type"] == "call":+ result = set_lights(**response["function_calls"][0]["arguments"])+ response = agent.complete(json.dumps(result)) # feed the result back+```++With a large catalogue, persist tool embeddings across runs with `needle.Needle(tools=..., tool_index_path="tools.idx")`. Every turn returns one JSON object:-**Tool schema:**```json{- "name": "get_weather",- "description": "Get current weather for a city.",- "parameters": {- "location": { "type": "string", "description": "City name.", "required": true }- }+ "type": "call",+ "success": true,+ "error": null,+ "error_code": null,+ "function_calls": [ { "name": "set_lights", "arguments": { "room": "living room", "on": true, "brightness": 30 } } ],+ "reasoning": "'living room' -> room; 'dim' -> on true, brightness 30",+ "confidence": 0.94,+ "prefill_tps": 4300.0,+ "decode_tps": 850.0,+ "peak_ram_mb": 28.0}```-**Answer schema:**-```json-{ "name": "get_weather", "arguments": { "location": "Paris" } }-```+## Behaviour++Needle solves every problem as a function call. The context declares what may be called; the model answers with calls. Performing an action and extracting structured data are the same operation, the only difference is what you declare.++- A request no declared tool can serve is refused with the empty call `[]`. That is the whole contract for off-topic input; there is no free-text fallback.+- Arguments contain only values evidenced by the input. An optional field with no evidence is omitted, not guessed; omission is the field-level `[]`.+- `reasoning` is the model's short derivation of each argument from its source span (`'ten minutes' -> minutes 10`). It is generated unconstrained; only the call itself is grammar-constrained, so the JSON cannot be malformed while the derivation stays legible.+- After you execute a call, pass the result back as the next `complete()`. The model continues from it, and later arguments may depend on earlier results: `search_for_contact` first, then `send_instant_message` with the returned `contact_id`. A final step may answer in plain text from the results: `"type": "respond"` with empty `function_calls`.+- A session shares one toolset. Later turns are bare queries against the same tools; `reset()` rewinds the conversation and keeps the tools loaded.++## System facts++An optional system turn carries environment state as facts, never instructions:-**Full JSONL example** (each line is one training example, `tools` and `answers` are JSON-encoded strings):-```jsonl-{"query": "What's the weather in Paris?", "tools": "[{\"name\":\"get_weather\",\"description\":\"Get current weather for a city.\",\"parameters\":{\"location\":{\"type\":\"string\",\"description\":\"City name.\",\"required\":true}}}]", "answers": "[{\"name\":\"get_weather\",\"arguments\":{\"location\":\"Paris\"}}]"}-{"query": "Turn off the lights", "tools": "[{\"name\":\"get_weather\",\"description\":\"Get current weather for a city.\",\"parameters\":{\"location\":{\"type\":\"string\",\"description\":\"City name.\",\"required\":true}}},{\"name\":\"toggle_lights\",\"description\":\"Toggle smart lights on or off.\",\"parameters\":{\"state\":{\"type\":\"string\",\"description\":\"on or off.\",\"required\":true}}}]", "answers": "[{\"name\":\"toggle_lights\",\"arguments\":{\"state\":\"off\"}}]"}```+date: 2026-07-21 Tue 14:30; locale: en-US; device: phone; battery: 62%+```++Recognized keys are `date`, `locale`, `device`, `battery`, `network`, `location`, `user`, and `assistant`. The model resolves relative language against them: "tomorrow at 7" becomes an absolute time only when a `date:` fact licenses it, otherwise the human phrase passes through verbatim. `assistant:` declares the identity the model binds to. Pass the turn with `--system system.txt` on the CLI or `needle.Needle(tools=tools, system="date: ...")` in Python. Needle trains with and without the turn, so omitting it is safe; instructions placed there do not steer the model.++## Deploy Needle++Download the folder for your platform from the release:-Provide at least **120 examples per tool** (100 train / 10 val / 10 test). Fewer examples will overfit — you'll see perfect training metrics but the model won't generalize. Vary query phrasing and include examples with multiple tools available.+| your device | folder | command-line | library |+| --- | --- | --- | --- |+| Mac (Apple Silicon) | `macos-arm64` | `needle` | `libneedle.a` |+| Linux x86-64 (PC, server, AMD) | `linux-x86_64` | `needle` | `libneedle.a` |+| Linux ARM64 (Raspberry Pi, server) | `linux-arm64` | `needle` | `libneedle.a` |+| Linux ARMv7 (32-bit) | `linux-armv7` | `needle` | `libneedle.a` |+| Linux RISC-V | `linux-riscv64` | `needle` | `libneedle.a` |+| Linux MIPS32el (Ingenic cameras, routers) | `linux-mipsel` | `needle` | `libneedle.a` |+| Windows x64 | `windows-x86_64` | `needle.exe` | `libneedle.a` |+| Windows ARM | `windows-arm64` | `needle.exe` | `libneedle.a` |+| Android | `android-arm64` / `android-armv7` / `android-riscv64` | `needle` | `libneedle.a` |+| iOS / watchOS / tvOS | `ios-arm64` / `watchos-arm64` / `tvos-arm64` | - | `libneedle.a` |+| Browser / Node (WebAssembly) | `wasm` | - | `needle.js` + `needle.wasm` |-### Using a finetuned model+To run it, use the command-line binary. On macOS, Linux, or Android:-Finetuning saves the best checkpoint as `checkpoints/needle_finetuned_<id>_best.pkl`:+```sh+# answer one query and exit+./needle --tools tools.json --prompt "dim the living room to 30"-```bash-needle run --checkpoint checkpoints/needle_finetuned_*_best.pkl \- --query "What's the weather?" --tools '[{"name":"get_weather","description":"Get current weather for a city.","parameters":{"location":{"type":"string","description":"City name.","required":true}}}]'+# or an HTTP server on localhost:8080 (POST /complete {"input": "..."})+./needle --tools tools.json --serve++# with a large tool catalogue, persist tool embeddings across runs+./needle --tools tools.json --tool-index tools.idx --serve```-```python-params, config = load_checkpoint("checkpoints/needle_finetuned_<id>_best.pkl")-model = SimpleAttentionNetwork(config)-result = generate(model, params, get_tokenizer(), query="...", tools='[...]', stream=False)+`tools.json` is a JSON array of the functions the assistant may call:++```json+[+ {+ "name": "set_lights",+ "description": "Turn a room's lights on or off and set brightness",+ "parameters": {+ "type": "object",+ "properties": {+ "room": { "type": "string" },+ "on": { "type": "boolean" },+ "brightness": { "type": "integer", "description": "0 to 100" }+ },+ "required": ["room", "on"]+ }+ },+ {+ "name": "play_music",+ "description": "Play music matching a mood, genre, or artist",+ "parameters": {+ "type": "object",+ "properties": { "query": { "type": "string" } },+ "required": ["query"]+ }+ },+ {+ "name": "send_message",+ "description": "Text a contact",+ "parameters": {+ "type": "object",+ "properties": {+ "to": { "type": "string" },+ "body": { "type": "string" }+ },+ "required": ["to", "body"]+ }+ }+]```-## CLI+## Tool retrieval++Five or fewer declared tools render directly. Above that, retrieval engages: at init every tool schema is embedded once by a built-in contrastive head, each turn embeds the query, and only the five highest-scoring tools enter the context, with the grammar rebuilt over just that subset. An unselected tool is unreachable, not merely unlikely. `--tool-index <path>` (CLI) or `tool_index_path` (Python) persists the embeddings on disk, keyed by a fingerprint over the schemas and the model; a matching fingerprint loads instantly, a changed schema re-embeds only what changed.++## Confidence++The `confidence` field is the minimum of two signals: a calibrated post-hoc head that scores the full prompt plus the call the model just produced, and the decoding probability of the call tokens. A call is accepted only when both agree, so the failure mode is escalation, not wrong execution. The contract: pick a threshold for your product, act at or above it, re-ask or route to a bigger model below it. Off-topic requests return the empty call `[]`.++## Fine-tuning+Needle is open and trainable end to end. Fine-tune it on your own tools and domains, then export to a `.cact` and ship it like the base model. See the [needle repo](https://github.com/cactus-compute/needle) for training and export.++## Extraction++Extraction is the same exchange as tool calling: declare the record schema as the only tool and pass the content as the prompt; the passage sits where the query sits, and the returned call's `arguments` are the extracted fields. With one declared tool the grammar admits exactly one call of that name, the `tool_choice` equivalent, so schema conformance is guaranteed rather than requested. There is no separate JSON mode.++`schema.json` describes the record to extract:++```json+[+ {+ "name": "receipt",+ "description": "A purchase receipt shared as text",+ "parameters": {+ "type": "object",+ "properties": {+ "merchant": { "type": "string" },+ "total": { "type": "number" },+ "currency": { "type": "string" },+ "line_items": { "type": "array", "items": { "type": "object" } }+ },+ "required": ["merchant", "total"]+ }+ }+]```-needle playground Test and finetune via web UI-needle finetune <data.jsonl> Finetune on your own data-needle run --query "..." --tools Single inference-needle train Full training run-needle pretrain Pretrain on PleIAs/SYNTH-needle eval --checkpoint <path> Evaluate a checkpoint-needle tokenize Tokenize dataset-needle generate-data Synthesize training data via Gemini-needle tpu <action> TPU management (see docs/tpu.md)++```sh+./needle --tools schema.json --prompt "GreenMart receipt: oat milk 3.50, total 7.75 paid by visa"```+```json+{ "type": "call", "function_calls": [ { "name": "receipt", "arguments": { "merchant": "GreenMart", "total": 7.75 } } ] }```-@misc{ndubuaku2026needle,- title={Needle},- author={Henry Ndubuaku, Jakub Mroz, Karen Mosoyan, Roman Shemet, Parkirat Sandhu, Satyajit Kumar, Noah Cylich, Justin H. Lee},- year={2026},- url={https://github.com/cactus-compute/needle}++## Citation++Needle 2 is built by the Cactus Compute team. If you use it in your work, please cite:++```bibtex+@misc{needle2_2026,+ title = {Needle 2: A 45M-Parameter Foundation Tool-Calling Model for Tiny Devices},+ author = {Ndubuaku, Henry and Mosoyan, Karen and Mroz, Jakub and Cylich, Noah and+ Kumar, Satyajit and Sandhu, Parkirat and Shemet, Roman and Lee, Justin H.},+ year = {2026},+ organization = {Cactus Compute, Inc.},+ howpublished = {\url{https://github.com/cactus-compute/needle}}}```++Reach out on [email protected] for partnerships, collaborations, synergies and deploying Needle2 in your product.\ No newline at end of filediff --git a/assets/architecture.png b/assets/architecture.pngnew file mode 100644index 0000000..d27c08eBinary files /dev/null and b/assets/architecture.png differdiff --git a/assets/frontier.png b/assets/frontier.pngnew file mode 100644index 0000000..8e79d6cBinary files /dev/null and b/assets/frontier.png differdiff --git a/assets/frontier.svg b/assets/frontier.svgnew file mode 100644index 0000000..f00812c--- /dev/null+++ b/assets/frontier.svg@@ -0,0 +1,42 @@+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 520" role="img" aria-label="Scatter plot of Mobile Actions accuracy versus model size">+ <rect x="0" y="0" width="900" height="520" fill="#111113"/>+ <line x1="90" y1="60" x2="850" y2="60" stroke="#232326" stroke-width="1"/>+ <line x1="90" y1="150" x2="850" y2="150" stroke="#232326" stroke-width="1"/>+ <line x1="90" y1="240" x2="850" y2="240" stroke="#232326" stroke-width="1"/>+ <line x1="90" y1="330" x2="850" y2="330" stroke="#232326" stroke-width="1"/>+ <line x1="90" y1="420" x2="850" y2="420" stroke="#8a877f" stroke-width="1"/>+ <line x1="90" y1="60" x2="90" y2="420" stroke="#8a877f" stroke-width="1"/>+ <g font-family="ui-monospace, Menlo, monospace" font-size="11" fill="#85827a">+ <text x="78" y="424" text-anchor="end">40</text>+ <text x="78" y="334" text-anchor="end">55</text>+ <text x="78" y="244" text-anchor="end">70</text>+ <text x="78" y="154" text-anchor="end">85</text>+ <text x="78" y="64" text-anchor="end">100</text>+ <text x="90" y="444" text-anchor="middle">0</text>+ <text x="209" y="444" text-anchor="middle">0.5B</text>+ <text x="328" y="444" text-anchor="middle">1B</text>+ <text x="446" y="444" text-anchor="middle">1.5B</text>+ <text x="565" y="444" text-anchor="middle">2B</text>+ <text x="684" y="444" text-anchor="middle">2.5B</text>+ <text x="802" y="444" text-anchor="middle">3B</text>+ <text x="470" y="480" text-anchor="middle">Total parameters</text>+ <text x="30" y="240" text-anchor="middle" transform="rotate(-90 30 240)">Mobile-Actions accuracy (%)</text>+ </g>+ <path d="M 144.6 245.4 C 149.3 245.4, 149.3 276.0, 154.1 276.0 C 478.3 276.0, 478.3 314.4, 802.5 314.4" fill="none" stroke="#8a877f" stroke-width="1.4"/>+ <g font-family="Inter, -apple-system, sans-serif" font-size="12.5" fill="#e8e6df">+ <circle cx="144.6" cy="245.4" r="5" fill="#e8e6df"/>+ <text x="156.6" y="237.4">LFM2.5 230M</text>+ <text x="156.6" y="250.4" font-size="10.5" fill="#85827a">f16 · vLLM</text>+ <circle cx="154.1" cy="276.0" r="5" fill="#e8e6df"/>+ <text x="166.1" y="292.0">FunctionGemma 270M</text>+ <text x="166.1" y="305.0" font-size="10.5" fill="#85827a">f16 · vLLM</text>+ <circle cx="802.5" cy="314.4" r="5" fill="#e8e6df"/>+ <text x="790.5" y="334.4" text-anchor="end">Apple FM</text>+ <text x="790.5" y="347.4" text-anchor="end" font-size="10.5" fill="#85827a">on-device</text>+ </g>+ <g>+ <circle cx="100.7" cy="277.8" r="6" fill="#e8593d"/>+ <text x="100.7" y="251.8" text-anchor="middle" font-family="Inter, sans-serif" font-size="13" font-weight="600" fill="#e8593d">Needle 2</text>+ <text x="100.7" y="264.8" text-anchor="middle" font-size="10.5" fill="#85827a">CQ2-bit</text>+ </g>+ </svg>diff --git a/docs/hf_config.json b/docs/hf_config.jsondeleted file mode 100644index b990d75..0000000--- a/docs/hf_config.json+++ /dev/null@@ -1,5 +0,0 @@-{- "library_name": "jax",- "model_type": "custom",- "architectures": ["SimpleAttentionNetwork"]-}diff --git a/docs/hf_model_card.md b/docs/hf_model_card.mddeleted file mode 100644index e356505..0000000--- a/docs/hf_model_card.md+++ /dev/null@@ -1,139 +0,0 @@-----license: mit-library_name: jax-tags:- - function-calling- - tool-use- - encoder-decoder- - edge- - on-device- - jax- - flax------# Needle--We distilled Gemini 3.1 into a 26m parameter "[Simple Attention Network](docs/simple_attention_networks.md)" that you can even finetune locally on your Mac/PC.-In production, Needle runs on [Cactus](https://github.com/cactus-compute/cactus) at 6000 toks/sec prefill and 1200 decode speed.-Weights are fully open on [Cactus-Compute/needle](https://huggingface.co/Cactus-Compute/needle), as well as the dataset generation.--| | |-|---|---|-| Parameters | 26M |-| Architecture | Encoder-decoder, pure attention (no FFN) |-| Encoder | 12 layers, GQA (8H/4KV), RoPE, gated residuals |-| Decoder | 8 layers, self-attn + cross-attn, gated residuals |-| d_model | 512 |-| Vocab | 8192 (SentencePiece BPE) |-| Norm | ZCRMSNorm (zero-centered, init=0) |-| Precision | bfloat16 (INT4 QAT during training) |-| Pretraining | 200B tokens on 16x TPU v6e (27hrs) |-| Post-training | 2B tokens of function call data (45mins) |--```-d=512, 8H/4KV, BPE=8192- ┌──────────────┐- │ Tool Call │- └──────┬───────┘- ┌┴──────────┐- │ Softmax │- └─────┬─────┘- ┌─────┴─────┐- │ Linear (T)│ <- tied- └─────┬─────┘- ┌─────┴─────┐- │ ZCRMSNorm │- └─────┬─────┘- ┌────────┴────────┐- │ Decoder x 8 │- │┌───────────────┐│- ││ ZCRMSNorm ││- ││ Masked Self ││- ││ Attn + RoPE ││- ││ Gated Residual││- │├───────────────┤│- ┌──────────────┐ ││ ZCRMSNorm ││- │ Encoder x 12 │─────────────────────>Cross Attn ││- │ │ ││ Gated Residual││- │ ┌──────────┐ │ │└───────────────┘│- │ │ZCRMSNorm │ │ └────────┬────────┘- │ │Self Attn │ │ ┌─────┴─────┐- │ │ GQA+RoPE │ │ │ Embedding │ <- shared- │ │Gated Res │ │ └─────┬─────┘- │ │ │ │ ┌───────┴────────┐- │ │ (no FFN) │ │ │[EOS]<tool_call>│- │ └──────────┘ │ │ + answer │- │ │ └────────────────┘- └──────┬───────┘- │- ┌────┴──────┐- │ Embedding │- └────┬──────┘- │- ┌────┴──────┐- │ Text │- │ query │- └───────────┘-```--## Quickstart--```bash-git clone https://github.com/cactus-compute/needle.git-cd needle && source ./setup-needle playground-```--Opens a web UI at http://127.0.0.1:7860 where you can test and finetune on your own tools. Weights are auto-downloaded.--## Usage (Python)--```python-from needle import load_checkpoint, generate, SimpleAttentionNetwork, get_tokenizer--params, config = load_checkpoint("checkpoints/needle.pkl")-model = SimpleAttentionNetwork(config)-tokenizer = get_tokenizer()--result = generate(- model, params, tokenizer,- query="What's the weather in San Francisco?",- tools='[{"name":"get_weather","description":"Get current weather for a city.","parameters":{"location":{"type":"string","description":"City name.","required":true}}}]',- stream=False,-)-print(result)-# [{"name":"get_weather","arguments":{"location":"San Francisco"}}]-```--## Finetuning--Finetune on your own tools via the web UI or CLI:--```bash-# Web UI (generates data via Gemini, trains, evaluates, bundles result)-needle playground--# CLI (auto-downloads weights if not local)-needle finetune data.jsonl-```--## Links--- [Needle](https://github.com/cactus-compute/needle) - training, finetuning, and inference code-- [Cactus](https://github.com/cactus-compute/cactus) - on-device runtime (6000 tok/s prefill, 1200 tok/s decode)-- [Simple Attention Networks](https://github.com/cactus-compute/needle/blob/main/docs/simple_attention_networks.md) - architecture details--## License--MIT--## Citation--```-@misc{ndubuaku2026needle,- title={Needle},- author={Henry Ndubuaku and Jakub Mroz and Karen Mosoyan and Roman Shemet and Parkirat Sandhu and Satyajit Kumar and Noah Cylich and Justin H. Lee},- year={2026},- url={https://github.com/cactus-compute/needle}-}-```diff --git a/docs/simple_attention_networks.md b/docs/simple_attention_networks.mddeleted file mode 100644index c7332aa..0000000--- a/docs/simple_attention_networks.md+++ /dev/null@@ -1,115 +0,0 @@-# Simple Attention Networks--Experiments at Cactus showed that MLPs can be completely dropped from transformer networks, as long as the model relies on external knowledge source.-Function calling relies on external tools list, so we designed a simple attention network for function calling.--```-d=512, 8H/4KV, BPE=8192- ┌──────────────┐- │ Tool Call │- └──────┬───────┘- ┌┴──────────┐- │ Softmax │- └─────┬─────┘- ┌─────┴─────┐- │ Linear (T)│ ← tied- └─────┬─────┘- ┌─────┴─────┐- │ ZCRMSNorm │- └─────┬─────┘- ┌────────┴────────┐- │ Decoder x 8 │- │┌───────────────┐│- ││ ZCRMSNorm ││- ││ Masked Self ││- ││ Attn + RoPE ││- ││ Gated Residual││- │├───────────────┤│- ┌──────────────┐ ││ ZCRMSNorm ││- │ Encoder x 12 │──────────────────────▶Cross Attn ││- │ │ ││ Gated Residual││- │ ┌──────────┐ │ │└───────────────┘│- │ │ZCRMSNorm │ │ └────────┬────────┘- │ │Self Attn │ │ ┌─────┴─────┐- │ │ GQA+RoPE │ │ │ Embedding │ ← shared- │ │Gated Res │ │ └─────┬─────┘- │ │ │ │ ┌───────┴───────-┐- │ │ (no FFN) │ │ │[EOS]<tool_call>│- │ └──────────┘ │ │ + answer │- │ │ └───────────────-┘- └──────┬───────┘- │- ┌────┴──────┐- │ Embedding │- └────┬──────┘- │- ┌────┴──────┐- │ Text │- │ query │- └───────────┘-```--## Why No FFN--- **Softmax is nonlinear.** `softmax(QK^T/sqrt(d)) * V` is a data-dependent nonlinear mixing operation. For a task that is about routing information (query -> tool alignment), attention is the right primitive.-- **Tool calling is retrieval-and-assembly.** Match query to tool name, extract argument values, assemble JSON. All three are aligning and copying between input and output -exactly what cross-attention does. No step requires per-position feature transformation (which is what FFN provides).-- **At small scale, FFN parameters are wasted.** ~2/3 of standard transformer parameters are FFN. For a <50M model on a structured task, those parameters contribute less than more attention layers (deeper cross-attention = better query-tool alignment).-- **Fewer parameters = faster inference.** FFNs have the biggest GEMM/GEMV dimensions -removing them cuts per-layer parameters by ~2/3, directly reducing the memory bandwidth bottleneck that dominates latency on edge devices.--## Why Encoder-Decoder--- **Bidirectional encoding.** Tools are structured objects -a bidirectional encoder sees the full definition at once. A causal model sees it left-to-right and must infer structure from partial context.-- **No input tokens in KV cache.** Encoder-decoder uses a fixed-size encoder representation for cross-attention, not re-attending the full input at every generation step.-- **Natural fit for multi-head design.** The encoder output feeds the decoder (generation) and the contrastive head (tool retrieval). Clean separation.--## Gated Residuals--Without FFN, there is no per-position nonlinear rewriting per layer. This makes residual connection design critical.--- **Standard residual** `x = x + Attn(Norm(x))` -attention can only ADD a delta. Without FFN to do the rewriting, purely additive is limiting.-- **No residual** `x = Attn(Norm(x))` -each layer fully rewrites, but we lose the gradient highway. Deep networks (12+ layers) will not train.-- **Gated residual (ours)** `x = x + sigmoid(gate) * Attn(Norm(x))` -per-sublayer learnable scalar, initialized to 0. sigmoid(0) = 0.5, so training starts with half-strength residual. The model can learn to sharpen useful layers (g->1) or suppress unhelpful ones (g->0) without losing gradient flow.--## ZCRMSNorm--- **Standard RMSNorm:** `x * gamma / RMS(x)`, gamma initialized to 1.-- **ZCRMSNorm:** `x * (1 + gamma) / RMS(x)`, gamma initialized to 0.-- At init, ZCRMSNorm is identity-up-to-scale. Pairs with gated residuals: the entire block starts as a damped identity + damped normalized attention. No component starts with a strong learned bias.-- From the nGPT / DeepSeek-V3 line of work. Applied to QK heads as well (QK-norm) for training stability.--## Contrastive Tool Selection Head--CLIP-style head for retrieving relevant tools before generation. Useful when the tool set is large and you want to filter to the top-k most relevant tools for a query.--- **Architecture:** encoder output -> mean pool over non-pad positions -> Dense(d_model/4) -> ReLU -> Dense(128) -> L2-normalize. Produces a unit vector per input.-- **Training:** symmetric contrastive loss (CLIP). Each batch pairs queries with their positive tools; in-batch negatives provide the contrastive signal. Learnable temperature (`log_temp`).-- **Inference:** encode query and each tool candidate into the shared embedding space, rank by cosine similarity, take top-k.-- Trained jointly with the main CE loss at 0.1x weight. Same encoder is used for both generation and retrieval.--## Muon for Attention-Only--- **Dual optimizer:** Muon (Q/K/V/O projections, LR 0.02, WD 0.01) + AdamW (everything else, LR 3e-4).-- Without FFN, the model is a deep stack of linear projections with softmax routing. Muon enforces orthogonality on weight updates via Newton-Schulz, preventing the representation collapse that can happen when stacking many linear layers without interleaving nonlinearities.--## INT4 QAT as Regularization--- **Fake quantization every 100 steps.** Weights are group-wise quantized to INT4 (symmetric, group_size=32) with straight-through estimator (STE), then used for the forward pass. Gradients flow through the rounding via STE.-- **Regularization effect.** Quantization noise acts as a form of weight noise regularization, discouraging the model from relying on precise weight values. For a small model with no FFN (fewer parameters, higher overfitting risk), this is beneficial.-- **Deploy-ready.** The model trains with the same quantization it will see at inference. No post-training quantization gap.--## Token-Level Loss Weighting--- Base (JSON structure): 1.0x, tool names: 2.0x, argument keys: 1.5x, argument values: 4.0x.-- The model achieves ~99% JSON parse rate early in training. The actual errors are values > names > keys > structure. Weighting matches the error distribution.-- Auxiliary: z-loss for logit stability, CLIP contrastive loss at 0.1x.--## Citation--```-@misc{ndubuaku2025needle,- title={Simple Attention Networks},- author={Henry Ndubuaku},- year={2026},- url={https://github.com/cactus-compute/needle/blob/main/docs/simple_attention_networks.md}-}-```diff --git a/docs/tpu.md b/docs/tpu.mddeleted file mode 100644index d8a8bcc..0000000--- a/docs/tpu.md+++ /dev/null@@ -1,135 +0,0 @@-# TPU Setup & Workflow--## TPU Factsheet--```- Trillium (v6e)- ──────────────────────────────────────────- HBM per chip 32 GB- BF16 FLOPS 918 TFLOPS- HBM bandwidth 1,640 GB/s- ICI bandwidth 3,584 Gbps- On-demand/chip/hr $2.70 (US regions)- ──────────────────────────────────────────-- Dataset- ──────────────────────────────────────────- Text 2M Tool-call pairs- (query, tools, answers)- synthesized via Gemini- Cactus-Compute/tool-calls- (HuggingFace)- ──────────────────────────────────────────-- ┌────────────────────┬──────────┬──────────┬──────────┐- │ │ v6e-8 │ v6e-16 │ v6e-32 │- ├────────────────────┼──────────┼──────────┼──────────┤- │ Chips │ 8 │ 16 │ 32 │- │ Total HBM │ 256 GB │ 512 GB │ 1024 GB │- │ Scaling eff. │ 0.9× │ 0.8× │ 0.7× │- │ Eff. TFLOPS │ 994 │ 1,766 │ 3,091 │- │ Est. time │ ~88h │ ~49h │ ~29h │- │ On-demand $/hr │ $21.60 │ $43.20 │ $86.40 │- │ Est. total cost │ ~$1,900 │ ~$2,120 │ ~$2,510 │- └────────────────────┴──────────┴──────────┴──────────┘-```--## CLI Reference--```- needle tpu-- create NAME Create TPU (auto-finds zone)- --type STR Accelerator (default: v6e-8)- --version STR TPU OS (auto-detected from --type)- --preemptible Create spot/preemptible instance- connect NAME SSH config + connect (auto-zone)- setup NAME Full setup: sync code + venv + deps- sync NAME Fast sync: code (no venv rebuild)- train NAME [-- ARGS] Launch training on all workers- pretrain NAME [-- ARGS] Launch pretraining on all workers- claude NAME Install Claude Code on instance- stop NAME Stop instance (auto-zone)- start NAME Start stopped instance (auto-zone)- delete NAME Delete instance (auto-zone)- list List all TPU instances- --zone ZONE Override auto-detected zone-```--## GCP Setup--- Download the `macOS ARM` gcloud SDK from [here](https://docs.cloud.google.com/sdk/docs/install-sdk) and unzip.-- Open terminal, cd to your downloads and run `./google-cloud-sdk/install.sh`-- Restart terminal and run `gcloud init`, sign in with cactus email, should prompt for project-- Else, set the project with `gcloud config set [PROJECT NAME]`-- Run `gcloud help` and read carefully--## Single-host (v6e-8) - SSH into instance--```-1. Create an instance (auto: finds zone -> installs Claude Code -> connects via SSH)- needle tpu create my-experiment- (exit with 'exit' or Ctrl+D)--2. Reconnect anytime- ssh my-experiment- or VS Code: click the '><' in the bottom left -> select my-experiment----- run from the instance -----3. Create a GitHub Personal Access Token (PAT)- GitHub -> Settings -> Developer settings -> Personal access tokens -> Tokens (classic)- Generate a token with 'repo' scope--4. Clone the repo on your instance using your PAT- git clone https://github.com/cactus-compute/needle.git- cd needle && source ./setup--5. Login to Hugging Face (required for private datasets)- hf auth login- (paste your HF token - get one at https://huggingface.co/settings/tokens)--6. Use tmux for long training runs (survives SSH disconnects)- tmux new -s train # start a named session- needle train --wandb # run training inside it- Ctrl+B, then D # detach (keeps running)- tmux attach -t train # reattach later----- back on your Mac -----7. Stop when done (saves disk, stops billing)- needle tpu stop my-experiment--8. (Optional) Delete instance when no longer needed- needle tpu delete my-experiment-```--## Multi-host (v6e-16+) - run from your Mac--For multi-host TPUs (v6e-16+), you drive everything from your Mac.-The CLI syncs code and launches training across all workers automatically.--```-1. Set auth tokens (workers need these to download data + log metrics)- export HF_TOKEN=hf_...- export WANDB_API_KEY=...--2. Add SSH key to agent (required for gcloud scp/ssh)- ssh-add ~/.ssh/google_compute_engine--3. Create a multi-host TPU (auto: finds zone -> syncs code -> installs deps)- needle tpu create my-experiment --type v6e-16--4. Launch training on all workers from your Mac- needle tpu train my-experiment -- --wandb --epochs 1--5. After code changes, sync without rebuilding venv (fast, ~seconds)- needle tpu sync my-experiment--6. Full re-setup if deps changed (slow, rebuilds venv)- needle tpu setup my-experiment--7. Stop/delete when done- needle tpu stop my-experiment- needle tpu delete my-experiment-```