Skip to content

Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.

fix: enhance documentation for offline setup and model sharing; add CLI commands for fetching and downloading models; implement environment variable support for library path

HenryNdubuakuc949b6ce7078
diff --git a/README.md b/README.md
index 76e15ac..9843272 100644
--- a/README.md
+++ b/README.md
@@ -4,7 +4,7 @@
Needle 2 is an open 45M-parameter model for tool calling, device use and structured extraction. The whole model is a single 14MB binary that runs a full session in about 28MB of RAM. It is built on our Simple Attention Network findings, compressed to CQ2-bit with Cactus Quants, and baked into its own engine. On the benchmarks below, Needle 2 trades wins with other small models like FunctionGemma 270M, LFM2.5 230M and Apple FM, at 5x to 70x smaller, and 2 bits against their f16.
-This repository is the Python package: inference, LoRA fine-tuning, and export. `pip install cactus-needle`, describe your tools, and call them from Python. The inference engine is fetched once from Hugging Face and cached; there is nothing else to build.
+This repository is the Python package: inference, LoRA fine-tuning, and export. `pip install cactus-needle`, describe your tools, and call them from Python. The inference engine is fetched once from Hugging Face and cached; there is nothing else to build, and offline setup for air gapped devices is covered in [doc/apis.md](doc/apis.md).
- **Self-contained**: weights baked into a single 14MB engine; no separate model files to manage, and inference does no network.
- **Simple contract**: tool calls come back as structured data, text in, JSON out; a byte-level grammar compiled from your schemas constrains every token.
@@ -121,7 +121,7 @@ pip install "cactus-needle[metal]"
needle build checkpoints/needle2.pkl --lora checkpoints/needle_lora.pkl --out my_needle.cact
```
-Add `--bits 2` (default 4) for a smaller model, or set `NEEDLE_HF_REPO=<you>/<model>` and pass `--upload` to publish the `.cact`.
+Add `--bits 2` (default 4) for a smaller model, or set `NEEDLE_HF_REPO=<you>/<model>` and pass `--upload` to publish the `.cact`. The counterpart `needle download <you>/<model>/my_needle.cact` pulls a published archive on any machine.
**4. Run it.** The engine is weights-agnostic, so a tuned `.cact` runs on it directly - no recompilation:
diff --git a/doc/apis.md b/doc/apis.md
index 54a9424..b2bec75 100644
--- a/doc/apis.md
+++ b/doc/apis.md
@@ -151,3 +151,13 @@ Five or fewer declared tools render directly. Above that, retrieval engages: at
The `confidence` field is the minimum of two signals: a calibrated post-hoc head that scores the full prompt plus the call the model just produced, and the decoding probability of the call tokens. A call is accepted only when both agree, so the failure mode is escalation, not wrong execution. The contract: pick a threshold for your product, act at or above it, re-ask or route to a bigger model below it. Off-topic requests return the empty call `[]`.
Calibration holds for the base model only. Fine-tuning does not update the head, so an agent running tuned weights reports `confidence` as `None` and warns once at construction.
+
+## Offline devices
+
+The engine is fetched once and cached at `~/.cache/cactus-needle/<engine version>/`. Inference itself never touches the network, so an air gapped device only needs that one file in place. Three ways to get it there:
+
+1. `needle fetch` downloads the engine for the current machine into the cache and prints the path. `--out <dir>` places it elsewhere. `--platform-tag manylinux2014_aarch64` fetches the build for a different device; tags follow the wheel names on the Hugging Face repo (`macosx_11_0_arm64`, `manylinux2014_x86_64`, `musllinux_1_2_aarch64`, `win_amd64`, `win_arm64`).
+2. Copy the file to the same cache path on the device, or drop it inside the installed `needle/` package directory, which wins over the cache.
+3. Set `NEEDLE_LIB_PATH=/path/to/libneedle.so` to load exactly that file and skip every lookup.
+
+The Python package itself installs offline the standard way: `pip download cactus-needle` on a connected machine, then `pip install --no-index --find-links <dir> cactus-needle` on the device.
diff --git a/doc/finetuning.md b/doc/finetuning.md
index c098369..1e15d43 100644
--- a/doc/finetuning.md
+++ b/doc/finetuning.md
@@ -31,6 +31,8 @@ needle build checkpoints/needle2.pkl --lora adapter.pkl --out tuned.cact
agent = needle.Needle(tools=[...], weights="tuned.cact")
```
+To share a tuned model, set `NEEDLE_HF_REPO=<you>/<model>` and pass `--upload` to `needle build`; on any other machine `needle download <you>/<model>/tuned.cact` pulls it back down (or pass just `<you>/<model>` when the repo holds a single archive).
+
Defaults: batch size 16, learning rate 0.0001 with warmup and cosine decay, gradient clipping at norm 1, rank 16, alpha 32, max length 1024, validation split 0.1. The base checkpoint downloads from Hugging Face on first run.
To grow a small hand written set, seed the generator with it (needs `OPENROUTER_API_KEY`; set `OPENROUTER_URL` to use another OpenAI compatible gateway):
diff --git a/needle/__init__.py b/needle/__init__.py
index 364ac10..16c93c2 100644
--- a/needle/__init__.py
+++ b/needle/__init__.py
@@ -13,6 +13,9 @@ __all__ = ["Needle", "tool", "Field", "extract", "__version__"]
def _library_path():
from .agent import fetch
+ override = os.environ.get("NEEDLE_LIB_PATH")
+ if override:
+ return override
here = os.path.dirname(os.path.abspath(__file__))
local = os.path.join(here, fetch._lib_name())
if os.path.exists(local):
diff --git a/needle/agent/fetch.py b/needle/agent/fetch.py
index dd2cdb7..5d5e2f2 100644
--- a/needle/agent/fetch.py
+++ b/needle/agent/fetch.py
@@ -8,6 +8,14 @@ HF_REPO = "Cactus-Compute/needle2"
ENGINE_VERSION = "2.0.1"
+def _lib_name_for(tag):
+ if tag.startswith("macosx"):
+ return "libneedle.dylib"
+ if tag.startswith("win"):
+ return "libneedle.dll"
+ return "libneedle.so"
+
+
def _lib_name():
if sys.platform == "darwin":
return "libneedle.dylib"
@@ -41,13 +49,13 @@ def _platform_tag():
return family + arch
-def fetch_library(version, dest_dir):
+def fetch_library(version, dest_dir, tag=None):
from huggingface_hub import hf_hub_download
- tag = _platform_tag()
+ tag = tag or _platform_tag()
wheel = "cactus_needle-{}-py3-none-{}.whl".format(version, tag)
path = hf_hub_download(repo_id=HF_REPO, filename="python/" + wheel, repo_type="model")
- lib = _lib_name()
+ lib = _lib_name_for(tag)
with zipfile.ZipFile(path) as archive:
data = archive.read("needle/" + lib)
out = os.path.join(dest_dir, lib)
diff --git a/needle/cli.py b/needle/cli.py
index dac81bb..53cc63e 100644
--- a/needle/cli.py
+++ b/needle/cli.py
@@ -7,6 +7,13 @@ import threading
HELP = """Check the readme"""
+def _weights_spec(spec):
+ parts = [p for p in spec.split("/") if p]
+ if len(parts) < 2:
+ raise SystemExit("pass <org>/<repo>/<file>.cact or <org>/<repo>")
+ return "/".join(parts[:2]), "/".join(parts[2:]) or None
+
+
_ABSL_LOG_START = re.compile(rb"^[EIWF]\d{4} \d\d:\d\d:\d\d")
_NOISY_LOG_HEADER = re.compile(
rb"\] (?:Fusion: .*gemm_fusion|Computation: .*_computation|Delay kernel timed out)"
@@ -148,6 +155,17 @@ def main():
p.add_argument("--upload", action="store_true", help="Push the .cact to $NEEDLE_HF_REPO")
p.add_argument("--bits", type=str, default=None, choices=["2", "4"])
+ p = sub.add_parser("download", add_help=False)
+ p.add_argument("spec", type=str,
+ help="Hugging Face spec: <org>/<repo>/<file>.cact, or <org>/<repo> if it holds one archive")
+ p.add_argument("--out", type=str, default=".", help="Directory to place the archive")
+
+ p = sub.add_parser("fetch", add_help=False)
+ p.add_argument("--out", type=str, default=None,
+ help="Directory to place the engine (default: the cache)")
+ p.add_argument("--platform-tag", type=str, default=None,
+ help="Fetch the build for another device, e.g. manylinux2014_aarch64")
+
p = sub.add_parser("playground", add_help=False)
p.add_argument("--weights", type=str, default=None,
help="Tuned .cact to serve (defaults to the base model from HuggingFace)")
@@ -172,6 +190,31 @@ def main():
elif args.command == "build":
from .model.finetune import build_main
build_main(args)
+ elif args.command == "download":
+ import shutil
+ from huggingface_hub import hf_hub_download, list_repo_files
+ repo, filename = _weights_spec(args.spec)
+ if not filename:
+ cacts = [f for f in list_repo_files(repo) if f.endswith(".cact")]
+ if len(cacts) != 1:
+ raise SystemExit(f"{repo} holds {len(cacts)} .cact files, name one: "
+ + ", ".join(cacts[:5]))
+ filename = cacts[0]
+ cached = hf_hub_download(repo_id=repo, filename=filename, repo_type="model")
+ os.makedirs(args.out, exist_ok=True)
+ dest = os.path.join(args.out, os.path.basename(filename))
+ shutil.copyfile(cached, dest)
+ print(f" {'weights':<9} {dest} {os.path.getsize(dest) / 1e6:.2f} MB")
+ print(f" {'next':<9} needle.Needle(weights={dest!r}, tools=[...])")
+ elif args.command == "fetch":
+ from .agent import fetch
+ dest = args.out or os.path.join(os.path.expanduser("~"), ".cache",
+ "cactus-needle", fetch.ENGINE_VERSION)
+ os.makedirs(dest, exist_ok=True)
+ path = fetch.fetch_library(fetch.ENGINE_VERSION, dest, tag=args.platform_tag)
+ print(f" {'engine':<9} {path}")
+ print(f" {'deploy':<9} copy to ~/.cache/cactus-needle/{fetch.ENGINE_VERSION}/ "
+ f"on the device, or point NEEDLE_LIB_PATH at the file")
elif args.command == "playground":
from .playground.server import main as playground_main
playground_main(args)
diff --git a/tests/test_fetch.py b/tests/test_fetch.py
new file mode 100644
index 0000000..fc9ad9e
--- /dev/null
+++ b/tests/test_fetch.py
@@ -0,0 +1,24 @@
+def test_lib_path_env_override(tmp_path, monkeypatch):
+ import needle
+
+ fake = tmp_path / "libneedle.dylib"
+ fake.write_bytes(b"x")
+ monkeypatch.setenv("NEEDLE_LIB_PATH", str(fake))
+ assert needle._library_path() == str(fake)
+
+
+def test_weights_spec_parsing():
+ from needle.cli import _weights_spec
+
+ assert _weights_spec("acme/tuned/model.cact") == ("acme/tuned", "model.cact")
+ assert _weights_spec("acme/tuned") == ("acme/tuned", None)
+ assert _weights_spec("acme/tuned/sub/dir/m.cact") == ("acme/tuned", "sub/dir/m.cact")
+
+
+def test_lib_name_for_tags():
+ from needle.agent.fetch import _lib_name_for
+
+ assert _lib_name_for("macosx_11_0_arm64") == "libneedle.dylib"
+ assert _lib_name_for("win_amd64") == "libneedle.dll"
+ assert _lib_name_for("manylinux2014_aarch64") == "libneedle.so"
+ assert _lib_name_for("musllinux_1_2_x86_64") == "libneedle.so"