Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.
feat: Introduce support for multiple Needle engine generations and enhance telemetry
HenryNdubuakucd2f97642643
diff --git a/doc/apis.md b/doc/apis.mdindex 77ecd13..38837d2 100644--- a/doc/apis.md+++ b/doc/apis.md@@ -8,7 +8,9 @@| `agent.reset()` | Rewind the conversation, keep the tools loaded. || `needle.tool` | Decorator that turns a function into a tool schema (attached as `fn._needle_tool`). || `needle.Field(...)` | Per argument constraints, attached inline with `typing.Annotated` or passed as a default. |-| `needle.extract(text, schema, system=None, max_new_tokens=256, weights=None)` | One shot extraction. Returns a Pydantic instance if `schema` is a model, else a dict, or `None` if nothing matched. `weights` selects a tuned `.cact` and defaults to whatever the engine already has loaded. |+| `needle.extract(text, schema, system=None, max_new_tokens=256, weights=None, strict=True)` | One shot extraction. Returns a Pydantic instance if `schema` is a model, else a dict, or `None` if nothing matched. Strict mode rejects temporal values that contradict literal source years and engine-reported ungrounded values. |++When one tuned archive is active, `extract(..., weights=None)` inherits it for backwards compatibility. If both a Needle 2 and Needle 3 tune are active, pass `weights=` explicitly; the client refuses to guess between generations.## Declaring tools@@ -104,8 +106,9 @@ Needle solves every problem as a function call. The context declares what may be- Arguments contain only values evidenced by the input. An optional field with no evidence is omitted, not guessed; omission is the field-level `[]`.- `reasoning` is the model's short derivation of each argument from its source span (`'ten minutes' -> minutes 10`). It is generated unconstrained; only the call itself is grammar-constrained, so the JSON cannot be malformed while the derivation stays legible.- After you execute a call, pass the result back as the next `complete()`. The model continues from it, and later arguments may depend on earlier results: `search_for_contact` first, then `send_instant_message` with the returned `contact_id`. A final `"type": "respond"` with empty `function_calls` signals the loop is done; the answer is the tool results themselves, which `run()` collects on the final response as `results`. No free text is generated.-- A session shares one toolset. Later turns are bare queries against the same tools; `reset()` rewinds the conversation and keeps the tools loaded.-- Weights stay loaded for the process. Once a tuned `.cact` is bound, constructing or calling a base-model agent raises instead of silently answering with those weights; construct base-model agents before tuned ones, or run tuned and base workloads in separate processes. Rebinding agents on the same weights is fine.+- An agent shares one toolset. Later turns are bare queries against the same tools; `reset()` rewinds the conversation and keeps the tools loaded.+- Each tuned agent runs in its own worker process. The worker loads its `.cact` once and owns an independent engine, KV cache, and conversation without extending the native C API.+- Needle 2 and Needle 3 archives carry different format tags. The package reads the tag and loads the matching native engine; it never feeds one generation's weights to the other engine.## Extraction@@ -155,10 +158,10 @@ Calibration holds for the base model only. Fine-tuning does not update the head,## Offline devices-The engine is fetched once and cached at `~/.cache/cactus-needle/<engine version>/`. Inference itself never touches the network, so an air gapped device only needs that one file in place. Three ways to get it there:+Each generation's engine is fetched once and cached under `~/.cache/cactus-needle/v2/` or `v3/`. Inference itself never touches the network, so an air gapped device only needs the matching file in place. Three ways to get it there:-1. `needle fetch` downloads the engine for the current machine into the cache and prints the path. `--out <dir>` places it elsewhere. `--platform-tag manylinux2014_aarch64` fetches the build for a different device; tags follow the wheel names on the Hugging Face repo (`macosx_11_0_arm64`, `manylinux2014_x86_64`, `musllinux_1_2_aarch64`, `win_amd64`, `win_arm64`). For the standalone engine runner, `needle download <platform> [--out <dir>]` (platform folder names like `macos-arm64`, `linux-x86_64`; the CLI prints the full list on an invalid name) copies that platform's engine files into `<out>/<platform>/` and marks the `needle` runner executable.+1. `needle fetch --generation 2|3` downloads that engine for the current machine into the cache and prints the path. `--out <dir>` places it elsewhere. `--platform-tag manylinux2014_aarch64` fetches the build for a different device; tags follow the wheel names on the Hugging Face repo (`macosx_11_0_arm64`, `manylinux2014_x86_64`, `musllinux_1_2_aarch64`, `win_amd64`, `win_arm64`). For the standalone engine runner, `needle download <platform> --generation 2|3 [--out <dir>]` (platform folders include `macos-arm64`, `linux-x86_64`, `wasm`, and `wasm-component`; the CLI prints the full list on an invalid name) copies that platform's engine files into `<out>/<platform>/` and marks a native `needle` runner executable when present. The `wasm-component` target contains `needle.component.wasm` and the `cactus:needle/[email protected]` WIT contract.2. Copy the file to the same cache path on the device, or drop it inside the installed `needle/` package directory, which wins over the cache.-3. Set `NEEDLE_LIB_PATH=/path/to/libneedle.so` to load exactly that file and skip every lookup.+3. Set `NEEDLE2_LIB_PATH=/path/to/libneedle.so` or `NEEDLE3_LIB_PATH=/path/to/libneedle.so` to override one generation. The legacy `NEEDLE_LIB_PATH` remains a Needle 2 alias; it is deliberately ignored for Needle 3 so a v3 archive cannot be sent to a v2 engine.The Python package itself installs offline the standard way: `pip download cactus-needle` on a connected machine, then `pip install --no-index --find-links <dir> cactus-needle` on the device. On a device that must never attempt the network, also set `HF_HUB_OFFLINE=1` so a missing engine fails fast with a clear error instead of trying to download.diff --git a/doc/finetuning.md b/doc/finetuning.mdindex 4129914..1a67b5a 100644--- a/doc/finetuning.md+++ b/doc/finetuning.md@@ -37,6 +37,11 @@ needle build checkpoints/needle2.pkl --lora adapter.pkl --out tuned.cactagent = needle.Needle(tools=[...], weights="tuned.cact")```+Fine-tuning is quantization-aware by default. `--qat-bits auto` trains through+the checkpoint's declared CQ export scheme and records that scheme in the+adapter; `needle build` rejects a conflicting bit-width override. Use+`--qat-bits none` only when intentionally targeting float evaluation.+To share a tuned model, set `NEEDLE_HF_REPO=<you>/<model>` and pass `--upload` to `needle build`; on any other machine `needle download <you>/<model>/tuned.cact` pulls it back down (or pass just `<you>/<model>` when the repo holds a single archive).Defaults: batch size 16, learning rate 0.0001 with warmup and cosine decay, gradient clipping at norm 1, rank 16, alpha 32, max length 1024, validation split 0.1. The base checkpoint downloads from Hugging Face on first run.diff --git a/llms.txt b/llms.txtindex cff97b2..d7cc5c3 100644--- a/llms.txt+++ b/llms.txt@@ -203,7 +203,7 @@ Adapt by swapping the Literal values for your own and keeping the shapes: closed- `needle generate-data` / `needle finetune` / `needle build` - the fine-tuning pipeline.- `needle playground` - browser UI.- `needle fetch [--platform-tag <tag>] [--out <dir>]` - pre-download the inference engine for this machine (or another platform) into the cache; prints the path. For air-gapped devices, also see `NEEDLE_LIB_PATH` and `HF_HUB_OFFLINE` in doc/apis.md.-- `needle download <org>/<repo>[/<file>.cact] | <platform> [--out <dir>]` - pull a published `.cact` (single-archive repos need only `<org>/<repo>`), or pass a platform folder name (`macos-arm64`, `linux-x86_64`, ...) to fetch that platform's engine files (including the `needle` runner, marked executable) into `<out>/<platform>/`.+- `needle download <org>/<repo>[/<file>.cact] | <platform> [--out <dir>]` - pull a published `.cact` (single-archive repos need only `<org>/<repo>`), or pass a platform folder name (`macos-arm64`, `linux-x86_64`, `wasm`, `wasm-component`, ...) to fetch that platform's engine files into `<out>/<platform>/`. Native runners are marked executable; `wasm-component` contains the WASI Preview 2 component and WIT contract.## Common mistakes to avoiddiff --git a/needle/__init__.py b/needle/__init__.pyindex 07cb146..67c1f9b 100644--- a/needle/__init__.py+++ b/needle/__init__.py@@ -3,44 +3,83 @@ from __future__ import annotationsimport ctypesimport jsonimport os+import reimport sysimport warningsfrom .agent.tools import Field, build_schema, pydantic_schema, tool, _is_pydantic_modelfrom ._telemetry import track as _track+from ._worker import FineTuneWorker-__version__ = "2.0.8"-__all__ = ["Needle", "tool", "Field", "extract", "__version__"]+__version__ = "2.0.12"+__all__ = ["Needle", "ExtractionValidationError", "tool", "Field", "extract",+ "__version__"]-def _library_path():+class ExtractionValidationError(ValueError):+ """The engine produced structured values that are not grounded in the input."""+++_CACT_GENERATIONS = {+ 0x05E12A83: 2,+ 0x05E12A84: 3,+}+++def _weight_generation(path):+ with open(path, "rb") as handle:+ tag_bytes = handle.read(4)+ if len(tag_bytes) != 4:+ raise RuntimeError(f"{path} is not a complete .cact archive")+ tag = int.from_bytes(tag_bytes, "little")+ try:+ return _CACT_GENERATIONS[tag]+ except KeyError as exc:+ raise RuntimeError(+ f"{path} has unknown .cact format tag 0x{tag:08x}; "+ "cannot choose a compatible Needle engine") from exc+++def _library_path(generation=2):from .agent import fetch- override = os.environ.get("NEEDLE_LIB_PATH")+ generation = int(generation)+ override = os.environ.get(f"NEEDLE{generation}_LIB_PATH")+ if generation == 2 and not override:+ # NEEDLE_LIB_PATH predates multi-generation dispatch and therefore+ # names the Needle 2 engine. Never route a v3 archive through it.+ override = os.environ.get("NEEDLE_LIB_PATH")if override:return overridehere = os.path.dirname(os.path.abspath(__file__))- local = os.path.join(here, fetch._lib_name())- if os.path.exists(local):- return local- cache = os.path.join(os.path.expanduser("~"), ".cache", "cactus-needle", fetch.ENGINE_VERSION)+ lib_name = fetch._lib_name()+ stem, suffix = os.path.splitext(lib_name)+ local_names = [f"{stem}{generation}{suffix}"]+ if generation == 2:+ # Wheels published before the split shipped Needle 2 as libneedle.*.+ local_names.append(lib_name)+ for name in local_names:+ local = os.path.join(here, name)+ if os.path.exists(local):+ return local+ version = fetch.engine_version(generation)+ cache = os.path.join(os.path.expanduser("~"), ".cache", "cactus-needle",+ f"v{generation}", version)cached = os.path.join(cache, fetch._lib_name())if os.path.exists(cached):return cachedos.makedirs(cache, exist_ok=True)- return fetch.fetch_library(fetch.ENGINE_VERSION, cache)+ return fetch.fetch_library(version, cache, generation=generation)-_lib_handle = None-_active = None-_active_weights = None-_active_blob = None+_lib_handles = {}+_active = {}-def _lib():- global _lib_handle- if _lib_handle is None:- lib = ctypes.CDLL(_library_path())+def _lib(generation=2):+ generation = int(generation)+ if generation not in _lib_handles:+ lib = ctypes.CDLL(_library_path(generation))lib.needle_init.argtypes = [ctypes.c_char_p, ctypes.c_char_p, ctypes.c_char_p]lib.needle_init.restype = ctypes.c_intlib.needle_complete.argtypes = [ctypes.c_char_p, ctypes.c_int, ctypes.c_char_p, ctypes.c_int]@@ -49,14 +88,17 @@ def _lib():lib.needle_reset.restype = Nonelib.needle_load.argtypes = [ctypes.c_char_p, ctypes.c_uint64]lib.needle_load.restype = ctypes.c_int- _lib_handle = lib- return _lib_handle+ _lib_handles[generation] = lib+ return _lib_handles[generation]class Needle:def __init__(self, tools=None, system=None, weights=None, tool_index_path=None, buffer_size=65536):self._functions = {}- self._weights = weights+ self._weights = os.fspath(weights) if weights is not None else None+ self._generation = _weight_generation(self._weights) if self._weights else 2+ self._worker = None+ self._closed = Falseif weights:warnings.warn("finetuning does not update the confidence head, so scores are ""uncalibrated for tuned weights; this agent reports confidence as None",@@ -70,33 +112,28 @@ class Needle:self._n_tools = Noneself._tool_index_path = tool_index_path.encode("utf-8") if tool_index_path else Noneself._buffer = ctypes.create_string_buffer(buffer_size)- self._bind()+ if self._weights:+ self._worker = FineTuneWorker(+ _library_path(self._generation), self._weights,+ self._system.decode("utf-8"), self._tools_json.decode("utf-8"),+ os.fspath(tool_index_path) if tool_index_path else None,+ buffer_size)+ else:+ self._bind()def _bind(self):- global _active, _active_weights, _active_blob- if _active is self:+ if self._closed:+ raise RuntimeError("Needle instance is closed")+ if self._worker is not None:return- if not self._weights and _active_weights:- raise RuntimeError(- f"{_active_weights} is loaded and the engine cannot unload it, "- f"so this agent would silently answer with those weights; "- f"construct agents that want the base model before any tuned "- f"one, or run them in separate processes")- if self._weights and _active_weights != self._weights:- with open(self._weights, "rb") as handle:- blob = handle.read()- _active_blob = blob- if _lib().needle_load(blob, len(blob)) != 0:- raise RuntimeError(- f"failed to load weights from {self._weights} - the .cact "- f"format is tied to the engine version, so an archive "- f"exported by an older cactus-needle will not load; re-run "- f"`needle build` on your checkpoint with this package version")- _active_weights = self._weights- if _lib().needle_init(self._system, self._tools_json, self._tool_index_path) < 0:- _active = None+ generation = self._generation+ lib = _lib(generation)+ if _active.get(generation) is self:+ return+ if lib.needle_init(self._system, self._tools_json, self._tool_index_path) < 0:+ _active.pop(generation, None)raise RuntimeError("needle_init failed")- _active = self+ _active[generation] = selfdef _resolve(self, tools):schemas = []@@ -114,7 +151,8 @@ class Needle:return schemasdef _track_props(self):- return {"n_tools": self._n_tools, "tuned": bool(self._weights)}+ return {"n_tools": self._n_tools, "tuned": bool(self._weights),+ "generation": self._generation}def complete(self, text: str, max_new_tokens: int = 256) -> dict:_track("complete", self._track_props())@@ -122,12 +160,18 @@ class Needle:def _complete(self, text: str, max_new_tokens: int = 256) -> dict:self._bind()- rc = _lib().needle_complete(text.encode("utf-8"), int(max_new_tokens),- self._buffer, len(self._buffer))- if rc < 0:- raise RuntimeError(f"needle_complete failed (code {rc})")+ if self._worker is not None:+ raw = self._worker.complete(text, max_new_tokens)+ else:+ lib = _lib(self._generation)+ rc = lib.needle_complete(+ text.encode("utf-8"), int(max_new_tokens), self._buffer,+ len(self._buffer))+ if rc < 0:+ raise RuntimeError(f"needle_complete failed (code {rc})")+ raw = self._buffer.value.decode("utf-8")try:- response = json.loads(self._buffer.value.decode("utf-8"))+ response = json.loads(raw)except json.JSONDecodeError as err:raise RuntimeError(f"engine returned an unparseable envelope ({err}); this is an "@@ -159,13 +203,37 @@ class Needle:response["results"] = executedreturn response- def extract(self, text: str, schema: type | dict, max_new_tokens: int = 256) -> object:+ def extract(self, text: str, schema: type | dict, max_new_tokens: int = 256,+ strict: bool = True) -> object:return extract(text, schema, max_new_tokens=max_new_tokens,- weights=self._weights)+ weights=self._weights, strict=strict)def reset(self):self._bind()- _lib().needle_reset()+ if self._worker is not None:+ self._worker.reset()+ else:+ _lib(self._generation).needle_reset()++ def close(self):+ if self._worker is not None:+ try:+ self._worker.close()+ finally:+ self._worker = None+ self._closed = True++ def __enter__(self):+ return self++ def __exit__(self, exc_type, exc, traceback):+ self.close()++ def __del__(self):+ try:+ self.close()+ except Exception:+ passdef _jsonable(value):@@ -176,17 +244,111 @@ def _jsonable(value):return str(value)+def _schema_parameters(schema):+ raw = pydantic_schema(schema) if _is_pydantic_model(schema) else schema+ return raw.get("parameters", raw) if isinstance(raw, dict) else {}+++def _resolve_ref(node, root):+ seen = set()+ while isinstance(node, dict) and "$ref" in node:+ if node["$ref"] in seen:+ break+ seen.add(node["$ref"])+ target = root+ for part in node["$ref"].removeprefix("#/").split("/"):+ target = target.get(part.replace("~1", "/").replace("~0", "~"), {})+ if target is node:+ break+ node = target+ return node+++def _source_years(text):+ months = (r"(?:jan(?:uary)?|feb(?:ruary)?|mar(?:ch)?|apr(?:il)?|may|"+ r"jun(?:e)?|jul(?:y)?|aug(?:ust)?|sep(?:t(?:ember)?)?|"+ r"oct(?:ober)?|nov(?:ember)?|dec(?:ember)?)")+ patterns = [+ rf"\b\d{{1,2}}(?:st|nd|rd|th)?\s+{months}[\s,]+(\d{{1,4}})(?![0-9A-Za-z])",+ rf"\b{months}\s+\d{{1,2}}(?:st|nd|rd|th)?\s*,?\s+(\d{{1,4}})(?![0-9A-Za-z])",+ rf"\b{months}[\s,]+(\d{{3,4}})(?![0-9A-Za-z])",+ r"\byear\s+(\d{1,4})(?![0-9A-Za-z])",+ r"(?<![0-9])(\d{1,4})(?=[-/]\d{1,2}[-/]\d{1,2}(?![0-9]))",+ ]+ lowered = text.lower()+ return {int(match.group(1)) for pattern in patterns+ for match in re.finditer(pattern, lowered)}+++def _temporal_grounding(text, schema, arguments):+ root = _schema_parameters(schema)+ years = _source_years(text)+ checked, failures = set(), set()++ def walk(value, node, path):+ node = _resolve_ref(node, root)+ variants = node.get("anyOf") or node.get("oneOf") or []+ concrete = [v for v in variants if _resolve_ref(v, root).get("type") != "null"]+ if len(concrete) == 1:+ node = _resolve_ref(concrete[0], root)+ fmt = node.get("format")+ if fmt in ("date", "date-time") and isinstance(value, str):+ match = re.match(r"^(\d{4})-", value)+ if match and years:+ checked.add(path)+ if int(match.group(1)) not in years:+ failures.add(path)+ return+ if isinstance(value, dict):+ properties = node.get("properties", {})+ for key, item in value.items():+ if key in properties:+ walk(item, properties[key], f"{path}.{key}" if path else key)+ elif isinstance(value, list) and "items" in node:+ for index, item in enumerate(value):+ walk(item, node["items"], f"{path}[{index}]")++ walk(arguments, root, "")+ return checked, failures+++def _validate_extraction(text, schema, arguments, response):+ checked, failures = _temporal_grounding(text, schema, arguments)+ validation = response.get("validation") or {}+ for name in validation.get("ungrounded") or []:+ path = name.split(".", 1)[-1]+ if path not in checked or path in failures:+ failures.add(path)+ if validation.get("negation"):+ failures.add("negated request")+ if failures:+ detail = ", ".join(sorted(failures))+ raise ExtractionValidationError(+ f"extraction returned values not grounded in the input: {detail}")++def extract(text: str, schema: type | dict, system: str | None = None,- max_new_tokens: int = 256, weights: str | None = None) -> object:- """One-shot structured extraction: declare `schema` as the only tool and return- the parsed object (a Pydantic instance if `schema` is a model, else a dict).- Re-initializes the shared engine with this single schema. Defaults to whatever- weights are already loaded, since the engine cannot unload them."""- _track("extract", {"n_tools": 1, "tuned": bool(weights or _active_weights)})- agent = Needle(tools=[schema], system=system, weights=weights or _active_weights)- response = agent._complete(text, max_new_tokens)+ max_new_tokens: int = 256, weights: str | None = None,+ strict: bool = True) -> object:+ """One-shot structured extraction using the matching native engine.++ With ``strict=True`` (the default), temporal values that contradict a literal+ year in the input, plus engine-reported fabricated or negated values, raise+ :class:`ExtractionValidationError` instead of being returned silently.+ """+ selected = weights+ generation = _weight_generation(selected) if selected else 2+ _track("extract", {"n_tools": 1, "tuned": bool(selected),+ "generation": generation})+ agent = Needle(tools=[schema], system=system, weights=selected)+ try:+ response = agent._complete(text, max_new_tokens)+ finally:+ agent.close()calls = response.get("function_calls") or []if not calls:return Nonearguments = calls[0].get("arguments") or {}+ if strict:+ _validate_extraction(text, schema, arguments, response)return schema(**arguments) if _is_pydantic_model(schema) else argumentsdiff --git a/needle/_telemetry.py b/needle/_telemetry.pyindex 449e0fe..762e2f9 100644--- a/needle/_telemetry.py+++ b/needle/_telemetry.py@@ -59,16 +59,18 @@ def _get_anon_id() -> str:def _send(event: str, props: dict | None) -> None:try:from . import __version__- from .agent.fetch import ENGINE_VERSION+ from .agent.fetch import engine_version+ properties = props or {}+ generation = int(properties.get("generation", 2))payload = json.dumps({"event": event,"anon_id": _get_anon_id(),"version": __version__,- "engine": ENGINE_VERSION,+ "engine": engine_version(generation),"os": platform.system(),"arch": platform.machine(),"python": platform.python_version(),- "props": props or {},+ "props": properties,}).encode("utf-8")req = urllib.request.Request(ENDPOINT, data=payload,diff --git a/needle/_worker.py b/needle/_worker.pynew file mode 100644index 0000000..bb0ac8b--- /dev/null+++ b/needle/_worker.py@@ -0,0 +1,246 @@+from __future__ import annotations++import ctypes+import json+import os+import queue+import struct+import subprocess+import sys+import threading+++_HEADER = struct.Struct("!Q")+_EOF = object()+++def _read_exact(stream, size):+ chunks = []+ remaining = size+ while remaining:+ chunk = stream.read(remaining)+ if not chunk:+ if remaining == size:+ return None+ raise EOFError("truncated worker message")+ chunks.append(chunk)+ remaining -= len(chunk)+ return b"".join(chunks)+++def _read_message(stream):+ header = _read_exact(stream, _HEADER.size)+ if header is None:+ return None+ size = _HEADER.unpack(header)[0]+ payload = _read_exact(stream, size)+ if payload is None:+ raise EOFError("truncated worker message")+ return json.loads(payload.decode("utf-8"))+++def _write_message(stream, message):+ payload = json.dumps(message, ensure_ascii=False,+ separators=(",", ":")).encode("utf-8")+ stream.write(_HEADER.pack(len(payload)))+ stream.write(payload)+ stream.flush()+++def _load_library(path):+ lib = ctypes.CDLL(path)+ lib.needle_init.argtypes = [ctypes.c_char_p, ctypes.c_char_p,+ ctypes.c_char_p]+ lib.needle_init.restype = ctypes.c_int+ lib.needle_complete.argtypes = [ctypes.c_char_p, ctypes.c_int,+ ctypes.c_char_p, ctypes.c_int]+ lib.needle_complete.restype = ctypes.c_int+ lib.needle_reset.argtypes = []+ lib.needle_reset.restype = None+ lib.needle_load.argtypes = [ctypes.c_char_p, ctypes.c_uint64]+ lib.needle_load.restype = ctypes.c_int+ return lib+++def _child():+ protocol = os.fdopen(os.dup(sys.stdout.fileno()), "wb", buffering=0)+ os.dup2(sys.stderr.fileno(), sys.stdout.fileno())+ source = sys.stdin.buffer+ try:+ config = _read_message(source)+ if config is None:+ return 1+ lib = _load_library(config["library"])+ with open(config["weights"], "rb") as handle:+ weights = handle.read()+ if lib.needle_load(weights, len(weights)) < 0:+ raise RuntimeError(f"needle_load failed for {config['weights']}")+ system = config["system"].encode("utf-8")+ tools = config["tools"].encode("utf-8")+ index = config.get("tool_index")+ index = index.encode("utf-8") if index else None+ prefix = lib.needle_init(system, tools, index)+ if prefix < 0:+ raise RuntimeError(f"needle_init failed (code {prefix})")+ output = ctypes.create_string_buffer(int(config["buffer_size"]))+ _write_message(protocol, {"status": "ready", "prefix_tokens": prefix})+ while True:+ request = _read_message(source)+ if request is None:+ break+ operation = request.get("operation")+ if operation == "complete":+ code = lib.needle_complete(+ request["text"].encode("utf-8"),+ int(request["max_new_tokens"]), output, len(output))+ if code < 0:+ _write_message(protocol, {+ "status": "error",+ "message": f"needle_complete failed (code {code})",+ })+ else:+ _write_message(protocol, {+ "status": "ok",+ "response": output.value.decode("utf-8"),+ })+ elif operation == "reset":+ lib.needle_reset()+ _write_message(protocol, {"status": "ok"})+ elif operation == "close":+ _write_message(protocol, {"status": "ok"})+ break+ else:+ _write_message(protocol, {+ "status": "error",+ "message": f"unknown worker operation: {operation}",+ })+ return 0+ except BaseException as exc:+ try:+ _write_message(protocol, {+ "status": "fatal",+ "message": f"{type(exc).__name__}: {exc}",+ })+ except BaseException:+ pass+ return 1+ finally:+ protocol.close()+++class FineTuneWorker:+ def __init__(self, library, weights, system, tools, tool_index,+ buffer_size, startup_timeout=300):+ self._lock = threading.Lock()+ self._messages = queue.Queue()+ self._closed = False+ self._process = subprocess.Popen(+ [sys.executable, os.path.abspath(__file__), "--child"],+ stdin=subprocess.PIPE, stdout=subprocess.PIPE)+ self._reader = threading.Thread(target=self._read_messages,+ daemon=True)+ self._reader.start()+ try:+ self._send({+ "library": os.fspath(library),+ "weights": os.path.abspath(os.fspath(weights)),+ "system": system,+ "tools": tools,+ "tool_index": tool_index,+ "buffer_size": int(buffer_size),+ })+ ready = self._receive(startup_timeout)+ if ready.get("status") != "ready":+ raise RuntimeError(ready.get("message", "worker failed to start"))+ except BaseException:+ self.close()+ raise++ @property+ def pid(self):+ return self._process.pid++ def _read_messages(self):+ try:+ while True:+ message = _read_message(self._process.stdout)+ if message is None:+ break+ self._messages.put(message)+ except BaseException as exc:+ self._messages.put(exc)+ finally:+ self._messages.put(_EOF)++ def _send(self, message):+ if self._process.stdin is None:+ raise RuntimeError("Needle worker input is closed")+ try:+ _write_message(self._process.stdin, message)+ except (BrokenPipeError, OSError) as exc:+ raise RuntimeError("Needle worker exited unexpectedly") from exc++ def _receive(self, timeout=None):+ try:+ message = self._messages.get(timeout=timeout)+ except queue.Empty as exc:+ raise TimeoutError("Needle worker did not initialize in time") from exc+ if message is _EOF:+ code = self._process.poll()+ raise RuntimeError(f"Needle worker exited unexpectedly (code {code})")+ if isinstance(message, BaseException):+ raise RuntimeError("Needle worker protocol failed") from message+ return message++ def _request(self, message):+ with self._lock:+ if self._closed:+ raise RuntimeError("Needle worker is closed")+ if self._process.poll() is not None:+ raise RuntimeError(+ f"Needle worker exited unexpectedly (code {self._process.returncode})")+ self._send(message)+ response = self._receive()+ if response.get("status") != "ok":+ raise RuntimeError(response.get("message", "Needle worker failed"))+ return response++ def complete(self, text, max_new_tokens):+ return self._request({+ "operation": "complete",+ "text": text,+ "max_new_tokens": int(max_new_tokens),+ })["response"]++ def reset(self):+ self._request({"operation": "reset"})++ def close(self):+ with self._lock:+ if self._closed:+ return+ self._closed = True+ if self._process.poll() is None:+ try:+ self._send({"operation": "close"})+ self._receive(2)+ except BaseException:+ pass+ if self._process.stdin is not None:+ try:+ self._process.stdin.close()+ except OSError:+ pass+ try:+ self._process.wait(timeout=2)+ except subprocess.TimeoutExpired:+ self._process.terminate()+ try:+ self._process.wait(timeout=2)+ except subprocess.TimeoutExpired:+ self._process.kill()+ self._process.wait()+++if __name__ == "__main__":+ raise SystemExit(_child() if sys.argv[1:] == ["--child"] else 2)diff --git a/needle/agent/fetch.py b/needle/agent/fetch.pyindex 8de6379..c1736a4 100644--- a/needle/agent/fetch.py+++ b/needle/agent/fetch.py@@ -3,14 +3,24 @@ import platformimport sysimport zipfile-HF_REPO = "Cactus-Compute/needle2"--ENGINE_VERSION = "2.0.3"+ENGINE_REPOS = {+ 2: "Cactus-Compute/needle2",+ 3: "Cactus-Compute/needle3",+}+ENGINE_VERSIONS = {+ 2: "2.0.4",+ 3: "3.0.0",+}++# Backwards-compatible aliases for callers that explicitly fetch Needle 2.+HF_REPO = ENGINE_REPOS[2]+ENGINE_VERSION = ENGINE_VERSIONS[2]PLATFORMS = ("macos-arm64", "linux-x86_64", "linux-arm64", "linux-armv7","linux-riscv64", "linux-mipsel", "windows-x86_64", "windows-arm64","android-arm64", "android-armv7", "android-riscv64",- "ios-arm64", "ios-sim-arm64", "tvos-arm64", "watchos-arm64", "wasm")+ "ios-arm64", "ios-sim-arm64", "tvos-arm64", "watchos-arm64", "wasm",+ "wasm-component")def _lib_name_for(tag):@@ -54,28 +64,43 @@ def _platform_tag():return family + arch-def _register_download():+def engine_repo(generation=2):+ try:+ return ENGINE_REPOS[int(generation)]+ except (KeyError, TypeError, ValueError) as exc:+ raise ValueError(f"unsupported Needle generation: {generation}") from exc+++def engine_version(generation=2):+ try:+ return ENGINE_VERSIONS[int(generation)]+ except (KeyError, TypeError, ValueError) as exc:+ raise ValueError(f"unsupported Needle generation: {generation}") from exc+++def _register_download(generation=2):from huggingface_hub import hf_hub_downloadtry:- hf_hub_download(repo_id=HF_REPO, filename="config.json", repo_type="model",+ hf_hub_download(repo_id=engine_repo(generation), filename="config.json", repo_type="model",force_download=True)except Exception:pass-def download_platform(name, out_dir):+def download_platform(name, out_dir, generation=2):import shutilimport statfrom huggingface_hub import hf_hub_download, list_repo_files- _register_download()- files = [f for f in list_repo_files(HF_REPO) if f.startswith(name + "/")]+ repo = engine_repo(generation)+ _register_download(generation)+ files = [f for f in list_repo_files(repo) if f.startswith(name + "/")]dest = os.path.join(out_dir, name)os.makedirs(dest, exist_ok=True)out = []for f in files:- cached = hf_hub_download(repo_id=HF_REPO, filename=f, repo_type="model")+ cached = hf_hub_download(repo_id=repo, filename=f, repo_type="model")target = os.path.join(dest, os.path.basename(f))shutil.copyfile(cached, target)if os.path.basename(target) in ("needle", "needle.exe"):@@ -85,14 +110,19 @@ def download_platform(name, out_dir):return out-def fetch_library(version, dest_dir, tag=None):+def fetch_library(version=None, dest_dir=None, tag=None, generation=2):from huggingface_hub import hf_hub_download+ if dest_dir is None:+ raise TypeError("dest_dir is required")+ version = version or engine_version(generation)tag = tag or _platform_tag()wheel = "cactus_needle-{}-py3-none-{}.whl".format(version, tag)- _register_download()- path = hf_hub_download(repo_id=HF_REPO, filename="python/" + wheel, repo_type="model")+ repo = engine_repo(generation)+ _register_download(generation)+ path = hf_hub_download(repo_id=repo, filename="python/" + wheel, repo_type="model")lib = _lib_name_for(tag)+ os.makedirs(dest_dir, exist_ok=True)with zipfile.ZipFile(path) as archive:data = archive.read("needle/" + lib)out = os.path.join(dest_dir, lib)diff --git a/needle/cli.py b/needle/cli.pyindex a606c19..192c383 100644--- a/needle/cli.py+++ b/needle/cli.py@@ -148,6 +148,10 @@ def main():help="Concurrent OpenRouter requests when generating (default: 8)")p.add_argument("--checkpoint-dir", type=str, default="checkpoints")p.add_argument("--out", type=str, default=None, help="Output adapter path (.pkl)")+ p.add_argument(+ "--qat-bits", choices=["auto", "none", "2", "4"], default="auto",+ help="LoRA training numerics: auto matches the checkpoint export scheme "+ "(default), or force none/2/4")p = sub.add_parser("generate-data")p.add_argument("--tools", type=str, default=None, help="Tool schemas JSON to seed generation")@@ -171,12 +175,16 @@ def main():help="Platform folder (e.g. macos-arm64), or Hugging Face spec: ""<org>/<repo>/<file>.cact, or <org>/<repo> if it holds one archive")p.add_argument("--out", type=str, default=".", help="Directory to place the files")+ p.add_argument("--generation", type=int, choices=[2, 3], default=2,+ help="Engine generation when downloading a platform build (default: 2)")p = sub.add_parser("fetch")p.add_argument("--out", type=str, default=None,help="Directory to place the engine (default: the cache)")p.add_argument("--platform-tag", type=str, default=None,help="Fetch the build for another device, e.g. manylinux2014_aarch64")+ p.add_argument("--generation", type=int, choices=[2, 3], default=2,+ help="Needle engine generation to fetch (default: 2)")p = sub.add_parser("playground")p.add_argument("--weights", type=str, default=None,@@ -191,7 +199,8 @@ def main():sys.exit(0)from ._telemetry import track- track("cli:" + args.command)+ track("cli:" + args.command,+ {"generation": getattr(args, "generation", 2)})if args.command == "run":from .model.run import main as run_main@@ -213,7 +222,8 @@ def main():if args.spec not in fetch.PLATFORMS:raise SystemExit("unknown platform, pick one of: "+ ", ".join(fetch.PLATFORMS))- paths = fetch.download_platform(args.spec, args.out)+ paths = fetch.download_platform(args.spec, args.out,+ generation=args.generation)for path in paths:print(f" {'file':<9} {path} {os.path.getsize(path) / 1e6:.2f} MB")runner = next((p for p in paths@@ -221,7 +231,7 @@ def main():if runner:print(f" {'next':<9} {runner} --tools tools.json --serve")else:- fetch._register_download()+ fetch._register_download(args.generation)repo, filename = _weights_spec(args.spec)if not filename:cacts = [f for f in list_repo_files(repo) if f.endswith(".cact")]@@ -237,13 +247,16 @@ def main():print(f" {'next':<9} needle.Needle(weights={dest!r}, tools=[...])")elif args.command == "fetch":from .agent import fetch+ generation = args.generation+ version = fetch.engine_version(generation)dest = args.out or os.path.join(os.path.expanduser("~"), ".cache",- "cactus-needle", fetch.ENGINE_VERSION)+ "cactus-needle", f"v{generation}", version)os.makedirs(dest, exist_ok=True)- path = fetch.fetch_library(fetch.ENGINE_VERSION, dest, tag=args.platform_tag)+ path = fetch.fetch_library(version, dest, tag=args.platform_tag,+ generation=generation)print(f" {'engine':<9} {path}")- print(f" {'deploy':<9} copy to ~/.cache/cactus-needle/{fetch.ENGINE_VERSION}/ "- f"on the device, or point NEEDLE_LIB_PATH at the file")+ print(f" {'deploy':<9} copy to ~/.cache/cactus-needle/v{generation}/{version}/ "+ f"on the device, or point NEEDLE{generation}_LIB_PATH at the file")elif args.command == "playground":from .playground.server import main as playground_mainplayground_main(args)diff --git a/needle/model/finetune.py b/needle/model/finetune.pyindex fe20877..08e63f6 100644--- a/needle/model/finetune.py+++ b/needle/model/finetune.py@@ -297,6 +297,8 @@ def finetune_local(args, progress=None):import optaxfrom .run import load_checkpointfrom .architecture import SimpleAttentionNetwork+ from .quantize import (configure_deploy, cq_ste_params,+ cq_ste_mixed_params, parse_bits_map)def emit(msg):print(msg, flush=True)@@ -327,6 +329,29 @@ def finetune_local(args, progress=None):emit(f" {'data':<9} {len(seqs)} examples seq_len {max_len} cap {args.max_len}")model = SimpleAttentionNetwork(config)+ qat_mode = getattr(args, "qat_bits", "auto")+ qat_mode = "none" if qat_mode is None else str(qat_mode).lower()+ qat_bits = None+ qat_bits_map = None+ parsed_bits_map = None+ if qat_mode == "auto":+ qat_bits_map = getattr(config, "weight_bits", "") or None+ if qat_bits_map:+ parsed_bits_map = parse_bits_map(qat_bits_map)+ else:+ qat_bits = 4+ elif qat_mode in ("2", "4"):+ qat_bits = int(qat_mode)+ elif qat_mode != "none":+ raise ValueError("--qat-bits must be auto, none, 2, or 4")+ qat_enabled = qat_bits is not None or qat_bits_map is not None+ if qat_enabled:+ configure_deploy(act_bits=getattr(config, "act_bits", 8),+ kv_bits=getattr(config, "kv_bits", 8))+ scheme = f"mixed[{qat_bits_map}]" if qat_bits_map else f"W{qat_bits}"+ emit(f" {'numerics':<9} CQ {scheme} STE + A8 (matches export)")+ else:+ emit(f" {'numerics':<9} full precision")paths = lora_target_paths(params)scale = args.lora_alpha / args.lora_ranklora = init_lora(params, paths, args.lora_rank, jax.random.PRNGKey(0))@@ -352,7 +377,13 @@ def finetune_local(args, progress=None):emit(f" {'schedule':<9} {total_steps} steps warmup {warmup} cosine decay clip 1.0 (compiling...)")def loss_fn(lora, ids, mask):- logits = model.apply({"params": merge_lora(params, lora, scale)}, ids)+ merged = merge_lora(params, lora, scale)+ if qat_bits_map is not None:+ bits_map, default_bits = parsed_bits_map+ merged = cq_ste_mixed_params(merged, bits_map, default_bits)+ elif qat_bits is not None:+ merged = cq_ste_params(merged, qat_bits)+ logits = model.apply({"params": merged}, ids, quant=qat_enabled)logits, targets, mask = logits[:, :-1], ids[:, 1:], mask[:, 1:]ce = optax.softmax_cross_entropy_with_integer_labels(logits, targets)return (ce * mask).sum() / jnp.maximum(mask.sum(), 1.0)@@ -395,6 +426,8 @@ def finetune_local(args, progress=None):"scale": float(scale),"base": base_path,"rank": args.lora_rank,+ "qat_bits": qat_bits,+ "qat_bits_map": qat_bits_map,}, handle)print(f" {'adapter':<9} {out}")print(f" {'next':<9} needle build {base_path} --lora {out}")@@ -409,6 +442,8 @@ def build_main(args):params, config, _ = load_checkpoint(args.checkpoint, return_run=True)+ adapter_qat_bits = None+ adapter_qat_bits_map = Noneif args.lora:with open(args.lora, "rb") as handle:adapter = pickle.load(handle)@@ -416,11 +451,29 @@ def build_main(args):for key, v in adapter["lora"].items()}params = merge_lora(params, lora, adapter["scale"])print(f" {'merged':<9} {len(lora)} weight groups {args.lora}")+ adapter_qat_bits = adapter.get("qat_bits")+ adapter_qat_bits_map = adapter.get("qat_bits_map")bits = args.bits- bits_map = None if bits else (getattr(config, "weight_bits", "") or None)- if not bits and not bits_map:- bits = "4"+ if adapter_qat_bits_map is not None:+ if bits is not None:+ raise ValueError(+ "adapter was trained for the checkpoint's mixed CQ bit map, but "+ f"--bits {bits} would deploy different numerics")+ bits_map = adapter_qat_bits_map+ print(f" {'scheme':<9} CQ mixed[{bits_map}] from QAT adapter metadata")+ elif adapter_qat_bits is not None:+ if bits is not None and int(bits) != int(adapter_qat_bits):+ raise ValueError(+ f"adapter was trained for CQ W{adapter_qat_bits}, but --bits {bits} "+ "would deploy different numerics; rebuild at the adapter bit width")+ bits = str(adapter_qat_bits)+ bits_map = None+ print(f" {'scheme':<9} CQ W{bits} from QAT adapter metadata")+ else:+ bits_map = None if bits else (getattr(config, "weight_bits", "") or None)+ if not bits and not bits_map:+ bits = "4"out = args.out or (os.path.splitext(os.path.basename(args.checkpoint))[0] + ".cact")info = write_export(params, config, out,diff --git a/pyproject.toml b/pyproject.tomlindex 3c33134..6d1e95a 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -1,6 +1,6 @@[project]name = "cactus-needle"-version = "2.0.8"+version = "2.0.12"description = "A 14MB foundation tool-calling model for tiny devices: inference, LoRA finetuning, and build."requires-python = ">=3.9"readme = "README.md"diff --git a/tests/test_fetch.py b/tests/test_fetch.pyindex fc9ad9e..9c579f7 100644--- a/tests/test_fetch.py+++ b/tests/test_fetch.py@@ -7,6 +7,24 @@ def test_lib_path_env_override(tmp_path, monkeypatch):assert needle._library_path() == str(fake)+def test_legacy_lib_override_cannot_capture_v3(tmp_path, monkeypatch):+ import needle+ from needle.agent import fetch++ v2 = tmp_path / "libneedle-v2.dylib"+ v3 = tmp_path / "libneedle-v3.dylib"+ v2.write_bytes(b"v2")+ v3.write_bytes(b"v3")+ monkeypatch.setenv("NEEDLE_LIB_PATH", str(v2))+ monkeypatch.setenv("NEEDLE3_LIB_PATH", str(v3))+ assert needle._library_path(2) == str(v2)+ assert needle._library_path(3) == str(v3)++ monkeypatch.delenv("NEEDLE3_LIB_PATH")+ monkeypatch.setattr(fetch, "fetch_library", lambda *args, **kwargs: str(v3))+ assert needle._library_path(3) == str(v3)++def test_weights_spec_parsing():from needle.cli import _weights_spec@@ -22,3 +40,25 @@ def test_lib_name_for_tags():assert _lib_name_for("win_amd64") == "libneedle.dll"assert _lib_name_for("manylinux2014_aarch64") == "libneedle.so"assert _lib_name_for("musllinux_1_2_x86_64") == "libneedle.so"+++def test_component_platform_is_downloadable():+ from needle.agent.fetch import PLATFORMS++ assert "wasm-component" in PLATFORMS+++def test_fetch_library_creates_destination(tmp_path, monkeypatch):+ import zipfile+ from needle.agent import fetch++ wheel = tmp_path / "engine.whl"+ with zipfile.ZipFile(wheel, "w") as archive:+ archive.writestr("needle/libneedle.so", b"engine")+ monkeypatch.setattr(fetch, "_register_download", lambda generation: None)+ monkeypatch.setattr("huggingface_hub.hf_hub_download",+ lambda **kwargs: str(wheel))+ out = fetch.fetch_library("2.0.4", tmp_path / "new", tag="manylinux2014_x86_64")++ assert (tmp_path / "new" / "libneedle.so").read_bytes() == b"engine"+ assert out == str(tmp_path / "new" / "libneedle.so")diff --git a/tests/test_finetune.py b/tests/test_finetune.pyindex 20067ed..9d7a95b 100644--- a/tests/test_finetune.py+++ b/tests/test_finetune.py@@ -24,11 +24,11 @@ def _write_data(path):handle.write(json.dumps(row) + "\n")-def _finetune_args(data, checkpoint, out, ckpt_dir):+def _finetune_args(data, checkpoint, out, ckpt_dir, qat_bits="auto"):return types.SimpleNamespace(jsonl_path=str(data), checkpoint=checkpoint, epochs=1, batch_size=2,lr=1e-3, lora_rank=4, lora_alpha=8.0, max_len=64, generate=0,- model=None, checkpoint_dir=str(ckpt_dir), out=str(out))+ model=None, checkpoint_dir=str(ckpt_dir), out=str(out), qat_bits=qat_bits)def test_finetune_writes_adapter(tiny_checkpoint, tmp_path):@@ -38,20 +38,28 @@ def test_finetune_writes_adapter(tiny_checkpoint, tmp_path):_write_data(data)out = tmp_path / "adapter.pkl"progress = []- finetune_local(_finetune_args(data, tiny_checkpoint, out, tmp_path / "ck"),+ finetune_local(_finetune_args(data, tiny_checkpoint, out, tmp_path / "ck", qat_bits=4),progress=progress.append)assert any("loss" in m for m in progress)+ assert any("CQ W4 STE + A8" in m for m in progress)assert out.exists()with open(out, "rb") as handle:adapter = pickle.load(handle)assert adapter["rank"] == 4assert abs(adapter["scale"] - 2.0) < 1e-6assert adapter["base"] == tiny_checkpoint+ assert adapter["qat_bits"] == 4assert adapter["lora"]for value in adapter["lora"].values():assert "A" in value and "B" in value+ from needle.model.finetune import build_main+ with pytest.raises(ValueError, match="trained for CQ W4"):+ build_main(types.SimpleNamespace(checkpoint=tiny_checkpoint, lora=str(out),+ out=str(tmp_path / "wrong.cact"),+ upload=False, bits="2"))+def test_finetune_then_build_merges(tiny_checkpoint, tmp_path):from needle.model.finetune import finetune_local, build_main@@ -68,3 +76,41 @@ def test_finetune_then_build_merges(tiny_checkpoint, tmp_path):assert os.path.exists(out)header, _ = read_export(out)assert header["num_tensors"] > 0+++def test_auto_qat_preserves_checkpoint_mixed_bit_map(tiny_checkpoint, tmp_path):+ from needle.model.finetune import finetune_local, build_main+ from needle.model.export import read_export++ with open(tiny_checkpoint, "rb") as handle:+ checkpoint = pickle.load(handle)+ bit_map = "embedding=4,mhc=4,default=2"+ checkpoint["config"]["weight_bits"] = bit_map+ mixed_checkpoint = tmp_path / "mixed.pkl"+ with open(mixed_checkpoint, "wb") as handle:+ pickle.dump(checkpoint, handle)++ data = tmp_path / "data.jsonl"+ _write_data(data)+ adapter_path = tmp_path / "mixed-adapter.pkl"+ progress = []+ finetune_local(_finetune_args(data, str(mixed_checkpoint), adapter_path,+ tmp_path / "ck"), progress=progress.append)+ assert any(f"mixed[{bit_map}]" in message for message in progress)+ with open(adapter_path, "rb") as handle:+ adapter = pickle.load(handle)+ assert adapter["qat_bits"] is None+ assert adapter["qat_bits_map"] == bit_map++ out = tmp_path / "mixed.cact"+ build_main(types.SimpleNamespace(checkpoint=str(mixed_checkpoint),+ lora=str(adapter_path), out=str(out),+ upload=False, bits=None))+ header, tensors = read_export(out)+ assert header["num_tensors"] == len(tensors)++ with pytest.raises(ValueError, match="mixed CQ bit map"):+ build_main(types.SimpleNamespace(checkpoint=str(mixed_checkpoint),+ lora=str(adapter_path),+ out=str(tmp_path / "wrong.cact"),+ upload=False, bits="4"))diff --git a/tests/test_weights.py b/tests/test_weights.pyindex d00594a..437a8f1 100644--- a/tests/test_weights.py+++ b/tests/test_weights.py@@ -29,23 +29,56 @@ class _Stub:self.calls.append("reset")+class _WorkerStub:+ next_pid = 1000++ def __init__(self, calls, library, weights, system, tools, tool_index,+ buffer_size):+ self.calls = calls+ self.pid = type(self).next_pid+ type(self).next_pid += 1+ self.calls.append(("worker_start", self.pid, library, weights))++ def complete(self, text, max_new_tokens):+ self.calls.append(("worker_complete", self.pid, text))+ return ENVELOPE.decode("utf-8")++ def reset(self):+ self.calls.append(("worker_reset", self.pid))++ def close(self):+ self.calls.append(("worker_close", self.pid))++@pytest.fixturedef engine(monkeypatch):import needlecalls = []- stub = _Stub(calls)- monkeypatch.setattr(needle, "_lib", lambda: stub)- monkeypatch.setattr(needle, "_active", None)- monkeypatch.setattr(needle, "_active_weights", None)- monkeypatch.setattr(needle, "_active_blob", None)+ stubs = {2: _Stub(calls), 3: _Stub(calls)}+ monkeypatch.setattr(needle, "_lib", lambda generation=2: stubs[generation])+ monkeypatch.setattr(needle, "_library_path",+ lambda generation=2: f"/tmp/libneedle{generation}")+ monkeypatch.setattr(+ needle, "FineTuneWorker",+ lambda library, weights, system, tools, tool_index, buffer_size:+ _WorkerStub(calls, library, weights, system, tools, tool_index,+ buffer_size))+ monkeypatch.setattr(needle, "_active", {})return calls@pytest.fixturedef tuned(tmp_path):path = tmp_path / "tuned.cact"- path.write_bytes(b"tuned weights")+ path.write_bytes((0x05E12A83).to_bytes(4, "little") + b"tuned weights")+ return str(path)+++def tuned_v3(tmp_path):+ path = tmp_path / "tuned-v3.cact"+ path.write_bytes((0x05E12A84).to_bytes(4, "little") + b"tuned weights")return str(path)@@ -57,40 +90,115 @@ def _tuned_agent(path):return needle.Needle(tools="[]", weights=path)-def test_new_base_agent_refuses_once_tuned_weights_are_loaded(engine, tuned):+def test_base_agent_and_tuned_worker_coexist(engine, tuned):import needle- _tuned_agent(tuned)- with pytest.raises(RuntimeError, match="cannot unload"):- needle.Needle(tools="[]")+ tuned_agent = _tuned_agent(tuned)+ base = needle.Needle(tools="[]")+ tuned_agent.complete("tuned")+ base.complete("base")+ assert any(call[0] == "worker_complete" for call in engine if isinstance(call, tuple))+ assert "complete" in engine-def test_existing_base_agent_refuses_instead_of_answering_on_tuned_weights(engine, tuned):+def test_tuned_worker_does_not_rebind_existing_base_agent(engine, tuned):import needlebase = needle.Needle(tools="[]")base.complete("hello")- _tuned_agent(tuned)- with pytest.raises(RuntimeError, match="cannot unload"):- base.complete("hello")+ tuned_agent = _tuned_agent(tuned)+ tuned_agent.complete("hello")+ base.complete("again")+ assert engine.count("load") == 0+ assert engine.count("complete") == 2-def test_tuned_agent_rebinds_without_reloading_its_own_weights(engine, tuned):+def test_tuned_agent_loads_one_worker_and_reuses_it(engine, tuned):agent = _tuned_agent(tuned)- other = _tuned_agent(tuned)- other.complete("hello")- agent.complete("hello")- assert engine.count("load") == 1+ agent.complete("one")+ agent.complete("two")+ starts = [call for call in engine+ if isinstance(call, tuple) and call[0] == "worker_start"]+ assert len(starts) == 1+ assert len([call for call in engine+ if isinstance(call, tuple) and call[0] == "worker_complete"]) == 2-def test_extract_inherits_the_loaded_weights(engine, tuned):+def test_extract_uses_an_explicit_tuned_worker(engine, tuned):import needle- _tuned_agent(tuned)schema = {"name": "City", "parameters": {"type": "object","properties": {"city": {"type": "string"}}}}with warnings.catch_warnings():warnings.simplefilter("ignore")- assert needle.extract("the weather in Paris", schema) == {"city": "Paris"}- assert needle._active_weights == tuned- assert engine.count("load") == 1+ assert needle.extract("the weather in Paris", schema, weights=tuned) == {+ "city": "Paris"}+ assert any(call[0] == "worker_start" and call[3] == tuned+ for call in engine if isinstance(call, tuple))+++def test_v2_and_v3_weights_route_to_independent_engines(engine, tuned, tuned_v3,+ monkeypatch):+ import needle++ routed = []+ monkeypatch.setattr(needle, "_library_path",+ lambda generation=2: routed.append(generation) or+ f"/tmp/libneedle{generation}")++ v2 = _tuned_agent(tuned)+ v3 = _tuned_agent(tuned_v3)+ assert v2._generation == 2+ assert v3._generation == 3+ assert routed == [2, 3]+++def test_unknown_archive_tag_is_rejected_before_loading(engine, tmp_path):+ path = tmp_path / "unknown.cact"+ path.write_bytes(b"NOPE")+ with pytest.raises(RuntimeError, match="unknown .cact format tag"):+ _tuned_agent(str(path))+++def test_two_finetunes_use_independent_workers(engine, tuned, tmp_path):+ other = tmp_path / "other.cact"+ other.write_bytes((0x05E12A83).to_bytes(4, "little") + b"other weights")++ first = _tuned_agent(tuned)+ second = _tuned_agent(str(other))+ assert first._worker.pid != second._worker.pid+ first.complete("one")+ second.complete("two")+ assert ("worker_complete", first._worker.pid, "one") in engine+ assert ("worker_complete", second._worker.pid, "two") in engine+ first.close()+ second.close()+++def test_extraction_rejects_fabricated_temporal_year():+ import datetime+ import needle+ import pydantic++ class Invoice(pydantic.BaseModel):+ due_date: datetime.datetime++ response = {"validation": {"ungrounded": ["Invoice.due_date"]}}+ correct = {"due_date": "2034-09-05T00:00:00Z"}+ needle._validate_extraction("due on 5th September 2034", Invoice,+ correct, response)++ fabricated = {"due_date": "2024-09-05T00:00:00Z"}+ with pytest.raises(needle.ExtractionValidationError, match="due_date"):+ needle._validate_extraction("due on 5th September 2034", Invoice,+ fabricated, response)++ with pytest.raises(needle.ExtractionValidationError, match="due_date"):+ needle._validate_extraction(+ "due on 5th September 42", Invoice,+ {"due_date": "2026-09-05T00:00:00Z"}, response)++ assert needle._source_years("due September 5, 2034") == {2034}+ assert needle._source_years("due September 5") == set()+ assert needle._source_years("due 5th September 42") == {42}+ assert needle._source_years("Invoice 42 is due tomorrow at 5") == set()diff --git a/tests/test_worker.py b/tests/test_worker.pynew file mode 100644index 0000000..c853e5c--- /dev/null+++ b/tests/test_worker.py@@ -0,0 +1,123 @@+import io+import json+import shutil+import subprocess+import sys+import warnings++import pytest++from needle._worker import FineTuneWorker, _read_message, _write_message+++def test_worker_protocol_round_trip():+ stream = io.BytesIO()+ message = {"operation": "complete", "text": "héllo\nworld",+ "max_new_tokens": 17}+ _write_message(stream, message)+ stream.seek(0)+ assert _read_message(stream) == message+++def stub_engine(tmp_path):+ compiler = shutil.which("cc")+ if compiler is None or sys.platform == "win32":+ pytest.skip("a C compiler is required for the worker integration test")+ source = tmp_path / "stub.c"+ source.write_text(r'''+#include <stdint.h>+#include <stdio.h>++static int model_id;+static int resets;++int needle_load(const unsigned char* data, unsigned long long size) {+ if (!data || size < 6 || data[4] == 0) return -1;+ model_id = data[4];+ return 0;+}++int needle_init(const char* system, const char* tools, const char* index) {+ (void)system;+ (void)tools;+ (void)index;+ return 7;+}++int needle_complete(const char* input, int max_new_tokens, char* output,+ int capacity) {+ (void)max_new_tokens;+ snprintf(output, (size_t)capacity,+ "{\"type\":\"text\",\"model\":%d,\"resets\":%d,"+ "\"input\":\"%s\"}", model_id, resets, input);+ return 1;+}++void needle_reset(void) {+ resets++;+}+''')+ if sys.platform == "darwin":+ library = tmp_path / "libstub.dylib"+ command = [compiler, "-dynamiclib", str(source), "-o", str(library)]+ else:+ library = tmp_path / "libstub.so"+ command = [compiler, "-shared", "-fPIC", str(source), "-o", str(library)]+ subprocess.run(command, check=True)+ return library+++def _weights(path, model_id):+ path.write_bytes((0x05E12A83).to_bytes(4, "little") ++ bytes([model_id, 1]))+ return path+++def test_finetunes_run_in_independent_processes(stub_engine, tmp_path):+ first_path = _weights(tmp_path / "first.cact", 11)+ second_path = _weights(tmp_path / "second.cact", 29)+ first = FineTuneWorker(stub_engine, first_path, "", "[]", None, 4096)+ second = FineTuneWorker(stub_engine, second_path, "", "[]", None, 4096)+ try:+ assert first.pid != second.pid+ assert json.loads(first.complete("first", 8))["model"] == 11+ assert json.loads(second.complete("second", 8))["model"] == 29+ first.reset()+ assert json.loads(first.complete("again", 8))["resets"] == 1+ assert json.loads(second.complete("again", 8))["resets"] == 0+ finally:+ first.close()+ second.close()+ assert first._process.poll() is not None+ assert second._process.poll() is not None+++def test_tuned_agents_use_independent_workers(stub_engine, tmp_path,+ monkeypatch):+ import needle++ monkeypatch.setattr(needle, "_library_path",+ lambda generation=2: str(stub_engine))+ first_path = _weights(tmp_path / "agent-first.cact", 7)+ second_path = _weights(tmp_path / "agent-second.cact", 13)+ with warnings.catch_warnings():+ warnings.simplefilter("ignore")+ first = needle.Needle(tools="[]", weights=first_path)+ second = needle.Needle(tools="[]", weights=second_path)+ try:+ assert first._worker.pid != second._worker.pid+ assert first.complete("first")["model"] == 7+ assert second.complete("second")["model"] == 13+ first.reset()+ assert first.complete("again")["resets"] == 1+ assert second.complete("again")["resets"] == 0+ finally:+ first.close()+ second.close()+++def test_worker_propagates_native_load_failure(stub_engine, tmp_path):+ bad = _weights(tmp_path / "bad.cact", 0)+ with pytest.raises(RuntimeError, match="needle_load failed"):+ FineTuneWorker(stub_engine, bad, "", "[]", None, 4096)