Skip to content

Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.

Implement platform integration for fine-tuning and model management

HenryNdubuakuf189b23ebf34
diff --git a/README.md b/README.md
index f5f574a..f39432d 100644
--- a/README.md
+++ b/README.md
@@ -62,13 +62,44 @@ Needle was designed to be customised. Its capacity is a ladder, and a subnetwork
![Every subnetwork before and after fine-tuning on DroidCall and on Mobile Actions](assets/finetune.svg)
+Two ways to fine-tune, from the same package:
+
+| | Local, `needle finetune` | Platform, `needle platform finetune` |
+| --- | --- | --- |
+| What trains | LoRA adapters on the attention projections, base frozen, merged at export | The full model, every depth from 2 layers up |
+| What it keeps | Your data only | Your data reinforced with Needle's original dataset, so nothing already learned is unlearned |
+| Confidence | Head untouched; `confidence` is `None` | Head fine-tuned with the model, calibrated on your tools |
+| Precision | 4-bit | 2-bit, the same post-training as the shipped model |
+| Data | Your JSONL, `query`/`answers` or chat format | Yours, or generated from your tool definitions, 100 to 10,000 examples per run |
+| Scores | Validation loss | Validation and test accuracy for every depth |
+| Compute | Your machine, JAX on CPU, CUDA or Metal | Cactus GPUs |
+| Runs from | The CLI | The CLI, Python, the [dashboard](https://cactuscompute.com/dashboard), or a coding agent holding your key |
+
+Local:
+
```sh
pip install "cactus-needle[train]"
needle finetune data.jsonl --epochs 10 --out adapter.safetensors
needle build --lora adapter.safetensors --layers 8 --out tuned.cact
```
-Local fine-tuning trains and exports at 4 bits; the [fine-tuning guide](https://cactuscompute.com/blog/finetuning-needle) has the rest. The 2-bit post-training and quantisation behind the shipped model, enriched with Cactus proprietary datasets, run on the [Cactus Platform](https://cactuscompute.com/dashboard).
+Platform, with a key from the [console](https://cactuscompute.com/dashboard/api-keys) in `NEEDLE_API_KEY`. One command uploads the files, trains and scores every size, and downloads the `.cact` files; once a job is submitted it can also be followed on the dashboard:
+
+```sh
+export NEEDLE_API_KEY=needle_ft_...
+needle platform generate --tools tools.json --examples 1000 --out ./data
+needle platform finetune data/train.jsonl data/validation.jsonl data/test.jsonl --suffix smart-home --out ./models
+```
+
+```python
+from needle.platform import Platform
+
+client = Platform()
+job = client.wait(client.finetune(["train.jsonl"], ["validation.jsonl"], ["test.jsonl"], suffix="smart-home"))
+paths = client.download(job["fine_tuned_model"], "models", depth=8)
+```
+
+Or hand the key to Claude Code or Codex with [cactuscompute.com/llms.txt](https://cactuscompute.com/llms.txt) and let the agent run the loop. `needle platform jobs | models | files | billing` list what the account holds, `needle download model-<id>` fetches a model by id, and the [fine-tuning guide](https://cactuscompute.com/blog/finetuning-needle) covers the data format and how to read the scores.
## Deploy
diff --git a/llms.txt b/llms.txt
index 3d8b5f8..24f3400 100644
--- a/llms.txt
+++ b/llms.txt
@@ -128,7 +128,7 @@ if r["type"] == "call":
## Confidence gating
-`confidence` is the min of a post-hoc calibration head and the decode probability of the call. Calibration holds for the base model only: an agent constructed with `weights=` reports `confidence` as None (fine-tuning does not update the head). Pick a threshold per product; act at/above it, escalate below it:
+`confidence` is the min of a post-hoc calibration head and the decode probability of the call. The head ships inside the archive when it was trained for those weights: the base model and platform fine-tunes have it, so they report a calibrated score; local `needle finetune` does not train it, `needle build --lora` leaves it out, and an agent constructed with such weights reports `confidence` as None. Pick a threshold per product; act at/above it, escalate below it:
```python
r = agent.complete(user_text)
@@ -187,6 +187,31 @@ agent.run("...")
Options: `finetune --lora-rank 16 --lora-alpha 32 --lr 1e-4 --batch-size 16 --max-len 1024 --val-split 0.1 --checkpoint <base.safetensors or .pkl> --checkpoint-dir <dir> --out <adapter.safetensors or .pkl> --generate <n> --model <id> --workers <n>`; `--model` defaults to `deepseek/deepseek-flash-latest`, `--workers` defaults to `8`, and `--checkpoint-dir` defaults to `checkpoints`. `--generate` synthesizes extra examples through the configured OpenRouter endpoint before training. `build --bits 2|4 --upload` (with `NEEDLE_HF_REPO=<you>/<model>`).
+## Platform (hosted fine-tuning)
+
+The hosted platform at https://cactuscompute.com/v1 trains every size from 2 layers up on GPUs, scores each on your test file, and serves the `.cact` files; the API reference is https://cactuscompute.com/llms/api. The package drives it with a key from https://cactuscompute.com/dashboard/api-keys in `NEEDLE_API_KEY` (never print it). Files are chat-format JSONL: `messages` (OpenAI chat turns, tool calls on the assistant turn as `tool_calls` with `arguments` as a JSON string) and OpenAI-form `tools`.
+
+```sh
+needle platform finetune train.jsonl validation.jsonl test.jsonl [--max-depth N] [--suffix name] [--out dir] [--depth N] [--no-wait]
+needle platform generate --tools tools.json --examples 1000 [--description "..."] [--message "..."] [--suffix name] [--out dir]
+needle platform jobs [ftjob-id [--wait] [--out dir] [--depth N]] | models [model-id] | files | billing
+needle download model-<id> [--depth N] [--out dir]
+```
+
+```python
+from needle.platform import Platform, PlatformError
+client = Platform() # key from NEEDLE_API_KEY, or Platform(api_key=...)
+client.billing() # plan, limits, usage; a fine-tune spends one job at submission
+job = client.finetune(["train.jsonl"], ["validation.jsonl"], ["test.jsonl"], max_depth=20, suffix="smart-home")
+job = client.wait(job, poll=10) # raises PlatformError(code) if the job fails
+job["evaluations"] # [{depth, validation: {correct, total, excluded}, test: {...}}, ...]
+client.download(job["fine_tuned_model"], "models") # every size as <name>-<depth>L.cact; depth=8 for one
+gen = client.wait(client.generate(tools, 1000, description="...", messages=["..."], suffix="smart-home"))
+client.download_file(gen["files"][0]["id"], "train.jsonl")
+```
+
+Submission prints the job's dashboard page, https://cactuscompute.com/dashboard/jobs/<id>; Ctrl-C leaves the job running and `needle platform jobs <id> --wait --out dir` resumes the wait and the download. `PlatformError.code` is the API's stable code (`unauthorized`, `subscription_required`, `quota_exceeded`, `job_active`, `rate_limited`, `invalid_request`, ...), with `.param` and `.url` when the API sends them; `rate_limited` is retried after `Retry-After` automatically. Ask before every `finetune` or `generate`: both spend allowance at submission. Platform files also train locally: `needle finetune train.jsonl` reads single-turn chat-format lines as well as the `query`/`answers` form.
+
## Playground (browser)
Hosted at https://cactuscompute.com/needle: preset demos, editable tools/prompt, multi-turn follow-ups, and a "Finetune on these tools" button that runs the pipeline above and returns a downloadable `.cact`. `needle playground [--weights my.cact]` serves the same UI locally at http://127.0.0.1:7860. Local fine-tuning is 4-bit; the 2-bit post-training and quantisation behind the shipped model, enriched with Cactus proprietary datasets, run on the Cactus Platform at https://cactuscompute.com/dashboard.
@@ -208,7 +233,8 @@ Adapt by swapping the Literal values for your own and keeping the shapes: closed
- `needle generate-data` / `needle finetune` / `needle build [checkpoint] [--lora adapter] [--layers N] [--platform <folder>] [--out <path>]` - the fine-tuning pipeline; fine-tuning runs at the full 20 layers, `build` merges the adapter, slices the `--layers` rung (default 20), exports at 4 bits, and fetches the published `needle3.cact` on every run (without an adapter at full depth it ships that published 2-bit archive as is). `--platform` also downloads that platform's engine and header and makes `--out` a folder holding them plus `needle3.cact`.
- `needle playground` - the browser UI locally (hosted at https://cactuscompute.com/needle).
- `needle fetch [--platform-tag <tag>] [--out <dir>]` - pre-download the inference engine for this machine (or another platform) into the cache; prints the path. For air-gapped devices, also see `NEEDLE3_LIB_PATH` and `HF_HUB_OFFLINE` at https://cactuscompute.com/blog/needle-python-docs.
-- `needle download needle3 | needle3.safetensors | <org>/<repo>[/<file>.cact] | <platform> [--out <dir>]` - pull the base Needle 3 archive, a checkpoint to fine-tune, a published `.cact` (single-archive repos need only `<org>/<repo>`), or pass a platform folder name (`macos-arm64`, `linux-x86_64`, `wasm`, `wasm-component`, ...) to fetch that platform's engine files into `<out>/<platform>/`. Native runners are marked executable; `wasm-component` contains the WASI Preview 2 component and WIT contract.
+- `needle platform finetune | generate | jobs | models | files | billing` - the hosted platform from the CLI (section above).
+- `needle download needle3 | needle3.safetensors | model-<id> | <org>/<repo>[/<file>.cact] | <platform> [--out <dir>]` - pull the base Needle 3 archive, a checkpoint to fine-tune, a platform-trained model by id (`--depth N` for one size), a published `.cact` (single-archive repos need only `<org>/<repo>`), or pass a platform folder name (`macos-arm64`, `linux-x86_64`, `wasm`, `wasm-component`, ...) to fetch that platform's engine files into `<out>/<platform>/`. Native runners are marked executable; `wasm-component` contains the WASI Preview 2 component and WIT contract.
## Common mistakes to avoid
diff --git a/needle/__init__.py b/needle/__init__.py
index cd24b20..9d3e7e5 100644
--- a/needle/__init__.py
+++ b/needle/__init__.py
@@ -41,6 +41,40 @@ def _weight_generation(path):
"cannot choose a compatible Needle engine") from exc
+_CACT_HEADER = "<48If"
+_CACT_RECORD = "<BBHIIIIQQII"
+_CACT_FP16, _CACT_RAW = 1, 4
+_CONFIDENCE_HEAD_CODE = 2
+
+
+def _confidence_head_present(path):
+ """Whether a Needle 3 archive carries a confidence head, read from its head manifest."""
+ import struct
+
+ with open(path, "rb") as handle:
+ header = handle.read(struct.calcsize(_CACT_HEADER))
+ if len(header) < struct.calcsize(_CACT_HEADER):
+ return False
+ fields = struct.unpack(_CACT_HEADER, header)
+ num_tensors, codebook = fields[1], fields[2]
+ handle.seek(codebook * 4, 1)
+ size = struct.calcsize(_CACT_RECORD)
+ records = [struct.unpack(_CACT_RECORD, handle.read(size)) for _ in range(num_tensors)]
+ if not records or records[-1][0] != _CACT_RAW:
+ return False
+ for heads in (1, 2, 3):
+ index = num_tensors - 2 - 6 * heads
+ if index < 0:
+ continue
+ dtype, ndim, _, length = records[index][:4]
+ if dtype != _CACT_FP16 or ndim != 1 or length != heads:
+ continue
+ handle.seek(records[index][7])
+ codes = struct.unpack(f"<{heads}e", handle.read(2 * heads))
+ return _CONFIDENCE_HEAD_CODE in {int(round(c)) for c in codes}
+ return False
+
+
def _library_path(generation=2):
from .agent import fetch
@@ -132,9 +166,11 @@ class Needle:
self._generation = int(generation or 3)
self._worker = None
self._closed = False
- if self._tuned:
- warnings.warn("finetuning does not update the confidence head, so scores are "
- "uncalibrated for tuned weights; this agent reports confidence as None",
+ self._calibrated = (not self._tuned
+ or (self._generation >= 3 and _confidence_head_present(self._weights)))
+ if not self._calibrated:
+ warnings.warn("these weights carry no confidence head trained for them, so this "
+ "agent reports confidence as None; platform fine-tunes keep the head",
stacklevel=2)
self._system_text = _with_date_fact(system or "") if auto_date else (system or "")
self._system = self._system_text.encode("utf-8")
@@ -220,7 +256,7 @@ class Needle:
raise RuntimeError(
f"engine returned an unparseable envelope ({err}); this is an "
f"engine bug - please report it with the prompt and schema") from err
- if self._tuned:
+ if not self._calibrated:
response["confidence"] = None
if ground:
_annotate_ungrounded(response, self._tool_schemas, self._seen_years,
diff --git a/needle/cli.py b/needle/cli.py
index bc649f6..89a6859 100644
--- a/needle/cli.py
+++ b/needle/cli.py
@@ -10,7 +10,8 @@ HELP = """usage: needle <command> [options]
finetune train a LoRA adapter on JSONL data (--layers N for a rung)
generate-data synthesise training data via OpenRouter
build export a checkpoint (+ adapter) to a .cact archive
- download needle3 | needle3.safetensors | <platform> | <org>/<repo>[/<file>.cact]
+ platform fine-tune, generate data and download models on cactuscompute.com
+ download needle3 | needle3.safetensors | <platform> | model-<id> | <org>/<repo>[/<file>.cact]
fetch fetch the engine library for this platform
playground serve the browser playground
@@ -31,6 +32,8 @@ def _download_target(spec):
if "/" in spec:
return "hub", spec
+ if spec.startswith("model-"):
+ return "hosted", spec
if spec in fetch.PLATFORMS:
return "platform", spec
name = spec[:-5] if spec.endswith(".cact") else spec
@@ -41,7 +44,8 @@ def _download_target(spec):
raise SystemExit(
f"unknown download {spec!r}: pass needle3 (base weights), needle3.safetensors "
f"(the checkpoint to fine-tune), a platform ("
- + ", ".join(fetch.PLATFORMS) + "), or <org>/<repo>[/<file>.cact]")
+ + ", ".join(fetch.PLATFORMS) + "), model-<id> from cactuscompute.com, "
+ "or <org>/<repo>[/<file>.cact]")
_ABSL_LOG_START = re.compile(rb"^[EIWF]\d{4} \d\d:\d\d:\d\d")
@@ -200,12 +204,50 @@ def main():
p.add_argument("spec", type=str,
help="needle3 (base weights), needle3.safetensors (the checkpoint to "
"fine-tune), a platform "
- "folder (e.g. macos-arm64), or a Hugging Face spec: "
+ "folder (e.g. macos-arm64), a fine-tuned model-<id> from "
+ "cactuscompute.com, or a Hugging Face spec: "
"<org>/<repo>/<file>.cact, or <org>/<repo> if it holds one archive")
p.add_argument("--out", type=str, default=".", help="Directory to place the files")
+ p.add_argument("--depth", type=int, default=None,
+ help="One size of a fine-tuned model-<id> (default: every size)")
p.add_argument("--generation", type=int, choices=[2, 3], default=3,
help="Engine generation when downloading a platform build (default: 3)")
+ p = sub.add_parser("platform")
+ verbs = p.add_subparsers(dest="verb")
+ v = verbs.add_parser("finetune")
+ v.add_argument("train", type=str, help="Training .jsonl (a path, or a file-<id> already uploaded)")
+ v.add_argument("validation", type=str, help="Validation .jsonl or file-<id>")
+ v.add_argument("test", type=str, help="Test .jsonl or file-<id>")
+ v.add_argument("--max-depth", type=int, default=None,
+ help="Largest size to train, 2 up to the base depth (default: the base depth)")
+ v.add_argument("--suffix", type=str, default=None, help="Name for the job and its models")
+ v.add_argument("--out", type=str, default=".", help="Directory for the downloaded .cact files")
+ v.add_argument("--depth", type=int, default=None,
+ help="Download one size only (default: every size)")
+ v.add_argument("--no-wait", action="store_true",
+ help="Submit and print the job id without polling or downloading")
+ v = verbs.add_parser("generate")
+ v.add_argument("--tools", type=str, required=True, help="Tool definitions .json, flat or OpenAI form")
+ v.add_argument("--examples", type=int, default=1000, help="Examples to generate, 100 to 10,000")
+ v.add_argument("--description", type=str, default=None, help="What the product does")
+ v.add_argument("--message", action="append", default=None,
+ help="An example user message; repeat for several")
+ v.add_argument("--suffix", type=str, default=None, help="Name for the job and its files")
+ v.add_argument("--out", type=str, default=".", help="Directory for the generated .jsonl files")
+ v.add_argument("--no-wait", action="store_true",
+ help="Submit and print the job id without polling or downloading")
+ v = verbs.add_parser("jobs")
+ v.add_argument("job_id", type=str, nargs="?", default=None, help="One job to show in full")
+ v.add_argument("--wait", action="store_true", help="Poll that job until it ends")
+ v.add_argument("--out", type=str, default=None,
+ help="Download what the finished job produced into this directory")
+ v.add_argument("--depth", type=int, default=None, help="One size of a fine-tuned model (default: every size)")
+ v = verbs.add_parser("models")
+ v.add_argument("model_id", type=str, nargs="?", default=None, help="One model to show with its sizes")
+ v = verbs.add_parser("files")
+ verbs.add_parser("billing")
+
p = sub.add_parser("fetch")
p.add_argument("--out", type=str, default=None,
help="Directory to place the engine (default: the cache)")
@@ -264,6 +306,11 @@ def main():
path = fetch.fetch_weights(target, args.out)
print(f" {'weights':<9} {path} {os.path.getsize(path) / 1e6:.2f} MB")
print(f" {'next':<9} needle.Needle(weights={path!r}, tools=[...])")
+ elif kind == "hosted":
+ from .platform import Platform
+ for path in Platform().download(target, args.out, depth=args.depth):
+ print(f" {'weights':<9} {path} {os.path.getsize(path) / 1e6:.2f} MB")
+ print(f" {'next':<9} needle.Needle(weights=<path>, tools=[...], auto_date=False)")
elif kind == "checkpoint":
generation = 2 if target.startswith("needle2") else 3
path = fetch.fetch_checkpoint(target, os.path.join(args.out, fetch.CHECKPOINT_PREFIX),
@@ -297,6 +344,9 @@ def main():
print(f" {'engine':<9} {path}")
print(f" {'deploy':<9} copy to ~/.cache/cactus-needle/v{generation}/{version}/ "
f"on the device, or point NEEDLE{generation}_LIB_PATH at the file")
+ elif args.command == "platform":
+ from .platform import main as platform_main
+ platform_main(args)
elif args.command == "playground":
from .playground.server import main as playground_main
playground_main(args)
diff --git a/needle/model/finetune.py b/needle/model/finetune.py
index bffba09..ee64b24 100644
--- a/needle/model/finetune.py
+++ b/needle/model/finetune.py
@@ -224,8 +224,35 @@ def _encode(tokenizer, example, max_len):
return ids + [PAD_ID] * pad, mask + [0.0] * pad
-def fit_max_len(path, tokenizer, cap):
- longest = 0
+def from_chat(example):
+ """A platform chat-format line (`messages` + OpenAI-form `tools`) as a local example.
+
+ Handles one user turn answered by one assistant turn, with an optional system
+ turn in front; returns None for anything else.
+ """
+ messages = example.get("messages")
+ if not isinstance(messages, list) or "query" in example:
+ return None
+ turns = [m for m in messages if m.get("role") != "system"]
+ if len(turns) != 2 or turns[0].get("role") != "user" or turns[1].get("role") != "assistant":
+ return None
+ answers = []
+ for call in turns[1].get("tool_calls") or []:
+ function = call.get("function") or call
+ arguments = function.get("arguments", {})
+ if isinstance(arguments, str):
+ arguments = json.loads(arguments) if arguments.strip() else {}
+ answers.append({"name": function.get("name"), "arguments": arguments})
+ tools = [t.get("function", t) if isinstance(t, dict) else t for t in example.get("tools") or []]
+ out = {"query": turns[0].get("content") or "", "tools": tools, "answers": answers}
+ system = [m.get("content") for m in messages if m.get("role") == "system"]
+ if system and system[0]:
+ out["system"] = system[0]
+ return out
+
+
+def read_examples(path, report=False):
+ skipped = 0
with open(path) as handle:
for line in handle:
line = line.strip()
@@ -233,9 +260,20 @@ def fit_max_len(path, tokenizer, cap):
continue
example = json.loads(line)
if "query" not in example:
+ example = from_chat(example)
+ if example is None:
+ skipped += 1
continue
- prompt, target = render_example(example)
- longest = max(longest, len(tokenizer.encode(prompt)) + len(tokenizer.encode(target)) + 2)
+ yield example
+ if skipped and report:
+ print(f" skipped {skipped} line(s) that are neither query/answers nor single-turn chat format")
+
+
+def fit_max_len(path, tokenizer, cap):
+ longest = 0
+ for example in read_examples(path):
+ prompt, target = render_example(example)
+ longest = max(longest, len(tokenizer.encode(prompt)) + len(tokenizer.encode(target)) + 2)
bucket = 128
while bucket < min(longest, cap):
bucket *= 2
@@ -244,17 +282,10 @@ def fit_max_len(path, tokenizer, cap):
def load_jsonl(path, tokenizer, max_len):
seqs, masks = [], []
- with open(path) as handle:
- for line in handle:
- line = line.strip()
- if not line:
- continue
- example = json.loads(line)
- if "query" not in example:
- continue
- ids, mask = _encode(tokenizer, example, max_len)
- seqs.append(ids)
- masks.append(mask)
+ for example in read_examples(path, report=True):
+ ids, mask = _encode(tokenizer, example, max_len)
+ seqs.append(ids)
+ masks.append(mask)
return np.array(seqs, np.int32), np.array(masks, np.float32)
@@ -427,7 +458,7 @@ def finetune_local(args, progress=None):
})
print(f" {'adapter':<9} {out}")
print(f" {'next':<9} needle build {base_path} --lora {out}")
- print(f" {'note':<9} confidence reports None with tuned weights; the head is not tuned")
+ print(f" {'note':<9} local tuning leaves the confidence head untrained; needle build drops it and confidence reports None")
def build_main(args):
@@ -435,7 +466,7 @@ def build_main(args):
import jax.numpy as jnp
from ..agent import fetch
from .run import load_checkpoint
- from .architecture import effective_kv_window
+ from .architecture import ConfidenceHead, effective_kv_window
from .export import read_layers, read_tokenizer_blob, write_export
from .quantize import WEIGHT_BITS
@@ -468,6 +499,9 @@ def build_main(args):
for key, v in adapter["lora"].items()}
params = merge_lora(params, lora, adapter["scale"])
print(f" {'merged':<9} {len(lora)} weight groups {args.lora}")
+ if ConfidenceHead.key in params:
+ params = {k: v for k, v in params.items() if k != ConfidenceHead.key}
+ print(f" {'dropped':<9} the confidence head; it is not trained locally, so confidence reports None")
params, config = rung(params, config, layers)
print(f" {'depth':<9} {config.num_layers} layers")
out = out or (os.path.splitext(os.path.basename(checkpoint))[0] + ".cact")
diff --git a/needle/platform.py b/needle/platform.py
new file mode 100644
index 0000000..6aa01f3
--- /dev/null
+++ b/needle/platform.py
@@ -0,0 +1,429 @@
+from __future__ import annotations
+
+import json
+import os
+import shutil
+import ssl
+import time
+import urllib.error
+import urllib.parse
+import urllib.request
+
+BASE_URL = "https://cactuscompute.com/v1"
+KEYS_URL = "https://cactuscompute.com/dashboard/api-keys"
+JOBS_URL = "https://cactuscompute.com/dashboard/jobs"
+BASE_MODEL = "needle-3"
+TERMINAL = ("succeeded", "failed", "cancelled")
+
+
+class PlatformError(RuntimeError):
+ def __init__(self, code, message, status=0, param=None, url=None, retry_after=0):
+ super().__init__(message)
+ self.code = code
+ self.status = status
+ self.param = param
+ self.url = url
+ self.retry_after = retry_after
+
+ def __str__(self):
+ text = f"{self.code}: {self.args[0]}"
+ if self.param:
+ text += f" (param: {self.param})"
+ if self.url:
+ text += f" -> {self.url}"
+ return text
+
+
+class _NoRedirect(urllib.request.HTTPRedirectHandler):
+ def redirect_request(self, req, fp, code, msg, headers, newurl):
+ return None
+
+
+def _ssl_context():
+ try:
+ import certifi
+ return ssl.create_default_context(cafile=certifi.where())
+ except ImportError:
+ return ssl.create_default_context()
+
+
+def flat_tools(tools):
+ """OpenAI-form tool definitions to the flat form `needle.Needle(tools=...)` takes."""
+ return [t["function"] if t.get("type") == "function" and "function" in t else t
+ for t in tools]
+
+
+def openai_tools(tools):
+ """Flat tool definitions to the OpenAI form the platform takes."""
+ return [t if t.get("type") == "function" else {"type": "function", "function": t}
+ for t in tools]
+
+
+class Platform:
+ """Client for the hosted fine-tuning API at cactuscompute.com/v1.
+
+ The key comes from `api_key` or `NEEDLE_API_KEY`; create one in the console
+ under API Keys. Every method raises `PlatformError` with the API's stable
+ error `code` on failure.
+ """
+
+ def __init__(self, api_key=None, base_url=BASE_URL, timeout=60, retries=3):
+ self.api_key = api_key or os.environ.get("NEEDLE_API_KEY")
+ self.base_url = base_url.rstrip("/")
+ self.timeout = timeout
+ self.retries = retries
+ https = urllib.request.HTTPSHandler(context=_ssl_context())
+ self._opener = urllib.request.build_opener(_NoRedirect, https)
+ self._follower = urllib.request.build_opener(https)
+
+ def _send(self, method, url, body=None, headers=None, auth=True):
+ headers = dict(headers or {})
+ if auth and self.api_key:
+ headers["Authorization"] = f"Bearer {self.api_key}"
+ data = None
+ if body is not None and not isinstance(body, (bytes, bytearray)):
+ data = json.dumps(body).encode("utf-8")
+ headers["Content-Type"] = "application/json"
+ elif body is not None:
+ data = bytes(body)
+ request = urllib.request.Request(url, data=data, method=method, headers=headers)
+ for attempt in range(self.retries + 1):
+ try:
+ with self._opener.open(request, timeout=self.timeout) as response:
+ return response.status, dict(response.headers), response.read()
+ except urllib.error.HTTPError as error:
+ payload = error.read()
+ if error.code in (301, 302, 303, 307, 308):
+ return error.code, dict(error.headers), payload
+ code, message, param, link = _parse_error(payload, error.code)
+ if error.code == 401 and not self.api_key:
+ message, link = "set NEEDLE_API_KEY to a key from the console", link or KEYS_URL
+ retry_after = _seconds(error.headers.get("Retry-After"))
+ if error.code == 429 and code == "rate_limited" and attempt < self.retries:
+ time.sleep(max(retry_after, 1))
+ continue
+ raise PlatformError(code, message, error.code, param, link, retry_after) from None
+ except urllib.error.URLError as error:
+ raise PlatformError("network_error", str(error.reason)) from None
+ raise PlatformError("rate_limited", "the API kept answering 429", 429)
+
+ def _call(self, method, path, body=None, params=None):
+ url = f"{self.base_url}/{path}"
+ if params:
+ url += "?" + urllib.parse.urlencode({k: v for k, v in params.items() if v is not None})
+ status, headers, payload = self._send(method, url, body)
+ return json.loads(payload.decode("utf-8")) if payload else {}
+
+ def _signed_link(self, path):
+ status, headers, _ = self._send("GET", f"{self.base_url}/{path}")
+ link = headers.get("Location") or headers.get("location")
+ if status not in (301, 302, 303, 307, 308) or not link:
+ raise PlatformError("unexpected_response", f"expected a redirect from {path}, got {status}", status)
+ return link
+
+ def _fetch_to(self, link, dest):
+ os.makedirs(os.path.dirname(os.path.abspath(dest)), exist_ok=True)
+ request = urllib.request.Request(link, method="GET")
+ with self._follower.open(request, timeout=self.timeout) as response, open(dest, "wb") as out:
+ shutil.copyfileobj(response, out)
+ return dest
+
+ def plans(self):
+ return self._call("GET", "plans")
+
+ def billing(self):
+ return self._call("GET", "billing")
+
+ def tool_schemas(self, description):
+ return self._call("POST", "tool_schemas", {"description": description})["tools"]
+
+ def files(self, search=None, limit=100):
+ return self._call("GET", "files", params={"search": search, "limit": limit})["data"]
+
+ def file(self, file_id):
+ return self._call("GET", f"files/{file_id}")
+
+ def delete_file(self, file_id):
+ return self._call("DELETE", f"files/{file_id}")
+
+ def upload(self, path, name=None):
+ """Upload a .jsonl file in the platform's three steps; returns the file record."""
+ size = os.path.getsize(path)
+ reservation = self._call("POST", "files", {"name": name or os.path.basename(path), "bytes": size})
+ with open(path, "rb") as handle:
+ self._send("PUT", reservation["url"], handle.read(),
+ {"Content-Type": "application/octet-stream"}, auth=False)
+ return self._call("POST", f"files/{reservation['id']}/complete")
+
+ def download_file(self, file_id, dest):
+ return self._fetch_to(self._signed_link(f"files/{file_id}/content"), dest)
+
+ def generate(self, tools, examples, description=None, messages=None, suffix=None):
+ body = {"tools": openai_tools(tools), "examples": int(examples)}
+ if description:
+ body["description"] = description
+ if messages:
+ body["messages"] = list(messages)
+ if suffix:
+ body["suffix"] = suffix
+ return self._call("POST", "generations", body)
+
+ def generation(self, job_id):
+ return self._call("GET", f"generations/{job_id}")
+
+ def generations(self, limit=20):
+ return self._call("GET", "generations", params={"limit": limit})["data"]
+
+ def cancel_generation(self, job_id):
+ return self._call("POST", f"generations/{job_id}/cancel")
+
+ def _file_ids(self, items):
+ ids = []
+ for item in items if isinstance(items, (list, tuple)) else [items]:
+ if isinstance(item, dict):
+ ids.append(item["id"])
+ elif isinstance(item, str) and item.startswith("file-"):
+ ids.append(item)
+ elif isinstance(item, str) and os.path.exists(item):
+ ids.append(self.upload(item)["id"])
+ else:
+ raise PlatformError("invalid_request", f"{item!r} is neither a file id nor a path", 400)
+ return ids
+
+ def finetune(self, training_files, validation_files, test_files, max_depth=None, suffix=None):
+ """Start a fine-tune. Each group takes paths (uploaded first), file ids, or file records."""
+ if max_depth is None:
+ max_depth = self.model(BASE_MODEL)["depth"]
+ body = {
+ "model": BASE_MODEL,
+ "training_files": self._file_ids(training_files),
+ "validation_files": self._file_ids(validation_files),
+ "test_files": self._file_ids(test_files),
+ "max_depth": int(max_depth),
+ }
+ if suffix:
+ body["suffix"] = suffix
+ return self._call("POST", "fine_tuning/jobs", body)
+
+ def job(self, job_id):
+ return self._call("GET", f"fine_tuning/jobs/{job_id}")
+
+ def jobs(self, limit=20):
+ return self._call("GET", "fine_tuning/jobs", params={"limit": limit})["data"]
+
+ def cancel(self, job_id):
+ return self._call("POST", f"fine_tuning/jobs/{job_id}/cancel")
+
+ def wait(self, job, poll=10, on_update=None):
+ """Poll a fine-tune or generation until it ends; returns the record, raises if it failed."""
+ job_id = job["id"] if isinstance(job, dict) else job
+ fetch = self.generation if (isinstance(job, dict) and job.get("object") == "generation") else self.job
+ while True:
+ record = fetch(job_id)
+ if on_update:
+ on_update(record)
+ status = record.get("status")
+ if status in TERMINAL:
+ break
+ time.sleep(poll)
+ if status == "succeeded":
+ return record
+ error = record.get("error") or {}
+ raise PlatformError(error.get("code") or status, error.get("message") or f"job {job_id} {status}")
+
+ def models(self, search=None, limit=100):
+ return self._call("GET", "models", params={"search": search, "limit": limit})["data"]
+
+ def model(self, model_id):
+ return self._call("GET", f"models/{model_id}")
+
+ def delete_model(self, model_id):
+ return self._call("DELETE", f"models/{model_id}")
+
+ def download(self, model_id, out=".", depth=None):
+ """Download a model's .cact files into `out`, every size or one `depth`; returns the paths."""
+ record = self.model(model_id)
+ stem = record.get("name") or record["id"]
+ variants = record.get("variants") or [{"id": record["id"], "depth": record.get("depth")}]
+ if depth is not None:
+ variants = [v for v in variants if v.get("depth") == int(depth)]
+ if not variants:
+ raise PlatformError("not_found", f"{model_id} has no {depth}-layer size", 404)
+ paths = []
+ for variant in variants:
+ dest = os.path.join(out, f"{stem}-{variant['depth']}L.cact")
+ paths.append(self._fetch_to(self._signed_link(f"models/{variant['id']}/content"), dest))
+ return paths
+
+
+def _seconds(value):
+ try:
+ return int(float(value or 0))
+ except ValueError:
+ return 0
+
+
+def _parse_error(payload, status):
+ try:
+ error = json.loads(payload.decode("utf-8")).get("error") or {}
+ except (ValueError, AttributeError):
+ error = {}
+ return (error.get("code") or f"http_{status}", error.get("message") or f"HTTP {status}",
+ error.get("param"), error.get("url"))
+
+
+def _print(label, value):
+ print(f" {label:<9} {value}")
+
+
+def _print_usage(client):
+ billing = client.billing()
+ limits, usage = billing.get("limits") or {}, billing.get("usage") or {}
+ _print("plan", f"{billing.get('plan')}: {usage.get('jobs', 0)}/{limits.get('jobs', '?')} fine-tunes, "
+ f"{usage.get('examples', 0)}/{limits.get('examples', '?')} generated examples this period")
+
+
+def _print_job(job):
+ line = f"{job['id']} {job.get('status')}"
+ if job.get("name"):
+ line += f" {job['name']}"
+ if job.get("max_depth"):
+ line += f" max_depth {job['max_depth']}"
+ if job.get("error"):
+ line += f" error {job['error'].get('code')}"
+ print(" " + line)
+
+
+def _print_evaluations(job):
+ rows = job.get("evaluations") or []
+ if not rows:
+ return
+ print(f" {'depth':>5} {'validation':>12} {'test':>12}")
+ for row in sorted(rows, key=lambda r: -r["depth"]):
+ v, t = row["validation"], row["test"]
+ print(f" {row['depth']:>4}L {v['correct']:>5}/{v['total']:<6} {t['correct']:>5}/{t['total']:<6}")
+
+
+def _progress(kind):
+ seen = {}
+
+ def show(record):
+ key = (record.get("status"), record.get("completed_steps"))
+ if key in seen:
+ return
+ seen[key] = True
+ status = record.get("status")
+ if status == "running" and record.get("total_steps"):
+ print(f" {kind:<9} running, step {record.get('completed_steps', 0)}/{record['total_steps']}", flush=True)
+ else:
+ print(f" {kind:<9} {status}", flush=True)
+ return show
+
+
+def _cmd_finetune(client, args):
+ _print_usage(client)
+ groups = {}
+ for label, item in (("train", args.train), ("validation", args.validation), ("test", args.test)):
+ if item.startswith("file-"):
+ groups[label] = item
+ else:
+ record = client.upload(item)
+ _print("upload", f"{record['id']} {record['filename']} {record['bytes'] / 1e6:.2f} MB")
+ groups[label] = record["id"]
+ job = client.finetune([groups["train"]], [groups["validation"]], [groups["test"]],
+ max_depth=args.max_depth, suffix=args.suffix)
+ _print("job", f"{job['id']} max_depth {job.get('max_depth')}")
+ _print("watch", f"{JOBS_URL}/{job['id']} (Ctrl-C leaves the job running; needle platform jobs {job['id']} --wait --out DIR resumes)")
+ if args.no_wait:
+ return
+ job = client.wait(job, on_update=_progress("job"))
+ _print_evaluations(job)
+ paths = client.download(job["fine_tuned_model"], args.out, depth=args.depth)
+ for path in paths:
+ _print("weights", f"{path} {os.path.getsize(path) / 1e6:.2f} MB")
+ _print("next", f"needle.Needle(weights={paths[-1]!r}, tools=[...], auto_date=False)")
+
+
+def _cmd_generate(client, args):
+ _print_usage(client)
+ with open(args.tools) as handle:
+ tools = json.load(handle)
+ job = client.generate(tools, args.examples, description=args.description,
+ messages=args.message, suffix=args.suffix)
+ _print("job", f"{job['id']} {args.examples} examples")
+ _print("watch", f"{JOBS_URL}/{job['id']} (Ctrl-C leaves the job running; needle platform jobs {job['id']} --wait --out DIR resumes)")
+ if args.no_wait:
+ return
+ job = client.wait(job, on_update=_progress("job"))
+ for record in job.get("files") or []:
+ dest = client.download_file(record["id"], os.path.join(args.out, record["filename"]))
+ _print(record["group"], f"{dest} {os.path.getsize(dest) / 1e6:.2f} MB")
+ _print("next", "needle platform finetune <train.jsonl> <validation.jsonl> <test.jsonl>")
+
+
+def _cmd_jobs(client, args):
+ if not args.job_id:
+ for job in client.jobs():
+ _print_job(job)
+ return
+ try:
+ job = client.job(args.job_id)
+ except PlatformError as error:
+ if error.status != 404:
+ raise
+ job = client.generation(args.job_id)
+ if args.wait and job.get("status") not in TERMINAL:
+ job = client.wait(job, on_update=_progress("job"))
+ _print_job(job)
+ _print_evaluations(job)
+ if job.get("fine_tuned_model"):
+ _print("model", job["fine_tuned_model"])
+ if args.out:
+ for path in client.download(job["fine_tuned_model"], args.out, depth=args.depth):
+ _print("weights", f"{path} {os.path.getsize(path) / 1e6:.2f} MB")
+ return
+ for record in job.get("files") or []:
+ if args.out and job.get("object") == "generation" and job.get("status") == "succeeded":
+ dest = client.download_file(record["id"], os.path.join(args.out, record["filename"]))
+ _print(record.get("group", "file"), f"{dest} {os.path.getsize(dest) / 1e6:.2f} MB")
+ else:
+ _print(record.get("group", "file"), f"{record['id']} {record['filename']}")
+
+
+def _cmd_models(client, args):
+ if args.model_id:
+ record = client.model(args.model_id)
+ _print("model", f"{record['id']} {record.get('name') or ''} {record.get('depth')} layers")
+ for variant in record.get("variants") or []:
+ size = f" {variant['bytes'] / 1e6:.2f} MB" if variant.get("bytes") else ""
+ _print(f"{variant['depth']}L", f"{variant['id']}{size}")
+ _print("next", f"needle download {record['id']} [--depth N]")
+ return
+ for record in client.models():
+ _print("model", f"{record['id']} {record.get('name') or ''} {record.get('depth')} layers")
+
+
+def _cmd_files(client, args):
+ for record in client.files():
+ _print("file", f"{record['id']} {record['filename']} {record['bytes'] / 1e6:.2f} MB")
+
+
+def _cmd_billing(client, args):
+ billing = client.billing()
+ _print("plan", billing.get("plan"))
+ for key, limit in (billing.get("limits") or {}).items():
+ _print(key, f"{(billing.get('usage') or {}).get(key, 0)} / {limit}")
+ if billing.get("period_end"):
+ _print("resets", time.strftime("%Y-%m-%d", time.gmtime(billing["period_end"])))
+
+
+def main(args):
+ client = Platform()
+ verbs = {"finetune": _cmd_finetune, "generate": _cmd_generate, "jobs": _cmd_jobs,
+ "models": _cmd_models, "files": _cmd_files, "billing": _cmd_billing}
+ if not args.verb:
+ raise SystemExit("needle platform finetune | generate | jobs | models | files | billing")
+ try:
+ verbs[args.verb](client, args)
+ except PlatformError as error:
+ raise SystemExit(f"platform error {error}")
diff --git a/tests/conftest.py b/tests/conftest.py
index d3d7c19..9d2bac4 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -98,3 +98,37 @@ def published_base(tiny_base_archive, monkeypatch):
monkeypatch.setattr(fetch, "fetch_weights", fake_fetch)
return calls
+
+
+def fake_cact():
+ return _fake_cact
+
+
+def _fake_cact(path, heads):
+ import struct
+
+ header_fmt, record_fmt = "<48If", "<BBHIIIIQQII"
+ body = bytearray()
+ records = []
+
+ def add(dtype, shape, blob):
+ records.append((dtype, len(shape), 0, *(list(shape) + [0] * (4 - len(shape))), len(body), len(blob), 0, 0))
+ body.extend(blob)
+
+ add(1, (4,), struct.pack("<4e", 0, 0, 0, 0))
+ if heads:
+ add(1, (len(heads),), struct.pack(f"<{len(heads)}e", *heads))
+ for _ in heads:
+ for _ in range(6):
+ add(1, (2,), struct.pack("<2e", 0, 0))
+ add(4, (3,), b"tok")
+ head = struct.calcsize(header_fmt) + struct.calcsize(record_fmt) * len(records)
+ fields = [0x05E12A84, len(records), 0] + [0] * 45
+ out = bytearray(struct.pack(header_fmt, *fields, 0.0))
+ for rec in records:
+ rec = list(rec)
+ rec[7] += head
+ out += struct.pack(record_fmt, *rec)
+ out += body
+ path.write_bytes(bytes(out))
diff --git a/tests/test_platform.py b/tests/test_platform.py
new file mode 100644
index 0000000..aa9690e
--- /dev/null
+++ b/tests/test_platform.py
@@ -0,0 +1,223 @@
+import json
+import threading
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+import pytest
+
+
+class FakePlatform(BaseHTTPRequestHandler):
+ state = None
+
+ def log_message(self, *args):
+ pass
+
+ def _body(self):
+ length = int(self.headers.get("Content-Length") or 0)
+ return self.rfile.read(length) if length else b""
+
+ def _json(self, status, payload, headers=None):
+ data = json.dumps(payload).encode()
+ self.send_response(status)
+ self.send_header("Content-Type", "application/json")
+ for key, value in (headers or {}).items():
+ self.send_header(key, value)
+ self.send_header("Content-Length", str(len(data)))
+ self.end_headers()
+ self.wfile.write(data)
+
+ def _error(self, status, code, headers=None, **extra):
+ self._json(status, {"error": {"message": code, "type": "platform_error", "code": code, **extra}}, headers)
+
+ def do_GET(self):
+ s = self.state
+ s["requests"].append(("GET", self.path, self.headers.get("Authorization")))
+ if self.path.startswith("/blob/"):
+ self.send_response(200)
+ self.send_header("Content-Length", "5")
+ self.end_headers()
+ self.wfile.write(b"CACT!")
+ elif self.path.startswith("/v1/") and self.headers.get("Authorization") != "Bearer needle_ft_test":
+ self._error(401, "unauthorized", url="https://cactuscompute.com/dashboard/api-keys")
+ elif self.path == "/v1/billing":
+ if s["rate_limit_once"]:
+ s["rate_limit_once"] = False
+ self._error(429, "rate_limited", {"Retry-After": "1"})
+ else:
+ self._json(200, {"plan": "standard", "limits": {"jobs": 10}, "usage": {"jobs": 1}})
+ elif self.path == "/v1/models/needle-3":
+ self._json(200, {"id": "needle-3", "object": "model", "owned_by": "cactus", "depth": 20, "variants": []})
+ elif self.path == "/v1/models/model-1":
+ self._json(200, {"id": "model-1", "object": "model", "owned_by": "me", "name": "smart-home", "depth": 8,
+ "variants": [{"id": "model-1-8", "depth": 8, "is_full": True, "bytes": 5},
+ {"id": "model-1-4", "depth": 4, "is_full": False, "bytes": 5}]})
+ elif self.path.startswith("/v1/models/model-1-") and self.path.endswith("/content"):
+ self.send_response(307)
+ self.send_header("Location", f"http://{self.headers['Host']}/blob/{self.path.split('/')[3]}")
+ self.send_header("Content-Length", "0")
+ self.end_headers()
+ elif self.path == "/v1/fine_tuning/jobs/ftjob-1":
+ s["polls"] += 1
+ status = "running" if s["polls"] < 2 else "succeeded"
+ self._json(200, {"id": "ftjob-1", "object": "fine_tuning.job", "status": status, "created_at": 0,
+ "completed_steps": min(s["polls"], 5), "total_steps": 5, "max_depth": 8,
+ "fine_tuned_model": "model-1" if status == "succeeded" else None,
+ "evaluations": [{"depth": 8, "validation": {"correct": 9, "total": 10, "excluded": 0},
+ "test": {"correct": 8, "total": 10, "excluded": 0}}] if status == "succeeded" else []})
+ else:
+ self._error(404, "not_found")
+
+ def do_POST(self):
+ s = self.state
+ body = self._body()
+ s["requests"].append(("POST", self.path, self.headers.get("Authorization")))
+ if self.path == "/v1/files":
+ payload = json.loads(body)
+ s["reserved"] = payload
+ self._json(201, {"id": "0123", "url": f"http://{self.headers['Host']}/upload/0123", "expires_at": "2026-01-01T00:00:00Z"})
+ elif self.path == "/v1/files/0123/complete":
+ self._json(201, {"id": "file-0123", "object": "file", "filename": s["reserved"]["name"],
+ "bytes": s["reserved"]["bytes"], "created_at": 0, "purpose": "fine-tune"})
+ elif self.path == "/v1/fine_tuning/jobs":
+ s["submitted"] = json.loads(body)
+ if s["submitted"].get("suffix") == "broke":
+ self._error(402, "quota_exceeded", param="jobs", url="https://cactuscompute.com/dashboard/usage")
+ else:
+ self._json(201, {"id": "ftjob-1", "object": "fine_tuning.job", "status": "queued", "created_at": 0,
+ "max_depth": s["submitted"]["max_depth"]})
+ else:
+ self._error(404, "not_found")
+
+ def do_PUT(self):
+ s = self.state
+ body = self._body()
+ s["requests"].append(("PUT", self.path, self.headers.get("Authorization")))
+ s["uploaded"] = body
+ self.send_response(200)
+ self.send_header("Content-Length", "0")
+ self.end_headers()
+
+
+def server():
+ state = {"requests": [], "polls": 0, "rate_limit_once": False}
+ FakePlatform.state = state
+ httpd = ThreadingHTTPServer(("127.0.0.1", 0), FakePlatform)
+ thread = threading.Thread(target=httpd.serve_forever, daemon=True)
+ thread.start()
+ try:
+ yield f"http://127.0.0.1:{httpd.server_address[1]}/v1", state
+ finally:
+ httpd.shutdown()
+
+
+def test_missing_key_points_at_the_console(server, monkeypatch):
+ from needle.platform import Platform, PlatformError
+
+ base, state = server
+ monkeypatch.delenv("NEEDLE_API_KEY", raising=False)
+ with pytest.raises(PlatformError) as caught:
+ Platform(base_url=base).billing()
+ assert caught.value.code == "unauthorized"
+ assert "NEEDLE_API_KEY" in str(caught.value)
+ assert "dashboard/api-keys" in caught.value.url
+ assert state["requests"][-1][2] is None
+
+
+def test_finetune_uploads_submits_waits_and_downloads(server, tmp_path):
+ from needle.platform import Platform
+
+ base, state = server
+ train = tmp_path / "train.jsonl"
+ train.write_text('{"messages":[],"tools":[]}\n')
+ client = Platform(api_key="needle_ft_test", base_url=base)
+
+ job = client.finetune([str(train)], ["file-val"], ["file-test"], suffix="smart-home")
+ assert job["id"] == "ftjob-1"
+ assert state["reserved"] == {"name": "train.jsonl", "bytes": train.stat().st_size}
+ assert state["uploaded"] == train.read_bytes()
+ put = next(r for r in state["requests"] if r[0] == "PUT")
+ assert put[1] == "/upload/0123" and put[2] is None
+ assert state["submitted"] == {"model": "needle-3", "training_files": ["file-0123"],
+ "validation_files": ["file-val"], "test_files": ["file-test"],
+ "max_depth": 20, "suffix": "smart-home"}
+
+ seen = []
+ done = client.wait(job, poll=0, on_update=lambda r: seen.append(r["status"]))
+ assert seen == ["running", "succeeded"]
+ assert done["fine_tuned_model"] == "model-1"
+
+ paths = client.download("model-1", str(tmp_path / "out"))
+ assert sorted(p.split("/")[-1] for p in paths) == ["smart-home-4L.cact", "smart-home-8L.cact"]
+ assert all(open(p, "rb").read() == b"CACT!" for p in paths)
+ blob = next(r for r in state["requests"] if r[1].startswith("/blob/"))
+ assert blob[2] is None
+
+ only = client.download("model-1", str(tmp_path / "one"), depth=4)
+ assert [p.split("/")[-1] for p in only] == ["smart-home-4L.cact"]
+
+
+def test_errors_carry_code_param_and_url(server, tmp_path):
+ from needle.platform import Platform, PlatformError
+
+ base, state = server
+ client = Platform(api_key="needle_ft_test", base_url=base)
+ with pytest.raises(PlatformError) as caught:
+ client.finetune(["file-a"], ["file-b"], ["file-c"], max_depth=8, suffix="broke")
+ assert (caught.value.code, caught.value.status, caught.value.param) == ("quota_exceeded", 402, "jobs")
+ assert "dashboard/usage" in str(caught.value)
+
+ state["rate_limit_once"] = True
+ assert client.billing()["plan"] == "standard"
+
+
+def test_download_target_routes_hosted_models():
+ from needle.cli import _download_target
+
+ assert _download_target("model-abc") == ("hosted", "model-abc")
+
+
+def test_tool_form_conversions():
+ from needle.platform import flat_tools, openai_tools
+
+ flat = [{"name": "set_lights", "parameters": {"type": "object", "properties": {}}}]
+ wrapped = openai_tools(flat)
+ assert wrapped == [{"type": "function", "function": flat[0]}]
+ assert flat_tools(wrapped) == flat
+ assert openai_tools(wrapped) == wrapped
+
+
+def test_local_finetune_reads_chat_format(tmp_path):
+ from needle.model.finetune import from_chat, read_examples
+
+ chat = {"messages": [{"role": "system", "content": "date: 2026-09-19"},
+ {"role": "user", "content": "turn the kitchen lights off"},
+ {"role": "assistant", "content": "", "tool_calls": [
+ {"id": "call_1", "type": "function",
+ "function": {"name": "set_lights", "arguments": "{\"room\":\"kitchen\",\"state\":\"off\"}"}}]}],
+ "tools": [{"type": "function", "function": {"name": "set_lights", "parameters": {"type": "object"}}}]}
+ example = from_chat(chat)
+ assert example == {"query": "turn the kitchen lights off", "system": "date: 2026-09-19",
+ "tools": [{"name": "set_lights", "parameters": {"type": "object"}}],
+ "answers": [{"name": "set_lights", "arguments": {"room": "kitchen", "state": "off"}}]}
+ silent = {"messages": [{"role": "user", "content": "hi"}, {"role": "assistant", "content": "Hello."}], "tools": []}
+ assert from_chat(silent)["answers"] == []
+ multi = {"messages": [{"role": "user", "content": "a"}, {"role": "assistant", "content": "b"},
+ {"role": "user", "content": "c"}, {"role": "assistant", "content": "d"}], "tools": []}
+ assert from_chat(multi) is None
+
+ path = tmp_path / "mixed.jsonl"
+ path.write_text("\n".join(json.dumps(x) for x in (chat, silent, multi,
+ {"query": "q", "tools": [], "answers": []})) + "\n")
+ rows = list(read_examples(str(path)))
+ assert [r["query"] for r in rows] == ["turn the kitchen lights off", "hi", "q"]
+
+
+def test_confidence_head_detected_from_manifest(tmp_path, fake_cact):
+ import needle
+
+ fake_cact(tmp_path / "platform.cact", [1, 2])
+ fake_cact(tmp_path / "local.cact", [1])
+ fake_cact(tmp_path / "bare.cact", [])
+ assert needle._confidence_head_present(str(tmp_path / "platform.cact")) is True
+ assert needle._confidence_head_present(str(tmp_path / "local.cact")) is False
+ assert needle._confidence_head_present(str(tmp_path / "bare.cact")) is False
diff --git a/tests/test_weights.py b/tests/test_weights.py
index d90ac89..6f15696 100644
--- a/tests/test_weights.py
+++ b/tests/test_weights.py
@@ -277,3 +277,17 @@ def test_tool_index_path_agrees_between_the_base_and_tuned_paths(engine, tuned,
if isinstance(call, tuple) and call[0] == "worker_start"]
assert started
assert base._tool_index_path.decode("utf-8") == os.fspath(index)
+
+
+def test_platform_tuned_archive_keeps_confidence(engine, tmp_path, fake_cact):
+ import needle
+
+ fake_cact(tmp_path / "platform.cact", [1, 2])
+ fake_cact(tmp_path / "local.cact", [1])
+ with warnings.catch_warnings():
+ warnings.simplefilter("error")
+ calibrated = needle.Needle(tools="[]", weights=str(tmp_path / "platform.cact"))
+ assert calibrated.complete("hello").get("confidence") is not None
+ with pytest.warns(UserWarning, match="confidence as None"):
+ uncalibrated = needle.Needle(tools="[]", weights=str(tmp_path / "local.cact"))
+ assert uncalibrated.complete("hello").get("confidence", "kept") is None