Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.
feat: enhance confidence validation and update agent construction logic
HenryNdubuaku42ae523fda02
diff --git a/doc/apis.md b/doc/apis.mdindex 9b8999a..cbead42 100644--- a/doc/apis.md+++ b/doc/apis.md@@ -168,7 +168,7 @@ Five or fewer declared tools render directly. Above that, retrieval engages: at## Confidence-The `confidence` field is the minimum of two signals: a calibrated post-hoc head that scores the full prompt plus the call the model just produced, and the decoding probability of the call tokens. A call is accepted only when both agree, so the failure mode is escalation, not wrong execution. The contract: pick a threshold for your product, act at or above it, re-ask or route to a bigger model below it. Off-topic requests return the empty call `[]`.+The `confidence` field is the minimum of two signals: a calibrated post-hoc head that scores the full prompt plus the call the model just produced, and the decoding probability of the call tokens. A call is accepted only when both agree, so the failure mode is escalation, not wrong execution. The contract: pick a threshold for your product, act at or above it, re-ask or route to a bigger model below it. Off-topic requests return the empty call `[]`. Every response also carries `escalate`, true when `confidence` is below the engine's default threshold of 0.15.Calibration holds for the base model only. Fine-tuning does not update the head, so an agent running tuned weights reports `confidence` as `None` and warns once at construction.diff --git a/doc/environments.md b/doc/environments.mdindex 421eb70..a7dbeea 100644--- a/doc/environments.md+++ b/doc/environments.md@@ -21,7 +21,7 @@ Every environment module exposes the same surface:| --- | --- || `TOOLS` | The decorated tool functions, each returning `{"ok": True, ...}` echoes of its arguments. || `SYSTEM` | The system prompt the agent is built with. |-| `agent` | A `needle.Needle(tools=TOOLS, system=SYSTEM)`, constructed lazily on first access so importing never fetches the engine. Construction sets `NEEDLE_STRICT_VALIDATE=1` (if unset) so out-of-bounds calls are suppressed to refusals. |+| `agent` | A `needle.Needle(tools=TOOLS, system=SYSTEM)`, constructed lazily on first access so importing never fetches the engine. `run_tests` treats a call whose `validation` reports ungrounded values or a negation as a refusal. || `TEST_CASES` | The frozen suite: dicts of `query`, expected `calls`, `category`, and optional `critical`. || `run_tests(min_confidence=0.0, verbose=True)` | Runs the suite against the shipped engine; returns `True` at >=90% pass with zero critical failures. |diff --git a/llms.txt b/llms.txtindex a9e2ca6..ae3fb2b 100644--- a/llms.txt+++ b/llms.txt@@ -124,7 +124,7 @@ if r["type"] == "call":## Confidence gating-`confidence` is the min of a post-hoc calibration head and the decode probability of the call. Calibration holds for the base model only: an agent constructed with `weights=` reports `confidence` as None (fine-tuning does not update the head). Pick a threshold per product; act at/above it, escalate below it:+`confidence` is the min of a post-hoc calibration head and the decode probability of the call. Calibration holds for the base model only: an agent constructed with `weights=` reports `confidence` as None (fine-tuning does not update the head). Pick a threshold per product; act at/above it, escalate below it (`escalate` in the response applies the engine default of 0.15):```pythonr = agent.complete(user_text)diff --git a/needle/environments/_harness.py b/needle/environments/_harness.pyindex fb561f9..e165fe1 100644--- a/needle/environments/_harness.py+++ b/needle/environments/_harness.py@@ -1,5 +1,4 @@import json-import osimport needle@@ -9,7 +8,6 @@ _agents = {}def agent_for(module):key = module.__name__if key not in _agents:- os.environ.setdefault("NEEDLE_STRICT_VALIDATE", "1")_agents[key] = needle.Needle(tools=module.TOOLS, system=module.SYSTEM)return _agents[key]@@ -25,6 +23,9 @@ def run_tests(module, min_confidence=0.0, verbose=True):agent.reset()response = agent.complete(case["query"])got = response.get("function_calls") or []+ validation = response.get("validation") or {}+ if got and (validation.get("ungrounded") or validation.get("negation")):+ got = []if got and response.get("confidence", 0.0) < min_confidence:got = []want = case["calls"]