Skip to content

Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.

feat: enhance confidence validation and update agent construction logic

HenryNdubuaku42ae523fda02
diff --git a/doc/apis.md b/doc/apis.md
index 9b8999a..cbead42 100644
--- a/doc/apis.md
+++ b/doc/apis.md
@@ -168,7 +168,7 @@ Five or fewer declared tools render directly. Above that, retrieval engages: at
## Confidence
-The `confidence` field is the minimum of two signals: a calibrated post-hoc head that scores the full prompt plus the call the model just produced, and the decoding probability of the call tokens. A call is accepted only when both agree, so the failure mode is escalation, not wrong execution. The contract: pick a threshold for your product, act at or above it, re-ask or route to a bigger model below it. Off-topic requests return the empty call `[]`.
+The `confidence` field is the minimum of two signals: a calibrated post-hoc head that scores the full prompt plus the call the model just produced, and the decoding probability of the call tokens. A call is accepted only when both agree, so the failure mode is escalation, not wrong execution. The contract: pick a threshold for your product, act at or above it, re-ask or route to a bigger model below it. Off-topic requests return the empty call `[]`. Every response also carries `escalate`, true when `confidence` is below the engine's default threshold of 0.15.
Calibration holds for the base model only. Fine-tuning does not update the head, so an agent running tuned weights reports `confidence` as `None` and warns once at construction.
diff --git a/doc/environments.md b/doc/environments.md
index 421eb70..a7dbeea 100644
--- a/doc/environments.md
+++ b/doc/environments.md
@@ -21,7 +21,7 @@ Every environment module exposes the same surface:
| --- | --- |
| `TOOLS` | The decorated tool functions, each returning `{"ok": True, ...}` echoes of its arguments. |
| `SYSTEM` | The system prompt the agent is built with. |
-| `agent` | A `needle.Needle(tools=TOOLS, system=SYSTEM)`, constructed lazily on first access so importing never fetches the engine. Construction sets `NEEDLE_STRICT_VALIDATE=1` (if unset) so out-of-bounds calls are suppressed to refusals. |
+| `agent` | A `needle.Needle(tools=TOOLS, system=SYSTEM)`, constructed lazily on first access so importing never fetches the engine. `run_tests` treats a call whose `validation` reports ungrounded values or a negation as a refusal. |
| `TEST_CASES` | The frozen suite: dicts of `query`, expected `calls`, `category`, and optional `critical`. |
| `run_tests(min_confidence=0.0, verbose=True)` | Runs the suite against the shipped engine; returns `True` at >=90% pass with zero critical failures. |
diff --git a/llms.txt b/llms.txt
index a9e2ca6..ae3fb2b 100644
--- a/llms.txt
+++ b/llms.txt
@@ -124,7 +124,7 @@ if r["type"] == "call":
## Confidence gating
-`confidence` is the min of a post-hoc calibration head and the decode probability of the call. Calibration holds for the base model only: an agent constructed with `weights=` reports `confidence` as None (fine-tuning does not update the head). Pick a threshold per product; act at/above it, escalate below it:
+`confidence` is the min of a post-hoc calibration head and the decode probability of the call. Calibration holds for the base model only: an agent constructed with `weights=` reports `confidence` as None (fine-tuning does not update the head). Pick a threshold per product; act at/above it, escalate below it (`escalate` in the response applies the engine default of 0.15):
```python
r = agent.complete(user_text)
diff --git a/needle/environments/_harness.py b/needle/environments/_harness.py
index fb561f9..e165fe1 100644
--- a/needle/environments/_harness.py
+++ b/needle/environments/_harness.py
@@ -1,5 +1,4 @@
import json
-import os
import needle
@@ -9,7 +8,6 @@ _agents = {}
def agent_for(module):
key = module.__name__
if key not in _agents:
- os.environ.setdefault("NEEDLE_STRICT_VALIDATE", "1")
_agents[key] = needle.Needle(tools=module.TOOLS, system=module.SYSTEM)
return _agents[key]
@@ -25,6 +23,9 @@ def run_tests(module, min_confidence=0.0, verbose=True):
agent.reset()
response = agent.complete(case["query"])
got = response.get("function_calls") or []
+ validation = response.get("validation") or {}
+ if got and (validation.get("ungrounded") or validation.get("negation")):
+ got = []
if got and response.get("confidence", 0.0) < min_confidence:
got = []
want = case["calls"]