graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs
Python
Document usage boundaries and align evaluation receipts
Rach Pradhan4a739145f1b7
diff --git a/docs/adr/0187-optional-judgments-preserve-usage-uncertainty.md b/docs/adr/0187-optional-judgments-preserve-usage-uncertainty.mdnew file mode 100644index 00000000..d346e0ad--- /dev/null+++ b/docs/adr/0187-optional-judgments-preserve-usage-uncertainty.md@@ -0,0 +1,27 @@+# 0187. Optional judgments preserve credential and cost boundaries++Status: accepted 2026-09-23++## Context++An optional judgment runs beside the main model request. Charging or+authenticating it as though it were part of that request hides its route and+can turn a missing receipt into an invented zero.++## Decision++Use the gateway login for this optional request, independently of the active+model credential. The engine owns the behavior for both terminal and ACP+sessions. If the login is absent or the request fails, skip the judgment+without retrying it in the same session.++Count reported tokens from a successful response, but leave its unsettled+charge unknown. Mark a completed response without valid usage as missing+usage, and a failed request attempt as uncertain. The session tally carries+those distinctions to the terminal summary and ACP usage notification.+Scripted mock judgments never affect the tally.++## Validation++`src/jev_tool.zig` and `src/acp_usage.zig` tests cover successful usage,+missing usage, failed attempts, unknown cost, and mock isolation.diff --git a/docs/adr/0188-streamed-tool-calls-retain-identity.md b/docs/adr/0188-streamed-tool-calls-retain-identity.mdnew file mode 100644index 00000000..923d11ad--- /dev/null+++ b/docs/adr/0188-streamed-tool-calls-retain-identity.md@@ -0,0 +1,27 @@+# 0188. Streamed tool calls retain their explicit identity++Status: accepted 2026-09-23++## Context++A stream can reuse a numeric tool index for a later call. Merging solely by+index can concatenate two argument objects, lose their separate identities,+and prevent an otherwise complete successor from executing.++## Decision++Assemble calls by explicit call ID. An ID-less fragment continues the most+recent call at its index, while a new ID at that index starts a distinct call.+If the predecessor lacks a complete argument object, preserve it as invalid;+do not repair it with fragments from the successor. Contradictory names also+invalidate the affected call.++Only structured call and result events establish tool execution. Text that+merely resembles a command, including prose in a response, is not execution+evidence.++## Validation++`src/agent_steps_tests.zig` covers reused indexes, incomplete predecessors,+and contradictory fragments. The scripted behavior case in+`evals/harness_behavior.jsonl` checks distinct calls through the request path.diff --git a/docs/adr/README.md b/docs/adr/README.mdindex 15a4372b..1f997fbe 100644--- a/docs/adr/README.md+++ b/docs/adr/README.md@@ -212,6 +212,8 @@ record only when you need the evidence or the edge cases.| [0184](0184-directory-scan-budgets-count-examined-entries.md) | Directory scan limits count examined entries before filtering; truncated listings and suggestions remain explicitly partial. || [0185](0185-input-semantics-belong-to-the-task-contract.md) | Empty-input, whitespace, and record-framing semantics come from the task contract, not global work instructions. || [0186](0186-tool-previews-use-original-source-ranges.md) | Tool previews select whole diagnostic lines from original omitted ranges within byte and line budgets. |+| [0187](0187-optional-judgments-preserve-usage-uncertainty.md) | Optional judgments use an independent gateway login; tokens and unsettled cost remain distinct in CLI and ACP usage. |+| [0188](0188-streamed-tool-calls-retain-identity.md) | Streamed tool calls use explicit IDs; reused indexes keep incomplete predecessors invalid, and prose is not execution evidence. |## When to write onediff --git a/graff-evals/measurement.py b/graff-evals/measurement.pyindex 73539b7e..3f2d3bb8 100644--- a/graff-evals/measurement.py+++ b/graff-evals/measurement.py@@ -26,7 +26,8 @@ def graff_usage(stderr):result['cost_usd'] = float(m.group(6))result['cost_kind'] = 'harness-reported'tail = m.group(7)- for key, pattern in [('sub_calls', r'(\d+) subscription call'), ('unpriced_calls', r'(\d+) call\(s\) on unpriced models')]:+ for key, pattern in [('sub_calls', r'(\d+) subscription call'),+ ('unpriced_calls', r'(\d+) call\(s\) (?:on unpriced models|with unknown cost)')]:if hit := re.search(pattern, tail):result[key] = int(hit.group(1))if result.get('unpriced_calls'):@@ -45,7 +46,13 @@ def graff_usage(stderr):# The prefix alone is sufficient evidence of incompleteness, even if a# newer runtime introduces an unfamiliar explanation in the tail.subtotal = re.match(r'^\[usage\]\s+known subtotal:', m.group(0)) is not None- if subtotal or result.get('missing_usage_calls') or result.get('unreported_failed_attempts'):+ cost_only_subtotal = subtotal and result.get('unpriced_calls') and re.fullmatch(+ r'\s*(?:·\s*\d+ subscription call\(s\), flat-rate \(not in \$\)\s*)?'+ r'·\s*\d+ call\(s\) (?:on unpriced models|with unknown cost)\s*', tail)+ if cost_only_subtotal:+ result['usage_complete'] = True+ result['known_cost_usd'] = float(m.group(6)) if m.group(6) is not None else None+ elif subtotal or result.get('missing_usage_calls') or result.get('unreported_failed_attempts'):result['usage_complete'] = Falsefor key in ('in', 'cached', 'writes', 'out'):result['known_' + key] = result[key]diff --git a/graff-evals/test_measurement.py b/graff-evals/test_measurement.pyindex 96502282..29a677f2 100644--- a/graff-evals/test_measurement.py+++ b/graff-evals/test_measurement.py@@ -34,6 +34,32 @@ class MeasurementTests(unittest.TestCase):self.assertIsNone(measurement.graff_usage(base + '2 subscription call(s), flat-rate (not in $)')['cost_usd'])self.assertEqual(measurement.graff_usage(base + '2 subscription call(s), flat-rate (not in $)')['cost_kind'], 'metered-only-subscription-excluded')+ def test_known_subtotal_from_unknown_cost_keeps_measured_tokens(self):+ footer = ('[usage] known subtotal: 2 api call(s) · 80 in (40 cached, 5 cache writes)'+ ' + 9 out tokens · $0.001234 · 1 call(s) with unknown cost')+ usage = measurement.graff_usage(footer)+ self.assertTrue(usage['usage_complete'])+ self.assertEqual((usage['in'], usage['cached'], usage['writes'], usage['out']), (80, 40, 5, 9))+ self.assertEqual(usage['known_cost_usd'], .001234)+ self.assertIsNone(usage['cost_usd'])+ self.assertEqual(usage['unpriced_calls'], 1)+ self.assertEqual(usage['cost_kind'], 'incomplete-unpriced')+ legacy = measurement.graff_usage(footer.replace('with unknown cost', 'on unpriced models'))+ self.assertEqual(legacy, usage)++ def test_unknown_cost_does_not_hide_missing_or_failed_usage(self):+ base = ('[usage] known subtotal: 2 api call(s) · 80 in (40 cached) + 9 out tokens · $0.001234'+ ' · 1 call(s) with unknown cost')+ for suffix in (' · totals incomplete: 1 call(s) missing usage (tokens and cost unknown)',+ ' · totals incomplete: 1 failed request attempt(s) without usage (tokens and cost unknown)',+ ' · future unexplained footer'):+ with self.subTest(suffix=suffix):+ usage = measurement.graff_usage(base + suffix)+ self.assertFalse(usage['usage_complete'])+ self.assertIsNone(usage['in'])+ self.assertEqual(usage['known_in'], 80)+ self.assertIsNone(usage['cost_usd'])+def test_missing_usage_preserves_subtotals_without_claiming_complete_totals(self):footer = ('[usage] 3 api call(s) · 80 in (40 cached, 5 cache writes) + 9 out tokens · $0.001234'' · totals incomplete: 1 call(s) missing usage (tokens and cost unknown)')diff --git a/scripts/eval/tier1-manifest.json b/scripts/eval/tier1-manifest.jsonindex 74658966..c99e4d47 100644--- a/scripts/eval/tier1-manifest.json+++ b/scripts/eval/tier1-manifest.json@@ -15,7 +15,7 @@"src/repl.zig","TUI/root.zig"],- "test_count_baseline": 2742,+ "test_count_baseline": 2750,"test_count_slack": 25,"required_invariants": [{