Skip to content

graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs

Python

Document usage boundaries and align evaluation receipts

Rach Pradhan4a739145f1b7
diff --git a/docs/adr/0187-optional-judgments-preserve-usage-uncertainty.md b/docs/adr/0187-optional-judgments-preserve-usage-uncertainty.md
new file mode 100644
index 00000000..d346e0ad
--- /dev/null
+++ b/docs/adr/0187-optional-judgments-preserve-usage-uncertainty.md
@@ -0,0 +1,27 @@
+# 0187. Optional judgments preserve credential and cost boundaries
+
+Status: accepted 2026-09-23
+
+## Context
+
+An optional judgment runs beside the main model request. Charging or
+authenticating it as though it were part of that request hides its route and
+can turn a missing receipt into an invented zero.
+
+## Decision
+
+Use the gateway login for this optional request, independently of the active
+model credential. The engine owns the behavior for both terminal and ACP
+sessions. If the login is absent or the request fails, skip the judgment
+without retrying it in the same session.
+
+Count reported tokens from a successful response, but leave its unsettled
+charge unknown. Mark a completed response without valid usage as missing
+usage, and a failed request attempt as uncertain. The session tally carries
+those distinctions to the terminal summary and ACP usage notification.
+Scripted mock judgments never affect the tally.
+
+## Validation
+
+`src/jev_tool.zig` and `src/acp_usage.zig` tests cover successful usage,
+missing usage, failed attempts, unknown cost, and mock isolation.
diff --git a/docs/adr/0188-streamed-tool-calls-retain-identity.md b/docs/adr/0188-streamed-tool-calls-retain-identity.md
new file mode 100644
index 00000000..923d11ad
--- /dev/null
+++ b/docs/adr/0188-streamed-tool-calls-retain-identity.md
@@ -0,0 +1,27 @@
+# 0188. Streamed tool calls retain their explicit identity
+
+Status: accepted 2026-09-23
+
+## Context
+
+A stream can reuse a numeric tool index for a later call. Merging solely by
+index can concatenate two argument objects, lose their separate identities,
+and prevent an otherwise complete successor from executing.
+
+## Decision
+
+Assemble calls by explicit call ID. An ID-less fragment continues the most
+recent call at its index, while a new ID at that index starts a distinct call.
+If the predecessor lacks a complete argument object, preserve it as invalid;
+do not repair it with fragments from the successor. Contradictory names also
+invalidate the affected call.
+
+Only structured call and result events establish tool execution. Text that
+merely resembles a command, including prose in a response, is not execution
+evidence.
+
+## Validation
+
+`src/agent_steps_tests.zig` covers reused indexes, incomplete predecessors,
+and contradictory fragments. The scripted behavior case in
+`evals/harness_behavior.jsonl` checks distinct calls through the request path.
diff --git a/docs/adr/README.md b/docs/adr/README.md
index 15a4372b..1f997fbe 100644
--- a/docs/adr/README.md
+++ b/docs/adr/README.md
@@ -212,6 +212,8 @@ record only when you need the evidence or the edge cases.
| [0184](0184-directory-scan-budgets-count-examined-entries.md) | Directory scan limits count examined entries before filtering; truncated listings and suggestions remain explicitly partial. |
| [0185](0185-input-semantics-belong-to-the-task-contract.md) | Empty-input, whitespace, and record-framing semantics come from the task contract, not global work instructions. |
| [0186](0186-tool-previews-use-original-source-ranges.md) | Tool previews select whole diagnostic lines from original omitted ranges within byte and line budgets. |
+| [0187](0187-optional-judgments-preserve-usage-uncertainty.md) | Optional judgments use an independent gateway login; tokens and unsettled cost remain distinct in CLI and ACP usage. |
+| [0188](0188-streamed-tool-calls-retain-identity.md) | Streamed tool calls use explicit IDs; reused indexes keep incomplete predecessors invalid, and prose is not execution evidence. |
## When to write one
diff --git a/graff-evals/measurement.py b/graff-evals/measurement.py
index 73539b7e..3f2d3bb8 100644
--- a/graff-evals/measurement.py
+++ b/graff-evals/measurement.py
@@ -26,7 +26,8 @@ def graff_usage(stderr):
result['cost_usd'] = float(m.group(6))
result['cost_kind'] = 'harness-reported'
tail = m.group(7)
- for key, pattern in [('sub_calls', r'(\d+) subscription call'), ('unpriced_calls', r'(\d+) call\(s\) on unpriced models')]:
+ for key, pattern in [('sub_calls', r'(\d+) subscription call'),
+ ('unpriced_calls', r'(\d+) call\(s\) (?:on unpriced models|with unknown cost)')]:
if hit := re.search(pattern, tail):
result[key] = int(hit.group(1))
if result.get('unpriced_calls'):
@@ -45,7 +46,13 @@ def graff_usage(stderr):
# The prefix alone is sufficient evidence of incompleteness, even if a
# newer runtime introduces an unfamiliar explanation in the tail.
subtotal = re.match(r'^\[usage\]\s+known subtotal:', m.group(0)) is not None
- if subtotal or result.get('missing_usage_calls') or result.get('unreported_failed_attempts'):
+ cost_only_subtotal = subtotal and result.get('unpriced_calls') and re.fullmatch(
+ r'\s*(?:·\s*\d+ subscription call\(s\), flat-rate \(not in \$\)\s*)?'
+ r'·\s*\d+ call\(s\) (?:on unpriced models|with unknown cost)\s*', tail)
+ if cost_only_subtotal:
+ result['usage_complete'] = True
+ result['known_cost_usd'] = float(m.group(6)) if m.group(6) is not None else None
+ elif subtotal or result.get('missing_usage_calls') or result.get('unreported_failed_attempts'):
result['usage_complete'] = False
for key in ('in', 'cached', 'writes', 'out'):
result['known_' + key] = result[key]
diff --git a/graff-evals/test_measurement.py b/graff-evals/test_measurement.py
index 96502282..29a677f2 100644
--- a/graff-evals/test_measurement.py
+++ b/graff-evals/test_measurement.py
@@ -34,6 +34,32 @@ class MeasurementTests(unittest.TestCase):
self.assertIsNone(measurement.graff_usage(base + '2 subscription call(s), flat-rate (not in $)')['cost_usd'])
self.assertEqual(measurement.graff_usage(base + '2 subscription call(s), flat-rate (not in $)')['cost_kind'], 'metered-only-subscription-excluded')
+ def test_known_subtotal_from_unknown_cost_keeps_measured_tokens(self):
+ footer = ('[usage] known subtotal: 2 api call(s) · 80 in (40 cached, 5 cache writes)'
+ ' + 9 out tokens · $0.001234 · 1 call(s) with unknown cost')
+ usage = measurement.graff_usage(footer)
+ self.assertTrue(usage['usage_complete'])
+ self.assertEqual((usage['in'], usage['cached'], usage['writes'], usage['out']), (80, 40, 5, 9))
+ self.assertEqual(usage['known_cost_usd'], .001234)
+ self.assertIsNone(usage['cost_usd'])
+ self.assertEqual(usage['unpriced_calls'], 1)
+ self.assertEqual(usage['cost_kind'], 'incomplete-unpriced')
+ legacy = measurement.graff_usage(footer.replace('with unknown cost', 'on unpriced models'))
+ self.assertEqual(legacy, usage)
+
+ def test_unknown_cost_does_not_hide_missing_or_failed_usage(self):
+ base = ('[usage] known subtotal: 2 api call(s) · 80 in (40 cached) + 9 out tokens · $0.001234'
+ ' · 1 call(s) with unknown cost')
+ for suffix in (' · totals incomplete: 1 call(s) missing usage (tokens and cost unknown)',
+ ' · totals incomplete: 1 failed request attempt(s) without usage (tokens and cost unknown)',
+ ' · future unexplained footer'):
+ with self.subTest(suffix=suffix):
+ usage = measurement.graff_usage(base + suffix)
+ self.assertFalse(usage['usage_complete'])
+ self.assertIsNone(usage['in'])
+ self.assertEqual(usage['known_in'], 80)
+ self.assertIsNone(usage['cost_usd'])
+
def test_missing_usage_preserves_subtotals_without_claiming_complete_totals(self):
footer = ('[usage] 3 api call(s) · 80 in (40 cached, 5 cache writes) + 9 out tokens · $0.001234'
' · totals incomplete: 1 call(s) missing usage (tokens and cost unknown)')
diff --git a/scripts/eval/tier1-manifest.json b/scripts/eval/tier1-manifest.json
index 74658966..c99e4d47 100644
--- a/scripts/eval/tier1-manifest.json
+++ b/scripts/eval/tier1-manifest.json
@@ -15,7 +15,7 @@
"src/repl.zig",
"TUI/root.zig"
],
- "test_count_baseline": 2742,
+ "test_count_baseline": 2750,
"test_count_slack": 25,
"required_invariants": [
{