Skip to content

Turbocharge Claude Code, Cursor, Codex, Gemini & every coding agent: faster, cheaper, with contextual understanding specific to your codebase.

feat(telemetry): count the turns that told the user what graft saved

anirudhkumar-nanonetsdd686597d6e0
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4bb1f8bc..6c166046 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -75,6 +75,33 @@
### Added
+- **Telemetry can now tell "graft saved tokens" apart from "the user was told".**
+ `saved_tokens_bucket` counted what graft *computed* — every
+ `[graft] tokens saved ≈ N` footer the PostToolUse accumulator swept up — so a
+ turn that saved 20,000 tokens in silence and a turn that saved nothing were the
+ same number. `session_summary` now also carries `graft_turns_bucket` and
+ `reported_turns_bucket`: of the turns that used graft, how many closed with the
+ one-line tally `SKILL.md` and `SAVINGS_TURN_NUDGE` both ask for.
+
+ The agent's own prose lives in one place a hook can reach, so at the end of a
+ turn that used graft the Stop hook reads the tail of the host transcript named
+ on its stdin and checks the reply for a "graft saved ~N tokens" line. Only a
+ count of turns is kept, and only as a bucket — no reply, no fragment, not even a
+ length. `TELEMETRY.md` documents the local read in full.
+
+ A turn that can't be checked — a host whose Stop hook names no transcript, an
+ unreadable file, a Stop racing the transcript write — is counted in *neither*
+ total, so the ratio means "of the turns we could read" rather than deflating to
+ zero on every editor but Claude Code. One reply is one turn however often Stop
+ fires, guarded by the id of the last reply examined.
+
+- **`scripts/tally-audit.mjs`**, the same ratio offline and at full resolution:
+ which turns were silent, whether the number the agent reported matched graft's
+ own footers, and whether it named the call count. Run it when the aggregate says
+ something surprising. It also flags per-turn savings estimates large enough to be
+ artifacts rather than savings.
+
+
- **`graft init` converges instead of accumulating, and `graft uninstall` removes
graft entirely.** `init` wrote the files the selected agents needed and never
looked at the rest, so a repo wired by an older version — or by the same version
diff --git a/TELEMETRY.md b/TELEMETRY.md
index a54fbfa4..8d00e542 100644
--- a/TELEMETRY.md
+++ b/TELEMETRY.md
@@ -39,7 +39,7 @@ The events:
| `build_completed` | `files_bucket`, `langs`, `mode` (`fast`/`deep`), `duration_bucket`, `incremental` | A build succeeds |
| `build_failed` | `stage`, `code` — both fixed enums | A build throws |
| `query` | `command`, `surface` (`cli`/`mcp`/`hook`), `hit` (`ask` only) | Any query command |
-| `session_summary` | `graft_reads_bucket`, `source_reads_bucket`, `saved_tokens_bucket` | Once, after an agent session ends |
+| `session_summary` | `graft_reads_bucket`, `source_reads_bucket`, `saved_tokens_bucket`, `graft_turns_bucket`, `reported_turns_bucket` | Once, after an agent session ends |
Two rules govern every value above, and both are enforced in code rather than by
review:
@@ -72,11 +72,32 @@ An example event, in full:
Run `graft telemetry debug` to print the exact batch your machine would send. It
sends nothing.
+### One thing graft reads locally
+
+Two of those properties — `graft_turns_bucket` and `reported_turns_bucket` — need
+to know something the others don't: whether the reply you actually read said what
+graft saved. graft computes a saving on every retrieval call, but a turn that
+saves 20,000 tokens in silence and a turn that saves nothing look identical in
+`saved_tokens_bucket`, and that difference is the whole question.
+
+The agent's own prose lives in one place a hook can reach: the transcript file
+your editor writes, named on the Stop hook's stdin. So at the end of a turn that
+used graft, graft reads the tail of that file, checks the reply for a "graft saved
+~N tokens" line, and increments one of two counters. Nothing out of the file is
+stored, and nothing out of it is sent — not the reply, not a fragment of it, not
+its length. Two counts of turns cross the wire, as buckets, once per session.
+
+A turn graft cannot check — an editor whose Stop hook names no transcript, an
+unreadable file — is counted in neither total, so the ratio always means "of the
+turns we could read".
+
## What is never sent
No source code. No file paths, repo names, organisation names, git remotes, or
branch names. No symbol names, node summaries, or anything out of the graph. No
-query strings, prompts, or agent output. No error messages or stack traces — a
+query strings, prompts, or agent output — graft reads the last reply of a
+graft-using turn locally to decide whether it mentioned the saving (see "One
+thing graft reads locally"); only the resulting count is sent, never the text. No error messages or stack traces — a
failure contributes a code from a fixed list and nothing else. No environment
variables, API keys, model names, hostnames, usernames, or email addresses.
diff --git a/scripts/tally-audit.mjs b/scripts/tally-audit.mjs
new file mode 100644
index 00000000..9891167a
--- /dev/null
+++ b/scripts/tally-audit.mjs
@@ -0,0 +1,140 @@
+/**
+ * "Saved vs said": how often does the agent actually TELL the user what graft saved?
+ *
+ * node scripts/tally-audit.mjs [transcript-dir] [--json]
+ *
+ * `savedTokens` (src/claude/state.ts) counts what graft *computed* — every
+ * `[graft] tokens saved ≈ N` footer the PostToolUse accumulator swept up. This
+ * script measures the other half, offline and in full detail: of the turns that
+ * used graft, how many closed with the one-line tally SKILL.md asks for, was the
+ * number right, and did it name the call count.
+ *
+ * The shipped metric (`graft_turns_bucket` / `reported_turns_bucket` on
+ * `session_summary`) is the same ratio at bucket resolution, which is all that
+ * can cross the wire. Run this locally when the aggregate says something
+ * surprising and you need to see which turns and why.
+ *
+ * Reads Claude Code's own transcript JSONL, which defaults to
+ * ~/.claude/projects/<slugified-cwd>/. Nothing leaves the machine.
+ */
+import { readFileSync, readdirSync, existsSync } from "node:fs";
+import { join } from "node:path";
+import { homedir } from "node:os";
+
+const SAVED_FOOTER = /\[graft\] tokens saved ≈ ([\d,]+)/g;
+const TALLY = /graft\s+saved\s*[~≈]?\s*([\d,.]+)\s*(k|m)?\s*(?:tok|tokens)/i;
+const CALLS = /(\d+)\s*calls?/i;
+/** How far past the tally to look for its call count. The tally is one line
+ * ("🌱 graft saved ~12,400 tokens this turn (3 calls)"); scanning the rest of
+ * the reply instead would match any unrelated "3 calls" further down and report
+ * near-perfect compliance for a field the agent mostly omits. */
+const TALLY_WINDOW = 60;
+/** Above this, a per-turn "saved" figure is an estimator artifact rather than a
+ * saving — a `--depth all` closure or a vendored tree summing whole-file
+ * baselines nobody would have read. Reported separately, never averaged in. */
+const IMPLAUSIBLE_TOKENS = 1_000_000;
+
+const args = process.argv.slice(2).filter((a) => a !== "--json");
+const asJson = process.argv.includes("--json");
+const dir = args[0] ?? defaultTranscriptDir();
+
+function defaultTranscriptDir() {
+ const slug = process.cwd().replace(/[/.]/g, "-");
+ return join(homedir(), ".claude", "projects", slug);
+}
+function textOf(content) {
+ if (typeof content === "string") return content;
+ if (!Array.isArray(content)) return "";
+ return content.map((p) => (p?.type === "text" ? p.text ?? "" : "")).join("\n");
+}
+function isUserPrompt(o) {
+ if (o.type !== "user" || o.isSidechain || o.isMeta) return false;
+ const c = o.message?.content;
+ if (typeof c === "string") return true;
+ return Array.isArray(c) && !c.some((p) => p?.type === "tool_result");
+}
+function median(ns) {
+ if (!ns.length) return 0;
+ const s = [...ns].sort((a, b) => a - b);
+ const mid = s.length >> 1;
+ return s.length % 2 ? s[mid] : Math.round((s[mid - 1] + s[mid]) / 2);
+}
+
+if (!existsSync(dir)) {
+ console.error(`no transcripts at ${dir}`);
+ process.exit(1);
+}
+
+const turns = [];
+for (const file of readdirSync(dir).filter((f) => f.endsWith(".jsonl"))) {
+ let turn = null;
+ const flush = () => { if (turn?.calls > 0) turns.push(turn); };
+ for (const line of readFileSync(join(dir, file), "utf8").split("\n")) {
+ if (!line.trim()) continue;
+ let o;
+ try { o = JSON.parse(line); } catch { continue; }
+ if (o.isSidechain) continue; // a subagent's prose is not what the user read
+ if (isUserPrompt(o)) { flush(); turn = { file, calls: 0, saved: 0, said: null, statedCalls: null }; continue; }
+ if (!turn) continue;
+ if (o.type === "user") { // a tool_result wearing the user role
+ for (const m of JSON.stringify(o.message?.content ?? "").matchAll(SAVED_FOOTER)) {
+ turn.calls++;
+ turn.saved += Number(m[1].replace(/,/g, "")) || 0;
+ }
+ }
+ if (o.type === "assistant") {
+ const m = TALLY.exec(textOf(o.message?.content));
+ if (m) {
+ let n = Number(m[1].replace(/,/g, ""));
+ if (m[2]?.toLowerCase() === "k") n *= 1e3;
+ if (m[2]?.toLowerCase() === "m") n *= 1e6;
+ turn.said = n;
+ const c = CALLS.exec(textOf(o.message?.content).slice(m.index, m.index + TALLY_WINDOW));
+ turn.statedCalls = c ? Number(c[1]) : null;
+ }
+ }
+ }
+ flush();
+}
+
+const said = turns.filter((t) => t.said !== null);
+const plausible = turns.filter((t) => t.saved < IMPLAUSIBLE_TOKENS);
+const outliers = turns.filter((t) => t.saved >= IMPLAUSIBLE_TOKENS);
+const fidelity = { exact: 0, over: 0, under: 0 };
+for (const t of said.filter((t) => t.saved > 0)) {
+ const r = t.said / t.saved;
+ if (r >= 0.9 && r <= 1.1) fidelity.exact++;
+ else if (r > 1.1) fidelity.over++;
+ else fidelity.under++;
+}
+
+const report = {
+ transcripts: readdirSync(dir).filter((f) => f.endsWith(".jsonl")).length,
+ graft_turns: turns.length,
+ reported: said.length,
+ silent: turns.length - said.length,
+ reported_pct: turns.length ? Number(((said.length / turns.length) * 100).toFixed(1)) : 0,
+ stated_call_count: said.filter((t) => t.statedCalls !== null).length,
+ median_saved_per_turn: median(plausible.map((t) => t.saved)),
+ fidelity,
+ implausible_turns: outliers.length,
+ implausible_max: outliers.length ? Math.max(...outliers.map((t) => t.saved)) : 0,
+};
+
+if (asJson) {
+ console.log(JSON.stringify(report, null, 2));
+} else {
+ const { graft_turns: n } = report;
+ console.log(`transcripts scanned: ${report.transcripts} (${dir})`);
+ console.log(`turns that used graft: ${n}`);
+ console.log(` ...reported a tally: ${report.reported} (${report.reported_pct}%)`);
+ console.log(` ...silent: ${report.silent}`);
+ console.log(` ...named the call count ${report.stated_call_count}/${report.reported}`);
+ console.log(`median saved per turn: ${report.median_saved_per_turn.toLocaleString()} tok`);
+ console.log(`reported number vs footer sum (±10%): exact ${fidelity.exact} · over ${fidelity.over} · under ${fidelity.under}`);
+ if (outliers.length) {
+ console.log(`\n⚠ ${outliers.length} turn(s) claim ≥${IMPLAUSIBLE_TOKENS.toLocaleString()} tokens saved ` +
+ `(max ${report.implausible_max.toLocaleString()}) — an estimator artifact, excluded from the median. ` +
+ `See savingsFor() in src/context/savings.ts.`);
+ }
+}
diff --git a/src/claude/hooks.ts b/src/claude/hooks.ts
index 3f055337..52854d32 100644
--- a/src/claude/hooks.ts
+++ b/src/claude/hooks.ts
@@ -10,6 +10,7 @@ import { graftCliPath, claudeScriptPath } from './paths.js';
import { runUpkeep } from '../upkeep-run.js';
import { runningVersion } from '../upkeep.js';
import { flushClosedSessions } from '../telemetry/sessions.js';
+import { hasSavingsTally, lastAssistantTurn } from './tally.js';
import { scopeOf, scopesOfGraph } from '../graph/scopes.js';
/** Prompts shorter than this never trigger retrieval — they are almost always
@@ -244,10 +245,47 @@ function handleToolSavings(input: any, dir: string): void {
const id = input?.session_id || 'default';
const s = readSession(dir, id);
s.savedTokens = (s.savedTokens ?? 0) + total;
+ // This turn used graft, so it is owed a tally in the reply. countTallyTurn()
+ // at Stop resolves whether it got one; a flag rather than a count because a
+ // turn with four graft calls is still one reply to the user.
+ s.turnUsedGraft = true;
writeSession(dir, id, s);
}
-function handleStop(dir: string): void {
+/**
+ * At turn end: did the reply the user just read say what graft saved?
+ *
+ * Runs only on turns the tool-savings hook flagged, so a conversational turn
+ * costs nothing. A turn we cannot observe — a host whose Stop hook names no
+ * transcript, an unreadable file, or a Stop that fires before the final prose
+ * is on disk — is counted in NEITHER total: the ratio these two numbers form
+ * has to mean "of the turns we could check", not "of the turns we tried to".
+ */
+function countTallyTurn(input: any, dir: string): void {
+ try {
+ const id = input?.session_id || 'default';
+ const s = readSession(dir, id);
+ if (!s.turnUsedGraft) return;
+ const turn = lastAssistantTurn(input?.transcript_path);
+ // Same reply as last time we looked: no new prose has landed, so this Stop
+ // is a duplicate or a race with the transcript write. Drop the turn rather
+ // than judge it on a stale message.
+ if (!turn || turn.uuid === s.lastTallyUuid) {
+ writeSession(dir, id, { ...s, turnUsedGraft: false });
+ return;
+ }
+ s.graftTurns = (s.graftTurns ?? 0) + 1;
+ if (hasSavingsTally(turn.text)) s.reportedTurns = (s.reportedTurns ?? 0) + 1;
+ s.turnUsedGraft = false;
+ s.lastTallyUuid = turn.uuid;
+ writeSession(dir, id, s);
+ } catch {
+ // A turn-end metric is never worth failing the graph sync over.
+ }
+}
+
+function handleStop(input: any, dir: string): void {
+ countTallyTurn(input, dir);
// sync-run.js ships next to this module inside the package, so it resolves in
// any repo that installs graft (not just graft's own). Defensive existsSync:
// if the package is somehow incomplete, skip rather than wedge on syncing:true.
@@ -292,9 +330,9 @@ export async function main(event: string): Promise<void> {
if (event === 'tool-savings') { handleToolSavings(input, dir); return; }
- if (event === 'stop') { handleStop(dir); return; }
+ if (event === 'stop') { handleStop(input, dir); return; }
- if (event === 'post-edit-sync') { await handlePostEdit(input, dir); handleStop(dir); return; }
+ if (event === 'post-edit-sync') { await handlePostEdit(input, dir); handleStop(input, dir); return; }
if (event === 'prompt') {
const prompt = String(input?.prompt ?? '').trim();
diff --git a/src/claude/state.ts b/src/claude/state.ts
index 45d9f252..8a142546 100644
--- a/src/claude/state.ts
+++ b/src/claude/state.ts
@@ -36,6 +36,19 @@ export interface SessionState {
/** Weak-match nudges spent this session, capped so the line stays signal.
* Optional for the same backwards-compatibility reason as above. */
nudges?: number;
+ /** Turns this session in which the agent used a graft retrieval tool — the
+ * denominator for "did it tell the user what that saved". Counted at Stop,
+ * and only for turns whose reply we could actually read (see claude/tally.ts). */
+ graftTurns?: number;
+ /** Of those turns, the ones whose reply carried a "graft saved ~N tokens"
+ * tally. `graftTurns - reportedTurns` is silent value: saved, never said. */
+ reportedTurns?: number;
+ /** Set by the tool-savings hook when the current turn touched graft, cleared
+ * at Stop once the turn has been counted. Transient, not a total. */
+ turnUsedGraft?: boolean;
+ /** `uuid` of the last assistant reply already examined, so a Stop that fires
+ * without new prose (or fires twice) can't count one reply as two turns. */
+ lastTallyUuid?: string;
/** Set once this session has been rolled up into a `session_summary`
* telemetry event, so a resumed or long-lived session is counted once.
* A flag rather than deleting the file: the file still holds `lastQuery` and
diff --git a/src/claude/tally.ts b/src/claude/tally.ts
new file mode 100644
index 00000000..1ec1a324
--- /dev/null
+++ b/src/claude/tally.ts
@@ -0,0 +1,131 @@
+/**
+ * Did the agent actually TELL the user what graft saved?
+ *
+ * `savedTokens` (state.ts) counts what graft *computed* — every
+ * `[graft] tokens saved ≈ N` footer the PostToolUse accumulator swept up. That
+ * number is real whether or not anyone sees it. This module measures the other
+ * half: whether the reply the user read closed with the one-line tally that
+ * SKILL.md and `SAVINGS_TURN_NUDGE` (context/savings.ts) both ask for. A turn
+ * that saved 20k tokens in silence is a turn where the product did its job and
+ * got no credit for it, and the two are indistinguishable in the numbers we
+ * ship today.
+ *
+ * The assistant's own prose exists in exactly one place a hook can reach: the
+ * host's transcript JSONL, named on the Stop hook's stdin as `transcript_path`.
+ * So this reads that file — locally, at turn end, for a boolean. Nothing out of
+ * it is stored or sent; the session keeps a count and the telemetry contract
+ * carries that count as a bucket. See TELEMETRY.md.
+ *
+ * Only the tail is read. A transcript grows for the life of a session and can
+ * reach many megabytes, while the answer is always in its last few entries — a
+ * hook that read the whole file would get slower every turn for no extra signal.
+ */
+import { closeSync, fstatSync, openSync, readSync } from 'node:fs';
+
+/**
+ * How much of the transcript's end to read. Comfortably larger than any single
+ * turn's worth of entries, small enough that the read cost is flat as the
+ * session grows.
+ */
+const TAIL_BYTES = 1024 * 1024;
+
+/**
+ * The tally, as the agent is actually asked to write it: a "graft saved ~N
+ * tokens" claim in prose. Deliberately loose about the decoration around it —
+ * the 🌱, the "this turn", the "(3 calls)" suffix and the thousands separator
+ * are all optional in practice, and a metric that only counted the one exact
+ * phrasing from the example would measure the agent's formatting rather than
+ * whether the user was told. Anchored on "graft saved" + a number + "tokens" so
+ * ordinary prose about graft can't trip it.
+ */
+const TALLY = /graft\s+saved\s*[~≈]?\s*[\d,.]+\s*[km]?\s*(?:tok|tokens)/i;
+
+/** Did this reply tell the user what graft saved? */
+export function hasSavingsTally(text: string): boolean {
+ return TALLY.test(text);
+}
+
+/** One turn's worth of assistant prose, plus the id of the entry it ended on. */
+export interface AssistantTurn {
+ /** `uuid` of the last assistant entry, so the same reply is never counted twice. */
+ uuid: string;
+ /** Every text block the agent emitted since the last user prompt, joined. */
+ text: string;
+}
+
+/** Read the last {@link TAIL_BYTES} of a file as utf8, dropping the leading
+ * partial line a byte-offset read leaves behind. Returns '' on any I/O trouble —
+ * a hook never fails over a metric. */
+function readTail(path: string): string {
+ let fd: number | undefined;
+ try {
+ fd = openSync(path, 'r');
+ const size = fstatSync(fd).size;
+ const len = Math.min(size, TAIL_BYTES);
+ const buf = Buffer.allocUnsafe(len);
+ readSync(fd, buf, 0, len, size - len);
+ const raw = buf.toString('utf8');
+ return len < size ? raw.slice(raw.indexOf('\n') + 1) : raw;
+ } catch {
+ return '';
+ } finally {
+ if (fd !== undefined) { try { closeSync(fd); } catch { /* already gone */ } }
+ }
+}
+
+/** Is this entry a user's own prompt, rather than a tool result wearing the
+ * `user` role? Tool results carry a `tool_result` part; a real prompt does not.
+ * The boundary matters because the tally can land in any text block of the turn,
+ * not only the last one. */
+function isUserPrompt(entry: any): boolean {
+ if (entry?.type !== 'user' || entry?.isMeta) return false;
+ const content = entry?.message?.content;
+ if (typeof content === 'string') return true;
+ return Array.isArray(content) && !content.some((p: any) => p?.type === 'tool_result');
+}
+
+function textOf(content: unknown): string {
+ if (typeof content === 'string') return content;
+ if (!Array.isArray(content)) return '';
+ return content.map((p: any) => (p?.type === 'text' ? String(p.text ?? '') : '')).join('\n');
+}
+
+/**
+ * The assistant prose of the turn the transcript ends on, or null when it can't
+ * be read (no path — a host whose Stop hook doesn't name one — unreadable file,
+ * or a tail with no assistant text in it).
+ *
+ * Null is meaningfully different from "no tally found": a turn we cannot observe
+ * is left out of BOTH the numerator and the denominator by the caller, so a host
+ * that exposes no transcript reports no ratio rather than a ratio of zero.
+ *
+ * Sidechain entries (subagents) are skipped throughout — a subagent's own prose
+ * is never what the user read.
+ */
+export function lastAssistantTurn(transcriptPath: unknown): AssistantTurn | null {
+ if (typeof transcriptPath !== 'string' || !transcriptPath) return null;
+ const tail = readTail(transcriptPath);
+ if (!tail) return null;
+
+ const entries: any[] = [];
+ for (const line of tail.split('\n')) {
+ if (!line.trim()) continue;
+ try { entries.push(JSON.parse(line)); } catch { /* clipped or partial line */ }
+ }
+
+ // Walk back to the turn boundary, collecting assistant text on the way.
+ const parts: string[] = [];
+ let uuid: string | null = null;
+ for (let i = entries.length - 1; i >= 0; i--) {
+ const e = entries[i];
+ if (e?.isSidechain) continue;
+ if (isUserPrompt(e)) break;
+ if (e?.type !== 'assistant') continue;
+ const text = textOf(e?.message?.content);
+ if (!text.trim()) continue;
+ if (uuid === null && typeof e?.uuid === 'string') uuid = e.uuid;
+ parts.unshift(text);
+ }
+ if (uuid === null || parts.length === 0) return null;
+ return { uuid, text: parts.join('\n') };
+}
diff --git a/src/telemetry/contract.ts b/src/telemetry/contract.ts
index cb380232..f62884d3 100644
--- a/src/telemetry/contract.ts
+++ b/src/telemetry/contract.ts
@@ -43,8 +43,13 @@ export const EVENTS: Record<string, ReadonlySet<string>> = {
/** One query, from any surface. The DAU backbone and the dead-command detector. */
query: new Set<string>(['command', 'surface', 'hit']),
/** One closed agent session, summarised. `graft_reads` vs `source_reads` is
- * the single number that says whether an agent prefers graft to grep. */
- session_summary: new Set<string>(['graft_reads_bucket', 'source_reads_bucket', 'saved_tokens_bucket']),
+ * the single number that says whether an agent prefers graft to grep; the two
+ * `*_turns` buckets are the follow-up question — of the turns that used graft,
+ * how many told the user what it saved. Both are counts of turns, never text. */
+ session_summary: new Set<string>([
+ 'graft_reads_bucket', 'source_reads_bucket', 'saved_tokens_bucket',
+ 'graft_turns_bucket', 'reported_turns_bucket',
+ ]),
};
/** Stamped on every event by `track()`; not listed per event above. */
diff --git a/src/telemetry/sessions.ts b/src/telemetry/sessions.ts
index 12dca218..126ba6a0 100644
--- a/src/telemetry/sessions.ts
+++ b/src/telemetry/sessions.ts
@@ -67,6 +67,12 @@ export function flushClosedSessions(
graft_reads_bucket: countBucket(s.graftReads ?? 0),
source_reads_bucket: countBucket(s.sourceReads ?? 0),
saved_tokens_bucket: savedTokensBucket(s.savedTokens ?? 0),
+ // Saved vs *said*: the gap between these two is value the user never
+ // heard about. Only turns whose reply was actually readable are in
+ // either count (claude/tally.ts), so a host that exposes no transcript
+ // reports 0/0 rather than a misleading 0-out-of-many.
+ graft_turns_bucket: countBucket(s.graftTurns ?? 0),
+ reported_turns_bucket: countBucket(s.reportedTurns ?? 0),
},
{ repo, host: 'claude-code', home, env },
);
diff --git a/test/claude-tally.test.ts b/test/claude-tally.test.ts
new file mode 100644
index 00000000..64ed0819
--- /dev/null
+++ b/test/claude-tally.test.ts
@@ -0,0 +1,168 @@
+/**
+ * "Saved" vs "said": the turn-end check that measures whether the agent actually
+ * told the user what graft saved. Its rules: read only the last turn, never
+ * count one reply twice, and count NEITHER total for a turn we cannot observe.
+ */
+import { test } from 'node:test';
+import assert from 'node:assert/strict';
+import { mkdtempSync, mkdirSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { hasSavingsTally, lastAssistantTurn } from '../src/claude/tally.js';
+import { main } from '../src/claude/hooks.js';
+import { readSession, writeSession } from '../src/claude/state.js';
+
+test('hasSavingsTally accepts the phrasings agents actually write', () => {
+ for (const yes of [
+ '🌱 graft saved ~12,400 tokens this turn (3 calls)',
+ 'graft saved ~12k tokens this turn',
+ 'Graft saved 8,100 tokens across 2 calls.',
+ 'graft saved ≈ 900 tok this turn',
+ ]) assert.equal(hasSavingsTally(yes), true, yes);
+});
+
+test('hasSavingsTally is not tripped by ordinary prose about graft', () => {
+ for (const no of [
+ 'I used graft to find the call sites.',
+ 'graft saved me a lot of time here', // no number, no unit
+ 'The 3 calls to graft returned 8 hits each.',
+ '',
+ ]) assert.equal(hasSavingsTally(no), false, no);
+});
+
+/** A transcript line as Claude Code writes it. */
+function assistant(uuid: string, text: string): string {
+ return JSON.stringify({ type: 'assistant', uuid, message: { role: 'assistant', content: [{ type: 'text', text }] } });
+}
+function userPrompt(text: string): string {
+ return JSON.stringify({ type: 'user', uuid: `u-${text}`, message: { role: 'user', content: text } });
+}
+function toolResult(text: string): string {
+ return JSON.stringify({ type: 'user', message: { role: 'user', content: [{ type: 'tool_result', content: text }] } });
+}
+function transcript(lines: string[]): string {
+ const p = join(mkdtempSync(join(tmpdir(), 'graft-tally-')), 'session.jsonl');
+ writeFileSync(p, lines.join('\n') + '\n');
+ return p;
+}
+
+test('the whole last turn is read, not just its final text block', () => {
+ const p = transcript([
+ userPrompt('older question'),
+ assistant('a1', 'no tally in this earlier turn'),
+ userPrompt('how does auth work'),
+ assistant('a2', '🌱 graft saved ~5,000 tokens this turn'),
+ toolResult('[graft] tokens saved ≈ 1,000'),
+ assistant('a3', 'Done.'),
+ ]);
+ const turn = lastAssistantTurn(p)!;
+ assert.equal(turn.uuid, 'a3', 'uuid is the last assistant entry of the turn');
+ assert.equal(hasSavingsTally(turn.text), true, 'a tally in an earlier block of the same turn still counts');
+});
+
+test('the previous turn is not read into this one', () => {
+ const p = transcript([
+ userPrompt('first'),
+ assistant('a1', '🌱 graft saved ~9,000 tokens this turn'),
+ userPrompt('second'),
+ assistant('a2', 'Nothing to report.'),
+ ]);
+ const turn = lastAssistantTurn(p)!;
+ assert.equal(hasSavingsTally(turn.text), false);
+});
+
+test('a subagent’s prose is never what the user read', () => {
+ const p = transcript([
+ userPrompt('go'),
+ JSON.stringify({ type: 'assistant', uuid: 's1', isSidechain: true,
+ message: { role: 'assistant', content: [{ type: 'text', text: '🌱 graft saved ~4,000 tokens this turn' }] } }),
+ assistant('a1', 'The subagent finished.'),
+ ]);
+ assert.equal(hasSavingsTally(lastAssistantTurn(p)!.text), false);
+});
+
+test('an unobservable turn yields null rather than a false negative', () => {
+ assert.equal(lastAssistantTurn(undefined), null, 'no transcript_path (e.g. Codex)');
+ assert.equal(lastAssistantTurn('/nope/missing.jsonl'), null, 'unreadable file');
+ assert.equal(lastAssistantTurn(transcript([userPrompt('go')])), null, 'no assistant prose yet');
+});
+
+/* ---------------------------------------------------------------------- */
+/* the hook path: tool-savings flags the turn, stop resolves it */
+/* ---------------------------------------------------------------------- */
+
+async function runHook(event: string, stdin: object): Promise<void> {
+ process.env.GRAFT_TEST_STDIN = JSON.stringify(stdin);
+ try { await main(event); } finally { delete process.env.GRAFT_TEST_STDIN; }
+}
+
+function repo(): string {
+ const d = mkdtempSync(join(tmpdir(), 'graft-tally-repo-'));
+ mkdirSync(join(d, 'graft', '.cache', 'session'), { recursive: true });
+ process.env.CLAUDE_PROJECT_DIR = d;
+ return d;
+}
+
+test('a graft turn that reports the tally counts in both totals', async () => {
+ const d = repo();
+ await runHook('tool-savings', { session_id: 's1', tool_response: '[graft] tokens saved ≈ 4,000' });
+ assert.equal(readSession(d, 's1').turnUsedGraft, true, 'the turn is flagged as owing a tally');
+ const p = transcript([userPrompt('how does auth work'), assistant('a1', '🌱 graft saved ~4,000 tokens this turn (1 call)')]);
+ await runHook('stop', { session_id: 's1', transcript_path: p });
+ const s = readSession(d, 's1');
+ assert.equal(s.graftTurns, 1);
+ assert.equal(s.reportedTurns, 1);
+ assert.equal(s.turnUsedGraft, false, 'the flag is cleared so the next turn starts clean');
+});
+
+test('a silent graft turn counts in the denominator only', async () => {
+ const d = repo();
+ await runHook('tool-savings', { session_id: 's1', tool_response: '[graft] tokens saved ≈ 20,000' });
+ const p = transcript([userPrompt('how does auth work'), assistant('a1', 'Auth is handled in src/auth.ts.')]);
+ await runHook('stop', { session_id: 's1', transcript_path: p });
+ const s = readSession(d, 's1');
+ assert.equal(s.graftTurns, 1);
+ assert.equal(s.reportedTurns ?? 0, 0, 'saved 20k, said nothing — that is the gap we ship');
+});
+
+test('a turn that never touched graft is not counted at all', async () => {
+ const d = repo();
+ const p = transcript([userPrompt('just chatting here'), assistant('a1', 'Sure.')]);
+ await runHook('stop', { session_id: 's1', transcript_path: p });
+ const s = readSession(d, 's1');
+ assert.equal(s.graftTurns, undefined);
+ assert.equal(s.reportedTurns, undefined);
+});
+
+test('a second Stop on the same reply does not count the turn twice', async () => {
+ const d = repo();
+ const p = transcript([userPrompt('how does auth work'), assistant('a1', '🌱 graft saved ~4,000 tokens this turn')]);
+ await runHook('tool-savings', { session_id: 's1', tool_response: '[graft] tokens saved ≈ 4,000' });
+ await runHook('stop', { session_id: 's1', transcript_path: p });
+ // The flag is re-set (another graft call), but the reply on disk is unchanged.
+ await runHook('tool-savings', { session_id: 's1', tool_response: '[graft] tokens saved ≈ 4,000' });
+ await runHook('stop', { session_id: 's1', transcript_path: p });
+ const s = readSession(d, 's1');
+ assert.equal(s.graftTurns, 1, 'one reply is one turn, however often Stop fires');
+ assert.equal(s.reportedTurns, 1);
+});
+
+test('a turn whose reply cannot be read is dropped from both totals', async () => {
+ const d = repo();
+ await runHook('tool-savings', { session_id: 's1', tool_response: '[graft] tokens saved ≈ 4,000' });
+ await runHook('stop', { session_id: 's1' }); // no transcript_path — a Codex-shaped Stop
+ const s = readSession(d, 's1');
+ assert.equal(s.graftTurns, undefined, 'an unobservable turn must not deflate the ratio');
+ assert.equal(s.turnUsedGraft, false, 'but the flag is still cleared, so it cannot leak forward');
+});
+
+test('a pre-existing session file with none of these fields still parses', async () => {
+ const d = repo();
+ writeSession(d, 's1', { lastQuery: null, perAgentQuery: {}, graftReads: 3, sourceReads: 1, savedTokens: 900 });
+ await runHook('tool-savings', { session_id: 's1', tool_response: '[graft] tokens saved ≈ 100' });
+ const p = transcript([userPrompt('how does auth work'), assistant('a1', 'graft saved ~1,000 tokens this turn')]);
+ await runHook('stop', { session_id: 's1', transcript_path: p });
+ const s = readSession(d, 's1');
+ assert.equal(s.savedTokens, 1000, 'the existing counter is preserved');
+ assert.equal(s.graftTurns, 1);
+});
diff --git a/test/telemetry-sessions.test.ts b/test/telemetry-sessions.test.ts
index 5cff3e3a..37f06f46 100644
--- a/test/telemetry-sessions.test.ts
+++ b/test/telemetry-sessions.test.ts
@@ -94,3 +94,27 @@ test('an empty session still counts — installed but unused is the signal we wa
const [ev] = peek(home) as { properties: Record<string, string> }[];
assert.equal(ev.properties.graft_reads_bucket, '0');
});
+
+test('the saved-vs-said turn counts ride along, bucketed', () => {
+ const { repo, home } = fixture('sess-tally');
+ writeSessionFile(repo, 's1',
+ { graftReads: 6, sourceReads: 1, savedTokens: 7400, graftTurns: 9, reportedTurns: 3 },
+ SESSION_IDLE_MS + 1000);
+ assert.equal(flushClosedSessions(repo, Date.now(), home, OPEN), 1);
+ const [ev] = peek(home) as { properties: Record<string, string> }[];
+ assert.equal(ev.properties.graft_turns_bucket, '5-19');
+ assert.equal(ev.properties.reported_turns_bucket, '1-4');
+ const values = Object.values(ev.properties);
+ for (const raw of ['9', '3']) {
+ assert.equal(values.includes(raw), false, `the raw turn count ${raw} was sent as a property value`);
+ }
+});
+
+test('a session file written before the turn counters exist reports zero, not undefined', () => {
+ const { repo, home } = fixture('sess-tally-old');
+ writeSessionFile(repo, 's1', { graftReads: 4, sourceReads: 0, savedTokens: 500 }, SESSION_IDLE_MS + 1000);
+ assert.equal(flushClosedSessions(repo, Date.now(), home, OPEN), 1);
+ const [ev] = peek(home) as { properties: Record<string, string> }[];
+ assert.equal(ev.properties.graft_turns_bucket, '0');
+ assert.equal(ev.properties.reported_turns_bucket, '0');
+});