Turbocharge Claude Code, Cursor, Codex, Gemini & every coding agent: faster, cheaper, with contextual understanding specific to your codebase.
TypeScript ai-agentsanthropicclaude-codeclicode-graphcodexcontext-engineeringcursordeveloper-toolsgeminiknowledge-graphllmmcpmcp-serveropen-sourceopenaitree-sitter
feat(telemetry): count the turns that told the user what graft saved
anirudhkumar-nanonetsdd686597d6e0
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 4bb1f8bc..6c166046 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -75,6 +75,33 @@### Added+- **Telemetry can now tell "graft saved tokens" apart from "the user was told".**+ `saved_tokens_bucket` counted what graft *computed* — every+ `[graft] tokens saved ≈ N` footer the PostToolUse accumulator swept up — so a+ turn that saved 20,000 tokens in silence and a turn that saved nothing were the+ same number. `session_summary` now also carries `graft_turns_bucket` and+ `reported_turns_bucket`: of the turns that used graft, how many closed with the+ one-line tally `SKILL.md` and `SAVINGS_TURN_NUDGE` both ask for.++ The agent's own prose lives in one place a hook can reach, so at the end of a+ turn that used graft the Stop hook reads the tail of the host transcript named+ on its stdin and checks the reply for a "graft saved ~N tokens" line. Only a+ count of turns is kept, and only as a bucket — no reply, no fragment, not even a+ length. `TELEMETRY.md` documents the local read in full.++ A turn that can't be checked — a host whose Stop hook names no transcript, an+ unreadable file, a Stop racing the transcript write — is counted in *neither*+ total, so the ratio means "of the turns we could read" rather than deflating to+ zero on every editor but Claude Code. One reply is one turn however often Stop+ fires, guarded by the id of the last reply examined.++- **`scripts/tally-audit.mjs`**, the same ratio offline and at full resolution:+ which turns were silent, whether the number the agent reported matched graft's+ own footers, and whether it named the call count. Run it when the aggregate says+ something surprising. It also flags per-turn savings estimates large enough to be+ artifacts rather than savings.++- **`graft init` converges instead of accumulating, and `graft uninstall` removesgraft entirely.** `init` wrote the files the selected agents needed and neverlooked at the rest, so a repo wired by an older version — or by the same versiondiff --git a/TELEMETRY.md b/TELEMETRY.mdindex a54fbfa4..8d00e542 100644--- a/TELEMETRY.md+++ b/TELEMETRY.md@@ -39,7 +39,7 @@ The events:| `build_completed` | `files_bucket`, `langs`, `mode` (`fast`/`deep`), `duration_bucket`, `incremental` | A build succeeds || `build_failed` | `stage`, `code` — both fixed enums | A build throws || `query` | `command`, `surface` (`cli`/`mcp`/`hook`), `hit` (`ask` only) | Any query command |-| `session_summary` | `graft_reads_bucket`, `source_reads_bucket`, `saved_tokens_bucket` | Once, after an agent session ends |+| `session_summary` | `graft_reads_bucket`, `source_reads_bucket`, `saved_tokens_bucket`, `graft_turns_bucket`, `reported_turns_bucket` | Once, after an agent session ends |Two rules govern every value above, and both are enforced in code rather than byreview:@@ -72,11 +72,32 @@ An example event, in full:Run `graft telemetry debug` to print the exact batch your machine would send. Itsends nothing.+### One thing graft reads locally++Two of those properties — `graft_turns_bucket` and `reported_turns_bucket` — need+to know something the others don't: whether the reply you actually read said what+graft saved. graft computes a saving on every retrieval call, but a turn that+saves 20,000 tokens in silence and a turn that saves nothing look identical in+`saved_tokens_bucket`, and that difference is the whole question.++The agent's own prose lives in one place a hook can reach: the transcript file+your editor writes, named on the Stop hook's stdin. So at the end of a turn that+used graft, graft reads the tail of that file, checks the reply for a "graft saved+~N tokens" line, and increments one of two counters. Nothing out of the file is+stored, and nothing out of it is sent — not the reply, not a fragment of it, not+its length. Two counts of turns cross the wire, as buckets, once per session.++A turn graft cannot check — an editor whose Stop hook names no transcript, an+unreadable file — is counted in neither total, so the ratio always means "of the+turns we could read".+## What is never sentNo source code. No file paths, repo names, organisation names, git remotes, orbranch names. No symbol names, node summaries, or anything out of the graph. No-query strings, prompts, or agent output. No error messages or stack traces — a+query strings, prompts, or agent output — graft reads the last reply of a+graft-using turn locally to decide whether it mentioned the saving (see "One+thing graft reads locally"); only the resulting count is sent, never the text. No error messages or stack traces — afailure contributes a code from a fixed list and nothing else. No environmentvariables, API keys, model names, hostnames, usernames, or email addresses.diff --git a/scripts/tally-audit.mjs b/scripts/tally-audit.mjsnew file mode 100644index 00000000..9891167a--- /dev/null+++ b/scripts/tally-audit.mjs@@ -0,0 +1,140 @@+/**+ * "Saved vs said": how often does the agent actually TELL the user what graft saved?+ *+ * node scripts/tally-audit.mjs [transcript-dir] [--json]+ *+ * `savedTokens` (src/claude/state.ts) counts what graft *computed* — every+ * `[graft] tokens saved ≈ N` footer the PostToolUse accumulator swept up. This+ * script measures the other half, offline and in full detail: of the turns that+ * used graft, how many closed with the one-line tally SKILL.md asks for, was the+ * number right, and did it name the call count.+ *+ * The shipped metric (`graft_turns_bucket` / `reported_turns_bucket` on+ * `session_summary`) is the same ratio at bucket resolution, which is all that+ * can cross the wire. Run this locally when the aggregate says something+ * surprising and you need to see which turns and why.+ *+ * Reads Claude Code's own transcript JSONL, which defaults to+ * ~/.claude/projects/<slugified-cwd>/. Nothing leaves the machine.+ */+import { readFileSync, readdirSync, existsSync } from "node:fs";+import { join } from "node:path";+import { homedir } from "node:os";++const SAVED_FOOTER = /\[graft\] tokens saved ≈ ([\d,]+)/g;+const TALLY = /graft\s+saved\s*[~≈]?\s*([\d,.]+)\s*(k|m)?\s*(?:tok|tokens)/i;+const CALLS = /(\d+)\s*calls?/i;+/** How far past the tally to look for its call count. The tally is one line+ * ("🌱 graft saved ~12,400 tokens this turn (3 calls)"); scanning the rest of+ * the reply instead would match any unrelated "3 calls" further down and report+ * near-perfect compliance for a field the agent mostly omits. */+const TALLY_WINDOW = 60;+/** Above this, a per-turn "saved" figure is an estimator artifact rather than a+ * saving — a `--depth all` closure or a vendored tree summing whole-file+ * baselines nobody would have read. Reported separately, never averaged in. */+const IMPLAUSIBLE_TOKENS = 1_000_000;++const args = process.argv.slice(2).filter((a) => a !== "--json");+const asJson = process.argv.includes("--json");+const dir = args[0] ?? defaultTranscriptDir();++function defaultTranscriptDir() {+ const slug = process.cwd().replace(/[/.]/g, "-");+ return join(homedir(), ".claude", "projects", slug);+}+function textOf(content) {+ if (typeof content === "string") return content;+ if (!Array.isArray(content)) return "";+ return content.map((p) => (p?.type === "text" ? p.text ?? "" : "")).join("\n");+}+function isUserPrompt(o) {+ if (o.type !== "user" || o.isSidechain || o.isMeta) return false;+ const c = o.message?.content;+ if (typeof c === "string") return true;+ return Array.isArray(c) && !c.some((p) => p?.type === "tool_result");+}+function median(ns) {+ if (!ns.length) return 0;+ const s = [...ns].sort((a, b) => a - b);+ const mid = s.length >> 1;+ return s.length % 2 ? s[mid] : Math.round((s[mid - 1] + s[mid]) / 2);+}++if (!existsSync(dir)) {+ console.error(`no transcripts at ${dir}`);+ process.exit(1);+}++const turns = [];+for (const file of readdirSync(dir).filter((f) => f.endsWith(".jsonl"))) {+ let turn = null;+ const flush = () => { if (turn?.calls > 0) turns.push(turn); };+ for (const line of readFileSync(join(dir, file), "utf8").split("\n")) {+ if (!line.trim()) continue;+ let o;+ try { o = JSON.parse(line); } catch { continue; }+ if (o.isSidechain) continue; // a subagent's prose is not what the user read+ if (isUserPrompt(o)) { flush(); turn = { file, calls: 0, saved: 0, said: null, statedCalls: null }; continue; }+ if (!turn) continue;+ if (o.type === "user") { // a tool_result wearing the user role+ for (const m of JSON.stringify(o.message?.content ?? "").matchAll(SAVED_FOOTER)) {+ turn.calls++;+ turn.saved += Number(m[1].replace(/,/g, "")) || 0;+ }+ }+ if (o.type === "assistant") {+ const m = TALLY.exec(textOf(o.message?.content));+ if (m) {+ let n = Number(m[1].replace(/,/g, ""));+ if (m[2]?.toLowerCase() === "k") n *= 1e3;+ if (m[2]?.toLowerCase() === "m") n *= 1e6;+ turn.said = n;+ const c = CALLS.exec(textOf(o.message?.content).slice(m.index, m.index + TALLY_WINDOW));+ turn.statedCalls = c ? Number(c[1]) : null;+ }+ }+ }+ flush();+}++const said = turns.filter((t) => t.said !== null);+const plausible = turns.filter((t) => t.saved < IMPLAUSIBLE_TOKENS);+const outliers = turns.filter((t) => t.saved >= IMPLAUSIBLE_TOKENS);+const fidelity = { exact: 0, over: 0, under: 0 };+for (const t of said.filter((t) => t.saved > 0)) {+ const r = t.said / t.saved;+ if (r >= 0.9 && r <= 1.1) fidelity.exact++;+ else if (r > 1.1) fidelity.over++;+ else fidelity.under++;+}++const report = {+ transcripts: readdirSync(dir).filter((f) => f.endsWith(".jsonl")).length,+ graft_turns: turns.length,+ reported: said.length,+ silent: turns.length - said.length,+ reported_pct: turns.length ? Number(((said.length / turns.length) * 100).toFixed(1)) : 0,+ stated_call_count: said.filter((t) => t.statedCalls !== null).length,+ median_saved_per_turn: median(plausible.map((t) => t.saved)),+ fidelity,+ implausible_turns: outliers.length,+ implausible_max: outliers.length ? Math.max(...outliers.map((t) => t.saved)) : 0,+};++if (asJson) {+ console.log(JSON.stringify(report, null, 2));+} else {+ const { graft_turns: n } = report;+ console.log(`transcripts scanned: ${report.transcripts} (${dir})`);+ console.log(`turns that used graft: ${n}`);+ console.log(` ...reported a tally: ${report.reported} (${report.reported_pct}%)`);+ console.log(` ...silent: ${report.silent}`);+ console.log(` ...named the call count ${report.stated_call_count}/${report.reported}`);+ console.log(`median saved per turn: ${report.median_saved_per_turn.toLocaleString()} tok`);+ console.log(`reported number vs footer sum (±10%): exact ${fidelity.exact} · over ${fidelity.over} · under ${fidelity.under}`);+ if (outliers.length) {+ console.log(`\n⚠ ${outliers.length} turn(s) claim ≥${IMPLAUSIBLE_TOKENS.toLocaleString()} tokens saved ` ++ `(max ${report.implausible_max.toLocaleString()}) — an estimator artifact, excluded from the median. ` ++ `See savingsFor() in src/context/savings.ts.`);+ }+}diff --git a/src/claude/hooks.ts b/src/claude/hooks.tsindex 3f055337..52854d32 100644--- a/src/claude/hooks.ts+++ b/src/claude/hooks.ts@@ -10,6 +10,7 @@ import { graftCliPath, claudeScriptPath } from './paths.js';import { runUpkeep } from '../upkeep-run.js';import { runningVersion } from '../upkeep.js';import { flushClosedSessions } from '../telemetry/sessions.js';+import { hasSavingsTally, lastAssistantTurn } from './tally.js';import { scopeOf, scopesOfGraph } from '../graph/scopes.js';/** Prompts shorter than this never trigger retrieval — they are almost always@@ -244,10 +245,47 @@ function handleToolSavings(input: any, dir: string): void {const id = input?.session_id || 'default';const s = readSession(dir, id);s.savedTokens = (s.savedTokens ?? 0) + total;+ // This turn used graft, so it is owed a tally in the reply. countTallyTurn()+ // at Stop resolves whether it got one; a flag rather than a count because a+ // turn with four graft calls is still one reply to the user.+ s.turnUsedGraft = true;writeSession(dir, id, s);}-function handleStop(dir: string): void {+/**+ * At turn end: did the reply the user just read say what graft saved?+ *+ * Runs only on turns the tool-savings hook flagged, so a conversational turn+ * costs nothing. A turn we cannot observe — a host whose Stop hook names no+ * transcript, an unreadable file, or a Stop that fires before the final prose+ * is on disk — is counted in NEITHER total: the ratio these two numbers form+ * has to mean "of the turns we could check", not "of the turns we tried to".+ */+function countTallyTurn(input: any, dir: string): void {+ try {+ const id = input?.session_id || 'default';+ const s = readSession(dir, id);+ if (!s.turnUsedGraft) return;+ const turn = lastAssistantTurn(input?.transcript_path);+ // Same reply as last time we looked: no new prose has landed, so this Stop+ // is a duplicate or a race with the transcript write. Drop the turn rather+ // than judge it on a stale message.+ if (!turn || turn.uuid === s.lastTallyUuid) {+ writeSession(dir, id, { ...s, turnUsedGraft: false });+ return;+ }+ s.graftTurns = (s.graftTurns ?? 0) + 1;+ if (hasSavingsTally(turn.text)) s.reportedTurns = (s.reportedTurns ?? 0) + 1;+ s.turnUsedGraft = false;+ s.lastTallyUuid = turn.uuid;+ writeSession(dir, id, s);+ } catch {+ // A turn-end metric is never worth failing the graph sync over.+ }+}++function handleStop(input: any, dir: string): void {+ countTallyTurn(input, dir);// sync-run.js ships next to this module inside the package, so it resolves in// any repo that installs graft (not just graft's own). Defensive existsSync:// if the package is somehow incomplete, skip rather than wedge on syncing:true.@@ -292,9 +330,9 @@ export async function main(event: string): Promise<void> {if (event === 'tool-savings') { handleToolSavings(input, dir); return; }- if (event === 'stop') { handleStop(dir); return; }+ if (event === 'stop') { handleStop(input, dir); return; }- if (event === 'post-edit-sync') { await handlePostEdit(input, dir); handleStop(dir); return; }+ if (event === 'post-edit-sync') { await handlePostEdit(input, dir); handleStop(input, dir); return; }if (event === 'prompt') {const prompt = String(input?.prompt ?? '').trim();diff --git a/src/claude/state.ts b/src/claude/state.tsindex 45d9f252..8a142546 100644--- a/src/claude/state.ts+++ b/src/claude/state.ts@@ -36,6 +36,19 @@ export interface SessionState {/** Weak-match nudges spent this session, capped so the line stays signal.* Optional for the same backwards-compatibility reason as above. */nudges?: number;+ /** Turns this session in which the agent used a graft retrieval tool — the+ * denominator for "did it tell the user what that saved". Counted at Stop,+ * and only for turns whose reply we could actually read (see claude/tally.ts). */+ graftTurns?: number;+ /** Of those turns, the ones whose reply carried a "graft saved ~N tokens"+ * tally. `graftTurns - reportedTurns` is silent value: saved, never said. */+ reportedTurns?: number;+ /** Set by the tool-savings hook when the current turn touched graft, cleared+ * at Stop once the turn has been counted. Transient, not a total. */+ turnUsedGraft?: boolean;+ /** `uuid` of the last assistant reply already examined, so a Stop that fires+ * without new prose (or fires twice) can't count one reply as two turns. */+ lastTallyUuid?: string;/** Set once this session has been rolled up into a `session_summary`* telemetry event, so a resumed or long-lived session is counted once.* A flag rather than deleting the file: the file still holds `lastQuery` anddiff --git a/src/claude/tally.ts b/src/claude/tally.tsnew file mode 100644index 00000000..1ec1a324--- /dev/null+++ b/src/claude/tally.ts@@ -0,0 +1,131 @@+/**+ * Did the agent actually TELL the user what graft saved?+ *+ * `savedTokens` (state.ts) counts what graft *computed* — every+ * `[graft] tokens saved ≈ N` footer the PostToolUse accumulator swept up. That+ * number is real whether or not anyone sees it. This module measures the other+ * half: whether the reply the user read closed with the one-line tally that+ * SKILL.md and `SAVINGS_TURN_NUDGE` (context/savings.ts) both ask for. A turn+ * that saved 20k tokens in silence is a turn where the product did its job and+ * got no credit for it, and the two are indistinguishable in the numbers we+ * ship today.+ *+ * The assistant's own prose exists in exactly one place a hook can reach: the+ * host's transcript JSONL, named on the Stop hook's stdin as `transcript_path`.+ * So this reads that file — locally, at turn end, for a boolean. Nothing out of+ * it is stored or sent; the session keeps a count and the telemetry contract+ * carries that count as a bucket. See TELEMETRY.md.+ *+ * Only the tail is read. A transcript grows for the life of a session and can+ * reach many megabytes, while the answer is always in its last few entries — a+ * hook that read the whole file would get slower every turn for no extra signal.+ */+import { closeSync, fstatSync, openSync, readSync } from 'node:fs';++/**+ * How much of the transcript's end to read. Comfortably larger than any single+ * turn's worth of entries, small enough that the read cost is flat as the+ * session grows.+ */+const TAIL_BYTES = 1024 * 1024;++/**+ * The tally, as the agent is actually asked to write it: a "graft saved ~N+ * tokens" claim in prose. Deliberately loose about the decoration around it —+ * the 🌱, the "this turn", the "(3 calls)" suffix and the thousands separator+ * are all optional in practice, and a metric that only counted the one exact+ * phrasing from the example would measure the agent's formatting rather than+ * whether the user was told. Anchored on "graft saved" + a number + "tokens" so+ * ordinary prose about graft can't trip it.+ */+const TALLY = /graft\s+saved\s*[~≈]?\s*[\d,.]+\s*[km]?\s*(?:tok|tokens)/i;++/** Did this reply tell the user what graft saved? */+export function hasSavingsTally(text: string): boolean {+ return TALLY.test(text);+}++/** One turn's worth of assistant prose, plus the id of the entry it ended on. */+export interface AssistantTurn {+ /** `uuid` of the last assistant entry, so the same reply is never counted twice. */+ uuid: string;+ /** Every text block the agent emitted since the last user prompt, joined. */+ text: string;+}++/** Read the last {@link TAIL_BYTES} of a file as utf8, dropping the leading+ * partial line a byte-offset read leaves behind. Returns '' on any I/O trouble —+ * a hook never fails over a metric. */+function readTail(path: string): string {+ let fd: number | undefined;+ try {+ fd = openSync(path, 'r');+ const size = fstatSync(fd).size;+ const len = Math.min(size, TAIL_BYTES);+ const buf = Buffer.allocUnsafe(len);+ readSync(fd, buf, 0, len, size - len);+ const raw = buf.toString('utf8');+ return len < size ? raw.slice(raw.indexOf('\n') + 1) : raw;+ } catch {+ return '';+ } finally {+ if (fd !== undefined) { try { closeSync(fd); } catch { /* already gone */ } }+ }+}++/** Is this entry a user's own prompt, rather than a tool result wearing the+ * `user` role? Tool results carry a `tool_result` part; a real prompt does not.+ * The boundary matters because the tally can land in any text block of the turn,+ * not only the last one. */+function isUserPrompt(entry: any): boolean {+ if (entry?.type !== 'user' || entry?.isMeta) return false;+ const content = entry?.message?.content;+ if (typeof content === 'string') return true;+ return Array.isArray(content) && !content.some((p: any) => p?.type === 'tool_result');+}++function textOf(content: unknown): string {+ if (typeof content === 'string') return content;+ if (!Array.isArray(content)) return '';+ return content.map((p: any) => (p?.type === 'text' ? String(p.text ?? '') : '')).join('\n');+}++/**+ * The assistant prose of the turn the transcript ends on, or null when it can't+ * be read (no path — a host whose Stop hook doesn't name one — unreadable file,+ * or a tail with no assistant text in it).+ *+ * Null is meaningfully different from "no tally found": a turn we cannot observe+ * is left out of BOTH the numerator and the denominator by the caller, so a host+ * that exposes no transcript reports no ratio rather than a ratio of zero.+ *+ * Sidechain entries (subagents) are skipped throughout — a subagent's own prose+ * is never what the user read.+ */+export function lastAssistantTurn(transcriptPath: unknown): AssistantTurn | null {+ if (typeof transcriptPath !== 'string' || !transcriptPath) return null;+ const tail = readTail(transcriptPath);+ if (!tail) return null;++ const entries: any[] = [];+ for (const line of tail.split('\n')) {+ if (!line.trim()) continue;+ try { entries.push(JSON.parse(line)); } catch { /* clipped or partial line */ }+ }++ // Walk back to the turn boundary, collecting assistant text on the way.+ const parts: string[] = [];+ let uuid: string | null = null;+ for (let i = entries.length - 1; i >= 0; i--) {+ const e = entries[i];+ if (e?.isSidechain) continue;+ if (isUserPrompt(e)) break;+ if (e?.type !== 'assistant') continue;+ const text = textOf(e?.message?.content);+ if (!text.trim()) continue;+ if (uuid === null && typeof e?.uuid === 'string') uuid = e.uuid;+ parts.unshift(text);+ }+ if (uuid === null || parts.length === 0) return null;+ return { uuid, text: parts.join('\n') };+}diff --git a/src/telemetry/contract.ts b/src/telemetry/contract.tsindex cb380232..f62884d3 100644--- a/src/telemetry/contract.ts+++ b/src/telemetry/contract.ts@@ -43,8 +43,13 @@ export const EVENTS: Record<string, ReadonlySet<string>> = {/** One query, from any surface. The DAU backbone and the dead-command detector. */query: new Set<string>(['command', 'surface', 'hit']),/** One closed agent session, summarised. `graft_reads` vs `source_reads` is- * the single number that says whether an agent prefers graft to grep. */- session_summary: new Set<string>(['graft_reads_bucket', 'source_reads_bucket', 'saved_tokens_bucket']),+ * the single number that says whether an agent prefers graft to grep; the two+ * `*_turns` buckets are the follow-up question — of the turns that used graft,+ * how many told the user what it saved. Both are counts of turns, never text. */+ session_summary: new Set<string>([+ 'graft_reads_bucket', 'source_reads_bucket', 'saved_tokens_bucket',+ 'graft_turns_bucket', 'reported_turns_bucket',+ ]),};/** Stamped on every event by `track()`; not listed per event above. */diff --git a/src/telemetry/sessions.ts b/src/telemetry/sessions.tsindex 12dca218..126ba6a0 100644--- a/src/telemetry/sessions.ts+++ b/src/telemetry/sessions.ts@@ -67,6 +67,12 @@ export function flushClosedSessions(graft_reads_bucket: countBucket(s.graftReads ?? 0),source_reads_bucket: countBucket(s.sourceReads ?? 0),saved_tokens_bucket: savedTokensBucket(s.savedTokens ?? 0),+ // Saved vs *said*: the gap between these two is value the user never+ // heard about. Only turns whose reply was actually readable are in+ // either count (claude/tally.ts), so a host that exposes no transcript+ // reports 0/0 rather than a misleading 0-out-of-many.+ graft_turns_bucket: countBucket(s.graftTurns ?? 0),+ reported_turns_bucket: countBucket(s.reportedTurns ?? 0),},{ repo, host: 'claude-code', home, env },);diff --git a/test/claude-tally.test.ts b/test/claude-tally.test.tsnew file mode 100644index 00000000..64ed0819--- /dev/null+++ b/test/claude-tally.test.ts@@ -0,0 +1,168 @@+/**+ * "Saved" vs "said": the turn-end check that measures whether the agent actually+ * told the user what graft saved. Its rules: read only the last turn, never+ * count one reply twice, and count NEITHER total for a turn we cannot observe.+ */+import { test } from 'node:test';+import assert from 'node:assert/strict';+import { mkdtempSync, mkdirSync, writeFileSync } from 'node:fs';+import { tmpdir } from 'node:os';+import { join } from 'node:path';+import { hasSavingsTally, lastAssistantTurn } from '../src/claude/tally.js';+import { main } from '../src/claude/hooks.js';+import { readSession, writeSession } from '../src/claude/state.js';++test('hasSavingsTally accepts the phrasings agents actually write', () => {+ for (const yes of [+ '🌱 graft saved ~12,400 tokens this turn (3 calls)',+ 'graft saved ~12k tokens this turn',+ 'Graft saved 8,100 tokens across 2 calls.',+ 'graft saved ≈ 900 tok this turn',+ ]) assert.equal(hasSavingsTally(yes), true, yes);+});++test('hasSavingsTally is not tripped by ordinary prose about graft', () => {+ for (const no of [+ 'I used graft to find the call sites.',+ 'graft saved me a lot of time here', // no number, no unit+ 'The 3 calls to graft returned 8 hits each.',+ '',+ ]) assert.equal(hasSavingsTally(no), false, no);+});++/** A transcript line as Claude Code writes it. */+function assistant(uuid: string, text: string): string {+ return JSON.stringify({ type: 'assistant', uuid, message: { role: 'assistant', content: [{ type: 'text', text }] } });+}+function userPrompt(text: string): string {+ return JSON.stringify({ type: 'user', uuid: `u-${text}`, message: { role: 'user', content: text } });+}+function toolResult(text: string): string {+ return JSON.stringify({ type: 'user', message: { role: 'user', content: [{ type: 'tool_result', content: text }] } });+}+function transcript(lines: string[]): string {+ const p = join(mkdtempSync(join(tmpdir(), 'graft-tally-')), 'session.jsonl');+ writeFileSync(p, lines.join('\n') + '\n');+ return p;+}++test('the whole last turn is read, not just its final text block', () => {+ const p = transcript([+ userPrompt('older question'),+ assistant('a1', 'no tally in this earlier turn'),+ userPrompt('how does auth work'),+ assistant('a2', '🌱 graft saved ~5,000 tokens this turn'),+ toolResult('[graft] tokens saved ≈ 1,000'),+ assistant('a3', 'Done.'),+ ]);+ const turn = lastAssistantTurn(p)!;+ assert.equal(turn.uuid, 'a3', 'uuid is the last assistant entry of the turn');+ assert.equal(hasSavingsTally(turn.text), true, 'a tally in an earlier block of the same turn still counts');+});++test('the previous turn is not read into this one', () => {+ const p = transcript([+ userPrompt('first'),+ assistant('a1', '🌱 graft saved ~9,000 tokens this turn'),+ userPrompt('second'),+ assistant('a2', 'Nothing to report.'),+ ]);+ const turn = lastAssistantTurn(p)!;+ assert.equal(hasSavingsTally(turn.text), false);+});++test('a subagent’s prose is never what the user read', () => {+ const p = transcript([+ userPrompt('go'),+ JSON.stringify({ type: 'assistant', uuid: 's1', isSidechain: true,+ message: { role: 'assistant', content: [{ type: 'text', text: '🌱 graft saved ~4,000 tokens this turn' }] } }),+ assistant('a1', 'The subagent finished.'),+ ]);+ assert.equal(hasSavingsTally(lastAssistantTurn(p)!.text), false);+});++test('an unobservable turn yields null rather than a false negative', () => {+ assert.equal(lastAssistantTurn(undefined), null, 'no transcript_path (e.g. Codex)');+ assert.equal(lastAssistantTurn('/nope/missing.jsonl'), null, 'unreadable file');+ assert.equal(lastAssistantTurn(transcript([userPrompt('go')])), null, 'no assistant prose yet');+});++/* ---------------------------------------------------------------------- */+/* the hook path: tool-savings flags the turn, stop resolves it */+/* ---------------------------------------------------------------------- */++async function runHook(event: string, stdin: object): Promise<void> {+ process.env.GRAFT_TEST_STDIN = JSON.stringify(stdin);+ try { await main(event); } finally { delete process.env.GRAFT_TEST_STDIN; }+}++function repo(): string {+ const d = mkdtempSync(join(tmpdir(), 'graft-tally-repo-'));+ mkdirSync(join(d, 'graft', '.cache', 'session'), { recursive: true });+ process.env.CLAUDE_PROJECT_DIR = d;+ return d;+}++test('a graft turn that reports the tally counts in both totals', async () => {+ const d = repo();+ await runHook('tool-savings', { session_id: 's1', tool_response: '[graft] tokens saved ≈ 4,000' });+ assert.equal(readSession(d, 's1').turnUsedGraft, true, 'the turn is flagged as owing a tally');+ const p = transcript([userPrompt('how does auth work'), assistant('a1', '🌱 graft saved ~4,000 tokens this turn (1 call)')]);+ await runHook('stop', { session_id: 's1', transcript_path: p });+ const s = readSession(d, 's1');+ assert.equal(s.graftTurns, 1);+ assert.equal(s.reportedTurns, 1);+ assert.equal(s.turnUsedGraft, false, 'the flag is cleared so the next turn starts clean');+});++test('a silent graft turn counts in the denominator only', async () => {+ const d = repo();+ await runHook('tool-savings', { session_id: 's1', tool_response: '[graft] tokens saved ≈ 20,000' });+ const p = transcript([userPrompt('how does auth work'), assistant('a1', 'Auth is handled in src/auth.ts.')]);+ await runHook('stop', { session_id: 's1', transcript_path: p });+ const s = readSession(d, 's1');+ assert.equal(s.graftTurns, 1);+ assert.equal(s.reportedTurns ?? 0, 0, 'saved 20k, said nothing — that is the gap we ship');+});++test('a turn that never touched graft is not counted at all', async () => {+ const d = repo();+ const p = transcript([userPrompt('just chatting here'), assistant('a1', 'Sure.')]);+ await runHook('stop', { session_id: 's1', transcript_path: p });+ const s = readSession(d, 's1');+ assert.equal(s.graftTurns, undefined);+ assert.equal(s.reportedTurns, undefined);+});++test('a second Stop on the same reply does not count the turn twice', async () => {+ const d = repo();+ const p = transcript([userPrompt('how does auth work'), assistant('a1', '🌱 graft saved ~4,000 tokens this turn')]);+ await runHook('tool-savings', { session_id: 's1', tool_response: '[graft] tokens saved ≈ 4,000' });+ await runHook('stop', { session_id: 's1', transcript_path: p });+ // The flag is re-set (another graft call), but the reply on disk is unchanged.+ await runHook('tool-savings', { session_id: 's1', tool_response: '[graft] tokens saved ≈ 4,000' });+ await runHook('stop', { session_id: 's1', transcript_path: p });+ const s = readSession(d, 's1');+ assert.equal(s.graftTurns, 1, 'one reply is one turn, however often Stop fires');+ assert.equal(s.reportedTurns, 1);+});++test('a turn whose reply cannot be read is dropped from both totals', async () => {+ const d = repo();+ await runHook('tool-savings', { session_id: 's1', tool_response: '[graft] tokens saved ≈ 4,000' });+ await runHook('stop', { session_id: 's1' }); // no transcript_path — a Codex-shaped Stop+ const s = readSession(d, 's1');+ assert.equal(s.graftTurns, undefined, 'an unobservable turn must not deflate the ratio');+ assert.equal(s.turnUsedGraft, false, 'but the flag is still cleared, so it cannot leak forward');+});++test('a pre-existing session file with none of these fields still parses', async () => {+ const d = repo();+ writeSession(d, 's1', { lastQuery: null, perAgentQuery: {}, graftReads: 3, sourceReads: 1, savedTokens: 900 });+ await runHook('tool-savings', { session_id: 's1', tool_response: '[graft] tokens saved ≈ 100' });+ const p = transcript([userPrompt('how does auth work'), assistant('a1', 'graft saved ~1,000 tokens this turn')]);+ await runHook('stop', { session_id: 's1', transcript_path: p });+ const s = readSession(d, 's1');+ assert.equal(s.savedTokens, 1000, 'the existing counter is preserved');+ assert.equal(s.graftTurns, 1);+});diff --git a/test/telemetry-sessions.test.ts b/test/telemetry-sessions.test.tsindex 5cff3e3a..37f06f46 100644--- a/test/telemetry-sessions.test.ts+++ b/test/telemetry-sessions.test.ts@@ -94,3 +94,27 @@ test('an empty session still counts — installed but unused is the signal we waconst [ev] = peek(home) as { properties: Record<string, string> }[];assert.equal(ev.properties.graft_reads_bucket, '0');});++test('the saved-vs-said turn counts ride along, bucketed', () => {+ const { repo, home } = fixture('sess-tally');+ writeSessionFile(repo, 's1',+ { graftReads: 6, sourceReads: 1, savedTokens: 7400, graftTurns: 9, reportedTurns: 3 },+ SESSION_IDLE_MS + 1000);+ assert.equal(flushClosedSessions(repo, Date.now(), home, OPEN), 1);+ const [ev] = peek(home) as { properties: Record<string, string> }[];+ assert.equal(ev.properties.graft_turns_bucket, '5-19');+ assert.equal(ev.properties.reported_turns_bucket, '1-4');+ const values = Object.values(ev.properties);+ for (const raw of ['9', '3']) {+ assert.equal(values.includes(raw), false, `the raw turn count ${raw} was sent as a property value`);+ }+});++test('a session file written before the turn counters exist reports zero, not undefined', () => {+ const { repo, home } = fixture('sess-tally-old');+ writeSessionFile(repo, 's1', { graftReads: 4, sourceReads: 0, savedTokens: 500 }, SESSION_IDLE_MS + 1000);+ assert.equal(flushClosedSessions(repo, Date.now(), home, OPEN), 1);+ const [ev] = peek(home) as { properties: Record<string, string> }[];+ assert.equal(ev.properties.graft_turns_bucket, '0');+ assert.equal(ev.properties.reported_turns_bucket, '0');+});