Open source Grok Bot alternative, Bring your own agent and subscription: message Claude Code, Codex, Gemini CLI, Cursor, Copilot and 40+ AI agents as bots on your own computer, from iPhone, Mac, Windows, Linux or terminal. Agent-agnostic and free. Also a Muse and Dots alternative. Built in Rust.
Rust acpagent-client-protocolai-agentsai-assistantclaudeclaude-codecodexcoding-agentscursorgeminigemini-cligrok-bot-alternativeiosiphonelinuxmacosmcpopen-sourcerustself-hosted
docs: measure Codex bots' prompt cache too
leepokaide4b70d3491c
diff --git a/docs/design/bot-memory.md b/docs/design/bot-memory.mdindex 5f92d3dc..87d57143 100644--- a/docs/design/bot-memory.md+++ b/docs/design/bot-memory.md@@ -775,12 +775,16 @@ for the owner's decision.- decisions.**Prompt cache.** The host logs each turn's cache reads and writes (`turn usage` in `host.log`).-`tools/memory-eval/cache.py` measures the hit rate from Claude Code's own session logs; on 2026-10-18:+`tools/memory-eval/cache.py` measures the hit rate from Claude Code's and Codex's own session logs; on 2026-10-18:| | Conversations | Calls | In a conversation | Every call ||---|---|---|---|---|-| The owner's bots (`~/.codync`, `~/.codync-dev`) | 27 | 229 | **97.6%** | 83.7% |-| The evaluation run | 173 | 472 | 98.1% | 71.1% |+| The owner's Claude bots (`~/.codync`, `~/.codync-dev`) | 27 | 229 | **97.6%** | 83.7% |+| The owner's Codex bots | 7 | 24 | 96.7% | 81.4% |+| The evaluation run (Claude) | 173 | 472 | 98.1% | 71.1% |++The owner's Cursor, oh-my-pi and custom bots aren't measured: the script reads only those two logs. Once the host+from this branch runs, its `turn usage` lines cover every harness whose adapter reports cached tokens.- **In a conversation** is the share of input read from cache, leaving out each conversation's first call andcalls more than an hour after the previous one. Per call it is 97.2%, and the median conversation 99.2%.@@ -792,7 +796,7 @@ for the owner's decision.cache-safe opening. The evaluation had none.- **Cost:** Claude Code writes every cache entry for an hour, at 2× input. Overall, input cost 36.9% of the samecalls uncached. A one-call conversation costs more than no cache would (157%): it writes the cache and never- reads it back.+ reads it back. Codex (OpenAI) caches on its own and charges nothing to write.**Product metrics** (PostHog counts only, never content):diff --git a/tools/memory-eval/README.md b/tools/memory-eval/README.mdindex 9c9a73c0..57d24185 100644--- a/tools/memory-eval/README.md+++ b/tools/memory-eval/README.md@@ -23,12 +23,13 @@ the day of the run.## Prompt cache-`cache.py` reads Claude Code's session logs (`~/.claude/projects`) and prints how much of each conversation's input-came from the prompt cache, why the cold calls were cold, and the input cost against no cache. It needs no host:+`cache.py` reads Claude Code's and Codex's session logs (`~/.claude/projects`, `~/.codex/sessions`) and prints,+per harness, how much of each conversation's input came from the prompt cache and why the cold calls were cold;+for Claude also the input cost against no cache. It needs no host:```shtools/memory-eval/cache.py # the bots of ~/.codync and ~/.codync-dev-tools/memory-eval/cache.py eval-home-bots # session folders whose name contains this+tools/memory-eval/cache.py eval-home-bots # workspaces whose path contains this ("/" and "." read as "-")```The latest numbers are under [prompt cache](../../docs/design/bot-memory.md#evaluation).diff --git a/tools/memory-eval/cache.py b/tools/memory-eval/cache.pyindex 4f94ad34..0527b2d6 100755--- a/tools/memory-eval/cache.py+++ b/tools/memory-eval/cache.py@@ -1,100 +1,142 @@#!/usr/bin/env python3-"""How well Claude bots' prompt cache holds, read from Claude Code's own session logs.+"""How well bots' prompt cache holds, read from the harnesses' own session logs.-Each session file in ~/.claude/projects is one conversation; each API call in it is counted once.-Usage: tools/memory-eval/cache.py [folder-name-part ...] (default: the bots of ~/.codync and ~/.codync-dev)+Claude Code keeps them in ~/.claude/projects, Codex in ~/.codex/sessions. Each session file is one conversation;+each API call in it is counted once. Other harnesses aren't read; the host's `turn usage` log covers any harness+whose adapter reports cached tokens.+Usage: tools/memory-eval/cache.py [workspace-part ...]+A part matches the workspace path with every non-alphanumeric character as "-", the way Claude Code names its+folders (default: the bots of ~/.codync and ~/.codync-dev)."""import globimport jsonimport os+import reimport sysfrom collections import Counterfrom datetime import datetimefrom statistics import mean, median-ROOT = os.path.expanduser('~/.claude/projects')-LIFE = 3600 # Claude Code writes 1-hour cache entries.+LIFE = 3600 # longest a cache entry lives: Claude Code writes 1-hour entries+PARTS = sys.argv[1:] or ['-codync-bots-', '-codync-dev-bots-']-def calls(path, seen):- """(read, total input, 1h writes, writes, model, seconds since the previous call) per API call."""- prev = None- for line in open(path, errors='replace'):- try:- rec = json.loads(line)- except ValueError:- continue- msg = rec.get('message') if isinstance(rec, dict) else None- if not isinstance(msg, dict) or msg.get('role') != 'assistant' or 'usage' not in msg:- continue- key = msg.get('id') or rec.get('uuid')- if key in seen: # a forked session repeats its parent's calls- continue- seen.add(key)- u = msg['usage']- read = u.get('cache_read_input_tokens') or 0- write = u.get('cache_creation_input_tokens') or 0- total = read + write + (u.get('input_tokens') or 0)- if total == 0:- continue- ts = datetime.fromisoformat(rec['timestamp'].replace('Z', '+00:00'))- gap = (ts - prev).total_seconds() if prev else None- prev = ts- w1h = (u.get('cache_creation') or {}).get('ephemeral_1h_input_tokens') or 0- yield read, total, w1h, write, msg.get('model', ''), gap+def wanted(path):+ return any(p in re.sub(r'[^A-Za-z0-9]', '-', path) for p in PARTS)-def cost(read, total, w1h, write, model):- """Input cost at API list multipliers: read 0.1x (0.05x on the 5.5 models), 5-minute write 1.25x, 1-hour 2x."""- return total - read - write + 1.25 * (write - w1h) + 2 * w1h + (0.05 if '-5-5' in model else 0.1) * read+def gaps(stamps):+ prev = None+ for ts in stamps:+ t = datetime.fromisoformat(ts.replace('Z', '+00:00'))+ yield (t - prev).total_seconds() if prev else None+ prev = t-def main():- parts = sys.argv[1:] or ['-codync-bots-', '-codync-dev-bots-']- sessions, causes, sums = [], Counter(), Counter()- for folder in glob.glob(os.path.join(ROOT, '*')):- if not any(p in os.path.basename(folder) for p in parts):+def claude():+ """Per conversation: (read, total input, 1h writes, writes, model, seconds since the previous call)."""+ sessions = []+ for folder in glob.glob(os.path.expanduser('~/.claude/projects/*')):+ if not wanted(os.path.basename(folder)):continueseen = set()for path in glob.glob(os.path.join(folder, '*.jsonl')):- rows = list(calls(path, seen))+ raw = []+ for line in open(path, errors='replace'):+ try:+ rec = json.loads(line)+ except ValueError:+ continue+ msg = rec.get('message') if isinstance(rec, dict) else None+ if not isinstance(msg, dict) or msg.get('role') != 'assistant' or 'usage' not in msg:+ continue+ key = msg.get('id') or rec.get('uuid')+ if key in seen: # a forked session repeats its parent's calls+ continue+ seen.add(key)+ u = msg['usage']+ read = u.get('cache_read_input_tokens') or 0+ write = u.get('cache_creation_input_tokens') or 0+ total = read + write + (u.get('input_tokens') or 0)+ w1h = (u.get('cache_creation') or {}).get('ephemeral_1h_input_tokens') or 0+ if total:+ raw.append((read, total, w1h, write, msg.get('model', ''), rec['timestamp']))+ rows = [r[:5] + (g,) for r, g in zip(raw, gaps(r[5] for r in raw))]if rows:sessions.append(rows)- if not sessions:- sys.exit('no sessions found')- every, warm, per_every, per_warm = [], [], [], []- for rows in sessions:- hot = [r for r in rows if r[5] is not None and r[5] <= LIFE]- every += rows- warm += hot- per_every.append(sum(r[0] for r in rows) / sum(r[1] for r in rows))- if hot:- per_warm.append(sum(r[0] for r in hot) / sum(r[1] for r in hot))- for read, total, w1h, write, model, gap in rows:- sums['w1h'] += w1h- sums['write'] += write- sums['cost'] += cost(read, total, w1h, write, model)- if read >= total / 2:+ return sessions+++def codex():+ """The same rows from Codex rollouts. OpenAI caches on its own and charges nothing to write."""+ sessions = []+ for path in glob.glob(os.path.expanduser('~/.codex/sessions/**/rollout-*.jsonl'), recursive=True):+ raw, last_total = [], None+ for line in open(path, errors='replace'):+ try:+ rec = json.loads(line)+ except ValueError:continue- causes['first call' if gap is None else 'after the cache expired' if gap > LIFE else 'mid-session miss'] += 1+ p = rec.get('payload') or {}+ if rec.get('type') == 'session_meta':+ if not wanted(p.get('cwd', '')):+ break+ continue+ if rec.get('type') != 'event_msg' or p.get('type') != 'token_count' or not p.get('info'):+ continue+ total = p['info']['total_token_usage']['total_tokens']+ if total == last_total: # the same usage reported again+ continue+ last_total = total+ u = p['info']['last_token_usage']+ if u['input_tokens']:+ raw.append((u['cached_input_tokens'], u['input_tokens'], 0, 0, '', rec['timestamp']))+ rows = [r[:5] + (g,) for r, g in zip(raw, gaps(r[5] for r in raw))]+ if rows:+ sessions.append(rows)+ return sessions- def pct(x):- return f'{100 * x:5.1f}%'- def line(name, rows, per):- by_tokens = sum(r[0] for r in rows) / sum(r[1] for r in rows)- return (f'{name:<22} {pct(by_tokens):>9} {pct(mean(r[0] / r[1] for r in rows)):>8} '- f'{pct(mean(per)):>11} {pct(median(per)):>9}')+def cost(read, total, w1h, write, model):+ """Anthropic input cost at list multipliers: read 0.1x (0.05x on the 5.5 models), 5-minute write 1.25x, 1-hour 2x."""+ return total - read - write + 1.25 * (write - w1h) + 2 * w1h + (0.05 if '-5-5' in model else 0.1) * read++def pct(x):+ return f'{100 * x:5.1f}%'+++def report(name, sessions, priced):+ if not sessions:+ print(f'{name}: no sessions found\n')+ return+ every = [r for s in sessions for r in s]+ warm = [r for r in every if r[5] is not None and r[5] <= LIFE]+ per_every = [sum(r[0] for r in s) / sum(r[1] for r in s) for s in sessions]+ per_warm = [sum(r[0] for r in h) / sum(r[1] for r in h) for s in sessions+ if (h := [r for r in s if r[5] is not None and r[5] <= LIFE])]+ causes = Counter('first call' if r[5] is None else 'after the cache expired' if r[5] > LIFE else 'mid-session miss'+ for r in every if r[0] < r[1] / 2)total = sum(r[1] for r in every)- print(f'{len(sessions)} conversations, {len(every)} calls, {total / 1e6:.2f}M input tokens\n')++ def line(label, rows, per):+ by_tokens = sum(r[0] for r in rows) / sum(r[1] for r in rows) if rows else 0+ spread = f'{pct(mean(per)):>11} {pct(median(per)):>9}' if per else ''+ per_call = pct(mean(r[0] / r[1] for r in rows)) if rows else ''+ return f'{label:<22} {pct(by_tokens):>9} {per_call:>8} {spread}'++ print(f'{name}: {len(sessions)} conversations, {len(every)} calls, {total / 1e6:.2f}M input tokens')print(f"{'hit rate':<22} {'by tokens':>9} {'per call':>8} {'per session':>11} {'median':>9}")print(line('in a conversation', warm, per_warm))print(line('every call', every, per_every))- print('\n"in a conversation" leaves out each first call and calls more than an hour after the previous one.')print(f'cold calls (under half read from cache): {dict(causes)}')- print(f"1-hour writes: {pct(sums['w1h'] / max(sums['write'], 1))} of cache writes; "- f"input cost vs. uncached: {pct(sums['cost'] / total)}")+ if priced:+ w1h, write = sum(r[2] for r in every), sum(r[3] for r in every)+ print(f'1-hour writes: {pct(w1h / max(write, 1))} of cache writes; input cost vs. uncached: '+ f'{pct(sum(cost(*r[:5]) for r in every) / total)}')+ print()-main()+report('Claude', claude(), priced=True)+report('Codex', codex(), priced=False)+print('"in a conversation" leaves out each first call and calls more than an hour after the previous one.')