Skip to content

Open source Grok Bot alternative, Bring your own agent and subscription: message Claude Code, Codex, Gemini CLI, Cursor, Copilot and 40+ AI agents as bots on your own computer, from iPhone, Mac, Windows, Linux or terminal. Agent-agnostic and free. Also a Muse and Dots alternative. Built in Rust.

docs: measure Codex bots' prompt cache too

leepokaide4b70d3491c
diff --git a/docs/design/bot-memory.md b/docs/design/bot-memory.md
index 5f92d3dc..87d57143 100644
--- a/docs/design/bot-memory.md
+++ b/docs/design/bot-memory.md
@@ -775,12 +775,16 @@ for the owner's decision.
- decisions.
**Prompt cache.** The host logs each turn's cache reads and writes (`turn usage` in `host.log`).
-`tools/memory-eval/cache.py` measures the hit rate from Claude Code's own session logs; on 2026-10-18:
+`tools/memory-eval/cache.py` measures the hit rate from Claude Code's and Codex's own session logs; on 2026-10-18:
| | Conversations | Calls | In a conversation | Every call |
|---|---|---|---|---|
-| The owner's bots (`~/.codync`, `~/.codync-dev`) | 27 | 229 | **97.6%** | 83.7% |
-| The evaluation run | 173 | 472 | 98.1% | 71.1% |
+| The owner's Claude bots (`~/.codync`, `~/.codync-dev`) | 27 | 229 | **97.6%** | 83.7% |
+| The owner's Codex bots | 7 | 24 | 96.7% | 81.4% |
+| The evaluation run (Claude) | 173 | 472 | 98.1% | 71.1% |
+
+The owner's Cursor, oh-my-pi and custom bots aren't measured: the script reads only those two logs. Once the host
+from this branch runs, its `turn usage` lines cover every harness whose adapter reports cached tokens.
- **In a conversation** is the share of input read from cache, leaving out each conversation's first call and
calls more than an hour after the previous one. Per call it is 97.2%, and the median conversation 99.2%.
@@ -792,7 +796,7 @@ for the owner's decision.
cache-safe opening. The evaluation had none.
- **Cost:** Claude Code writes every cache entry for an hour, at 2× input. Overall, input cost 36.9% of the same
calls uncached. A one-call conversation costs more than no cache would (157%): it writes the cache and never
- reads it back.
+ reads it back. Codex (OpenAI) caches on its own and charges nothing to write.
**Product metrics** (PostHog counts only, never content):
diff --git a/tools/memory-eval/README.md b/tools/memory-eval/README.md
index 9c9a73c0..57d24185 100644
--- a/tools/memory-eval/README.md
+++ b/tools/memory-eval/README.md
@@ -23,12 +23,13 @@ the day of the run.
## Prompt cache
-`cache.py` reads Claude Code's session logs (`~/.claude/projects`) and prints how much of each conversation's input
-came from the prompt cache, why the cold calls were cold, and the input cost against no cache. It needs no host:
+`cache.py` reads Claude Code's and Codex's session logs (`~/.claude/projects`, `~/.codex/sessions`) and prints,
+per harness, how much of each conversation's input came from the prompt cache and why the cold calls were cold;
+for Claude also the input cost against no cache. It needs no host:
```sh
tools/memory-eval/cache.py # the bots of ~/.codync and ~/.codync-dev
-tools/memory-eval/cache.py eval-home-bots # session folders whose name contains this
+tools/memory-eval/cache.py eval-home-bots # workspaces whose path contains this ("/" and "." read as "-")
```
The latest numbers are under [prompt cache](../../docs/design/bot-memory.md#evaluation).
diff --git a/tools/memory-eval/cache.py b/tools/memory-eval/cache.py
index 4f94ad34..0527b2d6 100755
--- a/tools/memory-eval/cache.py
+++ b/tools/memory-eval/cache.py
@@ -1,100 +1,142 @@
#!/usr/bin/env python3
-"""How well Claude bots' prompt cache holds, read from Claude Code's own session logs.
+"""How well bots' prompt cache holds, read from the harnesses' own session logs.
-Each session file in ~/.claude/projects is one conversation; each API call in it is counted once.
-Usage: tools/memory-eval/cache.py [folder-name-part ...] (default: the bots of ~/.codync and ~/.codync-dev)
+Claude Code keeps them in ~/.claude/projects, Codex in ~/.codex/sessions. Each session file is one conversation;
+each API call in it is counted once. Other harnesses aren't read; the host's `turn usage` log covers any harness
+whose adapter reports cached tokens.
+Usage: tools/memory-eval/cache.py [workspace-part ...]
+A part matches the workspace path with every non-alphanumeric character as "-", the way Claude Code names its
+folders (default: the bots of ~/.codync and ~/.codync-dev).
"""
import glob
import json
import os
+import re
import sys
from collections import Counter
from datetime import datetime
from statistics import mean, median
-ROOT = os.path.expanduser('~/.claude/projects')
-LIFE = 3600 # Claude Code writes 1-hour cache entries.
+LIFE = 3600 # longest a cache entry lives: Claude Code writes 1-hour entries
+PARTS = sys.argv[1:] or ['-codync-bots-', '-codync-dev-bots-']
-def calls(path, seen):
- """(read, total input, 1h writes, writes, model, seconds since the previous call) per API call."""
- prev = None
- for line in open(path, errors='replace'):
- try:
- rec = json.loads(line)
- except ValueError:
- continue
- msg = rec.get('message') if isinstance(rec, dict) else None
- if not isinstance(msg, dict) or msg.get('role') != 'assistant' or 'usage' not in msg:
- continue
- key = msg.get('id') or rec.get('uuid')
- if key in seen: # a forked session repeats its parent's calls
- continue
- seen.add(key)
- u = msg['usage']
- read = u.get('cache_read_input_tokens') or 0
- write = u.get('cache_creation_input_tokens') or 0
- total = read + write + (u.get('input_tokens') or 0)
- if total == 0:
- continue
- ts = datetime.fromisoformat(rec['timestamp'].replace('Z', '+00:00'))
- gap = (ts - prev).total_seconds() if prev else None
- prev = ts
- w1h = (u.get('cache_creation') or {}).get('ephemeral_1h_input_tokens') or 0
- yield read, total, w1h, write, msg.get('model', ''), gap
+def wanted(path):
+ return any(p in re.sub(r'[^A-Za-z0-9]', '-', path) for p in PARTS)
-def cost(read, total, w1h, write, model):
- """Input cost at API list multipliers: read 0.1x (0.05x on the 5.5 models), 5-minute write 1.25x, 1-hour 2x."""
- return total - read - write + 1.25 * (write - w1h) + 2 * w1h + (0.05 if '-5-5' in model else 0.1) * read
+def gaps(stamps):
+ prev = None
+ for ts in stamps:
+ t = datetime.fromisoformat(ts.replace('Z', '+00:00'))
+ yield (t - prev).total_seconds() if prev else None
+ prev = t
-def main():
- parts = sys.argv[1:] or ['-codync-bots-', '-codync-dev-bots-']
- sessions, causes, sums = [], Counter(), Counter()
- for folder in glob.glob(os.path.join(ROOT, '*')):
- if not any(p in os.path.basename(folder) for p in parts):
+def claude():
+ """Per conversation: (read, total input, 1h writes, writes, model, seconds since the previous call)."""
+ sessions = []
+ for folder in glob.glob(os.path.expanduser('~/.claude/projects/*')):
+ if not wanted(os.path.basename(folder)):
continue
seen = set()
for path in glob.glob(os.path.join(folder, '*.jsonl')):
- rows = list(calls(path, seen))
+ raw = []
+ for line in open(path, errors='replace'):
+ try:
+ rec = json.loads(line)
+ except ValueError:
+ continue
+ msg = rec.get('message') if isinstance(rec, dict) else None
+ if not isinstance(msg, dict) or msg.get('role') != 'assistant' or 'usage' not in msg:
+ continue
+ key = msg.get('id') or rec.get('uuid')
+ if key in seen: # a forked session repeats its parent's calls
+ continue
+ seen.add(key)
+ u = msg['usage']
+ read = u.get('cache_read_input_tokens') or 0
+ write = u.get('cache_creation_input_tokens') or 0
+ total = read + write + (u.get('input_tokens') or 0)
+ w1h = (u.get('cache_creation') or {}).get('ephemeral_1h_input_tokens') or 0
+ if total:
+ raw.append((read, total, w1h, write, msg.get('model', ''), rec['timestamp']))
+ rows = [r[:5] + (g,) for r, g in zip(raw, gaps(r[5] for r in raw))]
if rows:
sessions.append(rows)
- if not sessions:
- sys.exit('no sessions found')
- every, warm, per_every, per_warm = [], [], [], []
- for rows in sessions:
- hot = [r for r in rows if r[5] is not None and r[5] <= LIFE]
- every += rows
- warm += hot
- per_every.append(sum(r[0] for r in rows) / sum(r[1] for r in rows))
- if hot:
- per_warm.append(sum(r[0] for r in hot) / sum(r[1] for r in hot))
- for read, total, w1h, write, model, gap in rows:
- sums['w1h'] += w1h
- sums['write'] += write
- sums['cost'] += cost(read, total, w1h, write, model)
- if read >= total / 2:
+ return sessions
+
+
+def codex():
+ """The same rows from Codex rollouts. OpenAI caches on its own and charges nothing to write."""
+ sessions = []
+ for path in glob.glob(os.path.expanduser('~/.codex/sessions/**/rollout-*.jsonl'), recursive=True):
+ raw, last_total = [], None
+ for line in open(path, errors='replace'):
+ try:
+ rec = json.loads(line)
+ except ValueError:
continue
- causes['first call' if gap is None else 'after the cache expired' if gap > LIFE else 'mid-session miss'] += 1
+ p = rec.get('payload') or {}
+ if rec.get('type') == 'session_meta':
+ if not wanted(p.get('cwd', '')):
+ break
+ continue
+ if rec.get('type') != 'event_msg' or p.get('type') != 'token_count' or not p.get('info'):
+ continue
+ total = p['info']['total_token_usage']['total_tokens']
+ if total == last_total: # the same usage reported again
+ continue
+ last_total = total
+ u = p['info']['last_token_usage']
+ if u['input_tokens']:
+ raw.append((u['cached_input_tokens'], u['input_tokens'], 0, 0, '', rec['timestamp']))
+ rows = [r[:5] + (g,) for r, g in zip(raw, gaps(r[5] for r in raw))]
+ if rows:
+ sessions.append(rows)
+ return sessions
- def pct(x):
- return f'{100 * x:5.1f}%'
- def line(name, rows, per):
- by_tokens = sum(r[0] for r in rows) / sum(r[1] for r in rows)
- return (f'{name:<22} {pct(by_tokens):>9} {pct(mean(r[0] / r[1] for r in rows)):>8} '
- f'{pct(mean(per)):>11} {pct(median(per)):>9}')
+def cost(read, total, w1h, write, model):
+ """Anthropic input cost at list multipliers: read 0.1x (0.05x on the 5.5 models), 5-minute write 1.25x, 1-hour 2x."""
+ return total - read - write + 1.25 * (write - w1h) + 2 * w1h + (0.05 if '-5-5' in model else 0.1) * read
+
+def pct(x):
+ return f'{100 * x:5.1f}%'
+
+
+def report(name, sessions, priced):
+ if not sessions:
+ print(f'{name}: no sessions found\n')
+ return
+ every = [r for s in sessions for r in s]
+ warm = [r for r in every if r[5] is not None and r[5] <= LIFE]
+ per_every = [sum(r[0] for r in s) / sum(r[1] for r in s) for s in sessions]
+ per_warm = [sum(r[0] for r in h) / sum(r[1] for r in h) for s in sessions
+ if (h := [r for r in s if r[5] is not None and r[5] <= LIFE])]
+ causes = Counter('first call' if r[5] is None else 'after the cache expired' if r[5] > LIFE else 'mid-session miss'
+ for r in every if r[0] < r[1] / 2)
total = sum(r[1] for r in every)
- print(f'{len(sessions)} conversations, {len(every)} calls, {total / 1e6:.2f}M input tokens\n')
+
+ def line(label, rows, per):
+ by_tokens = sum(r[0] for r in rows) / sum(r[1] for r in rows) if rows else 0
+ spread = f'{pct(mean(per)):>11} {pct(median(per)):>9}' if per else ''
+ per_call = pct(mean(r[0] / r[1] for r in rows)) if rows else ''
+ return f'{label:<22} {pct(by_tokens):>9} {per_call:>8} {spread}'
+
+ print(f'{name}: {len(sessions)} conversations, {len(every)} calls, {total / 1e6:.2f}M input tokens')
print(f"{'hit rate':<22} {'by tokens':>9} {'per call':>8} {'per session':>11} {'median':>9}")
print(line('in a conversation', warm, per_warm))
print(line('every call', every, per_every))
- print('\n"in a conversation" leaves out each first call and calls more than an hour after the previous one.')
print(f'cold calls (under half read from cache): {dict(causes)}')
- print(f"1-hour writes: {pct(sums['w1h'] / max(sums['write'], 1))} of cache writes; "
- f"input cost vs. uncached: {pct(sums['cost'] / total)}")
+ if priced:
+ w1h, write = sum(r[2] for r in every), sum(r[3] for r in every)
+ print(f'1-hour writes: {pct(w1h / max(write, 1))} of cache writes; input cost vs. uncached: '
+ f'{pct(sum(cost(*r[:5]) for r in every) / total)}')
+ print()
-main()
+report('Claude', claude(), priced=True)
+report('Codex', codex(), priced=False)
+print('"in a conversation" leaves out each first call and calls more than an hour after the previous one.')