Experience-Driven Black-Box Cost Control for Coding Agents
Python agentic-aiagentic-codingai-codingai-coding-assistantanthropicartificial-intelligenceclaude-codecodexcoding-agentcoding-agentscost-optimizationcost-reductiondeveloper-toolsgithub-copilotllmopen-sourceopenaiproductivitytoken-optimization
Tool pruning: count prompts as well as sessions
Aaron508fb3c30694
diff --git a/README.md b/README.mdindex 6a3e028..190d71d 100644--- a/README.md+++ b/README.md@@ -41,8 +41,9 @@ what the next sessions in that repository pay for:that `pytest` is missing here and `python -m pytest` works, that the test runner wants labelswithout a `tests.` prefix -- are learned from recurring evidence and stated at the start ofthe next session. Learning them makes no model call.-- **Tool pruning.** Tools that stay loaded but go unused, session after session (five in a row- by default), are left out of the next sessions through the agent's own settings. By default+- **Tool pruning.** Tools that stay loaded but go unused (over ten prompts across at least two+ sessions, or five sessions however short) are left out of the next sessions through the+ agent's own settings. By defaultthis happens without asking you: the next session start says which tools were left out, andone command gives them back. Set `toolprune=propose` to approve each deny list yourself instead.- **Failure-loop intervention.** A turn that keeps repeating the same failing edit-and-test@@ -203,17 +204,20 @@ command (`repos`, `doctor`, `config unset`, ...) and how each agent asks beforeThe three mechanisms are `microskills`, `toolprune` and `loopguard`; each takes `on` or `off`,and each has numbers you can tune with the same `config set` command (in Claude Code's form here).-Tool pruning is `auto` by default: once a tool has gone unused for `toolprune_evidence` sessions-in a row (five), the plugin adds it to the deny list without asking, and it is left out from the-next session on. It writes the list once your sessions in the repository have been idle for an-hour (`toolprune_quiet_minutes`): a conversation resumed after its tools change writes its whole-history to the prompt cache again instead of reading it, and after an idle hour that cache has-expired anyway. With `toolprune=propose` the plugin only proposes the list, and nothing changes-until you run `toolprune apply`.+Tool pruning is `auto` by default: once a tool never used here has gone unused over+`toolprune_evidence` prompts (ten) across at least `toolprune_min_sessions` sessions (two), or+over `toolprune_sessions` sessions however short (five), the plugin adds it to the deny list+without asking, and it is left out from the next session on. Prompts let long sessions count for+more; the two-session floor keeps one session's work from deciding alone. It writes the list+once your sessions in the repository have been idle for an hour (`toolprune_quiet_minutes`): a+conversation resumed after its tools change writes its whole history to the prompt cache again+instead of reading it, and after an idle hour that cache has expired anyway. With+`toolprune=propose` the plugin only proposes the list, and nothing changes until you run+`toolprune apply`.```text/tokeneconomizer:config set microskills_max_facts=2 # inject at most two facts-/tokeneconomizer:config set toolprune_evidence=10 # wait for ten unused sessions+/tokeneconomizer:config set toolprune_evidence=20 # wait for twenty unused prompts/tokeneconomizer:config set loopguard_nudges=2 # redirect twice before telling you```diff --git a/docs/PLUGIN.md b/docs/PLUGIN.mdindex 64e3059..bc9d376 100644--- a/docs/PLUGIN.md+++ b/docs/PLUGIN.md@@ -54,7 +54,7 @@ Every hook fails open: any error becomes an empty output (no change) and a trace| Learning signal | the official grader's `resolved` | the envelope learns from turns that **ended green**: the agent stopped on its own, was not stopped by the guard, and no verification after its last edit failed | There is no grader. Micro-skills and evidence pruning never needed one || Injection channel | a prefix on the task prompt | `SessionStart` `additionalContext` | Verified to reach the model (§3) || Tool removal | `--disallowed-tools` / Codex config at launch | a **deny list**, applied by default once the evidence forms it and the repository's sessions have been idle longer than a prompt cache lives (§3), and announced at the next session start (`toolprune=propose` waits for approval instead), written to the repository's own settings file — tools in `.claude/settings.local.json`, Codex capability packages in `.codex/config.toml`, Copilot MCP servers in `.github/copilot/settings.local.json` — effective from the **next** session | The plugin launches nothing. A session keeps the tools it started with, but a conversation resumed after a change sends the new ones, re-writing its cached prefix, hence the idle wait. Each agent's file can express a different unit (§3) |-| Pruning candidates | `toolprune.tools.INAPPLICABLE_TOOLS` — orchestration, Q&A, plan mode, web — "inapplicable by construction" for a sealed headless task | nothing is inapplicable by construction; only **evidence** (loaded but unused for N sessions) proposes a tool, and a person's tools — AskUserQuestion, plan mode, sub-agents, the web — are protected | A person asks questions, plans and browses in a real session. A deny rule the plugin wrote for a tool that has since become protected is given back at the next fold |+| Pruning candidates | `toolprune.tools.INAPPLICABLE_TOOLS` — orchestration, Q&A, plan mode, web — "inapplicable by construction" for a sealed headless task | nothing is inapplicable by construction; only **evidence** (never used, and unused over N prompts across two or more sessions, or over N sessions) proposes a tool, and a person's tools — AskUserQuestion, plan mode, sub-agents, the web — are protected | A person asks questions, plans and browses in a real session. A deny rule the plugin wrote for a tool that has since become protected is given back at the next fold || Network | every arm, baseline included, carries an answer-key guard — a `PreToolUse` hook and an `/etc/hosts` block refuse the benchmarks' upstream sources — and drops the web tools | **no guard**: no `PreToolUse` hook (on Pi, no `tool_call` handler), no blocked command or host; the web tools stay | A benchmark's answer is a fetch away. A real session has no answer key, and the web is part of the work || Nudge | kill the agent, resume it with a continuation prompt | a mid-turn reminder attached to the tool result | Different intervention, so its effect is not inherited from the experiments || Still stuck after the redirect | terminal abort, in autonomous runs | a one-line **notice** to the person, ending with how to stop the turn on that agent and in that mode (§3); the plugin never stops a turn, and `loopguard: stop` is read as `nudge` | Not every agent's hooks can stop a turn (Codex and Copilot ignore `continue: false`), and the person decides: the benchmark guard's own design note says "interactive callers should map it to a hand-off" |@@ -418,6 +418,34 @@ OpenRouter (cache breakpoints, which move every turn, left out of the comparisonfact was retired on disk kept the original system message, and a new session took thecurrent facts.+**Counting prompts, 2026-10-24** (the `experimental` branch). Can the mechanisms learn from the+prompts of a session rather than from whole sessions, so a long first session counts for more?++* Tool pruning: yes, with a floor. Replaying this machine's recorded Claude Code sessions+ (12 projects, 61 sessions, 893 prompts) through the plugin's hooks and fold, against one tool+ catalog, and counting a tool a later session used as a wrong prune:++ | Rule | Projects with a list | Wrong prunes | Tools left out of the next session, summed |+ |---|---|---|---|+ | 5 sessions (`main`) | 7 | 1 | 340 |+ | 10 prompts over 2+ sessions, or 5 sessions (now) | 8 | 1 | 512 |+ | 10 prompts (`toolprune_min_sessions=1`) | 8 | 7 | 628 |++ Ten prompts in one session pruned tools the next session needed (`ListAgents`,+ `SendMessage`, a connector's tools): a session's prompts are one piece of work. Ten prompts+ alone also never prunes for one-prompt sessions, hence the five-session fallback. Prompts+ are counted from each agent's own log (`trace.*_prompt_count`); a session that ran none is+ no evidence.+* Micro-skills: no. Folded per prompt, a fact went active sooner, but a session meets its trap+ once and its later prompts use the working form the conversation already shows, so the fact+ fired in a quarter of the prompts that exercised it and never cleared the firing gate (40%+ over at least 4): in a stream of four-prompt sessions it was never injected, where per+ session it is injected at the fifth start. A replay that split prompts into sessions of+ their own had shown the opposite only because each such start recorded the fact as given.+ The facts still count sessions.+* The loop guard already counts prompt-turns: each turn that ends green feeds the envelope at+ its `Stop`, and the guard acts within the same session once it has five.+## 4. State and privacyAll state is local, under `$TOKENECONOMIZER_HOME` (default `~/.tokeneconomizer`), one directorydiff --git a/plugin/README.md b/plugin/README.mdindex a42ce23..a9b424b 100644--- a/plugin/README.md+++ b/plugin/README.md@@ -59,10 +59,10 @@ This records which tools a Claude Code session loads, with no model call. Do itmachine, and again after Claude Code updates. **Don't skip it:** without it the deny list seesonly the few tools Claude Code defers, not the ones every session loads.-**4. Work normally.** See [What to expect](#what-to-expect). Once tools have gone unused for-about five sessions, the plugin leaves them out of the next sessions without asking you (it-writes the list once your sessions here have been idle for an hour), and the session start says-so once:+**4. Work normally.** See [What to expect](#what-to-expect). Once a tool has gone unused over+ten prompts across two or more sessions (or five sessions, however short), the plugin leaves it+out of the next sessions without asking you (it writes the list once your sessions here have+been idle for an hour), and the session start says so once:```text/tokeneconomizer:toolprune # the tools it left out, and the evidence@@ -126,8 +126,9 @@ read-only, so Codex asks first: choose **Yes, proceed**. From a terminal, insidesession).**4. Work normally.** See [What to expect](#what-to-expect). Codex's deny list turns off the-sub-agent (`collaboration`), `goals` and image (`image_media`) packages, each only after it went-unused in every session where it was loaded; a package you used even once is kept. It is+sub-agent (`collaboration`), `goals` and image (`image_media`) packages, each only once it has+been loaded and unused long enough ([What to expect](#what-to-expect)); a package you used even+once is kept. It iswritten to the project's `.codex/config.toml`, which Codex reads **only in a trusted project**(the session start that announces it says so otherwise), and never into one your repositorycommits: that stays a proposal, and the session start says why. The web, asking you questions@@ -192,7 +193,8 @@ With no model call. Do it once on this machine, and again after Copilot updates.**4. Work normally.** See [What to expect](#what-to-expect). Copilot has no setting that removes asingle built-in tool, only one that turns off a whole MCP server, so the deny list turns off a-server only when **none** of its tools was used in about five sessions: one of yours from+server only when **none** of its tools was used long enough ([What to expect](#what-to-expect)):+one of yours from`~/.copilot/mcp-config.json`, or the built-in `github-mcp-server`, which loads only when you aresigned in to GitHub. It is written to `.github/copilot/settings.local.json`, and the next sessionstart says so; `/tokeneconomizer:toolprune restore` turns the server back on. `toolprune@@ -289,8 +291,11 @@ from); `pi config` turns it off without removing it. Optional: `rm -rf ~/.tokeneadds facts says so once, in one line. A resumed session keeps the facts it started with, wordfor word, so its cached prompt still matches. `microskills` shows the facts injected and thoseforming.-- **Deny list.** Applied without asking you, after about five sessions in which a tool, package- or server went unused, from the next session on; that session's start says so once. It is+- **Deny list.** Applied without asking you once a tool, package or server never used here has+ gone unused over ten prompts across at least two sessions, or over five sessions however short+ (three times as long for a few higher-risk tools), from the next session on; that session's+ start says so once. Prompts let long sessions count for more, and the two-session floor keeps+ one session's work from deciding alone. A session that ran no prompt counts for nothing. It iswritten once the repository's sessions have been idle for an hour: a conversation resumedafter its tools change writes its whole history to the prompt cache again instead of readingit, and after an idle hour that cache has expired anyway. `toolprune` shows a list still@@ -378,7 +383,7 @@ and are shared by every agent working in it. The examples use Claude Code's form```text/tokeneconomizer:config set microskills_max_facts=2 # inject at most two facts-/tokeneconomizer:config set toolprune_evidence=10 # wait for ten unused sessions+/tokeneconomizer:config set toolprune_evidence=20 # wait for twenty unused prompts/tokeneconomizer:config set loopguard_nudges=2 loopguard.min_turns=30/tokeneconomizer:config unset loopguard.min_turns # back to the default```@@ -388,7 +393,9 @@ and are shared by every agent working in it. The examples use Claude Code's form| `microskills_max_facts` | `4` | the most facts injected at session start || `microskills_max_tokens` | `150` | the size of the injected block, in tokens (about 4 characters each) || `microskills_propose_every` | `5` | sessions between runs of the paid fact proposer |-| `toolprune_evidence` | `5` | sessions a tool must go unused before it is left out (three times as long for a few higher-risk tools; the web is never proposed) |+| `toolprune_evidence` | `10` | prompts a never-used tool must go unused, over at least `toolprune_min_sessions` sessions, before it is left out (three times as many for a few higher-risk tools; the web is never proposed) |+| `toolprune_min_sessions` | `2` | sessions those prompts must span; `1` lets a single session's prompts decide |+| `toolprune_sessions` | `5` | or this many sessions, however few prompts they ran || `toolprune_quiet_minutes` | `60` | how long the repository's sessions must have been idle before `auto` writes a deny list, so that no conversation resumed while its prompt cache is warm loses it; `0` writes at once || `loopguard_aggr` | `default` | the guard's threshold preset: `conservative`, `default` or `aggressive` || `loopguard_nudges` | `1` | redirects per prompt-turn before the guard only notifies you |diff --git a/tests/test_live_core.py b/tests/test_live_core.pyindex 3472d67..5bcc679 100644--- a/tests/test_live_core.py+++ b/tests/test_live_core.py@@ -597,9 +597,11 @@ def test_a_job_that_never_answers_stops_blocking_the_next_one():# ---- Copilot session logs -------------------------------------------------------------------def copilot_events(path, calls, *, shutdown=True, model="gpt-5.4", cwd="/w/repo", sid="cp1"):- """A Copilot `session-state/<id>/events.jsonl`: session.start, one user message, the tool- calls as execution_start/complete pairs, and (unless killed) a session.shutdown."""+def copilot_events(path, calls, *, shutdown=True, model="gpt-5.4", cwd="/w/repo", sid="cp1",+ prompts=1):+ """A Copilot `session-state/<id>/events.jsonl`: session.start, `prompts` user messages (the+ tool calls, as execution_start/complete pairs, in the first), and (unless killed) a+ session.shutdown."""ev = [{"type": "session.start", "data": {"sessionId": sid, "copilotVersion": "1.0.83","selectedModel": model, "context": {"cwd": cwd}}},{"type": "user.message", "data": {"content": "fix it", "turnId": "0"}}]@@ -609,6 +611,8 @@ def copilot_events(path, calls, *, shutdown=True, model="gpt-5.4", cwd="/w/repo"ev.append({"type": "tool.execution_complete", "data": {"toolCallId": f"c{n}", "success": ok, "turnId": "0","result": {"content": text, "detailedContent": text}}})+ for k in range(1, prompts):+ ev.append({"type": "user.message", "data": {"content": "and then", "turnId": str(k)}})if shutdown:ev.append({"type": "session.shutdown", "data": {"modelMetrics": {model: {"requests": {"count": 4}, "usage": {diff --git a/tests/test_live_hooks.py b/tests/test_live_hooks.pyindex 33ef8cb..ca8f279 100644--- a/tests/test_live_hooks.py+++ b/tests/test_live_hooks.py@@ -139,6 +139,34 @@ def test_a_learned_fact_reaches_the_next_session_and_is_announced_once(env, repoassert starts[5]["hookSpecificOutput"]["additionalContext"] == ctx+def multi_prompt_trap_session(path, repo, i, prompts=4, every=False):+ """A session of `prompts` prompts that meets the missing `pytest` at its first test run (or,+ with `every`, at each): later prompts use the working form, as the conversation shows it."""+ def prompt(k):+ edit = ("Edit", {"file_path": f"{repo}/pkg/m{i}_{k}.py", "old_string": "a",+ "new_string": "b"}, "ok", False)+ known = ("Bash", {"command": "python -m pytest -q tests/test_a.py"}, "1 passed in 0.1s",+ False)+ trap = ("Bash", {"command": "pytest -q tests/test_a.py"},+ "bash: pytest: command not found", True)+ return (f"fix bug {i}.{k}", [edit, trap, known] if every or k == 0 else [edit, known])+ return write_session(path, [prompt(k) for k in range(prompts)])+++def test_facts_learn_per_session_however_many_prompts_it_ran(env, repo, tmp_path):+ """Unlike tool pruning, the facts count sessions, not prompts. Within a session the agent+ already knows the fix from its own earlier failure, so its later prompts would count as+ exercising the fact without the failure: per prompt, a trap met once a session fires in a+ quarter of them and never clears the firing gate. Four-prompt sessions reach the fact at the+ fifth start, as one-prompt sessions do (above)."""+ claim = "`pytest …` fails here with command not found; `python -m pytest …` works."+ starts = [run_session(env, repo, f"s{i}",+ multi_prompt_trap_session(str(tmp_path / f"s{i}.jsonl"), repo, i))+ for i in range(6)]+ assert all(s == {} for s in starts[:4])+ assert claim in starts[4]["hookSpecificOutput"]["additionalContext"]++def test_resume_does_not_inject_again_and_shadow_never_injects(env, repo, tmp_path):for i in range(5):run_session(env, repo, f"s{i}", trap_session(str(tmp_path / f"s{i}.jsonl"), repo, i))@@ -378,7 +406,7 @@ def test_with_propose_unused_tools_are_proposed_but_nothing_changes_until_appliefor i in range(5):t = write_session(str(tmp_path / f"u{i}.jsonl"), [("look", [("Read", {"file_path": f"{repo}/a.py"}, "x", False),- ("Bash", {"command": "ls"}, "a.py", False)])])+ ("Bash", {"command": "ls"}, "a.py", False)]), ("and again", [])])run_session(env, repo, f"u{i}", t)st = store_for(env, repo)prop = st.read("untool.json")["proposal"]@@ -402,7 +430,7 @@ def test_by_default_unused_tools_are_removed_from_the_next_session_and_it_says_sfor i in range(5):t = write_session(str(tmp_path / f"u{i}.jsonl"), [("look", [("Read", {"file_path": f"{repo}/a.py"}, "x", False),- ("Bash", {"command": "ls"}, "a.py", False)])])+ ("Bash", {"command": "ls"}, "a.py", False)]), ("and again", [])])run_session(env, repo, f"u{i}", t)st = store_for(env, repo)deny = json.load(open(denylist.settings_path(repo)))["permissions"]["deny"]@@ -431,7 +459,7 @@ def test_tools_given_back_stay_back_until_named_again(env, repo, tmp_path, monkedef sessions(prefix, n):for i in range(n):t = write_session(str(tmp_path / f"{prefix}{i}.jsonl"), [("look", [- ("Bash", {"command": "ls"}, "a.py", False)])])+ ("Bash", {"command": "ls"}, "a.py", False)]), ("and again", [])])run_session(env, repo, f"{prefix}{i}", t)sessions("a", 5)def deny():@@ -449,6 +477,74 @@ def test_tools_given_back_stay_back_until_named_again(env, repo, tmp_path, monkeassert denylist.kept_for(store_for(env, repo), repo) == ["CronCreate"]+def _prompts(env, repo, tmp_path, sid, n):+ """One Claude Code session of `n` prompts; only the first calls a tool (Bash)."""+ t = write_session(str(tmp_path / f"{sid}.jsonl"),+ [(f"prompt {k}", [("Bash", {"command": "ls"}, "a.py", False)] if k == 0+ else []) for k in range(n)])+ run_session(env, repo, sid, t)+++def _another_repo(tmp_path, name):+ r = tmp_path / name+ (r / ".git").mkdir(parents=True)+ (r / ".git" / "HEAD").write_text("ref: refs/heads/main\n")+ return str(r)+++def test_ten_prompts_over_two_sessions_are_enough_and_one_sessions_are_not(env, repo, tmp_path):+ """The evidence counts prompts as well as sessions: a never-used tool is left out after ten+ prompts over at least two sessions. One session's prompts are one piece of work, so ten of+ them in a single session are not enough -- unless `toolprune_min_sessions=1`."""+ env = dict(env, TE_TOOLPRUNE="propose")+ _catalog(env, ["Bash", "Read", "Workflow", "CronCreate"])+ _prompts(env, repo, tmp_path, "long", 10)+ st = store_for(env, repo)+ unt = st.read("untool.json")+ assert unt["proposal"] == []+ assert (unt["streaks"]["Workflow"], unt["prompt_streaks"]["Workflow"]) == (1, 10)+ _prompts(env, repo, tmp_path, "short", 1)+ assert sorted(st.read("untool.json")["proposal"]) == ["CronCreate", "Workflow"] # not Read++ one = dict(env, TE_TOOLPRUNE_MIN_SESSIONS="1")+ nine, ten = _another_repo(tmp_path, "nine"), _another_repo(tmp_path, "ten")+ _prompts(one, nine, tmp_path, "s9", 9)+ _prompts(one, ten, tmp_path, "s10", 10)+ assert store_for(one, nine).read("untool.json")["proposal"] == []+ assert sorted(store_for(one, ten).read("untool.json")["proposal"]) == ["CronCreate", "Workflow"]+++def test_five_sessions_are_enough_however_short(env, repo, tmp_path):+ env = dict(env, TE_TOOLPRUNE="propose")+ _catalog(env, ["Bash", "Workflow"])+ for i in range(4):+ _prompts(env, repo, tmp_path, f"s{i}", 1)+ assert store_for(env, repo).read("untool.json")["proposal"] == []+ _prompts(env, repo, tmp_path, "s4", 1)+ assert store_for(env, repo).read("untool.json")["proposal"] == ["Workflow"]+++def test_a_session_without_prompts_is_no_evidence(env, repo, tmp_path):+ env = dict(env, TE_TOOLPRUNE="propose")+ _catalog(env, ["Bash", "Workflow"])+ _prompts(env, repo, tmp_path, "some", 4)+ _prompts(env, repo, tmp_path, "none", 0)+ unt = store_for(env, repo).read("untool.json")+ assert (unt["sessions"], unt["streaks"]["Workflow"], unt["prompt_streaks"]["Workflow"]) == (+ 1, 1, 4)+++def test_a_high_risk_tool_needs_three_times_the_prompts(env, repo, tmp_path):+ env = dict(env, TE_TOOLPRUNE="propose")+ _catalog(env, ["Bash", "Workflow", "NotebookEdit"])+ _prompts(env, repo, tmp_path, "a", 20)+ _prompts(env, repo, tmp_path, "b", 9)+ assert store_for(env, repo).read("untool.json")["proposal"] == ["Workflow"]+ _prompts(env, repo, tmp_path, "c", 1)+ assert sorted(store_for(env, repo).read("untool.json")["proposal"]) == [+ "NotebookEdit", "Workflow"]++def _pending_repo(env, repo, tmp_path, quiet="60"):"""Five sessions that leave three tools unused, under the default wait before writing."""env = dict(env, TE_TOOLPRUNE_QUIET_MINUTES=quiet)@@ -457,7 +553,7 @@ def _pending_repo(env, repo, tmp_path, quiet="60"):for i in range(5):t = write_session(str(tmp_path / f"u{i}.jsonl"), [("look", [("Read", {"file_path": f"{repo}/a.py"}, "x", False),- ("Bash", {"command": "ls"}, "a.py", False)])])+ ("Bash", {"command": "ls"}, "a.py", False)]), ("and again", [])])run_session(env, repo, f"u{i}", t)return env, LiveContext.build(hosts.ClaudeCode(), repo, env)@@ -751,12 +847,13 @@ GITHUB_TOOLS = ["github-mcp-server-list_issues", "github-mcp-server-get_commit","github-mcp-server-search_code"]-def copilot_session(env, repo, tmp_path, sid, calls, monkeypatch, source="new"):+def copilot_session(env, repo, tmp_path, sid, calls, monkeypatch, source="new", prompts=1):"""One Copilot session, as its hook file delivers it: no transcript path before Stop (thelog's place is fixed by the session id), Claude tool names, a top-level context answer."""home = tmp_path / "copilot-home"monkeypatch.setenv("COPILOT_HOME", str(home))- copilot_events(str(home / "session-state" / sid / "events.jsonl"), calls, cwd=repo, sid=sid)+ copilot_events(str(home / "session-state" / sid / "events.jsonl"), calls, cwd=repo, sid=sid,+ prompts=prompts)base = {"session_id": sid, "cwd": repo, "timestamp": "2026-10-08T04:22:15.157Z"}start = call(env, dict(base, hook_event_name="SessionStart", source=source), "copilot")call(env, dict(base, hook_event_name="UserPromptSubmit", prompt="fix it"), "copilot")@@ -821,25 +918,39 @@ def test_copilot_proposes_only_whole_unused_mcp_servers(env, repo, tmp_path, monfor i in range(5):copilot_session(env, repo, tmp_path, f"u{i}", [("bash", {"command": "ls"}, "a.py\n<shellId: 0 completed with exit code 0>", True)],- monkeypatch)+ monkeypatch, prompts=2)st = store_for(env, repo, "copilot")unt = st.read("untool.json")# builtins went unused too, but Copilot has no persistent per-tool lever: only the serverassert unt["proposal"] == ["github-mcp-server"]assert unt["servers"]["github-mcp-server"] == {"tools": 3, "used": False,- "unused_sessions": 5}+ "unused_sessions": 5, "unused_prompts": 10}assert not os.path.exists(denylist.settings_path(repo, "copilot"))(notice,) = copilot_session(env, repo, tmp_path, "n", [], monkeypatch)["progress"]assert "1 MCP server(s) went unused" in notice and "/tokeneconomizer:toolprune" in notice+def test_two_copilot_sessions_of_five_prompts_are_enough_to_propose_a_server(+ env, repo, tmp_path, monkeypatch):+ env = dict(env, TE_TOOLPRUNE="propose")+ _copilot_catalog(env, ["bash", "view"] + GITHUB_TOOLS)+ for sid in ("u1", "u2"):+ copilot_session(env, repo, tmp_path, sid, [+ ("bash", {"command": "ls"}, "a.py\n<shellId: 0 completed with exit code 0>", True)],+ monkeypatch, prompts=5)+ unt = store_for(env, repo, "copilot").read("untool.json")+ assert unt["proposal"] == ["github-mcp-server"]+ assert unt["servers"]["github-mcp-server"]["unused_sessions"] == 2+ assert unt["servers"]["github-mcp-server"]["unused_prompts"] == 10++def test_by_default_copilot_turns_off_the_unused_server_and_says_so(env, repo, tmp_path,monkeypatch):_copilot_catalog(env, ["bash", "view"] + GITHUB_TOOLS)for i in range(5):copilot_session(env, repo, tmp_path, f"u{i}", [("bash", {"command": "ls"}, "a.py\n<shellId: 0 completed with exit code 0>", True)],- monkeypatch)+ monkeypatch, prompts=2)settings = json.load(open(denylist.settings_path(repo, "copilot")))assert settings["disabledMcpServers"] == ["github-mcp-server"](notice,) = copilot_session(env, repo, tmp_path, "n", [], monkeypatch)["progress"]@@ -875,18 +986,24 @@ def test_copilot_apply_and_restore_touch_only_the_plugins_own_servers(repo, tmp_# ---- Codex capability untooling -----------------------------------------------------------def codex_rollout(path, repo, i, exec_inputs=()):- """A Codex rollout: one shell command and, per `exec_inputs`, one code-mode exec call."""+def codex_rollout(path, repo, i, exec_inputs=(), prompts=1):+ """A Codex rollout of `prompts` turns: in the first, one shell command and, per+ `exec_inputs`, one code-mode exec call."""recs = [{"type": "session_meta", "payload": {"id": f"thread-{i}", "cwd": repo,- "cli_version": "0.153.1"}},- {"type": "turn_context", "payload": {"turn_id": "t1", "model": "gpt-5.6-sol"}},- {"type": "event_msg", "payload": {"type": "item_completed", "turn_id": "t1", "item": {- "type": "CommandExecution", "id": f"c{i}", "command": ["bash", "-lc", "ls"],- "aggregated_output": "a.py", "exit_code": 0, "status": "completed"}}}]- for k, src in enumerate(exec_inputs):- recs.append({"type": "response_item", "payload": {- "type": "custom_tool_call", "name": "exec", "call_id": f"x{i}-{k}",- "status": "completed", "input": src}})+ "cli_version": "0.153.1"}}]+ for k in range(1, prompts + 1):+ recs += [{"type": "turn_context", "payload": {"turn_id": f"t{k}", "model": "gpt-5.6-sol"}},+ {"type": "event_msg", "payload": {"type": "task_started", "turn_id": f"t{k}"}}]+ if k > 1:+ continue+ recs.append({"type": "event_msg", "payload": {"type": "item_completed", "turn_id": "t1",+ "item": {+ "type": "CommandExecution", "id": f"c{i}", "command": ["bash", "-lc", "ls"],+ "aggregated_output": "a.py", "exit_code": 0, "status": "completed"}}})+ for n, src in enumerate(exec_inputs):+ recs.append({"type": "response_item", "payload": {+ "type": "custom_tool_call", "name": "exec", "call_id": f"x{i}-{n}",+ "status": "completed", "input": src}})recs.append({"type": "event_msg", "payload": {"type": "token_count", "info": {"total_token_usage": {"input_tokens": 3000, "cached_input_tokens": 2000,"cache_write_input_tokens": 0, "output_tokens": 90,@@ -900,24 +1017,30 @@ def test_codex_proposes_capabilities_it_never_saw_used(env, repo, tmp_path):env = dict(env, TE_TOOLPRUNE="propose")for i in range(5):uses = ["await tools.view_image({path: 'a.png'})"] if i == 2 else []- t = codex_rollout(str(tmp_path / f"rollout-{i}.jsonl"), repo, i, uses)+ t = codex_rollout(str(tmp_path / f"rollout-{i}.jsonl"), repo, i, uses, prompts=2)run_session(env, repo, f"cx{i}", t, host="codex")unt = store_for(env, repo, "codex").read("untool.json")- # collaboration and goals went unused in all five; image was used once and is kept+ # collaboration and goals went unused over all ten prompts; image was used once: keptassert unt["proposal"] == ["collaboration", "goals"]assert unt["capabilities"]["image_media"]["used"] is True- assert unt["capabilities"]["goals"] == {"loaded_sessions": 5, "unused_sessions": 5,- "used": False}+ assert unt["capabilities"]["goals"] == {"loaded_sessions": 5, "loaded_prompts": 10,+ "unused_sessions": 5, "used": False}+ assert (unt["sessions"], unt["prompts"]) == (5, 10)assert not os.path.exists(denylist.settings_path(repo, "codex")) # nothing written yetdef test_codex_needs_the_full_evidence_before_proposing(env, repo, tmp_path):"""The experiments' controller demotes collaboration after ONE unused task; a person's- sessions need `untool_evidence` of them."""- for i in range(4):+ sessions need `toolprune_evidence` prompts of it, however they split into sessions."""+ env = dict(env, TE_TOOLPRUNE="propose")+ for i, prompts in enumerate((3, 3, 3)):run_session(env, repo, f"cx{i}", codex_rollout(str(tmp_path / f"rollout-{i}.jsonl"),- repo, i), host="codex")- assert store_for(env, repo, "codex").read("untool.json")["proposal"] == []+ repo, i, prompts=prompts), host="codex")+ assert store_for(env, repo, "codex").read("untool.json")["proposal"] == [] # nine+ run_session(env, repo, "cx3", codex_rollout(str(tmp_path / "rollout-3.jsonl"), repo, 3),+ host="codex")+ assert store_for(env, repo, "codex").read("untool.json")["proposal"] == [+ "collaboration", "goals"] # ten@pytest.fixture@@ -989,7 +1112,7 @@ def test_a_committed_codex_config_is_never_edited(tmp_path, codex_accepts):def _codex_sessions(env, repo, tmp_path, n=5):for i in range(n):- t = codex_rollout(str(tmp_path / f"rollout-{i}.jsonl"), repo, i)+ t = codex_rollout(str(tmp_path / f"rollout-{i}.jsonl"), repo, i, prompts=2)run_session(env, repo, f"cx{i}", t, host="codex")diff --git a/tests/test_trace.py b/tests/test_trace.pyindex a193016..94a3bd1 100644--- a/tests/test_trace.py+++ b/tests/test_trace.py@@ -177,3 +177,58 @@ def test_single_attempt_domain_collapses_to_resolved():"""SWE-bench rows carry no attempt fields; pass@1 and pass@N are the same number."""assert _p1p2({"resolved": True}) == (True, True)assert _p1p2({"resolved": False}) == (False, False)+++def _jsonl(path, records):+ import json+ with open(path, "w") as fh:+ fh.write("\n".join(json.dumps(r) for r in records) + "\n")+ return str(path)+++def test_a_claude_session_counts_the_prompts_the_person_gave(tmp_path):+ """Typed prompts and slash commands; not tool results, a local command's output, an+ interrupt marker, a compaction summary, meta text, or a subagent's prompts."""+ def user(content, **flags):+ return dict({"type": "user", "message": {"role": "user", "content": content}}, **flags)+ p = _jsonl(tmp_path / "c.jsonl", [+ user("fix the test"),+ user([{"type": "tool_result", "tool_use_id": "t1", "content": "ok"}]),+ user([{"type": "text", "text": "and the docs"}, {"type": "image", "source": {}}]),+ user("<command-name>/tokeneconomizer:status</command-name>"),+ user("<local-command-stdout>done</local-command-stdout>"),+ user("[Request interrupted by user]"),+ user("Caveat: the messages below were generated by the user", isMeta=True),+ user("This session is being continued...", isCompactSummary=True),+ user("subagent task", isSidechain=True),+ {"type": "assistant", "message": {"role": "assistant", "content": "done"}}])+ assert trace.claude_prompt_count(p) == 3+++def test_a_codex_session_counts_one_prompt_per_turn(tmp_path):+ turn = [{"type": "turn_context", "payload": {"turn_id": "t"}},+ {"type": "event_msg", "payload": {"type": "task_started"}},+ {"type": "response_item", "payload": {"type": "message", "role": "user"}}]+ assert trace.codex_prompt_count(_jsonl(tmp_path / "r.jsonl", turn * 3)) == 3+ # a rollout without task_started events counts its turn contexts+ older = [r for r in turn if r["type"] != "event_msg"]+ assert trace.codex_prompt_count(_jsonl(tmp_path / "o.jsonl", older * 2)) == 2+++def test_a_copilot_session_counts_its_user_messages(tmp_path):+ p = _jsonl(tmp_path / "events.jsonl", [+ {"type": "session.start", "data": {}}, {"type": "user.message", "data": {}},+ {"type": "tool.execution_start", "data": {}}, {"type": "user.message", "data": {}}])+ assert trace.copilot_prompt_count(p) == 2++++def test_a_pi_session_counts_the_prompts_on_its_active_branch(tmp_path):+ """Pi keeps a conversation as a tree; a prompt on a branch the person left is not counted."""+ def msg(eid, parent, role):+ return {"type": "message", "id": eid, "parentId": parent, "message": {"role": role}}+ p = _jsonl(tmp_path / "pi.jsonl", [+ {"type": "session", "id": "s"}, msg("a", None, "user"), msg("b", "a", "assistant"),+ msg("x", "b", "user"), msg("y", "x", "assistant"), # abandoned+ msg("c", "b", "user"), msg("d", "c", "toolResult"), msg("e", "d", "assistant")])+ assert trace.pi_prompt_count(p) == 2diff --git a/tokeneconomizer/plugin/cli.py b/tokeneconomizer/plugin/cli.pyindex 1e45efe..d1f7217 100644--- a/tokeneconomizer/plugin/cli.py+++ b/tokeneconomizer/plugin/cli.py@@ -123,6 +123,7 @@ def cmd_status(args):if ln.startswith("- ")),"sessions_folded": inj.get("sessions", 0)},"toolprune": {"mode": cfg.toolprune, "sessions": unt.get("sessions", 0),+ "prompts": unt.get("prompts", 0),"catalog_source": unt.get("catalog_source"),"proposal": unt.get("proposal", []),"applied": denylist.applied_for(ctx.store, ctx.repo.root,@@ -161,7 +162,8 @@ def cmd_status(args):print(" toolprune not available on this host")else:src = u["catalog_source"] or "none yet"- print(f" toolprune {u['sessions']} sessions observed, catalog: {src}; "+ print(f" toolprune {u['sessions']} sessions ({u['prompts']} prompts) observed, "+ f"catalog: {src}; "f"proposed {denylist.unit(ctx.host.name, plural=True)}: "f"{', '.join(u['proposal']) or '-'}; applied: {', '.join(u['applied']) or '-'}")sp = out["loopguard"]@@ -190,8 +192,8 @@ def cmd_microskills(args):# ---- toolprune -----------------------------------------------------------------------------def _toolprune_show(ctx, unt):host = ctx.host.name- print(f"sessions observed: {unt.get('sessions', 0)} catalog: "- f"{unt.get('catalog_source') or 'none yet'}")+ print(f"sessions observed: {unt.get('sessions', 0)} ({unt.get('prompts', 0)} prompts) "+ f"catalog: {unt.get('catalog_source') or 'none yet'}")print(f"applied ({denylist.unit(host, plural=True)} this plugin added to "f"{denylist.settings_path(ctx.repo.root, host)}): "f"{', '.join(denylist.applied_for(ctx.store, ctx.repo.root, host)) or '-'}")@@ -211,7 +213,8 @@ def _toolprune_show(ctx, unt):if caps:print("capabilities (removable from a project's .codex/config.toml):")for c, e in sorted(caps.items()):- print(f" {c:14s} loaded in {e['loaded_sessions']:3d} sessions, unused in "+ print(f" {c:14s} loaded in {e['loaded_sessions']:3d} sessions "+ f"({e.get('loaded_prompts', e['loaded_sessions']):3d} prompts), unused in "f"{e['unused_sessions']:3d}{' USED (kept)' if e['used'] else ''}")trusted = denylist.codex_project_trusted(ctx.repo.root)if trusted is False:@@ -223,15 +226,20 @@ def _toolprune_show(ctx, unt):if servers:print("MCP servers (Copilot can only remove a whole server):")for name, e in sorted(servers.items()):- state = "USED (kept)" if e["used"] else f"unused in {e['unused_sessions']} sessions"+ state = ("USED (kept)" if e["used"] else+ f"unused for {e.get('unused_sessions', 0)} sessions "+ f"({e.get('unused_prompts', 0)} prompts)")print(f" {name:24s} {e['tools']:3d} tools, {state}")streaks = unt.get("streaks") or {}if streaks and host == "claude":- print("unused-session streaks (tools never used are candidates once over "- f"{ctx.config.toolprune_evidence}):")- for t, n in sorted(streaks.items(), key=lambda kv: -kv[1])[:30]:+ cfg = ctx.config+ prompts = unt.get("prompt_streaks") or {}+ print(f"how long each tool has gone unused (a tool never used is left out after "+ f"{cfg.toolprune_evidence} prompts over {cfg.toolprune_min_sessions}+ sessions, "+ f"or after {cfg.toolprune_sessions} sessions):")+ for t, n in sorted(streaks.items(), key=lambda kv: (-kv[1], -prompts.get(kv[0], 0)))[:30]:used = "used" if t in set(unt.get("used_tools", [])) else ""- print(f" {n:4d} {t} {used}")+ print(f" {n:4d} sessions {prompts.get(t, n):5d} prompts {t} {used}")if not unt.get("catalog_source") or unt.get("catalog_source") == "observed":if os.path.exists(os.path.join(ctx.root, "hosts", host, "catalog.json")):# Recorded, but a repository reads the catalog when its next session is folded.@@ -424,6 +432,8 @@ def _config_view(cfg):"toolprune": {"state": "off" if cfg.toolprune == "off" else "on","toolprune": d.pop("toolprune"),"toolprune_evidence": d.pop("toolprune_evidence"),+ "toolprune_min_sessions": d.pop("toolprune_min_sessions"),+ "toolprune_sessions": d.pop("toolprune_sessions"),"toolprune_quiet_minutes": d.pop("toolprune_quiet_minutes")},"loopguard": {"state": "off" if cfg.loopguard == "off" else "on",diff --git a/tokeneconomizer/plugin/config.py b/tokeneconomizer/plugin/config.pyindex d734ad5..a5015b3 100644--- a/tokeneconomizer/plugin/config.py+++ b/tokeneconomizer/plugin/config.py@@ -26,7 +26,9 @@ person can tune by hand:* microskills_max_facts, microskills_max_tokens -- caps on the injected block;* microskills_propose_every -- sessions between paid proposer runs;- * toolprune_evidence -- unused sessions before a tool is proposed;+ * toolprune_evidence -- prompts a never-used tool must have stayed unused through,+ over at least toolprune_min_sessions sessions, before it is+ proposed -- or over toolprune_sessions sessions, however short;* toolprune_quiet_minutes -- how long every session of the repository must have been idlebefore `auto` writes a deny list (0: at once), so that noconversation resumed with a warm prompt cache loses it;@@ -127,7 +129,9 @@ class LiveConfig:microskills_max_tokens: int = 150microskills_propose_every: int = 5toolprune: str = "auto"- toolprune_evidence: int = 5+ toolprune_evidence: int = 10+ toolprune_min_sessions: int = 2+ toolprune_sessions: int = 5toolprune_quiet_minutes: int = 60loopguard: str = "nudge"loopguard_aggr: str = "default"@@ -243,6 +247,8 @@ _ENV = {"TE_MODE": "mode", "TE_PAID": "paid", "TE_SHADOW": "shadow","TE_MICROSKILLS_PROPOSE_EVERY": "microskills_propose_every","TE_TOOLPRUNE": "toolprune", "TE_TOOLPRUNE_EVIDENCE": "toolprune_evidence","TE_TOOLPRUNE_QUIET_MINUTES": "toolprune_quiet_minutes",+ "TE_TOOLPRUNE_MIN_SESSIONS": "toolprune_min_sessions",+ "TE_TOOLPRUNE_SESSIONS": "toolprune_sessions","TE_LOOPGUARD": "loopguard", "TE_LOOPGUARD_AGGR": "loopguard_aggr","TE_LOOPGUARD_NUDGES": "loopguard_nudges","TE_REPOMAP": "repomap", "TE_REPOMAP_TOKENS": "repomap_tokens",diff --git a/tokeneconomizer/plugin/denylist.py b/tokeneconomizer/plugin/denylist.pyindex 67799bc..f778ae5 100644--- a/tokeneconomizer/plugin/denylist.py+++ b/tokeneconomizer/plugin/denylist.py@@ -532,16 +532,6 @@ def restore(repo_root, store, tools=None, host="claude"):return _restore_json_list(repo_root, store, tools, host)-def filter_proposal(proposal, streaks, evidence):- """Hold back high-risk tools until they have HIGH_RISK_FACTOR x the usual evidence."""- out = []- for t in proposal:- need = evidence * (live_adapter.HIGH_RISK_FACTOR if t in live_adapter.HIGH_RISK else 1)- if int(streaks.get(t, 0)) >= need:- out.append(t)- return out--def copilot_server_proposal(raw, catalog, used, servers):"""MCP servers every one of whose catalog tools is in the evidence proposal `raw`."""by_server = {}diff --git a/tokeneconomizer/plugin/fold.py b/tokeneconomizer/plugin/fold.pyindex d1e7768..7804408 100644--- a/tokeneconomizer/plugin/fold.py+++ b/tokeneconomizer/plugin/fold.py@@ -91,20 +91,43 @@ def _fold_microskills(ctx, adapter, run_dir, session_id, injected=None):return {"optimizer_tokens": float(opt_tokens or 0), "rendered_chars": len(rendered)}-def _fold_toolprune(ctx, adapter, run_dir, session_id, transcript=None, workdir=None):+def enough_evidence(sessions, prompts, cfg, high_risk=False):+ """Has a never-used tool gone unused long enough to be left out? Over `toolprune_evidence`+ prompts spread across at least `toolprune_min_sessions` sessions, or over+ `toolprune_sessions` sessions however short; a high-risk tool needs HIGH_RISK_FACTOR times+ the prompts and the sessions. Prompts let long sessions count for more; the floor keeps one+ session's prompts -- one piece of work -- from deciding alone (in recorded sessions that is+ what pruned tools a later session needed)."""+ k = live_adapter.HIGH_RISK_FACTOR if high_risk else 1+ return ((sessions >= cfg.toolprune_min_sessions and prompts >= cfg.toolprune_evidence * k)+ or sessions >= cfg.toolprune_sessions * k)+++def _fold_toolprune(ctx, adapter, run_dir, session_id, transcript=None, workdir=None,+ prompts=None):+ """Tool pruning's evidence: the sessions a tool went unused (`ToolingStage`'s streaks) and+ the prompts those sessions ran (`prompt_streaks`), weighed by `enough_evidence`. A list+ applies from a later session, since a running session keeps the tools it started with.+ `prompts` is None where the log does not say, and the session counts as one prompt."""if not denylist.supports(ctx.host.name) or ctx.config.toolprune == "off":return Noneif ctx.host.name == "codex":- return _fold_toolprune_codex(ctx, adapter, session_id, transcript, workdir)+ return _fold_toolprune_codex(ctx, adapter, session_id, transcript, workdir, prompts)host = ctx.host.namefrom tokeneconomizer.config import ToolingConfigfrom tokeneconomizer.toolprune import ToolingStagest = ctx.store.read("untool.json") or {}- stage = ToolingStage(ToolingConfig(evidence=ctx.config.toolprune_evidence), host, "agent",+ # The stage proposes never-used tools unused for the session floor; `enough_evidence`+ # decides among them.+ stage = ToolingStage(ToolingConfig(evidence=max(1, ctx.config.toolprune_min_sessions)),+ host, "agent",adapter=adapter, repo=ctx.repo.label, results_root=ctx.mech_dir,fresh=True)stage.used_tools = set(st.get("used_tools", []))stage.tool_unused_streaks = {k: int(v) for k, v in (st.get("streaks") or {}).items()}+ # A session from before prompts were counted counts as one prompt.+ prompt_streaks = {k: int(v) for k, v in+ (st.get("prompt_streaks") or stage.tool_unused_streaks).items()}stage.catalog_tools = set(st.get("catalog_tools", []))stage.catalog_desc = dict(st.get("catalog_desc") or {})applied = denylist.applied_for(ctx.store, ctx.repo.root, ctx.host.name)@@ -128,26 +151,34 @@ def _fold_toolprune(ctx, adapter, run_dir, session_id, transcript=None, workdir=# observed catalog; the source (or its absence) is recorded in untool.json instead.stage._warned_no_catalog = Truesrow = {}+ before = dict(stage.tool_unused_streaks)stage.after_task({"instance_id": session_id}, {"untooled_excluded": applied}, run_dir, srow)+ for t, n in stage.tool_unused_streaks.items():+ if n == 0:+ prompt_streaks[t] = 0 # used+ elif n == before.get(t, 0) + 1: # unused this session+ prompt_streaks[t] = prompt_streaks.get(t, 0) + (1 if prompts is None else prompts)sessions = int(st.get("sessions", 0)) + 1- raw = stage.build_tools({"instance_id": f"after-{session_id}"}, sessions) or []+ raw = [t for t in stage.build_tools({"instance_id": f"after-{session_id}"}, sessions) or []+ if enough_evidence(stage.tool_unused_streaks.get(t, 0), prompt_streaks.get(t, 0),+ ctx.config, high_risk=t in live_adapter.HIGH_RISK)]doc = {"used_tools": sorted(stage.used_tools), "streaks": stage.tool_unused_streaks,+ "prompt_streaks": prompt_streaks,"catalog_tools": sorted(stage.catalog_tools), "catalog_desc": stage.catalog_desc,# A session run under an applied mask parses no catalog (the stage skips it, as the# masked list is not the full one); the last source still describes the catalog."catalog_source": srow.get("untool_catalog_source") or st.get("catalog_source"),- "sessions": sessions, "updated": time.time()}+ "sessions": sessions, "prompts": int(st.get("prompts", 0)) + (prompts or 0),+ "updated": time.time()}if host == "copilot":# Copilot's only persistent lever is a whole MCP server (see `denylist`).servers = trace.copilot_mcp_servers(ctx.env)proposal = [s for s in denylist.copilot_server_proposal(raw, stage.catalog_tools, stage.used_tools, servers)if s not in applied and s not in kept]- doc["servers"] = _server_summary(stage, servers)+ doc["servers"] = _server_summary(stage, servers, prompt_streaks)else:- proposal = [t for t in denylist.filter_proposal(raw, stage.tool_unused_streaks,- ctx.config.toolprune_evidence)- if t not in applied and t not in kept]+ proposal = [t for t in raw if t not in applied and t not in kept]doc["proposal"] = proposaldoc["new_in_proposal"] = sorted(set(proposal) - set(st.get("proposal", [])))if ctx.config.toolprune == "auto" and proposal and not ctx.config.shadow:@@ -159,32 +190,36 @@ def _fold_toolprune(ctx, adapter, run_dir, session_id, transcript=None, workdir="restored": restored}-def _server_summary(stage, servers):- """{server: {tools, used, unused_sessions}} -- the Copilot evidence at the unit it acts on."""+def _server_summary(stage, servers, prompt_streaks):+ """{server: {tools, used, unused_sessions, unused_prompts}} -- the Copilot evidence at the+ unit it acts on: a server's least-evidenced tool."""out = {}for t in stage.catalog_tools:s = trace.mcp_server_of(t, servers)if not s:continue- e = out.setdefault(s, {"tools": 0, "used": False, "unused_sessions": None})+ e = out.setdefault(s, {"tools": 0, "used": False, "unused_sessions": None,+ "unused_prompts": None})e["tools"] += 1e["used"] = e["used"] or t in stage.used_tools- n = int(stage.tool_unused_streaks.get(t, 0))- e["unused_sessions"] = n if e["unused_sessions"] is None else min(e["unused_sessions"], n)+ for key, n in (("unused_sessions", stage.tool_unused_streaks.get(t, 0)),+ ("unused_prompts", prompt_streaks.get(t, 0))):+ e[key] = int(n) if e[key] is None else min(e[key], int(n))return out-def _fold_toolprune_codex(ctx, adapter, session_id, transcript, workdir):+def _fold_toolprune_codex(ctx, adapter, session_id, transcript, workdir, prompts=None):"""Codex: the experiments' capability controller (`codex_untool.CodexUntoolProfile`, driventhrough `ToolingStage`) decides which packages went unused, over a trace that carries themodel's own `exec` calls (`trace.codex_capability_trace`). Two live adaptations:* the session ran under the capabilities the PERSON applied, not the profile's own mask(a plugin launches nothing), so the observation is recorded against the applied set;- * a package is proposed only once the profile has demoted it AND it went unused in- `toolprune_evidence` sessions (x3 for image/media, the high-risk package), with no use- ever -- the profile alone demotes collaboration after one unused task, which was right- for a sealed benchmark stream and is too quick for a person's sessions."""+ * a package is proposed only once the profile has demoted it AND it was loaded, and+ unused, long enough by `enough_evidence` (x3 for image/media, the high-risk package),+ with no use ever -- the profile alone demotes collaboration after one unused task, which+ was right for a sealed benchmark stream and is too quick for a person's sessions. The+ profile counts sessions; the prompts are counted here (`loaded_prompts`)."""from tokeneconomizer.config import ToolingConfigfrom tokeneconomizer.toolprune import ToolingStageif not transcript or not workdir:@@ -202,23 +237,34 @@ def _fold_toolprune_codex(ctx, adapter, session_id, transcript, workdir):applied = denylist.applied_for(ctx.store, ctx.repo.root, ctx.host.name)kept = set(denylist.kept_for(ctx.store, ctx.repo.root, ctx.host.name)) # given backstage.codex_profile._pending["logical_capabilities"] = sorted(applied)+ loaded_before = dict(stage.codex_profile.state.get("loaded_exposures") or {})srow = {}stage.after_task({"instance_id": session_id}, {}, run_dir, srow)state = stage.codex_profile.stateever_used = set(state.get("ever_completed") or ())loaded = state.get("loaded_exposures") or {}+ prior = ctx.store.read("untool.json") or {}+ # A session from before prompts were counted counts as one prompt.+ loaded_prompts = dict(prior.get("loaded_prompts") or+ {c: int(n) for c, n in loaded_before.items()})+ for c in denylist.CODEX_LIVE_CAPABILITIES:+ if int(loaded.get(c, 0)) > int(loaded_before.get(c, 0)): # loaded this session+ loaded_prompts[c] = int(loaded_prompts.get(c, 0)) + (1 if prompts is None+ else prompts)proposal = [c for c in denylist.CODEX_LIVE_CAPABILITIESif c in (state.get("current_pruned") or ()) and c not in ever_usedand c not in applied and c not in kept- and int(loaded.get(c, 0)) >= ctx.config.toolprune_evidence * (- live_adapter.HIGH_RISK_FACTOR if c in CODEX_HIGH_RISK else 1)]- prior = ctx.store.read("untool.json") or {}+ and enough_evidence(int(loaded.get(c, 0)), int(loaded_prompts.get(c, 0)),+ ctx.config, high_risk=c in CODEX_HIGH_RISK)]doc = {"sessions": int(state.get("tasks_completed") or 0), "catalog_source": "capabilities",+ "prompts": int(prior.get("prompts", 0)) + (prompts or 0),"capabilities": {c: {"loaded_sessions": int(loaded.get(c, 0)),+ "loaded_prompts": int(loaded_prompts.get(c, 0)),"unused_sessions": int((state.get("zero_attempt_exposures")or {}).get(c, 0)),"used": c in ever_used}for c in denylist.CODEX_LIVE_CAPABILITIES},+ "loaded_prompts": loaded_prompts,"proposal": proposal, "updated": time.time(),"new_in_proposal": sorted(set(proposal) - set(prior.get("proposal", [])))}if ctx.config.toolprune == "auto" and proposal and not ctx.config.shadow:@@ -324,6 +370,11 @@ def fold_session(ctx, session_id, transcript_path=None):workdir = tempfile.mkdtemp(prefix="te-fold-")try:trace_path = ctx.host.mechanism_trace(transcript, workdir) if transcript else None+ try:+ prompts = ctx.host.prompt_count(transcript) if transcript else None+ except (OSError, ValueError):+ prompts = None+ record["prompts"] = promptsif not trace_path:record["error"] = "transcript unreadable"else:@@ -333,8 +384,10 @@ def fold_session(ctx, session_id, transcript_path=None):with ctx.store.locked("mechanisms"):inj = _fold_microskills(ctx, adapter, run_dir, session_id,injected=sess.get("injected_ids"))- unt = _fold_toolprune(ctx, adapter, run_dir, session_id,- transcript=transcript, workdir=workdir)+ # A session that ran no prompt is no evidence about the tools.+ unt = None if prompts == 0 else _fold_toolprune(+ ctx, adapter, run_dir, session_id, transcript=transcript,+ workdir=workdir, prompts=prompts)record["injection"] = injrecord["untool"] = untif unt and unt.get("pending"):diff --git a/tokeneconomizer/plugin/hosts.py b/tokeneconomizer/plugin/hosts.pyindex df7aa32..51e2528 100644--- a/tokeneconomizer/plugin/hosts.py+++ b/tokeneconomizer/plugin/hosts.py@@ -180,6 +180,10 @@ class Host:def session_usage(self, transcript_path):raise NotImplementedError+ def prompt_count(self, transcript_path):+ """How many prompts the finished session ran; None where the log does not say."""+ return None+def settle(self, transcript_path):"""Wait, if this host needs to, until the finished session's log is complete."""@@ -248,6 +252,9 @@ class ClaudeCode(Host):def session_usage(self, transcript_path):return trace.claude_session_usage(transcript_path)+ def prompt_count(self, transcript_path):+ return trace.claude_prompt_count(transcript_path)+def tool_usage(self, trace_path):return trace.claude_tool_usage(os.path.realpath(trace_path))@@ -299,6 +306,9 @@ class Codex(Host):def session_usage(self, transcript_path):return trace.codex_session_usage(transcript_path)+ def prompt_count(self, transcript_path):+ return trace.codex_prompt_count(transcript_path)+class Pi(Host):"""Pi (earendil-works/pi), through the extension in `plugin/pi/`.@@ -331,6 +341,9 @@ class Pi(Host):n = trace.pi_session_to_claude(transcript_path, out)return out if n is not None else None+ def prompt_count(self, transcript_path):+ return trace.pi_prompt_count(transcript_path)+def session_usage(self, transcript_path):return trace.pi_session_usage(transcript_path)@@ -379,6 +392,9 @@ class Copilot(Host):def session_usage(self, transcript_path):return trace.copilot_session_usage(transcript_path)+ def prompt_count(self, transcript_path):+ return trace.copilot_prompt_count(transcript_path)+def settle(self, transcript_path, wait_s=15.0, fresh_s=60.0):# Copilot writes `session.shutdown` -- the event that carries usage -- just AFTER the# SessionEnd hook returns (observed order: hook.end, then session.shutdown), so thediff --git a/tokeneconomizer/trace/__init__.py b/tokeneconomizer/trace/__init__.pyindex e686d19..002a77a 100644--- a/tokeneconomizer/trace/__init__.py+++ b/tokeneconomizer/trace/__init__.py@@ -15,26 +15,28 @@ from .calls import RTK_PREFIX_RE, call_cwds, load_trajectory, strip_rtk_prefix,from .catalog import (CODEX_DEFAULT_TOOL_CATALOG, catalog_sources, parse_catalog,parse_catalog_with_source, parse_tool_catalog, parse_tool_catalog_claude,parse_tool_catalog_flat, parse_tool_names_copilot)-from .claude import (claude_calls, claude_deferred_tools, claude_session_usage,- claude_subagent_files, claude_tool_usage)-from .codex import (codex_capability_trace, codex_exec_calls, codex_rollout_to_exec,- codex_session_meta, codex_session_usage, rollout_item_to_exec)-from .copilot import (copilot_event_calls, copilot_mcp_servers, copilot_session_meta,- copilot_session_usage, copilot_turns, mcp_server_of)-from .pi import (pi_active_branch, pi_json_reply, pi_session_header, pi_session_to_claude,- pi_session_usage)+from .claude import (claude_calls, claude_deferred_tools, claude_prompt_count,+ claude_session_usage, claude_subagent_files, claude_tool_usage)+from .codex import (codex_capability_trace, codex_exec_calls, codex_prompt_count,+ codex_rollout_to_exec, codex_session_meta, codex_session_usage,+ rollout_item_to_exec)+from .copilot import (copilot_event_calls, copilot_mcp_servers, copilot_prompt_count,+ copilot_session_meta, copilot_session_usage, copilot_turns,+ mcp_server_of)+from .pi import (pi_active_branch, pi_json_reply, pi_prompt_count, pi_session_header,+ pi_session_to_claude, pi_session_usage)__all__ = ["RTK_PREFIX_RE", "call_cwds", "load_trajectory", "strip_rtk_prefix", "tool_usage","CODEX_DEFAULT_TOOL_CATALOG", "catalog_sources", "parse_catalog", "parse_catalog_with_source","parse_tool_catalog", "parse_tool_catalog_claude", "parse_tool_catalog_flat","parse_tool_names_copilot",- "claude_calls", "claude_deferred_tools", "claude_session_usage", "claude_subagent_files",- "claude_tool_usage",- "codex_capability_trace", "codex_exec_calls", "codex_rollout_to_exec", "codex_session_meta",- "codex_session_usage", "rollout_item_to_exec",- "copilot_event_calls", "copilot_mcp_servers", "copilot_session_meta",+ "claude_calls", "claude_deferred_tools", "claude_prompt_count", "claude_session_usage",+ "claude_subagent_files", "claude_tool_usage",+ "codex_capability_trace", "codex_exec_calls", "codex_prompt_count", "codex_rollout_to_exec",+ "codex_session_meta", "codex_session_usage", "rollout_item_to_exec",+ "copilot_event_calls", "copilot_mcp_servers", "copilot_prompt_count", "copilot_session_meta","copilot_session_usage", "copilot_turns", "mcp_server_of",- "pi_active_branch", "pi_json_reply", "pi_session_header", "pi_session_to_claude",- "pi_session_usage",+ "pi_active_branch", "pi_json_reply", "pi_prompt_count", "pi_session_header",+ "pi_session_to_claude", "pi_session_usage",]diff --git a/tokeneconomizer/trace/claude.py b/tokeneconomizer/trace/claude.pyindex 3ea1026..c8dbe99 100644--- a/tokeneconomizer/trace/claude.py+++ b/tokeneconomizer/trace/claude.py@@ -62,6 +62,36 @@ def claude_calls(lines):return out+def _is_prompt(ev):+ """Is this entry a prompt the person gave, typed or as a slash command? Its own `user`+ entries, less tool results, a local command's output, an interrupt marker, a compaction+ summary, the meta text Claude Code adds, and a subagent's."""+ if (ev.get("type") != "user" or ev.get("isMeta") or ev.get("isSidechain")+ or ev.get("isCompactSummary")):+ return False+ content = (ev.get("message") or {}).get("content")+ if isinstance(content, list):+ blocks = [b for b in content if isinstance(b, dict)]+ if any(b.get("type") == "tool_result" for b in blocks):+ return False+ content = " ".join(b.get("text") or "" for b in blocks if b.get("type") == "text")+ return isinstance(content, str) and not content.startswith(("<local-command",+ "[Request interrupted"))+++def claude_prompt_count(transcript_path):+ """How many prompts the person gave a Claude Code session (`_is_prompt`)."""+ n = 0+ for line in _lines(transcript_path):+ try:+ ev = json.loads(line)+ except ValueError:+ continue+ n += isinstance(ev, dict) and _is_prompt(ev)+ return n+++def claude_subagent_files(transcript_path):"""Subagent transcripts of a Claude Code session: `<dir>/<session-id>/subagents/*.jsonl`."""if not transcript_path:diff --git a/tokeneconomizer/trace/codex.py b/tokeneconomizer/trace/codex.pyindex 3afd549..5754091 100644--- a/tokeneconomizer/trace/codex.py+++ b/tokeneconomizer/trace/codex.py@@ -17,6 +17,7 @@ import reimport shlexfrom .jsonl import lines as _lines+from .jsonl import records as _records_SHELL_PATCH_FILE_RE = re.compile(r"^\*\*\* (?:Update|Add|Delete) File:\s*(.+?)\s*$", re.M)_SHELL_GIT_APPLY_PATH_RE = re.compile(r"^\+\+\+ b/(.+?)\s*$", re.M)@@ -197,6 +198,24 @@ def codex_capability_trace(rollout_path, out_path):return n+def codex_prompt_count(rollout_path):+ """How many prompts a Codex session ran: one turn each (`_turn_starts`)."""+ starts = _turn_starts(rollout_path)+ return sum(1 for d in _records(rollout_path) if starts(d))+++def _turn_starts(rollout_path):+ """What opens a turn in this rollout: its `task_started` events (0.163.0), or where it has+ none, its `turn_context` records."""+ def started(d):+ return (d.get("type") == "event_msg"+ and (d.get("payload") or {}).get("type") == "task_started")+ if any(started(d) for d in _records(rollout_path)):+ return started+ return lambda d: d.get("type") == "turn_context"+++def codex_session_meta(rollout_path):"""{cwd, model, cli_version} from a rollout's session_meta / turn_context records."""meta = {}diff --git a/tokeneconomizer/trace/copilot.py b/tokeneconomizer/trace/copilot.pyindex e21d5ed..bd305db 100644--- a/tokeneconomizer/trace/copilot.py+++ b/tokeneconomizer/trace/copilot.py@@ -133,6 +133,12 @@ def copilot_turns(path):return turns+def copilot_prompt_count(path):+ """How many prompts a Copilot session ran: one `user.message` event each."""+ return sum(1 for ev in _copilot_events(path) if ev.get("type") == "user.message")+++def copilot_mcp_servers(env=None):"""MCP server names a Copilot session can load: the built-in GitHub server plus everyserver in the user's `mcp-config.json`. Tool names are `<server>-<tool>`."""diff --git a/tokeneconomizer/trace/pi.py b/tokeneconomizer/trace/pi.pyindex 8f96fb3..0eb182e 100644--- a/tokeneconomizer/trace/pi.py+++ b/tokeneconomizer/trace/pi.py@@ -42,6 +42,12 @@ def pi_active_branch(entries):return list(reversed(path))+def pi_prompt_count(path):+ """How many prompts the conversation ran: `user` messages on its active branch."""+ return sum(1 for e in pi_active_branch(_pi_entries(path))+ if e.get("type") == "message" and (e.get("message") or {}).get("role") == "user")++def _pi_usage(u):if not isinstance(u, dict):return None