Skip to content

Experience-Driven Black-Box Cost Control for Coding Agents

Tool pruning: count prompts as well as sessions

Aaron508fb3c30694
diff --git a/README.md b/README.md
index 6a3e028..190d71d 100644
--- a/README.md
+++ b/README.md
@@ -41,8 +41,9 @@ what the next sessions in that repository pay for:
that `pytest` is missing here and `python -m pytest` works, that the test runner wants labels
without a `tests.` prefix -- are learned from recurring evidence and stated at the start of
the next session. Learning them makes no model call.
-- **Tool pruning.** Tools that stay loaded but go unused, session after session (five in a row
- by default), are left out of the next sessions through the agent's own settings. By default
+- **Tool pruning.** Tools that stay loaded but go unused (over ten prompts across at least two
+ sessions, or five sessions however short) are left out of the next sessions through the
+ agent's own settings. By default
this happens without asking you: the next session start says which tools were left out, and
one command gives them back. Set `toolprune=propose` to approve each deny list yourself instead.
- **Failure-loop intervention.** A turn that keeps repeating the same failing edit-and-test
@@ -203,17 +204,20 @@ command (`repos`, `doctor`, `config unset`, ...) and how each agent asks before
The three mechanisms are `microskills`, `toolprune` and `loopguard`; each takes `on` or `off`,
and each has numbers you can tune with the same `config set` command (in Claude Code's form here).
-Tool pruning is `auto` by default: once a tool has gone unused for `toolprune_evidence` sessions
-in a row (five), the plugin adds it to the deny list without asking, and it is left out from the
-next session on. It writes the list once your sessions in the repository have been idle for an
-hour (`toolprune_quiet_minutes`): a conversation resumed after its tools change writes its whole
-history to the prompt cache again instead of reading it, and after an idle hour that cache has
-expired anyway. With `toolprune=propose` the plugin only proposes the list, and nothing changes
-until you run `toolprune apply`.
+Tool pruning is `auto` by default: once a tool never used here has gone unused over
+`toolprune_evidence` prompts (ten) across at least `toolprune_min_sessions` sessions (two), or
+over `toolprune_sessions` sessions however short (five), the plugin adds it to the deny list
+without asking, and it is left out from the next session on. Prompts let long sessions count for
+more; the two-session floor keeps one session's work from deciding alone. It writes the list
+once your sessions in the repository have been idle for an hour (`toolprune_quiet_minutes`): a
+conversation resumed after its tools change writes its whole history to the prompt cache again
+instead of reading it, and after an idle hour that cache has expired anyway. With
+`toolprune=propose` the plugin only proposes the list, and nothing changes until you run
+`toolprune apply`.
```text
/tokeneconomizer:config set microskills_max_facts=2 # inject at most two facts
-/tokeneconomizer:config set toolprune_evidence=10 # wait for ten unused sessions
+/tokeneconomizer:config set toolprune_evidence=20 # wait for twenty unused prompts
/tokeneconomizer:config set loopguard_nudges=2 # redirect twice before telling you
```
diff --git a/docs/PLUGIN.md b/docs/PLUGIN.md
index 64e3059..bc9d376 100644
--- a/docs/PLUGIN.md
+++ b/docs/PLUGIN.md
@@ -54,7 +54,7 @@ Every hook fails open: any error becomes an empty output (no change) and a trace
| Learning signal | the official grader's `resolved` | the envelope learns from turns that **ended green**: the agent stopped on its own, was not stopped by the guard, and no verification after its last edit failed | There is no grader. Micro-skills and evidence pruning never needed one |
| Injection channel | a prefix on the task prompt | `SessionStart` `additionalContext` | Verified to reach the model (§3) |
| Tool removal | `--disallowed-tools` / Codex config at launch | a **deny list**, applied by default once the evidence forms it and the repository's sessions have been idle longer than a prompt cache lives (§3), and announced at the next session start (`toolprune=propose` waits for approval instead), written to the repository's own settings file — tools in `.claude/settings.local.json`, Codex capability packages in `.codex/config.toml`, Copilot MCP servers in `.github/copilot/settings.local.json` — effective from the **next** session | The plugin launches nothing. A session keeps the tools it started with, but a conversation resumed after a change sends the new ones, re-writing its cached prefix, hence the idle wait. Each agent's file can express a different unit (§3) |
-| Pruning candidates | `toolprune.tools.INAPPLICABLE_TOOLS` — orchestration, Q&A, plan mode, web — "inapplicable by construction" for a sealed headless task | nothing is inapplicable by construction; only **evidence** (loaded but unused for N sessions) proposes a tool, and a person's tools — AskUserQuestion, plan mode, sub-agents, the web — are protected | A person asks questions, plans and browses in a real session. A deny rule the plugin wrote for a tool that has since become protected is given back at the next fold |
+| Pruning candidates | `toolprune.tools.INAPPLICABLE_TOOLS` — orchestration, Q&A, plan mode, web — "inapplicable by construction" for a sealed headless task | nothing is inapplicable by construction; only **evidence** (never used, and unused over N prompts across two or more sessions, or over N sessions) proposes a tool, and a person's tools — AskUserQuestion, plan mode, sub-agents, the web — are protected | A person asks questions, plans and browses in a real session. A deny rule the plugin wrote for a tool that has since become protected is given back at the next fold |
| Network | every arm, baseline included, carries an answer-key guard — a `PreToolUse` hook and an `/etc/hosts` block refuse the benchmarks' upstream sources — and drops the web tools | **no guard**: no `PreToolUse` hook (on Pi, no `tool_call` handler), no blocked command or host; the web tools stay | A benchmark's answer is a fetch away. A real session has no answer key, and the web is part of the work |
| Nudge | kill the agent, resume it with a continuation prompt | a mid-turn reminder attached to the tool result | Different intervention, so its effect is not inherited from the experiments |
| Still stuck after the redirect | terminal abort, in autonomous runs | a one-line **notice** to the person, ending with how to stop the turn on that agent and in that mode (§3); the plugin never stops a turn, and `loopguard: stop` is read as `nudge` | Not every agent's hooks can stop a turn (Codex and Copilot ignore `continue: false`), and the person decides: the benchmark guard's own design note says "interactive callers should map it to a hand-off" |
@@ -418,6 +418,34 @@ OpenRouter (cache breakpoints, which move every turn, left out of the comparison
fact was retired on disk kept the original system message, and a new session took the
current facts.
+**Counting prompts, 2026-10-24** (the `experimental` branch). Can the mechanisms learn from the
+prompts of a session rather than from whole sessions, so a long first session counts for more?
+
+* Tool pruning: yes, with a floor. Replaying this machine's recorded Claude Code sessions
+ (12 projects, 61 sessions, 893 prompts) through the plugin's hooks and fold, against one tool
+ catalog, and counting a tool a later session used as a wrong prune:
+
+ | Rule | Projects with a list | Wrong prunes | Tools left out of the next session, summed |
+ |---|---|---|---|
+ | 5 sessions (`main`) | 7 | 1 | 340 |
+ | 10 prompts over 2+ sessions, or 5 sessions (now) | 8 | 1 | 512 |
+ | 10 prompts (`toolprune_min_sessions=1`) | 8 | 7 | 628 |
+
+ Ten prompts in one session pruned tools the next session needed (`ListAgents`,
+ `SendMessage`, a connector's tools): a session's prompts are one piece of work. Ten prompts
+ alone also never prunes for one-prompt sessions, hence the five-session fallback. Prompts
+ are counted from each agent's own log (`trace.*_prompt_count`); a session that ran none is
+ no evidence.
+* Micro-skills: no. Folded per prompt, a fact went active sooner, but a session meets its trap
+ once and its later prompts use the working form the conversation already shows, so the fact
+ fired in a quarter of the prompts that exercised it and never cleared the firing gate (40%
+ over at least 4): in a stream of four-prompt sessions it was never injected, where per
+ session it is injected at the fifth start. A replay that split prompts into sessions of
+ their own had shown the opposite only because each such start recorded the fact as given.
+ The facts still count sessions.
+* The loop guard already counts prompt-turns: each turn that ends green feeds the envelope at
+ its `Stop`, and the guard acts within the same session once it has five.
+
## 4. State and privacy
All state is local, under `$TOKENECONOMIZER_HOME` (default `~/.tokeneconomizer`), one directory
diff --git a/plugin/README.md b/plugin/README.md
index a42ce23..a9b424b 100644
--- a/plugin/README.md
+++ b/plugin/README.md
@@ -59,10 +59,10 @@ This records which tools a Claude Code session loads, with no model call. Do it
machine, and again after Claude Code updates. **Don't skip it:** without it the deny list sees
only the few tools Claude Code defers, not the ones every session loads.
-**4. Work normally.** See [What to expect](#what-to-expect). Once tools have gone unused for
-about five sessions, the plugin leaves them out of the next sessions without asking you (it
-writes the list once your sessions here have been idle for an hour), and the session start says
-so once:
+**4. Work normally.** See [What to expect](#what-to-expect). Once a tool has gone unused over
+ten prompts across two or more sessions (or five sessions, however short), the plugin leaves it
+out of the next sessions without asking you (it writes the list once your sessions here have
+been idle for an hour), and the session start says so once:
```text
/tokeneconomizer:toolprune # the tools it left out, and the evidence
@@ -126,8 +126,9 @@ read-only, so Codex asks first: choose **Yes, proceed**. From a terminal, inside
session).
**4. Work normally.** See [What to expect](#what-to-expect). Codex's deny list turns off the
-sub-agent (`collaboration`), `goals` and image (`image_media`) packages, each only after it went
-unused in every session where it was loaded; a package you used even once is kept. It is
+sub-agent (`collaboration`), `goals` and image (`image_media`) packages, each only once it has
+been loaded and unused long enough ([What to expect](#what-to-expect)); a package you used even
+once is kept. It is
written to the project's `.codex/config.toml`, which Codex reads **only in a trusted project**
(the session start that announces it says so otherwise), and never into one your repository
commits: that stays a proposal, and the session start says why. The web, asking you questions
@@ -192,7 +193,8 @@ With no model call. Do it once on this machine, and again after Copilot updates.
**4. Work normally.** See [What to expect](#what-to-expect). Copilot has no setting that removes a
single built-in tool, only one that turns off a whole MCP server, so the deny list turns off a
-server only when **none** of its tools was used in about five sessions: one of yours from
+server only when **none** of its tools was used long enough ([What to expect](#what-to-expect)):
+one of yours from
`~/.copilot/mcp-config.json`, or the built-in `github-mcp-server`, which loads only when you are
signed in to GitHub. It is written to `.github/copilot/settings.local.json`, and the next session
start says so; `/tokeneconomizer:toolprune restore` turns the server back on. `toolprune
@@ -289,8 +291,11 @@ from); `pi config` turns it off without removing it. Optional: `rm -rf ~/.tokene
adds facts says so once, in one line. A resumed session keeps the facts it started with, word
for word, so its cached prompt still matches. `microskills` shows the facts injected and those
forming.
-- **Deny list.** Applied without asking you, after about five sessions in which a tool, package
- or server went unused, from the next session on; that session's start says so once. It is
+- **Deny list.** Applied without asking you once a tool, package or server never used here has
+ gone unused over ten prompts across at least two sessions, or over five sessions however short
+ (three times as long for a few higher-risk tools), from the next session on; that session's
+ start says so once. Prompts let long sessions count for more, and the two-session floor keeps
+ one session's work from deciding alone. A session that ran no prompt counts for nothing. It is
written once the repository's sessions have been idle for an hour: a conversation resumed
after its tools change writes its whole history to the prompt cache again instead of reading
it, and after an idle hour that cache has expired anyway. `toolprune` shows a list still
@@ -378,7 +383,7 @@ and are shared by every agent working in it. The examples use Claude Code's form
```text
/tokeneconomizer:config set microskills_max_facts=2 # inject at most two facts
-/tokeneconomizer:config set toolprune_evidence=10 # wait for ten unused sessions
+/tokeneconomizer:config set toolprune_evidence=20 # wait for twenty unused prompts
/tokeneconomizer:config set loopguard_nudges=2 loopguard.min_turns=30
/tokeneconomizer:config unset loopguard.min_turns # back to the default
```
@@ -388,7 +393,9 @@ and are shared by every agent working in it. The examples use Claude Code's form
| `microskills_max_facts` | `4` | the most facts injected at session start |
| `microskills_max_tokens` | `150` | the size of the injected block, in tokens (about 4 characters each) |
| `microskills_propose_every` | `5` | sessions between runs of the paid fact proposer |
-| `toolprune_evidence` | `5` | sessions a tool must go unused before it is left out (three times as long for a few higher-risk tools; the web is never proposed) |
+| `toolprune_evidence` | `10` | prompts a never-used tool must go unused, over at least `toolprune_min_sessions` sessions, before it is left out (three times as many for a few higher-risk tools; the web is never proposed) |
+| `toolprune_min_sessions` | `2` | sessions those prompts must span; `1` lets a single session's prompts decide |
+| `toolprune_sessions` | `5` | or this many sessions, however few prompts they ran |
| `toolprune_quiet_minutes` | `60` | how long the repository's sessions must have been idle before `auto` writes a deny list, so that no conversation resumed while its prompt cache is warm loses it; `0` writes at once |
| `loopguard_aggr` | `default` | the guard's threshold preset: `conservative`, `default` or `aggressive` |
| `loopguard_nudges` | `1` | redirects per prompt-turn before the guard only notifies you |
diff --git a/tests/test_live_core.py b/tests/test_live_core.py
index 3472d67..5bcc679 100644
--- a/tests/test_live_core.py
+++ b/tests/test_live_core.py
@@ -597,9 +597,11 @@ def test_a_job_that_never_answers_stops_blocking_the_next_one():
# ---- Copilot session logs ------------------------------------------------------------------
-def copilot_events(path, calls, *, shutdown=True, model="gpt-5.4", cwd="/w/repo", sid="cp1"):
- """A Copilot `session-state/<id>/events.jsonl`: session.start, one user message, the tool
- calls as execution_start/complete pairs, and (unless killed) a session.shutdown."""
+def copilot_events(path, calls, *, shutdown=True, model="gpt-5.4", cwd="/w/repo", sid="cp1",
+ prompts=1):
+ """A Copilot `session-state/<id>/events.jsonl`: session.start, `prompts` user messages (the
+ tool calls, as execution_start/complete pairs, in the first), and (unless killed) a
+ session.shutdown."""
ev = [{"type": "session.start", "data": {"sessionId": sid, "copilotVersion": "1.0.83",
"selectedModel": model, "context": {"cwd": cwd}}},
{"type": "user.message", "data": {"content": "fix it", "turnId": "0"}}]
@@ -609,6 +611,8 @@ def copilot_events(path, calls, *, shutdown=True, model="gpt-5.4", cwd="/w/repo"
ev.append({"type": "tool.execution_complete", "data": {
"toolCallId": f"c{n}", "success": ok, "turnId": "0",
"result": {"content": text, "detailedContent": text}}})
+ for k in range(1, prompts):
+ ev.append({"type": "user.message", "data": {"content": "and then", "turnId": str(k)}})
if shutdown:
ev.append({"type": "session.shutdown", "data": {
"modelMetrics": {model: {"requests": {"count": 4}, "usage": {
diff --git a/tests/test_live_hooks.py b/tests/test_live_hooks.py
index 33ef8cb..ca8f279 100644
--- a/tests/test_live_hooks.py
+++ b/tests/test_live_hooks.py
@@ -139,6 +139,34 @@ def test_a_learned_fact_reaches_the_next_session_and_is_announced_once(env, repo
assert starts[5]["hookSpecificOutput"]["additionalContext"] == ctx
+def multi_prompt_trap_session(path, repo, i, prompts=4, every=False):
+ """A session of `prompts` prompts that meets the missing `pytest` at its first test run (or,
+ with `every`, at each): later prompts use the working form, as the conversation shows it."""
+ def prompt(k):
+ edit = ("Edit", {"file_path": f"{repo}/pkg/m{i}_{k}.py", "old_string": "a",
+ "new_string": "b"}, "ok", False)
+ known = ("Bash", {"command": "python -m pytest -q tests/test_a.py"}, "1 passed in 0.1s",
+ False)
+ trap = ("Bash", {"command": "pytest -q tests/test_a.py"},
+ "bash: pytest: command not found", True)
+ return (f"fix bug {i}.{k}", [edit, trap, known] if every or k == 0 else [edit, known])
+ return write_session(path, [prompt(k) for k in range(prompts)])
+
+
+def test_facts_learn_per_session_however_many_prompts_it_ran(env, repo, tmp_path):
+ """Unlike tool pruning, the facts count sessions, not prompts. Within a session the agent
+ already knows the fix from its own earlier failure, so its later prompts would count as
+ exercising the fact without the failure: per prompt, a trap met once a session fires in a
+ quarter of them and never clears the firing gate. Four-prompt sessions reach the fact at the
+ fifth start, as one-prompt sessions do (above)."""
+ claim = "`pytest …` fails here with command not found; `python -m pytest …` works."
+ starts = [run_session(env, repo, f"s{i}",
+ multi_prompt_trap_session(str(tmp_path / f"s{i}.jsonl"), repo, i))
+ for i in range(6)]
+ assert all(s == {} for s in starts[:4])
+ assert claim in starts[4]["hookSpecificOutput"]["additionalContext"]
+
+
def test_resume_does_not_inject_again_and_shadow_never_injects(env, repo, tmp_path):
for i in range(5):
run_session(env, repo, f"s{i}", trap_session(str(tmp_path / f"s{i}.jsonl"), repo, i))
@@ -378,7 +406,7 @@ def test_with_propose_unused_tools_are_proposed_but_nothing_changes_until_applie
for i in range(5):
t = write_session(str(tmp_path / f"u{i}.jsonl"), [("look", [
("Read", {"file_path": f"{repo}/a.py"}, "x", False),
- ("Bash", {"command": "ls"}, "a.py", False)])])
+ ("Bash", {"command": "ls"}, "a.py", False)]), ("and again", [])])
run_session(env, repo, f"u{i}", t)
st = store_for(env, repo)
prop = st.read("untool.json")["proposal"]
@@ -402,7 +430,7 @@ def test_by_default_unused_tools_are_removed_from_the_next_session_and_it_says_s
for i in range(5):
t = write_session(str(tmp_path / f"u{i}.jsonl"), [("look", [
("Read", {"file_path": f"{repo}/a.py"}, "x", False),
- ("Bash", {"command": "ls"}, "a.py", False)])])
+ ("Bash", {"command": "ls"}, "a.py", False)]), ("and again", [])])
run_session(env, repo, f"u{i}", t)
st = store_for(env, repo)
deny = json.load(open(denylist.settings_path(repo)))["permissions"]["deny"]
@@ -431,7 +459,7 @@ def test_tools_given_back_stay_back_until_named_again(env, repo, tmp_path, monke
def sessions(prefix, n):
for i in range(n):
t = write_session(str(tmp_path / f"{prefix}{i}.jsonl"), [("look", [
- ("Bash", {"command": "ls"}, "a.py", False)])])
+ ("Bash", {"command": "ls"}, "a.py", False)]), ("and again", [])])
run_session(env, repo, f"{prefix}{i}", t)
sessions("a", 5)
def deny():
@@ -449,6 +477,74 @@ def test_tools_given_back_stay_back_until_named_again(env, repo, tmp_path, monke
assert denylist.kept_for(store_for(env, repo), repo) == ["CronCreate"]
+def _prompts(env, repo, tmp_path, sid, n):
+ """One Claude Code session of `n` prompts; only the first calls a tool (Bash)."""
+ t = write_session(str(tmp_path / f"{sid}.jsonl"),
+ [(f"prompt {k}", [("Bash", {"command": "ls"}, "a.py", False)] if k == 0
+ else []) for k in range(n)])
+ run_session(env, repo, sid, t)
+
+
+def _another_repo(tmp_path, name):
+ r = tmp_path / name
+ (r / ".git").mkdir(parents=True)
+ (r / ".git" / "HEAD").write_text("ref: refs/heads/main\n")
+ return str(r)
+
+
+def test_ten_prompts_over_two_sessions_are_enough_and_one_sessions_are_not(env, repo, tmp_path):
+ """The evidence counts prompts as well as sessions: a never-used tool is left out after ten
+ prompts over at least two sessions. One session's prompts are one piece of work, so ten of
+ them in a single session are not enough -- unless `toolprune_min_sessions=1`."""
+ env = dict(env, TE_TOOLPRUNE="propose")
+ _catalog(env, ["Bash", "Read", "Workflow", "CronCreate"])
+ _prompts(env, repo, tmp_path, "long", 10)
+ st = store_for(env, repo)
+ unt = st.read("untool.json")
+ assert unt["proposal"] == []
+ assert (unt["streaks"]["Workflow"], unt["prompt_streaks"]["Workflow"]) == (1, 10)
+ _prompts(env, repo, tmp_path, "short", 1)
+ assert sorted(st.read("untool.json")["proposal"]) == ["CronCreate", "Workflow"] # not Read
+
+ one = dict(env, TE_TOOLPRUNE_MIN_SESSIONS="1")
+ nine, ten = _another_repo(tmp_path, "nine"), _another_repo(tmp_path, "ten")
+ _prompts(one, nine, tmp_path, "s9", 9)
+ _prompts(one, ten, tmp_path, "s10", 10)
+ assert store_for(one, nine).read("untool.json")["proposal"] == []
+ assert sorted(store_for(one, ten).read("untool.json")["proposal"]) == ["CronCreate", "Workflow"]
+
+
+def test_five_sessions_are_enough_however_short(env, repo, tmp_path):
+ env = dict(env, TE_TOOLPRUNE="propose")
+ _catalog(env, ["Bash", "Workflow"])
+ for i in range(4):
+ _prompts(env, repo, tmp_path, f"s{i}", 1)
+ assert store_for(env, repo).read("untool.json")["proposal"] == []
+ _prompts(env, repo, tmp_path, "s4", 1)
+ assert store_for(env, repo).read("untool.json")["proposal"] == ["Workflow"]
+
+
+def test_a_session_without_prompts_is_no_evidence(env, repo, tmp_path):
+ env = dict(env, TE_TOOLPRUNE="propose")
+ _catalog(env, ["Bash", "Workflow"])
+ _prompts(env, repo, tmp_path, "some", 4)
+ _prompts(env, repo, tmp_path, "none", 0)
+ unt = store_for(env, repo).read("untool.json")
+ assert (unt["sessions"], unt["streaks"]["Workflow"], unt["prompt_streaks"]["Workflow"]) == (
+ 1, 1, 4)
+
+
+def test_a_high_risk_tool_needs_three_times_the_prompts(env, repo, tmp_path):
+ env = dict(env, TE_TOOLPRUNE="propose")
+ _catalog(env, ["Bash", "Workflow", "NotebookEdit"])
+ _prompts(env, repo, tmp_path, "a", 20)
+ _prompts(env, repo, tmp_path, "b", 9)
+ assert store_for(env, repo).read("untool.json")["proposal"] == ["Workflow"]
+ _prompts(env, repo, tmp_path, "c", 1)
+ assert sorted(store_for(env, repo).read("untool.json")["proposal"]) == [
+ "NotebookEdit", "Workflow"]
+
+
def _pending_repo(env, repo, tmp_path, quiet="60"):
"""Five sessions that leave three tools unused, under the default wait before writing."""
env = dict(env, TE_TOOLPRUNE_QUIET_MINUTES=quiet)
@@ -457,7 +553,7 @@ def _pending_repo(env, repo, tmp_path, quiet="60"):
for i in range(5):
t = write_session(str(tmp_path / f"u{i}.jsonl"), [("look", [
("Read", {"file_path": f"{repo}/a.py"}, "x", False),
- ("Bash", {"command": "ls"}, "a.py", False)])])
+ ("Bash", {"command": "ls"}, "a.py", False)]), ("and again", [])])
run_session(env, repo, f"u{i}", t)
return env, LiveContext.build(hosts.ClaudeCode(), repo, env)
@@ -751,12 +847,13 @@ GITHUB_TOOLS = ["github-mcp-server-list_issues", "github-mcp-server-get_commit",
"github-mcp-server-search_code"]
-def copilot_session(env, repo, tmp_path, sid, calls, monkeypatch, source="new"):
+def copilot_session(env, repo, tmp_path, sid, calls, monkeypatch, source="new", prompts=1):
"""One Copilot session, as its hook file delivers it: no transcript path before Stop (the
log's place is fixed by the session id), Claude tool names, a top-level context answer."""
home = tmp_path / "copilot-home"
monkeypatch.setenv("COPILOT_HOME", str(home))
- copilot_events(str(home / "session-state" / sid / "events.jsonl"), calls, cwd=repo, sid=sid)
+ copilot_events(str(home / "session-state" / sid / "events.jsonl"), calls, cwd=repo, sid=sid,
+ prompts=prompts)
base = {"session_id": sid, "cwd": repo, "timestamp": "2026-10-08T04:22:15.157Z"}
start = call(env, dict(base, hook_event_name="SessionStart", source=source), "copilot")
call(env, dict(base, hook_event_name="UserPromptSubmit", prompt="fix it"), "copilot")
@@ -821,25 +918,39 @@ def test_copilot_proposes_only_whole_unused_mcp_servers(env, repo, tmp_path, mon
for i in range(5):
copilot_session(env, repo, tmp_path, f"u{i}", [
("bash", {"command": "ls"}, "a.py\n<shellId: 0 completed with exit code 0>", True)],
- monkeypatch)
+ monkeypatch, prompts=2)
st = store_for(env, repo, "copilot")
unt = st.read("untool.json")
# builtins went unused too, but Copilot has no persistent per-tool lever: only the server
assert unt["proposal"] == ["github-mcp-server"]
assert unt["servers"]["github-mcp-server"] == {"tools": 3, "used": False,
- "unused_sessions": 5}
+ "unused_sessions": 5, "unused_prompts": 10}
assert not os.path.exists(denylist.settings_path(repo, "copilot"))
(notice,) = copilot_session(env, repo, tmp_path, "n", [], monkeypatch)["progress"]
assert "1 MCP server(s) went unused" in notice and "/tokeneconomizer:toolprune" in notice
+def test_two_copilot_sessions_of_five_prompts_are_enough_to_propose_a_server(
+ env, repo, tmp_path, monkeypatch):
+ env = dict(env, TE_TOOLPRUNE="propose")
+ _copilot_catalog(env, ["bash", "view"] + GITHUB_TOOLS)
+ for sid in ("u1", "u2"):
+ copilot_session(env, repo, tmp_path, sid, [
+ ("bash", {"command": "ls"}, "a.py\n<shellId: 0 completed with exit code 0>", True)],
+ monkeypatch, prompts=5)
+ unt = store_for(env, repo, "copilot").read("untool.json")
+ assert unt["proposal"] == ["github-mcp-server"]
+ assert unt["servers"]["github-mcp-server"]["unused_sessions"] == 2
+ assert unt["servers"]["github-mcp-server"]["unused_prompts"] == 10
+
+
def test_by_default_copilot_turns_off_the_unused_server_and_says_so(env, repo, tmp_path,
monkeypatch):
_copilot_catalog(env, ["bash", "view"] + GITHUB_TOOLS)
for i in range(5):
copilot_session(env, repo, tmp_path, f"u{i}", [
("bash", {"command": "ls"}, "a.py\n<shellId: 0 completed with exit code 0>", True)],
- monkeypatch)
+ monkeypatch, prompts=2)
settings = json.load(open(denylist.settings_path(repo, "copilot")))
assert settings["disabledMcpServers"] == ["github-mcp-server"]
(notice,) = copilot_session(env, repo, tmp_path, "n", [], monkeypatch)["progress"]
@@ -875,18 +986,24 @@ def test_copilot_apply_and_restore_touch_only_the_plugins_own_servers(repo, tmp_
# ---- Codex capability untooling ----------------------------------------------------------
-def codex_rollout(path, repo, i, exec_inputs=()):
- """A Codex rollout: one shell command and, per `exec_inputs`, one code-mode exec call."""
+def codex_rollout(path, repo, i, exec_inputs=(), prompts=1):
+ """A Codex rollout of `prompts` turns: in the first, one shell command and, per
+ `exec_inputs`, one code-mode exec call."""
recs = [{"type": "session_meta", "payload": {"id": f"thread-{i}", "cwd": repo,
- "cli_version": "0.153.1"}},
- {"type": "turn_context", "payload": {"turn_id": "t1", "model": "gpt-5.6-sol"}},
- {"type": "event_msg", "payload": {"type": "item_completed", "turn_id": "t1", "item": {
- "type": "CommandExecution", "id": f"c{i}", "command": ["bash", "-lc", "ls"],
- "aggregated_output": "a.py", "exit_code": 0, "status": "completed"}}}]
- for k, src in enumerate(exec_inputs):
- recs.append({"type": "response_item", "payload": {
- "type": "custom_tool_call", "name": "exec", "call_id": f"x{i}-{k}",
- "status": "completed", "input": src}})
+ "cli_version": "0.153.1"}}]
+ for k in range(1, prompts + 1):
+ recs += [{"type": "turn_context", "payload": {"turn_id": f"t{k}", "model": "gpt-5.6-sol"}},
+ {"type": "event_msg", "payload": {"type": "task_started", "turn_id": f"t{k}"}}]
+ if k > 1:
+ continue
+ recs.append({"type": "event_msg", "payload": {"type": "item_completed", "turn_id": "t1",
+ "item": {
+ "type": "CommandExecution", "id": f"c{i}", "command": ["bash", "-lc", "ls"],
+ "aggregated_output": "a.py", "exit_code": 0, "status": "completed"}}})
+ for n, src in enumerate(exec_inputs):
+ recs.append({"type": "response_item", "payload": {
+ "type": "custom_tool_call", "name": "exec", "call_id": f"x{i}-{n}",
+ "status": "completed", "input": src}})
recs.append({"type": "event_msg", "payload": {"type": "token_count", "info": {
"total_token_usage": {"input_tokens": 3000, "cached_input_tokens": 2000,
"cache_write_input_tokens": 0, "output_tokens": 90,
@@ -900,24 +1017,30 @@ def test_codex_proposes_capabilities_it_never_saw_used(env, repo, tmp_path):
env = dict(env, TE_TOOLPRUNE="propose")
for i in range(5):
uses = ["await tools.view_image({path: 'a.png'})"] if i == 2 else []
- t = codex_rollout(str(tmp_path / f"rollout-{i}.jsonl"), repo, i, uses)
+ t = codex_rollout(str(tmp_path / f"rollout-{i}.jsonl"), repo, i, uses, prompts=2)
run_session(env, repo, f"cx{i}", t, host="codex")
unt = store_for(env, repo, "codex").read("untool.json")
- # collaboration and goals went unused in all five; image was used once and is kept
+ # collaboration and goals went unused over all ten prompts; image was used once: kept
assert unt["proposal"] == ["collaboration", "goals"]
assert unt["capabilities"]["image_media"]["used"] is True
- assert unt["capabilities"]["goals"] == {"loaded_sessions": 5, "unused_sessions": 5,
- "used": False}
+ assert unt["capabilities"]["goals"] == {"loaded_sessions": 5, "loaded_prompts": 10,
+ "unused_sessions": 5, "used": False}
+ assert (unt["sessions"], unt["prompts"]) == (5, 10)
assert not os.path.exists(denylist.settings_path(repo, "codex")) # nothing written yet
def test_codex_needs_the_full_evidence_before_proposing(env, repo, tmp_path):
"""The experiments' controller demotes collaboration after ONE unused task; a person's
- sessions need `untool_evidence` of them."""
- for i in range(4):
+ sessions need `toolprune_evidence` prompts of it, however they split into sessions."""
+ env = dict(env, TE_TOOLPRUNE="propose")
+ for i, prompts in enumerate((3, 3, 3)):
run_session(env, repo, f"cx{i}", codex_rollout(str(tmp_path / f"rollout-{i}.jsonl"),
- repo, i), host="codex")
- assert store_for(env, repo, "codex").read("untool.json")["proposal"] == []
+ repo, i, prompts=prompts), host="codex")
+ assert store_for(env, repo, "codex").read("untool.json")["proposal"] == [] # nine
+ run_session(env, repo, "cx3", codex_rollout(str(tmp_path / "rollout-3.jsonl"), repo, 3),
+ host="codex")
+ assert store_for(env, repo, "codex").read("untool.json")["proposal"] == [
+ "collaboration", "goals"] # ten
@pytest.fixture
@@ -989,7 +1112,7 @@ def test_a_committed_codex_config_is_never_edited(tmp_path, codex_accepts):
def _codex_sessions(env, repo, tmp_path, n=5):
for i in range(n):
- t = codex_rollout(str(tmp_path / f"rollout-{i}.jsonl"), repo, i)
+ t = codex_rollout(str(tmp_path / f"rollout-{i}.jsonl"), repo, i, prompts=2)
run_session(env, repo, f"cx{i}", t, host="codex")
diff --git a/tests/test_trace.py b/tests/test_trace.py
index a193016..94a3bd1 100644
--- a/tests/test_trace.py
+++ b/tests/test_trace.py
@@ -177,3 +177,58 @@ def test_single_attempt_domain_collapses_to_resolved():
"""SWE-bench rows carry no attempt fields; pass@1 and pass@N are the same number."""
assert _p1p2({"resolved": True}) == (True, True)
assert _p1p2({"resolved": False}) == (False, False)
+
+
+def _jsonl(path, records):
+ import json
+ with open(path, "w") as fh:
+ fh.write("\n".join(json.dumps(r) for r in records) + "\n")
+ return str(path)
+
+
+def test_a_claude_session_counts_the_prompts_the_person_gave(tmp_path):
+ """Typed prompts and slash commands; not tool results, a local command's output, an
+ interrupt marker, a compaction summary, meta text, or a subagent's prompts."""
+ def user(content, **flags):
+ return dict({"type": "user", "message": {"role": "user", "content": content}}, **flags)
+ p = _jsonl(tmp_path / "c.jsonl", [
+ user("fix the test"),
+ user([{"type": "tool_result", "tool_use_id": "t1", "content": "ok"}]),
+ user([{"type": "text", "text": "and the docs"}, {"type": "image", "source": {}}]),
+ user("<command-name>/tokeneconomizer:status</command-name>"),
+ user("<local-command-stdout>done</local-command-stdout>"),
+ user("[Request interrupted by user]"),
+ user("Caveat: the messages below were generated by the user", isMeta=True),
+ user("This session is being continued...", isCompactSummary=True),
+ user("subagent task", isSidechain=True),
+ {"type": "assistant", "message": {"role": "assistant", "content": "done"}}])
+ assert trace.claude_prompt_count(p) == 3
+
+
+def test_a_codex_session_counts_one_prompt_per_turn(tmp_path):
+ turn = [{"type": "turn_context", "payload": {"turn_id": "t"}},
+ {"type": "event_msg", "payload": {"type": "task_started"}},
+ {"type": "response_item", "payload": {"type": "message", "role": "user"}}]
+ assert trace.codex_prompt_count(_jsonl(tmp_path / "r.jsonl", turn * 3)) == 3
+ # a rollout without task_started events counts its turn contexts
+ older = [r for r in turn if r["type"] != "event_msg"]
+ assert trace.codex_prompt_count(_jsonl(tmp_path / "o.jsonl", older * 2)) == 2
+
+
+def test_a_copilot_session_counts_its_user_messages(tmp_path):
+ p = _jsonl(tmp_path / "events.jsonl", [
+ {"type": "session.start", "data": {}}, {"type": "user.message", "data": {}},
+ {"type": "tool.execution_start", "data": {}}, {"type": "user.message", "data": {}}])
+ assert trace.copilot_prompt_count(p) == 2
+
+
+
+def test_a_pi_session_counts_the_prompts_on_its_active_branch(tmp_path):
+ """Pi keeps a conversation as a tree; a prompt on a branch the person left is not counted."""
+ def msg(eid, parent, role):
+ return {"type": "message", "id": eid, "parentId": parent, "message": {"role": role}}
+ p = _jsonl(tmp_path / "pi.jsonl", [
+ {"type": "session", "id": "s"}, msg("a", None, "user"), msg("b", "a", "assistant"),
+ msg("x", "b", "user"), msg("y", "x", "assistant"), # abandoned
+ msg("c", "b", "user"), msg("d", "c", "toolResult"), msg("e", "d", "assistant")])
+ assert trace.pi_prompt_count(p) == 2
diff --git a/tokeneconomizer/plugin/cli.py b/tokeneconomizer/plugin/cli.py
index 1e45efe..d1f7217 100644
--- a/tokeneconomizer/plugin/cli.py
+++ b/tokeneconomizer/plugin/cli.py
@@ -123,6 +123,7 @@ def cmd_status(args):
if ln.startswith("- ")),
"sessions_folded": inj.get("sessions", 0)},
"toolprune": {"mode": cfg.toolprune, "sessions": unt.get("sessions", 0),
+ "prompts": unt.get("prompts", 0),
"catalog_source": unt.get("catalog_source"),
"proposal": unt.get("proposal", []),
"applied": denylist.applied_for(ctx.store, ctx.repo.root,
@@ -161,7 +162,8 @@ def cmd_status(args):
print(" toolprune not available on this host")
else:
src = u["catalog_source"] or "none yet"
- print(f" toolprune {u['sessions']} sessions observed, catalog: {src}; "
+ print(f" toolprune {u['sessions']} sessions ({u['prompts']} prompts) observed, "
+ f"catalog: {src}; "
f"proposed {denylist.unit(ctx.host.name, plural=True)}: "
f"{', '.join(u['proposal']) or '-'}; applied: {', '.join(u['applied']) or '-'}")
sp = out["loopguard"]
@@ -190,8 +192,8 @@ def cmd_microskills(args):
# ---- toolprune -----------------------------------------------------------------------------
def _toolprune_show(ctx, unt):
host = ctx.host.name
- print(f"sessions observed: {unt.get('sessions', 0)} catalog: "
- f"{unt.get('catalog_source') or 'none yet'}")
+ print(f"sessions observed: {unt.get('sessions', 0)} ({unt.get('prompts', 0)} prompts) "
+ f"catalog: {unt.get('catalog_source') or 'none yet'}")
print(f"applied ({denylist.unit(host, plural=True)} this plugin added to "
f"{denylist.settings_path(ctx.repo.root, host)}): "
f"{', '.join(denylist.applied_for(ctx.store, ctx.repo.root, host)) or '-'}")
@@ -211,7 +213,8 @@ def _toolprune_show(ctx, unt):
if caps:
print("capabilities (removable from a project's .codex/config.toml):")
for c, e in sorted(caps.items()):
- print(f" {c:14s} loaded in {e['loaded_sessions']:3d} sessions, unused in "
+ print(f" {c:14s} loaded in {e['loaded_sessions']:3d} sessions "
+ f"({e.get('loaded_prompts', e['loaded_sessions']):3d} prompts), unused in "
f"{e['unused_sessions']:3d}{' USED (kept)' if e['used'] else ''}")
trusted = denylist.codex_project_trusted(ctx.repo.root)
if trusted is False:
@@ -223,15 +226,20 @@ def _toolprune_show(ctx, unt):
if servers:
print("MCP servers (Copilot can only remove a whole server):")
for name, e in sorted(servers.items()):
- state = "USED (kept)" if e["used"] else f"unused in {e['unused_sessions']} sessions"
+ state = ("USED (kept)" if e["used"] else
+ f"unused for {e.get('unused_sessions', 0)} sessions "
+ f"({e.get('unused_prompts', 0)} prompts)")
print(f" {name:24s} {e['tools']:3d} tools, {state}")
streaks = unt.get("streaks") or {}
if streaks and host == "claude":
- print("unused-session streaks (tools never used are candidates once over "
- f"{ctx.config.toolprune_evidence}):")
- for t, n in sorted(streaks.items(), key=lambda kv: -kv[1])[:30]:
+ cfg = ctx.config
+ prompts = unt.get("prompt_streaks") or {}
+ print(f"how long each tool has gone unused (a tool never used is left out after "
+ f"{cfg.toolprune_evidence} prompts over {cfg.toolprune_min_sessions}+ sessions, "
+ f"or after {cfg.toolprune_sessions} sessions):")
+ for t, n in sorted(streaks.items(), key=lambda kv: (-kv[1], -prompts.get(kv[0], 0)))[:30]:
used = "used" if t in set(unt.get("used_tools", [])) else ""
- print(f" {n:4d} {t} {used}")
+ print(f" {n:4d} sessions {prompts.get(t, n):5d} prompts {t} {used}")
if not unt.get("catalog_source") or unt.get("catalog_source") == "observed":
if os.path.exists(os.path.join(ctx.root, "hosts", host, "catalog.json")):
# Recorded, but a repository reads the catalog when its next session is folded.
@@ -424,6 +432,8 @@ def _config_view(cfg):
"toolprune": {"state": "off" if cfg.toolprune == "off" else "on",
"toolprune": d.pop("toolprune"),
"toolprune_evidence": d.pop("toolprune_evidence"),
+ "toolprune_min_sessions": d.pop("toolprune_min_sessions"),
+ "toolprune_sessions": d.pop("toolprune_sessions"),
"toolprune_quiet_minutes": d.pop("toolprune_quiet_minutes")},
"loopguard": {
"state": "off" if cfg.loopguard == "off" else "on",
diff --git a/tokeneconomizer/plugin/config.py b/tokeneconomizer/plugin/config.py
index d734ad5..a5015b3 100644
--- a/tokeneconomizer/plugin/config.py
+++ b/tokeneconomizer/plugin/config.py
@@ -26,7 +26,9 @@ person can tune by hand:
* microskills_max_facts, microskills_max_tokens -- caps on the injected block;
* microskills_propose_every -- sessions between paid proposer runs;
- * toolprune_evidence -- unused sessions before a tool is proposed;
+ * toolprune_evidence -- prompts a never-used tool must have stayed unused through,
+ over at least toolprune_min_sessions sessions, before it is
+ proposed -- or over toolprune_sessions sessions, however short;
* toolprune_quiet_minutes -- how long every session of the repository must have been idle
before `auto` writes a deny list (0: at once), so that no
conversation resumed with a warm prompt cache loses it;
@@ -127,7 +129,9 @@ class LiveConfig:
microskills_max_tokens: int = 150
microskills_propose_every: int = 5
toolprune: str = "auto"
- toolprune_evidence: int = 5
+ toolprune_evidence: int = 10
+ toolprune_min_sessions: int = 2
+ toolprune_sessions: int = 5
toolprune_quiet_minutes: int = 60
loopguard: str = "nudge"
loopguard_aggr: str = "default"
@@ -243,6 +247,8 @@ _ENV = {"TE_MODE": "mode", "TE_PAID": "paid", "TE_SHADOW": "shadow",
"TE_MICROSKILLS_PROPOSE_EVERY": "microskills_propose_every",
"TE_TOOLPRUNE": "toolprune", "TE_TOOLPRUNE_EVIDENCE": "toolprune_evidence",
"TE_TOOLPRUNE_QUIET_MINUTES": "toolprune_quiet_minutes",
+ "TE_TOOLPRUNE_MIN_SESSIONS": "toolprune_min_sessions",
+ "TE_TOOLPRUNE_SESSIONS": "toolprune_sessions",
"TE_LOOPGUARD": "loopguard", "TE_LOOPGUARD_AGGR": "loopguard_aggr",
"TE_LOOPGUARD_NUDGES": "loopguard_nudges",
"TE_REPOMAP": "repomap", "TE_REPOMAP_TOKENS": "repomap_tokens",
diff --git a/tokeneconomizer/plugin/denylist.py b/tokeneconomizer/plugin/denylist.py
index 67799bc..f778ae5 100644
--- a/tokeneconomizer/plugin/denylist.py
+++ b/tokeneconomizer/plugin/denylist.py
@@ -532,16 +532,6 @@ def restore(repo_root, store, tools=None, host="claude"):
return _restore_json_list(repo_root, store, tools, host)
-def filter_proposal(proposal, streaks, evidence):
- """Hold back high-risk tools until they have HIGH_RISK_FACTOR x the usual evidence."""
- out = []
- for t in proposal:
- need = evidence * (live_adapter.HIGH_RISK_FACTOR if t in live_adapter.HIGH_RISK else 1)
- if int(streaks.get(t, 0)) >= need:
- out.append(t)
- return out
-
-
def copilot_server_proposal(raw, catalog, used, servers):
"""MCP servers every one of whose catalog tools is in the evidence proposal `raw`."""
by_server = {}
diff --git a/tokeneconomizer/plugin/fold.py b/tokeneconomizer/plugin/fold.py
index d1e7768..7804408 100644
--- a/tokeneconomizer/plugin/fold.py
+++ b/tokeneconomizer/plugin/fold.py
@@ -91,20 +91,43 @@ def _fold_microskills(ctx, adapter, run_dir, session_id, injected=None):
return {"optimizer_tokens": float(opt_tokens or 0), "rendered_chars": len(rendered)}
-def _fold_toolprune(ctx, adapter, run_dir, session_id, transcript=None, workdir=None):
+def enough_evidence(sessions, prompts, cfg, high_risk=False):
+ """Has a never-used tool gone unused long enough to be left out? Over `toolprune_evidence`
+ prompts spread across at least `toolprune_min_sessions` sessions, or over
+ `toolprune_sessions` sessions however short; a high-risk tool needs HIGH_RISK_FACTOR times
+ the prompts and the sessions. Prompts let long sessions count for more; the floor keeps one
+ session's prompts -- one piece of work -- from deciding alone (in recorded sessions that is
+ what pruned tools a later session needed)."""
+ k = live_adapter.HIGH_RISK_FACTOR if high_risk else 1
+ return ((sessions >= cfg.toolprune_min_sessions and prompts >= cfg.toolprune_evidence * k)
+ or sessions >= cfg.toolprune_sessions * k)
+
+
+def _fold_toolprune(ctx, adapter, run_dir, session_id, transcript=None, workdir=None,
+ prompts=None):
+ """Tool pruning's evidence: the sessions a tool went unused (`ToolingStage`'s streaks) and
+ the prompts those sessions ran (`prompt_streaks`), weighed by `enough_evidence`. A list
+ applies from a later session, since a running session keeps the tools it started with.
+ `prompts` is None where the log does not say, and the session counts as one prompt."""
if not denylist.supports(ctx.host.name) or ctx.config.toolprune == "off":
return None
if ctx.host.name == "codex":
- return _fold_toolprune_codex(ctx, adapter, session_id, transcript, workdir)
+ return _fold_toolprune_codex(ctx, adapter, session_id, transcript, workdir, prompts)
host = ctx.host.name
from tokeneconomizer.config import ToolingConfig
from tokeneconomizer.toolprune import ToolingStage
st = ctx.store.read("untool.json") or {}
- stage = ToolingStage(ToolingConfig(evidence=ctx.config.toolprune_evidence), host, "agent",
+ # The stage proposes never-used tools unused for the session floor; `enough_evidence`
+ # decides among them.
+ stage = ToolingStage(ToolingConfig(evidence=max(1, ctx.config.toolprune_min_sessions)),
+ host, "agent",
adapter=adapter, repo=ctx.repo.label, results_root=ctx.mech_dir,
fresh=True)
stage.used_tools = set(st.get("used_tools", []))
stage.tool_unused_streaks = {k: int(v) for k, v in (st.get("streaks") or {}).items()}
+ # A session from before prompts were counted counts as one prompt.
+ prompt_streaks = {k: int(v) for k, v in
+ (st.get("prompt_streaks") or stage.tool_unused_streaks).items()}
stage.catalog_tools = set(st.get("catalog_tools", []))
stage.catalog_desc = dict(st.get("catalog_desc") or {})
applied = denylist.applied_for(ctx.store, ctx.repo.root, ctx.host.name)
@@ -128,26 +151,34 @@ def _fold_toolprune(ctx, adapter, run_dir, session_id, transcript=None, workdir=
# observed catalog; the source (or its absence) is recorded in untool.json instead.
stage._warned_no_catalog = True
srow = {}
+ before = dict(stage.tool_unused_streaks)
stage.after_task({"instance_id": session_id}, {"untooled_excluded": applied}, run_dir, srow)
+ for t, n in stage.tool_unused_streaks.items():
+ if n == 0:
+ prompt_streaks[t] = 0 # used
+ elif n == before.get(t, 0) + 1: # unused this session
+ prompt_streaks[t] = prompt_streaks.get(t, 0) + (1 if prompts is None else prompts)
sessions = int(st.get("sessions", 0)) + 1
- raw = stage.build_tools({"instance_id": f"after-{session_id}"}, sessions) or []
+ raw = [t for t in stage.build_tools({"instance_id": f"after-{session_id}"}, sessions) or []
+ if enough_evidence(stage.tool_unused_streaks.get(t, 0), prompt_streaks.get(t, 0),
+ ctx.config, high_risk=t in live_adapter.HIGH_RISK)]
doc = {"used_tools": sorted(stage.used_tools), "streaks": stage.tool_unused_streaks,
+ "prompt_streaks": prompt_streaks,
"catalog_tools": sorted(stage.catalog_tools), "catalog_desc": stage.catalog_desc,
# A session run under an applied mask parses no catalog (the stage skips it, as the
# masked list is not the full one); the last source still describes the catalog.
"catalog_source": srow.get("untool_catalog_source") or st.get("catalog_source"),
- "sessions": sessions, "updated": time.time()}
+ "sessions": sessions, "prompts": int(st.get("prompts", 0)) + (prompts or 0),
+ "updated": time.time()}
if host == "copilot":
# Copilot's only persistent lever is a whole MCP server (see `denylist`).
servers = trace.copilot_mcp_servers(ctx.env)
proposal = [s for s in denylist.copilot_server_proposal(
raw, stage.catalog_tools, stage.used_tools, servers)
if s not in applied and s not in kept]
- doc["servers"] = _server_summary(stage, servers)
+ doc["servers"] = _server_summary(stage, servers, prompt_streaks)
else:
- proposal = [t for t in denylist.filter_proposal(raw, stage.tool_unused_streaks,
- ctx.config.toolprune_evidence)
- if t not in applied and t not in kept]
+ proposal = [t for t in raw if t not in applied and t not in kept]
doc["proposal"] = proposal
doc["new_in_proposal"] = sorted(set(proposal) - set(st.get("proposal", [])))
if ctx.config.toolprune == "auto" and proposal and not ctx.config.shadow:
@@ -159,32 +190,36 @@ def _fold_toolprune(ctx, adapter, run_dir, session_id, transcript=None, workdir=
"restored": restored}
-def _server_summary(stage, servers):
- """{server: {tools, used, unused_sessions}} -- the Copilot evidence at the unit it acts on."""
+def _server_summary(stage, servers, prompt_streaks):
+ """{server: {tools, used, unused_sessions, unused_prompts}} -- the Copilot evidence at the
+ unit it acts on: a server's least-evidenced tool."""
out = {}
for t in stage.catalog_tools:
s = trace.mcp_server_of(t, servers)
if not s:
continue
- e = out.setdefault(s, {"tools": 0, "used": False, "unused_sessions": None})
+ e = out.setdefault(s, {"tools": 0, "used": False, "unused_sessions": None,
+ "unused_prompts": None})
e["tools"] += 1
e["used"] = e["used"] or t in stage.used_tools
- n = int(stage.tool_unused_streaks.get(t, 0))
- e["unused_sessions"] = n if e["unused_sessions"] is None else min(e["unused_sessions"], n)
+ for key, n in (("unused_sessions", stage.tool_unused_streaks.get(t, 0)),
+ ("unused_prompts", prompt_streaks.get(t, 0))):
+ e[key] = int(n) if e[key] is None else min(e[key], int(n))
return out
-def _fold_toolprune_codex(ctx, adapter, session_id, transcript, workdir):
+def _fold_toolprune_codex(ctx, adapter, session_id, transcript, workdir, prompts=None):
"""Codex: the experiments' capability controller (`codex_untool.CodexUntoolProfile`, driven
through `ToolingStage`) decides which packages went unused, over a trace that carries the
model's own `exec` calls (`trace.codex_capability_trace`). Two live adaptations:
* the session ran under the capabilities the PERSON applied, not the profile's own mask
(a plugin launches nothing), so the observation is recorded against the applied set;
- * a package is proposed only once the profile has demoted it AND it went unused in
- `toolprune_evidence` sessions (x3 for image/media, the high-risk package), with no use
- ever -- the profile alone demotes collaboration after one unused task, which was right
- for a sealed benchmark stream and is too quick for a person's sessions."""
+ * a package is proposed only once the profile has demoted it AND it was loaded, and
+ unused, long enough by `enough_evidence` (x3 for image/media, the high-risk package),
+ with no use ever -- the profile alone demotes collaboration after one unused task, which
+ was right for a sealed benchmark stream and is too quick for a person's sessions. The
+ profile counts sessions; the prompts are counted here (`loaded_prompts`)."""
from tokeneconomizer.config import ToolingConfig
from tokeneconomizer.toolprune import ToolingStage
if not transcript or not workdir:
@@ -202,23 +237,34 @@ def _fold_toolprune_codex(ctx, adapter, session_id, transcript, workdir):
applied = denylist.applied_for(ctx.store, ctx.repo.root, ctx.host.name)
kept = set(denylist.kept_for(ctx.store, ctx.repo.root, ctx.host.name)) # given back
stage.codex_profile._pending["logical_capabilities"] = sorted(applied)
+ loaded_before = dict(stage.codex_profile.state.get("loaded_exposures") or {})
srow = {}
stage.after_task({"instance_id": session_id}, {}, run_dir, srow)
state = stage.codex_profile.state
ever_used = set(state.get("ever_completed") or ())
loaded = state.get("loaded_exposures") or {}
+ prior = ctx.store.read("untool.json") or {}
+ # A session from before prompts were counted counts as one prompt.
+ loaded_prompts = dict(prior.get("loaded_prompts") or
+ {c: int(n) for c, n in loaded_before.items()})
+ for c in denylist.CODEX_LIVE_CAPABILITIES:
+ if int(loaded.get(c, 0)) > int(loaded_before.get(c, 0)): # loaded this session
+ loaded_prompts[c] = int(loaded_prompts.get(c, 0)) + (1 if prompts is None
+ else prompts)
proposal = [c for c in denylist.CODEX_LIVE_CAPABILITIES
if c in (state.get("current_pruned") or ()) and c not in ever_used
and c not in applied and c not in kept
- and int(loaded.get(c, 0)) >= ctx.config.toolprune_evidence * (
- live_adapter.HIGH_RISK_FACTOR if c in CODEX_HIGH_RISK else 1)]
- prior = ctx.store.read("untool.json") or {}
+ and enough_evidence(int(loaded.get(c, 0)), int(loaded_prompts.get(c, 0)),
+ ctx.config, high_risk=c in CODEX_HIGH_RISK)]
doc = {"sessions": int(state.get("tasks_completed") or 0), "catalog_source": "capabilities",
+ "prompts": int(prior.get("prompts", 0)) + (prompts or 0),
"capabilities": {c: {"loaded_sessions": int(loaded.get(c, 0)),
+ "loaded_prompts": int(loaded_prompts.get(c, 0)),
"unused_sessions": int((state.get("zero_attempt_exposures")
or {}).get(c, 0)),
"used": c in ever_used}
for c in denylist.CODEX_LIVE_CAPABILITIES},
+ "loaded_prompts": loaded_prompts,
"proposal": proposal, "updated": time.time(),
"new_in_proposal": sorted(set(proposal) - set(prior.get("proposal", [])))}
if ctx.config.toolprune == "auto" and proposal and not ctx.config.shadow:
@@ -324,6 +370,11 @@ def fold_session(ctx, session_id, transcript_path=None):
workdir = tempfile.mkdtemp(prefix="te-fold-")
try:
trace_path = ctx.host.mechanism_trace(transcript, workdir) if transcript else None
+ try:
+ prompts = ctx.host.prompt_count(transcript) if transcript else None
+ except (OSError, ValueError):
+ prompts = None
+ record["prompts"] = prompts
if not trace_path:
record["error"] = "transcript unreadable"
else:
@@ -333,8 +384,10 @@ def fold_session(ctx, session_id, transcript_path=None):
with ctx.store.locked("mechanisms"):
inj = _fold_microskills(ctx, adapter, run_dir, session_id,
injected=sess.get("injected_ids"))
- unt = _fold_toolprune(ctx, adapter, run_dir, session_id,
- transcript=transcript, workdir=workdir)
+ # A session that ran no prompt is no evidence about the tools.
+ unt = None if prompts == 0 else _fold_toolprune(
+ ctx, adapter, run_dir, session_id, transcript=transcript,
+ workdir=workdir, prompts=prompts)
record["injection"] = inj
record["untool"] = unt
if unt and unt.get("pending"):
diff --git a/tokeneconomizer/plugin/hosts.py b/tokeneconomizer/plugin/hosts.py
index df7aa32..51e2528 100644
--- a/tokeneconomizer/plugin/hosts.py
+++ b/tokeneconomizer/plugin/hosts.py
@@ -180,6 +180,10 @@ class Host:
def session_usage(self, transcript_path):
raise NotImplementedError
+ def prompt_count(self, transcript_path):
+ """How many prompts the finished session ran; None where the log does not say."""
+ return None
+
def settle(self, transcript_path):
"""Wait, if this host needs to, until the finished session's log is complete."""
@@ -248,6 +252,9 @@ class ClaudeCode(Host):
def session_usage(self, transcript_path):
return trace.claude_session_usage(transcript_path)
+ def prompt_count(self, transcript_path):
+ return trace.claude_prompt_count(transcript_path)
+
def tool_usage(self, trace_path):
return trace.claude_tool_usage(os.path.realpath(trace_path))
@@ -299,6 +306,9 @@ class Codex(Host):
def session_usage(self, transcript_path):
return trace.codex_session_usage(transcript_path)
+ def prompt_count(self, transcript_path):
+ return trace.codex_prompt_count(transcript_path)
+
class Pi(Host):
"""Pi (earendil-works/pi), through the extension in `plugin/pi/`.
@@ -331,6 +341,9 @@ class Pi(Host):
n = trace.pi_session_to_claude(transcript_path, out)
return out if n is not None else None
+ def prompt_count(self, transcript_path):
+ return trace.pi_prompt_count(transcript_path)
+
def session_usage(self, transcript_path):
return trace.pi_session_usage(transcript_path)
@@ -379,6 +392,9 @@ class Copilot(Host):
def session_usage(self, transcript_path):
return trace.copilot_session_usage(transcript_path)
+ def prompt_count(self, transcript_path):
+ return trace.copilot_prompt_count(transcript_path)
+
def settle(self, transcript_path, wait_s=15.0, fresh_s=60.0):
# Copilot writes `session.shutdown` -- the event that carries usage -- just AFTER the
# SessionEnd hook returns (observed order: hook.end, then session.shutdown), so the
diff --git a/tokeneconomizer/trace/__init__.py b/tokeneconomizer/trace/__init__.py
index e686d19..002a77a 100644
--- a/tokeneconomizer/trace/__init__.py
+++ b/tokeneconomizer/trace/__init__.py
@@ -15,26 +15,28 @@ from .calls import RTK_PREFIX_RE, call_cwds, load_trajectory, strip_rtk_prefix,
from .catalog import (CODEX_DEFAULT_TOOL_CATALOG, catalog_sources, parse_catalog,
parse_catalog_with_source, parse_tool_catalog, parse_tool_catalog_claude,
parse_tool_catalog_flat, parse_tool_names_copilot)
-from .claude import (claude_calls, claude_deferred_tools, claude_session_usage,
- claude_subagent_files, claude_tool_usage)
-from .codex import (codex_capability_trace, codex_exec_calls, codex_rollout_to_exec,
- codex_session_meta, codex_session_usage, rollout_item_to_exec)
-from .copilot import (copilot_event_calls, copilot_mcp_servers, copilot_session_meta,
- copilot_session_usage, copilot_turns, mcp_server_of)
-from .pi import (pi_active_branch, pi_json_reply, pi_session_header, pi_session_to_claude,
- pi_session_usage)
+from .claude import (claude_calls, claude_deferred_tools, claude_prompt_count,
+ claude_session_usage, claude_subagent_files, claude_tool_usage)
+from .codex import (codex_capability_trace, codex_exec_calls, codex_prompt_count,
+ codex_rollout_to_exec, codex_session_meta, codex_session_usage,
+ rollout_item_to_exec)
+from .copilot import (copilot_event_calls, copilot_mcp_servers, copilot_prompt_count,
+ copilot_session_meta, copilot_session_usage, copilot_turns,
+ mcp_server_of)
+from .pi import (pi_active_branch, pi_json_reply, pi_prompt_count, pi_session_header,
+ pi_session_to_claude, pi_session_usage)
__all__ = [
"RTK_PREFIX_RE", "call_cwds", "load_trajectory", "strip_rtk_prefix", "tool_usage",
"CODEX_DEFAULT_TOOL_CATALOG", "catalog_sources", "parse_catalog", "parse_catalog_with_source",
"parse_tool_catalog", "parse_tool_catalog_claude", "parse_tool_catalog_flat",
"parse_tool_names_copilot",
- "claude_calls", "claude_deferred_tools", "claude_session_usage", "claude_subagent_files",
- "claude_tool_usage",
- "codex_capability_trace", "codex_exec_calls", "codex_rollout_to_exec", "codex_session_meta",
- "codex_session_usage", "rollout_item_to_exec",
- "copilot_event_calls", "copilot_mcp_servers", "copilot_session_meta",
+ "claude_calls", "claude_deferred_tools", "claude_prompt_count", "claude_session_usage",
+ "claude_subagent_files", "claude_tool_usage",
+ "codex_capability_trace", "codex_exec_calls", "codex_prompt_count", "codex_rollout_to_exec",
+ "codex_session_meta", "codex_session_usage", "rollout_item_to_exec",
+ "copilot_event_calls", "copilot_mcp_servers", "copilot_prompt_count", "copilot_session_meta",
"copilot_session_usage", "copilot_turns", "mcp_server_of",
- "pi_active_branch", "pi_json_reply", "pi_session_header", "pi_session_to_claude",
- "pi_session_usage",
+ "pi_active_branch", "pi_json_reply", "pi_prompt_count", "pi_session_header",
+ "pi_session_to_claude", "pi_session_usage",
]
diff --git a/tokeneconomizer/trace/claude.py b/tokeneconomizer/trace/claude.py
index 3ea1026..c8dbe99 100644
--- a/tokeneconomizer/trace/claude.py
+++ b/tokeneconomizer/trace/claude.py
@@ -62,6 +62,36 @@ def claude_calls(lines):
return out
+def _is_prompt(ev):
+ """Is this entry a prompt the person gave, typed or as a slash command? Its own `user`
+ entries, less tool results, a local command's output, an interrupt marker, a compaction
+ summary, the meta text Claude Code adds, and a subagent's."""
+ if (ev.get("type") != "user" or ev.get("isMeta") or ev.get("isSidechain")
+ or ev.get("isCompactSummary")):
+ return False
+ content = (ev.get("message") or {}).get("content")
+ if isinstance(content, list):
+ blocks = [b for b in content if isinstance(b, dict)]
+ if any(b.get("type") == "tool_result" for b in blocks):
+ return False
+ content = " ".join(b.get("text") or "" for b in blocks if b.get("type") == "text")
+ return isinstance(content, str) and not content.startswith(("<local-command",
+ "[Request interrupted"))
+
+
+def claude_prompt_count(transcript_path):
+ """How many prompts the person gave a Claude Code session (`_is_prompt`)."""
+ n = 0
+ for line in _lines(transcript_path):
+ try:
+ ev = json.loads(line)
+ except ValueError:
+ continue
+ n += isinstance(ev, dict) and _is_prompt(ev)
+ return n
+
+
+
def claude_subagent_files(transcript_path):
"""Subagent transcripts of a Claude Code session: `<dir>/<session-id>/subagents/*.jsonl`."""
if not transcript_path:
diff --git a/tokeneconomizer/trace/codex.py b/tokeneconomizer/trace/codex.py
index 3afd549..5754091 100644
--- a/tokeneconomizer/trace/codex.py
+++ b/tokeneconomizer/trace/codex.py
@@ -17,6 +17,7 @@ import re
import shlex
from .jsonl import lines as _lines
+from .jsonl import records as _records
_SHELL_PATCH_FILE_RE = re.compile(r"^\*\*\* (?:Update|Add|Delete) File:\s*(.+?)\s*$", re.M)
_SHELL_GIT_APPLY_PATH_RE = re.compile(r"^\+\+\+ b/(.+?)\s*$", re.M)
@@ -197,6 +198,24 @@ def codex_capability_trace(rollout_path, out_path):
return n
+def codex_prompt_count(rollout_path):
+ """How many prompts a Codex session ran: one turn each (`_turn_starts`)."""
+ starts = _turn_starts(rollout_path)
+ return sum(1 for d in _records(rollout_path) if starts(d))
+
+
+def _turn_starts(rollout_path):
+ """What opens a turn in this rollout: its `task_started` events (0.163.0), or where it has
+ none, its `turn_context` records."""
+ def started(d):
+ return (d.get("type") == "event_msg"
+ and (d.get("payload") or {}).get("type") == "task_started")
+ if any(started(d) for d in _records(rollout_path)):
+ return started
+ return lambda d: d.get("type") == "turn_context"
+
+
+
def codex_session_meta(rollout_path):
"""{cwd, model, cli_version} from a rollout's session_meta / turn_context records."""
meta = {}
diff --git a/tokeneconomizer/trace/copilot.py b/tokeneconomizer/trace/copilot.py
index e21d5ed..bd305db 100644
--- a/tokeneconomizer/trace/copilot.py
+++ b/tokeneconomizer/trace/copilot.py
@@ -133,6 +133,12 @@ def copilot_turns(path):
return turns
+def copilot_prompt_count(path):
+ """How many prompts a Copilot session ran: one `user.message` event each."""
+ return sum(1 for ev in _copilot_events(path) if ev.get("type") == "user.message")
+
+
+
def copilot_mcp_servers(env=None):
"""MCP server names a Copilot session can load: the built-in GitHub server plus every
server in the user's `mcp-config.json`. Tool names are `<server>-<tool>`."""
diff --git a/tokeneconomizer/trace/pi.py b/tokeneconomizer/trace/pi.py
index 8f96fb3..0eb182e 100644
--- a/tokeneconomizer/trace/pi.py
+++ b/tokeneconomizer/trace/pi.py
@@ -42,6 +42,12 @@ def pi_active_branch(entries):
return list(reversed(path))
+def pi_prompt_count(path):
+ """How many prompts the conversation ran: `user` messages on its active branch."""
+ return sum(1 for e in pi_active_branch(_pi_entries(path))
+ if e.get("type") == "message" and (e.get("message") or {}).get("role") == "user")
+
+
def _pi_usage(u):
if not isinstance(u, dict):
return None