Skip to content

Experience-Driven Black-Box Cost Control for Coding Agents

default mode changed

Aaron46f6069a972b
diff --git a/README.md b/README.md
index 4f8c1f4..a8ee3a7 100644
--- a/README.md
+++ b/README.md
@@ -37,22 +37,28 @@ TokenEconomizer is a plugin for **Claude Code**, **Codex**, the **GitHub Copilot
**Pi**. It leaves the agent as it is, reads each finished session in a repository, and changes
what the next sessions in that repository pay for:
-- **Micro-skills.** Environment facts your sessions keep rediscovering by failing at them --
- that `pytest` is missing here and `python -m pytest` works, that the test runner wants labels
- without a `tests.` prefix -- are learned from recurring evidence and stated at the start of
- the next session. Learning them makes no model call.
+- **Micro-skills** (off until you turn it on). Repository knowledge your sessions keep needing,
+ in any language -- that a build step must run before anything else works, which spelling of a
+ tool works here (`npx tsc` where `tsc` is missing, `python -m pytest` where `pytest` is), an
+ environment variable a command needs, where generated files live -- is learned from finished
+ sessions and stated at the start of the next one. A fact is kept only while later sessions
+ confirm it. Turned on, it calls a model through your own agent every three sessions to propose
+ facts, and that cost is counted with your sessions'. How much it helps depends on the
+ repository's state and the tasks you run in it.
- **Tool pruning.** Tools that stay loaded but are never used (over ten prompts across at least
two sessions, or five sessions however short) are left out of the next sessions through the
agent's own settings. By default this happens without asking you: the next session start says
which tools were left out, and one command gives them back. Set `toolprune=propose` to approve
each deny list yourself instead.
-- **Failure-loop intervention.** A turn that keeps repeating the same failing edit-and-test
- cycle, well past how long this repository's turns usually take, gets one redirect. If it is
- still stuck after that, the plugin tells you how to stop it; it never stops a turn itself.
+- **Failure-loop intervention.** A turn that keeps hitting the same failures is put to a judge,
+ a model called through your own agent, and gets one redirect only if the judge finds it stuck.
+ If it is still stuck after that, the plugin tells you how to stop it; it never stops a turn
+ itself.
-Each mechanism can be switched off or tuned on its own. The plugin's actions are announced in
-one line, every hook fails open, and by default it makes no model calls of its own. What it learns
-stays on your machine, under `~/.tokeneconomizer`.
+Tool pruning and the loop guard are on by default, and each mechanism can be switched on or off
+and tuned on its own. The plugin's actions are announced in one line, and every hook fails open.
+By default its only model calls are the loop guard's judge, asked only about a turn that shows
+signs of looping. What it learns stays on your machine, under `~/.tokeneconomizer`.
## Results
@@ -194,7 +200,7 @@ agent; only how you type them differs:
| `toolprune` | the tools it left out, and the evidence (not on Pi) |
| `toolprune restore` | give them back; they will not be removed again |
| `config` | every setting in effect |
-| `config set loopguard=off` | turn a mechanism off here; add `--global` for every repository |
+| `config set microskills=on` | turn a mechanism on (or `=off`) here; add `--global` for every repository |
Inside an agent each command is one model turn; from a terminal it costs nothing. Every other
command (`repos`, `doctor`, `config unset`, ...) and how each agent asks before running one:
@@ -202,20 +208,21 @@ command (`repos`, `doctor`, `config unset`, ...) and how each agent asks before
## Settings
-The three mechanisms are `microskills`, `toolprune` and `loopguard`; each takes `on` or `off`,
-and each has numbers you can tune with the same `config set` command (in Claude Code's form here).
-Tool pruning is `auto` by default: once a tool never used here has gone unused over
-`toolprune_evidence` prompts (ten) across at least `toolprune_min_sessions` sessions (two), or
-over `toolprune_sessions` sessions however short (five), the plugin adds it to the deny list
-without asking, and it is left out from the next session on. Prompts let long sessions count for
-more; the two-session floor keeps one session's work from deciding alone. It writes the list
-once your sessions in the repository have been idle for an hour (`toolprune_quiet_minutes`): a
-conversation resumed after its tools change writes its whole history to the prompt cache again
-instead of reading it, and after an idle hour that cache has expired anyway. With
-`toolprune=propose` the plugin only proposes the list, and nothing changes until you run
-`toolprune apply`.
+The three mechanisms are `microskills` (off by default), `toolprune` and `loopguard` (both on);
+each takes `on` or `off`, and each has numbers you can tune with the same `config set` command
+(in Claude Code's form here). Tool pruning is `auto` by default: once a tool never used here has
+gone unused over `toolprune_evidence` prompts (ten) across at least `toolprune_min_sessions`
+sessions (two), or over `toolprune_sessions` sessions however short (five), the plugin adds it
+to the deny list without asking, and it is left out from the next session on. Prompts let long
+sessions count for more; the two-session floor keeps one session's work from deciding alone. It
+writes the list once your sessions in the repository have been idle for an hour
+(`toolprune_quiet_minutes`): a conversation resumed after its tools change writes its whole
+history to the prompt cache again instead of reading it, and after an idle hour that cache has
+expired anyway. With `toolprune=propose` the plugin only proposes the list, and nothing changes
+until you run `toolprune apply`.
```text
+/tokeneconomizer:config set microskills=on # learn and state repository facts
/tokeneconomizer:config set microskills_max_facts=2 # inject at most two facts
/tokeneconomizer:config set toolprune_evidence=20 # wait for twenty unused prompts
/tokeneconomizer:config set loopguard_nudges=2 # redirect twice before telling you
@@ -223,10 +230,11 @@ instead of reading it, and after an idle hour that cache has expired anyway. Wit
For one session only, set an environment variable when you start the agent:
`TE_DISABLE=1 claude` turns the whole plugin off, and `TE_LOOPGUARD=off codex` (likewise
-`TE_MICROSKILLS`, `TE_TOOLPRUNE`) one mechanism. The paid helpers -- an LLM fact proposer and a
-judge for the loop guard -- are off unless you set `paid=true`; they run through your own agent's
-CLI, on the same plan or login as your sessions. Every setting, its default and its environment
-variable: [Settings](plugin/README.md#settings).
+`TE_MICROSKILLS`, `TE_TOOLPRUNE`) one mechanism. The plugin's model calls -- the loop guard's
+judge, and micro-skills' proposer when it is on -- run through your own agent's CLI, on the same
+plan or login as your sessions; `microskills=envfacts` keeps only the facts found without a model
+call, and `loopguard=off` turns the judge off with the guard. Every setting, its default and its
+environment variable: [Settings](plugin/README.md#settings).
## What it does on each agent
@@ -253,8 +261,12 @@ a path through a symlink), the list stays a proposal and the session start says
### When a turn gets stuck
-The first time a turn looks stuck, the loop guard adds a redirect to the model's context. If the
-turn is still stuck after that, it shows you one line, ending with how to stop the turn:
+When a turn shows signs of looping -- the same failures repeating, or errors piling up -- the
+loop guard asks its judge, and only if the judge finds the turn stuck does it add a redirect to
+the model's context. (One exception: a turn far past this repository's usual length that also
+keeps churning the same edits through the same failures is redirected even if the judge
+disagrees.) If the turn is still stuck after that, it shows you one line, ending with how to stop
+the turn:
> TokenEconomizer: this turn still looks stuck after a redirect (envelope, 12 tool calls). Press
> Esc to interrupt it.
diff --git a/docs/PLUGIN.md b/docs/PLUGIN.md
index bc9d376..c90df62 100644
--- a/docs/PLUGIN.md
+++ b/docs/PLUGIN.md
@@ -21,10 +21,11 @@ the hooks run) and `plugin/` (the packaging for each agent). To install and use
| `SessionEnd` | Hands the session to a detached `te fold` and returns at once | microskills, toolprune, ledger |
`te fold` runs the **same classes the experiments ran** over the finished session's log:
-`PrefixStage` (envfacts, plus the paid proposer when allowed) and `ToolingStage` (evidence
-pruning). The loop guard (`loopguard.TurnGuard`) is the benchmark harness's `NudgeGuard` control
-flow — `SpiralState.update`, `assess_abort`, `is_stuck`, `nudge_text`, the cumulative backstop —
-one tool call at a time. None of the thresholds were re-tuned. Three things it adds:
+`PrefixStage` (with micro-skills on, the free envfacts layer under the proposer) and
+`ToolingStage` (evidence pruning). The loop guard (`loopguard.TurnGuard`) is the benchmark
+harness's `NudgeGuard` control flow — `SpiralState.update`, `assess_abort`, `is_stuck`,
+`nudge_text`, the cumulative backstop — one tool call at a time. None of the thresholds were
+re-tuned. Three things it adds:
* It counts **edits made through the shell**: `sed -i`, `perl -pi`, a redirect or `tee` into a
source file, an inline Python script that writes a file back, `patch`/`git apply`, Codex's
@@ -39,7 +40,7 @@ one tool call at a time. None of the thresholds were re-tuned. Three things it a
that floor, so the first call that counts as stuck has also tripped the backstop, and the turn
would be stopped (or, in nudge mode, merely flagged) without ever being redirected. Now an
unspent nudge goes first, and the stop needs a second stuck segment after it.
-* With `paid` on, the **stuck judge and nudge writer run in a detached `te judge` job**, not in
+* The **stuck judge and nudge writer run in a detached `te judge` job**, not in
the hook. The hook starts the job and returns; a later PostToolUse picks up the verdict, with
the redirect already written. Each job's spend is charged to the ledger when it is spent.
@@ -56,9 +57,12 @@ Every hook fails open: any error becomes an empty output (no change) and a trace
| Tool removal | `--disallowed-tools` / Codex config at launch | a **deny list**, applied by default once the evidence forms it and the repository's sessions have been idle longer than a prompt cache lives (§3), and announced at the next session start (`toolprune=propose` waits for approval instead), written to the repository's own settings file — tools in `.claude/settings.local.json`, Codex capability packages in `.codex/config.toml`, Copilot MCP servers in `.github/copilot/settings.local.json` — effective from the **next** session | The plugin launches nothing. A session keeps the tools it started with, but a conversation resumed after a change sends the new ones, re-writing its cached prefix, hence the idle wait. Each agent's file can express a different unit (§3) |
| Pruning candidates | `toolprune.tools.INAPPLICABLE_TOOLS` — orchestration, Q&A, plan mode, web — "inapplicable by construction" for a sealed headless task | nothing is inapplicable by construction; only **evidence** (never used, and unused over N prompts across two or more sessions, or over N sessions) proposes a tool, and a person's tools — AskUserQuestion, plan mode, sub-agents, the web — are protected | A person asks questions, plans and browses in a real session. A deny rule the plugin wrote for a tool that has since become protected is given back at the next fold |
| Network | every arm, baseline included, carries an answer-key guard — a `PreToolUse` hook and an `/etc/hosts` block refuse the benchmarks' upstream sources — and drops the web tools | **no guard**: no `PreToolUse` hook (on Pi, no `tool_call` handler), no blocked command or host; the web tools stay | A benchmark's answer is a fetch away. A real session has no answer key, and the web is part of the work |
+| Who decides a turn is stuck | in the guard's arms, the trace judge, woken by the rule gate or by a turn past the envelope (the agents ran under a turn cap); without a judge, the strict rule | **always the judge** when the guard is on, woken only by a sign of trouble (a repeated failure, or dense errors) after `min_turns` calls, and never before the repository has an envelope; a `productive` or unsure verdict spares the turn. The hard backstop (far past the envelope AND the strict stall rule) still acts without it. While the agent's model calls are paused, the judge gives no verdict and only the backstop acts | The rule alone redirected turns that had found their cause and were working around it (§5); interactive turns outrun a small envelope all the time, so the envelope alone wakes nothing |
| Nudge | kill the agent, resume it with a continuation prompt | a mid-turn reminder attached to the tool result | Different intervention, so its effect is not inherited from the experiments |
| Still stuck after the redirect | terminal abort, in autonomous runs | a one-line **notice** to the person, ending with how to stop the turn on that agent and in that mode (§3); the plugin never stops a turn, and `loopguard: stop` is read as `nudge` | Not every agent's hooks can stop a turn (Codex and Copilot ignore `continue: false`), and the person decides: the benchmark guard's own design note says "interactive callers should map it to a hand-off" |
-| Optimizer model | the agent's own model, so overhead could not look cheaper than its arm | the cheapest capable model on the user's own agent CLI (`paid_model`, default `haiku`), and only when `paid` is on | Every paid call is still charged to the session ledger |
+| Micro-skills' reading | a domain vocabulary per benchmark (Python/git, Node/web); the proposer's digest counts edited Python files, lists test invocations and flags Python import errors, and its prompt leads with the test command | **generic**, whatever the repository: every ecosystem's interpreters, package tools and test runners at once, builds and checks (`make`, `tsc`, `npm run build`, `cargo check`, ...) as their own intent, every edited file, and a language-neutral digest (failing commands with a normalized error line, paths relative to the repository) and prompt, with tests one topic among several. `vocabulary=python-git` or `node-web` selects an experiment's | A person's repository may be in any language, or several; the experiments keep the paper's reading (its prompt is pinned by a test) |
+| Micro-skills | on in the arms that measured it: the proposer every 5 tasks (on Codex, every 10, over the free layer) | **off by default**; `on` runs the free layer under the proposer, every 3 sessions (after the 2nd, 5th, 8th, ...), since a real session usually carries more work than one benchmark task | The proposer calls a model on the person's plan, and what it injects is derived from command output a repository controls: turning it on is the person's call. In the experiments the free layer alone formed almost nothing once the harness set each environment up properly, so `on` keeps the proposer |
+| Optimizer model | the agent's own model, so overhead could not look cheaper than its arm | the cheapest capable model on the user's own agent CLI (`paid_model`, default `haiku`): the proposer when micro-skills is on, the loop guard's judge and redirect writer whenever the guard is on | Every call is still charged to the session ledger |
## 3. Verified agent behaviour
@@ -446,6 +450,51 @@ prompts of a session rather than from whole sessions, so a long first session co
* The loop guard already counts prompt-turns: each turn that ends green feeds the envelope at
its `Stop`, and the guard acts within the same session once it has five.
+**Copilot and Pi across repositories, 2026-10-25** (Copilot CLI 1.0.102 over BYOK, Pi 1.3.0;
+every model through OpenRouter; `microskills=on`, everything else default but a zero idle
+wait). Each agent ran eight one-prompt sessions: in a repository, in its `pkg/` subdirectory, in
+a second clone of it, in another repository, and in a plain folder.
+
+* The repository, its subdirectory and its clone shared one state; the other repository and the
+ folder had their own. `repos` and `status` named the right one from the clone and the
+ subdirectory.
+* On Pi the missing-`pytest` fact formed over four sessions and was injected at the fifth and
+ sixth, which ran `python -m pytest` first. On Copilot the model ran `python -m pytest` unprompted
+ in three sessions of six, and the fact stayed under its firing gate.
+* Copilot's unused MCP server was turned off after five sessions and announced at the next start;
+ the clone got the list at the end of its own next session.
+* Pi's proposer ran after the second and fifth sessions, through `pi --print`, and was charged.
+
+What the run found, now fixed: a session started in a subdirectory that ran
+`cd <repository> && pytest` learned that as a form of its own, spelled with the machine's path (a `cd` up to the
+repository now lands relative to it); and under BYOK Copilot's proposer asked to sign in every
+third session, leaving an empty session behind each time (a CLI that answers it is not signed
+in now pauses the plugin's model calls on that agent, and the next session start says so once).
+
+**The judge decides, 2026-10-25** (Claude Code 2.1.310, Claude Haiku 5.5 through OpenRouter, for
+the agent and for the judge; thresholds lowered so a loop forms quickly, the backstop raised to
+six envelopes so that it could not act first). A turn told to make 12 edit-and-test cycles
+against a test no edit can pass: the judge's first verdict spared it; the second, asked at call
+16, found it stuck (0.90), and its model's redirect -- naming the eight edits to one file and
+the repeated failure, and asking for the traceback to be read before another edit -- reached the
+agent at call 24, which stopped two calls later and said the test could not pass. The two
+verdicts and the redirect cost about 25k effective tokens, charged to the session.
+
+**Micro-skills read generically, 2026-10-25** (Pi 1.3.0, Claude Haiku 5.5 through OpenRouter;
+`microskills=on`). Seven sessions in a Node repository whose tests load a build output that each
+test run deletes. The model read `package.json` and `build.js` first every time and never met
+the trap; the proposer, run after the second and fifth sessions by `microskills=on` alone,
+admitted four facts -- where the build script, the tests and the manifest are, and how
+`src/ops.js` exports -- injected from the fifth session on. None was about tests. The two
+sessions given them still read the same files first, so in a repository this small they saved
+nothing.
+
+What the run's preparation found, now fixed: the proposer's window of recent sessions was not
+saved with its state. The experiments keep one proposer in memory over a stream; the plugin
+reloads it for every session, so each run saw only the session that had just ended, and a fact,
+which must cite two, could never be admitted. Every proposer call before this was spent for
+nothing.
+
## 4. State and privacy
All state is local, under `$TOKENECONOMIZER_HOME` (default `~/.tokeneconomizer`), one directory
@@ -486,10 +535,13 @@ Which repository a session belongs to (`plugin/paths.py`, `plugin/repos.py`):
own session logs it is derived from, it is private to you: the plugin creates it `0700` and its
files `0600`, and tightens a looser one it finds.
-Nothing leaves the machine unless `paid` is on. Then the proposer, judge and nudge writer send
-**digests** — command templates, edited file basenames, error signatures, never file contents —
-through your own agent CLI (`claude -p` / `codex exec` / `copilot -p` / `pi -p`), billed where
-your session is.
+Nothing leaves the machine but the model calls of the loop guard (its judge and nudge writer, on
+by default with the guard) and of micro-skills (its proposer, when it is on). Those calls send
+**digests** — command templates, edited file basenames, error lines with paths and numbers
+masked, never file contents — through your own agent CLI (`claude -p` / `codex exec` /
+`copilot -p` / `pi -p`), billed where your session is. The proposer keeps the digests of the last eight
+sessions with its state, raw commands and output capped, so that each run sees more than the
+session that just ended.
The deny-list writer adds and removes only deny rules it recorded adding, never one you wrote, and
adds a `settings.local.json` it creates to `.git/info/exclude`. A repository decides what its
@@ -499,9 +551,9 @@ parse rather than write over it.
A learned fact is text injected into your agent's context, derived from your sessions' command
output — which a repository controls. The free layer only fills fixed templates with observed
-command forms and hosts, and lints out completion advice; the paid proposer's facts are
-open-vocabulary. Read them with `te microskills` before enabling the proposer on a repository you do
-not trust.
+command forms and hosts, and lints out completion advice; the proposer's facts are
+open-vocabulary. On a repository you do not trust, use `microskills=envfacts`, or read what it
+learned with `te microskills`.
## 5. Known limitations
@@ -509,9 +561,10 @@ not trust.
per-tool deny list waits on Copilot gaining a persistent per-tool filter.
* **The loop guard, live** — both rungs have run on a provoked loop on every agent (2026-10-21),
not on a real spiral, and whether a nudge changes a looping model's course is not established
- (on Pi one run stopped after it, another went on). The guard counts edits and
- repeated failures, not understanding, so with a small learned envelope it can fire on a turn
- that has already found its cause and is working around it.
+ (on Pi one run stopped after it, another went on). The rule counts edits and repeated
+ failures, not understanding, so with a small learned envelope it could fire on a turn that had
+ already found its cause and was working around it; since 2026-10-25 the judge decides every
+ redirect but the backstop's, which still counts.
* **Pi** — tool pruning is not planned there, since Pi declares only four tools by default.
* **Known gaps:**
* A test run whose exit code the model masks (`python3 -m unittest; echo "exit=$?"`, seen on
diff --git a/plugin/README.md b/plugin/README.md
index a9b424b..7fcedfc 100644
--- a/plugin/README.md
+++ b/plugin/README.md
@@ -216,7 +216,7 @@ redirect shows "● TokenEconomizer: …" in the conversation (and on stdout und
**With your own model key (BYOK).** If Copilot answers *"Access denied by policy settings"*, your
organization's policy refuses its models, and Copilot can use your own provider instead. The
-plugin works the same way there, except for its paid helpers (below). In your shell profile:
+plugin works the same way there, except for its model calls (below). In your shell profile:
```bash
export COPILOT_PROVIDER_TYPE=anthropic # or openai, azure
@@ -237,10 +237,13 @@ export COPILOT_PROVIDER_MODEL_ID=claude-haiku-4.5 # the id Copilot config
export COPILOT_PROVIDER_WIRE_MODEL=anthropic/claude-haiku-4.5 # the id OpenRouter bills
```
-Copilot does not pass these variables to the plugin's hooks, so under BYOK the paid helpers
-(`paid=true`) cannot reach your provider: their calls answer nothing, and each leaves a line in
-`~/.tokeneconomizer/errors.log`. Everything else, including `refresh-catalog` and `measure`,
-needs no provider. `~/.tokeneconomizer/bin/te --host copilot doctor` shows the route it sees.
+Copilot does not pass these variables to the plugin's hooks, so under BYOK its model calls
+(micro-skills' proposer, the loop guard's judge) cannot reach your
+provider: `copilot -p` asks to sign in instead. The first such answer leaves a line in
+`~/.tokeneconomizer/errors.log` and pauses those calls, and the next session start says so once;
+the facts that need no model call are still learned (`microskills=envfacts` keeps only those).
+Everything else, including `refresh-catalog` and `measure`, needs no provider.
+`~/.tokeneconomizer/bin/te --host copilot doctor` shows the route it sees, and a pause.
To check the whole plugin on your own setup:
@@ -285,12 +288,18 @@ from); `pi config` turns it off without removing it. Optional: `rm -rf ~/.tokene
## What to expect
-- **Facts.** A fact forms once the same environment failure has come up in a few sessions (three
- confirming ones, among at least four where it applied), and is injected at every session start
- after that while the failure keeps occurring or keeps being prevented. A session start that
- adds facts says so once, in one line. A resumed session keeps the facts it started with, word
- for word, so its cached prompt still matches. `microskills` shows the facts injected and those
- forming.
+- **Facts** (once you turn micro-skills on; it is off by default). Two sources propose them, in
+ any language or toolchain: the same environment failure coming up in a few sessions (a tool,
+ interpreter, build or test command that fails as typed and works spelled another way), with
+ no model call; and, every three sessions, a model reading digests of the recent ones (the
+ commands run, the build, test and lint commands among them, the files edited, and each
+ failing command with its error line, paths relative to the repository) and writing facts
+ about the repository, each with the patterns that test it against later sessions. A fact is
+ injected at every session start once later sessions have confirmed it (three confirming ones,
+ among at least four where it applied), and retired when one contradicts it. How much this helps
+ depends on the repository's state and the tasks you run in it. A session start that adds facts
+ says so once, in one line. A resumed session keeps the facts it started with, word for word, so
+ its cached prompt still matches. `microskills` shows the facts injected and those forming.
- **Deny list.** Applied without asking you once a tool, package or server never used here has
gone unused over ten prompts across at least two sessions, or over five sessions however short
(three times as long for a few higher-risk tools), from the next session on; that session's
@@ -303,13 +312,20 @@ from); `pi config` turns it off without removing it. Optional: `rm -rf ~/.tokene
give back is never removed again (`toolprune apply NAME` removes one again). With
`toolprune=propose`, it is only proposed, the session start says so, and nothing changes
until you run `toolprune apply`. A file the plugin creates is added to `.git/info/exclude`.
-- **Loop guard.** Quiet on normal turns. On a turn that keeps failing the same way well past this
- repository's usual length, one redirect is attached to a tool result; if the turn is still stuck
- after it, a one-line notice tells you how to stop it. The plugin never stops a turn itself.
+ Each checkout keeps its own settings file: a repository's other clones share what it learned,
+ and each gets the list when one of its own sessions ends.
+- **Loop guard.** Quiet on normal turns, and it does nothing until a repository has a few turns
+ that ended with passing checks. A turn that keeps hitting the same failures, or piles up
+ errors, after `min_turns` tool calls is put to a judge model; only a confident "stuck" verdict
+ attaches one redirect (written by that model) to a tool result, and if the turn is still stuck
+ after it, a one-line notice tells you how to stop it. A long turn with no such sign is never
+ judged. A turn far past the repository's usual length that also keeps churning edits through
+ the same failures is redirected even if the judge disagrees. The plugin never stops a turn
+ itself.
Where the notice appears, and how to stop, on each agent and in each mode:
[When a turn gets stuck](../README.md#when-a-turn-gets-stuck).
- **Cost.** `status` shows each session's tokens, priced after it ends, and the plugin's own
- overhead, which is zero unless you turn on the paid helpers.
+ overhead: the loop guard's judge, and micro-skills' proposer when it is on.
## Commands
@@ -367,17 +383,17 @@ and are shared by every agent working in it. The examples use Claude Code's form
**Turn a mechanism on or off:**
```text
-/tokeneconomizer:config set loopguard=off # the loop guard, in this repository
-/tokeneconomizer:config set toolprune=off microskills=on
-/tokeneconomizer:config set loopguard=off --global # in every repository
+/tokeneconomizer:config set microskills=on # micro-skills, in this repository
+/tokeneconomizer:config set toolprune=off loopguard=off
+/tokeneconomizer:config set microskills=on --global # in every repository
/tokeneconomizer:config # what is on, and every value in effect
```
| Mechanism | Key | `on` means | Other values |
|---|---|---|---|
-| Micro-skills | `microskills` | `envfacts`: facts found without model calls | `off`; `envfacts+distiller`, `distiller` add a paid fact proposer (need `paid=true`) |
+| Micro-skills (off by default) | `microskills` | `envfacts+distiller`: facts found without model calls, plus a model proposing facts every `microskills_propose_every` sessions | `off`; `envfacts`: only the facts found without model calls; `distiller`: only the proposer |
| Tool pruning | `toolprune` | `auto`: applied once it forms and your sessions have been idle an hour, from the next session, and announced | `off`; `propose`: only proposed, applied when you run `toolprune apply` |
-| Loop guard | `loopguard` | `nudge`: one mid-turn redirect, then a notice to you if the turn is still stuck | `off`; `stop` is read as `nudge`, with a warning |
+| Loop guard | `loopguard` | `nudge`: a judge model decides whether a looping turn is stuck; if so, one mid-turn redirect, then a notice to you if the turn is still stuck | `off` (no judge either); `stop` is read as `nudge`, with a warning |
**Tune a mechanism:**
@@ -392,7 +408,7 @@ and are shared by every agent working in it. The examples use Claude Code's form
|---|---|---|
| `microskills_max_facts` | `4` | the most facts injected at session start |
| `microskills_max_tokens` | `150` | the size of the injected block, in tokens (about 4 characters each) |
-| `microskills_propose_every` | `5` | sessions between runs of the paid fact proposer |
+| `microskills_propose_every` | `3` | sessions between runs of the fact proposer (the first runs after the second session) |
| `toolprune_evidence` | `10` | prompts a never-used tool must go unused, over at least `toolprune_min_sessions` sessions, before it is left out (three times as many for a few higher-risk tools; the web is never proposed) |
| `toolprune_min_sessions` | `2` | sessions those prompts must span; `1` lets a single session's prompts decide |
| `toolprune_sessions` | `5` | or this many sessions, however few prompts they ran |
@@ -417,10 +433,10 @@ wins over both. Defaults at each preset:
| `envelope_pct` | 90 | 90 | 90 | percentile of the repository's passing turns the envelope starts from |
| `envelope_margin` | 1.5 | 1.25 | 1.0 | multiplier on that percentile |
| `envelope_min_samples` | 7 | 5 | 3 | passing turns needed before the envelope is used |
-| `backstop_mult` | 3.5 | 2.5 | 1.8 | past this multiple of the envelope, a stuck turn is redirected, then flagged to you |
-| `judge_confidence` | 0.85 | 0.7 | 0.55 | with `paid=true`: how sure the model judge must be |
-| `gate_min_turns` | 24 | 18 | 12 | with `paid=true`: tool calls before the judge is asked |
-| `gate_error_outputs` | 8 | 8 | 8 | with `paid=true`: failing results that ask the judge early |
+| `backstop_mult` | 3.5 | 2.5 | 1.8 | past this multiple of the envelope, a turn the rule calls stuck is redirected even if the judge disagrees, then flagged to you |
+| `judge_confidence` | 0.85 | 0.7 | 0.55 | how sure the judge must be that a turn is stuck |
+| `gate_min_turns` | 24 | 18 | 12 | tool calls before the judge is asked (never fewer than `min_turns`) |
+| `gate_error_outputs` | 8 | 8 | 8 | failing results that ask the judge without a repeated failure |
**Repository map** (off by default; see [Repository map](#repository-map)):
@@ -435,8 +451,7 @@ wins over both. Defaults at each preset:
| Key | Default | Values |
|---|---|---|
-| `paid` | `false` | allow the plugin's own model calls (see [Model calls](#model-calls-and-api-keys)) |
-| `paid_model` | `""` | model for those calls; empty: `haiku` on Claude Code, the agent's own default model on Codex and Pi, `COPILOT_MODEL` on Copilot (unless it is `auto`) |
+| `paid_model` | `""` | model for the plugin's own calls; empty: `haiku` on Claude Code, the agent's own default model on Codex and Pi, `COPILOT_MODEL` on Copilot (unless it is `auto`) |
| `shadow` | `false` | decide and log everything, act on nothing |
| `mode` | `auto` | `interactive` or `headless` to override what the plugin detects |
@@ -498,18 +513,21 @@ not rewritten, so the commands that fact recommends keep working.
## Model calls and API keys
-**By default the plugin makes no model calls.** Micro-skills, tool pruning and the loop guard
-all run locally.
-
-With `paid=true`, three helpers can call a model: the fact proposer, the loop guard's judge and
-the redirect writer. They run through **your own agent CLI** (`claude -p`; `codex exec` on
-Codex; `copilot -p`, with every tool removed, on Copilot; `pi -p`, with nothing loaded but the
-model, on Pi), so they use whatever your sessions already use: a Claude subscription or
-Anthropic API key, your Codex login or OpenAI key, Copilot's subscription, or the provider and
-login Pi is set up with. (Copilot with BYOK is the exception; see [Copilot CLI](#copilot-cli).)
-Their cost is billed there too, and shown in `status`. A call that gets no reply leaves a line in
-`~/.tokeneconomizer/errors.log`. `toolprune measure` on Claude Code makes its two calls the same
-way, and only when you run it.
+**By default the plugin's only model calls are the loop guard's**: its judge, asked about a turn
+that shows signs of looping, and, when the judge finds it stuck, the redirect it writes. Tool
+pruning never calls a model. Turning micro-skills on adds its fact proposer, once every
+`microskills_propose_every` sessions (`microskills=envfacts` keeps only the facts found without
+a model call); `loopguard=off` stops the judge with the guard. These calls run through **your
+own agent CLI** (`claude -p`; `codex exec` on Codex; `copilot -p`, with every tool removed, on
+Copilot; `pi -p`, with nothing loaded but the model, on Pi), so they use whatever your sessions
+already use: a Claude subscription or Anthropic API key, your Codex login or OpenAI key,
+Copilot's subscription, or the provider and login Pi is set up with. (Copilot with BYOK is the
+exception; see [Copilot CLI](#copilot-cli).) Their cost is billed there too, and shown in
+`status`. A call that gets no reply leaves a line in `~/.tokeneconomizer/errors.log`; one whose
+CLI answered that it is not signed in also pauses that agent's calls (the loop guard then
+redirects only at its backstop), the next session start says so once, and
+`config set microskills=on` (or `loopguard=on`) tries again. `toolprune measure` on Claude Code
+makes its two calls the same way, and only when you run it.
## Troubleshooting
@@ -520,5 +538,5 @@ way, and only when you run it.
| `python 3.x (needs >= 3.10)` | install Python 3.10+ or set `TE_PYTHON` to one |
| the deny list leaves nothing out after many sessions | run `toolprune refresh-catalog` (Claude Code, Copilot) |
| a Codex deny list was applied but nothing changed | trust the project in Codex; `~/.tokeneconomizer/bin/te --host codex doctor` says whether it is trusted |
-| errors.log has `sidecall copilot: no reply` | the paid helpers cannot use a Copilot BYOK provider ([Copilot CLI](#copilot-cli)); turn `paid` off |
+| errors.log has `sidecall copilot: no reply`, and a session start says the plugin's model calls are paused | they cannot use a Copilot BYOK provider ([Copilot CLI](#copilot-cli)): micro-skills keeps its facts that need no model call (`microskills=envfacts` stops trying the proposer), and the loop guard redirects only at its backstop. Once Copilot is signed in, `config set microskills=on` (or `loopguard=on`) tries again |
| `te` shows another agent's state | add `--host claude`, `codex`, `copilot` or `pi` |
diff --git a/tests/conftest.py b/tests/conftest.py
index 464475c..7f9dd96 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -2,8 +2,28 @@
import os
import sys
+import pytest
+
_TESTS = os.path.dirname(os.path.abspath(__file__))
_REPO = os.path.dirname(_TESTS)
for p in (_REPO, _TESTS):
if p not in sys.path:
sys.path.insert(0, p)
+
+
+def pytest_configure(config):
+ config.addinivalue_line("markers", "real_sidecalls: test the side-call functions themselves "
+ "(their subprocess is faked inside the test)")
+
+
[email protected](autouse=True)
+def _no_model_calls(request, monkeypatch):
+ """No test reaches a real agent CLI. With the loop guard on, a stuck-looking turn wakes its
+ judge: the plugin's side calls answer nothing here, and a judge job runs inline instead of
+ as a detached process. A test that wants a verdict fakes the judge (`stuck_judge`)."""
+ from tokeneconomizer.plugin import judge, sidecall
+ monkeypatch.setattr(judge, "spawn_job", judge.run_job)
+ if request.node.get_closest_marker("real_sidecalls"):
+ return
+ for name in ("claude_call", "codex_call", "copilot_call", "pi_call"):
+ monkeypatch.setattr(sidecall, name, lambda *a, **k: ("", None))
diff --git a/tests/test_envfacts_args.py b/tests/test_envfacts_args.py
index 2423b35..f6ee072 100644
--- a/tests/test_envfacts_args.py
+++ b/tests/test_envfacts_args.py
@@ -225,6 +225,51 @@ def test_a_cd_into_the_start_directory_by_absolute_path_is_no_cd():
assert o.form == "cd /elsewhere && python runtests.py" # left the start: as typed
+def test_a_cd_up_to_the_repository_from_a_subdirectory_is_no_cd():
+ """Seen live on Pi: a session started in `pkg/` ran `cd <repository> && pytest -q`, which
+ was learned as a form of its own, spelled with that machine's path."""
+ kw = dict(cwd="", root="/r/pkg", top="/r")
+ assert observe.classify_command("cd /r && pytest -q", **kw)[0].form == "pytest"
+ assert observe.classify_command("cd .. && pytest -q", **kw)[0].form == "pytest"
+ o = observe.classify_command("cd /r && cd tests && python runtests.py app", **kw)[0]
+ assert o.form == "cd tests && python runtests.py"
+ # Anywhere else above the start is taken as typed: relative to the repository it would hold
+ # only for sessions that start there (seen in a home directory that is a repository).
+ o = observe.classify_command("cd /r/tests && python runtests.py app", **kw)[0]
+ assert o.form == "cd /r/tests && python runtests.py"
+ o = observe.classify_command("cd /elsewhere && pytest -q", **kw)[0]
+ assert o.form == "cd /elsewhere && pytest"
+ o = observe.classify_command("cd /r && pytest -q", cwd="", root="/r/pkg")[0]
+ assert o.form == "cd /r && pytest" # no repository known
+
+
+def test_the_repository_is_the_nearest_directory_holding_git(tmp_path):
+ (tmp_path / "repo" / ".git").mkdir(parents=True)
+ (tmp_path / "repo" / "pkg" / "sub").mkdir(parents=True)
+ (tmp_path / "wt" / "pkg").mkdir(parents=True)
+ (tmp_path / "wt" / ".git").write_text("gitdir: /elsewhere/.git/worktrees/wt\n")
+ assert observe.repository_top(str(tmp_path / "repo" / "pkg" / "sub")) == str(tmp_path / "repo")
+ assert observe.repository_top(str(tmp_path / "wt" / "pkg")) == str(tmp_path / "wt")
+ assert observe.repository_top("relative/path") is None
+ assert observe.repository_top(str(tmp_path / "repo" / "gone")) is None
+
+
+def test_sessions_started_in_a_subdirectory_learn_the_repositorys_plain_form(tmp_path):
+ repo = tmp_path / "repo"
+ (repo / ".git").mkdir(parents=True)
+ (repo / "pkg").mkdir()
+ start = str(repo / "pkg")
+ tasks = [[_at(("Edit", {"file_path": f"{repo}/pkg/m{i}.py", "old_string": "a",
+ "new_string": "b"}, "ok"), start),
+ _at(_sh(f"cd {repo} && pytest -q", "bash: line 1: pytest: command not found"),
+ start),
+ _at(_sh(f"cd {repo} && python -m pytest -q", "1 passed in 0.1s"), start)]
+ for i in range(6)]
+ d, rendered = _stream(tmp_path, tasks)
+ assert "`pytest …` fails here with command not found" in rendered[-1]
+ assert str(repo) not in rendered[-1] and not any(str(repo) in k for k in d.facts)
+
+
def test_a_command_stopped_by_its_own_cd_teaches_nothing(tmp_path):
p = write_claude_transcript(str(tmp_path / "t.jsonl"), [
_at(_sh("cd tests && python runtests.py app", "Exit code 1\n/bin/bash: line 1: cd: "
diff --git a/tests/test_generic_microskills.py b/tests/test_generic_microskills.py
new file mode 100644
index 0000000..f5f0ccf
--- /dev/null
+++ b/tests/test_generic_microskills.py
@@ -0,0 +1,121 @@
+"""Micro-skills read generically in the plugin: any language or toolchain, builds and checks as
+well as tests, every edited file -- while the experiments keep the paper's reading."""
+import hashlib
+import json
+
+from tokeneconomizer.microskills import GENERIC, NODE_WEB, PYTHON_GIT, observe
+from tokeneconomizer.microskills import distiller as dist
+from tokeneconomizer.microskills import envfacts as ef
+from tokeneconomizer.plugin.adapter import detect_vocabulary
+from fixtures import write_claude_transcript
+
+
+def _form(cmd, vocab):
+ obs = observe.classify_command(cmd, vocab=vocab)
+ return (obs[0].intent, obs[0].form) if obs else None
+
+
+def test_the_generic_reading_knows_builds_and_other_ecosystems():
+ assert _form("npm run build", GENERIC) == ("run_build", "npm run build")
+ assert _form("npm run test:unit", GENERIC) == ("run_tests", "npm run test:unit")
+ assert _form("cargo build --release", GENERIC) == ("run_build", "cargo build")
+ assert _form("tsc -p .", GENERIC) == ("run_build", "tsc")
+ assert _form("go test ./...", GENERIC) == ("run_tests", "go test")
+ assert _form("./gradlew test", GENERIC) == ("run_tests", "gradlew test")
+ assert _form("bundle install", GENERIC) == ("package_tool", "bundle")
+ # The experiments' vocabularies read as they did.
+ assert _form("tsc -p .", PYTHON_GIT) is None and _form("npm run build", NODE_WEB) == (
+ "package_tool", "npm")
+ assert _form("./gradlew test", PYTHON_GIT) is None
+
+
+def _session(tmp_path, i, calls):
+ return write_claude_transcript(str(tmp_path / f"s{i}.jsonl"), calls)
+
+
+def test_a_build_tool_repair_becomes_a_fact(tmp_path):
+ """Not a test command at all: `tsc` missing, `npx tsc` working, session after session."""
+ d = ef.EnvFactsDistiller("acme/web", vocab=GENERIC)
+ for i in range(6):
+ d.update(_session(tmp_path, i, [
+ ("Edit", {"file_path": f"/r/src/m{i}.ts", "old_string": "a", "new_string": "b"}, "ok"),
+ ("Bash", {"command": "tsc -p ."}, "Exit code 127\nbash: tsc: command not found"),
+ ("Bash", {"command": "npx tsc -p ."}, "")]), iid=f"s{i}")
+ text = d.render()
+ assert text.startswith(ef.GENERIC_HEADER)
+ assert "`tsc …` fails here with command not found; `npx tsc …` works." in text
+
+
+def test_every_edited_file_counts_in_the_generic_reading(tmp_path):
+ p = _session(tmp_path, 0, [
+ ("Edit", {"file_path": "/r/src/main.go", "old_string": "a", "new_string": "b"}, "ok"),
+ ("Write", {"file_path": "/r/web/app.ts", "content": "x"}, "ok"),
+ ("Edit", {"file_path": "/r/pkg/m.py", "old_string": "a", "new_string": "b"}, "ok")])
+ assert observe.observe_task(p, GENERIC)["edited"] == ["app.ts", "m.py", "main.go"]
+ assert observe.observe_task(p, PYTHON_GIT)["edited"] == ["m.py"]
+
+
+def test_the_generic_digest_is_language_neutral_and_relative(tmp_path):
+ repo = tmp_path / "repo"
+ (repo / ".git").mkdir(parents=True)
+ r = str(repo)
+ p = write_claude_transcript(str(tmp_path / "t.jsonl"), [
+ ("Edit", {"file_path": f"{r}/src/main.ts", "old_string": "a", "new_string": "b"}, "ok", r),
+ ("Bash", {"command": f"cd {r} && npm run build"},
+ f"Exit code 2\n{r}/src/main.ts(3,5): error TS2304: Cannot find name 'foo'.", r),
+ ("Bash", {"command": "node scripts/gen.js"},
+ "Error: Cannot find module 'left-pad'\nExit code 1", r),
+ ("Bash", {"command": "cargo build"}, "Finished dev", r)])
+ dg = dist.build_generic_digest("acme/web", p, "s0")
+ assert dg["edited_files"] == ["main.ts"]
+ assert dg["check_commands"] == ["npm run build", "cargo build"]
+ assert dg["failed_commands"] == [
+ # line and column masked; the error's own code kept, as it names the failure
+ {"cmd": "npm run build", "error": "src/main.ts(N,N): error TS2304: Cannot find name 'foo'."},
+ {"cmd": "node scripts/gen.js", "error": "Error: Cannot find module 'left-pad'"}]
+ assert dg["missing_dependency"] and not dg["missing_command"]
+ shown = json.dumps(dist.proposer_view(dg))
+ assert r not in shown and "_cmd_blob" not in shown
+
+
+def test_the_plugin_reads_every_repository_generically(tmp_path):
+ (tmp_path / "package.json").write_text("{}")
+ v = detect_vocabulary(str(tmp_path))
+ assert v.generic and v.canonical_interpreter == "node"
+ assert detect_vocabulary(str(tmp_path / "nowhere")) is GENERIC
+ assert detect_vocabulary(str(tmp_path), "python-git") is PYTHON_GIT
+
+
+def test_the_proposer_keeps_its_window_across_folds(tmp_path, monkeypatch):
+ """The plugin reloads the proposer for every session. Its window of recent sessions was not
+ saved, so each tick saw one session, and a fact -- which must cite two -- could never be
+ admitted, however many calls were paid for."""
+ monkeypatch.setattr(dist, "PROPOSE_EVERY", 3)
+ prompts = []
+
+ def runner(prompt, model):
+ prompts.append(prompt)
+ return json.dumps({"extractors": [], "facts": [{
+ "id": "build-before-test", "claim": "make test needs make build first",
+ "injection": "`make test` runs the built binary: run `make build` first.",
+ "detector": {"scope": r"make test", "confirm": r"make build"},
+ "cite": ["s0", "s1"]}]}), 100.0
+ root = str(tmp_path / "state")
+ for i in range(2):
+ d = dist.Distiller.load("acme/tool", root, runner=runner, vocab=GENERIC)
+ d.update(_session(tmp_path, i, [
+ ("Bash", {"command": "make build"}, "built"),
+ ("Bash", {"command": "make test"}, "ok ./... 0.01s")]), iid=f"s{i}")
+ d.save(root)
+ assert len(prompts) == 1 and "whatever its language" in prompts[0]
+ assert "build-before-test" in d.facts
+ again = dist.Distiller.load("acme/tool", root, vocab=GENERIC)
+ assert [dg["iid"] for dg in again.window] == ["s0", "s1"]
+
+
+def test_the_experiments_proposer_prompt_is_the_papers():
+ """The generic prompt is the plugin's; the experiments' must stay what the paper ran."""
+ assert hashlib.sha256(dist.PROPOSER_SYSTEM.encode()).hexdigest()[:16] == "fad9adfb8fda5875"
+ g = dist.PROPOSER_SYSTEM_GENERIC
+ assert "repro_written" not in g and "importerror" not in g
+ assert g.count(", ".join(dist.GENERIC_PREDICATES)) == 2
diff --git a/tests/test_live_config.py b/tests/test_live_config.py
index 9d37f2f..bfb9cd8 100644
--- a/tests/test_live_config.py
+++ b/tests/test_live_config.py
@@ -17,7 +17,7 @@ from tests.test_live_hooks import (GREEN, _first, _turn, env, repo, run_session,
# ---- parsing ------------------------------------------------------------------------------
[email protected]("key,on", [("microskills", "envfacts"), ("toolprune", "auto"),
[email protected]("key,on", [("microskills", "envfacts+distiller"), ("toolprune", "auto"),
("loopguard", "nudge")])
def test_each_mechanism_takes_a_plain_on_and_off(key, on):
cfg = LiveConfig()
@@ -185,9 +185,9 @@ def test_a_higher_min_turns_holds_the_guard_back(env, repo):
def test_injection_off_and_a_tight_fact_cap_keep_the_context_clean(env, repo, tmp_path):
for i in range(4):
run_session(env, repo, f"s{i}", trap_session(str(tmp_path / f"s{i}.jsonl"), repo, i))
- off = run_session(dict(env, TE_INJECT="off"), repo, "s4",
+ off = run_session(dict(env, TE_MICROSKILLS="off"), repo, "s4",
trap_session(str(tmp_path / "s4.jsonl"), repo, 4))
- assert off == {} # the default injects here
+ assert off == {} # `envfacts` injects here
tight = dict(env, TE_INJECT_MAX_TOKENS="5") # smaller than the block's header
run_session(tight, repo, "s5", trap_session(str(tmp_path / "s5.jsonl"), repo, 5))
after = run_session(tight, repo, "s6", trap_session(str(tmp_path / "s6.jsonl"), repo, 6))
diff --git a/tests/test_live_core.py b/tests/test_live_core.py
index 5bcc679..fbfa1ec 100644
--- a/tests/test_live_core.py
+++ b/tests/test_live_core.py
@@ -99,16 +99,23 @@ def test_config_layers_and_never_raises_on_bad_values(tmp_path):
cfg = live_config.load_config(root, repo_dir=repo_dir, env={"TE_UNTOOL": "auto"})
assert cfg.loopguard == "stop" # the invalid repo value is ignored
assert cfg.toolprune_evidence == 7 and cfg.toolprune == "auto"
- assert cfg.microskills == "envfacts+distiller" and cfg.microskills_effective == "envfacts"
+ assert cfg.microskills == "envfacts+distiller" # its proposer needs no `paid`
joined = " ".join(cfg.warnings)
- assert "bogus" in joined and "nope" in joined and "paid=false" in joined
+ assert "bogus" in joined and "nope" in joined and "paid" not in joined
assert live_config.load_config(root, env={"TE_DISABLE": "1"}).enabled is False
def test_write_config_validates_before_writing(tmp_path):
path = str(tmp_path / "config.json")
- live_config.write_config(path, {"loopguard": "off", "paid": "true"})
- assert json.load(open(path)) == {"loopguard": "off", "paid": True}
+ live_config.write_config(path, {"loopguard": "off", "paid_model": "haiku"})
+ assert json.load(open(path)) == {"loopguard": "off", "paid_model": "haiku"}
+ with pytest.raises(ValueError, match="always asks its judge"):
+ live_config.write_config(path, {"paid": "true"}) # gone: refused, and why
+ assert "paid" not in json.load(open(path))
+ with open(path, "w") as fh:
+ json.dump({"paid": True, "loopguard": "off"}, fh) # an old file still reads
+ cfg = live_config.load_config(str(tmp_path), repo_dir=None, env={})
+ assert not cfg.warnings
with pytest.raises(ValueError):
live_config.write_config(path, {"loopguard": "sometimes"})
assert json.load(open(path))["loopguard"] == "off"
@@ -679,6 +686,7 @@ def test_copilot_tools_are_grouped_by_mcp_server(tmp_path):
"github-mcp-server", "acme-db"}
def test_a_copilot_side_call_is_a_bare_completion_priced_from_its_own_log(tmp_path,
monkeypatch):
import subprocess
@@ -701,12 +709,13 @@ def test_a_copilot_side_call_is_a_bare_completion_priced_from_its_own_log(tmp_pa
assert cmd[:3] == ["copilot", "-p", "propose facts"] and "--disable-builtin-mcps" in cmd
assert cmd.count("--excluded-tools") == len(sidecall.COPILOT_TOOLS)
assert seen["env"]["TE_SIDECALL"] == "1" # its own hooks must not recurse
- cfg = live_config.LiveConfig(paid=True)
+ cfg = live_config.LiveConfig()
assert sidecall.runner_for("copilot", cfg).__code__.co_names.count("copilot_call") == 1
assert sidecall.runner_for("claude", cfg).__code__.co_names.count("claude_call") == 1
assert sidecall.runner_for("pi", cfg).__code__.co_names.count("pi_call") == 1
def test_a_side_call_with_no_reply_is_noted_and_keeps_to_a_bare_completion(tmp_path,
monkeypatch):
"""Seen live: under Copilot's BYOK the provider's variables do not reach the hooks, so the
@@ -774,6 +783,7 @@ def test_a_pi_reply_is_read_from_its_json_run():
assert trace.pi_json_reply("") == ("", None)
def test_a_pi_side_call_is_a_bare_run_on_the_persons_own_pi(monkeypatch):
import subprocess
from tokeneconomizer.plugin import sidecall
diff --git a/tests/test_live_hooks.py b/tests/test_live_hooks.py
index ca8f279..eb09b8b 100644
--- a/tests/test_live_hooks.py
+++ b/tests/test_live_hooks.py
@@ -29,6 +29,9 @@ def env(tmp_path):
return {"TOKENECONOMIZER_HOME": str(tmp_path / "home"), "TE_FOLD_SYNC": "1",
# A deny list is written at the fold here; the wait before it has its own tests.
"TE_TOOLPRUNE_QUIET_MINUTES": "0",
+ # Facts are learned here without a model call. Micro-skills is off by default, and
+ # its `on` adds the proposer; both have their own tests.
+ "TE_MICROSKILLS": "envfacts",
"GIT_CEILING_DIRECTORIES": str(tmp_path), "PATH": os.environ.get("PATH", "")}
@@ -139,6 +142,44 @@ def test_a_learned_fact_reaches_the_next_session_and_is_announced_once(env, repo
assert starts[5]["hookSpecificOutput"]["additionalContext"] == ctx
+def test_micro_skills_is_off_by_default_and_learns_nothing(env, repo, tmp_path, monkeypatch,
+ capsys):
+ """The sessions that teach the fact above leave nothing behind unless it is turned on."""
+ from tokeneconomizer.plugin import cli
+ default = {k: v for k, v in env.items() if k != "TE_MICROSKILLS"}
+ starts = [run_session(default, repo, f"s{i}",
+ trap_session(str(tmp_path / f"s{i}.jsonl"), repo, i))
+ for i in range(6)]
+ assert all(s == {} for s in starts)
+ assert store_for(default, repo).read("injection.json") is None
+ monkeypatch.delenv("TE_MICROSKILLS", raising=False)
+ for k, v in default.items():
+ monkeypatch.setenv(k, v)
+ assert cli.main(["--host", "claude", "--cwd", repo, "microskills"]) == 0
+ assert "`config set microskills=on` turns it on" in capsys.readouterr().out
+
+
+def test_micro_skills_on_runs_the_proposer_without_paid_and_charges_it(
+ env, repo, tmp_path, monkeypatch):
+ """`on` is the experiments' micro-skills: the free facts plus the proposer, here every
+ third session (after the 2nd, 5th, ...), as a session carries more work than a benchmark
+ task. Turning it on is what allows the proposer's model call; `paid` is the loop guard's."""
+ from tokeneconomizer.plugin import sidecall
+ models = []
+
+ def fake_call(prompt, model="haiku", **kw):
+ models.append(model)
+ return "{}", 120.0
+ monkeypatch.setattr(sidecall, "claude_call", fake_call)
+ on = dict(env, TE_MICROSKILLS="on")
+ for i in range(5):
+ run_session(on, repo, f"s{i}", trap_session(str(tmp_path / f"s{i}.jsonl"), repo, i))
+ ledger = store_for(on, repo).read_lines("ledger.jsonl")
+ assert models == ["haiku", "haiku"]
+ assert [r["overhead"]["optimizer_tokens"] for r in ledger] == [0, 120.0, 0, 0, 120.0]
+ assert store_for(on, repo).read("injection.json")["profile"] == "envfacts+distiller"
+
+
def multi_prompt_trap_session(path, repo, i, prompts=4, every=False):
"""A session of `prompts` prompts that meets the missing `pytest` at its first test run (or,
with `every`, at each): later prompts use the working form, as the conversation shows it."""
@@ -322,12 +363,12 @@ def test_the_guard_redirects_then_tells_the_person_how_to_stop_on_every_agent(
assert len(to_model) == 1 # the redirect alone
-def _fake_judges(monkeypatch, seen):
+def _fake_judges(monkeypatch, seen, stuck=True):
from tokeneconomizer.loopguard import judge as reflect
def judge(digest, runner, model="agent"):
seen.append(digest)
- return {"stuck": True, "confidence": 0.95, "reason": "loop", "_tokens": 1000.0}
+ return {"stuck": stuck, "confidence": 0.95, "reason": "loop", "_tokens": 1000.0}
def writer(digest, runner, model="agent"):
return {"text": "Read the failing assertion before editing again.", "_tokens": 300.0}
@@ -335,13 +376,13 @@ def _fake_judges(monkeypatch, seen):
monkeypatch.setattr(reflect, "llm_nudge_writer", writer)
-def test_the_paid_judge_answers_from_a_job_and_its_spend_is_charged_as_it_happens(
+def test_the_judge_answers_from_a_job_and_its_spend_is_charged_as_it_happens(
env, repo, monkeypatch):
- """The job runs inline here (TE_JUDGE_SYNC=1); in the plugin it is a detached process."""
+ """The job runs inline here (see conftest); in the plugin it is a detached process."""
from tokeneconomizer.plugin import accounting
seen = []
_fake_judges(monkeypatch, seen)
- paid = dict(env, TE_PAID="1", TE_JUDGE_SYNC="1")
+ paid = env
for k in range(5):
_turn(paid, repo, "s", f"g{k}", GREEN)
outs = _turn(paid, repo, "s", "stuck", stuck_calls(40))
@@ -361,12 +402,12 @@ def test_the_paid_judge_answers_from_a_job_and_its_spend_is_charged_as_it_happen
1300.0 * len(charged))
-def test_a_hook_only_starts_the_paid_judge_and_never_waits_for_it(env, repo, monkeypatch):
+def test_a_hook_only_starts_the_judge_and_never_waits_for_it(env, repo, monkeypatch):
from tokeneconomizer.plugin import judge as live_judge
seen, spawned = [], []
_fake_judges(monkeypatch, seen)
monkeypatch.setattr(live_judge, "spawn_job", lambda ctx, sid, job: spawned.append(job))
- paid = dict(env, TE_PAID="1")
+ paid = env
for k in range(5):
_turn(paid, repo, "s", f"g{k}", GREEN)
_turn(paid, repo, "s", "stuck", stuck_calls(12))
@@ -375,6 +416,62 @@ def test_a_hook_only_starts_the_paid_judge_and_never_waits_for_it(env, repo, mon
assert req["id"] == spawned[0] and "NOT green" not in req["digest"]
+def _redirects(outs):
+ return [i for i, o in enumerate(outs) if "hookSpecificOutput" in o]
+
+
+def test_the_judge_decides_every_redirect_but_the_backstops(env, repo, monkeypatch):
+ """With the guard on, a turn that shows signs of looping goes to the judge, and only a
+ confident `stuck` redirects it; a `productive` verdict spares it, and only the backstop --
+ far past the usual turn AND the strict stall pattern, which the judge cannot veto -- still
+ acts. The rule alone, as the guard ran without a judge, redirected this turn at call 12."""
+ _warm(env, repo) # usual turn: 8 calls, envelope 10
+ seen = []
+ _fake_judges(monkeypatch, seen, stuck=False)
+ spared = _turn(env, repo, "s", "spared", stuck_calls(40))
+ first = _redirects(spared)
+ assert seen and first and first[0] + 1 >= 25 # the backstop: 2.5 x the envelope
+ _fake_judges(monkeypatch, seen, stuck=True)
+ judged = _turn(env, repo, "s", "judged", stuck_calls(40))
+ first = _redirects(judged)
+ assert first and first[0] + 1 < 25
+ assert judged[first[0]]["hookSpecificOutput"]["additionalContext"] == (
+ "Read the failing assertion before editing again.")
+
+
+def test_the_judge_is_not_asked_before_the_repository_is_learned(env, repo, monkeypatch):
+ seen = []
+ _fake_judges(monkeypatch, seen)
+ assert all(o == {} for o in _turn(env, repo, "s", "cold", stuck_calls(40)))
+ assert seen == []
+
+
+def test_a_long_turn_with_no_sign_of_trouble_never_wakes_the_judge(env, repo, monkeypatch):
+ """Interactive turns outrun a small envelope all the time: that alone asks nothing."""
+ _warm(env, repo)
+ seen = []
+ _fake_judges(monkeypatch, seen)
+ long_and_clean = [("Read", {"file_path": f"/r/m{k}.py"}, "ok") for k in range(60)]
+ assert all(o == {} for o in _turn(env, repo, "s", "long", long_and_clean))
+ assert seen == []
+
+
+def test_while_model_calls_are_paused_only_the_backstop_redirects(env, repo, monkeypatch):
+ from tokeneconomizer.plugin import sidecall
+ _warm(env, repo)
+ seen = []
+ _fake_judges(monkeypatch, seen)
+ monkeypatch.setattr(sidecall, "paused", lambda host, root=None: {"at": 1.0})
+ outs = _turn(env, repo, "s", "stuck", stuck_calls(40))
+ first = _redirects(outs)
+ assert seen == [] and first and first[0] + 1 >= 25
+
+
+def _warm(env, repo):
+ for k in range(5):
+ _turn(env, repo, "s", f"g{k}", GREEN)
+
+
def test_only_turns_that_end_green_feed_the_envelope(env, repo):
fail = {"stdout": "FAILED tests/test_a.py::t - AssertionError", "stderr": ""}
_turn(env, repo, "s", "red", [("Edit", {"file_path": "/r/a.py"}, "ok"),
@@ -897,6 +994,53 @@ def test_copilot_learns_from_its_own_logs_and_injects_at_top_level(env, repo, tm
assert len(turns) == 5 and all(t["tool_calls"] == 2 for t in turns)
+SIGNED_OUT = ("Error: No authentication information found. Copilot can be authenticated with "
+ "GitHub using an OAuth Token or a Fine-Grained Personal Access Token. To "
+ "authenticate, you can use any of the following methods: Start 'copilot' and run "
+ "the '/login' command")
+
+
+def test_a_signed_out_copilot_pauses_the_model_calls_and_says_so_once(env, repo, tmp_path,
+ monkeypatch):
+ """Seen live under BYOK: Copilot does not pass the provider to the plugin's hooks, so the
+ proposer's `copilot -p` asked to sign in, every third session, each time leaving an empty
+ session behind. One such answer pauses the calls; the next start says so once, the facts
+ that need no model call still form, and `config set microskills=on` tries again."""
+ import subprocess
+ from tokeneconomizer.plugin import cli, sidecall
+ for k, v in env.items():
+ monkeypatch.setenv(k, v) # `_no_reply` writes under the state root
+ tried = []
+
+ def signed_out(prompt, model="", **kw):
+ tried.append(prompt)
+ sidecall._no_reply("copilot", subprocess.CompletedProcess([], 1, "", SIGNED_OUT))
+ return "", None
+ monkeypatch.setattr(sidecall, "copilot_call", signed_out)
+ on = dict(env, TE_MICROSKILLS="on")
+ starts = [copilot_session(on, repo, tmp_path, f"b{i}", COPILOT_TRAP, monkeypatch)
+ for i in range(6)]
+ assert len(tried) == 1 # after the 2nd session; not after the 5th
+ said = [p for s in starts for p in s.get("progress", []) if "not signed in" in p]
+ assert len(said) == 1 and "Copilot does not pass it to the plugin's hooks" in said[0]
+ assert "progress" in starts[2] and said[0] in starts[2]["progress"]
+ assert "`python -m pytest …` works." in starts[4]["additionalContext"]
+ assert cli.main(["--host", "copilot", "--cwd", repo, "config", "set", "microskills=on"]) == 0
+ assert sidecall.paused("copilot") is None
+
+
+def test_only_a_signed_out_answer_pauses_the_model_calls(env, monkeypatch):
+ import subprocess
+ from tokeneconomizer.plugin import sidecall
+ monkeypatch.setenv("TOKENECONOMIZER_HOME", env["TOKENECONOMIZER_HOME"])
+ sidecall._no_reply("pi", subprocess.CompletedProcess([], 1, "", "429 rate limited"))
+ assert sidecall.paused("pi") is None
+ sidecall._no_reply("claude", subprocess.CompletedProcess([], 1, "Invalid API key · Please "
+ "run /login", ""))
+ assert sidecall.paused("claude") and sidecall.paused("pi") is None
+ assert sidecall.resume("claude") and sidecall.paused("claude") is None
+
+
def test_a_failed_copilot_tool_reaches_the_guard(env, repo, tmp_path, monkeypatch):
copilot_session(env, repo, tmp_path, "f", [
("view", {"path": f"{repo}/nope.py"}, "Path does not exist", False)], monkeypatch)
diff --git a/tests/test_live_repos.py b/tests/test_live_repos.py
index 33cad9d..673e009 100644
--- a/tests/test_live_repos.py
+++ b/tests/test_live_repos.py
@@ -47,6 +47,9 @@ def env(tmp_path):
return {"TOKENECONOMIZER_HOME": str(tmp_path / "te"), "TE_FOLD_SYNC": "1",
# A deny list is written at the fold here; the wait before it has its own tests.
"TE_TOOLPRUNE_QUIET_MINUTES": "0",
+ # Facts are learned here without a model call. Micro-skills is off by default, and
+ # its `on` adds the proposer; both have their own tests.
+ "TE_MICROSKILLS": "envfacts",
"GIT_CEILING_DIRECTORIES": str(tmp_path), "HOME": str(home),
"PATH": os.environ.get("PATH", "")}
diff --git a/tokeneconomizer/loopguard/rules.py b/tokeneconomizer/loopguard/rules.py
index ff252d9..37b1844 100644
--- a/tokeneconomizer/loopguard/rules.py
+++ b/tokeneconomizer/loopguard/rules.py
@@ -151,7 +151,9 @@ def assess_abort(state, envelope=None, judge=None, thr=None):
# an over-envelope run is worth a paid judge look even without an exact repeated
# failure yet, which is what lets the guard engage on a short-turn agent whose
# spirals hit the cap before accumulating a repeat signature.
- over_envelope = envelope is not None and state.turns >= envelope
+ # `gate_over_envelope=False` (the plugin's) leaves the gate to the trouble signs alone.
+ over_envelope = (envelope is not None and state.turns >= envelope
+ and thr.get("gate_over_envelope", True))
if not (gate_for_judge(state, thr) or over_envelope):
return False, "gate-closed"
try:
diff --git a/tokeneconomizer/microskills/README.md b/tokeneconomizer/microskills/README.md
index ca841d7..5effd05 100644
--- a/tokeneconomizer/microskills/README.md
+++ b/tokeneconomizer/microskills/README.md
@@ -5,16 +5,18 @@ How micro-skills (see the [main README](../../README.md)) are learned, kept and
Each finished session goes through three steps:
1. **Observe** (`observe.py`). Every shell command is split into segments and classified by
- *intent* (run the tests, run the interpreter, use the package tool, fetch, probe the VCS) and
- *form* (how it was spelled), and its outcome is read off the output as an environmental error
- class or a sign that tests ran. The words that name intents come from a `Vocabulary`
- (`PYTHON_GIT`, `NODE_WEB`), so the same reader handles a Python or a Node repository.
+ *intent* (run the tests, run the interpreter, build or check, use the package tool, fetch,
+ probe the VCS) and *form* (how it was spelled), and its outcome is read off the output as an
+ environmental error class or a sign that tests ran. The words that name intents come from a
+ `Vocabulary`. The plugin reads every repository with `GENERIC` -- every ecosystem at once,
+ every edited file, and the proposer's language-neutral digest and prompt; the experiments
+ use `PYTHON_GIT` or `NODE_WEB`, the paper's narrower reading.
2. **Propose.** `envfacts.py` turns recurring shapes of evidence into candidate facts, with no
model call: a failing form followed by a working one (*form repair*), every fetch to a host
refused (*boundary*), VCS probes that always come back empty (*empty probe*), a test label
that loads once rewritten (*arg repair*), a missing environment variable that a later run
- sets (*env value*). With `paid=true`, `distiller.py` adds an LLM proposer that authors facts
- with host-checked detectors.
+ sets (*env value*). `distiller.py` adds an LLM proposer that authors facts with host-checked
+ detectors; in the plugin, `microskills=on` runs both.
3. **Keep or retire** (`lifecycle.py`). A candidate becomes active after confirmations on
three sessions (for a proposed fact about the code, sessions that edited different files),
retires on contradiction, and is injected only while
diff --git a/tokeneconomizer/microskills/__init__.py b/tokeneconomizer/microskills/__init__.py
index 1e76c03..70ea1db 100644
--- a/tokeneconomizer/microskills/__init__.py
+++ b/tokeneconomizer/microskills/__init__.py
@@ -12,9 +12,9 @@ Stdlib only; never imports the plugin runtime.
from .distiller import Distiller
from .envfacts import EnvFactsDistiller
from .lifecycle import Lifecycle
-from .observe import NODE_WEB, PYTHON_GIT, Vocabulary, classify_command, observe_task
+from .observe import GENERIC, NODE_WEB, PYTHON_GIT, Vocabulary, classify_command, observe_task
from .stage import ComposedDistiller, PrefixStage
-__all__ = ["Distiller", "EnvFactsDistiller", "Lifecycle", "NODE_WEB", "PYTHON_GIT",
+__all__ = ["Distiller", "EnvFactsDistiller", "GENERIC", "Lifecycle", "NODE_WEB", "PYTHON_GIT",
"Vocabulary", "classify_command", "observe_task", "ComposedDistiller",
"PrefixStage"]
diff --git a/tokeneconomizer/microskills/distiller.py b/tokeneconomizer/microskills/distiller.py
index e647d56..79ee0dc 100644
--- a/tokeneconomizer/microskills/distiller.py
+++ b/tokeneconomizer/microskills/distiller.py
@@ -20,8 +20,9 @@ import json
import os
import re
+from ..trace import call_cwds
from ..trace.claude import CONVERSATION
-from .lifecycle import Lifecycle, edited_pyfiles
+from .lifecycle import Lifecycle, edited_files, edited_pyfiles
# ---- proposer cadence / caps / gates ---------------------------------------
PROPOSE_EVERY = 8 # one LLM proposer call per this many tasks (PrefixStage rebinds
@@ -33,6 +34,7 @@ MAX_EXTRACTORS = 12 # cap authored features (keeps the digest bounded)
RAW_SAMPLE = 3 # raw excerpts shown to the proposer on an abstaining tick
ABSTAIN_GATE = 0.5 # tick is "abstaining" if >= this frac of window cmds are unseen
STATE_SUBDIR = "manual_state"
+BLOB_CAP = 20_000 # characters of a saved digest's raw commands (twice that of output)
# ---- regexes ---------------------------------------------------------------
PYTEST_RE = re.compile(r"-m\s+pytest|\bpytest\b|\bpy\.test\b")
@@ -104,7 +106,9 @@ def bash_cmds(calls):
# ---- compact digest: one dict per finished task (the proposer's only view) --
-def build_digest(repo, transcript_path, iid, resolved=None):
+def build_digest(repo, transcript_path, iid, resolved=None, generic=False):
+ if generic:
+ return build_generic_digest(repo, transcript_path, iid, resolved)
try:
calls = load_calls(transcript_path)
except Exception:
@@ -132,6 +136,104 @@ def build_digest(repo, transcript_path, iid, resolved=None):
}
+# ---- the generic digest: any language or toolchain (`Vocabulary.generic`, the plugin's) ---
+# Build, test, lint and type-check commands across ecosystems, and test runs among them.
+CHECK_RE = re.compile(
+ r"\b(?:pytest|py\.test|tox|nox|unittest|runtests\.py|jest|vitest|mocha|rspec|phpunit|ctest|"
+ r"tsc|eslint|prettier|ruff|flake8|pylint|mypy|pyright|golangci-lint|cmake|ninja|bazel)\b|"
+ r"(?:^|[\s;&|(])make\b|"
+ r"\b(?:npm|yarn|pnpm|bun)\s+(?:run\s+)?(?:test|build|lint|check|typecheck)\b|"
+ r"\b(?:cargo|go|mvn|gradle|gradlew|dotnet|mix|swift)\s+"
+ r"(?:test|build|check|vet|clippy|compile|package|verify)\b")
+TEST_RE = re.compile(
+ r"\b(?:pytest|py\.test|tox|nox|unittest|runtests\.py|jest|vitest|mocha|rspec|phpunit|ctest)\b|"
+ r"\b(?:make|npm|yarn|pnpm|bun|cargo|go|mvn|gradle|gradlew|dotnet|mix|swift|rake)\s+"
+ r"(?:run\s+)?test\b")
+MISSING_CMD_RE = re.compile(r"command not found|: not found\b|is not recognized as", re.I)
+MISSING_DEP_RE = re.compile(
+ r"ModuleNotFoundError|No module named|ImportError|Cannot find module|ERR_MODULE_NOT_FOUND|"
+ r"cannot find package|could not find (?:package|crate|module|gem)|unresolved import|"
+ r"cannot load such file|undefined reference to|fatal error: [^\n]*: No such file", re.I)
+BAD_USAGE_RE = re.compile(r"(?m)^usage:|unrecognized arguments?|no such option|"
+ r"unknown (?:option|flag|command)|invalid option|unexpected argument|"
+ r"error: argument", re.I)
+_EXIT_RE = re.compile(r"(?i)\bexit(?:ed with)?(?: code| status)?[:=\s]+(\d+)")
+_FAILED_OUT_RE = re.compile(r"Traceback \(most recent call last\)|^\w*Error\b", re.M)
+_ERROR_LINE_RE = re.compile(r"(?i)\b(?:error|failed|failure|fatal|exception|traceback|cannot|"
+ r"not found|denied|refused|panic)\b")
+
+
+def _command_failed(out):
+ m = _EXIT_RE.search(out or "")
+ if m:
+ return int(m.group(1)) != 0
+ return bool(MISSING_CMD_RE.search(out or "") or MISSING_DEP_RE.search(out or "")
+ or _FAILED_OUT_RE.search(out or ""))
+
+
+def _relative(text, roots):
+ """A command or error line with the repository's own paths made relative and the home
+ directory written `~`: the same shape on every machine, and no one's paths in a prompt."""
+ for r in roots:
+ if r:
+ text = text.replace(r + "/", "").replace(r, ".")
+ text = re.sub(r"^cd \.\s*(?:&&|;)\s*", "", text) # into the repository: no `cd`
+ home = os.path.expanduser("~")
+ return text.replace(home, "~") if home and home != "/" else text
+
+
+def error_signature(out, roots=()):
+ """The first line of a failing output that names the failure, normalized so two sessions
+ that fail the same way read the same: paths, numbers and addresses masked."""
+ for ln in (out or "").splitlines():
+ if _ERROR_LINE_RE.search(ln) or MISSING_CMD_RE.search(ln):
+ s = _relative(ln.strip(), roots)
+ s = re.sub(r"0x[0-9a-fA-F]+", "<hex>", s)
+ s = re.sub(r"(?<![\w.~])/[^\s'\"`:,)]+", "<path>", s)
+ s = re.sub(r"\b\d+\b", "N", s)
+ return " ".join(s.split())[:120]
+ return ""
+
+
+def build_generic_digest(repo, transcript_path, iid, resolved=None):
+ """The digest the plugin's proposer reads: every edited file whatever its language, the
+ build/test/lint commands run, the commands that failed with a normalized error line, and
+ flags that hold in any ecosystem. Raw blobs stay host-side for the detectors."""
+ from .observe import repository_top
+ try:
+ calls = load_calls(transcript_path)
+ except Exception:
+ return None
+ cmds = bash_cmds(calls)
+ start = next((c for c in call_cwds(transcript_path) if c), None)
+ roots = sorted({r for r in (start, repository_top(start)) if r}, key=len, reverse=True)
+ out_blob = "\n".join(o for _, o in cmds)
+ failed, seen = [], set()
+ for c, o in cmds:
+ if not _command_failed(o):
+ continue
+ row = (_templates([_relative(c, roots)], cap=1) or [""])[0], error_signature(o, roots)
+ if row not in seen:
+ seen.add(row)
+ failed.append({"cmd": row[0], "error": row[1]})
+ return {
+ "iid": iid,
+ "resolved": bool(resolved) if resolved is not None else None,
+ "edited_files": sorted(edited_files(calls)),
+ "n_cmds": len(cmds),
+ "cmd_templates": _templates(_relative(c, roots) for c, _ in cmds),
+ "check_commands": _templates(_relative(c, roots) for c, _ in cmds
+ if CHECK_RE.search(c))[:6],
+ "failed_commands": failed[:6],
+ "missing_command": bool(MISSING_CMD_RE.search(out_blob)),
+ "missing_dependency": bool(MISSING_DEP_RE.search(out_blob)),
+ "bad_usage": bool(BAD_USAGE_RE.search(out_blob)),
+ "_ran_tests": any(TEST_RE.search(c) for c, _ in cmds),
+ "_cmd_blob": "\n".join(c for c, _ in cmds),
+ "_out_blob": out_blob,
+ }
+
+
def _templates(cmds, cap=12):
"""Dedup command shapes: drop long path/arg tails so the proposer sees FORMS."""
seen, out = set(), []
@@ -164,12 +266,22 @@ def _rx(pat):
# ---- detector DSL: host-executable, no LLM at eval time --------------------
+# Each reads either digest: a fact the experiments' proposer named keeps a meaning under the
+# generic digest, and the reverse.
PREDICATES = {
- "repro_written": lambda dg: dg["repro_written"],
- "importerror": lambda dg: dg["importerror"],
- "badarg": lambda dg: dg["badarg"],
- "ran_tests": lambda dg: bool(dg["test_invocations"]),
+ "repro_written": lambda dg: bool(dg.get("repro_written")),
+ "importerror": lambda dg: bool(dg.get("importerror", dg.get("missing_dependency"))),
+ "badarg": lambda dg: bool(dg.get("badarg", dg.get("bad_usage"))),
+ "ran_tests": lambda dg: bool(dg.get("test_invocations") or dg.get("_ran_tests")),
+ "ran_checks": lambda dg: bool(dg.get("check_commands") or dg.get("test_invocations")),
+ "command_failed": lambda dg: bool(dg.get("failed_commands")),
+ "missing_command": lambda dg: bool(dg.get("missing_command")),
+ "missing_dependency": lambda dg: bool(dg.get("missing_dependency", dg.get("importerror"))),
+ "bad_usage": lambda dg: bool(dg.get("bad_usage", dg.get("badarg"))),
+ "edited": lambda dg: bool(dg.get("edited_files")),
}
+GENERIC_PREDICATES = ("ran_checks", "ran_tests", "command_failed", "missing_command",
+ "missing_dependency", "bad_usage", "edited")
class Detector:
@@ -292,7 +404,7 @@ class Fact:
# ---- proposer prompt (facts + extractors + revisable contradiction) --------
-PROPOSER_SYSTEM = (
+_PROPOSER_HEAD = (
"You mine a DURABLE, REPO-SPECIFIC operating manual for a coding agent from a "
"window of finished-task digests in ONE repository. Each digest lists edited "
"files, command templates, test invocations, and boolean flags — NOT raw logs.\n\n"
@@ -301,7 +413,10 @@ PROPOSER_SYSTEM = (
"required env var, a build/reinstall step that's needed (or NOT needed), a lint/"
"format hook that reformats, etc. Do NOT emit task-specific fixes or which lines "
"to change — only durable interface facts true across tasks.\n\n"
- "For EACH fact return an object with:\n"
+)
+PROPOSER_SYSTEM = (
+ _PROPOSER_HEAD
+ + "For EACH fact return an object with:\n"
' id short kebab slug\n'
' claim one sentence, what is true about this repo\n'
' injection the exact line to inject into the agent prefix. State it as a '
@@ -338,10 +453,33 @@ PROPOSER_SYSTEM = (
)
+# The plugin's: any language or toolchain, with tests one topic among several. Its rules
+# (fields, detectors, extractors, revision) are the experiments' own.
+_GENERIC_HEAD = (
+ "You mine a DURABLE, REPO-SPECIFIC operating manual for a coding agent from a "
+ "window of finished-session digests in ONE repository, whatever its language and "
+ "toolchain. Each digest lists the files edited, command templates (paths relative to "
+ "the repository), the build, test and lint commands run, the commands that failed "
+ "with a normalized error line, and boolean flags — NOT raw logs or file contents. "
+ "(Each session is called a task below.)\n\n"
+ "Emit facts that a FUTURE session in this repository would want stated up front to "
+ "avoid wasted commands: how to build, run or check it here and the exact invocation "
+ "that works; the toolchain (interpreter, package manager, compiler, version manager) "
+ "and how it is reached; a setup, install, code-generation or rebuild step that is "
+ "needed (or NOT needed); an environment variable, service or working directory a "
+ "command needs; where generated, configured or vendored files live; a command that "
+ "fails here and the variant that works; a hook or formatter that rewrites files. "
+ "Tests are one such topic, not the focus. Do NOT emit task-specific fixes or which "
+ "lines to change — only durable facts true across sessions.\n\n"
+)
+PROPOSER_SYSTEM_GENERIC = (_GENERIC_HEAD + PROPOSER_SYSTEM[len(_PROPOSER_HEAD):]).replace(
+ "repro_written, importerror, badarg, ran_tests", ", ".join(GENERIC_PREDICATES))
+
+
def build_proposer_prompt(repo, window, existing_ids, existing_feats, raw_excerpts,
- external=()):
+ external=(), generic=False):
view = [proposer_view(d) for d in window]
- p = (PROPOSER_SYSTEM
+ p = ((PROPOSER_SYSTEM_GENERIC if generic else PROPOSER_SYSTEM)
+ f"\n\nRepo: {repo}"
+ "\n\nExisting fact ids (re-emit an id with ONLY detector.contradict changed to "
"revise its contradiction condition, else use a NEW id):\n"
@@ -397,8 +535,10 @@ class Distiller:
d.update(transcript_path, iid=..., resolved=...) # fold in the finished task
d.save(root)
"""
- def __init__(self, repo, runner=None, model="agent", external_facts=None):
+ def __init__(self, repo, runner=None, model="agent", external_facts=None, vocab=None):
self.repo = repo
+ # The plugin's generic reading (`Vocabulary.generic`): the generic digest and prompt.
+ self.generic = bool(getattr(vocab, "generic", False))
# Zero-arg callable returning claims another layer already injects (see
# build_proposer_prompt); None when the distiller runs alone.
self.external_facts = external_facts
@@ -426,13 +566,19 @@ class Distiller:
dg["features"] = feats
return dg
+ def _sensed(self, cmd):
+ """Does a built-in sense already read this command?"""
+ if self.generic:
+ return bool(CHECK_RE.search(cmd))
+ return bool(PYTEST_RE.search(cmd) or REPRO_RE.search(cmd))
+
def _coverage(self, cmds):
"""Fraction of commands matched by SOME extractor or built-in test/repro sense."""
if not cmds:
return 1.0
seen = 0
for c, _ in cmds:
- hit = (PYTEST_RE.search(c) or REPRO_RE.search(c)
+ hit = (self._sensed(c)
or any(ex.rx and ex.rx.search(c) for ex in self.extractors.values()))
if hit:
seen += 1
@@ -443,7 +589,7 @@ class Distiller:
out = []
for cmds in self.raw_window[-WINDOW:]:
for c, _ in cmds:
- if PYTEST_RE.search(c) or REPRO_RE.search(c):
+ if self._sensed(c):
continue
if any(ex.rx and ex.rx.search(c) for ex in self.extractors.values()):
continue
@@ -461,7 +607,8 @@ class Distiller:
excerpts = self._abstaining_excerpts()[:RAW_SAMPLE] if abstaining else []
external = list(self.external_facts()) if self.external_facts else []
prompt = build_proposer_prompt(self.repo, self.window, set(self.facts),
- set(self.extractors), excerpts, external=external)
+ set(self.extractors), excerpts, external=external,
+ generic=self.generic)
raw, eff = self.runner(prompt, self.model)
if eff is not None:
self.opt_tokens += eff
@@ -553,7 +700,8 @@ class Distiller:
if not lines:
return ""
head = (f"# repo operating manual ({self.repo.split('/')[-1]}) "
- f"— learned from prior tasks in this repo")
+ + ("— learned from earlier sessions in this repository" if self.generic
+ else "— learned from prior tasks in this repo"))
return head + "\n" + "\n".join(f"- {l}" for l in lines) + "\n"
def update(self, transcript_path, iid="", resolved=None, edited=None):
@@ -564,7 +712,7 @@ class Distiller:
self.now += 1
return
cmds = bash_cmds(calls)
- dg = build_digest(self.repo, transcript_path, iid, resolved)
+ dg = build_digest(self.repo, transcript_path, iid, resolved, generic=self.generic)
if dg is None:
self.now += 1
return
@@ -590,16 +738,25 @@ class Distiller:
return os.path.join(d, repo.replace("/", "__") + ".json")
def save(self, root):
+ # The window goes with the facts: a proposer reloaded for each session (the plugin's
+ # fold) must see the recent sessions, as one kept in memory over a stream (the
+ # experiments') does -- a fact needs MIN_CITE of them to be grounded, so a window of
+ # the current session alone could never admit one. Raw blobs are capped.
+ window = [dict(dg, _cmd_blob=dg.get("_cmd_blob", "")[-BLOB_CAP:],
+ _out_blob=dg.get("_out_blob", "")[-2 * BLOB_CAP:])
+ for dg in self.window[-WINDOW:]]
json.dump({"repo": self.repo, "now": self.now, "opt_tokens": self.opt_tokens,
"last_propose": self.last_propose,
"extractors": {k: e.to_dict() for k, e in self.extractors.items()},
- "facts": {k: f.to_dict() for k, f in self.facts.items()}},
+ "facts": {k: f.to_dict() for k, f in self.facts.items()},
+ "window": window,
+ "raw_cmds": [[c for c, _ in cmds] for cmds in self.raw_window[-WINDOW:]]},
open(self._path(root, self.repo), "w"), indent=2)
@classmethod
- def load(cls, repo, root, runner=None, model="agent", external_facts=None):
+ def load(cls, repo, root, runner=None, model="agent", external_facts=None, vocab=None):
p = cls._path(root, repo)
- self = cls(repo, runner, model, external_facts=external_facts)
+ self = cls(repo, runner, model, external_facts=external_facts, vocab=vocab)
if not os.path.exists(p):
return self
d = json.load(open(p))
@@ -608,6 +765,8 @@ class Distiller:
self.last_propose = d.get("last_propose", -1)
self.extractors = {k: Extractor(v) for k, v in d.get("extractors", {}).items()}
self.facts = {k: Fact.from_dict(v) for k, v in d.get("facts", {}).items()}
+ self.window = list(d.get("window") or [])[-WINDOW:]
+ self.raw_window = [[(c, "") for c in cmds] for cmds in d.get("raw_cmds") or []]
return self
def debug_row(self):
diff --git a/tokeneconomizer/microskills/envfacts.py b/tokeneconomizer/microskills/envfacts.py
index 7a0e857..a262f06 100644
--- a/tokeneconomizer/microskills/envfacts.py
+++ b/tokeneconomizer/microskills/envfacts.py
@@ -44,6 +44,8 @@ STATE_SUBDIR = "envfacts_state"
MAX_FACTS = 4
MAX_TOKENS = 150 # ~4 chars per token; the whole block, header included
HEADER = "# environment facts learned from earlier tasks in this checkout"
+# The plugin's (`Vocabulary.generic`): a person's work comes in sessions, not tasks.
+GENERIC_HEADER = "# environment facts learned from earlier sessions in this repository"
# Words that would turn a fact into completion or verification advice.
_LINT_RE = re.compile(r"\b(skip|no need|don'?t|do not|stop|done|completed?|finish(?:ed)?|"
@@ -173,9 +175,10 @@ def typed_ok(o):
def _repairs(form, a_form, intent):
- """Can `form` stand in for the failing `a_form`? Another form -- and, outside test runs,
- not a bare one: only a fuller spelling repairs a bare interpreter or package tool."""
- return form != a_form and (intent == "run_tests" or not is_bare(form))
+ """Can `form` stand in for the failing `a_form`? Another form -- and, outside test runs
+ and builds, not a bare one: only a fuller spelling repairs a bare interpreter or package
+ tool."""
+ return form != a_form and (intent in ("run_tests", "run_build") or not is_bare(form))
def _moves_shell(form):
@@ -244,12 +247,16 @@ class EnvFactsDistiller:
self.last_digest = None # the most recent per-task verdicts (telemetry)
# ---- render --------------------------------------------------------------------
+ @property
+ def header(self):
+ return GENERIC_HEADER if self.vocab.generic else HEADER
+
def render(self, task=None):
chosen = self._select()
self.last_injected = {f.id for f in chosen}
if not chosen:
return ""
- lines = [HEADER] + [f"- {t}" for t in self._merged_lines(chosen)]
+ lines = [self.header] + [f"- {t}" for t in self._merged_lines(chosen)]
return "\n".join(lines) + "\n"
def injected_ids(self):
@@ -280,7 +287,7 @@ class EnvFactsDistiller:
trial = chosen + [f]
if len(trial) > self.max_facts:
break
- text = HEADER + "\n" + "\n".join(f"- {t}" for t in self._merged_lines(trial))
+ text = self.header + "\n" + "\n".join(f"- {t}" for t in self._merged_lines(trial))
if len(text) > self.max_tokens * 4:
continue
chosen = trial
@@ -324,14 +331,14 @@ class EnvFactsDistiller:
by_intent = {}
for o in obs:
by_intent.setdefault(o.intent, []).append(o)
- for intent in ("run_interpreter", "run_tests", "package_tool"):
+ for intent in ("run_interpreter", "run_tests", "run_build", "package_tool"):
seq = by_intent.get(intent, [])
# A test run that failed because its LABEL did not load says nothing about the way
# the runner was called (Django reports a bad label as a ModuleNotFoundError): those
# are ArgRepair's. Counting them here once taught "`python tests/runtests.py` fails;
# `python runtests.py` works" from a stream of mistyped labels.
fails = [o for o in seq if form_failed(o)]
- if intent != "run_tests":
+ if intent not in ("run_tests", "run_build"):
# An explicit path or activation that fails is a missing dependency, not a
# resolution problem; only a bare form can be "repaired" by a fuller one.
fails = [o for o in fails if is_bare(o.form)]
diff --git a/tokeneconomizer/microskills/lifecycle.py b/tokeneconomizer/microskills/lifecycle.py
index e719d41..63cf4a8 100644
--- a/tokeneconomizer/microskills/lifecycle.py
+++ b/tokeneconomizer/microskills/lifecycle.py
@@ -35,6 +35,21 @@ def edited_pyfiles(calls):
return fs
+def edited_files(calls):
+ """Every file an edit tool wrote, by basename, whatever its language: the plugin's reading
+ (`Vocabulary.generic`). The experiments count Python files only (`edited_pyfiles`)."""
+ fs = set()
+ for tool, args, _ in calls:
+ if tool in ("edit", "multiedit", "write", "create", "str_replace_editor"):
+ p = args.get("file_path") or args.get("path") or ""
+ if p:
+ fs.add(os.path.basename(p))
+ elif tool == "apply_patch":
+ for p in args.get("paths") or ():
+ fs.add(os.path.basename(p))
+ return fs
+
+
class Lifecycle:
def __init__(self, key, high, drift=True):
self.key = key
diff --git a/tokeneconomizer/microskills/observe.py b/tokeneconomizer/microskills/observe.py
index e3c5380..d720ebd 100644
--- a/tokeneconomizer/microskills/observe.py
+++ b/tokeneconomizer/microskills/observe.py
@@ -15,7 +15,7 @@ import re
from dataclasses import dataclass
from ..trace import call_cwds, load_trajectory, strip_rtk_prefix
-from .lifecycle import edited_pyfiles
+from .lifecycle import edited_files, edited_pyfiles
# --------------------------------------------------------------------------------------
@@ -35,6 +35,14 @@ class Vocabulary:
package_tools: frozenset
fetch_heads: frozenset = frozenset({"curl", "wget"})
env_probe_heads: frozenset = frozenset({"which", "command", "type", "whereis"})
+ # Tools whose `<tool> test` runs the tests.
+ task_heads: frozenset = frozenset({"make", "npm", "yarn", "pnpm", "cargo", "go"})
+ # Executables that build, compile, lint or type-check (the `run_build` intent).
+ build_heads: frozenset = frozenset()
+ # The language-neutral reading the plugin uses: builds and checks are an intent of their
+ # own, every edited file counts, and the proposer reads the generic digest
+ # (`distiller.build_generic_digest`). The experiments' vocabularies keep their narrower one.
+ generic: bool = False
PYTHON_GIT = Vocabulary(
@@ -59,6 +67,31 @@ NODE_WEB = Vocabulary(
package_tools=frozenset({"npm", "yarn", "pnpm", "corepack", "apt", "apt-get"}),
)
+# The plugin's: every ecosystem at once, since a repository a person works in may mix them.
+GENERIC = Vocabulary(
+ name="generic",
+ interpreters=PYTHON_GIT.interpreters | NODE_WEB.interpreters
+ | frozenset({"ruby", "perl", "php", "Rscript", "julia", "lua", "elixir"}),
+ canonical_interpreter="python",
+ runner_heads=PYTHON_GIT.runner_heads | NODE_WEB.runner_heads
+ | frozenset({"rspec", "phpunit", "ctest", "bats"}),
+ runner_scripts=PYTHON_GIT.runner_scripts,
+ runner_modules=PYTHON_GIT.runner_modules,
+ package_tools=PYTHON_GIT.package_tools | NODE_WEB.package_tools
+ | frozenset({"gem", "bundle", "composer", "cargo", "go", "mvn", "gradle", "dotnet", "brew",
+ "rustup", "pyenv", "nvm", "asdf"}),
+ task_heads=frozenset({"make", "npm", "yarn", "pnpm", "cargo", "go", "mvn", "gradle",
+ "gradlew", "dotnet", "mix", "swift", "rake"}),
+ build_heads=frozenset({"make", "cmake", "ninja", "bazel", "tsc", "eslint", "prettier",
+ "ruff", "flake8", "pylint", "mypy", "pyright", "black", "isort",
+ "golangci-lint", "gcc", "g++", "clang", "javac", "webpack", "vite"}),
+ generic=True,
+)
+# `<tool> build`, `npm run lint`, ...: what a generic reading counts as building or checking.
+BUILD_SUBS = frozenset({"build", "check", "compile", "lint", "vet", "typecheck", "type-check",
+ "clippy", "package", "verify", "fmt", "format"})
+_RUN_SCRIPT_HEADS = frozenset({"npm", "yarn", "pnpm", "bun"})
+
DEFAULT_VOCAB = PYTHON_GIT
# Module-level aliases of the default vocabulary, kept for the offline analyses.
INTERPRETERS = PYTHON_GIT.interpreters
@@ -191,10 +224,32 @@ class Observation:
return f"Obs({self.intent},{self.form!r},{self.outcome})"
-def _cd_target(cur, target, root=None):
+def repository_top(start):
+ """The repository a directory belongs to: the nearest directory at or above `start` that
+ holds a `.git` (a directory, or a worktree's file); None when there is none on disk, or
+ `start` itself is gone (a removed checkout must not borrow an enclosing repository)."""
+ if not start or not os.path.isabs(start) or not os.path.isdir(start):
+ return None
+ d = os.path.normpath(start)
+ while True:
+ if os.path.exists(os.path.join(d, ".git")):
+ return d
+ up = os.path.dirname(d)
+ if up == d:
+ return None
+ d = up
+
+
+def _cd_target(cur, target, root=None, top=None):
"""Where `cd target` from `cur` lands, relative to the directory the session started in
("" for that directory itself); None when it leaves it or cannot be told (`~`, `$VAR`,
- `-`, an absolute path with no known start)."""
+ `-`, an absolute path with no known start).
+
+ A session started in a subdirectory of its repository (`top`) that changes up to the
+ repository's top runs what a session started there runs: `cd <top> && pytest` is the plain
+ `pytest`, not a form of its own spelled with one machine's path. Any other directory above
+ the start is taken as typed: relative to `top` it would hold only for sessions that start
+ there."""
if not target or target[0] in "~$-":
return None
if os.path.isabs(target):
@@ -205,10 +260,15 @@ def _cd_target(cur, target, root=None):
rel = os.path.normpath(os.path.join(cur or ".", target))
if rel == ".":
return ""
- return None if rel == ".." or rel.startswith("../") else rel
+ if rel == ".." or rel.startswith("../"):
+ if not (top and root):
+ return None
+ at = os.path.normpath(os.path.join(root, rel))
+ return "" if at == os.path.normpath(top) else None
+ return rel
-def classify_command(cmd, call_index=0, vocab=None, cwd="", root=None):
+def classify_command(cmd, call_index=0, vocab=None, cwd="", root=None, top=None):
"""Observations for one shell command (possibly compound). Activation prefixes carry
across the segments of one command, which is how `source activate X && python ...`
becomes a single form. `vocab` names the domain's intents (default Python/git).
@@ -216,7 +276,8 @@ def classify_command(cmd, call_index=0, vocab=None, cwd="", root=None):
`cwd` is the directory the command started in, relative to the one the session started
in (`root`, an absolute path when known): a shell that keeps its directory between calls
runs a bare `python runtests.py` in `tests/` after an earlier `cd tests`, and that is the
- same form as `cd tests && python runtests.py` typed at the start."""
+ same form as `cd tests && python runtests.py` typed at the start. `top` is the repository
+ `root` lies in (`repository_top`), for a `cd` up to it from a subdirectory (`_cd_target`)."""
vocab = vocab or DEFAULT_VOCAB
obs = []
carried = ""
@@ -240,7 +301,7 @@ def classify_command(cmd, call_index=0, vocab=None, cwd="", root=None):
# the forms that follow, as an activation is.
m = _CD_RE.match(seg)
if m:
- there = _cd_target(here, m.group(2), root) if here is not None else None
+ there = _cd_target(here, m.group(2), root, top) if here is not None else None
if there is None:
here, cd = None, f"cd {m.group(2)} &&"
else:
@@ -322,8 +383,22 @@ def _classify(head, base, rest, body, prefix, seg, call, vocab):
return Observation(call, "run_tests",
form(f"{vocab.canonical_interpreter} " + head.lstrip("./")),
head, prefix, seg)
- if base in ("make", "npm", "yarn", "pnpm", "cargo", "go") and rest and rest[0] == "test":
+ if base in vocab.task_heads and rest and rest[0] == "test":
return Observation(call, "run_tests", form(f"{base} test"), head, prefix, seg)
+ if vocab.generic:
+ if base in _RUN_SCRIPT_HEADS and len(rest) > 1 and rest[0] == "run":
+ script = rest[1]
+ kind = script.split(":", 1)[0]
+ if kind == "test":
+ return Observation(call, "run_tests", form(f"{base} run {script}"), head, prefix,
+ seg)
+ if kind in BUILD_SUBS:
+ return Observation(call, "run_build", form(f"{base} run {script}"), head, prefix,
+ seg)
+ if base in vocab.task_heads and rest and rest[0] in BUILD_SUBS:
+ return Observation(call, "run_build", form(f"{base} {rest[0]}"), head, prefix, seg)
+ if base in vocab.build_heads:
+ return Observation(call, "run_build", form(base), head, prefix, seg)
if base in vocab.env_probe_heads:
return Observation(call, "probe_env", form(base), head, prefix, seg)
if base in vocab.fetch_heads:
@@ -400,6 +475,12 @@ def _label_failed(labels, out):
# (`| grep FAILED`, a silent run) proves nothing either way.
_LOADED_RE = re.compile(r"\bRan [1-9]\d* tests?\b|\b[1-9]\d* (?:passed|failed|passing|failing)\b|"
r"\bcollected [1-9]\d* items?\b")
+# Other runners' summaries, for the generic reading: Go's `ok pkg 0.01s` and `--- PASS`, Rust's
+# `test result:`, RSpec's `3 examples, 0 failures`, .NET's `Passed!`, Maven's `Tests run: 3`.
+_LOADED_GENERIC_RE = re.compile(r"^ok\s+\S+\s+[\d.]+s\b|^--- (?:PASS|FAIL)\b|"
+ r"\btest result: (?:ok|FAILED)\b|"
+ r"\b[1-9]\d* examples?, \d+ failures?\b|"
+ r"\b(?:Passed|Failed)!\s+-|\bTests run: [1-9]\d*", re.M)
# Code asking for an environment variable by name, and the ways a command sets one.
_ENV_NEED_RE = re.compile(r"environment variable\s+['\"`]?([A-Z][A-Z0-9_]{2,})")
_ENV_PREFIX_RE = re.compile(r"(?:^|[\s;&(])(?:export\s+)?([A-Z][A-Z0-9_]{2,})=(['\"]?)([\w.\-/:]+)\2")
@@ -491,9 +572,11 @@ def observe_task(transcript_path, vocab=None):
having run; a failed call with no infrastructure error marks only the first relevant
segment as having run and leaves the others unknown (an `&&` chain may not have reached
them)."""
+ vocab = vocab or DEFAULT_VOCAB
calls = load_trajectory(transcript_path)
cwds = call_cwds(transcript_path)
root = next((c for c in cwds if c), None)
+ top = repository_top(root)
obs_all = []
triples = []
env_calls = []
@@ -508,14 +591,13 @@ def observe_task(transcript_path, vocab=None):
continue # stopped at its `cd`: nothing after it ran
here = ""
if root and idx < len(cwds) and cwds[idx]:
- here = _cd_target("", cwds[idx], root)
- here = here if here is not None else None
+ here = _cd_target("", cwds[idx], root, top)
if here is None:
# Started outside the session's directory: its own `cd`s are taken as typed.
obs = classify_command(cmd, idx, vocab)
else:
- obs = classify_command(cmd, idx, vocab, cwd=here, root=root)
- if "python" in cmd or "=" in cmd:
+ obs = classify_command(cmd, idx, vocab, cwd=here, root=root, top=top)
+ if vocab.generic or "python" in cmd or "=" in cmd:
ec = _env_calls(cmd, out, ok if _exit_code_hint(out) is None else
_exit_code_hint(out) == 0)
if ec["need"] or ec["sets"]:
@@ -527,7 +609,8 @@ def observe_task(transcript_path, vocab=None):
o.labels = _test_labels(o)
o.label_err = bool(o.labels) and _label_failed(o.labels, out)
o.loaded = bool(o.labels) and not o.label_err and bool(_LOADED_RE.search(out))
- o.tests_ran = bool(_LOADED_RE.search(out))
+ o.tests_ran = bool(_LOADED_RE.search(out) or (
+ vocab.generic and _LOADED_GENERIC_RE.search(out)))
# Pass-through segments (a leading `cd`, a trailing `| head`, an `echo ---`
# separator) neither add evidence nor pollute the output, so they do not break a
# call's homogeneity; filters that rewrite content (grep, sed, awk, wc) do.
@@ -539,7 +622,8 @@ def observe_task(transcript_path, vocab=None):
err = error_class(out)
if err in FORM_ERRORS or err in NETWORK_ERRORS:
targets = [o for o in obs if o.intent in ("run_interpreter", "run_tests",
- "package_tool", "fetch_remote")]
+ "run_build", "package_tool",
+ "fetch_remote")]
named = [o for o in targets
if (o.head and o.head in out) or (o.head and o.head.rsplit("/", 1)[-1] in
_error_line(out, err))]
@@ -560,7 +644,8 @@ def observe_task(transcript_path, vocab=None):
else:
first = True
for o in obs:
- if o.intent in ("run_interpreter", "run_tests", "package_tool", "fetch_remote"):
+ if o.intent in ("run_interpreter", "run_tests", "run_build", "package_tool",
+ "fetch_remote"):
o.outcome = "ran" if first else None
first = False
else:
@@ -581,7 +666,8 @@ def observe_task(transcript_path, vocab=None):
if not homogeneous or not success or not out.strip() or err in NETWORK_ERRORS:
o.outcome = None
obs_all.extend(obs)
- return {"obs": obs_all, "edited": sorted(edited_pyfiles(triples)), "n_calls": len(calls),
+ edited = edited_files(triples) if vocab.generic else edited_pyfiles(triples)
+ return {"obs": obs_all, "edited": sorted(edited), "n_calls": len(calls),
"env": env_calls}
diff --git a/tokeneconomizer/microskills/stage.py b/tokeneconomizer/microskills/stage.py
index 3208688..430a4ef 100644
--- a/tokeneconomizer/microskills/stage.py
+++ b/tokeneconomizer/microskills/stage.py
@@ -94,9 +94,10 @@ class PrefixStage(Stage):
# layer already injects, so it authors only facts beyond them.
external = env.injectable_claims if env is not None else None
dist = (Distiller(repo, runner=runner, model=optimizer_model,
- external_facts=external) if fresh
+ external_facts=external, vocab=vocab) if fresh
else Distiller.load(repo, results_root, runner=runner,
- model=optimizer_model, external_facts=external))
+ model=optimizer_model, external_facts=external,
+ vocab=vocab))
if env is not None:
self.reason = "envfacts+manual-distill"
self.distiller = ComposedDistiller(env, dist)
diff --git a/tokeneconomizer/plugin/README.md b/tokeneconomizer/plugin/README.md
index ba26484..10b4ace 100644
--- a/tokeneconomizer/plugin/README.md
+++ b/tokeneconomizer/plugin/README.md
@@ -19,7 +19,7 @@ What happens at each hook, and how each agent was verified to behave, is in
| `context.py`, `paths.py`, `store.py` | where state lives, which repository a session is in, locked atomic JSON |
| `adapter.py` | the `DomainAdapter` for a real repository |
| `accounting.py` | the session ledger |
-| `sidecall.py`, `judge.py` | opt-in paid calls through the agent's own CLI, off the hook's critical path |
+| `sidecall.py`, `judge.py` | the plugin's model calls (the loop guard's judge, micro-skills' proposer) through the agent's own CLI, off the hook's critical path |
| `repomap.py`, `repomap_gen.py` | the opt-in repository map |
| `probe.py` | `te --host copilot probe`: the plugin end to end in real Copilot sessions |
diff --git a/tokeneconomizer/plugin/__init__.py b/tokeneconomizer/plugin/__init__.py
index bfe5e1a..7306376 100644
--- a/tokeneconomizer/plugin/__init__.py
+++ b/tokeneconomizer/plugin/__init__.py
@@ -14,7 +14,8 @@ fails open.
* after a session -- `fold`: run `PrefixStage` and `ToolingStage` over the finished session,
off the agent's critical path; `denylist` writes an approved proposal into
the agent's own settings file;
- * paid, opt-in -- `sidecall`, `judge`: LLM calls through the agent's own CLI, charged to the
+ * model calls -- `sidecall`, `judge`: LLM calls through the agent's own CLI (the proposer
+ with micro-skills on, the judge with the loop guard on), charged to the
session's ledger (`accounting`);
* commands -- `hook.main` (what the hooks execute) and `cli.main` (`te`), with
`toolcheck`, `repomap` and `probe` behind it.
diff --git a/tokeneconomizer/plugin/adapter.py b/tokeneconomizer/plugin/adapter.py
index 9d79519..9c49ef9 100644
--- a/tokeneconomizer/plugin/adapter.py
+++ b/tokeneconomizer/plugin/adapter.py
@@ -15,16 +15,20 @@ answers fit an interactive session instead of a sealed benchmark task:
union of a probed snapshot (`te toolprune refresh-catalog`, stored under the state root), the
deferred tools the transcript lists, and the tools it used. Without a snapshot only deferred
tools can be proposed -- they cost one listing line each, so that is honest but small.
- * `env_vocabulary` picks the environment-fact vocabulary from the repository's own markers
- (`package.json` -> Node, Python packaging files -> Python), overridable by config.
+ * `env_vocabulary` is the generic one (`microskills.GENERIC`): every ecosystem at once, builds
+ and checks as well as tests, every edited file, and the proposer's language-neutral digest.
+ The repository's markers only pick how a script run through its shebang is spelled
+ (`package.json` alone -> `node`). The experiments' narrower vocabularies stay selectable
+ (`vocabulary=python-git` / `node-web`).
* `problem_statements` is empty: there is no issue text to give the per-task tool judge.
"""
+import dataclasses
import json
import os
from .. import trace
from ..loopguard import state_from_trajectory
-from ..microskills import NODE_WEB, PYTHON_GIT
+from ..microskills import GENERIC, NODE_WEB, PYTHON_GIT
from ..toolprune.tools import PROTECTED_TOOLS
# Tools a person relies on in an interactive session, beyond the benchmark work surface.
@@ -66,8 +70,8 @@ def detect_vocabulary(repo_root, preference="auto"):
return PYTHON_GIT
has = lambda names: any(os.path.exists(os.path.join(repo_root or "", n)) for n in names)
if has(_NODE_MARKERS) and not has(_PY_MARKERS):
- return NODE_WEB
- return PYTHON_GIT
+ return dataclasses.replace(GENERIC, canonical_interpreter="node")
+ return GENERIC
def catalog_path(state_root, host):
diff --git a/tokeneconomizer/plugin/cli.py b/tokeneconomizer/plugin/cli.py
index d1f7217..6e1785c 100644
--- a/tokeneconomizer/plugin/cli.py
+++ b/tokeneconomizer/plugin/cli.py
@@ -42,8 +42,8 @@ import sys
import time
from .. import loopguard
-from . import accounting, denylist, fold, hosts, paths, repomap, repos, toolcheck
-from .config import unset_config, write_config
+from . import accounting, denylist, fold, hosts, paths, repomap, repos, sidecall, toolcheck
+from .config import canonical, unset_config, write_config
from .context import LiveContext
from .paths import state_root
@@ -118,7 +118,7 @@ def cmd_status(args):
cfg.loopguard_thresholds)
ledger = accounting.summarize(ctx.store.read_lines("ledger.jsonl"))
out.update({
- "microskills": {"profile": cfg.microskills_effective,
+ "microskills": {"profile": cfg.microskills,
"injected_lines": sum(1 for ln in (inj.get("rendered") or "").splitlines()
if ln.startswith("- ")),
"sessions_folded": inj.get("sessions", 0)},
@@ -137,8 +137,8 @@ def cmd_status(args):
print(f"TokenEconomizer -- {out['repo']} ({ctx.host.name})")
print(f" repository {ctx.repo.root}, {repos.describe(ctx.repo)}")
print(f" state {out['state']}")
- print(f" config microskills={cfg.microskills_effective} toolprune={cfg.toolprune} "
- f"loopguard={cfg.loopguard} repomap={cfg.repomap} mode={cfg.mode} paid={cfg.paid} "
+ print(f" config microskills={cfg.microskills} toolprune={cfg.toolprune} "
+ f"loopguard={cfg.loopguard} repomap={cfg.repomap} mode={cfg.mode} "
f"shadow={cfg.shadow}")
if cfg.repomap != "off":
rm = repomap.read_state(ctx)
@@ -177,6 +177,12 @@ def cmd_status(args):
def cmd_microskills(args):
ctx = _ctx(args)
inj = ctx.store.read("injection.json") or {}
+ if ctx.config.microskills == "off":
+ print(f"micro-skills is off here (the default). `config set microskills=on` turns it on: "
+ f"facts are then learned from finished sessions, and a model proposes more every "
+ f"{ctx.config.microskills_propose_every} sessions, through this agent")
+ if not inj:
+ return 0
text = inj.get("rendered") or ""
print(text.rstrip() or "(nothing injected yet: no fact has cleared its firing gate)")
facts = inj.get("facts") or {}
@@ -481,6 +487,14 @@ def cmd_config(args):
print(f"{path}: {json.dumps(cur, sort_keys=True)}")
print("applies from the next hook call; a new fact-size cap applies once the next session "
"has been folded")
+ if {canonical(k) for k in updates} & {"microskills", "loopguard"} and sidecall.resume(
+ ctx.host.name, ctx.root):
+ print(f"the plugin's model calls through `{ctx.host.name} -p` will be tried again")
+ if "distiller" in str(cur.get("microskills", "")):
+ every = cur.get("microskills_propose_every") or ctx.config.microskills_propose_every
+ print(f"note: micro-skills calls a model through this agent every {every} sessions to "
+ "propose facts, charged to the session's cost; `microskills=envfacts` learns "
+ "without model calls")
if cur.get("loopguard") == "stop":
print("note: `stop` acts as `nudge`: the plugin never stops a turn; a turn still stuck "
"after the redirect gets a notice saying how to stop it")
@@ -499,7 +513,7 @@ def cmd_fold(args):
def cmd_judge(args):
- """The detached half of the paid stuck judge (see plugin/judge.py); not for people."""
+ """The detached half of the stuck judge (see plugin/judge.py); not for people."""
from . import judge
judge.run_job(_session_ctx(args, args.session), args.session, args.job)
return 0
@@ -613,6 +627,13 @@ def cmd_doctor(args):
# Only the agents installed here: a Codex-only machine has no Claude Code catalog to probe.
agents = [h for h in ("claude", "codex", "copilot", "pi") if _sh.which(h)]
print(f"agents {', '.join(agents) or 'none of claude, codex, copilot, pi on PATH'}")
+ for h in agents:
+ pause = sidecall.paused(h, root)
+ if pause:
+ print(f"{h:11s} model calls paused since "
+ f"{time.strftime('%Y-%m-%d %H:%M', time.localtime(pause['at']))}: `{h} -p` said "
+ f"it is not signed in; `config set microskills=on` (or `loopguard=on`) "
+ f"tries again")
if "claude" in agents:
cat = os.path.join(root, "hosts", "claude", "catalog.json")
print(f"claude catalog {'present' if os.path.exists(cat) else 'not probed'}")
@@ -633,7 +654,8 @@ def cmd_doctor(args):
f"{copilot_model() or '(Copilot default)'}")
if base:
print("copilot note: Copilot does not pass COPILOT_PROVIDER_* to the plugin's "
- "hooks, so with paid=true its side calls cannot use this provider")
+ "hooks, so its model calls (micro-skills' proposer, the loop guard's judge) "
+ "cannot use this provider")
ctrust = denylist.copilot_folder_trusted(args.cwd)
if ctrust is not None:
print(f"copilot this folder is {'trusted' if ctrust else 'NOT trusted'} (an "
diff --git a/tokeneconomizer/plugin/config.py b/tokeneconomizer/plugin/config.py
index a5015b3..557c131 100644
--- a/tokeneconomizer/plugin/config.py
+++ b/tokeneconomizer/plugin/config.py
@@ -2,21 +2,27 @@
The defaults:
- * microskills = "envfacts" -- the LLM-free facts only. The paid proposer (`envfacts+distiller`,
- `distiller`) is opt-in, and refused unless `paid` is also true;
+ * microskills = "off" -- nothing is learned or injected. `on` is "envfacts+distiller":
+ the LLM-free facts plus the proposer, which reads digests of
+ finished sessions every `microskills_propose_every` sessions and
+ calls a model through the agent's own CLI (charged to the
+ ledger) to write facts beyond them, as in the experiments.
+ "envfacts" alone makes no model call;
* toolprune = "auto" -- a deny list is applied once the evidence forms it and the
repository's sessions have been idle (toolprune_quiet_minutes),
from the next session, and announced once at that session's start;
`toolprune restore` gives it back. "propose" only proposes it,
and writes nothing to the agent's settings until the person
runs `toolprune apply`;
- * loopguard = "nudge" -- at most one mid-turn redirect per prompt-turn; a turn still
- stuck after it gets a notice to the person, saying how to stop
- it. The plugin never stops a turn itself. "stop", which used to
- stop a headless turn on Claude Code and Pi, is still read, and
- acts as "nudge" with a warning;
- * paid = False -- no mechanism makes an LLM call of its own. Every paid call that is
- allowed is charged to the session's ledger;
+ * loopguard = "nudge" -- a turn that shows signs of looping is put to a judge, a model
+ called through the agent's own CLI, and redirected mid-turn only
+ when the judge finds it stuck (the redirect written by that
+ model), at most once per prompt-turn; a turn still stuck after it
+ gets a notice to the person, saying how to stop it. The plugin
+ never stops a turn itself. "stop", which used to stop a headless
+ turn on Claude Code and Pi, is still read, and acts as "nudge"
+ with a warning. Every model call the plugin makes is charged to
+ the session's ledger;
* shadow = False -- with True, every decision is computed and logged but none
reaches the agent: the way to measure a mechanism before
trusting it.
@@ -25,7 +31,7 @@ Each mechanism also takes a plain `on` / `off` (`on` = the value above), and has
person can tune by hand:
* microskills_max_facts, microskills_max_tokens -- caps on the injected block;
- * microskills_propose_every -- sessions between paid proposer runs;
+ * microskills_propose_every -- sessions between proposer runs;
* toolprune_evidence -- prompts a never-used tool must have stayed unused through,
over at least toolprune_min_sessions sessions, before it is
proposed -- or over toolprune_sessions sessions, however short;
@@ -62,7 +68,7 @@ MODES = ("auto", "interactive", "headless")
AGGR = ("conservative", "default", "aggressive")
# What `on` turns each mechanism to, and the words read as on / off.
-ON_VALUE = {"microskills": "envfacts", "toolprune": "auto", "loopguard": "nudge",
+ON_VALUE = {"microskills": "envfacts+distiller", "toolprune": "auto", "loopguard": "nudge",
"repomap": "on"}
_ON = ("on", "true", "yes", "1", "enable", "enabled")
_OFF = ("off", "false", "no", "0", "disable", "disabled")
@@ -84,6 +90,14 @@ ALIASES = {
}
_THRESHOLD_PREFIXES = ("loopguard.", "spiral.")
+# Settings that no longer exist, and where what they did went. A config file that still has one
+# is read without complaint; `config set` says why it is refused.
+OBSOLETE = {
+ "paid": "the loop guard always asks its judge (`loopguard=off` turns it off), and "
+ "micro-skills' proposer runs whenever micro-skills is on (`microskills=envfacts` "
+ "keeps only the facts that need no model call)",
+}
+
# The loop guard's thresholds a person may pin (`loopguard.<name>=<value>`), with their type
# and range. They are the keys of `loopguard.DEFAULT_THRESHOLDS`; a turn's "turns" are its tool
# calls here.
@@ -98,9 +112,9 @@ LOOPGUARD_KEYS = {
"envelope_min_samples": (int, 1, None, "ended-green turns needed before the envelope is used"),
"backstop_mult": (float, 0.1, None, "stuck past this multiple of the envelope: redirect, "
"then stop"),
- "judge_confidence": (float, 0.0, 1.0, "paid judge confidence needed to act (paid=true)"),
- "gate_min_turns": (int, 1, None, "tool calls before the paid judge is consulted"),
- "gate_error_outputs": (int, 1, None, "failing results that wake the paid judge early"),
+ "judge_confidence": (float, 0.0, 1.0, "the judge's confidence needed to redirect"),
+ "gate_min_turns": (int, 1, None, "tool calls before the judge is consulted"),
+ "gate_error_outputs": (int, 1, None, "failing results that wake the judge without a repeat"),
}
@@ -124,10 +138,10 @@ def _threshold_name(key):
class LiveConfig:
enabled: bool = True
mode: str = "auto"
- microskills: str = "envfacts"
+ microskills: str = "off"
microskills_max_facts: int = 4
microskills_max_tokens: int = 150
- microskills_propose_every: int = 5
+ microskills_propose_every: int = 3
toolprune: str = "auto"
toolprune_evidence: int = 10
toolprune_min_sessions: int = 2
@@ -141,7 +155,6 @@ class LiveConfig:
repomap_tokens: int = 2048
repomap_python: str = ""
repomap_timeout: int = 8
- paid: bool = False
paid_model: str = ""
shadow: bool = False
vocabulary: str = "auto"
@@ -149,7 +162,7 @@ class LiveConfig:
_CHOICES = {"mode": MODES, "microskills": MICROSKILLS, "toolprune": TOOLPRUNE,
"loopguard": LOOPGUARD, "repomap": REPOMAP, "loopguard_aggr": AGGR,
- "vocabulary": ("auto", "python-git", "node-web")}
+ "vocabulary": ("auto", "generic", "python-git", "node-web")}
# The smallest value a number setting takes, where it is not 1.
_MIN = {"toolprune_quiet_minutes": 0}
@@ -164,6 +177,8 @@ class LiveConfig:
if name is not None:
return self._set_threshold(name, value, source)
key = canonical(key)
+ if key in OBSOLETE:
+ return True
names = {f.name: f for f in fields(self) if f.name != "warnings"}
if key not in names:
self.warnings.append(f"{source}: unknown key {key!r}")
@@ -224,17 +239,9 @@ class LiveConfig:
self.loopguard_thresholds[name] = v
return True
- @property
- def microskills_effective(self):
- """The micro-skill profile that will actually run: a paid profile without `paid` falls
- back to the free layer, and says so in `warnings` when the config is loaded."""
- if self.microskills in ("envfacts+distiller", "distiller") and not self.paid:
- return "envfacts"
- return self.microskills
-
# Environment variables, old names first so that a new name set alongside its old one wins.
-_ENV = {"TE_MODE": "mode", "TE_PAID": "paid", "TE_SHADOW": "shadow",
+_ENV = {"TE_MODE": "mode", "TE_SHADOW": "shadow",
"TE_PAID_MODEL": "paid_model", "TE_VOCABULARY": "vocabulary",
"TE_INJECT": "microskills", "TE_INJECT_MAX_FACTS": "microskills_max_facts",
"TE_INJECT_MAX_TOKENS": "microskills_max_tokens",
@@ -294,9 +301,6 @@ def load_config(root, repo_dir=None, env=None):
cfg.warnings.append("loopguard=stop acts as nudge: the plugin no longer stops a turn; a "
"turn still stuck after the redirect gets a notice saying how to stop "
"it (set loopguard=nudge to silence this)")
- if cfg.microskills != cfg.microskills_effective:
- cfg.warnings.append(f"microskills={cfg.microskills!r} makes paid calls but paid=false; "
- f"running {cfg.microskills_effective!r}")
return cfg
@@ -310,6 +314,10 @@ def _drop_old_names(current, key):
def write_config(path, updates):
"""Merge `updates` into the JSON config at `path` (validating each key first), under the
settings' current names. A `loopguard.<name>` key lands in `loopguard_thresholds`."""
+ gone = [k for k in updates if canonical(k) in OBSOLETE]
+ if gone:
+ raise ValueError("; ".join(f"`{k}` no longer exists: {OBSOLETE[canonical(k)]}"
+ for k in gone))
probe = LiveConfig()
bad = [k for k, v in updates.items() if not probe.set(k, v, source="te config")]
if bad:
diff --git a/tokeneconomizer/plugin/controller.py b/tokeneconomizer/plugin/controller.py
index 3d4f54f..4fa0f92 100644
--- a/tokeneconomizer/plugin/controller.py
+++ b/tokeneconomizer/plugin/controller.py
@@ -12,9 +12,9 @@
Every handler returns quickly and fails open. A hook that raises would break the user's session
for a cost optimization, so `hook.main` turns any exception into an empty output, and nothing
-here waits on the network: injection reads a cached render, and the only paid call a hook can
-cause -- the sampled stuck judge -- runs only when `paid` is on and the free gate is open, and
-then in a detached job (`judge`), never in the hook itself.
+here waits on the network: injection reads a cached render, and the only model call a hook can
+cause -- the sampled stuck judge -- runs only when the guard's free gate is open, and then in a
+detached job (`judge`), never in the hook itself.
Every visible action is announced in one `systemMessage` line, so a person can see why the agent
changed course; `shadow` computes and logs every decision without acting on any of them.
@@ -24,7 +24,7 @@ import os
import time
from .. import loopguard
-from . import denylist, fold, repomap, repos
+from . import denylist, fold, repomap, repos, sidecall
from .hosts import merge_outputs
from .paths import ensure_launcher
@@ -134,6 +134,20 @@ class LiveController:
and not self.cfg.shadow):
outputs.extend(self._toolprune_notices(seen))
+ pause = sidecall.paused(self.host.name, self.ctx.root)
+ calls_a_model = "distiller" in self.cfg.microskills or self.cfg.loopguard != "off"
+ if (pause and calls_a_model and not self.cfg.shadow
+ and seen.get("sidecall_paused") != pause["at"]):
+ byok = (" (with your own model key, Copilot does not pass it to the plugin's hooks)"
+ if self.host.name == "copilot" else "")
+ outputs.append(self.host.notice(
+ f"TokenEconomizer: the plugin's own model calls go through `{self.host.name} -p`, "
+ f"which answered that it is not signed in{byok}, so they are paused; facts that "
+ f"need no model call are still learned, and the loop guard redirects only at its "
+ f"backstop. `{self._cmd('config')} set microskills=on` (or `loopguard=on`) tries "
+ f"again."))
+ seen["sidecall_paused"] = pause["at"]
+
note = repos.migration_notice(self.ctx.root, self._cmd("repos"))
if note and not self.cfg.shadow:
outputs.append(self.host.notice(note))
@@ -261,13 +275,20 @@ class LiveController:
return {}
# ---- PostToolUse ---------------------------------------------------------------------
- def _judge(self, thr):
- """The paid stuck judge, run in the background (`judge.AsyncJudge`), or None: the
- hook only starts a job, and a later call picks up its verdict with the redirect already
- written, so the agent never waits on it."""
- if not self.cfg.paid:
+ def _judge(self, thr, envelope):
+ """The stuck judge, run in the background (`judge.AsyncJudge`): with the guard on, it
+ makes every call to redirect but the backstop's. The hook only starts a job, and a later
+ call picks up its verdict with the redirect already written, so the agent never waits.
+
+ None before this repository has an envelope -- the guard does not act on a repository
+ whose usual turns it has not learned (the rule fallback never acts without one either).
+ While this agent's model calls are paused, a judge that never answers: the turn is
+ spared, as a missing verdict is, rather than redirected on the rule alone."""
+ from .judge import AsyncJudge, NoVerdict
+ if envelope is None:
return None
- from .judge import AsyncJudge
+ if sidecall.paused(self.host.name, self.ctx.root):
+ return NoVerdict()
return AsyncJudge(self.ctx, self.sid, thr.get("judge_confidence"))
def post_tool(self):
@@ -289,8 +310,15 @@ class LiveController:
overrides=self.cfg.loopguard_thresholds)
envelope = tracker.envelope
thr = loopguard.thresholds(tracker, self.cfg.loopguard_thresholds)
+ # The judge is woken only by a sign of trouble -- a repeated failure, or dense errors --
+ # once a turn has run `min_turns` calls; never merely because the turn outran this
+ # repository's usual length, which interactive turns do all the time (the experiments'
+ # agents ran under a turn cap, hence their envelope gate).
+ thr = dict(thr, gate_over_envelope=False,
+ gate_min_turns=max(thr["gate_min_turns"], thr["min_turns"]))
decision = g.fold(name, args, text, code, envelope=envelope, thr=thr,
- judge=self._judge(thr), nudge_budget=self.cfg.loopguard_nudges)
+ judge=self._judge(thr, envelope),
+ nudge_budget=self.cfg.loopguard_nudges)
out = {}
if decision:
action, reason = decision
diff --git a/tokeneconomizer/plugin/fold.py b/tokeneconomizer/plugin/fold.py
index 7804408..2ba220c 100644
--- a/tokeneconomizer/plugin/fold.py
+++ b/tokeneconomizer/plugin/fold.py
@@ -64,7 +64,7 @@ def _run_dir(workdir, trace_path):
def _fold_microskills(ctx, adapter, run_dir, session_id, injected=None):
- profile = ctx.config.microskills_effective
+ profile = ctx.config.microskills
if profile == "off":
return None
from tokeneconomizer.config import InjectionConfig
@@ -73,7 +73,10 @@ def _fold_microskills(ctx, adapter, run_dir, session_id, injected=None):
propose_every=ctx.config.microskills_propose_every,
max_facts=ctx.config.microskills_max_facts,
max_tokens=ctx.config.microskills_max_tokens)
- runner = sidecall.runner_for(ctx.host.name, ctx.config) if icfg.paid else None
+ # Paused once the host's CLI said it is not signed in (`sidecall.paused`): the free layer
+ # still learns.
+ runner = (sidecall.runner_for(ctx.host.name, ctx.config)
+ if icfg.paid and not sidecall.paused(ctx.host.name, ctx.root) else None)
stage = PrefixStage(icfg, ctx.repo.label, ctx.mech_dir, False, runner=runner,
optimizer_model="agent", adapter=adapter, agent=ctx.host.name)
srow = {"resolved": None}
diff --git a/tokeneconomizer/plugin/judge.py b/tokeneconomizer/plugin/judge.py
index 25c0fe8..d4bd411 100644
--- a/tokeneconomizer/plugin/judge.py
+++ b/tokeneconomizer/plugin/judge.py
@@ -1,7 +1,7 @@
-"""The loop guard's paid judge, off the hook's critical path.
+"""The loop guard's judge, off the hook's critical path.
-With `paid` on, the guard's gate can ask an LLM whether a turn is stuck and, when it is, have a
-second call write the redirect. Both are side calls through the agent's own CLI (`sidecall`),
+With the guard on, its gate asks an LLM whether a turn is stuck and, when it is, has a second
+call write the redirect. Both are side calls through the agent's own CLI (`sidecall`),
and run inside a PostToolUse hook they would hold the agent for tens of seconds. So the hook
only STARTS a job here and returns: `AsyncJudge.start` writes the request and
spawns `te judge` detached; the job runs the judge, and the nudge writer only when the verdict is
@@ -50,13 +50,25 @@ class AsyncJudge:
return True, res.get("verdict") or None
+class NoVerdict:
+ """The judge while this agent's model calls are paused: it never answers, so the guard
+ spares the turn rather than act on its rule alone, and only the backstop remains."""
+
+ def start(self, guard, digest):
+ return None
+
+ def poll(self, guard):
+ return False, None
+
+
def run_job(ctx, session_id, job_id):
"""The detached half: judge the digest, write the redirect if stuck, charge the ledger."""
from ..loopguard import judge as loop_judge
req = ctx.store.read(REQUEST.format(sid=session_id)) or {}
if req.get("id") != job_id: # superseded by a newer request
return None
- runner = sidecall.runner_for(ctx.host.name, ctx.config)
+ runner = (None if sidecall.paused(ctx.host.name, ctx.root)
+ else sidecall.runner_for(ctx.host.name, ctx.config))
verdict, written = None, None
if runner is not None:
try:
diff --git a/tokeneconomizer/plugin/probe.py b/tokeneconomizer/plugin/probe.py
index c817b75..bd4c060 100644
--- a/tokeneconomizer/plugin/probe.py
+++ b/tokeneconomizer/plugin/probe.py
@@ -108,8 +108,10 @@ def _session_env(state, trap):
# one plugin per name (verified: one SessionStart hook with both present).
# The probe applies the deny list itself, for one measured session, and restores it: its
# sessions must only propose one, as `toolprune=auto` would apply it for all the rest.
+ # Micro-skills is off by default; the probe checks that a learned fact reaches the context,
+ # and the trap's fact is one the free layer learns without a model call.
env.update(TOKENECONOMIZER_HOME=state, PATH=trap + os.pathsep + env.get("PATH", ""),
- TE_TOOLPRUNE="propose")
+ TE_TOOLPRUNE="propose", TE_MICROSKILLS="envfacts")
return env
diff --git a/tokeneconomizer/plugin/sidecall.py b/tokeneconomizer/plugin/sidecall.py
index a38efb2..50fa543 100644
--- a/tokeneconomizer/plugin/sidecall.py
+++ b/tokeneconomizer/plugin/sidecall.py
@@ -1,4 +1,4 @@
-"""Paid side calls -- the proposer, the stuck judge, the nudge writer -- on the user's own agent.
+"""Side calls -- the proposer, the stuck judge, the nudge writer -- on the user's own agent.
The mechanisms call an optimizer as `runner(prompt, model) -> (text, effective_tokens)`
(`runner.AgentRunner.optimizer_call`). In the harness that call went to the agent's model through
@@ -11,8 +11,9 @@ so no extra API key is needed and the call is billed where the session is:
was charged to; in a user's session the cheapest capable model is the right default, and the
ledger records the real cost either way.
* Codex: `codex exec --json --disable hooks --ephemeral -s read-only`. Each such call carries
- Codex's full base prompt (~25k tokens, mostly cached) on the user's plan quota, which is why
- paid mechanisms default off and the Codex default injection is the free layer.
+ Codex's full base prompt (~25k tokens, mostly cached) on the user's plan quota, one reason
+ micro-skills (whose proposer calls it) defaults off and the loop guard's judge is woken
+ only by a sign of trouble.
* Copilot CLI: `copilot -p` as a pure text completion, as `te_harness.copilot.optimizer_call`
runs it: a neutral cwd, no custom instructions, no built-in MCP server, every tool denied
and excluded. Its usage is read from the call's own session log (`result.sessionId` ->
@@ -28,10 +29,14 @@ so no extra API key is needed and the call is billed where the session is:
Every side call runs with `TE_SIDECALL=1`, which the plugin's hook entry point treats as "do
nothing": without it the nested agent would run this plugin's hooks and recurse. A call that
returns no reply -- a CLI that is not signed in, say -- leaves one line in the plugin's
-errors.log, which `te doctor` shows. Nothing here runs unless `paid` is enabled.
+errors.log, which `te doctor` shows. One that answered it is not signed in also pauses that
+host's side calls (`paused`): every later call would fail the same way, and Copilot's leave an
+empty session behind each time. Nothing here runs unless micro-skills is on (the proposer)
+or the loop guard is (the judge and the nudge writer).
"""
import json
import os
+import re
import subprocess
import tempfile
@@ -45,22 +50,59 @@ def _env():
return dict(os.environ, TE_SIDECALL="1")
+# What a CLI says when it is not signed in -- Copilot's "To authenticate, ... run the '/login'
+# command" under BYOK, Claude Code's "Please run /login".
+_SIGNED_OUT_RE = re.compile(r"To authenticate|/login\b|\b[Nn]ot (?:logged|signed) in")
+
+
def _no_reply(host, out):
"""Note a side call that came back empty: the paid helpers otherwise fail silently -- seen
- with Copilot, which does not pass a BYOK provider's variables to the hooks that start it."""
+ with Copilot, which does not pass a BYOK provider's variables to the hooks that start it.
+ A CLI that said it is not signed in pauses the host's side calls."""
import time
from .paths import state_root
- tail = " ".join((getattr(out, "stderr", "") or "").split())[-300:]
+ stderr = getattr(out, "stderr", "") or ""
+ tail = " ".join(stderr.split())[-300:]
try:
root = state_root()
os.makedirs(root, exist_ok=True)
with open(os.path.join(root, "errors.log"), "a", encoding="utf-8") as fh:
fh.write(f"--- {time.strftime('%Y-%m-%dT%H:%M:%S')} sidecall {host}: no reply "
f"(exit {getattr(out, 'returncode', '?')}) {tail}\n")
+ if _SIGNED_OUT_RE.search(stderr + (getattr(out, "stdout", "") or "")):
+ path = _pause_path(host, root)
+ os.makedirs(os.path.dirname(path), exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ json.dump({"at": time.time(), "reason": tail[-200:]}, fh)
except OSError:
pass
+def _pause_path(host, root=None):
+ from .paths import state_root
+ return os.path.join(root or state_root(), "hosts", host, "sidecall.json")
+
+
+def paused(host, root=None):
+ """The pause on this host's side calls ({"at", "reason"}) since its CLI said it is not
+ signed in, or None. `te config set` of `microskills` or `loopguard` lifts it (`resume`)."""
+ try:
+ with open(_pause_path(host, root), encoding="utf-8") as fh:
+ d = json.load(fh)
+ except (OSError, ValueError):
+ return None
+ return d if isinstance(d, dict) and d.get("at") else None
+
+
+def resume(host, root=None):
+ """Lift the pause; True if there was one."""
+ try:
+ os.remove(_pause_path(host, root))
+ return True
+ except OSError:
+ return False
+
+
_FLAGS = {}
@@ -226,10 +268,9 @@ def copilot_call(prompt, model="", cli="copilot", timeout=TIMEOUT_S):
def runner_for(host_name, config):
- """The optimizer callable for this host, or None when paid calls are not allowed. Each host's
- calls go through that host's own CLI, never another agent's."""
- if not config.paid:
- return None
+ """The optimizer callable for this host. Each host's calls go through that host's own CLI,
+ never another agent's. The caller decides whether a call is allowed: micro-skills' proposer
+ when micro-skills is on, the loop guard's judge when the guard is."""
model = config.paid_model
if host_name == "pi":
return lambda prompt, _m="agent": pi_call(prompt, model=model)