The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.
docs(gateguard): report the pilot null result and the hidden-intent design
montjeffrey5588e27646b2
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 43da99709..66e4994d6 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -13,6 +13,7 @@- Add opt-in `GATEGUARD_FACT_FORCE_MAX_DENIALS`, a per-session denial cap: once that many first-touch denials have been issued, further new paths pass through (counted in `cap_allows`). Unset keeps the current behaviour; a malformed value is reported once on stderr and leaves the gate uncapped (#2755, thanks @SulimanAbdulrazzaq; closes #2608).- GateGuard loads its target-classification, transcript, search-matching and change-profile modules only on the calls that use them, and parses only the searches that name the target, so shell commands and edits of already checked files stay within a few milliseconds of the previous hook.- Add a development-only harness, `docker/gateguard-effectiveness/`, that runs real Claude sessions on six trapped coding tasks with no gate, the gate, the gate without the question meant to catch each trap, and a placebo gate, and grades the results with hidden graders (see `docs/gateguard/question-effectiveness.md`).+- Report that harness's pilot: 72 sessions in which every arm passed every trial, so it measured the gate's cost (about 45% more turns) and nothing about its effect. Add a second runner, `run-intent.js`, that measures effectiveness instead: the deciding fact exists only in a simulated user's hidden intent, sessions are multi-turn so the gate's question can actually be answered, scenarios are authored blind to the gate's question taxonomy, and each trial is classified into a hole worth fixing rather than checked against a declared target.- Add opt-in `GATEGUARD_METRICS` decision metrics (codes only, never paths, commands or content) and `scripts/gateguard-report.js` to summarise them.- Sensitive targets (`.env*`, keys and certificates, `credentials*`, `secrets.*`, `auth`/`security`/`payment(s)`/`billing`/`migrations` path segments, `.github/workflows/`) and files with more than one hard link get none of these allowances, judged on the path as written and its real location, and are gated in subagents too. `GATEGUARD_EXEMPT_GLOBS` still takes precedence.- Newly denied: a subagent's first touch of a sensitive or hard-linked file, and the first NotebookEdit of each notebook. Every other rule only removes denials and never returns an `allow` decision. See `skills/gateguard/SKILL.md`.diff --git a/docs/gateguard/question-effectiveness.md b/docs/gateguard/question-effectiveness.mdindex e863b52e4..e7f6c9582 100644--- a/docs/gateguard/question-effectiveness.md+++ b/docs/gateguard/question-effectiveness.md@@ -153,4 +153,145 @@ folder. `--arms off,gate,main` compares the routing of two commits.## Pilot results-Not run yet.+The pilot ran: 6 scenarios, 4 arms, 3 repetitions, 72 sessions, sonnet, hook+commit `bdf2dca6`. It returned no information at all.++| Arm | Passed | Mean score | Median denials | Median turns |+| --- | ---: | ---: | ---: | ---: |+| `off` | 18/18 | 1.000 | 0 | 4.5 |+| `gate` | 18/18 | 1.000 | 1 | 6 |+| `minus-target` | 18/18 | 1.000 | 1 | 5 |+| `placebo` | 18/18 | 1.000 | 1 | 6 |++Every arm passed every trial on every scenario. Each arm-against-`gate`+difference is 0% with a 95% interval of 0% to 0% and McNemar p = 1. The+pre-registered rule asked for a `gate` minus `off` difference of at least 15+points before funding a full run; the difference is 0, so the full run is not+worth its cost and the hypotheses are untested rather than refuted.++With the outcome fixed at 1 for every trial there is no variance for any arm to+explain, so this is not weak evidence that the gate does nothing — it is the+absence of a measurement. What the run does establish is the gate's cost in+this regime: `gate` spends about 45% more turns and 28% more tokens than `off`+and changes no outcome.++### Why it measured nothing++1. **Every trap hid its deciding fact in the repository.** `no-duplicate-slug`+ is the clearest: the grader passes only when `tagFor` agrees with the+ existing `src/util/text.js`, so an agent asked for a URL-safe slug greps,+ finds `slugify` and reuses it. The existing test, the CSV's later rows and+ the `.env` convention are the same shape. Agents are reliably good at+ discovering repository facts, so the gate was measured where it cannot help.+2. **The asking channel was switched off.** Trials ran one-shot with "Work+ autonomously in this repository and do not ask clarifying questions". A+ fact-forcing gate was evaluated with no user to answer it, so the only+ mechanism left was the agent restating facts to itself.+3. **The checks could not fail.** Each scenario declared `targetQuestions`, and+ the suite asserted the gate asks one of them for the reference edits. Those+ targets were chosen by reading the gate's own question taxonomy, so the+ assertion held by construction. The scenario set was likewise built around+ the six questions the gate already had, so the design could only confirm+ existing coverage and never discover a missing question.++The first `Limits` bullet anticipated (1) and (3) in part. It understated them:+the problem is not only that the author wrote both sides, but that the pass+criterion referred to the gate's own vocabulary.++## Hidden-intent evaluation++`docker/gateguard-effectiveness/scenarios-intent/` and `run-intent.js` answer a+different question, and are built so that a null result cannot be manufactured+by construction.++**The deciding fact is only in the user's head.** Each scenario carries an+`intent.json` holding an ambiguity the repository cannot settle and the facts+that resolve it, each marked `decisive` or not. `audit-retention-purge` asks for+"our audit retention policy" to be applied; the workspace offers+`config/retention.json` with `sessionDays: 30` and no audit key, while the real+policy is 400 days and a soft mark. Nothing in the tree says so, so an agent+that does not ask can only guess, and the ungated arm cannot sit at the ceiling.++**A simulated user answers only what is actually asked.** `user-sim.js` runs a+separate cheap model holding the facts, under instructions to answer the+question in front of it, never volunteer, and otherwise repeat a stonewall line.+It reports which fact ids it disclosed, so "asked something" and "asked the+deciding thing" are distinguishable. A judge whose reply cannot be parsed is+recorded as a judge failure rather than silently read as a stonewall.++**Sessions are multi-turn and symmetric.** `session.js` joins turns with+`--session-id` and `--resume`. Every arm, gated or not, is told the user is+reachable, so the gate's contribution is whether it makes the agent ask the+deciding question — not whether asking was permitted.++**Scenarios are authored blind.** `task.json` must not declare+`targetQuestions`; the loader rejects it. Which questions a scenario provokes is+observed from the run and reported.++### Hole classes++Each trial is classified from behaviour alone — whether the gate fired, whether+the agent asked, whether the deciding facts came out, and whether the outcome+was right. Nothing in the classification reads the taxonomy.++| Class | Meaning |+| --- | --- |+| `working` | the deciding fact was obtained and the outcome is right |+| `lucky` | right outcome without asking: the scenario does not force the question |+| `follow-through-hole` | the fact was obtained and the outcome is still wrong |+| `targeting-hole` | the agent asked, but not for the deciding fact |+| `silence-hole` | the gate fired and the agent still did not ask |+| `coverage-hole` | a deciding ambiguity the gate never engaged, and the outcome is wrong |++A scenario is reported as carrying no information when nothing varies across+arms — every trial passing or every trial failing. Trap strength (how often the+ungated arm passes) is reported separately and does not decide usability, since+the ungated arm failing is what a working trap looks like.++Graders are checked before any session is billed: the start workspace and the+`naive` overlay must both score below 1 and `reference` must score 1, or the run+stops.++### First hidden-intent run++Four trials, `off` and `gate`, two repetitions, sonnet under test and haiku as+the user:++| Arm | working | lucky | follow-through | targeting | silence | coverage |+| --- | ---: | ---: | ---: | ---: | ---: | ---: |+| `gate` | 1 | 0 | 0 | 0 | 1 | 0 |+| `off` | 0 | 0 | 0 | 0 | 0 | 2 |++`gate` passed 1 of 2 and `off` 0 of 2, so the scenario separates the arms where+the original six did not. Four trials settle nothing about the effect size; what+they show is that the instrument now has a scale.++The questions the gate actually raised on this scenario were `config-effect`,+`config-reader`, `data-schema`, `importers`, `no-plaintext-secrets`,+`public-api` and `quote-instruction` — seven questions, none of which asks about+a retention window or whether a delete is soft. In the one `silence-hole` trial+the gate fired and the agent proceeded without asking anyway. Both are+improvable gaps, and neither was expressible in the previous design.++### Reproduce++```bash+node docker/gateguard-effectiveness/run-intent.js --check-graders+node docker/gateguard-effectiveness/run-intent.js --out gg-intent --arms off,gate --dry-run+node docker/gateguard-effectiveness/run-intent.js --out gg-intent --model <model> --allow-real-provider+node docker/gateguard-effectiveness/run-intent.js --out gg-intent --summarize+node --test tests/docker/gateguard-intent.test.js+```++`minus-target` is refused here: it needs the declared targets these scenarios+deliberately lack. Each trial is several billed turns plus one cheap judge call+per user turn.++### Limits of this design++- The simulated user is a model, so disclosure is not perfectly reproducible;+ repetitions, not a single trial, carry the estimate.+- One scenario is not a corpus. The blind-authoring rule is only as good as the+ independence of whoever writes the next ones.+- Telling every arm that the user is reachable raises asking across the board,+ which is the right comparison but not the shipped default.