The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.
feat(gateguard): measure hook latency in fresh processes
Jeffrey Montoyad95aa3425e6e
diff --git a/docs/gateguard/evaluation.md b/docs/gateguard/evaluation.mdindex 080fe8aa3..10f799ae5 100644--- a/docs/gateguard/evaluation.md+++ b/docs/gateguard/evaluation.md@@ -43,8 +43,15 @@ Metrics per hook:not ask. Asking fewer questions must not drop the ones that matter.- **Estimated denial tokens**: characters of all denial reasons divided by 4.- **Allows with a note**: how the working tree avoided a denial.-- **Hook latency p50/p95**: wall time of `run()` per step, in process. The- only non-deterministic metric.+- **Hook latency, fresh process**: `require()` of the hook plus `run()`, timed+ inside a new Node process per step, which is how Claude Code runs a hook.+ Node start-up itself is excluded; it is the same for every hook. See+ [Latency](#latency).+- **run() latency, warm**: wall time of `run()` in the scenario's worker, after+ the hook module is loaded. Modules the hook loads lazily count on the step+ that first needs them.++The latency rows are the only non-deterministic metrics.Each scenario runs in its own temp project (files, symlinks and hard linkscreated from the fixture), with its own transcript, `GATEGUARD_STATE_DIR`, `HOME` and a@@ -57,30 +64,32 @@ relatively at that ref, into a temp tree that mirrors the repo layout.## ResultsWorking tree = branch `gateguard-full`; `upstream/main` = `c70874fa`;-`8438f040` = the pull request head before this round. Node 22, Linux.+`4b02f669` = the pull request head before this round. Node 22, Linux.```textCorpus: 20 scenarios, 184 steps.```-| Metric | working tree | upstream/main | 8438f040 |+| Metric | working tree | upstream/main | 4b02f669 || --- | ---: | ---: | ---: || Steps | 184 | 184 | 184 |-| Denials | 132 | 157 | 131 |+| Denials | 132 | 157 | 132 || Redundant denials | 0 | 38 | 0 |-| Must-deny bypasses | 0 | 8 | 1 |-| Expectation mismatches | 0 | 55 | 1 |+| Must-deny bypasses | 0 | 8 | 0 |+| Expectation mismatches | 0 | 55 | 0 || Irrelevant questions asked | 2 | 187 | 2 || Irrelevant questions in condensed denials | 2 | 111 | 2 || Warranted questions not asked | 1 | 57 | 1 |-| Estimated denial tokens | 25633 | 27796 | 25455 |-| Allows with a credit note | 12 | 0 | 13 |+| Estimated denial tokens | 25633 | 27796 | 25633 |+| Allows with a credit note | 12 | 0 | 12 || Allows with a sibling note | 11 | 0 | 11 || Allows with a trivial-edit note | 5 | 0 | 5 |-| Hook latency p50 (ms) | 2.28 | 1.20 | 2.14 |-| Hook latency p95 (ms) | 7.81 | 2.93 | 7.48 |+| Hook latency p50, fresh process (ms) | 21.18 | 10.58 | 25.11 |+| Hook latency p95, fresh process (ms) | 30.34 | 16.67 | 38.24 |+| run() latency p50, warm (ms) | 2.03 | 1.03 | 2.01 |+| run() latency p95, warm (ms) | 10.56 | 2.58 | 7.03 |-| Scenario | Steps | Denials: working tree | Denials: upstream/main | Denials: 8438f040 |+| Scenario | Steps | Denials: working tree | Denials: upstream/main | Denials: 4b02f669 || --- | ---: | ---: | ---: | ---: || docs-heavy-session | 14 | 6 | 12 | 6 || scaffold-module | 10 | 3 | 10 | 3 |@@ -94,7 +103,7 @@ Corpus: 20 scenarios, 184 steps.| subagent-edits | 6 | 3 | 1 | 3 || first-shell-commands | 20 | 11 | 17 | 11 || windows-paths | 8 | 5 | 7 | 5 |-| bypass-search-filters | 19 | 15 | 19 | 14 |+| bypass-search-filters | 19 | 15 | 19 | 15 || bypass-turn-and-batch | 7 | 7 | 7 | 7 || bypass-siblings | 11 | 11 | 11 | 11 || cap-with-sensitive | 6 | 4 | 6 | 4 |@@ -129,12 +138,12 @@ Against `upstream/main`:after a comment-only one (the comment edit no longer spends the file's firsttouch).-Against `8438f040` (the pull request head before this round): one more-denial and one fewer credit. `bypass-search-filters` now includes a search-whose directory-qualified include (`rg -g 'src/**' sweep .`) keeps the target-in `lib/` out of the search; `8438f040` dropped that include and credited the-edit, its one must-deny bypass. The control that searches the target's own-directory is still credited.+Against `4b02f669` (the pull request head before this round): the same+decisions, questions and notes on every step; only latency changes (see+[Latency](#latency)). `bypass-search-filters` includes a search whose+directory-qualified include (`rg -g 'src/**' sweep .`) keeps the target in+`lib/` out of the search; it must be denied, and the control that searches+the target's own directory is credited.Four scenarios come from security reviews of this change.`bypass-comment-context` holds 21 must-deny steps: comment-looking edits that@@ -173,10 +182,12 @@ What the working tree still gets wrong, by the corpus's own labels:(`c-comment-continuation`) has no data words, so the condensed hint nolonger mentions data schemas; the corpus labels that question as warranted(the 1 unasked one).-- Latency rises against `upstream/main` (p50 2.3 ms against 1.2 ms, p95- 7.8 ms against 2.9 ms) with the added transcript scanning, path resolution- and the target's link-count check; both stay far below the 200 ms budget- for blocking hooks. Checking a comment-only edit+- Latency: a first touch costs more than on `upstream/main` because it reads+ the transcript, matches the searches that name the target, reads the target+ file and profiles the change (fresh-process p50 21.2 ms against 10.6 ms,+ p95 30.3 ms against 16.7 ms on this corpus, where every step is a first+ touch or a gate). Shell commands and repeat edits stay within about 1 to+ 2.5 ms of `main`; see [Latency](#latency). Checking a comment-only editreads and scans the target file, which costs up to about 50 ms for a 1 MiBfile (the largest one read).@@ -185,14 +196,14 @@ What the working tree still gets wrong, by the corpus's own labels:```bashgit fetch upstream mainnode scripts/dev/gateguard-eval.js --markdown-node scripts/dev/gateguard-eval.js --markdown --baseline upstream/main --baseline 8438f040+node scripts/dev/gateguard-eval.js --markdown --baseline upstream/main --baseline 4b02f669node scripts/dev/gateguard-eval.js --json > gateguard-eval.jsonnode scripts/dev/gateguard-eval.js --baseline upstream/main --sarif gateguard-eval.sarifnode tests/hooks/gateguard-scenarios.test.js````--baseline <ref>` takes any ref and can be repeated (default-`upstream/main`); pass the pull request head (`8438f040` above) to compare+`upstream/main`); pass the pull request head (`4b02f669` above) to compareagainst it. `--corpus <dir>` points at anotherscenario directory. The script exits non-zero when the working tree has amismatch, a must-deny bypass, an explicit `allow` decision or a thrown error.@@ -201,6 +212,34 @@ mismatch, a must-deny bypass, an explicit `allow` decision or a thrown error.`warning`) with each hook's totals in the run properties. Timing is left out,so the file is identical across reruns of the same tree and corpus.+## Latency++The fresh-process pass replays every scenario again, with a new state+directory and transcript, and runs each step with+`node -e <probe> <hook> <payload>`. Its decisions must match the worker's;+a step decided differently fails the run. `--no-cold` skips the pass.++Per call type: 25 fresh processes each, p50 of `require()` plus p50 of+`run()` in ms, Node 22 on Linux, with a 20-search turn and a 50-function+target file:++| Call | this branch | `4b02f669` | `main` |+| --- | ---: | ---: | ---: |+| Shell command, first of session | 10.2 | 19.2 | 8.7 |+| Shell command, routine gate already passed | 12.3 | 21.3 | 11.6 |+| Edit of a file already checked | 11.1 | 20.6 | 8.7 |+| First-touch edit, denied | 22.4 | 29.7 | 9.6 |+| First-touch edit, comment-only pass | 25.7 | 38.5 | 9.5 |++Shell commands and repeat edits, most calls in a session, now load only the+modules they use and stay within about 1 to 2.5 ms of `main`. A first touch+reads the transcript, parses the turn's searches that name the target, reads+the target file and profiles the change; `main` does none of these. Each hook+call is a new Node process, so the code a call runs is also compiled on that+call, and that compilation, not the size of the files, is most of the+remaining difference. Node's module compile cache does not reduce it: it+caches top-level code, and the gate's functions are compiled on first call.+## Corpus formatOne JSON file per scenario in `tests/fixtures/gateguard-scenarios/`:diff --git a/scripts/dev/gateguard-eval.js b/scripts/dev/gateguard-eval.jsindex f44598d52..b97693634 100644--- a/scripts/dev/gateguard-eval.js+++ b/scripts/dev/gateguard-eval.js@@ -305,6 +305,59 @@ async function runCorpus(hookFile, scenarios) {}}+// --- cold latency ---+// see docs/gateguard/evaluation.md#latency++const COLD_PROBE = [+ "const { performance } = require('perf_hooks');",+ 'const started = performance.now();',+ 'const hook = require(process.argv[1]);',+ 'const result = hook.run(process.argv[2]);',+ 'const ms = performance.now() - started;',+ 'process.stdout.write(JSON.stringify({ ms, result: result === undefined ? null : result }));'+].join('\n');++/** Replay every scenario with one fresh Node process per step; returns per-step require+run latency and decisions. */+function measureColdLatency(hookFile, scenarios) {+ const workDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gateguard-cold-'));+ try {+ const steps = [];+ for (const scenario of scenarios) {+ const prepared = prepareScenario(scenario, workDir);+ if (prepared.skipped) continue;+ for (const step of prepared.steps) {+ if (step.records.length > 0) {+ fs.appendFileSync(prepared.transcript, step.records.map(record => `${JSON.stringify(record)}\n`).join(''), 'utf8');+ }+ const raw = JSON.stringify(step.payload);+ const stdout = execFileSync(process.execPath, ['-e', COLD_PROBE, hookFile, raw], {+ env: prepared.env,+ encoding: 'utf8',+ maxBuffer: 16 * 1024 * 1024,+ stdio: ['ignore', 'pipe', 'ignore']+ });+ const { ms, result } = JSON.parse(stdout);+ const output = hookOutput(result);+ steps.push({ scenario: scenario.name, step: step.id, latencyMs: ms, decision: output.decision === 'deny' ? 'deny' : 'allow' });+ }+ }+ return steps;+ } finally {+ fs.rmSync(workDir, { recursive: true, force: true });+ }+}++function coldSummary(coldSteps, warmSteps) {+ const sorted = coldSteps.map(step => step.latencyMs).sort((a, b) => a - b);+ const warmDecisions = new Map(warmSteps.map(step => [`${step.scenario}/${step.step}`, step.decision]));+ const disagreements = coldSteps.filter(step => warmDecisions.get(`${step.scenario}/${step.step}`) !== step.decision).length;+ return {+ p50Ms: Number(percentile(sorted, 0.5).toFixed(3)),+ p95Ms: Number(percentile(sorted, 0.95).toFixed(3)),+ disagreements+ };+}+// --- question mapping ---function questionTextTable() {@@ -439,8 +492,10 @@ const METRIC_ROWS = [['Allows with a credit note', s => s.totals.allowsByKind.credit],['Allows with a sibling note', s => s.totals.allowsByKind.sibling],['Allows with a trivial-edit note', s => s.totals.allowsByKind.trivial],- ['Hook latency p50 (ms)', s => s.latency.p50Ms.toFixed(2)],- ['Hook latency p95 (ms)', s => s.latency.p95Ms.toFixed(2)]+ ['Hook latency p50, fresh process (ms)', s => (s.cold ? s.cold.p50Ms.toFixed(2) : 'not run')],+ ['Hook latency p95, fresh process (ms)', s => (s.cold ? s.cold.p95Ms.toFixed(2) : 'not run')],+ ['run() latency p50, warm (ms)', s => s.latency.p50Ms.toFixed(2)],+ ['run() latency p95, warm (ms)', s => s.latency.p95Ms.toFixed(2)]];function markdownTable(header, rows) {@@ -474,6 +529,10 @@ function renderMarkdown(report) {lines.push('', `${report.hooks[0].label} mismatches:`, '');mismatched.forEach(step => lines.push(`- ${step.scenario} / ${step.step}: expected ${step.expect}, got ${step.decision} (${step.kind})`));}+ const cold = report.hooks[0].summary.cold;+ if (cold && cold.disagreements > 0) {+ lines.push('', `${report.hooks[0].label}: ${cold.disagreements} step(s) decided differently in a fresh process.`);+ }return `${lines.join('\n')}\n`;}@@ -586,16 +645,18 @@ const USAGE = [' --corpus <dir> scenario directory (default tests/fixtures/gateguard-scenarios)',' --markdown print markdown tables (default)',' --json print the full report as JSON',- ' --sarif <file> also write working-tree gate failures as SARIF 2.1.0'+ ' --sarif <file> also write working-tree gate failures as SARIF 2.1.0',+ ' --no-cold skip the fresh-process latency pass (one Node process per step)'].join('\n');function parseArgs(argv) {- const options = { format: 'markdown', baselines: [], corpus: DEFAULT_CORPUS_DIR, sarif: null, help: false };+ const options = { format: 'markdown', baselines: [], corpus: DEFAULT_CORPUS_DIR, sarif: null, cold: true, help: false };for (let i = 0; i < argv.length; i++) {const arg = argv[i];if (arg === '--markdown') options.format = 'markdown';else if (arg === '--json') options.format = 'json';else if (arg === '--help' || arg === '-h') options.help = true;+ else if (arg === '--no-cold') options.cold = false;else if ((arg === '--baseline' || arg === '--corpus' || arg === '--sarif') && i + 1 < argv.length) {if (arg === '--baseline') options.baselines.push(argv[++i]);else if (arg === '--sarif') options.sarif = path.resolve(argv[++i]);@@ -607,7 +668,7 @@ function parseArgs(argv) {}/** Evaluate the working-tree hook and each baseline ref over a corpus. */-async function evaluate({ baselines = [DEFAULT_BASELINE], corpus = DEFAULT_CORPUS_DIR } = {}) {+async function evaluate({ baselines = [DEFAULT_BASELINE], corpus = DEFAULT_CORPUS_DIR, cold = false } = {}) {const scenarios = loadCorpus(corpus);const table = questionTextTable();const hooks = [{ label: 'working tree', ref: null, file: path.join(REPO_ROOT, ...HOOK_RELATIVE_PATH.split('/')) }];@@ -618,7 +679,9 @@ async function evaluate({ baselines = [DEFAULT_BASELINE], corpus = DEFAULT_CORPU});const evaluated = [];for (const hook of hooks) {- evaluated.push({ label: hook.label, ref: hook.ref, summary: summarize(await runCorpus(hook.file, scenarios), table) });+ const summary = summarize(await runCorpus(hook.file, scenarios), table);+ if (cold) summary.cold = coldSummary(measureColdLatency(hook.file, scenarios), summary.steps);+ evaluated.push({ label: hook.label, ref: hook.ref, summary });}return {corpus: { scenarios: scenarios.length, steps: scenarios.reduce((sum, scenario) => sum + scenario.steps.length, 0) },@@ -640,7 +703,8 @@ async function main() {process.stdout.write(options.format === 'json' ? `${JSON.stringify(report, null, 2)}\n` : renderMarkdown(report));if (options.sarif) fs.writeFileSync(options.sarif, renderSarif(report), 'utf8');const working = report.hooks[0].summary.totals;- if (working.mustDenyBypasses > 0 || working.mismatches > 0 || working.explicitAllows > 0 || working.errors > 0) {+ const coldDisagreements = report.hooks[0].summary.cold ? report.hooks[0].summary.cold.disagreements : 0;+ if (working.mustDenyBypasses > 0 || working.mismatches > 0 || working.explicitAllows > 0 || working.errors > 0 || coldDisagreements > 0) {process.exitCode = 1;}}@@ -654,4 +718,4 @@ if (!isMainThread && workerData && workerData.gateguardEval) {});}-module.exports = { loadCorpus, materializeHook, runCorpus, summarize, questionsAsked, renderMarkdown, renderSarif, parseArgs, evaluate };+module.exports = { loadCorpus, materializeHook, runCorpus, measureColdLatency, summarize, questionsAsked, renderMarkdown, renderSarif, parseArgs, evaluate };diff --git a/tests/scripts/gateguard-eval.test.js b/tests/scripts/gateguard-eval.test.jsindex 9935322a3..f8eaf5bbc 100644--- a/tests/scripts/gateguard-eval.test.js+++ b/tests/scripts/gateguard-eval.test.js@@ -5,10 +5,11 @@'use strict';const assert = require('assert');+const fs = require('fs');const os = require('os');const path = require('path');const { pathToFileURL } = require('url');-const { parseArgs, renderSarif } = require('../../scripts/dev/gateguard-eval');+const { parseArgs, renderSarif, loadCorpus, runCorpus, summarize, measureColdLatency } = require('../../scripts/dev/gateguard-eval');let passed = 0;let failed = 0;@@ -164,6 +165,42 @@ test('output leaves out timing so reruns are byte-identical', () => {assert.strictEqual(renderSarif(slower), text);});-console.log(`\nPassed: ${passed}`);-console.log(`Failed: ${failed}`);-process.exitCode = failed > 0 ? 1 : 0;+test('--no-cold turns off the fresh-process pass', () => {+ assert.strictEqual(parseArgs([]).cold, true);+ assert.strictEqual(parseArgs(['--no-cold']).cold, false);+});++async function coldTests() {+ const corpusDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gateguard-eval-corpus-'));+ try {+ fs.copyFileSync(+ path.join(__dirname, '..', 'fixtures', 'gateguard-scenarios', '11-first-shell-commands.json'),+ path.join(corpusDir, 'shell.json')+ );+ const scenarios = loadCorpus(corpusDir);+ const hookFile = path.join(__dirname, '..', '..', 'scripts', 'hooks', 'gateguard-fact-force.js');+ const warm = summarize(await runCorpus(hookFile, scenarios));+ const cold = measureColdLatency(hookFile, scenarios);+ test('the fresh-process pass times every step', () => {+ assert.strictEqual(cold.length, scenarios[0].steps.length);+ assert.ok(cold.every(step => Number.isFinite(step.latencyMs) && step.latencyMs > 0));+ });+ test('fresh processes decide every step as the worker does', () => {+ assert.deepStrictEqual(cold.map(step => step.decision), warm.steps.map(step => step.decision));+ });+ } finally {+ fs.rmSync(corpusDir, { recursive: true, force: true });+ }+}++coldTests().then(finish, error => {+ console.log(` ✗ fresh-process pass failed: ${error.message}`);+ failed++;+ finish();+});++function finish() {+ console.log(`\nPassed: ${passed}`);+ console.log(`Failed: ${failed}`);+ process.exitCode = failed > 0 ? 1 : 0;+}