π The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated
TypeScript agentic-aiagentic-frameworkagentic-workflowagentsai-agentsai-assistantai-skillsautonomous-agentsclaudecodexdsh-pluginharnessmcp-servermulti-agentmulti-agent-systemsnpmskillsswarmswarm-intelligencetypescript
feat(harness): anti-overfitting β frozen public human eval + per-gen red/blue deltas + clean-room replay (ADR-176) (#2580)
rUvaee3b999102f
diff --git a/.github/workflows/v3-ci.yml b/.github/workflows/v3-ci.ymlindex c7f6167785..1699c641e7 100644--- a/.github/workflows/v3-ci.yml+++ b/.github/workflows/v3-ci.yml@@ -339,6 +339,14 @@ jobs:working-directory: v3/@claude-flow/clirun: node scripts/smoke-flywheel-proof.mjs+ - name: "ADR-176 β clean-room replay acceptance test (offline, identical hashes)"+ # Replay a PROMOTED generation from its receipt alone: recompute every+ # hash bit-identically and re-run accept/v1+sig to reproduce promoted=true,+ # with network access trapped. Proves promotions are reproducible without+ # trusting any service log.+ working-directory: v3/@claude-flow/cli+ run: node scripts/replay-generation.mjs+- name: Run testsworking-directory: v3# Tolerate exit code 139 (SIGSEGV) ONLY when it occurs at processdiff --git a/v3/@claude-flow/cli/.claude/eval/human-relevance-frozen-v1.json b/v3/@claude-flow/cli/.claude/eval/human-relevance-frozen-v1.jsonnew file mode 100644index 0000000000..d22dcc1874--- /dev/null+++ b/v3/@claude-flow/cli/.claude/eval/human-relevance-frozen-v1.json@@ -0,0 +1,17 @@+{+ "version": "human-relevance-frozen-v1",+ "description": "Frozen, public, human-labeled relevance eval set (ADR-081). Immutable β the red/blue anchor the flywheel must never regress AND against which per-generation human-relevance deltas are reported. Do not edit; supersede with a new version file.",+ "metric": "nDCG@3 over labelled substring-relevance",+ "tasks": [+ { "id": "hr-0", "q": "how was the Opus model alias fixed", "labels": ["opus 4.8", "opus alias", "opus model alias", "#2232"] },+ { "id": "hr-1", "q": "self-learning wiring task-completed pretrain", "labels": ["self-learning", "adr-074", "#2245", "task-completed"] },+ { "id": "hr-2", "q": "deterministic codemod engine var-to-const", "labels": ["deterministic tier-1 codemod", "adr-143", "codemod", "var-to-const"] },+ { "id": "hr-3", "q": "MCP server orphan leak parent-death", "labels": ["mcp orphan", "parent-death", "#2234", "orphan on every claude"] },+ { "id": "hr-4", "q": "unified learning stats aggregator", "labels": ["unified learning-stats", "adr-075"] },+ { "id": "hr-5", "q": "structured distillation 4-field schema", "labels": ["structured distillation", "adr-076", "4-field schema"] },+ { "id": "hr-6", "q": "SQL injection migrate.ts table identifier", "labels": ["sql injection", "migrate.ts", "agentdb", "cve"] },+ { "id": "hr-7", "q": "recall@k HNSW benchmark harness", "labels": ["hnsw", "recall@k", "benchmark intelligence"] },+ { "id": "hr-8", "q": "Q-learning encoder keyword block", "labels": ["q-state encoder", "keyword block", "#2239", "q-encoder"] },+ { "id": "hr-9", "q": "security hardening crypto random IDs", "labels": ["cwe-347", "crypto.randomuuid", "random id", "crypto random"] }+ ]+}diff --git a/v3/@claude-flow/cli/__tests__/harness-flywheel-generations.test.ts b/v3/@claude-flow/cli/__tests__/harness-flywheel-generations.test.tsindex b97e1b6965..6515661a8c 100644--- a/v3/@claude-flow/cli/__tests__/harness-flywheel-generations.test.ts+++ b/v3/@claude-flow/cli/__tests__/harness-flywheel-generations.test.ts@@ -135,6 +135,23 @@ describe('runFlywheelGeneration β compounding autonomy loop', () => {expect(servedChampion(root).championHash).toBeNull(); // reverted});+ it('records a per-generation human-relevance delta and exposes the overfitting signal', async () => {+ const root = mkdtempSync(join(tmpdir(), 'fwg-'));+ await runFlywheelGeneration(root, { ...makeDeps(1000), humanEvalHash: 'sha256:frozen-test' });+ const promos = loadPromotions(root);+ expect(promos.length).toBeGreaterThanOrEqual(1);+ // the human-relevance delta is recorded on every promotion (against the frozen set)+ expect(promos[0].deltas.humanRelevance).toBeDefined();+ expect(promos[0].humanEvalHash).toBe('sha256:frozen-test');++ const s = flywheelStatus(root);+ expect(s.cumulativeBenchmarkDelta).toBeGreaterThan(0); // proxy (self-retrieval) improved+ // stub's anchor is config-independent β human relevance flat β the overfitting+ // signal is now VISIBLE (proxy up, human ~0), not hidden.+ expect(Math.abs(s.cumulativeHumanRelevanceDelta)).toBeLessThan(0.01);+ expect(s.humanEvalHash).toBe('sha256:frozen-test');+ });+it('no-op (never throws) on a store too small to harvest', async () => {const root = mkdtempSync(join(tmpdir(), 'fwg-'));const r = await runFlywheelGeneration(root, { ...makeDeps(1), getPatterns: () => patterns.slice(0, 5) });diff --git a/v3/@claude-flow/cli/__tests__/harness-frozen-eval.test.ts b/v3/@claude-flow/cli/__tests__/harness-frozen-eval.test.tsnew file mode 100644index 0000000000..9c2e72e0b3--- /dev/null+++ b/v3/@claude-flow/cli/__tests__/harness-frozen-eval.test.ts@@ -0,0 +1,24 @@+/**+ * Frozen public human-labeled eval set (ADR-176 anti-overfitting).+ */+import { describe, it, expect } from 'vitest';+import {+ loadFrozenHumanEval, humanEvalHash, FROZEN_HUMAN_EVAL_HASH, FROZEN_HUMAN_EVAL_VERSION,+} from '../src/services/harness-frozen-eval.js';++describe('frozen human eval set', () => {+ it('loads and its content hash matches the pinned constant (frozen guarantee)', () => {+ const e = loadFrozenHumanEval();+ expect(e.version).toBe(FROZEN_HUMAN_EVAL_VERSION);+ expect(e.tasks.length).toBe(10);+ e.tasks.forEach((t) => { expect(t.q).toBeTruthy(); expect(t.labels.length).toBeGreaterThan(0); });+ expect(e.corpusHash).toBe(FROZEN_HUMAN_EVAL_HASH);+ });++ it('hash is order-independent but tamper-evident', () => {+ const e = loadFrozenHumanEval();+ expect(humanEvalHash([...e.tasks].reverse())).toBe(e.corpusHash); // order-independent+ const tampered = e.tasks.map((t, i) => (i === 0 ? { ...t, labels: [...t.labels, 'injected'] } : t));+ expect(humanEvalHash(tampered)).not.toBe(e.corpusHash); // any edit changes the hash+ });+});diff --git a/v3/@claude-flow/cli/scripts/flywheel-status.mjs b/v3/@claude-flow/cli/scripts/flywheel-status.mjsindex e8057f5379..af29af1e62 100644--- a/v3/@claude-flow/cli/scripts/flywheel-status.mjs+++ b/v3/@claude-flow/cli/scripts/flywheel-status.mjs@@ -24,6 +24,12 @@ console.log('='.repeat(64));console.log(`generations (promotions) : ${s.generations}`);console.log(`attempts (incl. refusals): ${s.attempts}`);console.log(`cumulative held-out Ξ : ${s.lineage.cumulativeHeldOutImprovement.toFixed(4)}`);+// Anti-overfitting view: proxy (self-supervised) gain vs FROZEN human-relevance gain.+const bench = s.cumulativeBenchmarkDelta ?? 0, human = s.cumulativeHumanRelevanceDelta ?? 0;+const overfit = bench > 0.02 && human <= 0.005;+console.log(`cumulative proxy Ξ : ${bench.toFixed(4)} (self-supervised self-retrieval)`);+console.log(`cumulative HUMAN Ξ : ${human.toFixed(4)} (frozen human-labeled eval)${overfit ? ' β οΈ proxy up but human flat β OVERFITTING' : ''}`);+console.log(`frozen human eval : ${s.humanEvalHash ? s.humanEvalHash.slice(0, 26) + 'β¦' : '(none)'}`);console.log(`lineage intact : ${s.lineage.lineageIntact} replayable: ${s.lineage.allReplayable}`);console.log(`immutable root : ${(s.lineage.rootHash || '(none yet)').slice(0, 26)}β¦`);console.log(`plateau : ${s.plateau.status} β ${s.plateau.rationale}`);diff --git a/v3/@claude-flow/cli/scripts/replay-generation.mjs b/v3/@claude-flow/cli/scripts/replay-generation.mjsnew file mode 100644index 0000000000..0c71e12486--- /dev/null+++ b/v3/@claude-flow/cli/scripts/replay-generation.mjs@@ -0,0 +1,77 @@+#!/usr/bin/env node+/**+ * Acceptance test (ADR-176) β clean-room replay of a promoted generation.+ *+ * From a clean install, replay ONE promoted generation bundle from+ * .claude/evolve-proof/ and verify the SAME candidate passes accept/v1+sig with+ * IDENTICAL hashes and NO NETWORK β i.e. the promotion is reproducible from the+ * receipt alone, without trusting any service log.+ *+ * Proves: (1) each embedded hash recomputes bit-identically, (2) re-running the+ * versioned rule on independently-recomputed inputs reproduces promoted=true,+ * (3) zero network access (fetch is trapped and any call fails the test).+ *+ * Usage: node scripts/replay-generation.mjs [--file <bundle.json>] [--dir <projectRoot>]+ */+import { readFileSync, existsSync, readdirSync } from 'node:fs';+import { fileURLToPath } from 'node:url';+import { dirname, join, resolve } from 'node:path';+import { createHash } from 'node:crypto';++// ββ Hard no-network guard: any network call during replay fails the test. ββ+for (const name of ['fetch']) {+ globalThis[name] = () => { throw new Error(`NETWORK ACCESS during replay (${name}) β replay must be offline`); };+}++const CLI_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..');+const ep = await import(`file://${join(CLI_ROOT, 'dist/src/services/evolve-proof.js')}`);++const arg = (f) => { const i = process.argv.indexOf(f); return i > -1 ? process.argv[i + 1] : null; };+const root = resolve(arg('--dir') || CLI_ROOT);+const evalDir = join(root, '.claude', 'evolve-proof');++// pick a promoted generation: explicit --file, else the highest-numbered real-generation-*.+let file = arg('--file');+if (!file) {+ const reals = existsSync(evalDir) ? readdirSync(evalDir).filter((f) => /^real-generation-\d+\.json$/.test(f)).sort() : [];+ if (!reals.length) { console.error(`no promoted generation found in ${evalDir}`); process.exit(1); }+ file = join(evalDir, reals[reals.length - 1]);+} else if (!existsSync(file)) { file = join(evalDir, file); }++const bundle = JSON.parse(readFileSync(file, 'utf-8'));+console.log(`clean-room replay of ${file}`);+console.log('='.repeat(70));+if (!bundle.decisionReceipt?.promoted) { console.error('β bundle is not a PROMOTED generation'); process.exit(1); }++// (1) hashes recompute bit-identically+const sha = (s) => 'sha256:' + createHash('sha256').update(s).digest('hex');+const canon = (v) => Array.isArray(v) ? v.map(canon) : (v && typeof v === 'object' ? Object.fromEntries(Object.keys(v).sort().map((k) => [k, canon(v[k])])) : v);+const cstr = (v) => JSON.stringify(canon(v));+// manifest hash mirrors evolve-proof.manifestHash (canonicalManifestBytes β sha256)+const { canonicalManifestBytes } = await import(`file://${join(CLI_ROOT, 'dist/src/config/proven-config.js')}`);+const mh = (m) => sha(canonicalManifestBytes(m).toString('utf-8'));++const checks = [+ ['input holdout hash', sha(cstr(bundle.holdout)), bundle.inputHoldoutHash],+ ['baseline manifest hash', mh(bundle.baselineManifest), bundle.baselineManifestHash],+ ['candidate manifest hash', mh(bundle.candidateManifest), bundle.candidateManifestHash],+];+let idOK = true;+for (const [label, got, want] of checks) {+ const same = got === want;+ idOK = idOK && same;+ console.log(`${same ? 'β' : 'β'} ${label}: ${same ? 'identical' : `MISMATCH\n recomputed ${got}\n recorded ${want}`}`);+}++// (2) re-run accept/v1+sig on independently-recomputed inputs β promoted=true+const v = ep.verifyReceiptBundle(bundle);+console.log(`${v.valid ? 'β' : 'β'} independent verify: ${v.explanation}`);+console.log(` decisionMatches=${v.decisionMatches} ruleVersion=${bundle.decisionReceipt.promotionRuleVersion} noAutoServe=${v.noAutoServe}`);+if (bundle.deltas?.humanRelevance !== undefined)+ console.log(` human-relevance Ξ (frozen eval ${(bundle.humanEvalHash||'').slice(0,18)}β¦): ${bundle.deltas.humanRelevance.toFixed(4)}`);++const pass = idOK && v.valid && v.decisionMatches;+console.log('='.repeat(70));+console.log(pass ? 'ACCEPTANCE TEST PASSED β promoted generation reproducible from its receipt, identical hashes, offline.'+ : 'ACCEPTANCE TEST FAILED');+process.exit(pass ? 0 : 1);diff --git a/v3/@claude-flow/cli/scripts/smoke-flywheel-proof.mjs b/v3/@claude-flow/cli/scripts/smoke-flywheel-proof.mjsindex ae14d5fa4b..b168b772a1 100644--- a/v3/@claude-flow/cli/scripts/smoke-flywheel-proof.mjs+++ b/v3/@claude-flow/cli/scripts/smoke-flywheel-proof.mjs@@ -26,6 +26,14 @@ const ok = (m) => console.log(`β ${m}`);const ep = await import(`file://${d('services/evolve-proof.js')}`);const rvfa = await import(`file://${d('config/proven-config-rvfa.js')}`);const pc = await import(`file://${d('config/proven-config.js')}`);+const fe = await import(`file://${d('services/harness-frozen-eval.js')}`);++// 0. the frozen public human-labeled eval set must not have drifted (pinned hash)+try {+ const e = fe.loadFrozenHumanEval();+ e.corpusHash === fe.FROZEN_HUMAN_EVAL_HASH ? ok(`frozen human eval set intact (${e.tasks.length} tasks, ${e.corpusHash.slice(0, 18)}β¦)`)+ : fail(`frozen human eval hash drifted`);+} catch (err) { fail(`frozen human eval load: ${err.message}`); }// 1. synthetic proof-of-mechanism fixtureconst g0 = JSON.parse(readFileSync(cl('evolve-proof/generation-0.json'), 'utf-8'));diff --git a/v3/@claude-flow/cli/src/services/evolve-proof.ts b/v3/@claude-flow/cli/src/services/evolve-proof.tsindex a6feb47b19..cbc5a3547a 100644--- a/v3/@claude-flow/cli/src/services/evolve-proof.ts+++ b/v3/@claude-flow/cli/src/services/evolve-proof.ts@@ -65,7 +65,13 @@ export interface ShadowRegistration {export interface CostReceipt { usd: number; llmCalls: number; tier: string; notes: string; }/** Multi-dimensional deltas vs the parent β distinguishes *why* a change is (un)safe. */-export interface ChangeDeltas { benchmark: number; security: number; cost: number; }+export interface ChangeDeltas {+ benchmark: number; // the optimization objective (e.g. self-supervised self-retrieval)+ security: number; // redblue verdict (-1 = FAILED)+ cost: number;+ humanRelevance?: number; // per-generation Ξ on the FROZEN human-labeled eval set (ADR-176 anti-overfitting).+ // ~0 across generations while `benchmark` > 0 β overfitting to the proxy β now visible.+}/*** Causal promotion record (not just provenance). Answers *why* a candidate won β@@ -110,6 +116,7 @@ export interface EvolveReceiptBundle {mutationClass: string;mutationSummary: string;deltas: ChangeDeltas;+ humanEvalHash?: string; // the frozen human eval set the humanRelevance delta is measured againstpromotion: PromotionRecord | null; // populated on passregression: RegressionRecord | null; // populated on reject// ββ embedded evidence (so verification needs no service logs) ββ@@ -145,6 +152,8 @@ export interface AssembleOpts {cost: { tier: string; notes: string };redblue?: 'PASS' | 'FAIL' | 'SKIPPED'; drift?: number;canaryRollbackRate?: number; // SEPARATE deployment-safety signal; default = strict per-held-out-task regression+ humanRelevanceDelta?: number; // per-gen Ξ on the frozen human eval set+ humanEvalHash?: string; // which frozen human eval set the delta is againstlayer?: string; corpus?: string;}@@ -190,7 +199,7 @@ export function assembleBundle(baseline: Record<string, number>, candidate: Reco} : null;const { mutationClass, mutationSummary } = classifyMutation(baseline, candidate);- const deltas: ChangeDeltas = { benchmark: candidateHeldOut - baselineHeldOut, security: o.redblue === 'FAIL' ? -1 : 0, cost: 0 };+ const deltas: ChangeDeltas = { benchmark: candidateHeldOut - baselineHeldOut, security: o.redblue === 'FAIL' ? -1 : 0, cost: 0, humanRelevance: o.humanRelevanceDelta };const promotion: PromotionRecord | null = promoted ? { parentManifestHash: o.parent, candidateManifestHash, mutationClass, mutationSummary, deltas, decisionReceipt } : null;const regression: RegressionRecord | null = promoted ? null : {candidateManifestHash, ancestor: o.parent ?? baselineManifestHash, mutationClass,@@ -203,7 +212,7 @@ export function assembleBundle(baseline: Record<string, number>, candidate: RecomeetsPromotionRule: { version: PROMOTION_RULE_VERSION, result: promoted },decisionReceipt, shadow,costReceipt: { usd: 0, llmCalls: 0, tier: o.cost.tier, notes: o.cost.notes },- mutationClass, mutationSummary, deltas, promotion, regression,+ mutationClass, mutationSummary, deltas, humanEvalHash: o.humanEvalHash, promotion, regression,holdout, baselineManifest, candidateManifest,};}@@ -218,12 +227,15 @@ export function assembleBundle(baseline: Record<string, number>, candidate: Recoexport function runRealEvolveRound(opts: {baseline: Record<string, number>; candidate: Record<string, number>; holdout: HoldoutTask[];generation: number; parent: string | null; branch?: string; now: number;- redblue?: 'PASS' | 'FAIL' | 'SKIPPED'; drift?: number; canaryRollbackRate?: number; corpus: string;+ redblue?: 'PASS' | 'FAIL' | 'SKIPPED'; drift?: number; canaryRollbackRate?: number;+ humanRelevanceDelta?: number; humanEvalHash?: string; corpus: string;}): EvolveReceiptBundle {return assembleBundle(opts.baseline, opts.candidate, opts.holdout, {generation: opts.generation, parent: opts.parent, branch: opts.branch ?? 'main', now: opts.now, kind: 'real',cost: { tier: 'real-local', notes: 'measured on the frozen anchor via live retrieval β no LLM, no network' },- redblue: opts.redblue, drift: opts.drift, canaryRollbackRate: opts.canaryRollbackRate, layer: 'real/retrieval', corpus: opts.corpus,+ redblue: opts.redblue, drift: opts.drift, canaryRollbackRate: opts.canaryRollbackRate,+ humanRelevanceDelta: opts.humanRelevanceDelta, humanEvalHash: opts.humanEvalHash,+ layer: 'real/retrieval', corpus: opts.corpus,});}diff --git a/v3/@claude-flow/cli/src/services/harness-flywheel-generations.ts b/v3/@claude-flow/cli/src/services/harness-flywheel-generations.tsindex bb604a70ac..5291d46565 100644--- a/v3/@claude-flow/cli/src/services/harness-flywheel-generations.ts+++ b/v3/@claude-flow/cli/src/services/harness-flywheel-generations.ts@@ -40,6 +40,7 @@ export interface GenerationDeps {getPatterns: () => HarvestPattern[] | Promise<HarvestPattern[]>;search: (q: string, cfg: RetrievalConfig) => Promise<RankedItem[]> | RankedItem[];anchorTasks: AnchorTask[];+ humanEvalHash?: string; // content hash of the FROZEN human eval set anchorTasks come fromsample?: number;now: number;applyFn?: (cfg: Record<string, number>, hash: string, generation: number) => void;@@ -236,8 +237,11 @@ export async function runFlywheelGeneration(root: string, deps: GenerationDeps):const canaryRollbackRate = CANARY.length ? cRoll / CANARY.length : 0;const candAnchor = await anchorMean(cand);const redblue: 'PASS' | 'FAIL' = candAnchor >= baseAnchor - ANCHOR_TOL ? 'PASS' : 'FAIL';+ // per-generation HUMAN-RELEVANCE delta on the frozen eval set (anti-overfitting+ // visibility): if this stays ~0 while benchmark Ξ > 0, the loop is overfitting.+ const humanRelevanceDelta = candAnchor - baseAnchor;- const bundle = runRealEvolveRound({ baseline: baseline as unknown as Record<string, number>, candidate: cand as unknown as Record<string, number>, holdout, generation, parent, branch: 'main', now: deps.now, redblue, canaryRollbackRate, corpus: FROZEN_CORPUS });+ const bundle = runRealEvolveRound({ baseline: baseline as unknown as Record<string, number>, candidate: cand as unknown as Record<string, number>, holdout, generation, parent, branch: 'main', now: deps.now, redblue, canaryRollbackRate, humanRelevanceDelta, humanEvalHash: deps.humanEvalHash, corpus: FROZEN_CORPUS });appendAttempt(root, bundle);if (bundle.decisionReceipt.promoted) appendPromotion(root, bundle);@@ -309,6 +313,9 @@ export interface FlywheelStatus {plateau: PlateauReport;mutation: MutationStat[];axisEffectiveness: AxisStat[]; // per-dimension payoff driving the meta-learning bias+ cumulativeBenchmarkDelta: number; // Ξ£ self-supervised (proxy) Ξ over promotions+ cumulativeHumanRelevanceDelta: number; // Ξ£ frozen human-eval Ξ over promotions β if β0 while benchmarkβ«0 β overfitting+ humanEvalHash: string | null; // frozen human eval set the deltas are againstserved: ServedState;champion: { config: Record<string, number>; hash: string | null };}@@ -325,6 +332,9 @@ export function flywheelStatus(root: string): FlywheelStatus {plateau: detectPlateau(attempts.length ? attempts : promotions, { window: 5 }),mutation: mutationEffectiveness(attempts.length ? attempts : promotions),axisEffectiveness: axisEffectiveness(promotions),+ cumulativeBenchmarkDelta: promotions.reduce((s, b) => s + (b.deltas.benchmark ?? 0), 0),+ cumulativeHumanRelevanceDelta: promotions.reduce((s, b) => s + (b.deltas.humanRelevance ?? 0), 0),+ humanEvalHash: promotions.length ? (promotions[promotions.length - 1].humanEvalHash ?? null) : null,served: servedChampion(root),champion: { config: champ.config, hash: champ.hash },};diff --git a/v3/@claude-flow/cli/src/services/harness-flywheel-runtime.ts b/v3/@claude-flow/cli/src/services/harness-flywheel-runtime.tsindex 969428fa3a..7301459770 100644--- a/v3/@claude-flow/cli/src/services/harness-flywheel-runtime.ts+++ b/v3/@claude-flow/cli/src/services/harness-flywheel-runtime.ts@@ -6,21 +6,8 @@*/import { harnessLoopOptedIn } from './harness-worker.js';import { runFlywheelTick, type FlywheelDeps, type FlywheelResult, type RetrievalConfig, type AnchorTask } from './harness-flywheel.js';-import { runFlywheelGeneration, checkServedChampionDrift, type GenerationResult, type GenerationDeps, type AnchorTask as GenAnchorTask } from './harness-flywheel-generations.js';--/** The frozen ADR-081 human-relevance anchor in the generation runner's shape. */-const GEN_ANCHOR: GenAnchorTask[] = [- ['how was the Opus model alias fixed', ['opus 4.8', 'opus alias', 'opus model alias', '#2232']],- ['self-learning wiring task-completed pretrain', ['self-learning', 'adr-074', '#2245', 'task-completed']],- ['deterministic codemod engine var-to-const', ['deterministic tier-1 codemod', 'adr-143', 'codemod', 'var-to-const']],- ['MCP server orphan leak parent-death', ['mcp orphan', 'parent-death', '#2234', 'orphan on every claude']],- ['unified learning stats aggregator', ['unified learning-stats', 'adr-075']],- ['structured distillation 4-field schema', ['structured distillation', 'adr-076', '4-field schema']],- ['SQL injection migrate.ts table identifier', ['sql injection', 'migrate.ts', 'agentdb', 'cve']],- ['recall@k HNSW benchmark harness', ['hnsw', 'recall@k', 'benchmark intelligence']],- ['Q-learning encoder keyword block', ['q-state encoder', 'keyword block', '#2239', 'q-encoder']],- ['security hardening crypto random IDs', ['cwe-347', 'crypto.randomuuid', 'random id', 'crypto random']],-].map(([q, labels], i) => ({ id: `anchor-${i}`, q: q as string, labels: labels as string[] }));+import { runFlywheelGeneration, checkServedChampionDrift, type GenerationResult, type GenerationDeps } from './harness-flywheel-generations.js';+import { loadFrozenHumanEval } from './harness-frozen-eval.js';/** The human-labeled ADR-081 anchor β the never-regress relevance set. */const ANCHOR: AnchorTask[] = [@@ -85,13 +72,16 @@ export async function runFlywheelGenerationWorker(const neural = await import('../mcp-tools/neural-tools.js');const tool = neural.neuralTools.find((t) => t.name === 'neural_patterns');if (!tool) return { ran: false, reason: 'neural_patterns tool unavailable', generation: 0 };+ // Load the FROZEN, hashed, public human eval set (throws if it has drifted).+ const frozen = loadFrozenHumanEval();const deps: GenerationDeps = {getPatterns: () => neural.getStorePatterns(),search: async (query, cfg: RetrievalConfig) => {const r = await tool.handler({ action: 'search', query, mode: 'hybrid', limit: 5, rerank: false, ...cfg }) as { results?: Array<{ id?: string; name?: string }> };return (r.results || []).slice(0, 5).map((m) => ({ id: m?.id ?? '', name: m?.name ?? '' }));},- anchorTasks: GEN_ANCHOR,+ anchorTasks: frozen.tasks,+ humanEvalHash: frozen.corpusHash,sample: opts.sample ?? 120,now: opts.now ?? Date.now(),};diff --git a/v3/@claude-flow/cli/src/services/harness-frozen-eval.ts b/v3/@claude-flow/cli/src/services/harness-frozen-eval.tsnew file mode 100644index 0000000000..5ad9d1fa57--- /dev/null+++ b/v3/@claude-flow/cli/src/services/harness-frozen-eval.ts@@ -0,0 +1,69 @@+/**+ * Frozen public human-labeled eval set (ADR-176 β anti-overfitting).+ *+ * The flywheel's compounding gains are measured on a self-supervised retrieval+ * benchmark, which could overfit β improving self-retrieval while human+ * relevance stays merely *preserved*, not improved. The defense is a SINGLE,+ * FROZEN, PUBLIC, hashed human-labeled eval set (`.claude/eval/β¦-v1.json`):+ * - it is the red/blue anchor the flywheel must never regress, AND+ * - the set against which a per-generation HUMAN-RELEVANCE DELTA is recorded+ * in every receipt β so "self-retrieval up, human relevance flat" is+ * observable in the lineage, not hidden.+ *+ * `loadFrozenHumanEval()` verifies the file's content hash against a PINNED+ * constant and throws on mismatch β the set cannot silently drift; changing it+ * means shipping a new versioned file, not editing this one.+ */+import * as fs from 'fs';+import * as path from 'path';+import { fileURLToPath } from 'url';+import { createRequire } from 'module';+import { createHash } from 'node:crypto';++const __dirname = path.dirname(fileURLToPath(import.meta.url));++export const FROZEN_HUMAN_EVAL_VERSION = 'human-relevance-frozen-v1';+export const FROZEN_HUMAN_EVAL_FILE = path.join('.claude', 'eval', `${FROZEN_HUMAN_EVAL_VERSION}.json`);+/** Pinned canonical hash of the frozen eval tasks β the tamper-evidence anchor. */+export const FROZEN_HUMAN_EVAL_HASH = 'sha256:6096e48ef8f2182e0f00348a953f0f00fe0415575b300234fe2316f37b768200';++export interface HumanEvalTask { id: string; q: string; labels: string[]; }+export interface FrozenHumanEval { version: string; tasks: HumanEvalTask[]; corpusHash: string; }++function canon(v: unknown): unknown {+ return Array.isArray(v) ? v.map(canon)+ : (v && typeof v === 'object') ? Object.fromEntries(Object.keys(v as object).sort().map((k) => [k, canon((v as Record<string, unknown>)[k])])) : v;+}+/** Canonical content hash over the tasks β order-independent (tasks sorted by id). */+export function humanEvalHash(tasks: HumanEvalTask[]): string {+ const sorted = [...tasks].sort((a, b) => a.id.localeCompare(b.id));+ return 'sha256:' + createHash('sha256').update(JSON.stringify(canon(sorted))).digest('hex');+}++function locate(): string | null {+ const candidates: string[] = [];+ try {+ const req = createRequire(import.meta.url);+ candidates.push(path.join(path.dirname(req.resolve('@claude-flow/cli/package.json')), FROZEN_HUMAN_EVAL_FILE));+ } catch { /* not resolvable in this context */ }+ candidates.push(path.resolve(__dirname, '..', '..', '..', FROZEN_HUMAN_EVAL_FILE)); // dist/src/services β pkg root+ candidates.push(path.resolve(__dirname, '..', '..', FROZEN_HUMAN_EVAL_FILE)); // src/services β pkg root+ for (const c of candidates) if (fs.existsSync(c)) return c;+ return null;+}++/**+ * Load + verify the frozen human eval set. Throws if missing or if its content+ * hash != the pinned FROZEN_HUMAN_EVAL_HASH (the "frozen" guarantee).+ */+export function loadFrozenHumanEval(): FrozenHumanEval {+ const p = locate();+ if (!p) throw new Error(`frozen human eval set not found (${FROZEN_HUMAN_EVAL_FILE})`);+ const parsed = JSON.parse(fs.readFileSync(p, 'utf-8')) as { version?: string; tasks?: HumanEvalTask[] };+ const tasks = parsed.tasks ?? [];+ const corpusHash = humanEvalHash(tasks);+ if (corpusHash !== FROZEN_HUMAN_EVAL_HASH) {+ throw new Error(`frozen human eval hash mismatch β set has drifted (got ${corpusHash}, pinned ${FROZEN_HUMAN_EVAL_HASH}); supersede with a new versioned file, do not edit`);+ }+ return { version: parsed.version ?? FROZEN_HUMAN_EVAL_VERSION, tasks, corpusHash };+}diff --git a/v3/docs/adr/ADR-176-proven-self-benchmarking-harness-loop.md b/v3/docs/adr/ADR-176-proven-self-benchmarking-harness-loop.mdindex 55de92bc84..633d019028 100644--- a/v3/docs/adr/ADR-176-proven-self-benchmarking-harness-loop.md+++ b/v3/docs/adr/ADR-176-proven-self-benchmarking-harness-loop.md@@ -158,6 +158,12 @@ The discovery was **autonomous** β a coarseβlocal multi-axis grid with **con**Meta-learning (evidence β action).** `axisEffectiveness()` attributes each promotion's held-out Ξ to the policy axes that moved, and `biasedGrid()` concentrates the search on axes that have historically paid off (a Β±1 exploration floor on every axis, but expanded range + pairwise joint moves on productive ones) β the optimizer *uses* the lineage-as-knowledge-base, not just records it. **Deployment-safety canary.** `checkServedChampionDrift()` runs each tick *before* the generation: it re-scores the currently-served champion against its predecessor on a **fresh harvest of the current store**, and auto-**rolls back** the active policy if the champion has drifted (self-retrieval or the human anchor). Real ongoing measurement + real rollback on real evolving data β the honest analogue of a live-traffic canary, without fabricating traffic.+**Biggest failure mode + fix (anti-overfitting).** The known risk: the loop overfits the self-supervised proxy while human relevance is *preserved but not improved*. Two defenses make that **visible and falsifiable** rather than hidden:+- a **frozen, public, hashed human-labeled eval set** (`.claude/eval/human-relevance-frozen-v1.json`, loaded via `harness-frozen-eval.ts` which pins the content hash and throws on drift) β the single source of truth for the red/blue anchor;+- a **per-generation human-relevance delta** recorded in every receipt (`deltas.humanRelevance` vs `humanEvalHash`) and surfaced by `flywheelStatus` β so "proxy Ξ β« 0 while human Ξ β 0" shows up in the status as an explicit **overfitting** flag instead of being buried.++**Acceptance test (clean-room replay).** `scripts/replay-generation.mjs` (a CI gate): from a clean install, replay one PROMOTED generation from its receipt alone β every embedded hash recomputes bit-identically and re-running `accept/v1+sig` on independently-recomputed inputs reproduces `promoted=true`, with **network access trapped** (offline). A promotion is reproducible without trusting any service log.+**Honest scope of the claim.** The measured improvement is on a **self-supervised self-retrieval** benchmark (find a doc from its own body), gated so human-labeled relevance does **not** regress. So the demonstrated capability is: *retrieval gets generation-over-generation better at self-retrieval while preserving human relevance and deployment safety* β it is **not** a claim that human-labeled relevance improved (that is held flat by design). Proving compounding gains on human-labeled relevance directly would need a large human-labeled suite (out of scope for $0 autonomous operation). The value shown is that **the wheel provably turns**: verified improvements accumulate into an auditable, replayable lineage without human intervention.**Local vs global trust.** A locally-mined, gate-cleared champion may be adopted **locally, unsigned** (the install trusting its own execution-verified evidence on its own data). Cross-install propagation still requires the config-signed champion (ADR-177). Local self-optimization and global distribution are separate trust domains.