Skip to content

armgit / ruflo

Public Imported from GitHub
0

🌊 The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated

feat(harness): anti-overfitting β€” frozen public human eval + per-gen red/blue deltas + clean-room replay (ADR-176) (#2580)

rUvaee3b999102f
diff --git a/.github/workflows/v3-ci.yml b/.github/workflows/v3-ci.yml
index c7f6167785..1699c641e7 100644
--- a/.github/workflows/v3-ci.yml
+++ b/.github/workflows/v3-ci.yml
@@ -339,6 +339,14 @@ jobs:
working-directory: v3/@claude-flow/cli
run: node scripts/smoke-flywheel-proof.mjs
+ - name: "ADR-176 β€” clean-room replay acceptance test (offline, identical hashes)"
+ # Replay a PROMOTED generation from its receipt alone: recompute every
+ # hash bit-identically and re-run accept/v1+sig to reproduce promoted=true,
+ # with network access trapped. Proves promotions are reproducible without
+ # trusting any service log.
+ working-directory: v3/@claude-flow/cli
+ run: node scripts/replay-generation.mjs
+
- name: Run tests
working-directory: v3
# Tolerate exit code 139 (SIGSEGV) ONLY when it occurs at process
diff --git a/v3/@claude-flow/cli/.claude/eval/human-relevance-frozen-v1.json b/v3/@claude-flow/cli/.claude/eval/human-relevance-frozen-v1.json
new file mode 100644
index 0000000000..d22dcc1874
--- /dev/null
+++ b/v3/@claude-flow/cli/.claude/eval/human-relevance-frozen-v1.json
@@ -0,0 +1,17 @@
+{
+ "version": "human-relevance-frozen-v1",
+ "description": "Frozen, public, human-labeled relevance eval set (ADR-081). Immutable β€” the red/blue anchor the flywheel must never regress AND against which per-generation human-relevance deltas are reported. Do not edit; supersede with a new version file.",
+ "metric": "nDCG@3 over labelled substring-relevance",
+ "tasks": [
+ { "id": "hr-0", "q": "how was the Opus model alias fixed", "labels": ["opus 4.8", "opus alias", "opus model alias", "#2232"] },
+ { "id": "hr-1", "q": "self-learning wiring task-completed pretrain", "labels": ["self-learning", "adr-074", "#2245", "task-completed"] },
+ { "id": "hr-2", "q": "deterministic codemod engine var-to-const", "labels": ["deterministic tier-1 codemod", "adr-143", "codemod", "var-to-const"] },
+ { "id": "hr-3", "q": "MCP server orphan leak parent-death", "labels": ["mcp orphan", "parent-death", "#2234", "orphan on every claude"] },
+ { "id": "hr-4", "q": "unified learning stats aggregator", "labels": ["unified learning-stats", "adr-075"] },
+ { "id": "hr-5", "q": "structured distillation 4-field schema", "labels": ["structured distillation", "adr-076", "4-field schema"] },
+ { "id": "hr-6", "q": "SQL injection migrate.ts table identifier", "labels": ["sql injection", "migrate.ts", "agentdb", "cve"] },
+ { "id": "hr-7", "q": "recall@k HNSW benchmark harness", "labels": ["hnsw", "recall@k", "benchmark intelligence"] },
+ { "id": "hr-8", "q": "Q-learning encoder keyword block", "labels": ["q-state encoder", "keyword block", "#2239", "q-encoder"] },
+ { "id": "hr-9", "q": "security hardening crypto random IDs", "labels": ["cwe-347", "crypto.randomuuid", "random id", "crypto random"] }
+ ]
+}
diff --git a/v3/@claude-flow/cli/__tests__/harness-flywheel-generations.test.ts b/v3/@claude-flow/cli/__tests__/harness-flywheel-generations.test.ts
index b97e1b6965..6515661a8c 100644
--- a/v3/@claude-flow/cli/__tests__/harness-flywheel-generations.test.ts
+++ b/v3/@claude-flow/cli/__tests__/harness-flywheel-generations.test.ts
@@ -135,6 +135,23 @@ describe('runFlywheelGeneration β€” compounding autonomy loop', () => {
expect(servedChampion(root).championHash).toBeNull(); // reverted
});
+ it('records a per-generation human-relevance delta and exposes the overfitting signal', async () => {
+ const root = mkdtempSync(join(tmpdir(), 'fwg-'));
+ await runFlywheelGeneration(root, { ...makeDeps(1000), humanEvalHash: 'sha256:frozen-test' });
+ const promos = loadPromotions(root);
+ expect(promos.length).toBeGreaterThanOrEqual(1);
+ // the human-relevance delta is recorded on every promotion (against the frozen set)
+ expect(promos[0].deltas.humanRelevance).toBeDefined();
+ expect(promos[0].humanEvalHash).toBe('sha256:frozen-test');
+
+ const s = flywheelStatus(root);
+ expect(s.cumulativeBenchmarkDelta).toBeGreaterThan(0); // proxy (self-retrieval) improved
+ // stub's anchor is config-independent β†’ human relevance flat β†’ the overfitting
+ // signal is now VISIBLE (proxy up, human ~0), not hidden.
+ expect(Math.abs(s.cumulativeHumanRelevanceDelta)).toBeLessThan(0.01);
+ expect(s.humanEvalHash).toBe('sha256:frozen-test');
+ });
+
it('no-op (never throws) on a store too small to harvest', async () => {
const root = mkdtempSync(join(tmpdir(), 'fwg-'));
const r = await runFlywheelGeneration(root, { ...makeDeps(1), getPatterns: () => patterns.slice(0, 5) });
diff --git a/v3/@claude-flow/cli/__tests__/harness-frozen-eval.test.ts b/v3/@claude-flow/cli/__tests__/harness-frozen-eval.test.ts
new file mode 100644
index 0000000000..9c2e72e0b3
--- /dev/null
+++ b/v3/@claude-flow/cli/__tests__/harness-frozen-eval.test.ts
@@ -0,0 +1,24 @@
+/**
+ * Frozen public human-labeled eval set (ADR-176 anti-overfitting).
+ */
+import { describe, it, expect } from 'vitest';
+import {
+ loadFrozenHumanEval, humanEvalHash, FROZEN_HUMAN_EVAL_HASH, FROZEN_HUMAN_EVAL_VERSION,
+} from '../src/services/harness-frozen-eval.js';
+
+describe('frozen human eval set', () => {
+ it('loads and its content hash matches the pinned constant (frozen guarantee)', () => {
+ const e = loadFrozenHumanEval();
+ expect(e.version).toBe(FROZEN_HUMAN_EVAL_VERSION);
+ expect(e.tasks.length).toBe(10);
+ e.tasks.forEach((t) => { expect(t.q).toBeTruthy(); expect(t.labels.length).toBeGreaterThan(0); });
+ expect(e.corpusHash).toBe(FROZEN_HUMAN_EVAL_HASH);
+ });
+
+ it('hash is order-independent but tamper-evident', () => {
+ const e = loadFrozenHumanEval();
+ expect(humanEvalHash([...e.tasks].reverse())).toBe(e.corpusHash); // order-independent
+ const tampered = e.tasks.map((t, i) => (i === 0 ? { ...t, labels: [...t.labels, 'injected'] } : t));
+ expect(humanEvalHash(tampered)).not.toBe(e.corpusHash); // any edit changes the hash
+ });
+});
diff --git a/v3/@claude-flow/cli/scripts/flywheel-status.mjs b/v3/@claude-flow/cli/scripts/flywheel-status.mjs
index e8057f5379..af29af1e62 100644
--- a/v3/@claude-flow/cli/scripts/flywheel-status.mjs
+++ b/v3/@claude-flow/cli/scripts/flywheel-status.mjs
@@ -24,6 +24,12 @@ console.log('='.repeat(64));
console.log(`generations (promotions) : ${s.generations}`);
console.log(`attempts (incl. refusals): ${s.attempts}`);
console.log(`cumulative held-out Ξ” : ${s.lineage.cumulativeHeldOutImprovement.toFixed(4)}`);
+// Anti-overfitting view: proxy (self-supervised) gain vs FROZEN human-relevance gain.
+const bench = s.cumulativeBenchmarkDelta ?? 0, human = s.cumulativeHumanRelevanceDelta ?? 0;
+const overfit = bench > 0.02 && human <= 0.005;
+console.log(`cumulative proxy Ξ” : ${bench.toFixed(4)} (self-supervised self-retrieval)`);
+console.log(`cumulative HUMAN Ξ” : ${human.toFixed(4)} (frozen human-labeled eval)${overfit ? ' ⚠️ proxy up but human flat β†’ OVERFITTING' : ''}`);
+console.log(`frozen human eval : ${s.humanEvalHash ? s.humanEvalHash.slice(0, 26) + '…' : '(none)'}`);
console.log(`lineage intact : ${s.lineage.lineageIntact} replayable: ${s.lineage.allReplayable}`);
console.log(`immutable root : ${(s.lineage.rootHash || '(none yet)').slice(0, 26)}…`);
console.log(`plateau : ${s.plateau.status} β€” ${s.plateau.rationale}`);
diff --git a/v3/@claude-flow/cli/scripts/replay-generation.mjs b/v3/@claude-flow/cli/scripts/replay-generation.mjs
new file mode 100644
index 0000000000..0c71e12486
--- /dev/null
+++ b/v3/@claude-flow/cli/scripts/replay-generation.mjs
@@ -0,0 +1,77 @@
+#!/usr/bin/env node
+/**
+ * Acceptance test (ADR-176) β€” clean-room replay of a promoted generation.
+ *
+ * From a clean install, replay ONE promoted generation bundle from
+ * .claude/evolve-proof/ and verify the SAME candidate passes accept/v1+sig with
+ * IDENTICAL hashes and NO NETWORK β€” i.e. the promotion is reproducible from the
+ * receipt alone, without trusting any service log.
+ *
+ * Proves: (1) each embedded hash recomputes bit-identically, (2) re-running the
+ * versioned rule on independently-recomputed inputs reproduces promoted=true,
+ * (3) zero network access (fetch is trapped and any call fails the test).
+ *
+ * Usage: node scripts/replay-generation.mjs [--file <bundle.json>] [--dir <projectRoot>]
+ */
+import { readFileSync, existsSync, readdirSync } from 'node:fs';
+import { fileURLToPath } from 'node:url';
+import { dirname, join, resolve } from 'node:path';
+import { createHash } from 'node:crypto';
+
+// ── Hard no-network guard: any network call during replay fails the test. ──
+for (const name of ['fetch']) {
+ globalThis[name] = () => { throw new Error(`NETWORK ACCESS during replay (${name}) β€” replay must be offline`); };
+}
+
+const CLI_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..');
+const ep = await import(`file://${join(CLI_ROOT, 'dist/src/services/evolve-proof.js')}`);
+
+const arg = (f) => { const i = process.argv.indexOf(f); return i > -1 ? process.argv[i + 1] : null; };
+const root = resolve(arg('--dir') || CLI_ROOT);
+const evalDir = join(root, '.claude', 'evolve-proof');
+
+// pick a promoted generation: explicit --file, else the highest-numbered real-generation-*.
+let file = arg('--file');
+if (!file) {
+ const reals = existsSync(evalDir) ? readdirSync(evalDir).filter((f) => /^real-generation-\d+\.json$/.test(f)).sort() : [];
+ if (!reals.length) { console.error(`no promoted generation found in ${evalDir}`); process.exit(1); }
+ file = join(evalDir, reals[reals.length - 1]);
+} else if (!existsSync(file)) { file = join(evalDir, file); }
+
+const bundle = JSON.parse(readFileSync(file, 'utf-8'));
+console.log(`clean-room replay of ${file}`);
+console.log('='.repeat(70));
+if (!bundle.decisionReceipt?.promoted) { console.error('βœ— bundle is not a PROMOTED generation'); process.exit(1); }
+
+// (1) hashes recompute bit-identically
+const sha = (s) => 'sha256:' + createHash('sha256').update(s).digest('hex');
+const canon = (v) => Array.isArray(v) ? v.map(canon) : (v && typeof v === 'object' ? Object.fromEntries(Object.keys(v).sort().map((k) => [k, canon(v[k])])) : v);
+const cstr = (v) => JSON.stringify(canon(v));
+// manifest hash mirrors evolve-proof.manifestHash (canonicalManifestBytes β†’ sha256)
+const { canonicalManifestBytes } = await import(`file://${join(CLI_ROOT, 'dist/src/config/proven-config.js')}`);
+const mh = (m) => sha(canonicalManifestBytes(m).toString('utf-8'));
+
+const checks = [
+ ['input holdout hash', sha(cstr(bundle.holdout)), bundle.inputHoldoutHash],
+ ['baseline manifest hash', mh(bundle.baselineManifest), bundle.baselineManifestHash],
+ ['candidate manifest hash', mh(bundle.candidateManifest), bundle.candidateManifestHash],
+];
+let idOK = true;
+for (const [label, got, want] of checks) {
+ const same = got === want;
+ idOK = idOK && same;
+ console.log(`${same ? 'βœ“' : 'βœ—'} ${label}: ${same ? 'identical' : `MISMATCH\n recomputed ${got}\n recorded ${want}`}`);
+}
+
+// (2) re-run accept/v1+sig on independently-recomputed inputs β†’ promoted=true
+const v = ep.verifyReceiptBundle(bundle);
+console.log(`${v.valid ? 'βœ“' : 'βœ—'} independent verify: ${v.explanation}`);
+console.log(` decisionMatches=${v.decisionMatches} ruleVersion=${bundle.decisionReceipt.promotionRuleVersion} noAutoServe=${v.noAutoServe}`);
+if (bundle.deltas?.humanRelevance !== undefined)
+ console.log(` human-relevance Ξ” (frozen eval ${(bundle.humanEvalHash||'').slice(0,18)}…): ${bundle.deltas.humanRelevance.toFixed(4)}`);
+
+const pass = idOK && v.valid && v.decisionMatches;
+console.log('='.repeat(70));
+console.log(pass ? 'ACCEPTANCE TEST PASSED β€” promoted generation reproducible from its receipt, identical hashes, offline.'
+ : 'ACCEPTANCE TEST FAILED');
+process.exit(pass ? 0 : 1);
diff --git a/v3/@claude-flow/cli/scripts/smoke-flywheel-proof.mjs b/v3/@claude-flow/cli/scripts/smoke-flywheel-proof.mjs
index ae14d5fa4b..b168b772a1 100644
--- a/v3/@claude-flow/cli/scripts/smoke-flywheel-proof.mjs
+++ b/v3/@claude-flow/cli/scripts/smoke-flywheel-proof.mjs
@@ -26,6 +26,14 @@ const ok = (m) => console.log(`βœ“ ${m}`);
const ep = await import(`file://${d('services/evolve-proof.js')}`);
const rvfa = await import(`file://${d('config/proven-config-rvfa.js')}`);
const pc = await import(`file://${d('config/proven-config.js')}`);
+const fe = await import(`file://${d('services/harness-frozen-eval.js')}`);
+
+// 0. the frozen public human-labeled eval set must not have drifted (pinned hash)
+try {
+ const e = fe.loadFrozenHumanEval();
+ e.corpusHash === fe.FROZEN_HUMAN_EVAL_HASH ? ok(`frozen human eval set intact (${e.tasks.length} tasks, ${e.corpusHash.slice(0, 18)}…)`)
+ : fail(`frozen human eval hash drifted`);
+} catch (err) { fail(`frozen human eval load: ${err.message}`); }
// 1. synthetic proof-of-mechanism fixture
const g0 = JSON.parse(readFileSync(cl('evolve-proof/generation-0.json'), 'utf-8'));
diff --git a/v3/@claude-flow/cli/src/services/evolve-proof.ts b/v3/@claude-flow/cli/src/services/evolve-proof.ts
index a6feb47b19..cbc5a3547a 100644
--- a/v3/@claude-flow/cli/src/services/evolve-proof.ts
+++ b/v3/@claude-flow/cli/src/services/evolve-proof.ts
@@ -65,7 +65,13 @@ export interface ShadowRegistration {
export interface CostReceipt { usd: number; llmCalls: number; tier: string; notes: string; }
/** Multi-dimensional deltas vs the parent β€” distinguishes *why* a change is (un)safe. */
-export interface ChangeDeltas { benchmark: number; security: number; cost: number; }
+export interface ChangeDeltas {
+ benchmark: number; // the optimization objective (e.g. self-supervised self-retrieval)
+ security: number; // redblue verdict (-1 = FAILED)
+ cost: number;
+ humanRelevance?: number; // per-generation Ξ” on the FROZEN human-labeled eval set (ADR-176 anti-overfitting).
+ // ~0 across generations while `benchmark` > 0 β‡’ overfitting to the proxy β€” now visible.
+}
/**
* Causal promotion record (not just provenance). Answers *why* a candidate won β€”
@@ -110,6 +116,7 @@ export interface EvolveReceiptBundle {
mutationClass: string;
mutationSummary: string;
deltas: ChangeDeltas;
+ humanEvalHash?: string; // the frozen human eval set the humanRelevance delta is measured against
promotion: PromotionRecord | null; // populated on pass
regression: RegressionRecord | null; // populated on reject
// ── embedded evidence (so verification needs no service logs) ──
@@ -145,6 +152,8 @@ export interface AssembleOpts {
cost: { tier: string; notes: string };
redblue?: 'PASS' | 'FAIL' | 'SKIPPED'; drift?: number;
canaryRollbackRate?: number; // SEPARATE deployment-safety signal; default = strict per-held-out-task regression
+ humanRelevanceDelta?: number; // per-gen Ξ” on the frozen human eval set
+ humanEvalHash?: string; // which frozen human eval set the delta is against
layer?: string; corpus?: string;
}
@@ -190,7 +199,7 @@ export function assembleBundle(baseline: Record<string, number>, candidate: Reco
} : null;
const { mutationClass, mutationSummary } = classifyMutation(baseline, candidate);
- const deltas: ChangeDeltas = { benchmark: candidateHeldOut - baselineHeldOut, security: o.redblue === 'FAIL' ? -1 : 0, cost: 0 };
+ const deltas: ChangeDeltas = { benchmark: candidateHeldOut - baselineHeldOut, security: o.redblue === 'FAIL' ? -1 : 0, cost: 0, humanRelevance: o.humanRelevanceDelta };
const promotion: PromotionRecord | null = promoted ? { parentManifestHash: o.parent, candidateManifestHash, mutationClass, mutationSummary, deltas, decisionReceipt } : null;
const regression: RegressionRecord | null = promoted ? null : {
candidateManifestHash, ancestor: o.parent ?? baselineManifestHash, mutationClass,
@@ -203,7 +212,7 @@ export function assembleBundle(baseline: Record<string, number>, candidate: Reco
meetsPromotionRule: { version: PROMOTION_RULE_VERSION, result: promoted },
decisionReceipt, shadow,
costReceipt: { usd: 0, llmCalls: 0, tier: o.cost.tier, notes: o.cost.notes },
- mutationClass, mutationSummary, deltas, promotion, regression,
+ mutationClass, mutationSummary, deltas, humanEvalHash: o.humanEvalHash, promotion, regression,
holdout, baselineManifest, candidateManifest,
};
}
@@ -218,12 +227,15 @@ export function assembleBundle(baseline: Record<string, number>, candidate: Reco
export function runRealEvolveRound(opts: {
baseline: Record<string, number>; candidate: Record<string, number>; holdout: HoldoutTask[];
generation: number; parent: string | null; branch?: string; now: number;
- redblue?: 'PASS' | 'FAIL' | 'SKIPPED'; drift?: number; canaryRollbackRate?: number; corpus: string;
+ redblue?: 'PASS' | 'FAIL' | 'SKIPPED'; drift?: number; canaryRollbackRate?: number;
+ humanRelevanceDelta?: number; humanEvalHash?: string; corpus: string;
}): EvolveReceiptBundle {
return assembleBundle(opts.baseline, opts.candidate, opts.holdout, {
generation: opts.generation, parent: opts.parent, branch: opts.branch ?? 'main', now: opts.now, kind: 'real',
cost: { tier: 'real-local', notes: 'measured on the frozen anchor via live retrieval β€” no LLM, no network' },
- redblue: opts.redblue, drift: opts.drift, canaryRollbackRate: opts.canaryRollbackRate, layer: 'real/retrieval', corpus: opts.corpus,
+ redblue: opts.redblue, drift: opts.drift, canaryRollbackRate: opts.canaryRollbackRate,
+ humanRelevanceDelta: opts.humanRelevanceDelta, humanEvalHash: opts.humanEvalHash,
+ layer: 'real/retrieval', corpus: opts.corpus,
});
}
diff --git a/v3/@claude-flow/cli/src/services/harness-flywheel-generations.ts b/v3/@claude-flow/cli/src/services/harness-flywheel-generations.ts
index bb604a70ac..5291d46565 100644
--- a/v3/@claude-flow/cli/src/services/harness-flywheel-generations.ts
+++ b/v3/@claude-flow/cli/src/services/harness-flywheel-generations.ts
@@ -40,6 +40,7 @@ export interface GenerationDeps {
getPatterns: () => HarvestPattern[] | Promise<HarvestPattern[]>;
search: (q: string, cfg: RetrievalConfig) => Promise<RankedItem[]> | RankedItem[];
anchorTasks: AnchorTask[];
+ humanEvalHash?: string; // content hash of the FROZEN human eval set anchorTasks come from
sample?: number;
now: number;
applyFn?: (cfg: Record<string, number>, hash: string, generation: number) => void;
@@ -236,8 +237,11 @@ export async function runFlywheelGeneration(root: string, deps: GenerationDeps):
const canaryRollbackRate = CANARY.length ? cRoll / CANARY.length : 0;
const candAnchor = await anchorMean(cand);
const redblue: 'PASS' | 'FAIL' = candAnchor >= baseAnchor - ANCHOR_TOL ? 'PASS' : 'FAIL';
+ // per-generation HUMAN-RELEVANCE delta on the frozen eval set (anti-overfitting
+ // visibility): if this stays ~0 while benchmark Ξ” > 0, the loop is overfitting.
+ const humanRelevanceDelta = candAnchor - baseAnchor;
- const bundle = runRealEvolveRound({ baseline: baseline as unknown as Record<string, number>, candidate: cand as unknown as Record<string, number>, holdout, generation, parent, branch: 'main', now: deps.now, redblue, canaryRollbackRate, corpus: FROZEN_CORPUS });
+ const bundle = runRealEvolveRound({ baseline: baseline as unknown as Record<string, number>, candidate: cand as unknown as Record<string, number>, holdout, generation, parent, branch: 'main', now: deps.now, redblue, canaryRollbackRate, humanRelevanceDelta, humanEvalHash: deps.humanEvalHash, corpus: FROZEN_CORPUS });
appendAttempt(root, bundle);
if (bundle.decisionReceipt.promoted) appendPromotion(root, bundle);
@@ -309,6 +313,9 @@ export interface FlywheelStatus {
plateau: PlateauReport;
mutation: MutationStat[];
axisEffectiveness: AxisStat[]; // per-dimension payoff driving the meta-learning bias
+ cumulativeBenchmarkDelta: number; // Ξ£ self-supervised (proxy) Ξ” over promotions
+ cumulativeHumanRelevanceDelta: number; // Ξ£ frozen human-eval Ξ” over promotions β€” if β‰ˆ0 while benchmark≫0 β‡’ overfitting
+ humanEvalHash: string | null; // frozen human eval set the deltas are against
served: ServedState;
champion: { config: Record<string, number>; hash: string | null };
}
@@ -325,6 +332,9 @@ export function flywheelStatus(root: string): FlywheelStatus {
plateau: detectPlateau(attempts.length ? attempts : promotions, { window: 5 }),
mutation: mutationEffectiveness(attempts.length ? attempts : promotions),
axisEffectiveness: axisEffectiveness(promotions),
+ cumulativeBenchmarkDelta: promotions.reduce((s, b) => s + (b.deltas.benchmark ?? 0), 0),
+ cumulativeHumanRelevanceDelta: promotions.reduce((s, b) => s + (b.deltas.humanRelevance ?? 0), 0),
+ humanEvalHash: promotions.length ? (promotions[promotions.length - 1].humanEvalHash ?? null) : null,
served: servedChampion(root),
champion: { config: champ.config, hash: champ.hash },
};
diff --git a/v3/@claude-flow/cli/src/services/harness-flywheel-runtime.ts b/v3/@claude-flow/cli/src/services/harness-flywheel-runtime.ts
index 969428fa3a..7301459770 100644
--- a/v3/@claude-flow/cli/src/services/harness-flywheel-runtime.ts
+++ b/v3/@claude-flow/cli/src/services/harness-flywheel-runtime.ts
@@ -6,21 +6,8 @@
*/
import { harnessLoopOptedIn } from './harness-worker.js';
import { runFlywheelTick, type FlywheelDeps, type FlywheelResult, type RetrievalConfig, type AnchorTask } from './harness-flywheel.js';
-import { runFlywheelGeneration, checkServedChampionDrift, type GenerationResult, type GenerationDeps, type AnchorTask as GenAnchorTask } from './harness-flywheel-generations.js';
-
-/** The frozen ADR-081 human-relevance anchor in the generation runner's shape. */
-const GEN_ANCHOR: GenAnchorTask[] = [
- ['how was the Opus model alias fixed', ['opus 4.8', 'opus alias', 'opus model alias', '#2232']],
- ['self-learning wiring task-completed pretrain', ['self-learning', 'adr-074', '#2245', 'task-completed']],
- ['deterministic codemod engine var-to-const', ['deterministic tier-1 codemod', 'adr-143', 'codemod', 'var-to-const']],
- ['MCP server orphan leak parent-death', ['mcp orphan', 'parent-death', '#2234', 'orphan on every claude']],
- ['unified learning stats aggregator', ['unified learning-stats', 'adr-075']],
- ['structured distillation 4-field schema', ['structured distillation', 'adr-076', '4-field schema']],
- ['SQL injection migrate.ts table identifier', ['sql injection', 'migrate.ts', 'agentdb', 'cve']],
- ['recall@k HNSW benchmark harness', ['hnsw', 'recall@k', 'benchmark intelligence']],
- ['Q-learning encoder keyword block', ['q-state encoder', 'keyword block', '#2239', 'q-encoder']],
- ['security hardening crypto random IDs', ['cwe-347', 'crypto.randomuuid', 'random id', 'crypto random']],
-].map(([q, labels], i) => ({ id: `anchor-${i}`, q: q as string, labels: labels as string[] }));
+import { runFlywheelGeneration, checkServedChampionDrift, type GenerationResult, type GenerationDeps } from './harness-flywheel-generations.js';
+import { loadFrozenHumanEval } from './harness-frozen-eval.js';
/** The human-labeled ADR-081 anchor β€” the never-regress relevance set. */
const ANCHOR: AnchorTask[] = [
@@ -85,13 +72,16 @@ export async function runFlywheelGenerationWorker(
const neural = await import('../mcp-tools/neural-tools.js');
const tool = neural.neuralTools.find((t) => t.name === 'neural_patterns');
if (!tool) return { ran: false, reason: 'neural_patterns tool unavailable', generation: 0 };
+ // Load the FROZEN, hashed, public human eval set (throws if it has drifted).
+ const frozen = loadFrozenHumanEval();
const deps: GenerationDeps = {
getPatterns: () => neural.getStorePatterns(),
search: async (query, cfg: RetrievalConfig) => {
const r = await tool.handler({ action: 'search', query, mode: 'hybrid', limit: 5, rerank: false, ...cfg }) as { results?: Array<{ id?: string; name?: string }> };
return (r.results || []).slice(0, 5).map((m) => ({ id: m?.id ?? '', name: m?.name ?? '' }));
},
- anchorTasks: GEN_ANCHOR,
+ anchorTasks: frozen.tasks,
+ humanEvalHash: frozen.corpusHash,
sample: opts.sample ?? 120,
now: opts.now ?? Date.now(),
};
diff --git a/v3/@claude-flow/cli/src/services/harness-frozen-eval.ts b/v3/@claude-flow/cli/src/services/harness-frozen-eval.ts
new file mode 100644
index 0000000000..5ad9d1fa57
--- /dev/null
+++ b/v3/@claude-flow/cli/src/services/harness-frozen-eval.ts
@@ -0,0 +1,69 @@
+/**
+ * Frozen public human-labeled eval set (ADR-176 β€” anti-overfitting).
+ *
+ * The flywheel's compounding gains are measured on a self-supervised retrieval
+ * benchmark, which could overfit β€” improving self-retrieval while human
+ * relevance stays merely *preserved*, not improved. The defense is a SINGLE,
+ * FROZEN, PUBLIC, hashed human-labeled eval set (`.claude/eval/…-v1.json`):
+ * - it is the red/blue anchor the flywheel must never regress, AND
+ * - the set against which a per-generation HUMAN-RELEVANCE DELTA is recorded
+ * in every receipt β€” so "self-retrieval up, human relevance flat" is
+ * observable in the lineage, not hidden.
+ *
+ * `loadFrozenHumanEval()` verifies the file's content hash against a PINNED
+ * constant and throws on mismatch β€” the set cannot silently drift; changing it
+ * means shipping a new versioned file, not editing this one.
+ */
+import * as fs from 'fs';
+import * as path from 'path';
+import { fileURLToPath } from 'url';
+import { createRequire } from 'module';
+import { createHash } from 'node:crypto';
+
+const __dirname = path.dirname(fileURLToPath(import.meta.url));
+
+export const FROZEN_HUMAN_EVAL_VERSION = 'human-relevance-frozen-v1';
+export const FROZEN_HUMAN_EVAL_FILE = path.join('.claude', 'eval', `${FROZEN_HUMAN_EVAL_VERSION}.json`);
+/** Pinned canonical hash of the frozen eval tasks β€” the tamper-evidence anchor. */
+export const FROZEN_HUMAN_EVAL_HASH = 'sha256:6096e48ef8f2182e0f00348a953f0f00fe0415575b300234fe2316f37b768200';
+
+export interface HumanEvalTask { id: string; q: string; labels: string[]; }
+export interface FrozenHumanEval { version: string; tasks: HumanEvalTask[]; corpusHash: string; }
+
+function canon(v: unknown): unknown {
+ return Array.isArray(v) ? v.map(canon)
+ : (v && typeof v === 'object') ? Object.fromEntries(Object.keys(v as object).sort().map((k) => [k, canon((v as Record<string, unknown>)[k])])) : v;
+}
+/** Canonical content hash over the tasks β€” order-independent (tasks sorted by id). */
+export function humanEvalHash(tasks: HumanEvalTask[]): string {
+ const sorted = [...tasks].sort((a, b) => a.id.localeCompare(b.id));
+ return 'sha256:' + createHash('sha256').update(JSON.stringify(canon(sorted))).digest('hex');
+}
+
+function locate(): string | null {
+ const candidates: string[] = [];
+ try {
+ const req = createRequire(import.meta.url);
+ candidates.push(path.join(path.dirname(req.resolve('@claude-flow/cli/package.json')), FROZEN_HUMAN_EVAL_FILE));
+ } catch { /* not resolvable in this context */ }
+ candidates.push(path.resolve(__dirname, '..', '..', '..', FROZEN_HUMAN_EVAL_FILE)); // dist/src/services β†’ pkg root
+ candidates.push(path.resolve(__dirname, '..', '..', FROZEN_HUMAN_EVAL_FILE)); // src/services β†’ pkg root
+ for (const c of candidates) if (fs.existsSync(c)) return c;
+ return null;
+}
+
+/**
+ * Load + verify the frozen human eval set. Throws if missing or if its content
+ * hash != the pinned FROZEN_HUMAN_EVAL_HASH (the "frozen" guarantee).
+ */
+export function loadFrozenHumanEval(): FrozenHumanEval {
+ const p = locate();
+ if (!p) throw new Error(`frozen human eval set not found (${FROZEN_HUMAN_EVAL_FILE})`);
+ const parsed = JSON.parse(fs.readFileSync(p, 'utf-8')) as { version?: string; tasks?: HumanEvalTask[] };
+ const tasks = parsed.tasks ?? [];
+ const corpusHash = humanEvalHash(tasks);
+ if (corpusHash !== FROZEN_HUMAN_EVAL_HASH) {
+ throw new Error(`frozen human eval hash mismatch β€” set has drifted (got ${corpusHash}, pinned ${FROZEN_HUMAN_EVAL_HASH}); supersede with a new versioned file, do not edit`);
+ }
+ return { version: parsed.version ?? FROZEN_HUMAN_EVAL_VERSION, tasks, corpusHash };
+}
diff --git a/v3/docs/adr/ADR-176-proven-self-benchmarking-harness-loop.md b/v3/docs/adr/ADR-176-proven-self-benchmarking-harness-loop.md
index 55de92bc84..633d019028 100644
--- a/v3/docs/adr/ADR-176-proven-self-benchmarking-harness-loop.md
+++ b/v3/docs/adr/ADR-176-proven-self-benchmarking-harness-loop.md
@@ -158,6 +158,12 @@ The discovery was **autonomous** — a coarse→local multi-axis grid with **con
**Meta-learning (evidence β†’ action).** `axisEffectiveness()` attributes each promotion's held-out Ξ” to the policy axes that moved, and `biasedGrid()` concentrates the search on axes that have historically paid off (a Β±1 exploration floor on every axis, but expanded range + pairwise joint moves on productive ones) β€” the optimizer *uses* the lineage-as-knowledge-base, not just records it. **Deployment-safety canary.** `checkServedChampionDrift()` runs each tick *before* the generation: it re-scores the currently-served champion against its predecessor on a **fresh harvest of the current store**, and auto-**rolls back** the active policy if the champion has drifted (self-retrieval or the human anchor). Real ongoing measurement + real rollback on real evolving data β€” the honest analogue of a live-traffic canary, without fabricating traffic.
+**Biggest failure mode + fix (anti-overfitting).** The known risk: the loop overfits the self-supervised proxy while human relevance is *preserved but not improved*. Two defenses make that **visible and falsifiable** rather than hidden:
+- a **frozen, public, hashed human-labeled eval set** (`.claude/eval/human-relevance-frozen-v1.json`, loaded via `harness-frozen-eval.ts` which pins the content hash and throws on drift) β€” the single source of truth for the red/blue anchor;
+- a **per-generation human-relevance delta** recorded in every receipt (`deltas.humanRelevance` vs `humanEvalHash`) and surfaced by `flywheelStatus` β€” so "proxy Ξ” ≫ 0 while human Ξ” β‰ˆ 0" shows up in the status as an explicit **overfitting** flag instead of being buried.
+
+**Acceptance test (clean-room replay).** `scripts/replay-generation.mjs` (a CI gate): from a clean install, replay one PROMOTED generation from its receipt alone β€” every embedded hash recomputes bit-identically and re-running `accept/v1+sig` on independently-recomputed inputs reproduces `promoted=true`, with **network access trapped** (offline). A promotion is reproducible without trusting any service log.
+
**Honest scope of the claim.** The measured improvement is on a **self-supervised self-retrieval** benchmark (find a doc from its own body), gated so human-labeled relevance does **not** regress. So the demonstrated capability is: *retrieval gets generation-over-generation better at self-retrieval while preserving human relevance and deployment safety* β€” it is **not** a claim that human-labeled relevance improved (that is held flat by design). Proving compounding gains on human-labeled relevance directly would need a large human-labeled suite (out of scope for $0 autonomous operation). The value shown is that **the wheel provably turns**: verified improvements accumulate into an auditable, replayable lineage without human intervention.
**Local vs global trust.** A locally-mined, gate-cleared champion may be adopted **locally, unsigned** (the install trusting its own execution-verified evidence on its own data). Cross-install propagation still requires the config-signed champion (ADR-177). Local self-optimization and global distribution are separate trust domains.