The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.
docs: add evaluator harness config scenario
Affaan Mustafa3dddfc827015
diff --git a/docs/ECC-2.0-GA-ROADMAP.md b/docs/ECC-2.0-GA-ROADMAP.mdindex 3033f07f4..b6e94b92a 100644--- a/docs/ECC-2.0-GA-ROADMAP.md+++ b/docs/ECC-2.0-GA-ROADMAP.md@@ -58,8 +58,8 @@ As of 2026-05-12:`examples/evaluator-rag-prototype/` define the first read-onlyself-improving harness prototype: scenario specs, traces, reports,candidate playbooks, verifier results, accepted maintainer-salvage,- billing-readiness, and CI-failure-diagnosis candidates, plus rejected- unsafe candidates.+ billing-readiness, CI-failure-diagnosis, and harness-config-quality+ candidates, plus rejected unsafe candidates.- The npm package surface now excludes Python bytecode/cache artifacts throughpackage `files` negation rules and a publish-surface regression test.- `docs/legacy-artifact-inventory.md` records that no `_legacy-documents-*`@@ -200,7 +200,7 @@ is not complete unless the evidence column exists and has been freshly verified.| AgentShield enterprise iteration | Policy gates, SARIF, packs, provenance, corpus, HTML reports, exception lifecycle audit | PRs #53, #55-#62 landed with test evidence | Needs PDF/export decision or next enterprise signal || ECC Tools next-level app | Billing audit, PR checks, deep analyzer, sync backlog | PRs #26-#39 landed with test evidence | Needs capacity-backed Linear rollout / broader evaluator corpus || GitGuardian/Dependabot/CodeRabbit-style checks | Non-blocking taxonomy and deterministic follow-up checks | ECC-Tools risk taxonomy check plus follow-up signals landed, including Skill Quality, Deep Analyzer Evidence, Analyzer Corpus Evidence, RAG/Evaluator Evidence, and PR Review/Salvage Evidence | Partially complete |-| Harness-agnostic learning system | Audit, adapter matrix, observability, traces, promotion loop | Audit/adapters/observability gates plus `docs/architecture/evaluator-rag-prototype.md` and `examples/evaluator-rag-prototype/` define read-only stale-salvage, billing-readiness, and CI-failure-diagnosis scenarios with trace, report, playbook, and verifier result artifacts | Needs broader evaluator corpus |+| Harness-agnostic learning system | Audit, adapter matrix, observability, traces, promotion loop | Audit/adapters/observability gates plus `docs/architecture/evaluator-rag-prototype.md` and `examples/evaluator-rag-prototype/` define read-only stale-salvage, billing-readiness, CI-failure-diagnosis, and harness-config-quality scenarios with trace, report, playbook, and verifier result artifacts | Needs AgentShield policy exception corpus || Linear roadmap is detailed | Linear project status plus repo mirror | Repo mirror exists; issue creation was retried on 2026-05-12 and remains blocked by the workspace free issue limit | Needs recurring status updates after each merge batch || Flow separation and progress tracking | Flow lanes with owner artifacts and update cadence | This roadmap defines lanes below | Active || Realtime Linear sync | Project updates while issue limit is blocked; issues later | ECC-Tools #39 implements opt-in Linear API sync for deferred follow-up backlog items | Needs workspace capacity/config rollout |@@ -219,7 +219,7 @@ back to the repo evidence and merge commits.| Queue hygiene and salvage | GitHub PR/issue state, salvage ledger | Append ledger entries for any future stale closures | Every cleanup batch || Release and publication | rc.1 release docs, publication readiness doc | Naming matrix and plugin submission/contact checklist | Before any tag || Harness OS core | Audit, adapter matrix, observability docs, `ecc2/` | HUD/session-control acceptance spec | Weekly until GA |-| Evaluation and RAG | Reference-set validation, harness audit, traces | Read-only evaluator/RAG prototype plus stale-salvage and billing-readiness fixtures | Expand to CI, harness-config, and AgentShield scenarios |+| Evaluation and RAG | Reference-set validation, harness audit, traces | Read-only evaluator/RAG prototype plus stale-salvage, billing-readiness, CI-failure-diagnosis, and harness-config-quality fixtures | Expand to AgentShield policy exception scenario || AgentShield enterprise | AgentShield PR evidence and roadmap notes | PDF-export decision or next enterprise signal | After value decision || ECC Tools app | ECC-Tools PR evidence, billing audit, risk taxonomy | Capacity-backed Linear rollout or broader evaluator/RAG corpus slice | Next implementation batch || Linear progress | Linear project status updates and this mirror | Status update with queue/evidence/missing gates | Every significant merge batch |diff --git a/docs/architecture/evaluator-rag-prototype.md b/docs/architecture/evaluator-rag-prototype.mdindex 60bb7a329..feba9d0f0 100644--- a/docs/architecture/evaluator-rag-prototype.md+++ b/docs/architecture/evaluator-rag-prototype.md@@ -12,7 +12,9 @@ lane has real inputs, real accepted work, and real rejected work. The corpus nowalso includes a billing/Marketplace readiness scenario so launch copy cannottreat dry-run release evidence or roadmap intent as live billing state. ACI-failure diagnosis scenario adds the log-first workflow needed before an-agent proposes fixes for red checks.+agent proposes fixes for red checks. A harness-config quality scenario keeps+MCP, plugin, hook, command, agent, and adapter recommendations tied to the+adapter matrix before they mutate setup guidance.## Reference Pressure@@ -100,6 +102,9 @@ Current corpus:launch claims before public copy says they are live.- `ci-failure-diagnosis`: requires failed-job logs, changed-file scope, and anamed regression command before a CI fix playbook can be promoted.+- `harness-config-quality`: requires adapter state, install/onramp path,+ verification commands, risk notes, and config-preservation behavior before a+ harness setup recommendation can be promoted.## ECC Tools Mapping@@ -133,5 +138,4 @@ A candidate can be promoted only when:The next evaluator/RAG corpus should add:-- a harness-config quality scenario covering MCP/plugin/hook drift;- an AgentShield policy exception scenario with SARIF and report evidence.diff --git a/examples/evaluator-rag-prototype/harness-config-quality/candidate-playbook.md b/examples/evaluator-rag-prototype/harness-config-quality/candidate-playbook.mdnew file mode 100644index 000000000..3065e360e--- /dev/null+++ b/examples/evaluator-rag-prototype/harness-config-quality/candidate-playbook.md@@ -0,0 +1,49 @@+# Harness Config Quality Playbook++Candidate id: `adapter-matrix-backed-drift-check`++Use this playbook when a PR, install change, or setup recommendation touches+MCP, plugins, hooks, commands, agents, rules, install targets, or harness+adapter surfaces.++## Accepted Path++1. Identify the touched harness/config surface.+2. Retrieve the adapter state from+ `docs/architecture/harness-adapter-compliance.md` or+ `scripts/lib/harness-adapter-compliance.js`.+3. Record whether the harness is `Native`, `Adapter-backed`,+ `Instruction-backed`, or `Reference-only`.+4. Name the install/onramp path and verification command from the matrix.+5. Preserve existing user and project config by using merge, dry-run, or+ explicit no-overwrite behavior.+6. Run the relevant validation gate:+ - `npm run harness:adapters -- --check`+ - `npm run harness:audit -- --format json`+ - `node tests/lib/install-targets.test.js`+ - `node tests/opencode-plugin-hooks.test.js`+ - `node tests/docs/mcp-management-docs.test.js`+7. Promote a config recommendation only when the evidence matches the harness+ state and the config preservation behavior is explicit.++## Rejected Path++Do not claim Claude hook parity for Codex, Gemini, Zed, OpenCode, or other+harnesses unless the adapter matrix and tests prove it.++Do not overwrite `settings.json`, MCP configs, plugin manifests, rule files, or+command surfaces without a merge/dry-run path and a rollback note.++Do not toggle live MCP servers, publish plugins, or edit user-level harness+config from the evaluator run.++## Minimum Validation++- `npm run harness:adapters -- --check`+- `npm run harness:audit -- --format json`+- Focused install, plugin, MCP, or hook test for the changed surface+- `git diff --check`+- Markdown lint when docs are touched++Record the adapter state, risk note, validation commands, and config+preservation behavior in the maintainer PR body or handoff.diff --git a/examples/evaluator-rag-prototype/harness-config-quality/report.json b/examples/evaluator-rag-prototype/harness-config-quality/report.jsonnew file mode 100644index 000000000..8d50f4942--- /dev/null+++ b/examples/evaluator-rag-prototype/harness-config-quality/report.json@@ -0,0 +1,35 @@+{+ "schema_version": "ecc.evaluator-rag.report.v1",+ "scenario_id": "harness-config-quality",+ "run_id": "2026-05-12-harness-config-quality-prototype",+ "result": "prototype_passed",+ "read_only": true,+ "scores": {+ "adapter_evidence": 0.94,+ "config_preservation": 0.88,+ "verification_specificity": 0.9,+ "parity_claim_safety": 1,+ "publication_safety": 1+ },+ "findings": [+ {+ "id": "adapter-state-required",+ "severity": "warning",+ "summary": "Harness recommendations must retrieve the adapter state before claiming native support or runtime enforcement."+ },+ {+ "id": "config-overwrite-risk",+ "severity": "warning",+ "summary": "MCP, hook, plugin, command, and rule changes must preserve existing user/project config and use dry-run or merge behavior when available."+ },+ {+ "id": "verification-command-needed",+ "severity": "info",+ "summary": "The accepted playbook names harness adapter, harness audit, install-target, or plugin-hook regression gates before a config change can merge."+ }+ ],+ "recommended_next_action": {+ "candidate_id": "adapter-matrix-backed-drift-check",+ "action": "Use the promoted harness-config quality playbook for PRs or setup work touching MCP, plugin, hook, command, agent, rule, or adapter surfaces."+ }+}diff --git a/examples/evaluator-rag-prototype/harness-config-quality/scenario.json b/examples/evaluator-rag-prototype/harness-config-quality/scenario.jsonnew file mode 100644index 000000000..90dccb946--- /dev/null+++ b/examples/evaluator-rag-prototype/harness-config-quality/scenario.json@@ -0,0 +1,57 @@+{+ "schema_version": "ecc.evaluator-rag.scenario.v1",+ "scenario_id": "harness-config-quality",+ "title": "Detect harness config drift before changing adapters or installs",+ "mode": "read_only_prototype",+ "objective": "Given a change to MCP, plugin, hook, command, agent, or harness adapter surfaces, retrieve the adapter matrix and validation evidence before promoting a setup recommendation or config change.",+ "sources": [+ {+ "kind": "repo_doc",+ "path": "docs/architecture/harness-adapter-compliance.md",+ "purpose": "Public adapter matrix that names harness state, install/onramp paths, verification commands, and risk notes"+ },+ {+ "kind": "repo_source",+ "path": "scripts/lib/harness-adapter-compliance.js",+ "purpose": "Structured source of truth for the adapter compliance matrix"+ },+ {+ "kind": "repo_config",+ "path": "hooks/hooks.json",+ "purpose": "Claude hook surface that must not be assumed portable without adapter evidence"+ },+ {+ "kind": "repo_config",+ "path": "mcp-configs/mcp-servers.json",+ "purpose": "Reference MCP config that can drift from harness-specific runtime semantics"+ },+ {+ "kind": "repo_test",+ "command": "npm run harness:adapters -- --check",+ "purpose": "Adapter matrix consistency gate"+ }+ ],+ "retrieval_questions": [+ "Which harness or config surface changed: MCP, plugin, hook, command, agent, rule, or adapter?",+ "Does the adapter matrix classify this harness as native, adapter-backed, instruction-backed, or reference-only?",+ "Which install path, verification command, risk note, owner, and source doc apply?",+ "Does the recommendation preserve existing user config rather than overwriting it?",+ "Which compatibility regression or harness audit command proves the setup still works?"+ ],+ "forbidden_actions": [+ "claiming native support for instruction-backed or reference-only harnesses",+ "copying Claude hook semantics into Codex, Gemini, Zed, or OpenCode without adapter evidence",+ "silently overwriting existing user MCP, hook, plugin, command, or rule config",+ "disabling or enabling live MCP servers from a read-only evaluator run",+ "shipping an adapter change without a verification command",+ "publishing packages or plugins from this evaluator run"+ ],+ "acceptance_gates": [+ "adapter state is retrieved from the matrix",+ "install or onramp path is named",+ "verification command is named",+ "risk note is preserved",+ "config-preservation behavior is explicit",+ "at least one unsupported parity claim is rejected"+ ]+}diff --git a/examples/evaluator-rag-prototype/harness-config-quality/trace.json b/examples/evaluator-rag-prototype/harness-config-quality/trace.jsonnew file mode 100644index 000000000..3a7c29b40--- /dev/null+++ b/examples/evaluator-rag-prototype/harness-config-quality/trace.json@@ -0,0 +1,45 @@+{+ "schema_version": "ecc.evaluator-rag.trace.v1",+ "scenario_id": "harness-config-quality",+ "run_id": "2026-05-12-harness-config-quality-prototype",+ "read_only": true,+ "events": [+ {+ "phase": "observation",+ "summary": "A setup recommendation or PR touches MCP, plugin, hook, command, agent, rule, or adapter surfaces. The evaluator records the surface without editing local or user-level config.",+ "evidence": [+ "docs/architecture/harness-adapter-compliance.md",+ "scripts/lib/harness-adapter-compliance.js"+ ]+ },+ {+ "phase": "retrieval",+ "summary": "Retrieved the adapter state, install/onramp path, verification commands, risk notes, and config-preservation tests for the affected harness.",+ "evidence": [+ "npm run harness:adapters -- --check",+ "npm run harness:audit -- --format json",+ "node tests/lib/install-targets.test.js"+ ]+ },+ {+ "phase": "proposal",+ "summary": "Generated two candidate playbooks: adapter-matrix-backed drift check, and unsupported hook parity claim that copies Claude semantics into every harness.",+ "candidate_ids": [+ "adapter-matrix-backed-drift-check",+ "unsupported-hook-parity-claim"+ ]+ },+ {+ "phase": "verification",+ "summary": "Accepted the matrix-backed drift check because it names state, install path, verification, and preservation behavior. Rejected unsupported hook parity because it overclaims portability.",+ "evidence": [+ "examples/evaluator-rag-prototype/harness-config-quality/verifier-result.json"+ ]+ },+ {+ "phase": "promotion",+ "summary": "Promoted only the read-only harness-config quality playbook. The evaluator does not overwrite configs, toggle MCP servers, publish plugins, or claim native support.",+ "promoted_candidate_id": "adapter-matrix-backed-drift-check"+ }+ ]+}diff --git a/examples/evaluator-rag-prototype/harness-config-quality/verifier-result.json b/examples/evaluator-rag-prototype/harness-config-quality/verifier-result.jsonnew file mode 100644index 000000000..d0c78391d--- /dev/null+++ b/examples/evaluator-rag-prototype/harness-config-quality/verifier-result.json@@ -0,0 +1,35 @@+{+ "schema_version": "ecc.evaluator-rag.verifier.v1",+ "scenario_id": "harness-config-quality",+ "run_id": "2026-05-12-harness-config-quality-prototype",+ "read_only": true,+ "candidates": [+ {+ "candidate_id": "adapter-matrix-backed-drift-check",+ "decision": "accepted",+ "score": 0.92,+ "reasons": [+ "retrieves adapter state before making a support claim",+ "names install or onramp path and verification commands",+ "preserves existing user and project config",+ "keeps runtime MCP toggles and plugin publication out of the evaluator run",+ "requires focused compatibility regression coverage"+ ],+ "rollback": "Revert the future adapter/config PR or restore the prior config merge behavior; no live user config is changed by this read-only playbook."+ },+ {+ "candidate_id": "unsupported-hook-parity-claim",+ "decision": "rejected",+ "score": 0.16,+ "reasons": [+ "claims native support without adapter matrix evidence",+ "copies Claude hook semantics into instruction-backed harnesses",+ "does not name a verification command",+ "does not preserve existing MCP or hook config",+ "risks publishing or installing unsupported plugin behavior"+ ],+ "rollback": "Do not publish this setup recommendation; restart from adapter state, risk note, and config-preservation evidence."+ }+ ],+ "promoted_candidate_id": "adapter-matrix-backed-drift-check"+}diff --git a/tests/docs/evaluator-rag-prototype.test.js b/tests/docs/evaluator-rag-prototype.test.jsindex 7c8531313..9714d10fb 100644--- a/tests/docs/evaluator-rag-prototype.test.js+++ b/tests/docs/evaluator-rag-prototype.test.js@@ -135,7 +135,7 @@ test('roadmap points to the evaluator RAG prototype and keeps broader corpus worassert.ok(roadmap.includes('docs/architecture/evaluator-rag-prototype.md'));assert.ok(roadmap.includes('examples/evaluator-rag-prototype/'));- assert.ok(roadmap.includes('Needs broader evaluator corpus'));+ assert.ok(roadmap.includes('Needs AgentShield policy exception corpus'));});test('billing readiness scenario rejects launch copy overclaims', () => {@@ -221,6 +221,52 @@ test('ci failure diagnosis scenario rejects rerun-only fixes', () => {assert.ok(playbook.includes('Full required GitHub Actions matrix before merge'));});+test('harness config quality scenario rejects unsupported parity claims', () => {+ const scenario = readFixtureJson('harness-config-quality/scenario.json');+ const trace = readFixtureJson('harness-config-quality/trace.json');+ const report = readFixtureJson('harness-config-quality/report.json');+ const verifier = readFixtureJson('harness-config-quality/verifier-result.json');+ const playbook = read('examples/evaluator-rag-prototype/harness-config-quality/candidate-playbook.md');++ assert.strictEqual(scenario.scenario_id, 'harness-config-quality');+ assert.strictEqual(trace.scenario_id, scenario.scenario_id);+ assert.strictEqual(report.scenario_id, scenario.scenario_id);+ assert.strictEqual(verifier.scenario_id, scenario.scenario_id);+ assert.strictEqual(trace.read_only, true);+ assert.strictEqual(report.read_only, true);+ assert.strictEqual(verifier.read_only, true);++ for (const blocked of [+ 'claiming native support for instruction-backed or reference-only harnesses',+ 'copying Claude hook semantics into Codex, Gemini, Zed, or OpenCode without adapter evidence',+ 'silently overwriting existing user MCP, hook, plugin, command, or rule config',+ 'publishing packages or plugins from this evaluator run'+ ]) {+ assert.ok(scenario.forbidden_actions.includes(blocked), `Missing harness forbidden action: ${blocked}`);+ }++ for (const required of [+ 'adapter state is retrieved from the matrix',+ 'install or onramp path is named',+ 'verification command is named',+ 'config-preservation behavior is explicit'+ ]) {+ assert.ok(scenario.acceptance_gates.includes(required), `Missing harness acceptance gate: ${required}`);+ }++ const accepted = verifier.candidates.find(candidate => candidate.candidate_id === 'adapter-matrix-backed-drift-check');+ const rejected = verifier.candidates.find(candidate => candidate.candidate_id === 'unsupported-hook-parity-claim');++ assert.ok(accepted, 'Missing accepted adapter-matrix candidate');+ assert.ok(rejected, 'Missing rejected unsupported parity candidate');+ assert.strictEqual(accepted.decision, 'accepted');+ assert.strictEqual(rejected.decision, 'rejected');+ assert.strictEqual(verifier.promoted_candidate_id, accepted.candidate_id);+ assert.ok(rejected.reasons.join('\n').includes('native support'));+ assert.ok(playbook.includes('npm run harness:adapters -- --check'));+ assert.ok(playbook.includes('node tests/docs/mcp-management-docs.test.js'));+});+if (failed > 0) {console.log(`\nFailed: ${failed}`);process.exit(1);