The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.
docs: add deep-analyzer evaluator scenario
Affaan Mustafa37c27a60fdaf
diff --git a/docs/ECC-2.0-GA-ROADMAP.md b/docs/ECC-2.0-GA-ROADMAP.mdindex 97954c341..899d652c9 100644--- a/docs/ECC-2.0-GA-ROADMAP.md+++ b/docs/ECC-2.0-GA-ROADMAP.md@@ -201,7 +201,7 @@ is not complete unless the evidence column exists and has been freshly verified.| AgentShield enterprise iteration | Policy gates, SARIF, packs, provenance, corpus, HTML reports, exception lifecycle audit | PRs #53, #55-#62 landed with test evidence | Needs PDF/export decision or next enterprise signal || ECC Tools next-level app | Billing audit, PR checks, deep analyzer, sync backlog | PRs #26-#39 landed with test evidence | Needs capacity-backed Linear rollout / broader evaluator corpus || GitGuardian/Dependabot/CodeRabbit-style checks | Non-blocking taxonomy and deterministic follow-up checks | ECC-Tools risk taxonomy check plus follow-up signals landed, including Skill Quality, Deep Analyzer Evidence, Analyzer Corpus Evidence, RAG/Evaluator Evidence, and PR Review/Salvage Evidence | Partially complete |-| Harness-agnostic learning system | Audit, adapter matrix, observability, traces, promotion loop | Audit/adapters/observability gates plus `docs/architecture/evaluator-rag-prototype.md` and `examples/evaluator-rag-prototype/` define read-only stale-salvage, billing-readiness, CI-failure-diagnosis, harness-config-quality, AgentShield policy-exception, and skill-quality evidence scenarios with trace, report, playbook, and verifier result artifacts | Needs deep-analyzer corpus |+| Harness-agnostic learning system | Audit, adapter matrix, observability, traces, promotion loop | Audit/adapters/observability gates plus `docs/architecture/evaluator-rag-prototype.md` and `examples/evaluator-rag-prototype/` define read-only stale-salvage, billing-readiness, CI-failure-diagnosis, harness-config-quality, AgentShield policy-exception, skill-quality evidence, and deep-analyzer evidence scenarios with trace, report, playbook, and verifier result artifacts | Local corpus complete; hosted integration remains future || Linear roadmap is detailed | Linear project status plus repo mirror | Repo mirror exists; issue creation was retried on 2026-05-12 and remains blocked by the workspace free issue limit | Needs recurring status updates after each merge batch || Flow separation and progress tracking | Flow lanes with owner artifacts and update cadence | This roadmap defines lanes below | Active || Realtime Linear sync | Project updates while issue limit is blocked; issues later | ECC-Tools #39 implements opt-in Linear API sync for deferred follow-up backlog items | Needs workspace capacity/config rollout |@@ -220,7 +220,7 @@ back to the repo evidence and merge commits.| Queue hygiene and salvage | GitHub PR/issue state, salvage ledger | Append ledger entries for any future stale closures | Every cleanup batch || Release and publication | rc.1 release docs, publication readiness doc | Naming matrix and plugin submission/contact checklist | Before any tag || Harness OS core | Audit, adapter matrix, observability docs, `ecc2/` | HUD/session-control acceptance spec | Weekly until GA |-| Evaluation and RAG | Reference-set validation, harness audit, traces | Read-only evaluator/RAG prototype plus stale-salvage, billing-readiness, CI-failure-diagnosis, harness-config-quality, AgentShield policy-exception, and skill-quality evidence fixtures | Expand to deep-analyzer evidence scenario |+| Evaluation and RAG | Reference-set validation, harness audit, traces | Read-only evaluator/RAG prototype plus stale-salvage, billing-readiness, CI-failure-diagnosis, harness-config-quality, AgentShield policy-exception, skill-quality evidence, and deep-analyzer evidence fixtures | Use as fixture contract before hosted retrieval/check-run automation || AgentShield enterprise | AgentShield PR evidence and roadmap notes | PDF-export decision or next enterprise signal | After value decision || ECC Tools app | ECC-Tools PR evidence, billing audit, risk taxonomy | Capacity-backed Linear rollout or broader evaluator/RAG corpus slice | Next implementation batch || Linear progress | Linear project status updates and this mirror | Status update with queue/evidence/missing gates | Every significant merge batch |@@ -419,6 +419,6 @@ Acceptance:executive report, corpus benchmark output, and exception lifecycle audit.2. Enable/configure the merged Linear backlog sync path after workspace issuecapacity clears or the Linear workspace is upgraded.-3. Expand the evaluator/RAG corpus beyond stale-salvage, billing, CI,- harness-config, AgentShield policy-exception, and skill-quality evidence- prototypes toward deep-analyzer evidence scenarios.+3. Consume the local evaluator/RAG corpus from ECC Tools before adding hosted+ retrieval, vector storage, model-backed judging, or automated check-run+ promotion.diff --git a/docs/architecture/evaluator-rag-prototype.md b/docs/architecture/evaluator-rag-prototype.mdindex 48b684cc8..4543e578b 100644--- a/docs/architecture/evaluator-rag-prototype.md+++ b/docs/architecture/evaluator-rag-prototype.md@@ -19,7 +19,9 @@ exception scenario gates security exceptions on SARIF/report evidence, ownerfields, expiry state, and remediation-versus-exception decisions. Askill-quality evidence scenario requires observed failure or feedback evidence,working examples, reference-set gaps, and validation commands before a skill-amendment can be promoted.+amendment can be promoted. A deep-analyzer evidence scenario requires analyzer+corpus cases, expected-output comparisons, and risk-taxonomy proof before+repository or commit-analysis behavior can change.## Reference Pressure@@ -116,6 +118,9 @@ Current corpus:- `skill-quality-evidence`: requires focused skill scope, observed failure oruser-feedback evidence, examples/reference-set coverage, validation commands,and publication safety before a skill amendment can be promoted.+- `deep-analyzer-evidence`: requires maintained analyzer corpus cases,+ expected-output comparisons, representative repository/commit histories, and+ regression commands before deep-analysis behavior can be promoted.## ECC Tools Mapping@@ -147,7 +152,7 @@ A candidate can be promoted only when:## Next Expansion-The next evaluator/RAG corpus should add:--- a deep-analyzer evidence scenario with maintained reference sets and rejected- low-evidence candidates.+The local evaluator/RAG corpus now covers the current evidence buckets. Future+work should consume these fixtures from ECC Tools before adding hosted+retrieval, vector storage, model-backed judging, or automated check-run+promotion.diff --git a/examples/evaluator-rag-prototype/deep-analyzer-evidence/candidate-playbook.md b/examples/evaluator-rag-prototype/deep-analyzer-evidence/candidate-playbook.mdnew file mode 100644index 000000000..86eaf4999--- /dev/null+++ b/examples/evaluator-rag-prototype/deep-analyzer-evidence/candidate-playbook.md@@ -0,0 +1,60 @@+# Deep Analyzer Evidence Playbook++Candidate id: `corpus-backed-analyzer-change`++Use this playbook when a PR changes repository analysis, commit analysis,+architecture classification, workflow detection, pattern detection, or+deep-analysis risk-taxonomy behavior.++## Accepted Path++1. Name the changed analyzer surface and source file.+2. Retrieve the Deep Analyzer Evidence contract from `../ECC-Tools/README.md`+ and the follow-up logic in `../ECC-Tools/src/lib/analyzer.ts`.+3. Match the change to maintained corpus or reference evidence:+ - `../ECC-Tools/src/analyzers/fixtures/deep-analyzer-corpus.ts`+ - `../ECC-Tools/src/analyzers/deep-analyzer-corpus.test.ts`+ - `../ECC-Tools/src/lib/analyzer.compare.test.ts`+4. Compare expected outputs for the affected behavior:+ - folder type;+ - module organization;+ - test location;+ - primary language;+ - commit message type;+ - detected workflow names.+5. Add or update analyzer corpus, expected-output snapshots, fixtures,+ benchmarks, golden cases, evals, or reference sets for the same changed+ surface.+6. Run the relevant validation gate from `../ECC-Tools/`:+ - `npm test -- src/analyzers/deep-analyzer-corpus.test.ts src/lib/analyzer.compare.test.ts`+ - `npm run typecheck`+ - `npm run lint`+7. Record the corpus case, expected-output comparison, validation output, and+ rollback notes in the maintainer PR body or handoff.++## Rejected Path++Do not promote analyzer threshold, classification, or risk-taxonomy changes+without corpus, snapshot, fixture, benchmark, golden, eval, or reference-set+evidence.++Do not suppress the `Deep Analyzer Evidence` PR-risk bucket just because the+change is small. Suppress it only when co-located evidence covers the same+analyzer surface.++Do not rely only on broad manual review notes. Analyzer changes need+representative repository shapes or commit-history cases with expected outputs.++Do not post PR comments, create check runs, sync Linear, publish packages, edit+plugins, or create release artifacts from the evaluator run.++## Minimum Validation++- `npm test -- src/analyzers/deep-analyzer-corpus.test.ts src/lib/analyzer.compare.test.ts`+- `npm run typecheck`+- `npm run lint`+- `git diff --check`+- Markdown lint when docs or playbooks are touched++Preserve source attribution for analyzer evidence and include rollback guidance+for the future maintainer PR.diff --git a/examples/evaluator-rag-prototype/deep-analyzer-evidence/report.json b/examples/evaluator-rag-prototype/deep-analyzer-evidence/report.jsonnew file mode 100644index 000000000..ee373ebcf--- /dev/null+++ b/examples/evaluator-rag-prototype/deep-analyzer-evidence/report.json@@ -0,0 +1,35 @@+{+ "schema_version": "ecc.evaluator-rag.report.v1",+ "scenario_id": "deep-analyzer-evidence",+ "run_id": "2026-05-12-deep-analyzer-evidence-prototype",+ "result": "prototype_passed",+ "read_only": true,+ "scores": {+ "corpus_retrieval": 0.95,+ "expected_output_comparison": 0.91,+ "representative_case_coverage": 0.89,+ "taxonomy_gap_safety": 0.93,+ "publication_safety": 1+ },+ "findings": [+ {+ "id": "corpus-required",+ "severity": "warning",+ "summary": "Deep-analysis behavior changes need maintained corpus, snapshot, fixture, benchmark, golden, eval, or reference-set evidence before promotion."+ },+ {+ "id": "expected-output-required",+ "severity": "warning",+ "summary": "Analyzer changes should compare expected folder type, module organization, test location, primary language, commit pattern, or workflow outputs."+ },+ {+ "id": "read-only-routing",+ "severity": "info",+ "summary": "The evaluator can recommend a maintainer PR but cannot post PR comments, check runs, Linear sync updates, packages, plugins, or release actions itself."+ }+ ],+ "recommended_next_action": {+ "candidate_id": "corpus-backed-analyzer-change",+ "action": "Use the promoted deep-analyzer evidence playbook for PRs that change repository, commit, architecture, workflow, pattern, or risk-taxonomy analysis behavior."+ }+}diff --git a/examples/evaluator-rag-prototype/deep-analyzer-evidence/scenario.json b/examples/evaluator-rag-prototype/deep-analyzer-evidence/scenario.jsonnew file mode 100644index 000000000..e2fb06a25--- /dev/null+++ b/examples/evaluator-rag-prototype/deep-analyzer-evidence/scenario.json@@ -0,0 +1,57 @@+{+ "schema_version": "ecc.evaluator-rag.scenario.v1",+ "scenario_id": "deep-analyzer-evidence",+ "title": "Require analyzer corpus evidence before promoting deep-analysis changes",+ "mode": "read_only_prototype",+ "objective": "Given a change to repository, commit, architecture, pattern, or deep-analysis logic, retrieve maintained analyzer corpus evidence and expected-output comparisons before promoting analyzer behavior or risk-taxonomy changes.",+ "sources": [+ {+ "kind": "sibling_repo_doc",+ "path": "../ECC-Tools/README.md",+ "purpose": "Public description of deep-analyzer predictive follow-ups and the Deep Analyzer Evidence PR-risk bucket"+ },+ {+ "kind": "sibling_repo_source",+ "path": "../ECC-Tools/src/lib/analyzer.ts",+ "purpose": "Predictive follow-up logic that flags analyzer changes without corpus, snapshot, fixture, or benchmark evidence"+ },+ {+ "kind": "sibling_repo_source",+ "path": "../ECC-Tools/src/lib/pr-risk-taxonomy.ts",+ "purpose": "Non-blocking PR-risk taxonomy bucket for deep-analyzer evidence"+ },+ {+ "kind": "sibling_repo_fixture",+ "path": "../ECC-Tools/src/analyzers/fixtures/deep-analyzer-corpus.ts",+ "purpose": "Maintained corpus cases for representative repository shapes, commit histories, and expected analyzer outputs"+ },+ {+ "kind": "sibling_repo_test",+ "command": "npm test -- src/analyzers/deep-analyzer-corpus.test.ts src/lib/analyzer.compare.test.ts",+ "purpose": "Regression evidence for analyzer corpus outputs and deep-analyzer follow-up generation"+ }+ ],+ "retrieval_questions": [+ "Which analyzer surface changed: repository structure, architecture, code style, commit messages, workflow detection, pattern detection, or risk taxonomy?",+ "Which maintained corpus case or reference set covers the same analyzer behavior?",+ "Do expected outputs compare folder type, module organization, test location, primary language, commit type, and workflow names?",+ "Does the PR add analyzer corpus, snapshot, fixture, benchmark, golden, eval, or reference-set evidence alongside analyzer code changes?",+ "Does the evaluator keep PR comments, check runs, Linear sync, package changes, and publication actions out of the read-only pass?"+ ],+ "forbidden_actions": [+ "promoting repository, commit, architecture, or deep-analysis changes without analyzer corpus evidence",+ "suppressing the Deep Analyzer Evidence risk bucket without co-located corpus, snapshot, fixture, or benchmark evidence",+ "changing analyzer thresholds or classifications without expected-output comparison",+ "relying only on broad manual review notes instead of representative repository and commit-history cases",+ "posting PR comments, check runs, or Linear sync updates from this read-only evaluator run",+ "changing package, plugin, release, or publication state from this evaluator run"+ ],+ "acceptance_gates": [+ "changed analyzer surface is named",+ "maintained corpus or reference-set path is included",+ "expected analyzer outputs are compared",+ "representative repository shape or commit history is described",+ "regression command is named",+ "at least one no-corpus analyzer change is rejected"+ ]+}diff --git a/examples/evaluator-rag-prototype/deep-analyzer-evidence/trace.json b/examples/evaluator-rag-prototype/deep-analyzer-evidence/trace.jsonnew file mode 100644index 000000000..d64117860--- /dev/null+++ b/examples/evaluator-rag-prototype/deep-analyzer-evidence/trace.json@@ -0,0 +1,45 @@+{+ "schema_version": "ecc.evaluator-rag.trace.v1",+ "scenario_id": "deep-analyzer-evidence",+ "run_id": "2026-05-12-deep-analyzer-evidence-prototype",+ "read_only": true,+ "events": [+ {+ "phase": "observation",+ "summary": "A deep-analysis PR changes repository, commit, architecture, workflow, pattern, or risk-taxonomy behavior. The evaluator records the touched analyzer surface and remains read-only.",+ "evidence": [+ "../ECC-Tools/src/lib/analyzer.ts",+ "../ECC-Tools/src/lib/pr-risk-taxonomy.ts"+ ]+ },+ {+ "phase": "retrieval",+ "summary": "Retrieved the maintained analyzer corpus, corpus regression test, and follow-up tests that distinguish corpus-backed analyzer changes from no-evidence analyzer rewrites.",+ "evidence": [+ "../ECC-Tools/src/analyzers/fixtures/deep-analyzer-corpus.ts",+ "../ECC-Tools/src/analyzers/deep-analyzer-corpus.test.ts",+ "../ECC-Tools/src/lib/analyzer.compare.test.ts"+ ]+ },+ {+ "phase": "proposal",+ "summary": "Generated two candidate playbooks: corpus-backed analyzer change, and threshold-only analyzer rewrite without expected-output evidence.",+ "candidate_ids": [+ "corpus-backed-analyzer-change",+ "threshold-only-analyzer-rewrite"+ ]+ },+ {+ "phase": "verification",+ "summary": "Accepted the corpus-backed analyzer change because it names representative repository/commit cases and expected-output comparisons. Rejected the threshold-only rewrite because it lacks corpus or benchmark evidence.",+ "evidence": [+ "examples/evaluator-rag-prototype/deep-analyzer-evidence/verifier-result.json"+ ]+ },+ {+ "phase": "promotion",+ "summary": "Promoted only the read-only deep-analyzer evidence playbook. Future analyzer edits must move through maintainer PRs with corpus evidence, regression commands, and rollback notes.",+ "promoted_candidate_id": "corpus-backed-analyzer-change"+ }+ ]+}diff --git a/examples/evaluator-rag-prototype/deep-analyzer-evidence/verifier-result.json b/examples/evaluator-rag-prototype/deep-analyzer-evidence/verifier-result.jsonnew file mode 100644index 000000000..ef3d7d83b--- /dev/null+++ b/examples/evaluator-rag-prototype/deep-analyzer-evidence/verifier-result.json@@ -0,0 +1,35 @@+{+ "schema_version": "ecc.evaluator-rag.verifier.v1",+ "scenario_id": "deep-analyzer-evidence",+ "run_id": "2026-05-12-deep-analyzer-evidence-prototype",+ "read_only": true,+ "candidates": [+ {+ "candidate_id": "corpus-backed-analyzer-change",+ "decision": "accepted",+ "score": 0.92,+ "reasons": [+ "names the changed analyzer surface and matching maintained corpus case",+ "compares expected analyzer outputs for representative repository and commit-history inputs",+ "keeps Deep Analyzer Evidence taxonomy behavior tied to co-located corpus or benchmark evidence",+ "names the regression command that exercises corpus and follow-up behavior",+ "keeps PR comments, check runs, Linear sync, and publication actions out of the evaluator run"+ ],+ "rollback": "Revert the future analyzer PR and restore the prior corpus expectations; no hosted check-run, Linear, package, or publication state changes in this read-only playbook."+ },+ {+ "candidate_id": "threshold-only-analyzer-rewrite",+ "decision": "rejected",+ "score": 0.13,+ "reasons": [+ "changes analyzer thresholds without corpus evidence",+ "does not compare expected outputs against representative repository or commit-history cases",+ "does not update analyzer corpus, snapshot, fixture, benchmark, golden, eval, or reference-set artifacts",+ "would suppress Deep Analyzer Evidence risk without proof",+ "does not name a regression command"+ ],+ "rollback": "Do not promote this analyzer rewrite; restart from maintained corpus inputs, expected-output snapshots, and a focused maintainer PR."+ }+ ],+ "promoted_candidate_id": "corpus-backed-analyzer-change"+}diff --git a/tests/docs/evaluator-rag-prototype.test.js b/tests/docs/evaluator-rag-prototype.test.jsindex 4655990cb..52459379f 100644--- a/tests/docs/evaluator-rag-prototype.test.js+++ b/tests/docs/evaluator-rag-prototype.test.js@@ -130,12 +130,12 @@ test('candidate playbook preserves stale-salvage operating rules', () => {}});-test('roadmap points to the evaluator RAG prototype and keeps broader corpus work open', () => {+test('roadmap points to the evaluator RAG prototype and keeps hosted integration open', () => {const roadmap = read('docs/ECC-2.0-GA-ROADMAP.md');assert.ok(roadmap.includes('docs/architecture/evaluator-rag-prototype.md'));assert.ok(roadmap.includes('examples/evaluator-rag-prototype/'));- assert.ok(roadmap.includes('Needs deep-analyzer corpus'));+ assert.ok(roadmap.includes('Local corpus complete; hosted integration remains future'));});test('billing readiness scenario rejects launch copy overclaims', () => {@@ -361,6 +361,54 @@ test('skill quality evidence scenario rejects vague rewrites', () => {assert.ok(playbook.includes('observed skill-run failure'));});+test('deep analyzer evidence scenario rejects no-corpus analyzer changes', () => {+ const scenario = readFixtureJson('deep-analyzer-evidence/scenario.json');+ const trace = readFixtureJson('deep-analyzer-evidence/trace.json');+ const report = readFixtureJson('deep-analyzer-evidence/report.json');+ const verifier = readFixtureJson('deep-analyzer-evidence/verifier-result.json');+ const playbook = read('examples/evaluator-rag-prototype/deep-analyzer-evidence/candidate-playbook.md');++ assert.strictEqual(scenario.scenario_id, 'deep-analyzer-evidence');+ assert.strictEqual(trace.scenario_id, scenario.scenario_id);+ assert.strictEqual(report.scenario_id, scenario.scenario_id);+ assert.strictEqual(verifier.scenario_id, scenario.scenario_id);+ assert.strictEqual(trace.read_only, true);+ assert.strictEqual(report.read_only, true);+ assert.strictEqual(verifier.read_only, true);++ for (const blocked of [+ 'promoting repository, commit, architecture, or deep-analysis changes without analyzer corpus evidence',+ 'suppressing the Deep Analyzer Evidence risk bucket without co-located corpus, snapshot, fixture, or benchmark evidence',+ 'changing analyzer thresholds or classifications without expected-output comparison',+ 'posting PR comments, check runs, or Linear sync updates from this read-only evaluator run'+ ]) {+ assert.ok(scenario.forbidden_actions.includes(blocked), `Missing deep-analyzer forbidden action: ${blocked}`);+ }++ for (const required of [+ 'changed analyzer surface is named',+ 'maintained corpus or reference-set path is included',+ 'expected analyzer outputs are compared',+ 'representative repository shape or commit history is described',+ 'regression command is named'+ ]) {+ assert.ok(scenario.acceptance_gates.includes(required), `Missing deep-analyzer acceptance gate: ${required}`);+ }++ const accepted = verifier.candidates.find(candidate => candidate.candidate_id === 'corpus-backed-analyzer-change');+ const rejected = verifier.candidates.find(candidate => candidate.candidate_id === 'threshold-only-analyzer-rewrite');++ assert.ok(accepted, 'Missing accepted deep-analyzer candidate');+ assert.ok(rejected, 'Missing rejected threshold-only analyzer candidate');+ assert.strictEqual(accepted.decision, 'accepted');+ assert.strictEqual(rejected.decision, 'rejected');+ assert.strictEqual(verifier.promoted_candidate_id, accepted.candidate_id);+ assert.ok(rejected.reasons.join('\n').includes('does not compare expected outputs'));+ assert.ok(playbook.includes('../ECC-Tools/src/analyzers/fixtures/deep-analyzer-corpus.ts'));+ assert.ok(playbook.includes('npm test -- src/analyzers/deep-analyzer-corpus.test.ts src/lib/analyzer.compare.test.ts'));+ assert.ok(playbook.includes('Deep Analyzer Evidence'));+});+if (failed > 0) {console.log(`\nFailed: ${failed}`);process.exit(1);