Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
feat(phase-4): de-circularise the DAC gate + detection content truth table (#429)
Beenu Arora9ae871362a40
diff --git a/.github/workflows/validate-detections.yml b/.github/workflows/validate-detections.ymlindex d4370524a..b87c25842 100644--- a/.github/workflows/validate-detections.yml+++ b/.github/workflows/validate-detections.yml@@ -34,6 +34,12 @@ jobs:- name: Verify curated v1.0 coverage manifestrun: python3 scripts/curate_detections.py --check+ - name: Verify detection content truth table (Phase 4 — honest coverage)+ # Fails if the committed docs/detections/truth-table.md drifts from the+ # on-disk reality, so the "executable rules" headline can never quietly+ # diverge from what the engine actually runs.+ run: python3 scripts/detection_truth_table.py --check+- name: Count rules by severityif: always()run: |diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 8e45f254a..648bc552e 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Added+- **Phase 4a — the Detection-as-Code gate is no longer circular.** The reality audit's #1 circular gate: the detection-proposal promote path shelled out to `run_evals.py` **without ever passing the proposed rule body**, so "passed" meant the repo-wide substrate MITRE accuracy didn't move — a value independent of the rule under review. A blind rule (matches nothing) or a noisy rule (matches everything) sailed through its own exam. New `services/api/app/services/detection_eval.py::evaluate_candidate_rule` runs the *candidate `rule_body` itself* through the **real** runtime engine (`rule_engine.execute_rule`) against caller-supplied positive/negative fixtures: it must fire on every positive and stay silent on every negative. New `POST /detection-proposals/{id}/evaluate-rule` stores the verdict under `eval_result["candidate_rule"]`, and `POST /decide` (approve) now **requires** it — a benchmark-only pass is no longer sufficient. `services/api/tests/test_detection_eval.py` (6 tests) is the mutation test that proves the gate rejects a blind rule, rejects a noisy rule, rejects a no-positive-fixture rule, and that the verdict genuinely depends on the rule body. Claim-to-gate matrix: DAC row PARTIAL(circular) → GATED.+- **Phase 4b — detection content truth table (honest coverage).** The README advertised "6000+ imported detection rules", but ~97% live under `_quarantine/` (`enabled: false`) because their upstream query language (SPL / YARA-L / CAR pseudocode) does not execute on the engine. New `scripts/detection_truth_table.py` walks `detections/` and classifies every rule **executable** (fires today) vs **non-executable** (provenance/coverage only), rendering `docs/detections/truth-table.md`. The honest headline: **939 executable** rules (861 native + 77 sigma-imported + 1 community) of 6975 on disk; 5921 quarantined, 115 disabled. `--check` gates the doc in `validate-detections.yml` so the number can never quietly drift from reality, and the README now cites the executable figure for *coverage* (the on-disk figure only describes the imported *library*). Claim-to-gate matrix: 6000+-imported row PARTIAL → GATED (14 GATED / 10 PARTIAL / 3 NO GATE). The LLM-dependent half of Phase 4 (live-agent Tier-1 eval, hallucination/calibration/abstention, model matrix, 150-payload prompt-injection adversarial) is tracked as 4c+ in `docs/audit/PROGRESS.md`.- **Phase 3.4 — cross-store tenant isolation, now proven against live containers.** The offline `isolation.yml` gate proved each read path *constructs* a tenant scope; it could not prove the scope actually isolates. New `.github/workflows/isolation-live.yml` + `tests/isolation/test_live_stores.py` seed tenant A and tenant B in **real containers** (Neo4j, Redis, ClickHouse, Redpanda/Kafka) and assert a read as A returns zero B data. ClickHouse runs the **production** `lake_sql.rewrite_for_tenant` rewriter against a live warehouse (verified: `SELECT … FROM aisoc.raw_events` becomes `… WHERE tenant_id = '<A>'`, so B's rows never return); Neo4j uses the `tenant_id` property filter; Redis uses the `aisoc:t:<tenant>:*` keyspace namespacing; Kafka replays the `graph_ws` per-tenant envelope filter. Every test also asserts the *unscoped* read sees both tenants, so a scoped pass can never be vacuous on an empty store. The isolation registry (`tests/isolation/stores.py`) flips Neo4j/Redis/ClickHouse/Kafka from `container_pending` → `container_gated`, and the claim-to-gate matrix moves "Cross-tenant isolation (Qdrant/Neo4j/Redis/ClickHouse/Kafka)" PARTIAL → GATED (12 GATED / 12 PARTIAL / 3 NO GATE). The heavy-demo-stack items (Playwright real-stack E2E, demo time-to-first-investigation budget) are tracked as non-blocking 3.5+ in `docs/audit/PROGRESS.md`.- **Phase 3.3 — upgrade-safety gate ("people upgrade; nothing tested it").** New `upgrade` job in `integration.yml` (matrix `v7.5.0 → HEAD` and `v7.3.1 → HEAD` against a real Postgres 16): install a prior release's migration set, seed 250 probe rows on that released schema, then apply HEAD's migration set — the actual self-host upgrade path — and assert the seeded rows survive **and** every HEAD migration applied. The signal this adds beyond the existing fresh-apply job is **destructive-migration detection**: a migration that drops or rewrites existing data fails here even though it applies cleanly on an empty database. The `v7.3.1 → HEAD` leg lands 8 incremental migrations on a pre-existing populated schema (verified locally: 250/250 rows survived, 55/55 migrations tracked). Uses `git checkout <ref> -- services/api/migrations` with an `rm -rf` first so the released set is exact (a `checkout … -- path` alone leaks HEAD-only files and silently degrades to a fresh apply).- **Phase 3.2 — Postgres-outage chaos gate (fail-soft + self-heal).** Extends the `integration.yml` spine job with a second chaos scenario beyond the consumer-kill test: kill Postgres mid-stream, assert the fusion consumer keeps running (the `AlertSink` degrades to "fused alerts still stream over Kafka/WS, persistence dropped for the outage window" instead of crashing the worker), then restart Postgres and assert a *fresh* event persists — proving the sink's asyncpg pool and the API's SQLAlchemy pool both self-heal with no service restart. Distinct event titles side-step fusion's in-memory dedup so the post-restart event is a genuine fresh persist. Verified locally against a real Postgres 16 before gating (persist during outage → `None` with no raise; first persist after restart → row written). `scripts/integration/spine_test.py` polling now swallows transient HTTP errors during the recovery window so a 5xx blip while the API reconnects extends the poll rather than failing the run.diff --git a/README.md b/README.mdindex 7fcc878a3..0c9305651 100644--- a/README.md+++ b/README.md@@ -75,7 +75,7 @@ The orchestrator is a ~600-line LangGraph in [`services/agents/`](services/agent| Autonomous AI investigation | LangGraph | no | partial (Splunk AI) | yes || Agent decision audit trail | public Investigation Ledger | n/a | n/a | not published || Public substrate eval harness | CI-gated, reproducible, with synthetic telemetry corpus + per-template macros | n/a | n/a | not published |-| Detection content | 800 native + 6 000+ imported (Sigma / Splunk / Chronicle / CAR) | 1 200+ rules | 1 000+ apps | curated |+| Detection content | 939 executable (861 native) + 6 000-rule provenance-tracked imported library ([truth table](docs/detections/truth-table.md)) | 1 200+ rules | 1 000+ apps | curated || Plugin SDK | Python / TypeScript / Go | YAML rules only | apps | proprietary || Data residency | your infra | your infra | partial | vendor cloud || Pricing | $0 (self-host) | $0 (self-host) | per ingest GB | enterprise |@@ -169,7 +169,7 @@ A handful of headline capabilities — the rest are catalogued in [`apps/docs/do- **69 click-and-connect data connectors** (EDR/XDR, SIEM, cloud, CNAPP, identity, SaaS, VCS, K8s audit, network) with schema-driven config, live `Test connection`, and vault-encrypted secrets. Walkthrough: [`apps/docs/docs/connectors/index.md`](apps/docs/docs/connectors/index.md).- **Investigation Rail + replayable Investigation Ledger** — every prompt, tool call, evidence chip, and rationale stored against a case, replayable in the UI. [`apps/docs/docs/console/investigation-rail.md`](apps/docs/docs/console/investigation-rail.md).-- **Detection-as-Code lifecycle** — propose → review → eval-gate → promote; CI rejects any candidate that regresses MITRE accuracy. [`apps/docs/docs/concepts/detections.md`](apps/docs/docs/concepts/detections.md) — and the 800+ native Sigma rules live in [`detections/`](detections/).+- **Detection-as-Code lifecycle** — propose → review → eval-gate → promote; CI rejects any candidate that fails its own positive/negative fixtures (the non-circular gate) or regresses MITRE accuracy. [`apps/docs/docs/concepts/detections.md`](apps/docs/docs/concepts/detections.md) — and the 861 native rules live in [`detections/`](detections/).- **L0–L4 automation maturity** — gate every autonomous action on per-action confidence thresholds and blast-radius. [`apps/docs/docs/concepts/automation-maturity.md`](apps/docs/docs/concepts/automation-maturity.md).- **Hunt-as-Code** — YAML hypotheses with MITRE tags, cron schedules, and natural-language `/hunt` workbench. [`hunts/`](hunts/) + [`apps/docs/docs/console/rule-tuning.md`](apps/docs/docs/console/rule-tuning.md).- **Public weekly benchmark scoreboard** — the same harness that gates PRs, run weekly against `main`. [`apps/docs/docs/benchmark-scoreboard.mdx`](apps/docs/docs/benchmark-scoreboard.mdx).diff --git a/ROADMAP.md b/ROADMAP.mdindex cdd45e909..3db286369 100644--- a/ROADMAP.md+++ b/ROADMAP.md@@ -26,7 +26,7 @@ A proof-first, security-first program to make every README claim gate-backed, cl- [x] Phase 1 — Four existential holes (prompt injection, memory poisoning, cross-store tenant isolation, data-exfiltration/redaction, cost DoS, vault)- [x] Phase 2 — Supply chain + truth (security.yml scanners, claim-gate ratchet, hard-fail insecure prod defaults, TRADEMARK, verifying-releases, license fix; continuation landed: per-image cosign signatures + CycloneDX SBOM attestations + SLSA provenance, all actions SHA-pinned)- [x] Phase 3 — Integration / E2E / chaos / DR (real-container spine test, backup-restore, chaos, upgrade, cross-store isolation live-replay; heavy-demo-stack Playwright E2E + demo-timing gate tracked as non-blocking 3.5+)-- [ ] Phase 4 — Real evals + detection content truth table (third-party-labeled corpus, hallucination/calibration/abstention, model matrix)+- [ ] Phase 4 — Real evals + detection content truth table (third-party-labeled corpus, hallucination/calibration/abstention, model matrix) — **4a/4b landed** (de-circularised DAC candidate-rule gate + honest executable-vs-imported truth table); 4c+ live-agent eval / calibration / model matrix pending- [ ] Phase 5 — Data spine correctness (schema registry, DLQ, idempotency, watermarking, backfill/replay, lineage)- [ ] Phase 6 — Performance + cost (throughput harness, $/TB, storage-consolidation ADR)- [ ] Phase 7 — Ingest-time graph + multi-model router (posture, effective permissions, bi-temporal, fusion-time ContextBundle, determinism contract)diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.mdindex d54429f5c..5b6426c7b 100644--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md@@ -17,9 +17,9 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`| Public eval harness gates every PR | README L62, L77 | `ci.yml :: p1-eval` | GATED (but suites are self-consistency; see reality report) | Phase 4 || Alert-reduction is a real measurement | README L62 | `ci.yml :: p1-eval` (`alert_reduction`) | PARTIAL (gates an in-test fusion re-impl, not `services/fusion`) | Phase 4 || Runs entirely on your infrastructure / no data exfiltration | README L63 | `ci.yml :: python-test` (agents) runs `test_privacy_redactor.py` (zero raw PII survives) | PARTIAL (redaction gated + README made precise per mode; air-gapped egress-blocked CI + Helm NetworkPolicy in continuation/Phase 2) | Phase 2 |-| Detection-as-Code rejects candidates that regress MITRE accuracy | README L170 | `ci.yml :: p1-eval` (w2-dac baseline) | PARTIAL (circular: candidate `rule_body` never evaluated) | Phase 4 |+| Detection-as-Code rejects candidates that regress MITRE accuracy | README L170 | `ci.yml :: python-test` (`test_detection_eval.py` — candidate `rule_body` run through the real engine vs its own positive/negative fixtures; approval requires it) + `p1-eval` w2-dac baseline | GATED | - || 800+ native detection rules | README L78, L170 | `validate-detections.yml` (strict fixture replay) | GATED | - |-| 6000+ imported detection rules | README L78 | `validate-detections.yml` (parse/provenance) | PARTIAL (97% quarantined/non-executable; heatmap is tag-based) | Phase 4 Tier 3 |+| 6000+ imported detection rules | README L78 | `validate-detections.yml` (parse/provenance + `detection_truth_table.py --check`); README cites the executable figure (939) not the on-disk figure for coverage | GATED | - || L0-L4 automation maturity gates every action | README L171 | `ci.yml` autonomy-policy drift test | PARTIAL (thresholds gated; rollback/post-verify/dry-run ungated) | Phase 9 || Hunt-as-Code + `/hunt` | README L172 | `ci.yml :: p1-eval` (`hunt_corpus`) | GATED | - || Weekly benchmark scoreboard runs live against `main` | README L173 | `wet-eval.yml` (weekly) | NO GATE (no-ops without secret; live-agent tables are placeholders) | Phase 4 Tier 1 |@@ -38,9 +38,9 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`## Summary-- GATED: 12-- PARTIAL: 12-- NO GATE: 3 (Phase 2 moved insecure-defaults-hard-fail to GATED and secret/IaC/container scanning to PARTIAL; the Phase 2 continuation moved signed/attested releases to GATED; Phase 3.4 moved cross-store isolation to GATED via the live-container replay; remaining NO GATE close in Phases 4/10/11)+- GATED: 14+- PARTIAL: 10+- NO GATE: 3 (Phase 2 moved insecure-defaults-hard-fail to GATED and secret/IaC/container scanning to PARTIAL; the Phase 2 continuation moved signed/attested releases to GATED; Phase 3.4 moved cross-store isolation to GATED via the live-container replay; Phase 4 moved the DAC candidate-rule gate and the imported-count honesty gate to GATED; remaining NO GATE close in Phases 4-continuation/10/11)The ratchet is enforced by `scripts/check_claim_gate_matrix.py` (wired into `security.yml`): the NO GATE count may only decrease.diff --git a/docs/detections/truth-table.md b/docs/detections/truth-table.mdnew file mode 100644index 000000000..147ac1ead--- /dev/null+++ b/docs/detections/truth-table.md@@ -0,0 +1,48 @@+# Detection content — truth table++> Generated by `scripts/detection_truth_table.py`. `--check` gates this+> file in `validate-detections.yml`, so the numbers below can never quietly+> diverge from what the engine actually runs. **Do not edit by hand.**++A rule is **executable** when it fires in AiSOC today: not quarantined+(`_quarantine/` or `enabled: false`) and its `detection` body is the native+AiSOC condition DSL, a Sigma selection/condition, or a runtime-engine language.+A rule is **non-executable** when quarantined or when its only body is an+untranslated upstream language (SPL / YARA-L / CAR pseudocode) — present for+provenance and coverage-mapping, not firing.++## Headline++| metric | count |+|--------|------:|+| rules on disk (total) | 6975 |+| **executable (fire today)** | **939** |+| non-executable (provenance/coverage only) | 6036 |++## By tier++| tier | on disk | executable |+|------|--------:|-----------:|+| car (imported) | 99 | 0 |+| chronicle (imported) | 877 | 0 |+| community | 1 | 1 |+| native | 861 | 861 |+| sigma (imported) | 3132 | 77 |+| splunk (imported) | 2005 | 0 |++## Why rules are non-executable++| reason | count |+|--------|------:|+| `enabled: false` | 115 |+| under `_quarantine/` (untranslated on import) | 5921 |++## How to read the README claim++The imported corpus is large (6975 rules on disk) and valuable as a+provenance-tracked ATT&CK-mapped library, but the number that matters+operationally is **939 executable rules** — the ones the engine+fires against live telemetry. The README and marketplace must cite the+executable figure when describing detection *coverage*, and may cite the+on-disk figure only when explicitly describing the imported *library*.+diff --git a/scripts/detection_truth_table.py b/scripts/detection_truth_table.pynew file mode 100644index 000000000..985570297--- /dev/null+++ b/scripts/detection_truth_table.py@@ -0,0 +1,256 @@+#!/usr/bin/env python3+"""Detection content truth table (Phase 4 — honest coverage).++The Phase 0 reality audit's overclaim #2: the README advertises "6000+ imported+detection rules", but ~97% of the imported set lives under `_quarantine/`+(`enabled: false`) because its upstream query language (SPL / YARA-L / CAR+pseudocode) does not execute on the AiSOC engine. The ATT&CK heatmap counts+metadata tags, not rules that fire.++This script walks `detections/` and classifies every rule as **executable**+(fires in AiSOC today) or **non-executable** (present for provenance /+coverage-mapping only), then renders an honest breakdown to+`docs/detections/truth-table.md`. `--check` fails CI when the committed doc+drifts from the on-disk reality — so the headline number can never quietly+diverge from what actually runs.++A rule is EXECUTABLE when it is not quarantined (not under `_quarantine/`, not+`enabled: false`) AND its `detection` body is in a form the engine evaluates:+the native AiSOC `condition` DSL, a Sigma `selection`/`condition`, or one of the+runtime-engine languages. A rule is NON-EXECUTABLE when quarantined or when its+only body is an untranslated upstream language (`splunk_spl`, `chronicle_yaral`,+CAR pseudocode).++Usage:+ python3 scripts/detection_truth_table.py # regenerate the doc+ python3 scripts/detection_truth_table.py --check # fail on drift+ python3 scripts/detection_truth_table.py --json # print counts as JSON+"""++from __future__ import annotations++import argparse+import json+import sys+from dataclasses import dataclass, field+from pathlib import Path++import yaml++ROOT = Path(__file__).resolve().parent.parent+DETECTIONS = ROOT / "detections"+DOC = ROOT / "docs" / "detections" / "truth-table.md"++# Directories under detections/ that are not rules.+SKIP_DIRS = {"fixtures", "playbooks"}++NATIVE_CATEGORIES = {"endpoint", "cloud", "identity", "network", "application", "data-exfil"}++# detection-body keys that the AiSOC engine can actually evaluate.+EXECUTABLE_BODY_KEYS = {"condition", "selection", "sigma", "yara", "kql", "eql", "lucene", "regex", "query", "keywords"}+# untranslated upstream query languages — present for provenance, do not fire.+NON_EXECUTABLE_BODY_KEYS = {"splunk_spl", "chronicle_yaral", "yaral", "spl", "car_pseudocode", "car"}+++@dataclass+class Counts:+ total: int = 0+ unparseable: int = 0+ executable: int = 0+ non_executable: int = 0+ by_tier: dict[str, int] = field(default_factory=dict)+ executable_by_tier: dict[str, int] = field(default_factory=dict)+ non_exec_reason: dict[str, int] = field(default_factory=dict)++ def bump(self, d: dict[str, int], key: str) -> None:+ d[key] = d.get(key, 0) + 1+++def _tier_for(path: Path) -> str:+ rel = path.relative_to(DETECTIONS)+ top = rel.parts[0]+ if top in NATIVE_CATEGORIES:+ return "native"+ if top.endswith("-imports"):+ return top[: -len("-imports")] + " (imported)"+ if top == "community":+ return "community"+ return top+++def _is_quarantined(path: Path, data: dict) -> tuple[bool, str]:+ if "_quarantine" in path.parts:+ return True, "quarantine_dir"+ enabled = data.get("enabled")+ if enabled is False:+ return True, "disabled"+ return False, ""+++def _has_executable_body(data: dict) -> bool:+ det = data.get("detection")+ if not isinstance(det, dict):+ return False+ keys = set(det.keys())+ if keys & EXECUTABLE_BODY_KEYS:+ return True+ # A body that is only an untranslated upstream language does not fire.+ if keys & NON_EXECUTABLE_BODY_KEYS:+ return False+ # Unknown shape — treat as non-executable so the honest count never+ # over-reports what fires.+ return False+++def _iter_rule_files() -> list[Path]:+ out: list[Path] = []+ for path in sorted(DETECTIONS.rglob("*.yaml")):+ rel = path.relative_to(DETECTIONS)+ if rel.parts[0] in SKIP_DIRS:+ continue+ if path.name.lower() in {"readme.yaml", "index.yaml"}:+ continue+ out.append(path)+ return out+++def compute() -> Counts:+ counts = Counts()+ for path in _iter_rule_files():+ counts.total += 1+ try:+ data = yaml.safe_load(path.read_text(encoding="utf-8"))+ except (yaml.YAMLError, OSError):+ counts.unparseable += 1+ counts.non_executable += 1+ counts.bump(counts.non_exec_reason, "unparseable")+ continue+ if not isinstance(data, dict):+ counts.unparseable += 1+ counts.non_executable += 1+ counts.bump(counts.non_exec_reason, "unparseable")+ continue++ tier = _tier_for(path)+ counts.bump(counts.by_tier, tier)++ quarantined, reason = _is_quarantined(path, data)+ if quarantined:+ counts.non_executable += 1+ counts.bump(counts.non_exec_reason, reason)+ continue+ if _has_executable_body(data):+ counts.executable += 1+ counts.bump(counts.executable_by_tier, tier)+ else:+ counts.non_executable += 1+ counts.bump(counts.non_exec_reason, "untranslated_upstream_language")+ return counts+++def render_markdown(c: Counts) -> str:+ lines: list[str] = []+ lines.append("# Detection content — truth table")+ lines.append("")+ lines.append("> Generated by `scripts/detection_truth_table.py`. `--check` gates this")+ lines.append("> file in `validate-detections.yml`, so the numbers below can never quietly")+ lines.append("> diverge from what the engine actually runs. **Do not edit by hand.**")+ lines.append("")+ lines.append("A rule is **executable** when it fires in AiSOC today: not quarantined")+ lines.append("(`_quarantine/` or `enabled: false`) and its `detection` body is the native")+ lines.append("AiSOC condition DSL, a Sigma selection/condition, or a runtime-engine language.")+ lines.append("A rule is **non-executable** when quarantined or when its only body is an")+ lines.append("untranslated upstream language (SPL / YARA-L / CAR pseudocode) — present for")+ lines.append("provenance and coverage-mapping, not firing.")+ lines.append("")+ lines.append("## Headline")+ lines.append("")+ lines.append("| metric | count |")+ lines.append("|--------|------:|")+ lines.append(f"| rules on disk (total) | {c.total} |")+ lines.append(f"| **executable (fire today)** | **{c.executable}** |")+ lines.append(f"| non-executable (provenance/coverage only) | {c.non_executable} |")+ lines.append("")+ lines.append("## By tier")+ lines.append("")+ lines.append("| tier | on disk | executable |")+ lines.append("|------|--------:|-----------:|")+ for tier in sorted(c.by_tier):+ lines.append(f"| {tier} | {c.by_tier[tier]} | {c.executable_by_tier.get(tier, 0)} |")+ lines.append("")+ lines.append("## Why rules are non-executable")+ lines.append("")+ lines.append("| reason | count |")+ lines.append("|--------|------:|")+ reason_labels = {+ "quarantine_dir": "under `_quarantine/` (untranslated on import)",+ "disabled": "`enabled: false`",+ "untranslated_upstream_language": "body is an untranslated upstream language",+ "unparseable": "unparseable YAML",+ }+ for reason in sorted(c.non_exec_reason):+ label = reason_labels.get(reason, reason)+ lines.append(f"| {label} | {c.non_exec_reason[reason]} |")+ lines.append("")+ lines.append("## How to read the README claim")+ lines.append("")+ lines.append(f"The imported corpus is large ({c.total} rules on disk) and valuable as a")+ lines.append("provenance-tracked ATT&CK-mapped library, but the number that matters")+ lines.append(f"operationally is **{c.executable} executable rules** — the ones the engine")+ lines.append("fires against live telemetry. The README and marketplace must cite the")+ lines.append("executable figure when describing detection *coverage*, and may cite the")+ lines.append("on-disk figure only when explicitly describing the imported *library*.")+ lines.append("")+ return "\n".join(lines)+++def main() -> int:+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--check", action="store_true", help="fail if the committed doc is stale")+ parser.add_argument("--json", dest="as_json", action="store_true", help="print counts as JSON")+ args = parser.parse_args()++ counts = compute()+ rendered = render_markdown(counts)++ if args.as_json:+ print(+ json.dumps(+ {+ "total": counts.total,+ "executable": counts.executable,+ "non_executable": counts.non_executable,+ "unparseable": counts.unparseable,+ "by_tier": counts.by_tier,+ "executable_by_tier": counts.executable_by_tier,+ "non_exec_reason": counts.non_exec_reason,+ },+ indent=2,+ sort_keys=True,+ )+ )+ return 0++ if args.check:+ if not DOC.exists():+ print(f"ERROR: {DOC.relative_to(ROOT)} does not exist — run scripts/detection_truth_table.py", file=sys.stderr)+ return 1+ current = DOC.read_text(encoding="utf-8")+ if current.strip() != rendered.strip():+ print(+ f"ERROR: {DOC.relative_to(ROOT)} is stale. The detection content changed but the "+ "truth table was not regenerated. Run: python3 scripts/detection_truth_table.py",+ file=sys.stderr,+ )+ return 1+ print(f"OK: detection truth table current — {counts.executable} executable / {counts.total} on disk")+ return 0++ DOC.parent.mkdir(parents=True, exist_ok=True)+ DOC.write_text(rendered + "\n", encoding="utf-8")+ print(f"wrote {DOC.relative_to(ROOT)} — {counts.executable} executable / {counts.total} on disk")+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/services/api/app/api/v1/endpoints/detection_proposals.py b/services/api/app/api/v1/endpoints/detection_proposals.pyindex cd51e47ac..5c9226cec 100644--- a/services/api/app/api/v1/endpoints/detection_proposals.py+++ b/services/api/app/api/v1/endpoints/detection_proposals.py@@ -1,11 +1,19 @@"""Detection-as-code proposal lifecycle endpoints (Wave 2 — w2-dac).Surfaces the propose → review → eval-gated → promote flow that brings-detections under the same CI gate as agent prompts. The eval gate is-satisfied by `scripts/run_evals.py` running offline; the result of that-run is stored back on the proposal as ``eval_result`` and a candidate-that regresses MITRE accuracy by ≥ ``max_regression_pp`` percentage-points cannot be promoted.+detections under the same CI gate as agent prompts.++Two independent gates guard promotion, and approval requires **both**:++* **Candidate-rule gate (Phase 4, non-circular).** ``/evaluate-rule`` runs the+ *proposed rule body itself* through the runtime engine against caller-supplied+ positive and negative fixtures — it must fire on every positive and stay+ silent on every negative. This closes the Phase 0 reality-audit finding that+ the promote path never evaluated the proposed rule.+* **Substrate-benchmark gate.** ``/run-eval`` / ``/eval`` run+ ``scripts/run_evals.py`` and reject a candidate that regresses repo-wide MITRE+ accuracy by ≥ ``max_regression_pp`` pp. This guards against a rule that passes+ its own fixtures but perturbs the shared substrate.Endpoints---------@@ -13,7 +21,8 @@ Endpoints* ``POST /detection-proposals`` Create a proposal.* ``GET /detection-proposals/{id}`` Proposal detail.* ``POST /detection-proposals/{id}/comment`` Add a review comment.-* ``POST /detection-proposals/{id}/eval`` Attach eval result + verdict.+* ``POST /detection-proposals/{id}/evaluate-rule`` Evaluate the candidate rule vs its fixtures (non-circular gate).+* ``POST /detection-proposals/{id}/eval`` Attach substrate-benchmark result + verdict.* ``POST /detection-proposals/{id}/decide`` Approve/reject a proposal.* ``POST /detection-proposals/{id}/promote`` Materialise into ``detection_rules``.* ``GET /detection-proposals/baselines`` List eval baselines.@@ -44,6 +53,7 @@ from app.models.detection_proposal import (DetectionRuleProposal,)from app.models.detection_rule import DetectionRule+from app.services.detection_eval import evaluate_candidate_rulefrom app.services.github import create_detection_prrouter = APIRouter(prefix="/detection-proposals", tags=["detection_rules", "dac"])@@ -165,6 +175,29 @@ class EvalAttachRequest(BaseModel):)+class EvaluateRuleRequest(BaseModel):+ """Evaluate the candidate rule body against its own fixtures (Phase 4).++ This is the non-circular gate: unlike ``/run-eval`` (which measures the+ repo-wide substrate benchmark and is independent of the proposed rule),+ this runs the *proposed rule body itself* through the runtime engine and+ asserts it fires on every positive fixture and stays silent on every+ negative one. Approval now requires this verdict to pass.+ """++ positive_fixtures: list[dict[str, Any]] = Field(+ ...,+ min_length=1,+ max_length=200,+ description="Events the rule MUST fire on (the attacks it claims to catch). At least one required.",+ )+ negative_fixtures: list[dict[str, Any]] = Field(+ default_factory=list,+ max_length=200,+ description="Benign near-miss events the rule MUST NOT fire on.",+ )++class DecisionRequest(BaseModel):decision: Literal["approve", "reject"]comment: str | None = Field(default=None, max_length=2000)@@ -643,6 +676,54 @@ async def run_eval_harness(_ = exc # best-effort cleanup; ignore unlink failures[email protected]("/{proposal_id}/evaluate-rule", response_model=ProposalResponse)+async def evaluate_rule(+ proposal_id: uuid.UUID,+ request: EvaluateRuleRequest,+ current_user: Annotated[AuthUser, Depends(require_permission("rules:write"))],+ db: DBSession,+) -> ProposalResponse:+ """Evaluate the candidate rule body against its own fixtures (Phase 4).++ Runs the proposed ``rule_language`` + ``rule_body`` through the runtime+ engine against the supplied fixtures and stores the verdict under+ ``eval_result["candidate_rule"]``. This is the gate that de-circularises+ promotion: a rule that misses its positives (blind) or fires on its+ negatives (noisy) cannot be approved, regardless of the global benchmark.+ """+ _ensure_dac_enabled()+ proposal = await _load_proposal(db, proposal_id, current_user.tenant_id)+ if proposal.status in {"promoted", "rejected"}:+ raise HTTPException(+ status_code=status.HTTP_409_CONFLICT,+ detail=f"Proposal is {proposal.status}; candidate-rule eval cannot be re-run",+ )++ result = evaluate_candidate_rule(+ rule_language=proposal.rule_language,+ rule_body=proposal.rule_body,+ positive_fixtures=request.positive_fixtures,+ negative_fixtures=request.negative_fixtures,+ )++ # Merge into eval_result so the benchmark verdict (if already attached) is+ # preserved alongside the candidate-rule verdict.+ merged = dict(proposal.eval_result or {})+ merged["candidate_rule"] = {**result.to_dict(), "ran_at": datetime.now(UTC).isoformat()}+ proposal.eval_result = merged++ # A candidate-rule failure is decisive — it cannot be approved. On pass,+ # only advance to eval_passed if the benchmark gate isn't failing.+ if not result.passed:+ proposal.status = "eval_failed"+ elif proposal.status not in {"eval_failed"}:+ proposal.status = "eval_passed"++ await db.commit()+ await db.refresh(proposal)+ return ProposalResponse.model_validate(proposal)++@router.post("/{proposal_id}/eval", response_model=ProposalResponse)async def attach_eval_result(proposal_id: uuid.UUID,@@ -686,8 +767,21 @@ async def attach_eval_result(max_regression_pp=request.max_regression_pp,)- proposal.eval_result = verdict- proposal.status = "eval_passed" if verdict["passed"] else "eval_failed"+ # Preserve any candidate-rule verdict already attached via /evaluate-rule.+ existing = dict(proposal.eval_result or {})+ candidate = existing.get("candidate_rule")+ merged = dict(verdict)+ if candidate is not None:+ merged["candidate_rule"] = candidate+ proposal.eval_result = merged++ # Benchmark pass alone is not sufficient if the candidate rule failed its+ # own fixtures — the non-circular gate is decisive.+ candidate_ok = (candidate or {}).get("passed", None)+ if candidate_ok is False:+ proposal.status = "eval_failed"+ else:+ proposal.status = "eval_passed" if verdict["passed"] else "eval_failed"await db.commit()await db.refresh(proposal)return ProposalResponse.model_validate(proposal)@@ -710,11 +804,33 @@ async def decide_proposal()if request.decision == "approve":- eval_passed = bool(proposal.eval_result.get("passed")) if proposal.eval_result else False- if not eval_passed:+ eval_result = proposal.eval_result or {}+ # Phase 4 — de-circularised gate. Approval requires the candidate rule+ # to have been evaluated against its own fixtures and passed (fires on+ # positives, silent on negatives). A benchmark-only "pass" is no longer+ # sufficient: the reality audit showed the benchmark is independent of+ # the proposed rule, so a blind/noisy rule could sail through.+ candidate = eval_result.get("candidate_rule")+ if not candidate:+ raise HTTPException(+ status_code=status.HTTP_412_PRECONDITION_FAILED,+ detail=(+ "Candidate-rule eval has not been run; cannot approve. "+ "POST /detection-proposals/{id}/evaluate-rule with positive "+ "and negative fixtures first."+ ),+ )+ if not bool(candidate.get("passed")):+ raise HTTPException(+ status_code=status.HTTP_412_PRECONDITION_FAILED,+ detail=f"Candidate-rule eval failed ({candidate.get('reason', 'rule did not pass its fixtures')}); cannot approve.",+ )+ # If a substrate-benchmark verdict is also attached it must not be a+ # regression failure.+ if "passed" in eval_result and not bool(eval_result.get("passed")):raise HTTPException(status_code=status.HTTP_412_PRECONDITION_FAILED,- detail="Eval gate has not passed; cannot approve. Run the eval suite and attach the report first.",+ detail="Substrate benchmark gate failed (MITRE regression); cannot approve.",)proposal.status = "approved"else:diff --git a/services/api/app/services/detection_eval.py b/services/api/app/services/detection_eval.pynew file mode 100644index 000000000..27c737c30--- /dev/null+++ b/services/api/app/services/detection_eval.py@@ -0,0 +1,158 @@+"""Candidate detection-rule evaluation (Phase 4 — de-circularise the DAC gate).++The Phase 0 reality audit's #1 circular gate: the detection-proposal promote+path shelled out to ``scripts/run_evals.py`` **without ever passing the proposed+rule body**. "Passed" meant the repo's global substrate MITRE accuracy did not+regress — a value entirely independent of the rule under review. A rule that+matches nothing (blind) or matches everything (noisy) sailed through its own+exam.++This module closes that hole. It evaluates the **candidate rule body itself**+against caller-supplied positive and negative fixtures using the *real* runtime+engine (:func:`app.services.rule_engine.execute_rule`) — the same code that runs+the rule in production. The contract is the one every detection must satisfy:++* it MUST fire on every positive fixture (a rule that misses the attack it+ claims to catch is worthless), and+* it MUST stay silent on every negative fixture (a rule that fires on the+ benign near-miss is a false-positive factory).++The proposal approve-gate now requires this verdict to pass, so a bad rule is+rejected on its own merits rather than on an unrelated global metric.+"""++from __future__ import annotations++from dataclasses import dataclass, field+from typing import Any++from app.services.rule_engine import execute_rule+++@dataclass(frozen=True)+class FixtureOutcome:+ """Per-fixture result: did the candidate rule fire, and was that expected?"""++ index: int+ kind: str # "positive" | "negative"+ fired: bool+ expected_fire: bool+ ok: bool+ error: str | None = None+++@dataclass(frozen=True)+class CandidateEvalResult:+ """Outcome of evaluating a candidate rule against its fixtures."""++ passed: bool+ positives_total: int+ positives_fired: int+ negatives_total: int+ negatives_fired: int+ outcomes: list[FixtureOutcome] = field(default_factory=list)+ reason: str = ""++ def to_dict(self) -> dict[str, Any]:+ return {+ "passed": self.passed,+ "positives_total": self.positives_total,+ "positives_fired": self.positives_fired,+ "negatives_total": self.negatives_total,+ "negatives_fired": self.negatives_fired,+ "reason": self.reason,+ "outcomes": [+ {+ "index": o.index,+ "kind": o.kind,+ "fired": o.fired,+ "expected_fire": o.expected_fire,+ "ok": o.ok,+ "error": o.error,+ }+ for o in self.outcomes+ ],+ }+++def _fires(rule_language: str, rule_body: str, event: dict[str, Any]) -> tuple[bool, str | None]:+ """Run the candidate rule against a single event via the real engine."""+ match = execute_rule(+ rule_id="candidate",+ rule_name="candidate",+ rule_language=rule_language,+ rule_body=rule_body,+ severity="medium",+ events=[event],+ )+ return bool(match.matched), match.error+++def evaluate_candidate_rule(+ *,+ rule_language: str,+ rule_body: str,+ positive_fixtures: list[dict[str, Any]],+ negative_fixtures: list[dict[str, Any]],+) -> CandidateEvalResult:+ """Evaluate a candidate rule against its own positive/negative fixtures.++ Returns a :class:`CandidateEvalResult`. ``passed`` is True only when the+ rule fired on *every* positive fixture, stayed silent on *every* negative+ fixture, and at least one positive fixture was supplied (a rule with no+ positive fixture has proven nothing — the gate would be vacuous).+ """+ outcomes: list[FixtureOutcome] = []++ if not positive_fixtures:+ return CandidateEvalResult(+ passed=False,+ positives_total=0,+ positives_fired=0,+ negatives_total=len(negative_fixtures),+ negatives_fired=0,+ reason="no positive fixtures supplied — cannot prove the rule catches anything",+ )++ positives_fired = 0+ for i, event in enumerate(positive_fixtures):+ fired, err = _fires(rule_language, rule_body, event)+ if fired:+ positives_fired += 1+ outcomes.append(+ FixtureOutcome(index=i, kind="positive", fired=fired, expected_fire=True, ok=fired, error=err)+ )++ negatives_fired = 0+ for i, event in enumerate(negative_fixtures):+ fired, err = _fires(rule_language, rule_body, event)+ if fired:+ negatives_fired += 1+ outcomes.append(+ FixtureOutcome(index=i, kind="negative", fired=fired, expected_fire=False, ok=not fired, error=err)+ )++ fired_on_all_positives = positives_fired == len(positive_fixtures)+ silent_on_all_negatives = negatives_fired == 0+ passed = fired_on_all_positives and silent_on_all_negatives++ n_pos = len(positive_fixtures)+ n_neg = len(negative_fixtures)+ if passed:+ reason = "rule fired on all positives and stayed silent on all negatives"+ elif not fired_on_all_positives and not silent_on_all_negatives:+ reason = f"rule is both blind ({positives_fired}/{n_pos} positives) and noisy ({negatives_fired}/{n_neg} negatives fired)"+ elif not fired_on_all_positives:+ reason = f"rule is blind: fired on only {positives_fired}/{n_pos} positive fixtures"+ else:+ reason = f"rule is noisy: fired on {negatives_fired}/{n_neg} negative fixtures (must be 0)"++ return CandidateEvalResult(+ passed=passed,+ positives_total=len(positive_fixtures),+ positives_fired=positives_fired,+ negatives_total=len(negative_fixtures),+ negatives_fired=negatives_fired,+ outcomes=outcomes,+ reason=reason,+ )diff --git a/services/api/tests/test_detection_eval.py b/services/api/tests/test_detection_eval.pynew file mode 100644index 000000000..287c1e6fc--- /dev/null+++ b/services/api/tests/test_detection_eval.py@@ -0,0 +1,139 @@+"""Phase 4 — the anti-circularity gate for detection-rule promotion.++The Phase 0 reality audit's #1 circular gate: the detection-proposal promote+path shelled out to the eval harness **without ever passing the proposed rule+body**, so a rule that matches nothing (blind) or matches everything (noisy)+passed its own exam. `app.services.detection_eval.evaluate_candidate_rule`+closes that hole by running the *candidate rule body itself* through the real+runtime engine (`app.services.rule_engine.execute_rule`) against its fixtures.++This suite is the mutation test that proves the gate is no longer circular: a+deliberately-broken rule is REJECTED, and a correct rule is accepted. It uses+the real engine, so the verdict genuinely depends on the rule body.+"""++from __future__ import annotations++from app.services.detection_eval import evaluate_candidate_rule++# A correct Sigma rule: fires on a root console login, silent on an IAM-user+# login. Evaluated by the runtime engine's Sigma path (pySigma if present, else+# the substring fallback — the assertions hold for both).+GOOD_RULE = """+title: AWS Root Console Login+logsource:+ product: aws+ service: cloudtrail+detection:+ selection:+ event_name: ConsoleLogin+ user_type: Root+ condition: selection+"""++POSITIVE = {"event_name": "ConsoleLogin", "user_type": "Root", "src_ip": "203.0.113.7"}+NEGATIVE = {"event_name": "ConsoleLogin", "user_type": "IAMUser", "src_ip": "203.0.113.7"}+++def test_good_rule_passes_its_own_fixtures():+ result = evaluate_candidate_rule(+ rule_language="sigma",+ rule_body=GOOD_RULE,+ positive_fixtures=[POSITIVE],+ negative_fixtures=[NEGATIVE],+ )+ assert result.passed, result.reason+ assert result.positives_fired == 1+ assert result.negatives_fired == 0+++def test_blind_rule_is_rejected():+ """A rule that never fires on its positive fixture must fail — the core+ circular-gate refutation: a blind rule no longer 'passes'."""+ blind = """+title: Never Matches Anything+logsource:+ product: aws+detection:+ selection:+ event_name: EventThatNeverOccursZZZ+ condition: selection+"""+ result = evaluate_candidate_rule(+ rule_language="sigma",+ rule_body=blind,+ positive_fixtures=[POSITIVE],+ negative_fixtures=[NEGATIVE],+ )+ assert not result.passed+ assert result.positives_fired == 0+ assert "blind" in result.reason+++def test_noisy_rule_is_rejected():+ """A rule that also fires on its negative (benign) fixture must fail."""+ noisy = """+title: Fires On Any Console Login+logsource:+ product: aws+detection:+ selection:+ event_name: ConsoleLogin+ condition: selection+"""+ result = evaluate_candidate_rule(+ rule_language="sigma",+ rule_body=noisy,+ positive_fixtures=[POSITIVE],+ negative_fixtures=[NEGATIVE],+ )+ assert not result.passed+ assert result.negatives_fired == 1+ assert "noisy" in result.reason+++def test_no_positive_fixtures_is_rejected_not_vacuous():+ """A rule with no positive fixture has proven nothing — it must not pass."""+ result = evaluate_candidate_rule(+ rule_language="sigma",+ rule_body=GOOD_RULE,+ positive_fixtures=[],+ negative_fixtures=[NEGATIVE],+ )+ assert not result.passed+ assert result.positives_total == 0+ assert "no positive" in result.reason.lower()+++def test_verdict_is_json_serialisable_with_per_fixture_detail():+ result = evaluate_candidate_rule(+ rule_language="sigma",+ rule_body=GOOD_RULE,+ positive_fixtures=[POSITIVE],+ negative_fixtures=[NEGATIVE],+ )+ d = result.to_dict()+ assert d["passed"] is True+ assert len(d["outcomes"]) == 2+ kinds = {o["kind"] for o in d["outcomes"]}+ assert kinds == {"positive", "negative"}+ assert all(o["ok"] for o in d["outcomes"])+++def test_the_gate_verdict_depends_on_the_rule_body():+ """The property the circular gate lacked: two different rule bodies against+ the *same* fixtures yield different verdicts. If the verdict were+ independent of the rule (the old bug), these would be identical."""+ good = evaluate_candidate_rule(+ rule_language="sigma",+ rule_body=GOOD_RULE,+ positive_fixtures=[POSITIVE],+ negative_fixtures=[NEGATIVE],+ )+ bad = evaluate_candidate_rule(+ rule_language="sigma",+ rule_body="title: X\ndetection:\n selection:\n event_name: NopeZZZ\n condition: selection\n",+ positive_fixtures=[POSITIVE],+ negative_fixtures=[NEGATIVE],+ )+ assert good.passed and not bad.passed