Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

feat(phase-4): de-circularise the DAC gate + detection content truth table (#429)

Beenu Arora9ae871362a40
diff --git a/.github/workflows/validate-detections.yml b/.github/workflows/validate-detections.yml
index d4370524a..b87c25842 100644
--- a/.github/workflows/validate-detections.yml
+++ b/.github/workflows/validate-detections.yml
@@ -34,6 +34,12 @@ jobs:
- name: Verify curated v1.0 coverage manifest
run: python3 scripts/curate_detections.py --check
+ - name: Verify detection content truth table (Phase 4 — honest coverage)
+ # Fails if the committed docs/detections/truth-table.md drifts from the
+ # on-disk reality, so the "executable rules" headline can never quietly
+ # diverge from what the engine actually runs.
+ run: python3 scripts/detection_truth_table.py --check
+
- name: Count rules by severity
if: always()
run: |
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8e45f254a..648bc552e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
+- **Phase 4a — the Detection-as-Code gate is no longer circular.** The reality audit's #1 circular gate: the detection-proposal promote path shelled out to `run_evals.py` **without ever passing the proposed rule body**, so "passed" meant the repo-wide substrate MITRE accuracy didn't move — a value independent of the rule under review. A blind rule (matches nothing) or a noisy rule (matches everything) sailed through its own exam. New `services/api/app/services/detection_eval.py::evaluate_candidate_rule` runs the *candidate `rule_body` itself* through the **real** runtime engine (`rule_engine.execute_rule`) against caller-supplied positive/negative fixtures: it must fire on every positive and stay silent on every negative. New `POST /detection-proposals/{id}/evaluate-rule` stores the verdict under `eval_result["candidate_rule"]`, and `POST /decide` (approve) now **requires** it — a benchmark-only pass is no longer sufficient. `services/api/tests/test_detection_eval.py` (6 tests) is the mutation test that proves the gate rejects a blind rule, rejects a noisy rule, rejects a no-positive-fixture rule, and that the verdict genuinely depends on the rule body. Claim-to-gate matrix: DAC row PARTIAL(circular) → GATED.
+- **Phase 4b — detection content truth table (honest coverage).** The README advertised "6000+ imported detection rules", but ~97% live under `_quarantine/` (`enabled: false`) because their upstream query language (SPL / YARA-L / CAR pseudocode) does not execute on the engine. New `scripts/detection_truth_table.py` walks `detections/` and classifies every rule **executable** (fires today) vs **non-executable** (provenance/coverage only), rendering `docs/detections/truth-table.md`. The honest headline: **939 executable** rules (861 native + 77 sigma-imported + 1 community) of 6975 on disk; 5921 quarantined, 115 disabled. `--check` gates the doc in `validate-detections.yml` so the number can never quietly drift from reality, and the README now cites the executable figure for *coverage* (the on-disk figure only describes the imported *library*). Claim-to-gate matrix: 6000+-imported row PARTIAL → GATED (14 GATED / 10 PARTIAL / 3 NO GATE). The LLM-dependent half of Phase 4 (live-agent Tier-1 eval, hallucination/calibration/abstention, model matrix, 150-payload prompt-injection adversarial) is tracked as 4c+ in `docs/audit/PROGRESS.md`.
- **Phase 3.4 — cross-store tenant isolation, now proven against live containers.** The offline `isolation.yml` gate proved each read path *constructs* a tenant scope; it could not prove the scope actually isolates. New `.github/workflows/isolation-live.yml` + `tests/isolation/test_live_stores.py` seed tenant A and tenant B in **real containers** (Neo4j, Redis, ClickHouse, Redpanda/Kafka) and assert a read as A returns zero B data. ClickHouse runs the **production** `lake_sql.rewrite_for_tenant` rewriter against a live warehouse (verified: `SELECT … FROM aisoc.raw_events` becomes `… WHERE tenant_id = '<A>'`, so B's rows never return); Neo4j uses the `tenant_id` property filter; Redis uses the `aisoc:t:<tenant>:*` keyspace namespacing; Kafka replays the `graph_ws` per-tenant envelope filter. Every test also asserts the *unscoped* read sees both tenants, so a scoped pass can never be vacuous on an empty store. The isolation registry (`tests/isolation/stores.py`) flips Neo4j/Redis/ClickHouse/Kafka from `container_pending` → `container_gated`, and the claim-to-gate matrix moves "Cross-tenant isolation (Qdrant/Neo4j/Redis/ClickHouse/Kafka)" PARTIAL → GATED (12 GATED / 12 PARTIAL / 3 NO GATE). The heavy-demo-stack items (Playwright real-stack E2E, demo time-to-first-investigation budget) are tracked as non-blocking 3.5+ in `docs/audit/PROGRESS.md`.
- **Phase 3.3 — upgrade-safety gate ("people upgrade; nothing tested it").** New `upgrade` job in `integration.yml` (matrix `v7.5.0 → HEAD` and `v7.3.1 → HEAD` against a real Postgres 16): install a prior release's migration set, seed 250 probe rows on that released schema, then apply HEAD's migration set — the actual self-host upgrade path — and assert the seeded rows survive **and** every HEAD migration applied. The signal this adds beyond the existing fresh-apply job is **destructive-migration detection**: a migration that drops or rewrites existing data fails here even though it applies cleanly on an empty database. The `v7.3.1 → HEAD` leg lands 8 incremental migrations on a pre-existing populated schema (verified locally: 250/250 rows survived, 55/55 migrations tracked). Uses `git checkout <ref> -- services/api/migrations` with an `rm -rf` first so the released set is exact (a `checkout … -- path` alone leaks HEAD-only files and silently degrades to a fresh apply).
- **Phase 3.2 — Postgres-outage chaos gate (fail-soft + self-heal).** Extends the `integration.yml` spine job with a second chaos scenario beyond the consumer-kill test: kill Postgres mid-stream, assert the fusion consumer keeps running (the `AlertSink` degrades to "fused alerts still stream over Kafka/WS, persistence dropped for the outage window" instead of crashing the worker), then restart Postgres and assert a *fresh* event persists — proving the sink's asyncpg pool and the API's SQLAlchemy pool both self-heal with no service restart. Distinct event titles side-step fusion's in-memory dedup so the post-restart event is a genuine fresh persist. Verified locally against a real Postgres 16 before gating (persist during outage → `None` with no raise; first persist after restart → row written). `scripts/integration/spine_test.py` polling now swallows transient HTTP errors during the recovery window so a 5xx blip while the API reconnects extends the poll rather than failing the run.
diff --git a/README.md b/README.md
index 7fcc878a3..0c9305651 100644
--- a/README.md
+++ b/README.md
@@ -75,7 +75,7 @@ The orchestrator is a ~600-line LangGraph in [`services/agents/`](services/agent
| Autonomous AI investigation | LangGraph | no | partial (Splunk AI) | yes |
| Agent decision audit trail | public Investigation Ledger | n/a | n/a | not published |
| Public substrate eval harness | CI-gated, reproducible, with synthetic telemetry corpus + per-template macros | n/a | n/a | not published |
-| Detection content | 800 native + 6 000+ imported (Sigma / Splunk / Chronicle / CAR) | 1 200+ rules | 1 000+ apps | curated |
+| Detection content | 939 executable (861 native) + 6 000-rule provenance-tracked imported library ([truth table](docs/detections/truth-table.md)) | 1 200+ rules | 1 000+ apps | curated |
| Plugin SDK | Python / TypeScript / Go | YAML rules only | apps | proprietary |
| Data residency | your infra | your infra | partial | vendor cloud |
| Pricing | $0 (self-host) | $0 (self-host) | per ingest GB | enterprise |
@@ -169,7 +169,7 @@ A handful of headline capabilities — the rest are catalogued in [`apps/docs/do
- **69 click-and-connect data connectors** (EDR/XDR, SIEM, cloud, CNAPP, identity, SaaS, VCS, K8s audit, network) with schema-driven config, live `Test connection`, and vault-encrypted secrets. Walkthrough: [`apps/docs/docs/connectors/index.md`](apps/docs/docs/connectors/index.md).
- **Investigation Rail + replayable Investigation Ledger** — every prompt, tool call, evidence chip, and rationale stored against a case, replayable in the UI. [`apps/docs/docs/console/investigation-rail.md`](apps/docs/docs/console/investigation-rail.md).
-- **Detection-as-Code lifecycle** — propose → review → eval-gate → promote; CI rejects any candidate that regresses MITRE accuracy. [`apps/docs/docs/concepts/detections.md`](apps/docs/docs/concepts/detections.md) — and the 800+ native Sigma rules live in [`detections/`](detections/).
+- **Detection-as-Code lifecycle** — propose → review → eval-gate → promote; CI rejects any candidate that fails its own positive/negative fixtures (the non-circular gate) or regresses MITRE accuracy. [`apps/docs/docs/concepts/detections.md`](apps/docs/docs/concepts/detections.md) — and the 861 native rules live in [`detections/`](detections/).
- **L0–L4 automation maturity** — gate every autonomous action on per-action confidence thresholds and blast-radius. [`apps/docs/docs/concepts/automation-maturity.md`](apps/docs/docs/concepts/automation-maturity.md).
- **Hunt-as-Code** — YAML hypotheses with MITRE tags, cron schedules, and natural-language `/hunt` workbench. [`hunts/`](hunts/) + [`apps/docs/docs/console/rule-tuning.md`](apps/docs/docs/console/rule-tuning.md).
- **Public weekly benchmark scoreboard** — the same harness that gates PRs, run weekly against `main`. [`apps/docs/docs/benchmark-scoreboard.mdx`](apps/docs/docs/benchmark-scoreboard.mdx).
diff --git a/ROADMAP.md b/ROADMAP.md
index cdd45e909..3db286369 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -26,7 +26,7 @@ A proof-first, security-first program to make every README claim gate-backed, cl
- [x] Phase 1 — Four existential holes (prompt injection, memory poisoning, cross-store tenant isolation, data-exfiltration/redaction, cost DoS, vault)
- [x] Phase 2 — Supply chain + truth (security.yml scanners, claim-gate ratchet, hard-fail insecure prod defaults, TRADEMARK, verifying-releases, license fix; continuation landed: per-image cosign signatures + CycloneDX SBOM attestations + SLSA provenance, all actions SHA-pinned)
- [x] Phase 3 — Integration / E2E / chaos / DR (real-container spine test, backup-restore, chaos, upgrade, cross-store isolation live-replay; heavy-demo-stack Playwright E2E + demo-timing gate tracked as non-blocking 3.5+)
-- [ ] Phase 4 — Real evals + detection content truth table (third-party-labeled corpus, hallucination/calibration/abstention, model matrix)
+- [ ] Phase 4 — Real evals + detection content truth table (third-party-labeled corpus, hallucination/calibration/abstention, model matrix) — **4a/4b landed** (de-circularised DAC candidate-rule gate + honest executable-vs-imported truth table); 4c+ live-agent eval / calibration / model matrix pending
- [ ] Phase 5 — Data spine correctness (schema registry, DLQ, idempotency, watermarking, backfill/replay, lineage)
- [ ] Phase 6 — Performance + cost (throughput harness, $/TB, storage-consolidation ADR)
- [ ] Phase 7 — Ingest-time graph + multi-model router (posture, effective permissions, bi-temporal, fusion-time ContextBundle, determinism contract)
diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.md
index d54429f5c..5b6426c7b 100644
--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md
+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md
@@ -17,9 +17,9 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
| Public eval harness gates every PR | README L62, L77 | `ci.yml :: p1-eval` | GATED (but suites are self-consistency; see reality report) | Phase 4 |
| Alert-reduction is a real measurement | README L62 | `ci.yml :: p1-eval` (`alert_reduction`) | PARTIAL (gates an in-test fusion re-impl, not `services/fusion`) | Phase 4 |
| Runs entirely on your infrastructure / no data exfiltration | README L63 | `ci.yml :: python-test` (agents) runs `test_privacy_redactor.py` (zero raw PII survives) | PARTIAL (redaction gated + README made precise per mode; air-gapped egress-blocked CI + Helm NetworkPolicy in continuation/Phase 2) | Phase 2 |
-| Detection-as-Code rejects candidates that regress MITRE accuracy | README L170 | `ci.yml :: p1-eval` (w2-dac baseline) | PARTIAL (circular: candidate `rule_body` never evaluated) | Phase 4 |
+| Detection-as-Code rejects candidates that regress MITRE accuracy | README L170 | `ci.yml :: python-test` (`test_detection_eval.py` — candidate `rule_body` run through the real engine vs its own positive/negative fixtures; approval requires it) + `p1-eval` w2-dac baseline | GATED | - |
| 800+ native detection rules | README L78, L170 | `validate-detections.yml` (strict fixture replay) | GATED | - |
-| 6000+ imported detection rules | README L78 | `validate-detections.yml` (parse/provenance) | PARTIAL (97% quarantined/non-executable; heatmap is tag-based) | Phase 4 Tier 3 |
+| 6000+ imported detection rules | README L78 | `validate-detections.yml` (parse/provenance + `detection_truth_table.py --check`); README cites the executable figure (939) not the on-disk figure for coverage | GATED | - |
| L0-L4 automation maturity gates every action | README L171 | `ci.yml` autonomy-policy drift test | PARTIAL (thresholds gated; rollback/post-verify/dry-run ungated) | Phase 9 |
| Hunt-as-Code + `/hunt` | README L172 | `ci.yml :: p1-eval` (`hunt_corpus`) | GATED | - |
| Weekly benchmark scoreboard runs live against `main` | README L173 | `wet-eval.yml` (weekly) | NO GATE (no-ops without secret; live-agent tables are placeholders) | Phase 4 Tier 1 |
@@ -38,9 +38,9 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
## Summary
-- GATED: 12
-- PARTIAL: 12
-- NO GATE: 3 (Phase 2 moved insecure-defaults-hard-fail to GATED and secret/IaC/container scanning to PARTIAL; the Phase 2 continuation moved signed/attested releases to GATED; Phase 3.4 moved cross-store isolation to GATED via the live-container replay; remaining NO GATE close in Phases 4/10/11)
+- GATED: 14
+- PARTIAL: 10
+- NO GATE: 3 (Phase 2 moved insecure-defaults-hard-fail to GATED and secret/IaC/container scanning to PARTIAL; the Phase 2 continuation moved signed/attested releases to GATED; Phase 3.4 moved cross-store isolation to GATED via the live-container replay; Phase 4 moved the DAC candidate-rule gate and the imported-count honesty gate to GATED; remaining NO GATE close in Phases 4-continuation/10/11)
The ratchet is enforced by `scripts/check_claim_gate_matrix.py` (wired into `security.yml`): the NO GATE count may only decrease.
diff --git a/docs/detections/truth-table.md b/docs/detections/truth-table.md
new file mode 100644
index 000000000..147ac1ead
--- /dev/null
+++ b/docs/detections/truth-table.md
@@ -0,0 +1,48 @@
+# Detection content — truth table
+
+> Generated by `scripts/detection_truth_table.py`. `--check` gates this
+> file in `validate-detections.yml`, so the numbers below can never quietly
+> diverge from what the engine actually runs. **Do not edit by hand.**
+
+A rule is **executable** when it fires in AiSOC today: not quarantined
+(`_quarantine/` or `enabled: false`) and its `detection` body is the native
+AiSOC condition DSL, a Sigma selection/condition, or a runtime-engine language.
+A rule is **non-executable** when quarantined or when its only body is an
+untranslated upstream language (SPL / YARA-L / CAR pseudocode) — present for
+provenance and coverage-mapping, not firing.
+
+## Headline
+
+| metric | count |
+|--------|------:|
+| rules on disk (total) | 6975 |
+| **executable (fire today)** | **939** |
+| non-executable (provenance/coverage only) | 6036 |
+
+## By tier
+
+| tier | on disk | executable |
+|------|--------:|-----------:|
+| car (imported) | 99 | 0 |
+| chronicle (imported) | 877 | 0 |
+| community | 1 | 1 |
+| native | 861 | 861 |
+| sigma (imported) | 3132 | 77 |
+| splunk (imported) | 2005 | 0 |
+
+## Why rules are non-executable
+
+| reason | count |
+|--------|------:|
+| `enabled: false` | 115 |
+| under `_quarantine/` (untranslated on import) | 5921 |
+
+## How to read the README claim
+
+The imported corpus is large (6975 rules on disk) and valuable as a
+provenance-tracked ATT&CK-mapped library, but the number that matters
+operationally is **939 executable rules** — the ones the engine
+fires against live telemetry. The README and marketplace must cite the
+executable figure when describing detection *coverage*, and may cite the
+on-disk figure only when explicitly describing the imported *library*.
+
diff --git a/scripts/detection_truth_table.py b/scripts/detection_truth_table.py
new file mode 100644
index 000000000..985570297
--- /dev/null
+++ b/scripts/detection_truth_table.py
@@ -0,0 +1,256 @@
+#!/usr/bin/env python3
+"""Detection content truth table (Phase 4 — honest coverage).
+
+The Phase 0 reality audit's overclaim #2: the README advertises "6000+ imported
+detection rules", but ~97% of the imported set lives under `_quarantine/`
+(`enabled: false`) because its upstream query language (SPL / YARA-L / CAR
+pseudocode) does not execute on the AiSOC engine. The ATT&CK heatmap counts
+metadata tags, not rules that fire.
+
+This script walks `detections/` and classifies every rule as **executable**
+(fires in AiSOC today) or **non-executable** (present for provenance /
+coverage-mapping only), then renders an honest breakdown to
+`docs/detections/truth-table.md`. `--check` fails CI when the committed doc
+drifts from the on-disk reality — so the headline number can never quietly
+diverge from what actually runs.
+
+A rule is EXECUTABLE when it is not quarantined (not under `_quarantine/`, not
+`enabled: false`) AND its `detection` body is in a form the engine evaluates:
+the native AiSOC `condition` DSL, a Sigma `selection`/`condition`, or one of the
+runtime-engine languages. A rule is NON-EXECUTABLE when quarantined or when its
+only body is an untranslated upstream language (`splunk_spl`, `chronicle_yaral`,
+CAR pseudocode).
+
+Usage:
+ python3 scripts/detection_truth_table.py # regenerate the doc
+ python3 scripts/detection_truth_table.py --check # fail on drift
+ python3 scripts/detection_truth_table.py --json # print counts as JSON
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from dataclasses import dataclass, field
+from pathlib import Path
+
+import yaml
+
+ROOT = Path(__file__).resolve().parent.parent
+DETECTIONS = ROOT / "detections"
+DOC = ROOT / "docs" / "detections" / "truth-table.md"
+
+# Directories under detections/ that are not rules.
+SKIP_DIRS = {"fixtures", "playbooks"}
+
+NATIVE_CATEGORIES = {"endpoint", "cloud", "identity", "network", "application", "data-exfil"}
+
+# detection-body keys that the AiSOC engine can actually evaluate.
+EXECUTABLE_BODY_KEYS = {"condition", "selection", "sigma", "yara", "kql", "eql", "lucene", "regex", "query", "keywords"}
+# untranslated upstream query languages — present for provenance, do not fire.
+NON_EXECUTABLE_BODY_KEYS = {"splunk_spl", "chronicle_yaral", "yaral", "spl", "car_pseudocode", "car"}
+
+
+@dataclass
+class Counts:
+ total: int = 0
+ unparseable: int = 0
+ executable: int = 0
+ non_executable: int = 0
+ by_tier: dict[str, int] = field(default_factory=dict)
+ executable_by_tier: dict[str, int] = field(default_factory=dict)
+ non_exec_reason: dict[str, int] = field(default_factory=dict)
+
+ def bump(self, d: dict[str, int], key: str) -> None:
+ d[key] = d.get(key, 0) + 1
+
+
+def _tier_for(path: Path) -> str:
+ rel = path.relative_to(DETECTIONS)
+ top = rel.parts[0]
+ if top in NATIVE_CATEGORIES:
+ return "native"
+ if top.endswith("-imports"):
+ return top[: -len("-imports")] + " (imported)"
+ if top == "community":
+ return "community"
+ return top
+
+
+def _is_quarantined(path: Path, data: dict) -> tuple[bool, str]:
+ if "_quarantine" in path.parts:
+ return True, "quarantine_dir"
+ enabled = data.get("enabled")
+ if enabled is False:
+ return True, "disabled"
+ return False, ""
+
+
+def _has_executable_body(data: dict) -> bool:
+ det = data.get("detection")
+ if not isinstance(det, dict):
+ return False
+ keys = set(det.keys())
+ if keys & EXECUTABLE_BODY_KEYS:
+ return True
+ # A body that is only an untranslated upstream language does not fire.
+ if keys & NON_EXECUTABLE_BODY_KEYS:
+ return False
+ # Unknown shape — treat as non-executable so the honest count never
+ # over-reports what fires.
+ return False
+
+
+def _iter_rule_files() -> list[Path]:
+ out: list[Path] = []
+ for path in sorted(DETECTIONS.rglob("*.yaml")):
+ rel = path.relative_to(DETECTIONS)
+ if rel.parts[0] in SKIP_DIRS:
+ continue
+ if path.name.lower() in {"readme.yaml", "index.yaml"}:
+ continue
+ out.append(path)
+ return out
+
+
+def compute() -> Counts:
+ counts = Counts()
+ for path in _iter_rule_files():
+ counts.total += 1
+ try:
+ data = yaml.safe_load(path.read_text(encoding="utf-8"))
+ except (yaml.YAMLError, OSError):
+ counts.unparseable += 1
+ counts.non_executable += 1
+ counts.bump(counts.non_exec_reason, "unparseable")
+ continue
+ if not isinstance(data, dict):
+ counts.unparseable += 1
+ counts.non_executable += 1
+ counts.bump(counts.non_exec_reason, "unparseable")
+ continue
+
+ tier = _tier_for(path)
+ counts.bump(counts.by_tier, tier)
+
+ quarantined, reason = _is_quarantined(path, data)
+ if quarantined:
+ counts.non_executable += 1
+ counts.bump(counts.non_exec_reason, reason)
+ continue
+ if _has_executable_body(data):
+ counts.executable += 1
+ counts.bump(counts.executable_by_tier, tier)
+ else:
+ counts.non_executable += 1
+ counts.bump(counts.non_exec_reason, "untranslated_upstream_language")
+ return counts
+
+
+def render_markdown(c: Counts) -> str:
+ lines: list[str] = []
+ lines.append("# Detection content — truth table")
+ lines.append("")
+ lines.append("> Generated by `scripts/detection_truth_table.py`. `--check` gates this")
+ lines.append("> file in `validate-detections.yml`, so the numbers below can never quietly")
+ lines.append("> diverge from what the engine actually runs. **Do not edit by hand.**")
+ lines.append("")
+ lines.append("A rule is **executable** when it fires in AiSOC today: not quarantined")
+ lines.append("(`_quarantine/` or `enabled: false`) and its `detection` body is the native")
+ lines.append("AiSOC condition DSL, a Sigma selection/condition, or a runtime-engine language.")
+ lines.append("A rule is **non-executable** when quarantined or when its only body is an")
+ lines.append("untranslated upstream language (SPL / YARA-L / CAR pseudocode) — present for")
+ lines.append("provenance and coverage-mapping, not firing.")
+ lines.append("")
+ lines.append("## Headline")
+ lines.append("")
+ lines.append("| metric | count |")
+ lines.append("|--------|------:|")
+ lines.append(f"| rules on disk (total) | {c.total} |")
+ lines.append(f"| **executable (fire today)** | **{c.executable}** |")
+ lines.append(f"| non-executable (provenance/coverage only) | {c.non_executable} |")
+ lines.append("")
+ lines.append("## By tier")
+ lines.append("")
+ lines.append("| tier | on disk | executable |")
+ lines.append("|------|--------:|-----------:|")
+ for tier in sorted(c.by_tier):
+ lines.append(f"| {tier} | {c.by_tier[tier]} | {c.executable_by_tier.get(tier, 0)} |")
+ lines.append("")
+ lines.append("## Why rules are non-executable")
+ lines.append("")
+ lines.append("| reason | count |")
+ lines.append("|--------|------:|")
+ reason_labels = {
+ "quarantine_dir": "under `_quarantine/` (untranslated on import)",
+ "disabled": "`enabled: false`",
+ "untranslated_upstream_language": "body is an untranslated upstream language",
+ "unparseable": "unparseable YAML",
+ }
+ for reason in sorted(c.non_exec_reason):
+ label = reason_labels.get(reason, reason)
+ lines.append(f"| {label} | {c.non_exec_reason[reason]} |")
+ lines.append("")
+ lines.append("## How to read the README claim")
+ lines.append("")
+ lines.append(f"The imported corpus is large ({c.total} rules on disk) and valuable as a")
+ lines.append("provenance-tracked ATT&CK-mapped library, but the number that matters")
+ lines.append(f"operationally is **{c.executable} executable rules** — the ones the engine")
+ lines.append("fires against live telemetry. The README and marketplace must cite the")
+ lines.append("executable figure when describing detection *coverage*, and may cite the")
+ lines.append("on-disk figure only when explicitly describing the imported *library*.")
+ lines.append("")
+ return "\n".join(lines)
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--check", action="store_true", help="fail if the committed doc is stale")
+ parser.add_argument("--json", dest="as_json", action="store_true", help="print counts as JSON")
+ args = parser.parse_args()
+
+ counts = compute()
+ rendered = render_markdown(counts)
+
+ if args.as_json:
+ print(
+ json.dumps(
+ {
+ "total": counts.total,
+ "executable": counts.executable,
+ "non_executable": counts.non_executable,
+ "unparseable": counts.unparseable,
+ "by_tier": counts.by_tier,
+ "executable_by_tier": counts.executable_by_tier,
+ "non_exec_reason": counts.non_exec_reason,
+ },
+ indent=2,
+ sort_keys=True,
+ )
+ )
+ return 0
+
+ if args.check:
+ if not DOC.exists():
+ print(f"ERROR: {DOC.relative_to(ROOT)} does not exist — run scripts/detection_truth_table.py", file=sys.stderr)
+ return 1
+ current = DOC.read_text(encoding="utf-8")
+ if current.strip() != rendered.strip():
+ print(
+ f"ERROR: {DOC.relative_to(ROOT)} is stale. The detection content changed but the "
+ "truth table was not regenerated. Run: python3 scripts/detection_truth_table.py",
+ file=sys.stderr,
+ )
+ return 1
+ print(f"OK: detection truth table current — {counts.executable} executable / {counts.total} on disk")
+ return 0
+
+ DOC.parent.mkdir(parents=True, exist_ok=True)
+ DOC.write_text(rendered + "\n", encoding="utf-8")
+ print(f"wrote {DOC.relative_to(ROOT)} — {counts.executable} executable / {counts.total} on disk")
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/services/api/app/api/v1/endpoints/detection_proposals.py b/services/api/app/api/v1/endpoints/detection_proposals.py
index cd51e47ac..5c9226cec 100644
--- a/services/api/app/api/v1/endpoints/detection_proposals.py
+++ b/services/api/app/api/v1/endpoints/detection_proposals.py
@@ -1,11 +1,19 @@
"""Detection-as-code proposal lifecycle endpoints (Wave 2 — w2-dac).
Surfaces the propose → review → eval-gated → promote flow that brings
-detections under the same CI gate as agent prompts. The eval gate is
-satisfied by `scripts/run_evals.py` running offline; the result of that
-run is stored back on the proposal as ``eval_result`` and a candidate
-that regresses MITRE accuracy by ≥ ``max_regression_pp`` percentage
-points cannot be promoted.
+detections under the same CI gate as agent prompts.
+
+Two independent gates guard promotion, and approval requires **both**:
+
+* **Candidate-rule gate (Phase 4, non-circular).** ``/evaluate-rule`` runs the
+ *proposed rule body itself* through the runtime engine against caller-supplied
+ positive and negative fixtures — it must fire on every positive and stay
+ silent on every negative. This closes the Phase 0 reality-audit finding that
+ the promote path never evaluated the proposed rule.
+* **Substrate-benchmark gate.** ``/run-eval`` / ``/eval`` run
+ ``scripts/run_evals.py`` and reject a candidate that regresses repo-wide MITRE
+ accuracy by ≥ ``max_regression_pp`` pp. This guards against a rule that passes
+ its own fixtures but perturbs the shared substrate.
Endpoints
---------
@@ -13,7 +21,8 @@ Endpoints
* ``POST /detection-proposals`` Create a proposal.
* ``GET /detection-proposals/{id}`` Proposal detail.
* ``POST /detection-proposals/{id}/comment`` Add a review comment.
-* ``POST /detection-proposals/{id}/eval`` Attach eval result + verdict.
+* ``POST /detection-proposals/{id}/evaluate-rule`` Evaluate the candidate rule vs its fixtures (non-circular gate).
+* ``POST /detection-proposals/{id}/eval`` Attach substrate-benchmark result + verdict.
* ``POST /detection-proposals/{id}/decide`` Approve/reject a proposal.
* ``POST /detection-proposals/{id}/promote`` Materialise into ``detection_rules``.
* ``GET /detection-proposals/baselines`` List eval baselines.
@@ -44,6 +53,7 @@ from app.models.detection_proposal import (
DetectionRuleProposal,
)
from app.models.detection_rule import DetectionRule
+from app.services.detection_eval import evaluate_candidate_rule
from app.services.github import create_detection_pr
router = APIRouter(prefix="/detection-proposals", tags=["detection_rules", "dac"])
@@ -165,6 +175,29 @@ class EvalAttachRequest(BaseModel):
)
+class EvaluateRuleRequest(BaseModel):
+ """Evaluate the candidate rule body against its own fixtures (Phase 4).
+
+ This is the non-circular gate: unlike ``/run-eval`` (which measures the
+ repo-wide substrate benchmark and is independent of the proposed rule),
+ this runs the *proposed rule body itself* through the runtime engine and
+ asserts it fires on every positive fixture and stays silent on every
+ negative one. Approval now requires this verdict to pass.
+ """
+
+ positive_fixtures: list[dict[str, Any]] = Field(
+ ...,
+ min_length=1,
+ max_length=200,
+ description="Events the rule MUST fire on (the attacks it claims to catch). At least one required.",
+ )
+ negative_fixtures: list[dict[str, Any]] = Field(
+ default_factory=list,
+ max_length=200,
+ description="Benign near-miss events the rule MUST NOT fire on.",
+ )
+
+
class DecisionRequest(BaseModel):
decision: Literal["approve", "reject"]
comment: str | None = Field(default=None, max_length=2000)
@@ -643,6 +676,54 @@ async def run_eval_harness(
_ = exc # best-effort cleanup; ignore unlink failures
[email protected]("/{proposal_id}/evaluate-rule", response_model=ProposalResponse)
+async def evaluate_rule(
+ proposal_id: uuid.UUID,
+ request: EvaluateRuleRequest,
+ current_user: Annotated[AuthUser, Depends(require_permission("rules:write"))],
+ db: DBSession,
+) -> ProposalResponse:
+ """Evaluate the candidate rule body against its own fixtures (Phase 4).
+
+ Runs the proposed ``rule_language`` + ``rule_body`` through the runtime
+ engine against the supplied fixtures and stores the verdict under
+ ``eval_result["candidate_rule"]``. This is the gate that de-circularises
+ promotion: a rule that misses its positives (blind) or fires on its
+ negatives (noisy) cannot be approved, regardless of the global benchmark.
+ """
+ _ensure_dac_enabled()
+ proposal = await _load_proposal(db, proposal_id, current_user.tenant_id)
+ if proposal.status in {"promoted", "rejected"}:
+ raise HTTPException(
+ status_code=status.HTTP_409_CONFLICT,
+ detail=f"Proposal is {proposal.status}; candidate-rule eval cannot be re-run",
+ )
+
+ result = evaluate_candidate_rule(
+ rule_language=proposal.rule_language,
+ rule_body=proposal.rule_body,
+ positive_fixtures=request.positive_fixtures,
+ negative_fixtures=request.negative_fixtures,
+ )
+
+ # Merge into eval_result so the benchmark verdict (if already attached) is
+ # preserved alongside the candidate-rule verdict.
+ merged = dict(proposal.eval_result or {})
+ merged["candidate_rule"] = {**result.to_dict(), "ran_at": datetime.now(UTC).isoformat()}
+ proposal.eval_result = merged
+
+ # A candidate-rule failure is decisive — it cannot be approved. On pass,
+ # only advance to eval_passed if the benchmark gate isn't failing.
+ if not result.passed:
+ proposal.status = "eval_failed"
+ elif proposal.status not in {"eval_failed"}:
+ proposal.status = "eval_passed"
+
+ await db.commit()
+ await db.refresh(proposal)
+ return ProposalResponse.model_validate(proposal)
+
+
@router.post("/{proposal_id}/eval", response_model=ProposalResponse)
async def attach_eval_result(
proposal_id: uuid.UUID,
@@ -686,8 +767,21 @@ async def attach_eval_result(
max_regression_pp=request.max_regression_pp,
)
- proposal.eval_result = verdict
- proposal.status = "eval_passed" if verdict["passed"] else "eval_failed"
+ # Preserve any candidate-rule verdict already attached via /evaluate-rule.
+ existing = dict(proposal.eval_result or {})
+ candidate = existing.get("candidate_rule")
+ merged = dict(verdict)
+ if candidate is not None:
+ merged["candidate_rule"] = candidate
+ proposal.eval_result = merged
+
+ # Benchmark pass alone is not sufficient if the candidate rule failed its
+ # own fixtures — the non-circular gate is decisive.
+ candidate_ok = (candidate or {}).get("passed", None)
+ if candidate_ok is False:
+ proposal.status = "eval_failed"
+ else:
+ proposal.status = "eval_passed" if verdict["passed"] else "eval_failed"
await db.commit()
await db.refresh(proposal)
return ProposalResponse.model_validate(proposal)
@@ -710,11 +804,33 @@ async def decide_proposal(
)
if request.decision == "approve":
- eval_passed = bool(proposal.eval_result.get("passed")) if proposal.eval_result else False
- if not eval_passed:
+ eval_result = proposal.eval_result or {}
+ # Phase 4 — de-circularised gate. Approval requires the candidate rule
+ # to have been evaluated against its own fixtures and passed (fires on
+ # positives, silent on negatives). A benchmark-only "pass" is no longer
+ # sufficient: the reality audit showed the benchmark is independent of
+ # the proposed rule, so a blind/noisy rule could sail through.
+ candidate = eval_result.get("candidate_rule")
+ if not candidate:
+ raise HTTPException(
+ status_code=status.HTTP_412_PRECONDITION_FAILED,
+ detail=(
+ "Candidate-rule eval has not been run; cannot approve. "
+ "POST /detection-proposals/{id}/evaluate-rule with positive "
+ "and negative fixtures first."
+ ),
+ )
+ if not bool(candidate.get("passed")):
+ raise HTTPException(
+ status_code=status.HTTP_412_PRECONDITION_FAILED,
+ detail=f"Candidate-rule eval failed ({candidate.get('reason', 'rule did not pass its fixtures')}); cannot approve.",
+ )
+ # If a substrate-benchmark verdict is also attached it must not be a
+ # regression failure.
+ if "passed" in eval_result and not bool(eval_result.get("passed")):
raise HTTPException(
status_code=status.HTTP_412_PRECONDITION_FAILED,
- detail="Eval gate has not passed; cannot approve. Run the eval suite and attach the report first.",
+ detail="Substrate benchmark gate failed (MITRE regression); cannot approve.",
)
proposal.status = "approved"
else:
diff --git a/services/api/app/services/detection_eval.py b/services/api/app/services/detection_eval.py
new file mode 100644
index 000000000..27c737c30
--- /dev/null
+++ b/services/api/app/services/detection_eval.py
@@ -0,0 +1,158 @@
+"""Candidate detection-rule evaluation (Phase 4 — de-circularise the DAC gate).
+
+The Phase 0 reality audit's #1 circular gate: the detection-proposal promote
+path shelled out to ``scripts/run_evals.py`` **without ever passing the proposed
+rule body**. "Passed" meant the repo's global substrate MITRE accuracy did not
+regress — a value entirely independent of the rule under review. A rule that
+matches nothing (blind) or matches everything (noisy) sailed through its own
+exam.
+
+This module closes that hole. It evaluates the **candidate rule body itself**
+against caller-supplied positive and negative fixtures using the *real* runtime
+engine (:func:`app.services.rule_engine.execute_rule`) — the same code that runs
+the rule in production. The contract is the one every detection must satisfy:
+
+* it MUST fire on every positive fixture (a rule that misses the attack it
+ claims to catch is worthless), and
+* it MUST stay silent on every negative fixture (a rule that fires on the
+ benign near-miss is a false-positive factory).
+
+The proposal approve-gate now requires this verdict to pass, so a bad rule is
+rejected on its own merits rather than on an unrelated global metric.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any
+
+from app.services.rule_engine import execute_rule
+
+
+@dataclass(frozen=True)
+class FixtureOutcome:
+ """Per-fixture result: did the candidate rule fire, and was that expected?"""
+
+ index: int
+ kind: str # "positive" | "negative"
+ fired: bool
+ expected_fire: bool
+ ok: bool
+ error: str | None = None
+
+
+@dataclass(frozen=True)
+class CandidateEvalResult:
+ """Outcome of evaluating a candidate rule against its fixtures."""
+
+ passed: bool
+ positives_total: int
+ positives_fired: int
+ negatives_total: int
+ negatives_fired: int
+ outcomes: list[FixtureOutcome] = field(default_factory=list)
+ reason: str = ""
+
+ def to_dict(self) -> dict[str, Any]:
+ return {
+ "passed": self.passed,
+ "positives_total": self.positives_total,
+ "positives_fired": self.positives_fired,
+ "negatives_total": self.negatives_total,
+ "negatives_fired": self.negatives_fired,
+ "reason": self.reason,
+ "outcomes": [
+ {
+ "index": o.index,
+ "kind": o.kind,
+ "fired": o.fired,
+ "expected_fire": o.expected_fire,
+ "ok": o.ok,
+ "error": o.error,
+ }
+ for o in self.outcomes
+ ],
+ }
+
+
+def _fires(rule_language: str, rule_body: str, event: dict[str, Any]) -> tuple[bool, str | None]:
+ """Run the candidate rule against a single event via the real engine."""
+ match = execute_rule(
+ rule_id="candidate",
+ rule_name="candidate",
+ rule_language=rule_language,
+ rule_body=rule_body,
+ severity="medium",
+ events=[event],
+ )
+ return bool(match.matched), match.error
+
+
+def evaluate_candidate_rule(
+ *,
+ rule_language: str,
+ rule_body: str,
+ positive_fixtures: list[dict[str, Any]],
+ negative_fixtures: list[dict[str, Any]],
+) -> CandidateEvalResult:
+ """Evaluate a candidate rule against its own positive/negative fixtures.
+
+ Returns a :class:`CandidateEvalResult`. ``passed`` is True only when the
+ rule fired on *every* positive fixture, stayed silent on *every* negative
+ fixture, and at least one positive fixture was supplied (a rule with no
+ positive fixture has proven nothing — the gate would be vacuous).
+ """
+ outcomes: list[FixtureOutcome] = []
+
+ if not positive_fixtures:
+ return CandidateEvalResult(
+ passed=False,
+ positives_total=0,
+ positives_fired=0,
+ negatives_total=len(negative_fixtures),
+ negatives_fired=0,
+ reason="no positive fixtures supplied — cannot prove the rule catches anything",
+ )
+
+ positives_fired = 0
+ for i, event in enumerate(positive_fixtures):
+ fired, err = _fires(rule_language, rule_body, event)
+ if fired:
+ positives_fired += 1
+ outcomes.append(
+ FixtureOutcome(index=i, kind="positive", fired=fired, expected_fire=True, ok=fired, error=err)
+ )
+
+ negatives_fired = 0
+ for i, event in enumerate(negative_fixtures):
+ fired, err = _fires(rule_language, rule_body, event)
+ if fired:
+ negatives_fired += 1
+ outcomes.append(
+ FixtureOutcome(index=i, kind="negative", fired=fired, expected_fire=False, ok=not fired, error=err)
+ )
+
+ fired_on_all_positives = positives_fired == len(positive_fixtures)
+ silent_on_all_negatives = negatives_fired == 0
+ passed = fired_on_all_positives and silent_on_all_negatives
+
+ n_pos = len(positive_fixtures)
+ n_neg = len(negative_fixtures)
+ if passed:
+ reason = "rule fired on all positives and stayed silent on all negatives"
+ elif not fired_on_all_positives and not silent_on_all_negatives:
+ reason = f"rule is both blind ({positives_fired}/{n_pos} positives) and noisy ({negatives_fired}/{n_neg} negatives fired)"
+ elif not fired_on_all_positives:
+ reason = f"rule is blind: fired on only {positives_fired}/{n_pos} positive fixtures"
+ else:
+ reason = f"rule is noisy: fired on {negatives_fired}/{n_neg} negative fixtures (must be 0)"
+
+ return CandidateEvalResult(
+ passed=passed,
+ positives_total=len(positive_fixtures),
+ positives_fired=positives_fired,
+ negatives_total=len(negative_fixtures),
+ negatives_fired=negatives_fired,
+ outcomes=outcomes,
+ reason=reason,
+ )
diff --git a/services/api/tests/test_detection_eval.py b/services/api/tests/test_detection_eval.py
new file mode 100644
index 000000000..287c1e6fc
--- /dev/null
+++ b/services/api/tests/test_detection_eval.py
@@ -0,0 +1,139 @@
+"""Phase 4 — the anti-circularity gate for detection-rule promotion.
+
+The Phase 0 reality audit's #1 circular gate: the detection-proposal promote
+path shelled out to the eval harness **without ever passing the proposed rule
+body**, so a rule that matches nothing (blind) or matches everything (noisy)
+passed its own exam. `app.services.detection_eval.evaluate_candidate_rule`
+closes that hole by running the *candidate rule body itself* through the real
+runtime engine (`app.services.rule_engine.execute_rule`) against its fixtures.
+
+This suite is the mutation test that proves the gate is no longer circular: a
+deliberately-broken rule is REJECTED, and a correct rule is accepted. It uses
+the real engine, so the verdict genuinely depends on the rule body.
+"""
+
+from __future__ import annotations
+
+from app.services.detection_eval import evaluate_candidate_rule
+
+# A correct Sigma rule: fires on a root console login, silent on an IAM-user
+# login. Evaluated by the runtime engine's Sigma path (pySigma if present, else
+# the substring fallback — the assertions hold for both).
+GOOD_RULE = """
+title: AWS Root Console Login
+logsource:
+ product: aws
+ service: cloudtrail
+detection:
+ selection:
+ event_name: ConsoleLogin
+ user_type: Root
+ condition: selection
+"""
+
+POSITIVE = {"event_name": "ConsoleLogin", "user_type": "Root", "src_ip": "203.0.113.7"}
+NEGATIVE = {"event_name": "ConsoleLogin", "user_type": "IAMUser", "src_ip": "203.0.113.7"}
+
+
+def test_good_rule_passes_its_own_fixtures():
+ result = evaluate_candidate_rule(
+ rule_language="sigma",
+ rule_body=GOOD_RULE,
+ positive_fixtures=[POSITIVE],
+ negative_fixtures=[NEGATIVE],
+ )
+ assert result.passed, result.reason
+ assert result.positives_fired == 1
+ assert result.negatives_fired == 0
+
+
+def test_blind_rule_is_rejected():
+ """A rule that never fires on its positive fixture must fail — the core
+ circular-gate refutation: a blind rule no longer 'passes'."""
+ blind = """
+title: Never Matches Anything
+logsource:
+ product: aws
+detection:
+ selection:
+ event_name: EventThatNeverOccursZZZ
+ condition: selection
+"""
+ result = evaluate_candidate_rule(
+ rule_language="sigma",
+ rule_body=blind,
+ positive_fixtures=[POSITIVE],
+ negative_fixtures=[NEGATIVE],
+ )
+ assert not result.passed
+ assert result.positives_fired == 0
+ assert "blind" in result.reason
+
+
+def test_noisy_rule_is_rejected():
+ """A rule that also fires on its negative (benign) fixture must fail."""
+ noisy = """
+title: Fires On Any Console Login
+logsource:
+ product: aws
+detection:
+ selection:
+ event_name: ConsoleLogin
+ condition: selection
+"""
+ result = evaluate_candidate_rule(
+ rule_language="sigma",
+ rule_body=noisy,
+ positive_fixtures=[POSITIVE],
+ negative_fixtures=[NEGATIVE],
+ )
+ assert not result.passed
+ assert result.negatives_fired == 1
+ assert "noisy" in result.reason
+
+
+def test_no_positive_fixtures_is_rejected_not_vacuous():
+ """A rule with no positive fixture has proven nothing — it must not pass."""
+ result = evaluate_candidate_rule(
+ rule_language="sigma",
+ rule_body=GOOD_RULE,
+ positive_fixtures=[],
+ negative_fixtures=[NEGATIVE],
+ )
+ assert not result.passed
+ assert result.positives_total == 0
+ assert "no positive" in result.reason.lower()
+
+
+def test_verdict_is_json_serialisable_with_per_fixture_detail():
+ result = evaluate_candidate_rule(
+ rule_language="sigma",
+ rule_body=GOOD_RULE,
+ positive_fixtures=[POSITIVE],
+ negative_fixtures=[NEGATIVE],
+ )
+ d = result.to_dict()
+ assert d["passed"] is True
+ assert len(d["outcomes"]) == 2
+ kinds = {o["kind"] for o in d["outcomes"]}
+ assert kinds == {"positive", "negative"}
+ assert all(o["ok"] for o in d["outcomes"])
+
+
+def test_the_gate_verdict_depends_on_the_rule_body():
+ """The property the circular gate lacked: two different rule bodies against
+ the *same* fixtures yield different verdicts. If the verdict were
+ independent of the rule (the old bug), these would be identical."""
+ good = evaluate_candidate_rule(
+ rule_language="sigma",
+ rule_body=GOOD_RULE,
+ positive_fixtures=[POSITIVE],
+ negative_fixtures=[NEGATIVE],
+ )
+ bad = evaluate_candidate_rule(
+ rule_language="sigma",
+ rule_body="title: X\ndetection:\n selection:\n event_name: NopeZZZ\n condition: selection\n",
+ positive_fixtures=[POSITIVE],
+ negative_fixtures=[NEGATIVE],
+ )
+ assert good.passed and not bad.passed