Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
fix: retract the 75.3% alert-reduction figure everywhere, and gate the numbers (#874)
Beenu Arora908df8a44f07
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex a949a318e..7cdfa7188 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -368,6 +368,30 @@ jobs:# commit the result.- name: Verify connector count is in sync (Phase 1.1)run: python scripts/generate_connector_count.py --check+ # Same shape, for the detection / marketplace figures. The landing page+ # carried three different wrong answers to "how many detections" — 6,998+ # in four places, 7,117 for the marketplace, 218 on the contributor+ # leaderboard — with no gate on any of them. It also quoted the imported+ # corpus as the detection capability, while ~85% of it sits under+ # `_quarantine/` and is never loaded.+ #+ # The self-test hand-edits the generated artefact and requires the drift+ # to be caught, so a green check means the comparison still happens.+ - name: Corpus stats — self-test+ run: python3 scripts/generate_corpus_stats.py --self-test+ - name: Verify detection / marketplace counts are in sync+ run: python3 scripts/generate_corpus_stats.py --check+ # Prose cannot be generated the way a count can, so the retraction of+ # the 75.3% alert-reduction figure is gated instead. `benchmark.md`+ # withdrew the claim; `BenchmarkResults.tsx` went on rendering a "Real+ # measurement" badge on it and `benchmarks/alert-reduction.md` went on+ # calling the harness a faithful re-implementation of production+ # grouping. A correction that reaches one surface of three is not a+ # correction.+ - name: Alert-reduction claims — self-test+ run: python3 scripts/check_alert_reduction_claims.py --self-test+ - name: Verify the retracted alert-reduction figure stays retracted+ run: python3 scripts/check_alert_reduction_claims.py --check# The count comes from the registry; so, now, does the console's# `ConnectorType` union. It used to be written by hand, which is how ten# members came to name nothing the platform ingests — `ibm_qradar`diff --git a/CHANGELOG.md b/CHANGELOG.mdindex ea749a2ae..57a1b82bc 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,84 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]+### Fixed++- **A retracted benchmark figure was still badged "Real measurement".**+ `apps/docs/docs/benchmark.md` withdrew the 75.3 % alert-reduction claim —+ the harness that produced it groups on four tiers of `(rule_id, host,+ user)` while the shipping `RawAlert.correlation_key()` groups on+ `{tenant}:{entity}:{tactic}`, so it does not merely re-implement fusion's+ grouping, it implements *different* grouping. The retraction reached one+ surface of five. `BenchmarkResults.tsx` rendered a green **"Real+ measurement"** badge on `0.753` with a blurb claiming the harness used the+ production rules, "same logic"; `benchmarks/alert-reduction.md`+ (`sidebar_position: 1`) called it a "faithful in-harness re-implementation"+ and headlined 75.3 %; `ComparisonTable.tsx` qualified it as "measured on+ fixed noisy stream". Two more were found while gating the invariant:+ `benchmark-methodology.md`, and `benchmark.md` itself, which re-asserted+ the claim in its own intro blockquote. All five now carry the wording+ `benchmark.md` already uses. The comparison table quotes **33.3 %**, the+ figure measured against the key the product actually runs.+- **The landing page published three different wrong corpus counts.**+ "6,998 detections" in four places (the tree indexes 7,016, of which 5,937+ are quarantined and the engine loads **833**), "57 plugins" (77), "7,117+ community items" (7,155), and `218 rules across 5 categories` on the+ contributor leaderboard (833 across 6). The 6,998 figure also quoted the+ imported corpus as the detection capability, presenting quarantined rules+ as executable.+- **Governance figures had gone stale.** `ROADMAP.md` published "136 rows —+ 128 GATED / 8 PARTIAL" against a matrix holding 139 / 131, in the same+ sentence that tells the reader to recount with the script "rather than+ trusting a figure quoted in prose — this line has gone stale before".+ `CLAIM_TO_GATE_MATRIX.md` carried a stale executable-rule figure (939) in+ a row note. The scoreboard's newest substrate row was labelled `v8.1.1`+ while `VERSION` read `10.0.0`; it is refreshed from a fresh deterministic+ run (0.97, unchanged) and now carries the tree's version. The scoreboard+ keeps its three rows, all `substrate: true`, and no live-LLM row was+ invented.+- **The "Design partners" block on the landing page was removed** rather+ than updated. Four dashed "Partner A–D" chips under the caption+ "Reference partners onboarding through Q2 2026": placeholders rather than+ fabricated logos, but four of them assert a partner count nothing in the+ repository supports, and the window closed in June 2026 while still being+ advertised as upcoming.++### Added++- **`scripts/generate_corpus_stats.py`** — generates+ `apps/web/src/data/corpus-stats.json` + `corpusStats.ts` from the compiled+ engine ruleset, the generated detection truth table, and the marketplace+ index, reconciling all three against each other and refusing to publish if+ they disagree. Every landing surface imports the constants, so none can+ carry its own literal. `--check` is wired into `ci.yml :: python-lint`;+ `--self-test` hand-edits the artefact and requires the drift to be caught.+ The artefact keeps `executable` and `onDisk`/`quarantined` as separate+ fields, and the UI leads with the executable count.+- **`scripts/check_alert_reduction_claims.py`** — prose cannot be generated+ the way a count can, so the retraction is gated instead. No published+ surface may assert the legacy harness runs the production grouping; any+ surface quoting 75.3 % must carry the retraction; the `alert_reduction`+ suite card may not be declared `kind: 'measurement'`; and every surface+ publishing the real figure must quote `PUBLISHED_REDUCTION_PCT`, a new+ constant in `services/fusion/tests/test_alert_reduction_real.py` that the+ test asserts against its own measurement — so the chain from measurement+ to published prose has no hand-copied link. A paragraph that dates or+ negates the claim is exempt, so the retraction can quote the wording it+ retracts.++### Changed++- **`readme_gates.py` covers the governance documents.** Its `FIGURE_DOCS`+ list named one compliance page, which is why `ROADMAP.md` drifted with CI+ green. `ROADMAP.md` and `RELEASES.md` are now on the list, the matrix+ **row total** is compared as well as the GATED/PARTIAL split, and a figure+ the prose explicitly dates ("the count at that time") is exempt so history+ need not be rewritten.+- **`check_scoreboard.py --check` verifies `agent_version` against+ `VERSION`.** `--refresh` already stamped it; nothing compared it, and the+ freshness gate reads only the date — which a refresh keeps current — so a+ row could be two days old and still labelled two majors behind.+## [10.0.0] — 2026-09-25**Two ways for a control to be absent: not written, or written and notdiff --git a/RELEASES.md b/RELEASES.mdindex c3b7d29fd..fb9af3fe7 100644--- a/RELEASES.md+++ b/RELEASES.md@@ -2,7 +2,7 @@This file mirrors what used to live in the "What's new" section of [`README.md`](README.md). The complete, machine-readable inventory (with file paths, env-var diffs, and per-release test counts) lives in [`CHANGELOG.md`](CHANGELOG.md).-> **TL;DR for first-time visitors:** AiSOC is on `v10.0.0`, released 2026-09-25 — a release about controls that were written and not reachable. Row-level security covered 92 tables and filtered nothing, because every service connected to Postgres as a superuser; fifty-eight routes carried no authentication at all; UEBA reported healthy and could not write an anomaly; AI triage was wired to a gateway both of its resolvers ignored. **It is a major because upgrading requires action** — services now connect as a DML-only `aisoc_app` role. Every product claim is backed by a failing CI test (claim-to-gate matrix: 139 rows — 131 GATED / 8 PARTIAL / 0 NO GATE). The latest GitHub release with notes and downloads: <https://github.com/beenuar/AiSOC/releases/latest>.+> **TL;DR for first-time visitors:** AiSOC is on `v10.0.0`, released 2026-09-25 — a release about controls that were written and not reachable. Row-level security covered 92 tables and filtered nothing, because every service connected to Postgres as a superuser; fifty-eight routes carried no authentication at all; UEBA reported healthy and could not write an anomaly; AI triage was wired to a gateway both of its resolvers ignored. **It is a major because upgrading requires action** — services now connect as a DML-only `aisoc_app` role. Every product claim is backed by a failing CI test (claim-to-gate matrix: 142 rows — 134 GATED / 8 PARTIAL / 0 NO GATE). The latest GitHub release with notes and downloads: <https://github.com/beenuar/AiSOC/releases/latest>.---@@ -108,7 +108,7 @@ The full inventory (every file, env-var, and test count) lives under `[8.0.0]` i- **Phase D — breadth.** Eight new connectors — IBM QRadar, Exabeam, Securonix, Devo, Netskope, Windows/Sysmon (WEF), Zeek/Suricata NDR, and a generic syslog/CEF listener (D1); an AI/LLM-usage audit connector + eight `llm-*` detections + hot/cold ClickHouse lake tiering (D2); and a live-vendor mock-server smoke suite that exercises each connector's real HTTP client (D3).- **Phase E — prove it.** The public benchmark scoreboard is now CI-gated against a deterministic live-agent MITRE-accuracy run, closing the last `NO GATE` and ratcheting `MAX_NO_GATE` to 0 (E1).-The full inventory (every file, env-var, and test count) lives under `[7.6.0]` in [`CHANGELOG.md`](CHANGELOG.md). Note that the claim-to-gate figure quoted in the v7.6.0 announcement (33 GATED / 7 PARTIAL) was the count at that time; the matrix has grown to 108 rows since.+The full inventory (every file, env-var, and test count) lives under `[7.6.0]` in [`CHANGELOG.md`](CHANGELOG.md). Note that the claim-to-gate figure quoted in the v7.6.0 announcement (33 GATED / 7 PARTIAL) was the count at that time; the matrix has grown since — recount with `python3 scripts/check_claim_gate_matrix.py` rather than reading a number off this line.---diff --git a/ROADMAP.md b/ROADMAP.mdindex 1ace56766..65cffc439 100644--- a/ROADMAP.md+++ b/ROADMAP.md@@ -38,13 +38,13 @@ A proof-first, security-first program to make every README claim gate-backed, cl**Program status:** all 13 hardening phases (0–12) are landed. Building on them, the **Fully-Operational AI-SOC roadmap** (Phases A1–E1) is now complete — it wired the three end-to-end paths the reality audit found unwired and pushed the platform to competitive parity + beyond:-- **Phase A (SIEM foundation):** A1 ClickHouse lake writer · A2 live detection-evaluation worker (825 executable rules on the stream) · A3 default-on one-command deploy (connectors + graph on) · A4 UEBA behavioral-model fusion.+- **Phase A (SIEM foundation):** A1 ClickHouse lake writer · A2 live detection-evaluation worker (833 executable rules on the stream) · A3 default-on one-command deploy (connectors + graph on) · A4 UEBA behavioral-model fusion.- **Phase B (SOAR foundation):** B1 auto-triage worker (copilot default) · B2 credential resolver + `decide()`-governed live dispatch + 10 vendor adapters · B3 real rollback + post-action verification + durable approval-SLA timers · B4 Business Context Rules in the hot path.- **Phase C (parity differentiators):** C1 Advanced Data Explorer · C2 Effective-Permissions live posture loader · C3 autopilot/copilot posture scorecard · C4 fuse-time attack-chain auto-grouping.- **Phase D (breadth):** D1 eight connectors (QRadar/Exabeam/Securonix/Devo/Netskope/Windows-Sysmon/Zeek-Suricata/syslog-CEF) · D2 AI/LLM-usage audit connector + `llm-*` detections + hot/cold lake tiering · D3 live-vendor mock-server smoke.- **Phase E (prove it):** E1 CI-gated benchmark scoreboard tied to a deterministic live-agent MITRE-accuracy run.-The claim-to-gate matrix stands at **136 rows — 128 GATED / 8 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test** and the ratchet (`MAX_NO_GATE=0`) forbids any regression. The 8 remaining PARTIAL rows are honest, named deferrals; none was relabelled without building the gate it names. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).+The claim-to-gate matrix stands at **142 rows — 134 GATED / 8 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test** and the ratchet (`MAX_NO_GATE=0`) forbids any regression. The 8 remaining PARTIAL rows are honest, named deferrals; none was relabelled without building the gate it names. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).## v4.0 — Shippeddiff --git a/apps/docs/docs/benchmark-methodology.md b/apps/docs/docs/benchmark-methodology.mdindex f96119705..7af5ee3e9 100644--- a/apps/docs/docs/benchmark-methodology.md+++ b/apps/docs/docs/benchmark-methodology.md@@ -98,9 +98,12 @@ it here in detail.The substrate suite drives deterministic code only:- **Keyword extractors** — pulled out of `services/agents/app/extractors/`-- **Fusion grouping** — a faithful in-harness re-implementation of the- Tier 1/2/3 rules used by `services/fusion`, minus the DB-backed dedup- and the ML scorer+- **Fusion grouping** — a four-tier scheme implemented inside the test,+ keyed on `(rule_id, host, user)`. It is **not** the grouping+ `services/fusion` runs, which keys on `{tenant}:{entity}:{tactic}`;+ it is retained as a regression gate and its number does not describe+ this product. See [Why there are two alert-reduction+ numbers](./benchmark.md#why-there-are-two-alert-reduction-numbers)- **Report and plan templates** — the deterministic synthesisers used asfall-back when an LLM call fails- **Offline judges** — keyword-coverage and rubric-score checks@@ -150,7 +153,8 @@ The summary, with class labels:| Suite | Class | Headline metric ||-------|-------|-----------------|| `mitre_accuracy` | Substrate self-check | Per-template macro accuracy of the keyword extractor |-| `alert_reduction` | Real measurement | 1 000-alert noisy stream → fused incident count |+| `alert_reduction` | Legacy — does not describe this product | 1 000-alert noisy stream → four-tier in-test grouping, retained as a regression gate ([why](./benchmark.md#why-there-are-two-alert-reduction-numbers)) |+| alert reduction (product logic) | Real measurement | The same stream grouped with `RawAlert.correlation_key()`, the method `Correlator` calls || `investigation_completeness` | Substrate self-check | Mean keyword coverage of the report template || `response_quality` | Substrate self-check | Mean rubric score of the synthesised plan || `playbook_completion_rate` | Operational coverage gate | Fraction of in-scope incidents with a matched playbook |diff --git a/apps/docs/docs/benchmark.md b/apps/docs/docs/benchmark.mdindex 75c5dccaf..d3a27eca2 100644--- a/apps/docs/docs/benchmark.md+++ b/apps/docs/docs/benchmark.md@@ -39,9 +39,11 @@ agent performance**. Every table below is labelled with its class.>> This page is _not_ a leaderboard for AI SOC agents. It is a CI-gated harness> that exercises the deterministic substrate underneath AiSOC — the keyword-> extractors, the in-harness fusion grouping (a faithful re-implementation of-> the production Tier 1/2/3 logic in `services/fusion`, minus the DB-backed-> dedup and ML scoring), the report and response templates, and the offline+> extractors, the in-harness fusion grouping (a four-tier scheme implemented+> inside the test, which groups on different dimensions from `services/fusion`+> and is retained for continuity — see [Why there are two alert-reduction+> numbers](#why-there-are-two-alert-reduction-numbers)), the report and+> response templates, and the offline> judges that grade them. The dataset, the harness, and the CI gate are all in> the repo. You can reproduce every number on this page in under 10 seconds on> a laptop.@@ -512,29 +514,38 @@ python3 scripts/run_evals.py --ci --out report.json## What each suite actually measures-### 1. Alert reduction ratio — `Real measurement`+### 1. Alert reduction ratio — two numbers-**Source:** [`services/agents/tests/test_alert_reduction.py`](https://github.com/beenuar/AiSOC/blob/main/services/agents/tests/test_alert_reduction.py)+**Product logic:** [`services/fusion/tests/test_alert_reduction_real.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/tests/test_alert_reduction_real.py)+groups the stream with `RawAlert.correlation_key()` — `{tenant}:{entity}:{tactic}`+over a one-hour window, the method `Correlator` actually calls — and reports+**33.3 %**. `Correlator` itself needs Redis, but Redis is where it *stores*+incidents; the grouping decision is the key plus the window, so the ratio is+computed from the real method without the storage layer. The gate is bounded+on both sides, because a floor alone is satisfied by a key that collapses+everything into one incident.++**Legacy suite:** [`services/agents/tests/test_alert_reduction.py`](https://github.com/beenuar/AiSOC/blob/main/services/agents/tests/test_alert_reduction.py)A 1 000-alert noisy stream — pure duplicates, near-duplicates within a30-minute host window, multi-host rule storms, and benign low-score chatter —-is fed into the in-harness `fuse_alerts` function. That function is a-deterministic, in-memory re-implementation of the same Tier 1/2/3 grouping-rules used by the production `services/fusion` engine — minus the-DB-backed deduplicator and the ML scorer. The grouping logic itself is the-same:+is fed into the in-harness `fuse_alerts` function. Four tiers keyed on+`(rule_id, host, user)`:- **Tier 1** — same `(rule, host, user)` within 10 minutes → 1 incident- **Tier 2** — same `(rule, host)` within 30 minutes → merge into a Tier-1 incident- **Tier 3** — same rule within 5 minutes across ≥ 3 hosts → "storm" incident-Incidents below the noise threshold (`score < 0.35`) are dropped. The output is-whatever the code produces — a fusion-rule regression will move the number.-This is a legitimate measurement of grouping behavior on a controlled dataset,-but it is **not** end-to-end coverage of the production fusion service.+Incidents below the noise threshold (`score < 0.35`) are dropped.-The reported ~75 % is the actual output of the in-harness grouping function on-this fixed dataset. It is not tuned to match a marketing number.+This page previously described that function as a re-implementation of the+production rules "minus the DB-backed deduplicator and the ML scorer", and+said the grouping logic itself was the same. It is not: different dimensions,+different windows, different answer — see [Why there are two alert-reduction+numbers](#why-there-are-two-alert-reduction-numbers). The reported ~75 % is+the honest output of *that* function on this fixed dataset — it is not tuned+to match a marketing number — but it describes an algorithm this product does+not run, and is retained as a regression gate only.### 2. MITRE ATT&CK tactic accuracy — `Substrate self-consistency`diff --git a/apps/docs/docs/benchmarks/alert-reduction.md b/apps/docs/docs/benchmarks/alert-reduction.mdindex 314cef9d6..c31515b06 100644--- a/apps/docs/docs/benchmarks/alert-reduction.md+++ b/apps/docs/docs/benchmarks/alert-reduction.md@@ -1,47 +1,61 @@---sidebar_position: 1title: Alert Reduction-description: How the AiSOC alert-fusion substrate collapses a noisy 1,000-alert stream into ~247 incidents while losing zero critical alerts. Methodology, latest measured numbers, reproduction, and CI artefact location.+description: Two alert-reduction numbers, why both are published, and which one describes AiSOC. Methodology, the retraction of the legacy 75.3% figure, reproduction, and CI artefact location.---# Alert Reduction> **Synthetic, deterministic, reproducible in under a second.** This page-> documents the AiSOC alert-reduction benchmark — what it measures, how the-> 3-tier fusion logic collapses a noisy alert stream into actionable-> incidents, and how to reproduce the numbers locally.+> documents the AiSOC alert-reduction benchmark — what it measures, how a+> noisy alert stream collapses into actionable incidents, and how to+> reproduce the numbers locally.>-> The headline number on the [Public Eval Harness](../benchmark.md) page-> (`75.3 %`) is the output of this benchmark. This page is the long-form-> companion: the dataset shape, the per-tier collapse breakdown, the-> critical-alert preservation guarantee, and the road to per-rule false-positive-> rates (tracked under [issue #5](https://github.com/beenuar/AiSOC/issues/5)).+> **Two numbers are published, and only one of them describes this product.**+> **33.3 %** is measured against `RawAlert.correlation_key()`, the grouping+> `Correlator` actually calls. **75.3 %** comes from a four-tier scheme+> implemented inside a test; it is retained for continuity and **does not+> describe this product**. See [Why there are two+> numbers](#why-there-are-two-numbers) before quoting either.[](#latest-results):::warning Read this first-This is a **synthetic** benchmark. The 1,000-alert stream is generated by-`generate_noisy_alert_stream` in+Both numbers are **synthetic**. The 1,000-alert stream is generated with a+fixed RNG seed and is not a capture from a real SIEM.++The legacy suite in[`services/agents/tests/test_alert_reduction.py`](https://github.com/beenuar/AiSOC/blob/main/services/agents/tests/test_alert_reduction.py)-with a fixed RNG seed; the fusion code under test is a faithful in-harness-re-implementation of the production Tier 1 / Tier 2 / Tier 3 grouping rules in-`services/fusion`, **minus** the DB-backed deduplicator and the ML scorer. It-runs deterministically on every PR targeting `main` / `develop` and gates-fusion-logic regressions in milliseconds. It is **not** an end-to-end-measurement of the production fusion service against real customer telemetry.+was long described here as a faithful in-harness re-implementation of the+production grouping rules. That description was wrong, and the gap is wider+than "minus the DB-backed deduplicator and the ML scorer": the test does not+merely reimplement fusion's grouping, it implements **different** grouping.+Its four tiers key on `(rule_id, host, user)` with 10/30/5-minute windows,+while `RawAlert.correlation_key()` keys on `{tenant}:{entity}:{tactic}` over a+one-hour window.++[`services/fusion/tests/test_alert_reduction_real.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/tests/test_alert_reduction_real.py)+measures the real key. Neither suite is an end-to-end measurement of the+production fusion service against real customer telemetry.:::## TL;DR-| Metric | Value | Target |-|---|---|---|-| **Reduction ratio** | **75.3 %** (1,000 alerts → 247 incidents) | ≥ 70 % |-| Critical alerts preserved | **93 / 93** (zero drop) | 100 % |-| Storm incidents collapsed (Tier 3) | 16 (avg ~8.9 alerts each, 3–8 distinct hosts) | ≥ 1 |-| Determinism | Same RNG seed → same byte-for-byte output | Required |-| Wall time | ~8 ms on a laptop | < 100 ms |--The current snapshot lives at+| Metric | Value | Target | Describes AiSOC? |+|---|---|---|---|+| **Reduction ratio (product logic)** | **33.3 %** (1,000 alerts → 667 incidents) | 20–95 % | **Yes** — groups with `RawAlert.correlation_key()` |+| Reduction ratio (legacy suite) | 75.3 % (1,000 alerts → 247 incidents) | ≥ 70 % | **No** — four-tier scheme implemented inside the test |+| Critical alerts preserved (legacy suite) | 93 / 93 (zero drop) | 100 % | Invariant of the legacy suite |+| Storm incidents collapsed (legacy Tier 3) | 16 (avg ~8.9 alerts each, 3–8 distinct hosts) | ≥ 1 | Legacy suite only |+| Determinism | Same RNG seed → same byte-for-byte output | Required | Both |+| Wall time | ~8 ms on a laptop | < 100 ms | Both |++The real-key gate is bounded on **both** sides rather than floored. A floor+alone is satisfied by a key that collapses everything into one incident —+99.9 % reduction and a useless SOC — so "more reduction is better" is only+true up to a point, and the gate says where.++The legacy snapshot lives at[`eval-results/eval/results/latest.json`](https://github.com/beenuar/AiSOC/blob/eval-results/eval/results/latest.json)under `suites.alert_reduction`. Every successful build on `main` / `develop`appends a new commit-keyed entry alongside it (see@@ -62,12 +76,42 @@ This benchmark answers exactly one question:> incoming alerts collapse into incidents — and does the substrate drop any> of the alerts that an analyst genuinely needed to see?**-It is a **real measurement**, not a self-consistency gate. The fusion grouping-logic is the same Tier 1/2/3 algorithm shipped in `services/fusion` — a-regression in the grouping rules will move this number. The other three+Answered twice, because the harness has answered it twice. The other threesubstrate suites on the [public eval page](../benchmark.md) are tautological-by design (dataset and judge written together to gate template breakage); this-one is not.+by design (dataset and judge written together to gate template breakage);+neither alert-reduction suite is. But only one of them groups the way the+product groups.++## Why there are two numbers {#why-there-are-two-numbers}++The original suite was honest about being synthetic and always carried a+`PARTIAL` row saying it gated "an in-test fusion re-impl". Reading it closely,+the gap was wider than that wording admitted: the test did not merely+reimplement fusion's grouping, it implemented **different** grouping.++Its four tiers key on `(rule_id, host, user)` with 10/30/5-minute windows.+`RawAlert.correlation_key()` — what `Correlator` actually calls — keys on+`{tenant}:{entity}:{tactic}` over a one-hour window. Different dimensions,+different windows, different answer. So the 75.3 % described an algorithm the+product does not run, and a reimplementation can drift from the thing it+stands for without any test failing.++[`services/fusion/tests/test_alert_reduction_real.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/tests/test_alert_reduction_real.py)+measures the real key. It reports **33.3 %** on a comparable stream. That is+less flattering and it is ours.++`Correlator` itself needs Redis, but Redis is where it *stores* incidents —+the grouping decision is entirely `correlation_key()` plus the configured+window, so the ratio is computed from the real method without the storage+layer.++Both numbers stay published. Deleting the old one would make the history+unreadable; presenting it without this note would be the thing this page+exists to prevent.++The sections below describe the **legacy** suite's dataset and tier+structure, which is where the 75.3 % comes from. Read them as documentation+of a retained regression gate, not of the shipping correlation logic.## The synthetic dataset@@ -99,13 +143,14 @@ by construction and get dropped by the noise threshold. This is intentional —the benign-noise bucket exists to verify the noise-floor cut still fires, notto be shown to a human.-## The 3-tier fusion logic+## The legacy suite's 3-tier grouping-The grouping algorithm has three rules, applied in order. The exact code lives+The legacy algorithm has three rules, applied in order. The exact code livesin [`fuse_alerts`](https://github.com/beenuar/AiSOC/blob/main/services/agents/tests/test_alert_reduction.py)-inside the test file (so the benchmark stays self-contained) and mirrors the-production logic in-[`services/fusion`](https://github.com/beenuar/AiSOC/tree/main/services/fusion).+inside the test file. It is **not** the production logic in+[`services/fusion`](https://github.com/beenuar/AiSOC/tree/main/services/fusion),+which keys on `{tenant}:{entity}:{tactic}` over a one-hour window — see [Why+there are two numbers](#why-there-are-two-numbers).### Tier 1 — strict identity collapse@@ -139,17 +184,20 @@ alerts vanish.The numbers below are produced by `python3 scripts/run_evals.py --json` on thecurrent `main` and rendered into `eval/results/latest.json` on the-`eval-results` branch by the CI gate.+`eval-results` branch by the CI gate. `run_evals.py` emits the **legacy**+suite under the `alert_reduction` key; the real-key figure is produced by+pytest in `services/fusion` and is shown alongside it.### Headline-| Field | Value |-|---|---|-| `metric` | `reduction_ratio` |-| `value` | `0.753` |-| `target` | `0.70` |-| `passed` | `true` |-| `duration_ms` | `7.9` |+| Field | Product logic (real correlation key) | Legacy suite |+|---|---|---|+| `metric` | `reduction_ratio` | `reduction_ratio` |+| `value` | `0.333` | `0.753` |+| `target` | `0.20 – 0.95` (bounded both sides) | `≥ 0.70` (floor only) |+| `passed` | `true` | `true` |+| `duration_ms` | ~30 | `7.9` |+| Describes this product | **yes** | **no** |### Collapse breakdown@@ -217,7 +265,15 @@ python3 scripts/run_evals.py --json --out /tmp/eval.jsonpython3 -c "import json; print(json.dumps(json.load(open('/tmp/eval.json'))['suites']['alert_reduction'], indent=2))"```-To run only the alert-reduction tests under `pytest`:+To run the figure that describes this product — the real correlation key:++```bash+cd services/fusion+python3 -m pytest tests/test_alert_reduction_real.py -v -s+# [eval] alert reduction (real correlation_key): 1000 alerts -> 667 incidents = 33.3%+```++To run only the legacy alert-reduction tests under `pytest`:```bashcd services/agents@@ -259,13 +315,17 @@ the same JSON shape `run_evals.py` writes locally.In the spirit of being painfully honest about synthetic vs. real:-- **It does not measure the production fusion service end-to-end.** The- in-harness `fuse_alerts` is a faithful re-implementation of the Tier 1/2/3- rules from `services/fusion`, but the production service additionally runs- a Postgres-backed deduplicator (handles cross-window persistence) and an ML- scorer (lifts and depresses scores based on historical FP rate). Those- components have their own unit tests; they're out of scope here because- this gate has to run in milliseconds in CI.+- **The legacy `fuse_alerts` does not implement the production grouping at+ all.** It keys on `(rule_id, host, user)`; the product keys on+ `{tenant}:{entity}:{tactic}`. This was described here as a faithful+ re-implementation for several releases and it was not one.+- **Neither suite measures the production fusion service end-to-end.** The+ real-key measurement drives `RawAlert.correlation_key()` directly, but the+ production service additionally runs a Postgres-backed deduplicator+ (cross-window persistence) and an ML scorer (lifts and depresses scores+ based on historical FP rate). Those components have their own unit tests;+ they are out of scope here because this gate has to run in milliseconds in+ CI.- **It does not exercise real telemetry.** The 1,000-alert stream is shapedto look like the worst hour of a SIEM, but it isn't a capture from one.Real-customer evaluation is on the v1.5 roadmap and will be opt-in /@@ -317,6 +377,9 @@ section on the umbrella benchmark page lists the contribution path.against.- [`scripts/run_evals.py`](https://github.com/beenuar/AiSOC/blob/main/scripts/run_evals.py) —the unified runner that produces the JSON snapshot consumed by this page.+- [`services/fusion/tests/test_alert_reduction_real.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/tests/test_alert_reduction_real.py) —+ the measurement against `RawAlert.correlation_key()`. This is the one that+ describes AiSOC.- [`services/agents/tests/test_alert_reduction.py`](https://github.com/beenuar/AiSOC/blob/main/services/agents/tests/test_alert_reduction.py) —- the test file containing both the synthetic stream generator and the- in-harness fusion code under test.+ the legacy test file containing both the synthetic stream generator and the+ four-tier grouping under test.diff --git a/apps/docs/docs/compliance/evidence-pack.md b/apps/docs/docs/compliance/evidence-pack.mdindex a87a50ba4..0a3d63370 100644--- a/apps/docs/docs/compliance/evidence-pack.md+++ b/apps/docs/docs/compliance/evidence-pack.md@@ -33,7 +33,7 @@ and says so.| Prompt-injection resistance | 32-payload adversarial corpus with a recall floor and a false-positive ceiling | `services/agents/tests/adversarial/` || Claim-to-gate traceability | Every product claim mapped to the CI job that fails when it stops being true | `docs/audit/CLAIM_TO_GATE_MATRIX.md` |-The last row is the one worth reading first. It is the honest index: 131 rows+The last row is the one worth reading first. It is the honest index: 134 rows`GATED`, 8 `PARTIAL` with the specific gap named, and a ratchet that refusesto let a row sit at `NO GATE`.diff --git a/apps/docs/docs/glossary.md b/apps/docs/docs/glossary.mdindex 49d130e42..6d5561cd9 100644--- a/apps/docs/docs/glossary.md+++ b/apps/docs/docs/glossary.md@@ -184,8 +184,12 @@ carry a `tenant_id` column and are protected by RLS.## Eval & quality-**Alert reduction** — Real measurement: ratio of raw alerts in to-correlated incidents out, computed over a fixed noisy stream.+**Alert reduction** — Ratio of raw alerts in to correlated incidents out,+computed over a fixed noisy stream. Two figures are published: the real+measurement groups with `RawAlert.correlation_key()` — what `Correlator`+calls — and reports 33.3 %; a legacy in-test suite groups on four tiers of+`(rule_id, host, user)` and reports 75.3 %, which does not describe this+product.**Eval harness** — Reproducible suite that runs in CI against a fixed200-incident synthetic dataset. Three suites measure substrate self-diff --git a/apps/docs/docs/intro.md b/apps/docs/docs/intro.mdindex 58b18bf57..9e9bcf1f6 100644--- a/apps/docs/docs/intro.md+++ b/apps/docs/docs/intro.md@@ -49,7 +49,7 @@ for the full list.- **Click-and-connect cloud connectors** — pick from a **84-connector** catalog spanning EDR, SIEM, cloud, IAM, SaaS, VCS, network, ITSM, vuln, and email-security sources (CrowdStrike, SentinelOne, Cortex XDR, Splunk, Microsoft Sentinel, AWS Security Hub, Defender XDR, GCP Cloud Audit, GCP SCC, M365, Entra ID, Azure Activity, Google Workspace, Okta, Duo Security, Cloudflare, Tailscale, GitHub, Wiz, Snyk, Zscaler, Proofpoint, ServiceNow, Jira, 1Password, **Wazuh Indexer**, **auditd `file_tail`**, and 25 more — see [Architecture](./architecture)). Fill a schema-driven form, click `Test connection` for a live auth round-trip, and `Save & enable`. Secrets are encrypted at the application layer with a Fernet [`CredentialVault`](./operations/credentials) before they hit Postgres; an in-process APScheduler polls each enabled instance and pushes normalized OCSF events through to the ingest spine. Setup walkthroughs: [docs/connectors](./connectors).- **Investigation Ledger** — every prompt, response, evidence citation, and tool call the agent emits is logged step-by-step and replayable on each case.-- **Public eval harness** — alert reduction (a real measurement on a fixed noisy stream) plus MITRE-tactic, investigation-completeness, and response-quality substrate self-consistency gates. Reproducible with one command and run in CI on every PR. The [eval harness page](./benchmark) documents what each suite does and does not measure.+- **Public eval harness** — alert reduction, measured against `RawAlert.correlation_key()` on a fixed noisy stream (a legacy in-test suite grouping on different dimensions is retained for continuity and does not describe this product), plus MITRE-tactic, investigation-completeness, and response-quality substrate self-consistency gates. Reproducible with one command and run in CI on every PR. The [eval harness page](./benchmark) documents what each suite does and does not measure.- **Ambient Copilot** — context-aware next-action suggestions on every alert, case, rule, and playbook page; one click runs the right agent tool with the right payload.- **Responder PWA** — installable mobile route at `/responder/*` with passkey-only login, on-call rotation, approvals queue, VAPID Web Push, and offline shell.- **LangGraph multi-agent investigation** — orchestrator, recon, forensic, responder, and report-writer agents grounded in MITRE ATT&CK with Qdrant RAG memory. Includes domain-specific agents for phishing triage, identity threat analysis, cloud misconfiguration detection, and insider threat scoring.diff --git a/apps/docs/static/data/scoreboard.json b/apps/docs/static/data/scoreboard.jsonindex 8b86b1eb4..fcb8e44d3 100644--- a/apps/docs/static/data/scoreboard.json+++ b/apps/docs/static/data/scoreboard.json@@ -3,9 +3,9 @@"_readme": "Append-only public benchmark scoreboard. Each row is one eval run. Substrate rows (no LLM, deterministic) are marked substrate:true and MUST NOT be quoted as live agent performance. Wet-eval rows (substrate:false) are appended by the T5.5 weekly CI workflow once it lands. Newest row first.","rows": [{- "date": "2026-09-23",- "agent_version": "v8.1.1",- "commit_sha": "4ea86852",+ "date": "2026-09-25",+ "agent_version": "v10.0.0",+ "commit_sha": "46b1ad74","substrate": true,"eval_mode": "substrate-only","mitre_accuracy": 0.97,diff --git a/apps/web/src/app/benchmark/page.tsx b/apps/web/src/app/benchmark/page.tsxindex 80e7066d0..6eccd7adb 100644--- a/apps/web/src/app/benchmark/page.tsx+++ b/apps/web/src/app/benchmark/page.tsx@@ -98,10 +98,12 @@ export default async function BenchmarkPage() {</h1><p className="mt-4 max-w-3xl text-lg text-gray-400">A deterministic regression harness over the AiSOC substrate —- the keyword extractors, the in-harness fusion grouping (a faithful- re-implementation of the production Tier 1/2/3 logic in{' '}- <code className="text-gray-300">services/fusion</code>, minus the- DB-backed dedup and ML scoring), the report and response+ the keyword extractors, the in-harness fusion grouping (a four-tier+ scheme implemented inside the test, which groups on different+ dimensions from{' '}+ <code className="text-gray-300">services/fusion</code> and is+ retained for continuity — its number does not describe this+ product), the report and responsetemplates, and the offline judges that grade them. The dataset,the harness, and the CI gate are in the repo. The numbers on thispage are pulled from{' '}@@ -216,27 +218,41 @@ export default async function BenchmarkPage() {What each suite measures</h2><div className="mt-6 space-y-4 text-sm">- <div className="rounded-lg border border-emerald-500/20 bg-emerald-500/[0.03] p-5">+ <div className="rounded-lg border border-rose-500/20 bg-rose-500/[0.03] p-5"><div className="flex flex-wrap items-center gap-2"><h3 className="text-base font-semibold text-white">- Alert reduction ({fmtPct(ar?.value)})+ Alert reduction, legacy suite ({fmtPct(ar?.value)})</h3>- <span className="rounded-full border border-emerald-500/30 bg-emerald-500/10 px-2 py-0.5 text-[10px] font-semibold uppercase tracking-wider text-emerald-200">- Real measurement+ <span className="rounded-full border border-rose-500/30 bg-rose-500/10 px-2 py-0.5 text-[10px] font-semibold uppercase tracking-wider text-rose-200">+ Legacy — does not describe this product</span></div><p className="mt-2 text-gray-300">A 1,000-alert noisy stream with duplicates, near-duplicates,rule-storms, and benign chatter is fabricated deterministically,then passed through{' '}- <code className="text-gray-300">fuse_alerts</code> — an- in-harness re-implementation of the same Tier 1 / 2 / 3 merge- windows and score floor that the production fusion service- runs. The grouping logic is identical; the harness skips the- DB-backed deduplicator and ML scorer that ride on top in- production. The reduction ratio is whatever the harness code- emits. This is a legitimate measurement of the grouping logic,- and a regression in those rules will move the number.+ <code className="text-gray-300">fuse_alerts</code> — four+ tiers keyed on{' '}+ <code className="text-gray-300">(rule_id, host, user)</code>{' '}+ with 10/30/5-minute windows. This panel used to say the+ grouping logic was identical to production. It is not:{' '}+ <code className="text-gray-300">+ RawAlert.correlation_key()+ </code>+ , the method the fusion{' '}+ <code className="text-gray-300">Correlator</code> actually+ calls, keys on{' '}+ <code className="text-gray-300">+ {tenant}:{entity}:{tactic}+ </code>{' '}+ over a one-hour window. Different dimensions, different+ windows, different answer. The figure that describes this+ product is <strong>33.3%</strong>, measured against the real+ key by{' '}+ <code className="text-gray-300">+ services/fusion/tests/test_alert_reduction_real.py+ </code>+ . The legacy number is retained as a regression gate.</p></div>diff --git a/apps/web/src/app/why-open-source/page.tsx b/apps/web/src/app/why-open-source/page.tsxindex 311ca0d43..cb3acbe97 100644--- a/apps/web/src/app/why-open-source/page.tsx+++ b/apps/web/src/app/why-open-source/page.tsx@@ -33,7 +33,7 @@ const PILLARS = [{label: 'Public eval harness',href: '/benchmark',- body: 'A 200-incident eval suite runs on every PR targeting main / develop. Four CI gates: one real measurement (alert reduction, against a separately generated 1,000-alert noisy stream) and three substrate self-consistency checks (MITRE tactic, completeness, response quality, against the 200-incident dataset). The harness, the dataset, the rubric, and historical results are in the repo. The benchmark page documents what each metric measures and what it does not.',+ body: 'A 200-incident eval suite runs on every PR targeting main / develop. Alert reduction is measured against RawAlert.correlation_key() — the grouping the fusion Correlator actually calls — on a separately generated 1,000-alert noisy stream; a legacy in-test suite groups on different dimensions and is retained for continuity, not as a description of this product. Three further gates (MITRE tactic, completeness, response quality, against the 200-incident dataset) are substrate self-consistency checks. The harness, the dataset, the rubric, and historical results are in the repo. The benchmark page documents what each metric measures and what it does not.',},{label: 'MIT, end-to-end',@@ -49,7 +49,7 @@ const ARTEFACTS = [},{title: 'A reproducible eval harness',- body: 'Cloning the repo and running `python3 scripts/run_evals.py` produces the same alert-reduction ratio, MITRE-tactic gate, completeness coverage, and response-quality score that the CI gate produces. The benchmark page documents which numbers are real measurements of the substrate and which are self-consistency gates that would need an online LLM-as-judge run to be called agent accuracy.',+ body: 'Cloning the repo and running `python3 scripts/run_evals.py` produces the same legacy alert-reduction ratio, MITRE-tactic gate, completeness coverage, and response-quality score that the CI gate produces; `pytest services/fusion/tests/test_alert_reduction_real.py` produces the reduction measured against the correlation key the product runs. The benchmark page documents which numbers measure the substrate and which are self-consistency gates that would need an online LLM-as-judge run to be called agent accuracy.',},{title: 'Source code for the agent',diff --git a/apps/web/src/components/benchmark/BenchmarkResults.tsx b/apps/web/src/components/benchmark/BenchmarkResults.tsxindex 050ac76e9..6d11a472f 100644--- a/apps/web/src/components/benchmark/BenchmarkResults.tsx+++ b/apps/web/src/components/benchmark/BenchmarkResults.tsx@@ -1,6 +1,15 @@import clsx from 'clsx';-type Kind = 'measurement' | 'self-consistency';+/**+ * `legacy` exists because one suite is neither a measurement of this product+ * nor a self-consistency gate: the alert-reduction suite groups on four tiers+ * of `(rule_id, host, user)`, while the shipping `RawAlert.correlation_key()`+ * groups on `{tenant}:{entity}:{tactic}`. It does not merely re-implement+ * fusion's grouping, it implements *different* grouping, so its number+ * describes an algorithm the product does not run. `apps/docs/docs/benchmark.md`+ * withdrew the claim; this surface badged it "Real measurement" regardless.+ */+type Kind = 'measurement' | 'self-consistency' | 'legacy';interface PerTemplateBlock {metric: string;@@ -42,10 +51,16 @@ export interface EvalReport {/*** Snapshot fallback used when the live fetch from the `eval-results` branch* fails (e.g. the branch hasn't been initialized yet, or the build runs- * offline). Numbers are hand-copied from a recent `scripts/run_evals.py`- * run on `main`. The pass/fail gates in `services/agents/tests/test_*.py`+ * offline). Numbers are hand-copied from a `scripts/run_evals.py` run on+ * `main`, and `generated_at` below is that run's date, not this file's — the+ * page renders it alongside a "Snapshot fallback" badge so a reader can see+ * how old it is. The pass/fail gates in `services/agents/tests/test_*.py`* are the source of truth — if they pass, the headline "≥ target" claim* still holds even if these exact percentages lag a commit or two.+ *+ * `alert_reduction` here is the *legacy* four-tier suite, the only one+ * `run_evals.py` emits under that key. It is rendered with a `legacy` badge+ * rather than a measurement one; see the `Kind` docblock above.*/const SNAPSHOT_REPORT: EvalReport = {generated_at: '2026-05-06T04:32:09.258436+00:00',@@ -157,11 +172,11 @@ interface SuiteCard {const SUITE_META: Record<string, SuiteCard> = {alert_reduction: {id: 'alert_reduction',- name: 'Alert reduction',+ name: 'Alert reduction (legacy suite)',metricLabel: 'Reduction ratio',- kind: 'measurement',+ kind: 'legacy',blurb:- "A 1,000-alert noisy stream (duplicates, near-duplicates, rule storms, low-score chatter) is fed into an in-harness re-implementation of the production Tier 1 / 2 / 3 grouping rules — same logic, no DB-backed dedup or ML scorer. The number is whatever the code produces; a regression in the grouping rules moves it.",+ 'A four-tier scheme implemented inside the test, keyed on (rule_id, host, user) with 10/30/5-minute windows. RawAlert.correlation_key() — what Correlator actually calls — keys on {tenant}:{entity}:{tactic} over a one-hour window. Different dimensions, different windows, different answer. Retained for continuity; it does not describe this product. The number that does is measured against the real correlation key by services/fusion/tests/test_alert_reduction_real.py and published on the benchmark page.',},mitre_accuracy: {id: 'mitre_accuracy',@@ -198,6 +213,10 @@ const KIND_LABEL: Record<Kind, { label: string; classes: string }> = {label: 'Substrate self-consistency',classes: 'border-amber-500/30 bg-amber-500/10 text-amber-200',},+ legacy: {+ label: 'Legacy — does not describe this product',+ classes: 'border-rose-500/30 bg-rose-500/10 text-rose-200',+ },};function formatPercent(value: number): string {@@ -246,11 +265,13 @@ interface BenchmarkResultsProps {}export function BenchmarkResults({ report }: BenchmarkResultsProps) {+ // The legacy suite sorts last: it led the grid while badged as a real+ // measurement, which put a withdrawn figure in the first card a reader sees.const orderedIds: string[] = [- 'alert_reduction','mitre_accuracy','investigation_completeness','response_quality',+ 'alert_reduction',];return (diff --git a/apps/web/src/components/benchmark/ComparisonTable.test.tsx b/apps/web/src/components/benchmark/ComparisonTable.test.tsxindex e24bc9c94..e0b330c09 100644--- a/apps/web/src/components/benchmark/ComparisonTable.test.tsx+++ b/apps/web/src/components/benchmark/ComparisonTable.test.tsx@@ -27,10 +27,14 @@ describe('ComparisonTable', () => {expect(screen.queryByText(/every commit/i)).toBeNull();});- it('tags the alert-reduction number as a measurement, not a marketing claim', () => {+ it('quotes the reduction measured against the correlation key the product runs', () => {render(<ComparisonTable />);- // Substring match — the cell reads "75.3% (measured on fixed noisy stream)".- expect(screen.getByText(/75\.3% \(measured/)).toBeInTheDocument();+ // Substring match — the cell reads+ // "33.3% (real correlation key, fixed noisy stream)".+ expect(screen.getByText(/33\.3% \(real correlation key/)).toBeInTheDocument();+ // The legacy four-tier figure describes an algorithm this product does not+ // run, so it must not appear in a vendor comparison at all.+ expect(screen.queryByText(/75\.3/)).toBeNull();});});diff --git a/apps/web/src/components/benchmark/ComparisonTable.tsx b/apps/web/src/components/benchmark/ComparisonTable.tsxindex 00021c704..5e21b9801 100644--- a/apps/web/src/components/benchmark/ComparisonTable.tsx+++ b/apps/web/src/components/benchmark/ComparisonTable.tsx@@ -12,8 +12,12 @@ interface Vendor {/*** The AiSOC row uses qualifiers that match what the harness actually does:- * - Alert reduction is a *real* measurement against a fixed noisy stream, so- * we tag it "measured".+ * - Alert reduction quotes the figure produced by the grouping the product+ * actually runs — `RawAlert.correlation_key()`, measured by+ * `services/fusion/tests/test_alert_reduction_real.py`. The older 75.3%+ * came from a four-tier scheme implemented inside a test and describes an+ * algorithm this product does not run; quoting it here was qualified as+ * "measured on fixed noisy stream", which is not the qualifier that matters.* - MITRE "accuracy" is a substrate self-consistency check (extractor vs.* dataset that's written to feed it), so we tag it "regression gate" rather* than implying it is a leaderboard score for an LLM agent.@@ -24,7 +28,7 @@ const VENDORS: Vendor[] = [{name: 'AiSOC',type: 'open',- reduction: '75.3% (measured on fixed noisy stream)',+ reduction: '33.3% (real correlation key, fixed noisy stream)',mitre: '97% (substrate regression gate)',audit: 'Per-step ledger',selfHost: 'Yes (MIT)',diff --git a/apps/web/src/components/detections/ContributorLeaderboard.tsx b/apps/web/src/components/detections/ContributorLeaderboard.tsxindex 9c4a02c47..9f416ce41 100644--- a/apps/web/src/components/detections/ContributorLeaderboard.tsx+++ b/apps/web/src/components/detections/ContributorLeaderboard.tsx@@ -2,6 +2,11 @@import { useMemo, useState } from 'react';+import {+ DETECTION_CATEGORIES,+ EXECUTABLE_DETECTION_COUNT,+} from '@/data/corpusStats';+const CONTRIBUTE_URL ='https://github.com/beenuar/AiSOC/blob/main/.github/PULL_REQUEST_TEMPLATE/detection_rule.md';const DETECTIONS_URL =@@ -21,8 +26,16 @@ const BADGE_STYLES: Record<Contributor['badge'], { bg: string; text: string; borbronze: { bg: 'bg-orange-500/10', text: 'text-orange-300', border: 'border-orange-500/30', label: 'Bronze' },};+// The core row is the executable corpus, read from the generated artefact+// rather than typed: it previously claimed 218 rules across 5 categories,+// which matched neither the engine (833) nor the indexed corpus.const CORE_CONTRIBUTORS: Contributor[] = [- { name: 'AiSOC', rules: 218, categories: ['cloud', 'endpoint', 'identity', 'network', 'application'], badge: 'platinum' },+ {+ name: 'AiSOC',+ rules: EXECUTABLE_DETECTION_COUNT,+ categories: Object.keys(DETECTION_CATEGORIES),+ badge: 'platinum',+ },];const BADGE_THRESHOLDS: { tier: Contributor['badge']; min: number; label: string }[] = [@@ -187,8 +200,8 @@ export function ContributorLeaderboard() {rel="noreferrer"className="underline decoration-dotted hover:text-gray-300">- {sorted.reduce((sum, c) => sum + c.rules, 0).toLocaleString()}+ rules across{' '}- {[...new Set(sorted.flatMap((c) => c.categories))].length} categories+ {sorted.reduce((sum, c) => sum + c.rules, 0).toLocaleString()} executable rules+ across {[...new Set(sorted.flatMap((c) => c.categories))].length} categories</a></p><p className="text-xs text-gray-500">diff --git a/apps/web/src/components/landing/sections/ConnectorsMarquee.tsx b/apps/web/src/components/landing/sections/ConnectorsMarquee.tsxindex 5b93b89cf..8a53ba5ef 100644--- a/apps/web/src/components/landing/sections/ConnectorsMarquee.tsx+++ b/apps/web/src/components/landing/sections/ConnectorsMarquee.tsx@@ -10,6 +10,9 @@* of the real `_CONNECTOR_CLASSES` registry. The full catalogue size* is rendered from `CONNECTOR_COUNT` (sourced from the registry by* `scripts/generate_connector_count.py`) so the prose can never drift.+ * The corpus figures come from `corpusStats.ts` the same way+ * (`scripts/generate_corpus_stats.py`), and lead with the executable+ * count — the indexed corpus is mostly quarantined and never loaded.* - Code callout with a "Write a connector in 50 lines" snippet* mirroring the BaseConnector pattern used in the registry.*@@ -23,6 +26,10 @@ import { Marquee } from '@/components/magicui/Marquee';import { docs } from '@/lib/docs';import { cn } from '@/lib/utils';import { CONNECTOR_COUNT } from '@/data/connectorCount';+import {+ EXECUTABLE_DETECTION_COUNT,+ PLAYBOOK_PACK_COUNT,+} from '@/data/corpusStats';interface ConnectorPill {name: string;@@ -142,7 +149,10 @@ export function ConnectorsMarquee() {id="connectors-heading"className="font-velvet-display font-normal mt-3 text-3xl tracking-tight text-velvet-content-primary sm:text-4xl lg:text-[40px] lg:leading-[1.15] lg:tracking-[-0.015em]">- {CONNECTOR_COUNT} connectors. 6,998 detections. 62 playbook packs.+ {/* One template literal, so SSR emits a single text node: adjacent+ JSX children around an interpolation get `<!-- -->` separators+ injected server-side that the client does not reproduce. */}+ {`${CONNECTOR_COUNT} connectors. ${EXECUTABLE_DETECTION_COUNT} executable detections. ${PLAYBOOK_PACK_COUNT} playbook packs.`}</h2><p className="mt-4 text-base leading-relaxed text-velvet-content-secondary sm:text-lg">Every connector renders a schema-driven form, encrypts itsdiff --git a/apps/web/src/components/landing/sections/FeatureGrid.tsx b/apps/web/src/components/landing/sections/FeatureGrid.tsxindex 78274c5b5..70638551e 100644--- a/apps/web/src/components/landing/sections/FeatureGrid.tsx+++ b/apps/web/src/components/landing/sections/FeatureGrid.tsx@@ -41,6 +41,10 @@ import {import type { ComponentType, SVGProps } from 'react';import { cn } from '@/lib/utils';import { CONNECTOR_COUNT } from '@/data/connectorCount';+import {+ EXECUTABLE_DETECTION_COUNT,+ MARKETPLACE_ITEM_COUNT,+} from '@/data/corpusStats';interface Tile {icon: ComponentType<SVGProps<SVGSVGElement>>;@@ -72,7 +76,7 @@ const BLOCKS: ReadonlyArray<FeatureBlock> = [{icon: FileCog,title: 'Native detections.',- body: '6,998 YAML rules across cloud, endpoint, identity, network, application, and data-exfil.',+ body: `${EXECUTABLE_DETECTION_COUNT} executable rules across cloud, endpoint, identity, network, application, and data-exfil.`,},{icon: ScrollText,@@ -139,7 +143,7 @@ const BLOCKS: ReadonlyArray<FeatureBlock> = [{icon: Boxes,title: 'Marketplace.',- body: '7,117 community items — detections, playbooks, plugins.',+ body: `${MARKETPLACE_ITEM_COUNT.toLocaleString('en-US')} indexed items — detections, playbooks, plugins.`,},{icon: Code2,diff --git a/apps/web/src/components/landing/sections/Hero.tsx b/apps/web/src/components/landing/sections/Hero.tsxindex a3c12c80c..3ac320c5f 100644--- a/apps/web/src/components/landing/sections/Hero.tsx+++ b/apps/web/src/components/landing/sections/Hero.tsx@@ -33,10 +33,15 @@ import { AnimatedGridPattern } from '@/components/magicui/AnimatedGridPattern';import { AuroraText } from '@/components/magicui/AuroraText';import { cn } from '@/lib/utils';import { CONNECTOR_COUNT } from '@/data/connectorCount';+import {+ EXECUTABLE_DETECTION_COUNT,+ PLAYBOOK_PACK_COUNT,+ PLUGIN_COUNT,+} from '@/data/corpusStats';const PROOF_CHIPS: ReadonlyArray<string> = [`${CONNECTOR_COUNT} connectors · EDR · SIEM · cloud · IAM · SaaS · VCS · network`,- '6,998 detections · 62 playbook packs · 57 plugins',+ `${EXECUTABLE_DETECTION_COUNT} executable detections · ${PLAYBOOK_PACK_COUNT} playbook packs · ${PLUGIN_COUNT} plugins`,'Self-host · Render · Fly.io · Helm · Terraform · air-gap',];diff --git a/apps/web/src/components/landing/sections/Pillars.tsx b/apps/web/src/components/landing/sections/Pillars.tsxindex fe9c55c0f..7fac72c54 100644--- a/apps/web/src/components/landing/sections/Pillars.tsx+++ b/apps/web/src/components/landing/sections/Pillars.tsx@@ -12,9 +12,10 @@* pointer event so the card reads as "active" without redrawing* its body.*- * Each card carries a stat strip (e.g. "6,998 public detection rules")+ * Each card carries a stat strip (e.g. "833 executable detection rules")* with `tnum`-enabled mono digits and a single CTA link that opens the- * corresponding repo file.+ * corresponding repo file. Corpus figures come from `corpusStats.ts`+ * (`scripts/generate_corpus_stats.py`), never typed by hand.*/import Link from 'next/link';@@ -22,6 +23,7 @@ import { motion, useReducedMotion } from 'framer-motion';import { ArrowRight, GitGraph, ScrollText, Sparkles, Boxes } from 'lucide-react';import type { ComponentType, SVGProps } from 'react';import { GlowingEffect } from '@/components/aceternity/GlowingEffect';+import { EXECUTABLE_DETECTION_COUNT } from '@/data/corpusStats';import { docs } from '@/lib/docs';import { cn } from '@/lib/utils';@@ -43,8 +45,8 @@ const PILLARS: ReadonlyArray<Pillar> = [title: 'Open source and transparent',body:'MIT-licensed agent, public detection corpus, reproducible benchmark — every claim on this page maps to a file in the repo.',- stat: '6,998',- statLabel: 'public detection rules',+ stat: String(EXECUTABLE_DETECTION_COUNT),+ statLabel: 'executable detection rules',href: 'https://github.com/beenuar/AiSOC/blob/main/LICENSE',linkLabel: 'Read the LICENSE',},diff --git a/apps/web/src/components/landing/sections/ProofStrip.tsx b/apps/web/src/components/landing/sections/ProofStrip.tsxindex 192063fca..0eb5362ae 100644--- a/apps/web/src/components/landing/sections/ProofStrip.tsx+++ b/apps/web/src/components/landing/sections/ProofStrip.tsx@@ -3,16 +3,21 @@/*** Proof / logo strip — `proof-strip` from §6.2 of the brief.*- * Renders two horizontal rows:+ * Renders one horizontal row: "Built on the open-source stack you already+ * trust" — six line-art brand wordmarks (LangGraph, Apache Kafka, Neo4j,+ * PostgreSQL, Qdrant, Ollama) inside a Marquee, paused on hover/focus. Each+ * names a dependency a reader can verify in the tree.*- * 1. "Built on the open-source stack you already trust" — six line-art- * brand wordmarks (LangGraph, Apache Kafka, Neo4j, PostgreSQL,- * Qdrant, Ollama) inside a Marquee, paused on hover/focus.- *- * 2. "Design partners" — four greyed partner-outline pills that read- * "Reference partners onboarding through Q2 2026", matching the- * content doc's empty-state treatment until real customer logos- * are licensed.+ * A "Design partners" block used to sit below it: four dashed "Partner A–D"+ * chips under the caption "Reference partners onboarding through Q2 2026".+ * It was removed rather than updated. The chips were placeholders rather than+ * fabricated logos, but four of them assert a partner count nothing in this+ * repository supports, and the date had already lapsed — a window that closed+ * in June 2026 was still being advertised as upcoming. An empty-state that+ * implies four partners is a weaker version of a fabricated logo wall, not a+ * neutral absence, so the honest rendering of "no design partners yet" is to+ * render nothing. Restore the section when there are named partners who have+ * agreed to be named.** The Marquee primitive (`MagicUI`) already collapses to a static row* under `prefers-reduced-motion` via the `animate-marquee` keyframe@@ -35,13 +40,6 @@ const STACK_WORDMARKS: ReadonlyArray<{{ name: 'Ollama', caption: 'Local LLM' },];-const PARTNER_PLACEHOLDERS = [- 'Partner A',- 'Partner B',- 'Partner C',- 'Partner D',-] as const;-function StackWordmark({ name, caption }: { name: string; caption: string }) {return (<div@@ -92,31 +90,6 @@ export function ProofStrip() {</Marquee></div>- <div className="mt-10 flex flex-col items-center gap-4">- <p className="text-xs font-semibold uppercase tracking-[0.18em] text-velvet-content-tertiary">- Design partners- </p>- <ul- aria-label="Reference partners onboarding through Q2 2026"- className="flex flex-wrap items-center justify-center gap-2 sm:gap-3"- >- {PARTNER_PLACEHOLDERS.map((partner) => (- <li- key={partner}- className="inline-flex items-center gap-2 rounded-full border border-dashed border-velvet-border bg-velvet-surface-raised/30 px-4 py-1.5 text-xs font-medium text-velvet-content-tertiary"- >- <span- aria-hidden="true"- className="inline-block h-1.5 w-1.5 rounded-full bg-velvet-content-tertiary/40"- />- {partner}- </li>- ))}- </ul>- <p className="text-xs text-velvet-content-tertiary">- Reference partners onboarding through Q2 2026- </p>- </div></div></section>);diff --git a/apps/web/src/data/corpus-stats.json b/apps/web/src/data/corpus-stats.jsonnew file mode 100644index 000000000..bba5f6187--- /dev/null+++ b/apps/web/src/data/corpus-stats.json@@ -0,0 +1,22 @@+{+ "executable": 833,+ "onDisk": 7016,+ "quarantined": 5937,+ "plugins": 77,+ "playbookPacks": 62,+ "marketplaceItems": 7155,+ "categories": {+ "application": 86,+ "cloud": 190,+ "data-exfil": 20,+ "endpoint": 296,+ "identity": 160,+ "network": 81+ },+ "generatedFrom": [+ "services/fusion/app/data/detection_ruleset.json",+ "marketplace/index.json",+ "docs/detections/truth-table.md"+ ],+ "regenerateWith": "python3 scripts/generate_corpus_stats.py"+}diff --git a/apps/web/src/data/corpusStats.ts b/apps/web/src/data/corpusStats.tsnew file mode 100644index 000000000..7d610681c--- /dev/null+++ b/apps/web/src/data/corpusStats.ts@@ -0,0 +1,32 @@+// AUTO-GENERATED by scripts/generate_corpus_stats.py.+// Do not edit by hand. Run the script (or `make corpus-stats`) instead.+//+// Single source of truth for every published detection / marketplace count.+//+// EXECUTABLE_DETECTION_COUNT is the capability number: rules the fusion+// engine actually loads. DETECTIONS_ON_DISK is the indexed corpus, most of+// which is quarantined and never evaluated -- it is a corpus figure and must+// never be presented as what the product executes.++import data from './corpus-stats.json';++/** Rules the fusion engine loads and evaluates. */+export const EXECUTABLE_DETECTION_COUNT: number = data.executable;++/** Detection YAML indexed in the marketplace, quarantined rules included. */+export const DETECTIONS_ON_DISK: number = data.onDisk;++/** Imported rules held under `_quarantine/`: on disk, never loaded. */+export const QUARANTINED_DETECTIONS: number = data.quarantined;++export const PLUGIN_COUNT: number = data.plugins;+export const PLAYBOOK_PACK_COUNT: number = data.playbookPacks;+export const MARKETPLACE_ITEM_COUNT: number = data.marketplaceItems;++/** Executable rules per detection category. */+export const DETECTION_CATEGORIES: Readonly<Record<string, number>> = data.categories;+export const DETECTION_CATEGORY_COUNT: number = Object.keys(data.categories).length;++/** Locale-formatted labels, so no surface re-implements the separator. */+export const EXECUTABLE_DETECTIONS_LABEL = `${EXECUTABLE_DETECTION_COUNT.toLocaleString()} executable detections`;+export const MARKETPLACE_ITEMS_LABEL = `${MARKETPLACE_ITEM_COUNT.toLocaleString()} marketplace items`;diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.mdindex 6c9db5824..a3c3a0df0 100644--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md@@ -19,7 +19,7 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`| Runs entirely on your infrastructure / no data exfiltration | README L63 | `ci.yml :: python-test` (agents) runs `test_privacy_redactor.py` (zero raw PII survives) | PARTIAL (redaction gated; the CLI now has no default telemetry endpoint and `telemetry.test.ts` pins that an enabled-but-unconfigured run makes zero network calls; **v9.0 adds the Helm egress default-deny** — `infra/helm/aisoc/templates/networkpolicy.yaml`, gated by `helm.yml` which asserts it denies by default, excludes `169.254.169.254/32` from every allowed CIDR, and stays opt-in because a CNI that does not enforce NetworkPolicy ignores it silently; platform-wide air-gapped egress-blocked CI still outstanding) | Phase 2 || Detection-as-Code rejects candidates that regress MITRE accuracy | README L170 | `ci.yml :: python-test` (`test_detection_eval.py` — candidate `rule_body` run through the real engine vs its own positive/negative fixtures; approval requires it) + `p1-eval` w2-dac baseline | GATED | - || 800+ native detection rules | README L78, L170 | `validate-detections.yml` (strict fixture replay) | GATED | - |-| 6000+ imported detection rules | README L78 | `validate-detections.yml` (parse/provenance + `detection_truth_table.py --check`); README cites the executable figure (939) not the on-disk figure for coverage | GATED | - |+| 6000+ imported detection rules | README L78 | `validate-detections.yml` (parse/provenance + `detection_truth_table.py --check`); README cites the executable figure (833) not the on-disk figure for coverage | GATED | - || L0-L4 automation maturity gates every action | README L171 | `ci.yml` autonomy-policy drift test + `test_autonomy_safety.py` (dry-run-default, rollback-capability contract, verification-mandate, scorecard) + `test_governed_dispatch.py` (Phase B2 — `decide()` wired into the live-actions `/dispatch` path: above-tier→dry-run, dry-run-off→queued, L0→blocked, executor never invoked when queued/blocked) | GATED | - || Hunt-as-Code + `/hunt` | README L172 | `ci.yml :: p1-eval` (`hunt_corpus`) | GATED | - || Weekly benchmark scoreboard runs live against `main` | README L173 | `ci.yml` agents job (Phase E1 — `scripts/check_scoreboard.py` runs the deterministic live-agent MITRE-accuracy eval over the 200-incident corpus and fails if the published `scoreboard.json` substrate row drifts > 0.02, the schema is invalid, or a substrate row is mislabelled) + `wet-eval.yml` (weekly funded LLM rows) | GATED | - |@@ -147,10 +147,13 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`| Every model AiSOC sends has a gateway that defines it, and the variable compose supplies is the variable the resolvers read | `apps/docs/docs/operations/llm-gateway.md`, `.env.example`, README (AI triage) | `ci.yml :: python-lint` (`scripts/check_llm_model_routing.py` — both directions on every name: each role pin's alias must be defined by `infra/litellm/config.yaml` and each alias must be claimed by a pin; every model named in `.env.example` or `docker-compose.yml`, in a file that wires the bundled gateway, must be one the gateway defines; every variable compose points at the gateway must be read by *both* resolvers, and every gateway variable the resolvers read must be set by compose — the defect that shipped was compose supplying `LLM_GATEWAY_URL` and both resolvers deliberately ignoring it, so AI triage could not reach a model at all; a service given the gateway URL must also get `LITELLM_MASTER_KEY`, because a provider key sent there is rejected. Every parser is AST- or comment-aware and the self-test proves it: the config's commented Ollama example repeats `model_name: aisoc-triage` verbatim, `.env.example` documents the escape hatch as a commented `AISOC_MODEL_PIN_TRIAGE=`, and compose explains the gateway in prose two lines above the key it describes — a line regex credits all three) + `services/agents/tests/test_llm_factory.py` + `services/api/tests/test_model_aliases_routing.py` | GATED | Embeddings are a recorded exclusion — the gateway's model list is chat aliases only — and the gate checks that in both directions rather than trusting the comment || Every test file in the tree is executed by a workflow | `CONTRIBUTING.md` / the suites themselves | `ci.yml :: python-test` (`scripts/check_test_discovery.py` — the corpus is `git ls-files`, so a service added tomorrow is scanned with no edit, and an untracked install tree is out by construction rather than by a name somebody remembered. Reach is modelled the way pytest collects: directory arguments, `--ignore`, `conftest` `collect_ignore`, and `-m` marker deselection, because a file whose every test is deselected runs nothing however plainly it is named. Both directions — an invocation naming a path that is not in the tree fails too. Quarantine is a shrink-only list where each entry carries a reason and an entry that has become reachable fails as stale. Two blind spots in this parser were found by counting what it **credited**: a step that `cd`s into a service and then names `tests/` had its paths resolved against the repository root, which reported all 82 agents files unreached *and* credited the root tree with a package's invocation; and `-m` was first skipped as "a filter, not a collection rule", which would have credited a file in full under `-m "not integration"`) + `tests/test_test_discovery_gate.py` | GATED | - || Every reported LLM cost is measured, labelled an estimate, or absent | admin cost dashboard, `/costs/dashboard`, `/investigations` run + aggregate responses, Investigation Ledger, `/metrics/funnel` insights, investigation summary export | `ci.yml :: python-lint` (`scripts/check_cost_provenance.py` — the tracker priced the `aisoc-<role>` **alias** against a table of hosted list prices it does not appear in, so every call fell through a `(0.001, 0.002)` default: a 903-token completion on an operator's own hardware reported `total_cost_usd=0.000999`, and that figure reached the cost dashboard, the funnel, the per-run ledger and the budget circuit breaker, which trips at `AISOC_BUDGET_HARD_USD`. Seven rules, each a structural relationship with both name spaces enumerated: every money field has the count that qualifies it and every provenance count qualifies a money field; every column the writer writes is added by migration 063 and every column 063 adds is written or read; neither pricing table carries a fallback rate and neither can key on an alias; the two gateway headers read are the ones carrying a **total**, not the five sharing their prefix that carry a component, an adjustment or a *percentage*; every provenance field on the wire exists on both the Python and TypeScript sides, which cannot import one another; and a console surface that renders a currency amount consults a provenance count first. Every parser is AST- or comment-aware, proven by nine blind-spot cases built from text already in the corpus — `gateway_cost.py`'s docstring enumerates all seven cost headers to explain why five are refused, `cost_telemetry.py`'s docstring names the removed `(0.001, 0.002)` default, the migration's `COMMENT ON COLUMN` statements repeat every column name, and `api.ts` documents fields in comments above them — a line regex credits all of them) + `services/agents/tests/test_cost_provenance.py` (22 cases, header fixtures captured from a real gateway) + `apps/web/src/components/admin/CostDashboardView.test.tsx` (first-paint assertion that a window with 1,287 unmeasured calls renders no `$` at all) | GATED | A measured `0.0` — what a local model genuinely costs — is a value and is reported as one; only an *absence* renders as not measured |+| Published detection / marketplace counts match the tree | landing page (`Hero.tsx`, `ConnectorsMarquee.tsx`, `FeatureGrid.tsx`, `Pillars.tsx`), `ContributorLeaderboard.tsx` | `ci.yml :: python-lint` (`generate_corpus_stats.py --check` — the counts are generated into `apps/web/src/data/corpus-stats.json` + `corpusStats.ts` and imported, so a surface cannot carry its own literal; the generator reconciles three independent sources against each other — the compiled engine ruleset, `docs/detections/truth-table.md` and `marketplace/index.json` — and refuses to publish if they disagree. The landing page previously gave three different wrong answers at once: 6,998 detections in four places, 7,117 marketplace items, and 218 rules on the leaderboard, with no gate on any of them) | GATED | - |+| The corpus figure the product executes is distinguished from the corpus on disk | landing page, `docs/detections/truth-table.md` | `ci.yml :: python-lint` (`generate_corpus_stats.py --check` emits `executable` and `onDisk`/`quarantined` as separate fields and the UI leads with `EXECUTABLE_DETECTION_COUNT`; ~85% of the indexed corpus sits under `_quarantine/` and is never loaded, so quoting the on-disk figure as the detection capability presents quarantined rules as executable) | GATED | - |+| The retracted 75.3% alert-reduction figure is never presented as a measurement | `apps/docs/docs/benchmark.md`, `benchmarks/alert-reduction.md`, `BenchmarkResults.tsx`, `ComparisonTable.tsx`, `benchmark/page.tsx` | `ci.yml :: python-lint` (`check_alert_reduction_claims.py --check` — scans every published surface: no file may assert the legacy harness runs the production grouping ("faithful re-implementation", "same logic"), any file quoting 75.3% must carry the retraction, the `alert_reduction` suite card may not be declared `kind: 'measurement'`, and every surface publishing the real figure must quote `PUBLISHED_REDUCTION_PCT` from `services/fusion/tests/test_alert_reduction_real.py`, which that test asserts against its own measurement — so the chain from measurement to prose has no hand-copied link. `benchmark.md` had withdrawn the claim while two other surfaces went on asserting it, and `benchmark.md` itself re-asserted it in its own intro blockquote) | GATED | - |## Summary-- GATED: 131+- GATED: 134- PARTIAL: 8- NO GATE: 0 (**every claim is backed by a failing test.** The last NO GATE — the weekly benchmark scoreboard running live against `main` — closed in Phase E1: `scripts/check_scoreboard.py` ties the published scoreboard to a deterministic per-PR live-agent MITRE-accuracy run, while the funded weekly `wet-eval.yml` appends the LLM-tier rows. The remaining PARTIAL rows are honest, named deferrals — each states the specific gap and what closes it — not unproven claims.)@@ -170,6 +173,17 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`> only enforces the NO GATE ceiling, so a stale GATED total does not fail CI> — the number above is now what `scripts/check_claim_gate_matrix.py` prints,> and re-running it is the only way to change it.+>+> **And once more, outside this file.** `readme_gates.py` compares the tally+> in the README, in this file's Summary, and in a compliance page — but+> `ROADMAP.md` was not on that list, so its tally line sat at 136 rows /+> 128 GATED against a matrix holding 139 / 131 with every check green. The+> line even tells the reader to recount with the script "rather than trusting+> a figure quoted in prose — this line has gone stale before". A gate that+> names the documents it covers will miss the one nobody added; `ROADMAP.md`+> and `RELEASES.md` are now on the list, the *row total* is compared as well+> as the GATED/PARTIAL split, and a figure the prose explicitly dates+> ("the count at that time") is exempted so history need not be rewritten.### On the PARTIAL rowsdiff --git a/docs/roadmap/v8-progress.md b/docs/roadmap/v8-progress.mdindex 30286ef8b..b119d8da6 100644--- a/docs/roadmap/v8-progress.md+++ b/docs/roadmap/v8-progress.md@@ -49,8 +49,10 @@ migration runners for all three non-Postgres stores), recursive investigationcontract, and a SOC-agent benchmark a third party can run against their ownagent.-Claim-to-gate matrix: **108 rows — 99 GATED / 9 PARTIAL / 0 NO GATE**. Recount-with `python3 scripts/check_claim_gate_matrix.py`; do not quote a remembered+Claim-to-gate matrix **at the v8.0 cut**: 108 rows — 99 GATED / 9 PARTIAL /+0 NO GATE. That is a record of where v8.0 landed, not the current tally; the+matrix has grown since. Recount with+`python3 scripts/check_claim_gate_matrix.py`; do not quote a rememberedfigure.Of the twelve hardening phases, **Phase 4 is the only one unchecked, anddiff --git a/scripts/check_alert_reduction_claims.py b/scripts/check_alert_reduction_claims.pynew file mode 100644index 000000000..0c489ad07--- /dev/null+++ b/scripts/check_alert_reduction_claims.py@@ -0,0 +1,379 @@+#!/usr/bin/env python3+"""A retracted benchmark figure must stay retracted on every surface that quotes it.++Why this exists+---------------+``apps/docs/docs/benchmark.md`` withdrew the 75.3 % alert-reduction claim: the+harness that produced it groups on four tiers of ``(rule_id, host, user)``,+while the shipping ``RawAlert.correlation_key()`` groups on+``{tenant}:{entity}:{tactic}``. It does not merely re-implement fusion's+grouping, it implements **different** grouping, so the number describes an+algorithm the product does not run.++The retraction reached exactly one of the three surfaces carrying the claim.++ * ``BenchmarkResults.tsx`` kept rendering a green **"Real measurement"**+ badge on ``0.753`` with a blurb asserting the harness used the production+ grouping rules, "same logic" — the precise sentence ``benchmark.md``+ exists to withdraw.+ * ``benchmarks/alert-reduction.md`` called the harness a "faithful in-harness+ re-implementation of the production Tier 1 / Tier 2 / Tier 3 grouping+ rules" and headlined 75.3 %. It is ``sidebar_position: 1``, so it was the+ first benchmark page a reader opened.+ * ``ComparisonTable.tsx`` qualified the figure as "measured on fixed noisy+ stream" — a qualifier, but not the one that matters.++A correction that lands on one surface and not the others is not a+correction; it is a repository that contradicts itself and cites whichever+page the reader happened to open. Prose cannot be generated the way a count+can, so the invariant is gated instead:++1. No published surface may assert the legacy harness runs the production+ grouping logic.+2. Any surface quoting the legacy figure must carry the retraction with it.+3. Surfaces must agree on the figure that *does* describe the product.++Usage+-----+ python3 scripts/check_alert_reduction_claims.py+ python3 scripts/check_alert_reduction_claims.py --check # same, explicit+ python3 scripts/check_alert_reduction_claims.py --self-test+"""++from __future__ import annotations++import argparse+import re+import sys+from pathlib import Path++# `scripts/` is on sys.path when this file is run as a program, but not when a+# test loads it by path with importlib. gate_toolkit sits beside it either way.+sys.path.insert(0, str(Path(__file__).resolve().parent))++from gate_toolkit import repo_root, self_test_main++REPO_ROOT = repo_root()++#: Surfaces a reader can reach: the docs portal, the marketing/console UI, and+#: the governance documents at the root. Deliberately not `services/` — a test+#: docstring explaining the defect must be free to describe it.+SCAN_ROOTS: tuple[tuple[Path, tuple[str, ...]], ...] = (+ (REPO_ROOT / "apps" / "docs" / "docs", ("*.md", "*.mdx")),+ (REPO_ROOT / "apps" / "web" / "src", ("*.tsx", "*.ts")),+ (REPO_ROOT / "docs" / "audit", ("*.md",)),+)++#: Root-level documents that publish claims.+SCAN_FILES: tuple[Path, ...] = (+ REPO_ROOT / "README.md",+ REPO_ROOT / "ROADMAP.md",+ REPO_ROOT / "RELEASES.md",+)++#: Where the real figure is pinned, beside the measurement that produces it.+REAL_MEASUREMENT = REPO_ROOT / "services" / "fusion" / "tests" / "test_alert_reduction_real.py"+_PINNED = re.compile(r"^PUBLISHED_REDUCTION_PCT\s*=\s*(?P<pct>\d+(?:\.\d+)?)\s*$", re.MULTILINE)++#: The legacy figure, in every spelling a surface has used for it.+LEGACY_FIGURE = re.compile(r"\b75\.3\s*%|\b0\.753\b")+++def published_reduction_pct() -> float:+ """The real figure, read from the test that measures it.++ Not a literal here. A gate that carries its own copy of the number it is+ policing cannot detect the number changing — it can only detect surfaces+ disagreeing with the gate, which is the circularity this repository has+ been bitten by before.+ """+ if not REAL_MEASUREMENT.is_file():+ raise SystemExit(f"{Path(sys.argv[0]).name}: {REAL_MEASUREMENT} missing — nothing pins the published figure")+ match = _PINNED.search(REAL_MEASUREMENT.read_text(encoding="utf-8"))+ if match is None:+ raise SystemExit(f"{Path(sys.argv[0]).name}: no PUBLISHED_REDUCTION_PCT in {REAL_MEASUREMENT.relative_to(REPO_ROOT)}")+ return float(match.group("pct"))+++def real_figure_pattern(pct: float) -> re.Pattern[str]:+ """Match the pinned figure in both spellings surfaces use: `33.3 %`, `0.333`."""+ as_pct = f"{pct:g}".replace(".", r"\.")+ as_ratio = f"{pct / 100:g}".replace(".", r"\.")+ return re.compile(rf"\b{as_pct}\s*%|\b{as_ratio}\b")+++#: Surfaces that publish the real figure in prose. Explicit, like+#: ``COUNT_BEARING_FILES`` in ``generate_connector_count.py``: when the+#: measurement moves, each of these has to move with it, and naming them is+#: what makes that a red build rather than a silent contradiction.+FIGURE_BEARING_SURFACES: tuple[Path, ...] = (+ REPO_ROOT / "apps" / "docs" / "docs" / "benchmark.md",+ REPO_ROOT / "apps" / "docs" / "docs" / "benchmarks" / "alert-reduction.md",+ REPO_ROOT / "apps" / "web" / "src" / "components" / "benchmark" / "ComparisonTable.tsx",+)++#: Any one of these, in the same file, counts as carrying the retraction.+#: Several spellings because the surfaces legitimately differ in register: a+#: claim-to-gate matrix row is terser than a benchmark page.+RETRACTION_MARKERS: tuple[re.Pattern[str], ...] = (+ re.compile(r"does not describe this product", re.IGNORECASE),+ re.compile(r"not describing this product", re.IGNORECASE),+ re.compile(r"\blegacy suite\b", re.IGNORECASE),+ re.compile(r"correlation_key", re.IGNORECASE),+)++#: A paragraph carrying one of these is *reporting* the withdrawn claim, not+#: making it. Documenting a retraction necessarily quotes the wording being+#: retracted, and a gate that cannot tell those apart forces the correction to+#: be written in circumlocutions.+NEGATION_MARKERS: tuple[re.Pattern[str], ...] = (+ re.compile(r"\bwas wrong\b", re.IGNORECASE),+ re.compile(r"\blong described\b", re.IGNORECASE),+ re.compile(r"\bretract", re.IGNORECASE),+ re.compile(r"\bwithdraw", re.IGNORECASE),+ re.compile(r"\b(is|was) not one\b", re.IGNORECASE),+ re.compile(r"\bnot a faithful\b", re.IGNORECASE),+)++#: Assertions that the legacy harness is equivalent to what ships. Each one+#: was live in the tree when this gate was written.+BANNED_CLAIMS: tuple[tuple[re.Pattern[str], str], ...] = (+ (+ re.compile(r"faithful[\s\-]+(in[\s\-]harness[\s\-]+)?re[\s\-]?implementation", re.IGNORECASE),+ 'calls the legacy harness a "faithful re-implementation" of production grouping',+ ),+ (+ re.compile(r"grouping rules\s*[-—,]\s*same logic", re.IGNORECASE),+ 'asserts the harness runs the "same logic" as the production grouping rules',+ ),+ (+ re.compile(r"same Tier 1\s*/\s*2\s*/\s*3 algorithm shipped", re.IGNORECASE),+ "claims the harness is the algorithm shipped in services/fusion",+ ),+)++#: The "Real measurement" label attached to the legacy suite, in the four+#: shapes it took: a section heading, a table row, a glossary entry, and a+#: badge on a console panel. None of these quotes 75.3 % or uses the word+#: "faithful", so the checks above saw nothing wrong with any of them — the+#: claim was carried entirely by the label. Matched within one paragraph so+#: an unrelated "Real measurement" elsewhere on the same page (the wet-eval+#: latency and token rows genuinely are measurements) is not swept up.+_MEASUREMENT_LABEL = re.compile(r"Real measurement", re.IGNORECASE)+_LEGACY_SUBJECT = re.compile(r"`?alert_reduction`?|alert[- ]reduction|alert reduction", re.IGNORECASE)+_PRODUCT_LOGIC = re.compile(r"correlation_key|product logic", re.IGNORECASE)++#: The alert-reduction suite card must not be badged as a real measurement.+#: Matched structurally rather than by prose: the badge is chosen by a `kind`+#: field inside the `alert_reduction` block of `SUITE_META`.+_SUITE_META_ALERT_REDUCTION = re.compile(+ r"alert_reduction:\s*\{(?P<body>.*?)\n\s{2}\},",+ re.DOTALL,+)+++def _enclosing_paragraph(text: str, index: int) -> str:+ """The blank-line-delimited block containing ``index``."""+ start = text.rfind("\n\n", 0, index)+ end = text.find("\n\n", index)+ return text[(0 if start < 0 else start) : (len(text) if end < 0 else end)]+++def _iter_surfaces() -> list[Path]:+ """Every published file this gate renders a verdict about."""+ found: list[Path] = []+ for root, globs in SCAN_ROOTS:+ if not root.is_dir():+ continue+ for pattern in globs:+ found.extend(p for p in root.rglob(pattern) if "node_modules" not in p.parts)+ found.extend(p for p in SCAN_FILES if p.is_file())+ return sorted(set(found))+++def violations(files: dict[Path, str], pct: float, *, whole_tree: bool = False) -> list[str]:+ """Every finding, as a rendered line. Pure, so the self-test can drive it.++ ``whole_tree`` enables the checks that are only meaningful over a complete+ scan — chiefly "a surface that must publish the figure is absent". Without+ it the self-test's injected snippets would trip that check and every case+ would report a finding regardless of what it was testing, which is a+ self-test that passes because it is broken.+ """+ found: list[str] = []+ real_figure = real_figure_pattern(pct)+ required = {p.relative_to(REPO_ROOT) for p in FIGURE_BEARING_SURFACES}+ seen_bearing: set[Path] = set()++ for path, text in sorted(files.items()):+ rel = path.relative_to(REPO_ROOT) if path.is_absolute() else path++ for pattern, description in BANNED_CLAIMS:+ for match in pattern.finditer(text):+ if any(n.search(_enclosing_paragraph(text, match.start())) for n in NEGATION_MARKERS):+ continue+ line = text[: match.start()].count("\n") + 1+ found.append(f"{rel}:{line}: {description} — {match.group(0)!r}")++ legacy = LEGACY_FIGURE.search(text)+ if legacy and not (real_figure.search(text) or any(m.search(text) for m in RETRACTION_MARKERS)):+ line = text[: legacy.start()].count("\n") + 1+ found.append(+ f"{rel}:{line}: quotes the legacy 75.3% alert-reduction figure without the retraction. "+ "Say that it describes a four-tier scheme the product does not run, and point at the "+ f"{pct:g}% measured against RawAlert.correlation_key()."+ )++ if rel in required:+ seen_bearing.add(rel)+ if not real_figure.search(text):+ found.append(+ f"{rel}: publishes the alert-reduction figure but does not quote {pct:g}%, which is what "+ "PUBLISHED_REDUCTION_PCT in services/fusion/tests/test_alert_reduction_real.py measures."+ )++ for match in _MEASUREMENT_LABEL.finditer(text):+ para = _enclosing_paragraph(text, match.start())+ if not _LEGACY_SUBJECT.search(para):+ continue+ # The real-key measurement is a measurement, and says so.+ if _PRODUCT_LOGIC.search(para) or any(n.search(para) for n in NEGATION_MARKERS):+ continue+ line = text[: match.start()].count("\n") + 1+ found.append(+ f'{rel}:{line}: labels the legacy alert-reduction suite a "Real measurement". '+ "It groups on four tiers of (rule_id, host, user); the product groups on "+ "{tenant}:{entity}:{tactic}."+ )++ block = _SUITE_META_ALERT_REDUCTION.search(text)+ if block and re.search(r"kind:\s*'measurement'", block.group("body")):+ line = text[: block.start()].count("\n") + 1+ found.append(+ f"{rel}:{line}: the alert_reduction suite is declared kind: 'measurement', which renders a "+ '"Real measurement" badge on a figure that describes an algorithm this product does not run.'+ )++ if whole_tree:+ for missing in sorted(required - seen_bearing):+ found.append(f"{missing}: named in FIGURE_BEARING_SURFACES but not present — remove it or restore the page")++ return found+++def check() -> int:+ surfaces = _iter_surfaces()+ if not surfaces:+ print(+ f"{Path(sys.argv[0]).name}: found no published surfaces under {REPO_ROOT} — refusing to report a clean tree it never opened",+ file=sys.stderr,+ )+ return 2++ pct = published_reduction_pct()+ files = {p: p.read_text(encoding="utf-8", errors="replace") for p in surfaces}+ quoting = sum(1 for text in files.values() if LEGACY_FIGURE.search(text))+ found = violations(files, pct, whole_tree=True)++ print(+ f"alert-reduction claims: scanned {len(files)} published surfaces, "+ f"{quoting} quote the legacy figure; the measured figure is {pct:g}%"+ )+ if found:+ print("\nthe 75.3% alert-reduction figure was retracted; these surfaces still assert it:", file=sys.stderr)+ for line in found:+ print(f" - {line}", file=sys.stderr)+ print(+ "\nSee apps/docs/docs/benchmark.md#why-there-are-two-alert-reduction-numbers for the wording to reuse.",+ file=sys.stderr,+ )+ return 1++ print("OK: every surface quoting the legacy figure carries the retraction.")+ return 0+++def _self_test() -> int:+ """Prove the gate detects each claim it forbids, on injected text."""+ pct = published_reduction_pct()+ real = f"{pct:g}%"+ fake = Path("apps/docs/docs/injected.md")++ cases: list[tuple[str, str]] = [+ (+ "flags a bare legacy figure with no retraction",+ "The harness reports a 75.3 % reduction ratio on a noisy stream.",+ ),+ (+ 'flags a "faithful re-implementation" claim',+ f"The code under test is a faithful in-harness re-implementation of the production rules. {real}",+ ),+ (+ 'flags a "same logic" claim',+ f"fed into the production Tier 1 / 2 / 3 grouping rules — same logic, no DB-backed dedup. {real}",+ ),+ (+ 'flags a "Real measurement" label on the legacy suite',+ "| `alert_reduction` | Real measurement | 1 000-alert noisy stream |",+ ),+ (+ "flags a Real measurement badge beside an alert-reduction heading",+ "<h3>Alert reduction (75.3%)</h3>\n<span>Real measurement</span>\nlegacy suite, 33.3%",+ ),+ (+ "flags kind: 'measurement' on the alert_reduction suite card",+ "const SUITE_META = {\n"+ " alert_reduction: {\n"+ " id: 'alert_reduction',\n"+ " kind: 'measurement',\n"+ f" blurb: 'see the {real} legacy suite note',\n"+ " },\n"+ "};\n",+ ),+ ]+ extra = [(name, bool(violations({fake: text}, pct))) for name, text in cases]++ # A surface on the bearing list that quotes the *old* figure after the+ # measurement moved. This is the case the previous design could not see,+ # because the gate carried its own copy of the number.+ extra.append(+ (+ "flags a bearing surface left on a superseded figure",+ bool(+ violations(+ {FIGURE_BEARING_SURFACES[0]: f"reduction is {pct + 1:g}% — legacy suite retained"},+ pct,+ )+ ),+ )+ )+ extra.append(+ (+ "passes the retracted wording benchmark.md already uses",+ not violations(+ {+ fake: (+ "| Alert reduction (legacy suite) | 75.3 % | A four-tier scheme implemented inside "+ "the test. Retained for continuity; does not describe this product. The real "+ f"correlation_key() measurement reports {real}."+ )+ },+ pct,+ ),+ )+ )+ extra.append(("passes the tree as committed", check() == 0))++ return self_test_main(Path(__file__).name, ["--check"], extra)+++def main(argv: list[str]) -> int:+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--check", action="store_true", help="Render the verdict (the default).")+ parser.add_argument("--self-test", action="store_true", help="Prove this gate detects the drift it claims to.")+ args = parser.parse_args(argv)+ return _self_test() if args.self_test else check()+++if __name__ == "__main__":+ sys.exit(main(sys.argv[1:]))diff --git a/scripts/check_scoreboard.py b/scripts/check_scoreboard.pyindex 5c46f22d4..6e1d3fcb7 100644--- a/scripts/check_scoreboard.py+++ b/scripts/check_scoreboard.py@@ -124,6 +124,32 @@ def _staleness_errors(data: dict, today: date | None = None) -> list[str]:]+def _version_errors(data: dict) -> list[str]:+ """The newest substrate row must name the version in the tree.++ `--refresh` already stamps `agent_version` from `VERSION`, but nothing+ verified it, so the newest row sat at `v8.1.1` while the tree read+ `10.0.0` and every check passed. The freshness gate below only reads the+ *date*, which a refresh keeps current — so a row could be two days old and+ still be labelled with a version two majors behind, which is the more+ misleading of the two. Older rows are history and are left alone.+ """+ tree = _tree_version()+ if not tree:+ return []+ row = _newest_substrate_row(data)+ if row is None:+ return []+ published = str(row.get("agent_version", ""))+ if published == tree:+ return []+ return [+ f"the newest substrate row is labelled {published or '(unset)'} but VERSION reads {tree}. "+ "A reader takes the top row as describing the shipping agent. Run: "+ "python3 scripts/check_scoreboard.py --refresh"+ ]++def _validate_schema(data: dict) -> list[str]:errors: list[str] = []try:@@ -156,7 +182,7 @@ def check() -> int:print(f"ERROR: {SCOREBOARD.relative_to(ROOT)} missing", file=sys.stderr)return 1data = _load()- errors = _validate_schema(data) + _validate_honesty(data) + _staleness_errors(data)+ errors = _validate_schema(data) + _validate_honesty(data) + _staleness_errors(data) + _version_errors(data)row = _newest_substrate_row(data)if row is None:diff --git a/scripts/generate_corpus_stats.py b/scripts/generate_corpus_stats.pynew file mode 100644index 000000000..383a85ed6--- /dev/null+++ b/scripts/generate_corpus_stats.py@@ -0,0 +1,366 @@+#!/usr/bin/env python3+"""Generate the single source of truth for AiSOC's detection / marketplace counts.++Why this exists+---------------+``generate_connector_count.py`` already proved the shape: derive the number+from the tree, commit the artefact, gate the artefact. Every *other* corpus+figure on the landing page was still hand-typed, and every one of them had+drifted:++ * ``Hero.tsx`` "6,998 detections ... 57 plugins"+ * ``ConnectorsMarquee.tsx`` "6,998 detections"+ * ``FeatureGrid.tsx`` "6,998 YAML rules", "7,117 community items"+ * ``Pillars.tsx`` stat "6,998" / "public detection rules"+ * ``ContributorLeaderboard.tsx`` "218 rules across 5 categories"++Three different wrong answers to one question, and no gate covering any of+them. Worse than stale: "6,998 detections" counts the imported corpus, ~85% of+which sits under ``detections/*/\\_quarantine/`` and is never loaded. Quoting it+as the detection capability presents quarantined rules as executable, which+this repository has a standing rule against.++So the artefact separates the two figures rather than picking one:++``executable``+ What the engine loads — ``services/fusion/app/data/detection_ruleset.json``.+ This is the capability number and the one the UI leads with.+``on_disk`` / ``quarantined``+ The corpus as indexed by ``marketplace/index.json``. Real, published, and+ only ever quoted *as* a corpus.++Cross-checking, in both directions+----------------------------------+The recurring defect in this tree's gates is the one-directional comparison:+it checks A against B, never B against A, so drift in the direction things+actually change slips through while the gate prints OK. Three independent+places count detections here — the compiled engine ruleset, the generated+truth table, and the marketplace index — so this reconciles all three against+each other before writing anything. A disagreement is a hard error, not a+silently-preferred source.++Outputs+-------+ * ``apps/web/src/data/corpus-stats.json`` — machine-readable payload.+ * ``apps/web/src/data/corpusStats.ts`` — TypeScript constants imported by+ every landing / marketplace / detections surface.++Usage+-----++ python3 scripts/generate_corpus_stats.py # write outputs+ python3 scripts/generate_corpus_stats.py --check # fail if drift+ python3 scripts/generate_corpus_stats.py --self-test # prove the gate works+"""++from __future__ import annotations++import argparse+import json+import re+import sys+from collections import Counter+from pathlib import Path+from typing import Any++# `scripts/` is on sys.path when this file is run as a program, but not when a+# test loads it by path with importlib. gate_toolkit sits beside it either way.+sys.path.insert(0, str(Path(__file__).resolve().parent))++from gate_toolkit import repo_root, self_test_main++REPO_ROOT = repo_root()+RULESET = REPO_ROOT / "services" / "fusion" / "app" / "data" / "detection_ruleset.json"+MARKETPLACE = REPO_ROOT / "marketplace" / "index.json"+TRUTH_TABLE = REPO_ROOT / "docs" / "detections" / "truth-table.md"+JSON_OUT = REPO_ROOT / "apps" / "web" / "src" / "data" / "corpus-stats.json"+TS_OUT = REPO_ROOT / "apps" / "web" / "src" / "data" / "corpusStats.ts"++#: Prose that quotes the executable-rule count outside the TypeScript surfaces.+#: Same contract as ``generate_connector_count.py``: each regex must match, and+#: the ``n`` group is rewritten in place (or reported, under ``--check``).+COUNT_BEARING_FILES: tuple[tuple[Path, tuple[re.Pattern[str], ...]], ...] = (+ (+ REPO_ROOT / "README.md",+ (+ re.compile(r"(?P<pre>fusion runs )(?P<n>\d+)(?P<post> executable detection rules)"),+ re.compile(r"(?P<pre>\| Detection engine \()(?P<n>\d+)(?P<post> executable rules\))"),+ ),+ ),+ (+ REPO_ROOT / "ROADMAP.md",+ (re.compile(r"(?P<pre>detection-evaluation worker \()(?P<n>\d+)(?P<post> executable rules on the stream\))"),),+ ),+)+++def _read_json(path: Path, label: str) -> Any:+ if not path.is_file():+ raise SystemExit(f"{Path(sys.argv[0]).name}: {label} missing at {path} — nothing to count")+ try:+ return json.loads(path.read_text(encoding="utf-8"))+ except json.JSONDecodeError as exc:+ raise SystemExit(f"{Path(sys.argv[0]).name}: {label} is not valid JSON: {exc}") from exc+++def read_engine_ruleset() -> tuple[int, dict[str, int]]:+ """Executable rule count and per-category split, from the compiled ruleset."""+ data = _read_json(RULESET, "detection ruleset")+ rules = data.get("rules")+ if not isinstance(rules, list) or not rules:+ raise SystemExit(f"{Path(sys.argv[0]).name}: {RULESET} declares no rules — refusing to publish a zero corpus")+ declared = data.get("count")+ if isinstance(declared, int) and declared != len(rules):+ raise SystemExit(f"{Path(sys.argv[0]).name}: ruleset `count` is {declared} but it holds {len(rules)} rules")+ categories = Counter(str(rule.get("category", "uncategorized")) for rule in rules)+ return len(rules), dict(sorted(categories.items()))+++def read_marketplace() -> dict[str, int]:+ """Corpus figures from the marketplace index's own published stats.++ The stats block is read rather than recomputed from ``items`` so the+ artefact and the marketplace page can never disagree, and the two are+ reconciled below so a broken ``build_marketplace.py`` cannot pass either.+ """+ data = _read_json(MARKETPLACE, "marketplace index")+ stats = data.get("stats")+ items = data.get("items")+ if not isinstance(stats, dict) or not isinstance(items, list) or not items:+ raise SystemExit(f"{Path(sys.argv[0]).name}: {MARKETPLACE} has no stats/items — refusing to publish a zero corpus")++ counted = Counter(str(item.get("type", "")) for item in items)+ for key, kind in (("detections", "detection"), ("plugins", "plugin"), ("playbooks", "playbook")):+ published = stats.get(key)+ if published != counted[kind]:+ raise SystemExit(+ f"{Path(sys.argv[0]).name}: marketplace stats.{key}={published} but {counted[kind]} "+ f"{kind!r} items are indexed — regenerate with `python3 scripts/build_marketplace.py`"+ )++ out = {+ "on_disk": int(stats["detections"]),+ "quarantined": int(stats.get("quarantined", 0)),+ "plugins": int(stats["plugins"]),+ "playbook_packs": int(stats["playbooks"]),+ "marketplace_items": int(stats.get("total", len(items))),+ }+ if out["marketplace_items"] != len(items):+ raise SystemExit(f"{Path(sys.argv[0]).name}: marketplace stats.total={out['marketplace_items']} but {len(items)} items are indexed")+ return out+++_TRUTH_TABLE_EXECUTABLE = re.compile(r"\|\s*\*\*executable \(loaded by the engine\)\*\*\s*\|\s*\*\*(?P<n>\d+)\*\*\s*\|")+++def read_truth_table_executable() -> int:+ """The executable count as published in ``docs/detections/truth-table.md``.++ A third opinion on the same number. The truth table is itself generated+ (``scripts/detection_truth_table.py``), so this is not belt-and-braces: it+ catches the case where one generator has been re-run and the other has not.+ """+ if not TRUTH_TABLE.is_file():+ raise SystemExit(f"{Path(sys.argv[0]).name}: detection truth table missing at {TRUTH_TABLE}")+ match = _TRUTH_TABLE_EXECUTABLE.search(TRUTH_TABLE.read_text(encoding="utf-8"))+ if match is None:+ raise SystemExit(f"{Path(sys.argv[0]).name}: could not find the executable row in {TRUTH_TABLE.relative_to(REPO_ROOT)}")+ return int(match.group("n"))+++def build_payload() -> dict[str, Any]:+ executable, categories = read_engine_ruleset()+ corpus = read_marketplace()+ documented = read_truth_table_executable()++ if documented != executable:+ raise SystemExit(+ f"{Path(sys.argv[0]).name}: the engine loads {executable} rules but "+ f"docs/detections/truth-table.md publishes {documented}. Re-run "+ "`python3 scripts/detection_truth_table.py` and `python3 scripts/export_detection_ruleset.py`."+ )+ if executable > corpus["on_disk"]:+ raise SystemExit(f"{Path(sys.argv[0]).name}: {executable} executable rules exceeds {corpus['on_disk']} indexed on disk")+ if corpus["quarantined"] > corpus["on_disk"]:+ raise SystemExit(f"{Path(sys.argv[0]).name}: {corpus['quarantined']} quarantined exceeds {corpus['on_disk']} on disk")++ return {+ "executable": executable,+ "onDisk": corpus["on_disk"],+ "quarantined": corpus["quarantined"],+ "plugins": corpus["plugins"],+ "playbookPacks": corpus["playbook_packs"],+ "marketplaceItems": corpus["marketplace_items"],+ "categories": categories,+ "generatedFrom": [+ "services/fusion/app/data/detection_ruleset.json",+ "marketplace/index.json",+ "docs/detections/truth-table.md",+ ],+ "regenerateWith": "python3 scripts/generate_corpus_stats.py",+ }+++def render_typescript(payload: dict[str, Any]) -> str:+ """Render the TS constants module imported by the landing / detections UI."""+ return (+ "// AUTO-GENERATED by scripts/generate_corpus_stats.py.\n"+ "// Do not edit by hand. Run the script (or `make corpus-stats`) instead.\n"+ "//\n"+ "// Single source of truth for every published detection / marketplace count.\n"+ "//\n"+ "// EXECUTABLE_DETECTION_COUNT is the capability number: rules the fusion\n"+ "// engine actually loads. DETECTIONS_ON_DISK is the indexed corpus, most of\n"+ "// which is quarantined and never evaluated -- it is a corpus figure and must\n"+ "// never be presented as what the product executes.\n"+ "\n"+ "import data from './corpus-stats.json';\n"+ "\n"+ "/** Rules the fusion engine loads and evaluates. */\n"+ "export const EXECUTABLE_DETECTION_COUNT: number = data.executable;\n"+ "\n"+ "/** Detection YAML indexed in the marketplace, quarantined rules included. */\n"+ "export const DETECTIONS_ON_DISK: number = data.onDisk;\n"+ "\n"+ "/** Imported rules held under `_quarantine/`: on disk, never loaded. */\n"+ "export const QUARANTINED_DETECTIONS: number = data.quarantined;\n"+ "\n"+ "export const PLUGIN_COUNT: number = data.plugins;\n"+ "export const PLAYBOOK_PACK_COUNT: number = data.playbookPacks;\n"+ "export const MARKETPLACE_ITEM_COUNT: number = data.marketplaceItems;\n"+ "\n"+ "/** Executable rules per detection category. */\n"+ "export const DETECTION_CATEGORIES: Readonly<Record<string, number>> = data.categories;\n"+ "export const DETECTION_CATEGORY_COUNT: number = Object.keys(data.categories).length;\n"+ "\n"+ "/** Locale-formatted labels, so no surface re-implements the separator. */\n"+ "export const EXECUTABLE_DETECTIONS_LABEL = `${EXECUTABLE_DETECTION_COUNT.toLocaleString()} executable detections`;\n"+ "export const MARKETPLACE_ITEMS_LABEL = `${MARKETPLACE_ITEM_COUNT.toLocaleString()} marketplace items`;\n"+ )+++def write_outputs(payload: dict[str, Any]) -> None:+ JSON_OUT.parent.mkdir(parents=True, exist_ok=True)+ JSON_OUT.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")+ TS_OUT.write_text(render_typescript(payload), encoding="utf-8")+++def artefact_drift(payload: dict[str, Any], existing_json: str | None, existing_ts: str | None) -> list[str]:+ """Drift between a freshly-derived payload and what is committed.++ Pure, so ``--self-test`` can drive it with a hand-edited copy instead of+ mutating the working tree to prove the gate still bites.+ """+ drift: list[str] = []+ want_json = json.dumps(payload, indent=2) + "\n"+ if existing_json != want_json:+ drift.append(f"{JSON_OUT.relative_to(REPO_ROOT)}: regenerate with `python3 scripts/generate_corpus_stats.py`")+ if existing_ts != render_typescript(payload):+ drift.append(f"{TS_OUT.relative_to(REPO_ROOT)}: regenerate with `python3 scripts/generate_corpus_stats.py`")+ return drift+++def reconcile_prose(executable: int, *, check_only: bool) -> list[str]:+ """Rewrite or verify every COUNT_BEARING_FILES entry."""+ drift: list[str] = []+ for path, patterns in COUNT_BEARING_FILES:+ if not path.exists():+ drift.append(f"missing file {path.relative_to(REPO_ROOT)}")+ continue+ before = path.read_text(encoding="utf-8")+ after = before+ for pattern in patterns:+ matches = list(pattern.finditer(after))+ if not matches:+ drift.append(f"{path.relative_to(REPO_ROOT)}: pattern {pattern.pattern!r} did not match")+ continue+ # Rewrite right-to-left so earlier match offsets stay valid.+ for match in reversed(matches):+ if match.group("n") != str(executable):+ after = after[: match.start("n")] + str(executable) + after[match.end("n") :]+ if after != before:+ if check_only:+ drift.append(f"{path.relative_to(REPO_ROOT)}: executable-rule count drift (expected {executable})")+ else:+ path.write_text(after, encoding="utf-8")+ return drift+++def _self_test() -> int:+ """Prove the gate detects a hand edit, then that it refuses an empty tree."""+ payload = build_payload()+ committed_json = JSON_OUT.read_text(encoding="utf-8") if JSON_OUT.exists() else None+ committed_ts = TS_OUT.read_text(encoding="utf-8") if TS_OUT.exists() else None++ tampered = dict(payload)+ tampered["executable"] = int(payload["executable"]) + 1+ hand_edited_json = json.dumps(tampered, indent=2) + "\n"++ # The TS module reads its numbers from the JSON, so tampering with the+ # payload leaves the rendered TS byte-identical. A hand edit to that file+ # therefore means editing the *text* — the shape this catches is someone+ # pinning a literal to stop the import "changing under them".+ hand_edited_ts = render_typescript(payload).replace(+ "export const EXECUTABLE_DETECTION_COUNT: number = data.executable;",+ "export const EXECUTABLE_DETECTION_COUNT: number = 6998;",+ )++ extra = [+ (+ "flags a hand-edited count in corpus-stats.json",+ bool(artefact_drift(payload, hand_edited_json, committed_ts)),+ ),+ (+ "flags a hand-edited literal in corpusStats.ts",+ bool(artefact_drift(payload, committed_json, hand_edited_ts)),+ ),+ (+ "passes on the committed artefact",+ not artefact_drift(payload, committed_json, committed_ts),+ ),+ ]+ return self_test_main(Path(__file__).name, ["--check"], extra)+++def main(argv: list[str]) -> int:+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--check", action="store_true", help="Fail (exit 1) on any drift instead of rewriting files.")+ parser.add_argument("--self-test", action="store_true", help="Prove this gate detects the drift it claims to.")+ args = parser.parse_args(argv)++ if args.self_test:+ return _self_test()++ payload = build_payload()+ executable = int(payload["executable"])+ drift: list[str] = []++ if args.check:+ drift.extend(+ artefact_drift(+ payload,+ JSON_OUT.read_text(encoding="utf-8") if JSON_OUT.exists() else None,+ TS_OUT.read_text(encoding="utf-8") if TS_OUT.exists() else None,+ )+ )+ else:+ write_outputs(payload)++ drift.extend(reconcile_prose(executable, check_only=args.check))++ if drift:+ print(f"corpus-stats drift detected (executable={executable}):", file=sys.stderr)+ for line in drift:+ print(f" - {line}", file=sys.stderr)+ return 1++ print(+ f"corpus-stats OK — {executable} executable across {len(payload['categories'])} categories, "+ f"{payload['onDisk']} indexed on disk ({payload['quarantined']} quarantined), "+ f"{payload['plugins']} plugins, {payload['playbookPacks']} playbook packs"+ )+ return 0+++if __name__ == "__main__":+ sys.exit(main(sys.argv[1:]))diff --git a/scripts/readme_gates.py b/scripts/readme_gates.pyindex ce3dfff48..60bdb8625 100644--- a/scripts/readme_gates.py+++ b/scripts/readme_gates.py@@ -398,7 +398,34 @@ CLAIM_MATRIX = REPO_ROOT / "docs" / "audit" / "CLAIM_TO_GATE_MATRIX.md"#: same way the README is: a figure repeated in prose drifts from its#: source the first time the source changes, and a compliance page#: quoting a stale number is worse than one quoting none.-FIGURE_DOCS = (REPO_ROOT / "apps" / "docs" / "docs" / "compliance" / "evidence-pack.md",)+#:+#: `ROADMAP.md` was the omission that proved the point. Its tally line even+#: instructs the reader to recount with the script "rather than trusting a+#: figure quoted in prose — this line has gone stale before", and it had gone+#: stale again (136/128 against a matrix holding 139/131) with every check in+#: the repository green, because this tuple listed one compliance page and the+#: governance documents were not in it.+FIGURE_DOCS = (+ REPO_ROOT / "apps" / "docs" / "docs" / "compliance" / "evidence-pack.md",+ REPO_ROOT / "ROADMAP.md",+ REPO_ROOT / "RELEASES.md",+)++#: A figure the prose explicitly dates is a record, not a claim about now.+#: `RELEASES.md` deliberately quotes the tally as it stood at an older+#: release; holding that to today's count would force the history to be+#: rewritten every time the matrix grows.+_HISTORICAL_QUOTE = re.compile(+ r"at that time|at the time|as it stood|as of v\d|at the v\d[\d.]* cut",+ re.IGNORECASE,+)+++def _is_historical(text: str, index: int) -> bool:+ """Whether the figure at ``index`` sits in a sentence that dates itself."""+ start = text.rfind("\n", 0, index) + 1+ end = text.find("\n", index)+ return bool(_HISTORICAL_QUOTE.search(text[start : len(text) if end < 0 else end]))def _truth_table_executable() -> int | None:@@ -429,6 +456,24 @@ def _matrix_counts() -> tuple[int, int] | None:return (gated, partial) if (gated or partial) else None+def _matrix_row_total() -> int | None:+ """Every data row, `NO GATE` included.++ `_matrix_counts` deliberately skips `NO GATE` rows, so the tally prose —+ "N rows — G GATED / P PARTIAL / Z NO GATE" — had its *leading* figure+ checked by nothing. Returned separately rather than widening that+ function's tuple, which is unpacked as a pair by its tests.+ """+ if not CLAIM_MATRIX.exists():+ return None+ rows = sum(+ 1+ for line in _read(CLAIM_MATRIX).splitlines()+ if line.lstrip().startswith("|") and ("NO GATE" in line or "PARTIAL" in line or "GATED" in line)+ )+ return rows or None++def gate_readme_figures() -> list[GateFailure]:"""Detection and claim-gate figures in the README must match their source."""failures: list[GateFailure] = []@@ -480,6 +525,8 @@ def gate_readme_figures() -> list[GateFailure]:for label, text in sources:for m in re.finditer(r"(\d+)\s+(?:rows\s+)?`?GATED`?[,/\s]+(?:and\s+)?(\d+)\s+`?PARTIAL", text):+ if _is_historical(text, m.start()):+ continueif (int(m.group(1)), int(m.group(2))) != (gated, partial):failures.append(GateFailure(@@ -489,6 +536,18 @@ def gate_readme_figures() -> list[GateFailure]:f"{gated} GATED / {partial} PARTIAL.",))+ total = _matrix_row_total()+ if total is not None:+ for m in re.finditer(r"(\d+)\s+rows\s*[—–-]\s*\d+\s+`?GATED", text):+ if _is_historical(text, m.start()):+ continue+ if int(m.group(1)) != total:+ failures.append(+ GateFailure(+ "readme-figures",+ f"{label} claims {m.group(1)} matrix rows; docs/audit/CLAIM_TO_GATE_MATRIX.md holds {total}.",+ )+ )# The bullet-list form the matrix's Summary uses. The inline# pattern above needs both figures on one line and silently# matched nothing here, which is how the stale count survived.diff --git a/services/fusion/tests/test_alert_reduction_real.py b/services/fusion/tests/test_alert_reduction_real.pyindex 70e9fb8e5..a078120d8 100644--- a/services/fusion/tests/test_alert_reduction_real.py+++ b/services/fusion/tests/test_alert_reduction_real.py@@ -43,6 +43,17 @@ WINDOW = timedelta(seconds=3600)#: comparable, which is the point of keeping both.STREAM_SIZE = 1000+#: The figure the docs and the console publish, as a percentage.+#:+#: Pinned here, beside the measurement, because the plausible band below is+#: deliberately wide and would not notice the ratio moving from 33.3 % to+#: 34.1 % — while every published surface went on quoting the old one. The+#: band asks "is the correlation still sane"; this asks "is the number we+#: printed still the number". `scripts/check_alert_reduction_claims.py` reads+#: this constant and holds every published surface to it, so the chain from+#: measurement to prose has no hand-copied link in it.+PUBLISHED_REDUCTION_PCT = 33.3+TACTICS = ["initial-access","execution",@@ -140,6 +151,12 @@ def test_reduction_is_measured_against_the_real_correlation_key() -> None:f"reduction {reduction:.1%} is outside the plausible band; either correlation stopped grouping or it is collapsing unrelated alerts")+ assert round(reduction * 100, 1) == PUBLISHED_REDUCTION_PCT, (+ f"reduction is now {reduction:.1%} but PUBLISHED_REDUCTION_PCT says {PUBLISHED_REDUCTION_PCT} %, "+ "which is what the benchmark pages print. Update the constant and re-run "+ "`python3 scripts/check_alert_reduction_claims.py` so the published surfaces move with it."+ )+def test_the_key_is_the_products_key_not_a_local_one() -> None:"""Pins the dimensions, so a change to `correlation_key()` that alters