Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
docs: add alert-reduction benchmark page (Stage 2 #7) (#95)
Beenu Arora4cab499983f7
diff --git a/apps/docs/docs/benchmarks/alert-reduction.md b/apps/docs/docs/benchmarks/alert-reduction.mdnew file mode 100644index 000000000..314cef9d6--- /dev/null+++ b/apps/docs/docs/benchmarks/alert-reduction.md@@ -0,0 +1,322 @@+---+sidebar_position: 1+title: Alert Reduction+description: How the AiSOC alert-fusion substrate collapses a noisy 1,000-alert stream into ~247 incidents while losing zero critical alerts. Methodology, latest measured numbers, reproduction, and CI artefact location.+---++# Alert Reduction++> **Synthetic, deterministic, reproducible in under a second.** This page+> documents the AiSOC alert-reduction benchmark — what it measures, how the+> 3-tier fusion logic collapses a noisy alert stream into actionable+> incidents, and how to reproduce the numbers locally.+>+> The headline number on the [Public Eval Harness](../benchmark.md) page+> (`75.3 %`) is the output of this benchmark. This page is the long-form+> companion: the dataset shape, the per-tier collapse breakdown, the+> critical-alert preservation guarantee, and the road to per-rule false-positive+> rates (tracked under [issue #5](https://github.com/beenuar/AiSOC/issues/5)).++[](#latest-results)++:::warning Read this first+This is a **synthetic** benchmark. The 1,000-alert stream is generated by+`generate_noisy_alert_stream` in+[`services/agents/tests/test_alert_reduction.py`](https://github.com/beenuar/AiSOC/blob/main/services/agents/tests/test_alert_reduction.py)+with a fixed RNG seed; the fusion code under test is a faithful in-harness+re-implementation of the production Tier 1 / Tier 2 / Tier 3 grouping rules in+`services/fusion`, **minus** the DB-backed deduplicator and the ML scorer. It+runs deterministically on every PR targeting `main` / `develop` and gates+fusion-logic regressions in milliseconds. It is **not** an end-to-end+measurement of the production fusion service against real customer telemetry.+:::++## TL;DR++| Metric | Value | Target |+|---|---|---|+| **Reduction ratio** | **75.3 %** (1,000 alerts → 247 incidents) | ≥ 70 % |+| Critical alerts preserved | **93 / 93** (zero drop) | 100 % |+| Storm incidents collapsed (Tier 3) | 16 (avg ~8.9 alerts each, 3–8 distinct hosts) | ≥ 1 |+| Determinism | Same RNG seed → same byte-for-byte output | Required |+| Wall time | ~8 ms on a laptop | < 100 ms |++The current snapshot lives at+[`eval-results/eval/results/latest.json`](https://github.com/beenuar/AiSOC/blob/eval-results/eval/results/latest.json)+under `suites.alert_reduction`. Every successful build on `main` / `develop`+appends a new commit-keyed entry alongside it (see+[CI artefact location](#ci-artefact-location)).++## What this measures++A SOC's first-order pain isn't "did we catch the threat" — it's "did we drown+in the catch." Modern detection stacks routinely emit thousands of alerts per+hour, and the same underlying event (a single misconfigured agent, one phish+campaign, one credential-stuffing burst) can light up dozens of rules at once.+The job of fusion is to take that raw rule output and emit a small,+deduplicated, human-reviewable list of **incidents**.++This benchmark answers exactly one question:++> **Given a controlled, realistic distribution of noise, what fraction of+> incoming alerts collapse into incidents — and does the substrate drop any+> of the alerts that an analyst genuinely needed to see?**++It is a **real measurement**, not a self-consistency gate. The fusion grouping+logic is the same Tier 1/2/3 algorithm shipped in `services/fusion` — a+regression in the grouping rules will move this number. The other three+substrate suites on the [public eval page](../benchmark.md) are tautological+by design (dataset and judge written together to gate template breakage); this+one is not.++## The synthetic dataset++The 1,000-alert stream is generated by+[`generate_noisy_alert_stream`](https://github.com/beenuar/AiSOC/blob/main/services/agents/tests/test_alert_reduction.py)+with a fixed seed so every run is byte-for-byte identical. The mix is+deliberately shaped to look like the worst hour of a real SIEM:++| Bucket | Share | What it represents |+|---|---|---|+| Pure duplicates | ~25 % | Same `(rule, host, user)` within the same minute — the agent re-firing on the same telemetry |+| Near-duplicates | ~30 % | Same `(rule, host)`, different user **or** ±5-minute window — same root event, slight metadata drift |+| Storms | ~15 % | One rule firing across many hosts within a 5-minute window — campaign / worm / misconfigured push |+| Benign noise | ~10 % | Low-score rules (`score < 0.35`) — the chatter every SOC silences but the SIEM still emits |+| Unique high-signal events | ~20 % | Distinct `(rule, host, user)` events with no friends — the things you actually want to look at |++The full distribution from the latest run:++| Severity in | Alerts | Incidents out (post-fusion) |+|---|---:|---:|+| `critical` | 93 | 38 |+| `high` | 417 | 130 |+| `medium` | 303 | 79 |+| `low` | 187 | 0 (collapsed below the noise floor) |+| **Total** | **1,000** | **247** |++The `low`-severity bucket disappears entirely: those alerts carry `score < 0.35`+by construction and get dropped by the noise threshold. This is intentional —+the benign-noise bucket exists to verify the noise-floor cut still fires, not+to be shown to a human.++## The 3-tier fusion logic++The grouping algorithm has three rules, applied in order. The exact code lives+in [`fuse_alerts`](https://github.com/beenuar/AiSOC/blob/main/services/agents/tests/test_alert_reduction.py)+inside the test file (so the benchmark stays self-contained) and mirrors the+production logic in+[`services/fusion`](https://github.com/beenuar/AiSOC/tree/main/services/fusion).++### Tier 1 — strict identity collapse++Same `(rule_id, host, user)` within a **10-minute window** → 1 incident.+This is the "the agent re-fired on the same telemetry" case. Most pure+duplicates and same-source loops collapse here.++### Tier 2 — host-scoped merge++Same `(rule_id, host)` within a **30-minute window**, but spanning multiple+users (or the same user with ±5-minute drift) → merged into one Tier-1+incident. This catches "same root event, slightly different metadata"+duplicates, including a single attacker pivoting between local accounts on+the same host.++### Tier 3 — rule-storm collapse++Same `rule_id` firing within a **5-minute window** across **≥ 3 distinct+hosts** → one "storm" incident with `host = <storm:N-hosts>`. This is the+"campaign / worm / misconfigured deployment push" case — the kind of event+that historically generates a hundred near-identical tickets.++### Noise-floor drop++Any incident with aggregate `score < 0.35` is dropped before being emitted.+This is the configurable knob a tenant uses to silence chronically chatty+rules without de-listing them; in the benchmark it means the 187 `low`+alerts vanish.++## Latest results {#latest-results}++The numbers below are produced by `python3 scripts/run_evals.py --json` on the+current `main` and rendered into `eval/results/latest.json` on the+`eval-results` branch by the CI gate.++### Headline++| Field | Value |+|---|---|+| `metric` | `reduction_ratio` |+| `value` | `0.753` |+| `target` | `0.70` |+| `passed` | `true` |+| `duration_ms` | `7.9` |++### Collapse breakdown++| Bucket | Count | Avg alerts collapsed | Notes |+|---|---:|---:|---|+| Tier 1/2 multi-member incidents (non-storm) | 56 | ~8.8 alerts each | Same root event, deduplicated |+| Tier 3 storm incidents | 16 | ~8.9 alerts each | 3 – 8 distinct hosts per storm |+| Single-member incidents | 175 | 1 each | Genuine unique signal — exactly what an analyst needs to triage |+| **Total emitted incidents** | **247** | — | from 1,000 raw alerts |++The 175 single-member incidents are the floor: those are the alerts that+*should* survive fusion, because they represent distinct events. Anything+beyond that floor is a deduplication win.++### Critical-alert preservation++| Severity tier | Alerts in | Preserved in incidents | Dropped |+|---|---:|---:|---:|+| `critical` | 93 | **93 (100 %)** | **0** |++This is the non-negotiable invariant: **fusion never silently drops a critical+alert.** The substrate may merge a critical alert into a multi-member incident+(it does — 93 critical alerts are represented across the 38 critical incidents+emitted), but it will never delete one. The+[`test_critical_alerts_never_dropped`](https://github.com/beenuar/AiSOC/blob/main/services/agents/tests/test_alert_reduction.py)+test asserts this directly and is part of the same CI gate.++### Determinism++Two consecutive runs on the same commit produce byte-for-byte identical+output. The benchmark uses a fixed RNG seed inside `generate_noisy_alert_stream`,+and the fusion algorithm is order-stable on its inputs. This matters for two+reasons:++1. **CI flakiness has a single root cause.** If two CI runs disagree on the+ reduction ratio, it's a code change, not a coincidence.+2. **Diff-friendly regression review.** When a PR moves the number, you can+ inspect *which* incidents changed shape (single-member ↔ multi-member ↔+ storm) instead of being told "the noise moved."++The+[`test_run_is_deterministic`](https://github.com/beenuar/AiSOC/blob/main/services/agents/tests/test_alert_reduction.py)+test enforces this — it runs `fuse_alerts` twice and compares the incident+lists element-for-element.++## Reproduce locally++From a fresh clone:++```bash+git clone https://github.com/beenuar/AiSOC && cd AiSOC+python3 scripts/run_evals.py+```++Expected output (excerpt):++```text+[PASS] alert_reduction reduction_ratio 0.753 (target >= 0.70)+```++For machine-readable output:++```bash+python3 scripts/run_evals.py --json --out /tmp/eval.json+python3 -c "import json; print(json.dumps(json.load(open('/tmp/eval.json'))['suites']['alert_reduction'], indent=2))"+```++To run only the alert-reduction tests under `pytest`:++```bash+cd services/agents+python3 -m pytest tests/test_alert_reduction.py -v+```++That covers four assertions:++- `test_reduction_ratio_meets_target` — the headline `≥ 70 %` gate+- `test_critical_alerts_never_dropped` — the zero-loss invariant+- `test_run_is_deterministic` — same seed → same output+- `test_storms_collapse` — Tier 3 fires when ≥ 3 hosts share a rule in a 5-minute window++## CI artefact location++Every push to `main` or `develop` triggers+[`.github/workflows/ci.yml`](https://github.com/beenuar/AiSOC/blob/main/.github/workflows/ci.yml),+which runs `scripts/run_evals.py --ci --out report.json`. On success, the+report is committed to the orphan+[`eval-results`](https://github.com/beenuar/AiSOC/tree/eval-results) branch:++```text+eval/results/<commit_sha>.json # one snapshot per commit+eval/results/latest.json # most recent passing build on main+eval/results/badge-reduction.json # shields.io endpoint backing the badge above+```++Reading the latest snapshot from CI:++```bash+curl -s https://raw.githubusercontent.com/beenuar/AiSOC/eval-results/eval/results/latest.json \+ | python3 -c "import json,sys; d=json.load(sys.stdin); print(json.dumps(d['suites']['alert_reduction'], indent=2))"+```++Drop that into a dashboard, a Slack notifier, or a tenant-facing report — it's+the same JSON shape `run_evals.py` writes locally.++## What this benchmark does *not* tell you++In the spirit of being painfully honest about synthetic vs. real:++- **It does not measure the production fusion service end-to-end.** The+ in-harness `fuse_alerts` is a faithful re-implementation of the Tier 1/2/3+ rules from `services/fusion`, but the production service additionally runs+ a Postgres-backed deduplicator (handles cross-window persistence) and an ML+ scorer (lifts and depresses scores based on historical FP rate). Those+ components have their own unit tests; they're out of scope here because+ this gate has to run in milliseconds in CI.+- **It does not exercise real telemetry.** The 1,000-alert stream is shaped+ to look like the worst hour of a SIEM, but it isn't a capture from one.+ Real-customer evaluation is on the v1.5 roadmap and will be opt-in /+ federated; see [`apps/docs/docs/benchmark.md`](../benchmark.md#what-this-is-not).+- **It does not compute per-rule false-positive rates.** That is the next+ stage of this benchmark, tracked under+ [issue #5 — per-rule cross-fire FP eval gate](https://github.com/beenuar/AiSOC/issues/5)+ ([PR #89](https://github.com/beenuar/AiSOC/pull/89), in flight). Once that+ ships, this page will gain a "per-rule FP rate" section that surfaces which+ rules contribute most to the noise floor and how their merge rate trends+ over time.+- **It does not score "alerts an analyst would have wanted."** That requires+ labeled human-judgement data, which we don't have synthetically. The+ closest proxy here is the critical-alert preservation invariant — *zero*+ alerts above the noise floor are silently dropped.++## Roadmap++The pieces that turn this from a substrate gate into a buyer-grade benchmark:++- **Per-rule FP rate breakdown** — depends on+ [issue #5](https://github.com/beenuar/AiSOC/issues/5) /+ [PR #89](https://github.com/beenuar/AiSOC/pull/89). Will surface per-rule+ contribution to the noise floor and gate net-new noisy detections in CI.+- **Per-tenant fusion config replay** — let an operator paste their fusion+ knobs (window sizes, noise threshold, severity overrides) and replay the+ stream against them. Lets a buyer answer "what would *my* number look like"+ before deploying.+- **Live-fusion online benchmark** — once the production+ [`services/fusion`](https://github.com/beenuar/AiSOC/tree/main/services/fusion)+ service has a stable replay API, run the same 1,000-alert stream through the+ real DB-backed deduplicator + ML scorer in a nightly job. The delta vs.+ this in-harness number is the "what does the substrate gate miss" signal.+- **Federated, opt-in real-tenant evaluation** — collapsed against tenant+ telemetry under DP-style aggregation so the headline number reflects a+ population, not a single deployment.++PRs against any of those are welcome — the+[`Help us harden the harness`](../benchmark.md#help-us-harden-the-harness)+section on the umbrella benchmark page lists the contribution path.++## See also++- [Public Eval Harness](../benchmark.md) — the umbrella page covering all five+ CI-gated suites (MITRE accuracy, alert reduction, investigation+ completeness, response quality, playbook completion).+- [Synthetic telemetry corpus](../benchmark.md#5-synthetic-telemetry-corpus) —+ the 14-source backing event corpus that connector and Sigma rule PRs wire+ against.+- [`scripts/run_evals.py`](https://github.com/beenuar/AiSOC/blob/main/scripts/run_evals.py) —+ the unified runner that produces the JSON snapshot consumed by this page.+- [`services/agents/tests/test_alert_reduction.py`](https://github.com/beenuar/AiSOC/blob/main/services/agents/tests/test_alert_reduction.py) —+ the test file containing both the synthetic stream generator and the+ in-harness fusion code under test.diff --git a/apps/docs/docs/connectors/osctrl.md b/apps/docs/docs/connectors/osctrl.mdindex 24cc62331..696501a51 100644--- a/apps/docs/docs/connectors/osctrl.md+++ b/apps/docs/docs/connectors/osctrl.md@@ -14,10 +14,10 @@ from the osquery table that produced the row, so detection rules in`startup_items`) to high without having to reason about vendor-specific severityfields.-This connector is read-only on its own. The companion [aisoc-direct-TLS endpoint](/docs/connectors/aisoc-direct) and the live-query response action-(see [agent capabilities](/docs/concepts/capabilities)) build on the same-auth surface to dispatch ad-hoc queries from playbooks.+This connector is read-only on its own. The companion AiSOC TLS endpoint+(`aisoc-direct`, ships as a separate connector page) and the live-query+response action (see [agent capabilities](/docs/concepts/capabilities)) build+on the same auth surface to dispatch ad-hoc queries from playbooks.## What you getdiff --git a/apps/docs/docs/operations/air-gapped.md b/apps/docs/docs/operations/air-gapped.mdindex a1e1ae45b..8e1e553d3 100644--- a/apps/docs/docs/operations/air-gapped.md+++ b/apps/docs/docs/operations/air-gapped.md@@ -215,5 +215,5 @@ enforcement.## Related pages* [Credentials & Vault](credentials.md) — how tenant secrets are encrypted at rest-* [API configuration](../getting-started/configuration.md) — all environment variables+* [Environment variables](../deployment/env-vars.md) — all configuration variables for the API and worker services* [Plugin SDK](../plugins/overview.md) — build connectors that work under air-gapdiff --git a/apps/docs/sidebars.ts b/apps/docs/sidebars.tsindex 470a9baae..213196b51 100644--- a/apps/docs/sidebars.ts+++ b/apps/docs/sidebars.ts@@ -67,6 +67,11 @@ const sidebars: SidebarsConfig = {label: "Detections",items: ["detections/coverage"],},+ {+ type: "category",+ label: "Benchmarks",+ items: ["benchmarks/alert-reduction"],+ },{type: "category",label: "Operations",