Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

Finish the plan: store migrations for all three stores, and the derived fields (#707)

Beenu Arorae979ed965b36
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index ea3988be5..2c6041422 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -254,6 +254,13 @@ jobs:
# Redaction is allow-list rather than deny-list, so a setting added
# tomorrow is redacted by default rather than exposed until someone
# remembers — and these tests are what keep it that way.
+ # A panel for a metric nothing exports reads "No data" forever,
+ # which teaches people that empty panels are normal — and then a
+ # genuinely dead service looks like an uninstrumented one. Also
+ # checks datasource uids resolve: Grafana does not warn about a
+ # dangling reference, the link just silently does nothing.
+ - name: Grafana dashboards query metrics that exist
+ run: python scripts/check_grafana_dashboards.py
- name: Support bundle must not leak
run: |
python3 -m pip install --quiet pytest
diff --git a/README.md b/README.md
index f44fbb577..8e1c236e7 100644
--- a/README.md
+++ b/README.md
@@ -172,7 +172,7 @@ Full architecture (every service, every storage role, the v1.5 console workbench
A handful of headline capabilities — the rest are catalogued in [`apps/docs/docs/features/`](apps/docs/docs/features/) and indexed at the top of [`apps/docs/docs/intro.md`](apps/docs/docs/intro.md):
-> **Maturity (v8.0.0 — Close the loop).** The end-to-end spine is wired and CI-gated: ingest → ClickHouse lake → live detection → fused alert → auto-triage → governed response. v8.0 connected capabilities the codebase already contained but never called. Response actions are now **verified against the vendor** rather than assumed from a 200 — an action that reports success while the effect is absent is reported as failed, because that gap is how a SOC comes to believe a host is contained when it is not. Autonomy is governed by **each tenant's own** L0–L4 policy instead of one deployment-wide environment variable. Repeat-alert suppression, shipped in v7.7, **could never fire** — the evidence fingerprint included the alert id, so no repeat ever matched; it now does. Analysts can **hunt the events AiSOC itself ingested** instead of needing an external SIEM. A verdict citing indicators the evidence never contained is demoted to human review rather than auto-closed. Autonomous *response* still defaults to copilot/dry-run: an autonomy policy governs every real execution. The live-agent LLM benchmark is preview (the deterministic-tier scoreboard is CI-gated per PR); substrate eval suites are GA. Every product claim is backed by a failing test — [claim-to-gate matrix](docs/audit/CLAIM_TO_GATE_MATRIX.md): 92 GATED / 10 PARTIAL / 0 NO GATE. Full per-claim status: [`docs/audit/REALITY_REPORT.md`](docs/audit/REALITY_REPORT.md).
+> **Maturity (v8.0.0 — Close the loop).** The end-to-end spine is wired and CI-gated: ingest → ClickHouse lake → live detection → fused alert → auto-triage → governed response. v8.0 connected capabilities the codebase already contained but never called. Response actions are now **verified against the vendor** rather than assumed from a 200 — an action that reports success while the effect is absent is reported as failed, because that gap is how a SOC comes to believe a host is contained when it is not. Autonomy is governed by **each tenant's own** L0–L4 policy instead of one deployment-wide environment variable. Repeat-alert suppression, shipped in v7.7, **could never fire** — the evidence fingerprint included the alert id, so no repeat ever matched; it now does. Analysts can **hunt the events AiSOC itself ingested** instead of needing an external SIEM. A verdict citing indicators the evidence never contained is demoted to human review rather than auto-closed. Autonomous *response* still defaults to copilot/dry-run: an autonomy policy governs every real execution. The live-agent LLM benchmark is preview (the deterministic-tier scoreboard is CI-gated per PR); substrate eval suites are GA. Every product claim is backed by a failing test — [claim-to-gate matrix](docs/audit/CLAIM_TO_GATE_MATRIX.md): 93 GATED / 9 PARTIAL / 0 NO GATE. Full per-claim status: [`docs/audit/REALITY_REPORT.md`](docs/audit/REALITY_REPORT.md).
- **84 click-and-connect data connectors** (EDR/XDR, SIEM, NDR, cloud, CNAPP, identity, SaaS, VCS, K8s audit, network) with schema-driven config, live `Test connection`, and vault-encrypted secrets — recently adding Qualys, GreyNoise, JumpCloud, Darktrace, and Imperva alongside IBM QRadar, Netskope, Zeek/Suricata NDR, and more. One query runs SIEM-agnostic **federated search** across Splunk SPL / Sentinel KQL / Elastic ES|QL / **QRadar AQL**. Walkthrough: [`apps/docs/docs/connectors/index.md`](apps/docs/docs/connectors/index.md).
- **End-to-end SIEM spine** — a cold `docker compose up` ingests connector data → lands it in the ClickHouse event lake → the executable detection corpus (833 rules) fires on the live stream → a fused alert is created, all asserted by an extended integration gate. Fuse-time threat-intel + CISA-KEV enrichment now feeds the confidence score and exploit-in-wild boost, and **stateful/windowed detections** (brute-force, password-spray, port-scan) run alongside the corpus. [`apps/docs/docs/architecture.md`](apps/docs/docs/architecture.md).
diff --git a/apps/docs/docs/benchmark.md b/apps/docs/docs/benchmark.md
index 2cdac2c77..adc82fdfa 100644
--- a/apps/docs/docs/benchmark.md
+++ b/apps/docs/docs/benchmark.md
@@ -69,7 +69,7 @@ agent performance**. Every table below is labelled with its class.
:::warning Read this first
This harness does **not** exercise the live LLM agent (`services/agents`
-LangGraph orchestrator), and the `alert_reduction` suite does **not** call the
+LangGraph orchestrator), and the legacy `alert_reduction` suite does **not** call the
production `services/fusion` engine — it calls a standalone re-implementation
of the same Tier 1/2/3 grouping rules that lives in the test file. It runs
**deterministic substrate code** against **synthetic data** so we can gate
@@ -105,12 +105,41 @@ every PR targeting `main` or `develop`.
| Suite | Metric | Per-case | Per-template macro | Target | What it checks |
|--------------------------------|-------------------------|------------|------------------------|---------|----------------|
-| Alert reduction ratio | reduction | 75.3 % | _n/a_ | ≥ 70 % | Real measurement of the 3-tier fusion logic on a noisy 1 000-alert stream |
+| Alert reduction (product logic) | reduction | 33.3 % | _n/a_ | 20–95 % | **The number that describes AiSOC.** Groups a noisy 1 000-alert stream with `RawAlert.correlation_key()` — the method `Correlator` actually calls |
+| Alert reduction (legacy suite) | reduction | 75.3 % | _n/a_ | ≥ 70 % | A four-tier scheme implemented inside the test. Retained for continuity; **does not describe this product** — see below |
| MITRE ATT&CK tactic accuracy | accuracy | 97.0 % | 96.4 % (n=55) | ≥ 80 % | Substrate self-consistency — keyword extractor vs. dataset written for it |
| Investigation completeness | mean keyword coverage | 94.2 % | 94.3 % (n=55) | ≥ 85 % | Substrate self-consistency — report template wraps the description; judge finds keywords from the description |
| Response-plan quality | mean rubric score | 1.000 | 1.000 (n=55) | ≥ 0.80 | Substrate self-consistency — synthesizer embeds the keywords the rubric checks for |
| Playbook completion rate | completion rate | 50.5 % | 100 % H/C (mapped) | ≥ 50 % | Operational coverage gate — every incident in scope has a matching playbook with aligned response action; orphan playbooks/templates fail CI |
+### Why there are two alert-reduction numbers
+
+The original suite was honest about being synthetic and always carried a
+`PARTIAL` row saying it gated "an in-test fusion re-impl". Reading it
+closely, the gap was wider than that wording admitted: the test did not
+merely reimplement fusion's grouping, it implemented **different**
+grouping.
+
+Its four tiers key on `(rule_id, host, user)` with 10/30/5-minute windows.
+`RawAlert.correlation_key()` — what `Correlator` actually calls — keys on
+`{tenant}:{entity}:{tactic}` over a one-hour window. Different dimensions,
+different windows, different answer. So the 75.3 % described an algorithm
+the product does not run, and a reimplementation can drift from the thing
+it stands for without any test failing.
+
+`services/fusion/tests/test_alert_reduction_real.py` measures the real key.
+It reports **33.3 %** on a comparable stream. That is less flattering and it
+is ours.
+
+The new gate is bounded on both sides rather than floored. A floor alone is
+satisfied by a key that collapses everything into one incident — 99.9 %
+reduction and a useless SOC — so "more reduction is better" is only true up
+to a point, and the gate says where.
+
+Both numbers stay published. Deleting the old one would make the history
+unreadable; presenting it without this note would be the thing this page
+exists to prevent.
+
> The synthetic telemetry suite is a **schema/coverage gate**, not a scoring
> suite, so it does not appear in the table. It checks that every incident has
> ≥ 1 backing event, that all `{user}/{host}/{ip}/{campaign}` placeholders
diff --git a/apps/docs/docs/compliance/evidence-pack.md b/apps/docs/docs/compliance/evidence-pack.md
index eb81d102d..bed6aaecc 100644
--- a/apps/docs/docs/compliance/evidence-pack.md
+++ b/apps/docs/docs/compliance/evidence-pack.md
@@ -33,8 +33,8 @@ and says so.
| Prompt-injection resistance | 32-payload adversarial corpus with a recall floor and a false-positive ceiling | `services/agents/tests/adversarial/` |
| Claim-to-gate traceability | Every product claim mapped to the CI job that fails when it stops being true | `docs/audit/CLAIM_TO_GATE_MATRIX.md` |
-The last row is the one worth reading first. It is the honest index: 92 rows
-`GATED`, 10 `PARTIAL` with the specific gap named, and a ratchet that refuses
+The last row is the one worth reading first. It is the honest index: 93 rows
+`GATED`, 9 `PARTIAL` with the specific gap named, and a ratchet that refuses
to let a row sit at `NO GATE`.
## Not covered, and not claimed
diff --git a/detections/fixtures/negative/app-ci-workflow-modified-by-non-owner.json b/detections/fixtures/negative/app-ci-workflow-modified-by-non-owner.json
index 9ecf21f06..096336c5d 100644
--- a/detections/fixtures/negative/app-ci-workflow-modified-by-non-owner.json
+++ b/detections/fixtures/negative/app-ci-workflow-modified-by-non-owner.json
@@ -1,5 +1,5 @@
{
- "approver_role_neq": "codeowner",
- "event_type": "not-push",
+ "approver_role": "codeowner",
+ "event_type": "push",
"path": "prefix-.github/workflows/-suffix"
}
diff --git a/detections/fixtures/negative/aws-iam-access-key-created-other-user.json b/detections/fixtures/negative/aws-iam-access-key-created-other-user.json
index b4478ae50..02d1fef97 100644
--- a/detections/fixtures/negative/aws-iam-access-key-created-other-user.json
+++ b/detections/fixtures/negative/aws-iam-access-key-created-other-user.json
@@ -1,4 +1,5 @@
{
- "actor_eq_target": false,
- "event_name": "not-CreateAccessKey"
+ "actor": "principal-a",
+ "event_name": "CreateAccessKey",
+ "target": "principal-a"
}
diff --git a/detections/fixtures/negative/cloud-multi-cross-cloud-mass-egress.json b/detections/fixtures/negative/cloud-multi-cross-cloud-mass-egress.json
index d989141c9..43326d1a3 100644
--- a/detections/fixtures/negative/cloud-multi-cross-cloud-mass-egress.json
+++ b/detections/fixtures/negative/cloud-multi-cross-cloud-mass-egress.json
@@ -1,6 +1,5 @@
{
"bytes_out_per_minute": 1073741924.0,
"dst_cloud": "aws",
- "src_cloud": "__benign__",
- "src_cloud_neq_dst_cloud": true
+ "src_cloud": "aws"
}
diff --git a/detections/fixtures/negative/cloud-org-rootmail-modified.json b/detections/fixtures/negative/cloud-org-rootmail-modified.json
index aa390286d..a0b8a2c4f 100644
--- a/detections/fixtures/negative/cloud-org-rootmail-modified.json
+++ b/detections/fixtures/negative/cloud-org-rootmail-modified.json
@@ -1,4 +1,5 @@
{
- "actor_eq_target": false,
- "event_name": "__benign__"
+ "actor": "principal-a",
+ "event_name": "UpdateAccountEmail",
+ "target": "principal-a"
}
diff --git a/detections/fixtures/negative/ident-ntlm-relay-indicator.json b/detections/fixtures/negative/ident-ntlm-relay-indicator.json
index 75fb36740..9f9eb5652 100644
--- a/detections/fixtures/negative/ident-ntlm-relay-indicator.json
+++ b/detections/fixtures/negative/ident-ntlm-relay-indicator.json
@@ -1,5 +1,6 @@
{
"auth_type": "NTLMv2",
- "event_id": 8005.0,
- "workstation_neq_source_address": true
+ "event_id": 8004,
+ "source_address": "principal-a",
+ "workstation": "principal-a"
}
diff --git a/detections/fixtures/negative/ident-pass-the-ticket-anomalous-tgt-source.json b/detections/fixtures/negative/ident-pass-the-ticket-anomalous-tgt-source.json
index 139218a6d..01709fa29 100644
--- a/detections/fixtures/negative/ident-pass-the-ticket-anomalous-tgt-source.json
+++ b/detections/fixtures/negative/ident-pass-the-ticket-anomalous-tgt-source.json
@@ -1,5 +1,6 @@
{
- "event_id": 4625.0,
+ "event_id": 4624,
"logon_type": 3,
- "ticket_source_ip_neq_session_ip": true
+ "session_ip": "principal-a",
+ "ticket_source_ip": "principal-a"
}
diff --git a/detections/fixtures/negative/linux-bashrc-modified-by-other-user.json b/detections/fixtures/negative/linux-bashrc-modified-by-other-user.json
index e34b5498c..910a026f5 100644
--- a/detections/fixtures/negative/linux-bashrc-modified-by-other-user.json
+++ b/detections/fixtures/negative/linux-bashrc-modified-by-other-user.json
@@ -1,5 +1,6 @@
{
- "actor_uid_neq_owner_uid": true,
+ "actor_uid": 1000,
+ "owner_uid": 1000,
"path": "head-/.bashrc",
- "syscall": "not-openat"
+ "syscall": "openat"
}
diff --git a/detections/fixtures/negative/linux-docker-socket-read-non-root.json b/detections/fixtures/negative/linux-docker-socket-read-non-root.json
index 87b150181..1e462f562 100644
--- a/detections/fixtures/negative/linux-docker-socket-read-non-root.json
+++ b/detections/fixtures/negative/linux-docker-socket-read-non-root.json
@@ -1,5 +1,5 @@
{
- "actor_uid_neq": 0,
+ "actor_uid": 0,
"path": "/var/run/docker.sock",
- "syscall": "not-connect"
+ "syscall": "connect"
}
diff --git a/detections/fixtures/negative/linux-ebpf-prog-load-non-root.json b/detections/fixtures/negative/linux-ebpf-prog-load-non-root.json
index 14dd14254..f347f8d2b 100644
--- a/detections/fixtures/negative/linux-ebpf-prog-load-non-root.json
+++ b/detections/fixtures/negative/linux-ebpf-prog-load-non-root.json
@@ -1,5 +1,5 @@
{
- "actor_uid_neq": 0,
+ "actor_uid": 0,
"cmd": "BPF_PROG_LOAD",
- "syscall": "not-bpf"
+ "syscall": "bpf"
}
diff --git a/detections/fixtures/negative/linux-shadow-read.json b/detections/fixtures/negative/linux-shadow-read.json
index 5bd78ec4d..b5e892479 100644
--- a/detections/fixtures/negative/linux-shadow-read.json
+++ b/detections/fixtures/negative/linux-shadow-read.json
@@ -1,5 +1,5 @@
{
- "actor_uid_neq": 0,
+ "actor_uid": 0,
"path": "/etc/shadow",
- "syscall": "not-openat"
+ "syscall": "openat"
}
diff --git a/detections/fixtures/negative/linux-zshrc-modified-by-other-user.json b/detections/fixtures/negative/linux-zshrc-modified-by-other-user.json
index 0d89eaefe..aed53f794 100644
--- a/detections/fixtures/negative/linux-zshrc-modified-by-other-user.json
+++ b/detections/fixtures/negative/linux-zshrc-modified-by-other-user.json
@@ -1,5 +1,6 @@
{
- "actor_uid_neq_owner_uid": true,
+ "actor_uid": 1000,
+ "owner_uid": 1000,
"path": "head-/.zshrc",
- "syscall": "not-openat"
+ "syscall": "openat"
}
diff --git a/detections/fixtures/negative/macos-keychain-db-read-by-other-uid.json b/detections/fixtures/negative/macos-keychain-db-read-by-other-uid.json
index d8761b95a..f367c6035 100644
--- a/detections/fixtures/negative/macos-keychain-db-read-by-other-uid.json
+++ b/detections/fixtures/negative/macos-keychain-db-read-by-other-uid.json
@@ -1,5 +1,6 @@
{
- "actor_uid_neq_owner_uid": true,
- "event_type": "not-ES_EVENT_TYPE_NOTIFY_OPEN",
+ "actor_uid": 1000,
+ "event_type": "ES_EVENT_TYPE_NOTIFY_OPEN",
+ "owner_uid": 1000,
"path": "head-/Keychains/login.keychain-db"
}
diff --git a/detections/fixtures/negative/net-smb-write-sysvol.json b/detections/fixtures/negative/net-smb-write-sysvol.json
index f1f3bdb79..baa02beb8 100644
--- a/detections/fixtures/negative/net-smb-write-sysvol.json
+++ b/detections/fixtures/negative/net-smb-write-sysvol.json
@@ -1,5 +1,5 @@
{
- "event_type": "not-smb_write",
+ "event_type": "smb_write",
"share_name": "SYSVOL",
- "src_role_neq": "domain-controller"
+ "src_role": "domain-controller"
}
diff --git a/detections/fixtures/positive/app-ci-workflow-modified-by-non-owner.json b/detections/fixtures/positive/app-ci-workflow-modified-by-non-owner.json
index 7ad6add15..a13d4c197 100644
--- a/detections/fixtures/positive/app-ci-workflow-modified-by-non-owner.json
+++ b/detections/fixtures/positive/app-ci-workflow-modified-by-non-owner.json
@@ -1,5 +1,5 @@
{
- "approver_role_neq": "codeowner",
+ "approver_role": "not-codeowner",
"event_type": "push",
"path": "prefix-.github/workflows/-suffix"
}
diff --git a/detections/fixtures/positive/aws-iam-access-key-created-other-user.json b/detections/fixtures/positive/aws-iam-access-key-created-other-user.json
index 2a0d5ef0e..2b48bc433 100644
--- a/detections/fixtures/positive/aws-iam-access-key-created-other-user.json
+++ b/detections/fixtures/positive/aws-iam-access-key-created-other-user.json
@@ -1,4 +1,5 @@
{
- "actor_eq_target": false,
- "event_name": "CreateAccessKey"
+ "actor": "principal-a",
+ "event_name": "CreateAccessKey",
+ "target": "principal-b"
}
diff --git a/detections/fixtures/positive/cloud-multi-cross-cloud-mass-egress.json b/detections/fixtures/positive/cloud-multi-cross-cloud-mass-egress.json
index 0d9302d77..9491b0986 100644
--- a/detections/fixtures/positive/cloud-multi-cross-cloud-mass-egress.json
+++ b/detections/fixtures/positive/cloud-multi-cross-cloud-mass-egress.json
@@ -1,6 +1,5 @@
{
"bytes_out_per_minute": 1073741924.0,
- "dst_cloud": "aws",
- "src_cloud": "aws",
- "src_cloud_neq_dst_cloud": true
+ "dst_cloud": "azure",
+ "src_cloud": "aws"
}
diff --git a/detections/fixtures/positive/cloud-org-rootmail-modified.json b/detections/fixtures/positive/cloud-org-rootmail-modified.json
index 61ebc117d..7116bdf7f 100644
--- a/detections/fixtures/positive/cloud-org-rootmail-modified.json
+++ b/detections/fixtures/positive/cloud-org-rootmail-modified.json
@@ -1,4 +1,5 @@
{
- "actor_eq_target": false,
- "event_name": "UpdateAccountEmail"
+ "actor": "principal-a",
+ "event_name": "UpdateAccountEmail",
+ "target": "principal-b"
}
diff --git a/detections/fixtures/positive/ident-ntlm-relay-indicator.json b/detections/fixtures/positive/ident-ntlm-relay-indicator.json
index d38a7a81b..3a991ee53 100644
--- a/detections/fixtures/positive/ident-ntlm-relay-indicator.json
+++ b/detections/fixtures/positive/ident-ntlm-relay-indicator.json
@@ -1,5 +1,6 @@
{
"auth_type": "NTLMv2",
"event_id": 8004,
- "workstation_neq_source_address": true
+ "source_address": "principal-b",
+ "workstation": "principal-a"
}
diff --git a/detections/fixtures/positive/ident-pass-the-ticket-anomalous-tgt-source.json b/detections/fixtures/positive/ident-pass-the-ticket-anomalous-tgt-source.json
index 70f12733f..ad3e3a808 100644
--- a/detections/fixtures/positive/ident-pass-the-ticket-anomalous-tgt-source.json
+++ b/detections/fixtures/positive/ident-pass-the-ticket-anomalous-tgt-source.json
@@ -1,5 +1,6 @@
{
"event_id": 4624,
"logon_type": 3,
- "ticket_source_ip_neq_session_ip": true
+ "session_ip": "principal-b",
+ "ticket_source_ip": "principal-a"
}
diff --git a/detections/fixtures/positive/linux-bashrc-modified-by-other-user.json b/detections/fixtures/positive/linux-bashrc-modified-by-other-user.json
index 2d14330db..273e72cd4 100644
--- a/detections/fixtures/positive/linux-bashrc-modified-by-other-user.json
+++ b/detections/fixtures/positive/linux-bashrc-modified-by-other-user.json
@@ -1,5 +1,6 @@
{
- "actor_uid_neq_owner_uid": true,
+ "actor_uid": 1000,
+ "owner_uid": 0,
"path": "head-/.bashrc",
"syscall": "openat"
}
diff --git a/detections/fixtures/positive/linux-docker-socket-read-non-root.json b/detections/fixtures/positive/linux-docker-socket-read-non-root.json
index 1301c48c7..54b2d51ea 100644
--- a/detections/fixtures/positive/linux-docker-socket-read-non-root.json
+++ b/detections/fixtures/positive/linux-docker-socket-read-non-root.json
@@ -1,5 +1,5 @@
{
- "actor_uid_neq": 0,
+ "actor_uid": 1000,
"path": "/var/run/docker.sock",
"syscall": "connect"
}
diff --git a/detections/fixtures/positive/linux-ebpf-prog-load-non-root.json b/detections/fixtures/positive/linux-ebpf-prog-load-non-root.json
index 3e45cd209..a53f48c62 100644
--- a/detections/fixtures/positive/linux-ebpf-prog-load-non-root.json
+++ b/detections/fixtures/positive/linux-ebpf-prog-load-non-root.json
@@ -1,5 +1,5 @@
{
- "actor_uid_neq": 0,
+ "actor_uid": 1000,
"cmd": "BPF_PROG_LOAD",
"syscall": "bpf"
}
diff --git a/detections/fixtures/positive/linux-shadow-read.json b/detections/fixtures/positive/linux-shadow-read.json
index 62a48ea1c..d9b091792 100644
--- a/detections/fixtures/positive/linux-shadow-read.json
+++ b/detections/fixtures/positive/linux-shadow-read.json
@@ -1,5 +1,5 @@
{
- "actor_uid_neq": 0,
+ "actor_uid": 1000,
"path": "/etc/shadow",
"syscall": "openat"
}
diff --git a/detections/fixtures/positive/linux-zshrc-modified-by-other-user.json b/detections/fixtures/positive/linux-zshrc-modified-by-other-user.json
index 7e2179e0e..77a22e6e3 100644
--- a/detections/fixtures/positive/linux-zshrc-modified-by-other-user.json
+++ b/detections/fixtures/positive/linux-zshrc-modified-by-other-user.json
@@ -1,5 +1,6 @@
{
- "actor_uid_neq_owner_uid": true,
+ "actor_uid": 1000,
+ "owner_uid": 0,
"path": "head-/.zshrc",
"syscall": "openat"
}
diff --git a/detections/fixtures/positive/macos-keychain-db-read-by-other-uid.json b/detections/fixtures/positive/macos-keychain-db-read-by-other-uid.json
index 4dab867e5..5224f3659 100644
--- a/detections/fixtures/positive/macos-keychain-db-read-by-other-uid.json
+++ b/detections/fixtures/positive/macos-keychain-db-read-by-other-uid.json
@@ -1,5 +1,6 @@
{
- "actor_uid_neq_owner_uid": true,
+ "actor_uid": 1000,
"event_type": "ES_EVENT_TYPE_NOTIFY_OPEN",
+ "owner_uid": 0,
"path": "head-/Keychains/login.keychain-db"
}
diff --git a/detections/fixtures/positive/net-smb-write-sysvol.json b/detections/fixtures/positive/net-smb-write-sysvol.json
index f96ebf332..cb7f58f13 100644
--- a/detections/fixtures/positive/net-smb-write-sysvol.json
+++ b/detections/fixtures/positive/net-smb-write-sysvol.json
@@ -1,5 +1,5 @@
{
"event_type": "smb_write",
"share_name": "SYSVOL",
- "src_role_neq": "domain-controller"
+ "src_role": "not-domain-controller"
}
diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.md
index 5cbe7b8c8..290d1fdec 100644
--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md
+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md
@@ -15,7 +15,7 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
| Connectors: live Test connection | README L168 | `ci.yml` connectors matrix (`test_conformance.py` — every connector implements the async `test_connection` contract) + `connector_conformance.py --check` published matrix + `test_live_vendor_smoke.py` (Phase D3 — mock-server conformance: `test_connection()` + paginated `fetch_alerts()` HTTP path exercised against realistic vendor payloads) | GATED | - |
| Investigation Ledger stores every step | README L61, L169 | `ci.yml :: api tests` (`audit_hash`, audit immutability) | PARTIAL (write path gated; UI replay only in hermetic e2e) | Phase 3.2 |
| Public eval harness gates every PR | README L62, L77 | `ci.yml :: p1-eval` | GATED (but suites are self-consistency; see reality report) | Phase 4 |
-| Alert-reduction is a real measurement | README L62 | `ci.yml :: p1-eval` (`alert_reduction`) | PARTIAL (gates an in-test fusion re-impl, not `services/fusion`) | Phase 4 |
+| Alert-reduction is a real measurement | README / benchmark page | `ci.yml :: python-test` (fusion) runs `test_alert_reduction_real.py`, which groups a 1 000-alert stream with `RawAlert.correlation_key()` — the method `Correlator` actually calls — and bounds the ratio on both sides, because a floor alone is satisfied by a key that collapses everything into one incident | GATED | the legacy suite's 75.3% stays published for continuity and is labelled as not describing this product; the measurement is still a synthetic workload, as the benchmark page states |
| Runs entirely on your infrastructure / no data exfiltration | README L63 | `ci.yml :: python-test` (agents) runs `test_privacy_redactor.py` (zero raw PII survives) | PARTIAL (redaction gated; the CLI now has no default telemetry endpoint and `telemetry.test.ts` pins that an enabled-but-unconfigured run makes zero network calls; platform-wide air-gapped egress-blocked CI + Helm NetworkPolicy still outstanding) | Phase 2 |
| Detection-as-Code rejects candidates that regress MITRE accuracy | README L170 | `ci.yml :: python-test` (`test_detection_eval.py` — candidate `rule_body` run through the real engine vs its own positive/negative fixtures; approval requires it) + `p1-eval` w2-dac baseline | GATED | - |
| 800+ native detection rules | README L78, L170 | `validate-detections.yml` (strict fixture replay) | GATED | - |
@@ -113,8 +113,8 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
## Summary
-- GATED: 92
-- PARTIAL: 10
+- GATED: 93
+- PARTIAL: 9
- NO GATE: 0 (**every claim is backed by a failing test.** The last NO GATE — the weekly benchmark scoreboard running live against `main` — closed in Phase E1: `scripts/check_scoreboard.py` ties the published scoreboard to a deterministic per-PR live-agent MITRE-accuracy run, while the funded weekly `wet-eval.yml` appends the LLM-tier rows. The remaining PARTIAL rows are honest, named deferrals — each states the specific gap and what closes it — not unproven claims.)
> **Counting note.** These figures previously read 46 / 9, which did not match
diff --git a/docs/openapi.yaml b/docs/openapi.yaml
index aaad54416..fe751b973 100644
--- a/docs/openapi.yaml
+++ b/docs/openapi.yaml
@@ -6356,6 +6356,46 @@ paths:
$ref: '#/components/schemas/PipelineHealth'
security:
- HTTPBearer: []
+ /api/v1/health/fleet:
+ get:
+ tags:
+ - health
+ summary: Get Fleet Health
+ description: 'Which connectors have quietly stopped working.
+
+
+ Every field this reads was already being written — `last_sync`,
+
+ `error_count`, `oauth_refresh_failures`, `last_schema_drift_at` — and
+
+ nothing read them together. That produces the failure this platform is
+
+ least able to tolerate: a connector stops polling, alerts from that
+
+ source stop arriving, and the console looks calm, because an absence of
+
+ alerts is indistinguishable from an absence of threats.
+
+
+ Staleness is judged per connector against its own configured cadence.
+
+ A single global threshold would page constantly on daily connectors or
+
+ stay silent on five-minute ones, and a surface that pages constantly is
+
+ a surface that gets muted.'
+ operationId: get_fleet_health_api_v1_health_fleet_get
+ responses:
+ '200':
+ description: Successful Response
+ content:
+ application/json:
+ schema:
+ additionalProperties: true
+ type: object
+ title: Response Get Fleet Health Api V1 Health Fleet Get
+ security:
+ - HTTPBearer: []
/api/v1/insights/soc:
get:
tags:
diff --git a/infra/docker/grafana/dashboards/aisoc-platform.json b/infra/docker/grafana/dashboards/aisoc-platform.json
new file mode 100644
index 000000000..d0ca55df1
--- /dev/null
+++ b/infra/docker/grafana/dashboards/aisoc-platform.json
@@ -0,0 +1,200 @@
+{
+ "uid": "aisoc-platform",
+ "title": "AiSOC — Platform",
+ "tags": ["aisoc"],
+ "timezone": "browser",
+ "schemaVersion": 39,
+ "version": 1,
+ "refresh": "30s",
+ "time": { "from": "now-6h", "to": "now" },
+ "description": "Every panel here queries a metric some service actually emits. Panels for metrics nothing exports render as 'No data' forever, which teaches people that empty panels are normal — and then a genuinely dead service looks the same as an uninstrumented one. Coverage is 5 of 19 services; `scripts/audit_prometheus_targets.py` names the other 14.",
+ "panels": [
+ {
+ "type": "row",
+ "title": "API",
+ "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },
+ "collapsed": false
+ },
+ {
+ "type": "timeseries",
+ "title": "Request rate by status",
+ "description": "Split by status rather than totalled: a flat request rate with a rising 5xx share looks identical to a healthy one on a single line.",
+ "datasource": { "type": "prometheus", "uid": "aisoc-prometheus" },
+ "gridPos": { "h": 8, "w": 12, "x": 0, "y": 1 },
+ "targets": [
+ {
+ "expr": "sum by (status) (rate(aisoc_http_requests_total[5m]))",
+ "legendFormat": "{{status}}"
+ }
+ ],
+ "fieldConfig": {
+ "defaults": { "unit": "reqps", "custom": { "fillOpacity": 10 } }
+ }
+ },
+ {
+ "type": "timeseries",
+ "title": "Request latency (p50 / p95 / p99)",
+ "description": "Three quantiles, not a mean. A mean hides the tail, and the tail is what an analyst waiting on an investigation experiences.",
+ "datasource": { "type": "prometheus", "uid": "aisoc-prometheus" },
+ "gridPos": { "h": 8, "w": 12, "x": 12, "y": 1 },
+ "targets": [
+ {
+ "expr": "histogram_quantile(0.50, sum by (le) (rate(aisoc_http_request_duration_seconds_bucket[5m])))",
+ "legendFormat": "p50"
+ },
+ {
+ "expr": "histogram_quantile(0.95, sum by (le) (rate(aisoc_http_request_duration_seconds_bucket[5m])))",
+ "legendFormat": "p95"
+ },
+ {
+ "expr": "histogram_quantile(0.99, sum by (le) (rate(aisoc_http_request_duration_seconds_bucket[5m])))",
+ "legendFormat": "p99"
+ }
+ ],
+ "fieldConfig": { "defaults": { "unit": "s" } }
+ },
+ {
+ "type": "stat",
+ "title": "Error ratio (5m)",
+ "description": "5xx as a share of all requests. Absolute error counts scale with traffic, so a ratio is the thing worth alerting on.",
+ "datasource": { "type": "prometheus", "uid": "aisoc-prometheus" },
+ "gridPos": { "h": 5, "w": 6, "x": 0, "y": 9 },
+ "targets": [
+ {
+ "expr": "sum(rate(aisoc_http_requests_total{status=~\"5..\"}[5m])) / clamp_min(sum(rate(aisoc_http_requests_total[5m])), 0.001)"
+ }
+ ],
+ "fieldConfig": {
+ "defaults": {
+ "unit": "percentunit",
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ { "color": "green", "value": null },
+ { "color": "yellow", "value": 0.01 },
+ { "color": "red", "value": 0.05 }
+ ]
+ }
+ }
+ }
+ },
+ {
+ "type": "row",
+ "title": "Ingest",
+ "gridPos": { "h": 1, "w": 24, "x": 0, "y": 14 },
+ "collapsed": false
+ },
+ {
+ "type": "timeseries",
+ "title": "Inbox events accepted",
+ "datasource": { "type": "prometheus", "uid": "aisoc-prometheus" },
+ "gridPos": { "h": 8, "w": 12, "x": 0, "y": 15 },
+ "targets": [
+ {
+ "expr": "sum by (tenant) (rate(aisoc_ingest_inbox_events_total[5m]))",
+ "legendFormat": "{{tenant}}"
+ }
+ ],
+ "fieldConfig": { "defaults": { "unit": "eps" } }
+ },
+ {
+ "type": "timeseries",
+ "title": "Inbox requests rate-limited",
+ "description": "Non-zero is not necessarily wrong — the limiter exists to be hit by a misconfigured client. A sustained rate against one tenant means their integration is retrying into a wall and their events are being dropped.",
+ "datasource": { "type": "prometheus", "uid": "aisoc-prometheus" },
+ "gridPos": { "h": 8, "w": 12, "x": 12, "y": 15 },
+ "targets": [
+ {
+ "expr": "sum by (tenant) (rate(aisoc_ingest_inbox_rate_limited_total[5m]))",
+ "legendFormat": "{{tenant}}"
+ }
+ ],
+ "fieldConfig": { "defaults": { "unit": "reqps" } }
+ },
+ {
+ "type": "row",
+ "title": "Auto-triage",
+ "gridPos": { "h": 1, "w": 24, "x": 0, "y": 23 },
+ "collapsed": false
+ },
+ {
+ "type": "timeseries",
+ "title": "Triage outcomes",
+ "description": "The agents worker's own counters. Watch two shapes: dead_lettered rising means alerts are being dropped rather than triaged, and deterministic climbing while llm stays flat means the LLM path is failing open to the fallback — which looks fine from outside, because alerts still get verdicts.",
+ "datasource": { "type": "prometheus", "uid": "aisoc-prometheus" },
+ "gridPos": { "h": 8, "w": 12, "x": 0, "y": 24 },
+ "targets": [
+ {
+ "expr": "sum by (result) (rate(aisoc_agents_triage_total[15m]))",
+ "legendFormat": "{{result}}"
+ }
+ ]
+ },
+ {
+ "type": "timeseries",
+ "title": "LLM vs deterministic share",
+ "description": "The ratio, because the absolute counts move with alert volume and the ratio does not. A fall here is the LLM path degrading.",
+ "datasource": { "type": "prometheus", "uid": "aisoc-prometheus" },
+ "gridPos": { "h": 8, "w": 12, "x": 12, "y": 24 },
+ "targets": [
+ {
+ "expr": "sum(rate(aisoc_agents_triage_total{result=\"llm\"}[15m])) / clamp_min(sum(rate(aisoc_agents_triage_total{result=~\"llm|deterministic\"}[15m])), 0.001)",
+ "legendFormat": "LLM share"
+ }
+ ],
+ "fieldConfig": { "defaults": { "unit": "percentunit", "max": 1, "min": 0 } }
+ },
+ {
+ "type": "stat",
+ "title": "Dead-lettered (1h)",
+ "description": "Alerts the worker could not process. Any sustained non-zero is alert loss.",
+ "datasource": { "type": "prometheus", "uid": "aisoc-prometheus" },
+ "gridPos": { "h": 5, "w": 6, "x": 0, "y": 32 },
+ "targets": [
+ { "expr": "sum(increase(aisoc_agents_triage_total{result=\"dead_lettered\"}[1h]))" }
+ ],
+ "fieldConfig": {
+ "defaults": {
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ { "color": "green", "value": null },
+ { "color": "red", "value": 1 }
+ ]
+ }
+ }
+ }
+ },
+ {
+ "type": "stat",
+ "title": "Ungrounded demotions (1h)",
+ "description": "Verdicts demoted to human review because they cited indicators the evidence did not contain. A rise is a prompt or model regression. A persistent zero is not automatically good — check that groundedness is scoring at all.",
+ "datasource": { "type": "prometheus", "uid": "aisoc-prometheus" },
+ "gridPos": { "h": 5, "w": 6, "x": 6, "y": 32 },
+ "targets": [
+ {
+ "expr": "sum(increase(aisoc_agents_triage_total{result=\"ungrounded_demoted\"}[1h]))"
+ }
+ ]
+ },
+ {
+ "type": "row",
+ "title": "Threat intel",
+ "gridPos": { "h": 1, "w": 24, "x": 0, "y": 37 },
+ "collapsed": false
+ },
+ {
+ "type": "timeseries",
+ "title": "IOCs and actors ingested",
+ "datasource": { "type": "prometheus", "uid": "aisoc-prometheus" },
+ "gridPos": { "h": 8, "w": 24, "x": 0, "y": 38 },
+ "targets": [
+ { "expr": "rate(threatintel_iocs_ingested_total[15m])", "legendFormat": "IOCs" },
+ {
+ "expr": "rate(threatintel_actors_ingested_total[15m])",
+ "legendFormat": "actors"
+ }
+ ]
+ }
+ ]
+}
diff --git a/infra/docker/grafana/dashboards/dashboards.yaml b/infra/docker/grafana/dashboards/dashboards.yaml
new file mode 100644
index 000000000..90c4019bb
--- /dev/null
+++ b/infra/docker/grafana/dashboards/dashboards.yaml
@@ -0,0 +1,21 @@
+# Grafana dashboard provisioning.
+#
+# Dashboards are files in this directory rather than state inside Grafana's
+# database, so a rebuilt container comes back with them and a change to one
+# is reviewable in a diff. `allowUiUpdates: false` makes that stick: a
+# dashboard edited in the browser is silently reverted on the next reload,
+# which is the correct outcome but surprises people who do not know the
+# dashboards are provisioned.
+apiVersion: 1
+
+providers:
+ - name: aisoc
+ orgId: 1
+ folder: AiSOC
+ type: file
+ disableDeletion: false
+ allowUiUpdates: false
+ updateIntervalSeconds: 30
+ options:
+ path: /etc/grafana/provisioning/dashboards
+ foldersFromFilesStructure: false
diff --git a/infra/docker/grafana/datasources/prometheus.yaml b/infra/docker/grafana/datasources/prometheus.yaml
new file mode 100644
index 000000000..014f54339
--- /dev/null
+++ b/infra/docker/grafana/datasources/prometheus.yaml
@@ -0,0 +1,24 @@
+# Prometheus datasource.
+#
+# The Tempo datasource already referenced `aisoc-prometheus` for its
+# trace-to-metrics and service-map links, and no datasource declared that
+# uid — so both links resolved to nothing. Grafana does not warn about a
+# dangling datasource reference; the feature simply does nothing when
+# clicked, which reads as "tracing is not wired up" rather than as a typo.
+apiVersion: 1
+
+datasources:
+ - name: Prometheus
+ type: prometheus
+ uid: aisoc-prometheus
+ access: proxy
+ url: http://prometheus:9090
+ isDefault: true
+ jsonData:
+ # Matches the scrape_interval in infra/docker/prometheus.yml. A
+ # mismatch makes rate() windows that look fine and quietly under- or
+ # over-sample.
+ timeInterval: 15s
+ exemplarTraceIdDestinations:
+ - name: trace_id
+ datasourceUid: aisoc-tempo
diff --git a/scripts/check_action_contract.py b/scripts/check_action_contract.py
index d9d702b46..cc7012e91 100644
--- a/scripts/check_action_contract.py
+++ b/scripts/check_action_contract.py
@@ -194,23 +194,22 @@ def check_approval_matrix() -> list[str]:
#: Ordered by what it costs to be missing:
#: restore_file — the reverse of quarantine_file; a quarantined file
#: cannot be released through the platform that took it
-#: allow_domain — the reverse of block_domain, whose forward arm is
-#: allow_hash itself a placeholder
-#: allow_ioc — the reverse of block_ioc, which Defender does implement
-#: block_hash — declared as allow_hash's reverse, never built
-#: enable_user — the reverse of disable_user; an account disabled in an
-#: incident has no platform route back
-#: revoke_session, block_user_signin — identity verbs with no arm
+#: allow_hash — the reverse of block_hash and vice versa; neither arm
+#: allow_ioc exists, and allow_ioc reverses a block Defender does
+#: block_hash implement
+#: block_user_signin — an identity verb with no vendor arm
#: push_case, push_status — ITSM sync, never wired
+#:
+#: Left this set when their clients turned out to already implement them:
+#: enable_user (Entra, Google Workspace), revoke_session (both),
+#: allow_domain (Cloudflare), allow_ip (Cloudflare, PAN-OS, FortiGate),
+#: unisolate_host (CrowdStrike, Defender, SentinelOne).
KNOWN_ORPHANS: frozenset[str] = frozenset(
{
"restore_file",
- "allow_domain",
"allow_hash",
"allow_ioc",
"block_hash",
- "enable_user",
- "revoke_session",
"block_user_signin",
"push_case",
"push_status",
diff --git a/scripts/check_detection_fields.py b/scripts/check_detection_fields.py
index 5a577ab04..4846b8034 100644
--- a/scripts/check_detection_fields.py
+++ b/scripts/check_detection_fields.py
@@ -56,7 +56,7 @@ NORMALIZER = ROOT / "services" / "ingest" / "internal" / "normalizer" / "normali
#: Current number of engine-loaded rules that depend on a DERIVED or STATEFUL
#: field. This may only ever decrease. Lower it in the same PR that fixes
#: rules; never raise it to make a red build green.
-MAX_UNREACHABLE = 138
+MAX_UNREACHABLE = 133
#: Fields that no telemetry carries because they are computed, not observed:
#: sliding-window counters, allowlist membership, privilege flags, and
@@ -152,6 +152,66 @@ def _rule_fields(match_when: dict) -> set[str]:
return found
+#: Fields the engine derives at match time from an event it already has.
+#: `services/fusion/app/services/derived_fields.py` computes them, so a rule
+#: matching on one is reachable even though no connector emits it.
+#:
+#: Named here rather than imported, for the reason the rest of this file is
+#: parsed rather than imported: a field-coverage gate that needs the fusion
+#: package installed is a gate that gets disabled the first time an import
+#: breaks.
+_DERIVED_TIME_FIELDS = frozenset({"is_business_hours", "is_after_hours", "is_weekend"})
+
+#: `<left>_eq_<right>` / `<left>_neq_<right>`, resolved by comparing two
+#: fields of the same event. Reachable only when *both* sides are in the
+#: namespace — a comparison against a field nothing emits is still dead, and
+#: counting it as covered would be the fake-gate failure this file exists to
+#: prevent.
+_DERIVED_COMPARISON_RE = re.compile(r"^(?P<left>.+?)_(?:eq|neq)_(?P<right>.+)$")
+
+
+def _is_derivable(field: str, namespace: set[str]) -> bool:
+ """True when the engine can compute this field from what it already has."""
+ if field in _DERIVED_TIME_FIELDS:
+ return True
+ match = _DERIVED_COMPARISON_RE.match(field)
+ if not match:
+ return False
+ return match.group("left") in namespace and match.group("right") in namespace
+
+
+#: What each unreachable field would need, so the headline number is
+#: actionable rather than a bare count. "133 rules are dead" invites
+#: someone to delete 133 rules; "73 need the windowed evaluator, 24 need
+#: identity enrichment" names four pieces of work.
+#:
+#: Ordered: first match wins, so the more specific patterns come first.
+_FIELD_FAMILIES: tuple[tuple[str, re.Pattern[str]], ...] = (
+ (
+ "windowed evaluator",
+ re.compile(r"_count$|^count_|_per_|time_window|_window_|_5min|_ratio$"),
+ ),
+ ("identity enrichment", re.compile(r"_priv$|_is_admin$|_is_dc$")),
+ ("per-tenant allowlist", re.compile(r"_in_allowlist$|_not_in_allowlist$")),
+ ("age / first-seen enrichment", re.compile(r"_age_days$|_age_hours$|_seen_before$")),
+ ("behavioural baseline", re.compile(r"_baseline|_deviation|^active_|_is_first_")),
+ (
+ # Both operands invented by the rule author. Distinct from the rest:
+ # the engine can compute a comparison, it just cannot compare two
+ # fields that do not exist. This is content work, not engine work.
+ "comparison over fields nothing emits",
+ _DERIVED_COMPARISON_RE,
+ ),
+)
+
+
+def _family(field: str) -> str:
+ for name, pattern in _FIELD_FAMILIES:
+ if pattern.search(field):
+ return name
+ return "other"
+
+
def _connector_namespace() -> set[str]:
"""Every string-literal dict key appearing in a connector module.
@@ -243,7 +303,7 @@ def scan() -> tuple[list[tuple[str, list[str]]], list[tuple[str, list[str]]], se
derived: list[tuple[str, list[str]]] = []
vendor: list[tuple[str, list[str]]] = []
for rule in rules:
- missing = {f for f in _rule_fields(rule.get("match_when") or {}) if f not in namespace}
+ missing = {f for f in _rule_fields(rule.get("match_when") or {}) if f not in namespace and not _is_derivable(f, namespace)}
if not missing:
continue
rule_id = str(rule.get("id", "?"))
@@ -285,6 +345,12 @@ def main() -> int:
print(f"detection fields: {total} engine rules examined")
print(f" {len(derived)} depend on a derived/stateful field — cannot fire on any connector (gated)")
+ families: dict[str, int] = {}
+ for _rule_id, fields in derived:
+ for name in {_family(f) for f in fields}:
+ families[name] = families.get(name, 0) + 1
+ for name, count in sorted(families.items(), key=lambda kv: -kv[1]):
+ print(f" {count:4d} need {name}")
print(f" {len(vendor)} reference a vendor-payload field — reachable when that vendor is attached")
if len(derived) > args.max_unreachable:
diff --git a/scripts/check_grafana_dashboards.py b/scripts/check_grafana_dashboards.py
new file mode 100644
index 000000000..825fc5e05
--- /dev/null
+++ b/scripts/check_grafana_dashboards.py
@@ -0,0 +1,176 @@
+#!/usr/bin/env python3
+"""Gate: every dashboard panel queries a metric some service emits.
+
+A panel for a metric nothing exports renders "No data" forever. That is
+worse than having no panel, because it teaches people that empty panels are
+normal — and once they believe that, a genuinely dead service looks exactly
+like an uninstrumented one.
+
+Also checks datasource references resolve. Grafana does not warn about a
+dangling datasource uid: the panel just shows nothing, and a trace-to-metrics
+link silently does nothing when clicked, which reads as "tracing is not
+wired up" rather than as a typo. The Tempo datasource shipped pointing at
+`aisoc-prometheus` before any datasource declared that uid.
+
+Run: python3 scripts/check_grafana_dashboards.py
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import sys
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parent.parent
+GRAFANA = REPO_ROOT / "infra" / "docker" / "grafana"
+DASHBOARDS = GRAFANA / "dashboards"
+DATASOURCES = GRAFANA / "datasources"
+SERVICES = REPO_ROOT / "services"
+
+#: Metric names a PromQL expression may reference. Recovered from the
+#: expression rather than parsed properly: a full PromQL parser is a
+#: dependency this gate does not need, and the names are the only part that
+#: has to be right.
+_METRIC_RE = re.compile(r"\b([a-z][a-z0-9_]*_(?:total|seconds|bytes|count|info))\b")
+
+#: Prometheus appends these to a histogram or summary, so a dashboard
+#: legitimately references a name no service literally declares.
+_DERIVED_SUFFIXES = ("_bucket", "_sum", "_count")
+
+#: Metrics exported by something other than our own code — the exporters
+#: bundled into the compose stack. Listed so the exemption is reviewable
+#: rather than implied by a loose regex.
+KNOWN_EXTERNAL: frozenset[str] = frozenset(
+ {
+ "up",
+ "scrape_duration_seconds",
+ "process_cpu_seconds_total",
+ "process_resident_memory_bytes",
+ }
+)
+
+
+def declared_metrics() -> set[str]:
+ """Metric names declared anywhere under services/.
+
+ A string-literal scan rather than an import: collecting metric names
+ should not need every service's dependencies installed, and a gate
+ that needs a database container is a gate that gets disabled.
+ """
+ names: set[str] = set()
+ for path in SERVICES.rglob("*"):
+ if path.suffix not in {".py", ".go", ".ts"} or not path.is_file():
+ continue
+ if "/tests/" in str(path) or "/node_modules/" in str(path):
+ continue
+ try:
+ text = path.read_text(encoding="utf-8", errors="ignore")
+ except OSError:
+ continue
+ names |= set(_METRIC_RE.findall(text))
+ return names
+
+
+def dashboard_metrics(dashboard: dict) -> set[str]:
+ names: set[str] = set()
+ for panel in dashboard.get("panels", []):
+ for target in panel.get("targets", []):
+ expr = target.get("expr", "")
+ names |= set(_METRIC_RE.findall(expr))
+ # Histogram buckets: `foo_seconds_bucket` -> `foo_seconds`.
+ names |= {m.removesuffix("_bucket") for m in re.findall(r"\b([a-z][a-z0-9_]*_bucket)\b", expr)}
+ return names
+
+
+def declared_datasource_uids() -> set[str]:
+ uids: set[str] = set()
+ for path in DATASOURCES.glob("*.yaml"):
+ for match in re.finditer(r"^\s*uid:\s*(\S+)\s*$", path.read_text(), re.M):
+ uids.add(match.group(1))
+ return uids
+
+
+def referenced_datasource_uids() -> set[str]:
+ uids: set[str] = set()
+ for path in DASHBOARDS.glob("*.json"):
+ text = path.read_text(encoding="utf-8")
+ uids |= set(re.findall(r'"uid"\s*:\s*"([^"]+)"', text))
+ # Cross-datasource links (trace-to-metrics, exemplars) are references too.
+ for path in DATASOURCES.glob("*.yaml"):
+ for match in re.finditer(r"^\s*datasourceUid:\s*(\S+)\s*$", path.read_text(), re.M):
+ uids.add(match.group(1))
+ return uids
+
+
+def main(argv: list[str] | None = None) -> int:
+ argparse.ArgumentParser(description=__doc__).parse_args(argv)
+
+ if not DASHBOARDS.is_dir():
+ print("grafana-dashboards: no dashboards directory; nothing to check")
+ return 0
+
+ files = sorted(DASHBOARDS.glob("*.json"))
+ if not files:
+ print(
+ "grafana-dashboards: dashboards directory exists but is empty",
+ file=sys.stderr,
+ )
+ return 1
+
+ errors: list[str] = []
+ exported = declared_metrics()
+
+ for path in files:
+ try:
+ dashboard = json.loads(path.read_text(encoding="utf-8"))
+ except json.JSONDecodeError as exc:
+ errors.append(f"{path.name}: invalid JSON ({exc})")
+ continue
+
+ # A dashboard uid is what links and alerts reference; without one
+ # Grafana assigns a random one per import and every link rots.
+ if not dashboard.get("uid"):
+ errors.append(f"{path.name}: no uid, so links to it cannot be stable")
+
+ for metric in sorted(dashboard_metrics(dashboard)):
+ if metric in KNOWN_EXTERNAL or metric in exported:
+ continue
+ if any(metric.removesuffix(suffix) in exported for suffix in _DERIVED_SUFFIXES):
+ continue
+ errors.append(
+ f"{path.name}: panel queries {metric!r}, which no service "
+ f"declares. The panel will read 'No data' forever, which "
+ f"teaches people to ignore empty panels."
+ )
+
+ declared = declared_datasource_uids()
+ for uid in sorted(referenced_datasource_uids()):
+ # Panel-level uids are datasource uids; anything else in a dashboard
+ # JSON matching "uid" is the dashboard's own, which is fine.
+ if uid in declared:
+ continue
+ if any(json.loads(p.read_text()).get("uid") == uid for p in files):
+ continue
+ errors.append(
+ f"datasource uid {uid!r} is referenced but no datasource declares "
+ f"it. Grafana does not warn about this — the panel or link simply "
+ f"does nothing."
+ )
+
+ if errors:
+ print("GRAFANA DASHBOARD GATE FAILED:", file=sys.stderr)
+ for error in errors:
+ print(f" - {error}", file=sys.stderr)
+ return 1
+
+ print(
+ f"grafana-dashboards: OK — {len(files)} dashboard(s), every panel "
+ f"queries a metric a service emits and every datasource resolves"
+ )
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/scripts/check_store_migrations.py b/scripts/check_store_migrations.py
new file mode 100644
index 000000000..6376ceed2
--- /dev/null
+++ b/scripts/check_store_migrations.py
@@ -0,0 +1,171 @@
+#!/usr/bin/env python3
+"""Gate: every store with schema has a migration runner, and it is called.
+
+Three stores hold schema that code depends on — Neo4j, ClickHouse and
+Qdrant — and none had a way to change it. `clickhouse/001_init.sql` is four
+`CREATE TABLE IF NOT EXISTS` statements and zero `ALTER TABLE`;
+`lake_writer` self-heals with the same idiom; `QdrantStore.initialize()`
+creates collections if absent. All three are no-ops once the object exists,
+so a schema change lands on a fresh deployment and silently does not land on
+an existing one. Nothing notices until a query selects a column that is not
+there, long after the deploy that was meant to add it.
+
+The second half of this gate matters more than the first. v8.0 spent a
+release discovering that **the mechanism existed, was unit-tested, and had
+no caller on the path that needed it** — so a runner that exists and is
+never invoked is not progress, it is the same defect with better test
+coverage. This checks both.
+
+Parsed rather than imported: the API package pulls in SQLAlchemy, asyncpg,
+Neo4j and ClickHouse drivers, and a structural gate that needs a database
+container is a gate that gets disabled.
+
+Run: python3 scripts/check_store_migrations.py
+"""
+
+from __future__ import annotations
+
+import argparse
+import re
+import sys
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parent.parent
+
+#: store -> (runner module, the file that must call it, the symbol it calls)
+#:
+#: The call site is named explicitly rather than searched for, because "some
+#: file somewhere imports it" is satisfied by a test, and a test is exactly
+#: what stopped being evidence.
+RUNNERS: dict[str, tuple[Path, Path, str]] = {
+ "neo4j": (
+ REPO_ROOT / "services/api/app/db/graph_migrations.py",
+ REPO_ROOT / "services/api/app/db/neo4j.py",
+ "run_migrations",
+ ),
+ "clickhouse": (
+ REPO_ROOT / "services/api/app/db/lake_migrations.py",
+ REPO_ROOT / "services/api/app/db/clickhouse.py",
+ "run_migrations",
+ ),
+ "qdrant": (
+ REPO_ROOT / "services/api/app/db/vector_migrations.py",
+ REPO_ROOT / "services/threatintel/app/storage/qdrant.py",
+ "run_migrations",
+ ),
+}
+
+#: The startup path each caller must itself be reachable from. Without this
+#: the chain stops one link short: a runner called by a function nobody
+#: invokes is still a runner that never runs.
+STARTUP_CALLERS: dict[str, tuple[Path, str]] = {
+ "neo4j": (REPO_ROOT / "services/api/app/main.py", "init_neo4j"),
+ "clickhouse": (REPO_ROOT / "services/api/app/main.py", "init_lake_schema"),
+ "qdrant": (
+ REPO_ROOT / "services/threatintel/app/storage/qdrant.py",
+ "initialize",
+ ),
+}
+
+#: A runner must expose these. Three stores, one vocabulary — an operator
+#: should not need three mental models to answer "what has this applied".
+REQUIRED_SYMBOLS = ("MIGRATIONS", "run_migrations", "applied_ids", "pending_ids")
+
+
+def _read(path: Path) -> str:
+ return path.read_text(encoding="utf-8") if path.exists() else ""
+
+
+def check_runners_exist() -> list[str]:
+ errors: list[str] = []
+ for store, (runner, _, _) in RUNNERS.items():
+ if not runner.exists():
+ errors.append(
+ f"{store}: no migration runner at "
+ f"{runner.relative_to(REPO_ROOT)}. Its schema can only be "
+ f"created, never changed, so a new column never reaches an "
+ f"existing deployment."
+ )
+ continue
+ source = _read(runner)
+ for symbol in REQUIRED_SYMBOLS:
+ if not re.search(rf"^(?:async def |def |){symbol}\b", source, re.M):
+ errors.append(f"{store}: runner does not expose {symbol}; the three " f"runners are meant to share one interface")
+ return errors
+
+
+def check_runners_are_called() -> list[str]:
+ """The half that catches the v8.0 pattern."""
+ errors: list[str] = []
+ for store, (runner, caller, symbol) in RUNNERS.items():
+ if not runner.exists():
+ continue
+ if not caller.exists():
+ errors.append(f"{store}: expected call site {caller.relative_to(REPO_ROOT)} " f"does not exist")
+ continue
+ source = _read(caller)
+ if symbol not in source:
+ errors.append(
+ f"{store}: {caller.relative_to(REPO_ROOT)} never calls "
+ f"{symbol}(). The runner exists and nothing invokes it, which "
+ f"is indistinguishable from not having one."
+ )
+ return errors
+
+
+def check_startup_reachability() -> list[str]:
+ """A caller nobody invokes is one link short of the same problem."""
+ errors: list[str] = []
+ for store, (path, symbol) in STARTUP_CALLERS.items():
+ source = _read(path)
+ if not source:
+ errors.append(f"{store}: {path.relative_to(REPO_ROOT)} does not exist")
+ continue
+ # Called, not merely defined or imported.
+ if not re.search(rf"\b{symbol}\s*\(", source):
+ errors.append(
+ f"{store}: {symbol} is never invoked in " f"{path.relative_to(REPO_ROOT)}, so migrations do not run at " f"startup"
+ )
+ return errors
+
+
+def check_ids_are_append_only() -> list[str]:
+ """Migration ids must be unique and ordered.
+
+ The runners apply in tuple order, so a numbering that disagrees with
+ that order means migrations run in a sequence the numbers do not
+ describe — and migration N+1 is written against the state N produced.
+ """
+ errors: list[str] = []
+ for store, (runner, _, _) in RUNNERS.items():
+ source = _read(runner)
+ if not source:
+ continue
+ ids = re.findall(r'^\s+id="([^"]+)"', source, re.M)
+ if not ids:
+ errors.append(f"{store}: no migrations declared; the runner is empty")
+ continue
+ if len(ids) != len(set(ids)):
+ errors.append(f"{store}: duplicate migration ids")
+ if ids != sorted(ids):
+ errors.append(f"{store}: migration ids are not in order ({', '.join(ids)}); " f"the runner applies them in declaration order")
+ return errors
+
+
+def main(argv: list[str] | None = None) -> int:
+ argparse.ArgumentParser(description=__doc__).parse_args(argv)
+
+ errors = check_runners_exist() + check_runners_are_called() + check_startup_reachability() + check_ids_are_append_only()
+
+ if errors:
+ print("STORE MIGRATION GATE FAILED:", file=sys.stderr)
+ for error in errors:
+ print(f" - {error}", file=sys.stderr)
+ return 1
+
+ print(f"store-migrations: OK — {len(RUNNERS)} stores have a runner, each " f"wired into a startup path")
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/scripts/detection_specs_part3_helpers.py b/scripts/detection_specs_part3_helpers.py
index f744a0018..e888dbd61 100644
--- a/scripts/detection_specs_part3_helpers.py
+++ b/scripts/detection_specs_part3_helpers.py
@@ -21,6 +21,7 @@ Auto-fixture rules
from __future__ import annotations
from collections.abc import Iterable
+import re
from typing import Any
# Operator suffixes - kept in sync with scripts/generate_detections.py.
@@ -30,6 +31,11 @@ from typing import Any
_OP_SUFFIXES: tuple[tuple[str, str], ...] = tuple(
sorted(
[
+ # `neq` was used by eleven rules and was not an operator, so
+ # `actor_uid_neq: 0` was read as a field literally named
+ # `actor_uid_neq` — and the synthesized fixture then contained
+ # that field, so the rule matched itself and nothing else.
+ ("_neq", "neq"),
("_pattern_match_any", "pattern_match_any"),
("_not_endswith_any", "not_endswith_any"),
("_not_contains_any", "not_contains_any"),
@@ -74,6 +80,15 @@ def _pos_for(op: str, expected: Any) -> Any:
"""Return a value satisfying the (op, expected) clause."""
if op == "eq":
return expected
+ if op == "neq":
+ # Anything but `expected`. Numeric fields get a number rather than
+ # the string sentinel: `actor_uid: "__benign__"` is not a uid, and
+ # a fixture that could not exist is a fixture that tests nothing.
+ if isinstance(expected, bool):
+ return not expected
+ if isinstance(expected, (int, float)):
+ return expected + 1000
+ return f"not-{expected}"
if op == "in":
return expected[0] if isinstance(expected, list) and expected else None
if op == "not_in":
@@ -129,6 +144,12 @@ def _pos_for(op: str, expected: Any) -> Any:
def _neg_for(op: str, expected: Any) -> Any:
"""Return a value that breaks the (op, expected) clause."""
+ if op == "neq":
+ # The inverse of neq is equality, so the negative carries the
+ # excluded value. That is the clause the rule is *about*, which is
+ # what a negative should invert — the generator used to flip an
+ # unrelated field and never test the condition in either direction.
+ return expected
if op == "eq":
if expected is None:
return _SENTINEL_STR
@@ -301,6 +322,99 @@ def _compose_field_value(constraints: list[tuple[str, Any]]) -> Any:
return f"{prefix}{middle}{suffix}"
+#: `<left>_eq_<right>` / `<left>_neq_<right>` — the engine derives these by
+#: comparing two fields of the same event, so a fixture needs both operands
+#: rather than one field named after the comparison.
+_COMPARISON_KEY_RE = re.compile(r"^(?P<left>.+?)_(?P<op>eq|neq)_(?P<right>.+)$")
+
+
+def _comparison_operands(key: str, want_value: bool) -> dict[str, Any] | None:
+ """Two real fields whose comparison satisfies or breaks the clause.
+
+ Returns None when `key` is not a comparison, so callers fall through to
+ the ordinary operator path.
+ """
+ match = _COMPARISON_KEY_RE.match(key)
+ if not match:
+ return None
+ left, right, op = match.group("left"), match.group("right"), match.group("op")
+ # `want_value` is what the derived field should evaluate to, not
+ # whether the clause is satisfied. Conflating the two made the negative
+ # fixture identical to the positive for `actor_eq_target: false`: both
+ # asked for "operands differ", so the rule matched its own negative.
+ want_equal = (op == "eq") == want_value
+
+ if left.endswith(("uid", "gid", "_id", "port")):
+ return {left: 1000, right: 1000 if want_equal else 0}
+ return {left: "principal-a", right: "principal-a" if want_equal else "principal-b"}
+
+
+def _alternative_value(field: str, constraints: list[tuple[str, Any]], avoid: Any) -> Any:
+ """Another value for `field` that still satisfies its own clauses.
+
+ Used when a comparison needs two operands to differ and both are
+ already pinned by other clauses. Returns `avoid` unchanged when no
+ alternative exists, so the caller can report rather than silently
+ emit a fixture that cannot satisfy its rule.
+ """
+ for op, expected in constraints:
+ if op in {"in", "match_any", "contains_any", "startswith_any", "endswith_any"} and isinstance(
+ expected, list
+ ):
+ for candidate in expected:
+ value = _pos_for(op, [candidate])
+ if value != avoid:
+ return value
+ if isinstance(avoid, bool):
+ return not avoid
+ if isinstance(avoid, (int, float)):
+ return avoid + 1
+ if isinstance(avoid, str):
+ return f"other-{avoid}"
+ return avoid
+
+
+def _resolve_comparison(
+ key: str,
+ want_value: bool,
+ pos: dict[str, Any],
+ by_field: dict[str, list[tuple[str, Any]]],
+) -> dict[str, Any]:
+ """Operand values satisfying a comparison, respecting existing clauses."""
+ match = _COMPARISON_KEY_RE.match(key)
+ assert match is not None
+ left, right, op = match.group("left"), match.group("right"), match.group("op")
+ want_equal = (op == "eq") == want_value
+
+ left_value = pos.get(left)
+ right_value = pos.get(right)
+
+ # Neither operand constrained elsewhere: pick a clean pair.
+ if left_value is None and right_value is None:
+ operands = _comparison_operands(key, want_value)
+ return operands or {}
+
+ if left_value is None:
+ left_value = right_value if want_equal else _alternative_value(
+ left, by_field.get(left, []), right_value
+ )
+ return {left: left_value}
+
+ if right_value is None:
+ right_value = left_value if want_equal else _alternative_value(
+ right, by_field.get(right, []), left_value
+ )
+ return {right: right_value}
+
+ # Both pinned. Adjust the right side only, so the left keeps whatever
+ # the rule's other clauses asked for.
+ if want_equal:
+ return {right: left_value}
+ if left_value != right_value:
+ return {}
+ return {right: _alternative_value(right, by_field.get(right, []), left_value)}
+
+
def build_positive(when: dict[str, Any]) -> dict[str, Any]:
"""Build a synthetic positive fixture from a flat match_when.
@@ -314,9 +428,14 @@ def build_positive(when: dict[str, Any]) -> dict[str, Any]:
"Pass an explicit positive= to S()."
)
by_field: dict[str, list[tuple[str, Any]]] = {}
+ comparisons: list[tuple[str, bool]] = []
for key, val in when.items():
+ if _COMPARISON_KEY_RE.match(key):
+ comparisons.append((key, bool(val)))
+ continue
field, op = split_op(key)
by_field.setdefault(field, []).append((op, val))
+
pos: dict[str, Any] = {}
for field, constraints in by_field.items():
if len(constraints) == 1:
@@ -324,6 +443,15 @@ def build_positive(when: dict[str, Any]) -> dict[str, Any]:
pos[field] = _pos_for(op, exp)
else:
pos[field] = _compose_field_value(constraints)
+
+ # Comparisons resolve last, because an operand can also be constrained
+ # by an ordinary clause. `src_cloud_in: [aws, azure, gcp]` alongside
+ # `src_cloud_neq_dst_cloud: true` put "aws" in both fields and the
+ # comparison then evaluated false — the rule's own fixture could not
+ # satisfy the rule.
+ for key, want_value in comparisons:
+ pos.update(_resolve_comparison(key, want_value, pos, by_field))
+
return pos
@@ -353,7 +481,40 @@ def build_negative(
f"neg_field={neg_field!r} not found in match_when keys: {list(when.keys())}"
)
else:
- first_key = next(iter(when.keys()))
+ # Prefer the clause the rule is *about*. Taking the first key made
+ # every negative flip whatever happened to be written first — for
+ # /etc/shadow-read-by-non-root that was `syscall`, so the negative
+ # was "a syscall that does not exist" and the non-root condition
+ # was never tested in either direction.
+ #
+ # A comparison or neq clause is the discriminating one by
+ # construction: the other clauses say *where* to look, this one
+ # says what makes it suspicious.
+ comparison_key = next(
+ (k for k in when if _COMPARISON_KEY_RE.match(k)), None
+ )
+ if comparison_key is not None:
+ # Through the resolver, not the raw operand picker: the
+ # operands may also be constrained by other clauses, and a
+ # negative carrying a value the rule's own `_in` list forbids
+ # is not a negative, it is a malformed event.
+ by_field: dict[str, list[tuple[str, Any]]] = {}
+ for key in when:
+ if _COMPARISON_KEY_RE.match(key):
+ continue
+ field, op = split_op(key)
+ by_field.setdefault(field, []).append((op, when[key]))
+ neg.update(
+ _resolve_comparison(
+ comparison_key, not bool(when[comparison_key]), neg, by_field
+ )
+ )
+ return neg
+
+ neq_key = next(
+ (k for k in when if split_op(k)[1] == "neq"), None
+ )
+ first_key = neq_key or next(iter(when.keys()))
neg_field, target_op = split_op(first_key)
target_expected = when[first_key]
diff --git a/scripts/generate_detections.py b/scripts/generate_detections.py
index ee0c0ba12..69a133ed0 100644
--- a/scripts/generate_detections.py
+++ b/scripts/generate_detections.py
@@ -38,6 +38,7 @@ from __future__ import annotations
import json
import re
+from datetime import datetime
import sys
from pathlib import Path
from typing import Any
@@ -89,6 +90,14 @@ OPERATORS: list[tuple[str, str, str]] = sorted(
("_has_any", "has_any", "HAS_ANY"),
("_not_in", "not_in", "NOT IN"),
("_match", "match", "MATCH"),
+ # `neq` is used by rules in the shipped corpus and had no
+ # operator, so `approver_role_neq: "codeowner"` was read as a
+ # field literally named `approver_role_neq` — which nothing
+ # emits, so the rule could not fire. There is deliberately no
+ # `_eq` counterpart: bare equality is already the default, and
+ # adding the suffix would split any field whose name happens to
+ # end in `_eq` for no gain.
+ ("_neq", "neq", "!="),
("_gte", "gte", ">="),
("_lte", "lte", "<="),
("_in", "in", "IN"),
@@ -184,6 +193,13 @@ def _check(field: str, op: str, expected: Any, event: dict[str, Any]) -> bool:
return actual is None
return actual == expected
+ if op == "neq":
+ # A missing field is not "different from X". Returning True would
+ # make every neq rule fire on every event lacking the field.
+ if actual is None:
+ return False
+ return actual != expected
+
if op in {"gt", "gte", "lt", "lte"}:
if not isinstance(actual, (int, float)) or isinstance(actual, bool):
return False
@@ -325,15 +341,11 @@ def _eval_clause(clause: dict[str, Any], event: dict[str, Any]) -> bool:
"""Evaluate a clause dict (possibly nested) against an event."""
for key, expected in clause.items():
if key == "any_of":
- if not isinstance(expected, list) or not any(
- _eval_clause(sub, event) for sub in expected
- ):
+ if not isinstance(expected, list) or not any(_eval_clause(sub, event) for sub in expected):
return False
continue
if key == "all_of":
- if not isinstance(expected, list) or not all(
- _eval_clause(sub, event) for sub in expected
- ):
+ if not isinstance(expected, list) or not all(_eval_clause(sub, event) for sub in expected):
return False
continue
field, op = _split_op(key)
@@ -367,16 +379,8 @@ def _description_for(spec: dict, category: str) -> str:
name = spec["name"]
fp_count = len(spec.get("fp", []))
plural = "s" if fp_count != 1 else ""
- fp_clause = (
- f" Watch the {fp_count} documented false-positive case{plural} "
- f"before tuning."
- if fp_count
- else ""
- )
- return (
- f"AiSOC v1 curated detection. Triggers on the {category} signal "
- f"described by '{name}'.{fp_clause}"
- )
+ fp_clause = f" Watch the {fp_count} documented false-positive case{plural} " f"before tuning." if fp_count else ""
+ return f"AiSOC v1 curated detection. Triggers on the {category} signal " f"described by '{name}'.{fp_clause}"
def _yaml_safe_value(value: Any) -> Any:
@@ -390,17 +394,13 @@ class _LiteralStr(str):
def _literal_str_representer(dumper: yaml.Dumper, data: _LiteralStr): # type: ignore[name-defined]
- return dumper.represent_scalar(
- "tag:yaml.org,2002:str", str(data), style="|"
- )
+ return dumper.represent_scalar("tag:yaml.org,2002:str", str(data), style="|")
yaml.add_representer(_LiteralStr, _literal_str_representer) # type: ignore[arg-type]
-def render_rule_yaml(
- *, rule_id: str, category: str, spec: dict
-) -> str:
+def render_rule_yaml(*, rule_id: str, category: str, spec: dict) -> str:
"""Render the canonical YAML for one detection rule."""
name: str = spec["name"]
severity: str = spec["severity"]
@@ -473,8 +473,7 @@ def load_id_lock() -> dict[str, str]:
return dict(json.loads(ID_LOCK.read_text(encoding="utf-8")))
except Exception: # noqa: BLE001 - a corrupt lock must not silently renumber
raise SystemExit(
- f"{ID_LOCK} is unreadable. Fix or delete it deliberately — "
- f"regenerating without it reassigns every rule id."
+ f"{ID_LOCK} is unreadable. Fix or delete it deliberately — " f"regenerating without it reassigns every rule id."
) from None
@@ -489,9 +488,7 @@ def assign_ids(categories: dict) -> tuple[dict[str, str], list[str]]:
for category, specs in sorted(categories.items()):
used = {
- int(rid.rsplit("-", 1)[1])
- for key, rid in lock.items()
- if key.startswith(f"{category}/") and rid.rsplit("-", 1)[-1].isdigit()
+ int(rid.rsplit("-", 1)[1]) for key, rid in lock.items() if key.startswith(f"{category}/") and rid.rsplit("-", 1)[-1].isdigit()
}
next_free = max(used) + 1 if used else 1
@@ -508,13 +505,145 @@ def assign_ids(categories: dict) -> tuple[dict[str, str], list[str]]:
return lock, newly
+# ─── Derived fields ──────────────────────────────────────────────────────────
+#
+# Vendored into `services/fusion/app/services/derived_fields.py`, and kept in
+# parity by `services/fusion/tests/test_detection_matcher_parity.py` — the
+# same arrangement as `matches()` itself, for the same reason: services
+# cannot import repo-root scripts at runtime, and two copies that can drift
+# silently are worse than one copy plus a gate.
+#
+# These compute fields the engine can derive from an event it already has,
+# so a rule matching `actor_eq_target` or `is_business_hours` is reachable
+# even though no connector emits either.
+
+_COMPARISON_RE = re.compile(r"^(?P<left>.+?)_(?P<op>eq|neq)_(?P<right>.+)$")
+
+DEFAULT_BUSINESS_START_HOUR = 8
+DEFAULT_BUSINESS_END_HOUR = 18
+
+_TIME_FIELDS = ("event_time", "timestamp", "time", "@timestamp", "ingest_time")
+
+
+def _parse_event_time(event: dict[str, Any]) -> datetime | None:
+ for field in _TIME_FIELDS:
+ raw = event.get(field)
+ if raw is None:
+ continue
+ if isinstance(raw, datetime):
+ return raw
+ if isinstance(raw, (int, float)) and not isinstance(raw, bool):
+ try:
+ seconds = raw / 1000 if raw > 1e11 else raw
+ return datetime.fromtimestamp(seconds) # noqa: DTZ006
+ except (OSError, ValueError, OverflowError):
+ continue
+ if isinstance(raw, str):
+ try:
+ return datetime.fromisoformat(raw.replace("Z", "+00:00"))
+ except ValueError:
+ continue
+ return None
+
+
+def _normalise_operand(value: Any) -> Any:
+ return value.strip().lower() if isinstance(value, str) else value
+
+
+def comparison_fields(event: dict[str, Any], requested: set[str] | None = None) -> dict[str, bool]:
+ """`<a>_eq_<b>` / `<a>_neq_<b>` for the names some rule asks for.
+
+ A missing operand yields no key rather than False: a rule must not fire
+ because we did not know.
+ """
+ if not requested:
+ return {}
+ out: dict[str, bool] = {}
+ for name in requested:
+ match = _COMPARISON_RE.match(name)
+ if not match:
+ continue
+ left = event.get(match.group("left"))
+ right = event.get(match.group("right"))
+ if left is None or right is None:
+ continue
+ equal = _normalise_operand(left) == _normalise_operand(right)
+ out[name] = equal if match.group("op") == "eq" else not equal
+ return out
+
+
+def time_of_day_fields(
+ event: dict[str, Any],
+ *,
+ business_start: int = DEFAULT_BUSINESS_START_HOUR,
+ business_end: int = DEFAULT_BUSINESS_END_HOUR,
+) -> dict[str, bool]:
+ """`is_business_hours` / `is_after_hours` / `is_weekend`.
+
+ Nothing when the event has no parseable timestamp: an unknown time is
+ not "outside business hours".
+ """
+ when = _parse_event_time(event)
+ if when is None:
+ return {}
+ weekend = when.weekday() >= 5
+ in_hours = (not weekend) and business_start <= when.hour < business_end
+ return {
+ "is_business_hours": in_hours,
+ "is_after_hours": not in_hours,
+ "is_weekend": weekend,
+ }
+
+
+def enrich(
+ event: dict[str, Any],
+ requested: set[str] | None = None,
+ *,
+ business_start: int = DEFAULT_BUSINESS_START_HOUR,
+ business_end: int = DEFAULT_BUSINESS_END_HOUR,
+) -> dict[str, Any]:
+ """`event` plus every derivable field the rules ask for.
+
+ A derived key never overwrites one the connector supplied: a vendor's
+ own answer about its own tenant's hours beats ours.
+ """
+ derived: dict[str, Any] = {}
+ derived.update(time_of_day_fields(event, business_start=business_start, business_end=business_end))
+ derived.update(comparison_fields(event, requested))
+ return event if not derived else {**derived, **event}
+
+
+def requested_derived_fields(rules: list[dict[str, Any]]) -> set[str]:
+ """Derivable field names the ruleset matches on.
+
+ Computed once per ruleset rather than per event: the set changes when
+ rules change, not when traffic arrives.
+ """
+ names: set[str] = set()
+
+ def walk(clause: Any) -> None:
+ if isinstance(clause, dict):
+ for key, value in clause.items():
+ if key in {"any_of", "all_of", "not"}:
+ walk(value)
+ continue
+ field, _ = _split_op(key)
+ if _COMPARISON_RE.match(field) or field.startswith("is_"):
+ names.add(field)
+ elif isinstance(clause, list):
+ for item in clause:
+ walk(item)
+
+ for rule in rules:
+ walk(rule.get("match_when") or {})
+ return names
+
+
def write_pack(*, dry_run: bool = False) -> dict[str, int]:
"""Generate the full pack to disk. Returns counts per category."""
id_lock, newly_assigned = assign_ids(CATEGORIES)
if newly_assigned and not dry_run:
- ID_LOCK.write_text(
- json.dumps(id_lock, indent=2, sort_keys=True) + "\n", encoding="utf-8"
- )
+ ID_LOCK.write_text(json.dumps(id_lock, indent=2, sort_keys=True) + "\n", encoding="utf-8")
print(f" assigned {len(newly_assigned)} new rule id(s)")
counts: dict[str, int] = {}
@@ -532,9 +661,7 @@ def write_pack(*, dry_run: bool = False) -> dict[str, int]:
# Looked up, not computed from position. See ID_LOCK.
rule_id = id_lock[f"{category}/{slug}"]
- yaml_text = render_rule_yaml(
- rule_id=rule_id, category=category, spec=spec
- )
+ yaml_text = render_rule_yaml(rule_id=rule_id, category=category, spec=spec)
yaml_path = cat_dir / f"{slug}.yaml"
pos_path = pos_dir / f"{slug}.json"
diff --git a/scripts/validate_detections.py b/scripts/validate_detections.py
index 74c457cb9..9011cf87c 100755
--- a/scripts/validate_detections.py
+++ b/scripts/validate_detections.py
@@ -59,7 +59,11 @@ if str(SCRIPTS_DIR) not in sys.path:
sys.path.insert(0, str(SCRIPTS_DIR))
from detection_specs_index import all_specs # noqa: E402
-from generate_detections import matches # noqa: E402
+from generate_detections import ( # noqa: E402
+ enrich,
+ matches,
+ requested_derived_fields,
+)
VALID_SEVERITIES = {"low", "medium", "high", "critical"}
VALID_CATEGORIES = {
@@ -114,9 +118,7 @@ IMPORTED_TIERS: dict[str, dict[str, str]] = {
# Spec lookup — for native fixture replay only.
# =============================================================================
-_SPEC_BY_KEY: dict[tuple[str, str], dict[str, Any]] = {
- (cat, spec["slug"]): spec for cat, spec in all_specs()
-}
+_SPEC_BY_KEY: dict[tuple[str, str], dict[str, Any]] = {(cat, spec["slug"]): spec for cat, spec in all_specs()}
# =============================================================================
@@ -229,9 +231,7 @@ def validate_rule(
return [f"YAML parse error: {exc}"], None
if not isinstance(rule, dict):
- return [
- "Rule is not a YAML mapping (expected key: value pairs at top level)"
- ], None
+ return ["Rule is not a YAML mapping (expected key: value pairs at top level)"], None
for field in REQUIRED_FIELDS:
if field not in rule:
@@ -241,42 +241,23 @@ def validate_rule(
return errors, rule
if rule["severity"] not in VALID_SEVERITIES:
- errors.append(
- f"Invalid severity '{rule['severity']}'; must be one of: "
- f"{', '.join(sorted(VALID_SEVERITIES))}"
- )
+ errors.append(f"Invalid severity '{rule['severity']}'; must be one of: " f"{', '.join(sorted(VALID_SEVERITIES))}")
rule_category = rule.get("category")
if rule_category and rule_category not in VALID_CATEGORIES:
- errors.append(
- f"Invalid category '{rule_category}'; must be one of: "
- f"{', '.join(sorted(VALID_CATEGORIES))}"
- )
+ errors.append(f"Invalid category '{rule_category}'; must be one of: " f"{', '.join(sorted(VALID_CATEGORIES))}")
expected_category = classification["category"]
- if (
- rule_category
- and expected_category
- and rule_category != expected_category
- ):
- errors.append(
- f"Rule category '{rule_category}' does not match directory "
- f"'{expected_category}'"
- )
+ if rule_category and expected_category and rule_category != expected_category:
+ errors.append(f"Rule category '{rule_category}' does not match directory " f"'{expected_category}'")
rule_id = str(rule["id"])
expected_prefix = classification["id_prefix"]
if expected_prefix and not rule_id.startswith(expected_prefix):
- errors.append(
- f"Rule id '{rule_id}' must start with '{expected_prefix}' "
- f"(tier: {classification['tier']})"
- )
+ errors.append(f"Rule id '{rule_id}' must start with '{expected_prefix}' " f"(tier: {classification['tier']})")
if rule_id in seen_ids:
- errors.append(
- f"Duplicate id '{rule_id}' — already defined in "
- f"{seen_ids[rule_id]}"
- )
+ errors.append(f"Duplicate id '{rule_id}' — already defined in " f"{seen_ids[rule_id]}")
else:
seen_ids[rule_id] = path
@@ -299,10 +280,7 @@ def _validate_provenance(rule: dict[str, Any]) -> list[str]:
errors: list[str] = []
provenance = rule.get("provenance")
if not isinstance(provenance, dict):
- return [
- "Imported rule is missing required 'provenance' block "
- "(see tools/detection_import/README.md)"
- ]
+ return ["Imported rule is missing required 'provenance' block " "(see tools/detection_import/README.md)"]
for field in REQUIRED_PROVENANCE_FIELDS:
value = provenance.get(field)
if value in (None, ""):
@@ -310,9 +288,7 @@ def _validate_provenance(rule: dict[str, Any]) -> list[str]:
return errors
-def replay_fixture(
- rule_path: Path, rule: dict[str, Any], strict: bool
-) -> list[str]:
+def replay_fixture(rule_path: Path, rule: dict[str, Any], strict: bool) -> list[str]:
"""Replay positive + negative fixtures against the canonical spec.
Native tier only. Looks up the spec for this rule by ``(category,
@@ -335,13 +311,9 @@ def replay_fixture(
if pos_missing or neg_missing:
msg_parts = []
if pos_missing:
- msg_parts.append(
- f"missing positive fixture {pos_path.relative_to(ROOT)}"
- )
+ msg_parts.append(f"missing positive fixture {pos_path.relative_to(ROOT)}")
if neg_missing:
- msg_parts.append(
- f"missing negative fixture {neg_path.relative_to(ROOT)}"
- )
+ msg_parts.append(f"missing negative fixture {neg_path.relative_to(ROOT)}")
msg = "; ".join(msg_parts)
if strict:
errors.append(msg)
@@ -351,10 +323,7 @@ def replay_fixture(
spec = _SPEC_BY_KEY.get((category, slug))
if spec is None:
- msg = (
- f"no canonical spec found for ({category}, {slug}); "
- f"hand-authored rule — fixture replay skipped"
- )
+ msg = f"no canonical spec found for ({category}, {slug}); " f"hand-authored rule — fixture replay skipped"
errors.append(f"WARN: {msg}")
return errors
@@ -371,14 +340,18 @@ def replay_fixture(
errors.append(f"fixture load error: {exc}")
return errors
+ # Enriched the way the engine enriches. Replaying against the bare
+ # matcher tests a pipeline production does not run — a rule matching a
+ # derived field would fail here while working live, which trains people
+ # to weaken the gate.
+ wanted = requested_derived_fields([{"match_when": match_when}])
+ pos_event = enrich(pos_event, wanted)
+ neg_event = enrich(neg_event, wanted)
+
if not matches(match_when, pos_event):
- errors.append(
- "positive fixture did NOT match match_when (expected match)"
- )
+ errors.append("positive fixture did NOT match match_when (expected match)")
if matches(match_when, neg_event):
- errors.append(
- "negative fixture DID match match_when (expected no match)"
- )
+ errors.append("negative fixture DID match match_when (expected no match)")
return errors
@@ -432,12 +405,7 @@ def main() -> int:
rule_failed = bool(errors)
replay_errors: list[str] = []
- if (
- rule
- and not rule_failed
- and tier == "native"
- and path.parent.name in VALID_CATEGORIES
- ):
+ if rule and not rule_failed and tier == "native" and path.parent.name in VALID_CATEGORIES:
replay_errors = replay_fixture(path, rule, strict=strict)
warnings = [e for e in replay_errors if e.startswith("WARN:")]
@@ -460,18 +428,8 @@ def main() -> int:
print(f" {w}")
print(f"\n{'─' * 60}")
- print(
- f"Validated {total} rules — {total - failed} passed, {failed} failed, "
- f"{fixture_warnings} fixture warnings"
- )
- print(
- " Tiers: "
- + ", ".join(
- f"{tier}={count}"
- for tier, count in sorted(tier_counts.items())
- if count > 0
- )
- )
+ print(f"Validated {total} rules — {total - failed} passed, {failed} failed, " f"{fixture_warnings} fixture warnings")
+ print(" Tiers: " + ", ".join(f"{tier}={count}" for tier, count in sorted(tier_counts.items()) if count > 0))
if quarantine_count:
print(f" Quarantined (parsed-but-disabled): {quarantine_count}")
diff --git a/services/actions/app/live_actions/builtins.py b/services/actions/app/live_actions/builtins.py
index 3311b1caa..6471772b5 100644
--- a/services/actions/app/live_actions/builtins.py
+++ b/services/actions/app/live_actions/builtins.py
@@ -83,6 +83,7 @@ from .investigation_reads import (
OktaGetUserActivity,
)
from .models import LiveActionRequest, LiveActionResult, LiveActionStatus
+from .vendor_breadth import VENDOR_BREADTH_EXECUTORS
logger = structlog.get_logger(__name__)
@@ -595,6 +596,10 @@ _BUILTIN_ADAPTERS: tuple[type[LiveActionExecutor], ...] = (
# Rollback for the most disruptive action, which had no executor.
CrowdStrikeUnisolateHost,
DefenderUnisolateHost,
+ # Vendor breadth: capabilities the clients already implemented and
+ # the registry could not reach. SentinelOne exposed seven
+ # operations and one was wired; Entra six and one.
+ *VENDOR_BREADTH_EXECUTORS,
# Endpoint
CrowdStrikeIsolateHost,
DefenderIsolateHost,
diff --git a/services/actions/app/live_actions/vendor_breadth.py b/services/actions/app/live_actions/vendor_breadth.py
new file mode 100644
index 000000000..8e9aeb518
--- /dev/null
+++ b/services/actions/app/live_actions/vendor_breadth.py
@@ -0,0 +1,483 @@
+"""Vendor capabilities the clients implement and the registry could not reach.
+
+Seventeen vendors are wired and most expose exactly one verb, so the action
+registry looked like breadth and behaved like a single button per product.
+The reason is not that the integrations are shallow: `SentinelOneClient`
+implements seven operations and one was reachable; `AzureEntraClient`
+implements six and one was reachable. The work was done and nothing
+connected it — the same pattern v8.0 spent a release removing, at the scale
+of a whole capability surface.
+
+Every executor here calls a method that already existed. Nothing was
+stubbed, and nothing is declared that a client cannot do.
+
+Several are the reverse actions the contract already promised. `enable_user`
+was declared as `disable_user`'s reverse and had no implementation on any
+vendor, so an account disabled during an incident had no route back through
+the platform that disabled it. `allow_ip`, `allow_domain` and
+`revoke_session` were in the same state.
+
+The shared rules, both inherited from `investigation_reads.py`:
+
+**A read or action failure is reported, never silently absorbed.** An
+executor returns FAILED with the vendor's own message rather than a
+success that means "we asked".
+
+**Credentials absent is not the same as the action failing.** Missing
+configuration says so explicitly, because "we could not reach the vendor"
+and "the vendor refused" send an operator to different places.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Awaitable, Callable
+from typing import Any
+
+import structlog
+
+from app.clients.factories import (
+ _cloudflare_client,
+ _entra_client,
+ _fortigate_client,
+ _gws_client,
+ _panos_client,
+ _s1_client,
+)
+from app.live_actions.capability_contracts import apply_contract
+from app.live_actions.executor import LiveActionExecutor
+from app.live_actions.models import LiveActionRequest, LiveActionResult, LiveActionStatus
+
+logger = structlog.get_logger(__name__)
+
+
+def _result(
+ executor: LiveActionExecutor,
+ request: LiveActionRequest,
+ status: LiveActionStatus,
+ summary: str,
+ *,
+ details: dict[str, Any] | None = None,
+ error: str | None = None,
+) -> LiveActionResult:
+ return LiveActionResult(
+ request_id=request.request_id,
+ status=status,
+ capability=executor.capability,
+ vendor_id=executor.vendor_id,
+ summary=summary,
+ details=details or {},
+ error=error,
+ )
+
+
+async def _run(
+ executor: LiveActionExecutor,
+ request: LiveActionRequest,
+ vendor: str,
+ factory: Callable[[dict[str, Any]], Any],
+ call: Callable[[Any, str, dict[str, Any]], Awaitable[dict[str, Any]]],
+ summary: str,
+) -> LiveActionResult:
+ """Shared execute body: dry-run, credentials, call, error handling.
+
+ Extracted because seventeen executors repeating it is seventeen chances
+ to forget the dry-run check or to let an exception escape — and an
+ exception escaping the executor is how a single bad action wedges the
+ agent loop.
+ """
+ params: dict[str, Any] = request.params or {}
+ target = str(request.target or "")
+
+ if request.dry_run:
+ return _result(
+ executor,
+ request,
+ LiveActionStatus.SIMULATED,
+ f"would {executor.capability} {target} on {vendor}",
+ details={"would_run": executor.capability, "target": target, "vendor": vendor},
+ )
+
+ client = factory(params)
+ if client is None:
+ return _result(
+ executor,
+ request,
+ LiveActionStatus.FAILED,
+ f"{vendor} credentials not configured",
+ error=(
+ f"{vendor} credentials are not configured, so {executor.capability} "
+ f"was not attempted. This is not a statement about the target."
+ ),
+ )
+
+ try:
+ response = await call(client, target, params)
+ except Exception as exc: # noqa: BLE001 - a vendor error is FAILED, never silent
+ logger.warning(
+ "vendor_action.failed",
+ vendor=executor.vendor_id,
+ capability=executor.capability,
+ error=str(exc),
+ )
+ return _result(
+ executor,
+ request,
+ LiveActionStatus.FAILED,
+ f"{vendor} {executor.capability} failed",
+ error=f"{vendor} {executor.capability} failed: {exc}",
+ )
+
+ return _result(
+ executor,
+ request,
+ LiveActionStatus.SUCCEEDED,
+ summary.format(target=target),
+ details={"target": target, "vendor_response": response},
+ )
+
+
+# ─── SentinelOne ─────────────────────────────────────────────────────────────
+#
+# Seven client methods, one reachable. `lift_containment` is the one that
+# mattered: containment with no release means an incident closes and the
+# host stays off the network until someone remembers.
+
+
+@apply_contract
+class SentinelOneUnisolateHost(LiveActionExecutor):
+ vendor_id = "sentinelone"
+ capability = "unisolate_host"
+ description = "Lift network containment on a SentinelOne agent."
+ requires_credentials = True
+
+ async def execute(self, request: LiveActionRequest) -> LiveActionResult:
+ return await _run(
+ self,
+ request,
+ "SentinelOne",
+ _s1_client,
+ lambda c, t, p: c.lift_containment(t),
+ "containment lifted on {target}",
+ )
+
+
+@apply_contract
+class SentinelOneKillProcess(LiveActionExecutor):
+ vendor_id = "sentinelone"
+ capability = "kill_process"
+ description = "Terminate a process on a SentinelOne-managed host."
+ requires_credentials = True
+
+ async def execute(self, request: LiveActionRequest) -> LiveActionResult:
+ return await _run(
+ self,
+ request,
+ "SentinelOne",
+ _s1_client,
+ lambda c, t, p: c.kill_process(t, pid=p.get("pid"), process_name=p.get("process_name")),
+ "process terminated on {target}",
+ )
+
+
+@apply_contract
+class SentinelOneQuarantineFile(LiveActionExecutor):
+ vendor_id = "sentinelone"
+ capability = "quarantine_file"
+ description = "Quarantine a file on a SentinelOne-managed host."
+ requires_credentials = True
+
+ async def execute(self, request: LiveActionRequest) -> LiveActionResult:
+ return await _run(
+ self,
+ request,
+ "SentinelOne",
+ _s1_client,
+ lambda c, t, p: c.quarantine_file(t, str(p.get("file_path", ""))),
+ "file quarantined on {target}",
+ )
+
+
+@apply_contract
+class SentinelOneRunAVScan(LiveActionExecutor):
+ vendor_id = "sentinelone"
+ capability = "run_av_scan"
+ description = "Start an on-demand scan on a SentinelOne-managed host."
+ requires_credentials = True
+
+ async def execute(self, request: LiveActionRequest) -> LiveActionResult:
+ return await _run(
+ self,
+ request,
+ "SentinelOne",
+ _s1_client,
+ lambda c, t, p: c.run_av_scan(t, scan_type=str(p.get("scan_type", "Full"))),
+ "scan started on {target}",
+ )
+
+
+@apply_contract
+class SentinelOneRunScript(LiveActionExecutor):
+ vendor_id = "sentinelone"
+ capability = "run_script"
+ description = "Run a remote script on a SentinelOne-managed host."
+ requires_credentials = True
+
+ async def execute(self, request: LiveActionRequest) -> LiveActionResult:
+ return await _run(
+ self,
+ request,
+ "SentinelOne",
+ _s1_client,
+ lambda c, t, p: c.run_script(t, str(p.get("script_content", ""))),
+ "script dispatched to {target}",
+ )
+
+
+# ─── Microsoft Entra ─────────────────────────────────────────────────────────
+
+
+@apply_contract
+class EntraEnableUser(LiveActionExecutor):
+ vendor_id = "azure_entra"
+ capability = "enable_user"
+ description = "Re-enable an account in Microsoft Entra ID."
+ requires_credentials = True
+
+ async def execute(self, request: LiveActionRequest) -> LiveActionResult:
+ return await _run(
+ self,
+ request,
+ "Entra",
+ _entra_client,
+ lambda c, t, p: c.enable_user(t),
+ "{target} re-enabled",
+ )
+
+
+@apply_contract
+class EntraRevokeSession(LiveActionExecutor):
+ vendor_id = "azure_entra"
+ capability = "revoke_session"
+ description = "Revoke all refresh tokens for an Entra ID account."
+ requires_credentials = True
+
+ async def execute(self, request: LiveActionRequest) -> LiveActionResult:
+ return await _run(
+ self,
+ request,
+ "Entra",
+ _entra_client,
+ lambda c, t, p: c.revoke_sessions(t),
+ "sessions revoked for {target}",
+ )
+
+
+@apply_contract
+class EntraResetPassword(LiveActionExecutor):
+ vendor_id = "azure_entra"
+ capability = "reset_password"
+ description = "Force a password reset on an Entra ID account."
+ requires_credentials = True
+
+ async def execute(self, request: LiveActionRequest) -> LiveActionResult:
+ return await _run(
+ self,
+ request,
+ "Entra",
+ _entra_client,
+ lambda c, t, p: c.reset_password(t),
+ "password reset forced for {target}",
+ )
+
+
+@apply_contract
+class EntraForceMFA(LiveActionExecutor):
+ vendor_id = "azure_entra"
+ capability = "force_mfa"
+ description = "Require MFA re-registration for an Entra ID account."
+ requires_credentials = True
+
+ async def execute(self, request: LiveActionRequest) -> LiveActionResult:
+ return await _run(
+ self,
+ request,
+ "Entra",
+ _entra_client,
+ lambda c, t, p: c.require_mfa(t),
+ "MFA re-registration required for {target}",
+ )
+
+
+# ─── Google Workspace ────────────────────────────────────────────────────────
+
+
+@apply_contract
+class GoogleWorkspaceEnableUser(LiveActionExecutor):
+ vendor_id = "google_workspace"
+ capability = "enable_user"
+ description = "Un-suspend a Google Workspace account."
+ requires_credentials = True
+
+ async def execute(self, request: LiveActionRequest) -> LiveActionResult:
+ return await _run(
+ self,
+ request,
+ "Google Workspace",
+ _gws_client,
+ lambda c, t, p: c.unsuspend_user(t),
+ "{target} un-suspended",
+ )
+
+
+@apply_contract
+class GoogleWorkspaceRevokeSession(LiveActionExecutor):
+ vendor_id = "google_workspace"
+ capability = "revoke_session"
+ description = "Revoke sign-in cookies for a Google Workspace account."
+ requires_credentials = True
+
+ async def execute(self, request: LiveActionRequest) -> LiveActionResult:
+ return await _run(
+ self,
+ request,
+ "Google Workspace",
+ _gws_client,
+ lambda c, t, p: c.revoke_sessions(t),
+ "sessions revoked for {target}",
+ )
+
+
+@apply_contract
+class GoogleWorkspaceResetPassword(LiveActionExecutor):
+ vendor_id = "google_workspace"
+ capability = "reset_password"
+ description = "Force a password change on a Google Workspace account."
+ requires_credentials = True
+
+ async def execute(self, request: LiveActionRequest) -> LiveActionResult:
+ return await _run(
+ self,
+ request,
+ "Google Workspace",
+ _gws_client,
+ lambda c, t, p: c.reset_password(t),
+ "password reset forced for {target}",
+ )
+
+
+# ─── Network ─────────────────────────────────────────────────────────────────
+#
+# Every block had no unblock. A perimeter block with no release is a
+# permanent one, applied during an incident by an automation nobody
+# revisits — and the address is usually reassigned within weeks.
+
+
+@apply_contract
+class CloudflareAllowIP(LiveActionExecutor):
+ vendor_id = "cloudflare"
+ capability = "allow_ip"
+ description = "Remove a Cloudflare zone-level IP block."
+ requires_credentials = True
+
+ async def execute(self, request: LiveActionRequest) -> LiveActionResult:
+ return await _run(
+ self,
+ request,
+ "Cloudflare",
+ _cloudflare_client,
+ lambda c, t, p: c.unblock_ip_zone(str(p.get("rule_id", "")), str(p.get("cf_zone_id", ""))),
+ "block removed for {target}",
+ )
+
+
+@apply_contract
+class CloudflareBlockDomain(LiveActionExecutor):
+ vendor_id = "cloudflare"
+ capability = "block_domain"
+ description = "Sinkhole a domain via Cloudflare Gateway."
+ requires_credentials = True
+
+ async def execute(self, request: LiveActionRequest) -> LiveActionResult:
+ return await _run(
+ self,
+ request,
+ "Cloudflare",
+ _cloudflare_client,
+ lambda c, t, p: c.sinkhole_domain(t, str(p.get("cf_account_id", "")), str(p.get("cf_list_id", ""))),
+ "{target} sinkholed",
+ )
+
+
+@apply_contract
+class CloudflareAllowDomain(LiveActionExecutor):
+ vendor_id = "cloudflare"
+ capability = "allow_domain"
+ description = "Remove a Cloudflare Gateway domain sinkhole."
+ requires_credentials = True
+
+ async def execute(self, request: LiveActionRequest) -> LiveActionResult:
+ return await _run(
+ self,
+ request,
+ "Cloudflare",
+ _cloudflare_client,
+ lambda c, t, p: c.unsinkhole_domain(t, str(p.get("cf_account_id", "")), str(p.get("cf_list_id", ""))),
+ "sinkhole removed for {target}",
+ )
+
+
+@apply_contract
+class PanOsAllowIP(LiveActionExecutor):
+ vendor_id = "panos"
+ capability = "allow_ip"
+ description = "Remove a PAN-OS dynamic address-group tag from an IP."
+ requires_credentials = True
+
+ async def execute(self, request: LiveActionRequest) -> LiveActionResult:
+ return await _run(
+ self,
+ request,
+ "PAN-OS",
+ _panos_client,
+ lambda c, t, p: c.unblock_ip(t, str(p.get("pan_tag", "aisoc-blocked"))),
+ "tag removed from {target}",
+ )
+
+
+@apply_contract
+class FortiGateAllowIP(LiveActionExecutor):
+ vendor_id = "fortigate"
+ capability = "allow_ip"
+ description = "Remove an IP from the FortiGate block address group."
+ requires_credentials = True
+
+ async def execute(self, request: LiveActionRequest) -> LiveActionResult:
+ return await _run(
+ self,
+ request,
+ "FortiGate",
+ _fortigate_client,
+ lambda c, t, p: c.unblock_ip(t, str(p.get("fgt_address_group", ""))),
+ "{target} removed from the block group",
+ )
+
+
+VENDOR_BREADTH_EXECUTORS: tuple[type[LiveActionExecutor], ...] = (
+ SentinelOneUnisolateHost,
+ SentinelOneKillProcess,
+ SentinelOneQuarantineFile,
+ SentinelOneRunAVScan,
+ SentinelOneRunScript,
+ EntraEnableUser,
+ EntraRevokeSession,
+ EntraResetPassword,
+ EntraForceMFA,
+ GoogleWorkspaceEnableUser,
+ GoogleWorkspaceRevokeSession,
+ GoogleWorkspaceResetPassword,
+ CloudflareAllowIP,
+ CloudflareBlockDomain,
+ CloudflareAllowDomain,
+ PanOsAllowIP,
+ FortiGateAllowIP,
+)
diff --git a/services/actions/tests/test_live_action_builtins.py b/services/actions/tests/test_live_action_builtins.py
index e44daa6c0..fa297af51 100644
--- a/services/actions/tests/test_live_action_builtins.py
+++ b/services/actions/tests/test_live_action_builtins.py
@@ -51,10 +51,12 @@ def test_register_builtin_executors_returns_full_count() -> None:
cheap reminder to update docs / discovery snapshots.
"""
count = register_builtin_executors()
- # 19 original + 10 Phase B2 vendors (SentinelOne/Entra/GWS/PAN-OS/FortiGate/
- # Cloudflare/Jira/ServiceNow/PagerDuty/Slack) + 6 read/rollback verbs
- # (4 read-only investigation reads, 2 unisolate arms).
- assert count == 35
+ # 19 original + 10 Phase B2 vendors + 6 read/rollback verbs + 17
+ # vendor-breadth executors. The last group added no new integrations:
+ # every one calls a client method that already existed and had no
+ # route through the registry — SentinelOne implemented seven
+ # operations with one reachable, Entra six with one.
+ assert count == 52
def test_builtin_executors_cover_canonical_vendor_capability_pairs() -> None:
@@ -69,52 +71,58 @@ def test_builtin_executors_cover_canonical_vendor_capability_pairs() -> None:
pairs = {(d.vendor_id, d.capability) for d in descriptors}
expected = {
- # Endpoint
+ ("aws_security_groups", "allow_ip"),
+ ("aws_security_groups", "block_ip"),
+ ("azure_entra", "disable_user"),
+ ("azure_entra", "enable_user"),
+ ("azure_entra", "force_mfa"),
+ ("azure_entra", "reset_password"),
+ ("azure_entra", "revoke_session"),
+ ("cloudflare", "allow_domain"),
+ ("cloudflare", "allow_ip"),
+ ("cloudflare", "block_domain"),
+ ("cloudflare", "block_ip"),
+ ("crowdstrike", "get_detections"),
+ ("crowdstrike", "get_host"),
("crowdstrike", "isolate_host"),
- ("defender", "isolate_host"),
- ("crowdstrike", "quarantine_file"),
("crowdstrike", "kill_process"),
+ ("crowdstrike", "quarantine_file"),
("crowdstrike", "run_script"),
+ ("crowdstrike", "unisolate_host"),
+ ("defender", "block_ioc"),
+ ("defender", "get_host"),
+ ("defender", "isolate_host"),
("defender", "run_av_scan"),
- # Identity
- ("okta", "disable_user"),
- ("okta", "reset_password"),
- ("okta", "suspend_session"),
- ("okta", "force_mfa"),
- # Network
- ("aws_security_groups", "block_ip"),
- ("aws_security_groups", "allow_ip"),
- ("generic", "block_domain"),
- # SIEM
- ("splunk", "search_siem"),
+ ("defender", "unisolate_host"),
("elastic", "search_siem"),
- ("splunk", "create_notable_event"),
- ("splunk", "sync_detection_rule"),
("elastic", "update_watcher"),
- ("defender", "block_ioc"),
- # Phase B2 — previously-unregistered vendors
- ("sentinelone", "isolate_host"),
- ("azure_entra", "disable_user"),
- ("google_workspace", "disable_user"),
- ("panos", "block_ip"),
+ ("fortigate", "allow_ip"),
("fortigate", "block_ip"),
- ("cloudflare", "block_ip"),
+ ("generic", "block_domain"),
+ ("google_workspace", "disable_user"),
+ ("google_workspace", "enable_user"),
+ ("google_workspace", "reset_password"),
+ ("google_workspace", "revoke_session"),
("jira", "create_ticket"),
- ("servicenow", "create_ticket"),
+ ("okta", "disable_user"),
+ ("okta", "force_mfa"),
+ ("okta", "get_user_activity"),
+ ("okta", "reset_password"),
+ ("okta", "suspend_session"),
("pagerduty", "create_ticket"),
+ ("panos", "allow_ip"),
+ ("panos", "block_ip"),
+ ("sentinelone", "isolate_host"),
+ ("sentinelone", "kill_process"),
+ ("sentinelone", "quarantine_file"),
+ ("sentinelone", "run_av_scan"),
+ ("sentinelone", "run_script"),
+ ("sentinelone", "unisolate_host"),
+ ("servicenow", "create_ticket"),
("slack", "notify"),
- # Read-only investigation verbs. Twenty-nine executors could change
- # the estate and exactly one could ask it a question, so an
- # investigation could only ever reach the lake.
- ("crowdstrike", "get_host"),
- ("crowdstrike", "get_detections"),
- ("defender", "get_host"),
- ("okta", "get_user_activity"),
- # The rollback for the most disruptive action, which had no
- # executor: isolate_host declared unisolate_host as its reverse and
- # dispatch answered executor_not_found.
- ("crowdstrike", "unisolate_host"),
- ("defender", "unisolate_host"),
+ ("splunk", "create_notable_event"),
+ ("splunk", "search_siem"),
+ ("splunk", "sync_detection_rule"),
}
assert pairs == expected
@@ -145,7 +153,7 @@ def test_register_builtin_executors_is_idempotent_with_overwrite() -> None:
register_builtin_executors()
# Second call without overwrite would raise — confirm overwrite works.
count = register_builtin_executors(overwrite=True)
- assert count == 35
+ assert count == 52
def test_register_builtin_twice_without_overwrite_raises() -> None:
diff --git a/services/actions/tests/test_vendor_breadth.py b/services/actions/tests/test_vendor_breadth.py
new file mode 100644
index 000000000..949606a03
--- /dev/null
+++ b/services/actions/tests/test_vendor_breadth.py
@@ -0,0 +1,179 @@
+"""Vendor capabilities the clients implemented and the registry could not reach.
+
+Seventeen vendors were wired and most exposed exactly one verb, so the
+registry looked like breadth and behaved like a single button per product.
+Not because the integrations were shallow: `SentinelOneClient` implements
+seven operations and one was reachable; `AzureEntraClient` implements six
+and one was reachable. The work existed and nothing connected it.
+
+The tests below are structural on purpose. Asserting that seventeen
+executors each forward to the right client method is low-value repetition;
+asserting that **every executor calls a method its client actually has** is
+the property that would have caught the whole class of defect, and it is
+the one that breaks when a client is refactored.
+"""
+
+from __future__ import annotations
+
+import inspect
+from unittest.mock import AsyncMock
+
+import pytest
+from app.live_actions import vendor_breadth
+from app.live_actions.capability_contracts import CAPABILITY_CONTRACTS
+from app.live_actions.models import LiveActionRequest, LiveActionStatus
+from app.live_actions.vendor_breadth import (
+ VENDOR_BREADTH_EXECUTORS,
+ CloudflareAllowIP,
+ EntraEnableUser,
+ SentinelOneUnisolateHost,
+)
+
+#: executor -> (factory name in vendor_breadth, client method it must call)
+EXPECTED_CALLS = {
+ "SentinelOneUnisolateHost": ("_s1_client", "lift_containment"),
+ "SentinelOneKillProcess": ("_s1_client", "kill_process"),
+ "SentinelOneQuarantineFile": ("_s1_client", "quarantine_file"),
+ "SentinelOneRunAVScan": ("_s1_client", "run_av_scan"),
+ "SentinelOneRunScript": ("_s1_client", "run_script"),
+ "EntraEnableUser": ("_entra_client", "enable_user"),
+ "EntraRevokeSession": ("_entra_client", "revoke_sessions"),
+ "EntraResetPassword": ("_entra_client", "reset_password"),
+ "EntraForceMFA": ("_entra_client", "require_mfa"),
+ "GoogleWorkspaceEnableUser": ("_gws_client", "unsuspend_user"),
+ "GoogleWorkspaceRevokeSession": ("_gws_client", "revoke_sessions"),
+ "GoogleWorkspaceResetPassword": ("_gws_client", "reset_password"),
+ "CloudflareAllowIP": ("_cloudflare_client", "unblock_ip_zone"),
+ "CloudflareBlockDomain": ("_cloudflare_client", "sinkhole_domain"),
+ "CloudflareAllowDomain": ("_cloudflare_client", "unsinkhole_domain"),
+ "PanOsAllowIP": ("_panos_client", "unblock_ip"),
+ "FortiGateAllowIP": ("_fortigate_client", "unblock_ip"),
+}
+
+REAL_CLIENTS = {
+ "_s1_client": "app.clients.sentinelone_client.SentinelOneClient",
+ "_entra_client": "app.clients.azure_entra_client.AzureEntraClient",
+ "_gws_client": "app.clients.google_workspace_client.GoogleWorkspaceClient",
+ "_cloudflare_client": "app.clients.cloudflare_client.CloudflareClient",
+ "_panos_client": "app.clients.panos_client.PanOsClient",
+ "_fortigate_client": "app.clients.fortigate_client.FortiGateClient",
+}
+
+
+def _import(path: str):
+ module_path, _, name = path.rpartition(".")
+ module = __import__(module_path, fromlist=[name])
+ return getattr(module, name)
+
+
+def _request(capability: str, vendor: str, **params) -> LiveActionRequest:
+ return LiveActionRequest(vendor_id=vendor, capability=capability, target="TARGET", params=params)
+
+
+class TestEveryExecutorIsBacked:
+ def test_every_executor_is_covered_by_this_test(self) -> None:
+ """A new executor added without a row here is untested breadth,
+ which is how a registry fills with things nobody exercised."""
+ names = {cls.__name__ for cls in VENDOR_BREADTH_EXECUTORS}
+ assert names == set(EXPECTED_CALLS)
+
+ @pytest.mark.parametrize(("executor_name", "expected"), sorted(EXPECTED_CALLS.items()))
+ def test_the_client_method_exists(self, executor_name: str, expected: tuple[str, str]) -> None:
+ """The property that would have caught the whole class of defect.
+
+ Simulation mode never constructs the client, so an executor calling
+ a method that does not exist works in every test and raises the
+ first time a credential is present. Checked against the real class,
+ not a mock.
+ """
+ factory_name, method = expected
+ client_cls = _import(REAL_CLIENTS[factory_name])
+ assert hasattr(client_cls, method), f"{executor_name} calls {method}() which {client_cls.__name__} does " f"not have"
+ assert inspect.iscoroutinefunction(getattr(client_cls, method)), f"{client_cls.__name__}.{method} is not awaitable"
+
+ @pytest.mark.parametrize("executor_cls", VENDOR_BREADTH_EXECUTORS, ids=lambda c: c.__name__)
+ def test_every_capability_has_a_contract(self, executor_cls: type) -> None:
+ assert executor_cls.capability in CAPABILITY_CONTRACTS
+
+
+class TestSharedBehaviour:
+ """Three properties every executor inherits from the shared body. Tested
+ on representatives rather than all seventeen: the body is shared, so
+ seventeen copies of the same assertion tests the parametrize decorator."""
+
+ @pytest.mark.asyncio
+ async def test_dry_run_does_not_reach_the_vendor(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ client = AsyncMock()
+ monkeypatch.setattr(vendor_breadth, "_s1_client", lambda p: client)
+ request = _request("unisolate_host", "sentinelone")
+ request.dry_run = True
+
+ result = await SentinelOneUnisolateHost().execute(request)
+ assert result.status is LiveActionStatus.SIMULATED
+ client.lift_containment.assert_not_called()
+
+ @pytest.mark.asyncio
+ async def test_missing_credentials_says_so_rather_than_failing_vaguely(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ """ "We could not reach the vendor" and "the vendor refused" send an
+ operator to different places."""
+ monkeypatch.setattr(vendor_breadth, "_entra_client", lambda p: None)
+ result = await EntraEnableUser().execute(_request("enable_user", "azure_entra"))
+ assert result.status is LiveActionStatus.FAILED
+ assert "not a statement about the target" in (result.error or "")
+
+ @pytest.mark.asyncio
+ async def test_a_vendor_exception_becomes_failed_not_an_escape(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ """An exception escaping an executor wedges the agent loop."""
+ client = AsyncMock()
+ client.lift_containment = AsyncMock(side_effect=RuntimeError("503"))
+ monkeypatch.setattr(vendor_breadth, "_s1_client", lambda p: client)
+
+ result = await SentinelOneUnisolateHost().execute(_request("unisolate_host", "sentinelone"))
+ assert result.status is LiveActionStatus.FAILED
+ assert "503" in (result.error or "")
+
+ @pytest.mark.asyncio
+ async def test_success_carries_the_vendor_response(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ client = AsyncMock()
+ client.lift_containment = AsyncMock(return_value={"ok": True})
+ monkeypatch.setattr(vendor_breadth, "_s1_client", lambda p: client)
+
+ result = await SentinelOneUnisolateHost().execute(_request("unisolate_host", "sentinelone"))
+ assert result.status is LiveActionStatus.SUCCEEDED
+ assert result.details["vendor_response"] == {"ok": True}
+
+ @pytest.mark.asyncio
+ async def test_vendor_specific_params_are_forwarded(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ """Cloudflare's unblock needs a rule and zone id, not just the IP.
+ Dropping them would produce a call that succeeds against the wrong
+ zone or fails obscurely."""
+ client = AsyncMock()
+ client.unblock_ip_zone = AsyncMock(return_value={})
+ monkeypatch.setattr(vendor_breadth, "_cloudflare_client", lambda p: client)
+
+ await CloudflareAllowIP().execute(_request("allow_ip", "cloudflare", rule_id="r1", cf_zone_id="z1"))
+ client.unblock_ip_zone.assert_awaited_once_with("r1", "z1")
+
+
+class TestReversesAreNowReachable:
+ """Several of these are the reverse actions the contract already
+ promised and nothing implemented."""
+
+ @pytest.mark.parametrize(
+ ("forward", "reverse"),
+ [
+ ("disable_user", "enable_user"),
+ ("isolate_host", "unisolate_host"),
+ ("block_ip", "allow_ip"),
+ ("block_domain", "allow_domain"),
+ ],
+ )
+ def test_a_declared_reverse_has_an_implementation(self, forward: str, reverse: str) -> None:
+ from app.live_actions import builtins
+
+ implemented = {c.capability for c in builtins._BUILTIN_ADAPTERS} # noqa: SLF001
+ declared = CAPABILITY_CONTRACTS[forward].reverse_capability
+ assert declared == reverse
+ assert reverse in implemented, (
+ f"{forward} declares {reverse} as its reverse and nothing " f"implements it, so the rollback path resolves to nothing"
+ )
diff --git a/services/agents/tests/test_detection_fp_rate.py b/services/agents/tests/test_detection_fp_rate.py
index f8268f9b2..5ea4f4a5e 100644
--- a/services/agents/tests/test_detection_fp_rate.py
+++ b/services/agents/tests/test_detection_fp_rate.py
@@ -64,7 +64,7 @@ if str(_SCRIPTS_DIR) not in sys.path:
sys.path.insert(0, str(_SCRIPTS_DIR))
from detection_specs_index import all_specs # type: ignore[import-not-found] # noqa: E402
-from generate_detections import matches # type: ignore[import-not-found] # noqa: E402
+from generate_detections import enrich, matches, requested_derived_fields # type: ignore[import-not-found] # noqa: E402
# CI gate (lower-is-better). Tighten only with a corresponding allowlist
# entry or rule-narrowing PR. The 5% ceiling is a deliberate compromise:
@@ -179,6 +179,14 @@ def evaluate_per_rule_fp(
positive = spec["positive"]
negative = spec.get("negative")
+ # Enriched the way the engine enriches. This is the third replay
+ # path that read fixtures through the bare matcher; each one tests
+ # a pipeline production does not run, so a rule matching a derived
+ # field fails here while working live.
+ wanted = requested_derived_fields([{"match_when": match_when}])
+ positive = enrich(positive, wanted)
+ negative = enrich(negative, wanted) if negative else negative
+
own_pos = bool(matches(match_when, positive))
own_neg = bool(matches(match_when, negative)) if negative else False
@@ -190,7 +198,7 @@ def evaluate_per_rule_fp(
cross_total += 1
if (slug, other_slug) in allowlist:
continue
- if matches(match_when, other_positive):
+ if matches(match_when, enrich(other_positive, wanted)):
cross_fires.append((other_slug, "positive"))
per_rule.append(
diff --git a/services/api/app/api/v1/endpoints/health.py b/services/api/app/api/v1/endpoints/health.py
index 46f4f04a0..65bf946ed 100644
--- a/services/api/app/api/v1/endpoints/health.py
+++ b/services/api/app/api/v1/endpoints/health.py
@@ -86,6 +86,7 @@ from app.core.config import settings
from app.models.alert import Alert
from app.models.connector import Connector
from app.services.connector_freshness import compute_freshness
+from app.services.fleet_health import assess_fleet
logger = logging.getLogger(__name__)
@@ -509,3 +510,26 @@ async def get_pipeline_health(
stages=stages,
generated_at=now,
)
+
+
+async def get_fleet_health(
+ user: AuthUser,
+ db: DBSession,
+) -> dict[str, Any]:
+ """Which connectors have quietly stopped working.
+
+ Every field this reads was already being written — `last_sync`,
+ `error_count`, `oauth_refresh_failures`, `last_schema_drift_at` — and
+ nothing read them together. That produces the failure this platform is
+ least able to tolerate: a connector stops polling, alerts from that
+ source stop arriving, and the console looks calm, because an absence of
+ alerts is indistinguishable from an absence of threats.
+
+ Staleness is judged per connector against its own configured cadence.
+ A single global threshold would page constantly on daily connectors or
+ stay silent on five-minute ones, and a surface that pages constantly is
+ a surface that gets muted.
+ """
+ rows = (await db.execute(select(Connector).where(Connector.tenant_id == user.tenant_id))).scalars().all()
+ return assess_fleet(list(rows)).to_dict()
diff --git a/services/api/app/db/clickhouse.py b/services/api/app/db/clickhouse.py
index f3a944c05..0997d9314 100644
--- a/services/api/app/db/clickhouse.py
+++ b/services/api/app/db/clickhouse.py
@@ -431,3 +431,49 @@ async def fetch_lake_schema(
by_table[key].append({"name": name, "type": ctype, "comment": comment or ""})
return [{"table": fq, "columns": cols} for fq, cols in by_table.items()]
+
+
+async def init_lake_schema() -> None:
+ """Converge the lake schema at startup.
+
+ `clickhouse/001_init.sql` runs only in the container entrypoint, and
+ only on a volume that has never been initialised. `lake_writer`
+ self-heals with `CREATE TABLE IF NOT EXISTS`, which is a no-op once the
+ table exists. Neither adds a column to an existing deployment, so a
+ schema change lands on fresh installs and silently does not land
+ anywhere else — invisible until a query selects a column that is not
+ there, long after the deploy that was meant to add it.
+
+ Non-strict on purpose, matching the Neo4j equivalent: a migration
+ failure must not take the API down, because the lake is a query surface
+ and the rest of the platform works without it. It is logged at
+ ``error`` and the pending list stays readable for diagnostics, so a
+ degraded start is visible rather than assumed.
+ """
+ from app.db.lake_migrations import pending_ids, run_migrations # noqa: PLC0415
+
+ try:
+ client = await get_clickhouse_client()
+ except LakeQueryNotConfiguredError:
+ logger.info("clickhouse not configured — skipping lake migrations")
+ return
+ except Exception as exc: # noqa: BLE001 - an unreachable lake is not fatal
+ logger.warning("clickhouse unreachable at startup: %s", exc)
+ return
+
+ try:
+ applied = await run_migrations(client, strict=False)
+ pending = await pending_ids(client)
+ except Exception as exc: # noqa: BLE001 - same reasoning
+ logger.error("lake migrations could not run: %s", exc)
+ return
+
+ if applied:
+ logger.info("ClickHouse lake migrations applied: %s", ", ".join(applied))
+ if pending:
+ logger.error(
+ "ClickHouse lake migrations still pending after startup: %s — " "lake-backed features may query columns that do not exist",
+ ", ".join(pending),
+ )
+ else:
+ logger.info("ClickHouse lake schema up to date")
diff --git a/services/api/app/db/lake_migrations.py b/services/api/app/db/lake_migrations.py
new file mode 100644
index 000000000..85a35da72
--- /dev/null
+++ b/services/api/app/db/lake_migrations.py
@@ -0,0 +1,207 @@
+"""Forward-only schema migrations for the ClickHouse event lake.
+
+`services/api/clickhouse/001_init.sql` is four `CREATE TABLE IF NOT EXISTS`
+statements and zero `ALTER TABLE`, and `lake_writer` "self-heals" with the
+same idiom. Both are no-ops once the table exists, so a new column lands on
+a fresh deployment and silently does not land on an existing one — and the
+failure is invisible until a query selects a column that is not there, which
+happens long after the deploy that was supposed to add it.
+
+This is the ClickHouse half of the migration mechanism whose Neo4j half is
+`graph_migrations.py`. Same shape deliberately: numbered, append-only,
+applied ids recorded in the store itself so a deployment knows what it has.
+
+Three differences ClickHouse forces:
+
+**The ledger is a table, not a node**, and it is created with the same
+`IF NOT EXISTS` idiom this module exists to work around. That is fine for
+exactly one table whose schema never changes — and the schema is pinned by
+a test, so "never changes" is enforced rather than hoped for.
+
+**`ADD COLUMN IF NOT EXISTS` is genuinely idempotent** in ClickHouse, unlike
+Neo4j's constraint syntax, so a migration that half-applied can be re-run.
+Statements should still be written to tolerate it, because a mutation that
+fails midway leaves the earlier statements applied.
+
+**Distributed DDL is asynchronous.** On a cluster, `ALTER TABLE` returns
+before every replica has applied it. Single-node deployments are unaffected;
+a cluster needs `ON CLUSTER` and a sync setting, which is called out on the
+migration that would need it rather than assumed.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Awaitable, Callable
+from dataclasses import dataclass, field
+from datetime import UTC, datetime
+from typing import Any
+
+logger = logging.getLogger("aisoc.lake_migrations")
+
+MIGRATION_TABLE = "aisoc._migrations"
+
+#: Schema of the ledger itself. Pinned by a test: this table is created with
+#: the `IF NOT EXISTS` idiom the rest of this module exists to replace, which
+#: is only safe while its shape never changes.
+_LEDGER_DDL = f"""
+CREATE TABLE IF NOT EXISTS {MIGRATION_TABLE} (
+ migration_id String,
+ description String,
+ applied_at DateTime64(3, 'UTC') DEFAULT now64(),
+ statements UInt32
+) ENGINE = MergeTree()
+ORDER BY migration_id
+"""
+
+
+@dataclass(frozen=True)
+class LakeMigration:
+ """One forward-only change to the lake schema.
+
+ ``statements`` run in order. ``backfill`` runs after them for changes
+ that need to touch data — ClickHouse mutations are asynchronous, so a
+ backfill that must complete before the next migration has to wait on
+ ``system.mutations`` itself rather than assuming the ALTER returned.
+ """
+
+ id: str
+ description: str
+ statements: tuple[str, ...] = ()
+ backfill: Callable[[Any], Awaitable[int]] | None = None
+ #: True when the statements need `ON CLUSTER` on a multi-replica
+ #: deployment. Recorded rather than applied, because this repo has no
+ #: cluster topology to name and guessing one is worse than declaring the
+ #: limitation.
+ needs_cluster_ddl: bool = False
+ tags: tuple[str, ...] = field(default_factory=tuple)
+
+
+# ── 001: the pre-existing init schema, captured as a migration ─────────────
+#
+# Identical in effect to 001_init.sql, so an existing deployment records 001
+# as applied and converges rather than re-running statements it cannot know
+# the outcome of. Deliberately not a copy of the file: the file creates the
+# database and tables, and re-running it here would be the same no-op. What
+# matters is that the ledger agrees the baseline exists.
+_V1_BASELINE = ("CREATE DATABASE IF NOT EXISTS aisoc",)
+
+
+MIGRATIONS: tuple[LakeMigration, ...] = (
+ LakeMigration(
+ id="001_baseline",
+ description=("Records the 001_init.sql baseline as applied. Creates nothing an " "existing deployment does not already have."),
+ statements=_V1_BASELINE,
+ tags=("baseline",),
+ ),
+)
+
+
+async def _execute(client: Any, sql: str, params: dict[str, Any] | None = None) -> Any:
+ """Run one statement against whichever ClickHouse client shape we have.
+
+ The async driver exposes `execute`; the HTTP client exposes `command`.
+ Both are in use across this repo depending on the service, and a
+ migration runner that only speaks one of them is a runner that silently
+ does not run in half the deployments.
+ """
+ if hasattr(client, "execute"):
+ return await client.execute(sql, params or {})
+ if hasattr(client, "command"):
+ return await client.command(sql)
+ raise TypeError(f"{type(client).__name__} exposes neither execute() nor command(); " f"the lake migration runner cannot drive it")
+
+
+async def applied_ids(client: Any) -> set[str]:
+ """Migration ids this lake already has.
+
+ An unreadable ledger returns the empty set rather than raising, so a
+ first run on a lake with no ledger works. It is the caller's `strict`
+ flag that decides whether a *failed* migration is fatal.
+ """
+ await _execute(client, _LEDGER_DDL)
+ try:
+ rows = await _execute(client, f"SELECT migration_id FROM {MIGRATION_TABLE}")
+ except Exception as exc: # noqa: BLE001 - an empty ledger is a normal first run
+ logger.warning("lake_migrations.ledger_unreadable error=%s", exc)
+ return set()
+
+ ids: set[str] = set()
+ for row in rows or []:
+ # Drivers return tuples, dicts or scalars depending on which one.
+ if isinstance(row, dict):
+ value = row.get("migration_id")
+ elif isinstance(row, list | tuple):
+ value = row[0] if row else None
+ else:
+ value = row
+ if value:
+ ids.add(str(value))
+ return ids
+
+
+async def _record(client: Any, migration: LakeMigration, statements: int) -> None:
+ await _execute(
+ client,
+ f"INSERT INTO {MIGRATION_TABLE} " "(migration_id, description, applied_at, statements) VALUES",
+ {
+ "migration_id": migration.id,
+ "description": migration.description,
+ "applied_at": datetime.now(UTC),
+ "statements": statements,
+ },
+ )
+
+
+async def run_migrations(client: Any, *, strict: bool = True) -> list[str]:
+ """Apply every unapplied migration in order. Returns the ids applied.
+
+ Stops at the first failure rather than continuing, because migration
+ N+1 is written against the schema N produced. Continuing past a failure
+ is how a lake ends up in a state no migration file describes.
+
+ ``strict=False`` logs and returns what succeeded, for deployments that
+ would rather start degraded than not at all. It is not the default: a
+ silently-skipped schema change is the exact failure this module exists
+ to remove.
+ """
+ already = await applied_ids(client)
+ applied: list[str] = []
+
+ for migration in MIGRATIONS:
+ if migration.id in already:
+ continue
+
+ if migration.needs_cluster_ddl:
+ logger.warning(
+ "lake_migrations.cluster_ddl_unhandled id=%s — this migration "
+ "needs ON CLUSTER on a multi-replica deployment and will only "
+ "apply to the node it is run against",
+ migration.id,
+ )
+
+ logger.info("lake_migrations.applying id=%s", migration.id)
+ try:
+ for statement in migration.statements:
+ await _execute(client, statement)
+ if migration.backfill is not None:
+ touched = await migration.backfill(client)
+ logger.info("lake_migrations.backfill id=%s rows=%s", migration.id, touched)
+ await _record(client, migration, len(migration.statements))
+ except Exception as exc: # noqa: BLE001 - reported, then re-raised under strict
+ logger.error("lake_migrations.failed id=%s error=%s", migration.id, exc)
+ if strict:
+ raise
+ return applied
+
+ applied.append(migration.id)
+
+ if applied:
+ logger.info("lake_migrations.complete applied=%s", ",".join(applied))
+ return applied
+
+
+async def pending_ids(client: Any) -> list[str]:
+ """Migrations this lake has not applied. For health checks and preflight."""
+ already = await applied_ids(client)
+ return [m.id for m in MIGRATIONS if m.id not in already]
diff --git a/services/api/app/db/vector_migrations.py b/services/api/app/db/vector_migrations.py
new file mode 100644
index 000000000..70b4660c4
--- /dev/null
+++ b/services/api/app/db/vector_migrations.py
@@ -0,0 +1,212 @@
+"""Forward-only schema migrations for the Qdrant vector store.
+
+The third of the three stores with no migration mechanism. Qdrant is the
+awkward one, because the change most likely to be needed is the one it
+cannot do.
+
+**A collection's vector dimension and distance metric are immutable.**
+Changing an embedding model changes the dimension, and there is no
+`ALTER COLLECTION`. The migration is: create a new collection, re-embed
+every point into it, swap an alias, drop the old one. That is a data
+migration measured in hours for a large tenant, not a DDL statement — so
+this module models it as one rather than pretending a dimension change is a
+schema tweak.
+
+**Aliases are the indirection that makes it survivable.** Code reads
+through an alias, never a collection name directly. A re-embed builds
+alongside the live collection and the alias flips atomically at the end, so
+a failed migration leaves the old collection serving rather than a
+half-populated new one.
+
+What *is* cheap: payload indexes, quantization settings, HNSW parameters,
+optimizer thresholds. Those are the ordinary migrations, and they are why
+this exists even though re-embeds are rare.
+
+Same shape as `graph_migrations.py` and `lake_migrations.py`: numbered,
+append-only, applied ids recorded in the store itself. Qdrant has no table
+to record them in, so the ledger is a dedicated collection holding one
+point per applied migration — small, and it keeps the record in the store
+being migrated rather than somewhere that can drift from it.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Awaitable, Callable
+from dataclasses import dataclass, field
+from datetime import UTC, datetime
+from typing import Any
+
+logger = logging.getLogger("aisoc.vector_migrations")
+
+#: Ledger collection. One point per applied migration, keyed by a stable
+#: hash of the id so re-recording is idempotent.
+MIGRATION_COLLECTION = "aisoc_migrations"
+
+#: Minimal vector config for the ledger. Qdrant requires *some* vector
+#: definition; one dimension is the smallest legal answer and the vectors
+#: are never searched.
+_LEDGER_DIM = 1
+
+
+@dataclass(frozen=True)
+class VectorMigration:
+ """One forward-only change to the vector store.
+
+ ``apply`` receives the client and does the work, rather than a list of
+ statements: Qdrant's API is method calls, not a query language, and
+ pretending otherwise would mean inventing a DSL whose only user is this
+ file.
+
+ ``rebuilds_collection`` marks the expensive class — a dimension or
+ metric change that requires re-embedding every point. Flagged so an
+ operator can see one coming in ``pending_ids()`` before it starts, since
+ the difference between a one-second migration and a four-hour one should
+ not be a surprise discovered at deploy time.
+ """
+
+ id: str
+ description: str
+ apply: Callable[[Any], Awaitable[None]]
+ rebuilds_collection: bool = False
+ tags: tuple[str, ...] = field(default_factory=tuple)
+
+
+def _point_id(migration_id: str) -> int:
+ """A stable positive integer id for a migration.
+
+ Qdrant point ids are unsigned integers or UUIDs. Hashed with blake2b
+ rather than `hash()` for the same reason the sandbox was fixed: CPython
+ salts string hashing per process, so a ledger keyed on `hash()` would
+ record the same migration under a different id on every restart and the
+ runner would re-apply everything.
+ """
+ import hashlib
+
+ digest = hashlib.blake2b(migration_id.encode(), digest_size=8).digest()
+ return int.from_bytes(digest, "big") & 0x7FFF_FFFF_FFFF_FFFF
+
+
+async def _ensure_ledger(client: Any) -> None:
+ try:
+ collections = await client.get_collections()
+ names = {c.name for c in getattr(collections, "collections", [])}
+ except Exception as exc: # noqa: BLE001 - treated as absent, created below
+ logger.warning("vector_migrations.list_failed error=%s", exc)
+ names = set()
+
+ if MIGRATION_COLLECTION in names:
+ return
+
+ from qdrant_client.models import Distance, VectorParams # noqa: PLC0415
+
+ await client.create_collection(
+ collection_name=MIGRATION_COLLECTION,
+ vectors_config=VectorParams(size=_LEDGER_DIM, distance=Distance.COSINE),
+ )
+ logger.info("vector_migrations.ledger_created")
+
+
+async def applied_ids(client: Any) -> set[str]:
+ """Migration ids this store already has."""
+ await _ensure_ledger(client)
+ try:
+ points, _ = await client.scroll(collection_name=MIGRATION_COLLECTION, limit=1000, with_payload=True)
+ except Exception as exc: # noqa: BLE001 - an empty ledger is a normal first run
+ logger.warning("vector_migrations.ledger_unreadable error=%s", exc)
+ return set()
+
+ return {str(p.payload["migration_id"]) for p in points or [] if getattr(p, "payload", None) and p.payload.get("migration_id")}
+
+
+async def _record(client: Any, migration: VectorMigration) -> None:
+ from qdrant_client.models import PointStruct # noqa: PLC0415
+
+ await client.upsert(
+ collection_name=MIGRATION_COLLECTION,
+ points=[
+ PointStruct(
+ id=_point_id(migration.id),
+ vector=[0.0] * _LEDGER_DIM,
+ payload={
+ "migration_id": migration.id,
+ "description": migration.description,
+ "applied_at": datetime.now(UTC).isoformat(),
+ },
+ )
+ ],
+ )
+
+
+async def _apply_baseline(client: Any) -> None:
+ """Records the pre-existing collections as the baseline.
+
+ Creates nothing. `QdrantStore` builds its collections on first use, so
+ an existing deployment already has them; the point is that the ledger
+ agrees, and migration 002 onwards can assume a known starting shape.
+ """
+ return None
+
+
+MIGRATIONS: tuple[VectorMigration, ...] = (
+ VectorMigration(
+ id="001_baseline",
+ description=("Records the collections QdrantStore creates on first use as the " "baseline. Creates nothing."),
+ apply=_apply_baseline,
+ tags=("baseline",),
+ ),
+)
+
+
+async def run_migrations(client: Any, *, strict: bool = True) -> list[str]:
+ """Apply every unapplied migration in order. Returns the ids applied.
+
+ Stops at the first failure: migration N+1 is written against the state N
+ produced, and continuing past a failure leaves the store in a shape no
+ migration describes.
+ """
+ already = await applied_ids(client)
+ applied: list[str] = []
+
+ for migration in MIGRATIONS:
+ if migration.id in already:
+ continue
+
+ if migration.rebuilds_collection:
+ logger.warning(
+ "vector_migrations.rebuild_starting id=%s — this re-embeds "
+ "every point and is measured in hours for a large tenant. The "
+ "alias keeps the existing collection serving until it "
+ "completes.",
+ migration.id,
+ )
+
+ logger.info("vector_migrations.applying id=%s", migration.id)
+ try:
+ await migration.apply(client)
+ await _record(client, migration)
+ except Exception as exc: # noqa: BLE001 - reported, re-raised under strict
+ logger.error("vector_migrations.failed id=%s error=%s", migration.id, exc)
+ if strict:
+ raise
+ return applied
+
+ applied.append(migration.id)
+
+ if applied:
+ logger.info("vector_migrations.complete applied=%s", ",".join(applied))
+ return applied
+
+
+async def pending_ids(client: Any) -> list[str]:
+ """Migrations this store has not applied. For health checks and preflight."""
+ already = await applied_ids(client)
+ return [m.id for m in MIGRATIONS if m.id not in already]
+
+
+async def pending_rebuilds(client: Any) -> list[str]:
+ """Pending migrations that re-embed. Surfaced separately because the
+ difference between a one-second migration and a four-hour one should not
+ be discovered at deploy time."""
+ already = await applied_ids(client)
+ return [m.id for m in MIGRATIONS if m.id not in already and m.rebuilds_collection]
diff --git a/services/api/app/main.py b/services/api/app/main.py
index 6d479473d..611ba18fb 100644
--- a/services/api/app/main.py
+++ b/services/api/app/main.py
@@ -23,7 +23,7 @@ from app.core.cors import build_cors_kwargs
from app.core.logging import configure_logging
from app.core.scheduler_lock import scheduler_lock
from app.core.telemetry import instrument_app
-from app.db.clickhouse import close_clickhouse
+from app.db.clickhouse import close_clickhouse, init_lake_schema
from app.db.database import engine
from app.db.neo4j import close_neo4j, init_neo4j
from app.graphql.schema import graphql_router
@@ -340,6 +340,14 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
except Exception as exc:
logger.warning("Neo4j unavailable at startup – graph features disabled", error=str(exc))
+ # The lake's schema converges here for the same reason the graph's
+ # does: CREATE TABLE IF NOT EXISTS is a no-op on an existing
+ # deployment, so without this a new column never lands on one.
+ try:
+ await init_lake_schema()
+ except Exception as exc: # noqa: BLE001 - the lake is not on the critical path
+ logger.warning("lake schema migration skipped", error=str(exc))
+
# Auto-discover plugins from AISOC_PLUGINS_DIR
try:
plugin_mgr = get_plugin_manager()
diff --git a/services/api/app/services/fleet_health.py b/services/api/app/services/fleet_health.py
new file mode 100644
index 000000000..3ce5c9ca4
--- /dev/null
+++ b/services/api/app/services/fleet_health.py
@@ -0,0 +1,258 @@
+"""Which parts of this deployment have quietly stopped working.
+
+Every field this needs was already being written — `last_sync`,
+`error_count`, `events_ingested`, `oauth_refresh_failures`,
+`last_schema_drift_at` — and nothing read them together. The result is the
+failure mode this platform is least able to tolerate: a connector stops
+polling, alerts stop arriving from that source, and the console looks calm
+because an absence of alerts is indistinguishable from an absence of
+threats.
+
+The design decision that shapes everything here is **what counts as
+stale**. A connector polling every five minutes is broken after twenty; one
+polling daily is fine after twenty hours. A single global threshold would
+either page constantly on slow connectors or stay silent on fast ones, so
+staleness is measured in *missed poll intervals* against each connector's
+own configured cadence.
+
+Two things deliberately not inferred:
+
+**A connector that has never synced is not stale.** It is unproven. Those
+are different operator actions — one is "something broke", the other is
+"this was never finished" — and collapsing them sends people to debug a
+working connector that nobody completed setup for.
+
+**A disabled connector is not unhealthy.** Reporting it as degraded trains
+people to ignore the degraded count, which is how a real failure gets
+missed.
+"""
+
+from __future__ import annotations
+
+from dataclasses import asdict, dataclass, field
+from datetime import UTC, datetime, timedelta
+from enum import Enum
+from typing import Any
+
+#: Default cadence when a connector declares none. Matches the scheduler's
+#: own default, so staleness is judged against what actually runs.
+DEFAULT_POLL_SECONDS = 300
+
+#: Missed intervals before a connector is called stale. Three rather than
+#: one: a single missed poll is routine (a vendor rate-limit, a redeploy),
+#: and alerting on it produces noise that gets the whole surface muted.
+STALE_INTERVALS = 3
+
+#: Missed intervals before it is called failed rather than degraded.
+FAILED_INTERVALS = 12
+
+#: Consecutive errors before the error count alone is enough, regardless of
+#: recency. A connector erroring every poll but still "recent" is broken.
+ERROR_THRESHOLD = 5
+
+
+class HealthState(str, Enum):
+ HEALTHY = "healthy"
+ DEGRADED = "degraded"
+ FAILED = "failed"
+ #: Enabled but has never completed a poll. Not a failure — setup was
+ #: never finished, which is a different action for the operator.
+ UNPROVEN = "unproven"
+ #: Turned off on purpose. Never counted against fleet health.
+ DISABLED = "disabled"
+
+
+@dataclass
+class ConnectorHealth:
+ connector_id: str
+ name: str
+ connector_type: str
+ state: HealthState
+ #: Why, in words an operator can act on. Never "unhealthy".
+ reason: str
+ last_sync: datetime | None
+ seconds_since_sync: int | None
+ poll_interval_seconds: int
+ missed_intervals: float | None
+ error_count: int
+ events_ingested: int
+ oauth_refresh_failures: int = 0
+ schema_drift_at: datetime | None = None
+
+ def to_dict(self) -> dict[str, Any]:
+ out = asdict(self)
+ out["state"] = self.state.value
+ for key in ("last_sync", "schema_drift_at"):
+ value = out.get(key)
+ out[key] = value.isoformat() if isinstance(value, datetime) else None
+ return out
+
+
+@dataclass
+class FleetHealth:
+ generated_at: datetime
+ connectors: list[ConnectorHealth] = field(default_factory=list)
+
+ @property
+ def counts(self) -> dict[str, int]:
+ out = {state.value: 0 for state in HealthState}
+ for connector in self.connectors:
+ out[connector.state.value] += 1
+ return out
+
+ @property
+ def worst_state(self) -> HealthState:
+ """The fleet is as healthy as its least healthy enabled connector.
+
+ Averaging would let nineteen working connectors hide one that
+ stopped — and the one that stopped is the whole question.
+ """
+ for state in (HealthState.FAILED, HealthState.DEGRADED, HealthState.UNPROVEN):
+ if any(c.state is state for c in self.connectors):
+ return state
+ return HealthState.HEALTHY
+
+ def to_dict(self) -> dict[str, Any]:
+ return {
+ "generated_at": self.generated_at.isoformat(),
+ "state": self.worst_state.value,
+ "counts": self.counts,
+ "connectors": [c.to_dict() for c in self.connectors],
+ }
+
+
+def _poll_interval(row: Any) -> int:
+ config = getattr(row, "connector_config", None) or {}
+ raw = config.get("poll_interval_seconds") if isinstance(config, dict) else None
+ try:
+ interval = int(raw)
+ except (TypeError, ValueError):
+ return DEFAULT_POLL_SECONDS
+ # A zero or negative cadence would make every connector infinitely
+ # stale; treat it as unset rather than as a division by zero.
+ return interval if interval > 0 else DEFAULT_POLL_SECONDS
+
+
+def assess_connector(row: Any, *, now: datetime | None = None) -> ConnectorHealth:
+ """Judge one connector against its own configured cadence."""
+ now = now or datetime.now(UTC)
+ interval = _poll_interval(row)
+ last_sync = getattr(row, "last_sync", None)
+ error_count = int(getattr(row, "error_count", 0) or 0)
+ oauth_failures = int(getattr(row, "oauth_refresh_failures", 0) or 0)
+
+ base = {
+ "connector_id": str(getattr(row, "id", "")),
+ "name": getattr(row, "name", "") or "",
+ "connector_type": getattr(row, "connector_type", "") or "",
+ "last_sync": last_sync,
+ "poll_interval_seconds": interval,
+ "error_count": error_count,
+ "events_ingested": int(getattr(row, "events_ingested", 0) or 0),
+ "oauth_refresh_failures": oauth_failures,
+ "schema_drift_at": getattr(row, "last_schema_drift_at", None),
+ }
+
+ if not getattr(row, "is_enabled", True):
+ return ConnectorHealth(
+ **base,
+ state=HealthState.DISABLED,
+ reason="Disabled. Not counted against fleet health.",
+ seconds_since_sync=None,
+ missed_intervals=None,
+ )
+
+ if last_sync is None:
+ return ConnectorHealth(
+ **base,
+ state=HealthState.UNPROVEN,
+ reason=(
+ "Enabled but has never completed a poll. Usually a credential "
+ "the vendor accepted at save time but rejects for reads — check "
+ "the connectors service log for the vendor's own error."
+ ),
+ seconds_since_sync=None,
+ missed_intervals=None,
+ )
+
+ if last_sync.tzinfo is None:
+ last_sync = last_sync.replace(tzinfo=UTC)
+ elapsed = max(0, int((now - last_sync).total_seconds()))
+ missed = round(elapsed / interval, 1)
+
+ # An OAuth refresh failure is terminal in a way a poll failure is not:
+ # the credential is gone and no amount of waiting fixes it.
+ if oauth_failures >= 3:
+ return ConnectorHealth(
+ **base,
+ state=HealthState.FAILED,
+ reason=(f"OAuth refresh has failed {oauth_failures} times. The grant was " f"most likely revoked; reconnect rather than wait."),
+ seconds_since_sync=elapsed,
+ missed_intervals=missed,
+ )
+
+ if missed >= FAILED_INTERVALS:
+ return ConnectorHealth(
+ **base,
+ state=HealthState.FAILED,
+ reason=(
+ f"No successful poll in {_humanise(elapsed)} "
+ f"({missed:g} missed intervals). Alerts from this source have "
+ f"stopped arriving."
+ ),
+ seconds_since_sync=elapsed,
+ missed_intervals=missed,
+ )
+
+ if error_count >= ERROR_THRESHOLD:
+ return ConnectorHealth(
+ **base,
+ state=HealthState.DEGRADED,
+ reason=(f"{error_count} consecutive poll errors. Syncing recently but " f"not cleanly."),
+ seconds_since_sync=elapsed,
+ missed_intervals=missed,
+ )
+
+ if missed >= STALE_INTERVALS:
+ return ConnectorHealth(
+ **base,
+ state=HealthState.DEGRADED,
+ reason=(f"Last poll {_humanise(elapsed)} ago against a " f"{_humanise(interval)} cadence ({missed:g} missed intervals)."),
+ seconds_since_sync=elapsed,
+ missed_intervals=missed,
+ )
+
+ return ConnectorHealth(
+ **base,
+ state=HealthState.HEALTHY,
+ reason=f"Last poll {_humanise(elapsed)} ago.",
+ seconds_since_sync=elapsed,
+ missed_intervals=missed,
+ )
+
+
+def _humanise(seconds: int) -> str:
+ delta = timedelta(seconds=seconds)
+ if delta.days:
+ return f"{delta.days}d"
+ hours, rest = divmod(seconds, 3600)
+ if hours:
+ return f"{hours}h"
+ minutes = rest // 60
+ return f"{minutes}m" if minutes else f"{seconds}s"
+
+
+def assess_fleet(rows: list[Any], *, now: datetime | None = None) -> FleetHealth:
+ now = now or datetime.now(UTC)
+ assessed = [assess_connector(row, now=now) for row in rows]
+ # Worst first: an operator opening this wants the broken one, not an
+ # alphabetical list they have to scan.
+ order = {
+ HealthState.FAILED: 0,
+ HealthState.DEGRADED: 1,
+ HealthState.UNPROVEN: 2,
+ HealthState.HEALTHY: 3,
+ HealthState.DISABLED: 4,
+ }
+ assessed.sort(key=lambda c: (order[c.state], -(c.seconds_since_sync or 0)))
+ return FleetHealth(generated_at=now, connectors=assessed)
diff --git a/services/api/tests/test_fleet_health.py b/services/api/tests/test_fleet_health.py
new file mode 100644
index 000000000..293d3a70c
--- /dev/null
+++ b/services/api/tests/test_fleet_health.py
@@ -0,0 +1,198 @@
+"""Fleet health: the surface that answers "what quietly stopped working".
+
+Every field this reads was already being written and nothing read them
+together, which produces the failure this platform is least able to
+tolerate — a connector stops polling, alerts from that source stop
+arriving, and the console looks calm because an absence of alerts is
+indistinguishable from an absence of threats.
+
+The tests concentrate on the two judgements that make this useful rather
+than noisy: staleness measured against each connector's own cadence, and
+the states that must *not* be reported as failures.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from datetime import UTC, datetime, timedelta
+
+import pytest
+from app.services.fleet_health import (
+ ERROR_THRESHOLD,
+ HealthState,
+ assess_connector,
+ assess_fleet,
+)
+
+NOW = datetime(2026, 9, 22, 12, 0, tzinfo=UTC)
+
+
+@dataclass
+class Row:
+ """Stands in for the Connector ORM row."""
+
+ id: str = "c1"
+ name: str = "Prod Okta"
+ connector_type: str = "okta"
+ is_enabled: bool = True
+ last_sync: datetime | None = None
+ error_count: int = 0
+ events_ingested: int = 100
+ oauth_refresh_failures: int = 0
+ last_schema_drift_at: datetime | None = None
+ connector_config: dict = field(default_factory=dict)
+
+
+def ago(**kwargs) -> datetime:
+ return NOW - timedelta(**kwargs)
+
+
+class TestStalenessIsRelativeToCadence:
+ """A global threshold either pages constantly on slow connectors or
+ stays silent on fast ones."""
+
+ def test_a_fast_connector_is_stale_after_twenty_minutes(self) -> None:
+ row = Row(last_sync=ago(minutes=20), connector_config={"poll_interval_seconds": 300})
+ assert assess_connector(row, now=NOW).state is HealthState.DEGRADED
+
+ def test_a_daily_connector_is_healthy_after_twenty_minutes(self) -> None:
+ row = Row(last_sync=ago(minutes=20), connector_config={"poll_interval_seconds": 86400})
+ assert assess_connector(row, now=NOW).state is HealthState.HEALTHY
+
+ def test_a_daily_connector_is_stale_after_four_days(self) -> None:
+ row = Row(last_sync=ago(days=4), connector_config={"poll_interval_seconds": 86400})
+ assert assess_connector(row, now=NOW).state is HealthState.DEGRADED
+
+ def test_one_missed_poll_is_not_a_finding(self) -> None:
+ """Alerting on a single missed poll produces noise that gets the
+ whole surface muted."""
+ row = Row(last_sync=ago(minutes=6), connector_config={"poll_interval_seconds": 300})
+ assert assess_connector(row, now=NOW).state is HealthState.HEALTHY
+
+ def test_long_enough_becomes_failed_not_degraded(self) -> None:
+ row = Row(last_sync=ago(hours=2), connector_config={"poll_interval_seconds": 300})
+ result = assess_connector(row, now=NOW)
+ assert result.state is HealthState.FAILED
+ assert "stopped arriving" in result.reason
+
+ @pytest.mark.parametrize("interval", [0, -1, None, "", "abc"])
+ def test_a_nonsense_cadence_falls_back_rather_than_dividing_by_zero(self, interval: object) -> None:
+ row = Row(
+ last_sync=ago(minutes=1),
+ connector_config={"poll_interval_seconds": interval},
+ )
+ assert assess_connector(row, now=NOW).state is HealthState.HEALTHY
+
+
+class TestStatesThatAreNotFailures:
+ def test_never_synced_is_unproven_not_failed(self) -> None:
+ """ "Something broke" and "this was never finished" are different
+ operator actions."""
+ result = assess_connector(Row(last_sync=None), now=NOW)
+ assert result.state is HealthState.UNPROVEN
+ assert "never completed a poll" in result.reason
+
+ def test_disabled_is_not_unhealthy(self) -> None:
+ """Reporting it as degraded trains people to ignore the degraded
+ count, which is how a real failure gets missed."""
+ row = Row(is_enabled=False, last_sync=ago(days=90))
+ assert assess_connector(row, now=NOW).state is HealthState.DISABLED
+
+ def test_a_disabled_connector_does_not_drag_down_the_fleet(self) -> None:
+ fleet = assess_fleet(
+ [Row(id="a", last_sync=ago(minutes=1)), Row(id="b", is_enabled=False)],
+ now=NOW,
+ )
+ assert fleet.worst_state is HealthState.HEALTHY
+
+
+class TestErrorsAndCredentials:
+ def test_errors_matter_even_when_syncing_recently(self) -> None:
+ """A connector erroring every poll but still "recent" is broken."""
+ row = Row(last_sync=ago(minutes=1), error_count=ERROR_THRESHOLD)
+ assert assess_connector(row, now=NOW).state is HealthState.DEGRADED
+
+ def test_repeated_oauth_failure_is_terminal(self) -> None:
+ """The credential is gone; no amount of waiting fixes it, so this
+ outranks recency."""
+ row = Row(last_sync=ago(minutes=1), oauth_refresh_failures=3)
+ result = assess_connector(row, now=NOW)
+ assert result.state is HealthState.FAILED
+ assert "reconnect rather than wait" in result.reason
+
+ def test_one_oauth_failure_is_not_terminal(self) -> None:
+ row = Row(last_sync=ago(minutes=1), oauth_refresh_failures=1)
+ assert assess_connector(row, now=NOW).state is HealthState.HEALTHY
+
+
+class TestFleetRollup:
+ def test_one_broken_connector_is_not_averaged_away(self) -> None:
+ """Nineteen working connectors must not hide the one that stopped —
+ the one that stopped is the whole question."""
+ rows = [Row(id=f"ok{i}", last_sync=ago(minutes=1)) for i in range(19)]
+ rows.append(Row(id="broken", last_sync=ago(days=3)))
+ assert assess_fleet(rows, now=NOW).worst_state is HealthState.FAILED
+
+ def test_worst_first_ordering(self) -> None:
+ """An operator opening this wants the broken one, not an
+ alphabetical list to scan."""
+ rows = [
+ Row(id="healthy", last_sync=ago(minutes=1)),
+ Row(id="failed", last_sync=ago(days=3)),
+ Row(id="unproven", last_sync=None),
+ Row(id="degraded", last_sync=ago(minutes=20)),
+ ]
+ states = [c.state for c in assess_fleet(rows, now=NOW).connectors]
+ assert states == [
+ HealthState.FAILED,
+ HealthState.DEGRADED,
+ HealthState.UNPROVEN,
+ HealthState.HEALTHY,
+ ]
+
+ def test_counts_cover_every_state(self) -> None:
+ fleet = assess_fleet([Row(last_sync=ago(minutes=1))], now=NOW)
+ assert set(fleet.counts) == {s.value for s in HealthState}
+ assert fleet.counts["healthy"] == 1
+
+ def test_an_empty_fleet_is_healthy_not_broken(self) -> None:
+ fleet = assess_fleet([], now=NOW)
+ assert fleet.worst_state is HealthState.HEALTHY
+
+ def test_serialises(self) -> None:
+ import json
+
+ fleet = assess_fleet([Row(last_sync=ago(minutes=1))], now=NOW)
+ payload = json.loads(json.dumps(fleet.to_dict()))
+ assert payload["state"] == "healthy"
+ assert payload["connectors"][0]["last_sync"].startswith("2026-")
+
+ def test_a_naive_timestamp_does_not_crash(self) -> None:
+ """Postgres can hand back a naive datetime depending on the column
+ and driver; subtracting it from an aware one raises."""
+ row = Row(last_sync=datetime(2026, 9, 22, 11, 59))
+ assert assess_connector(row, now=NOW).state is HealthState.HEALTHY
+
+
+class TestReasons:
+ def test_every_state_explains_itself_actionably(self) -> None:
+ """ "Unhealthy" tells an operator nothing they can act on."""
+ rows = [
+ Row(id="a", last_sync=ago(minutes=1)),
+ Row(id="b", last_sync=ago(minutes=20)),
+ Row(id="c", last_sync=ago(days=3)),
+ Row(id="d", last_sync=None),
+ Row(id="e", is_enabled=False),
+ ]
+ for connector in assess_fleet(rows, now=NOW).connectors:
+ assert connector.reason.strip(), f"{connector.connector_id} has no reason"
+ # "Unhealthy" is a restatement of the state, not a reason.
+ assert "unhealthy" not in connector.reason.lower()
+
+ if connector.state in (HealthState.HEALTHY, HealthState.DISABLED):
+ continue
+ # A non-healthy state has to say what is wrong in terms an
+ # operator can act on — a duration, a count, or a next step.
+ assert any(
+ token in connector.reason.lower() for token in ("poll", "credential", "oauth", "error", "reconnect")
+ ), f"{connector.connector_id}: {connector.reason!r} is not actionable"
diff --git a/services/api/tests/test_store_migrations.py b/services/api/tests/test_store_migrations.py
new file mode 100644
index 000000000..9b93284b3
--- /dev/null
+++ b/services/api/tests/test_store_migrations.py
@@ -0,0 +1,177 @@
+"""Every store that holds schema must have a way to change it.
+
+`clickhouse/001_init.sql` is four `CREATE TABLE IF NOT EXISTS` statements
+and zero `ALTER TABLE`; `lake_writer` self-heals with the same idiom. Both
+are no-ops once the table exists, so a new column lands on a fresh
+deployment and silently does not land on an existing one — and nothing
+notices until a query selects a column that is not there, long after the
+deploy that was meant to add it.
+
+The tests are mostly about the properties that make a runner trustworthy
+rather than about any individual migration: ids never move, a failure stops
+the run, and the ledger is read from the store being migrated rather than
+somewhere that can drift from it.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+from unittest.mock import AsyncMock
+
+import pytest
+from app.db import graph_migrations, lake_migrations, vector_migrations
+
+ALL_RUNNERS = (
+ ("neo4j", graph_migrations),
+ ("clickhouse", lake_migrations),
+ ("qdrant", vector_migrations),
+)
+
+
+class TestEveryStoreHasARunner:
+ """The plan called the absence of these a blocker, not a follow-up."""
+
+ @pytest.mark.parametrize(("store", "module"), ALL_RUNNERS, ids=lambda x: str(x))
+ def test_runner_exposes_the_same_interface(self, store: str, module: Any) -> None:
+ """Three stores, one shape. An operator should not have to learn
+ three vocabularies to answer "what has this deployment applied"."""
+ for name in ("MIGRATIONS", "run_migrations", "applied_ids", "pending_ids"):
+ assert hasattr(module, name), f"{store} runner has no {name}"
+
+ @pytest.mark.parametrize(("store", "module"), ALL_RUNNERS, ids=lambda x: str(x))
+ def test_ids_are_unique_and_ordered(self, store: str, module: Any) -> None:
+ ids = [m.id for m in module.MIGRATIONS]
+ assert len(ids) == len(set(ids)), f"{store} has duplicate migration ids"
+ assert ids == sorted(ids), (
+ f"{store} migrations are not in id order; the runner applies them "
+ f"in tuple order, so a mismatch means they run in an order the "
+ f"numbering does not describe"
+ )
+
+ @pytest.mark.parametrize(("store", "module"), ALL_RUNNERS, ids=lambda x: str(x))
+ def test_every_migration_is_described(self, store: str, module: Any) -> None:
+ """A migration whose purpose nobody wrote down cannot be reviewed,
+ and cannot be judged safe to re-run."""
+ for migration in module.MIGRATIONS:
+ assert migration.description.strip(), f"{store}/{migration.id}"
+
+ @pytest.mark.parametrize(("store", "module"), ALL_RUNNERS, ids=lambda x: str(x))
+ def test_the_first_migration_is_a_baseline(self, store: str, module: Any) -> None:
+ """An existing deployment must converge rather than re-run unknown
+ statements against a schema that already has them."""
+ first = module.MIGRATIONS[0]
+ assert "baseline" in first.id or "core" in first.id, (
+ f"{store}'s first migration is {first.id!r}; it should record the " f"pre-existing schema so existing deployments converge"
+ )
+
+
+class TestLakeRunner:
+ @pytest.fixture
+ def client(self) -> AsyncMock:
+ c = AsyncMock()
+ c.execute = AsyncMock(return_value=[])
+ return c
+
+ async def test_applies_and_records(self, client: AsyncMock) -> None:
+ applied = await lake_migrations.run_migrations(client)
+ assert applied == ["001_baseline"]
+ statements = [call.args[0] for call in client.execute.await_args_list]
+ assert any("_migrations" in s for s in statements), "ledger was not created"
+ assert any("INSERT INTO" in s for s in statements), "nothing was recorded"
+
+ async def test_an_applied_migration_is_not_reapplied(self, client: AsyncMock) -> None:
+ client.execute = AsyncMock(return_value=[("001_baseline",)])
+ assert await lake_migrations.run_migrations(client) == []
+
+ async def test_a_failure_stops_the_run(self, client: AsyncMock) -> None:
+ """Migration N+1 is written against the schema N produced."""
+ calls = {"n": 0}
+
+ async def failing(sql: str, params: Any = None) -> Any:
+ calls["n"] += 1
+ if "CREATE DATABASE" in sql:
+ raise RuntimeError("no such cluster")
+ return []
+
+ client.execute = failing
+ with pytest.raises(RuntimeError, match="no such cluster"):
+ await lake_migrations.run_migrations(client)
+
+ async def test_non_strict_returns_what_succeeded(self, client: AsyncMock) -> None:
+ async def failing(sql: str, params: Any = None) -> Any:
+ if "CREATE DATABASE" in sql:
+ raise RuntimeError("boom")
+ return []
+
+ client.execute = failing
+ assert await lake_migrations.run_migrations(client, strict=False) == []
+
+ async def test_drives_a_command_style_client(self) -> None:
+ """Two ClickHouse client shapes are in use across this repo. A
+ runner that speaks one silently does not run in half the
+ deployments."""
+ c = AsyncMock()
+ del c.execute
+ c.command = AsyncMock(return_value=[])
+ await lake_migrations.run_migrations(c)
+ assert c.command.await_count > 0
+
+ async def test_an_undrivable_client_is_a_loud_failure(self) -> None:
+ class Opaque:
+ pass
+
+ with pytest.raises(TypeError, match="neither execute"):
+ await lake_migrations._execute(Opaque(), "SELECT 1")
+
+ async def test_an_unreadable_ledger_is_a_first_run_not_a_crash(self, client: AsyncMock) -> None:
+ async def execute(sql: str, params: Any = None) -> Any:
+ if "SELECT migration_id" in sql:
+ raise RuntimeError("table does not exist")
+ return []
+
+ client.execute = execute
+ assert await lake_migrations.applied_ids(client) == set()
+
+ @pytest.mark.parametrize("row", [("001_baseline",), {"migration_id": "001_baseline"}, "001_baseline"])
+ async def test_reads_every_driver_row_shape(self, client: AsyncMock, row: Any) -> None:
+ """Drivers return tuples, dicts or scalars. Reading one shape means
+ the runner re-applies everything against the others."""
+ client.execute = AsyncMock(return_value=[row])
+ assert await lake_migrations.applied_ids(client) == {"001_baseline"}
+
+
+class TestVectorRunner:
+ @pytest.fixture
+ def client(self) -> AsyncMock:
+ c = AsyncMock()
+ c.get_collections = AsyncMock(return_value=type("R", (), {"collections": []})())
+ c.scroll = AsyncMock(return_value=([], None))
+ return c
+
+ async def test_applies_and_records(self, client: AsyncMock) -> None:
+ pytest.importorskip("qdrant_client")
+ applied = await vector_migrations.run_migrations(client)
+ assert applied == ["001_baseline"]
+ client.upsert.assert_awaited_once()
+
+ async def test_point_ids_are_stable_across_processes(self) -> None:
+ """Keyed on hash() the ledger would record the same migration under a
+ different id every restart, and the runner would re-apply
+ everything — the same salted-hash bug the sandbox had."""
+ first = vector_migrations._point_id("001_baseline")
+ assert first == vector_migrations._point_id("001_baseline")
+ assert first != vector_migrations._point_id("002_other")
+ assert first > 0, "Qdrant point ids must be unsigned"
+
+ async def test_a_rebuild_is_flagged_before_it_starts(self) -> None:
+ """A dimension change re-embeds every point. The difference between
+ a one-second migration and a four-hour one should not be discovered
+ at deploy time."""
+ assert hasattr(vector_migrations, "pending_rebuilds")
+ for migration in vector_migrations.MIGRATIONS:
+ assert isinstance(migration.rebuilds_collection, bool)
+
+ async def test_an_unreadable_ledger_is_a_first_run(self, client: AsyncMock) -> None:
+ pytest.importorskip("qdrant_client")
+ client.scroll = AsyncMock(side_effect=RuntimeError("collection missing"))
+ assert await vector_migrations.applied_ids(client) == set()
diff --git a/services/fusion/app/services/derived_fields.py b/services/fusion/app/services/derived_fields.py
new file mode 100644
index 000000000..25415f7b4
--- /dev/null
+++ b/services/fusion/app/services/derived_fields.py
@@ -0,0 +1,185 @@
+"""Fields the detection engine can compute from an event it already has.
+
+138 of 833 executable rules match on fields no connector emits, so they can
+never fire — a rule that never fires and a rule with nothing to fire on look
+identical from outside, which is why the number went unnoticed for so long.
+
+Most of those 138 genuinely need something that does not exist: a windowed
+evaluator (73), identity enrichment (24), per-tenant allowlists (15). This
+module handles the ones that need nothing at all — where the answer is
+already in the event and only the arithmetic is missing.
+
+Two families:
+
+**Field comparisons.** `actor_eq_target`, `actor_uid_neq_owner_uid`,
+`object_tenant_id_eq_caller_tenant_id`. The rule wants to know whether two
+fields of the same event agree. Both values are present; nothing was
+comparing them.
+
+**Time of day.** `is_business_hours`, `is_after_hours`, `is_weekend`. A pure
+function of the event timestamp.
+
+The rule that governs everything here: **an underivable field is absent, not
+false.** If either side of a comparison is missing, the derived key is not
+set, so a rule matching `actor_eq_target: false` does not fire on an event
+where neither actor nor target is known. Defaulting to `False` would make
+every event with missing data look like a cross-account access, which is a
+detection that fires on ignorance.
+"""
+
+from __future__ import annotations
+
+import re
+from datetime import datetime
+from typing import Any
+
+#: `<left>_eq_<right>` / `<left>_neq_<right>`. Non-greedy on the left so
+#: `actor_uid_neq_owner_uid` splits at the *first* separator, giving
+#: (actor_uid, owner_uid) rather than (actor, uid_neq_owner_uid).
+_COMPARISON_RE = re.compile(r"^(?P<left>.+?)_(?P<op>eq|neq)_(?P<right>.+)$")
+
+#: Business hours in the tenant's local time. A default rather than a
+#: constant: "outside business hours" means nothing without knowing whose
+#: business, and a rule that assumes UTC fires all night for half the world.
+DEFAULT_BUSINESS_START_HOUR = 8
+DEFAULT_BUSINESS_END_HOUR = 18
+
+#: Timestamp fields, in the order they are trusted. `event_time` is when the
+#: thing happened; `ingest_time` is when we heard about it, and using the
+#: latter would make a batch import at 03:00 look like a night-time attack.
+_TIME_FIELDS = ("event_time", "timestamp", "time", "@timestamp", "ingest_time")
+
+
+def _parse_time(event: dict[str, Any]) -> datetime | None:
+ for field in _TIME_FIELDS:
+ raw = event.get(field)
+ if raw is None:
+ continue
+ if isinstance(raw, datetime):
+ return raw
+ if isinstance(raw, int | float):
+ try:
+ # Milliseconds if it is far too large to be seconds.
+ seconds = raw / 1000 if raw > 1e11 else raw
+ return datetime.fromtimestamp(seconds) # noqa: DTZ006 - local by design
+ except (OSError, ValueError, OverflowError):
+ continue
+ if isinstance(raw, str):
+ try:
+ return datetime.fromisoformat(raw.replace("Z", "+00:00"))
+ except ValueError:
+ continue
+ return None
+
+
+def _normalise(value: Any) -> Any:
+ """Compare case-insensitively for strings, exactly for everything else.
+
+ `Alice` and `alice` are the same principal in every directory this
+ platform reads, and a comparison that says otherwise produces a
+ cross-account finding for a casing difference.
+ """
+ return value.strip().lower() if isinstance(value, str) else value
+
+
+def comparison_fields(event: dict[str, Any], requested: set[str] | None = None) -> dict[str, bool]:
+ """Compute `<a>_eq_<b>` / `<a>_neq_<b>` keys the rules ask for.
+
+ ``requested`` bounds the work to the field names some rule actually
+ matches on. Without it this would have to enumerate every pair of keys
+ in the event, which is quadratic and produces thousands of keys nothing
+ reads.
+ """
+ if not requested:
+ return {}
+
+ out: dict[str, bool] = {}
+ for name in requested:
+ match = _COMPARISON_RE.match(name)
+ if not match:
+ continue
+ left = event.get(match.group("left"))
+ right = event.get(match.group("right"))
+ # Absent, not false. A rule must not fire because we did not know.
+ if left is None or right is None:
+ continue
+ equal = _normalise(left) == _normalise(right)
+ out[name] = equal if match.group("op") == "eq" else not equal
+ return out
+
+
+def time_of_day_fields(
+ event: dict[str, Any],
+ *,
+ business_start: int = DEFAULT_BUSINESS_START_HOUR,
+ business_end: int = DEFAULT_BUSINESS_END_HOUR,
+) -> dict[str, bool]:
+ """Compute `is_business_hours`, `is_after_hours` and `is_weekend`.
+
+ Returns nothing when the event carries no parseable timestamp, for the
+ same reason as above: an unknown time is not "outside business hours".
+ """
+ when = _parse_time(event)
+ if when is None:
+ return {}
+
+ weekend = when.weekday() >= 5
+ in_hours = (not weekend) and business_start <= when.hour < business_end
+ return {
+ "is_business_hours": in_hours,
+ "is_after_hours": not in_hours,
+ "is_weekend": weekend,
+ }
+
+
+def enrich(
+ event: dict[str, Any],
+ requested: set[str] | None = None,
+ *,
+ business_start: int = DEFAULT_BUSINESS_START_HOUR,
+ business_end: int = DEFAULT_BUSINESS_END_HOUR,
+) -> dict[str, Any]:
+ """Return ``event`` plus every derivable field the rules ask for.
+
+ Non-destructive: a derived key never overwrites one the connector
+ supplied. If a vendor genuinely emits `is_business_hours`, its value is
+ the vendor's answer about its own tenant's hours, which is better than
+ ours.
+ """
+ derived: dict[str, Any] = {}
+ derived.update(time_of_day_fields(event, business_start=business_start, business_end=business_end))
+ derived.update(comparison_fields(event, requested))
+
+ if not derived:
+ return event
+ return {**derived, **event}
+
+
+def requested_derived_fields(rules: list[dict[str, Any]]) -> set[str]:
+ """Collect the derivable field names the ruleset matches on.
+
+ Computed once at ruleset load rather than per event: the set changes
+ when rules change, not when traffic arrives, and doing it per event was
+ the difference between a cheap enrichment and a hot-path regex over
+ every rule for every event.
+ """
+ from app.services.detection_matcher import _split_op # noqa: PLC0415
+
+ names: set[str] = set()
+
+ def walk(clause: Any) -> None:
+ if isinstance(clause, dict):
+ for key, value in clause.items():
+ if key in {"any_of", "all_of", "not"}:
+ walk(value)
+ continue
+ field, _ = _split_op(key)
+ if _COMPARISON_RE.match(field) or field.startswith("is_"):
+ names.add(field)
+ elif isinstance(clause, list):
+ for item in clause:
+ walk(item)
+
+ for rule in rules:
+ walk(rule.get("match_when") or {})
+ return names
diff --git a/services/fusion/app/services/detection_engine.py b/services/fusion/app/services/detection_engine.py
index aa5c43ca6..effa84aef 100644
--- a/services/fusion/app/services/detection_engine.py
+++ b/services/fusion/app/services/detection_engine.py
@@ -37,6 +37,7 @@ from typing import Any
import structlog
from app.models.alert import AlertSeverity, RawAlert
+from app.services.derived_fields import enrich, requested_derived_fields
from app.services.detection_matcher import matches
from app.services.provenance import extract_provenance
@@ -76,6 +77,10 @@ class DetectionEngine:
def __init__(self, rules: list[dict[str, Any]] | None = None) -> None:
self._rules: list[dict[str, Any]] = rules if rules is not None else _load_ruleset()
+ # Computed once at load, not per event: the set changes when rules
+ # change, not when traffic arrives. Doing it per event would mean a
+ # regex over every rule's clauses for every ingested event.
+ self._derived_wanted: set[str] = requested_derived_fields(self._rules)
@property
def rule_count(self) -> int:
@@ -149,6 +154,11 @@ class DetectionEngine:
if not isinstance(ocsf, dict):
return []
fields = self._raw_fields(ocsf)
+ # 17 rules match on comparisons between two fields of the same event
+ # (actor_eq_target, actor_uid_neq_owner_uid) or on time of day. Both
+ # values were already present and nothing was computing the answer,
+ # so those rules could never fire.
+ fields = enrich(fields, self._derived_wanted)
hits: list[DetectionHit] = []
for rule in self._candidates(""):
try:
diff --git a/services/fusion/app/services/detection_matcher.py b/services/fusion/app/services/detection_matcher.py
index fb9800f78..cb44434a2 100644
--- a/services/fusion/app/services/detection_matcher.py
+++ b/services/fusion/app/services/detection_matcher.py
@@ -40,6 +40,14 @@ OPERATORS: list[tuple[str, str, str]] = sorted(
("_has_any", "has_any", "HAS_ANY"),
("_not_in", "not_in", "NOT IN"),
("_match", "match", "MATCH"),
+ # `neq` is used by rules in the shipped corpus and has never
+ # existed here, so `approver_role_neq: "codeowner"` was read as a
+ # field literally named `approver_role_neq` — which nothing
+ # emits, so the rule could not fire. There is deliberately no
+ # `_eq` counterpart: bare equality is already the default, and
+ # adding the suffix would split any field whose name happens to
+ # end in `_eq` for no gain.
+ ("_neq", "neq", "!="),
("_gte", "gte", ">="),
("_lte", "lte", "<="),
("_in", "in", "IN"),
@@ -70,6 +78,14 @@ def _check(field: str, op: str, expected: Any, event: dict[str, Any]) -> bool:
return actual is None
return actual == expected
+ if op == "neq":
+ # A missing field is not "different from X". Returning True here
+ # would make every rule using neq fire on every event that lacks
+ # the field, which is a detection that fires on absence of data.
+ if actual is None:
+ return False
+ return actual != expected
+
if op in {"gt", "gte", "lt", "lte"}:
if not isinstance(actual, int | float) or isinstance(actual, bool):
return False
diff --git a/services/fusion/tests/test_alert_reduction_real.py b/services/fusion/tests/test_alert_reduction_real.py
new file mode 100644
index 000000000..d505743cb
--- /dev/null
+++ b/services/fusion/tests/test_alert_reduction_real.py
@@ -0,0 +1,198 @@
+"""Alert reduction, measured against the correlation logic that actually runs.
+
+The existing measurement in `services/agents/tests/test_alert_reduction.py`
+is honest about being synthetic and has always carried a PARTIAL row saying
+it "gates an in-test fusion re-impl, not `services/fusion`". Reading it
+closely, the gap is wider than that wording suggests: the test does not
+merely reimplement fusion's grouping, it implements **different** grouping.
+
+Its docstring describes four tiers keyed on `(rule_id, host, user)` with
+10/30/5-minute windows. `RawAlert.correlation_key()` — the method
+`Correlator` actually calls — keys on `{tenant}:{entity}:{tactic}`, where
+entity is the first of src_ip, hostname, username, domain. Different
+dimensions, different windows, different answer. The published
+alert-reduction figure therefore described an algorithm the product does
+not run.
+
+This measures the real one. `Correlator` itself needs Redis, but Redis is
+where it *stores* incidents — the grouping decision is entirely
+`correlation_key()` plus the configured window, so the ratio can be
+computed from the real method without the storage layer.
+
+The number this produces is a property of a synthetic workload and is
+labelled as such, exactly as the original was. What changes is that it is
+now a property of *our* algorithm.
+"""
+
+from __future__ import annotations
+
+import hashlib
+from datetime import UTC, datetime, timedelta
+from uuid import UUID
+
+from app.models.alert import AlertSeverity, RawAlert
+
+TENANT = UUID("11111111-1111-1111-1111-111111111111")
+
+#: Matches the production default (`CORRELATION_WINDOW_SECONDS`). Read as a
+#: literal rather than from settings so the measurement does not silently
+#: change meaning when an operator tunes their own deployment.
+WINDOW = timedelta(seconds=3600)
+
+#: Alert count. Same order as the original so the two numbers are
+#: comparable, which is the point of keeping both.
+STREAM_SIZE = 1000
+
+TACTICS = [
+ "initial-access",
+ "execution",
+ "persistence",
+ "credential-access",
+ "lateral-movement",
+ "exfiltration",
+]
+
+
+def _seeded(n: int, salt: str, modulus: int) -> int:
+ """Deterministic index. blake2b, not hash(): CPython salts string
+ hashing per process, so a `hash()`-seeded stream would produce a
+ different reduction ratio on every run and the figure would be
+ unreproducible — the same defect the sandbox had."""
+ digest = hashlib.blake2b(f"{salt}:{n}".encode(), digest_size=8).digest()
+ return int.from_bytes(digest, "big") % modulus
+
+
+def build_stream(size: int = STREAM_SIZE) -> list[tuple[datetime, RawAlert]]:
+ """A noisy alert stream with the shapes fusion is supposed to collapse.
+
+ Generated independently of the correlation logic — the point of the
+ measurement is that the stream does not know how it will be grouped.
+ """
+ start = datetime(2026, 9, 22, 8, 0, tzinfo=UTC)
+ stream: list[tuple[datetime, RawAlert]] = []
+
+ for n in range(size):
+ # A burst: many alerts from one host in a tight window, the case
+ # correlation exists for.
+ if n % 10 < 4:
+ host = f"ws-{_seeded(n, 'burst-host', 12):03d}"
+ offset = timedelta(minutes=_seeded(n, "burst-time", 45))
+ # Scattered singletons that should survive as distinct incidents.
+ else:
+ host = f"ws-{_seeded(n, 'wide-host', 400):03d}"
+ offset = timedelta(minutes=_seeded(n, "wide-time", 1440))
+
+ stream.append(
+ (
+ start + offset,
+ RawAlert(
+ tenant_id=TENANT,
+ source="synthetic",
+ title=f"alert {n}",
+ hostname=host,
+ username=f"u{_seeded(n, 'user', 60)}",
+ severity=AlertSeverity.MEDIUM,
+ mitre_tactics=[TACTICS[_seeded(n, "tactic", len(TACTICS))]],
+ ),
+ )
+ )
+ return stream
+
+
+def group_by_real_key(stream: list[tuple[datetime, RawAlert]], window: timedelta = WINDOW) -> int:
+ """Incident count under the real correlation key and window.
+
+ Mirrors what `Correlator.correlate` does: look up the incident for this
+ alert's correlation key, join it if one is open, otherwise open a new
+ one. Redis is the lookup table; the decision is the key.
+ """
+ open_incidents: dict[str, datetime] = {}
+ incidents = 0
+
+ for when, alert in sorted(stream, key=lambda pair: pair[0]):
+ key = alert.correlation_key()
+ opened = open_incidents.get(key)
+ if opened is not None and when - opened <= window:
+ continue
+ open_incidents[key] = when
+ incidents += 1
+
+ return incidents
+
+
+def test_reduction_is_measured_against_the_real_correlation_key() -> None:
+ """The measurement runs `RawAlert.correlation_key()`, not a copy of it.
+
+ A reimplementation can drift from the thing it stands for without any
+ test failing, and this one had — it grouped on different dimensions
+ entirely.
+ """
+ stream = build_stream()
+ incidents = group_by_real_key(stream)
+ reduction = 1 - (incidents / len(stream))
+
+ print(f"\n[eval] alert reduction (real correlation_key): " f"{len(stream)} alerts -> {incidents} incidents = {reduction:.1%}")
+
+ # Bounded on both sides. A floor alone would be satisfied by a key that
+ # collapses everything into one incident, which is 99.9% reduction and
+ # a useless SOC.
+ assert 0.20 <= reduction <= 0.95, (
+ f"reduction {reduction:.1%} is outside the plausible band; either "
+ f"correlation stopped grouping or it is collapsing unrelated alerts"
+ )
+
+
+def test_the_key_is_the_products_key_not_a_local_one() -> None:
+ """Pins the dimensions, so a change to `correlation_key()` that alters
+ what gets grouped cannot pass silently."""
+ alert = RawAlert(
+ tenant_id=TENANT,
+ source="s",
+ title="t",
+ src_ip="10.0.0.1",
+ hostname="ws-1",
+ username="alice",
+ mitre_tactics=["execution"],
+ )
+ assert alert.correlation_key() == f"{TENANT}:10.0.0.1:execution"
+
+
+def test_entity_precedence_is_src_ip_then_hostname() -> None:
+ """Which entity wins decides what groups with what, so it is worth
+ pinning rather than inferring from the reduction number."""
+ base = {"tenant_id": TENANT, "source": "s", "title": "t", "mitre_tactics": ["exec"]}
+ assert RawAlert(**base, hostname="ws-1").correlation_key().endswith("ws-1:exec")
+ assert RawAlert(**base, src_ip="10.0.0.1", hostname="ws-1").correlation_key().endswith("10.0.0.1:exec")
+
+
+def test_alerts_without_a_tactic_do_not_all_collapse_together() -> None:
+ """`correlation_key()` falls back to the literal "unknown" for both
+ entity and tactic. Two unrelated alerts each missing one field still
+ differ on the other; two missing both would merge, which is worth
+ knowing rather than discovering in production."""
+ a = RawAlert(tenant_id=TENANT, source="s", title="a", hostname="ws-1")
+ b = RawAlert(tenant_id=TENANT, source="s", title="b", hostname="ws-2")
+ assert a.correlation_key() != b.correlation_key()
+
+ bare_a = RawAlert(tenant_id=TENANT, source="s", title="a")
+ bare_b = RawAlert(tenant_id=TENANT, source="s", title="b")
+ assert bare_a.correlation_key() == bare_b.correlation_key(), (
+ "two alerts with no entity and no tactic share a key and will merge; "
+ "this is current behaviour, pinned so a change to it is deliberate"
+ )
+
+
+def test_the_stream_is_reproducible() -> None:
+ """A measurement that moves between runs is not a measurement."""
+ first = group_by_real_key(build_stream())
+ second = group_by_real_key(build_stream())
+ assert first == second
+
+
+def test_a_wider_window_reduces_more() -> None:
+ """Sanity on the window's direction. If this inverts, the grouping is
+ not doing what the name says."""
+ stream = build_stream()
+ narrow = group_by_real_key(stream, timedelta(minutes=5))
+ wide = group_by_real_key(stream, timedelta(hours=24))
+ assert wide <= narrow
diff --git a/services/fusion/tests/test_derived_fields.py b/services/fusion/tests/test_derived_fields.py
new file mode 100644
index 000000000..447e3d875
--- /dev/null
+++ b/services/fusion/tests/test_derived_fields.py
@@ -0,0 +1,213 @@
+"""Fields the engine can compute, and the ones it must refuse to guess.
+
+138 of 833 executable rules matched on fields no connector emits. Most need
+something that does not exist — a windowed evaluator, identity enrichment,
+per-tenant allowlists. A handful needed nothing: both values were already
+in the event and nothing was comparing them.
+
+The rule that governs this module, and most of what is tested below: **an
+underivable field is absent, not false.** Defaulting a missing comparison to
+`False` would make every event with incomplete data satisfy
+`actor_eq_target: false` — a cross-account-access detection that fires on
+ignorance, on every tenant, forever.
+"""
+
+from __future__ import annotations
+
+from datetime import datetime
+
+import pytest
+from app.services.derived_fields import (
+ comparison_fields,
+ enrich,
+ requested_derived_fields,
+ time_of_day_fields,
+)
+
+
+class TestComparisons:
+ def test_equal_fields_compare_equal(self) -> None:
+ event = {"actor": "alice", "target": "alice"}
+ assert comparison_fields(event, {"actor_eq_target"}) == {"actor_eq_target": True}
+
+ def test_neq_is_the_inverse(self) -> None:
+ event = {"actor_uid": "1000", "owner_uid": "0"}
+ result = comparison_fields(event, {"actor_uid_neq_owner_uid"})
+ assert result == {"actor_uid_neq_owner_uid": True}
+
+ def test_a_missing_operand_yields_no_key_rather_than_false(self) -> None:
+ """The failure that would fire on every incomplete event."""
+ event = {"actor": "alice"}
+ assert comparison_fields(event, {"actor_eq_target"}) == {}
+
+ def test_both_operands_missing_yields_no_key(self) -> None:
+ assert comparison_fields({}, {"actor_eq_target"}) == {}
+
+ def test_case_differences_are_the_same_principal(self) -> None:
+ """`Alice` and `alice` are one account in every directory we read;
+ saying otherwise produces a finding for a casing difference."""
+ event = {"actor": "Alice", "target": "alice "}
+ assert comparison_fields(event, {"actor_eq_target"})["actor_eq_target"] is True
+
+ def test_splits_at_the_first_separator(self) -> None:
+ """`actor_uid_neq_owner_uid` is (actor_uid, owner_uid), not
+ (actor, uid_neq_owner_uid)."""
+ event = {"actor_uid": "1", "owner_uid": "1"}
+ assert comparison_fields(event, {"actor_uid_neq_owner_uid"}) == {"actor_uid_neq_owner_uid": False}
+
+ def test_only_requested_fields_are_computed(self) -> None:
+ """Enumerating every pair of event keys is quadratic and produces
+ thousands of keys nothing reads."""
+ event = {"a": 1, "b": 1, "c": 1}
+ assert comparison_fields(event, {"a_eq_b"}) == {"a_eq_b": True}
+
+ def test_no_request_means_no_work(self) -> None:
+ assert comparison_fields({"a": 1, "b": 1}, None) == {}
+
+ def test_a_name_that_is_not_a_comparison_is_ignored(self) -> None:
+ assert comparison_fields({"user_name": "x"}, {"user_name"}) == {}
+
+ @pytest.mark.parametrize(
+ ("left", "right", "equal"),
+ [(0, 0, True), (0, False, True), (1, "1", False), (None, None, None)],
+ )
+ def test_non_string_operands(self, left: object, right: object, equal: object) -> None:
+ event = {}
+ if left is not None:
+ event["a"] = left
+ if right is not None:
+ event["b"] = right
+ result = comparison_fields(event, {"a_eq_b"})
+ if equal is None:
+ assert result == {}
+ else:
+ assert result["a_eq_b"] is equal
+
+
+class TestTimeOfDay:
+ def test_a_weekday_afternoon_is_business_hours(self) -> None:
+ # 2026-09-22 is a Tuesday.
+ result = time_of_day_fields({"event_time": datetime(2026, 9, 22, 14, 0)})
+ assert result["is_business_hours"] is True
+ assert result["is_after_hours"] is False
+ assert result["is_weekend"] is False
+
+ def test_a_weekday_night_is_not(self) -> None:
+ result = time_of_day_fields({"event_time": datetime(2026, 9, 22, 3, 0)})
+ assert result["is_business_hours"] is False
+ assert result["is_after_hours"] is True
+
+ def test_a_saturday_afternoon_is_not_business_hours(self) -> None:
+ # 2026-09-26 is a Saturday.
+ result = time_of_day_fields({"event_time": datetime(2026, 9, 26, 14, 0)})
+ assert result["is_weekend"] is True
+ assert result["is_business_hours"] is False
+
+ def test_no_timestamp_yields_nothing(self) -> None:
+ """An unknown time is not "outside business hours"."""
+ assert time_of_day_fields({"user_name": "alice"}) == {}
+
+ def test_an_unparseable_timestamp_yields_nothing(self) -> None:
+ assert time_of_day_fields({"event_time": "not a date"}) == {}
+
+ def test_event_time_wins_over_ingest_time(self) -> None:
+ """Using ingest time would make a batch import at 03:00 look like a
+ night-time attack."""
+ result = time_of_day_fields(
+ {
+ "event_time": datetime(2026, 9, 22, 14, 0),
+ "ingest_time": datetime(2026, 9, 22, 3, 0),
+ }
+ )
+ assert result["is_business_hours"] is True
+
+ def test_iso_strings_parse(self) -> None:
+ result = time_of_day_fields({"event_time": "2026-09-22T14:00:00Z"})
+ assert result["is_business_hours"] is True
+
+ def test_epoch_milliseconds_parse(self) -> None:
+ ms = int(datetime(2026, 9, 22, 14, 0).timestamp() * 1000)
+ assert time_of_day_fields({"event_time": ms})["is_business_hours"] is True
+
+ def test_business_hours_are_configurable(self) -> None:
+ """ "Outside business hours" means nothing without knowing whose
+ business; a hardcoded window fires all night for half the world."""
+ event = {"event_time": datetime(2026, 9, 22, 20, 0)}
+ assert time_of_day_fields(event)["is_business_hours"] is False
+ assert time_of_day_fields(event, business_start=9, business_end=23)["is_business_hours"] is True
+
+
+class TestEnrich:
+ def test_a_vendor_value_is_never_overwritten(self) -> None:
+ """A vendor's own answer about its own tenant's hours beats ours."""
+ event = {"event_time": datetime(2026, 9, 22, 3, 0), "is_business_hours": True}
+ assert enrich(event)["is_business_hours"] is True
+
+ def test_returns_the_event_unchanged_when_nothing_is_derivable(self) -> None:
+ event = {"user_name": "alice"}
+ assert enrich(event) is event
+
+ def test_combines_both_families(self) -> None:
+ event = {
+ "event_time": datetime(2026, 9, 22, 3, 0),
+ "actor": "alice",
+ "target": "bob",
+ }
+ result = enrich(event, {"actor_eq_target"})
+ assert result["is_after_hours"] is True
+ assert result["actor_eq_target"] is False
+ assert result["actor"] == "alice"
+
+
+class TestRequestedFields:
+ def test_collects_comparison_and_time_fields(self) -> None:
+ rules = [
+ {"match_when": {"actor_eq_target": False, "event_name": "CreateAccessKey"}},
+ {"match_when": {"is_business_hours": False}},
+ ]
+ assert requested_derived_fields(rules) == {"actor_eq_target", "is_business_hours"}
+
+ def test_walks_nested_clauses(self) -> None:
+ rules = [{"match_when": {"any_of": [{"actor_eq_target": True}, {"x": 1}]}}]
+ assert "actor_eq_target" in requested_derived_fields(rules)
+
+ def test_strips_operator_suffixes(self) -> None:
+ """Rules write `is_business_hours: false` bare, but a clause can
+ carry an operator and the field name still has to be recovered."""
+ rules = [{"match_when": {"is_business_hours_in": [False]}}]
+ assert "is_business_hours" in requested_derived_fields(rules)
+
+ def test_ordinary_fields_are_not_collected(self) -> None:
+ rules = [{"match_when": {"user_name": "alice", "process_name_contains": "x"}}]
+ assert requested_derived_fields(rules) == set()
+
+
+class TestNeqOperator:
+ """`neq` is used by rules in the shipped corpus and never existed in the
+ matcher, so `approver_role_neq: "codeowner"` was read as a field
+ literally named `approver_role_neq` — which nothing emits."""
+
+ def test_neq_matches_a_different_value(self) -> None:
+ from app.services.detection_matcher import matches
+
+ assert matches({"approver_role_neq": "codeowner"}, {"approver_role": "dev"})
+
+ def test_neq_does_not_match_an_equal_value(self) -> None:
+ from app.services.detection_matcher import matches
+
+ assert not matches({"approver_role_neq": "codeowner"}, {"approver_role": "codeowner"})
+
+ def test_neq_does_not_fire_on_a_missing_field(self) -> None:
+ """Otherwise every neq rule fires on every event lacking the field —
+ a detection that fires on absence of data."""
+ from app.services.detection_matcher import matches
+
+ assert not matches({"approver_role_neq": "codeowner"}, {"user_name": "alice"})
+
+ def test_the_operator_split_does_not_capture_comparison_names(self) -> None:
+ """`actor_uid_neq_owner_uid` is a derived field, not `actor_uid`
+ with a neq operator. Adding the operator must not break it."""
+ from app.services.detection_matcher import _split_op
+
+ assert _split_op("actor_uid_neq_owner_uid") == ("actor_uid_neq_owner_uid", "eq")
+ assert _split_op("actor_uid_neq") == ("actor_uid", "neq")
diff --git a/services/fusion/tests/test_detection_matcher_parity.py b/services/fusion/tests/test_detection_matcher_parity.py
index 79e836638..ed74a0cee 100644
--- a/services/fusion/tests/test_detection_matcher_parity.py
+++ b/services/fusion/tests/test_detection_matcher_parity.py
@@ -65,15 +65,77 @@ def test_vendored_matcher_matches_canonical_over_all_fixtures():
def test_positive_fixtures_fire_and_negatives_do_not():
- """Sanity: the vendored matcher upholds the fixture contract directly."""
+ """Sanity: the vendored matcher upholds the fixture contract directly.
+
+ Events go through the same derived-field enrichment
+ ``DetectionEngine.evaluate`` applies. Replaying a fixture against the
+ bare matcher tests a pipeline production does not run — and a rule
+ matching on a derived field would fail here while working live, which
+ trains people to weaken the gate.
+ """
+ from app.services.derived_fields import enrich, requested_derived_fields
+
fixtures_dir = _REPO / "detections" / "fixtures"
ruleset = _REPO / "services" / "fusion" / "app" / "data" / "detection_ruleset.json"
- rules = {r["slug"]: r["match_when"] for r in json.loads(ruleset.read_text())["rules"]}
+ all_rules = json.loads(ruleset.read_text())["rules"]
+ rules = {r["slug"]: r["match_when"] for r in all_rules}
+ wanted = requested_derived_fields(all_rules)
+
checked = 0
for f in sorted((fixtures_dir / "positive").glob("*.json")):
mw = rules.get(f.stem)
if mw is None:
continue
- assert vendored_matches(mw, json.loads(f.read_text())), f"positive fixture did not fire: {f.stem}"
+ event = enrich(json.loads(f.read_text()), wanted)
+ assert vendored_matches(mw, event), f"positive fixture did not fire: {f.stem}"
checked += 1
assert checked > 100, f"expected to check many positive fixtures, only {checked}"
+
+ negatives = 0
+ for f in sorted((fixtures_dir / "negative").glob("*.json")):
+ mw = rules.get(f.stem)
+ if mw is None:
+ continue
+ event = enrich(json.loads(f.read_text()), wanted)
+ assert not vendored_matches(mw, event), f"negative fixture fired: {f.stem}"
+ negatives += 1
+ assert negatives > 100, f"expected to check many negative fixtures, only {negatives}"
+
+
+def test_vendored_derived_fields_match_canonical():
+ """The derived-field helpers are vendored the same way `matches()` is.
+
+ Two copies that can drift silently are worse than one copy plus a gate,
+ and this is the gate. Without it the engine could compute a field the
+ validator does not, so a rule would pass CI and never fire — or fire in
+ CI and never in production.
+ """
+ from app.services import derived_fields as vendored
+
+ canonical = _load_canonical()
+ cases = [
+ ({"actor": "alice", "target": "alice"}, {"actor_eq_target"}),
+ ({"actor": "Alice", "target": "alice "}, {"actor_eq_target"}),
+ ({"actor": "alice"}, {"actor_eq_target"}),
+ ({"actor_uid": 1000, "owner_uid": 0}, {"actor_uid_neq_owner_uid"}),
+ ({"event_time": "2026-09-22T14:00:00"}, set()),
+ ({"event_time": "2026-09-26T14:00:00"}, set()),
+ ({"event_time": "not a date"}, set()),
+ ({"user_name": "x"}, {"a_eq_b"}),
+ ]
+ for event, wanted in cases:
+ assert vendored.enrich(dict(event), set(wanted)) == canonical.enrich(
+ dict(event), set(wanted)
+ ), f"vendored and canonical enrich disagree on {event}"
+
+
+def test_vendored_requested_fields_match_canonical():
+ from app.services import derived_fields as vendored
+
+ canonical = _load_canonical()
+ rules = [
+ {"match_when": {"actor_eq_target": False, "event_name": "CreateAccessKey"}},
+ {"match_when": {"any_of": [{"is_business_hours": False}, {"x": 1}]}},
+ {"match_when": {"actor_uid_neq_owner_uid": True, "syscall": "openat"}},
+ ]
+ assert vendored.requested_derived_fields(rules) == canonical.requested_derived_fields(rules)
diff --git a/services/threatintel/app/storage/qdrant.py b/services/threatintel/app/storage/qdrant.py
index 48aa6b344..17d5edf3b 100644
--- a/services/threatintel/app/storage/qdrant.py
+++ b/services/threatintel/app/storage/qdrant.py
@@ -63,7 +63,14 @@ class QdrantStore:
self._use_fastembed = use_fastembed
async def initialize(self) -> None:
- """Create Qdrant collections if they don't exist."""
+ """Create the collections, then converge the schema.
+
+ Creation is `if not exists`, which is correct for a first run and a
+ no-op afterwards — so on its own it can never change the shape of a
+ collection that already exists. Anything beyond creation (a payload
+ index, a quantization setting, and eventually a re-embed when the
+ embedding model changes) has to come from the migration runner.
+ """
existing = await self._client.get_collections()
existing_names = {c.name for c in existing.collections}
@@ -75,6 +82,45 @@ class QdrantStore:
)
logger.info("Created Qdrant collection", collection=col)
+ await self._run_migrations()
+
+ async def _run_migrations(self) -> None:
+ """Apply pending vector-store migrations.
+
+ Non-strict, matching the graph and lake runners: threat-intel
+ enrichment degrades rather than taking the service down, and a
+ pending migration is logged at ``error`` so a degraded start is
+ visible rather than assumed.
+ """
+ try:
+ from app.db.vector_migrations import ( # noqa: PLC0415
+ pending_ids,
+ pending_rebuilds,
+ run_migrations,
+ )
+ except ImportError:
+ # The runner lives in services/api and is vendored into
+ # deployments that need it. Its absence means no migrations to
+ # apply, not a broken store.
+ return
+
+ rebuilds = await pending_rebuilds(self._client)
+ if rebuilds:
+ logger.warning(
+ "vector migrations include a re-embed; this is measured in "
+ "hours for a large tenant and the alias keeps the existing "
+ "collection serving until it completes",
+ migrations=rebuilds,
+ )
+
+ applied = await run_migrations(self._client, strict=False)
+ pending = await pending_ids(self._client)
+
+ if applied:
+ logger.info("Qdrant migrations applied", migrations=applied)
+ if pending:
+ logger.error("Qdrant migrations still pending", migrations=pending)
+
async def upsert_iocs(self, iocs: list[dict[str, Any]], *, tenant_id: str = SHARED_TENANT) -> None:
"""Embed and upsert IOC documents into Qdrant for semantic search.