Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
Finish the plan: store migrations for all three stores, and the derived fields (#707)
Beenu Arorae979ed965b36
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex ea3988be5..2c6041422 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -254,6 +254,13 @@ jobs:# Redaction is allow-list rather than deny-list, so a setting added# tomorrow is redacted by default rather than exposed until someone# remembers — and these tests are what keep it that way.+ # A panel for a metric nothing exports reads "No data" forever,+ # which teaches people that empty panels are normal — and then a+ # genuinely dead service looks like an uninstrumented one. Also+ # checks datasource uids resolve: Grafana does not warn about a+ # dangling reference, the link just silently does nothing.+ - name: Grafana dashboards query metrics that exist+ run: python scripts/check_grafana_dashboards.py- name: Support bundle must not leakrun: |python3 -m pip install --quiet pytestdiff --git a/README.md b/README.mdindex f44fbb577..8e1c236e7 100644--- a/README.md+++ b/README.md@@ -172,7 +172,7 @@ Full architecture (every service, every storage role, the v1.5 console workbenchA handful of headline capabilities — the rest are catalogued in [`apps/docs/docs/features/`](apps/docs/docs/features/) and indexed at the top of [`apps/docs/docs/intro.md`](apps/docs/docs/intro.md):-> **Maturity (v8.0.0 — Close the loop).** The end-to-end spine is wired and CI-gated: ingest → ClickHouse lake → live detection → fused alert → auto-triage → governed response. v8.0 connected capabilities the codebase already contained but never called. Response actions are now **verified against the vendor** rather than assumed from a 200 — an action that reports success while the effect is absent is reported as failed, because that gap is how a SOC comes to believe a host is contained when it is not. Autonomy is governed by **each tenant's own** L0–L4 policy instead of one deployment-wide environment variable. Repeat-alert suppression, shipped in v7.7, **could never fire** — the evidence fingerprint included the alert id, so no repeat ever matched; it now does. Analysts can **hunt the events AiSOC itself ingested** instead of needing an external SIEM. A verdict citing indicators the evidence never contained is demoted to human review rather than auto-closed. Autonomous *response* still defaults to copilot/dry-run: an autonomy policy governs every real execution. The live-agent LLM benchmark is preview (the deterministic-tier scoreboard is CI-gated per PR); substrate eval suites are GA. Every product claim is backed by a failing test — [claim-to-gate matrix](docs/audit/CLAIM_TO_GATE_MATRIX.md): 92 GATED / 10 PARTIAL / 0 NO GATE. Full per-claim status: [`docs/audit/REALITY_REPORT.md`](docs/audit/REALITY_REPORT.md).+> **Maturity (v8.0.0 — Close the loop).** The end-to-end spine is wired and CI-gated: ingest → ClickHouse lake → live detection → fused alert → auto-triage → governed response. v8.0 connected capabilities the codebase already contained but never called. Response actions are now **verified against the vendor** rather than assumed from a 200 — an action that reports success while the effect is absent is reported as failed, because that gap is how a SOC comes to believe a host is contained when it is not. Autonomy is governed by **each tenant's own** L0–L4 policy instead of one deployment-wide environment variable. Repeat-alert suppression, shipped in v7.7, **could never fire** — the evidence fingerprint included the alert id, so no repeat ever matched; it now does. Analysts can **hunt the events AiSOC itself ingested** instead of needing an external SIEM. A verdict citing indicators the evidence never contained is demoted to human review rather than auto-closed. Autonomous *response* still defaults to copilot/dry-run: an autonomy policy governs every real execution. The live-agent LLM benchmark is preview (the deterministic-tier scoreboard is CI-gated per PR); substrate eval suites are GA. Every product claim is backed by a failing test — [claim-to-gate matrix](docs/audit/CLAIM_TO_GATE_MATRIX.md): 93 GATED / 9 PARTIAL / 0 NO GATE. Full per-claim status: [`docs/audit/REALITY_REPORT.md`](docs/audit/REALITY_REPORT.md).- **84 click-and-connect data connectors** (EDR/XDR, SIEM, NDR, cloud, CNAPP, identity, SaaS, VCS, K8s audit, network) with schema-driven config, live `Test connection`, and vault-encrypted secrets — recently adding Qualys, GreyNoise, JumpCloud, Darktrace, and Imperva alongside IBM QRadar, Netskope, Zeek/Suricata NDR, and more. One query runs SIEM-agnostic **federated search** across Splunk SPL / Sentinel KQL / Elastic ES|QL / **QRadar AQL**. Walkthrough: [`apps/docs/docs/connectors/index.md`](apps/docs/docs/connectors/index.md).- **End-to-end SIEM spine** — a cold `docker compose up` ingests connector data → lands it in the ClickHouse event lake → the executable detection corpus (833 rules) fires on the live stream → a fused alert is created, all asserted by an extended integration gate. Fuse-time threat-intel + CISA-KEV enrichment now feeds the confidence score and exploit-in-wild boost, and **stateful/windowed detections** (brute-force, password-spray, port-scan) run alongside the corpus. [`apps/docs/docs/architecture.md`](apps/docs/docs/architecture.md).diff --git a/apps/docs/docs/benchmark.md b/apps/docs/docs/benchmark.mdindex 2cdac2c77..adc82fdfa 100644--- a/apps/docs/docs/benchmark.md+++ b/apps/docs/docs/benchmark.md@@ -69,7 +69,7 @@ agent performance**. Every table below is labelled with its class.:::warning Read this firstThis harness does **not** exercise the live LLM agent (`services/agents`-LangGraph orchestrator), and the `alert_reduction` suite does **not** call the+LangGraph orchestrator), and the legacy `alert_reduction` suite does **not** call theproduction `services/fusion` engine — it calls a standalone re-implementationof the same Tier 1/2/3 grouping rules that lives in the test file. It runs**deterministic substrate code** against **synthetic data** so we can gate@@ -105,12 +105,41 @@ every PR targeting `main` or `develop`.| Suite | Metric | Per-case | Per-template macro | Target | What it checks ||--------------------------------|-------------------------|------------|------------------------|---------|----------------|-| Alert reduction ratio | reduction | 75.3 % | _n/a_ | ≥ 70 % | Real measurement of the 3-tier fusion logic on a noisy 1 000-alert stream |+| Alert reduction (product logic) | reduction | 33.3 % | _n/a_ | 20–95 % | **The number that describes AiSOC.** Groups a noisy 1 000-alert stream with `RawAlert.correlation_key()` — the method `Correlator` actually calls |+| Alert reduction (legacy suite) | reduction | 75.3 % | _n/a_ | ≥ 70 % | A four-tier scheme implemented inside the test. Retained for continuity; **does not describe this product** — see below || MITRE ATT&CK tactic accuracy | accuracy | 97.0 % | 96.4 % (n=55) | ≥ 80 % | Substrate self-consistency — keyword extractor vs. dataset written for it || Investigation completeness | mean keyword coverage | 94.2 % | 94.3 % (n=55) | ≥ 85 % | Substrate self-consistency — report template wraps the description; judge finds keywords from the description || Response-plan quality | mean rubric score | 1.000 | 1.000 (n=55) | ≥ 0.80 | Substrate self-consistency — synthesizer embeds the keywords the rubric checks for || Playbook completion rate | completion rate | 50.5 % | 100 % H/C (mapped) | ≥ 50 % | Operational coverage gate — every incident in scope has a matching playbook with aligned response action; orphan playbooks/templates fail CI |+### Why there are two alert-reduction numbers++The original suite was honest about being synthetic and always carried a+`PARTIAL` row saying it gated "an in-test fusion re-impl". Reading it+closely, the gap was wider than that wording admitted: the test did not+merely reimplement fusion's grouping, it implemented **different**+grouping.++Its four tiers key on `(rule_id, host, user)` with 10/30/5-minute windows.+`RawAlert.correlation_key()` — what `Correlator` actually calls — keys on+`{tenant}:{entity}:{tactic}` over a one-hour window. Different dimensions,+different windows, different answer. So the 75.3 % described an algorithm+the product does not run, and a reimplementation can drift from the thing+it stands for without any test failing.++`services/fusion/tests/test_alert_reduction_real.py` measures the real key.+It reports **33.3 %** on a comparable stream. That is less flattering and it+is ours.++The new gate is bounded on both sides rather than floored. A floor alone is+satisfied by a key that collapses everything into one incident — 99.9 %+reduction and a useless SOC — so "more reduction is better" is only true up+to a point, and the gate says where.++Both numbers stay published. Deleting the old one would make the history+unreadable; presenting it without this note would be the thing this page+exists to prevent.+> The synthetic telemetry suite is a **schema/coverage gate**, not a scoring> suite, so it does not appear in the table. It checks that every incident has> ≥ 1 backing event, that all `{user}/{host}/{ip}/{campaign}` placeholdersdiff --git a/apps/docs/docs/compliance/evidence-pack.md b/apps/docs/docs/compliance/evidence-pack.mdindex eb81d102d..bed6aaecc 100644--- a/apps/docs/docs/compliance/evidence-pack.md+++ b/apps/docs/docs/compliance/evidence-pack.md@@ -33,8 +33,8 @@ and says so.| Prompt-injection resistance | 32-payload adversarial corpus with a recall floor and a false-positive ceiling | `services/agents/tests/adversarial/` || Claim-to-gate traceability | Every product claim mapped to the CI job that fails when it stops being true | `docs/audit/CLAIM_TO_GATE_MATRIX.md` |-The last row is the one worth reading first. It is the honest index: 92 rows-`GATED`, 10 `PARTIAL` with the specific gap named, and a ratchet that refuses+The last row is the one worth reading first. It is the honest index: 93 rows+`GATED`, 9 `PARTIAL` with the specific gap named, and a ratchet that refusesto let a row sit at `NO GATE`.## Not covered, and not claimeddiff --git a/detections/fixtures/negative/app-ci-workflow-modified-by-non-owner.json b/detections/fixtures/negative/app-ci-workflow-modified-by-non-owner.jsonindex 9ecf21f06..096336c5d 100644--- a/detections/fixtures/negative/app-ci-workflow-modified-by-non-owner.json+++ b/detections/fixtures/negative/app-ci-workflow-modified-by-non-owner.json@@ -1,5 +1,5 @@{- "approver_role_neq": "codeowner",- "event_type": "not-push",+ "approver_role": "codeowner",+ "event_type": "push","path": "prefix-.github/workflows/-suffix"}diff --git a/detections/fixtures/negative/aws-iam-access-key-created-other-user.json b/detections/fixtures/negative/aws-iam-access-key-created-other-user.jsonindex b4478ae50..02d1fef97 100644--- a/detections/fixtures/negative/aws-iam-access-key-created-other-user.json+++ b/detections/fixtures/negative/aws-iam-access-key-created-other-user.json@@ -1,4 +1,5 @@{- "actor_eq_target": false,- "event_name": "not-CreateAccessKey"+ "actor": "principal-a",+ "event_name": "CreateAccessKey",+ "target": "principal-a"}diff --git a/detections/fixtures/negative/cloud-multi-cross-cloud-mass-egress.json b/detections/fixtures/negative/cloud-multi-cross-cloud-mass-egress.jsonindex d989141c9..43326d1a3 100644--- a/detections/fixtures/negative/cloud-multi-cross-cloud-mass-egress.json+++ b/detections/fixtures/negative/cloud-multi-cross-cloud-mass-egress.json@@ -1,6 +1,5 @@{"bytes_out_per_minute": 1073741924.0,"dst_cloud": "aws",- "src_cloud": "__benign__",- "src_cloud_neq_dst_cloud": true+ "src_cloud": "aws"}diff --git a/detections/fixtures/negative/cloud-org-rootmail-modified.json b/detections/fixtures/negative/cloud-org-rootmail-modified.jsonindex aa390286d..a0b8a2c4f 100644--- a/detections/fixtures/negative/cloud-org-rootmail-modified.json+++ b/detections/fixtures/negative/cloud-org-rootmail-modified.json@@ -1,4 +1,5 @@{- "actor_eq_target": false,- "event_name": "__benign__"+ "actor": "principal-a",+ "event_name": "UpdateAccountEmail",+ "target": "principal-a"}diff --git a/detections/fixtures/negative/ident-ntlm-relay-indicator.json b/detections/fixtures/negative/ident-ntlm-relay-indicator.jsonindex 75fb36740..9f9eb5652 100644--- a/detections/fixtures/negative/ident-ntlm-relay-indicator.json+++ b/detections/fixtures/negative/ident-ntlm-relay-indicator.json@@ -1,5 +1,6 @@{"auth_type": "NTLMv2",- "event_id": 8005.0,- "workstation_neq_source_address": true+ "event_id": 8004,+ "source_address": "principal-a",+ "workstation": "principal-a"}diff --git a/detections/fixtures/negative/ident-pass-the-ticket-anomalous-tgt-source.json b/detections/fixtures/negative/ident-pass-the-ticket-anomalous-tgt-source.jsonindex 139218a6d..01709fa29 100644--- a/detections/fixtures/negative/ident-pass-the-ticket-anomalous-tgt-source.json+++ b/detections/fixtures/negative/ident-pass-the-ticket-anomalous-tgt-source.json@@ -1,5 +1,6 @@{- "event_id": 4625.0,+ "event_id": 4624,"logon_type": 3,- "ticket_source_ip_neq_session_ip": true+ "session_ip": "principal-a",+ "ticket_source_ip": "principal-a"}diff --git a/detections/fixtures/negative/linux-bashrc-modified-by-other-user.json b/detections/fixtures/negative/linux-bashrc-modified-by-other-user.jsonindex e34b5498c..910a026f5 100644--- a/detections/fixtures/negative/linux-bashrc-modified-by-other-user.json+++ b/detections/fixtures/negative/linux-bashrc-modified-by-other-user.json@@ -1,5 +1,6 @@{- "actor_uid_neq_owner_uid": true,+ "actor_uid": 1000,+ "owner_uid": 1000,"path": "head-/.bashrc",- "syscall": "not-openat"+ "syscall": "openat"}diff --git a/detections/fixtures/negative/linux-docker-socket-read-non-root.json b/detections/fixtures/negative/linux-docker-socket-read-non-root.jsonindex 87b150181..1e462f562 100644--- a/detections/fixtures/negative/linux-docker-socket-read-non-root.json+++ b/detections/fixtures/negative/linux-docker-socket-read-non-root.json@@ -1,5 +1,5 @@{- "actor_uid_neq": 0,+ "actor_uid": 0,"path": "/var/run/docker.sock",- "syscall": "not-connect"+ "syscall": "connect"}diff --git a/detections/fixtures/negative/linux-ebpf-prog-load-non-root.json b/detections/fixtures/negative/linux-ebpf-prog-load-non-root.jsonindex 14dd14254..f347f8d2b 100644--- a/detections/fixtures/negative/linux-ebpf-prog-load-non-root.json+++ b/detections/fixtures/negative/linux-ebpf-prog-load-non-root.json@@ -1,5 +1,5 @@{- "actor_uid_neq": 0,+ "actor_uid": 0,"cmd": "BPF_PROG_LOAD",- "syscall": "not-bpf"+ "syscall": "bpf"}diff --git a/detections/fixtures/negative/linux-shadow-read.json b/detections/fixtures/negative/linux-shadow-read.jsonindex 5bd78ec4d..b5e892479 100644--- a/detections/fixtures/negative/linux-shadow-read.json+++ b/detections/fixtures/negative/linux-shadow-read.json@@ -1,5 +1,5 @@{- "actor_uid_neq": 0,+ "actor_uid": 0,"path": "/etc/shadow",- "syscall": "not-openat"+ "syscall": "openat"}diff --git a/detections/fixtures/negative/linux-zshrc-modified-by-other-user.json b/detections/fixtures/negative/linux-zshrc-modified-by-other-user.jsonindex 0d89eaefe..aed53f794 100644--- a/detections/fixtures/negative/linux-zshrc-modified-by-other-user.json+++ b/detections/fixtures/negative/linux-zshrc-modified-by-other-user.json@@ -1,5 +1,6 @@{- "actor_uid_neq_owner_uid": true,+ "actor_uid": 1000,+ "owner_uid": 1000,"path": "head-/.zshrc",- "syscall": "not-openat"+ "syscall": "openat"}diff --git a/detections/fixtures/negative/macos-keychain-db-read-by-other-uid.json b/detections/fixtures/negative/macos-keychain-db-read-by-other-uid.jsonindex d8761b95a..f367c6035 100644--- a/detections/fixtures/negative/macos-keychain-db-read-by-other-uid.json+++ b/detections/fixtures/negative/macos-keychain-db-read-by-other-uid.json@@ -1,5 +1,6 @@{- "actor_uid_neq_owner_uid": true,- "event_type": "not-ES_EVENT_TYPE_NOTIFY_OPEN",+ "actor_uid": 1000,+ "event_type": "ES_EVENT_TYPE_NOTIFY_OPEN",+ "owner_uid": 1000,"path": "head-/Keychains/login.keychain-db"}diff --git a/detections/fixtures/negative/net-smb-write-sysvol.json b/detections/fixtures/negative/net-smb-write-sysvol.jsonindex f1f3bdb79..baa02beb8 100644--- a/detections/fixtures/negative/net-smb-write-sysvol.json+++ b/detections/fixtures/negative/net-smb-write-sysvol.json@@ -1,5 +1,5 @@{- "event_type": "not-smb_write",+ "event_type": "smb_write","share_name": "SYSVOL",- "src_role_neq": "domain-controller"+ "src_role": "domain-controller"}diff --git a/detections/fixtures/positive/app-ci-workflow-modified-by-non-owner.json b/detections/fixtures/positive/app-ci-workflow-modified-by-non-owner.jsonindex 7ad6add15..a13d4c197 100644--- a/detections/fixtures/positive/app-ci-workflow-modified-by-non-owner.json+++ b/detections/fixtures/positive/app-ci-workflow-modified-by-non-owner.json@@ -1,5 +1,5 @@{- "approver_role_neq": "codeowner",+ "approver_role": "not-codeowner","event_type": "push","path": "prefix-.github/workflows/-suffix"}diff --git a/detections/fixtures/positive/aws-iam-access-key-created-other-user.json b/detections/fixtures/positive/aws-iam-access-key-created-other-user.jsonindex 2a0d5ef0e..2b48bc433 100644--- a/detections/fixtures/positive/aws-iam-access-key-created-other-user.json+++ b/detections/fixtures/positive/aws-iam-access-key-created-other-user.json@@ -1,4 +1,5 @@{- "actor_eq_target": false,- "event_name": "CreateAccessKey"+ "actor": "principal-a",+ "event_name": "CreateAccessKey",+ "target": "principal-b"}diff --git a/detections/fixtures/positive/cloud-multi-cross-cloud-mass-egress.json b/detections/fixtures/positive/cloud-multi-cross-cloud-mass-egress.jsonindex 0d9302d77..9491b0986 100644--- a/detections/fixtures/positive/cloud-multi-cross-cloud-mass-egress.json+++ b/detections/fixtures/positive/cloud-multi-cross-cloud-mass-egress.json@@ -1,6 +1,5 @@{"bytes_out_per_minute": 1073741924.0,- "dst_cloud": "aws",- "src_cloud": "aws",- "src_cloud_neq_dst_cloud": true+ "dst_cloud": "azure",+ "src_cloud": "aws"}diff --git a/detections/fixtures/positive/cloud-org-rootmail-modified.json b/detections/fixtures/positive/cloud-org-rootmail-modified.jsonindex 61ebc117d..7116bdf7f 100644--- a/detections/fixtures/positive/cloud-org-rootmail-modified.json+++ b/detections/fixtures/positive/cloud-org-rootmail-modified.json@@ -1,4 +1,5 @@{- "actor_eq_target": false,- "event_name": "UpdateAccountEmail"+ "actor": "principal-a",+ "event_name": "UpdateAccountEmail",+ "target": "principal-b"}diff --git a/detections/fixtures/positive/ident-ntlm-relay-indicator.json b/detections/fixtures/positive/ident-ntlm-relay-indicator.jsonindex d38a7a81b..3a991ee53 100644--- a/detections/fixtures/positive/ident-ntlm-relay-indicator.json+++ b/detections/fixtures/positive/ident-ntlm-relay-indicator.json@@ -1,5 +1,6 @@{"auth_type": "NTLMv2","event_id": 8004,- "workstation_neq_source_address": true+ "source_address": "principal-b",+ "workstation": "principal-a"}diff --git a/detections/fixtures/positive/ident-pass-the-ticket-anomalous-tgt-source.json b/detections/fixtures/positive/ident-pass-the-ticket-anomalous-tgt-source.jsonindex 70f12733f..ad3e3a808 100644--- a/detections/fixtures/positive/ident-pass-the-ticket-anomalous-tgt-source.json+++ b/detections/fixtures/positive/ident-pass-the-ticket-anomalous-tgt-source.json@@ -1,5 +1,6 @@{"event_id": 4624,"logon_type": 3,- "ticket_source_ip_neq_session_ip": true+ "session_ip": "principal-b",+ "ticket_source_ip": "principal-a"}diff --git a/detections/fixtures/positive/linux-bashrc-modified-by-other-user.json b/detections/fixtures/positive/linux-bashrc-modified-by-other-user.jsonindex 2d14330db..273e72cd4 100644--- a/detections/fixtures/positive/linux-bashrc-modified-by-other-user.json+++ b/detections/fixtures/positive/linux-bashrc-modified-by-other-user.json@@ -1,5 +1,6 @@{- "actor_uid_neq_owner_uid": true,+ "actor_uid": 1000,+ "owner_uid": 0,"path": "head-/.bashrc","syscall": "openat"}diff --git a/detections/fixtures/positive/linux-docker-socket-read-non-root.json b/detections/fixtures/positive/linux-docker-socket-read-non-root.jsonindex 1301c48c7..54b2d51ea 100644--- a/detections/fixtures/positive/linux-docker-socket-read-non-root.json+++ b/detections/fixtures/positive/linux-docker-socket-read-non-root.json@@ -1,5 +1,5 @@{- "actor_uid_neq": 0,+ "actor_uid": 1000,"path": "/var/run/docker.sock","syscall": "connect"}diff --git a/detections/fixtures/positive/linux-ebpf-prog-load-non-root.json b/detections/fixtures/positive/linux-ebpf-prog-load-non-root.jsonindex 3e45cd209..a53f48c62 100644--- a/detections/fixtures/positive/linux-ebpf-prog-load-non-root.json+++ b/detections/fixtures/positive/linux-ebpf-prog-load-non-root.json@@ -1,5 +1,5 @@{- "actor_uid_neq": 0,+ "actor_uid": 1000,"cmd": "BPF_PROG_LOAD","syscall": "bpf"}diff --git a/detections/fixtures/positive/linux-shadow-read.json b/detections/fixtures/positive/linux-shadow-read.jsonindex 62a48ea1c..d9b091792 100644--- a/detections/fixtures/positive/linux-shadow-read.json+++ b/detections/fixtures/positive/linux-shadow-read.json@@ -1,5 +1,5 @@{- "actor_uid_neq": 0,+ "actor_uid": 1000,"path": "/etc/shadow","syscall": "openat"}diff --git a/detections/fixtures/positive/linux-zshrc-modified-by-other-user.json b/detections/fixtures/positive/linux-zshrc-modified-by-other-user.jsonindex 7e2179e0e..77a22e6e3 100644--- a/detections/fixtures/positive/linux-zshrc-modified-by-other-user.json+++ b/detections/fixtures/positive/linux-zshrc-modified-by-other-user.json@@ -1,5 +1,6 @@{- "actor_uid_neq_owner_uid": true,+ "actor_uid": 1000,+ "owner_uid": 0,"path": "head-/.zshrc","syscall": "openat"}diff --git a/detections/fixtures/positive/macos-keychain-db-read-by-other-uid.json b/detections/fixtures/positive/macos-keychain-db-read-by-other-uid.jsonindex 4dab867e5..5224f3659 100644--- a/detections/fixtures/positive/macos-keychain-db-read-by-other-uid.json+++ b/detections/fixtures/positive/macos-keychain-db-read-by-other-uid.json@@ -1,5 +1,6 @@{- "actor_uid_neq_owner_uid": true,+ "actor_uid": 1000,"event_type": "ES_EVENT_TYPE_NOTIFY_OPEN",+ "owner_uid": 0,"path": "head-/Keychains/login.keychain-db"}diff --git a/detections/fixtures/positive/net-smb-write-sysvol.json b/detections/fixtures/positive/net-smb-write-sysvol.jsonindex f96ebf332..cb7f58f13 100644--- a/detections/fixtures/positive/net-smb-write-sysvol.json+++ b/detections/fixtures/positive/net-smb-write-sysvol.json@@ -1,5 +1,5 @@{"event_type": "smb_write","share_name": "SYSVOL",- "src_role_neq": "domain-controller"+ "src_role": "not-domain-controller"}diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.mdindex 5cbe7b8c8..290d1fdec 100644--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md@@ -15,7 +15,7 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`| Connectors: live Test connection | README L168 | `ci.yml` connectors matrix (`test_conformance.py` — every connector implements the async `test_connection` contract) + `connector_conformance.py --check` published matrix + `test_live_vendor_smoke.py` (Phase D3 — mock-server conformance: `test_connection()` + paginated `fetch_alerts()` HTTP path exercised against realistic vendor payloads) | GATED | - || Investigation Ledger stores every step | README L61, L169 | `ci.yml :: api tests` (`audit_hash`, audit immutability) | PARTIAL (write path gated; UI replay only in hermetic e2e) | Phase 3.2 || Public eval harness gates every PR | README L62, L77 | `ci.yml :: p1-eval` | GATED (but suites are self-consistency; see reality report) | Phase 4 |-| Alert-reduction is a real measurement | README L62 | `ci.yml :: p1-eval` (`alert_reduction`) | PARTIAL (gates an in-test fusion re-impl, not `services/fusion`) | Phase 4 |+| Alert-reduction is a real measurement | README / benchmark page | `ci.yml :: python-test` (fusion) runs `test_alert_reduction_real.py`, which groups a 1 000-alert stream with `RawAlert.correlation_key()` — the method `Correlator` actually calls — and bounds the ratio on both sides, because a floor alone is satisfied by a key that collapses everything into one incident | GATED | the legacy suite's 75.3% stays published for continuity and is labelled as not describing this product; the measurement is still a synthetic workload, as the benchmark page states || Runs entirely on your infrastructure / no data exfiltration | README L63 | `ci.yml :: python-test` (agents) runs `test_privacy_redactor.py` (zero raw PII survives) | PARTIAL (redaction gated; the CLI now has no default telemetry endpoint and `telemetry.test.ts` pins that an enabled-but-unconfigured run makes zero network calls; platform-wide air-gapped egress-blocked CI + Helm NetworkPolicy still outstanding) | Phase 2 || Detection-as-Code rejects candidates that regress MITRE accuracy | README L170 | `ci.yml :: python-test` (`test_detection_eval.py` — candidate `rule_body` run through the real engine vs its own positive/negative fixtures; approval requires it) + `p1-eval` w2-dac baseline | GATED | - || 800+ native detection rules | README L78, L170 | `validate-detections.yml` (strict fixture replay) | GATED | - |@@ -113,8 +113,8 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`## Summary-- GATED: 92-- PARTIAL: 10+- GATED: 93+- PARTIAL: 9- NO GATE: 0 (**every claim is backed by a failing test.** The last NO GATE — the weekly benchmark scoreboard running live against `main` — closed in Phase E1: `scripts/check_scoreboard.py` ties the published scoreboard to a deterministic per-PR live-agent MITRE-accuracy run, while the funded weekly `wet-eval.yml` appends the LLM-tier rows. The remaining PARTIAL rows are honest, named deferrals — each states the specific gap and what closes it — not unproven claims.)> **Counting note.** These figures previously read 46 / 9, which did not matchdiff --git a/docs/openapi.yaml b/docs/openapi.yamlindex aaad54416..fe751b973 100644--- a/docs/openapi.yaml+++ b/docs/openapi.yaml@@ -6356,6 +6356,46 @@ paths:$ref: '#/components/schemas/PipelineHealth'security:- HTTPBearer: []+ /api/v1/health/fleet:+ get:+ tags:+ - health+ summary: Get Fleet Health+ description: 'Which connectors have quietly stopped working.+++ Every field this reads was already being written — `last_sync`,++ `error_count`, `oauth_refresh_failures`, `last_schema_drift_at` — and++ nothing read them together. That produces the failure this platform is++ least able to tolerate: a connector stops polling, alerts from that++ source stop arriving, and the console looks calm, because an absence of++ alerts is indistinguishable from an absence of threats.+++ Staleness is judged per connector against its own configured cadence.++ A single global threshold would page constantly on daily connectors or++ stay silent on five-minute ones, and a surface that pages constantly is++ a surface that gets muted.'+ operationId: get_fleet_health_api_v1_health_fleet_get+ responses:+ '200':+ description: Successful Response+ content:+ application/json:+ schema:+ additionalProperties: true+ type: object+ title: Response Get Fleet Health Api V1 Health Fleet Get+ security:+ - HTTPBearer: []/api/v1/insights/soc:get:tags:diff --git a/infra/docker/grafana/dashboards/aisoc-platform.json b/infra/docker/grafana/dashboards/aisoc-platform.jsonnew file mode 100644index 000000000..d0ca55df1--- /dev/null+++ b/infra/docker/grafana/dashboards/aisoc-platform.json@@ -0,0 +1,200 @@+{+ "uid": "aisoc-platform",+ "title": "AiSOC — Platform",+ "tags": ["aisoc"],+ "timezone": "browser",+ "schemaVersion": 39,+ "version": 1,+ "refresh": "30s",+ "time": { "from": "now-6h", "to": "now" },+ "description": "Every panel here queries a metric some service actually emits. Panels for metrics nothing exports render as 'No data' forever, which teaches people that empty panels are normal — and then a genuinely dead service looks the same as an uninstrumented one. Coverage is 5 of 19 services; `scripts/audit_prometheus_targets.py` names the other 14.",+ "panels": [+ {+ "type": "row",+ "title": "API",+ "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },+ "collapsed": false+ },+ {+ "type": "timeseries",+ "title": "Request rate by status",+ "description": "Split by status rather than totalled: a flat request rate with a rising 5xx share looks identical to a healthy one on a single line.",+ "datasource": { "type": "prometheus", "uid": "aisoc-prometheus" },+ "gridPos": { "h": 8, "w": 12, "x": 0, "y": 1 },+ "targets": [+ {+ "expr": "sum by (status) (rate(aisoc_http_requests_total[5m]))",+ "legendFormat": "{{status}}"+ }+ ],+ "fieldConfig": {+ "defaults": { "unit": "reqps", "custom": { "fillOpacity": 10 } }+ }+ },+ {+ "type": "timeseries",+ "title": "Request latency (p50 / p95 / p99)",+ "description": "Three quantiles, not a mean. A mean hides the tail, and the tail is what an analyst waiting on an investigation experiences.",+ "datasource": { "type": "prometheus", "uid": "aisoc-prometheus" },+ "gridPos": { "h": 8, "w": 12, "x": 12, "y": 1 },+ "targets": [+ {+ "expr": "histogram_quantile(0.50, sum by (le) (rate(aisoc_http_request_duration_seconds_bucket[5m])))",+ "legendFormat": "p50"+ },+ {+ "expr": "histogram_quantile(0.95, sum by (le) (rate(aisoc_http_request_duration_seconds_bucket[5m])))",+ "legendFormat": "p95"+ },+ {+ "expr": "histogram_quantile(0.99, sum by (le) (rate(aisoc_http_request_duration_seconds_bucket[5m])))",+ "legendFormat": "p99"+ }+ ],+ "fieldConfig": { "defaults": { "unit": "s" } }+ },+ {+ "type": "stat",+ "title": "Error ratio (5m)",+ "description": "5xx as a share of all requests. Absolute error counts scale with traffic, so a ratio is the thing worth alerting on.",+ "datasource": { "type": "prometheus", "uid": "aisoc-prometheus" },+ "gridPos": { "h": 5, "w": 6, "x": 0, "y": 9 },+ "targets": [+ {+ "expr": "sum(rate(aisoc_http_requests_total{status=~\"5..\"}[5m])) / clamp_min(sum(rate(aisoc_http_requests_total[5m])), 0.001)"+ }+ ],+ "fieldConfig": {+ "defaults": {+ "unit": "percentunit",+ "thresholds": {+ "mode": "absolute",+ "steps": [+ { "color": "green", "value": null },+ { "color": "yellow", "value": 0.01 },+ { "color": "red", "value": 0.05 }+ ]+ }+ }+ }+ },+ {+ "type": "row",+ "title": "Ingest",+ "gridPos": { "h": 1, "w": 24, "x": 0, "y": 14 },+ "collapsed": false+ },+ {+ "type": "timeseries",+ "title": "Inbox events accepted",+ "datasource": { "type": "prometheus", "uid": "aisoc-prometheus" },+ "gridPos": { "h": 8, "w": 12, "x": 0, "y": 15 },+ "targets": [+ {+ "expr": "sum by (tenant) (rate(aisoc_ingest_inbox_events_total[5m]))",+ "legendFormat": "{{tenant}}"+ }+ ],+ "fieldConfig": { "defaults": { "unit": "eps" } }+ },+ {+ "type": "timeseries",+ "title": "Inbox requests rate-limited",+ "description": "Non-zero is not necessarily wrong — the limiter exists to be hit by a misconfigured client. A sustained rate against one tenant means their integration is retrying into a wall and their events are being dropped.",+ "datasource": { "type": "prometheus", "uid": "aisoc-prometheus" },+ "gridPos": { "h": 8, "w": 12, "x": 12, "y": 15 },+ "targets": [+ {+ "expr": "sum by (tenant) (rate(aisoc_ingest_inbox_rate_limited_total[5m]))",+ "legendFormat": "{{tenant}}"+ }+ ],+ "fieldConfig": { "defaults": { "unit": "reqps" } }+ },+ {+ "type": "row",+ "title": "Auto-triage",+ "gridPos": { "h": 1, "w": 24, "x": 0, "y": 23 },+ "collapsed": false+ },+ {+ "type": "timeseries",+ "title": "Triage outcomes",+ "description": "The agents worker's own counters. Watch two shapes: dead_lettered rising means alerts are being dropped rather than triaged, and deterministic climbing while llm stays flat means the LLM path is failing open to the fallback — which looks fine from outside, because alerts still get verdicts.",+ "datasource": { "type": "prometheus", "uid": "aisoc-prometheus" },+ "gridPos": { "h": 8, "w": 12, "x": 0, "y": 24 },+ "targets": [+ {+ "expr": "sum by (result) (rate(aisoc_agents_triage_total[15m]))",+ "legendFormat": "{{result}}"+ }+ ]+ },+ {+ "type": "timeseries",+ "title": "LLM vs deterministic share",+ "description": "The ratio, because the absolute counts move with alert volume and the ratio does not. A fall here is the LLM path degrading.",+ "datasource": { "type": "prometheus", "uid": "aisoc-prometheus" },+ "gridPos": { "h": 8, "w": 12, "x": 12, "y": 24 },+ "targets": [+ {+ "expr": "sum(rate(aisoc_agents_triage_total{result=\"llm\"}[15m])) / clamp_min(sum(rate(aisoc_agents_triage_total{result=~\"llm|deterministic\"}[15m])), 0.001)",+ "legendFormat": "LLM share"+ }+ ],+ "fieldConfig": { "defaults": { "unit": "percentunit", "max": 1, "min": 0 } }+ },+ {+ "type": "stat",+ "title": "Dead-lettered (1h)",+ "description": "Alerts the worker could not process. Any sustained non-zero is alert loss.",+ "datasource": { "type": "prometheus", "uid": "aisoc-prometheus" },+ "gridPos": { "h": 5, "w": 6, "x": 0, "y": 32 },+ "targets": [+ { "expr": "sum(increase(aisoc_agents_triage_total{result=\"dead_lettered\"}[1h]))" }+ ],+ "fieldConfig": {+ "defaults": {+ "thresholds": {+ "mode": "absolute",+ "steps": [+ { "color": "green", "value": null },+ { "color": "red", "value": 1 }+ ]+ }+ }+ }+ },+ {+ "type": "stat",+ "title": "Ungrounded demotions (1h)",+ "description": "Verdicts demoted to human review because they cited indicators the evidence did not contain. A rise is a prompt or model regression. A persistent zero is not automatically good — check that groundedness is scoring at all.",+ "datasource": { "type": "prometheus", "uid": "aisoc-prometheus" },+ "gridPos": { "h": 5, "w": 6, "x": 6, "y": 32 },+ "targets": [+ {+ "expr": "sum(increase(aisoc_agents_triage_total{result=\"ungrounded_demoted\"}[1h]))"+ }+ ]+ },+ {+ "type": "row",+ "title": "Threat intel",+ "gridPos": { "h": 1, "w": 24, "x": 0, "y": 37 },+ "collapsed": false+ },+ {+ "type": "timeseries",+ "title": "IOCs and actors ingested",+ "datasource": { "type": "prometheus", "uid": "aisoc-prometheus" },+ "gridPos": { "h": 8, "w": 24, "x": 0, "y": 38 },+ "targets": [+ { "expr": "rate(threatintel_iocs_ingested_total[15m])", "legendFormat": "IOCs" },+ {+ "expr": "rate(threatintel_actors_ingested_total[15m])",+ "legendFormat": "actors"+ }+ ]+ }+ ]+}diff --git a/infra/docker/grafana/dashboards/dashboards.yaml b/infra/docker/grafana/dashboards/dashboards.yamlnew file mode 100644index 000000000..90c4019bb--- /dev/null+++ b/infra/docker/grafana/dashboards/dashboards.yaml@@ -0,0 +1,21 @@+# Grafana dashboard provisioning.+#+# Dashboards are files in this directory rather than state inside Grafana's+# database, so a rebuilt container comes back with them and a change to one+# is reviewable in a diff. `allowUiUpdates: false` makes that stick: a+# dashboard edited in the browser is silently reverted on the next reload,+# which is the correct outcome but surprises people who do not know the+# dashboards are provisioned.+apiVersion: 1++providers:+ - name: aisoc+ orgId: 1+ folder: AiSOC+ type: file+ disableDeletion: false+ allowUiUpdates: false+ updateIntervalSeconds: 30+ options:+ path: /etc/grafana/provisioning/dashboards+ foldersFromFilesStructure: falsediff --git a/infra/docker/grafana/datasources/prometheus.yaml b/infra/docker/grafana/datasources/prometheus.yamlnew file mode 100644index 000000000..014f54339--- /dev/null+++ b/infra/docker/grafana/datasources/prometheus.yaml@@ -0,0 +1,24 @@+# Prometheus datasource.+#+# The Tempo datasource already referenced `aisoc-prometheus` for its+# trace-to-metrics and service-map links, and no datasource declared that+# uid — so both links resolved to nothing. Grafana does not warn about a+# dangling datasource reference; the feature simply does nothing when+# clicked, which reads as "tracing is not wired up" rather than as a typo.+apiVersion: 1++datasources:+ - name: Prometheus+ type: prometheus+ uid: aisoc-prometheus+ access: proxy+ url: http://prometheus:9090+ isDefault: true+ jsonData:+ # Matches the scrape_interval in infra/docker/prometheus.yml. A+ # mismatch makes rate() windows that look fine and quietly under- or+ # over-sample.+ timeInterval: 15s+ exemplarTraceIdDestinations:+ - name: trace_id+ datasourceUid: aisoc-tempodiff --git a/scripts/check_action_contract.py b/scripts/check_action_contract.pyindex d9d702b46..cc7012e91 100644--- a/scripts/check_action_contract.py+++ b/scripts/check_action_contract.py@@ -194,23 +194,22 @@ def check_approval_matrix() -> list[str]:#: Ordered by what it costs to be missing:#: restore_file — the reverse of quarantine_file; a quarantined file#: cannot be released through the platform that took it-#: allow_domain — the reverse of block_domain, whose forward arm is-#: allow_hash itself a placeholder-#: allow_ioc — the reverse of block_ioc, which Defender does implement-#: block_hash — declared as allow_hash's reverse, never built-#: enable_user — the reverse of disable_user; an account disabled in an-#: incident has no platform route back-#: revoke_session, block_user_signin — identity verbs with no arm+#: allow_hash — the reverse of block_hash and vice versa; neither arm+#: allow_ioc exists, and allow_ioc reverses a block Defender does+#: block_hash implement+#: block_user_signin — an identity verb with no vendor arm#: push_case, push_status — ITSM sync, never wired+#:+#: Left this set when their clients turned out to already implement them:+#: enable_user (Entra, Google Workspace), revoke_session (both),+#: allow_domain (Cloudflare), allow_ip (Cloudflare, PAN-OS, FortiGate),+#: unisolate_host (CrowdStrike, Defender, SentinelOne).KNOWN_ORPHANS: frozenset[str] = frozenset({"restore_file",- "allow_domain","allow_hash","allow_ioc","block_hash",- "enable_user",- "revoke_session","block_user_signin","push_case","push_status",diff --git a/scripts/check_detection_fields.py b/scripts/check_detection_fields.pyindex 5a577ab04..4846b8034 100644--- a/scripts/check_detection_fields.py+++ b/scripts/check_detection_fields.py@@ -56,7 +56,7 @@ NORMALIZER = ROOT / "services" / "ingest" / "internal" / "normalizer" / "normali#: Current number of engine-loaded rules that depend on a DERIVED or STATEFUL#: field. This may only ever decrease. Lower it in the same PR that fixes#: rules; never raise it to make a red build green.-MAX_UNREACHABLE = 138+MAX_UNREACHABLE = 133#: Fields that no telemetry carries because they are computed, not observed:#: sliding-window counters, allowlist membership, privilege flags, and@@ -152,6 +152,66 @@ def _rule_fields(match_when: dict) -> set[str]:return found+#: Fields the engine derives at match time from an event it already has.+#: `services/fusion/app/services/derived_fields.py` computes them, so a rule+#: matching on one is reachable even though no connector emits it.+#:+#: Named here rather than imported, for the reason the rest of this file is+#: parsed rather than imported: a field-coverage gate that needs the fusion+#: package installed is a gate that gets disabled the first time an import+#: breaks.+_DERIVED_TIME_FIELDS = frozenset({"is_business_hours", "is_after_hours", "is_weekend"})++#: `<left>_eq_<right>` / `<left>_neq_<right>`, resolved by comparing two+#: fields of the same event. Reachable only when *both* sides are in the+#: namespace — a comparison against a field nothing emits is still dead, and+#: counting it as covered would be the fake-gate failure this file exists to+#: prevent.+_DERIVED_COMPARISON_RE = re.compile(r"^(?P<left>.+?)_(?:eq|neq)_(?P<right>.+)$")+++def _is_derivable(field: str, namespace: set[str]) -> bool:+ """True when the engine can compute this field from what it already has."""+ if field in _DERIVED_TIME_FIELDS:+ return True+ match = _DERIVED_COMPARISON_RE.match(field)+ if not match:+ return False+ return match.group("left") in namespace and match.group("right") in namespace+++#: What each unreachable field would need, so the headline number is+#: actionable rather than a bare count. "133 rules are dead" invites+#: someone to delete 133 rules; "73 need the windowed evaluator, 24 need+#: identity enrichment" names four pieces of work.+#:+#: Ordered: first match wins, so the more specific patterns come first.+_FIELD_FAMILIES: tuple[tuple[str, re.Pattern[str]], ...] = (+ (+ "windowed evaluator",+ re.compile(r"_count$|^count_|_per_|time_window|_window_|_5min|_ratio$"),+ ),+ ("identity enrichment", re.compile(r"_priv$|_is_admin$|_is_dc$")),+ ("per-tenant allowlist", re.compile(r"_in_allowlist$|_not_in_allowlist$")),+ ("age / first-seen enrichment", re.compile(r"_age_days$|_age_hours$|_seen_before$")),+ ("behavioural baseline", re.compile(r"_baseline|_deviation|^active_|_is_first_")),+ (+ # Both operands invented by the rule author. Distinct from the rest:+ # the engine can compute a comparison, it just cannot compare two+ # fields that do not exist. This is content work, not engine work.+ "comparison over fields nothing emits",+ _DERIVED_COMPARISON_RE,+ ),+)+++def _family(field: str) -> str:+ for name, pattern in _FIELD_FAMILIES:+ if pattern.search(field):+ return name+ return "other"++def _connector_namespace() -> set[str]:"""Every string-literal dict key appearing in a connector module.@@ -243,7 +303,7 @@ def scan() -> tuple[list[tuple[str, list[str]]], list[tuple[str, list[str]]], sederived: list[tuple[str, list[str]]] = []vendor: list[tuple[str, list[str]]] = []for rule in rules:- missing = {f for f in _rule_fields(rule.get("match_when") or {}) if f not in namespace}+ missing = {f for f in _rule_fields(rule.get("match_when") or {}) if f not in namespace and not _is_derivable(f, namespace)}if not missing:continuerule_id = str(rule.get("id", "?"))@@ -285,6 +345,12 @@ def main() -> int:print(f"detection fields: {total} engine rules examined")print(f" {len(derived)} depend on a derived/stateful field — cannot fire on any connector (gated)")+ families: dict[str, int] = {}+ for _rule_id, fields in derived:+ for name in {_family(f) for f in fields}:+ families[name] = families.get(name, 0) + 1+ for name, count in sorted(families.items(), key=lambda kv: -kv[1]):+ print(f" {count:4d} need {name}")print(f" {len(vendor)} reference a vendor-payload field — reachable when that vendor is attached")if len(derived) > args.max_unreachable:diff --git a/scripts/check_grafana_dashboards.py b/scripts/check_grafana_dashboards.pynew file mode 100644index 000000000..825fc5e05--- /dev/null+++ b/scripts/check_grafana_dashboards.py@@ -0,0 +1,176 @@+#!/usr/bin/env python3+"""Gate: every dashboard panel queries a metric some service emits.++A panel for a metric nothing exports renders "No data" forever. That is+worse than having no panel, because it teaches people that empty panels are+normal — and once they believe that, a genuinely dead service looks exactly+like an uninstrumented one.++Also checks datasource references resolve. Grafana does not warn about a+dangling datasource uid: the panel just shows nothing, and a trace-to-metrics+link silently does nothing when clicked, which reads as "tracing is not+wired up" rather than as a typo. The Tempo datasource shipped pointing at+`aisoc-prometheus` before any datasource declared that uid.++Run: python3 scripts/check_grafana_dashboards.py+"""++from __future__ import annotations++import argparse+import json+import re+import sys+from pathlib import Path++REPO_ROOT = Path(__file__).resolve().parent.parent+GRAFANA = REPO_ROOT / "infra" / "docker" / "grafana"+DASHBOARDS = GRAFANA / "dashboards"+DATASOURCES = GRAFANA / "datasources"+SERVICES = REPO_ROOT / "services"++#: Metric names a PromQL expression may reference. Recovered from the+#: expression rather than parsed properly: a full PromQL parser is a+#: dependency this gate does not need, and the names are the only part that+#: has to be right.+_METRIC_RE = re.compile(r"\b([a-z][a-z0-9_]*_(?:total|seconds|bytes|count|info))\b")++#: Prometheus appends these to a histogram or summary, so a dashboard+#: legitimately references a name no service literally declares.+_DERIVED_SUFFIXES = ("_bucket", "_sum", "_count")++#: Metrics exported by something other than our own code — the exporters+#: bundled into the compose stack. Listed so the exemption is reviewable+#: rather than implied by a loose regex.+KNOWN_EXTERNAL: frozenset[str] = frozenset(+ {+ "up",+ "scrape_duration_seconds",+ "process_cpu_seconds_total",+ "process_resident_memory_bytes",+ }+)+++def declared_metrics() -> set[str]:+ """Metric names declared anywhere under services/.++ A string-literal scan rather than an import: collecting metric names+ should not need every service's dependencies installed, and a gate+ that needs a database container is a gate that gets disabled.+ """+ names: set[str] = set()+ for path in SERVICES.rglob("*"):+ if path.suffix not in {".py", ".go", ".ts"} or not path.is_file():+ continue+ if "/tests/" in str(path) or "/node_modules/" in str(path):+ continue+ try:+ text = path.read_text(encoding="utf-8", errors="ignore")+ except OSError:+ continue+ names |= set(_METRIC_RE.findall(text))+ return names+++def dashboard_metrics(dashboard: dict) -> set[str]:+ names: set[str] = set()+ for panel in dashboard.get("panels", []):+ for target in panel.get("targets", []):+ expr = target.get("expr", "")+ names |= set(_METRIC_RE.findall(expr))+ # Histogram buckets: `foo_seconds_bucket` -> `foo_seconds`.+ names |= {m.removesuffix("_bucket") for m in re.findall(r"\b([a-z][a-z0-9_]*_bucket)\b", expr)}+ return names+++def declared_datasource_uids() -> set[str]:+ uids: set[str] = set()+ for path in DATASOURCES.glob("*.yaml"):+ for match in re.finditer(r"^\s*uid:\s*(\S+)\s*$", path.read_text(), re.M):+ uids.add(match.group(1))+ return uids+++def referenced_datasource_uids() -> set[str]:+ uids: set[str] = set()+ for path in DASHBOARDS.glob("*.json"):+ text = path.read_text(encoding="utf-8")+ uids |= set(re.findall(r'"uid"\s*:\s*"([^"]+)"', text))+ # Cross-datasource links (trace-to-metrics, exemplars) are references too.+ for path in DATASOURCES.glob("*.yaml"):+ for match in re.finditer(r"^\s*datasourceUid:\s*(\S+)\s*$", path.read_text(), re.M):+ uids.add(match.group(1))+ return uids+++def main(argv: list[str] | None = None) -> int:+ argparse.ArgumentParser(description=__doc__).parse_args(argv)++ if not DASHBOARDS.is_dir():+ print("grafana-dashboards: no dashboards directory; nothing to check")+ return 0++ files = sorted(DASHBOARDS.glob("*.json"))+ if not files:+ print(+ "grafana-dashboards: dashboards directory exists but is empty",+ file=sys.stderr,+ )+ return 1++ errors: list[str] = []+ exported = declared_metrics()++ for path in files:+ try:+ dashboard = json.loads(path.read_text(encoding="utf-8"))+ except json.JSONDecodeError as exc:+ errors.append(f"{path.name}: invalid JSON ({exc})")+ continue++ # A dashboard uid is what links and alerts reference; without one+ # Grafana assigns a random one per import and every link rots.+ if not dashboard.get("uid"):+ errors.append(f"{path.name}: no uid, so links to it cannot be stable")++ for metric in sorted(dashboard_metrics(dashboard)):+ if metric in KNOWN_EXTERNAL or metric in exported:+ continue+ if any(metric.removesuffix(suffix) in exported for suffix in _DERIVED_SUFFIXES):+ continue+ errors.append(+ f"{path.name}: panel queries {metric!r}, which no service "+ f"declares. The panel will read 'No data' forever, which "+ f"teaches people to ignore empty panels."+ )++ declared = declared_datasource_uids()+ for uid in sorted(referenced_datasource_uids()):+ # Panel-level uids are datasource uids; anything else in a dashboard+ # JSON matching "uid" is the dashboard's own, which is fine.+ if uid in declared:+ continue+ if any(json.loads(p.read_text()).get("uid") == uid for p in files):+ continue+ errors.append(+ f"datasource uid {uid!r} is referenced but no datasource declares "+ f"it. Grafana does not warn about this — the panel or link simply "+ f"does nothing."+ )++ if errors:+ print("GRAFANA DASHBOARD GATE FAILED:", file=sys.stderr)+ for error in errors:+ print(f" - {error}", file=sys.stderr)+ return 1++ print(+ f"grafana-dashboards: OK — {len(files)} dashboard(s), every panel "+ f"queries a metric a service emits and every datasource resolves"+ )+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/scripts/check_store_migrations.py b/scripts/check_store_migrations.pynew file mode 100644index 000000000..6376ceed2--- /dev/null+++ b/scripts/check_store_migrations.py@@ -0,0 +1,171 @@+#!/usr/bin/env python3+"""Gate: every store with schema has a migration runner, and it is called.++Three stores hold schema that code depends on — Neo4j, ClickHouse and+Qdrant — and none had a way to change it. `clickhouse/001_init.sql` is four+`CREATE TABLE IF NOT EXISTS` statements and zero `ALTER TABLE`;+`lake_writer` self-heals with the same idiom; `QdrantStore.initialize()`+creates collections if absent. All three are no-ops once the object exists,+so a schema change lands on a fresh deployment and silently does not land on+an existing one. Nothing notices until a query selects a column that is not+there, long after the deploy that was meant to add it.++The second half of this gate matters more than the first. v8.0 spent a+release discovering that **the mechanism existed, was unit-tested, and had+no caller on the path that needed it** — so a runner that exists and is+never invoked is not progress, it is the same defect with better test+coverage. This checks both.++Parsed rather than imported: the API package pulls in SQLAlchemy, asyncpg,+Neo4j and ClickHouse drivers, and a structural gate that needs a database+container is a gate that gets disabled.++Run: python3 scripts/check_store_migrations.py+"""++from __future__ import annotations++import argparse+import re+import sys+from pathlib import Path++REPO_ROOT = Path(__file__).resolve().parent.parent++#: store -> (runner module, the file that must call it, the symbol it calls)+#:+#: The call site is named explicitly rather than searched for, because "some+#: file somewhere imports it" is satisfied by a test, and a test is exactly+#: what stopped being evidence.+RUNNERS: dict[str, tuple[Path, Path, str]] = {+ "neo4j": (+ REPO_ROOT / "services/api/app/db/graph_migrations.py",+ REPO_ROOT / "services/api/app/db/neo4j.py",+ "run_migrations",+ ),+ "clickhouse": (+ REPO_ROOT / "services/api/app/db/lake_migrations.py",+ REPO_ROOT / "services/api/app/db/clickhouse.py",+ "run_migrations",+ ),+ "qdrant": (+ REPO_ROOT / "services/api/app/db/vector_migrations.py",+ REPO_ROOT / "services/threatintel/app/storage/qdrant.py",+ "run_migrations",+ ),+}++#: The startup path each caller must itself be reachable from. Without this+#: the chain stops one link short: a runner called by a function nobody+#: invokes is still a runner that never runs.+STARTUP_CALLERS: dict[str, tuple[Path, str]] = {+ "neo4j": (REPO_ROOT / "services/api/app/main.py", "init_neo4j"),+ "clickhouse": (REPO_ROOT / "services/api/app/main.py", "init_lake_schema"),+ "qdrant": (+ REPO_ROOT / "services/threatintel/app/storage/qdrant.py",+ "initialize",+ ),+}++#: A runner must expose these. Three stores, one vocabulary — an operator+#: should not need three mental models to answer "what has this applied".+REQUIRED_SYMBOLS = ("MIGRATIONS", "run_migrations", "applied_ids", "pending_ids")+++def _read(path: Path) -> str:+ return path.read_text(encoding="utf-8") if path.exists() else ""+++def check_runners_exist() -> list[str]:+ errors: list[str] = []+ for store, (runner, _, _) in RUNNERS.items():+ if not runner.exists():+ errors.append(+ f"{store}: no migration runner at "+ f"{runner.relative_to(REPO_ROOT)}. Its schema can only be "+ f"created, never changed, so a new column never reaches an "+ f"existing deployment."+ )+ continue+ source = _read(runner)+ for symbol in REQUIRED_SYMBOLS:+ if not re.search(rf"^(?:async def |def |){symbol}\b", source, re.M):+ errors.append(f"{store}: runner does not expose {symbol}; the three " f"runners are meant to share one interface")+ return errors+++def check_runners_are_called() -> list[str]:+ """The half that catches the v8.0 pattern."""+ errors: list[str] = []+ for store, (runner, caller, symbol) in RUNNERS.items():+ if not runner.exists():+ continue+ if not caller.exists():+ errors.append(f"{store}: expected call site {caller.relative_to(REPO_ROOT)} " f"does not exist")+ continue+ source = _read(caller)+ if symbol not in source:+ errors.append(+ f"{store}: {caller.relative_to(REPO_ROOT)} never calls "+ f"{symbol}(). The runner exists and nothing invokes it, which "+ f"is indistinguishable from not having one."+ )+ return errors+++def check_startup_reachability() -> list[str]:+ """A caller nobody invokes is one link short of the same problem."""+ errors: list[str] = []+ for store, (path, symbol) in STARTUP_CALLERS.items():+ source = _read(path)+ if not source:+ errors.append(f"{store}: {path.relative_to(REPO_ROOT)} does not exist")+ continue+ # Called, not merely defined or imported.+ if not re.search(rf"\b{symbol}\s*\(", source):+ errors.append(+ f"{store}: {symbol} is never invoked in " f"{path.relative_to(REPO_ROOT)}, so migrations do not run at " f"startup"+ )+ return errors+++def check_ids_are_append_only() -> list[str]:+ """Migration ids must be unique and ordered.++ The runners apply in tuple order, so a numbering that disagrees with+ that order means migrations run in a sequence the numbers do not+ describe — and migration N+1 is written against the state N produced.+ """+ errors: list[str] = []+ for store, (runner, _, _) in RUNNERS.items():+ source = _read(runner)+ if not source:+ continue+ ids = re.findall(r'^\s+id="([^"]+)"', source, re.M)+ if not ids:+ errors.append(f"{store}: no migrations declared; the runner is empty")+ continue+ if len(ids) != len(set(ids)):+ errors.append(f"{store}: duplicate migration ids")+ if ids != sorted(ids):+ errors.append(f"{store}: migration ids are not in order ({', '.join(ids)}); " f"the runner applies them in declaration order")+ return errors+++def main(argv: list[str] | None = None) -> int:+ argparse.ArgumentParser(description=__doc__).parse_args(argv)++ errors = check_runners_exist() + check_runners_are_called() + check_startup_reachability() + check_ids_are_append_only()++ if errors:+ print("STORE MIGRATION GATE FAILED:", file=sys.stderr)+ for error in errors:+ print(f" - {error}", file=sys.stderr)+ return 1++ print(f"store-migrations: OK — {len(RUNNERS)} stores have a runner, each " f"wired into a startup path")+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/scripts/detection_specs_part3_helpers.py b/scripts/detection_specs_part3_helpers.pyindex f744a0018..e888dbd61 100644--- a/scripts/detection_specs_part3_helpers.py+++ b/scripts/detection_specs_part3_helpers.py@@ -21,6 +21,7 @@ Auto-fixture rulesfrom __future__ import annotationsfrom collections.abc import Iterable+import refrom typing import Any# Operator suffixes - kept in sync with scripts/generate_detections.py.@@ -30,6 +31,11 @@ from typing import Any_OP_SUFFIXES: tuple[tuple[str, str], ...] = tuple(sorted([+ # `neq` was used by eleven rules and was not an operator, so+ # `actor_uid_neq: 0` was read as a field literally named+ # `actor_uid_neq` — and the synthesized fixture then contained+ # that field, so the rule matched itself and nothing else.+ ("_neq", "neq"),("_pattern_match_any", "pattern_match_any"),("_not_endswith_any", "not_endswith_any"),("_not_contains_any", "not_contains_any"),@@ -74,6 +80,15 @@ def _pos_for(op: str, expected: Any) -> Any:"""Return a value satisfying the (op, expected) clause."""if op == "eq":return expected+ if op == "neq":+ # Anything but `expected`. Numeric fields get a number rather than+ # the string sentinel: `actor_uid: "__benign__"` is not a uid, and+ # a fixture that could not exist is a fixture that tests nothing.+ if isinstance(expected, bool):+ return not expected+ if isinstance(expected, (int, float)):+ return expected + 1000+ return f"not-{expected}"if op == "in":return expected[0] if isinstance(expected, list) and expected else Noneif op == "not_in":@@ -129,6 +144,12 @@ def _pos_for(op: str, expected: Any) -> Any:def _neg_for(op: str, expected: Any) -> Any:"""Return a value that breaks the (op, expected) clause."""+ if op == "neq":+ # The inverse of neq is equality, so the negative carries the+ # excluded value. That is the clause the rule is *about*, which is+ # what a negative should invert — the generator used to flip an+ # unrelated field and never test the condition in either direction.+ return expectedif op == "eq":if expected is None:return _SENTINEL_STR@@ -301,6 +322,99 @@ def _compose_field_value(constraints: list[tuple[str, Any]]) -> Any:return f"{prefix}{middle}{suffix}"+#: `<left>_eq_<right>` / `<left>_neq_<right>` — the engine derives these by+#: comparing two fields of the same event, so a fixture needs both operands+#: rather than one field named after the comparison.+_COMPARISON_KEY_RE = re.compile(r"^(?P<left>.+?)_(?P<op>eq|neq)_(?P<right>.+)$")+++def _comparison_operands(key: str, want_value: bool) -> dict[str, Any] | None:+ """Two real fields whose comparison satisfies or breaks the clause.++ Returns None when `key` is not a comparison, so callers fall through to+ the ordinary operator path.+ """+ match = _COMPARISON_KEY_RE.match(key)+ if not match:+ return None+ left, right, op = match.group("left"), match.group("right"), match.group("op")+ # `want_value` is what the derived field should evaluate to, not+ # whether the clause is satisfied. Conflating the two made the negative+ # fixture identical to the positive for `actor_eq_target: false`: both+ # asked for "operands differ", so the rule matched its own negative.+ want_equal = (op == "eq") == want_value++ if left.endswith(("uid", "gid", "_id", "port")):+ return {left: 1000, right: 1000 if want_equal else 0}+ return {left: "principal-a", right: "principal-a" if want_equal else "principal-b"}+++def _alternative_value(field: str, constraints: list[tuple[str, Any]], avoid: Any) -> Any:+ """Another value for `field` that still satisfies its own clauses.++ Used when a comparison needs two operands to differ and both are+ already pinned by other clauses. Returns `avoid` unchanged when no+ alternative exists, so the caller can report rather than silently+ emit a fixture that cannot satisfy its rule.+ """+ for op, expected in constraints:+ if op in {"in", "match_any", "contains_any", "startswith_any", "endswith_any"} and isinstance(+ expected, list+ ):+ for candidate in expected:+ value = _pos_for(op, [candidate])+ if value != avoid:+ return value+ if isinstance(avoid, bool):+ return not avoid+ if isinstance(avoid, (int, float)):+ return avoid + 1+ if isinstance(avoid, str):+ return f"other-{avoid}"+ return avoid+++def _resolve_comparison(+ key: str,+ want_value: bool,+ pos: dict[str, Any],+ by_field: dict[str, list[tuple[str, Any]]],+) -> dict[str, Any]:+ """Operand values satisfying a comparison, respecting existing clauses."""+ match = _COMPARISON_KEY_RE.match(key)+ assert match is not None+ left, right, op = match.group("left"), match.group("right"), match.group("op")+ want_equal = (op == "eq") == want_value++ left_value = pos.get(left)+ right_value = pos.get(right)++ # Neither operand constrained elsewhere: pick a clean pair.+ if left_value is None and right_value is None:+ operands = _comparison_operands(key, want_value)+ return operands or {}++ if left_value is None:+ left_value = right_value if want_equal else _alternative_value(+ left, by_field.get(left, []), right_value+ )+ return {left: left_value}++ if right_value is None:+ right_value = left_value if want_equal else _alternative_value(+ right, by_field.get(right, []), left_value+ )+ return {right: right_value}++ # Both pinned. Adjust the right side only, so the left keeps whatever+ # the rule's other clauses asked for.+ if want_equal:+ return {right: left_value}+ if left_value != right_value:+ return {}+ return {right: _alternative_value(right, by_field.get(right, []), left_value)}++def build_positive(when: dict[str, Any]) -> dict[str, Any]:"""Build a synthetic positive fixture from a flat match_when.@@ -314,9 +428,14 @@ def build_positive(when: dict[str, Any]) -> dict[str, Any]:"Pass an explicit positive= to S().")by_field: dict[str, list[tuple[str, Any]]] = {}+ comparisons: list[tuple[str, bool]] = []for key, val in when.items():+ if _COMPARISON_KEY_RE.match(key):+ comparisons.append((key, bool(val)))+ continuefield, op = split_op(key)by_field.setdefault(field, []).append((op, val))+pos: dict[str, Any] = {}for field, constraints in by_field.items():if len(constraints) == 1:@@ -324,6 +443,15 @@ def build_positive(when: dict[str, Any]) -> dict[str, Any]:pos[field] = _pos_for(op, exp)else:pos[field] = _compose_field_value(constraints)++ # Comparisons resolve last, because an operand can also be constrained+ # by an ordinary clause. `src_cloud_in: [aws, azure, gcp]` alongside+ # `src_cloud_neq_dst_cloud: true` put "aws" in both fields and the+ # comparison then evaluated false — the rule's own fixture could not+ # satisfy the rule.+ for key, want_value in comparisons:+ pos.update(_resolve_comparison(key, want_value, pos, by_field))+return pos@@ -353,7 +481,40 @@ def build_negative(f"neg_field={neg_field!r} not found in match_when keys: {list(when.keys())}")else:- first_key = next(iter(when.keys()))+ # Prefer the clause the rule is *about*. Taking the first key made+ # every negative flip whatever happened to be written first — for+ # /etc/shadow-read-by-non-root that was `syscall`, so the negative+ # was "a syscall that does not exist" and the non-root condition+ # was never tested in either direction.+ #+ # A comparison or neq clause is the discriminating one by+ # construction: the other clauses say *where* to look, this one+ # says what makes it suspicious.+ comparison_key = next(+ (k for k in when if _COMPARISON_KEY_RE.match(k)), None+ )+ if comparison_key is not None:+ # Through the resolver, not the raw operand picker: the+ # operands may also be constrained by other clauses, and a+ # negative carrying a value the rule's own `_in` list forbids+ # is not a negative, it is a malformed event.+ by_field: dict[str, list[tuple[str, Any]]] = {}+ for key in when:+ if _COMPARISON_KEY_RE.match(key):+ continue+ field, op = split_op(key)+ by_field.setdefault(field, []).append((op, when[key]))+ neg.update(+ _resolve_comparison(+ comparison_key, not bool(when[comparison_key]), neg, by_field+ )+ )+ return neg++ neq_key = next(+ (k for k in when if split_op(k)[1] == "neq"), None+ )+ first_key = neq_key or next(iter(when.keys()))neg_field, target_op = split_op(first_key)target_expected = when[first_key]diff --git a/scripts/generate_detections.py b/scripts/generate_detections.pyindex ee0c0ba12..69a133ed0 100644--- a/scripts/generate_detections.py+++ b/scripts/generate_detections.py@@ -38,6 +38,7 @@ from __future__ import annotationsimport jsonimport re+from datetime import datetimeimport sysfrom pathlib import Pathfrom typing import Any@@ -89,6 +90,14 @@ OPERATORS: list[tuple[str, str, str]] = sorted(("_has_any", "has_any", "HAS_ANY"),("_not_in", "not_in", "NOT IN"),("_match", "match", "MATCH"),+ # `neq` is used by rules in the shipped corpus and had no+ # operator, so `approver_role_neq: "codeowner"` was read as a+ # field literally named `approver_role_neq` — which nothing+ # emits, so the rule could not fire. There is deliberately no+ # `_eq` counterpart: bare equality is already the default, and+ # adding the suffix would split any field whose name happens to+ # end in `_eq` for no gain.+ ("_neq", "neq", "!="),("_gte", "gte", ">="),("_lte", "lte", "<="),("_in", "in", "IN"),@@ -184,6 +193,13 @@ def _check(field: str, op: str, expected: Any, event: dict[str, Any]) -> bool:return actual is Nonereturn actual == expected+ if op == "neq":+ # A missing field is not "different from X". Returning True would+ # make every neq rule fire on every event lacking the field.+ if actual is None:+ return False+ return actual != expected+if op in {"gt", "gte", "lt", "lte"}:if not isinstance(actual, (int, float)) or isinstance(actual, bool):return False@@ -325,15 +341,11 @@ def _eval_clause(clause: dict[str, Any], event: dict[str, Any]) -> bool:"""Evaluate a clause dict (possibly nested) against an event."""for key, expected in clause.items():if key == "any_of":- if not isinstance(expected, list) or not any(- _eval_clause(sub, event) for sub in expected- ):+ if not isinstance(expected, list) or not any(_eval_clause(sub, event) for sub in expected):return Falsecontinueif key == "all_of":- if not isinstance(expected, list) or not all(- _eval_clause(sub, event) for sub in expected- ):+ if not isinstance(expected, list) or not all(_eval_clause(sub, event) for sub in expected):return Falsecontinuefield, op = _split_op(key)@@ -367,16 +379,8 @@ def _description_for(spec: dict, category: str) -> str:name = spec["name"]fp_count = len(spec.get("fp", []))plural = "s" if fp_count != 1 else ""- fp_clause = (- f" Watch the {fp_count} documented false-positive case{plural} "- f"before tuning."- if fp_count- else ""- )- return (- f"AiSOC v1 curated detection. Triggers on the {category} signal "- f"described by '{name}'.{fp_clause}"- )+ fp_clause = f" Watch the {fp_count} documented false-positive case{plural} " f"before tuning." if fp_count else ""+ return f"AiSOC v1 curated detection. Triggers on the {category} signal " f"described by '{name}'.{fp_clause}"def _yaml_safe_value(value: Any) -> Any:@@ -390,17 +394,13 @@ class _LiteralStr(str):def _literal_str_representer(dumper: yaml.Dumper, data: _LiteralStr): # type: ignore[name-defined]- return dumper.represent_scalar(- "tag:yaml.org,2002:str", str(data), style="|"- )+ return dumper.represent_scalar("tag:yaml.org,2002:str", str(data), style="|")yaml.add_representer(_LiteralStr, _literal_str_representer) # type: ignore[arg-type]-def render_rule_yaml(- *, rule_id: str, category: str, spec: dict-) -> str:+def render_rule_yaml(*, rule_id: str, category: str, spec: dict) -> str:"""Render the canonical YAML for one detection rule."""name: str = spec["name"]severity: str = spec["severity"]@@ -473,8 +473,7 @@ def load_id_lock() -> dict[str, str]:return dict(json.loads(ID_LOCK.read_text(encoding="utf-8")))except Exception: # noqa: BLE001 - a corrupt lock must not silently renumberraise SystemExit(- f"{ID_LOCK} is unreadable. Fix or delete it deliberately — "- f"regenerating without it reassigns every rule id."+ f"{ID_LOCK} is unreadable. Fix or delete it deliberately — " f"regenerating without it reassigns every rule id.") from None@@ -489,9 +488,7 @@ def assign_ids(categories: dict) -> tuple[dict[str, str], list[str]]:for category, specs in sorted(categories.items()):used = {- int(rid.rsplit("-", 1)[1])- for key, rid in lock.items()- if key.startswith(f"{category}/") and rid.rsplit("-", 1)[-1].isdigit()+ int(rid.rsplit("-", 1)[1]) for key, rid in lock.items() if key.startswith(f"{category}/") and rid.rsplit("-", 1)[-1].isdigit()}next_free = max(used) + 1 if used else 1@@ -508,13 +505,145 @@ def assign_ids(categories: dict) -> tuple[dict[str, str], list[str]]:return lock, newly+# ─── Derived fields ──────────────────────────────────────────────────────────+#+# Vendored into `services/fusion/app/services/derived_fields.py`, and kept in+# parity by `services/fusion/tests/test_detection_matcher_parity.py` — the+# same arrangement as `matches()` itself, for the same reason: services+# cannot import repo-root scripts at runtime, and two copies that can drift+# silently are worse than one copy plus a gate.+#+# These compute fields the engine can derive from an event it already has,+# so a rule matching `actor_eq_target` or `is_business_hours` is reachable+# even though no connector emits either.++_COMPARISON_RE = re.compile(r"^(?P<left>.+?)_(?P<op>eq|neq)_(?P<right>.+)$")++DEFAULT_BUSINESS_START_HOUR = 8+DEFAULT_BUSINESS_END_HOUR = 18++_TIME_FIELDS = ("event_time", "timestamp", "time", "@timestamp", "ingest_time")+++def _parse_event_time(event: dict[str, Any]) -> datetime | None:+ for field in _TIME_FIELDS:+ raw = event.get(field)+ if raw is None:+ continue+ if isinstance(raw, datetime):+ return raw+ if isinstance(raw, (int, float)) and not isinstance(raw, bool):+ try:+ seconds = raw / 1000 if raw > 1e11 else raw+ return datetime.fromtimestamp(seconds) # noqa: DTZ006+ except (OSError, ValueError, OverflowError):+ continue+ if isinstance(raw, str):+ try:+ return datetime.fromisoformat(raw.replace("Z", "+00:00"))+ except ValueError:+ continue+ return None+++def _normalise_operand(value: Any) -> Any:+ return value.strip().lower() if isinstance(value, str) else value+++def comparison_fields(event: dict[str, Any], requested: set[str] | None = None) -> dict[str, bool]:+ """`<a>_eq_<b>` / `<a>_neq_<b>` for the names some rule asks for.++ A missing operand yields no key rather than False: a rule must not fire+ because we did not know.+ """+ if not requested:+ return {}+ out: dict[str, bool] = {}+ for name in requested:+ match = _COMPARISON_RE.match(name)+ if not match:+ continue+ left = event.get(match.group("left"))+ right = event.get(match.group("right"))+ if left is None or right is None:+ continue+ equal = _normalise_operand(left) == _normalise_operand(right)+ out[name] = equal if match.group("op") == "eq" else not equal+ return out+++def time_of_day_fields(+ event: dict[str, Any],+ *,+ business_start: int = DEFAULT_BUSINESS_START_HOUR,+ business_end: int = DEFAULT_BUSINESS_END_HOUR,+) -> dict[str, bool]:+ """`is_business_hours` / `is_after_hours` / `is_weekend`.++ Nothing when the event has no parseable timestamp: an unknown time is+ not "outside business hours".+ """+ when = _parse_event_time(event)+ if when is None:+ return {}+ weekend = when.weekday() >= 5+ in_hours = (not weekend) and business_start <= when.hour < business_end+ return {+ "is_business_hours": in_hours,+ "is_after_hours": not in_hours,+ "is_weekend": weekend,+ }+++def enrich(+ event: dict[str, Any],+ requested: set[str] | None = None,+ *,+ business_start: int = DEFAULT_BUSINESS_START_HOUR,+ business_end: int = DEFAULT_BUSINESS_END_HOUR,+) -> dict[str, Any]:+ """`event` plus every derivable field the rules ask for.++ A derived key never overwrites one the connector supplied: a vendor's+ own answer about its own tenant's hours beats ours.+ """+ derived: dict[str, Any] = {}+ derived.update(time_of_day_fields(event, business_start=business_start, business_end=business_end))+ derived.update(comparison_fields(event, requested))+ return event if not derived else {**derived, **event}+++def requested_derived_fields(rules: list[dict[str, Any]]) -> set[str]:+ """Derivable field names the ruleset matches on.++ Computed once per ruleset rather than per event: the set changes when+ rules change, not when traffic arrives.+ """+ names: set[str] = set()++ def walk(clause: Any) -> None:+ if isinstance(clause, dict):+ for key, value in clause.items():+ if key in {"any_of", "all_of", "not"}:+ walk(value)+ continue+ field, _ = _split_op(key)+ if _COMPARISON_RE.match(field) or field.startswith("is_"):+ names.add(field)+ elif isinstance(clause, list):+ for item in clause:+ walk(item)++ for rule in rules:+ walk(rule.get("match_when") or {})+ return names++def write_pack(*, dry_run: bool = False) -> dict[str, int]:"""Generate the full pack to disk. Returns counts per category."""id_lock, newly_assigned = assign_ids(CATEGORIES)if newly_assigned and not dry_run:- ID_LOCK.write_text(- json.dumps(id_lock, indent=2, sort_keys=True) + "\n", encoding="utf-8"- )+ ID_LOCK.write_text(json.dumps(id_lock, indent=2, sort_keys=True) + "\n", encoding="utf-8")print(f" assigned {len(newly_assigned)} new rule id(s)")counts: dict[str, int] = {}@@ -532,9 +661,7 @@ def write_pack(*, dry_run: bool = False) -> dict[str, int]:# Looked up, not computed from position. See ID_LOCK.rule_id = id_lock[f"{category}/{slug}"]- yaml_text = render_rule_yaml(- rule_id=rule_id, category=category, spec=spec- )+ yaml_text = render_rule_yaml(rule_id=rule_id, category=category, spec=spec)yaml_path = cat_dir / f"{slug}.yaml"pos_path = pos_dir / f"{slug}.json"diff --git a/scripts/validate_detections.py b/scripts/validate_detections.pyindex 74c457cb9..9011cf87c 100755--- a/scripts/validate_detections.py+++ b/scripts/validate_detections.py@@ -59,7 +59,11 @@ if str(SCRIPTS_DIR) not in sys.path:sys.path.insert(0, str(SCRIPTS_DIR))from detection_specs_index import all_specs # noqa: E402-from generate_detections import matches # noqa: E402+from generate_detections import ( # noqa: E402+ enrich,+ matches,+ requested_derived_fields,+)VALID_SEVERITIES = {"low", "medium", "high", "critical"}VALID_CATEGORIES = {@@ -114,9 +118,7 @@ IMPORTED_TIERS: dict[str, dict[str, str]] = {# Spec lookup — for native fixture replay only.# =============================================================================-_SPEC_BY_KEY: dict[tuple[str, str], dict[str, Any]] = {- (cat, spec["slug"]): spec for cat, spec in all_specs()-}+_SPEC_BY_KEY: dict[tuple[str, str], dict[str, Any]] = {(cat, spec["slug"]): spec for cat, spec in all_specs()}# =============================================================================@@ -229,9 +231,7 @@ def validate_rule(return [f"YAML parse error: {exc}"], Noneif not isinstance(rule, dict):- return [- "Rule is not a YAML mapping (expected key: value pairs at top level)"- ], None+ return ["Rule is not a YAML mapping (expected key: value pairs at top level)"], Nonefor field in REQUIRED_FIELDS:if field not in rule:@@ -241,42 +241,23 @@ def validate_rule(return errors, ruleif rule["severity"] not in VALID_SEVERITIES:- errors.append(- f"Invalid severity '{rule['severity']}'; must be one of: "- f"{', '.join(sorted(VALID_SEVERITIES))}"- )+ errors.append(f"Invalid severity '{rule['severity']}'; must be one of: " f"{', '.join(sorted(VALID_SEVERITIES))}")rule_category = rule.get("category")if rule_category and rule_category not in VALID_CATEGORIES:- errors.append(- f"Invalid category '{rule_category}'; must be one of: "- f"{', '.join(sorted(VALID_CATEGORIES))}"- )+ errors.append(f"Invalid category '{rule_category}'; must be one of: " f"{', '.join(sorted(VALID_CATEGORIES))}")expected_category = classification["category"]- if (- rule_category- and expected_category- and rule_category != expected_category- ):- errors.append(- f"Rule category '{rule_category}' does not match directory "- f"'{expected_category}'"- )+ if rule_category and expected_category and rule_category != expected_category:+ errors.append(f"Rule category '{rule_category}' does not match directory " f"'{expected_category}'")rule_id = str(rule["id"])expected_prefix = classification["id_prefix"]if expected_prefix and not rule_id.startswith(expected_prefix):- errors.append(- f"Rule id '{rule_id}' must start with '{expected_prefix}' "- f"(tier: {classification['tier']})"- )+ errors.append(f"Rule id '{rule_id}' must start with '{expected_prefix}' " f"(tier: {classification['tier']})")if rule_id in seen_ids:- errors.append(- f"Duplicate id '{rule_id}' — already defined in "- f"{seen_ids[rule_id]}"- )+ errors.append(f"Duplicate id '{rule_id}' — already defined in " f"{seen_ids[rule_id]}")else:seen_ids[rule_id] = path@@ -299,10 +280,7 @@ def _validate_provenance(rule: dict[str, Any]) -> list[str]:errors: list[str] = []provenance = rule.get("provenance")if not isinstance(provenance, dict):- return [- "Imported rule is missing required 'provenance' block "- "(see tools/detection_import/README.md)"- ]+ return ["Imported rule is missing required 'provenance' block " "(see tools/detection_import/README.md)"]for field in REQUIRED_PROVENANCE_FIELDS:value = provenance.get(field)if value in (None, ""):@@ -310,9 +288,7 @@ def _validate_provenance(rule: dict[str, Any]) -> list[str]:return errors-def replay_fixture(- rule_path: Path, rule: dict[str, Any], strict: bool-) -> list[str]:+def replay_fixture(rule_path: Path, rule: dict[str, Any], strict: bool) -> list[str]:"""Replay positive + negative fixtures against the canonical spec.Native tier only. Looks up the spec for this rule by ``(category,@@ -335,13 +311,9 @@ def replay_fixture(if pos_missing or neg_missing:msg_parts = []if pos_missing:- msg_parts.append(- f"missing positive fixture {pos_path.relative_to(ROOT)}"- )+ msg_parts.append(f"missing positive fixture {pos_path.relative_to(ROOT)}")if neg_missing:- msg_parts.append(- f"missing negative fixture {neg_path.relative_to(ROOT)}"- )+ msg_parts.append(f"missing negative fixture {neg_path.relative_to(ROOT)}")msg = "; ".join(msg_parts)if strict:errors.append(msg)@@ -351,10 +323,7 @@ def replay_fixture(spec = _SPEC_BY_KEY.get((category, slug))if spec is None:- msg = (- f"no canonical spec found for ({category}, {slug}); "- f"hand-authored rule — fixture replay skipped"- )+ msg = f"no canonical spec found for ({category}, {slug}); " f"hand-authored rule — fixture replay skipped"errors.append(f"WARN: {msg}")return errors@@ -371,14 +340,18 @@ def replay_fixture(errors.append(f"fixture load error: {exc}")return errors+ # Enriched the way the engine enriches. Replaying against the bare+ # matcher tests a pipeline production does not run — a rule matching a+ # derived field would fail here while working live, which trains people+ # to weaken the gate.+ wanted = requested_derived_fields([{"match_when": match_when}])+ pos_event = enrich(pos_event, wanted)+ neg_event = enrich(neg_event, wanted)+if not matches(match_when, pos_event):- errors.append(- "positive fixture did NOT match match_when (expected match)"- )+ errors.append("positive fixture did NOT match match_when (expected match)")if matches(match_when, neg_event):- errors.append(- "negative fixture DID match match_when (expected no match)"- )+ errors.append("negative fixture DID match match_when (expected no match)")return errors@@ -432,12 +405,7 @@ def main() -> int:rule_failed = bool(errors)replay_errors: list[str] = []- if (- rule- and not rule_failed- and tier == "native"- and path.parent.name in VALID_CATEGORIES- ):+ if rule and not rule_failed and tier == "native" and path.parent.name in VALID_CATEGORIES:replay_errors = replay_fixture(path, rule, strict=strict)warnings = [e for e in replay_errors if e.startswith("WARN:")]@@ -460,18 +428,8 @@ def main() -> int:print(f" {w}")print(f"\n{'─' * 60}")- print(- f"Validated {total} rules — {total - failed} passed, {failed} failed, "- f"{fixture_warnings} fixture warnings"- )- print(- " Tiers: "- + ", ".join(- f"{tier}={count}"- for tier, count in sorted(tier_counts.items())- if count > 0- )- )+ print(f"Validated {total} rules — {total - failed} passed, {failed} failed, " f"{fixture_warnings} fixture warnings")+ print(" Tiers: " + ", ".join(f"{tier}={count}" for tier, count in sorted(tier_counts.items()) if count > 0))if quarantine_count:print(f" Quarantined (parsed-but-disabled): {quarantine_count}")diff --git a/services/actions/app/live_actions/builtins.py b/services/actions/app/live_actions/builtins.pyindex 3311b1caa..6471772b5 100644--- a/services/actions/app/live_actions/builtins.py+++ b/services/actions/app/live_actions/builtins.py@@ -83,6 +83,7 @@ from .investigation_reads import (OktaGetUserActivity,)from .models import LiveActionRequest, LiveActionResult, LiveActionStatus+from .vendor_breadth import VENDOR_BREADTH_EXECUTORSlogger = structlog.get_logger(__name__)@@ -595,6 +596,10 @@ _BUILTIN_ADAPTERS: tuple[type[LiveActionExecutor], ...] = (# Rollback for the most disruptive action, which had no executor.CrowdStrikeUnisolateHost,DefenderUnisolateHost,+ # Vendor breadth: capabilities the clients already implemented and+ # the registry could not reach. SentinelOne exposed seven+ # operations and one was wired; Entra six and one.+ *VENDOR_BREADTH_EXECUTORS,# EndpointCrowdStrikeIsolateHost,DefenderIsolateHost,diff --git a/services/actions/app/live_actions/vendor_breadth.py b/services/actions/app/live_actions/vendor_breadth.pynew file mode 100644index 000000000..8e9aeb518--- /dev/null+++ b/services/actions/app/live_actions/vendor_breadth.py@@ -0,0 +1,483 @@+"""Vendor capabilities the clients implement and the registry could not reach.++Seventeen vendors are wired and most expose exactly one verb, so the action+registry looked like breadth and behaved like a single button per product.+The reason is not that the integrations are shallow: `SentinelOneClient`+implements seven operations and one was reachable; `AzureEntraClient`+implements six and one was reachable. The work was done and nothing+connected it — the same pattern v8.0 spent a release removing, at the scale+of a whole capability surface.++Every executor here calls a method that already existed. Nothing was+stubbed, and nothing is declared that a client cannot do.++Several are the reverse actions the contract already promised. `enable_user`+was declared as `disable_user`'s reverse and had no implementation on any+vendor, so an account disabled during an incident had no route back through+the platform that disabled it. `allow_ip`, `allow_domain` and+`revoke_session` were in the same state.++The shared rules, both inherited from `investigation_reads.py`:++**A read or action failure is reported, never silently absorbed.** An+executor returns FAILED with the vendor's own message rather than a+success that means "we asked".++**Credentials absent is not the same as the action failing.** Missing+configuration says so explicitly, because "we could not reach the vendor"+and "the vendor refused" send an operator to different places.+"""++from __future__ import annotations++from collections.abc import Awaitable, Callable+from typing import Any++import structlog++from app.clients.factories import (+ _cloudflare_client,+ _entra_client,+ _fortigate_client,+ _gws_client,+ _panos_client,+ _s1_client,+)+from app.live_actions.capability_contracts import apply_contract+from app.live_actions.executor import LiveActionExecutor+from app.live_actions.models import LiveActionRequest, LiveActionResult, LiveActionStatus++logger = structlog.get_logger(__name__)+++def _result(+ executor: LiveActionExecutor,+ request: LiveActionRequest,+ status: LiveActionStatus,+ summary: str,+ *,+ details: dict[str, Any] | None = None,+ error: str | None = None,+) -> LiveActionResult:+ return LiveActionResult(+ request_id=request.request_id,+ status=status,+ capability=executor.capability,+ vendor_id=executor.vendor_id,+ summary=summary,+ details=details or {},+ error=error,+ )+++async def _run(+ executor: LiveActionExecutor,+ request: LiveActionRequest,+ vendor: str,+ factory: Callable[[dict[str, Any]], Any],+ call: Callable[[Any, str, dict[str, Any]], Awaitable[dict[str, Any]]],+ summary: str,+) -> LiveActionResult:+ """Shared execute body: dry-run, credentials, call, error handling.++ Extracted because seventeen executors repeating it is seventeen chances+ to forget the dry-run check or to let an exception escape — and an+ exception escaping the executor is how a single bad action wedges the+ agent loop.+ """+ params: dict[str, Any] = request.params or {}+ target = str(request.target or "")++ if request.dry_run:+ return _result(+ executor,+ request,+ LiveActionStatus.SIMULATED,+ f"would {executor.capability} {target} on {vendor}",+ details={"would_run": executor.capability, "target": target, "vendor": vendor},+ )++ client = factory(params)+ if client is None:+ return _result(+ executor,+ request,+ LiveActionStatus.FAILED,+ f"{vendor} credentials not configured",+ error=(+ f"{vendor} credentials are not configured, so {executor.capability} "+ f"was not attempted. This is not a statement about the target."+ ),+ )++ try:+ response = await call(client, target, params)+ except Exception as exc: # noqa: BLE001 - a vendor error is FAILED, never silent+ logger.warning(+ "vendor_action.failed",+ vendor=executor.vendor_id,+ capability=executor.capability,+ error=str(exc),+ )+ return _result(+ executor,+ request,+ LiveActionStatus.FAILED,+ f"{vendor} {executor.capability} failed",+ error=f"{vendor} {executor.capability} failed: {exc}",+ )++ return _result(+ executor,+ request,+ LiveActionStatus.SUCCEEDED,+ summary.format(target=target),+ details={"target": target, "vendor_response": response},+ )+++# ─── SentinelOne ─────────────────────────────────────────────────────────────+#+# Seven client methods, one reachable. `lift_containment` is the one that+# mattered: containment with no release means an incident closes and the+# host stays off the network until someone remembers.+++@apply_contract+class SentinelOneUnisolateHost(LiveActionExecutor):+ vendor_id = "sentinelone"+ capability = "unisolate_host"+ description = "Lift network containment on a SentinelOne agent."+ requires_credentials = True++ async def execute(self, request: LiveActionRequest) -> LiveActionResult:+ return await _run(+ self,+ request,+ "SentinelOne",+ _s1_client,+ lambda c, t, p: c.lift_containment(t),+ "containment lifted on {target}",+ )+++@apply_contract+class SentinelOneKillProcess(LiveActionExecutor):+ vendor_id = "sentinelone"+ capability = "kill_process"+ description = "Terminate a process on a SentinelOne-managed host."+ requires_credentials = True++ async def execute(self, request: LiveActionRequest) -> LiveActionResult:+ return await _run(+ self,+ request,+ "SentinelOne",+ _s1_client,+ lambda c, t, p: c.kill_process(t, pid=p.get("pid"), process_name=p.get("process_name")),+ "process terminated on {target}",+ )+++@apply_contract+class SentinelOneQuarantineFile(LiveActionExecutor):+ vendor_id = "sentinelone"+ capability = "quarantine_file"+ description = "Quarantine a file on a SentinelOne-managed host."+ requires_credentials = True++ async def execute(self, request: LiveActionRequest) -> LiveActionResult:+ return await _run(+ self,+ request,+ "SentinelOne",+ _s1_client,+ lambda c, t, p: c.quarantine_file(t, str(p.get("file_path", ""))),+ "file quarantined on {target}",+ )+++@apply_contract+class SentinelOneRunAVScan(LiveActionExecutor):+ vendor_id = "sentinelone"+ capability = "run_av_scan"+ description = "Start an on-demand scan on a SentinelOne-managed host."+ requires_credentials = True++ async def execute(self, request: LiveActionRequest) -> LiveActionResult:+ return await _run(+ self,+ request,+ "SentinelOne",+ _s1_client,+ lambda c, t, p: c.run_av_scan(t, scan_type=str(p.get("scan_type", "Full"))),+ "scan started on {target}",+ )+++@apply_contract+class SentinelOneRunScript(LiveActionExecutor):+ vendor_id = "sentinelone"+ capability = "run_script"+ description = "Run a remote script on a SentinelOne-managed host."+ requires_credentials = True++ async def execute(self, request: LiveActionRequest) -> LiveActionResult:+ return await _run(+ self,+ request,+ "SentinelOne",+ _s1_client,+ lambda c, t, p: c.run_script(t, str(p.get("script_content", ""))),+ "script dispatched to {target}",+ )+++# ─── Microsoft Entra ─────────────────────────────────────────────────────────+++@apply_contract+class EntraEnableUser(LiveActionExecutor):+ vendor_id = "azure_entra"+ capability = "enable_user"+ description = "Re-enable an account in Microsoft Entra ID."+ requires_credentials = True++ async def execute(self, request: LiveActionRequest) -> LiveActionResult:+ return await _run(+ self,+ request,+ "Entra",+ _entra_client,+ lambda c, t, p: c.enable_user(t),+ "{target} re-enabled",+ )+++@apply_contract+class EntraRevokeSession(LiveActionExecutor):+ vendor_id = "azure_entra"+ capability = "revoke_session"+ description = "Revoke all refresh tokens for an Entra ID account."+ requires_credentials = True++ async def execute(self, request: LiveActionRequest) -> LiveActionResult:+ return await _run(+ self,+ request,+ "Entra",+ _entra_client,+ lambda c, t, p: c.revoke_sessions(t),+ "sessions revoked for {target}",+ )+++@apply_contract+class EntraResetPassword(LiveActionExecutor):+ vendor_id = "azure_entra"+ capability = "reset_password"+ description = "Force a password reset on an Entra ID account."+ requires_credentials = True++ async def execute(self, request: LiveActionRequest) -> LiveActionResult:+ return await _run(+ self,+ request,+ "Entra",+ _entra_client,+ lambda c, t, p: c.reset_password(t),+ "password reset forced for {target}",+ )+++@apply_contract+class EntraForceMFA(LiveActionExecutor):+ vendor_id = "azure_entra"+ capability = "force_mfa"+ description = "Require MFA re-registration for an Entra ID account."+ requires_credentials = True++ async def execute(self, request: LiveActionRequest) -> LiveActionResult:+ return await _run(+ self,+ request,+ "Entra",+ _entra_client,+ lambda c, t, p: c.require_mfa(t),+ "MFA re-registration required for {target}",+ )+++# ─── Google Workspace ────────────────────────────────────────────────────────+++@apply_contract+class GoogleWorkspaceEnableUser(LiveActionExecutor):+ vendor_id = "google_workspace"+ capability = "enable_user"+ description = "Un-suspend a Google Workspace account."+ requires_credentials = True++ async def execute(self, request: LiveActionRequest) -> LiveActionResult:+ return await _run(+ self,+ request,+ "Google Workspace",+ _gws_client,+ lambda c, t, p: c.unsuspend_user(t),+ "{target} un-suspended",+ )+++@apply_contract+class GoogleWorkspaceRevokeSession(LiveActionExecutor):+ vendor_id = "google_workspace"+ capability = "revoke_session"+ description = "Revoke sign-in cookies for a Google Workspace account."+ requires_credentials = True++ async def execute(self, request: LiveActionRequest) -> LiveActionResult:+ return await _run(+ self,+ request,+ "Google Workspace",+ _gws_client,+ lambda c, t, p: c.revoke_sessions(t),+ "sessions revoked for {target}",+ )+++@apply_contract+class GoogleWorkspaceResetPassword(LiveActionExecutor):+ vendor_id = "google_workspace"+ capability = "reset_password"+ description = "Force a password change on a Google Workspace account."+ requires_credentials = True++ async def execute(self, request: LiveActionRequest) -> LiveActionResult:+ return await _run(+ self,+ request,+ "Google Workspace",+ _gws_client,+ lambda c, t, p: c.reset_password(t),+ "password reset forced for {target}",+ )+++# ─── Network ─────────────────────────────────────────────────────────────────+#+# Every block had no unblock. A perimeter block with no release is a+# permanent one, applied during an incident by an automation nobody+# revisits — and the address is usually reassigned within weeks.+++@apply_contract+class CloudflareAllowIP(LiveActionExecutor):+ vendor_id = "cloudflare"+ capability = "allow_ip"+ description = "Remove a Cloudflare zone-level IP block."+ requires_credentials = True++ async def execute(self, request: LiveActionRequest) -> LiveActionResult:+ return await _run(+ self,+ request,+ "Cloudflare",+ _cloudflare_client,+ lambda c, t, p: c.unblock_ip_zone(str(p.get("rule_id", "")), str(p.get("cf_zone_id", ""))),+ "block removed for {target}",+ )+++@apply_contract+class CloudflareBlockDomain(LiveActionExecutor):+ vendor_id = "cloudflare"+ capability = "block_domain"+ description = "Sinkhole a domain via Cloudflare Gateway."+ requires_credentials = True++ async def execute(self, request: LiveActionRequest) -> LiveActionResult:+ return await _run(+ self,+ request,+ "Cloudflare",+ _cloudflare_client,+ lambda c, t, p: c.sinkhole_domain(t, str(p.get("cf_account_id", "")), str(p.get("cf_list_id", ""))),+ "{target} sinkholed",+ )+++@apply_contract+class CloudflareAllowDomain(LiveActionExecutor):+ vendor_id = "cloudflare"+ capability = "allow_domain"+ description = "Remove a Cloudflare Gateway domain sinkhole."+ requires_credentials = True++ async def execute(self, request: LiveActionRequest) -> LiveActionResult:+ return await _run(+ self,+ request,+ "Cloudflare",+ _cloudflare_client,+ lambda c, t, p: c.unsinkhole_domain(t, str(p.get("cf_account_id", "")), str(p.get("cf_list_id", ""))),+ "sinkhole removed for {target}",+ )+++@apply_contract+class PanOsAllowIP(LiveActionExecutor):+ vendor_id = "panos"+ capability = "allow_ip"+ description = "Remove a PAN-OS dynamic address-group tag from an IP."+ requires_credentials = True++ async def execute(self, request: LiveActionRequest) -> LiveActionResult:+ return await _run(+ self,+ request,+ "PAN-OS",+ _panos_client,+ lambda c, t, p: c.unblock_ip(t, str(p.get("pan_tag", "aisoc-blocked"))),+ "tag removed from {target}",+ )+++@apply_contract+class FortiGateAllowIP(LiveActionExecutor):+ vendor_id = "fortigate"+ capability = "allow_ip"+ description = "Remove an IP from the FortiGate block address group."+ requires_credentials = True++ async def execute(self, request: LiveActionRequest) -> LiveActionResult:+ return await _run(+ self,+ request,+ "FortiGate",+ _fortigate_client,+ lambda c, t, p: c.unblock_ip(t, str(p.get("fgt_address_group", ""))),+ "{target} removed from the block group",+ )+++VENDOR_BREADTH_EXECUTORS: tuple[type[LiveActionExecutor], ...] = (+ SentinelOneUnisolateHost,+ SentinelOneKillProcess,+ SentinelOneQuarantineFile,+ SentinelOneRunAVScan,+ SentinelOneRunScript,+ EntraEnableUser,+ EntraRevokeSession,+ EntraResetPassword,+ EntraForceMFA,+ GoogleWorkspaceEnableUser,+ GoogleWorkspaceRevokeSession,+ GoogleWorkspaceResetPassword,+ CloudflareAllowIP,+ CloudflareBlockDomain,+ CloudflareAllowDomain,+ PanOsAllowIP,+ FortiGateAllowIP,+)diff --git a/services/actions/tests/test_live_action_builtins.py b/services/actions/tests/test_live_action_builtins.pyindex e44daa6c0..fa297af51 100644--- a/services/actions/tests/test_live_action_builtins.py+++ b/services/actions/tests/test_live_action_builtins.py@@ -51,10 +51,12 @@ def test_register_builtin_executors_returns_full_count() -> None:cheap reminder to update docs / discovery snapshots."""count = register_builtin_executors()- # 19 original + 10 Phase B2 vendors (SentinelOne/Entra/GWS/PAN-OS/FortiGate/- # Cloudflare/Jira/ServiceNow/PagerDuty/Slack) + 6 read/rollback verbs- # (4 read-only investigation reads, 2 unisolate arms).- assert count == 35+ # 19 original + 10 Phase B2 vendors + 6 read/rollback verbs + 17+ # vendor-breadth executors. The last group added no new integrations:+ # every one calls a client method that already existed and had no+ # route through the registry — SentinelOne implemented seven+ # operations with one reachable, Entra six with one.+ assert count == 52def test_builtin_executors_cover_canonical_vendor_capability_pairs() -> None:@@ -69,52 +71,58 @@ def test_builtin_executors_cover_canonical_vendor_capability_pairs() -> None:pairs = {(d.vendor_id, d.capability) for d in descriptors}expected = {- # Endpoint+ ("aws_security_groups", "allow_ip"),+ ("aws_security_groups", "block_ip"),+ ("azure_entra", "disable_user"),+ ("azure_entra", "enable_user"),+ ("azure_entra", "force_mfa"),+ ("azure_entra", "reset_password"),+ ("azure_entra", "revoke_session"),+ ("cloudflare", "allow_domain"),+ ("cloudflare", "allow_ip"),+ ("cloudflare", "block_domain"),+ ("cloudflare", "block_ip"),+ ("crowdstrike", "get_detections"),+ ("crowdstrike", "get_host"),("crowdstrike", "isolate_host"),- ("defender", "isolate_host"),- ("crowdstrike", "quarantine_file"),("crowdstrike", "kill_process"),+ ("crowdstrike", "quarantine_file"),("crowdstrike", "run_script"),+ ("crowdstrike", "unisolate_host"),+ ("defender", "block_ioc"),+ ("defender", "get_host"),+ ("defender", "isolate_host"),("defender", "run_av_scan"),- # Identity- ("okta", "disable_user"),- ("okta", "reset_password"),- ("okta", "suspend_session"),- ("okta", "force_mfa"),- # Network- ("aws_security_groups", "block_ip"),- ("aws_security_groups", "allow_ip"),- ("generic", "block_domain"),- # SIEM- ("splunk", "search_siem"),+ ("defender", "unisolate_host"),("elastic", "search_siem"),- ("splunk", "create_notable_event"),- ("splunk", "sync_detection_rule"),("elastic", "update_watcher"),- ("defender", "block_ioc"),- # Phase B2 — previously-unregistered vendors- ("sentinelone", "isolate_host"),- ("azure_entra", "disable_user"),- ("google_workspace", "disable_user"),- ("panos", "block_ip"),+ ("fortigate", "allow_ip"),("fortigate", "block_ip"),- ("cloudflare", "block_ip"),+ ("generic", "block_domain"),+ ("google_workspace", "disable_user"),+ ("google_workspace", "enable_user"),+ ("google_workspace", "reset_password"),+ ("google_workspace", "revoke_session"),("jira", "create_ticket"),- ("servicenow", "create_ticket"),+ ("okta", "disable_user"),+ ("okta", "force_mfa"),+ ("okta", "get_user_activity"),+ ("okta", "reset_password"),+ ("okta", "suspend_session"),("pagerduty", "create_ticket"),+ ("panos", "allow_ip"),+ ("panos", "block_ip"),+ ("sentinelone", "isolate_host"),+ ("sentinelone", "kill_process"),+ ("sentinelone", "quarantine_file"),+ ("sentinelone", "run_av_scan"),+ ("sentinelone", "run_script"),+ ("sentinelone", "unisolate_host"),+ ("servicenow", "create_ticket"),("slack", "notify"),- # Read-only investigation verbs. Twenty-nine executors could change- # the estate and exactly one could ask it a question, so an- # investigation could only ever reach the lake.- ("crowdstrike", "get_host"),- ("crowdstrike", "get_detections"),- ("defender", "get_host"),- ("okta", "get_user_activity"),- # The rollback for the most disruptive action, which had no- # executor: isolate_host declared unisolate_host as its reverse and- # dispatch answered executor_not_found.- ("crowdstrike", "unisolate_host"),- ("defender", "unisolate_host"),+ ("splunk", "create_notable_event"),+ ("splunk", "search_siem"),+ ("splunk", "sync_detection_rule"),}assert pairs == expected@@ -145,7 +153,7 @@ def test_register_builtin_executors_is_idempotent_with_overwrite() -> None:register_builtin_executors()# Second call without overwrite would raise — confirm overwrite works.count = register_builtin_executors(overwrite=True)- assert count == 35+ assert count == 52def test_register_builtin_twice_without_overwrite_raises() -> None:diff --git a/services/actions/tests/test_vendor_breadth.py b/services/actions/tests/test_vendor_breadth.pynew file mode 100644index 000000000..949606a03--- /dev/null+++ b/services/actions/tests/test_vendor_breadth.py@@ -0,0 +1,179 @@+"""Vendor capabilities the clients implemented and the registry could not reach.++Seventeen vendors were wired and most exposed exactly one verb, so the+registry looked like breadth and behaved like a single button per product.+Not because the integrations were shallow: `SentinelOneClient` implements+seven operations and one was reachable; `AzureEntraClient` implements six+and one was reachable. The work existed and nothing connected it.++The tests below are structural on purpose. Asserting that seventeen+executors each forward to the right client method is low-value repetition;+asserting that **every executor calls a method its client actually has** is+the property that would have caught the whole class of defect, and it is+the one that breaks when a client is refactored.+"""++from __future__ import annotations++import inspect+from unittest.mock import AsyncMock++import pytest+from app.live_actions import vendor_breadth+from app.live_actions.capability_contracts import CAPABILITY_CONTRACTS+from app.live_actions.models import LiveActionRequest, LiveActionStatus+from app.live_actions.vendor_breadth import (+ VENDOR_BREADTH_EXECUTORS,+ CloudflareAllowIP,+ EntraEnableUser,+ SentinelOneUnisolateHost,+)++#: executor -> (factory name in vendor_breadth, client method it must call)+EXPECTED_CALLS = {+ "SentinelOneUnisolateHost": ("_s1_client", "lift_containment"),+ "SentinelOneKillProcess": ("_s1_client", "kill_process"),+ "SentinelOneQuarantineFile": ("_s1_client", "quarantine_file"),+ "SentinelOneRunAVScan": ("_s1_client", "run_av_scan"),+ "SentinelOneRunScript": ("_s1_client", "run_script"),+ "EntraEnableUser": ("_entra_client", "enable_user"),+ "EntraRevokeSession": ("_entra_client", "revoke_sessions"),+ "EntraResetPassword": ("_entra_client", "reset_password"),+ "EntraForceMFA": ("_entra_client", "require_mfa"),+ "GoogleWorkspaceEnableUser": ("_gws_client", "unsuspend_user"),+ "GoogleWorkspaceRevokeSession": ("_gws_client", "revoke_sessions"),+ "GoogleWorkspaceResetPassword": ("_gws_client", "reset_password"),+ "CloudflareAllowIP": ("_cloudflare_client", "unblock_ip_zone"),+ "CloudflareBlockDomain": ("_cloudflare_client", "sinkhole_domain"),+ "CloudflareAllowDomain": ("_cloudflare_client", "unsinkhole_domain"),+ "PanOsAllowIP": ("_panos_client", "unblock_ip"),+ "FortiGateAllowIP": ("_fortigate_client", "unblock_ip"),+}++REAL_CLIENTS = {+ "_s1_client": "app.clients.sentinelone_client.SentinelOneClient",+ "_entra_client": "app.clients.azure_entra_client.AzureEntraClient",+ "_gws_client": "app.clients.google_workspace_client.GoogleWorkspaceClient",+ "_cloudflare_client": "app.clients.cloudflare_client.CloudflareClient",+ "_panos_client": "app.clients.panos_client.PanOsClient",+ "_fortigate_client": "app.clients.fortigate_client.FortiGateClient",+}+++def _import(path: str):+ module_path, _, name = path.rpartition(".")+ module = __import__(module_path, fromlist=[name])+ return getattr(module, name)+++def _request(capability: str, vendor: str, **params) -> LiveActionRequest:+ return LiveActionRequest(vendor_id=vendor, capability=capability, target="TARGET", params=params)+++class TestEveryExecutorIsBacked:+ def test_every_executor_is_covered_by_this_test(self) -> None:+ """A new executor added without a row here is untested breadth,+ which is how a registry fills with things nobody exercised."""+ names = {cls.__name__ for cls in VENDOR_BREADTH_EXECUTORS}+ assert names == set(EXPECTED_CALLS)++ @pytest.mark.parametrize(("executor_name", "expected"), sorted(EXPECTED_CALLS.items()))+ def test_the_client_method_exists(self, executor_name: str, expected: tuple[str, str]) -> None:+ """The property that would have caught the whole class of defect.++ Simulation mode never constructs the client, so an executor calling+ a method that does not exist works in every test and raises the+ first time a credential is present. Checked against the real class,+ not a mock.+ """+ factory_name, method = expected+ client_cls = _import(REAL_CLIENTS[factory_name])+ assert hasattr(client_cls, method), f"{executor_name} calls {method}() which {client_cls.__name__} does " f"not have"+ assert inspect.iscoroutinefunction(getattr(client_cls, method)), f"{client_cls.__name__}.{method} is not awaitable"++ @pytest.mark.parametrize("executor_cls", VENDOR_BREADTH_EXECUTORS, ids=lambda c: c.__name__)+ def test_every_capability_has_a_contract(self, executor_cls: type) -> None:+ assert executor_cls.capability in CAPABILITY_CONTRACTS+++class TestSharedBehaviour:+ """Three properties every executor inherits from the shared body. Tested+ on representatives rather than all seventeen: the body is shared, so+ seventeen copies of the same assertion tests the parametrize decorator."""++ @pytest.mark.asyncio+ async def test_dry_run_does_not_reach_the_vendor(self, monkeypatch: pytest.MonkeyPatch) -> None:+ client = AsyncMock()+ monkeypatch.setattr(vendor_breadth, "_s1_client", lambda p: client)+ request = _request("unisolate_host", "sentinelone")+ request.dry_run = True++ result = await SentinelOneUnisolateHost().execute(request)+ assert result.status is LiveActionStatus.SIMULATED+ client.lift_containment.assert_not_called()++ @pytest.mark.asyncio+ async def test_missing_credentials_says_so_rather_than_failing_vaguely(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """ "We could not reach the vendor" and "the vendor refused" send an+ operator to different places."""+ monkeypatch.setattr(vendor_breadth, "_entra_client", lambda p: None)+ result = await EntraEnableUser().execute(_request("enable_user", "azure_entra"))+ assert result.status is LiveActionStatus.FAILED+ assert "not a statement about the target" in (result.error or "")++ @pytest.mark.asyncio+ async def test_a_vendor_exception_becomes_failed_not_an_escape(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """An exception escaping an executor wedges the agent loop."""+ client = AsyncMock()+ client.lift_containment = AsyncMock(side_effect=RuntimeError("503"))+ monkeypatch.setattr(vendor_breadth, "_s1_client", lambda p: client)++ result = await SentinelOneUnisolateHost().execute(_request("unisolate_host", "sentinelone"))+ assert result.status is LiveActionStatus.FAILED+ assert "503" in (result.error or "")++ @pytest.mark.asyncio+ async def test_success_carries_the_vendor_response(self, monkeypatch: pytest.MonkeyPatch) -> None:+ client = AsyncMock()+ client.lift_containment = AsyncMock(return_value={"ok": True})+ monkeypatch.setattr(vendor_breadth, "_s1_client", lambda p: client)++ result = await SentinelOneUnisolateHost().execute(_request("unisolate_host", "sentinelone"))+ assert result.status is LiveActionStatus.SUCCEEDED+ assert result.details["vendor_response"] == {"ok": True}++ @pytest.mark.asyncio+ async def test_vendor_specific_params_are_forwarded(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """Cloudflare's unblock needs a rule and zone id, not just the IP.+ Dropping them would produce a call that succeeds against the wrong+ zone or fails obscurely."""+ client = AsyncMock()+ client.unblock_ip_zone = AsyncMock(return_value={})+ monkeypatch.setattr(vendor_breadth, "_cloudflare_client", lambda p: client)++ await CloudflareAllowIP().execute(_request("allow_ip", "cloudflare", rule_id="r1", cf_zone_id="z1"))+ client.unblock_ip_zone.assert_awaited_once_with("r1", "z1")+++class TestReversesAreNowReachable:+ """Several of these are the reverse actions the contract already+ promised and nothing implemented."""++ @pytest.mark.parametrize(+ ("forward", "reverse"),+ [+ ("disable_user", "enable_user"),+ ("isolate_host", "unisolate_host"),+ ("block_ip", "allow_ip"),+ ("block_domain", "allow_domain"),+ ],+ )+ def test_a_declared_reverse_has_an_implementation(self, forward: str, reverse: str) -> None:+ from app.live_actions import builtins++ implemented = {c.capability for c in builtins._BUILTIN_ADAPTERS} # noqa: SLF001+ declared = CAPABILITY_CONTRACTS[forward].reverse_capability+ assert declared == reverse+ assert reverse in implemented, (+ f"{forward} declares {reverse} as its reverse and nothing " f"implements it, so the rollback path resolves to nothing"+ )diff --git a/services/agents/tests/test_detection_fp_rate.py b/services/agents/tests/test_detection_fp_rate.pyindex f8268f9b2..5ea4f4a5e 100644--- a/services/agents/tests/test_detection_fp_rate.py+++ b/services/agents/tests/test_detection_fp_rate.py@@ -64,7 +64,7 @@ if str(_SCRIPTS_DIR) not in sys.path:sys.path.insert(0, str(_SCRIPTS_DIR))from detection_specs_index import all_specs # type: ignore[import-not-found] # noqa: E402-from generate_detections import matches # type: ignore[import-not-found] # noqa: E402+from generate_detections import enrich, matches, requested_derived_fields # type: ignore[import-not-found] # noqa: E402# CI gate (lower-is-better). Tighten only with a corresponding allowlist# entry or rule-narrowing PR. The 5% ceiling is a deliberate compromise:@@ -179,6 +179,14 @@ def evaluate_per_rule_fp(positive = spec["positive"]negative = spec.get("negative")+ # Enriched the way the engine enriches. This is the third replay+ # path that read fixtures through the bare matcher; each one tests+ # a pipeline production does not run, so a rule matching a derived+ # field fails here while working live.+ wanted = requested_derived_fields([{"match_when": match_when}])+ positive = enrich(positive, wanted)+ negative = enrich(negative, wanted) if negative else negative+own_pos = bool(matches(match_when, positive))own_neg = bool(matches(match_when, negative)) if negative else False@@ -190,7 +198,7 @@ def evaluate_per_rule_fp(cross_total += 1if (slug, other_slug) in allowlist:continue- if matches(match_when, other_positive):+ if matches(match_when, enrich(other_positive, wanted)):cross_fires.append((other_slug, "positive"))per_rule.append(diff --git a/services/api/app/api/v1/endpoints/health.py b/services/api/app/api/v1/endpoints/health.pyindex 46f4f04a0..65bf946ed 100644--- a/services/api/app/api/v1/endpoints/health.py+++ b/services/api/app/api/v1/endpoints/health.py@@ -86,6 +86,7 @@ from app.core.config import settingsfrom app.models.alert import Alertfrom app.models.connector import Connectorfrom app.services.connector_freshness import compute_freshness+from app.services.fleet_health import assess_fleetlogger = logging.getLogger(__name__)@@ -509,3 +510,26 @@ async def get_pipeline_health(stages=stages,generated_at=now,)++[email protected]("/fleet")+async def get_fleet_health(+ user: AuthUser,+ db: DBSession,+) -> dict[str, Any]:+ """Which connectors have quietly stopped working.++ Every field this reads was already being written — `last_sync`,+ `error_count`, `oauth_refresh_failures`, `last_schema_drift_at` — and+ nothing read them together. That produces the failure this platform is+ least able to tolerate: a connector stops polling, alerts from that+ source stop arriving, and the console looks calm, because an absence of+ alerts is indistinguishable from an absence of threats.++ Staleness is judged per connector against its own configured cadence.+ A single global threshold would page constantly on daily connectors or+ stay silent on five-minute ones, and a surface that pages constantly is+ a surface that gets muted.+ """+ rows = (await db.execute(select(Connector).where(Connector.tenant_id == user.tenant_id))).scalars().all()+ return assess_fleet(list(rows)).to_dict()diff --git a/services/api/app/db/clickhouse.py b/services/api/app/db/clickhouse.pyindex f3a944c05..0997d9314 100644--- a/services/api/app/db/clickhouse.py+++ b/services/api/app/db/clickhouse.py@@ -431,3 +431,49 @@ async def fetch_lake_schema(by_table[key].append({"name": name, "type": ctype, "comment": comment or ""})return [{"table": fq, "columns": cols} for fq, cols in by_table.items()]+++async def init_lake_schema() -> None:+ """Converge the lake schema at startup.++ `clickhouse/001_init.sql` runs only in the container entrypoint, and+ only on a volume that has never been initialised. `lake_writer`+ self-heals with `CREATE TABLE IF NOT EXISTS`, which is a no-op once the+ table exists. Neither adds a column to an existing deployment, so a+ schema change lands on fresh installs and silently does not land+ anywhere else — invisible until a query selects a column that is not+ there, long after the deploy that was meant to add it.++ Non-strict on purpose, matching the Neo4j equivalent: a migration+ failure must not take the API down, because the lake is a query surface+ and the rest of the platform works without it. It is logged at+ ``error`` and the pending list stays readable for diagnostics, so a+ degraded start is visible rather than assumed.+ """+ from app.db.lake_migrations import pending_ids, run_migrations # noqa: PLC0415++ try:+ client = await get_clickhouse_client()+ except LakeQueryNotConfiguredError:+ logger.info("clickhouse not configured — skipping lake migrations")+ return+ except Exception as exc: # noqa: BLE001 - an unreachable lake is not fatal+ logger.warning("clickhouse unreachable at startup: %s", exc)+ return++ try:+ applied = await run_migrations(client, strict=False)+ pending = await pending_ids(client)+ except Exception as exc: # noqa: BLE001 - same reasoning+ logger.error("lake migrations could not run: %s", exc)+ return++ if applied:+ logger.info("ClickHouse lake migrations applied: %s", ", ".join(applied))+ if pending:+ logger.error(+ "ClickHouse lake migrations still pending after startup: %s — " "lake-backed features may query columns that do not exist",+ ", ".join(pending),+ )+ else:+ logger.info("ClickHouse lake schema up to date")diff --git a/services/api/app/db/lake_migrations.py b/services/api/app/db/lake_migrations.pynew file mode 100644index 000000000..85a35da72--- /dev/null+++ b/services/api/app/db/lake_migrations.py@@ -0,0 +1,207 @@+"""Forward-only schema migrations for the ClickHouse event lake.++`services/api/clickhouse/001_init.sql` is four `CREATE TABLE IF NOT EXISTS`+statements and zero `ALTER TABLE`, and `lake_writer` "self-heals" with the+same idiom. Both are no-ops once the table exists, so a new column lands on+a fresh deployment and silently does not land on an existing one — and the+failure is invisible until a query selects a column that is not there, which+happens long after the deploy that was supposed to add it.++This is the ClickHouse half of the migration mechanism whose Neo4j half is+`graph_migrations.py`. Same shape deliberately: numbered, append-only,+applied ids recorded in the store itself so a deployment knows what it has.++Three differences ClickHouse forces:++**The ledger is a table, not a node**, and it is created with the same+`IF NOT EXISTS` idiom this module exists to work around. That is fine for+exactly one table whose schema never changes — and the schema is pinned by+a test, so "never changes" is enforced rather than hoped for.++**`ADD COLUMN IF NOT EXISTS` is genuinely idempotent** in ClickHouse, unlike+Neo4j's constraint syntax, so a migration that half-applied can be re-run.+Statements should still be written to tolerate it, because a mutation that+fails midway leaves the earlier statements applied.++**Distributed DDL is asynchronous.** On a cluster, `ALTER TABLE` returns+before every replica has applied it. Single-node deployments are unaffected;+a cluster needs `ON CLUSTER` and a sync setting, which is called out on the+migration that would need it rather than assumed.+"""++from __future__ import annotations++import logging+from collections.abc import Awaitable, Callable+from dataclasses import dataclass, field+from datetime import UTC, datetime+from typing import Any++logger = logging.getLogger("aisoc.lake_migrations")++MIGRATION_TABLE = "aisoc._migrations"++#: Schema of the ledger itself. Pinned by a test: this table is created with+#: the `IF NOT EXISTS` idiom the rest of this module exists to replace, which+#: is only safe while its shape never changes.+_LEDGER_DDL = f"""+CREATE TABLE IF NOT EXISTS {MIGRATION_TABLE} (+ migration_id String,+ description String,+ applied_at DateTime64(3, 'UTC') DEFAULT now64(),+ statements UInt32+) ENGINE = MergeTree()+ORDER BY migration_id+"""+++@dataclass(frozen=True)+class LakeMigration:+ """One forward-only change to the lake schema.++ ``statements`` run in order. ``backfill`` runs after them for changes+ that need to touch data — ClickHouse mutations are asynchronous, so a+ backfill that must complete before the next migration has to wait on+ ``system.mutations`` itself rather than assuming the ALTER returned.+ """++ id: str+ description: str+ statements: tuple[str, ...] = ()+ backfill: Callable[[Any], Awaitable[int]] | None = None+ #: True when the statements need `ON CLUSTER` on a multi-replica+ #: deployment. Recorded rather than applied, because this repo has no+ #: cluster topology to name and guessing one is worse than declaring the+ #: limitation.+ needs_cluster_ddl: bool = False+ tags: tuple[str, ...] = field(default_factory=tuple)+++# ── 001: the pre-existing init schema, captured as a migration ─────────────+#+# Identical in effect to 001_init.sql, so an existing deployment records 001+# as applied and converges rather than re-running statements it cannot know+# the outcome of. Deliberately not a copy of the file: the file creates the+# database and tables, and re-running it here would be the same no-op. What+# matters is that the ledger agrees the baseline exists.+_V1_BASELINE = ("CREATE DATABASE IF NOT EXISTS aisoc",)+++MIGRATIONS: tuple[LakeMigration, ...] = (+ LakeMigration(+ id="001_baseline",+ description=("Records the 001_init.sql baseline as applied. Creates nothing an " "existing deployment does not already have."),+ statements=_V1_BASELINE,+ tags=("baseline",),+ ),+)+++async def _execute(client: Any, sql: str, params: dict[str, Any] | None = None) -> Any:+ """Run one statement against whichever ClickHouse client shape we have.++ The async driver exposes `execute`; the HTTP client exposes `command`.+ Both are in use across this repo depending on the service, and a+ migration runner that only speaks one of them is a runner that silently+ does not run in half the deployments.+ """+ if hasattr(client, "execute"):+ return await client.execute(sql, params or {})+ if hasattr(client, "command"):+ return await client.command(sql)+ raise TypeError(f"{type(client).__name__} exposes neither execute() nor command(); " f"the lake migration runner cannot drive it")+++async def applied_ids(client: Any) -> set[str]:+ """Migration ids this lake already has.++ An unreadable ledger returns the empty set rather than raising, so a+ first run on a lake with no ledger works. It is the caller's `strict`+ flag that decides whether a *failed* migration is fatal.+ """+ await _execute(client, _LEDGER_DDL)+ try:+ rows = await _execute(client, f"SELECT migration_id FROM {MIGRATION_TABLE}")+ except Exception as exc: # noqa: BLE001 - an empty ledger is a normal first run+ logger.warning("lake_migrations.ledger_unreadable error=%s", exc)+ return set()++ ids: set[str] = set()+ for row in rows or []:+ # Drivers return tuples, dicts or scalars depending on which one.+ if isinstance(row, dict):+ value = row.get("migration_id")+ elif isinstance(row, list | tuple):+ value = row[0] if row else None+ else:+ value = row+ if value:+ ids.add(str(value))+ return ids+++async def _record(client: Any, migration: LakeMigration, statements: int) -> None:+ await _execute(+ client,+ f"INSERT INTO {MIGRATION_TABLE} " "(migration_id, description, applied_at, statements) VALUES",+ {+ "migration_id": migration.id,+ "description": migration.description,+ "applied_at": datetime.now(UTC),+ "statements": statements,+ },+ )+++async def run_migrations(client: Any, *, strict: bool = True) -> list[str]:+ """Apply every unapplied migration in order. Returns the ids applied.++ Stops at the first failure rather than continuing, because migration+ N+1 is written against the schema N produced. Continuing past a failure+ is how a lake ends up in a state no migration file describes.++ ``strict=False`` logs and returns what succeeded, for deployments that+ would rather start degraded than not at all. It is not the default: a+ silently-skipped schema change is the exact failure this module exists+ to remove.+ """+ already = await applied_ids(client)+ applied: list[str] = []++ for migration in MIGRATIONS:+ if migration.id in already:+ continue++ if migration.needs_cluster_ddl:+ logger.warning(+ "lake_migrations.cluster_ddl_unhandled id=%s — this migration "+ "needs ON CLUSTER on a multi-replica deployment and will only "+ "apply to the node it is run against",+ migration.id,+ )++ logger.info("lake_migrations.applying id=%s", migration.id)+ try:+ for statement in migration.statements:+ await _execute(client, statement)+ if migration.backfill is not None:+ touched = await migration.backfill(client)+ logger.info("lake_migrations.backfill id=%s rows=%s", migration.id, touched)+ await _record(client, migration, len(migration.statements))+ except Exception as exc: # noqa: BLE001 - reported, then re-raised under strict+ logger.error("lake_migrations.failed id=%s error=%s", migration.id, exc)+ if strict:+ raise+ return applied++ applied.append(migration.id)++ if applied:+ logger.info("lake_migrations.complete applied=%s", ",".join(applied))+ return applied+++async def pending_ids(client: Any) -> list[str]:+ """Migrations this lake has not applied. For health checks and preflight."""+ already = await applied_ids(client)+ return [m.id for m in MIGRATIONS if m.id not in already]diff --git a/services/api/app/db/vector_migrations.py b/services/api/app/db/vector_migrations.pynew file mode 100644index 000000000..70b4660c4--- /dev/null+++ b/services/api/app/db/vector_migrations.py@@ -0,0 +1,212 @@+"""Forward-only schema migrations for the Qdrant vector store.++The third of the three stores with no migration mechanism. Qdrant is the+awkward one, because the change most likely to be needed is the one it+cannot do.++**A collection's vector dimension and distance metric are immutable.**+Changing an embedding model changes the dimension, and there is no+`ALTER COLLECTION`. The migration is: create a new collection, re-embed+every point into it, swap an alias, drop the old one. That is a data+migration measured in hours for a large tenant, not a DDL statement — so+this module models it as one rather than pretending a dimension change is a+schema tweak.++**Aliases are the indirection that makes it survivable.** Code reads+through an alias, never a collection name directly. A re-embed builds+alongside the live collection and the alias flips atomically at the end, so+a failed migration leaves the old collection serving rather than a+half-populated new one.++What *is* cheap: payload indexes, quantization settings, HNSW parameters,+optimizer thresholds. Those are the ordinary migrations, and they are why+this exists even though re-embeds are rare.++Same shape as `graph_migrations.py` and `lake_migrations.py`: numbered,+append-only, applied ids recorded in the store itself. Qdrant has no table+to record them in, so the ledger is a dedicated collection holding one+point per applied migration — small, and it keeps the record in the store+being migrated rather than somewhere that can drift from it.+"""++from __future__ import annotations++import logging+from collections.abc import Awaitable, Callable+from dataclasses import dataclass, field+from datetime import UTC, datetime+from typing import Any++logger = logging.getLogger("aisoc.vector_migrations")++#: Ledger collection. One point per applied migration, keyed by a stable+#: hash of the id so re-recording is idempotent.+MIGRATION_COLLECTION = "aisoc_migrations"++#: Minimal vector config for the ledger. Qdrant requires *some* vector+#: definition; one dimension is the smallest legal answer and the vectors+#: are never searched.+_LEDGER_DIM = 1+++@dataclass(frozen=True)+class VectorMigration:+ """One forward-only change to the vector store.++ ``apply`` receives the client and does the work, rather than a list of+ statements: Qdrant's API is method calls, not a query language, and+ pretending otherwise would mean inventing a DSL whose only user is this+ file.++ ``rebuilds_collection`` marks the expensive class — a dimension or+ metric change that requires re-embedding every point. Flagged so an+ operator can see one coming in ``pending_ids()`` before it starts, since+ the difference between a one-second migration and a four-hour one should+ not be a surprise discovered at deploy time.+ """++ id: str+ description: str+ apply: Callable[[Any], Awaitable[None]]+ rebuilds_collection: bool = False+ tags: tuple[str, ...] = field(default_factory=tuple)+++def _point_id(migration_id: str) -> int:+ """A stable positive integer id for a migration.++ Qdrant point ids are unsigned integers or UUIDs. Hashed with blake2b+ rather than `hash()` for the same reason the sandbox was fixed: CPython+ salts string hashing per process, so a ledger keyed on `hash()` would+ record the same migration under a different id on every restart and the+ runner would re-apply everything.+ """+ import hashlib++ digest = hashlib.blake2b(migration_id.encode(), digest_size=8).digest()+ return int.from_bytes(digest, "big") & 0x7FFF_FFFF_FFFF_FFFF+++async def _ensure_ledger(client: Any) -> None:+ try:+ collections = await client.get_collections()+ names = {c.name for c in getattr(collections, "collections", [])}+ except Exception as exc: # noqa: BLE001 - treated as absent, created below+ logger.warning("vector_migrations.list_failed error=%s", exc)+ names = set()++ if MIGRATION_COLLECTION in names:+ return++ from qdrant_client.models import Distance, VectorParams # noqa: PLC0415++ await client.create_collection(+ collection_name=MIGRATION_COLLECTION,+ vectors_config=VectorParams(size=_LEDGER_DIM, distance=Distance.COSINE),+ )+ logger.info("vector_migrations.ledger_created")+++async def applied_ids(client: Any) -> set[str]:+ """Migration ids this store already has."""+ await _ensure_ledger(client)+ try:+ points, _ = await client.scroll(collection_name=MIGRATION_COLLECTION, limit=1000, with_payload=True)+ except Exception as exc: # noqa: BLE001 - an empty ledger is a normal first run+ logger.warning("vector_migrations.ledger_unreadable error=%s", exc)+ return set()++ return {str(p.payload["migration_id"]) for p in points or [] if getattr(p, "payload", None) and p.payload.get("migration_id")}+++async def _record(client: Any, migration: VectorMigration) -> None:+ from qdrant_client.models import PointStruct # noqa: PLC0415++ await client.upsert(+ collection_name=MIGRATION_COLLECTION,+ points=[+ PointStruct(+ id=_point_id(migration.id),+ vector=[0.0] * _LEDGER_DIM,+ payload={+ "migration_id": migration.id,+ "description": migration.description,+ "applied_at": datetime.now(UTC).isoformat(),+ },+ )+ ],+ )+++async def _apply_baseline(client: Any) -> None:+ """Records the pre-existing collections as the baseline.++ Creates nothing. `QdrantStore` builds its collections on first use, so+ an existing deployment already has them; the point is that the ledger+ agrees, and migration 002 onwards can assume a known starting shape.+ """+ return None+++MIGRATIONS: tuple[VectorMigration, ...] = (+ VectorMigration(+ id="001_baseline",+ description=("Records the collections QdrantStore creates on first use as the " "baseline. Creates nothing."),+ apply=_apply_baseline,+ tags=("baseline",),+ ),+)+++async def run_migrations(client: Any, *, strict: bool = True) -> list[str]:+ """Apply every unapplied migration in order. Returns the ids applied.++ Stops at the first failure: migration N+1 is written against the state N+ produced, and continuing past a failure leaves the store in a shape no+ migration describes.+ """+ already = await applied_ids(client)+ applied: list[str] = []++ for migration in MIGRATIONS:+ if migration.id in already:+ continue++ if migration.rebuilds_collection:+ logger.warning(+ "vector_migrations.rebuild_starting id=%s — this re-embeds "+ "every point and is measured in hours for a large tenant. The "+ "alias keeps the existing collection serving until it "+ "completes.",+ migration.id,+ )++ logger.info("vector_migrations.applying id=%s", migration.id)+ try:+ await migration.apply(client)+ await _record(client, migration)+ except Exception as exc: # noqa: BLE001 - reported, re-raised under strict+ logger.error("vector_migrations.failed id=%s error=%s", migration.id, exc)+ if strict:+ raise+ return applied++ applied.append(migration.id)++ if applied:+ logger.info("vector_migrations.complete applied=%s", ",".join(applied))+ return applied+++async def pending_ids(client: Any) -> list[str]:+ """Migrations this store has not applied. For health checks and preflight."""+ already = await applied_ids(client)+ return [m.id for m in MIGRATIONS if m.id not in already]+++async def pending_rebuilds(client: Any) -> list[str]:+ """Pending migrations that re-embed. Surfaced separately because the+ difference between a one-second migration and a four-hour one should not+ be discovered at deploy time."""+ already = await applied_ids(client)+ return [m.id for m in MIGRATIONS if m.id not in already and m.rebuilds_collection]diff --git a/services/api/app/main.py b/services/api/app/main.pyindex 6d479473d..611ba18fb 100644--- a/services/api/app/main.py+++ b/services/api/app/main.py@@ -23,7 +23,7 @@ from app.core.cors import build_cors_kwargsfrom app.core.logging import configure_loggingfrom app.core.scheduler_lock import scheduler_lockfrom app.core.telemetry import instrument_app-from app.db.clickhouse import close_clickhouse+from app.db.clickhouse import close_clickhouse, init_lake_schemafrom app.db.database import enginefrom app.db.neo4j import close_neo4j, init_neo4jfrom app.graphql.schema import graphql_router@@ -340,6 +340,14 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:except Exception as exc:logger.warning("Neo4j unavailable at startup – graph features disabled", error=str(exc))+ # The lake's schema converges here for the same reason the graph's+ # does: CREATE TABLE IF NOT EXISTS is a no-op on an existing+ # deployment, so without this a new column never lands on one.+ try:+ await init_lake_schema()+ except Exception as exc: # noqa: BLE001 - the lake is not on the critical path+ logger.warning("lake schema migration skipped", error=str(exc))+# Auto-discover plugins from AISOC_PLUGINS_DIRtry:plugin_mgr = get_plugin_manager()diff --git a/services/api/app/services/fleet_health.py b/services/api/app/services/fleet_health.pynew file mode 100644index 000000000..3ce5c9ca4--- /dev/null+++ b/services/api/app/services/fleet_health.py@@ -0,0 +1,258 @@+"""Which parts of this deployment have quietly stopped working.++Every field this needs was already being written — `last_sync`,+`error_count`, `events_ingested`, `oauth_refresh_failures`,+`last_schema_drift_at` — and nothing read them together. The result is the+failure mode this platform is least able to tolerate: a connector stops+polling, alerts stop arriving from that source, and the console looks calm+because an absence of alerts is indistinguishable from an absence of+threats.++The design decision that shapes everything here is **what counts as+stale**. A connector polling every five minutes is broken after twenty; one+polling daily is fine after twenty hours. A single global threshold would+either page constantly on slow connectors or stay silent on fast ones, so+staleness is measured in *missed poll intervals* against each connector's+own configured cadence.++Two things deliberately not inferred:++**A connector that has never synced is not stale.** It is unproven. Those+are different operator actions — one is "something broke", the other is+"this was never finished" — and collapsing them sends people to debug a+working connector that nobody completed setup for.++**A disabled connector is not unhealthy.** Reporting it as degraded trains+people to ignore the degraded count, which is how a real failure gets+missed.+"""++from __future__ import annotations++from dataclasses import asdict, dataclass, field+from datetime import UTC, datetime, timedelta+from enum import Enum+from typing import Any++#: Default cadence when a connector declares none. Matches the scheduler's+#: own default, so staleness is judged against what actually runs.+DEFAULT_POLL_SECONDS = 300++#: Missed intervals before a connector is called stale. Three rather than+#: one: a single missed poll is routine (a vendor rate-limit, a redeploy),+#: and alerting on it produces noise that gets the whole surface muted.+STALE_INTERVALS = 3++#: Missed intervals before it is called failed rather than degraded.+FAILED_INTERVALS = 12++#: Consecutive errors before the error count alone is enough, regardless of+#: recency. A connector erroring every poll but still "recent" is broken.+ERROR_THRESHOLD = 5+++class HealthState(str, Enum):+ HEALTHY = "healthy"+ DEGRADED = "degraded"+ FAILED = "failed"+ #: Enabled but has never completed a poll. Not a failure — setup was+ #: never finished, which is a different action for the operator.+ UNPROVEN = "unproven"+ #: Turned off on purpose. Never counted against fleet health.+ DISABLED = "disabled"+++@dataclass+class ConnectorHealth:+ connector_id: str+ name: str+ connector_type: str+ state: HealthState+ #: Why, in words an operator can act on. Never "unhealthy".+ reason: str+ last_sync: datetime | None+ seconds_since_sync: int | None+ poll_interval_seconds: int+ missed_intervals: float | None+ error_count: int+ events_ingested: int+ oauth_refresh_failures: int = 0+ schema_drift_at: datetime | None = None++ def to_dict(self) -> dict[str, Any]:+ out = asdict(self)+ out["state"] = self.state.value+ for key in ("last_sync", "schema_drift_at"):+ value = out.get(key)+ out[key] = value.isoformat() if isinstance(value, datetime) else None+ return out+++@dataclass+class FleetHealth:+ generated_at: datetime+ connectors: list[ConnectorHealth] = field(default_factory=list)++ @property+ def counts(self) -> dict[str, int]:+ out = {state.value: 0 for state in HealthState}+ for connector in self.connectors:+ out[connector.state.value] += 1+ return out++ @property+ def worst_state(self) -> HealthState:+ """The fleet is as healthy as its least healthy enabled connector.++ Averaging would let nineteen working connectors hide one that+ stopped — and the one that stopped is the whole question.+ """+ for state in (HealthState.FAILED, HealthState.DEGRADED, HealthState.UNPROVEN):+ if any(c.state is state for c in self.connectors):+ return state+ return HealthState.HEALTHY++ def to_dict(self) -> dict[str, Any]:+ return {+ "generated_at": self.generated_at.isoformat(),+ "state": self.worst_state.value,+ "counts": self.counts,+ "connectors": [c.to_dict() for c in self.connectors],+ }+++def _poll_interval(row: Any) -> int:+ config = getattr(row, "connector_config", None) or {}+ raw = config.get("poll_interval_seconds") if isinstance(config, dict) else None+ try:+ interval = int(raw)+ except (TypeError, ValueError):+ return DEFAULT_POLL_SECONDS+ # A zero or negative cadence would make every connector infinitely+ # stale; treat it as unset rather than as a division by zero.+ return interval if interval > 0 else DEFAULT_POLL_SECONDS+++def assess_connector(row: Any, *, now: datetime | None = None) -> ConnectorHealth:+ """Judge one connector against its own configured cadence."""+ now = now or datetime.now(UTC)+ interval = _poll_interval(row)+ last_sync = getattr(row, "last_sync", None)+ error_count = int(getattr(row, "error_count", 0) or 0)+ oauth_failures = int(getattr(row, "oauth_refresh_failures", 0) or 0)++ base = {+ "connector_id": str(getattr(row, "id", "")),+ "name": getattr(row, "name", "") or "",+ "connector_type": getattr(row, "connector_type", "") or "",+ "last_sync": last_sync,+ "poll_interval_seconds": interval,+ "error_count": error_count,+ "events_ingested": int(getattr(row, "events_ingested", 0) or 0),+ "oauth_refresh_failures": oauth_failures,+ "schema_drift_at": getattr(row, "last_schema_drift_at", None),+ }++ if not getattr(row, "is_enabled", True):+ return ConnectorHealth(+ **base,+ state=HealthState.DISABLED,+ reason="Disabled. Not counted against fleet health.",+ seconds_since_sync=None,+ missed_intervals=None,+ )++ if last_sync is None:+ return ConnectorHealth(+ **base,+ state=HealthState.UNPROVEN,+ reason=(+ "Enabled but has never completed a poll. Usually a credential "+ "the vendor accepted at save time but rejects for reads — check "+ "the connectors service log for the vendor's own error."+ ),+ seconds_since_sync=None,+ missed_intervals=None,+ )++ if last_sync.tzinfo is None:+ last_sync = last_sync.replace(tzinfo=UTC)+ elapsed = max(0, int((now - last_sync).total_seconds()))+ missed = round(elapsed / interval, 1)++ # An OAuth refresh failure is terminal in a way a poll failure is not:+ # the credential is gone and no amount of waiting fixes it.+ if oauth_failures >= 3:+ return ConnectorHealth(+ **base,+ state=HealthState.FAILED,+ reason=(f"OAuth refresh has failed {oauth_failures} times. The grant was " f"most likely revoked; reconnect rather than wait."),+ seconds_since_sync=elapsed,+ missed_intervals=missed,+ )++ if missed >= FAILED_INTERVALS:+ return ConnectorHealth(+ **base,+ state=HealthState.FAILED,+ reason=(+ f"No successful poll in {_humanise(elapsed)} "+ f"({missed:g} missed intervals). Alerts from this source have "+ f"stopped arriving."+ ),+ seconds_since_sync=elapsed,+ missed_intervals=missed,+ )++ if error_count >= ERROR_THRESHOLD:+ return ConnectorHealth(+ **base,+ state=HealthState.DEGRADED,+ reason=(f"{error_count} consecutive poll errors. Syncing recently but " f"not cleanly."),+ seconds_since_sync=elapsed,+ missed_intervals=missed,+ )++ if missed >= STALE_INTERVALS:+ return ConnectorHealth(+ **base,+ state=HealthState.DEGRADED,+ reason=(f"Last poll {_humanise(elapsed)} ago against a " f"{_humanise(interval)} cadence ({missed:g} missed intervals)."),+ seconds_since_sync=elapsed,+ missed_intervals=missed,+ )++ return ConnectorHealth(+ **base,+ state=HealthState.HEALTHY,+ reason=f"Last poll {_humanise(elapsed)} ago.",+ seconds_since_sync=elapsed,+ missed_intervals=missed,+ )+++def _humanise(seconds: int) -> str:+ delta = timedelta(seconds=seconds)+ if delta.days:+ return f"{delta.days}d"+ hours, rest = divmod(seconds, 3600)+ if hours:+ return f"{hours}h"+ minutes = rest // 60+ return f"{minutes}m" if minutes else f"{seconds}s"+++def assess_fleet(rows: list[Any], *, now: datetime | None = None) -> FleetHealth:+ now = now or datetime.now(UTC)+ assessed = [assess_connector(row, now=now) for row in rows]+ # Worst first: an operator opening this wants the broken one, not an+ # alphabetical list they have to scan.+ order = {+ HealthState.FAILED: 0,+ HealthState.DEGRADED: 1,+ HealthState.UNPROVEN: 2,+ HealthState.HEALTHY: 3,+ HealthState.DISABLED: 4,+ }+ assessed.sort(key=lambda c: (order[c.state], -(c.seconds_since_sync or 0)))+ return FleetHealth(generated_at=now, connectors=assessed)diff --git a/services/api/tests/test_fleet_health.py b/services/api/tests/test_fleet_health.pynew file mode 100644index 000000000..293d3a70c--- /dev/null+++ b/services/api/tests/test_fleet_health.py@@ -0,0 +1,198 @@+"""Fleet health: the surface that answers "what quietly stopped working".++Every field this reads was already being written and nothing read them+together, which produces the failure this platform is least able to+tolerate — a connector stops polling, alerts from that source stop+arriving, and the console looks calm because an absence of alerts is+indistinguishable from an absence of threats.++The tests concentrate on the two judgements that make this useful rather+than noisy: staleness measured against each connector's own cadence, and+the states that must *not* be reported as failures.+"""++from __future__ import annotations++from dataclasses import dataclass, field+from datetime import UTC, datetime, timedelta++import pytest+from app.services.fleet_health import (+ ERROR_THRESHOLD,+ HealthState,+ assess_connector,+ assess_fleet,+)++NOW = datetime(2026, 9, 22, 12, 0, tzinfo=UTC)+++@dataclass+class Row:+ """Stands in for the Connector ORM row."""++ id: str = "c1"+ name: str = "Prod Okta"+ connector_type: str = "okta"+ is_enabled: bool = True+ last_sync: datetime | None = None+ error_count: int = 0+ events_ingested: int = 100+ oauth_refresh_failures: int = 0+ last_schema_drift_at: datetime | None = None+ connector_config: dict = field(default_factory=dict)+++def ago(**kwargs) -> datetime:+ return NOW - timedelta(**kwargs)+++class TestStalenessIsRelativeToCadence:+ """A global threshold either pages constantly on slow connectors or+ stays silent on fast ones."""++ def test_a_fast_connector_is_stale_after_twenty_minutes(self) -> None:+ row = Row(last_sync=ago(minutes=20), connector_config={"poll_interval_seconds": 300})+ assert assess_connector(row, now=NOW).state is HealthState.DEGRADED++ def test_a_daily_connector_is_healthy_after_twenty_minutes(self) -> None:+ row = Row(last_sync=ago(minutes=20), connector_config={"poll_interval_seconds": 86400})+ assert assess_connector(row, now=NOW).state is HealthState.HEALTHY++ def test_a_daily_connector_is_stale_after_four_days(self) -> None:+ row = Row(last_sync=ago(days=4), connector_config={"poll_interval_seconds": 86400})+ assert assess_connector(row, now=NOW).state is HealthState.DEGRADED++ def test_one_missed_poll_is_not_a_finding(self) -> None:+ """Alerting on a single missed poll produces noise that gets the+ whole surface muted."""+ row = Row(last_sync=ago(minutes=6), connector_config={"poll_interval_seconds": 300})+ assert assess_connector(row, now=NOW).state is HealthState.HEALTHY++ def test_long_enough_becomes_failed_not_degraded(self) -> None:+ row = Row(last_sync=ago(hours=2), connector_config={"poll_interval_seconds": 300})+ result = assess_connector(row, now=NOW)+ assert result.state is HealthState.FAILED+ assert "stopped arriving" in result.reason++ @pytest.mark.parametrize("interval", [0, -1, None, "", "abc"])+ def test_a_nonsense_cadence_falls_back_rather_than_dividing_by_zero(self, interval: object) -> None:+ row = Row(+ last_sync=ago(minutes=1),+ connector_config={"poll_interval_seconds": interval},+ )+ assert assess_connector(row, now=NOW).state is HealthState.HEALTHY+++class TestStatesThatAreNotFailures:+ def test_never_synced_is_unproven_not_failed(self) -> None:+ """ "Something broke" and "this was never finished" are different+ operator actions."""+ result = assess_connector(Row(last_sync=None), now=NOW)+ assert result.state is HealthState.UNPROVEN+ assert "never completed a poll" in result.reason++ def test_disabled_is_not_unhealthy(self) -> None:+ """Reporting it as degraded trains people to ignore the degraded+ count, which is how a real failure gets missed."""+ row = Row(is_enabled=False, last_sync=ago(days=90))+ assert assess_connector(row, now=NOW).state is HealthState.DISABLED++ def test_a_disabled_connector_does_not_drag_down_the_fleet(self) -> None:+ fleet = assess_fleet(+ [Row(id="a", last_sync=ago(minutes=1)), Row(id="b", is_enabled=False)],+ now=NOW,+ )+ assert fleet.worst_state is HealthState.HEALTHY+++class TestErrorsAndCredentials:+ def test_errors_matter_even_when_syncing_recently(self) -> None:+ """A connector erroring every poll but still "recent" is broken."""+ row = Row(last_sync=ago(minutes=1), error_count=ERROR_THRESHOLD)+ assert assess_connector(row, now=NOW).state is HealthState.DEGRADED++ def test_repeated_oauth_failure_is_terminal(self) -> None:+ """The credential is gone; no amount of waiting fixes it, so this+ outranks recency."""+ row = Row(last_sync=ago(minutes=1), oauth_refresh_failures=3)+ result = assess_connector(row, now=NOW)+ assert result.state is HealthState.FAILED+ assert "reconnect rather than wait" in result.reason++ def test_one_oauth_failure_is_not_terminal(self) -> None:+ row = Row(last_sync=ago(minutes=1), oauth_refresh_failures=1)+ assert assess_connector(row, now=NOW).state is HealthState.HEALTHY+++class TestFleetRollup:+ def test_one_broken_connector_is_not_averaged_away(self) -> None:+ """Nineteen working connectors must not hide the one that stopped —+ the one that stopped is the whole question."""+ rows = [Row(id=f"ok{i}", last_sync=ago(minutes=1)) for i in range(19)]+ rows.append(Row(id="broken", last_sync=ago(days=3)))+ assert assess_fleet(rows, now=NOW).worst_state is HealthState.FAILED++ def test_worst_first_ordering(self) -> None:+ """An operator opening this wants the broken one, not an+ alphabetical list to scan."""+ rows = [+ Row(id="healthy", last_sync=ago(minutes=1)),+ Row(id="failed", last_sync=ago(days=3)),+ Row(id="unproven", last_sync=None),+ Row(id="degraded", last_sync=ago(minutes=20)),+ ]+ states = [c.state for c in assess_fleet(rows, now=NOW).connectors]+ assert states == [+ HealthState.FAILED,+ HealthState.DEGRADED,+ HealthState.UNPROVEN,+ HealthState.HEALTHY,+ ]++ def test_counts_cover_every_state(self) -> None:+ fleet = assess_fleet([Row(last_sync=ago(minutes=1))], now=NOW)+ assert set(fleet.counts) == {s.value for s in HealthState}+ assert fleet.counts["healthy"] == 1++ def test_an_empty_fleet_is_healthy_not_broken(self) -> None:+ fleet = assess_fleet([], now=NOW)+ assert fleet.worst_state is HealthState.HEALTHY++ def test_serialises(self) -> None:+ import json++ fleet = assess_fleet([Row(last_sync=ago(minutes=1))], now=NOW)+ payload = json.loads(json.dumps(fleet.to_dict()))+ assert payload["state"] == "healthy"+ assert payload["connectors"][0]["last_sync"].startswith("2026-")++ def test_a_naive_timestamp_does_not_crash(self) -> None:+ """Postgres can hand back a naive datetime depending on the column+ and driver; subtracting it from an aware one raises."""+ row = Row(last_sync=datetime(2026, 9, 22, 11, 59))+ assert assess_connector(row, now=NOW).state is HealthState.HEALTHY+++class TestReasons:+ def test_every_state_explains_itself_actionably(self) -> None:+ """ "Unhealthy" tells an operator nothing they can act on."""+ rows = [+ Row(id="a", last_sync=ago(minutes=1)),+ Row(id="b", last_sync=ago(minutes=20)),+ Row(id="c", last_sync=ago(days=3)),+ Row(id="d", last_sync=None),+ Row(id="e", is_enabled=False),+ ]+ for connector in assess_fleet(rows, now=NOW).connectors:+ assert connector.reason.strip(), f"{connector.connector_id} has no reason"+ # "Unhealthy" is a restatement of the state, not a reason.+ assert "unhealthy" not in connector.reason.lower()++ if connector.state in (HealthState.HEALTHY, HealthState.DISABLED):+ continue+ # A non-healthy state has to say what is wrong in terms an+ # operator can act on — a duration, a count, or a next step.+ assert any(+ token in connector.reason.lower() for token in ("poll", "credential", "oauth", "error", "reconnect")+ ), f"{connector.connector_id}: {connector.reason!r} is not actionable"diff --git a/services/api/tests/test_store_migrations.py b/services/api/tests/test_store_migrations.pynew file mode 100644index 000000000..9b93284b3--- /dev/null+++ b/services/api/tests/test_store_migrations.py@@ -0,0 +1,177 @@+"""Every store that holds schema must have a way to change it.++`clickhouse/001_init.sql` is four `CREATE TABLE IF NOT EXISTS` statements+and zero `ALTER TABLE`; `lake_writer` self-heals with the same idiom. Both+are no-ops once the table exists, so a new column lands on a fresh+deployment and silently does not land on an existing one — and nothing+notices until a query selects a column that is not there, long after the+deploy that was meant to add it.++The tests are mostly about the properties that make a runner trustworthy+rather than about any individual migration: ids never move, a failure stops+the run, and the ledger is read from the store being migrated rather than+somewhere that can drift from it.+"""++from __future__ import annotations++from typing import Any+from unittest.mock import AsyncMock++import pytest+from app.db import graph_migrations, lake_migrations, vector_migrations++ALL_RUNNERS = (+ ("neo4j", graph_migrations),+ ("clickhouse", lake_migrations),+ ("qdrant", vector_migrations),+)+++class TestEveryStoreHasARunner:+ """The plan called the absence of these a blocker, not a follow-up."""++ @pytest.mark.parametrize(("store", "module"), ALL_RUNNERS, ids=lambda x: str(x))+ def test_runner_exposes_the_same_interface(self, store: str, module: Any) -> None:+ """Three stores, one shape. An operator should not have to learn+ three vocabularies to answer "what has this deployment applied"."""+ for name in ("MIGRATIONS", "run_migrations", "applied_ids", "pending_ids"):+ assert hasattr(module, name), f"{store} runner has no {name}"++ @pytest.mark.parametrize(("store", "module"), ALL_RUNNERS, ids=lambda x: str(x))+ def test_ids_are_unique_and_ordered(self, store: str, module: Any) -> None:+ ids = [m.id for m in module.MIGRATIONS]+ assert len(ids) == len(set(ids)), f"{store} has duplicate migration ids"+ assert ids == sorted(ids), (+ f"{store} migrations are not in id order; the runner applies them "+ f"in tuple order, so a mismatch means they run in an order the "+ f"numbering does not describe"+ )++ @pytest.mark.parametrize(("store", "module"), ALL_RUNNERS, ids=lambda x: str(x))+ def test_every_migration_is_described(self, store: str, module: Any) -> None:+ """A migration whose purpose nobody wrote down cannot be reviewed,+ and cannot be judged safe to re-run."""+ for migration in module.MIGRATIONS:+ assert migration.description.strip(), f"{store}/{migration.id}"++ @pytest.mark.parametrize(("store", "module"), ALL_RUNNERS, ids=lambda x: str(x))+ def test_the_first_migration_is_a_baseline(self, store: str, module: Any) -> None:+ """An existing deployment must converge rather than re-run unknown+ statements against a schema that already has them."""+ first = module.MIGRATIONS[0]+ assert "baseline" in first.id or "core" in first.id, (+ f"{store}'s first migration is {first.id!r}; it should record the " f"pre-existing schema so existing deployments converge"+ )+++class TestLakeRunner:+ @pytest.fixture+ def client(self) -> AsyncMock:+ c = AsyncMock()+ c.execute = AsyncMock(return_value=[])+ return c++ async def test_applies_and_records(self, client: AsyncMock) -> None:+ applied = await lake_migrations.run_migrations(client)+ assert applied == ["001_baseline"]+ statements = [call.args[0] for call in client.execute.await_args_list]+ assert any("_migrations" in s for s in statements), "ledger was not created"+ assert any("INSERT INTO" in s for s in statements), "nothing was recorded"++ async def test_an_applied_migration_is_not_reapplied(self, client: AsyncMock) -> None:+ client.execute = AsyncMock(return_value=[("001_baseline",)])+ assert await lake_migrations.run_migrations(client) == []++ async def test_a_failure_stops_the_run(self, client: AsyncMock) -> None:+ """Migration N+1 is written against the schema N produced."""+ calls = {"n": 0}++ async def failing(sql: str, params: Any = None) -> Any:+ calls["n"] += 1+ if "CREATE DATABASE" in sql:+ raise RuntimeError("no such cluster")+ return []++ client.execute = failing+ with pytest.raises(RuntimeError, match="no such cluster"):+ await lake_migrations.run_migrations(client)++ async def test_non_strict_returns_what_succeeded(self, client: AsyncMock) -> None:+ async def failing(sql: str, params: Any = None) -> Any:+ if "CREATE DATABASE" in sql:+ raise RuntimeError("boom")+ return []++ client.execute = failing+ assert await lake_migrations.run_migrations(client, strict=False) == []++ async def test_drives_a_command_style_client(self) -> None:+ """Two ClickHouse client shapes are in use across this repo. A+ runner that speaks one silently does not run in half the+ deployments."""+ c = AsyncMock()+ del c.execute+ c.command = AsyncMock(return_value=[])+ await lake_migrations.run_migrations(c)+ assert c.command.await_count > 0++ async def test_an_undrivable_client_is_a_loud_failure(self) -> None:+ class Opaque:+ pass++ with pytest.raises(TypeError, match="neither execute"):+ await lake_migrations._execute(Opaque(), "SELECT 1")++ async def test_an_unreadable_ledger_is_a_first_run_not_a_crash(self, client: AsyncMock) -> None:+ async def execute(sql: str, params: Any = None) -> Any:+ if "SELECT migration_id" in sql:+ raise RuntimeError("table does not exist")+ return []++ client.execute = execute+ assert await lake_migrations.applied_ids(client) == set()++ @pytest.mark.parametrize("row", [("001_baseline",), {"migration_id": "001_baseline"}, "001_baseline"])+ async def test_reads_every_driver_row_shape(self, client: AsyncMock, row: Any) -> None:+ """Drivers return tuples, dicts or scalars. Reading one shape means+ the runner re-applies everything against the others."""+ client.execute = AsyncMock(return_value=[row])+ assert await lake_migrations.applied_ids(client) == {"001_baseline"}+++class TestVectorRunner:+ @pytest.fixture+ def client(self) -> AsyncMock:+ c = AsyncMock()+ c.get_collections = AsyncMock(return_value=type("R", (), {"collections": []})())+ c.scroll = AsyncMock(return_value=([], None))+ return c++ async def test_applies_and_records(self, client: AsyncMock) -> None:+ pytest.importorskip("qdrant_client")+ applied = await vector_migrations.run_migrations(client)+ assert applied == ["001_baseline"]+ client.upsert.assert_awaited_once()++ async def test_point_ids_are_stable_across_processes(self) -> None:+ """Keyed on hash() the ledger would record the same migration under a+ different id every restart, and the runner would re-apply+ everything — the same salted-hash bug the sandbox had."""+ first = vector_migrations._point_id("001_baseline")+ assert first == vector_migrations._point_id("001_baseline")+ assert first != vector_migrations._point_id("002_other")+ assert first > 0, "Qdrant point ids must be unsigned"++ async def test_a_rebuild_is_flagged_before_it_starts(self) -> None:+ """A dimension change re-embeds every point. The difference between+ a one-second migration and a four-hour one should not be discovered+ at deploy time."""+ assert hasattr(vector_migrations, "pending_rebuilds")+ for migration in vector_migrations.MIGRATIONS:+ assert isinstance(migration.rebuilds_collection, bool)++ async def test_an_unreadable_ledger_is_a_first_run(self, client: AsyncMock) -> None:+ pytest.importorskip("qdrant_client")+ client.scroll = AsyncMock(side_effect=RuntimeError("collection missing"))+ assert await vector_migrations.applied_ids(client) == set()diff --git a/services/fusion/app/services/derived_fields.py b/services/fusion/app/services/derived_fields.pynew file mode 100644index 000000000..25415f7b4--- /dev/null+++ b/services/fusion/app/services/derived_fields.py@@ -0,0 +1,185 @@+"""Fields the detection engine can compute from an event it already has.++138 of 833 executable rules match on fields no connector emits, so they can+never fire — a rule that never fires and a rule with nothing to fire on look+identical from outside, which is why the number went unnoticed for so long.++Most of those 138 genuinely need something that does not exist: a windowed+evaluator (73), identity enrichment (24), per-tenant allowlists (15). This+module handles the ones that need nothing at all — where the answer is+already in the event and only the arithmetic is missing.++Two families:++**Field comparisons.** `actor_eq_target`, `actor_uid_neq_owner_uid`,+`object_tenant_id_eq_caller_tenant_id`. The rule wants to know whether two+fields of the same event agree. Both values are present; nothing was+comparing them.++**Time of day.** `is_business_hours`, `is_after_hours`, `is_weekend`. A pure+function of the event timestamp.++The rule that governs everything here: **an underivable field is absent, not+false.** If either side of a comparison is missing, the derived key is not+set, so a rule matching `actor_eq_target: false` does not fire on an event+where neither actor nor target is known. Defaulting to `False` would make+every event with missing data look like a cross-account access, which is a+detection that fires on ignorance.+"""++from __future__ import annotations++import re+from datetime import datetime+from typing import Any++#: `<left>_eq_<right>` / `<left>_neq_<right>`. Non-greedy on the left so+#: `actor_uid_neq_owner_uid` splits at the *first* separator, giving+#: (actor_uid, owner_uid) rather than (actor, uid_neq_owner_uid).+_COMPARISON_RE = re.compile(r"^(?P<left>.+?)_(?P<op>eq|neq)_(?P<right>.+)$")++#: Business hours in the tenant's local time. A default rather than a+#: constant: "outside business hours" means nothing without knowing whose+#: business, and a rule that assumes UTC fires all night for half the world.+DEFAULT_BUSINESS_START_HOUR = 8+DEFAULT_BUSINESS_END_HOUR = 18++#: Timestamp fields, in the order they are trusted. `event_time` is when the+#: thing happened; `ingest_time` is when we heard about it, and using the+#: latter would make a batch import at 03:00 look like a night-time attack.+_TIME_FIELDS = ("event_time", "timestamp", "time", "@timestamp", "ingest_time")+++def _parse_time(event: dict[str, Any]) -> datetime | None:+ for field in _TIME_FIELDS:+ raw = event.get(field)+ if raw is None:+ continue+ if isinstance(raw, datetime):+ return raw+ if isinstance(raw, int | float):+ try:+ # Milliseconds if it is far too large to be seconds.+ seconds = raw / 1000 if raw > 1e11 else raw+ return datetime.fromtimestamp(seconds) # noqa: DTZ006 - local by design+ except (OSError, ValueError, OverflowError):+ continue+ if isinstance(raw, str):+ try:+ return datetime.fromisoformat(raw.replace("Z", "+00:00"))+ except ValueError:+ continue+ return None+++def _normalise(value: Any) -> Any:+ """Compare case-insensitively for strings, exactly for everything else.++ `Alice` and `alice` are the same principal in every directory this+ platform reads, and a comparison that says otherwise produces a+ cross-account finding for a casing difference.+ """+ return value.strip().lower() if isinstance(value, str) else value+++def comparison_fields(event: dict[str, Any], requested: set[str] | None = None) -> dict[str, bool]:+ """Compute `<a>_eq_<b>` / `<a>_neq_<b>` keys the rules ask for.++ ``requested`` bounds the work to the field names some rule actually+ matches on. Without it this would have to enumerate every pair of keys+ in the event, which is quadratic and produces thousands of keys nothing+ reads.+ """+ if not requested:+ return {}++ out: dict[str, bool] = {}+ for name in requested:+ match = _COMPARISON_RE.match(name)+ if not match:+ continue+ left = event.get(match.group("left"))+ right = event.get(match.group("right"))+ # Absent, not false. A rule must not fire because we did not know.+ if left is None or right is None:+ continue+ equal = _normalise(left) == _normalise(right)+ out[name] = equal if match.group("op") == "eq" else not equal+ return out+++def time_of_day_fields(+ event: dict[str, Any],+ *,+ business_start: int = DEFAULT_BUSINESS_START_HOUR,+ business_end: int = DEFAULT_BUSINESS_END_HOUR,+) -> dict[str, bool]:+ """Compute `is_business_hours`, `is_after_hours` and `is_weekend`.++ Returns nothing when the event carries no parseable timestamp, for the+ same reason as above: an unknown time is not "outside business hours".+ """+ when = _parse_time(event)+ if when is None:+ return {}++ weekend = when.weekday() >= 5+ in_hours = (not weekend) and business_start <= when.hour < business_end+ return {+ "is_business_hours": in_hours,+ "is_after_hours": not in_hours,+ "is_weekend": weekend,+ }+++def enrich(+ event: dict[str, Any],+ requested: set[str] | None = None,+ *,+ business_start: int = DEFAULT_BUSINESS_START_HOUR,+ business_end: int = DEFAULT_BUSINESS_END_HOUR,+) -> dict[str, Any]:+ """Return ``event`` plus every derivable field the rules ask for.++ Non-destructive: a derived key never overwrites one the connector+ supplied. If a vendor genuinely emits `is_business_hours`, its value is+ the vendor's answer about its own tenant's hours, which is better than+ ours.+ """+ derived: dict[str, Any] = {}+ derived.update(time_of_day_fields(event, business_start=business_start, business_end=business_end))+ derived.update(comparison_fields(event, requested))++ if not derived:+ return event+ return {**derived, **event}+++def requested_derived_fields(rules: list[dict[str, Any]]) -> set[str]:+ """Collect the derivable field names the ruleset matches on.++ Computed once at ruleset load rather than per event: the set changes+ when rules change, not when traffic arrives, and doing it per event was+ the difference between a cheap enrichment and a hot-path regex over+ every rule for every event.+ """+ from app.services.detection_matcher import _split_op # noqa: PLC0415++ names: set[str] = set()++ def walk(clause: Any) -> None:+ if isinstance(clause, dict):+ for key, value in clause.items():+ if key in {"any_of", "all_of", "not"}:+ walk(value)+ continue+ field, _ = _split_op(key)+ if _COMPARISON_RE.match(field) or field.startswith("is_"):+ names.add(field)+ elif isinstance(clause, list):+ for item in clause:+ walk(item)++ for rule in rules:+ walk(rule.get("match_when") or {})+ return namesdiff --git a/services/fusion/app/services/detection_engine.py b/services/fusion/app/services/detection_engine.pyindex aa5c43ca6..effa84aef 100644--- a/services/fusion/app/services/detection_engine.py+++ b/services/fusion/app/services/detection_engine.py@@ -37,6 +37,7 @@ from typing import Anyimport structlogfrom app.models.alert import AlertSeverity, RawAlert+from app.services.derived_fields import enrich, requested_derived_fieldsfrom app.services.detection_matcher import matchesfrom app.services.provenance import extract_provenance@@ -76,6 +77,10 @@ class DetectionEngine:def __init__(self, rules: list[dict[str, Any]] | None = None) -> None:self._rules: list[dict[str, Any]] = rules if rules is not None else _load_ruleset()+ # Computed once at load, not per event: the set changes when rules+ # change, not when traffic arrives. Doing it per event would mean a+ # regex over every rule's clauses for every ingested event.+ self._derived_wanted: set[str] = requested_derived_fields(self._rules)@propertydef rule_count(self) -> int:@@ -149,6 +154,11 @@ class DetectionEngine:if not isinstance(ocsf, dict):return []fields = self._raw_fields(ocsf)+ # 17 rules match on comparisons between two fields of the same event+ # (actor_eq_target, actor_uid_neq_owner_uid) or on time of day. Both+ # values were already present and nothing was computing the answer,+ # so those rules could never fire.+ fields = enrich(fields, self._derived_wanted)hits: list[DetectionHit] = []for rule in self._candidates(""):try:diff --git a/services/fusion/app/services/detection_matcher.py b/services/fusion/app/services/detection_matcher.pyindex fb9800f78..cb44434a2 100644--- a/services/fusion/app/services/detection_matcher.py+++ b/services/fusion/app/services/detection_matcher.py@@ -40,6 +40,14 @@ OPERATORS: list[tuple[str, str, str]] = sorted(("_has_any", "has_any", "HAS_ANY"),("_not_in", "not_in", "NOT IN"),("_match", "match", "MATCH"),+ # `neq` is used by rules in the shipped corpus and has never+ # existed here, so `approver_role_neq: "codeowner"` was read as a+ # field literally named `approver_role_neq` — which nothing+ # emits, so the rule could not fire. There is deliberately no+ # `_eq` counterpart: bare equality is already the default, and+ # adding the suffix would split any field whose name happens to+ # end in `_eq` for no gain.+ ("_neq", "neq", "!="),("_gte", "gte", ">="),("_lte", "lte", "<="),("_in", "in", "IN"),@@ -70,6 +78,14 @@ def _check(field: str, op: str, expected: Any, event: dict[str, Any]) -> bool:return actual is Nonereturn actual == expected+ if op == "neq":+ # A missing field is not "different from X". Returning True here+ # would make every rule using neq fire on every event that lacks+ # the field, which is a detection that fires on absence of data.+ if actual is None:+ return False+ return actual != expected+if op in {"gt", "gte", "lt", "lte"}:if not isinstance(actual, int | float) or isinstance(actual, bool):return Falsediff --git a/services/fusion/tests/test_alert_reduction_real.py b/services/fusion/tests/test_alert_reduction_real.pynew file mode 100644index 000000000..d505743cb--- /dev/null+++ b/services/fusion/tests/test_alert_reduction_real.py@@ -0,0 +1,198 @@+"""Alert reduction, measured against the correlation logic that actually runs.++The existing measurement in `services/agents/tests/test_alert_reduction.py`+is honest about being synthetic and has always carried a PARTIAL row saying+it "gates an in-test fusion re-impl, not `services/fusion`". Reading it+closely, the gap is wider than that wording suggests: the test does not+merely reimplement fusion's grouping, it implements **different** grouping.++Its docstring describes four tiers keyed on `(rule_id, host, user)` with+10/30/5-minute windows. `RawAlert.correlation_key()` — the method+`Correlator` actually calls — keys on `{tenant}:{entity}:{tactic}`, where+entity is the first of src_ip, hostname, username, domain. Different+dimensions, different windows, different answer. The published+alert-reduction figure therefore described an algorithm the product does+not run.++This measures the real one. `Correlator` itself needs Redis, but Redis is+where it *stores* incidents — the grouping decision is entirely+`correlation_key()` plus the configured window, so the ratio can be+computed from the real method without the storage layer.++The number this produces is a property of a synthetic workload and is+labelled as such, exactly as the original was. What changes is that it is+now a property of *our* algorithm.+"""++from __future__ import annotations++import hashlib+from datetime import UTC, datetime, timedelta+from uuid import UUID++from app.models.alert import AlertSeverity, RawAlert++TENANT = UUID("11111111-1111-1111-1111-111111111111")++#: Matches the production default (`CORRELATION_WINDOW_SECONDS`). Read as a+#: literal rather than from settings so the measurement does not silently+#: change meaning when an operator tunes their own deployment.+WINDOW = timedelta(seconds=3600)++#: Alert count. Same order as the original so the two numbers are+#: comparable, which is the point of keeping both.+STREAM_SIZE = 1000++TACTICS = [+ "initial-access",+ "execution",+ "persistence",+ "credential-access",+ "lateral-movement",+ "exfiltration",+]+++def _seeded(n: int, salt: str, modulus: int) -> int:+ """Deterministic index. blake2b, not hash(): CPython salts string+ hashing per process, so a `hash()`-seeded stream would produce a+ different reduction ratio on every run and the figure would be+ unreproducible — the same defect the sandbox had."""+ digest = hashlib.blake2b(f"{salt}:{n}".encode(), digest_size=8).digest()+ return int.from_bytes(digest, "big") % modulus+++def build_stream(size: int = STREAM_SIZE) -> list[tuple[datetime, RawAlert]]:+ """A noisy alert stream with the shapes fusion is supposed to collapse.++ Generated independently of the correlation logic — the point of the+ measurement is that the stream does not know how it will be grouped.+ """+ start = datetime(2026, 9, 22, 8, 0, tzinfo=UTC)+ stream: list[tuple[datetime, RawAlert]] = []++ for n in range(size):+ # A burst: many alerts from one host in a tight window, the case+ # correlation exists for.+ if n % 10 < 4:+ host = f"ws-{_seeded(n, 'burst-host', 12):03d}"+ offset = timedelta(minutes=_seeded(n, "burst-time", 45))+ # Scattered singletons that should survive as distinct incidents.+ else:+ host = f"ws-{_seeded(n, 'wide-host', 400):03d}"+ offset = timedelta(minutes=_seeded(n, "wide-time", 1440))++ stream.append(+ (+ start + offset,+ RawAlert(+ tenant_id=TENANT,+ source="synthetic",+ title=f"alert {n}",+ hostname=host,+ username=f"u{_seeded(n, 'user', 60)}",+ severity=AlertSeverity.MEDIUM,+ mitre_tactics=[TACTICS[_seeded(n, "tactic", len(TACTICS))]],+ ),+ )+ )+ return stream+++def group_by_real_key(stream: list[tuple[datetime, RawAlert]], window: timedelta = WINDOW) -> int:+ """Incident count under the real correlation key and window.++ Mirrors what `Correlator.correlate` does: look up the incident for this+ alert's correlation key, join it if one is open, otherwise open a new+ one. Redis is the lookup table; the decision is the key.+ """+ open_incidents: dict[str, datetime] = {}+ incidents = 0++ for when, alert in sorted(stream, key=lambda pair: pair[0]):+ key = alert.correlation_key()+ opened = open_incidents.get(key)+ if opened is not None and when - opened <= window:+ continue+ open_incidents[key] = when+ incidents += 1++ return incidents+++def test_reduction_is_measured_against_the_real_correlation_key() -> None:+ """The measurement runs `RawAlert.correlation_key()`, not a copy of it.++ A reimplementation can drift from the thing it stands for without any+ test failing, and this one had — it grouped on different dimensions+ entirely.+ """+ stream = build_stream()+ incidents = group_by_real_key(stream)+ reduction = 1 - (incidents / len(stream))++ print(f"\n[eval] alert reduction (real correlation_key): " f"{len(stream)} alerts -> {incidents} incidents = {reduction:.1%}")++ # Bounded on both sides. A floor alone would be satisfied by a key that+ # collapses everything into one incident, which is 99.9% reduction and+ # a useless SOC.+ assert 0.20 <= reduction <= 0.95, (+ f"reduction {reduction:.1%} is outside the plausible band; either "+ f"correlation stopped grouping or it is collapsing unrelated alerts"+ )+++def test_the_key_is_the_products_key_not_a_local_one() -> None:+ """Pins the dimensions, so a change to `correlation_key()` that alters+ what gets grouped cannot pass silently."""+ alert = RawAlert(+ tenant_id=TENANT,+ source="s",+ title="t",+ src_ip="10.0.0.1",+ hostname="ws-1",+ username="alice",+ mitre_tactics=["execution"],+ )+ assert alert.correlation_key() == f"{TENANT}:10.0.0.1:execution"+++def test_entity_precedence_is_src_ip_then_hostname() -> None:+ """Which entity wins decides what groups with what, so it is worth+ pinning rather than inferring from the reduction number."""+ base = {"tenant_id": TENANT, "source": "s", "title": "t", "mitre_tactics": ["exec"]}+ assert RawAlert(**base, hostname="ws-1").correlation_key().endswith("ws-1:exec")+ assert RawAlert(**base, src_ip="10.0.0.1", hostname="ws-1").correlation_key().endswith("10.0.0.1:exec")+++def test_alerts_without_a_tactic_do_not_all_collapse_together() -> None:+ """`correlation_key()` falls back to the literal "unknown" for both+ entity and tactic. Two unrelated alerts each missing one field still+ differ on the other; two missing both would merge, which is worth+ knowing rather than discovering in production."""+ a = RawAlert(tenant_id=TENANT, source="s", title="a", hostname="ws-1")+ b = RawAlert(tenant_id=TENANT, source="s", title="b", hostname="ws-2")+ assert a.correlation_key() != b.correlation_key()++ bare_a = RawAlert(tenant_id=TENANT, source="s", title="a")+ bare_b = RawAlert(tenant_id=TENANT, source="s", title="b")+ assert bare_a.correlation_key() == bare_b.correlation_key(), (+ "two alerts with no entity and no tactic share a key and will merge; "+ "this is current behaviour, pinned so a change to it is deliberate"+ )+++def test_the_stream_is_reproducible() -> None:+ """A measurement that moves between runs is not a measurement."""+ first = group_by_real_key(build_stream())+ second = group_by_real_key(build_stream())+ assert first == second+++def test_a_wider_window_reduces_more() -> None:+ """Sanity on the window's direction. If this inverts, the grouping is+ not doing what the name says."""+ stream = build_stream()+ narrow = group_by_real_key(stream, timedelta(minutes=5))+ wide = group_by_real_key(stream, timedelta(hours=24))+ assert wide <= narrowdiff --git a/services/fusion/tests/test_derived_fields.py b/services/fusion/tests/test_derived_fields.pynew file mode 100644index 000000000..447e3d875--- /dev/null+++ b/services/fusion/tests/test_derived_fields.py@@ -0,0 +1,213 @@+"""Fields the engine can compute, and the ones it must refuse to guess.++138 of 833 executable rules matched on fields no connector emits. Most need+something that does not exist — a windowed evaluator, identity enrichment,+per-tenant allowlists. A handful needed nothing: both values were already+in the event and nothing was comparing them.++The rule that governs this module, and most of what is tested below: **an+underivable field is absent, not false.** Defaulting a missing comparison to+`False` would make every event with incomplete data satisfy+`actor_eq_target: false` — a cross-account-access detection that fires on+ignorance, on every tenant, forever.+"""++from __future__ import annotations++from datetime import datetime++import pytest+from app.services.derived_fields import (+ comparison_fields,+ enrich,+ requested_derived_fields,+ time_of_day_fields,+)+++class TestComparisons:+ def test_equal_fields_compare_equal(self) -> None:+ event = {"actor": "alice", "target": "alice"}+ assert comparison_fields(event, {"actor_eq_target"}) == {"actor_eq_target": True}++ def test_neq_is_the_inverse(self) -> None:+ event = {"actor_uid": "1000", "owner_uid": "0"}+ result = comparison_fields(event, {"actor_uid_neq_owner_uid"})+ assert result == {"actor_uid_neq_owner_uid": True}++ def test_a_missing_operand_yields_no_key_rather_than_false(self) -> None:+ """The failure that would fire on every incomplete event."""+ event = {"actor": "alice"}+ assert comparison_fields(event, {"actor_eq_target"}) == {}++ def test_both_operands_missing_yields_no_key(self) -> None:+ assert comparison_fields({}, {"actor_eq_target"}) == {}++ def test_case_differences_are_the_same_principal(self) -> None:+ """`Alice` and `alice` are one account in every directory we read;+ saying otherwise produces a finding for a casing difference."""+ event = {"actor": "Alice", "target": "alice "}+ assert comparison_fields(event, {"actor_eq_target"})["actor_eq_target"] is True++ def test_splits_at_the_first_separator(self) -> None:+ """`actor_uid_neq_owner_uid` is (actor_uid, owner_uid), not+ (actor, uid_neq_owner_uid)."""+ event = {"actor_uid": "1", "owner_uid": "1"}+ assert comparison_fields(event, {"actor_uid_neq_owner_uid"}) == {"actor_uid_neq_owner_uid": False}++ def test_only_requested_fields_are_computed(self) -> None:+ """Enumerating every pair of event keys is quadratic and produces+ thousands of keys nothing reads."""+ event = {"a": 1, "b": 1, "c": 1}+ assert comparison_fields(event, {"a_eq_b"}) == {"a_eq_b": True}++ def test_no_request_means_no_work(self) -> None:+ assert comparison_fields({"a": 1, "b": 1}, None) == {}++ def test_a_name_that_is_not_a_comparison_is_ignored(self) -> None:+ assert comparison_fields({"user_name": "x"}, {"user_name"}) == {}++ @pytest.mark.parametrize(+ ("left", "right", "equal"),+ [(0, 0, True), (0, False, True), (1, "1", False), (None, None, None)],+ )+ def test_non_string_operands(self, left: object, right: object, equal: object) -> None:+ event = {}+ if left is not None:+ event["a"] = left+ if right is not None:+ event["b"] = right+ result = comparison_fields(event, {"a_eq_b"})+ if equal is None:+ assert result == {}+ else:+ assert result["a_eq_b"] is equal+++class TestTimeOfDay:+ def test_a_weekday_afternoon_is_business_hours(self) -> None:+ # 2026-09-22 is a Tuesday.+ result = time_of_day_fields({"event_time": datetime(2026, 9, 22, 14, 0)})+ assert result["is_business_hours"] is True+ assert result["is_after_hours"] is False+ assert result["is_weekend"] is False++ def test_a_weekday_night_is_not(self) -> None:+ result = time_of_day_fields({"event_time": datetime(2026, 9, 22, 3, 0)})+ assert result["is_business_hours"] is False+ assert result["is_after_hours"] is True++ def test_a_saturday_afternoon_is_not_business_hours(self) -> None:+ # 2026-09-26 is a Saturday.+ result = time_of_day_fields({"event_time": datetime(2026, 9, 26, 14, 0)})+ assert result["is_weekend"] is True+ assert result["is_business_hours"] is False++ def test_no_timestamp_yields_nothing(self) -> None:+ """An unknown time is not "outside business hours"."""+ assert time_of_day_fields({"user_name": "alice"}) == {}++ def test_an_unparseable_timestamp_yields_nothing(self) -> None:+ assert time_of_day_fields({"event_time": "not a date"}) == {}++ def test_event_time_wins_over_ingest_time(self) -> None:+ """Using ingest time would make a batch import at 03:00 look like a+ night-time attack."""+ result = time_of_day_fields(+ {+ "event_time": datetime(2026, 9, 22, 14, 0),+ "ingest_time": datetime(2026, 9, 22, 3, 0),+ }+ )+ assert result["is_business_hours"] is True++ def test_iso_strings_parse(self) -> None:+ result = time_of_day_fields({"event_time": "2026-09-22T14:00:00Z"})+ assert result["is_business_hours"] is True++ def test_epoch_milliseconds_parse(self) -> None:+ ms = int(datetime(2026, 9, 22, 14, 0).timestamp() * 1000)+ assert time_of_day_fields({"event_time": ms})["is_business_hours"] is True++ def test_business_hours_are_configurable(self) -> None:+ """ "Outside business hours" means nothing without knowing whose+ business; a hardcoded window fires all night for half the world."""+ event = {"event_time": datetime(2026, 9, 22, 20, 0)}+ assert time_of_day_fields(event)["is_business_hours"] is False+ assert time_of_day_fields(event, business_start=9, business_end=23)["is_business_hours"] is True+++class TestEnrich:+ def test_a_vendor_value_is_never_overwritten(self) -> None:+ """A vendor's own answer about its own tenant's hours beats ours."""+ event = {"event_time": datetime(2026, 9, 22, 3, 0), "is_business_hours": True}+ assert enrich(event)["is_business_hours"] is True++ def test_returns_the_event_unchanged_when_nothing_is_derivable(self) -> None:+ event = {"user_name": "alice"}+ assert enrich(event) is event++ def test_combines_both_families(self) -> None:+ event = {+ "event_time": datetime(2026, 9, 22, 3, 0),+ "actor": "alice",+ "target": "bob",+ }+ result = enrich(event, {"actor_eq_target"})+ assert result["is_after_hours"] is True+ assert result["actor_eq_target"] is False+ assert result["actor"] == "alice"+++class TestRequestedFields:+ def test_collects_comparison_and_time_fields(self) -> None:+ rules = [+ {"match_when": {"actor_eq_target": False, "event_name": "CreateAccessKey"}},+ {"match_when": {"is_business_hours": False}},+ ]+ assert requested_derived_fields(rules) == {"actor_eq_target", "is_business_hours"}++ def test_walks_nested_clauses(self) -> None:+ rules = [{"match_when": {"any_of": [{"actor_eq_target": True}, {"x": 1}]}}]+ assert "actor_eq_target" in requested_derived_fields(rules)++ def test_strips_operator_suffixes(self) -> None:+ """Rules write `is_business_hours: false` bare, but a clause can+ carry an operator and the field name still has to be recovered."""+ rules = [{"match_when": {"is_business_hours_in": [False]}}]+ assert "is_business_hours" in requested_derived_fields(rules)++ def test_ordinary_fields_are_not_collected(self) -> None:+ rules = [{"match_when": {"user_name": "alice", "process_name_contains": "x"}}]+ assert requested_derived_fields(rules) == set()+++class TestNeqOperator:+ """`neq` is used by rules in the shipped corpus and never existed in the+ matcher, so `approver_role_neq: "codeowner"` was read as a field+ literally named `approver_role_neq` — which nothing emits."""++ def test_neq_matches_a_different_value(self) -> None:+ from app.services.detection_matcher import matches++ assert matches({"approver_role_neq": "codeowner"}, {"approver_role": "dev"})++ def test_neq_does_not_match_an_equal_value(self) -> None:+ from app.services.detection_matcher import matches++ assert not matches({"approver_role_neq": "codeowner"}, {"approver_role": "codeowner"})++ def test_neq_does_not_fire_on_a_missing_field(self) -> None:+ """Otherwise every neq rule fires on every event lacking the field —+ a detection that fires on absence of data."""+ from app.services.detection_matcher import matches++ assert not matches({"approver_role_neq": "codeowner"}, {"user_name": "alice"})++ def test_the_operator_split_does_not_capture_comparison_names(self) -> None:+ """`actor_uid_neq_owner_uid` is a derived field, not `actor_uid`+ with a neq operator. Adding the operator must not break it."""+ from app.services.detection_matcher import _split_op++ assert _split_op("actor_uid_neq_owner_uid") == ("actor_uid_neq_owner_uid", "eq")+ assert _split_op("actor_uid_neq") == ("actor_uid", "neq")diff --git a/services/fusion/tests/test_detection_matcher_parity.py b/services/fusion/tests/test_detection_matcher_parity.pyindex 79e836638..ed74a0cee 100644--- a/services/fusion/tests/test_detection_matcher_parity.py+++ b/services/fusion/tests/test_detection_matcher_parity.py@@ -65,15 +65,77 @@ def test_vendored_matcher_matches_canonical_over_all_fixtures():def test_positive_fixtures_fire_and_negatives_do_not():- """Sanity: the vendored matcher upholds the fixture contract directly."""+ """Sanity: the vendored matcher upholds the fixture contract directly.++ Events go through the same derived-field enrichment+ ``DetectionEngine.evaluate`` applies. Replaying a fixture against the+ bare matcher tests a pipeline production does not run — and a rule+ matching on a derived field would fail here while working live, which+ trains people to weaken the gate.+ """+ from app.services.derived_fields import enrich, requested_derived_fields+fixtures_dir = _REPO / "detections" / "fixtures"ruleset = _REPO / "services" / "fusion" / "app" / "data" / "detection_ruleset.json"- rules = {r["slug"]: r["match_when"] for r in json.loads(ruleset.read_text())["rules"]}+ all_rules = json.loads(ruleset.read_text())["rules"]+ rules = {r["slug"]: r["match_when"] for r in all_rules}+ wanted = requested_derived_fields(all_rules)+checked = 0for f in sorted((fixtures_dir / "positive").glob("*.json")):mw = rules.get(f.stem)if mw is None:continue- assert vendored_matches(mw, json.loads(f.read_text())), f"positive fixture did not fire: {f.stem}"+ event = enrich(json.loads(f.read_text()), wanted)+ assert vendored_matches(mw, event), f"positive fixture did not fire: {f.stem}"checked += 1assert checked > 100, f"expected to check many positive fixtures, only {checked}"++ negatives = 0+ for f in sorted((fixtures_dir / "negative").glob("*.json")):+ mw = rules.get(f.stem)+ if mw is None:+ continue+ event = enrich(json.loads(f.read_text()), wanted)+ assert not vendored_matches(mw, event), f"negative fixture fired: {f.stem}"+ negatives += 1+ assert negatives > 100, f"expected to check many negative fixtures, only {negatives}"+++def test_vendored_derived_fields_match_canonical():+ """The derived-field helpers are vendored the same way `matches()` is.++ Two copies that can drift silently are worse than one copy plus a gate,+ and this is the gate. Without it the engine could compute a field the+ validator does not, so a rule would pass CI and never fire — or fire in+ CI and never in production.+ """+ from app.services import derived_fields as vendored++ canonical = _load_canonical()+ cases = [+ ({"actor": "alice", "target": "alice"}, {"actor_eq_target"}),+ ({"actor": "Alice", "target": "alice "}, {"actor_eq_target"}),+ ({"actor": "alice"}, {"actor_eq_target"}),+ ({"actor_uid": 1000, "owner_uid": 0}, {"actor_uid_neq_owner_uid"}),+ ({"event_time": "2026-09-22T14:00:00"}, set()),+ ({"event_time": "2026-09-26T14:00:00"}, set()),+ ({"event_time": "not a date"}, set()),+ ({"user_name": "x"}, {"a_eq_b"}),+ ]+ for event, wanted in cases:+ assert vendored.enrich(dict(event), set(wanted)) == canonical.enrich(+ dict(event), set(wanted)+ ), f"vendored and canonical enrich disagree on {event}"+++def test_vendored_requested_fields_match_canonical():+ from app.services import derived_fields as vendored++ canonical = _load_canonical()+ rules = [+ {"match_when": {"actor_eq_target": False, "event_name": "CreateAccessKey"}},+ {"match_when": {"any_of": [{"is_business_hours": False}, {"x": 1}]}},+ {"match_when": {"actor_uid_neq_owner_uid": True, "syscall": "openat"}},+ ]+ assert vendored.requested_derived_fields(rules) == canonical.requested_derived_fields(rules)diff --git a/services/threatintel/app/storage/qdrant.py b/services/threatintel/app/storage/qdrant.pyindex 48aa6b344..17d5edf3b 100644--- a/services/threatintel/app/storage/qdrant.py+++ b/services/threatintel/app/storage/qdrant.py@@ -63,7 +63,14 @@ class QdrantStore:self._use_fastembed = use_fastembedasync def initialize(self) -> None:- """Create Qdrant collections if they don't exist."""+ """Create the collections, then converge the schema.++ Creation is `if not exists`, which is correct for a first run and a+ no-op afterwards — so on its own it can never change the shape of a+ collection that already exists. Anything beyond creation (a payload+ index, a quantization setting, and eventually a re-embed when the+ embedding model changes) has to come from the migration runner.+ """existing = await self._client.get_collections()existing_names = {c.name for c in existing.collections}@@ -75,6 +82,45 @@ class QdrantStore:)logger.info("Created Qdrant collection", collection=col)+ await self._run_migrations()++ async def _run_migrations(self) -> None:+ """Apply pending vector-store migrations.++ Non-strict, matching the graph and lake runners: threat-intel+ enrichment degrades rather than taking the service down, and a+ pending migration is logged at ``error`` so a degraded start is+ visible rather than assumed.+ """+ try:+ from app.db.vector_migrations import ( # noqa: PLC0415+ pending_ids,+ pending_rebuilds,+ run_migrations,+ )+ except ImportError:+ # The runner lives in services/api and is vendored into+ # deployments that need it. Its absence means no migrations to+ # apply, not a broken store.+ return++ rebuilds = await pending_rebuilds(self._client)+ if rebuilds:+ logger.warning(+ "vector migrations include a re-embed; this is measured in "+ "hours for a large tenant and the alias keeps the existing "+ "collection serving until it completes",+ migrations=rebuilds,+ )++ applied = await run_migrations(self._client, strict=False)+ pending = await pending_ids(self._client)++ if applied:+ logger.info("Qdrant migrations applied", migrations=applied)+ if pending:+ logger.error("Qdrant migrations still pending", migrations=pending)+async def upsert_iocs(self, iocs: list[dict[str, Any]], *, tenant_id: str = SHARED_TENANT) -> None:"""Embed and upsert IOC documents into Qdrant for semantic search.