Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
merge: union the ledgers and recompute the published figures
Beenu Arora179a8814f61a
diff --cc DEPTH_PROGRESS.mdindex 759dad539,8168ad034..03630e6d7--- a/DEPTH_PROGRESS.md+++ b/DEPTH_PROGRESS.md@@@ -738,14 -454,8 +738,15 @@@ endpoint produces once triage falls thr| 2026-10-09 | 3.2 | Ordered and unordered sequences in the windowed engine; all four translatable Sigma correlation types compile, seven refusal reasons recorded. No upstream correlation rule exists in this tree to import (D8), so the corpus is first-party and labelled as such. || 2026-10-09 | 4.1 | Reproduced: `check_cloud_native_collection.py` on the unmodified tree reported all three collection paths absent. Implemented `aws_cloudtrail_s3` (SQS-notified S3 objects; management events, data events and VPC flow logs; delete-after-emit), `gcp_pubsub` (REST pull on a log-sink subscription; ack-after-read) and `azure_event_hubs` (Capture blobs over the Data Lake Gen2 JSON API, with a focused Avro OCF reader). Negative controls recorded in the PR: deleting the SQS message before the object is read, acknowledging the Pub/Sub batch before it is built, ignoring the Avro union branch index, and ignoring the flow-log header line each fail a named test, and all were restored. Connectors suite 1014 -> 1098 passed. || 2026-10-09 | 3.1 | All 74 decided. 50 translated into `wd-*` rules derived from each original's own clauses and replayed through the real engine (162 assertions); 24 refused with a reason across five kinds. `MAX_UNREACHABLE` 119 → 45, published executable 2,603 → 2,529, windowed 18 → 68. Three findings recorded as D4–D6 below. |+| 2026-10-11 | 8.1 | parity 4.1's containerised-IdP leg **observed to pass**, both halves: `keycloak-oidc` verifies a real `id_token` against Keycloak's published JWKS with an audience-mismatch negative control, and `keycloak-saml` walks the whole SP-initiated flow to the real ACS and signs `[email protected]` in as `infosec`, with an untrusted-key control answering 400. Run `38124780886`, as `ENVIRONMENT=test` against the DML-only `aisoc_app` role. The SSO claim row moves PARTIAL → GATED on that evidence. Building it found two shipped defects no self-authored provider could expose: `resolve_saml_connection` bound `:i` untyped so `:i IS NULL` raised on asyncpg, and Keycloak's default role-list mapper emits one `<Attribute Name="Role">` per role, which python3-saml refuses. |+| 2026-10-11 | 8.1 | WebAuthn as a console second factor and MFA enforced per role. Fixing the branch's own gates found three `passkey_credentials` reads scoped by user id alone on a tenant-scoped table, and a permission-model violation where the MFA policy route imported `ROLE_PERMISSIONS` directly; the role *vocabulary* now has one owner in `role_grants.known_roles()`. 23 of 23 live cases pass against real Postgres, and deleting the login integration fails 14 of them — measured, where the workflow had said eight. |+| 2026-10-11 | 8.1 | An enforced audit retention window with a re-anchored checkpoint, ADR-0011, and 12 live cases. Writing them found that migration `111` declared `ON DELETE CASCADE` on the checkpoint table and then made the cascade impossible; and, separately, that `audit_log`'s own trigger from migration `101` blocks `DELETE FROM tenants` for every tenant that has written a row, so tenant deletion is impossible on `main`. Reproduced and filed as #1330 rather than fixed here. |+| 2026-10-11 | 8.1 | Opt-in per-tenant limit enforcement (migration `113`, D12), wired to `POST /connectors` and `POST /tenants/me/users` through one helper, with `GET`/`PUT /usage/enforcement` as the operator surface. Nine live cases, half of them positive controls: a guard that never refuses fails 1, a guard stuck closed fails 4. |+| 2026-10-11 | 8.1 | axe extended to the case list and case workspace, the two of parity 4.7's five views that were left out. Four real defects found and fixed: two `<select>` controls with no accessible name (one of them the control that writes a case's status), two indistinguishable `<aside>` landmarks, and panel headings at `h3` directly under the page's `h1`. |++| 2026-10-11 | 7.1 | Reproduced: against the unmodified tree, `registry.list_vendors_for_capability` returned `[]` for all five mailbox verbs and `dispatcher.dispatch` answered `executor_not_found` for `purge_email` over 79 registered builtin arms -- a phishing response that could block the sender's domain at the firewall and could not remove the message it arrived in. Added the five verbs to both halves of the vocabulary, two mail clients (Graph, Gmail with per-mailbox domain-wide delegation), ten vendor arms, migration `104`, three intake adapters, campaign grouping, the governed fan-out and a report-mailbox worker. Only documented vendor APIs: purge is a soft delete on both and `restore_email` is a real reverse; there is no tenant-wide sender block because neither vendor exposes one through these APIs, and the per-mailbox scope is in the summary an analyst reads rather than only in a docstring. Everything ships off. Negative controls recorded in the PR: defaulting the feature flag on reds four tests, inverting the verification probe's polarity reds two, reading the envelope sender instead of the preserved headers reds four adapter tests including the one asserting the report mailbox is not proposed for purge, incrementing `recipient_count` instead of recomputing it reds the grouping test, and dropping `recipient_count` from the per-mailbox dispatch reds the fan-out test -- which is the quiet way a SEVERE action becomes a MODERATE one. Gates: `check_action_contract` 86 -> 97 executors, `check_module_reachability` 981 -> 986 of 1014, `check_route_authz` 262 -> 265 authorizing with both ceilings unmoved, matrix 302 -> 303 rows. Actions suite 1158 -> 1228; api 44 new cases, 17 of them against a real Postgres with the full chain. Recorded as **PARTIAL**, not GATED: no Microsoft 365 or Google Workspace tenant was exercised (D-7.1). |+ | 2026-10-11 | 7.2 | Reproduced: `grep -rl "authentication-results" services/` returned no match on the unmodified tree, so nothing in the product read a sender-authentication header and a message whose SPF and DMARC had both failed produced a verdict that had never looked. Built header analysis with a three-valued result, URL submission through the existing sandbox contract, the attachment hash lookup the submit path already had, and `evidence_cited` citations into the investigation ledger. Migration `111` stores the headers on the submission so a retriage replays them rather than re-fetching a message the purge has usually removed. `analysed` and `unanalysed` are counted separately in three places and summed in none. Negative controls: treating `none`/`neutral` as passes reds two tests; silently skipping an unanalysable URL reds three. **That second control is the finding worth keeping** -- it failed to fail on the first attempt, because every test built its own `ArtefactResult` list and never ran the producer, so five cases were added that drive `analyse_message` with the registry patched empty. A second finding came from the ledger suite's teardown: deleting a run cascades to its events and migration 008's trigger refuses, so the fixture was dropped rather than the append-only control defeated. Gates: matrix 303 -> 304 rows (300 GATED), `check_text_bind_casts` and `check_raw_sql_columns` clean over 6 new columns, `check_route_authz` unmoved. 45 new api cases, 6 of them against a real Postgres with the full chain. |### D-7.1 -- No Microsoft 365 or Google Workspace tenant was exerciseddiff --cc ROADMAP.mdindex 15d47af48,125374b2a..c55c8730d--- a/ROADMAP.md+++ b/ROADMAP.md@@@ -45,9 -45,7 +45,11 @@@ A proof-first, security-first program t- **Phase D (breadth):** D1 eight connectors (QRadar/Exabeam/Securonix/Devo/Netskope/Windows-Sysmon/Zeek-Suricata/syslog-CEF) · D2 AI/LLM-usage audit connector + `llm-*` detections + hot/cold lake tiering · D3 live-vendor mock-server smoke.- **Phase E (prove it):** E1 CI-gated benchmark scoreboard tied to a deterministic live-agent MITRE-accuracy run.- The claim-to-gate matrix stands at **303 rows — 300 GATED / 3 PARTIAL / 0 NO GATE**. Two rows are PARTIAL. "The retention window a tenant chooses is the one the lake honours" stays PARTIAL until a live ClickHouse test watches a row survive past the old 90-day boundary. "Cloud-native collection" stays PARTIAL until a capture recorded from a live Event Hub replaces the fixture synthesised from the Avro specification. "SSO completes a sign-in" was the third, and depth 8.1 closed it by building the gate it named rather than by relabelling: `sso-live.yml` now stands up a containerised Keycloak and both legs have been observed to pass — `keycloak-oidc` takes a real `id_token` through `_verify_id_token` against the provider's published JWKS, and `keycloak-saml` walks the whole SP-initiated flow to the real ACS. Every other product claim is backed by a failing test, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).-The claim-to-gate matrix stands at **304 rows — 300 GATED / 4 PARTIAL / 0 NO GATE**. Four rows are PARTIAL. "A reported phishing message can be grouped into a campaign and purged across its recipients" is PARTIAL because no Microsoft 365 or Google Workspace tenant is reachable from CI or from this maintainer, so every vendor claim in depth 7.1 rests on the documented request shape and a recorded mock; it returns to GATED when a live tenant is available. "The retention window a tenant chooses is the one the lake honours" stays PARTIAL until a live ClickHouse test watches a row survive past the old 90-day boundary. "Cloud-native collection" stays PARTIAL until a capture recorded from a live Event Hub replaces the fixture synthesised from the Avro specification. And "SSO completes a sign-in": fix pass 4.1 drove both handlers end to end against real Postgres and real signed assertions, and it stays PARTIAL until the containerised-IdP leg of `sso-live.yml` has been observed to pass, because the identity providers in that suite are trust material this repository authors. Every other product claim is backed by a failing test, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).++The claim-to-gate matrix stands at **304 rows — 301 GATED / 3 PARTIAL / 0 NO GATE**. Four rows are PARTIAL. "A reported phishing message can be grouped into a campaign and purged across its recipients" is PARTIAL because no Microsoft 365 or Google Workspace tenant is reachable from CI or from this maintainer, so every vendor claim in depth 7.1 rests on the documented request shape and a recorded mock; it returns to GATED when a live tenant is available. "The retention window a tenant chooses is the one the lake honours" stays PARTIAL until a live ClickHouse test watches a row survive past the old 90-day boundary. "Cloud-native collection" stays PARTIAL until a capture recorded from a live Event Hub replaces the fixture synthesised from the Avro specification. And "SSO completes a sign-in": fix pass 4.1 drove both handlers end to end against real Postgres and real signed assertions, and it stays PARTIAL until the containerised-IdP leg of `sso-live.yml` has been observed to pass, because the identity providers in that suite are trust material this repository authors. Every other product claim is backed by a failing test, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).+- The claim-to-gate matrix stands at **303 rows — 300 GATED / 3 PARTIAL / 0 NO GATE**. Four rows are PARTIAL. "A reported phishing message can be grouped into a campaign and purged across its recipients" is PARTIAL because no Microsoft 365 or Google Workspace tenant is reachable from CI or from this maintainer, so every vendor claim in depth 7.1 rests on the documented request shape and a recorded mock; it returns to GATED when a live tenant is available. "The retention window a tenant chooses is the one the lake honours" stays PARTIAL until a live ClickHouse test watches a row survive past the old 90-day boundary. "Cloud-native collection" stays PARTIAL until a capture recorded from a live Event Hub replaces the fixture synthesised from the Avro specification. And "SSO completes a sign-in": fix pass 4.1 drove both handlers end to end against real Postgres and real signed assertions, and it stays PARTIAL until the containerised-IdP leg of `sso-live.yml` has been observed to pass, because the identity providers in that suite are trust material this repository authors. Every other product claim is backed by a failing test, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).++The claim-to-gate matrix stands at **304 rows — 301 GATED / 3 PARTIAL / 0 NO GATE**. Two rows are PARTIAL. "The retention window a tenant chooses is the one the lake honours" stays PARTIAL until a live ClickHouse test watches a row survive past the old 90-day boundary. "Cloud-native collection" stays PARTIAL until a capture recorded from a live Event Hub replaces the fixture synthesised from the Avro specification. "SSO completes a sign-in" was the third, and depth 8.1 closed it by building the gate it named rather than by relabelling: `sso-live.yml` now stands up a containerised Keycloak and both legs have been observed to pass — `keycloak-oidc` takes a real `id_token` through `_verify_id_token` against the provider's published JWKS, and `keycloak-saml` walks the whole SP-initiated flow to the real ACS. Every other product claim is backed by a failing test, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).++++The claim-to-gate matrix stands at **304 rows — 301 GATED / 3 PARTIAL / 0 NO GATE**. Four rows are PARTIAL. "A reported phishing message can be grouped into a campaign and purged across its recipients" is PARTIAL because no Microsoft 365 or Google Workspace tenant is reachable from CI or from this maintainer, so every vendor claim in depth 7.1 rests on the documented request shape and a recorded mock; it returns to GATED when a live tenant is available. "The retention window a tenant chooses is the one the lake honours" stays PARTIAL until a live ClickHouse test watches a row survive past the old 90-day boundary. "Cloud-native collection" stays PARTIAL until a capture recorded from a live Event Hub replaces the fixture synthesised from the Avro specification. And "SSO completes a sign-in": fix pass 4.1 drove both handlers end to end against real Postgres and real signed assertions, and it stays PARTIAL until the containerised-IdP leg of `sso-live.yml` has been observed to pass, because the identity providers in that suite are trust material this repository authors. Every other product claim is backed by a failing test, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).## v4.0 — Shippeddiff --cc apps/docs/docs/compliance/evidence-pack.mdindex bbe5113ba,ef13f1440..a321d17c9--- a/apps/docs/docs/compliance/evidence-pack.md+++ b/apps/docs/docs/compliance/evidence-pack.md@@@ -33,10 -33,7 +33,11 @@@ and says so| Prompt-injection resistance | Three adversarial corpora: 32 payloads scanned in isolation and 71 incident pairs whose payloads are placed in attacker-controllable fields, both against a clean twin with a recall floor and a false-positive ceiling, plus 34 held-out pairs authored after the guard was frozen and carrying no floor at all, on which the guard measures 7.1% | `services/agents/tests/adversarial/` || Claim-to-gate traceability | Every product claim mapped to the CI job that fails when it stops being true | `docs/audit/CLAIM_TO_GATE_MATRIX.md` |--The last row is the one worth reading first. It is the honest index: 300 rows++The last row is the one worth reading first. It is the honest index: 301 rows+++`GATED`, 2 `PARTIAL`, and a ratchet that refuses to let a row sit at+- The last row is the one worth reading first. It is the honest index: 300 rows++The last row is the one worth reading first. It is the honest index: 301 rows`GATED`, 4 `PARTIAL`, and a ratchet that refuses to let a row sit at`NO GATE`. Re-read the figure from the table rather than from here — a countcopied into prose goes stale in silence, which is whydiff --cc docs/audit/CLAIM_TO_GATE_MATRIX.mdindex 9e88bea27,4f0dd3e33..77f91c061--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md@@@ -314,9 -315,9 +315,9 @@@ Statuses: `GATED` (a CI job fails when## Summary- - Total: 303- - GATED: 300+ - Total: 304-- GATED: 300-- PARTIAL: 4++- GATED: 301+- PARTIAL: 3- NO GATE: 0Counts come from `scripts/check_claim_gate_matrix.py`, which is also the gate. Recompute