Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
Merge remote-tracking branch 'origin/main' into depth/8.1-entitlement-enforcement
Beenu Arorab6b684ba086e
diff --cc ROADMAP.mdindex 0ed1c578e,d029c22f9..a16d78c2c--- a/ROADMAP.md+++ b/ROADMAP.md@@@ -45,7 -45,7 +45,9 @@@ A proof-first, security-first program t- **Phase D (breadth):** D1 eight connectors (QRadar/Exabeam/Securonix/Devo/Netskope/Windows-Sysmon/Zeek-Suricata/syslog-CEF) · D2 AI/LLM-usage audit connector + `llm-*` detections + hot/cold lake tiering · D3 live-vendor mock-server smoke.- **Phase E (prove it):** E1 CI-gated benchmark scoreboard tied to a deterministic live-agent MITRE-accuracy run.+ The claim-to-gate matrix stands at **302 rows — 300 GATED / 2 PARTIAL / 0 NO GATE**. Two rows are PARTIAL. "The retention window a tenant chooses is the one the lake honours" stays PARTIAL until a live ClickHouse test watches a row survive past the old 90-day boundary. "Cloud-native collection" stays PARTIAL until a capture recorded from a live Event Hub replaces the fixture synthesised from the Avro specification. "SSO completes a sign-in" was the third, and depth 8.1 closed it by building the gate it named rather than by relabelling: `sso-live.yml` now stands up a containerised Keycloak and both legs have been observed to pass — `keycloak-oidc` takes a real `id_token` through `_verify_id_token` against the provider's published JWKS, and `keycloak-saml` walks the whole SP-initiated flow to the real ACS. Every other product claim is backed by a failing test, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).+++The claim-to-gate matrix stands at **303 rows — 300 GATED / 3 PARTIAL / 0 NO GATE**. Three rows are PARTIAL. "The retention window a tenant chooses is the one the lake honours" stays PARTIAL until a live ClickHouse test watches a row survive past the old 90-day boundary. "Cloud-native collection" stays PARTIAL until a capture recorded from a live Event Hub replaces the fixture synthesised from the Avro specification. And "SSO completes a sign-in": fix pass 4.1 drove both handlers end to end against real Postgres and real signed assertions, and it stays PARTIAL until the containerised-IdP leg of `sso-live.yml` has been observed to pass, because the identity providers in that suite are trust material this repository authors. Every other product claim is backed by a failing test, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).## v4.0 — Shippeddiff --cc apps/docs/docs/compliance/evidence-pack.mdindex fb5e360ec,7890b5387..d7f443afd--- a/apps/docs/docs/compliance/evidence-pack.md+++ b/apps/docs/docs/compliance/evidence-pack.md@@@ -34,7 -34,7 +34,9 @@@ and says so| Claim-to-gate traceability | Every product claim mapped to the CI job that fails when it stops being true | `docs/audit/CLAIM_TO_GATE_MATRIX.md` |The last row is the one worth reading first. It is the honest index: 300 rows+ `GATED`, 2 `PARTIAL`, and a ratchet that refuses to let a row sit at+++`GATED`, 3 `PARTIAL`, and a ratchet that refuses to let a row sit at`NO GATE`. Re-read the figure from the table rather than from here — a countcopied into prose goes stale in silence, which is why`scripts/readme_gates.py` cross-checks this sentence against the matrix.diff --cc docs/audit/CLAIM_TO_GATE_MATRIX.mdindex 8aa2b7a64,712149e18..c3254deed--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md@@@ -277,8 -277,7 +277,10 @@@ Statuses: `GATED` (a CI job fails when| CORE gives the investigation agent real indicator evidence | `apps/docs/docs/deployment/docker.md` | `ci.yml :: Python — Service unit tests` (`services/fusion/tests/test_tenant_ioc_match.py`) | core | GATED | `AlertEnricher` asks the enrichment service, which runs in the `full` profile, so on CORE the call failed, was caught, logged at `debug`, and `{}` returned: the agent received "could not check" for every indicator on every alert. The tenant's own `threat_intel_iocs` is in the Postgres fusion already uses and CORE ships a real CISA KEV feed. Expiry is a hard gate in SQL; decay is a per-type half-life on `last_seen`, because an address reassigns in weeks and a file hash identifies the same bytes forever. The half-lives are judgement rather than measurement and say so. A faded hit is reported weakly rather than dropped, because "seen weakly" is different information from "never seen" || Injection flip rate and unsafe-action rate are measured per family | `apps/docs/docs/benchmark.md` | `ci.yml :: Python — Tests` (`tests/adversarial/test_behavioural_injection.py`) | core | GATED | **27 pairs across the 9 families the plan names**, each payload written to fit the field it arrives in: overall flip 25.9%, unsafe action 3.7%, with a per-family ceiling so a regression in one cannot hide behind an improvement in another. It immediately found a real blind spot: **`fake_tool_output` flips every verdict and the guard catches none of it**, because a payload shaped like a tool's own answer reads as evidence rather than as an instruction. Recorded at 1.0 rather than patched, and the guard is deliberately not tuned against this corpus in the same change, because hardening against a tuned corpus has already been measured here moving 66.7% to 98.1% while held-out payloads moved 3.6% to 7.1%. **Measures the pipeline, not a model**: the stub is deliberately obedient, so the rates are an upper bound on harm and the artefact carries `llm_resistance_measured: false` || Closure accuracy is measured on the tenant's own data, not only on a synthetic corpus | `apps/docs/docs/operations/shadow-mode.md` | `ci.yml :: Python — Tests` (agents, `test_closure_qa_sampling.py`) | core | GATED | Nothing measured this. The eval harness grades a synthetic corpus and the funnel counts how many alerts were closed, neither of which answers "of the ones the agent closed on your data, how many should it have". A tenant-configurable sample (5% default) of auto-closures goes to an analyst, scored on the five-part rubric. The sample is a **deterministic hash of the alert id, not `random()`**: a rolled sample fires at the configured rate *per replica*, so three replicas at 5% sample 15%, and a redelivered Kafka message gets a second roll and weights one closure twice. Bucketing also makes a higher rate a superset of a lower one, so a tenant's figure stays comparable across a rate change. Zero reviews reports **not measured** rather than 0.0, and every mean travels with the count it was taken over |+ | SSO completes a sign-in, into the right tenant with the mapped role | `apps/docs/docs/operations/security.md` | `ci.yml :: Python — Tests` (`services/api/tests/test_sso_provisioning.py`, `test_saml_connection_metadata.py`) + `sso-live.yml :: sso-live` (`tests/isolation/test_sso_live.py` against real Postgres and the real application) + `sso-live.yml :: keycloak-oidc` (a real Keycloak `id_token` through `_verify_id_token` against its published JWKS) + `sso-live.yml :: keycloak-saml` (`scripts/ci/keycloak_saml_probe.py`: the whole SP-initiated flow against a containerised Keycloak — the app's AuthnRequest from the connection's own trust material, Keycloak's own login page, a signed Response, and the real ACS against real Postgres as the DML-only `aisoc_app` role) | core | GATED | Both handlers used to end a successful sign-in by putting a JWT in an `aisoc_token` cookie carrying `sub`, `email`, `name` and `picture`. That token authenticated nothing three times over: signed with `JWT_SECRET` rather than the key the API verifies with, carrying no `tenant_id` or `role`, and in a cookie when the API reads `Authorization: Bearer`. A user could complete the whole dance and find every request unauthenticated. **The security property the tests exist for is that the tenant does not come from the assertion**: an IdP that can name its own tenant can name somebody else's, so it comes from the connection row an administrator configured, and `complete_sso_login` is asserted to take no tenant argument at all. A group cannot map to `admin` or `platform_admin`. The token travels in the URL fragment, which is not sent to the server and does not reach an access log or a `Referer`. **Fix pass 4.1** found the half none of that could see: `_saml_settings()` read its IdP trust anchor from `SAML_IDP_*` while the connection's own `metadata_url` and `metadata_xml` were written by `POST /sso-connections` and read by nothing, so on a stock install — which sets no SAML environment at all — every assertion was refused while the console showed a configured connection. The SAML callback also redirected straight to its RelayState destination, whose auth guard rebuilds its bounce target from `pathname + search` and drops the fragment the token travels in. Both are fixed and driven end to end. **This row was PARTIAL because the identity providers in that suite are a keypair and a metadata document this repository authors, so they cannot exhibit a vendor's own quirks, and the containerised leg had never been observed to pass.** Both halves are now closed. `keycloak-oidc` passed on `main` in run 38118951217 (2026-10-11), and depth 8.1 added `keycloak-saml`, which walks the entire SP-initiated flow against a real Keycloak and passed on the pull request that added it. **It immediately found two shipped defects neither the self-authored provider nor any unit test could see**: `resolve_saml_connection` bound `:i` with no type, so `:i IS NULL` — the branch `/auth/saml/login` takes when the console links to it without an issuer, which is every console — raised on asyncpg and was swallowed into "no enabled SAML connection is configured"; and Keycloak's default role-list mapper emits one `<Attribute Name="Role">` per role, which python3-saml refuses, so the ACS answered an unhandled 500 to a stock Keycloak. Both fixed, both with negative controls in the PR. One limit stated rather than implied: there is no browser in either job, so the SAML hop is driven over HTTP with the login form parsed out of Keycloak's own HTML — everything a browser does except JavaScript, which Keycloak's login form does not require |+++| SSO completes a sign-in, into the right tenant with the mapped role | `apps/docs/docs/operations/security.md` | `ci.yml :: Python — Tests` (`services/api/tests/test_sso_provisioning.py`, `test_saml_connection_metadata.py`) + `sso-live.yml :: sso-live` (`tests/isolation/test_sso_live.py` against real Postgres and the real application: a SAML Response signed by the connection's own IdP key, and an RS256 `id_token` from a provider serving real discovery and a real JWKS, each taken through to a token `GET /auth/me` accepts) | core | PARTIAL | Both handlers used to end a successful sign-in by putting a JWT in an `aisoc_token` cookie carrying `sub`, `email`, `name` and `picture`. That token authenticated nothing three times over: signed with `JWT_SECRET` rather than the key the API verifies with, carrying no `tenant_id` or `role`, and in a cookie when the API reads `Authorization: Bearer`. A user could complete the whole dance and find every request unauthenticated. **The security property the tests exist for is that the tenant does not come from the assertion**: an IdP that can name its own tenant can name somebody else's, so it comes from the connection row an administrator configured, and `complete_sso_login` is asserted to take no tenant argument at all. A group cannot map to `admin` or `platform_admin`. The token travels in the URL fragment, which is not sent to the server and does not reach an access log or a `Referer`. **Fix pass 4.1** found the half none of that could see: `_saml_settings()` read its IdP trust anchor from `SAML_IDP_*` while the connection's own `metadata_url` and `metadata_xml` were written by `POST /sso-connections` and read by nothing, so on a stock install — which sets no SAML environment at all — every assertion was refused while the console showed a configured connection. The SAML callback also redirected straight to its RelayState destination, whose auth guard rebuilds its bounce target from `pathname + search` and drops the fragment the token travels in. Both are fixed and driven end to end. **Why this is PARTIAL and not GATED**: the identity providers in that suite are a keypair and a metadata document this repository authors, so they cannot exhibit a vendor's own quirks. `sso-live.yml :: keycloak-oidc` is the containerised leg and **has never been observed to pass** — it was written where no container runtime would start. The row returns to GATED on the first green run of that job, not before |+| A per-tenant limit an operator configured can refuse work, and only if they opted in | `services/api/app/services/entitlements.py`; parity plan 4.5 | `entitlements-live.yml :: entitlements-live` (`tests/isolation/test_entitlement_enforcement_live.py`, 9 cases against real Postgres as the DML-only `aisoc_app` role. The job additionally asserts a **fresh database enforces nothing**, and carries two negative controls: a guard that never refuses fails 1 of 9, and a guard stuck closed fails 4 of 9 — every one of those a positive control) | core | GATED | `entitlements.py` said in its own docstring that enforcement was not its job, which was the right first move and left an operator who configured a ceiling with a number the product reported and never applied. Enforcement is **opt-in per tenant per key**, because `tenants.limits` has been a reporting input since it existed and the numbers in it were never chosen as hard caps — arriving switched on would turn every one into a refusal on upgrade. A measurement failure **allows** the request, logged at `error`: a quota that cannot count must not stop a SOC, and the fail-closed instinct here is the one that shipped a control withholding every grant on every deployment. The refusal is **402** with the key, the usage and the ceiling in the body — 403 says the caller may not do this (false) and 429 invites a retry that will fail identically. Wired rather than available: `POST /connectors` checks before the vault call and `POST /tenants/me/users` after the role grant is authorized, both through one helper. **Six console pages parity 4.5 also names — branding, SCIM tokens, retention, organisation members and grants, rule packs, delegations — are not built; this row covers the enforcement bullet only** || Accessibility is checked on the views an analyst uses, not only on the landing page | `ROADMAP.md` | `ci.yml :: Web — Tests` (`apps/web/src/test/a11y.test.tsx`) | core | GATED | 1.1 narrowed the "WCAG AA full accessibility pass" claim because axe covered the landing and chrome components only. 4.7 adds **three of the five** operator views the plan names: the alerts queue and list, the investigation rail, and settings. Each asserts it rendered real markup **before** axe runs, because axe passes on an empty div and a view that rendered a spinner would report coverage the suite does not have. That guard immediately caught one: the rail keys SWR on a tuple, the fixture read only the string form, and it was rendering its error state. **Limits stated**: the case workspace is not covered, and axe checks a subset of WCAG rather than all of it || A playbook can start from an alert, and previews before it acts | `apps/docs/docs/console/playbooks.md` | `ci.yml :: Python — Tests` (agents, `test_alert_triggered_playbooks.py`) | core | GATED | `find_matching()` had **no production caller**, so no playbook ever ran from an alert. Three switches must agree before one acts (deployment, tenant, playbook) and the default of every one is off; anything short of all three runs in **preview**, with its plan and simulated steps attached to the alert. Preview is the default state rather than a mode somebody has to remember, so a deployment that enables the feature and forgets the per-playbook list gets previews rather than containment. The live list is read per call, so turning one off takes effect on the next alert rather than the next restart. It runs **after** triage, because a playbook's conditions read the verdict || An approval step is a durable pause that survives a restart | `apps/docs/docs/console/playbooks.md` | `ci.yml :: Python — Tests` (agents, `test_playbook_pause_resume.py`) | core | GATED | The engine is a single-threaded index walk with no pause and no resume, so `approval` failed closed and **12 shipped playbooks aborted on it**. Failing closed was right (it previously returned `{"skipped": true}` while reporting SUCCESS, letting a run continue into the action a human was meant to authorise) but it is not an approval mechanism. Migration `081` stores the position **and the context**, because the resumed half would otherwise see an empty context and every templated parameter would resolve to nothing. The stored index is the approval step itself and resume advances past it, so a replayed decision continues rather than pausing again. The pause is resolved **before** the run continues, so a double-tap in the responder app resumes once. Expiry is mandatory with a recorded outcome: `expired` is a decision, not the absence of one |@@@ -314,9 -313,9 +316,12 @@@## Summary- - Total: 303+ - Total: 302- GATED: 300+ - PARTIAL: 2++++- Total: 303+- PARTIAL: 3- NO GATE: 0Counts come from `scripts/check_claim_gate_matrix.py`, which is also the gate. Recompute