Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
Merge remote-tracking branch 'origin/main' into depth/4.4-throughput
Beenu Arora0ab0c65a0454
diff --cc .github/workflows/ci.ymlindex 2c12e5b74,8de41fb49..73b4b53a1--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@@ -701,21 -701,18 +701,34 @@@ jobsrun: python3 scripts/check_cloud_native_collection.py --self-test- name: Cloud-native collection gaterun: python3 scripts/check_cloud_native_collection.py+ # Depth 9.2. The probe set behind the recall measurement has to stay+ # usable: every probe must name an item that is in the corpus, the+ # probes must stay low-overlap with their answers (a probe written in+ # the document's own words hands the lexical baseline a score it could+ # not earn on a real question), and the baseline must still be a real+ # retriever rather than a stub. The scorer itself runs too: with no+ # embedding backend on a runner it prints "not measured" and exits 0,+ # which is the honest row and also proves the refusal path works.+ - name: Memory recall scorer — self-test+ run: python3 scripts/score_memory_recall.py --self-test+ - name: Memory recall scorer+ run: python3 scripts/score_memory_recall.py+++ # The three standard inputs the syslog gate does not cover: OTLP+ # logs, a customer Kafka topic and a TAXII 2.1 server. All three are+ # things a third party points at, and all three fail the same three+ # ways — unwired, tenant taken from the payload, and (for the Kafka+ # consumer) a loop that cannot tell a permanent fault from a+ # transient one. Self-test first, on the same tree: it injects+ # nineteen violations, one per rule. Writing it found two real+ # defects — the OTLP template was not in ALLOWED_TEMPLATE_IDS, so no+ # operator could mint a token the receiver would accept, and the+ # media-type rule matched the error message that names JSON rather+ # than the case arm that accepts it.+ - name: Standard inputs gate — self-test+ run: python3 scripts/check_standard_inputs.py --self-test+ - name: Standard inputs gate+ run: python3 scripts/check_standard_inputs.py# One layer in from the name: the profile gate asks whether a connector# reaches a normalization path, this asks which OCSF class it reaches# when it gets there. The class uid divided by a thousand is thediff --cc DEPTH_PROGRESS.mdindex e1640b09c,5567c305c..5e9439fef--- a/DEPTH_PROGRESS.md+++ b/DEPTH_PROGRESS.md@@@ -64,11 -64,12 +64,17 @@@ is written. Next free at capture: migra| migration `107` | 9.4 — `107_case_war_rooms.sql` | claimed || migration `108` | 9.5 — `108_hunt_reports.sql` | claimed || migration `109` | 9.6 — `109_push_devices.sql` | claimed |+ | migration `110` | 8.1 — `110_console_webauthn.sql` | claimed |+ | migration `111` | 8.1 — `111_audit_retention.sql` | claimed |+ | migration `112` | 8.1 — released, taken by another lane (D12) | **struck** |+ | migration `113` | 8.1 — `113_entitlement_enforcement.sql` | claimed |+ | ADR `0011` | 8.1 — truncating a hash-chained audit log | claimed |++| migration `110` | 4.2 — `110_stream_inputs.sql` (Kafka customer topics, TAXII collections) | claimed |+| migration `111` | 4.3 — `111_lake_query_jobs.sql` (cold-query cost estimate and progress state) | claimed |+| migration `112` | 4.5 — `112_scheduler_leases.sql` (connector-scheduler leader election) | claimed |+| ADR `0011` | 4.4 — partitioning fusion by entity rather than by alert | claimed |+Highest migration on `main` at claim time was `101_audit_immutable_nullify.sql`,so `095`--`101` were taken by sibling lanes between the capture above and thisclaim; re-read `services/api/migrations/` rather than trusting the capture row.@@@ -79,13 -80,13 +85,20 @@@ and `103` were already spoken for in ththrough `109`, one per item. A number left unused at the end of the phase isreleased by striking its row rather than by silently leaving a gap.+ Item 8.1 re-read `services/api/migrations/` on 2026-10-11 before claiming:+ `105_semantic_memory.sql` was the highest on `main`, and `102`--`109` were+ already spoken for above, so 8.1 takes `110`--`112` and ADR `0011`. The three+ sub-items that need no schema — the containerised-IdP CI leg,+ internationalisation and accessibility — claim no number. The same release+ rule applies: an unused number is struck, not silently skipped.++Phase 4 (items 4.2--4.5) re-read the directory on 2026-10-11 immediately+before claiming: `101_audit_immutable_nullify.sql` is still the highest file+on `main`, `102`--`109` are spoken for above, so this lane takes `110`--`112`+and ADR `0011`. The numbers are claimed before any file is written, because+two lanes recently picked `102`/`103` within minutes of each other and one+had to renumber. Unused numbers are struck at the end of the phase.+## Phase 0: Kickoff- [x] **0.1** `DEPTH_PROGRESS.md` exists (this file), modelled on@@@ -647,8 -723,11 +931,14 @@@ endpoint produces once triage falls thr| 2026-10-07 | 0.3 | `make up` and `make smoke` (10/10) pass. Injection suite and load-harness baselines committed. `make up-full` deferred (D2) and hosted model rows blocked (D3). Fixing D1 was a precondition for the load-harness baseline. || 2026-10-09 | 3.3 | Identity privilege (18 rules) and a per-tenant first-seen store (2) built; 5 rules were never unreachable (gate operator drift, D9); 18 retired with a reason (D10). `MAX_UNREACHABLE` 45 → 2, the two remaining both needing the 3.4 baseline. Two matcher operators three shipped rules already used were implemented. || 2026-10-09 | 3.2 | Ordered and unordered sequences in the windowed engine; all four translatable Sigma correlation types compile, seven refusal reasons recorded. No upstream correlation rule exists in this tree to import (D8), so the corpus is first-party and labelled as such. |+| 2026-10-11 | 4.4 | Every parity 6.9 bullet closed; the cloud-hardware run is not possible from here and is recorded as unmeasured rather than implied. Rule index on required fields (lossless, proven by replaying every fixture both ways; 68.3% fewer candidates, 2.6x on evaluation), entity partition key held equal to the correlation key, per-entity concurrent triage, lag-based scale hint derived from the published run. The realistic mix found that eight of twelve connector profiles are category 2 and are promoted unconditionally, so on those connectors no event can be routine — the first run measured a routine alert rate of 1.00. ADR-0011; matrix 309 rows, 306 GATED. |+| 2026-10-11 | 4.3 | Third bullet closed. Reproduced: `check_retention_window.py` with the new rules reported no cold-query estimator, no job surface and no job table on the pre-fix tree. Built `lake_cold_query.py` (EXPLAIN ESTIMATE + system.parts + system.processes, no invented throughput or price), `lake_query_runner.py` and migration `111`. Negative control: removing the `status = 'queued'` predicate makes both interleaved transactions claim one job and the live suite goes red. The first version of that test — five gathered claims — passed under the broken implementation and was replaced. 26 pure + 12 live tests; matrix 306 rows, 303 GATED. |+| 2026-10-11 | 4.2 | Reproduced: `check_standard_inputs.py` against the pre-fix tree reported no OTLP package, no Kafka input and no TAXII server — the only TAXII-shaped route being a demo-gated sample list. Built all three, after moving the consumer-health classifier out of `graph_ws` into `streamhealth` so the new Kafka consumer shares the table rather than copying it. Negative controls recorded in the PR: a timestamp-only TAXII cursor loses an indicator in the single-row pagination walk (measured — 3 of 4), and nineteen gate injections each go red. Writing it found four defects, including an OTLP template nobody could mint a token for and a gate rule that read an error message instead of the behaviour. Ingest Go suite green; api projection suite 37; live TAXII suite 15. || 2026-10-09 | 4.1 | Reproduced: `check_cloud_native_collection.py` on the unmodified tree reported all three collection paths absent. Implemented `aws_cloudtrail_s3` (SQS-notified S3 objects; management events, data events and VPC flow logs; delete-after-emit), `gcp_pubsub` (REST pull on a log-sink subscription; ack-after-read) and `azure_event_hubs` (Capture blobs over the Data Lake Gen2 JSON API, with a focused Avro OCF reader). Negative controls recorded in the PR: deleting the SQS message before the object is read, acknowledging the Pub/Sub batch before it is built, ignoring the Avro union branch index, and ignoring the flow-log header line each fail a named test, and all were restored. Connectors suite 1014 -> 1098 passed. || 2026-10-09 | 3.1 | All 74 decided. 50 translated into `wd-*` rules derived from each original's own clauses and replayed through the real engine (162 assertions); 24 refused with a reason across five kinds. `MAX_UNREACHABLE` 119 → 45, published executable 2,603 → 2,529, windowed 18 → 68. Three findings recorded as D4–D6 below. |+ | 2026-10-11 | 8.1 | parity 4.1's containerised-IdP leg **observed to pass**, both halves: `keycloak-oidc` verifies a real `id_token` against Keycloak's published JWKS with an audience-mismatch negative control, and `keycloak-saml` walks the whole SP-initiated flow to the real ACS and signs `[email protected]` in as `infosec`, with an untrusted-key control answering 400. Run `38124780886`, as `ENVIRONMENT=test` against the DML-only `aisoc_app` role. The SSO claim row moves PARTIAL → GATED on that evidence. Building it found two shipped defects no self-authored provider could expose: `resolve_saml_connection` bound `:i` untyped so `:i IS NULL` raised on asyncpg, and Keycloak's default role-list mapper emits one `<Attribute Name="Role">` per role, which python3-saml refuses. |+ | 2026-10-11 | 8.1 | WebAuthn as a console second factor and MFA enforced per role. Fixing the branch's own gates found three `passkey_credentials` reads scoped by user id alone on a tenant-scoped table, and a permission-model violation where the MFA policy route imported `ROLE_PERMISSIONS` directly; the role *vocabulary* now has one owner in `role_grants.known_roles()`. 23 of 23 live cases pass against real Postgres, and deleting the login integration fails 14 of them — measured, where the workflow had said eight. |+ | 2026-10-11 | 8.1 | An enforced audit retention window with a re-anchored checkpoint, ADR-0011, and 12 live cases. Writing them found that migration `111` declared `ON DELETE CASCADE` on the checkpoint table and then made the cascade impossible; and, separately, that `audit_log`'s own trigger from migration `101` blocks `DELETE FROM tenants` for every tenant that has written a row, so tenant deletion is impossible on `main`. Reproduced and filed as #1330 rather than fixed here. |+ | 2026-10-11 | 8.1 | Opt-in per-tenant limit enforcement (migration `113`, D12), wired to `POST /connectors` and `POST /tenants/me/users` through one helper, with `GET`/`PUT /usage/enforcement` as the operator surface. Nine live cases, half of them positive controls: a guard that never refuses fails 1, a guard stuck closed fails 4. |+ | 2026-10-11 | 8.1 | axe extended to the case list and case workspace, the two of parity 4.7's five views that were left out. Four real defects found and fixed: two `<select>` controls with no accessible name (one of them the control that writes a case's status), two indistinguishable `<aside>` landmarks, and panel headings at `h3` directly under the page's `h1`. |+diff --cc docs/audit/CLAIM_TO_GATE_MATRIX.mdindex 6f8dcb532,712149e18..32851f6df--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md@@@ -320,9 -313,9 +320,13 @@@ Statuses: `GATED` (a CI job fails when## Summary+ - Total: 302+ - GATED: 300+ - PARTIAL: 2+++- Total: 309+- GATED: 306+- PARTIAL: 3- NO GATE: 0Counts come from `scripts/check_claim_gate_matrix.py`, which is also the gate. Recompute