Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
docs(fix-pass): close the books — claim rows re-run, siblings reconciled, final report (#1302)
Beenu Aroraf06745525ec5
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex b368ea898..59b54ed67 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -31,6 +31,46 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Fixed+- **A fix pass closed the gap between what the enterprise-identity surfaces+ claimed and what they did.** Six items, each reproduced before it was+ fixed. **White-label** reached the surfaces the docs listed: both report+ workers resolve branding (and neither could store an artefact on real+ Postgres at all — a status-vocabulary mismatch the test double could not+ show), `send_approval_email` gained the caller it had never had, and the+ Slack and Teams prompts carry the tenant's product name through a+ read-only lookup in `services/actions`. Two surfaces were **narrowed+ rather than built**, with the reason recorded: the sign-in page resolves+ branding from a credential it does not yet have, and nothing maps a+ hostname to an organisation, so baking one name in would be wrong for+ every organisation but one *while looking configured*.++ **Usage metering** counts the path that answered rather than a column+ both paths write — a deployment with no model had been reporting 100% AI+ triage — and now classifies all four tiers the worker stamps, not just+ the deterministic one, with an AST pin that fails on any tier no meter+ classifies. **SCIM** gained PATCH, group membership and deprovisioning+ against live Postgres through the real app, and the console+ token-minting the documentation had promised.++ **The maturity table's evidence was audited and the gate that graded it+ was wrong.** One row covered three capabilities — SCIM, white-label and+ metering — citing a single SCIM test, and the gate reported OK on it for+ four releases, because it matches a row on a prefix and the first+ capability's evidence was real. Three rows now, three live suites, and+ metering's cited evidence had been wrong as well as missing: it runs on+ ClickHouse, not Postgres. `bundles_several_capabilities()` refuses the+ shape, distinguishing a list from a subordinate clause, with a self-test+ asserting both directions.++- **Two gates read the shell as if it were pytest's argument list.**+ `check_ci_install_parity` treated the `tee` target in+ `pytest … 2>&1 | tee /tmp/wl.txt` as a directory argument, concluding a+ job that named one file ran a whole service suite and demanding its+ entire dependency set — thirteen findings from one `tee`. Arguments now+ stop at the first shell operator, with a redirection consuming its+ target whether attached or detached. Proven in both directions: a+ genuine `pytest tests/` still produces all thirteen findings.+- **The copilot stopped inventing investigations, and stopped blaming amissing API key for a model that timed out** (#1275). The templatefallback returned one of five rotating paragraphs asserting findingsdiff --git a/FIX_PASS_PROGRESS.md b/FIX_PASS_PROGRESS.mdindex e9d1e2c48..56dbd1b2a 100644--- a/FIX_PASS_PROGRESS.md+++ b/FIX_PASS_PROGRESS.md@@ -146,10 +146,14 @@ conditional `numpy` pins collide into an unresolvable install. Parse with## Wave 7: Close the books-- [ ] **7.1** Re-run every claim row this pass touched-- [ ] **7.2** Update `GAP_CLOSURE_PROGRESS.md` and `PARITY_PROGRESS.md`-- [ ] **7.3** One `[Unreleased]` entry in `CHANGELOG.md`-- [ ] **7.4** Final report in this file+- [x] **7.1** Re-run every claim row this pass touched+ - Matrix: **302 rows, 299 GATED, 3 PARTIAL, 0 NO GATE**, counted by `check_claim_gate_matrix.py` rather than read off the summary line. Every static gate the touched rows name was re-run green: the matrix ratchet, maturity table (and its self-test), README gates, profile service counts, mypy baseline, comment paths, merge markers, attribution across 13,372 tracked files, competitor names, SCIM contract, detection fields, perf results, the one-case-table check, CI install parity and dependency pins. No row was promoted to GATED by this pass without the gate existing first; the three PARTIAL rows stay PARTIAL, including SSO, whose containerised-IdP leg has never been observed to pass.+- [x] **7.2** Update `GAP_CLOSURE_PROGRESS.md` and `PARITY_PROGRESS.md`+ - Parity **4.2 Console MFA** flipped to `[x]`, verified against the tree rather than against this file: `endpoints/mfa.py` mounted in the v1 router, migration `096`, `MfaView.tsx` in the console, `test_console_mfa.py` offline and `tests/isolation/test_console_mfa_live.py` live. Parity **4.1 SSO end to end** flipped to `[~]`, not `[x]` — the SAML metadata defect is fixed and driven end to end against real Postgres with a real signed assertion, but the containerised-IdP leg has never been watched to pass, and a box resting on a job nobody has seen succeed is not done. Everything else in both files was left alone: the remaining open boxes belong to the depth and gap-closure programmes, which are still running, and flipping them here would claim another plan's work.+- [x] **7.3** One `[Unreleased]` entry in `CHANGELOG.md`+ - Two entries under `### Fixed`, in house style, no tag: the enterprise-identity pass (white-label, metering, SCIM, maturity evidence) and the gate-parser fix. Both name what reproduced, what was narrowed instead of built, and why.+- [x] **7.4** Final report in this file+ - See **Final report** at the end of this file.## Maintainer-only (recorded, not attempted)@@ -657,3 +661,83 @@ comment above it says the figure is "recorded so a future change that givestriage a tool shows up here as a number moving off zero", and a literal cannever move. The fix derives the same zero from the ledger so it can changewhen the run does.++## Final report++**Where it landed: 40 of 48 items `[x]`, 2 `[~]`, 4 `[!]` maintainer-only.** The+two still showing `[ ]` here — 4.3 white-label and 4.6 maturity evidence — are+complete and flipped on the branch carrying their PR; they read open only in a+worktree cut before it merged. Re-read this file on `main` rather than trusting+that sentence.++### What this pass was for++It added no capability. It existed to repair what two other programmes had+recorded as shipped and an audit found broken, which is why nearly every item+begins with a reproduction rather than a design.++### The pattern that recurred most++**A mechanism existed, passed its tests, and nothing called it on the path that+mattered.** `send_approval_email` had no caller at all, so the signed+email-approval fallback was unreachable. The retro-hunt flag passed through no+compose file, so a tenant could not turn the feature on. `find_matching()` had+no production caller, so no playbook had ever run from an alert. A passing test+on an uncalled function is indistinguishable from a working feature until+somebody traces the call graph.++**Its second half is worse and reachability analysis cannot see it: the code+runs and the table it reads is one nothing writes.** `triages_model` counted a+column the deterministic path also writes, so a deployment with no model+reported 100% AI triage — and the figure looked plausible, which is why it+survived four releases.++### What the gates themselves got wrong++Three gates were found grading something other than what they claimed.++- `check_maturity_table` matched a row on a **prefix**, so one row covering+ SCIM, white-label and metering satisfied the lookup with a single SCIM test+ and the other two capabilities inherited a pass. It printed OK for four+ releases.+- `check_ci_install_parity` read the `tee` target in+ `pytest … | tee /tmp/wl.txt` as a pytest **directory** argument, concluding a+ job naming one file ran a whole service suite. Thirteen findings from one+ `tee`.+- `check_test_discovery` read a shell redirection as a test path.++All three are the same shape: **a detector that knows one syntax of the+language it parses.** Each fix ships with a self-test asserting both+directions, because a parser fix that stops detecting is worse than the bug.++### What was narrowed rather than built, and why++Narrowing is in the plan as a legitimate outcome, and it was used twice, both+times with the reason recorded beside the claim.++- **The sign-in page is not branded.** Branding resolves from the caller's+ credential and there is none yet on that page; nothing maps a hostname to an+ organisation, because `organizations` has a slug and a name and no domain.+ One name baked into the process would be wrong for every organisation on the+ deployment except one, *while looking configured* — worse than the platform+ name.+- **`services/actions` reads only `product_name`.** The API resolver also+ returns colours, a logo, support contacts and a footer; none has anywhere to+ go in a Block Kit payload or a MessageCard, and resolving fields nobody+ renders is how a second resolver starts drifting from the first.++### What is left, and why it is not a gap++Four items are `[!]` maintainer-only and were recorded rather than attempted:+making the live jobs required checks under branch protection, funding provider+keys so the weekly evaluation produces real numbers, a real multi-node cluster+for the scale run, and accepting or overruling the closure default in the 1.4+ADR. None is a code change.++Two are `[~]`, and both are deliberate rather than unfinished. **3.9** installs+from the lockfile and fails loudly but cannot *run* without a funded key (M2).+**4.1** SSO is fixed and driven end to end against real Postgres with a real+signed assertion, but its containerised-IdP leg has never been observed to+pass, so the claim row is PARTIAL — a row resting on a job nobody has watched+succeed is not gated, and relabelling it would make this file a liability+instead of a control.diff --git a/PARITY_PROGRESS.md b/PARITY_PROGRESS.mdindex 93dd00a32..fd42f8bea 100644--- a/PARITY_PROGRESS.md+++ b/PARITY_PROGRESS.md@@ -177,8 +177,10 @@ a reachability finding for 1.4 rather than a claim edit.## Phase 4: Enterprise identity and the operator console-- [ ] **4.1** SSO end to end-- [ ] **4.2** Console MFA+- [~] **4.1** SSO end to end+ - Closed under the fix pass as item 4.1, and **partial on purpose**. Two of the three reported defects did not reproduce — the console fragment consumer and the login-screen SSO entry both landed in v17.1.0 — and six tests now pin them. The third was real: `_saml_settings()` read its IdP trust anchor from `SAML_IDP_*` while the connection's own `metadata_url` / `metadata_xml` were read by nothing, so a correctly signed assertion came back `invalid_response`; fixed in `app/auth/saml_metadata.py`, driven end to end against real Postgres and a real signed assertion in `sso-live.yml`. It stays `[~]` because the containerised-IdP leg (`sso-live.yml :: keycloak-oidc`) has never been observed to pass, so the claim row is PARTIAL rather than GATED. A row resting on a job nobody has watched succeed is not gated.+- [x] **4.2** Console MFA+ - Closed under the fix pass as item 4.2. Reproduced first: ten of eleven assertions against the real login route and real Postgres failed with `{"detail":"Not Found"}` — the surface did not exist. Now TOTP pinned to RFC 6238's own vectors rather than to a round trip, ten single-use recovery codes, a vault-encrypted secret, a `last_used_step` high-water mark so a code cannot be replayed inside its own 30-second window, per-tenant enforcement with no backfill row, and audited enrolment, disable, recovery-code use, policy change and administrative reset. A correct password for an enrolled account answers **202** with a challenge rather than a session — 202 rather than a widened 200 body, because `access_token` is required and making it optional breaks every generated client. Migration `096`; 28 offline tests, 13 live, and a CI negative control that deletes the challenge branch and requires the suite to go red. Verified present on `main`: `endpoints/mfa.py` mounted in the v1 router, `MfaView.tsx` in the console, `test_console_mfa.py` and `tests/isolation/test_console_mfa_live.py`.- [ ] **4.3** Custom roles everywhere — ships as security S13- [ ] **4.4** Audit log for tenant admins- [ ] **4.5** Operator pages