Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

docs(fix-pass): close the books — claim rows re-run, siblings reconciled, final report (#1302)

Beenu Aroraf06745525ec5
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b368ea898..59b54ed67 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -31,6 +31,46 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
+- **A fix pass closed the gap between what the enterprise-identity surfaces
+ claimed and what they did.** Six items, each reproduced before it was
+ fixed. **White-label** reached the surfaces the docs listed: both report
+ workers resolve branding (and neither could store an artefact on real
+ Postgres at all — a status-vocabulary mismatch the test double could not
+ show), `send_approval_email` gained the caller it had never had, and the
+ Slack and Teams prompts carry the tenant's product name through a
+ read-only lookup in `services/actions`. Two surfaces were **narrowed
+ rather than built**, with the reason recorded: the sign-in page resolves
+ branding from a credential it does not yet have, and nothing maps a
+ hostname to an organisation, so baking one name in would be wrong for
+ every organisation but one *while looking configured*.
+
+ **Usage metering** counts the path that answered rather than a column
+ both paths write — a deployment with no model had been reporting 100% AI
+ triage — and now classifies all four tiers the worker stamps, not just
+ the deterministic one, with an AST pin that fails on any tier no meter
+ classifies. **SCIM** gained PATCH, group membership and deprovisioning
+ against live Postgres through the real app, and the console
+ token-minting the documentation had promised.
+
+ **The maturity table's evidence was audited and the gate that graded it
+ was wrong.** One row covered three capabilities — SCIM, white-label and
+ metering — citing a single SCIM test, and the gate reported OK on it for
+ four releases, because it matches a row on a prefix and the first
+ capability's evidence was real. Three rows now, three live suites, and
+ metering's cited evidence had been wrong as well as missing: it runs on
+ ClickHouse, not Postgres. `bundles_several_capabilities()` refuses the
+ shape, distinguishing a list from a subordinate clause, with a self-test
+ asserting both directions.
+
+- **Two gates read the shell as if it were pytest's argument list.**
+ `check_ci_install_parity` treated the `tee` target in
+ `pytest … 2>&1 | tee /tmp/wl.txt` as a directory argument, concluding a
+ job that named one file ran a whole service suite and demanding its
+ entire dependency set — thirteen findings from one `tee`. Arguments now
+ stop at the first shell operator, with a redirection consuming its
+ target whether attached or detached. Proven in both directions: a
+ genuine `pytest tests/` still produces all thirteen findings.
+
- **The copilot stopped inventing investigations, and stopped blaming a
missing API key for a model that timed out** (#1275). The template
fallback returned one of five rotating paragraphs asserting findings
diff --git a/FIX_PASS_PROGRESS.md b/FIX_PASS_PROGRESS.md
index e9d1e2c48..56dbd1b2a 100644
--- a/FIX_PASS_PROGRESS.md
+++ b/FIX_PASS_PROGRESS.md
@@ -146,10 +146,14 @@ conditional `numpy` pins collide into an unresolvable install. Parse with
## Wave 7: Close the books
-- [ ] **7.1** Re-run every claim row this pass touched
-- [ ] **7.2** Update `GAP_CLOSURE_PROGRESS.md` and `PARITY_PROGRESS.md`
-- [ ] **7.3** One `[Unreleased]` entry in `CHANGELOG.md`
-- [ ] **7.4** Final report in this file
+- [x] **7.1** Re-run every claim row this pass touched
+ - Matrix: **302 rows, 299 GATED, 3 PARTIAL, 0 NO GATE**, counted by `check_claim_gate_matrix.py` rather than read off the summary line. Every static gate the touched rows name was re-run green: the matrix ratchet, maturity table (and its self-test), README gates, profile service counts, mypy baseline, comment paths, merge markers, attribution across 13,372 tracked files, competitor names, SCIM contract, detection fields, perf results, the one-case-table check, CI install parity and dependency pins. No row was promoted to GATED by this pass without the gate existing first; the three PARTIAL rows stay PARTIAL, including SSO, whose containerised-IdP leg has never been observed to pass.
+- [x] **7.2** Update `GAP_CLOSURE_PROGRESS.md` and `PARITY_PROGRESS.md`
+ - Parity **4.2 Console MFA** flipped to `[x]`, verified against the tree rather than against this file: `endpoints/mfa.py` mounted in the v1 router, migration `096`, `MfaView.tsx` in the console, `test_console_mfa.py` offline and `tests/isolation/test_console_mfa_live.py` live. Parity **4.1 SSO end to end** flipped to `[~]`, not `[x]` — the SAML metadata defect is fixed and driven end to end against real Postgres with a real signed assertion, but the containerised-IdP leg has never been watched to pass, and a box resting on a job nobody has seen succeed is not done. Everything else in both files was left alone: the remaining open boxes belong to the depth and gap-closure programmes, which are still running, and flipping them here would claim another plan's work.
+- [x] **7.3** One `[Unreleased]` entry in `CHANGELOG.md`
+ - Two entries under `### Fixed`, in house style, no tag: the enterprise-identity pass (white-label, metering, SCIM, maturity evidence) and the gate-parser fix. Both name what reproduced, what was narrowed instead of built, and why.
+- [x] **7.4** Final report in this file
+ - See **Final report** at the end of this file.
## Maintainer-only (recorded, not attempted)
@@ -657,3 +661,83 @@ comment above it says the figure is "recorded so a future change that gives
triage a tool shows up here as a number moving off zero", and a literal can
never move. The fix derives the same zero from the ledger so it can change
when the run does.
+
+## Final report
+
+**Where it landed: 40 of 48 items `[x]`, 2 `[~]`, 4 `[!]` maintainer-only.** The
+two still showing `[ ]` here — 4.3 white-label and 4.6 maturity evidence — are
+complete and flipped on the branch carrying their PR; they read open only in a
+worktree cut before it merged. Re-read this file on `main` rather than trusting
+that sentence.
+
+### What this pass was for
+
+It added no capability. It existed to repair what two other programmes had
+recorded as shipped and an audit found broken, which is why nearly every item
+begins with a reproduction rather than a design.
+
+### The pattern that recurred most
+
+**A mechanism existed, passed its tests, and nothing called it on the path that
+mattered.** `send_approval_email` had no caller at all, so the signed
+email-approval fallback was unreachable. The retro-hunt flag passed through no
+compose file, so a tenant could not turn the feature on. `find_matching()` had
+no production caller, so no playbook had ever run from an alert. A passing test
+on an uncalled function is indistinguishable from a working feature until
+somebody traces the call graph.
+
+**Its second half is worse and reachability analysis cannot see it: the code
+runs and the table it reads is one nothing writes.** `triages_model` counted a
+column the deterministic path also writes, so a deployment with no model
+reported 100% AI triage — and the figure looked plausible, which is why it
+survived four releases.
+
+### What the gates themselves got wrong
+
+Three gates were found grading something other than what they claimed.
+
+- `check_maturity_table` matched a row on a **prefix**, so one row covering
+ SCIM, white-label and metering satisfied the lookup with a single SCIM test
+ and the other two capabilities inherited a pass. It printed OK for four
+ releases.
+- `check_ci_install_parity` read the `tee` target in
+ `pytest … | tee /tmp/wl.txt` as a pytest **directory** argument, concluding a
+ job naming one file ran a whole service suite. Thirteen findings from one
+ `tee`.
+- `check_test_discovery` read a shell redirection as a test path.
+
+All three are the same shape: **a detector that knows one syntax of the
+language it parses.** Each fix ships with a self-test asserting both
+directions, because a parser fix that stops detecting is worse than the bug.
+
+### What was narrowed rather than built, and why
+
+Narrowing is in the plan as a legitimate outcome, and it was used twice, both
+times with the reason recorded beside the claim.
+
+- **The sign-in page is not branded.** Branding resolves from the caller's
+ credential and there is none yet on that page; nothing maps a hostname to an
+ organisation, because `organizations` has a slug and a name and no domain.
+ One name baked into the process would be wrong for every organisation on the
+ deployment except one, *while looking configured* — worse than the platform
+ name.
+- **`services/actions` reads only `product_name`.** The API resolver also
+ returns colours, a logo, support contacts and a footer; none has anywhere to
+ go in a Block Kit payload or a MessageCard, and resolving fields nobody
+ renders is how a second resolver starts drifting from the first.
+
+### What is left, and why it is not a gap
+
+Four items are `[!]` maintainer-only and were recorded rather than attempted:
+making the live jobs required checks under branch protection, funding provider
+keys so the weekly evaluation produces real numbers, a real multi-node cluster
+for the scale run, and accepting or overruling the closure default in the 1.4
+ADR. None is a code change.
+
+Two are `[~]`, and both are deliberate rather than unfinished. **3.9** installs
+from the lockfile and fails loudly but cannot *run* without a funded key (M2).
+**4.1** SSO is fixed and driven end to end against real Postgres with a real
+signed assertion, but its containerised-IdP leg has never been observed to
+pass, so the claim row is PARTIAL — a row resting on a job nobody has watched
+succeed is not gated, and relabelling it would make this file a liability
+instead of a control.
diff --git a/PARITY_PROGRESS.md b/PARITY_PROGRESS.md
index 93dd00a32..fd42f8bea 100644
--- a/PARITY_PROGRESS.md
+++ b/PARITY_PROGRESS.md
@@ -177,8 +177,10 @@ a reachability finding for 1.4 rather than a claim edit.
## Phase 4: Enterprise identity and the operator console
-- [ ] **4.1** SSO end to end
-- [ ] **4.2** Console MFA
+- [~] **4.1** SSO end to end
+ - Closed under the fix pass as item 4.1, and **partial on purpose**. Two of the three reported defects did not reproduce — the console fragment consumer and the login-screen SSO entry both landed in v17.1.0 — and six tests now pin them. The third was real: `_saml_settings()` read its IdP trust anchor from `SAML_IDP_*` while the connection's own `metadata_url` / `metadata_xml` were read by nothing, so a correctly signed assertion came back `invalid_response`; fixed in `app/auth/saml_metadata.py`, driven end to end against real Postgres and a real signed assertion in `sso-live.yml`. It stays `[~]` because the containerised-IdP leg (`sso-live.yml :: keycloak-oidc`) has never been observed to pass, so the claim row is PARTIAL rather than GATED. A row resting on a job nobody has watched succeed is not gated.
+- [x] **4.2** Console MFA
+ - Closed under the fix pass as item 4.2. Reproduced first: ten of eleven assertions against the real login route and real Postgres failed with `{"detail":"Not Found"}` — the surface did not exist. Now TOTP pinned to RFC 6238's own vectors rather than to a round trip, ten single-use recovery codes, a vault-encrypted secret, a `last_used_step` high-water mark so a code cannot be replayed inside its own 30-second window, per-tenant enforcement with no backfill row, and audited enrolment, disable, recovery-code use, policy change and administrative reset. A correct password for an enrolled account answers **202** with a challenge rather than a session — 202 rather than a widened 200 body, because `access_token` is required and making it optional breaks every generated client. Migration `096`; 28 offline tests, 13 live, and a CI negative control that deletes the challenge branch and requires the suite to go red. Verified present on `main`: `endpoints/mfa.py` mounted in the v1 router, `MfaView.tsx` in the console, `test_console_mfa.py` and `tests/isolation/test_console_mfa_live.py`.
- [ ] **4.3** Custom roles everywhere — ships as security S13
- [ ] **4.4** Audit log for tenant admins
- [ ] **4.5** Operator pages