Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
AiSOC depth plan: progress
Mirrors plans/aisoc_depth_plan.plan.md, which is locked and is the source of
truth. This file is the mutable half.
Legend: [ ] not started, [~] in progress, [x] done, [!] blocked (with
exactly what is needed).
An item is [x] only when all three of these are recorded against it: the test
or gate that failed on the pre-fix tree for the stated reason, the change, and
the negative control showing that reverting it fails again.
- Base commit:
1b8bc2d4(origin/main, v17.1.0). - Plan captured at:
90ea2fd. Figures in the plan body are hints at that commit; Phase 0.2 below re-derives each one and the tree wins.
Reconciliation, 2026-10-09
Eleven items had merged to main and every one still read [ ] here. The
tracker is the mutable half of a locked plan, so a tick that lags the tree
makes it describe a repository that no longer exists -- the same failure the
plan exists to catch, one level up.
Reconciled against the tree rather than against the pull requests: each item
below was confirmed by the artefact it was supposed to produce being present
on main (the corpus file, the OCSF decision table, the event catalogue
directory, the scorecard fields, the notify arms, the report scheduler, the
step bounds, the access-conditions routes, the security pack), not by a merged
title. Each carries the reproduce, change and negative control the legend
requires in its own pull request body.
| Item | Landed in | Confirmed by |
|---|---|---|
| 1.1 | #1205 | services/agents/tests/eval_data/verdict/verdict_corpus_v1.json |
| 1.2 | #1191 | nine scorecard fields in aisoc_benchmark/replay.py |
| 1.3 | #1197 | scripts/check_verdict_corpus.py, prompting/tool_results.py |
| 2.1 | #1195 | internal/normalizer/ocsf_classes.go |
| 2.2 | #1204 | scripts/check_activity_projection.py |
| 2.3 | #1209 | schemas/event_catalog/ |
| 5.1 | #1203 | live_actions/notify_arms.py |
| 5.2 | #1211 | app/workers/report_scheduler.py |
| 5.3 | #1207 | app/playbook/bounds.py |
| 8.2 | #1206 | endpoints/access_conditions.py |
| 8.3 | #1210 | docs/security/ |
Not started: phases 3, 4, 6, 7, 9 and the whole of 10, and
2.4--2.7. Those remain [ ] and are not claimed here.
Not started as at 2026-10-09: phases 3, 4, 6, 7, 9 and the whole of 10, plus 1.4--1.6 and 2.4--2.7. That list was true on the day and is not maintained -- lanes have landed items in several of those phases since. The per-phase checklists below and the item log at the end are the live answer; this paragraph is a dated snapshot and reading it as current is the exact failure the reconciliation above was written to fix.
Migration and ADR numbers claimed
Phases run in parallel lanes, so a number is claimed here before the file
is written. Next free at capture: migration 095, ADR 0010
(docs/decisions/, not docs/adr/ -- the plan's wording; the directory is
decisions/ and the highest is 0009-auto-close-without-an-earned-grant.md).
Renumbered at the 2026-10-12 merge train. Nineteen lanes merged at once and several had claimed the same free numbers before any of them landed — four lanes on migration
110, three on111, three on ADR0011. First lane merged kept its number; the rest were renumbered in one commit, with every in-tree reference updated. The file name in the Status column is authoritative; the Number column records what the lane claimed at the time. Final ladder: migrations102–118contiguous, ADRs0011–0013. The109claimed by 9.6 for109_push_devices.sqlwas never written (9.6 shipped no migration), so109went to109_mcp_access.sql; a future 9.6 follow-up re-derives againstmainas the rule below already requires. The111claimed by 6.4 is likewise re-derived when that lane lands.
| Number | Claimed by | Status |
|---|---|---|
ADR 0010 |
2.7 — entity and session context in CORE | landed, docs/decisions/0010-entity-and-session-context-in-core.md, accepted |
migration 102 |
2.4 — 102_sessions.sql |
landed, sessions table plus three alerts columns |
migration 103 |
2.5 — 103_posture_snapshots.sql |
claimed |
migration 104 |
9.1 — 109_mcp_access.sql |
claimed |
migration 105 |
9.2 — 105_semantic_memory.sql |
claimed |
migration 106 |
9.3 — 106_case_depth.sql |
claimed |
migration 107 |
9.4 — 107_case_war_rooms.sql |
claimed |
migration 108 |
9.5 — 108_hunt_reports.sql |
claimed |
migration 109 |
9.6 — 109_push_devices.sql |
claimed |
migration 110 |
8.1 — 116_console_webauthn.sql |
claimed |
migration 111 |
8.1 — 117_audit_retention.sql |
claimed |
migration 112 |
8.1 — released, taken by another lane (D12) | struck |
migration 113 |
8.1 — 118_entitlement_enforcement.sql |
claimed |
ADR 0011 |
8.1 — truncating a hash-chained audit log | claimed |
| migration 104 | 5.4 stateful user/manager verification (verification requests, answers, tenant opt-in) | claimed |
| migration 105 | 5.6 plain-language playbook drafts (draft, preview, approval) | claimed |
Lane B claimed 102/103 and landed that claim on main first, so Lane C's
two moved to 104/105 and 102_playbook_verification.sql was renamed with
it. Both lanes had claimed before writing, as the plan requires -- what the
rule cannot prevent is two lanes claiming the same free number before either
merges, so re-read services/api/migrations/ and this table before
picking one.
| migration 110 | 6.1 — 110_ai_assets.sql | claimed |
| migration 111 | 6.4 — 111_ai_agent_identities.sql | claimed |
| ADR 0011 | 6.6 — AI containment arms off by default, and the rollback a kill-switch needs | claimed |
| — | 5.4 — 104_playbook_verification.sql on depth/5.4-5.6-lane-c, and 7.1 — 104_phishing_campaigns.sql on depth/7.1-7.3-phishing | collision, unresolved here. Both lanes wrote a 104 on a branch without a row in this table. Phase 9's 104 is the one with a claim, so these two need renumbering by their own lanes; recorded rather than renamed from outside, because renaming another lane's migration from here is how a branch comes back to a file that no longer exists |
| migration 110 | 7.1 — 111_phishing_campaigns.sql | claimed |
| migration 111 | 7.2 — 112_phishing_message_analysis.sql | claimed |
| migration | 1103.4 — | released, ADR-0011 put the baselines in the Redis fusion already runs, so there is no table |
| migration 110_entity_baselines.sql111 | 3.7 — 111_detection_coverage.sql | claimed |
| ADR 0011 | 3.4 — behavioural baselines in CORE | landed, docs/decisions/0011-behavioural-baselines-in-core.md, accepted |
| migration 110 | 4.2 — 110_stream_inputs.sql (Kafka customer topics, TAXII collections) | claimed |
| migration 111 | 4.3 — 114_lake_query_jobs.sql (cold-query cost estimate and progress state) | claimed |
| migration 112 | 4.5 — 115_scheduler_leases.sql (connector-scheduler leader election) | claimed |
| ADR 0011 | 4.4 — partitioning fusion by entity rather than by alert | claimed |
Highest migration on main at claim time was 101_audit_immutable_nullify.sql,
so 095--101 were taken by sibling lanes between the capture above and this
claim; re-read services/api/migrations/ rather than trusting the capture row.
The same is true of this table: 104 was already written on two sibling
lanes' branches with no row here, which is how lanes collide. Phase 6's
numbers were picked after reading services/api/migrations/ on main and
git ls-tree over every origin/* branch, not from the row above.
Phase 9 re-read services/api/migrations/ on 2026-10-11 before claiming:
101_audit_immutable_nullify.sql was still the highest on main, and 102
and 103 were already spoken for in the table above, so Phase 9 takes 104
through 109, one per item. A number left unused at the end of the phase is
released by striking its row rather than by silently leaving a gap.
Item 8.1 re-read services/api/migrations/ on 2026-10-11 before claiming:
105_semantic_memory.sql was the highest on main, and 102--109 were
already spoken for above, so 8.1 takes 110--112 and ADR 0011. The three
sub-items that need no schema — the containerised-IdP CI leg,
internationalisation and accessibility — claim no number. The same release
rule applies: an unused number is struck, not silently skipped.
Phase 6 moved 105/106 to 110/111 on 2026-10-11, because Phase 9's
claim for 104--109 reached this table on main first and a claim table
nobody defers to is not a coordination mechanism. The lesson is the one this
section already states and that three lanes still fell to: the table is
authoritative only for lanes that write to it before the file, and four
branches now carry a 104 or a 105 that was never claimed here. Reading
main is not enough; read git ls-tree over every origin/* branch as
well, which is what found this.
Phase 7 claimed 104 and 105 on the same day and lost the race: Phase 9
merged to main first, so Phase 7 renumbered to 110 and 111 rather than
leaving two files with one number each. Both lanes re-read the directory
before claiming and both read 101, which is the shape a claim table cannot
prevent on its own — the table is the tiebreak, and whoever lands second
moves. Recorded because the next lane will hit it too: re-read
DEPTH_PROGRESS.md on origin/main, not only the migrations directory, and
expect to renumber if a sibling merged between the claim and the pull
request.
Phase 3 re-read services/api/migrations/ on 2026-10-11 before claiming:
101_audit_immutable_nullify.sql was still the highest on main and 102
through 109 were already spoken for above, so Phase 3 takes 110 for the
3.4 baseline store and 111 for the 3.7 coverage store. ADR 0011 is the
next free number in docs/decisions/ after 2.7 landed 0010.
Phase 4 (items 4.2--4.5) re-read the directory on 2026-10-11 immediately
before claiming: 101_audit_immutable_nullify.sql is still the highest file
on main, 102--109 are spoken for above, so this lane takes 110--112
and ADR 0011. The numbers are claimed before any file is written, because
two lanes recently picked 102/103 within minutes of each other and one
had to renumber. Unused numbers are struck at the end of the phase.
Carried to the wave-2 depth plan, 2026-11-01
This plan's open items were handed over to
plans/aisoc_depth_wave2.plan.md at its kickoff and are tracked in
DEPTH2_PROGRESS.md from that date. They are deliberately not ticked
here: a tick in both files is a double count, and the wave-2 plan's Phase 10
closes this plan's Phase 10 as well.
Carried: 0.3's make up-full baseline (recorded below as D2 — now captured,
see DEPTH2_PROGRESS.md 0.3), 2.6, 3.6, 3.7, the 4.4 cloud-hardware run,
the 4.5 managed-cluster run, 5.5, 5.6, 6.3 to 6.6, 8.1's sub-items 4.5 and
4.6, the 9.4 Teams half, the 9.6 remainder, and Phase 10. The hand-over
table in DEPTH2_PROGRESS.md names the wave-2 phase each one became.
Phase 0: Kickoff
- 0.1
DEPTH_PROGRESS.mdexists (this file), modelled onPARITY_PROGRESS.md. - 0.2 Every figure re-derived against the base commit. See below.
- [
] 0.3 Baselines captured. Four of five recorded;]make up-fulland the hosted-model matrix are noted under Deviations.make up-fullwas captured on 2026-11-01 under the wave-2 plan's Phase 0.3 — 22 containers on thefullprofile, recorded indocs/audit/depth2-baseline/README.md. This item stays `[here and is closed inDEPTH2_PROGRESS.md`.
0.2 Re-derived figures
Measured on 1b8bc2d4, 2026-10-07, each by the generator or gate the plan
names. Where the captured and measured values differ, the measured value is
the one later phases are graded against.
| Figure | Captured in plan | Measured | Note |
|---|---|---|---|
| Executable detections | 2,603 | 2,603 | generate_corpus_stats.py --check |
| On disk | 6,991 | 6,991 | same |
| Quarantined | — | 4,388 | same |
| Unreachable native rules | 119 | 119 | check_detection_fields.py; MAX_UNREACHABLE = 119 |
| … needing a windowed evaluator | 74 | 74 | the family Phase 3.1 translates |
| … identity enrichment | 24 | 24 | Phase 3.3 |
| … fields nothing emits | 8 | 8 | Phase 3.3 |
| … first-seen / age | 8 | 8 | Phase 3.3 |
| … other | 6 | 6 | |
| … behavioural baseline | 2 | 2 | Phase 3.4 |
| Windowed rules | 18 | 18 | services/fusion/app/data/windowed_ruleset.json |
| Executor arms | 73 | 74 | classes declaring capability = "…" under services/actions/app/live_actions/. 33 distinct capability verbs. Phase 5.5 counts arms, so its target is measured from 74 |
| Claim-to-gate matrix | 290 rows, all GATED | 290 rows, 290 GATED, 0 PARTIAL, 0 NO GATE | check_claim_gate_matrix.py |
| CORE / full services | 16 / 22 | 16 / 22 | check_profile_service_counts.py (production stack: 16 / 21) |
| Next migration | 095 | 095 | highest is 094_user_delete_fk.sql |
| Next ADR | 0010 | 0010 | in docs/decisions/, highest 0009 |
Cloud, identity, SaaS and code executable rules: captured 461, measured
395. The gap is a definition the plan does not pin, not a change in the
tree, so it is recorded here rather than reconciled silently. Measured by
log_source on executable entries in marketplace/index.json, grouping:
| Group | Count | log_source values counted |
|---|---|---|
| cloud | 225 | aws 76, azure 50, gcp 40, kubernetes 36, docker 10, cloud 8, multi-cloud 5 |
| identity | 123 | identity 47, okta 35, active-directory 22, auth0 5, duo 5, ping 5, keycloak 4 |
| saas | 34 | google-workspace 8, m365 7, slack 6, email 3, salesforce 3, + 7 others |
| code | 13 | github 12, gitlab 1 |
| total | 395 |
Phase 3.6 must therefore raise 395 → 800 and ship the make stats line
with a --check gate that fixes this grouping in code, so the figure stops
being re-derivable two ways.
Two measurement caveats found while re-deriving, both worth carrying:
- 1,724 of the 2,603 executable rules live under
detections/sigma-imports/_quarantine/. "Executable" and "quarantined" overlap, because the Sigma compiler began translating rules in place without moving them. Any later statement of the form "N executable rules" must not be read as "N rules outside quarantine". - The plan's "1,798 of 2,603 are Windows (69%)" does not reproduce from
the index. Only 111 executable entries carry
log_source: windows, and 218 are Windows by path or log source (8%). The 1,770 Sigma imports carry nolog_sourceat all, which is where the difference lives: the captured figure counted the Sigma corpus as Windows, which is approximately but not exactly true. Phase 3.6's gate should classify by a field that exists rather than by corpus membership.
0.3 Baseline artefacts
Hardware: Apple M5 Max, 18 cores, 128 GiB host; Docker 29.5.2 in a 6-CPU /
15.6 GiB Linux VM. Date 2026-10-07. Commit 1b8bc2d4.
| Baseline | Result | Artefact |
|---|---|---|
make up (CORE) |
PASS -- 16 services healthy, administrator created | — |
make smoke |
PASS 10/10 stages -- event traversed ingest → Kafka → fusion → alert, severity and source attribution survived | — |
| Behavioural injection suite | 19 passed. Overall flip rate 0.2593, unsafe-action rate 0.037 over 27 cases | services/agents/tests/eval_data/behavioural_injection.json |
| Live-agent model matrix | 0 of 2 models measured -- "no live LLM key configured", reported as not measured rather than zeros | — |
| Load harness, 20,000 events | pipeline 130.3 alerts/s, ingest accepted 397.6 events/s, p50 54.9 s, p95 100.2 s, p99 103.9 s, delivery ratio 1.0, duplicates 0, dead letters 0 | docs/perf/results/2026-10-07-compose-depth-baseline.json |
Per-family injection flip rates at baseline, from the artefact above:
| Family | Cases | Flip rate | Guard catch rate |
|---|---|---|---|
| command_line | 4 | 0.25 | 0.75 |
| email_subject | 3 | 0.3333 | 0.6667 |
| fake_analyst_note | 2 | 0.0 | 1.0 |
| fake_tool_output | — | 1.0 | 0.0 |
| file_path, process_name, url, username | — | 0.0 | — |
| persona | — | 0.67 | — |
fake_tool_output at 1.0 flip / 0.0 catch is the figure Phase 1.3 must bring
to at or below 0.10 on tuned and held-out payloads.
The load-harness row is the "before" for Phase 4.4, whose target is a sustained 5,000 events/s at p95 under 2 s on documented cloud hardware. The measured p95 of 100 s is a drain latency under a 20,000-event burst on a 6-CPU laptop VM, not a steady-state figure, and must be labelled that way wherever it is quoted.
Deviations
Recorded as they happen. A deviation is not a failure; an unrecorded one is.
D1 -- The load harness could not run from a clean checkout (fixed here)
scripts/perf/load_harness.py invoked go run ./services/demo-producer with
cwd= the repository root. There is no go.mod at the root -- the producer
is its own module at services/demo-producer/go.mod -- so the subprocess
failed before starting:
producer failed (1): go: go.mod file not found in current directory or any
parent directory; see 'go help modules'
The harness has therefore never produced a run on the path it documents; the
published laptop figures must have come from a prebuilt --producer-bin,
which needs no module context. Fixed in the Phase 0 commit by running go run . from the producer's own directory, which is what produced the baseline
above. Phase 4.4 extends this harness and inherits the fix.
D2 -- make up-full not captured on this host
Deferred rather than claimed. The full profile is 22 services at ~12 GiB
against a 15.6 GiB VM that is already running the 16 CORE services, and a
baseline taken under memory pressure would describe the host rather than the
stack. Recorded as unmeasured; Phase 4.5 (helm install brings up all of
CORE) and Phase 3.4 (baselines in CORE) are the items that need it, and both
will re-take it on a sized host.
D3 -- Hosted model rows are maintainer-blocked (M2)
scripts/run_model_matrix.py reports 0 of 2 models measured with "no live
LLM key configured". This is the gate behaving correctly: it refuses to
publish 0.000 where it means "not measured". Phase 1.4 has since measured
two local candidates, which need no key: see the verdict-quality table on the
benchmark page and the 1.4 note below. The hosted rows stay [!] until the
maintainer funds a key, and still read "not measured" rather than zero.
D4 -- Not all 74 "windowed" rules need a window
The reachability gate attributes a rule to the windowed family by matching
its field names against _count$|^count_|_per_|time_window|_window_|_5min|_ratio$.
Eight of the 74 are per-event properties the pattern misreads: row_count is
the size of one export record, answer_count the number of records in one DNS
response, insecure_registries_count the length of a list in one daemon
config event, subdomain_hex_ratio a property of one query name, and
active_keys_per_account the state of an account at one moment. They are
refused under the not-windowed kind rather than translated, and the family
label in check_detection_fields.py is left as it is: changing the classifier
would move the published family counts that Phase 0.2 recorded, for no gain
now that each rule carries an individual reason.
D5 -- Five windowed rules that predate this work group by a field nothing emits
wd-secret-enumeration (distinct_by=secret_name), wd-windows-password-spray,
wd-sysmon-remote-thread-fanout, wd-sysmon-dns-query-fanout and
wd-sysmon-process-spawn-burst name entity fields outside the statically
recovered namespace. Four of the five are Windows EventData keys the
windows_event connector lifts wholesale, so the namespace under-approximates
and the rules are probably fine; secret_name has no such explanation. The
new gate reports these rather than failing on them, because failing would
be a false alarm about working rules, and the entity question is answered by
replay rather than by a name lookup. Not fixed here: changing a shipped
rule's group_by is a content decision with its own blast radius.
D6 -- The windowed engine ran a narrower field namespace than the stateless one
WindowedDetectionEngine._fields carried a docstring saying it matched the
stateless engine's namespace "exactly". It did not: the stateless engine then
applies derived_fields.enrich() and the per-tenant allowlist overlay, and
the windowed engine applied neither. Two of the rules translated here carry an
<x>_in_allowlist clause, so without fixing this they would have moved from
one engine that could not fire them to another. Both passes are now applied in
evaluate(), which takes the overlay the consumer had already resolved for
the stateless engine.
D7 -- There is no "existing watermark" in fusion to order sequences by
Item 3.2 says to order sequences "by event time with the existing
watermark". No watermark exists anywhere in the detection path: the only two
in the tree belong to services/actions' shadow-reconcile router and the
dead-letter replay, which are unrelated subsystems. Rather than invent one or
quietly order by arrival, the engine orders strictly by event time and
declares its own bound,
SEQUENCE_REORDER_TOLERANCE_SECONDS = 300, for how late an event may arrive
and still be stitched into a sequence. The bound is stated in the module
rather than inherited from something that does not exist.
Ordering by arrival would have been the easy implementation and is wrong
here: almost every connector in this tree polls, so a batch arrives in
the vendor's order and the event that starts a sequence routinely lands after
the one that finishes it. test_a_batch_delivered_out_of_arrival_order_still_fires
is the regression test for that, and reverting the event-time read makes it
and two others fail.
D11 -- Semantic recall measured worse than the full-text baseline on every embedding backend reachable from here
Parity 6.5 asks for the accuracy delta. It is measured, by
scripts/score_memory_recall.py over 12 hand-authored probes against a
12-item corpus, and the delta is negative:
| metric | lexical | semantic (qwen2.5:0.5b) |
delta |
|---|---|---|---|
| recall@1 | 0.667 | 0.333 | -0.333 |
| recall@3 | 0.750 | 0.417 | -0.333 |
| recall@5 | 0.750 | 0.667 | -0.083 |
| MRR | 0.708 | 0.465 | -0.243 |
The honest reading is not "semantic recall does not work". It is that
no embedding model is reachable from this project without a funded key.
The two models served locally are generative models used as embedders,
which is a known-poor substitute; nomic-embed-text was attempted and the
pull did not complete in ten minutes. Hosted embedding rows are blocked by
the same thing as D3 (maintainer item M2).
Three things follow, and all three are in the tree rather than in this note:
- The default did not move.
POST /kb/querystays full-text; semantic recall isrecall_memory: falseand strictly additive, so a caller that does not ask for it gets byte-identical behaviour. Putting a measurably worse retriever in front of the ranker would have made analyst search worse on exactly the deployments that cannot pay for an embedding model. The field's docstring carries the number and says to move the default when the measurement does. - The measurement found a real defect in the first implementation. The
3,072-dimension model initially scored 0.167 at recall@1 — and that was
measuring
_fit()truncating the vector to the collection's 1,536, not measuring the model. Padding a short vector is harmless under cosine; truncating a long one discards most of what the model said. Truncation is now refused with a message naming both dimensions, which is a configuration an operator can fix rather than a silently worse retriever they cannot see. - What is still not measured, stated plainly: the downstream effect of memory recall on verdict quality, which is what the Phase 1 scorecard grades. That needs an agent run per corpus item against a model worth grading, and no such model is funded or locally servable here. The retrieval delta above is the part that can be measured honestly today, and it is the part that is actually about what 9.2 builds.
D12 -- Migration 112 was claimed here and taken by another lane first
The claim table above reserved 110--112 for 8.1 after re-reading
services/api/migrations/ against main, which is the documented procedure
and was not enough. Sequentially-numbered migrations are a first-come
namespace across branches, not across main: by the time 8.1 came to write
its third file, 115_scheduler_leases.sql existed on another lane's pushed
branch. 110 and 111 had three and three other claimants respectively and
survived only because this item pushed first.
Re-derived immediately before committing by scanning the migration directory
of all 259 remote branches, not just main, which put the first free
number at 113. The 112 row is struck rather than silently reused, per the
release rule at the top of that table.
The general lesson is that "re-read main before claiming" understates the
problem. main is the one place a number cannot yet be taken.
D13 -- Two honest gates disagreed about the claim matrix, and the older one was wrong
scripts/check_claim_gate_matrix.py read the matrix as 300 GATED / 2 PARTIAL
while readme_gates._matrix_counts read 299 / 3, so the summary block was
reported as stale against a table that was correct.
readme_gates classified a row by scanning the whole line for NO GATE,
then PARTIAL, then GATED. A row's Notes cell is prose, so the first GATED
row to record why it had been PARTIAL -- which is exactly what a row should
say when a gate is built and the status earned -- counted as PARTIAL.
The cheap fix was to reword the row and lose the history. It now reads the cell the column header names, refuses a table whose Status column cannot be located rather than counting it as empty, and both properties are pinned by tests that fail against the previous parser.
D14 -- 4.6 is not started, and 4.5 is half done. Neither is claimed
Stated plainly rather than folded into a tick.
4.6 internationalisation (gap-closure 13.4) is not started. It asks
for a Next.js i18n library, externalised strings on five surfaces, a locale
switcher, RTL, locale-aware dates and numbers, a pilot locale and a CI gate
for missing and unused keys. Nothing of it exists: there is no i18n
directory, no message catalogue and no i18n dependency in apps/web. It is
the one sub-item of 8.1 with no code behind it, and the honest record is a
[ ], not a partial credit for the parts that are easy.
4.5 asks for two things. The opt-in limit enforcement the depth plan names by file is built, wired to two routes and gated (#1332). The six console pages -- branding, SCIM tokens, retention, organisation members and grants, rule packs, delegations -- are not. Its claim row says so in the row itself rather than relying on this note, because a reader of the matrix does not read this file.
Neither is blocking 8.1's stated "Done when": a CI test signs in through OIDC
and SAML with MFA enforced for the role, an elevation expires on schedule
(8.2, merged), and the security pack passes the competitor-name and
comment-path gates. All three hold. The item is [~] rather than [x]
because the spec lists more than its own done-when.
D8 -- No upstream Sigma correlation rule exists in this tree to import
detections/sigma-imports/ holds 3,132 rule documents and not one carries
a correlation: block, so the missing importer and the missing corpus hid
each other: there was nothing for an importer to have failed on. There is
also no fetcher for the upstream corpus (scripts/ has compile_sigma_ruleset.py,
sigma_compiler.py and sigma_proof_event.py, none of which downloads
anything).
So the importer is built and exercised against first-party correlation
documents hand-authored in the upstream format under
detections/sigma-correlations/, four accepted and two refused. The figure
they contribute is first-party content and is labelled that way in the truth
table and in the corpus README; it is not imported coverage. Vendoring the
upstream correlation corpus needs a fetcher and a licence review, and is not
attempted here.
One standard is weaker here than for the stateless Sigma imports, and stays
weaker: those are replayed through the real connector normalize() by an
out-of-process worker, and these are replayed through the real windowed
engine only. The connector step is not applied, because importing
services/connectors into a fusion test process shadows fusion's own app
package — the documented reason compile_sigma_ruleset.py uses a subprocess.
Instead, every selector field, group-by and distinct_by is checked
against the emitted-field namespace with no ceiling, which caught three
invented fields on the first run of the gate.
D15 -- A durable wait suspended the run and then carried on
Found while building 5.4 on 5.3's pause, and fixed here because 5.4 inherits
it. _control_wait writes a pause row and returns StepStatus.PENDING with
paused: True, and the run loop broke out only for StepType.APPROVAL --
so a wait long enough to become durable, or any wait with
until: callback, wrote its row and then ran every step after it. When the
callback or the sweeper later resumed from step_index + 1, those steps ran
a second time.
Nothing caught it because the two tests that exercise the durable path assert
on the pause row and on the resume, not on what the first half of the run did
afterwards -- and the shipped packs contain no wait, so no replay exercised
it either. The run loop now stops on any step that returns PENDING with a
pause_id, keyed on the result rather than on the step type, because the same
control step both sleeps in process (no pause) and suspends (a pause) and the
type cannot tell those apart.
Negative control: reverting that break fails
test_a_durable_pause_stops_the_run plus two others, recorded in the pull
request.
D16 -- The manager comes from SCIM only, and the lookup is three sources deep
The plan says "from SCIM or IdP data". Implemented as the step's own
manager parameter, then the run context (where identity enrichment puts one
when the alert carried it), then aisoc_scim_users.manager_ref -- the
enterprise extension's manager.value, which the column was missing
entirely, so every sync delivered the attribute and every sync dropped it.
A live per-request lookup against Entra or Okta is not implemented, and
is named here rather than left to be inferred: it needs a vendor credential
in the agents service, which today holds none, and the escalation runs on a
sweeper with no tenant session. A tenant that provisions through SCIM has the
data; a tenant that does not will see no_manager and the run resumes with
that outcome rather than hanging.
context_import.py writes a manager_id edge into Neo4j, which would be a
fourth source. Not used: the agents service has an asyncpg pool and no Neo4j
driver, and adding one for an escalation is a dependency the item does not
justify.
D17 -- The plan says valid_to; this schema says valid_until
Item 2.5 asks for versioned snapshots with valid_from / valid_to. The one
existing table in this database that bounds a fact in time is identity_edges
(migration 018), and it spells the closing column valid_until. Adopting the
plan's spelling would put two names for one concept in one schema, and the
next person to write a point-in-time query would have to know which table
they were in before they could write a WHERE. The existing spelling wins;
posture_snapshots uses valid_from / valid_until, and the deviation is
recorded here rather than made silently.
D18 -- Eleven API tests are already failing on main, and nothing grades them
The full services/api suite run for 2.5 reported 4,774 passed and 11 failed.
All eleven are in test_alert_list_priority_sort.py and
test_alert_tiles_cover_the_whole_result_set.py, neither of which this branch
touches, and all eleven fail identically on a clean git archive origin/main
export with the same error: list_alerts() missing 1 required positional argument: 'http_request'. The handler grew a Request parameter; the two
test files call it directly and were not updated.
What makes this worth a deviation rather than a footnote is why it is still
open on a branch-protected main with twenty-eight required checks. Both
files open with "Run against live Postgres: the defect is an ORDER BY, and a
fake session that answers any query cannot tell one ordering from another" --
and their fixture skips when no database answers. The only job that runs them
is Python — Tests, which sets DATABASE_URL to a host with no Postgres
behind it, so the fixture skips and the job is green. Neither file is named by
any workflow in .github/workflows/, so no live-Postgres job runs them
either. The suite that exists to prove the priority ordering has therefore
never graded it, and the signature break has been invisible since it landed.
Fixing them belongs to item 2.6, which wires the priority score and the queue ordering these very tests cover; the fix lands there together with a workflow that actually runs them against a database, because repairing the assertions without that would restore a test nothing executes.
D9 -- Five of the 45 "unreachable" rules were never unreachable
scripts/check_detection_fields.py carries its own copy of the matcher's
operator suffixes, under a comment saying it mirrors
detection_matcher.OPERATORS. It did not. neq had been added to the
matcher and not to the gate, so approver_role_neq: "codeowner" — which the
engine reads correctly as approver_role with != — was reported as a rule
naming a field called approver_role_neq. Five rules were counted against
the ratchet for a defect that had already been fixed.
The same comparison in the other direction found a worse one: the gate
listed not_startswith_any and not_endswith, which the matcher did not
implement. Three shipped rules use path_not_startswith_any, so the matcher
read the whole string as a field name and all three could never fire — while
the gate, stripping a suffix nothing implemented, reported them reachable.
Both operators are now implemented in the vendored matcher and the canonical
one, the fixture synthesiser knows them, and
test_the_gate_and_the_matcher_agree_on_every_operator compares the two
lists in both directions.
D10 -- 18 rules are retired rather than fixed, and why each one is
The plan says of one family: "build them where cheap, otherwise quarantine
with a reason". Applied to all three families, that leaves 18 rules that need
work belonging to another item, recorded individually in
scripts/enrichment_decisions.py:
| kind | n | needs |
|---|---|---|
needs-connector-field |
9 | a field the vendor has and no connector surfaces — depth plan 2.2 |
needs-external-source |
6 | a registry, a credential report or a factor record — 2.5 and beyond |
needs-inventory |
3 | an inventory this platform does not hold (DC machine accounts, privileged OAuth scopes) |
scripts/check_enrichment_decisions.py is what stops this being a way to
move the ratchet: it fails if a retired rule's fields are all resolvable.
Proved by retiring a working rule and watching it fail.
The line that mattered most was refusing a plausible substitute.
domain_age_days wants a domain's registration age from a registry; the
first-seen store could have answered "when this deployment first saw it" and
the number would have looked right in a diff. It would also have fired the
rule on most of the internet. Five such age-of-a-thing fields are named in a
test that forbids the first-seen store from ever growing them.
D19 -- Identity privilege reads a table a tenant must populate
The *_priv booleans resolve from identity_nodes.privilege_tier >= 2, the
column migration 018 created for the question. The table's only writer is
POST /api/v1/identity-graph/nodes, so a tenant that has imported no
directory gets no key at all and those 18 rules stay silent — they are
reachable "when the tenant has identity data", the same standard this gate
already applies to a vendor-payload field. That is weaker than "reachable on
any deployment" and is stated rather than glossed. Depth plan 2.5's posture
collectors are what will populate it automatically.
Four booleans in the same family are deliberately not computed —
scope_priv, act_as_user_priv, gpo_link_priv and account_is_dc — and
are among the 18 above. Each has no subject field any source emits, and
guessing a subject would produce a boolean that is confidently wrong rather
than absent.
D20 -- The "behavioural baseline" family named the wrong piece of work
check_detection_fields.py classified an unreachable field into a family so
the headline number would be actionable. Its behavioural pattern was
_baseline|_deviation|^active_|_is_first_, and ^active_ is not a
behavioural shape at all. It matched the only two rules left on the ratchet:
| rule | field | what it actually needs |
|---|---|---|
det-endpoint-146 |
active_state |
a systemd unit state from a host agent |
det-identity-015 |
active_p0_or_p1_incident |
the tenant's ITSM incident queue |
So the gate reported "2 rules need depth plan 3.4" when the true answer was zero, and building 3.4 exactly as specified would not have moved the number by one. A gate pointing at the wrong piece of work is worse than one pointing at none, because the named thing gets built and the next reader concludes the fix failed.
The pattern now matches the shape baselines.py actually produces
(^baseline_|_baseline$|_deviation$), and a new external system context
family carries the ITSM rule. MAX_UNREACHABLE stays at 2 — this is a
reclassification and not a fix, and lowering a ceiling on the strength of
relabelling is the move this file exists to refuse.
Two consequences worth stating. The behavioural family is now genuinely empty, so 3.4 closes no unreachable rule and is not credited with any; it earns its place by making 3.5's rules possible, which is a different claim. And D9's lesson repeated one level up: a gate that keeps its own description of another component drifts from it, and here the drift was in the classifier rather than in the operator list.
D21 -- D9, D10 and D19 were written, merged, and lost on main
PR #1226 (depth 3.3) added three deviation sections. None of them is on
main. The item log still cites "(gate operator drift, D9)" and "18 retired
with a reason (D10)", so the tracker has spent two days pointing at
explanations a reader could not find, and the same merge truncated 3.2's
checklist paragraph mid-sentence and left a stale unchecked duplicate
beneath it.
Nothing was wrong with the branch; the loss happened in a conflict
resolution that took main's side of this file wholesale. That is the
documented failure mode for a file several lanes write to at once, and the
documented remedy — keep both sides' paragraphs, never take one side whole —
is the one that would have prevented it.
All three are restored here verbatim from the merge commit rather than
rewritten from memory. At restoration the trio kept its numbers; the 2026-10-12
merge train renumbered them D19/D20/D22 because parallel lanes had taken D11-D14 on main first.
D22 -- A rule of this lane's own was a bare anomaly, and its own gate caught it
Depth plan 3.5 is explicit that an anomaly is "a typed signal, not an alert
on its own". ta-saas-oauth-app-volume-spike was written with
baseline_oauth_app_volume_z_gte as its entire clause -- an alert on an
anomaly, which is the one thing the item says not to build. It passed
generate_detections --check, the truth table, the reachability gate and
the generic fixture replay, because every one of those asks whether a rule
is well-formed and can fire, and it was and it could.
What caught it was the family's own gate asserting that each rule reads a
threat condition as well as a baseline. The rule now also requires
event_type: oauth_token_event.
The reusable half is that the generic gates could not have caught it. A corpus-wide gate checks the properties every rule shares; a property that belongs to one family has to be asserted by that family, and writing that assertion is part of adding the family rather than a later tidy-up.
Phase 1: Verdict quality you can publish
- 1.1 Balanced, labelled verdict corpus
- 1.2 Scorecard: balanced accuracy, MCC, per-class precision/recall, Wilson interval, false negatives, auto-close precision, escalation rate, time to verdict
- 1.3 Close
fake_tool_outputstructurally (≤ 0.10 tuned and held-out) - 1.4 A 7--8B local model in the matrix, chosen by score
- 1.5
make replay-evaldesign-partner kit - 1.6 Published "Verdict quality" table with a
--checkdrift gate
1.4 -- five local models measured; the winner is generation, not size
All five graded the same corpus digest through the production triage path, 72 model calls each, zero provider errors.
| Model | Balanced acc. | MCC | Malicious recall | Malicious prec. | ECE | Auto-close prec. |
|---|---|---|---|---|---|---|
llama3.1:8b-instruct-q4_K_M |
0.357 | 0.209 | 0.583 / 36 | 0.724 | 0.349 | 0.111 (9) |
llama3.2:3b-instruct-q4_K_M (CPU default) |
0.285 | 0.047 | 0.250 / 36 | 0.500 | 0.442 | 0.571 (7) |
llama3:8b |
0.279 | 0.113 | 0.639 / 36 | 0.657 | 0.421 | 0.091 (11) |
qwen2.5:7b-instruct-q4_K_M |
0.260 | 0.032 | 0.083 / 36 | 0.750 | 0.594 | 0.243 (37) |
mistral:7b-instruct-v0.3-q4_K_M |
0.210 | 0.202 | 0.639 / 36 | 0.657 | 0.366 | 0.143 (14) |
Four classes: a constant answer scores 0.250 balanced accuracy, and always guessing the majority class scores 0.500 plain accuracy.
Decision, strictly by balanced accuracy: llama3.1:8b-instruct-q4_K_M
becomes the default on docker-compose.gpu.yml and
docker-compose.host-llm.yml. It also leads on MCC, malicious precision and
calibration, and is the only row with ECE below 0.40. The 3B stays the
default for CPU make up, where a ~5 GB model does not fit the 8 GB CORE
budget -- a deployment constraint, not a score.
Size is the wrong unit. The previous-generation llama3:8b scores
below the 3B it would replace, so "upgrade to an 8B" is not advice that
survives measurement. An earlier revision of this file recorded exactly that
conclusion from llama3:8b alone and was wrong; the LiteLLM config's hint
now carries all five numbers instead.
Two findings outrank the ranking. Every model measured is far closer to chance than to useful -- the best is 0.357 against a 0.250 floor, and three of five fall below the 0.500 plain accuracy of guessing the majority class. And auto-close precision is poor across the board (0.091-0.571 on 7-37 candidates), so nothing measured here supports autonomous closure. That is a finding about the auto-close gate, not about which model to pick.
Caveats published rather than smoothed over:
- Latency is excluded from the comparison. The 3B ran CPU-bound inside the stack container and the four 7--8B rows GPU-accelerated on the host, so the figures measure different machines. All are still printed.
- Per-decision model attribution is unverified. Every call was placed
and answered -- each decision carries reasoning specific to its alert --
but
resolved_modelscame back empty on every run, so the scorecard shows that 72 calls happened and not which model served each. - Auto-close precision rests on 7 to 37 candidates. Small denominators; read the column as a warning sign, not a ranking.
Writing the winner into the GPU overlay broke the GPU overlay, and CI
caught it. Adding the model default left the
deploy.resources.reservations.devices block nested under litellm
instead of ollama. The file still parsed, docker compose config still
resolved, and make up-gpu would still have come up -- with the card
reserved by a proxy that runs no inference while generation fell back to
the CPU. There is no error anywhere in that sequence, which is why the
existing test_it_touches_only_ollama was the only thing between it and a
release.
Its companion assertion failed for a legitimate reason at the same time:
the overlay now must touch three services, since the measured default
needs a pull instruction on ollama-pull and routing on litellm. The
assertion was re-expressed rather than relaxed, and two properties added --
a device reserved on any service but ollama fails, and the model pulled
must be the model routed to. That second one guards the quiet failure: the
gateway 404s on a model nothing pulled, triage swallows it into the
deterministic fallback, and the stack looks healthy while measuring no
model at all. Each was proven by injecting the defect it names.
1.5 -- run end to end against a live stack, not a mock
make replay-eval / make replay-eval-verify drove a real run through the
API against a mocked Splunk ES holding 200 recorded closed notables: 200
read, 60 replayed in the test window, 240 production writes intercepted, a
signed bundle written and verified, and a wrong key refused with exit 1.
The live run is what found the three defects the unit tests could not: the decisions route answers a bare array and the kit assumed an envelope; a machine named only in the agent's reasoning survived redaction; and the prose and identifier passes named their counter namespace in different cases, so one host could take two tokens. All three are fixed with regression tests, each proven to fail against the pre-fix tree.
Known gap: the run abstained on all 60 decisions because the stack's triage reached no model, so the kit has been exercised on a deterministic run end to end but not yet on a run with real model verdicts in it.
1.6 -- the section it replaced was wrong, not merely stale
apps/docs/docs/benchmark.md said verdict accuracy was not measured here
and could not be. Item 1.1's corpus made that false. The reasoning is kept
(it is why score_replay_set.py still refuses this tree's two all-malicious
corpora) and the conclusion is replaced by the measurement.
scripts/check_verdict_quality.py --check is wired into ci.yml and fails
on drift between the committed scorecard and the published block, proven by
reverting a number and watching it red. It also refuses five scorecard
shapes that would each publish a score-shaped cell with no measurement
behind it -- the load-bearing one being a row claiming a live measurement
that placed zero model calls, which is precisely what a run against a dead
endpoint produces once triage falls through to its deterministic scorer.
Phase 2: A semantic layer and live posture
- 2.1 OCSF classes beyond five
- 2.2 Activity projection on every event
- 2.3 Event classification catalogue
- 2.4 Sessions per principal — one producer in the activity
projection, three consumers (Postgres, ClickHouse, Neo4j), held by
check_session_projection.py - 2.5
__posture_snapshot__collectors (AWS, Azure, GCP, Workspace, GitHub) - 2.4 Sessions per principal
- 2.5
__posture_snapshot__collectors (AWS, Azure, GCP, Workspace, GitHub) - 2.6 Use the posture everywhere
- 2.7 Entity context in CORE (ADR first) — ADR-0010 accepted;
Postgres in CORE, Neo4j stays in
full
Phase 3: Detection depth for cloud, identity and SaaS
- 3.1 Translate the 74 windowed
det-*rules — 50 translated, 24 refused with a reason.MAX_UNREACHABLE119 → 45. - 3.2 Ordered sequences and Sigma correlations — the windowed engine
stages ordered and unordered sequences (
SequenceRuleinservices/fusion/app/services/windowed_detection.py, loaded byload_sequence_rulesfrom the same artefact the counting rules use); all four translatable Sigma correlation types compile throughscripts/sigma_correlation.py, held byscripts/check_sigma_correlations.py, with seven refusal reasons recorded rather than approximated. No upstream Sigma correlation rule exists in this tree to import (D8), so the corpus is first-party and labelled as such. - 3.3 Enrichment inputs (parity 5.5) — identity privilege (18 rules)
resolved from the Phase 2.5 posture and SCIM attributes, and a per-tenant
first-seen store keyed by principal and attribute
(
services/fusion/app/services/first_seen.py, Redis with a lake backstop) closing 2 more. 5 rules were never unreachable — the gate kept its own copy of the matcher's operator list and had drifted from it (D9) — and 18 were retired with a reason rather than given fields nothing emits (D10).MAX_UNREACHABLE45 → 2, and the two that remain both need the 3.4 baseline, so they are named rather than hidden. Two matcher operators that three shipped rules already used were implemented. - 3.4 Behavioural baselines in CORE — the scoring core moved into
fusion as
services/fusion/app/services/baselines.pyrather than UEBA moving into CORE, measured and recorded in ADR-0011: the UEBA image is 145.4 MB on disk with a 63.3 MiB import floor, against zero new containers and a measured 2,412 bytes of Redis per principal (200 principals over 10,000 events,used_memoryread before and after against a real Redis 7). All three families the plan names are there — per-principal rarity of actions, resources, client families and locations; peer-group deviation; volume per principal and per OAuth app — each a typedAnomalySignal, never an alert on its own. Wired intoDetectionEngine.evaluate_asyncbeside the 3.3 first-seen store, so a CORE install computes them with nofullservice running. 22 tests, one of which caught this work's own worst defect: the first draft resolved client-family rarity from three field names no connector emits, so that signal could never have been computed anywhere. Every subject chain is now held against the gate's own emitted-field namespace, proved by reinstating the broken chain. The reachability gate's "behavioural baseline" family was narrowed: it matched^active_, which swept in a systemd unit state and an ITSM lookup and reported both as blocked on a baseline that would never have helped either (D20).MAX_UNREACHABLEstays at 2, now with accurate reasons. - 3.5 Threat-and-anomaly condition rules — 24 in a new
threat-anomalycategory (scripts/detection_specs_part4_threat_anomaly.py), each pairing a threat condition with a 3.4 baseline score, across AWS 6, Okta 4, Workspace 3, Azure/Entra 2, GitHub 2, Kubernetes 2 and one each of M365, Slack, Salesforce, GitLab and generic identity. Executable 2,511 → 2,535 (generate_corpus_stats.py --check), on disk 6,991 → 7,015, counted as its ownmake statsline because the claim differs: a rule here also needs a warmed baseline, so a day-one install has the rule before it has the signal. Every fixture is hand-authored and every negative is the benign twin of its positive — the same threat action by a principal for whom it is routine — so the pair differs only in the clause the rule is about.test_threat_anomaly_replay.py(147 assertions) drives the realDetectionEngineand holds four properties a synthesised fixture cannot: the twin moves a baseline field, no negative uses thenot-sentinel, stripping the baseline from a positive silences the rule, and one rule fires end to end in CORE on a baseline earned from replayed traffic with nofullservice running — the phase's second "Done when", as an assertion. The decorative-clause control caught a rule of this lane's own that was a bare anomaly (D22). - 3.6 395 → at least 800 cloud, identity, SaaS and code conditions
- 3.7 Detection engineering agent (gap-closure Phase 9 / parity 6.2)
Phase 4: Collection and scale
4.1 Cloud-native collection (S3+SQS org trail, Pub/Sub, Event Hubs)
4.2 Standard inputs (syslog, OTLP, Kafka, TAXII 2.1). All four bullets closed. The syslog bullet landed earlier (its entry is below and is unchanged); the three that were open are done here.
OTLP logs.
POST /v1/logsinservices/ingest/internal/otlp, readingExportLogsServiceRequestin both encodings the specification defines and gzip bodies, because real collectors emit both and a receiver that reads one of them is a receiver half the estates cannot use. The path is not ours to choose: an exporter appends/v1/logsto its endpoint. The partial-success contract is implemented rather than skipped — a partly unprojectable batch answers 200 withrejected_log_records, and a publish failure answers 503 withRetry-After, because a blanket 200 tells the collector records it never delivered have landed and a 400 makes it discard them.Kafka input.
services/ingest/internal/kafkainput, one source per tenant, the tenant from a minted token and from nothing in the record. The failure handling is the whole of it, and it is the sharedstreamhealthtable rather than a second copy: a rebalance is transient and the loop continues, aTopicAuthorizationFailedstops it naming the operator action, and an undecodable record is poison — skipped and committed, because a poison record that is not committed is re-read forever. A publish failure is the one case that must not commit, Kafka being the only input here that can withhold an acknowledgement.TAXII 2.1. A real server at
/api/v1/taxii2/overthreat_intel_iocs(migration110), serving discovery, API root, collections, objects, manifest andadded_afterpagination. A collection is a filtered view, never a copy. The API root is a fixed string because a tenant in the path is a segment any caller can retype, on an endpoint whose purpose is to be pointed at by a third party. Writes are refused with a 403 that names the posture.The refactor that came first, because it decides whether the Kafka input is a second copy of a table or a second user of one: the permanent/transient/poison classifier, the stuck-after escalation and the kafka-go reader-error bridge were private to
graph_ws. They moved toservices/ingest/internal/streamhealth, parameterised by the two genuinely per-subsystem things — the published metric stem, which keeps its name, and the environment variables the advice tells an operator to edit.graph_wskeeps its API through aliases and its 555-line test passes unchanged.Gated by
scripts/check_standard_inputs.py(self-test: nineteen injected violations, one per rule, plus a refusal on both an absent and an empty tree) wired intoci.yml :: python-lint, and bytaxii-live.ymlagainst real Postgres. 49 new Go tests (27 OTLP, 22 Kafka input), 37 Python projection tests, 15 live TAXII tests. Three claim rows added; the matrix reads 305 rows, 302 GATED.Four defects found by writing this, recorded rather than quietly fixed. A well-formed empty OTLP batch answered 400, because an
ExportLogsServiceRequestwith no fields set encodes to zero protobuf bytes and the body reader called that unreadable. The partial-success path had no reachable trigger, so the contract was a mechanism with no caller; a record carrying nothing of its own is now refused and counted.otlp-logswas absent fromALLOWED_TEMPLATE_IDS, so no operator could mint a token the receiver accepts — the receiver, its route, its template and its readiness entry were all correct and the feature was unusable. And the gate's own media-type rule matched the error message that names JSON rather than the case arm that accepts it, so a receiver that had stopped reading JSON entirely passed.Three verification gaps, stated rather than implied. No live OpenTelemetry Collector, no live Kafka broker and no third-party TAXII client has been exercised against any of this. The OTLP bodies are built with the vendor's own protobuf types, the Kafka loop is driven through a scripted reader so what is proven is the failure handling rather than kafka-go's wire protocol, and TAXII conformance is asserted against the specification's shapes rather than against a client's behaviour.
One thing found and left.
GET /api/v1/threatintel/stix/taxii/collectionsis a demo-gated stub returning a fixed sample list under the description "TAXII 2.1 collections for server compatibility". Removing a published operation is a breaking OpenAPI change, so it stays; its docstring now says plainly that it is not the server, and so does the new page.[~] 4.2 (historical) The syslog bullet, closed earlier.
services/ingestaccepts UDP and TCP and reads RFC 5424, RFC 3164, CEF and LEEF, with the tenant taken from a minted token because syslog carries nothing a sender does not control. Gated byscripts/check_syslog_listener.py(self-test: one injected violation per rule, including each of the three Go syntaxes a tenant could leak through) wired intoci.yml :: python-lint. 33 new Go tests. At the time this was written three bullets were open and were named here rather than left to be inferred: an OTLP logs receiver, a Kafka input for customer topics, and a TAXII 2.1 server over the tenant IOC store. All three are now closed; see the[x]entry above. This paragraph is kept as the record of what the syslog bullet alone covered, not as a live statement of what is missing.4.1 Cloud-native collection (S3+SQS org trail, Pub/Sub, Event Hubs). Three connectors (
aws_cloudtrail_s3,gcp_pubsub,azure_event_hubs), each resumable and each declaring a boundedcollection_budgetthat says where the overflow goes. Gated byscripts/check_cloud_native_collection.py(self-test: one injected violation per rule, plus a refusal on both an absent and an empty tree), wired intoci.yml :: python-lint. 83 new tests, connectors suite 1014 ->- One verification gap, stated rather than buried: no Azure
subscription is reachable from here, so the Event Hubs capture fixture was
synthesised from the Avro specification rather than recorded from a live
hub, and the Data Lake Gen2 listing path has never run against a real
storage account. The claim row is
PARTIALfor exactly that reason.
- One verification gap, stated rather than buried: no Azure
subscription is reachable from here, so the Event Hubs capture fixture was
synthesised from the Avro specification rather than recorded from a live
hub, and the Data Lake Gen2 listing path has never run against a real
storage account. The claim row is
4.2 Standard inputs (syslog, OTLP, Kafka, TAXII 2.1)
4.3 Retention the tenant chooses. All three bullets closed. The lake TTL is now a 400-day ceiling rather than a fixed 90 days, in
001_init.sql, in the tiering DELETE and in aMODIFY TTLmigration so an existing volume converges;MAX_LAKE_DAYSbounds what a tenant may choose and the three numbers are held equal. Legal hold now blocks the purge, throughmay_purgeso the decision has one home, with unreadable hold evidence withholding rather than proceeding, and the run reporting what it withheld. Gated byscripts/check_retention_window.py(self-test: eight injected violations, one per rule) wired intoci.yml :: python-lint. 13 new tests, api suite 4,569 -> 4,582.The third bullet is now closed too.
/lake/sql'scostwas the rate limiter's token price — a number about how often you may ask, not about how much the asking will read — and there was no progress state at all: a cold query either returned inside the published 30-second bound or raised a bare timeout.Every figure in the new estimate is ClickHouse's own: rows and parts from
EXPLAIN ESTIMATE, the warm/cold split fromsystem.partsjoined to the storage policy's volumes (a part is cold because the server says it is on the cold volume, not because of its age — those disagree whenever a move is in flight), and progress fromsystem.processeskeyed on a query id we set. There is deliberately no invented throughput constant: a predicted duration needs a bytes-per-second figure and the only honest source is a measurement on the deployment's own hardware, so the routing rule is structural — a query goes to a job when the estimate says it must read the cold volume.estimated_usdis null unless the deployment configuresAISOC_LAKE_COLD_RETRIEVAL_USD_PER_GB.docs/decisions/storage-cost-model.jsonprices residency, not reads; quoting it for a scan would be the wrong number stated confidently.nullmeans "we do not know what this costs" and0.0means "this costs nothing", and a test pins that they are never confused.Jobs are rows (migration
111), not an in-process registry: the API runs more than one replica, and that shape has been shipped and regretted three times here. 26 pure tests, 12 live ones against real Postgres, eight new rules incheck_retention_window.py, andlake-query-jobs-live.yml.The test that had to be rewritten, because the first one proved nothing. The claim race originally gathered five concurrent claims and asserted one winner — and it passed under a SELECT-then-UPDATE implementation as well, measured by subverting the claim and watching it stay green. It is now two explicitly interleaved transactions, and that version goes red when the predicate is removed.
Two of the eight new gate rules were wrong in the same direction and were caught by their own injections: the terminal-state rule matched a status string
list_stuck_jobslegitimately also uses, and the routing rule matched a response field name rather than the branch, so an endpoint that reportstouches_coldand ignores it passed.Not measured, and why: no ClickHouse was exercised. There is none in CI and none on the machine this was written on, so
EXPLAIN ESTIMATE, thesystem.partsjoin and thesystem.processesread are asserted against ClickHouse's documented shapes with the parsing tested on recorded rows. What is proven live is the job lifecycle, which is where the concurrency defects are.Two limits on what is closed, recorded rather than left to be found: the lake half is gated statically — no test in CI watches a row survive past day 90 on a real ClickHouse, so what is proven is that the numbers agree and the migration exists; and a hold stops the whole tenant's sweep rather than only its own subjects, because both purges are bulk statements. Narrowing that needs a per-subject predicate on each store and is follow-up.
[~] 4.4 Throughput (parity 6.9) and a cloud-hardware run. Every parity 6.9 bullet is closed; the cloud-hardware run is not, and cannot be from here.
Rules indexed — on required fields, not log source. The engine ran every rule against every event, and its own docstring already recorded why a log-source index could not be the fix: connector product names do not line up with spec products, a product-based pre-filter was tried once and reverted for silently dropping real matches, and a mapping table leaves every new connector one missing entry away from a detection that stops firing.
match_whenis an AND at the top level, so a rule that requires a field the event does not carry cannot match it — strictly more selective than log source and provably lossless. "Required" is decided by running the real matcher against an empty event, which is how writing the test found thatneqrequires its field in this matcher despite reading like a negation; a hand-written operator list would have been wrong in the direction that drops detections. Measured: 2,511 rules, 1,751 indexed (69.74%), 797 candidates for a routine cloud event (68.3% fewer), 2.760 → 1.066 ms/event over 2,000 iterations.Ordering per entity. Ingest keyed every message
tenant:event_id, unique per event, so nothing about one entity was ordered relative to anything — while fusion correlates on{tenant}:{entity}:{tactic}. Nowtenant:entityin the same precedence the correlation key uses, held equal by a test, because a partition key disagreeing with the correlation key routes an entity's events to a partition and then correlates them under a different name.Concurrent triage. Auto-triage awaited one LLM call at a time. Concurrency is per entity, not global: alerts about one host are the ones most likely to be one incident and triage reads memory earlier triages of that entity wrote, so racing them turns a deterministic verdict into a coin toss. Offsets commit only after the in-flight window drains, with a time bound as well as a count.
Lag hints.
GET /scale-hintpublishes the backlog and a suggested replica count. AiSOC does not act on it — ADR-0012 records why — and the per-replica budget is derived from the published realistic-mix run rather than chosen, with a test that reads the committed result file and fails if the two drift.Realistic mix.
--alert-every Nsends one alerting event every Nth and routine telemetry in between. Published run: 6,000 events saturating at 1-in-100, 0 of 5,940 routine events raised an alert, 0 missing, 0 duplicates, p95 event-to-alert 1,195 ms, ingest 122.6 events/s.The first attempt at that run measured a routine alert rate of 1.00 — every "benign" event became an alert. Not the corpus firing: the load path sends an undeclared connector type, the ingest normalizer's generic fallback files it as class 2001 Security Finding, and the promoter promotes OCSF category 2 unconditionally. Eight of the twelve declared profiles are category 2, so on those connectors no event can be routine whatever it contains. Also fixed: "missing events" was counted against every accepted event, so a perfect mix run reported losing 99% of its traffic.
ADR
0011. 51 new tests (12 index, 8 partition key, 18 entity lanes, 13 scale hint). Three claim rows; matrix 309 rows, 306 GATED.[~]and not[x], for one reason: there is no cloud-hardware run. No cloud instance has been provisioned for this project, so every figure on the performance page comes from a laptop or from a kind cluster on it. The plan's 5,000 events/s at p95 under 2 s target is therefore an aim rather than a result, and the page says so, with the exact command that would close it once an instance exists. A second, smaller caveat recorded there too: the published mix run went through the published images, not a build of this branch, so it is not a measurement of the rule index — that is measured separately and in process.[~] 4.5 Deployment completeness (parity 6.8). Three of four bullets closed; the managed-cluster run is maintainer-blocked.
The chart installs CORE.
infra/helm/aisoc/values.yamldeclared seven application services against CORE's sixteen: no response execution, no connector polling, no threat-intel store, no LLM gateway, no local model, no vector store.helm installproduced a deployment that could not act, could not collect and could not triage — and reported success, because nothing compared the two files.actions,connectorsandthreatintelare now entries in.Values.services;litellm,ollamaandqdrantare deliberately not, because that map defaults every image tag to.Chart.AppVersionand tagging a third-party image with AiSOC's version renders perfectly and pulls a tag that does not exist. Two of them need storage, so they are StatefulSets with volume claims.Restore covers backup. The two scripts already agreed on all six components; nothing held them that way, and a component added to one and not the other is discovered during an incident. Now compared in both directions.
Leader election. The scheduler adds one job per connector instance in process, so two replicas each polled every vendor on every cycle — and on a rate-limited API the second replica's poll is the one that gets throttled, which looks exactly like expired credentials. A Postgres lease (migration
112), not a Kubernetes Lease: the scheduler also runs under Compose and systemd, and an election that only works on Kubernetes is off wherever most first runs happen. The mechanism isUPDATE ... WHERE expires_at < now(), proved by two explicitly interleaved transactions rather than a race — a race between coroutines on one loop passes under a SELECT-then-UPDATE too, which this phase measured on its own lake-job claim. A follower removes its poll jobs rather than declining to add more.Gated by
scripts/check_deployment_completeness.py(self-test: nine injected violations, one per rule, plus a refusal on both an absent and an empty tree) wired intoci.yml :: python-lint, and byscheduler-lease-live.yml. 13 live lease tests; matrix 311 rows, 308 GATED.[~], for three things that could not be verified from here, named rather than implied:- No managed-Kubernetes run. The plan already says this one waits
on the maintainer providing a cluster. No managed cluster is
reachable from this machine, so
docs/perf/gains no row. helm installhas not been run on a cluster.helm lintis clean andhelm templaterenders 14 workload components, which is static evidence that the chart is well-formed — not evidence that it comes up. The kind install the plan's "Done when" names is the open half of this item.scripts/check_published_images.pycould not reach GHCR ("ghcr.io unreachable: The read operation timed out") and correctly refused a vacuous pass rather than reporting the tree clean. So the three new third-party image references and the three new first-party ones are not confirmed to resolve in the registry.
- No managed-Kubernetes run. The plan already says this one waits
on the maintainer providing a cluster. No managed cluster is
reachable from this machine, so
Phase 5: Response that finishes
- 5.1 Steps that act (parity 5.3)
- 5.2 Delivery (parity 5.7)
- 5.3
wait,parallel,loopwith idempotency keys - 5.4 Stateful user and manager verification. A
verify_userstep suspends the run on averificationpause (migration 102, a thirdkindonaisoc_playbook_pausesrather than a second table), asks the affected person over the configured Slack, Teams or email destination through the 5.1notifyarms, escalates to their manager fromaisoc_scim_users.manager_refwhen the stage deadline passes, and resumes the run with the signed answer in the context so an ordinaryconditionstep branches on it. Five outcomes a playbook can read:acknowledged,denied,unsure,timed_out,no_manager. The answer is HMAC-signed over(token, answer, stage)— the token alone would make a guess yield three working links, and without the stage the subject's own link still answers after the question has moved to their manager. The contract's caution is enforced inmay_auto_sendin that order: a confirmed true positive is never asked, with no opt-out and no exception for a human-initiated run, and everything else needs a row inaisoc_verification_policies— default-deny, with a writer atPUT /api/v1/verification-policiesso the control can actually be switched on. Ships off:AISOC_PLAYBOOK_VERIFY_EXECUTEpreviews and names who would be asked. Gated byscripts/check_stateful_verification.py(self-test: eleven injected violations, one per rule) wired intoci.yml :: python-lint, plusverification-live.ymldriving the real SQL against real Postgres. 20 new offline tests, 12 live, 5 SCIM; shipped playbookuser-verification-containment.playbook.json. Deviation D15 records thewaitdefect this item found and fixed. - 5.5 Executor arms 74 → at least 150
- 5.6 Plain-language playbooks
Phase 6: AI agent security
6.1 AI asset inventory. Reproduced:
scripts/check_ai_asset_inventory.pyon the unmodified tree reported nine problems across five of its eight rules — no migration creates the table, no model maps it, no route reads it, no writer exists, and the graph vocabulary has no label for an AI asset. Implementedaisoc_ai_assets(migration110, tenant-scoped,FORCE ROW LEVEL SECURITY, granted toaisoc_app), the ORM model,GET /ai-assets,GET /ai-assets/{id}andPOST /ai-assets/{id}/sanctionbehind a newai_estate:read/ai_estate:writepair held by five roles and mintable into an API key,AiAssetRecorderin the fusion consumer's raw-events branch, theAIAssetgraph label withOPERATESandHOSTSedges emitted at ingest (schemav1.2 -> v1.3), and the/ai-estateconsole page. 21 new fusion tests (suite 662 -> 683), 8 new Go tests ininternal/graph, 5 live-Postgres cases intests/isolation/test_ai_estate_live.py.The writer is the whole item, and the gate says so rather than the prose: the
WRITERrule requires the consumer to call the recorder andmain.pyto construct it, because an unconstructed recorder is aNoneevery call site skips in silence and the page would render "no AI estate" to a tenant that has one — the second half of this repository's recurring defect, the half module reachability cannot see.Two negative controls, both re-injecting a real defect rather than an invented one, both run against a real PostgreSQL 16 with every migration applied and the suite connected as the DML-only
aisoc_approle: reverting the identity constraint fromUNIQUE NULLS NOT DISTINCTto plainUNIQUEturns one provider-key row into three, each claiming to be first seen now (measured, not predicted — a provider key has no device, so under the default NULL semantics every sighting inserts); and a tenant policy ofUSING (true)lets tenant A read tenant B's estate. Each fails the suite and each is a step inai-estate-live.yml. The gate's own self-test injects one violation per rule and found two blind spots in the gate while being written — it read its own component's docstring as a fabrication, and aNodeAIAsset,substring match would have passed on a commented-out// NodeAIAsset,.Two limits, stated rather than left to be found. Phase 2.5 is open, so
effective_permissionsis a column with a resolver behind it and nothing writing it yet; the risk scorer returnsFalsefor an absent snapshot rather than treating unknown as privileged, and the console says in words that permissions did not contribute. And the three discovery sources see genuinely different things — runtime telemetry never carries a device, a provider audit log never carries either — sodiscovery_sourceis stored per row and the page prints what each source can and cannot see, because an unlinked device and an unlinkable one are different facts.6.2 Endpoint discovery with no new agent. Reproduced:
services/fusion/tests/test_ai_endpoint_discovery.pywas written first and 13 of its 16 cases failed on the tree as 6.1 left it, driving payloads in the shapeswindows_event,auditd,fleetdmandosctrlactually emit — read out of each connector's ownnormalize()rather than invented.The root cause is the one this repository keeps paying for, in its second form. 6.1's endpoint branch read
process.nameoff the OCSF event. The canonical OCSF profile mapstitle,external_idand the identity aliases and nothing else; all four endpoint connectors put their process fields in the connector payload, whichservices/ingeststores as a JSON string underraw_data. So the branch resolvedNoneon every real source and could never have produced one row — and its own unit tests passed throughout, because they built events in the shape the module expected. The recorder now reconstructs the same flat namespaceDetectionEngine._raw_fieldsdoes, andPROCESS_FIELDS/COMMAND_LINE_FIELDSname each connector's own spelling.Built:
packs/aisoc-ai-estate.yaml(5 queries — coding-agent processes, MCP server processes, MCP client configuration files, AI IDE extensions, provider key files; loaded through the realpack_loader, 5 packs -> 6),SOURCE_CAPABILITIESdeclaring what each of the five discovery sources sees and cannot see,apps/docs/docs/concepts/ai-estate-discovery.md, andscripts/check_ai_endpoint_discovery.py(6 rules, self-test injects one violation each). Fusion suite 683 -> 700.Negative controls: an ordinary process (
falcon-sensor,osqueryd,nginx,chrome,MsMpEng.exe,node), an unrelated IDE extension (esbenp.prettier-vscode) and an unrelatedpackage.jsoneach produce no asset, and an endpoint sighting is asserted to carry no model or provider — because filling those from a default would make the inventory assert something no source observed. The gate's own self-test re-injects the original defect (fields = _connector_fields(ocsf)->fields = dict(ocsf)) and requires NAMESPACE to catch it; writing it found the same read-your-own-docstring blind spot a second time, now fixed by stripping the docstring before matching.Honest limit, and the reason the claim row is PARTIAL:
crowdstrike,sentineloneandazure_defenderpoll the vendor's own detections, not process telemetry —crowdstrike.pyreads/detects/queries/detects/v1— so they cannot supply a process start, and the plan's wording that endpoint discovery comes from them is not satisfiable with the connectors this tree has. The documentation says so in a callout rather than leaving a reader to assume their EDR covers it. Separately, no osquery agent runs in CI: the pack is asserted to load and to select the columns the recorder reads, but no test has watched a realosquerydexecute one of its queries.6.3 Provider coverage beyond OpenAI and Anthropic
6.4 Agent identity and baselines (parity 6.7)
6.5 AI rule pack
6.6 AI containment and kill-switch
Phase 7: Phishing operations
- 7.1 Intake to purge (gap-closure Phase 10 / parity 6.4)
- 7.2 Header, URL and attachment analysis
- 7.3 Adversary-in-the-middle session reuse
Phase 8: Enterprise gates
- [~] 8.1 Identity and administration. Five of six sub-items done, each
in its own pull request; the shortfall is itemised in D14 below.
- parity 4.1 — a containerised Keycloak signs somebody in over OIDC and SAML, both legs observed to pass (#1327). The SSO claim row moves PARTIAL → GATED on that evidence, not on a relabel.
- 4.2 — WebAuthn as a console second factor and MFA enforced per role (#1328), on top of the TOTP, recovery codes and per-tenant enforcement fix pass 4.2 already shipped.
- 4.4 — the audit retention window is enforced and a truncated chain
still verifies (#1329). Export already existed on
AuditLogView.tsx. - [~] 4.5 — opt-in limit enforcement in
entitlements.py(#1332). The six console pages 4.5 also names are not built: see D14. - 4.6 — internationalisation. Not started: see D14.
- 4.7 — axe covers the case list and case workspace, the two views 4.7 left out, and the four defects it found are fixed (#1331).
- 8.2 ABAC, elevation and workload identities
- 8.3 Buyer security pack
Phase 9: Analyst surfaces
9.1 MCP over streamable HTTP (parity 6.6). All six bullets of parity 6.6 closed.
services/mcp/src/http.tsserves the SDK'sStreamableHTTPServerTransportatPOST /mcpwith an unauthenticatedGET /healthz;docker-compose.ymlgains anmcpservice under its own profile andinfra/helm/aisocanmcpcomponent, both off unless an operator asks for them. The credential is per request and there is no environment fallback — the transport is stateless with a fresh server per request, so revoking a key ends the conversations it opened rather than leaving a session id that outlives it. Every call goes throughPOST /api/v1/mcp/tool-calls/authorize, which resolves the tool's required permission from a catalogue generated from the TypeScript registry, checks it through the principal's ownrequire_permission(not a second copy of the logic), consults the tenant's grant, and writes anmcp:tool_callaudit row before the tool runs — reads included, which was the whole gap. The outcome is a secondmcp:tool_resultrow becauseaudit_logrefuses UPDATE by trigger. A tool whose registry entry saysreadOnlyHint: falsecannot run on a bearer token: it queues an approval fingerprinted over its exact arguments, the requester cannot approve it, and the approval is spent by the call it authorised. Gated byscripts/check_mcp_over_http.py(self-test: one injected violation per rule across all seven, plus the empty-tree refusal) wired intoci.yml :: python-lint. 32 new tests — 16 API, 16 over a real socket against a real API stub. Live end-to-end against a real Postgres 16 and the real API recorded in the PR: 19 tools advertised, a read allowed, an out-of-scope tool denied, the write tool refused at the tenant switch and then at the approval, approved by a second principal, allowed, and spent. 18 of 18 audit rows hash-chained, andDELETE FROM audit_logrefused by the trigger while the run was in progress.9.2 Semantic memory (parity 6.5). All three bullets of parity 6.5 closed, and the third one — "replay shows the accuracy delta" — produced a result that changed the design rather than confirming it.
aisoc_memory_items(migration105) is the store of record for past cases, runbooks and institutional memory; Qdrant is an index over it, reached through its REST API so this service gains no dependency and its lockfile does not move. Every item carries an author, a source, a scope, an expiry and a state, and the author is the authenticated principal, never the body — provenance a caller can set is not provenance. Two claims about one subject suspend both sides and recall returns neither until an analyst settles it; preferring the newer one silently restores a bad entry the next time the job that wrote it runs, which is the case the feature exists for. Recall is tenant-scoped in the Qdrant filter itself andsearch()raises rather than offering an unscoped branch. With no embedding backend recall is 503 with the reason, never an empty list.kind='case'is written automatically by the case-closure path, so the half of the "exists unwired" family that reachability analysis cannot see — a table every surface reads and nothing writes — does not apply. Measured byscripts/score_memory_recall.py(12 hand-authored low-overlap probes, mean lexical overlap 0.166,is_synthetic: true); its self-test is wired intoci.yml :: python-lint. 17 new tests, api suite +17. Live end to end against real Postgres 16, real Qdrant v1.7.0 and a real embedding backend, recorded in the PR. The measurement is published as it came out, and it is unflattering: see D11.9.3 Case depth (parity 5.6 remainder). All four open bullets. Alert merge needed no schema at all:
is_merged,parent_alert_idandchild_alert_idshave been onalertssince migration 001 with the ORM model as their only mention anywhere — the repository's recurring defect, and worth naming so "new" means something. Merge keeps every row: a duplicate is marked and pointed at its parent, the survivor's children are appended not replaced, re-merging and merging into a merged alert are both refused because each would orphan or cycle the chain, and the survivor inherits the case a duplicate was already on. Case merge (migration106) repoints the alerts and keeps both case rows, because deleting loses the audit chain, the custody entries and any vendor ticket already naming that id. Bulk triage is new in the route and the console: a checkbox per row, a select-all that selects this page only, and a bar whose Apply performs a real write and surfaces the preflight 409 with the real matched count instead of silently retrying.closedis deliberately unreachable there —bulk-closesnapshots every row first and a weaker second path to one destination is how the weaker one wins. Tenant custom fields are two tables rather than a JSONB blob so a field can be enumerated, rendered, validated at write time and retired cleanly; values are canonicalised on write. Workload is counted per request over rows the query selected, a merged-away case is not counted as work, and a mean over nothing isnull, never0.0— with the count it was taken over beside it. Graded by.github/workflows/case-depth-live.ymlagainst a real Postgres with three negative controls, because every statement here is Postgres-only and a SQLite stand-in would be a different query. 29 new tests (23 API live, 6 web). The live suite earned its keep immediately: it caught thataisoc_cases.alert_idsisuuid[]while the two columns beside it are JSONB, which a stand-in would have accepted.9.4 War rooms
9.3 Case depth (parity 5.6 remainder)
[~] 9.4 War rooms. Slack is complete; Teams is built and maintainer-blocked, and the block is stated in the product rather than here. One action (
POST /war-rooms/cases/{id}, and a console panel on the case) creates the channel, invites the assignees, posts the case header and records the room; a sweep worker posts the timeline and verdicts as they change; a decision taken in the channel comes back onto the case through a service-token-only route. Idempotency is the design rather than a refinement: every post is keyed by a fingerprint of the event value, unique per room, so a retry is a no-op, two replicas cannot double-post, and a case that goesopen → closed → openposts three times because a re-opened case is news. The sweep derives its events from the case row rather than a cursor, so a missed tick cannot lose one. Opening fails loudly (502 carrying the vendor's own sentence) because an analyst pressed a button; syncing fails soft and recordslast_error, which the console renders — a room that has silently stopped mirroring is worse than one that was never opened, and "never synced" renders differently from "synced with nothing to say". A chat button reaches four verbs andcloseis not one of them. 37 new tests (17 slack-bot, 9 teams-bot, 11 API). Open and stated, not inferred: creating a Teams channel needs Microsoft Graph with an Entra application registration and admin consent, which no maintainer has provisioned, so that route answers 501 naming all four missing settings and deliberately does not fall back to the incoming webhook — one shared channel per deployment would put the next incident's traffic in the last one's thread. The Graph calls themselves have therefore never run against a real tenant. Neither has the Slack path against a real workspace: it is covered by 17 tests against a fake client, and no Slack token exists here.9.5 Hunt to report
9.5 Hunt to report. A natural-language hunt or a retro-hunt sighting becomes a stored report answering what, how, where and when, with ledger citations, readable as JSON or HTML and exportable as PDF through the same WeasyPrint path and the same unavailability error the weekly digest uses — one renderer, because two drift and the PDF stops matching the page somebody approved. Three refusals carry the item: a hunt that has never run is refused (a report about an unexecuted query is a statement about the query, not the estate), an uncited report is refused at persist time (an uncited report is an assertion), and a section that cannot be derived stays empty and renders as "not recorded" rather than as a confident sentence about nothing. The retro-hunt builder computes the gap between when intelligence arrived and when the estate first saw the indicator, which is the number a retro-hunt exists to produce and which a reader would otherwise get by subtracting two timestamps. Graded by
.github/workflows/hunt-report-live.ymlagainst a real Postgres with three negative controls. 14 new tests. The live suite earned its keep three times on its first run:alertshas no singularaffected_host(it isaffected_hosts, a JSONB array, andentitiesis what fusion actually writes),investigation_eventshas nocase_idat all so the ledger is only reachable by joininginvestigation_runs, and that table'scase_idis TEXT rather than UUID so binding a UUID raises in asyncpg instead of coercing. Each would have shipped under a SQLite stand-in, and the middle one would have cited an empty set on every report.9.6 Native responder
[~] 9.6 Native responder. Two of three bullets closed; the third is maintainer-blocked and the block is in the product, not in this note. Device build:
mobile.ymlgains an Android job that runsexpo prebuildand GradleassembleDebug, asserts the APK really containsclasses.dex(a green build that produced nothing is a shape this repository has shipped) and uploads it — an APK that installs and runs on a handset. Not store-signable (a Google Play account) and not a release build::app:createBundleReleaseJsAndAssetsdies inside Metro's asset hashing on the runner, Node 20 fixes it, andscripts/check_toolchain_pins.pycorrectly refuses a second Node version because 20 left security support in April 2026 and no image ships it. Lowering that gate to make this job pass would have been the wrong trade, so the limit is stated instead. iOS compiles and links for the simulator on a macOS runner — the Expo config generates a valid Xcode project, the pods resolve and every native module builds. It deliberately does not run Metro's bundle phase, which dies on that runner inside one of expo-router's PNGs for a Metro/Node reason unrelated to this app; the Android job bundles, so that half is still graded. A device build needs an Apple developer account. Voice:apps/mobile/src/lib/voice.tstakes a recogniser that returns a string, so no audio type appears anywhere in the module and there is nothing for a later edit to upload; a device with no offline model is refused rather than silently using the platform's cloud recogniser; the transcript is shown before it is sent, because on-device recognition is worse and a security question misheard is a question answered about the wrong host. Gated byscripts/check_no_audio_egress.py(self-test: six checks, including that a comment saying the app does not record is not mistaken for the app recording), wired into bothci.ymlunconditionally andmobile.yml. Push:services/realtimesent every subscription throughweb-push, which signs a VAPID JWT and POSTs to the endpoint URL — a bare APNs device token is not a URL, so a subscription saved by the mobile app was accepted, counted and failed on every send, which is the worst of the three states because the count looked healthy.native-push.tsspeaks APNs and FCM HTTP v1 and distinguishes delivered, device-gone and not-configured, the last naming each missing variable and never reporting a delivery. 19 new tests (11 realtime, 8 mobile); mobile suite 34 → 42. Open and stated: no APNs.p8and no FCM service account exist here, so nothing has been delivered to a real handset, and no microphone has exercised the voice path. The README's claim table was inverted rather than deleted — it now has to keep saying that an iPhone device build and a real-handset push have not happened, andmobile.ymlfails if it stops.
Phase 10: Close the books
- 10 Re-run touched claim rows, update sibling progress files, one
[Unreleased]entry per phase, final report here
Item log
| Date | Item | What happened |
|---|---|---|
| 2026-10-11 | 9.3 | Reproduced: the alert merge columns had zero writers outside the ORM model; grep -rn custom_field services/api and grep -rn workload (excluding workload identities) both counted zero; and grep -rn bulk apps/web/src/components/alerts found only unrelated comments, so the only bulk verb in the product was POST /alerts/bulk-close with no console surface at all. The two-case-table trap named in the brief was already closed by migration 083 and check_one_case_table.py reports OK, so aisoc_cases is the table this item deepens. Three negative controls recorded in the PR and wired into the live workflow. The live suite found a real defect on its first run: aisoc_cases.alert_ids is uuid[] while ioc_ids and artifact_ids beside it are JSONB, and the merge statement had assumed JSONB. |
| 2026-10-11 | 9.1 | Reproduced: on origin/main the greps for StreamableHTTP in services/mcp/src/, for aisoc-mcp in docker-compose.yml and infra/helm/, for a per-tenant tool grant in services/api/app/, and for an mcp: audit action all counted zero, and check_mcp_over_http.py reported seven problems. Built the HTTP transport, the authorize/complete pair, the tenant grant table, the approval queue bound to a call's arguments, and the generated tool catalogue held in parity with the TypeScript registry. Four negative controls recorded in the PR; one of them found a hole in the new gate itself, which reported http.ts for reading the process-wide API key when the only mention was a comment saying it deliberately does not — the detector now strips comments before matching, and refuses a {...cfg} spread as well, because a spread is correct today and a credential fallback the day somebody reorders the literal. |
| 2026-10-11 | 9.4 | Reproduced: grep -rn 'conversations.create|channels.create|war_room' across every service and the console returned four hits, all prose — a playbook template describing a "Notify war-room" step and an eval fixture quoting the phrase. Both bots could only answer: slack-bot had one outbound route (an approval card) and teams-bot posts to a hand-configured webhook. Built the channel creation, the keyed sweep, the decision route and the console panel. Teams is 501 with its four missing settings named, which is the honest state and the one a stock install is in. |
| 2026-10-11 | 9.5 | Reproduced: grep -n hunt across app/services/report_builder.py and app/api/v1/endpoints/reports.py returned nothing — the report surface and the hunt surface did not know about each other, so a hunt produced rows and a retro-hunt produced a sightings row and neither produced anything a person could read or forward. Built the four-section report, the citation rule, the two builders and the JSON/HTML/PDF export. Three negative controls in the live workflow: allowing an uncited report, filling an empty section with a sentence, and reporting a hunt that never ran each have to turn the suite red. The live run found three schema facts a stand-in would have hidden, recorded in the item above. |
| 2026-10-11 | 9.6 | Reproduced: grep -c 'eas build|gradlew|xcodebuild|expo prebuild' .github/workflows/mobile.yml returned 0 and the workflow actively required the README to say no device build had happened; grep -rni 'speech|voice' apps/mobile/src apps/mobile/app returned 0; and services/realtime/src/push.ts sent every record through webpush.sendNotification, so a native token was accepted and could never be delivered to. Built the Android APK job, the simulator job, the on-device voice module and the APNs/FCM sender. Negative control: removing the on-device refusal so the cloud recogniser could be used fails two of the eight voice tests. The README guard was inverted rather than deleted — "we added a build so the honesty check is no longer needed" is how an ungated claim comes back. |
| 2026-10-11 | 9.2 | Reproduced: knowledge_base.py searched with to_tsvector/plainto_tsquery and nothing else; grep -rn qdrant services/api/app/ found only the migration runner and the tenant-deletion sweep, no recall path; and no memory row anywhere carried an author, a source, a scope, an expiry or a state. Built the store of record, the Qdrant index over it, the conflict state machine, the case-closure writer and the recall scorer. Two negative controls recorded in the PR: restoring last-write-wins on a contradiction fails three tests, and restoring vector truncation fails one. The measurement is negative and is published as such — see D11, which also records that it caught the truncation defect. |
| 2026-10-07 | 0.1 | This file created at base commit 1b8bc2d4. |
| 2026-10-07 | 0.2 | Every figure re-derived. Two matched exactly (detections, unreachable families); executor arms measured 74 against a captured 73; the cloud/identity/SaaS/code figure measured 395 against a captured 461 on a grouping the plan does not pin, recorded above. Two measurement caveats found: executable and quarantined overlap by 1,724 rules, and the "69% Windows" figure does not reproduce from the index. |
| 2026-10-10 | 2.7 | Reproduced: ADR-0007 closed by naming this decision and declining to make it, so the question was open and nothing in the tree answered it. Measured rather than estimated, on a running CORE stack: 16 services idle at 3,649.8 MiB, and ollama measures 2.96 GiB under inference against a 140.6 MiB idle, so CORE's working peak is 6,540 MiB and the headroom under 8 GiB is 1,652 MiB — which is what either option is spent out of, and is the number the estimate-based version of this decision would have missed. Neo4j 5.15-community with the full profile's own settings, loaded with 20,000 sessions, 500 users and 20,000 edges and then read with the aggregation 2.4 needs: 696.8 MiB resident, 42.2% of the headroom. The same rows in a tenant-scoped Postgres table with the three indexes the store needs: +10.9 MiB marginal resident in a container CORE already runs, 8,288 kB of relation, aggregation in 5.148 ms — 0.7% of the headroom and 64× less memory for the same data. Accepted: Postgres in CORE, Neo4j keeps the Session nodes in full. The ADR states what it does not decide — ClickHouse stays in full, so CORE still has no event lake. |
| 2026-10-10 | 2.4 | Reproduced: check_session_projection.py against origin/main reports all three declarations missing and exits 1 — there was no session concept in the tree at all, in any of the three stores. Fixed by deriving the key in exactly one place (services/ingest/internal/activity/session.go) and reading it in three (sessions.py for Postgres, lake_writer.py for ClickHouse, graph/session.go for Neo4j), on ADR-0010's condition that the arrangement is held by a gate rather than a comment. The id is a deterministic uuidv5 of tenant, source and key, so the three stores agree without any of them coordinating at write time. Negative controls: reverting the cloudTrailSession reordering fails three subtests plus TestAnAnonymousAssumeRoleNamesNobody; shrinking the closed set in either the Python tuple or the SQL CHECK is caught; a second deriver and an invented fallback key are each caught, performed as the exact edit a contributor would make. Writing the tests found the defect that mattered: the chain edge back to the originating human was being dropped on precisely the two calls that carry it, because AssumeRoleWithSAML and AssumeRoleWithWebIdentity have no userIdentity at all (the caller is not yet an AWS principal) and the early return on a missing identity skipped the issued-session read. Pointing the new gate at the pre-fix tree also found a defect in the gate: it died with FileNotFoundError instead of naming which declaration was missing, so a fourth self-test case now asserts a missing declaration produces a sentence rather than a traceback. Ingest suite green (12 packages); fusion 677 passed, 2 skipped. |
| 2026-10-09 | 4.3 | Reproduced: check_retention_window.py on the unmodified tree reported six problems — the lake deleting at 90 days against a 3,650-day cap, no MODIFY TTL migration, tiering deleting at 90, and the purge worker calling neither alerts_under_legal_hold nor may_purge and reporting nothing withheld. Fixed all but the cold-query bullet. Four negative controls recorded in the PR, including the positive control: a purge stuck closed withholds every tenant and fails four of the eight hold tests, which is the only thing that separates a working control from commendable caution. One pre-existing test (test_storage_tiering.py) pinned the 90-day tiering DELETE and now reads the shared constant; test_retention_worker.py's session double could not answer the new holds query, so the worker failed closed and five of its cases failed — the double was taught the query rather than the fail-closed behaviour relaxed. |
| 2026-10-09 | 4.2 | Reproduced: check_syslog_listener.py against origin/main reported "no Go sources under services/ingest/internal/syslog", and the only syslog path was POST /v1/inbox/cef, which needs a forwarder that already speaks HTTP. Implemented services/ingest/internal/syslog (UDP + TCP, both RFC 6587 framings, four wire formats, token-derived tenant, batched publish, /readyz subscription). Five negative controls recorded in the PR; one of them found a hole in the new gate itself, which reported OK while the tenant came from msg.Hostname through a struct field — the detector now knows all three Go syntaxes and the self-test exercises each. Writing the tests also found a shutdown deadlock: Stop waited on goroutines watching a context the caller had not cancelled yet. Ingest suite green, internal/syslog 33 cases. |
| 2026-10-07 | 0.3 | make up and make smoke (10/10) pass. Injection suite and load-harness baselines committed. make up-full deferred (D2) and hosted model rows blocked (D3). Fixing D1 was a precondition for the load-harness baseline. |
| 2026-10-09 | 3.3 | Identity privilege (18 rules) and a per-tenant first-seen store (2) built; 5 rules were never unreachable (gate operator drift, D9); 18 retired with a reason (D10). MAX_UNREACHABLE 45 → 2, the two remaining both needing the 3.4 baseline. Two matcher operators three shipped rules already used were implemented. |
| 2026-10-09 | 3.2 | Ordered and unordered sequences in the windowed engine; all four translatable Sigma correlation types compile, seven refusal reasons recorded. No upstream correlation rule exists in this tree to import (D8), so the corpus is first-party and labelled as such. |
| 2026-10-11 | 4.5 | Three of four bullets closed. Reproduced: check_deployment_completeness.py on the pre-fix tree reported six CORE services the chart renders no workload for. Added them (three in .Values.services, three as their own templates because a third-party image must not inherit .Chart.AppVersion and two need volumes), compared restore against backup in both directions, and built Postgres leader election for the connector scheduler (migration 112). Negative control: removing the lease's expiry predicate makes both interleaved transactions lead and three live cases go red. Not verified here: no managed-cluster run, no helm install on a cluster, and check_published_images.py could not reach GHCR and refused to pass vacuously. |
| 2026-10-11 | 4.4 | Every parity 6.9 bullet closed; the cloud-hardware run is not possible from here and is recorded as unmeasured rather than implied. Rule index on required fields (lossless, proven by replaying every fixture both ways; 68.3% fewer candidates, 2.6x on evaluation), entity partition key held equal to the correlation key, per-entity concurrent triage, lag-based scale hint derived from the published run. The realistic mix found that eight of twelve connector profiles are category 2 and are promoted unconditionally, so on those connectors no event can be routine — the first run measured a routine alert rate of 1.00. ADR-0012; matrix 309 rows, 306 GATED. |
| 2026-10-11 | 4.3 | Third bullet closed. Reproduced: check_retention_window.py with the new rules reported no cold-query estimator, no job surface and no job table on the pre-fix tree. Built lake_cold_query.py (EXPLAIN ESTIMATE + system.parts + system.processes, no invented throughput or price), lake_query_runner.py and migration 111. Negative control: removing the status = 'queued' predicate makes both interleaved transactions claim one job and the live suite goes red. The first version of that test — five gathered claims — passed under the broken implementation and was replaced. 26 pure + 12 live tests; matrix 306 rows, 303 GATED. |
| 2026-10-11 | 4.2 | Reproduced: check_standard_inputs.py against the pre-fix tree reported no OTLP package, no Kafka input and no TAXII server — the only TAXII-shaped route being a demo-gated sample list. Built all three, after moving the consumer-health classifier out of graph_ws into streamhealth so the new Kafka consumer shares the table rather than copying it. Negative controls recorded in the PR: a timestamp-only TAXII cursor loses an indicator in the single-row pagination walk (measured — 3 of 4), and nineteen gate injections each go red. Writing it found four defects, including an OTLP template nobody could mint a token for and a gate rule that read an error message instead of the behaviour. Ingest Go suite green; api projection suite 37; live TAXII suite 15. |
| 2026-10-09 | 4.1 | Reproduced: check_cloud_native_collection.py on the unmodified tree reported all three collection paths absent. Implemented aws_cloudtrail_s3 (SQS-notified S3 objects; management events, data events and VPC flow logs; delete-after-emit), gcp_pubsub (REST pull on a log-sink subscription; ack-after-read) and azure_event_hubs (Capture blobs over the Data Lake Gen2 JSON API, with a focused Avro OCF reader). Negative controls recorded in the PR: deleting the SQS message before the object is read, acknowledging the Pub/Sub batch before it is built, ignoring the Avro union branch index, and ignoring the flow-log header line each fail a named test, and all were restored. Connectors suite 1014 -> 1098 passed. |
| 2026-10-11 | 5.4 | Reproduced: check_stateful_verification.py against origin/main reported eight problems — no verification.py at all, verify_user absent from the control-flow table, no resume path, no escalation, no timed_out, no sweep, no migration and no writer for the opt-in table. Built the stage machine on the existing pause row rather than a second table. Three negative controls recorded in the PR: reverting the run-loop break fails three tests including the wait regression (D15), removing manager_ref=_extract_manager(body) fails three SCIM tests, and removing ENABLE ROW LEVEL SECURITY from migration 102 fails the live tenancy test with "tenant B read tenant A's verification policy". Two pre-existing tests pinned the old shape and were widened rather than relaxed: the control-flow table's membership (still an equality, now of four) and the sweeper's return dict. Writing the gate found a hole in the gate: its first block extractor matched \n\n\n, which finds nothing for the last function in a module, so the ordering rule was skipped entirely over an injected violation. |
| 2026-10-11 | 6.2 | Reproduced: 13 of 16 new cases failed, driving the shapes the four endpoint connectors actually emit. Root cause was 6.1's endpoint branch reading process.name off OCSF, which the canonical profile never populates — it found None on every real source while its own tests passed. Fixed by reconstructing the connector namespace. Shipped the osquery pack (5 queries, packs 5 -> 6), the source-capability table, the operator doc and a 6-rule gate. Fusion suite 683 -> 700. Claim row is PARTIAL: no osquery agent runs in CI, and the EDR connectors return vendor detections rather than process telemetry, so the plan's "CrowdStrike, Defender, SentinelOne" wording is not satisfiable with these connectors. |
| 2026-10-11 | 6.1 | Reproduced: the new gate reported nine problems across five of eight rules on the unmodified tree. Built aisoc_ai_assets (migration 110), its routes behind a new ai_estate:* permission pair, the fusion recorder that writes it on the default profile, the AIAsset graph label with OPERATES/HOSTS edges, and the /ai-estate page. Two negative controls against a real PostgreSQL: reverting NULLS NOT DISTINCT turns one provider-key row into three, and a USING (true) policy leaks across tenants. Claim matrix 302 -> 303 rows, 300 GATED. Writing the gate found two blind spots in the gate itself. |
| 2026-10-09 | 3.1 | All 74 decided. 50 translated into wd-* rules derived from each original's own clauses and replayed through the real engine (162 assertions); 24 refused with a reason across five kinds. MAX_UNREACHABLE 119 → 45, published executable 2,603 → 2,529, windowed 18 → 68. Three findings recorded as D4–D6 below. |
| 2026-10-11 | 8.1 | parity 4.1's containerised-IdP leg observed to pass, both halves: keycloak-oidc verifies a real id_token against Keycloak's published JWKS with an audience-mismatch negative control, and keycloak-saml walks the whole SP-initiated flow to the real ACS and signs [email protected] in as infosec, with an untrusted-key control answering 400. Run 38124780886, as ENVIRONMENT=test against the DML-only aisoc_app role. The SSO claim row moves PARTIAL → GATED on that evidence. Building it found two shipped defects no self-authored provider could expose: resolve_saml_connection bound :i untyped so :i IS NULL raised on asyncpg, and Keycloak's default role-list mapper emits one <Attribute Name="Role"> per role, which python3-saml refuses. |
| 2026-10-11 | 8.1 | WebAuthn as a console second factor and MFA enforced per role. Fixing the branch's own gates found three passkey_credentials reads scoped by user id alone on a tenant-scoped table, and a permission-model violation where the MFA policy route imported ROLE_PERMISSIONS directly; the role vocabulary now has one owner in role_grants.known_roles(). 23 of 23 live cases pass against real Postgres, and deleting the login integration fails 14 of them — measured, where the workflow had said eight. |
| 2026-10-11 | 8.1 | An enforced audit retention window with a re-anchored checkpoint, ADR-0013, and 12 live cases. Writing them found that migration 111 declared ON DELETE CASCADE on the checkpoint table and then made the cascade impossible; and, separately, that audit_log's own trigger from migration 101 blocks DELETE FROM tenants for every tenant that has written a row, so tenant deletion is impossible on main. Reproduced and filed as #1330 rather than fixed here. |
| 2026-10-11 | 8.1 | Opt-in per-tenant limit enforcement (migration 113, D12), wired to POST /connectors and POST /tenants/me/users through one helper, with GET/PUT /usage/enforcement as the operator surface. Nine live cases, half of them positive controls: a guard that never refuses fails 1, a guard stuck closed fails 4. |
| 2026-10-11 | 8.1 | axe extended to the case list and case workspace, the two of parity 4.7's five views that were left out. Four real defects found and fixed: two <select> controls with no accessible name (one of them the control that writes a case's status), two indistinguishable <aside> landmarks, and panel headings at h3 directly under the page's h1. |
| 2026-10-11 | 7.1 | Reproduced: against the unmodified tree, registry.list_vendors_for_capability returned [] for all five mailbox verbs and dispatcher.dispatch answered executor_not_found for purge_email over 79 registered builtin arms -- a phishing response that could block the sender's domain at the firewall and could not remove the message it arrived in. Added the five verbs to both halves of the vocabulary, two mail clients (Graph, Gmail with per-mailbox domain-wide delegation), ten vendor arms, migration 104, three intake adapters, campaign grouping, the governed fan-out and a report-mailbox worker. Only documented vendor APIs: purge is a soft delete on both and restore_email is a real reverse; there is no tenant-wide sender block because neither vendor exposes one through these APIs, and the per-mailbox scope is in the summary an analyst reads rather than only in a docstring. Everything ships off. Negative controls recorded in the PR: defaulting the feature flag on reds four tests, inverting the verification probe's polarity reds two, reading the envelope sender instead of the preserved headers reds four adapter tests including the one asserting the report mailbox is not proposed for purge, incrementing recipient_count instead of recomputing it reds the grouping test, and dropping recipient_count from the per-mailbox dispatch reds the fan-out test -- which is the quiet way a SEVERE action becomes a MODERATE one. Gates: check_action_contract 86 -> 97 executors, check_module_reachability 981 -> 986 of 1014, check_route_authz 262 -> 265 authorizing with both ceilings unmoved, matrix 302 -> 303 rows. Actions suite 1158 -> 1228; api 44 new cases, 17 of them against a real Postgres with the full chain. Recorded as PARTIAL, not GATED: no Microsoft 365 or Google Workspace tenant was exercised (D-7.1). |
| 2026-10-11 | 7.2 | Reproduced: grep -rl "authentication-results" services/ returned no match on the unmodified tree, so nothing in the product read a sender-authentication header and a message whose SPF and DMARC had both failed produced a verdict that had never looked. Built header analysis with a three-valued result, URL submission through the existing sandbox contract, the attachment hash lookup the submit path already had, and evidence_cited citations into the investigation ledger. Migration 111 stores the headers on the submission so a retriage replays them rather than re-fetching a message the purge has usually removed. analysed and unanalysed are counted separately in three places and summed in none. Negative controls: treating none/neutral as passes reds two tests; silently skipping an unanalysable URL reds three. That second control is the finding worth keeping -- it failed to fail on the first attempt, because every test built its own ArtefactResult list and never ran the producer, so five cases were added that drive analyse_message with the registry patched empty. A second finding came from the ledger suite's teardown: deleting a run cascades to its events and migration 008's trigger refuses, so the fixture was dropped rather than the append-only control defeated. Gates: matrix 303 -> 304 rows (300 GATED), check_text_bind_casts and check_raw_sql_columns clean over 6 new columns, check_route_authz unmoved. 45 new api cases, 6 of them against a real Postgres with the full chain. |
D-7.1 -- No Microsoft 365 or Google Workspace tenant was exercised
Every mailbox verb in 7.1 is built against the vendors' documented request
shapes and asserted with a recorded respx mock: the Graph $filter on a
bracketed internetMessageId, the move to recoverableitemsdeletions, the
inbox messageRules body, Gmail's rfc822msgid: operator, trash/untrash
and the settings filter. Nobody has watched either vendor accept any of
them. The maintainer-only list already carries "provide cloud and SaaS test
tenants"; this is the same block, named here so the claim row's PARTIAL has a
reason a reader can find.
Two specific consequences, stated rather than implied. A decoder and a fixture
that share an author share a misreading, so a field Graph spells differently
from the reference would pass every test here. And the Gmail filter body and
the Graph inbox-rule body were written from the API reference rather than
taken from a working call, which is the half of block_sender least likely
to be right first time.
The claim row returns to GATED when a live tenant is available. Nothing was
weakened to accommodate this: the row says PARTIAL, the operations page says
"treat the first run against your own tenant as the first run", and the
CHANGELOG entry says it too.
| 2026-10-11 | 2.5 | Reproduced: test_posture_snapshot_coverage.py on the unmodified tree asserted the gap — every resolver in effective_permissions/ was pure and the only production snapshot source returned {}, so every live call 412'd and the subsystem had never answered a question about a real estate. Five collectors now exist under services/connectors/app/connectors/posture/, reached by a __posture_snapshot__ sentinel through each connector's get_resource_config, and the coverage test is inverted: it now fails when a registered resolver has no declared collector. Building it found three things worth recording. (a) BaseConnector.get_resource_config declares at_ts: str | None = None while AWS, Okta and GitHub each required a positional ts, so a caller written against the base class raised TypeError on three of the connectors — standardised, with test_every_implementation_matches_the_base_signature holding it. (b) Google Workspace's directory read needs scopes the audit-log credential is not consented for, so the JWT is now built per scope: asking for both in one assertion would have broken alert polling on every tenant that had not pre-consented the new ones. (c) A snapshot is only an answer if you know when it was true — an alert is triaged minutes to days after the event, and a single mutable "current posture" row answers "what can this identity do now" while appearing to answer "what could it do when it acted", which is the worse failure because the number looks right. Migration 103 therefore versions snapshots on valid_from/valid_until (valid_to in the plan; valid_from/valid_until is the existing spelling in migration 018 and two spellings of one concept in one schema is worse than a recorded deviation — D17). Negative controls, both run against live Postgres and both restored: dropping posture_snapshots_live_idx leaves test_a_second_live_row_is_refused_by_the_database reporting DID NOT RAISE IntegrityError, and defeating the content-hash comparison in save_snapshot fails test_unchanged_recollection_does_not_open_a_version with two distinct row ids — without that comparison an unchanged estate re-collected hourly writes a version an hour and "when did this posture last change" becomes unanswerable because every row is a change. Evidence: 12 posture-store tests against live Postgres with the migration chain applied, 65 across the posture and effective-permission suites, 1,145 connector tests, 4,774 API tests. Eleven API failures are pre-existing on main and unrelated — recorded as D18. |
| 2026-10-11 | 7.3 | Reproduced: OktaConnector.normalize() returned None for session_id, src_asn, user_agent, client_browser and mfa_satisfied on a System Log row carrying all five, so the family had nothing to group by and could not have been written. Same shape as the outcome hoist in #1279 -- the engine flattens raw_event one level and authenticationContext arrives as a dict. Hoisted the session, network and client fields on Okta and Entra, then wrote four rules grouping by session rather than by user: three distinct-value rules at threshold 2 and one ordered sequence at critical. Fixtures hand-authored from an Okta System Log row and a Graph signIn resource; the negative twin inverts only the clause each rule is about. Windowed 72 -> 76; executable detections unchanged at 2,511, because a windowed rule is not in that corpus. Negative controls: removing the session_id hoist reds 4 connector tests; switching one rule's group_by to actor_email reds 2 engine tests. Two findings recorded as D-7.3a (the threshold engine windows by arrival and only sequences window by event time) and D-7.3b (the reported-message link is deferred with a reason). |
D-7.3a -- The threshold engine windows by arrival, not by event time
Found by a test that failed. An earlier version of
test_a_replay_outside_the_sequence_window_does_not_fire fed two events whose
event_time fields were two hours apart and expected the threshold rules to
stay quiet. They fired, because _evaluate_thresholds scores each Redis
member with time.time() while _evaluate_sequences reads
parse_event_time. Both events arrived in the same second, so the window
never moved.
This is recorded rather than patched. Arrival-time scoring is what makes a
threshold cheap, and a connector polling a vendor every five minutes delivers
a burst whose arrival order is the only clock fusion has for it. The
consequence is specific and worth knowing: the one-hour figure on the three
wd-aitm-* threshold rules is a budget on ingestion, not on the vendor's
timestamps. A replay that happened an hour ago and is polled now lands in
the same window as the sign-in. For this family that direction is the safe
one -- it widens rather than narrows -- but a future rule whose correctness
depends on the vendor's clock needs a sequence, not a threshold.
D-7.3b -- The reported-message half of 7.3 is deferred, with the reason
The plan's 7.3 asks for the session-reuse family "plus the link from a reported message whose URL precedes that sign-in". The session half is implemented and gated. The link is not, and inventing it would have been worse than deferring it.
The obstacle is not effort, it is that nothing can feed it today. The
reported-message URLs live in aisoc_phishing_submissions in services/api
(depth 7.1), and the windowed engine consumes Kafka events in
services/fusion; fusion has no read of the API's tables and should not grow
one for a detection. The alternative -- staging a gateway's phishing
verdict instead of AiSOC's own reported-message corpus -- was examined and
refused: the one secure-email-gateway connector in the tree, proofpoint,
fetches /v2/siem/messages/blocked only, so it never sees a click, and its
normalize() emits neither the recipient nor the URL. Hoisting fields for an
event the connector does not fetch is a rule that cannot fire wearing a rule
that can.
What would close it, in the order it has to happen: proofpoint (or a
netskope / zscaler web arm) fetching click events and emitting
recipient, url and the gateway's classification; then a sequence rule
staging that click and the MFA-satisfied sign-in by the same principal. The
AiSOC-authored half -- joining a tenant's own reported-message URLs to a
sign-in -- belongs in the API beside the campaign store rather than in
fusion, as an enrichment on the alert the session rules raise.