Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
AiSOC depth plan, wave 2: progress
Mirrors plans/aisoc_depth_wave2.plan.md, which is locked and is the source of
truth. This file is the mutable half. The only permitted edit to the plan file
is flipping a todo's status in its front matter; every deviation is recorded
here.
Legend: [ ] not started, [~] in progress, [x] done, [!] blocked (with
exactly what is needed).
An item is [x] only when all three of these are recorded against it: the test
or gate that failed on the pre-change tree for the stated reason, the change,
and the negative control showing that reverting it fails again.
- Base commit:
152c6d5f(origin/main, v21.0.0). - Plan captured at:
152c6d5. Figures in the plan body are hints at that commit; Phase 0.2 below re-derives each one and the tree wins. - Predecessor:
DEPTH_PROGRESS.mdtracks the first depth plan. Items that plan left open are carried here (see "Hand-over" below) and are never ticked in both files.
Hand-over from the first depth plan
Recorded once, at kickoff, as the plan's precondition requires. These items
move from plans/aisoc_depth_plan.plan.md to this plan and are tracked here
from now on; DEPTH_PROGRESS.md gains a cross-reference rather than a tick.
| First depth plan item | Carried to | Note at hand-over |
|---|---|---|
0.3 make up-full baseline |
wave 2 Phase 0.3 | DEPTH_PROGRESS.md D2 records why it was not captured there |
| 2.6 effective permissions | wave 2 Phase 1.1 | wave 2 adds usage overlay, computed tags, AWS semantics |
| 3.6 content depth | wave 2 Phase 2.1 | wave 2 adds the scenario families |
| 3.7 detection engineering agent | wave 2 Phase 2.5 | unchanged scope |
| 4.4 cloud-hardware run | wave 2 Phase 8.4 | maintainer-blocked at hand-over |
| 4.5 managed-cluster run | wave 2 Phase 8.4 | maintainer-blocked at hand-over |
| 5.5 verb breadth | wave 2 Phase 6.1 | wave 2 raises the floor to 150 arms |
| 5.6 plain-language playbooks | wave 2 Phase 6.2 | wave 2 adds the bounded model step |
| 6.3–6.6 AI estate | wave 2 Phases 7.1, 7.3, 7.4, 7.5 | wave 2 adds hosted agents and repo skills (7.2) |
| 8.1 sub-item 4.5 operator pages | wave 2 Phase 9.5 | unchanged scope |
| 8.1 sub-item 4.6 internationalisation | wave 2 Phase 9.3 | unchanged scope |
| 9.4 Teams half | wave 2 Phase 9.4 | maintainer-blocked at hand-over |
| 9.6 remainder (device evidence) | wave 2 Phase 9.2 | maintainer-blocked at hand-over |
| Phase 10 close | wave 2 Phase 10 | wave 2's Phase 10 closes both |
Lane B status (sources: 1.3, 1.6, 8.1 to 8.4)
| Item | State | Where |
|---|---|---|
| 1.6 Anonymizer and hosting classes at ingest | done, in review | PR #1586 |
| 1.3 HR and device context | done for three of six connectors, in review | PR #1589 |
| 8.1 CNAPP depth | not started, gap reproduced | item log |
| 8.2 NDR connectors | not started; ExtraHop buildable, Corelight blocked | item log |
| 8.3 Enrichment per tenant, cited | not started, gap reproduced | item log |
| 8.4 Scale and deployment evidence | blocked, recorded with exactly what is needed | item log |
Every row above has its reproduction recorded in the item log as a command or a failing test, including the ones that were not built: a gap nobody reproduced is a rumour, and the three not-started items are written up so the next lane starts from evidence rather than from the plan's prose.
Migration and ADR numbers claimed
Claim a number here, push the claim, then write the file. Two lanes once claimed the same migration number simultaneously and one had to renumber.
| Number | Kind | Lane / item | Claimed | File |
|---|---|---|---|---|
| 0015 | ADR | Wave 0 / Phase 0.4 | 2026-11-01 | docs/decisions/0015-siem-replacement-scope.md |
| 142 | migration | Lane D / Phase 6.3 | 2026-11-01 | tenant response schedule (business hours, on-call) |
| 143 | migration | Lane D / Phase 6.3 | 2026-11-01 | approval timeout outcomes on the durable pause |
| 144 | migration | Lane D / Phase 6.4 | 2026-11-01 | reserved — ticket-sync state, if 6.4 reproduces a gap |
| 145 | migration | Lane D / Phase 7.5 | 2026-11-01 | reserved — AI containment state, if needed |
| 0020 | ADR | Lane D | 2026-11-01 | reserved |
| 124 | migration | Lane A / 1.1 | 2026-11-01 | services/api/migrations/124_identity_privilege_facts.sql |
| 125 | migration | Lane A / 1.2 | 2026-11-01 | services/api/migrations/125_access_review_findings.sql |
| 126 | migration | Lane A / 1.4 | 2026-11-01 | released unused — 1.4 lives in the existing fusion baseline store (Redis), as the plan requires; no table was needed |
| 127 | migration | Lane A / 1.5 | 2026-11-01 | released unused — 1.5 reads posture_snapshots (103) and the lake; no new table |
| 128 | migration | Lane A / 3.1 | 2026-11-01 | services/api/migrations/128_identity_links.sql |
| 129 | migration | Lane A | 2026-11-01 | released unused |
| 0016 | ADR | Lane A / 1.1 | 2026-11-01 | docs/decisions/0016-privilege-context-in-scoring.md |
| 0017 | ADR | Lane A / 3.1 | 2026-11-01 | released unused — the confidence model is a table of eight rules in identity_resolution.py, each with its reasoning beside it, and the merge threshold's justification is in the module docstring. An ADR restating them would be a second place for them to drift |
| 146 | migration | Lane E / 7.1 | 2026-11-01 | AI provider coverage + chat compliance (labels and bounded snippet only) |
| 147 | migration | Lane E / 7.2 | 2026-11-01 | hosted agents and repository AI artefacts as asset kinds |
| 148 | migration | Lane E / 7.3 | 2026-11-01 | agent identities and per-agent baselines |
| 149 | migration | Lane E / 7.3 | 2026-11-01 | reserved: agent baseline observations |
| 150 | migration | Lane E / 7.4 | 2026-11-01 | reserved: AI rule-pack supporting state |
| 0021 | ADR | Lane E / 7.1 | 2026-11-01 | AI-chat compliance-log minimisation |
| 151–158 | migration | Lane F (analyst: 3.2, 3.3, 4.x, 5.x, 9.x) | 2026-11-01 | reserved block, see below |
| 0022 | ADR | Lane F | 2026-11-01 | reserved |
Highest at kickoff: migration 123, ADR 0014. Lane E holds the exclusive
block migrations 146–150 and ADR 0021.
Pre-allocated exclusive blocks, claimed at lane start so two lanes cannot pick
the same number: Lane D holds migrations 142–145 and ADR 0020. A
reserved number that is not used is released by striking it here at lane
close, never by a second lane assuming it is free.
| 0018 | ADR | Lane B / 1.6 | 2026-11-01 | docs/decisions/0018-ip-class-dataset-provenance.md |
| 130 | migration | Lane B / 1.6 | 2026-11-01 | services/api/migrations/130_alert_ip_class.sql |
| 131 | migration | Lane B / 1.3 | 2026-11-01 | services/api/migrations/131_workforce_context.sql |
| 132 | migration | Lane B / 8.3 | 2026-11-01 | services/api/migrations/132_tenant_enrichment_providers.sql |
| 133 | migration | Lane B / 8.1 | 2026-11-01 | services/api/migrations/133_cloud_security_findings.sql |
Lane B holds an exclusive pre-allocated block: migrations 130-137 and ADR
0018. Numbers 134-137 are reserved and unused unless an item below
claims one here first.
Highest at kickoff: migration 123, ADR 0014. Next free outside Lane B's
block: migration 124, ADR 0016.
Highest at kickoff: migration 123, ADR 0014. Next free sequential number:
migration 124, ADR 0016. Outside that sequence, Lane E holds the exclusive
block migrations 146–150 and ADR 0021, and Lane F holds 151–158 and
ADR 0022 (below).
Lane F block (exclusive, pre-allocated)
Lane F holds migrations 151–158 and ADR 0022. The block is pre-allocated
rather than taken from the next-free counter so that Lane F's many items cannot
collide with lanes A, B, D and E, which take theirs sequentially. Files as they
land:
| Number | File | Item |
|---|---|---|
| 151 | services/api/migrations/151_alert_occurrences.sql |
3.3 linked occurrences |
| 152 | services/api/migrations/152_root_cause_findings.sql |
3.3 cross-vendor root cause |
| 153 | services/api/migrations/153_attack_stories.sql |
3.2 stories |
| 154 | services/api/migrations/154_triage_questions.sql |
4.3 questions |
| 155 | services/api/migrations/155_hunt_source_reports.sql |
5.1 source reports |
| 156 | services/api/migrations/156_software_inventory.sql |
5.4 inventory |
| 157 | unused | — |
| 158 | unused | — |
Highest at kickoff: migration 123, ADR 0014. Lane A claimed migrations |
||
124–129 and ADRs 0016–0017, and used 124, 125, 128 and 0016. |
||
| The rest are released: a claimed-and-unused number is kept in this table | ||
| rather than deleted, so a later lane reading it can see the number is free | ||
| without having to prove the claim was abandoned rather than forgotten. | ||
Next free outside the block: migration 130, ADR 0018. |
Phase 0: Kickoff
- 0.1 This file exists, modelled on
DEPTH_PROGRESS.md, with the hand-over recorded above. - 0.2 Every figure re-derived against the base commit. See below.
- 0.3 Baseline artefacts committed under
docs/audit/depth2-baseline/with hardware, date and commit. - 0.4 ADR-0015 (SIEM-replacement scope) written as proposed: AiSOC federates to the SIEM rather than replacing it, so the README's "Not a drop-in SIEM replacement" is now a decided position with its cost argument recorded. Nothing is built from it in this plan.
0.2 Re-derived figures
Measured on 152c6d5f (the base commit) on 2026-11-01, Apple M-series
(arm64, macOS), from the generators the plan names. Captured values are the
plan body's; measured values are this tree's. The tree wins.
| Figure | Captured | Measured | Source | Verdict |
|---|---|---|---|---|
| Executable detections | 2,530 | 2,530 | scripts/project_stats.py |
matches |
| Detection rules on disk | 7,015 | 7,015 | scripts/project_stats.py |
matches |
| Threat-and-anomaly family | 24 | 24 | scripts/project_stats.py |
matches |
| Windowed rules | 76 | 76 | services/fusion/app/data/windowed_ruleset.json |
matches |
| Longest window | 86,400 s | 86,400 s | same file | matches |
| Registered connectors | 87 | 87 | scripts/generate_connector_count.py --check |
matches |
| Connector type ids | 98 | 98 | packages/types/src/generated/connector-types.json (members) |
matches |
| Executor arms (strict) | 89 | 89 | capability = "..." declarations under services/actions/app/live_actions/ |
matches |
| Distinct verbs | 39 | 39 | same, deduplicated | matches |
| Executor arms (loose grep) | 111 | 111 | same pattern without the line anchor, counting reverse_capability |
matches — and is still not the figure |
| Claim-to-gate matrix | 334 / 329 / 5 | 334 / 329 / 5 | scripts/check_claim_gate_matrix.py |
matches |
| Verdict best balanced accuracy | 0.357 | 0.357 (llama3.1:8b-instruct-q4_K_M) |
apps/docs/docs/benchmark.md |
matches |
| Constant-answer floor | 0.250 | 0.250 | same | matches |
| Hosted model rows | not measured | not measured | same | matches |
| Auto-close precision range | 0.091–0.571 | 0.091–0.571 | same | matches |
| Alert reduction (production key) | 33.3 % | 33.3 % | same | matches |
| Alert reduction (detection-shaped) | 24.5 % | 24.5 % | same | matches |
| Highest migration | 123 | 123 | services/api/migrations/ |
matches |
| Highest ADR | 0014 | 0014 | docs/decisions/ |
matches |
| Cloud/identity/SaaS/code conditions | ~358 | 344 by the method below | compiled rulesets | moved — see D1 |
Every figure the plan named reproduces exactly except the last, which the plan itself flagged as approximate ("about 358 ... record why it moved").
D1 — The cloud/identity/SaaS/code condition count has no formal rule yet
The first depth plan's 3.6 counts "executable cloud, identity, SaaS and code
conditions (native rules, windowed rules and threat-and-anomaly conditions
together)", by log_source product. No script in the tree implements that
sentence, so every published value of it — 461 at the first plan's capture,
395 at its re-derivation, "about 358" in this plan — was counted by hand with
a slightly different product set. That is the defect this plan's Phase 2.1
fixes by requiring a make stats line held by a --check gate.
Measured here, stating the method so the next reader can reproduce it: native
compiled rules whose product falls in the cloud (aws, gcp, azure,
kubernetes, docker, cloudflare), identity (okta, identity,
active-directory, entra, auth0, jumpcloud, ping, onelogin, duo),
SaaS (google-workspace, m365, slack, salesforce, snowflake, box,
dropbox, 1password, atlassian, jira, confluence, zoom,
servicenow, notion, hubspot) and code (github, gitlab, bitbucket,
npm, pypi) sets: 344.
Top contributors: aws 73, azure 46, kubernetes 37, gcp 35, okta 33,
identity 33, active-directory 17, google-workspace 10, docker 9,
m365 8, github 7, slack 7.
Windowed rules contribute 0 to that figure, and the reason is a finding in
itself: windowed_ruleset.json entries carry category but no product
field at all, so a product-keyed count cannot see them and a category-keyed
count would be counting a different thing. Phase 2.1's gate has to decide this
explicitly — either windowed rules gain a product, or the published figure
says "native only" — and whichever it picks, the number moves again. Recorded
here so that move reads as a definition being fixed rather than as content
disappearing.
The gap between 344 and "about 358" is therefore a difference of product sets,
not of content: nothing was removed from the tree between the plan's capture
commit and this one (they are the same commit, 152c6d5).
0.3 Baseline artefacts
Committed under docs/audit/depth2-baseline/ with hardware, date and commit
on each. These are the "before" for every later delta.
| Artefact | Status |
|---|---|
| Re-derived figures | captured (the table above) |
| Executor arm count | captured (89 strict / 39 verbs / 111 loose) |
| Cloud/identity/SaaS/code conditions | captured (344, method recorded in D1) |
| Verdict scorecard | the published matrix is the baseline; re-running it needs a local model pulled (see below) |
| Alert-reduction suite | captured |
make up + make smoke |
captured |
make up-full |
captured — closes the first depth plan's D2 |
Deviations
Recorded as they are found. D1 is above.
D2 — make up-full failed its first run on a port bind, and what that does and does not prove
The baseline host had nine unrelated containers running from other projects,
one of them lodestone-postgres-1 published on 0.0.0.0:5432. That is the
exact condition the port resolver was built for, and it is a more honest
baseline host than a clean one.
What happened, in order:
make upsucceeded. CORE came up, an administrator was created, andmake smokepassed 10 of 10 stages. Nodocker-compose.ports.ymlwas written during this run.make up-fullrecreatedpostgres(thefullprofile changes its environment) and the recreate failed:Bind for 0.0.0.0:5432 failed: port is already allocated.- Running
python3 scripts/resolve_port_conflicts.pyby hand detected the conflict immediately and wrote the overlay:postgres 5432 → 15432 (5432 is held by container lodestone-postgres-1). make up-fullthen succeeded: 22 AiSOC containers, postgres published on
Both up and up-full declare _ports as a prerequisite, and COMPOSE is
a recursively-expanded variable whose $(shell test -f ...) re-runs on every
reference, so the overlay is picked up by both targets once it exists. The
unexplained step is (1): the resolver ran as a prerequisite of make up and
did not write an overlay, while the same script three minutes later did.
What this does not prove. Step 1 ran against a tree where an earlier
backgrounded make up had been killed mid-pull, so the container and
network state at that moment is not reconstructable, and the two obvious
hypotheses — a probe that reads a held port as free (the SO_REUSEADDR
class, recorded as fixed) versus a resolver that correctly saw AiSOC's own
already-running postgres — cannot be separated from this evidence. A grep is
not a reproduction and neither is a sequence I cannot replay.
Carried as a follow-up, not claimed as a defect: reproduce on a clean
host by starting a foreign Postgres on 5432 first, then make up from a tree
with no overlay present, and record whether the overlay is written. If it is
not, that is a real first-run defect for the most common conflict there is
and belongs in its own fix with a negative control. Until that run exists,
this is an observation with its evidence attached.
D3 — the queue has never read identity privilege, contrary to the plan
The plan's 1.1 says "Prioritisation already reads identity privilege (v16)
and must read this". It does not. score_alert_priority has taken an
identity argument and carried a _PRIVILEGE_FACTOR table since v16, and
its only production caller — alert_queue.build_queue — passed
identity=None at alert_queue.py:570. queue_priority(), the one other
function that would pass one, has no production caller at all.
So the privilege factor was unreachable from the only surface where prioritisation changes anything. Recorded here because it changes 1.1's scope: the item is not "make the existing reader read a better source", it is "wire the reader", which is the repository's defining defect found again inside a module whose own docstring explains why an ordering nobody can interrogate stops being trusted.
Closed by Lane A's 1.1: resolve_identity_contexts() resolves each alert's
actor against the privilege version covering that alert's own event time,
one batched query per page, and build_queue passes it.
D4 — the container clock runs ahead of the host's, and a live
point-in-time test trips over it
identity_privilege_facts.valid_from defaults to Postgres now(). On this
machine the Docker Postgres container's clock measured ~17 ms ahead of
the host's Python clock, so a row written and immediately read back at
datetime.now(tz=UTC) landed before its own valid_from and the
point-in-time select correctly returned nothing.
That is the store behaving exactly as specified — an event predating a
collection has no posture — so the fix is in the test, which now reads one
minute ahead. Widening valid_from <= :at to absorb the skew would have
answered historical questions with newer snapshots, which is precisely the
defect migration 103 is versioned to prevent. Recorded because the symptom
("the row is there and the point-in-time read cannot see it") reads as a
query bug and is not one.
D5 — the verdict baseline in the plan is for a model this host cannot run
The published best balanced accuracy of 0.357 is
llama3.1:8b-instruct-q4_K_M. The only model pulled on this host is
llama3.2:3b-instruct-q4_K_M, so a run here is not comparable to that row.
Rather than quote the 8B figure as a baseline it did not measure, Lane A ran
the same 3B model on both trees — origin/main at 152c6d5ff and the
lane branch — so the comparison has one variable. Both numbers are recorded
in the item log below, and neither is published as the project's headline:
the scoreboard row for the 8B model is untouched.
Phase 7: AI agent security (Lane E)
- 7.1 Provider coverage (depth 6.3) plus enterprise AI-chat compliance logs stored as labels and bounded snippets only. #1585.
- 7.2 Hosted agents and repository AI artefacts as new asset kinds. #1585.
- [!] 7.3 Agent identity and per-agent baselines (depth 6.4 / parity 6.7) — not delivered; reproduced gap and what is needed recorded below.
- [~] 7.4 AI rule pack — four of the listed targets shipped and replayed (#1585); the rest need a producer or 7.3's baselines, each recorded below.
D5 — Lane F did not reach eleven of its sixteen items, and this is where they stand
Recorded rather than left implicit. Lane F covered 4.2–4.6, 5.1–5.5, 9.1–9.5 and 3.2–3.3. Five landed (3.2, 3.3 first half, 5.2, 4.6, 9.2/9.4 records); eleven did not. The reproduction work for each was done before the lane ran out of time, so what follows is measured state, not a guess, and is worth more to whoever picks them up than a fresh grep would be.
| Item | Measured state on this tree | What is actually left to build |
|---|---|---|
| 3.3 second half | first half shipped | Cross-vendor root-cause collapse (migration 152 is claimed and unused). alerts.external_id is the natural anchor; parent_alert_id / child_alert_ids / is_merged have existed since migration 042 with no writer, so check that vocabulary before minting a new one. |
| 4.2 reviewer pass | not started | The decision contract is assembled by reflex_observability.decision_contract and is not persisted — absent from persist_auto_triage's 24 kwargs. A reviewer pass needs it durable first. Closure is chokepointed at closure/authorization.authorize_auto_closure, so a reviewer cannot reach closure by accident. Ship only if it moves balanced accuracy or auto-close precision without lowering malicious recall. |
| 4.3 questions + rules | not started | Both engines are in grammar lockstep today. Operators are eq/ne/lt/lte/gt/gte/contains/startswith/endswith/in/not_in/exists/not_exists; neq is not one and a rule using it silently never matches. Migration 154 is claimed and unused. |
| 4.4 precedent + re-triage | verified, not built | See D3. Precedent reaches triage by two routes but matches only on exact fingerprint or rule/entity string; semantic_memory.py is a complete Qdrant recall with provenance that the triage path never touches. Re-triage is genuinely absent, and run_id is uuid5(alert_row_id:WORKFLOW_VERSION) with ON CONFLICT (run_id, seq) DO NOTHING, so a naive re-triage writes nothing — it needs a re-triage discriminator in the run id, and seq 0/1 are taken. |
| 4.5 vendor evidence | verified, not built | See D3. investigation_reads.py exists with 5 verbs over 7 vendors. Two real gaps: no GuardDuty finding-read verb (the alias resolves GuardDuty to a CloudTrail read), and run_auto_triage has no tool channel at all and says so in a comment, so a vendor alert that auto-triages never pulls vendor evidence. A deterministic context fetch, like _attach_identity_context, is a better shape than a tool channel: the ledger citation is then real because the fetch happened. |
| 5.1 hunt from a report | not started | Every piece exists separately: ssrf_guard.validate_outbound_url (a validator, not a fetcher — it does not bound response size, so 5.1 must add its own max_bytes and timeout), EvidenceEnvelope.wrap for nonce fences (use it, not wrap_untrusted), and the closed hunt schema in hunt/plan.py with hunt_plan_sql.py binding every value as a parameter. Migration 155 is claimed and unused. |
| 5.3 explorer | partly exists | The Data Explorer exists and already shows the generated query; /graph?entity= pivot exists. Missing: CSV export, cross-filters and a per-entity view. One latent bug to fix while there: ExploreView.tsx assigns res.esql into a field labelled SQL and runs it through lakeApi.sql (ClickHouse). |
| 5.4 software inventory | absent | No connector reads a code-platform SBOM API. packs/aisoc-inventory-baseline.yaml queries deb_packages, rpm_packages and homebrew_packages — but not the Windows programs table — and nothing consumes the results. assets and asset_vulnerabilities are host-level; there is no package-level table. Migration 156 is claimed and unused. |
| 5.5 retro-hunt at scale | absent | The retro-hunt subsystem is substantial and sweeps two stores with honest failure states. scripts/perf/ is ingest/throughput oriented; nothing measures sweep cost against indicator volume. Needs recorded hardware. |
| 9.1 voice | absent | voice.ts is an interface with no recogniser and no screen (see D4). expo-local-authentication is already installed and app/approval/[id].tsx already names the exact action in its biometric prompt, which is the 9.1 requirement. @react-native-voice/voice is already denied by check_no_audio_egress.py as cloud-capable — an on-device package must be added and allowlisted, with the self-test extended, since the self-test is what makes that gate non-vacuous. |
| 9.3 i18n | absent | Nothing at all: no library, no messages/, no RTL, no gate. Scope is the five surfaces gap-closure 13.4 names ( |
| 9.5 operator pages | 2 of 6 built; the matrix row is stale | SCIM tokens is built (ScimTokensPanel.tsx, wired at SettingsView.tsx:424) and organisation members is built, but the claim row at CLAIM_TO_GATE_MATRIX.md:299 still lists both as not built. Branding, retention, rule packs, delegations and the console over-limit state are genuinely absent. The enforcement bullet of parity 4.5 is already GATED and done. |
D3 — Four Lane F items did not reproduce as the plan describes them
Recorded under the "reproduce before building" rule: a grep is a hint, and four of Lane F's gaps turned out to be narrower than the plan's wording, or already closed. Nothing was rebuilt on the strength of the plan's sentence.
| Item | Plan says | Measured on this tree |
|---|---|---|
| 4.4(a) | analyst reclassifications do not reach triage | They do, by two independent routes. human_priors.record_human_prior writes author: "human", confidence: 1.0 into aisoc_institutional_memory under outcome:<evidence_fingerprint>, and context/dispositions.py renders the last five analyst decisions into the triage preamble. The real gap is narrower: matching is exact fingerprint (SHA-256 over 18 canonical fields) or rule-id/entity string, never similarity. services/api/app/services/semantic_memory.py is a complete Qdrant-backed vector recall with tenant filtering and provenance — and the triage path never touches it; its only consumers are /kb/query (opt-in), remember_closed_case and its own router. |
| 4.5 | read verbs "in investigation_reads.py" are to be added |
The file exists, at exactly that path, 987 lines, 5 read verbs across 7 vendors with contracts, executors, a closed agent allowlist, a parameter allowlist and autospec'd signature tests. Defender/EDR evidence pulling is fully built. The two real gaps: no GuardDuty finding-read verb (only an alias that resolves GuardDuty to a CloudTrail read), and reads are reachable only from the escalation path — run_auto_triage has no tool channel at all, and says so in a comment, so a vendor-originated alert that auto-triages never pulls the vendor's evidence. |
| 5.2 | the hunt scheduler is "Elasticsearch only" | Splunk is a live registered provider. SUPPORTED_PROVIDERS held two drivers, and backend choice is driven by what the tenant connected rather than by the translation. Separately, the plan's "scheduled hunts resolved their SIEM from process settings that were never declared fields" defect is already fixed and documented at length in event_warehouse/credentials.py. The real gap was the half that fix could not reach: a tenant who has connected nothing. |
| 5.2 tail | "add the missing link from a hunt into the governed detection-proposal flow" | The link exists. hunt_scheduler._propose_detection_from_hunt inserts a DRAFT into detection_rule_proposals with source='hunt-finding', gated by AISOC_HUNT_TO_DETECTION (default on), since wave 1. |
D4 — The mobile README claimed voice input that does not exist
Not a plan defect; a tree defect found while reproducing 9.1/9.2. The
capability table read "Voice input — Transcription is on-device by
construction", which is true of the module and reads as a working
feature. apps/mobile/src/lib/voice.ts has no recogniser behind it and no
screen importing it: the only file in the repository that mentions it is
that README. Corrected to say so, and the mobile.yml README-honesty guard
extended to pin both the voice row and the handset row, with the guard's
own vacuous-pass case (an empty README) asserted.
D3 - Three of the five anonymizer classes have no redistributable source
Item 1.6 names five classes and anticipates that one of them (residential proxies) may have no redistributable source. Measured, three do not.
The widely cited community list of VPN and datacentre ranges was the obvious candidate and was refused on the licence, not on the data: its repository carries no licence file at all. Recorded as a command rather than as a claim, because this is the kind of statement that goes stale:
$ curl -s -o /dev/null -w "%{http_code}" \
https://raw.githubusercontent.com/X4BNet/lists_vpn/main/LICENSE
404
An MIT-licensed repository cannot redistribute data whose terms nobody can read. Commercial anonymizer feeds exist and are licensed per deployment, which makes them a per-tenant credential rather than something to commit.
So vpn, public_proxy and residential_proxy ship with zero records and
are tenant-supplied only; tor_exit and hosting_provider ship populated.
The docs page says this in a table and in prose, the ADR records the
alternatives, and the claim-matrix row leads with it.
D4 - Tenant-supplied class lists are designed and not built
The fallback the plan names for an uncovered class is a tenant-supplied
list. It is not implemented, and three classes are therefore unavailable
rather than empty-by-choice. Said plainly in
apps/docs/docs/operations/ip-classification.md under "What is not here
yet" rather than implied to exist.
The design, recorded so the next lane does not re-derive it:
- the stamper is in Go at ingest and has no per-tenant database access, so the per-tenant layer belongs in fusion, where the tenant is known;
services/fusion/app/services/tenant_overlay.pyis the extension point. It already loads per-tenant rows from Postgres with hot reload and already exposes exactly this shape inderived_allowlist_fields(event)andderived_identity_fields(event): per-tenant derived fields applied at match time;- it needs a tenant-scoped table (CIDR, class, author, reason) with RLS, a
third source in
TenantOverlay._fetch(whose signature is a 2-tuple today and would become a 3-tuple), anip_classapplied only where the global dataset had no answer, and an API route plus a settings surface so something writes the table. A table fusion reads and nothing writes is the same defect as a table the console writes and nothing reads.
Not done because landing 1.6 and 1.3 unblocks Lane C, and a half-wired per-tenant path would have been worse than an honest gap. Nothing shipped depends on it; the three classes simply carry no records.
D5 - Workday and Kandji are blocked; three of the six named sources are built
Item 1.3 names six connectors. Three are built against public references verified on 2026-11-01 and cited by URL in each module: BambooHR, Rippling and Microsoft Intune. Three are not, for different reasons.
Workday. community.workday.com/sites/default/files/file-hosting/restapi/
answers 200, but the Staffing REST surface a tenant actually exposes is
per-tenant and the report-as-a-service shape most deployments use is defined
by the customer's own report definition, not by a fixed schema. There is no
public response document from which an honest mock payload can be written.
Needed: a Workday sandbox tenant, or one recorded (redacted) response
from Get_Workers or from a customer's staffing RaaS report.
Kandji. api-docs.kandji.io answers 200 and the landing page documents
the base URL, the token header, the rate limit and the response codes. The
device-list response's field names are rendered client-side and could not be
extracted. Needed: one recorded (redacted) GET /api/v1/devices
response, or the field list from the vendor's device-detail page.
Jamf Pro. developer.jamf.com/jamf-pro/reference/get_v1-computers-inventory
answers 200 and the field names are verified -- general.name,
operatingSystem.version, operatingSystem.fileVault2Status,
diskEncryption.fileVault2Enabled, userAndLocation.username,
userAndLocation.email, security.firewallEnabled -- so unlike the other
two this one is not blocked on the vendor. It is not built in this pass
for time, and plugins/jamf-mdm stays a manifest without a connector class.
The promotion is a known quantity: one module in the shape of
microsoft_intune.py reading GET /v1/computers-inventory with
section=GENERAL§ion=OPERATING_SYSTEM§ion=DISK_ENCRYPTION§ion=USER_AND_LOCATION§ion=SECURITY.
Recorded rather than built against an invented shape, which is the plan's own rule for exactly these vendors.
D6 - HR and MDM connectors get no ingest profile, and the first evidence for that was the wrong test
Corrected after CI. The conclusion held; the evidence did not, which is the half worth recording.
These three connectors publish no events at all. fetch_alerts returns an
empty list on each by design: an HR system reports slowly-changing state,
not security events, and their output is context the scheduler writes to
workforce_members and workforce_devices, never to ingest. A
connectorProfiles entry would be an entry nothing can reach.
What was wrong was the check behind it. The original entry said go test ./internal/normalizer/ passed and cited
TestConnectorTypeCanonicalTargetsAreDeclaredAndUnshadowed, which asserts
that every alias target is declared and asks nothing about a connector
having a profile. The test that does ask is
TestEveryDeclaredConnectorHasAnOCSFDecision, and it failed in CI:
connector "bamboohr" has no entry in connectorOCSFClass: its events
take the 2001 default, which is always promoted, and nothing records
that as a decision
connectorOCSFClass exists precisely to separate "decided to take the
generic path" from "nobody looked", and a connector with no entry reads as
the second. All three now carry a genericReason naming what they are.
The lesson is the one this programme keeps paying for: a local suite passing is not evidence that the specific question was asked. The original entry named a test by title and did not read what it asserted.
D7 - Zero check runs was a conflicted PR, not a saturated pool, and the two look identical from outside
Corrected. This entry first recorded the wrong cause, and the correction is worth more to the next lane than the original observation was.
What was true: the head commit of each Lane B branch returned
total_count: 0 check runs. What was wrong: attributing that to the shared
runner pool because 19 runs were queued and 18 in progress at the same
moment. The pool had in fact drained to a single queued run.
The real cause. Wave 0 was squash-merged into main, so main and the
Lane B base depth2/wave0-kickoff each carried DEPTH2_PROGRESS.md and
plans/aisoc_depth_wave2.plan.md by different histories - an add/add
conflict. Both pull requests were therefore mergeStateStatus: DIRTY, and
GitHub cannot compute a merge ref for a conflicted pull request, so it
creates no check runs at all. Zero runs was the conflict, not the queue.
The one call that tells the two apart, which is the durable half of this entry:
gh pr view <n> --repo beenuar/AiSOC --json mergeStateStatus -q .mergeStateStatus
DIRTY- the merge ref cannot be computed. Nothing can be graded, and no amount of waiting or re-pushing will change that. Merge the base in and the runs appear.BLOCKEDorUNSTABLEwithtotal_count: 0- the pull request is gradeable and the runs have genuinely not started. That is the pool.
A queue snapshot alone cannot distinguish them, because a busy queue is the ordinary state of this repository and is true at the same moment either way. Checking the queue first is what produced the wrong answer here.
Standing rule for this lane from now on: after any push, read
mergeStateStatus before concluding anything about the pool.
Resolved by merging origin/main into both branches, keeping both sides of
the tracker. Measured immediately afterwards: 108 check runs on
depth2/lane-b-sources and 109 on depth2/lane-b-workforce, against zero
before.
Item log
One entry per item as it lands: what failed before, the change, the negative control, and the measurement delta.
6.1 Verb breadth — done
What failed before. scripts/check_executor_arm_count.py --check on the
pre-change tree:
executor-arm-count: FAIL
89 executor arms over 39 verbs; the plan's floor is 150. Counted by the
declared rule (class-level `capability = "..."`), not the looser grep that
also matches reverse_capability arguments.
That is the reproduction, and it is also the fix for a second defect: the
figure had been published three ways. The loose grep reports 111 because it
counts reverse_capability= arguments, which are declarations about a verb
rather than arms of it.
The change. 62 new arms across AWS, GCP, Entra, Google Workspace, Okta,
GitHub, Slack, Salesforce, Snowflake, 1Password, Cloudflare Gateway, Zscaler
and Netskope, with 20 new verbs in the Capability enum and the
KNOWN_CAPABILITIES mirror, a capability contract each, and nine new or
extended vendor clients. Every endpoint was verified against the vendor's
current public reference and every URL committed in a module docstring was
fetched and required to answer 200 first — which caught two errors before any
code was written: admin.users.setInactive does not exist (404), so Slack
containment is removal plus admin.users.assign; and
developer.salesforce.com answers 403 to every non-browser request, so the
Salesforce citations are the help.salesforce.com pages carrying the same
semantics.
Measurement. 89 arms / 39 verbs → 151 arms / 60 verbs / 34 vendors,
by the plan's declared rule, read from the generated artefact
docs/audit/executor-arms.json.
Negative controls, all five run and recorded.
- Dropping
NetskopeAllowDomainfromCLOUD_SAAS_EXECUTORSwhile leaving the class in place: the gate exits 1 naming it as a declaration that "counts towards the arm total and answers executor_not_found at dispatch" — the padding shape the cross-check exists for. - Raising
REQUIRED_ARMSto 152: the gate exits 1 on the floor. - Making
_runcall the client factory on the dry-run path: 62 of 62test_dry_run_never_constructs_a_clientcases fail. - Renaming a keyword in one arm's client call (
user_name=→username=): the autospec'd double raisesmissing a required argument: 'user_name'and the signature test fails — which is the defect class that shipped twice (SearchSIEMExecutor,CreateNotableEventExecutor) because simulation mode never constructs the client. - Adding the Authorization header to GitHub's credential-revocation call:
test_github_credential_revocation_sends_no_authorizationfails. GitHub documents that an authenticated request there returns 403, so without this the leaked token stays live while an operator debugs a missing scope.
Wiring, per the seven-point rule. Enum member and KNOWN_CAPABILITIES
mirror for all 20 new verbs (check_action_contract.py compares them);
contract per verb, declared on the verb and not the vendor; executor wired to
a real client method with a create_autospec signature test covering every one
of the 62; credential tuple per arm re-derived from the factory's source by
ast and graded behaviourally; verification recorded as a gap with its reason
rather than claimed (see below); rollback where the vendor allows it and the
limitation recorded where it does not; KNOWN_ORPHANS unchanged at 7 and not
raised — no new verb was added to it.
The verification gap, recorded rather than papered over.
PostActionVerifier is keyed on ActionType, and these are live-action
capabilities with no ActionType, so the probe registry cannot be reached for
them — the same gap unisolate_host has carried since it was built. Every one
of these clients does implement its read-back, and those read-backs ship as
dispatchable read verbs (get_cloud_identity, get_oauth_grants,
get_secret_scanning_alerts), so a playbook can read the effect even though
governance does not treat the verb as self-verifying. The consequence is
enforced rather than assumed: a test asserts no new verb above read_only is
AUTOMATIC, because unverifiable means not autonomous.
Vendor limitations recorded, not worked around.
- Microsoft 365 application passwords: Graph publishes no endpoint for them,
so there is no M365 arm for that half of the plan's "Workspace and M365
OAuth-grant and app-password revocation". The Workspace half ships
(
asps.delete); the M365 half is absent because the API is. - 1Password: no
revoke_tokenarm. The SCIM bridge sees no user-issued API tokens, and the Connect and Service Account tokens that exist are managed outside it with no documented revocation endpoint. Suspension stops sessions. - GitHub deploy keys: deletion is not platform-reversible, because the host stores only the public half and re-adding needs key material AiSOC never holds.
6.2 Plain-language playbooks and a bounded model step — done
Reproduced first, and half of it did not reproduce. Depth 5.6's
drafter already exists: services/agents/app/playbook/nl_drafter.py turns a
sentence into a playbook, validates it against both the Pydantic model and
schemas/playbook.schema.json, falls back to a deterministic substrate when
no model is configured, and lands it with enabled=false so the editor is
the gate. POST /playbooks/draft-from-nl wraps it. Nothing there needed
building. The one part of 5.6 that does not exist is "previews it against a
chosen past alert" — see D3.
What this plan adds did reproduce: there was no step type that asks the
model a bounded question, and StepType had 26 members with no way to
express a decision the author did not want to make themselves.
The change. ask_model: a closed answer set written by the author, a
required on_invalid fallback branch, and a branch id taken from the
author's own map rather than from the model's text. Five surfaces in sync,
held by scripts/check_playbook_schema_parity.py, which reports
schema 27 | StepType 27 | runnable 27 | packages/types 27 and scans every
.ts/.tsx file for a sixth vocabulary rather than naming the files it
knows about.
"Cannot call a verb" is pinned by a test that tries. Three layers: the
step is routed through _CONTROL_FLOW rather than _HANDLERS (which is
where action_bridge is reached); an AST test asserts the module imports
nothing that can dispatch; and test_the_step_cannot_dispatch_a_verb makes
the model answer with the name of a containment verb, replaces the bridge
with a tripwire that fails the test if called, and asserts the step decided
nothing — because a verb name is just an invalid answer here.
Negative controls. Removing ask_model from _CONTROL_FLOW fails three
branch tests; deleting the enum-membership check in parse_answer fails
three answer-validation tests.
6.3 Time gates and timeouts — done
Reproduced. approval_rules.evaluate() took impact, declared approval,
confidence and tier, and nothing else — there was no way for a tenant to say
"not at three in the morning". And pause.expire_due had exactly one
behaviour: mark the pause expired and log it, so every approval in the
product behaved identically at its deadline.
The change. ResponseSchedule + apply_schedule in approval_rules.py
(vendored byte-identically into the API), migration 142 for the per-tenant
schedule and its windows with RLS, and migration 143 for the pause row's
declared timeout outcome.
The direction is the property, and it is asserted rather than described.
test_it_never_produces_an_automatic_decision sweeps every ActionImpact ×
ApprovalRequirement pair and asserts the schedule never lowers one. The
positive control (test_inside_business_hours_leaves_it_automatic) is the
one that catches a gate stuck closed, which reads as working caution.
One statement, not two. The first cut read the declared outcomes in a
separate cross-tenant sweep before expire_due. That was a second statement
racing the first and would have taken the tenant-predicate ratchet from 80
to 81 — which the gate refuses, correctly. Collapsing both into one
UPDATE ... RETURNING removes the race, removes the ordering hazard, and
moves the ratchet entry rather than adding one.
Negative controls. Letting apply_schedule touch a non-automatic
decision, or emit AUTOMATIC, fails six tests; treating on_call_now = None
as cover fails the one test named for it.
6.4 Ticket sync — verified, and one direction was broken
The outbound direction already works and needed no change. Driven end to
end for the first time — the production push_case and push_status_change
routes, the production Jira and ServiceNow connectors, and a mock HTTP
server returning recorded vendor shapes — all five assertions passed on the
first run: a case mints an issue carrying the aisoc-case-<id> label the
return leg resolves by, a ServiceNow incident carries its correlation id, a
status change patches the existing ticket rather than minting a second one,
a vendor refusal is a 502 that does not forward the vendor's body, and a
connector without the capability is refused.
The inbound direction had a defect that four correct unit tests could not
see, because each mocks the next layer: case_fanout patches
httpx.AsyncClient, the connectors push routes are exercised by nothing,
test_push_capabilities.py patches httpx.AsyncClient with a MagicMock,
and the inbound test mocks the database session. A chain of four correct
unit tests is not a loop.
The defect: the idempotency check was new_status == ref_row.status, an
exact equality. Jira Done maps to the intermediate resolved, so a
case a human had closed in AiSOC was moved backwards out of its terminal
state by an ordinary redelivery — and Jira retries up to five times while
ServiceNow retries until it gets a 2xx. No unusual sequence was needed, only
a close followed by a retry.
The fix refuses an inbound status that would move a case out of a
terminal state, scoped to terminal states only: an ITSM moving a ticket from
Resolved back to In Progress is a real transition this product should
mirror, and only at closed is a retry indistinguishable from a reopen.
Negative control. Reverting the guard fails two tests, including the redelivery test that found it.
7.5 AI containment — done
Reproduced. ai_inventory.py scores a provider key's exposure and
llm_usage reads OpenAI and Anthropic audit logs. Zero arms could act on
any of it, so a provider key found in a public repository was a finding with
a recommendation and no verb.
The change. 14 arms across OpenAI, Anthropic and an AI gateway: key revocation, role demotion, workspace removal, spend caps, and the reads behind them. Arms 151 → 165 over 64 verbs across 37 vendors.
The property this phase turns on: three providers, three different answers
to "can this be undone", carried honestly. OpenAI deletes a project key,
so that arm declares no rollback. Anthropic deactivates one, so it has a
real platform reverse and is the reason restore_token exists as a verb at
all. A gateway block is a documented pair. One revoke_token contract
claiming a platform rollback because Anthropic can would have given the
rollback path a route back it does not have on the other two.
Two things deliberately not duplicated, each pinned by a test. AI OAuth
grant revocation reuses the Workspace, Entra and Okta arms from 6.1, because
an AI application's grant is an ordinary consent and a fourth arm would be a
second code path to the same vendor endpoint. Agent process kill reuses the
EDR kill_process arms, because the agent is a process on a host.
A recorded vendor limitation. Neither OpenAI's nor Anthropic's public
admin API exposes a budget this platform can set, so set_spend_cap has
exactly one vendor arm and test_no_spend_cap_arm_exists_for_a_provider_that_has_no_budget_api
says so.
The kill-switch playbook ships enabled: false, asks the bounded
question before the approval rather than after it, and the approval
declares its timeout outcome: if nobody answers within two hours the gateway
key's spend cap goes to zero, which is the narrow containment rather than
the wide one. It is not services/actions/app/services/kill_switch.py,
which is the platform-wide autonomy stop and is not an agent control.
D3 — Depth 5.6's "preview against a chosen past alert" is still open
The drafter produces a schema-valid draft and lands it disabled, and
alert_trigger.py previews playbooks when an alert fires. What does not
exist is the affordance 5.6 names: pick a past alert, and see what this
draft would have done against it. POST /playbooks/draft-from-nl takes a
prompt and nothing else.
Recorded rather than built here because it needs a surface this lane does not own — the draft endpoint would have to resolve a historical alert into a trigger context, which is the console's alert reader rather than the playbook engine. The bounded-step half of 6.2, which is what this plan adds to 5.6, is complete.
D4 — The new verbs carry a recorded verification gap, not a probe
Every client added in 6.1 and 7.5 implements its read-back, and none of them
is reachable from PostActionVerifier, because that registry is keyed on
ActionType and these are live-action capabilities with no ActionType —
the same gap unisolate_host has carried since it was built.
Closing it properly means either adding 24 ActionType members (each of
which the contract gate then requires a legacy executor for, by design) or
re-keying the verifier on capability strings. Both are larger than this
lane's scope and neither should be done by widening an exemption list.
The consequence is enforced rather than left implicit: a test asserts no new
verb above read_only is AUTOMATIC, so unverifiable means not autonomous,
and the read-backs ship as dispatchable read verbs (get_cloud_identity,
get_oauth_grants, get_secret_scanning_alerts) so a playbook can read the
effect even though governance does not treat the verb as self-verifying.
D5 — The kill-switch playbook is recorded as outside the benchmark corpus
test_playbook_completion_rate.py requires every shipped playbook to match
at least one incident in the 200-case synthetic corpus, and every pack
category to cover at least one — "a category that matches no incidents is
dead weight". The AI-agent kill switch matches nothing, because the corpus
contains no AI-estate template at all.
Two wrong fixes were available and both were refused. Tagging the playbook
supply-chain would make it match github-pat-leak, which is exactly the
"matches by tag only" case the gate's own docstring says it exists to
prevent: blocking a gateway key does nothing about a leaked GitHub PAT.
Adding an AI-estate template to the corpus moves a published measurement
and belongs to the lane that owns detection content.
What shipped instead: one entry in the existing
_PLAYBOOKS_NOT_IN_BENCHMARK_DATASET list with the reason, the category
gate's exemption derived from that list rather than kept as a second
hand-written one (so the two cannot disagree), and a new ratchet —
test_the_not_in_benchmark_list_is_a_ratchet re-runs the evaluation with
the exemption emptied and fails if an entry would no longer be an orphan.
It uses the evaluator's own definition rather than a second predicate,
because a ratchet with its own idea of "covered" is a ratchet that drifts.
Proven by adding a covered playbook to the list and watching it fail.
To close this: an AI-estate incident template in
services/agents/tests/eval_data/synthetic_incidents.json, with the
scorecard and alert-reduction suite re-run, after which the entry must be
removed or the ratchet fails.
8.1 CNAPP depth - Lane B - NOT STARTED, gap reproduced
Reproduced, recorded as commands rather than asserted.
services/connectors/app/connectors/wiz.py declares exactly one GraphQL
operation, query IssuesTable. There is no vulnerability query, no
configuration-finding query, no secrets query and no runtime-detection
query. The four sibling cloud-security connectors each read one endpoint
and it is their alert stream in every case:
| Connector | The one read it makes |
|---|---|
orca |
two GETs against the alerts API |
prisma_cloud |
two POSTs against the search API |
lacework |
GET /api/v2/Alerts |
sysdig |
GET /api/v1/secureEvents |
So the gap is real on all five and is as the plan describes it.
What remains, scoped: extend the Wiz connector to vulnerabilities,
configuration findings, secrets and runtime detections where the vendor API
offers them; record the same depth check for the other four; surface the
findings as triage context keyed on the resource the alert touched,
not as alerts by default. The triage-context half is the part that needs a
design decision rather than a read: a CNAPP finding on a resource is an
input to a verdict about an event on that resource, which means a lookup
keyed on resource identity at triage time, in the shape
services/agents/app/context/identity.py uses for principals.
Risk the plan already names: Wiz's newer GraphQL surfaces are behind a customer tenant. Verify each field against the vendor's current public reference before writing it, and record BLOCKED rather than inventing a payload shape, exactly as D5 does for Workday and Kandji.
8.2 NDR connectors - Lane B - NOT STARTED, one half blocked
Reproduced. ls services/connectors/app/connectors/ | grep -iE "extrahop|corelight" returns nothing. The two network connectors that
exist are zeek_suricata and darktrace.
Reference availability, measured 2026-11-01:
| Vendor | Public reference | Verdict |
|---|---|---|
| ExtraHop RevealX 360 | https://docs.extrahop.com/current/rest-api-guide/ answers 200 |
buildable |
| Corelight | https://docs.corelight.com/ answers 401 |
BLOCKED |
So ExtraHop is a build and Corelight is not. Corelight needs either a
documentation account, or one recorded (redacted) detection payload plus the
endpoint and auth shape, before an honest mock can be written. Corelight's
Zeek output is open and is already covered by the zeek_suricata
connector, which is a different thing from the sensor's own detection API
and must not be passed off as it.
8.3 Enrichment per tenant, cited - Lane B - NOT STARTED, gap reproduced
Reproduced, and the gap is wider than the plan states.
services/enrichment is a Go service, not Python. Measured on this
tree:
grep -c tenantoverinternal/enricher/enricher.goandinternal/config/config.goreturns 0 and 0. There is no per-tenant dimension anywhere in the service, so per-tenant opt-in is absent rather than partial.- No reference to the credential vault anywhere under
services/enrichment/. Keys are per-deployment environment variables. RATE_LIMIT_RPSandRATE_LIMIT_BURSTare parsed inconfig.golines 59-60, assigned to struct fields at lines 92-93, and read by nothing else in the service. The plan's "reads and never enforces" reproduces exactly.URLSCAN_API_KEYis a config field (config.go:71) with no client.- No enrichment result is cited in the ledger.
What remains, and the one design decision it needs. The per-tenant
opt-in is a console settings surface with the tenant's own key in the
credential vault, which services/api owns the write path for. A Go
service cannot hold that write path, so either the enrichment service
learns to read vault tokens through a vendored read-path decrypt_dict
(the pattern services/connectors already uses), or the fan-out moves
behind an API route that resolves the tenant's credentials first. That
choice should be made explicitly rather than discovered during
implementation.
The rate limiter needs a test that proves it refuses, per the brief: a limiter asserted only on the allowed path is indistinguishable from no limiter.
8.4 Scale and deployment evidence - Lane B - BLOCKED, which is the correct outcome
The plan's own text allows for this: "published with hardware, date and commit, or recorded as blocked with exactly what is needed". Both halves are blocked on hardware nobody here has, and neither is blocked on code.
4.4, the cloud-hardware run. Already stated honestly on
apps/docs/docs/operations/performance.md under "No cloud-hardware run
exists yet, and that is a gap", re-read on 2026-11-01 and still accurate.
Every figure on that page comes from an 18-core Apple-silicon laptop, a
6-CPU Linux VM on it, or a kind cluster on the same machine. The target is
a sustained 5,000 events/s through detection at p95 under 2 s on one
documented instance size.
Needed: a cloud account and one named instance type. Nothing else. The
harness already records the hardware fingerprint, the load shape and the
commit into every result file, and scripts/check_perf_results.py already
refuses a result that drops any of them. The command that produces the row
is published on that page and is reproduced here so this entry stands alone:
python3 scripts/perf/load_harness.py \
--target compose --events 500000 --workers 32 --batch 100 \
--alert-every 100 --target-eps 5000 --drain-timeout 1800 \
--assert-benign-alert-rate-max 0.001 \
--out docs/perf/results/<date>-<instance-type>-realistic-mix.json
4.5, the managed-cluster run. The only cluster evidence in the tree is the three-node kind run of 2026-09-27, and the page already says what that does and does not prove: "kind nodes are containers on the same kernel, so this is not two machines' worth of anything". A kind run is not a managed-cluster run and must not be published as one.
Needed: a managed Kubernetes cluster (EKS, GKE or AKS) with at least three worker nodes and a load balancer, plus permission to install the chart into it. The first depth plan recorded the same requirement as maintainer action M3; this entry carries it forward rather than restating it as new.
What was deliberately not done: no figure was produced by extrapolation from the laptop runs. A laptop VM's disk, network and CPU scheduling are not an instance type's, and a number derived that way becomes a promise nobody can keep. "Not measured" with the exact requirement is the honest record, and it is the record this item asks for.
1.3 HR and device context - Lane B
Reproduced. On the current tree:
ModuleNotFoundError: No module named 'app.connectors.bamboohr'
and the state behind it, measured rather than asserted: 88 connectors in
_CONNECTOR_CLASSES, none HR and none MDM; import_context with exactly
one production caller, the manual POST at
services/api/app/api/v1/endpoints/graph.py:725.
Shipped.
| Piece | Where |
|---|---|
| HR connectors | bamboohr, rippling |
| MDM connector | microsoft_intune |
| Capabilities | PULL_WORKFORCE, PULL_DEVICE_INVENTORY |
| Context store | services/connectors/app/workforce.py, migration 131 |
| Scheduler caller | ConnectorScheduler._sync_workforce_context, on every poll |
| CORE read | services/api/app/services/workforce_context.py, merged into /graph/identity-context |
| Triage derivations | derive_employment_facts and _render_device_line in services/agents/app/context/identity.py |
| Citations | identity.basis names departed, departing, contractor and device counts |
| Marketplace | three plugins/<id>/plugin.yaml, index regenerated |
| Docs | three pages generated, all three in the sidebar |
| Count | regenerated, not edited: 87 -> 90 |
Negative control. Removing the _sync_workforce_context call from
_poll_one fails test_the_scheduler_calls_a_workforce_sync_on_every_poll
with the reason stated: the connector would be registered, configured,
polled and silently ignored. Removing each derived fact from
render_for_prompt fails its paired prompt assertion, and removing it from
basis fails its paired citation assertion, which is why both halves are
asserted for all four facts.
Numbers, from the generators.
| Figure | Value | Source |
|---|---|---|
| Registered connectors | 87 -> 90 | generate_connector_count.py --check |
| Connector docs | 90 documented, 41 generated / 49 hand-written | generate_connector_docs.py --check |
| Marketplace plugins | 80 -> 83 | build_marketplace.py |
| Claim matrix | 335/330/5 -> 336/331/5 | check_claim_gate_matrix.py |
One trap avoided and worth recording. generate_connector_docs._slug
writes the hyphenated filename for a connector with no existing page, so
microsoft_intune documents at /docs/connectors/microsoft-intune. The
connector's docs_url was written with the underscore first and would have
404'd; corrected before commit and verified against the generated sidebar
entry.
Deviations: D5 (Workday and Kandji blocked, Jamf deferred with its shape recorded), D6 (no ingest profile, with the reason and the check).
1.1 Effective permissions everywhere — shipped
Reproduced first. services/api/tests/test_identity_privilege_reproduction.py
and services/agents/tests/test_identity_privilege_tool_reproduction.py
failed on the pre-change tree: 15 of 20 api assertions and all 3 agents
assertions. The 5 api assertions that passed are the negative arms of the
condition-operator matrix, and they passed for the wrong reason — the
resolver denied every operator it could not evaluate, so "correctly refuses"
and "under-reports real access" were indistinguishable until the operators
were modelled.
What shipped.
effective_permissions/iam_conditions.py: 24 condition operators plus theIfExistssuffix and theForAnyValue:/ForAllValues:qualifiers, with a three-valued result. Undecidable is resolved per effect — an undecidableAllowdoes not grant, an undecidableDenystill denies — so both arms refuse to over-report access.effective_permissions/aws.py:NotActionandNotResourceinvert their sets; session policies cap the identity arm (intersected across several, never unioned) and not the resource-policy arm; withheld actions are reported indeny_actionswith the reason innotes.identity_privilege.py: tier derived from what an identity can write, never from the name of the role it holds, pluscan_escalate,data_access,third_party,cross_account, each with its evidence.- Migration 124:
identity_privilege_facts(versioned on the same interval as the snapshot it came from, one live row per principal enforced by a partial unique index) andidentity_permission_usage(the runtime overlay), both under RLS. identity_usage_collector.py: the overlay from the lake, returningmeasured=Falsewith a reason whenever the lake cannot be asked.
Wiring — the production callers.
| Surface | Caller |
|---|---|
| Fusion, per alert | fusion_engine.process() step 2.7 → IdentityPrivilegeResolver.resolve(), stamped at the alert's event time |
| Confidence | ConfidenceScorer.score() factor identity_privilege, weight 0.06, the eight existing weights scaled by 0.94 so the total is still exactly 1.0 |
| Console queue | alert_queue.build_queue() → resolve_identity_contexts() → score_alert_priority(identity=...) (see D3) |
| Typed pivots | investigation_tools.TOOLS["identity_effective_permissions"], reachable from POST /graph/investigate/query |
| Agent tool registry | tools/registry.py::default_registry, so the model is offered it and the call lands in the ledger |
Triage prompt + confidence_basis |
fused_alert_consumer._attach_privilege_context() → context/privilege.py, through the same /investigate/query door the registered tool uses |
Negative controls. Ten live-Postgres tests, each with one: the point-in-time read is contrasted against the live-row read it replaced and the two disagree; the unchanged-rederivation test is paired with a changed one that does open a version; the queue test reverts the context and the score collapses to the unknown baseline.
Gates. test_tool_result_channel.py caught the new tool missing from
AGENT_TOOL_NAMES and test_investigation_tools.py caught it missing from
the dispatch-coverage table. Both were the gate working, and both are fixed
rather than relaxed.
Measurements.
Alert reduction, services/fusion/tests/test_alert_reduction_real.py:
| Measure | Before | After |
|---|---|---|
| Production correlation key | 1000 → 667 = 33.3 % | 1000 → 667 = 33.3 % |
| Detection-shaped cohort | 1000 → 755 = 24.5 % | 1000 → 755 = 24.5 % |
Verdict scorecard, scripts/run_verdict_eval.py, same corpus and the same
model on both trees (see D5), Apple M-series arm64 / macOS, 2026-11-01,
llama3.2:3b-instruct-q4_K_M served by Ollama, 72 live model calls on each
run, 0 errored:
| Metric | Before (152c6d5ff) |
After | Delta |
|---|---|---|---|
| Balanced accuracy | 0.2604 | 0.2715 | +0.0111 |
| Malicious recall | 0.1944 | 0.1944 | 0.0000 |
| Abstention rate | 0.0139 | 0.0139 | 0.0000 |
| Auto-close precision | 0.4286 (3/7) | 0.3750 (3/8) | −0.0536 |
Both movements are run-to-run variance, not an effect of this change, and
the reason is structural rather than a judgement call. The verdict harness
sets no service token, so fetch_privilege_context returns empty,
render_privilege(None) returns "", and the prompt the model receives is
byte-identical on both trees. The auto-close figure moved because its
denominator moved from 7 to 8 — one additional item crossed the
consideration threshold — which is the smallest denominator on the
scorecard and the one most sensitive to a single sampled token.
Nothing here is published to the scoreboard: the project's headline row is
llama3.1:8b-instruct-q4_K_M at 0.357, which this host cannot run.
1.2 Access review and right-sizing — shipped
Reproduced first. services/api/tests/test_access_review_reproduction.py
failed 7/7: no module in the tree compared granted with used, and a grep for
access_review, right-sizing, best_fit_role or dormant_admin across
services/ and apps/web/src matched only Lane A's own 1.1 files.
What shipped. access_review.py (pure), access_review_store.py,
migration 125, five routes behind compliance:read, and a console page
at /access-review with a sidebar entry. Six finding kinds, each with a
required object_hint and recommendation — a read-only review whose
findings do not name the object to change leaves the reviewer with the hard
half of the work, which is why reviews do not happen.
The honesty property this item turns on. Usage-dependent findings are
suppressed wholesale when UsageOverlay.measured is false, not emitted
with a caveat. A caveat on "revoke this" is read by nobody, and the finding
is wrong in the direction that costs an outage. best_fit_role returns
None for the same reason, and also when no built-in covers the used set —
recommending a role that does not cover something a principal actually used
is a recommendation to cause an outage, which is the one error a
right-sizing feature must not make.
Three distinct empty states, because only one of them is good news:
never_run, nothing_reviewed (a review over zero principals) and
no_findings (a review over a real inventory). Rendering all three as
"0 findings" presents an estate nobody looked at as an estate with nothing
wrong.
Negative controls. Eight live-Postgres tests: unmeasured usage suppresses the usage findings and then the same principal with measured usage produces them; three runs produce one row and the run table proves three runs happened; a finding that stops reproducing is retired; an acknowledgement survives the next run; another tenant's finding cannot be acknowledged (404, not 403, so the route is not an existence oracle).
Defect found while building. The retire query hit the recorded
:name::type bind-cast trap — SQLAlchemy's regex backtracks one character
over a Postgres cast, so :principals::text[] declared a parameter named
principal. Seven of eight live tests failed with a syntax error. Spelled
CAST(:principals AS text[]); check_text_bind_casts.py stays green.
1.4 Sub-principal baselines — shipped
Reproduced first.
services/fusion/tests/test_sub_principal_baselines_reproduction.py failed
7/7. baselines.py kept a baseline per principal plus one volume
baseline per OAuth app; a token, a service account and an AI agent were all
folded into the human's baseline, which is backwards for the case these
baselines exist to catch.
What shipped. In baselines.py itself, as the plan requires — not a new
store. api_token, oauth_app, service_account and ai_agent become
subjects of their own, with four signals: acting outside its own baseline, a
machine cadence turning interactive, a new ASN within an hour of the token
first appearing, and reactivation after 60 days idle. All seven new fields
are in DERIVED_FIELDS, so a rule can clause on them.
classify_cadence tests interactive first, and the order is the
classifier. A person working 09:00–17:00 produces consecutive active hours
whose gaps are almost all 1, which is indistinguishable from an hourly cron
by regularity alone. What separates them is that a person calls several
times within one hour and a timer does not, so the repeat count has to be
consulted before the regularity is. The first implementation checked
regularity first and classified a human as scheduled.
The cross-language contract, and the second defect it caught. The GitHub
join key is written by Go (session.go, strings.TrimSpace, no case
folding) and consumed by Python, whose general field helper lowercases —
correct for a username, wrong for a base64 digest, and the obvious
implementation is the broken one. Both sides now read one committed fixture,
services/ingest/internal/activity/testdata/github_token_key.json, and it
immediately found a second divergence nobody had looked for: Go requires a
string type, while the Python side coerced a numeric hashed_token into
"12345" — a baseline key Go never writes. Go: 8 subtests pass. Python: 9
assertions pass, including a negative control that fails if the resolver
ever starts folding case.
Gate caught drift. check_detection_fields.py keeps its own mirror of
DERIVED_FIELDS and test_baselines.py compares the two in both
directions; it failed on the seven new fields, which is the gate working.
1.5 What changed before this alert — shipped
Reproduced first.
services/api/tests/test_change_timeline_reproduction.py failed 5/5.
What shipped. change_timeline.py joins versioned posture (what the
estate looked like, complete for collected providers) with audit events (who
did it, complete for ingested sources). Both are kept because a timeline
from one alone is missing either the attribution or the coverage. Six change
kinds over a closed action vocabulary — an unfamiliar action is not returned
rather than guessed into a kind, because a reader cannot tell a guessed
classification from a known one.
ChangeTimeline.measured distinguishes "read both sources and found
nothing" from "could not read either". An empty list with no such field
turns a lake outage into evidence of a quiet week.
Wiring. Registered as the typed pivot changes_before_alert and in the
agents tool registry, reaching the API through the same /investigate/query
door the privilege tool uses. The lake table is aisoc.raw_events, already
declared ClickHouse-owned in check_raw_sql_columns.py's UNMIGRATED_TABLES,
so no new declaration was needed — verified by running the gate.
3.1 Identity resolution — shipped
Reproduced first.
services/api/tests/test_identity_resolution_reproduction.py failed 7/7.
What shipped. identity_resolution.py (pure) over all six sources the
plan names, identity_resolution_store.py, and migration 128. Eight
rules, each carrying its own confidence, the evidence that produced it, and
a merged flag derived from one threshold.
The failure mode designed against is a wrong link, not a missing one. A
wrong merge makes the identity look richer, not broken, and nothing
downstream can detect it. The three canonical wrong merges are refused and
each has a test: [email protected] against [email protected] (stored as
a visible weak link, never merged), admin against admin (not even
returned as weak — it would fill a review page with noise that reads as
near-misses), and a shared role mailbox (demoted rather than treated as a
verified address match).
MERGE_THRESHOLD is 0.90 because the two errors are asymmetric: a missed
link costs an analyst one pivot by hand, a wrong link silently attributes
one person's alerts to another.
Both stores, one read interface. Postgres for CORE per ADR-0010,
SAME_PERSON edges mirrored to the graph for full, and reads come from
Postgres only — a read that preferred the graph would answer differently on
the two profiles for reasons no operator could see. Only merge-strength
links are mirrored, because a weak edge in the graph gets walked by every
traversal whose author did not know to filter it.
Negative controls. Seven live-Postgres tests. The ordered-pair CHECK is
exercised with a deliberate out-of-order insert that must be refused by the
database; the weak-chain test is paired with a strong link proving the walk
works at all; the bounded walk is compared at two depths; a human decision
is tested in both directions (confirm promotes a weak link, reject demotes
a strong one, neither touches confidence).
Measurements for 1.2, 1.4, 1.5 and 3.1
Alert reduction re-run after all four: 33.3 % (1000 → 667) and 24.5 % (1000 → 755) — identical to the Phase 0 baseline.
The verdict scorecard was not re-run for this batch, and the reason is stated rather than skipped: none of these four changes what the triage prompt contains on the eval corpus. 1.2 is a separate console surface with no triage caller; 1.4 adds fields to fusion's derived namespace that no corpus alert carries; 1.5 and 3.1 add tools the model may call, and the harness places no service token, so both return "could not read" exactly as the privilege tool did in the 1.1 measurement — where the prompt was proved byte-identical. A re-run would re-measure the same sampling noise at ~17 minutes per pass. If a later lane wires any of these into the prompt by default, the scorecard has to move with it.
7.1 — Provider coverage and AI-chat compliance logs
Landed early because Lane D's 7.5 depends on the provider work.
What already existed, and was therefore not rebuilt. Verified before
writing anything: the AI estate inventory (services/api/app/services/ ai_inventory.py, one score_risk, six asset kinds, tri-state sanctioned),
migration 110 and its RLS, the fusion recorder with its three discovery
sources, endpoint discovery, the ai_gateway connector, the two OCSF 6003
webhook templates, packages/aisoc-ai-sdk, the AI estate console page, and
sixteen executable AI rules — eight llm-* on the OpenAI/Anthropic
control plane (det-application-071…078) and eight ai-* on runtime and
guardrail findings (079…086). None of that was touched.
Reproduced gap A. LlmUsageConnector accepted openai and anthropic
and nothing else. Six of seven tests in
services/connectors/tests/connectors/test_llm_usage_provider_coverage.py
fail on the base commit; four of them drive the real HTTP path against a
mock server and get a ValueError from the constructor.
Reproduced gap B, found while wiring A and the more interesting one.
test_ai_asset_recorder.py writes finding.evidence.scopes into a
hand-built OCSF event and asserts the recorder reads
finding.evidence.scopes. Both halves are the test's own invention: no
connector produces that path and ingest does not synthesise it, so the
assertion held whether or not the production path worked. Driven through the
shape ingest really produces — the connector's dict as a JSON string under
raw_data — an Entra consent grant reached the inventory with no scopes at
all, so an AI notetaker consented into every mailbox scored identically to
one granted nothing.
Reproduced gap C. No table anywhere held AI-chat compliance data; the migration sweep for one returned nothing.
What shipped. Four more providers on the existing connector
(google_gemini, github_copilot, microsoft_copilot, entra_consent),
each against the vendor's current public reference, every URL fetched and
200 before committing. _from_provider_audit now reads through
_connector_fields, so scopes reach the one existing score_risk; an OAuth
grant is keyed on the application rather than the audit record. Migration
146 with five RLS'd tables, ADR-0021, a classifier, a client, three API
routes and a console panel.
Wiring proof. One score_risk path, unchanged — the new providers feed
the existing oauth_grant and provider_key kinds and add no second
scorer. Minimisation is schema-enforced (no column can hold a message;
VARCHAR(280); CHECK-constrained label vocabulary; no client method returns
text) rather than promised in prose. Erasure and retention come from the
tenant_id column name that discover_tenant_tables selects on, asserted
per table.
Negative controls. (1) Reverting only the fields.get("scopes") read
fails both recorder tests again with KeyError: 'scopes'. (2) Adding a
transcript Text column to AiChatContentLabel and changing nothing else
fails the forbidden-column and unbounded-TEXT sweeps.
Measurement delta. No change to the executable-detection figure: this item adds no rules. Rules reading these feeds are 7.4.
Deviation D3 — one provider, not several, for the compliance half. The
plan says "where a provider offers a compliance API". Exactly one does with
a complete public reference: Anthropic's, at
platform.claude.com/docs/en/manage-claude/compliance-*, whose pages were
fetched and returned 200. OpenAI's equivalent is behind a Cloudflare
challenge that answers 403 to every non-browser request, including its
ordinary audit-log reference, so no honest mock can be written from it. The
provider column is a CHECK constraint with one value today; adding a
second is a one-line migration. Blocked, with exactly what is needed: a
reachable public reference for the OpenAI Compliance API, or a recorded
payload from a customer tenant.
7.2 — Hosted agents and repository AI artefacts
What already existed, and was therefore not rebuilt. The whole asset
table, the one score_risk, the tri-state sanctioned, the console page and
its "not learned" rendering, and the recorder's three discovery sources. 7.2
widened two constraints and added two producers; it changed no scoring.
Reproduced gap. .laneE/repro_7x.py against the base commit:
AI_ASSET_KINDS held six values, hosted_agent present: False,
repo artefact kind present: False. Migration 110's CHECK constraint rejected
a seventh kind deliberately, so this is that refusal answered by a reviewed
migration rather than by dropping the constraint.
What shipped. Migration 147 (two kinds, two discovery sources, a
repository expression index), ai_repo_scan.py, hosted_agents.py, the
generated _vendor/ai_instruction_rules.py with its sync gate wired into
ci.yml, and the console's kind list and source-limitation copy.
Wiring proof — the single scorer. score_risk gained three signals
(unvetted_mcp_server, instruction_injection, unpinned_skill_source) and
both producers emit attributes keyed to what it already reads (tools,
mcp_servers, skills), so neither needs a special case and no second
scorer exists. DEVICELESS_KINDS is shared between the model and the scorer
so the four kinds with no device are not charged for lacking one.
Wiring proof — the injection channel. Repository instruction text and a
hosted agent's system prompt both pass scan_instruction_text at capture.
The patterns are generated from the agents guard's own _RULES table, so the
scanner and the triage guard cannot disagree about the same text; only
pattern names are stored, never the matching span. Proven in both
directions: a hostile instruction file is flagged and scores higher, an
ordinary one is flagged by nothing and scores exactly 0 with an empty reason
list.
Negative controls. Clean twins on every signal (clean instruction file,
clean hosted agent, vetted MCP server, every deviceless kind) each score 0
with no reasons, so every delta is attributable to one field. The vendored
rule table's --check fails against a stale copy, and the generator fails if
an excluded rule name leaves the guard.
Two defects found while building it, both recorded rather than quietly
fixed. path.lstrip("./") strips the leading dot of a dotfolder, which
would have turned most of the scan off silently — caught because the
classification test enumerates real paths instead of two examples. And a
committed JWT-shaped literal in the 7.1 test, which the repository's secret
scan correctly refused.
Measurement delta. No change to the executable-detection figure.
D4 — One hosted-agent provider, for the same reason as D3
GET /v1/agents is the one provider hosted-agent list with a complete public
reference. The GitHub Copilot coding agent is reachable only through its
audit events (copilot.swe_agent_*), which 7.1 now ingests but which are
events rather than an inventory; docs.github.com/en/rest/copilot/copilot-agents
is a 404. Blocked, with exactly what is needed: a published REST reference
for a second provider's hosted-agent list, or a recorded response from one.
7.4 — AI rule pack (PARTIAL, four of the listed targets)
What already existed. Sixteen executable AI rules: eight llm-* on the
OpenAI/Anthropic control plane and eight ai-* on runtime and guardrail
findings. Not rebuilt.
Reproduced gap. .laneE/repro_7x.py showed no rule anywhere clausing on
a Copilot audit action, a consent scope or an assistant's resource-access
volume, because until 7.1 no connector emitted any of them.
What shipped. Four rules on the feeds 7.1 added:
llm-copilot-content-exclusion-changed (critical),
llm-copilot-agent-mcp-config-changed (high),
llm-ai-app-granted-write-scope (high),
llm-copilot-bulk-resource-access (medium).
Replay proof. services/fusion/tests/test_ai_rule_pack_replay.py drives
the real DetectionEngine over the real compiled ruleset against fixtures
that are the committed output of LlmUsageConnector.normalize() — not
synthesized from the rules. Each negative twin differs in exactly one
decisive field, and a test asserts that property on the fixtures rather than
trusting it.
Negative control. Rewriting one rule's clause to the unprefixed vendor action — the exact shape of the measured GitHub failure where 1 of 12 rules fired on real output — and re-exporting fails that rule's positive replay while the other three stay green.
Measurement delta. Executable detections 2,530 → 2,534; on disk
7,015 → 7,019. Re-derived with python3 scripts/project_stats.py and
propagated by the generators to the README, ROADMAP.md, the truth table,
corpus-stats.json, both marketplace indexes and the packaged API copy. No
figure was hand-typed; test_readme_figures_gate.py passes (23).
Not delivered, and why — the remaining targets each need a producer that does not exist yet. Recorded rather than shipped as rules clausing on fields nothing emits, which is the failure this plan's replay rule exists to prevent:
| Target | What is missing |
|---|---|
| Provider key used from a new ASN | No provider audit log read here carries a source IP, so there is no address to resolve to an ASN. Needs 1.6's anonymiser/ASN dataset plus a provider feed that emits the IP. |
| Agent invoking a new high-risk tool | "New" is a per-agent baseline, which is 7.3 and is not built. The stateless engine can see the tool, not its novelty. |
| Rise in guardrail violations | A rate over a window per agent; same dependency on 7.3. |
| Token-spend anomalies per key and per agent | cost_telemetry.py records per-call cost and the dashboard aggregates it, but neither writes a per-agent baseline the engine can read. Wiring a second meter was explicitly refused. |
| Secrets or personal data labelled in AI chats | The labels exist (7.1, migration 146) but live in Postgres, not on the event stream the engine evaluates. Needs a promoter from aisoc_ai_chat_content_labels to an event, which is a design decision rather than a rule. |
| Agent process reading browser cookie or credential stores | Endpoint discovery sees AI processes; nothing emits a file-access event scoped to cookie or credential stores. Needs an osquery/EDR producer. |
7.3 — Agent identity and baselines: NOT DELIVERED
Reproduced gap, recorded for whoever picks it up. Two halves, both open:
- Agent identity. No module in the tree defines an identity for AiSOC's own
agents:
grep -rln "agent_identity\|AgentIdentity\|agent_principal"acrossservices/returns nothing. Parity 6.7 asks for short-lived credentials and least-privilege tool scopes visible in the audit log. - Per-agent baselines.
services/fusion/app/services/baselines.pydefines zerobaseline_*functions; the only AI-adjacent signal isbaseline_oauth_app_volume_z, emitted insideBaselineStoreand keyed on the OAuth application, not on an agent. Nothing keys a baseline on an agent identity, andcost_telemetry.pyhas no baseline writer at all.
Why it is not started rather than half-built. It is the dependency under
four of 7.4's six remaining targets, so a partial version would have produced
rules that cannot fire — and 7.5 (Lane D) depends on 7.1, which landed first
as instructed. Starting a per-agent baseline store without the agent identity
half would key it on whatever field happened to be present, which is how
baseline_oauth_app_volume_z came to be the only one.
What is needed: the full item, in order — agent identities with
short-lived credentials and least-privilege tool scopes first, then baselines
keyed on them (tools, resources, volume, token spend, hours) reading the
existing cost_telemetry.py rather than a second meter.
| Date | Item | Entry |
|---|---|---|
| 2026-11-01 | 3.3 | Reproduced: 5 of 7 cases in services/fusion/tests/test_alert_occurrences.py fail on the pre-change tree with "no statement touched alert_occurrences". A repeat's only residue was a source_event_ids append — capped at 200 and shared with multi-event correlation, so its length is not a repeat count — and alerts.first_seen/last_seen have had no writer since 001_init.sql. Shipped migration 151 (alert_occurrences, a rollup keyed (tenant_id, alert_id): exact count including the survivor, real window, newest-first bounded payload-id window), the sink write, and the measurement as its own labelled row: 28.4 % repeat folding (1000 → 716, 284 folded) published beside 33.3 % and 24.5 % correlation and never added to either. The reduction is identical before and after, which the docs state: folding is not new, keeping what it folded is. Balance asserted four ways summing to the denominator (658 retained alone + 58 attached to a group + 284 folded + 0 pending), plus a resolvability assertion that each folded repeat maps to the retained alert's own id. Writing the live suite found a defect in this item's own migration: the conventional WITH CHECK (tenant_id = current_tenant_id()) makes the row unwritable by fusion, which binds no tenant session — measured as aisoc_app (NOBYPASSRLS), the identical session inserts into alerts and is refused here, and because the write is best-effort it would have failed in silence (migration 123's shape). Negative controls, both measured: reverting the WITH CHECK fails all six live cases; removing the jsonb containment guard takes five redeliveries of one message from a count of 2 to 7. Baselines unmoved. #1588 |
| 2026-11-01 | 4.6 | Not measured, with the reproduction. make replay-eval reads a window of closed findings somebody's analysts already labelled; no design partner has offered one, so false-positive reduction, time to verdict and escalation rate are recorded as not measured on the replay page with the exact two-command reproduction. scripts/check_outcome_claims.py is what makes the record worth anything: while the page says an outcome is unmeasured, no published surface may print a number for it — including 0, which is the more damaging case because it reads as measured-and-reduced-nothing and looks like data. The prohibition is read off the page rather than hardcoded, so a real measurement retires it without editing the gate. Its first run flagged a true sentence (a table row naming escalation rate beside an unrelated "auto-refresh every 60s"), so the rule narrowed from same-line to same-clause with both that case and a real claim in the identical row shape pinned. 13 self-test cases; negative control names file and line. #1588 |
| 2026-11-01 | 9.4 | Blocked, verified rather than assumed. The Teams war room is built and POST /war-room answers 501 naming each absent Microsoft Graph setting instead of falling back to the incoming webhook, because one shared channel per deployment puts the next incident's traffic in the last one's thread. Already pinned by services/teams-bot/tests/test_war_room.py. Needs: an Entra application registration with Channel.Create, ChannelMessage.Send and Group.ReadWrite.All, admin consent, and a team id. Recorded as a claim row so the honest refusal cannot quietly become a fallback. #1588 |
| 2026-11-01 | 5.2 | Reproduced: resolve_provider required connected_types.intersection(provider.connector_types), and _execute_hunt turns the resulting UnsupportedTranslation into a hit count of zero — not an error state, so no case opens, nothing logs above INFO, last_run_at moves, and it is indistinguishable from "your hunt found nothing". That was the answer on every deployment whose data lives in AiSOC's own lake. Added LakeProvider (no connector types; the registry now allows such a provider, checked after the connector-backed chain so a tenant's own SIEM always wins) with tenant binding through compile_hunt, an unreachable lake raising rather than returning zero, and dropped conditions logged by name. ADR-0015's HUNTING OUTPUT label did not exist anywhere outside the ADR — now in the banner, the page title and the mail subject, with a sentence saying what the document does not claim. Emailed delivery did not exist either (aisoc_hunt_reports and the report scheduler's ReportArtefact were never connected); it ships off behind AISOC_HUNT_REPORT_RECIPIENTS. Negative control: removing the empty-connector_types arm fails the reproduction and the scheduler-wiring case. API suite 5042 passed. #1588 |
| 2026-11-01 | 9.2 | Blocked, and the README was overstating. See D4. Corrected the capability table and extended the mobile.yml honesty guard to pin both rows. Needs: 9.1's recogniser and screen first — nothing can speak to the app today — then an Apple developer account and provisioning profile for iOS, or a Google Play account for a store-signable Android build. The Android assembleDebug APK already installs on a handset, so once 9.1 lands the Android half becomes a device question rather than a credentials one. #1588 |
| 2026-11-01 | 3.2 | Reproduced: the grouper chained on a shared host, user or source-IP string, first match wins in a fixed order, with nothing on the result saying which of the three had done it — so a NAT egress address welds two intrusions into one story; an alert carrying a session_id (written by fusion since migration 102, alongside originating_human) got no more weight than one carrying a hostname; and ChainMember carried a per-member severity that nothing aggregated, so "a member benign alone may be escalated by its story" had nowhere to happen. Built on the grouper: typed links with three strengths (session/human > hash/domain > host/user/ip), story severity raised by exactly one tier only when the story spans three or more kill-chain stages with the reason carried on the assignment, and actor mode held at unknown until Lane A's 1.4 supplies cadence — two tempting inferences rejected in code and pinned by test. Measured: story recall 100 % on all three labelled scenarios (4/4, 3/3, 2/2), purity 100 % (5/5), 0 cross-rule stories of 63 on a detection-shaped cohort. The purity rule was found by the measurement, not reasoned out: the first cut refused a weak link only across detection rules and the suite measured 75 % purity, because the scenario alerts carry no rule_id and an antivirus quarantine sharing the NAT address joined a phishing-to-exfiltration story. A weak link now also may not join a story anchored on strong evidence; the asymmetry (a strong link may still join a string-held pile) is deliberate and tested. Negative controls, both measured: removing either refusal fails two cases. Reduction baselines unmoved at 33.3 % and 24.5 %. Open and recorded, not approximated: stories are still Redis-only — the assignment reaches the Kafka envelope and the realtime socket and evaporates, so no API read or case can show one — and the resolved-identity link waits on Lane A's 3.1. #1588 |
1.6 Anonymizer and hosting classes at ingest - Lane B
Reproduced. A Go test driving the real Normalize entry point, not the
classifier in isolation, so that a classifier package with passing unit
tests and no caller could not satisfy it. On the pre-change tree:
internal/normalizer/ipclass_stamp_test.go:68:20:
act.Location.IPClass undefined (type activity.Location has no field)
Nothing in the tree classified an address, and no dataset existed to classify from.
Shipped.
|---|---|
| Downloader | scripts/fetch_ip_class_dataset.py |
| Pinned snapshot | services/ingest/internal/activity/ipclass/data/ (21,726 records) |
| Go classifier | services/ingest/internal/activity/ipclass/ipclass.go, embedded |
| Production caller | activity.Project -> stampIPClass, so every caller gets it |
| Python loader | scripts/ip_class_dataset.py, vendored to services/fusion/app/services/ip_class.py |
| Live -> matcher | DetectionEngine._with_activity_location lifts the nested stamp flat |
| Replay -> matcher | ip_class_fields in both copies of the derived-field lockstep |
| Lake | 3 columns, 001_init.sql and lake migration 005_ip_class in lockstep |
| Alert | migration 130, fusion RawAlert, promoter, _INSERT_SQL, API AlertResponse |
| Console | SourceAddressClassSection on the investigation rail |
| Gate | scripts/check_ip_class_dataset.py, wired in ci.yml twice |
| Decision | ADR 0018 |
| Docs | apps/docs/docs/operations/ip-classification.md, in the sidebar |
Negative control. Removing the single stampIPClass(&act.Location) call
from Project, leaving the fields and the dataset in place:
--- FAIL: TestNormalizeStampsAnonymizerClass/tor_exit_relay
ip 2.56.10.36: ip_class = "", want "tor_exit"
--- FAIL: TestNormalizeStampsAnonymizerClass/cloud_hosting_range
ip 1.178.1.0: ip_class = "", want "hosting_provider"
Restored, the suite passes again.
Numbers, from the generators.
|---|---|---|
| Dataset records | 21,726 | check_ip_class_dataset.py |
| tor_exit records | 1,228 | same |
| hosting_provider records | 20,498 | same |
| Sources | 8 | same |
| Classes shipped | 2 of 5 | same (see D3) |
| Go/Python parity sample | 400 addresses, 0 disagreements | same |
| Claim matrix | 334/329/5 -> 335/330/5 | check_claim_gate_matrix.py |
Scorecard and reduction suite. Not re-run: this item adds a field that no
shipped rule matches on yet (Lane C's 2.1 is what will), changes no
correlation key, no confidence input and no priority input, and the fusion
suite's 1,065 tests pass unchanged. Recorded as a deliberate judgement
rather than an omission; Lane C's first rule reading ip_class is the PR
that has to carry the delta.
Deviations: D3 (three classes have no redistributable source), D4 (tenant-supplied lists designed, not built).
7.3 — Agent identity and per-agent baselines (depth 6.4 / parity 6.7)
#1601, base main.
Reproduced gap. docs/audit/depth2-baseline/lane-e-phase73-reproduction.py,
runnable from a bare checkout: no module under services/agents defines an
agent identity; ToolRegistry.execute takes no caller, declares no scope and
writes no audit row; baselines.py declares seven derived fields and none is
keyed on an AI agent.
Half A. Short-lived identity per run, least-privilege tool scope enforced
at ToolRegistry.execute — the one chokepoint every tool call passes
through, because a check an agent performs on itself is one a steered agent
can be talked out of. Identity is required, not optional. Nothing in the
scope table changes an estate; a test forbids it.
Half B — one mechanism, built for Lane A. SUB_PRINCIPAL_FIELDS replaces
the hardcoded OAuth-app block, with the dict key as the field prefix so
baseline_oauth_app_volume_z keeps its published name. Lane A's 1.4 adds
token and service_account as one entry each. Eight new derived fields.
Negative controls. Making identity optional fails one named test;
removing the agent entry fails eight baseline tests.
7.4 — four more targets shipped, two still blocked
Executable detections 2,530 → 2,534 on main's base (and 2,534 again on
the landing branch, which carries the other four — the two PRs touch
different rules and the figure is regenerated in each).
Shipped and replayed through the real connector, the real BaselineStore
and the real engine: ai-agent-new-high-risk-tool,
ai-agent-guardrail-violation-rate, ai-agent-token-spend-anomaly,
ai-gateway-key-spend-anomaly.
D5 — The 7.1 labels reach the engine as events, not as a query
Decided explicitly, as the brief requires, and not yet implemented.
The chat-content labels from 7.1 live in aisoc_ai_chat_content_labels in
Postgres; the detection engine evaluates a stream. Two options:
The rule reads it where it lives — refused. The engine is stateless and evaluates per event with no database handle; giving it one would mean a Postgres round trip per event on the hot path, and the one place that pattern exists (the tenant tuning overlay) is a per-tenant cache loaded once, not a per-event read.
The label reaches the stream — chosen. The collector emits one OCSF event
per newly labelled conversation, through the same ingest path a connector
uses, carrying the labels, the shape names and the bounded snippet it
already stores. A rule then clauses on labels_has_any exactly as every
other rule clauses on connector fields, with no new mechanism and no new
hot-path cost.
What it needs: the API service has no ingest client — IngestClient
lives in services/connectors — so this is either a small vendored client
in the API or a move of the collector into the connectors service. That is
a structural decision touching another lane's tree, which is why it is
recorded here rather than guessed at. Nothing is shipped clausing on
labels until it exists.
D6 — The enrichment-decision gate judges reachability globally
Found while landing 7.1: scripts/check_enrichment_decisions.py asks
whether a retired rule's fields are resolvable against the union of every
connector's namespace. Promoting a field called scopes on Entra consent
events therefore made ident-github-pat-classic-created-with-broad-scopes
look reachable, and the gate demanded it be un-retired — while GitHub's
audit log still does not carry a created classic PAT's scopes, so the rule
could not fire.
Worked around correctly by renaming the field to granted_scopes, which is
more precise anyway. The gate weakness is not fixed: a rule's
reachability should be judged against the fields its own log_source
emits. Left recorded rather than changed, because it is a shared gate
several lanes depend on and a scoping change belongs with its owner.
D7 — Lane A's Phase 1.4 had not landed when 7.3 was built
origin/main was fetched before starting and again before committing;
baselines.py on main still had no sub-principal mechanism. 7.3 therefore
built the shared one rather than waiting. It is additive — a dict entry per
kind, plus one name appended to PRINCIPAL_FIELDS — so if 1.4 lands first
the two combine rather than conflict. If both touch the file, combine; never
take one side.