Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
docs(audit): record where each lettered deferral now stands (#1041)
Beenu Arora2b92b039371b
diff --git a/docs/audit/DEFERRED_SUBPHASES.md b/docs/audit/DEFERRED_SUBPHASES.mdindex 80f6df345..09d68d8ef 100644--- a/docs/audit/DEFERRED_SUBPHASES.md+++ b/docs/audit/DEFERRED_SUBPHASES.md@@ -1,6 +1,6 @@# The lettered deferrals-**Last updated:** 2026-09-28 (v12.0.0)+**Last updated:** 2026-09-29 (v12.2.0)Eight sub-phases of the hardening program were deferred with a letter suffix— 3.5+, 5b, 6b, 7b+, 8b, 9b, 10b, 11b. Six of them were named in `ROADMAP.md`,@@ -41,14 +41,31 @@ decision is worse than admitting the record was lost.**From the phase line:** *"heavy-demo-stack Playwright E2E + demo-timing gatetracked as non-blocking 3.5+"*.-**Status: open.** Playwright E2E exists against the hermetic stack. What does-not exist is a run against the full demo stack, or a gate on how long the demo-takes to become usable.--**Why it is worth doing:** `packages/aisoc-sandbox` has a cold-start gate-(`aisoc-sandbox demo` in under 30 s) and the devcontainer has one. The demo-stack — the thing `make demo` starts and a first-time reader actually meets —-has neither, so it can get slower indefinitely without anyone noticing.+**Status: both halves built, in [#1040](https://github.com/beenuar/AiSOC/pull/1040).**+`apps/web/e2e/demo-stack/` asserts against the stack `pnpm aisoc:demo`+starts — kept separate from `journey`, which stubs the network, and from+`screenshots`, whose own header calls its tests recorders rather than+assertions. `docs/perf/demo-timing.json` records the runs the bound came+from and `scripts/check_demo_timing.py` enforces it, refusing a declaration+without its provenance and a ceiling below the worst run it cites.++**Two defects the gate found while being written.** `isPortFree` binds+127.0.0.1 and claimed to be "the same test Docker is going to run"; Docker's+allocator refuses a port another *container* holds, so the demo took 5432+anyway and postgres started with no network attached — postgres healthy, API+healthy, seed exited 1, console empty. And the harness polled `/v1/cases`+where the API serves `/api/v1/cases`, which is why the showcase lookup cost+exactly 60 s on every run. Fixing the first took the stack from 3m06s to+1m39s.++**What stays open, recorded beside the number rather than implicitly.**+`seed_demo.py` still fails on a fresh demo stack, so the console has no+cases in it: the ORM maps `CaseTask` to `case_tasks` while migration 027+creates `aisoc_case_tasks` with a different column set. `create_all` papered+over it until `061_runtime_app_role.sql` took `CREATE` on schema public+away, and the permission error is swallowed as "create_all skipped (likely+already applied)". Reconciling two designs for one table is a schema change,+not a timing gate.---@@ -56,14 +73,26 @@ has neither, so it can get slower indefinitely without anyone noticing.**From the phase line:** *"backfill/replay-from-offset tracked as 5b"*.-**Status: open.** Phase 5 shipped the schema registry, the dead-letter queue-and event-time watermarking. What is missing is the operator action that-follows a DLQ: having captured a poison batch, replay it from a given Kafka-offset after fixing the cause.--**Why it is worth doing:** a dead-letter queue nobody can drain is an audit-trail, not a recovery mechanism. `GET /api/v1/health/dead-letters` reports the-backlog and nothing consumes it.+**Status: closed in [#1036](https://github.com/beenuar/AiSOC/pull/1036).**+`POST /api/v1/health/dead-letters/replay` re-reads a bounded range from a+given `(topic, partition, offset)`, re-validates it and produces what now+passes.++The row could never have been the thing replayed: `aisoc_dead_letters`+stores a 2,000-character excerpt, truncated on purpose. The faithful copy is+in Kafka, and reaching it needed the partition and offset the consumer had+in hand and discarded — now carried through `_dead_letter` and persisted,+nullable, because a fabricated offset replays somebody else's message.++The property that matters is not the bound or the permission: **a message is+re-validated by the validator that refused it, and one that still fails is+refused again rather than produced.** Replaying a poison batch into the+consumer that rejected it reproduces the outage, so a preview whose+`would_pass` is zero is the answer "your fix has not landed". Dry run is the+default at three layers, and on a dry run the producer is never constructed.+Proved against Redpanda with a negative control: deleting `consumer.seek()`+makes the live test fail, which every fake-client test in the suite would+have passed.---@@ -74,24 +103,22 @@ managed-mode sizing guide and the LLM-cost dashboard so a tenant's storage$/mo shows next to its LLM $/mo"* —[`docs/decisions/0005-storage-consolidation.md`](../decisions/0005-storage-consolidation.md).-**Status: open, and it was on no list of these until now.** Phase 6 shipped-the model and its drift gate, and both are real: `scripts/storage_cost_model.py`-computes ≈$902/mo and ≈$30/raw-TB at 1 TB/day, `docs/decisions/storage-cost-model.json`-is the committed worked example, and `.github/workflows/perf.yml` fails when-the two diverge.--Neither half of the follow-up exists. `services/api/app/services/cost_dashboard.py`-and `apps/web/src/app/(admin)/costs/page.tsx` mention storage nowhere — the-dashboard reports LLM spend only — and the "Billing and cost transparency"-section of `apps/docs/docs/operations/managed-instance.md` points at that LLM-dashboard and says nothing about storage. Outside the ADR, the CHANGELOG and-the perf workflow, the model's only readers are two ClickHouse tiering files-citing it in a comment.--**Why it is worth doing:** the ADR's stated reason for computing the number-at all was sizing and the managed-mode pricing shape (ADR-0003). A cost model-whose only consumer is the gate that checks the cost model is a well-tested-constant.+**Status: closed in [#1019](https://github.com/beenuar/AiSOC/pull/1019).**+`GET /api/v1/costs/dashboard` carries a `storage` block and the console+renders it. It rests on one measurement — the uncompressed bytes a tenant's+events occupied in the lake over the window — and runs the committed model+over it.++Three properties are asserted rather than asserted-to: it is badged a+projection and no field is named `*_cost_usd`, so the naming is itself the+label; it is a sibling of the headline and never a term in it, because one+number cannot be labelled two ways; and on a CORE deployment, where the lake+does not run, every money field is null with the reason attached rather than+a confident `$0.00`. The rate card is mirrored into the API because the+script cannot be imported there, and `storage_cost_model.py --check` now+reads the mirror back — closing "the model has no consumer" by adding one+that could silently quote different prices would have been the worse+outcome.---@@ -100,14 +127,34 @@ constant.**From the phase line:** *"7b+ (posture collection, effective-permissionssnapshot loader, bi-temporal valid_from/valid_to, fusion-time ContextBundle)"*.-**Status: partially closed, and the remaining gap is precisely named.**--v8.1's audit found all five effective-permissions resolvers already shipped-and reporting `coverage: "full"`. The gap is not the resolvers: **no connector-answers `__posture_snapshot__`, so four of the five return 412.** Bi-temporal-`valid_from`/`valid_to` landed in T1.2.--So what remains of 7b is one thing: a connector-side posture snapshot.+**Status: still open, and still exactly one thing.** No connector answers+`__posture_snapshot__`, so four of the five effective-permissions resolvers+return 412. Bi-temporal `valid_from`/`valid_to` landed in T1.2 and the+resolvers themselves all ship reporting `coverage: "full"`.++**Re-derived rather than restated, because the shape of the remaining work+was not previously written down.** `__posture_snapshot__` is not a method or+an attribute — it is a sentinel `resource_id` passed to+`get_resource_config(resource_id, at_ts)`, and a connector answers it by+branching on the literal value and returning a reconciled snapshot in the+shape its resolver expects. Only five of the 84 connectors implement+`get_resource_config` at all, and of the four providers that 412+(`aws_security_hub`, `azure_entra`, `gcp_scc`, `google_workspace`), two+implement it for an unrelated id shape and two do not implement it at all.+Okta is the one that works, and it works because `posture_loader` assembles+its snapshot from several ordinary `get_resource_config` reads rather than+from the sentinel.++So closing this is four provider-specific snapshot collectors over real+vendor APIs — IAM policies and SCPs for AWS, role assignments for Azure,+IAM bindings for GCP, roles and privileges for Workspace — each returning+the shape its resolver already consumes, and each needing credentials to+verify against. That is a piece of work in its own right, not a loose end,+and it is left open rather than part-built.++`services/api/tests/test_posture_snapshot_coverage.py` already encodes the+gap as data, with a test per provider that fails the moment a connector+starts referencing the sentinel. That test is the thing to delete last.---@@ -117,24 +164,28 @@ So what remains of 7b is one thing: a connector-side posture snapshot.tracked as 8b; this seeds the registry + gate with the canonicaltriage/summary prompts"* — `services/agents/app/llm/prompt_registry.py`.-**Status: open, and the gap is countable.** Phase 8 shipped the registry, the-committed `prompts.lock.json` and `scripts/check_prompt_lock.py --check`,-which fails when a prompt's text changes without a version bump. That is the-mechanism making the `AGENTS.md` rule — a prompt change obliges an eval-re-grade — enforceable rather than aspirational.--It is enforceable over three prompts. One production module reads the-registry: `services/agents/app/hunt/agent.py` takes `hunt.system`.-`triage.system` and `summary.system` are registered, hash-pinned and gated,-and nothing under `services/agents/app/` asks for either — while ten modules-there still declare a module-level `_SYSTEM_PROMPT` of their own, the triage,-cloud, identity, insider-threat and phishing agents and the recon, forensic,-responder, report-writer and playbook-drafter among them.--**Why it is worth doing:** every prompt still inline can be edited without a-version bump, without a lock change, and therefore without the re-grade —-which is the exact hole the registry was written to close, left open for all-but one of them.+**Status: closed in [#1016](https://github.com/beenuar/AiSOC/pull/1016).**+Coverage went from 3 registered / 1 read / 21 shipped-unpinned to 22+registered / 22 read / 0 shipped-unpinned.++The gate runs in both directions now. `inline-prompt` catches a module-level+string reaching a system-message sink, detected by what the constant *is*+rather than by its name — a sweep for `_SYSTEM_PROMPT` missed+`deep_investigation._SYSTEM_PREAMBLE` and the ten-prompt+`contextual._SYSTEM_PROMPTS` dict — and follows assignment to a fixed point,+because three of the twelve reached their sink through a local first.+`unread-prompt` catches the other direction: `summary.system` was hash-pinned+for a summariser this service does not have.++**`register()` no longer strips, and the measurement is the reason.** Ten of+the migrated prompts ended in a newline, and trimming it changed the+completion for six of those ten on `qwen2.5:0.5b` at temperature 0 — against+a control where the same prompt asked twice was identical 10/10. A+normalisation the author cannot see is an undeclared prompt change performed+by the registry itself. All 21 migrated prompts hash identically to the+literal they were moved from, so no model call changed; the one deliberate+text change is `triage.system` v1 → v2, a placeholder nobody read replaced+by the production prompt.---@@ -143,35 +194,24 @@ but one of them.**From the phase line:** *"live-router wiring + durable approval-SLA timertable tracked as 9b"*.-**Status: substantially closed, and what is left is narrower than this-section used to say.**--The live-router half is done: `services/api/app/api/v1/endpoints/approvals.py`-now carries a decision through to `services/actions` and records on the row-whether it executed, and `POST /actions` consults the confidence × impact-matrix rather than blast radius alone. Durable storage arrived with-`055_action_records.sql`.--**The timer table landed too, and this section was wrong about it.** It-recorded the Slack bot's `ApprovalTimeoutScheduler` as in-process only. A-Postgres-backed store had in fact existed since Phase B3-(`services/slack-bot/app/services/timer_store.py`) — and it genuinely was not-durable, for a reason worth keeping: it created `approval_timers` itself with-`CREATE TABLE IF NOT EXISTS`, outside every migration chain, so once-`061_runtime_app_role.sql` took `CREATE` on schema public away from the-runtime role that call raised at startup and `main.py` fell back to the-in-memory store. Durable approval timers silently stopped being durable.-`062_approval_timers.sql` brought the table into the chain with a `tenant_id`-and an RLS policy; the store now refuses to create it and names the migration-instead, and `recover()` re-arms surviving timers on start.--**What remains is every approval the bot never sees.** The expiry lives in-`services/slack-bot`, a `chatops`-profile service, and is armed per ChatOps-approval. The API's own `agent_approvals` row carries an `expires_at` and its-status vocabulary includes `expired`; nothing writes that status, and no-worker under `services/api/app/workers/` sweeps the column. An approval-raised in the console and never answered still waits forever rather than-timing out to its declared safe default.+**Status: closed in [#1038](https://github.com/beenuar/AiSOC/pull/1038).**+The live-router half and the durable timer table had already landed; what+remained was every approval the ChatOps bot never sees.++`agent_approvals` has carried an `expires_at` and an `expired` status since+migration 009, and nothing wrote that status and no worker swept the column,+so a console-raised approval nobody answered waited forever — invisibly,+because the row stayed `pending` and nobody could tell it from one still+under consideration. `app/workers/approval_expiry.py` sweeps it.++Three decisions worth keeping. The safe default is `rejected` and is read+back from the two places that already declare it rather than restated, since+two halves timing out differently would be worse than either. Expiring is+not deciding: nothing is dispatched, and `/decide` still accepts an+`expired` row, so a human returning to a timed-out request can still act —+a test pins that guard. And an approval with no `expires_at` is *counted*,+not given one, because inventing a window would start expiring containments+on a schedule nobody chose.---@@ -180,26 +220,24 @@ timing out to its declared safe default.**From the phase line:** *"Live-vendor sandbox smoke + rate-limit/checkpointdurability tracked as 10b"*.-**Status: one half is credential-blocked; the other has a mechanism and one-adopter.**--Live-vendor smoke needs sandbox accounts with real vendors — an account-action, the same class of blocker as the npm publish and the funded eval key,-not an engineering task. It stays open and says so.--**Checkpoint durability is no longer absent, which this section used to say-it was.** `services/connectors/app/db/connector_repo.py::record_checkpoint`-persists a connector's ingest checkpoint into `connector_config.checkpoint`,-and `services/connectors/app/scheduler.py` seeds it back into the connector-before each poll and writes the advanced value only after the batch was-accepted, so a failed ingest never moves it forward.--What is missing is adoption. `splunk` is the only one of the 84 connectors-implementing `set_checkpoint`/`get_checkpoint`; the scheduler seeds through a-`getattr(connector, "set_checkpoint", None)` that is simply absent on the-other 83. For those, poll state still does not survive a restart — the-connector re-reads its overlap window, which is safe, and means a long outage-silently loses events older than that window.+**Status: one half closed in [#1039](https://github.com/beenuar/AiSOC/pull/1039);+the other is credential-blocked and stays open.**++**Checkpoint adoption: 1 connector of 84 → 5.** The durable machinery+already worked; what was missing was adoption, and the reason it stayed+missing is the point. The scheduler reached the connector through+`getattr(connector, "set_checkpoint", None)`, absent on 83 of 84 — a+duck-typed optional protocol has no failing state, only a quiet one, and+nothing could report which connectors resumed. The contract is on+`BaseConnector` now, adopting is two class attributes, `checkpoints()`+answers the question, and a test asserts the adopter set is exactly the+recorded one so losing one fails loudly. `splunk` moved onto the shared+machinery with its eight existing tests unchanged. The remaining 79 are+recorded as not-checkpointing rather than described as done.++**Live-vendor sandbox smoke stays open.** It needs sandbox accounts with+real vendors — an account action, the same class of blocker as the npm+publish and the funded eval key, not an engineering task.---