Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

docs(4.5): two claim rows and the 4.5 progress entry

Beenu Arora16ffe001364d
diff --git a/DEPTH_PROGRESS.md b/DEPTH_PROGRESS.md
index e1640b09c..84aba16c1 100644
--- a/DEPTH_PROGRESS.md
+++ b/DEPTH_PROGRESS.md
@@ -590,7 +590,62 @@ invented fields on the first run of the gate.
through the **published images**, not a build of this branch, so it is
not a measurement of the rule index — that is measured separately and
in process.
-- [ ] **4.5** Deployment completeness (parity 6.8)
+- [~] **4.5** Deployment completeness (parity 6.8). **Three of four
+ bullets closed; the managed-cluster run is maintainer-blocked.**
+
+ **The chart installs CORE.** `infra/helm/aisoc/values.yaml` declared
+ **seven** application services against CORE's sixteen: no response
+ execution, no connector polling, no threat-intel store, no LLM
+ gateway, no local model, no vector store. `helm install` produced a
+ deployment that could not act, could not collect and could not triage
+ — and reported success, because nothing compared the two files.
+ `actions`, `connectors` and `threatintel` are now entries in
+ `.Values.services`; `litellm`, `ollama` and `qdrant` are deliberately
+ not, because that map defaults every image tag to
+ `.Chart.AppVersion` and tagging a third-party image with AiSOC's
+ version renders perfectly and pulls a tag that does not exist. Two of
+ them need storage, so they are StatefulSets with volume claims.
+
+ **Restore covers backup.** The two scripts already agreed on all six
+ components; nothing held them that way, and a component added to one
+ and not the other is discovered during an incident. Now compared in
+ both directions.
+
+ **Leader election.** The scheduler adds one job per connector
+ instance in process, so two replicas each polled every vendor on every
+ cycle — and on a rate-limited API the second replica's poll is the one
+ that gets throttled, which looks exactly like expired credentials. A
+ Postgres lease (migration `112`), not a Kubernetes Lease: the
+ scheduler also runs under Compose and systemd, and an election that
+ only works on Kubernetes is off wherever most first runs happen. The
+ mechanism is `UPDATE ... WHERE expires_at < now()`, proved by two
+ **explicitly interleaved** transactions rather than a race — a race
+ between coroutines on one loop passes under a SELECT-then-UPDATE too,
+ which this phase measured on its own lake-job claim. A follower
+ *removes* its poll jobs rather than declining to add more.
+
+ Gated by `scripts/check_deployment_completeness.py` (self-test: nine
+ injected violations, one per rule, plus a refusal on both an absent
+ and an empty tree) wired into `ci.yml :: python-lint`, and by
+ `scheduler-lease-live.yml`. 13 live lease tests; matrix 311 rows, 308
+ GATED.
+
+ **`[~]`, for three things that could not be verified from here**,
+ named rather than implied:
+
+ 1. **No managed-Kubernetes run.** The plan already says this one waits
+ on the maintainer providing a cluster. No managed cluster is
+ reachable from this machine, so `docs/perf/` gains no row.
+ 2. **`helm install` has not been run on a cluster.** `helm lint` is
+ clean and `helm template` renders 14 workload components, which is
+ static evidence that the chart is well-formed — not evidence that
+ it comes up. The kind install the plan's "Done when" names is the
+ open half of this item.
+ 3. **`scripts/check_published_images.py` could not reach GHCR**
+ ("ghcr.io unreachable: The read operation timed out") and correctly
+ refused a vacuous pass rather than reporting the tree clean. So the
+ three new third-party image references and the three new
+ first-party ones are **not** confirmed to resolve in the registry.
## Phase 5: Response that finishes
@@ -647,6 +702,7 @@ invented fields on the first run of the gate.
| 2026-10-07 | 0.3 | `make up` and `make smoke` (10/10) pass. Injection suite and load-harness baselines committed. `make up-full` deferred (D2) and hosted model rows blocked (D3). Fixing D1 was a precondition for the load-harness baseline. |
| 2026-10-09 | 3.3 | Identity privilege (18 rules) and a per-tenant first-seen store (2) built; 5 rules were never unreachable (gate operator drift, D9); 18 retired with a reason (D10). `MAX_UNREACHABLE` 45 → 2, the two remaining both needing the 3.4 baseline. Two matcher operators three shipped rules already used were implemented. |
| 2026-10-09 | 3.2 | Ordered and unordered sequences in the windowed engine; all four translatable Sigma correlation types compile, seven refusal reasons recorded. No upstream correlation rule exists in this tree to import (D8), so the corpus is first-party and labelled as such. |
+| 2026-10-11 | 4.5 | Three of four bullets closed. Reproduced: `check_deployment_completeness.py` on the pre-fix tree reported six CORE services the chart renders no workload for. Added them (three in `.Values.services`, three as their own templates because a third-party image must not inherit `.Chart.AppVersion` and two need volumes), compared restore against backup in both directions, and built Postgres leader election for the connector scheduler (migration `112`). Negative control: removing the lease's expiry predicate makes both interleaved transactions lead and three live cases go red. **Not verified here:** no managed-cluster run, no `helm install` on a cluster, and `check_published_images.py` could not reach GHCR and refused to pass vacuously. |
| 2026-10-11 | 4.4 | Every parity 6.9 bullet closed; the cloud-hardware run is not possible from here and is recorded as unmeasured rather than implied. Rule index on required fields (lossless, proven by replaying every fixture both ways; 68.3% fewer candidates, 2.6x on evaluation), entity partition key held equal to the correlation key, per-entity concurrent triage, lag-based scale hint derived from the published run. The realistic mix found that eight of twelve connector profiles are category 2 and are promoted unconditionally, so on those connectors no event can be routine — the first run measured a routine alert rate of 1.00. ADR-0011; matrix 309 rows, 306 GATED. |
| 2026-10-11 | 4.3 | Third bullet closed. Reproduced: `check_retention_window.py` with the new rules reported no cold-query estimator, no job surface and no job table on the pre-fix tree. Built `lake_cold_query.py` (EXPLAIN ESTIMATE + system.parts + system.processes, no invented throughput or price), `lake_query_runner.py` and migration `111`. Negative control: removing the `status = 'queued'` predicate makes both interleaved transactions claim one job and the live suite goes red. The first version of that test — five gathered claims — passed under the broken implementation and was replaced. 26 pure + 12 live tests; matrix 306 rows, 303 GATED. |
| 2026-10-11 | 4.2 | Reproduced: `check_standard_inputs.py` against the pre-fix tree reported no OTLP package, no Kafka input and no TAXII server — the only TAXII-shaped route being a demo-gated sample list. Built all three, after moving the consumer-health classifier out of `graph_ws` into `streamhealth` so the new Kafka consumer shares the table rather than copying it. Negative controls recorded in the PR: a timestamp-only TAXII cursor loses an indicator in the single-row pagination walk (measured — 3 of 4), and nineteen gate injections each go red. Writing it found four defects, including an OTLP template nobody could mint a token for and a gate rule that read an error message instead of the behaviour. Ingest Go suite green; api projection suite 37; live TAXII suite 15. |
diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.md
index 6f8dcb532..de95098a6 100644
--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md
+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md
@@ -21,6 +21,8 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
| Fusion does not run every detection rule against every event | `docs/decisions/0011-how-fusion-scales.md`; depth plan 4.4 / parity 6.9 | `ci.yml :: Python — Tests` (fusion, `test_rule_index_is_lossless.py`, 12 cases: **every committed detection fixture is replayed through the indexed path and the full corpus and the hit lists must be identical and in the same order**, plus 200 random non-matching events and the empty event, because fixtures are synthesised from the rules they test and only exercise the matching case. The classification is asserted to be executed rather than enumerated — for every field the index calls required, the real matcher is asserted to refuse an empty event) | core | GATED | Indexed on **required fields, not log source**, which is what the plan asks for: connector product names do not line up with spec products, a product-based pre-filter was tried once and reverted for silently dropping real matches, and a mapping table leaves every new connector one missing entry away from a detection that stops firing. Measured: 2,511 rules, 1,751 indexed (69.74%), 797 candidates for a routine cloud event (68.3% fewer), 2.760 → 1.066 ms/event over 2,000 iterations. The indexed fraction is pinned just under the measurement so it can only ratchet up. Writing the test found that `neq` **requires** its field in this matcher despite reading like a negation, which a hand-written operator list would have got wrong in the direction that drops detections |
| One entity's events are ordered, and triage runs concurrently across entities | `docs/decisions/0011-how-fusion-scales.md`; depth plan 4.4 / parity 6.9 | `ci.yml` ingest Go job (`internal/publisher`, 8 cases: two events about one host share a partition key, two hosts do not, the same hostname in two tenants does not, the precedence matches `RawAlert.correlation_key()` field for field, and an event naming nobody falls back to its own id rather than to the tenant — which would be a hot partition) + `ci.yml :: Python — Tests` (agents, `test_entity_lanes.py`, 18 cases: one entity's alerts complete in arrival order **with the first deliberately the slowest**, two entities overlap, peak concurrency is the bound, a failing alert does not take its lane down, and 50 entities leave zero lanes retained) | core | GATED | The partition key was `tenant:event_id`, unique per event, so nothing about one entity was ordered relative to anything — while fusion correlates on `{tenant}:{entity}:{tactic}`. A partition key disagreeing with the correlation key would route an entity's events to a partition and correlate them under another name, correct-looking from either end, so a test holds the two orders equal. Triage commits offsets only after the in-flight window drains, with a time bound as well as a count: without it a topic quieter than the concurrency bound leaves finished work uncommitted and every restart replays it |
| The load harness measures a pipeline where most events do not alert, and the autoscaling hint is derived from that run | `apps/docs/docs/operations/performance.md`; depth plan 4.4 / parity 6.9 | `ci.yml :: Python — Tests` (`tests/test_load_harness_realistic_mix.py`, 8 cases: the Go and Python benign markers are equal and the SQL cannot confuse them even though one is a prefix of the other, the delivery ratio is taken against the alerting population so a lossless mix run reports 1.0 rather than 0.01, an all-alerting run reports the benign rate as **not measured** rather than 0.0, a result file written before the mix existed still grades, and a ceiling on an unmeasured rate fails rather than passes) + fusion `test_scale_hint.py` (13 cases, including one that **reads the committed result file** and fails if `PUBLISHED_REPLICA_EPS` drifts from the run it names) + `check_perf_results.py` (every committed result carries its hardware, its date and the not-an-SLO label) | core | GATED | Published run: 6,000 events saturating at 1-in-100, **0 of 5,940 routine events raised an alert**, 0 missing, 0 duplicates, p95 event-to-alert 1,195 ms, ingest 122.6 events/s. The first attempt measured a routine alert rate of **1.00** — the load path's undeclared connector type falls to a generic profile filed as class 2001, and the promoter promotes category 2 unconditionally, so on eight of the twelve declared profiles no event can be routine. **No cloud-hardware run exists**: no instance has been provisioned for this project, so the plan's 5,000 events/s target is an aim rather than a result and the page says so with the command that would close it. The published run also predates the rule index — it ran against the published images — which the page states |
+| `helm install` brings up every CORE service, and a restore covers everything the backup writes | `infra/helm/aisoc/values.yaml`; depth plan 4.5 / parity 6.8 | `ci.yml :: python-lint` (`check_deployment_completeness.py --self-test` then the gate: every profile-less service in `docker-compose.yml` must have a chart workload or a **named reason** not to, in both directions, and every `backup_*` function must have a `restore_*` counterpart and the reverse. Nine injected violations, one per rule, including a CORE service the chart drops and a component backed up but not restorable) | core | GATED | The chart declared **seven** application services against CORE's sixteen: no response execution, no connector polling, no threat-intel store, no LLM gateway, no local model, no vector store — and `helm install` reported success, because nothing compared the two files. `litellm`, `ollama` and `qdrant` are deliberately outside `.Values.services`: that map defaults every image tag to `.Chart.AppVersion`, and tagging a third-party image with AiSOC's version renders perfectly and pulls a tag that does not exist. **`helm install` has not been run on a cluster from CI** — the gate is static, `helm lint` and `helm template` are the dynamic checks that exist, and the kind install remains the open half of this row's claim |
+| Two connector-scheduler replicas cannot both poll every vendor | `services/connectors/app/leader_lease.py`; depth plan 4.5 / parity 6.8 | `scheduler-lease-live.yml :: Two replicas cannot both poll every vendor` (real Postgres, every migration applied, 13 cases: a free lease taken once, a held lease not stealable, an expired one taken over, a process paused past its own expiry unable to resume and renew, the term advancing on takeover and **not** on renewal, and a release letting the next replica in without waiting out the lease. The decisive case interleaves two transactions explicitly rather than racing them, because a race between coroutines on one loop passes under a SELECT-then-UPDATE as well — measured on this repository's own lake-job claim. The job asserts the lease table exists, greps the run for a skip, and carries a negative control removing the expiry predicate, after which three cases go red) | core | GATED | A Postgres lease rather than a Kubernetes Lease, because the scheduler also runs under Compose, under `make up` and in a systemd unit, and an election that only works on Kubernetes is off wherever most first runs happen. Off by default and enabled in the Helm values, which is where a second replica is possible. A follower **removes** its poll jobs rather than declining to add more: a replica that loses leadership while holding jobs keeps polling, which is the same double-poll with the replica now reporting itself as a follower |
| Investigation Ledger stores every step | README L61, L169 | `ci.yml :: Python — Lint & Type-check` (`check_ledger_replay_contract.py` — response model ↔ console type, client requests ↔ router decorators, and `seq` ordering/uniqueness, each in both directions) + `ci.yml :: Python — Tests` (`test_investigation_replay.py`, 37 cases over `/replay`, `/events`, `/explain`; removing `ORDER BY seq` fails five of them) | core | GATED | — |
| Public eval harness gates every PR | README L62, L77 | `ci.yml :: p1-eval` | core | GATED (but suites are self-consistency; see reality report) | Phase 4 |
| Alert-reduction is a real measurement | README / benchmark page | `ci.yml :: python-test` (fusion) runs `test_alert_reduction_real.py`, which groups a 1 000-alert stream with `RawAlert.correlation_key()` — the method `Correlator` actually calls — and bounds the ratio on both sides, because a floor alone is satisfied by a key that collapses everything into one incident | core | GATED | the legacy suite's 75.3% stays published for continuity and is labelled as not describing this product; the measurement is still a synthetic workload, as the benchmark page states |
@@ -320,8 +322,8 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
## Summary
-- Total: 309
-- GATED: 306
+- Total: 311
+- GATED: 308
- PARTIAL: 3
- NO GATE: 0