Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
chore(release): v8.1.1 (#728)
Beenu Arora8d3d0560657f
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 7856cc126..37a2b7a0b 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,124 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]+## [8.1.1] — 2026-09-23++**The v8.1.0 notes described a platform its own quick start never started.**+This release contains no new capability. It exists because an adoption audit+of the repository — installability, architecture comprehension, data+provenance, pipeline connectivity — found that the single most-followed path+into AiSOC did not run AiSOC, and that finding accounts for most of the+recurring feedback on its own.++`./install.sh` handed off to a nine-service compose file with **no ingest+service, no fusion service, and `AISOC_DISABLE_KAFKA: true`**. Everything+visible in the resulting console came from `seed_demo.py` writing fifteen+fabricated incidents straight into Postgres. A reader followed the README, saw+a populated console, and concluded the platform worked — having never run the+platform. The installer printed *"AiSOC is up and running"* because the+compose command exited 0.++The good news the audit also produced: **the core pipeline does work.** It had+simply never been demonstrated. `make smoke` now pushes one real event through+ingest, Kafka, fusion and detection and reads the alert back from the API,+reaching past nothing, and CI fails if any stage does.++### Fixed++- **The documented quick start never ran the product** ([#726](https://github.com/beenuar/AiSOC/pull/726)).+ `install.sh` now runs `make up` — the same CORE stack the README documents+ and CI tests — and then runs the golden pipeline, so the installer's success+ banner reports a verified event traversal rather than a process exit code.+ `infra/compose/docker-compose.demo.yml` is kept for screenshots and UI work+ and now says at the top of the file that it cannot answer whether AiSOC+ works.+- **`services/ingest` answered `/health` unconditionally** ([#727](https://github.com/beenuar/AiSOC/pull/727)),+ so *"ingest is healthy"* and *"every event is being dropped"* could both be+ true at once — precisely the state a broker outage produces. `/readyz` now+ dials Kafka rather than reading a cached connection flag, because kafka-go+ reconnects lazily and a cached flag is stale in exactly the situation this+ is meant to catch. Verified live: 200 with Kafka up, 503 with a reason and a+ log hint with Kafka stopped.+- **Three of the eight publishable packages could not be built** ([#725](https://github.com/beenuar/AiSOC/pull/725)).+ The release pipeline builds and packs every package on each tag precisely so+ it cannot rot while the upload is credential-gated — and on the v8.1.0 tag+ it earned that design back. `aisoc` and `@aisoc/mcp` failed with+ `Could not resolve "@aisoc/report-card"`: that workspace dependency's `main`+ points at a build artifact and `pnpm --filter <pkg> build` never builds+ dependencies, so the `...` suffix was missing. `aisoc-cli` could not produce+ a wheel at all — `pyproject.toml` declared `packages = ["src/aisoc_cli"]`+ *and* a `force-include` mapping the same templates directory to the same+ wheel path, so hatchling refused on the duplicate. Neither was visible+ anywhere but a tag. `test_packaging.py` now builds the wheel and looks+ inside it, because a test that read the config would have declared both+ fine.+- **Alerts were labelled `"crowdstrike crowdstrike"`** — vendor and product+ joined without deduplication, in two copies of the same helper, so fixing+ one left the other. There is one shared implementation now+ (`fusion/app/services/provenance.py::product_label`). In the same pass, an+ alert's `description` was `str(raw_data)`, so the console showed+ `{"command_line": "powershell.exe -nop ...` where a sentence belongs,+ discarding the vendor's own description.+- **Fusion defaulted its enrichment URL to `localhost:8082` — itself** — and+ swallowed the resulting failure at `DEBUG`; and fusion declared no+ `depends_on` for Postgres, so it raced the database on cold boot.+- **Five surfaces rendered fabricated data outside demo mode**: the MSSP+ overview and its managed-tenant and cross-tenant-incident lists, the+ Copilot's reply on API error, the air-gap status endpoint reporting+ unchecked conditions as satisfied, a hard-coded analyst identity in+ Settings, and the investigation timeline. All are gated behind demo mode and+ return honest empties otherwise. `seed_demo.py` refuses to run outside+ development unless `AISOC_ALLOW_SEED=1`.++### Added++- **`tests/e2e/golden_pipeline/`** — one deterministic end-to-end test that+ pushes raw telemetry into ingest and asserts each boundary independently, so+ a break names the stage rather than the suite. Wired into CI as+ `golden-pipeline.yml`, which also verifies the gate fails when the pipeline+ is broken.+- **`scripts/doctor.sh` / `make doctor`** — diagnoses Docker, Compose, disk,+ ports, configuration, Postgres, Redis, Kafka and every service, and prints+ the command to run next. It checks disk before Kafka and probes the broker+ directly, because the Docker VM filling up produced a Kafka that reported+ `healthy` while refusing every request.+- **`/livez` and `/readyz` on `services/ingest`**, matching `services/api` and+ `services/fusion`.+- **`scripts/project_stats.py --check`** — derives the connector count,+ executable detection count, compose service count and claim-to-gate totals+ from the tree and fails CI when the README disagrees.+- **`docs/audit/REPOSITORY_REALITY.md`** — every major component classified+ WORKING / PARTIAL / BROKEN / DEMO-ONLY / EXPERIMENTAL / DEAD CODE by tracing+ the implementation, not the filename. It records that OpenSearch is started+ by the `full` profile and read by nothing, rather than drawing it into a+ diagram as though it were part of the design.+- **`docs/architecture/README.md`** — rewritten around *what happens when+ AiSOC receives one security event*, with five diagrams whose every box links+ to the directory that implements it.+- **`docs/testing/CLEAN_INSTALL.md`** — the clean-machine walkthrough, from a+ real run.+- Migration `054_alert_provenance.sql` adds `is_synthetic`,+ `synthetic_source` and `synthetic_scenario` to `alerts`.++### Changed++- **CORE is the default deployment profile**: ten services, roughly 6 GB, and+ the smallest deployment that takes a real event and produces a real alert —+ not a cut-down toy. ClickHouse, Neo4j, Qdrant, OpenSearch, enrichment and+ connectors moved to `full`. Both heavy dependencies were already+ environment-gated, so nothing was weakened to do this. `make up-full` sets+ `AISOC_LAKE_WRITER_ENABLED` and `AISOC_GRAPH_ENABLED`, which the profile+ needs and did not previously set.+- **README rewritten for adoption** rather than release history: what AiSOC+ does, quick start, how to tell demo data from real data, connecting real+ sources, the data flow, project maturity per capability, and what AiSOC is+ not.+- **One command set.** `make install / up / up-full / down / restart / status+ / doctor / smoke / demo / logs / test / test-unit / test-integration /+ test-e2e / stats / clean`, consolidating the previous scattered scripts.+- The integration workflow boots the `full` profile with the lake writer and+ graph enabled, so it exercises the architecture it claims to.+## [8.1.0] — 2026-09-23**Wave-2 features, and the gaps behind them.** Every item in the wave-2@@ -184,19 +302,6 @@ Claim-to-gate matrix: **108 rows — 99 GATED, 9 PARTIAL, 0 NO GATE**.### Fixed-- **Three of the eight publishable packages could not be built.** The release- pipeline builds and packs every package on each tag precisely so it cannot- rot while the upload is credential-gated — and on this tag it earned that- design back. `aisoc` and `@aisoc/mcp` failed with- `Could not resolve "@aisoc/report-card"`: that workspace dependency's `main`- points at a build artifact and `pnpm --filter <pkg> build` never builds- dependencies, so the `...` suffix was missing. `aisoc-cli` could not produce- a wheel at all — `pyproject.toml` declared `packages = ["src/aisoc_cli"]`- *and* a `force-include` mapping the same templates directory to the same- wheel path, so hatchling refused on the duplicate. Both would have blocked- the first real publish, and neither was visible anywhere but a tag.- `test_packaging.py` now builds the wheel and looks inside it, because a test- that read the config would have declared both fine.- **The sandbox determinism test compared a timer.** Its `VOLATILE` list namedfour fields the CLI does not emit — the real one is `elapsed_ms` — so thecomparison included a wall-clock value and failed whenever two runs@@ -4066,7 +4171,10 @@ demo profile. Details below.- Helm chart for Kubernetes deployment (`infra/helm/aisoc/`)- MIT License-[Unreleased]: https://github.com/beenuar/AiSOC/compare/v5.2.0...HEAD+[Unreleased]: https://github.com/beenuar/AiSOC/compare/v8.1.1...HEAD+[8.1.1]: https://github.com/beenuar/AiSOC/compare/v8.1.0...v8.1.1+[8.1.0]: https://github.com/beenuar/AiSOC/compare/v8.0.0...v8.1.0+[8.0.0]: https://github.com/beenuar/AiSOC/compare/v7.7.0...v8.0.0[5.2.0]: https://github.com/beenuar/AiSOC/compare/v5.1.0...v5.2.0[5.1.0]: https://github.com/beenuar/AiSOC/compare/v5.0.0...v5.1.0[5.0.0]: https://github.com/beenuar/AiSOC/compare/v4.1.0...v5.0.0diff --git a/README.md b/README.mdindex 9c5bf8dcb..736363b62 100644--- a/README.md+++ b/README.md@@ -7,7 +7,7 @@**An open-source, self-hostable AI Security Operations Center.** It ingests your security telemetry, detects and correlates threats, investigates them with AI agents whose reasoning is fully auditable, and proposes responses a human approves.[](https://opensource.org/licenses/MIT)-[](CHANGELOG.md)+[](CHANGELOG.md)[](https://github.com/beenuar/AiSOC/actions/workflows/ci.yml)[](https://github.com/beenuar/AiSOC/actions/workflows/codeql.yml)[](https://securityscorecards.dev/viewer/?uri=github.com/beenuar/AiSOC)diff --git a/RELEASES.md b/RELEASES.mdindex 5497cab1f..cce03760b 100644--- a/RELEASES.md+++ b/RELEASES.md@@ -2,13 +2,29 @@This file mirrors what used to live in the "What's new" section of [`README.md`](README.md). The complete, machine-readable inventory (with file paths, env-var diffs, and per-release test counts) lives in [`CHANGELOG.md`](CHANGELOG.md).-> **TL;DR for first-time visitors:** AiSOC is on `v8.1.0`, released 2026-09-23 — wave-2 features plus the gaps behind them. Every wave-2 backlog item was audited against the tree before any code was written, and the backlog was wrong in both directions: two items were already built, four had the capability present with the path that feeds it broken. One inverted what the feature appeared to do — a Slack or Teams approval authorized nobody. Every product claim is backed by a failing CI test (claim-to-gate matrix: 99 GATED / 9 PARTIAL / 0 NO GATE). The latest GitHub release with notes and downloads: <https://github.com/beenuar/AiSOC/releases/latest>.+> **TL;DR for first-time visitors:** AiSOC is on `v8.1.1`, released 2026-09-23 — an adoption-audit release with no new capability. The audit found that the single most-followed path into AiSOC did not run AiSOC: `./install.sh` started a compose file with no ingest service, no fusion service and Kafka disabled, and everything in the resulting console had been written straight into Postgres by a seed script. The core pipeline does work; it had never been demonstrated, and `make smoke` now demonstrates it in CI. Every product claim is backed by a failing CI test (claim-to-gate matrix: 99 GATED / 9 PARTIAL / 0 NO GATE). The latest GitHub release with notes and downloads: <https://github.com/beenuar/AiSOC/releases/latest>.---## What's new-`VERSION` is `8.1.0`. The **v8.1.0** release (2026-09-23) delivers the wave-2 backlog and the defects auditing it surfaced.+`VERSION` is `8.1.1`. The **v8.1.1** release (2026-09-23) adds no capability. It exists because an audit of how the repository is adopted — installability, architecture comprehension, data provenance, pipeline connectivity — found the quick start did not start the product.++**v8.1.1 highlights (September 23, 2026)**+- **The documented quick start never ran the product.** `./install.sh` handed off to a nine-service compose file with no ingest service, no fusion service, and `AISOC_DISABLE_KAFKA: true`. Everything visible in the console came from `seed_demo.py` writing fifteen fabricated incidents straight into Postgres, and the installer printed "AiSOC is up and running" because the compose command exited 0. It now runs `make up` — the same CORE stack the README documents and CI tests — then proves the pipeline before claiming success.+- **`make smoke` is the claim.** One real event enters ingest, travels Kafka and fusion, matches a detection, and is read back from the API, with each of the eight stages reporting independently so a break names the boundary. It reaches past nothing: no stubbed Kafka, no inserted alert. CI fails if any stage does, and separately verifies the gate fails when the pipeline is broken.+- **`services/ingest` answered `/health` unconditionally**, so "ingest is healthy" and "every event is being dropped" could both be true at once — exactly the state a broker outage produces. `/readyz` now dials Kafka; verified live at 200 up and 503 down, with a reason and the log command to run.+- **CORE is the default: ten services, roughly 6 GB**, and it is the smallest deployment that turns a real event into a real alert rather than a cut-down toy. ClickHouse, Neo4j, Qdrant, OpenSearch, enrichment and connectors moved to `full`. OpenSearch is started by `full` and read by nothing, and is recorded that way instead of appearing in a diagram.+- **Five surfaces rendered fabricated data outside demo mode** — the MSSP overview, the Copilot's reply on API error, the air-gap status endpoint, an analyst identity in Settings, and the investigation timeline. All are gated now and return honest empties otherwise.+- **Three of the eight publishable packages could not be built.** The v8.1.0 tag surfaced it, which is the credential-gated build design working: an unresolvable workspace dependency for `aisoc` and `@aisoc/mcp`, and a duplicate-path wheel failure for `aisoc-cli`. Both would have blocked the first real publish.++`make doctor`, `docs/audit/REPOSITORY_REALITY.md`, a data-flow rewrite of `docs/architecture/README.md`, and `docs/testing/CLEAN_INSTALL.md` came out of the same pass. The full inventory lives under `[8.1.1]` in [`CHANGELOG.md`](CHANGELOG.md).++---++## v8.1.0 (2026-09-23)++`VERSION` was `8.1.0`. The **v8.1.0** release (2026-09-23) delivers the wave-2 backlog and the defects auditing it surfaced.**v8.1.0 highlights (September 23, 2026)**- **A ChatOps approval authorized nobody.** The Slack and Teams bots verified who clicked — Slack signs every interaction payload, Teams payloads carry an HMAC — recorded that person in an audit event, and then called the actions service with no approver. The permission-tier check and separation of duties were both skipped. A bot cannot supply permissions (it knows a Slack user id and has no idea what that person may do in AiSOC), so it now asserts identity only and the actions service maps it through operator configuration. **If you use ChatOps approvals you must populate `AISOC_CHATOPS_APPROVERS` or they will be refused** — that is the intended failure.diff --git a/ROADMAP.md b/ROADMAP.mdindex 69e5b6054..4946e78db 100644--- a/ROADMAP.md+++ b/ROADMAP.md@@ -625,6 +625,32 @@ against **v8.2**, and it becomes a re-tag the moment the credentials exist.---+## v8.1.1 — Shipped (2026-09-23)++An adoption audit, and no new capability. The recurring feedback on this+project — hard to install, fabricated data, architecture nobody could follow,+documentation describing things a reader could not reproduce — turned out to+share one root cause: **the documented quick start did not run the product.**+`./install.sh` started a compose file with no ingest service, no fusion+service and `AISOC_DISABLE_KAFKA: true`, and the populated console a reader+saw was a seed script writing rows into Postgres.++The pipeline itself works. It had simply never been demonstrated, and now+`make smoke` demonstrates it on every CI run: one real event through ingest,+Kafka, fusion and detection, read back from the API, eight stages each+reporting independently.++Also in this release: `/readyz` on `services/ingest` that dials Kafka rather+than answering unconditionally, `make doctor`, CORE as the default ten-service+profile, five fabricated-data surfaces gated behind demo mode, the three+packages that could not be built at the v8.1.0 tag, and three new documents —+`docs/audit/REPOSITORY_REALITY.md`, a data-flow rewrite of+`docs/architecture/README.md`, and `docs/testing/CLEAN_INSTALL.md`.++Full inventory under `[8.1.1]` in [`CHANGELOG.md`](CHANGELOG.md).++---+## Ideas Backlog (unscheduled)- "Explain this alert" button using LLM with enrichment contextdiff --git a/VERSION b/VERSIONindex 8104cabd3..0e7915245 100644--- a/VERSION+++ b/VERSION@@ -1 +1 @@-8.1.0+8.1.1diff --git a/apps/docs/docs/integrations/github-action.md b/apps/docs/docs/integrations/github-action.mdindex dceb67cec..8b8ce8a49 100644--- a/apps/docs/docs/integrations/github-action.md+++ b/apps/docs/docs/integrations/github-action.md@@ -16,7 +16,7 @@ as a PR comment or job summary. No LLM, no data leaves your CI runner.today points at the action's directory inside the monorepo:```yaml-- uses: beenuar/AiSOC/packages/[email protected]+- uses: beenuar/AiSOC/packages/[email protected]```Pin a tag rather than a branch, so a workflow cannot change underneath you.@@ -41,7 +41,7 @@ jobs:triage:runs-on: ubuntu-lateststeps:- - uses: beenuar/AiSOC/packages/[email protected]+ - uses: beenuar/AiSOC/packages/[email protected]with:mode: pr-commentmin-severity: low@@ -70,7 +70,7 @@ jobs:digest:runs-on: ubuntu-lateststeps:- - uses: beenuar/AiSOC/packages/[email protected]+ - uses: beenuar/AiSOC/packages/[email protected]with:mode: digest```diff --git a/apps/web/package.json b/apps/web/package.jsonindex e05c1fb81..017c6a4a5 100644--- a/apps/web/package.json+++ b/apps/web/package.json@@ -1,6 +1,6 @@{"name": "@aisoc/web",- "version": "8.1.0",+ "version": "8.1.1","private": true,"description": "AiSOC SOC Console - Next.js 14 Frontend","license": "MIT",diff --git a/docs/operations/publishing.md b/docs/operations/publishing.mdindex dfceceaa6..1f2862b93 100644--- a/docs/operations/publishing.md+++ b/docs/operations/publishing.md@@ -91,7 +91,7 @@ There is no API token to store or rotate in this path.reference that resolves is the action's directory inside this monorepo:```yaml-- uses: beenuar/AiSOC/packages/[email protected]+- uses: beenuar/AiSOC/packages/[email protected]```That form needs no setup and is what the README anddiff --git a/docs/roadmap/v8-progress.md b/docs/roadmap/v8-progress.mdindex 325fbf4a4..a02ab9616 100644--- a/docs/roadmap/v8-progress.md+++ b/docs/roadmap/v8-progress.md@@ -1,7 +1,7 @@# AiSOC v8 — progress tracker**Last updated:** 2026-09-23-**Current release:** `v8.1.0` (2026-09-23) · **Next:** `v8.2` (packaging)+**Current release:** `v8.1.1` (2026-09-23) · **Next:** `v8.2` (packaging)This tracker is the at-a-glance view of what has landed across the v8 line andwhat is still open. It is deliberately short and dated. When it disagrees with@@ -92,6 +92,37 @@ optional one.---+## v8.1.1 — shipped 2026-09-23++An adoption audit with no new capability, run against the whole repository:+installability, architecture comprehension, data provenance, pipeline+connectivity, and whether documented commands work. The headline finding is+that `./install.sh` started a compose file with **no ingest service, no fusion+service and Kafka disabled** — so the most-followed path into the project did+not run the project, and the console a reader saw had been populated by a seed+script.++| Item | Status |+|------|--------|+| `install.sh` runs `make up` (CORE) and then the golden pipeline | done |+| `tests/e2e/golden_pipeline/` + `golden-pipeline.yml`, verified to fail when broken | done |+| `scripts/doctor.sh` / `make doctor` | done |+| `/livez` + `/readyz` on `services/ingest`, dialling Kafka | done |+| CORE as the default profile; ClickHouse / Neo4j / Qdrant / OpenSearch / enrichment / connectors to `full` | done |+| Five fabricated-data surfaces gated behind demo mode; `seed_demo.py` refuses outside development | done |+| `054_alert_provenance.sql` — `is_synthetic` on `alerts` | done |+| `scripts/project_stats.py --check` — README figures derived from the tree | done |+| `docs/audit/REPOSITORY_REALITY.md` | done |+| `docs/architecture/README.md` rewritten around one event's journey | done |+| `docs/testing/CLEAN_INSTALL.md` | done |+| README rewritten for adoption | done |+| Consolidated `Makefile` | done |+| The three packages that could not be built at the v8.1.0 tag | done |+| CrowdStrike has no normalizer profile and uses the generic one | open — recorded in the reality audit |+| OpenSearch is started by `full` and read by nothing | open — recorded, not drawn into a diagram |++---+## Not in v8.1 either: packaging`release.yml` already builds, packs and would upload all eight packages on