Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

chore(release): v8.1.1 (#728)

Beenu Arora8d3d0560657f
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7856cc126..37a2b7a0b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,124 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
+## [8.1.1] — 2026-09-23
+
+**The v8.1.0 notes described a platform its own quick start never started.**
+This release contains no new capability. It exists because an adoption audit
+of the repository — installability, architecture comprehension, data
+provenance, pipeline connectivity — found that the single most-followed path
+into AiSOC did not run AiSOC, and that finding accounts for most of the
+recurring feedback on its own.
+
+`./install.sh` handed off to a nine-service compose file with **no ingest
+service, no fusion service, and `AISOC_DISABLE_KAFKA: true`**. Everything
+visible in the resulting console came from `seed_demo.py` writing fifteen
+fabricated incidents straight into Postgres. A reader followed the README, saw
+a populated console, and concluded the platform worked — having never run the
+platform. The installer printed *"AiSOC is up and running"* because the
+compose command exited 0.
+
+The good news the audit also produced: **the core pipeline does work.** It had
+simply never been demonstrated. `make smoke` now pushes one real event through
+ingest, Kafka, fusion and detection and reads the alert back from the API,
+reaching past nothing, and CI fails if any stage does.
+
+### Fixed
+
+- **The documented quick start never ran the product** ([#726](https://github.com/beenuar/AiSOC/pull/726)).
+ `install.sh` now runs `make up` — the same CORE stack the README documents
+ and CI tests — and then runs the golden pipeline, so the installer's success
+ banner reports a verified event traversal rather than a process exit code.
+ `infra/compose/docker-compose.demo.yml` is kept for screenshots and UI work
+ and now says at the top of the file that it cannot answer whether AiSOC
+ works.
+- **`services/ingest` answered `/health` unconditionally** ([#727](https://github.com/beenuar/AiSOC/pull/727)),
+ so *"ingest is healthy"* and *"every event is being dropped"* could both be
+ true at once — precisely the state a broker outage produces. `/readyz` now
+ dials Kafka rather than reading a cached connection flag, because kafka-go
+ reconnects lazily and a cached flag is stale in exactly the situation this
+ is meant to catch. Verified live: 200 with Kafka up, 503 with a reason and a
+ log hint with Kafka stopped.
+- **Three of the eight publishable packages could not be built** ([#725](https://github.com/beenuar/AiSOC/pull/725)).
+ The release pipeline builds and packs every package on each tag precisely so
+ it cannot rot while the upload is credential-gated — and on the v8.1.0 tag
+ it earned that design back. `aisoc` and `@aisoc/mcp` failed with
+ `Could not resolve "@aisoc/report-card"`: that workspace dependency's `main`
+ points at a build artifact and `pnpm --filter <pkg> build` never builds
+ dependencies, so the `...` suffix was missing. `aisoc-cli` could not produce
+ a wheel at all — `pyproject.toml` declared `packages = ["src/aisoc_cli"]`
+ *and* a `force-include` mapping the same templates directory to the same
+ wheel path, so hatchling refused on the duplicate. Neither was visible
+ anywhere but a tag. `test_packaging.py` now builds the wheel and looks
+ inside it, because a test that read the config would have declared both
+ fine.
+- **Alerts were labelled `"crowdstrike crowdstrike"`** — vendor and product
+ joined without deduplication, in two copies of the same helper, so fixing
+ one left the other. There is one shared implementation now
+ (`fusion/app/services/provenance.py::product_label`). In the same pass, an
+ alert's `description` was `str(raw_data)`, so the console showed
+ `{"command_line": "powershell.exe -nop ...` where a sentence belongs,
+ discarding the vendor's own description.
+- **Fusion defaulted its enrichment URL to `localhost:8082` — itself** — and
+ swallowed the resulting failure at `DEBUG`; and fusion declared no
+ `depends_on` for Postgres, so it raced the database on cold boot.
+- **Five surfaces rendered fabricated data outside demo mode**: the MSSP
+ overview and its managed-tenant and cross-tenant-incident lists, the
+ Copilot's reply on API error, the air-gap status endpoint reporting
+ unchecked conditions as satisfied, a hard-coded analyst identity in
+ Settings, and the investigation timeline. All are gated behind demo mode and
+ return honest empties otherwise. `seed_demo.py` refuses to run outside
+ development unless `AISOC_ALLOW_SEED=1`.
+
+### Added
+
+- **`tests/e2e/golden_pipeline/`** — one deterministic end-to-end test that
+ pushes raw telemetry into ingest and asserts each boundary independently, so
+ a break names the stage rather than the suite. Wired into CI as
+ `golden-pipeline.yml`, which also verifies the gate fails when the pipeline
+ is broken.
+- **`scripts/doctor.sh` / `make doctor`** — diagnoses Docker, Compose, disk,
+ ports, configuration, Postgres, Redis, Kafka and every service, and prints
+ the command to run next. It checks disk before Kafka and probes the broker
+ directly, because the Docker VM filling up produced a Kafka that reported
+ `healthy` while refusing every request.
+- **`/livez` and `/readyz` on `services/ingest`**, matching `services/api` and
+ `services/fusion`.
+- **`scripts/project_stats.py --check`** — derives the connector count,
+ executable detection count, compose service count and claim-to-gate totals
+ from the tree and fails CI when the README disagrees.
+- **`docs/audit/REPOSITORY_REALITY.md`** — every major component classified
+ WORKING / PARTIAL / BROKEN / DEMO-ONLY / EXPERIMENTAL / DEAD CODE by tracing
+ the implementation, not the filename. It records that OpenSearch is started
+ by the `full` profile and read by nothing, rather than drawing it into a
+ diagram as though it were part of the design.
+- **`docs/architecture/README.md`** — rewritten around *what happens when
+ AiSOC receives one security event*, with five diagrams whose every box links
+ to the directory that implements it.
+- **`docs/testing/CLEAN_INSTALL.md`** — the clean-machine walkthrough, from a
+ real run.
+- Migration `054_alert_provenance.sql` adds `is_synthetic`,
+ `synthetic_source` and `synthetic_scenario` to `alerts`.
+
+### Changed
+
+- **CORE is the default deployment profile**: ten services, roughly 6 GB, and
+ the smallest deployment that takes a real event and produces a real alert —
+ not a cut-down toy. ClickHouse, Neo4j, Qdrant, OpenSearch, enrichment and
+ connectors moved to `full`. Both heavy dependencies were already
+ environment-gated, so nothing was weakened to do this. `make up-full` sets
+ `AISOC_LAKE_WRITER_ENABLED` and `AISOC_GRAPH_ENABLED`, which the profile
+ needs and did not previously set.
+- **README rewritten for adoption** rather than release history: what AiSOC
+ does, quick start, how to tell demo data from real data, connecting real
+ sources, the data flow, project maturity per capability, and what AiSOC is
+ not.
+- **One command set.** `make install / up / up-full / down / restart / status
+ / doctor / smoke / demo / logs / test / test-unit / test-integration /
+ test-e2e / stats / clean`, consolidating the previous scattered scripts.
+- The integration workflow boots the `full` profile with the lake writer and
+ graph enabled, so it exercises the architecture it claims to.
+
## [8.1.0] — 2026-09-23
**Wave-2 features, and the gaps behind them.** Every item in the wave-2
@@ -184,19 +302,6 @@ Claim-to-gate matrix: **108 rows — 99 GATED, 9 PARTIAL, 0 NO GATE**.
### Fixed
-- **Three of the eight publishable packages could not be built.** The release
- pipeline builds and packs every package on each tag precisely so it cannot
- rot while the upload is credential-gated — and on this tag it earned that
- design back. `aisoc` and `@aisoc/mcp` failed with
- `Could not resolve "@aisoc/report-card"`: that workspace dependency's `main`
- points at a build artifact and `pnpm --filter <pkg> build` never builds
- dependencies, so the `...` suffix was missing. `aisoc-cli` could not produce
- a wheel at all — `pyproject.toml` declared `packages = ["src/aisoc_cli"]`
- *and* a `force-include` mapping the same templates directory to the same
- wheel path, so hatchling refused on the duplicate. Both would have blocked
- the first real publish, and neither was visible anywhere but a tag.
- `test_packaging.py` now builds the wheel and looks inside it, because a test
- that read the config would have declared both fine.
- **The sandbox determinism test compared a timer.** Its `VOLATILE` list named
four fields the CLI does not emit — the real one is `elapsed_ms` — so the
comparison included a wall-clock value and failed whenever two runs
@@ -4066,7 +4171,10 @@ demo profile. Details below.
- Helm chart for Kubernetes deployment (`infra/helm/aisoc/`)
- MIT License
-[Unreleased]: https://github.com/beenuar/AiSOC/compare/v5.2.0...HEAD
+[Unreleased]: https://github.com/beenuar/AiSOC/compare/v8.1.1...HEAD
+[8.1.1]: https://github.com/beenuar/AiSOC/compare/v8.1.0...v8.1.1
+[8.1.0]: https://github.com/beenuar/AiSOC/compare/v8.0.0...v8.1.0
+[8.0.0]: https://github.com/beenuar/AiSOC/compare/v7.7.0...v8.0.0
[5.2.0]: https://github.com/beenuar/AiSOC/compare/v5.1.0...v5.2.0
[5.1.0]: https://github.com/beenuar/AiSOC/compare/v5.0.0...v5.1.0
[5.0.0]: https://github.com/beenuar/AiSOC/compare/v4.1.0...v5.0.0
diff --git a/README.md b/README.md
index 9c5bf8dcb..736363b62 100644
--- a/README.md
+++ b/README.md
@@ -7,7 +7,7 @@
**An open-source, self-hostable AI Security Operations Center.** It ingests your security telemetry, detects and correlates threats, investigates them with AI agents whose reasoning is fully auditable, and proposes responses a human approves.
[![License: MIT](https://img.shields.io/badge/License-MIT-22c55e.svg?style=flat-square)](https://opensource.org/licenses/MIT)
-[![Version](https://img.shields.io/badge/version-8.1.0-f59e0b?style=flat-square)](CHANGELOG.md)
+[![Version](https://img.shields.io/badge/version-8.1.1-f59e0b?style=flat-square)](CHANGELOG.md)
[![CI](https://img.shields.io/github/actions/workflow/status/beenuar/AiSOC/ci.yml?branch=main&label=CI&style=flat-square)](https://github.com/beenuar/AiSOC/actions/workflows/ci.yml)
[![CodeQL](https://img.shields.io/github/actions/workflow/status/beenuar/AiSOC/codeql.yml?branch=main&label=CodeQL&style=flat-square)](https://github.com/beenuar/AiSOC/actions/workflows/codeql.yml)
[![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/beenuar/AiSOC/badge)](https://securityscorecards.dev/viewer/?uri=github.com/beenuar/AiSOC)
diff --git a/RELEASES.md b/RELEASES.md
index 5497cab1f..cce03760b 100644
--- a/RELEASES.md
+++ b/RELEASES.md
@@ -2,13 +2,29 @@
This file mirrors what used to live in the "What's new" section of [`README.md`](README.md). The complete, machine-readable inventory (with file paths, env-var diffs, and per-release test counts) lives in [`CHANGELOG.md`](CHANGELOG.md).
-> **TL;DR for first-time visitors:** AiSOC is on `v8.1.0`, released 2026-09-23 — wave-2 features plus the gaps behind them. Every wave-2 backlog item was audited against the tree before any code was written, and the backlog was wrong in both directions: two items were already built, four had the capability present with the path that feeds it broken. One inverted what the feature appeared to do — a Slack or Teams approval authorized nobody. Every product claim is backed by a failing CI test (claim-to-gate matrix: 99 GATED / 9 PARTIAL / 0 NO GATE). The latest GitHub release with notes and downloads: <https://github.com/beenuar/AiSOC/releases/latest>.
+> **TL;DR for first-time visitors:** AiSOC is on `v8.1.1`, released 2026-09-23 — an adoption-audit release with no new capability. The audit found that the single most-followed path into AiSOC did not run AiSOC: `./install.sh` started a compose file with no ingest service, no fusion service and Kafka disabled, and everything in the resulting console had been written straight into Postgres by a seed script. The core pipeline does work; it had never been demonstrated, and `make smoke` now demonstrates it in CI. Every product claim is backed by a failing CI test (claim-to-gate matrix: 99 GATED / 9 PARTIAL / 0 NO GATE). The latest GitHub release with notes and downloads: <https://github.com/beenuar/AiSOC/releases/latest>.
---
## What's new
-`VERSION` is `8.1.0`. The **v8.1.0** release (2026-09-23) delivers the wave-2 backlog and the defects auditing it surfaced.
+`VERSION` is `8.1.1`. The **v8.1.1** release (2026-09-23) adds no capability. It exists because an audit of how the repository is adopted — installability, architecture comprehension, data provenance, pipeline connectivity — found the quick start did not start the product.
+
+**v8.1.1 highlights (September 23, 2026)**
+- **The documented quick start never ran the product.** `./install.sh` handed off to a nine-service compose file with no ingest service, no fusion service, and `AISOC_DISABLE_KAFKA: true`. Everything visible in the console came from `seed_demo.py` writing fifteen fabricated incidents straight into Postgres, and the installer printed "AiSOC is up and running" because the compose command exited 0. It now runs `make up` — the same CORE stack the README documents and CI tests — then proves the pipeline before claiming success.
+- **`make smoke` is the claim.** One real event enters ingest, travels Kafka and fusion, matches a detection, and is read back from the API, with each of the eight stages reporting independently so a break names the boundary. It reaches past nothing: no stubbed Kafka, no inserted alert. CI fails if any stage does, and separately verifies the gate fails when the pipeline is broken.
+- **`services/ingest` answered `/health` unconditionally**, so "ingest is healthy" and "every event is being dropped" could both be true at once — exactly the state a broker outage produces. `/readyz` now dials Kafka; verified live at 200 up and 503 down, with a reason and the log command to run.
+- **CORE is the default: ten services, roughly 6 GB**, and it is the smallest deployment that turns a real event into a real alert rather than a cut-down toy. ClickHouse, Neo4j, Qdrant, OpenSearch, enrichment and connectors moved to `full`. OpenSearch is started by `full` and read by nothing, and is recorded that way instead of appearing in a diagram.
+- **Five surfaces rendered fabricated data outside demo mode** — the MSSP overview, the Copilot's reply on API error, the air-gap status endpoint, an analyst identity in Settings, and the investigation timeline. All are gated now and return honest empties otherwise.
+- **Three of the eight publishable packages could not be built.** The v8.1.0 tag surfaced it, which is the credential-gated build design working: an unresolvable workspace dependency for `aisoc` and `@aisoc/mcp`, and a duplicate-path wheel failure for `aisoc-cli`. Both would have blocked the first real publish.
+
+`make doctor`, `docs/audit/REPOSITORY_REALITY.md`, a data-flow rewrite of `docs/architecture/README.md`, and `docs/testing/CLEAN_INSTALL.md` came out of the same pass. The full inventory lives under `[8.1.1]` in [`CHANGELOG.md`](CHANGELOG.md).
+
+---
+
+## v8.1.0 (2026-09-23)
+
+`VERSION` was `8.1.0`. The **v8.1.0** release (2026-09-23) delivers the wave-2 backlog and the defects auditing it surfaced.
**v8.1.0 highlights (September 23, 2026)**
- **A ChatOps approval authorized nobody.** The Slack and Teams bots verified who clicked — Slack signs every interaction payload, Teams payloads carry an HMAC — recorded that person in an audit event, and then called the actions service with no approver. The permission-tier check and separation of duties were both skipped. A bot cannot supply permissions (it knows a Slack user id and has no idea what that person may do in AiSOC), so it now asserts identity only and the actions service maps it through operator configuration. **If you use ChatOps approvals you must populate `AISOC_CHATOPS_APPROVERS` or they will be refused** — that is the intended failure.
diff --git a/ROADMAP.md b/ROADMAP.md
index 69e5b6054..4946e78db 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -625,6 +625,32 @@ against **v8.2**, and it becomes a re-tag the moment the credentials exist.
---
+## v8.1.1 — Shipped (2026-09-23)
+
+An adoption audit, and no new capability. The recurring feedback on this
+project — hard to install, fabricated data, architecture nobody could follow,
+documentation describing things a reader could not reproduce — turned out to
+share one root cause: **the documented quick start did not run the product.**
+`./install.sh` started a compose file with no ingest service, no fusion
+service and `AISOC_DISABLE_KAFKA: true`, and the populated console a reader
+saw was a seed script writing rows into Postgres.
+
+The pipeline itself works. It had simply never been demonstrated, and now
+`make smoke` demonstrates it on every CI run: one real event through ingest,
+Kafka, fusion and detection, read back from the API, eight stages each
+reporting independently.
+
+Also in this release: `/readyz` on `services/ingest` that dials Kafka rather
+than answering unconditionally, `make doctor`, CORE as the default ten-service
+profile, five fabricated-data surfaces gated behind demo mode, the three
+packages that could not be built at the v8.1.0 tag, and three new documents —
+`docs/audit/REPOSITORY_REALITY.md`, a data-flow rewrite of
+`docs/architecture/README.md`, and `docs/testing/CLEAN_INSTALL.md`.
+
+Full inventory under `[8.1.1]` in [`CHANGELOG.md`](CHANGELOG.md).
+
+---
+
## Ideas Backlog (unscheduled)
- "Explain this alert" button using LLM with enrichment context
diff --git a/VERSION b/VERSION
index 8104cabd3..0e7915245 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-8.1.0
+8.1.1
diff --git a/apps/docs/docs/integrations/github-action.md b/apps/docs/docs/integrations/github-action.md
index dceb67cec..8b8ce8a49 100644
--- a/apps/docs/docs/integrations/github-action.md
+++ b/apps/docs/docs/integrations/github-action.md
@@ -16,7 +16,7 @@ as a PR comment or job summary. No LLM, no data leaves your CI runner.
today points at the action's directory inside the monorepo:
```yaml
-- uses: beenuar/AiSOC/packages/[email protected]
+- uses: beenuar/AiSOC/packages/[email protected]
```
Pin a tag rather than a branch, so a workflow cannot change underneath you.
@@ -41,7 +41,7 @@ jobs:
triage:
runs-on: ubuntu-latest
steps:
- - uses: beenuar/AiSOC/packages/[email protected]
+ - uses: beenuar/AiSOC/packages/[email protected]
with:
mode: pr-comment
min-severity: low
@@ -70,7 +70,7 @@ jobs:
digest:
runs-on: ubuntu-latest
steps:
- - uses: beenuar/AiSOC/packages/[email protected]
+ - uses: beenuar/AiSOC/packages/[email protected]
with:
mode: digest
```
diff --git a/apps/web/package.json b/apps/web/package.json
index e05c1fb81..017c6a4a5 100644
--- a/apps/web/package.json
+++ b/apps/web/package.json
@@ -1,6 +1,6 @@
{
"name": "@aisoc/web",
- "version": "8.1.0",
+ "version": "8.1.1",
"private": true,
"description": "AiSOC SOC Console - Next.js 14 Frontend",
"license": "MIT",
diff --git a/docs/operations/publishing.md b/docs/operations/publishing.md
index dfceceaa6..1f2862b93 100644
--- a/docs/operations/publishing.md
+++ b/docs/operations/publishing.md
@@ -91,7 +91,7 @@ There is no API token to store or rotate in this path.
reference that resolves is the action's directory inside this monorepo:
```yaml
-- uses: beenuar/AiSOC/packages/[email protected]
+- uses: beenuar/AiSOC/packages/[email protected]
```
That form needs no setup and is what the README and
diff --git a/docs/roadmap/v8-progress.md b/docs/roadmap/v8-progress.md
index 325fbf4a4..a02ab9616 100644
--- a/docs/roadmap/v8-progress.md
+++ b/docs/roadmap/v8-progress.md
@@ -1,7 +1,7 @@
# AiSOC v8 — progress tracker
**Last updated:** 2026-09-23
-**Current release:** `v8.1.0` (2026-09-23) · **Next:** `v8.2` (packaging)
+**Current release:** `v8.1.1` (2026-09-23) · **Next:** `v8.2` (packaging)
This tracker is the at-a-glance view of what has landed across the v8 line and
what is still open. It is deliberately short and dated. When it disagrees with
@@ -92,6 +92,37 @@ optional one.
---
+## v8.1.1 — shipped 2026-09-23
+
+An adoption audit with no new capability, run against the whole repository:
+installability, architecture comprehension, data provenance, pipeline
+connectivity, and whether documented commands work. The headline finding is
+that `./install.sh` started a compose file with **no ingest service, no fusion
+service and Kafka disabled** — so the most-followed path into the project did
+not run the project, and the console a reader saw had been populated by a seed
+script.
+
+| Item | Status |
+|------|--------|
+| `install.sh` runs `make up` (CORE) and then the golden pipeline | done |
+| `tests/e2e/golden_pipeline/` + `golden-pipeline.yml`, verified to fail when broken | done |
+| `scripts/doctor.sh` / `make doctor` | done |
+| `/livez` + `/readyz` on `services/ingest`, dialling Kafka | done |
+| CORE as the default profile; ClickHouse / Neo4j / Qdrant / OpenSearch / enrichment / connectors to `full` | done |
+| Five fabricated-data surfaces gated behind demo mode; `seed_demo.py` refuses outside development | done |
+| `054_alert_provenance.sql` — `is_synthetic` on `alerts` | done |
+| `scripts/project_stats.py --check` — README figures derived from the tree | done |
+| `docs/audit/REPOSITORY_REALITY.md` | done |
+| `docs/architecture/README.md` rewritten around one event's journey | done |
+| `docs/testing/CLEAN_INSTALL.md` | done |
+| README rewritten for adoption | done |
+| Consolidated `Makefile` | done |
+| The three packages that could not be built at the v8.1.0 tag | done |
+| CrowdStrike has no normalizer profile and uses the generic one | open — recorded in the reality audit |
+| OpenSearch is started by `full` and read by nothing | open — recorded, not drawn into a diagram |
+
+---
+
## Not in v8.1 either: packaging
`release.yml` already builds, packs and would upload all eight packages on