Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

docs: post-merge Investigation Rail + correlation narrative (PR #106)

Beenu Arora36d2c80d0fd2
diff --git a/AGENTS.md b/AGENTS.md
index 5ced3fcdf..07f0aca85 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -32,6 +32,7 @@
- Severity ladder is exactly five tiers: `info | low | medium | high | critical` (v1.5+). Vendor-native ladders that publish a distinct `critical` (Azure 5-tier, GCP SCC 5-tier, GitHub `critical`, ServiceNow priority 1, AWS GuardDuty ≥8.0, AuditD identity-destruction events, K8s `cluster-admin` bindings, Tailscale tailnet lockdown failures) MUST map to `critical` in their `normalize()` and NOT be collapsed into `high`. Confidence (`alert.confidence`, int 0–100 with band `low | medium | high`) is independent of severity and is emitted by `services/fusion` `ConfidenceScorer`.
- Every connector ships a marketplace manifest at `plugins/<connector-id>/plugin.yaml` mirroring its `schema()`. Run `pnpm marketplace:sync` after adding one.
- Per-connector setup walkthroughs live under `apps/docs/docs/connectors/<connector-id>.md` and are indexed by `apps/docs/sidebars.ts` under the `Connectors` category. The vault threat model + rotation procedure live in `apps/docs/docs/operations/credentials.md`.
+- **Alerts / Investigation Rail (v1.5):** `/alerts` is a two-pane workbench with `InvestigationRail.tsx` on the right. `GET /api/v1/alerts/{id}` returns an envelope (narrative, related entities with `pivotPath`, six-event mini-timeline, `recommended_actions`). Fusion writes deterministic correlation copy at fuse time (`services/fusion/app/services/narrative.py`); API uses `alert_rail.py`, `narrative_projection.py`, and vendored `app/_vendor/narrative.py` (keep in sync via `scripts/sync_vendored_narrative.py`). User doc: `apps/docs/docs/console/investigation-rail.md`.
- v1.4 eval harness conventions:
- Synthetic dataset is fixed at 200 incidents (`services/agents/tests/eval_data/synthetic_incidents.json`) plus an aligned synthetic telemetry corpus (`synthetic_telemetry.jsonl`). Three of the four metrics (`alert_reduction`, `investigation_completeness`, `response_quality`) are substrate self-consistency gates, not agent accuracy scores; only `mitre_accuracy` measures the live agent. The benchmark page (`apps/docs/docs/benchmark.md`) explains which is which.
- PRs touching the agent, orchestrator graph, prompts, tools, RAG corpus, or detection content must re-grade against the harness and include before/after deltas in the PR body if any axis regresses.
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 404ca6c10..019e1d28f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
+### Alerts console — Investigation Rail & correlation narrative (v1.5 W6 / PR-4)
+
+The `/alerts` queue is now a two-pane workbench: the existing queue on the left and an **Investigation Rail** on the right. Selecting a row hydrates narrative, related entities (with pivot links), a six-event mini-timeline, and structured recommended actions — no drawer hop for first-pass triage.
+
+- **`services/fusion`** — `fusion_engine` emits a deterministic **correlation narrative** at fuse time; `narrative.py` mirrors the vendored builder shared with the API.
+- **`services/api`** — `alert_rail.py` builds the rail payload; `narrative_projection.py` + `narrative_loader.py` back-fill legacy alerts on first `GET /api/v1/alerts/{id}`; migration `041_alert_correlation_narrative.sql` stores cached narrative on the alert row; OpenAPI + tests extended for the detail envelope.
+- **`apps/web`** — `InvestigationRail.tsx` + `AlertsView.tsx` integration; Vitest coverage in `InvestigationRail.test.tsx`.
+- **`apps/docs/docs/console/investigation-rail.md`** — operator/analyst guide; sidebar entry under Console.
+- **`scripts/sync_vendored_narrative.py`** — keeps `services/api/app/_vendor/narrative.py` aligned with fusion.
+
### Connectors — Wazuh Indexer ingest (Stage 2)
New first-class endpoint connector for Wazuh deployments. AiSOC now polls the
diff --git a/README.md b/README.md
index e0ffda939..63d26127b 100644
--- a/README.md
+++ b/README.md
@@ -349,15 +349,17 @@ flowchart LR
MCP --> API
```
+**v1.5 console:** On `/alerts/[id]`, the **Investigation Rail** surfaces fusion correlation narrative, evidence chips, and Deep Explain (LLM) with audit logging. See [Investigation Rail](apps/docs/docs/console/investigation-rail.md) in the docs site.
+
### Service map
| Service | Lang | Port | Role |
|---|---|---|---|
-| `web` | Next.js 14 + React | 3000 | SOC console, benchmark scoreboard, marketing landing |
-| `api` | Python · FastAPI | 8000 | Alerts, cases, RBAC, graph, rules, audit, compliance, detection proposals (DAC), federated search fan-out, SLA tracking |
+| `web` | Next.js 14 + React | 3000 | SOC console (alerts detail + Investigation Rail), benchmark scoreboard, marketing landing |
+| `api` | Python · FastAPI | 8000 | Alerts (detail envelope + correlation narrative), cases, RBAC, graph, rules, audit, compliance, detection proposals (DAC), federated search fan-out, SLA tracking |
| `realtime` | Node.js · `ws` | 8086 | Per-channel WebSocket fan-out + VAPID Web Push |
| `agents` | Python · LangGraph | 8001 | Multi-agent reasoning + Qdrant RAG + Hunt-as-Code engine & scheduler |
-| `fusion` | Python | 8003 | Dedup + ML scoring (LightGBM, IsoForest), alert confidence, entity risk / RBA |
+| `fusion` | Python | 8003 | Dedup + ML scoring (LightGBM, IsoForest), alert confidence, entity risk / RBA, correlation narrative projection for API |
| `actions` | Python | 8002 | SOAR with blast-radius gating + ChatOps verification |
| `connectors` | Python | — | Connector polling (APScheduler), credential vault, federated query translators |
| `threatintel` | Python | 8005 | TAXII / MISP / OTX / KEV polling |
@@ -756,7 +758,7 @@ The full OpenAPI 3.1 spec lives at [`docs/openapi.yaml`](docs/openapi.yaml). End
| Tag | Prefix | Notes |
|---|---|---|
| `auth` | `/api/v1/auth/` | JWT login, SAML ACS, OIDC callback |
-| `alerts` | `/api/v1/alerts/` | CRUD, bulk status, timeline |
+| `alerts` | `/api/v1/alerts/` | CRUD, bulk status, timeline, detail envelope (`investigation_rail`, `correlation_narrative`, `deep_explain`) |
| `cases` | `/api/v1/cases/` | Create, link alerts, evidence |
| `rules` | `/api/v1/rules/` | Sigma / YARA / KQL CRUD + test |
| `detections` | `/api/v1/detections/` | Catalog browse + install |
diff --git a/apps/docs/docs/architecture.md b/apps/docs/docs/architecture.md
index 4c32930b5..001a5f235 100644
--- a/apps/docs/docs/architecture.md
+++ b/apps/docs/docs/architecture.md
@@ -73,7 +73,8 @@ The v1.5 release ships from a review of G2, Gartner Peer Insights, and customer
- **Shift handoff dashboard** — `/shifts` shows outgoing/incoming analysts the active cases, in-flight investigations, and queued approvals on one screen. Endpoints in `services/api/app/api/v1/endpoints/shifts.py`.
- **EASM (External Attack Surface Management)** — `/easm` discovers assets, exposed services, and certificate-expiry risks for everything the org points at the public internet.
- **MSSP executive dashboard** — `/mssp` rolls up KPIs, cross-tenant alert volumes, and per-customer SLA posture into a multi-tenant pane.
-- **Alert noise-tuning dashboard** — `/noise-tuning` surfaces per-rule false-positive rate, suppression candidates, and one-click tuning.
+- **Investigation Rail on `/alerts`** — two-pane alert queue: selecting a row opens a right-hand rail with **deterministic correlation narrative** (fusion-time, no LLM on read), **related entities** with pivot links, a **mini-timeline** (case + audit, six events), and **recommended actions**. `GET /api/v1/alerts/{id}` returns a structured envelope; legacy rows lazy back-fill narrative on first read. See [Investigation Rail](./console/investigation-rail).
+- **Alert noise-tuning dashboard** — `/noise-tuning` (and `/detection/tuning` where deployed) surfaces per-rule false-positive rate, suppression candidates, and one-click tuning.
- **Team analytics & gamification** — `/analytics/team` ships analyst leaderboard, MTTR per analyst, dispositions accuracy, and shift workload balance.
- **STIX 2.1 / TAXII 2.1 publishing** — `services/api/app/api/v1/endpoints/stix_taxii.py` pushes the tenant's IOCs and threat-actor profiles to upstream / community feeds.
- **Automated compliance evidence** — `services/api/app/api/v1/endpoints/compliance.py` collects point-in-time evidence for SOC 2 / ISO 27001 / NIST CSF / PCI-DSS / HIPAA / DORA.
@@ -184,12 +185,12 @@ AiSOC/
| Service | Port | Language | Responsibility |
|---------|------|----------|----------------|
-| `api` | 8000 | Python (FastAPI) | REST gateway, auth, RBAC, RLS, audit log, **Investigation Ledger**, Ambient Copilot, marketplace, approvals, on-call, passkeys, push subscriptions, **Detection Proposals** (DAC lifecycle), **Federated Search** fan-out, SLA tracking, **Shifts** handoff, **STIX/TAXII** publishing, **Compliance evidence** collection, **Deployment / air-gap** configuration |
+| `api` | 8000 | Python (FastAPI) | REST gateway, auth, RBAC, RLS, audit log, **Investigation Ledger**, **alert detail envelope** (Investigation Rail: narrative, entities, mini-timeline, actions), Ambient Copilot, marketplace, approvals, on-call, passkeys, push subscriptions, **Detection Proposals** (DAC lifecycle), **Federated Search** fan-out, SLA tracking, **Shifts** handoff, **STIX/TAXII** publishing, **Compliance evidence** collection, **Deployment / air-gap** configuration |
| `agents` | 8001 | Python (LangGraph) | Orchestrator + recon + forensic + responder + report-writer agents, **autonomous triage agent** + phishing / identity / cloud / insider-threat sub-agents, **conversational investigation chat**, playbook engine, ledger writes, **Hunt-as-Code** engine + scheduler |
| `realtime` | 8086 | TypeScript (Node.js) | WebSocket streaming of agent steps; **VAPID Web Push** delivery for the Responder PWA |
| `ingest` | 8081 | Go | OCSF normalisation, Bloom-filter dedup, Kafka publish |
| `enrichment` | 8080 | Go | Enrichment fan-out (IP, domain, hash, email, user) |
-| `fusion` | 8003 | Python | ML scoring (LightGBM + Isolation Forest), correlation, **alert confidence scoring**, **entity risk / RBA** |
+| `fusion` | 8003 | Python | ML scoring (LightGBM + Isolation Forest), correlation, **deterministic correlation narratives** on fused alerts, **alert confidence scoring**, **entity risk / RBA** |
| `actions` | 8002 | Python | Plugin action executor, blast-radius gating, **ChatOps verification** (HMAC-signed Slack/Teams prompts) |
| `threatintel` | 8005 | Python | TAXII 2.1 / MISP / OTX / KEV ingestion + triple storage |
| `ueba` | 8007 | Python | Welford baseline, Z-score scoring, anomaly stream |
@@ -201,7 +202,7 @@ AiSOC/
| `osquery-tls` | 8443 | Go | TLS server implementing osquery's enrol/config/distributed/log endpoints; ships normalised host events into `ingest` |
| `osquery-extensions` | — | Go | Out-of-band osquery extensions registering custom virtual tables and decorators consumed by `osquery-tls` |
| `slack-bot` | — | Python | ChatOps surface: posts approval prompts, exposes `/aisoc` slash command, verifies inbound interactions with HMAC-signed Slack request signatures |
-| `web` | 3000 | TypeScript (Next.js) | React console + Responder PWA route group, **benchmark scoreboard**, conversational investigation chat |
+| `web` | 3000 | TypeScript (Next.js) | React console + Responder PWA route group, **`/alerts` Investigation Rail**, **benchmark scoreboard**, conversational investigation chat |
## Storage Tier
@@ -230,6 +231,27 @@ The agent-side writer lives in
and the UI consumer is
[`apps/web/src/components/cases/InvestigationLedger.tsx`](https://github.com/beenuar/AiSOC/blob/main/apps/web/src/components/cases/InvestigationLedger.tsx).
+## Investigation Rail and correlation narrative
+
+The alert queue (`/alerts`) pairs the sortable table with an **Investigation Rail**
+([`InvestigationRail.tsx`](https://github.com/beenuar/AiSOC/blob/main/apps/web/src/components/alerts/InvestigationRail.tsx))
+fed by `GET /api/v1/alerts/{id}`. The response envelope is assembled in
+[`services/api/app/services/alert_rail.py`](https://github.com/beenuar/AiSOC/blob/main/services/api/app/services/alert_rail.py)
+(narrative text, entity buckets, merged mini-timeline, recommended actions).
+
+**Narrative** — At fusion time, `services/fusion` runs the same deterministic
+builder as the API vendored copy ([`narrative.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/narrative.py))
+so promoted alerts persist a short explanation of *which* signals correlated
+and *why*. Reads do not call an LLM. Alerts created before this shipped get a
+lazy projection on first detail fetch via
+[`narrative_projection.py`](https://github.com/beenuar/AiSOC/blob/main/services/api/app/services/narrative_projection.py),
+then the text is cached on the row (see migration `041_alert_correlation_narrative.sql`).
+
+**Sync** — When the narrative builder changes, run `scripts/sync_vendored_narrative.py`
+so `services/api/app/_vendor/narrative.py` stays aligned with fusion.
+
+Operator-facing detail: [Investigation Rail](./console/investigation-rail).
+
## Responder PWA
The Responder PWA is mounted under the Next.js route group