Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
docs: post-merge Investigation Rail + correlation narrative (PR #106)
Beenu Arora36d2c80d0fd2
diff --git a/AGENTS.md b/AGENTS.mdindex 5ced3fcdf..07f0aca85 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -32,6 +32,7 @@- Severity ladder is exactly five tiers: `info | low | medium | high | critical` (v1.5+). Vendor-native ladders that publish a distinct `critical` (Azure 5-tier, GCP SCC 5-tier, GitHub `critical`, ServiceNow priority 1, AWS GuardDuty ≥8.0, AuditD identity-destruction events, K8s `cluster-admin` bindings, Tailscale tailnet lockdown failures) MUST map to `critical` in their `normalize()` and NOT be collapsed into `high`. Confidence (`alert.confidence`, int 0–100 with band `low | medium | high`) is independent of severity and is emitted by `services/fusion` `ConfidenceScorer`.- Every connector ships a marketplace manifest at `plugins/<connector-id>/plugin.yaml` mirroring its `schema()`. Run `pnpm marketplace:sync` after adding one.- Per-connector setup walkthroughs live under `apps/docs/docs/connectors/<connector-id>.md` and are indexed by `apps/docs/sidebars.ts` under the `Connectors` category. The vault threat model + rotation procedure live in `apps/docs/docs/operations/credentials.md`.+- **Alerts / Investigation Rail (v1.5):** `/alerts` is a two-pane workbench with `InvestigationRail.tsx` on the right. `GET /api/v1/alerts/{id}` returns an envelope (narrative, related entities with `pivotPath`, six-event mini-timeline, `recommended_actions`). Fusion writes deterministic correlation copy at fuse time (`services/fusion/app/services/narrative.py`); API uses `alert_rail.py`, `narrative_projection.py`, and vendored `app/_vendor/narrative.py` (keep in sync via `scripts/sync_vendored_narrative.py`). User doc: `apps/docs/docs/console/investigation-rail.md`.- v1.4 eval harness conventions:- Synthetic dataset is fixed at 200 incidents (`services/agents/tests/eval_data/synthetic_incidents.json`) plus an aligned synthetic telemetry corpus (`synthetic_telemetry.jsonl`). Three of the four metrics (`alert_reduction`, `investigation_completeness`, `response_quality`) are substrate self-consistency gates, not agent accuracy scores; only `mitre_accuracy` measures the live agent. The benchmark page (`apps/docs/docs/benchmark.md`) explains which is which.- PRs touching the agent, orchestrator graph, prompts, tools, RAG corpus, or detection content must re-grade against the harness and include before/after deltas in the PR body if any axis regresses.diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 404ca6c10..019e1d28f 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]+### Alerts console — Investigation Rail & correlation narrative (v1.5 W6 / PR-4)++The `/alerts` queue is now a two-pane workbench: the existing queue on the left and an **Investigation Rail** on the right. Selecting a row hydrates narrative, related entities (with pivot links), a six-event mini-timeline, and structured recommended actions — no drawer hop for first-pass triage.++- **`services/fusion`** — `fusion_engine` emits a deterministic **correlation narrative** at fuse time; `narrative.py` mirrors the vendored builder shared with the API.+- **`services/api`** — `alert_rail.py` builds the rail payload; `narrative_projection.py` + `narrative_loader.py` back-fill legacy alerts on first `GET /api/v1/alerts/{id}`; migration `041_alert_correlation_narrative.sql` stores cached narrative on the alert row; OpenAPI + tests extended for the detail envelope.+- **`apps/web`** — `InvestigationRail.tsx` + `AlertsView.tsx` integration; Vitest coverage in `InvestigationRail.test.tsx`.+- **`apps/docs/docs/console/investigation-rail.md`** — operator/analyst guide; sidebar entry under Console.+- **`scripts/sync_vendored_narrative.py`** — keeps `services/api/app/_vendor/narrative.py` aligned with fusion.+### Connectors — Wazuh Indexer ingest (Stage 2)New first-class endpoint connector for Wazuh deployments. AiSOC now polls thediff --git a/README.md b/README.mdindex e0ffda939..63d26127b 100644--- a/README.md+++ b/README.md@@ -349,15 +349,17 @@ flowchart LRMCP --> API```+**v1.5 console:** On `/alerts/[id]`, the **Investigation Rail** surfaces fusion correlation narrative, evidence chips, and Deep Explain (LLM) with audit logging. See [Investigation Rail](apps/docs/docs/console/investigation-rail.md) in the docs site.+### Service map| Service | Lang | Port | Role ||---|---|---|---|-| `web` | Next.js 14 + React | 3000 | SOC console, benchmark scoreboard, marketing landing |-| `api` | Python · FastAPI | 8000 | Alerts, cases, RBAC, graph, rules, audit, compliance, detection proposals (DAC), federated search fan-out, SLA tracking |+| `web` | Next.js 14 + React | 3000 | SOC console (alerts detail + Investigation Rail), benchmark scoreboard, marketing landing |+| `api` | Python · FastAPI | 8000 | Alerts (detail envelope + correlation narrative), cases, RBAC, graph, rules, audit, compliance, detection proposals (DAC), federated search fan-out, SLA tracking || `realtime` | Node.js · `ws` | 8086 | Per-channel WebSocket fan-out + VAPID Web Push || `agents` | Python · LangGraph | 8001 | Multi-agent reasoning + Qdrant RAG + Hunt-as-Code engine & scheduler |-| `fusion` | Python | 8003 | Dedup + ML scoring (LightGBM, IsoForest), alert confidence, entity risk / RBA |+| `fusion` | Python | 8003 | Dedup + ML scoring (LightGBM, IsoForest), alert confidence, entity risk / RBA, correlation narrative projection for API || `actions` | Python | 8002 | SOAR with blast-radius gating + ChatOps verification || `connectors` | Python | — | Connector polling (APScheduler), credential vault, federated query translators || `threatintel` | Python | 8005 | TAXII / MISP / OTX / KEV polling |@@ -756,7 +758,7 @@ The full OpenAPI 3.1 spec lives at [`docs/openapi.yaml`](docs/openapi.yaml). End| Tag | Prefix | Notes ||---|---|---|| `auth` | `/api/v1/auth/` | JWT login, SAML ACS, OIDC callback |-| `alerts` | `/api/v1/alerts/` | CRUD, bulk status, timeline |+| `alerts` | `/api/v1/alerts/` | CRUD, bulk status, timeline, detail envelope (`investigation_rail`, `correlation_narrative`, `deep_explain`) || `cases` | `/api/v1/cases/` | Create, link alerts, evidence || `rules` | `/api/v1/rules/` | Sigma / YARA / KQL CRUD + test || `detections` | `/api/v1/detections/` | Catalog browse + install |diff --git a/apps/docs/docs/architecture.md b/apps/docs/docs/architecture.mdindex 4c32930b5..001a5f235 100644--- a/apps/docs/docs/architecture.md+++ b/apps/docs/docs/architecture.md@@ -73,7 +73,8 @@ The v1.5 release ships from a review of G2, Gartner Peer Insights, and customer- **Shift handoff dashboard** — `/shifts` shows outgoing/incoming analysts the active cases, in-flight investigations, and queued approvals on one screen. Endpoints in `services/api/app/api/v1/endpoints/shifts.py`.- **EASM (External Attack Surface Management)** — `/easm` discovers assets, exposed services, and certificate-expiry risks for everything the org points at the public internet.- **MSSP executive dashboard** — `/mssp` rolls up KPIs, cross-tenant alert volumes, and per-customer SLA posture into a multi-tenant pane.-- **Alert noise-tuning dashboard** — `/noise-tuning` surfaces per-rule false-positive rate, suppression candidates, and one-click tuning.+- **Investigation Rail on `/alerts`** — two-pane alert queue: selecting a row opens a right-hand rail with **deterministic correlation narrative** (fusion-time, no LLM on read), **related entities** with pivot links, a **mini-timeline** (case + audit, six events), and **recommended actions**. `GET /api/v1/alerts/{id}` returns a structured envelope; legacy rows lazy back-fill narrative on first read. See [Investigation Rail](./console/investigation-rail).+- **Alert noise-tuning dashboard** — `/noise-tuning` (and `/detection/tuning` where deployed) surfaces per-rule false-positive rate, suppression candidates, and one-click tuning.- **Team analytics & gamification** — `/analytics/team` ships analyst leaderboard, MTTR per analyst, dispositions accuracy, and shift workload balance.- **STIX 2.1 / TAXII 2.1 publishing** — `services/api/app/api/v1/endpoints/stix_taxii.py` pushes the tenant's IOCs and threat-actor profiles to upstream / community feeds.- **Automated compliance evidence** — `services/api/app/api/v1/endpoints/compliance.py` collects point-in-time evidence for SOC 2 / ISO 27001 / NIST CSF / PCI-DSS / HIPAA / DORA.@@ -184,12 +185,12 @@ AiSOC/| Service | Port | Language | Responsibility ||---------|------|----------|----------------|-| `api` | 8000 | Python (FastAPI) | REST gateway, auth, RBAC, RLS, audit log, **Investigation Ledger**, Ambient Copilot, marketplace, approvals, on-call, passkeys, push subscriptions, **Detection Proposals** (DAC lifecycle), **Federated Search** fan-out, SLA tracking, **Shifts** handoff, **STIX/TAXII** publishing, **Compliance evidence** collection, **Deployment / air-gap** configuration |+| `api` | 8000 | Python (FastAPI) | REST gateway, auth, RBAC, RLS, audit log, **Investigation Ledger**, **alert detail envelope** (Investigation Rail: narrative, entities, mini-timeline, actions), Ambient Copilot, marketplace, approvals, on-call, passkeys, push subscriptions, **Detection Proposals** (DAC lifecycle), **Federated Search** fan-out, SLA tracking, **Shifts** handoff, **STIX/TAXII** publishing, **Compliance evidence** collection, **Deployment / air-gap** configuration || `agents` | 8001 | Python (LangGraph) | Orchestrator + recon + forensic + responder + report-writer agents, **autonomous triage agent** + phishing / identity / cloud / insider-threat sub-agents, **conversational investigation chat**, playbook engine, ledger writes, **Hunt-as-Code** engine + scheduler || `realtime` | 8086 | TypeScript (Node.js) | WebSocket streaming of agent steps; **VAPID Web Push** delivery for the Responder PWA || `ingest` | 8081 | Go | OCSF normalisation, Bloom-filter dedup, Kafka publish || `enrichment` | 8080 | Go | Enrichment fan-out (IP, domain, hash, email, user) |-| `fusion` | 8003 | Python | ML scoring (LightGBM + Isolation Forest), correlation, **alert confidence scoring**, **entity risk / RBA** |+| `fusion` | 8003 | Python | ML scoring (LightGBM + Isolation Forest), correlation, **deterministic correlation narratives** on fused alerts, **alert confidence scoring**, **entity risk / RBA** || `actions` | 8002 | Python | Plugin action executor, blast-radius gating, **ChatOps verification** (HMAC-signed Slack/Teams prompts) || `threatintel` | 8005 | Python | TAXII 2.1 / MISP / OTX / KEV ingestion + triple storage || `ueba` | 8007 | Python | Welford baseline, Z-score scoring, anomaly stream |@@ -201,7 +202,7 @@ AiSOC/| `osquery-tls` | 8443 | Go | TLS server implementing osquery's enrol/config/distributed/log endpoints; ships normalised host events into `ingest` || `osquery-extensions` | — | Go | Out-of-band osquery extensions registering custom virtual tables and decorators consumed by `osquery-tls` || `slack-bot` | — | Python | ChatOps surface: posts approval prompts, exposes `/aisoc` slash command, verifies inbound interactions with HMAC-signed Slack request signatures |-| `web` | 3000 | TypeScript (Next.js) | React console + Responder PWA route group, **benchmark scoreboard**, conversational investigation chat |+| `web` | 3000 | TypeScript (Next.js) | React console + Responder PWA route group, **`/alerts` Investigation Rail**, **benchmark scoreboard**, conversational investigation chat |## Storage Tier@@ -230,6 +231,27 @@ The agent-side writer lives inand the UI consumer is[`apps/web/src/components/cases/InvestigationLedger.tsx`](https://github.com/beenuar/AiSOC/blob/main/apps/web/src/components/cases/InvestigationLedger.tsx).+## Investigation Rail and correlation narrative++The alert queue (`/alerts`) pairs the sortable table with an **Investigation Rail**+([`InvestigationRail.tsx`](https://github.com/beenuar/AiSOC/blob/main/apps/web/src/components/alerts/InvestigationRail.tsx))+fed by `GET /api/v1/alerts/{id}`. The response envelope is assembled in+[`services/api/app/services/alert_rail.py`](https://github.com/beenuar/AiSOC/blob/main/services/api/app/services/alert_rail.py)+(narrative text, entity buckets, merged mini-timeline, recommended actions).++**Narrative** — At fusion time, `services/fusion` runs the same deterministic+builder as the API vendored copy ([`narrative.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/narrative.py))+so promoted alerts persist a short explanation of *which* signals correlated+and *why*. Reads do not call an LLM. Alerts created before this shipped get a+lazy projection on first detail fetch via+[`narrative_projection.py`](https://github.com/beenuar/AiSOC/blob/main/services/api/app/services/narrative_projection.py),+then the text is cached on the row (see migration `041_alert_correlation_narrative.sql`).++**Sync** — When the narrative builder changes, run `scripts/sync_vendored_narrative.py`+so `services/api/app/_vendor/narrative.py` stays aligned with fusion.++Operator-facing detail: [Investigation Rail](./console/investigation-rail).+## Responder PWAThe Responder PWA is mounted under the Next.js route group