Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
AiSOC System Design
This document describes the end-to-end architecture of the AiSOC platform after the v2 enterprise upgrade. It covers data flow, the new knowledge graph, the detection rule engine, the threat intelligence pipeline, and the ML-augmented alert fusion pipeline.
1. High-level Topology
┌──────────────────────────────────────────────┐
│ External Sources │
│ EDR / SIEM / Cloud / Identity / Network │
│ CrowdStrike · Splunk · AWS · Okta · Sentinel│
└─────────┬───────────────────────┬────────────┘
│ │
Webhooks / Email Vendor APIs
(Universal Capture) (poll loop)
│ │
▼ ▼
┌─────────────────┐ ┌──────────────────────┐
│ services/api │ │ services/connectors │
│ inbox tokens + │ │ APScheduler poll │
│ HMAC webhooks │ │ 90 connector classes│
│ CEF / HEC / DNS│ │ push_case + status │
└────────┬────────┘ └──────────┬───────────┘
│ │
└───────────┬───────────┘
│
▼
┌──────────────────────────────────────────────────────┐
│ services/ingest (Go) │
│ ─ OCSF normalizer │
│ ─ MITRE ATT&CK technique tagger (in-process index) │
│ ─ Shodan enrichment (TTL cache) │
│ ─ CISA KEV correlator → VULNERABILITY_MATCH events │
└────────┬───────────────────────────┬─────────────────┘
│ │
ocsf.events vulnerability.matches
│ │
▼ ▼
┌────────────────────┐ ┌────────────────────┐
│ Apache Kafka │ │ Apache Kafka │
└────────┬───────────┘ └─────────┬──────────┘
│ │
▼ ▼
┌──────────────────────────────────────────────────┐
│ services/fusion (Python) │
│ ─ Simhash dedup (Redis) │
│ ─ Entity + ATT&CK correlation │
│ ─ MLScorer: Isolation Forest + LightGBM ranker │
│ ─ Analyst feedback loop │
└────────┬─────────────────────────────────────────┘
│
fused.alerts
│
▼
┌──────────────────────────────────────────────────┐
│ services/api (FastAPI) │
│ ─ REST + WebSocket endpoints │
│ ─ Detection rule engine (Sigma / YARA / KQL) │
│ ─ Neo4j graph_service (entity + attack-path) │
│ ─ Cases / RBAC / audit │
└────┬───────────┬───────────────┬─────────────────┘
│ │ │
▼ ▼ ▼
┌────────┐ ┌────────┐ ┌────────────────┐
│PostgreSQL│ │Neo4j │ │OpenSearch + QDR│
│ (state) │ │(graph)│ │ (search + RAG) │
└──────────┘ └───────┘ └────────────────┘
┌──────────────────────────────────┐
│ services/threatintel │
│ TAXII 2.1 · MISP · OTX · KEV │
│ → OpenSearch · Qdrant · Neo4j │
└──────────────────────────────────┘
┌──────────────────────────────────┐
│ services/agents │
│ LangGraph + full ATT&CK + RAG │
│ (Anthropic Claude 3.7 / GPT-4o) │
└──────────────────────────────────┘
2. Service Responsibilities
| Service | Language | Hot path | Async path |
|---|---|---|---|
services/ingest |
Go 1.21 | Normalize raw events → OCSF, tag ATT&CK, enrich Shodan | Emit vulnerability.matches topic |
services/enrichment |
Go 1.21 | Per-IOC TTL-cached enrichment lookups | n/a |
services/fusion |
Python 3.11 | Simhash dedup → correlation → ML scoring → publish fused alerts | Background ML retrain on feedback |
services/api |
Python 3.11 | REST + WS, RBAC, case mgmt, rule engine, graph queries, ITSM webhook inbox, case fan-out | Schema migrations on boot |
services/connectors |
Python 3.11 | 90 connector classes, APScheduler poll, push_case / push_status_change for Jira & ServiceNow |
CredentialVault decrypt at poll time |
services/agents |
Python 3.11 | LangGraph multi-agent investigation runs | Loads full ATT&CK STIX bundle on boot, optional Qdrant embed |
services/actions |
Python 3.11 | SOAR action execution with blast-radius gating | Approval workflows |
services/threatintel |
Python 3.11 | IOC/actor search API | APScheduler poll loop for TAXII/MISP/OTX/KEV |
services/ueba |
Python 3.11 | Welford / Z-score user-behavior analytics, peer-group baselines | Background baseline rebuild |
services/honeytokens |
Python 3.11 | Deception platform: mints AWS keys, doc lures, DNS canaries; emits honeytoken.tripped |
n/a |
services/purple-team |
Python 3.11 | Adversary emulation orchestrator (ATT&CK technique runner) | Detection-coverage scorer |
services/osquery-tls |
Go 1.25 | TLS server for osquery enrol / config / distributed / log endpoints | Ships normalised host events into services/ingest |
services/osquery-extensions |
Go 1.25 | Out-of-band osquery extensions (custom virtual tables + decorators) | Loaded by osquery-tls-managed agents |
services/slack-bot |
Python 3.11 | ChatOps surface: posts approval prompts, /aisoc slash command |
Verifies inbound interactions with HMAC-signed Slack request signatures |
services/realtime |
Node 22 | WebSocket fan-out + Web Push for the web console + Responder PWA | n/a |
services/mcp |
Node 22 (TypeScript) | Model Context Protocol stdio server, 19 tools for IDE-side agents (Claude / Cursor / Continue / Cody) — discovery, deep-dive, action/replay, and the warm-tier lake query pair (aisoc_lake_schema, aisoc_lake_query) |
n/a |
apps/web |
Next.js 14 | Server components + SWR + Responder PWA route group + benchmark scoreboard | n/a |
3. Knowledge Graph (Neo4j)
The knowledge graph is the connective tissue between alerts, entities, and the MITRE ATT&CK matrix.
3.1 Node Labels
| Label | Key | Purpose |
|---|---|---|
Host |
id (host UUID or hostname) |
Endpoint context |
User |
id (sub or upn) |
Identity context |
Alert |
id (alert UUID) |
Detected event |
Case |
id (case UUID) |
Triage container |
IOC |
value (sha256/ip/domain) |
Atomic indicators |
Technique |
id (e.g. T1059.001) |
MITRE ATT&CK |
Tactic |
id (e.g. TA0002) |
MITRE ATT&CK |
ThreatActor |
id |
Attribution |
3.2 Relationships
(Host)-[:LOGGED_IN]->(User)
(User)-[:OWNS]->(Host)
(Alert)-[:ON_HOST]->(Host)
(Alert)-[:AFFECTS]->(User)
(Alert)-[:USES]->(Technique)
(Technique)-[:PART_OF]->(Tactic)
(IOC)-[:OBSERVED_IN]->(Alert)
(ThreatActor)-[:USES]->(Technique)
(ThreatActor)-[:OBSERVED_AS]->(IOC)
(Case)-[:CONTAINS]->(Alert)
3.3 Critical Queries
- Attack path (
get_attack_path): given aCase, traverse(:Case)-[:CONTAINS]->(:Alert)-[:USES]->(:Technique)-[:PART_OF]->(:Tactic)ordered by tactic kill-chain index. - Blast radius (
get_blast_radius): variable-length path query from any entity, capped at 3 hops, filtered by relationship type, used by theactionsservice's gating layer. - Entity neighbors: 1-hop graph used by the SOC console "show context" panel.
- MITRE coverage: aggregated counts of distinct techniques observed per tenant per time window.
3.4 Bootstrapping
services/api/app/db/neo4j.py initializes a singleton AsyncDriver, verifies connectivity, and creates uniqueness constraints on (Host.id), (User.id), (Alert.id), (Case.id), (IOC.value), (Technique.id) at startup.
4. Detection Rule Engine
services/api/app/services/rule_engine.py implements multi-language rule execution.
| Rule type | Backend | Notes |
|---|---|---|
| Sigma | pySigma → OpenSearch query |
Most common path; supports field mappings |
| YARA | yara-python |
Targets file/memory artifacts |
| KQL | Translated to ClickHouse SQL | For event analytics |
| Lucene / Regex | Direct OpenSearch | For full-text patterns |
4.1 Endpoints
| Method | Path | Purpose |
|---|---|---|
GET /v1/rules |
List rules with filtering | |
POST /v1/rules |
Create a new rule | |
POST /v1/rules/{id}/execute |
Execute single rule on demand | |
POST /v1/rules/hunt |
Multi-rule, time-bounded hunt | |
PATCH /v1/rules/{id} |
Update rule (validate, version) |
4.2 Lifecycle
- Rule authored in the web console IDE (Monaco editor with live test panel).
- Validated against a sample event; on save, written to
detection_rulestable. - Scheduler service evaluates
enabled=truerules at their cadence. - Hits emit fused alert candidates onto Kafka via the
fusionpipeline.
5. Threat Intelligence Pipeline
services/threatintel runs an APScheduler-driven poll loop with per-feed handlers.
TAXII 2.1 ─┐
MISP ──────┤ ┌──────────────────────────┐ ┌────────────┐
OTX ──────┼──▶│ ThreatIntelPipeline │──▶│ OpenSearch │
CISA KEV ──┘ │ ─ STIX 2.1 parser │ │ Qdrant │
│ ─ Redis bloom dedup │ │ Neo4j │
└──────────────────────────┘ └────────────┘
5.1 Components
| Module | Role |
|---|---|
clients/taxii.py |
Async TAXII 2.1 client |
clients/misp.py |
MISP REST client |
clients/otx.py |
AlienVault OTX client |
clients/cisa_kev.py |
KEV catalog fetcher |
parsers/stix.py |
Normalize STIX 2.1 SDO/SRO objects |
storage/bloom.py |
Redis-backed Bloom filter (configurable size + FP rate) |
storage/opensearch.py |
Index iocs-* and actors-* |
storage/qdrant.py |
Embed via text-embedding-3-large for semantic recall |
storage/neo4j.py |
Persist actor↔TTP and actor↔IOC graph |
feeds/scheduler.py |
APScheduler interval jobs |
feeds/pipeline.py |
Fan-out to all sinks with idempotency |
5.2 Configuration
All feeds toggle via env vars (see docs/runbooks/LOCAL_DEVELOPMENT.md).
6. ML-Augmented Fusion
services/fusion/app/services/ml_scorer.py adds two scoring dimensions to every fused alert.
| Score | Model | Trigger | Cold-start |
|---|---|---|---|
anomaly_score |
Isolation Forest (sklearn) | Auto-train at ≥50 alerts | Heuristic on event rarity, hour-of-day, severity |
priority_score |
LightGBM LambdaRank |
Trains when ≥100 analyst feedback rows | Heuristic blending severity, anomaly score, asset criticality |
6.1 Feature Vector (_featurize)
- Numerical:
severity(1-5),asset_criticality(0-1),hour_of_day(cyclical),dedup_count,anomaly_score(when ranking) - Categorical: top-N MITRE tactic, source product, alert kind (one-hot)
6.2 Feedback Loop
- Analyst submits
POST /ml/feedbackwithis_true_positiveandassigned_priority1-5. - Fusion buffers feedback; once buffer ≥100, schedules a background retrain.
- New model is hot-swapped behind a
RWLock; old model is retained until validation. GET /ml/statusexposes counts, last-trained timestamp, buffer size.
7. AI Agents (LangGraph)
services/agents boots with the full MITRE ATT&CK STIX bundle and (optionally) embeds it into Qdrant for semantic retrieval.
7.1 Agent DAG
PlannerAgent
│
├─▶ EndpointAgent (CrowdStrike, Velociraptor, GRR)
├─▶ IdentityAgent (Okta, Azure AD, Sentinel)
├─▶ NetworkAgent (Zeek, NetFlow, Darktrace)
├─▶ CloudAgent (AWS, Azure, GCP)
├─▶ EmailAgent (M365 Defender, Gmail)
├─▶ ThreatIntelAgent (calls services/threatintel)
└─▶ VulnerabilityAgent(reads VULNERABILITY_MATCH stream)
│
▼
SynthesisAgent ──▶ ActionAgent (with blast-radius gate)
Each agent emits chain-of-thought traces persisted in PostgreSQL for explainability.
7.2 RAG
- Threat reports: indexed by the threat intel service in Qdrant
- MITRE corpus: technique descriptions embedded on agent boot (toggle via
ATTCK_QDRANT_EMBED=true) - Internal runbooks: ingested via
POST /v1/agents/runbooks
8. Multi-tenancy & Security
- Tenant isolation — All Postgres tables include
tenant_idand are protected by RLS policies. - Neo4j scoping — Every node carries
tenant_idand queries useWHERE n.tenant_id = $tid. - Auth — JWT + API keys; per-tenant scope checks live in
app/api/v1/deps.py. - Audit log — Every mutating endpoint emits an audit entry to
audit_log(Postgres) and Kafka. - Action gating — Blast-radius traversal in Neo4j must return ≤ configured cap before high-impact actions execute.
9. Observability
- Metrics — Prometheus
/metricson every service; Grafana dashboards underinfra/grafana/. - Tracing — OpenTelemetry SDK in Python services; Jaeger for visualization.
- Logging —
structlog(Python) +log/slog(Go) in JSON for ELK ingestion.
10. Deployment Topologies
| Environment | Recommended |
|---|---|
| Dev / demo | docker compose up (this repo) |
| Single-cluster prod | Helm chart at infra/helm/aisoc |
| Multi-region | Terraform infra/terraform + EKS + cross-region MSK |
| Air-gapped | Swap LLM to Ollama-compatible endpoint (OPENAI_BASE_URL) |
11. References
- API Reference
- Local Development
- The lettered deferrals — this entry used to
link
PROGRESS.md, which is gitignored and was never committed.
12. v2.1 Additions (Connector Platform, Case Fan-out, Bidirectional ITSM)
The following subsystems were added on top of the v2 design described above. They are additive — none of the previous architecture changed semantically; the data flow into services/ingest simply gained two new front doors.
12.1 Connector Platform (services/connectors)
A first-class, in-process polling tier with 90 registered connector classes spanning seven categories — edr, siem, cloud, iam, saas, vcs, network. Every connector subclasses BaseConnector and declares:
schema()— self-describingConnectorSchema(name, label, category, fields, oauth, default_poll_interval_seconds)consumed by the web console for the connect form.capabilities()— tuple ofCapabilityenum values (PULL_ALERTS,PUSH_CASE,PUSH_STATUS,FEDERATED_SEARCH, …) that the orchestrator inspects at runtime.normalize()— collapses each vendor's severity ladder into the four-tierinfo | low | medium | highscheme.
Sensitive auth_config fields (marked secret=True in the schema) are encrypted at the application layer by CredentialVault (Fernet AES-128-CBC + HMAC-SHA256, vault token format vault:v1:<base64>). Key rotation is supported via MultiFernet with AISOC_CREDENTIAL_KEY_ROTATION_FROM. The API service holds the encrypt/decrypt keypair authority; services/connectors ships a vendored read-path decrypt_dict() so the scheduler can decrypt at poll time without owning the write path.
Polling runs in-process via ConnectorScheduler (APScheduler): one job per enabled instance, 5-minute default cadence, overridable per-instance via connector_config.poll_interval_seconds. The scheduler reloads jobs every 30 seconds. Disable in tests with AISOC_CONNECTORS_DISABLE_SCHEDULER=1.
Normalized events flow through IngestClient to services/ingest's /v1/ingest/batch endpoint with an X-Tenant-ID header — preserving the OCSF normalization, ATT&CK tagging, and KEV correlation already described in §1.
→ Connector capability matrix → Credential vault threat model
12.2 Universal Capture (/v1/inbox/...)
For sources that don't fit a polled connector — internal SIEM forwarders, syslog/CEF gateways, vendor webhooks, email alerts — services/api exposes a per-tenant token-authenticated webhook inbox. Each token is bound to a template_id (generic-json, cef, splunk-hec, dns-zonefile, itsm-inbound, …) that controls how the body is parsed before it hits the same OCSF normalizer.
Tokens are minted via POST /v1/inbox/tokens, stored hashed (never plaintext), and rate-limited per token. The HMAC verification path (X-AiSOC-Signature: sha256=<hex>) is shared between universal capture and the ITSM inbound webhook described in §12.4.
12.3 Tenant Lake API
A scoped, tenant-isolated query surface over the OpenSearch events-* indices. Every query is rewritten to inject tenant_id as a top-level filter before reaching OpenSearch — defense in depth on top of the existing index-level RBAC. Query timeouts and result-size caps are enforced per-tenant.
12.4 Case Fan-out and Bidirectional ITSM (WS8)
AiSOC is the source of truth for case state. ITSM systems (Jira, ServiceNow) are projections — analysts working in their familiar ticket UI still see correct context, but the canonical state lives in AiSOC.
Outbound (AiSOC → ITSM): When a case is created or its status changes, services/api/app/services/case_fanout.py looks up every connector instance the tenant has enabled with Capability.PUSH_CASE (or PUSH_STATUS) and projects the change. Successes write a row to case_external_refs (case_id, connector_id, external_system, external_id, external_url, external_status, last_synced_at) so subsequent updates target the correct external record. Failures are logged but never block the AiSOC case write — the canonical state is already durable.
Inbound (ITSM → AiSOC): A public-facing webhook at POST /v1/inbox/itsm accepts Jira and ServiceNow status-change payloads. The endpoint:
- Verifies the per-tenant HMAC (
X-AiSOC-Signature). - Parses the vendor-specific payload (
issue.key+issue.fields.status.namefor Jira;sys_id+statefor ServiceNow). - Maps the vendor status into the AiSOC enum via
_JIRA_INBOUND_STATUS/_SNOW_INBOUND_STATUS. - Looks up the matching
case_external_refsrow to find the AiSOCcase_id. - Idempotently applies the status change to the AiSOC case (no-op if the case is already in that state).
The mapping is intentionally lossy in one direction: vendor-specific fields (Jira priority, ServiceNow assignment_group, etc.) are not synced back into AiSOC. The contract is "case status converges, ITSM-specific metadata stays in ITSM."
→ ITSM as a projection of AiSOC (full architecture doc)
12.5 Test Coverage
The v2.1 surfaces ship with focused unit tests rather than mocked integration tests against live vendor APIs:
| Surface | Tests | Location |
|---|---|---|
| Inbound ITSM webhook | 69 | services/api/tests/test_inbox_itsm_endpoint.py |
| Case fan-out | 21 | services/api/tests/test_case_fanout.py |
push_case / push_status_change (Jira + ServiceNow) |
23 | services/connectors/tests/test_push_capabilities.py |
| Total | 113 |
All 113 are passing on the current main. Vendor HTTP calls are mocked at the httpx.AsyncClient layer; database calls are mocked at the SQLAlchemy text(...).bindparams(...) layer with a _bind_params helper that tolerates SQLAlchemy 2.x.
12.6 Plugin SDK and Marketplace
Connectors are also the unit of distribution. Every connector ships a marketplace manifest at plugins/<connector-id>/plugin.yaml mirroring its schema(). The marketplace index at marketplace/index.json is synced to apps/web/public/marketplace/ via pnpm marketplace:sync and consumed by the in-app connector picker. Third-party authors can ship plugins via the same SDK without modifying core.
→ Plugin SDK (Python) → Plugin SDK (Go)
13. v2.2 Additions (Endpoint Telemetry, ChatOps, Responder PWA, MCP, One-Click Install)
The v2.2 increment is scoped to first-mile ergonomics — getting events into AiSOC from every realistic source, getting analysts out of the console for routine approvals, and getting the entire stack onto a freshly-imaged laptop in one command.
13.1 Endpoint Telemetry (osquery TLS server + extensions)
services/osquery-tls is a Go TLS server that implements the four osquery endpoints — /enroll, /config, /distributed/read, /distributed/write, /log — so that any host running osquery can be enrolled into AiSOC with a single enrol secret. Result rows are normalised into OCSF and shipped to services/ingest's /v1/ingest/batch endpoint, sharing the same KEV correlator and ATT&CK tagger described in §1.
services/osquery-extensions registers custom virtual tables and decorators that the standard osquery distribution doesn't ship — for example, AiSOC-specific process-lineage and EDR-correlated host metadata — and is loaded out-of-band by agents managed by osquery-tls. The split (server vs extensions) keeps the TLS server's surface area minimal and lets the extensions iterate without touching the enrol path.
This makes AiSOC self-sufficient for endpoint telemetry: a tenant who doesn't own a CrowdStrike or SentinelOne licence can still get host events into the platform without standing up a separate fleet manager.
13.2 ChatOps (services/slack-bot)
A first-class Slack surface that closes the loop on the action-gating layer described in §8. When services/actions requires approval for a high-blast-radius action, the slack-bot posts an interactive message into the configured channel; an authorised approver clicks the button; Slack posts back to the bot, which verifies the request with an HMAC-signed Slack signature and forwards the approval to the actions service. The bot also exposes a /aisoc slash command for ad-hoc queries (case lookup, ledger snippet, IOC reputation).
Crucially, AiSOC remains the source of truth — Slack is a projection. If the bot is offline, approvals fall back to the web console with no state divergence.
13.3 Responder PWA
A route group inside apps/web (Next.js 14) registers as an installable PWA targeted at on-call responders. It connects to services/realtime for WebSocket case updates and accepts Web Push notifications from the same service, so a responder can be paged on their phone with a tappable approve / deny / escalate prompt without opening the desktop console. The full set of console actions is available offline-tolerant: actions queued while disconnected are replayed when the WebSocket reconnects, with optimistic local state and server-side conflict resolution.
13.4 Model Context Protocol Server (services/mcp)
A TypeScript stdio MCP server that exposes 11 AiSOC tools (case search, alert detail, IOC pivot, ledger query, detection-as-code lookup, …) to IDE-side AI agents — Claude Code, Cursor, Continue, Cody. This makes AiSOC a first-class context source for any analyst writing detections, runbooks, or post-mortems in their editor. The MCP server is read-only by default; write tools require an explicit per-tool capability token.
13.5 Investigation Ledger and Ambient Copilot
Every prompt, tool invocation, and agent step inside services/agents is persisted into the investigation ledger — a per-case, append-only log that is replayable end-to-end. The ledger is the substrate for the Ambient Copilot in apps/web: a sidebar that shows what the agent is doing right now, why it picked the next tool, and what the analyst can do to redirect it. Combined with the LangGraph DAG in §7, this gives AiSOC a debugging surface for AI-driven investigations that mirrors the kind of structured logging analysts already expect from deterministic systems.
13.6 One-Click Install Pipeline
Two cross-platform bootstrap installers live at the repo root:
install.sh— Linux + macOS bash; supportsapt,dnf,pacman,zypper,apk, andbrew. Idempotent; safe to re-run.install.ps1— Windows PowerShell; useswingetand handles WSL2 enablement for Docker Desktop.
Both detect the OS, install missing prerequisites (git, Docker Engine + Compose v2 / Docker Desktop, Node.js 22 LTS, pnpm 8+ via corepack), clone the repo into ~/aisoc, then invoke pnpm aisoc:demo to bring up the slim demo stack from the published GHCR images and open the browser at the seeded LockBit case. Companion uninstallers (uninstall.sh, uninstall.ps1) provide graduated cleanup — stop stack only, drop volumes, remove pulled images, delete node_modules, delete the repo clone — gated behind interactive confirmation prompts unless --all --yes is supplied.
→ One-click install (Docusaurus) → Quick install reference
13.7 Updated Service Inventory (post v2.2)
The full set of services in services/ after v2.2 is:
actions/ agents/ api/ connectors/
demo-producer/ enrichment/ fusion/ honeytokens/
ingest/ mcp/ osquery-extensions/ osquery-tls/
purple-team/ realtime/ slack-bot/ threatintel/
ueba/
17 services in total. The mcp/, osquery-extensions/, osquery-tls/, and slack-bot/ directories are the v2.2 additions; the rest are described in §2 and §12 above.