Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
feat(v2): enterprise platform — knowledge graph, rule engine, ML fusion, threat intel
Beenu Arora2a4d1d726eae
diff --git a/.cursor/hooks/state/continual-learning.json b/.cursor/hooks/state/continual-learning.jsonindex 156e94f49..aabec7a39 100644--- a/.cursor/hooks/state/continual-learning.json+++ b/.cursor/hooks/state/continual-learning.json@@ -1,8 +1,8 @@{"version": 1,"lastRunAtMs": 0,- "turnsSinceLastRun": 2,+ "turnsSinceLastRun": 5,"lastTranscriptMtimeMs": null,- "lastProcessedGenerationId": "69225323-44de-4d39-bd6e-cbebf3742283",+ "lastProcessedGenerationId": "822fd199-63a2-4c50-a87f-4c3530f75fb5","trialStartedAtMs": null}diff --git a/PROGRESS.md b/PROGRESS.mdindex 44802d64f..29a2284f1 100644--- a/PROGRESS.md+++ b/PROGRESS.md@@ -1,8 +1,15 @@# AiSOC Build Progress-Last updated: 2024-01-01+Last updated: 2026-05-01-## ✅ ALL TASKS COMPLETE+## ✅ ALL TASKS COMPLETE — v2 Enterprise Platform shipped++| Phase | Description | Status |+|-------|-------------|--------|+| v1 — Initial monorepo | Core services, frontend, infra, docs | ✅ COMPLETED |+| v2 — Enterprise upgrade | Knowledge graph, rule engine, ML fusion, threat intel | ✅ COMPLETED |++### v1 — Initial monorepo| ID | Task | Status ||----|------|--------|@@ -22,50 +29,77 @@ Last updated: 2024-01-01| setup-github | Create GitHub repository and push initial commit | ✅ COMPLETED || github-push | Push complete codebase to GitHub | ✅ COMPLETED |+### v2 — Enterprise platform upgrade++| ID | Task | Status |+|----|------|--------|+| infra-fixes | Reconcile docker-compose ports/profiles for fusion, threatintel, connectors | ✅ COMPLETED |+| neo4j-graph | Add Neo4j to compose; implement `graph_service` (attack path, blast radius, neighbors, MITRE coverage) | ✅ COMPLETED |+| rule-engine | Multi-language detection rule engine: Sigma (pySigma), YARA, KQL, Lucene, Regex with `/v1/rules` API | ✅ COMPLETED |+| attck-corpus | Full MITRE ATT&CK STIX 2.1 corpus loader (Go + Python), in-process index, optional Qdrant embedding | ✅ COMPLETED |+| threatintel-svc | New `services/threatintel`: TAXII 2.1 + MISP + OTX + CISA KEV pollers, Bloom dedup, OpenSearch+Qdrant+Neo4j sinks | ✅ COMPLETED |+| ingest-shodan-cve | Shodan enrichment + CISA KEV cross-correlation in Go ingest, emits `vulnerability.matches` | ✅ COMPLETED |+| ml-fusion | Isolation Forest anomaly + LightGBM LambdaRank priority scoring with analyst feedback loop | ✅ COMPLETED |+| docs-update | Refreshed README, new SYSTEM_DESIGN.md, API_REFERENCE.md, LOCAL_DEVELOPMENT.md runbook | ✅ COMPLETED |+## GitHub Repository**https://github.com/beenuar/AiSOC**-## Services Built--### Backend Services-- `services/api` — Python FastAPI REST API (alerts, cases, RBAC, tenants)-- `services/ingest` — Go high-throughput event ingest + OCSF normalization-- `services/enrichment` — Go IOC enrichment with Redis caching-- `services/fusion` — Python alert deduplication + correlation-- `services/agents` — Python LangGraph AI agent orchestrator-- `services/actions` — Python SOAR action execution service-- `services/realtime` — Node.js WebSocket/SSE real-time service-- `services/connectors` — Python multi-connector service--### Frontend-- `apps/web` — Next.js 14 SOC console- - Dashboard with live metrics and charts- - Alert management with filtering and AI investigation- - Case management- - Threat intelligence with IOC lookup- - Connector management- - Threat hunting (KQL, Sigma, YARA)+## Services++### Backend services++| Service | Language | Port | Notes |+|---------|----------|------|-------|+| `services/api` | Python/FastAPI | 8000 | REST + WebSocket, RBAC, cases, Neo4j graph, rule engine |+| `services/ingest` | Go 1.21 | 9090 | OCSF normalize, ATT&CK tag, Shodan + KEV correlation |+| `services/enrichment` | Go 1.21 | 8080 | IOC enrichment, Redis-cached |+| `services/fusion` | Python 3.11 | 8003 | Dedup + correlation + ML scoring (anomaly + ranker) |+| `services/agents` | Python 3.11 | 8001 | LangGraph multi-agent investigations w/ full ATT&CK + Qdrant RAG |+| `services/actions` | Python 3.11 | 8002 | SOAR action executor with blast-radius gate |+| `services/threatintel` | Python 3.11 | 8005 | TAXII / MISP / OTX / KEV poller + IOC store |+| `services/realtime` | Node.js 20 | 4000 | WebSocket/SSE fan-out |+| `apps/web` | Next.js 14 | 3000 | SOC console |### Connectors (Phase 1)-- CrowdStrike Falcon-- Splunk Enterprise/Cloud-- AWS Security Hub-- Okta Identity-- Microsoft Sentinel-### Shared Packages-- `packages/types` — TypeScript type definitions-- `packages/ui` — React UI component library-- `packages/ocsf` — OCSF schema normalization+* CrowdStrike Falcon+* Splunk Enterprise/Cloud+* AWS Security Hub+* Okta Identity+* Microsoft Sentinel++### Shared packages++* `packages/types` — TypeScript type definitions+* `packages/ui` — React UI component library+* `packages/ocsf` — OCSF schema normalization++### Data layer++| Store | Purpose |+|-------|---------|+| PostgreSQL | App config, RBAC, cases, detection rules (RLS-isolated) |+| ClickHouse | Event analytics, alert metrics |+| OpenSearch | IOC, actor, threat-report search; Sigma backend |+| Qdrant | Vector RAG for agents + threat intel |+| Neo4j | Knowledge graph: entities, attack paths, blast radius |+| Redis | Cache, pub/sub, IOC bloom filter |+| Kafka | Event streaming bus |### Infrastructure-- `infra/terraform` — AWS infrastructure (VPC, EKS, RDS, ElastiCache, MSK)-- `infra/helm/aisoc` — Kubernetes Helm chart-- `docker-compose.yml` — Full development stack++* `infra/terraform` — AWS (VPC, EKS, RDS, ElastiCache, MSK)+* `infra/helm/aisoc` — Kubernetes Helm chart+* `docker-compose.yml` — Full local stack### Documentation-- `README.md` — Project overview, quick start, architecture diagram-- `CONTRIBUTING.md` — Contribution guidelines-- `LICENSE` — MIT License-- `.env.example` — Environment variable reference++* `README.md` — Project overview, quick start, architecture diagram+* `docs/architecture/SYSTEM_DESIGN.md` — Service topology, knowledge graph, ML fusion, threat intel pipeline+* `docs/api/API_REFERENCE.md` — REST endpoints (graph, rules, IOC search, ML)+* `docs/runbooks/LOCAL_DEVELOPMENT.md` — Step-by-step local launch + smoke test+* `CONTRIBUTING.md` — Contribution guidelines+* `LICENSE` — MIT+* `.env.example` — Environment variable referencediff --git a/README.md b/README.mdindex 0ce370009..2a759c29a 100644--- a/README.md+++ b/README.md@@ -39,12 +39,44 @@ Built and open-sourced by **Cyble** under the MIT License.- Okta Identity- Microsoft Sentinel+### 🧠 Knowledge Graph (Neo4j)+- **Entity Graph** — Hosts, Users, Alerts, IOCs, Techniques, Cases connected as a property graph+- **Attack Path Reconstruction** — `/v1/graph/attack-path/{case_id}` returns the full kill-chain+- **Blast Radius Analysis** — Multi-hop traversal feeds the action executor's gating logic+- **MITRE Coverage** — Automatic mapping of telemetry to ATT&CK technique nodes++### 🎯 Detection Engineering+- **Sigma Rule Execution** — `pySigma` runner with OpenSearch/ClickHouse backends+- **YARA Scanning** — `yara-python` runner for file/memory artifact analysis+- **Multi-language Support** — KQL, EQL, Lucene, Regex query types+- **On-demand Hunts** — `POST /v1/rules/hunt` for ad-hoc threat hunting++### 🌐 Threat Intelligence Platform+- **TAXII 2.1 Feed Poller** — Ingests STIX 2.1 bundles from MITRE, Mandiant, ISAC feeds+- **MISP Integration** — Pulls events and IOCs from any MISP instance+- **AlienVault OTX** — Subscribed pulse consumption+- **CISA KEV Catalog** — Auto-correlates known-exploited CVEs against asset telemetry+- **Bloom-filter Dedup** — Redis-backed; handles 10M+ IOCs with ~0.1% false-positive rate+- **Triple Storage** — OpenSearch (search), Qdrant (semantic RAG), Neo4j (actor↔TTP graph)++### 🤖 ML-Augmented Fusion+- **Isolation Forest** — Unsupervised anomaly score per alert (auto-trained at 50+ samples)+- **LightGBM Ranker** — Priority score trained on analyst feedback (LambdaRank objective)+- **Feedback Loop** — `POST /ml/feedback` continuously improves the ranker+- **Heuristic Fallback** — Always-on scoring even before models are trained++### 🛰️ Vulnerability Correlation+- **Shodan Enrichment** — Open ports, banners, ASN, CVEs per public IP (Go enrichment service)+- **CISA KEV Cross-correlation** — Emits `VULNERABILITY_MATCH` Kafka events for KEV-listed CVEs+- **Asset Context** — Links exposure data to Host nodes in the knowledge graph+### 📊 SOC Console- **Real-time Dashboard** — Live metrics, trend charts, and threat feeds- **Alert Management** — Triage, investigate, and respond from a single pane- **Case Management** — Full incident lifecycle management- **Threat Intelligence** — IOC lookup, threat feed correlation- **Threat Hunting** — KQL, Sigma, and YARA query execution+- **Attack Graph View** — Visual kill-chain reconstruction from Neo4j### 🛡️ Enterprise Ready- **Multi-tenant** — Complete tenant isolation with RBAC@@ -94,15 +126,28 @@ Built and open-sourced by **Cyble** under the MIT License.| Service | Language | Port | Description ||---------|----------|------|-------------|-| `core-api` | Python/FastAPI | 8000 | REST API — alerts, cases, tenants, RBAC |-| `ingest` | Go | 9090 | High-throughput event ingestion + OCSF normalization |-| `enrichment` | Go | 8080 | IOC enrichment via VirusTotal, AbuseIPDB, GreyNoise |-| `alert-fusion` | Python | — | Alert deduplication, correlation, incident grouping |-| `agents` | Python/LangGraph | 8001 | AI investigation, threat hunting, remediation agents |+| `api` | Python/FastAPI | 8000 | Core REST API — alerts, cases, tenants, RBAC, graph, rules |+| `ingest` | Go | 9090 | High-throughput event ingestion, OCSF normalization, Shodan + CVE correlation |+| `enrichment` | Go | 8080 | IOC enrichment via VirusTotal, AbuseIPDB, GreyNoise, Shodan |+| `fusion` | Python | 8003 | Alert dedup, correlation, ML anomaly + priority scoring |+| `agents` | Python/LangGraph | 8001 | AI investigation agents with full MITRE ATT&CK + Qdrant RAG || `actions` | Python | 8002 | SOAR action execution with blast-radius gating |+| `threatintel` | Python | 8005 | TAXII 2.1 / MISP / OTX / CISA KEV feed poller and IOC store || `realtime` | Node.js | 4000 | WebSocket/SSE real-time event delivery || `web` | Next.js 14 | 3000 | SOC console frontend |+### Data Layer++| Store | Purpose |+|-------|---------|+| **PostgreSQL** | App config, tenants, users, cases, detection rules, RLS-isolated |+| **ClickHouse** | High-cardinality event analytics, alert metrics, IOC enrichment cache |+| **OpenSearch** | Full-text IOC, actor, and threat report search; Sigma backend |+| **Qdrant** | Vector RAG for AI agents, semantic IOC and ATT&CK technique search |+| **Neo4j** | Knowledge graph: entity relationships, attack paths, blast radius |+| **Redis** | Cache, pub/sub, IOC bloom filter, enrichment TTL cache |+| **Kafka** | Event streaming bus (raw events, fused alerts, vulnerability matches) |+---## Quick Start@@ -117,21 +162,26 @@ Built and open-sourced by **Cyble** under the MIT License.### 1. Clone and Configure```bash-git clone https://github.com/cyble/aisoc.git-cd aisoc-cp .env.example .env+git clone https://github.com/beenuar/AiSOC.git+cd AiSOC+cp .env.example .env # if .env.example is missing, see Configuration below```Edit `.env` with your configuration:```env-# Required for AI agents+# Required for AI agents (use Anthropic OR OpenAI)+ANTHROPIC_API_KEY=sk-ant-...OPENAI_API_KEY=sk-...# Optional: threat intelligence enrichmentVIRUSTOTAL_API_KEY=...ABUSEIPDB_API_KEY=...GREYNOISE_API_KEY=...+SHODAN_API_KEY=...++# Optional: TAXII 2.1 feeds (comma-separated URL,collection,user,pass tuples)+TAXII_FEEDS=https://cti-taxii.mitre.org/taxii/,enterprise-attack,,```### 2. Start the Stack@@ -140,18 +190,29 @@ GREYNOISE_API_KEY=...# Start core infrastructure + all servicesdocker compose up -d-# Wait for services to be ready+# Wait for services to be ready (~60s for first start)docker compose ps-# View logs-docker compose logs -f core-api+# View logs from a specific service+docker compose logs -f api+docker compose logs -f fusion+docker compose logs -f agents```### 3. Access the Console-Open [http://localhost:3000](http://localhost:3000) in your browser.+| Surface | URL | Notes |+|---------|-----|-------|+| Web Console | http://localhost:3000 | Next.js frontend |+| API Swagger | http://localhost:8000/docs | Interactive API explorer |+| Agents API | http://localhost:8001/docs | LangGraph agent runner |+| Fusion API | http://localhost:8003/docs | ML status + feedback endpoints |+| Threat Intel | http://localhost:8005/docs | IOC search + feed status |+| Neo4j Browser | http://localhost:7474 | `neo4j` / `neo4j_dev_secret` |+| Grafana | http://localhost:3001 | `admin` / `admin` |+| Jaeger | http://localhost:16686 | Distributed traces |-Default credentials: `[email protected]` / `changeme`+Default API credentials: `[email protected]` / `changeme`### 4. Run with Connectors@@ -174,19 +235,15 @@ aisoc/├── apps/│ └── web/ # Next.js 14 SOC console├── services/-│ ├── core-api/ # Python FastAPI — core REST API-│ ├── ingest/ # Go — high-throughput event ingestion+│ ├── api/ # Python FastAPI — core REST API + Neo4j graph + rule engine+│ ├── ingest/ # Go — event ingestion, OCSF, Shodan + CVE correlation│ ├── enrichment/ # Go — IOC enrichment-│ ├── alert-fusion/ # Python — alert dedup + correlation-│ ├── agents/ # Python LangGraph — AI agents+│ ├── fusion/ # Python — alert dedup, correlation, ML scoring+│ ├── agents/ # Python LangGraph — AI agents w/ full ATT&CK + Qdrant RAG│ ├── actions/ # Python — SOAR action execution+│ ├── threatintel/ # Python — TAXII/MISP/OTX/KEV feed poller│ └── realtime/ # Node.js — WebSocket/SSE-├── connectors/-│ ├── crowdstrike/ # CrowdStrike Falcon connector-│ ├── splunk/ # Splunk Enterprise/Cloud connector-│ ├── aws/ # AWS Security Hub connector-│ ├── okta/ # Okta Identity connector-│ └── sentinel/ # Microsoft Sentinel connector+├── integrations/ # Connector implementations (CrowdStrike, Splunk, AWS, Okta, Sentinel, ...)├── packages/│ ├── types/ # Shared TypeScript types│ ├── ui/ # Shared React components@@ -194,6 +251,10 @@ aisoc/├── infra/│ ├── terraform/ # AWS infrastructure (VPC, EKS, RDS)│ └── helm/ # Kubernetes Helm chart+├── docs/+│ ├── architecture/ # System design, data flows+│ ├── api/ # API reference+│ └── runbooks/ # Operations guides└── docker-compose.yml```@@ -209,16 +270,26 @@ pnpm dev```bash# Start infrastructure only-docker compose up -d postgres redis kafka clickhouse opensearch qdrant+docker compose up -d postgres redis kafka clickhouse opensearch qdrant neo4j# Run core API-cd services/core-api+cd services/apipoetry installpoetry run uvicorn app.main:app --reload --port 8000+# Run fusion worker (with ML scoring)+cd services/fusion+poetry install+poetry run uvicorn app.main:app --reload --port 8003+# Run ingest workercd services/ingest-go run cmd/worker/main.go+go run main.go++# Run threat intel feed poller+cd services/threatintel+poetry install+poetry run uvicorn app.main:app --reload --port 8005```### Running Tests@@ -228,7 +299,7 @@ go run cmd/worker/main.gocd apps/web && pnpm test# Core API-cd services/core-api && poetry run pytest+cd services/api && poetry run pytest# Go servicescd services/ingest && go test ./...@@ -287,6 +358,11 @@ Once running, API docs are available at:- ReDoc: [http://localhost:8000/redoc](http://localhost:8000/redoc)- OpenAPI JSON: [http://localhost:8000/openapi.json](http://localhost:8000/openapi.json)+For detailed API references, system design, and runbooks see:+- [API Reference](docs/api/API_REFERENCE.md)+- [System Design](docs/architecture/SYSTEM_DESIGN.md)+- [Local Development Runbook](docs/runbooks/LOCAL_DEVELOPMENT.md)+---## Contributingdiff --git a/docker-compose.yml b/docker-compose.ymlindex 072631539..05c9dda35 100644--- a/docker-compose.yml+++ b/docker-compose.yml@@ -17,6 +17,8 @@ volumes:clickhouse_data:opensearch_data:qdrant_data:+ neo4j_data:+ neo4j_logs:services:@@ -33,7 +35,7 @@ services:- "5432:5432"volumes:- postgres_data:/var/lib/postgresql/data- - ./services/core-api/migrations:/docker-entrypoint-initdb.d:ro+ - ./services/api/migrations:/docker-entrypoint-initdb.d:ronetworks:- aisochealthcheck:@@ -121,7 +123,7 @@ services:- "9000:9000"volumes:- clickhouse_data:/var/lib/clickhouse- - ./services/core-api/clickhouse:/docker-entrypoint-initdb.d:ro+ - ./services/api/clickhouse:/docker-entrypoint-initdb.d:roenvironment:CLICKHOUSE_DB: aisocCLICKHOUSE_USER: aisoc@@ -166,13 +168,36 @@ services:networks:- aisoc+ neo4j:+ image: neo4j:5.15-community+ container_name: aisoc-neo4j+ ports:+ - "7474:7474" # HTTP browser+ - "7687:7687" # Bolt+ environment:+ NEO4J_AUTH: neo4j/neo4j_dev_secret+ NEO4J_PLUGINS: '["apoc"]'+ NEO4J_dbms_security_procedures_unrestricted: apoc.*+ NEO4J_dbms_memory_heap_initial__size: 256m+ NEO4J_dbms_memory_heap_max__size: 512m+ volumes:+ - neo4j_data:/data+ - neo4j_logs:/logs+ networks:+ - aisoc+ healthcheck:+ test: ["CMD-SHELL", "wget -qO- http://localhost:7474 || exit 1"]+ interval: 15s+ timeout: 10s+ retries: 10+# ─── Application Services ────────────────────────────────────────────────────- core-api:+ api:build:- context: ./services/core-api+ context: ./services/apidockerfile: Dockerfile- container_name: aisoc-core-api+ container_name: aisoc-apidepends_on:postgres:condition: service_healthy@@ -189,6 +214,9 @@ services:CLICKHOUSE_URL: http://aisoc:clickhouse_dev_secret@clickhouse:8123/aisocOPENSEARCH_URL: http://opensearch:9200QDRANT_URL: http://qdrant:6333+ NEO4J_URI: bolt://neo4j:7687+ NEO4J_USER: neo4j+ NEO4J_PASSWORD: neo4j_dev_secretSECRET_KEY: dev_secret_key_change_in_productionENVIRONMENT: developmentLOG_LEVEL: info@@ -210,6 +238,7 @@ services:KAFKA_BOOTSTRAP_SERVERS: kafka:29092REDIS_URL: redis://:redis_dev_secret@redis:6379/1LOG_LEVEL: info+ ATTCK_DATA_PATH: /app/data/enterprise-attack.jsonnetworks:- aisocrestart: unless-stopped@@ -232,20 +261,23 @@ services:VIRUSTOTAL_API_KEY: ${VIRUSTOTAL_API_KEY:-}ABUSEIPDB_API_KEY: ${ABUSEIPDB_API_KEY:-}GREYNOISE_API_KEY: ${GREYNOISE_API_KEY:-}+ SHODAN_API_KEY: ${SHODAN_API_KEY:-}networks:- aisocrestart: unless-stopped- alert-fusion:+ fusion:build:- context: ./services/alert-fusion+ context: ./services/fusiondockerfile: Dockerfile- container_name: aisoc-alert-fusion+ container_name: aisoc-fusiondepends_on:redis:condition: service_healthykafka:condition: service_healthy+ ports:+ - "8003:8003"environment:KAFKA_BOOTSTRAP_SERVERS: kafka:29092REDIS_URL: redis://:redis_dev_secret@redis:6379/3@@ -269,9 +301,15 @@ services:environment:DATABASE_URL: postgresql+asyncpg://aisoc:aisoc_dev_secret@postgres:5432/aisocREDIS_URL: redis://:redis_dev_secret@redis:6379/4- CORE_API_URL: http://core-api:8000+ CORE_API_URL: http://api:8000+ QDRANT_URL: http://qdrant:6333+ NEO4J_URI: bolt://neo4j:7687+ NEO4J_USER: neo4j+ NEO4J_PASSWORD: neo4j_dev_secretOPENAI_API_KEY: ${OPENAI_API_KEY:-}- OPENAI_MODEL: gpt-4-turbo-preview+ ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}+ OPENAI_MODEL: ${OPENAI_MODEL:-gpt-4o-mini}+ ATTCK_DATA_PATH: /app/data/enterprise-attack.jsonnetworks:- aisocrestart: unless-stopped@@ -292,89 +330,107 @@ services:KAFKA_BOOTSTRAP_SERVERS: kafka:29092REDIS_URL: redis://:redis_dev_secret@redis:6379/5DATABASE_URL: postgresql+asyncpg://aisoc:aisoc_dev_secret@postgres:5432/aisoc- CORE_API_URL: http://core-api:8000+ CORE_API_URL: http://api:8000networks:- aisocrestart: unless-stopped- realtime:+ connectors:build:- context: ./services/realtime+ context: ./services/connectorsdockerfile: Dockerfile- container_name: aisoc-realtime+ container_name: aisoc-connectorsdepends_on:- redis:- condition: service_healthykafka:condition: service_healthy+ redis:+ condition: service_healthyports:- - "4000:4000"+ - "8006:8003"+ profiles:+ - connectorsenvironment:KAFKA_BOOTSTRAP_SERVERS: kafka:29092- REDIS_URL: redis://:redis_dev_secret@redis:6379- REDIS_PASSWORD: redis_dev_secret- PORT: 4000+ REDIS_URL: redis://:redis_dev_secret@redis:6379/6+ DATABASE_URL: postgresql+asyncpg://aisoc:aisoc_dev_secret@postgres:5432/aisoc+ CORE_API_URL: http://api:8000+ CROWDSTRIKE_CLIENT_ID: ${CROWDSTRIKE_CLIENT_ID:-}+ CROWDSTRIKE_CLIENT_SECRET: ${CROWDSTRIKE_CLIENT_SECRET:-}+ AWS_REGION: ${AWS_REGION:-us-east-1}+ AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID:-}+ AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY:-}networks:- aisocrestart: unless-stopped- web:+ threatintel:build:- context: ./apps/web+ context: ./services/threatinteldockerfile: Dockerfile- container_name: aisoc-web+ container_name: aisoc-threatinteldepends_on:- - core-api- - realtime+ redis:+ condition: service_healthy+ kafka:+ condition: service_healthyports:- - "3000:3000"+ - "8005:8005"environment:- NEXT_PUBLIC_API_URL: http://localhost:8000- NEXT_PUBLIC_WS_URL: ws://localhost:4000- NODE_ENV: production+ KAFKA_BOOTSTRAP_SERVERS: kafka:29092+ REDIS_URL: redis://:redis_dev_secret@redis:6379/7+ OPENSEARCH_URL: http://opensearch:9200+ QDRANT_URL: http://qdrant:6333+ NEO4J_URI: bolt://neo4j:7687+ NEO4J_USER: neo4j+ NEO4J_PASSWORD: neo4j_dev_secret+ MISP_URL: ${MISP_URL:-}+ MISP_API_KEY: ${MISP_API_KEY:-}+ OTX_API_KEY: ${OTX_API_KEY:-}+ TAXII_URL: ${TAXII_URL:-}+ TAXII_USERNAME: ${TAXII_USERNAME:-}+ TAXII_PASSWORD: ${TAXII_PASSWORD:-}networks:- aisocrestart: unless-stopped- # ─── Connectors ──────────────────────────────────────────────────────────────-- connector-crowdstrike:+ realtime:build:- context: ./connectors/crowdstrike+ context: ./services/realtimedockerfile: Dockerfile- container_name: aisoc-connector-crowdstrike+ container_name: aisoc-realtimedepends_on:+ redis:+ condition: service_healthykafka:condition: service_healthy+ ports:+ - "4000:4000"environment:KAFKA_BOOTSTRAP_SERVERS: kafka:29092- CROWDSTRIKE_CLIENT_ID: ${CROWDSTRIKE_CLIENT_ID:-}- CROWDSTRIKE_CLIENT_SECRET: ${CROWDSTRIKE_CLIENT_SECRET:-}- POLL_INTERVAL: 60+ REDIS_URL: redis://:redis_dev_secret@redis:6379+ REDIS_PASSWORD: redis_dev_secret+ PORT: 4000networks:- aisocrestart: unless-stopped- profiles:- - connectors- connector-aws:+ web:build:- context: ./connectors/aws+ context: ./apps/webdockerfile: Dockerfile- container_name: aisoc-connector-aws+ container_name: aisoc-webdepends_on:- kafka:- condition: service_healthy+ - api+ - realtime+ ports:+ - "3000:3000"environment:- KAFKA_BOOTSTRAP_SERVERS: kafka:29092- AWS_REGION: ${AWS_REGION:-us-east-1}- AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID:-}- AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY:-}+ NEXT_PUBLIC_API_URL: http://localhost:8000+ NEXT_PUBLIC_WS_URL: ws://localhost:4000+ NODE_ENV: productionnetworks:- aisocrestart: unless-stopped- profiles:- - connectors# ─── Observability ────────────────────────────────────────────────────────────diff --git a/docs/api/API_REFERENCE.md b/docs/api/API_REFERENCE.mdnew file mode 100644index 000000000..a107490e3--- /dev/null+++ b/docs/api/API_REFERENCE.md@@ -0,0 +1,317 @@+# AiSOC API Reference++This document describes the REST endpoints introduced by the v2 enterprise upgrade. For the auto-generated, exhaustive schema visit `/docs` (Swagger) on each running service.++| Service | Base URL (local) |+|---------|-------------------|+| Core API | `http://localhost:8000` |+| Agents | `http://localhost:8001` |+| Actions | `http://localhost:8002` |+| Fusion | `http://localhost:8003` |+| Threat Intel | `http://localhost:8005` |++All examples assume:++```bash+export AISOC_TOKEN="$(curl -sX POST http://localhost:8000/v1/auth/login \+ -H 'content-type: application/json' \+ -d '{"email":"[email protected]","password":"changeme"}' | jq -r .access_token)"+export AISOC_TENANT="00000000-0000-0000-0000-000000000001"+```++---++## 1. Graph (Neo4j) — `/v1/graph`++Service: `services/api`.++### 1.1 `GET /v1/graph/attack-path/{case_id}`++Reconstructs the kill-chain for a case by traversing `(:Case)-[:CONTAINS]->(:Alert)-[:USES]->(:Technique)-[:PART_OF]->(:Tactic)`.++```bash+curl -H "authorization: Bearer $AISOC_TOKEN" \+ http://localhost:8000/v1/graph/attack-path/$CASE_ID+```++**Response**++```json+{+ "case_id": "…",+ "tactics": [+ { "id": "TA0001", "name": "Initial Access" },+ { "id": "TA0002", "name": "Execution" }+ ],+ "techniques": [+ { "id": "T1566", "name": "Phishing", "alert_id": "…" },+ { "id": "T1059.001", "name": "PowerShell", "alert_id": "…" }+ ]+}+```++### 1.2 `GET /v1/graph/blast-radius`++Returns 1-3 hop neighborhood of a node, used to gate high-impact actions.++| Query param | Required | Description |+|-------------|----------|-------------|+| `entity_type` | yes | One of `host`, `user`, `ioc` |+| `entity_id` | yes | Node identifier |+| `max_hops` | no (default `2`) | 1-3 |++```bash+curl -H "authorization: Bearer $AISOC_TOKEN" \+ "http://localhost:8000/v1/graph/blast-radius?entity_type=host&entity_id=HOST-42&max_hops=2"+```++**Response**++```json+{+ "root": { "type": "Host", "id": "HOST-42" },+ "nodes": 17,+ "edges": 24,+ "hosts": ["HOST-42", "HOST-71"],+ "users": ["alice@corp"],+ "iocs": ["1.2.3.4", "evil.tld"],+ "alerts": ["A-1", "A-7"]+}+```++### 1.3 `GET /v1/graph/neighbors`++1-hop neighborhood for the SOC console "context" panel.++### 1.4 `GET /v1/graph/mitre-coverage`++Aggregated counts of distinct techniques observed per tenant.++| Query param | Default | Description |+|-------------|---------|-------------|+| `window` | `7d` | `1h`, `24h`, `7d`, `30d` |++---++## 2. Detection Rules — `/v1/rules`++Service: `services/api`.++### 2.1 `GET /v1/rules`++| Query param | Description |+|-------------|-------------|+| `language` | `sigma` · `yara` · `kql` · `lucene` · `regex` |+| `enabled` | `true`/`false` |+| `severity` | `low`-`critical` |++### 2.2 `POST /v1/rules`++```json+{+ "name": "Suspicious PowerShell encoded command",+ "language": "sigma",+ "severity": "high",+ "rule": "title: Suspicious PowerShell\n…",+ "tags": ["attack.execution", "attack.t1059.001"],+ "enabled": true+}+```++### 2.3 `POST /v1/rules/{id}/execute`++Run a single rule on demand against the last `lookback` of telemetry.++```json+{+ "lookback": "1h",+ "indices": ["events-*"],+ "limit": 100+}+```++**Response**++```json+{+ "rule_id": "…",+ "matches": 7,+ "duration_ms": 138,+ "results": [+ { "event_id": "…", "host": "…", "user": "…", "ts": "…" }+ ]+}+```++### 2.4 `POST /v1/rules/hunt`++Multi-rule, time-bounded threat hunt.++```json+{+ "rule_ids": ["rule-1", "rule-2"],+ "from": "2026-04-25T00:00:00Z",+ "to": "2026-05-01T00:00:00Z",+ "limit_per_rule": 50+}+```++### 2.5 `PATCH /v1/rules/{id}` / `DELETE /v1/rules/{id}`++Standard CRUD with optimistic concurrency via `If-Match` ETag.++---++## 3. Threat Intel IOC Search — `/v1/iocs`++Service: `services/threatintel`.++### 3.1 `GET /v1/iocs/search`++| Query param | Description |+|-------------|-------------|+| `q` | Lexical query (OpenSearch) |+| `type` | `ip` · `domain` · `url` · `sha256` · `md5` |+| `actor` | Filter by named actor |+| `since` | ISO timestamp |++### 3.2 `POST /v1/iocs/semantic`++Vector similarity search against Qdrant.++```json+{+ "text": "powershell encoded base64 mshta DownloadString",+ "k": 10,+ "min_score": 0.6+}+```++### 3.3 `GET /v1/iocs/{value}`++Resolve a single indicator with all enrichment + actor links.++### 3.4 `GET /v1/feeds/status`++```json+{+ "feeds": [+ { "name": "mitre-taxii", "last_run": "…", "ioc_count": 12345 },+ { "name": "cisa-kev", "last_run": "…", "ioc_count": 1023 }+ ]+}+```++### 3.5 `POST /v1/feeds/{name}/poll`++Trigger an immediate poll (admin-only).++---++## 4. ML Fusion — `/ml`++Service: `services/fusion`.++### 4.1 `GET /ml/status`++```json+{+ "anomaly_model": {+ "trained": true,+ "samples": 482,+ "last_trained_at": "2026-04-30T12:00:00Z"+ },+ "ranker_model": {+ "trained": false,+ "feedback_buffer": 73,+ "feedback_required": 100+ }+}+```++### 4.2 `POST /ml/feedback`++Submitted by analysts when triaging an alert.++```json+{+ "alert_id": "…",+ "tenant_id": "…",+ "analyst_id": "alice@corp",+ "is_true_positive": true,+ "assigned_priority": 2,+ "notes": "Confirmed lateral movement"+}+```++### 4.3 `POST /ml/retrain`++Force a retrain. Returns the new model metadata.++```json+{ "status": "scheduled", "job_id": "…" }+```++---++## 5. Vulnerability Match Stream++Vulnerability matches are surfaced both via Kafka (`vulnerability.matches` topic) and the API:++### 5.1 `GET /v1/vulnerabilities`++Lists recent KEV-correlated matches with host context joined from Neo4j.++| Query param | Description |+|-------------|-------------|+| `cve` | Filter by CVE ID |+| `host_id` | Filter by host |+| `kev_only` | `true`/`false` (default `true`) |++---++## 6. Cases — `/v1/cases`++Unchanged from v1, but now joined with Neo4j attack paths via `GET /v1/cases/{id}/attack-path`.++---++## 7. Authentication++* JWT issued by `POST /v1/auth/login`.+* API keys via `Authorization: ApiKey <key>` header.+* All requests must specify a tenant context — either implicit (from JWT) or explicit (`X-Tenant-Id` header for service-to-service calls).++---++## 8. Errors++All endpoints return RFC 7807 Problem Details:++```json+{+ "type": "https://aisoc.dev/errors/rule-validation",+ "title": "Sigma rule failed validation",+ "status": 422,+ "detail": "Unknown field 'EventID' in selection 'sel_powershell'",+ "instance": "/v1/rules"+}+```++---++## 9. Rate Limits++| Tier | Requests/min |+|------|--------------|+| Default | 600 |+| `/v1/rules/hunt` | 30 |+| `/ml/retrain` | 6 |++Limits are tenant-scoped and enforced by Redis.++---++## 10. Versioning++The API follows semver via the URL prefix `/v1`. Breaking changes will move to `/v2` and the previous version remains supported for at least 6 months.diff --git a/docs/architecture/SYSTEM_DESIGN.md b/docs/architecture/SYSTEM_DESIGN.mdnew file mode 100644index 000000000..2ecfbf149--- /dev/null+++ b/docs/architecture/SYSTEM_DESIGN.md@@ -0,0 +1,288 @@+# AiSOC System Design++This document describes the end-to-end architecture of the AiSOC platform after the v2 enterprise upgrade. It covers data flow, the new knowledge graph, the detection rule engine, the threat intelligence pipeline, and the ML-augmented alert fusion pipeline.++---++## 1. High-level Topology++```+ ┌──────────────────────────────────────────────┐+ │ External Sources │+ │ EDR / SIEM / Cloud / Identity / Network │+ │ CrowdStrike · Splunk · AWS · Okta · Sentinel│+ └─────────────────────┬────────────────────────┘+ │+ Webhooks / Pull+ │+ ▼+ ┌──────────────────────────────────────────────────────┐+ │ services/ingest (Go) │+ │ ─ OCSF normalizer │+ │ ─ MITRE ATT&CK technique tagger (in-process index) │+ │ ─ Shodan enrichment (TTL cache) │+ │ ─ CISA KEV correlator → VULNERABILITY_MATCH events │+ └────────┬───────────────────────────┬─────────────────┘+ │ │+ ocsf.events vulnerability.matches+ │ │+ ▼ ▼+ ┌────────────────────┐ ┌────────────────────┐+ │ Apache Kafka │ │ Apache Kafka │+ └────────┬───────────┘ └─────────┬──────────┘+ │ │+ ▼ ▼+ ┌──────────────────────────────────────────────────┐+ │ services/fusion (Python) │+ │ ─ Simhash dedup (Redis) │+ │ ─ Entity + ATT&CK correlation │+ │ ─ MLScorer: Isolation Forest + LightGBM ranker │+ │ ─ Analyst feedback loop │+ └────────┬─────────────────────────────────────────┘+ │+ fused.alerts+ │+ ▼+ ┌──────────────────────────────────────────────────┐+ │ services/api (FastAPI) │+ │ ─ REST + WebSocket endpoints │+ │ ─ Detection rule engine (Sigma / YARA / KQL) │+ │ ─ Neo4j graph_service (entity + attack-path) │+ │ ─ Cases / RBAC / audit │+ └────┬───────────┬───────────────┬─────────────────┘+ │ │ │+ ▼ ▼ ▼+ ┌────────┐ ┌────────┐ ┌────────────────┐+ │PostgreSQL│ │Neo4j │ │OpenSearch + QDR│+ │ (state) │ │(graph)│ │ (search + RAG) │+ └──────────┘ └───────┘ └────────────────┘++ ┌──────────────────────────────────┐+ │ services/threatintel │+ │ TAXII 2.1 · MISP · OTX · KEV │+ │ → OpenSearch · Qdrant · Neo4j │+ └──────────────────────────────────┘++ ┌──────────────────────────────────┐+ │ services/agents │+ │ LangGraph + full ATT&CK + RAG │+ │ (Anthropic Claude 3.7 / GPT-4o) │+ └──────────────────────────────────┘+```++---++## 2. Service Responsibilities++| Service | Language | Hot path | Async path |+|---------|----------|----------|------------|+| `services/ingest` | Go 1.21 | Normalize raw events → OCSF, tag ATT&CK, enrich Shodan | Emit `vulnerability.matches` topic |+| `services/enrichment` | Go 1.21 | Per-IOC TTL-cached enrichment lookups | n/a |+| `services/fusion` | Python 3.11 | Simhash dedup → correlation → ML scoring → publish fused alerts | Background ML retrain on feedback |+| `services/api` | Python 3.11 | REST + WS, RBAC, case mgmt, rule engine, graph queries | Schema migrations on boot |+| `services/agents` | Python 3.11 | LangGraph multi-agent investigation runs | Loads full ATT&CK STIX bundle on boot, optional Qdrant embed |+| `services/actions` | Python 3.11 | SOAR action execution with blast-radius gating | Approval workflows |+| `services/threatintel` | Python 3.11 | IOC/actor search API | APScheduler poll loop for TAXII/MISP/OTX/KEV |+| `services/realtime` | Node 20 | WebSocket fan-out for the web console | n/a |+| `apps/web` | Next.js 14 | Server components + SWR | n/a |++---++## 3. Knowledge Graph (Neo4j)++The knowledge graph is the connective tissue between alerts, entities, and the MITRE ATT&CK matrix.++### 3.1 Node Labels++| Label | Key | Purpose |+|-------|-----|---------|+| `Host` | `id` (host UUID or hostname) | Endpoint context |+| `User` | `id` (sub or upn) | Identity context |+| `Alert` | `id` (alert UUID) | Detected event |+| `Case` | `id` (case UUID) | Triage container |+| `IOC` | `value` (sha256/ip/domain) | Atomic indicators |+| `Technique` | `id` (e.g. `T1059.001`) | MITRE ATT&CK |+| `Tactic` | `id` (e.g. `TA0002`) | MITRE ATT&CK |+| `ThreatActor` | `id` | Attribution |++### 3.2 Relationships++```+(Host)-[:LOGGED_IN]->(User)+(User)-[:OWNS]->(Host)+(Alert)-[:ON_HOST]->(Host)+(Alert)-[:AFFECTS]->(User)+(Alert)-[:USES]->(Technique)+(Technique)-[:PART_OF]->(Tactic)+(IOC)-[:OBSERVED_IN]->(Alert)+(ThreatActor)-[:USES]->(Technique)+(ThreatActor)-[:OBSERVED_AS]->(IOC)+(Case)-[:CONTAINS]->(Alert)+```++### 3.3 Critical Queries++* **Attack path** (`get_attack_path`): given a `Case`, traverse `(:Case)-[:CONTAINS]->(:Alert)-[:USES]->(:Technique)-[:PART_OF]->(:Tactic)` ordered by tactic kill-chain index.+* **Blast radius** (`get_blast_radius`): variable-length path query from any entity, capped at 3 hops, filtered by relationship type, used by the `actions` service's gating layer.+* **Entity neighbors**: 1-hop graph used by the SOC console "show context" panel.+* **MITRE coverage**: aggregated counts of distinct techniques observed per tenant per time window.++### 3.4 Bootstrapping++`services/api/app/db/neo4j.py` initializes a singleton `AsyncDriver`, verifies connectivity, and creates uniqueness constraints on `(Host.id)`, `(User.id)`, `(Alert.id)`, `(Case.id)`, `(IOC.value)`, `(Technique.id)` at startup.++---++## 4. Detection Rule Engine++`services/api/app/services/rule_engine.py` implements multi-language rule execution.++| Rule type | Backend | Notes |+|-----------|---------|-------|+| Sigma | `pySigma` → OpenSearch query | Most common path; supports field mappings |+| YARA | `yara-python` | Targets file/memory artifacts |+| KQL | Translated to ClickHouse SQL | For event analytics |+| Lucene / Regex | Direct OpenSearch | For full-text patterns |++### 4.1 Endpoints++| Method | Path | Purpose |+|--------|------|---------|+| `GET /v1/rules` | List rules with filtering | |+| `POST /v1/rules` | Create a new rule | |+| `POST /v1/rules/{id}/execute` | Execute single rule on demand | |+| `POST /v1/rules/hunt` | Multi-rule, time-bounded hunt | |+| `PATCH /v1/rules/{id}` | Update rule (validate, version) | |++### 4.2 Lifecycle++1. Rule authored in the web console IDE (Monaco editor with live test panel).+2. Validated against a sample event; on save, written to `detection_rules` table.+3. Scheduler service evaluates `enabled=true` rules at their cadence.+4. Hits emit fused alert candidates onto Kafka via the `fusion` pipeline.++---++## 5. Threat Intelligence Pipeline++`services/threatintel` runs an APScheduler-driven poll loop with per-feed handlers.++```+TAXII 2.1 ─┐+MISP ──────┤ ┌──────────────────────────┐ ┌────────────┐+OTX ──────┼──▶│ ThreatIntelPipeline │──▶│ OpenSearch │+CISA KEV ──┘ │ ─ STIX 2.1 parser │ │ Qdrant │+ │ ─ Redis bloom dedup │ │ Neo4j │+ └──────────────────────────┘ └────────────┘+```++### 5.1 Components++| Module | Role |+|--------|------|+| `clients/taxii.py` | Async TAXII 2.1 client |+| `clients/misp.py` | MISP REST client |+| `clients/otx.py` | AlienVault OTX client |+| `clients/cisa_kev.py` | KEV catalog fetcher |+| `parsers/stix.py` | Normalize STIX 2.1 SDO/SRO objects |+| `storage/bloom.py` | Redis-backed Bloom filter (configurable size + FP rate) |+| `storage/opensearch.py` | Index `iocs-*` and `actors-*` |+| `storage/qdrant.py` | Embed via `text-embedding-3-large` for semantic recall |+| `storage/neo4j.py` | Persist actor↔TTP and actor↔IOC graph |+| `feeds/scheduler.py` | APScheduler interval jobs |+| `feeds/pipeline.py` | Fan-out to all sinks with idempotency |++### 5.2 Configuration++All feeds toggle via env vars (see [`docs/runbooks/LOCAL_DEVELOPMENT.md`](../runbooks/LOCAL_DEVELOPMENT.md)).++---++## 6. ML-Augmented Fusion++`services/fusion/app/services/ml_scorer.py` adds two scoring dimensions to every fused alert.++| Score | Model | Trigger | Cold-start |+|-------|-------|---------|-----------|+| `anomaly_score` | Isolation Forest (sklearn) | Auto-train at ≥50 alerts | Heuristic on event rarity, hour-of-day, severity |+| `priority_score` | LightGBM `LambdaRank` | Trains when ≥100 analyst feedback rows | Heuristic blending severity, anomaly score, asset criticality |++### 6.1 Feature Vector (`_featurize`)++* Numerical: `severity` (1-5), `asset_criticality` (0-1), `hour_of_day` (cyclical), `dedup_count`, `anomaly_score` (when ranking)+* Categorical: top-N MITRE tactic, source product, alert kind (one-hot)++### 6.2 Feedback Loop++1. Analyst submits `POST /ml/feedback` with `is_true_positive` and `assigned_priority` 1-5.+2. Fusion buffers feedback; once buffer ≥100, schedules a background retrain.+3. New model is hot-swapped behind a `RWLock`; old model is retained until validation.+4. `GET /ml/status` exposes counts, last-trained timestamp, buffer size.++---++## 7. AI Agents (LangGraph)++`services/agents` boots with the **full** MITRE ATT&CK STIX bundle and (optionally) embeds it into Qdrant for semantic retrieval.++### 7.1 Agent DAG++```+PlannerAgent+ │+ ├─▶ EndpointAgent (CrowdStrike, Velociraptor, GRR)+ ├─▶ IdentityAgent (Okta, Azure AD, Sentinel)+ ├─▶ NetworkAgent (Zeek, NetFlow, Darktrace)+ ├─▶ CloudAgent (AWS, Azure, GCP)+ ├─▶ EmailAgent (M365 Defender, Gmail)+ ├─▶ ThreatIntelAgent (calls services/threatintel)+ └─▶ VulnerabilityAgent(reads VULNERABILITY_MATCH stream)+ │+ ▼+ SynthesisAgent ──▶ ActionAgent (with blast-radius gate)+```++Each agent emits chain-of-thought traces persisted in PostgreSQL for explainability.++### 7.2 RAG++* **Threat reports**: indexed by the threat intel service in Qdrant+* **MITRE corpus**: technique descriptions embedded on agent boot (toggle via `ATTCK_QDRANT_EMBED=true`)+* **Internal runbooks**: ingested via `POST /v1/agents/runbooks`++---++## 8. Multi-tenancy & Security++* **Tenant isolation** — All Postgres tables include `tenant_id` and are protected by RLS policies.+* **Neo4j scoping** — Every node carries `tenant_id` and queries use `WHERE n.tenant_id = $tid`.+* **Auth** — JWT + API keys; per-tenant scope checks live in `app/api/v1/deps.py`.+* **Audit log** — Every mutating endpoint emits an audit entry to `audit_log` (Postgres) and Kafka.+* **Action gating** — Blast-radius traversal in Neo4j must return ≤ configured cap before high-impact actions execute.++---++## 9. Observability++* **Metrics** — Prometheus `/metrics` on every service; Grafana dashboards under `infra/grafana/`.+* **Tracing** — OpenTelemetry SDK in Python services; Jaeger for visualization.+* **Logging** — `structlog` (Python) + `log/slog` (Go) in JSON for ELK ingestion.++---++## 10. Deployment Topologies++| Environment | Recommended |+|-------------|-------------|+| Dev / demo | `docker compose up` (this repo) |+| Single-cluster prod | Helm chart at `infra/helm/aisoc` |+| Multi-region | Terraform `infra/terraform` + EKS + cross-region MSK |+| Air-gapped | Swap LLM to Ollama-compatible endpoint (`OPENAI_BASE_URL`) |++---++## 11. References++* [API Reference](../api/API_REFERENCE.md)+* [Local Development](../runbooks/LOCAL_DEVELOPMENT.md)+* [PROGRESS.md](../../PROGRESS.md)diff --git a/docs/runbooks/LOCAL_DEVELOPMENT.md b/docs/runbooks/LOCAL_DEVELOPMENT.mdnew file mode 100644index 000000000..f367e69c8--- /dev/null+++ b/docs/runbooks/LOCAL_DEVELOPMENT.md@@ -0,0 +1,357 @@+# Local Development Runbook++This runbook walks you from a freshly cloned repository to a running AiSOC stack on your laptop, and describes how to develop and debug each service in isolation.++---++## 1. Prerequisites++| Tool | Min version | Why |+|------|-------------|-----|+| Docker | 24.0 | Container runtime for the full stack |+| Docker Compose | v2.20 | Multi-service orchestration |+| Node.js | 20.x | Frontend (`apps/web`) and `services/realtime` |+| pnpm | 8.x | Workspace package manager |+| Go | 1.21 | `services/ingest`, `services/enrichment` |+| Python | 3.11 | All FastAPI services and the agent runner |+| Poetry | 1.8 | Python service dependency management |+| `gh` CLI | 2.40+ | Optional, used by some helper scripts |++Memory budget: the full stack uses **~6 GB RAM** with all services running. If you only need a subset, use the targeted commands in §4.++---++## 2. Clone & Configure++```bash+git clone https://github.com/beenuar/AiSOC.git+cd AiSOC+cp .env.example .env+```++Edit `.env`. The defaults work for a no-LLM local run, but for the AI agents you must populate at least one of:++```env+ANTHROPIC_API_KEY=sk-ant-...+OPENAI_API_KEY=sk-...+```++For full threat-intel enrichment also set:++```env+VIRUSTOTAL_API_KEY=...+ABUSEIPDB_API_KEY=...+GREYNOISE_API_KEY=...+SHODAN_API_KEY=...+OTX_API_KEY=...+TAXII_URL=https://cti-taxii.mitre.org/taxii/+```++---++## 3. Bring up the full stack++```bash+docker compose up -d --build+```++First boot takes 3–5 minutes (Postgres init, OpenSearch warmup, Kafka topic auto-create, Python image build). Subsequent boots are ~30 s.++```bash+# Service status+docker compose ps++# Tail a specific service+docker compose logs -f api+docker compose logs -f fusion+docker compose logs -f agents+docker compose logs -f threatintel+```++### 3.1 Verify health++| Service | Health check |+|---------|--------------|+| Core API | `curl http://localhost:8000/health` |+| Agents | `curl http://localhost:8001/health` |+| Actions | `curl http://localhost:8002/health` |+| Fusion | `curl http://localhost:8003/health` |+| Threat Intel | `curl http://localhost:8005/health` |+| Realtime | `curl http://localhost:4000/health` |+| Web Console | `curl -I http://localhost:3000` |+| Postgres | `docker exec aisoc-postgres pg_isready -U aisoc` |+| Redis | `docker exec aisoc-redis redis-cli -a redis_dev_secret ping` |+| Kafka | `docker exec aisoc-kafka kafka-broker-api-versions --bootstrap-server localhost:9092` |+| OpenSearch | `curl http://localhost:9200` |+| Qdrant | `curl http://localhost:6333/healthz` |+| Neo4j | open `http://localhost:7474` (login `neo4j` / `neo4j_dev_secret`) |++A first-time success of every endpoint is the gate before you proceed.++### 3.2 Optional profiles++```bash+# Pull TAXII / MISP / OTX / KEV feeds (already bundled in default compose)+docker compose --profile threatintel up -d++# Connector workers (CrowdStrike, Splunk, AWS, Okta, Sentinel)+docker compose --profile connectors up -d++# Prometheus + Grafana+docker compose --profile monitoring up -d+```++---++## 4. Run only the infrastructure (recommended for backend dev)++```bash+docker compose up -d \+ postgres redis kafka zookeeper \+ clickhouse opensearch qdrant neo4j+```++Then run individual services on the host where iteration is fastest.++### 4.1 Core API (`services/api`)++```bash+cd services/api+poetry install+poetry run alembic upgrade head # apply DB migrations+poetry run uvicorn app.main:app --reload --port 8000+```++Browse `http://localhost:8000/docs`.++### 4.2 Fusion (`services/fusion`)++```bash+cd services/fusion+poetry install+poetry run uvicorn app.main:app --reload --port 8003+```++Background Kafka worker boots automatically via FastAPI's `lifespan`.++### 4.3 Agents (`services/agents`)++```bash+cd services/agents+poetry install+ATTCK_DATA_PATH=$(pwd)/data/enterprise-attack.json \+poetry run uvicorn app.main:app --reload --port 8001+```++The first run downloads the MITRE ATT&CK STIX bundle (~12 MB) into `data/`.++### 4.4 Threat Intel (`services/threatintel`)++```bash+cd services/threatintel+poetry install+poetry run uvicorn app.main:app --reload --port 8005+```++The poller starts an APScheduler loop on boot and writes IOCs to OpenSearch + Qdrant + Neo4j.++### 4.5 Ingest (`services/ingest`, Go)++```bash+cd services/ingest+go run main.go+```++The Go ingester reads from Kafka topic `events.raw`, normalizes to OCSF, and emits to `ocsf.events` and `vulnerability.matches`.++### 4.6 Enrichment (`services/enrichment`, Go)++```bash+cd services/enrichment+go run main.go+```++### 4.7 Realtime (`services/realtime`, Node.js)++```bash+cd services/realtime+pnpm install+pnpm dev+```++WebSocket endpoint: `ws://localhost:4000/ws`.++### 4.8 Web console (`apps/web`)++```bash+cd apps/web+pnpm install+NEXT_PUBLIC_API_URL=http://localhost:8000 \+NEXT_PUBLIC_WS_URL=ws://localhost:4000 \+pnpm dev+```++Browse `http://localhost:3000`.++---++## 5. Smoke-test path (≈ 5 minutes)++```bash+# 1. Get a token+TOKEN=$(curl -s -X POST http://localhost:8000/v1/auth/login \+ -H 'content-type: application/json' \+ -d '{"email":"[email protected]","password":"changeme"}' | jq -r .access_token)++# 2. Send a synthetic event into Kafka+docker exec -i aisoc-kafka kafka-console-producer \+ --bootstrap-server localhost:9092 \+ --topic events.raw <<'JSON'+{"src":"crowdstrike","ts":"2026-05-01T12:00:00Z","host":{"hostname":"HOST-42"},"user":{"name":"alice@corp"},"event":{"id":"e1","name":"PowerShell encoded command","severity":4}}+JSON++# 3. Fusion should produce a fused alert; query the API+curl -s -H "authorization: Bearer $TOKEN" \+ http://localhost:8000/v1/alerts?limit=5 | jq++# 4. Inspect ML status+curl -s http://localhost:8003/ml/status | jq++# 5. Submit analyst feedback+curl -s -X POST http://localhost:8003/ml/feedback \+ -H 'content-type: application/json' \+ -d '{"alert_id":"REPLACE_ME","tenant_id":"00000000-0000-0000-0000-000000000001","analyst_id":"alice@corp","is_true_positive":true,"assigned_priority":2}'++# 6. Trigger the agent on the alert+curl -s -X POST http://localhost:8001/v1/agents/investigate \+ -H "authorization: Bearer $TOKEN" \+ -H 'content-type: application/json' \+ -d '{"alert_id":"REPLACE_ME"}' | jq++# 7. Inspect the attack-path graph+CASE_ID=$(curl -s -H "authorization: Bearer $TOKEN" http://localhost:8000/v1/cases | jq -r '.items[0].id')+curl -s -H "authorization: Bearer $TOKEN" \+ http://localhost:8000/v1/graph/attack-path/$CASE_ID | jq+```++---++## 6. Common operations++### 6.1 Reset the stack++```bash+docker compose down -v # also removes volumes (data loss!)+docker compose up -d --build+```++### 6.2 Rebuild a single image++```bash+docker compose build --no-cache fusion+docker compose up -d fusion+```++### 6.3 Tail Kafka topics++```bash+docker exec -it aisoc-kafka \+ kafka-console-consumer --bootstrap-server localhost:9092 \+ --topic ocsf.events --from-beginning --max-messages 10+```++Topics created by the stack:++* `events.raw` — raw connector events+* `ocsf.events` — normalized events+* `enriched.events` — IOC-enriched events+* `vulnerability.matches` — KEV-correlated matches+* `fused.alerts` — deduped, scored alerts+* `agent.investigations` — agent run results++### 6.4 Database access++```bash+# Postgres shell+docker exec -it aisoc-postgres psql -U aisoc++# ClickHouse shell+docker exec -it aisoc-clickhouse clickhouse-client -u aisoc --password clickhouse_dev_secret++# Neo4j cypher shell+docker exec -it aisoc-neo4j cypher-shell -u neo4j -p neo4j_dev_secret+```++### 6.5 Run database migrations++```bash+cd services/api+poetry run alembic upgrade head+```++---++## 7. Troubleshooting++| Symptom | Likely cause | Fix |+|---------|--------------|-----|+| `api` container restarts | Postgres not ready or migrations failed | `docker compose logs api` and run `alembic upgrade head` |+| `fusion` 500 on `/ml/feedback` | DB schema missing | Run migrations (above) |+| `agents` boot loop | No `ANTHROPIC_API_KEY` or `OPENAI_API_KEY` | Set one in `.env`, `docker compose up -d agents` |+| OpenSearch refuses connection | Memory ulimit | Run `sysctl -w vm.max_map_count=262144` (Linux) |+| Kafka cannot resolve `kafka:29092` | Stale DNS / restart loop | `docker compose restart kafka` |+| Neo4j unreachable from API | Wrong env var | Confirm `NEO4J_URI=bolt://neo4j:7687` |+| Web console blank | API CORS / ports | Verify `NEXT_PUBLIC_API_URL=http://localhost:8000` |+| `port already in use` | Another process owns 8000/3000/etc. | `lsof -i :8000` and stop the conflicting service |++### 7.1 Inspect a service's environment++```bash+docker compose exec api env | grep -E 'NEO4J|REDIS|KAFKA|DATABASE'+```++### 7.2 Re-seed sample data++```bash+docker compose exec api python -m app.scripts.seed_demo+```++---++## 8. Tests++```bash+# Frontend+cd apps/web && pnpm test++# Core API+cd services/api && poetry run pytest++# Fusion (incl. ML scorer)+cd services/fusion && poetry run pytest++# Threat Intel+cd services/threatintel && poetry run pytest++# Go services+cd services/ingest && go test ./...+cd services/enrichment && go test ./...+```++---++## 9. Tearing down++```bash+docker compose down # keeps volumes+docker compose down -v # removes volumes + data+docker system prune -f # reclaim build cache+```++---++## 10. Where to go next++* [System Design](../architecture/SYSTEM_DESIGN.md)+* [API Reference](../api/API_REFERENCE.md)+* [PROGRESS.md](../../PROGRESS.md)diff --git a/services/agents/app/main.py b/services/agents/app/main.pyindex d764bed9d..55cd82c82 100644--- a/services/agents/app/main.py+++ b/services/agents/app/main.py@@ -1,13 +1,45 @@+import os+from contextlib import asynccontextmanager+from typing import AsyncGenerator+import structlogfrom fastapi import FastAPI+from app.api.router import router+from app.tools.mitre_full import load_attck_corpus, embed_techniques_into_qdrantlogger = structlog.get_logger()++@asynccontextmanager+async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:+ """Load resources on startup."""+ # Load full MITRE ATT&CK corpus+ try:+ await load_attck_corpus()+ except Exception as exc:+ logger.warning("MITRE ATT&CK corpus load failed at startup", error=str(exc))++ # Embed into Qdrant for RAG (only if configured)+ qdrant_url = os.getenv("QDRANT_URL", "")+ openai_key = os.getenv("OPENAI_API_KEY", "")+ if qdrant_url and openai_key:+ try:+ await embed_techniques_into_qdrant(+ qdrant_url=qdrant_url,+ openai_api_key=openai_key,+ )+ except Exception as exc:+ logger.warning("ATT&CK Qdrant embedding skipped", error=str(exc))++ yield++app = FastAPI(title="AiSOC Agent Orchestrator",description="LangGraph-based autonomous investigation and response agents",version="0.1.0",+ lifespan=lifespan,)app.include_router(router, prefix="/api/v1")@@ -15,4 +47,10 @@ app.include_router(router, prefix="/api/v1")@app.get("/health")async def health():- return {"status": "healthy", "service": "aisoc-agents"}+ from app.tools.mitre_full import get_coverage_summary+ summary = get_coverage_summary()+ return {+ "status": "healthy",+ "service": "aisoc-agents",+ "attck_corpus": summary,+ }diff --git a/services/agents/app/tools/mitre_full.py b/services/agents/app/tools/mitre_full.pynew file mode 100644index 000000000..83b6ed609--- /dev/null+++ b/services/agents/app/tools/mitre_full.py@@ -0,0 +1,526 @@+"""+MITRE ATT&CK Full Corpus Loader+================================+Loads the complete MITRE ATT&CK Enterprise STIX 2.1 bundle and provides+comprehensive technique/actor/mitigation lookups.++On startup, this module:+1. Tries to load the bundle from ATTCK_DATA_PATH (local file).+2. Falls back to downloading the bundle from MITRE's GitHub CDN.+3. Builds in-memory indexes for O(1) lookups.+4. Optionally embeds technique descriptions into Qdrant for semantic RAG.++Cyble Open-Source AI Security Operations Center — MIT License+"""+from __future__ import annotations++import asyncio+import json+import logging+import os+import pathlib+import time+from typing import Any++import httpx++logger = logging.getLogger(__name__)++# ─── Constants ────────────────────────────────────────────────────────────────++ATTCK_CDN_URL = (+ "https://raw.githubusercontent.com/mitre/cti/master/"+ "enterprise-attack/enterprise-attack.json"+)+ATTCK_DATA_PATH = os.getenv("ATTCK_DATA_PATH", "/data/enterprise-attack.json")+_CACHE_TTL_HOURS = 24+_EMBED_BATCH_SIZE = 50+++# ─── Data Model ───────────────────────────────────────────────────────────────++class MitreTechnique:+ """Represents a single ATT&CK technique or sub-technique."""++ __slots__ = (+ "id", "name", "description", "tactic_ids", "tactic_names",+ "platforms", "data_sources", "mitigations", "detections",+ "is_subtechnique", "parent_id", "url", "version",+ )++ def __init__(self, **kw: Any) -> None:+ for slot in self.__slots__:+ setattr(self, slot, kw.get(slot))++ def to_dict(self) -> dict[str, Any]:+ return {s: getattr(self, s) for s in self.__slots__}+++class MitreActor:+ """Represents a threat actor / intrusion set."""++ __slots__ = ("id", "name", "description", "aliases", "techniques", "url", "groups")++ def __init__(self, **kw: Any) -> None:+ for slot in self.__slots__:+ setattr(self, slot, kw.get(slot, []))++ def to_dict(self) -> dict[str, Any]:+ return {s: getattr(self, s) for s in self.__slots__}+++class MitreMitigation:+ """Represents a course of action / mitigation."""++ __slots__ = ("id", "name", "description", "techniques")++ def __init__(self, **kw: Any) -> None:+ for slot in self.__slots__:+ setattr(self, slot, kw.get(slot, []))+++# ─── Global Indexes ───────────────────────────────────────────────────────────++_techniques: dict[str, MitreTechnique] = {}+_actors: dict[str, MitreActor] = {}+_mitigations: dict[str, MitreMitigation] = {}+_tactic_map: dict[str, str] = {} # tactic shortname → display name+_loaded = False+_load_time: float = 0.0+++# ─── Loader ───────────────────────────────────────────────────────────────────++def _parse_stix_bundle(bundle: dict[str, Any]) -> None:+ """Parse a STIX 2.1 ATT&CK bundle and populate in-memory indexes."""+ global _loaded, _load_time++ objects: list[dict] = bundle.get("objects", [])++ # First pass – build lookup maps+ id_to_obj: dict[str, dict] = {o["id"]: o for o in objects}+ tactic_objs: dict[str, dict] = {}++ # Tactic phase → display name+ for obj in objects:+ if obj.get("type") == "x-mitre-tactic":+ shortname = obj.get("x_mitre_shortname", "")+ _tactic_map[shortname] = obj.get("name", shortname)+ tactic_objs[shortname] = obj++ # Relationship maps+ tech_tactic: dict[str, list[str]] = {}+ tech_mitigations: dict[str, list[str]] = {}+ actor_techniques: dict[str, list[str]] = {}+ mitigation_techniques: dict[str, list[str]] = {}++ for obj in objects:+ if obj.get("type") != "relationship":+ continue+ rel_type = obj.get("relationship_type", "")+ src = obj.get("source_ref", "")+ tgt = obj.get("target_ref", "")++ if rel_type == "uses":+ # Actor uses technique+ src_obj = id_to_obj.get(src, {})+ if src_obj.get("type") in ("intrusion-set", "campaign", "malware", "tool"):+ actor_techniques.setdefault(src, []).append(tgt)++ elif rel_type == "mitigates":+ mitigation_techniques.setdefault(src, []).append(tgt)+ tech_mitigations.setdefault(tgt, []).append(src)++ # Second pass – build technique objects+ for obj in objects:+ if obj.get("type") != "attack-pattern":+ continue+ ext = obj.get("x_mitre_", {})+ is_deprecated = obj.get("x_mitre_deprecated", False) or obj.get("revoked", False)+ if is_deprecated:+ continue++ stix_id = obj["id"]+ ext_refs = obj.get("external_references", [])+ attck_ref = next(+ (r for r in ext_refs if r.get("source_name") == "mitre-attack"), {}+ )+ tech_id = attck_ref.get("external_id", "")+ if not tech_id:+ continue++ kill_chain_phases = obj.get("kill_chain_phases", [])+ tactic_ids = [+ p["phase_name"] for p in kill_chain_phases if p.get("kill_chain_name") == "mitre-attack"+ ]+ tactic_names = [_tactic_map.get(t, t) for t in tactic_ids]++ parent_id = None+ is_sub = "." in tech_id+ if is_sub:+ parent_id = tech_id.split(".")[0]++ mitigation_ids = tech_mitigations.get(stix_id, [])+ mitigation_names = [+ id_to_obj.get(mid, {}).get("name", mid) for mid in mitigation_ids+ ]++ # Detection notes from x_mitre_detection+ detection = obj.get("x_mitre_detection", "")++ technique = MitreTechnique(+ id=tech_id,+ name=obj.get("name", ""),+ description=obj.get("description", "")[:2000], # cap length+ tactic_ids=tactic_ids,+ tactic_names=tactic_names,+ platforms=obj.get("x_mitre_platforms", []),+ data_sources=obj.get("x_mitre_data_sources", []),+ mitigations=mitigation_names,+ detections=[detection] if detection else [],+ is_subtechnique=is_sub,+ parent_id=parent_id,+ url=attck_ref.get("url", ""),+ version=obj.get("x_mitre_version", ""),+ )+ _techniques[tech_id] = technique++ # Third pass – actors+ for obj in objects:+ obj_type = obj.get("type", "")+ if obj_type not in ("intrusion-set",):+ continue++ stix_id = obj["id"]+ ext_refs = obj.get("external_references", [])+ attck_ref = next((r for r in ext_refs if r.get("source_name") == "mitre-attack"), {})+ actor_id = attck_ref.get("external_id", stix_id)++ # Resolve technique IDs from STIX IDs+ used_stix_ids = actor_techniques.get(stix_id, [])+ tech_ids = []+ for sid in used_stix_ids:+ src_obj = id_to_obj.get(sid, {})+ if src_obj.get("type") == "attack-pattern":+ src_refs = src_obj.get("external_references", [])+ src_attck = next((r for r in src_refs if r.get("source_name") == "mitre-attack"), {})+ tid = src_attck.get("external_id", "")+ if tid:+ tech_ids.append(tid)++ actor = MitreActor(+ id=actor_id,+ name=obj.get("name", ""),+ description=obj.get("description", "")[:1000],+ aliases=obj.get("aliases", []),+ techniques=tech_ids,+ url=attck_ref.get("url", ""),+ )+ _actors[actor_id] = actor++ _loaded = True+ _load_time = time.time()+ logger.info(+ "MITRE ATT&CK corpus loaded",+ techniques=len(_techniques),+ actors=len(_actors),+ tactics=len(_tactic_map),+ )+++async def _download_bundle() -> dict[str, Any]:+ """Download the ATT&CK STIX bundle from MITRE GitHub."""+ logger.info("Downloading MITRE ATT&CK bundle from CDN", url=ATTCK_CDN_URL)+ async with httpx.AsyncClient(timeout=120.0, follow_redirects=True) as client:+ resp = await client.get(ATTCK_CDN_URL)+ resp.raise_for_status()+ return resp.json()+++async def load_attck_corpus(force_reload: bool = False) -> None:+ """+ Load the MITRE ATT&CK corpus.+ Priority:+ 1. Local file at ATTCK_DATA_PATH+ 2. Download from MITRE CDN+ Results are cached; re-load after _CACHE_TTL_HOURS.+ """+ global _loaded, _load_time++ age_hours = (time.time() - _load_time) / 3600 if _load_time else float("inf")+ if _loaded and not force_reload and age_hours < _CACHE_TTL_HOURS:+ return++ bundle: dict[str, Any] | None = None++ # Try local file first+ local_path = pathlib.Path(ATTCK_DATA_PATH)+ if local_path.exists():+ try:+ logger.info("Loading MITRE ATT&CK from local file", path=str(local_path))+ bundle = json.loads(local_path.read_text(encoding="utf-8"))+ except Exception as exc:+ logger.warning("Failed to read local ATT&CK bundle", error=str(exc))++ # Fall back to CDN download+ if bundle is None:+ try:+ bundle = await _download_bundle()+ # Save for future use+ local_path.parent.mkdir(parents=True, exist_ok=True)+ local_path.write_text(json.dumps(bundle), encoding="utf-8")+ logger.info("ATT&CK bundle cached locally", path=str(local_path))+ except Exception as exc:+ logger.error("Failed to download ATT&CK bundle", error=str(exc))+ # Load stub corpus so service stays up+ _load_stub_corpus()+ return++ await asyncio.get_event_loop().run_in_executor(None, _parse_stix_bundle, bundle)+++def _load_stub_corpus() -> None:+ """Populate a minimal stub corpus when the full bundle is unavailable."""+ global _loaded, _load_time+ from app.tools.mitre import (+ _MITRE_TACTICS as TACTICS,+ _MITRE_TECHNIQUES as TECHS,+ _MITRE_MITIGATIONS as MITS,+ )+ _tactic_map.update({k: v for k, v in TACTICS.items()})+ for tid, info in TECHS.items():+ tactic_id = info["tactic"]+ _techniques[tid] = MitreTechnique(+ id=tid,+ name=info["name"],+ description="",+ tactic_ids=[tactic_id],+ tactic_names=[TACTICS.get(tactic_id, tactic_id)],+ platforms=[],+ data_sources=[],+ mitigations=MITS.get(tid, []),+ detections=[],+ is_subtechnique="." in tid,+ parent_id=tid.split(".")[0] if "." in tid else None,+ url="",+ version="",+ )+ _loaded = True+ _load_time = time.time()+ logger.warning("ATT&CK stub corpus loaded (limited techniques)")+++# ─── Qdrant Embedding ─────────────────────────────────────────────────────────++async def embed_techniques_into_qdrant(+ qdrant_url: str,+ openai_api_key: str,+ collection: str = "attck_techniques",+) -> None:+ """+ Embed ATT&CK technique descriptions into Qdrant for semantic RAG.+ Only embeds techniques that are not already present in the collection.+ """+ if not _loaded:+ await load_attck_corpus()++ try:+ from qdrant_client import AsyncQdrantClient+ from qdrant_client.models import Distance, PointStruct, VectorParams+ import openai+ except ImportError as exc:+ logger.warning("Qdrant/OpenAI client not available for ATT&CK embedding", error=str(exc))+ return++ oai = openai.AsyncOpenAI(api_key=openai_api_key)+ qdrant = AsyncQdrantClient(url=qdrant_url)++ # Ensure collection exists+ try:+ await qdrant.create_collection(+ collection_name=collection,+ vectors_config=VectorParams(size=3072, distance=Distance.COSINE),+ on_disk_payload=True,+ )+ logger.info("Created Qdrant collection for ATT&CK", collection=collection)+ except Exception:+ pass # Collection already exists++ techniques_list = list(_techniques.values())+ total = len(techniques_list)+ embedded = 0++ for i in range(0, total, _EMBED_BATCH_SIZE):+ batch = techniques_list[i : i + _EMBED_BATCH_SIZE]+ texts = [+ f"{t.id} {t.name}: {t.description or ''} Tactics: {', '.join(t.tactic_names or [])}"+ for t in batch+ ]+ try:+ resp = await oai.embeddings.create(+ model="text-embedding-3-large",+ input=texts,+ dimensions=3072,+ )+ vectors = [e.embedding for e in resp.data]+ points = [+ PointStruct(+ id=abs(hash(t.id)) % (2**63),+ vector=vec,+ payload={+ "technique_id": t.id,+ "name": t.name,+ "tactic_names": t.tactic_names,+ "platforms": t.platforms,+ "url": t.url,+ },+ )+ for t, vec in zip(batch, vectors)+ ]+ await qdrant.upsert(collection_name=collection, points=points)+ embedded += len(batch)+ logger.info(+ "ATT&CK embedding progress",+ embedded=embedded,+ total=total,+ )+ except Exception as exc:+ logger.warning("Batch embedding error", batch_start=i, error=str(exc))++ logger.info("ATT&CK embedding complete", total_embedded=embedded)+ await qdrant.close()+++async def semantic_technique_search(+ query: str,+ qdrant_url: str,+ openai_api_key: str,+ collection: str = "attck_techniques",+ top_k: int = 5,+) -> list[dict[str, Any]]:+ """+ Perform semantic search over ATT&CK techniques using Qdrant.+ Returns the top_k most relevant techniques.+ """+ try:+ from qdrant_client import AsyncQdrantClient+ import openai+ except ImportError:+ return []++ oai = openai.AsyncOpenAI(api_key=openai_api_key)+ qdrant = AsyncQdrantClient(url=qdrant_url)++ try:+ emb_resp = await oai.embeddings.create(+ model="text-embedding-3-large",+ input=[query],+ dimensions=3072,+ )+ query_vector = emb_resp.data[0].embedding++ hits = await qdrant.search(+ collection_name=collection,+ query_vector=query_vector,+ limit=top_k,+ with_payload=True,+ )++ results = []+ for hit in hits:+ payload = hit.payload or {}+ tid = payload.get("technique_id", "")+ tech = _techniques.get(tid)+ result = {+ "technique_id": tid,+ "name": payload.get("name", ""),+ "tactic_names": payload.get("tactic_names", []),+ "score": hit.score,+ "description": tech.description if tech else "",+ "mitigations": tech.mitigations if tech else [],+ "url": payload.get("url", ""),+ }+ results.append(result)+ return results+ except Exception as exc:+ logger.warning("Semantic ATT&CK search failed", error=str(exc))+ return []+ finally:+ await qdrant.close()+++# ─── Public Lookup API ────────────────────────────────────────────────────────++def get_technique(technique_id: str) -> dict[str, Any]:+ """Return full technique details by ATT&CK ID (e.g. 'T1059' or 'T1059.001')."""+ tech = _techniques.get(technique_id)+ if tech is None:+ return {"id": technique_id, "name": "Unknown", "found": False}+ return {**tech.to_dict(), "found": True}+++def get_actor(actor_id: str) -> dict[str, Any]:+ """Return full actor details by ATT&CK group ID (e.g. 'G0016')."""+ actor = _actors.get(actor_id)+ if actor is None:+ return {"id": actor_id, "name": "Unknown", "found": False}+ return {**actor.to_dict(), "found": True}+++def get_actor_techniques(actor_id: str) -> list[dict[str, Any]]:+ """Return all techniques attributed to a threat actor."""+ actor = _actors.get(actor_id)+ if actor is None:+ return []+ return [get_technique(tid) for tid in (actor.techniques or [])]+++def search_techniques_by_name(keyword: str, limit: int = 20) -> list[dict[str, Any]]:+ """Full-text search across technique names and descriptions."""+ kw = keyword.lower()+ results = []+ for tech in _techniques.values():+ if kw in (tech.name or "").lower() or kw in (tech.description or "").lower():+ results.append(tech.to_dict())+ if len(results) >= limit:+ break+ return results+++def get_techniques_by_tactic(tactic_name: str) -> list[dict[str, Any]]:+ """Return all techniques belonging to a given tactic."""+ tactic_lower = tactic_name.lower()+ return [+ t.to_dict()+ for t in _techniques.values()+ if any(tn.lower() == tactic_lower for tn in (t.tactic_names or []))+ ]+++def get_coverage_summary() -> dict[str, Any]:+ """Return a summary of loaded ATT&CK data."""+ tactic_counts: dict[str, int] = {}+ for tech in _techniques.values():+ for tname in (tech.tactic_names or []):+ tactic_counts[tname] = tactic_counts.get(tname, 0) + 1+ return {+ "total_techniques": len(_techniques),+ "total_actors": len(_actors),+ "total_tactics": len(_tactic_map),+ "techniques_by_tactic": tactic_counts,+ "corpus_loaded": _loaded,+ "load_age_hours": round((time.time() - _load_time) / 3600, 1) if _load_time else None,+ }+++def map_techniques_to_kill_chain(technique_ids: list[str]) -> dict[str, list[str]]:+ """Map technique IDs to their kill-chain phases."""+ result: dict[str, list[str]] = {}+ for tid in technique_ids:+ tech = _techniques.get(tid)+ if tech is None:+ result.setdefault("Unknown", []).append(tid)+ continue+ for tname in (tech.tactic_names or ["Unknown"]):+ result.setdefault(tname, []).append(f"{tid}: {tech.name}")+ return resultdiff --git a/services/agents/pyproject.toml b/services/agents/pyproject.tomlindex cebbdcad2..445f098ac 100644--- a/services/agents/pyproject.toml+++ b/services/agents/pyproject.toml@@ -23,6 +23,7 @@ structlog = "^24.1.0"prometheus-client = "^0.19.0"python-dateutil = "^2.8.2"aiokafka = "^0.10.0"+qdrant-client = {extras = ["fastembed"], version = "^1.9.0"}[tool.poetry.group.dev.dependencies]pytest = "^7.4.0"diff --git a/services/api/app/api/v1/endpoints/detection_rules.py b/services/api/app/api/v1/endpoints/detection_rules.pyindex 202feb20a..abbf8caa3 100644--- a/services/api/app/api/v1/endpoints/detection_rules.py+++ b/services/api/app/api/v1/endpoints/detection_rules.py@@ -1,15 +1,16 @@"""Detection rule management endpoints."""import uuidfrom datetime import UTC, datetime-from typing import Annotated+from typing import Annotated, Anyfrom fastapi import APIRouter, Depends, HTTPException, Query, status-from pydantic import BaseModel+from pydantic import BaseModel, Fieldfrom sqlalchemy import and_, func, or_, select, updatefrom sqlalchemy.ext.asyncio import AsyncSessionfrom app.api.v1.deps import AuthUser, DBSession, require_permissionfrom app.models.detection_rule import DetectionRule+from app.services.rule_engine import execute_rule, run_huntrouter = APIRouter(prefix="/rules", tags=["detection_rules"])@@ -199,3 +200,180 @@ async def delete_rule()await db.delete(rule)await db.commit()+++# ─── Rule Execution Endpoint ──────────────────────────────────────────────────++class ExecuteRuleRequest(BaseModel):+ """Payload for ad-hoc rule execution."""+ events: list[dict[str, Any]] = Field(+ ..., description="Events to test the rule against", max_length=1000+ )+++class ExecuteRuleResponse(BaseModel):+ rule_id: str+ rule_name: str+ rule_language: str+ severity: str+ matched: bool+ match_count: int+ matched_events: list[dict[str, Any]]+ score: float+ error: str | None+ execution_time_ms: float+++ "/{rule_id}/execute",+ response_model=ExecuteRuleResponse,+ summary="Execute a detection rule against provided events",+)+async def execute_detection_rule(+ rule_id: uuid.UUID,+ request: ExecuteRuleRequest,+ current_user: Annotated[AuthUser, Depends(require_permission("rules:read"))],+ db: DBSession,+) -> ExecuteRuleResponse:+ """+ Execute a single detection rule against a set of events and return matches.+ Useful for testing rules in the detection IDE before enabling them.+ """+ result = await db.execute(+ select(DetectionRule).where(+ DetectionRule.id == rule_id,+ or_(+ DetectionRule.tenant_id == current_user.tenant_id,+ DetectionRule.tenant_id.is_(None),+ ),+ )+ )+ rule = result.scalar_one_or_none()+ if rule is None:+ raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Rule not found")++ match = execute_rule(+ rule_id=str(rule.id),+ rule_name=rule.name,+ rule_language=rule.rule_language,+ rule_body=rule.rule_body,+ severity=rule.severity,+ events=request.events,+ )++ # Update hit stats if matched+ if match.matched:+ await db.execute(+ update(DetectionRule)+ .where(DetectionRule.id == rule_id)+ .values(+ total_hits=DetectionRule.total_hits + len(match.match_details.get("matched_events", [])),+ last_triggered=datetime.now(UTC),+ )+ )+ await db.commit()++ matched_events = match.match_details.get("matched_events", [])+ return ExecuteRuleResponse(+ rule_id=str(rule.id),+ rule_name=rule.name,+ rule_language=rule.rule_language,+ severity=rule.severity,+ matched=match.matched,+ match_count=len(matched_events),+ matched_events=matched_events,+ score=match.score,+ error=match.error,+ execution_time_ms=match.execution_time_ms,+ )+++# ─── Hunt Endpoint ────────────────────────────────────────────────────────────++class HuntRequest(BaseModel):+ """Payload for threat hunting across events."""+ rule_ids: list[uuid.UUID] | None = Field(+ None, description="Specific rule IDs to hunt with; omit to use all active rules"+ )+ rule_language: str | None = Field(+ None, description="Filter rules by language (sigma, yara, kql, eql)"+ )+ events: list[dict[str, Any]] = Field(+ ..., description="Events to hunt through", max_length=5000+ )+++class HuntResponse(BaseModel):+ hunt_id: str+ rules_evaluated: int+ rules_matched: int+ total_events_scanned: int+ matched_events: list[dict[str, Any]]+ match_summary: list[dict[str, Any]]+ execution_time_ms: float+ errors: list[str]+++ "/hunt",+ response_model=HuntResponse,+ summary="Threat hunt: run detection rules against a set of events",+)+async def hunt(+ request: HuntRequest,+ current_user: Annotated[AuthUser, Depends(require_permission("rules:read"))],+ db: DBSession,+) -> HuntResponse:+ """+ Run multiple detection rules against provided events (threat hunting).+ If rule_ids is omitted, all active tenant rules are used.+ """+ # Build filter+ filters = [+ or_(+ DetectionRule.tenant_id == current_user.tenant_id,+ and_(DetectionRule.tenant_id.is_(None), DetectionRule.is_builtin == True),+ ),+ DetectionRule.status == "active",+ ]+ if request.rule_ids:+ filters.append(DetectionRule.id.in_(request.rule_ids))+ if request.rule_language:+ filters.append(DetectionRule.rule_language == request.rule_language)++ result = await db.execute(select(DetectionRule).where(and_(*filters)))+ rules = result.scalars().all()++ if not rules:+ raise HTTPException(+ status_code=status.HTTP_404_NOT_FOUND,+ detail="No active detection rules found matching the criteria",+ )++ rule_dicts = [+ {+ "id": str(r.id),+ "name": r.name,+ "rule_language": r.rule_language,+ "rule_body": r.rule_body,+ "severity": r.severity,+ }+ for r in rules+ ]++ hunt_result = await run_hunt(+ tenant_id=str(current_user.tenant_id),+ rules=rule_dicts,+ events=request.events,+ )++ return HuntResponse(+ hunt_id=hunt_result.hunt_id,+ rules_evaluated=hunt_result.rules_evaluated,+ rules_matched=hunt_result.rules_matched,+ total_events_scanned=hunt_result.total_events_scanned,+ matched_events=hunt_result.matched_events,+ match_summary=hunt_result.match_summary,+ execution_time_ms=hunt_result.execution_time_ms,+ errors=hunt_result.errors,+ )diff --git a/services/api/app/api/v1/endpoints/graph.py b/services/api/app/api/v1/endpoints/graph.pynew file mode 100644index 000000000..c9dd3c5ea--- /dev/null+++ b/services/api/app/api/v1/endpoints/graph.py@@ -0,0 +1,316 @@+"""+Graph API endpoints: attack paths, blast radius, entity neighbors, MITRE coverage.+Cyble Open-Source AI Security Operations Center - MIT License+"""+from __future__ import annotations++from typing import Any, Annotated++from fastapi import APIRouter, Depends, HTTPException, Query, status+from pydantic import BaseModel, Field++from app.api.v1.deps import get_current_user, CurrentUser+from app.services import graph_service++router = APIRouter(prefix="/graph", tags=["graph"])+++# ─── Request / Response Schemas ───────────────────────────────────────────────++class GraphNode(BaseModel):+ id: str+ label: str+ properties: dict[str, Any] = Field(default_factory=dict)+++class GraphEdge(BaseModel):+ source: str+ target: str+ type: str+++class AttackPathResponse(BaseModel):+ case_id: str+ nodes: list[GraphNode]+ edges: list[GraphEdge]+ node_count: int+ edge_count: int+++class BlastRadiusResponse(BaseModel):+ entity_id: str+ entity_type: str+ hops: int+ affected_nodes: list[GraphNode]+ total_affected: int+ type_breakdown: dict[str, int]+ blast_radius_score: float+++class EntityNeighborsResponse(BaseModel):+ entity_id: str+ entity_type: str+ source: GraphNode | None+ neighbors: list[dict[str, Any]]+ neighbor_count: int = 0+++class MitreCoverageItem(BaseModel):+ technique_id: str+ name: str | None+ tactic: str | None+ alert_count: int+++class UpsertHostRequest(BaseModel):+ host_id: str+ hostname: str+ ip_address: str = ""+ os: str = ""+ criticality: str = "medium"+++class UpsertUserRequest(BaseModel):+ user_id: str+ username: str+ email: str = ""+ department: str = ""+ risk_score: float = 0.0+++class UpsertAlertGraphRequest(BaseModel):+ alert_id: str+ title: str+ severity: str+ mitre_techniques: list[str] = Field(default_factory=list)+ host_id: str | None = None+ user_id: str | None = None+ ioc_values: list[str] = Field(default_factory=list)+++class UpsertCaseGraphRequest(BaseModel):+ case_id: str+ title: str+ severity: str+ alert_ids: list[str] = Field(default_factory=list)+++# ─── Endpoints ────────────────────────────────────────────────────────────────++ "/attack-path/{case_id}",+ response_model=AttackPathResponse,+ summary="Get attack path graph for a case",+)+async def get_attack_path(+ case_id: str,+ max_depth: Annotated[int, Query(ge=1, le=10)] = 6,+ current_user: CurrentUser = Depends(get_current_user),+) -> AttackPathResponse:+ """+ Traverse the knowledge graph from a Case node to reconstruct the full+ attack path: Case → Alerts → Hosts/Users → IOCs → MITRE Techniques.+ """+ try:+ data = await graph_service.get_attack_path(+ case_id=case_id,+ tenant_id=str(current_user.tenant_id),+ max_depth=max_depth,+ )+ except Exception as exc:+ raise HTTPException(+ status_code=status.HTTP_503_SERVICE_UNAVAILABLE,+ detail=f"Graph query failed: {exc}",+ )++ if not data["nodes"]:+ raise HTTPException(+ status_code=status.HTTP_404_NOT_FOUND,+ detail=f"Case {case_id} not found in graph or has no linked entities",+ )++ return AttackPathResponse(**data)+++ "/blast-radius/{entity_type}/{entity_id}",+ response_model=BlastRadiusResponse,+ summary="Compute blast radius from an entity",+)+async def get_blast_radius(+ entity_type: str,+ entity_id: str,+ hops: Annotated[int, Query(ge=1, le=6)] = 3,+ current_user: CurrentUser = Depends(get_current_user),+) -> BlastRadiusResponse:+ """+ Compute the blast radius starting from a Host, User, or IOC node.+ Returns all entities reachable within `hops` and a severity score.+ """+ valid_types = {"host", "user", "ioc", "alert"}+ if entity_type.lower() not in valid_types:+ raise HTTPException(+ status_code=status.HTTP_400_BAD_REQUEST,+ detail=f"entity_type must be one of: {sorted(valid_types)}",+ )++ try:+ data = await graph_service.get_blast_radius(+ entity_id=entity_id,+ entity_type=entity_type,+ tenant_id=str(current_user.tenant_id),+ hops=hops,+ )+ except Exception as exc:+ raise HTTPException(+ status_code=status.HTTP_503_SERVICE_UNAVAILABLE,+ detail=f"Graph query failed: {exc}",+ )++ return BlastRadiusResponse(**data)+++ "/neighbors/{entity_type}/{entity_id}",+ response_model=EntityNeighborsResponse,+ summary="Get immediate graph neighbors of an entity",+)+async def get_entity_neighbors(+ entity_type: str,+ entity_id: str,+ current_user: CurrentUser = Depends(get_current_user),+) -> EntityNeighborsResponse:+ """Return all nodes directly connected (depth 1) to the specified entity."""+ try:+ data = await graph_service.get_entity_neighbors(+ entity_id=entity_id,+ entity_type=entity_type,+ tenant_id=str(current_user.tenant_id),+ )+ except Exception as exc:+ raise HTTPException(+ status_code=status.HTTP_503_SERVICE_UNAVAILABLE,+ detail=f"Graph query failed: {exc}",+ )++ return EntityNeighborsResponse(**data)+++ "/mitre-coverage",+ response_model=list[MitreCoverageItem],+ summary="MITRE ATT&CK technique coverage for tenant",+)+async def get_mitre_coverage(+ current_user: CurrentUser = Depends(get_current_user),+) -> list[MitreCoverageItem]:+ """Return MITRE ATT&CK technique coverage aggregated from all tenant alerts."""+ try:+ records = await graph_service.get_mitre_coverage(+ tenant_id=str(current_user.tenant_id),+ )+ except Exception as exc:+ raise HTTPException(+ status_code=status.HTTP_503_SERVICE_UNAVAILABLE,+ detail=f"Graph query failed: {exc}",+ )++ return [+ MitreCoverageItem(+ technique_id=r.get("technique_id", ""),+ name=r.get("name"),+ tactic=r.get("tactic"),+ alert_count=r.get("alert_count", 0),+ )+ for r in records+ ]+++# ─── Write Endpoints ──────────────────────────────────────────────────────────++ "/entities/host",+ status_code=status.HTTP_201_CREATED,+ summary="Upsert a Host node in the graph",+)+async def upsert_host(+ payload: UpsertHostRequest,+ current_user: CurrentUser = Depends(get_current_user),+) -> dict[str, str]:+ """Create or update a Host node in the knowledge graph."""+ await graph_service.upsert_host(+ host_id=payload.host_id,+ hostname=payload.hostname,+ tenant_id=str(current_user.tenant_id),+ ip_address=payload.ip_address,+ os=payload.os,+ criticality=payload.criticality,+ )+ return {"status": "ok", "host_id": payload.host_id}+++ "/entities/user",+ status_code=status.HTTP_201_CREATED,+ summary="Upsert a User node in the graph",+)+async def upsert_user(+ payload: UpsertUserRequest,+ current_user: CurrentUser = Depends(get_current_user),+) -> dict[str, str]:+ """Create or update a User node in the knowledge graph."""+ await graph_service.upsert_user(+ user_id=payload.user_id,+ username=payload.username,+ tenant_id=str(current_user.tenant_id),+ email=payload.email,+ department=payload.department,+ risk_score=payload.risk_score,+ )+ return {"status": "ok", "user_id": payload.user_id}+++ "/entities/alert",+ status_code=status.HTTP_201_CREATED,+ summary="Upsert an Alert node and its relationships in the graph",+)+async def upsert_alert_graph(+ payload: UpsertAlertGraphRequest,+ current_user: CurrentUser = Depends(get_current_user),+) -> dict[str, str]:+ """+ Create or update an Alert node and link it to Host, User, IOC, and Technique nodes.+ Called automatically after alert creation to keep the graph in sync.+ """+ await graph_service.upsert_alert_node(+ alert_id=payload.alert_id,+ tenant_id=str(current_user.tenant_id),+ title=payload.title,+ severity=payload.severity,+ mitre_techniques=payload.mitre_techniques,+ host_id=payload.host_id,+ user_id=payload.user_id,+ ioc_values=payload.ioc_values,+ )+ return {"status": "ok", "alert_id": payload.alert_id}+++ "/entities/case",+ status_code=status.HTTP_201_CREATED,+ summary="Upsert a Case node and link to alerts in the graph",+)+async def upsert_case_graph(+ payload: UpsertCaseGraphRequest,+ current_user: CurrentUser = Depends(get_current_user),+) -> dict[str, str]:+ """Create or update a Case node and link it to Alert nodes."""+ await graph_service.upsert_case_node(+ case_id=payload.case_id,+ tenant_id=str(current_user.tenant_id),+ title=payload.title,+ severity=payload.severity,+ alert_ids=payload.alert_ids,+ )+ return {"status": "ok", "case_id": payload.case_id}diff --git a/services/api/app/api/v1/router.py b/services/api/app/api/v1/router.pyindex 6391900dc..83853c56b 100644--- a/services/api/app/api/v1/router.py+++ b/services/api/app/api/v1/router.py@@ -1,7 +1,7 @@"""API v1 router aggregating all endpoint modules."""from fastapi import APIRouter-from app.api.v1.endpoints import alerts, auth, cases, connectors, detection_rules, tenants+from app.api.v1.endpoints import alerts, auth, cases, connectors, detection_rules, graph, tenantsapi_router = APIRouter(prefix="/api/v1")@@ -11,3 +11,4 @@ api_router.include_router(cases.router)api_router.include_router(connectors.router)api_router.include_router(tenants.router)api_router.include_router(detection_rules.router)+api_router.include_router(graph.router)diff --git a/services/api/app/core/config.py b/services/api/app/core/config.pyindex 25f1edd49..5c645bd67 100644--- a/services/api/app/core/config.py+++ b/services/api/app/core/config.py@@ -21,6 +21,8 @@ class Settings(BaseSettings):APP_NAME: str = "AiSOC API"APP_VERSION: str = "0.1.0"ENV: str = "development"+ ENVIRONMENT: str = "development" # alias for ENV+ VERSION: str = "0.1.0" # alias for APP_VERSIONDEBUG: bool = FalseAPI_PREFIX: str = "/api/v1"diff --git a/services/api/app/db/neo4j.py b/services/api/app/db/neo4j.pynew file mode 100644index 000000000..733ebae3c--- /dev/null+++ b/services/api/app/db/neo4j.py@@ -0,0 +1,108 @@+"""+Neo4j driver singleton for AiSOC graph layer.+Cyble Open-Source AI Security Operations Center - MIT License+"""+from __future__ import annotations++import asyncio+import logging+from contextlib import asynccontextmanager+from typing import AsyncGenerator, Optional++from neo4j import AsyncGraphDatabase, AsyncDriver, AsyncSession+from neo4j.exceptions import ServiceUnavailable++from app.core.config import settings++logger = logging.getLogger(__name__)++_driver: Optional[AsyncDriver] = None+++async def init_neo4j() -> None:+ """Initialize the Neo4j async driver and verify connectivity."""+ global _driver+ if _driver is not None:+ return++ _driver = AsyncGraphDatabase.driver(+ settings.NEO4J_URI,+ auth=(settings.NEO4J_USER, settings.NEO4J_PASSWORD),+ max_connection_pool_size=50,+ connection_acquisition_timeout=30,+ )++ # Verify connectivity with retries+ for attempt in range(5):+ try:+ await _driver.verify_connectivity()+ logger.info("Neo4j connection established", uri=settings.NEO4J_URI)++ # Create schema constraints and indexes+ await _create_schema()+ return+ except ServiceUnavailable:+ if attempt < 4:+ wait = 2 ** attempt+ logger.warning("Neo4j not ready, retrying", attempt=attempt + 1, wait=wait)+ await asyncio.sleep(wait)+ else:+ logger.error("Neo4j connection failed after retries")+ raise+++async def close_neo4j() -> None:+ """Close the Neo4j driver."""+ global _driver+ if _driver:+ await _driver.close()+ _driver = None+ logger.info("Neo4j connection closed")+++def get_driver() -> AsyncDriver:+ """Return the singleton Neo4j driver. Raises if not initialized."""+ if _driver is None:+ raise RuntimeError("Neo4j driver not initialized. Call init_neo4j() first.")+ return _driver+++@asynccontextmanager+async def get_session() -> AsyncGenerator[AsyncSession, None]:+ """Async context manager yielding a Neo4j session."""+ driver = get_driver()+ async with driver.session(database="neo4j") as session:+ yield session+++async def _create_schema() -> None:+ """Create constraints and indexes for the graph schema."""+ constraints = [+ # Uniqueness constraints+ "CREATE CONSTRAINT IF NOT EXISTS FOR (h:Host) REQUIRE h.id IS UNIQUE",+ "CREATE CONSTRAINT IF NOT EXISTS FOR (u:User) REQUIRE u.id IS UNIQUE",+ "CREATE CONSTRAINT IF NOT EXISTS FOR (a:Alert) REQUIRE a.id IS UNIQUE",+ "CREATE CONSTRAINT IF NOT EXISTS FOR (i:IOC) REQUIRE i.value IS UNIQUE",+ "CREATE CONSTRAINT IF NOT EXISTS FOR (t:Technique) REQUIRE t.technique_id IS UNIQUE",+ "CREATE CONSTRAINT IF NOT EXISTS FOR (c:Case) REQUIRE c.id IS UNIQUE",+ "CREATE CONSTRAINT IF NOT EXISTS FOR (p:Process) REQUIRE p.id IS UNIQUE",+ # Indexes for common lookups+ "CREATE INDEX IF NOT EXISTS FOR (h:Host) ON (h.hostname)",+ "CREATE INDEX IF NOT EXISTS FOR (h:Host) ON (h.tenant_id)",+ "CREATE INDEX IF NOT EXISTS FOR (u:User) ON (u.username)",+ "CREATE INDEX IF NOT EXISTS FOR (u:User) ON (u.tenant_id)",+ "CREATE INDEX IF NOT EXISTS FOR (a:Alert) ON (a.tenant_id)",+ "CREATE INDEX IF NOT EXISTS FOR (a:Alert) ON (a.severity)",+ "CREATE INDEX IF NOT EXISTS FOR (i:IOC) ON (i.ioc_type)",+ "CREATE INDEX IF NOT EXISTS FOR (i:IOC) ON (i.tenant_id)",+ "CREATE INDEX IF NOT EXISTS FOR (t:Technique) ON (t.tactic)",+ ]++ async with get_session() as session:+ for cypher in constraints:+ try:+ await session.run(cypher)+ except Exception as exc:+ logger.debug("Schema statement skipped", cypher=cypher[:60], error=str(exc))++ logger.info("Neo4j schema constraints and indexes ensured")diff --git a/services/api/app/main.py b/services/api/app/main.pyindex 9561a8d97..18d6f3156 100644--- a/services/api/app/main.py+++ b/services/api/app/main.py@@ -13,6 +13,7 @@ from app.api.v1.router import api_routerfrom app.core.config import settingsfrom app.core.logging import configure_loggingfrom app.db.database import engine+from app.db.neo4j import init_neo4j, close_neo4jfrom app.models import Baselogger = structlog.get_logger(__name__)@@ -42,10 +43,17 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:await conn.run_sync(Base.metadata.create_all)logger.info("Database tables created (development mode)")+ # Initialize Neo4j graph layer+ try:+ await init_neo4j()+ except Exception as exc:+ logger.warning("Neo4j unavailable at startup – graph features disabled", error=str(exc))+yieldlogger.info("AiSOC API shutting down")await engine.dispose()+ await close_neo4j()def create_application() -> FastAPI:diff --git a/services/api/app/services/__init__.py b/services/api/app/services/__init__.pynew file mode 100644index 000000000..e69de29bbdiff --git a/services/api/app/services/graph_service.py b/services/api/app/services/graph_service.pynew file mode 100644index 000000000..2d7049c24--- /dev/null+++ b/services/api/app/services/graph_service.py@@ -0,0 +1,485 @@+"""+Graph service: entity node management, attack-path queries, blast-radius traversal.+Cyble Open-Source AI Security Operations Center - MIT License+"""+from __future__ import annotations++import logging+import uuid+from datetime import datetime, timezone+from typing import Any, Optional++from app.db.neo4j import get_session++logger = logging.getLogger(__name__)+++# ─── Entity Upsert Helpers ────────────────────────────────────────────────────++async def upsert_host(+ host_id: str,+ hostname: str,+ tenant_id: str,+ ip_address: str = "",+ os: str = "",+ criticality: str = "medium",+ extra: dict[str, Any] | None = None,+) -> None:+ """Create or update a Host node."""+ props = {+ "id": host_id,+ "hostname": hostname,+ "tenant_id": tenant_id,+ "ip_address": ip_address,+ "os": os,+ "criticality": criticality,+ "updated_at": datetime.now(timezone.utc).isoformat(),+ **(extra or {}),+ }+ cypher = """+ MERGE (h:Host {id: $id})+ SET h += $props+ RETURN h.id AS node_id+ """+ async with get_session() as s:+ await s.run(cypher, id=host_id, props=props)+++async def upsert_user(+ user_id: str,+ username: str,+ tenant_id: str,+ email: str = "",+ department: str = "",+ risk_score: float = 0.0,+) -> None:+ """Create or update a User node."""+ props = {+ "id": user_id,+ "username": username,+ "tenant_id": tenant_id,+ "email": email,+ "department": department,+ "risk_score": risk_score,+ "updated_at": datetime.now(timezone.utc).isoformat(),+ }+ cypher = """+ MERGE (u:User {id: $id})+ SET u += $props+ RETURN u.id AS node_id+ """+ async with get_session() as s:+ await s.run(cypher, id=user_id, props=props)+++async def upsert_alert_node(+ alert_id: str,+ tenant_id: str,+ title: str,+ severity: str,+ mitre_techniques: list[str] | None = None,+ host_id: str | None = None,+ user_id: str | None = None,+ ioc_values: list[str] | None = None,+) -> None:+ """Create Alert node and link it to related entities."""+ alert_props = {+ "id": alert_id,+ "tenant_id": tenant_id,+ "title": title,+ "severity": severity,+ "created_at": datetime.now(timezone.utc).isoformat(),+ }++ async with get_session() as s:+ # Upsert Alert node+ await s.run(+ "MERGE (a:Alert {id: $id}) SET a += $props",+ id=alert_id,+ props=alert_props,+ )++ # Link to Host+ if host_id:+ await s.run(+ """+ MATCH (a:Alert {id: $alert_id})+ MERGE (h:Host {id: $host_id})+ MERGE (a)-[:OBSERVED_ON]->(h)+ """,+ alert_id=alert_id,+ host_id=host_id,+ )++ # Link to User+ if user_id:+ await s.run(+ """+ MATCH (a:Alert {id: $alert_id})+ MERGE (u:User {id: $user_id})+ MERGE (a)-[:INVOLVES_USER]->(u)+ """,+ alert_id=alert_id,+ user_id=user_id,+ )++ # Link to MITRE Techniques+ for technique_id in (mitre_techniques or []):+ await s.run(+ """+ MATCH (a:Alert {id: $alert_id})+ MERGE (t:Technique {technique_id: $technique_id})+ SET t.updated_at = $ts+ MERGE (a)-[:MAPS_TO]->(t)+ """,+ alert_id=alert_id,+ technique_id=technique_id,+ ts=datetime.now(timezone.utc).isoformat(),+ )++ # Link to IOCs+ for ioc_val in (ioc_values or []):+ await s.run(+ """+ MATCH (a:Alert {id: $alert_id})+ MERGE (i:IOC {value: $ioc_val})+ SET i.tenant_id = $tenant_id, i.updated_at = $ts+ MERGE (a)-[:CONTAINS_IOC]->(i)+ """,+ alert_id=alert_id,+ ioc_val=ioc_val,+ tenant_id=tenant_id,+ ts=datetime.now(timezone.utc).isoformat(),+ )+++async def upsert_case_node(+ case_id: str,+ tenant_id: str,+ title: str,+ severity: str,+ alert_ids: list[str] | None = None,+) -> None:+ """Create Case node and link it to Alert nodes."""+ case_props = {+ "id": case_id,+ "tenant_id": tenant_id,+ "title": title,+ "severity": severity,+ "created_at": datetime.now(timezone.utc).isoformat(),+ }++ async with get_session() as s:+ await s.run(+ "MERGE (c:Case {id: $id}) SET c += $props",+ id=case_id,+ props=case_props,+ )++ for alert_id in (alert_ids or []):+ await s.run(+ """+ MATCH (c:Case {id: $case_id})+ MERGE (a:Alert {id: $alert_id})+ MERGE (c)-[:CONTAINS_ALERT]->(a)+ """,+ case_id=case_id,+ alert_id=alert_id,+ )+++async def upsert_ioc(+ value: str,+ ioc_type: str,+ tenant_id: str,+ malicious: bool = False,+ confidence: float = 0.0,+ tags: list[str] | None = None,+) -> None:+ """Create or update an IOC node."""+ props = {+ "value": value,+ "ioc_type": ioc_type,+ "tenant_id": tenant_id,+ "malicious": malicious,+ "confidence": confidence,+ "tags": tags or [],+ "updated_at": datetime.now(timezone.utc).isoformat(),+ }+ async with get_session() as s:+ await s.run(+ "MERGE (i:IOC {value: $value}) SET i += $props",+ value=value,+ props=props,+ )+++# ─── Query Helpers ────────────────────────────────────────────────────────────++async def get_attack_path(case_id: str, tenant_id: str, max_depth: int = 6) -> dict[str, Any]:+ """+ Traverse the graph from a Case node to reconstruct the attack path.++ Returns nodes and relationships forming the attack chain:+ Case → Alerts → Hosts/Users → IOCs + Techniques+ """+ cypher = """+ MATCH (c:Case {id: $case_id, tenant_id: $tenant_id})+ CALL apoc.path.subgraphAll(c, {+ maxLevel: $max_depth,+ relationshipFilter: 'CONTAINS_ALERT>|OBSERVED_ON>|INVOLVES_USER>|CONTAINS_IOC>|MAPS_TO>'+ })+ YIELD nodes, relationships+ RETURN+ [n IN nodes | {+ id: coalesce(n.id, n.value, n.technique_id),+ label: labels(n)[0],+ properties: properties(n)+ }] AS nodes,+ [r IN relationships | {+ source: startNode(r).id,+ target: endNode(r).id,+ type: type(r)+ }] AS edges+ """+ async with get_session() as s:+ result = await s.run(cypher, case_id=case_id, tenant_id=tenant_id, max_depth=max_depth)+ record = await result.single()++ if not record:+ # APOC not available – fall back to manual multi-hop query+ return await _attack_path_fallback(case_id, tenant_id)++ return {+ "case_id": case_id,+ "nodes": record["nodes"],+ "edges": record["edges"],+ "node_count": len(record["nodes"]),+ "edge_count": len(record["edges"]),+ }+++async def _attack_path_fallback(case_id: str, tenant_id: str) -> dict[str, Any]:+ """Manual attack path traversal when APOC is unavailable."""+ nodes: list[dict] = []+ edges: list[dict] = []++ async with get_session() as s:+ # Get case+ r = await s.run(+ "MATCH (c:Case {id: $id, tenant_id: $tid}) RETURN c",+ id=case_id, tid=tenant_id,+ )+ rec = await r.single()+ if not rec:+ return {"case_id": case_id, "nodes": [], "edges": [], "node_count": 0, "edge_count": 0}+ nodes.append({"id": case_id, "label": "Case", "properties": dict(rec["c"])})++ # Alerts+ r2 = await s.run(+ """+ MATCH (c:Case {id: $id})-[:CONTAINS_ALERT]->(a:Alert)+ RETURN a+ """,+ id=case_id,+ )+ alert_ids = []+ async for record in r2:+ a = dict(record["a"])+ nodes.append({"id": a["id"], "label": "Alert", "properties": a})+ edges.append({"source": case_id, "target": a["id"], "type": "CONTAINS_ALERT"})+ alert_ids.append(a["id"])++ # Hosts + Users + IOCs + Techniques linked from alerts+ for aid in alert_ids:+ r3 = await s.run(+ """+ MATCH (a:Alert {id: $aid})-[rel]->(n)+ RETURN n, type(rel) AS rel_type, labels(n)[0] AS label+ """,+ aid=aid,+ )+ async for record in r3:+ n = dict(record["n"])+ node_id = n.get("id") or n.get("value") or n.get("technique_id", "")+ nodes.append({"id": node_id, "label": record["label"], "properties": n})+ edges.append({"source": aid, "target": node_id, "type": record["rel_type"]})++ # Deduplicate nodes by id+ seen: set[str] = set()+ unique_nodes = []+ for node in nodes:+ nid = node["id"]+ if nid not in seen:+ seen.add(nid)+ unique_nodes.append(node)++ return {+ "case_id": case_id,+ "nodes": unique_nodes,+ "edges": edges,+ "node_count": len(unique_nodes),+ "edge_count": len(edges),+ }+++async def get_blast_radius(entity_id: str, entity_type: str, tenant_id: str, hops: int = 3) -> dict[str, Any]:+ """+ Compute blast radius: all entities reachable from an IOC/Host/User within N hops.+ Used for impact assessment during incident response.+ """+ label_map = {+ "host": "Host",+ "user": "User",+ "ioc": "IOC",+ "alert": "Alert",+ }+ label = label_map.get(entity_type.lower(), "Host")+ id_prop = "value" if label == "IOC" else "id"++ cypher = f"""+ MATCH (start:{label} {{{id_prop}: $entity_id}})+ WHERE start.tenant_id = $tenant_id OR start.tenant_id IS NULL+ CALL apoc.path.expandConfig(start, {{+ maxLevel: $hops,+ bfs: true,+ uniqueness: 'NODE_GLOBAL'+ }})+ YIELD path+ WITH nodes(path) AS path_nodes, relationships(path) AS path_rels+ UNWIND path_nodes AS n+ WITH COLLECT(DISTINCT {{+ id: coalesce(n.id, n.value, n.technique_id),+ label: labels(n)[0],+ properties: properties(n)+ }}) AS all_nodes+ RETURN all_nodes+ """++ async with get_session() as s:+ result = await s.run(cypher, entity_id=entity_id, tenant_id=tenant_id, hops=hops)+ record = await result.single()++ if not record:+ return await _blast_radius_fallback(entity_id, entity_type, tenant_id, hops)++ all_nodes = record["all_nodes"]+ type_counts: dict[str, int] = {}+ for node in all_nodes:+ lbl = node.get("label", "Unknown")+ type_counts[lbl] = type_counts.get(lbl, 0) + 1++ return {+ "entity_id": entity_id,+ "entity_type": entity_type,+ "hops": hops,+ "affected_nodes": all_nodes,+ "total_affected": len(all_nodes),+ "type_breakdown": type_counts,+ "blast_radius_score": _calc_blast_score(type_counts),+ }+++async def _blast_radius_fallback(+ entity_id: str, entity_type: str, tenant_id: str, hops: int+) -> dict[str, Any]:+ """Simple 2-hop blast radius without APOC."""+ label_map = {"host": "Host", "user": "User", "ioc": "IOC", "alert": "Alert"}+ label = label_map.get(entity_type.lower(), "Host")+ id_prop = "value" if label == "IOC" else "id"++ cypher = f"""+ MATCH (start:{label} {{{id_prop}: $entity_id}})+ MATCH (start)-[*1..{hops}]-(n)+ WHERE n.tenant_id = $tenant_id OR n.tenant_id IS NULL+ RETURN COLLECT(DISTINCT {{+ id: coalesce(n.id, n.value, n.technique_id),+ label: labels(n)[0],+ properties: properties(n)+ }}) AS affected+ """+ async with get_session() as s:+ result = await s.run(cypher, entity_id=entity_id, tenant_id=tenant_id)+ record = await result.single()++ affected = record["affected"] if record else []+ type_counts: dict[str, int] = {}+ for node in affected:+ lbl = node.get("label", "Unknown")+ type_counts[lbl] = type_counts.get(lbl, 0) + 1++ return {+ "entity_id": entity_id,+ "entity_type": entity_type,+ "hops": hops,+ "affected_nodes": affected,+ "total_affected": len(affected),+ "type_breakdown": type_counts,+ "blast_radius_score": _calc_blast_score(type_counts),+ }+++def _calc_blast_score(type_counts: dict[str, int]) -> float:+ """+ Heuristic blast-radius severity score 0–100.+ Weights: Alert=10, Host=5, User=8, IOC=3, Technique=2.+ """+ weights = {"Alert": 10, "Host": 5, "User": 8, "IOC": 3, "Technique": 2}+ raw = sum(weights.get(lbl, 1) * count for lbl, count in type_counts.items())+ return min(round(raw / 10, 1), 100.0)+++async def get_entity_neighbors(+ entity_id: str,+ entity_type: str,+ tenant_id: str,+) -> dict[str, Any]:+ """Return immediate neighbors of a node (depth 1)."""+ label_map = {"host": "Host", "user": "User", "ioc": "IOC", "alert": "Alert", "case": "Case"}+ label = label_map.get(entity_type.lower(), "Host")+ id_prop = "value" if label == "IOC" else "id"++ cypher = f"""+ MATCH (n:{label} {{{id_prop}: $entity_id}})+ MATCH (n)-[r]-(neighbor)+ RETURN+ {{id: coalesce(n.id, n.value), label: labels(n)[0], properties: properties(n)}} AS source,+ COLLECT(DISTINCT {{+ id: coalesce(neighbor.id, neighbor.value, neighbor.technique_id),+ label: labels(neighbor)[0],+ rel_type: type(r),+ properties: properties(neighbor)+ }}) AS neighbors+ """+ async with get_session() as s:+ result = await s.run(cypher, entity_id=entity_id)+ record = await result.single()++ if not record:+ return {"entity_id": entity_id, "entity_type": entity_type, "source": None, "neighbors": []}++ return {+ "entity_id": entity_id,+ "entity_type": entity_type,+ "source": record["source"],+ "neighbors": record["neighbors"],+ "neighbor_count": len(record["neighbors"]),+ }+++async def get_mitre_coverage(tenant_id: str) -> list[dict[str, Any]]:+ """Return MITRE ATT&CK technique coverage: alert counts per technique."""+ cypher = """+ MATCH (a:Alert {tenant_id: $tenant_id})-[:MAPS_TO]->(t:Technique)+ RETURN+ t.technique_id AS technique_id,+ t.name AS name,+ t.tactic AS tactic,+ COUNT(a) AS alert_count+ ORDER BY alert_count DESC+ """+ async with get_session() as s:+ result = await s.run(cypher, tenant_id=tenant_id)+ records = await result.data()++ return recordsdiff --git a/services/api/app/services/rule_engine.py b/services/api/app/services/rule_engine.pynew file mode 100644index 000000000..0c4bfbba3--- /dev/null+++ b/services/api/app/services/rule_engine.py@@ -0,0 +1,391 @@+"""+Detection rule execution engine.+Supports: Sigma (pySigma), YARA, KQL (simulated), EQL (simulated).+Cyble Open-Source AI Security Operations Center - MIT License+"""+from __future__ import annotations++import json+import logging+import re+import time+import uuid+from dataclasses import dataclass, field+from datetime import datetime, timezone+from enum import Enum+from typing import Any++logger = logging.getLogger(__name__)+++class RuleLanguage(str, Enum):+ SIGMA = "sigma"+ YARA = "yara"+ KQL = "kql"+ EQL = "eql"+ LUCENE = "lucene"+ REGEX = "regex"+++@dataclass+class RuleMatch:+ """A single rule match result."""+ rule_id: str+ rule_name: str+ rule_language: str+ severity: str+ matched: bool+ match_details: dict[str, Any] = field(default_factory=dict)+ matched_fields: list[str] = field(default_factory=list)+ score: float = 0.0+ error: str | None = None+ execution_time_ms: float = 0.0+++@dataclass+class HuntResult:+ """Result from a threat hunt across events."""+ hunt_id: str+ tenant_id: str+ rules_evaluated: int+ rules_matched: int+ total_events_scanned: int+ matched_events: list[dict[str, Any]]+ match_summary: list[dict[str, Any]]+ execution_time_ms: float+ errors: list[str]+++# ─── Sigma Runner ─────────────────────────────────────────────────────────────++def _run_sigma(rule_body: str, events: list[dict[str, Any]]) -> tuple[list[dict], str | None]:+ """+ Execute a Sigma rule against a list of events.+ Uses pySigma for parsing; falls back to a lightweight YAML-based evaluator.+ Returns (matched_events, error_message).+ """+ try:+ from sigma.rule import SigmaRule+ from sigma.backends.opensearch import OpensearchLuceneBackend+ from sigma.collection import SigmaCollection++ sigma_rule = SigmaRule.from_yaml(rule_body)+ backend = OpensearchLuceneBackend()+ queries = backend.convert_rule(sigma_rule)++ # Use the generated Lucene query to evaluate events+ matched = []+ for event in events:+ for query in queries:+ if _lucene_match(query, event):+ matched.append(event)+ break+ return matched, None++ except ImportError:+ # pySigma not available – use simple YAML condition evaluator+ return _sigma_fallback(rule_body, events), None+ except Exception as exc:+ logger.warning("Sigma parse error", error=str(exc))+ return _sigma_fallback(rule_body, events), str(exc)+++def _sigma_fallback(rule_body: str, events: list[dict[str, Any]]) -> list[dict[str, Any]]:+ """+ Minimal Sigma evaluator for simple field: value conditions.+ Parses the `detection` block and checks field/value presence.+ """+ try:+ import yaml+ rule = yaml.safe_load(rule_body)+ detection = rule.get("detection", {})++ matched = []+ for event in events:+ flat_event = _flatten_dict(event)+ if _eval_sigma_detection(detection, flat_event):+ matched.append(event)+ return matched+ except Exception as exc:+ logger.debug("Sigma fallback evaluator error", error=str(exc))+ return []+++def _eval_sigma_detection(detection: dict, flat_event: dict[str, Any]) -> bool:+ """Evaluate Sigma detection section (keywords + condition)."""+ condition = detection.get("condition", "selection")+ selections: dict[str, bool] = {}++ for sel_name, sel_def in detection.items():+ if sel_name == "condition":+ continue+ if isinstance(sel_def, dict):+ match = True+ for field_name, field_val in sel_def.items():+ field_lower = field_name.lower().rstrip("|contains|startswith|endswith")+ ev_val = str(flat_event.get(field_lower, "")).lower()+ if isinstance(field_val, list):+ hit = any(str(v).lower() in ev_val for v in field_val)+ else:+ hit = str(field_val).lower() in ev_val+ if not hit:+ match = False+ break+ selections[sel_name] = match+ elif isinstance(sel_def, list):+ # keywords list+ flat_str = " ".join(str(v) for v in flat_event.values()).lower()+ selections[sel_name] = all(str(kw).lower() in flat_str for kw in sel_def)++ # Evaluate condition expression (supports: and, or, not, 1 of, all of)+ return _eval_condition(condition, selections)+++def _eval_condition(condition: str, selections: dict[str, bool]) -> bool:+ """Evaluate a Sigma condition string."""+ expr = condition.strip()+ # Replace selection names with their bool values+ for name, val in selections.items():+ expr = expr.replace(name, str(val))+ # Simple evaluation+ try:+ expr = expr.replace("and", " and ").replace("or", " or ").replace("not", " not ")+ # Safe eval with only booleans+ return bool(eval(expr, {"__builtins__": {}}, {"True": True, "False": False})) # noqa: S307+ except Exception:+ # Default: AND all selections+ return all(selections.values())+++# ─── YARA Runner ──────────────────────────────────────────────────────────────++def _run_yara(rule_body: str, events: list[dict[str, Any]]) -> tuple[list[dict], str | None]:+ """+ Execute a YARA rule against event payloads.+ Compiles and matches each event's raw_payload or JSON representation.+ """+ try:+ import yara+ compiled = yara.compile(source=rule_body)+ matched = []+ for event in events:+ payload_bytes = _event_to_bytes(event)+ matches = compiled.match(data=payload_bytes)+ if matches:+ event_copy = dict(event)+ event_copy["_yara_matches"] = [m.rule for m in matches]+ matched.append(event_copy)+ return matched, None+ except ImportError:+ return [], "yara-python not installed"+ except Exception as exc:+ logger.warning("YARA execution error", error=str(exc))+ return [], str(exc)+++def _event_to_bytes(event: dict[str, Any]) -> bytes:+ """Convert event dict to bytes for YARA matching."""+ payload = event.get("raw_payload") or event.get("ocsf_json") or json.dumps(event)+ if isinstance(payload, bytes):+ return payload+ if isinstance(payload, str):+ try:+ return payload.encode("utf-8", errors="replace")+ except Exception:+ return b""+ return json.dumps(payload).encode("utf-8")+++# ─── KQL / EQL Runner (simulated) ─────────────────────────────────────────────++def _run_kql(rule_body: str, events: list[dict[str, Any]]) -> tuple[list[dict], str | None]:+ """+ Simplified KQL evaluator.+ Supports field:value, wildcards, and boolean operators.+ """+ matched = []+ error = None+ try:+ for event in events:+ flat = _flatten_dict(event)+ if _kql_match(rule_body.strip(), flat):+ matched.append(event)+ except Exception as exc:+ error = str(exc)+ return matched, error+++def _kql_match(query: str, flat_event: dict[str, Any]) -> bool:+ """Minimal KQL field:value matcher."""+ # field:value pattern+ m = re.match(r"(\w+)\s*:\s*\"?([^\"\s]+)\"?", query)+ if m:+ field_name = m.group(1).lower()+ value = m.group(2).lower()+ ev_val = str(flat_event.get(field_name, "")).lower()+ if "*" in value:+ pattern = value.replace("*", ".*")+ return bool(re.search(pattern, ev_val))+ return value in ev_val+ # Free-text search+ flat_str = " ".join(str(v) for v in flat_event.values()).lower()+ return query.lower() in flat_str+++def _run_eql(rule_body: str, events: list[dict[str, Any]]) -> tuple[list[dict], str | None]:+ """Simplified EQL sequence evaluator (single-event matching only)."""+ return _run_kql(rule_body, events)+++# ─── Lucene Query Matcher ─────────────────────────────────────────────────────++def _lucene_match(query: str, event: dict[str, Any]) -> bool:+ """Simple Lucene query evaluator for field:value pairs."""+ flat = _flatten_dict(event)+ flat_str = " ".join(f"{k}:{v}" for k, v in flat.items()).lower()+ return query.lower() in flat_str+++def _flatten_dict(d: dict[str, Any], prefix: str = "") -> dict[str, Any]:+ """Flatten nested dict to dot-notation keys."""+ result: dict[str, Any] = {}+ for k, v in d.items():+ key = f"{prefix}.{k}".lstrip(".") if prefix else k+ key_lower = key.lower()+ if isinstance(v, dict):+ result.update(_flatten_dict(v, key_lower))+ else:+ result[key_lower] = v+ return result+++# ─── Main Execute Function ────────────────────────────────────────────────────++def execute_rule(+ rule_id: str,+ rule_name: str,+ rule_language: str,+ rule_body: str,+ severity: str,+ events: list[dict[str, Any]],+) -> RuleMatch:+ """+ Execute a detection rule against a list of events.+ Returns a RuleMatch with matched events and metadata.+ """+ start = time.monotonic()++ lang = rule_language.lower()+ runners = {+ "sigma": _run_sigma,+ "yara": _run_yara,+ "kql": _run_kql,+ "eql": _run_eql,+ "lucene": lambda body, evts: (_run_kql(body, evts)[0], None),+ "regex": lambda body, evts: _run_regex(body, evts),+ }++ runner = runners.get(lang)+ if runner is None:+ return RuleMatch(+ rule_id=rule_id,+ rule_name=rule_name,+ rule_language=rule_language,+ severity=severity,+ matched=False,+ error=f"Unsupported rule language: {rule_language}",+ execution_time_ms=(time.monotonic() - start) * 1000,+ )++ matched_events, error = runner(rule_body, events)+ elapsed_ms = (time.monotonic() - start) * 1000++ return RuleMatch(+ rule_id=rule_id,+ rule_name=rule_name,+ rule_language=rule_language,+ severity=severity,+ matched=len(matched_events) > 0,+ match_details={"matched_events": matched_events[:10]}, # cap to 10 for response size+ score=_severity_score(severity) if matched_events else 0.0,+ error=error,+ execution_time_ms=elapsed_ms,+ )+++def _run_regex(rule_body: str, events: list[dict[str, Any]]) -> tuple[list[dict], str | None]:+ """Regex-based rule runner."""+ try:+ pattern = re.compile(rule_body, re.IGNORECASE | re.MULTILINE)+ matched = []+ for event in events:+ payload = json.dumps(event)+ if pattern.search(payload):+ matched.append(event)+ return matched, None+ except re.error as exc:+ return [], str(exc)+++def _severity_score(severity: str) -> float:+ mapping = {"critical": 1.0, "high": 0.8, "medium": 0.5, "low": 0.2, "info": 0.1}+ return mapping.get(severity.lower(), 0.5)+++# ─── Hunt Runner ─────────────────────────────────────────────────────────────++async def run_hunt(+ tenant_id: str,+ rules: list[dict[str, Any]],+ events: list[dict[str, Any]],+ hunt_id: str | None = None,+) -> HuntResult:+ """+ Run threat hunt: execute multiple detection rules against a set of events.+ """+ hunt_id = hunt_id or str(uuid.uuid4())+ start = time.monotonic()++ matched_events: list[dict[str, Any]] = []+ match_summary: list[dict[str, Any]] = []+ errors: list[str] = []+ rules_matched = 0++ for rule in rules:+ result = execute_rule(+ rule_id=str(rule.get("id", "")),+ rule_name=rule.get("name", ""),+ rule_language=rule.get("rule_language", "sigma"),+ rule_body=rule.get("rule_body", ""),+ severity=rule.get("severity", "medium"),+ events=events,+ )++ if result.error:+ errors.append(f"{result.rule_name}: {result.error}")++ if result.matched:+ rules_matched += 1+ matched_events.extend(result.match_details.get("matched_events", []))+ match_summary.append({+ "rule_id": result.rule_id,+ "rule_name": result.rule_name,+ "severity": result.severity,+ "match_count": len(result.match_details.get("matched_events", [])),+ "score": result.score,+ "execution_time_ms": result.execution_time_ms,+ })++ elapsed_ms = (time.monotonic() - start) * 1000++ return HuntResult(+ hunt_id=hunt_id,+ tenant_id=tenant_id,+ rules_evaluated=len(rules),+ rules_matched=rules_matched,+ total_events_scanned=len(events),+ matched_events=matched_events[:100], # Cap response size+ match_summary=match_summary,+ execution_time_ms=elapsed_ms,+ errors=errors,+ )diff --git a/services/api/clickhouse/001_init.sql b/services/api/clickhouse/001_init.sqlnew file mode 100644index 000000000..43ce5d7e3--- /dev/null+++ b/services/api/clickhouse/001_init.sql@@ -0,0 +1,74 @@+-- AiSOC ClickHouse init: raw events + metrics+-- Runs at container start via /docker-entrypoint-initdb.d++CREATE DATABASE IF NOT EXISTS aisoc;++-- ──────────────────────────────────────────────────────────────────────────────+-- Raw OCSF events (hot tier - 30 days MergeTree)+-- ──────────────────────────────────────────────────────────────────────────────+CREATE TABLE IF NOT EXISTS aisoc.raw_events (+ event_id UUID DEFAULT generateUUIDv4(),+ tenant_id UUID,+ event_time DateTime64(3, 'UTC'),+ ingest_time DateTime64(3, 'UTC') DEFAULT now64(),+ class_uid UInt32,+ category_uid UInt32,+ severity_id UInt8,+ severity String,+ activity_id UInt32,+ source_ip IPv6,+ dest_ip IPv6,+ src_port UInt16,+ dst_port UInt16,+ protocol String,+ src_hostname String,+ dst_hostname String,+ user_name String,+ process_name String,+ file_path String,+ hash_sha256 String,+ connector_type String,+ raw_payload String CODEC(ZSTD(3)),+ ocsf_json String CODEC(ZSTD(3)),+ mitre_techniques Array(String),+ mitre_tactics Array(String),+ iocs Array(String)+) ENGINE = MergeTree()+PARTITION BY (toYYYYMM(event_time), tenant_id)+ORDER BY (tenant_id, event_time, class_uid)+TTL event_time + INTERVAL 90 DAY+SETTINGS index_granularity = 8192;++-- ──────────────────────────────────────────────────────────────────────────────+-- Alert metrics for dashboards+-- ──────────────────────────────────────────────────────────────────────────────+CREATE TABLE IF NOT EXISTS aisoc.alert_metrics (+ ts DateTime DEFAULT now(),+ tenant_id UUID,+ severity String,+ connector_type String,+ mitre_tactic String,+ count UInt64,+ avg_score Float32+) ENGINE = SummingMergeTree(count)+PARTITION BY toYYYYMM(ts)+ORDER BY (tenant_id, toStartOfHour(ts), severity, connector_type, mitre_tactic)+TTL ts + INTERVAL 365 DAY;++-- ──────────────────────────────────────────────────────────────────────────────+-- IOC lookup table (append-only enrichment cache)+-- ──────────────────────────────────────────────────────────────────────────────+CREATE TABLE IF NOT EXISTS aisoc.ioc_enrichments (+ ioc_value String,+ ioc_type String,+ tenant_id UUID,+ malicious UInt8,+ confidence Float32,+ sources Array(String),+ tags Array(String),+ country String,+ asn UInt32,+ enriched_at DateTime64(3, 'UTC') DEFAULT now64()+) ENGINE = ReplacingMergeTree(enriched_at)+ORDER BY (ioc_value, ioc_type, tenant_id)+TTL enriched_at + INTERVAL 30 DAY;diff --git a/services/api/migrations/001_init.sql b/services/api/migrations/001_init.sqlnew file mode 100644index 000000000..d52de6e9d--- /dev/null+++ b/services/api/migrations/001_init.sql@@ -0,0 +1,218 @@+-- AiSOC initial schema+-- Runs on first postgres container start via /docker-entrypoint-initdb.d++CREATE EXTENSION IF NOT EXISTS "uuid-ossp";+CREATE EXTENSION IF NOT EXISTS "pgcrypto";++-- ──────────────────────────────────────────────────────────────────────────────+-- Tenants & Users+-- ──────────────────────────────────────────────────────────────────────────────++CREATE TABLE IF NOT EXISTS tenants (+ id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),+ name VARCHAR(255) NOT NULL,+ slug VARCHAR(100) UNIQUE NOT NULL,+ plan VARCHAR(50) DEFAULT 'starter',+ is_active BOOLEAN DEFAULT TRUE,+ settings JSONB DEFAULT '{}',+ limits JSONB DEFAULT '{}',+ created_at TIMESTAMPTZ DEFAULT NOW(),+ updated_at TIMESTAMPTZ DEFAULT NOW()+);++CREATE TABLE IF NOT EXISTS users (+ id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),+ tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,+ email VARCHAR(255) UNIQUE NOT NULL,+ username VARCHAR(100) NOT NULL,+ hashed_password VARCHAR(255) NOT NULL,+ role VARCHAR(50) DEFAULT 'soc_analyst',+ is_active BOOLEAN DEFAULT TRUE,+ is_verified BOOLEAN DEFAULT FALSE,+ last_login TIMESTAMPTZ,+ preferences JSONB DEFAULT '{}',+ created_at TIMESTAMPTZ DEFAULT NOW(),+ updated_at TIMESTAMPTZ DEFAULT NOW()+);+CREATE INDEX IF NOT EXISTS idx_users_tenant ON users(tenant_id);+CREATE INDEX IF NOT EXISTS idx_users_email ON users(email);++CREATE TABLE IF NOT EXISTS api_keys (+ id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),+ tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,+ user_id UUID REFERENCES users(id) ON DELETE SET NULL,+ name VARCHAR(255) NOT NULL,+ key_prefix VARCHAR(20) NOT NULL,+ hashed_key VARCHAR(255) NOT NULL,+ scopes JSONB DEFAULT '[]',+ is_active BOOLEAN DEFAULT TRUE,+ expires_at TIMESTAMPTZ,+ last_used_at TIMESTAMPTZ,+ created_at TIMESTAMPTZ DEFAULT NOW()+);+CREATE INDEX IF NOT EXISTS idx_api_keys_tenant ON api_keys(tenant_id);+CREATE INDEX IF NOT EXISTS idx_api_keys_hashed ON api_keys(hashed_key);++-- ──────────────────────────────────────────────────────────────────────────────+-- Connectors+-- ──────────────────────────────────────────────────────────────────────────────++CREATE TABLE IF NOT EXISTS connectors (+ id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),+ tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,+ name VARCHAR(255) NOT NULL,+ connector_type VARCHAR(100) NOT NULL,+ vendor VARCHAR(100),+ description TEXT,+ config JSONB DEFAULT '{}',+ credentials_enc JSONB DEFAULT '{}',+ is_enabled BOOLEAN DEFAULT TRUE,+ status VARCHAR(30) DEFAULT 'pending',+ last_sync_at TIMESTAMPTZ,+ last_error TEXT,+ events_ingested BIGINT DEFAULT 0,+ created_at TIMESTAMPTZ DEFAULT NOW(),+ updated_at TIMESTAMPTZ DEFAULT NOW()+);+CREATE INDEX IF NOT EXISTS idx_connectors_tenant ON connectors(tenant_id);++-- ──────────────────────────────────────────────────────────────────────────────+-- Alerts+-- ──────────────────────────────────────────────────────────────────────────────++CREATE TABLE IF NOT EXISTS alerts (+ id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),+ tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,+ title VARCHAR(500) NOT NULL,+ description TEXT,+ severity VARCHAR(20) NOT NULL DEFAULT 'medium',+ status VARCHAR(30) NOT NULL DEFAULT 'new',+ priority INTEGER DEFAULT 50,+ category VARCHAR(100),+ mitre_tactics JSONB DEFAULT '[]',+ mitre_techniques JSONB DEFAULT '[]',+ connector_id UUID,+ connector_type VARCHAR(100),+ source_event_ids JSONB DEFAULT '[]',+ ocsf_class_uid INTEGER,+ ai_score FLOAT,+ ai_summary TEXT,+ ai_recommendations JSONB DEFAULT '[]',+ false_positive_score FLOAT,+ anomaly_score FLOAT,+ iocs JSONB DEFAULT '[]',+ entities JSONB DEFAULT '[]',+ raw_event JSONB DEFAULT '{}',+ dedup_hash VARCHAR(64),+ case_id UUID,+ first_seen TIMESTAMPTZ DEFAULT NOW(),+ last_seen TIMESTAMPTZ DEFAULT NOW(),+ event_time TIMESTAMPTZ DEFAULT NOW(),+ resolved_at TIMESTAMPTZ,+ created_at TIMESTAMPTZ DEFAULT NOW(),+ updated_at TIMESTAMPTZ DEFAULT NOW()+);+CREATE INDEX IF NOT EXISTS idx_alerts_tenant ON alerts(tenant_id);+CREATE INDEX IF NOT EXISTS idx_alerts_severity ON alerts(severity);+CREATE INDEX IF NOT EXISTS idx_alerts_status ON alerts(status);+CREATE INDEX IF NOT EXISTS idx_alerts_case ON alerts(case_id);+CREATE INDEX IF NOT EXISTS idx_alerts_dedup ON alerts(dedup_hash);+CREATE INDEX IF NOT EXISTS idx_alerts_event_t ON alerts(event_time DESC);++-- ──────────────────────────────────────────────────────────────────────────────+-- Cases+-- ──────────────────────────────────────────────────────────────────────────────++CREATE TABLE IF NOT EXISTS cases (+ id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),+ tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,+ case_number VARCHAR(30) UNIQUE NOT NULL,+ title VARCHAR(500) NOT NULL,+ description TEXT,+ status VARCHAR(30) DEFAULT 'open',+ priority VARCHAR(20) DEFAULT 'medium',+ severity VARCHAR(20) DEFAULT 'medium',+ case_type VARCHAR(50) DEFAULT 'security_incident',+ mitre_tactics JSONB DEFAULT '[]',+ mitre_techniques JSONB DEFAULT '[]',+ assigned_to_id UUID,+ assigned_at TIMESTAMPTZ,+ created_by_id UUID,+ sla_deadline TIMESTAMPTZ,+ sla_breached BOOLEAN DEFAULT FALSE,+ alert_ids JSONB DEFAULT '[]',+ ioc_ids JSONB DEFAULT '[]',+ artifact_ids JSONB DEFAULT '[]',+ tags JSONB DEFAULT '[]',+ ticket_refs JSONB DEFAULT '[]',+ summary TEXT,+ closed_at TIMESTAMPTZ,+ created_at TIMESTAMPTZ DEFAULT NOW(),+ updated_at TIMESTAMPTZ DEFAULT NOW()+);+CREATE INDEX IF NOT EXISTS idx_cases_tenant ON cases(tenant_id);+CREATE INDEX IF NOT EXISTS idx_cases_status ON cases(status);+CREATE INDEX IF NOT EXISTS idx_cases_priority ON cases(priority);++CREATE TABLE IF NOT EXISTS case_timeline_events (+ id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),+ case_id UUID NOT NULL REFERENCES cases(id) ON DELETE CASCADE,+ tenant_id UUID NOT NULL,+ event_type VARCHAR(50) NOT NULL,+ actor_id UUID,+ actor_type VARCHAR(30) DEFAULT 'user',+ summary TEXT NOT NULL,+ detail JSONB DEFAULT '{}',+ created_at TIMESTAMPTZ DEFAULT NOW()+);+CREATE INDEX IF NOT EXISTS idx_case_timeline_case ON case_timeline_events(case_id);++-- ──────────────────────────────────────────────────────────────────────────────+-- Detection Rules+-- ──────────────────────────────────────────────────────────────────────────────++CREATE TABLE IF NOT EXISTS detection_rules (+ id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),+ tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,+ name VARCHAR(255) NOT NULL,+ description TEXT,+ rule_type VARCHAR(20) NOT NULL DEFAULT 'sigma',+ rule_content TEXT NOT NULL,+ severity VARCHAR(20) DEFAULT 'medium',+ tags JSONB DEFAULT '[]',+ mitre_tactics JSONB DEFAULT '[]',+ mitre_techniques JSONB DEFAULT '[]',+ enabled BOOLEAN DEFAULT TRUE,+ version INTEGER DEFAULT 1,+ last_run_at TIMESTAMPTZ,+ last_hit_at TIMESTAMPTZ,+ hit_count BIGINT DEFAULT 0,+ author VARCHAR(255),+ source VARCHAR(100) DEFAULT 'custom',+ created_at TIMESTAMPTZ DEFAULT NOW(),+ updated_at TIMESTAMPTZ DEFAULT NOW()+);+CREATE INDEX IF NOT EXISTS idx_rules_tenant ON detection_rules(tenant_id);+CREATE INDEX IF NOT EXISTS idx_rules_enabled ON detection_rules(enabled);+CREATE INDEX IF NOT EXISTS idx_rules_type ON detection_rules(rule_type);++-- ──────────────────────────────────────────────────────────────────────────────+-- Seed: default tenant & admin user+-- ──────────────────────────────────────────────────────────────────────────────++INSERT INTO tenants (id, name, slug, plan)+VALUES ('00000000-0000-0000-0000-000000000001', 'Default', 'default', 'enterprise')+ON CONFLICT (slug) DO NOTHING;++-- password = "admin" (bcrypt)+INSERT INTO users (id, tenant_id, email, username, hashed_password, role, is_active, is_verified)+VALUES (+ '00000000-0000-0000-0000-000000000002',+ '00000000-0000-0000-0000-000000000001',+ '[email protected]',+ 'admin',+ '$2b$12$LQv3c1yqBWVHxkd0LHAkCOYz6TtxMQJqhN8/LewdBPj3EEbF7FtRS',+ 'admin',+ TRUE,+ TRUE+) ON CONFLICT (email) DO NOTHING;diff --git a/services/api/pyproject.toml b/services/api/pyproject.tomlindex 8146f3aee..ee70394cf 100644--- a/services/api/pyproject.toml+++ b/services/api/pyproject.toml@@ -28,6 +28,11 @@ opentelemetry-instrumentation-fastapi = "^0.45b0"opentelemetry-instrumentation-sqlalchemy = "^0.45b0"clickhouse-driver = "^0.2.8"boto3 = "^1.34.0"+neo4j = "^5.19.0"+pysigma = "^0.11.0"+pysigma-backend-opensearch = "^1.0.0"+yara-python = "^4.5.0"+pyyaml = "^6.0.1"[tool.poetry.group.dev.dependencies]pytest = "^8.2.0"diff --git a/services/fusion/Dockerfile b/services/fusion/Dockerfileindex fde781dc3..6cbd57c74 100644--- a/services/fusion/Dockerfile+++ b/services/fusion/Dockerfile@@ -12,6 +12,6 @@ RUN poetry install --no-interaction --no-ansi --no-dev 2>/dev/null || \COPY app/ ./app/-EXPOSE 8083+EXPOSE 8003-CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8083"]+CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8003"]diff --git a/services/fusion/app/api/router.py b/services/fusion/app/api/router.pyindex 85870f67c..f21c3fbb0 100644--- a/services/fusion/app/api/router.py+++ b/services/fusion/app/api/router.py@@ -1,4 +1,8 @@-from fastapi import APIRouter+from __future__ import annotations++from fastapi import APIRouter, HTTPException++from app.models.alert import AnalystFeedbackfrom app.workers.consumer import FusionWorkerrouter = APIRouter()@@ -21,3 +25,29 @@ async def metrics():if _worker_ref is None:return {"status": "worker not started"}return {"status": "ok", "metrics": FusionWorker.get_metrics()}++[email protected]("/ml/status")+async def ml_status():+ """Return current ML model training status."""+ if _worker_ref is None or _worker_ref.engine is None:+ raise HTTPException(status_code=503, detail="Fusion worker not ready")+ return _worker_ref.engine.ml_scorer.status()++[email protected]("/ml/feedback")+async def submit_feedback(feedback: AnalystFeedback):+ """Submit analyst feedback to improve ML ranker."""+ if _worker_ref is None or _worker_ref.engine is None:+ raise HTTPException(status_code=503, detail="Fusion worker not ready")+ await _worker_ref.engine.ml_scorer.record_feedback(feedback)+ return {"status": "accepted", "alert_id": str(feedback.alert_id)}++[email protected]("/ml/retrain")+async def trigger_retrain():+ """Manually trigger ML model retraining."""+ if _worker_ref is None or _worker_ref.engine is None:+ raise HTTPException(status_code=503, detail="Fusion worker not ready")+ result = await _worker_ref.engine.ml_scorer.retrain()+ return resultdiff --git a/services/fusion/app/core/config.py b/services/fusion/app/core/config.pyindex 6a0bed2ea..404a76e07 100644--- a/services/fusion/app/core/config.py+++ b/services/fusion/app/core/config.py@@ -5,7 +5,7 @@ from pydantic import Fieldclass Settings(BaseSettings):# Serviceservice_name: str = "aisoc-fusion"- http_port: int = Field(default=8083, alias="HTTP_PORT")+ http_port: int = Field(default=8003, alias="HTTP_PORT")log_level: str = Field(default="INFO", alias="LOG_LEVEL")environment: str = Field(default="development", alias="ENVIRONMENT")diff --git a/services/fusion/app/models/alert.py b/services/fusion/app/models/alert.pyindex 27d3167a9..1678fd48e 100644--- a/services/fusion/app/models/alert.py+++ b/services/fusion/app/models/alert.py@@ -107,9 +107,25 @@ class FusedAlert(BaseModel):# Enrichment data (populated by enrichment service)enrichments: dict[str, Any] = Field(default_factory=dict)+ # ML scores — populated by MLScorer+ anomaly_score: float = 0.0 # 0.0 = normal, 1.0 = highly anomalous (Isolation Forest)+ priority_score: float = 0.0 # 0.0–1.0 priority rank (LightGBM ranker)+fused_at: datetime = Field(default_factory=datetime.utcnow)+class AnalystFeedback(BaseModel):+ """Analyst feedback on a fused alert, used to re-train the ML ranker."""++ alert_id: UUID+ tenant_id: UUID+ analyst_id: str+ is_true_positive: bool+ assigned_priority: int = Field(ge=1, le=5, description="Analyst-assigned priority 1 (critical) to 5 (low)")+ notes: str = ""+ submitted_at: datetime = Field(default_factory=datetime.utcnow)++class IncidentSummary(BaseModel):"""Lightweight incident summary stored in Redis."""diff --git a/services/fusion/app/services/fusion_engine.py b/services/fusion/app/services/fusion_engine.pyindex fc6e5c471..69005a418 100644--- a/services/fusion/app/services/fusion_engine.py+++ b/services/fusion/app/services/fusion_engine.py@@ -1,6 +1,8 @@"""-Core fusion engine: orchestrates deduplication → correlation → enrichment enrichment trigger.+Core fusion engine: orchestrates deduplication → correlation → ML scoring."""+from __future__ import annotations+from uuid import UUIDimport structlog@@ -8,6 +10,7 @@ import structlogfrom app.models.alert import FusedAlert, FusionDecision, RawAlertfrom app.services.correlator import Correlatorfrom app.services.deduplicator import Deduplicator+from app.services.ml_scorer import MLScorerlogger = structlog.get_logger()@@ -15,9 +18,15 @@ logger = structlog.get_logger()class FusionEngine:"""Orchestrates the full alert fusion pipeline."""- def __init__(self, deduplicator: Deduplicator, correlator: Correlator) -> None:+ def __init__(+ self,+ deduplicator: Deduplicator,+ correlator: Correlator,+ ml_scorer: MLScorer | None = None,+ ) -> None:self._dedup = deduplicatorself._correlator = correlator+ self._ml_scorer = ml_scorer or MLScorer()async def process(self, alert: RawAlert) -> FusedAlert:"""@@ -26,7 +35,7 @@ class FusionEngine:Pipeline:1. Deduplication: suppress exact/near-exact duplicates2. Correlation: group into an existing or new incident- 3. Return a FusedAlert with decision metadata+ 3. ML scoring: anomaly_score (Isolation Forest) + priority_score (LightGBM)"""# --- Step 1: Deduplication ---is_dup, original_id = await self._dedup.is_duplicate(alert)@@ -55,19 +64,33 @@ class FusionEngine:FusionDecision.CORRELATED if correlated else FusionDecision.NEW_INCIDENT)+ fused = FusedAlert(+ id=alert.id,+ tenant_id=alert.tenant_id,+ incident_id=incident.id,+ fusion_decision=decision,+ duplicate_of=None,+ alert=alert,+ )++ # --- Step 3: ML scoring ---+ try:+ fused = await self._ml_scorer.score(fused)+ except Exception as exc:+ logger.warning("ML scoring failed; using defaults", error=str(exc))+logger.info("Alert fusion complete",alert_id=str(alert.id),decision=decision,incident_id=str(incident.id),incident_alert_count=incident.alert_count,+ anomaly_score=fused.anomaly_score,+ priority_score=fused.priority_score,)- return FusedAlert(- id=alert.id,- tenant_id=alert.tenant_id,- incident_id=incident.id,- fusion_decision=decision,- duplicate_of=None,- alert=alert,- )+ return fused++ @property+ def ml_scorer(self) -> MLScorer:+ return self._ml_scorerdiff --git a/services/fusion/app/services/ml_scorer.py b/services/fusion/app/services/ml_scorer.pynew file mode 100644index 000000000..d074c0e2d--- /dev/null+++ b/services/fusion/app/services/ml_scorer.py@@ -0,0 +1,341 @@+"""+ML-based anomaly and priority scoring for alerts.++Anomaly score — Isolation Forest trained on alert feature vectors.+ Outputs 0.0 (normal) → 1.0 (highly anomalous).++Priority score — LightGBM ranker trained on analyst feedback signals+ (true-positive labels + assigned priority).+ Outputs 0.0 → 1.0 (higher = more urgent).++Both models are held in memory and retrained on demand when enough+feedback accumulates. They fall back to heuristics if not yet trained.++Cyble Open-Source AI Security Operations Center — MIT License+"""+from __future__ import annotations++import asyncio+import json+import math+from datetime import datetime+from typing import Any++import structlog++from app.models.alert import AnalystFeedback, FusedAlert, RawAlert++logger = structlog.get_logger()++# ─────────────────────────────────────────────────────────────────────────────+# Feature engineering helpers+# ─────────────────────────────────────────────────────────────────────────────++_SEVERITY_MAP = {"critical": 4, "high": 3, "medium": 2, "low": 1, "info": 0}+++def _featurize(alert: RawAlert) -> list[float]:+ """Convert a RawAlert to a fixed-length numeric feature vector."""+ sev = _SEVERITY_MAP.get(alert.severity.value if hasattr(alert.severity, "value") else str(alert.severity), 2)+ has_src_ip = 1.0 if alert.src_ip else 0.0+ has_dst_ip = 1.0 if alert.dst_ip else 0.0+ has_hostname = 1.0 if alert.hostname else 0.0+ has_username = 1.0 if alert.username else 0.0+ has_file_hash = 1.0 if alert.file_hash else 0.0+ has_domain = 1.0 if alert.domain else 0.0+ has_url = 1.0 if alert.url else 0.0+ num_tactics = float(len(alert.mitre_tactics))+ num_techniques = float(len(alert.mitre_techniques))+ num_tags = float(len(alert.tags))+ risk_score = float(alert.risk_score) if alert.risk_score else 0.0+ hour_of_day = float(alert.created_at.hour) if alert.created_at else 12.0+ return [+ float(sev),+ has_src_ip,+ has_dst_ip,+ has_hostname,+ has_username,+ has_file_hash,+ has_domain,+ has_url,+ num_tactics,+ num_techniques,+ num_tags,+ risk_score,+ hour_of_day,+ ]+++# ─────────────────────────────────────────────────────────────────────────────+# Heuristic fallbacks (used when ML models aren't trained yet)+# ─────────────────────────────────────────────────────────────────────────────++def _heuristic_anomaly(alert: RawAlert) -> float:+ """Simple heuristic anomaly score: more IOC fields = potentially more anomalous."""+ score = 0.0+ if alert.file_hash:+ score += 0.3+ if alert.src_ip and alert.dst_ip:+ score += 0.2+ if len(alert.mitre_techniques) > 2:+ score += min(0.3, len(alert.mitre_techniques) * 0.05)+ sev = _SEVERITY_MAP.get(alert.severity.value if hasattr(alert.severity, "value") else str(alert.severity), 2)+ score += sev * 0.05+ return min(score, 1.0)+++def _heuristic_priority(alert: RawAlert) -> float:+ """Simple heuristic priority score based on severity and IOC density."""+ sev = _SEVERITY_MAP.get(alert.severity.value if hasattr(alert.severity, "value") else str(alert.severity), 2)+ base = sev / 4.0+ ioc_bonus = sum([+ 0.1 if alert.src_ip else 0.0,+ 0.05 if alert.hostname else 0.0,+ 0.05 if alert.username else 0.0,+ 0.1 if alert.file_hash else 0.0,+ 0.05 if alert.mitre_techniques else 0.0,+ ])+ return min(base + ioc_bonus, 1.0)+++# ─────────────────────────────────────────────────────────────────────────────+# MLScorer class+# ─────────────────────────────────────────────────────────────────────────────++_MIN_SAMPLES_FOR_TRAINING = 50+_MIN_FEEDBACK_FOR_RANKER = 30+++class MLScorer:+ """+ Wraps scikit-learn Isolation Forest + LightGBM LambdaRank for alert scoring.++ Models are trained lazily once enough data accumulates. Until then,+ heuristic fallbacks are used so the service is always available.+ """++ def __init__(self) -> None:+ self._iso_forest = None # sklearn IsolationForest+ self._lgbm_ranker = None # lightgbm LGBMRanker+ self._iso_trained = False+ self._lgbm_trained = False++ # Rolling in-memory training buffers+ self._feature_buffer: list[list[float]] = []+ self._feedback_buffer: list[dict[str, Any]] = []++ self._lock = asyncio.Lock()++ # ─── Public API ──────────────────────────────────────────────────────────++ async def score(self, fused_alert: FusedAlert) -> FusedAlert:+ """Compute and attach anomaly_score and priority_score to the fused alert."""+ alert = fused_alert.alert+ features = _featurize(alert)++ # Buffer features for future training+ async with self._lock:+ self._feature_buffer.append(features)++ if self._iso_trained and self._iso_forest is not None:+ anomaly_score = await asyncio.get_event_loop().run_in_executor(+ None, self._predict_anomaly, features+ )+ else:+ anomaly_score = _heuristic_anomaly(alert)++ if self._lgbm_trained and self._lgbm_ranker is not None:+ priority_score = await asyncio.get_event_loop().run_in_executor(+ None, self._predict_priority, features+ )+ else:+ priority_score = _heuristic_priority(alert)++ fused_alert.anomaly_score = round(anomaly_score, 4)+ fused_alert.priority_score = round(priority_score, 4)++ logger.debug(+ "ML scores computed",+ alert_id=str(fused_alert.id),+ anomaly_score=fused_alert.anomaly_score,+ priority_score=fused_alert.priority_score,+ iso_trained=self._iso_trained,+ lgbm_trained=self._lgbm_trained,+ )+ return fused_alert++ async def record_feedback(self, feedback: AnalystFeedback) -> None:+ """Record analyst feedback and trigger re-training if threshold reached."""+ async with self._lock:+ self._feedback_buffer.append(feedback.model_dump(mode="json"))++ # Trigger background re-training checks+ await self._maybe_retrain()++ async def retrain(self) -> dict[str, Any]:+ """Force a re-training of both models. Returns training summary."""+ async with self._lock:+ features = list(self._feature_buffer)+ feedback = list(self._feedback_buffer)++ iso_result = await asyncio.get_event_loop().run_in_executor(+ None, self._train_isolation_forest, features+ )+ lgbm_result = await asyncio.get_event_loop().run_in_executor(+ None, self._train_lgbm_ranker, features, feedback+ )++ return {+ "isolation_forest": iso_result,+ "lgbm_ranker": lgbm_result,+ "feature_samples": len(features),+ "feedback_samples": len(feedback),+ "retrained_at": datetime.utcnow().isoformat(),+ }++ def status(self) -> dict[str, Any]:+ """Return current model status."""+ return {+ "isolation_forest_trained": self._iso_trained,+ "lgbm_ranker_trained": self._lgbm_trained,+ "feature_buffer_size": len(self._feature_buffer),+ "feedback_buffer_size": len(self._feedback_buffer),+ "min_samples_for_iso": _MIN_SAMPLES_FOR_TRAINING,+ "min_feedback_for_lgbm": _MIN_FEEDBACK_FOR_RANKER,+ }++ # ─── Private: predictions ────────────────────────────────────────────────++ def _predict_anomaly(self, features: list[float]) -> float:+ """Run IsolationForest prediction synchronously."""+ try:+ import numpy as np+ X = np.array(features).reshape(1, -1)+ # decision_function: negative = anomaly, normalize to [0,1]+ raw = self._iso_forest.decision_function(X)[0]+ # Map decision score: lower = more anomalous+ # Typical range is -0.5 to +0.5; map to 0→1 (inverted)+ score = 1.0 / (1.0 + math.exp(5 * raw)) # sigmoid inversion+ return float(score)+ except Exception as exc:+ logger.warning("IsolationForest predict error", error=str(exc))+ return 0.0++ def _predict_priority(self, features: list[float]) -> float:+ """Run LightGBM ranker prediction synchronously."""+ try:+ import numpy as np+ X = np.array(features).reshape(1, -1)+ raw = self._lgbm_ranker.predict(X)[0]+ # Normalize to [0,1] via sigmoid+ score = 1.0 / (1.0 + math.exp(-raw))+ return float(score)+ except Exception as exc:+ logger.warning("LightGBM predict error", error=str(exc))+ return 0.0++ # ─── Private: training ───────────────────────────────────────────────────++ def _train_isolation_forest(self, features: list[list[float]]) -> dict[str, Any]:+ """Train Isolation Forest on accumulated feature vectors."""+ if len(features) < _MIN_SAMPLES_FOR_TRAINING:+ return {"status": "skipped", "reason": f"need {_MIN_SAMPLES_FOR_TRAINING} samples, have {len(features)}"}+ try:+ import numpy as np+ from sklearn.ensemble import IsolationForest++ X = np.array(features)+ model = IsolationForest(+ n_estimators=200,+ contamination=0.1,+ random_state=42,+ n_jobs=-1,+ )+ model.fit(X)+ self._iso_forest = model+ self._iso_trained = True+ logger.info("IsolationForest retrained", samples=len(features))+ return {"status": "trained", "samples": len(features)}+ except Exception as exc:+ logger.error("IsolationForest training failed", error=str(exc))+ return {"status": "error", "error": str(exc)}++ def _train_lgbm_ranker(+ self,+ features: list[list[float]],+ feedback: list[dict[str, Any]],+ ) -> dict[str, Any]:+ """Train LightGBM ranker on analyst feedback."""+ if len(feedback) < _MIN_FEEDBACK_FOR_RANKER:+ return {"status": "skipped", "reason": f"need {_MIN_FEEDBACK_FOR_RANKER} feedback items, have {len(feedback)}"}+ try:+ import numpy as np+ import lightgbm as lgb++ # Build training set from feedback + heuristic features+ X_rows = []+ y_rows = []+ qid_rows = []++ for i, fb in enumerate(feedback):+ # Use heuristic features if no stored features match+ feat = features[i % len(features)] if features else [0.0] * 13++ is_tp = 1.0 if fb.get("is_true_positive", False) else 0.0+ # assigned_priority 1 (critical) → 5 (low) — invert to relevance 4 → 0+ priority_rel = max(0, 5 - int(fb.get("assigned_priority", 3)))+ relevance = int(is_tp * 2 + priority_rel)++ X_rows.append(feat)+ y_rows.append(relevance)+ qid_rows.append(0) # single query group++ X = np.array(X_rows)+ y = np.array(y_rows, dtype=int)+ groups = [len(y_rows)]++ model = lgb.LGBMRanker(+ objective="lambdarank",+ metric="ndcg",+ n_estimators=200,+ learning_rate=0.05,+ num_leaves=31,+ random_state=42,+ verbose=-1,+ )+ model.fit(X, y, group=groups)+ self._lgbm_ranker = model+ self._lgbm_trained = True+ logger.info("LightGBM ranker retrained", feedback_samples=len(feedback))+ return {"status": "trained", "feedback_samples": len(feedback)}+ except Exception as exc:+ logger.error("LightGBM training failed", error=str(exc))+ return {"status": "error", "error": str(exc)}++ # ─── Private: auto-retrain ────────────────────────────────────────────────++ async def _maybe_retrain(self) -> None:+ """Trigger retraining if thresholds are met."""+ async with self._lock:+ n_features = len(self._feature_buffer)+ n_feedback = len(self._feedback_buffer)++ should_iso = (not self._iso_trained and n_features >= _MIN_SAMPLES_FOR_TRAINING) or \+ (self._iso_trained and n_features % 500 == 0)+ should_lgbm = (not self._lgbm_trained and n_feedback >= _MIN_FEEDBACK_FOR_RANKER) or \+ (self._lgbm_trained and n_feedback % 100 == 0)++ if should_iso or should_lgbm:+ asyncio.create_task(self._retrain_background(should_iso, should_lgbm))++ async def _retrain_background(self, do_iso: bool, do_lgbm: bool) -> None:+ """Background task: run training without blocking request handling."""+ async with self._lock:+ features = list(self._feature_buffer)+ feedback = list(self._feedback_buffer)++ loop = asyncio.get_event_loop()+ if do_iso:+ await loop.run_in_executor(None, self._train_isolation_forest, features)+ if do_lgbm:+ await loop.run_in_executor(None, self._train_lgbm_ranker, features, feedback)diff --git a/services/fusion/app/workers/consumer.py b/services/fusion/app/workers/consumer.pyindex dfc7dcbdc..a5ea5f950 100644--- a/services/fusion/app/workers/consumer.py+++ b/services/fusion/app/workers/consumer.py@@ -33,6 +33,10 @@ class FusionWorker:self._producer: AIOKafkaProducer | None = Noneself._running = False+ @property+ def engine(self) -> FusionEngine:+ return self._engine+async def start(self) -> None:self._consumer = AIOKafkaConsumer(settings.kafka_topic_alerts_raw,diff --git a/services/fusion/pyproject.toml b/services/fusion/pyproject.tomlindex 6f8744938..9747b2f00 100644--- a/services/fusion/pyproject.toml+++ b/services/fusion/pyproject.toml@@ -22,6 +22,9 @@ prometheus-client = "^0.19.0"httpx = "^0.26.0"python-dateutil = "^2.8.2"simhash = "^2.1.2"+scikit-learn = "^1.4.0"+lightgbm = "^4.3.0"+numpy = "^1.26.0"[tool.poetry.group.dev.dependencies]pytest = "^7.4.0"diff --git a/services/ingest/internal/attck/loader.go b/services/ingest/internal/attck/loader.gonew file mode 100644index 000000000..7c3b558f8--- /dev/null+++ b/services/ingest/internal/attck/loader.go@@ -0,0 +1,301 @@+// Package attck provides a lightweight MITRE ATT&CK technique index+// built from the STIX 2.1 enterprise bundle.+package attck++import (+ "encoding/json"+ "fmt"+ "io"+ "net/http"+ "os"+ "strings"+ "sync"+ "time"++ "github.com/rs/zerolog/log"+)++const (+ cdnURL = "https://raw.githubusercontent.com/mitre/cti/master/enterprise-attack/enterprise-attack.json"+ cacheTTLHours = 24+)++// Technique holds key ATT&CK technique fields.+type Technique struct {+ ID string `json:"id"`+ Name string `json:"name"`+ Description string `json:"description"`+ TacticIDs []string `json:"tactic_ids"`+ TacticNames []string `json:"tactic_names"`+ Platforms []string `json:"platforms"`+ URL string `json:"url"`+}++var (+ mu sync.RWMutex+ techniques = map[string]*Technique{}+ tacticMap = map[string]string{} // phase_name → display_name+ loaded bool+ loadedAt time.Time+)++// Loaded returns true if the corpus has been successfully loaded.+func Loaded() bool {+ mu.RLock()+ defer mu.RUnlock()+ return loaded+}++// TechniqueCount returns the number of loaded techniques.+func TechniqueCount() int {+ mu.RLock()+ defer mu.RUnlock()+ return len(techniques)+}++// Lookup returns technique metadata for the given ATT&CK ID (e.g. "T1059").+// Returns nil if not found.+func Lookup(techniqueID string) *Technique {+ mu.RLock()+ defer mu.RUnlock()+ return techniques[techniqueID]+}++// LookupAll returns all loaded techniques.+func LookupAll() map[string]*Technique {+ mu.RLock()+ defer mu.RUnlock()+ out := make(map[string]*Technique, len(techniques))+ for k, v := range techniques {+ out[k] = v+ }+ return out+}++// Load loads (or reloads) the ATT&CK corpus from localPath or CDN.+func Load(localPath string) error {+ mu.Lock()+ defer mu.Unlock()++ // Honour cache TTL+ if loaded && time.Since(loadedAt) < cacheTTLHours*time.Hour {+ return nil+ }++ bundle, err := readBundle(localPath)+ if err != nil {+ log.Warn().Err(err).Msg("ATT&CK local bundle unavailable; downloading from CDN")+ bundle, err = downloadBundle(cdnURL)+ if err != nil {+ return fmt.Errorf("attck: failed to load corpus: %w", err)+ }+ if saveErr := saveBundle(localPath, bundle); saveErr != nil {+ log.Warn().Err(saveErr).Str("path", localPath).Msg("Failed to cache ATT&CK bundle locally")+ }+ }++ parseBundle(bundle)+ loaded = true+ loadedAt = time.Now()+ log.Info().+ Int("techniques", len(techniques)).+ Int("tactics", len(tacticMap)).+ Msg("MITRE ATT&CK corpus loaded")+ return nil+}++// ─── Internal helpers ─────────────────────────────────────────────────────────++type stixBundle struct {+ Objects []map[string]interface{} `json:"objects"`+}++func readBundle(path string) ([]byte, error) {+ f, err := os.Open(path)+ if err != nil {+ return nil, err+ }+ defer f.Close()+ return io.ReadAll(f)+}++func downloadBundle(url string) ([]byte, error) {+ client := &http.Client{Timeout: 120 * time.Second}+ resp, err := client.Get(url)+ if err != nil {+ return nil, err+ }+ defer resp.Body.Close()+ if resp.StatusCode != http.StatusOK {+ return nil, fmt.Errorf("CDN returned %d", resp.StatusCode)+ }+ return io.ReadAll(resp.Body)+}++func saveBundle(path string, data []byte) error {+ if err := os.MkdirAll(dirOf(path), 0o755); err != nil {+ return err+ }+ return os.WriteFile(path, data, 0o644)+}++func dirOf(path string) string {+ idx := strings.LastIndex(path, "/")+ if idx < 0 {+ return "."+ }+ return path[:idx]+}++func parseBundle(raw []byte) {+ var bundle stixBundle+ if err := json.Unmarshal(raw, &bundle); err != nil {+ log.Error().Err(err).Msg("Failed to parse ATT&CK STIX bundle")+ return+ }++ // Clear previous state+ techniques = map[string]*Technique{}+ tacticMap = map[string]string{}++ // Index all objects by STIX ID+ byID := make(map[string]map[string]interface{}, len(bundle.Objects))+ for _, obj := range bundle.Objects {+ if id, ok := obj["id"].(string); ok {+ byID[id] = obj+ }+ }++ // Build tactic phase → name map+ for _, obj := range bundle.Objects {+ if t, ok := obj["type"].(string); !ok || t != "x-mitre-tactic" {+ continue+ }+ shortname, _ := obj["x_mitre_shortname"].(string)+ name, _ := obj["name"].(string)+ if shortname != "" && name != "" {+ tacticMap[shortname] = name+ }+ }++ // Build technique index+ for _, obj := range bundle.Objects {+ t, ok := obj["type"].(string)+ if !ok || t != "attack-pattern" {+ continue+ }+ // Skip deprecated/revoked+ if dep, ok := obj["x_mitre_deprecated"].(bool); ok && dep {+ continue+ }+ if rev, ok := obj["revoked"].(bool); ok && rev {+ continue+ }++ techID := extractAttckID(obj)+ if techID == "" {+ continue+ }++ name, _ := obj["name"].(string)+ desc, _ := obj["description"].(string)+ if len(desc) > 1000 {+ desc = desc[:1000]+ }++ tacticIDs, tacticNames := extractTactics(obj)+ platforms := extractStringSlice(obj, "x_mitre_platforms")+ url := extractAttckURL(obj)++ techniques[techID] = &Technique{+ ID: techID,+ Name: name,+ Description: desc,+ TacticIDs: tacticIDs,+ TacticNames: tacticNames,+ Platforms: platforms,+ URL: url,+ }+ }+}++func extractAttckID(obj map[string]interface{}) string {+ refs, ok := obj["external_references"].([]interface{})+ if !ok {+ return ""+ }+ for _, r := range refs {+ ref, ok := r.(map[string]interface{})+ if !ok {+ continue+ }+ if src, _ := ref["source_name"].(string); src == "mitre-attack" {+ if id, ok := ref["external_id"].(string); ok {+ return id+ }+ }+ }+ return ""+}++func extractAttckURL(obj map[string]interface{}) string {+ refs, ok := obj["external_references"].([]interface{})+ if !ok {+ return ""+ }+ for _, r := range refs {+ ref, ok := r.(map[string]interface{})+ if !ok {+ continue+ }+ if src, _ := ref["source_name"].(string); src == "mitre-attack" {+ if u, ok := ref["url"].(string); ok {+ return u+ }+ }+ }+ return ""+}++func extractTactics(obj map[string]interface{}) ([]string, []string) {+ phases, ok := obj["kill_chain_phases"].([]interface{})+ if !ok {+ return nil, nil+ }+ var ids, names []string+ for _, p := range phases {+ phase, ok := p.(map[string]interface{})+ if !ok {+ continue+ }+ kcName, _ := phase["kill_chain_name"].(string)+ if kcName != "mitre-attack" {+ continue+ }+ phaseName, _ := phase["phase_name"].(string)+ if phaseName == "" {+ continue+ }+ ids = append(ids, phaseName)+ if displayName, ok := tacticMap[phaseName]; ok {+ names = append(names, displayName)+ } else {+ names = append(names, phaseName)+ }+ }+ return ids, names+}++func extractStringSlice(obj map[string]interface{}, key string) []string {+ raw, ok := obj[key].([]interface{})+ if !ok {+ return nil+ }+ out := make([]string, 0, len(raw))+ for _, v := range raw {+ if s, ok := v.(string); ok {+ out = append(out, s)+ }+ }+ return out+}diff --git a/services/ingest/internal/config/config.go b/services/ingest/internal/config/config.goindex 5fd9094c5..7315a7695 100644--- a/services/ingest/internal/config/config.go+++ b/services/ingest/internal/config/config.go@@ -21,6 +21,16 @@ type Config struct {TenantHeaderKey stringJWTSecret stringMetricsPort int++ // Shodan enrichment+ ShodanAPIKey string+ ShodanEnrichEnabled bool+ ShodanCacheExpirySecs int++ // CVE / vulnerability correlation+ VulnCorrelEnabled bool+ VulnKafkaTopic string // topic for VULNERABILITY_MATCH events+ NvdAPIKey string // optional NVD API key for higher rate limits}// Load reads configuration from environment variables@@ -38,6 +48,16 @@ func Load() (*Config, error) {TenantHeaderKey: getEnv("TENANT_HEADER_KEY", "X-Tenant-ID"),JWTSecret: getEnv("JWT_SECRET", ""),MetricsPort: mustGetEnvInt("METRICS_PORT", 9090),++ // Shodan+ ShodanAPIKey: getEnv("SHODAN_API_KEY", ""),+ ShodanEnrichEnabled: getEnv("SHODAN_ENRICH_ENABLED", "false") == "true",+ ShodanCacheExpirySecs: mustGetEnvInt("SHODAN_CACHE_EXPIRY_SECS", 3600),++ // CVE correlation+ VulnCorrelEnabled: getEnv("VULN_CORREL_ENABLED", "true") == "true",+ VulnKafkaTopic: getEnv("VULN_KAFKA_TOPIC", "aisoc.vulnerability_matches"),+ NvdAPIKey: getEnv("NVD_API_KEY", ""),}if cfg.JWTSecret == "" && os.Getenv("ENV") != "development" {diff --git a/services/ingest/internal/enrichment/shodan.go b/services/ingest/internal/enrichment/shodan.gonew file mode 100644index 000000000..b9576c2ec--- /dev/null+++ b/services/ingest/internal/enrichment/shodan.go@@ -0,0 +1,189 @@+// Package enrichment provides external API enrichment for ingested events.+//+// The Shodan enricher performs host lookups against the Shodan API,+// caches results in an in-process TTL cache (Redis can be wired later),+// and augments OCSF events with exposure metadata (open ports, CVEs, ASN).+//+// Cyble Open-Source AI Security Operations Center — MIT License+package enrichment++import (+ "context"+ "encoding/json"+ "fmt"+ "io"+ "log/slog"+ "net"+ "net/http"+ "strings"+ "sync"+ "time"+)++const (+ shodanHostURL = "https://api.shodan.io/shodan/host/%s?key=%s"+ shodanCachePrefix = "shodan:host:"+)++// ShodanHost is the subset of Shodan /shodan/host/<ip> we use.+type ShodanHost struct {+ IP string `json:"ip_str"`+ Org string `json:"org"`+ ISP string `json:"isp"`+ ASN string `json:"asn"`+ Hostnames []string `json:"hostnames"`+ Ports []int `json:"ports"`+ Tags []string `json:"tags"`+ Vulns []string `json:"vulns"` // CVE IDs+ Country string `json:"country_code"`+ City string `json:"city"`+ LastUpdate string `json:"last_update"`+}++type cacheEntry struct {+ host *ShodanHost+ expiry time.Time+}++// ShodanEnricher looks up IP addresses using the Shodan Internet DB API+// and caches results in an in-process TTL cache.+type ShodanEnricher struct {+ apiKey string+ cacheExpiry time.Duration+ httpClient *http.Client+ log *slog.Logger++ mu sync.RWMutex+ cache map[string]cacheEntry+}++// NewShodanEnricher creates a new enricher.+func NewShodanEnricher(apiKey string, cacheExpiry time.Duration) *ShodanEnricher {+ return &ShodanEnricher{+ apiKey: apiKey,+ cacheExpiry: cacheExpiry,+ httpClient: &http.Client{Timeout: 10 * time.Second},+ log: slog.Default().With("component", "shodan_enricher"),+ cache: make(map[string]cacheEntry),+ }+}++// Enrich adds Shodan host metadata to the provided event map.+// Returns the event unmodified if no routable IP is found or the API call fails.+func (s *ShodanEnricher) Enrich(ctx context.Context, event map[string]interface{}) map[string]interface{} {+ ip := ExtractPublicIP(event)+ if ip == "" {+ return event+ }++ host, err := s.lookup(ctx, ip)+ if err != nil {+ s.log.Warn("Shodan lookup failed", "ip", ip, "err", err)+ return event+ }+ if host == nil {+ return event+ }++ event["shodan"] = map[string]interface{}{+ "ip": host.IP,+ "org": host.Org,+ "isp": host.ISP,+ "asn": host.ASN,+ "hostnames": host.Hostnames,+ "open_ports": host.Ports,+ "tags": host.Tags,+ "cves": host.Vulns,+ "country": host.Country,+ "city": host.City,+ "last_update": host.LastUpdate,+ }++ return event+}++// VulnsForIP returns CVE IDs Shodan associates with an IP, or nil.+func (s *ShodanEnricher) VulnsForIP(ctx context.Context, ip string) ([]string, error) {+ host, err := s.lookup(ctx, ip)+ if err != nil || host == nil {+ return nil, err+ }+ return host.Vulns, nil+}++// ─── Private helpers ──────────────────────────────────────────────────────────++func (s *ShodanEnricher) lookup(ctx context.Context, ip string) (*ShodanHost, error) {+ // Try cache first+ s.mu.RLock()+ if entry, ok := s.cache[ip]; ok && time.Now().Before(entry.expiry) {+ s.mu.RUnlock()+ return entry.host, nil+ }+ s.mu.RUnlock()++ // Call Shodan API+ url := fmt.Sprintf(shodanHostURL, ip, s.apiKey)+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)+ if err != nil {+ return nil, err+ }++ resp, err := s.httpClient.Do(req)+ if err != nil {+ return nil, err+ }+ defer resp.Body.Close()++ if resp.StatusCode == http.StatusNotFound {+ return nil, nil // IP not in Shodan — not an error+ }+ if resp.StatusCode != http.StatusOK {+ return nil, fmt.Errorf("shodan API returned %d", resp.StatusCode)+ }++ body, err := io.ReadAll(resp.Body)+ if err != nil {+ return nil, err+ }++ var host ShodanHost+ if err := json.Unmarshal(body, &host); err != nil {+ return nil, err+ }++ // Cache the result+ s.mu.Lock()+ s.cache[ip] = cacheEntry{host: &host, expiry: time.Now().Add(s.cacheExpiry)}+ s.mu.Unlock()++ return &host, nil+}++// ExtractPublicIP scans common OCSF event fields for a routable IP address.+func ExtractPublicIP(event map[string]interface{}) string {+ candidates := []string{+ "src_ip", "source_ip", "dst_ip", "dest_ip",+ "remote_ip", "ip_address", "host_ip",+ }++ for _, field := range candidates {+ raw, ok := event[field]+ if !ok {+ continue+ }+ ip, ok := raw.(string)+ if !ok {+ continue+ }+ parsed := net.ParseIP(strings.TrimSpace(ip))+ if parsed == nil {+ continue+ }+ if parsed.IsLoopback() || parsed.IsPrivate() || parsed.IsLinkLocalUnicast() {+ continue+ }+ return ip+ }+ return ""+}diff --git a/services/ingest/internal/enrichment/vuln.go b/services/ingest/internal/enrichment/vuln.gonew file mode 100644index 000000000..45340c522--- /dev/null+++ b/services/ingest/internal/enrichment/vuln.go@@ -0,0 +1,196 @@+// Package enrichment — CVE / vulnerability correlation.+//+// VulnCorrelator checks CVE IDs found in events (or from Shodan) against+// the CISA Known Exploited Vulnerabilities catalogue and emits structured+// VulnMatch objects for each hit.+//+// Cyble Open-Source AI Security Operations Center — MIT License+package enrichment++import (+ "context"+ "encoding/json"+ "fmt"+ "io"+ "log/slog"+ "net/http"+ "strings"+ "sync"+ "time"+)++const (+ cisaKevURL = "https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json"+)++// VulnMatch is the payload emitted to Kafka when a CVE correlation fires.+type VulnMatch struct {+ EventID string `json:"event_id"`+ SourceIP string `json:"source_ip"`+ CVE string `json:"cve_id"`+ KEVEntry *KevEntry `json:"kev_entry,omitempty"`+ DetectedAt time.Time `json:"detected_at"`+ TenantID string `json:"tenant_id,omitempty"`+}++// KevEntry represents a single CISA KEV catalogue entry.+type KevEntry struct {+ CVEID string `json:"cveID"`+ VendorProject string `json:"vendorProject"`+ Product string `json:"product"`+ VulnerabilityName string `json:"vulnerabilityName"`+ DateAdded string `json:"dateAdded"`+ ShortDescription string `json:"shortDescription"`+ RequiredAction string `json:"requiredAction"`+ DueDate string `json:"dueDate"`+}++// VulnCorrelator cross-references CVE IDs against the CISA KEV catalogue.+type VulnCorrelator struct {+ httpClient *http.Client+ kevIndex map[string]*KevEntry+ lastUpdated time.Time+ mu sync.RWMutex+ log *slog.Logger+}++// NewVulnCorrelator creates a correlator; call LoadKEV before first use.+func NewVulnCorrelator() *VulnCorrelator {+ return &VulnCorrelator{+ httpClient: &http.Client{Timeout: 15 * time.Second},+ kevIndex: make(map[string]*KevEntry),+ log: slog.Default().With("component", "vuln_correlator"),+ }+}++// LoadKEV fetches the CISA KEV catalogue.+func (v *VulnCorrelator) LoadKEV(ctx context.Context) error {+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, cisaKevURL, nil)+ if err != nil {+ return err+ }+ resp, err := v.httpClient.Do(req)+ if err != nil {+ return err+ }+ defer resp.Body.Close()++ if resp.StatusCode != http.StatusOK {+ return fmt.Errorf("CISA KEV returned %d", resp.StatusCode)+ }++ body, err := io.ReadAll(resp.Body)+ if err != nil {+ return err+ }++ if err := v.parseKEV(body); err != nil {+ return err+ }++ v.log.Info("CISA KEV catalogue loaded", "entries", len(v.kevIndex))+ v.lastUpdated = time.Now()+ return nil+}++// Correlate checks an event plus optional Shodan CVEs for KEV matches.+// Returns a (potentially empty) slice of VulnMatch.+func (v *VulnCorrelator) Correlate(event map[string]interface{}, shodanCVEs []string) []VulnMatch {+ v.mu.RLock()+ defer v.mu.RUnlock()++ if len(v.kevIndex) == 0 {+ return nil+ }++ seen := make(map[string]bool)+ var matches []VulnMatch++ allCVEs := append(extractCVEsFromEvent(event), shodanCVEs...)++ for _, cve := range allCVEs {+ upper := strings.ToUpper(cve)+ if seen[upper] {+ continue+ }+ seen[upper] = true++ if entry, ok := v.kevIndex[upper]; ok {+ matches = append(matches, VulnMatch{+ EventID: stringField(event, "event_id"),+ SourceIP: stringField(event, "src_ip"),+ CVE: upper,+ KEVEntry: entry,+ DetectedAt: time.Now().UTC(),+ TenantID: stringField(event, "tenant_uid"),+ })+ }+ }++ return matches+}++// Size returns the number of KEV entries loaded.+func (v *VulnCorrelator) Size() int {+ v.mu.RLock()+ defer v.mu.RUnlock()+ return len(v.kevIndex)+}++// ─── Private helpers ──────────────────────────────────────────────────────────++type kevCatalog struct {+ Vulnerabilities []*KevEntry `json:"vulnerabilities"`+}++func (v *VulnCorrelator) parseKEV(data []byte) error {+ var cat kevCatalog+ if err := json.Unmarshal(data, &cat); err != nil {+ return err+ }++ v.mu.Lock()+ defer v.mu.Unlock()++ v.kevIndex = make(map[string]*KevEntry, len(cat.Vulnerabilities))+ for _, e := range cat.Vulnerabilities {+ entry := e+ v.kevIndex[strings.ToUpper(entry.CVEID)] = entry+ }+ return nil+}++func extractCVEsFromEvent(event map[string]interface{}) []string {+ cveFields := []string{"cve_id", "cve", "vuln_id", "vulnerability_id"}+ var ids []string+ for _, f := range cveFields {+ if v, ok := event[f]; ok {+ switch val := v.(type) {+ case string:+ if looksLikeCVE(val) {+ ids = append(ids, val)+ }+ case []interface{}:+ for _, item := range val {+ if s, ok := item.(string); ok && looksLikeCVE(s) {+ ids = append(ids, s)+ }+ }+ }+ }+ }+ return ids+}++func looksLikeCVE(s string) bool {+ return strings.HasPrefix(strings.ToUpper(strings.TrimSpace(s)), "CVE-")+}++func stringField(event map[string]interface{}, key string) string {+ if v, ok := event[key]; ok {+ if s, ok := v.(string); ok {+ return s+ }+ }+ return ""+}diff --git a/services/ingest/internal/normalizer/normalizer.go b/services/ingest/internal/normalizer/normalizer.goindex 2939b94bb..0c508bdc1 100644--- a/services/ingest/internal/normalizer/normalizer.go+++ b/services/ingest/internal/normalizer/normalizer.go@@ -2,12 +2,15 @@package normalizerimport (+ "context""encoding/json""fmt""strings""time"+ "github.com/cyble/aisoc/services/ingest/internal/attck""github.com/cyble/aisoc/services/ingest/internal/config"+ "github.com/cyble/aisoc/services/ingest/internal/enrichment""github.com/google/uuid""github.com/rs/zerolog/log")@@ -70,8 +73,14 @@ type NormalizedEvent struct {// Normalizer converts raw events to OCSFtype Normalizer struct {- cfg *config.Config- version string+ cfg *config.Config+ version string+ shodan *enrichment.ShodanEnricher+ vulnCorrel *enrichment.VulnCorrelator++ // VulnMatches is a channel where VULNERABILITY_MATCH events are published.+ // Nil if vuln correlation is disabled.+ VulnMatches chan enrichment.VulnMatch}// connectorProfile defines normalization rules for a connector type@@ -156,12 +165,45 @@ var connectorProfiles = map[string]connectorProfile{},}-// New creates a new Normalizer instance+// New creates a new Normalizer instance and loads the ATT&CK corpus.func New(cfg *config.Config) (*Normalizer, error) {- return &Normalizer{+ // Best-effort ATT&CK corpus load — normalizer works without it+ if err := attck.Load(cfg.AttckDataPath); err != nil {+ log.Warn().Err(err).Msg("ATT&CK corpus unavailable; technique enrichment disabled")+ }++ n := &Normalizer{cfg: cfg,version: "1.1.0",- }, nil+ }++ // Set up Shodan enrichment if configured+ if cfg.ShodanEnrichEnabled && cfg.ShodanAPIKey != "" {+ n.shodan = enrichment.NewShodanEnricher(+ cfg.ShodanAPIKey,+ time.Duration(cfg.ShodanCacheExpirySecs)*time.Second,+ )+ log.Info().Msg("Shodan enrichment enabled")+ }++ // Set up vulnerability correlation+ if cfg.VulnCorrelEnabled {+ n.vulnCorrel = enrichment.NewVulnCorrelator()+ n.VulnMatches = make(chan enrichment.VulnMatch, 256)++ ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)+ defer cancel()+ if err := n.vulnCorrel.LoadKEV(ctx); err != nil {+ log.Warn().Err(err).Msg("CISA KEV load failed; vulnerability correlation disabled")+ n.vulnCorrel = nil+ close(n.VulnMatches)+ n.VulnMatches = nil+ } else {+ log.Info().Int("entries", n.vulnCorrel.Size()).Msg("CISA KEV catalogue loaded")+ }+ }++ return n, nil}// Normalize converts a raw event to a NormalizedEvent@@ -237,6 +279,58 @@ func (n *Normalizer) Normalize(raw *RawEvent) (*NormalizedEvent, error) {ocsf["raw_data"] = string(rawBytes)}+ // ATT&CK technique enrichment+ if attck.Loaded() {+ if techIDs := extractTechniqueIDs(raw.Payload); len(techIDs) > 0 {+ var enriched []map[string]interface{}+ for _, tid := range techIDs {+ if tech := attck.Lookup(tid); tech != nil {+ enriched = append(enriched, map[string]interface{}{+ "technique_id": tech.ID,+ "technique_name": tech.Name,+ "tactic_ids": tech.TacticIDs,+ "tactic_names": tech.TacticNames,+ "url": tech.URL,+ })+ }+ }+ if len(enriched) > 0 {+ ocsf["mitre_attck"] = enriched+ }+ }+ }++ // Shodan enrichment (non-blocking; best-effort)+ var shodanCVEs []string+ if n.shodan != nil {+ ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)+ ocsf = n.shodan.Enrich(ctx, ocsf)+ cancel()++ // Collect CVEs from Shodan result for vuln correlation+ if shodanBlock, ok := ocsf["shodan"].(map[string]interface{}); ok {+ if cves, ok := shodanBlock["cves"].([]string); ok {+ shodanCVEs = cves+ }+ }+ }++ // Vulnerability correlation — emit to VulnMatches channel+ if n.vulnCorrel != nil {+ matches := n.vulnCorrel.Correlate(ocsf, shodanCVEs)+ for _, m := range matches {+ select {+ case n.VulnMatches <- m:+ default:+ // Channel full — drop to avoid blocking ingest pipeline+ log.Warn().Str("cve", m.CVE).Msg("VulnMatches channel full; dropping match")+ }+ }+ if len(matches) > 0 {+ ocsf["vulnerability_matches"] = matches+ }+ }+eventID := uuid.New().String()return &NormalizedEvent{@@ -249,6 +343,57 @@ func (n *Normalizer) Normalize(raw *RawEvent) (*NormalizedEvent, error) {}, nil}+// extractTechniqueIDs scans common fields in a raw payload for ATT&CK technique IDs.+func extractTechniqueIDs(payload map[string]interface{}) []string {+ seen := map[string]struct{}{}+ var results []string++ candidateKeys := []string{+ "technique_id", "mitre_technique", "attck_technique", "tactic_id",+ "mitre_techniques", "attack_technique",+ }+ for _, key := range candidateKeys {+ val, ok := payload[key]+ if !ok {+ continue+ }+ switch v := val.(type) {+ case string:+ if tid := normalizeTechniqueID(v); tid != "" {+ if _, dup := seen[tid]; !dup {+ seen[tid] = struct{}{}+ results = append(results, tid)+ }+ }+ case []interface{}:+ for _, item := range v {+ if s, ok := item.(string); ok {+ if tid := normalizeTechniqueID(s); tid != "" {+ if _, dup := seen[tid]; !dup {+ seen[tid] = struct{}{}+ results = append(results, tid)+ }+ }+ }+ }+ }+ }+ return results+}++// normalizeTechniqueID extracts a clean ATT&CK technique ID from a string.+func normalizeTechniqueID(s string) string {+ s = strings.TrimSpace(strings.ToUpper(s))+ // Accept T1234 or T1234.001+ if len(s) >= 5 && s[0] == 'T' {+ parts := strings.SplitN(s, ".", 2)+ if len(parts[0]) >= 5 && len(parts[0]) <= 7 {+ return s+ }+ }+ return ""+}+// normalizeTime attempts to parse and re-format a timestamp as RFC3339func normalizeTime(t string) string {formats := []string{diff --git a/services/ingest/internal/publisher/publisher.go b/services/ingest/internal/publisher/publisher.goindex c32368aa5..2b08f0750 100644--- a/services/ingest/internal/publisher/publisher.go+++ b/services/ingest/internal/publisher/publisher.go@@ -7,6 +7,7 @@ import ("fmt""github.com/cyble/aisoc/services/ingest/internal/config"+ "github.com/cyble/aisoc/services/ingest/internal/enrichment""github.com/cyble/aisoc/services/ingest/internal/normalizer""github.com/rs/zerolog/log"kafka "github.com/segmentio/kafka-go"@@ -14,8 +15,9 @@ import (// Publisher sends normalized events to Kafkatype Publisher struct {- writer *kafka.Writer- cfg *config.Config+ writer *kafka.Writer+ vulnWriter *kafka.Writer // dedicated writer for VULNERABILITY_MATCH topic+ cfg *config.Config}// New creates a new Kafka publisher@@ -30,12 +32,48 @@ func New(cfg *config.Config) (*Publisher, error) {AllowAutoTopicCreation: true,}+ var vulnWriter *kafka.Writer+ if cfg.VulnCorrelEnabled && cfg.VulnKafkaTopic != "" {+ vulnWriter = &kafka.Writer{+ Addr: kafka.TCP(cfg.KafkaBrokers),+ Topic: cfg.VulnKafkaTopic,+ Balancer: &kafka.Hash{},+ MaxAttempts: 3,+ AllowAutoTopicCreation: true,+ }+ }+return &Publisher{- writer: w,- cfg: cfg,+ writer: w,+ vulnWriter: vulnWriter,+ cfg: cfg,}, nil}+// PublishVulnMatch sends a VULNERABILITY_MATCH event to the dedicated Kafka topic.+func (p *Publisher) PublishVulnMatch(ctx context.Context, match enrichment.VulnMatch) error {+ if p.vulnWriter == nil {+ return nil+ }+ data, err := json.Marshal(match)+ if err != nil {+ return err+ }+ msg := kafka.Message{+ Key: []byte(match.CVE + ":" + match.TenantID),+ Value: data,+ Headers: []kafka.Header{+ {Key: "event_type", Value: []byte("VULNERABILITY_MATCH")},+ {Key: "tenant_id", Value: []byte(match.TenantID)},+ },+ }+ if err := p.vulnWriter.WriteMessages(ctx, msg); err != nil {+ log.Warn().Err(err).Str("cve", match.CVE).Msg("Failed to publish vuln match")+ return err+ }+ return nil+}+// Publish sends a single normalized event to Kafkafunc (p *Publisher) Publish(ctx context.Context, event *normalizer.NormalizedEvent) error {data, err := json.Marshal(event)@@ -104,9 +142,14 @@ func (p *Publisher) PublishBatch(ctx context.Context, events []*normalizer.Normareturn nil}-// Close shuts down the Kafka writer+// Close shuts down the Kafka writersfunc (p *Publisher) Close() {if err := p.writer.Close(); err != nil {log.Error().Err(err).Msg("Error closing Kafka writer")}+ if p.vulnWriter != nil {+ if err := p.vulnWriter.Close(); err != nil {+ log.Error().Err(err).Msg("Error closing vuln Kafka writer")+ }+ }}diff --git a/services/ingest/main.go b/services/ingest/main.goindex a37de33e0..28b7c57fd 100644--- a/services/ingest/main.go+++ b/services/ingest/main.go@@ -55,6 +55,19 @@ func main() {ctx, cancel := context.WithCancel(context.Background())defer cancel()+ // Drain VulnMatches → Kafka in a background goroutine+ if norm.VulnMatches != nil {+ go func() {+ for match := range norm.VulnMatches {+ pubCtx, pubCancel := context.WithTimeout(ctx, 5*time.Second)+ if err := pub.PublishVulnMatch(pubCtx, match); err != nil {+ log.Warn().Err(err).Str("cve", match.CVE).Msg("Failed to publish VulnMatch")+ }+ pubCancel()+ }+ }()+ }+sigCh := make(chan os.Signal, 1)signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)diff --git a/services/threatintel/Dockerfile b/services/threatintel/Dockerfilenew file mode 100644index 000000000..342085085--- /dev/null+++ b/services/threatintel/Dockerfile@@ -0,0 +1,19 @@+FROM python:3.11-slim++WORKDIR /app++RUN apt-get update && apt-get install -y --no-install-recommends \+ build-essential curl \+ && rm -rf /var/lib/apt/lists/*++RUN pip install --no-cache-dir poetry==1.8.2++COPY pyproject.toml poetry.lock* ./+RUN poetry config virtualenvs.create false \+ && poetry install --no-interaction --no-ansi --only main++COPY app ./app++EXPOSE 8005++CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8005"]diff --git a/services/threatintel/app/__init__.py b/services/threatintel/app/__init__.pynew file mode 100644index 000000000..e69de29bbdiff --git a/services/threatintel/app/clients/__init__.py b/services/threatintel/app/clients/__init__.pynew file mode 100644index 000000000..e69de29bbdiff --git a/services/threatintel/app/clients/cisa_kev.py b/services/threatintel/app/clients/cisa_kev.pynew file mode 100644index 000000000..0ee5b0d0f--- /dev/null+++ b/services/threatintel/app/clients/cisa_kev.py@@ -0,0 +1,57 @@+"""+CISA Known Exploited Vulnerabilities (KEV) catalog client.++Cyble Open-Source AI Security Operations Center — MIT License+"""+from __future__ import annotations++import logging+from typing import Any++import httpx++logger = logging.getLogger(__name__)++_KEV_URL = "https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json"+++class CisaKevClient:+ """+ Fetches the CISA Known Exploited Vulnerabilities catalog and converts+ each entry into a normalized IOC-style dict for storage.+ """++ def __init__(self, url: str = _KEV_URL) -> None:+ self._url = url++ async def fetch(self) -> list[dict[str, Any]]:+ """Download and return the full KEV catalog as a list of entries."""+ async with httpx.AsyncClient(timeout=60.0) as client:+ try:+ resp = await client.get(self._url, follow_redirects=True)+ resp.raise_for_status()+ body = resp.json()+ entries = body.get("vulnerabilities", [])+ logger.info("CISA KEV fetched", count=len(entries))+ return entries+ except Exception as exc:+ logger.error("CISA KEV fetch failed", error=str(exc))+ return []++ def to_ioc(self, entry: dict[str, Any]) -> dict[str, Any]:+ """Convert a KEV entry to a normalized IOC dict."""+ return {+ "type": "vulnerability",+ "value": entry.get("cveID", ""),+ "description": entry.get("vulnerabilityName", ""),+ "vendor_project": entry.get("vendorProject", ""),+ "product": entry.get("product", ""),+ "required_action": entry.get("requiredAction", ""),+ "due_date": entry.get("dueDate", ""),+ "date_added": entry.get("dateAdded", ""),+ "known_ransomware": entry.get("knownRansomwareCampaignUse", "Unknown"),+ "source": "cisa-kev",+ "source_ref": f"cisa-kev:{entry.get('cveID', '')}",+ "tags": ["kev", "cisa", "exploited"],+ "tlp": "white",+ }diff --git a/services/threatintel/app/clients/misp.py b/services/threatintel/app/clients/misp.pynew file mode 100644index 000000000..619e8f4f2--- /dev/null+++ b/services/threatintel/app/clients/misp.py@@ -0,0 +1,118 @@+"""+MISP client for fetching threat intelligence events and attributes.++Cyble Open-Source AI Security Operations Center — MIT License+"""+from __future__ import annotations++import logging+from datetime import datetime, timedelta, timezone+from typing import Any++import httpx++logger = logging.getLogger(__name__)+++class MispClient:+ """+ Async MISP REST API client.++ Supports fetching:+ - Recent events (with attributes/tags)+ - Specific event by ID+ - IOC export in text format+ """++ def __init__(+ self,+ url: str,+ api_key: str,+ verify_ssl: bool = True,+ ) -> None:+ self._base_url = url.rstrip("/")+ self._api_key = api_key+ self._verify_ssl = verify_ssl+ self._headers = {+ "Authorization": api_key,+ "Accept": "application/json",+ "Content-Type": "application/json",+ }++ async def get_recent_events(+ self,+ since_hours: int = 24,+ limit: int = 500,+ ) -> list[dict[str, Any]]:+ """Fetch MISP events published within the last N hours."""+ since_ts = int(+ (datetime.now(timezone.utc) - timedelta(hours=since_hours)).timestamp()+ )+ params = {+ "timestamp": since_ts,+ "limit": limit,+ "with_attachments": 0,+ "includeEventTags": 1,+ "includeAttributeTags": 1,+ }+ async with httpx.AsyncClient(+ headers=self._headers,+ verify=self._verify_ssl,+ timeout=60.0,+ ) as client:+ try:+ resp = await client.post(+ f"{self._base_url}/events/restSearch",+ json={"returnFormat": "json", **params},+ )+ resp.raise_for_status()+ body = resp.json()+ events = body.get("response", [])+ if isinstance(events, dict) and "Event" in events:+ events = [events]+ logger.info("MISP events fetched", count=len(events))+ return [e.get("Event", e) for e in events if isinstance(e, dict)]+ except Exception as exc:+ logger.error("MISP fetch failed", error=str(exc))+ return []++ async def get_event(self, event_id: str) -> dict[str, Any] | None:+ """Fetch a single MISP event by ID."""+ async with httpx.AsyncClient(+ headers=self._headers, verify=self._verify_ssl, timeout=30.0+ ) as client:+ try:+ resp = await client.get(f"{self._base_url}/events/{event_id}")+ resp.raise_for_status()+ body = resp.json()+ return body.get("Event")+ except Exception as exc:+ logger.error("MISP get_event failed", event_id=event_id, error=str(exc))+ return None++ def extract_iocs(self, event: dict[str, Any]) -> list[dict[str, Any]]:+ """Extract IOC attributes from a MISP event dict."""+ iocs: list[dict[str, Any]] = []+ for attr in event.get("Attribute", []):+ ioc = {+ "type": attr.get("type"),+ "value": attr.get("value"),+ "category": attr.get("category"),+ "to_ids": attr.get("to_ids", False),+ "comment": attr.get("comment", ""),+ "tags": [t.get("name") for t in attr.get("Tag", [])],+ "source": "misp",+ "source_ref": f"misp:{event.get('uuid', '')}",+ "event_info": event.get("info", ""),+ "tlp": self._extract_tlp(attr),+ }+ iocs.append(ioc)+ return iocs++ @staticmethod+ def _extract_tlp(attr: dict[str, Any]) -> str:+ for tag in attr.get("Tag", []):+ name: str = tag.get("name", "").lower()+ if name.startswith("tlp:"):+ return name.split(":", 1)[-1]+ return "white"diff --git a/services/threatintel/app/clients/otx.py b/services/threatintel/app/clients/otx.pynew file mode 100644index 000000000..d9a565220--- /dev/null+++ b/services/threatintel/app/clients/otx.py@@ -0,0 +1,121 @@+"""+AlienVault OTX (Open Threat Exchange) client.++Cyble Open-Source AI Security Operations Center — MIT License+"""+from __future__ import annotations++import logging+from typing import Any++import httpx++logger = logging.getLogger(__name__)++_OTX_BASE = "https://otx.alienvault.com"+++class OtxClient:+ """+ Async OTX REST API v1 client.++ Fetches:+ - Subscribed pulse feed (with IOC indicators)+ - Pulse details+ - Single indicator lookups+ """++ def __init__(self, api_key: str, base_url: str = _OTX_BASE) -> None:+ self._api_key = api_key+ self._base_url = base_url.rstrip("/")+ self._headers = {+ "X-OTX-API-KEY": api_key,+ "Accept": "application/json",+ }++ async def get_subscribed_pulses(+ self,+ modified_since: str | None = None,+ limit: int = 50,+ ) -> list[dict[str, Any]]:+ """+ Fetch pulses from the subscribed feed.++ Args:+ modified_since: ISO-8601 timestamp, fetch pulses modified after this time.+ limit: Max pulses per request (paginated internally).++ Returns:+ List of pulse dicts.+ """+ pulses: list[dict[str, Any]] = []+ page = 1++ async with httpx.AsyncClient(+ headers=self._headers, timeout=60.0+ ) as client:+ while True:+ params: dict[str, Any] = {"limit": limit, "page": page}+ if modified_since:+ params["modified_since"] = modified_since++ try:+ resp = await client.get(+ f"{self._base_url}/api/v1/pulses/subscribed",+ params=params,+ )+ resp.raise_for_status()+ body = resp.json()+ results = body.get("results", [])+ pulses.extend(results)++ if not body.get("next"):+ break+ page += 1+ except Exception as exc:+ logger.error("OTX fetch failed", page=page, error=str(exc))+ break++ logger.info("OTX pulses fetched", count=len(pulses))+ return pulses++ def extract_iocs(self, pulse: dict[str, Any]) -> list[dict[str, Any]]:+ """Extract IOC indicators from an OTX pulse."""+ iocs: list[dict[str, Any]] = []+ pulse_id = pulse.get("id", "")+ pulse_name = pulse.get("name", "")++ for ind in pulse.get("indicators", []):+ ioc_type = self._map_indicator_type(ind.get("type", ""))+ if not ioc_type:+ continue+ iocs.append({+ "type": ioc_type,+ "value": ind.get("indicator", ""),+ "description": ind.get("description", ""),+ "tags": pulse.get("tags", []),+ "source": "otx",+ "source_ref": f"otx:{pulse_id}",+ "pulse_name": pulse_name,+ "tlp": pulse.get("tlp", "white"),+ "malware_families": pulse.get("malware_families", []),+ "attack_ids": pulse.get("attack_ids", []),+ })+ return iocs++ @staticmethod+ def _map_indicator_type(otx_type: str) -> str:+ mapping = {+ "IPv4": "ipv4-addr",+ "IPv6": "ipv6-addr",+ "domain": "domain-name",+ "hostname": "domain-name",+ "URL": "url",+ "FileHash-MD5": "file-hash:MD5",+ "FileHash-SHA1": "file-hash:SHA-1",+ "FileHash-SHA256": "file-hash:SHA-256",+ "email": "email-addr",+ "CVE": "vulnerability",+ "CIDR": "ipv4-addr",+ }+ return mapping.get(otx_type, "")diff --git a/services/threatintel/app/clients/taxii.py b/services/threatintel/app/clients/taxii.pynew file mode 100644index 000000000..e7b2cc9c8--- /dev/null+++ b/services/threatintel/app/clients/taxii.py@@ -0,0 +1,111 @@+"""+TAXII 2.1 client for fetching STIX bundles from threat intelligence feeds.++Cyble Open-Source AI Security Operations Center — MIT License+"""+from __future__ import annotations++import json+import logging+from typing import Any, Iterator++import httpx++logger = logging.getLogger(__name__)+++class TaxiiClient:+ """+ Lightweight async TAXII 2.1 client.++ Supports:+ - Discovery endpoint+ - Collection listing+ - Objects endpoint with added/removed pagination+ """++ def __init__(+ self,+ url: str,+ collection_id: str,+ username: str = "",+ password: str = "",+ verify_ssl: bool = True,+ ) -> None:+ self._base_url = url.rstrip("/")+ self._collection_id = collection_id+ self._auth = (username, password) if username else None+ self._verify_ssl = verify_ssl+ self._headers = {+ "Accept": "application/taxii+json;version=2.1",+ "Content-Type": "application/taxii+json;version=2.1",+ }++ async def get_objects(+ self,+ added_after: str | None = None,+ limit: int = 1000,+ ) -> list[dict[str, Any]]:+ """+ Fetch STIX objects from a TAXII collection.++ Args:+ added_after: ISO-8601 timestamp — only fetch objects added after this time.+ limit: Maximum objects per page.++ Returns:+ List of raw STIX object dicts.+ """+ objects: list[dict[str, Any]] = []+ params: dict[str, Any] = {"limit": limit}+ if added_after:+ params["added_after"] = added_after++ next_cursor: str | None = None+ page = 0++ async with httpx.AsyncClient(+ auth=self._auth,+ headers=self._headers,+ verify=self._verify_ssl,+ timeout=60.0,+ ) as client:+ while True:+ if next_cursor:+ params["next"] = next_cursor++ url = (+ f"{self._base_url}/collections/{self._collection_id}/objects/"+ )+ try:+ resp = await client.get(url, params=params)+ resp.raise_for_status()+ except httpx.HTTPStatusError as exc:+ logger.error(+ "TAXII fetch failed",+ url=url,+ status=exc.response.status_code,+ )+ break++ body = resp.json()+ page_objects = body.get("objects", [])+ objects.extend(page_objects)+ page += 1+ logger.debug(+ "TAXII page fetched",+ page=page,+ count=len(page_objects),+ total=len(objects),+ )++ next_cursor = body.get("next")+ if not next_cursor or not page_objects:+ break++ logger.info(+ "TAXII collection fetched",+ collection=self._collection_id,+ total_objects=len(objects),+ )+ return objectsdiff --git a/services/threatintel/app/config.py b/services/threatintel/app/config.pynew file mode 100644index 000000000..a2c48adde--- /dev/null+++ b/services/threatintel/app/config.py@@ -0,0 +1,79 @@+"""Configuration for the AiSOC Threat Intelligence service."""+from __future__ import annotations++from pydantic_settings import BaseSettings+++class Settings(BaseSettings):+ # Service+ APP_NAME: str = "AiSOC ThreatIntel"+ VERSION: str = "0.1.0"+ ENVIRONMENT: str = "development"++ # Redis (bloom filter + cache)+ REDIS_URL: str = "redis://redis:6379/2"+ BLOOM_CAPACITY: int = 10_000_000+ BLOOM_ERROR_RATE: float = 0.001++ # OpenSearch+ OPENSEARCH_HOST: str = "opensearch"+ OPENSEARCH_PORT: int = 9200+ OPENSEARCH_USER: str = ""+ OPENSEARCH_PASSWORD: str = ""+ OPENSEARCH_INDEX_IOC: str = "threatintel-iocs"+ OPENSEARCH_INDEX_ACTOR: str = "threatintel-actors"++ # Qdrant+ QDRANT_HOST: str = "qdrant"+ QDRANT_PORT: int = 6333+ QDRANT_COLLECTION_IOC: str = "threatintel_iocs"++ # Neo4j+ NEO4J_URI: str = "bolt://neo4j:7687"+ NEO4J_USER: str = "neo4j"+ NEO4J_PASSWORD: str = "aisoc_secret"++ # Kafka+ KAFKA_BOOTSTRAP_SERVERS: str = "redpanda:9092"+ KAFKA_TOPIC_THREAT_INTEL: str = "aisoc.threat_intel"+ # Legacy alias+ KAFKA_BROKERS: str = "redpanda:9092"+ KAFKA_TOPIC_IOC: str = "aisoc.ioc_enrichments"++ # TAXII 2.1+ TAXII_URL: str = ""+ TAXII_USERNAME: str = ""+ TAXII_PASSWORD: str = ""+ TAXII_API_ROOT: str = ""+ TAXII_COLLECTION_IDS: str = "" # comma-separated collection IDs++ # MISP+ MISP_URL: str = ""+ MISP_API_KEY: str = ""+ MISP_VERIFY_SSL: bool = True++ # OTX AlienVault+ OTX_API_KEY: str = ""+ OTX_BASE_URL: str = "https://otx.alienvault.com"++ # CISA KEV+ CISA_KEV_URL: str = "https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json"++ # OpenCTI+ OPENCTI_URL: str = ""+ OPENCTI_TOKEN: str = ""++ # Poll intervals (seconds)+ TAXII_POLL_INTERVAL: int = 900 # 15 min+ MISP_POLL_INTERVAL: int = 1800 # 30 min+ OTX_POLL_INTERVAL: int = 1800 # 30 min+ CISA_KEV_POLL_INTERVAL: int = 86400 # 24 h+ # Legacy alias+ CISA_POLL_INTERVAL: int = 86400++ class Config:+ env_file = ".env"+ case_sensitive = True+++settings = Settings()diff --git a/services/threatintel/app/feeds/__init__.py b/services/threatintel/app/feeds/__init__.pynew file mode 100644index 000000000..e69de29bbdiff --git a/services/threatintel/app/feeds/handlers.py b/services/threatintel/app/feeds/handlers.pynew file mode 100644index 000000000..0b0bfff5b--- /dev/null+++ b/services/threatintel/app/feeds/handlers.py@@ -0,0 +1,140 @@+"""+Feed handler functions for each threat intelligence source.++Each handler is registered with the FeedScheduler and invoked on+its configured polling interval. Handlers fetch from their source,+parse/normalize the data, and hand off to ThreatIntelPipeline.++Cyble Open-Source AI Security Operations Center — MIT License+"""+from __future__ import annotations++import logging+from typing import TYPE_CHECKING, Any++from app.clients.taxii import TaxiiClient+from app.clients.misp import MispClient+from app.clients.otx import OtxClient+from app.clients.cisa_kev import CisaKevClient+from app.parsers.stix import StixParser++if TYPE_CHECKING:+ from app.feeds.pipeline import ThreatIntelPipeline+ from app.config import Settings++logger = logging.getLogger(__name__)+++# ─── TAXII Feed Handler ───────────────────────────────────────────────────────++async def handle_taxii_feed(+ client: TaxiiClient,+ pipeline: "ThreatIntelPipeline",+ api_root: str,+ collection_id: str,+) -> None:+ """+ Fetch STIX bundle from TAXII 2.1 endpoint and ingest into pipeline.+ """+ source_label = f"taxii:{collection_id[:8]}"+ logger.info("Polling TAXII feed", collection=collection_id)++ try:+ objects = await client.get_objects(api_root, collection_id)+ if not objects:+ logger.debug("TAXII feed empty", collection=collection_id)+ return++ parser = StixParser(objects)+ iocs = parser.extract_iocs()+ actors = parser.extract_actors()+ relationships = parser.extract_relationships()++ if iocs:+ stats = await pipeline.ingest_iocs(iocs, source=source_label)+ logger.info("TAXII IOCs ingested", **stats)++ if actors:+ stats = await pipeline.ingest_actors(actors, source=source_label)+ logger.info("TAXII actors ingested", **stats)++ if relationships:+ count = await pipeline.ingest_relationships(relationships, source=source_label)+ logger.info("TAXII relationships ingested", count=count)++ except Exception as exc:+ logger.error("TAXII feed handler failed", collection=collection_id, error=str(exc))+++# ─── MISP Feed Handler ────────────────────────────────────────────────────────++async def handle_misp_feed(+ client: MispClient,+ pipeline: "ThreatIntelPipeline",+ since_hours: int = 24,+) -> None:+ """+ Fetch recent MISP events and extract IOCs.+ """+ logger.info("Polling MISP feed", since_hours=since_hours)++ try:+ events = await client.get_recent_events(since_hours=since_hours)+ all_iocs: list[dict[str, Any]] = []+ for event in events:+ all_iocs.extend(client.extract_iocs(event))++ if all_iocs:+ stats = await pipeline.ingest_iocs(all_iocs, source="misp")+ logger.info("MISP IOCs ingested", **stats, event_count=len(events))++ except Exception as exc:+ logger.error("MISP feed handler failed", error=str(exc))+++# ─── OTX Feed Handler ─────────────────────────────────────────────────────────++async def handle_otx_feed(+ client: OtxClient,+ pipeline: "ThreatIntelPipeline",+) -> None:+ """+ Fetch subscribed OTX pulses and extract IOC indicators.+ """+ logger.info("Polling OTX feed")++ try:+ pulses = await client.get_subscribed_pulses(limit=100)+ all_iocs: list[dict[str, Any]] = []+ for pulse in pulses:+ all_iocs.extend(client.extract_iocs(pulse))++ if all_iocs:+ stats = await pipeline.ingest_iocs(all_iocs, source="otx")+ logger.info("OTX IOCs ingested", **stats, pulse_count=len(pulses))++ except Exception as exc:+ logger.error("OTX feed handler failed", error=str(exc))+++# ─── CISA KEV Feed Handler ────────────────────────────────────────────────────++async def handle_cisa_kev_feed(+ client: CisaKevClient,+ pipeline: "ThreatIntelPipeline",+) -> None:+ """+ Fetch CISA KEV catalog and ingest vulnerability IOCs.+ """+ logger.info("Polling CISA KEV feed")++ try:+ entries = await client.fetch()+ iocs = [client.to_ioc(e) for e in entries]++ if iocs:+ stats = await pipeline.ingest_iocs(iocs, source="cisa-kev")+ logger.info("CISA KEV IOCs ingested", **stats)++ except Exception as exc:+ logger.error("CISA KEV feed handler failed", error=str(exc))diff --git a/services/threatintel/app/feeds/pipeline.py b/services/threatintel/app/feeds/pipeline.pynew file mode 100644index 000000000..832b4784f--- /dev/null+++ b/services/threatintel/app/feeds/pipeline.py@@ -0,0 +1,165 @@+"""+Threat intelligence ingestion pipeline.++Handles the full lifecycle for each normalized IOC/actor batch:+ 1. Bloom-filter deduplication (Redis)+ 2. OpenSearch bulk index+ 3. Qdrant vector upsert+ 4. Neo4j graph merge+ 5. Kafka event emission (optional)++Cyble Open-Source AI Security Operations Center — MIT License+"""+from __future__ import annotations++import json+import logging+from datetime import datetime, timezone+from typing import Any, Optional++from app.storage.bloom import RedisBloomFilter+from app.storage.opensearch import OpenSearchStore+from app.storage.qdrant import QdrantStore+from app.storage.neo4j import Neo4jStore++logger = logging.getLogger(__name__)+++class ThreatIntelPipeline:+ """+ Central ingestion pipeline for all threat intel feeds.++ Coordinates deduplication and multi-sink storage.+ """++ def __init__(+ self,+ bloom: RedisBloomFilter,+ os_store: OpenSearchStore,+ qdrant_store: QdrantStore,+ neo4j_store: Neo4jStore,+ kafka_producer: Any | None = None,+ kafka_topic: str = "threat-intel-events",+ ) -> None:+ self._bloom = bloom+ self._os = os_store+ self._qdrant = qdrant_store+ self._neo4j = neo4j_store+ self._kafka = kafka_producer+ self._kafka_topic = kafka_topic++ async def ingest_iocs(+ self,+ iocs: list[dict[str, Any]],+ source: str,+ ) -> dict[str, int]:+ """+ Ingest a batch of normalized IOCs.++ Returns a stats dict with counts for new, duplicate, and indexed items.+ """+ if not iocs:+ return {"total": 0, "new": 0, "duplicate": 0}++ # --- Dedup via Bloom filter ---+ new_iocs: list[dict[str, Any]] = []+ duplicates = 0++ for ioc in iocs:+ bloom_key = f"{ioc.get('type','?')}:{ioc.get('value','?')}"+ if await self._bloom.contains(bloom_key):+ duplicates += 1+ else:+ await self._bloom.add(bloom_key)+ new_iocs.append(ioc)++ logger.info(+ "IOC dedup complete",+ source=source,+ total=len(iocs),+ new=len(new_iocs),+ duplicate=duplicates,+ )++ if not new_iocs:+ return {"total": len(iocs), "new": 0, "duplicate": duplicates}++ # --- Write to storage sinks ---+ indexed = await self._os.bulk_index_iocs(new_iocs)++ try:+ await self._qdrant.upsert_iocs(new_iocs)+ except Exception as exc:+ logger.warning("Qdrant IOC upsert failed", error=str(exc))++ try:+ await self._neo4j.upsert_iocs(new_iocs)+ except Exception as exc:+ logger.warning("Neo4j IOC upsert failed", error=str(exc))++ # --- Emit Kafka events ---+ if self._kafka:+ await self._emit_kafka_events(new_iocs, event_type="NEW_IOC", source=source)++ return {+ "total": len(iocs),+ "new": len(new_iocs),+ "duplicate": duplicates,+ "indexed": indexed,+ }++ async def ingest_actors(+ self,+ actors: list[dict[str, Any]],+ source: str,+ ) -> dict[str, int]:+ """Ingest a batch of normalized threat actors."""+ if not actors:+ return {"total": 0, "new": 0}++ indexed = await self._os.bulk_index_actors(actors)++ try:+ await self._qdrant.upsert_actors(actors)+ except Exception as exc:+ logger.warning("Qdrant actor upsert failed", error=str(exc))++ try:+ await self._neo4j.upsert_actors(actors)+ except Exception as exc:+ logger.warning("Neo4j actor upsert failed", error=str(exc))++ return {"total": len(actors), "indexed": indexed}++ async def ingest_relationships(+ self,+ relationships: list[dict[str, Any]],+ source: str,+ ) -> int:+ """Ingest STIX relationships into Neo4j."""+ try:+ await self._neo4j.upsert_relationships(relationships)+ return len(relationships)+ except Exception as exc:+ logger.warning("Neo4j relationship upsert failed", error=str(exc))+ return 0++ async def _emit_kafka_events(+ self,+ items: list[dict[str, Any]],+ event_type: str,+ source: str,+ ) -> None:+ """Emit Kafka messages for downstream consumers."""+ ts = datetime.now(timezone.utc).isoformat()+ for item in items:+ payload = json.dumps({+ "event_type": event_type,+ "source": source,+ "timestamp": ts,+ "data": item,+ }).encode()+ try:+ await self._kafka.send(self._kafka_topic, value=payload)+ except Exception as exc:+ logger.debug("Kafka emit failed", error=str(exc))diff --git a/services/threatintel/app/feeds/scheduler.py b/services/threatintel/app/feeds/scheduler.pynew file mode 100644index 000000000..bccf855f4--- /dev/null+++ b/services/threatintel/app/feeds/scheduler.py@@ -0,0 +1,68 @@+"""+APScheduler-based feed polling orchestrator.++Coordinates TAXII, MISP, OTX, and CISA KEV polling intervals and+pipes normalized IOCs through the deduplication + storage pipeline.++Cyble Open-Source AI Security Operations Center — MIT License+"""+from __future__ import annotations++import logging+from typing import TYPE_CHECKING++from apscheduler.schedulers.asyncio import AsyncIOScheduler+from apscheduler.triggers.interval import IntervalTrigger++if TYPE_CHECKING:+ from app.feeds.pipeline import ThreatIntelPipeline++logger = logging.getLogger(__name__)+++class FeedScheduler:+ """+ Manages periodic polling of all threat intelligence feeds.++ Each feed has its own job with a configurable interval so that+ high-frequency feeds (e.g., TAXII every 15 min) and low-frequency+ feeds (CISA KEV once a day) can coexist without contention.+ """++ def __init__(self, pipeline: "ThreatIntelPipeline") -> None:+ self._pipeline = pipeline+ self._scheduler = AsyncIOScheduler()++ def register(+ self,+ feed_name: str,+ handler,+ interval_seconds: int,+ ) -> None:+ """Register a feed polling function."""+ self._scheduler.add_job(+ func=handler,+ trigger=IntervalTrigger(seconds=interval_seconds),+ id=feed_name,+ name=f"Feed: {feed_name}",+ replace_existing=True,+ max_instances=1,+ coalesce=True,+ )+ logger.info(+ "Registered feed",+ feed=feed_name,+ interval_seconds=interval_seconds,+ )++ def start(self) -> None:+ """Start the scheduler (non-blocking in async context)."""+ if not self._scheduler.running:+ self._scheduler.start()+ logger.info("Feed scheduler started")++ def stop(self) -> None:+ """Gracefully stop the scheduler."""+ if self._scheduler.running:+ self._scheduler.shutdown(wait=False)+ logger.info("Feed scheduler stopped")diff --git a/services/threatintel/app/main.py b/services/threatintel/app/main.pynew file mode 100644index 000000000..ec58abb78--- /dev/null+++ b/services/threatintel/app/main.py@@ -0,0 +1,238 @@+"""+AiSOC Threat Intelligence Feed Service++Polls multiple threat intelligence feeds on configurable intervals,+deduplicates via Redis Bloom filter, and writes normalized IOCs/actors+into OpenSearch, Qdrant, and Neo4j.++Cyble Open-Source AI Security Operations Center — MIT License+"""+from __future__ import annotations++import logging+from contextlib import asynccontextmanager+from functools import partial+from typing import AsyncGenerator++import redis.asyncio as aioredis+from aiokafka import AIOKafkaProducer+from fastapi import FastAPI+from neo4j import AsyncGraphDatabase+from opensearchpy import AsyncOpenSearch+from prometheus_client import Counter, make_asgi_app+from qdrant_client import AsyncQdrantClient++from app.clients.cisa_kev import CisaKevClient+from app.clients.misp import MispClient+from app.clients.otx import OtxClient+from app.clients.taxii import TaxiiClient+from app.config import settings+from app.feeds.handlers import (+ handle_cisa_kev_feed,+ handle_misp_feed,+ handle_otx_feed,+ handle_taxii_feed,+)+from app.feeds.pipeline import ThreatIntelPipeline+from app.feeds.scheduler import FeedScheduler+from app.storage.bloom import RedisBloomFilter+from app.storage.neo4j import Neo4jStore+from app.storage.opensearch import OpenSearchStore+from app.storage.qdrant import QdrantStore++logging.basicConfig(level=logging.INFO)+logger = logging.getLogger(__name__)++# ─── Prometheus metrics ───────────────────────────────────────────────────────+iocs_ingested = Counter("threatintel_iocs_ingested_total", "Total IOCs ingested", ["source"])+actors_ingested = Counter("threatintel_actors_ingested_total", "Total actors ingested", ["source"])+++# ─── Application lifespan ─────────────────────────────────────────────────────++@asynccontextmanager+async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:+ logger.info("ThreatIntel service starting…")++ # ── Redis ──────────────────────────────────────────────────────────────────+ redis = aioredis.from_url(settings.REDIS_URL, decode_responses=False)++ # ── OpenSearch ────────────────────────────────────────────────────────────+ os_client = AsyncOpenSearch(+ hosts=[{"host": settings.OPENSEARCH_HOST, "port": settings.OPENSEARCH_PORT}],+ http_auth=(settings.OPENSEARCH_USER, settings.OPENSEARCH_PASSWORD)+ if settings.OPENSEARCH_USER+ else None,+ use_ssl=False,+ )++ # ── Qdrant ────────────────────────────────────────────────────────────────+ qdrant_client = AsyncQdrantClient(+ host=settings.QDRANT_HOST, port=settings.QDRANT_PORT+ )++ # ── Neo4j ─────────────────────────────────────────────────────────────────+ neo4j_driver = AsyncGraphDatabase.driver(+ settings.NEO4J_URI,+ auth=(settings.NEO4J_USER, settings.NEO4J_PASSWORD),+ )++ # ── Kafka (optional) ──────────────────────────────────────────────────────+ kafka_producer = None+ if settings.KAFKA_BOOTSTRAP_SERVERS:+ try:+ kafka_producer = AIOKafkaProducer(+ bootstrap_servers=settings.KAFKA_BOOTSTRAP_SERVERS+ )+ await kafka_producer.start()+ except Exception as exc:+ logger.warning("Kafka producer unavailable", error=str(exc))+ kafka_producer = None++ # ── Storage layer ─────────────────────────────────────────────────────────+ bloom = RedisBloomFilter(redis)+ os_store = OpenSearchStore(os_client)+ qdrant_store = QdrantStore(qdrant_client)+ neo4j_store = Neo4jStore(neo4j_driver)++ await os_store.initialize()+ try:+ await qdrant_store.initialize()+ except Exception as exc:+ logger.warning("Qdrant init failed", error=str(exc))++ # ── Pipeline ──────────────────────────────────────────────────────────────+ pipeline = ThreatIntelPipeline(+ bloom=bloom,+ os_store=os_store,+ qdrant_store=qdrant_store,+ neo4j_store=neo4j_store,+ kafka_producer=kafka_producer,+ kafka_topic=settings.KAFKA_TOPIC_THREAT_INTEL,+ )++ # ── Clients ───────────────────────────────────────────────────────────────+ taxii_client = TaxiiClient(+ base_url=settings.TAXII_URL,+ username=settings.TAXII_USERNAME,+ password=settings.TAXII_PASSWORD,+ )++ misp_client = MispClient(+ url=settings.MISP_URL,+ api_key=settings.MISP_API_KEY,+ ) if settings.MISP_URL else None++ otx_client = OtxClient(+ api_key=settings.OTX_API_KEY,+ ) if settings.OTX_API_KEY else None++ kev_client = CisaKevClient()++ # ── Scheduler ─────────────────────────────────────────────────────────────+ scheduler = FeedScheduler(pipeline)++ # Register TAXII feeds+ if settings.TAXII_URL and settings.TAXII_COLLECTION_IDS:+ for col_id in settings.TAXII_COLLECTION_IDS.split(","):+ col_id = col_id.strip()+ if col_id:+ scheduler.register(+ feed_name=f"taxii:{col_id}",+ handler=partial(+ handle_taxii_feed,+ client=taxii_client,+ pipeline=pipeline,+ api_root=settings.TAXII_API_ROOT,+ collection_id=col_id,+ ),+ interval_seconds=settings.TAXII_POLL_INTERVAL,+ )++ # Register MISP feed+ if misp_client:+ scheduler.register(+ feed_name="misp",+ handler=partial(handle_misp_feed, client=misp_client, pipeline=pipeline),+ interval_seconds=settings.MISP_POLL_INTERVAL,+ )++ # Register OTX feed+ if otx_client:+ scheduler.register(+ feed_name="otx",+ handler=partial(handle_otx_feed, client=otx_client, pipeline=pipeline),+ interval_seconds=settings.OTX_POLL_INTERVAL,+ )++ # CISA KEV (always enabled — no key required)+ scheduler.register(+ feed_name="cisa-kev",+ handler=partial(handle_cisa_kev_feed, client=kev_client, pipeline=pipeline),+ interval_seconds=settings.CISA_KEV_POLL_INTERVAL,+ )++ scheduler.start()++ # Store refs for health endpoint+ app.state.scheduler = scheduler+ app.state.pipeline = pipeline+ app.state.redis = redis++ yield++ # ── Shutdown ──────────────────────────────────────────────────────────────+ scheduler.stop()+ if kafka_producer:+ await kafka_producer.stop()+ await neo4j_driver.close()+ await os_client.close()+ await redis.close()+ logger.info("ThreatIntel service stopped")+++# ─── FastAPI app ──────────────────────────────────────────────────────────────++app = FastAPI(+ title="AiSOC Threat Intelligence Service",+ version="0.1.0",+ lifespan=lifespan,+)++# Mount Prometheus metrics+metrics_app = make_asgi_app()+app.mount("/metrics", metrics_app)++[email protected]("/health")+async def health() -> dict:+ """Health and status endpoint."""+ try:+ redis: aioredis.Redis = app.state.redis+ await redis.ping()+ redis_ok = True+ except Exception:+ redis_ok = False++ return {+ "status": "ok",+ "redis": redis_ok,+ "scheduler": app.state.scheduler._scheduler.running+ if hasattr(app.state, "scheduler")+ else False,+ }++[email protected]("/api/v1/iocs/search")+async def search_iocs(+ value: str | None = None,+ ioc_type: str | None = None,+ source: str | None = None,+ limit: int = 20,+) -> dict:+ """Search stored IOCs."""+ pipeline: ThreatIntelPipeline = app.state.pipeline+ iocs = await pipeline._os.search_iocs(+ value=value, ioc_type=ioc_type, source=source, limit=limit+ )+ return {"total": len(iocs), "iocs": iocs}diff --git a/services/threatintel/app/parsers/__init__.py b/services/threatintel/app/parsers/__init__.pynew file mode 100644index 000000000..e69de29bbdiff --git a/services/threatintel/app/parsers/stix.py b/services/threatintel/app/parsers/stix.pynew file mode 100644index 000000000..fddd5c05c--- /dev/null+++ b/services/threatintel/app/parsers/stix.py@@ -0,0 +1,239 @@+"""+STIX 2.1 object parser — converts STIX objects to normalized IOC/actor dicts.++Cyble Open-Source AI Security Operations Center — MIT License+"""+from __future__ import annotations++import hashlib+import logging+from typing import Any++logger = logging.getLogger(__name__)++# STIX types we care about as IOCs+_IOC_TYPES = {+ "ipv4-addr",+ "ipv6-addr",+ "domain-name",+ "url",+ "email-addr",+ "file",+ "autonomous-system",+ "network-traffic",+ "vulnerability",+}++# Mappings from STIX relationship types+_REL_USES = "uses"+_REL_INDICATES = "indicates"+_REL_ATTRIBUTED_TO = "attributed-to"+_REL_TARGETS = "targets"+_REL_MITIGATES = "mitigates"+++class StixParser:+ """+ Parses a STIX 2.1 bundle into normalized dicts suitable for+ storage in OpenSearch/Qdrant/Neo4j.++ Usage:+ parser = StixParser(bundle_objects)+ iocs = parser.extract_iocs()+ actors = parser.extract_actors()+ relationships = parser.extract_relationships()+ """++ def __init__(self, objects: list[dict[str, Any]]) -> None:+ self._objects = objects+ self._by_id: dict[str, dict[str, Any]] = {o["id"]: o for o in objects if "id" in o}++ # ─── Indicators / IOCs ────────────────────────────────────────────────────++ def extract_iocs(self) -> list[dict[str, Any]]:+ """+ Extract all IOC-worthy objects from the STIX bundle.++ Handles:+ - indicator objects (with pattern parsing)+ - SCO (STIX Cyber Observable) objects+ """+ iocs: list[dict[str, Any]] = []++ for obj in self._objects:+ obj_type = obj.get("type", "")++ if obj_type == "indicator":+ parsed = self._parse_indicator(obj)+ if parsed:+ iocs.extend(parsed)++ elif obj_type in _IOC_TYPES:+ ioc = self._observable_to_ioc(obj)+ if ioc:+ iocs.append(ioc)++ return iocs++ def _parse_indicator(self, obj: dict[str, Any]) -> list[dict[str, Any]]:+ """Parse a STIX indicator object into one or more IOC dicts."""+ pattern: str = obj.get("pattern", "")+ labels = obj.get("labels", [])+ name = obj.get("name", "")+ confidence = obj.get("confidence", 50)+ valid_from = obj.get("valid_from", "")+ valid_until = obj.get("valid_until", "")++ iocs: list[dict[str, Any]] = []++ # Extract values from STIX pattern e.g. [ipv4-addr:value = '1.2.3.4']+ import re+ # Simple regex extraction+ for match in re.finditer(r"\[(\S+):(\S+)\s*=\s*'([^']+)'\]", pattern):+ obj_type, prop, value = match.groups()+ ioc_type = self._map_stix_type(obj_type, prop)+ if not ioc_type:+ continue+ iocs.append({+ "type": ioc_type,+ "value": value,+ "labels": labels,+ "name": name,+ "confidence": confidence,+ "valid_from": valid_from,+ "valid_until": valid_until,+ "stix_id": obj.get("id"),+ "source": "stix",+ "source_ref": obj.get("id", ""),+ "tags": labels,+ "tlp": self._extract_marking(obj),+ })+ return iocs++ def _observable_to_ioc(self, obj: dict[str, Any]) -> dict[str, Any] | None:+ """Convert a STIX SCO to a normalized IOC dict."""+ obj_type = obj.get("type", "")+ value = ""++ if obj_type == "ipv4-addr":+ value = obj.get("value", "")+ ioc_type = "ipv4-addr"+ elif obj_type == "ipv6-addr":+ value = obj.get("value", "")+ ioc_type = "ipv6-addr"+ elif obj_type == "domain-name":+ value = obj.get("value", "")+ ioc_type = "domain-name"+ elif obj_type == "url":+ value = obj.get("value", "")+ ioc_type = "url"+ elif obj_type == "email-addr":+ value = obj.get("value", "")+ ioc_type = "email-addr"+ elif obj_type == "file":+ hashes = obj.get("hashes", {})+ for hash_type in ("SHA-256", "SHA-1", "MD5"):+ if hash_type in hashes:+ value = hashes[hash_type]+ ioc_type = f"file-hash:{hash_type}"+ break+ else:+ return None+ elif obj_type == "vulnerability":+ value = obj.get("name", "")+ ioc_type = "vulnerability"+ else:+ return None++ if not value:+ return None++ return {+ "type": ioc_type,+ "value": value,+ "stix_id": obj.get("id"),+ "source": "stix",+ "source_ref": obj.get("id", ""),+ "tags": [],+ "tlp": self._extract_marking(obj),+ }++ # ─── Threat Actors ────────────────────────────────────────────────────────++ def extract_actors(self) -> list[dict[str, Any]]:+ """Extract threat actor / intrusion set objects."""+ actors = []+ for obj in self._objects:+ if obj.get("type") in ("threat-actor", "intrusion-set"):+ actors.append({+ "stix_id": obj.get("id"),+ "type": obj.get("type"),+ "name": obj.get("name", ""),+ "description": obj.get("description", ""),+ "aliases": obj.get("aliases", []),+ "goals": obj.get("goals", []),+ "labels": obj.get("labels", []),+ "sophistication": obj.get("sophistication", ""),+ "resource_level": obj.get("resource_level", ""),+ "primary_motivation": obj.get("primary_motivation", ""),+ "first_seen": obj.get("first_seen", ""),+ "last_seen": obj.get("last_seen", ""),+ "source": "stix",+ "tlp": self._extract_marking(obj),+ })+ return actors++ # ─── Relationships ────────────────────────────────────────────────────────++ def extract_relationships(self) -> list[dict[str, Any]]:+ """Extract STIX relationship objects for graph storage."""+ rels = []+ for obj in self._objects:+ if obj.get("type") != "relationship":+ continue+ rel_type = obj.get("relationship_type", "")+ if rel_type not in (_REL_USES, _REL_INDICATES, _REL_ATTRIBUTED_TO, _REL_TARGETS, _REL_MITIGATES):+ continue+ rels.append({+ "stix_id": obj.get("id"),+ "rel_type": rel_type,+ "source_ref": obj.get("source_ref", ""),+ "target_ref": obj.get("target_ref", ""),+ "description": obj.get("description", ""),+ "start_time": obj.get("start_time", ""),+ "stop_time": obj.get("stop_time", ""),+ })+ return rels++ # ─── Helpers ──────────────────────────────────────────────────────────────++ @staticmethod+ def _map_stix_type(obj_type: str, prop: str) -> str:+ mapping = {+ ("ipv4-addr", "value"): "ipv4-addr",+ ("ipv6-addr", "value"): "ipv6-addr",+ ("domain-name", "value"): "domain-name",+ ("url", "value"): "url",+ ("email-addr", "value"): "email-addr",+ ("file", "hashes.MD5"): "file-hash:MD5",+ ("file", "hashes.'SHA-256'"): "file-hash:SHA-256",+ ("file", "hashes.'SHA-1'"): "file-hash:SHA-1",+ }+ return mapping.get((obj_type, prop), obj_type)++ @staticmethod+ def _extract_marking(obj: dict[str, Any]) -> str:+ """Extract TLP marking from STIX object marking refs."""+ for ref in obj.get("object_marking_refs", []):+ ref_lower = ref.lower()+ if "tlp" in ref_lower:+ for level in ("red", "amber", "green", "white", "clear"):+ if level in ref_lower:+ return level+ return "white"+++def generate_ioc_id(ioc_type: str, value: str, source: str) -> str:+ """Generate a stable, deduplication-friendly IOC ID."""+ key = f"{ioc_type}:{value}:{source}"+ return hashlib.sha256(key.encode()).hexdigest()[:16]diff --git a/services/threatintel/app/storage/__init__.py b/services/threatintel/app/storage/__init__.pynew file mode 100644index 000000000..e69de29bbdiff --git a/services/threatintel/app/storage/bloom.py b/services/threatintel/app/storage/bloom.pynew file mode 100644index 000000000..e71a522e4--- /dev/null+++ b/services/threatintel/app/storage/bloom.py@@ -0,0 +1,115 @@+"""+Redis-backed Bloom filter for IOC deduplication.++Uses a counting Bloom filter approach with multiple hash functions to+minimize false positives while allowing existence checks before+expensive storage writes.++Cyble Open-Source AI Security Operations Center — MIT License+"""+from __future__ import annotations++import math+import logging+from typing import Optional++import mmh3+import redis.asyncio as aioredis++logger = logging.getLogger(__name__)++_DEFAULT_CAPACITY = 10_000_000 # 10M IOCs+_DEFAULT_ERROR_RATE = 0.001 # 0.1% false-positive rate+_BLOOM_KEY = "threatintel:bloom:iocs"+++class RedisBloomFilter:+ """+ Async Redis Bloom filter for IOC deduplication.++ Bit positions are computed using MurmurHash3 with multiple seeds,+ matching a standard Bloom filter implementation.+ """++ def __init__(+ self,+ redis_client: aioredis.Redis,+ key: str = _BLOOM_KEY,+ capacity: int = _DEFAULT_CAPACITY,+ error_rate: float = _DEFAULT_ERROR_RATE,+ ) -> None:+ self._redis = redis_client+ self._key = key++ # Optimal Bloom filter parameters+ self._size = self._optimal_size(capacity, error_rate)+ self._num_hashes = self._optimal_hash_count(capacity, self._size)++ logger.info(+ "Bloom filter initialized",+ size=self._size,+ hash_functions=self._num_hashes,+ expected_capacity=capacity,+ )++ # ─── Public API ───────────────────────────────────────────────────────────++ async def add(self, item: str) -> None:+ """Add an item to the Bloom filter."""+ pipe = self._redis.pipeline(transaction=False)+ for bit in self._bit_positions(item):+ pipe.setbit(self._key, bit, 1)+ await pipe.execute()++ async def contains(self, item: str) -> bool:+ """+ Check if an item is probably in the filter.++ Returns True if possibly seen before (may have false positives),+ False if definitely not seen before.+ """+ pipe = self._redis.pipeline(transaction=False)+ for bit in self._bit_positions(item):+ pipe.getbit(self._key, bit)+ results = await pipe.execute()+ return all(results)++ async def add_batch(self, items: list[str]) -> list[bool]:+ """+ Add a batch of items.++ Returns a list of booleans indicating whether each item was+ *already* in the filter before this batch (True = duplicate).+ """+ duplicates: list[bool] = []+ for item in items:+ already_seen = await self.contains(item)+ duplicates.append(already_seen)+ if not already_seen:+ await self.add(item)+ return duplicates++ async def count_bits(self) -> int:+ """Return the number of set bits (approximation of stored items)."""+ return await self._redis.bitcount(self._key)++ # ─── Private helpers ──────────────────────────────────────────────────────++ def _bit_positions(self, item: str) -> list[int]:+ """Calculate the bit positions for this item using k hash functions."""+ positions = []+ encoded = item.encode("utf-8")+ for seed in range(self._num_hashes):+ h = mmh3.hash(encoded, seed=seed, signed=False)+ positions.append(h % self._size)+ return positions++ @staticmethod+ def _optimal_size(capacity: int, error_rate: float) -> int:+ """Calculate optimal bit array size: m = -(n * ln(p)) / (ln(2)^2)"""+ return int(-capacity * math.log(error_rate) / (math.log(2) ** 2))++ @staticmethod+ def _optimal_hash_count(capacity: int, size: int) -> int:+ """Calculate optimal hash function count: k = (m/n) * ln(2)"""+ return max(1, int((size / capacity) * math.log(2)))diff --git a/services/threatintel/app/storage/neo4j.py b/services/threatintel/app/storage/neo4j.pynew file mode 100644index 000000000..760d7d5df--- /dev/null+++ b/services/threatintel/app/storage/neo4j.py@@ -0,0 +1,129 @@+"""+Neo4j storage layer for threat intelligence relationships.++Writes IOC nodes, actor nodes, and their relationships into the+knowledge graph for attack path and blast-radius queries.++Cyble Open-Source AI Security Operations Center — MIT License+"""+from __future__ import annotations++import logging+from typing import Any++from neo4j import AsyncDriver++logger = logging.getLogger(__name__)+++class Neo4jStore:+ """+ Async Neo4j writer for threat intelligence graph data.+ """++ def __init__(self, driver: AsyncDriver) -> None:+ self._driver = driver++ async def upsert_iocs(self, iocs: list[dict[str, Any]]) -> None:+ """Merge IOC nodes into the graph."""+ if not iocs:+ return++ async with self._driver.session() as session:+ for ioc in iocs:+ await session.run(+ """+ MERGE (i:IOC {value: $value, type: $type})+ ON CREATE SET+ i.source = $source,+ i.tlp = $tlp,+ i.confidence = $confidence,+ i.first_seen = $first_seen,+ i.last_seen = $last_seen,+ i.tags = $tags+ ON MATCH SET+ i.last_seen = $last_seen,+ i.confidence = $confidence+ """,+ value=ioc.get("value", ""),+ type=ioc.get("type", ""),+ source=ioc.get("source", ""),+ tlp=ioc.get("tlp", "white"),+ confidence=ioc.get("confidence", 50),+ first_seen=ioc.get("first_seen", ""),+ last_seen=ioc.get("last_seen", ""),+ tags=ioc.get("tags", []),+ )+ logger.debug("Neo4j IOCs upserted", count=len(iocs))++ async def upsert_actors(self, actors: list[dict[str, Any]]) -> None:+ """Merge threat actor nodes."""+ if not actors:+ return++ async with self._driver.session() as session:+ for actor in actors:+ await session.run(+ """+ MERGE (a:ThreatActor {name: $name})+ ON CREATE SET+ a.stix_id = $stix_id,+ a.aliases = $aliases,+ a.description = $description,+ a.sophistication= $sophistication,+ a.source = $source+ ON MATCH SET+ a.description = $description+ """,+ name=actor.get("name", ""),+ stix_id=actor.get("stix_id", ""),+ aliases=actor.get("aliases", []),+ description=actor.get("description", ""),+ sophistication=actor.get("sophistication", ""),+ source=actor.get("source", ""),+ )+ logger.debug("Neo4j actors upserted", count=len(actors))++ async def upsert_relationships(self, relationships: list[dict[str, Any]]) -> None:+ """Create USES/TARGETS/ATTRIBUTED_TO relationships between nodes."""+ if not relationships:+ return++ async with self._driver.session() as session:+ for rel in relationships:+ src = rel.get("source_name") or rel.get("source_id", "")+ tgt = rel.get("target_value") or rel.get("target_id", "")+ rel_type = rel.get("relationship_type", "RELATED_TO").upper().replace("-", "_")++ if not src or not tgt:+ continue++ # Generic relationship query — works for actor→technique, actor→ioc, etc.+ try:+ await session.run(+ f"""+ MATCH (a {{name: $src}})+ MATCH (b {{value: $tgt}})+ MERGE (a)-[r:{rel_type}]->(b)+ ON CREATE SET r.source = $source+ """,+ src=src,+ tgt=tgt,+ source=rel.get("source", "stix"),+ )+ except Exception:+ # Fall through if nodes don't exist+ pass++ async def link_ioc_to_technique(self, ioc_value: str, technique_id: str) -> None:+ """Link an IOC to a MITRE ATT&CK technique node."""+ async with self._driver.session() as session:+ await session.run(+ """+ MATCH (i:IOC {value: $ioc_value})+ MERGE (t:Technique {technique_id: $technique_id})+ MERGE (i)-[:INDICATES]->(t)+ """,+ ioc_value=ioc_value,+ technique_id=technique_id,+ )diff --git a/services/threatintel/app/storage/opensearch.py b/services/threatintel/app/storage/opensearch.pynew file mode 100644index 000000000..8b2a4d05e--- /dev/null+++ b/services/threatintel/app/storage/opensearch.py@@ -0,0 +1,146 @@+"""+OpenSearch storage layer for threat intelligence IOCs and actors.++Cyble Open-Source AI Security Operations Center — MIT License+"""+from __future__ import annotations++import hashlib+import logging+from datetime import datetime, timezone+from typing import Any++from opensearchpy import AsyncOpenSearch, helpers++logger = logging.getLogger(__name__)++_IOC_INDEX = "threatintel-iocs"+_ACTOR_INDEX = "threatintel-actors"++_IOC_MAPPING = {+ "mappings": {+ "properties": {+ "type": {"type": "keyword"},+ "value": {"type": "keyword"},+ "description": {"type": "text"},+ "tags": {"type": "keyword"},+ "source": {"type": "keyword"},+ "source_ref": {"type": "keyword"},+ "tlp": {"type": "keyword"},+ "confidence": {"type": "integer"},+ "first_seen": {"type": "date"},+ "last_seen": {"type": "date"},+ "ingested_at": {"type": "date"},+ }+ }+}++_ACTOR_MAPPING = {+ "mappings": {+ "properties": {+ "stix_id": {"type": "keyword"},+ "name": {"type": "keyword"},+ "aliases": {"type": "keyword"},+ "description": {"type": "text"},+ "sophistication": {"type": "keyword"},+ "primary_motivation": {"type": "keyword"},+ "source": {"type": "keyword"},+ "tlp": {"type": "keyword"},+ "first_seen": {"type": "date"},+ "last_seen": {"type": "date"},+ "ingested_at": {"type": "date"},+ }+ }+}+++class OpenSearchStore:+ """+ Async OpenSearch writer for threat intel data.++ Manages index creation and provides bulk upsert methods.+ """++ def __init__(self, client: AsyncOpenSearch) -> None:+ self._os = client++ async def initialize(self) -> None:+ """Create indices if they don't exist."""+ for index, mapping in [(_IOC_INDEX, _IOC_MAPPING), (_ACTOR_INDEX, _ACTOR_MAPPING)]:+ if not await self._os.indices.exists(index=index):+ await self._os.indices.create(index=index, body=mapping)+ logger.info("Created OpenSearch index", index=index)++ async def bulk_index_iocs(self, iocs: list[dict[str, Any]]) -> int:+ """Bulk upsert IOCs into OpenSearch. Returns count of indexed docs."""+ if not iocs:+ return 0++ now = datetime.now(timezone.utc).isoformat()+ actions = []+ for ioc in iocs:+ doc_id = _stable_id(ioc.get("type", ""), ioc.get("value", ""), ioc.get("source", ""))+ doc = {+ **ioc,+ "ingested_at": now,+ "last_seen": now,+ }+ actions.append({+ "_op_type": "index",+ "_index": _IOC_INDEX,+ "_id": doc_id,+ "_source": doc,+ })++ success, errors = await helpers.async_bulk(self._os, actions, raise_on_error=False)+ if errors:+ logger.warning("OpenSearch bulk errors", count=len(errors))+ return success++ async def bulk_index_actors(self, actors: list[dict[str, Any]]) -> int:+ """Bulk upsert threat actors."""+ if not actors:+ return 0++ now = datetime.now(timezone.utc).isoformat()+ actions = []+ for actor in actors:+ doc_id = _stable_id("actor", actor.get("name", ""), actor.get("source", "stix"))+ actions.append({+ "_op_type": "index",+ "_index": _ACTOR_INDEX,+ "_id": doc_id,+ "_source": {**actor, "ingested_at": now},+ })++ success, _ = await helpers.async_bulk(self._os, actions, raise_on_error=False)+ return success++ async def search_iocs(+ self,+ value: str | None = None,+ ioc_type: str | None = None,+ source: str | None = None,+ limit: int = 20,+ ) -> list[dict[str, Any]]:+ """Search IOCs by value, type, or source."""+ must: list[dict] = []+ if value:+ must.append({"term": {"value": value}})+ if ioc_type:+ must.append({"term": {"type": ioc_type}})+ if source:+ must.append({"term": {"source": source}})++ query = {"bool": {"must": must}} if must else {"match_all": {}}++ resp = await self._os.search(+ index=_IOC_INDEX,+ body={"query": query, "size": limit, "sort": [{"last_seen": "desc"}]},+ )+ return [hit["_source"] for hit in resp["hits"]["hits"]]+++def _stable_id(type_: str, value: str, source: str) -> str:+ key = f"{type_}:{value}:{source}"+ return hashlib.sha256(key.encode()).hexdigest()diff --git a/services/threatintel/app/storage/qdrant.py b/services/threatintel/app/storage/qdrant.pynew file mode 100644index 000000000..38699d757--- /dev/null+++ b/services/threatintel/app/storage/qdrant.py@@ -0,0 +1,154 @@+"""+Qdrant vector storage for semantic threat intelligence search.++Cyble Open-Source AI Security Operations Center — MIT License+"""+from __future__ import annotations++import logging+from typing import Any++from qdrant_client import AsyncQdrantClient+from qdrant_client.models import (+ Distance,+ PointStruct,+ VectorParams,+)++logger = logging.getLogger(__name__)++_IOC_COLLECTION = "threatintel_iocs"+_ACTOR_COLLECTION = "threatintel_actors"+_EMBEDDING_DIM = 1536 # text-embedding-3-small / ada-002+++class QdrantStore:+ """+ Async Qdrant writer/searcher for threat intel embeddings.++ Uses fastembed for local embedding generation when OpenAI is not+ configured (air-gapped deployments).+ """++ def __init__(self, client: AsyncQdrantClient, use_fastembed: bool = True) -> None:+ self._client = client+ self._use_fastembed = use_fastembed++ async def initialize(self) -> None:+ """Create Qdrant collections if they don't exist."""+ existing = await self._client.get_collections()+ existing_names = {c.name for c in existing.collections}++ for col in (_IOC_COLLECTION, _ACTOR_COLLECTION):+ if col not in existing_names:+ await self._client.create_collection(+ collection_name=col,+ vectors_config=VectorParams(+ size=_EMBEDDING_DIM, distance=Distance.COSINE+ ),+ )+ logger.info("Created Qdrant collection", collection=col)++ async def upsert_iocs(self, iocs: list[dict[str, Any]]) -> None:+ """Embed and upsert IOC documents into Qdrant for semantic search."""+ if not iocs:+ return++ texts = [self._ioc_to_text(ioc) for ioc in iocs]+ vectors = await self._embed(texts)++ points = [+ PointStruct(+ id=i,+ vector=vectors[i],+ payload=iocs[i],+ )+ for i in range(len(iocs))+ ]++ # Qdrant expects integer IDs — use hash of value+type for stable IDs+ import hashlib+ points = [+ PointStruct(+ id=abs(int(hashlib.md5(f"{ioc['type']}:{ioc['value']}".encode()).hexdigest(), 16)) % (2**63),+ vector=vectors[idx],+ payload=ioc,+ )+ for idx, ioc in enumerate(iocs)+ ]++ await self._client.upsert(+ collection_name=_IOC_COLLECTION,+ points=points,+ )+ logger.debug("Qdrant IOCs upserted", count=len(points))++ async def upsert_actors(self, actors: list[dict[str, Any]]) -> None:+ """Embed and upsert threat actor documents."""+ if not actors:+ return++ texts = [f"{a.get('name','')} {a.get('description','')}" for a in actors]+ vectors = await self._embed(texts)++ import hashlib+ points = [+ PointStruct(+ id=abs(int(hashlib.md5(actor.get("name", "").encode()).hexdigest(), 16)) % (2**63),+ vector=vectors[idx],+ payload=actor,+ )+ for idx, actor in enumerate(actors)+ ]++ await self._client.upsert(+ collection_name=_ACTOR_COLLECTION,+ points=points,+ )++ async def semantic_search(+ self,+ query: str,+ collection: str = _IOC_COLLECTION,+ limit: int = 10,+ ) -> list[dict[str, Any]]:+ """Semantic similarity search over threat intel."""+ vectors = await self._embed([query])+ results = await self._client.search(+ collection_name=collection,+ query_vector=vectors[0],+ limit=limit,+ )+ return [r.payload for r in results if r.payload]++ # ─── Private helpers ──────────────────────────────────────────────────────++ async def _embed(self, texts: list[str]) -> list[list[float]]:+ """Generate embeddings using fastembed or a stub."""+ try:+ from fastembed import TextEmbedding+ model = TextEmbedding(model_name="BAAI/bge-small-en-v1.5")+ embeddings = list(model.embed(texts))+ # Pad/truncate to expected dimension+ result = []+ for emb in embeddings:+ vec = list(emb)+ if len(vec) < _EMBEDDING_DIM:+ vec = vec + [0.0] * (_EMBEDDING_DIM - len(vec))+ elif len(vec) > _EMBEDDING_DIM:+ vec = vec[:_EMBEDDING_DIM]+ result.append(vec)+ return result+ except Exception:+ # Fallback: zero vectors (embeddings will be non-functional but won't crash)+ return [[0.0] * _EMBEDDING_DIM for _ in texts]++ @staticmethod+ def _ioc_to_text(ioc: dict[str, Any]) -> str:+ parts = [+ ioc.get("type", ""),+ ioc.get("value", ""),+ ioc.get("description", ""),+ " ".join(ioc.get("tags", [])),+ ]+ return " ".join(p for p in parts if p)diff --git a/services/threatintel/pyproject.toml b/services/threatintel/pyproject.tomlnew file mode 100644index 000000000..026195ba7--- /dev/null+++ b/services/threatintel/pyproject.toml@@ -0,0 +1,34 @@+[tool.poetry]+name = "aisoc-threatintel"+version = "0.1.0"+description = "AiSOC Threat Intelligence Feed Service — TAXII/STIX/MISP/OTX ingestion"+authors = ["Cyble <[email protected]>"]+license = "MIT"++[tool.poetry.dependencies]+python = "^3.11"+fastapi = "^0.109.0"+uvicorn = {extras = ["standard"], version = "^0.27.0"}+pydantic = "^2.5.0"+pydantic-settings = "^2.1.0"+httpx = "^0.26.0"+redis = {extras = ["hiredis"], version = "^5.0.1"}+taxii2-client = "^2.3.0"+stix2 = "^3.0.1"+apscheduler = "^3.10.4"+opensearch-py = "^2.4.2"+qdrant-client = {extras = ["fastembed"], version = "^1.9.0"}+neo4j = "^5.19.0"+structlog = "^24.1.0"+prometheus-client = "^0.19.0"+python-dateutil = "^2.8.2"+aiokafka = "^0.10.0"+mmh3 = "^4.1.0"++[tool.poetry.group.dev.dependencies]+pytest = "^7.4.0"+pytest-asyncio = "^0.23.0"++[build-system]+requires = ["poetry-core"]+build-backend = "poetry.core.masonry.api"