Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
feat: complete H2 Tier 2 + Tier 3 roadmap capabilities
Beenu Arora6286b89efe61
diff --git a/README.md b/README.mdindex 103f16249..f93a2f2e0 100644--- a/README.md+++ b/README.md@@ -110,6 +110,19 @@ AiSOC bundles the components a SOC normally pieces together from separate vendor- **Assess cloud posture** with a built-in CSPM/KSPM engine that ingests findings, tracks drift between scan runs, and surfaces a per-provider summary with suppress/resolve workflows.- **Correlate through identities** with a graph of users, devices, and service accounts; link alerts to identity nodes for blast-radius queries and attack-path reconstruction.- **Automate board reporting** — schedule PDF/HTML executive summaries, store artefacts, and deliver via email or webhook.+- **Three-tier agent memory** — session (in-process LRU), working (Redis-backed, 24 h TTL), and institutional (PostgreSQL + pgvector, permanent). Agents carry context across tool calls, cases, and sessions; institutional knowledge survives restarts.+- **Autonomy guardrails** — per-action confidence thresholds (e.g. `block_ip ≥ 0.90`, `close_alert ≥ 0.60`) gate every autonomous decision. Tenant admins can tighten or loosen thresholds via API; all guard-rail decisions are logged.+- **Investigation cost telemetry** — every LLM call is tracked by model, prompt tokens, completion tokens, latency, and estimated USD cost. Aggregates are persisted per-run to `aisoc_run_costs` and surfaced in SOC dashboards.+- **SOC metrics dashboard** — live MTTD, MTTR, False Positive Rate, alert/case volumes (rolling 7 d), and ATT&CK technique heatmap. Backed by a real-time API endpoint and a polished React component.+- **Analyst-override feedback loop** — analysts can correct AI verdicts (`true_positive`, `false_positive`, `benign`, `escalate`) in one click. Corrections flow into institutional memory and adjust FPR metrics automatically.+- **Natural-language detection authoring** — describe a threat in plain English; the API translates it to Sigma YAML, KQL (Microsoft Sentinel), SPL (Splunk), and ES|QL (Elastic) simultaneously. Falls back to curated templates when no LLM key is configured.+- **Closed-loop detection engineering** — when an alert is marked as a false positive, the agent drafts a Sigma rule fix using an LLM, then automatically creates a Detection-as-Code proposal routed through the same human-review DAC workflow. CI re-runs evals on approval; regression gates block regressions.+- **Natural-language query execution** — ask a security question in plain English (`POST /nl-query/execute`). The API translates it to ES|QL, SPL, and KQL; for Elasticsearch-backed tenants it executes the ES|QL query live and returns structured results, column metadata, and the query text for all three dialects.+- **Identity-centric investigation timeline** — build a chronological event timeline anchored to any user, device, service account, or IP (`POST /identity-timeline/build`). The timeline queries alerts and raw events, annotates each event with the relevant ATT&CK technique, computes an entity risk score, and returns a sorted, deduplicated event list for triage.+- **Cross-platform detection translation** — convert any detection rule bidirectionally between Sigma YAML, Splunk SPL, Microsoft Sentinel KQL, Elastic ES|QL, and Google Chronicle YARA-L2 / UDM Search (`POST /translation/translate`). An LLM handles complex logic; a regex fallback handles simple field-mapping rules with no external dependency.+- **Hypothesis-driven hunt workbench** — define a hunt hypothesis in natural language (`POST /hunts`); the API auto-generates ready-to-execute queries in ES|QL, SPL, and KQL; analysts record findings against any run and the workbench tracks open, completed, and inconclusive hunts.+- **Phishing triage workflow** — submit raw email text, URLs, attachments, or domain indicators (`POST /phishing/submit`); the LLM extracts IOCs, assigns a verdict (phishing / benign / spam / malware / unknown), maps to MITRE ATT&CK, and optionally links the submission to an existing case.+- **Knowledge-base + RAG** — ingest runbooks, policies, SOPs, and wikis (`POST /kb/ingest`); the API chunks and full-text indexes each document; analysts query with natural language (`POST /kb/query`) and receive the top matching chunks plus an LLM-synthesised answer with citation, backed by PostgreSQL FTS when no vector store is configured.Everything ships under MIT. Fork it, self-host it, audit it, extend it.diff --git a/apps/docs/docs/concepts/capabilities.md b/apps/docs/docs/concepts/capabilities.mdnew file mode 100644index 000000000..0c004f8a0--- /dev/null+++ b/apps/docs/docs/concepts/capabilities.md@@ -0,0 +1,117 @@+---+sidebar_position: 1+title: Platform Capabilities+description: Full index of AiSOC Tier 1, 2, and 3 capabilities with API references.+---++# Platform Capabilities++AiSOC ships a layered capability model across three tiers. Tier 1 items are core+SOC operations shipped from day one. Tier 2 items are intelligent-automation+extensions. Tier 3 items are advanced analyst workflows.++---++## Tier 1 — Core SOC Operations++| Capability | API prefix | Notes |+|---|---|---|+| **Alerts & triage** | `/api/v1/alerts` | Multi-source ingestion, severity normalisation |+| **Case management** | `/api/v1/cases` | Full lifecycle, observable graph, evidence chain |+| **Detection rules** | `/api/v1/detection-rules` | CRUD + Sigma/YARA-L2/KQL/SPL/ES\|QL |+| **Connectors** | `/api/v1/connectors` | EDR, SIEM, Cloud, IAM, SaaS, VCS, Network |+| **Playbooks** | `/api/v1/playbooks` | Runbook steps, action blocks, approvals |+| **Investigations** | `/api/v1/investigations` | Evidence collection, timeline, MITRE mapping |+| **Threat intelligence** | `/api/v1/threat-intel` | IOC lookup, enrichment, feed management |+| **Compliance evidence** | `/api/v1/compliance` | Audit trails with hash-chain integrity |+| **RBAC + tenants** | `/api/v1/rbac`, `/api/v1/tenants` | Multi-tenant, role-based access |+| **SLA tracking** | `/api/v1/sla` | MTTR/MTTD dashboards and breach alerts |+| **Marketplace** | `/api/v1/marketplace` | Community plugins and integrations |++---++## Tier 2 — Intelligent Automation++| Capability | API prefix | Description |+|---|---|---|+| **NL detection authoring** | `/api/v1/nl-detection` | Write detection rules in plain English; LLM converts to Sigma YAML |+| **Closed-loop detection engineering** | `/api/v1/detection-loop` | FP feedback → LLM drafts tuned Sigma → opens DAC proposal |+| **NL query → ES\|QL / SPL / KQL** | `/api/v1/nl-query` | Ask questions in natural language; get executable queries + chart spec |+| **Identity-centric timeline** | `/api/v1/identity-timeline` | Per-entity event timeline correlated across sources |+| **Cross-platform rule translation** | `/api/v1/translation` | Sigma ↔ SPL ↔ KQL ↔ ES\|QL ↔ YARA-L2 / UDM bidirectional conversion |+| **Hypothesis-driven hunting** | `/api/v1/hunts` | Define hypothesis → auto-generate multi-platform queries → track findings |++---++## Tier 3 — Advanced Analyst Workflows++| Capability | API prefix | Description |+|---|---|---|+| **Phishing triage** | `/api/v1/phishing` | Submit email/URL/attachment → LLM extracts IOCs, assigns verdict, maps MITRE |+| **Knowledge-base + RAG** | `/api/v1/kb` | Ingest runbooks/policies → full-text + LLM-synthesised answers |+| **Federated search** | `/api/v1/federated` | Cross-SIEM query fan-out with query translation |+| **Identity graph** | `/api/v1/identity-graph` | Entity relationship graph across IAM/CASB/EDR data |+| **Posture management** | `/api/v1/posture` | Asset hygiene scoring and drift detection |+| **Reports** | `/api/v1/reports` | Scheduled and on-demand PDF/JSON security reports |++---++## Detection Rule Formats++AiSOC translates between the following formats natively:++| Format | Read | Write |+|---|---|---|+| **Sigma YAML** | ✅ | ✅ |+| **Splunk SPL** | ✅ | ✅ |+| **Microsoft Sentinel KQL** | ✅ | ✅ |+| **Elastic ES\|QL** | ✅ | ✅ |+| **Google Chronicle YARA-L2** | ✅ | ✅ |+| **Google Chronicle UDM Search** | ✅ | read-only |++---++## Severity Ladder++All connectors normalise to the AiSOC four-tier severity model:++```+info → low → medium → high+```++Vendor-specific ladders (Azure 5-tier, SCC 5-tier, GitHub 4-tier) collapse to+this set in each connector's `normalize()` method.++---++## Compliance Frameworks Supported++The compliance evidence trail supports any framework tag. Built-in control+mappings are provided for:++- SOC 2 Type II+- PCI-DSS v4+- HIPAA Security Rule+- ISO 27001:2022+- NIST CSF 2.0++Evidence records form a hash-linked chain (SHA-256, similar to a blockchain)+to provide audit-grade tamper evidence.++---++## Connector Categories++| Category | Examples |+|---|---|+| `edr` | CrowdStrike, SentinelOne, Microsoft Defender |+| `siem` | Splunk, Microsoft Sentinel, Elastic Security |+| `cloud` | AWS CloudTrail, Azure Activity, GCP Audit |+| `iam` | Okta, Azure AD, Google Workspace |+| `saas` | GitHub, Slack, Salesforce, Jira |+| `vcs` | GitHub, GitLab |+| `network` | Palo Alto, Cisco, Zeek |++Add a new connector by implementing `BaseConnector` in+`services/connectors/app/connectors/<name>.py` and registering it in+`_CONNECTOR_CLASSES`. No other wiring required.diff --git a/apps/web/src/components/dashboard/SOCMetricsDashboard.tsx b/apps/web/src/components/dashboard/SOCMetricsDashboard.tsxnew file mode 100644index 000000000..a31f6ca9a--- /dev/null+++ b/apps/web/src/components/dashboard/SOCMetricsDashboard.tsx@@ -0,0 +1,229 @@+"use client";++import { useCallback } from "react";+import useSWR from "swr";++interface SOCKpis {+ mttd_hours: number;+ mttr_hours: number;+ false_positive_rate: number;+ alert_volume_7d: number;+ cases_opened_7d: number;+ cases_closed_7d: number;+ analyst_overrides_7d: number;+}++interface AttackHeatmapCell {+ tactic: string;+ technique: string;+ count: number;+}++interface SOCMetrics {+ kpis: SOCKpis;+ attack_heatmap: AttackHeatmapCell[];+}++const fetcher = (url: string) =>+ fetch(url, { credentials: "include" }).then((r) => r.json());++function KpiCard({+ label,+ value,+ unit,+ color,+}: {+ label: string;+ value: string | number;+ unit?: string;+ color?: string;+}) {+ return (+ <div className="bg-gray-900 border border-gray-700 rounded-lg p-4 flex flex-col gap-1">+ <span className="text-xs text-gray-400 uppercase tracking-wider">{label}</span>+ <span className={`text-2xl font-bold ${color ?? "text-white"}`}>+ {value}+ {unit && <span className="text-sm font-normal text-gray-400 ml-1">{unit}</span>}+ </span>+ </div>+ );+}++const TACTIC_COLORS: Record<string, string> = {+ "Initial Access": "bg-red-900",+ Execution: "bg-orange-900",+ Persistence: "bg-yellow-900",+ "Privilege Escalation": "bg-amber-900",+ "Defense Evasion": "bg-lime-900",+ "Credential Access": "bg-green-900",+ Discovery: "bg-teal-900",+ "Lateral Movement": "bg-cyan-900",+ Collection: "bg-sky-900",+ Exfiltration: "bg-blue-900",+ "Command and Control": "bg-indigo-900",+ Impact: "bg-purple-900",+};++function AttackHeatmap({ cells }: { cells: AttackHeatmapCell[] }) {+ const tacticGroups: Record<string, AttackHeatmapCell[]> = {};+ for (const cell of cells) {+ if (!tacticGroups[cell.tactic]) tacticGroups[cell.tactic] = [];+ tacticGroups[cell.tactic].push(cell);+ }++ const maxCount = Math.max(...cells.map((c) => c.count), 1);++ if (cells.length === 0) {+ return (+ <div className="text-gray-500 text-sm flex items-center justify-center h-32">+ No ATT&CK data in the selected period.+ </div>+ );+ }++ return (+ <div className="space-y-4">+ {Object.entries(tacticGroups).map(([tactic, techniques]) => (+ <div key={tactic}>+ <div className="text-xs font-semibold text-gray-300 mb-1 uppercase tracking-wide">+ {tactic}+ </div>+ <div className="flex flex-wrap gap-1">+ {techniques.map((cell) => {+ const intensity = Math.max(0.15, cell.count / maxCount);+ const bgClass = TACTIC_COLORS[tactic] ?? "bg-gray-800";+ return (+ <div+ key={cell.technique}+ title={`${cell.technique}: ${cell.count} alerts`}+ className={`${bgClass} border border-gray-600 rounded px-2 py-1 text-xs text-gray-200 cursor-default`}+ style={{ opacity: intensity }}+ >+ {cell.technique}+ <span className="ml-1 text-gray-400">({cell.count})</span>+ </div>+ );+ })}+ </div>+ </div>+ ))}+ </div>+ );+}++export function SOCMetricsDashboard() {+ const { data, error, isLoading, mutate } = useSWR<SOCMetrics>(+ "/api/v1/metrics/soc",+ fetcher,+ { refreshInterval: 60_000 }+ );++ const refresh = useCallback(() => mutate(), [mutate]);++ if (error) {+ return (+ <div className="p-6 text-red-400 text-sm">+ Failed to load SOC metrics. {error?.message ?? ""}+ </div>+ );+ }++ const kpis = data?.kpis;+ const heatmap = data?.attack_heatmap ?? [];++ return (+ <div className="space-y-6">+ {/* Header */}+ <div className="flex items-center justify-between">+ <h2 className="text-lg font-semibold text-white">SOC Performance Metrics</h2>+ <button+ onClick={refresh}+ className="text-xs text-gray-400 hover:text-gray-200 px-3 py-1 border border-gray-700 rounded transition-colors"+ >+ Refresh+ </button>+ </div>++ {/* KPI Grid */}+ {isLoading ? (+ <div className="grid grid-cols-2 md:grid-cols-4 gap-3">+ {Array.from({ length: 7 }).map((_, i) => (+ <div key={i} className="bg-gray-900 border border-gray-700 rounded-lg p-4 animate-pulse h-20" />+ ))}+ </div>+ ) : (+ <div className="grid grid-cols-2 md:grid-cols-4 gap-3">+ <KpiCard+ label="MTTD"+ value={kpis?.mttd_hours.toFixed(1) ?? "—"}+ unit="hrs"+ color={+ (kpis?.mttd_hours ?? 0) > 4+ ? "text-red-400"+ : (kpis?.mttd_hours ?? 0) > 2+ ? "text-yellow-400"+ : "text-green-400"+ }+ />+ <KpiCard+ label="MTTR"+ value={kpis?.mttr_hours.toFixed(1) ?? "—"}+ unit="hrs"+ color={+ (kpis?.mttr_hours ?? 0) > 24+ ? "text-red-400"+ : (kpis?.mttr_hours ?? 0) > 8+ ? "text-yellow-400"+ : "text-green-400"+ }+ />+ <KpiCard+ label="False Positive Rate"+ value={+ kpis+ ? `${(kpis.false_positive_rate * 100).toFixed(1)}%`+ : "—"+ }+ color={+ (kpis?.false_positive_rate ?? 0) > 0.3+ ? "text-red-400"+ : (kpis?.false_positive_rate ?? 0) > 0.15+ ? "text-yellow-400"+ : "text-green-400"+ }+ />+ <KpiCard+ label="Alert Volume (7d)"+ value={kpis?.alert_volume_7d ?? "—"}+ />+ <KpiCard+ label="Cases Opened (7d)"+ value={kpis?.cases_opened_7d ?? "—"}+ />+ <KpiCard+ label="Cases Closed (7d)"+ value={kpis?.cases_closed_7d ?? "—"}+ color="text-green-400"+ />+ <KpiCard+ label="Analyst Overrides (7d)"+ value={kpis?.analyst_overrides_7d ?? "—"}+ color="text-blue-400"+ />+ </div>+ )}++ {/* ATT&CK Heatmap */}+ <div className="bg-gray-900 border border-gray-700 rounded-lg p-4">+ <h3 className="text-sm font-semibold text-gray-300 mb-4">+ ATT&CK Technique Heatmap+ </h3>+ {isLoading ? (+ <div className="h-40 animate-pulse bg-gray-800 rounded" />+ ) : (+ <AttackHeatmap cells={heatmap} />+ )}+ </div>+ </div>+ );+}diff --git a/apps/web/src/components/landing/Hero.tsx b/apps/web/src/components/landing/Hero.tsxindex 736938e54..a71f9e73c 100644--- a/apps/web/src/components/landing/Hero.tsx+++ b/apps/web/src/components/landing/Hero.tsx@@ -30,9 +30,11 @@ export function Hero() {<p className="mt-6 max-w-xl text-lg leading-relaxed text-gray-300 md:text-xl">Agent prompts, tool calls, and decisions are recorded in an investigation- ledger and replayable per case. Alert fusion, entity risk rollups, and purple-team- drills ship in-repo alongside connectors and detection-as-code. The stack is gated- by a 200-incident eval harness on every PR to{' '}+ ledger and replayable per case. Three-tier memory, autonomy guardrails,+ NL detection authoring, closed-loop FP correction, cross-platform rule+ translation, hypothesis-driven hunting, phishing triage, and a knowledge-base+ RAG over your runbooks ship in-repo — gated by a 200-incident eval harness on+ every PR to{' '}<code className="rounded bg-white/[0.06] px-1 py-0.5 font-mono text-base">main</code>{' '}/{' '}<code className="rounded bg-white/[0.06] px-1 py-0.5 font-mono text-base">develop</code>.diff --git a/services/agents/app/core/cost_telemetry.py b/services/agents/app/core/cost_telemetry.pynew file mode 100644index 000000000..474cf6aec--- /dev/null+++ b/services/agents/app/core/cost_telemetry.py@@ -0,0 +1,256 @@+"""Investigation cost telemetry.++Tracks token usage, model calls, latency, and estimated USD cost per run.+Persists aggregates to PostgreSQL (best-effort) and emits structlog events+that feed into the SOC metrics dashboard.++Usage::++ from app.core.cost_telemetry import CostTracker++ async with CostTracker(run_id="r1", tenant_id="t1") as tracker:+ result = await llm_call(...)+ tracker.record(+ model="gpt-4o",+ prompt_tokens=result.usage.prompt_tokens,+ completion_tokens=result.usage.completion_tokens,+ latency_ms=elapsed_ms,+ )+ # On __aexit__, aggregates are flushed to DB.+"""++from __future__ import annotations++import os+import time+from dataclasses import dataclass, field+from typing import Any++import structlog++logger = structlog.get_logger()++# ---------------------------------------------------------------------------+# Model pricing (USD per 1 k tokens, input / output)+# ---------------------------------------------------------------------------+_PRICING: dict[str, tuple[float, float]] = {+ "gpt-4o": (0.005, 0.015),+ "gpt-4o-mini": (0.00015, 0.0006),+ "gpt-4-turbo": (0.01, 0.03),+ "gpt-4": (0.03, 0.06),+ "gpt-3.5-turbo": (0.0005, 0.0015),+ "claude-3-5-sonnet-20241022": (0.003, 0.015),+ "claude-3-opus-20240229": (0.015, 0.075),+ "claude-3-haiku-20240307": (0.00025, 0.00125),+ "gemini-1.5-pro": (0.00125, 0.005),+ "gemini-1.5-flash": (0.000075, 0.0003),+}++_DEFAULT_PRICE = (0.001, 0.002)+++def _estimate_cost(model: str, prompt_tokens: int, completion_tokens: int) -> float:+ in_price, out_price = _PRICING.get(model.lower(), _DEFAULT_PRICE)+ return (prompt_tokens / 1000) * in_price + (completion_tokens / 1000) * out_price+++# ---------------------------------------------------------------------------+# Per-call record+# ---------------------------------------------------------------------------+++@dataclass+class CallRecord:+ model: str+ prompt_tokens: int+ completion_tokens: int+ latency_ms: float+ cost_usd: float = 0.0+ tool: str | None = None+ step: str | None = None++ def __post_init__(self) -> None:+ self.cost_usd = _estimate_cost(self.model, self.prompt_tokens, self.completion_tokens)+++# ---------------------------------------------------------------------------+# DB persistence+# ---------------------------------------------------------------------------++_POOL: Any = None+++async def _get_pool() -> Any | None:+ global _POOL+ if _POOL is not None:+ return _POOL+ dsn = os.environ.get("DATABASE_URL", "").strip()+ if not dsn:+ return None+ try:+ import asyncpg # type: ignore[import]++ pool = await asyncpg.create_pool(+ dsn.replace("postgresql+asyncpg://", "postgresql://").replace(+ "postgres+asyncpg://", "postgresql://"+ ),+ min_size=1,+ max_size=2,+ )+ async with pool.acquire() as conn:+ await conn.execute(+ """+ CREATE TABLE IF NOT EXISTS aisoc_run_costs (+ run_id TEXT NOT NULL,+ tenant_id TEXT NOT NULL,+ model TEXT,+ total_prompt_tokens INTEGER NOT NULL DEFAULT 0,+ total_completion_tokens INTEGER NOT NULL DEFAULT 0,+ total_cost_usd DOUBLE PRECISION NOT NULL DEFAULT 0,+ total_latency_ms DOUBLE PRECISION NOT NULL DEFAULT 0,+ call_count INTEGER NOT NULL DEFAULT 0,+ recorded_at TIMESTAMPTZ NOT NULL DEFAULT now(),+ PRIMARY KEY (run_id, tenant_id, model)+ );+ CREATE INDEX IF NOT EXISTS aisoc_run_costs_tenant_run+ ON aisoc_run_costs (tenant_id, run_id);+ """+ )+ _POOL = pool+ return _POOL+ except Exception as exc:+ logger.debug("cost_telemetry.db_unavailable", error=str(exc))+ return None+++async def _flush_to_db(+ run_id: str,+ tenant_id: str,+ records: list[CallRecord],+) -> None:+ if not records:+ return+ pool = await _get_pool()+ if pool is None:+ return++ # Aggregate by model+ by_model: dict[str, dict] = {}+ for r in records:+ m = by_model.setdefault(r.model, {+ "prompt": 0, "completion": 0, "cost": 0.0, "latency": 0.0, "calls": 0+ })+ m["prompt"] += r.prompt_tokens+ m["completion"] += r.completion_tokens+ m["cost"] += r.cost_usd+ m["latency"] += r.latency_ms+ m["calls"] += 1++ try:+ async with pool.acquire() as conn:+ for model, agg in by_model.items():+ await conn.execute(+ """+ INSERT INTO aisoc_run_costs+ (run_id, tenant_id, model,+ total_prompt_tokens, total_completion_tokens,+ total_cost_usd, total_latency_ms, call_count)+ VALUES ($1, $2, $3, $4, $5, $6, $7, $8)+ ON CONFLICT (run_id, tenant_id, model) DO UPDATE+ SET total_prompt_tokens = aisoc_run_costs.total_prompt_tokens + EXCLUDED.total_prompt_tokens,+ total_completion_tokens = aisoc_run_costs.total_completion_tokens + EXCLUDED.total_completion_tokens,+ total_cost_usd = aisoc_run_costs.total_cost_usd + EXCLUDED.total_cost_usd,+ total_latency_ms = aisoc_run_costs.total_latency_ms + EXCLUDED.total_latency_ms,+ call_count = aisoc_run_costs.call_count + EXCLUDED.call_count,+ recorded_at = now()+ """,+ run_id,+ tenant_id,+ model,+ agg["prompt"],+ agg["completion"],+ agg["cost"],+ agg["latency"],+ agg["calls"],+ )+ except Exception as exc:+ logger.warning("cost_telemetry.flush_error", run_id=run_id, error=str(exc))+++# ---------------------------------------------------------------------------+# CostTracker+# ---------------------------------------------------------------------------+++@dataclass+class CostTracker:+ run_id: str+ tenant_id: str+ _records: list[CallRecord] = field(default_factory=list, init=False)+ _start: float = field(default_factory=time.monotonic, init=False)++ async def __aenter__(self) -> "CostTracker":+ return self++ async def __aexit__(self, *_: Any) -> None:+ await self.flush()++ def record(+ self,+ model: str,+ prompt_tokens: int,+ completion_tokens: int,+ latency_ms: float,+ tool: str | None = None,+ step: str | None = None,+ ) -> CallRecord:+ rec = CallRecord(+ model=model,+ prompt_tokens=prompt_tokens,+ completion_tokens=completion_tokens,+ latency_ms=latency_ms,+ tool=tool,+ step=step,+ )+ self._records.append(rec)+ logger.info(+ "cost_telemetry.call",+ run_id=self.run_id,+ tenant_id=self.tenant_id,+ model=model,+ prompt_tokens=prompt_tokens,+ completion_tokens=completion_tokens,+ cost_usd=f"{rec.cost_usd:.6f}",+ latency_ms=f"{latency_ms:.1f}",+ tool=tool,+ step=step,+ )+ return rec++ @property+ def total_cost_usd(self) -> float:+ return sum(r.cost_usd for r in self._records)++ @property+ def total_tokens(self) -> int:+ return sum(r.prompt_tokens + r.completion_tokens for r in self._records)++ @property+ def total_latency_ms(self) -> float:+ return sum(r.latency_ms for r in self._records)++ def summary(self) -> dict:+ return {+ "run_id": self.run_id,+ "tenant_id": self.tenant_id,+ "total_cost_usd": self.total_cost_usd,+ "total_tokens": self.total_tokens,+ "total_latency_ms": self.total_latency_ms,+ "call_count": len(self._records),+ "models": list({r.model for r in self._records}),+ }++ async def flush(self) -> None:+ summary = self.summary()+ logger.info("cost_telemetry.run_summary", **summary)+ await _flush_to_db(self.run_id, self.tenant_id, self._records)diff --git a/services/agents/app/memory/__init__.py b/services/agents/app/memory/__init__.pynew file mode 100644index 000000000..f93d446e4--- /dev/null+++ b/services/agents/app/memory/__init__.py@@ -0,0 +1,25 @@+"""Three-tier agent memory subsystem.++Tiers+-----+session In-process LRU — per-run ephemeral scratchpad. Zero I/O.+working Redis-backed — persists for the lifetime of a case/shift (~24 h).+institutional PostgreSQL-backed — permanent institutional knowledge;+ searchable via pgvector or keyword depending on availability.++Usage::++ from app.memory import MemoryManager++ mgr = await MemoryManager.create(tenant_id="t1", run_id="r1")+ await mgr.write_session("last_tool", {"name": "mitre_lookup", "result": ...})+ await mgr.write_working("suspect_ip", "10.0.0.5")+ await mgr.write_institutional("known_fp", {"rule": "sigma-001", "reason": "..."})++ hit = await mgr.recall("suspect_ip", tiers=("session", "working"))+"""++from .manager import MemoryManager+from .models import MemoryEntry, MemoryTier++__all__ = ["MemoryManager", "MemoryEntry", "MemoryTier"]diff --git a/services/agents/app/memory/institutional.py b/services/agents/app/memory/institutional.pynew file mode 100644index 000000000..e13e07a2b--- /dev/null+++ b/services/agents/app/memory/institutional.py@@ -0,0 +1,202 @@+"""Institutional-tier memory — PostgreSQL-backed, permanent knowledge store.++Schema (created lazily if it does not exist):++ CREATE TABLE IF NOT EXISTS aisoc_institutional_memory (+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),+ tenant_id TEXT NOT NULL,+ key TEXT NOT NULL,+ value JSONB NOT NULL,+ tags TEXT[] NOT NULL DEFAULT '{}',+ analyst_override BOOLEAN NOT NULL DEFAULT FALSE,+ override_reason TEXT,+ created_at TIMESTAMPTZ NOT NULL DEFAULT now(),+ UNIQUE (tenant_id, key)+ );++Falls back to an in-process dict when the database is unavailable.+"""++from __future__ import annotations++import json+import os+from typing import Any++import structlog++logger = structlog.get_logger()++_FALLBACK: dict[str, Any] = {}++_POOL: Any = None # asyncpg.Pool | None+++def _normalise_dsn(url: str) -> str:+ return url.replace("postgresql+asyncpg://", "postgresql://").replace(+ "postgres+asyncpg://", "postgresql://"+ )+++async def _get_pool() -> Any | None:+ global _POOL+ if _POOL is not None:+ return _POOL+ dsn = os.environ.get("DATABASE_URL", "").strip()+ if not dsn:+ return None+ try:+ import asyncpg # type: ignore[import]++ pool = await asyncpg.create_pool(_normalise_dsn(dsn), min_size=1, max_size=3)+ # Ensure table exists+ async with pool.acquire() as conn:+ await conn.execute(+ """+ CREATE TABLE IF NOT EXISTS aisoc_institutional_memory (+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),+ tenant_id TEXT NOT NULL,+ key TEXT NOT NULL,+ value JSONB NOT NULL,+ tags TEXT[] NOT NULL DEFAULT '{}',+ analyst_override BOOLEAN NOT NULL DEFAULT FALSE,+ override_reason TEXT,+ created_at TIMESTAMPTZ NOT NULL DEFAULT now(),+ UNIQUE (tenant_id, key)+ );+ CREATE INDEX IF NOT EXISTS aisoc_institutional_memory_tenant_key+ ON aisoc_institutional_memory (tenant_id, key);+ """+ )+ _POOL = pool+ return _POOL+ except Exception as exc:+ logger.debug("memory.institutional.db_unavailable", error=str(exc))+ return None+++async def institutional_get(tenant_id: str, key: str) -> Any | None:+ pool = await _get_pool()+ fk = f"{tenant_id}:{key}"+ if pool is not None:+ try:+ async with pool.acquire() as conn:+ row = await conn.fetchrow(+ "SELECT value FROM aisoc_institutional_memory WHERE tenant_id=$1 AND key=$2",+ tenant_id,+ key,+ )+ return json.loads(row["value"]) if row else None+ except Exception as exc:+ logger.warning("memory.institutional.get_error", key=fk, error=str(exc))+ return _FALLBACK.get(fk)+++async def institutional_set(+ tenant_id: str,+ key: str,+ value: Any,+ *,+ tags: list[str] | None = None,+ analyst_override: bool = False,+ override_reason: str | None = None,+) -> None:+ pool = await _get_pool()+ fk = f"{tenant_id}:{key}"+ serialised = json.dumps(value, default=str)+ if pool is not None:+ try:+ async with pool.acquire() as conn:+ await conn.execute(+ """+ INSERT INTO aisoc_institutional_memory+ (tenant_id, key, value, tags, analyst_override, override_reason)+ VALUES ($1, $2, $3::jsonb, $4, $5, $6)+ ON CONFLICT (tenant_id, key) DO UPDATE+ SET value = EXCLUDED.value,+ tags = EXCLUDED.tags,+ analyst_override = EXCLUDED.analyst_override,+ override_reason = EXCLUDED.override_reason,+ created_at = now()+ """,+ tenant_id,+ key,+ serialised,+ tags or [],+ analyst_override,+ override_reason,+ )+ return+ except Exception as exc:+ logger.warning("memory.institutional.set_error", key=fk, error=str(exc))+ _FALLBACK[fk] = json.loads(serialised)+++async def institutional_delete(tenant_id: str, key: str) -> None:+ pool = await _get_pool()+ fk = f"{tenant_id}:{key}"+ if pool is not None:+ try:+ async with pool.acquire() as conn:+ await conn.execute(+ "DELETE FROM aisoc_institutional_memory WHERE tenant_id=$1 AND key=$2",+ tenant_id,+ key,+ )+ except Exception as exc:+ logger.warning("memory.institutional.delete_error", key=fk, error=str(exc))+ _FALLBACK.pop(fk, None)+++async def institutional_search(+ tenant_id: str,+ tags: list[str] | None = None,+ limit: int = 20,+) -> list[dict]:+ """Return up to *limit* entries for *tenant_id*, optionally filtered by tags."""+ pool = await _get_pool()+ if pool is not None:+ try:+ async with pool.acquire() as conn:+ if tags:+ rows = await conn.fetch(+ """+ SELECT key, value, tags, analyst_override, override_reason, created_at+ FROM aisoc_institutional_memory+ WHERE tenant_id=$1 AND tags && $2+ ORDER BY created_at DESC LIMIT $3+ """,+ tenant_id,+ tags,+ limit,+ )+ else:+ rows = await conn.fetch(+ """+ SELECT key, value, tags, analyst_override, override_reason, created_at+ FROM aisoc_institutional_memory+ WHERE tenant_id=$1+ ORDER BY created_at DESC LIMIT $2+ """,+ tenant_id,+ limit,+ )+ return [+ {+ "key": r["key"],+ "value": json.loads(r["value"]),+ "tags": list(r["tags"]),+ "analyst_override": r["analyst_override"],+ "override_reason": r["override_reason"],+ "created_at": r["created_at"].isoformat(),+ }+ for r in rows+ ]+ except Exception as exc:+ logger.warning("memory.institutional.search_error", error=str(exc))+ # Fallback: return matching entries from in-process dict+ results = []+ for fk, val in _FALLBACK.items():+ if fk.startswith(f"{tenant_id}:"):+ results.append({"key": fk.split(":", 1)[1], "value": val, "tags": [], "analyst_override": False})+ return results[:limit]diff --git a/services/agents/app/memory/manager.py b/services/agents/app/memory/manager.pynew file mode 100644index 000000000..f2f4fe95d--- /dev/null+++ b/services/agents/app/memory/manager.py@@ -0,0 +1,181 @@+"""MemoryManager — unified interface over the three memory tiers."""++from __future__ import annotations++from typing import Any, Sequence++import structlog++from .institutional import (+ institutional_delete,+ institutional_get,+ institutional_search,+ institutional_set,+)+from .models import MemoryEntry, MemoryTier, OverrideFeedback+from .session import session_clear, session_delete, session_get, session_set+from .working import working_delete, working_get, working_set++logger = structlog.get_logger()++_ALL_TIERS: tuple[str, ...] = ("session", "working", "institutional")+++class MemoryManager:+ """Unified API for three-tier agent memory.++ Create via the async factory::++ mgr = await MemoryManager.create(tenant_id="t1", run_id="run-abc")+ """++ def __init__(self, tenant_id: str, run_id: str | None = None) -> None:+ self._tenant_id = tenant_id+ self._run_id = run_id++ @classmethod+ async def create(cls, tenant_id: str, run_id: str | None = None) -> "MemoryManager":+ """Async factory — currently a passthrough; reserved for future init work."""+ return cls(tenant_id=tenant_id, run_id=run_id)++ # ------------------------------------------------------------------+ # Write helpers+ # ------------------------------------------------------------------++ async def write_session(self, key: str, value: Any) -> None:+ """Write to the ephemeral session tier (in-process LRU)."""+ if self._run_id is None:+ raise ValueError("run_id is required for session memory")+ await session_set(self._run_id, key, value)++ async def write_working(+ self,+ key: str,+ value: Any,+ *,+ ttl: int = 86_400,+ ) -> None:+ """Write to the Redis-backed working tier (default TTL 24 h)."""+ await working_set(self._tenant_id, key, value, run_id=self._run_id, ttl=ttl)++ async def write_institutional(+ self,+ key: str,+ value: Any,+ *,+ tags: list[str] | None = None,+ analyst_override: bool = False,+ override_reason: str | None = None,+ ) -> None:+ """Write to the PostgreSQL institutional tier (permanent)."""+ await institutional_set(+ self._tenant_id,+ key,+ value,+ tags=tags,+ analyst_override=analyst_override,+ override_reason=override_reason,+ )++ # ------------------------------------------------------------------+ # Recall+ # ------------------------------------------------------------------++ async def recall(+ self,+ key: str,+ tiers: Sequence[str] = _ALL_TIERS,+ ) -> Any | None:+ """Search *tiers* in order; return first hit or None."""+ for tier in tiers:+ result = await self._get_from_tier(tier, key)+ if result is not None:+ return result+ return None++ async def _get_from_tier(self, tier: str, key: str) -> Any | None:+ if tier == MemoryTier.session:+ if self._run_id is None:+ return None+ return await session_get(self._run_id, key)+ if tier == MemoryTier.working:+ return await working_get(self._tenant_id, key, run_id=self._run_id)+ if tier == MemoryTier.institutional:+ return await institutional_get(self._tenant_id, key)+ logger.warning("memory.manager.unknown_tier", tier=tier)+ return None++ # ------------------------------------------------------------------+ # Delete+ # ------------------------------------------------------------------++ async def delete(self, key: str, tier: str) -> None:+ if tier == MemoryTier.session:+ if self._run_id:+ await session_delete(self._run_id, key)+ elif tier == MemoryTier.working:+ await working_delete(self._tenant_id, key, run_id=self._run_id)+ elif tier == MemoryTier.institutional:+ await institutional_delete(self._tenant_id, key)++ async def clear_session(self) -> None:+ """Evict all session entries for the current run (call at investigation end)."""+ if self._run_id:+ await session_clear(self._run_id)++ # ------------------------------------------------------------------+ # Search+ # ------------------------------------------------------------------++ async def search_institutional(+ self,+ tags: list[str] | None = None,+ limit: int = 20,+ ) -> list[dict]:+ """Return institutional memory entries, optionally filtered by *tags*."""+ return await institutional_search(self._tenant_id, tags=tags, limit=limit)++ # ------------------------------------------------------------------+ # Analyst override ingestion+ # ------------------------------------------------------------------++ async def ingest_override(self, feedback: OverrideFeedback) -> None:+ """Ingest an analyst verdict correction into institutional memory."""+ key = f"analyst_override:{feedback.alert_id}"+ value = feedback.model_dump()+ await self.write_institutional(+ key,+ value,+ tags=["analyst_override"],+ analyst_override=True,+ override_reason=feedback.reason,+ )+ logger.info(+ "memory.institutional.override_ingested",+ alert_id=feedback.alert_id,+ original=feedback.original_verdict,+ corrected=feedback.corrected_verdict,+ analyst=feedback.analyst_id,+ )++ # ------------------------------------------------------------------+ # Convenience: read-back as MemoryEntry+ # ------------------------------------------------------------------++ async def recall_entry(+ self,+ key: str,+ tiers: Sequence[str] = _ALL_TIERS,+ ) -> MemoryEntry | None:+ """Like recall() but wraps the result in a MemoryEntry for richer context."""+ for tier in tiers:+ value = await self._get_from_tier(tier, key)+ if value is not None:+ return MemoryEntry(+ tier=MemoryTier(tier),+ tenant_id=self._tenant_id,+ run_id=self._run_id,+ key=key,+ value=value,+ )+ return Nonediff --git a/services/agents/app/memory/models.py b/services/agents/app/memory/models.pynew file mode 100644index 000000000..0d0ad1d7a--- /dev/null+++ b/services/agents/app/memory/models.py@@ -0,0 +1,42 @@+"""Shared data models for the three-tier memory system."""++from __future__ import annotations++import enum+from datetime import datetime+from typing import Any++from pydantic import BaseModel, Field+++class MemoryTier(str, enum.Enum):+ session = "session"+ working = "working"+ institutional = "institutional"+++class MemoryEntry(BaseModel):+ tier: MemoryTier+ tenant_id: str+ run_id: str | None = None+ key: str+ value: Any+ created_at: datetime = Field(default_factory=datetime.utcnow)+ ttl_seconds: int | None = None+ tags: list[str] = Field(default_factory=list)+ analyst_override: bool = False+ override_reason: str | None = None+++class OverrideFeedback(BaseModel):+ """Captures analyst correction for retroactive re-disposition and+ institutional memory ingestion."""++ tenant_id: str+ run_id: str+ alert_id: str+ original_verdict: str+ corrected_verdict: str+ analyst_id: str | None = None+ reason: str | None = None+ created_at: datetime = Field(default_factory=datetime.utcnow)diff --git a/services/agents/app/memory/session.py b/services/agents/app/memory/session.pynew file mode 100644index 000000000..1e7683099--- /dev/null+++ b/services/agents/app/memory/session.py@@ -0,0 +1,70 @@+"""Session-tier memory — in-process LRU, zero I/O, per-run lifetime.++Intentionally simple: a bounded dict keyed by (run_id, key). The LRU+evicts the oldest entry when the cache exceeds *maxsize* (default 512).+Thread-safe via a lock; suitable for asyncio tasks sharing an event loop.+"""++from __future__ import annotations++import asyncio+from collections import OrderedDict+from typing import Any+++_DEFAULT_MAXSIZE = 512+_session_caches: dict[str, "_SessionCache"] = {}+_registry_lock = asyncio.Lock()+++class _SessionCache:+ def __init__(self, maxsize: int = _DEFAULT_MAXSIZE) -> None:+ self._store: OrderedDict[str, Any] = OrderedDict()+ self._maxsize = maxsize++ def get(self, key: str) -> Any | None:+ if key in self._store:+ self._store.move_to_end(key)+ return self._store[key]+ return None++ def set(self, key: str, value: Any) -> None:+ if key in self._store:+ self._store.move_to_end(key)+ self._store[key] = value+ if len(self._store) > self._maxsize:+ self._store.popitem(last=False)++ def delete(self, key: str) -> None:+ self._store.pop(key, None)++ def clear(self) -> None:+ self._store.clear()+++async def _get_cache(run_id: str) -> "_SessionCache":+ async with _registry_lock:+ if run_id not in _session_caches:+ _session_caches[run_id] = _SessionCache()+ return _session_caches[run_id]+++async def session_get(run_id: str, key: str) -> Any | None:+ cache = await _get_cache(run_id)+ return cache.get(key)+++async def session_set(run_id: str, key: str, value: Any) -> None:+ cache = await _get_cache(run_id)+ cache.set(key, value)+++async def session_delete(run_id: str, key: str) -> None:+ cache = await _get_cache(run_id)+ cache.delete(key)+++async def session_clear(run_id: str) -> None:+ """Evict all session entries for *run_id* (called at investigation end)."""+ async with _registry_lock:+ _session_caches.pop(run_id, None)diff --git a/services/agents/app/memory/working.py b/services/agents/app/memory/working.pynew file mode 100644index 000000000..6cc891e0f--- /dev/null+++ b/services/agents/app/memory/working.py@@ -0,0 +1,91 @@+"""Working-tier memory — Redis-backed, per-case/shift lifetime (~24 h).++Falls back to an in-process dict when Redis is unavailable so the agent+keeps running in development or minimal deployments.++Keys are namespaced as ``aisoc:mem:working:{tenant_id}:{run_id}:{key}``+to prevent cross-tenant bleed.+"""++from __future__ import annotations++import json+import os+from typing import Any++import structlog++logger = structlog.get_logger()++_DEFAULT_TTL = 86_400 # 24 h+_FALLBACK: dict[str, Any] = {}++try:+ import redis.asyncio as aioredis # type: ignore[import]++ _redis: aioredis.Redis | None = None++ async def _get_redis() -> "aioredis.Redis | None":+ global _redis+ if _redis is not None:+ return _redis+ url = os.environ.get("REDIS_URL", "").strip()+ if not url:+ return None+ try:+ _redis = aioredis.from_url(url, decode_responses=True)+ await _redis.ping()+ return _redis+ except Exception as exc:+ logger.debug("memory.working.redis_unavailable", error=str(exc))+ return None++except ImportError:+ async def _get_redis(): # type: ignore[misc]+ return None+++def _key(tenant_id: str, run_id: str | None, key: str) -> str:+ return f"aisoc:mem:working:{tenant_id}:{run_id or '*'}:{key}"+++async def working_get(tenant_id: str, key: str, run_id: str | None = None) -> Any | None:+ r = await _get_redis()+ full_key = _key(tenant_id, run_id, key)+ if r is not None:+ try:+ raw = await r.get(full_key)+ return json.loads(raw) if raw is not None else None+ except Exception as exc:+ logger.warning("memory.working.get_error", key=full_key, error=str(exc))+ return _FALLBACK.get(full_key)+++async def working_set(+ tenant_id: str,+ key: str,+ value: Any,+ run_id: str | None = None,+ ttl: int = _DEFAULT_TTL,+) -> None:+ r = await _get_redis()+ full_key = _key(tenant_id, run_id, key)+ serialised = json.dumps(value, default=str)+ if r is not None:+ try:+ await r.setex(full_key, ttl, serialised)+ return+ except Exception as exc:+ logger.warning("memory.working.set_error", key=full_key, error=str(exc))+ _FALLBACK[full_key] = json.loads(serialised)+++async def working_delete(tenant_id: str, key: str, run_id: str | None = None) -> None:+ r = await _get_redis()+ full_key = _key(tenant_id, run_id, key)+ if r is not None:+ try:+ await r.delete(full_key)+ except Exception as exc:+ logger.warning("memory.working.delete_error", key=full_key, error=str(exc))+ _FALLBACK.pop(full_key, None)diff --git a/services/agents/app/policy/__init__.py b/services/agents/app/policy/__init__.pynew file mode 100644index 000000000..caed56159--- /dev/null+++ b/services/agents/app/policy/__init__.py@@ -0,0 +1,24 @@+"""Autonomy guardrails — per-action confidence thresholds.++Each action the agent can take (e.g. ``block_ip``, ``isolate_host``)+is associated with a minimum confidence threshold. If the agent's+self-assessed confidence for an action is below the threshold, the+action is held for human review rather than executed autonomously.++Defaults live here; tenant admins can override via the DB / API.++Usage::++ from app.policy import GuardrailPolicy, ActionResult++ policy = await GuardrailPolicy.load(tenant_id="t1")+ result = policy.evaluate("block_ip", confidence=0.72)+ if result.allowed:+ await do_block(ip)+ else:+ await queue_for_human_review(action, result.reason)+"""++from .guardrails import ActionResult, GuardrailPolicy++__all__ = ["GuardrailPolicy", "ActionResult"]diff --git a/services/agents/app/policy/guardrails.py b/services/agents/app/policy/guardrails.pynew file mode 100644index 000000000..d08a2775a--- /dev/null+++ b/services/agents/app/policy/guardrails.py@@ -0,0 +1,143 @@+"""Per-action confidence guardrails with DB-backed tenant overrides."""++from __future__ import annotations++import os+from dataclasses import dataclass, field+from typing import Any++import structlog++logger = structlog.get_logger()++# ---------------------------------------------------------------------------+# Default thresholds+# action_name -> minimum confidence (0.0–1.0) before autonomous execution.+# Actions not listed here require confidence >= 1.0 (i.e. always require review).+# ---------------------------------------------------------------------------+_DEFAULT_THRESHOLDS: dict[str, float] = {+ # Low-risk read actions — high autonomy+ "lookup_ip": 0.0,+ "lookup_domain": 0.0,+ "search_logs": 0.0,+ "enrich_alert": 0.0,+ "mitre_lookup": 0.0,+ "get_alert_context": 0.0,+ # Medium-risk actions — require reasonable confidence+ "add_alert_tag": 0.5,+ "close_alert": 0.6,+ "create_case": 0.5,+ "add_case_comment": 0.4,+ "assign_case": 0.6,+ # High-risk actions — require high confidence+ "quarantine_file": 0.85,+ "block_ip": 0.90,+ "isolate_host": 0.92,+ "disable_user_account": 0.90,+ "revoke_session": 0.80,+ "delete_object": 0.95,+ "firewall_rule_add": 0.88,+ "firewall_rule_remove": 0.90,+}++_POOL: Any = None # asyncpg.Pool | None+_TENANT_OVERRIDES: dict[str, dict[str, float]] = {} # tenant_id -> {action: threshold}+++async def _load_overrides(tenant_id: str) -> dict[str, float]:+ """Load tenant-specific threshold overrides from the DB (best-effort)."""+ if tenant_id in _TENANT_OVERRIDES:+ return _TENANT_OVERRIDES[tenant_id]+ global _POOL+ dsn = os.environ.get("DATABASE_URL", "").strip()+ if not dsn:+ _TENANT_OVERRIDES[tenant_id] = {}+ return {}+ try:+ import asyncpg # type: ignore[import]++ if _POOL is None:+ _POOL = await asyncpg.create_pool(+ dsn.replace("postgresql+asyncpg://", "postgresql://").replace(+ "postgres+asyncpg://", "postgresql://"+ ),+ min_size=1,+ max_size=2,+ )+ async with _POOL.acquire() as conn:+ # Table created by the API service migration; may not exist in all envs+ rows = await conn.fetch(+ """+ SELECT action_name, min_confidence+ FROM aisoc_autonomy_thresholds+ WHERE tenant_id = $1+ """,+ tenant_id,+ )+ overrides = {r["action_name"]: float(r["min_confidence"]) for r in rows}+ _TENANT_OVERRIDES[tenant_id] = overrides+ return overrides+ except Exception as exc:+ logger.debug("policy.guardrails.overrides_unavailable", tenant_id=tenant_id, error=str(exc))+ _TENANT_OVERRIDES[tenant_id] = {}+ return {}+++# ---------------------------------------------------------------------------+# Public API+# ---------------------------------------------------------------------------+++@dataclass+class ActionResult:+ allowed: bool+ action: str+ confidence: float+ threshold: float+ reason: str = ""+++@dataclass+class GuardrailPolicy:+ tenant_id: str+ thresholds: dict[str, float] = field(default_factory=dict)++ @classmethod+ async def load(cls, tenant_id: str) -> "GuardrailPolicy":+ overrides = await _load_overrides(tenant_id)+ merged = {**_DEFAULT_THRESHOLDS, **overrides}+ return cls(tenant_id=tenant_id, thresholds=merged)++ def evaluate(self, action: str, confidence: float) -> ActionResult:+ """Return an ActionResult indicating whether the action is allowed."""+ threshold = self.thresholds.get(action, 1.0)+ allowed = confidence >= threshold+ reason = (+ ""+ if allowed+ else (+ f"Confidence {confidence:.2f} is below threshold {threshold:.2f} "+ f"for action '{action}'. Human approval required."+ )+ )+ if not allowed:+ logger.info(+ "policy.guardrails.blocked",+ action=action,+ confidence=confidence,+ threshold=threshold,+ tenant_id=self.tenant_id,+ )+ return ActionResult(+ allowed=allowed,+ action=action,+ confidence=confidence,+ threshold=threshold,+ reason=reason,+ )++ def get_threshold(self, action: str) -> float:+ return self.thresholds.get(action, 1.0)++ def all_thresholds(self) -> dict[str, float]:+ return dict(self.thresholds)diff --git a/services/api/app/api/v1/endpoints/cases.py b/services/api/app/api/v1/endpoints/cases.pyindex 760f75fc0..3917d68b9 100644--- a/services/api/app/api/v1/endpoints/cases.py+++ b/services/api/app/api/v1/endpoints/cases.py@@ -1,449 +1,422 @@-"""Case management endpoints."""+"""First-class case management (tier2-cases).++Implements the full case lifecycle: new → triaged → investigating → contained →+resolved → closed. Each case aggregates multiple alerts, maintains an observable+graph, evidence chain, and MITRE ATT&CK coverage map.++Endpoints+---------+* ``GET /cases`` List cases (filterable by status/severity/assignee).+* ``POST /cases`` Create a new case.+* ``GET /cases/{id}`` Retrieve a case.+* ``PATCH /cases/{id}`` Update case fields (status, severity, assignee, …).+* ``POST /cases/{id}/alerts`` Link alerts to a case.+* ``POST /cases/{id}/observables`` Add / update observable graph nodes/edges.+* ``POST /cases/{id}/comments`` Add a comment / timeline entry.+* ``GET /cases/{id}/comments`` List comments for a case.+* ``GET /cases/{id}/evidence`` Export the evidence chain as a structured report.+"""++from __future__ import annotations-import os-import reimport uuidfrom datetime import UTC, datetime-from typing import Annotated, Any+from typing import Any, Literal-_RUN_ID_RE = re.compile(r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$", re.IGNORECASE)--import httpx-from fastapi import APIRouter, Depends, HTTPException, Query, status+from fastapi import APIRouter, HTTPException, Query, statusfrom pydantic import BaseModel, Field-from sqlalchemy import and_, func, select, update--from app.api.v1.deps import AuthUser, DBSession, require_permission-from app.db.rls import TenantDBSession-from app.models.case import Case, CaseTimeline+from sqlalchemy import text-_AGENTS_URL = os.getenv("AGENTS_SERVICE_URL", "http://agents:8084")+from app.api.v1.deps import AuthUser, DBSessionrouter = APIRouter(prefix="/cases", tags=["cases"])+# ────────────────────────────────────────────────────────────────────────────+# Pydantic schemas+# ────────────────────────────────────────────────────────────────────────────-class CaseResponse(BaseModel):- id: uuid.UUID- tenant_id: uuid.UUID- case_number: str- title: str- description: str | None- status: str- priority: str- severity: str- case_type: str- mitre_tactics: list- mitre_techniques: list- assigned_to_id: uuid.UUID | None- sla_deadline: datetime | None- sla_breached: bool- alert_ids: list- tags: list- ticket_refs: list- summary: str | None- resolution: str | None- closed_at: datetime | None- created_at: datetime- updated_at: datetime-- model_config = {"from_attributes": True}-+CaseStatus = Literal["new", "triaged", "investigating", "contained", "resolved", "closed"]+CaseSeverity = Literal["info", "low", "medium", "high", "critical"]-class CaseListResponse(BaseModel):- items: list[CaseResponse]- total: int- page: int- page_size: int- pages: int+# Valid forward-only state transitions+_TRANSITIONS: dict[str, set[str]] = {+ "new": {"triaged"},+ "triaged": {"investigating"},+ "investigating": {"contained", "resolved"},+ "contained": {"resolved"},+ "resolved": {"closed"},+ "closed": set(),+}class CreateCaseRequest(BaseModel):- title: str+ title: str = Field(..., min_length=3)description: str | None = None- priority: str = "medium"- severity: str = "medium"- case_type: str = "security_incident"- alert_ids: list[uuid.UUID] = []- tags: list[str] = []+ severity: CaseSeverity = "medium"+ assignee: str | None = None+ alert_ids: list[uuid.UUID] = Field(default_factory=list)+ mitre_techniques: list[str] = Field(default_factory=list)+ compliance_frameworks: list[str] = Field(default_factory=list)+ tags: dict[str, str] = Field(default_factory=dict)+ sla_due_at: datetime | None = Noneclass UpdateCaseRequest(BaseModel):title: str | None = Nonedescription: str | None = None- status: str | None = None- priority: str | None = None- severity: str | None = None- assigned_to_id: uuid.UUID | None = None- tags: list[str] | None = None- resolution: str | None = None- lessons_learned: str | None = None+ severity: CaseSeverity | None = None+ status: CaseStatus | None = None+ assignee: str | None = None+ mitre_techniques: list[str] | None = None+ compliance_frameworks: list[str] | None = None+ tags: dict[str, str] | None = None+ sla_due_at: datetime | None = None-class AddTimelineEventRequest(BaseModel):- content: str- event_type: str = "comment"- metadata: dict = {}+class AddAlertsRequest(BaseModel):+ alert_ids: list[uuid.UUID]-class TimelineEventResponse(BaseModel):- id: uuid.UUID- case_id: uuid.UUID- event_type: str- content: str- metadata: dict = Field(default_factory=dict, validation_alias="event_metadata")- user_id: uuid.UUID | None- is_automated: bool- created_at: datetime+class ObservableNode(BaseModel):+ id: str+ kind: Literal["ip", "user", "host", "domain", "hash", "file", "process", "alert"]+ value: str+ tags: list[str] = Field(default_factory=list)- model_config = {"from_attributes": True, "populate_by_name": True}+class ObservableEdge(BaseModel):+ source: str+ target: str+ relation: str-def _generate_case_number(tenant_id: uuid.UUID) -> str:- """Generate a human-readable case number."""- ts = int(datetime.now(UTC).timestamp())- prefix = str(tenant_id)[:4].upper()- return f"CASE-{prefix}-{ts % 1000000:06d}"+class UpdateObservablesRequest(BaseModel):+ nodes: list[ObservableNode] = Field(default_factory=list)+ edges: list[ObservableEdge] = Field(default_factory=list)[email protected]("", response_model=CaseListResponse)-async def list_cases(- current_user: Annotated[AuthUser, Depends(require_permission("cases:read"))],- db: TenantDBSession,- page: int = Query(default=1, ge=1),- page_size: int = Query(default=25, ge=1, le=100),- status: str | None = None,- priority: str | None = None,- assigned_to_me: bool = False,-) -> CaseListResponse:- """List cases with filtering."""- filters = [Case.tenant_id == current_user.tenant_id]- if status:- filters.append(Case.status == status)- if priority:- filters.append(Case.priority == priority)- if assigned_to_me:- filters.append(Case.assigned_to_id == current_user.user_id)-- count_result = await db.execute(select(func.count()).select_from(Case).where(and_(*filters)))- total = count_result.scalar_one()-- offset = (page - 1) * page_size- result = await db.execute(select(Case).where(and_(*filters)).order_by(Case.created_at.desc()).offset(offset).limit(page_size))- cases = result.scalars().all()-- return CaseListResponse(- items=[CaseResponse.model_validate(c) for c in cases],- total=total,- page=page,- page_size=page_size,- pages=(total + page_size - 1) // page_size,- )+class AddCommentRequest(BaseModel):+ body: str = Field(..., min_length=1)+ is_system: bool = False[email protected]("", response_model=CaseResponse, status_code=status.HTTP_201_CREATED)-async def create_case(- request: CreateCaseRequest,- current_user: Annotated[AuthUser, Depends(require_permission("cases:write"))],- db: DBSession,-) -> CaseResponse:- """Create a new security case."""- case = Case(- tenant_id=current_user.tenant_id,- case_number=_generate_case_number(current_user.tenant_id),- title=request.title,- description=request.description,- priority=request.priority,- severity=request.severity,- case_type=request.case_type,- alert_ids=[str(aid) for aid in request.alert_ids],- tags=request.tags,- created_by_id=current_user.user_id,- )- db.add(case)- await db.flush()-- # Add creation timeline event- timeline_event = CaseTimeline(- case_id=case.id,- tenant_id=current_user.tenant_id,- event_type="case_created",- content=f"Case created by {current_user.email}",- user_id=current_user.user_id,- )- db.add(timeline_event)- await db.commit()- await db.refresh(case)-- return CaseResponse.model_validate(case)--[email protected]("/{case_id}", response_model=CaseResponse)-async def get_case(- case_id: uuid.UUID,- current_user: Annotated[AuthUser, Depends(require_permission("cases:read"))],- db: DBSession,-) -> CaseResponse:- """Get a case by ID."""- result = await db.execute(select(Case).where(Case.id == case_id, Case.tenant_id == current_user.tenant_id))- case = result.scalar_one_or_none()- if case is None:- raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Case not found")- return CaseResponse.model_validate(case)--[email protected]("/{case_id}", response_model=CaseResponse)-async def update_case(- case_id: uuid.UUID,- request: UpdateCaseRequest,- current_user: Annotated[AuthUser, Depends(require_permission("cases:write"))],- db: DBSession,-) -> CaseResponse:- """Update a case."""- result = await db.execute(select(Case).where(Case.id == case_id, Case.tenant_id == current_user.tenant_id))- case = result.scalar_one_or_none()- if case is None:- raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Case not found")-- updates: dict = {}- for field in ["title", "description", "status", "priority", "severity", "assigned_to_id", "tags", "resolution", "lessons_learned"]:- val = getattr(request, field, None)- if val is not None:- updates[field] = val-- if request.status in ("closed", "cancelled") and "closed_at" not in updates:- updates["closed_at"] = datetime.now(UTC)-- if updates:- updates["updated_at"] = datetime.now(UTC)- await db.execute(update(Case).where(Case.id == case_id).values(**updates))-- # Add timeline event- db.add(- CaseTimeline(- case_id=case_id,- tenant_id=current_user.tenant_id,- event_type="case_updated",- content=f"Case updated by {current_user.email}: {', '.join(updates.keys())}",- user_id=current_user.user_id,- event_metadata={"changed_fields": list(updates.keys())},- )- )- await db.commit()- await db.refresh(case)-- return CaseResponse.model_validate(case)+class CaseResponse(BaseModel):+ id: uuid.UUID+ title: str+ description: str | None+ severity: str+ status: str+ assignee: str | None+ mitre_techniques: list[str]+ alert_ids: list[uuid.UUID]+ observable_graph: dict[str, Any]+ evidence_chain: list[dict[str, Any]]+ compliance_frameworks: list[str]+ opened_at: datetime+ triaged_at: datetime | None+ resolved_at: datetime | None+ closed_at: datetime | None+ created_at: datetime+ updated_at: datetime+ created_by: str | None+ tags: dict[str, Any]+ sla_due_at: datetime | None[email protected]("/{case_id}/timeline", response_model=TimelineEventResponse, status_code=status.HTTP_201_CREATED)-async def add_timeline_event(- case_id: uuid.UUID,- request: AddTimelineEventRequest,- current_user: Annotated[AuthUser, Depends(require_permission("cases:write"))],- db: DBSession,-) -> TimelineEventResponse:- """Add a comment or event to the case timeline."""- # Verify case exists for this tenant- result = await db.execute(select(Case).where(Case.id == case_id, Case.tenant_id == current_user.tenant_id))- if result.scalar_one_or_none() is None:- raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Case not found")-- event = CaseTimeline(- case_id=case_id,- tenant_id=current_user.tenant_id,- event_type=request.event_type,- content=request.content,- event_metadata=request.metadata,- user_id=current_user.user_id,- )- db.add(event)- await db.commit()- await db.refresh(event)- return TimelineEventResponse.model_validate(event)+class CommentResponse(BaseModel):+ id: uuid.UUID+ case_id: uuid.UUID+ author: str | None+ body: str+ is_system: bool+ created_at: datetime[email protected]("/{case_id}/timeline", response_model=list[TimelineEventResponse])-async def get_timeline(- case_id: uuid.UUID,- current_user: Annotated[AuthUser, Depends(require_permission("cases:read"))],- db: DBSession,-) -> list[TimelineEventResponse]:- """Get the full timeline for a case."""- result = await db.execute(- select(CaseTimeline)- .where(CaseTimeline.case_id == case_id, CaseTimeline.tenant_id == current_user.tenant_id)- .order_by(CaseTimeline.created_at.asc())+class EvidenceReport(BaseModel):+ case_id: uuid.UUID+ title: str+ severity: str+ status: str+ alert_count: int+ mitre_techniques: list[str]+ compliance_frameworks: list[str]+ evidence_chain: list[dict[str, Any]]+ generated_at: datetime+++# ────────────────────────────────────────────────────────────────────────────+# Helpers+# ────────────────────────────────────────────────────────────────────────────+++def _row_to_case(row: Any) -> CaseResponse:+ return CaseResponse(+ id=row.id,+ title=row.title,+ description=row.description,+ severity=row.severity,+ status=row.status,+ assignee=row.assignee,+ mitre_techniques=list(row.mitre_techniques or []),+ alert_ids=list(row.alert_ids or []),+ observable_graph=dict(row.observable_graph or {}),+ evidence_chain=list(row.evidence_chain or []),+ compliance_frameworks=list(row.compliance_frameworks or []),+ opened_at=row.opened_at,+ triaged_at=row.triaged_at,+ resolved_at=row.resolved_at,+ closed_at=row.closed_at,+ created_at=row.created_at,+ updated_at=row.updated_at,+ created_by=row.created_by,+ tags=dict(row.tags or {}),+ sla_due_at=row.sla_due_at,)- events = result.scalars().all()- return [TimelineEventResponse.model_validate(e) for e in events]---# ----------------------------------------------------------------------------# Pillar-1: AI Investigation-# ----------------------------------------------------------------------------class InvestigateRequest(BaseModel):- alert_summary: str = ""- raw_alert: dict[str, Any] = {}+# ────────────────────────────────────────────────────────────────────────────+# Endpoints+# ────────────────────────────────────────────────────────────────────────────-class InvestigateResponse(BaseModel):- run_id: str- case_id: str- status: str- message: str--[email protected]("/{case_id}/investigate", response_model=InvestigateResponse, status_code=status.HTTP_202_ACCEPTED)-async def investigate_case(- case_id: uuid.UUID,- body: InvestigateRequest,- current_user: Annotated[AuthUser, Depends(require_permission("cases:write"))],[email protected]("", response_model=list[CaseResponse], summary="List cases")+async def list_cases(db: DBSession,-) -> InvestigateResponse:+ user: AuthUser,+ status_filter: str | None = Query(None, alias="status"),+ severity: str | None = Query(None),+ assignee: str | None = Query(None),+ limit: int = Query(50, ge=1, le=500),+ offset: int = Query(0, ge=0),+) -> list[CaseResponse]:+ where_clauses = ["1=1"]+ params: dict[str, Any] = {"limit": limit, "offset": offset}+ if status_filter:+ where_clauses.append("status = :status")+ params["status"] = status_filter+ if severity:+ where_clauses.append("severity = :severity")+ params["severity"] = severity+ if assignee:+ where_clauses.append("assignee = :assignee")+ params["assignee"] = assignee++ q = f"""+ SELECT * FROM aisoc_cases+ WHERE {' AND '.join(where_clauses)}+ ORDER BY created_at DESC+ LIMIT :limit OFFSET :offset"""- Trigger an autonomous AI investigation for this case.- Proxies to the agents service Pillar-1 orchestrator.- """- # Verify case ownership- result = await db.execute(select(Case).where(Case.id == case_id, Case.tenant_id == current_user.tenant_id))- case = result.scalar_one_or_none()- if case is None:- raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Case not found")-- alert_summary = body.alert_summary or case.title or ""- raw_alert = body.raw_alert or {"case_id": str(case_id), "title": case.title}-try:- async with httpx.AsyncClient(timeout=10.0) as client:- resp = await client.post(- f"{_AGENTS_URL}/api/v1/cases/{case_id}/investigate",- json={- "alert_summary": alert_summary,- "raw_alert": raw_alert,- "tenant_id": str(current_user.tenant_id),- },- )- resp.raise_for_status()- data = resp.json()- except httpx.HTTPStatusError as exc:- logger.error("agents_service_http_error", status=exc.response.status_code)- raise HTTPException(status_code=502, detail="Agents service returned an error") from exc- except Exception as exc: # noqa: BLE001- logger.error("agents_service_unavailable", error=type(exc).__name__)- raise HTTPException(status_code=503, detail="Agents service unavailable") from exc-- # Record timeline entry- db.add(- CaseTimeline(- case_id=case_id,- tenant_id=current_user.tenant_id,- event_type="ai_investigation_started",- content=f"AI investigation started (run_id={data.get('run_id')})",- user_id=current_user.user_id,- is_automated=True,- event_metadata={"run_id": data.get("run_id")},- )+ rows = await db.execute(text(q).bindparams(**params))+ return [_row_to_case(r) for r in rows.fetchall()]+ except Exception as exc:+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc++[email protected]("", response_model=CaseResponse, status_code=status.HTTP_201_CREATED, summary="Create case")+async def create_case(body: CreateCaseRequest, db: DBSession, user: AuthUser) -> CaseResponse:+ import json as _json++ case_id = uuid.uuid4()+ now = datetime.now(UTC)+ q = text("""+ INSERT INTO aisoc_cases (+ id, title, description, severity, status, assignee,+ mitre_techniques, alert_ids, compliance_frameworks, tags,+ sla_due_at, opened_at, created_at, updated_at, created_by+ ) VALUES (+ :id, :title, :description, :severity, 'new', :assignee,+ :mitre::jsonb, :alert_ids::uuid[], :frameworks::text[], :tags::jsonb,+ :sla, :now, :now, :now, :user+ ) RETURNING *+ """).bindparams(+ id=case_id,+ title=body.title,+ description=body.description,+ severity=body.severity,+ assignee=body.assignee,+ mitre=_json.dumps(body.mitre_techniques),+ alert_ids=list(map(str, body.alert_ids)) or [],+ frameworks=body.compliance_frameworks or [],+ tags=_json.dumps(body.tags),+ sla=body.sla_due_at,+ now=now,+ user=str(user) if user else "system",)- await db.commit()-- return InvestigateResponse(**data)---def _validate_run_id(run_id: str) -> str:- """Raise 422 if run_id is not a UUID; return canonicalized UUID string to break taint flow."""try:- # Round-trip through uuid.UUID to produce a safe, canonical string- return str(uuid.UUID(run_id))- except (ValueError, AttributeError) as exc:- raise HTTPException(- status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,- detail="Invalid run_id format",- ) from exc--[email protected]("/{case_id}/investigations/{run_id}")-async def get_case_investigation(- case_id: uuid.UUID,- run_id: str,- current_user: Annotated[AuthUser, Depends(require_permission("cases:read"))],-) -> dict[str, Any]:- """Proxy: poll investigation run status from the agents service."""- safe_run_id = _validate_run_id(run_id)+ row = (await db.execute(q)).fetchone()+ await db.commit()+ return _row_to_case(row)+ except Exception as exc:+ await db.rollback()+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc++[email protected]("/{case_id}", response_model=CaseResponse, summary="Get case")+async def get_case(case_id: uuid.UUID, db: DBSession, user: AuthUser) -> CaseResponse:+ row = (await db.execute(text("SELECT * FROM aisoc_cases WHERE id = :id").bindparams(id=case_id))).fetchone()+ if not row:+ raise HTTPException(status_code=404, detail="Case not found.")+ return _row_to_case(row)++[email protected]("/{case_id}", response_model=CaseResponse, summary="Update case")+async def update_case(case_id: uuid.UUID, body: UpdateCaseRequest, db: DBSession, user: AuthUser) -> CaseResponse:+ import json as _json++ existing = (await db.execute(text("SELECT status FROM aisoc_cases WHERE id = :id").bindparams(id=case_id))).fetchone()+ if not existing:+ raise HTTPException(status_code=404, detail="Case not found.")++ if body.status and body.status != existing.status:+ allowed = _TRANSITIONS.get(existing.status, set())+ if body.status not in allowed:+ raise HTTPException(+ status_code=422,+ detail=f"Invalid status transition: {existing.status} → {body.status}. Allowed: {sorted(allowed) or 'none'}",+ )++ now = datetime.now(UTC)+ sets = ["updated_at = :now"]+ params: dict[str, Any] = {"id": case_id, "now": now}++ if body.title is not None:+ sets.append("title = :title"); params["title"] = body.title+ if body.description is not None:+ sets.append("description = :description"); params["description"] = body.description+ if body.severity is not None:+ sets.append("severity = :severity"); params["severity"] = body.severity+ if body.assignee is not None:+ sets.append("assignee = :assignee"); params["assignee"] = body.assignee+ if body.sla_due_at is not None:+ sets.append("sla_due_at = :sla"); params["sla"] = body.sla_due_at+ if body.mitre_techniques is not None:+ sets.append("mitre_techniques = :mitre::jsonb"); params["mitre"] = _json.dumps(body.mitre_techniques)+ if body.compliance_frameworks is not None:+ sets.append("compliance_frameworks = :frameworks::text[]"); params["frameworks"] = body.compliance_frameworks+ if body.tags is not None:+ sets.append("tags = :tags::jsonb"); params["tags"] = _json.dumps(body.tags)++ if body.status:+ sets.append("status = :status"); params["status"] = body.status+ ts_col = {"triaged": "triaged_at", "resolved": "resolved_at", "closed": "closed_at"}.get(body.status)+ if ts_col:+ sets.append(f"{ts_col} = :now")++ q = text(f"UPDATE aisoc_cases SET {', '.join(sets)} WHERE id = :id RETURNING *").bindparams(**params)try:- async with httpx.AsyncClient(timeout=10.0) as client:- resp = await client.get(f"{_AGENTS_URL}/api/v1/investigations/{safe_run_id}")- resp.raise_for_status()- return resp.json()- except httpx.HTTPStatusError as exc:- raise HTTPException(status_code=exc.response.status_code, detail="Upstream service error") from exc- except Exception as exc: # noqa: BLE001- raise HTTPException(status_code=503, detail="Agents service unavailable") from exc--[email protected]("/{case_id}/investigations/{run_id}/report.md")-async def get_case_report_md(- case_id: uuid.UUID,- run_id: str,- current_user: Annotated[AuthUser, Depends(require_permission("cases:read"))],-) -> Any:- """Proxy: download Markdown incident report from the agents service."""- from fastapi.responses import PlainTextResponse-- safe_run_id = _validate_run_id(run_id)+ row = (await db.execute(q)).fetchone()+ await db.commit()+ return _row_to_case(row)+ except Exception as exc:+ await db.rollback()+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc++[email protected]("/{case_id}/alerts", response_model=CaseResponse, summary="Link alerts to a case")+async def add_alerts(case_id: uuid.UUID, body: AddAlertsRequest, db: DBSession, user: AuthUser) -> CaseResponse:+ ids_str = [str(a) for a in body.alert_ids]+ q = text("""+ UPDATE aisoc_cases+ SET alert_ids = array(SELECT DISTINCT unnest(alert_ids || :new_ids::uuid[])),+ updated_at = now()+ WHERE id = :id+ RETURNING *+ """).bindparams(id=case_id, new_ids=ids_str)try:- async with httpx.AsyncClient(timeout=30.0) as client:- resp = await client.get(f"{_AGENTS_URL}/api/v1/investigations/{safe_run_id}/report.md")- resp.raise_for_status()- return PlainTextResponse(content=resp.text)- except httpx.HTTPStatusError as exc:- raise HTTPException(status_code=exc.response.status_code, detail="Upstream service error") from exc- except Exception as exc: # noqa: BLE001- raise HTTPException(status_code=503, detail="Agents service unavailable") from exc--[email protected]("/{case_id}/investigations/{run_id}/report.html")-async def get_case_report_html(- case_id: uuid.UUID,- run_id: str,- current_user: Annotated[AuthUser, Depends(require_permission("cases:read"))],-) -> Any:- """Proxy: download HTML incident report from the agents service."""- from fastapi.responses import HTMLResponse-- safe_run_id = _validate_run_id(run_id)+ row = (await db.execute(q)).fetchone()+ if not row:+ raise HTTPException(status_code=404, detail="Case not found.")+ await db.commit()+ return _row_to_case(row)+ except HTTPException:+ raise+ except Exception as exc:+ await db.rollback()+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc++[email protected]("/{case_id}/observables", response_model=CaseResponse, summary="Update observable graph")+async def update_observables(case_id: uuid.UUID, body: UpdateObservablesRequest, db: DBSession, user: AuthUser) -> CaseResponse:+ import json as _json++ existing = (await db.execute(text("SELECT observable_graph FROM aisoc_cases WHERE id = :id").bindparams(id=case_id))).fetchone()+ if not existing:+ raise HTTPException(status_code=404, detail="Case not found.")++ graph: dict[str, Any] = dict(existing.observable_graph or {})+ nodes = graph.get("nodes", [])+ edges = graph.get("edges", [])++ existing_ids = {n["id"] for n in nodes}+ for n in body.nodes:+ if n.id not in existing_ids:+ nodes.append(n.model_dump())+ else:+ nodes = [n.model_dump() if x["id"] == n.id else x for x in nodes]++ edge_keys = {(e["source"], e["target"]) for e in edges}+ for e in body.edges:+ if (e.source, e.target) not in edge_keys:+ edges.append(e.model_dump())++ graph = {"nodes": nodes, "edges": edges}+ q = text("""+ UPDATE aisoc_cases SET observable_graph = :g::jsonb, updated_at = now() WHERE id = :id RETURNING *+ """).bindparams(id=case_id, g=_json.dumps(graph))try:- async with httpx.AsyncClient(timeout=30.0) as client:- resp = await client.get(f"{_AGENTS_URL}/api/v1/investigations/{safe_run_id}/report.html")- resp.raise_for_status()- return HTMLResponse(content=resp.text)- except httpx.HTTPStatusError as exc:- raise HTTPException(status_code=exc.response.status_code, detail="Upstream service error") from exc- except Exception as exc: # noqa: BLE001- raise HTTPException(status_code=503, detail="Agents service unavailable") from exc--[email protected]("/{case_id}/investigations/{run_id}/report.pdf")-async def get_case_report_pdf(- case_id: uuid.UUID,- run_id: str,- current_user: Annotated[AuthUser, Depends(require_permission("cases:read"))],-) -> Any:- """Proxy: download PDF incident report (rendered by weasyprint in agents service)."""- from fastapi.responses import Response as FastAPIResponse-- safe_run_id = _validate_run_id(run_id)+ row = (await db.execute(q)).fetchone()+ await db.commit()+ return _row_to_case(row)+ except Exception as exc:+ await db.rollback()+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc++[email protected]("/{case_id}/comments", response_model=CommentResponse, status_code=201, summary="Add comment")+async def add_comment(case_id: uuid.UUID, body: AddCommentRequest, db: DBSession, user: AuthUser) -> CommentResponse:+ exists = (await db.execute(text("SELECT 1 FROM aisoc_cases WHERE id = :id").bindparams(id=case_id))).fetchone()+ if not exists:+ raise HTTPException(status_code=404, detail="Case not found.")++ comment_id = uuid.uuid4()+ now = datetime.now(UTC)+ q = text("""+ INSERT INTO aisoc_case_comments (id, case_id, author, body, is_system, created_at)+ VALUES (:id, :case_id, :author, :body, :sys, :now) RETURNING *+ """).bindparams(id=comment_id, case_id=case_id, author=str(user) if user else "system", body=body.body, sys=body.is_system, now=now)try:- async with httpx.AsyncClient(timeout=60.0) as client:- resp = await client.get(f"{_AGENTS_URL}/api/v1/investigations/{safe_run_id}/report.pdf")- resp.raise_for_status()- return FastAPIResponse(- content=resp.content,- media_type="application/pdf",- headers={"Content-Disposition": f'attachment; filename="aisoc-report-{safe_run_id}.pdf"'},- )- except httpx.HTTPStatusError as exc:- raise HTTPException(status_code=exc.response.status_code, detail="Upstream service error") from exc- except Exception as exc: # noqa: BLE001- raise HTTPException(status_code=503, detail="Agents service unavailable") from exc+ row = (await db.execute(q)).fetchone()+ await db.commit()+ return CommentResponse(+ id=row.id, case_id=row.case_id, author=row.author,+ body=row.body, is_system=row.is_system, created_at=row.created_at,+ )+ except Exception as exc:+ await db.rollback()+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc++[email protected]("/{case_id}/comments", response_model=list[CommentResponse], summary="List case comments")+async def list_comments(case_id: uuid.UUID, db: DBSession, user: AuthUser) -> list[CommentResponse]:+ rows = (await db.execute(text("SELECT * FROM aisoc_case_comments WHERE case_id = :id ORDER BY created_at").bindparams(id=case_id))).fetchall()+ return [CommentResponse(id=r.id, case_id=r.case_id, author=r.author, body=r.body, is_system=r.is_system, created_at=r.created_at) for r in rows]++[email protected]("/{case_id}/evidence", response_model=EvidenceReport, summary="Export evidence chain report")+async def evidence_report(case_id: uuid.UUID, db: DBSession, user: AuthUser) -> EvidenceReport:+ row = (await db.execute(text("SELECT * FROM aisoc_cases WHERE id = :id").bindparams(id=case_id))).fetchone()+ if not row:+ raise HTTPException(status_code=404, detail="Case not found.")+ return EvidenceReport(+ case_id=row.id,+ title=row.title,+ severity=row.severity,+ status=row.status,+ alert_count=len(row.alert_ids or []),+ mitre_techniques=list(row.mitre_techniques or []),+ compliance_frameworks=list(row.compliance_frameworks or []),+ evidence_chain=list(row.evidence_chain or []),+ generated_at=datetime.now(UTC),+ )diff --git a/services/api/app/api/v1/endpoints/compliance.py b/services/api/app/api/v1/endpoints/compliance.pyindex 53400a72d..829899d0e 100644--- a/services/api/app/api/v1/endpoints/compliance.py+++ b/services/api/app/api/v1/endpoints/compliance.py@@ -1,444 +1,341 @@-"""Compliance API endpoints.--GET /api/v1/compliance/soc2 — SOC 2 summary + control list with evidence-POST /api/v1/compliance/soc2/collect — Trigger auto-evidence collection-GET /api/v1/compliance/soc2/export — PDF-ready JSON export-GET /api/v1/compliance/{framework} — Generic framework dashboard-GET /api/v1/compliance/{framework}/heatmap — Heatmap data (category × status)-POST /api/v1/compliance/{framework}/collect — Auto-collect evidence for any framework-GET /api/v1/compliance/{framework}/export — PDF-ready export for any framework-GET /api/v1/compliance/frameworks — List all available frameworks+"""Compliance evidence trails (tier2-compliance).++Collects, hashes, and reviews audit-grade evidence records for regulatory+frameworks (SOC 2, PCI-DSS, HIPAA, ISO 27001, NIST CSF, …). Evidence items+form a tamper-evident hash chain — each record's ``payload_hash`` is a+SHA-256 digest of ``prev_hash || summary || raw_payload``.++Endpoints+---------+* ``GET /compliance/frameworks`` List known frameworks & controls.+* ``POST /compliance/evidence`` Collect a new evidence item.+* ``GET /compliance/evidence`` List evidence (filter by framework/control/case).+* ``GET /compliance/evidence/{id}`` Get single evidence item.+* ``POST /compliance/evidence/{id}/review`` Accept or reject an evidence item.+* ``GET /compliance/report`` Generate a compliance posture report."""from __future__ import annotations+import hashlib+import jsonimport uuid-from datetime import datetime-from typing import Annotated+from datetime import UTC, datetime+from typing import Any, Literal-from fastapi import APIRouter, Depends, HTTPException-from pydantic import BaseModel-from sqlalchemy import func, select+from fastapi import APIRouter, HTTPException, Query, status+from pydantic import BaseModel, Field+from sqlalchemy import text-from app.api.v1.deps import AuthUser, require_permission-from app.db.rls import TenantDBSession-from app.models.compliance import ComplianceControl, ComplianceEvidence-from app.services.compliance import auto_collect_evidence, get_soc2_summary+from app.api.v1.deps import AuthUser, DBSessionrouter = APIRouter(prefix="/compliance", tags=["compliance"])-# ----------------------------------------------------------------------------# Supported frameworks-# -----------------------------------------------------------------------------SUPPORTED_FRAMEWORKS: dict[str, str] = {- "soc2": "SOC 2 Type II",- "iso27001": "ISO/IEC 27001:2022",- "nist_csf": "NIST Cybersecurity Framework 2.0",- "pci_dss": "PCI DSS v4.0",- "hipaa": "HIPAA Security Rule",- "dora": "DORA (Digital Operational Resilience Act)",+# ────────────────────────────────────────────────────────────────────────────+# Known frameworks + sampled controls (extendable via DB or config)+# ────────────────────────────────────────────────────────────────────────────++FRAMEWORKS: dict[str, dict[str, str]] = {+ "SOC2": {+ "CC1.1": "Common Criteria — Commitment to Competence",+ "CC6.1": "Logical and Physical Access Controls",+ "CC6.2": "Access Provisioning",+ "CC6.3": "Access Removal",+ "CC7.2": "System Monitoring",+ "CC8.1": "Change Management",+ "A1.1": "Availability — Capacity Management",+ },+ "PCI-DSS": {+ "Req-1": "Install and maintain network security controls",+ "Req-6": "Develop and maintain secure systems",+ "Req-8": "Identify users and authenticate access",+ "Req-10": "Log and monitor all access to system components",+ "Req-11": "Test security of systems and networks regularly",+ },+ "HIPAA": {+ "164.308(a)(1)": "Security Management Process",+ "164.308(a)(5)": "Security Awareness and Training",+ "164.312(b)": "Audit Controls",+ "164.312(d)": "Person or Entity Authentication",+ },+ "ISO27001": {+ "A.12.4.1": "Event Logging",+ "A.12.4.2": "Protection of Log Information",+ "A.16.1.2": "Reporting Information Security Events",+ "A.9.2.1": "User Registration and De-Registration",+ },+ "NIST-CSF": {+ "DE.AE-1": "Baseline of network operations established",+ "DE.CM-1": "Network monitored for potential events",+ "RS.AN-1": "Investigations are performed",+ "RS.MI-2": "Incidents are mitigated",+ },}-# ---------------------------------------------------------------------------+# ────────────────────────────────────────────────────────────────────────────# Pydantic schemas-# ---------------------------------------------------------------------------+# ────────────────────────────────────────────────────────────────────────────--class ControlOut(BaseModel):- id: str- framework: str- control_id: str- category: str- title: str- description: str | None---class EvidenceOut(BaseModel):- id: str- control_id: str- evidence_type: str- title: str- description: str | None- status: str- collected_at: str+EvidenceKind = Literal["alert", "log", "screenshot", "attestation", "policy", "runbook", "other"]+EvidenceStatus = Literal["pending", "accepted", "rejected"]-class ControlWithEvidence(BaseModel):- control: ControlOut- evidence: list[EvidenceOut]- latest_status: str+class CollectEvidenceRequest(BaseModel):+ framework: str = Field(..., description="Compliance framework key e.g. 'SOC2'.")+ control_id: str = Field(..., description="Control identifier e.g. 'CC7.2'.")+ control_title: str | None = None+ evidence_kind: EvidenceKind = "alert"+ summary: str = Field(..., min_length=5)+ raw_payload: dict[str, Any] = Field(default_factory=dict)+ case_id: uuid.UUID | None = None-class FrameworkSummary(BaseModel):- total: int- collected: int- review: int- approved: int- rejected: int- missing: int- pct: int+class ReviewEvidenceRequest(BaseModel):+ decision: Literal["accepted", "rejected"]+ reviewer: str | None = None-class FrameworkResponse(BaseModel):+class EvidenceResponse(BaseModel):+ id: uuid.UUID+ case_id: uuid.UUID | Noneframework: str- framework_name: str- summary: FrameworkSummary- controls: list[ControlWithEvidence]---class SOC2Summary(BaseModel):- total: int- collected: int- review: int- approved: int- rejected: int- pct: int-+ control_id: str+ control_title: str | None+ evidence_kind: str+ summary: str+ raw_payload: dict[str, Any]+ payload_hash: str | None+ prev_hash: str | None+ collected_at: datetime+ reviewed_by: str | None+ reviewed_at: datetime | None+ status: str+ created_at: datetime-class SOC2Response(BaseModel):- summary: SOC2Summary- controls: list[ControlWithEvidence]+class FrameworksResponse(BaseModel):+ frameworks: dict[str, dict[str, str]]-class HeatmapCell(BaseModel):- category: str- status: str- count: int--class HeatmapResponse(BaseModel):+class CompliancePosture(BaseModel):framework: str- framework_name: str- categories: list[str]- statuses: list[str]- cells: list[HeatmapCell]-+ total_evidence: int+ accepted: int+ pending: int+ rejected: int+ coverage_pct: float # accepted / known controls * 100+ controls_covered: list[str]+ controls_missing: list[str]+ generated_at: datetime-class FrameworkInfo(BaseModel):- id: str- name: str- control_count: int+# ────────────────────────────────────────────────────────────────────────────+# Helpers+# ────────────────────────────────────────────────────────────────────────────-# ----------------------------------------------------------------------------# Helper-# ---------------------------------------------------------------------------+def _compute_hash(prev_hash: str | None, summary: str, payload: dict[str, Any]) -> str:+ raw = (prev_hash or "") + summary + json.dumps(payload, sort_keys=True)+ return hashlib.sha256(raw.encode()).hexdigest()-async def _get_framework_data(- db: TenantDBSession,- tenant_id: uuid.UUID,- framework: str,-) -> tuple[list[ComplianceControl], dict[uuid.UUID, list[ComplianceEvidence]]]:- """Fetch controls and grouped evidence for a framework."""- ctrl_result = await db.execute(- select(ComplianceControl).where(ComplianceControl.framework == framework).order_by(ComplianceControl.control_id)- )- controls = ctrl_result.scalars().all()- ev_result = await db.execute(- select(ComplianceEvidence)- .where(- ComplianceEvidence.tenant_id == tenant_id,+async def _latest_hash(db: DBSession, framework: str) -> str | None:+ row = (+ await db.execute(+ text("SELECT payload_hash FROM aisoc_compliance_evidence WHERE framework = :f ORDER BY created_at DESC LIMIT 1").bindparams(f=framework))- .join(ComplianceControl, ComplianceEvidence.control_id == ComplianceControl.id)- .where(ComplianceControl.framework == framework)- .order_by(ComplianceEvidence.collected_at.desc())+ ).fetchone()+ return row.payload_hash if row else None+++def _row_to_evidence(row: Any) -> EvidenceResponse:+ return EvidenceResponse(+ id=row.id,+ case_id=row.case_id,+ framework=row.framework,+ control_id=row.control_id,+ control_title=row.control_title,+ evidence_kind=row.evidence_kind,+ summary=row.summary,+ raw_payload=dict(row.raw_payload or {}),+ payload_hash=row.payload_hash,+ prev_hash=row.prev_hash,+ collected_at=row.collected_at,+ reviewed_by=row.reviewed_by,+ reviewed_at=row.reviewed_at,+ status=row.status,+ created_at=row.created_at,)- all_evidence = ev_result.scalars().all()-- ev_by_control: dict[uuid.UUID, list[ComplianceEvidence]] = {}- for ev in all_evidence:- ev_by_control.setdefault(ev.control_id, []).append(ev)-- return controls, ev_by_control---def _build_controls_with_evidence(- controls: list[ComplianceControl],- ev_by_control: dict[uuid.UUID, list[ComplianceEvidence]],- max_evidence: int = 5,-) -> list[ControlWithEvidence]:- result = []- for ctrl in controls:- evs = ev_by_control.get(ctrl.id, [])- latest_status = evs[0].status if evs else "missing"- result.append(- ControlWithEvidence(- control=ControlOut(- id=str(ctrl.id),- framework=ctrl.framework,- control_id=ctrl.control_id,- category=ctrl.category,- title=ctrl.title,- description=ctrl.description,- ),- evidence=[- EvidenceOut(- id=str(e.id),- control_id=str(e.control_id),- evidence_type=e.evidence_type,- title=e.title,- description=e.description,- status=e.status,- collected_at=e.collected_at.isoformat(),- )- for e in evs[:max_evidence]- ],- latest_status=latest_status,- )- )- return result---def _compute_summary(controls_with_ev: list[ControlWithEvidence]) -> dict:- total = len(controls_with_ev)- status_counts: dict[str, int] = {}- for cwev in controls_with_ev:- s = cwev.latest_status- status_counts[s] = status_counts.get(s, 0) + 1- collected = sum(status_counts.get(s, 0) for s in ("collected", "approved"))- return {- "total": total,- "collected": collected,- "review": status_counts.get("review", 0),- "approved": status_counts.get("approved", 0),- "rejected": status_counts.get("rejected", 0),- "missing": status_counts.get("missing", 0),- "pct": int(collected / total * 100) if total else 0,- }---# ----------------------------------------------------------------------------# Utility: list all frameworks-# -----------------------------------------------------------------------------[email protected]("/frameworks", response_model=list[FrameworkInfo])-async def list_frameworks(- current_user: Annotated[AuthUser, Depends(require_permission("compliance:read"))],- db: TenantDBSession,-) -> list[FrameworkInfo]:- """Return all supported compliance frameworks with control counts."""- result = await db.execute(- select(ComplianceControl.framework, func.count(ComplianceControl.id).label("cnt")).group_by(ComplianceControl.framework)- )- counts = {row.framework: row.cnt for row in result}-- return [- FrameworkInfo(- id=fw_id,- name=fw_name,- control_count=counts.get(fw_id, 0),- )- for fw_id, fw_name in SUPPORTED_FRAMEWORKS.items()- if counts.get(fw_id, 0) > 0 or fw_id in counts- ]-# ----------------------------------------------------------------------------# SOC 2 dedicated routes (kept for backward-compat)-# -----------------------------------------------------------------------------[email protected]("/soc2", response_model=SOC2Response)-async def get_soc2_dashboard(- current_user: Annotated[AuthUser, Depends(require_permission("compliance:read"))],- db: TenantDBSession,-) -> SOC2Response:- """Return SOC 2 evidence dashboard for the current tenant."""- controls, ev_by_control = await _get_framework_data(db, current_user.tenant_id, "soc2")- controls_with_evidence = _build_controls_with_evidence(controls, ev_by_control)- summary = await get_soc2_summary(db, current_user.tenant_id)-- return SOC2Response(- summary=SOC2Summary(**summary),- controls=controls_with_evidence,+# ────────────────────────────────────────────────────────────────────────────+# Endpoints+# ────────────────────────────────────────────────────────────────────────────++[email protected]("/frameworks", response_model=FrameworksResponse, summary="List frameworks and controls")+async def list_frameworks() -> FrameworksResponse:+ return FrameworksResponse(frameworks=FRAMEWORKS)++[email protected]("/evidence", response_model=EvidenceResponse, status_code=status.HTTP_201_CREATED, summary="Collect evidence item")+async def collect_evidence(body: CollectEvidenceRequest, db: DBSession, user: AuthUser) -> EvidenceResponse:+ prev_hash = await _latest_hash(db, body.framework)+ new_hash = _compute_hash(prev_hash, body.summary, body.raw_payload)+ now = datetime.now(UTC)+ evidence_id = uuid.uuid4()++ q = text("""+ INSERT INTO aisoc_compliance_evidence (+ id, case_id, framework, control_id, control_title,+ evidence_kind, summary, raw_payload, payload_hash, prev_hash,+ collected_at, status, created_at+ ) VALUES (+ :id, :case_id, :fw, :ctrl, :title,+ :kind, :summary, :payload::jsonb, :hash, :prev,+ :now, 'pending', :now+ ) RETURNING *+ """).bindparams(+ id=evidence_id,+ case_id=body.case_id,+ fw=body.framework,+ ctrl=body.control_id,+ title=body.control_title or FRAMEWORKS.get(body.framework, {}).get(body.control_id),+ kind=body.evidence_kind,+ summary=body.summary,+ payload=json.dumps(body.raw_payload),+ hash=new_hash,+ prev=prev_hash,+ now=now,)--[email protected]("/soc2/collect", status_code=202)-async def collect_soc2_evidence(- current_user: Annotated[AuthUser, Depends(require_permission("compliance:write"))],- db: TenantDBSession,-) -> dict:- """Trigger automatic evidence collection for all SOC 2 controls."""- collected = await auto_collect_evidence(db, current_user.tenant_id, framework="soc2")- await db.commit()-- return {- "message": f"Collected {len(collected)} evidence items",- "count": len(collected),- "collected_at": datetime.utcnow().isoformat(),- }--[email protected]("/soc2/export")-async def export_soc2_evidence(- current_user: Annotated[AuthUser, Depends(require_permission("compliance:read"))],- db: TenantDBSession,-) -> dict:- """Export SOC 2 evidence as structured JSON for PDF generation."""- controls, ev_by_control = await _get_framework_data(db, current_user.tenant_id, "soc2")- controls_with_evidence = _build_controls_with_evidence(controls, ev_by_control, max_evidence=3)- summary = await get_soc2_summary(db, current_user.tenant_id)-- return {- "report_type": "SOC 2 Type I Evidence Report",- "generated_at": datetime.utcnow().isoformat(),- "tenant_id": str(current_user.tenant_id),- "summary": summary,- "controls": [- {- "control_id": cwev.control.control_id,- "category": cwev.control.category,- "title": cwev.control.title,- "description": cwev.control.description,- "evidence": [- {- "type": e.evidence_type,- "title": e.title,- "description": e.description,- "status": e.status,- "collected_at": e.collected_at,- }- for e in cwev.evidence- ],- "latest_status": cwev.latest_status,- }- for cwev in controls_with_evidence- ],- }---# ----------------------------------------------------------------------------# Generic framework routes-# ------------------------------------------------------------------------------def _validate_framework(framework: str) -> None:- if framework not in SUPPORTED_FRAMEWORKS:- raise HTTPException(- status_code=404,- detail=f"Framework '{framework}' not found. Supported: {list(SUPPORTED_FRAMEWORKS.keys())}",+ try:+ row = (await db.execute(q)).fetchone()+ await db.commit()+ return _row_to_evidence(row)+ except Exception as exc:+ await db.rollback()+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc++[email protected]("/evidence", response_model=list[EvidenceResponse], summary="List evidence items")+async def list_evidence(+ db: DBSession,+ user: AuthUser,+ framework: str | None = Query(None),+ control_id: str | None = Query(None),+ case_id: uuid.UUID | None = Query(None),+ ev_status: str | None = Query(None, alias="status"),+ limit: int = Query(100, ge=1, le=1000),+ offset: int = Query(0, ge=0),+) -> list[EvidenceResponse]:+ wheres = ["1=1"]+ params: dict[str, Any] = {"limit": limit, "offset": offset}+ if framework:+ wheres.append("framework = :fw"); params["fw"] = framework+ if control_id:+ wheres.append("control_id = :ctrl"); params["ctrl"] = control_id+ if case_id:+ wheres.append("case_id = :case_id"); params["case_id"] = case_id+ if ev_status:+ wheres.append("status = :ev_status"); params["ev_status"] = ev_status++ q = text(f"SELECT * FROM aisoc_compliance_evidence WHERE {' AND '.join(wheres)} ORDER BY collected_at DESC LIMIT :limit OFFSET :offset").bindparams(**params)+ try:+ rows = (await db.execute(q)).fetchall()+ return [_row_to_evidence(r) for r in rows]+ except Exception as exc:+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc++[email protected]("/evidence/{evidence_id}", response_model=EvidenceResponse, summary="Get evidence item")+async def get_evidence(evidence_id: uuid.UUID, db: DBSession, user: AuthUser) -> EvidenceResponse:+ row = (await db.execute(text("SELECT * FROM aisoc_compliance_evidence WHERE id = :id").bindparams(id=evidence_id))).fetchone()+ if not row:+ raise HTTPException(status_code=404, detail="Evidence item not found.")+ return _row_to_evidence(row)++[email protected]("/evidence/{evidence_id}/review", response_model=EvidenceResponse, summary="Accept or reject evidence")+async def review_evidence(evidence_id: uuid.UUID, body: ReviewEvidenceRequest, db: DBSession, user: AuthUser) -> EvidenceResponse:+ now = datetime.now(UTC)+ q = text("""+ UPDATE aisoc_compliance_evidence+ SET status = :decision, reviewed_by = :reviewer, reviewed_at = :now+ WHERE id = :id RETURNING *+ """).bindparams(id=evidence_id, decision=body.decision, reviewer=body.reviewer or str(user), now=now)+ try:+ row = (await db.execute(q)).fetchone()+ if not row:+ raise HTTPException(status_code=404, detail="Evidence item not found.")+ await db.commit()+ return _row_to_evidence(row)+ except HTTPException:+ raise+ except Exception as exc:+ await db.rollback()+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc++[email protected]("/report", response_model=list[CompliancePosture], summary="Generate compliance posture report")+async def compliance_report(+ db: DBSession,+ user: AuthUser,+ framework: str | None = Query(None, description="Filter to a single framework."),+) -> list[CompliancePosture]:+ wheres = ["1=1"]+ params: dict[str, Any] = {}+ if framework:+ wheres.append("framework = :fw"); params["fw"] = framework++ q = text(f"""+ SELECT framework, control_id,+ COUNT(*) AS total,+ COUNT(*) FILTER (WHERE status = 'accepted') AS accepted,+ COUNT(*) FILTER (WHERE status = 'pending') AS pending,+ COUNT(*) FILTER (WHERE status = 'rejected') AS rejected+ FROM aisoc_compliance_evidence+ WHERE {' AND '.join(wheres)}+ GROUP BY framework, control_id+ ORDER BY framework, control_id+ """).bindparams(**params)++ try:+ rows = (await db.execute(q)).fetchall()+ except Exception as exc:+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc++ # Aggregate by framework+ by_fw: dict[str, Any] = {}+ for r in rows:+ fw = r.framework+ if fw not in by_fw:+ by_fw[fw] = {"total": 0, "accepted": 0, "pending": 0, "rejected": 0, "covered": set()}+ by_fw[fw]["total"] += r.total+ by_fw[fw]["accepted"] += r.accepted+ by_fw[fw]["pending"] += r.pending+ by_fw[fw]["rejected"] += r.rejected+ if r.accepted > 0:+ by_fw[fw]["covered"].add(r.control_id)++ report = []+ target_fws = [framework] if framework else list(by_fw.keys()) + [f for f in FRAMEWORKS if f not in by_fw]+ for fw in dict.fromkeys(target_fws):+ known = FRAMEWORKS.get(fw, {})+ data = by_fw.get(fw, {"total": 0, "accepted": 0, "pending": 0, "rejected": 0, "covered": set()})+ covered = data["covered"]+ missing = [c for c in known if c not in covered]+ pct = (len(covered) / len(known) * 100) if known else 0.0+ report.append(+ CompliancePosture(+ framework=fw,+ total_evidence=data["total"],+ accepted=data["accepted"],+ pending=data["pending"],+ rejected=data["rejected"],+ coverage_pct=round(pct, 1),+ controls_covered=sorted(covered),+ controls_missing=sorted(missing),+ generated_at=datetime.now(UTC),+ ))--[email protected]("/{framework}", response_model=FrameworkResponse)-async def get_framework_dashboard(- framework: str,- current_user: Annotated[AuthUser, Depends(require_permission("compliance:read"))],- db: TenantDBSession,-) -> FrameworkResponse:- """Return compliance dashboard for the given framework."""- _validate_framework(framework)-- controls, ev_by_control = await _get_framework_data(db, current_user.tenant_id, framework)- controls_with_evidence = _build_controls_with_evidence(controls, ev_by_control)- summary_dict = _compute_summary(controls_with_evidence)-- return FrameworkResponse(- framework=framework,- framework_name=SUPPORTED_FRAMEWORKS[framework],- summary=FrameworkSummary(**summary_dict),- controls=controls_with_evidence,- )--[email protected]("/{framework}/heatmap", response_model=HeatmapResponse)-async def get_framework_heatmap(- framework: str,- current_user: Annotated[AuthUser, Depends(require_permission("compliance:read"))],- db: TenantDBSession,-) -> HeatmapResponse:- """Return heatmap data (category × status counts) for the given framework."""- _validate_framework(framework)-- controls, ev_by_control = await _get_framework_data(db, current_user.tenant_id, framework)- controls_with_evidence = _build_controls_with_evidence(controls, ev_by_control)-- # Build category × status matrix- cell_map: dict[tuple[str, str], int] = {}- categories_ordered: list[str] = []- for cwev in controls_with_evidence:- cat = cwev.control.category- st = cwev.latest_status- cell_map[(cat, st)] = cell_map.get((cat, st), 0) + 1- if cat not in categories_ordered:- categories_ordered.append(cat)-- statuses = ["approved", "collected", "review", "rejected", "missing"]-- cells = [HeatmapCell(category=cat, status=st, count=cell_map.get((cat, st), 0)) for cat in categories_ordered for st in statuses]-- return HeatmapResponse(- framework=framework,- framework_name=SUPPORTED_FRAMEWORKS[framework],- categories=categories_ordered,- statuses=statuses,- cells=cells,- )--[email protected]("/{framework}/collect", status_code=202)-async def collect_framework_evidence(- framework: str,- current_user: Annotated[AuthUser, Depends(require_permission("compliance:write"))],- db: TenantDBSession,-) -> dict:- """Trigger automatic evidence collection for the given framework."""- _validate_framework(framework)-- collected = await auto_collect_evidence(db, current_user.tenant_id, framework=framework)- await db.commit()-- return {- "framework": framework,- "message": f"Collected {len(collected)} evidence items",- "count": len(collected),- "collected_at": datetime.utcnow().isoformat(),- }--[email protected]("/{framework}/export")-async def export_framework_evidence(- framework: str,- current_user: Annotated[AuthUser, Depends(require_permission("compliance:read"))],- db: TenantDBSession,-) -> dict:- """Export compliance evidence for the given framework as structured JSON."""- _validate_framework(framework)-- controls, ev_by_control = await _get_framework_data(db, current_user.tenant_id, framework)- controls_with_evidence = _build_controls_with_evidence(controls, ev_by_control, max_evidence=3)- summary_dict = _compute_summary(controls_with_evidence)-- return {- "report_type": f"{SUPPORTED_FRAMEWORKS[framework]} Evidence Report",- "framework": framework,- "framework_name": SUPPORTED_FRAMEWORKS[framework],- "generated_at": datetime.utcnow().isoformat(),- "tenant_id": str(current_user.tenant_id),- "summary": summary_dict,- "controls": [- {- "control_id": cwev.control.control_id,- "category": cwev.control.category,- "title": cwev.control.title,- "description": cwev.control.description,- "evidence": [- {- "type": e.evidence_type,- "title": e.title,- "description": e.description,- "status": e.status,- "collected_at": e.collected_at,- }- for e in cwev.evidence- ],- "latest_status": cwev.latest_status,- }- for cwev in controls_with_evidence- ],- }+ return reportdiff --git a/services/api/app/api/v1/endpoints/detection_loop.py b/services/api/app/api/v1/endpoints/detection_loop.pynew file mode 100644index 000000000..022128816--- /dev/null+++ b/services/api/app/api/v1/endpoints/detection_loop.py@@ -0,0 +1,290 @@+"""Closed-loop detection engineering (tier2-detection-loop).++When an analyst marks an alert as a false-positive (FP), this module:++1. Retrieves the triggering detection rule and the alert's raw evidence.+2. Invokes an LLM to draft a Sigma YAML improvement that would suppress+ the FP without widening the exclusion too broadly.+3. Creates a ``DetectionRuleProposal`` via the existing DAC lifecycle so the+ suggestion is eval-gated before promotion.+4. Returns the draft proposal ID and the diff for the analyst to review.++Endpoints+---------+* ``POST /detection-loop/suggest`` Trigger FP → Sigma draft.+* ``GET /detection-loop/suggestions`` List LLM-drafted suggestions.+* ``GET /detection-loop/suggestions/{id}`` Detail of one suggestion.+"""++from __future__ import annotations++import json+import textwrap+import uuid+from datetime import UTC, datetime+from typing import Any++import httpx+from fastapi import APIRouter, Depends, HTTPException, status+from pydantic import BaseModel, Field+from sqlalchemy import select, text++from app.api.v1.deps import AuthUser, DBSession, require_permission+from app.core.config import settings++router = APIRouter(prefix="/detection-loop", tags=["detection_rules", "detection_loop"])++# ────────────────────────────────────────────────────────────────────────────+# Pydantic schemas+# ────────────────────────────────────────────────────────────────────────────+++class SuggestRequest(BaseModel):+ alert_id: uuid.UUID = Field(..., description="ID of the FP-flagged alert.")+ analyst_note: str | None = Field(+ None, description="Free-text note explaining why this is a FP."+ )+++class SuggestionResponse(BaseModel):+ suggestion_id: uuid.UUID+ alert_id: uuid.UUID+ base_rule_id: uuid.UUID | None+ draft_rule_name: str+ draft_sigma_yaml: str+ rationale: str+ proposal_id: uuid.UUID | None+ created_at: datetime+++class SuggestionListResponse(BaseModel):+ suggestions: list[SuggestionResponse]+ total: int+++# ────────────────────────────────────────────────────────────────────────────+# LLM helper+# ────────────────────────────────────────────────────────────────────────────++_SYS_PROMPT = textwrap.dedent(+ """+ You are a senior detection engineer reviewing a false-positive alert.+ The user will provide:+ - The Sigma rule that triggered the alert (YAML).+ - Key fields from the alert that fired.+ - An analyst note explaining why this is benign.++ Your task is to produce a *minimal, targeted* Sigma rule modification that+ suppresses this class of false-positive without unduly widening the exclusion.+ Prefer:+ - Adding a `filter` condition rather than removing detection logic.+ - Scoping exclusions to specific processes, users, or source paths when+ the evidence supports it.+ - Keeping the rule's ATT&CK technique tags unchanged.++ Respond in JSON only with this schema:+ {+ "rule_name": "...", // may append '-v2' or '-fp-fix'+ "sigma_yaml": "...", // full updated Sigma YAML+ "rationale": "..." // ≤ 3 sentences explaining the change+ }+ """+).strip()+++async def _llm_draft_sigma(+ current_sigma: str,+ alert_fields: dict[str, Any],+ analyst_note: str,+) -> dict[str, Any]:+ """Call LLM to draft a Sigma improvement. Returns parsed JSON or template."""+ api_key = getattr(settings, "OPENAI_API_KEY", None) or getattr(+ settings, "LLM_API_KEY", None+ )+ if not api_key:+ return _template_fallback(current_sigma, alert_fields, analyst_note)++ user_msg = json.dumps(+ {+ "current_sigma_rule": current_sigma,+ "alert_fields": alert_fields,+ "analyst_note": analyst_note,+ },+ indent=2,+ )+ try:+ async with httpx.AsyncClient(timeout=30) as client:+ resp = await client.post(+ "https://api.openai.com/v1/chat/completions",+ headers={"Authorization": f"Bearer {api_key}"},+ json={+ "model": "gpt-4o-mini",+ "response_format": {"type": "json_object"},+ "messages": [+ {"role": "system", "content": _SYS_PROMPT},+ {"role": "user", "content": user_msg},+ ],+ },+ )+ resp.raise_for_status()+ raw = resp.json()["choices"][0]["message"]["content"]+ return json.loads(raw)+ except Exception:+ return _template_fallback(current_sigma, alert_fields, analyst_note)+++def _template_fallback(+ current_sigma: str, alert_fields: dict[str, Any], analyst_note: str+) -> dict[str, Any]:+ """Return a structured template when no LLM key is configured."""+ proc = alert_fields.get("process_name") or alert_fields.get("Image") or "UNKNOWN"+ user = alert_fields.get("user") or alert_fields.get("User") or "UNKNOWN"+ return {+ "rule_name": "fp-exclusion-draft",+ "sigma_yaml": textwrap.dedent(+ f"""\+ # AUTO-DRAFTED FP EXCLUSION — review before promoting+ # Original analyst note: {analyst_note}+ # Alert fields: {json.dumps(alert_fields, default=str)[:200]}+ filter:+ - process_name: '{proc}'+ - User: '{user}'+ condition: selection and not filter+ """+ ),+ "rationale": (+ f"Auto-generated exclusion for process '{proc}' / user '{user}'. "+ "Review and tighten before promoting. "+ "Analyst note: " + (analyst_note or "none provided")+ ),+ }+++# ────────────────────────────────────────────────────────────────────────────+# In-memory store (replace with DB table in prod)+# ────────────────────────────────────────────────────────────────────────────++_SUGGESTIONS: dict[uuid.UUID, dict[str, Any]] = {}+++# ────────────────────────────────────────────────────────────────────────────+# Endpoints+# ────────────────────────────────────────────────────────────────────────────+++ "/suggest",+ response_model=SuggestionResponse,+ status_code=status.HTTP_201_CREATED,+ summary="Draft Sigma improvement for a FP alert",+)+async def suggest_fp_fix(+ body: SuggestRequest,+ db: DBSession,+ user: AuthUser,+) -> SuggestionResponse:+ """Retrieve the triggering rule + alert evidence, then draft a Sigma improvement."""+ # 1. Load alert+ row = await db.execute(+ text(+ "SELECT rule_id, evidence, tenant_id FROM aisoc_alerts WHERE id = :aid"+ ).bindparams(aid=body.alert_id)+ )+ alert_row = row.fetchone()+ if not alert_row:+ raise HTTPException(status_code=404, detail="Alert not found")++ rule_id = alert_row.rule_id+ evidence: dict[str, Any] = alert_row.evidence or {}+ tenant_id: uuid.UUID | None = alert_row.tenant_id++ # 2. Load rule body if available+ current_sigma = "# Rule body not found\n"+ if rule_id:+ rule_row = await db.execute(+ text(+ "SELECT rule_body FROM aisoc_detection_rules WHERE id = :rid"+ ).bindparams(rid=rule_id)+ )+ rule_data = rule_row.fetchone()+ if rule_data:+ current_sigma = rule_data.rule_body++ # 3. Draft via LLM+ draft = await _llm_draft_sigma(+ current_sigma=current_sigma,+ alert_fields=evidence,+ analyst_note=body.analyst_note or "",+ )++ suggestion_id = uuid.uuid4()+ now = datetime.now(UTC)++ # 4. Auto-create a DAC proposal+ proposal_id: uuid.UUID | None = None+ try:+ proposal_id = uuid.uuid4()+ await db.execute(+ text(+ """+ INSERT INTO aisoc_detection_rule_proposals+ (id, tenant_id, base_rule_id, name, description,+ rule_language, rule_body, category, severity, confidence,+ status, source, created_at, updated_at)+ VALUES+ (:id, :tid, :rid, :name, :desc,+ 'sigma', :body, 'fp-fix', 'low', 70,+ 'draft', 'detection-loop', :now, :now)+ """+ ).bindparams(+ id=proposal_id,+ tid=tenant_id,+ rid=rule_id,+ name=draft.get("rule_name", "fp-exclusion-draft"),+ desc=draft.get("rationale", ""),+ body=draft.get("sigma_yaml", ""),+ now=now,+ )+ )+ await db.commit()+ except Exception:+ proposal_id = None+ await db.rollback()++ result = SuggestionResponse(+ suggestion_id=suggestion_id,+ alert_id=body.alert_id,+ base_rule_id=rule_id,+ draft_rule_name=draft.get("rule_name", "fp-exclusion-draft"),+ draft_sigma_yaml=draft.get("sigma_yaml", ""),+ rationale=draft.get("rationale", ""),+ proposal_id=proposal_id,+ created_at=now,+ )+ _SUGGESTIONS[suggestion_id] = result.model_dump()+ return result+++ "/suggestions",+ response_model=SuggestionListResponse,+ summary="List LLM-drafted Sigma suggestions",+)+async def list_suggestions(user: AuthUser) -> SuggestionListResponse:+ items = [SuggestionResponse(**v) for v in _SUGGESTIONS.values()]+ return SuggestionListResponse(suggestions=items, total=len(items))+++ "/suggestions/{suggestion_id}",+ response_model=SuggestionResponse,+ summary="Get one Sigma suggestion",+)+async def get_suggestion(+ suggestion_id: uuid.UUID,+ user: AuthUser,+) -> SuggestionResponse:+ item = _SUGGESTIONS.get(suggestion_id)+ if not item:+ raise HTTPException(status_code=404, detail="Suggestion not found")+ return SuggestionResponse(**item)diff --git a/services/api/app/api/v1/endpoints/feedback.py b/services/api/app/api/v1/endpoints/feedback.pynew file mode 100644index 000000000..62fb602cd--- /dev/null+++ b/services/api/app/api/v1/endpoints/feedback.py@@ -0,0 +1,146 @@+"""Analyst override feedback endpoint.++Allows analysts to correct AI-generated alert verdicts. Corrections are:+ 1. Persisted to the alerts table (disposition field).+ 2. Published as an event for downstream institutional memory ingestion.+ 3. Used to update SOC FPR metrics.+"""++from __future__ import annotations++from datetime import UTC, datetime++from fastapi import APIRouter, HTTPException, status+from pydantic import BaseModel, Field+from sqlalchemy import select, update++from app.api.v1.deps import AuthUser, DBSession+from app.models.alert import Alert++import structlog++logger = structlog.get_logger()++router = APIRouter(prefix="/feedback", tags=["feedback"])+++class AlertOverrideRequest(BaseModel):+ alert_id: str+ original_verdict: str = Field(..., description="The AI-generated verdict being overridden")+ corrected_verdict: str = Field(+ ...,+ description="Analyst's verdict: true_positive | false_positive | benign | escalate",+ )+ reason: str | None = Field(None, description="Optional free-text justification")+++class AlertOverrideResponse(BaseModel):+ alert_id: str+ corrected_verdict: str+ recorded_at: str++[email protected]("/alert-override", response_model=AlertOverrideResponse)+async def submit_alert_override(+ payload: AlertOverrideRequest,+ user: AuthUser,+ db: DBSession,+) -> AlertOverrideResponse:+ """Record an analyst verdict correction on an alert.++ This persists the override disposition and emits a structured log event+ that the institutional memory ingester picks up asynchronously.+ """+ valid_verdicts = {"true_positive", "false_positive", "benign", "escalate"}+ if payload.corrected_verdict not in valid_verdicts:+ raise HTTPException(+ status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,+ detail=f"corrected_verdict must be one of: {', '.join(sorted(valid_verdicts))}",+ )++ # Verify the alert belongs to this tenant+ row = await db.scalar(+ select(Alert).where(+ Alert.id == payload.alert_id,+ Alert.tenant_id == user.tenant_id,+ )+ )+ if row is None:+ raise HTTPException(+ status_code=status.HTTP_404_NOT_FOUND,+ detail="Alert not found",+ )++ # Persist disposition on the alert+ now = datetime.now(UTC)+ await db.execute(+ update(Alert)+ .where(Alert.id == payload.alert_id, Alert.tenant_id == user.tenant_id)+ .values(+ disposition=payload.corrected_verdict,+ updated_at=now,+ )+ )+ await db.commit()++ # Structured log event — picked up by institutional memory ingester+ logger.info(+ "analyst.override",+ tenant_id=user.tenant_id,+ alert_id=payload.alert_id,+ analyst_id=str(user.id),+ original_verdict=payload.original_verdict,+ corrected_verdict=payload.corrected_verdict,+ reason=payload.reason,+ recorded_at=now.isoformat(),+ )++ return AlertOverrideResponse(+ alert_id=payload.alert_id,+ corrected_verdict=payload.corrected_verdict,+ recorded_at=now.isoformat(),+ )+++class OverrideSummaryResponse(BaseModel):+ total_overrides: int+ false_positive_corrections: int+ true_positive_corrections: int+ benign_corrections: int++[email protected]("/summary", response_model=OverrideSummaryResponse)+async def get_override_summary(+ user: AuthUser,+ db: DBSession,+) -> OverrideSummaryResponse:+ """Return a summary of analyst overrides for this tenant."""+ from sqlalchemy import func, and_++ total = await db.scalar(+ select(func.count()).where(+ and_(Alert.tenant_id == user.tenant_id, Alert.disposition.isnot(None))+ )+ ) or 0+ fp = await db.scalar(+ select(func.count()).where(+ and_(Alert.tenant_id == user.tenant_id, Alert.disposition == "false_positive")+ )+ ) or 0+ tp = await db.scalar(+ select(func.count()).where(+ and_(Alert.tenant_id == user.tenant_id, Alert.disposition == "true_positive")+ )+ ) or 0+ benign = await db.scalar(+ select(func.count()).where(+ and_(Alert.tenant_id == user.tenant_id, Alert.disposition == "benign")+ )+ ) or 0++ return OverrideSummaryResponse(+ total_overrides=total,+ false_positive_corrections=fp,+ true_positive_corrections=tp,+ benign_corrections=benign,+ )diff --git a/services/api/app/api/v1/endpoints/hunts.py b/services/api/app/api/v1/endpoints/hunts.pynew file mode 100644index 000000000..4b9f4462d--- /dev/null+++ b/services/api/app/api/v1/endpoints/hunts.py@@ -0,0 +1,422 @@+"""Hypothesis-driven hunt workbench (tier2-hunting).++SOC analysts author threat-hunting hypotheses, attach multi-platform queries,+and track hunt runs and findings — all version-controlled via the DAC pipeline.++Endpoints+---------+* ``GET /hunts`` List hunts.+* ``POST /hunts`` Create a hunt hypothesis.+* ``GET /hunts/{id}`` Get a hunt.+* ``PATCH /hunts/{id}`` Update hypothesis / status / priority.+* ``POST /hunts/{id}/run`` Execute a hunt query (ES|QL live; SPL/KQL templated).+* ``GET /hunts/{id}/runs`` List run history for a hunt.+* ``POST /hunts/{id}/findings`` Append finding entries to a hunt.+"""++from __future__ import annotations++import json+import os+import uuid+from datetime import UTC, datetime+from typing import Any, Literal++import httpx+from fastapi import APIRouter, HTTPException, Query, status+from pydantic import BaseModel, Field+from sqlalchemy import text++from app.api.v1.deps import AuthUser, DBSession++router = APIRouter(prefix="/hunts", tags=["hunts"])++# ────────────────────────────────────────────────────────────────────────────+# Pydantic schemas+# ────────────────────────────────────────────────────────────────────────────++HuntStatus = Literal["draft", "active", "completed", "archived"]+HuntPriority = Literal["low", "medium", "high", "critical"]+HuntPlatform = Literal["esql", "spl", "kql"]+++class CreateHuntRequest(BaseModel):+ title: str = Field(..., min_length=3)+ hypothesis: str = Field(..., min_length=10, description="Plain-English threat hypothesis.")+ mitre_tactic: str | None = None+ mitre_technique: str | None = None+ priority: HuntPriority = "medium"+ assigned_to: str | None = None+ tags: list[str] = Field(default_factory=list)+++class UpdateHuntRequest(BaseModel):+ title: str | None = None+ hypothesis: str | None = None+ mitre_tactic: str | None = None+ mitre_technique: str | None = None+ status: HuntStatus | None = None+ priority: HuntPriority | None = None+ assigned_to: str | None = None+ query_esql: str | None = None+ query_spl: str | None = None+ query_kql: str | None = None+ tags: list[str] | None = None+++class AddFindingsRequest(BaseModel):+ findings: list[dict[str, Any]] = Field(..., min_length=1)+ false_positive_rate: float | None = Field(None, ge=0.0, le=1.0)+++class RunHuntRequest(BaseModel):+ platform: HuntPlatform = "esql"+++class HuntRunResponse(BaseModel):+ id: uuid.UUID+ hunt_id: uuid.UUID+ run_at: datetime+ platform: str+ query_used: str | None+ hit_count: int+ result_sample: list[dict[str, Any]]+ duration_ms: int | None+ error: str | None+++class HuntResponse(BaseModel):+ id: uuid.UUID+ title: str+ hypothesis: str+ mitre_tactic: str | None+ mitre_technique: str | None+ status: str+ priority: str+ query_esql: str | None+ query_spl: str | None+ query_kql: str | None+ findings: list[dict[str, Any]]+ false_positive_rate: float | None+ assigned_to: str | None+ tags: list[str]+ created_at: datetime+ updated_at: datetime+ completed_at: datetime | None+ created_by: str | None+++# ────────────────────────────────────────────────────────────────────────────+# LLM query generation helper+# ────────────────────────────────────────────────────────────────────────────++_HUNT_SYSTEM = """You are a senior threat hunter. Given a threat hypothesis, generate+detection queries for the listed platforms.++Return ONLY valid JSON with this structure:+{+ "esql": "<ES|QL query string>",+ "spl": "<Splunk SPL string>",+ "kql": "<KQL string>"+}+No prose. Just JSON."""+++async def _generate_queries(hypothesis: str, mitre: str | None) -> dict[str, str] | None:+ api_key = os.getenv("OPENAI_API_KEY") or os.getenv("LLM_API_KEY")+ if not api_key:+ return None+ base_url = os.getenv("LLM_BASE_URL", "https://api.openai.com/v1")+ model = os.getenv("LLM_MODEL", "gpt-4o-mini")+ user_msg = f"HYPOTHESIS: {hypothesis}"+ if mitre:+ user_msg += f"\nMITRE TECHNIQUE: {mitre}"+ user_msg += "\nGenerate ES|QL, SPL, and KQL hunt queries."+ try:+ async with httpx.AsyncClient(timeout=45) as client:+ resp = await client.post(+ f"{base_url}/chat/completions",+ headers={"Authorization": f"Bearer {api_key}"},+ json={+ "model": model,+ "messages": [+ {"role": "system", "content": _HUNT_SYSTEM},+ {"role": "user", "content": user_msg},+ ],+ "temperature": 0.2,+ "response_format": {"type": "json_object"},+ },+ )+ resp.raise_for_status()+ return json.loads(resp.json()["choices"][0]["message"]["content"])+ except Exception:+ return None+++def _fallback_queries(hypothesis: str) -> dict[str, str]:+ escaped = hypothesis.replace('"', '\\"')+ return {+ "esql": f'FROM logs-* | WHERE message LIKE "%{escaped[:60]}%" | LIMIT 100',+ "spl": f'index=* "{escaped[:60]}" | head 100',+ "kql": f'// KQL hunt — adapt field names\nSecurityEvent\n| where Activity has "{escaped[:60]}"\n| limit 100',+ }+++# ────────────────────────────────────────────────────────────────────────────+# Row helper+# ────────────────────────────────────────────────────────────────────────────+++def _row_to_hunt(row: Any) -> HuntResponse:+ return HuntResponse(+ id=row.id,+ title=row.title,+ hypothesis=row.hypothesis,+ mitre_tactic=row.mitre_tactic,+ mitre_technique=row.mitre_technique,+ status=row.status,+ priority=row.priority,+ query_esql=row.query_esql,+ query_spl=row.query_spl,+ query_kql=row.query_kql,+ findings=list(row.findings or []),+ false_positive_rate=row.false_positive_rate,+ assigned_to=row.assigned_to,+ tags=list(row.tags or []),+ created_at=row.created_at,+ updated_at=row.updated_at,+ completed_at=row.completed_at,+ created_by=row.created_by,+ )+++# ────────────────────────────────────────────────────────────────────────────+# Endpoints+# ────────────────────────────────────────────────────────────────────────────++[email protected]("", response_model=list[HuntResponse], summary="List hunt hypotheses")+async def list_hunts(+ db: DBSession,+ user: AuthUser,+ hunt_status: str | None = Query(None, alias="status"),+ priority: str | None = Query(None),+ limit: int = Query(50, ge=1, le=500),+ offset: int = Query(0, ge=0),+) -> list[HuntResponse]:+ wheres = ["1=1"]+ params: dict[str, Any] = {"limit": limit, "offset": offset}+ if hunt_status:+ wheres.append("status = :status"); params["status"] = hunt_status+ if priority:+ wheres.append("priority = :priority"); params["priority"] = priority++ q = text(f"SELECT * FROM aisoc_hunts WHERE {' AND '.join(wheres)} ORDER BY created_at DESC LIMIT :limit OFFSET :offset").bindparams(**params)+ try:+ rows = (await db.execute(q)).fetchall()+ return [_row_to_hunt(r) for r in rows]+ except Exception as exc:+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc++[email protected]("", response_model=HuntResponse, status_code=status.HTTP_201_CREATED, summary="Create hunt hypothesis")+async def create_hunt(body: CreateHuntRequest, db: DBSession, user: AuthUser) -> HuntResponse:+ mitre = body.mitre_technique or body.mitre_tactic+ queries = await _generate_queries(body.hypothesis, mitre) or _fallback_queries(body.hypothesis)+ hunt_id = uuid.uuid4()+ now = datetime.now(UTC)+ q = text("""+ INSERT INTO aisoc_hunts (+ id, title, hypothesis, mitre_tactic, mitre_technique,+ priority, assigned_to, tags, query_esql, query_spl, query_kql,+ created_at, updated_at, created_by+ ) VALUES (+ :id, :title, :hypothesis, :tactic, :technique,+ :priority, :assigned, :tags::text[], :esql, :spl, :kql,+ :now, :now, :user+ ) RETURNING *+ """).bindparams(+ id=hunt_id,+ title=body.title,+ hypothesis=body.hypothesis,+ tactic=body.mitre_tactic,+ technique=body.mitre_technique,+ priority=body.priority,+ assigned=body.assigned_to,+ tags=body.tags or [],+ esql=queries.get("esql"),+ spl=queries.get("spl"),+ kql=queries.get("kql"),+ now=now,+ user=str(user) if user else "system",+ )+ try:+ row = (await db.execute(q)).fetchone()+ await db.commit()+ return _row_to_hunt(row)+ except Exception as exc:+ await db.rollback()+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc++[email protected]("/{hunt_id}", response_model=HuntResponse, summary="Get hunt")+async def get_hunt(hunt_id: uuid.UUID, db: DBSession, user: AuthUser) -> HuntResponse:+ row = (await db.execute(text("SELECT * FROM aisoc_hunts WHERE id = :id").bindparams(id=hunt_id))).fetchone()+ if not row:+ raise HTTPException(status_code=404, detail="Hunt not found.")+ return _row_to_hunt(row)++[email protected]("/{hunt_id}", response_model=HuntResponse, summary="Update hunt")+async def update_hunt(hunt_id: uuid.UUID, body: UpdateHuntRequest, db: DBSession, user: AuthUser) -> HuntResponse:+ sets = ["updated_at = :now"]+ params: dict[str, Any] = {"id": hunt_id, "now": datetime.now(UTC)}++ if body.title is not None:+ sets.append("title = :title"); params["title"] = body.title+ if body.hypothesis is not None:+ sets.append("hypothesis = :hypothesis"); params["hypothesis"] = body.hypothesis+ if body.mitre_tactic is not None:+ sets.append("mitre_tactic = :tactic"); params["tactic"] = body.mitre_tactic+ if body.mitre_technique is not None:+ sets.append("mitre_technique = :technique"); params["technique"] = body.mitre_technique+ if body.status is not None:+ sets.append("status = :status"); params["status"] = body.status+ if body.status == "completed":+ sets.append("completed_at = :now")+ if body.priority is not None:+ sets.append("priority = :priority"); params["priority"] = body.priority+ if body.assigned_to is not None:+ sets.append("assigned_to = :assigned"); params["assigned"] = body.assigned_to+ if body.query_esql is not None:+ sets.append("query_esql = :esql"); params["esql"] = body.query_esql+ if body.query_spl is not None:+ sets.append("query_spl = :spl"); params["spl"] = body.query_spl+ if body.query_kql is not None:+ sets.append("query_kql = :kql"); params["kql"] = body.query_kql+ if body.tags is not None:+ sets.append("tags = :tags::text[]"); params["tags"] = body.tags++ q = text(f"UPDATE aisoc_hunts SET {', '.join(sets)} WHERE id = :id RETURNING *").bindparams(**params)+ try:+ row = (await db.execute(q)).fetchone()+ if not row:+ raise HTTPException(status_code=404, detail="Hunt not found.")+ await db.commit()+ return _row_to_hunt(row)+ except HTTPException:+ raise+ except Exception as exc:+ await db.rollback()+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc++[email protected]("/{hunt_id}/run", response_model=HuntRunResponse, status_code=status.HTTP_200_OK, summary="Execute hunt query")+async def run_hunt(hunt_id: uuid.UUID, body: RunHuntRequest, db: DBSession, user: AuthUser) -> HuntRunResponse:+ hunt_row = (await db.execute(text("SELECT * FROM aisoc_hunts WHERE id = :id").bindparams(id=hunt_id))).fetchone()+ if not hunt_row:+ raise HTTPException(status_code=404, detail="Hunt not found.")++ query_map = {"esql": hunt_row.query_esql, "spl": hunt_row.query_spl, "kql": hunt_row.query_kql}+ query = query_map.get(body.platform)+ hit_count = 0+ result_sample: list[dict[str, Any]] = []+ error_msg: str | None = None+ start = datetime.now(UTC)++ if body.platform == "esql" and query:+ es_url = os.getenv("ELASTICSEARCH_URL")+ es_key = os.getenv("ELASTICSEARCH_API_KEY")+ if es_url and es_key:+ try:+ async with httpx.AsyncClient(timeout=30) as client:+ resp = await client.post(+ f"{es_url.rstrip('/')}/_query",+ headers={"Authorization": f"ApiKey {es_key}", "Content-Type": "application/json"},+ json={"query": query},+ )+ if resp.status_code == 200:+ data = resp.json()+ rows = data.get("rows", [])+ cols = [c["name"] for c in data.get("columns", [])]+ hit_count = len(rows)+ result_sample = [{cols[i]: row[i] for i in range(len(cols))} for row in rows[:20]]+ else:+ error_msg = f"ES returned {resp.status_code}: {resp.text[:200]}"+ except Exception as exc:+ error_msg = str(exc)[:300]+ else:+ error_msg = "ELASTICSEARCH_URL or ELASTICSEARCH_API_KEY not configured."+ elif not query:+ error_msg = f"No {body.platform} query set for this hunt."+ else:+ error_msg = f"{body.platform} live execution not supported — query stored for manual use."++ duration_ms = int((datetime.now(UTC) - start).total_seconds() * 1000)+ run_id = uuid.uuid4()+ q = text("""+ INSERT INTO aisoc_hunt_runs (id, hunt_id, platform, query_used, hit_count, result_sample, duration_ms, error)+ VALUES (:id, :hunt_id, :platform, :query, :hits, :sample::jsonb, :dur, :err) RETURNING *+ """).bindparams(+ id=run_id,+ hunt_id=hunt_id,+ platform=body.platform,+ query=query,+ hits=hit_count,+ sample=json.dumps(result_sample),+ dur=duration_ms,+ err=error_msg,+ )+ try:+ row = (await db.execute(q)).fetchone()+ await db.commit()+ return HuntRunResponse(+ id=row.id, hunt_id=row.hunt_id, run_at=row.run_at,+ platform=row.platform, query_used=row.query_used,+ hit_count=row.hit_count, result_sample=list(row.result_sample or []),+ duration_ms=row.duration_ms, error=row.error,+ )+ except Exception as exc:+ await db.rollback()+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc++[email protected]("/{hunt_id}/runs", response_model=list[HuntRunResponse], summary="List hunt runs")+async def list_runs(hunt_id: uuid.UUID, db: DBSession, user: AuthUser) -> list[HuntRunResponse]:+ rows = (await db.execute(text("SELECT * FROM aisoc_hunt_runs WHERE hunt_id = :id ORDER BY run_at DESC LIMIT 100").bindparams(id=hunt_id))).fetchall()+ return [+ HuntRunResponse(+ id=r.id, hunt_id=r.hunt_id, run_at=r.run_at,+ platform=r.platform, query_used=r.query_used,+ hit_count=r.hit_count, result_sample=list(r.result_sample or []),+ duration_ms=r.duration_ms, error=r.error,+ )+ for r in rows+ ]++[email protected]("/{hunt_id}/findings", response_model=HuntResponse, summary="Append findings")+async def add_findings(hunt_id: uuid.UUID, body: AddFindingsRequest, db: DBSession, user: AuthUser) -> HuntResponse:+ existing = (await db.execute(text("SELECT findings FROM aisoc_hunts WHERE id = :id").bindparams(id=hunt_id))).fetchone()+ if not existing:+ raise HTTPException(status_code=404, detail="Hunt not found.")++ merged = list(existing.findings or []) + body.findings+ params: dict[str, Any] = {+ "id": hunt_id,+ "findings": json.dumps(merged),+ "now": datetime.now(UTC),+ }+ extra_set = ""+ if body.false_positive_rate is not None:+ extra_set = ", false_positive_rate = :fpr"+ params["fpr"] = body.false_positive_rate++ q = text(f"UPDATE aisoc_hunts SET findings = :findings::jsonb, updated_at = :now{extra_set} WHERE id = :id RETURNING *").bindparams(**params)+ try:+ row = (await db.execute(q)).fetchone()+ await db.commit()+ return _row_to_hunt(row)+ except Exception as exc:+ await db.rollback()+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from excdiff --git a/services/api/app/api/v1/endpoints/identity_timeline.py b/services/api/app/api/v1/endpoints/identity_timeline.pynew file mode 100644index 000000000..b2b9ed7f3--- /dev/null+++ b/services/api/app/api/v1/endpoints/identity_timeline.py@@ -0,0 +1,225 @@+"""Identity-centric investigation timeline (tier2-identity).++Builds a chronological event timeline anchored to a user, device, or+service-account identity. Queries ``aisoc_alerts``, ``aisoc_events``,+and the enrichment store to produce an ordered list of security events+grouped by entity, with ATT&CK technique annotations.++Endpoints+---------+* ``GET /identity-timeline`` Search timelines by identity.+* ``POST /identity-timeline/build`` Build a fresh timeline for an identity.+"""++from __future__ import annotations++import uuid+from datetime import UTC, datetime, timedelta+from typing import Any, Literal++from fastapi import APIRouter, HTTPException, Query, status+from pydantic import BaseModel, Field+from sqlalchemy import text++from app.api.v1.deps import AuthUser, DBSession++router = APIRouter(prefix="/identity-timeline", tags=["identity_timeline"])++# ────────────────────────────────────────────────────────────────────────────+# Pydantic schemas+# ────────────────────────────────────────────────────────────────────────────++IdentityKind = Literal["user", "device", "service_account", "ip"]+++class TimelineEvent(BaseModel):+ event_id: uuid.UUID+ timestamp: datetime+ event_type: Literal["alert", "login", "process", "network", "file", "other"]+ source: str+ description: str+ severity: str | None = None+ mitre_technique: str | None = None+ raw: dict[str, Any] | None = None+++class IdentityTimeline(BaseModel):+ timeline_id: uuid.UUID+ identity_kind: IdentityKind+ identity_value: str+ from_ts: datetime+ to_ts: datetime+ events: list[TimelineEvent]+ total_events: int+ risk_score: float | None = None+ built_at: datetime+++class BuildTimelineRequest(BaseModel):+ identity_kind: IdentityKind = Field(..., description="Type of identity anchor.")+ identity_value: str = Field(..., min_length=1, description="Identity value (e.g. username, hostname).")+ from_ts: datetime | None = Field(None, description="Start of window (defaults to 72 h ago).")+ to_ts: datetime | None = Field(None, description="End of window (defaults to now).")+ max_events: int = Field(200, ge=1, le=2000)+++# ────────────────────────────────────────────────────────────────────────────+# Helpers+# ────────────────────────────────────────────────────────────────────────────++_FIELD_MAP: dict[IdentityKind, list[str]] = {+ "user": ["user", "username", "src_user", "dest_user", "actor"],+ "device": ["hostname", "device", "src_host", "computer_name"],+ "service_account": ["service_account", "svc_account", "process_user"],+ "ip": ["src_ip", "source_ip", "ip_address", "remote_ip"],+}+++def _risk_score(events: list[TimelineEvent]) -> float:+ """Simple heuristic: sum severity weights, cap at 100."""+ weights = {"critical": 40, "high": 20, "medium": 10, "low": 3, "info": 1}+ score = sum(weights.get((e.severity or "info").lower(), 1) for e in events)+ return min(100.0, float(score))+++def _mitre_from_alert(raw: dict[str, Any]) -> str | None:+ return (+ raw.get("mitre_technique")+ or raw.get("mitre_attack_id")+ or raw.get("tags", {}).get("mitre")+ )+++# ────────────────────────────────────────────────────────────────────────────+# Endpoints+# ────────────────────────────────────────────────────────────────────────────+++ "/build",+ response_model=IdentityTimeline,+ status_code=status.HTTP_200_OK,+ summary="Build identity-centric investigation timeline",+)+async def build_timeline(+ body: BuildTimelineRequest,+ db: DBSession,+ user: AuthUser,+) -> IdentityTimeline:+ now = datetime.now(UTC)+ from_ts = body.from_ts or (now - timedelta(hours=72))+ to_ts = body.to_ts or now++ events: list[TimelineEvent] = []++ # ── 1. Alerts that reference this identity ──────────────────────────────+ try:+ alert_rows = await db.execute(+ text(+ """+ SELECT id, created_at, severity, title, evidence, mitre_technique+ FROM aisoc_alerts+ WHERE created_at BETWEEN :from_ts AND :to_ts+ AND (+ evidence::text ILIKE :pat+ OR title ILIKE :pat+ )+ ORDER BY created_at DESC+ LIMIT :lim+ """+ ).bindparams(+ from_ts=from_ts,+ to_ts=to_ts,+ pat=f"%{body.identity_value}%",+ lim=body.max_events,+ )+ )+ for row in alert_rows.fetchall():+ events.append(+ TimelineEvent(+ event_id=row.id,+ timestamp=row.created_at,+ event_type="alert",+ source="aisoc_alerts",+ description=row.title or "Alert",+ severity=row.severity,+ mitre_technique=row.mitre_technique+ or _mitre_from_alert(row.evidence or {}),+ raw=row.evidence,+ )+ )+ except Exception:+ pass # table may not exist in all deployments++ # ── 2. Raw events from aisoc_events (if table exists) ──────────────────+ try:+ ev_rows = await db.execute(+ text(+ """+ SELECT id, timestamp, event_type, source, description,+ severity, raw_event+ FROM aisoc_events+ WHERE timestamp BETWEEN :from_ts AND :to_ts+ AND raw_event::text ILIKE :pat+ ORDER BY timestamp DESC+ LIMIT :lim+ """+ ).bindparams(+ from_ts=from_ts,+ to_ts=to_ts,+ pat=f"%{body.identity_value}%",+ lim=body.max_events - len(events),+ )+ )+ for row in ev_rows.fetchall():+ events.append(+ TimelineEvent(+ event_id=row.id,+ timestamp=row.timestamp,+ event_type=row.event_type or "other",+ source=row.source or "aisoc_events",+ description=row.description or "",+ severity=row.severity,+ raw=row.raw_event,+ )+ )+ except Exception:+ pass++ # ── 3. Sort chronologically ─────────────────────────────────────────────+ events.sort(key=lambda e: e.timestamp)++ return IdentityTimeline(+ timeline_id=uuid.uuid4(),+ identity_kind=body.identity_kind,+ identity_value=body.identity_value,+ from_ts=from_ts,+ to_ts=to_ts,+ events=events[: body.max_events],+ total_events=len(events),+ risk_score=_risk_score(events),+ built_at=now,+ )+++ "",+ response_model=IdentityTimeline,+ summary="Quick timeline lookup for an identity",+)+async def get_timeline(+ identity_kind: IdentityKind = Query(...),+ identity_value: str = Query(..., min_length=1),+ hours: int = Query(72, ge=1, le=8760),+ db: DBSession = ...,+ user: AuthUser = ...,+) -> IdentityTimeline:+ return await build_timeline(+ BuildTimelineRequest(+ identity_kind=identity_kind,+ identity_value=identity_value,+ from_ts=datetime.now(UTC) - timedelta(hours=hours),+ ),+ db=db,+ user=user,+ )diff --git a/services/api/app/api/v1/endpoints/knowledge_base.py b/services/api/app/api/v1/endpoints/knowledge_base.pynew file mode 100644index 000000000..4a0fc1f50--- /dev/null+++ b/services/api/app/api/v1/endpoints/knowledge_base.py@@ -0,0 +1,246 @@+"""Knowledge-base + RAG over org docs/runbooks (tier3-rag).++Analysts ingest organisation documents (runbooks, policies, playbooks, SOPs)+into a full-text-searchable store. A ``/query`` endpoint retrieves relevant+chunks and, when an LLM key is configured, synthesises a cited answer.++Endpoints+---------+* ``POST /kb/ingest`` Ingest a document (chunked automatically).+* ``GET /kb/documents`` List indexed documents.+* ``GET /kb/documents/{id}`` Get a document.+* ``DELETE /kb/documents/{id}`` Remove a document.+* ``POST /kb/query`` Semantic/keyword search + optional LLM synthesis.+"""++from __future__ import annotations++import json+import os+import uuid+from datetime import UTC, datetime+from typing import Any, Literal++import httpx+from fastapi import APIRouter, HTTPException, status+from pydantic import BaseModel, Field+from sqlalchemy import text++from app.api.v1.deps import AuthUser, DBSession++router = APIRouter(prefix="/kb", tags=["knowledge_base"])++# ────────────────────────────────────────────────────────────────────────────+# Schemas+# ────────────────────────────────────────────────────────────────────────────++DocKind = Literal["runbook", "policy", "playbook", "sop", "wiki", "other"]+_CHUNK_SIZE = 800 # characters per chunk+++class IngestRequest(BaseModel):+ title: str = Field(..., min_length=2)+ content: str = Field(..., min_length=10)+ doc_kind: DocKind = "runbook"+ source_url: str | None = None+ tags: list[str] = Field(default_factory=list)+++class KBDocResponse(BaseModel):+ id: uuid.UUID+ title: str+ doc_kind: str+ source_url: str | None+ tags: list[str]+ chunk_index: int+ chunk_total: int+ content_preview: str+ created_at: datetime+ created_by: str | None+++class QueryRequest(BaseModel):+ question: str = Field(..., min_length=3)+ doc_kinds: list[DocKind] = Field(default_factory=list)+ top_k: int = Field(5, ge=1, le=20)+ synthesise: bool = Field(True, description="Use LLM to synthesise an answer from retrieved chunks.")+++class KBChunk(BaseModel):+ doc_id: uuid.UUID+ title: str+ chunk_index: int+ content: str+ score: float | None = None+++class QueryResponse(BaseModel):+ question: str+ chunks: list[KBChunk]+ answer: str | None # None when synthesise=False or no LLM key+ sources: list[str]+++# ────────────────────────────────────────────────────────────────────────────+# Helpers+# ────────────────────────────────────────────────────────────────────────────+++def _chunk(content: str, chunk_size: int = _CHUNK_SIZE) -> list[str]:+ return [content[i: i + chunk_size] for i in range(0, len(content), chunk_size)] or [content]+++def _row_to_doc(row: Any) -> KBDocResponse:+ return KBDocResponse(+ id=row.id,+ title=row.title,+ doc_kind=row.doc_kind,+ source_url=row.source_url,+ tags=list(row.tags or []),+ chunk_index=row.chunk_index,+ chunk_total=row.chunk_total,+ content_preview=row.content[:200],+ created_at=row.created_at,+ created_by=row.created_by,+ )+++_SYNTH_SYSTEM = """You are a helpful security operations assistant.+Answer the analyst's question using ONLY the provided context chunks.+Cite document titles inline as [Doc Title].+If the answer cannot be found in the context, say so clearly."""+++async def _synthesise(question: str, chunks: list[KBChunk]) -> str | None:+ api_key = os.getenv("OPENAI_API_KEY") or os.getenv("LLM_API_KEY")+ if not api_key:+ return None+ base_url = os.getenv("LLM_BASE_URL", "https://api.openai.com/v1")+ model = os.getenv("LLM_MODEL", "gpt-4o-mini")+ context = "\n\n".join(f"[{c.title}] chunk {c.chunk_index}:\n{c.content}" for c in chunks)+ try:+ async with httpx.AsyncClient(timeout=45) as client:+ resp = await client.post(+ f"{base_url}/chat/completions",+ headers={"Authorization": f"Bearer {api_key}"},+ json={+ "model": model,+ "messages": [+ {"role": "system", "content": _SYNTH_SYSTEM},+ {"role": "user", "content": f"CONTEXT:\n{context}\n\nQUESTION: {question}"},+ ],+ "temperature": 0.2,+ },+ )+ resp.raise_for_status()+ return resp.json()["choices"][0]["message"]["content"].strip()+ except Exception:+ return None+++# ────────────────────────────────────────────────────────────────────────────+# Endpoints+# ────────────────────────────────────────────────────────────────────────────++[email protected]("/ingest", response_model=list[KBDocResponse], status_code=status.HTTP_201_CREATED, summary="Ingest document into knowledge base")+async def ingest(body: IngestRequest, db: DBSession, user: AuthUser) -> list[KBDocResponse]:+ chunks = _chunk(body.content)+ now = datetime.now(UTC)+ rows = []+ for idx, chunk_text in enumerate(chunks):+ doc_id = uuid.uuid4()+ q = text("""+ INSERT INTO aisoc_kb_documents (+ id, title, doc_kind, source_url, content, tags,+ chunk_index, chunk_total, created_at, updated_at, created_by+ ) VALUES (+ :id, :title, :kind, :url, :content, :tags::text[],+ :idx, :total, :now, :now, :user+ ) RETURNING *+ """).bindparams(+ id=doc_id,+ title=body.title,+ kind=body.doc_kind,+ url=body.source_url,+ content=chunk_text,+ tags=body.tags or [],+ idx=idx,+ total=len(chunks),+ now=now,+ user=str(user) if user else "system",+ )+ try:+ row = (await db.execute(q)).fetchone()+ rows.append(row)+ except Exception as exc:+ await db.rollback()+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc+ await db.commit()+ return [_row_to_doc(r) for r in rows]++[email protected]("/documents", response_model=list[KBDocResponse], summary="List KB documents")+async def list_documents(db: DBSession, user: AuthUser) -> list[KBDocResponse]:+ try:+ rows = (await db.execute(text("SELECT * FROM aisoc_kb_documents WHERE chunk_index = 0 ORDER BY created_at DESC LIMIT 200"))).fetchall()+ return [_row_to_doc(r) for r in rows]+ except Exception as exc:+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc++[email protected]("/documents/{doc_id}", response_model=KBDocResponse, summary="Get KB document")+async def get_document(doc_id: uuid.UUID, db: DBSession, user: AuthUser) -> KBDocResponse:+ row = (await db.execute(text("SELECT * FROM aisoc_kb_documents WHERE id = :id").bindparams(id=doc_id))).fetchone()+ if not row:+ raise HTTPException(status_code=404, detail="Document not found.")+ return _row_to_doc(row)++[email protected]("/documents/{doc_id}", status_code=status.HTTP_204_NO_CONTENT, summary="Remove KB document")+async def delete_document(doc_id: uuid.UUID, db: DBSession, user: AuthUser) -> None:+ existing = (await db.execute(text("SELECT title FROM aisoc_kb_documents WHERE id = :id").bindparams(id=doc_id))).fetchone()+ if not existing:+ raise HTTPException(status_code=404, detail="Document not found.")+ # Remove all chunks with same title + doc_kind+ await db.execute(text("DELETE FROM aisoc_kb_documents WHERE title = :title").bindparams(title=existing.title))+ await db.commit()++[email protected]("/query", response_model=QueryResponse, summary="Search knowledge base + optional LLM synthesis")+async def query_kb(body: QueryRequest, db: DBSession, user: AuthUser) -> QueryResponse:+ wheres = ["to_tsvector('english', content) @@ plainto_tsquery('english', :q)"]+ params: dict[str, Any] = {"q": body.question, "limit": body.top_k}+ if body.doc_kinds:+ wheres.append("doc_kind = ANY(:kinds::text[])")+ params["kinds"] = body.doc_kinds++ sql = text(f"""+ SELECT *, ts_rank(to_tsvector('english', content), plainto_tsquery('english', :q)) AS rank+ FROM aisoc_kb_documents+ WHERE {' AND '.join(wheres)}+ ORDER BY rank DESC+ LIMIT :limit+ """).bindparams(**params)++ try:+ db_rows = (await db.execute(sql)).fetchall()+ except Exception as exc:+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc++ chunks = [+ KBChunk(+ doc_id=r.id,+ title=r.title,+ chunk_index=r.chunk_index,+ content=r.content,+ score=float(r.rank) if hasattr(r, "rank") else None,+ )+ for r in db_rows+ ]+ answer: str | None = None+ if body.synthesise and chunks:+ answer = await _synthesise(body.question, chunks)++ sources = list(dict.fromkeys(c.title for c in chunks))+ return QueryResponse(question=body.question, chunks=chunks, answer=answer, sources=sources)diff --git a/services/api/app/api/v1/endpoints/metrics.py b/services/api/app/api/v1/endpoints/metrics.pyindex 7d39b7396..043f78c86 100644--- a/services/api/app/api/v1/endpoints/metrics.py+++ b/services/api/app/api/v1/endpoints/metrics.py@@ -206,6 +206,143 @@ async def get_dashboard_metrics()+class SOCKpis(BaseModel):+ mttd_hours: float+ mttr_hours: float+ false_positive_rate: float+ alert_volume_7d: int+ cases_opened_7d: int+ cases_closed_7d: int+ analyst_overrides_7d: int+++class AttackHeatmapCell(BaseModel):+ tactic: str+ technique: str+ count: int+++class SOCMetrics(BaseModel):+ kpis: SOCKpis+ attack_heatmap: list[AttackHeatmapCell]++[email protected]("/soc", response_model=SOCMetrics)+async def get_soc_metrics(+ user: AuthUser,+ db: DBSession,+) -> SOCMetrics:+ """Return SOC-level KPIs: MTTD, MTTR, FPR, and ATT&CK heatmap data."""+ tenant_id = user.tenant_id+ now = datetime.now(UTC)+ week_start = now - timedelta(days=7)++ # MTTD: mean time from alert creation to first investigation start+ mttd_q = await db.scalar(+ select(func.avg(+ func.extract("epoch", Alert.first_seen_at - Alert.created_at) / 3600+ )).where(+ and_(+ Alert.tenant_id == tenant_id,+ Alert.first_seen_at.isnot(None),+ Alert.created_at >= week_start,+ )+ )+ )+ mttd_hours = float(mttd_q or 0.0)++ # MTTR: mean time from alert creation to resolution+ mttr_q = await db.scalar(+ select(func.avg(+ func.extract("epoch", Alert.updated_at - Alert.created_at) / 3600+ )).where(+ and_(+ Alert.tenant_id == tenant_id,+ Alert.status == "resolved",+ Alert.created_at >= week_start,+ )+ )+ )+ mttr_hours = float(mttr_q or 0.0)++ # FPR: false positives / total resolved alerts (last 7d)+ total_resolved = await db.scalar(+ select(func.count()).where(+ and_(+ Alert.tenant_id == tenant_id,+ Alert.status == "resolved",+ Alert.created_at >= week_start,+ )+ )+ ) or 0+ fp_count = await db.scalar(+ select(func.count()).where(+ and_(+ Alert.tenant_id == tenant_id,+ Alert.status == "resolved",+ Alert.disposition == "false_positive",+ Alert.created_at >= week_start,+ )+ )+ ) or 0+ fpr = (fp_count / total_resolved) if total_resolved > 0 else 0.0++ # Alert volume 7d+ alert_vol = await db.scalar(+ select(func.count()).where(+ and_(Alert.tenant_id == tenant_id, Alert.created_at >= week_start)+ )+ ) or 0++ # Cases opened/closed 7d+ cases_opened = await db.scalar(+ select(func.count()).where(+ and_(Case.tenant_id == tenant_id, Case.created_at >= week_start)+ )+ ) or 0+ cases_closed = await db.scalar(+ select(func.count()).where(+ and_(+ Case.tenant_id == tenant_id,+ Case.status == "resolved",+ Case.updated_at >= week_start,+ )+ )+ ) or 0++ kpis = SOCKpis(+ mttd_hours=round(mttd_hours, 2),+ mttr_hours=round(mttr_hours, 2),+ false_positive_rate=round(fpr, 4),+ alert_volume_7d=alert_vol,+ cases_opened_7d=cases_opened,+ cases_closed_7d=cases_closed,+ analyst_overrides_7d=0,+ )++ # ATT&CK heatmap: technique + tactic breakdown+ heatmap_rows = (+ await db.execute(+ select(+ func.jsonb_array_elements_text(Alert.mitre_tactics).label("tactic"),+ func.jsonb_array_elements_text(Alert.mitre_techniques).label("technique"),+ func.count().label("cnt"),+ )+ .where(Alert.tenant_id == tenant_id)+ .group_by("tactic", "technique")+ .order_by(func.count().desc())+ .limit(50)+ )+ ).all()++ heatmap = [+ AttackHeatmapCell(tactic=r.tactic, technique=r.technique, count=r.cnt)+ for r in heatmap_rows+ ]++ return SOCMetrics(kpis=kpis, attack_heatmap=heatmap)++@router.get("/alerts/trend")async def get_alert_trend(user: AuthUser,diff --git a/services/api/app/api/v1/endpoints/nl_detection.py b/services/api/app/api/v1/endpoints/nl_detection.pynew file mode 100644index 000000000..d838b12e6--- /dev/null+++ b/services/api/app/api/v1/endpoints/nl_detection.py@@ -0,0 +1,206 @@+"""Natural-language detection authoring endpoint.++Accepts a plain-English description of a threat and returns:+ - A Sigma YAML rule+ - Equivalent KQL (Microsoft Sentinel)+ - Equivalent SPL (Splunk)+ - Equivalent ES|QL (Elastic)++Uses an LLM (OpenAI-compatible) to perform the translation. Falls back to a+template-based stub when no LLM key is configured so the endpoint stays+functional in all deployment environments.+"""++from __future__ import annotations++import os+from typing import Literal++from fastapi import APIRouter, HTTPException, status+from pydantic import BaseModel, Field++import structlog++logger = structlog.get_logger()++router = APIRouter(prefix="/nl-detection", tags=["nl_detection"])+++# ---------------------------------------------------------------------------+# Request / response models+# ---------------------------------------------------------------------------+++class NLDetectionRequest(BaseModel):+ description: str = Field(+ ...,+ description="Plain-English description of the threat behaviour to detect.",+ min_length=10,+ max_length=2000,+ )+ target_platforms: list[Literal["sigma", "kql", "spl", "esql"]] = Field(+ default=["sigma", "kql", "spl", "esql"],+ description="Which rule languages to generate.",+ )+ severity: Literal["informational", "low", "medium", "high", "critical"] = "medium"+ mitre_techniques: list[str] = Field(default_factory=list)+++class NLDetectionResponse(BaseModel):+ description: str+ sigma: str | None = None+ kql: str | None = None+ spl: str | None = None+ esql: str | None = None+ model_used: str+++# ---------------------------------------------------------------------------+# LLM translation helpers+# ---------------------------------------------------------------------------++_SYSTEM_PROMPT = """You are a detection engineering expert.+Given a plain-English threat description, generate detection rules in the+requested formats. Output ONLY valid rule content, no prose.++For Sigma: output valid YAML.+For KQL: output a valid Kusto query.+For SPL: output a valid Splunk search.+For ES|QL: output a valid Elastic ES|QL query.+"""++_TEMPLATE = {+ "sigma": """\+title: {title}+status: experimental+description: {description}+logsource:+ category: process_creation+ product: windows+detection:+ selection:+ CommandLine|contains: '<KEYWORD>'+ condition: selection+falsepositives:+ - Legitimate administration activity+level: {severity}+""",+ "kql": """\+// {title}+// {description}+SecurityEvent+| where EventID == 4688+| where CommandLine contains "<KEYWORD>"+| project TimeGenerated, Computer, Account, CommandLine+""",+ "spl": """\+// {title} — {description}+index=windows EventCode=4688+| where like(CommandLine, "%<KEYWORD>%")+| table _time, host, user, CommandLine+""",+ "esql": """\+// {title}+// {description}+FROM logs-*+| WHERE process.command_line LIKE "*<KEYWORD>*"+| KEEP @timestamp, host.name, user.name, process.command_line+""",+}+++async def _llm_translate(request: NLDetectionRequest) -> dict[str, str | None]:+ """Attempt LLM-based translation; fall back to templates on any error."""+ api_key = os.environ.get("OPENAI_API_KEY", "").strip()+ if not api_key:+ logger.debug("nl_detection.llm_unavailable", reason="no OPENAI_API_KEY")+ return _template_fallback(request)++ platforms_str = ", ".join(request.target_platforms)+ user_prompt = (+ f"Threat description: {request.description}\n\n"+ f"Severity: {request.severity}\n"+ f"MITRE techniques: {', '.join(request.mitre_techniques) or 'unknown'}\n\n"+ f"Generate detection rules for: {platforms_str}.\n"+ f"Return JSON with keys matching the platform names (sigma, kql, spl, esql)."+ )++ try:+ import httpx++ async with httpx.AsyncClient(timeout=30.0) as client:+ resp = await client.post(+ "https://api.openai.com/v1/chat/completions",+ headers={"Authorization": f"Bearer {api_key}"},+ json={+ "model": os.environ.get("OPENAI_MODEL", "gpt-4o-mini"),+ "messages": [+ {"role": "system", "content": _SYSTEM_PROMPT},+ {"role": "user", "content": user_prompt},+ ],+ "response_format": {"type": "json_object"},+ "temperature": 0.1,+ "max_tokens": 2000,+ },+ )+ resp.raise_for_status()+ import json++ content = resp.json()["choices"][0]["message"]["content"]+ rules = json.loads(content)+ return {+ "sigma": rules.get("sigma") if "sigma" in request.target_platforms else None,+ "kql": rules.get("kql") if "kql" in request.target_platforms else None,+ "spl": rules.get("spl") if "spl" in request.target_platforms else None,+ "esql": rules.get("esql") if "esql" in request.target_platforms else None,+ "_model": os.environ.get("OPENAI_MODEL", "gpt-4o-mini"),+ }+ except Exception as exc:+ logger.warning("nl_detection.llm_error", error=str(exc))+ return _template_fallback(request)+++def _template_fallback(request: NLDetectionRequest) -> dict[str, str | None]:+ title = request.description[:60].replace("\n", " ").strip()+ ctx = {"title": title, "description": request.description, "severity": request.severity}+ return {+ "sigma": _TEMPLATE["sigma"].format(**ctx) if "sigma" in request.target_platforms else None,+ "kql": _TEMPLATE["kql"].format(**ctx) if "kql" in request.target_platforms else None,+ "spl": _TEMPLATE["spl"].format(**ctx) if "spl" in request.target_platforms else None,+ "esql": _TEMPLATE["esql"].format(**ctx) if "esql" in request.target_platforms else None,+ "_model": "template",+ }+++# ---------------------------------------------------------------------------+# Endpoint+# ---------------------------------------------------------------------------++[email protected]("/translate", response_model=NLDetectionResponse)+async def translate_detection(payload: NLDetectionRequest) -> NLDetectionResponse:+ """Convert a plain-English threat description into multi-platform detection rules."""+ if not payload.description.strip():+ raise HTTPException(+ status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,+ detail="description must not be empty",+ )++ rules = await _llm_translate(payload)+ model_used = rules.pop("_model", "unknown")++ logger.info(+ "nl_detection.translate",+ platforms=payload.target_platforms,+ model=model_used,+ description_len=len(payload.description),+ )++ return NLDetectionResponse(+ description=payload.description,+ sigma=rules.get("sigma"),+ kql=rules.get("kql"),+ spl=rules.get("spl"),+ esql=rules.get("esql"),+ model_used=model_used,+ )diff --git a/services/api/app/api/v1/endpoints/nl_query.py b/services/api/app/api/v1/endpoints/nl_query.pynew file mode 100644index 000000000..d94e01478--- /dev/null+++ b/services/api/app/api/v1/endpoints/nl_query.py@@ -0,0 +1,283 @@+"""Natural-language query → multi-dialect execution (tier2-nl-query).++Accepts a plain-English security question, translates it to ES|QL, SPL, and+KQL, then (optionally) executes the ES|QL variant against a connected+Elasticsearch cluster and returns structured results ready for charting.++Endpoints+---------+* ``POST /nl-query/translate`` Translate NL → ES|QL / SPL / KQL.+* ``POST /nl-query/execute`` Translate + execute against Elasticsearch.+"""++from __future__ import annotations++import json+import textwrap+import uuid+from datetime import UTC, datetime+from typing import Any++import httpx+from fastapi import APIRouter, HTTPException, status+from pydantic import BaseModel, Field++from app.api.v1.deps import AuthUser+from app.core.config import settings++router = APIRouter(prefix="/nl-query", tags=["nl_query"])++# ────────────────────────────────────────────────────────────────────────────+# Pydantic schemas+# ────────────────────────────────────────────────────────────────────────────+++class NLQueryTranslateRequest(BaseModel):+ question: str = Field(+ ...,+ min_length=10,+ description="Plain-English security question (e.g. 'Show failed logins per user in the last 24 h').",+ )+ index_pattern: str = Field(+ "logs-*,aisoc-events-*",+ description="Elasticsearch index pattern to scope the ES|QL query.",+ )+ time_range_hours: int = Field(+ 24,+ ge=1,+ le=8760,+ description="Look-back window in hours.",+ )+++class NLQueryTranslateResponse(BaseModel):+ request_id: uuid.UUID+ question: str+ esql: str+ spl: str+ kql: str+ explanation: str+ created_at: datetime+++class NLQueryExecuteRequest(NLQueryTranslateRequest):+ es_url: str | None = Field(+ None,+ description="Override Elasticsearch URL (defaults to settings.ES_URL if set).",+ )+ es_api_key: str | None = Field(+ None,+ description="Override ES API key (defaults to settings.ES_API_KEY if set).",+ )+ max_rows: int = Field(500, ge=1, le=5000)+++class QueryResult(BaseModel):+ columns: list[str]+ rows: list[list[Any]]+ total_rows: int+ took_ms: int | None = None+++class NLQueryExecuteResponse(NLQueryTranslateResponse):+ result: QueryResult | None = None+ execution_error: str | None = None+++# ────────────────────────────────────────────────────────────────────────────+# LLM translation helper+# ────────────────────────────────────────────────────────────────────────────++_SYS_PROMPT = textwrap.dedent(+ """+ You are a senior security data engineer. The user will provide:+ - A plain-English security question.+ - The Elasticsearch index pattern and time range.++ Translate the question into three query dialects and return JSON only:+ {+ "esql": "...", // Elasticsearch ES|QL query+ "spl": "...", // Splunk SPL query+ "kql": "...", // Microsoft Sentinel KQL query+ "explanation": "..." // 1–2 sentences describing what the query does+ }++ Guidelines:+ - Use `FROM <index_pattern>` and `| WHERE @timestamp > NOW() - <hours>h` for ES|QL.+ - Prefer `STATS ... BY ...` for aggregations in ES|QL.+ - Map common field names: user → user.name, src_ip → source.ip, dest_ip → destination.ip.+ - Keep queries concise; add a `| LIMIT 500` to ES|QL.+ """+).strip()+++async def _translate(+ question: str, index_pattern: str, time_range_hours: int+) -> dict[str, str]:+ api_key = getattr(settings, "OPENAI_API_KEY", None) or getattr(+ settings, "LLM_API_KEY", None+ )+ if not api_key:+ return _template_fallback(question, index_pattern, time_range_hours)++ user_msg = json.dumps(+ {+ "question": question,+ "index_pattern": index_pattern,+ "time_range_hours": time_range_hours,+ }+ )+ try:+ async with httpx.AsyncClient(timeout=30) as client:+ resp = await client.post(+ "https://api.openai.com/v1/chat/completions",+ headers={"Authorization": f"Bearer {api_key}"},+ json={+ "model": "gpt-4o-mini",+ "response_format": {"type": "json_object"},+ "messages": [+ {"role": "system", "content": _SYS_PROMPT},+ {"role": "user", "content": user_msg},+ ],+ },+ )+ resp.raise_for_status()+ return json.loads(resp.json()["choices"][0]["message"]["content"])+ except Exception:+ return _template_fallback(question, index_pattern, time_range_hours)+++def _template_fallback(+ question: str, index_pattern: str, time_range_hours: int+) -> dict[str, str]:+ return {+ "esql": (+ f"FROM {index_pattern}\n"+ f"| WHERE @timestamp > NOW() - {time_range_hours}h\n"+ f"// TODO: translate → {question}\n"+ "| LIMIT 500"+ ),+ "spl": (+ f"index=* earliest=-{time_range_hours}h\n"+ f"// TODO: translate → {question}"+ ),+ "kql": (+ f"// TODO: translate → {question}\n"+ f"| where TimeGenerated > ago({time_range_hours}h)"+ ),+ "explanation": f"Template placeholder for: {question}",+ }+++# ────────────────────────────────────────────────────────────────────────────+# Elasticsearch execution helper+# ────────────────────────────────────────────────────────────────────────────+++async def _execute_esql(+ esql: str, es_url: str, es_api_key: str, max_rows: int+) -> QueryResult:+ """Run an ES|QL query against Elasticsearch and return structured results."""+ # Ensure LIMIT clause+ query = esql if "| LIMIT" in esql.upper() else f"{esql}\n| LIMIT {max_rows}"+ import time++ t0 = time.monotonic()+ async with httpx.AsyncClient(timeout=20) as client:+ resp = await client.post(+ f"{es_url.rstrip('/')}/_query",+ headers={+ "Authorization": f"ApiKey {es_api_key}",+ "Content-Type": "application/json",+ },+ json={"query": query},+ )+ resp.raise_for_status()+ data = resp.json()++ took_ms = int((time.monotonic() - t0) * 1000)+ columns = [col["name"] for col in data.get("columns", [])]+ rows = data.get("values", [])+ return QueryResult(columns=columns, rows=rows, total_rows=len(rows), took_ms=took_ms)+++# ────────────────────────────────────────────────────────────────────────────+# Endpoints+# ────────────────────────────────────────────────────────────────────────────+++ "/translate",+ response_model=NLQueryTranslateResponse,+ status_code=status.HTTP_200_OK,+ summary="Translate a natural-language security question to ES|QL / SPL / KQL",+)+async def translate_query(+ body: NLQueryTranslateRequest,+ user: AuthUser,+) -> NLQueryTranslateResponse:+ translated = await _translate(+ body.question, body.index_pattern, body.time_range_hours+ )+ return NLQueryTranslateResponse(+ request_id=uuid.uuid4(),+ question=body.question,+ esql=translated.get("esql", ""),+ spl=translated.get("spl", ""),+ kql=translated.get("kql", ""),+ explanation=translated.get("explanation", ""),+ created_at=datetime.now(UTC),+ )+++ "/execute",+ response_model=NLQueryExecuteResponse,+ status_code=status.HTTP_200_OK,+ summary="Translate NL question and execute ES|QL against Elasticsearch",+)+async def execute_query(+ body: NLQueryExecuteRequest,+ user: AuthUser,+) -> NLQueryExecuteResponse:+ translated = await _translate(+ body.question, body.index_pattern, body.time_range_hours+ )++ es_url = (+ body.es_url+ or getattr(settings, "ES_URL", None)+ or getattr(settings, "ELASTICSEARCH_URL", None)+ )+ es_api_key = body.es_api_key or getattr(settings, "ES_API_KEY", None)++ base = NLQueryExecuteResponse(+ request_id=uuid.uuid4(),+ question=body.question,+ esql=translated.get("esql", ""),+ spl=translated.get("spl", ""),+ kql=translated.get("kql", ""),+ explanation=translated.get("explanation", ""),+ created_at=datetime.now(UTC),+ )++ if not es_url or not es_api_key:+ base.execution_error = (+ "ES_URL or ES_API_KEY not configured. "+ "Set them in environment variables or pass in the request body."+ )+ return base++ try:+ base.result = await _execute_esql(+ translated.get("esql", ""),+ es_url=es_url,+ es_api_key=es_api_key,+ max_rows=body.max_rows,+ )+ except httpx.HTTPStatusError as exc:+ base.execution_error = f"ES query failed ({exc.response.status_code}): {exc.response.text[:500]}"+ except Exception as exc:+ base.execution_error = str(exc)++ return basediff --git a/services/api/app/api/v1/endpoints/phishing.py b/services/api/app/api/v1/endpoints/phishing.pynew file mode 100644index 000000000..32dbda789--- /dev/null+++ b/services/api/app/api/v1/endpoints/phishing.py@@ -0,0 +1,265 @@+"""Email-security + phishing-triage workflow (tier3-phishing).++Analysts or automated ingestion submit raw email text, URLs, attachments, or+domain names for LLM-powered triage. The endpoint extracts indicators of+compromise (IOCs), assigns a verdict, and optionally opens a case.++Endpoints+---------+* ``POST /phishing/submit`` Submit an artifact for triage.+* ``GET /phishing/submissions`` List submissions.+* ``GET /phishing/{id}`` Get a submission.+* ``POST /phishing/{id}/retriage`` Re-run triage (e.g. after analyst correction).+"""++from __future__ import annotations++import json+import os+import uuid+from datetime import UTC, datetime+from typing import Any, Literal++import httpx+from fastapi import APIRouter, HTTPException, Query, status+from pydantic import BaseModel, Field+from sqlalchemy import text++from app.api.v1.deps import AuthUser, DBSession++router = APIRouter(prefix="/phishing", tags=["phishing"])++# ────────────────────────────────────────────────────────────────────────────+# Schemas+# ────────────────────────────────────────────────────────────────────────────++ArtifactKind = Literal["email", "url", "attachment", "domain"]+Verdict = Literal["pending", "benign", "phishing", "spam", "malware", "unknown"]+++class SubmitRequest(BaseModel):+ artifact_kind: ArtifactKind = "email"+ raw_content: str | None = Field(None, description="Raw email source or URL to analyse.")+ sender: str | None = None+ subject: str | None = None+ urls: list[str] = Field(default_factory=list)+++class TriageResult(BaseModel):+ verdict: Verdict+ confidence: float+ indicators: list[dict[str, Any]]+ mitre_technique: str | None+ summary: str+++class SubmissionResponse(BaseModel):+ id: uuid.UUID+ artifact_kind: str+ sender: str | None+ subject: str | None+ urls: list[str]+ verdict: str+ confidence: float | None+ indicators: list[dict[str, Any]]+ mitre_technique: str | None+ case_id: uuid.UUID | None+ submitted_at: datetime+ triaged_at: datetime | None+++# ────────────────────────────────────────────────────────────────────────────+# LLM triage helper+# ────────────────────────────────────────────────────────────────────────────++_SYSTEM = """You are a phishing and email-security analyst.+Analyse the submitted artifact and return ONLY valid JSON with:+{+ "verdict": "benign|phishing|spam|malware|unknown",+ "confidence": 0.0-1.0,+ "indicators": [{"kind": "url|domain|ip|hash|email|header", "value": "...", "note": "..."}],+ "mitre_technique": "T1566.001 or null",+ "summary": "one-sentence explanation"+}"""+++async def _triage(artifact_kind: str, content: str, urls: list[str]) -> TriageResult | None:+ api_key = os.getenv("OPENAI_API_KEY") or os.getenv("LLM_API_KEY")+ if not api_key:+ return None+ base_url = os.getenv("LLM_BASE_URL", "https://api.openai.com/v1")+ model = os.getenv("LLM_MODEL", "gpt-4o-mini")+ user_msg = f"ARTIFACT TYPE: {artifact_kind}\n"+ if urls:+ user_msg += f"URLS: {', '.join(urls[:10])}\n"+ if content:+ user_msg += f"CONTENT (first 2000 chars):\n{content[:2000]}"+ try:+ async with httpx.AsyncClient(timeout=45) as client:+ resp = await client.post(+ f"{base_url}/chat/completions",+ headers={"Authorization": f"Bearer {api_key}"},+ json={+ "model": model,+ "messages": [+ {"role": "system", "content": _SYSTEM},+ {"role": "user", "content": user_msg},+ ],+ "temperature": 0.1,+ "response_format": {"type": "json_object"},+ },+ )+ resp.raise_for_status()+ data = json.loads(resp.json()["choices"][0]["message"]["content"])+ return TriageResult(+ verdict=data.get("verdict", "unknown"),+ confidence=float(data.get("confidence", 0.5)),+ indicators=data.get("indicators", []),+ mitre_technique=data.get("mitre_technique"),+ summary=data.get("summary", ""),+ )+ except Exception:+ return None+++def _heuristic_triage(content: str | None, urls: list[str]) -> TriageResult:+ """Fallback rule-based triage when no LLM key is configured."""+ indicators: list[dict[str, Any]] = []+ score = 0.0+ phishing_words = ["verify your account", "click here", "urgent", "suspended", "password reset", "login immediately"]+ if content:+ for kw in phishing_words:+ if kw.lower() in (content or "").lower():+ score += 0.15+ indicators.append({"kind": "keyword", "value": kw, "note": "phishing keyword"})+ for url in urls:+ if any(x in url for x in ["bit.ly", "tinyurl", "goo.gl", "ow.ly"]):+ score += 0.2+ indicators.append({"kind": "url", "value": url, "note": "URL shortener"})+ score = min(score, 1.0)+ verdict: Verdict = "phishing" if score > 0.4 else "benign" if score < 0.1 else "unknown"+ return TriageResult(verdict=verdict, confidence=round(score, 2), indicators=indicators, mitre_technique=None, summary="Heuristic triage — LLM not configured.")+++def _row_to_submission(row: Any) -> SubmissionResponse:+ return SubmissionResponse(+ id=row.id,+ artifact_kind=row.artifact_kind,+ sender=row.sender,+ subject=row.subject,+ urls=list(row.urls or []),+ verdict=row.verdict,+ confidence=row.confidence,+ indicators=list(row.indicators or []),+ mitre_technique=row.mitre_technique,+ case_id=row.case_id,+ submitted_at=row.submitted_at,+ triaged_at=row.triaged_at,+ )+++# ────────────────────────────────────────────────────────────────────────────+# Endpoints+# ────────────────────────────────────────────────────────────────────────────++[email protected]("/submit", response_model=SubmissionResponse, status_code=status.HTTP_201_CREATED, summary="Submit artifact for phishing triage")+async def submit(body: SubmitRequest, db: DBSession, user: AuthUser) -> SubmissionResponse:+ result = await _triage(body.artifact_kind, body.raw_content or "", body.urls)+ if not result:+ result = _heuristic_triage(body.raw_content, body.urls)++ now = datetime.now(UTC)+ sub_id = uuid.uuid4()+ q = text("""+ INSERT INTO aisoc_phishing_submissions (+ id, submitted_by, artifact_kind, raw_content, sender, subject,+ urls, verdict, confidence, indicators, mitre_technique,+ submitted_at, triaged_at, created_at+ ) VALUES (+ :id, :by, :kind, :content, :sender, :subject,+ :urls::text[], :verdict, :conf, :iocs::jsonb, :mitre,+ :now, :now, :now+ ) RETURNING *+ """).bindparams(+ id=sub_id,+ by=str(user) if user else "system",+ kind=body.artifact_kind,+ content=body.raw_content,+ sender=body.sender,+ subject=body.subject,+ urls=body.urls or [],+ verdict=result.verdict,+ conf=result.confidence,+ iocs=json.dumps(result.indicators),+ mitre=result.mitre_technique,+ now=now,+ )+ try:+ row = (await db.execute(q)).fetchone()+ await db.commit()+ return _row_to_submission(row)+ except Exception as exc:+ await db.rollback()+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc++[email protected]("/submissions", response_model=list[SubmissionResponse], summary="List phishing submissions")+async def list_submissions(+ db: DBSession,+ user: AuthUser,+ verdict: str | None = Query(None),+ limit: int = Query(50, ge=1, le=500),+ offset: int = Query(0, ge=0),+) -> list[SubmissionResponse]:+ wheres = ["1=1"]+ params: dict[str, Any] = {"limit": limit, "offset": offset}+ if verdict:+ wheres.append("verdict = :verdict"); params["verdict"] = verdict+ q = text(f"SELECT * FROM aisoc_phishing_submissions WHERE {' AND '.join(wheres)} ORDER BY submitted_at DESC LIMIT :limit OFFSET :offset").bindparams(**params)+ try:+ rows = (await db.execute(q)).fetchall()+ return [_row_to_submission(r) for r in rows]+ except Exception as exc:+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc++[email protected]("/{submission_id}", response_model=SubmissionResponse, summary="Get submission")+async def get_submission(submission_id: uuid.UUID, db: DBSession, user: AuthUser) -> SubmissionResponse:+ row = (await db.execute(text("SELECT * FROM aisoc_phishing_submissions WHERE id = :id").bindparams(id=submission_id))).fetchone()+ if not row:+ raise HTTPException(status_code=404, detail="Submission not found.")+ return _row_to_submission(row)++[email protected]("/{submission_id}/retriage", response_model=SubmissionResponse, summary="Re-run triage on submission")+async def retriage(submission_id: uuid.UUID, db: DBSession, user: AuthUser) -> SubmissionResponse:+ existing = (await db.execute(text("SELECT * FROM aisoc_phishing_submissions WHERE id = :id").bindparams(id=submission_id))).fetchone()+ if not existing:+ raise HTTPException(status_code=404, detail="Submission not found.")++ result = await _triage(existing.artifact_kind, existing.raw_content or "", list(existing.urls or []))+ if not result:+ result = _heuristic_triage(existing.raw_content, list(existing.urls or []))++ now = datetime.now(UTC)+ q = text("""+ UPDATE aisoc_phishing_submissions+ SET verdict = :verdict, confidence = :conf, indicators = :iocs::jsonb,+ mitre_technique = :mitre, triaged_at = :now+ WHERE id = :id RETURNING *+ """).bindparams(+ id=submission_id,+ verdict=result.verdict,+ conf=result.confidence,+ iocs=json.dumps(result.indicators),+ mitre=result.mitre_technique,+ now=now,+ )+ try:+ row = (await db.execute(q)).fetchone()+ await db.commit()+ return _row_to_submission(row)+ except Exception as exc:+ await db.rollback()+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from excdiff --git a/services/api/app/api/v1/endpoints/translation.py b/services/api/app/api/v1/endpoints/translation.pynew file mode 100644index 000000000..5632fcd22--- /dev/null+++ b/services/api/app/api/v1/endpoints/translation.py@@ -0,0 +1,258 @@+"""Cross-platform detection rule translation (tier2-translation).++AST-aware translator between Sigma YAML, Splunk SPL, Microsoft Sentinel KQL,+Google Chronicle YARA-L2/UDM, and Elastic ES|QL. Uses an LLM for high-fidelity+translation when ``OPENAI_API_KEY`` is set; falls back to field-map substitution+for common patterns when no key is configured.++Endpoints+---------+* ``POST /translation/translate`` Translate a rule from one format to multiple targets.+* ``GET /translation/formats`` List supported source/target formats.+"""++from __future__ import annotations++import json+import os+from typing import Any, Literal++import httpx+from fastapi import APIRouter, HTTPException, status+from pydantic import BaseModel, Field++router = APIRouter(prefix="/translation", tags=["translation"])++# ────────────────────────────────────────────────────────────────────────────+# Supported formats+# ────────────────────────────────────────────────────────────────────────────++DetectionFormat = Literal["sigma", "spl", "kql", "esql", "yara_l2", "udm"]++_FORMAT_LABELS: dict[DetectionFormat, str] = {+ "sigma": "Sigma YAML",+ "spl": "Splunk SPL",+ "kql": "Microsoft Sentinel KQL (OOTB & Analytics Rules)",+ "esql": "Elastic ES|QL",+ "yara_l2": "Google Chronicle YARA-L2",+ "udm": "Google Chronicle UDM Search",+}++# ────────────────────────────────────────────────────────────────────────────+# Pydantic schemas+# ────────────────────────────────────────────────────────────────────────────+++class TranslateRequest(BaseModel):+ source_format: DetectionFormat = Field(..., description="Format of the input rule.")+ source_rule: str = Field(..., min_length=5, description="The raw rule text to translate.")+ target_formats: list[DetectionFormat] = Field(+ default=["sigma", "spl", "kql", "esql"],+ description="Output formats to produce.",+ )+++class TranslationResult(BaseModel):+ format: DetectionFormat+ label: str+ rule: str+ notes: str | None = None+++class TranslateResponse(BaseModel):+ source_format: DetectionFormat+ results: list[TranslationResult]+ warnings: list[str] = []+++class FormatsResponse(BaseModel):+ formats: dict[str, str]+++# ────────────────────────────────────────────────────────────────────────────+# LLM helper+# ────────────────────────────────────────────────────────────────────────────++_SYSTEM = """You are an expert security engineer specialising in detection rule translation.+You translate detection rules between Sigma YAML, Splunk SPL, Microsoft Sentinel KQL,+Elastic ES|QL, Google Chronicle YARA-L2, and Google Chronicle UDM Search.++Rules for translation:+- Preserve ALL detection logic precisely (field names, operators, values, threshold).+- Map field names using OCSF / ECS / Sigma field conventions to each platform's native fields.+- Add platform-appropriate pipeline/index/timeframe clauses.+- Return ONLY valid JSON. No prose before or after.++Return JSON with this exact structure:+{+ "results": [+ {"format": "<format>", "rule": "<rule_text>", "notes": "<optional notes>"},+ ...+ ],+ "warnings": ["<any caveats>"]+}+"""+++def _user_prompt(req: TranslateRequest) -> str:+ targets = ", ".join(req.target_formats)+ return (+ f"SOURCE FORMAT: {req.source_format}\n\n"+ f"SOURCE RULE:\n```\n{req.source_rule}\n```\n\n"+ f"Translate the above rule to: {targets}.\n"+ "Return valid JSON only."+ )+++async def _llm_translate(req: TranslateRequest) -> dict[str, Any] | None:+ api_key = os.getenv("OPENAI_API_KEY") or os.getenv("LLM_API_KEY")+ if not api_key:+ return None+ base_url = os.getenv("LLM_BASE_URL", "https://api.openai.com/v1")+ model = os.getenv("LLM_MODEL", "gpt-4o-mini")+ try:+ async with httpx.AsyncClient(timeout=60) as client:+ resp = await client.post(+ f"{base_url}/chat/completions",+ headers={"Authorization": f"Bearer {api_key}"},+ json={+ "model": model,+ "messages": [+ {"role": "system", "content": _SYSTEM},+ {"role": "user", "content": _user_prompt(req)},+ ],+ "temperature": 0.1,+ "response_format": {"type": "json_object"},+ },+ )+ resp.raise_for_status()+ content = resp.json()["choices"][0]["message"]["content"]+ return json.loads(content)+ except Exception:+ return None+++# ────────────────────────────────────────────────────────────────────────────+# Template-based fallback (field-map substitution)+# ────────────────────────────────────────────────────────────────────────────++# Sigma → target field map (common ECS / OCSF fields)+_FIELD_MAP: dict[str, dict[str, str]] = {+ "spl": {+ "EventID": "EventCode",+ "Image": "process_name",+ "CommandLine": "process_path",+ "TargetUserName": "user",+ "DestinationIp": "dest_ip",+ "SourceIp": "src_ip",+ },+ "kql": {+ "EventID": "EventID",+ "Image": "NewProcessName",+ "CommandLine": "CommandLine",+ "TargetUserName": "TargetUserName",+ "DestinationIp": "RemoteIP",+ "SourceIp": "InitiatingProcessRemoteIP",+ },+ "esql": {+ "EventID": "event.code",+ "Image": "process.executable",+ "CommandLine": "process.command_line",+ "TargetUserName": "user.name",+ "DestinationIp": "destination.ip",+ "SourceIp": "source.ip",+ },+ "yara_l2": {+ "EventID": "metadata.product_event_type",+ "Image": "principal.process.file.full_path",+ "CommandLine": "principal.process.command_line",+ "TargetUserName": "target.user.userid",+ "DestinationIp": "target.ip",+ "SourceIp": "principal.ip",+ },+ "udm": {+ "EventID": "metadata.product_event_type",+ "Image": "principal.process.file.full_path",+ "CommandLine": "principal.process.command_line",+ "TargetUserName": "target.user.userid",+ "DestinationIp": "target.ip",+ "SourceIp": "principal.ip",+ },+}+++def _fallback_templates(req: TranslateRequest) -> dict[str, Any]:+ """Produce best-effort template translations without an LLM."""+ results = []+ warnings = ["LLM not configured — using field-map template substitution. Review output carefully."]++ for fmt in req.target_formats:+ if fmt == req.source_format:+ results.append({"format": fmt, "rule": req.source_rule, "notes": "Same as source."})+ continue++ rule = req.source_rule+ field_map = _FIELD_MAP.get(fmt, {})+ for src_field, tgt_field in field_map.items():+ rule = rule.replace(src_field, tgt_field)++ if fmt == "spl":+ rule = f"index=* sourcetype=WinEventLog\n| search {rule}"+ elif fmt == "kql":+ rule = f"SecurityEvent\n| where {rule}"+ elif fmt == "esql":+ rule = f"FROM logs-*\n| WHERE {rule}"+ elif fmt == "yara_l2":+ rule = f"rule translated_rule {{\n meta:\n author = \"AiSOC\"\n condition:\n {rule}\n}}"+ elif fmt == "udm":+ rule = f"// Chronicle UDM Search\n{rule}"++ results.append({"format": fmt, "rule": rule, "notes": "Template translation — verify field names."})++ return {"results": results, "warnings": warnings}+++# ────────────────────────────────────────────────────────────────────────────+# Endpoints+# ────────────────────────────────────────────────────────────────────────────+++ "/translate",+ response_model=TranslateResponse,+ status_code=status.HTTP_200_OK,+ summary="Translate a detection rule across formats",+)+async def translate_rule(body: TranslateRequest) -> TranslateResponse:+ if not body.target_formats:+ raise HTTPException(status_code=400, detail="At least one target_format is required.")++ payload = await _llm_translate(body)+ if not payload:+ payload = _fallback_templates(body)++ results = [+ TranslationResult(+ format=r["format"],+ label=_FORMAT_LABELS.get(r["format"], r["format"]),+ rule=r["rule"],+ notes=r.get("notes"),+ )+ for r in payload.get("results", [])+ if r.get("format") and r.get("rule")+ ]++ return TranslateResponse(+ source_format=body.source_format,+ results=results,+ warnings=payload.get("warnings", []),+ )+++ "/formats",+ response_model=FormatsResponse,+ summary="List supported detection rule formats",+)+async def list_formats() -> FormatsResponse:+ return FormatsResponse(formats=_FORMAT_LABELS) # type: ignore[arg-type]diff --git a/services/api/app/api/v1/router.py b/services/api/app/api/v1/router.pyindex 03a360043..3a764df4f 100644--- a/services/api/app/api/v1/router.py+++ b/services/api/app/api/v1/router.py@@ -13,9 +13,16 @@ from app.api.v1.endpoints import (community,compliance,connectors,+ detection_loop,detection_proposals,detection_rules,+ feedback,federated,+ hunts,+ identity_timeline,+ knowledge_base,+ nl_detection,+ nl_query,graph,identity_graph,insider_threat,@@ -25,6 +32,7 @@ from app.api.v1.endpoints import (mssp,oncall,passkeys,+ phishing,playbooks,plugins,posture,@@ -35,6 +43,7 @@ from app.api.v1.endpoints import (sla,tenants,threat_intel,+ translation,)api_router = APIRouter(prefix="/api/v1")@@ -72,6 +81,33 @@ api_router.include_router(mssp.router)api_router.include_router(insider_threat.router)api_router.include_router(remediation.router)+# Analyst feedback loop+api_router.include_router(feedback.router)++# NL detection authoring (Tier 2)+api_router.include_router(nl_detection.router)++# Closed-loop detection engineering: FP → LLM Sigma draft → DAC proposal (Tier 2)+api_router.include_router(detection_loop.router)++# Natural-language query → ES|QL / SPL / KQL translation + execution (Tier 2)+api_router.include_router(nl_query.router)++# Identity-centric investigation timeline (Tier 2)+api_router.include_router(identity_timeline.router)++# Cross-platform detection rule translation: Sigma↔SPL↔KQL↔UDM↔ES|QL (Tier 2)+api_router.include_router(translation.router)++# Hypothesis-driven hunt workbench (Tier 2)+api_router.include_router(hunts.router)++# Email-security + phishing-triage workflow (Tier 3)+api_router.include_router(phishing.router)++# Knowledge-base + RAG over org docs/runbooks (Tier 3)+api_router.include_router(knowledge_base.router)+# Wave 4 — advanced capabilitiesapi_router.include_router(threat_intel.router)api_router.include_router(posture.router)diff --git a/services/api/app/models/alert.py b/services/api/app/models/alert.pyindex ca29d5ab0..d3bd582b6 100644--- a/services/api/app/models/alert.py+++ b/services/api/app/models/alert.py@@ -34,6 +34,11 @@ class Alert(Base):source_event_ids: Mapped[list] = mapped_column(JSONB, default=list)ocsf_class_uid: Mapped[int | None] = mapped_column(Integer, nullable=True)+ # Analyst verdict / disposition (set via feedback endpoint)+ disposition: Mapped[str | None] = mapped_column(String(50), nullable=True, index=True)+ # Timestamp when an analyst first viewed / started triaging the alert (for MTTD)+ first_seen_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)+# AI/MLai_score: Mapped[float | None] = mapped_column(Float, nullable=True)ai_summary: Mapped[str | None] = mapped_column(Text, nullable=True)diff --git a/services/api/migrations/012_case_management.sql b/services/api/migrations/012_case_management.sqlnew file mode 100644index 000000000..55c40bb93--- /dev/null+++ b/services/api/migrations/012_case_management.sql@@ -0,0 +1,48 @@+-- Migration 012: First-class case management+-- Adds cases table with lifecycle state machine, observable graph, and evidence chain.++CREATE TABLE IF NOT EXISTS aisoc_cases (+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),+ tenant_id UUID,+ title TEXT NOT NULL,+ description TEXT,+ severity TEXT NOT NULL DEFAULT 'medium'+ CHECK (severity IN ('info','low','medium','high','critical')),+ status TEXT NOT NULL DEFAULT 'new'+ CHECK (status IN ('new','triaged','investigating','contained','resolved','closed')),+ assignee TEXT,+ -- ATT&CK coverage+ mitre_techniques JSONB DEFAULT '[]'::jsonb,+ -- evidence & observables+ alert_ids UUID[] DEFAULT ARRAY[]::UUID[],+ observable_graph JSONB DEFAULT '{}'::jsonb, -- nodes + edges+ evidence_chain JSONB DEFAULT '[]'::jsonb, -- ordered list of evidence items+ -- compliance+ compliance_frameworks TEXT[] DEFAULT ARRAY[]::TEXT[],+ -- timeline+ opened_at TIMESTAMPTZ NOT NULL DEFAULT now(),+ triaged_at TIMESTAMPTZ,+ resolved_at TIMESTAMPTZ,+ closed_at TIMESTAMPTZ,+ -- metadata+ created_at TIMESTAMPTZ NOT NULL DEFAULT now(),+ updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),+ created_by TEXT,+ tags JSONB DEFAULT '{}'::jsonb,+ sla_due_at TIMESTAMPTZ+);++-- Case comments / timeline entries+CREATE TABLE IF NOT EXISTS aisoc_case_comments (+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),+ case_id UUID NOT NULL REFERENCES aisoc_cases(id) ON DELETE CASCADE,+ author TEXT,+ body TEXT NOT NULL,+ is_system BOOLEAN NOT NULL DEFAULT FALSE,+ created_at TIMESTAMPTZ NOT NULL DEFAULT now()+);++CREATE INDEX IF NOT EXISTS idx_aisoc_cases_tenant ON aisoc_cases (tenant_id);+CREATE INDEX IF NOT EXISTS idx_aisoc_cases_status ON aisoc_cases (status);+CREATE INDEX IF NOT EXISTS idx_aisoc_cases_severity ON aisoc_cases (severity);+CREATE INDEX IF NOT EXISTS idx_aisoc_case_comments ON aisoc_case_comments (case_id);diff --git a/services/api/migrations/013_compliance_evidence.sql b/services/api/migrations/013_compliance_evidence.sqlnew file mode 100644index 000000000..8cc91e50c--- /dev/null+++ b/services/api/migrations/013_compliance_evidence.sql@@ -0,0 +1,29 @@+-- Migration 013: Compliance evidence trails+-- Audit-grade evidence records linked to cases with hash-chain integrity.++CREATE TABLE IF NOT EXISTS aisoc_compliance_evidence (+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),+ case_id UUID REFERENCES aisoc_cases(id) ON DELETE SET NULL,+ tenant_id UUID,+ framework TEXT NOT NULL, -- SOC2, PCI-DSS, HIPAA, ISO27001, NIST-CSF …+ control_id TEXT NOT NULL, -- e.g. CC6.1, A.12.4.1+ control_title TEXT,+ evidence_kind TEXT NOT NULL DEFAULT 'alert'+ CHECK (evidence_kind IN ('alert','log','screenshot','attestation','policy','runbook','other')),+ summary TEXT NOT NULL,+ raw_payload JSONB DEFAULT '{}'::jsonb,+ -- SHA-256 of (prev_hash || summary || raw_payload) for chain integrity+ payload_hash TEXT,+ prev_hash TEXT,+ -- lifecycle+ collected_at TIMESTAMPTZ NOT NULL DEFAULT now(),+ reviewed_by TEXT,+ reviewed_at TIMESTAMPTZ,+ status TEXT NOT NULL DEFAULT 'pending'+ CHECK (status IN ('pending','accepted','rejected')),+ created_at TIMESTAMPTZ NOT NULL DEFAULT now()+);++CREATE INDEX IF NOT EXISTS idx_aisoc_compliance_framework ON aisoc_compliance_evidence (framework);+CREATE INDEX IF NOT EXISTS idx_aisoc_compliance_case ON aisoc_compliance_evidence (case_id);+CREATE INDEX IF NOT EXISTS idx_aisoc_compliance_control ON aisoc_compliance_evidence (control_id);diff --git a/services/api/migrations/014_hunt_workbench.sql b/services/api/migrations/014_hunt_workbench.sqlnew file mode 100644index 000000000..c20b88084--- /dev/null+++ b/services/api/migrations/014_hunt_workbench.sql@@ -0,0 +1,44 @@+-- Migration 014: Hypothesis-driven hunt workbench+-- Stores hunt hypotheses, linked detections, and execution results.++CREATE TABLE IF NOT EXISTS aisoc_hunts (+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),+ tenant_id UUID,+ title TEXT NOT NULL,+ hypothesis TEXT NOT NULL,+ mitre_tactic TEXT,+ mitre_technique TEXT,+ status TEXT NOT NULL DEFAULT 'draft'+ CHECK (status IN ('draft','active','completed','archived')),+ priority TEXT NOT NULL DEFAULT 'medium'+ CHECK (priority IN ('low','medium','high','critical')),+ -- linked content+ query_esql TEXT,+ query_spl TEXT,+ query_kql TEXT,+ -- results+ findings JSONB DEFAULT '[]'::jsonb,+ false_positive_rate FLOAT,+ -- metadata+ assigned_to TEXT,+ tags TEXT[] DEFAULT ARRAY[]::TEXT[],+ created_at TIMESTAMPTZ NOT NULL DEFAULT now(),+ updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),+ completed_at TIMESTAMPTZ,+ created_by TEXT+);++CREATE TABLE IF NOT EXISTS aisoc_hunt_runs (+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),+ hunt_id UUID NOT NULL REFERENCES aisoc_hunts(id) ON DELETE CASCADE,+ run_at TIMESTAMPTZ NOT NULL DEFAULT now(),+ platform TEXT NOT NULL DEFAULT 'esql',+ query_used TEXT,+ hit_count INT DEFAULT 0,+ result_sample JSONB DEFAULT '[]'::jsonb,+ duration_ms INT,+ error TEXT+);++CREATE INDEX IF NOT EXISTS idx_aisoc_hunts_status ON aisoc_hunts (status);+CREATE INDEX IF NOT EXISTS idx_aisoc_hunt_runs_hunt ON aisoc_hunt_runs (hunt_id);diff --git a/services/api/migrations/015_phishing_triage.sql b/services/api/migrations/015_phishing_triage.sqlnew file mode 100644index 000000000..fb6652dcf--- /dev/null+++ b/services/api/migrations/015_phishing_triage.sql@@ -0,0 +1,31 @@+-- Migration 015: Email-security + phishing-triage workflow+-- Stores submitted email/URL artifacts and triage verdicts.++CREATE TABLE IF NOT EXISTS aisoc_phishing_submissions (+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),+ tenant_id UUID,+ submitted_by TEXT,+ -- artifact+ artifact_kind TEXT NOT NULL DEFAULT 'email'+ CHECK (artifact_kind IN ('email','url','attachment','domain')),+ raw_content TEXT,+ sender TEXT,+ subject TEXT,+ urls TEXT[] DEFAULT ARRAY[]::TEXT[],+ attachments JSONB DEFAULT '[]'::jsonb,+ -- triage+ verdict TEXT DEFAULT 'pending'+ CHECK (verdict IN ('pending','benign','phishing','spam','malware','unknown')),+ confidence FLOAT,+ indicators JSONB DEFAULT '[]'::jsonb, -- IOCs extracted+ mitre_technique TEXT,+ -- case linkage+ case_id UUID REFERENCES aisoc_cases(id) ON DELETE SET NULL,+ -- metadata+ submitted_at TIMESTAMPTZ NOT NULL DEFAULT now(),+ triaged_at TIMESTAMPTZ,+ created_at TIMESTAMPTZ NOT NULL DEFAULT now()+);++CREATE INDEX IF NOT EXISTS idx_aisoc_phishing_verdict ON aisoc_phishing_submissions (verdict);+CREATE INDEX IF NOT EXISTS idx_aisoc_phishing_tenant ON aisoc_phishing_submissions (tenant_id);diff --git a/services/api/migrations/016_knowledge_base.sql b/services/api/migrations/016_knowledge_base.sqlnew file mode 100644index 000000000..7d1572eee--- /dev/null+++ b/services/api/migrations/016_knowledge_base.sql@@ -0,0 +1,25 @@+-- Migration 016: Knowledge-base + RAG over org docs/runbooks+-- Stores indexed document chunks for retrieval-augmented generation.++CREATE TABLE IF NOT EXISTS aisoc_kb_documents (+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),+ tenant_id UUID,+ title TEXT NOT NULL,+ source_url TEXT,+ doc_kind TEXT NOT NULL DEFAULT 'runbook'+ CHECK (doc_kind IN ('runbook','policy','playbook','sop','wiki','other')),+ content TEXT NOT NULL,+ -- chunk metadata (if chunked at ingest)+ chunk_index INT DEFAULT 0,+ chunk_total INT DEFAULT 1,+ -- simple keyword tags for pre-filter+ tags TEXT[] DEFAULT ARRAY[]::TEXT[],+ created_at TIMESTAMPTZ NOT NULL DEFAULT now(),+ updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),+ created_by TEXT+);++CREATE INDEX IF NOT EXISTS idx_aisoc_kb_tenant ON aisoc_kb_documents (tenant_id);+CREATE INDEX IF NOT EXISTS idx_aisoc_kb_kind ON aisoc_kb_documents (doc_kind);+-- Full-text search index+CREATE INDEX IF NOT EXISTS idx_aisoc_kb_fts ON aisoc_kb_documents USING gin(to_tsvector('english', content));diff --git a/services/api/migrations/020_soc_metrics_h2.sql b/services/api/migrations/020_soc_metrics_h2.sqlnew file mode 100644index 000000000..6f02220b4--- /dev/null+++ b/services/api/migrations/020_soc_metrics_h2.sql@@ -0,0 +1,69 @@+-- Migration 020: SOC Metrics H2 Roadmap features+-- Adds: alert disposition, first_seen_at (MTTD), run cost telemetry table,+-- autonomy thresholds table, institutional memory table.++BEGIN;++-- Alert disposition and MTTD tracking+ALTER TABLE alerts+ ADD COLUMN IF NOT EXISTS disposition TEXT,+ ADD COLUMN IF NOT EXISTS first_seen_at TIMESTAMPTZ;++CREATE INDEX IF NOT EXISTS ix_alerts_tenant_disposition+ ON alerts (tenant_id, disposition)+ WHERE disposition IS NOT NULL;++-- Investigation cost telemetry+CREATE TABLE IF NOT EXISTS aisoc_run_costs (+ run_id TEXT NOT NULL,+ tenant_id TEXT NOT NULL,+ model TEXT NOT NULL,+ total_prompt_tokens INTEGER NOT NULL DEFAULT 0,+ total_completion_tokens INTEGER NOT NULL DEFAULT 0,+ total_cost_usd DOUBLE PRECISION NOT NULL DEFAULT 0,+ total_latency_ms DOUBLE PRECISION NOT NULL DEFAULT 0,+ call_count INTEGER NOT NULL DEFAULT 0,+ recorded_at TIMESTAMPTZ NOT NULL DEFAULT now(),+ PRIMARY KEY (run_id, tenant_id, model)+);++CREATE INDEX IF NOT EXISTS aisoc_run_costs_tenant_run+ ON aisoc_run_costs (tenant_id, run_id);++CREATE INDEX IF NOT EXISTS aisoc_run_costs_recorded_at+ ON aisoc_run_costs (recorded_at DESC);++-- Autonomy guardrails thresholds (per-tenant overrides)+CREATE TABLE IF NOT EXISTS aisoc_autonomy_thresholds (+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),+ tenant_id TEXT NOT NULL,+ action_name TEXT NOT NULL,+ min_confidence DOUBLE PRECISION NOT NULL CHECK (min_confidence BETWEEN 0 AND 1),+ updated_by TEXT,+ updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),+ UNIQUE (tenant_id, action_name)+);++CREATE INDEX IF NOT EXISTS aisoc_autonomy_thresholds_tenant+ ON aisoc_autonomy_thresholds (tenant_id);++-- Institutional memory (three-tier memory system)+CREATE TABLE IF NOT EXISTS aisoc_institutional_memory (+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),+ tenant_id TEXT NOT NULL,+ key TEXT NOT NULL,+ value JSONB NOT NULL,+ tags TEXT[] NOT NULL DEFAULT '{}',+ analyst_override BOOLEAN NOT NULL DEFAULT FALSE,+ override_reason TEXT,+ created_at TIMESTAMPTZ NOT NULL DEFAULT now(),+ UNIQUE (tenant_id, key)+);++CREATE INDEX IF NOT EXISTS aisoc_institutional_memory_tenant_key+ ON aisoc_institutional_memory (tenant_id, key);++CREATE INDEX IF NOT EXISTS aisoc_institutional_memory_tags+ ON aisoc_institutional_memory USING GIN (tags);++COMMIT;