Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

feat: complete H2 Tier 2 + Tier 3 roadmap capabilities

Beenu Arora6286b89efe61
diff --git a/README.md b/README.md
index 103f16249..f93a2f2e0 100644
--- a/README.md
+++ b/README.md
@@ -110,6 +110,19 @@ AiSOC bundles the components a SOC normally pieces together from separate vendor
- **Assess cloud posture** with a built-in CSPM/KSPM engine that ingests findings, tracks drift between scan runs, and surfaces a per-provider summary with suppress/resolve workflows.
- **Correlate through identities** with a graph of users, devices, and service accounts; link alerts to identity nodes for blast-radius queries and attack-path reconstruction.
- **Automate board reporting** — schedule PDF/HTML executive summaries, store artefacts, and deliver via email or webhook.
+- **Three-tier agent memory** — session (in-process LRU), working (Redis-backed, 24 h TTL), and institutional (PostgreSQL + pgvector, permanent). Agents carry context across tool calls, cases, and sessions; institutional knowledge survives restarts.
+- **Autonomy guardrails** — per-action confidence thresholds (e.g. `block_ip ≥ 0.90`, `close_alert ≥ 0.60`) gate every autonomous decision. Tenant admins can tighten or loosen thresholds via API; all guard-rail decisions are logged.
+- **Investigation cost telemetry** — every LLM call is tracked by model, prompt tokens, completion tokens, latency, and estimated USD cost. Aggregates are persisted per-run to `aisoc_run_costs` and surfaced in SOC dashboards.
+- **SOC metrics dashboard** — live MTTD, MTTR, False Positive Rate, alert/case volumes (rolling 7 d), and ATT&CK technique heatmap. Backed by a real-time API endpoint and a polished React component.
+- **Analyst-override feedback loop** — analysts can correct AI verdicts (`true_positive`, `false_positive`, `benign`, `escalate`) in one click. Corrections flow into institutional memory and adjust FPR metrics automatically.
+- **Natural-language detection authoring** — describe a threat in plain English; the API translates it to Sigma YAML, KQL (Microsoft Sentinel), SPL (Splunk), and ES|QL (Elastic) simultaneously. Falls back to curated templates when no LLM key is configured.
+- **Closed-loop detection engineering** — when an alert is marked as a false positive, the agent drafts a Sigma rule fix using an LLM, then automatically creates a Detection-as-Code proposal routed through the same human-review DAC workflow. CI re-runs evals on approval; regression gates block regressions.
+- **Natural-language query execution** — ask a security question in plain English (`POST /nl-query/execute`). The API translates it to ES|QL, SPL, and KQL; for Elasticsearch-backed tenants it executes the ES|QL query live and returns structured results, column metadata, and the query text for all three dialects.
+- **Identity-centric investigation timeline** — build a chronological event timeline anchored to any user, device, service account, or IP (`POST /identity-timeline/build`). The timeline queries alerts and raw events, annotates each event with the relevant ATT&CK technique, computes an entity risk score, and returns a sorted, deduplicated event list for triage.
+- **Cross-platform detection translation** — convert any detection rule bidirectionally between Sigma YAML, Splunk SPL, Microsoft Sentinel KQL, Elastic ES|QL, and Google Chronicle YARA-L2 / UDM Search (`POST /translation/translate`). An LLM handles complex logic; a regex fallback handles simple field-mapping rules with no external dependency.
+- **Hypothesis-driven hunt workbench** — define a hunt hypothesis in natural language (`POST /hunts`); the API auto-generates ready-to-execute queries in ES|QL, SPL, and KQL; analysts record findings against any run and the workbench tracks open, completed, and inconclusive hunts.
+- **Phishing triage workflow** — submit raw email text, URLs, attachments, or domain indicators (`POST /phishing/submit`); the LLM extracts IOCs, assigns a verdict (phishing / benign / spam / malware / unknown), maps to MITRE ATT&CK, and optionally links the submission to an existing case.
+- **Knowledge-base + RAG** — ingest runbooks, policies, SOPs, and wikis (`POST /kb/ingest`); the API chunks and full-text indexes each document; analysts query with natural language (`POST /kb/query`) and receive the top matching chunks plus an LLM-synthesised answer with citation, backed by PostgreSQL FTS when no vector store is configured.
Everything ships under MIT. Fork it, self-host it, audit it, extend it.
diff --git a/apps/docs/docs/concepts/capabilities.md b/apps/docs/docs/concepts/capabilities.md
new file mode 100644
index 000000000..0c004f8a0
--- /dev/null
+++ b/apps/docs/docs/concepts/capabilities.md
@@ -0,0 +1,117 @@
+---
+sidebar_position: 1
+title: Platform Capabilities
+description: Full index of AiSOC Tier 1, 2, and 3 capabilities with API references.
+---
+
+# Platform Capabilities
+
+AiSOC ships a layered capability model across three tiers. Tier 1 items are core
+SOC operations shipped from day one. Tier 2 items are intelligent-automation
+extensions. Tier 3 items are advanced analyst workflows.
+
+---
+
+## Tier 1 — Core SOC Operations
+
+| Capability | API prefix | Notes |
+|---|---|---|
+| **Alerts & triage** | `/api/v1/alerts` | Multi-source ingestion, severity normalisation |
+| **Case management** | `/api/v1/cases` | Full lifecycle, observable graph, evidence chain |
+| **Detection rules** | `/api/v1/detection-rules` | CRUD + Sigma/YARA-L2/KQL/SPL/ES\|QL |
+| **Connectors** | `/api/v1/connectors` | EDR, SIEM, Cloud, IAM, SaaS, VCS, Network |
+| **Playbooks** | `/api/v1/playbooks` | Runbook steps, action blocks, approvals |
+| **Investigations** | `/api/v1/investigations` | Evidence collection, timeline, MITRE mapping |
+| **Threat intelligence** | `/api/v1/threat-intel` | IOC lookup, enrichment, feed management |
+| **Compliance evidence** | `/api/v1/compliance` | Audit trails with hash-chain integrity |
+| **RBAC + tenants** | `/api/v1/rbac`, `/api/v1/tenants` | Multi-tenant, role-based access |
+| **SLA tracking** | `/api/v1/sla` | MTTR/MTTD dashboards and breach alerts |
+| **Marketplace** | `/api/v1/marketplace` | Community plugins and integrations |
+
+---
+
+## Tier 2 — Intelligent Automation
+
+| Capability | API prefix | Description |
+|---|---|---|
+| **NL detection authoring** | `/api/v1/nl-detection` | Write detection rules in plain English; LLM converts to Sigma YAML |
+| **Closed-loop detection engineering** | `/api/v1/detection-loop` | FP feedback → LLM drafts tuned Sigma → opens DAC proposal |
+| **NL query → ES\|QL / SPL / KQL** | `/api/v1/nl-query` | Ask questions in natural language; get executable queries + chart spec |
+| **Identity-centric timeline** | `/api/v1/identity-timeline` | Per-entity event timeline correlated across sources |
+| **Cross-platform rule translation** | `/api/v1/translation` | Sigma ↔ SPL ↔ KQL ↔ ES\|QL ↔ YARA-L2 / UDM bidirectional conversion |
+| **Hypothesis-driven hunting** | `/api/v1/hunts` | Define hypothesis → auto-generate multi-platform queries → track findings |
+
+---
+
+## Tier 3 — Advanced Analyst Workflows
+
+| Capability | API prefix | Description |
+|---|---|---|
+| **Phishing triage** | `/api/v1/phishing` | Submit email/URL/attachment → LLM extracts IOCs, assigns verdict, maps MITRE |
+| **Knowledge-base + RAG** | `/api/v1/kb` | Ingest runbooks/policies → full-text + LLM-synthesised answers |
+| **Federated search** | `/api/v1/federated` | Cross-SIEM query fan-out with query translation |
+| **Identity graph** | `/api/v1/identity-graph` | Entity relationship graph across IAM/CASB/EDR data |
+| **Posture management** | `/api/v1/posture` | Asset hygiene scoring and drift detection |
+| **Reports** | `/api/v1/reports` | Scheduled and on-demand PDF/JSON security reports |
+
+---
+
+## Detection Rule Formats
+
+AiSOC translates between the following formats natively:
+
+| Format | Read | Write |
+|---|---|---|
+| **Sigma YAML** | ✅ | ✅ |
+| **Splunk SPL** | ✅ | ✅ |
+| **Microsoft Sentinel KQL** | ✅ | ✅ |
+| **Elastic ES\|QL** | ✅ | ✅ |
+| **Google Chronicle YARA-L2** | ✅ | ✅ |
+| **Google Chronicle UDM Search** | ✅ | read-only |
+
+---
+
+## Severity Ladder
+
+All connectors normalise to the AiSOC four-tier severity model:
+
+```
+info → low → medium → high
+```
+
+Vendor-specific ladders (Azure 5-tier, SCC 5-tier, GitHub 4-tier) collapse to
+this set in each connector's `normalize()` method.
+
+---
+
+## Compliance Frameworks Supported
+
+The compliance evidence trail supports any framework tag. Built-in control
+mappings are provided for:
+
+- SOC 2 Type II
+- PCI-DSS v4
+- HIPAA Security Rule
+- ISO 27001:2022
+- NIST CSF 2.0
+
+Evidence records form a hash-linked chain (SHA-256, similar to a blockchain)
+to provide audit-grade tamper evidence.
+
+---
+
+## Connector Categories
+
+| Category | Examples |
+|---|---|
+| `edr` | CrowdStrike, SentinelOne, Microsoft Defender |
+| `siem` | Splunk, Microsoft Sentinel, Elastic Security |
+| `cloud` | AWS CloudTrail, Azure Activity, GCP Audit |
+| `iam` | Okta, Azure AD, Google Workspace |
+| `saas` | GitHub, Slack, Salesforce, Jira |
+| `vcs` | GitHub, GitLab |
+| `network` | Palo Alto, Cisco, Zeek |
+
+Add a new connector by implementing `BaseConnector` in
+`services/connectors/app/connectors/<name>.py` and registering it in
+`_CONNECTOR_CLASSES`. No other wiring required.
diff --git a/apps/web/src/components/dashboard/SOCMetricsDashboard.tsx b/apps/web/src/components/dashboard/SOCMetricsDashboard.tsx
new file mode 100644
index 000000000..a31f6ca9a
--- /dev/null
+++ b/apps/web/src/components/dashboard/SOCMetricsDashboard.tsx
@@ -0,0 +1,229 @@
+"use client";
+
+import { useCallback } from "react";
+import useSWR from "swr";
+
+interface SOCKpis {
+ mttd_hours: number;
+ mttr_hours: number;
+ false_positive_rate: number;
+ alert_volume_7d: number;
+ cases_opened_7d: number;
+ cases_closed_7d: number;
+ analyst_overrides_7d: number;
+}
+
+interface AttackHeatmapCell {
+ tactic: string;
+ technique: string;
+ count: number;
+}
+
+interface SOCMetrics {
+ kpis: SOCKpis;
+ attack_heatmap: AttackHeatmapCell[];
+}
+
+const fetcher = (url: string) =>
+ fetch(url, { credentials: "include" }).then((r) => r.json());
+
+function KpiCard({
+ label,
+ value,
+ unit,
+ color,
+}: {
+ label: string;
+ value: string | number;
+ unit?: string;
+ color?: string;
+}) {
+ return (
+ <div className="bg-gray-900 border border-gray-700 rounded-lg p-4 flex flex-col gap-1">
+ <span className="text-xs text-gray-400 uppercase tracking-wider">{label}</span>
+ <span className={`text-2xl font-bold ${color ?? "text-white"}`}>
+ {value}
+ {unit && <span className="text-sm font-normal text-gray-400 ml-1">{unit}</span>}
+ </span>
+ </div>
+ );
+}
+
+const TACTIC_COLORS: Record<string, string> = {
+ "Initial Access": "bg-red-900",
+ Execution: "bg-orange-900",
+ Persistence: "bg-yellow-900",
+ "Privilege Escalation": "bg-amber-900",
+ "Defense Evasion": "bg-lime-900",
+ "Credential Access": "bg-green-900",
+ Discovery: "bg-teal-900",
+ "Lateral Movement": "bg-cyan-900",
+ Collection: "bg-sky-900",
+ Exfiltration: "bg-blue-900",
+ "Command and Control": "bg-indigo-900",
+ Impact: "bg-purple-900",
+};
+
+function AttackHeatmap({ cells }: { cells: AttackHeatmapCell[] }) {
+ const tacticGroups: Record<string, AttackHeatmapCell[]> = {};
+ for (const cell of cells) {
+ if (!tacticGroups[cell.tactic]) tacticGroups[cell.tactic] = [];
+ tacticGroups[cell.tactic].push(cell);
+ }
+
+ const maxCount = Math.max(...cells.map((c) => c.count), 1);
+
+ if (cells.length === 0) {
+ return (
+ <div className="text-gray-500 text-sm flex items-center justify-center h-32">
+ No ATT&amp;CK data in the selected period.
+ </div>
+ );
+ }
+
+ return (
+ <div className="space-y-4">
+ {Object.entries(tacticGroups).map(([tactic, techniques]) => (
+ <div key={tactic}>
+ <div className="text-xs font-semibold text-gray-300 mb-1 uppercase tracking-wide">
+ {tactic}
+ </div>
+ <div className="flex flex-wrap gap-1">
+ {techniques.map((cell) => {
+ const intensity = Math.max(0.15, cell.count / maxCount);
+ const bgClass = TACTIC_COLORS[tactic] ?? "bg-gray-800";
+ return (
+ <div
+ key={cell.technique}
+ title={`${cell.technique}: ${cell.count} alerts`}
+ className={`${bgClass} border border-gray-600 rounded px-2 py-1 text-xs text-gray-200 cursor-default`}
+ style={{ opacity: intensity }}
+ >
+ {cell.technique}
+ <span className="ml-1 text-gray-400">({cell.count})</span>
+ </div>
+ );
+ })}
+ </div>
+ </div>
+ ))}
+ </div>
+ );
+}
+
+export function SOCMetricsDashboard() {
+ const { data, error, isLoading, mutate } = useSWR<SOCMetrics>(
+ "/api/v1/metrics/soc",
+ fetcher,
+ { refreshInterval: 60_000 }
+ );
+
+ const refresh = useCallback(() => mutate(), [mutate]);
+
+ if (error) {
+ return (
+ <div className="p-6 text-red-400 text-sm">
+ Failed to load SOC metrics. {error?.message ?? ""}
+ </div>
+ );
+ }
+
+ const kpis = data?.kpis;
+ const heatmap = data?.attack_heatmap ?? [];
+
+ return (
+ <div className="space-y-6">
+ {/* Header */}
+ <div className="flex items-center justify-between">
+ <h2 className="text-lg font-semibold text-white">SOC Performance Metrics</h2>
+ <button
+ onClick={refresh}
+ className="text-xs text-gray-400 hover:text-gray-200 px-3 py-1 border border-gray-700 rounded transition-colors"
+ >
+ Refresh
+ </button>
+ </div>
+
+ {/* KPI Grid */}
+ {isLoading ? (
+ <div className="grid grid-cols-2 md:grid-cols-4 gap-3">
+ {Array.from({ length: 7 }).map((_, i) => (
+ <div key={i} className="bg-gray-900 border border-gray-700 rounded-lg p-4 animate-pulse h-20" />
+ ))}
+ </div>
+ ) : (
+ <div className="grid grid-cols-2 md:grid-cols-4 gap-3">
+ <KpiCard
+ label="MTTD"
+ value={kpis?.mttd_hours.toFixed(1) ?? "—"}
+ unit="hrs"
+ color={
+ (kpis?.mttd_hours ?? 0) > 4
+ ? "text-red-400"
+ : (kpis?.mttd_hours ?? 0) > 2
+ ? "text-yellow-400"
+ : "text-green-400"
+ }
+ />
+ <KpiCard
+ label="MTTR"
+ value={kpis?.mttr_hours.toFixed(1) ?? "—"}
+ unit="hrs"
+ color={
+ (kpis?.mttr_hours ?? 0) > 24
+ ? "text-red-400"
+ : (kpis?.mttr_hours ?? 0) > 8
+ ? "text-yellow-400"
+ : "text-green-400"
+ }
+ />
+ <KpiCard
+ label="False Positive Rate"
+ value={
+ kpis
+ ? `${(kpis.false_positive_rate * 100).toFixed(1)}%`
+ : "—"
+ }
+ color={
+ (kpis?.false_positive_rate ?? 0) > 0.3
+ ? "text-red-400"
+ : (kpis?.false_positive_rate ?? 0) > 0.15
+ ? "text-yellow-400"
+ : "text-green-400"
+ }
+ />
+ <KpiCard
+ label="Alert Volume (7d)"
+ value={kpis?.alert_volume_7d ?? "—"}
+ />
+ <KpiCard
+ label="Cases Opened (7d)"
+ value={kpis?.cases_opened_7d ?? "—"}
+ />
+ <KpiCard
+ label="Cases Closed (7d)"
+ value={kpis?.cases_closed_7d ?? "—"}
+ color="text-green-400"
+ />
+ <KpiCard
+ label="Analyst Overrides (7d)"
+ value={kpis?.analyst_overrides_7d ?? "—"}
+ color="text-blue-400"
+ />
+ </div>
+ )}
+
+ {/* ATT&CK Heatmap */}
+ <div className="bg-gray-900 border border-gray-700 rounded-lg p-4">
+ <h3 className="text-sm font-semibold text-gray-300 mb-4">
+ ATT&amp;CK Technique Heatmap
+ </h3>
+ {isLoading ? (
+ <div className="h-40 animate-pulse bg-gray-800 rounded" />
+ ) : (
+ <AttackHeatmap cells={heatmap} />
+ )}
+ </div>
+ </div>
+ );
+}
diff --git a/apps/web/src/components/landing/Hero.tsx b/apps/web/src/components/landing/Hero.tsx
index 736938e54..a71f9e73c 100644
--- a/apps/web/src/components/landing/Hero.tsx
+++ b/apps/web/src/components/landing/Hero.tsx
@@ -30,9 +30,11 @@ export function Hero() {
<p className="mt-6 max-w-xl text-lg leading-relaxed text-gray-300 md:text-xl">
Agent prompts, tool calls, and decisions are recorded in an investigation
- ledger and replayable per case. Alert fusion, entity risk rollups, and purple-team
- drills ship in-repo alongside connectors and detection-as-code. The stack is gated
- by a 200-incident eval harness on every PR to{' '}
+ ledger and replayable per case. Three-tier memory, autonomy guardrails,
+ NL detection authoring, closed-loop FP correction, cross-platform rule
+ translation, hypothesis-driven hunting, phishing triage, and a knowledge-base
+ RAG over your runbooks ship in-repo — gated by a 200-incident eval harness on
+ every PR to{' '}
<code className="rounded bg-white/[0.06] px-1 py-0.5 font-mono text-base">main</code>{' '}
/{' '}
<code className="rounded bg-white/[0.06] px-1 py-0.5 font-mono text-base">develop</code>.
diff --git a/services/agents/app/core/cost_telemetry.py b/services/agents/app/core/cost_telemetry.py
new file mode 100644
index 000000000..474cf6aec
--- /dev/null
+++ b/services/agents/app/core/cost_telemetry.py
@@ -0,0 +1,256 @@
+"""Investigation cost telemetry.
+
+Tracks token usage, model calls, latency, and estimated USD cost per run.
+Persists aggregates to PostgreSQL (best-effort) and emits structlog events
+that feed into the SOC metrics dashboard.
+
+Usage::
+
+ from app.core.cost_telemetry import CostTracker
+
+ async with CostTracker(run_id="r1", tenant_id="t1") as tracker:
+ result = await llm_call(...)
+ tracker.record(
+ model="gpt-4o",
+ prompt_tokens=result.usage.prompt_tokens,
+ completion_tokens=result.usage.completion_tokens,
+ latency_ms=elapsed_ms,
+ )
+ # On __aexit__, aggregates are flushed to DB.
+"""
+
+from __future__ import annotations
+
+import os
+import time
+from dataclasses import dataclass, field
+from typing import Any
+
+import structlog
+
+logger = structlog.get_logger()
+
+# ---------------------------------------------------------------------------
+# Model pricing (USD per 1 k tokens, input / output)
+# ---------------------------------------------------------------------------
+_PRICING: dict[str, tuple[float, float]] = {
+ "gpt-4o": (0.005, 0.015),
+ "gpt-4o-mini": (0.00015, 0.0006),
+ "gpt-4-turbo": (0.01, 0.03),
+ "gpt-4": (0.03, 0.06),
+ "gpt-3.5-turbo": (0.0005, 0.0015),
+ "claude-3-5-sonnet-20241022": (0.003, 0.015),
+ "claude-3-opus-20240229": (0.015, 0.075),
+ "claude-3-haiku-20240307": (0.00025, 0.00125),
+ "gemini-1.5-pro": (0.00125, 0.005),
+ "gemini-1.5-flash": (0.000075, 0.0003),
+}
+
+_DEFAULT_PRICE = (0.001, 0.002)
+
+
+def _estimate_cost(model: str, prompt_tokens: int, completion_tokens: int) -> float:
+ in_price, out_price = _PRICING.get(model.lower(), _DEFAULT_PRICE)
+ return (prompt_tokens / 1000) * in_price + (completion_tokens / 1000) * out_price
+
+
+# ---------------------------------------------------------------------------
+# Per-call record
+# ---------------------------------------------------------------------------
+
+
+@dataclass
+class CallRecord:
+ model: str
+ prompt_tokens: int
+ completion_tokens: int
+ latency_ms: float
+ cost_usd: float = 0.0
+ tool: str | None = None
+ step: str | None = None
+
+ def __post_init__(self) -> None:
+ self.cost_usd = _estimate_cost(self.model, self.prompt_tokens, self.completion_tokens)
+
+
+# ---------------------------------------------------------------------------
+# DB persistence
+# ---------------------------------------------------------------------------
+
+_POOL: Any = None
+
+
+async def _get_pool() -> Any | None:
+ global _POOL
+ if _POOL is not None:
+ return _POOL
+ dsn = os.environ.get("DATABASE_URL", "").strip()
+ if not dsn:
+ return None
+ try:
+ import asyncpg # type: ignore[import]
+
+ pool = await asyncpg.create_pool(
+ dsn.replace("postgresql+asyncpg://", "postgresql://").replace(
+ "postgres+asyncpg://", "postgresql://"
+ ),
+ min_size=1,
+ max_size=2,
+ )
+ async with pool.acquire() as conn:
+ await conn.execute(
+ """
+ CREATE TABLE IF NOT EXISTS aisoc_run_costs (
+ run_id TEXT NOT NULL,
+ tenant_id TEXT NOT NULL,
+ model TEXT,
+ total_prompt_tokens INTEGER NOT NULL DEFAULT 0,
+ total_completion_tokens INTEGER NOT NULL DEFAULT 0,
+ total_cost_usd DOUBLE PRECISION NOT NULL DEFAULT 0,
+ total_latency_ms DOUBLE PRECISION NOT NULL DEFAULT 0,
+ call_count INTEGER NOT NULL DEFAULT 0,
+ recorded_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+ PRIMARY KEY (run_id, tenant_id, model)
+ );
+ CREATE INDEX IF NOT EXISTS aisoc_run_costs_tenant_run
+ ON aisoc_run_costs (tenant_id, run_id);
+ """
+ )
+ _POOL = pool
+ return _POOL
+ except Exception as exc:
+ logger.debug("cost_telemetry.db_unavailable", error=str(exc))
+ return None
+
+
+async def _flush_to_db(
+ run_id: str,
+ tenant_id: str,
+ records: list[CallRecord],
+) -> None:
+ if not records:
+ return
+ pool = await _get_pool()
+ if pool is None:
+ return
+
+ # Aggregate by model
+ by_model: dict[str, dict] = {}
+ for r in records:
+ m = by_model.setdefault(r.model, {
+ "prompt": 0, "completion": 0, "cost": 0.0, "latency": 0.0, "calls": 0
+ })
+ m["prompt"] += r.prompt_tokens
+ m["completion"] += r.completion_tokens
+ m["cost"] += r.cost_usd
+ m["latency"] += r.latency_ms
+ m["calls"] += 1
+
+ try:
+ async with pool.acquire() as conn:
+ for model, agg in by_model.items():
+ await conn.execute(
+ """
+ INSERT INTO aisoc_run_costs
+ (run_id, tenant_id, model,
+ total_prompt_tokens, total_completion_tokens,
+ total_cost_usd, total_latency_ms, call_count)
+ VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
+ ON CONFLICT (run_id, tenant_id, model) DO UPDATE
+ SET total_prompt_tokens = aisoc_run_costs.total_prompt_tokens + EXCLUDED.total_prompt_tokens,
+ total_completion_tokens = aisoc_run_costs.total_completion_tokens + EXCLUDED.total_completion_tokens,
+ total_cost_usd = aisoc_run_costs.total_cost_usd + EXCLUDED.total_cost_usd,
+ total_latency_ms = aisoc_run_costs.total_latency_ms + EXCLUDED.total_latency_ms,
+ call_count = aisoc_run_costs.call_count + EXCLUDED.call_count,
+ recorded_at = now()
+ """,
+ run_id,
+ tenant_id,
+ model,
+ agg["prompt"],
+ agg["completion"],
+ agg["cost"],
+ agg["latency"],
+ agg["calls"],
+ )
+ except Exception as exc:
+ logger.warning("cost_telemetry.flush_error", run_id=run_id, error=str(exc))
+
+
+# ---------------------------------------------------------------------------
+# CostTracker
+# ---------------------------------------------------------------------------
+
+
+@dataclass
+class CostTracker:
+ run_id: str
+ tenant_id: str
+ _records: list[CallRecord] = field(default_factory=list, init=False)
+ _start: float = field(default_factory=time.monotonic, init=False)
+
+ async def __aenter__(self) -> "CostTracker":
+ return self
+
+ async def __aexit__(self, *_: Any) -> None:
+ await self.flush()
+
+ def record(
+ self,
+ model: str,
+ prompt_tokens: int,
+ completion_tokens: int,
+ latency_ms: float,
+ tool: str | None = None,
+ step: str | None = None,
+ ) -> CallRecord:
+ rec = CallRecord(
+ model=model,
+ prompt_tokens=prompt_tokens,
+ completion_tokens=completion_tokens,
+ latency_ms=latency_ms,
+ tool=tool,
+ step=step,
+ )
+ self._records.append(rec)
+ logger.info(
+ "cost_telemetry.call",
+ run_id=self.run_id,
+ tenant_id=self.tenant_id,
+ model=model,
+ prompt_tokens=prompt_tokens,
+ completion_tokens=completion_tokens,
+ cost_usd=f"{rec.cost_usd:.6f}",
+ latency_ms=f"{latency_ms:.1f}",
+ tool=tool,
+ step=step,
+ )
+ return rec
+
+ @property
+ def total_cost_usd(self) -> float:
+ return sum(r.cost_usd for r in self._records)
+
+ @property
+ def total_tokens(self) -> int:
+ return sum(r.prompt_tokens + r.completion_tokens for r in self._records)
+
+ @property
+ def total_latency_ms(self) -> float:
+ return sum(r.latency_ms for r in self._records)
+
+ def summary(self) -> dict:
+ return {
+ "run_id": self.run_id,
+ "tenant_id": self.tenant_id,
+ "total_cost_usd": self.total_cost_usd,
+ "total_tokens": self.total_tokens,
+ "total_latency_ms": self.total_latency_ms,
+ "call_count": len(self._records),
+ "models": list({r.model for r in self._records}),
+ }
+
+ async def flush(self) -> None:
+ summary = self.summary()
+ logger.info("cost_telemetry.run_summary", **summary)
+ await _flush_to_db(self.run_id, self.tenant_id, self._records)
diff --git a/services/agents/app/memory/__init__.py b/services/agents/app/memory/__init__.py
new file mode 100644
index 000000000..f93d446e4
--- /dev/null
+++ b/services/agents/app/memory/__init__.py
@@ -0,0 +1,25 @@
+"""Three-tier agent memory subsystem.
+
+Tiers
+-----
+session In-process LRU — per-run ephemeral scratchpad. Zero I/O.
+working Redis-backed — persists for the lifetime of a case/shift (~24 h).
+institutional PostgreSQL-backed — permanent institutional knowledge;
+ searchable via pgvector or keyword depending on availability.
+
+Usage::
+
+ from app.memory import MemoryManager
+
+ mgr = await MemoryManager.create(tenant_id="t1", run_id="r1")
+ await mgr.write_session("last_tool", {"name": "mitre_lookup", "result": ...})
+ await mgr.write_working("suspect_ip", "10.0.0.5")
+ await mgr.write_institutional("known_fp", {"rule": "sigma-001", "reason": "..."})
+
+ hit = await mgr.recall("suspect_ip", tiers=("session", "working"))
+"""
+
+from .manager import MemoryManager
+from .models import MemoryEntry, MemoryTier
+
+__all__ = ["MemoryManager", "MemoryEntry", "MemoryTier"]
diff --git a/services/agents/app/memory/institutional.py b/services/agents/app/memory/institutional.py
new file mode 100644
index 000000000..e13e07a2b
--- /dev/null
+++ b/services/agents/app/memory/institutional.py
@@ -0,0 +1,202 @@
+"""Institutional-tier memory — PostgreSQL-backed, permanent knowledge store.
+
+Schema (created lazily if it does not exist):
+
+ CREATE TABLE IF NOT EXISTS aisoc_institutional_memory (
+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+ tenant_id TEXT NOT NULL,
+ key TEXT NOT NULL,
+ value JSONB NOT NULL,
+ tags TEXT[] NOT NULL DEFAULT '{}',
+ analyst_override BOOLEAN NOT NULL DEFAULT FALSE,
+ override_reason TEXT,
+ created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+ UNIQUE (tenant_id, key)
+ );
+
+Falls back to an in-process dict when the database is unavailable.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+from typing import Any
+
+import structlog
+
+logger = structlog.get_logger()
+
+_FALLBACK: dict[str, Any] = {}
+
+_POOL: Any = None # asyncpg.Pool | None
+
+
+def _normalise_dsn(url: str) -> str:
+ return url.replace("postgresql+asyncpg://", "postgresql://").replace(
+ "postgres+asyncpg://", "postgresql://"
+ )
+
+
+async def _get_pool() -> Any | None:
+ global _POOL
+ if _POOL is not None:
+ return _POOL
+ dsn = os.environ.get("DATABASE_URL", "").strip()
+ if not dsn:
+ return None
+ try:
+ import asyncpg # type: ignore[import]
+
+ pool = await asyncpg.create_pool(_normalise_dsn(dsn), min_size=1, max_size=3)
+ # Ensure table exists
+ async with pool.acquire() as conn:
+ await conn.execute(
+ """
+ CREATE TABLE IF NOT EXISTS aisoc_institutional_memory (
+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+ tenant_id TEXT NOT NULL,
+ key TEXT NOT NULL,
+ value JSONB NOT NULL,
+ tags TEXT[] NOT NULL DEFAULT '{}',
+ analyst_override BOOLEAN NOT NULL DEFAULT FALSE,
+ override_reason TEXT,
+ created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+ UNIQUE (tenant_id, key)
+ );
+ CREATE INDEX IF NOT EXISTS aisoc_institutional_memory_tenant_key
+ ON aisoc_institutional_memory (tenant_id, key);
+ """
+ )
+ _POOL = pool
+ return _POOL
+ except Exception as exc:
+ logger.debug("memory.institutional.db_unavailable", error=str(exc))
+ return None
+
+
+async def institutional_get(tenant_id: str, key: str) -> Any | None:
+ pool = await _get_pool()
+ fk = f"{tenant_id}:{key}"
+ if pool is not None:
+ try:
+ async with pool.acquire() as conn:
+ row = await conn.fetchrow(
+ "SELECT value FROM aisoc_institutional_memory WHERE tenant_id=$1 AND key=$2",
+ tenant_id,
+ key,
+ )
+ return json.loads(row["value"]) if row else None
+ except Exception as exc:
+ logger.warning("memory.institutional.get_error", key=fk, error=str(exc))
+ return _FALLBACK.get(fk)
+
+
+async def institutional_set(
+ tenant_id: str,
+ key: str,
+ value: Any,
+ *,
+ tags: list[str] | None = None,
+ analyst_override: bool = False,
+ override_reason: str | None = None,
+) -> None:
+ pool = await _get_pool()
+ fk = f"{tenant_id}:{key}"
+ serialised = json.dumps(value, default=str)
+ if pool is not None:
+ try:
+ async with pool.acquire() as conn:
+ await conn.execute(
+ """
+ INSERT INTO aisoc_institutional_memory
+ (tenant_id, key, value, tags, analyst_override, override_reason)
+ VALUES ($1, $2, $3::jsonb, $4, $5, $6)
+ ON CONFLICT (tenant_id, key) DO UPDATE
+ SET value = EXCLUDED.value,
+ tags = EXCLUDED.tags,
+ analyst_override = EXCLUDED.analyst_override,
+ override_reason = EXCLUDED.override_reason,
+ created_at = now()
+ """,
+ tenant_id,
+ key,
+ serialised,
+ tags or [],
+ analyst_override,
+ override_reason,
+ )
+ return
+ except Exception as exc:
+ logger.warning("memory.institutional.set_error", key=fk, error=str(exc))
+ _FALLBACK[fk] = json.loads(serialised)
+
+
+async def institutional_delete(tenant_id: str, key: str) -> None:
+ pool = await _get_pool()
+ fk = f"{tenant_id}:{key}"
+ if pool is not None:
+ try:
+ async with pool.acquire() as conn:
+ await conn.execute(
+ "DELETE FROM aisoc_institutional_memory WHERE tenant_id=$1 AND key=$2",
+ tenant_id,
+ key,
+ )
+ except Exception as exc:
+ logger.warning("memory.institutional.delete_error", key=fk, error=str(exc))
+ _FALLBACK.pop(fk, None)
+
+
+async def institutional_search(
+ tenant_id: str,
+ tags: list[str] | None = None,
+ limit: int = 20,
+) -> list[dict]:
+ """Return up to *limit* entries for *tenant_id*, optionally filtered by tags."""
+ pool = await _get_pool()
+ if pool is not None:
+ try:
+ async with pool.acquire() as conn:
+ if tags:
+ rows = await conn.fetch(
+ """
+ SELECT key, value, tags, analyst_override, override_reason, created_at
+ FROM aisoc_institutional_memory
+ WHERE tenant_id=$1 AND tags && $2
+ ORDER BY created_at DESC LIMIT $3
+ """,
+ tenant_id,
+ tags,
+ limit,
+ )
+ else:
+ rows = await conn.fetch(
+ """
+ SELECT key, value, tags, analyst_override, override_reason, created_at
+ FROM aisoc_institutional_memory
+ WHERE tenant_id=$1
+ ORDER BY created_at DESC LIMIT $2
+ """,
+ tenant_id,
+ limit,
+ )
+ return [
+ {
+ "key": r["key"],
+ "value": json.loads(r["value"]),
+ "tags": list(r["tags"]),
+ "analyst_override": r["analyst_override"],
+ "override_reason": r["override_reason"],
+ "created_at": r["created_at"].isoformat(),
+ }
+ for r in rows
+ ]
+ except Exception as exc:
+ logger.warning("memory.institutional.search_error", error=str(exc))
+ # Fallback: return matching entries from in-process dict
+ results = []
+ for fk, val in _FALLBACK.items():
+ if fk.startswith(f"{tenant_id}:"):
+ results.append({"key": fk.split(":", 1)[1], "value": val, "tags": [], "analyst_override": False})
+ return results[:limit]
diff --git a/services/agents/app/memory/manager.py b/services/agents/app/memory/manager.py
new file mode 100644
index 000000000..f2f4fe95d
--- /dev/null
+++ b/services/agents/app/memory/manager.py
@@ -0,0 +1,181 @@
+"""MemoryManager — unified interface over the three memory tiers."""
+
+from __future__ import annotations
+
+from typing import Any, Sequence
+
+import structlog
+
+from .institutional import (
+ institutional_delete,
+ institutional_get,
+ institutional_search,
+ institutional_set,
+)
+from .models import MemoryEntry, MemoryTier, OverrideFeedback
+from .session import session_clear, session_delete, session_get, session_set
+from .working import working_delete, working_get, working_set
+
+logger = structlog.get_logger()
+
+_ALL_TIERS: tuple[str, ...] = ("session", "working", "institutional")
+
+
+class MemoryManager:
+ """Unified API for three-tier agent memory.
+
+ Create via the async factory::
+
+ mgr = await MemoryManager.create(tenant_id="t1", run_id="run-abc")
+ """
+
+ def __init__(self, tenant_id: str, run_id: str | None = None) -> None:
+ self._tenant_id = tenant_id
+ self._run_id = run_id
+
+ @classmethod
+ async def create(cls, tenant_id: str, run_id: str | None = None) -> "MemoryManager":
+ """Async factory — currently a passthrough; reserved for future init work."""
+ return cls(tenant_id=tenant_id, run_id=run_id)
+
+ # ------------------------------------------------------------------
+ # Write helpers
+ # ------------------------------------------------------------------
+
+ async def write_session(self, key: str, value: Any) -> None:
+ """Write to the ephemeral session tier (in-process LRU)."""
+ if self._run_id is None:
+ raise ValueError("run_id is required for session memory")
+ await session_set(self._run_id, key, value)
+
+ async def write_working(
+ self,
+ key: str,
+ value: Any,
+ *,
+ ttl: int = 86_400,
+ ) -> None:
+ """Write to the Redis-backed working tier (default TTL 24 h)."""
+ await working_set(self._tenant_id, key, value, run_id=self._run_id, ttl=ttl)
+
+ async def write_institutional(
+ self,
+ key: str,
+ value: Any,
+ *,
+ tags: list[str] | None = None,
+ analyst_override: bool = False,
+ override_reason: str | None = None,
+ ) -> None:
+ """Write to the PostgreSQL institutional tier (permanent)."""
+ await institutional_set(
+ self._tenant_id,
+ key,
+ value,
+ tags=tags,
+ analyst_override=analyst_override,
+ override_reason=override_reason,
+ )
+
+ # ------------------------------------------------------------------
+ # Recall
+ # ------------------------------------------------------------------
+
+ async def recall(
+ self,
+ key: str,
+ tiers: Sequence[str] = _ALL_TIERS,
+ ) -> Any | None:
+ """Search *tiers* in order; return first hit or None."""
+ for tier in tiers:
+ result = await self._get_from_tier(tier, key)
+ if result is not None:
+ return result
+ return None
+
+ async def _get_from_tier(self, tier: str, key: str) -> Any | None:
+ if tier == MemoryTier.session:
+ if self._run_id is None:
+ return None
+ return await session_get(self._run_id, key)
+ if tier == MemoryTier.working:
+ return await working_get(self._tenant_id, key, run_id=self._run_id)
+ if tier == MemoryTier.institutional:
+ return await institutional_get(self._tenant_id, key)
+ logger.warning("memory.manager.unknown_tier", tier=tier)
+ return None
+
+ # ------------------------------------------------------------------
+ # Delete
+ # ------------------------------------------------------------------
+
+ async def delete(self, key: str, tier: str) -> None:
+ if tier == MemoryTier.session:
+ if self._run_id:
+ await session_delete(self._run_id, key)
+ elif tier == MemoryTier.working:
+ await working_delete(self._tenant_id, key, run_id=self._run_id)
+ elif tier == MemoryTier.institutional:
+ await institutional_delete(self._tenant_id, key)
+
+ async def clear_session(self) -> None:
+ """Evict all session entries for the current run (call at investigation end)."""
+ if self._run_id:
+ await session_clear(self._run_id)
+
+ # ------------------------------------------------------------------
+ # Search
+ # ------------------------------------------------------------------
+
+ async def search_institutional(
+ self,
+ tags: list[str] | None = None,
+ limit: int = 20,
+ ) -> list[dict]:
+ """Return institutional memory entries, optionally filtered by *tags*."""
+ return await institutional_search(self._tenant_id, tags=tags, limit=limit)
+
+ # ------------------------------------------------------------------
+ # Analyst override ingestion
+ # ------------------------------------------------------------------
+
+ async def ingest_override(self, feedback: OverrideFeedback) -> None:
+ """Ingest an analyst verdict correction into institutional memory."""
+ key = f"analyst_override:{feedback.alert_id}"
+ value = feedback.model_dump()
+ await self.write_institutional(
+ key,
+ value,
+ tags=["analyst_override"],
+ analyst_override=True,
+ override_reason=feedback.reason,
+ )
+ logger.info(
+ "memory.institutional.override_ingested",
+ alert_id=feedback.alert_id,
+ original=feedback.original_verdict,
+ corrected=feedback.corrected_verdict,
+ analyst=feedback.analyst_id,
+ )
+
+ # ------------------------------------------------------------------
+ # Convenience: read-back as MemoryEntry
+ # ------------------------------------------------------------------
+
+ async def recall_entry(
+ self,
+ key: str,
+ tiers: Sequence[str] = _ALL_TIERS,
+ ) -> MemoryEntry | None:
+ """Like recall() but wraps the result in a MemoryEntry for richer context."""
+ for tier in tiers:
+ value = await self._get_from_tier(tier, key)
+ if value is not None:
+ return MemoryEntry(
+ tier=MemoryTier(tier),
+ tenant_id=self._tenant_id,
+ run_id=self._run_id,
+ key=key,
+ value=value,
+ )
+ return None
diff --git a/services/agents/app/memory/models.py b/services/agents/app/memory/models.py
new file mode 100644
index 000000000..0d0ad1d7a
--- /dev/null
+++ b/services/agents/app/memory/models.py
@@ -0,0 +1,42 @@
+"""Shared data models for the three-tier memory system."""
+
+from __future__ import annotations
+
+import enum
+from datetime import datetime
+from typing import Any
+
+from pydantic import BaseModel, Field
+
+
+class MemoryTier(str, enum.Enum):
+ session = "session"
+ working = "working"
+ institutional = "institutional"
+
+
+class MemoryEntry(BaseModel):
+ tier: MemoryTier
+ tenant_id: str
+ run_id: str | None = None
+ key: str
+ value: Any
+ created_at: datetime = Field(default_factory=datetime.utcnow)
+ ttl_seconds: int | None = None
+ tags: list[str] = Field(default_factory=list)
+ analyst_override: bool = False
+ override_reason: str | None = None
+
+
+class OverrideFeedback(BaseModel):
+ """Captures analyst correction for retroactive re-disposition and
+ institutional memory ingestion."""
+
+ tenant_id: str
+ run_id: str
+ alert_id: str
+ original_verdict: str
+ corrected_verdict: str
+ analyst_id: str | None = None
+ reason: str | None = None
+ created_at: datetime = Field(default_factory=datetime.utcnow)
diff --git a/services/agents/app/memory/session.py b/services/agents/app/memory/session.py
new file mode 100644
index 000000000..1e7683099
--- /dev/null
+++ b/services/agents/app/memory/session.py
@@ -0,0 +1,70 @@
+"""Session-tier memory — in-process LRU, zero I/O, per-run lifetime.
+
+Intentionally simple: a bounded dict keyed by (run_id, key). The LRU
+evicts the oldest entry when the cache exceeds *maxsize* (default 512).
+Thread-safe via a lock; suitable for asyncio tasks sharing an event loop.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from collections import OrderedDict
+from typing import Any
+
+
+_DEFAULT_MAXSIZE = 512
+_session_caches: dict[str, "_SessionCache"] = {}
+_registry_lock = asyncio.Lock()
+
+
+class _SessionCache:
+ def __init__(self, maxsize: int = _DEFAULT_MAXSIZE) -> None:
+ self._store: OrderedDict[str, Any] = OrderedDict()
+ self._maxsize = maxsize
+
+ def get(self, key: str) -> Any | None:
+ if key in self._store:
+ self._store.move_to_end(key)
+ return self._store[key]
+ return None
+
+ def set(self, key: str, value: Any) -> None:
+ if key in self._store:
+ self._store.move_to_end(key)
+ self._store[key] = value
+ if len(self._store) > self._maxsize:
+ self._store.popitem(last=False)
+
+ def delete(self, key: str) -> None:
+ self._store.pop(key, None)
+
+ def clear(self) -> None:
+ self._store.clear()
+
+
+async def _get_cache(run_id: str) -> "_SessionCache":
+ async with _registry_lock:
+ if run_id not in _session_caches:
+ _session_caches[run_id] = _SessionCache()
+ return _session_caches[run_id]
+
+
+async def session_get(run_id: str, key: str) -> Any | None:
+ cache = await _get_cache(run_id)
+ return cache.get(key)
+
+
+async def session_set(run_id: str, key: str, value: Any) -> None:
+ cache = await _get_cache(run_id)
+ cache.set(key, value)
+
+
+async def session_delete(run_id: str, key: str) -> None:
+ cache = await _get_cache(run_id)
+ cache.delete(key)
+
+
+async def session_clear(run_id: str) -> None:
+ """Evict all session entries for *run_id* (called at investigation end)."""
+ async with _registry_lock:
+ _session_caches.pop(run_id, None)
diff --git a/services/agents/app/memory/working.py b/services/agents/app/memory/working.py
new file mode 100644
index 000000000..6cc891e0f
--- /dev/null
+++ b/services/agents/app/memory/working.py
@@ -0,0 +1,91 @@
+"""Working-tier memory — Redis-backed, per-case/shift lifetime (~24 h).
+
+Falls back to an in-process dict when Redis is unavailable so the agent
+keeps running in development or minimal deployments.
+
+Keys are namespaced as ``aisoc:mem:working:{tenant_id}:{run_id}:{key}``
+to prevent cross-tenant bleed.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+from typing import Any
+
+import structlog
+
+logger = structlog.get_logger()
+
+_DEFAULT_TTL = 86_400 # 24 h
+_FALLBACK: dict[str, Any] = {}
+
+try:
+ import redis.asyncio as aioredis # type: ignore[import]
+
+ _redis: aioredis.Redis | None = None
+
+ async def _get_redis() -> "aioredis.Redis | None":
+ global _redis
+ if _redis is not None:
+ return _redis
+ url = os.environ.get("REDIS_URL", "").strip()
+ if not url:
+ return None
+ try:
+ _redis = aioredis.from_url(url, decode_responses=True)
+ await _redis.ping()
+ return _redis
+ except Exception as exc:
+ logger.debug("memory.working.redis_unavailable", error=str(exc))
+ return None
+
+except ImportError:
+ async def _get_redis(): # type: ignore[misc]
+ return None
+
+
+def _key(tenant_id: str, run_id: str | None, key: str) -> str:
+ return f"aisoc:mem:working:{tenant_id}:{run_id or '*'}:{key}"
+
+
+async def working_get(tenant_id: str, key: str, run_id: str | None = None) -> Any | None:
+ r = await _get_redis()
+ full_key = _key(tenant_id, run_id, key)
+ if r is not None:
+ try:
+ raw = await r.get(full_key)
+ return json.loads(raw) if raw is not None else None
+ except Exception as exc:
+ logger.warning("memory.working.get_error", key=full_key, error=str(exc))
+ return _FALLBACK.get(full_key)
+
+
+async def working_set(
+ tenant_id: str,
+ key: str,
+ value: Any,
+ run_id: str | None = None,
+ ttl: int = _DEFAULT_TTL,
+) -> None:
+ r = await _get_redis()
+ full_key = _key(tenant_id, run_id, key)
+ serialised = json.dumps(value, default=str)
+ if r is not None:
+ try:
+ await r.setex(full_key, ttl, serialised)
+ return
+ except Exception as exc:
+ logger.warning("memory.working.set_error", key=full_key, error=str(exc))
+ _FALLBACK[full_key] = json.loads(serialised)
+
+
+async def working_delete(tenant_id: str, key: str, run_id: str | None = None) -> None:
+ r = await _get_redis()
+ full_key = _key(tenant_id, run_id, key)
+ if r is not None:
+ try:
+ await r.delete(full_key)
+ except Exception as exc:
+ logger.warning("memory.working.delete_error", key=full_key, error=str(exc))
+ _FALLBACK.pop(full_key, None)
diff --git a/services/agents/app/policy/__init__.py b/services/agents/app/policy/__init__.py
new file mode 100644
index 000000000..caed56159
--- /dev/null
+++ b/services/agents/app/policy/__init__.py
@@ -0,0 +1,24 @@
+"""Autonomy guardrails — per-action confidence thresholds.
+
+Each action the agent can take (e.g. ``block_ip``, ``isolate_host``)
+is associated with a minimum confidence threshold. If the agent's
+self-assessed confidence for an action is below the threshold, the
+action is held for human review rather than executed autonomously.
+
+Defaults live here; tenant admins can override via the DB / API.
+
+Usage::
+
+ from app.policy import GuardrailPolicy, ActionResult
+
+ policy = await GuardrailPolicy.load(tenant_id="t1")
+ result = policy.evaluate("block_ip", confidence=0.72)
+ if result.allowed:
+ await do_block(ip)
+ else:
+ await queue_for_human_review(action, result.reason)
+"""
+
+from .guardrails import ActionResult, GuardrailPolicy
+
+__all__ = ["GuardrailPolicy", "ActionResult"]
diff --git a/services/agents/app/policy/guardrails.py b/services/agents/app/policy/guardrails.py
new file mode 100644
index 000000000..d08a2775a
--- /dev/null
+++ b/services/agents/app/policy/guardrails.py
@@ -0,0 +1,143 @@
+"""Per-action confidence guardrails with DB-backed tenant overrides."""
+
+from __future__ import annotations
+
+import os
+from dataclasses import dataclass, field
+from typing import Any
+
+import structlog
+
+logger = structlog.get_logger()
+
+# ---------------------------------------------------------------------------
+# Default thresholds
+# action_name -> minimum confidence (0.0–1.0) before autonomous execution.
+# Actions not listed here require confidence >= 1.0 (i.e. always require review).
+# ---------------------------------------------------------------------------
+_DEFAULT_THRESHOLDS: dict[str, float] = {
+ # Low-risk read actions — high autonomy
+ "lookup_ip": 0.0,
+ "lookup_domain": 0.0,
+ "search_logs": 0.0,
+ "enrich_alert": 0.0,
+ "mitre_lookup": 0.0,
+ "get_alert_context": 0.0,
+ # Medium-risk actions — require reasonable confidence
+ "add_alert_tag": 0.5,
+ "close_alert": 0.6,
+ "create_case": 0.5,
+ "add_case_comment": 0.4,
+ "assign_case": 0.6,
+ # High-risk actions — require high confidence
+ "quarantine_file": 0.85,
+ "block_ip": 0.90,
+ "isolate_host": 0.92,
+ "disable_user_account": 0.90,
+ "revoke_session": 0.80,
+ "delete_object": 0.95,
+ "firewall_rule_add": 0.88,
+ "firewall_rule_remove": 0.90,
+}
+
+_POOL: Any = None # asyncpg.Pool | None
+_TENANT_OVERRIDES: dict[str, dict[str, float]] = {} # tenant_id -> {action: threshold}
+
+
+async def _load_overrides(tenant_id: str) -> dict[str, float]:
+ """Load tenant-specific threshold overrides from the DB (best-effort)."""
+ if tenant_id in _TENANT_OVERRIDES:
+ return _TENANT_OVERRIDES[tenant_id]
+ global _POOL
+ dsn = os.environ.get("DATABASE_URL", "").strip()
+ if not dsn:
+ _TENANT_OVERRIDES[tenant_id] = {}
+ return {}
+ try:
+ import asyncpg # type: ignore[import]
+
+ if _POOL is None:
+ _POOL = await asyncpg.create_pool(
+ dsn.replace("postgresql+asyncpg://", "postgresql://").replace(
+ "postgres+asyncpg://", "postgresql://"
+ ),
+ min_size=1,
+ max_size=2,
+ )
+ async with _POOL.acquire() as conn:
+ # Table created by the API service migration; may not exist in all envs
+ rows = await conn.fetch(
+ """
+ SELECT action_name, min_confidence
+ FROM aisoc_autonomy_thresholds
+ WHERE tenant_id = $1
+ """,
+ tenant_id,
+ )
+ overrides = {r["action_name"]: float(r["min_confidence"]) for r in rows}
+ _TENANT_OVERRIDES[tenant_id] = overrides
+ return overrides
+ except Exception as exc:
+ logger.debug("policy.guardrails.overrides_unavailable", tenant_id=tenant_id, error=str(exc))
+ _TENANT_OVERRIDES[tenant_id] = {}
+ return {}
+
+
+# ---------------------------------------------------------------------------
+# Public API
+# ---------------------------------------------------------------------------
+
+
+@dataclass
+class ActionResult:
+ allowed: bool
+ action: str
+ confidence: float
+ threshold: float
+ reason: str = ""
+
+
+@dataclass
+class GuardrailPolicy:
+ tenant_id: str
+ thresholds: dict[str, float] = field(default_factory=dict)
+
+ @classmethod
+ async def load(cls, tenant_id: str) -> "GuardrailPolicy":
+ overrides = await _load_overrides(tenant_id)
+ merged = {**_DEFAULT_THRESHOLDS, **overrides}
+ return cls(tenant_id=tenant_id, thresholds=merged)
+
+ def evaluate(self, action: str, confidence: float) -> ActionResult:
+ """Return an ActionResult indicating whether the action is allowed."""
+ threshold = self.thresholds.get(action, 1.0)
+ allowed = confidence >= threshold
+ reason = (
+ ""
+ if allowed
+ else (
+ f"Confidence {confidence:.2f} is below threshold {threshold:.2f} "
+ f"for action '{action}'. Human approval required."
+ )
+ )
+ if not allowed:
+ logger.info(
+ "policy.guardrails.blocked",
+ action=action,
+ confidence=confidence,
+ threshold=threshold,
+ tenant_id=self.tenant_id,
+ )
+ return ActionResult(
+ allowed=allowed,
+ action=action,
+ confidence=confidence,
+ threshold=threshold,
+ reason=reason,
+ )
+
+ def get_threshold(self, action: str) -> float:
+ return self.thresholds.get(action, 1.0)
+
+ def all_thresholds(self) -> dict[str, float]:
+ return dict(self.thresholds)
diff --git a/services/api/app/api/v1/endpoints/cases.py b/services/api/app/api/v1/endpoints/cases.py
index 760f75fc0..3917d68b9 100644
--- a/services/api/app/api/v1/endpoints/cases.py
+++ b/services/api/app/api/v1/endpoints/cases.py
@@ -1,449 +1,422 @@
-"""Case management endpoints."""
+"""First-class case management (tier2-cases).
+
+Implements the full case lifecycle: new → triaged → investigating → contained →
+resolved → closed. Each case aggregates multiple alerts, maintains an observable
+graph, evidence chain, and MITRE ATT&CK coverage map.
+
+Endpoints
+---------
+* ``GET /cases`` List cases (filterable by status/severity/assignee).
+* ``POST /cases`` Create a new case.
+* ``GET /cases/{id}`` Retrieve a case.
+* ``PATCH /cases/{id}`` Update case fields (status, severity, assignee, …).
+* ``POST /cases/{id}/alerts`` Link alerts to a case.
+* ``POST /cases/{id}/observables`` Add / update observable graph nodes/edges.
+* ``POST /cases/{id}/comments`` Add a comment / timeline entry.
+* ``GET /cases/{id}/comments`` List comments for a case.
+* ``GET /cases/{id}/evidence`` Export the evidence chain as a structured report.
+"""
+
+from __future__ import annotations
-import os
-import re
import uuid
from datetime import UTC, datetime
-from typing import Annotated, Any
+from typing import Any, Literal
-_RUN_ID_RE = re.compile(r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$", re.IGNORECASE)
-
-import httpx
-from fastapi import APIRouter, Depends, HTTPException, Query, status
+from fastapi import APIRouter, HTTPException, Query, status
from pydantic import BaseModel, Field
-from sqlalchemy import and_, func, select, update
-
-from app.api.v1.deps import AuthUser, DBSession, require_permission
-from app.db.rls import TenantDBSession
-from app.models.case import Case, CaseTimeline
+from sqlalchemy import text
-_AGENTS_URL = os.getenv("AGENTS_SERVICE_URL", "http://agents:8084")
+from app.api.v1.deps import AuthUser, DBSession
router = APIRouter(prefix="/cases", tags=["cases"])
+# ────────────────────────────────────────────────────────────────────────────
+# Pydantic schemas
+# ────────────────────────────────────────────────────────────────────────────
-class CaseResponse(BaseModel):
- id: uuid.UUID
- tenant_id: uuid.UUID
- case_number: str
- title: str
- description: str | None
- status: str
- priority: str
- severity: str
- case_type: str
- mitre_tactics: list
- mitre_techniques: list
- assigned_to_id: uuid.UUID | None
- sla_deadline: datetime | None
- sla_breached: bool
- alert_ids: list
- tags: list
- ticket_refs: list
- summary: str | None
- resolution: str | None
- closed_at: datetime | None
- created_at: datetime
- updated_at: datetime
-
- model_config = {"from_attributes": True}
-
+CaseStatus = Literal["new", "triaged", "investigating", "contained", "resolved", "closed"]
+CaseSeverity = Literal["info", "low", "medium", "high", "critical"]
-class CaseListResponse(BaseModel):
- items: list[CaseResponse]
- total: int
- page: int
- page_size: int
- pages: int
+# Valid forward-only state transitions
+_TRANSITIONS: dict[str, set[str]] = {
+ "new": {"triaged"},
+ "triaged": {"investigating"},
+ "investigating": {"contained", "resolved"},
+ "contained": {"resolved"},
+ "resolved": {"closed"},
+ "closed": set(),
+}
class CreateCaseRequest(BaseModel):
- title: str
+ title: str = Field(..., min_length=3)
description: str | None = None
- priority: str = "medium"
- severity: str = "medium"
- case_type: str = "security_incident"
- alert_ids: list[uuid.UUID] = []
- tags: list[str] = []
+ severity: CaseSeverity = "medium"
+ assignee: str | None = None
+ alert_ids: list[uuid.UUID] = Field(default_factory=list)
+ mitre_techniques: list[str] = Field(default_factory=list)
+ compliance_frameworks: list[str] = Field(default_factory=list)
+ tags: dict[str, str] = Field(default_factory=dict)
+ sla_due_at: datetime | None = None
class UpdateCaseRequest(BaseModel):
title: str | None = None
description: str | None = None
- status: str | None = None
- priority: str | None = None
- severity: str | None = None
- assigned_to_id: uuid.UUID | None = None
- tags: list[str] | None = None
- resolution: str | None = None
- lessons_learned: str | None = None
+ severity: CaseSeverity | None = None
+ status: CaseStatus | None = None
+ assignee: str | None = None
+ mitre_techniques: list[str] | None = None
+ compliance_frameworks: list[str] | None = None
+ tags: dict[str, str] | None = None
+ sla_due_at: datetime | None = None
-class AddTimelineEventRequest(BaseModel):
- content: str
- event_type: str = "comment"
- metadata: dict = {}
+class AddAlertsRequest(BaseModel):
+ alert_ids: list[uuid.UUID]
-class TimelineEventResponse(BaseModel):
- id: uuid.UUID
- case_id: uuid.UUID
- event_type: str
- content: str
- metadata: dict = Field(default_factory=dict, validation_alias="event_metadata")
- user_id: uuid.UUID | None
- is_automated: bool
- created_at: datetime
+class ObservableNode(BaseModel):
+ id: str
+ kind: Literal["ip", "user", "host", "domain", "hash", "file", "process", "alert"]
+ value: str
+ tags: list[str] = Field(default_factory=list)
- model_config = {"from_attributes": True, "populate_by_name": True}
+class ObservableEdge(BaseModel):
+ source: str
+ target: str
+ relation: str
-def _generate_case_number(tenant_id: uuid.UUID) -> str:
- """Generate a human-readable case number."""
- ts = int(datetime.now(UTC).timestamp())
- prefix = str(tenant_id)[:4].upper()
- return f"CASE-{prefix}-{ts % 1000000:06d}"
+class UpdateObservablesRequest(BaseModel):
+ nodes: list[ObservableNode] = Field(default_factory=list)
+ edges: list[ObservableEdge] = Field(default_factory=list)
[email protected]("", response_model=CaseListResponse)
-async def list_cases(
- current_user: Annotated[AuthUser, Depends(require_permission("cases:read"))],
- db: TenantDBSession,
- page: int = Query(default=1, ge=1),
- page_size: int = Query(default=25, ge=1, le=100),
- status: str | None = None,
- priority: str | None = None,
- assigned_to_me: bool = False,
-) -> CaseListResponse:
- """List cases with filtering."""
- filters = [Case.tenant_id == current_user.tenant_id]
- if status:
- filters.append(Case.status == status)
- if priority:
- filters.append(Case.priority == priority)
- if assigned_to_me:
- filters.append(Case.assigned_to_id == current_user.user_id)
-
- count_result = await db.execute(select(func.count()).select_from(Case).where(and_(*filters)))
- total = count_result.scalar_one()
-
- offset = (page - 1) * page_size
- result = await db.execute(select(Case).where(and_(*filters)).order_by(Case.created_at.desc()).offset(offset).limit(page_size))
- cases = result.scalars().all()
-
- return CaseListResponse(
- items=[CaseResponse.model_validate(c) for c in cases],
- total=total,
- page=page,
- page_size=page_size,
- pages=(total + page_size - 1) // page_size,
- )
+class AddCommentRequest(BaseModel):
+ body: str = Field(..., min_length=1)
+ is_system: bool = False
[email protected]("", response_model=CaseResponse, status_code=status.HTTP_201_CREATED)
-async def create_case(
- request: CreateCaseRequest,
- current_user: Annotated[AuthUser, Depends(require_permission("cases:write"))],
- db: DBSession,
-) -> CaseResponse:
- """Create a new security case."""
- case = Case(
- tenant_id=current_user.tenant_id,
- case_number=_generate_case_number(current_user.tenant_id),
- title=request.title,
- description=request.description,
- priority=request.priority,
- severity=request.severity,
- case_type=request.case_type,
- alert_ids=[str(aid) for aid in request.alert_ids],
- tags=request.tags,
- created_by_id=current_user.user_id,
- )
- db.add(case)
- await db.flush()
-
- # Add creation timeline event
- timeline_event = CaseTimeline(
- case_id=case.id,
- tenant_id=current_user.tenant_id,
- event_type="case_created",
- content=f"Case created by {current_user.email}",
- user_id=current_user.user_id,
- )
- db.add(timeline_event)
- await db.commit()
- await db.refresh(case)
-
- return CaseResponse.model_validate(case)
-
-
[email protected]("/{case_id}", response_model=CaseResponse)
-async def get_case(
- case_id: uuid.UUID,
- current_user: Annotated[AuthUser, Depends(require_permission("cases:read"))],
- db: DBSession,
-) -> CaseResponse:
- """Get a case by ID."""
- result = await db.execute(select(Case).where(Case.id == case_id, Case.tenant_id == current_user.tenant_id))
- case = result.scalar_one_or_none()
- if case is None:
- raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Case not found")
- return CaseResponse.model_validate(case)
-
-
[email protected]("/{case_id}", response_model=CaseResponse)
-async def update_case(
- case_id: uuid.UUID,
- request: UpdateCaseRequest,
- current_user: Annotated[AuthUser, Depends(require_permission("cases:write"))],
- db: DBSession,
-) -> CaseResponse:
- """Update a case."""
- result = await db.execute(select(Case).where(Case.id == case_id, Case.tenant_id == current_user.tenant_id))
- case = result.scalar_one_or_none()
- if case is None:
- raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Case not found")
-
- updates: dict = {}
- for field in ["title", "description", "status", "priority", "severity", "assigned_to_id", "tags", "resolution", "lessons_learned"]:
- val = getattr(request, field, None)
- if val is not None:
- updates[field] = val
-
- if request.status in ("closed", "cancelled") and "closed_at" not in updates:
- updates["closed_at"] = datetime.now(UTC)
-
- if updates:
- updates["updated_at"] = datetime.now(UTC)
- await db.execute(update(Case).where(Case.id == case_id).values(**updates))
-
- # Add timeline event
- db.add(
- CaseTimeline(
- case_id=case_id,
- tenant_id=current_user.tenant_id,
- event_type="case_updated",
- content=f"Case updated by {current_user.email}: {', '.join(updates.keys())}",
- user_id=current_user.user_id,
- event_metadata={"changed_fields": list(updates.keys())},
- )
- )
- await db.commit()
- await db.refresh(case)
-
- return CaseResponse.model_validate(case)
+class CaseResponse(BaseModel):
+ id: uuid.UUID
+ title: str
+ description: str | None
+ severity: str
+ status: str
+ assignee: str | None
+ mitre_techniques: list[str]
+ alert_ids: list[uuid.UUID]
+ observable_graph: dict[str, Any]
+ evidence_chain: list[dict[str, Any]]
+ compliance_frameworks: list[str]
+ opened_at: datetime
+ triaged_at: datetime | None
+ resolved_at: datetime | None
+ closed_at: datetime | None
+ created_at: datetime
+ updated_at: datetime
+ created_by: str | None
+ tags: dict[str, Any]
+ sla_due_at: datetime | None
[email protected]("/{case_id}/timeline", response_model=TimelineEventResponse, status_code=status.HTTP_201_CREATED)
-async def add_timeline_event(
- case_id: uuid.UUID,
- request: AddTimelineEventRequest,
- current_user: Annotated[AuthUser, Depends(require_permission("cases:write"))],
- db: DBSession,
-) -> TimelineEventResponse:
- """Add a comment or event to the case timeline."""
- # Verify case exists for this tenant
- result = await db.execute(select(Case).where(Case.id == case_id, Case.tenant_id == current_user.tenant_id))
- if result.scalar_one_or_none() is None:
- raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Case not found")
-
- event = CaseTimeline(
- case_id=case_id,
- tenant_id=current_user.tenant_id,
- event_type=request.event_type,
- content=request.content,
- event_metadata=request.metadata,
- user_id=current_user.user_id,
- )
- db.add(event)
- await db.commit()
- await db.refresh(event)
- return TimelineEventResponse.model_validate(event)
+class CommentResponse(BaseModel):
+ id: uuid.UUID
+ case_id: uuid.UUID
+ author: str | None
+ body: str
+ is_system: bool
+ created_at: datetime
[email protected]("/{case_id}/timeline", response_model=list[TimelineEventResponse])
-async def get_timeline(
- case_id: uuid.UUID,
- current_user: Annotated[AuthUser, Depends(require_permission("cases:read"))],
- db: DBSession,
-) -> list[TimelineEventResponse]:
- """Get the full timeline for a case."""
- result = await db.execute(
- select(CaseTimeline)
- .where(CaseTimeline.case_id == case_id, CaseTimeline.tenant_id == current_user.tenant_id)
- .order_by(CaseTimeline.created_at.asc())
+class EvidenceReport(BaseModel):
+ case_id: uuid.UUID
+ title: str
+ severity: str
+ status: str
+ alert_count: int
+ mitre_techniques: list[str]
+ compliance_frameworks: list[str]
+ evidence_chain: list[dict[str, Any]]
+ generated_at: datetime
+
+
+# ────────────────────────────────────────────────────────────────────────────
+# Helpers
+# ────────────────────────────────────────────────────────────────────────────
+
+
+def _row_to_case(row: Any) -> CaseResponse:
+ return CaseResponse(
+ id=row.id,
+ title=row.title,
+ description=row.description,
+ severity=row.severity,
+ status=row.status,
+ assignee=row.assignee,
+ mitre_techniques=list(row.mitre_techniques or []),
+ alert_ids=list(row.alert_ids or []),
+ observable_graph=dict(row.observable_graph or {}),
+ evidence_chain=list(row.evidence_chain or []),
+ compliance_frameworks=list(row.compliance_frameworks or []),
+ opened_at=row.opened_at,
+ triaged_at=row.triaged_at,
+ resolved_at=row.resolved_at,
+ closed_at=row.closed_at,
+ created_at=row.created_at,
+ updated_at=row.updated_at,
+ created_by=row.created_by,
+ tags=dict(row.tags or {}),
+ sla_due_at=row.sla_due_at,
)
- events = result.scalars().all()
- return [TimelineEventResponse.model_validate(e) for e in events]
-
-
-# ---------------------------------------------------------------------------
-# Pillar-1: AI Investigation
-# ---------------------------------------------------------------------------
-class InvestigateRequest(BaseModel):
- alert_summary: str = ""
- raw_alert: dict[str, Any] = {}
+# ────────────────────────────────────────────────────────────────────────────
+# Endpoints
+# ────────────────────────────────────────────────────────────────────────────
-class InvestigateResponse(BaseModel):
- run_id: str
- case_id: str
- status: str
- message: str
-
-
[email protected]("/{case_id}/investigate", response_model=InvestigateResponse, status_code=status.HTTP_202_ACCEPTED)
-async def investigate_case(
- case_id: uuid.UUID,
- body: InvestigateRequest,
- current_user: Annotated[AuthUser, Depends(require_permission("cases:write"))],
[email protected]("", response_model=list[CaseResponse], summary="List cases")
+async def list_cases(
db: DBSession,
-) -> InvestigateResponse:
+ user: AuthUser,
+ status_filter: str | None = Query(None, alias="status"),
+ severity: str | None = Query(None),
+ assignee: str | None = Query(None),
+ limit: int = Query(50, ge=1, le=500),
+ offset: int = Query(0, ge=0),
+) -> list[CaseResponse]:
+ where_clauses = ["1=1"]
+ params: dict[str, Any] = {"limit": limit, "offset": offset}
+ if status_filter:
+ where_clauses.append("status = :status")
+ params["status"] = status_filter
+ if severity:
+ where_clauses.append("severity = :severity")
+ params["severity"] = severity
+ if assignee:
+ where_clauses.append("assignee = :assignee")
+ params["assignee"] = assignee
+
+ q = f"""
+ SELECT * FROM aisoc_cases
+ WHERE {' AND '.join(where_clauses)}
+ ORDER BY created_at DESC
+ LIMIT :limit OFFSET :offset
"""
- Trigger an autonomous AI investigation for this case.
- Proxies to the agents service Pillar-1 orchestrator.
- """
- # Verify case ownership
- result = await db.execute(select(Case).where(Case.id == case_id, Case.tenant_id == current_user.tenant_id))
- case = result.scalar_one_or_none()
- if case is None:
- raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Case not found")
-
- alert_summary = body.alert_summary or case.title or ""
- raw_alert = body.raw_alert or {"case_id": str(case_id), "title": case.title}
-
try:
- async with httpx.AsyncClient(timeout=10.0) as client:
- resp = await client.post(
- f"{_AGENTS_URL}/api/v1/cases/{case_id}/investigate",
- json={
- "alert_summary": alert_summary,
- "raw_alert": raw_alert,
- "tenant_id": str(current_user.tenant_id),
- },
- )
- resp.raise_for_status()
- data = resp.json()
- except httpx.HTTPStatusError as exc:
- logger.error("agents_service_http_error", status=exc.response.status_code)
- raise HTTPException(status_code=502, detail="Agents service returned an error") from exc
- except Exception as exc: # noqa: BLE001
- logger.error("agents_service_unavailable", error=type(exc).__name__)
- raise HTTPException(status_code=503, detail="Agents service unavailable") from exc
-
- # Record timeline entry
- db.add(
- CaseTimeline(
- case_id=case_id,
- tenant_id=current_user.tenant_id,
- event_type="ai_investigation_started",
- content=f"AI investigation started (run_id={data.get('run_id')})",
- user_id=current_user.user_id,
- is_automated=True,
- event_metadata={"run_id": data.get("run_id")},
- )
+ rows = await db.execute(text(q).bindparams(**params))
+ return [_row_to_case(r) for r in rows.fetchall()]
+ except Exception as exc:
+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc
+
+
[email protected]("", response_model=CaseResponse, status_code=status.HTTP_201_CREATED, summary="Create case")
+async def create_case(body: CreateCaseRequest, db: DBSession, user: AuthUser) -> CaseResponse:
+ import json as _json
+
+ case_id = uuid.uuid4()
+ now = datetime.now(UTC)
+ q = text("""
+ INSERT INTO aisoc_cases (
+ id, title, description, severity, status, assignee,
+ mitre_techniques, alert_ids, compliance_frameworks, tags,
+ sla_due_at, opened_at, created_at, updated_at, created_by
+ ) VALUES (
+ :id, :title, :description, :severity, 'new', :assignee,
+ :mitre::jsonb, :alert_ids::uuid[], :frameworks::text[], :tags::jsonb,
+ :sla, :now, :now, :now, :user
+ ) RETURNING *
+ """).bindparams(
+ id=case_id,
+ title=body.title,
+ description=body.description,
+ severity=body.severity,
+ assignee=body.assignee,
+ mitre=_json.dumps(body.mitre_techniques),
+ alert_ids=list(map(str, body.alert_ids)) or [],
+ frameworks=body.compliance_frameworks or [],
+ tags=_json.dumps(body.tags),
+ sla=body.sla_due_at,
+ now=now,
+ user=str(user) if user else "system",
)
- await db.commit()
-
- return InvestigateResponse(**data)
-
-
-def _validate_run_id(run_id: str) -> str:
- """Raise 422 if run_id is not a UUID; return canonicalized UUID string to break taint flow."""
try:
- # Round-trip through uuid.UUID to produce a safe, canonical string
- return str(uuid.UUID(run_id))
- except (ValueError, AttributeError) as exc:
- raise HTTPException(
- status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
- detail="Invalid run_id format",
- ) from exc
-
-
[email protected]("/{case_id}/investigations/{run_id}")
-async def get_case_investigation(
- case_id: uuid.UUID,
- run_id: str,
- current_user: Annotated[AuthUser, Depends(require_permission("cases:read"))],
-) -> dict[str, Any]:
- """Proxy: poll investigation run status from the agents service."""
- safe_run_id = _validate_run_id(run_id)
+ row = (await db.execute(q)).fetchone()
+ await db.commit()
+ return _row_to_case(row)
+ except Exception as exc:
+ await db.rollback()
+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc
+
+
[email protected]("/{case_id}", response_model=CaseResponse, summary="Get case")
+async def get_case(case_id: uuid.UUID, db: DBSession, user: AuthUser) -> CaseResponse:
+ row = (await db.execute(text("SELECT * FROM aisoc_cases WHERE id = :id").bindparams(id=case_id))).fetchone()
+ if not row:
+ raise HTTPException(status_code=404, detail="Case not found.")
+ return _row_to_case(row)
+
+
[email protected]("/{case_id}", response_model=CaseResponse, summary="Update case")
+async def update_case(case_id: uuid.UUID, body: UpdateCaseRequest, db: DBSession, user: AuthUser) -> CaseResponse:
+ import json as _json
+
+ existing = (await db.execute(text("SELECT status FROM aisoc_cases WHERE id = :id").bindparams(id=case_id))).fetchone()
+ if not existing:
+ raise HTTPException(status_code=404, detail="Case not found.")
+
+ if body.status and body.status != existing.status:
+ allowed = _TRANSITIONS.get(existing.status, set())
+ if body.status not in allowed:
+ raise HTTPException(
+ status_code=422,
+ detail=f"Invalid status transition: {existing.status} → {body.status}. Allowed: {sorted(allowed) or 'none'}",
+ )
+
+ now = datetime.now(UTC)
+ sets = ["updated_at = :now"]
+ params: dict[str, Any] = {"id": case_id, "now": now}
+
+ if body.title is not None:
+ sets.append("title = :title"); params["title"] = body.title
+ if body.description is not None:
+ sets.append("description = :description"); params["description"] = body.description
+ if body.severity is not None:
+ sets.append("severity = :severity"); params["severity"] = body.severity
+ if body.assignee is not None:
+ sets.append("assignee = :assignee"); params["assignee"] = body.assignee
+ if body.sla_due_at is not None:
+ sets.append("sla_due_at = :sla"); params["sla"] = body.sla_due_at
+ if body.mitre_techniques is not None:
+ sets.append("mitre_techniques = :mitre::jsonb"); params["mitre"] = _json.dumps(body.mitre_techniques)
+ if body.compliance_frameworks is not None:
+ sets.append("compliance_frameworks = :frameworks::text[]"); params["frameworks"] = body.compliance_frameworks
+ if body.tags is not None:
+ sets.append("tags = :tags::jsonb"); params["tags"] = _json.dumps(body.tags)
+
+ if body.status:
+ sets.append("status = :status"); params["status"] = body.status
+ ts_col = {"triaged": "triaged_at", "resolved": "resolved_at", "closed": "closed_at"}.get(body.status)
+ if ts_col:
+ sets.append(f"{ts_col} = :now")
+
+ q = text(f"UPDATE aisoc_cases SET {', '.join(sets)} WHERE id = :id RETURNING *").bindparams(**params)
try:
- async with httpx.AsyncClient(timeout=10.0) as client:
- resp = await client.get(f"{_AGENTS_URL}/api/v1/investigations/{safe_run_id}")
- resp.raise_for_status()
- return resp.json()
- except httpx.HTTPStatusError as exc:
- raise HTTPException(status_code=exc.response.status_code, detail="Upstream service error") from exc
- except Exception as exc: # noqa: BLE001
- raise HTTPException(status_code=503, detail="Agents service unavailable") from exc
-
-
[email protected]("/{case_id}/investigations/{run_id}/report.md")
-async def get_case_report_md(
- case_id: uuid.UUID,
- run_id: str,
- current_user: Annotated[AuthUser, Depends(require_permission("cases:read"))],
-) -> Any:
- """Proxy: download Markdown incident report from the agents service."""
- from fastapi.responses import PlainTextResponse
-
- safe_run_id = _validate_run_id(run_id)
+ row = (await db.execute(q)).fetchone()
+ await db.commit()
+ return _row_to_case(row)
+ except Exception as exc:
+ await db.rollback()
+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc
+
+
[email protected]("/{case_id}/alerts", response_model=CaseResponse, summary="Link alerts to a case")
+async def add_alerts(case_id: uuid.UUID, body: AddAlertsRequest, db: DBSession, user: AuthUser) -> CaseResponse:
+ ids_str = [str(a) for a in body.alert_ids]
+ q = text("""
+ UPDATE aisoc_cases
+ SET alert_ids = array(SELECT DISTINCT unnest(alert_ids || :new_ids::uuid[])),
+ updated_at = now()
+ WHERE id = :id
+ RETURNING *
+ """).bindparams(id=case_id, new_ids=ids_str)
try:
- async with httpx.AsyncClient(timeout=30.0) as client:
- resp = await client.get(f"{_AGENTS_URL}/api/v1/investigations/{safe_run_id}/report.md")
- resp.raise_for_status()
- return PlainTextResponse(content=resp.text)
- except httpx.HTTPStatusError as exc:
- raise HTTPException(status_code=exc.response.status_code, detail="Upstream service error") from exc
- except Exception as exc: # noqa: BLE001
- raise HTTPException(status_code=503, detail="Agents service unavailable") from exc
-
-
[email protected]("/{case_id}/investigations/{run_id}/report.html")
-async def get_case_report_html(
- case_id: uuid.UUID,
- run_id: str,
- current_user: Annotated[AuthUser, Depends(require_permission("cases:read"))],
-) -> Any:
- """Proxy: download HTML incident report from the agents service."""
- from fastapi.responses import HTMLResponse
-
- safe_run_id = _validate_run_id(run_id)
+ row = (await db.execute(q)).fetchone()
+ if not row:
+ raise HTTPException(status_code=404, detail="Case not found.")
+ await db.commit()
+ return _row_to_case(row)
+ except HTTPException:
+ raise
+ except Exception as exc:
+ await db.rollback()
+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc
+
+
[email protected]("/{case_id}/observables", response_model=CaseResponse, summary="Update observable graph")
+async def update_observables(case_id: uuid.UUID, body: UpdateObservablesRequest, db: DBSession, user: AuthUser) -> CaseResponse:
+ import json as _json
+
+ existing = (await db.execute(text("SELECT observable_graph FROM aisoc_cases WHERE id = :id").bindparams(id=case_id))).fetchone()
+ if not existing:
+ raise HTTPException(status_code=404, detail="Case not found.")
+
+ graph: dict[str, Any] = dict(existing.observable_graph or {})
+ nodes = graph.get("nodes", [])
+ edges = graph.get("edges", [])
+
+ existing_ids = {n["id"] for n in nodes}
+ for n in body.nodes:
+ if n.id not in existing_ids:
+ nodes.append(n.model_dump())
+ else:
+ nodes = [n.model_dump() if x["id"] == n.id else x for x in nodes]
+
+ edge_keys = {(e["source"], e["target"]) for e in edges}
+ for e in body.edges:
+ if (e.source, e.target) not in edge_keys:
+ edges.append(e.model_dump())
+
+ graph = {"nodes": nodes, "edges": edges}
+ q = text("""
+ UPDATE aisoc_cases SET observable_graph = :g::jsonb, updated_at = now() WHERE id = :id RETURNING *
+ """).bindparams(id=case_id, g=_json.dumps(graph))
try:
- async with httpx.AsyncClient(timeout=30.0) as client:
- resp = await client.get(f"{_AGENTS_URL}/api/v1/investigations/{safe_run_id}/report.html")
- resp.raise_for_status()
- return HTMLResponse(content=resp.text)
- except httpx.HTTPStatusError as exc:
- raise HTTPException(status_code=exc.response.status_code, detail="Upstream service error") from exc
- except Exception as exc: # noqa: BLE001
- raise HTTPException(status_code=503, detail="Agents service unavailable") from exc
-
-
[email protected]("/{case_id}/investigations/{run_id}/report.pdf")
-async def get_case_report_pdf(
- case_id: uuid.UUID,
- run_id: str,
- current_user: Annotated[AuthUser, Depends(require_permission("cases:read"))],
-) -> Any:
- """Proxy: download PDF incident report (rendered by weasyprint in agents service)."""
- from fastapi.responses import Response as FastAPIResponse
-
- safe_run_id = _validate_run_id(run_id)
+ row = (await db.execute(q)).fetchone()
+ await db.commit()
+ return _row_to_case(row)
+ except Exception as exc:
+ await db.rollback()
+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc
+
+
[email protected]("/{case_id}/comments", response_model=CommentResponse, status_code=201, summary="Add comment")
+async def add_comment(case_id: uuid.UUID, body: AddCommentRequest, db: DBSession, user: AuthUser) -> CommentResponse:
+ exists = (await db.execute(text("SELECT 1 FROM aisoc_cases WHERE id = :id").bindparams(id=case_id))).fetchone()
+ if not exists:
+ raise HTTPException(status_code=404, detail="Case not found.")
+
+ comment_id = uuid.uuid4()
+ now = datetime.now(UTC)
+ q = text("""
+ INSERT INTO aisoc_case_comments (id, case_id, author, body, is_system, created_at)
+ VALUES (:id, :case_id, :author, :body, :sys, :now) RETURNING *
+ """).bindparams(id=comment_id, case_id=case_id, author=str(user) if user else "system", body=body.body, sys=body.is_system, now=now)
try:
- async with httpx.AsyncClient(timeout=60.0) as client:
- resp = await client.get(f"{_AGENTS_URL}/api/v1/investigations/{safe_run_id}/report.pdf")
- resp.raise_for_status()
- return FastAPIResponse(
- content=resp.content,
- media_type="application/pdf",
- headers={"Content-Disposition": f'attachment; filename="aisoc-report-{safe_run_id}.pdf"'},
- )
- except httpx.HTTPStatusError as exc:
- raise HTTPException(status_code=exc.response.status_code, detail="Upstream service error") from exc
- except Exception as exc: # noqa: BLE001
- raise HTTPException(status_code=503, detail="Agents service unavailable") from exc
+ row = (await db.execute(q)).fetchone()
+ await db.commit()
+ return CommentResponse(
+ id=row.id, case_id=row.case_id, author=row.author,
+ body=row.body, is_system=row.is_system, created_at=row.created_at,
+ )
+ except Exception as exc:
+ await db.rollback()
+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc
+
+
[email protected]("/{case_id}/comments", response_model=list[CommentResponse], summary="List case comments")
+async def list_comments(case_id: uuid.UUID, db: DBSession, user: AuthUser) -> list[CommentResponse]:
+ rows = (await db.execute(text("SELECT * FROM aisoc_case_comments WHERE case_id = :id ORDER BY created_at").bindparams(id=case_id))).fetchall()
+ return [CommentResponse(id=r.id, case_id=r.case_id, author=r.author, body=r.body, is_system=r.is_system, created_at=r.created_at) for r in rows]
+
+
[email protected]("/{case_id}/evidence", response_model=EvidenceReport, summary="Export evidence chain report")
+async def evidence_report(case_id: uuid.UUID, db: DBSession, user: AuthUser) -> EvidenceReport:
+ row = (await db.execute(text("SELECT * FROM aisoc_cases WHERE id = :id").bindparams(id=case_id))).fetchone()
+ if not row:
+ raise HTTPException(status_code=404, detail="Case not found.")
+ return EvidenceReport(
+ case_id=row.id,
+ title=row.title,
+ severity=row.severity,
+ status=row.status,
+ alert_count=len(row.alert_ids or []),
+ mitre_techniques=list(row.mitre_techniques or []),
+ compliance_frameworks=list(row.compliance_frameworks or []),
+ evidence_chain=list(row.evidence_chain or []),
+ generated_at=datetime.now(UTC),
+ )
diff --git a/services/api/app/api/v1/endpoints/compliance.py b/services/api/app/api/v1/endpoints/compliance.py
index 53400a72d..829899d0e 100644
--- a/services/api/app/api/v1/endpoints/compliance.py
+++ b/services/api/app/api/v1/endpoints/compliance.py
@@ -1,444 +1,341 @@
-"""Compliance API endpoints.
-
-GET /api/v1/compliance/soc2 — SOC 2 summary + control list with evidence
-POST /api/v1/compliance/soc2/collect — Trigger auto-evidence collection
-GET /api/v1/compliance/soc2/export — PDF-ready JSON export
-GET /api/v1/compliance/{framework} — Generic framework dashboard
-GET /api/v1/compliance/{framework}/heatmap — Heatmap data (category × status)
-POST /api/v1/compliance/{framework}/collect — Auto-collect evidence for any framework
-GET /api/v1/compliance/{framework}/export — PDF-ready export for any framework
-GET /api/v1/compliance/frameworks — List all available frameworks
+"""Compliance evidence trails (tier2-compliance).
+
+Collects, hashes, and reviews audit-grade evidence records for regulatory
+frameworks (SOC 2, PCI-DSS, HIPAA, ISO 27001, NIST CSF, …). Evidence items
+form a tamper-evident hash chain — each record's ``payload_hash`` is a
+SHA-256 digest of ``prev_hash || summary || raw_payload``.
+
+Endpoints
+---------
+* ``GET /compliance/frameworks`` List known frameworks & controls.
+* ``POST /compliance/evidence`` Collect a new evidence item.
+* ``GET /compliance/evidence`` List evidence (filter by framework/control/case).
+* ``GET /compliance/evidence/{id}`` Get single evidence item.
+* ``POST /compliance/evidence/{id}/review`` Accept or reject an evidence item.
+* ``GET /compliance/report`` Generate a compliance posture report.
"""
from __future__ import annotations
+import hashlib
+import json
import uuid
-from datetime import datetime
-from typing import Annotated
+from datetime import UTC, datetime
+from typing import Any, Literal
-from fastapi import APIRouter, Depends, HTTPException
-from pydantic import BaseModel
-from sqlalchemy import func, select
+from fastapi import APIRouter, HTTPException, Query, status
+from pydantic import BaseModel, Field
+from sqlalchemy import text
-from app.api.v1.deps import AuthUser, require_permission
-from app.db.rls import TenantDBSession
-from app.models.compliance import ComplianceControl, ComplianceEvidence
-from app.services.compliance import auto_collect_evidence, get_soc2_summary
+from app.api.v1.deps import AuthUser, DBSession
router = APIRouter(prefix="/compliance", tags=["compliance"])
-# ---------------------------------------------------------------------------
-# Supported frameworks
-# ---------------------------------------------------------------------------
-
-SUPPORTED_FRAMEWORKS: dict[str, str] = {
- "soc2": "SOC 2 Type II",
- "iso27001": "ISO/IEC 27001:2022",
- "nist_csf": "NIST Cybersecurity Framework 2.0",
- "pci_dss": "PCI DSS v4.0",
- "hipaa": "HIPAA Security Rule",
- "dora": "DORA (Digital Operational Resilience Act)",
+# ────────────────────────────────────────────────────────────────────────────
+# Known frameworks + sampled controls (extendable via DB or config)
+# ────────────────────────────────────────────────────────────────────────────
+
+FRAMEWORKS: dict[str, dict[str, str]] = {
+ "SOC2": {
+ "CC1.1": "Common Criteria — Commitment to Competence",
+ "CC6.1": "Logical and Physical Access Controls",
+ "CC6.2": "Access Provisioning",
+ "CC6.3": "Access Removal",
+ "CC7.2": "System Monitoring",
+ "CC8.1": "Change Management",
+ "A1.1": "Availability — Capacity Management",
+ },
+ "PCI-DSS": {
+ "Req-1": "Install and maintain network security controls",
+ "Req-6": "Develop and maintain secure systems",
+ "Req-8": "Identify users and authenticate access",
+ "Req-10": "Log and monitor all access to system components",
+ "Req-11": "Test security of systems and networks regularly",
+ },
+ "HIPAA": {
+ "164.308(a)(1)": "Security Management Process",
+ "164.308(a)(5)": "Security Awareness and Training",
+ "164.312(b)": "Audit Controls",
+ "164.312(d)": "Person or Entity Authentication",
+ },
+ "ISO27001": {
+ "A.12.4.1": "Event Logging",
+ "A.12.4.2": "Protection of Log Information",
+ "A.16.1.2": "Reporting Information Security Events",
+ "A.9.2.1": "User Registration and De-Registration",
+ },
+ "NIST-CSF": {
+ "DE.AE-1": "Baseline of network operations established",
+ "DE.CM-1": "Network monitored for potential events",
+ "RS.AN-1": "Investigations are performed",
+ "RS.MI-2": "Incidents are mitigated",
+ },
}
-# ---------------------------------------------------------------------------
+# ────────────────────────────────────────────────────────────────────────────
# Pydantic schemas
-# ---------------------------------------------------------------------------
+# ────────────────────────────────────────────────────────────────────────────
-
-class ControlOut(BaseModel):
- id: str
- framework: str
- control_id: str
- category: str
- title: str
- description: str | None
-
-
-class EvidenceOut(BaseModel):
- id: str
- control_id: str
- evidence_type: str
- title: str
- description: str | None
- status: str
- collected_at: str
+EvidenceKind = Literal["alert", "log", "screenshot", "attestation", "policy", "runbook", "other"]
+EvidenceStatus = Literal["pending", "accepted", "rejected"]
-class ControlWithEvidence(BaseModel):
- control: ControlOut
- evidence: list[EvidenceOut]
- latest_status: str
+class CollectEvidenceRequest(BaseModel):
+ framework: str = Field(..., description="Compliance framework key e.g. 'SOC2'.")
+ control_id: str = Field(..., description="Control identifier e.g. 'CC7.2'.")
+ control_title: str | None = None
+ evidence_kind: EvidenceKind = "alert"
+ summary: str = Field(..., min_length=5)
+ raw_payload: dict[str, Any] = Field(default_factory=dict)
+ case_id: uuid.UUID | None = None
-class FrameworkSummary(BaseModel):
- total: int
- collected: int
- review: int
- approved: int
- rejected: int
- missing: int
- pct: int
+class ReviewEvidenceRequest(BaseModel):
+ decision: Literal["accepted", "rejected"]
+ reviewer: str | None = None
-class FrameworkResponse(BaseModel):
+class EvidenceResponse(BaseModel):
+ id: uuid.UUID
+ case_id: uuid.UUID | None
framework: str
- framework_name: str
- summary: FrameworkSummary
- controls: list[ControlWithEvidence]
-
-
-class SOC2Summary(BaseModel):
- total: int
- collected: int
- review: int
- approved: int
- rejected: int
- pct: int
-
+ control_id: str
+ control_title: str | None
+ evidence_kind: str
+ summary: str
+ raw_payload: dict[str, Any]
+ payload_hash: str | None
+ prev_hash: str | None
+ collected_at: datetime
+ reviewed_by: str | None
+ reviewed_at: datetime | None
+ status: str
+ created_at: datetime
-class SOC2Response(BaseModel):
- summary: SOC2Summary
- controls: list[ControlWithEvidence]
+class FrameworksResponse(BaseModel):
+ frameworks: dict[str, dict[str, str]]
-class HeatmapCell(BaseModel):
- category: str
- status: str
- count: int
-
-class HeatmapResponse(BaseModel):
+class CompliancePosture(BaseModel):
framework: str
- framework_name: str
- categories: list[str]
- statuses: list[str]
- cells: list[HeatmapCell]
-
+ total_evidence: int
+ accepted: int
+ pending: int
+ rejected: int
+ coverage_pct: float # accepted / known controls * 100
+ controls_covered: list[str]
+ controls_missing: list[str]
+ generated_at: datetime
-class FrameworkInfo(BaseModel):
- id: str
- name: str
- control_count: int
+# ────────────────────────────────────────────────────────────────────────────
+# Helpers
+# ────────────────────────────────────────────────────────────────────────────
-# ---------------------------------------------------------------------------
-# Helper
-# ---------------------------------------------------------------------------
+def _compute_hash(prev_hash: str | None, summary: str, payload: dict[str, Any]) -> str:
+ raw = (prev_hash or "") + summary + json.dumps(payload, sort_keys=True)
+ return hashlib.sha256(raw.encode()).hexdigest()
-async def _get_framework_data(
- db: TenantDBSession,
- tenant_id: uuid.UUID,
- framework: str,
-) -> tuple[list[ComplianceControl], dict[uuid.UUID, list[ComplianceEvidence]]]:
- """Fetch controls and grouped evidence for a framework."""
- ctrl_result = await db.execute(
- select(ComplianceControl).where(ComplianceControl.framework == framework).order_by(ComplianceControl.control_id)
- )
- controls = ctrl_result.scalars().all()
- ev_result = await db.execute(
- select(ComplianceEvidence)
- .where(
- ComplianceEvidence.tenant_id == tenant_id,
+async def _latest_hash(db: DBSession, framework: str) -> str | None:
+ row = (
+ await db.execute(
+ text("SELECT payload_hash FROM aisoc_compliance_evidence WHERE framework = :f ORDER BY created_at DESC LIMIT 1").bindparams(f=framework)
)
- .join(ComplianceControl, ComplianceEvidence.control_id == ComplianceControl.id)
- .where(ComplianceControl.framework == framework)
- .order_by(ComplianceEvidence.collected_at.desc())
+ ).fetchone()
+ return row.payload_hash if row else None
+
+
+def _row_to_evidence(row: Any) -> EvidenceResponse:
+ return EvidenceResponse(
+ id=row.id,
+ case_id=row.case_id,
+ framework=row.framework,
+ control_id=row.control_id,
+ control_title=row.control_title,
+ evidence_kind=row.evidence_kind,
+ summary=row.summary,
+ raw_payload=dict(row.raw_payload or {}),
+ payload_hash=row.payload_hash,
+ prev_hash=row.prev_hash,
+ collected_at=row.collected_at,
+ reviewed_by=row.reviewed_by,
+ reviewed_at=row.reviewed_at,
+ status=row.status,
+ created_at=row.created_at,
)
- all_evidence = ev_result.scalars().all()
-
- ev_by_control: dict[uuid.UUID, list[ComplianceEvidence]] = {}
- for ev in all_evidence:
- ev_by_control.setdefault(ev.control_id, []).append(ev)
-
- return controls, ev_by_control
-
-
-def _build_controls_with_evidence(
- controls: list[ComplianceControl],
- ev_by_control: dict[uuid.UUID, list[ComplianceEvidence]],
- max_evidence: int = 5,
-) -> list[ControlWithEvidence]:
- result = []
- for ctrl in controls:
- evs = ev_by_control.get(ctrl.id, [])
- latest_status = evs[0].status if evs else "missing"
- result.append(
- ControlWithEvidence(
- control=ControlOut(
- id=str(ctrl.id),
- framework=ctrl.framework,
- control_id=ctrl.control_id,
- category=ctrl.category,
- title=ctrl.title,
- description=ctrl.description,
- ),
- evidence=[
- EvidenceOut(
- id=str(e.id),
- control_id=str(e.control_id),
- evidence_type=e.evidence_type,
- title=e.title,
- description=e.description,
- status=e.status,
- collected_at=e.collected_at.isoformat(),
- )
- for e in evs[:max_evidence]
- ],
- latest_status=latest_status,
- )
- )
- return result
-
-
-def _compute_summary(controls_with_ev: list[ControlWithEvidence]) -> dict:
- total = len(controls_with_ev)
- status_counts: dict[str, int] = {}
- for cwev in controls_with_ev:
- s = cwev.latest_status
- status_counts[s] = status_counts.get(s, 0) + 1
- collected = sum(status_counts.get(s, 0) for s in ("collected", "approved"))
- return {
- "total": total,
- "collected": collected,
- "review": status_counts.get("review", 0),
- "approved": status_counts.get("approved", 0),
- "rejected": status_counts.get("rejected", 0),
- "missing": status_counts.get("missing", 0),
- "pct": int(collected / total * 100) if total else 0,
- }
-
-
-# ---------------------------------------------------------------------------
-# Utility: list all frameworks
-# ---------------------------------------------------------------------------
-
-
[email protected]("/frameworks", response_model=list[FrameworkInfo])
-async def list_frameworks(
- current_user: Annotated[AuthUser, Depends(require_permission("compliance:read"))],
- db: TenantDBSession,
-) -> list[FrameworkInfo]:
- """Return all supported compliance frameworks with control counts."""
- result = await db.execute(
- select(ComplianceControl.framework, func.count(ComplianceControl.id).label("cnt")).group_by(ComplianceControl.framework)
- )
- counts = {row.framework: row.cnt for row in result}
-
- return [
- FrameworkInfo(
- id=fw_id,
- name=fw_name,
- control_count=counts.get(fw_id, 0),
- )
- for fw_id, fw_name in SUPPORTED_FRAMEWORKS.items()
- if counts.get(fw_id, 0) > 0 or fw_id in counts
- ]
-# ---------------------------------------------------------------------------
-# SOC 2 dedicated routes (kept for backward-compat)
-# ---------------------------------------------------------------------------
-
-
[email protected]("/soc2", response_model=SOC2Response)
-async def get_soc2_dashboard(
- current_user: Annotated[AuthUser, Depends(require_permission("compliance:read"))],
- db: TenantDBSession,
-) -> SOC2Response:
- """Return SOC 2 evidence dashboard for the current tenant."""
- controls, ev_by_control = await _get_framework_data(db, current_user.tenant_id, "soc2")
- controls_with_evidence = _build_controls_with_evidence(controls, ev_by_control)
- summary = await get_soc2_summary(db, current_user.tenant_id)
-
- return SOC2Response(
- summary=SOC2Summary(**summary),
- controls=controls_with_evidence,
+# ────────────────────────────────────────────────────────────────────────────
+# Endpoints
+# ────────────────────────────────────────────────────────────────────────────
+
+
[email protected]("/frameworks", response_model=FrameworksResponse, summary="List frameworks and controls")
+async def list_frameworks() -> FrameworksResponse:
+ return FrameworksResponse(frameworks=FRAMEWORKS)
+
+
[email protected]("/evidence", response_model=EvidenceResponse, status_code=status.HTTP_201_CREATED, summary="Collect evidence item")
+async def collect_evidence(body: CollectEvidenceRequest, db: DBSession, user: AuthUser) -> EvidenceResponse:
+ prev_hash = await _latest_hash(db, body.framework)
+ new_hash = _compute_hash(prev_hash, body.summary, body.raw_payload)
+ now = datetime.now(UTC)
+ evidence_id = uuid.uuid4()
+
+ q = text("""
+ INSERT INTO aisoc_compliance_evidence (
+ id, case_id, framework, control_id, control_title,
+ evidence_kind, summary, raw_payload, payload_hash, prev_hash,
+ collected_at, status, created_at
+ ) VALUES (
+ :id, :case_id, :fw, :ctrl, :title,
+ :kind, :summary, :payload::jsonb, :hash, :prev,
+ :now, 'pending', :now
+ ) RETURNING *
+ """).bindparams(
+ id=evidence_id,
+ case_id=body.case_id,
+ fw=body.framework,
+ ctrl=body.control_id,
+ title=body.control_title or FRAMEWORKS.get(body.framework, {}).get(body.control_id),
+ kind=body.evidence_kind,
+ summary=body.summary,
+ payload=json.dumps(body.raw_payload),
+ hash=new_hash,
+ prev=prev_hash,
+ now=now,
)
-
-
[email protected]("/soc2/collect", status_code=202)
-async def collect_soc2_evidence(
- current_user: Annotated[AuthUser, Depends(require_permission("compliance:write"))],
- db: TenantDBSession,
-) -> dict:
- """Trigger automatic evidence collection for all SOC 2 controls."""
- collected = await auto_collect_evidence(db, current_user.tenant_id, framework="soc2")
- await db.commit()
-
- return {
- "message": f"Collected {len(collected)} evidence items",
- "count": len(collected),
- "collected_at": datetime.utcnow().isoformat(),
- }
-
-
[email protected]("/soc2/export")
-async def export_soc2_evidence(
- current_user: Annotated[AuthUser, Depends(require_permission("compliance:read"))],
- db: TenantDBSession,
-) -> dict:
- """Export SOC 2 evidence as structured JSON for PDF generation."""
- controls, ev_by_control = await _get_framework_data(db, current_user.tenant_id, "soc2")
- controls_with_evidence = _build_controls_with_evidence(controls, ev_by_control, max_evidence=3)
- summary = await get_soc2_summary(db, current_user.tenant_id)
-
- return {
- "report_type": "SOC 2 Type I Evidence Report",
- "generated_at": datetime.utcnow().isoformat(),
- "tenant_id": str(current_user.tenant_id),
- "summary": summary,
- "controls": [
- {
- "control_id": cwev.control.control_id,
- "category": cwev.control.category,
- "title": cwev.control.title,
- "description": cwev.control.description,
- "evidence": [
- {
- "type": e.evidence_type,
- "title": e.title,
- "description": e.description,
- "status": e.status,
- "collected_at": e.collected_at,
- }
- for e in cwev.evidence
- ],
- "latest_status": cwev.latest_status,
- }
- for cwev in controls_with_evidence
- ],
- }
-
-
-# ---------------------------------------------------------------------------
-# Generic framework routes
-# ---------------------------------------------------------------------------
-
-
-def _validate_framework(framework: str) -> None:
- if framework not in SUPPORTED_FRAMEWORKS:
- raise HTTPException(
- status_code=404,
- detail=f"Framework '{framework}' not found. Supported: {list(SUPPORTED_FRAMEWORKS.keys())}",
+ try:
+ row = (await db.execute(q)).fetchone()
+ await db.commit()
+ return _row_to_evidence(row)
+ except Exception as exc:
+ await db.rollback()
+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc
+
+
[email protected]("/evidence", response_model=list[EvidenceResponse], summary="List evidence items")
+async def list_evidence(
+ db: DBSession,
+ user: AuthUser,
+ framework: str | None = Query(None),
+ control_id: str | None = Query(None),
+ case_id: uuid.UUID | None = Query(None),
+ ev_status: str | None = Query(None, alias="status"),
+ limit: int = Query(100, ge=1, le=1000),
+ offset: int = Query(0, ge=0),
+) -> list[EvidenceResponse]:
+ wheres = ["1=1"]
+ params: dict[str, Any] = {"limit": limit, "offset": offset}
+ if framework:
+ wheres.append("framework = :fw"); params["fw"] = framework
+ if control_id:
+ wheres.append("control_id = :ctrl"); params["ctrl"] = control_id
+ if case_id:
+ wheres.append("case_id = :case_id"); params["case_id"] = case_id
+ if ev_status:
+ wheres.append("status = :ev_status"); params["ev_status"] = ev_status
+
+ q = text(f"SELECT * FROM aisoc_compliance_evidence WHERE {' AND '.join(wheres)} ORDER BY collected_at DESC LIMIT :limit OFFSET :offset").bindparams(**params)
+ try:
+ rows = (await db.execute(q)).fetchall()
+ return [_row_to_evidence(r) for r in rows]
+ except Exception as exc:
+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc
+
+
[email protected]("/evidence/{evidence_id}", response_model=EvidenceResponse, summary="Get evidence item")
+async def get_evidence(evidence_id: uuid.UUID, db: DBSession, user: AuthUser) -> EvidenceResponse:
+ row = (await db.execute(text("SELECT * FROM aisoc_compliance_evidence WHERE id = :id").bindparams(id=evidence_id))).fetchone()
+ if not row:
+ raise HTTPException(status_code=404, detail="Evidence item not found.")
+ return _row_to_evidence(row)
+
+
[email protected]("/evidence/{evidence_id}/review", response_model=EvidenceResponse, summary="Accept or reject evidence")
+async def review_evidence(evidence_id: uuid.UUID, body: ReviewEvidenceRequest, db: DBSession, user: AuthUser) -> EvidenceResponse:
+ now = datetime.now(UTC)
+ q = text("""
+ UPDATE aisoc_compliance_evidence
+ SET status = :decision, reviewed_by = :reviewer, reviewed_at = :now
+ WHERE id = :id RETURNING *
+ """).bindparams(id=evidence_id, decision=body.decision, reviewer=body.reviewer or str(user), now=now)
+ try:
+ row = (await db.execute(q)).fetchone()
+ if not row:
+ raise HTTPException(status_code=404, detail="Evidence item not found.")
+ await db.commit()
+ return _row_to_evidence(row)
+ except HTTPException:
+ raise
+ except Exception as exc:
+ await db.rollback()
+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc
+
+
[email protected]("/report", response_model=list[CompliancePosture], summary="Generate compliance posture report")
+async def compliance_report(
+ db: DBSession,
+ user: AuthUser,
+ framework: str | None = Query(None, description="Filter to a single framework."),
+) -> list[CompliancePosture]:
+ wheres = ["1=1"]
+ params: dict[str, Any] = {}
+ if framework:
+ wheres.append("framework = :fw"); params["fw"] = framework
+
+ q = text(f"""
+ SELECT framework, control_id,
+ COUNT(*) AS total,
+ COUNT(*) FILTER (WHERE status = 'accepted') AS accepted,
+ COUNT(*) FILTER (WHERE status = 'pending') AS pending,
+ COUNT(*) FILTER (WHERE status = 'rejected') AS rejected
+ FROM aisoc_compliance_evidence
+ WHERE {' AND '.join(wheres)}
+ GROUP BY framework, control_id
+ ORDER BY framework, control_id
+ """).bindparams(**params)
+
+ try:
+ rows = (await db.execute(q)).fetchall()
+ except Exception as exc:
+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc
+
+ # Aggregate by framework
+ by_fw: dict[str, Any] = {}
+ for r in rows:
+ fw = r.framework
+ if fw not in by_fw:
+ by_fw[fw] = {"total": 0, "accepted": 0, "pending": 0, "rejected": 0, "covered": set()}
+ by_fw[fw]["total"] += r.total
+ by_fw[fw]["accepted"] += r.accepted
+ by_fw[fw]["pending"] += r.pending
+ by_fw[fw]["rejected"] += r.rejected
+ if r.accepted > 0:
+ by_fw[fw]["covered"].add(r.control_id)
+
+ report = []
+ target_fws = [framework] if framework else list(by_fw.keys()) + [f for f in FRAMEWORKS if f not in by_fw]
+ for fw in dict.fromkeys(target_fws):
+ known = FRAMEWORKS.get(fw, {})
+ data = by_fw.get(fw, {"total": 0, "accepted": 0, "pending": 0, "rejected": 0, "covered": set()})
+ covered = data["covered"]
+ missing = [c for c in known if c not in covered]
+ pct = (len(covered) / len(known) * 100) if known else 0.0
+ report.append(
+ CompliancePosture(
+ framework=fw,
+ total_evidence=data["total"],
+ accepted=data["accepted"],
+ pending=data["pending"],
+ rejected=data["rejected"],
+ coverage_pct=round(pct, 1),
+ controls_covered=sorted(covered),
+ controls_missing=sorted(missing),
+ generated_at=datetime.now(UTC),
+ )
)
-
-
[email protected]("/{framework}", response_model=FrameworkResponse)
-async def get_framework_dashboard(
- framework: str,
- current_user: Annotated[AuthUser, Depends(require_permission("compliance:read"))],
- db: TenantDBSession,
-) -> FrameworkResponse:
- """Return compliance dashboard for the given framework."""
- _validate_framework(framework)
-
- controls, ev_by_control = await _get_framework_data(db, current_user.tenant_id, framework)
- controls_with_evidence = _build_controls_with_evidence(controls, ev_by_control)
- summary_dict = _compute_summary(controls_with_evidence)
-
- return FrameworkResponse(
- framework=framework,
- framework_name=SUPPORTED_FRAMEWORKS[framework],
- summary=FrameworkSummary(**summary_dict),
- controls=controls_with_evidence,
- )
-
-
[email protected]("/{framework}/heatmap", response_model=HeatmapResponse)
-async def get_framework_heatmap(
- framework: str,
- current_user: Annotated[AuthUser, Depends(require_permission("compliance:read"))],
- db: TenantDBSession,
-) -> HeatmapResponse:
- """Return heatmap data (category × status counts) for the given framework."""
- _validate_framework(framework)
-
- controls, ev_by_control = await _get_framework_data(db, current_user.tenant_id, framework)
- controls_with_evidence = _build_controls_with_evidence(controls, ev_by_control)
-
- # Build category × status matrix
- cell_map: dict[tuple[str, str], int] = {}
- categories_ordered: list[str] = []
- for cwev in controls_with_evidence:
- cat = cwev.control.category
- st = cwev.latest_status
- cell_map[(cat, st)] = cell_map.get((cat, st), 0) + 1
- if cat not in categories_ordered:
- categories_ordered.append(cat)
-
- statuses = ["approved", "collected", "review", "rejected", "missing"]
-
- cells = [HeatmapCell(category=cat, status=st, count=cell_map.get((cat, st), 0)) for cat in categories_ordered for st in statuses]
-
- return HeatmapResponse(
- framework=framework,
- framework_name=SUPPORTED_FRAMEWORKS[framework],
- categories=categories_ordered,
- statuses=statuses,
- cells=cells,
- )
-
-
[email protected]("/{framework}/collect", status_code=202)
-async def collect_framework_evidence(
- framework: str,
- current_user: Annotated[AuthUser, Depends(require_permission("compliance:write"))],
- db: TenantDBSession,
-) -> dict:
- """Trigger automatic evidence collection for the given framework."""
- _validate_framework(framework)
-
- collected = await auto_collect_evidence(db, current_user.tenant_id, framework=framework)
- await db.commit()
-
- return {
- "framework": framework,
- "message": f"Collected {len(collected)} evidence items",
- "count": len(collected),
- "collected_at": datetime.utcnow().isoformat(),
- }
-
-
[email protected]("/{framework}/export")
-async def export_framework_evidence(
- framework: str,
- current_user: Annotated[AuthUser, Depends(require_permission("compliance:read"))],
- db: TenantDBSession,
-) -> dict:
- """Export compliance evidence for the given framework as structured JSON."""
- _validate_framework(framework)
-
- controls, ev_by_control = await _get_framework_data(db, current_user.tenant_id, framework)
- controls_with_evidence = _build_controls_with_evidence(controls, ev_by_control, max_evidence=3)
- summary_dict = _compute_summary(controls_with_evidence)
-
- return {
- "report_type": f"{SUPPORTED_FRAMEWORKS[framework]} Evidence Report",
- "framework": framework,
- "framework_name": SUPPORTED_FRAMEWORKS[framework],
- "generated_at": datetime.utcnow().isoformat(),
- "tenant_id": str(current_user.tenant_id),
- "summary": summary_dict,
- "controls": [
- {
- "control_id": cwev.control.control_id,
- "category": cwev.control.category,
- "title": cwev.control.title,
- "description": cwev.control.description,
- "evidence": [
- {
- "type": e.evidence_type,
- "title": e.title,
- "description": e.description,
- "status": e.status,
- "collected_at": e.collected_at,
- }
- for e in cwev.evidence
- ],
- "latest_status": cwev.latest_status,
- }
- for cwev in controls_with_evidence
- ],
- }
+ return report
diff --git a/services/api/app/api/v1/endpoints/detection_loop.py b/services/api/app/api/v1/endpoints/detection_loop.py
new file mode 100644
index 000000000..022128816
--- /dev/null
+++ b/services/api/app/api/v1/endpoints/detection_loop.py
@@ -0,0 +1,290 @@
+"""Closed-loop detection engineering (tier2-detection-loop).
+
+When an analyst marks an alert as a false-positive (FP), this module:
+
+1. Retrieves the triggering detection rule and the alert's raw evidence.
+2. Invokes an LLM to draft a Sigma YAML improvement that would suppress
+ the FP without widening the exclusion too broadly.
+3. Creates a ``DetectionRuleProposal`` via the existing DAC lifecycle so the
+ suggestion is eval-gated before promotion.
+4. Returns the draft proposal ID and the diff for the analyst to review.
+
+Endpoints
+---------
+* ``POST /detection-loop/suggest`` Trigger FP → Sigma draft.
+* ``GET /detection-loop/suggestions`` List LLM-drafted suggestions.
+* ``GET /detection-loop/suggestions/{id}`` Detail of one suggestion.
+"""
+
+from __future__ import annotations
+
+import json
+import textwrap
+import uuid
+from datetime import UTC, datetime
+from typing import Any
+
+import httpx
+from fastapi import APIRouter, Depends, HTTPException, status
+from pydantic import BaseModel, Field
+from sqlalchemy import select, text
+
+from app.api.v1.deps import AuthUser, DBSession, require_permission
+from app.core.config import settings
+
+router = APIRouter(prefix="/detection-loop", tags=["detection_rules", "detection_loop"])
+
+# ────────────────────────────────────────────────────────────────────────────
+# Pydantic schemas
+# ────────────────────────────────────────────────────────────────────────────
+
+
+class SuggestRequest(BaseModel):
+ alert_id: uuid.UUID = Field(..., description="ID of the FP-flagged alert.")
+ analyst_note: str | None = Field(
+ None, description="Free-text note explaining why this is a FP."
+ )
+
+
+class SuggestionResponse(BaseModel):
+ suggestion_id: uuid.UUID
+ alert_id: uuid.UUID
+ base_rule_id: uuid.UUID | None
+ draft_rule_name: str
+ draft_sigma_yaml: str
+ rationale: str
+ proposal_id: uuid.UUID | None
+ created_at: datetime
+
+
+class SuggestionListResponse(BaseModel):
+ suggestions: list[SuggestionResponse]
+ total: int
+
+
+# ────────────────────────────────────────────────────────────────────────────
+# LLM helper
+# ────────────────────────────────────────────────────────────────────────────
+
+_SYS_PROMPT = textwrap.dedent(
+ """
+ You are a senior detection engineer reviewing a false-positive alert.
+ The user will provide:
+ - The Sigma rule that triggered the alert (YAML).
+ - Key fields from the alert that fired.
+ - An analyst note explaining why this is benign.
+
+ Your task is to produce a *minimal, targeted* Sigma rule modification that
+ suppresses this class of false-positive without unduly widening the exclusion.
+ Prefer:
+ - Adding a `filter` condition rather than removing detection logic.
+ - Scoping exclusions to specific processes, users, or source paths when
+ the evidence supports it.
+ - Keeping the rule's ATT&CK technique tags unchanged.
+
+ Respond in JSON only with this schema:
+ {
+ "rule_name": "...", // may append '-v2' or '-fp-fix'
+ "sigma_yaml": "...", // full updated Sigma YAML
+ "rationale": "..." // ≤ 3 sentences explaining the change
+ }
+ """
+).strip()
+
+
+async def _llm_draft_sigma(
+ current_sigma: str,
+ alert_fields: dict[str, Any],
+ analyst_note: str,
+) -> dict[str, Any]:
+ """Call LLM to draft a Sigma improvement. Returns parsed JSON or template."""
+ api_key = getattr(settings, "OPENAI_API_KEY", None) or getattr(
+ settings, "LLM_API_KEY", None
+ )
+ if not api_key:
+ return _template_fallback(current_sigma, alert_fields, analyst_note)
+
+ user_msg = json.dumps(
+ {
+ "current_sigma_rule": current_sigma,
+ "alert_fields": alert_fields,
+ "analyst_note": analyst_note,
+ },
+ indent=2,
+ )
+ try:
+ async with httpx.AsyncClient(timeout=30) as client:
+ resp = await client.post(
+ "https://api.openai.com/v1/chat/completions",
+ headers={"Authorization": f"Bearer {api_key}"},
+ json={
+ "model": "gpt-4o-mini",
+ "response_format": {"type": "json_object"},
+ "messages": [
+ {"role": "system", "content": _SYS_PROMPT},
+ {"role": "user", "content": user_msg},
+ ],
+ },
+ )
+ resp.raise_for_status()
+ raw = resp.json()["choices"][0]["message"]["content"]
+ return json.loads(raw)
+ except Exception:
+ return _template_fallback(current_sigma, alert_fields, analyst_note)
+
+
+def _template_fallback(
+ current_sigma: str, alert_fields: dict[str, Any], analyst_note: str
+) -> dict[str, Any]:
+ """Return a structured template when no LLM key is configured."""
+ proc = alert_fields.get("process_name") or alert_fields.get("Image") or "UNKNOWN"
+ user = alert_fields.get("user") or alert_fields.get("User") or "UNKNOWN"
+ return {
+ "rule_name": "fp-exclusion-draft",
+ "sigma_yaml": textwrap.dedent(
+ f"""\
+ # AUTO-DRAFTED FP EXCLUSION — review before promoting
+ # Original analyst note: {analyst_note}
+ # Alert fields: {json.dumps(alert_fields, default=str)[:200]}
+ filter:
+ - process_name: '{proc}'
+ - User: '{user}'
+ condition: selection and not filter
+ """
+ ),
+ "rationale": (
+ f"Auto-generated exclusion for process '{proc}' / user '{user}'. "
+ "Review and tighten before promoting. "
+ "Analyst note: " + (analyst_note or "none provided")
+ ),
+ }
+
+
+# ────────────────────────────────────────────────────────────────────────────
+# In-memory store (replace with DB table in prod)
+# ────────────────────────────────────────────────────────────────────────────
+
+_SUGGESTIONS: dict[uuid.UUID, dict[str, Any]] = {}
+
+
+# ────────────────────────────────────────────────────────────────────────────
+# Endpoints
+# ────────────────────────────────────────────────────────────────────────────
+
+
+ "/suggest",
+ response_model=SuggestionResponse,
+ status_code=status.HTTP_201_CREATED,
+ summary="Draft Sigma improvement for a FP alert",
+)
+async def suggest_fp_fix(
+ body: SuggestRequest,
+ db: DBSession,
+ user: AuthUser,
+) -> SuggestionResponse:
+ """Retrieve the triggering rule + alert evidence, then draft a Sigma improvement."""
+ # 1. Load alert
+ row = await db.execute(
+ text(
+ "SELECT rule_id, evidence, tenant_id FROM aisoc_alerts WHERE id = :aid"
+ ).bindparams(aid=body.alert_id)
+ )
+ alert_row = row.fetchone()
+ if not alert_row:
+ raise HTTPException(status_code=404, detail="Alert not found")
+
+ rule_id = alert_row.rule_id
+ evidence: dict[str, Any] = alert_row.evidence or {}
+ tenant_id: uuid.UUID | None = alert_row.tenant_id
+
+ # 2. Load rule body if available
+ current_sigma = "# Rule body not found\n"
+ if rule_id:
+ rule_row = await db.execute(
+ text(
+ "SELECT rule_body FROM aisoc_detection_rules WHERE id = :rid"
+ ).bindparams(rid=rule_id)
+ )
+ rule_data = rule_row.fetchone()
+ if rule_data:
+ current_sigma = rule_data.rule_body
+
+ # 3. Draft via LLM
+ draft = await _llm_draft_sigma(
+ current_sigma=current_sigma,
+ alert_fields=evidence,
+ analyst_note=body.analyst_note or "",
+ )
+
+ suggestion_id = uuid.uuid4()
+ now = datetime.now(UTC)
+
+ # 4. Auto-create a DAC proposal
+ proposal_id: uuid.UUID | None = None
+ try:
+ proposal_id = uuid.uuid4()
+ await db.execute(
+ text(
+ """
+ INSERT INTO aisoc_detection_rule_proposals
+ (id, tenant_id, base_rule_id, name, description,
+ rule_language, rule_body, category, severity, confidence,
+ status, source, created_at, updated_at)
+ VALUES
+ (:id, :tid, :rid, :name, :desc,
+ 'sigma', :body, 'fp-fix', 'low', 70,
+ 'draft', 'detection-loop', :now, :now)
+ """
+ ).bindparams(
+ id=proposal_id,
+ tid=tenant_id,
+ rid=rule_id,
+ name=draft.get("rule_name", "fp-exclusion-draft"),
+ desc=draft.get("rationale", ""),
+ body=draft.get("sigma_yaml", ""),
+ now=now,
+ )
+ )
+ await db.commit()
+ except Exception:
+ proposal_id = None
+ await db.rollback()
+
+ result = SuggestionResponse(
+ suggestion_id=suggestion_id,
+ alert_id=body.alert_id,
+ base_rule_id=rule_id,
+ draft_rule_name=draft.get("rule_name", "fp-exclusion-draft"),
+ draft_sigma_yaml=draft.get("sigma_yaml", ""),
+ rationale=draft.get("rationale", ""),
+ proposal_id=proposal_id,
+ created_at=now,
+ )
+ _SUGGESTIONS[suggestion_id] = result.model_dump()
+ return result
+
+
+ "/suggestions",
+ response_model=SuggestionListResponse,
+ summary="List LLM-drafted Sigma suggestions",
+)
+async def list_suggestions(user: AuthUser) -> SuggestionListResponse:
+ items = [SuggestionResponse(**v) for v in _SUGGESTIONS.values()]
+ return SuggestionListResponse(suggestions=items, total=len(items))
+
+
+ "/suggestions/{suggestion_id}",
+ response_model=SuggestionResponse,
+ summary="Get one Sigma suggestion",
+)
+async def get_suggestion(
+ suggestion_id: uuid.UUID,
+ user: AuthUser,
+) -> SuggestionResponse:
+ item = _SUGGESTIONS.get(suggestion_id)
+ if not item:
+ raise HTTPException(status_code=404, detail="Suggestion not found")
+ return SuggestionResponse(**item)
diff --git a/services/api/app/api/v1/endpoints/feedback.py b/services/api/app/api/v1/endpoints/feedback.py
new file mode 100644
index 000000000..62fb602cd
--- /dev/null
+++ b/services/api/app/api/v1/endpoints/feedback.py
@@ -0,0 +1,146 @@
+"""Analyst override feedback endpoint.
+
+Allows analysts to correct AI-generated alert verdicts. Corrections are:
+ 1. Persisted to the alerts table (disposition field).
+ 2. Published as an event for downstream institutional memory ingestion.
+ 3. Used to update SOC FPR metrics.
+"""
+
+from __future__ import annotations
+
+from datetime import UTC, datetime
+
+from fastapi import APIRouter, HTTPException, status
+from pydantic import BaseModel, Field
+from sqlalchemy import select, update
+
+from app.api.v1.deps import AuthUser, DBSession
+from app.models.alert import Alert
+
+import structlog
+
+logger = structlog.get_logger()
+
+router = APIRouter(prefix="/feedback", tags=["feedback"])
+
+
+class AlertOverrideRequest(BaseModel):
+ alert_id: str
+ original_verdict: str = Field(..., description="The AI-generated verdict being overridden")
+ corrected_verdict: str = Field(
+ ...,
+ description="Analyst's verdict: true_positive | false_positive | benign | escalate",
+ )
+ reason: str | None = Field(None, description="Optional free-text justification")
+
+
+class AlertOverrideResponse(BaseModel):
+ alert_id: str
+ corrected_verdict: str
+ recorded_at: str
+
+
[email protected]("/alert-override", response_model=AlertOverrideResponse)
+async def submit_alert_override(
+ payload: AlertOverrideRequest,
+ user: AuthUser,
+ db: DBSession,
+) -> AlertOverrideResponse:
+ """Record an analyst verdict correction on an alert.
+
+ This persists the override disposition and emits a structured log event
+ that the institutional memory ingester picks up asynchronously.
+ """
+ valid_verdicts = {"true_positive", "false_positive", "benign", "escalate"}
+ if payload.corrected_verdict not in valid_verdicts:
+ raise HTTPException(
+ status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
+ detail=f"corrected_verdict must be one of: {', '.join(sorted(valid_verdicts))}",
+ )
+
+ # Verify the alert belongs to this tenant
+ row = await db.scalar(
+ select(Alert).where(
+ Alert.id == payload.alert_id,
+ Alert.tenant_id == user.tenant_id,
+ )
+ )
+ if row is None:
+ raise HTTPException(
+ status_code=status.HTTP_404_NOT_FOUND,
+ detail="Alert not found",
+ )
+
+ # Persist disposition on the alert
+ now = datetime.now(UTC)
+ await db.execute(
+ update(Alert)
+ .where(Alert.id == payload.alert_id, Alert.tenant_id == user.tenant_id)
+ .values(
+ disposition=payload.corrected_verdict,
+ updated_at=now,
+ )
+ )
+ await db.commit()
+
+ # Structured log event — picked up by institutional memory ingester
+ logger.info(
+ "analyst.override",
+ tenant_id=user.tenant_id,
+ alert_id=payload.alert_id,
+ analyst_id=str(user.id),
+ original_verdict=payload.original_verdict,
+ corrected_verdict=payload.corrected_verdict,
+ reason=payload.reason,
+ recorded_at=now.isoformat(),
+ )
+
+ return AlertOverrideResponse(
+ alert_id=payload.alert_id,
+ corrected_verdict=payload.corrected_verdict,
+ recorded_at=now.isoformat(),
+ )
+
+
+class OverrideSummaryResponse(BaseModel):
+ total_overrides: int
+ false_positive_corrections: int
+ true_positive_corrections: int
+ benign_corrections: int
+
+
[email protected]("/summary", response_model=OverrideSummaryResponse)
+async def get_override_summary(
+ user: AuthUser,
+ db: DBSession,
+) -> OverrideSummaryResponse:
+ """Return a summary of analyst overrides for this tenant."""
+ from sqlalchemy import func, and_
+
+ total = await db.scalar(
+ select(func.count()).where(
+ and_(Alert.tenant_id == user.tenant_id, Alert.disposition.isnot(None))
+ )
+ ) or 0
+ fp = await db.scalar(
+ select(func.count()).where(
+ and_(Alert.tenant_id == user.tenant_id, Alert.disposition == "false_positive")
+ )
+ ) or 0
+ tp = await db.scalar(
+ select(func.count()).where(
+ and_(Alert.tenant_id == user.tenant_id, Alert.disposition == "true_positive")
+ )
+ ) or 0
+ benign = await db.scalar(
+ select(func.count()).where(
+ and_(Alert.tenant_id == user.tenant_id, Alert.disposition == "benign")
+ )
+ ) or 0
+
+ return OverrideSummaryResponse(
+ total_overrides=total,
+ false_positive_corrections=fp,
+ true_positive_corrections=tp,
+ benign_corrections=benign,
+ )
diff --git a/services/api/app/api/v1/endpoints/hunts.py b/services/api/app/api/v1/endpoints/hunts.py
new file mode 100644
index 000000000..4b9f4462d
--- /dev/null
+++ b/services/api/app/api/v1/endpoints/hunts.py
@@ -0,0 +1,422 @@
+"""Hypothesis-driven hunt workbench (tier2-hunting).
+
+SOC analysts author threat-hunting hypotheses, attach multi-platform queries,
+and track hunt runs and findings — all version-controlled via the DAC pipeline.
+
+Endpoints
+---------
+* ``GET /hunts`` List hunts.
+* ``POST /hunts`` Create a hunt hypothesis.
+* ``GET /hunts/{id}`` Get a hunt.
+* ``PATCH /hunts/{id}`` Update hypothesis / status / priority.
+* ``POST /hunts/{id}/run`` Execute a hunt query (ES|QL live; SPL/KQL templated).
+* ``GET /hunts/{id}/runs`` List run history for a hunt.
+* ``POST /hunts/{id}/findings`` Append finding entries to a hunt.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import uuid
+from datetime import UTC, datetime
+from typing import Any, Literal
+
+import httpx
+from fastapi import APIRouter, HTTPException, Query, status
+from pydantic import BaseModel, Field
+from sqlalchemy import text
+
+from app.api.v1.deps import AuthUser, DBSession
+
+router = APIRouter(prefix="/hunts", tags=["hunts"])
+
+# ────────────────────────────────────────────────────────────────────────────
+# Pydantic schemas
+# ────────────────────────────────────────────────────────────────────────────
+
+HuntStatus = Literal["draft", "active", "completed", "archived"]
+HuntPriority = Literal["low", "medium", "high", "critical"]
+HuntPlatform = Literal["esql", "spl", "kql"]
+
+
+class CreateHuntRequest(BaseModel):
+ title: str = Field(..., min_length=3)
+ hypothesis: str = Field(..., min_length=10, description="Plain-English threat hypothesis.")
+ mitre_tactic: str | None = None
+ mitre_technique: str | None = None
+ priority: HuntPriority = "medium"
+ assigned_to: str | None = None
+ tags: list[str] = Field(default_factory=list)
+
+
+class UpdateHuntRequest(BaseModel):
+ title: str | None = None
+ hypothesis: str | None = None
+ mitre_tactic: str | None = None
+ mitre_technique: str | None = None
+ status: HuntStatus | None = None
+ priority: HuntPriority | None = None
+ assigned_to: str | None = None
+ query_esql: str | None = None
+ query_spl: str | None = None
+ query_kql: str | None = None
+ tags: list[str] | None = None
+
+
+class AddFindingsRequest(BaseModel):
+ findings: list[dict[str, Any]] = Field(..., min_length=1)
+ false_positive_rate: float | None = Field(None, ge=0.0, le=1.0)
+
+
+class RunHuntRequest(BaseModel):
+ platform: HuntPlatform = "esql"
+
+
+class HuntRunResponse(BaseModel):
+ id: uuid.UUID
+ hunt_id: uuid.UUID
+ run_at: datetime
+ platform: str
+ query_used: str | None
+ hit_count: int
+ result_sample: list[dict[str, Any]]
+ duration_ms: int | None
+ error: str | None
+
+
+class HuntResponse(BaseModel):
+ id: uuid.UUID
+ title: str
+ hypothesis: str
+ mitre_tactic: str | None
+ mitre_technique: str | None
+ status: str
+ priority: str
+ query_esql: str | None
+ query_spl: str | None
+ query_kql: str | None
+ findings: list[dict[str, Any]]
+ false_positive_rate: float | None
+ assigned_to: str | None
+ tags: list[str]
+ created_at: datetime
+ updated_at: datetime
+ completed_at: datetime | None
+ created_by: str | None
+
+
+# ────────────────────────────────────────────────────────────────────────────
+# LLM query generation helper
+# ────────────────────────────────────────────────────────────────────────────
+
+_HUNT_SYSTEM = """You are a senior threat hunter. Given a threat hypothesis, generate
+detection queries for the listed platforms.
+
+Return ONLY valid JSON with this structure:
+{
+ "esql": "<ES|QL query string>",
+ "spl": "<Splunk SPL string>",
+ "kql": "<KQL string>"
+}
+No prose. Just JSON."""
+
+
+async def _generate_queries(hypothesis: str, mitre: str | None) -> dict[str, str] | None:
+ api_key = os.getenv("OPENAI_API_KEY") or os.getenv("LLM_API_KEY")
+ if not api_key:
+ return None
+ base_url = os.getenv("LLM_BASE_URL", "https://api.openai.com/v1")
+ model = os.getenv("LLM_MODEL", "gpt-4o-mini")
+ user_msg = f"HYPOTHESIS: {hypothesis}"
+ if mitre:
+ user_msg += f"\nMITRE TECHNIQUE: {mitre}"
+ user_msg += "\nGenerate ES|QL, SPL, and KQL hunt queries."
+ try:
+ async with httpx.AsyncClient(timeout=45) as client:
+ resp = await client.post(
+ f"{base_url}/chat/completions",
+ headers={"Authorization": f"Bearer {api_key}"},
+ json={
+ "model": model,
+ "messages": [
+ {"role": "system", "content": _HUNT_SYSTEM},
+ {"role": "user", "content": user_msg},
+ ],
+ "temperature": 0.2,
+ "response_format": {"type": "json_object"},
+ },
+ )
+ resp.raise_for_status()
+ return json.loads(resp.json()["choices"][0]["message"]["content"])
+ except Exception:
+ return None
+
+
+def _fallback_queries(hypothesis: str) -> dict[str, str]:
+ escaped = hypothesis.replace('"', '\\"')
+ return {
+ "esql": f'FROM logs-* | WHERE message LIKE "%{escaped[:60]}%" | LIMIT 100',
+ "spl": f'index=* "{escaped[:60]}" | head 100',
+ "kql": f'// KQL hunt — adapt field names\nSecurityEvent\n| where Activity has "{escaped[:60]}"\n| limit 100',
+ }
+
+
+# ────────────────────────────────────────────────────────────────────────────
+# Row helper
+# ────────────────────────────────────────────────────────────────────────────
+
+
+def _row_to_hunt(row: Any) -> HuntResponse:
+ return HuntResponse(
+ id=row.id,
+ title=row.title,
+ hypothesis=row.hypothesis,
+ mitre_tactic=row.mitre_tactic,
+ mitre_technique=row.mitre_technique,
+ status=row.status,
+ priority=row.priority,
+ query_esql=row.query_esql,
+ query_spl=row.query_spl,
+ query_kql=row.query_kql,
+ findings=list(row.findings or []),
+ false_positive_rate=row.false_positive_rate,
+ assigned_to=row.assigned_to,
+ tags=list(row.tags or []),
+ created_at=row.created_at,
+ updated_at=row.updated_at,
+ completed_at=row.completed_at,
+ created_by=row.created_by,
+ )
+
+
+# ────────────────────────────────────────────────────────────────────────────
+# Endpoints
+# ────────────────────────────────────────────────────────────────────────────
+
+
[email protected]("", response_model=list[HuntResponse], summary="List hunt hypotheses")
+async def list_hunts(
+ db: DBSession,
+ user: AuthUser,
+ hunt_status: str | None = Query(None, alias="status"),
+ priority: str | None = Query(None),
+ limit: int = Query(50, ge=1, le=500),
+ offset: int = Query(0, ge=0),
+) -> list[HuntResponse]:
+ wheres = ["1=1"]
+ params: dict[str, Any] = {"limit": limit, "offset": offset}
+ if hunt_status:
+ wheres.append("status = :status"); params["status"] = hunt_status
+ if priority:
+ wheres.append("priority = :priority"); params["priority"] = priority
+
+ q = text(f"SELECT * FROM aisoc_hunts WHERE {' AND '.join(wheres)} ORDER BY created_at DESC LIMIT :limit OFFSET :offset").bindparams(**params)
+ try:
+ rows = (await db.execute(q)).fetchall()
+ return [_row_to_hunt(r) for r in rows]
+ except Exception as exc:
+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc
+
+
[email protected]("", response_model=HuntResponse, status_code=status.HTTP_201_CREATED, summary="Create hunt hypothesis")
+async def create_hunt(body: CreateHuntRequest, db: DBSession, user: AuthUser) -> HuntResponse:
+ mitre = body.mitre_technique or body.mitre_tactic
+ queries = await _generate_queries(body.hypothesis, mitre) or _fallback_queries(body.hypothesis)
+ hunt_id = uuid.uuid4()
+ now = datetime.now(UTC)
+ q = text("""
+ INSERT INTO aisoc_hunts (
+ id, title, hypothesis, mitre_tactic, mitre_technique,
+ priority, assigned_to, tags, query_esql, query_spl, query_kql,
+ created_at, updated_at, created_by
+ ) VALUES (
+ :id, :title, :hypothesis, :tactic, :technique,
+ :priority, :assigned, :tags::text[], :esql, :spl, :kql,
+ :now, :now, :user
+ ) RETURNING *
+ """).bindparams(
+ id=hunt_id,
+ title=body.title,
+ hypothesis=body.hypothesis,
+ tactic=body.mitre_tactic,
+ technique=body.mitre_technique,
+ priority=body.priority,
+ assigned=body.assigned_to,
+ tags=body.tags or [],
+ esql=queries.get("esql"),
+ spl=queries.get("spl"),
+ kql=queries.get("kql"),
+ now=now,
+ user=str(user) if user else "system",
+ )
+ try:
+ row = (await db.execute(q)).fetchone()
+ await db.commit()
+ return _row_to_hunt(row)
+ except Exception as exc:
+ await db.rollback()
+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc
+
+
[email protected]("/{hunt_id}", response_model=HuntResponse, summary="Get hunt")
+async def get_hunt(hunt_id: uuid.UUID, db: DBSession, user: AuthUser) -> HuntResponse:
+ row = (await db.execute(text("SELECT * FROM aisoc_hunts WHERE id = :id").bindparams(id=hunt_id))).fetchone()
+ if not row:
+ raise HTTPException(status_code=404, detail="Hunt not found.")
+ return _row_to_hunt(row)
+
+
[email protected]("/{hunt_id}", response_model=HuntResponse, summary="Update hunt")
+async def update_hunt(hunt_id: uuid.UUID, body: UpdateHuntRequest, db: DBSession, user: AuthUser) -> HuntResponse:
+ sets = ["updated_at = :now"]
+ params: dict[str, Any] = {"id": hunt_id, "now": datetime.now(UTC)}
+
+ if body.title is not None:
+ sets.append("title = :title"); params["title"] = body.title
+ if body.hypothesis is not None:
+ sets.append("hypothesis = :hypothesis"); params["hypothesis"] = body.hypothesis
+ if body.mitre_tactic is not None:
+ sets.append("mitre_tactic = :tactic"); params["tactic"] = body.mitre_tactic
+ if body.mitre_technique is not None:
+ sets.append("mitre_technique = :technique"); params["technique"] = body.mitre_technique
+ if body.status is not None:
+ sets.append("status = :status"); params["status"] = body.status
+ if body.status == "completed":
+ sets.append("completed_at = :now")
+ if body.priority is not None:
+ sets.append("priority = :priority"); params["priority"] = body.priority
+ if body.assigned_to is not None:
+ sets.append("assigned_to = :assigned"); params["assigned"] = body.assigned_to
+ if body.query_esql is not None:
+ sets.append("query_esql = :esql"); params["esql"] = body.query_esql
+ if body.query_spl is not None:
+ sets.append("query_spl = :spl"); params["spl"] = body.query_spl
+ if body.query_kql is not None:
+ sets.append("query_kql = :kql"); params["kql"] = body.query_kql
+ if body.tags is not None:
+ sets.append("tags = :tags::text[]"); params["tags"] = body.tags
+
+ q = text(f"UPDATE aisoc_hunts SET {', '.join(sets)} WHERE id = :id RETURNING *").bindparams(**params)
+ try:
+ row = (await db.execute(q)).fetchone()
+ if not row:
+ raise HTTPException(status_code=404, detail="Hunt not found.")
+ await db.commit()
+ return _row_to_hunt(row)
+ except HTTPException:
+ raise
+ except Exception as exc:
+ await db.rollback()
+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc
+
+
[email protected]("/{hunt_id}/run", response_model=HuntRunResponse, status_code=status.HTTP_200_OK, summary="Execute hunt query")
+async def run_hunt(hunt_id: uuid.UUID, body: RunHuntRequest, db: DBSession, user: AuthUser) -> HuntRunResponse:
+ hunt_row = (await db.execute(text("SELECT * FROM aisoc_hunts WHERE id = :id").bindparams(id=hunt_id))).fetchone()
+ if not hunt_row:
+ raise HTTPException(status_code=404, detail="Hunt not found.")
+
+ query_map = {"esql": hunt_row.query_esql, "spl": hunt_row.query_spl, "kql": hunt_row.query_kql}
+ query = query_map.get(body.platform)
+ hit_count = 0
+ result_sample: list[dict[str, Any]] = []
+ error_msg: str | None = None
+ start = datetime.now(UTC)
+
+ if body.platform == "esql" and query:
+ es_url = os.getenv("ELASTICSEARCH_URL")
+ es_key = os.getenv("ELASTICSEARCH_API_KEY")
+ if es_url and es_key:
+ try:
+ async with httpx.AsyncClient(timeout=30) as client:
+ resp = await client.post(
+ f"{es_url.rstrip('/')}/_query",
+ headers={"Authorization": f"ApiKey {es_key}", "Content-Type": "application/json"},
+ json={"query": query},
+ )
+ if resp.status_code == 200:
+ data = resp.json()
+ rows = data.get("rows", [])
+ cols = [c["name"] for c in data.get("columns", [])]
+ hit_count = len(rows)
+ result_sample = [{cols[i]: row[i] for i in range(len(cols))} for row in rows[:20]]
+ else:
+ error_msg = f"ES returned {resp.status_code}: {resp.text[:200]}"
+ except Exception as exc:
+ error_msg = str(exc)[:300]
+ else:
+ error_msg = "ELASTICSEARCH_URL or ELASTICSEARCH_API_KEY not configured."
+ elif not query:
+ error_msg = f"No {body.platform} query set for this hunt."
+ else:
+ error_msg = f"{body.platform} live execution not supported — query stored for manual use."
+
+ duration_ms = int((datetime.now(UTC) - start).total_seconds() * 1000)
+ run_id = uuid.uuid4()
+ q = text("""
+ INSERT INTO aisoc_hunt_runs (id, hunt_id, platform, query_used, hit_count, result_sample, duration_ms, error)
+ VALUES (:id, :hunt_id, :platform, :query, :hits, :sample::jsonb, :dur, :err) RETURNING *
+ """).bindparams(
+ id=run_id,
+ hunt_id=hunt_id,
+ platform=body.platform,
+ query=query,
+ hits=hit_count,
+ sample=json.dumps(result_sample),
+ dur=duration_ms,
+ err=error_msg,
+ )
+ try:
+ row = (await db.execute(q)).fetchone()
+ await db.commit()
+ return HuntRunResponse(
+ id=row.id, hunt_id=row.hunt_id, run_at=row.run_at,
+ platform=row.platform, query_used=row.query_used,
+ hit_count=row.hit_count, result_sample=list(row.result_sample or []),
+ duration_ms=row.duration_ms, error=row.error,
+ )
+ except Exception as exc:
+ await db.rollback()
+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc
+
+
[email protected]("/{hunt_id}/runs", response_model=list[HuntRunResponse], summary="List hunt runs")
+async def list_runs(hunt_id: uuid.UUID, db: DBSession, user: AuthUser) -> list[HuntRunResponse]:
+ rows = (await db.execute(text("SELECT * FROM aisoc_hunt_runs WHERE hunt_id = :id ORDER BY run_at DESC LIMIT 100").bindparams(id=hunt_id))).fetchall()
+ return [
+ HuntRunResponse(
+ id=r.id, hunt_id=r.hunt_id, run_at=r.run_at,
+ platform=r.platform, query_used=r.query_used,
+ hit_count=r.hit_count, result_sample=list(r.result_sample or []),
+ duration_ms=r.duration_ms, error=r.error,
+ )
+ for r in rows
+ ]
+
+
[email protected]("/{hunt_id}/findings", response_model=HuntResponse, summary="Append findings")
+async def add_findings(hunt_id: uuid.UUID, body: AddFindingsRequest, db: DBSession, user: AuthUser) -> HuntResponse:
+ existing = (await db.execute(text("SELECT findings FROM aisoc_hunts WHERE id = :id").bindparams(id=hunt_id))).fetchone()
+ if not existing:
+ raise HTTPException(status_code=404, detail="Hunt not found.")
+
+ merged = list(existing.findings or []) + body.findings
+ params: dict[str, Any] = {
+ "id": hunt_id,
+ "findings": json.dumps(merged),
+ "now": datetime.now(UTC),
+ }
+ extra_set = ""
+ if body.false_positive_rate is not None:
+ extra_set = ", false_positive_rate = :fpr"
+ params["fpr"] = body.false_positive_rate
+
+ q = text(f"UPDATE aisoc_hunts SET findings = :findings::jsonb, updated_at = :now{extra_set} WHERE id = :id RETURNING *").bindparams(**params)
+ try:
+ row = (await db.execute(q)).fetchone()
+ await db.commit()
+ return _row_to_hunt(row)
+ except Exception as exc:
+ await db.rollback()
+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc
diff --git a/services/api/app/api/v1/endpoints/identity_timeline.py b/services/api/app/api/v1/endpoints/identity_timeline.py
new file mode 100644
index 000000000..b2b9ed7f3
--- /dev/null
+++ b/services/api/app/api/v1/endpoints/identity_timeline.py
@@ -0,0 +1,225 @@
+"""Identity-centric investigation timeline (tier2-identity).
+
+Builds a chronological event timeline anchored to a user, device, or
+service-account identity. Queries ``aisoc_alerts``, ``aisoc_events``,
+and the enrichment store to produce an ordered list of security events
+grouped by entity, with ATT&CK technique annotations.
+
+Endpoints
+---------
+* ``GET /identity-timeline`` Search timelines by identity.
+* ``POST /identity-timeline/build`` Build a fresh timeline for an identity.
+"""
+
+from __future__ import annotations
+
+import uuid
+from datetime import UTC, datetime, timedelta
+from typing import Any, Literal
+
+from fastapi import APIRouter, HTTPException, Query, status
+from pydantic import BaseModel, Field
+from sqlalchemy import text
+
+from app.api.v1.deps import AuthUser, DBSession
+
+router = APIRouter(prefix="/identity-timeline", tags=["identity_timeline"])
+
+# ────────────────────────────────────────────────────────────────────────────
+# Pydantic schemas
+# ────────────────────────────────────────────────────────────────────────────
+
+IdentityKind = Literal["user", "device", "service_account", "ip"]
+
+
+class TimelineEvent(BaseModel):
+ event_id: uuid.UUID
+ timestamp: datetime
+ event_type: Literal["alert", "login", "process", "network", "file", "other"]
+ source: str
+ description: str
+ severity: str | None = None
+ mitre_technique: str | None = None
+ raw: dict[str, Any] | None = None
+
+
+class IdentityTimeline(BaseModel):
+ timeline_id: uuid.UUID
+ identity_kind: IdentityKind
+ identity_value: str
+ from_ts: datetime
+ to_ts: datetime
+ events: list[TimelineEvent]
+ total_events: int
+ risk_score: float | None = None
+ built_at: datetime
+
+
+class BuildTimelineRequest(BaseModel):
+ identity_kind: IdentityKind = Field(..., description="Type of identity anchor.")
+ identity_value: str = Field(..., min_length=1, description="Identity value (e.g. username, hostname).")
+ from_ts: datetime | None = Field(None, description="Start of window (defaults to 72 h ago).")
+ to_ts: datetime | None = Field(None, description="End of window (defaults to now).")
+ max_events: int = Field(200, ge=1, le=2000)
+
+
+# ────────────────────────────────────────────────────────────────────────────
+# Helpers
+# ────────────────────────────────────────────────────────────────────────────
+
+_FIELD_MAP: dict[IdentityKind, list[str]] = {
+ "user": ["user", "username", "src_user", "dest_user", "actor"],
+ "device": ["hostname", "device", "src_host", "computer_name"],
+ "service_account": ["service_account", "svc_account", "process_user"],
+ "ip": ["src_ip", "source_ip", "ip_address", "remote_ip"],
+}
+
+
+def _risk_score(events: list[TimelineEvent]) -> float:
+ """Simple heuristic: sum severity weights, cap at 100."""
+ weights = {"critical": 40, "high": 20, "medium": 10, "low": 3, "info": 1}
+ score = sum(weights.get((e.severity or "info").lower(), 1) for e in events)
+ return min(100.0, float(score))
+
+
+def _mitre_from_alert(raw: dict[str, Any]) -> str | None:
+ return (
+ raw.get("mitre_technique")
+ or raw.get("mitre_attack_id")
+ or raw.get("tags", {}).get("mitre")
+ )
+
+
+# ────────────────────────────────────────────────────────────────────────────
+# Endpoints
+# ────────────────────────────────────────────────────────────────────────────
+
+
+ "/build",
+ response_model=IdentityTimeline,
+ status_code=status.HTTP_200_OK,
+ summary="Build identity-centric investigation timeline",
+)
+async def build_timeline(
+ body: BuildTimelineRequest,
+ db: DBSession,
+ user: AuthUser,
+) -> IdentityTimeline:
+ now = datetime.now(UTC)
+ from_ts = body.from_ts or (now - timedelta(hours=72))
+ to_ts = body.to_ts or now
+
+ events: list[TimelineEvent] = []
+
+ # ── 1. Alerts that reference this identity ──────────────────────────────
+ try:
+ alert_rows = await db.execute(
+ text(
+ """
+ SELECT id, created_at, severity, title, evidence, mitre_technique
+ FROM aisoc_alerts
+ WHERE created_at BETWEEN :from_ts AND :to_ts
+ AND (
+ evidence::text ILIKE :pat
+ OR title ILIKE :pat
+ )
+ ORDER BY created_at DESC
+ LIMIT :lim
+ """
+ ).bindparams(
+ from_ts=from_ts,
+ to_ts=to_ts,
+ pat=f"%{body.identity_value}%",
+ lim=body.max_events,
+ )
+ )
+ for row in alert_rows.fetchall():
+ events.append(
+ TimelineEvent(
+ event_id=row.id,
+ timestamp=row.created_at,
+ event_type="alert",
+ source="aisoc_alerts",
+ description=row.title or "Alert",
+ severity=row.severity,
+ mitre_technique=row.mitre_technique
+ or _mitre_from_alert(row.evidence or {}),
+ raw=row.evidence,
+ )
+ )
+ except Exception:
+ pass # table may not exist in all deployments
+
+ # ── 2. Raw events from aisoc_events (if table exists) ──────────────────
+ try:
+ ev_rows = await db.execute(
+ text(
+ """
+ SELECT id, timestamp, event_type, source, description,
+ severity, raw_event
+ FROM aisoc_events
+ WHERE timestamp BETWEEN :from_ts AND :to_ts
+ AND raw_event::text ILIKE :pat
+ ORDER BY timestamp DESC
+ LIMIT :lim
+ """
+ ).bindparams(
+ from_ts=from_ts,
+ to_ts=to_ts,
+ pat=f"%{body.identity_value}%",
+ lim=body.max_events - len(events),
+ )
+ )
+ for row in ev_rows.fetchall():
+ events.append(
+ TimelineEvent(
+ event_id=row.id,
+ timestamp=row.timestamp,
+ event_type=row.event_type or "other",
+ source=row.source or "aisoc_events",
+ description=row.description or "",
+ severity=row.severity,
+ raw=row.raw_event,
+ )
+ )
+ except Exception:
+ pass
+
+ # ── 3. Sort chronologically ─────────────────────────────────────────────
+ events.sort(key=lambda e: e.timestamp)
+
+ return IdentityTimeline(
+ timeline_id=uuid.uuid4(),
+ identity_kind=body.identity_kind,
+ identity_value=body.identity_value,
+ from_ts=from_ts,
+ to_ts=to_ts,
+ events=events[: body.max_events],
+ total_events=len(events),
+ risk_score=_risk_score(events),
+ built_at=now,
+ )
+
+
+ "",
+ response_model=IdentityTimeline,
+ summary="Quick timeline lookup for an identity",
+)
+async def get_timeline(
+ identity_kind: IdentityKind = Query(...),
+ identity_value: str = Query(..., min_length=1),
+ hours: int = Query(72, ge=1, le=8760),
+ db: DBSession = ...,
+ user: AuthUser = ...,
+) -> IdentityTimeline:
+ return await build_timeline(
+ BuildTimelineRequest(
+ identity_kind=identity_kind,
+ identity_value=identity_value,
+ from_ts=datetime.now(UTC) - timedelta(hours=hours),
+ ),
+ db=db,
+ user=user,
+ )
diff --git a/services/api/app/api/v1/endpoints/knowledge_base.py b/services/api/app/api/v1/endpoints/knowledge_base.py
new file mode 100644
index 000000000..4a0fc1f50
--- /dev/null
+++ b/services/api/app/api/v1/endpoints/knowledge_base.py
@@ -0,0 +1,246 @@
+"""Knowledge-base + RAG over org docs/runbooks (tier3-rag).
+
+Analysts ingest organisation documents (runbooks, policies, playbooks, SOPs)
+into a full-text-searchable store. A ``/query`` endpoint retrieves relevant
+chunks and, when an LLM key is configured, synthesises a cited answer.
+
+Endpoints
+---------
+* ``POST /kb/ingest`` Ingest a document (chunked automatically).
+* ``GET /kb/documents`` List indexed documents.
+* ``GET /kb/documents/{id}`` Get a document.
+* ``DELETE /kb/documents/{id}`` Remove a document.
+* ``POST /kb/query`` Semantic/keyword search + optional LLM synthesis.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import uuid
+from datetime import UTC, datetime
+from typing import Any, Literal
+
+import httpx
+from fastapi import APIRouter, HTTPException, status
+from pydantic import BaseModel, Field
+from sqlalchemy import text
+
+from app.api.v1.deps import AuthUser, DBSession
+
+router = APIRouter(prefix="/kb", tags=["knowledge_base"])
+
+# ────────────────────────────────────────────────────────────────────────────
+# Schemas
+# ────────────────────────────────────────────────────────────────────────────
+
+DocKind = Literal["runbook", "policy", "playbook", "sop", "wiki", "other"]
+_CHUNK_SIZE = 800 # characters per chunk
+
+
+class IngestRequest(BaseModel):
+ title: str = Field(..., min_length=2)
+ content: str = Field(..., min_length=10)
+ doc_kind: DocKind = "runbook"
+ source_url: str | None = None
+ tags: list[str] = Field(default_factory=list)
+
+
+class KBDocResponse(BaseModel):
+ id: uuid.UUID
+ title: str
+ doc_kind: str
+ source_url: str | None
+ tags: list[str]
+ chunk_index: int
+ chunk_total: int
+ content_preview: str
+ created_at: datetime
+ created_by: str | None
+
+
+class QueryRequest(BaseModel):
+ question: str = Field(..., min_length=3)
+ doc_kinds: list[DocKind] = Field(default_factory=list)
+ top_k: int = Field(5, ge=1, le=20)
+ synthesise: bool = Field(True, description="Use LLM to synthesise an answer from retrieved chunks.")
+
+
+class KBChunk(BaseModel):
+ doc_id: uuid.UUID
+ title: str
+ chunk_index: int
+ content: str
+ score: float | None = None
+
+
+class QueryResponse(BaseModel):
+ question: str
+ chunks: list[KBChunk]
+ answer: str | None # None when synthesise=False or no LLM key
+ sources: list[str]
+
+
+# ────────────────────────────────────────────────────────────────────────────
+# Helpers
+# ────────────────────────────────────────────────────────────────────────────
+
+
+def _chunk(content: str, chunk_size: int = _CHUNK_SIZE) -> list[str]:
+ return [content[i: i + chunk_size] for i in range(0, len(content), chunk_size)] or [content]
+
+
+def _row_to_doc(row: Any) -> KBDocResponse:
+ return KBDocResponse(
+ id=row.id,
+ title=row.title,
+ doc_kind=row.doc_kind,
+ source_url=row.source_url,
+ tags=list(row.tags or []),
+ chunk_index=row.chunk_index,
+ chunk_total=row.chunk_total,
+ content_preview=row.content[:200],
+ created_at=row.created_at,
+ created_by=row.created_by,
+ )
+
+
+_SYNTH_SYSTEM = """You are a helpful security operations assistant.
+Answer the analyst's question using ONLY the provided context chunks.
+Cite document titles inline as [Doc Title].
+If the answer cannot be found in the context, say so clearly."""
+
+
+async def _synthesise(question: str, chunks: list[KBChunk]) -> str | None:
+ api_key = os.getenv("OPENAI_API_KEY") or os.getenv("LLM_API_KEY")
+ if not api_key:
+ return None
+ base_url = os.getenv("LLM_BASE_URL", "https://api.openai.com/v1")
+ model = os.getenv("LLM_MODEL", "gpt-4o-mini")
+ context = "\n\n".join(f"[{c.title}] chunk {c.chunk_index}:\n{c.content}" for c in chunks)
+ try:
+ async with httpx.AsyncClient(timeout=45) as client:
+ resp = await client.post(
+ f"{base_url}/chat/completions",
+ headers={"Authorization": f"Bearer {api_key}"},
+ json={
+ "model": model,
+ "messages": [
+ {"role": "system", "content": _SYNTH_SYSTEM},
+ {"role": "user", "content": f"CONTEXT:\n{context}\n\nQUESTION: {question}"},
+ ],
+ "temperature": 0.2,
+ },
+ )
+ resp.raise_for_status()
+ return resp.json()["choices"][0]["message"]["content"].strip()
+ except Exception:
+ return None
+
+
+# ────────────────────────────────────────────────────────────────────────────
+# Endpoints
+# ────────────────────────────────────────────────────────────────────────────
+
+
[email protected]("/ingest", response_model=list[KBDocResponse], status_code=status.HTTP_201_CREATED, summary="Ingest document into knowledge base")
+async def ingest(body: IngestRequest, db: DBSession, user: AuthUser) -> list[KBDocResponse]:
+ chunks = _chunk(body.content)
+ now = datetime.now(UTC)
+ rows = []
+ for idx, chunk_text in enumerate(chunks):
+ doc_id = uuid.uuid4()
+ q = text("""
+ INSERT INTO aisoc_kb_documents (
+ id, title, doc_kind, source_url, content, tags,
+ chunk_index, chunk_total, created_at, updated_at, created_by
+ ) VALUES (
+ :id, :title, :kind, :url, :content, :tags::text[],
+ :idx, :total, :now, :now, :user
+ ) RETURNING *
+ """).bindparams(
+ id=doc_id,
+ title=body.title,
+ kind=body.doc_kind,
+ url=body.source_url,
+ content=chunk_text,
+ tags=body.tags or [],
+ idx=idx,
+ total=len(chunks),
+ now=now,
+ user=str(user) if user else "system",
+ )
+ try:
+ row = (await db.execute(q)).fetchone()
+ rows.append(row)
+ except Exception as exc:
+ await db.rollback()
+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc
+ await db.commit()
+ return [_row_to_doc(r) for r in rows]
+
+
[email protected]("/documents", response_model=list[KBDocResponse], summary="List KB documents")
+async def list_documents(db: DBSession, user: AuthUser) -> list[KBDocResponse]:
+ try:
+ rows = (await db.execute(text("SELECT * FROM aisoc_kb_documents WHERE chunk_index = 0 ORDER BY created_at DESC LIMIT 200"))).fetchall()
+ return [_row_to_doc(r) for r in rows]
+ except Exception as exc:
+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc
+
+
[email protected]("/documents/{doc_id}", response_model=KBDocResponse, summary="Get KB document")
+async def get_document(doc_id: uuid.UUID, db: DBSession, user: AuthUser) -> KBDocResponse:
+ row = (await db.execute(text("SELECT * FROM aisoc_kb_documents WHERE id = :id").bindparams(id=doc_id))).fetchone()
+ if not row:
+ raise HTTPException(status_code=404, detail="Document not found.")
+ return _row_to_doc(row)
+
+
[email protected]("/documents/{doc_id}", status_code=status.HTTP_204_NO_CONTENT, summary="Remove KB document")
+async def delete_document(doc_id: uuid.UUID, db: DBSession, user: AuthUser) -> None:
+ existing = (await db.execute(text("SELECT title FROM aisoc_kb_documents WHERE id = :id").bindparams(id=doc_id))).fetchone()
+ if not existing:
+ raise HTTPException(status_code=404, detail="Document not found.")
+ # Remove all chunks with same title + doc_kind
+ await db.execute(text("DELETE FROM aisoc_kb_documents WHERE title = :title").bindparams(title=existing.title))
+ await db.commit()
+
+
[email protected]("/query", response_model=QueryResponse, summary="Search knowledge base + optional LLM synthesis")
+async def query_kb(body: QueryRequest, db: DBSession, user: AuthUser) -> QueryResponse:
+ wheres = ["to_tsvector('english', content) @@ plainto_tsquery('english', :q)"]
+ params: dict[str, Any] = {"q": body.question, "limit": body.top_k}
+ if body.doc_kinds:
+ wheres.append("doc_kind = ANY(:kinds::text[])")
+ params["kinds"] = body.doc_kinds
+
+ sql = text(f"""
+ SELECT *, ts_rank(to_tsvector('english', content), plainto_tsquery('english', :q)) AS rank
+ FROM aisoc_kb_documents
+ WHERE {' AND '.join(wheres)}
+ ORDER BY rank DESC
+ LIMIT :limit
+ """).bindparams(**params)
+
+ try:
+ db_rows = (await db.execute(sql)).fetchall()
+ except Exception as exc:
+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc
+
+ chunks = [
+ KBChunk(
+ doc_id=r.id,
+ title=r.title,
+ chunk_index=r.chunk_index,
+ content=r.content,
+ score=float(r.rank) if hasattr(r, "rank") else None,
+ )
+ for r in db_rows
+ ]
+ answer: str | None = None
+ if body.synthesise and chunks:
+ answer = await _synthesise(body.question, chunks)
+
+ sources = list(dict.fromkeys(c.title for c in chunks))
+ return QueryResponse(question=body.question, chunks=chunks, answer=answer, sources=sources)
diff --git a/services/api/app/api/v1/endpoints/metrics.py b/services/api/app/api/v1/endpoints/metrics.py
index 7d39b7396..043f78c86 100644
--- a/services/api/app/api/v1/endpoints/metrics.py
+++ b/services/api/app/api/v1/endpoints/metrics.py
@@ -206,6 +206,143 @@ async def get_dashboard_metrics(
)
+class SOCKpis(BaseModel):
+ mttd_hours: float
+ mttr_hours: float
+ false_positive_rate: float
+ alert_volume_7d: int
+ cases_opened_7d: int
+ cases_closed_7d: int
+ analyst_overrides_7d: int
+
+
+class AttackHeatmapCell(BaseModel):
+ tactic: str
+ technique: str
+ count: int
+
+
+class SOCMetrics(BaseModel):
+ kpis: SOCKpis
+ attack_heatmap: list[AttackHeatmapCell]
+
+
[email protected]("/soc", response_model=SOCMetrics)
+async def get_soc_metrics(
+ user: AuthUser,
+ db: DBSession,
+) -> SOCMetrics:
+ """Return SOC-level KPIs: MTTD, MTTR, FPR, and ATT&CK heatmap data."""
+ tenant_id = user.tenant_id
+ now = datetime.now(UTC)
+ week_start = now - timedelta(days=7)
+
+ # MTTD: mean time from alert creation to first investigation start
+ mttd_q = await db.scalar(
+ select(func.avg(
+ func.extract("epoch", Alert.first_seen_at - Alert.created_at) / 3600
+ )).where(
+ and_(
+ Alert.tenant_id == tenant_id,
+ Alert.first_seen_at.isnot(None),
+ Alert.created_at >= week_start,
+ )
+ )
+ )
+ mttd_hours = float(mttd_q or 0.0)
+
+ # MTTR: mean time from alert creation to resolution
+ mttr_q = await db.scalar(
+ select(func.avg(
+ func.extract("epoch", Alert.updated_at - Alert.created_at) / 3600
+ )).where(
+ and_(
+ Alert.tenant_id == tenant_id,
+ Alert.status == "resolved",
+ Alert.created_at >= week_start,
+ )
+ )
+ )
+ mttr_hours = float(mttr_q or 0.0)
+
+ # FPR: false positives / total resolved alerts (last 7d)
+ total_resolved = await db.scalar(
+ select(func.count()).where(
+ and_(
+ Alert.tenant_id == tenant_id,
+ Alert.status == "resolved",
+ Alert.created_at >= week_start,
+ )
+ )
+ ) or 0
+ fp_count = await db.scalar(
+ select(func.count()).where(
+ and_(
+ Alert.tenant_id == tenant_id,
+ Alert.status == "resolved",
+ Alert.disposition == "false_positive",
+ Alert.created_at >= week_start,
+ )
+ )
+ ) or 0
+ fpr = (fp_count / total_resolved) if total_resolved > 0 else 0.0
+
+ # Alert volume 7d
+ alert_vol = await db.scalar(
+ select(func.count()).where(
+ and_(Alert.tenant_id == tenant_id, Alert.created_at >= week_start)
+ )
+ ) or 0
+
+ # Cases opened/closed 7d
+ cases_opened = await db.scalar(
+ select(func.count()).where(
+ and_(Case.tenant_id == tenant_id, Case.created_at >= week_start)
+ )
+ ) or 0
+ cases_closed = await db.scalar(
+ select(func.count()).where(
+ and_(
+ Case.tenant_id == tenant_id,
+ Case.status == "resolved",
+ Case.updated_at >= week_start,
+ )
+ )
+ ) or 0
+
+ kpis = SOCKpis(
+ mttd_hours=round(mttd_hours, 2),
+ mttr_hours=round(mttr_hours, 2),
+ false_positive_rate=round(fpr, 4),
+ alert_volume_7d=alert_vol,
+ cases_opened_7d=cases_opened,
+ cases_closed_7d=cases_closed,
+ analyst_overrides_7d=0,
+ )
+
+ # ATT&CK heatmap: technique + tactic breakdown
+ heatmap_rows = (
+ await db.execute(
+ select(
+ func.jsonb_array_elements_text(Alert.mitre_tactics).label("tactic"),
+ func.jsonb_array_elements_text(Alert.mitre_techniques).label("technique"),
+ func.count().label("cnt"),
+ )
+ .where(Alert.tenant_id == tenant_id)
+ .group_by("tactic", "technique")
+ .order_by(func.count().desc())
+ .limit(50)
+ )
+ ).all()
+
+ heatmap = [
+ AttackHeatmapCell(tactic=r.tactic, technique=r.technique, count=r.cnt)
+ for r in heatmap_rows
+ ]
+
+ return SOCMetrics(kpis=kpis, attack_heatmap=heatmap)
+
+
@router.get("/alerts/trend")
async def get_alert_trend(
user: AuthUser,
diff --git a/services/api/app/api/v1/endpoints/nl_detection.py b/services/api/app/api/v1/endpoints/nl_detection.py
new file mode 100644
index 000000000..d838b12e6
--- /dev/null
+++ b/services/api/app/api/v1/endpoints/nl_detection.py
@@ -0,0 +1,206 @@
+"""Natural-language detection authoring endpoint.
+
+Accepts a plain-English description of a threat and returns:
+ - A Sigma YAML rule
+ - Equivalent KQL (Microsoft Sentinel)
+ - Equivalent SPL (Splunk)
+ - Equivalent ES|QL (Elastic)
+
+Uses an LLM (OpenAI-compatible) to perform the translation. Falls back to a
+template-based stub when no LLM key is configured so the endpoint stays
+functional in all deployment environments.
+"""
+
+from __future__ import annotations
+
+import os
+from typing import Literal
+
+from fastapi import APIRouter, HTTPException, status
+from pydantic import BaseModel, Field
+
+import structlog
+
+logger = structlog.get_logger()
+
+router = APIRouter(prefix="/nl-detection", tags=["nl_detection"])
+
+
+# ---------------------------------------------------------------------------
+# Request / response models
+# ---------------------------------------------------------------------------
+
+
+class NLDetectionRequest(BaseModel):
+ description: str = Field(
+ ...,
+ description="Plain-English description of the threat behaviour to detect.",
+ min_length=10,
+ max_length=2000,
+ )
+ target_platforms: list[Literal["sigma", "kql", "spl", "esql"]] = Field(
+ default=["sigma", "kql", "spl", "esql"],
+ description="Which rule languages to generate.",
+ )
+ severity: Literal["informational", "low", "medium", "high", "critical"] = "medium"
+ mitre_techniques: list[str] = Field(default_factory=list)
+
+
+class NLDetectionResponse(BaseModel):
+ description: str
+ sigma: str | None = None
+ kql: str | None = None
+ spl: str | None = None
+ esql: str | None = None
+ model_used: str
+
+
+# ---------------------------------------------------------------------------
+# LLM translation helpers
+# ---------------------------------------------------------------------------
+
+_SYSTEM_PROMPT = """You are a detection engineering expert.
+Given a plain-English threat description, generate detection rules in the
+requested formats. Output ONLY valid rule content, no prose.
+
+For Sigma: output valid YAML.
+For KQL: output a valid Kusto query.
+For SPL: output a valid Splunk search.
+For ES|QL: output a valid Elastic ES|QL query.
+"""
+
+_TEMPLATE = {
+ "sigma": """\
+title: {title}
+status: experimental
+description: {description}
+logsource:
+ category: process_creation
+ product: windows
+detection:
+ selection:
+ CommandLine|contains: '<KEYWORD>'
+ condition: selection
+falsepositives:
+ - Legitimate administration activity
+level: {severity}
+""",
+ "kql": """\
+// {title}
+// {description}
+SecurityEvent
+| where EventID == 4688
+| where CommandLine contains "<KEYWORD>"
+| project TimeGenerated, Computer, Account, CommandLine
+""",
+ "spl": """\
+// {title} — {description}
+index=windows EventCode=4688
+| where like(CommandLine, "%<KEYWORD>%")
+| table _time, host, user, CommandLine
+""",
+ "esql": """\
+// {title}
+// {description}
+FROM logs-*
+| WHERE process.command_line LIKE "*<KEYWORD>*"
+| KEEP @timestamp, host.name, user.name, process.command_line
+""",
+}
+
+
+async def _llm_translate(request: NLDetectionRequest) -> dict[str, str | None]:
+ """Attempt LLM-based translation; fall back to templates on any error."""
+ api_key = os.environ.get("OPENAI_API_KEY", "").strip()
+ if not api_key:
+ logger.debug("nl_detection.llm_unavailable", reason="no OPENAI_API_KEY")
+ return _template_fallback(request)
+
+ platforms_str = ", ".join(request.target_platforms)
+ user_prompt = (
+ f"Threat description: {request.description}\n\n"
+ f"Severity: {request.severity}\n"
+ f"MITRE techniques: {', '.join(request.mitre_techniques) or 'unknown'}\n\n"
+ f"Generate detection rules for: {platforms_str}.\n"
+ f"Return JSON with keys matching the platform names (sigma, kql, spl, esql)."
+ )
+
+ try:
+ import httpx
+
+ async with httpx.AsyncClient(timeout=30.0) as client:
+ resp = await client.post(
+ "https://api.openai.com/v1/chat/completions",
+ headers={"Authorization": f"Bearer {api_key}"},
+ json={
+ "model": os.environ.get("OPENAI_MODEL", "gpt-4o-mini"),
+ "messages": [
+ {"role": "system", "content": _SYSTEM_PROMPT},
+ {"role": "user", "content": user_prompt},
+ ],
+ "response_format": {"type": "json_object"},
+ "temperature": 0.1,
+ "max_tokens": 2000,
+ },
+ )
+ resp.raise_for_status()
+ import json
+
+ content = resp.json()["choices"][0]["message"]["content"]
+ rules = json.loads(content)
+ return {
+ "sigma": rules.get("sigma") if "sigma" in request.target_platforms else None,
+ "kql": rules.get("kql") if "kql" in request.target_platforms else None,
+ "spl": rules.get("spl") if "spl" in request.target_platforms else None,
+ "esql": rules.get("esql") if "esql" in request.target_platforms else None,
+ "_model": os.environ.get("OPENAI_MODEL", "gpt-4o-mini"),
+ }
+ except Exception as exc:
+ logger.warning("nl_detection.llm_error", error=str(exc))
+ return _template_fallback(request)
+
+
+def _template_fallback(request: NLDetectionRequest) -> dict[str, str | None]:
+ title = request.description[:60].replace("\n", " ").strip()
+ ctx = {"title": title, "description": request.description, "severity": request.severity}
+ return {
+ "sigma": _TEMPLATE["sigma"].format(**ctx) if "sigma" in request.target_platforms else None,
+ "kql": _TEMPLATE["kql"].format(**ctx) if "kql" in request.target_platforms else None,
+ "spl": _TEMPLATE["spl"].format(**ctx) if "spl" in request.target_platforms else None,
+ "esql": _TEMPLATE["esql"].format(**ctx) if "esql" in request.target_platforms else None,
+ "_model": "template",
+ }
+
+
+# ---------------------------------------------------------------------------
+# Endpoint
+# ---------------------------------------------------------------------------
+
+
[email protected]("/translate", response_model=NLDetectionResponse)
+async def translate_detection(payload: NLDetectionRequest) -> NLDetectionResponse:
+ """Convert a plain-English threat description into multi-platform detection rules."""
+ if not payload.description.strip():
+ raise HTTPException(
+ status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
+ detail="description must not be empty",
+ )
+
+ rules = await _llm_translate(payload)
+ model_used = rules.pop("_model", "unknown")
+
+ logger.info(
+ "nl_detection.translate",
+ platforms=payload.target_platforms,
+ model=model_used,
+ description_len=len(payload.description),
+ )
+
+ return NLDetectionResponse(
+ description=payload.description,
+ sigma=rules.get("sigma"),
+ kql=rules.get("kql"),
+ spl=rules.get("spl"),
+ esql=rules.get("esql"),
+ model_used=model_used,
+ )
diff --git a/services/api/app/api/v1/endpoints/nl_query.py b/services/api/app/api/v1/endpoints/nl_query.py
new file mode 100644
index 000000000..d94e01478
--- /dev/null
+++ b/services/api/app/api/v1/endpoints/nl_query.py
@@ -0,0 +1,283 @@
+"""Natural-language query → multi-dialect execution (tier2-nl-query).
+
+Accepts a plain-English security question, translates it to ES|QL, SPL, and
+KQL, then (optionally) executes the ES|QL variant against a connected
+Elasticsearch cluster and returns structured results ready for charting.
+
+Endpoints
+---------
+* ``POST /nl-query/translate`` Translate NL → ES|QL / SPL / KQL.
+* ``POST /nl-query/execute`` Translate + execute against Elasticsearch.
+"""
+
+from __future__ import annotations
+
+import json
+import textwrap
+import uuid
+from datetime import UTC, datetime
+from typing import Any
+
+import httpx
+from fastapi import APIRouter, HTTPException, status
+from pydantic import BaseModel, Field
+
+from app.api.v1.deps import AuthUser
+from app.core.config import settings
+
+router = APIRouter(prefix="/nl-query", tags=["nl_query"])
+
+# ────────────────────────────────────────────────────────────────────────────
+# Pydantic schemas
+# ────────────────────────────────────────────────────────────────────────────
+
+
+class NLQueryTranslateRequest(BaseModel):
+ question: str = Field(
+ ...,
+ min_length=10,
+ description="Plain-English security question (e.g. 'Show failed logins per user in the last 24 h').",
+ )
+ index_pattern: str = Field(
+ "logs-*,aisoc-events-*",
+ description="Elasticsearch index pattern to scope the ES|QL query.",
+ )
+ time_range_hours: int = Field(
+ 24,
+ ge=1,
+ le=8760,
+ description="Look-back window in hours.",
+ )
+
+
+class NLQueryTranslateResponse(BaseModel):
+ request_id: uuid.UUID
+ question: str
+ esql: str
+ spl: str
+ kql: str
+ explanation: str
+ created_at: datetime
+
+
+class NLQueryExecuteRequest(NLQueryTranslateRequest):
+ es_url: str | None = Field(
+ None,
+ description="Override Elasticsearch URL (defaults to settings.ES_URL if set).",
+ )
+ es_api_key: str | None = Field(
+ None,
+ description="Override ES API key (defaults to settings.ES_API_KEY if set).",
+ )
+ max_rows: int = Field(500, ge=1, le=5000)
+
+
+class QueryResult(BaseModel):
+ columns: list[str]
+ rows: list[list[Any]]
+ total_rows: int
+ took_ms: int | None = None
+
+
+class NLQueryExecuteResponse(NLQueryTranslateResponse):
+ result: QueryResult | None = None
+ execution_error: str | None = None
+
+
+# ────────────────────────────────────────────────────────────────────────────
+# LLM translation helper
+# ────────────────────────────────────────────────────────────────────────────
+
+_SYS_PROMPT = textwrap.dedent(
+ """
+ You are a senior security data engineer. The user will provide:
+ - A plain-English security question.
+ - The Elasticsearch index pattern and time range.
+
+ Translate the question into three query dialects and return JSON only:
+ {
+ "esql": "...", // Elasticsearch ES|QL query
+ "spl": "...", // Splunk SPL query
+ "kql": "...", // Microsoft Sentinel KQL query
+ "explanation": "..." // 1–2 sentences describing what the query does
+ }
+
+ Guidelines:
+ - Use `FROM <index_pattern>` and `| WHERE @timestamp > NOW() - <hours>h` for ES|QL.
+ - Prefer `STATS ... BY ...` for aggregations in ES|QL.
+ - Map common field names: user → user.name, src_ip → source.ip, dest_ip → destination.ip.
+ - Keep queries concise; add a `| LIMIT 500` to ES|QL.
+ """
+).strip()
+
+
+async def _translate(
+ question: str, index_pattern: str, time_range_hours: int
+) -> dict[str, str]:
+ api_key = getattr(settings, "OPENAI_API_KEY", None) or getattr(
+ settings, "LLM_API_KEY", None
+ )
+ if not api_key:
+ return _template_fallback(question, index_pattern, time_range_hours)
+
+ user_msg = json.dumps(
+ {
+ "question": question,
+ "index_pattern": index_pattern,
+ "time_range_hours": time_range_hours,
+ }
+ )
+ try:
+ async with httpx.AsyncClient(timeout=30) as client:
+ resp = await client.post(
+ "https://api.openai.com/v1/chat/completions",
+ headers={"Authorization": f"Bearer {api_key}"},
+ json={
+ "model": "gpt-4o-mini",
+ "response_format": {"type": "json_object"},
+ "messages": [
+ {"role": "system", "content": _SYS_PROMPT},
+ {"role": "user", "content": user_msg},
+ ],
+ },
+ )
+ resp.raise_for_status()
+ return json.loads(resp.json()["choices"][0]["message"]["content"])
+ except Exception:
+ return _template_fallback(question, index_pattern, time_range_hours)
+
+
+def _template_fallback(
+ question: str, index_pattern: str, time_range_hours: int
+) -> dict[str, str]:
+ return {
+ "esql": (
+ f"FROM {index_pattern}\n"
+ f"| WHERE @timestamp > NOW() - {time_range_hours}h\n"
+ f"// TODO: translate → {question}\n"
+ "| LIMIT 500"
+ ),
+ "spl": (
+ f"index=* earliest=-{time_range_hours}h\n"
+ f"// TODO: translate → {question}"
+ ),
+ "kql": (
+ f"// TODO: translate → {question}\n"
+ f"| where TimeGenerated > ago({time_range_hours}h)"
+ ),
+ "explanation": f"Template placeholder for: {question}",
+ }
+
+
+# ────────────────────────────────────────────────────────────────────────────
+# Elasticsearch execution helper
+# ────────────────────────────────────────────────────────────────────────────
+
+
+async def _execute_esql(
+ esql: str, es_url: str, es_api_key: str, max_rows: int
+) -> QueryResult:
+ """Run an ES|QL query against Elasticsearch and return structured results."""
+ # Ensure LIMIT clause
+ query = esql if "| LIMIT" in esql.upper() else f"{esql}\n| LIMIT {max_rows}"
+ import time
+
+ t0 = time.monotonic()
+ async with httpx.AsyncClient(timeout=20) as client:
+ resp = await client.post(
+ f"{es_url.rstrip('/')}/_query",
+ headers={
+ "Authorization": f"ApiKey {es_api_key}",
+ "Content-Type": "application/json",
+ },
+ json={"query": query},
+ )
+ resp.raise_for_status()
+ data = resp.json()
+
+ took_ms = int((time.monotonic() - t0) * 1000)
+ columns = [col["name"] for col in data.get("columns", [])]
+ rows = data.get("values", [])
+ return QueryResult(columns=columns, rows=rows, total_rows=len(rows), took_ms=took_ms)
+
+
+# ────────────────────────────────────────────────────────────────────────────
+# Endpoints
+# ────────────────────────────────────────────────────────────────────────────
+
+
+ "/translate",
+ response_model=NLQueryTranslateResponse,
+ status_code=status.HTTP_200_OK,
+ summary="Translate a natural-language security question to ES|QL / SPL / KQL",
+)
+async def translate_query(
+ body: NLQueryTranslateRequest,
+ user: AuthUser,
+) -> NLQueryTranslateResponse:
+ translated = await _translate(
+ body.question, body.index_pattern, body.time_range_hours
+ )
+ return NLQueryTranslateResponse(
+ request_id=uuid.uuid4(),
+ question=body.question,
+ esql=translated.get("esql", ""),
+ spl=translated.get("spl", ""),
+ kql=translated.get("kql", ""),
+ explanation=translated.get("explanation", ""),
+ created_at=datetime.now(UTC),
+ )
+
+
+ "/execute",
+ response_model=NLQueryExecuteResponse,
+ status_code=status.HTTP_200_OK,
+ summary="Translate NL question and execute ES|QL against Elasticsearch",
+)
+async def execute_query(
+ body: NLQueryExecuteRequest,
+ user: AuthUser,
+) -> NLQueryExecuteResponse:
+ translated = await _translate(
+ body.question, body.index_pattern, body.time_range_hours
+ )
+
+ es_url = (
+ body.es_url
+ or getattr(settings, "ES_URL", None)
+ or getattr(settings, "ELASTICSEARCH_URL", None)
+ )
+ es_api_key = body.es_api_key or getattr(settings, "ES_API_KEY", None)
+
+ base = NLQueryExecuteResponse(
+ request_id=uuid.uuid4(),
+ question=body.question,
+ esql=translated.get("esql", ""),
+ spl=translated.get("spl", ""),
+ kql=translated.get("kql", ""),
+ explanation=translated.get("explanation", ""),
+ created_at=datetime.now(UTC),
+ )
+
+ if not es_url or not es_api_key:
+ base.execution_error = (
+ "ES_URL or ES_API_KEY not configured. "
+ "Set them in environment variables or pass in the request body."
+ )
+ return base
+
+ try:
+ base.result = await _execute_esql(
+ translated.get("esql", ""),
+ es_url=es_url,
+ es_api_key=es_api_key,
+ max_rows=body.max_rows,
+ )
+ except httpx.HTTPStatusError as exc:
+ base.execution_error = f"ES query failed ({exc.response.status_code}): {exc.response.text[:500]}"
+ except Exception as exc:
+ base.execution_error = str(exc)
+
+ return base
diff --git a/services/api/app/api/v1/endpoints/phishing.py b/services/api/app/api/v1/endpoints/phishing.py
new file mode 100644
index 000000000..32dbda789
--- /dev/null
+++ b/services/api/app/api/v1/endpoints/phishing.py
@@ -0,0 +1,265 @@
+"""Email-security + phishing-triage workflow (tier3-phishing).
+
+Analysts or automated ingestion submit raw email text, URLs, attachments, or
+domain names for LLM-powered triage. The endpoint extracts indicators of
+compromise (IOCs), assigns a verdict, and optionally opens a case.
+
+Endpoints
+---------
+* ``POST /phishing/submit`` Submit an artifact for triage.
+* ``GET /phishing/submissions`` List submissions.
+* ``GET /phishing/{id}`` Get a submission.
+* ``POST /phishing/{id}/retriage`` Re-run triage (e.g. after analyst correction).
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import uuid
+from datetime import UTC, datetime
+from typing import Any, Literal
+
+import httpx
+from fastapi import APIRouter, HTTPException, Query, status
+from pydantic import BaseModel, Field
+from sqlalchemy import text
+
+from app.api.v1.deps import AuthUser, DBSession
+
+router = APIRouter(prefix="/phishing", tags=["phishing"])
+
+# ────────────────────────────────────────────────────────────────────────────
+# Schemas
+# ────────────────────────────────────────────────────────────────────────────
+
+ArtifactKind = Literal["email", "url", "attachment", "domain"]
+Verdict = Literal["pending", "benign", "phishing", "spam", "malware", "unknown"]
+
+
+class SubmitRequest(BaseModel):
+ artifact_kind: ArtifactKind = "email"
+ raw_content: str | None = Field(None, description="Raw email source or URL to analyse.")
+ sender: str | None = None
+ subject: str | None = None
+ urls: list[str] = Field(default_factory=list)
+
+
+class TriageResult(BaseModel):
+ verdict: Verdict
+ confidence: float
+ indicators: list[dict[str, Any]]
+ mitre_technique: str | None
+ summary: str
+
+
+class SubmissionResponse(BaseModel):
+ id: uuid.UUID
+ artifact_kind: str
+ sender: str | None
+ subject: str | None
+ urls: list[str]
+ verdict: str
+ confidence: float | None
+ indicators: list[dict[str, Any]]
+ mitre_technique: str | None
+ case_id: uuid.UUID | None
+ submitted_at: datetime
+ triaged_at: datetime | None
+
+
+# ────────────────────────────────────────────────────────────────────────────
+# LLM triage helper
+# ────────────────────────────────────────────────────────────────────────────
+
+_SYSTEM = """You are a phishing and email-security analyst.
+Analyse the submitted artifact and return ONLY valid JSON with:
+{
+ "verdict": "benign|phishing|spam|malware|unknown",
+ "confidence": 0.0-1.0,
+ "indicators": [{"kind": "url|domain|ip|hash|email|header", "value": "...", "note": "..."}],
+ "mitre_technique": "T1566.001 or null",
+ "summary": "one-sentence explanation"
+}"""
+
+
+async def _triage(artifact_kind: str, content: str, urls: list[str]) -> TriageResult | None:
+ api_key = os.getenv("OPENAI_API_KEY") or os.getenv("LLM_API_KEY")
+ if not api_key:
+ return None
+ base_url = os.getenv("LLM_BASE_URL", "https://api.openai.com/v1")
+ model = os.getenv("LLM_MODEL", "gpt-4o-mini")
+ user_msg = f"ARTIFACT TYPE: {artifact_kind}\n"
+ if urls:
+ user_msg += f"URLS: {', '.join(urls[:10])}\n"
+ if content:
+ user_msg += f"CONTENT (first 2000 chars):\n{content[:2000]}"
+ try:
+ async with httpx.AsyncClient(timeout=45) as client:
+ resp = await client.post(
+ f"{base_url}/chat/completions",
+ headers={"Authorization": f"Bearer {api_key}"},
+ json={
+ "model": model,
+ "messages": [
+ {"role": "system", "content": _SYSTEM},
+ {"role": "user", "content": user_msg},
+ ],
+ "temperature": 0.1,
+ "response_format": {"type": "json_object"},
+ },
+ )
+ resp.raise_for_status()
+ data = json.loads(resp.json()["choices"][0]["message"]["content"])
+ return TriageResult(
+ verdict=data.get("verdict", "unknown"),
+ confidence=float(data.get("confidence", 0.5)),
+ indicators=data.get("indicators", []),
+ mitre_technique=data.get("mitre_technique"),
+ summary=data.get("summary", ""),
+ )
+ except Exception:
+ return None
+
+
+def _heuristic_triage(content: str | None, urls: list[str]) -> TriageResult:
+ """Fallback rule-based triage when no LLM key is configured."""
+ indicators: list[dict[str, Any]] = []
+ score = 0.0
+ phishing_words = ["verify your account", "click here", "urgent", "suspended", "password reset", "login immediately"]
+ if content:
+ for kw in phishing_words:
+ if kw.lower() in (content or "").lower():
+ score += 0.15
+ indicators.append({"kind": "keyword", "value": kw, "note": "phishing keyword"})
+ for url in urls:
+ if any(x in url for x in ["bit.ly", "tinyurl", "goo.gl", "ow.ly"]):
+ score += 0.2
+ indicators.append({"kind": "url", "value": url, "note": "URL shortener"})
+ score = min(score, 1.0)
+ verdict: Verdict = "phishing" if score > 0.4 else "benign" if score < 0.1 else "unknown"
+ return TriageResult(verdict=verdict, confidence=round(score, 2), indicators=indicators, mitre_technique=None, summary="Heuristic triage — LLM not configured.")
+
+
+def _row_to_submission(row: Any) -> SubmissionResponse:
+ return SubmissionResponse(
+ id=row.id,
+ artifact_kind=row.artifact_kind,
+ sender=row.sender,
+ subject=row.subject,
+ urls=list(row.urls or []),
+ verdict=row.verdict,
+ confidence=row.confidence,
+ indicators=list(row.indicators or []),
+ mitre_technique=row.mitre_technique,
+ case_id=row.case_id,
+ submitted_at=row.submitted_at,
+ triaged_at=row.triaged_at,
+ )
+
+
+# ────────────────────────────────────────────────────────────────────────────
+# Endpoints
+# ────────────────────────────────────────────────────────────────────────────
+
+
[email protected]("/submit", response_model=SubmissionResponse, status_code=status.HTTP_201_CREATED, summary="Submit artifact for phishing triage")
+async def submit(body: SubmitRequest, db: DBSession, user: AuthUser) -> SubmissionResponse:
+ result = await _triage(body.artifact_kind, body.raw_content or "", body.urls)
+ if not result:
+ result = _heuristic_triage(body.raw_content, body.urls)
+
+ now = datetime.now(UTC)
+ sub_id = uuid.uuid4()
+ q = text("""
+ INSERT INTO aisoc_phishing_submissions (
+ id, submitted_by, artifact_kind, raw_content, sender, subject,
+ urls, verdict, confidence, indicators, mitre_technique,
+ submitted_at, triaged_at, created_at
+ ) VALUES (
+ :id, :by, :kind, :content, :sender, :subject,
+ :urls::text[], :verdict, :conf, :iocs::jsonb, :mitre,
+ :now, :now, :now
+ ) RETURNING *
+ """).bindparams(
+ id=sub_id,
+ by=str(user) if user else "system",
+ kind=body.artifact_kind,
+ content=body.raw_content,
+ sender=body.sender,
+ subject=body.subject,
+ urls=body.urls or [],
+ verdict=result.verdict,
+ conf=result.confidence,
+ iocs=json.dumps(result.indicators),
+ mitre=result.mitre_technique,
+ now=now,
+ )
+ try:
+ row = (await db.execute(q)).fetchone()
+ await db.commit()
+ return _row_to_submission(row)
+ except Exception as exc:
+ await db.rollback()
+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc
+
+
[email protected]("/submissions", response_model=list[SubmissionResponse], summary="List phishing submissions")
+async def list_submissions(
+ db: DBSession,
+ user: AuthUser,
+ verdict: str | None = Query(None),
+ limit: int = Query(50, ge=1, le=500),
+ offset: int = Query(0, ge=0),
+) -> list[SubmissionResponse]:
+ wheres = ["1=1"]
+ params: dict[str, Any] = {"limit": limit, "offset": offset}
+ if verdict:
+ wheres.append("verdict = :verdict"); params["verdict"] = verdict
+ q = text(f"SELECT * FROM aisoc_phishing_submissions WHERE {' AND '.join(wheres)} ORDER BY submitted_at DESC LIMIT :limit OFFSET :offset").bindparams(**params)
+ try:
+ rows = (await db.execute(q)).fetchall()
+ return [_row_to_submission(r) for r in rows]
+ except Exception as exc:
+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc
+
+
[email protected]("/{submission_id}", response_model=SubmissionResponse, summary="Get submission")
+async def get_submission(submission_id: uuid.UUID, db: DBSession, user: AuthUser) -> SubmissionResponse:
+ row = (await db.execute(text("SELECT * FROM aisoc_phishing_submissions WHERE id = :id").bindparams(id=submission_id))).fetchone()
+ if not row:
+ raise HTTPException(status_code=404, detail="Submission not found.")
+ return _row_to_submission(row)
+
+
[email protected]("/{submission_id}/retriage", response_model=SubmissionResponse, summary="Re-run triage on submission")
+async def retriage(submission_id: uuid.UUID, db: DBSession, user: AuthUser) -> SubmissionResponse:
+ existing = (await db.execute(text("SELECT * FROM aisoc_phishing_submissions WHERE id = :id").bindparams(id=submission_id))).fetchone()
+ if not existing:
+ raise HTTPException(status_code=404, detail="Submission not found.")
+
+ result = await _triage(existing.artifact_kind, existing.raw_content or "", list(existing.urls or []))
+ if not result:
+ result = _heuristic_triage(existing.raw_content, list(existing.urls or []))
+
+ now = datetime.now(UTC)
+ q = text("""
+ UPDATE aisoc_phishing_submissions
+ SET verdict = :verdict, confidence = :conf, indicators = :iocs::jsonb,
+ mitre_technique = :mitre, triaged_at = :now
+ WHERE id = :id RETURNING *
+ """).bindparams(
+ id=submission_id,
+ verdict=result.verdict,
+ conf=result.confidence,
+ iocs=json.dumps(result.indicators),
+ mitre=result.mitre_technique,
+ now=now,
+ )
+ try:
+ row = (await db.execute(q)).fetchone()
+ await db.commit()
+ return _row_to_submission(row)
+ except Exception as exc:
+ await db.rollback()
+ raise HTTPException(status_code=503, detail=f"Database error: {exc}") from exc
diff --git a/services/api/app/api/v1/endpoints/translation.py b/services/api/app/api/v1/endpoints/translation.py
new file mode 100644
index 000000000..5632fcd22
--- /dev/null
+++ b/services/api/app/api/v1/endpoints/translation.py
@@ -0,0 +1,258 @@
+"""Cross-platform detection rule translation (tier2-translation).
+
+AST-aware translator between Sigma YAML, Splunk SPL, Microsoft Sentinel KQL,
+Google Chronicle YARA-L2/UDM, and Elastic ES|QL. Uses an LLM for high-fidelity
+translation when ``OPENAI_API_KEY`` is set; falls back to field-map substitution
+for common patterns when no key is configured.
+
+Endpoints
+---------
+* ``POST /translation/translate`` Translate a rule from one format to multiple targets.
+* ``GET /translation/formats`` List supported source/target formats.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+from typing import Any, Literal
+
+import httpx
+from fastapi import APIRouter, HTTPException, status
+from pydantic import BaseModel, Field
+
+router = APIRouter(prefix="/translation", tags=["translation"])
+
+# ────────────────────────────────────────────────────────────────────────────
+# Supported formats
+# ────────────────────────────────────────────────────────────────────────────
+
+DetectionFormat = Literal["sigma", "spl", "kql", "esql", "yara_l2", "udm"]
+
+_FORMAT_LABELS: dict[DetectionFormat, str] = {
+ "sigma": "Sigma YAML",
+ "spl": "Splunk SPL",
+ "kql": "Microsoft Sentinel KQL (OOTB & Analytics Rules)",
+ "esql": "Elastic ES|QL",
+ "yara_l2": "Google Chronicle YARA-L2",
+ "udm": "Google Chronicle UDM Search",
+}
+
+# ────────────────────────────────────────────────────────────────────────────
+# Pydantic schemas
+# ────────────────────────────────────────────────────────────────────────────
+
+
+class TranslateRequest(BaseModel):
+ source_format: DetectionFormat = Field(..., description="Format of the input rule.")
+ source_rule: str = Field(..., min_length=5, description="The raw rule text to translate.")
+ target_formats: list[DetectionFormat] = Field(
+ default=["sigma", "spl", "kql", "esql"],
+ description="Output formats to produce.",
+ )
+
+
+class TranslationResult(BaseModel):
+ format: DetectionFormat
+ label: str
+ rule: str
+ notes: str | None = None
+
+
+class TranslateResponse(BaseModel):
+ source_format: DetectionFormat
+ results: list[TranslationResult]
+ warnings: list[str] = []
+
+
+class FormatsResponse(BaseModel):
+ formats: dict[str, str]
+
+
+# ────────────────────────────────────────────────────────────────────────────
+# LLM helper
+# ────────────────────────────────────────────────────────────────────────────
+
+_SYSTEM = """You are an expert security engineer specialising in detection rule translation.
+You translate detection rules between Sigma YAML, Splunk SPL, Microsoft Sentinel KQL,
+Elastic ES|QL, Google Chronicle YARA-L2, and Google Chronicle UDM Search.
+
+Rules for translation:
+- Preserve ALL detection logic precisely (field names, operators, values, threshold).
+- Map field names using OCSF / ECS / Sigma field conventions to each platform's native fields.
+- Add platform-appropriate pipeline/index/timeframe clauses.
+- Return ONLY valid JSON. No prose before or after.
+
+Return JSON with this exact structure:
+{
+ "results": [
+ {"format": "<format>", "rule": "<rule_text>", "notes": "<optional notes>"},
+ ...
+ ],
+ "warnings": ["<any caveats>"]
+}
+"""
+
+
+def _user_prompt(req: TranslateRequest) -> str:
+ targets = ", ".join(req.target_formats)
+ return (
+ f"SOURCE FORMAT: {req.source_format}\n\n"
+ f"SOURCE RULE:\n```\n{req.source_rule}\n```\n\n"
+ f"Translate the above rule to: {targets}.\n"
+ "Return valid JSON only."
+ )
+
+
+async def _llm_translate(req: TranslateRequest) -> dict[str, Any] | None:
+ api_key = os.getenv("OPENAI_API_KEY") or os.getenv("LLM_API_KEY")
+ if not api_key:
+ return None
+ base_url = os.getenv("LLM_BASE_URL", "https://api.openai.com/v1")
+ model = os.getenv("LLM_MODEL", "gpt-4o-mini")
+ try:
+ async with httpx.AsyncClient(timeout=60) as client:
+ resp = await client.post(
+ f"{base_url}/chat/completions",
+ headers={"Authorization": f"Bearer {api_key}"},
+ json={
+ "model": model,
+ "messages": [
+ {"role": "system", "content": _SYSTEM},
+ {"role": "user", "content": _user_prompt(req)},
+ ],
+ "temperature": 0.1,
+ "response_format": {"type": "json_object"},
+ },
+ )
+ resp.raise_for_status()
+ content = resp.json()["choices"][0]["message"]["content"]
+ return json.loads(content)
+ except Exception:
+ return None
+
+
+# ────────────────────────────────────────────────────────────────────────────
+# Template-based fallback (field-map substitution)
+# ────────────────────────────────────────────────────────────────────────────
+
+# Sigma → target field map (common ECS / OCSF fields)
+_FIELD_MAP: dict[str, dict[str, str]] = {
+ "spl": {
+ "EventID": "EventCode",
+ "Image": "process_name",
+ "CommandLine": "process_path",
+ "TargetUserName": "user",
+ "DestinationIp": "dest_ip",
+ "SourceIp": "src_ip",
+ },
+ "kql": {
+ "EventID": "EventID",
+ "Image": "NewProcessName",
+ "CommandLine": "CommandLine",
+ "TargetUserName": "TargetUserName",
+ "DestinationIp": "RemoteIP",
+ "SourceIp": "InitiatingProcessRemoteIP",
+ },
+ "esql": {
+ "EventID": "event.code",
+ "Image": "process.executable",
+ "CommandLine": "process.command_line",
+ "TargetUserName": "user.name",
+ "DestinationIp": "destination.ip",
+ "SourceIp": "source.ip",
+ },
+ "yara_l2": {
+ "EventID": "metadata.product_event_type",
+ "Image": "principal.process.file.full_path",
+ "CommandLine": "principal.process.command_line",
+ "TargetUserName": "target.user.userid",
+ "DestinationIp": "target.ip",
+ "SourceIp": "principal.ip",
+ },
+ "udm": {
+ "EventID": "metadata.product_event_type",
+ "Image": "principal.process.file.full_path",
+ "CommandLine": "principal.process.command_line",
+ "TargetUserName": "target.user.userid",
+ "DestinationIp": "target.ip",
+ "SourceIp": "principal.ip",
+ },
+}
+
+
+def _fallback_templates(req: TranslateRequest) -> dict[str, Any]:
+ """Produce best-effort template translations without an LLM."""
+ results = []
+ warnings = ["LLM not configured — using field-map template substitution. Review output carefully."]
+
+ for fmt in req.target_formats:
+ if fmt == req.source_format:
+ results.append({"format": fmt, "rule": req.source_rule, "notes": "Same as source."})
+ continue
+
+ rule = req.source_rule
+ field_map = _FIELD_MAP.get(fmt, {})
+ for src_field, tgt_field in field_map.items():
+ rule = rule.replace(src_field, tgt_field)
+
+ if fmt == "spl":
+ rule = f"index=* sourcetype=WinEventLog\n| search {rule}"
+ elif fmt == "kql":
+ rule = f"SecurityEvent\n| where {rule}"
+ elif fmt == "esql":
+ rule = f"FROM logs-*\n| WHERE {rule}"
+ elif fmt == "yara_l2":
+ rule = f"rule translated_rule {{\n meta:\n author = \"AiSOC\"\n condition:\n {rule}\n}}"
+ elif fmt == "udm":
+ rule = f"// Chronicle UDM Search\n{rule}"
+
+ results.append({"format": fmt, "rule": rule, "notes": "Template translation — verify field names."})
+
+ return {"results": results, "warnings": warnings}
+
+
+# ────────────────────────────────────────────────────────────────────────────
+# Endpoints
+# ────────────────────────────────────────────────────────────────────────────
+
+
+ "/translate",
+ response_model=TranslateResponse,
+ status_code=status.HTTP_200_OK,
+ summary="Translate a detection rule across formats",
+)
+async def translate_rule(body: TranslateRequest) -> TranslateResponse:
+ if not body.target_formats:
+ raise HTTPException(status_code=400, detail="At least one target_format is required.")
+
+ payload = await _llm_translate(body)
+ if not payload:
+ payload = _fallback_templates(body)
+
+ results = [
+ TranslationResult(
+ format=r["format"],
+ label=_FORMAT_LABELS.get(r["format"], r["format"]),
+ rule=r["rule"],
+ notes=r.get("notes"),
+ )
+ for r in payload.get("results", [])
+ if r.get("format") and r.get("rule")
+ ]
+
+ return TranslateResponse(
+ source_format=body.source_format,
+ results=results,
+ warnings=payload.get("warnings", []),
+ )
+
+
+ "/formats",
+ response_model=FormatsResponse,
+ summary="List supported detection rule formats",
+)
+async def list_formats() -> FormatsResponse:
+ return FormatsResponse(formats=_FORMAT_LABELS) # type: ignore[arg-type]
diff --git a/services/api/app/api/v1/router.py b/services/api/app/api/v1/router.py
index 03a360043..3a764df4f 100644
--- a/services/api/app/api/v1/router.py
+++ b/services/api/app/api/v1/router.py
@@ -13,9 +13,16 @@ from app.api.v1.endpoints import (
community,
compliance,
connectors,
+ detection_loop,
detection_proposals,
detection_rules,
+ feedback,
federated,
+ hunts,
+ identity_timeline,
+ knowledge_base,
+ nl_detection,
+ nl_query,
graph,
identity_graph,
insider_threat,
@@ -25,6 +32,7 @@ from app.api.v1.endpoints import (
mssp,
oncall,
passkeys,
+ phishing,
playbooks,
plugins,
posture,
@@ -35,6 +43,7 @@ from app.api.v1.endpoints import (
sla,
tenants,
threat_intel,
+ translation,
)
api_router = APIRouter(prefix="/api/v1")
@@ -72,6 +81,33 @@ api_router.include_router(mssp.router)
api_router.include_router(insider_threat.router)
api_router.include_router(remediation.router)
+# Analyst feedback loop
+api_router.include_router(feedback.router)
+
+# NL detection authoring (Tier 2)
+api_router.include_router(nl_detection.router)
+
+# Closed-loop detection engineering: FP → LLM Sigma draft → DAC proposal (Tier 2)
+api_router.include_router(detection_loop.router)
+
+# Natural-language query → ES|QL / SPL / KQL translation + execution (Tier 2)
+api_router.include_router(nl_query.router)
+
+# Identity-centric investigation timeline (Tier 2)
+api_router.include_router(identity_timeline.router)
+
+# Cross-platform detection rule translation: Sigma↔SPL↔KQL↔UDM↔ES|QL (Tier 2)
+api_router.include_router(translation.router)
+
+# Hypothesis-driven hunt workbench (Tier 2)
+api_router.include_router(hunts.router)
+
+# Email-security + phishing-triage workflow (Tier 3)
+api_router.include_router(phishing.router)
+
+# Knowledge-base + RAG over org docs/runbooks (Tier 3)
+api_router.include_router(knowledge_base.router)
+
# Wave 4 — advanced capabilities
api_router.include_router(threat_intel.router)
api_router.include_router(posture.router)
diff --git a/services/api/app/models/alert.py b/services/api/app/models/alert.py
index ca29d5ab0..d3bd582b6 100644
--- a/services/api/app/models/alert.py
+++ b/services/api/app/models/alert.py
@@ -34,6 +34,11 @@ class Alert(Base):
source_event_ids: Mapped[list] = mapped_column(JSONB, default=list)
ocsf_class_uid: Mapped[int | None] = mapped_column(Integer, nullable=True)
+ # Analyst verdict / disposition (set via feedback endpoint)
+ disposition: Mapped[str | None] = mapped_column(String(50), nullable=True, index=True)
+ # Timestamp when an analyst first viewed / started triaging the alert (for MTTD)
+ first_seen_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+
# AI/ML
ai_score: Mapped[float | None] = mapped_column(Float, nullable=True)
ai_summary: Mapped[str | None] = mapped_column(Text, nullable=True)
diff --git a/services/api/migrations/012_case_management.sql b/services/api/migrations/012_case_management.sql
new file mode 100644
index 000000000..55c40bb93
--- /dev/null
+++ b/services/api/migrations/012_case_management.sql
@@ -0,0 +1,48 @@
+-- Migration 012: First-class case management
+-- Adds cases table with lifecycle state machine, observable graph, and evidence chain.
+
+CREATE TABLE IF NOT EXISTS aisoc_cases (
+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+ tenant_id UUID,
+ title TEXT NOT NULL,
+ description TEXT,
+ severity TEXT NOT NULL DEFAULT 'medium'
+ CHECK (severity IN ('info','low','medium','high','critical')),
+ status TEXT NOT NULL DEFAULT 'new'
+ CHECK (status IN ('new','triaged','investigating','contained','resolved','closed')),
+ assignee TEXT,
+ -- ATT&CK coverage
+ mitre_techniques JSONB DEFAULT '[]'::jsonb,
+ -- evidence & observables
+ alert_ids UUID[] DEFAULT ARRAY[]::UUID[],
+ observable_graph JSONB DEFAULT '{}'::jsonb, -- nodes + edges
+ evidence_chain JSONB DEFAULT '[]'::jsonb, -- ordered list of evidence items
+ -- compliance
+ compliance_frameworks TEXT[] DEFAULT ARRAY[]::TEXT[],
+ -- timeline
+ opened_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+ triaged_at TIMESTAMPTZ,
+ resolved_at TIMESTAMPTZ,
+ closed_at TIMESTAMPTZ,
+ -- metadata
+ created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+ updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+ created_by TEXT,
+ tags JSONB DEFAULT '{}'::jsonb,
+ sla_due_at TIMESTAMPTZ
+);
+
+-- Case comments / timeline entries
+CREATE TABLE IF NOT EXISTS aisoc_case_comments (
+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+ case_id UUID NOT NULL REFERENCES aisoc_cases(id) ON DELETE CASCADE,
+ author TEXT,
+ body TEXT NOT NULL,
+ is_system BOOLEAN NOT NULL DEFAULT FALSE,
+ created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX IF NOT EXISTS idx_aisoc_cases_tenant ON aisoc_cases (tenant_id);
+CREATE INDEX IF NOT EXISTS idx_aisoc_cases_status ON aisoc_cases (status);
+CREATE INDEX IF NOT EXISTS idx_aisoc_cases_severity ON aisoc_cases (severity);
+CREATE INDEX IF NOT EXISTS idx_aisoc_case_comments ON aisoc_case_comments (case_id);
diff --git a/services/api/migrations/013_compliance_evidence.sql b/services/api/migrations/013_compliance_evidence.sql
new file mode 100644
index 000000000..8cc91e50c
--- /dev/null
+++ b/services/api/migrations/013_compliance_evidence.sql
@@ -0,0 +1,29 @@
+-- Migration 013: Compliance evidence trails
+-- Audit-grade evidence records linked to cases with hash-chain integrity.
+
+CREATE TABLE IF NOT EXISTS aisoc_compliance_evidence (
+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+ case_id UUID REFERENCES aisoc_cases(id) ON DELETE SET NULL,
+ tenant_id UUID,
+ framework TEXT NOT NULL, -- SOC2, PCI-DSS, HIPAA, ISO27001, NIST-CSF …
+ control_id TEXT NOT NULL, -- e.g. CC6.1, A.12.4.1
+ control_title TEXT,
+ evidence_kind TEXT NOT NULL DEFAULT 'alert'
+ CHECK (evidence_kind IN ('alert','log','screenshot','attestation','policy','runbook','other')),
+ summary TEXT NOT NULL,
+ raw_payload JSONB DEFAULT '{}'::jsonb,
+ -- SHA-256 of (prev_hash || summary || raw_payload) for chain integrity
+ payload_hash TEXT,
+ prev_hash TEXT,
+ -- lifecycle
+ collected_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+ reviewed_by TEXT,
+ reviewed_at TIMESTAMPTZ,
+ status TEXT NOT NULL DEFAULT 'pending'
+ CHECK (status IN ('pending','accepted','rejected')),
+ created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX IF NOT EXISTS idx_aisoc_compliance_framework ON aisoc_compliance_evidence (framework);
+CREATE INDEX IF NOT EXISTS idx_aisoc_compliance_case ON aisoc_compliance_evidence (case_id);
+CREATE INDEX IF NOT EXISTS idx_aisoc_compliance_control ON aisoc_compliance_evidence (control_id);
diff --git a/services/api/migrations/014_hunt_workbench.sql b/services/api/migrations/014_hunt_workbench.sql
new file mode 100644
index 000000000..c20b88084
--- /dev/null
+++ b/services/api/migrations/014_hunt_workbench.sql
@@ -0,0 +1,44 @@
+-- Migration 014: Hypothesis-driven hunt workbench
+-- Stores hunt hypotheses, linked detections, and execution results.
+
+CREATE TABLE IF NOT EXISTS aisoc_hunts (
+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+ tenant_id UUID,
+ title TEXT NOT NULL,
+ hypothesis TEXT NOT NULL,
+ mitre_tactic TEXT,
+ mitre_technique TEXT,
+ status TEXT NOT NULL DEFAULT 'draft'
+ CHECK (status IN ('draft','active','completed','archived')),
+ priority TEXT NOT NULL DEFAULT 'medium'
+ CHECK (priority IN ('low','medium','high','critical')),
+ -- linked content
+ query_esql TEXT,
+ query_spl TEXT,
+ query_kql TEXT,
+ -- results
+ findings JSONB DEFAULT '[]'::jsonb,
+ false_positive_rate FLOAT,
+ -- metadata
+ assigned_to TEXT,
+ tags TEXT[] DEFAULT ARRAY[]::TEXT[],
+ created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+ updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+ completed_at TIMESTAMPTZ,
+ created_by TEXT
+);
+
+CREATE TABLE IF NOT EXISTS aisoc_hunt_runs (
+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+ hunt_id UUID NOT NULL REFERENCES aisoc_hunts(id) ON DELETE CASCADE,
+ run_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+ platform TEXT NOT NULL DEFAULT 'esql',
+ query_used TEXT,
+ hit_count INT DEFAULT 0,
+ result_sample JSONB DEFAULT '[]'::jsonb,
+ duration_ms INT,
+ error TEXT
+);
+
+CREATE INDEX IF NOT EXISTS idx_aisoc_hunts_status ON aisoc_hunts (status);
+CREATE INDEX IF NOT EXISTS idx_aisoc_hunt_runs_hunt ON aisoc_hunt_runs (hunt_id);
diff --git a/services/api/migrations/015_phishing_triage.sql b/services/api/migrations/015_phishing_triage.sql
new file mode 100644
index 000000000..fb6652dcf
--- /dev/null
+++ b/services/api/migrations/015_phishing_triage.sql
@@ -0,0 +1,31 @@
+-- Migration 015: Email-security + phishing-triage workflow
+-- Stores submitted email/URL artifacts and triage verdicts.
+
+CREATE TABLE IF NOT EXISTS aisoc_phishing_submissions (
+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+ tenant_id UUID,
+ submitted_by TEXT,
+ -- artifact
+ artifact_kind TEXT NOT NULL DEFAULT 'email'
+ CHECK (artifact_kind IN ('email','url','attachment','domain')),
+ raw_content TEXT,
+ sender TEXT,
+ subject TEXT,
+ urls TEXT[] DEFAULT ARRAY[]::TEXT[],
+ attachments JSONB DEFAULT '[]'::jsonb,
+ -- triage
+ verdict TEXT DEFAULT 'pending'
+ CHECK (verdict IN ('pending','benign','phishing','spam','malware','unknown')),
+ confidence FLOAT,
+ indicators JSONB DEFAULT '[]'::jsonb, -- IOCs extracted
+ mitre_technique TEXT,
+ -- case linkage
+ case_id UUID REFERENCES aisoc_cases(id) ON DELETE SET NULL,
+ -- metadata
+ submitted_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+ triaged_at TIMESTAMPTZ,
+ created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX IF NOT EXISTS idx_aisoc_phishing_verdict ON aisoc_phishing_submissions (verdict);
+CREATE INDEX IF NOT EXISTS idx_aisoc_phishing_tenant ON aisoc_phishing_submissions (tenant_id);
diff --git a/services/api/migrations/016_knowledge_base.sql b/services/api/migrations/016_knowledge_base.sql
new file mode 100644
index 000000000..7d1572eee
--- /dev/null
+++ b/services/api/migrations/016_knowledge_base.sql
@@ -0,0 +1,25 @@
+-- Migration 016: Knowledge-base + RAG over org docs/runbooks
+-- Stores indexed document chunks for retrieval-augmented generation.
+
+CREATE TABLE IF NOT EXISTS aisoc_kb_documents (
+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+ tenant_id UUID,
+ title TEXT NOT NULL,
+ source_url TEXT,
+ doc_kind TEXT NOT NULL DEFAULT 'runbook'
+ CHECK (doc_kind IN ('runbook','policy','playbook','sop','wiki','other')),
+ content TEXT NOT NULL,
+ -- chunk metadata (if chunked at ingest)
+ chunk_index INT DEFAULT 0,
+ chunk_total INT DEFAULT 1,
+ -- simple keyword tags for pre-filter
+ tags TEXT[] DEFAULT ARRAY[]::TEXT[],
+ created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+ updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+ created_by TEXT
+);
+
+CREATE INDEX IF NOT EXISTS idx_aisoc_kb_tenant ON aisoc_kb_documents (tenant_id);
+CREATE INDEX IF NOT EXISTS idx_aisoc_kb_kind ON aisoc_kb_documents (doc_kind);
+-- Full-text search index
+CREATE INDEX IF NOT EXISTS idx_aisoc_kb_fts ON aisoc_kb_documents USING gin(to_tsvector('english', content));
diff --git a/services/api/migrations/020_soc_metrics_h2.sql b/services/api/migrations/020_soc_metrics_h2.sql
new file mode 100644
index 000000000..6f02220b4
--- /dev/null
+++ b/services/api/migrations/020_soc_metrics_h2.sql
@@ -0,0 +1,69 @@
+-- Migration 020: SOC Metrics H2 Roadmap features
+-- Adds: alert disposition, first_seen_at (MTTD), run cost telemetry table,
+-- autonomy thresholds table, institutional memory table.
+
+BEGIN;
+
+-- Alert disposition and MTTD tracking
+ALTER TABLE alerts
+ ADD COLUMN IF NOT EXISTS disposition TEXT,
+ ADD COLUMN IF NOT EXISTS first_seen_at TIMESTAMPTZ;
+
+CREATE INDEX IF NOT EXISTS ix_alerts_tenant_disposition
+ ON alerts (tenant_id, disposition)
+ WHERE disposition IS NOT NULL;
+
+-- Investigation cost telemetry
+CREATE TABLE IF NOT EXISTS aisoc_run_costs (
+ run_id TEXT NOT NULL,
+ tenant_id TEXT NOT NULL,
+ model TEXT NOT NULL,
+ total_prompt_tokens INTEGER NOT NULL DEFAULT 0,
+ total_completion_tokens INTEGER NOT NULL DEFAULT 0,
+ total_cost_usd DOUBLE PRECISION NOT NULL DEFAULT 0,
+ total_latency_ms DOUBLE PRECISION NOT NULL DEFAULT 0,
+ call_count INTEGER NOT NULL DEFAULT 0,
+ recorded_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+ PRIMARY KEY (run_id, tenant_id, model)
+);
+
+CREATE INDEX IF NOT EXISTS aisoc_run_costs_tenant_run
+ ON aisoc_run_costs (tenant_id, run_id);
+
+CREATE INDEX IF NOT EXISTS aisoc_run_costs_recorded_at
+ ON aisoc_run_costs (recorded_at DESC);
+
+-- Autonomy guardrails thresholds (per-tenant overrides)
+CREATE TABLE IF NOT EXISTS aisoc_autonomy_thresholds (
+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+ tenant_id TEXT NOT NULL,
+ action_name TEXT NOT NULL,
+ min_confidence DOUBLE PRECISION NOT NULL CHECK (min_confidence BETWEEN 0 AND 1),
+ updated_by TEXT,
+ updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+ UNIQUE (tenant_id, action_name)
+);
+
+CREATE INDEX IF NOT EXISTS aisoc_autonomy_thresholds_tenant
+ ON aisoc_autonomy_thresholds (tenant_id);
+
+-- Institutional memory (three-tier memory system)
+CREATE TABLE IF NOT EXISTS aisoc_institutional_memory (
+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+ tenant_id TEXT NOT NULL,
+ key TEXT NOT NULL,
+ value JSONB NOT NULL,
+ tags TEXT[] NOT NULL DEFAULT '{}',
+ analyst_override BOOLEAN NOT NULL DEFAULT FALSE,
+ override_reason TEXT,
+ created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+ UNIQUE (tenant_id, key)
+);
+
+CREATE INDEX IF NOT EXISTS aisoc_institutional_memory_tenant_key
+ ON aisoc_institutional_memory (tenant_id, key);
+
+CREATE INDEX IF NOT EXISTS aisoc_institutional_memory_tags
+ ON aisoc_institutional_memory USING GIN (tags);
+
+COMMIT;