Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

feat: v1.6 — attack-path agent, connector health, tiered retention pipeline

Beenu Arora6131b1468d9b
diff --git a/.gitignore b/.gitignore
index e64c6d55e..bc38517ed 100644
--- a/.gitignore
+++ b/.gitignore
@@ -85,3 +85,6 @@ PR_TRIAGE.md
# Docusaurus generated cache (apps/docs)
apps/docs/.docusaurus/
apps/docs/build/
+
+# Local QA artifacts (screenshots from manual UI checks)
+apps/web/.qa-screenshots/
diff --git a/apps/docs/docs/features/attack-path-agent.md b/apps/docs/docs/features/attack-path-agent.md
new file mode 100644
index 000000000..768047687
--- /dev/null
+++ b/apps/docs/docs/features/attack-path-agent.md
@@ -0,0 +1,82 @@
+---
+sidebar_position: 3
+title: Attack-path investigation agent
+description: How AiSOC's investigation graph adds an attack-path step that walks the knowledge graph, scores blast radius, and proposes containment actions on the highest-value pivots.
+---
+
+# Attack-path investigation agent
+
+When the investigation agent finishes its analysis of a case, it has a story but not a map. The attack-path agent takes the case, walks the knowledge graph, finds the chain of entities that links the alert to its blast radius, and turns that into concrete containment proposals.
+
+## Where it sits in the workflow
+
+The orchestrator graph (`services/agents/app/graph/workflow.py`) runs:
+
+```
+ingest → triage → investigation → attack_path → END
+```
+
+The attack-path node runs *after* the investigation agent, so it inherits the case's findings and can layer graph-derived insights on top. It is purely additive: if the graph service has nothing to say (no path, no neighbors), the node records that and exits without modifying the proposed-action queue.
+
+## What it does, step by step
+
+1. **Calls the graph API.** `GET /api/v1/graph/attack-path/{case_id}` returns the path of entities and relationships that the case touches, plus a MITRE tactic chain extracted from the relationship types.
+2. **Summarizes the graph.** Records node counts by type (user, host, asset, ip, etc.), edge counts, and the tactic chain into `state.threat_intel["attack_path"]`. This becomes part of the case's durable record.
+3. **Scores blast radius for pivot entities.** For every host or service-account on the path, the agent calls `GET /api/v1/graph/blast-radius/{type}/{id}` to compute how many downstream entities depend on it. High blast radius means a single containment action has high impact.
+4. **Proposes isolation actions.** For the top pivot entities (those above a configurable blast-radius threshold), the agent emits a `ProposedAction` with type `isolate_host` (or the equivalent for the entity type), an `ActionRisk` band, and the parameters needed to execute — `entity_type`, `entity_id`, `blast_radius_score`.
+
+These proposed actions land in the same queue as everything else the investigation produced, so the existing approval flow (auto-approve for low-risk, human-in-the-loop for high-risk) governs whether they execute.
+
+## Why the graph and not just rules
+
+Three reasons the agent reads from the graph instead of running rules over the raw alert:
+
+1. **Lateral movement is a graph problem.** A SIEM rule can tell you a user authenticated from a new IP. Only a graph can tell you that user has access to seven critical hosts, three of which talk to the database tier.
+2. **Blast radius is a graph property.** You can't compute it by counting log lines. You compute it by walking edges.
+3. **The same graph powers the SOC analyst's UI.** The agent and the human are looking at the same data structure, which means an analyst can audit *why* the agent proposed a specific isolation in seconds, not minutes.
+
+## Output contract
+
+Inside `InvestigationState`, the agent adds:
+
+- One or more `Finding` entries summarizing the path and the high-blast pivots.
+- Zero or more `ProposedAction` entries with `action_type="isolate_host"` (or the appropriate verb for the entity type) and a `risk` band.
+- A compact summary at `state.threat_intel["attack_path"]`:
+
+```json
+{
+ "node_count": 12,
+ "edge_count": 18,
+ "tactic_chain": ["initial_access", "credential_access", "lateral_movement"],
+ "high_blast_pivots": [
+ {
+ "label": "Host",
+ "name": "build-server-01",
+ "entity_type": "host",
+ "entity_id": "host-1f9...",
+ "blast_radius_score": 47
+ }
+ ]
+}
+```
+
+Downstream consumers (the case timeline, the response queue, the analytics layer) read from this summary rather than re-walking the graph.
+
+## When it stays silent
+
+The agent records nothing in the proposed-action queue if:
+
+- The graph service returned 404 (no path discoverable for this case yet — common for very new cases or low-fidelity alerts).
+- All pivot blast-radius scores are below the threshold.
+- The case has no entities to walk from.
+
+Silence is correct behavior. The agent is not required to fire on every case.
+
+## Where it lives in the code
+
+- Agent entry point: `services/agents/app/agents/attack_path_agent.py`
+- Graph client tool: `services/agents/app/tools/graph.py`
+- LangGraph wiring: `services/agents/app/graph/workflow.py`
+- Tests: `services/agents/tests/test_attack_path_agent.py`
+- Underlying graph endpoints: `services/api/app/api/v1/endpoints/graph.py`
+- Graph service: `services/api/app/services/graph_service.py`
diff --git a/apps/docs/docs/features/connector-health.md b/apps/docs/docs/features/connector-health.md
new file mode 100644
index 000000000..b2a2b05d9
--- /dev/null
+++ b/apps/docs/docs/features/connector-health.md
@@ -0,0 +1,86 @@
+---
+sidebar_position: 1
+title: Connector health & schema-drift sentinel
+description: How AiSOC fingerprints every connector's event schema, detects drift the moment a vendor changes its API, and gives you a single pane of health across every integration.
+---
+
+# Connector health and schema-drift sentinel
+
+Every SOC integration breaks the same way: a vendor ships an API change in their next release, your normalizer silently starts dropping fields, and three weeks later somebody notices a detection went quiet. The schema-drift sentinel exists so that hour-zero is the same as hour-three-weeks: AiSOC notices the drift on the next poll and surfaces it before any downstream rule misfires.
+
+This page documents the model, the API surface, and how to operate it.
+
+## What it does
+
+Every time the connector scheduler polls a vendor and normalizes a batch of events, AiSOC computes a stable **schema fingerprint** — a SHA-256 hash of the sorted set of top-level field names in the normalized payload. That fingerprint is stored on the connector instance.
+
+On the next poll:
+
+- If the fingerprint matches, we record a successful run.
+- If it changes, we **diff the previous and current field sets**, persist the drift event with `added`/`removed` fields and the previous/current fingerprint, and surface it via the connector health API.
+
+Because the hash is over the *normalized* schema (not the raw vendor payload), it is stable across order-of-fields changes, cosmetic vendor renames that the connector silently rewrites, and value-only changes — but it changes the moment a real field appears or disappears.
+
+## Why fingerprints, not full diffs
+
+The naive design is "store the whole schema and diff it every time." We don't, for three reasons:
+
+1. **Cheap to compute, cheap to store.** A 32-byte hash per connector beats keeping every prior schema around.
+2. **No PII or vendor data leaks into the audit trail.** The fingerprint reveals nothing about the events themselves — it's a one-way hash of the field-name set.
+3. **Equality check is O(1).** The hot path on every poll is a single string compare.
+
+When drift is detected, we then *also* persist the human-readable diff (`added`/`removed`/`previous_fingerprint`) so an operator can read what changed without recomputing it.
+
+## Health surface
+
+`GET /api/v1/connectors/health` returns a tenant-scoped health summary:
+
+```json
+{
+ "total": 12,
+ "healthy": 10,
+ "degraded": 1,
+ "error": 0,
+ "drift_in_last_24h": 1,
+ "events_dropped_total": 247,
+ "instances": [
+ {
+ "id": "...",
+ "connector_id": "github",
+ "name": "github-prod",
+ "health_status": "healthy",
+ "schema_fingerprint": "a3f9...",
+ "last_schema_drift_at": null,
+ "last_drift_details": null,
+ "events_dropped": 0,
+ "last_run_at": "2026-05-07T22:31:04Z"
+ }
+ ]
+}
+```
+
+The per-instance fields:
+
+| Field | Meaning |
+|---|---|
+| `schema_fingerprint` | Current normalized-schema hash. |
+| `last_schema_drift_at` | Timestamp of the most recent drift event (null if never). |
+| `last_drift_details` | `{ "added": [...], "removed": [...], "previous_fingerprint": "..." }`. |
+| `events_dropped` | Cumulative count of events the pre-ingest filter dropped (see [Security data pipeline](./data-pipeline.md)). |
+| `health_status` | `healthy`, `degraded`, or `error` — derived from recent run history. |
+
+## Operational notes
+
+- **Drift is not a failure.** A drift event is informational: it says "the shape of your incoming events changed". The connector keeps polling. You decide whether to investigate, update the normalizer, or accept the change.
+- **First run sets the baseline.** A connector with no prior fingerprint adopts the first observed one without raising drift; only subsequent polls trigger drift events.
+- **Empty batches don't reset the fingerprint.** A poll that returns zero events leaves the prior fingerprint intact, so transient vendor outages don't show up as drift.
+
+## Where it lives in the code
+
+- Fingerprinting helpers: `services/connectors/app/pipeline/fingerprint.py`
+- Filter rules (used to compute `events_dropped`): `services/connectors/app/pipeline/filter_rules.py`
+- Drift integration in the polling loop: `services/connectors/app/scheduler.py`
+- API surface: `services/api/app/api/v1/endpoints/connectors.py` (`/health`)
+- Schema migration: `services/api/migrations/026_connector_schema_drift.sql`
+
+If you're adding a new connector, you don't need to do anything — the scheduler computes the fingerprint from the normalized batch your `normalize()` returns. Drift detection is implicit.
diff --git a/apps/docs/docs/features/data-pipeline.md b/apps/docs/docs/features/data-pipeline.md
new file mode 100644
index 000000000..5339d1abf
--- /dev/null
+++ b/apps/docs/docs/features/data-pipeline.md
@@ -0,0 +1,93 @@
+---
+sidebar_position: 2
+title: Security data pipeline & tiered retention
+description: Declarative, per-connector pre-ingest filter rules that drop or keep events before they hit hot storage, plus the tiered retention model that backs them.
+---
+
+# Security data pipeline and tiered retention
+
+Most SIEMs treat ingest as a tax: every event goes in at the same cost, and you pay until the bill forces you to drop sources. AiSOC takes a different stance — **decide what to keep before it crosses the wire**, with simple, declarative rules that are easy to read in a code review.
+
+## The pre-ingest filter
+
+Every connector instance can carry a `filter_rules` block in its config. On every poll, after normalization but before the events are pushed to ingest, the scheduler runs the filter and either drops or keeps each event.
+
+```yaml
+filter_rules:
+ - field: "raw.severity"
+ op: "eq"
+ value: "informational"
+ action: "drop"
+ - field: "raw.action"
+ op: "in"
+ value: ["allow", "ignore"]
+ action: "drop"
+ - field: "raw.user.email"
+ op: "ends_with"
+ value: "@partner.example.com"
+ action: "keep"
+```
+
+Rules are evaluated **first match wins**. An event with no matching rule is kept by default — filters are an opt-in deny list, not a strict allow list.
+
+### Supported operators
+
+| Operator | Meaning |
+|---|---|
+| `eq` / `ne` | Exact equality / inequality. |
+| `contains` | Substring match (case-sensitive). |
+| `starts_with` / `ends_with` | Prefix / suffix match. |
+| `in` | Value is in a list of allowed values. |
+
+The `field` selector is a dotted path into the normalized event (e.g. `raw.severity`, `entities.user.email`). Missing fields cause the rule to skip rather than crash, so a malformed rule degrades gracefully — the event passes through.
+
+### Why declarative, not a Lua hook
+
+We deliberately keep the filter language small:
+
+- **It reads in a code review.** A junior analyst can audit a YAML block.
+- **It serializes cleanly.** Filter rules live in `connector_config.filter_rules` and are versioned with the rest of the connector instance.
+- **It can't fork, exec, or call out.** The pre-ingest path is on the hot poll loop. Limiting expressiveness is a feature, not a bug.
+
+If you genuinely need a Turing-complete enrichment step, that belongs in the enrichment service, not the filter.
+
+## Drop counters
+
+Every dropped event increments `events_dropped` on the connector instance. That counter is surfaced both per-instance and aggregated across the tenant in `GET /api/v1/connectors/health`:
+
+```json
+{
+ "events_dropped_total": 247,
+ "instances": [
+ { "name": "cloudflare-prod", "events_dropped": 247, ... }
+ ]
+}
+```
+
+This lets you answer "is the filter doing what I think it's doing?" without grepping logs.
+
+## Tiered retention model
+
+The pipeline assumes a three-tier storage layout:
+
+| Tier | Backend | Default retention | Purpose |
+|---|---|---|---|
+| **Hot** | OpenSearch | 7-30 days | Detection rules, real-time queries, the last week of activity. |
+| **Warm** | ClickHouse / Parquet on object store | 90 days | Historical investigation, hunt queries, compliance look-back. |
+| **Cold** | Object store (S3 / R2) | 1 year+ | Compliance archive. Queryable on demand, cheap at rest. |
+
+The pre-ingest filter sits *in front* of all three: an event the filter drops never lands in any tier. That makes it the highest-leverage knob you have for cost control — drops are free, retries on hot storage are not.
+
+## Operational guidance
+
+- **Start permissive, then tighten.** First week of any new connector: no filter rules. Watch the volume. Then add rules to drop the noisy floor (informational severity, repeated noisy-actor health checks, etc.).
+- **Filter rules are per-instance.** Two GitHub connectors for two orgs can have completely different rule sets.
+- **`events_dropped` is monotonic.** Reset the counter by re-creating the instance if you want a clean window.
+- **Don't filter what detections need.** A detection that triggers on `severity=informational` will silently never fire if a filter drops those events. Review filter rules whenever you ship a new detection rule.
+
+## Where it lives in the code
+
+- Filter engine: `services/connectors/app/pipeline/filter_rules.py`
+- Filter integration in the polling loop: `services/connectors/app/scheduler.py`
+- Drop counter persistence: `services/connectors/app/db/connector_repo.py`
+- API surface: `services/api/app/api/v1/endpoints/connectors.py`
diff --git a/apps/docs/sidebars.ts b/apps/docs/sidebars.ts
index a18202209..911212918 100644
--- a/apps/docs/sidebars.ts
+++ b/apps/docs/sidebars.ts
@@ -12,6 +12,15 @@ const sidebars: SidebarsConfig = {
label: "Core Concepts",
items: ["concepts/cases", "concepts/playbooks", "concepts/detections"],
},
+ {
+ type: "category",
+ label: "Features",
+ items: [
+ "features/connector-health",
+ "features/data-pipeline",
+ "features/attack-path-agent",
+ ],
+ },
{
type: "category",
label: "Connectors",
diff --git a/apps/web/src/components/cases/CaseWorkspace.tsx b/apps/web/src/components/cases/CaseWorkspace.tsx
index 07a646fb2..002327b4e 100644
--- a/apps/web/src/components/cases/CaseWorkspace.tsx
+++ b/apps/web/src/components/cases/CaseWorkspace.tsx
@@ -24,7 +24,9 @@ import { format, formatDistanceToNow } from 'date-fns';
import toast from 'react-hot-toast';
import {
casesApi,
+ graphApi,
type Case,
+ type CaseAttackPath,
type CaseSeverity,
type CaseStatus,
type CaseTask,
@@ -36,9 +38,20 @@ import { EmptyState } from '@/components/ui/EmptyState';
import { InvestigationLedger } from './InvestigationLedger';
import { ContextualActions } from '@/components/copilot/ContextualActions';
-type WorkspaceTab = 'overview' | 'investigation' | 'ledger' | 'report';
-
-const VALID_TABS: readonly WorkspaceTab[] = ['overview', 'investigation', 'ledger', 'report'];
+type WorkspaceTab =
+ | 'overview'
+ | 'investigation'
+ | 'attack-path'
+ | 'ledger'
+ | 'report';
+
+const VALID_TABS: readonly WorkspaceTab[] = [
+ 'overview',
+ 'investigation',
+ 'attack-path',
+ 'ledger',
+ 'report',
+];
function isWorkspaceTab(value: string | null): value is WorkspaceTab {
return value !== null && (VALID_TABS as readonly string[]).includes(value);
@@ -701,7 +714,7 @@ export function CaseWorkspace({ caseId }: { caseId: string }) {
{/* Tabs */}
<div className="flex items-center gap-1 border-b border-slate-800/70 text-xs">
- {(['overview', 'investigation', 'ledger', 'report'] as WorkspaceTab[]).map((tab) => (
+ {VALID_TABS.map((tab) => (
<button
key={tab}
onClick={() => setActiveTab(tab)}
@@ -725,6 +738,16 @@ export function CaseWorkspace({ caseId }: { caseId: string }) {
audit
</span>
</span>
+ ) : tab === 'attack-path' ? (
+ <span className="inline-flex items-center gap-1">
+ Attack path
+ <span
+ className="rounded bg-orange-500/15 px-1 py-0.5 text-[8px] font-bold uppercase tracking-wider text-orange-300 ring-1 ring-orange-500/30"
+ title="Reconstructed attack graph from the Attack-Path Investigation Agent"
+ >
+ graph
+ </span>
+ </span>
) : (
tab
)}
@@ -745,6 +768,11 @@ export function CaseWorkspace({ caseId }: { caseId: string }) {
/>
)}
+ {/* Attack path tab — reconstructed graph for this case */}
+ {activeTab === 'attack-path' && (
+ <AttackPathPanel caseId={caseRecord.id} />
+ )}
+
{/* Ledger tab — persistent, replayable agent decision log */}
{activeTab === 'ledger' && (
<InvestigationLedger
@@ -1099,6 +1127,148 @@ function InvestigationPanel({
);
}
+// ─── Attack path panel ────────────────────────────────────────────────────────
+
+/**
+ * AttackPathPanel renders the case-scoped attack-path graph reconstructed by
+ * the Attack-Path Investigation Agent. Cases without graph entities (404 from
+ * the backend) fall back to an explanatory empty state so the analyst knows
+ * the agent is wired up but simply has no graph context yet.
+ */
+function AttackPathPanel({ caseId }: { caseId: string }) {
+ const { data, error, isLoading } = useSWR<CaseAttackPath | null>(
+ `case:${caseId}:attack-path`,
+ () => graphApi.getCaseAttackPath(caseId, { maxDepth: 4 }),
+ { revalidateOnFocus: false },
+ );
+
+ if (isLoading) {
+ return (
+ <div className="space-y-3">
+ <Skeleton className="h-6 w-48" />
+ <Skeleton className="h-32 w-full" />
+ <Skeleton className="h-32 w-full" />
+ </div>
+ );
+ }
+
+ if (error) {
+ return (
+ <ErrorState
+ title="Failed to load attack path"
+ description={error instanceof Error ? error.message : 'Unknown error'}
+ />
+ );
+ }
+
+ if (!data || data.nodeCount === 0) {
+ return (
+ <div className="flex flex-col items-center justify-center rounded-xl border border-dashed border-slate-700/60 bg-slate-900/30 py-16 text-center">
+ <div className="mb-3 flex h-10 w-10 items-center justify-center rounded-full bg-orange-500/10 ring-1 ring-orange-500/30">
+ <svg className="h-5 w-5 text-orange-300" fill="none" viewBox="0 0 24 24" stroke="currentColor">
+ <path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M13 10V3L4 14h7v7l9-11h-7z" />
+ </svg>
+ </div>
+ <p className="text-sm font-medium text-slate-300">No attack path yet</p>
+ <p className="mt-1 max-w-md text-xs text-slate-500">
+ The Attack-Path Investigation Agent reconstructs lateral movement and blast radius from linked alerts and graph entities. Run an investigation or link more alerts to surface a path here.
+ </p>
+ </div>
+ );
+ }
+
+ // Bucket nodes by entity type so the panel reads as &ldquo;hosts /
+ // identities / processes&rdquo; rather than an undifferentiated soup.
+ const nodesByType = data.nodes.reduce<Record<string, CaseAttackPath['nodes']>>(
+ (acc, node) => {
+ const type = (node.properties?.type as string) ?? node.label ?? 'entity';
+ (acc[type] ||= []).push(node);
+ return acc;
+ },
+ {},
+ );
+
+ return (
+ <div className="space-y-4">
+ <div className="flex items-center justify-between">
+ <div>
+ <h3 className="text-sm font-medium text-slate-200">Reconstructed attack path</h3>
+ <p className="text-xs text-slate-500">
+ {data.nodeCount} node{data.nodeCount === 1 ? '' : 's'} · {data.edgeCount} edge{data.edgeCount === 1 ? '' : 's'} · depth ≤ 4
+ </p>
+ </div>
+ <span className="inline-flex items-center gap-1.5 rounded-full bg-orange-500/10 px-2.5 py-0.5 text-xs font-medium text-orange-300 ring-1 ring-orange-500/30">
+ attack path
+ </span>
+ </div>
+
+ {/* Entities grouped by type — keeps the panel scannable even when the
+ backend returns dense graphs (10s of nodes is normal). */}
+ <div className="grid grid-cols-1 gap-4 lg:grid-cols-2">
+ {Object.entries(nodesByType).map(([type, nodes]) => (
+ <div
+ key={type}
+ className="rounded-xl border border-slate-800/80 bg-slate-900/40 p-4"
+ >
+ <h4 className="mb-2 text-xs font-semibold uppercase tracking-wide text-slate-400">
+ {type} ({nodes.length})
+ </h4>
+ <ul className="space-y-1.5">
+ {nodes.slice(0, 8).map((node) => (
+ <li
+ key={node.id}
+ className="flex items-start gap-2 text-xs"
+ >
+ <span className="mt-0.5 inline-block h-1.5 w-1.5 rounded-full bg-orange-400/60 shrink-0" />
+ <div className="min-w-0">
+ <p className="truncate font-mono text-slate-300">{node.label || node.id}</p>
+ {typeof node.properties?.risk_score === 'number' && (
+ <p className="text-[10px] text-slate-500">
+ risk score: {(node.properties.risk_score as number).toFixed(2)}
+ </p>
+ )}
+ </div>
+ </li>
+ ))}
+ {nodes.length > 8 && (
+ <li className="text-[10px] italic text-slate-500">
+ +{nodes.length - 8} more
+ </li>
+ )}
+ </ul>
+ </div>
+ ))}
+ </div>
+
+ {/* Edges — show the relationship verbs because that's what differentiates
+ a benign topology snapshot from an actual attack chain. */}
+ {data.edges.length > 0 && (
+ <div className="rounded-xl border border-slate-800/80 bg-slate-900/40 p-4">
+ <h4 className="mb-2 text-xs font-semibold uppercase tracking-wide text-slate-400">
+ Relationships
+ </h4>
+ <ul className="space-y-1">
+ {data.edges.slice(0, 12).map((edge, i) => (
+ <li key={`${edge.source}-${edge.target}-${i}`} className="flex items-center gap-2 text-[11px] font-mono">
+ <span className="truncate text-slate-300">{edge.source}</span>
+ <span className="rounded bg-slate-800 px-1.5 py-0.5 text-[10px] text-orange-300">
+ {edge.type}
+ </span>
+ <span className="truncate text-slate-300">{edge.target}</span>
+ </li>
+ ))}
+ {data.edges.length > 12 && (
+ <li className="text-[10px] italic text-slate-500">
+ +{data.edges.length - 12} more relationships
+ </li>
+ )}
+ </ul>
+ </div>
+ )}
+ </div>
+ );
+}
+
function AuditLogPreview({ log }: { log: Array<{ kind: string; agent: string; summary: string }> }) {
return (
<ul className="p-4 space-y-1.5">
diff --git a/apps/web/src/components/connectors/ConnectorInstanceList.tsx b/apps/web/src/components/connectors/ConnectorInstanceList.tsx
index d69b2a74d..29922e783 100644
--- a/apps/web/src/components/connectors/ConnectorInstanceList.tsx
+++ b/apps/web/src/components/connectors/ConnectorInstanceList.tsx
@@ -103,6 +103,28 @@ function ConnectorCard({
}: ConnectorCardProps) {
const statusCfg = STATUS_CONFIG[connector.status] ?? STATUS_CONFIG.configuring;
const alertCount = connector.alertCount ?? connector.alertsIngested ?? 0;
+ const eventsDropped = connector.eventsDropped ?? 0;
+ const driftAt = connector.lastSchemaDriftAt;
+ const drift = connector.lastDriftDetails;
+ const driftAdded = drift?.added?.length ?? 0;
+ const driftRemoved = drift?.removed?.length ?? 0;
+ const driftRecent =
+ driftAt &&
+ Date.now() - new Date(driftAt).getTime() < 24 * 60 * 60 * 1000;
+
+ // Compose the drift tooltip lazily — only the schema-drift sentinel emits a
+ // diff so we keep the UI quiet when nothing has changed.
+ const driftTooltip = drift
+ ? [
+ driftAdded > 0 ? `+${driftAdded} new field${driftAdded === 1 ? '' : 's'}` : null,
+ driftRemoved > 0
+ ? `-${driftRemoved} removed field${driftRemoved === 1 ? '' : 's'}`
+ : null,
+ driftAt ? `at ${new Date(driftAt).toLocaleString()}` : null,
+ ]
+ .filter(Boolean)
+ .join(' • ')
+ : '';
return (
<div className="bg-gray-900/60 border border-gray-800/60 rounded-xl p-5 hover:border-gray-700/60 transition-colors flex flex-col">
@@ -132,6 +154,57 @@ function ConnectorCard({
<p className="text-xs text-gray-500 mb-4 line-clamp-2">{connector.description}</p>
)}
+ {/* Schema-drift + filter-drop badges. Only render when there is
+ something to surface so a healthy connector keeps its clean look. */}
+ {(driftAt || eventsDropped > 0) && (
+ <div className="flex flex-wrap gap-1.5 mb-3">
+ {driftAt && (
+ <span
+ title={driftTooltip}
+ className={clsx(
+ 'text-[10px] px-2 py-0.5 rounded-full border inline-flex items-center gap-1',
+ driftRecent
+ ? 'text-amber-300 bg-amber-500/10 border-amber-500/30'
+ : 'text-gray-400 bg-gray-800/60 border-gray-700/50',
+ )}
+ >
+ <svg className="w-3 h-3" fill="none" viewBox="0 0 24 24" stroke="currentColor">
+ <path
+ strokeLinecap="round"
+ strokeLinejoin="round"
+ strokeWidth={2}
+ d="M12 9v2m0 4h.01M5 19h14a2 2 0 001.74-2.99l-7-12a2 2 0 00-3.48 0l-7 12A2 2 0 005 19z"
+ />
+ </svg>
+ {driftRecent ? 'Schema drift' : 'Schema changed'}
+ {(driftAdded > 0 || driftRemoved > 0) && (
+ <span className="opacity-80">
+ {driftAdded > 0 && `+${driftAdded}`}
+ {driftAdded > 0 && driftRemoved > 0 && '/'}
+ {driftRemoved > 0 && `-${driftRemoved}`}
+ </span>
+ )}
+ </span>
+ )}
+ {eventsDropped > 0 && (
+ <span
+ title={`${eventsDropped.toLocaleString()} event${eventsDropped === 1 ? '' : 's'} dropped by pre-ingest filter rules`}
+ className="text-[10px] px-2 py-0.5 rounded-full border inline-flex items-center gap-1 text-blue-300 bg-blue-500/10 border-blue-500/30"
+ >
+ <svg className="w-3 h-3" fill="none" viewBox="0 0 24 24" stroke="currentColor">
+ <path
+ strokeLinecap="round"
+ strokeLinejoin="round"
+ strokeWidth={2}
+ d="M3 4a1 1 0 011-1h16a1 1 0 011 1v2.586a1 1 0 01-.293.707l-6.414 6.414a1 1 0 00-.293.707V17l-4 4v-6.586a1 1 0 00-.293-.707L3.293 7.293A1 1 0 013 6.586V4z"
+ />
+ </svg>
+ {eventsDropped.toLocaleString()} dropped
+ </span>
+ )}
+ </div>
+ )}
+
<div className="grid grid-cols-2 gap-2 mb-4">
<div className="bg-gray-800/60 rounded-lg p-2">
<p className="text-xs text-gray-500">Events ingested</p>
diff --git a/apps/web/src/components/connectors/ConnectorsView.tsx b/apps/web/src/components/connectors/ConnectorsView.tsx
index c6c535e55..d3d3b5460 100644
--- a/apps/web/src/components/connectors/ConnectorsView.tsx
+++ b/apps/web/src/components/connectors/ConnectorsView.tsx
@@ -20,7 +20,11 @@ import useSWR from 'swr';
import toast from 'react-hot-toast';
import { clsx } from 'clsx';
-import { connectorsApi, type Connector } from '@/lib/api';
+import {
+ connectorsApi,
+ type Connector,
+ type ConnectorHealthSummary,
+} from '@/lib/api';
import { ErrorState } from '@/components/ui/ErrorState';
import { AddConnectorModal } from './AddConnectorModal';
import { ConnectorInstanceList } from './ConnectorInstanceList';
@@ -73,18 +77,56 @@ export function ConnectorsView() {
{ revalidateOnFocus: false, fallbackData: { connectors: DEMO_CONNECTORS } },
);
+ // Health summary is a separate endpoint so the empty/error case is silent —
+ // we just fall back to the locally-derived stats. SWR keys are namespaced so
+ // the two queries don't fight over the cache.
+ const { data: healthSummary } = useSWR<ConnectorHealthSummary | null>(
+ 'connectors:health',
+ async () => {
+ try {
+ return await connectorsApi.health();
+ } catch {
+ return null;
+ }
+ },
+ { revalidateOnFocus: false, refreshInterval: 60_000 },
+ );
+
const connectors: Connector[] = useMemo(() => data?.connectors ?? [], [data]);
- const stats = useMemo(() => {
+ const localStats = useMemo(() => {
const active = connectors.filter((c) => c.status === 'active').length;
const errored = connectors.filter((c) => c.status === 'error').length;
const totalEvents = connectors.reduce(
(sum, c) => sum + (c.alertCount ?? c.alertsIngested ?? 0),
0,
);
- return { active, errored, totalEvents };
+ const driftedRecently = connectors.filter((c) => {
+ if (!c.lastSchemaDriftAt) return false;
+ return Date.now() - new Date(c.lastSchemaDriftAt).getTime() < 24 * 60 * 60 * 1000;
+ }).length;
+ const totalDropped = connectors.reduce(
+ (sum, c) => sum + (c.eventsDropped ?? 0),
+ 0,
+ );
+ return { active, errored, totalEvents, driftedRecently, totalDropped };
}, [connectors]);
+ // Prefer backend-aggregated stats when the API is reachable so the tile
+ // matches the source of truth (the schema-drift sentinel runs server-side).
+ const stats = useMemo(() => {
+ if (healthSummary) {
+ return {
+ active: healthSummary.healthy,
+ errored: healthSummary.unhealthy,
+ totalEvents: healthSummary.totalEventsIngested,
+ driftedRecently: healthSummary.driftedRecently,
+ totalDropped: healthSummary.totalEventsDropped,
+ };
+ }
+ return localStats;
+ }, [healthSummary, localStats]);
+
// ─── Actions ──────────────────────────────────────────────────────────────
const handleTest = async (id: string) => {
@@ -164,26 +206,61 @@ export function ConnectorsView() {
</div>
{/* Stats — render even with zero connectors so the layout is stable
- between empty/populated states. */}
- <div className="grid grid-cols-2 md:grid-cols-4 gap-3">
+ between empty/populated states. The drift/dropped tiles only render
+ when the schema-drift sentinel has actually fired so a clean tenant
+ keeps a calm 4-tile layout. */}
+ <div
+ className={clsx(
+ 'grid grid-cols-2 gap-3',
+ stats.driftedRecently > 0 || stats.totalDropped > 0
+ ? 'md:grid-cols-6'
+ : 'md:grid-cols-4',
+ )}
+ >
{[
- { label: 'Total Connectors', value: connectors.length, color: 'text-blue-400' },
- { label: 'Active', value: stats.active, color: 'text-green-400' },
- { label: 'Errors', value: stats.errored, color: 'text-red-400' },
+ {
+ label: 'Total Connectors',
+ value: healthSummary?.total ?? connectors.length,
+ color: 'text-blue-400',
+ show: true,
+ },
+ { label: 'Active', value: stats.active, color: 'text-green-400', show: true },
+ { label: 'Errors', value: stats.errored, color: 'text-red-400', show: true },
{
label: 'Events Ingested',
value: stats.totalEvents.toLocaleString(),
color: 'text-purple-400',
+ show: true,
+ },
+ {
+ label: 'Schema Drift (24h)',
+ value: stats.driftedRecently,
+ color: 'text-amber-400',
+ show: stats.driftedRecently > 0,
+ tooltip:
+ healthSummary?.lastDriftAt
+ ? `Most recent drift: ${new Date(healthSummary.lastDriftAt).toLocaleString()}`
+ : undefined,
+ },
+ {
+ label: 'Events Dropped',
+ value: stats.totalDropped.toLocaleString(),
+ color: 'text-cyan-400',
+ show: stats.totalDropped > 0,
+ tooltip: 'Filtered out by pre-ingest pipeline rules',
},
- ].map((stat) => (
- <div
- key={stat.label}
- className="bg-gray-900/60 border border-gray-800/60 rounded-xl p-4"
- >
- <p className={clsx('text-2xl font-bold mb-1', stat.color)}>{stat.value}</p>
- <p className="text-xs text-gray-500">{stat.label}</p>
- </div>
- ))}
+ ]
+ .filter((stat) => stat.show)
+ .map((stat) => (
+ <div
+ key={stat.label}
+ title={stat.tooltip}
+ className="bg-gray-900/60 border border-gray-800/60 rounded-xl p-4"
+ >
+ <p className={clsx('text-2xl font-bold mb-1', stat.color)}>{stat.value}</p>
+ <p className="text-xs text-gray-500">{stat.label}</p>
+ </div>
+ ))}
</div>
{/* Body */}
diff --git a/apps/web/src/components/landing/Features.tsx b/apps/web/src/components/landing/Features.tsx
index 659de6e55..f67ba1936 100644
--- a/apps/web/src/components/landing/Features.tsx
+++ b/apps/web/src/components/landing/Features.tsx
@@ -11,6 +11,14 @@ const FEATURES = [
<path d="M4 4h16v4H4zM4 10h10v4H4zM4 16h16v4H4z" />
),
},
+ {
+ title: 'Tiered retention pipeline',
+ description:
+ 'Declarative pre-ingest filter rules drop noisy events at the connector edge before they hit Kafka — cutting indexing cost without losing forensic depth. Every drop is counted per-connector and surfaced in the health summary.',
+ icon: (
+ <path d="M3 4a1 1 0 011-1h16a1 1 0 011 1v2.586a1 1 0 01-.293.707l-6.414 6.414a1 1 0 00-.293.707V17l-4 4v-6.586a1 1 0 00-.293-.707L3.293 7.293A1 1 0 013 6.586V4z" />
+ ),
+ },
{
title: 'Three-tier agent memory',
description:
@@ -75,6 +83,14 @@ const FEATURES = [
<path d="M5 5a3 3 0 116 0 3 3 0 01-6 0zm8 14a3 3 0 116 0 3 3 0 01-6 0zM7.5 8l5 8" />
),
},
+ {
+ title: 'Attack-path investigation agent',
+ description:
+ 'A dedicated LangGraph agent reconstructs lateral movement and blast radius from the live graph, summarises high-risk pivots, and proposes contained response actions (host isolation, account disable) gated by autonomy guardrails.',
+ icon: (
+ <path d="M13 10V3L4 14h7v7l9-11h-7z" />
+ ),
+ },
{
title: 'SOC metrics dashboard',
description:
@@ -91,6 +107,14 @@ const FEATURES = [
<path d="M9 3H5a2 2 0 00-2 2v4m6-6h10a2 2 0 012 2v4M9 3v18m0 0h10a2 2 0 002-2V9M9 21H5a2 2 0 01-2-2V9m0 0h18" />
),
},
+ {
+ title: 'Schema-drift sentinel',
+ description:
+ 'Every poll fingerprints the upstream payload shape and diffs it against the last known-good schema. New, removed, or renamed fields surface in the connector card and aggregate health tile before they silently break detections.',
+ icon: (
+ <path d="M12 9v2m0 4h.01M5 19h14a2 2 0 001.74-2.99l-7-12a2 2 0 00-3.48 0l-7 12A2 2 0 005 19z" />
+ ),
+ },
{
title: 'Compliance & governance',
description:
diff --git a/apps/web/src/lib/api.ts b/apps/web/src/lib/api.ts
index 658ad8ba8..460160572 100644
--- a/apps/web/src/lib/api.ts
+++ b/apps/web/src/lib/api.ts
@@ -761,6 +761,18 @@ export interface Connector {
description?: string;
createdAt?: string;
updatedAt?: string;
+ /** SHA-256 fingerprint of the most recent normalized event schema. */
+ schemaFingerprint?: string;
+ /** Timestamp of the most recent schema-drift event (null/undef if never). */
+ lastSchemaDriftAt?: string;
+ /** Human-readable diff for the most recent drift event. */
+ lastDriftDetails?: {
+ added?: string[];
+ removed?: string[];
+ previous_fingerprint?: string;
+ };
+ /** Cumulative count of events the pre-ingest filter dropped. */
+ eventsDropped?: number;
}
export interface ConnectorsResponse {
@@ -830,6 +842,32 @@ interface BackendConnectorResponse {
tags: string[];
created_at: string;
updated_at: string;
+ schema_fingerprint?: string | null;
+ last_schema_drift_at?: string | null;
+ last_drift_details?: {
+ added?: string[];
+ removed?: string[];
+ previous_fingerprint?: string;
+ } | null;
+ events_dropped?: number | null;
+}
+
+/** Aggregated health summary for the connectors fleet (a single tenant). */
+export interface ConnectorHealthSummary {
+ total: number;
+ healthy: number;
+ unhealthy: number;
+ unknown: number;
+ /** Connectors whose schema has ever drifted. */
+ drifted: number;
+ /** Connectors whose schema drifted in the last 24h. */
+ driftedRecently: number;
+ /** Most-recent drift timestamp across the fleet. */
+ lastDriftAt?: string;
+ /** Cumulative events ingested across all enabled connectors. */
+ totalEventsIngested: number;
+ /** Cumulative events dropped by pre-ingest filter rules. */
+ totalEventsDropped: number;
}
/**
@@ -870,6 +908,10 @@ function mapBackendConnector(row: BackendConnectorResponse): Connector {
alertsIngested: row.events_ingested,
createdAt: row.created_at,
updatedAt: row.updated_at,
+ schemaFingerprint: row.schema_fingerprint ?? undefined,
+ lastSchemaDriftAt: row.last_schema_drift_at ?? undefined,
+ lastDriftDetails: row.last_drift_details ?? undefined,
+ eventsDropped: row.events_dropped ?? 0,
};
}
@@ -960,6 +1002,39 @@ export const connectorsApi = {
delete: (id: string) =>
request<void>(`/api/v1/connectors/${id}`, { method: 'DELETE' }),
+
+ /**
+ * Aggregated health summary for the connectors fleet.
+ *
+ * Surfaces healthy/degraded/error counts, schema-drift events seen in the
+ * last 24h, and the cumulative count of events the pre-ingest filter
+ * dropped. Used by the Connectors page header strip and the new
+ * "Connector Health" widget on the dashboard.
+ */
+ health: async (): Promise<ConnectorHealthSummary> => {
+ const raw = await request<{
+ total: number;
+ healthy: number;
+ unhealthy: number;
+ unknown: number;
+ drifted: number;
+ drifted_recently: number;
+ last_drift_at: string | null;
+ total_events_ingested: number;
+ total_events_dropped: number;
+ }>('/api/v1/connectors/health');
+ return {
+ total: raw.total,
+ healthy: raw.healthy,
+ unhealthy: raw.unhealthy,
+ unknown: raw.unknown,
+ drifted: raw.drifted,
+ driftedRecently: raw.drifted_recently,
+ lastDriftAt: raw.last_drift_at ?? undefined,
+ totalEventsIngested: raw.total_events_ingested,
+ totalEventsDropped: raw.total_events_dropped,
+ };
+ },
};
// ─── Threat Intel ─────────────────────────────────────────────────────────────
@@ -1169,6 +1244,41 @@ export interface MitreCoverage {
generatedAt: string;
}
+/**
+ * Case-scoped attack path graph as returned by the Attack-Path Investigation
+ * Agent backend (`/api/v1/graph/attack-path/{case_id}`). Distinct from the
+ * tenant-level `AttackGraph` overview because the backend payload schema is
+ * leaner: each node carries a `properties` dict and edges carry a relationship
+ * `type` instead of label/weight metadata.
+ */
+export interface CaseAttackPathNode {
+ id: string;
+ label: string;
+ properties: Record<string, unknown>;
+}
+
+export interface CaseAttackPathEdge {
+ source: string;
+ target: string;
+ type: string;
+}
+
+export interface CaseAttackPath {
+ caseId: string;
+ nodes: CaseAttackPathNode[];
+ edges: CaseAttackPathEdge[];
+ nodeCount: number;
+ edgeCount: number;
+}
+
+interface BackendCaseAttackPathResponse {
+ case_id: string;
+ nodes: CaseAttackPathNode[];
+ edges: CaseAttackPathEdge[];
+ node_count: number;
+ edge_count: number;
+}
+
export const graphApi = {
getOverview: (filters: { entity?: string; depth?: number } = {}) =>
request<AttackGraph>('/api/v1/graph', {
@@ -1188,6 +1298,37 @@ export const graphApi = {
`/api/v1/graph/blast-radius`,
{ params: { entity } },
),
+
+ /**
+ * Fetch the reconstructed attack-path graph for a single case.
+ * Backed by the Attack-Path Investigation Agent endpoint
+ * `/api/v1/graph/attack-path/{case_id}`.
+ * Returns `null` when the case has no linked graph entities (404 from API).
+ */
+ getCaseAttackPath: async (
+ caseId: string,
+ options: { maxDepth?: number } = {},
+ ): Promise<CaseAttackPath | null> => {
+ try {
+ const raw = await request<BackendCaseAttackPathResponse>(
+ `/api/v1/graph/attack-path/${encodeURIComponent(caseId)}`,
+ {
+ params: options.maxDepth ? { max_depth: options.maxDepth } : {},
+ },
+ );
+ return {
+ caseId: raw.case_id,
+ nodes: raw.nodes ?? [],
+ edges: raw.edges ?? [],
+ nodeCount: raw.node_count ?? 0,
+ edgeCount: raw.edge_count ?? 0,
+ };
+ } catch (err) {
+ const apiErr = err as { status?: number };
+ if (apiErr?.status === 404) return null;
+ throw err;
+ }
+ },
};
// ─── Detection Rules ─────────────────────────────────────────────────────────
diff --git a/services/agents/app/agents/attack_path_agent.py b/services/agents/app/agents/attack_path_agent.py
new file mode 100644
index 000000000..b9f9b218c
--- /dev/null
+++ b/services/agents/app/agents/attack_path_agent.py
@@ -0,0 +1,255 @@
+"""
+Attack-Path Investigation Agent.
+
+Runs after the standard investigation node. Walks the knowledge graph
+(Case → Alerts → Hosts/Users → IOCs → MITRE techniques) and computes blast
+radius for the highest-impact pivot points. Adds graph-derived findings and
+proposed actions to the InvestigationState.
+
+This is the Neo4j + LangGraph integration step: it converts the static
+investigation report into a connected story that explains *how an attacker
+could reach the crown jewels from the observed indicators*.
+"""
+
+from __future__ import annotations
+
+import os
+from collections import Counter
+from typing import Any
+
+import structlog
+
+from app.models.state import ActionRisk, AgentStatus, InvestigationState, ProposedAction
+from app.tools.graph import get_attack_path, get_blast_radius
+
+logger = structlog.get_logger()
+
+# Internal service token used when the agents service calls the API. In dev
+# this is set via the AGENTS_API_TOKEN env var; in production it is rotated
+# alongside the rest of the service-mesh creds.
+_SERVICE_TOKEN = os.getenv("AGENTS_API_TOKEN")
+
+# Score thresholds (0.0–1.0) that mirror the buckets in graph_service.py.
+_BLAST_HIGH = 0.7
+_BLAST_MED = 0.4
+
+# Cap how many entities we run blast-radius on per run to keep tail latency
+# bounded — the graph traversal is the most expensive part of the workflow.
+_MAX_BLAST_ENTITIES = 5
+
+
+async def run_attack_path(state: InvestigationState) -> InvestigationState:
+ """Walk the graph for the case linked to this investigation and enrich state."""
+ logger.info("Attack-path agent starting", incident_id=str(state.incident_id))
+ state.iteration_count += 1
+
+ case_id = _resolve_case_id(state)
+ if not case_id:
+ # No case linkage yet — agent has nothing to traverse. Stay silent so
+ # we don't pollute the report with noise; the workflow is still
+ # complete because investigation_node already set status.
+ logger.debug("No case_id on state; skipping attack-path agent")
+ return state
+
+ path = await get_attack_path(case_id=case_id, api_token=_SERVICE_TOKEN)
+ if path.get("error") or not path.get("nodes"):
+ state.add_finding(
+ f"Attack-path graph empty for case {case_id} "
+ f"(node_count={path.get('node_count', 0)})."
+ )
+ return state
+
+ # ---- Summarise the path ----
+ nodes: list[dict[str, Any]] = path.get("nodes", [])
+ edges: list[dict[str, Any]] = path.get("edges", [])
+ label_counts = Counter(n.get("label", "Unknown") for n in nodes)
+ state.add_finding(
+ "Attack-path graph: "
+ + ", ".join(f"{c} {label}" for label, c in label_counts.most_common())
+ + f" across {len(edges)} relationships."
+ )
+
+ # Tactic chain (deduplicated, order-preserving) gives a quick read on
+ # how far through the kill chain the attacker has progressed.
+ tactic_chain = _extract_tactic_chain(nodes)
+ if tactic_chain:
+ state.add_finding(
+ f"Kill-chain stages reached: {' → '.join(tactic_chain)}."
+ )
+
+ # ---- Pick high-value entities to compute blast radius for ----
+ pivot_targets = _select_blast_targets(nodes)
+
+ high_blast: list[dict[str, Any]] = []
+ for pivot in pivot_targets[:_MAX_BLAST_ENTITIES]:
+ blast = await get_blast_radius(
+ entity_type=pivot["entity_type"],
+ entity_id=pivot["entity_id"],
+ api_token=_SERVICE_TOKEN,
+ )
+ if blast.get("error"):
+ continue
+ score = float(blast.get("blast_radius_score", 0.0))
+ if score >= _BLAST_HIGH:
+ high_blast.append(
+ {
+ **blast,
+ "label": pivot["label"],
+ "name": pivot.get("name", pivot["entity_id"]),
+ "entity_id": pivot["entity_id"],
+ "entity_type": pivot["entity_type"],
+ }
+ )
+
+ if high_blast:
+ for b in high_blast:
+ state.add_finding(
+ f"High blast radius from {b['label']} '{b['name']}': "
+ f"{b['total_affected']} entities reachable "
+ f"(score={b['blast_radius_score']:.2f}, "
+ f"breakdown={dict(b.get('type_breakdown', {}))})."
+ )
+
+ # Propose an isolation action for the highest-scoring pivot.
+ top = max(high_blast, key=lambda b: b["blast_radius_score"])
+ if top["label"] in ("Host", "User"):
+ state.proposed_actions.append(
+ ProposedAction(
+ action_type=("isolate_host" if top["label"] == "Host" else "disable_user"),
+ description=(
+ f"Isolate {top['label'].lower()} '{top['name']}' — "
+ f"{top['total_affected']} downstream entities at risk."
+ ),
+ risk_level=ActionRisk.HIGH,
+ target=top["name"],
+ requires_approval=True,
+ rationale=(
+ f"Graph blast radius {top['blast_radius_score']:.2f} "
+ f"with {top['total_affected']} reachable entities."
+ ),
+ parameters={
+ "entity_type": top["entity_type"],
+ "entity_id": top["entity_id"],
+ "blast_radius_score": top["blast_radius_score"],
+ },
+ )
+ )
+
+ # Persist a compact summary on the state so downstream services (UI,
+ # report writer) can render it without re-querying the graph.
+ state.threat_intel.setdefault("attack_path", {})
+ state.threat_intel["attack_path"] = {
+ "case_id": case_id,
+ "node_count": path.get("node_count", len(nodes)),
+ "edge_count": path.get("edge_count", len(edges)),
+ "tactic_chain": tactic_chain,
+ "label_counts": dict(label_counts),
+ "high_blast_entities": [
+ {
+ "label": b["label"],
+ "name": b["name"],
+ "score": b["blast_radius_score"],
+ "total_affected": b["total_affected"],
+ }
+ for b in high_blast
+ ],
+ }
+
+ # Don't override a terminal status set by the previous node, but if the
+ # workflow is still running, flag completion now.
+ if state.status == AgentStatus.RUNNING:
+ state.status = AgentStatus.COMPLETED
+
+ logger.info(
+ "Attack-path complete",
+ case_id=case_id,
+ node_count=len(nodes),
+ edge_count=len(edges),
+ high_blast=len(high_blast),
+ )
+ return state
+
+
+# ─── Helpers ──────────────────────────────────────────────────────────────────
+
+
+def _resolve_case_id(state: InvestigationState) -> str | None:
+ """Find the case id on the state, tolerant of where the orchestrator put it."""
+ raw = state.raw_alert or {}
+ return (
+ raw.get("case_id")
+ or raw.get("caseId")
+ or (state.threat_intel or {}).get("case_id")
+ )
+
+
+def _extract_tactic_chain(nodes: list[dict[str, Any]]) -> list[str]:
+ """Pull MITRE tactic names from Technique nodes, preserving first-seen order."""
+ seen: list[str] = []
+ for node in nodes:
+ if node.get("label") != "Technique":
+ continue
+ props = node.get("properties", {}) or {}
+ tactic = props.get("tactic_name") or props.get("tactic")
+ if tactic and tactic not in seen:
+ seen.append(tactic)
+ return seen
+
+
+def _select_blast_targets(nodes: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ """Rank Host/User/IOC nodes for blast-radius traversal.
+
+ Hosts and users with high criticality / risk_score are evaluated first;
+ IOCs are included as a tail because they're cheaper to score but rarely
+ surface new pivot points the existing investigation hasn't already used.
+ """
+ candidates: list[tuple[float, dict[str, Any]]] = []
+ for node in nodes:
+ label = node.get("label")
+ props = node.get("properties", {}) or {}
+ nid = node.get("id") or props.get("id") or props.get("value")
+ if not nid:
+ continue
+
+ if label == "Host":
+ crit = (props.get("criticality") or "medium").lower()
+ score = {"critical": 1.0, "high": 0.8, "medium": 0.5, "low": 0.2}.get(crit, 0.5)
+ candidates.append(
+ (
+ score,
+ {
+ "entity_type": "host",
+ "entity_id": nid,
+ "label": "Host",
+ "name": props.get("hostname", nid),
+ },
+ )
+ )
+ elif label == "User":
+ risk = float(props.get("risk_score", 0.0) or 0.0)
+ candidates.append(
+ (
+ min(1.0, risk + 0.3),
+ {
+ "entity_type": "user",
+ "entity_id": nid,
+ "label": "User",
+ "name": props.get("username", nid),
+ },
+ )
+ )
+ elif label == "IOC":
+ candidates.append(
+ (
+ 0.2,
+ {
+ "entity_type": "ioc",
+ "entity_id": nid,
+ "label": "IOC",
+ "name": props.get("value", nid),
+ },
+ )
+ )
+
+ candidates.sort(key=lambda x: x[0], reverse=True)
+ return [c[1] for c in candidates]
diff --git a/services/agents/app/graph/workflow.py b/services/agents/app/graph/workflow.py
index 3545cf04e..fc9acdda6 100644
--- a/services/agents/app/graph/workflow.py
+++ b/services/agents/app/graph/workflow.py
@@ -1,9 +1,11 @@
"""
-LangGraph workflow: wires Auto-Triage → Triage → Enrichment → Investigation agents.
+LangGraph workflow: wires Auto-Triage → Triage → Enrichment → Investigation
+→ Attack-Path agents.
Auto-triage runs first. If the LLM classifies the alert as FP/benign with
confidence above the auto-close threshold the graph terminates early.
-Otherwise the alert flows through the full manual pipeline.
+Otherwise the alert flows through the full manual pipeline, ending with the
+graph-aware Attack-Path agent that walks Neo4j to compute blast radius.
"""
from __future__ import annotations
@@ -11,6 +13,7 @@ from __future__ import annotations
import structlog
from langgraph.graph import END, StateGraph
+from app.agents.attack_path_agent import run_attack_path
from app.agents.auto_triage_agent import run_auto_triage
from app.agents.enrichment_agent import run_enrichment
from app.agents.investigation_agent import run_investigation
@@ -55,6 +58,12 @@ async def investigation_node(state: dict) -> dict:
return s.to_dict()
+async def attack_path_node(state: dict) -> dict:
+ s = _from_dict(state)
+ s = await run_attack_path(s)
+ return s.to_dict()
+
+
def _should_continue(state: dict) -> str:
"""Conditional edge: stop if max iterations reached or status is terminal."""
s = _from_dict(state)
@@ -78,7 +87,8 @@ def build_investigation_graph() -> StateGraph:
Flow:
auto_triage ─┬─ (high-confidence FP/benign) ──► END
- └─ (else) ──► triage ──► enrichment ──► investigation ──► END
+ └─ (else) ──► triage ──► enrichment ──► investigation
+ ──► attack_path ──► END
"""
graph = StateGraph(dict)
@@ -86,6 +96,7 @@ def build_investigation_graph() -> StateGraph:
graph.add_node("triage", triage_node)
graph.add_node("enrichment", enrichment_node)
graph.add_node("investigation", investigation_node)
+ graph.add_node("attack_path", attack_path_node)
graph.set_entry_point("auto_triage")
@@ -96,7 +107,8 @@ def build_investigation_graph() -> StateGraph:
)
graph.add_edge("triage", "enrichment")
graph.add_edge("enrichment", "investigation")
- graph.add_edge("investigation", END)
+ graph.add_edge("investigation", "attack_path")
+ graph.add_edge("attack_path", END)
return graph.compile()
diff --git a/services/agents/app/tools/graph.py b/services/agents/app/tools/graph.py
new file mode 100644
index 000000000..494ac73cc
--- /dev/null
+++ b/services/agents/app/tools/graph.py
@@ -0,0 +1,113 @@
+"""
+Tool: knowledge-graph queries via the API service.
+
+Wraps the `/api/v1/graph/*` endpoints exposed by `services/api`. The agents
+service uses these to walk attack paths, compute blast-radius severity, and
+discover the immediate neighbourhood of an entity during an investigation.
+"""
+
+from __future__ import annotations
+
+import os
+from typing import Any
+
+import httpx
+import structlog
+
+logger = structlog.get_logger()
+
+_API_URL = os.getenv("API_SERVICE_URL", "http://api:8000")
+_TIMEOUT = float(os.getenv("AGENTS_API_TIMEOUT", "10.0"))
+
+
+def _headers(api_token: str | None) -> dict[str, str]:
+ return {"Authorization": f"Bearer {api_token}"} if api_token else {}
+
+
+async def get_attack_path(
+ case_id: str,
+ api_token: str | None = None,
+ max_depth: int = 6,
+) -> dict[str, Any]:
+ """Return the Case → Alert → Host/User → IOC → Technique attack path."""
+ try:
+ async with httpx.AsyncClient(timeout=_TIMEOUT) as client:
+ resp = await client.get(
+ f"{_API_URL}/api/v1/graph/attack-path/{case_id}",
+ params={"max_depth": max_depth},
+ headers=_headers(api_token),
+ )
+ resp.raise_for_status()
+ return resp.json()
+ except httpx.HTTPStatusError as exc:
+ if exc.response.status_code == 404:
+ return {"case_id": case_id, "nodes": [], "edges": [], "node_count": 0, "edge_count": 0}
+ logger.warning("attack_path query failed", case_id=case_id, error=str(exc))
+ return {"error": str(exc), "case_id": case_id, "nodes": [], "edges": []}
+ except Exception as exc: # noqa: BLE001
+ logger.warning("attack_path query failed", case_id=case_id, error=str(exc))
+ return {"error": str(exc), "case_id": case_id, "nodes": [], "edges": []}
+
+
+async def get_blast_radius(
+ entity_type: str,
+ entity_id: str,
+ api_token: str | None = None,
+ hops: int = 3,
+) -> dict[str, Any]:
+ """Compute the blast radius starting from a Host/User/IOC/Alert node."""
+ try:
+ async with httpx.AsyncClient(timeout=_TIMEOUT) as client:
+ resp = await client.get(
+ f"{_API_URL}/api/v1/graph/blast-radius/{entity_type}/{entity_id}",
+ params={"hops": hops},
+ headers=_headers(api_token),
+ )
+ resp.raise_for_status()
+ return resp.json()
+ except Exception as exc: # noqa: BLE001
+ logger.warning(
+ "blast_radius query failed",
+ entity_type=entity_type,
+ entity_id=entity_id,
+ error=str(exc),
+ )
+ return {
+ "error": str(exc),
+ "entity_id": entity_id,
+ "entity_type": entity_type,
+ "affected_nodes": [],
+ "total_affected": 0,
+ "type_breakdown": {},
+ "blast_radius_score": 0.0,
+ }
+
+
+async def get_entity_neighbors(
+ entity_type: str,
+ entity_id: str,
+ api_token: str | None = None,
+) -> dict[str, Any]:
+ """Return all nodes directly connected (depth 1) to the specified entity."""
+ try:
+ async with httpx.AsyncClient(timeout=_TIMEOUT) as client:
+ resp = await client.get(
+ f"{_API_URL}/api/v1/graph/neighbors/{entity_type}/{entity_id}",
+ headers=_headers(api_token),
+ )
+ resp.raise_for_status()
+ return resp.json()
+ except Exception as exc: # noqa: BLE001
+ logger.warning(
+ "entity_neighbors query failed",
+ entity_type=entity_type,
+ entity_id=entity_id,
+ error=str(exc),
+ )
+ return {
+ "error": str(exc),
+ "entity_id": entity_id,
+ "entity_type": entity_type,
+ "neighbors": [],
+ "neighbor_count": 0,
+ }
diff --git a/services/agents/tests/test_attack_path_agent.py b/services/agents/tests/test_attack_path_agent.py
new file mode 100644
index 000000000..371deb69a
--- /dev/null
+++ b/services/agents/tests/test_attack_path_agent.py
@@ -0,0 +1,156 @@
+"""Tests for the Attack-Path Investigation Agent.
+
+These tests do not require a live Neo4j or API service; they monkeypatch the
+graph-tool calls so the agent can be exercised in isolation. The intent is to
+lock down:
+
+* findings / proposed_actions are populated only when the graph returns data
+* tactic chain extraction preserves order and deduplicates
+* high blast-radius hosts produce an isolation `ProposedAction`
+* the agent stays silent when no `case_id` is on state (no false findings)
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+from uuid import uuid4
+
+import pytest
+
+_AGENTS_ROOT = Path(__file__).resolve().parents[1]
+if str(_AGENTS_ROOT) not in sys.path:
+ sys.path.insert(0, str(_AGENTS_ROOT))
+
+# Imports must come after path mutation.
+from app.agents import attack_path_agent # noqa: E402
+from app.models.state import ( # noqa: E402
+ ActionRisk,
+ AgentStatus,
+ InvestigationState,
+)
+
+
+def _state(case_id: str | None = "case-123") -> InvestigationState:
+ return InvestigationState(
+ incident_id=uuid4(),
+ tenant_id=uuid4(),
+ alert_summary="EDR detection",
+ raw_alert={"case_id": case_id} if case_id else {},
+ status=AgentStatus.RUNNING,
+ )
+
+
+async def test_attack_path_skipped_when_no_case_id(monkeypatch: pytest.MonkeyPatch) -> None:
+ called = {"path": False, "blast": False}
+
+ async def _fake_path(*_a, **_kw):
+ called["path"] = True
+ return {"nodes": [], "edges": []}
+
+ async def _fake_blast(*_a, **_kw):
+ called["blast"] = True
+ return {}
+
+ monkeypatch.setattr(attack_path_agent, "get_attack_path", _fake_path)
+ monkeypatch.setattr(attack_path_agent, "get_blast_radius", _fake_blast)
+
+ state = _state(case_id=None)
+ result = await attack_path_agent.run_attack_path(state)
+
+ assert called["path"] is False
+ assert called["blast"] is False
+ assert result.findings == []
+ assert result.proposed_actions == []
+
+
+async def test_attack_path_records_summary_and_tactic_chain(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ nodes = [
+ {"id": "case-123", "label": "Case", "properties": {}},
+ {"id": "alert-1", "label": "Alert", "properties": {}},
+ {
+ "id": "host-1",
+ "label": "Host",
+ "properties": {"hostname": "web-01", "criticality": "high"},
+ },
+ {
+ "id": "tech-T1059",
+ "label": "Technique",
+ "properties": {"tactic_name": "Execution"},
+ },
+ {
+ "id": "tech-T1071",
+ "label": "Technique",
+ "properties": {"tactic_name": "C2"},
+ },
+ # Duplicate tactic should not appear twice in the chain.
+ {
+ "id": "tech-T1105",
+ "label": "Technique",
+ "properties": {"tactic_name": "C2"},
+ },
+ ]
+ edges = [{"from": "case-123", "to": "alert-1", "type": "HAS_ALERT"}] * 5
+
+ async def _fake_path(case_id: str, **_kw):
+ assert case_id == "case-123"
+ return {"nodes": nodes, "edges": edges, "node_count": 6, "edge_count": 5}
+
+ async def _fake_blast(entity_type: str, entity_id: str, **_kw):
+ # Only the high-criticality host should be queried first; return a
+ # high-blast result for it.
+ if entity_type == "host" and entity_id == "host-1":
+ return {
+ "blast_radius_score": 0.82,
+ "total_affected": 17,
+ "type_breakdown": {"User": 4, "IOC": 13},
+ "affected_nodes": [],
+ }
+ return {"blast_radius_score": 0.1, "total_affected": 0, "type_breakdown": {}}
+
+ monkeypatch.setattr(attack_path_agent, "get_attack_path", _fake_path)
+ monkeypatch.setattr(attack_path_agent, "get_blast_radius", _fake_blast)
+
+ state = _state()
+ result = await attack_path_agent.run_attack_path(state)
+
+ assert any("Attack-path graph" in f for f in result.findings)
+ assert any("Kill-chain stages reached" in f for f in result.findings)
+ assert any("High blast radius from Host 'web-01'" in f for f in result.findings)
+
+ chain = result.threat_intel["attack_path"]["tactic_chain"]
+ assert chain == ["Execution", "C2"], "duplicate tactics should be deduplicated in order"
+
+ isolate_actions = [a for a in result.proposed_actions if a.action_type == "isolate_host"]
+ assert len(isolate_actions) == 1
+ action = isolate_actions[0]
+ assert action.target == "web-01"
+ assert action.risk_level == ActionRisk.HIGH
+ assert action.requires_approval is True
+ assert action.parameters["entity_type"] == "host"
+ assert action.parameters["blast_radius_score"] == pytest.approx(0.82)
+
+
+async def test_attack_path_no_findings_for_empty_graph(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ async def _fake_path(*_a, **_kw):
+ return {"nodes": [], "edges": [], "node_count": 0, "edge_count": 0}
+
+ async def _fake_blast(*_a, **_kw):
+ # Should never be called when the graph is empty.
+ raise AssertionError("blast radius queried for empty graph")
+
+ monkeypatch.setattr(attack_path_agent, "get_attack_path", _fake_path)
+ monkeypatch.setattr(attack_path_agent, "get_blast_radius", _fake_blast)
+
+ state = _state()
+ result = await attack_path_agent.run_attack_path(state)
+
+ assert any("Attack-path graph empty" in f for f in result.findings)
+ assert result.proposed_actions == []
diff --git a/services/api/app/api/v1/endpoints/connectors.py b/services/api/app/api/v1/endpoints/connectors.py
index 4014478b6..24c8383c1 100644
--- a/services/api/app/api/v1/endpoints/connectors.py
+++ b/services/api/app/api/v1/endpoints/connectors.py
@@ -105,7 +105,15 @@ class ConnectorResponse(BaseModel):
last_health_check: datetime | None
last_sync: datetime | None
events_ingested: int
+ events_dropped: int = 0
error_count: int
+ # Schema-drift sentinel state surfaced to the wizard so the
+ # connector card can show "schema changed at <ts>" without a
+ # second round-trip. Both fields are nullable for connectors
+ # that haven't drifted (the steady-state expectation).
+ schema_fingerprint: str | None = None
+ last_schema_drift_at: datetime | None = None
+ last_drift_details: dict[str, Any] | None = None
tags: list
created_at: datetime
updated_at: datetime
@@ -113,6 +121,25 @@ class ConnectorResponse(BaseModel):
model_config = {"from_attributes": True}
+class ConnectorHealthSummary(BaseModel):
+ """Tenant-wide rollup of connector health for the dashboard.
+
+ The web console shows a small "Connector Health" tile at the top
+ of the Connectors page; this endpoint feeds that tile so the UI
+ can render it without iterating the full list client-side.
+ """
+
+ total: int
+ healthy: int
+ unhealthy: int
+ unknown: int
+ drifted: int
+ drifted_recently: int # last 24 h
+ last_drift_at: datetime | None
+ total_events_ingested: int
+ total_events_dropped: int
+
+
class CreateConnectorRequest(BaseModel):
name: str = Field(min_length=1, max_length=255)
connector_type: str = Field(min_length=1, max_length=100)
@@ -304,6 +331,73 @@ async def list_catalog(
return {"connectors": schemas}
+# ------------------------------------------------------------------ health summary
+
+
[email protected]("/health", response_model=ConnectorHealthSummary)
+async def connector_health_summary(
+ current_user: Annotated[AuthUser, Depends(require_permission("connectors:read"))],
+ db: DBSession,
+) -> ConnectorHealthSummary:
+ """Aggregate per-tenant connector health for the dashboard tile.
+
+ Single SELECT scan; we deliberately do *not* group in SQL because
+ Postgres' filter expressions over a few hundred rows are negligible
+ compared to the round-trip, and the resulting Python is a lot easier
+ to read than five COUNT(*) FILTER clauses.
+ """
+ result = await db.execute(
+ select(Connector).where(Connector.tenant_id == current_user.tenant_id)
+ )
+ connectors = list(result.scalars().all())
+
+ now = datetime.now(UTC)
+ twenty_four_hours_ago_seconds = 24 * 60 * 60
+
+ total = len(connectors)
+ healthy = 0
+ unhealthy = 0
+ unknown = 0
+ drifted = 0
+ drifted_recently = 0
+ last_drift_at: datetime | None = None
+ total_events_ingested = 0
+ total_events_dropped = 0
+
+ for c in connectors:
+ if c.health_status == "healthy":
+ healthy += 1
+ elif c.health_status == "unhealthy":
+ unhealthy += 1
+ else:
+ unknown += 1
+
+ total_events_ingested += int(c.events_ingested or 0)
+ total_events_dropped += int(getattr(c, "events_dropped", 0) or 0)
+
+ drift_at = getattr(c, "last_schema_drift_at", None)
+ if drift_at is not None:
+ drifted += 1
+ if last_drift_at is None or drift_at > last_drift_at:
+ last_drift_at = drift_at
+ # Compare in seconds so we don't depend on timedelta arithmetic
+ # being available on every datetime variant Postgres can return.
+ if (now - drift_at).total_seconds() <= twenty_four_hours_ago_seconds:
+ drifted_recently += 1
+
+ return ConnectorHealthSummary(
+ total=total,
+ healthy=healthy,
+ unhealthy=unhealthy,
+ unknown=unknown,
+ drifted=drifted,
+ drifted_recently=drifted_recently,
+ last_drift_at=last_drift_at,
+ total_events_ingested=total_events_ingested,
+ total_events_dropped=total_events_dropped,
+ )
+
+
# ------------------------------------------------------------------ pre-save test
diff --git a/services/api/app/models/connector.py b/services/api/app/models/connector.py
index 6f6a655a7..7aa38e6fb 100644
--- a/services/api/app/models/connector.py
+++ b/services/api/app/models/connector.py
@@ -3,7 +3,7 @@
import uuid
from datetime import UTC, datetime
-from sqlalchemy import Boolean, DateTime, ForeignKey, String
+from sqlalchemy import Boolean, DateTime, ForeignKey, String, Text
from sqlalchemy.dialects.postgresql import JSONB, UUID
from sqlalchemy.orm import Mapped, mapped_column, relationship
@@ -25,7 +25,13 @@ class Connector(Base):
last_health_check: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
last_sync: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
events_ingested: Mapped[int] = mapped_column(default=0)
+ events_dropped: Mapped[int] = mapped_column(default=0)
error_count: Mapped[int] = mapped_column(default=0)
+ # Schema-drift sentinel state (migration 026). NULL until the first
+ # non-empty poll has populated a fingerprint baseline.
+ schema_fingerprint: Mapped[str | None] = mapped_column(Text, nullable=True)
+ last_schema_drift_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+ last_drift_details: Mapped[dict | None] = mapped_column(JSONB, nullable=True)
tags: Mapped[list] = mapped_column(JSONB, default=list)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(UTC))
updated_at: Mapped[datetime] = mapped_column(
diff --git a/services/api/migrations/026_connector_schema_drift.sql b/services/api/migrations/026_connector_schema_drift.sql
new file mode 100644
index 000000000..bf1386453
--- /dev/null
+++ b/services/api/migrations/026_connector_schema_drift.sql
@@ -0,0 +1,47 @@
+-- 026_connector_schema_drift.sql
+-- Connector Health & Schema-Drift Sentinel
+--
+-- Adds three columns to the `connectors` table so the polling scheduler can
+-- record schema fingerprints from each connector's most recent payload and
+-- raise an alert (drift) when the field set changes between polls.
+--
+-- schema_fingerprint : SHA-256 hex of the sorted unique top-level
+-- field names observed in the most recent batch.
+-- NULL until the connector has produced its
+-- first non-empty batch.
+--
+-- last_schema_drift_at : Timestamp of the most recent confirmed drift
+-- (fingerprint changed vs prior baseline).
+--
+-- last_drift_details : JSONB describing what changed — added fields,
+-- removed fields, sample event count, etc.
+-- Surfaced in the UI so an operator can decide
+-- whether the upstream API legitimately changed
+-- or whether a parser is breaking.
+--
+-- events_dropped : Counter incremented when pre-ingest filter
+-- rules (Security Data Pipeline feature) drop
+-- an event before it reaches the ingest service.
+--
+-- This migration is additive and idempotent; running it twice is safe.
+
+BEGIN;
+
+ALTER TABLE connectors
+ ADD COLUMN IF NOT EXISTS schema_fingerprint TEXT,
+ ADD COLUMN IF NOT EXISTS last_schema_drift_at TIMESTAMPTZ,
+ ADD COLUMN IF NOT EXISTS last_drift_details JSONB,
+ ADD COLUMN IF NOT EXISTS events_dropped INTEGER NOT NULL DEFAULT 0;
+
+-- Index on health_status so the new health-summary endpoint can group
+-- counts in a single scan instead of a tenant-wide table scan.
+CREATE INDEX IF NOT EXISTS idx_connectors_health_status
+ ON connectors(tenant_id, health_status);
+
+-- Index on last_schema_drift_at to surface "drifted in the last 24 h"
+-- rows quickly on the dashboard.
+CREATE INDEX IF NOT EXISTS idx_connectors_last_drift
+ ON connectors(tenant_id, last_schema_drift_at)
+ WHERE last_schema_drift_at IS NOT NULL;
+
+COMMIT;
diff --git a/services/connectors/app/db/connector_repo.py b/services/connectors/app/db/connector_repo.py
index 86ac5cb2c..fe16b7378 100644
--- a/services/connectors/app/db/connector_repo.py
+++ b/services/connectors/app/db/connector_repo.py
@@ -29,6 +29,7 @@ from sqlalchemy import (
MetaData,
String,
Table,
+ Text,
select,
update,
)
@@ -54,7 +55,13 @@ connectors_table = Table(
Column("last_health_check", DateTime(timezone=True)),
Column("last_sync", DateTime(timezone=True)),
Column("events_ingested", Integer, nullable=False),
+ Column("events_dropped", Integer, nullable=False),
Column("error_count", Integer, nullable=False),
+ # Schema-drift sentinel columns (migration 026). Nullable: a brand-new
+ # connector hasn't established a fingerprint baseline yet.
+ Column("schema_fingerprint", Text),
+ Column("last_schema_drift_at", DateTime(timezone=True)),
+ Column("last_drift_details", JSONB),
Column("tags", JSON, nullable=False),
Column("created_at", DateTime(timezone=True), nullable=False),
Column("updated_at", DateTime(timezone=True), nullable=False),
@@ -80,7 +87,14 @@ class ConnectorInstance:
health_status: str
last_sync: datetime | None
events_ingested: int
+ events_dropped: int
error_count: int
+ # Drift-sentinel state. ``schema_fingerprint`` is the SHA-256 of the
+ # sorted, deduped top-level field names from the previous successful
+ # poll. NULL on first poll.
+ schema_fingerprint: str | None
+ last_schema_drift_at: datetime | None
+ last_drift_details: dict[str, Any] | None
async def fetch_enabled_connectors(connection: Any) -> list[ConnectorInstance]:
@@ -101,7 +115,11 @@ async def fetch_enabled_connectors(connection: Any) -> list[ConnectorInstance]:
connectors_table.c.health_status,
connectors_table.c.last_sync,
connectors_table.c.events_ingested,
+ connectors_table.c.events_dropped,
connectors_table.c.error_count,
+ connectors_table.c.schema_fingerprint,
+ connectors_table.c.last_schema_drift_at,
+ connectors_table.c.last_drift_details,
).where(connectors_table.c.is_enabled.is_(True))
result = await connection.execute(stmt)
@@ -118,7 +136,11 @@ async def fetch_enabled_connectors(connection: Any) -> list[ConnectorInstance]:
health_status=row.health_status,
last_sync=row.last_sync,
events_ingested=row.events_ingested,
+ events_dropped=row.events_dropped,
error_count=row.error_count,
+ schema_fingerprint=row.schema_fingerprint,
+ last_schema_drift_at=row.last_schema_drift_at,
+ last_drift_details=row.last_drift_details,
)
for row in rows
]
@@ -129,19 +151,31 @@ async def record_poll_success(
connector_id: uuid.UUID,
*,
events_added: int,
+ events_dropped: int = 0,
+ schema_fingerprint: str | None = None,
) -> None:
- """Update last_sync, increment events_ingested, mark healthy."""
+ """Update last_sync, increment counters, mark healthy.
+
+ If ``schema_fingerprint`` is provided it is written to the row; the
+ drift bookkeeping (``last_schema_drift_at`` / ``last_drift_details``)
+ is updated by ``record_schema_drift`` so the two callers don't race
+ on the same UPDATE.
+ """
now = datetime.now(UTC)
+ values: dict[str, Any] = {
+ "last_sync": now,
+ "last_health_check": now,
+ "health_status": "healthy",
+ "events_ingested": connectors_table.c.events_ingested + events_added,
+ "events_dropped": connectors_table.c.events_dropped + events_dropped,
+ "updated_at": now,
+ }
+ if schema_fingerprint is not None:
+ values["schema_fingerprint"] = schema_fingerprint
stmt = (
update(connectors_table)
.where(connectors_table.c.id == connector_id)
- .values(
- last_sync=now,
- last_health_check=now,
- health_status="healthy",
- events_ingested=connectors_table.c.events_ingested + events_added,
- updated_at=now,
- )
+ .values(**values)
)
await connection.execute(stmt)
@@ -165,6 +199,33 @@ async def record_poll_failure(
await connection.execute(stmt)
+async def record_schema_drift(
+ connection: Any,
+ connector_id: uuid.UUID,
+ *,
+ fingerprint: str,
+ details: dict[str, Any],
+) -> None:
+ """Record a confirmed schema drift event.
+
+ Called by the scheduler when the new poll's fingerprint differs from
+ the stored baseline. Writes the new fingerprint, drift timestamp and
+ details so the dashboard can surface "what changed" to operators.
+ """
+ now = datetime.now(UTC)
+ stmt = (
+ update(connectors_table)
+ .where(connectors_table.c.id == connector_id)
+ .values(
+ schema_fingerprint=fingerprint,
+ last_schema_drift_at=now,
+ last_drift_details=details,
+ updated_at=now,
+ )
+ )
+ await connection.execute(stmt)
+
+
__all__ = [
"ConnectorInstance",
"connectors_table",
@@ -172,4 +233,5 @@ __all__ = [
"metadata",
"record_poll_failure",
"record_poll_success",
+ "record_schema_drift",
]
diff --git a/services/connectors/app/pipeline/__init__.py b/services/connectors/app/pipeline/__init__.py
new file mode 100644
index 000000000..c29593fa4
--- /dev/null
+++ b/services/connectors/app/pipeline/__init__.py
@@ -0,0 +1,28 @@
+"""Pre-ingest pipeline: schema fingerprinting and event filtering.
+
+This package owns the *connector-side* half of two related features:
+
+* **Connector Health & Schema-Drift Sentinel** — fingerprints each batch
+ of normalized events so the scheduler can detect when an upstream API
+ starts returning a different field set (silent integration breakage).
+
+* **Security Data Pipeline + Tiered Retention** — applies a list of
+ declarative filter rules (drop / route) before events leave the
+ connectors microservice, so noisy events never reach the hot tier.
+
+Both helpers are pure functions that take a list of events and return
+something useful. They never touch the database, the network, or the
+scheduler state — that's the scheduler's job.
+"""
+
+from __future__ import annotations
+
+from app.pipeline.fingerprint import compute_fingerprint, diff_fingerprints
+from app.pipeline.filter_rules import FilterDecision, apply_filter_rules
+
+__all__ = [
+ "FilterDecision",
+ "apply_filter_rules",
+ "compute_fingerprint",
+ "diff_fingerprints",
+]
diff --git a/services/connectors/app/pipeline/filter_rules.py b/services/connectors/app/pipeline/filter_rules.py
new file mode 100644
index 000000000..7b6ea9048
--- /dev/null
+++ b/services/connectors/app/pipeline/filter_rules.py
@@ -0,0 +1,151 @@
+"""Pre-ingest filter rules — Security Data Pipeline (MVP).
+
+The data-pipeline feature is a "drop or keep before it costs you anything"
+layer: every connector instance can carry a small list of declarative rules
+in its ``connector_config["filter_rules"]`` array. The scheduler runs these
+rules over each normalized batch *before* shipping events to the ingest
+service, so a noisy informational alert can be discarded at the connector
+edge instead of consuming hot-tier storage and triage capacity.
+
+Rule shape (kept intentionally tiny so it's safe to author by hand):
+
+.. code-block:: yaml
+
+ filter_rules:
+ - field: severity
+ op: eq
+ value: info
+ action: drop
+ - field: rule_name
+ op: contains
+ value: "test alert"
+ action: drop
+
+The same operator vocabulary as ``federated.query`` is reused so SOC
+operators don't have to learn two filter dialects.
+
+Supported actions:
+
+* ``drop`` — discard the event entirely. It never reaches ingest.
+* ``keep`` — explicitly retain (used to override a later catch-all drop).
+
+Anything that doesn't match any rule is kept by default. Rules are
+evaluated top-to-bottom; first match wins. This is the same semantics
+``iptables`` and most firewall vendors use, so it's familiar and
+debuggable.
+
+The function is pure — it does not log, ingest, or mutate the events.
+The scheduler is responsible for incrementing ``events_dropped`` and for
+writing a structured log line per drop so an operator can audit which
+events were filtered.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any
+
+# Operator vocabulary mirrors federated/query.py to avoid teaching the
+# user two slightly different dialects of "this field equals that value".
+_VALID_OPS: frozenset[str] = frozenset(
+ {"eq", "ne", "contains", "starts_with", "ends_with", "in"}
+)
+_VALID_ACTIONS: frozenset[str] = frozenset({"drop", "keep"})
+
+
+@dataclass(slots=True)
+class FilterDecision:
+ """Result of running rules over a single event.
+
+ ``action`` is always either ``"drop"`` or ``"keep"`` (never None) so
+ the caller can branch unambiguously. ``rule_index`` is the 0-based
+ index of the rule that matched, or ``None`` if the default
+ keep-everything fallback fired. ``rule_index`` is logged so an
+ operator can audit which rule discarded which event.
+ """
+
+ action: str
+ rule_index: int | None
+
+
+def _matches(event: dict[str, Any], field: str, op: str, value: Any) -> bool:
+ """Evaluate a single ``field <op> value`` predicate against an event.
+
+ Missing fields are treated as a non-match (return False) for every
+ operator. We don't raise: a poorly authored rule referencing a field
+ that an upstream vendor renamed shouldn't blow up the polling loop.
+ """
+ actual = event.get(field)
+ if actual is None:
+ return False
+
+ # Cheapest comparisons first.
+ if op == "eq":
+ return actual == value
+ if op == "ne":
+ return actual != value
+ if op == "in":
+ # Caller is expected to pass a list/tuple/set for `value`. If they
+ # passed a scalar, fall back to `==` so we don't crash.
+ if isinstance(value, list | tuple | set):
+ return actual in value
+ return actual == value
+
+ # String operators — coerce to str so "severity == 5" with a string
+ # rule still does the obvious thing.
+ actual_str = str(actual)
+ value_str = str(value)
+ if op == "contains":
+ return value_str.lower() in actual_str.lower()
+ if op == "starts_with":
+ return actual_str.lower().startswith(value_str.lower())
+ if op == "ends_with":
+ return actual_str.lower().endswith(value_str.lower())
+
+ # Unknown operator: refuse to drop. Better to over-keep than to silently
+ # discard events because someone fat-fingered the operator.
+ return False
+
+
+def apply_filter_rules(
+ event: dict[str, Any],
+ rules: list[dict[str, Any]] | None,
+) -> FilterDecision:
+ """Evaluate the rule list against one event and return the decision.
+
+ Iterates rules top-to-bottom; the first rule whose predicate matches
+ decides the action (drop or keep). If no rule matches, the default
+ is to keep the event — the scheduler must not silently discard
+ everything just because a tenant left their rule list empty.
+
+ Malformed rules (missing keys, bad operator, bad action) are skipped
+ silently. We log noisily about this elsewhere (the scheduler emits a
+ structured warning the first time a malformed rule is seen) so the
+ user gets visibility without each evaluation costing log spam.
+ """
+ if not rules:
+ return FilterDecision(action="keep", rule_index=None)
+
+ for index, rule in enumerate(rules):
+ if not isinstance(rule, dict):
+ continue
+ field = rule.get("field")
+ op = rule.get("op")
+ value = rule.get("value")
+ action = rule.get("action", "drop")
+
+ if not isinstance(field, str) or not field:
+ continue
+ if op not in _VALID_OPS:
+ continue
+ if action not in _VALID_ACTIONS:
+ continue
+
+ if _matches(event, field, op, value):
+ return FilterDecision(action=action, rule_index=index)
+
+ # No rule matched: default keep.
+ return FilterDecision(action="keep", rule_index=None)
+
+
+__all__ = ["FilterDecision", "apply_filter_rules"]
diff --git a/services/connectors/app/pipeline/fingerprint.py b/services/connectors/app/pipeline/fingerprint.py
new file mode 100644
index 000000000..205210d6e
--- /dev/null
+++ b/services/connectors/app/pipeline/fingerprint.py
@@ -0,0 +1,98 @@
+"""Schema fingerprinting for the Connector Health & Schema-Drift Sentinel.
+
+We compute a stable hash of the *set of top-level field names* that appear
+across a batch of normalized events. The hash is deliberately simple so that
+two polls of the same upstream API produce the same fingerprint as long as
+the field set is unchanged — even if values change, even if events arrive in
+a different order, even if some events omit optional fields.
+
+The intent is not to detect every possible upstream change (a value-type
+change from string→int wouldn't be caught here, for example). The intent is
+to catch the **single most common silent failure mode**: an upstream vendor
+renames or removes a field, the connector's normalize() logic now produces
+events missing critical attributes, and detections silently start firing
+less frequently. Even this coarse fingerprint catches that case quickly.
+
+Algorithm
+---------
+
+For a batch of N normalized events:
+
+1. Union all top-level keys across all N events into a set.
+2. Sort lexicographically (stable across runs, so the hash is stable).
+3. Join with ``\\x1f`` (ASCII unit separator — guaranteed not to appear in
+ a JSON field name).
+4. SHA-256 the joined string and return the hex digest.
+
+Two batches with identical field sets produce identical fingerprints.
+A batch with one extra field (``new_field``) or one missing field produces
+a different fingerprint, which the scheduler treats as drift.
+
+Empty batches return ``None`` rather than a fingerprint of the empty set,
+because we don't want a quiet hour (no events) to clobber the baseline
+fingerprint we recorded the previous time the connector returned events.
+"""
+
+from __future__ import annotations
+
+import hashlib
+from collections.abc import Iterable
+from typing import Any
+
+# We do *not* recurse into nested dicts. The drift sentinel is intentionally
+# scoped to top-level fields because (a) normalize() outputs are flat by
+# convention in this project (``severity``, ``timestamp``, ``user``, etc.)
+# and (b) hashing arbitrarily nested structures would catch too many false
+# positives — every alert has different ``raw_event`` contents, but those
+# differences don't represent schema drift in any meaningful sense.
+
+_SEPARATOR = "\x1f"
+
+
+def compute_fingerprint(events: Iterable[dict[str, Any]]) -> str | None:
+ """Return a stable SHA-256 hex digest of the union of top-level field names.
+
+ Returns ``None`` if no events have any keys (e.g. an empty list, or a
+ list of empty dicts). Returning ``None`` lets the caller distinguish
+ "no events this poll" from "the upstream changed schema to the empty
+ set", which is essentially never what we want to flag as drift.
+ """
+ keys: set[str] = set()
+ for event in events:
+ if not isinstance(event, dict):
+ # Defensive: we only fingerprint structured events. A connector
+ # that misbehaves and yields a plain string is the scheduler's
+ # problem, not the fingerprinter's.
+ continue
+ keys.update(event.keys())
+
+ if not keys:
+ return None
+
+ joined = _SEPARATOR.join(sorted(keys))
+ digest = hashlib.sha256(joined.encode("utf-8")).hexdigest()
+ return digest
+
+
+def diff_fingerprints(
+ previous_keys: Iterable[str],
+ current_keys: Iterable[str],
+) -> dict[str, list[str]]:
+ """Describe what changed between two field sets.
+
+ Used by the scheduler to populate ``last_drift_details`` so the UI can
+ show "added: X; removed: Y" instead of just "fingerprint changed".
+ Both arguments are the raw key sets — the caller is responsible for
+ pulling them from the most recent batch and the prior baseline batch
+ (we cache the prior batch's keys in-memory in the scheduler).
+ """
+ prev = set(previous_keys)
+ curr = set(current_keys)
+ return {
+ "added": sorted(curr - prev),
+ "removed": sorted(prev - curr),
+ "unchanged_count": len(prev & curr),
+ }
+
+
+__all__ = ["compute_fingerprint", "diff_fingerprints"]
diff --git a/services/connectors/app/scheduler.py b/services/connectors/app/scheduler.py
index 1e65b1b71..94a310abd 100644
--- a/services/connectors/app/scheduler.py
+++ b/services/connectors/app/scheduler.py
@@ -64,9 +64,15 @@ from app.db.connector_repo import (
fetch_enabled_connectors,
record_poll_failure,
record_poll_success,
+ record_schema_drift,
)
from app.db.engine import get_engine
from app.ingest_client import IngestClient, IngestClientError
+from app.pipeline import (
+ apply_filter_rules,
+ compute_fingerprint,
+ diff_fingerprints,
+)
from app.security.credential_vault import CredentialVault, CredentialVaultError, get_vault
logger = logging.getLogger("aisoc.connectors.scheduler")
@@ -385,48 +391,174 @@ class ConnectorScheduler:
# raw events still produces consistent shapes downstream.
normalized = [_safe_normalize(connector, e) for e in raw_events]
+ # ---- Security Data Pipeline: pre-ingest filter rules ----
+ #
+ # Rules live on the connector's ``connector_config["filter_rules"]``
+ # array. ``apply_filter_rules`` is pure: it just decides drop/keep.
+ # We count drops here so they show up in ``events_dropped`` on the
+ # row and emit one info-level log per drop so an operator can audit
+ # filter behavior without trawling the database.
+ filter_rules = (target.connector_config or {}).get("filter_rules") or []
+ kept: list[dict[str, Any]] = []
+ dropped_count = 0
+ for event in normalized:
+ decision = apply_filter_rules(event, filter_rules)
+ if decision.action == "drop":
+ dropped_count += 1
+ # Keep the log line short — high-volume connectors can drop
+ # hundreds of events per poll, and we don't want the log to
+ # become the bottleneck. The rule_index is enough to find
+ # the offending rule in the connector config.
+ logger.debug(
+ "connector.scheduler.filter_drop id=%s rule_index=%s",
+ connector_id,
+ decision.rule_index,
+ )
+ continue
+ kept.append(event)
+
+ # ---- Schema-Drift Sentinel: fingerprint the kept batch ----
+ #
+ # We fingerprint *kept* events (post-filter) because the filter
+ # rules are tenant-controlled and shouldn't be allowed to mask a
+ # legitimate upstream schema change by dropping all instances of
+ # the new field. Empty batches return None and we don't update
+ # the baseline in that case (see fingerprint.compute_fingerprint).
+ new_fingerprint = compute_fingerprint(kept)
+ drift_detected = False
+ drift_details: dict[str, Any] | None = None
+
+ if new_fingerprint is not None and target.schema_fingerprint is not None:
+ if new_fingerprint != target.schema_fingerprint:
+ drift_detected = True
+ # Recompute the union of keys so we can describe what
+ # changed. We don't store prior keys in the row (only the
+ # hash), so we approximate "removed" by diffing against
+ # the current batch's keys vs. the keys set we'd expect
+ # from the prior fingerprint. Since we don't have the
+ # prior keys, we record the new key set and ``added`` is
+ # the entire new set. This is good enough for the UI to
+ # surface "schema changed at <timestamp>"; a future
+ # iteration can persist the prior key list.
+ current_keys = sorted({k for ev in kept for k in ev.keys()})
+ drift_details = diff_fingerprints(
+ previous_keys=[], # baseline keys not persisted yet
+ current_keys=current_keys,
+ )
+ drift_details["sample_event_count"] = len(kept)
+ drift_details["new_fingerprint"] = new_fingerprint
+ drift_details["previous_fingerprint"] = target.schema_fingerprint
+ logger.warning(
+ "connector.scheduler.schema_drift id=%s type=%s added=%d",
+ connector_id,
+ target.connector_type,
+ len(drift_details.get("added", [])),
+ )
+
try:
result = await self._ingest_client.push_events(
tenant_id=target.tenant_id,
connector_id=target.id,
connector_type=target.connector_type,
- events=normalized,
+ events=kept,
)
except IngestClientError:
logger.exception(
"connector.scheduler.ingest_failed id=%s events=%d",
connector_id,
- len(normalized),
+ len(kept),
)
await self._record_failure(target.id)
return
accepted = int(result.get("accepted", 0) or 0)
elapsed_ms = int((datetime.now(UTC) - started).total_seconds() * 1000)
- await self._record_success(target.id, events_added=accepted)
+
+ # If drift was detected, persist that fact *before* marking the
+ # poll successful so the UI reflects "drifted" status alongside
+ # "healthy". Both writes are independent transactions; the order
+ # protects against a race where the operator views the row mid-
+ # poll and sees a stale fingerprint.
+ if drift_detected and drift_details is not None:
+ await self._record_drift(
+ target.id,
+ fingerprint=new_fingerprint or "",
+ details=drift_details,
+ )
+
+ await self._record_success(
+ target.id,
+ events_added=accepted,
+ events_dropped=dropped_count,
+ schema_fingerprint=new_fingerprint,
+ )
logger.info(
- "connector.scheduler.poll_complete id=%s type=%s accepted=%d rejected=%d elapsed_ms=%d",
+ "connector.scheduler.poll_complete id=%s type=%s accepted=%d rejected=%d dropped=%d drift=%s elapsed_ms=%d",
connector_id,
target.connector_type,
accepted,
int(result.get("rejected", 0) or 0),
+ dropped_count,
+ "yes" if drift_detected else "no",
elapsed_ms,
)
# ------------------------------------------------------------------ db helpers
- async def _record_success(self, connector_id: uuid.UUID, *, events_added: int) -> None:
+ async def _record_success(
+ self,
+ connector_id: uuid.UUID,
+ *,
+ events_added: int,
+ events_dropped: int = 0,
+ schema_fingerprint: str | None = None,
+ ) -> None:
if self._engine is None: # pragma: no cover
return
try:
async with self._engine.begin() as conn:
- await record_poll_success(conn, connector_id, events_added=events_added)
+ await record_poll_success(
+ conn,
+ connector_id,
+ events_added=events_added,
+ events_dropped=events_dropped,
+ schema_fingerprint=schema_fingerprint,
+ )
except Exception: # pragma: no cover
logger.exception(
"connector.scheduler.record_success_failed id=%s",
connector_id,
)
+ async def _record_drift(
+ self,
+ connector_id: uuid.UUID,
+ *,
+ fingerprint: str,
+ details: dict[str, Any],
+ ) -> None:
+ """Persist a detected schema drift for the connector row.
+
+ Kept separate from ``_record_success`` so a drift write that
+ races with a future poll's success write doesn't clobber the
+ ``last_schema_drift_at`` timestamp.
+ """
+ if self._engine is None: # pragma: no cover
+ return
+ try:
+ async with self._engine.begin() as conn:
+ await record_schema_drift(
+ conn,
+ connector_id,
+ fingerprint=fingerprint,
+ details=details,
+ )
+ except Exception: # pragma: no cover
+ logger.exception(
+ "connector.scheduler.record_drift_failed id=%s",
+ connector_id,
+ )
+
async def _record_failure(self, connector_id: uuid.UUID) -> None:
if self._engine is None: # pragma: no cover
return
diff --git a/services/connectors/tests/test_pipeline_filter_rules.py b/services/connectors/tests/test_pipeline_filter_rules.py
new file mode 100644
index 000000000..2f3687951
--- /dev/null
+++ b/services/connectors/tests/test_pipeline_filter_rules.py
@@ -0,0 +1,130 @@
+"""Tests for declarative pre-ingest filter rules.
+
+These rules run on every normalized event before it leaves the connector
+service. Getting the semantics right is critical: a buggy rule engine could
+silently drop alerts that should have been triaged. The tests here pin down:
+
+* default behaviour (no rules → keep)
+* each operator (eq, ne, contains, starts_with, ends_with, in)
+* drop vs. keep actions
+* malformed rules don't crash and don't drop events
+* missing fields don't match (so a renamed upstream field doesn't
+ accidentally match every rule and discard everything)
+"""
+
+from __future__ import annotations
+
+from app.pipeline.filter_rules import FilterDecision, apply_filter_rules
+
+
+def test_no_rules_keeps_event() -> None:
+ decision = apply_filter_rules({"severity": "info"}, None)
+ assert decision == FilterDecision(action="keep", rule_index=None)
+
+
+def test_empty_rules_keeps_event() -> None:
+ decision = apply_filter_rules({"severity": "info"}, [])
+ assert decision == FilterDecision(action="keep", rule_index=None)
+
+
+def test_eq_drops_matching_event() -> None:
+ rules = [{"field": "severity", "op": "eq", "value": "info", "action": "drop"}]
+ decision = apply_filter_rules({"severity": "info"}, rules)
+ assert decision.action == "drop"
+ assert decision.rule_index == 0
+
+
+def test_eq_keeps_non_matching_event() -> None:
+ rules = [{"field": "severity", "op": "eq", "value": "info", "action": "drop"}]
+ decision = apply_filter_rules({"severity": "high"}, rules)
+ assert decision.action == "keep"
+ assert decision.rule_index is None
+
+
+def test_contains_is_case_insensitive() -> None:
+ rules = [{"field": "rule_name", "op": "contains", "value": "TEST", "action": "drop"}]
+ decision = apply_filter_rules({"rule_name": "lab test alert"}, rules)
+ assert decision.action == "drop"
+
+
+def test_starts_with_and_ends_with() -> None:
+ starts = [{"field": "host", "op": "starts_with", "value": "lab-", "action": "drop"}]
+ ends = [{"field": "host", "op": "ends_with", "value": "-stage", "action": "drop"}]
+
+ assert apply_filter_rules({"host": "lab-web-01"}, starts).action == "drop"
+ assert apply_filter_rules({"host": "prod-web-01"}, starts).action == "keep"
+ assert apply_filter_rules({"host": "web-01-stage"}, ends).action == "drop"
+
+
+def test_in_operator_with_list() -> None:
+ rules = [{"field": "severity", "op": "in", "value": ["info", "low"], "action": "drop"}]
+ assert apply_filter_rules({"severity": "info"}, rules).action == "drop"
+ assert apply_filter_rules({"severity": "low"}, rules).action == "drop"
+ assert apply_filter_rules({"severity": "high"}, rules).action == "keep"
+
+
+def test_in_operator_with_scalar_falls_back_to_eq() -> None:
+ rules = [{"field": "severity", "op": "in", "value": "info", "action": "drop"}]
+ assert apply_filter_rules({"severity": "info"}, rules).action == "drop"
+ assert apply_filter_rules({"severity": "low"}, rules).action == "keep"
+
+
+def test_ne_drops_when_field_differs() -> None:
+ rules = [{"field": "tenant", "op": "ne", "value": "acme", "action": "drop"}]
+ assert apply_filter_rules({"tenant": "evil-corp"}, rules).action == "drop"
+ assert apply_filter_rules({"tenant": "acme"}, rules).action == "keep"
+
+
+def test_first_match_wins() -> None:
+ rules = [
+ {"field": "severity", "op": "eq", "value": "info", "action": "keep"},
+ {"field": "severity", "op": "eq", "value": "info", "action": "drop"},
+ ]
+ decision = apply_filter_rules({"severity": "info"}, rules)
+ assert decision.action == "keep"
+ assert decision.rule_index == 0
+
+
+def test_keep_action_overrides_later_drop() -> None:
+ rules = [
+ {"field": "rule_name", "op": "contains", "value": "important", "action": "keep"},
+ {"field": "severity", "op": "eq", "value": "info", "action": "drop"},
+ ]
+ decision = apply_filter_rules(
+ {"rule_name": "important info alert", "severity": "info"},
+ rules,
+ )
+ assert decision.action == "keep"
+ assert decision.rule_index == 0
+
+
+def test_missing_field_does_not_match() -> None:
+ # If a rule references a field the event doesn't have, the rule should
+ # be ignored rather than matching (which would silently drop everything
+ # when an upstream vendor renames a field).
+ rules = [{"field": "definitely_missing", "op": "eq", "value": "x", "action": "drop"}]
+ decision = apply_filter_rules({"severity": "high"}, rules)
+ assert decision.action == "keep"
+
+
+def test_malformed_rule_skipped() -> None:
+ rules = [
+ "not a dict", # type: ignore[list-item]
+ {"field": ""},
+ {"field": "severity", "op": "bogus_op", "value": "info", "action": "drop"},
+ {"field": "severity", "op": "eq", "value": "info", "action": "delete"},
+ {"field": "severity", "op": "eq", "value": "info", "action": "drop"},
+ ]
+ decision = apply_filter_rules({"severity": "info"}, rules)
+ # Only the last rule is well-formed; that's the one that should fire.
+ assert decision.action == "drop"
+ assert decision.rule_index == 4
+
+
+def test_string_op_coerces_non_string_actuals() -> None:
+ # Some upstreams report severity as an int; rule authors usually write
+ # the value as a string. Coercing both sides to str avoids the
+ # surprise of "5".contains("5") not matching int 5.
+ rules = [{"field": "severity", "op": "contains", "value": "5", "action": "drop"}]
+ decision = apply_filter_rules({"severity": 5}, rules)
+ assert decision.action == "drop"
diff --git a/services/connectors/tests/test_pipeline_fingerprint.py b/services/connectors/tests/test_pipeline_fingerprint.py
new file mode 100644
index 000000000..17ae8e36b
--- /dev/null
+++ b/services/connectors/tests/test_pipeline_fingerprint.py
@@ -0,0 +1,79 @@
+"""Tests for the schema-fingerprint module.
+
+The fingerprint is the foundation of the Schema-Drift Sentinel: it has to be
+**stable** across reorderings and value changes, and **distinct** when the set
+of top-level keys actually changes. These tests lock those properties down so
+the scheduler can safely use the digest as a primary-key for "schema we last
+saw on this connector".
+"""
+
+from __future__ import annotations
+
+from app.pipeline.fingerprint import compute_fingerprint, diff_fingerprints
+
+
+def test_fingerprint_stable_across_event_order() -> None:
+ a = [{"severity": "high", "host": "h1"}, {"severity": "low", "host": "h2"}]
+ b = list(reversed(a))
+ assert compute_fingerprint(a) == compute_fingerprint(b)
+
+
+def test_fingerprint_stable_across_value_changes() -> None:
+ a = [{"severity": "high", "host": "h1"}]
+ b = [{"severity": "low", "host": "h2"}]
+ # Same key set, different values — fingerprint must be identical.
+ assert compute_fingerprint(a) == compute_fingerprint(b)
+
+
+def test_fingerprint_changes_when_field_added() -> None:
+ baseline = [{"severity": "high"}]
+ drifted = [{"severity": "high", "new_field": "x"}]
+ assert compute_fingerprint(baseline) != compute_fingerprint(drifted)
+
+
+def test_fingerprint_changes_when_field_removed() -> None:
+ baseline = [{"severity": "high", "user": "u1"}]
+ drifted = [{"severity": "high"}]
+ assert compute_fingerprint(baseline) != compute_fingerprint(drifted)
+
+
+def test_fingerprint_unions_keys_across_events() -> None:
+ # The union of keys is what we hash, so an event missing 'user' shouldn't
+ # change the fingerprint as long as some other event has it.
+ a = [{"severity": "high", "user": "u1"}, {"severity": "low"}]
+ b = [{"severity": "high"}, {"severity": "low", "user": "u1"}]
+ assert compute_fingerprint(a) == compute_fingerprint(b)
+
+
+def test_fingerprint_empty_returns_none() -> None:
+ assert compute_fingerprint([]) is None
+ assert compute_fingerprint([{}]) is None
+
+
+def test_fingerprint_skips_non_dict_events() -> None:
+ # A misbehaving connector returning a plain string shouldn't crash;
+ # we just skip the offender and fingerprint the rest.
+ a = [{"severity": "high"}, "garbage"] # type: ignore[list-item]
+ b = [{"severity": "high"}]
+ assert compute_fingerprint(a) == compute_fingerprint(b)
+
+
+def test_diff_added_and_removed() -> None:
+ diff = diff_fingerprints({"severity", "host"}, {"severity", "user"})
+ assert diff["added"] == ["user"]
+ assert diff["removed"] == ["host"]
+ assert diff["unchanged_count"] == 1
+
+
+def test_diff_unchanged() -> None:
+ diff = diff_fingerprints({"severity", "host"}, {"severity", "host"})
+ assert diff["added"] == []
+ assert diff["removed"] == []
+ assert diff["unchanged_count"] == 2
+
+
+def test_diff_handles_empty_previous() -> None:
+ diff = diff_fingerprints(set(), {"severity", "host"})
+ assert sorted(diff["added"]) == ["host", "severity"]
+ assert diff["removed"] == []
+ assert diff["unchanged_count"] == 0
diff --git a/services/connectors/tests/test_scheduler.py b/services/connectors/tests/test_scheduler.py
index 31c1c4b4b..cf24e4ce5 100644
--- a/services/connectors/tests/test_scheduler.py
+++ b/services/connectors/tests/test_scheduler.py
@@ -43,6 +43,10 @@ def _make_instance(
last_sync=None,
events_ingested=0,
error_count=0,
+ events_dropped=0,
+ schema_fingerprint=None,
+ last_schema_drift_at=None,
+ last_drift_details=None,
)
@@ -113,6 +117,236 @@ async def test_poll_one_happy_path(monkeypatch):
assert fake_engine.failure_calls == []
+# ---------------------------------------------------------------------------
+# Schema-Drift Sentinel + filter-rules pipeline integration
+# ---------------------------------------------------------------------------
+
+
+async def test_poll_one_filter_rules_drop_events(monkeypatch):
+ """Pre-ingest filter rules should drop matching events and bump the
+ drop counter passed to record_poll_success without ever shipping the
+ dropped events to ingest."""
+ inst = _make_instance(
+ connector_type="fake_connector",
+ connector_config={
+ "poll_interval_seconds": 60,
+ "filter_rules": [
+ {"field": "severity", "op": "eq", "value": "info", "action": "drop"},
+ ],
+ },
+ )
+
+ fake_connector = MagicMock()
+ fake_connector.fetch_alerts = AsyncMock(
+ return_value=[
+ {"severity": "info", "id": "drop-me"},
+ {"severity": "high", "id": "keep-me"},
+ ]
+ )
+ fake_connector.normalize = MagicMock(side_effect=lambda e: e)
+
+ monkeypatch.setattr(
+ "app.scheduler.CONNECTOR_REGISTRY",
+ {"fake_connector": MagicMock(return_value=fake_connector)},
+ )
+
+ fake_engine = _FakeEngine([inst])
+ fake_ingest = _FakeIngestClient(accepted=1)
+
+ captured_success: dict[str, Any] = {}
+
+ async def capture_record_poll_success(
+ conn: Any,
+ connector_id: uuid.UUID,
+ *,
+ events_added: int,
+ events_dropped: int = 0,
+ schema_fingerprint: str | None = None,
+ ) -> None:
+ conn.engine.success_calls.append(connector_id)
+ captured_success["events_added"] = events_added
+ captured_success["events_dropped"] = events_dropped
+ captured_success["schema_fingerprint"] = schema_fingerprint
+
+ monkeypatch.setattr("app.scheduler.record_poll_success", capture_record_poll_success)
+
+ scheduler = ConnectorScheduler(engine=fake_engine, ingest_client=fake_ingest, vault=_FakeVault())
+ await scheduler._poll_one(connector_id=inst.id)
+
+ # Only the high-severity event should reach ingest.
+ pushed = fake_ingest.last_payload
+ assert pushed is not None
+ assert pushed["events"] == [{"severity": "high", "id": "keep-me"}]
+ assert captured_success["events_dropped"] == 1
+ assert captured_success["events_added"] == 1
+ assert captured_success["schema_fingerprint"] is not None
+
+
+async def test_poll_one_first_poll_records_fingerprint_no_drift(monkeypatch):
+ """First poll on a connector with no baseline should record the new
+ fingerprint via record_poll_success but NOT trigger record_schema_drift."""
+ inst = _make_instance(
+ connector_type="fake_connector",
+ connector_config={"poll_interval_seconds": 60},
+ )
+ # Baseline is None on the ConnectorInstance.
+
+ fake_connector = MagicMock()
+ fake_connector.fetch_alerts = AsyncMock(return_value=[{"severity": "high", "host": "h1"}])
+ fake_connector.normalize = MagicMock(side_effect=lambda e: e)
+
+ monkeypatch.setattr(
+ "app.scheduler.CONNECTOR_REGISTRY",
+ {"fake_connector": MagicMock(return_value=fake_connector)},
+ )
+
+ fake_engine = _FakeEngine([inst])
+ scheduler = ConnectorScheduler(
+ engine=fake_engine,
+ ingest_client=_FakeIngestClient(accepted=1),
+ vault=_FakeVault(),
+ )
+ await scheduler._poll_one(connector_id=inst.id)
+
+ assert fake_engine.success_calls == [inst.id]
+ # No drift expected on first poll.
+ assert fake_engine.drift_calls == []
+
+
+async def test_poll_one_detects_schema_drift(monkeypatch):
+ """When the new fingerprint differs from the stored baseline,
+ record_schema_drift should be called with details describing the
+ change before record_poll_success is invoked."""
+ # Synthesize a baseline fingerprint that differs from what the
+ # current poll will produce.
+ from app.pipeline.fingerprint import compute_fingerprint as cf
+
+ stale_fp = cf([{"severity": "high"}]) # baseline: only one field
+ inst = _make_instance(
+ connector_type="fake_connector",
+ connector_config={"poll_interval_seconds": 60},
+ )
+ inst.schema_fingerprint = stale_fp
+
+ fake_connector = MagicMock()
+ fake_connector.fetch_alerts = AsyncMock(
+ return_value=[{"severity": "high", "host": "h1", "new_field": "x"}]
+ )
+ fake_connector.normalize = MagicMock(side_effect=lambda e: e)
+
+ monkeypatch.setattr(
+ "app.scheduler.CONNECTOR_REGISTRY",
+ {"fake_connector": MagicMock(return_value=fake_connector)},
+ )
+
+ fake_engine = _FakeEngine([inst])
+ scheduler = ConnectorScheduler(
+ engine=fake_engine,
+ ingest_client=_FakeIngestClient(accepted=1),
+ vault=_FakeVault(),
+ )
+ await scheduler._poll_one(connector_id=inst.id)
+
+ assert fake_engine.success_calls == [inst.id]
+ assert len(fake_engine.drift_calls) == 1
+ drift = fake_engine.drift_calls[0]
+ assert drift["connector_id"] == inst.id
+ assert drift["fingerprint"] != stale_fp
+ assert "added" in drift["details"]
+ assert "previous_fingerprint" in drift["details"]
+ assert drift["details"]["previous_fingerprint"] == stale_fp
+
+
+async def test_poll_one_stable_schema_no_drift(monkeypatch):
+ """If the new fingerprint matches the stored baseline, the row's
+ fingerprint is rewritten via success but no drift event fires."""
+ from app.pipeline.fingerprint import compute_fingerprint as cf
+
+ events = [{"severity": "high", "host": "h1"}]
+ baseline_fp = cf(events)
+ inst = _make_instance(
+ connector_type="fake_connector",
+ connector_config={"poll_interval_seconds": 60},
+ )
+ inst.schema_fingerprint = baseline_fp
+
+ fake_connector = MagicMock()
+ fake_connector.fetch_alerts = AsyncMock(return_value=events)
+ fake_connector.normalize = MagicMock(side_effect=lambda e: e)
+
+ monkeypatch.setattr(
+ "app.scheduler.CONNECTOR_REGISTRY",
+ {"fake_connector": MagicMock(return_value=fake_connector)},
+ )
+
+ fake_engine = _FakeEngine([inst])
+ scheduler = ConnectorScheduler(
+ engine=fake_engine,
+ ingest_client=_FakeIngestClient(accepted=1),
+ vault=_FakeVault(),
+ )
+ await scheduler._poll_one(connector_id=inst.id)
+
+ assert fake_engine.success_calls == [inst.id]
+ assert fake_engine.drift_calls == []
+
+
+async def test_poll_one_quiet_hour_does_not_clobber_fingerprint(monkeypatch):
+ """If a poll returns zero events the fingerprint is None and we should
+ NOT overwrite the baseline in the row (ConnectorInstance fingerprint
+ stays as-is)."""
+ from app.pipeline.fingerprint import compute_fingerprint as cf
+
+ baseline_fp = cf([{"severity": "high", "host": "h1"}])
+ inst = _make_instance(
+ connector_type="fake_connector",
+ connector_config={"poll_interval_seconds": 60},
+ )
+ inst.schema_fingerprint = baseline_fp
+
+ fake_connector = MagicMock()
+ fake_connector.fetch_alerts = AsyncMock(return_value=[])
+ fake_connector.normalize = MagicMock(side_effect=lambda e: e)
+
+ monkeypatch.setattr(
+ "app.scheduler.CONNECTOR_REGISTRY",
+ {"fake_connector": MagicMock(return_value=fake_connector)},
+ )
+
+ captured_success: dict[str, Any] = {}
+
+ async def capture_record_poll_success(
+ conn: Any,
+ connector_id: uuid.UUID,
+ *,
+ events_added: int,
+ events_dropped: int = 0,
+ schema_fingerprint: str | None = None,
+ ) -> None:
+ conn.engine.success_calls.append(connector_id)
+ captured_success["schema_fingerprint"] = schema_fingerprint
+
+ monkeypatch.setattr("app.scheduler.record_poll_success", capture_record_poll_success)
+
+ fake_engine = _FakeEngine([inst])
+ scheduler = ConnectorScheduler(
+ engine=fake_engine,
+ ingest_client=_FakeIngestClient(accepted=0),
+ vault=_FakeVault(),
+ )
+ await scheduler._poll_one(connector_id=inst.id)
+
+ # Empty batch → fingerprint should be None so the DB layer leaves the
+ # baseline alone.
+ assert captured_success["schema_fingerprint"] is None
+ assert fake_engine.drift_calls == []
+
+
# ---------------------------------------------------------------------------
# Failure paths each flip the row to unhealthy without raising
# ---------------------------------------------------------------------------
@@ -338,6 +572,7 @@ class _FakeEngine:
self.instances = list(instances)
self.success_calls: list[uuid.UUID] = []
self.failure_calls: list[uuid.UUID] = []
+ self.drift_calls: list[dict[str, Any]] = []
def begin(self) -> _FakeAsyncContextManager:
return _FakeAsyncContextManager(_FakeConnection(self))
@@ -389,12 +624,31 @@ def _patch_repo_calls(monkeypatch):
async def fake_fetch_enabled_connectors(conn: Any) -> list[ConnectorInstance]:
return list(conn.engine.instances)
- async def fake_record_poll_success(conn: Any, connector_id: uuid.UUID, *, events_added: int) -> None:
+ async def fake_record_poll_success(
+ conn: Any,
+ connector_id: uuid.UUID,
+ *,
+ events_added: int,
+ events_dropped: int = 0,
+ schema_fingerprint: str | None = None,
+ ) -> None:
conn.engine.success_calls.append(connector_id)
async def fake_record_poll_failure(conn: Any, connector_id: uuid.UUID) -> None:
conn.engine.failure_calls.append(connector_id)
+ async def fake_record_schema_drift(
+ conn: Any,
+ connector_id: uuid.UUID,
+ *,
+ fingerprint: str,
+ details: dict[str, Any],
+ ) -> None:
+ getattr(conn.engine, "drift_calls", []).append(
+ {"connector_id": connector_id, "fingerprint": fingerprint, "details": details}
+ )
+
monkeypatch.setattr("app.scheduler.fetch_enabled_connectors", fake_fetch_enabled_connectors)
monkeypatch.setattr("app.scheduler.record_poll_success", fake_record_poll_success)
monkeypatch.setattr("app.scheduler.record_poll_failure", fake_record_poll_failure)
+ monkeypatch.setattr("app.scheduler.record_schema_drift", fake_record_schema_drift)