Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
feat: v1.6 — attack-path agent, connector health, tiered retention pipeline
Beenu Arora6131b1468d9b
diff --git a/.gitignore b/.gitignoreindex e64c6d55e..bc38517ed 100644--- a/.gitignore+++ b/.gitignore@@ -85,3 +85,6 @@ PR_TRIAGE.md# Docusaurus generated cache (apps/docs)apps/docs/.docusaurus/apps/docs/build/++# Local QA artifacts (screenshots from manual UI checks)+apps/web/.qa-screenshots/diff --git a/apps/docs/docs/features/attack-path-agent.md b/apps/docs/docs/features/attack-path-agent.mdnew file mode 100644index 000000000..768047687--- /dev/null+++ b/apps/docs/docs/features/attack-path-agent.md@@ -0,0 +1,82 @@+---+sidebar_position: 3+title: Attack-path investigation agent+description: How AiSOC's investigation graph adds an attack-path step that walks the knowledge graph, scores blast radius, and proposes containment actions on the highest-value pivots.+---++# Attack-path investigation agent++When the investigation agent finishes its analysis of a case, it has a story but not a map. The attack-path agent takes the case, walks the knowledge graph, finds the chain of entities that links the alert to its blast radius, and turns that into concrete containment proposals.++## Where it sits in the workflow++The orchestrator graph (`services/agents/app/graph/workflow.py`) runs:++```+ingest → triage → investigation → attack_path → END+```++The attack-path node runs *after* the investigation agent, so it inherits the case's findings and can layer graph-derived insights on top. It is purely additive: if the graph service has nothing to say (no path, no neighbors), the node records that and exits without modifying the proposed-action queue.++## What it does, step by step++1. **Calls the graph API.** `GET /api/v1/graph/attack-path/{case_id}` returns the path of entities and relationships that the case touches, plus a MITRE tactic chain extracted from the relationship types.+2. **Summarizes the graph.** Records node counts by type (user, host, asset, ip, etc.), edge counts, and the tactic chain into `state.threat_intel["attack_path"]`. This becomes part of the case's durable record.+3. **Scores blast radius for pivot entities.** For every host or service-account on the path, the agent calls `GET /api/v1/graph/blast-radius/{type}/{id}` to compute how many downstream entities depend on it. High blast radius means a single containment action has high impact.+4. **Proposes isolation actions.** For the top pivot entities (those above a configurable blast-radius threshold), the agent emits a `ProposedAction` with type `isolate_host` (or the equivalent for the entity type), an `ActionRisk` band, and the parameters needed to execute — `entity_type`, `entity_id`, `blast_radius_score`.++These proposed actions land in the same queue as everything else the investigation produced, so the existing approval flow (auto-approve for low-risk, human-in-the-loop for high-risk) governs whether they execute.++## Why the graph and not just rules++Three reasons the agent reads from the graph instead of running rules over the raw alert:++1. **Lateral movement is a graph problem.** A SIEM rule can tell you a user authenticated from a new IP. Only a graph can tell you that user has access to seven critical hosts, three of which talk to the database tier.+2. **Blast radius is a graph property.** You can't compute it by counting log lines. You compute it by walking edges.+3. **The same graph powers the SOC analyst's UI.** The agent and the human are looking at the same data structure, which means an analyst can audit *why* the agent proposed a specific isolation in seconds, not minutes.++## Output contract++Inside `InvestigationState`, the agent adds:++- One or more `Finding` entries summarizing the path and the high-blast pivots.+- Zero or more `ProposedAction` entries with `action_type="isolate_host"` (or the appropriate verb for the entity type) and a `risk` band.+- A compact summary at `state.threat_intel["attack_path"]`:++```json+{+ "node_count": 12,+ "edge_count": 18,+ "tactic_chain": ["initial_access", "credential_access", "lateral_movement"],+ "high_blast_pivots": [+ {+ "label": "Host",+ "name": "build-server-01",+ "entity_type": "host",+ "entity_id": "host-1f9...",+ "blast_radius_score": 47+ }+ ]+}+```++Downstream consumers (the case timeline, the response queue, the analytics layer) read from this summary rather than re-walking the graph.++## When it stays silent++The agent records nothing in the proposed-action queue if:++- The graph service returned 404 (no path discoverable for this case yet — common for very new cases or low-fidelity alerts).+- All pivot blast-radius scores are below the threshold.+- The case has no entities to walk from.++Silence is correct behavior. The agent is not required to fire on every case.++## Where it lives in the code++- Agent entry point: `services/agents/app/agents/attack_path_agent.py`+- Graph client tool: `services/agents/app/tools/graph.py`+- LangGraph wiring: `services/agents/app/graph/workflow.py`+- Tests: `services/agents/tests/test_attack_path_agent.py`+- Underlying graph endpoints: `services/api/app/api/v1/endpoints/graph.py`+- Graph service: `services/api/app/services/graph_service.py`diff --git a/apps/docs/docs/features/connector-health.md b/apps/docs/docs/features/connector-health.mdnew file mode 100644index 000000000..b2a2b05d9--- /dev/null+++ b/apps/docs/docs/features/connector-health.md@@ -0,0 +1,86 @@+---+sidebar_position: 1+title: Connector health & schema-drift sentinel+description: How AiSOC fingerprints every connector's event schema, detects drift the moment a vendor changes its API, and gives you a single pane of health across every integration.+---++# Connector health and schema-drift sentinel++Every SOC integration breaks the same way: a vendor ships an API change in their next release, your normalizer silently starts dropping fields, and three weeks later somebody notices a detection went quiet. The schema-drift sentinel exists so that hour-zero is the same as hour-three-weeks: AiSOC notices the drift on the next poll and surfaces it before any downstream rule misfires.++This page documents the model, the API surface, and how to operate it.++## What it does++Every time the connector scheduler polls a vendor and normalizes a batch of events, AiSOC computes a stable **schema fingerprint** — a SHA-256 hash of the sorted set of top-level field names in the normalized payload. That fingerprint is stored on the connector instance.++On the next poll:++- If the fingerprint matches, we record a successful run.+- If it changes, we **diff the previous and current field sets**, persist the drift event with `added`/`removed` fields and the previous/current fingerprint, and surface it via the connector health API.++Because the hash is over the *normalized* schema (not the raw vendor payload), it is stable across order-of-fields changes, cosmetic vendor renames that the connector silently rewrites, and value-only changes — but it changes the moment a real field appears or disappears.++## Why fingerprints, not full diffs++The naive design is "store the whole schema and diff it every time." We don't, for three reasons:++1. **Cheap to compute, cheap to store.** A 32-byte hash per connector beats keeping every prior schema around.+2. **No PII or vendor data leaks into the audit trail.** The fingerprint reveals nothing about the events themselves — it's a one-way hash of the field-name set.+3. **Equality check is O(1).** The hot path on every poll is a single string compare.++When drift is detected, we then *also* persist the human-readable diff (`added`/`removed`/`previous_fingerprint`) so an operator can read what changed without recomputing it.++## Health surface++`GET /api/v1/connectors/health` returns a tenant-scoped health summary:++```json+{+ "total": 12,+ "healthy": 10,+ "degraded": 1,+ "error": 0,+ "drift_in_last_24h": 1,+ "events_dropped_total": 247,+ "instances": [+ {+ "id": "...",+ "connector_id": "github",+ "name": "github-prod",+ "health_status": "healthy",+ "schema_fingerprint": "a3f9...",+ "last_schema_drift_at": null,+ "last_drift_details": null,+ "events_dropped": 0,+ "last_run_at": "2026-05-07T22:31:04Z"+ }+ ]+}+```++The per-instance fields:++| Field | Meaning |+|---|---|+| `schema_fingerprint` | Current normalized-schema hash. |+| `last_schema_drift_at` | Timestamp of the most recent drift event (null if never). |+| `last_drift_details` | `{ "added": [...], "removed": [...], "previous_fingerprint": "..." }`. |+| `events_dropped` | Cumulative count of events the pre-ingest filter dropped (see [Security data pipeline](./data-pipeline.md)). |+| `health_status` | `healthy`, `degraded`, or `error` — derived from recent run history. |++## Operational notes++- **Drift is not a failure.** A drift event is informational: it says "the shape of your incoming events changed". The connector keeps polling. You decide whether to investigate, update the normalizer, or accept the change.+- **First run sets the baseline.** A connector with no prior fingerprint adopts the first observed one without raising drift; only subsequent polls trigger drift events.+- **Empty batches don't reset the fingerprint.** A poll that returns zero events leaves the prior fingerprint intact, so transient vendor outages don't show up as drift.++## Where it lives in the code++- Fingerprinting helpers: `services/connectors/app/pipeline/fingerprint.py`+- Filter rules (used to compute `events_dropped`): `services/connectors/app/pipeline/filter_rules.py`+- Drift integration in the polling loop: `services/connectors/app/scheduler.py`+- API surface: `services/api/app/api/v1/endpoints/connectors.py` (`/health`)+- Schema migration: `services/api/migrations/026_connector_schema_drift.sql`++If you're adding a new connector, you don't need to do anything — the scheduler computes the fingerprint from the normalized batch your `normalize()` returns. Drift detection is implicit.diff --git a/apps/docs/docs/features/data-pipeline.md b/apps/docs/docs/features/data-pipeline.mdnew file mode 100644index 000000000..5339d1abf--- /dev/null+++ b/apps/docs/docs/features/data-pipeline.md@@ -0,0 +1,93 @@+---+sidebar_position: 2+title: Security data pipeline & tiered retention+description: Declarative, per-connector pre-ingest filter rules that drop or keep events before they hit hot storage, plus the tiered retention model that backs them.+---++# Security data pipeline and tiered retention++Most SIEMs treat ingest as a tax: every event goes in at the same cost, and you pay until the bill forces you to drop sources. AiSOC takes a different stance — **decide what to keep before it crosses the wire**, with simple, declarative rules that are easy to read in a code review.++## The pre-ingest filter++Every connector instance can carry a `filter_rules` block in its config. On every poll, after normalization but before the events are pushed to ingest, the scheduler runs the filter and either drops or keeps each event.++```yaml+filter_rules:+ - field: "raw.severity"+ op: "eq"+ value: "informational"+ action: "drop"+ - field: "raw.action"+ op: "in"+ value: ["allow", "ignore"]+ action: "drop"+ - field: "raw.user.email"+ op: "ends_with"+ value: "@partner.example.com"+ action: "keep"+```++Rules are evaluated **first match wins**. An event with no matching rule is kept by default — filters are an opt-in deny list, not a strict allow list.++### Supported operators++| Operator | Meaning |+|---|---|+| `eq` / `ne` | Exact equality / inequality. |+| `contains` | Substring match (case-sensitive). |+| `starts_with` / `ends_with` | Prefix / suffix match. |+| `in` | Value is in a list of allowed values. |++The `field` selector is a dotted path into the normalized event (e.g. `raw.severity`, `entities.user.email`). Missing fields cause the rule to skip rather than crash, so a malformed rule degrades gracefully — the event passes through.++### Why declarative, not a Lua hook++We deliberately keep the filter language small:++- **It reads in a code review.** A junior analyst can audit a YAML block.+- **It serializes cleanly.** Filter rules live in `connector_config.filter_rules` and are versioned with the rest of the connector instance.+- **It can't fork, exec, or call out.** The pre-ingest path is on the hot poll loop. Limiting expressiveness is a feature, not a bug.++If you genuinely need a Turing-complete enrichment step, that belongs in the enrichment service, not the filter.++## Drop counters++Every dropped event increments `events_dropped` on the connector instance. That counter is surfaced both per-instance and aggregated across the tenant in `GET /api/v1/connectors/health`:++```json+{+ "events_dropped_total": 247,+ "instances": [+ { "name": "cloudflare-prod", "events_dropped": 247, ... }+ ]+}+```++This lets you answer "is the filter doing what I think it's doing?" without grepping logs.++## Tiered retention model++The pipeline assumes a three-tier storage layout:++| Tier | Backend | Default retention | Purpose |+|---|---|---|---|+| **Hot** | OpenSearch | 7-30 days | Detection rules, real-time queries, the last week of activity. |+| **Warm** | ClickHouse / Parquet on object store | 90 days | Historical investigation, hunt queries, compliance look-back. |+| **Cold** | Object store (S3 / R2) | 1 year+ | Compliance archive. Queryable on demand, cheap at rest. |++The pre-ingest filter sits *in front* of all three: an event the filter drops never lands in any tier. That makes it the highest-leverage knob you have for cost control — drops are free, retries on hot storage are not.++## Operational guidance++- **Start permissive, then tighten.** First week of any new connector: no filter rules. Watch the volume. Then add rules to drop the noisy floor (informational severity, repeated noisy-actor health checks, etc.).+- **Filter rules are per-instance.** Two GitHub connectors for two orgs can have completely different rule sets.+- **`events_dropped` is monotonic.** Reset the counter by re-creating the instance if you want a clean window.+- **Don't filter what detections need.** A detection that triggers on `severity=informational` will silently never fire if a filter drops those events. Review filter rules whenever you ship a new detection rule.++## Where it lives in the code++- Filter engine: `services/connectors/app/pipeline/filter_rules.py`+- Filter integration in the polling loop: `services/connectors/app/scheduler.py`+- Drop counter persistence: `services/connectors/app/db/connector_repo.py`+- API surface: `services/api/app/api/v1/endpoints/connectors.py`diff --git a/apps/docs/sidebars.ts b/apps/docs/sidebars.tsindex a18202209..911212918 100644--- a/apps/docs/sidebars.ts+++ b/apps/docs/sidebars.ts@@ -12,6 +12,15 @@ const sidebars: SidebarsConfig = {label: "Core Concepts",items: ["concepts/cases", "concepts/playbooks", "concepts/detections"],},+ {+ type: "category",+ label: "Features",+ items: [+ "features/connector-health",+ "features/data-pipeline",+ "features/attack-path-agent",+ ],+ },{type: "category",label: "Connectors",diff --git a/apps/web/src/components/cases/CaseWorkspace.tsx b/apps/web/src/components/cases/CaseWorkspace.tsxindex 07a646fb2..002327b4e 100644--- a/apps/web/src/components/cases/CaseWorkspace.tsx+++ b/apps/web/src/components/cases/CaseWorkspace.tsx@@ -24,7 +24,9 @@ import { format, formatDistanceToNow } from 'date-fns';import toast from 'react-hot-toast';import {casesApi,+ graphApi,type Case,+ type CaseAttackPath,type CaseSeverity,type CaseStatus,type CaseTask,@@ -36,9 +38,20 @@ import { EmptyState } from '@/components/ui/EmptyState';import { InvestigationLedger } from './InvestigationLedger';import { ContextualActions } from '@/components/copilot/ContextualActions';-type WorkspaceTab = 'overview' | 'investigation' | 'ledger' | 'report';--const VALID_TABS: readonly WorkspaceTab[] = ['overview', 'investigation', 'ledger', 'report'];+type WorkspaceTab =+ | 'overview'+ | 'investigation'+ | 'attack-path'+ | 'ledger'+ | 'report';++const VALID_TABS: readonly WorkspaceTab[] = [+ 'overview',+ 'investigation',+ 'attack-path',+ 'ledger',+ 'report',+];function isWorkspaceTab(value: string | null): value is WorkspaceTab {return value !== null && (VALID_TABS as readonly string[]).includes(value);@@ -701,7 +714,7 @@ export function CaseWorkspace({ caseId }: { caseId: string }) {{/* Tabs */}<div className="flex items-center gap-1 border-b border-slate-800/70 text-xs">- {(['overview', 'investigation', 'ledger', 'report'] as WorkspaceTab[]).map((tab) => (+ {VALID_TABS.map((tab) => (<buttonkey={tab}onClick={() => setActiveTab(tab)}@@ -725,6 +738,16 @@ export function CaseWorkspace({ caseId }: { caseId: string }) {audit</span></span>+ ) : tab === 'attack-path' ? (+ <span className="inline-flex items-center gap-1">+ Attack path+ <span+ className="rounded bg-orange-500/15 px-1 py-0.5 text-[8px] font-bold uppercase tracking-wider text-orange-300 ring-1 ring-orange-500/30"+ title="Reconstructed attack graph from the Attack-Path Investigation Agent"+ >+ graph+ </span>+ </span>) : (tab)}@@ -745,6 +768,11 @@ export function CaseWorkspace({ caseId }: { caseId: string }) {/>)}+ {/* Attack path tab — reconstructed graph for this case */}+ {activeTab === 'attack-path' && (+ <AttackPathPanel caseId={caseRecord.id} />+ )}+{/* Ledger tab — persistent, replayable agent decision log */}{activeTab === 'ledger' && (<InvestigationLedger@@ -1099,6 +1127,148 @@ function InvestigationPanel({);}+// ─── Attack path panel ────────────────────────────────────────────────────────++/**+ * AttackPathPanel renders the case-scoped attack-path graph reconstructed by+ * the Attack-Path Investigation Agent. Cases without graph entities (404 from+ * the backend) fall back to an explanatory empty state so the analyst knows+ * the agent is wired up but simply has no graph context yet.+ */+function AttackPathPanel({ caseId }: { caseId: string }) {+ const { data, error, isLoading } = useSWR<CaseAttackPath | null>(+ `case:${caseId}:attack-path`,+ () => graphApi.getCaseAttackPath(caseId, { maxDepth: 4 }),+ { revalidateOnFocus: false },+ );++ if (isLoading) {+ return (+ <div className="space-y-3">+ <Skeleton className="h-6 w-48" />+ <Skeleton className="h-32 w-full" />+ <Skeleton className="h-32 w-full" />+ </div>+ );+ }++ if (error) {+ return (+ <ErrorState+ title="Failed to load attack path"+ description={error instanceof Error ? error.message : 'Unknown error'}+ />+ );+ }++ if (!data || data.nodeCount === 0) {+ return (+ <div className="flex flex-col items-center justify-center rounded-xl border border-dashed border-slate-700/60 bg-slate-900/30 py-16 text-center">+ <div className="mb-3 flex h-10 w-10 items-center justify-center rounded-full bg-orange-500/10 ring-1 ring-orange-500/30">+ <svg className="h-5 w-5 text-orange-300" fill="none" viewBox="0 0 24 24" stroke="currentColor">+ <path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M13 10V3L4 14h7v7l9-11h-7z" />+ </svg>+ </div>+ <p className="text-sm font-medium text-slate-300">No attack path yet</p>+ <p className="mt-1 max-w-md text-xs text-slate-500">+ The Attack-Path Investigation Agent reconstructs lateral movement and blast radius from linked alerts and graph entities. Run an investigation or link more alerts to surface a path here.+ </p>+ </div>+ );+ }++ // Bucket nodes by entity type so the panel reads as “hosts /+ // identities / processes” rather than an undifferentiated soup.+ const nodesByType = data.nodes.reduce<Record<string, CaseAttackPath['nodes']>>(+ (acc, node) => {+ const type = (node.properties?.type as string) ?? node.label ?? 'entity';+ (acc[type] ||= []).push(node);+ return acc;+ },+ {},+ );++ return (+ <div className="space-y-4">+ <div className="flex items-center justify-between">+ <div>+ <h3 className="text-sm font-medium text-slate-200">Reconstructed attack path</h3>+ <p className="text-xs text-slate-500">+ {data.nodeCount} node{data.nodeCount === 1 ? '' : 's'} · {data.edgeCount} edge{data.edgeCount === 1 ? '' : 's'} · depth ≤ 4+ </p>+ </div>+ <span className="inline-flex items-center gap-1.5 rounded-full bg-orange-500/10 px-2.5 py-0.5 text-xs font-medium text-orange-300 ring-1 ring-orange-500/30">+ attack path+ </span>+ </div>++ {/* Entities grouped by type — keeps the panel scannable even when the+ backend returns dense graphs (10s of nodes is normal). */}+ <div className="grid grid-cols-1 gap-4 lg:grid-cols-2">+ {Object.entries(nodesByType).map(([type, nodes]) => (+ <div+ key={type}+ className="rounded-xl border border-slate-800/80 bg-slate-900/40 p-4"+ >+ <h4 className="mb-2 text-xs font-semibold uppercase tracking-wide text-slate-400">+ {type} ({nodes.length})+ </h4>+ <ul className="space-y-1.5">+ {nodes.slice(0, 8).map((node) => (+ <li+ key={node.id}+ className="flex items-start gap-2 text-xs"+ >+ <span className="mt-0.5 inline-block h-1.5 w-1.5 rounded-full bg-orange-400/60 shrink-0" />+ <div className="min-w-0">+ <p className="truncate font-mono text-slate-300">{node.label || node.id}</p>+ {typeof node.properties?.risk_score === 'number' && (+ <p className="text-[10px] text-slate-500">+ risk score: {(node.properties.risk_score as number).toFixed(2)}+ </p>+ )}+ </div>+ </li>+ ))}+ {nodes.length > 8 && (+ <li className="text-[10px] italic text-slate-500">+ +{nodes.length - 8} more+ </li>+ )}+ </ul>+ </div>+ ))}+ </div>++ {/* Edges — show the relationship verbs because that's what differentiates+ a benign topology snapshot from an actual attack chain. */}+ {data.edges.length > 0 && (+ <div className="rounded-xl border border-slate-800/80 bg-slate-900/40 p-4">+ <h4 className="mb-2 text-xs font-semibold uppercase tracking-wide text-slate-400">+ Relationships+ </h4>+ <ul className="space-y-1">+ {data.edges.slice(0, 12).map((edge, i) => (+ <li key={`${edge.source}-${edge.target}-${i}`} className="flex items-center gap-2 text-[11px] font-mono">+ <span className="truncate text-slate-300">{edge.source}</span>+ <span className="rounded bg-slate-800 px-1.5 py-0.5 text-[10px] text-orange-300">+ {edge.type}+ </span>+ <span className="truncate text-slate-300">{edge.target}</span>+ </li>+ ))}+ {data.edges.length > 12 && (+ <li className="text-[10px] italic text-slate-500">+ +{data.edges.length - 12} more relationships+ </li>+ )}+ </ul>+ </div>+ )}+ </div>+ );+}+function AuditLogPreview({ log }: { log: Array<{ kind: string; agent: string; summary: string }> }) {return (<ul className="p-4 space-y-1.5">diff --git a/apps/web/src/components/connectors/ConnectorInstanceList.tsx b/apps/web/src/components/connectors/ConnectorInstanceList.tsxindex d69b2a74d..29922e783 100644--- a/apps/web/src/components/connectors/ConnectorInstanceList.tsx+++ b/apps/web/src/components/connectors/ConnectorInstanceList.tsx@@ -103,6 +103,28 @@ function ConnectorCard({}: ConnectorCardProps) {const statusCfg = STATUS_CONFIG[connector.status] ?? STATUS_CONFIG.configuring;const alertCount = connector.alertCount ?? connector.alertsIngested ?? 0;+ const eventsDropped = connector.eventsDropped ?? 0;+ const driftAt = connector.lastSchemaDriftAt;+ const drift = connector.lastDriftDetails;+ const driftAdded = drift?.added?.length ?? 0;+ const driftRemoved = drift?.removed?.length ?? 0;+ const driftRecent =+ driftAt &&+ Date.now() - new Date(driftAt).getTime() < 24 * 60 * 60 * 1000;++ // Compose the drift tooltip lazily — only the schema-drift sentinel emits a+ // diff so we keep the UI quiet when nothing has changed.+ const driftTooltip = drift+ ? [+ driftAdded > 0 ? `+${driftAdded} new field${driftAdded === 1 ? '' : 's'}` : null,+ driftRemoved > 0+ ? `-${driftRemoved} removed field${driftRemoved === 1 ? '' : 's'}`+ : null,+ driftAt ? `at ${new Date(driftAt).toLocaleString()}` : null,+ ]+ .filter(Boolean)+ .join(' • ')+ : '';return (<div className="bg-gray-900/60 border border-gray-800/60 rounded-xl p-5 hover:border-gray-700/60 transition-colors flex flex-col">@@ -132,6 +154,57 @@ function ConnectorCard({<p className="text-xs text-gray-500 mb-4 line-clamp-2">{connector.description}</p>)}+ {/* Schema-drift + filter-drop badges. Only render when there is+ something to surface so a healthy connector keeps its clean look. */}+ {(driftAt || eventsDropped > 0) && (+ <div className="flex flex-wrap gap-1.5 mb-3">+ {driftAt && (+ <span+ title={driftTooltip}+ className={clsx(+ 'text-[10px] px-2 py-0.5 rounded-full border inline-flex items-center gap-1',+ driftRecent+ ? 'text-amber-300 bg-amber-500/10 border-amber-500/30'+ : 'text-gray-400 bg-gray-800/60 border-gray-700/50',+ )}+ >+ <svg className="w-3 h-3" fill="none" viewBox="0 0 24 24" stroke="currentColor">+ <path+ strokeLinecap="round"+ strokeLinejoin="round"+ strokeWidth={2}+ d="M12 9v2m0 4h.01M5 19h14a2 2 0 001.74-2.99l-7-12a2 2 0 00-3.48 0l-7 12A2 2 0 005 19z"+ />+ </svg>+ {driftRecent ? 'Schema drift' : 'Schema changed'}+ {(driftAdded > 0 || driftRemoved > 0) && (+ <span className="opacity-80">+ {driftAdded > 0 && `+${driftAdded}`}+ {driftAdded > 0 && driftRemoved > 0 && '/'}+ {driftRemoved > 0 && `-${driftRemoved}`}+ </span>+ )}+ </span>+ )}+ {eventsDropped > 0 && (+ <span+ title={`${eventsDropped.toLocaleString()} event${eventsDropped === 1 ? '' : 's'} dropped by pre-ingest filter rules`}+ className="text-[10px] px-2 py-0.5 rounded-full border inline-flex items-center gap-1 text-blue-300 bg-blue-500/10 border-blue-500/30"+ >+ <svg className="w-3 h-3" fill="none" viewBox="0 0 24 24" stroke="currentColor">+ <path+ strokeLinecap="round"+ strokeLinejoin="round"+ strokeWidth={2}+ d="M3 4a1 1 0 011-1h16a1 1 0 011 1v2.586a1 1 0 01-.293.707l-6.414 6.414a1 1 0 00-.293.707V17l-4 4v-6.586a1 1 0 00-.293-.707L3.293 7.293A1 1 0 013 6.586V4z"+ />+ </svg>+ {eventsDropped.toLocaleString()} dropped+ </span>+ )}+ </div>+ )}+<div className="grid grid-cols-2 gap-2 mb-4"><div className="bg-gray-800/60 rounded-lg p-2"><p className="text-xs text-gray-500">Events ingested</p>diff --git a/apps/web/src/components/connectors/ConnectorsView.tsx b/apps/web/src/components/connectors/ConnectorsView.tsxindex c6c535e55..d3d3b5460 100644--- a/apps/web/src/components/connectors/ConnectorsView.tsx+++ b/apps/web/src/components/connectors/ConnectorsView.tsx@@ -20,7 +20,11 @@ import useSWR from 'swr';import toast from 'react-hot-toast';import { clsx } from 'clsx';-import { connectorsApi, type Connector } from '@/lib/api';+import {+ connectorsApi,+ type Connector,+ type ConnectorHealthSummary,+} from '@/lib/api';import { ErrorState } from '@/components/ui/ErrorState';import { AddConnectorModal } from './AddConnectorModal';import { ConnectorInstanceList } from './ConnectorInstanceList';@@ -73,18 +77,56 @@ export function ConnectorsView() {{ revalidateOnFocus: false, fallbackData: { connectors: DEMO_CONNECTORS } },);+ // Health summary is a separate endpoint so the empty/error case is silent —+ // we just fall back to the locally-derived stats. SWR keys are namespaced so+ // the two queries don't fight over the cache.+ const { data: healthSummary } = useSWR<ConnectorHealthSummary | null>(+ 'connectors:health',+ async () => {+ try {+ return await connectorsApi.health();+ } catch {+ return null;+ }+ },+ { revalidateOnFocus: false, refreshInterval: 60_000 },+ );+const connectors: Connector[] = useMemo(() => data?.connectors ?? [], [data]);- const stats = useMemo(() => {+ const localStats = useMemo(() => {const active = connectors.filter((c) => c.status === 'active').length;const errored = connectors.filter((c) => c.status === 'error').length;const totalEvents = connectors.reduce((sum, c) => sum + (c.alertCount ?? c.alertsIngested ?? 0),0,);- return { active, errored, totalEvents };+ const driftedRecently = connectors.filter((c) => {+ if (!c.lastSchemaDriftAt) return false;+ return Date.now() - new Date(c.lastSchemaDriftAt).getTime() < 24 * 60 * 60 * 1000;+ }).length;+ const totalDropped = connectors.reduce(+ (sum, c) => sum + (c.eventsDropped ?? 0),+ 0,+ );+ return { active, errored, totalEvents, driftedRecently, totalDropped };}, [connectors]);+ // Prefer backend-aggregated stats when the API is reachable so the tile+ // matches the source of truth (the schema-drift sentinel runs server-side).+ const stats = useMemo(() => {+ if (healthSummary) {+ return {+ active: healthSummary.healthy,+ errored: healthSummary.unhealthy,+ totalEvents: healthSummary.totalEventsIngested,+ driftedRecently: healthSummary.driftedRecently,+ totalDropped: healthSummary.totalEventsDropped,+ };+ }+ return localStats;+ }, [healthSummary, localStats]);+// ─── Actions ──────────────────────────────────────────────────────────────const handleTest = async (id: string) => {@@ -164,26 +206,61 @@ export function ConnectorsView() {</div>{/* Stats — render even with zero connectors so the layout is stable- between empty/populated states. */}- <div className="grid grid-cols-2 md:grid-cols-4 gap-3">+ between empty/populated states. The drift/dropped tiles only render+ when the schema-drift sentinel has actually fired so a clean tenant+ keeps a calm 4-tile layout. */}+ <div+ className={clsx(+ 'grid grid-cols-2 gap-3',+ stats.driftedRecently > 0 || stats.totalDropped > 0+ ? 'md:grid-cols-6'+ : 'md:grid-cols-4',+ )}+ >{[- { label: 'Total Connectors', value: connectors.length, color: 'text-blue-400' },- { label: 'Active', value: stats.active, color: 'text-green-400' },- { label: 'Errors', value: stats.errored, color: 'text-red-400' },+ {+ label: 'Total Connectors',+ value: healthSummary?.total ?? connectors.length,+ color: 'text-blue-400',+ show: true,+ },+ { label: 'Active', value: stats.active, color: 'text-green-400', show: true },+ { label: 'Errors', value: stats.errored, color: 'text-red-400', show: true },{label: 'Events Ingested',value: stats.totalEvents.toLocaleString(),color: 'text-purple-400',+ show: true,+ },+ {+ label: 'Schema Drift (24h)',+ value: stats.driftedRecently,+ color: 'text-amber-400',+ show: stats.driftedRecently > 0,+ tooltip:+ healthSummary?.lastDriftAt+ ? `Most recent drift: ${new Date(healthSummary.lastDriftAt).toLocaleString()}`+ : undefined,+ },+ {+ label: 'Events Dropped',+ value: stats.totalDropped.toLocaleString(),+ color: 'text-cyan-400',+ show: stats.totalDropped > 0,+ tooltip: 'Filtered out by pre-ingest pipeline rules',},- ].map((stat) => (- <div- key={stat.label}- className="bg-gray-900/60 border border-gray-800/60 rounded-xl p-4"- >- <p className={clsx('text-2xl font-bold mb-1', stat.color)}>{stat.value}</p>- <p className="text-xs text-gray-500">{stat.label}</p>- </div>- ))}+ ]+ .filter((stat) => stat.show)+ .map((stat) => (+ <div+ key={stat.label}+ title={stat.tooltip}+ className="bg-gray-900/60 border border-gray-800/60 rounded-xl p-4"+ >+ <p className={clsx('text-2xl font-bold mb-1', stat.color)}>{stat.value}</p>+ <p className="text-xs text-gray-500">{stat.label}</p>+ </div>+ ))}</div>{/* Body */}diff --git a/apps/web/src/components/landing/Features.tsx b/apps/web/src/components/landing/Features.tsxindex 659de6e55..f67ba1936 100644--- a/apps/web/src/components/landing/Features.tsx+++ b/apps/web/src/components/landing/Features.tsx@@ -11,6 +11,14 @@ const FEATURES = [<path d="M4 4h16v4H4zM4 10h10v4H4zM4 16h16v4H4z" />),},+ {+ title: 'Tiered retention pipeline',+ description:+ 'Declarative pre-ingest filter rules drop noisy events at the connector edge before they hit Kafka — cutting indexing cost without losing forensic depth. Every drop is counted per-connector and surfaced in the health summary.',+ icon: (+ <path d="M3 4a1 1 0 011-1h16a1 1 0 011 1v2.586a1 1 0 01-.293.707l-6.414 6.414a1 1 0 00-.293.707V17l-4 4v-6.586a1 1 0 00-.293-.707L3.293 7.293A1 1 0 013 6.586V4z" />+ ),+ },{title: 'Three-tier agent memory',description:@@ -75,6 +83,14 @@ const FEATURES = [<path d="M5 5a3 3 0 116 0 3 3 0 01-6 0zm8 14a3 3 0 116 0 3 3 0 01-6 0zM7.5 8l5 8" />),},+ {+ title: 'Attack-path investigation agent',+ description:+ 'A dedicated LangGraph agent reconstructs lateral movement and blast radius from the live graph, summarises high-risk pivots, and proposes contained response actions (host isolation, account disable) gated by autonomy guardrails.',+ icon: (+ <path d="M13 10V3L4 14h7v7l9-11h-7z" />+ ),+ },{title: 'SOC metrics dashboard',description:@@ -91,6 +107,14 @@ const FEATURES = [<path d="M9 3H5a2 2 0 00-2 2v4m6-6h10a2 2 0 012 2v4M9 3v18m0 0h10a2 2 0 002-2V9M9 21H5a2 2 0 01-2-2V9m0 0h18" />),},+ {+ title: 'Schema-drift sentinel',+ description:+ 'Every poll fingerprints the upstream payload shape and diffs it against the last known-good schema. New, removed, or renamed fields surface in the connector card and aggregate health tile before they silently break detections.',+ icon: (+ <path d="M12 9v2m0 4h.01M5 19h14a2 2 0 001.74-2.99l-7-12a2 2 0 00-3.48 0l-7 12A2 2 0 005 19z" />+ ),+ },{title: 'Compliance & governance',description:diff --git a/apps/web/src/lib/api.ts b/apps/web/src/lib/api.tsindex 658ad8ba8..460160572 100644--- a/apps/web/src/lib/api.ts+++ b/apps/web/src/lib/api.ts@@ -761,6 +761,18 @@ export interface Connector {description?: string;createdAt?: string;updatedAt?: string;+ /** SHA-256 fingerprint of the most recent normalized event schema. */+ schemaFingerprint?: string;+ /** Timestamp of the most recent schema-drift event (null/undef if never). */+ lastSchemaDriftAt?: string;+ /** Human-readable diff for the most recent drift event. */+ lastDriftDetails?: {+ added?: string[];+ removed?: string[];+ previous_fingerprint?: string;+ };+ /** Cumulative count of events the pre-ingest filter dropped. */+ eventsDropped?: number;}export interface ConnectorsResponse {@@ -830,6 +842,32 @@ interface BackendConnectorResponse {tags: string[];created_at: string;updated_at: string;+ schema_fingerprint?: string | null;+ last_schema_drift_at?: string | null;+ last_drift_details?: {+ added?: string[];+ removed?: string[];+ previous_fingerprint?: string;+ } | null;+ events_dropped?: number | null;+}++/** Aggregated health summary for the connectors fleet (a single tenant). */+export interface ConnectorHealthSummary {+ total: number;+ healthy: number;+ unhealthy: number;+ unknown: number;+ /** Connectors whose schema has ever drifted. */+ drifted: number;+ /** Connectors whose schema drifted in the last 24h. */+ driftedRecently: number;+ /** Most-recent drift timestamp across the fleet. */+ lastDriftAt?: string;+ /** Cumulative events ingested across all enabled connectors. */+ totalEventsIngested: number;+ /** Cumulative events dropped by pre-ingest filter rules. */+ totalEventsDropped: number;}/**@@ -870,6 +908,10 @@ function mapBackendConnector(row: BackendConnectorResponse): Connector {alertsIngested: row.events_ingested,createdAt: row.created_at,updatedAt: row.updated_at,+ schemaFingerprint: row.schema_fingerprint ?? undefined,+ lastSchemaDriftAt: row.last_schema_drift_at ?? undefined,+ lastDriftDetails: row.last_drift_details ?? undefined,+ eventsDropped: row.events_dropped ?? 0,};}@@ -960,6 +1002,39 @@ export const connectorsApi = {delete: (id: string) =>request<void>(`/api/v1/connectors/${id}`, { method: 'DELETE' }),++ /**+ * Aggregated health summary for the connectors fleet.+ *+ * Surfaces healthy/degraded/error counts, schema-drift events seen in the+ * last 24h, and the cumulative count of events the pre-ingest filter+ * dropped. Used by the Connectors page header strip and the new+ * "Connector Health" widget on the dashboard.+ */+ health: async (): Promise<ConnectorHealthSummary> => {+ const raw = await request<{+ total: number;+ healthy: number;+ unhealthy: number;+ unknown: number;+ drifted: number;+ drifted_recently: number;+ last_drift_at: string | null;+ total_events_ingested: number;+ total_events_dropped: number;+ }>('/api/v1/connectors/health');+ return {+ total: raw.total,+ healthy: raw.healthy,+ unhealthy: raw.unhealthy,+ unknown: raw.unknown,+ drifted: raw.drifted,+ driftedRecently: raw.drifted_recently,+ lastDriftAt: raw.last_drift_at ?? undefined,+ totalEventsIngested: raw.total_events_ingested,+ totalEventsDropped: raw.total_events_dropped,+ };+ },};// ─── Threat Intel ─────────────────────────────────────────────────────────────@@ -1169,6 +1244,41 @@ export interface MitreCoverage {generatedAt: string;}+/**+ * Case-scoped attack path graph as returned by the Attack-Path Investigation+ * Agent backend (`/api/v1/graph/attack-path/{case_id}`). Distinct from the+ * tenant-level `AttackGraph` overview because the backend payload schema is+ * leaner: each node carries a `properties` dict and edges carry a relationship+ * `type` instead of label/weight metadata.+ */+export interface CaseAttackPathNode {+ id: string;+ label: string;+ properties: Record<string, unknown>;+}++export interface CaseAttackPathEdge {+ source: string;+ target: string;+ type: string;+}++export interface CaseAttackPath {+ caseId: string;+ nodes: CaseAttackPathNode[];+ edges: CaseAttackPathEdge[];+ nodeCount: number;+ edgeCount: number;+}++interface BackendCaseAttackPathResponse {+ case_id: string;+ nodes: CaseAttackPathNode[];+ edges: CaseAttackPathEdge[];+ node_count: number;+ edge_count: number;+}+export const graphApi = {getOverview: (filters: { entity?: string; depth?: number } = {}) =>request<AttackGraph>('/api/v1/graph', {@@ -1188,6 +1298,37 @@ export const graphApi = {`/api/v1/graph/blast-radius`,{ params: { entity } },),++ /**+ * Fetch the reconstructed attack-path graph for a single case.+ * Backed by the Attack-Path Investigation Agent endpoint+ * `/api/v1/graph/attack-path/{case_id}`.+ * Returns `null` when the case has no linked graph entities (404 from API).+ */+ getCaseAttackPath: async (+ caseId: string,+ options: { maxDepth?: number } = {},+ ): Promise<CaseAttackPath | null> => {+ try {+ const raw = await request<BackendCaseAttackPathResponse>(+ `/api/v1/graph/attack-path/${encodeURIComponent(caseId)}`,+ {+ params: options.maxDepth ? { max_depth: options.maxDepth } : {},+ },+ );+ return {+ caseId: raw.case_id,+ nodes: raw.nodes ?? [],+ edges: raw.edges ?? [],+ nodeCount: raw.node_count ?? 0,+ edgeCount: raw.edge_count ?? 0,+ };+ } catch (err) {+ const apiErr = err as { status?: number };+ if (apiErr?.status === 404) return null;+ throw err;+ }+ },};// ─── Detection Rules ─────────────────────────────────────────────────────────diff --git a/services/agents/app/agents/attack_path_agent.py b/services/agents/app/agents/attack_path_agent.pynew file mode 100644index 000000000..b9f9b218c--- /dev/null+++ b/services/agents/app/agents/attack_path_agent.py@@ -0,0 +1,255 @@+"""+Attack-Path Investigation Agent.++Runs after the standard investigation node. Walks the knowledge graph+(Case → Alerts → Hosts/Users → IOCs → MITRE techniques) and computes blast+radius for the highest-impact pivot points. Adds graph-derived findings and+proposed actions to the InvestigationState.++This is the Neo4j + LangGraph integration step: it converts the static+investigation report into a connected story that explains *how an attacker+could reach the crown jewels from the observed indicators*.+"""++from __future__ import annotations++import os+from collections import Counter+from typing import Any++import structlog++from app.models.state import ActionRisk, AgentStatus, InvestigationState, ProposedAction+from app.tools.graph import get_attack_path, get_blast_radius++logger = structlog.get_logger()++# Internal service token used when the agents service calls the API. In dev+# this is set via the AGENTS_API_TOKEN env var; in production it is rotated+# alongside the rest of the service-mesh creds.+_SERVICE_TOKEN = os.getenv("AGENTS_API_TOKEN")++# Score thresholds (0.0–1.0) that mirror the buckets in graph_service.py.+_BLAST_HIGH = 0.7+_BLAST_MED = 0.4++# Cap how many entities we run blast-radius on per run to keep tail latency+# bounded — the graph traversal is the most expensive part of the workflow.+_MAX_BLAST_ENTITIES = 5+++async def run_attack_path(state: InvestigationState) -> InvestigationState:+ """Walk the graph for the case linked to this investigation and enrich state."""+ logger.info("Attack-path agent starting", incident_id=str(state.incident_id))+ state.iteration_count += 1++ case_id = _resolve_case_id(state)+ if not case_id:+ # No case linkage yet — agent has nothing to traverse. Stay silent so+ # we don't pollute the report with noise; the workflow is still+ # complete because investigation_node already set status.+ logger.debug("No case_id on state; skipping attack-path agent")+ return state++ path = await get_attack_path(case_id=case_id, api_token=_SERVICE_TOKEN)+ if path.get("error") or not path.get("nodes"):+ state.add_finding(+ f"Attack-path graph empty for case {case_id} "+ f"(node_count={path.get('node_count', 0)})."+ )+ return state++ # ---- Summarise the path ----+ nodes: list[dict[str, Any]] = path.get("nodes", [])+ edges: list[dict[str, Any]] = path.get("edges", [])+ label_counts = Counter(n.get("label", "Unknown") for n in nodes)+ state.add_finding(+ "Attack-path graph: "+ + ", ".join(f"{c} {label}" for label, c in label_counts.most_common())+ + f" across {len(edges)} relationships."+ )++ # Tactic chain (deduplicated, order-preserving) gives a quick read on+ # how far through the kill chain the attacker has progressed.+ tactic_chain = _extract_tactic_chain(nodes)+ if tactic_chain:+ state.add_finding(+ f"Kill-chain stages reached: {' → '.join(tactic_chain)}."+ )++ # ---- Pick high-value entities to compute blast radius for ----+ pivot_targets = _select_blast_targets(nodes)++ high_blast: list[dict[str, Any]] = []+ for pivot in pivot_targets[:_MAX_BLAST_ENTITIES]:+ blast = await get_blast_radius(+ entity_type=pivot["entity_type"],+ entity_id=pivot["entity_id"],+ api_token=_SERVICE_TOKEN,+ )+ if blast.get("error"):+ continue+ score = float(blast.get("blast_radius_score", 0.0))+ if score >= _BLAST_HIGH:+ high_blast.append(+ {+ **blast,+ "label": pivot["label"],+ "name": pivot.get("name", pivot["entity_id"]),+ "entity_id": pivot["entity_id"],+ "entity_type": pivot["entity_type"],+ }+ )++ if high_blast:+ for b in high_blast:+ state.add_finding(+ f"High blast radius from {b['label']} '{b['name']}': "+ f"{b['total_affected']} entities reachable "+ f"(score={b['blast_radius_score']:.2f}, "+ f"breakdown={dict(b.get('type_breakdown', {}))})."+ )++ # Propose an isolation action for the highest-scoring pivot.+ top = max(high_blast, key=lambda b: b["blast_radius_score"])+ if top["label"] in ("Host", "User"):+ state.proposed_actions.append(+ ProposedAction(+ action_type=("isolate_host" if top["label"] == "Host" else "disable_user"),+ description=(+ f"Isolate {top['label'].lower()} '{top['name']}' — "+ f"{top['total_affected']} downstream entities at risk."+ ),+ risk_level=ActionRisk.HIGH,+ target=top["name"],+ requires_approval=True,+ rationale=(+ f"Graph blast radius {top['blast_radius_score']:.2f} "+ f"with {top['total_affected']} reachable entities."+ ),+ parameters={+ "entity_type": top["entity_type"],+ "entity_id": top["entity_id"],+ "blast_radius_score": top["blast_radius_score"],+ },+ )+ )++ # Persist a compact summary on the state so downstream services (UI,+ # report writer) can render it without re-querying the graph.+ state.threat_intel.setdefault("attack_path", {})+ state.threat_intel["attack_path"] = {+ "case_id": case_id,+ "node_count": path.get("node_count", len(nodes)),+ "edge_count": path.get("edge_count", len(edges)),+ "tactic_chain": tactic_chain,+ "label_counts": dict(label_counts),+ "high_blast_entities": [+ {+ "label": b["label"],+ "name": b["name"],+ "score": b["blast_radius_score"],+ "total_affected": b["total_affected"],+ }+ for b in high_blast+ ],+ }++ # Don't override a terminal status set by the previous node, but if the+ # workflow is still running, flag completion now.+ if state.status == AgentStatus.RUNNING:+ state.status = AgentStatus.COMPLETED++ logger.info(+ "Attack-path complete",+ case_id=case_id,+ node_count=len(nodes),+ edge_count=len(edges),+ high_blast=len(high_blast),+ )+ return state+++# ─── Helpers ──────────────────────────────────────────────────────────────────+++def _resolve_case_id(state: InvestigationState) -> str | None:+ """Find the case id on the state, tolerant of where the orchestrator put it."""+ raw = state.raw_alert or {}+ return (+ raw.get("case_id")+ or raw.get("caseId")+ or (state.threat_intel or {}).get("case_id")+ )+++def _extract_tactic_chain(nodes: list[dict[str, Any]]) -> list[str]:+ """Pull MITRE tactic names from Technique nodes, preserving first-seen order."""+ seen: list[str] = []+ for node in nodes:+ if node.get("label") != "Technique":+ continue+ props = node.get("properties", {}) or {}+ tactic = props.get("tactic_name") or props.get("tactic")+ if tactic and tactic not in seen:+ seen.append(tactic)+ return seen+++def _select_blast_targets(nodes: list[dict[str, Any]]) -> list[dict[str, Any]]:+ """Rank Host/User/IOC nodes for blast-radius traversal.++ Hosts and users with high criticality / risk_score are evaluated first;+ IOCs are included as a tail because they're cheaper to score but rarely+ surface new pivot points the existing investigation hasn't already used.+ """+ candidates: list[tuple[float, dict[str, Any]]] = []+ for node in nodes:+ label = node.get("label")+ props = node.get("properties", {}) or {}+ nid = node.get("id") or props.get("id") or props.get("value")+ if not nid:+ continue++ if label == "Host":+ crit = (props.get("criticality") or "medium").lower()+ score = {"critical": 1.0, "high": 0.8, "medium": 0.5, "low": 0.2}.get(crit, 0.5)+ candidates.append(+ (+ score,+ {+ "entity_type": "host",+ "entity_id": nid,+ "label": "Host",+ "name": props.get("hostname", nid),+ },+ )+ )+ elif label == "User":+ risk = float(props.get("risk_score", 0.0) or 0.0)+ candidates.append(+ (+ min(1.0, risk + 0.3),+ {+ "entity_type": "user",+ "entity_id": nid,+ "label": "User",+ "name": props.get("username", nid),+ },+ )+ )+ elif label == "IOC":+ candidates.append(+ (+ 0.2,+ {+ "entity_type": "ioc",+ "entity_id": nid,+ "label": "IOC",+ "name": props.get("value", nid),+ },+ )+ )++ candidates.sort(key=lambda x: x[0], reverse=True)+ return [c[1] for c in candidates]diff --git a/services/agents/app/graph/workflow.py b/services/agents/app/graph/workflow.pyindex 3545cf04e..fc9acdda6 100644--- a/services/agents/app/graph/workflow.py+++ b/services/agents/app/graph/workflow.py@@ -1,9 +1,11 @@"""-LangGraph workflow: wires Auto-Triage → Triage → Enrichment → Investigation agents.+LangGraph workflow: wires Auto-Triage → Triage → Enrichment → Investigation+→ Attack-Path agents.Auto-triage runs first. If the LLM classifies the alert as FP/benign withconfidence above the auto-close threshold the graph terminates early.-Otherwise the alert flows through the full manual pipeline.+Otherwise the alert flows through the full manual pipeline, ending with the+graph-aware Attack-Path agent that walks Neo4j to compute blast radius."""from __future__ import annotations@@ -11,6 +13,7 @@ from __future__ import annotationsimport structlogfrom langgraph.graph import END, StateGraph+from app.agents.attack_path_agent import run_attack_pathfrom app.agents.auto_triage_agent import run_auto_triagefrom app.agents.enrichment_agent import run_enrichmentfrom app.agents.investigation_agent import run_investigation@@ -55,6 +58,12 @@ async def investigation_node(state: dict) -> dict:return s.to_dict()+async def attack_path_node(state: dict) -> dict:+ s = _from_dict(state)+ s = await run_attack_path(s)+ return s.to_dict()++def _should_continue(state: dict) -> str:"""Conditional edge: stop if max iterations reached or status is terminal."""s = _from_dict(state)@@ -78,7 +87,8 @@ def build_investigation_graph() -> StateGraph:Flow:auto_triage ─┬─ (high-confidence FP/benign) ──► END- └─ (else) ──► triage ──► enrichment ──► investigation ──► END+ └─ (else) ──► triage ──► enrichment ──► investigation+ ──► attack_path ──► END"""graph = StateGraph(dict)@@ -86,6 +96,7 @@ def build_investigation_graph() -> StateGraph:graph.add_node("triage", triage_node)graph.add_node("enrichment", enrichment_node)graph.add_node("investigation", investigation_node)+ graph.add_node("attack_path", attack_path_node)graph.set_entry_point("auto_triage")@@ -96,7 +107,8 @@ def build_investigation_graph() -> StateGraph:)graph.add_edge("triage", "enrichment")graph.add_edge("enrichment", "investigation")- graph.add_edge("investigation", END)+ graph.add_edge("investigation", "attack_path")+ graph.add_edge("attack_path", END)return graph.compile()diff --git a/services/agents/app/tools/graph.py b/services/agents/app/tools/graph.pynew file mode 100644index 000000000..494ac73cc--- /dev/null+++ b/services/agents/app/tools/graph.py@@ -0,0 +1,113 @@+"""+Tool: knowledge-graph queries via the API service.++Wraps the `/api/v1/graph/*` endpoints exposed by `services/api`. The agents+service uses these to walk attack paths, compute blast-radius severity, and+discover the immediate neighbourhood of an entity during an investigation.+"""++from __future__ import annotations++import os+from typing import Any++import httpx+import structlog++logger = structlog.get_logger()++_API_URL = os.getenv("API_SERVICE_URL", "http://api:8000")+_TIMEOUT = float(os.getenv("AGENTS_API_TIMEOUT", "10.0"))+++def _headers(api_token: str | None) -> dict[str, str]:+ return {"Authorization": f"Bearer {api_token}"} if api_token else {}+++async def get_attack_path(+ case_id: str,+ api_token: str | None = None,+ max_depth: int = 6,+) -> dict[str, Any]:+ """Return the Case → Alert → Host/User → IOC → Technique attack path."""+ try:+ async with httpx.AsyncClient(timeout=_TIMEOUT) as client:+ resp = await client.get(+ f"{_API_URL}/api/v1/graph/attack-path/{case_id}",+ params={"max_depth": max_depth},+ headers=_headers(api_token),+ )+ resp.raise_for_status()+ return resp.json()+ except httpx.HTTPStatusError as exc:+ if exc.response.status_code == 404:+ return {"case_id": case_id, "nodes": [], "edges": [], "node_count": 0, "edge_count": 0}+ logger.warning("attack_path query failed", case_id=case_id, error=str(exc))+ return {"error": str(exc), "case_id": case_id, "nodes": [], "edges": []}+ except Exception as exc: # noqa: BLE001+ logger.warning("attack_path query failed", case_id=case_id, error=str(exc))+ return {"error": str(exc), "case_id": case_id, "nodes": [], "edges": []}+++async def get_blast_radius(+ entity_type: str,+ entity_id: str,+ api_token: str | None = None,+ hops: int = 3,+) -> dict[str, Any]:+ """Compute the blast radius starting from a Host/User/IOC/Alert node."""+ try:+ async with httpx.AsyncClient(timeout=_TIMEOUT) as client:+ resp = await client.get(+ f"{_API_URL}/api/v1/graph/blast-radius/{entity_type}/{entity_id}",+ params={"hops": hops},+ headers=_headers(api_token),+ )+ resp.raise_for_status()+ return resp.json()+ except Exception as exc: # noqa: BLE001+ logger.warning(+ "blast_radius query failed",+ entity_type=entity_type,+ entity_id=entity_id,+ error=str(exc),+ )+ return {+ "error": str(exc),+ "entity_id": entity_id,+ "entity_type": entity_type,+ "affected_nodes": [],+ "total_affected": 0,+ "type_breakdown": {},+ "blast_radius_score": 0.0,+ }+++async def get_entity_neighbors(+ entity_type: str,+ entity_id: str,+ api_token: str | None = None,+) -> dict[str, Any]:+ """Return all nodes directly connected (depth 1) to the specified entity."""+ try:+ async with httpx.AsyncClient(timeout=_TIMEOUT) as client:+ resp = await client.get(+ f"{_API_URL}/api/v1/graph/neighbors/{entity_type}/{entity_id}",+ headers=_headers(api_token),+ )+ resp.raise_for_status()+ return resp.json()+ except Exception as exc: # noqa: BLE001+ logger.warning(+ "entity_neighbors query failed",+ entity_type=entity_type,+ entity_id=entity_id,+ error=str(exc),+ )+ return {+ "error": str(exc),+ "entity_id": entity_id,+ "entity_type": entity_type,+ "neighbors": [],+ "neighbor_count": 0,+ }diff --git a/services/agents/tests/test_attack_path_agent.py b/services/agents/tests/test_attack_path_agent.pynew file mode 100644index 000000000..371deb69a--- /dev/null+++ b/services/agents/tests/test_attack_path_agent.py@@ -0,0 +1,156 @@+"""Tests for the Attack-Path Investigation Agent.++These tests do not require a live Neo4j or API service; they monkeypatch the+graph-tool calls so the agent can be exercised in isolation. The intent is to+lock down:++* findings / proposed_actions are populated only when the graph returns data+* tactic chain extraction preserves order and deduplicates+* high blast-radius hosts produce an isolation `ProposedAction`+* the agent stays silent when no `case_id` is on state (no false findings)+"""++from __future__ import annotations++import sys+from pathlib import Path+from uuid import uuid4++import pytest++_AGENTS_ROOT = Path(__file__).resolve().parents[1]+if str(_AGENTS_ROOT) not in sys.path:+ sys.path.insert(0, str(_AGENTS_ROOT))++# Imports must come after path mutation.+from app.agents import attack_path_agent # noqa: E402+from app.models.state import ( # noqa: E402+ ActionRisk,+ AgentStatus,+ InvestigationState,+)+++def _state(case_id: str | None = "case-123") -> InvestigationState:+ return InvestigationState(+ incident_id=uuid4(),+ tenant_id=uuid4(),+ alert_summary="EDR detection",+ raw_alert={"case_id": case_id} if case_id else {},+ status=AgentStatus.RUNNING,+ )+++async def test_attack_path_skipped_when_no_case_id(monkeypatch: pytest.MonkeyPatch) -> None:+ called = {"path": False, "blast": False}++ async def _fake_path(*_a, **_kw):+ called["path"] = True+ return {"nodes": [], "edges": []}++ async def _fake_blast(*_a, **_kw):+ called["blast"] = True+ return {}++ monkeypatch.setattr(attack_path_agent, "get_attack_path", _fake_path)+ monkeypatch.setattr(attack_path_agent, "get_blast_radius", _fake_blast)++ state = _state(case_id=None)+ result = await attack_path_agent.run_attack_path(state)++ assert called["path"] is False+ assert called["blast"] is False+ assert result.findings == []+ assert result.proposed_actions == []+++async def test_attack_path_records_summary_and_tactic_chain(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ nodes = [+ {"id": "case-123", "label": "Case", "properties": {}},+ {"id": "alert-1", "label": "Alert", "properties": {}},+ {+ "id": "host-1",+ "label": "Host",+ "properties": {"hostname": "web-01", "criticality": "high"},+ },+ {+ "id": "tech-T1059",+ "label": "Technique",+ "properties": {"tactic_name": "Execution"},+ },+ {+ "id": "tech-T1071",+ "label": "Technique",+ "properties": {"tactic_name": "C2"},+ },+ # Duplicate tactic should not appear twice in the chain.+ {+ "id": "tech-T1105",+ "label": "Technique",+ "properties": {"tactic_name": "C2"},+ },+ ]+ edges = [{"from": "case-123", "to": "alert-1", "type": "HAS_ALERT"}] * 5++ async def _fake_path(case_id: str, **_kw):+ assert case_id == "case-123"+ return {"nodes": nodes, "edges": edges, "node_count": 6, "edge_count": 5}++ async def _fake_blast(entity_type: str, entity_id: str, **_kw):+ # Only the high-criticality host should be queried first; return a+ # high-blast result for it.+ if entity_type == "host" and entity_id == "host-1":+ return {+ "blast_radius_score": 0.82,+ "total_affected": 17,+ "type_breakdown": {"User": 4, "IOC": 13},+ "affected_nodes": [],+ }+ return {"blast_radius_score": 0.1, "total_affected": 0, "type_breakdown": {}}++ monkeypatch.setattr(attack_path_agent, "get_attack_path", _fake_path)+ monkeypatch.setattr(attack_path_agent, "get_blast_radius", _fake_blast)++ state = _state()+ result = await attack_path_agent.run_attack_path(state)++ assert any("Attack-path graph" in f for f in result.findings)+ assert any("Kill-chain stages reached" in f for f in result.findings)+ assert any("High blast radius from Host 'web-01'" in f for f in result.findings)++ chain = result.threat_intel["attack_path"]["tactic_chain"]+ assert chain == ["Execution", "C2"], "duplicate tactics should be deduplicated in order"++ isolate_actions = [a for a in result.proposed_actions if a.action_type == "isolate_host"]+ assert len(isolate_actions) == 1+ action = isolate_actions[0]+ assert action.target == "web-01"+ assert action.risk_level == ActionRisk.HIGH+ assert action.requires_approval is True+ assert action.parameters["entity_type"] == "host"+ assert action.parameters["blast_radius_score"] == pytest.approx(0.82)+++async def test_attack_path_no_findings_for_empty_graph(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ async def _fake_path(*_a, **_kw):+ return {"nodes": [], "edges": [], "node_count": 0, "edge_count": 0}++ async def _fake_blast(*_a, **_kw):+ # Should never be called when the graph is empty.+ raise AssertionError("blast radius queried for empty graph")++ monkeypatch.setattr(attack_path_agent, "get_attack_path", _fake_path)+ monkeypatch.setattr(attack_path_agent, "get_blast_radius", _fake_blast)++ state = _state()+ result = await attack_path_agent.run_attack_path(state)++ assert any("Attack-path graph empty" in f for f in result.findings)+ assert result.proposed_actions == []diff --git a/services/api/app/api/v1/endpoints/connectors.py b/services/api/app/api/v1/endpoints/connectors.pyindex 4014478b6..24c8383c1 100644--- a/services/api/app/api/v1/endpoints/connectors.py+++ b/services/api/app/api/v1/endpoints/connectors.py@@ -105,7 +105,15 @@ class ConnectorResponse(BaseModel):last_health_check: datetime | Nonelast_sync: datetime | Noneevents_ingested: int+ events_dropped: int = 0error_count: int+ # Schema-drift sentinel state surfaced to the wizard so the+ # connector card can show "schema changed at <ts>" without a+ # second round-trip. Both fields are nullable for connectors+ # that haven't drifted (the steady-state expectation).+ schema_fingerprint: str | None = None+ last_schema_drift_at: datetime | None = None+ last_drift_details: dict[str, Any] | None = Nonetags: listcreated_at: datetimeupdated_at: datetime@@ -113,6 +121,25 @@ class ConnectorResponse(BaseModel):model_config = {"from_attributes": True}+class ConnectorHealthSummary(BaseModel):+ """Tenant-wide rollup of connector health for the dashboard.++ The web console shows a small "Connector Health" tile at the top+ of the Connectors page; this endpoint feeds that tile so the UI+ can render it without iterating the full list client-side.+ """++ total: int+ healthy: int+ unhealthy: int+ unknown: int+ drifted: int+ drifted_recently: int # last 24 h+ last_drift_at: datetime | None+ total_events_ingested: int+ total_events_dropped: int++class CreateConnectorRequest(BaseModel):name: str = Field(min_length=1, max_length=255)connector_type: str = Field(min_length=1, max_length=100)@@ -304,6 +331,73 @@ async def list_catalog(return {"connectors": schemas}+# ------------------------------------------------------------------ health summary++[email protected]("/health", response_model=ConnectorHealthSummary)+async def connector_health_summary(+ current_user: Annotated[AuthUser, Depends(require_permission("connectors:read"))],+ db: DBSession,+) -> ConnectorHealthSummary:+ """Aggregate per-tenant connector health for the dashboard tile.++ Single SELECT scan; we deliberately do *not* group in SQL because+ Postgres' filter expressions over a few hundred rows are negligible+ compared to the round-trip, and the resulting Python is a lot easier+ to read than five COUNT(*) FILTER clauses.+ """+ result = await db.execute(+ select(Connector).where(Connector.tenant_id == current_user.tenant_id)+ )+ connectors = list(result.scalars().all())++ now = datetime.now(UTC)+ twenty_four_hours_ago_seconds = 24 * 60 * 60++ total = len(connectors)+ healthy = 0+ unhealthy = 0+ unknown = 0+ drifted = 0+ drifted_recently = 0+ last_drift_at: datetime | None = None+ total_events_ingested = 0+ total_events_dropped = 0++ for c in connectors:+ if c.health_status == "healthy":+ healthy += 1+ elif c.health_status == "unhealthy":+ unhealthy += 1+ else:+ unknown += 1++ total_events_ingested += int(c.events_ingested or 0)+ total_events_dropped += int(getattr(c, "events_dropped", 0) or 0)++ drift_at = getattr(c, "last_schema_drift_at", None)+ if drift_at is not None:+ drifted += 1+ if last_drift_at is None or drift_at > last_drift_at:+ last_drift_at = drift_at+ # Compare in seconds so we don't depend on timedelta arithmetic+ # being available on every datetime variant Postgres can return.+ if (now - drift_at).total_seconds() <= twenty_four_hours_ago_seconds:+ drifted_recently += 1++ return ConnectorHealthSummary(+ total=total,+ healthy=healthy,+ unhealthy=unhealthy,+ unknown=unknown,+ drifted=drifted,+ drifted_recently=drifted_recently,+ last_drift_at=last_drift_at,+ total_events_ingested=total_events_ingested,+ total_events_dropped=total_events_dropped,+ )++# ------------------------------------------------------------------ pre-save testdiff --git a/services/api/app/models/connector.py b/services/api/app/models/connector.pyindex 6f6a655a7..7aa38e6fb 100644--- a/services/api/app/models/connector.py+++ b/services/api/app/models/connector.py@@ -3,7 +3,7 @@import uuidfrom datetime import UTC, datetime-from sqlalchemy import Boolean, DateTime, ForeignKey, String+from sqlalchemy import Boolean, DateTime, ForeignKey, String, Textfrom sqlalchemy.dialects.postgresql import JSONB, UUIDfrom sqlalchemy.orm import Mapped, mapped_column, relationship@@ -25,7 +25,13 @@ class Connector(Base):last_health_check: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)last_sync: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)events_ingested: Mapped[int] = mapped_column(default=0)+ events_dropped: Mapped[int] = mapped_column(default=0)error_count: Mapped[int] = mapped_column(default=0)+ # Schema-drift sentinel state (migration 026). NULL until the first+ # non-empty poll has populated a fingerprint baseline.+ schema_fingerprint: Mapped[str | None] = mapped_column(Text, nullable=True)+ last_schema_drift_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)+ last_drift_details: Mapped[dict | None] = mapped_column(JSONB, nullable=True)tags: Mapped[list] = mapped_column(JSONB, default=list)created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(UTC))updated_at: Mapped[datetime] = mapped_column(diff --git a/services/api/migrations/026_connector_schema_drift.sql b/services/api/migrations/026_connector_schema_drift.sqlnew file mode 100644index 000000000..bf1386453--- /dev/null+++ b/services/api/migrations/026_connector_schema_drift.sql@@ -0,0 +1,47 @@+-- 026_connector_schema_drift.sql+-- Connector Health & Schema-Drift Sentinel+--+-- Adds three columns to the `connectors` table so the polling scheduler can+-- record schema fingerprints from each connector's most recent payload and+-- raise an alert (drift) when the field set changes between polls.+--+-- schema_fingerprint : SHA-256 hex of the sorted unique top-level+-- field names observed in the most recent batch.+-- NULL until the connector has produced its+-- first non-empty batch.+--+-- last_schema_drift_at : Timestamp of the most recent confirmed drift+-- (fingerprint changed vs prior baseline).+--+-- last_drift_details : JSONB describing what changed — added fields,+-- removed fields, sample event count, etc.+-- Surfaced in the UI so an operator can decide+-- whether the upstream API legitimately changed+-- or whether a parser is breaking.+--+-- events_dropped : Counter incremented when pre-ingest filter+-- rules (Security Data Pipeline feature) drop+-- an event before it reaches the ingest service.+--+-- This migration is additive and idempotent; running it twice is safe.++BEGIN;++ALTER TABLE connectors+ ADD COLUMN IF NOT EXISTS schema_fingerprint TEXT,+ ADD COLUMN IF NOT EXISTS last_schema_drift_at TIMESTAMPTZ,+ ADD COLUMN IF NOT EXISTS last_drift_details JSONB,+ ADD COLUMN IF NOT EXISTS events_dropped INTEGER NOT NULL DEFAULT 0;++-- Index on health_status so the new health-summary endpoint can group+-- counts in a single scan instead of a tenant-wide table scan.+CREATE INDEX IF NOT EXISTS idx_connectors_health_status+ ON connectors(tenant_id, health_status);++-- Index on last_schema_drift_at to surface "drifted in the last 24 h"+-- rows quickly on the dashboard.+CREATE INDEX IF NOT EXISTS idx_connectors_last_drift+ ON connectors(tenant_id, last_schema_drift_at)+ WHERE last_schema_drift_at IS NOT NULL;++COMMIT;diff --git a/services/connectors/app/db/connector_repo.py b/services/connectors/app/db/connector_repo.pyindex 86ac5cb2c..fe16b7378 100644--- a/services/connectors/app/db/connector_repo.py+++ b/services/connectors/app/db/connector_repo.py@@ -29,6 +29,7 @@ from sqlalchemy import (MetaData,String,Table,+ Text,select,update,)@@ -54,7 +55,13 @@ connectors_table = Table(Column("last_health_check", DateTime(timezone=True)),Column("last_sync", DateTime(timezone=True)),Column("events_ingested", Integer, nullable=False),+ Column("events_dropped", Integer, nullable=False),Column("error_count", Integer, nullable=False),+ # Schema-drift sentinel columns (migration 026). Nullable: a brand-new+ # connector hasn't established a fingerprint baseline yet.+ Column("schema_fingerprint", Text),+ Column("last_schema_drift_at", DateTime(timezone=True)),+ Column("last_drift_details", JSONB),Column("tags", JSON, nullable=False),Column("created_at", DateTime(timezone=True), nullable=False),Column("updated_at", DateTime(timezone=True), nullable=False),@@ -80,7 +87,14 @@ class ConnectorInstance:health_status: strlast_sync: datetime | Noneevents_ingested: int+ events_dropped: interror_count: int+ # Drift-sentinel state. ``schema_fingerprint`` is the SHA-256 of the+ # sorted, deduped top-level field names from the previous successful+ # poll. NULL on first poll.+ schema_fingerprint: str | None+ last_schema_drift_at: datetime | None+ last_drift_details: dict[str, Any] | Noneasync def fetch_enabled_connectors(connection: Any) -> list[ConnectorInstance]:@@ -101,7 +115,11 @@ async def fetch_enabled_connectors(connection: Any) -> list[ConnectorInstance]:connectors_table.c.health_status,connectors_table.c.last_sync,connectors_table.c.events_ingested,+ connectors_table.c.events_dropped,connectors_table.c.error_count,+ connectors_table.c.schema_fingerprint,+ connectors_table.c.last_schema_drift_at,+ connectors_table.c.last_drift_details,).where(connectors_table.c.is_enabled.is_(True))result = await connection.execute(stmt)@@ -118,7 +136,11 @@ async def fetch_enabled_connectors(connection: Any) -> list[ConnectorInstance]:health_status=row.health_status,last_sync=row.last_sync,events_ingested=row.events_ingested,+ events_dropped=row.events_dropped,error_count=row.error_count,+ schema_fingerprint=row.schema_fingerprint,+ last_schema_drift_at=row.last_schema_drift_at,+ last_drift_details=row.last_drift_details,)for row in rows]@@ -129,19 +151,31 @@ async def record_poll_success(connector_id: uuid.UUID,*,events_added: int,+ events_dropped: int = 0,+ schema_fingerprint: str | None = None,) -> None:- """Update last_sync, increment events_ingested, mark healthy."""+ """Update last_sync, increment counters, mark healthy.++ If ``schema_fingerprint`` is provided it is written to the row; the+ drift bookkeeping (``last_schema_drift_at`` / ``last_drift_details``)+ is updated by ``record_schema_drift`` so the two callers don't race+ on the same UPDATE.+ """now = datetime.now(UTC)+ values: dict[str, Any] = {+ "last_sync": now,+ "last_health_check": now,+ "health_status": "healthy",+ "events_ingested": connectors_table.c.events_ingested + events_added,+ "events_dropped": connectors_table.c.events_dropped + events_dropped,+ "updated_at": now,+ }+ if schema_fingerprint is not None:+ values["schema_fingerprint"] = schema_fingerprintstmt = (update(connectors_table).where(connectors_table.c.id == connector_id)- .values(- last_sync=now,- last_health_check=now,- health_status="healthy",- events_ingested=connectors_table.c.events_ingested + events_added,- updated_at=now,- )+ .values(**values))await connection.execute(stmt)@@ -165,6 +199,33 @@ async def record_poll_failure(await connection.execute(stmt)+async def record_schema_drift(+ connection: Any,+ connector_id: uuid.UUID,+ *,+ fingerprint: str,+ details: dict[str, Any],+) -> None:+ """Record a confirmed schema drift event.++ Called by the scheduler when the new poll's fingerprint differs from+ the stored baseline. Writes the new fingerprint, drift timestamp and+ details so the dashboard can surface "what changed" to operators.+ """+ now = datetime.now(UTC)+ stmt = (+ update(connectors_table)+ .where(connectors_table.c.id == connector_id)+ .values(+ schema_fingerprint=fingerprint,+ last_schema_drift_at=now,+ last_drift_details=details,+ updated_at=now,+ )+ )+ await connection.execute(stmt)++__all__ = ["ConnectorInstance","connectors_table",@@ -172,4 +233,5 @@ __all__ = ["metadata","record_poll_failure","record_poll_success",+ "record_schema_drift",]diff --git a/services/connectors/app/pipeline/__init__.py b/services/connectors/app/pipeline/__init__.pynew file mode 100644index 000000000..c29593fa4--- /dev/null+++ b/services/connectors/app/pipeline/__init__.py@@ -0,0 +1,28 @@+"""Pre-ingest pipeline: schema fingerprinting and event filtering.++This package owns the *connector-side* half of two related features:++* **Connector Health & Schema-Drift Sentinel** — fingerprints each batch+ of normalized events so the scheduler can detect when an upstream API+ starts returning a different field set (silent integration breakage).++* **Security Data Pipeline + Tiered Retention** — applies a list of+ declarative filter rules (drop / route) before events leave the+ connectors microservice, so noisy events never reach the hot tier.++Both helpers are pure functions that take a list of events and return+something useful. They never touch the database, the network, or the+scheduler state — that's the scheduler's job.+"""++from __future__ import annotations++from app.pipeline.fingerprint import compute_fingerprint, diff_fingerprints+from app.pipeline.filter_rules import FilterDecision, apply_filter_rules++__all__ = [+ "FilterDecision",+ "apply_filter_rules",+ "compute_fingerprint",+ "diff_fingerprints",+]diff --git a/services/connectors/app/pipeline/filter_rules.py b/services/connectors/app/pipeline/filter_rules.pynew file mode 100644index 000000000..7b6ea9048--- /dev/null+++ b/services/connectors/app/pipeline/filter_rules.py@@ -0,0 +1,151 @@+"""Pre-ingest filter rules — Security Data Pipeline (MVP).++The data-pipeline feature is a "drop or keep before it costs you anything"+layer: every connector instance can carry a small list of declarative rules+in its ``connector_config["filter_rules"]`` array. The scheduler runs these+rules over each normalized batch *before* shipping events to the ingest+service, so a noisy informational alert can be discarded at the connector+edge instead of consuming hot-tier storage and triage capacity.++Rule shape (kept intentionally tiny so it's safe to author by hand):++.. code-block:: yaml++ filter_rules:+ - field: severity+ op: eq+ value: info+ action: drop+ - field: rule_name+ op: contains+ value: "test alert"+ action: drop++The same operator vocabulary as ``federated.query`` is reused so SOC+operators don't have to learn two filter dialects.++Supported actions:++* ``drop`` — discard the event entirely. It never reaches ingest.+* ``keep`` — explicitly retain (used to override a later catch-all drop).++Anything that doesn't match any rule is kept by default. Rules are+evaluated top-to-bottom; first match wins. This is the same semantics+``iptables`` and most firewall vendors use, so it's familiar and+debuggable.++The function is pure — it does not log, ingest, or mutate the events.+The scheduler is responsible for incrementing ``events_dropped`` and for+writing a structured log line per drop so an operator can audit which+events were filtered.+"""++from __future__ import annotations++from dataclasses import dataclass+from typing import Any++# Operator vocabulary mirrors federated/query.py to avoid teaching the+# user two slightly different dialects of "this field equals that value".+_VALID_OPS: frozenset[str] = frozenset(+ {"eq", "ne", "contains", "starts_with", "ends_with", "in"}+)+_VALID_ACTIONS: frozenset[str] = frozenset({"drop", "keep"})+++@dataclass(slots=True)+class FilterDecision:+ """Result of running rules over a single event.++ ``action`` is always either ``"drop"`` or ``"keep"`` (never None) so+ the caller can branch unambiguously. ``rule_index`` is the 0-based+ index of the rule that matched, or ``None`` if the default+ keep-everything fallback fired. ``rule_index`` is logged so an+ operator can audit which rule discarded which event.+ """++ action: str+ rule_index: int | None+++def _matches(event: dict[str, Any], field: str, op: str, value: Any) -> bool:+ """Evaluate a single ``field <op> value`` predicate against an event.++ Missing fields are treated as a non-match (return False) for every+ operator. We don't raise: a poorly authored rule referencing a field+ that an upstream vendor renamed shouldn't blow up the polling loop.+ """+ actual = event.get(field)+ if actual is None:+ return False++ # Cheapest comparisons first.+ if op == "eq":+ return actual == value+ if op == "ne":+ return actual != value+ if op == "in":+ # Caller is expected to pass a list/tuple/set for `value`. If they+ # passed a scalar, fall back to `==` so we don't crash.+ if isinstance(value, list | tuple | set):+ return actual in value+ return actual == value++ # String operators — coerce to str so "severity == 5" with a string+ # rule still does the obvious thing.+ actual_str = str(actual)+ value_str = str(value)+ if op == "contains":+ return value_str.lower() in actual_str.lower()+ if op == "starts_with":+ return actual_str.lower().startswith(value_str.lower())+ if op == "ends_with":+ return actual_str.lower().endswith(value_str.lower())++ # Unknown operator: refuse to drop. Better to over-keep than to silently+ # discard events because someone fat-fingered the operator.+ return False+++def apply_filter_rules(+ event: dict[str, Any],+ rules: list[dict[str, Any]] | None,+) -> FilterDecision:+ """Evaluate the rule list against one event and return the decision.++ Iterates rules top-to-bottom; the first rule whose predicate matches+ decides the action (drop or keep). If no rule matches, the default+ is to keep the event — the scheduler must not silently discard+ everything just because a tenant left their rule list empty.++ Malformed rules (missing keys, bad operator, bad action) are skipped+ silently. We log noisily about this elsewhere (the scheduler emits a+ structured warning the first time a malformed rule is seen) so the+ user gets visibility without each evaluation costing log spam.+ """+ if not rules:+ return FilterDecision(action="keep", rule_index=None)++ for index, rule in enumerate(rules):+ if not isinstance(rule, dict):+ continue+ field = rule.get("field")+ op = rule.get("op")+ value = rule.get("value")+ action = rule.get("action", "drop")++ if not isinstance(field, str) or not field:+ continue+ if op not in _VALID_OPS:+ continue+ if action not in _VALID_ACTIONS:+ continue++ if _matches(event, field, op, value):+ return FilterDecision(action=action, rule_index=index)++ # No rule matched: default keep.+ return FilterDecision(action="keep", rule_index=None)+++__all__ = ["FilterDecision", "apply_filter_rules"]diff --git a/services/connectors/app/pipeline/fingerprint.py b/services/connectors/app/pipeline/fingerprint.pynew file mode 100644index 000000000..205210d6e--- /dev/null+++ b/services/connectors/app/pipeline/fingerprint.py@@ -0,0 +1,98 @@+"""Schema fingerprinting for the Connector Health & Schema-Drift Sentinel.++We compute a stable hash of the *set of top-level field names* that appear+across a batch of normalized events. The hash is deliberately simple so that+two polls of the same upstream API produce the same fingerprint as long as+the field set is unchanged — even if values change, even if events arrive in+a different order, even if some events omit optional fields.++The intent is not to detect every possible upstream change (a value-type+change from string→int wouldn't be caught here, for example). The intent is+to catch the **single most common silent failure mode**: an upstream vendor+renames or removes a field, the connector's normalize() logic now produces+events missing critical attributes, and detections silently start firing+less frequently. Even this coarse fingerprint catches that case quickly.++Algorithm+---------++For a batch of N normalized events:++1. Union all top-level keys across all N events into a set.+2. Sort lexicographically (stable across runs, so the hash is stable).+3. Join with ``\\x1f`` (ASCII unit separator — guaranteed not to appear in+ a JSON field name).+4. SHA-256 the joined string and return the hex digest.++Two batches with identical field sets produce identical fingerprints.+A batch with one extra field (``new_field``) or one missing field produces+a different fingerprint, which the scheduler treats as drift.++Empty batches return ``None`` rather than a fingerprint of the empty set,+because we don't want a quiet hour (no events) to clobber the baseline+fingerprint we recorded the previous time the connector returned events.+"""++from __future__ import annotations++import hashlib+from collections.abc import Iterable+from typing import Any++# We do *not* recurse into nested dicts. The drift sentinel is intentionally+# scoped to top-level fields because (a) normalize() outputs are flat by+# convention in this project (``severity``, ``timestamp``, ``user``, etc.)+# and (b) hashing arbitrarily nested structures would catch too many false+# positives — every alert has different ``raw_event`` contents, but those+# differences don't represent schema drift in any meaningful sense.++_SEPARATOR = "\x1f"+++def compute_fingerprint(events: Iterable[dict[str, Any]]) -> str | None:+ """Return a stable SHA-256 hex digest of the union of top-level field names.++ Returns ``None`` if no events have any keys (e.g. an empty list, or a+ list of empty dicts). Returning ``None`` lets the caller distinguish+ "no events this poll" from "the upstream changed schema to the empty+ set", which is essentially never what we want to flag as drift.+ """+ keys: set[str] = set()+ for event in events:+ if not isinstance(event, dict):+ # Defensive: we only fingerprint structured events. A connector+ # that misbehaves and yields a plain string is the scheduler's+ # problem, not the fingerprinter's.+ continue+ keys.update(event.keys())++ if not keys:+ return None++ joined = _SEPARATOR.join(sorted(keys))+ digest = hashlib.sha256(joined.encode("utf-8")).hexdigest()+ return digest+++def diff_fingerprints(+ previous_keys: Iterable[str],+ current_keys: Iterable[str],+) -> dict[str, list[str]]:+ """Describe what changed between two field sets.++ Used by the scheduler to populate ``last_drift_details`` so the UI can+ show "added: X; removed: Y" instead of just "fingerprint changed".+ Both arguments are the raw key sets — the caller is responsible for+ pulling them from the most recent batch and the prior baseline batch+ (we cache the prior batch's keys in-memory in the scheduler).+ """+ prev = set(previous_keys)+ curr = set(current_keys)+ return {+ "added": sorted(curr - prev),+ "removed": sorted(prev - curr),+ "unchanged_count": len(prev & curr),+ }+++__all__ = ["compute_fingerprint", "diff_fingerprints"]diff --git a/services/connectors/app/scheduler.py b/services/connectors/app/scheduler.pyindex 1e65b1b71..94a310abd 100644--- a/services/connectors/app/scheduler.py+++ b/services/connectors/app/scheduler.py@@ -64,9 +64,15 @@ from app.db.connector_repo import (fetch_enabled_connectors,record_poll_failure,record_poll_success,+ record_schema_drift,)from app.db.engine import get_enginefrom app.ingest_client import IngestClient, IngestClientError+from app.pipeline import (+ apply_filter_rules,+ compute_fingerprint,+ diff_fingerprints,+)from app.security.credential_vault import CredentialVault, CredentialVaultError, get_vaultlogger = logging.getLogger("aisoc.connectors.scheduler")@@ -385,48 +391,174 @@ class ConnectorScheduler:# raw events still produces consistent shapes downstream.normalized = [_safe_normalize(connector, e) for e in raw_events]+ # ---- Security Data Pipeline: pre-ingest filter rules ----+ #+ # Rules live on the connector's ``connector_config["filter_rules"]``+ # array. ``apply_filter_rules`` is pure: it just decides drop/keep.+ # We count drops here so they show up in ``events_dropped`` on the+ # row and emit one info-level log per drop so an operator can audit+ # filter behavior without trawling the database.+ filter_rules = (target.connector_config or {}).get("filter_rules") or []+ kept: list[dict[str, Any]] = []+ dropped_count = 0+ for event in normalized:+ decision = apply_filter_rules(event, filter_rules)+ if decision.action == "drop":+ dropped_count += 1+ # Keep the log line short — high-volume connectors can drop+ # hundreds of events per poll, and we don't want the log to+ # become the bottleneck. The rule_index is enough to find+ # the offending rule in the connector config.+ logger.debug(+ "connector.scheduler.filter_drop id=%s rule_index=%s",+ connector_id,+ decision.rule_index,+ )+ continue+ kept.append(event)++ # ---- Schema-Drift Sentinel: fingerprint the kept batch ----+ #+ # We fingerprint *kept* events (post-filter) because the filter+ # rules are tenant-controlled and shouldn't be allowed to mask a+ # legitimate upstream schema change by dropping all instances of+ # the new field. Empty batches return None and we don't update+ # the baseline in that case (see fingerprint.compute_fingerprint).+ new_fingerprint = compute_fingerprint(kept)+ drift_detected = False+ drift_details: dict[str, Any] | None = None++ if new_fingerprint is not None and target.schema_fingerprint is not None:+ if new_fingerprint != target.schema_fingerprint:+ drift_detected = True+ # Recompute the union of keys so we can describe what+ # changed. We don't store prior keys in the row (only the+ # hash), so we approximate "removed" by diffing against+ # the current batch's keys vs. the keys set we'd expect+ # from the prior fingerprint. Since we don't have the+ # prior keys, we record the new key set and ``added`` is+ # the entire new set. This is good enough for the UI to+ # surface "schema changed at <timestamp>"; a future+ # iteration can persist the prior key list.+ current_keys = sorted({k for ev in kept for k in ev.keys()})+ drift_details = diff_fingerprints(+ previous_keys=[], # baseline keys not persisted yet+ current_keys=current_keys,+ )+ drift_details["sample_event_count"] = len(kept)+ drift_details["new_fingerprint"] = new_fingerprint+ drift_details["previous_fingerprint"] = target.schema_fingerprint+ logger.warning(+ "connector.scheduler.schema_drift id=%s type=%s added=%d",+ connector_id,+ target.connector_type,+ len(drift_details.get("added", [])),+ )+try:result = await self._ingest_client.push_events(tenant_id=target.tenant_id,connector_id=target.id,connector_type=target.connector_type,- events=normalized,+ events=kept,)except IngestClientError:logger.exception("connector.scheduler.ingest_failed id=%s events=%d",connector_id,- len(normalized),+ len(kept),)await self._record_failure(target.id)returnaccepted = int(result.get("accepted", 0) or 0)elapsed_ms = int((datetime.now(UTC) - started).total_seconds() * 1000)- await self._record_success(target.id, events_added=accepted)++ # If drift was detected, persist that fact *before* marking the+ # poll successful so the UI reflects "drifted" status alongside+ # "healthy". Both writes are independent transactions; the order+ # protects against a race where the operator views the row mid-+ # poll and sees a stale fingerprint.+ if drift_detected and drift_details is not None:+ await self._record_drift(+ target.id,+ fingerprint=new_fingerprint or "",+ details=drift_details,+ )++ await self._record_success(+ target.id,+ events_added=accepted,+ events_dropped=dropped_count,+ schema_fingerprint=new_fingerprint,+ )logger.info(- "connector.scheduler.poll_complete id=%s type=%s accepted=%d rejected=%d elapsed_ms=%d",+ "connector.scheduler.poll_complete id=%s type=%s accepted=%d rejected=%d dropped=%d drift=%s elapsed_ms=%d",connector_id,target.connector_type,accepted,int(result.get("rejected", 0) or 0),+ dropped_count,+ "yes" if drift_detected else "no",elapsed_ms,)# ------------------------------------------------------------------ db helpers- async def _record_success(self, connector_id: uuid.UUID, *, events_added: int) -> None:+ async def _record_success(+ self,+ connector_id: uuid.UUID,+ *,+ events_added: int,+ events_dropped: int = 0,+ schema_fingerprint: str | None = None,+ ) -> None:if self._engine is None: # pragma: no coverreturntry:async with self._engine.begin() as conn:- await record_poll_success(conn, connector_id, events_added=events_added)+ await record_poll_success(+ conn,+ connector_id,+ events_added=events_added,+ events_dropped=events_dropped,+ schema_fingerprint=schema_fingerprint,+ )except Exception: # pragma: no coverlogger.exception("connector.scheduler.record_success_failed id=%s",connector_id,)+ async def _record_drift(+ self,+ connector_id: uuid.UUID,+ *,+ fingerprint: str,+ details: dict[str, Any],+ ) -> None:+ """Persist a detected schema drift for the connector row.++ Kept separate from ``_record_success`` so a drift write that+ races with a future poll's success write doesn't clobber the+ ``last_schema_drift_at`` timestamp.+ """+ if self._engine is None: # pragma: no cover+ return+ try:+ async with self._engine.begin() as conn:+ await record_schema_drift(+ conn,+ connector_id,+ fingerprint=fingerprint,+ details=details,+ )+ except Exception: # pragma: no cover+ logger.exception(+ "connector.scheduler.record_drift_failed id=%s",+ connector_id,+ )+async def _record_failure(self, connector_id: uuid.UUID) -> None:if self._engine is None: # pragma: no coverreturndiff --git a/services/connectors/tests/test_pipeline_filter_rules.py b/services/connectors/tests/test_pipeline_filter_rules.pynew file mode 100644index 000000000..2f3687951--- /dev/null+++ b/services/connectors/tests/test_pipeline_filter_rules.py@@ -0,0 +1,130 @@+"""Tests for declarative pre-ingest filter rules.++These rules run on every normalized event before it leaves the connector+service. Getting the semantics right is critical: a buggy rule engine could+silently drop alerts that should have been triaged. The tests here pin down:++* default behaviour (no rules → keep)+* each operator (eq, ne, contains, starts_with, ends_with, in)+* drop vs. keep actions+* malformed rules don't crash and don't drop events+* missing fields don't match (so a renamed upstream field doesn't+ accidentally match every rule and discard everything)+"""++from __future__ import annotations++from app.pipeline.filter_rules import FilterDecision, apply_filter_rules+++def test_no_rules_keeps_event() -> None:+ decision = apply_filter_rules({"severity": "info"}, None)+ assert decision == FilterDecision(action="keep", rule_index=None)+++def test_empty_rules_keeps_event() -> None:+ decision = apply_filter_rules({"severity": "info"}, [])+ assert decision == FilterDecision(action="keep", rule_index=None)+++def test_eq_drops_matching_event() -> None:+ rules = [{"field": "severity", "op": "eq", "value": "info", "action": "drop"}]+ decision = apply_filter_rules({"severity": "info"}, rules)+ assert decision.action == "drop"+ assert decision.rule_index == 0+++def test_eq_keeps_non_matching_event() -> None:+ rules = [{"field": "severity", "op": "eq", "value": "info", "action": "drop"}]+ decision = apply_filter_rules({"severity": "high"}, rules)+ assert decision.action == "keep"+ assert decision.rule_index is None+++def test_contains_is_case_insensitive() -> None:+ rules = [{"field": "rule_name", "op": "contains", "value": "TEST", "action": "drop"}]+ decision = apply_filter_rules({"rule_name": "lab test alert"}, rules)+ assert decision.action == "drop"+++def test_starts_with_and_ends_with() -> None:+ starts = [{"field": "host", "op": "starts_with", "value": "lab-", "action": "drop"}]+ ends = [{"field": "host", "op": "ends_with", "value": "-stage", "action": "drop"}]++ assert apply_filter_rules({"host": "lab-web-01"}, starts).action == "drop"+ assert apply_filter_rules({"host": "prod-web-01"}, starts).action == "keep"+ assert apply_filter_rules({"host": "web-01-stage"}, ends).action == "drop"+++def test_in_operator_with_list() -> None:+ rules = [{"field": "severity", "op": "in", "value": ["info", "low"], "action": "drop"}]+ assert apply_filter_rules({"severity": "info"}, rules).action == "drop"+ assert apply_filter_rules({"severity": "low"}, rules).action == "drop"+ assert apply_filter_rules({"severity": "high"}, rules).action == "keep"+++def test_in_operator_with_scalar_falls_back_to_eq() -> None:+ rules = [{"field": "severity", "op": "in", "value": "info", "action": "drop"}]+ assert apply_filter_rules({"severity": "info"}, rules).action == "drop"+ assert apply_filter_rules({"severity": "low"}, rules).action == "keep"+++def test_ne_drops_when_field_differs() -> None:+ rules = [{"field": "tenant", "op": "ne", "value": "acme", "action": "drop"}]+ assert apply_filter_rules({"tenant": "evil-corp"}, rules).action == "drop"+ assert apply_filter_rules({"tenant": "acme"}, rules).action == "keep"+++def test_first_match_wins() -> None:+ rules = [+ {"field": "severity", "op": "eq", "value": "info", "action": "keep"},+ {"field": "severity", "op": "eq", "value": "info", "action": "drop"},+ ]+ decision = apply_filter_rules({"severity": "info"}, rules)+ assert decision.action == "keep"+ assert decision.rule_index == 0+++def test_keep_action_overrides_later_drop() -> None:+ rules = [+ {"field": "rule_name", "op": "contains", "value": "important", "action": "keep"},+ {"field": "severity", "op": "eq", "value": "info", "action": "drop"},+ ]+ decision = apply_filter_rules(+ {"rule_name": "important info alert", "severity": "info"},+ rules,+ )+ assert decision.action == "keep"+ assert decision.rule_index == 0+++def test_missing_field_does_not_match() -> None:+ # If a rule references a field the event doesn't have, the rule should+ # be ignored rather than matching (which would silently drop everything+ # when an upstream vendor renames a field).+ rules = [{"field": "definitely_missing", "op": "eq", "value": "x", "action": "drop"}]+ decision = apply_filter_rules({"severity": "high"}, rules)+ assert decision.action == "keep"+++def test_malformed_rule_skipped() -> None:+ rules = [+ "not a dict", # type: ignore[list-item]+ {"field": ""},+ {"field": "severity", "op": "bogus_op", "value": "info", "action": "drop"},+ {"field": "severity", "op": "eq", "value": "info", "action": "delete"},+ {"field": "severity", "op": "eq", "value": "info", "action": "drop"},+ ]+ decision = apply_filter_rules({"severity": "info"}, rules)+ # Only the last rule is well-formed; that's the one that should fire.+ assert decision.action == "drop"+ assert decision.rule_index == 4+++def test_string_op_coerces_non_string_actuals() -> None:+ # Some upstreams report severity as an int; rule authors usually write+ # the value as a string. Coercing both sides to str avoids the+ # surprise of "5".contains("5") not matching int 5.+ rules = [{"field": "severity", "op": "contains", "value": "5", "action": "drop"}]+ decision = apply_filter_rules({"severity": 5}, rules)+ assert decision.action == "drop"diff --git a/services/connectors/tests/test_pipeline_fingerprint.py b/services/connectors/tests/test_pipeline_fingerprint.pynew file mode 100644index 000000000..17ae8e36b--- /dev/null+++ b/services/connectors/tests/test_pipeline_fingerprint.py@@ -0,0 +1,79 @@+"""Tests for the schema-fingerprint module.++The fingerprint is the foundation of the Schema-Drift Sentinel: it has to be+**stable** across reorderings and value changes, and **distinct** when the set+of top-level keys actually changes. These tests lock those properties down so+the scheduler can safely use the digest as a primary-key for "schema we last+saw on this connector".+"""++from __future__ import annotations++from app.pipeline.fingerprint import compute_fingerprint, diff_fingerprints+++def test_fingerprint_stable_across_event_order() -> None:+ a = [{"severity": "high", "host": "h1"}, {"severity": "low", "host": "h2"}]+ b = list(reversed(a))+ assert compute_fingerprint(a) == compute_fingerprint(b)+++def test_fingerprint_stable_across_value_changes() -> None:+ a = [{"severity": "high", "host": "h1"}]+ b = [{"severity": "low", "host": "h2"}]+ # Same key set, different values — fingerprint must be identical.+ assert compute_fingerprint(a) == compute_fingerprint(b)+++def test_fingerprint_changes_when_field_added() -> None:+ baseline = [{"severity": "high"}]+ drifted = [{"severity": "high", "new_field": "x"}]+ assert compute_fingerprint(baseline) != compute_fingerprint(drifted)+++def test_fingerprint_changes_when_field_removed() -> None:+ baseline = [{"severity": "high", "user": "u1"}]+ drifted = [{"severity": "high"}]+ assert compute_fingerprint(baseline) != compute_fingerprint(drifted)+++def test_fingerprint_unions_keys_across_events() -> None:+ # The union of keys is what we hash, so an event missing 'user' shouldn't+ # change the fingerprint as long as some other event has it.+ a = [{"severity": "high", "user": "u1"}, {"severity": "low"}]+ b = [{"severity": "high"}, {"severity": "low", "user": "u1"}]+ assert compute_fingerprint(a) == compute_fingerprint(b)+++def test_fingerprint_empty_returns_none() -> None:+ assert compute_fingerprint([]) is None+ assert compute_fingerprint([{}]) is None+++def test_fingerprint_skips_non_dict_events() -> None:+ # A misbehaving connector returning a plain string shouldn't crash;+ # we just skip the offender and fingerprint the rest.+ a = [{"severity": "high"}, "garbage"] # type: ignore[list-item]+ b = [{"severity": "high"}]+ assert compute_fingerprint(a) == compute_fingerprint(b)+++def test_diff_added_and_removed() -> None:+ diff = diff_fingerprints({"severity", "host"}, {"severity", "user"})+ assert diff["added"] == ["user"]+ assert diff["removed"] == ["host"]+ assert diff["unchanged_count"] == 1+++def test_diff_unchanged() -> None:+ diff = diff_fingerprints({"severity", "host"}, {"severity", "host"})+ assert diff["added"] == []+ assert diff["removed"] == []+ assert diff["unchanged_count"] == 2+++def test_diff_handles_empty_previous() -> None:+ diff = diff_fingerprints(set(), {"severity", "host"})+ assert sorted(diff["added"]) == ["host", "severity"]+ assert diff["removed"] == []+ assert diff["unchanged_count"] == 0diff --git a/services/connectors/tests/test_scheduler.py b/services/connectors/tests/test_scheduler.pyindex 31c1c4b4b..cf24e4ce5 100644--- a/services/connectors/tests/test_scheduler.py+++ b/services/connectors/tests/test_scheduler.py@@ -43,6 +43,10 @@ def _make_instance(last_sync=None,events_ingested=0,error_count=0,+ events_dropped=0,+ schema_fingerprint=None,+ last_schema_drift_at=None,+ last_drift_details=None,)@@ -113,6 +117,236 @@ async def test_poll_one_happy_path(monkeypatch):assert fake_engine.failure_calls == []+# ---------------------------------------------------------------------------+# Schema-Drift Sentinel + filter-rules pipeline integration+# ---------------------------------------------------------------------------+++async def test_poll_one_filter_rules_drop_events(monkeypatch):+ """Pre-ingest filter rules should drop matching events and bump the+ drop counter passed to record_poll_success without ever shipping the+ dropped events to ingest."""+ inst = _make_instance(+ connector_type="fake_connector",+ connector_config={+ "poll_interval_seconds": 60,+ "filter_rules": [+ {"field": "severity", "op": "eq", "value": "info", "action": "drop"},+ ],+ },+ )++ fake_connector = MagicMock()+ fake_connector.fetch_alerts = AsyncMock(+ return_value=[+ {"severity": "info", "id": "drop-me"},+ {"severity": "high", "id": "keep-me"},+ ]+ )+ fake_connector.normalize = MagicMock(side_effect=lambda e: e)++ monkeypatch.setattr(+ "app.scheduler.CONNECTOR_REGISTRY",+ {"fake_connector": MagicMock(return_value=fake_connector)},+ )++ fake_engine = _FakeEngine([inst])+ fake_ingest = _FakeIngestClient(accepted=1)++ captured_success: dict[str, Any] = {}++ async def capture_record_poll_success(+ conn: Any,+ connector_id: uuid.UUID,+ *,+ events_added: int,+ events_dropped: int = 0,+ schema_fingerprint: str | None = None,+ ) -> None:+ conn.engine.success_calls.append(connector_id)+ captured_success["events_added"] = events_added+ captured_success["events_dropped"] = events_dropped+ captured_success["schema_fingerprint"] = schema_fingerprint++ monkeypatch.setattr("app.scheduler.record_poll_success", capture_record_poll_success)++ scheduler = ConnectorScheduler(engine=fake_engine, ingest_client=fake_ingest, vault=_FakeVault())+ await scheduler._poll_one(connector_id=inst.id)++ # Only the high-severity event should reach ingest.+ pushed = fake_ingest.last_payload+ assert pushed is not None+ assert pushed["events"] == [{"severity": "high", "id": "keep-me"}]+ assert captured_success["events_dropped"] == 1+ assert captured_success["events_added"] == 1+ assert captured_success["schema_fingerprint"] is not None+++async def test_poll_one_first_poll_records_fingerprint_no_drift(monkeypatch):+ """First poll on a connector with no baseline should record the new+ fingerprint via record_poll_success but NOT trigger record_schema_drift."""+ inst = _make_instance(+ connector_type="fake_connector",+ connector_config={"poll_interval_seconds": 60},+ )+ # Baseline is None on the ConnectorInstance.++ fake_connector = MagicMock()+ fake_connector.fetch_alerts = AsyncMock(return_value=[{"severity": "high", "host": "h1"}])+ fake_connector.normalize = MagicMock(side_effect=lambda e: e)++ monkeypatch.setattr(+ "app.scheduler.CONNECTOR_REGISTRY",+ {"fake_connector": MagicMock(return_value=fake_connector)},+ )++ fake_engine = _FakeEngine([inst])+ scheduler = ConnectorScheduler(+ engine=fake_engine,+ ingest_client=_FakeIngestClient(accepted=1),+ vault=_FakeVault(),+ )+ await scheduler._poll_one(connector_id=inst.id)++ assert fake_engine.success_calls == [inst.id]+ # No drift expected on first poll.+ assert fake_engine.drift_calls == []+++async def test_poll_one_detects_schema_drift(monkeypatch):+ """When the new fingerprint differs from the stored baseline,+ record_schema_drift should be called with details describing the+ change before record_poll_success is invoked."""+ # Synthesize a baseline fingerprint that differs from what the+ # current poll will produce.+ from app.pipeline.fingerprint import compute_fingerprint as cf++ stale_fp = cf([{"severity": "high"}]) # baseline: only one field+ inst = _make_instance(+ connector_type="fake_connector",+ connector_config={"poll_interval_seconds": 60},+ )+ inst.schema_fingerprint = stale_fp++ fake_connector = MagicMock()+ fake_connector.fetch_alerts = AsyncMock(+ return_value=[{"severity": "high", "host": "h1", "new_field": "x"}]+ )+ fake_connector.normalize = MagicMock(side_effect=lambda e: e)++ monkeypatch.setattr(+ "app.scheduler.CONNECTOR_REGISTRY",+ {"fake_connector": MagicMock(return_value=fake_connector)},+ )++ fake_engine = _FakeEngine([inst])+ scheduler = ConnectorScheduler(+ engine=fake_engine,+ ingest_client=_FakeIngestClient(accepted=1),+ vault=_FakeVault(),+ )+ await scheduler._poll_one(connector_id=inst.id)++ assert fake_engine.success_calls == [inst.id]+ assert len(fake_engine.drift_calls) == 1+ drift = fake_engine.drift_calls[0]+ assert drift["connector_id"] == inst.id+ assert drift["fingerprint"] != stale_fp+ assert "added" in drift["details"]+ assert "previous_fingerprint" in drift["details"]+ assert drift["details"]["previous_fingerprint"] == stale_fp+++async def test_poll_one_stable_schema_no_drift(monkeypatch):+ """If the new fingerprint matches the stored baseline, the row's+ fingerprint is rewritten via success but no drift event fires."""+ from app.pipeline.fingerprint import compute_fingerprint as cf++ events = [{"severity": "high", "host": "h1"}]+ baseline_fp = cf(events)+ inst = _make_instance(+ connector_type="fake_connector",+ connector_config={"poll_interval_seconds": 60},+ )+ inst.schema_fingerprint = baseline_fp++ fake_connector = MagicMock()+ fake_connector.fetch_alerts = AsyncMock(return_value=events)+ fake_connector.normalize = MagicMock(side_effect=lambda e: e)++ monkeypatch.setattr(+ "app.scheduler.CONNECTOR_REGISTRY",+ {"fake_connector": MagicMock(return_value=fake_connector)},+ )++ fake_engine = _FakeEngine([inst])+ scheduler = ConnectorScheduler(+ engine=fake_engine,+ ingest_client=_FakeIngestClient(accepted=1),+ vault=_FakeVault(),+ )+ await scheduler._poll_one(connector_id=inst.id)++ assert fake_engine.success_calls == [inst.id]+ assert fake_engine.drift_calls == []+++async def test_poll_one_quiet_hour_does_not_clobber_fingerprint(monkeypatch):+ """If a poll returns zero events the fingerprint is None and we should+ NOT overwrite the baseline in the row (ConnectorInstance fingerprint+ stays as-is)."""+ from app.pipeline.fingerprint import compute_fingerprint as cf++ baseline_fp = cf([{"severity": "high", "host": "h1"}])+ inst = _make_instance(+ connector_type="fake_connector",+ connector_config={"poll_interval_seconds": 60},+ )+ inst.schema_fingerprint = baseline_fp++ fake_connector = MagicMock()+ fake_connector.fetch_alerts = AsyncMock(return_value=[])+ fake_connector.normalize = MagicMock(side_effect=lambda e: e)++ monkeypatch.setattr(+ "app.scheduler.CONNECTOR_REGISTRY",+ {"fake_connector": MagicMock(return_value=fake_connector)},+ )++ captured_success: dict[str, Any] = {}++ async def capture_record_poll_success(+ conn: Any,+ connector_id: uuid.UUID,+ *,+ events_added: int,+ events_dropped: int = 0,+ schema_fingerprint: str | None = None,+ ) -> None:+ conn.engine.success_calls.append(connector_id)+ captured_success["schema_fingerprint"] = schema_fingerprint++ monkeypatch.setattr("app.scheduler.record_poll_success", capture_record_poll_success)++ fake_engine = _FakeEngine([inst])+ scheduler = ConnectorScheduler(+ engine=fake_engine,+ ingest_client=_FakeIngestClient(accepted=0),+ vault=_FakeVault(),+ )+ await scheduler._poll_one(connector_id=inst.id)++ # Empty batch → fingerprint should be None so the DB layer leaves the+ # baseline alone.+ assert captured_success["schema_fingerprint"] is None+ assert fake_engine.drift_calls == []++# ---------------------------------------------------------------------------# Failure paths each flip the row to unhealthy without raising# ---------------------------------------------------------------------------@@ -338,6 +572,7 @@ class _FakeEngine:self.instances = list(instances)self.success_calls: list[uuid.UUID] = []self.failure_calls: list[uuid.UUID] = []+ self.drift_calls: list[dict[str, Any]] = []def begin(self) -> _FakeAsyncContextManager:return _FakeAsyncContextManager(_FakeConnection(self))@@ -389,12 +624,31 @@ def _patch_repo_calls(monkeypatch):async def fake_fetch_enabled_connectors(conn: Any) -> list[ConnectorInstance]:return list(conn.engine.instances)- async def fake_record_poll_success(conn: Any, connector_id: uuid.UUID, *, events_added: int) -> None:+ async def fake_record_poll_success(+ conn: Any,+ connector_id: uuid.UUID,+ *,+ events_added: int,+ events_dropped: int = 0,+ schema_fingerprint: str | None = None,+ ) -> None:conn.engine.success_calls.append(connector_id)async def fake_record_poll_failure(conn: Any, connector_id: uuid.UUID) -> None:conn.engine.failure_calls.append(connector_id)+ async def fake_record_schema_drift(+ conn: Any,+ connector_id: uuid.UUID,+ *,+ fingerprint: str,+ details: dict[str, Any],+ ) -> None:+ getattr(conn.engine, "drift_calls", []).append(+ {"connector_id": connector_id, "fingerprint": fingerprint, "details": details}+ )+monkeypatch.setattr("app.scheduler.fetch_enabled_connectors", fake_fetch_enabled_connectors)monkeypatch.setattr("app.scheduler.record_poll_success", fake_record_poll_success)monkeypatch.setattr("app.scheduler.record_poll_failure", fake_record_poll_failure)+ monkeypatch.setattr("app.scheduler.record_schema_drift", fake_record_schema_drift)