Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

feat(phase-5): data spine correctness — schema registry + DLQ + lineage (#430)

Beenu Arora3d8d2622b5cc
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 648bc552e..e4951af29 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
+- **Phase 5 — data spine correctness: schema registry + dead-letter queue + lineage.** The reality audit's load-bearing gap: the fusion consumer logged a warning and **silently dropped** any malformed message — silent data loss. Now every message is schema-validated against a versioned envelope registry (`services/fusion/app/services/event_schema.py`: `aisoc.raw_events` and `aisoc.alerts.raw`, each pinned to `v1`, with a drift-guard test so the wire contract can't move unnoticed) *before* the promoter sees it. Anything that fails — non-object payload, unknown `schema_version`, missing `ocsf_event`, non-UUID tenant, or a RawAlert that fails deep Pydantic validation — is routed to a **fail-soft dead-letter queue** (`services/fusion/app/services/dlq.py`: `LoggingDLQ` default, `InMemoryDLQ` for tests, `KafkaDLQ` republishing to `aisoc.alerts.dlq`) with its reason, schema version, source-event lineage, and a truncated payload, instead of vanishing. `safe_record` guarantees a DLQ that itself throws never crashes the consumer. Each processed alert logs `fusion.lineage` (source event id + schema version). 21 new tests (`test_event_schema.py`, `test_dlq.py`, `test_consumer_dlq.py`) prove a valid event is processed with an empty DLQ, every poison shape is captured (not dropped), and a failing DLQ sink doesn't crash the consumer; full fusion suite 92/92 at 59% coverage (floor 48). Idempotency (AlertSink dedup fingerprint, Phase 3.1) and event-time watermarking are already present; backfill/replay-from-offset is tracked as 5b in `docs/audit/PROGRESS.md`.
- **Phase 4a — the Detection-as-Code gate is no longer circular.** The reality audit's #1 circular gate: the detection-proposal promote path shelled out to `run_evals.py` **without ever passing the proposed rule body**, so "passed" meant the repo-wide substrate MITRE accuracy didn't move — a value independent of the rule under review. A blind rule (matches nothing) or a noisy rule (matches everything) sailed through its own exam. New `services/api/app/services/detection_eval.py::evaluate_candidate_rule` runs the *candidate `rule_body` itself* through the **real** runtime engine (`rule_engine.execute_rule`) against caller-supplied positive/negative fixtures: it must fire on every positive and stay silent on every negative. New `POST /detection-proposals/{id}/evaluate-rule` stores the verdict under `eval_result["candidate_rule"]`, and `POST /decide` (approve) now **requires** it — a benchmark-only pass is no longer sufficient. `services/api/tests/test_detection_eval.py` (6 tests) is the mutation test that proves the gate rejects a blind rule, rejects a noisy rule, rejects a no-positive-fixture rule, and that the verdict genuinely depends on the rule body. Claim-to-gate matrix: DAC row PARTIAL(circular) → GATED.
- **Phase 4b — detection content truth table (honest coverage).** The README advertised "6000+ imported detection rules", but ~97% live under `_quarantine/` (`enabled: false`) because their upstream query language (SPL / YARA-L / CAR pseudocode) does not execute on the engine. New `scripts/detection_truth_table.py` walks `detections/` and classifies every rule **executable** (fires today) vs **non-executable** (provenance/coverage only), rendering `docs/detections/truth-table.md`. The honest headline: **939 executable** rules (861 native + 77 sigma-imported + 1 community) of 6975 on disk; 5921 quarantined, 115 disabled. `--check` gates the doc in `validate-detections.yml` so the number can never quietly drift from reality, and the README now cites the executable figure for *coverage* (the on-disk figure only describes the imported *library*). Claim-to-gate matrix: 6000+-imported row PARTIAL → GATED (14 GATED / 10 PARTIAL / 3 NO GATE). The LLM-dependent half of Phase 4 (live-agent Tier-1 eval, hallucination/calibration/abstention, model matrix, 150-payload prompt-injection adversarial) is tracked as 4c+ in `docs/audit/PROGRESS.md`.
- **Phase 3.4 — cross-store tenant isolation, now proven against live containers.** The offline `isolation.yml` gate proved each read path *constructs* a tenant scope; it could not prove the scope actually isolates. New `.github/workflows/isolation-live.yml` + `tests/isolation/test_live_stores.py` seed tenant A and tenant B in **real containers** (Neo4j, Redis, ClickHouse, Redpanda/Kafka) and assert a read as A returns zero B data. ClickHouse runs the **production** `lake_sql.rewrite_for_tenant` rewriter against a live warehouse (verified: `SELECT … FROM aisoc.raw_events` becomes `… WHERE tenant_id = '<A>'`, so B's rows never return); Neo4j uses the `tenant_id` property filter; Redis uses the `aisoc:t:<tenant>:*` keyspace namespacing; Kafka replays the `graph_ws` per-tenant envelope filter. Every test also asserts the *unscoped* read sees both tenants, so a scoped pass can never be vacuous on an empty store. The isolation registry (`tests/isolation/stores.py`) flips Neo4j/Redis/ClickHouse/Kafka from `container_pending` → `container_gated`, and the claim-to-gate matrix moves "Cross-tenant isolation (Qdrant/Neo4j/Redis/ClickHouse/Kafka)" PARTIAL → GATED (12 GATED / 12 PARTIAL / 3 NO GATE). The heavy-demo-stack items (Playwright real-stack E2E, demo time-to-first-investigation budget) are tracked as non-blocking 3.5+ in `docs/audit/PROGRESS.md`.
diff --git a/ROADMAP.md b/ROADMAP.md
index 3db286369..3582dc13c 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -27,7 +27,7 @@ A proof-first, security-first program to make every README claim gate-backed, cl
- [x] Phase 2 — Supply chain + truth (security.yml scanners, claim-gate ratchet, hard-fail insecure prod defaults, TRADEMARK, verifying-releases, license fix; continuation landed: per-image cosign signatures + CycloneDX SBOM attestations + SLSA provenance, all actions SHA-pinned)
- [x] Phase 3 — Integration / E2E / chaos / DR (real-container spine test, backup-restore, chaos, upgrade, cross-store isolation live-replay; heavy-demo-stack Playwright E2E + demo-timing gate tracked as non-blocking 3.5+)
- [ ] Phase 4 — Real evals + detection content truth table (third-party-labeled corpus, hallucination/calibration/abstention, model matrix) — **4a/4b landed** (de-circularised DAC candidate-rule gate + honest executable-vs-imported truth table); 4c+ live-agent eval / calibration / model matrix pending
-- [ ] Phase 5 — Data spine correctness (schema registry, DLQ, idempotency, watermarking, backfill/replay, lineage)
+- [x] Phase 5 — Data spine correctness (versioned event-schema registry + dead-letter queue + source-event lineage in the fusion consumer; idempotency via AlertSink dedup + event-time watermarking; backfill/replay-from-offset tracked as 5b)
- [ ] Phase 6 — Performance + cost (throughput harness, $/TB, storage-consolidation ADR)
- [ ] Phase 7 — Ingest-time graph + multi-model router (posture, effective permissions, bi-temporal, fusion-time ContextBundle, determinism contract)
- [ ] Phase 8 — LLMOps (prompt registry, model pinning, provider fallback, response caching, structured-output validation)
diff --git a/services/fusion/app/services/dlq.py b/services/fusion/app/services/dlq.py
new file mode 100644
index 000000000..e3f1074a2
--- /dev/null
+++ b/services/fusion/app/services/dlq.py
@@ -0,0 +1,138 @@
+"""Dead-letter queue for the fusion consumer (Phase 5).
+
+A poison message — malformed JSON shape, an unknown schema version, a non-UUID
+tenant, or a RawAlert that fails deep validation — must not crash the consumer
+and must not vanish. Before Phase 5 the consumer logged a warning and dropped
+it (silent data loss the reality audit flagged). Now every such message is
+routed here: captured with its reason, schema version, source-event lineage,
+and a truncated copy of the payload, so an operator can see exactly what was
+rejected and why.
+
+Sinks are pluggable and **fail-soft** — a DLQ that itself throws must never
+take down the consumer (that would turn a single poison message into an
+outage). :func:`safe_record` wraps every sink call.
+
+* :class:`LoggingDLQ` — default. Emits a structured ``fusion.dead_letter``
+ warning. Always available, zero dependencies.
+* :class:`InMemoryDLQ` — test double; keeps records in a list.
+* :class:`KafkaDLQ` — production option that republishes to a dead-letter
+ topic (``aisoc.alerts.dlq``) via the consumer's existing producer, so DLQ
+ volume is observable on the same bus as everything else.
+"""
+
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass, field
+from datetime import UTC, datetime
+from typing import Any, Protocol, runtime_checkable
+
+import structlog
+
+logger = structlog.get_logger()
+
+# Cap the stored payload so a pathological megabyte message can't bloat the DLQ
+# topic / log line. The reason + lineage are what an operator triages on.
+_MAX_PAYLOAD_CHARS = 4000
+
+
+@dataclass(frozen=True)
+class DeadLetter:
+ """One rejected message, with everything needed to triage it."""
+
+ topic: str
+ reason: str
+ schema_version: str
+ payload_excerpt: str
+ source_event_id: str | None = None
+ tenant_id: str | None = None
+ ts: str = field(default_factory=lambda: datetime.now(UTC).isoformat())
+
+ @classmethod
+ def build(
+ cls,
+ *,
+ topic: str,
+ reason: str,
+ schema_version: str,
+ payload: Any,
+ source_event_id: str | None = None,
+ tenant_id: str | None = None,
+ ) -> DeadLetter:
+ try:
+ excerpt = json.dumps(payload, default=str)[:_MAX_PAYLOAD_CHARS]
+ except (TypeError, ValueError):
+ excerpt = str(payload)[:_MAX_PAYLOAD_CHARS]
+ return cls(
+ topic=topic,
+ reason=reason,
+ schema_version=schema_version,
+ payload_excerpt=excerpt,
+ source_event_id=source_event_id,
+ tenant_id=tenant_id,
+ )
+
+ def to_dict(self) -> dict[str, Any]:
+ return {
+ "topic": self.topic,
+ "reason": self.reason,
+ "schema_version": self.schema_version,
+ "payload_excerpt": self.payload_excerpt,
+ "source_event_id": self.source_event_id,
+ "tenant_id": self.tenant_id,
+ "ts": self.ts,
+ }
+
+
+@runtime_checkable
+class DeadLetterQueue(Protocol):
+ async def record(self, dead_letter: DeadLetter) -> None: ...
+
+
+class LoggingDLQ:
+ """Default sink — structured warning. Alert on ``fusion.dead_letter``."""
+
+ async def record(self, dead_letter: DeadLetter) -> None:
+ logger.warning("fusion.dead_letter", **dead_letter.to_dict())
+
+
+class InMemoryDLQ:
+ """Test double. Keeps records so assertions can inspect them."""
+
+ def __init__(self) -> None:
+ self.records: list[DeadLetter] = []
+
+ async def record(self, dead_letter: DeadLetter) -> None:
+ self.records.append(dead_letter)
+
+
+class KafkaDLQ:
+ """Republish to a dead-letter topic via the consumer's producer.
+
+ ``producer`` is the worker's ``AIOKafkaProducer`` (already started). We do
+ not own its lifecycle. If the send fails, :func:`safe_record` upstream
+ swallows it — a broken DLQ must not wedge the consumer.
+ """
+
+ def __init__(self, producer: Any, topic: str = "aisoc.alerts.dlq") -> None:
+ self._producer = producer
+ self._topic = topic
+
+ async def record(self, dead_letter: DeadLetter) -> None:
+ await self._producer.send(self._topic, value=dead_letter.to_dict())
+
+
+async def safe_record(dlq: DeadLetterQueue | None, dead_letter: DeadLetter) -> bool:
+ """Record to the DLQ, swallowing any sink error. Returns True on success.
+
+ A DLQ that raises must never propagate — that would let one poison message
+ crash the consumer, the exact failure mode the DLQ exists to prevent.
+ """
+ if dlq is None:
+ return False
+ try:
+ await dlq.record(dead_letter)
+ return True
+ except Exception as exc: # noqa: BLE001 — fail-soft by contract
+ logger.error("fusion.dlq_sink_failed", error=str(exc), topic=dead_letter.topic, reason=dead_letter.reason)
+ return False
diff --git a/services/fusion/app/services/event_schema.py b/services/fusion/app/services/event_schema.py
new file mode 100644
index 000000000..bbaa3808b
--- /dev/null
+++ b/services/fusion/app/services/event_schema.py
@@ -0,0 +1,137 @@
+"""Versioned event-schema registry for the fusion consumer (Phase 5).
+
+The reality audit's spine work (Phase 3.1) proved raw events flow end-to-end.
+Phase 5 makes that flow *correct under adversity*: a malformed, mis-versioned,
+or mis-tenanted message must never (a) crash the consumer or (b) vanish
+silently. This module is the schema half of that contract — it validates every
+message against a versioned envelope schema before the promoter sees it, and
+returns a structured verdict. The dead-letter queue (`dlq.py`) then captures
+anything that fails, with the reason, instead of dropping it on the floor.
+
+Two envelope schemas are registered, keyed by the topic they arrive on:
+
+* ``aisoc.raw_events`` — ingest's normalized-event envelope. Must carry an
+ ``ocsf_event`` object and resolve to a UUID tenant. ``schema_version``
+ defaults to ``v1`` when absent (every event ingest emits today is v1); an
+ unrecognised version is dead-lettered rather than mis-parsed.
+* ``aisoc.alerts.raw`` — the vendor RawAlert envelope. Structural check here;
+ the deep Pydantic validation stays in the consumer and its failure is also
+ dead-lettered.
+
+Adding a field to an envelope is backward-compatible (unknown fields are
+ignored). Renaming/removing a required field, or bumping the wire format, must
+add a new version string to :data:`KNOWN_SCHEMA_VERSIONS` — the drift test in
+``test_event_schema.py`` fails if the registered set changes without a
+conscious edit, so the wire contract can't move unnoticed.
+"""
+
+from __future__ import annotations
+
+import uuid
+from dataclasses import dataclass
+from typing import Any
+
+SCHEMA_VERSION_FIELD = "schema_version"
+DEFAULT_SCHEMA_VERSION = "v1"
+
+# The wire contract. Bump deliberately; the drift test guards this set.
+KNOWN_SCHEMA_VERSIONS: dict[str, frozenset[str]] = {
+ "aisoc.raw_events": frozenset({"v1"}),
+ "aisoc.alerts.raw": frozenset({"v1"}),
+}
+
+
+@dataclass(frozen=True)
+class EventValidation:
+ """Verdict for one message. ``ok`` gates processing; the rest is for the
+ DLQ record and lineage logging."""
+
+ ok: bool
+ schema_version: str
+ reason: str = ""
+ source_event_id: str | None = None
+ tenant_id: str | None = None
+
+
+def resolve_schema_version(payload: dict[str, Any]) -> str:
+ v = payload.get(SCHEMA_VERSION_FIELD)
+ return str(v) if isinstance(v, str | int) and str(v).strip() else DEFAULT_SCHEMA_VERSION
+
+
+def _resolve_source_event_id(payload: dict[str, Any], ocsf: dict[str, Any] | None) -> str | None:
+ for candidate in (payload.get("event_id"), payload.get("id")):
+ if isinstance(candidate, str) and candidate:
+ return candidate
+ if ocsf is not None:
+ meta = ocsf.get("metadata")
+ if isinstance(meta, dict):
+ uid = meta.get("uid")
+ if isinstance(uid, str) and uid:
+ return uid
+ return None
+
+
+def validate_raw_event(payload: Any, *, topic: str = "aisoc.raw_events") -> EventValidation:
+ """Validate an ingest normalized-event envelope."""
+ version = DEFAULT_SCHEMA_VERSION
+ if not isinstance(payload, dict):
+ return EventValidation(ok=False, schema_version=version, reason="payload is not a JSON object")
+
+ version = resolve_schema_version(payload)
+ known = KNOWN_SCHEMA_VERSIONS.get(topic, frozenset())
+ if version not in known:
+ return EventValidation(ok=False, schema_version=version, reason=f"unknown schema_version '{version}' for {topic}")
+
+ ocsf = payload.get("ocsf_event")
+ if not isinstance(ocsf, dict):
+ return EventValidation(ok=False, schema_version=version, reason="missing or non-object 'ocsf_event'")
+
+ tenant_raw = payload.get("tenant_id") or ocsf.get("tenant_uid")
+ try:
+ tenant_id = str(uuid.UUID(str(tenant_raw)))
+ except (ValueError, TypeError):
+ return EventValidation(
+ ok=False,
+ schema_version=version,
+ reason="tenant_id is not a UUID (mis-configured connector)",
+ source_event_id=_resolve_source_event_id(payload, ocsf),
+ )
+
+ return EventValidation(
+ ok=True,
+ schema_version=version,
+ source_event_id=_resolve_source_event_id(payload, ocsf),
+ tenant_id=tenant_id,
+ )
+
+
+def validate_raw_alert(payload: Any, *, topic: str = "aisoc.alerts.raw") -> EventValidation:
+ """Structural check for a vendor RawAlert envelope.
+
+ Deep field validation stays in the consumer (Pydantic); this catches the
+ gross-shape failures (non-object, unknown version) so they are dead-lettered
+ with a clear reason rather than raising deep in model construction.
+ """
+ version = DEFAULT_SCHEMA_VERSION
+ if not isinstance(payload, dict):
+ return EventValidation(ok=False, schema_version=version, reason="payload is not a JSON object")
+ version = resolve_schema_version(payload)
+ known = KNOWN_SCHEMA_VERSIONS.get(topic, frozenset())
+ if version not in known:
+ return EventValidation(ok=False, schema_version=version, reason=f"unknown schema_version '{version}' for {topic}")
+ source_event_id = _resolve_source_event_id(payload, None)
+ tenant = payload.get("tenant_id")
+ return EventValidation(
+ ok=True,
+ schema_version=version,
+ source_event_id=source_event_id,
+ tenant_id=str(tenant) if tenant is not None else None,
+ )
+
+
+def validate_event(topic: str, payload: Any, *, raw_events_topic: str, alerts_raw_topic: str) -> EventValidation:
+ """Dispatch validation by the topic a message arrived on."""
+ if topic == raw_events_topic:
+ return validate_raw_event(payload, topic="aisoc.raw_events")
+ # Everything else is treated as a RawAlert envelope.
+ return validate_raw_alert(payload, topic="aisoc.alerts.raw")
diff --git a/services/fusion/app/workers/consumer.py b/services/fusion/app/workers/consumer.py
index 446d35c72..77860fd6c 100644
--- a/services/fusion/app/workers/consumer.py
+++ b/services/fusion/app/workers/consumer.py
@@ -12,6 +12,8 @@ from aiokafka import AIOKafkaConsumer, AIOKafkaProducer
from app.core.config import settings
from app.models.alert import FusionDecision, RawAlert
from app.services.alert_sink import AlertSink
+from app.services.dlq import DeadLetter, DeadLetterQueue, LoggingDLQ, safe_record
+from app.services.event_schema import validate_event
from app.services.fusion_engine import FusionEngine
from app.services.promoter import promote_normalized_event
@@ -25,6 +27,7 @@ _METRICS = {
"promoted": 0,
"not_promoted": 0,
"persisted": 0,
+ "dead_lettered": 0,
"errors": 0,
}
@@ -32,9 +35,17 @@ _METRICS = {
class FusionWorker:
"""Kafka consumer/producer pair that drives the fusion pipeline."""
- def __init__(self, engine: FusionEngine, sink: AlertSink | None = None) -> None:
+ def __init__(
+ self,
+ engine: FusionEngine,
+ sink: AlertSink | None = None,
+ dlq: DeadLetterQueue | None = None,
+ ) -> None:
self._engine = engine
self._sink = sink
+ # A poison message must never vanish silently; default to a structured
+ # logging DLQ so persistence-free deployments still get the signal.
+ self._dlq: DeadLetterQueue = dlq or LoggingDLQ()
self._consumer: AIOKafkaConsumer | None = None
self._producer: AIOKafkaProducer | None = None
self._running = False
@@ -93,7 +104,53 @@ class FusionWorker:
_METRICS["errors"] += 1
logger.error("Failed to process message", error=str(exc), exc_info=True)
+ async def _dead_letter(
+ self,
+ *,
+ topic: str,
+ payload,
+ reason: str,
+ schema_version: str = "v1",
+ source_event_id: str | None = None,
+ tenant_id: str | None = None,
+ ) -> None:
+ """Route a poison message to the DLQ instead of dropping it silently."""
+ _METRICS["dead_lettered"] += 1
+ await safe_record(
+ self._dlq,
+ DeadLetter.build(
+ topic=topic,
+ reason=reason,
+ schema_version=schema_version,
+ payload=payload,
+ source_event_id=source_event_id,
+ tenant_id=tenant_id,
+ ),
+ )
+
async def _process_message(self, payload: dict, topic: str | None = None) -> None:
+ resolved_topic = topic or settings.kafka_topic_alerts_raw
+
+ # Phase 5 — schema-validate the envelope BEFORE the promoter sees it.
+ # A malformed / mis-versioned / mis-tenanted message is dead-lettered
+ # (captured with its reason + lineage), never silently dropped.
+ validation = validate_event(
+ resolved_topic,
+ payload,
+ raw_events_topic=settings.kafka_topic_raw_events,
+ alerts_raw_topic=settings.kafka_topic_alerts_raw,
+ )
+ if not validation.ok:
+ await self._dead_letter(
+ topic=resolved_topic,
+ payload=payload,
+ reason=validation.reason,
+ schema_version=validation.schema_version,
+ source_event_id=validation.source_event_id,
+ tenant_id=validation.tenant_id,
+ )
+ return
+
if topic == settings.kafka_topic_raw_events:
# Ingest-normalized OCSF event — run the deterministic promotion
# policy (see app/services/promoter.py). Non-promoted events are
@@ -107,10 +164,26 @@ class FusionWorker:
try:
alert = RawAlert.model_validate(payload)
except Exception as exc:
- logger.warning("Invalid alert payload", error=str(exc))
- _METRICS["errors"] += 1
+ # Deep validation failure — dead-letter it (the schema layer
+ # only caught gross shape); never silently drop.
+ await self._dead_letter(
+ topic=resolved_topic,
+ payload=payload,
+ reason=f"RawAlert validation failed: {exc}",
+ schema_version=validation.schema_version,
+ source_event_id=validation.source_event_id,
+ tenant_id=validation.tenant_id,
+ )
return
+ # Lineage — trace each fused alert back to its source event + schema.
+ logger.debug(
+ "fusion.lineage",
+ topic=resolved_topic,
+ schema_version=validation.schema_version,
+ source_event_id=validation.source_event_id,
+ )
+
fused = await self._engine.process(alert)
_METRICS["processed"] += 1
diff --git a/services/fusion/tests/test_consumer_dlq.py b/services/fusion/tests/test_consumer_dlq.py
new file mode 100644
index 000000000..a347db1c7
--- /dev/null
+++ b/services/fusion/tests/test_consumer_dlq.py
@@ -0,0 +1,109 @@
+"""Phase 5 — the fusion consumer dead-letters poison messages (never drops).
+
+Before Phase 5 the consumer logged a warning and dropped a malformed message —
+silent data loss. These tests drive ``FusionWorker._process_message`` directly
+(no Kafka) with an in-memory DLQ and prove: a valid event is processed and the
+DLQ stays empty; every poison shape is captured in the DLQ with a reason; and a
+DLQ that itself fails does not crash the consumer.
+"""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+from unittest.mock import AsyncMock
+
+import pytest
+from app.core.config import settings
+from app.models.alert import FusionDecision
+from app.services.dlq import InMemoryDLQ
+from app.workers.consumer import FusionWorker
+
+pytestmark = pytest.mark.asyncio
+
+TENANT = "00000000-0000-0000-0000-000000000001"
+RAW_EVENTS = settings.kafka_topic_raw_events
+ALERTS_RAW = settings.kafka_topic_alerts_raw
+
+
+def _valid_raw_event() -> dict:
+ return {
+ "tenant_id": TENANT,
+ "ocsf_event": {
+ "class_uid": 2001,
+ "category_uid": 2,
+ "severity_id": 5,
+ "message": "Credential dumping",
+ "metadata": {"uid": "ev-1", "product": {"name": "Falcon", "vendor_name": "CrowdStrike"}},
+ },
+ }
+
+
+def _worker_with_dlq() -> tuple[FusionWorker, InMemoryDLQ]:
+ dlq = InMemoryDLQ()
+ engine = SimpleNamespace(
+ process=AsyncMock(
+ return_value=SimpleNamespace(
+ fusion_decision=FusionDecision.NEW_ALERT,
+ model_dump=lambda mode="json": {"id": "fused-1"},
+ )
+ )
+ )
+ worker = FusionWorker(engine=engine, sink=None, dlq=dlq)
+ worker._producer = AsyncMock() # noqa: SLF001 — test drives the internals
+ return worker, dlq
+
+
+async def test_valid_event_is_processed_and_not_dead_lettered():
+ worker, dlq = _worker_with_dlq()
+ await worker._process_message(_valid_raw_event(), topic=RAW_EVENTS) # noqa: SLF001
+ assert dlq.records == []
+ worker._engine.process.assert_awaited_once() # noqa: SLF001
+ worker._producer.send.assert_awaited_once() # noqa: SLF001
+
+
+async def test_missing_ocsf_event_is_dead_lettered_not_dropped():
+ worker, dlq = _worker_with_dlq()
+ await worker._process_message({"tenant_id": TENANT}, topic=RAW_EVENTS) # noqa: SLF001
+ assert len(dlq.records) == 1
+ assert "ocsf_event" in dlq.records[0].reason
+ worker._engine.process.assert_not_awaited() # noqa: SLF001
+
+
+async def test_unknown_schema_version_is_dead_lettered():
+ worker, dlq = _worker_with_dlq()
+ poison = _valid_raw_event()
+ poison["schema_version"] = "v99"
+ await worker._process_message(poison, topic=RAW_EVENTS) # noqa: SLF001
+ assert len(dlq.records) == 1
+ assert "unknown schema_version" in dlq.records[0].reason
+
+
+async def test_non_uuid_tenant_is_dead_lettered_with_lineage():
+ worker, dlq = _worker_with_dlq()
+ poison = _valid_raw_event()
+ poison["tenant_id"] = "not-a-uuid"
+ poison["ocsf_event"].pop("tenant_uid", None)
+ await worker._process_message(poison, topic=RAW_EVENTS) # noqa: SLF001
+ assert len(dlq.records) == 1
+ assert dlq.records[0].source_event_id == "ev-1"
+
+
+async def test_invalid_raw_alert_is_dead_lettered_not_silently_dropped():
+ worker, dlq = _worker_with_dlq()
+ # Missing every required RawAlert field — deep Pydantic validation fails.
+ await worker._process_message({"tenant_id": TENANT, "nonsense": True}, topic=ALERTS_RAW) # noqa: SLF001
+ assert len(dlq.records) == 1
+ assert "RawAlert validation failed" in dlq.records[0].reason
+ worker._engine.process.assert_not_awaited() # noqa: SLF001
+
+
+async def test_dead_letter_never_crashes_the_consumer_when_sink_fails():
+ class ExplodingDLQ:
+ async def record(self, dead_letter): # noqa: ANN001, ARG002
+ raise RuntimeError("dlq sink down")
+
+ engine = SimpleNamespace(process=AsyncMock())
+ worker = FusionWorker(engine=engine, sink=None, dlq=ExplodingDLQ())
+ worker._producer = AsyncMock() # noqa: SLF001
+ # Poison message + failing DLQ: must return cleanly, not raise.
+ await worker._process_message({"tenant_id": TENANT}, topic=RAW_EVENTS) # noqa: SLF001
diff --git a/services/fusion/tests/test_dlq.py b/services/fusion/tests/test_dlq.py
new file mode 100644
index 000000000..15a8f3731
--- /dev/null
+++ b/services/fusion/tests/test_dlq.py
@@ -0,0 +1,63 @@
+"""Phase 5 — dead-letter queue tests."""
+
+from __future__ import annotations
+
+import pytest
+from app.services.dlq import DeadLetter, InMemoryDLQ, LoggingDLQ, safe_record
+
+
+def test_dead_letter_build_truncates_payload_and_keeps_lineage():
+ huge = {"blob": "x" * 10000, "n": 1}
+ dl = DeadLetter.build(
+ topic="aisoc.raw_events",
+ reason="test",
+ schema_version="v1",
+ payload=huge,
+ source_event_id="ev-9",
+ tenant_id="t-1",
+ )
+ assert len(dl.payload_excerpt) <= 4000
+ assert dl.source_event_id == "ev-9"
+ assert dl.tenant_id == "t-1"
+ assert dl.ts # timestamped
+
+
+def test_dead_letter_build_handles_unserialisable_payload():
+ dl = DeadLetter.build(topic="t", reason="r", schema_version="v1", payload=object())
+ assert dl.payload_excerpt # falls back to str(), never raises
+
+
+async def test_in_memory_dlq_records():
+ dlq = InMemoryDLQ()
+ dl = DeadLetter.build(topic="t", reason="r", schema_version="v1", payload={"a": 1})
+ ok = await safe_record(dlq, dl)
+ assert ok
+ assert len(dlq.records) == 1
+ assert dlq.records[0].reason == "r"
+
+
+async def test_safe_record_is_none_safe():
+ assert await safe_record(None, DeadLetter.build(topic="t", reason="r", schema_version="v1", payload={})) is False
+
+
+async def test_safe_record_swallows_sink_failure():
+ class ExplodingDLQ:
+ async def record(self, dead_letter): # noqa: ANN001, ARG002
+ raise RuntimeError("kafka down")
+
+ # A DLQ that itself fails must never propagate — one poison message can't
+ # be allowed to crash the consumer.
+ ok = await safe_record(ExplodingDLQ(), DeadLetter.build(topic="t", reason="r", schema_version="v1", payload={}))
+ assert ok is False
+
+
+async def test_logging_dlq_does_not_raise():
+ await LoggingDLQ().record(DeadLetter.build(topic="t", reason="r", schema_version="v1", payload={"x": 1}))
+
+
+if __name__ == "__main__": # pragma: no cover
+ raise SystemExit(pytest.main([__file__, "-v"]))
diff --git a/services/fusion/tests/test_event_schema.py b/services/fusion/tests/test_event_schema.py
new file mode 100644
index 000000000..d8fa49a52
--- /dev/null
+++ b/services/fusion/tests/test_event_schema.py
@@ -0,0 +1,81 @@
+"""Phase 5 — versioned event-schema registry tests."""
+
+from __future__ import annotations
+
+from app.services.event_schema import (
+ KNOWN_SCHEMA_VERSIONS,
+ validate_event,
+ validate_raw_alert,
+ validate_raw_event,
+)
+
+TENANT = "00000000-0000-0000-0000-000000000001"
+
+
+def _raw_event(**overrides) -> dict:
+ base = {"tenant_id": TENANT, "ocsf_event": {"class_uid": 2001, "metadata": {"uid": "ev-1"}}}
+ base.update(overrides)
+ return base
+
+
+def test_valid_raw_event_passes_and_carries_lineage():
+ v = validate_raw_event(_raw_event())
+ assert v.ok
+ assert v.schema_version == "v1"
+ assert v.source_event_id == "ev-1"
+ assert v.tenant_id == TENANT
+
+
+def test_non_object_payload_is_rejected():
+ v = validate_raw_event("not a dict")
+ assert not v.ok
+ assert "not a JSON object" in v.reason
+
+
+def test_missing_ocsf_event_is_rejected():
+ v = validate_raw_event({"tenant_id": TENANT})
+ assert not v.ok
+ assert "ocsf_event" in v.reason
+
+
+def test_unknown_schema_version_is_rejected():
+ v = validate_raw_event(_raw_event(schema_version="v99"))
+ assert not v.ok
+ assert "unknown schema_version" in v.reason
+
+
+def test_non_uuid_tenant_is_rejected_with_lineage():
+ v = validate_raw_event(_raw_event(tenant_id="not-a-uuid"))
+ assert not v.ok
+ assert "tenant_id is not a UUID" in v.reason
+ # Lineage is still captured so an operator can find the offending event.
+ assert v.source_event_id == "ev-1"
+
+
+def test_tenant_uid_fallback_inside_ocsf():
+ payload = {"ocsf_event": {"tenant_uid": TENANT, "class_uid": 2001}}
+ v = validate_raw_event(payload)
+ assert v.ok
+ assert v.tenant_id == TENANT
+
+
+def test_raw_alert_structural_check():
+ assert validate_raw_alert({"tenant_id": TENANT, "title": "x"}).ok
+ bad = validate_raw_alert([1, 2, 3])
+ assert not bad.ok
+
+
+def test_dispatch_by_topic():
+ ev = validate_event("aisoc.raw_events", _raw_event(), raw_events_topic="aisoc.raw_events", alerts_raw_topic="aisoc.alerts.raw")
+ assert ev.ok
+ al = validate_event("aisoc.alerts.raw", {"tenant_id": TENANT}, raw_events_topic="aisoc.raw_events", alerts_raw_topic="aisoc.alerts.raw")
+ assert al.ok
+
+
+def test_registered_schema_set_is_stable_drift_guard():
+ """The wire contract must not move unnoticed. If you intentionally add or
+ bump an envelope version, update this expected set in the same commit."""
+ assert KNOWN_SCHEMA_VERSIONS == {
+ "aisoc.raw_events": frozenset({"v1"}),
+ "aisoc.alerts.raw": frozenset({"v1"}),
+ }