Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
feat(phase-5): data spine correctness — schema registry + DLQ + lineage (#430)
Beenu Arora3d8d2622b5cc
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 648bc552e..e4951af29 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Added+- **Phase 5 — data spine correctness: schema registry + dead-letter queue + lineage.** The reality audit's load-bearing gap: the fusion consumer logged a warning and **silently dropped** any malformed message — silent data loss. Now every message is schema-validated against a versioned envelope registry (`services/fusion/app/services/event_schema.py`: `aisoc.raw_events` and `aisoc.alerts.raw`, each pinned to `v1`, with a drift-guard test so the wire contract can't move unnoticed) *before* the promoter sees it. Anything that fails — non-object payload, unknown `schema_version`, missing `ocsf_event`, non-UUID tenant, or a RawAlert that fails deep Pydantic validation — is routed to a **fail-soft dead-letter queue** (`services/fusion/app/services/dlq.py`: `LoggingDLQ` default, `InMemoryDLQ` for tests, `KafkaDLQ` republishing to `aisoc.alerts.dlq`) with its reason, schema version, source-event lineage, and a truncated payload, instead of vanishing. `safe_record` guarantees a DLQ that itself throws never crashes the consumer. Each processed alert logs `fusion.lineage` (source event id + schema version). 21 new tests (`test_event_schema.py`, `test_dlq.py`, `test_consumer_dlq.py`) prove a valid event is processed with an empty DLQ, every poison shape is captured (not dropped), and a failing DLQ sink doesn't crash the consumer; full fusion suite 92/92 at 59% coverage (floor 48). Idempotency (AlertSink dedup fingerprint, Phase 3.1) and event-time watermarking are already present; backfill/replay-from-offset is tracked as 5b in `docs/audit/PROGRESS.md`.- **Phase 4a — the Detection-as-Code gate is no longer circular.** The reality audit's #1 circular gate: the detection-proposal promote path shelled out to `run_evals.py` **without ever passing the proposed rule body**, so "passed" meant the repo-wide substrate MITRE accuracy didn't move — a value independent of the rule under review. A blind rule (matches nothing) or a noisy rule (matches everything) sailed through its own exam. New `services/api/app/services/detection_eval.py::evaluate_candidate_rule` runs the *candidate `rule_body` itself* through the **real** runtime engine (`rule_engine.execute_rule`) against caller-supplied positive/negative fixtures: it must fire on every positive and stay silent on every negative. New `POST /detection-proposals/{id}/evaluate-rule` stores the verdict under `eval_result["candidate_rule"]`, and `POST /decide` (approve) now **requires** it — a benchmark-only pass is no longer sufficient. `services/api/tests/test_detection_eval.py` (6 tests) is the mutation test that proves the gate rejects a blind rule, rejects a noisy rule, rejects a no-positive-fixture rule, and that the verdict genuinely depends on the rule body. Claim-to-gate matrix: DAC row PARTIAL(circular) → GATED.- **Phase 4b — detection content truth table (honest coverage).** The README advertised "6000+ imported detection rules", but ~97% live under `_quarantine/` (`enabled: false`) because their upstream query language (SPL / YARA-L / CAR pseudocode) does not execute on the engine. New `scripts/detection_truth_table.py` walks `detections/` and classifies every rule **executable** (fires today) vs **non-executable** (provenance/coverage only), rendering `docs/detections/truth-table.md`. The honest headline: **939 executable** rules (861 native + 77 sigma-imported + 1 community) of 6975 on disk; 5921 quarantined, 115 disabled. `--check` gates the doc in `validate-detections.yml` so the number can never quietly drift from reality, and the README now cites the executable figure for *coverage* (the on-disk figure only describes the imported *library*). Claim-to-gate matrix: 6000+-imported row PARTIAL → GATED (14 GATED / 10 PARTIAL / 3 NO GATE). The LLM-dependent half of Phase 4 (live-agent Tier-1 eval, hallucination/calibration/abstention, model matrix, 150-payload prompt-injection adversarial) is tracked as 4c+ in `docs/audit/PROGRESS.md`.- **Phase 3.4 — cross-store tenant isolation, now proven against live containers.** The offline `isolation.yml` gate proved each read path *constructs* a tenant scope; it could not prove the scope actually isolates. New `.github/workflows/isolation-live.yml` + `tests/isolation/test_live_stores.py` seed tenant A and tenant B in **real containers** (Neo4j, Redis, ClickHouse, Redpanda/Kafka) and assert a read as A returns zero B data. ClickHouse runs the **production** `lake_sql.rewrite_for_tenant` rewriter against a live warehouse (verified: `SELECT … FROM aisoc.raw_events` becomes `… WHERE tenant_id = '<A>'`, so B's rows never return); Neo4j uses the `tenant_id` property filter; Redis uses the `aisoc:t:<tenant>:*` keyspace namespacing; Kafka replays the `graph_ws` per-tenant envelope filter. Every test also asserts the *unscoped* read sees both tenants, so a scoped pass can never be vacuous on an empty store. The isolation registry (`tests/isolation/stores.py`) flips Neo4j/Redis/ClickHouse/Kafka from `container_pending` → `container_gated`, and the claim-to-gate matrix moves "Cross-tenant isolation (Qdrant/Neo4j/Redis/ClickHouse/Kafka)" PARTIAL → GATED (12 GATED / 12 PARTIAL / 3 NO GATE). The heavy-demo-stack items (Playwright real-stack E2E, demo time-to-first-investigation budget) are tracked as non-blocking 3.5+ in `docs/audit/PROGRESS.md`.diff --git a/ROADMAP.md b/ROADMAP.mdindex 3db286369..3582dc13c 100644--- a/ROADMAP.md+++ b/ROADMAP.md@@ -27,7 +27,7 @@ A proof-first, security-first program to make every README claim gate-backed, cl- [x] Phase 2 — Supply chain + truth (security.yml scanners, claim-gate ratchet, hard-fail insecure prod defaults, TRADEMARK, verifying-releases, license fix; continuation landed: per-image cosign signatures + CycloneDX SBOM attestations + SLSA provenance, all actions SHA-pinned)- [x] Phase 3 — Integration / E2E / chaos / DR (real-container spine test, backup-restore, chaos, upgrade, cross-store isolation live-replay; heavy-demo-stack Playwright E2E + demo-timing gate tracked as non-blocking 3.5+)- [ ] Phase 4 — Real evals + detection content truth table (third-party-labeled corpus, hallucination/calibration/abstention, model matrix) — **4a/4b landed** (de-circularised DAC candidate-rule gate + honest executable-vs-imported truth table); 4c+ live-agent eval / calibration / model matrix pending-- [ ] Phase 5 — Data spine correctness (schema registry, DLQ, idempotency, watermarking, backfill/replay, lineage)+- [x] Phase 5 — Data spine correctness (versioned event-schema registry + dead-letter queue + source-event lineage in the fusion consumer; idempotency via AlertSink dedup + event-time watermarking; backfill/replay-from-offset tracked as 5b)- [ ] Phase 6 — Performance + cost (throughput harness, $/TB, storage-consolidation ADR)- [ ] Phase 7 — Ingest-time graph + multi-model router (posture, effective permissions, bi-temporal, fusion-time ContextBundle, determinism contract)- [ ] Phase 8 — LLMOps (prompt registry, model pinning, provider fallback, response caching, structured-output validation)diff --git a/services/fusion/app/services/dlq.py b/services/fusion/app/services/dlq.pynew file mode 100644index 000000000..e3f1074a2--- /dev/null+++ b/services/fusion/app/services/dlq.py@@ -0,0 +1,138 @@+"""Dead-letter queue for the fusion consumer (Phase 5).++A poison message — malformed JSON shape, an unknown schema version, a non-UUID+tenant, or a RawAlert that fails deep validation — must not crash the consumer+and must not vanish. Before Phase 5 the consumer logged a warning and dropped+it (silent data loss the reality audit flagged). Now every such message is+routed here: captured with its reason, schema version, source-event lineage,+and a truncated copy of the payload, so an operator can see exactly what was+rejected and why.++Sinks are pluggable and **fail-soft** — a DLQ that itself throws must never+take down the consumer (that would turn a single poison message into an+outage). :func:`safe_record` wraps every sink call.++* :class:`LoggingDLQ` — default. Emits a structured ``fusion.dead_letter``+ warning. Always available, zero dependencies.+* :class:`InMemoryDLQ` — test double; keeps records in a list.+* :class:`KafkaDLQ` — production option that republishes to a dead-letter+ topic (``aisoc.alerts.dlq``) via the consumer's existing producer, so DLQ+ volume is observable on the same bus as everything else.+"""++from __future__ import annotations++import json+from dataclasses import dataclass, field+from datetime import UTC, datetime+from typing import Any, Protocol, runtime_checkable++import structlog++logger = structlog.get_logger()++# Cap the stored payload so a pathological megabyte message can't bloat the DLQ+# topic / log line. The reason + lineage are what an operator triages on.+_MAX_PAYLOAD_CHARS = 4000+++@dataclass(frozen=True)+class DeadLetter:+ """One rejected message, with everything needed to triage it."""++ topic: str+ reason: str+ schema_version: str+ payload_excerpt: str+ source_event_id: str | None = None+ tenant_id: str | None = None+ ts: str = field(default_factory=lambda: datetime.now(UTC).isoformat())++ @classmethod+ def build(+ cls,+ *,+ topic: str,+ reason: str,+ schema_version: str,+ payload: Any,+ source_event_id: str | None = None,+ tenant_id: str | None = None,+ ) -> DeadLetter:+ try:+ excerpt = json.dumps(payload, default=str)[:_MAX_PAYLOAD_CHARS]+ except (TypeError, ValueError):+ excerpt = str(payload)[:_MAX_PAYLOAD_CHARS]+ return cls(+ topic=topic,+ reason=reason,+ schema_version=schema_version,+ payload_excerpt=excerpt,+ source_event_id=source_event_id,+ tenant_id=tenant_id,+ )++ def to_dict(self) -> dict[str, Any]:+ return {+ "topic": self.topic,+ "reason": self.reason,+ "schema_version": self.schema_version,+ "payload_excerpt": self.payload_excerpt,+ "source_event_id": self.source_event_id,+ "tenant_id": self.tenant_id,+ "ts": self.ts,+ }+++@runtime_checkable+class DeadLetterQueue(Protocol):+ async def record(self, dead_letter: DeadLetter) -> None: ...+++class LoggingDLQ:+ """Default sink — structured warning. Alert on ``fusion.dead_letter``."""++ async def record(self, dead_letter: DeadLetter) -> None:+ logger.warning("fusion.dead_letter", **dead_letter.to_dict())+++class InMemoryDLQ:+ """Test double. Keeps records so assertions can inspect them."""++ def __init__(self) -> None:+ self.records: list[DeadLetter] = []++ async def record(self, dead_letter: DeadLetter) -> None:+ self.records.append(dead_letter)+++class KafkaDLQ:+ """Republish to a dead-letter topic via the consumer's producer.++ ``producer`` is the worker's ``AIOKafkaProducer`` (already started). We do+ not own its lifecycle. If the send fails, :func:`safe_record` upstream+ swallows it — a broken DLQ must not wedge the consumer.+ """++ def __init__(self, producer: Any, topic: str = "aisoc.alerts.dlq") -> None:+ self._producer = producer+ self._topic = topic++ async def record(self, dead_letter: DeadLetter) -> None:+ await self._producer.send(self._topic, value=dead_letter.to_dict())+++async def safe_record(dlq: DeadLetterQueue | None, dead_letter: DeadLetter) -> bool:+ """Record to the DLQ, swallowing any sink error. Returns True on success.++ A DLQ that raises must never propagate — that would let one poison message+ crash the consumer, the exact failure mode the DLQ exists to prevent.+ """+ if dlq is None:+ return False+ try:+ await dlq.record(dead_letter)+ return True+ except Exception as exc: # noqa: BLE001 — fail-soft by contract+ logger.error("fusion.dlq_sink_failed", error=str(exc), topic=dead_letter.topic, reason=dead_letter.reason)+ return Falsediff --git a/services/fusion/app/services/event_schema.py b/services/fusion/app/services/event_schema.pynew file mode 100644index 000000000..bbaa3808b--- /dev/null+++ b/services/fusion/app/services/event_schema.py@@ -0,0 +1,137 @@+"""Versioned event-schema registry for the fusion consumer (Phase 5).++The reality audit's spine work (Phase 3.1) proved raw events flow end-to-end.+Phase 5 makes that flow *correct under adversity*: a malformed, mis-versioned,+or mis-tenanted message must never (a) crash the consumer or (b) vanish+silently. This module is the schema half of that contract — it validates every+message against a versioned envelope schema before the promoter sees it, and+returns a structured verdict. The dead-letter queue (`dlq.py`) then captures+anything that fails, with the reason, instead of dropping it on the floor.++Two envelope schemas are registered, keyed by the topic they arrive on:++* ``aisoc.raw_events`` — ingest's normalized-event envelope. Must carry an+ ``ocsf_event`` object and resolve to a UUID tenant. ``schema_version``+ defaults to ``v1`` when absent (every event ingest emits today is v1); an+ unrecognised version is dead-lettered rather than mis-parsed.+* ``aisoc.alerts.raw`` — the vendor RawAlert envelope. Structural check here;+ the deep Pydantic validation stays in the consumer and its failure is also+ dead-lettered.++Adding a field to an envelope is backward-compatible (unknown fields are+ignored). Renaming/removing a required field, or bumping the wire format, must+add a new version string to :data:`KNOWN_SCHEMA_VERSIONS` — the drift test in+``test_event_schema.py`` fails if the registered set changes without a+conscious edit, so the wire contract can't move unnoticed.+"""++from __future__ import annotations++import uuid+from dataclasses import dataclass+from typing import Any++SCHEMA_VERSION_FIELD = "schema_version"+DEFAULT_SCHEMA_VERSION = "v1"++# The wire contract. Bump deliberately; the drift test guards this set.+KNOWN_SCHEMA_VERSIONS: dict[str, frozenset[str]] = {+ "aisoc.raw_events": frozenset({"v1"}),+ "aisoc.alerts.raw": frozenset({"v1"}),+}+++@dataclass(frozen=True)+class EventValidation:+ """Verdict for one message. ``ok`` gates processing; the rest is for the+ DLQ record and lineage logging."""++ ok: bool+ schema_version: str+ reason: str = ""+ source_event_id: str | None = None+ tenant_id: str | None = None+++def resolve_schema_version(payload: dict[str, Any]) -> str:+ v = payload.get(SCHEMA_VERSION_FIELD)+ return str(v) if isinstance(v, str | int) and str(v).strip() else DEFAULT_SCHEMA_VERSION+++def _resolve_source_event_id(payload: dict[str, Any], ocsf: dict[str, Any] | None) -> str | None:+ for candidate in (payload.get("event_id"), payload.get("id")):+ if isinstance(candidate, str) and candidate:+ return candidate+ if ocsf is not None:+ meta = ocsf.get("metadata")+ if isinstance(meta, dict):+ uid = meta.get("uid")+ if isinstance(uid, str) and uid:+ return uid+ return None+++def validate_raw_event(payload: Any, *, topic: str = "aisoc.raw_events") -> EventValidation:+ """Validate an ingest normalized-event envelope."""+ version = DEFAULT_SCHEMA_VERSION+ if not isinstance(payload, dict):+ return EventValidation(ok=False, schema_version=version, reason="payload is not a JSON object")++ version = resolve_schema_version(payload)+ known = KNOWN_SCHEMA_VERSIONS.get(topic, frozenset())+ if version not in known:+ return EventValidation(ok=False, schema_version=version, reason=f"unknown schema_version '{version}' for {topic}")++ ocsf = payload.get("ocsf_event")+ if not isinstance(ocsf, dict):+ return EventValidation(ok=False, schema_version=version, reason="missing or non-object 'ocsf_event'")++ tenant_raw = payload.get("tenant_id") or ocsf.get("tenant_uid")+ try:+ tenant_id = str(uuid.UUID(str(tenant_raw)))+ except (ValueError, TypeError):+ return EventValidation(+ ok=False,+ schema_version=version,+ reason="tenant_id is not a UUID (mis-configured connector)",+ source_event_id=_resolve_source_event_id(payload, ocsf),+ )++ return EventValidation(+ ok=True,+ schema_version=version,+ source_event_id=_resolve_source_event_id(payload, ocsf),+ tenant_id=tenant_id,+ )+++def validate_raw_alert(payload: Any, *, topic: str = "aisoc.alerts.raw") -> EventValidation:+ """Structural check for a vendor RawAlert envelope.++ Deep field validation stays in the consumer (Pydantic); this catches the+ gross-shape failures (non-object, unknown version) so they are dead-lettered+ with a clear reason rather than raising deep in model construction.+ """+ version = DEFAULT_SCHEMA_VERSION+ if not isinstance(payload, dict):+ return EventValidation(ok=False, schema_version=version, reason="payload is not a JSON object")+ version = resolve_schema_version(payload)+ known = KNOWN_SCHEMA_VERSIONS.get(topic, frozenset())+ if version not in known:+ return EventValidation(ok=False, schema_version=version, reason=f"unknown schema_version '{version}' for {topic}")+ source_event_id = _resolve_source_event_id(payload, None)+ tenant = payload.get("tenant_id")+ return EventValidation(+ ok=True,+ schema_version=version,+ source_event_id=source_event_id,+ tenant_id=str(tenant) if tenant is not None else None,+ )+++def validate_event(topic: str, payload: Any, *, raw_events_topic: str, alerts_raw_topic: str) -> EventValidation:+ """Dispatch validation by the topic a message arrived on."""+ if topic == raw_events_topic:+ return validate_raw_event(payload, topic="aisoc.raw_events")+ # Everything else is treated as a RawAlert envelope.+ return validate_raw_alert(payload, topic="aisoc.alerts.raw")diff --git a/services/fusion/app/workers/consumer.py b/services/fusion/app/workers/consumer.pyindex 446d35c72..77860fd6c 100644--- a/services/fusion/app/workers/consumer.py+++ b/services/fusion/app/workers/consumer.py@@ -12,6 +12,8 @@ from aiokafka import AIOKafkaConsumer, AIOKafkaProducerfrom app.core.config import settingsfrom app.models.alert import FusionDecision, RawAlertfrom app.services.alert_sink import AlertSink+from app.services.dlq import DeadLetter, DeadLetterQueue, LoggingDLQ, safe_record+from app.services.event_schema import validate_eventfrom app.services.fusion_engine import FusionEnginefrom app.services.promoter import promote_normalized_event@@ -25,6 +27,7 @@ _METRICS = {"promoted": 0,"not_promoted": 0,"persisted": 0,+ "dead_lettered": 0,"errors": 0,}@@ -32,9 +35,17 @@ _METRICS = {class FusionWorker:"""Kafka consumer/producer pair that drives the fusion pipeline."""- def __init__(self, engine: FusionEngine, sink: AlertSink | None = None) -> None:+ def __init__(+ self,+ engine: FusionEngine,+ sink: AlertSink | None = None,+ dlq: DeadLetterQueue | None = None,+ ) -> None:self._engine = engineself._sink = sink+ # A poison message must never vanish silently; default to a structured+ # logging DLQ so persistence-free deployments still get the signal.+ self._dlq: DeadLetterQueue = dlq or LoggingDLQ()self._consumer: AIOKafkaConsumer | None = Noneself._producer: AIOKafkaProducer | None = Noneself._running = False@@ -93,7 +104,53 @@ class FusionWorker:_METRICS["errors"] += 1logger.error("Failed to process message", error=str(exc), exc_info=True)+ async def _dead_letter(+ self,+ *,+ topic: str,+ payload,+ reason: str,+ schema_version: str = "v1",+ source_event_id: str | None = None,+ tenant_id: str | None = None,+ ) -> None:+ """Route a poison message to the DLQ instead of dropping it silently."""+ _METRICS["dead_lettered"] += 1+ await safe_record(+ self._dlq,+ DeadLetter.build(+ topic=topic,+ reason=reason,+ schema_version=schema_version,+ payload=payload,+ source_event_id=source_event_id,+ tenant_id=tenant_id,+ ),+ )+async def _process_message(self, payload: dict, topic: str | None = None) -> None:+ resolved_topic = topic or settings.kafka_topic_alerts_raw++ # Phase 5 — schema-validate the envelope BEFORE the promoter sees it.+ # A malformed / mis-versioned / mis-tenanted message is dead-lettered+ # (captured with its reason + lineage), never silently dropped.+ validation = validate_event(+ resolved_topic,+ payload,+ raw_events_topic=settings.kafka_topic_raw_events,+ alerts_raw_topic=settings.kafka_topic_alerts_raw,+ )+ if not validation.ok:+ await self._dead_letter(+ topic=resolved_topic,+ payload=payload,+ reason=validation.reason,+ schema_version=validation.schema_version,+ source_event_id=validation.source_event_id,+ tenant_id=validation.tenant_id,+ )+ return+if topic == settings.kafka_topic_raw_events:# Ingest-normalized OCSF event — run the deterministic promotion# policy (see app/services/promoter.py). Non-promoted events are@@ -107,10 +164,26 @@ class FusionWorker:try:alert = RawAlert.model_validate(payload)except Exception as exc:- logger.warning("Invalid alert payload", error=str(exc))- _METRICS["errors"] += 1+ # Deep validation failure — dead-letter it (the schema layer+ # only caught gross shape); never silently drop.+ await self._dead_letter(+ topic=resolved_topic,+ payload=payload,+ reason=f"RawAlert validation failed: {exc}",+ schema_version=validation.schema_version,+ source_event_id=validation.source_event_id,+ tenant_id=validation.tenant_id,+ )return+ # Lineage — trace each fused alert back to its source event + schema.+ logger.debug(+ "fusion.lineage",+ topic=resolved_topic,+ schema_version=validation.schema_version,+ source_event_id=validation.source_event_id,+ )+fused = await self._engine.process(alert)_METRICS["processed"] += 1diff --git a/services/fusion/tests/test_consumer_dlq.py b/services/fusion/tests/test_consumer_dlq.pynew file mode 100644index 000000000..a347db1c7--- /dev/null+++ b/services/fusion/tests/test_consumer_dlq.py@@ -0,0 +1,109 @@+"""Phase 5 — the fusion consumer dead-letters poison messages (never drops).++Before Phase 5 the consumer logged a warning and dropped a malformed message —+silent data loss. These tests drive ``FusionWorker._process_message`` directly+(no Kafka) with an in-memory DLQ and prove: a valid event is processed and the+DLQ stays empty; every poison shape is captured in the DLQ with a reason; and a+DLQ that itself fails does not crash the consumer.+"""++from __future__ import annotations++from types import SimpleNamespace+from unittest.mock import AsyncMock++import pytest+from app.core.config import settings+from app.models.alert import FusionDecision+from app.services.dlq import InMemoryDLQ+from app.workers.consumer import FusionWorker++pytestmark = pytest.mark.asyncio++TENANT = "00000000-0000-0000-0000-000000000001"+RAW_EVENTS = settings.kafka_topic_raw_events+ALERTS_RAW = settings.kafka_topic_alerts_raw+++def _valid_raw_event() -> dict:+ return {+ "tenant_id": TENANT,+ "ocsf_event": {+ "class_uid": 2001,+ "category_uid": 2,+ "severity_id": 5,+ "message": "Credential dumping",+ "metadata": {"uid": "ev-1", "product": {"name": "Falcon", "vendor_name": "CrowdStrike"}},+ },+ }+++def _worker_with_dlq() -> tuple[FusionWorker, InMemoryDLQ]:+ dlq = InMemoryDLQ()+ engine = SimpleNamespace(+ process=AsyncMock(+ return_value=SimpleNamespace(+ fusion_decision=FusionDecision.NEW_ALERT,+ model_dump=lambda mode="json": {"id": "fused-1"},+ )+ )+ )+ worker = FusionWorker(engine=engine, sink=None, dlq=dlq)+ worker._producer = AsyncMock() # noqa: SLF001 — test drives the internals+ return worker, dlq+++async def test_valid_event_is_processed_and_not_dead_lettered():+ worker, dlq = _worker_with_dlq()+ await worker._process_message(_valid_raw_event(), topic=RAW_EVENTS) # noqa: SLF001+ assert dlq.records == []+ worker._engine.process.assert_awaited_once() # noqa: SLF001+ worker._producer.send.assert_awaited_once() # noqa: SLF001+++async def test_missing_ocsf_event_is_dead_lettered_not_dropped():+ worker, dlq = _worker_with_dlq()+ await worker._process_message({"tenant_id": TENANT}, topic=RAW_EVENTS) # noqa: SLF001+ assert len(dlq.records) == 1+ assert "ocsf_event" in dlq.records[0].reason+ worker._engine.process.assert_not_awaited() # noqa: SLF001+++async def test_unknown_schema_version_is_dead_lettered():+ worker, dlq = _worker_with_dlq()+ poison = _valid_raw_event()+ poison["schema_version"] = "v99"+ await worker._process_message(poison, topic=RAW_EVENTS) # noqa: SLF001+ assert len(dlq.records) == 1+ assert "unknown schema_version" in dlq.records[0].reason+++async def test_non_uuid_tenant_is_dead_lettered_with_lineage():+ worker, dlq = _worker_with_dlq()+ poison = _valid_raw_event()+ poison["tenant_id"] = "not-a-uuid"+ poison["ocsf_event"].pop("tenant_uid", None)+ await worker._process_message(poison, topic=RAW_EVENTS) # noqa: SLF001+ assert len(dlq.records) == 1+ assert dlq.records[0].source_event_id == "ev-1"+++async def test_invalid_raw_alert_is_dead_lettered_not_silently_dropped():+ worker, dlq = _worker_with_dlq()+ # Missing every required RawAlert field — deep Pydantic validation fails.+ await worker._process_message({"tenant_id": TENANT, "nonsense": True}, topic=ALERTS_RAW) # noqa: SLF001+ assert len(dlq.records) == 1+ assert "RawAlert validation failed" in dlq.records[0].reason+ worker._engine.process.assert_not_awaited() # noqa: SLF001+++async def test_dead_letter_never_crashes_the_consumer_when_sink_fails():+ class ExplodingDLQ:+ async def record(self, dead_letter): # noqa: ANN001, ARG002+ raise RuntimeError("dlq sink down")++ engine = SimpleNamespace(process=AsyncMock())+ worker = FusionWorker(engine=engine, sink=None, dlq=ExplodingDLQ())+ worker._producer = AsyncMock() # noqa: SLF001+ # Poison message + failing DLQ: must return cleanly, not raise.+ await worker._process_message({"tenant_id": TENANT}, topic=RAW_EVENTS) # noqa: SLF001diff --git a/services/fusion/tests/test_dlq.py b/services/fusion/tests/test_dlq.pynew file mode 100644index 000000000..15a8f3731--- /dev/null+++ b/services/fusion/tests/test_dlq.py@@ -0,0 +1,63 @@+"""Phase 5 — dead-letter queue tests."""++from __future__ import annotations++import pytest+from app.services.dlq import DeadLetter, InMemoryDLQ, LoggingDLQ, safe_record+++def test_dead_letter_build_truncates_payload_and_keeps_lineage():+ huge = {"blob": "x" * 10000, "n": 1}+ dl = DeadLetter.build(+ topic="aisoc.raw_events",+ reason="test",+ schema_version="v1",+ payload=huge,+ source_event_id="ev-9",+ tenant_id="t-1",+ )+ assert len(dl.payload_excerpt) <= 4000+ assert dl.source_event_id == "ev-9"+ assert dl.tenant_id == "t-1"+ assert dl.ts # timestamped+++def test_dead_letter_build_handles_unserialisable_payload():+ dl = DeadLetter.build(topic="t", reason="r", schema_version="v1", payload=object())+ assert dl.payload_excerpt # falls back to str(), never raises+++async def test_in_memory_dlq_records():+ dlq = InMemoryDLQ()+ dl = DeadLetter.build(topic="t", reason="r", schema_version="v1", payload={"a": 1})+ ok = await safe_record(dlq, dl)+ assert ok+ assert len(dlq.records) == 1+ assert dlq.records[0].reason == "r"+++async def test_safe_record_is_none_safe():+ assert await safe_record(None, DeadLetter.build(topic="t", reason="r", schema_version="v1", payload={})) is False+++async def test_safe_record_swallows_sink_failure():+ class ExplodingDLQ:+ async def record(self, dead_letter): # noqa: ANN001, ARG002+ raise RuntimeError("kafka down")++ # A DLQ that itself fails must never propagate — one poison message can't+ # be allowed to crash the consumer.+ ok = await safe_record(ExplodingDLQ(), DeadLetter.build(topic="t", reason="r", schema_version="v1", payload={}))+ assert ok is False+++async def test_logging_dlq_does_not_raise():+ await LoggingDLQ().record(DeadLetter.build(topic="t", reason="r", schema_version="v1", payload={"x": 1}))+++if __name__ == "__main__": # pragma: no cover+ raise SystemExit(pytest.main([__file__, "-v"]))diff --git a/services/fusion/tests/test_event_schema.py b/services/fusion/tests/test_event_schema.pynew file mode 100644index 000000000..d8fa49a52--- /dev/null+++ b/services/fusion/tests/test_event_schema.py@@ -0,0 +1,81 @@+"""Phase 5 — versioned event-schema registry tests."""++from __future__ import annotations++from app.services.event_schema import (+ KNOWN_SCHEMA_VERSIONS,+ validate_event,+ validate_raw_alert,+ validate_raw_event,+)++TENANT = "00000000-0000-0000-0000-000000000001"+++def _raw_event(**overrides) -> dict:+ base = {"tenant_id": TENANT, "ocsf_event": {"class_uid": 2001, "metadata": {"uid": "ev-1"}}}+ base.update(overrides)+ return base+++def test_valid_raw_event_passes_and_carries_lineage():+ v = validate_raw_event(_raw_event())+ assert v.ok+ assert v.schema_version == "v1"+ assert v.source_event_id == "ev-1"+ assert v.tenant_id == TENANT+++def test_non_object_payload_is_rejected():+ v = validate_raw_event("not a dict")+ assert not v.ok+ assert "not a JSON object" in v.reason+++def test_missing_ocsf_event_is_rejected():+ v = validate_raw_event({"tenant_id": TENANT})+ assert not v.ok+ assert "ocsf_event" in v.reason+++def test_unknown_schema_version_is_rejected():+ v = validate_raw_event(_raw_event(schema_version="v99"))+ assert not v.ok+ assert "unknown schema_version" in v.reason+++def test_non_uuid_tenant_is_rejected_with_lineage():+ v = validate_raw_event(_raw_event(tenant_id="not-a-uuid"))+ assert not v.ok+ assert "tenant_id is not a UUID" in v.reason+ # Lineage is still captured so an operator can find the offending event.+ assert v.source_event_id == "ev-1"+++def test_tenant_uid_fallback_inside_ocsf():+ payload = {"ocsf_event": {"tenant_uid": TENANT, "class_uid": 2001}}+ v = validate_raw_event(payload)+ assert v.ok+ assert v.tenant_id == TENANT+++def test_raw_alert_structural_check():+ assert validate_raw_alert({"tenant_id": TENANT, "title": "x"}).ok+ bad = validate_raw_alert([1, 2, 3])+ assert not bad.ok+++def test_dispatch_by_topic():+ ev = validate_event("aisoc.raw_events", _raw_event(), raw_events_topic="aisoc.raw_events", alerts_raw_topic="aisoc.alerts.raw")+ assert ev.ok+ al = validate_event("aisoc.alerts.raw", {"tenant_id": TENANT}, raw_events_topic="aisoc.raw_events", alerts_raw_topic="aisoc.alerts.raw")+ assert al.ok+++def test_registered_schema_set_is_stable_drift_guard():+ """The wire contract must not move unnoticed. If you intentionally add or+ bump an envelope version, update this expected set in the same commit."""+ assert KNOWN_SCHEMA_VERSIONS == {+ "aisoc.raw_events": frozenset({"v1"}),+ "aisoc.alerts.raw": frozenset({"v1"}),+ }