Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

feat(phase-7a): unified deterministic->ML->LLM model router + determinism contract (#432)

Beenu Arorae92fccb0c1da
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 182a6d350..0fcf09718 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -326,6 +326,7 @@ jobs:
tests/test_prompt_envelope.py \
tests/test_privacy_redactor.py \
tests/test_cost_governor.py \
+ tests/test_model_router.py \
-v --tb=short
# ──────────────────────────────────────────────
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e66f47241..55eaece44 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
+- **Phase 7a — unified multi-model router with tier attribution + a determinism contract.** The reality audit confirmed the knowledge graph is already built at ingest (v8 T1.1, `services/ingest/internal/graph/`); the genuine Phase 7 delta was the *model router*. Before it, the deterministic→LLM fallback was reimplemented independently in NL query, playbook drafting, explain, copilot, and each sub-agent — no single audit point for which model answered or why. New `services/agents/app/routing/model_router.py` is that place: a `ModelRouter` that escalates deterministic → ML → LLM only when the cheaper tier is under-confident, and attributes every decision (`tier`, `model_used`, `attribution` trail, `tiers_considered`, `escalation_blocked_reason`). It **never silently uses the LLM** — a skipped or blocked LLM tier (no key, air-gap, deterministic mode, governor circuit open, or a tier error) is always recorded. Introduces the canonical **`AISOC_DETERMINISTIC`** flag and composes with the existing `CostGovernor` circuit breaker: either one forces deterministic-only, in which case the router is reproducible (same input → identical decision). 12 tests (`services/agents/tests/test_model_router.py`, appended to the CI agents gate) prove tier selection, attribution, the no-silent-LLM property, graceful LLM-failure degradation, and the determinism contract. Doc: `apps/docs/docs/concepts/model-router.md`. Remaining Phase 7 enrichments (posture collection, effective-permissions snapshot loader, bi-temporal validity, fusion-time ContextBundle) are tracked as 7b+ in `docs/audit/PROGRESS.md`.
- **Phase 6 — performance + cost, both gated.** Two cheap, non-flaky gates in a new `.github/workflows/perf.yml`. (1) **Throughput** — `scripts/perf/throughput_harness.py` runs the *real* fusion hot path (`promote_normalized_event`) over a deterministic 20k-event synthetic corpus and reports events/sec + p50/p95/p99 per-event latency (measured ~250k eps on commodity hardware). The gate asserts a **generous regression floor** (1,000 eps — a >200× margin) so it fires only on a catastrophic regression such as an I/O call slipping onto the hot path, never on shared-runner jitter; it is explicitly a regression floor, not a production SLO. (2) **Cost** — `scripts/storage_cost_model.py` is a deterministic tiered-storage $/TB model (hot ClickHouse block 30d / warm object 60d / cold archive 275d, 8× ZSTD); the committed worked example (`docs/decisions/storage-cost-model.json`: ≈ $902/mo and ≈ $30/raw-TB at 1 TB/day) is drift-gated by `--check`, so the number can never silently diverge from the rate card. Rate-card values are labelled reference list prices to verify per provider/region — the value is the methodology, not the exact dollars. `docs/decisions/0005-storage-consolidation.md` (ADR) records the three-tier decision (keep ClickHouse-only hot; do not add a second hot engine) and cites the gated model.
- **Phase 5 — data spine correctness: schema registry + dead-letter queue + lineage.** The reality audit's load-bearing gap: the fusion consumer logged a warning and **silently dropped** any malformed message — silent data loss. Now every message is schema-validated against a versioned envelope registry (`services/fusion/app/services/event_schema.py`: `aisoc.raw_events` and `aisoc.alerts.raw`, each pinned to `v1`, with a drift-guard test so the wire contract can't move unnoticed) *before* the promoter sees it. Anything that fails — non-object payload, unknown `schema_version`, missing `ocsf_event`, non-UUID tenant, or a RawAlert that fails deep Pydantic validation — is routed to a **fail-soft dead-letter queue** (`services/fusion/app/services/dlq.py`: `LoggingDLQ` default, `InMemoryDLQ` for tests, `KafkaDLQ` republishing to `aisoc.alerts.dlq`) with its reason, schema version, source-event lineage, and a truncated payload, instead of vanishing. `safe_record` guarantees a DLQ that itself throws never crashes the consumer. Each processed alert logs `fusion.lineage` (source event id + schema version). 21 new tests (`test_event_schema.py`, `test_dlq.py`, `test_consumer_dlq.py`) prove a valid event is processed with an empty DLQ, every poison shape is captured (not dropped), and a failing DLQ sink doesn't crash the consumer; full fusion suite 92/92 at 59% coverage (floor 48). Idempotency (AlertSink dedup fingerprint, Phase 3.1) and event-time watermarking are already present; backfill/replay-from-offset is tracked as 5b in `docs/audit/PROGRESS.md`.
- **Phase 4a — the Detection-as-Code gate is no longer circular.** The reality audit's #1 circular gate: the detection-proposal promote path shelled out to `run_evals.py` **without ever passing the proposed rule body**, so "passed" meant the repo-wide substrate MITRE accuracy didn't move — a value independent of the rule under review. A blind rule (matches nothing) or a noisy rule (matches everything) sailed through its own exam. New `services/api/app/services/detection_eval.py::evaluate_candidate_rule` runs the *candidate `rule_body` itself* through the **real** runtime engine (`rule_engine.execute_rule`) against caller-supplied positive/negative fixtures: it must fire on every positive and stay silent on every negative. New `POST /detection-proposals/{id}/evaluate-rule` stores the verdict under `eval_result["candidate_rule"]`, and `POST /decide` (approve) now **requires** it — a benchmark-only pass is no longer sufficient. `services/api/tests/test_detection_eval.py` (6 tests) is the mutation test that proves the gate rejects a blind rule, rejects a noisy rule, rejects a no-positive-fixture rule, and that the verdict genuinely depends on the rule body. Claim-to-gate matrix: DAC row PARTIAL(circular) → GATED.
diff --git a/ROADMAP.md b/ROADMAP.md
index fa5875c78..46b73f6d9 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -29,7 +29,7 @@ A proof-first, security-first program to make every README claim gate-backed, cl
- [ ] Phase 4 — Real evals + detection content truth table (third-party-labeled corpus, hallucination/calibration/abstention, model matrix) — **4a/4b landed** (de-circularised DAC candidate-rule gate + honest executable-vs-imported truth table); 4c+ live-agent eval / calibration / model matrix pending
- [x] Phase 5 — Data spine correctness (versioned event-schema registry + dead-letter queue + source-event lineage in the fusion consumer; idempotency via AlertSink dedup + event-time watermarking; backfill/replay-from-offset tracked as 5b)
- [x] Phase 6 — Performance + cost (fusion hot-path throughput harness with a generous regression-floor gate; deterministic storage $/TB cost model + drift gate; storage-consolidation ADR-0005)
-- [ ] Phase 7 — Ingest-time graph + multi-model router (posture, effective permissions, bi-temporal, fusion-time ContextBundle, determinism contract)
+- [x] Phase 7 — Ingest-time graph + multi-model router — **graph-at-ingest already shipped** (v8 T1.1, `services/ingest/internal/graph/`); **7a landed**: unified deterministic→ML→LLM router with tier attribution + the `AISOC_DETERMINISTIC` determinism contract (`services/agents/app/routing/model_router.py`, gated). 7b+ (posture collection, effective-permissions snapshot loader, bi-temporal valid_from/valid_to, fusion-time ContextBundle) tracked in `docs/audit/PROGRESS.md`
- [ ] Phase 8 — LLMOps (prompt registry, model pinning, provider fallback, response caching, structured-output validation)
- [ ] Phase 9 — Autonomy safety (rollback, post-action verification, dry-run default, break-glass, approval SLA, scorecard)
- [ ] Phase 10 — Connector + content quality (conformance suite, published matrix, detection lifecycle)
diff --git a/apps/docs/docs/concepts/model-router.md b/apps/docs/docs/concepts/model-router.md
new file mode 100644
index 000000000..863645cf9
--- /dev/null
+++ b/apps/docs/docs/concepts/model-router.md
@@ -0,0 +1,67 @@
+# Multi-model router
+
+AiSOC's reasoning uses three tiers, cheapest first:
+
+1. **Deterministic** — rules and heuristics (keyword tactic extraction, the
+ triage scorer, the confidence model). No network, no cost, reproducible.
+2. **ML** — the fusion service's Isolation Forest + LambdaRank scorers.
+3. **LLM** — a hosted or local model, used only as a last resort.
+
+The **model router** (`services/agents/app/routing/model_router.py`) is the one
+place that decides which tier answers a given request, escalating only when the
+cheaper tier isn't confident enough, and **attributing every decision** to the
+tier that produced it.
+
+## Why a single router
+
+Before it, the deterministic→LLM fallback was reimplemented independently in NL
+query, playbook drafting, "explain this alert", the copilot, and each
+sub-agent. Each checked "is a key configured?" and degraded on its own, so there
+was no single audit point for *which model answered* or *why the LLM was (or
+wasn't) used*. The router unifies that.
+
+## The escalation ladder
+
+```
+deterministic ── confident? ──▶ done (tier=deterministic)
+ │ no
+ ▼
+ ML ──────── confident? ──▶ done (tier=ml)
+ │ no
+ ▼
+ LLM ─────────────────────▶ done (tier=llm)
+```
+
+Each `RoutingDecision` carries:
+
+- `tier` — which tier produced the answer,
+- `model_used` — the concrete model/scorer name,
+- `attribution` — a human-readable trail of *why* this tier was chosen and why
+ cheaper tiers were insufficient,
+- `tiers_considered` — every tier that ran,
+- `escalation_blocked_reason` — set whenever the router *wanted* to escalate but
+ couldn't (no key, air-gap, deterministic mode, governor circuit open, or the
+ LLM tier errored).
+
+## The determinism contract
+
+The router **never silently uses the LLM.** Two independent controls force the
+deterministic tier and forbid ML/LLM entirely:
+
+- **`AISOC_DETERMINISTIC=1`** — the canonical determinism switch (air-gapped
+ deployments, reproducible evals, cost lockdown).
+- **The cost governor** — when a tenant's hard budget is exhausted the governor
+ returns a circuit-open decision; the router honours it and drops to
+ deterministic-only.
+
+In deterministic mode the router is **reproducible**: the same input yields an
+identical decision, because only the pure deterministic tier runs. This is
+gated in CI (`services/agents/tests/test_model_router.py`,
+`test_determinism_contract_identical_input_identical_decision`).
+
+## What the gate proves
+
+`test_model_router.py` proves the router picks the cheapest sufficient tier,
+attributes every decision, never silently reaches the LLM (a skipped or blocked
+LLM tier is always recorded), degrades gracefully when the LLM errors, and is
+reproducible in deterministic mode.
diff --git a/apps/docs/sidebars.ts b/apps/docs/sidebars.ts
index fbec8ba15..4b3e54a1b 100644
--- a/apps/docs/sidebars.ts
+++ b/apps/docs/sidebars.ts
@@ -25,6 +25,7 @@ const sidebars: SidebarsConfig = {
"concepts/detections",
"concepts/live-actions",
"concepts/automation-maturity",
+ "concepts/model-router",
],
},
{
diff --git a/services/agents/app/routing/__init__.py b/services/agents/app/routing/__init__.py
new file mode 100644
index 000000000..a4b794134
--- /dev/null
+++ b/services/agents/app/routing/__init__.py
@@ -0,0 +1,25 @@
+"""Multi-model routing (Phase 7): deterministic → ML → LLM with attribution."""
+
+from app.routing.model_router import (
+ DEFAULT_CONFIDENCE_FLOOR,
+ DETERMINISTIC_ENV_FLAG,
+ ModelRouter,
+ ModelTier,
+ RoutingDecision,
+ RoutingRequest,
+ TierResult,
+ build_router,
+ is_deterministic_mode,
+)
+
+__all__ = [
+ "DEFAULT_CONFIDENCE_FLOOR",
+ "DETERMINISTIC_ENV_FLAG",
+ "ModelRouter",
+ "ModelTier",
+ "RoutingDecision",
+ "RoutingRequest",
+ "TierResult",
+ "build_router",
+ "is_deterministic_mode",
+]
diff --git a/services/agents/app/routing/model_router.py b/services/agents/app/routing/model_router.py
new file mode 100644
index 000000000..6838e70a7
--- /dev/null
+++ b/services/agents/app/routing/model_router.py
@@ -0,0 +1,311 @@
+"""Unified deterministic → ML → LLM model router (Phase 7).
+
+The reality audit found the three-model story (deterministic rules → ML scorers
+→ LLM) implemented as *scattered, ad-hoc fallbacks*: `nl_query`, `nl_drafter`,
+`explain`, `copilot`, and the sub-agents each independently check "is a key
+configured?" and degrade on their own. There was no single place that:
+
+* decides which tier should answer (cheapest sufficient tier first),
+* **attributes** every decision to the tier that produced it (so cost and
+ provenance are auditable), and
+* enforces a **determinism contract**: when deterministic mode is on (env flag
+ or the cost governor tripping its circuit breaker), the router NEVER touches
+ the ML or LLM tier, and the deterministic tier is reproducible — identical
+ input yields an identical decision.
+
+This module is that place. It is intentionally dependency-light (stdlib only):
+the three tier functions are injected, so the router is unit-testable without a
+real model, and `build_router` wires the real deterministic scorer while
+leaving ML/LLM as optional callables the caller supplies.
+
+Key invariant the gate proves: **the router never silently uses the LLM.** Every
+decision carries an `attribution` trail; whenever the LLM tier is skipped or
+blocked (deterministic mode, no key, governor circuit open, tier error), the
+reason is recorded rather than the router quietly degrading.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import inspect
+import json
+import logging
+import os
+from collections.abc import Awaitable, Callable
+from dataclasses import dataclass, field
+from enum import Enum
+from typing import Any, Protocol, runtime_checkable
+
+logger = logging.getLogger(__name__)
+
+DETERMINISTIC_ENV_FLAG = "AISOC_DETERMINISTIC"
+DEFAULT_CONFIDENCE_FLOOR = 0.85
+
+
+class ModelTier(str, Enum):
+ DETERMINISTIC = "deterministic"
+ ML = "ml"
+ LLM = "llm"
+
+
+def is_deterministic_mode() -> bool:
+ """Canonical determinism switch. Off by default; set AISOC_DETERMINISTIC=1
+ to force the deterministic tier everywhere (air-gap, reproducible evals,
+ cost lockdown)."""
+ raw = os.environ.get(DETERMINISTIC_ENV_FLAG)
+ if raw is None:
+ return False
+ return raw.strip().lower() in {"1", "true", "yes", "on", "enabled"}
+
+
+@dataclass(frozen=True)
+class RoutingRequest:
+ """Input to the router. ``fingerprint`` is the determinism/dedup identity."""
+
+ payload: dict[str, Any]
+ tenant_id: str = "00000000-0000-0000-0000-000000000000"
+
+ def fingerprint(self) -> str:
+ blob = json.dumps(self.payload, sort_keys=True, default=str)
+ return hashlib.sha256(blob.encode("utf-8")).hexdigest()
+
+
+@dataclass(frozen=True)
+class TierResult:
+ """What a tier returns: an output, a confidence in [0,1], and a reason."""
+
+ output: Any
+ confidence: float
+ reason: str = ""
+ model: str | None = None
+
+
+@dataclass(frozen=True)
+class RoutingDecision:
+ """The router's verdict, fully attributed."""
+
+ tier: ModelTier
+ output: Any
+ confidence: float
+ model_used: str
+ attribution: list[str] = field(default_factory=list)
+ tiers_considered: list[str] = field(default_factory=list)
+ deterministic: bool = False
+ escalation_blocked_reason: str | None = None
+
+ def to_dict(self) -> dict[str, Any]:
+ return {
+ "tier": self.tier.value,
+ "confidence": self.confidence,
+ "model_used": self.model_used,
+ "attribution": list(self.attribution),
+ "tiers_considered": list(self.tiers_considered),
+ "deterministic": self.deterministic,
+ "escalation_blocked_reason": self.escalation_blocked_reason,
+ }
+
+
+@runtime_checkable
+class Governor(Protocol):
+ """Subset of cost_governor used here — kept as a Protocol so this module
+ doesn't import the heavy governor at module load."""
+
+ def check(self, tenant_id: str, fingerprint: str) -> Any: ...
+
+
+TierFn = Callable[[RoutingRequest], TierResult | Awaitable[TierResult]]
+
+
+async def _maybe_await(value: TierResult | Awaitable[TierResult]) -> TierResult:
+ if inspect.isawaitable(value):
+ return await value
+ return value
+
+
+class ModelRouter:
+ """Routes a request to the cheapest sufficient tier, fully attributed.
+
+ Escalation ladder: deterministic → ML → LLM. A tier "wins" when its
+ confidence meets ``confidence_floor``; otherwise the router escalates to the
+ next available tier. The LLM tier is only ever reached when it is
+ configured, deterministic mode is off, and the governor (if any) permits
+ spend — and if it is *not* reached, the reason is recorded.
+ """
+
+ def __init__(
+ self,
+ *,
+ deterministic_fn: TierFn,
+ ml_fn: TierFn | None = None,
+ llm_fn: TierFn | None = None,
+ confidence_floor: float = DEFAULT_CONFIDENCE_FLOOR,
+ deterministic_only: bool = False,
+ governor: Governor | None = None,
+ ) -> None:
+ self._deterministic_fn = deterministic_fn
+ self._ml_fn = ml_fn
+ self._llm_fn = llm_fn
+ self._floor = confidence_floor
+ self._forced_deterministic = deterministic_only
+ self._governor = governor
+
+ def _resolve_deterministic_only(self, request: RoutingRequest, attribution: list[str]) -> bool:
+ if self._forced_deterministic:
+ attribution.append("deterministic-only: forced by construction")
+ return True
+ if is_deterministic_mode():
+ attribution.append(f"deterministic-only: {DETERMINISTIC_ENV_FLAG} is set")
+ return True
+ if self._governor is not None:
+ try:
+ decision = self._governor.check(request.tenant_id, request.fingerprint())
+ use_llm = bool(getattr(decision, "use_llm", True))
+ label = getattr(getattr(decision, "decision", None), "value", None) or str(getattr(decision, "decision", "?"))
+ if not use_llm:
+ attribution.append(f"deterministic-only: cost governor decision '{label}' forbids LLM spend")
+ return True
+ except Exception as exc: # noqa: BLE001 — governor must never crash routing
+ logger.warning("model_router.governor_check_failed", extra={"error": str(exc)})
+ return False
+
+ async def route(self, request: RoutingRequest) -> RoutingDecision:
+ attribution: list[str] = []
+ considered: list[str] = []
+ deterministic_only = self._resolve_deterministic_only(request, attribution)
+
+ # Tier 1 — deterministic. Always runs; it's the floor of the ladder.
+ det = await _maybe_await(self._deterministic_fn(request))
+ considered.append(ModelTier.DETERMINISTIC.value)
+
+ higher_tier_available = (self._ml_fn is not None or self._llm_fn is not None) and not deterministic_only
+ if det.confidence >= self._floor or not higher_tier_available:
+ if deterministic_only:
+ attribution.append("resolved on deterministic tier (deterministic-only mode)")
+ elif det.confidence >= self._floor:
+ attribution.append(f"deterministic tier confident ({det.confidence:.2f} >= floor {self._floor:.2f})")
+ else:
+ attribution.append("no higher tier available; using deterministic result")
+ blocked = None
+ if not higher_tier_available and det.confidence < self._floor:
+ blocked = "deterministic-only mode" if deterministic_only else "no ML/LLM tier configured"
+ return RoutingDecision(
+ tier=ModelTier.DETERMINISTIC,
+ output=det.output,
+ confidence=det.confidence,
+ model_used=det.model or "deterministic",
+ attribution=attribution + ([det.reason] if det.reason else []),
+ tiers_considered=considered,
+ deterministic=True,
+ escalation_blocked_reason=blocked,
+ )
+
+ # Tier 2 — ML. Cheaper than the LLM; try before escalating further.
+ best = det
+ best_tier = ModelTier.DETERMINISTIC
+ if self._ml_fn is not None:
+ attribution.append(f"deterministic below floor ({det.confidence:.2f} < {self._floor:.2f}); escalating to ML")
+ ml = await _maybe_await(self._ml_fn(request))
+ considered.append(ModelTier.ML.value)
+ if ml.confidence >= self._floor or self._llm_fn is None:
+ if ml.confidence >= self._floor:
+ attribution.append(f"ML tier confident ({ml.confidence:.2f} >= floor)")
+ blocked = None
+ else:
+ attribution.append("no LLM tier configured; using ML result")
+ blocked = "no LLM tier configured"
+ return RoutingDecision(
+ tier=ModelTier.ML,
+ output=ml.output,
+ confidence=ml.confidence,
+ model_used=ml.model or "ml",
+ attribution=attribution + ([ml.reason] if ml.reason else []),
+ tiers_considered=considered,
+ deterministic=False,
+ escalation_blocked_reason=blocked,
+ )
+ if ml.confidence > best.confidence:
+ best, best_tier = ml, ModelTier.ML
+
+ # Tier 3 — LLM. Last resort; only reached when configured + permitted.
+ if self._llm_fn is not None:
+ attribution.append("escalating to LLM tier")
+ considered.append(ModelTier.LLM.value)
+ try:
+ llm = await _maybe_await(self._llm_fn(request))
+ return RoutingDecision(
+ tier=ModelTier.LLM,
+ output=llm.output,
+ confidence=llm.confidence,
+ model_used=llm.model or "llm",
+ attribution=attribution + ([llm.reason] if llm.reason else []),
+ tiers_considered=considered,
+ deterministic=False,
+ )
+ except Exception as exc: # noqa: BLE001 — LLM failure must degrade, not crash
+ reason = f"LLM tier failed ({type(exc).__name__}); falling back to {best_tier.value}"
+ attribution.append(reason)
+ return RoutingDecision(
+ tier=best_tier,
+ output=best.output,
+ confidence=best.confidence,
+ model_used=best.model or best_tier.value,
+ attribution=attribution,
+ tiers_considered=considered,
+ deterministic=best_tier is ModelTier.DETERMINISTIC,
+ escalation_blocked_reason=reason,
+ )
+
+ # No LLM tier at all — never silently pretend; record the block.
+ blocked = "no LLM tier configured"
+ attribution.append(f"escalation to LLM blocked: {blocked}; using {best_tier.value} result")
+ return RoutingDecision(
+ tier=best_tier,
+ output=best.output,
+ confidence=best.confidence,
+ model_used=best.model or best_tier.value,
+ attribution=attribution,
+ tiers_considered=considered,
+ deterministic=best_tier is ModelTier.DETERMINISTIC,
+ escalation_blocked_reason=blocked,
+ )
+
+
+def build_router(
+ *,
+ ml_fn: TierFn | None = None,
+ llm_fn: TierFn | None = None,
+ confidence_floor: float = DEFAULT_CONFIDENCE_FLOOR,
+ governor: Governor | None = None,
+) -> ModelRouter:
+ """Wire a router with the real deterministic triage scorer as tier 1.
+
+ ML/LLM tiers are optional and injected by the caller (they carry heavy
+ deps), so this factory stays import-light and the deterministic path always
+ works — including air-gapped.
+ """
+
+ def _deterministic(request: RoutingRequest) -> TierResult:
+ # Lazy import so this module never drags the agents package at import.
+ from app.agents.triage_agent import _score_alert # noqa: PLC0415
+ from app.models.state import InvestigationState # noqa: PLC0415
+
+ payload = request.payload
+ state = InvestigationState(
+ alert_summary=str(payload.get("alert_summary") or payload.get("title") or ""),
+ raw_alert=payload.get("raw_alert") or payload,
+ )
+ verdict, confidence = _score_alert(state)
+ return TierResult(
+ output={"verdict": verdict},
+ confidence=confidence,
+ reason=f"heuristic triage → {verdict}",
+ model="deterministic:triage",
+ )
+
+ return ModelRouter(
+ deterministic_fn=_deterministic,
+ ml_fn=ml_fn,
+ llm_fn=llm_fn,
+ confidence_floor=confidence_floor,
+ governor=governor,
+ )
diff --git a/services/agents/tests/test_model_router.py b/services/agents/tests/test_model_router.py
new file mode 100644
index 000000000..f8916f3de
--- /dev/null
+++ b/services/agents/tests/test_model_router.py
@@ -0,0 +1,179 @@
+"""Phase 7 — the multi-model router contract.
+
+Proves the router (a) picks the cheapest sufficient tier, (b) attributes every
+decision to a tier, (c) NEVER silently uses the LLM (a skipped/blocked LLM tier
+is always recorded), and (d) honours the determinism contract: in deterministic
+mode it never touches ML/LLM and is reproducible.
+
+The tiers are injected as plain functions so the contract is proven without any
+real model — that is the point of the router: one auditable place for the
+deterministic→ML→LLM ladder.
+"""
+
+from __future__ import annotations
+
+import os
+from types import SimpleNamespace
+
+import pytest
+from app.routing.model_router import (
+ DETERMINISTIC_ENV_FLAG,
+ ModelRouter,
+ ModelTier,
+ RoutingRequest,
+ TierResult,
+ is_deterministic_mode,
+)
+
+pytestmark = pytest.mark.asyncio
+
+REQ = RoutingRequest(payload={"title": "suspicious login", "severity": "high"})
+
+
+def _tier(conf: float, label: str):
+ def fn(_request):
+ return TierResult(output={"by": label}, confidence=conf, reason=f"{label} says {conf}", model=label)
+
+ return fn
+
+
+def _exploding_tier(_request):
+ raise RuntimeError("provider 500")
+
+
+async def test_confident_deterministic_wins_and_llm_never_called():
+ llm_calls = {"n": 0}
+
+ def llm(_r):
+ llm_calls["n"] += 1
+ return TierResult(output={}, confidence=1.0, model="llm")
+
+ router = ModelRouter(deterministic_fn=_tier(0.95, "det"), ml_fn=_tier(0.99, "ml"), llm_fn=llm)
+ decision = await router.route(REQ)
+
+ assert decision.tier is ModelTier.DETERMINISTIC
+ assert decision.deterministic is True
+ assert llm_calls["n"] == 0 # the whole point — cheap tier answered
+ assert decision.tiers_considered == ["deterministic"]
+ assert any("confident" in a for a in decision.attribution)
+
+
+async def test_low_confidence_escalates_deterministic_to_ml_to_llm():
+ router = ModelRouter(
+ deterministic_fn=_tier(0.2, "det"),
+ ml_fn=_tier(0.4, "ml"),
+ llm_fn=_tier(0.9, "llm"),
+ )
+ decision = await router.route(REQ)
+ assert decision.tier is ModelTier.LLM
+ assert decision.tiers_considered == ["deterministic", "ml", "llm"]
+
+
+async def test_ml_wins_when_confident_and_llm_not_reached():
+ llm_calls = {"n": 0}
+
+ def llm(_r):
+ llm_calls["n"] += 1
+ return TierResult(output={}, confidence=1.0, model="llm")
+
+ router = ModelRouter(deterministic_fn=_tier(0.1, "det"), ml_fn=_tier(0.95, "ml"), llm_fn=llm)
+ decision = await router.route(REQ)
+ assert decision.tier is ModelTier.ML
+ assert llm_calls["n"] == 0
+
+
+async def test_no_silent_llm_when_llm_tier_absent():
+ """Low-confidence deterministic + no LLM tier: must fall back to
+ deterministic AND record why the escalation was blocked — never silent."""
+ router = ModelRouter(deterministic_fn=_tier(0.3, "det"), ml_fn=None, llm_fn=None)
+ decision = await router.route(REQ)
+ assert decision.tier is ModelTier.DETERMINISTIC
+ assert decision.escalation_blocked_reason == "no ML/LLM tier configured"
+
+
+async def test_llm_failure_degrades_to_best_lower_tier_with_reason():
+ router = ModelRouter(
+ deterministic_fn=_tier(0.3, "det"),
+ ml_fn=_tier(0.5, "ml"),
+ llm_fn=_exploding_tier,
+ )
+ decision = await router.route(REQ)
+ # ML was the best lower tier (0.5 > 0.3); LLM blew up → fall back to ML.
+ assert decision.tier is ModelTier.ML
+ assert decision.escalation_blocked_reason is not None
+ assert "LLM tier failed" in decision.escalation_blocked_reason
+
+
+async def test_deterministic_only_never_touches_ml_or_llm():
+ ml_calls = {"n": 0}
+ llm_calls = {"n": 0}
+
+ def ml(_r):
+ ml_calls["n"] += 1
+ return TierResult(output={}, confidence=1.0, model="ml")
+
+ def llm(_r):
+ llm_calls["n"] += 1
+ return TierResult(output={}, confidence=1.0, model="llm")
+
+ # Deterministic tier is LOW confidence, so absent the flag it would escalate.
+ router = ModelRouter(deterministic_fn=_tier(0.1, "det"), ml_fn=ml, llm_fn=llm, deterministic_only=True)
+ decision = await router.route(REQ)
+ assert decision.tier is ModelTier.DETERMINISTIC
+ assert decision.deterministic is True
+ assert ml_calls["n"] == 0 and llm_calls["n"] == 0
+ assert any("deterministic-only" in a for a in decision.attribution)
+
+
+async def test_env_flag_forces_deterministic_mode(monkeypatch):
+ monkeypatch.setenv(DETERMINISTIC_ENV_FLAG, "1")
+ assert is_deterministic_mode() is True
+ router = ModelRouter(deterministic_fn=_tier(0.1, "det"), ml_fn=_tier(0.99, "ml"), llm_fn=_tier(0.99, "llm"))
+ decision = await router.route(REQ)
+ assert decision.tier is ModelTier.DETERMINISTIC
+
+
+async def test_governor_circuit_open_forces_deterministic():
+ class OpenGovernor:
+ def check(self, tenant_id, fingerprint): # noqa: ANN001, ARG002
+ return SimpleNamespace(use_llm=False, decision=SimpleNamespace(value="circuit_open"))
+
+ router = ModelRouter(
+ deterministic_fn=_tier(0.1, "det"),
+ ml_fn=_tier(0.99, "ml"),
+ llm_fn=_tier(0.99, "llm"),
+ governor=OpenGovernor(),
+ )
+ decision = await router.route(REQ)
+ assert decision.tier is ModelTier.DETERMINISTIC
+ assert any("cost governor" in a for a in decision.attribution)
+
+
+async def test_determinism_contract_identical_input_identical_decision():
+ """The headline contract: in deterministic mode, the same input twice
+ yields an identical decision (the deterministic tier is pure)."""
+ router = ModelRouter(deterministic_fn=_tier(0.42, "det"), ml_fn=_tier(0.9, "ml"), deterministic_only=True)
+ d1 = await router.route(REQ)
+ d2 = await router.route(REQ)
+ assert d1.to_dict() == d2.to_dict()
+
+
+async def test_every_decision_is_attributed():
+ for det_conf in (0.05, 0.5, 0.95):
+ router = ModelRouter(deterministic_fn=_tier(det_conf, "det"), ml_fn=_tier(0.5, "ml"), llm_fn=_tier(0.99, "llm"))
+ decision = await router.route(REQ)
+ assert decision.attribution, "every decision must carry an attribution trail"
+ assert decision.model_used
+ assert decision.tiers_considered
+
+
+async def test_fingerprint_is_stable_and_order_independent():
+ a = RoutingRequest(payload={"x": 1, "y": 2})
+ b = RoutingRequest(payload={"y": 2, "x": 1})
+ assert a.fingerprint() == b.fingerprint()
+
+
+async def test_is_deterministic_mode_default_off(monkeypatch):
+ monkeypatch.delenv(DETERMINISTIC_ENV_FLAG, raising=False)
+ assert is_deterministic_mode() is False
+ assert os.environ.get(DETERMINISTIC_ENV_FLAG) is None