Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
feat(phase-7a): unified deterministic->ML->LLM model router + determinism contract (#432)
Beenu Arorae92fccb0c1da
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 182a6d350..0fcf09718 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -326,6 +326,7 @@ jobs:tests/test_prompt_envelope.py \tests/test_privacy_redactor.py \tests/test_cost_governor.py \+ tests/test_model_router.py \-v --tb=short# ──────────────────────────────────────────────diff --git a/CHANGELOG.md b/CHANGELOG.mdindex e66f47241..55eaece44 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Added+- **Phase 7a — unified multi-model router with tier attribution + a determinism contract.** The reality audit confirmed the knowledge graph is already built at ingest (v8 T1.1, `services/ingest/internal/graph/`); the genuine Phase 7 delta was the *model router*. Before it, the deterministic→LLM fallback was reimplemented independently in NL query, playbook drafting, explain, copilot, and each sub-agent — no single audit point for which model answered or why. New `services/agents/app/routing/model_router.py` is that place: a `ModelRouter` that escalates deterministic → ML → LLM only when the cheaper tier is under-confident, and attributes every decision (`tier`, `model_used`, `attribution` trail, `tiers_considered`, `escalation_blocked_reason`). It **never silently uses the LLM** — a skipped or blocked LLM tier (no key, air-gap, deterministic mode, governor circuit open, or a tier error) is always recorded. Introduces the canonical **`AISOC_DETERMINISTIC`** flag and composes with the existing `CostGovernor` circuit breaker: either one forces deterministic-only, in which case the router is reproducible (same input → identical decision). 12 tests (`services/agents/tests/test_model_router.py`, appended to the CI agents gate) prove tier selection, attribution, the no-silent-LLM property, graceful LLM-failure degradation, and the determinism contract. Doc: `apps/docs/docs/concepts/model-router.md`. Remaining Phase 7 enrichments (posture collection, effective-permissions snapshot loader, bi-temporal validity, fusion-time ContextBundle) are tracked as 7b+ in `docs/audit/PROGRESS.md`.- **Phase 6 — performance + cost, both gated.** Two cheap, non-flaky gates in a new `.github/workflows/perf.yml`. (1) **Throughput** — `scripts/perf/throughput_harness.py` runs the *real* fusion hot path (`promote_normalized_event`) over a deterministic 20k-event synthetic corpus and reports events/sec + p50/p95/p99 per-event latency (measured ~250k eps on commodity hardware). The gate asserts a **generous regression floor** (1,000 eps — a >200× margin) so it fires only on a catastrophic regression such as an I/O call slipping onto the hot path, never on shared-runner jitter; it is explicitly a regression floor, not a production SLO. (2) **Cost** — `scripts/storage_cost_model.py` is a deterministic tiered-storage $/TB model (hot ClickHouse block 30d / warm object 60d / cold archive 275d, 8× ZSTD); the committed worked example (`docs/decisions/storage-cost-model.json`: ≈ $902/mo and ≈ $30/raw-TB at 1 TB/day) is drift-gated by `--check`, so the number can never silently diverge from the rate card. Rate-card values are labelled reference list prices to verify per provider/region — the value is the methodology, not the exact dollars. `docs/decisions/0005-storage-consolidation.md` (ADR) records the three-tier decision (keep ClickHouse-only hot; do not add a second hot engine) and cites the gated model.- **Phase 5 — data spine correctness: schema registry + dead-letter queue + lineage.** The reality audit's load-bearing gap: the fusion consumer logged a warning and **silently dropped** any malformed message — silent data loss. Now every message is schema-validated against a versioned envelope registry (`services/fusion/app/services/event_schema.py`: `aisoc.raw_events` and `aisoc.alerts.raw`, each pinned to `v1`, with a drift-guard test so the wire contract can't move unnoticed) *before* the promoter sees it. Anything that fails — non-object payload, unknown `schema_version`, missing `ocsf_event`, non-UUID tenant, or a RawAlert that fails deep Pydantic validation — is routed to a **fail-soft dead-letter queue** (`services/fusion/app/services/dlq.py`: `LoggingDLQ` default, `InMemoryDLQ` for tests, `KafkaDLQ` republishing to `aisoc.alerts.dlq`) with its reason, schema version, source-event lineage, and a truncated payload, instead of vanishing. `safe_record` guarantees a DLQ that itself throws never crashes the consumer. Each processed alert logs `fusion.lineage` (source event id + schema version). 21 new tests (`test_event_schema.py`, `test_dlq.py`, `test_consumer_dlq.py`) prove a valid event is processed with an empty DLQ, every poison shape is captured (not dropped), and a failing DLQ sink doesn't crash the consumer; full fusion suite 92/92 at 59% coverage (floor 48). Idempotency (AlertSink dedup fingerprint, Phase 3.1) and event-time watermarking are already present; backfill/replay-from-offset is tracked as 5b in `docs/audit/PROGRESS.md`.- **Phase 4a — the Detection-as-Code gate is no longer circular.** The reality audit's #1 circular gate: the detection-proposal promote path shelled out to `run_evals.py` **without ever passing the proposed rule body**, so "passed" meant the repo-wide substrate MITRE accuracy didn't move — a value independent of the rule under review. A blind rule (matches nothing) or a noisy rule (matches everything) sailed through its own exam. New `services/api/app/services/detection_eval.py::evaluate_candidate_rule` runs the *candidate `rule_body` itself* through the **real** runtime engine (`rule_engine.execute_rule`) against caller-supplied positive/negative fixtures: it must fire on every positive and stay silent on every negative. New `POST /detection-proposals/{id}/evaluate-rule` stores the verdict under `eval_result["candidate_rule"]`, and `POST /decide` (approve) now **requires** it — a benchmark-only pass is no longer sufficient. `services/api/tests/test_detection_eval.py` (6 tests) is the mutation test that proves the gate rejects a blind rule, rejects a noisy rule, rejects a no-positive-fixture rule, and that the verdict genuinely depends on the rule body. Claim-to-gate matrix: DAC row PARTIAL(circular) → GATED.diff --git a/ROADMAP.md b/ROADMAP.mdindex fa5875c78..46b73f6d9 100644--- a/ROADMAP.md+++ b/ROADMAP.md@@ -29,7 +29,7 @@ A proof-first, security-first program to make every README claim gate-backed, cl- [ ] Phase 4 — Real evals + detection content truth table (third-party-labeled corpus, hallucination/calibration/abstention, model matrix) — **4a/4b landed** (de-circularised DAC candidate-rule gate + honest executable-vs-imported truth table); 4c+ live-agent eval / calibration / model matrix pending- [x] Phase 5 — Data spine correctness (versioned event-schema registry + dead-letter queue + source-event lineage in the fusion consumer; idempotency via AlertSink dedup + event-time watermarking; backfill/replay-from-offset tracked as 5b)- [x] Phase 6 — Performance + cost (fusion hot-path throughput harness with a generous regression-floor gate; deterministic storage $/TB cost model + drift gate; storage-consolidation ADR-0005)-- [ ] Phase 7 — Ingest-time graph + multi-model router (posture, effective permissions, bi-temporal, fusion-time ContextBundle, determinism contract)+- [x] Phase 7 — Ingest-time graph + multi-model router — **graph-at-ingest already shipped** (v8 T1.1, `services/ingest/internal/graph/`); **7a landed**: unified deterministic→ML→LLM router with tier attribution + the `AISOC_DETERMINISTIC` determinism contract (`services/agents/app/routing/model_router.py`, gated). 7b+ (posture collection, effective-permissions snapshot loader, bi-temporal valid_from/valid_to, fusion-time ContextBundle) tracked in `docs/audit/PROGRESS.md`- [ ] Phase 8 — LLMOps (prompt registry, model pinning, provider fallback, response caching, structured-output validation)- [ ] Phase 9 — Autonomy safety (rollback, post-action verification, dry-run default, break-glass, approval SLA, scorecard)- [ ] Phase 10 — Connector + content quality (conformance suite, published matrix, detection lifecycle)diff --git a/apps/docs/docs/concepts/model-router.md b/apps/docs/docs/concepts/model-router.mdnew file mode 100644index 000000000..863645cf9--- /dev/null+++ b/apps/docs/docs/concepts/model-router.md@@ -0,0 +1,67 @@+# Multi-model router++AiSOC's reasoning uses three tiers, cheapest first:++1. **Deterministic** — rules and heuristics (keyword tactic extraction, the+ triage scorer, the confidence model). No network, no cost, reproducible.+2. **ML** — the fusion service's Isolation Forest + LambdaRank scorers.+3. **LLM** — a hosted or local model, used only as a last resort.++The **model router** (`services/agents/app/routing/model_router.py`) is the one+place that decides which tier answers a given request, escalating only when the+cheaper tier isn't confident enough, and **attributing every decision** to the+tier that produced it.++## Why a single router++Before it, the deterministic→LLM fallback was reimplemented independently in NL+query, playbook drafting, "explain this alert", the copilot, and each+sub-agent. Each checked "is a key configured?" and degraded on its own, so there+was no single audit point for *which model answered* or *why the LLM was (or+wasn't) used*. The router unifies that.++## The escalation ladder++```+deterministic ── confident? ──▶ done (tier=deterministic)+ │ no+ ▼+ ML ──────── confident? ──▶ done (tier=ml)+ │ no+ ▼+ LLM ─────────────────────▶ done (tier=llm)+```++Each `RoutingDecision` carries:++- `tier` — which tier produced the answer,+- `model_used` — the concrete model/scorer name,+- `attribution` — a human-readable trail of *why* this tier was chosen and why+ cheaper tiers were insufficient,+- `tiers_considered` — every tier that ran,+- `escalation_blocked_reason` — set whenever the router *wanted* to escalate but+ couldn't (no key, air-gap, deterministic mode, governor circuit open, or the+ LLM tier errored).++## The determinism contract++The router **never silently uses the LLM.** Two independent controls force the+deterministic tier and forbid ML/LLM entirely:++- **`AISOC_DETERMINISTIC=1`** — the canonical determinism switch (air-gapped+ deployments, reproducible evals, cost lockdown).+- **The cost governor** — when a tenant's hard budget is exhausted the governor+ returns a circuit-open decision; the router honours it and drops to+ deterministic-only.++In deterministic mode the router is **reproducible**: the same input yields an+identical decision, because only the pure deterministic tier runs. This is+gated in CI (`services/agents/tests/test_model_router.py`,+`test_determinism_contract_identical_input_identical_decision`).++## What the gate proves++`test_model_router.py` proves the router picks the cheapest sufficient tier,+attributes every decision, never silently reaches the LLM (a skipped or blocked+LLM tier is always recorded), degrades gracefully when the LLM errors, and is+reproducible in deterministic mode.diff --git a/apps/docs/sidebars.ts b/apps/docs/sidebars.tsindex fbec8ba15..4b3e54a1b 100644--- a/apps/docs/sidebars.ts+++ b/apps/docs/sidebars.ts@@ -25,6 +25,7 @@ const sidebars: SidebarsConfig = {"concepts/detections","concepts/live-actions","concepts/automation-maturity",+ "concepts/model-router",],},{diff --git a/services/agents/app/routing/__init__.py b/services/agents/app/routing/__init__.pynew file mode 100644index 000000000..a4b794134--- /dev/null+++ b/services/agents/app/routing/__init__.py@@ -0,0 +1,25 @@+"""Multi-model routing (Phase 7): deterministic → ML → LLM with attribution."""++from app.routing.model_router import (+ DEFAULT_CONFIDENCE_FLOOR,+ DETERMINISTIC_ENV_FLAG,+ ModelRouter,+ ModelTier,+ RoutingDecision,+ RoutingRequest,+ TierResult,+ build_router,+ is_deterministic_mode,+)++__all__ = [+ "DEFAULT_CONFIDENCE_FLOOR",+ "DETERMINISTIC_ENV_FLAG",+ "ModelRouter",+ "ModelTier",+ "RoutingDecision",+ "RoutingRequest",+ "TierResult",+ "build_router",+ "is_deterministic_mode",+]diff --git a/services/agents/app/routing/model_router.py b/services/agents/app/routing/model_router.pynew file mode 100644index 000000000..6838e70a7--- /dev/null+++ b/services/agents/app/routing/model_router.py@@ -0,0 +1,311 @@+"""Unified deterministic → ML → LLM model router (Phase 7).++The reality audit found the three-model story (deterministic rules → ML scorers+→ LLM) implemented as *scattered, ad-hoc fallbacks*: `nl_query`, `nl_drafter`,+`explain`, `copilot`, and the sub-agents each independently check "is a key+configured?" and degrade on their own. There was no single place that:++* decides which tier should answer (cheapest sufficient tier first),+* **attributes** every decision to the tier that produced it (so cost and+ provenance are auditable), and+* enforces a **determinism contract**: when deterministic mode is on (env flag+ or the cost governor tripping its circuit breaker), the router NEVER touches+ the ML or LLM tier, and the deterministic tier is reproducible — identical+ input yields an identical decision.++This module is that place. It is intentionally dependency-light (stdlib only):+the three tier functions are injected, so the router is unit-testable without a+real model, and `build_router` wires the real deterministic scorer while+leaving ML/LLM as optional callables the caller supplies.++Key invariant the gate proves: **the router never silently uses the LLM.** Every+decision carries an `attribution` trail; whenever the LLM tier is skipped or+blocked (deterministic mode, no key, governor circuit open, tier error), the+reason is recorded rather than the router quietly degrading.+"""++from __future__ import annotations++import hashlib+import inspect+import json+import logging+import os+from collections.abc import Awaitable, Callable+from dataclasses import dataclass, field+from enum import Enum+from typing import Any, Protocol, runtime_checkable++logger = logging.getLogger(__name__)++DETERMINISTIC_ENV_FLAG = "AISOC_DETERMINISTIC"+DEFAULT_CONFIDENCE_FLOOR = 0.85+++class ModelTier(str, Enum):+ DETERMINISTIC = "deterministic"+ ML = "ml"+ LLM = "llm"+++def is_deterministic_mode() -> bool:+ """Canonical determinism switch. Off by default; set AISOC_DETERMINISTIC=1+ to force the deterministic tier everywhere (air-gap, reproducible evals,+ cost lockdown)."""+ raw = os.environ.get(DETERMINISTIC_ENV_FLAG)+ if raw is None:+ return False+ return raw.strip().lower() in {"1", "true", "yes", "on", "enabled"}+++@dataclass(frozen=True)+class RoutingRequest:+ """Input to the router. ``fingerprint`` is the determinism/dedup identity."""++ payload: dict[str, Any]+ tenant_id: str = "00000000-0000-0000-0000-000000000000"++ def fingerprint(self) -> str:+ blob = json.dumps(self.payload, sort_keys=True, default=str)+ return hashlib.sha256(blob.encode("utf-8")).hexdigest()+++@dataclass(frozen=True)+class TierResult:+ """What a tier returns: an output, a confidence in [0,1], and a reason."""++ output: Any+ confidence: float+ reason: str = ""+ model: str | None = None+++@dataclass(frozen=True)+class RoutingDecision:+ """The router's verdict, fully attributed."""++ tier: ModelTier+ output: Any+ confidence: float+ model_used: str+ attribution: list[str] = field(default_factory=list)+ tiers_considered: list[str] = field(default_factory=list)+ deterministic: bool = False+ escalation_blocked_reason: str | None = None++ def to_dict(self) -> dict[str, Any]:+ return {+ "tier": self.tier.value,+ "confidence": self.confidence,+ "model_used": self.model_used,+ "attribution": list(self.attribution),+ "tiers_considered": list(self.tiers_considered),+ "deterministic": self.deterministic,+ "escalation_blocked_reason": self.escalation_blocked_reason,+ }+++@runtime_checkable+class Governor(Protocol):+ """Subset of cost_governor used here — kept as a Protocol so this module+ doesn't import the heavy governor at module load."""++ def check(self, tenant_id: str, fingerprint: str) -> Any: ...+++TierFn = Callable[[RoutingRequest], TierResult | Awaitable[TierResult]]+++async def _maybe_await(value: TierResult | Awaitable[TierResult]) -> TierResult:+ if inspect.isawaitable(value):+ return await value+ return value+++class ModelRouter:+ """Routes a request to the cheapest sufficient tier, fully attributed.++ Escalation ladder: deterministic → ML → LLM. A tier "wins" when its+ confidence meets ``confidence_floor``; otherwise the router escalates to the+ next available tier. The LLM tier is only ever reached when it is+ configured, deterministic mode is off, and the governor (if any) permits+ spend — and if it is *not* reached, the reason is recorded.+ """++ def __init__(+ self,+ *,+ deterministic_fn: TierFn,+ ml_fn: TierFn | None = None,+ llm_fn: TierFn | None = None,+ confidence_floor: float = DEFAULT_CONFIDENCE_FLOOR,+ deterministic_only: bool = False,+ governor: Governor | None = None,+ ) -> None:+ self._deterministic_fn = deterministic_fn+ self._ml_fn = ml_fn+ self._llm_fn = llm_fn+ self._floor = confidence_floor+ self._forced_deterministic = deterministic_only+ self._governor = governor++ def _resolve_deterministic_only(self, request: RoutingRequest, attribution: list[str]) -> bool:+ if self._forced_deterministic:+ attribution.append("deterministic-only: forced by construction")+ return True+ if is_deterministic_mode():+ attribution.append(f"deterministic-only: {DETERMINISTIC_ENV_FLAG} is set")+ return True+ if self._governor is not None:+ try:+ decision = self._governor.check(request.tenant_id, request.fingerprint())+ use_llm = bool(getattr(decision, "use_llm", True))+ label = getattr(getattr(decision, "decision", None), "value", None) or str(getattr(decision, "decision", "?"))+ if not use_llm:+ attribution.append(f"deterministic-only: cost governor decision '{label}' forbids LLM spend")+ return True+ except Exception as exc: # noqa: BLE001 — governor must never crash routing+ logger.warning("model_router.governor_check_failed", extra={"error": str(exc)})+ return False++ async def route(self, request: RoutingRequest) -> RoutingDecision:+ attribution: list[str] = []+ considered: list[str] = []+ deterministic_only = self._resolve_deterministic_only(request, attribution)++ # Tier 1 — deterministic. Always runs; it's the floor of the ladder.+ det = await _maybe_await(self._deterministic_fn(request))+ considered.append(ModelTier.DETERMINISTIC.value)++ higher_tier_available = (self._ml_fn is not None or self._llm_fn is not None) and not deterministic_only+ if det.confidence >= self._floor or not higher_tier_available:+ if deterministic_only:+ attribution.append("resolved on deterministic tier (deterministic-only mode)")+ elif det.confidence >= self._floor:+ attribution.append(f"deterministic tier confident ({det.confidence:.2f} >= floor {self._floor:.2f})")+ else:+ attribution.append("no higher tier available; using deterministic result")+ blocked = None+ if not higher_tier_available and det.confidence < self._floor:+ blocked = "deterministic-only mode" if deterministic_only else "no ML/LLM tier configured"+ return RoutingDecision(+ tier=ModelTier.DETERMINISTIC,+ output=det.output,+ confidence=det.confidence,+ model_used=det.model or "deterministic",+ attribution=attribution + ([det.reason] if det.reason else []),+ tiers_considered=considered,+ deterministic=True,+ escalation_blocked_reason=blocked,+ )++ # Tier 2 — ML. Cheaper than the LLM; try before escalating further.+ best = det+ best_tier = ModelTier.DETERMINISTIC+ if self._ml_fn is not None:+ attribution.append(f"deterministic below floor ({det.confidence:.2f} < {self._floor:.2f}); escalating to ML")+ ml = await _maybe_await(self._ml_fn(request))+ considered.append(ModelTier.ML.value)+ if ml.confidence >= self._floor or self._llm_fn is None:+ if ml.confidence >= self._floor:+ attribution.append(f"ML tier confident ({ml.confidence:.2f} >= floor)")+ blocked = None+ else:+ attribution.append("no LLM tier configured; using ML result")+ blocked = "no LLM tier configured"+ return RoutingDecision(+ tier=ModelTier.ML,+ output=ml.output,+ confidence=ml.confidence,+ model_used=ml.model or "ml",+ attribution=attribution + ([ml.reason] if ml.reason else []),+ tiers_considered=considered,+ deterministic=False,+ escalation_blocked_reason=blocked,+ )+ if ml.confidence > best.confidence:+ best, best_tier = ml, ModelTier.ML++ # Tier 3 — LLM. Last resort; only reached when configured + permitted.+ if self._llm_fn is not None:+ attribution.append("escalating to LLM tier")+ considered.append(ModelTier.LLM.value)+ try:+ llm = await _maybe_await(self._llm_fn(request))+ return RoutingDecision(+ tier=ModelTier.LLM,+ output=llm.output,+ confidence=llm.confidence,+ model_used=llm.model or "llm",+ attribution=attribution + ([llm.reason] if llm.reason else []),+ tiers_considered=considered,+ deterministic=False,+ )+ except Exception as exc: # noqa: BLE001 — LLM failure must degrade, not crash+ reason = f"LLM tier failed ({type(exc).__name__}); falling back to {best_tier.value}"+ attribution.append(reason)+ return RoutingDecision(+ tier=best_tier,+ output=best.output,+ confidence=best.confidence,+ model_used=best.model or best_tier.value,+ attribution=attribution,+ tiers_considered=considered,+ deterministic=best_tier is ModelTier.DETERMINISTIC,+ escalation_blocked_reason=reason,+ )++ # No LLM tier at all — never silently pretend; record the block.+ blocked = "no LLM tier configured"+ attribution.append(f"escalation to LLM blocked: {blocked}; using {best_tier.value} result")+ return RoutingDecision(+ tier=best_tier,+ output=best.output,+ confidence=best.confidence,+ model_used=best.model or best_tier.value,+ attribution=attribution,+ tiers_considered=considered,+ deterministic=best_tier is ModelTier.DETERMINISTIC,+ escalation_blocked_reason=blocked,+ )+++def build_router(+ *,+ ml_fn: TierFn | None = None,+ llm_fn: TierFn | None = None,+ confidence_floor: float = DEFAULT_CONFIDENCE_FLOOR,+ governor: Governor | None = None,+) -> ModelRouter:+ """Wire a router with the real deterministic triage scorer as tier 1.++ ML/LLM tiers are optional and injected by the caller (they carry heavy+ deps), so this factory stays import-light and the deterministic path always+ works — including air-gapped.+ """++ def _deterministic(request: RoutingRequest) -> TierResult:+ # Lazy import so this module never drags the agents package at import.+ from app.agents.triage_agent import _score_alert # noqa: PLC0415+ from app.models.state import InvestigationState # noqa: PLC0415++ payload = request.payload+ state = InvestigationState(+ alert_summary=str(payload.get("alert_summary") or payload.get("title") or ""),+ raw_alert=payload.get("raw_alert") or payload,+ )+ verdict, confidence = _score_alert(state)+ return TierResult(+ output={"verdict": verdict},+ confidence=confidence,+ reason=f"heuristic triage → {verdict}",+ model="deterministic:triage",+ )++ return ModelRouter(+ deterministic_fn=_deterministic,+ ml_fn=ml_fn,+ llm_fn=llm_fn,+ confidence_floor=confidence_floor,+ governor=governor,+ )diff --git a/services/agents/tests/test_model_router.py b/services/agents/tests/test_model_router.pynew file mode 100644index 000000000..f8916f3de--- /dev/null+++ b/services/agents/tests/test_model_router.py@@ -0,0 +1,179 @@+"""Phase 7 — the multi-model router contract.++Proves the router (a) picks the cheapest sufficient tier, (b) attributes every+decision to a tier, (c) NEVER silently uses the LLM (a skipped/blocked LLM tier+is always recorded), and (d) honours the determinism contract: in deterministic+mode it never touches ML/LLM and is reproducible.++The tiers are injected as plain functions so the contract is proven without any+real model — that is the point of the router: one auditable place for the+deterministic→ML→LLM ladder.+"""++from __future__ import annotations++import os+from types import SimpleNamespace++import pytest+from app.routing.model_router import (+ DETERMINISTIC_ENV_FLAG,+ ModelRouter,+ ModelTier,+ RoutingRequest,+ TierResult,+ is_deterministic_mode,+)++pytestmark = pytest.mark.asyncio++REQ = RoutingRequest(payload={"title": "suspicious login", "severity": "high"})+++def _tier(conf: float, label: str):+ def fn(_request):+ return TierResult(output={"by": label}, confidence=conf, reason=f"{label} says {conf}", model=label)++ return fn+++def _exploding_tier(_request):+ raise RuntimeError("provider 500")+++async def test_confident_deterministic_wins_and_llm_never_called():+ llm_calls = {"n": 0}++ def llm(_r):+ llm_calls["n"] += 1+ return TierResult(output={}, confidence=1.0, model="llm")++ router = ModelRouter(deterministic_fn=_tier(0.95, "det"), ml_fn=_tier(0.99, "ml"), llm_fn=llm)+ decision = await router.route(REQ)++ assert decision.tier is ModelTier.DETERMINISTIC+ assert decision.deterministic is True+ assert llm_calls["n"] == 0 # the whole point — cheap tier answered+ assert decision.tiers_considered == ["deterministic"]+ assert any("confident" in a for a in decision.attribution)+++async def test_low_confidence_escalates_deterministic_to_ml_to_llm():+ router = ModelRouter(+ deterministic_fn=_tier(0.2, "det"),+ ml_fn=_tier(0.4, "ml"),+ llm_fn=_tier(0.9, "llm"),+ )+ decision = await router.route(REQ)+ assert decision.tier is ModelTier.LLM+ assert decision.tiers_considered == ["deterministic", "ml", "llm"]+++async def test_ml_wins_when_confident_and_llm_not_reached():+ llm_calls = {"n": 0}++ def llm(_r):+ llm_calls["n"] += 1+ return TierResult(output={}, confidence=1.0, model="llm")++ router = ModelRouter(deterministic_fn=_tier(0.1, "det"), ml_fn=_tier(0.95, "ml"), llm_fn=llm)+ decision = await router.route(REQ)+ assert decision.tier is ModelTier.ML+ assert llm_calls["n"] == 0+++async def test_no_silent_llm_when_llm_tier_absent():+ """Low-confidence deterministic + no LLM tier: must fall back to+ deterministic AND record why the escalation was blocked — never silent."""+ router = ModelRouter(deterministic_fn=_tier(0.3, "det"), ml_fn=None, llm_fn=None)+ decision = await router.route(REQ)+ assert decision.tier is ModelTier.DETERMINISTIC+ assert decision.escalation_blocked_reason == "no ML/LLM tier configured"+++async def test_llm_failure_degrades_to_best_lower_tier_with_reason():+ router = ModelRouter(+ deterministic_fn=_tier(0.3, "det"),+ ml_fn=_tier(0.5, "ml"),+ llm_fn=_exploding_tier,+ )+ decision = await router.route(REQ)+ # ML was the best lower tier (0.5 > 0.3); LLM blew up → fall back to ML.+ assert decision.tier is ModelTier.ML+ assert decision.escalation_blocked_reason is not None+ assert "LLM tier failed" in decision.escalation_blocked_reason+++async def test_deterministic_only_never_touches_ml_or_llm():+ ml_calls = {"n": 0}+ llm_calls = {"n": 0}++ def ml(_r):+ ml_calls["n"] += 1+ return TierResult(output={}, confidence=1.0, model="ml")++ def llm(_r):+ llm_calls["n"] += 1+ return TierResult(output={}, confidence=1.0, model="llm")++ # Deterministic tier is LOW confidence, so absent the flag it would escalate.+ router = ModelRouter(deterministic_fn=_tier(0.1, "det"), ml_fn=ml, llm_fn=llm, deterministic_only=True)+ decision = await router.route(REQ)+ assert decision.tier is ModelTier.DETERMINISTIC+ assert decision.deterministic is True+ assert ml_calls["n"] == 0 and llm_calls["n"] == 0+ assert any("deterministic-only" in a for a in decision.attribution)+++async def test_env_flag_forces_deterministic_mode(monkeypatch):+ monkeypatch.setenv(DETERMINISTIC_ENV_FLAG, "1")+ assert is_deterministic_mode() is True+ router = ModelRouter(deterministic_fn=_tier(0.1, "det"), ml_fn=_tier(0.99, "ml"), llm_fn=_tier(0.99, "llm"))+ decision = await router.route(REQ)+ assert decision.tier is ModelTier.DETERMINISTIC+++async def test_governor_circuit_open_forces_deterministic():+ class OpenGovernor:+ def check(self, tenant_id, fingerprint): # noqa: ANN001, ARG002+ return SimpleNamespace(use_llm=False, decision=SimpleNamespace(value="circuit_open"))++ router = ModelRouter(+ deterministic_fn=_tier(0.1, "det"),+ ml_fn=_tier(0.99, "ml"),+ llm_fn=_tier(0.99, "llm"),+ governor=OpenGovernor(),+ )+ decision = await router.route(REQ)+ assert decision.tier is ModelTier.DETERMINISTIC+ assert any("cost governor" in a for a in decision.attribution)+++async def test_determinism_contract_identical_input_identical_decision():+ """The headline contract: in deterministic mode, the same input twice+ yields an identical decision (the deterministic tier is pure)."""+ router = ModelRouter(deterministic_fn=_tier(0.42, "det"), ml_fn=_tier(0.9, "ml"), deterministic_only=True)+ d1 = await router.route(REQ)+ d2 = await router.route(REQ)+ assert d1.to_dict() == d2.to_dict()+++async def test_every_decision_is_attributed():+ for det_conf in (0.05, 0.5, 0.95):+ router = ModelRouter(deterministic_fn=_tier(det_conf, "det"), ml_fn=_tier(0.5, "ml"), llm_fn=_tier(0.99, "llm"))+ decision = await router.route(REQ)+ assert decision.attribution, "every decision must carry an attribution trail"+ assert decision.model_used+ assert decision.tiers_considered+++async def test_fingerprint_is_stable_and_order_independent():+ a = RoutingRequest(payload={"x": 1, "y": 2})+ b = RoutingRequest(payload={"y": 2, "x": 1})+ assert a.fingerprint() == b.fingerprint()+++async def test_is_deterministic_mode_default_off(monkeypatch):+ monkeypatch.delenv(DETERMINISTIC_ENV_FLAG, raising=False)+ assert is_deterministic_mode() is False+ assert os.environ.get(DETERMINISTIC_ENV_FLAG) is None