Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
Model matrix — the last open item in Phase 4 (#715)
Beenu Aroraffd2ee90264e
diff --git a/.github/workflows/wet-eval.yml b/.github/workflows/wet-eval.ymlindex 9408c1868..05172389f 100644--- a/.github/workflows/wet-eval.yml+++ b/.github/workflows/wet-eval.yml@@ -158,6 +158,31 @@ jobs:--out /tmp/wet-eval/report.json \--wet-out /tmp/wet-eval/wet-block.json+ # ──────────────────────────────────────────────+ # Step 3b — model matrix (Phase 4c).+ # The single-model wet eval above says how the agent performs on one+ # backend; it cannot say whether the result is a property of the+ # agent or of the model. The matrix grades the same corpus across+ # several, one variable at a time.+ #+ # Wrapper, not a second evaluator: it sets the model pins and+ # re-invokes run_evals.py, so there is one definition of accuracy+ # rather than two that can drift.+ # ──────────────────────────────────────────────+ - name: Run model matrix+ if: steps.preflight.outputs.should_run == 'True'+ env:+ WET_EVAL_OPENAI_KEY: ${{ secrets.WET_EVAL_OPENAI_KEY }}+ OPENAI_API_KEY: ${{ secrets.WET_EVAL_OPENAI_KEY }}+ AISOC_MATRIX_MODELS: ${{ vars.AISOC_MATRIX_MODELS || 'gpt-4o-mini,gpt-4o' }}+ run: |+ set -euo pipefail+ python3 scripts/run_model_matrix.py \+ --out /tmp/wet-eval/model-matrix.json \+ --markdown-out /tmp/wet-eval/model-matrix.md+ echo "### Model matrix" >> "${GITHUB_STEP_SUMMARY}"+ cat /tmp/wet-eval/model-matrix.md >> "${GITHUB_STEP_SUMMARY}"+# ──────────────────────────────────────────────# Step 4 — rewrite the wet-eval tables in benchmark.md.# The `--check` flag fails the step if the markdown scaffolddiff --git a/ROADMAP.md b/ROADMAP.mdindex 661b1ebe2..fde259ea2 100644--- a/ROADMAP.md+++ b/ROADMAP.md@@ -26,7 +26,7 @@ A proof-first, security-first program to make every README claim gate-backed, cl- [x] Phase 1 — Four existential holes (prompt injection, memory poisoning, cross-store tenant isolation, data-exfiltration/redaction, cost DoS, vault)- [x] Phase 2 — Supply chain + truth (security.yml scanners, claim-gate ratchet, hard-fail insecure prod defaults, TRADEMARK, verifying-releases, license fix; continuation landed: per-image cosign signatures + CycloneDX SBOM attestations + SLSA provenance, all actions SHA-pinned)- [x] Phase 3 — Integration / E2E / chaos / DR (real-container spine test, backup-restore, chaos, upgrade, cross-store isolation live-replay; heavy-demo-stack Playwright E2E + demo-timing gate tracked as non-blocking 3.5+)-- [ ] Phase 4 — Real evals + detection content truth table (third-party-labeled corpus, hallucination/calibration/abstention, model matrix) — **4a/4b landed** (de-circularised DAC candidate-rule gate + honest executable-vs-imported truth table); 4c+ live-agent eval / calibration / model matrix pending+- [ ] Phase 4 — Real evals + detection content truth table (third-party-labeled corpus, hallucination/calibration/abstention, model matrix) — **4a/4b/4c landed**: de-circularised DAC candidate-rule gate; honest executable-vs-imported truth table; hallucination, abstention, calibration and containment metrics in `packages/aisoc-benchmark` with a documented adapter so a third-party agent can be graded on the same corpus; confidence calibration gated in `test_confidence_calibration.py`; and a **model matrix** (`scripts/run_model_matrix.py`) wired into the weekly wet eval, which grades the same corpus across several models by re-invoking the existing evaluator rather than defining a second notion of accuracy. **Remains open on one thing only, and it is not code:** the live-agent numbers need a funded provider key. Without one the matrix reports *not measured* per model rather than emitting zeros, because a zero is a measurement and "we did not run this" is not.- [x] Phase 5 — Data spine correctness (versioned event-schema registry + dead-letter queue + source-event lineage in the fusion consumer; idempotency via AlertSink dedup + event-time watermarking; backfill/replay-from-offset tracked as 5b)- [x] Phase 6 — Performance + cost (fusion hot-path throughput harness with a generous regression-floor gate; deterministic storage $/TB cost model + drift gate; storage-consolidation ADR-0005)- [x] Phase 7 — Ingest-time graph + multi-model router — **graph-at-ingest already shipped** (v8 T1.1, `services/ingest/internal/graph/`); **7a landed**: unified deterministic→ML→LLM router with tier attribution + the `AISOC_DETERMINISTIC` determinism contract (`services/agents/app/routing/model_router.py`, gated). 7b+ (posture collection, effective-permissions snapshot loader, bi-temporal valid_from/valid_to, fusion-time ContextBundle) tracked in `docs/audit/PROGRESS.md`@@ -230,8 +230,15 @@ Implemented and reviewed by Beenu Arora <[email protected]>.- [x] Case auto-summary + PDF export (`case_summary.py` + `case_summary_html.py`)- [x] Investigation timeline (replayable) (`InvestigationTimeline.tsx`)- [x] Playbook gallery with 12 curated packs + GitHub PR integration for detection proposals-- [ ] Mobile responder console (React Native) — triage and acknowledge from phone _(deferred to v8.0)_-- [ ] Plugin publishing marketplace v3 (commercial plugins, revenue sharing) _(deferred to v8.0)_+- [ ] Mobile responder console (React Native) — triage and acknowledge from phone+ _(deferred past v8.0; **not started** — no React Native code exists in the+ tree. v8.0 shipped without it and nothing in `docs/roadmap/v8-progress.md`+ claimed otherwise, so the previous "deferred to v8.0" note had quietly+ become false.)_+- [ ] Plugin publishing marketplace v3 (commercial plugins, revenue sharing)+ _(deferred past v8.0; **not started**. Revenue sharing is a commercial+ decision rather than an engineering one, and the free packaging path is+ itself still blocked on registry credentials — see v8.1.)_---diff --git a/scripts/run_model_matrix.py b/scripts/run_model_matrix.pynew file mode 100644index 000000000..4de68061a--- /dev/null+++ b/scripts/run_model_matrix.py@@ -0,0 +1,244 @@+#!/usr/bin/env python3+"""Grade the same corpus across several models and publish the comparison.++Phase 4's last open item. The wet eval runs one model — whichever the pins+resolve to — so the published numbers describe the agent *on that model*+and say nothing about whether the result is a property of the agent or of+the backend. Those are different claims, and only one of them is about+this project.++A matrix answers it: same corpus, same prompts, same grader, one variable.+If accuracy collapses on a cheaper model, that is a fact a reader deciding+what to self-host needs, and it is a fact no single-model run can produce.++Deliberately a wrapper, not a second evaluator. It sets the model pin and+invokes `scripts/run_evals.py --wet` per model, then collates the blocks+that script already emits. Reimplementing the eval would create a second+definition of "accuracy" that could drift from the one on the scoreboard —+the exact defect the alert-reduction suite had, where an in-test+reimplementation published a number the product did not produce.++Without a funded key it reports **not measured** and exits zero. It does+not emit zeros: a zero is a measurement, and "we did not run this" is not.++Run: python3 scripts/run_model_matrix.py --models gpt-4o-mini,gpt-4o+"""++from __future__ import annotations++import argparse+import json+import os+import subprocess+import sys+import tempfile+from dataclasses import dataclass, field+from pathlib import Path+from typing import Any++REPO_ROOT = Path(__file__).resolve().parent.parent+RUN_EVALS = REPO_ROOT / "scripts" / "run_evals.py"++#: Roles the pin override has to cover. Setting only one leaves the other+#: agents on their default model, which would make the matrix a comparison+#: of one role rather than of the system.+PIN_ROLES = ("triage", "investigation", "report", "recon")+++@dataclass+class ModelResult:+ model: str+ measured: bool+ #: Populated only when measured. Absent rather than zeroed, because a+ #: zero accuracy and an unrun model must not render the same.+ metrics: dict[str, Any] = field(default_factory=dict)+ error: str | None = None++ def to_dict(self) -> dict[str, Any]:+ out: dict[str, Any] = {"model": self.model, "measured": self.measured}+ if self.measured:+ out["metrics"] = self.metrics+ if self.error:+ out["error"] = self.error+ return out+++def has_live_key() -> bool:+ return bool((os.getenv("WET_EVAL_OPENAI_KEY") or os.getenv("OPENAI_API_KEY") or "").strip())+++def run_one(model: str, *, limit: int | None, timeout: int) -> ModelResult:+ """Run the existing wet eval with every model pin set to `model`."""+ env = dict(os.environ)+ for role in PIN_ROLES:+ env[f"AISOC_MODEL_PIN_{role.upper()}"] = model+ # The agent's resolver reads OPENAI_API_KEY; the workflow supplies the+ # key under the wet-eval name.+ if env.get("WET_EVAL_OPENAI_KEY") and not env.get("OPENAI_API_KEY"):+ env["OPENAI_API_KEY"] = env["WET_EVAL_OPENAI_KEY"]++ with tempfile.TemporaryDirectory() as tmp:+ wet_out = Path(tmp) / "wet.json"+ cmd = [+ sys.executable,+ str(RUN_EVALS),+ "--wet",+ "--out",+ str(Path(tmp) / "report.json"),+ "--wet-out",+ str(wet_out),+ ]+ if limit:+ cmd += ["--limit", str(limit)]++ try:+ completed = subprocess.run(+ cmd,+ env=env,+ capture_output=True,+ text=True,+ timeout=timeout,+ cwd=REPO_ROOT,+ )+ except subprocess.TimeoutExpired:+ return ModelResult(+ model,+ measured=False,+ error=f"timed out after {timeout}s",+ )++ if completed.returncode != 0:+ # The model's own failure is a result about that model, not a+ # reason to abandon the matrix — a model that cannot complete+ # the corpus is exactly what a reader wants to know.+ tail = (completed.stderr or completed.stdout or "").strip().splitlines()+ return ModelResult(+ model,+ measured=False,+ error=tail[-1][:300] if tail else f"exit {completed.returncode}",+ )++ if not wet_out.exists():+ return ModelResult(model, measured=False, error="no wet block produced")++ block = json.loads(wet_out.read_text(encoding="utf-8"))++ return ModelResult(model, measured=True, metrics=_project(block))+++def _project(block: dict[str, Any]) -> dict[str, Any]:+ """The comparable subset. A full wet block is large and most of it is+ identical across models; the point of the matrix is the part that is+ not."""+ latency = block.get("latency_seconds") or {}+ tokens = (block.get("tokens") or {}).get("total") or {}+ usd = block.get("usd") or {}+ return {+ "incidents": block.get("incidents"),+ "mitre_accuracy": block.get("mitre_accuracy"),+ "abstention_rate": block.get("abstention_rate"),+ "mean_groundedness": block.get("mean_groundedness"),+ "latency_p95_s": latency.get("p95"),+ "tokens_mean": tokens.get("mean"),+ "usd_total": usd.get("total"),+ "usd_per_incident": usd.get("mean"),+ }+++def render_markdown(results: list[ModelResult]) -> str:+ lines = [+ "| Model | Incidents | MITRE accuracy | Abstention | Groundedness | p95 latency | Tokens/inv | $/incident |",+ "|-------|-----------|----------------|------------|--------------|-------------|------------|------------|",+ ]+ for result in results:+ if not result.measured:+ # One cell, spanning the intent: this row was not measured.+ # Filling the columns with dashes reads as zero at a glance.+ reason = result.error or "no live key configured"+ lines.append(f"| `{result.model}` | _not measured — {reason}_ |||||||")+ continue+ m = result.metrics++ # `m` is bound as a default rather than captured: the closure is+ # used inside this iteration, so capturing works today and breaks+ # the moment anyone defers the call.+ def fmt(key: str, spec: str = "{:.3f}", metrics: dict = m) -> str:+ value = metrics.get(key)+ return spec.format(value) if isinstance(value, int | float) else "—"++ lines.append(+ f"| `{result.model}` | {m.get('incidents') or '—'} "+ f"| {fmt('mitre_accuracy', '{:.1%}')} "+ f"| {fmt('abstention_rate', '{:.1%}')} "+ f"| {fmt('mean_groundedness')} "+ f"| {fmt('latency_p95_s', '{:.2f}s')} "+ f"| {fmt('tokens_mean', '{:.0f}')} "+ f"| {fmt('usd_per_incident', '${:.4f}')} |"+ )+ return "\n".join(lines)+++def main(argv: list[str] | None = None) -> int:+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument(+ "--models",+ default=os.getenv("AISOC_MATRIX_MODELS", "gpt-4o-mini,gpt-4o"),+ help="Comma-separated model names to grade.",+ )+ parser.add_argument("--out", type=Path, help="Write the JSON result here.")+ parser.add_argument("--markdown-out", type=Path, help="Write the table here.")+ parser.add_argument(+ "--limit",+ type=int,+ help="Grade only the first N incidents. For a smoke run; a limited " "matrix must not be published as a full one.",+ )+ parser.add_argument("--timeout", type=int, default=3600)+ args = parser.parse_args(argv)++ models = [m.strip() for m in args.models.split(",") if m.strip()]+ if not models:+ print("model-matrix: no models requested", file=sys.stderr)+ return 2++ live = has_live_key()+ if not live:+ print("model-matrix: no live LLM key configured — reporting 'not measured' " "for every model rather than emitting zeros.")++ results = [+ run_one(model, limit=args.limit, timeout=args.timeout)+ if live+ else ModelResult(model, measured=False, error="no live key configured")+ for model in models+ ]++ payload = {+ "corpus": "synthetic_incidents.json",+ "limit": args.limit,+ "partial": bool(args.limit),+ "live": live,+ "results": [r.to_dict() for r in results],+ }++ if args.out:+ args.out.parent.mkdir(parents=True, exist_ok=True)+ args.out.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")+ table = render_markdown(results)+ if args.markdown_out:+ args.markdown_out.parent.mkdir(parents=True, exist_ok=True)+ args.markdown_out.write_text(table + "\n", encoding="utf-8")++ print()+ print(table)+ print()+ measured = sum(1 for r in results if r.measured)+ print(f"model-matrix: {measured}/{len(results)} model(s) measured")+ if args.limit:+ print(f" PARTIAL: limited to {args.limit} incidents — not comparable " f"with a full run and must not be published as one.")+ # Exit zero even with nothing measured: an absent key is a+ # configuration state, not a build failure, and failing here would+ # make every fork's CI red.+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/tests/test_model_matrix.py b/tests/test_model_matrix.pynew file mode 100644index 000000000..80d28b48c--- /dev/null+++ b/tests/test_model_matrix.py@@ -0,0 +1,151 @@+"""The model matrix must never invent a number it did not measure.++Phase 4's last open item was a model matrix: the wet eval runs one model,+so its numbers describe the agent *on that model* and say nothing about+whether the result is a property of the agent or of the backend. Those are+different claims and only one is about this project.++Which makes the honesty requirement sharper than usual. A matrix exists to+be compared across rows, so a row that reads `0.000` because nothing ran+is worse than a missing row — it invites the conclusion that the model was+graded and scored zero.+"""++from __future__ import annotations++import importlib.util+import json+import subprocess+import sys+from pathlib import Path++import pytest++REPO_ROOT = Path(__file__).resolve().parent.parent+MODULE = REPO_ROOT / "scripts" / "run_model_matrix.py"++spec = importlib.util.spec_from_file_location("run_model_matrix", MODULE)+assert spec and spec.loader+matrix = importlib.util.module_from_spec(spec)+sys.modules["run_model_matrix"] = matrix+spec.loader.exec_module(matrix)+++WET_BLOCK = {+ "model": "stub",+ "incidents": 200,+ "mitre_accuracy": 0.97,+ "abstention_rate": 0.04,+ "mean_groundedness": 0.93,+ "latency_seconds": {"p95": 4.2},+ "tokens": {"total": {"mean": 3100}},+ "usd": {"total": 1.84, "mean": 0.0092},+}+++def stub_eval(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:+ """A stand-in for run_evals.py emitting the same wet-block shape."""+ script = tmp_path / "fake_run_evals.py"+ script.write_text(+ "import json, os, sys\n"+ "from pathlib import Path\n"+ "args = sys.argv[1:]\n"+ "out = Path(args[args.index('--wet-out') + 1])\n"+ "model = os.environ.get('AISOC_MODEL_PIN_TRIAGE', 'unknown')\n"+ "if model == 'broken':\n"+ " sys.stderr.write('provider returned 400\\n'); raise SystemExit(1)\n"+ "out.parent.mkdir(parents=True, exist_ok=True)\n"+ f"block = {WET_BLOCK!r}\n"+ "block['model'] = model\n"+ "out.write_text(json.dumps(block))\n",+ encoding="utf-8",+ )+ monkeypatch.setattr(matrix, "RUN_EVALS", script)+ monkeypatch.setenv("OPENAI_API_KEY", "test-key")+ return script+++class TestUnmeasuredIsNotZero:+ def test_no_key_reports_not_measured(self, monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.delenv("OPENAI_API_KEY", raising=False)+ monkeypatch.delenv("WET_EVAL_OPENAI_KEY", raising=False)+ assert matrix.has_live_key() is False++ result = matrix.ModelResult("gpt-4o", measured=False, error="no live key")+ assert "metrics" not in result.to_dict()++ def test_the_table_says_not_measured_rather_than_dashes(self) -> None:+ """Dashes across seven columns read as zero at a glance."""+ table = matrix.render_markdown([matrix.ModelResult("gpt-4o", measured=False, error="no live key")])+ assert "not measured" in table+ assert "0.0%" not in table++ def test_absent_key_exits_zero(self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture) -> None:+ """An absent key is a configuration state, not a build failure.+ Failing here would make every fork's CI red."""+ monkeypatch.delenv("OPENAI_API_KEY", raising=False)+ monkeypatch.delenv("WET_EVAL_OPENAI_KEY", raising=False)+ assert matrix.main(["--models", "gpt-4o"]) == 0+++class TestMeasurement:+ def test_each_model_is_graded(self, stub_eval: Path, tmp_path: Path) -> None:+ out = tmp_path / "matrix.json"+ matrix.main(["--models", "a,b", "--out", str(out)])+ payload = json.loads(out.read_text())+ assert [r["model"] for r in payload["results"]] == ["a", "b"]+ assert all(r["measured"] for r in payload["results"])++ def test_every_pin_role_is_set(self, stub_eval: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:+ """Setting one role leaves the others on their default model, which+ makes the matrix a comparison of one agent rather than the system."""+ captured: dict = {}+ real_run = subprocess.run++ def spy(cmd, **kwargs):+ captured.update(kwargs.get("env") or {})+ return real_run(cmd, **kwargs)++ monkeypatch.setattr(matrix.subprocess, "run", spy)+ matrix.main(["--models", "modelX"])+ for role in matrix.PIN_ROLES:+ assert captured.get(f"AISOC_MODEL_PIN_{role.upper()}") == "modelX"++ def test_a_failing_model_is_a_result_not_an_abort(self, stub_eval: Path, tmp_path: Path) -> None:+ """A model that cannot complete the corpus is exactly what a reader+ wants to know, and must not take the other rows with it."""+ out = tmp_path / "matrix.json"+ matrix.main(["--models", "good,broken,alsogood", "--out", str(out)])+ payload = json.loads(out.read_text())+ by_model = {r["model"]: r for r in payload["results"]}+ assert by_model["good"]["measured"] is True+ assert by_model["alsogood"]["measured"] is True+ assert by_model["broken"]["measured"] is False+ assert "400" in by_model["broken"]["error"]++ def test_the_projection_keeps_the_comparable_fields(self) -> None:+ projected = matrix._project(WET_BLOCK)+ for key in ("mitre_accuracy", "abstention_rate", "usd_per_incident", "latency_p95_s"):+ assert projected[key] is not None++ def test_a_limited_run_is_marked_partial(self, stub_eval: Path, tmp_path: Path) -> None:+ """A limited matrix is not comparable with a full one, and must not+ be published as one."""+ out = tmp_path / "matrix.json"+ matrix.main(["--models", "a", "--limit", "10", "--out", str(out)])+ payload = json.loads(out.read_text())+ assert payload["partial"] is True+ assert payload["limit"] == 10+++def test_it_wraps_the_real_eval_rather_than_reimplementing_it() -> None:+ """A second definition of "accuracy" can drift from the one on the+ scoreboard — which is exactly what the alert-reduction suite did.+ """+ source = MODULE.read_text(encoding="utf-8")+ assert "run_evals.py" in source+ for reimplementation in ("def grade", "def score_incident", "def _accuracy"):+ assert reimplementation not in source, (+ f"{reimplementation} suggests the matrix is computing its own " f"metrics rather than collating the eval's"+ )