Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

Model matrix — the last open item in Phase 4 (#715)

Beenu Aroraffd2ee90264e
diff --git a/.github/workflows/wet-eval.yml b/.github/workflows/wet-eval.yml
index 9408c1868..05172389f 100644
--- a/.github/workflows/wet-eval.yml
+++ b/.github/workflows/wet-eval.yml
@@ -158,6 +158,31 @@ jobs:
--out /tmp/wet-eval/report.json \
--wet-out /tmp/wet-eval/wet-block.json
+ # ──────────────────────────────────────────────
+ # Step 3b — model matrix (Phase 4c).
+ # The single-model wet eval above says how the agent performs on one
+ # backend; it cannot say whether the result is a property of the
+ # agent or of the model. The matrix grades the same corpus across
+ # several, one variable at a time.
+ #
+ # Wrapper, not a second evaluator: it sets the model pins and
+ # re-invokes run_evals.py, so there is one definition of accuracy
+ # rather than two that can drift.
+ # ──────────────────────────────────────────────
+ - name: Run model matrix
+ if: steps.preflight.outputs.should_run == 'True'
+ env:
+ WET_EVAL_OPENAI_KEY: ${{ secrets.WET_EVAL_OPENAI_KEY }}
+ OPENAI_API_KEY: ${{ secrets.WET_EVAL_OPENAI_KEY }}
+ AISOC_MATRIX_MODELS: ${{ vars.AISOC_MATRIX_MODELS || 'gpt-4o-mini,gpt-4o' }}
+ run: |
+ set -euo pipefail
+ python3 scripts/run_model_matrix.py \
+ --out /tmp/wet-eval/model-matrix.json \
+ --markdown-out /tmp/wet-eval/model-matrix.md
+ echo "### Model matrix" >> "${GITHUB_STEP_SUMMARY}"
+ cat /tmp/wet-eval/model-matrix.md >> "${GITHUB_STEP_SUMMARY}"
+
# ──────────────────────────────────────────────
# Step 4 — rewrite the wet-eval tables in benchmark.md.
# The `--check` flag fails the step if the markdown scaffold
diff --git a/ROADMAP.md b/ROADMAP.md
index 661b1ebe2..fde259ea2 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -26,7 +26,7 @@ A proof-first, security-first program to make every README claim gate-backed, cl
- [x] Phase 1 — Four existential holes (prompt injection, memory poisoning, cross-store tenant isolation, data-exfiltration/redaction, cost DoS, vault)
- [x] Phase 2 — Supply chain + truth (security.yml scanners, claim-gate ratchet, hard-fail insecure prod defaults, TRADEMARK, verifying-releases, license fix; continuation landed: per-image cosign signatures + CycloneDX SBOM attestations + SLSA provenance, all actions SHA-pinned)
- [x] Phase 3 — Integration / E2E / chaos / DR (real-container spine test, backup-restore, chaos, upgrade, cross-store isolation live-replay; heavy-demo-stack Playwright E2E + demo-timing gate tracked as non-blocking 3.5+)
-- [ ] Phase 4 — Real evals + detection content truth table (third-party-labeled corpus, hallucination/calibration/abstention, model matrix) — **4a/4b landed** (de-circularised DAC candidate-rule gate + honest executable-vs-imported truth table); 4c+ live-agent eval / calibration / model matrix pending
+- [ ] Phase 4 — Real evals + detection content truth table (third-party-labeled corpus, hallucination/calibration/abstention, model matrix) — **4a/4b/4c landed**: de-circularised DAC candidate-rule gate; honest executable-vs-imported truth table; hallucination, abstention, calibration and containment metrics in `packages/aisoc-benchmark` with a documented adapter so a third-party agent can be graded on the same corpus; confidence calibration gated in `test_confidence_calibration.py`; and a **model matrix** (`scripts/run_model_matrix.py`) wired into the weekly wet eval, which grades the same corpus across several models by re-invoking the existing evaluator rather than defining a second notion of accuracy. **Remains open on one thing only, and it is not code:** the live-agent numbers need a funded provider key. Without one the matrix reports *not measured* per model rather than emitting zeros, because a zero is a measurement and "we did not run this" is not.
- [x] Phase 5 — Data spine correctness (versioned event-schema registry + dead-letter queue + source-event lineage in the fusion consumer; idempotency via AlertSink dedup + event-time watermarking; backfill/replay-from-offset tracked as 5b)
- [x] Phase 6 — Performance + cost (fusion hot-path throughput harness with a generous regression-floor gate; deterministic storage $/TB cost model + drift gate; storage-consolidation ADR-0005)
- [x] Phase 7 — Ingest-time graph + multi-model router — **graph-at-ingest already shipped** (v8 T1.1, `services/ingest/internal/graph/`); **7a landed**: unified deterministic→ML→LLM router with tier attribution + the `AISOC_DETERMINISTIC` determinism contract (`services/agents/app/routing/model_router.py`, gated). 7b+ (posture collection, effective-permissions snapshot loader, bi-temporal valid_from/valid_to, fusion-time ContextBundle) tracked in `docs/audit/PROGRESS.md`
@@ -230,8 +230,15 @@ Implemented and reviewed by Beenu Arora <[email protected]>.
- [x] Case auto-summary + PDF export (`case_summary.py` + `case_summary_html.py`)
- [x] Investigation timeline (replayable) (`InvestigationTimeline.tsx`)
- [x] Playbook gallery with 12 curated packs + GitHub PR integration for detection proposals
-- [ ] Mobile responder console (React Native) — triage and acknowledge from phone _(deferred to v8.0)_
-- [ ] Plugin publishing marketplace v3 (commercial plugins, revenue sharing) _(deferred to v8.0)_
+- [ ] Mobile responder console (React Native) — triage and acknowledge from phone
+ _(deferred past v8.0; **not started** — no React Native code exists in the
+ tree. v8.0 shipped without it and nothing in `docs/roadmap/v8-progress.md`
+ claimed otherwise, so the previous "deferred to v8.0" note had quietly
+ become false.)_
+- [ ] Plugin publishing marketplace v3 (commercial plugins, revenue sharing)
+ _(deferred past v8.0; **not started**. Revenue sharing is a commercial
+ decision rather than an engineering one, and the free packaging path is
+ itself still blocked on registry credentials — see v8.1.)_
---
diff --git a/scripts/run_model_matrix.py b/scripts/run_model_matrix.py
new file mode 100644
index 000000000..4de68061a
--- /dev/null
+++ b/scripts/run_model_matrix.py
@@ -0,0 +1,244 @@
+#!/usr/bin/env python3
+"""Grade the same corpus across several models and publish the comparison.
+
+Phase 4's last open item. The wet eval runs one model — whichever the pins
+resolve to — so the published numbers describe the agent *on that model*
+and say nothing about whether the result is a property of the agent or of
+the backend. Those are different claims, and only one of them is about
+this project.
+
+A matrix answers it: same corpus, same prompts, same grader, one variable.
+If accuracy collapses on a cheaper model, that is a fact a reader deciding
+what to self-host needs, and it is a fact no single-model run can produce.
+
+Deliberately a wrapper, not a second evaluator. It sets the model pin and
+invokes `scripts/run_evals.py --wet` per model, then collates the blocks
+that script already emits. Reimplementing the eval would create a second
+definition of "accuracy" that could drift from the one on the scoreboard —
+the exact defect the alert-reduction suite had, where an in-test
+reimplementation published a number the product did not produce.
+
+Without a funded key it reports **not measured** and exits zero. It does
+not emit zeros: a zero is a measurement, and "we did not run this" is not.
+
+Run: python3 scripts/run_model_matrix.py --models gpt-4o-mini,gpt-4o
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import subprocess
+import sys
+import tempfile
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any
+
+REPO_ROOT = Path(__file__).resolve().parent.parent
+RUN_EVALS = REPO_ROOT / "scripts" / "run_evals.py"
+
+#: Roles the pin override has to cover. Setting only one leaves the other
+#: agents on their default model, which would make the matrix a comparison
+#: of one role rather than of the system.
+PIN_ROLES = ("triage", "investigation", "report", "recon")
+
+
+@dataclass
+class ModelResult:
+ model: str
+ measured: bool
+ #: Populated only when measured. Absent rather than zeroed, because a
+ #: zero accuracy and an unrun model must not render the same.
+ metrics: dict[str, Any] = field(default_factory=dict)
+ error: str | None = None
+
+ def to_dict(self) -> dict[str, Any]:
+ out: dict[str, Any] = {"model": self.model, "measured": self.measured}
+ if self.measured:
+ out["metrics"] = self.metrics
+ if self.error:
+ out["error"] = self.error
+ return out
+
+
+def has_live_key() -> bool:
+ return bool((os.getenv("WET_EVAL_OPENAI_KEY") or os.getenv("OPENAI_API_KEY") or "").strip())
+
+
+def run_one(model: str, *, limit: int | None, timeout: int) -> ModelResult:
+ """Run the existing wet eval with every model pin set to `model`."""
+ env = dict(os.environ)
+ for role in PIN_ROLES:
+ env[f"AISOC_MODEL_PIN_{role.upper()}"] = model
+ # The agent's resolver reads OPENAI_API_KEY; the workflow supplies the
+ # key under the wet-eval name.
+ if env.get("WET_EVAL_OPENAI_KEY") and not env.get("OPENAI_API_KEY"):
+ env["OPENAI_API_KEY"] = env["WET_EVAL_OPENAI_KEY"]
+
+ with tempfile.TemporaryDirectory() as tmp:
+ wet_out = Path(tmp) / "wet.json"
+ cmd = [
+ sys.executable,
+ str(RUN_EVALS),
+ "--wet",
+ "--out",
+ str(Path(tmp) / "report.json"),
+ "--wet-out",
+ str(wet_out),
+ ]
+ if limit:
+ cmd += ["--limit", str(limit)]
+
+ try:
+ completed = subprocess.run(
+ cmd,
+ env=env,
+ capture_output=True,
+ text=True,
+ timeout=timeout,
+ cwd=REPO_ROOT,
+ )
+ except subprocess.TimeoutExpired:
+ return ModelResult(
+ model,
+ measured=False,
+ error=f"timed out after {timeout}s",
+ )
+
+ if completed.returncode != 0:
+ # The model's own failure is a result about that model, not a
+ # reason to abandon the matrix — a model that cannot complete
+ # the corpus is exactly what a reader wants to know.
+ tail = (completed.stderr or completed.stdout or "").strip().splitlines()
+ return ModelResult(
+ model,
+ measured=False,
+ error=tail[-1][:300] if tail else f"exit {completed.returncode}",
+ )
+
+ if not wet_out.exists():
+ return ModelResult(model, measured=False, error="no wet block produced")
+
+ block = json.loads(wet_out.read_text(encoding="utf-8"))
+
+ return ModelResult(model, measured=True, metrics=_project(block))
+
+
+def _project(block: dict[str, Any]) -> dict[str, Any]:
+ """The comparable subset. A full wet block is large and most of it is
+ identical across models; the point of the matrix is the part that is
+ not."""
+ latency = block.get("latency_seconds") or {}
+ tokens = (block.get("tokens") or {}).get("total") or {}
+ usd = block.get("usd") or {}
+ return {
+ "incidents": block.get("incidents"),
+ "mitre_accuracy": block.get("mitre_accuracy"),
+ "abstention_rate": block.get("abstention_rate"),
+ "mean_groundedness": block.get("mean_groundedness"),
+ "latency_p95_s": latency.get("p95"),
+ "tokens_mean": tokens.get("mean"),
+ "usd_total": usd.get("total"),
+ "usd_per_incident": usd.get("mean"),
+ }
+
+
+def render_markdown(results: list[ModelResult]) -> str:
+ lines = [
+ "| Model | Incidents | MITRE accuracy | Abstention | Groundedness | p95 latency | Tokens/inv | $/incident |",
+ "|-------|-----------|----------------|------------|--------------|-------------|------------|------------|",
+ ]
+ for result in results:
+ if not result.measured:
+ # One cell, spanning the intent: this row was not measured.
+ # Filling the columns with dashes reads as zero at a glance.
+ reason = result.error or "no live key configured"
+ lines.append(f"| `{result.model}` | _not measured — {reason}_ |||||||")
+ continue
+ m = result.metrics
+
+ # `m` is bound as a default rather than captured: the closure is
+ # used inside this iteration, so capturing works today and breaks
+ # the moment anyone defers the call.
+ def fmt(key: str, spec: str = "{:.3f}", metrics: dict = m) -> str:
+ value = metrics.get(key)
+ return spec.format(value) if isinstance(value, int | float) else "—"
+
+ lines.append(
+ f"| `{result.model}` | {m.get('incidents') or '—'} "
+ f"| {fmt('mitre_accuracy', '{:.1%}')} "
+ f"| {fmt('abstention_rate', '{:.1%}')} "
+ f"| {fmt('mean_groundedness')} "
+ f"| {fmt('latency_p95_s', '{:.2f}s')} "
+ f"| {fmt('tokens_mean', '{:.0f}')} "
+ f"| {fmt('usd_per_incident', '${:.4f}')} |"
+ )
+ return "\n".join(lines)
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument(
+ "--models",
+ default=os.getenv("AISOC_MATRIX_MODELS", "gpt-4o-mini,gpt-4o"),
+ help="Comma-separated model names to grade.",
+ )
+ parser.add_argument("--out", type=Path, help="Write the JSON result here.")
+ parser.add_argument("--markdown-out", type=Path, help="Write the table here.")
+ parser.add_argument(
+ "--limit",
+ type=int,
+ help="Grade only the first N incidents. For a smoke run; a limited " "matrix must not be published as a full one.",
+ )
+ parser.add_argument("--timeout", type=int, default=3600)
+ args = parser.parse_args(argv)
+
+ models = [m.strip() for m in args.models.split(",") if m.strip()]
+ if not models:
+ print("model-matrix: no models requested", file=sys.stderr)
+ return 2
+
+ live = has_live_key()
+ if not live:
+ print("model-matrix: no live LLM key configured — reporting 'not measured' " "for every model rather than emitting zeros.")
+
+ results = [
+ run_one(model, limit=args.limit, timeout=args.timeout)
+ if live
+ else ModelResult(model, measured=False, error="no live key configured")
+ for model in models
+ ]
+
+ payload = {
+ "corpus": "synthetic_incidents.json",
+ "limit": args.limit,
+ "partial": bool(args.limit),
+ "live": live,
+ "results": [r.to_dict() for r in results],
+ }
+
+ if args.out:
+ args.out.parent.mkdir(parents=True, exist_ok=True)
+ args.out.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
+ table = render_markdown(results)
+ if args.markdown_out:
+ args.markdown_out.parent.mkdir(parents=True, exist_ok=True)
+ args.markdown_out.write_text(table + "\n", encoding="utf-8")
+
+ print()
+ print(table)
+ print()
+ measured = sum(1 for r in results if r.measured)
+ print(f"model-matrix: {measured}/{len(results)} model(s) measured")
+ if args.limit:
+ print(f" PARTIAL: limited to {args.limit} incidents — not comparable " f"with a full run and must not be published as one.")
+ # Exit zero even with nothing measured: an absent key is a
+ # configuration state, not a build failure, and failing here would
+ # make every fork's CI red.
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/tests/test_model_matrix.py b/tests/test_model_matrix.py
new file mode 100644
index 000000000..80d28b48c
--- /dev/null
+++ b/tests/test_model_matrix.py
@@ -0,0 +1,151 @@
+"""The model matrix must never invent a number it did not measure.
+
+Phase 4's last open item was a model matrix: the wet eval runs one model,
+so its numbers describe the agent *on that model* and say nothing about
+whether the result is a property of the agent or of the backend. Those are
+different claims and only one is about this project.
+
+Which makes the honesty requirement sharper than usual. A matrix exists to
+be compared across rows, so a row that reads `0.000` because nothing ran
+is worse than a missing row — it invites the conclusion that the model was
+graded and scored zero.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+import json
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+REPO_ROOT = Path(__file__).resolve().parent.parent
+MODULE = REPO_ROOT / "scripts" / "run_model_matrix.py"
+
+spec = importlib.util.spec_from_file_location("run_model_matrix", MODULE)
+assert spec and spec.loader
+matrix = importlib.util.module_from_spec(spec)
+sys.modules["run_model_matrix"] = matrix
+spec.loader.exec_module(matrix)
+
+
+WET_BLOCK = {
+ "model": "stub",
+ "incidents": 200,
+ "mitre_accuracy": 0.97,
+ "abstention_rate": 0.04,
+ "mean_groundedness": 0.93,
+ "latency_seconds": {"p95": 4.2},
+ "tokens": {"total": {"mean": 3100}},
+ "usd": {"total": 1.84, "mean": 0.0092},
+}
+
+
+def stub_eval(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
+ """A stand-in for run_evals.py emitting the same wet-block shape."""
+ script = tmp_path / "fake_run_evals.py"
+ script.write_text(
+ "import json, os, sys\n"
+ "from pathlib import Path\n"
+ "args = sys.argv[1:]\n"
+ "out = Path(args[args.index('--wet-out') + 1])\n"
+ "model = os.environ.get('AISOC_MODEL_PIN_TRIAGE', 'unknown')\n"
+ "if model == 'broken':\n"
+ " sys.stderr.write('provider returned 400\\n'); raise SystemExit(1)\n"
+ "out.parent.mkdir(parents=True, exist_ok=True)\n"
+ f"block = {WET_BLOCK!r}\n"
+ "block['model'] = model\n"
+ "out.write_text(json.dumps(block))\n",
+ encoding="utf-8",
+ )
+ monkeypatch.setattr(matrix, "RUN_EVALS", script)
+ monkeypatch.setenv("OPENAI_API_KEY", "test-key")
+ return script
+
+
+class TestUnmeasuredIsNotZero:
+ def test_no_key_reports_not_measured(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.delenv("OPENAI_API_KEY", raising=False)
+ monkeypatch.delenv("WET_EVAL_OPENAI_KEY", raising=False)
+ assert matrix.has_live_key() is False
+
+ result = matrix.ModelResult("gpt-4o", measured=False, error="no live key")
+ assert "metrics" not in result.to_dict()
+
+ def test_the_table_says_not_measured_rather_than_dashes(self) -> None:
+ """Dashes across seven columns read as zero at a glance."""
+ table = matrix.render_markdown([matrix.ModelResult("gpt-4o", measured=False, error="no live key")])
+ assert "not measured" in table
+ assert "0.0%" not in table
+
+ def test_absent_key_exits_zero(self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture) -> None:
+ """An absent key is a configuration state, not a build failure.
+ Failing here would make every fork's CI red."""
+ monkeypatch.delenv("OPENAI_API_KEY", raising=False)
+ monkeypatch.delenv("WET_EVAL_OPENAI_KEY", raising=False)
+ assert matrix.main(["--models", "gpt-4o"]) == 0
+
+
+class TestMeasurement:
+ def test_each_model_is_graded(self, stub_eval: Path, tmp_path: Path) -> None:
+ out = tmp_path / "matrix.json"
+ matrix.main(["--models", "a,b", "--out", str(out)])
+ payload = json.loads(out.read_text())
+ assert [r["model"] for r in payload["results"]] == ["a", "b"]
+ assert all(r["measured"] for r in payload["results"])
+
+ def test_every_pin_role_is_set(self, stub_eval: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+ """Setting one role leaves the others on their default model, which
+ makes the matrix a comparison of one agent rather than the system."""
+ captured: dict = {}
+ real_run = subprocess.run
+
+ def spy(cmd, **kwargs):
+ captured.update(kwargs.get("env") or {})
+ return real_run(cmd, **kwargs)
+
+ monkeypatch.setattr(matrix.subprocess, "run", spy)
+ matrix.main(["--models", "modelX"])
+ for role in matrix.PIN_ROLES:
+ assert captured.get(f"AISOC_MODEL_PIN_{role.upper()}") == "modelX"
+
+ def test_a_failing_model_is_a_result_not_an_abort(self, stub_eval: Path, tmp_path: Path) -> None:
+ """A model that cannot complete the corpus is exactly what a reader
+ wants to know, and must not take the other rows with it."""
+ out = tmp_path / "matrix.json"
+ matrix.main(["--models", "good,broken,alsogood", "--out", str(out)])
+ payload = json.loads(out.read_text())
+ by_model = {r["model"]: r for r in payload["results"]}
+ assert by_model["good"]["measured"] is True
+ assert by_model["alsogood"]["measured"] is True
+ assert by_model["broken"]["measured"] is False
+ assert "400" in by_model["broken"]["error"]
+
+ def test_the_projection_keeps_the_comparable_fields(self) -> None:
+ projected = matrix._project(WET_BLOCK)
+ for key in ("mitre_accuracy", "abstention_rate", "usd_per_incident", "latency_p95_s"):
+ assert projected[key] is not None
+
+ def test_a_limited_run_is_marked_partial(self, stub_eval: Path, tmp_path: Path) -> None:
+ """A limited matrix is not comparable with a full one, and must not
+ be published as one."""
+ out = tmp_path / "matrix.json"
+ matrix.main(["--models", "a", "--limit", "10", "--out", str(out)])
+ payload = json.loads(out.read_text())
+ assert payload["partial"] is True
+ assert payload["limit"] == 10
+
+
+def test_it_wraps_the_real_eval_rather_than_reimplementing_it() -> None:
+ """A second definition of "accuracy" can drift from the one on the
+ scoreboard — which is exactly what the alert-reduction suite did.
+ """
+ source = MODULE.read_text(encoding="utf-8")
+ assert "run_evals.py" in source
+ for reimplementation in ("def grade", "def score_incident", "def _accuracy"):
+ assert reimplementation not in source, (
+ f"{reimplementation} suggests the matrix is computing its own " f"metrics rather than collating the eval's"
+ )