Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

Release v21.0.0 — the eval-honesty major (#1582)

Beenu Arora152c6d5ff37e
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3f93888bb..0aa3470a9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
+## [21.0.0] - 2026-10-31
+
+### BREAKING
+
+- **Correlation grouping narrows on missing identifiers** (the Changed
+ entry below). On upgrade, *future* alerts that carry no tactic group
+ per rule instead of per entity, and alerts with no identifiers at all
+ stop cross-rule merging entirely. Existing incidents are untouched;
+ the measured effect on a detection-shaped cohort is reduction moving
+ 42.6% → 24.5% because unrelated alerts stop counting as grouped.
+ There is no API change in this release; the major marks this grouping
+ change plus the redefinition of published evaluation semantics
+ (serving-model provenance, full-cohort denominators, closure
+ contamination) that older scorecards do not carry.
+
### Added
- **Serving-model attribution, end to end.** Every published verdict-eval
diff --git a/README.md b/README.md
index 810995780..38b534ac7 100644
--- a/README.md
+++ b/README.md
@@ -6,8 +6,8 @@
**An open-source, self-hostable AI Security Operations Center.** It ingests your security telemetry, detects and correlates threats, investigates them with AI agents whose reasoning is fully auditable, and proposes responses a human approves.
-[![License: MIT](https://img.shields.io/badge/License-MIT-22c55e.svg?style=flat-square)](https://opensource.org/licenses/MIT) [![Version](https://img.shields.io/badge/version-20.4.0-f59e0b?style=flat-square)](CHANGELOG.md) [![CI](https://img.shields.io/github/actions/workflow/status/beenuar/AiSOC/ci.yml?branch=main&label=CI&style=flat-square)](https://github.com/beenuar/AiSOC/actions/workflows/ci.yml)
-[![CodeQL](https://img.shields.io/github/actions/workflow/status/beenuar/AiSOC/codeql.yml?branch=main&label=CodeQL&style=flat-square)](https://github.com/beenuar/AiSOC/actions/workflows/codeql.yml) [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/beenuar/AiSOC/badge)](https://securityscorecards.dev/viewer/?uri=github.com/beenuar/AiSOC) [![Technical Guide](https://img.shields.io/badge/Technical%20Guide-26%20page%20PDF-dc2626?style=flat-square)](https://raw.githubusercontent.com/beenuar/AiSOC/main/apps/web/public/papers/aisoc-technical-guide.pdf)
+[![License: MIT](https://img.shields.io/badge/License-MIT-22c55e.svg?style=flat-square)](https://opensource.org/licenses/MIT) [![Version](https://img.shields.io/badge/version-21.0.0-f59e0b?style=flat-square)](CHANGELOG.md) [![CI](https://img.shields.io/github/actions/workflow/status/beenuar/AiSOC/ci.yml?branch=main&label=CI&style=flat-square)](https://github.com/beenuar/AiSOC/actions/workflows/ci.yml)
+[![CodeQL](https://img.shields.io/github/actions/workflow/status/beenuar/AiSOC/codeql.yml?branch=main&label=CodeQL&style=flat-square)](https://github.com/beenuar/AiSOC/actions/workflows/codeql.yml) [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/beenuar/AiSOC/badge)](https://securityscorecards.dev/viewer/?uri=github.com/beenuar/AiSOC) [![Technical Guide](https://img.shields.io/badge/Technical%20Guide-27%20page%20PDF-dc2626?style=flat-square)](https://raw.githubusercontent.com/beenuar/AiSOC/main/apps/web/public/papers/aisoc-technical-guide.pdf)
**[Technical Guide (PDF)](https://raw.githubusercontent.com/beenuar/AiSOC/main/apps/web/public/papers/aisoc-technical-guide.pdf)** · [Docs](https://beenuar.github.io/AiSOC/) · [Architecture](docs/architecture/README.md) · [What actually works](docs/audit/REPOSITORY_REALITY.md) · [Discussions](https://github.com/beenuar/AiSOC/discussions)
@@ -159,17 +159,15 @@ exercised here** — there is no funded key, so per-model rows read *not measure
Vulnerabilities catalog — public, no API key — into the console's Threat Intelligence page: the one
thing in a fresh install that is neither synthetic nor yours.
-## Real vs synthetic data
+## Evaluation, reflex and replay
-| Kind | Where | How you can tell |
-|---|---|---|
-| **Real** | Your connectors and the ingest API | `is_synthetic = false` (the default) |
-| **Real, and not yours** | The CISA KEV feed on the Threat Intelligence page | Every row carries `source: cisa-kev`; it is the public catalog, unmodified |
-| **Sample** | The wizard's **Load sample data** | Source column reads `AiSOC`; RFC 5737 / RFC 2606 reserved addresses only |
-| **Demo** | `make demo` | `is_synthetic = true`, labelled in the console |
-| **Benchmark / fixtures** | `services/agents/tests/eval_data/`, `**/tests/` | Published rows carry `substrate: true`; fixtures never ship in an image |
+The published numbers carry their own provenance. ([methodology](https://beenuar.github.io/AiSOC/docs/benchmark) · [real vs synthetic data](apps/docs/docs/concepts/real-vs-synthetic.md))
-**Production never silently falls back to synthetic data.** An unreachable backend makes the console name the failure rather than invent an investigation, and an unmeasured figure reads *not measured*, never `0`. It was not always so: [the reality audit](docs/audit/REPOSITORY_REALITY.md) has each case.
+- **Every verdict scorecard names the model that actually served it** — gateway-attested or self-reported, with an explicit provenance status. A run with missing attribution, zero real calls or a deterministic fallback cannot be advertised as a named model's performance, and a measurement that cannot be bound to an attributed call cannot authorize closures.
+- **Denominators are published, and the uncomfortable direction wins.** Full-cohort recall keeps abstentions and failures in the malicious denominator; closure claims ship with their contamination rate; alert reduction has three labelled figures (75.3% legacy grouping, 33.3% production key, 24.5% on a detection-shaped cohort after a missing identifier was fixed to *narrow* the group instead of widening it). Unmeasured reads *not measured*, never `0`.
+- **The reflex path is observable** — nine named stages from queue delay to verification, and a decision contract carrying supporting evidence, contradictory evidence, missing prerequisites and the escalation reason.
+- **Replay is time-frozen; re-hunting is labelled.** Historical replay enforces evidence cutoffs in the retrieval queries (never by prompt), discloses the model-knowledge limit, and cannot execute real actions — the mutation boundary is the write adapter. Applying new intelligence to old telemetry is a distinct, rehearsable mode, and its discoveries are never counted as predictions at the original event time.
+- **Model suitability is per task.** The matrix grades each configured model on the same locked evidence against a deterministic baseline floor (balanced accuracy exactly 0.250); unfunded rows read *not measured*, with one-command reproduction against any local OpenAI-compatible endpoint.
## AI agents
diff --git a/RELEASES.md b/RELEASES.md
index 4286dc872..1934b42d4 100644
--- a/RELEASES.md
+++ b/RELEASES.md
@@ -2,7 +2,9 @@
This file mirrors what used to live in the "What's new" section of [`README.md`](README.md). The complete, machine-readable inventory (with file paths, env-var diffs, and per-release test counts) lives in [`CHANGELOG.md`](CHANGELOG.md).
-> **TL;DR for first-time visitors:** AiSOC is on `v20.4.0`, released 2026-10-30. **A fix release closing two defects a self-hoster hit the moment they turned SSO on.** Activating SSO OIDC locked the administrator out of their own tenant — 403 on users, investigations and the alert queue all at once (reported in [discussion #1568](https://github.com/beenuar/AiSOC/discussions/1568)). The cause: permissions fall back to the static role map only while a tenant holds zero role *grants*, and the bootstrap admin carries `users.role = 'admin'` with no grant row (it is created after the seed that would have backfilled one). That is harmless until the first SSO sign-in grants the provisioned user a role — which flips the whole tenant to database-backed RBAC, and from that instant the grant-less admin resolves to zero permissions. `bootstrap_admin` now grants the admin its catalog role on every startup, in an isolated session, a no-op when the catalog is unseeded, and **self-healing: a deployment already locked out recovers on the next API restart**. The same report surfaced a second bug: the alert funnel's "events of interest" count queried `aisoc.raw_events.ingested_at`, a column the lake does not have (it has `event_time` and `ingest_time`), so the query failed with ClickHouse `Code: 47` on every call and the console silently served a Postgres estimate; it now reads `ingest_time`, with a test that checks the query's columns against the committed ClickHouse DDL. Both fixes are verified against a real Postgres with the full migration chain — the lockout is reproduced in a test and then shown fixed, and the deprovisioning semantics the resolver relies on are untouched and still separately pinned. **2 defects were fixed in this release, 22 across it and the two before** (2 here, 10 in v20.3.0, 10 in v20.2.1 — counting distinct product or CI defects named in each release's Fixed and Security sections, dependency-version advisories excluded). No breaking changes; **neither `stable` channel moves**. The chart publishes as **`7.13.0`** — an `appVersion`-only move mirrors the app's minor, as `7.10.0`/`7.12.0` did before it. Everything below describes v20.3.0 and earlier, which this release does not change.
+> **TL;DR for first-time visitors:** AiSOC is on `v21.0.0`, released 2026-10-31. **The eval-honesty major: the platform's published numbers now carry their own provenance, denominators and safety outcomes — and three communities' requests landed as shipped code in one wave.** Every verdict scorecard records which model *actually served* each call (gateway-verified where the gateway attests it, self-reported where only the response body says so) with an explicit provenance status — a run with missing attribution, zero real calls or a deterministic fallback can no longer be advertised as a named model's performance, and an autonomy measurement that cannot be bound to an attributed call can no longer authorize closures. The replay scorer publishes full-cohort recall (abstentions and failures stay in the malicious denominator), the four closure metrics including contamination, and reliability counts; the model matrix grades the verdict task per model with runtime, sampling and cost recorded, over a deterministic baseline floor (balanced accuracy exactly 0.250 — the number models are read against). Alert reduction now has three labelled figures instead of one flattering one: 75.3% (legacy grouping, not what production runs), 33.3% (the production key), and **24.5% on a detection-shaped cohort after the correlation fix** — a missing identifier now narrows the group instead of widening it (cross-rule merges on that cohort: 36 → 0), proven against live Redis including the creation race. The reflex path reports nine named stages and a decision contract (supporting *and* contradictory evidence, missing prerequisites, escalation reason); retro-hunt gained a rehearsal mode whose mutation boundary is enforced at the write adapter, never the prompt. The same wave shipped the **business-context plan** — tenant rules reach the triage model and impact statements must cite the rules that fired or be dropped and counted; asset criticality orders the queue, weighs fusion confidence (unknown renders *unknown*, never zero-as-evidence) and tighten-only raises approval impact on critical assets — plus the **Agentmetry ingest template** for AI coding-agent telemetry ([discussion #1577](https://github.com/beenuar/AiSOC/discussions/1577): their untested "generic-json should work" claim was verified half-true — events landed but every severity flattened to Unknown — and the bundled template fixes exactly that), and the externally contributed **ack-clock escalation bridge**. **This is the behavior change behind the major:** correlation grouping narrows on missing identifiers (future alerts only; existing incidents untouched), and the evaluation-semantics redefinition means pre-v21 scorecards are not comparable to v21 ones — there is no API break. **3 defects were fixed in this release, 15 across it and the two before** (3 here, 2 in v20.4.0, 10 in v20.3.0 — counting distinct product or CI defects named in each release's Fixed and Security sections, dependency-version advisories excluded). **Neither `stable` channel moves** — v21.0.0 is an application major, and both channel guards hold by design until a deliberate dispatch promotes them. The chart publishes as **`7.14.0`** — an `appVersion`-only move mirroring the app's step, exactly as `7.9.0` shipped the v20 major. Everything below describes v20.4.0 and earlier, which this release does not change.
+>
+> **Previously:** AiSOC was on `v20.4.0`, released 2026-10-30. **A fix release closing two defects a self-hoster hit the moment they turned SSO on.** Activating SSO OIDC locked the administrator out of their own tenant — 403 on users, investigations and the alert queue all at once (reported in [discussion #1568](https://github.com/beenuar/AiSOC/discussions/1568)). The cause: permissions fall back to the static role map only while a tenant holds zero role *grants*, and the bootstrap admin carries `users.role = 'admin'` with no grant row (it is created after the seed that would have backfilled one). That is harmless until the first SSO sign-in grants the provisioned user a role — which flips the whole tenant to database-backed RBAC, and from that instant the grant-less admin resolves to zero permissions. `bootstrap_admin` now grants the admin its catalog role on every startup, in an isolated session, a no-op when the catalog is unseeded, and **self-healing: a deployment already locked out recovers on the next API restart**. The same report surfaced a second bug: the alert funnel's "events of interest" count queried `aisoc.raw_events.ingested_at`, a column the lake does not have (it has `event_time` and `ingest_time`), so the query failed with ClickHouse `Code: 47` on every call and the console silently served a Postgres estimate; it now reads `ingest_time`, with a test that checks the query's columns against the committed ClickHouse DDL. Both fixes are verified against a real Postgres with the full migration chain — the lockout is reproduced in a test and then shown fixed, and the deprovisioning semantics the resolver relies on are untouched and still separately pinned. **2 defects were fixed in this release, 22 across it and the two before** (2 here, 10 in v20.3.0, 10 in v20.2.1 — counting distinct product or CI defects named in each release's Fixed and Security sections, dependency-version advisories excluded). No breaking changes; **neither `stable` channel moves**. The chart publishes as **`7.13.0`** — an `appVersion`-only move mirrors the app's minor, as `7.10.0`/`7.12.0` did before it. Everything below describes v20.3.0 and earlier, which this release does not change.
>
> **Previously:** AiSOC was on `v20.3.0`, released 2026-10-30. **The community-contributions release: six externally contributed fixes merged in one pass, the console's front door fixed, and the detection corpus made more honest again.** A clean self-hosted install's `/` rendered the project's *marketing page* — neither the dashboard nor the login page an operator expects (reported in discussion #1549); it now redirects to `/login`, which routes onward by itself (first run → setup wizard, signed in → dashboard), and a deployment that *is* the public site opts back in with the runtime `AISOC_MARKETING_HOME`. The `custom` LLM provider's model id no longer has to be typed by hand: **`POST /api/v1/llm/credentials/models` asks any OpenAI-compatible endpoint what chat models it serves** (externally contributed — the exact vendor-neutral gap the #1481 review offered), bounded, SSRF/air-gap-guarded, and the stored key goes only to the stored base URL. An external report that **1 of 12 GitHub detection rules fired on real connector output was substantially correct**: five rules re-claused to GitHub's documented audit vocabulary (the deploy-key rule matched an action that does not exist — deploy keys log as `public_key.create`), five quarantined with recorded reasons, one claim refuted with measurement, a CI gate now replays every GitHub rule through the real connector (11 findings on the pre-fix tree), and the executable count honestly moves **2,535 → 2,530**. Community fixes from [alexmateescu](https://github.com/alexmateescu) close a retry storm (deterministic gateway 400s now dead-letter on first sight; prompt bytes capped at the egress chokepoint with water-filling truncation), a saved connector 422-ing every test-connection once it had polled, fresh chatops deploys crash-looping on a missing sibling mount, playbook agent-events dialling a port realtime never served, and triage verdicts clipped mid-JSON at 512 output tokens (now 2,048, with the reliability harness moving in lockstep — its parity test going red on the one-line bump is the drift guard working). Three dependency advisories cleared (`@hono/node-server` 2.1.5, `decode-uri-component` 0.5.1, archived-SDK `zod` 4.6.6). **10 defects were fixed in this release, 22 across it and the two before** (10 here, 1 in v20.2.1, 11 in v20.2.0 — counting distinct product or CI defects named in each release's Fixed and Security sections, dependency-version advisories excluded). No breaking changes; **neither `stable` channel moves**. The chart publishes as **`7.12.0`** — nothing inside the chart directory moves except `appVersion`, and the chart's step mirrors the app's, exactly as `7.10.0` shipped v20.1.0's minor on an `appVersion`-only change (a chart *patch* is reserved for shipping an app patch, as `7.11.1` did for v20.2.1). Everything below describes v20.2.1 and earlier, which this release does not change.
>
@@ -32,7 +34,46 @@ This file mirrors what used to live in the "What's new" section of [`README.md`]
## What's new
-`VERSION` is `20.4.0`. The **v20.4.0** release (2026-10-30) closes two
+`VERSION` is `21.0.0`. The **v21.0.0** release (2026-10-31) is the
+eval-honesty major: the platform's published numbers now carry their own
+provenance, denominators and safety outcomes. Serving-model attribution flows
+from the gateway header (verified) or the response body (self-reported,
+credited only when it can witness itself) into every scorecard, ledger
+exchange and eligibility decision — a run with missing attribution, zero real
+calls or a deterministic fallback cannot be advertised as a named model's
+performance, and an autonomy measurement that cannot be bound to an
+attributed call cannot authorize closures. The replay scorer publishes
+full-cohort recall (abstentions and failures stay in the malicious
+denominator), the four closure metrics including contamination, reliability
+counts (requested / attempted / completed / failed / unattempted) and
+per-severity segments. The correlation engine's **behavior change** is the
+reason this is a major: a missing identifier now narrows the group instead of
+widening it — alerts with no tactic group per rule, alerts with no
+identifiers never cross-rule merge — measured on a detection-shaped cohort as
+reduction falling 42.6% → 24.5% while cross-rule merges fell 36 → 0, because
+unrelated alerts stopped counting as grouped; the production-key headline
+stays 33.3%, and a live-Redis suite (expiry, tenant boundary, restart, the
+creation race fixed with `SET NX`) now runs in CI. The reflex path reports
+nine named stages and a structured decision contract; retro-hunt gained a
+rehearsal mode (`RETRO_HUNT_DRY_RUN`) whose mutation boundary is enforced at
+the write adapter; verdict scorecards disclose the model-knowledge limit on
+historical claims. The same wave landed the business-context plan (tenant
+rules reach the triage model with cited impact statements; asset criticality
+orders the queue, weighs confidence and tighten-only raises approval impact),
+the Agentmetry ingest template for AI coding-agent telemetry (discussion
+#1577 — their untested generic-json claim verified half-true and fixed by the
+bundled template), the externally contributed ack-clock escalation bridge,
+an annotated unsupported-claim measurement that found and fixed a real
+claim-support defect, and the verdict model matrix with a deterministic
+baseline floor (balanced accuracy exactly 0.250) against which every model
+row is read — rows without a funded run say *not measured* with one-command
+reproduction. No API breaks. Neither `stable` channel moves. The chart
+publishes as **`7.14.0`**. Everything below describes v20.4.0 and earlier,
+which this release does not change.
+
+---
+
+`VERSION` was `20.4.0`. The **v20.4.0** release (2026-10-30) closes two
defects a self-hoster hit when they enabled SSO (discussion #1568).
Activating SSO OIDC locked the administrator out — 403 on users,
investigations and the alert queue — because permissions fall back to
diff --git a/VERSION b/VERSION
index bd45cd74a..fb5b51303 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-20.4.0
+21.0.0
diff --git a/apps/docs/docs/concepts/real-vs-synthetic.md b/apps/docs/docs/concepts/real-vs-synthetic.md
new file mode 100644
index 000000000..05eb4d574
--- /dev/null
+++ b/apps/docs/docs/concepts/real-vs-synthetic.md
@@ -0,0 +1,31 @@
+---
+id: real-vs-synthetic
+title: Real vs synthetic data
+sidebar_label: Real vs synthetic data
+---
+
+# Real vs synthetic data
+
+Every row in a running AiSOC is one of five kinds, and each kind is
+distinguishable from the console and the API — this table moved here from the
+README front page, unchanged in substance, when the release that redefined the
+published evaluation semantics needed that space.
+
+| Kind | Where | How you can tell |
+|---|---|---|
+| **Real** | Your connectors and the ingest API | `is_synthetic = false` (the default) |
+| **Real, and not yours** | The CISA KEV feed on the Threat Intelligence page | Every row carries `source: cisa-kev`; it is the public catalog, unmodified |
+| **Sample** | The wizard's **Load sample data** | Source column reads `AiSOC`; RFC 5737 / RFC 2606 reserved addresses only |
+| **Demo** | `make demo` | `is_synthetic = true`, labelled in the console |
+| **Benchmark / fixtures** | `services/agents/tests/eval_data/`, `**/tests/` | Published rows carry `substrate: true`; fixtures never ship in an image |
+
+**Production never silently falls back to synthetic data.** An unreachable
+backend makes the console name the failure rather than invent an
+investigation, and an unmeasured figure reads *not measured*, never `0`. It
+was not always so: [the reality audit](https://github.com/beenuar/AiSOC/blob/main/docs/audit/REPOSITORY_REALITY.md)
+has each case.
+
+The same discipline governs the published numbers themselves — which model
+actually served an evaluation, what sits in each metric's denominator, and
+what a replay was allowed to know — described on the
+[benchmark methodology page](../benchmark.md).
diff --git a/apps/docs/sidebars.ts b/apps/docs/sidebars.ts
index 518cc4ae8..d7d95a9da 100644
--- a/apps/docs/sidebars.ts
+++ b/apps/docs/sidebars.ts
@@ -30,6 +30,7 @@ const sidebars: SidebarsConfig = {
"concepts/ai-estate",
"concepts/ai-estate-discovery",
"concepts/groundedness",
+ "concepts/real-vs-synthetic",
"concepts/live-actions",
"concepts/retro-hunts",
"concepts/hunting-agent",
diff --git a/apps/docs/static/data/scoreboard.json b/apps/docs/static/data/scoreboard.json
index f2799ae95..b0e2088cc 100644
--- a/apps/docs/static/data/scoreboard.json
+++ b/apps/docs/static/data/scoreboard.json
@@ -3,9 +3,9 @@
"_readme": "Append-only public benchmark scoreboard. Each row is one eval run. Substrate rows (no LLM, deterministic) are marked substrate:true and MUST NOT be quoted as live agent performance. Wet-eval rows (substrate:false) are appended by the T5.5 weekly CI workflow once it lands. Newest row first.",
"rows": [
{
- "date": "2026-10-30",
- "agent_version": "v20.4.0",
- "commit_sha": "013ecd38",
+ "date": "2026-10-31",
+ "agent_version": "v21.0.0",
+ "commit_sha": "f840cbe2",
"substrate": true,
"eval_mode": "substrate-only",
"mitre_accuracy": 0.97,
diff --git a/apps/docs/static/llms-full.txt b/apps/docs/static/llms-full.txt
index 6f1505a23..5cd15a65a 100644
--- a/apps/docs/static/llms-full.txt
+++ b/apps/docs/static/llms-full.txt
@@ -3,7 +3,7 @@
<!-- Generated by scripts/generate_llms_txt.py. Do not edit by hand; run the script and commit the result. -->
> AiSOC is an open-source, self-hostable AI Security Operations Center, MIT
-> licensed and maintained by the AiSOC community. Version 20.4.0.
+> licensed and maintained by the AiSOC community. Version 21.0.0.
> Repository: https://github.com/beenuar/AiSOC. Documentation: https://beenuar.github.io/AiSOC/.
This file exists so an automated reader can answer questions about AiSOC
@@ -127,7 +127,7 @@ plans/ Historical prototype subtree — reference only, never edited
Every number in this file is generated from the artefact that owns it. If you
need to state one, re-derive it rather than quoting a figure from prose —
-including from this file, which is a snapshot of version 20.4.0.
+including from this file, which is a snapshot of version 21.0.0.
```
python3 scripts/generate_corpus_stats.py --check # executable + on-disk rules
diff --git a/apps/docs/static/llms.txt b/apps/docs/static/llms.txt
index 952277fba..8ff41f3af 100644
--- a/apps/docs/static/llms.txt
+++ b/apps/docs/static/llms.txt
@@ -6,7 +6,7 @@
> security telemetry from 87 first-party connectors, normalises it to OCSF,
> correlates it into alerts, and runs LLM agents that triage and investigate them.
> Every agent decision is written to an investigation ledger, so the reasoning
-> behind a verdict can be replayed step by step. MIT licensed. Current version 20.4.0.
+> behind a verdict can be replayed step by step. MIT licensed. Current version 21.0.0.
AiSOC is built and maintained by the AiSOC community. It is a complete SOC
platform rather than a single tool: ingestion, detection, correlation, agent
diff --git a/apps/web/content/papers/aisoc-technical-guide.md b/apps/web/content/papers/aisoc-technical-guide.md
index da15a3c41..3b56f8b78 100644
--- a/apps/web/content/papers/aisoc-technical-guide.md
+++ b/apps/web/content/papers/aisoc-technical-guide.md
@@ -2,8 +2,8 @@
title: "AiSOC Technical Guide"
subtitle: "Install, operate and integrate — on a workstation and on a server"
author: "AiSOC project"
-version: "v1.1"
-date: "2026-10-28"
+version: "v1.2"
+date: "2026-10-31"
license: "MIT (same as the AiSOC project)"
abstract: |
A complete operator's guide to AiSOC: installing it on a laptop and on a
@@ -778,7 +778,62 @@ the restore, and failures are counted and surfaced.
---
-## 10. Honest limits
+## 10. Evaluation, replay and model suitability
+
+Added in v21.0.0, this is how the platform keeps its own numbers honest. The
+full methodology lives on the benchmark page; what an operator needs is below.
+
+**Serving-model attribution.** Every verdict scorecard, ledger exchange and
+eligibility decision records which model *actually served* each call, with a
+provenance status: `gateway_verified` when the LiteLLM gateway attests the
+serving model in a response header, `self_reported` when only the response
+body names it (credited only when the body can witness itself — a name merely
+equal to the requested alias is never copied in), and `unattributed`
+otherwise. A run that placed zero real calls, fell back to the deterministic
+path, or cannot name its serving model is preserved as a diagnostic and
+refused as "verified performance of model X". The same status gates autonomy:
+a clearing measurement that cannot be bound to an attributed call cannot
+authorize automatic closures.
+
+**Metrics with denominators.** The replay scorer publishes precision, recall,
+per-class F1, confusion matrices, balanced accuracy and MCC with
+numerator/denominator counts; full-cohort malicious recall keeps abstentions
+and runtime failures in the denominator; closure candidates carry four
+honest cuts (malicious closure rate, closure contamination, benign closure
+coverage, unknown-label share), always over *proposed* closures — a simulated
+closure never appears as a production outcome. Requested, attempted,
+completed, failed and unattempted counts travel with every run, so a failure
+that stops a run cannot vanish from the headline.
+
+**Alert reduction, three figures.** 75.3% is the legacy four-tier grouping
+benchmark (not what production runs); 33.3% is the production correlation
+key on the tactic-carrying cohort; 24.5% is a detection-shaped cohort after
+v21.0.0's grouping change, where a missing identifier narrows the group
+instead of widening it. The drop is the honest direction: the difference was
+unrelated alerts counted as grouped.
+
+**Replay and re-hunting.** Historical replay freezes evidence at each
+decision time, enforces the cutoff in retrieval and storage queries rather
+than in the prompt, keeps truth labels reachable by the evaluator and
+unreachable by the agent, and discloses that pretrained weights may postdate
+the events (the scorecard carries the model-knowledge limit, or "unknown").
+Replay cannot execute real actions — the mutation boundary is the write
+adapter, not an instruction. Re-hunting old telemetry with newly published
+intelligence is a separate, labelled mode with a rehearsal switch
+(`RETRO_HUNT_DRY_RUN=1`) that records proposed reopens without mutating
+anything; its discoveries are never counted as predictions at the original
+event time.
+
+**Model suitability is per task.** `scripts/run_model_matrix.py --task
+verdict` grades each configured model on the same locked corpus and records
+serving identity, sampling, runtime and token use per row. Every row is read
+against a deterministic baseline floor (balanced accuracy exactly 0.250 on
+the published corpus — what answering without a model scores). A model or
+configuration with no funded run is listed as *not measured*; reproduction is
+one command against any OpenAI-compatible endpoint, including a local
+Ollama.
+
+## 11. Honest limits
Stated plainly, because a guide that only lists strengths is a brochure.
@@ -787,9 +842,11 @@ Stated plainly, because a guide that only lists strengths is a brochure.
- **Long-duration scale figures do not exist.** The harness has ramp, burst,
backpressure and 24/72-hour soak profiles; the long ones have not been run,
and no throughput number is claimed from them.
-- **Package distribution is blocked on credentials.** The release workflow
- builds and packs all eight npm and PyPI packages on every tag; the upload
- step is credential-gated. Install from source.
+- **Package distribution is half-shipped.** The five PyPI packages are live
+ and resolve at the tree's versions (`pip install aisoc-sandbox` works
+ today); the three npm packages still await the first-upload `NPM_TOKEN`,
+ an account action. The release workflow builds and packs all eight on
+ every tag.
- **Benchmark numbers are labelled.** Three of the four eval metrics are
substrate self-consistency gates, not agent accuracy. Only MITRE accuracy
measures the live agent, and the benchmark page says which is which.
@@ -798,7 +855,7 @@ Stated plainly, because a guide that only lists strengths is a brochure.
---
-## 11. Where to go next
+## 12. Where to go next
| You want | Read |
|---|---|
diff --git a/apps/web/package.json b/apps/web/package.json
index fe850a704..7899f1e35 100644
--- a/apps/web/package.json
+++ b/apps/web/package.json
@@ -1,6 +1,6 @@
{
"name": "@aisoc/web",
- "version": "20.4.0",
+ "version": "21.0.0",
"private": true,
"description": "AiSOC SOC Console - Next.js 14 Frontend",
"license": "MIT",
diff --git a/apps/web/public/papers/aisoc-technical-guide.pdf b/apps/web/public/papers/aisoc-technical-guide.pdf
index 357b046fb..2574c4a57 100644
Binary files a/apps/web/public/papers/aisoc-technical-guide.pdf and b/apps/web/public/papers/aisoc-technical-guide.pdf differ
diff --git a/apps/web/public/papers/render-manifest.json b/apps/web/public/papers/render-manifest.json
index f6d26dca6..545204ba4 100644
--- a/apps/web/public/papers/render-manifest.json
+++ b/apps/web/public/papers/render-manifest.json
@@ -1,6 +1,6 @@
{
"sources": {
- "aisoc-technical-guide.md": "d5920e586e9a1538cb67b0e25532f79319a0fad668556a3dbcb03a1cad24d898",
+ "aisoc-technical-guide.md": "d0f93cba1d8acadcb9bd533c3e81a8ca6f8bdbb118586c8a741f2e2d674c1608",
"l0-l4-automation-maturity.md": "4da4b02d8e7b6d2839a593f09303191cd71c6b9ef28b34555a1a2086cf4c8781"
}
}
diff --git a/infra/helm/aisoc/Chart.yaml b/infra/helm/aisoc/Chart.yaml
index 0d85091b3..d710165df 100644
--- a/infra/helm/aisoc/Chart.yaml
+++ b/infra/helm/aisoc/Chart.yaml
@@ -25,13 +25,13 @@ type: application
# the chart refuses grants it used to accept (see the v14.0.0 BREAKING
# section), and the chart's major is the only coordinate a consumer pulling
# `oci://ghcr.io/beenuar/charts/aisoc` sees before they install it.
-version: 7.13.0
+version: 7.14.0
# Used as the image tag wherever a `tag:` in values.yaml is left empty, so
# it has to be a tag the registry actually holds — it read "5.2.0", which
# exists for no image, so every pod of a default install would have gone to
# ImagePullBackOff. scripts/check_published_images.py resolves it the same
# way the templates do and fails when it names nothing.
-appVersion: "v20.4.0"
+appVersion: "v21.0.0"
keywords:
- security