Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
Release v21.0.0 — the eval-honesty major (#1582)
Beenu Arora152c6d5ff37e
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 3f93888bb..0aa3470a9 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]+## [21.0.0] - 2026-10-31++### BREAKING++- **Correlation grouping narrows on missing identifiers** (the Changed+ entry below). On upgrade, *future* alerts that carry no tactic group+ per rule instead of per entity, and alerts with no identifiers at all+ stop cross-rule merging entirely. Existing incidents are untouched;+ the measured effect on a detection-shaped cohort is reduction moving+ 42.6% → 24.5% because unrelated alerts stop counting as grouped.+ There is no API change in this release; the major marks this grouping+ change plus the redefinition of published evaluation semantics+ (serving-model provenance, full-cohort denominators, closure+ contamination) that older scorecards do not carry.+### Added- **Serving-model attribution, end to end.** Every published verdict-evaldiff --git a/README.md b/README.mdindex 810995780..38b534ac7 100644--- a/README.md+++ b/README.md@@ -6,8 +6,8 @@**An open-source, self-hostable AI Security Operations Center.** It ingests your security telemetry, detects and correlates threats, investigates them with AI agents whose reasoning is fully auditable, and proposes responses a human approves.-[](https://opensource.org/licenses/MIT) [](CHANGELOG.md) [](https://github.com/beenuar/AiSOC/actions/workflows/ci.yml)-[](https://github.com/beenuar/AiSOC/actions/workflows/codeql.yml) [](https://securityscorecards.dev/viewer/?uri=github.com/beenuar/AiSOC) [](https://raw.githubusercontent.com/beenuar/AiSOC/main/apps/web/public/papers/aisoc-technical-guide.pdf)+[](https://opensource.org/licenses/MIT) [](CHANGELOG.md) [](https://github.com/beenuar/AiSOC/actions/workflows/ci.yml)+[](https://github.com/beenuar/AiSOC/actions/workflows/codeql.yml) [](https://securityscorecards.dev/viewer/?uri=github.com/beenuar/AiSOC) [](https://raw.githubusercontent.com/beenuar/AiSOC/main/apps/web/public/papers/aisoc-technical-guide.pdf)**[Technical Guide (PDF)](https://raw.githubusercontent.com/beenuar/AiSOC/main/apps/web/public/papers/aisoc-technical-guide.pdf)** · [Docs](https://beenuar.github.io/AiSOC/) · [Architecture](docs/architecture/README.md) · [What actually works](docs/audit/REPOSITORY_REALITY.md) · [Discussions](https://github.com/beenuar/AiSOC/discussions)@@ -159,17 +159,15 @@ exercised here** — there is no funded key, so per-model rows read *not measureVulnerabilities catalog — public, no API key — into the console's Threat Intelligence page: the onething in a fresh install that is neither synthetic nor yours.-## Real vs synthetic data+## Evaluation, reflex and replay-| Kind | Where | How you can tell |-|---|---|---|-| **Real** | Your connectors and the ingest API | `is_synthetic = false` (the default) |-| **Real, and not yours** | The CISA KEV feed on the Threat Intelligence page | Every row carries `source: cisa-kev`; it is the public catalog, unmodified |-| **Sample** | The wizard's **Load sample data** | Source column reads `AiSOC`; RFC 5737 / RFC 2606 reserved addresses only |-| **Demo** | `make demo` | `is_synthetic = true`, labelled in the console |-| **Benchmark / fixtures** | `services/agents/tests/eval_data/`, `**/tests/` | Published rows carry `substrate: true`; fixtures never ship in an image |+The published numbers carry their own provenance. ([methodology](https://beenuar.github.io/AiSOC/docs/benchmark) · [real vs synthetic data](apps/docs/docs/concepts/real-vs-synthetic.md))-**Production never silently falls back to synthetic data.** An unreachable backend makes the console name the failure rather than invent an investigation, and an unmeasured figure reads *not measured*, never `0`. It was not always so: [the reality audit](docs/audit/REPOSITORY_REALITY.md) has each case.+- **Every verdict scorecard names the model that actually served it** — gateway-attested or self-reported, with an explicit provenance status. A run with missing attribution, zero real calls or a deterministic fallback cannot be advertised as a named model's performance, and a measurement that cannot be bound to an attributed call cannot authorize closures.+- **Denominators are published, and the uncomfortable direction wins.** Full-cohort recall keeps abstentions and failures in the malicious denominator; closure claims ship with their contamination rate; alert reduction has three labelled figures (75.3% legacy grouping, 33.3% production key, 24.5% on a detection-shaped cohort after a missing identifier was fixed to *narrow* the group instead of widening it). Unmeasured reads *not measured*, never `0`.+- **The reflex path is observable** — nine named stages from queue delay to verification, and a decision contract carrying supporting evidence, contradictory evidence, missing prerequisites and the escalation reason.+- **Replay is time-frozen; re-hunting is labelled.** Historical replay enforces evidence cutoffs in the retrieval queries (never by prompt), discloses the model-knowledge limit, and cannot execute real actions — the mutation boundary is the write adapter. Applying new intelligence to old telemetry is a distinct, rehearsable mode, and its discoveries are never counted as predictions at the original event time.+- **Model suitability is per task.** The matrix grades each configured model on the same locked evidence against a deterministic baseline floor (balanced accuracy exactly 0.250); unfunded rows read *not measured*, with one-command reproduction against any local OpenAI-compatible endpoint.## AI agentsdiff --git a/RELEASES.md b/RELEASES.mdindex 4286dc872..1934b42d4 100644--- a/RELEASES.md+++ b/RELEASES.md@@ -2,7 +2,9 @@This file mirrors what used to live in the "What's new" section of [`README.md`](README.md). The complete, machine-readable inventory (with file paths, env-var diffs, and per-release test counts) lives in [`CHANGELOG.md`](CHANGELOG.md).-> **TL;DR for first-time visitors:** AiSOC is on `v20.4.0`, released 2026-10-30. **A fix release closing two defects a self-hoster hit the moment they turned SSO on.** Activating SSO OIDC locked the administrator out of their own tenant — 403 on users, investigations and the alert queue all at once (reported in [discussion #1568](https://github.com/beenuar/AiSOC/discussions/1568)). The cause: permissions fall back to the static role map only while a tenant holds zero role *grants*, and the bootstrap admin carries `users.role = 'admin'` with no grant row (it is created after the seed that would have backfilled one). That is harmless until the first SSO sign-in grants the provisioned user a role — which flips the whole tenant to database-backed RBAC, and from that instant the grant-less admin resolves to zero permissions. `bootstrap_admin` now grants the admin its catalog role on every startup, in an isolated session, a no-op when the catalog is unseeded, and **self-healing: a deployment already locked out recovers on the next API restart**. The same report surfaced a second bug: the alert funnel's "events of interest" count queried `aisoc.raw_events.ingested_at`, a column the lake does not have (it has `event_time` and `ingest_time`), so the query failed with ClickHouse `Code: 47` on every call and the console silently served a Postgres estimate; it now reads `ingest_time`, with a test that checks the query's columns against the committed ClickHouse DDL. Both fixes are verified against a real Postgres with the full migration chain — the lockout is reproduced in a test and then shown fixed, and the deprovisioning semantics the resolver relies on are untouched and still separately pinned. **2 defects were fixed in this release, 22 across it and the two before** (2 here, 10 in v20.3.0, 10 in v20.2.1 — counting distinct product or CI defects named in each release's Fixed and Security sections, dependency-version advisories excluded). No breaking changes; **neither `stable` channel moves**. The chart publishes as **`7.13.0`** — an `appVersion`-only move mirrors the app's minor, as `7.10.0`/`7.12.0` did before it. Everything below describes v20.3.0 and earlier, which this release does not change.+> **TL;DR for first-time visitors:** AiSOC is on `v21.0.0`, released 2026-10-31. **The eval-honesty major: the platform's published numbers now carry their own provenance, denominators and safety outcomes — and three communities' requests landed as shipped code in one wave.** Every verdict scorecard records which model *actually served* each call (gateway-verified where the gateway attests it, self-reported where only the response body says so) with an explicit provenance status — a run with missing attribution, zero real calls or a deterministic fallback can no longer be advertised as a named model's performance, and an autonomy measurement that cannot be bound to an attributed call can no longer authorize closures. The replay scorer publishes full-cohort recall (abstentions and failures stay in the malicious denominator), the four closure metrics including contamination, and reliability counts; the model matrix grades the verdict task per model with runtime, sampling and cost recorded, over a deterministic baseline floor (balanced accuracy exactly 0.250 — the number models are read against). Alert reduction now has three labelled figures instead of one flattering one: 75.3% (legacy grouping, not what production runs), 33.3% (the production key), and **24.5% on a detection-shaped cohort after the correlation fix** — a missing identifier now narrows the group instead of widening it (cross-rule merges on that cohort: 36 → 0), proven against live Redis including the creation race. The reflex path reports nine named stages and a decision contract (supporting *and* contradictory evidence, missing prerequisites, escalation reason); retro-hunt gained a rehearsal mode whose mutation boundary is enforced at the write adapter, never the prompt. The same wave shipped the **business-context plan** — tenant rules reach the triage model and impact statements must cite the rules that fired or be dropped and counted; asset criticality orders the queue, weighs fusion confidence (unknown renders *unknown*, never zero-as-evidence) and tighten-only raises approval impact on critical assets — plus the **Agentmetry ingest template** for AI coding-agent telemetry ([discussion #1577](https://github.com/beenuar/AiSOC/discussions/1577): their untested "generic-json should work" claim was verified half-true — events landed but every severity flattened to Unknown — and the bundled template fixes exactly that), and the externally contributed **ack-clock escalation bridge**. **This is the behavior change behind the major:** correlation grouping narrows on missing identifiers (future alerts only; existing incidents untouched), and the evaluation-semantics redefinition means pre-v21 scorecards are not comparable to v21 ones — there is no API break. **3 defects were fixed in this release, 15 across it and the two before** (3 here, 2 in v20.4.0, 10 in v20.3.0 — counting distinct product or CI defects named in each release's Fixed and Security sections, dependency-version advisories excluded). **Neither `stable` channel moves** — v21.0.0 is an application major, and both channel guards hold by design until a deliberate dispatch promotes them. The chart publishes as **`7.14.0`** — an `appVersion`-only move mirroring the app's step, exactly as `7.9.0` shipped the v20 major. Everything below describes v20.4.0 and earlier, which this release does not change.+>+> **Previously:** AiSOC was on `v20.4.0`, released 2026-10-30. **A fix release closing two defects a self-hoster hit the moment they turned SSO on.** Activating SSO OIDC locked the administrator out of their own tenant — 403 on users, investigations and the alert queue all at once (reported in [discussion #1568](https://github.com/beenuar/AiSOC/discussions/1568)). The cause: permissions fall back to the static role map only while a tenant holds zero role *grants*, and the bootstrap admin carries `users.role = 'admin'` with no grant row (it is created after the seed that would have backfilled one). That is harmless until the first SSO sign-in grants the provisioned user a role — which flips the whole tenant to database-backed RBAC, and from that instant the grant-less admin resolves to zero permissions. `bootstrap_admin` now grants the admin its catalog role on every startup, in an isolated session, a no-op when the catalog is unseeded, and **self-healing: a deployment already locked out recovers on the next API restart**. The same report surfaced a second bug: the alert funnel's "events of interest" count queried `aisoc.raw_events.ingested_at`, a column the lake does not have (it has `event_time` and `ingest_time`), so the query failed with ClickHouse `Code: 47` on every call and the console silently served a Postgres estimate; it now reads `ingest_time`, with a test that checks the query's columns against the committed ClickHouse DDL. Both fixes are verified against a real Postgres with the full migration chain — the lockout is reproduced in a test and then shown fixed, and the deprovisioning semantics the resolver relies on are untouched and still separately pinned. **2 defects were fixed in this release, 22 across it and the two before** (2 here, 10 in v20.3.0, 10 in v20.2.1 — counting distinct product or CI defects named in each release's Fixed and Security sections, dependency-version advisories excluded). No breaking changes; **neither `stable` channel moves**. The chart publishes as **`7.13.0`** — an `appVersion`-only move mirrors the app's minor, as `7.10.0`/`7.12.0` did before it. Everything below describes v20.3.0 and earlier, which this release does not change.>> **Previously:** AiSOC was on `v20.3.0`, released 2026-10-30. **The community-contributions release: six externally contributed fixes merged in one pass, the console's front door fixed, and the detection corpus made more honest again.** A clean self-hosted install's `/` rendered the project's *marketing page* — neither the dashboard nor the login page an operator expects (reported in discussion #1549); it now redirects to `/login`, which routes onward by itself (first run → setup wizard, signed in → dashboard), and a deployment that *is* the public site opts back in with the runtime `AISOC_MARKETING_HOME`. The `custom` LLM provider's model id no longer has to be typed by hand: **`POST /api/v1/llm/credentials/models` asks any OpenAI-compatible endpoint what chat models it serves** (externally contributed — the exact vendor-neutral gap the #1481 review offered), bounded, SSRF/air-gap-guarded, and the stored key goes only to the stored base URL. An external report that **1 of 12 GitHub detection rules fired on real connector output was substantially correct**: five rules re-claused to GitHub's documented audit vocabulary (the deploy-key rule matched an action that does not exist — deploy keys log as `public_key.create`), five quarantined with recorded reasons, one claim refuted with measurement, a CI gate now replays every GitHub rule through the real connector (11 findings on the pre-fix tree), and the executable count honestly moves **2,535 → 2,530**. Community fixes from [alexmateescu](https://github.com/alexmateescu) close a retry storm (deterministic gateway 400s now dead-letter on first sight; prompt bytes capped at the egress chokepoint with water-filling truncation), a saved connector 422-ing every test-connection once it had polled, fresh chatops deploys crash-looping on a missing sibling mount, playbook agent-events dialling a port realtime never served, and triage verdicts clipped mid-JSON at 512 output tokens (now 2,048, with the reliability harness moving in lockstep — its parity test going red on the one-line bump is the drift guard working). Three dependency advisories cleared (`@hono/node-server` 2.1.5, `decode-uri-component` 0.5.1, archived-SDK `zod` 4.6.6). **10 defects were fixed in this release, 22 across it and the two before** (10 here, 1 in v20.2.1, 11 in v20.2.0 — counting distinct product or CI defects named in each release's Fixed and Security sections, dependency-version advisories excluded). No breaking changes; **neither `stable` channel moves**. The chart publishes as **`7.12.0`** — nothing inside the chart directory moves except `appVersion`, and the chart's step mirrors the app's, exactly as `7.10.0` shipped v20.1.0's minor on an `appVersion`-only change (a chart *patch* is reserved for shipping an app patch, as `7.11.1` did for v20.2.1). Everything below describes v20.2.1 and earlier, which this release does not change.>@@ -32,7 +34,46 @@ This file mirrors what used to live in the "What's new" section of [`README.md`]## What's new-`VERSION` is `20.4.0`. The **v20.4.0** release (2026-10-30) closes two+`VERSION` is `21.0.0`. The **v21.0.0** release (2026-10-31) is the+eval-honesty major: the platform's published numbers now carry their own+provenance, denominators and safety outcomes. Serving-model attribution flows+from the gateway header (verified) or the response body (self-reported,+credited only when it can witness itself) into every scorecard, ledger+exchange and eligibility decision — a run with missing attribution, zero real+calls or a deterministic fallback cannot be advertised as a named model's+performance, and an autonomy measurement that cannot be bound to an+attributed call cannot authorize closures. The replay scorer publishes+full-cohort recall (abstentions and failures stay in the malicious+denominator), the four closure metrics including contamination, reliability+counts (requested / attempted / completed / failed / unattempted) and+per-severity segments. The correlation engine's **behavior change** is the+reason this is a major: a missing identifier now narrows the group instead of+widening it — alerts with no tactic group per rule, alerts with no+identifiers never cross-rule merge — measured on a detection-shaped cohort as+reduction falling 42.6% → 24.5% while cross-rule merges fell 36 → 0, because+unrelated alerts stopped counting as grouped; the production-key headline+stays 33.3%, and a live-Redis suite (expiry, tenant boundary, restart, the+creation race fixed with `SET NX`) now runs in CI. The reflex path reports+nine named stages and a structured decision contract; retro-hunt gained a+rehearsal mode (`RETRO_HUNT_DRY_RUN`) whose mutation boundary is enforced at+the write adapter; verdict scorecards disclose the model-knowledge limit on+historical claims. The same wave landed the business-context plan (tenant+rules reach the triage model with cited impact statements; asset criticality+orders the queue, weighs confidence and tighten-only raises approval impact),+the Agentmetry ingest template for AI coding-agent telemetry (discussion+#1577 — their untested generic-json claim verified half-true and fixed by the+bundled template), the externally contributed ack-clock escalation bridge,+an annotated unsupported-claim measurement that found and fixed a real+claim-support defect, and the verdict model matrix with a deterministic+baseline floor (balanced accuracy exactly 0.250) against which every model+row is read — rows without a funded run say *not measured* with one-command+reproduction. No API breaks. Neither `stable` channel moves. The chart+publishes as **`7.14.0`**. Everything below describes v20.4.0 and earlier,+which this release does not change.++---++`VERSION` was `20.4.0`. The **v20.4.0** release (2026-10-30) closes twodefects a self-hoster hit when they enabled SSO (discussion #1568).Activating SSO OIDC locked the administrator out — 403 on users,investigations and the alert queue — because permissions fall back todiff --git a/VERSION b/VERSIONindex bd45cd74a..fb5b51303 100644--- a/VERSION+++ b/VERSION@@ -1 +1 @@-20.4.0+21.0.0diff --git a/apps/docs/docs/concepts/real-vs-synthetic.md b/apps/docs/docs/concepts/real-vs-synthetic.mdnew file mode 100644index 000000000..05eb4d574--- /dev/null+++ b/apps/docs/docs/concepts/real-vs-synthetic.md@@ -0,0 +1,31 @@+---+id: real-vs-synthetic+title: Real vs synthetic data+sidebar_label: Real vs synthetic data+---++# Real vs synthetic data++Every row in a running AiSOC is one of five kinds, and each kind is+distinguishable from the console and the API — this table moved here from the+README front page, unchanged in substance, when the release that redefined the+published evaluation semantics needed that space.++| Kind | Where | How you can tell |+|---|---|---|+| **Real** | Your connectors and the ingest API | `is_synthetic = false` (the default) |+| **Real, and not yours** | The CISA KEV feed on the Threat Intelligence page | Every row carries `source: cisa-kev`; it is the public catalog, unmodified |+| **Sample** | The wizard's **Load sample data** | Source column reads `AiSOC`; RFC 5737 / RFC 2606 reserved addresses only |+| **Demo** | `make demo` | `is_synthetic = true`, labelled in the console |+| **Benchmark / fixtures** | `services/agents/tests/eval_data/`, `**/tests/` | Published rows carry `substrate: true`; fixtures never ship in an image |++**Production never silently falls back to synthetic data.** An unreachable+backend makes the console name the failure rather than invent an+investigation, and an unmeasured figure reads *not measured*, never `0`. It+was not always so: [the reality audit](https://github.com/beenuar/AiSOC/blob/main/docs/audit/REPOSITORY_REALITY.md)+has each case.++The same discipline governs the published numbers themselves — which model+actually served an evaluation, what sits in each metric's denominator, and+what a replay was allowed to know — described on the+[benchmark methodology page](../benchmark.md).diff --git a/apps/docs/sidebars.ts b/apps/docs/sidebars.tsindex 518cc4ae8..d7d95a9da 100644--- a/apps/docs/sidebars.ts+++ b/apps/docs/sidebars.ts@@ -30,6 +30,7 @@ const sidebars: SidebarsConfig = {"concepts/ai-estate","concepts/ai-estate-discovery","concepts/groundedness",+ "concepts/real-vs-synthetic","concepts/live-actions","concepts/retro-hunts","concepts/hunting-agent",diff --git a/apps/docs/static/data/scoreboard.json b/apps/docs/static/data/scoreboard.jsonindex f2799ae95..b0e2088cc 100644--- a/apps/docs/static/data/scoreboard.json+++ b/apps/docs/static/data/scoreboard.json@@ -3,9 +3,9 @@"_readme": "Append-only public benchmark scoreboard. Each row is one eval run. Substrate rows (no LLM, deterministic) are marked substrate:true and MUST NOT be quoted as live agent performance. Wet-eval rows (substrate:false) are appended by the T5.5 weekly CI workflow once it lands. Newest row first.","rows": [{- "date": "2026-10-30",- "agent_version": "v20.4.0",- "commit_sha": "013ecd38",+ "date": "2026-10-31",+ "agent_version": "v21.0.0",+ "commit_sha": "f840cbe2","substrate": true,"eval_mode": "substrate-only","mitre_accuracy": 0.97,diff --git a/apps/docs/static/llms-full.txt b/apps/docs/static/llms-full.txtindex 6f1505a23..5cd15a65a 100644--- a/apps/docs/static/llms-full.txt+++ b/apps/docs/static/llms-full.txt@@ -3,7 +3,7 @@<!-- Generated by scripts/generate_llms_txt.py. Do not edit by hand; run the script and commit the result. -->> AiSOC is an open-source, self-hostable AI Security Operations Center, MIT-> licensed and maintained by the AiSOC community. Version 20.4.0.+> licensed and maintained by the AiSOC community. Version 21.0.0.> Repository: https://github.com/beenuar/AiSOC. Documentation: https://beenuar.github.io/AiSOC/.This file exists so an automated reader can answer questions about AiSOC@@ -127,7 +127,7 @@ plans/ Historical prototype subtree — reference only, never editedEvery number in this file is generated from the artefact that owns it. If youneed to state one, re-derive it rather than quoting a figure from prose —-including from this file, which is a snapshot of version 20.4.0.+including from this file, which is a snapshot of version 21.0.0.```python3 scripts/generate_corpus_stats.py --check # executable + on-disk rulesdiff --git a/apps/docs/static/llms.txt b/apps/docs/static/llms.txtindex 952277fba..8ff41f3af 100644--- a/apps/docs/static/llms.txt+++ b/apps/docs/static/llms.txt@@ -6,7 +6,7 @@> security telemetry from 87 first-party connectors, normalises it to OCSF,> correlates it into alerts, and runs LLM agents that triage and investigate them.> Every agent decision is written to an investigation ledger, so the reasoning-> behind a verdict can be replayed step by step. MIT licensed. Current version 20.4.0.+> behind a verdict can be replayed step by step. MIT licensed. Current version 21.0.0.AiSOC is built and maintained by the AiSOC community. It is a complete SOCplatform rather than a single tool: ingestion, detection, correlation, agentdiff --git a/apps/web/content/papers/aisoc-technical-guide.md b/apps/web/content/papers/aisoc-technical-guide.mdindex da15a3c41..3b56f8b78 100644--- a/apps/web/content/papers/aisoc-technical-guide.md+++ b/apps/web/content/papers/aisoc-technical-guide.md@@ -2,8 +2,8 @@title: "AiSOC Technical Guide"subtitle: "Install, operate and integrate — on a workstation and on a server"author: "AiSOC project"-version: "v1.1"-date: "2026-10-28"+version: "v1.2"+date: "2026-10-31"license: "MIT (same as the AiSOC project)"abstract: |A complete operator's guide to AiSOC: installing it on a laptop and on a@@ -778,7 +778,62 @@ the restore, and failures are counted and surfaced.----## 10. Honest limits+## 10. Evaluation, replay and model suitability++Added in v21.0.0, this is how the platform keeps its own numbers honest. The+full methodology lives on the benchmark page; what an operator needs is below.++**Serving-model attribution.** Every verdict scorecard, ledger exchange and+eligibility decision records which model *actually served* each call, with a+provenance status: `gateway_verified` when the LiteLLM gateway attests the+serving model in a response header, `self_reported` when only the response+body names it (credited only when the body can witness itself — a name merely+equal to the requested alias is never copied in), and `unattributed`+otherwise. A run that placed zero real calls, fell back to the deterministic+path, or cannot name its serving model is preserved as a diagnostic and+refused as "verified performance of model X". The same status gates autonomy:+a clearing measurement that cannot be bound to an attributed call cannot+authorize automatic closures.++**Metrics with denominators.** The replay scorer publishes precision, recall,+per-class F1, confusion matrices, balanced accuracy and MCC with+numerator/denominator counts; full-cohort malicious recall keeps abstentions+and runtime failures in the denominator; closure candidates carry four+honest cuts (malicious closure rate, closure contamination, benign closure+coverage, unknown-label share), always over *proposed* closures — a simulated+closure never appears as a production outcome. Requested, attempted,+completed, failed and unattempted counts travel with every run, so a failure+that stops a run cannot vanish from the headline.++**Alert reduction, three figures.** 75.3% is the legacy four-tier grouping+benchmark (not what production runs); 33.3% is the production correlation+key on the tactic-carrying cohort; 24.5% is a detection-shaped cohort after+v21.0.0's grouping change, where a missing identifier narrows the group+instead of widening it. The drop is the honest direction: the difference was+unrelated alerts counted as grouped.++**Replay and re-hunting.** Historical replay freezes evidence at each+decision time, enforces the cutoff in retrieval and storage queries rather+than in the prompt, keeps truth labels reachable by the evaluator and+unreachable by the agent, and discloses that pretrained weights may postdate+the events (the scorecard carries the model-knowledge limit, or "unknown").+Replay cannot execute real actions — the mutation boundary is the write+adapter, not an instruction. Re-hunting old telemetry with newly published+intelligence is a separate, labelled mode with a rehearsal switch+(`RETRO_HUNT_DRY_RUN=1`) that records proposed reopens without mutating+anything; its discoveries are never counted as predictions at the original+event time.++**Model suitability is per task.** `scripts/run_model_matrix.py --task+verdict` grades each configured model on the same locked corpus and records+serving identity, sampling, runtime and token use per row. Every row is read+against a deterministic baseline floor (balanced accuracy exactly 0.250 on+the published corpus — what answering without a model scores). A model or+configuration with no funded run is listed as *not measured*; reproduction is+one command against any OpenAI-compatible endpoint, including a local+Ollama.++## 11. Honest limitsStated plainly, because a guide that only lists strengths is a brochure.@@ -787,9 +842,11 @@ Stated plainly, because a guide that only lists strengths is a brochure.- **Long-duration scale figures do not exist.** The harness has ramp, burst,backpressure and 24/72-hour soak profiles; the long ones have not been run,and no throughput number is claimed from them.-- **Package distribution is blocked on credentials.** The release workflow- builds and packs all eight npm and PyPI packages on every tag; the upload- step is credential-gated. Install from source.+- **Package distribution is half-shipped.** The five PyPI packages are live+ and resolve at the tree's versions (`pip install aisoc-sandbox` works+ today); the three npm packages still await the first-upload `NPM_TOKEN`,+ an account action. The release workflow builds and packs all eight on+ every tag.- **Benchmark numbers are labelled.** Three of the four eval metrics aresubstrate self-consistency gates, not agent accuracy. Only MITRE accuracymeasures the live agent, and the benchmark page says which is which.@@ -798,7 +855,7 @@ Stated plainly, because a guide that only lists strengths is a brochure.----## 11. Where to go next+## 12. Where to go next| You want | Read ||---|---|diff --git a/apps/web/package.json b/apps/web/package.jsonindex fe850a704..7899f1e35 100644--- a/apps/web/package.json+++ b/apps/web/package.json@@ -1,6 +1,6 @@{"name": "@aisoc/web",- "version": "20.4.0",+ "version": "21.0.0","private": true,"description": "AiSOC SOC Console - Next.js 14 Frontend","license": "MIT",diff --git a/apps/web/public/papers/aisoc-technical-guide.pdf b/apps/web/public/papers/aisoc-technical-guide.pdfindex 357b046fb..2574c4a57 100644Binary files a/apps/web/public/papers/aisoc-technical-guide.pdf and b/apps/web/public/papers/aisoc-technical-guide.pdf differdiff --git a/apps/web/public/papers/render-manifest.json b/apps/web/public/papers/render-manifest.jsonindex f6d26dca6..545204ba4 100644--- a/apps/web/public/papers/render-manifest.json+++ b/apps/web/public/papers/render-manifest.json@@ -1,6 +1,6 @@{"sources": {- "aisoc-technical-guide.md": "d5920e586e9a1538cb67b0e25532f79319a0fad668556a3dbcb03a1cad24d898",+ "aisoc-technical-guide.md": "d0f93cba1d8acadcb9bd533c3e81a8ca6f8bdbb118586c8a741f2e2d674c1608","l0-l4-automation-maturity.md": "4da4b02d8e7b6d2839a593f09303191cd71c6b9ef28b34555a1a2086cf4c8781"}}diff --git a/infra/helm/aisoc/Chart.yaml b/infra/helm/aisoc/Chart.yamlindex 0d85091b3..d710165df 100644--- a/infra/helm/aisoc/Chart.yaml+++ b/infra/helm/aisoc/Chart.yaml@@ -25,13 +25,13 @@ type: application# the chart refuses grants it used to accept (see the v14.0.0 BREAKING# section), and the chart's major is the only coordinate a consumer pulling# `oci://ghcr.io/beenuar/charts/aisoc` sees before they install it.-version: 7.13.0+version: 7.14.0# Used as the image tag wherever a `tag:` in values.yaml is left empty, so# it has to be a tag the registry actually holds — it read "5.2.0", which# exists for no image, so every pod of a default install would have gone to# ImagePullBackOff. scripts/check_published_images.py resolves it the same# way the templates do and fails when it names nothing.-appVersion: "v20.4.0"+appVersion: "v21.0.0"keywords:- security