Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

feat(evaluation): replay a customer's closed findings through production triage (#904)

Beenu Arora68cf12a6bce5
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 49d048e17..f72bacf22 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -544,6 +544,16 @@ jobs:
run: python3 scripts/check_cost_provenance.py --self-test
- name: Cost provenance gate
run: python3 scripts/check_cost_provenance.py
+ # Replay evaluation spans three trees that cannot import one another:
+ # the reader in services/actions, the runner in services/agents and the
+ # grader in packages/aisoc-benchmark. The contract between them is three
+ # declarations agreeing by convention, and this compares them in both
+ # directions so a field added on one side cannot be dropped silently at
+ # the boundary.
+ - name: Replay contract parity gate — self-test
+ run: python3 scripts/check_replay_contract_parity.py --self-test
+ - name: Replay contract parity gate
+ run: python3 scripts/check_replay_contract_parity.py
# The count claim above says how many connectors exist; this says
# whether a reader can find out how to use them. 35 of 84 had no page
# at all and the sidebar listed 34 of 96, so most connector
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b5b94b19b..a4f389716 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,60 +9,63 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
-<<<<<<< HEAD
-- **Closed-finding history readers: the analyst labels a replay evaluation
- grades against** (gap-closure Phase 1.1). AiSOC could already push a verdict
- into somebody else's SIEM. It could not read back the findings a customer's
- own analysts had already closed, so there was no way for an operator to
- measure triage against their own decisions on their own data before being
- asked to trust it. The published benchmark is substrate self-consistency on
- a synthetic corpus, which answers a different question.
-
- Five readers now list closed findings in a time window, each row carrying
- the analyst's disposition, their reason, who closed it and when:
- `SplunkClient.list_closed_notables`, `SentinelClient.list_closed_incidents`,
- `ElasticClient.list_closed_signals`, `QRadarClient.list_closed_offenses` and
- `DefenderClient.list_resolved_alerts`. They extend the clients in
- `services/actions`, which already own the credential path and already hold
- the writeback going the other way.
-
- **The rule that carries the value is the refusal to guess.** Vendor labels
- map to the canonical taxonomy in one place,
- `app/services/alert_history.py`, rather than five places that could disagree
- about what "benign positive" means. Anything outside that taxonomy becomes
- `unlabeled` and is excluded from accuracy. That is not an edge case: Splunk
- ES ships dispositions literally named "Other" and "Undetermined", and
- Sentinel and Defender both ship an explicit `Undetermined` or `Unknown`
- classification. An analyst who picked one of those said they did not know,
- and folding it into `true_positive` because the finding happened to be
- closed would manufacture agreement out of an admission of uncertainty.
+- **Replay evaluation: the production triage path, run over a customer's own
+ closed findings, writing nothing** (gap-closure Phase 1.2 and 1.3). Phase
+ 1.1 made an operator's analyst labels readable. Nothing could grade against
+ them, and the obvious way to build that grading is the wrong one: a second
+ triage implementation would measure the evaluation harness rather than the
+ product, and would diverge from it the first time anybody changed a prompt.
+
+ **The gap, and how it was closed.** `FusedAlertTriageWorker.triage` could
+ not run without writing. It recorded the verdict to the Investigation
+ Ledger and the `alerts` row, wrote a per-signature outcome prior, queued
+ approvals, cached the verdict for deduplication and pushed the disposition
+ back to the source SIEM, all inline. Persistence is now injected through
+ `app/workers/triage_persistence.py`: `LiveTriageWriter` is the default and
+ does exactly what the worker used to do, and `ShadowTriageWriter` counts
+ each call and performs none of them. There is no `if replay` on the hot
+ path, so there is nothing to drift. `CostTracker` grew a `persist` flag for
+ the same reason: a replay must measure spend and must not bill it.
+
+ **Point-in-time context, because the test window must not answer itself.**
+ Production writes every verdict back as an outcome prior and reads that
+ prior *before* triage, so in a naive replay the verdict on one finding
+ auto-closes the next finding with the same evidence and the evaluation
+ grades an answer it supplied itself. The deduplication cache does the same
+ thing one layer earlier and leaves no trace in either store. Organisation
+ memory and outcome priors are now captured once at the 70/30 time split and
+ served from that snapshot for the whole run, and rows recorded after the
+ split are dropped from it.
+
+ **Scoring extends `packages/aisoc-benchmark` rather than standing beside
+ it.** The hallucination machinery is the existing
+ `_INDICATOR_PATTERNS`, so an indicator counted here is one the synthetic
+ corpus grader would also count. New: per-class precision and recall with
+ malicious recall first, a confusion matrix, abstention rate, reliability
+ bins with an expected calibration error, per-rule and per-source
+ breakdowns, and seeded bootstrap confidence intervals. Below 30 malicious
+ cases the report prints the count and refuses a headline accuracy, because
+ on a queue that is almost entirely false positives an agent that calls
+ everything benign scores well and the figure would describe the queue. A
+ rate with no denominator reads "not measured", never 0.
+
+ **Measured:** the `services/agents` suite goes from 1252 to 1280 passing,
+ `packages/aisoc-benchmark` from 35 to 57, and `services/connectors` from
+ 880 to 886. Two runs of a 20-finding replay on the deterministic tier
+ produce identical decisions on every field but wall-clock latency, and two
+ renderings of one score are byte-identical.
+
+ **Gates:** `ci.yml` agents job (`test_replay_shadow_no_writes.py`,
+ `test_replay_leakage.py`), `ci.yml` benchmark job
+ (`test_replay_metrics.py`), and a new
+ `scripts/check_replay_contract_parity.py` wired into `ci.yml`, which
+ compares the finding shape and the disposition taxonomy across the three
+ trees that hold them and cannot import one another, in both directions. The
+ leakage tests each carry a sensitivity half that runs the unprotected
+ configuration and asserts it leaks, so the file cannot go quietly green
+ over a property it has stopped testing. Three claim-to-gate rows; the
+ matrix goes 148 rows to 151, GATED 140 to 143, NO GATE stays 0.
- Two vendor-specific decisions are worth stating rather than burying.
- Elastic Security ships no disposition field at all: closing a signal records
- no reason, so the honest reader returns `unlabeled` unless the deployment
- has adopted a workflow tag, and the convention it reads is documented rather
- than assumed. QRadar's "Non-Issue" maps to `benign` and not
- `benign_true_positive`, because it makes no claim about whether the rule was
- right, which is the distinction `benign` exists to carry.
-
- **Measured:** 30 new tests drive each reader's real HTTP path against
- vendor-shaped payloads with a mock transport, so a wrong endpoint, a wrong
- filter or a parser that fails on the documented response shape is caught
- here rather than on a customer's history. The `services/actions` suite goes
- from 737 to 767 passing.
-
- **Gate:** `ci.yml` actions job (`test_alert_history.py`), with a new
- claim-to-gate row. The assertions that hold the claim are the negative ones:
- every vendor's explicit "I do not know" is asserted to land on `unlabeled`,
- two conflicting Elastic tags yield `unlabeled` rather than a coin flip, and
- `unlabeled` is asserted not to be a member of `CANONICAL_DISPOSITIONS` so
- nothing downstream can score it as a verdict. `ClosedFinding.__post_init__`
- refuses a disposition that is neither canonical nor `unlabeled`, so a mapper
- cannot introduce a third vocabulary. An unparseable close time raises rather
- than defaulting to now, because a silently wrong close time would put a row
- on the wrong side of the train/test split and leak the answer into its own
- evaluation.
-=======
- **File and URL analysis behind one provider contract, with uploading a
customer file off by default.** AiSOC could enrich an IP, a domain and a URL
and had nothing at all for a file hash: `services/enrichment` fans out to
@@ -149,7 +152,59 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
No file was uploaded to any third party while building this. The recorded
fixture is a GET by hash for the published EICAR test file; every
submit and poll fixture is hand-written and labelled as such.
->>>>>>> c51f197b (feat(sandbox): file and URL analysis behind one provider contract)
+
+- **Closed-finding history readers: the analyst labels a replay evaluation
+ grades against** (gap-closure Phase 1.1). AiSOC could already push a verdict
+ into somebody else's SIEM. It could not read back the findings a customer's
+ own analysts had already closed, so there was no way for an operator to
+ measure triage against their own decisions on their own data before being
+ asked to trust it. The published benchmark is substrate self-consistency on
+ a synthetic corpus, which answers a different question.
+
+ Five readers now list closed findings in a time window, each row carrying
+ the analyst's disposition, their reason, who closed it and when:
+ `SplunkClient.list_closed_notables`, `SentinelClient.list_closed_incidents`,
+ `ElasticClient.list_closed_signals`, `QRadarClient.list_closed_offenses` and
+ `DefenderClient.list_resolved_alerts`. They extend the clients in
+ `services/actions`, which already own the credential path and already hold
+ the writeback going the other way.
+
+ **The rule that carries the value is the refusal to guess.** Vendor labels
+ map to the canonical taxonomy in one place,
+ `app/services/alert_history.py`, rather than five places that could disagree
+ about what "benign positive" means. Anything outside that taxonomy becomes
+ `unlabeled` and is excluded from accuracy. That is not an edge case: Splunk
+ ES ships dispositions literally named "Other" and "Undetermined", and
+ Sentinel and Defender both ship an explicit `Undetermined` or `Unknown`
+ classification. An analyst who picked one of those said they did not know,
+ and folding it into `true_positive` because the finding happened to be
+ closed would manufacture agreement out of an admission of uncertainty.
+
+ Two vendor-specific decisions are worth stating rather than burying.
+ Elastic Security ships no disposition field at all: closing a signal records
+ no reason, so the honest reader returns `unlabeled` unless the deployment
+ has adopted a workflow tag, and the convention it reads is documented rather
+ than assumed. QRadar's "Non-Issue" maps to `benign` and not
+ `benign_true_positive`, because it makes no claim about whether the rule was
+ right, which is the distinction `benign` exists to carry.
+
+ **Measured:** 30 new tests drive each reader's real HTTP path against
+ vendor-shaped payloads with a mock transport, so a wrong endpoint, a wrong
+ filter or a parser that fails on the documented response shape is caught
+ here rather than on a customer's history. The `services/actions` suite goes
+ from 737 to 767 passing.
+
+ **Gate:** `ci.yml` actions job (`test_alert_history.py`), with a new
+ claim-to-gate row. The assertions that hold the claim are the negative ones:
+ every vendor's explicit "I do not know" is asserted to land on `unlabeled`,
+ two conflicting Elastic tags yield `unlabeled` rather than a coin flip, and
+ `unlabeled` is asserted not to be a member of `CANONICAL_DISPOSITIONS` so
+ nothing downstream can score it as a verdict. `ClosedFinding.__post_init__`
+ refuses a disposition that is neither canonical nor `unlabeled`, so a mapper
+ cannot introduce a third vocabulary. An unparseable close time raises rather
+ than defaulting to now, because a silently wrong close time would put a row
+ on the wrong side of the train/test split and leak the answer into its own
+ evaluation.
## [11.2.0] — 2026-09-26
diff --git a/GAP_CLOSURE_PROGRESS.md b/GAP_CLOSURE_PROGRESS.md
index 262953c51..35c34826e 100644
--- a/GAP_CLOSURE_PROGRESS.md
+++ b/GAP_CLOSURE_PROGRESS.md
@@ -143,15 +143,60 @@ replay runners and evaluation scoring. None of the five clients
Phase 1.1 is genuinely new work, and the writeback direction it mirrors
(`disposition_writeback.py`) already exists and supplies the taxonomy.
+### D6. No two services can be imported into one process, so the connector `normalize()` is reached over HTTP
+
+The plan says "normalize each finding with the same connector `normalize()`
+production uses", and it is right to. Grading the agent on an input shape the
+product never produces measures a pipeline nobody runs.
+
+It cannot be done by importing. `services/agents`, `services/connectors` and
+`services/actions` all package their code as top-level `app`, so one Python
+process can hold exactly one of them, and no path manipulation changes that.
+The two remaining options were a copy of every vendor's field mapping inside
+the agents service, kept in step by discipline, or a round trip to the service
+that owns the mapping.
+
+**Resolution:** a round trip, through a new `POST /connectors/{id}/normalize`.
+This is the same reasoning that already put organisation memory and SIEM
+writeback behind HTTP calls to the API service rather than behind a second copy
+of their SQL. The handler builds the connector with `__new__` and never runs
+`__init__`, so the route holds no credential and cannot call a customer's SIEM;
+a connector whose `normalize` does reach for instance state gets a 422 naming
+itself rather than returning a half-mapped envelope. Eleven of the 84
+registered connectors take that path today, and a parametrised test asserts
+every connector either normalizes or says why.
+
+The runner keeps a `FindingNormalizer` port with **no fallback**. A replay that
+cannot reach the production mapping raises `NormalizerUnavailable` and names
+what is missing, because a "close enough" mapping written in the agents service
+would be invisible in the report: its output has the same shape as the real
+thing.
+
+### D7. What a replay measures is triage, not the pipeline
+
+The plan's chain is connector `normalize()` then triage. Production's chain has
+`services/ingest` and `services/fusion` in between, and fusion is where an
+alert gains correlation across related events, its fused confidence score, the
+deterministic narrative and entity resolution.
+
+A replayed finding carries none of those. Reproducing them in the agents
+service would be the reimplementation the plan forbids, one layer down.
+
+**Resolution:** the gap is published rather than closed. `ENVELOPE_LIMITS` in
+`app/replay/normalize.py` lists the four missing enrichments and travels in
+every report's method section, and `confidence_score` is deliberately left
+unset rather than invented, since a fabricated fusion confidence would enter
+the prompt that decides the verdict being graded.
+
---
## Phase 1: Replay evaluation on a customer's own history
- [x] **1.1 History readers.** Shipped in [#903](https://github.com/beenuar/AiSOC/pull/903). Five readers on the clients in `services/actions`, which already own the credential path and already hold the writeback going the other way: `SplunkClient.list_closed_notables`, `SentinelClient.list_closed_incidents`, `ElasticClient.list_closed_signals`, `QRadarClient.list_closed_offenses`, `DefenderClient.list_resolved_alerts`. One taxonomy module (`app/services/alert_history.py`) rather than five that could disagree. 30 tests drive each reader's real HTTP path against vendor-shaped payloads; the `services/actions` suite goes 737 to 767. Claim-to-gate row added, matrix 147 rows to 148, GATED 139 to 140. Two vendor decisions recorded in `apps/docs/docs/evaluation/replay.md`: Elastic ships no disposition field so an untagged deployment yields no labels, and QRadar "Non-Issue" is `benign` not `benign_true_positive` because it makes no claim about whether the rule was right.
-- [ ] **1.2 Replay runner.** New module in `services/agents`. Production `normalize()`, time ordering, 70/30 time split, the production triage path in shadow mode with persistence injected, memory and context frozen at the split point, and verdict, confidence, evidence, tool calls, model id, tokens, measured cost and latency recorded.
-- [ ] **1.3 Scoring.** Extend `packages/aisoc-benchmark`: per-class precision and recall with malicious recall first, confusion matrix, abstention rate, calibration with expected calibration error, hallucination rate, per-rule and per-source breakdowns, bootstrap confidence intervals, and no headline accuracy below 30 malicious cases.
-- [ ] **1.4 Surfaces.** CLI `aisoc replay`, async API job with tenant-scoped tables, the "Evaluate on your history" console page, and JSON, Markdown and PDF export.
-- [ ] **1.5 Gates and docs.** Recorded vendor payload tests per reader, the leakage test, claim-to-gate rows, and `apps/docs/docs/evaluation/replay.md` stating that data leaves the deployment only when the configured model is hosted.
+- [x] **1.2 Replay runner.** Shipped in [#904](https://github.com/beenuar/AiSOC/pull/904). `services/agents/app/replay/` holds the split, the shadow sinks and the runner; it holds no triage. Persistence is injected through `app/workers/triage_persistence.py`, whose default is `LiveTriageWriter` doing exactly what the worker did inline, so the measured path is the production one rather than a copy. `CostTracker` gained a `persist` flag so a replay measures spend without billing it. Normalisation reaches the real connector through a new `POST /connectors/{id}/normalize`, because both services package their code as top-level `app` and one process can hold one of them. Verdict, confidence, evidence, tool calls, model id, tokens, measured cost and latency are all recorded per decision. See D6 and D7 below for the two places the plan and the tree disagreed.
+- [x] **1.3 Scoring.** Shipped in [#904](https://github.com/beenuar/AiSOC/pull/904). `packages/aisoc-benchmark/aisoc_benchmark/replay.py` reuses the existing `_INDICATOR_PATTERNS` for hallucination so there is one definition, and adds per-class precision and recall with malicious recall first, a confusion matrix, abstention rate, reliability bins with an expected calibration error, per-rule and per-source breakdowns, and seeded bootstrap intervals. Below 30 malicious cases the headline accuracy is withheld with the count and the reason. A rate with no denominator reads "not measured".
+- [ ] **1.4 Surfaces.** CLI `aisoc replay`, async API job with tenant-scoped tables, the "Evaluate on your history" console page, and JSON, Markdown and PDF export. Markdown rendering already ships with 1.3 (`format_replay_report`).
+- [~] **1.5 Gates and docs.** Recorded vendor payload tests per reader shipped with 1.1. The leakage test shipped in [#904](https://github.com/beenuar/AiSOC/pull/904) (`services/agents/tests/test_replay_leakage.py`), covering all three stores a test-window decision can travel back through, each with a sensitivity half that runs the unprotected configuration and asserts it leaks. Three claim-to-gate rows added, matrix 148 rows to 151, GATED 140 to 143. `apps/docs/docs/evaluation/replay.md` covers the method, the limits and the privacy position. What remains is the documentation of the 1.4 surfaces once they exist.
**Done when:** the CLI, run against a mocked Splunk ES holding 200 recorded
closed notables, produces a report that reproduces byte for byte on a second run
@@ -283,6 +328,10 @@ Nothing yet beyond this kickoff. Each entry below will name its PR.
baseline above captured and recorded.
- [x] **Phase 1.1, history readers.**
[#903](https://github.com/beenuar/AiSOC/pull/903).
+- [x] **Phase 1.2 replay runner and 1.3 scoring.**
+ [#904](https://github.com/beenuar/AiSOC/pull/904). Suites: `services/agents`
+ 1252 to 1280, `packages/aisoc-benchmark` 35 to 57, `services/connectors` 880
+ to 886. Every other suite unchanged and passing.
### Notes for the next session
@@ -296,6 +345,17 @@ made a `ruff format --check` comparison against `main` look like `main` had two
unformatted files: `git stash` leaves untracked files in place, so the "base"
run was still seeing the new ones.
+**A test that reconfigures a gate before pointing it at production is not
+testing production.** `tests/test_readme_figures_gate.py::test_live_repo_is_consistent`
+passed against this tree while `scripts/readme_gates.py --skip-sandbox` failed
+against the same commit in CI. Its helper narrows `FIGURE_DOCS` to the one
+scratch path its fixtures write, and passing the real repository root through
+that helper left the narrowing in place, so the "real tree" check read the
+compliance page and neither `ROADMAP.md` nor `RELEASES.md`. Both had gone stale
+and the test had never been able to say so. Fixed by importing the gate
+unmodified for that one test and pinning the surface at three documents, and
+the fix was proven by re-staling `ROADMAP.md` and watching the test fail.
+
**Backgrounding a long run kills it here.** The first baseline attempt was
started with `nohup ... &` and was dead within a minute, having written 41
progress dots. Long suites run in the foreground.
diff --git a/ROADMAP.md b/ROADMAP.md
index 414e3ad01..feab76e50 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -44,7 +44,7 @@ A proof-first, security-first program to make every README claim gate-backed, cl
- **Phase D (breadth):** D1 eight connectors (QRadar/Exabeam/Securonix/Devo/Netskope/Windows-Sysmon/Zeek-Suricata/syslog-CEF) · D2 AI/LLM-usage audit connector + `llm-*` detections + hot/cold lake tiering · D3 live-vendor mock-server smoke.
- **Phase E (prove it):** E1 CI-gated benchmark scoreboard tied to a deterministic live-agent MITRE-accuracy run.
-The claim-to-gate matrix stands at **153 rows — 145 GATED / 8 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test** and the ratchet (`MAX_NO_GATE=0`) forbids any regression. The 8 remaining PARTIAL rows are honest, named deferrals; none was relabelled without building the gate it names. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).
+The claim-to-gate matrix stands at **156 rows — 148 GATED / 8 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test** and the ratchet (`MAX_NO_GATE=0`) forbids any regression. The 8 remaining PARTIAL rows are honest, named deferrals; none was relabelled without building the gate it names. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).
## v4.0 — Shipped
diff --git a/apps/docs/docs/compliance/evidence-pack.md b/apps/docs/docs/compliance/evidence-pack.md
index d0ed78cb0..3ae4a06f0 100644
--- a/apps/docs/docs/compliance/evidence-pack.md
+++ b/apps/docs/docs/compliance/evidence-pack.md
@@ -33,7 +33,7 @@ and says so.
| Prompt-injection resistance | 32-payload adversarial corpus with a recall floor and a false-positive ceiling | `services/agents/tests/adversarial/` |
| Claim-to-gate traceability | Every product claim mapped to the CI job that fails when it stops being true | `docs/audit/CLAIM_TO_GATE_MATRIX.md` |
-The last row is the one worth reading first. It is the honest index: 145 rows
+The last row is the one worth reading first. It is the honest index: 148 rows
`GATED`, 8 `PARTIAL` with the specific gap named, and a ratchet that refuses
to let a row sit at `NO GATE`.
diff --git a/apps/docs/docs/evaluation/replay.md b/apps/docs/docs/evaluation/replay.md
index 39687eef4..68f0fda8c 100644
--- a/apps/docs/docs/evaluation/replay.md
+++ b/apps/docs/docs/evaluation/replay.md
@@ -14,11 +14,10 @@ Replay evaluation answers the question that actually matters: **how does AiSOC
triage compare with what your analysts already decided, on your data?**
:::note What exists today
-This page currently documents the **history readers**, which are the first half
-of the feature: the part that reads your closed findings and turns your
-analysts' labels into something gradeable. The replay runner, the scoring
-report and the `aisoc replay` command land next and this page grows with them.
-Nothing here describes a capability that is not in the tree.
+This page documents the **history readers**, the **replay runner** and the
+**scoring report**. The `aisoc replay` command and the console page land next
+and this page grows with them. Nothing here describes a capability that is not
+in the tree.
:::
## The one rule worth reading first
@@ -153,6 +152,76 @@ category error.
| `FalsePositive` | `false_positive` |
| `Unknown` | `unlabeled` |
+## How a replay runs
+
+### Shadow mode writes nothing
+
+A replay runs the same triage path a live alert takes. Not a copy of it: the
+same `FusedAlertTriageWorker.triage` the Kafka consumer calls on every fused
+alert, constructed with two different sinks.
+
+Persistence is injected. In production the worker holds a writer that records
+the verdict to the Investigation Ledger and the `alerts` row, writes the
+outcome prior, queues approvals, caches the verdict for deduplication and
+pushes the disposition back to your SIEM. In a replay it holds one that counts
+each of those and performs none of them. The count is reported, because a
+replay that quietly stopped replaying and a replay whose writes were suppressed
+both write nothing, and only the count tells them apart.
+
+The full investigation graph is not run during a replay, because every node it
+executes records itself to the ledger. That does not change the measurement:
+the verdict is fixed before escalation is reached, and a test drives the real
+worker with a graph runner that tries to rewrite the verdict to prove it.
+
+### Point-in-time context, so the test window cannot answer itself
+
+History is ordered by close time and cut by time, 70/30 by default. The later
+period is the test window. A finding closed at the exact split instant stays on
+the train side.
+
+Organisation memory and outcome priors are captured **once**, at the split, and
+served from that snapshot for the whole run. Without this, a replay feeds
+itself: production writes every verdict back as a per-signature prior and reads
+that prior before triage, so the verdict on one alert auto-closes the next
+alert with the same evidence, and the evaluation grades an answer it supplied
+three seconds earlier. The deduplication cache does the same thing one layer
+earlier and leaves no trace in either store.
+
+There is one honest gap. Organisation-memory statements, as the API serves
+them, carry no creation time, so a statement cannot be tested against the split
+and is kept. The report publishes how many statements were in that position, as
+`statements_without_timestamp`, rather than claiming a tighter freeze than the
+data supports.
+
+### What the report says
+
+Recall on malicious leads, because it is the number you are deciding on and the
+one an imbalanced queue hides. Beside it: per-class precision and recall, a
+confusion matrix, the abstention rate, reliability bins with an expected
+calibration error, the hallucination rate with the indicators behind it,
+per-rule and per-source breakdowns, and bootstrap confidence intervals.
+
+Two rules govern what it prints.
+
+**Below 30 malicious cases in the test window, no headline accuracy is
+printed.** The count and the reason are printed instead. On a queue where
+almost everything is a false positive, an agent that calls everything benign
+scores well, and that figure would describe your queue rather than the product.
+
+**A rate with no denominator reads "not measured", never 0.** A zero in a
+recall column says the agent missed every case of that class; having never been
+asked is a different fact.
+
+### What a replay is not
+
+A replayed finding is normalized by the same connector `normalize()` your live
+pipeline uses, reached over HTTP in the connectors service so there is no
+second copy of any vendor's field mapping to drift. What it does not carry is
+everything fusion adds to a live alert: correlation across related events, the
+fused confidence score, the deterministic narrative and entity resolution.
+Every report states this in its method section. Replay measures triage on a
+single normalized finding, not the whole pipeline.
+
## Privacy: where your data goes
Reading your history happens inside your deployment, against credentials you
@@ -181,3 +250,14 @@ through the same triage path production uses.
- **Elastic yields nothing without tags**, as above.
- **The vendor's own label is kept verbatim** alongside the mapped one, so you
can audit a mapping you disagree with rather than having to trust it.
+- **Fusion's enrichments are absent**, as above. A replay grades triage on one
+ normalized finding at a time.
+- **Tool calls are structurally zero.** Shadow mode declines escalation, and
+ escalation is the only stage of this path that calls tools. The field is
+ recorded rather than omitted so a future change that gives triage a tool
+ shows up as a number moving off zero.
+- **Hallucination counting errs high.** Indicators are extracted from the
+ agent's own reasoning with the same pattern set the synthetic-corpus grader
+ checks against, so a phrase shaped like a domain is checked and, if absent
+ from the evidence, counted. The published rate is a ceiling, and the
+ indicators behind it travel with the report so you can re-derive it.
diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.md
index 7bc2d0850..c7d153146 100644
--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md
+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md
@@ -161,10 +161,13 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
| Air-gapped mode permits local analysis providers only | `operations/sandbox-providers.md`; `core/airgap.py` | `ci.yml :: python-lint` (`check_sandbox_upload_policy.py`, which requires every adapter must declare `local=` on its `ProviderCapabilities`, so a new provider cannot be added without air-gap mode being able to classify it) + `ci.yml :: api tests` (`test_air_gap_mode_refuses_non_local_providers`, where the hosted provider is excluded from `usable_names()` while remaining visible in `names()` with its reason, and the local one becomes the default) | GATED | the registry gate is over the adapters in `providers/`; a provider constructed outside it would not be seen, which is why `build_registry` is the only assembly path |
| A file-analysis failure reaches an agent as "could not check", never as a clean result | `operations/sandbox-providers.md`; agent tool `lookup_file_hash` | `ci.yml :: python-lint` (`check_sandbox_upload_policy.py`, whose AST pass fails any `except` block that constructs a `SandboxReport` or returns `outcome="known"`/`"not_seen"`, which is the shape that turns a provider outage into a benign verdict) + `ci.yml :: api tests` (`TestFailureIsNotACleanResult`, where a miss and a failure are asserted to produce different outcomes and different wording, and an unreachable provider is reported under `could_not_check` rather than folded into "no result") | GATED | the agent-side tool's wording is asserted in `services/agents/tests/test_sandbox_tool.py`; the model's interpretation of that wording is not gated and cannot be |
| A report field the provider does not publish reads "unavailable", never zero or empty | `operations/sandbox-providers.md` (provider contract) | `ci.yml :: api tests` (`test_sandbox_providers.py`, driven against a **recorded** report whose `attackTechniques` is `[]` beside `behavior.analyzed: false`; the adapter must return `UNAVAILABLE` with the stage's own reason attached, and the same payload with the stage marked run must return a real empty tuple. A missing score must not read as 0 and a missing IOC section must not read as "none found") | GATED | the recorded payload covers one provider and one shape of absence; CAPEv2's equivalents are synthetic because no live CAPEv2 instance was available |
+| Replay evaluation runs the production triage path, not a copy of it, and writes nothing while it does | `apps/docs/docs/evaluation/replay.md`; `services/agents/app/replay/runner.py` | `ci.yml` agents job (`test_replay_shadow_no_writes.py` — every durable write on the triage path is monkeypatched on the **real** module to raise a `BaseException`, then a replay is run: a `BaseException` because the worker wraps most of its writes in `contextlib.suppress(Exception)`, which is correct production behaviour and is exactly what would let a leaked write pass a gentler guard. Zero writes is asserted as a *count* rather than as an absence, since a replay that stopped replaying and a replay whose writes were suppressed both write nothing and only the counter tells them apart. That the path is production's is pinned two ways: the worker's default sinks are asserted to be `LiveTriageWriter`/`LiveTriageContextReader`, and an AST pass over `services/agents/app/main.py` asserts the deployed worker is constructed without overriding either. `test_escalation_cannot_change_the_graded_verdict` drives the real worker with a graph runner that rewrites the verdict and asserts the returned summary is unmoved, which is what makes declining escalation in shadow mode sound rather than convenient) | GATED | Triage only. Fusion's correlation, fused confidence score, narrative and entity resolution are absent from a replayed finding; `ENVELOPE_LIMITS` is published in every report's method section rather than left as a comment |
+| The test window of a replay cannot influence its own triage | `apps/docs/docs/evaluation/replay.md` (point-in-time context); `services/agents/app/replay/shadow.py` | `ci.yml` agents job (`test_replay_leakage.py` — three routes from a test-window decision back into a later one, each with its own test because one combined assertion would not say which broke: the outcome-memory prior that auto-closes a repeat without re-triage, the cost governor's dedup cache that short-circuits earlier still and leaves no ledger row or prior behind, and organisation memory read live into the prompt. Each is checked for **sensitivity** as well as for the guard: the memory test runs the identical corpus twice, once with the shadow sinks and once with a live reader and a recording writer, and asserts the second leaks at tier `memory`; the statements test asserts the frozen reader hands the agent the snapshot and that the same worker with the live reader picks up the poison. `capture_context` drops rows recorded after the split in both stores, and a snapshot taken at any other instant is refused rather than used) | GATED | Organisation-memory statements as `GET /feedback/context-statements` serves them carry no creation time, so the split filter has nothing to test them against and they are kept. The count is published as `statements_without_timestamp` in the report's method section rather than the freeze being claimed tighter than it is |
+| A replay report prints "not measured" where there was no denominator, and no headline accuracy on a corpus too thin to support one | `apps/docs/docs/evaluation/replay.md`; `packages/aisoc-benchmark/aisoc_benchmark/replay.py` | `ci.yml` benchmark job (`test_replay_metrics.py` — a class the agent never predicted reports `None` precision and renders as "not measured", because `0.0` in a recall column says the agent missed every case and "never asked" is a different fact; below 30 malicious cases the headline is withheld with the count and the reason, and the rendered report is asserted to contain no percentage in that section. Three populations are asserted apart: unlabelled findings excluded from accuracy and counted, triage refusals scored as neither answer nor abstention, and abstentions counted against malicious recall. A verdict outside the taxonomy gets its own confusion-matrix column rather than being folded into one the agent did not say. Bootstrap intervals are seeded and the seed and resample count travel in the report, so two runs over one corpus render byte-identically) | GATED | Hallucination extraction reuses the existing `_INDICATOR_PATTERNS`, which over-counts rather than under-counts: a phrase shaped like a domain is checked and, if absent from the evidence, counted. The published rate is a ceiling and the per-decision list travels with it so a disputed count can be re-derived |
## Summary
-- GATED: 145
+- GATED: 148
- PARTIAL: 8
- NO GATE: 0 (**every claim is backed by a failing test.** The last NO GATE — the weekly benchmark scoreboard running live against `main` — closed in Phase E1: `scripts/check_scoreboard.py` ties the published scoreboard to a deterministic per-PR live-agent MITRE-accuracy run, while the funded weekly `wet-eval.yml` appends the LLM-tier rows. The remaining PARTIAL rows are honest, named deferrals — each states the specific gap and what closes it — not unproven claims.)
diff --git a/packages/aisoc-benchmark/aisoc_benchmark/__init__.py b/packages/aisoc-benchmark/aisoc_benchmark/__init__.py
index 6c4c4f464..c9abe6343 100644
--- a/packages/aisoc-benchmark/aisoc_benchmark/__init__.py
+++ b/packages/aisoc-benchmark/aisoc_benchmark/__init__.py
@@ -6,20 +6,45 @@ method, get graded on the same corpus by the same code.
"""
from .adapter import AgentVerdict, BenchmarkIncident, HTTPAgent, SOCAgent
-from .metrics import BenchmarkResult, IncidentScore, aggregate, score_incident
+from .metrics import (
+ BenchmarkResult,
+ IncidentScore,
+ aggregate,
+ extract_checkable_indicators,
+ score_incident,
+)
+from .replay import (
+ MALICIOUS,
+ MIN_MALICIOUS_FOR_HEADLINE,
+ CalibrationBin,
+ ClassScore,
+ ReplayScore,
+ SegmentScore,
+ format_replay_report,
+ score_replay,
+)
from .runner import format_report, load_corpus, run_benchmark
__all__ = [
+ "MALICIOUS",
+ "MIN_MALICIOUS_FOR_HEADLINE",
"AgentVerdict",
"BenchmarkIncident",
"BenchmarkResult",
+ "CalibrationBin",
+ "ClassScore",
"HTTPAgent",
"IncidentScore",
+ "ReplayScore",
"SOCAgent",
+ "SegmentScore",
"aggregate",
+ "extract_checkable_indicators",
"format_report",
"load_corpus",
"run_benchmark",
"score_incident",
+ "format_replay_report",
+ "score_replay",
]
__version__ = "0.1.0"
diff --git a/packages/aisoc-benchmark/aisoc_benchmark/metrics.py b/packages/aisoc-benchmark/aisoc_benchmark/metrics.py
index a0796e2fc..77bb551ab 100644
--- a/packages/aisoc-benchmark/aisoc_benchmark/metrics.py
+++ b/packages/aisoc-benchmark/aisoc_benchmark/metrics.py
@@ -42,7 +42,13 @@ _INDICATOR_PATTERNS = (
re.compile(r"\b[a-fA-F0-9]{64}\b"), # SHA-256
re.compile(r"\b[a-fA-F0-9]{40}\b"), # SHA-1
re.compile(r"\b[a-fA-F0-9]{32}\b"), # MD5
- re.compile(r"\b[a-zA-Z0-9-]+\.[a-zA-Z]{2,}\b"), # domain-ish
+ # Domain-ish, including every label. The single-label form this used to
+ # carry matched only `evil.example` out of `evil.example.com`, which was
+ # invisible while the pattern was used solely as a yes/no test and became
+ # visible the moment `extract_checkable_indicators` started returning the
+ # matched text. Anything the narrower form matched this one still matches,
+ # so `is_checkable_indicator` answers exactly as before.
+ re.compile(r"\b[a-zA-Z0-9-]+(?:\.[a-zA-Z0-9-]+)*\.[a-zA-Z]{2,}\b"),
)
@@ -54,6 +60,43 @@ def is_checkable_indicator(value: str) -> bool:
return any(pattern.search(text) for pattern in _INDICATOR_PATTERNS)
+def extract_checkable_indicators(text: str) -> list[str]:
+ """Pull every checkable indicator out of free-text reasoning.
+
+ Replay evaluation (gap-closure Phase 1.3) grades what the production
+ triage path wrote, and that path emits prose: findings and a confidence
+ basis, not a structured ``cited_indicators`` list. So the indicators have
+ to be read back out of the sentences.
+
+ The pattern set is :data:`_INDICATOR_PATTERNS`, unchanged, which is what
+ makes this an extension of the existing hallucination measurement rather
+ than a second one. Anything this returns is by construction something
+ :func:`is_checkable_indicator` accepts, so both paths grade the same
+ vocabulary.
+
+ Over-counting is the safe direction and is the one this errs in: a phrase
+ shaped like a domain that is really an English abbreviation will be
+ checked against the evidence and, if absent, counted as hallucinated. That
+ makes the published rate a ceiling rather than a flattering floor, and the
+ per-decision list travels with the report so a disputed count can be
+ re-derived by a reader who disagrees.
+
+ Ordered by first appearance and de-duplicated, so the same address cited
+ four times is one indicator rather than four.
+ """
+ found: list[str] = []
+ seen: set[str] = set()
+ for pattern in _INDICATOR_PATTERNS:
+ for match in pattern.finditer(text or ""):
+ value = match.group(0)
+ key = value.lower()
+ if key in seen:
+ continue
+ seen.add(key)
+ found.append(value)
+ return sorted(found, key=lambda v: ((text or "").find(v), v))
+
+
def evidence_corpus(incident: Any) -> str:
"""Everything the agent was given, flattened for containment checks."""
parts = [
diff --git a/packages/aisoc-benchmark/aisoc_benchmark/replay.py b/packages/aisoc-benchmark/aisoc_benchmark/replay.py
new file mode 100644
index 000000000..cf8a1a412
--- /dev/null
+++ b/packages/aisoc-benchmark/aisoc_benchmark/replay.py
@@ -0,0 +1,617 @@
+"""Scoring for replay evaluation: the agent against a customer's own analysts.
+
+Gap-closure Phase 1.3.
+
+This extends the benchmark package rather than standing beside it. The
+hallucination machinery is :func:`aisoc_benchmark.metrics.is_checkable_indicator`
+and :func:`~aisoc_benchmark.metrics.extract_checkable_indicators`, unchanged, so
+an indicator counted here is one the synthetic-corpus grader would also have
+counted. A second grader would eventually disagree with the first about what
+"hallucinated" means, and the published numbers would stop being comparable
+while still looking like they were.
+
+What replay measures that the synthetic corpus cannot
+------------------------------------------------------
+The synthetic corpus has balanced classes by construction. A real queue does
+not: most closed findings are false positives, and a handful are the ones that
+mattered. Three consequences shape this module.
+
+**Headline accuracy is withheld on a thin corpus.** Below
+:data:`MIN_MALICIOUS_FOR_HEADLINE` true positives the report prints the count
+and refuses the number. Accuracy over 200 findings of which three were
+malicious is 98% for an agent that calls everything benign, and publishing
+that figure would be actively misleading in the direction that sells.
+
+**Recall on malicious leads.** It is the number an operator is actually
+deciding on, and it is the one an imbalanced corpus hides. :class:`ReplayScore`
+puts it first and the renderers keep that order.
+
+**Every mean travels with the count it was computed over.** A precision of
+1.00 across two predictions is not a precision of 1.00 across two hundred, and
+a table that prints only the ratio has thrown away the distinction.
+
+Absent is not zero
+------------------
+A rate with no denominator is ``None`` and renders as "not measured". This is
+not fussiness: ``0.0`` in a precision column reads as "the agent got every one
+of these wrong", and "the agent never predicted this class" is a different
+fact with a different remedy.
+
+Determinism
+-----------
+Bootstrap confidence intervals resample, so they need a source of randomness,
+and a report that has to reproduce byte for byte cannot have an unseeded one.
+:data:`BOOTSTRAP_SEED` fixes it, :class:`ReplayScore` records the seed and the
+resample count it used, and :func:`score_replay` takes both as arguments so a
+caller who wants a different interval can ask for one and have the report say
+so.
+"""
+
+from __future__ import annotations
+
+import json
+import random
+from collections import Counter
+from dataclasses import asdict, dataclass, field
+from typing import Any
+
+from .metrics import extract_checkable_indicators
+
+__all__ = [
+ "ABSTENTION_VERDICTS",
+ "BOOTSTRAP_RESAMPLES",
+ "BOOTSTRAP_SEED",
+ "CALIBRATION_BINS",
+ "GRADED_DISPOSITIONS",
+ "MALICIOUS",
+ "MIN_MALICIOUS_FOR_HEADLINE",
+ "UNLABELED",
+ "CalibrationBin",
+ "ClassScore",
+ "ReplayScore",
+ "SegmentScore",
+ "format_replay_report",
+ "score_replay",
+]
+
+#: The canonical disposition that means "this was a real threat". Written once
+#: because "malicious recall" has to mean the same thing in the metric, the
+#: renderer and the promotion gate Phase 2 builds on top of it.
+MALICIOUS = "true_positive"
+
+#: The four canonical dispositions an analyst can close a finding as. Mirrors
+#: ``CANONICAL_DISPOSITIONS`` in ``services/actions``; this package does not
+#: import from a service, and ``scripts/check_replay_contract_parity.py``
+#: keeps the two in step in both directions.
+GRADED_DISPOSITIONS: tuple[str, ...] = (
+ MALICIOUS,
+ "benign_true_positive",
+ "false_positive",
+ "benign",
+)
+
+#: An analyst who declined to classify. Excluded from accuracy, never guessed.
+UNLABELED = "unlabeled"
+
+#: Agent outputs that are a refusal to decide rather than a decision. These are
+#: abstentions: scored as neither right nor wrong, and counted separately so
+#: accuracy and abstention cannot be traded off invisibly. ``escalate`` belongs
+#: here for the same reason ``needs_review`` does: both route the alert to a
+#: human, which is a decision about who decides, not about what happened.
+ABSTENTION_VERDICTS: frozenset[str] = frozenset({"needs_review", "escalate", "unknown", ""})
+
+#: Below this many malicious cases in the test window, no headline accuracy is
+#: printed. Set by the plan.
+MIN_MALICIOUS_FOR_HEADLINE = 30
+
+#: Reliability-diagram bin edges. Ten equal-width bins over [0, 1].
+CALIBRATION_BINS = 10
+
+BOOTSTRAP_RESAMPLES = 1000
+BOOTSTRAP_SEED = 20260926
+
+
+@dataclass(frozen=True)
+class ClassScore:
+ """Precision and recall for one disposition, with the counts behind them."""
+
+ label: str
+ #: Ground-truth occurrences. The denominator of recall.
+ support: int
+ #: Times the agent predicted this class. The denominator of precision.
+ predicted: int
+ correct: int
+ precision: float | None
+ recall: float | None
+ f1: float | None
+
+
+@dataclass(frozen=True)
+class CalibrationBin:
+ """One reliability bin: how often the agent was right when this sure."""
+
+ lower: float
+ upper: float
+ count: int
+ mean_confidence: float | None
+ accuracy: float | None
+
+
+@dataclass(frozen=True)
+class SegmentScore:
+ """A per-rule or per-source slice. Small slices are reported, not hidden."""
+
+ key: str
+ graded: int
+ correct: int
+ accuracy: float | None
+ malicious_support: int
+ malicious_recall: float | None
+
+
+@dataclass
+class ReplayScore:
+ """The report. Field order is the reading order the renderers follow."""
+
+ # ---- what was graded -------------------------------------------------
+ decisions: int = 0
+ labelled: int = 0
+ unlabeled: int = 0
+ errored: int = 0
+ graded: int = 0
+ abstained: int = 0
+
+ # ---- the number an operator is deciding on ---------------------------
+ malicious_support: int = 0
+ malicious_recall: float | None = None
+ malicious_recall_ci: tuple[float, float] | None = None
+ malicious_precision: float | None = None
+
+ # ---- headline, withheld on a thin corpus -----------------------------
+ headline_accuracy: float | None = None
+ headline_accuracy_ci: tuple[float, float] | None = None
+ headline_withheld_reason: str | None = None
+
+ # ---- the rest --------------------------------------------------------
+ abstention_rate: float | None = None
+ per_class: list[ClassScore] = field(default_factory=list)
+ confusion: dict[str, dict[str, int]] = field(default_factory=dict)
+ calibration: list[CalibrationBin] = field(default_factory=list)
+ expected_calibration_error: float | None = None
+ hallucination_rate: float | None = None
+ hallucinated_total: int = 0
+ indicators_checked: int = 0
+ hallucinated_examples: list[str] = field(default_factory=list)
+ per_rule: list[SegmentScore] = field(default_factory=list)
+ per_source: list[SegmentScore] = field(default_factory=list)
+
+ # ---- cost and latency, reported not scored ---------------------------
+ total_tokens: int = 0
+ measured_usd: float | None = None
+ estimated_usd: float | None = None
+ unpriced_calls: int = 0
+ mean_latency_ms: float | None = None
+ p95_latency_ms: float | None = None
+ models: list[str] = field(default_factory=list)
+
+ # ---- provenance ------------------------------------------------------
+ bootstrap_resamples: int = BOOTSTRAP_RESAMPLES
+ bootstrap_seed: int = BOOTSTRAP_SEED
+ min_malicious_for_headline: int = MIN_MALICIOUS_FOR_HEADLINE
+
+ def as_dict(self) -> dict[str, Any]:
+ return asdict(self)
+
+
+def _ratio(numerator: int, denominator: int) -> float | None:
+ """A rate, or ``None`` when there was nothing to divide by."""
+ return (numerator / denominator) if denominator else None
+
+
+def _f1(precision: float | None, recall: float | None) -> float | None:
+ if precision is None or recall is None or (precision + recall) == 0:
+ return None
+ return 2 * precision * recall / (precision + recall)
+
+
+def _percentile(values: list[float], fraction: float) -> float | None:
+ if not values:
+ return None
+ ordered = sorted(values)
+ index = min(len(ordered) - 1, int(round(fraction * (len(ordered) - 1))))
+ return ordered[index]
+
+
+def _bootstrap_ci(
+ outcomes: list[int],
+ *,
+ resamples: int,
+ seed: int,
+ confidence: float = 0.95,
+) -> tuple[float, float] | None:
+ """Percentile bootstrap over a list of 0/1 outcomes.
+
+ Returns ``None`` below five observations. An interval over three samples
+ is arithmetic, not evidence, and printing one invites a reader to treat a
+ handful of cases as a measurement.
+ """
+ if len(outcomes) < 5 or resamples <= 0:
+ return None
+ rng = random.Random(seed)
+ n = len(outcomes)
+ means: list[float] = []
+ for _ in range(resamples):
+ total = 0
+ for _ in range(n):
+ total += outcomes[rng.randrange(n)]
+ means.append(total / n)
+ means.sort()
+ tail = (1.0 - confidence) / 2.0
+ low = means[min(len(means) - 1, int(tail * len(means)))]
+ high = means[min(len(means) - 1, int((1.0 - tail) * len(means)))]
+ return (round(low, 6), round(high, 6))
+
+
+def _evidence_text(decision: dict[str, Any]) -> str:
+ """Everything the agent was given, flattened, for the hallucination check."""
+ return repr(decision.get("evidence") or {}).lower()
+
+
+def _reasoning_text(decision: dict[str, Any]) -> str:
+ parts = [*(decision.get("findings") or []), *(decision.get("confidence_basis") or [])]
+ return " ".join(str(p) for p in parts)
+
+
+def _segment(
+ decisions: list[dict[str, Any]],
+ key_name: str,
+) -> list[SegmentScore]:
+ """Group graded decisions by a field and score each group."""
+ groups: dict[str, list[dict[str, Any]]] = {}
+ for decision in decisions:
+ key = str(decision.get(key_name) or "unattributed")
+ groups.setdefault(key, []).append(decision)
+
+ scores: list[SegmentScore] = []
+ for key in sorted(groups):
+ rows = groups[key]
+ answered = [d for d in rows if not _is_abstention(d)]
+ correct = sum(1 for d in answered if _is_correct(d))
+ malicious = [d for d in rows if d.get("expected_disposition") == MALICIOUS]
+ malicious_hit = sum(1 for d in malicious if d.get("verdict") == MALICIOUS)
+ scores.append(
+ SegmentScore(
+ key=key,
+ graded=len(answered),
+ correct=correct,
+ accuracy=_ratio(correct, len(answered)),
+ malicious_support=len(malicious),
+ malicious_recall=_ratio(malicious_hit, len(malicious)),
+ )
+ )
+ return scores
+
+
+def _is_abstention(decision: dict[str, Any]) -> bool:
+ return str(decision.get("verdict") or "") in ABSTENTION_VERDICTS
+
+
+def _is_correct(decision: dict[str, Any]) -> bool:
+ return decision.get("verdict") == decision.get("expected_disposition")
+
+
+def score_replay(
+ decisions: list[dict[str, Any]],
+ *,
+ bootstrap_resamples: int = BOOTSTRAP_RESAMPLES,
+ bootstrap_seed: int = BOOTSTRAP_SEED,
+) -> ReplayScore:
+ """Grade a replay run's decisions against the analysts' own labels.
+
+ ``decisions`` are :meth:`app.replay.runner.ReplayDecision.as_dict`
+ payloads. Plain dicts rather than an imported type, because this package
+ must stay installable without a service on the path: a vendor grading
+ their own agent against their own history should not need ``services/``.
+
+ Three populations, kept apart on purpose:
+
+ * **errored** - triage refused the finding. Neither a verdict nor an
+ abstention. Folding these into abstentions would let a run that crashed
+ on every hard case look appropriately humble.
+ * **unlabeled** - the analyst declined to classify. Excluded from
+ accuracy entirely, and counted so a customer whose history is mostly
+ unlabeled sees that rather than a confident number over a remnant.
+ * **graded** - everything else, split into answered and abstained.
+ """
+ score = ReplayScore(
+ decisions=len(decisions),
+ bootstrap_resamples=bootstrap_resamples,
+ bootstrap_seed=bootstrap_seed,
+ )
+
+ errored = [d for d in decisions if d.get("error")]
+ usable = [d for d in decisions if not d.get("error")]
+ labelled = [d for d in usable if d.get("labelled") and d.get("expected_disposition") in GRADED_DISPOSITIONS]
+ score.errored = len(errored)
+ score.unlabeled = len(usable) - len(labelled)
+ score.labelled = len(labelled)
+
+ # Hallucination spans every decision that produced reasoning, answered or
+ # not, and errored or not: an agent that invented an address before
+ # failing has still invented it.
+ checked = 0
+ hallucinated: list[str] = []
+ for decision in decisions:
+ corpus = _evidence_text(decision)
+ for indicator in extract_checkable_indicators(_reasoning_text(decision)):
+ checked += 1
+ if indicator.lower() not in corpus:
+ hallucinated.append(indicator)
+ score.indicators_checked = checked
+ score.hallucinated_total = len(hallucinated)
+ score.hallucination_rate = _ratio(len(hallucinated), checked)
+ score.hallucinated_examples = sorted(set(hallucinated))[:20]
+
+ # Cost and latency describe the whole run, including the decisions that
+ # are not graded: they were still paid for.
+ score.total_tokens = sum(int(d.get("tokens") or 0) for d in decisions)
+ measured = [float(d["measured_usd"]) for d in decisions if d.get("measured_usd") is not None]
+ estimated = [float(d["estimated_usd"]) for d in decisions if d.get("estimated_usd") is not None]
+ score.measured_usd = round(sum(measured), 6) if measured else None
+ score.estimated_usd = round(sum(estimated), 6) if estimated else None
+ score.unpriced_calls = sum(int(d.get("unpriced_calls") or 0) for d in decisions)
+ latencies = [float(d.get("latency_ms") or 0.0) for d in decisions]
+ score.mean_latency_ms = (sum(latencies) / len(latencies)) if latencies else None
+ score.p95_latency_ms = _percentile(latencies, 0.95)
+ score.models = sorted({str(m) for d in decisions for m in (d.get("resolved_models") or [])})
+
+ if not labelled:
+ score.headline_withheld_reason = (
+ "No finding in the test window carried an analyst disposition this platform can name, so there is nothing to grade against."
+ )
+ return score
+
+ answered = [d for d in labelled if not _is_abstention(d)]
+ score.abstained = len(labelled) - len(answered)
+ score.graded = len(answered)
+ score.abstention_rate = _ratio(score.abstained, len(labelled))
+
+ # ---- confusion matrix, over labelled decisions including abstentions --
+ matrix: dict[str, dict[str, int]] = {
+ expected: dict.fromkeys([*GRADED_DISPOSITIONS, "abstained"], 0) for expected in GRADED_DISPOSITIONS
+ }
+ for decision in labelled:
+ expected = str(decision.get("expected_disposition"))
+ predicted = "abstained" if _is_abstention(decision) else str(decision.get("verdict") or "")
+ if predicted not in matrix[expected]:
+ # A verdict outside the taxonomy. Recorded under its own column
+ # rather than dropped, because a column nobody expected is a
+ # finding about the agent.
+ for row in matrix.values():
+ row.setdefault(predicted, 0)
+ matrix[expected][predicted] += 1
+ score.confusion = matrix
+
+ # ---- per class, precision over answered only -------------------------
+ support = Counter(str(d.get("expected_disposition")) for d in labelled)
+ predicted_counts = Counter(str(d.get("verdict") or "") for d in answered)
+ for label in GRADED_DISPOSITIONS:
+ correct = sum(1 for d in answered if d.get("verdict") == label and _is_correct(d))
+ precision = _ratio(correct, predicted_counts.get(label, 0))
+ recall = _ratio(correct, support.get(label, 0))
+ score.per_class.append(
+ ClassScore(
+ label=label,
+ support=support.get(label, 0),
+ predicted=predicted_counts.get(label, 0),
+ correct=correct,
+ precision=precision,
+ recall=recall,
+ f1=_f1(precision, recall),
+ )
+ )
+
+ malicious_rows = [d for d in labelled if d.get("expected_disposition") == MALICIOUS]
+ score.malicious_support = len(malicious_rows)
+ malicious_class = next(c for c in score.per_class if c.label == MALICIOUS)
+ score.malicious_recall = malicious_class.recall
+ score.malicious_precision = malicious_class.precision
+ # Recall's outcomes are per malicious case: did the agent call it
+ # malicious? An abstention counts against recall, because an alert routed
+ # to a human was not caught by the agent.
+ score.malicious_recall_ci = _bootstrap_ci(
+ [1 if d.get("verdict") == MALICIOUS else 0 for d in malicious_rows],
+ resamples=bootstrap_resamples,
+ seed=bootstrap_seed,
+ )
+
+ # ---- calibration ------------------------------------------------------
+ score.calibration, score.expected_calibration_error = _calibration(answered)
+
+ # ---- headline, withheld below the floor -------------------------------
+ if score.malicious_support < MIN_MALICIOUS_FOR_HEADLINE:
+ score.headline_withheld_reason = (
+ f"The test window holds {score.malicious_support} malicious case(s), below the "
+ f"{MIN_MALICIOUS_FOR_HEADLINE} this report requires before printing a headline accuracy. "
+ f"On a queue where almost everything is a false positive, an agent that calls everything "
+ f"benign scores well, so the figure would describe the queue rather than the agent. "
+ f"Per-class recall and the confusion matrix below are reported in full."
+ )
+ elif answered:
+ correct = sum(1 for d in answered if _is_correct(d))
+ score.headline_accuracy = correct / len(answered)
+ score.headline_accuracy_ci = _bootstrap_ci(
+ [1 if _is_correct(d) else 0 for d in answered],
+ resamples=bootstrap_resamples,
+ seed=bootstrap_seed,
+ )
+
+ score.per_rule = _segment(labelled, "rule_id")
+ score.per_source = _segment(labelled, "vendor")
+ return score
+
+
+def _pct(value: float | None) -> str:
+ """A rate as a percentage, or the words that mean there was no denominator.
+
+ "not measured" rather than ``0.0`` everywhere. A zero in a recall column
+ says the agent missed every malicious case; no denominator says it was
+ never asked. Printing the first when the second is true is the single most
+ misleading thing a report of this kind can do.
+ """
+ return "not measured" if value is None else f"{value * 100:.1f}%"
+
+
+def _interval(bounds: tuple[float, float] | None) -> str:
+ return "not measured" if bounds is None else f"{bounds[0] * 100:.1f}% to {bounds[1] * 100:.1f}%"
+
+
+def format_replay_report(score: ReplayScore, *, method: dict[str, Any] | None = None) -> str:
+ """Render the report as Markdown, malicious recall first.
+
+ Deterministic by construction: no timestamps, no host names, no dict
+ iteration that is not sorted. A report that reproduces byte for byte is
+ the phase's acceptance test, and the easiest way to fail it is a renderer
+ that prints "generated at".
+ """
+ lines: list[str] = ["# Replay evaluation", ""]
+
+ lines += [
+ "## What was graded",
+ "",
+ f"- Decisions replayed: {score.decisions}",
+ f"- Carried an analyst label: {score.labelled}",
+ f"- Unlabelled, excluded from accuracy: {score.unlabeled}",
+ f"- Refused by triage, scored as neither: {score.errored}",
+ f"- Answered: {score.graded}. Abstained: {score.abstained} ({_pct(score.abstention_rate)})",
+ "",
+ "## Recall on malicious",
+ "",
+ f"- Malicious cases in the test window: {score.malicious_support}",
+ f"- Recall: {_pct(score.malicious_recall)} (95% CI {_interval(score.malicious_recall_ci)})",
+ f"- Precision: {_pct(score.malicious_precision)}",
+ "",
+ "## Headline accuracy",
+ "",
+ ]
+ if score.headline_accuracy is None:
+ lines += [f"Withheld. {score.headline_withheld_reason}", ""]
+ else:
+ lines += [
+ f"{_pct(score.headline_accuracy)} over {score.graded} answered decisions (95% CI {_interval(score.headline_accuracy_ci)}).",
+ "",
+ ]
+
+ lines += ["## Per class", "", "| Disposition | Support | Predicted | Precision | Recall | F1 |", "|---|---|---|---|---|---|"]
+ for row in score.per_class:
+ lines.append(f"| {row.label} | {row.support} | {row.predicted} | {_pct(row.precision)} | {_pct(row.recall)} | {_pct(row.f1)} |")
+
+ lines += ["", "## Confusion matrix", "", "Rows are the analyst's label, columns the agent's verdict.", ""]
+ columns = sorted({column for row in score.confusion.values() for column in row})
+ lines.append("| analyst \\ agent | " + " | ".join(columns) + " |")
+ lines.append("|---" * (len(columns) + 1) + "|")
+ for expected in sorted(score.confusion):
+ cells = " | ".join(str(score.confusion[expected].get(column, 0)) for column in columns)
+ lines.append(f"| {expected} | {cells} |")
+
+ lines += [
+ "",
+ "## Calibration",
+ "",
+ f"Expected calibration error: {_pct(score.expected_calibration_error)}",
+ "",
+ "| Confidence bin | Decisions | Mean confidence | Accuracy |",
+ "|---|---|---|---|",
+ ]
+ for cbin in score.calibration:
+ lines.append(f"| {cbin.lower:.1f} to {cbin.upper:.1f} | {cbin.count} | {_pct(cbin.mean_confidence)} | {_pct(cbin.accuracy)} |")
+
+ lines += [
+ "",
+ "## Hallucination",
+ "",
+ f"- Checkable indicators cited: {score.indicators_checked}",
+ f"- Absent from the evidence: {score.hallucinated_total} ({_pct(score.hallucination_rate)})",
+ ]
+ if score.hallucinated_examples:
+ lines.append(f"- Examples: {', '.join(score.hallucinated_examples[:10])}")
+
+ breakdowns: tuple[tuple[str, list[SegmentScore]], ...] = (
+ ("Per rule", score.per_rule),
+ ("Per source", score.per_source),
+ )
+ for title, segments in breakdowns:
+ lines += ["", f"## {title}", "", "| Key | Answered | Accuracy | Malicious | Malicious recall |", "|---|---|---|---|---|"]
+ for segment in segments:
+ lines.append(
+ f"| {segment.key} | {segment.graded} | {_pct(segment.accuracy)} | "
+ f"{segment.malicious_support} | {_pct(segment.malicious_recall)} |"
+ )
+
+ measured = "not measured" if score.measured_usd is None else f"${score.measured_usd:.6f}"
+ estimated = "not measured" if score.estimated_usd is None else f"${score.estimated_usd:.6f}"
+ lines += [
+ "",
+ "## Cost and latency",
+ "",
+ f"- Tokens: {score.total_tokens}",
+ f"- Measured spend: {measured} (list-price estimate for unbilled calls: {estimated}; "
+ f"calls with no price at all: {score.unpriced_calls})",
+ f"- Mean latency: {'not measured' if score.mean_latency_ms is None else f'{score.mean_latency_ms:.0f} ms'}, "
+ f"p95 {'not measured' if score.p95_latency_ms is None else f'{score.p95_latency_ms:.0f} ms'}",
+ f"- Models: {', '.join(score.models) if score.models else 'none, no model call was placed'}",
+ "",
+ "## Method",
+ "",
+ f"- Confidence intervals: percentile bootstrap, {score.bootstrap_resamples} resamples, seed {score.bootstrap_seed}.",
+ f"- Headline accuracy requires at least {score.min_malicious_for_headline} malicious cases.",
+ ]
+ if method:
+ for key in sorted(method):
+ lines.append(f"- {key}: {json.dumps(method[key], sort_keys=True)}")
+ lines.append("")
+ return "\n".join(lines)
+
+
+def _calibration(answered: list[dict[str, Any]]) -> tuple[list[CalibrationBin], float | None]:
+ """Reliability bins and expected calibration error.
+
+ ECE is the support-weighted mean gap between confidence and accuracy
+ across the bins. Empty bins contribute nothing and are still emitted, so
+ the diagram shows where an agent never expressed a confidence rather than
+ silently closing the gap.
+ """
+ if not answered:
+ return [], None
+
+ width = 1.0 / CALIBRATION_BINS
+ bins: list[CalibrationBin] = []
+ weighted_gap = 0.0
+ for index in range(CALIBRATION_BINS):
+ lower = index * width
+ upper = lower + width
+ # The last bin is closed at 1.0 so a confidence of exactly 1.0 lands
+ # somewhere rather than being dropped.
+ members = [
+ d
+ for d in answered
+ if lower <= float(d.get("confidence") or 0.0) < upper
+ or (index == CALIBRATION_BINS - 1 and float(d.get("confidence") or 0.0) == 1.0)
+ ]
+ if members:
+ mean_conf = sum(float(d.get("confidence") or 0.0) for d in members) / len(members)
+ accuracy = sum(1 for d in members if _is_correct(d)) / len(members)
+ weighted_gap += len(members) * abs(mean_conf - accuracy)
+ else:
+ mean_conf = None
+ accuracy = None
+ bins.append(
+ CalibrationBin(
+ lower=round(lower, 4),
+ upper=round(upper, 4),
+ count=len(members),
+ mean_confidence=(round(mean_conf, 6) if mean_conf is not None else None),
+ accuracy=(round(accuracy, 6) if accuracy is not None else None),
+ )
+ )
+ return bins, round(weighted_gap / len(answered), 6)
diff --git a/packages/aisoc-benchmark/tests/test_replay_metrics.py b/packages/aisoc-benchmark/tests/test_replay_metrics.py
new file mode 100644
index 000000000..083ae668a
--- /dev/null
+++ b/packages/aisoc-benchmark/tests/test_replay_metrics.py
@@ -0,0 +1,305 @@
+"""Replay scoring: the numbers, and the ones it refuses to print.
+
+Gap-closure Phase 1.3.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+# The baseline gate runs mypy with `--no-site-packages`, so pytest is
+# unresolvable there while pytest itself imports it without difficulty. The
+# same finding is recorded against `tests/test_benchmark.py`; silenced here
+# rather than added to the baseline, which may only shrink.
+import pytest # type: ignore[import-not-found]
+from aisoc_benchmark.metrics import extract_checkable_indicators
+from aisoc_benchmark.replay import (
+ MIN_MALICIOUS_FOR_HEADLINE,
+ format_replay_report,
+ score_replay,
+)
+
+MALICIOUS = "true_positive"
+FP = "false_positive"
+
+
+def _decision(
+ index: int,
+ expected: str,
+ verdict: str | None,
+ *,
+ confidence: float = 0.8,
+ labelled: bool = True,
+ findings: list[str] | None = None,
+ evidence: dict[str, Any] | None = None,
+ error: str | None = None,
+ rule_id: str = "rule-1",
+ vendor: str = "splunk",
+) -> dict[str, Any]:
+ return {
+ "finding_id": f"F-{index:03d}",
+ "vendor": vendor,
+ "rule_id": rule_id,
+ "closed_at": "2026-05-01T00:00:00+00:00",
+ "expected_disposition": expected,
+ "vendor_disposition": "disposition:1",
+ "labelled": labelled,
+ "verdict": verdict,
+ "confidence": confidence,
+ "tier": "deterministic",
+ "findings": findings or [],
+ "confidence_basis": [],
+ "evidence": evidence or {},
+ "tokens": 0,
+ "measured_usd": None,
+ "estimated_usd": None,
+ "unpriced_calls": 0,
+ "latency_ms": 10,
+ "resolved_models": [],
+ "error": error,
+ }
+
+
+def _corpus(malicious: int, benign: int, *, hits: int | None = None) -> list[dict[str, Any]]:
+ """``hits`` malicious cases the agent caught; the rest it called benign."""
+ caught = malicious if hits is None else hits
+ rows = [_decision(i, MALICIOUS, MALICIOUS if i < caught else FP) for i in range(malicious)]
+ rows += [_decision(1000 + i, FP, FP) for i in range(benign)]
+ return rows
+
+
+# --------------------------------------------------------------------------
+# The headline floor
+# --------------------------------------------------------------------------
+
+
+def test_a_thin_malicious_corpus_prints_no_headline_accuracy() -> None:
+ score = score_replay(_corpus(malicious=5, benign=195))
+
+ assert score.headline_accuracy is None
+ assert score.headline_accuracy_ci is None
+ assert "5 malicious case(s)" in (score.headline_withheld_reason or "")
+ # Everything else is still reported. Withholding one figure is not an
+ # excuse to withhold the ones that make the thin corpus visible.
+ assert score.malicious_support == 5
+ assert score.malicious_recall == 1.0
+
+
+def test_the_headline_appears_once_the_floor_is_met() -> None:
+ score = score_replay(_corpus(malicious=MIN_MALICIOUS_FOR_HEADLINE, benign=30))
+
+ assert score.headline_withheld_reason is None
+ assert score.headline_accuracy == pytest.approx(1.0)
+
+
+def test_the_withheld_headline_renders_as_words_not_as_zero() -> None:
+ """A reader must never see a number where a refusal belongs."""
+ report = format_replay_report(score_replay(_corpus(malicious=2, benign=50)))
+
+ assert "Withheld." in report
+ assert "0.0%" not in report.split("## Per class")[0].split("## Headline accuracy")[1]
+
+
+# --------------------------------------------------------------------------
+# Per class, and absent versus zero
+# --------------------------------------------------------------------------
+
+
+def test_a_class_the_agent_never_predicted_reads_not_measured() -> None:
+ score = score_replay(_corpus(malicious=10, benign=10))
+
+ benign_true_positive = next(c for c in score.per_class if c.label == "benign_true_positive")
+ assert benign_true_positive.support == 0
+ assert benign_true_positive.precision is None
+ assert benign_true_positive.recall is None
+ assert "not measured" in format_replay_report(score)
+
+
+def test_recall_counts_a_missed_malicious_case_against_the_agent() -> None:
+ score = score_replay(_corpus(malicious=10, benign=0, hits=6))
+
+ assert score.malicious_recall == pytest.approx(0.6)
+ assert score.malicious_precision == pytest.approx(1.0)
+
+
+def test_an_abstention_counts_against_malicious_recall() -> None:
+ """An alert routed to a human was not caught by the agent."""
+ rows = [_decision(i, MALICIOUS, "needs_review") for i in range(10)]
+
+ score = score_replay(rows)
+
+ assert score.abstained == 10
+ assert score.graded == 0
+ assert score.malicious_recall == 0.0
+ assert score.abstention_rate == 1.0
+
+
+def test_escalate_is_an_abstention_not_a_verdict() -> None:
+ score = score_replay([_decision(i, MALICIOUS, "escalate") for i in range(4)])
+
+ assert score.abstained == 4
+
+
+# --------------------------------------------------------------------------
+# Populations kept apart
+# --------------------------------------------------------------------------
+
+
+def test_unlabelled_findings_are_excluded_from_accuracy_and_counted() -> None:
+ rows = _corpus(malicious=10, benign=10)
+ rows += [_decision(2000 + i, "unlabeled", FP, labelled=False) for i in range(40)]
+
+ score = score_replay(rows)
+
+ assert score.decisions == 60
+ assert score.labelled == 20
+ assert score.unlabeled == 40
+ assert score.graded == 20
+
+
+def test_a_refusal_is_neither_an_answer_nor_an_abstention() -> None:
+ rows = _corpus(malicious=4, benign=4)
+ rows += [_decision(3000 + i, MALICIOUS, None, error="normalize failed") for i in range(3)]
+
+ score = score_replay(rows)
+
+ assert score.errored == 3
+ assert score.labelled == 8
+ assert score.abstained == 0
+
+
+def test_a_verdict_outside_the_taxonomy_gets_its_own_column() -> None:
+ """An unrecognised verdict is a finding about the product, not a rounding error."""
+ rows = [_decision(i, MALICIOUS, "wildly_unexpected") for i in range(5)]
+
+ score = score_replay(rows)
+
+ assert score.confusion[MALICIOUS]["wildly_unexpected"] == 5
+
+
+# --------------------------------------------------------------------------
+# Calibration
+# --------------------------------------------------------------------------
+
+
+def test_a_perfectly_calibrated_agent_has_near_zero_calibration_error() -> None:
+ # Eight of ten right, all claiming 0.8.
+ rows = [_decision(i, MALICIOUS, MALICIOUS if i < 8 else FP, confidence=0.8) for i in range(10)]
+
+ score = score_replay(rows)
+
+ assert score.expected_calibration_error == pytest.approx(0.0, abs=1e-6)
+
+
+def test_a_confidently_wrong_agent_has_a_large_calibration_error() -> None:
+ rows = [_decision(i, MALICIOUS, FP, confidence=0.95) for i in range(10)]
+
+ score = score_replay(rows)
+
+ assert score.expected_calibration_error == pytest.approx(0.95, abs=1e-6)
+
+
+def test_every_bin_is_emitted_so_an_unused_confidence_range_is_visible() -> None:
+ score = score_replay(_corpus(malicious=10, benign=0))
+
+ assert len(score.calibration) == 10
+ assert any(b.count == 0 and b.accuracy is None for b in score.calibration)
+
+
+# --------------------------------------------------------------------------
+# Hallucination
+# --------------------------------------------------------------------------
+
+
+def test_an_indicator_absent_from_the_evidence_is_counted_as_hallucinated() -> None:
+ rows = [
+ _decision(
+ 0,
+ MALICIOUS,
+ MALICIOUS,
+ findings=["Beaconing to 203.0.113.77 from 10.0.0.5"],
+ evidence={"src_ip": "10.0.0.5"},
+ )
+ ]
+
+ score = score_replay(rows)
+
+ assert score.indicators_checked == 2
+ assert score.hallucinated_total == 1
+ assert score.hallucinated_examples == ["203.0.113.77"]
+ assert score.hallucination_rate == pytest.approx(0.5)
+
+
+def test_hallucination_is_graded_on_a_decision_that_errored() -> None:
+ """An agent that invented an address before failing has still invented it."""
+ rows = [_decision(0, MALICIOUS, None, findings=["saw 203.0.113.9"], error="triage failed")]
+
+ score = score_replay(rows)
+
+ assert score.hallucinated_total == 1
+
+
+def test_extraction_uses_the_same_vocabulary_the_existing_grader_checks() -> None:
+ found = extract_checkable_indicators("contacted evil.example.com at 198.51.100.4 twice")
+
+ assert found == ["evil.example.com", "198.51.100.4"]
+
+
+# --------------------------------------------------------------------------
+# Breakdowns, intervals, determinism
+# --------------------------------------------------------------------------
+
+
+def test_per_rule_and_per_source_slices_report_their_own_sample_size() -> None:
+ rows = [_decision(i, MALICIOUS, MALICIOUS, rule_id="rule-a", vendor="splunk") for i in range(6)]
+ rows += [_decision(100 + i, MALICIOUS, FP, rule_id="rule-b", vendor="sentinel") for i in range(4)]
+
+ score = score_replay(rows)
+
+ by_rule = {s.key: s for s in score.per_rule}
+ assert by_rule["rule-a"].graded == 6
+ assert by_rule["rule-a"].malicious_recall == 1.0
+ assert by_rule["rule-b"].malicious_recall == 0.0
+ assert {s.key for s in score.per_source} == {"splunk", "sentinel"}
+
+
+def test_a_finding_with_no_rule_id_is_grouped_rather_than_dropped() -> None:
+ score = score_replay([_decision(0, MALICIOUS, MALICIOUS, rule_id="")])
+
+ assert [s.key for s in score.per_rule] == ["unattributed"]
+
+
+def test_an_interval_needs_more_than_a_handful_of_cases() -> None:
+ score = score_replay(_corpus(malicious=3, benign=0))
+
+ assert score.malicious_recall == 1.0
+ assert score.malicious_recall_ci is None
+
+
+def test_scoring_the_same_decisions_twice_gives_the_same_report() -> None:
+ """The bootstrap is seeded, so the report reproduces byte for byte."""
+ rows = _corpus(malicious=40, benign=60, hits=31)
+
+ first = format_replay_report(score_replay(rows))
+ second = format_replay_report(score_replay(rows))
+
+ assert first == second
+ assert "seed" in first
+
+
+def test_a_different_seed_is_recorded_in_the_report() -> None:
+ rows = _corpus(malicious=40, benign=60, hits=31)
+
+ report = format_replay_report(score_replay(rows, bootstrap_seed=7, bootstrap_resamples=200))
+
+ assert "200 resamples, seed 7" in report
+
+
+def test_a_history_with_no_labels_at_all_says_so() -> None:
+ rows = [_decision(i, "unlabeled", FP, labelled=False) for i in range(50)]
+
+ score = score_replay(rows)
+
+ assert score.headline_accuracy is None
+ assert "nothing to grade against" in (score.headline_withheld_reason or "")
+ assert score.unlabeled == 50
diff --git a/scripts/check_replay_contract_parity.py b/scripts/check_replay_contract_parity.py
new file mode 100644
index 000000000..0a098f83f
--- /dev/null
+++ b/scripts/check_replay_contract_parity.py
@@ -0,0 +1,172 @@
+#!/usr/bin/env python3
+"""The replay contract spans three trees that cannot import each other. Pin it.
+
+Gap-closure Phase 1.2 and 1.3.
+
+``services/actions`` reads a customer's closed findings and produces
+``ClosedFinding``. ``services/agents`` replays them and consumes
+``HistoricalFinding``. ``packages/aisoc-benchmark`` grades the result against
+``GRADED_DISPOSITIONS``. All three package their code as top-level ``app`` or
+as a standalone distribution, so none of them can import the others, and the
+contract between them is three declarations that agree by convention.
+
+Conventions drift. This gate reads all three with ``ast`` and compares them
+**in both directions**, because the failure this tree keeps finding is a gate
+that checks A against B and prints OK while B grows a field A never hears
+about. A field added to the reader and not to the replay type would be dropped
+silently at the boundary; a disposition added to the taxonomy and not to the
+grader would be scored as an unknown verdict.
+
+Nothing is imported. The gate runs on a bare interpreter, and importing
+``services/actions`` would drag in httpx and the whole client stack to compare
+a list of attribute names.
+"""
+
+from __future__ import annotations
+
+import argparse
+import ast
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+
+from gate_toolkit import repo_root, self_test_if_requested # noqa: E402
+
+self_test_if_requested(__file__)
+
+#: (path, class name) for the two dataclasses that must carry the same fields.
+_READER = ("services/actions/app/services/alert_history.py", "ClosedFinding")
+_REPLAY = ("services/agents/app/replay/findings.py", "HistoricalFinding")
+
+#: The canonical taxonomy, declared once per tree.
+_TAXONOMY_SOURCE = ("services/actions/app/services/disposition_writeback.py", "CANONICAL_DISPOSITIONS")
+_GRADER_SOURCE = ("packages/aisoc-benchmark/aisoc_benchmark/replay.py", "GRADED_DISPOSITIONS")
+
+#: ``CANONICAL_DISPOSITIONS`` covers everything a verdict may be, including the
+#: two that route to a human. The grader's ``GRADED_DISPOSITIONS`` is only the
+#: subset an analyst can *close* a finding as, because "needs review" is not an
+#: outcome anyone closed anything with. These are the members that may appear
+#: in one and not the other, and they are listed rather than inferred so adding
+#: a third does not silently widen the exemption.
+_NOT_A_CLOSING_LABEL = frozenset({"needs_review", "escalate"})
+
+#: The spelling of "the analyst declined to classify", which both trees hold
+#: as a module constant and neither may rename alone.
+_UNLABELED = ("unlabeled", [_READER[0], _REPLAY[0]])
+
+
+def _module(root: Path, relative: str) -> ast.Module:
+ path = root / relative
+ if not path.is_file():
+ raise SystemExit(f"FAIL: {relative} is missing; the replay contract cannot be checked")
+ return ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
+
+
+def _dataclass_fields(tree: ast.Module, class_name: str, relative: str) -> set[str]:
+ for node in ast.walk(tree):
+ if isinstance(node, ast.ClassDef) and node.name == class_name:
+ fields = {item.target.id for item in node.body if isinstance(item, ast.AnnAssign) and isinstance(item.target, ast.Name)}
+ if not fields:
+ raise SystemExit(f"FAIL: {class_name} in {relative} declares no annotated fields")
+ return fields
+ raise SystemExit(f"FAIL: {class_name} was not found in {relative}")
+
+
+def _string_members(tree: ast.Module, name: str, relative: str) -> set[str]:
+ """Read a module-level frozenset/tuple/set of string literals."""
+ for node in ast.walk(tree):
+ if isinstance(node, ast.Assign):
+ targets: list[ast.expr] = list(node.targets)
+ elif isinstance(node, ast.AnnAssign):
+ targets = [node.target]
+ else:
+ continue
+ if not any(isinstance(t, ast.Name) and t.id == name for t in targets):
+ continue
+ value: ast.expr | None = node.value
+ if isinstance(value, ast.Call) and value.args:
+ value = value.args[0]
+ if isinstance(value, ast.Tuple | ast.List | ast.Set):
+ members = {e.value for e in value.elts if isinstance(e, ast.Constant) and isinstance(e.value, str)}
+ names = {e.id for e in value.elts if isinstance(e, ast.Name)}
+ if members or names:
+ return members | {_resolve_name(tree, n) for n in names}
+ raise SystemExit(f"FAIL: {name} was not found as a literal collection in {relative}")
+
+
+def _resolve_name(tree: ast.Module, name: str) -> str:
+ """Resolve a module-level ``NAME = "literal"`` binding."""
+ for node in ast.walk(tree):
+ if isinstance(node, ast.Assign) and any(isinstance(t, ast.Name) and t.id == name for t in node.targets):
+ if isinstance(node.value, ast.Constant) and isinstance(node.value.value, str):
+ return str(node.value.value)
+ raise SystemExit(f"FAIL: {name} is referenced in a taxonomy collection but is not a module-level string")
+
+
+def _constant(tree: ast.Module, name: str, relative: str) -> str:
+ for node in ast.walk(tree):
+ if isinstance(node, ast.Assign) and any(isinstance(t, ast.Name) and t.id == name for t in node.targets):
+ if isinstance(node.value, ast.Constant) and isinstance(node.value.value, str):
+ return str(node.value.value)
+ raise SystemExit(f"FAIL: {name} was not found as a string constant in {relative}")
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--self-test", action="store_true", help="prove the gate refuses an empty tree")
+ parser.parse_args(argv)
+
+ root = repo_root()
+ failures: list[str] = []
+ checked = 0
+
+ reader_fields = _dataclass_fields(_module(root, _READER[0]), _READER[1], _READER[0])
+ replay_fields = _dataclass_fields(_module(root, _REPLAY[0]), _REPLAY[1], _REPLAY[0])
+ checked += len(reader_fields) + len(replay_fields)
+
+ missing_here = sorted(reader_fields - replay_fields)
+ missing_there = sorted(replay_fields - reader_fields)
+ if missing_here:
+ failures.append(
+ f"{_READER[1]} carries {missing_here} and {_REPLAY[1]} does not: the reader sends fields replay would drop at the boundary"
+ )
+ if missing_there:
+ failures.append(f"{_REPLAY[1]} carries {missing_there} and {_READER[1]} does not: replay expects fields no reader produces")
+
+ canonical = _string_members(_module(root, _TAXONOMY_SOURCE[0]), _TAXONOMY_SOURCE[1], _TAXONOMY_SOURCE[0])
+ graded = _string_members(_module(root, _GRADER_SOURCE[0]), _GRADER_SOURCE[1], _GRADER_SOURCE[0])
+ checked += len(canonical) + len(graded)
+
+ ungraded = sorted((canonical - graded) - _NOT_A_CLOSING_LABEL)
+ invented = sorted(graded - canonical)
+ if ungraded:
+ failures.append(f"{_TAXONOMY_SOURCE[1]} holds {ungraded}, which an analyst can close a finding as and the grader does not score")
+ if invented:
+ failures.append(f"{_GRADER_SOURCE[1]} holds {invented}, which the canonical taxonomy does not define")
+
+ name, files = _UNLABELED
+ spellings = {relative: _constant(_module(root, relative), "UNLABELED", relative) for relative in files}
+ checked += len(spellings)
+ if len(set(spellings.values())) != 1 or next(iter(spellings.values())) != name:
+ failures.append(f"UNLABELED is spelled inconsistently across the replay contract: {spellings}")
+
+ if not checked:
+ print("FAIL: the gate compared nothing, which is indistinguishable from a clean result")
+ return 1
+
+ if failures:
+ print(f"FAIL: the replay contract disagrees across trees ({len(failures)} finding(s))")
+ for failure in failures:
+ print(f" - {failure}")
+ return 1
+
+ print(
+ f"OK: replay contract agrees across three trees "
+ f"({len(reader_fields)} finding fields, {len(graded)} graded dispositions, {len(spellings)} UNLABELED spellings)"
+ )
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/scripts/mypy_baseline.json b/scripts/mypy_baseline.json
index 12681f1a7..f70342544 100644
--- a/scripts/mypy_baseline.json
+++ b/scripts/mypy_baseline.json
@@ -50,9 +50,6 @@
"tests/test_detection_id_stability.py": {
"attr-defined": 2
},
- "tests/test_readme_figures_gate.py": {
- "attr-defined": 5
- },
"tests/test_toolchain_pin_gate.py": {
"name-defined": 1
}
diff --git a/services/agents/app/core/cost_telemetry.py b/services/agents/app/core/cost_telemetry.py
index 0ce8ccb61..0e1395cbd 100644
--- a/services/agents/app/core/cost_telemetry.py
+++ b/services/agents/app/core/cost_telemetry.py
@@ -239,6 +239,12 @@ class CostSummary:
estimated_calls: int = 0
unpriced_calls: int = 0
+ #: What the gateway resolved the aliases to, when it said. Empty means no
+ #: model call was placed on this run, which is the honest answer for the
+ #: deterministic tier and is a different fact from "an unknown model".
+ #: Replay records it per decision so a report names the model it graded.
+ resolved_models: tuple[str, ...] = ()
+
@classmethod
def from_tracker(cls, tracker: CostTracker) -> CostSummary:
return cls(
@@ -247,6 +253,7 @@ class CostSummary:
estimated_usd=tracker.estimated_cost_usd,
estimated_calls=tracker.estimated_call_count,
unpriced_calls=tracker.unpriced_call_count,
+ resolved_models=tracker.resolved_models,
)
@@ -435,6 +442,18 @@ async def _flush_to_db(
class CostTracker:
run_id: str
tenant_id: str
+
+ #: Whether :meth:`flush` may write to ``aisoc_run_costs``.
+ #:
+ #: Gap-closure Phase 1.2. Shadow-mode replay has to *measure* spend, because
+ #: tokens, latency and measured cost are fields the replay report records
+ #: per decision. It must not *persist* it: a replay of somebody's closed
+ #: history is not a run their cost dashboard should bill them for, and the
+ #: run ids are synthetic. Measurement and persistence were the same step
+ #: until this flag, so the only way to decline the row was to decline the
+ #: numbers with it.
+ persist: bool = True
+
_records: list[CallRecord] = field(default_factory=list, init=False)
_start: float = field(default_factory=time.monotonic, init=False)
@@ -534,6 +553,11 @@ class CostTracker:
def total_latency_ms(self) -> float:
return sum(r.latency_ms for r in self._records)
+ @property
+ def resolved_models(self) -> tuple[str, ...]:
+ """Models the gateway said it billed, sorted. Empty when no call was placed."""
+ return tuple(sorted({r.resolved_model for r in self._records if r.resolved_model}))
+
def summary(self) -> dict:
"""The run's spend, with the provenance of every figure in it.
@@ -560,7 +584,9 @@ class CostTracker:
async def flush(self) -> None:
summary = self.summary()
- logger.info("cost_telemetry.run_summary", **summary)
+ logger.info("cost_telemetry.run_summary", **summary, persisted=self.persist)
+ if not self.persist:
+ return
await _flush_to_db(self.run_id, self.tenant_id, self._records)
diff --git a/services/agents/app/replay/__init__.py b/services/agents/app/replay/__init__.py
new file mode 100644
index 000000000..d2661153c
--- /dev/null
+++ b/services/agents/app/replay/__init__.py
@@ -0,0 +1,50 @@
+"""Replay evaluation: measure triage against a customer's own closed findings.
+
+Gap-closure Phase 1.2. See :mod:`app.replay.runner` for what runs, and
+:mod:`app.replay.shadow` for why it writes nothing and reads a frozen world.
+"""
+
+from __future__ import annotations
+
+from app.replay.findings import UNLABELED, HistoricalFinding
+from app.replay.normalize import (
+ ENVELOPE_LIMITS,
+ ConnectorNormalizer,
+ FindingNormalizer,
+ NormalizerUnavailable,
+ to_fused_envelope,
+)
+from app.replay.runner import (
+ DEFAULT_TRAIN_FRACTION,
+ ReplayDecision,
+ ReplayRun,
+ ReplayRunner,
+ TimeSplit,
+ split_by_time,
+)
+from app.replay.shadow import (
+ ContextSnapshot,
+ FrozenTriageContextReader,
+ ShadowTriageWriter,
+ capture_context,
+)
+
+__all__ = [
+ "DEFAULT_TRAIN_FRACTION",
+ "ENVELOPE_LIMITS",
+ "UNLABELED",
+ "ConnectorNormalizer",
+ "ContextSnapshot",
+ "FindingNormalizer",
+ "FrozenTriageContextReader",
+ "HistoricalFinding",
+ "NormalizerUnavailable",
+ "ReplayDecision",
+ "ReplayRun",
+ "ReplayRunner",
+ "ShadowTriageWriter",
+ "TimeSplit",
+ "capture_context",
+ "split_by_time",
+ "to_fused_envelope",
+]
diff --git a/services/agents/app/replay/connector_normalizer.py b/services/agents/app/replay/connector_normalizer.py
new file mode 100644
index 000000000..628da6c55
--- /dev/null
+++ b/services/agents/app/replay/connector_normalizer.py
@@ -0,0 +1,141 @@
+"""Normalize replay input through the connectors service, not through a copy of it.
+
+Gap-closure Phase 1.2.
+
+Why HTTP
+--------
+``services/connectors`` and ``services/agents`` both package their code as
+top-level ``app``. One Python process can therefore hold one of them, and no
+amount of path juggling changes that. So the choice was between a second copy
+of every vendor's field mapping living here, and one round trip to the service
+that owns the mapping.
+
+The copy loses. Replay exists to measure the product on a customer's own data;
+a mapping that drifted from the connectors service would grade the agent on an
+input shape the product never produces, and it would keep reporting confident
+numbers while doing so. This is the same reasoning that put organisation
+memory and SIEM writeback behind HTTP calls to the API service rather than
+behind a second copy of their SQL.
+
+Batched on purpose
+------------------
+:func:`fetch_normalized` sends the whole window in one request and returns a
+lookup. Two hundred findings are two hundred round trips otherwise, on a path
+an operator runs interactively.
+
+The lookup is keyed by a hash of the canonical row rather than by position,
+because the runner walks findings in split order and the request is sent in
+input order, and a positional mapping that silently slipped by one would
+attach every verdict to the wrong analyst label. A row the service did not
+return is a :class:`NormalizerUnavailable`, never a substituted mapping.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+from collections.abc import Mapping, Sequence
+from typing import Any
+
+import httpx
+import structlog
+
+from app.replay.normalize import NormalizerUnavailable
+
+logger = structlog.get_logger()
+
+__all__ = ["PrenormalizedRows", "fetch_normalized", "row_key"]
+
+_SERVICE_URL = os.getenv("CONNECTORS_SERVICE_URL", "http://connectors:8003")
+_TIMEOUT_S = float(os.getenv("AISOC_REPLAY_NORMALIZE_TIMEOUT_S", "60"))
+
+
+def row_key(row: Mapping[str, Any]) -> str:
+ """Stable identity for a vendor row, independent of dict ordering."""
+ blob = json.dumps(row, sort_keys=True, default=str)
+ return hashlib.sha256(blob.encode("utf-8")).hexdigest()
+
+
+class PrenormalizedRows:
+ """A :class:`FindingNormalizer` backed by an already-fetched batch.
+
+ Implements the same one-method port a connector does, so the runner does
+ not know or care whether it is holding a connector or a lookup.
+ """
+
+ def __init__(self, mapping: Mapping[str, Mapping[str, Any]], *, connector_id: str) -> None:
+ self._mapping = dict(mapping)
+ self.connector_id = connector_id
+
+ def __len__(self) -> int:
+ return len(self._mapping)
+
+ def normalize(self, raw: dict[str, Any]) -> dict[str, Any]:
+ envelope = self._mapping.get(row_key(raw))
+ if envelope is None:
+ raise NormalizerUnavailable(
+ f"the connectors service returned no normalized envelope for this {self.connector_id} row; "
+ f"replay will not substitute its own mapping"
+ )
+ return dict(envelope)
+
+
+async def fetch_normalized(
+ connector_id: str,
+ rows: Sequence[Mapping[str, Any]],
+ *,
+ tenant_id: str,
+ service_url: str | None = None,
+ service_token: str | None = None,
+) -> PrenormalizedRows:
+ """Map every row through the connectors service and return a lookup.
+
+ Raises :class:`NormalizerUnavailable` on any failure. Replay has no
+ degraded mode here: a run that could not normalize its input has nothing
+ to measure, and continuing with partial coverage would publish a number
+ over whichever rows happened to succeed.
+ """
+ if not rows:
+ return PrenormalizedRows({}, connector_id=connector_id)
+
+ base = (service_url or _SERVICE_URL).rstrip("/")
+ token = service_token if service_token is not None else os.getenv("AISOC_SERVICE_TOKEN", "").strip()
+ if not token:
+ raise NormalizerUnavailable(
+ "AISOC_SERVICE_TOKEN is unset, so the connectors service will refuse the service path "
+ "and replay cannot reach the production normalizer"
+ )
+
+ url = f"{base}/connectors/{connector_id}/normalize"
+ payload = {"rows": [dict(row) for row in rows]}
+ # A service token must declare the tenant it acts for; the connectors
+ # service refuses one that does not. The tenant comes from the caller's
+ # credential-derived scope, never from a field in the request body.
+ headers = {"Authorization": f"Bearer {token}", "X-AiSOC-Tenant-ID": tenant_id}
+ try:
+ async with httpx.AsyncClient(timeout=_TIMEOUT_S) as client:
+ response = await client.post(url, json=payload, headers=headers)
+ except httpx.HTTPError as exc:
+ raise NormalizerUnavailable(f"connectors service unreachable at {base}: {exc}") from exc
+
+ if response.status_code >= 400:
+ raise NormalizerUnavailable(
+ f"connectors service refused the normalize request for '{connector_id}' with HTTP {response.status_code}: {response.text[:300]}"
+ )
+
+ try:
+ body = response.json()
+ except ValueError as exc:
+ raise NormalizerUnavailable("connectors service returned a non-JSON normalize response") from exc
+
+ returned = body.get("rows") if isinstance(body, dict) else None
+ if not isinstance(returned, list) or len(returned) != len(rows):
+ raise NormalizerUnavailable(
+ f"connectors service returned {len(returned) if isinstance(returned, list) else 'no'} envelopes "
+ f"for {len(rows)} rows; a partial batch would silently drop findings from the evaluation"
+ )
+
+ mapping = {row_key(row): envelope for row, envelope in zip(rows, returned, strict=True)}
+ logger.info("replay.normalized", connector_id=connector_id, rows=len(rows), distinct=len(mapping))
+ return PrenormalizedRows(mapping, connector_id=connector_id)
diff --git a/services/agents/app/replay/findings.py b/services/agents/app/replay/findings.py
new file mode 100644
index 000000000..676970314
--- /dev/null
+++ b/services/agents/app/replay/findings.py
@@ -0,0 +1,97 @@
+"""The finding shape replay grades against, as this service sees it.
+
+Gap-closure Phase 1.2.
+
+Phase 1.1 put the readers on ``services/actions``, because that service owns
+the five SIEM credential paths and already holds the writeback going the other
+way. It produces ``app.services.alert_history.ClosedFinding``.
+
+This service cannot import that one. So :class:`HistoricalFinding` is the same
+row seen from here, and the two are kept in step by
+``scripts/check_replay_contract_parity.py``, which parses both dataclasses and
+compares the field sets **in both directions**. A one-directional check would
+pass while this side quietly lost a field the reader still sends, which is the
+failure shape this repository keeps finding in its own gates.
+
+The mapping is deliberately by name and nothing else. There is no positional
+construction anywhere, so a field appearing on one side and not the other is a
+loud ``TypeError`` at the boundary rather than a value landing in the wrong
+column.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Mapping
+from dataclasses import dataclass, field
+from datetime import UTC, datetime
+from typing import Any
+
+#: An analyst closed the finding without recording a disposition the platform
+#: can name. Mirrors ``app.services.alert_history.UNLABELED`` in
+#: ``services/actions``; the parity gate pins the two spellings together.
+UNLABELED = "unlabeled"
+
+__all__ = ["UNLABELED", "HistoricalFinding"]
+
+
+@dataclass(frozen=True)
+class HistoricalFinding:
+ """One finding a human already closed, with the label they chose."""
+
+ vendor: str
+ finding_id: str
+ title: str
+ disposition: str
+ vendor_disposition: str
+ closed_at: datetime
+ closed_by: str | None = None
+ reason: str | None = None
+ rule_id: str | None = None
+ severity: str | None = None
+ raw: Mapping[str, Any] = field(default_factory=dict)
+
+ @property
+ def is_labelled(self) -> bool:
+ """Whether this row may contribute to accuracy."""
+ return self.disposition != UNLABELED
+
+ @classmethod
+ def from_mapping(cls, row: Mapping[str, Any]) -> HistoricalFinding:
+ """Build from a ``ClosedFinding`` serialised across the service boundary.
+
+ ``closed_at`` is required and is not defaulted to "now" when it is
+ missing or unparseable. A finding with an invented close time lands on
+ whichever side of the train/test split the default happens to fall, so
+ a silent default here is a leak with a plausible-looking timestamp on
+ it.
+ """
+ closed_at = _require_time(row.get("closed_at"), finding_id=row.get("finding_id"))
+ return cls(
+ vendor=str(row.get("vendor") or ""),
+ finding_id=str(row.get("finding_id") or ""),
+ title=str(row.get("title") or ""),
+ disposition=str(row.get("disposition") or UNLABELED),
+ vendor_disposition=str(row.get("vendor_disposition") or ""),
+ closed_at=closed_at,
+ closed_by=_opt(row.get("closed_by")),
+ reason=_opt(row.get("reason")),
+ rule_id=_opt(row.get("rule_id")),
+ severity=_opt(row.get("severity")),
+ raw=dict(row.get("raw") or {}),
+ )
+
+
+def _opt(value: Any) -> str | None:
+ return str(value) if value not in (None, "") else None
+
+
+def _require_time(value: Any, *, finding_id: Any) -> datetime:
+ if isinstance(value, datetime):
+ return value if value.tzinfo else value.replace(tzinfo=UTC)
+ if isinstance(value, str) and value.strip():
+ try:
+ parsed = datetime.fromisoformat(value.strip().replace("Z", "+00:00"))
+ except ValueError as exc:
+ raise ValueError(f"finding {finding_id!r} has an unparseable closed_at {value!r}") from exc
+ return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC)
+ raise ValueError(f"finding {finding_id!r} has no usable closed_at (got {value!r})")
diff --git a/services/agents/app/replay/normalize.py b/services/agents/app/replay/normalize.py
new file mode 100644
index 000000000..9f7149339
--- /dev/null
+++ b/services/agents/app/replay/normalize.py
@@ -0,0 +1,170 @@
+"""Turn a closed vendor finding into the message production triage consumes.
+
+Gap-closure Phase 1.2.
+
+The plan's requirement is "normalize each finding with the same connector
+``normalize()`` production uses", and the reason is the one Phase 1.1 wrote
+into ``ClosedFinding.raw``: grading the agent on an input shape it never sees
+in production measures a pipeline nobody runs.
+
+Where the boundary actually falls
+---------------------------------
+Production's chain is::
+
+ connector.fetch_alerts -> connector.normalize -> services/ingest (OCSF)
+ -> Kafka raw_events -> services/fusion (promote + fuse)
+ -> aisoc.alerts.fused -> build_state -> triage
+
+This module covers the first and last links and is explicit that it does not
+cover the middle. :class:`ConnectorNormalizer` calls the **real** connector
+object, duck-typed on the one method connectors expose for this
+(``normalize(raw) -> dict``), so there is no second copy of any vendor's field
+mapping here. :func:`to_fused_envelope` then builds the
+``aisoc.alerts.fused`` message that ``build_state`` reads.
+
+What is missing between those two is everything fusion adds on live traffic:
+correlation across events, the fused confidence score, the deterministic
+narrative, and entity resolution. A replay is therefore a measurement of
+triage on a single normalized finding, not of the whole pipeline, and
+:data:`ENVELOPE_LIMITS` says so in the report rather than in a comment nobody
+reads. Calling it a full-pipeline number would be the more flattering claim
+and the false one.
+
+Why the normalizer is injected rather than imported
+---------------------------------------------------
+The connector classes live in ``services/connectors`` and this is
+``services/agents``. They are separate deployables with separate lockfiles and
+neither can import the other. So the port is what this module defines, and the
+caller supplies the connector: the CLI and the evaluation job both run where a
+checkout is present and can hand in the real class.
+
+There is deliberately no fallback normalizer. A replay that cannot reach the
+production mapping raises :class:`NormalizerUnavailable` and names what is
+missing. The alternative, a "close enough" mapping written here, is exactly
+the second implementation the plan forbids, and it would be invisible in the
+report because its output has the same shape as the real thing.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Mapping
+from typing import Any, Protocol, runtime_checkable
+
+from app.replay.findings import HistoricalFinding
+
+__all__ = [
+ "ENVELOPE_LIMITS",
+ "ConnectorNormalizer",
+ "FindingNormalizer",
+ "NormalizerUnavailable",
+ "to_fused_envelope",
+]
+
+#: Published in every replay report's method section. These are the fusion
+#: enrichments a live alert carries and a replayed finding does not.
+ENVELOPE_LIMITS: tuple[str, ...] = (
+ "Correlation across related events is not applied: each finding is replayed on its own.",
+ "The fused confidence score is absent, so triage sees no prior fusion confidence.",
+ "The deterministic correlation narrative is absent.",
+ "Entity resolution and graph context from the live pipeline are absent.",
+)
+
+
+class NormalizerUnavailable(RuntimeError):
+ """No production connector normalizer could be reached for a vendor.
+
+ Raised rather than substituted. A replay run that silently normalized its
+ own way would publish a number about a pipeline the product does not have.
+ """
+
+
+@runtime_checkable
+class FindingNormalizer(Protocol):
+ """The one method every connector in this tree exposes for this job."""
+
+ def normalize(self, raw: dict[str, Any]) -> dict[str, Any]:
+ """Map a raw vendor row to the canonical connector envelope."""
+
+
+class ConnectorNormalizer:
+ """Adapter over a real connector instance.
+
+ Holds the connector rather than subclassing it, so the object graded
+ against is the object production constructs. ``connector_id`` is read off
+ it for provenance: a report that says which connector produced its input
+ can be checked, and one that says "splunk" because a caller typed it
+ cannot.
+ """
+
+ def __init__(self, connector: FindingNormalizer, *, connector_id: str | None = None) -> None:
+ if not hasattr(connector, "normalize"):
+ raise NormalizerUnavailable(f"{type(connector).__name__} has no normalize(); replay will not substitute its own mapping")
+ self._connector = connector
+ self.connector_id = connector_id or str(getattr(connector, "connector_id", "") or "")
+
+ def normalize(self, raw: dict[str, Any]) -> dict[str, Any]:
+ return self._connector.normalize(dict(raw))
+
+
+def _first(row: Mapping[str, Any], *keys: str) -> Any:
+ for key in keys:
+ value = row.get(key)
+ if value not in (None, ""):
+ return value
+ return None
+
+
+def to_fused_envelope(
+ finding: HistoricalFinding,
+ normalized: Mapping[str, Any],
+ *,
+ tenant_id: str,
+ connector_id: str = "",
+) -> dict[str, Any]:
+ """Build the ``aisoc.alerts.fused`` message ``build_state`` consumes.
+
+ Field selection follows ``build_state`` exactly: every key set here is one
+ that function reads, and nothing is set that it ignores. The IOC fields
+ are read from the connector envelope first and from the untouched vendor
+ row second, because a connector lifts only the fields it has a canonical
+ home for and the rest stay in ``raw_event``.
+
+ ``confidence_score`` is left unset on purpose. Fusion computes it from
+ correlated evidence that a single replayed finding does not have, and
+ inventing a number for it would put a fabricated confidence into the
+ prompt that decides the verdict being graded.
+ """
+ raw_event = normalized.get("raw_event")
+ vendor_row: Mapping[str, Any] = raw_event if isinstance(raw_event, Mapping) else finding.raw
+
+ def pick(*keys: str) -> Any:
+ return _first(normalized, *keys) or _first(vendor_row, *keys)
+
+ alert: dict[str, Any] = {
+ "id": finding.finding_id,
+ "title": normalized.get("title") or finding.title,
+ "rule_id": finding.rule_id or pick("rule_id", "search_name"),
+ "rule_name": pick("rule_name", "search_name", "title"),
+ "severity": normalized.get("severity") or finding.severity,
+ "src_ip": pick("src_ip", "src"),
+ "dst_ip": pick("dst_ip", "dest", "dest_ip"),
+ "hostname": pick("hostname", "host", "dvc"),
+ "username": pick("username", "user", "src_user"),
+ "file_hash": pick("file_hash", "file_hash_sha256", "sha256"),
+ "domain": pick("domain"),
+ "url": pick("url"),
+ "mitre_techniques": pick("mitre_techniques", "annotations.mitre_attack") or [],
+ "risk_score": pick("risk_score") or 0.0,
+ "raw_event": dict(vendor_row),
+ "connector_id": connector_id,
+ "connector_type": finding.vendor,
+ "source_event_ids": [finding.finding_id] if finding.finding_id else [],
+ "tenant_id": tenant_id,
+ }
+ return {
+ "id": finding.finding_id,
+ "alert_row_id": finding.finding_id,
+ "tenant_id": tenant_id,
+ "incident_id": finding.finding_id,
+ "alert": alert,
+ }
diff --git a/services/agents/app/replay/runner.py b/services/agents/app/replay/runner.py
new file mode 100644
index 000000000..276d26b5e
--- /dev/null
+++ b/services/agents/app/replay/runner.py
@@ -0,0 +1,354 @@
+"""Replay a customer's closed findings through production triage, writing nothing.
+
+Gap-closure Phase 1.2.
+
+The whole point of this module is what it does *not* contain. There is no
+triage here. :class:`ReplayRunner` builds the message production builds, hands
+it to :meth:`FusedAlertTriageWorker.triage` (the same method the Kafka
+consumer calls on every live alert), and records what comes back. Swapping the
+persistence sink is the only difference between a replay and a live triage,
+and that sink is a constructor argument on the worker rather than a branch
+inside it, so there is no "if replay" anywhere on the hot path to drift.
+
+That matters more than it sounds. The measurement a reader cares about is "how
+does the product do on my alerts". A second triage implementation would answer
+"how does the evaluation harness do on my alerts", and the two diverge the
+first time anybody changes a prompt.
+
+The split
+---------
+Findings are ordered by close time and cut at a fraction (default 0.7). The cut
+is by **time**, not by row count, so the test window is a contiguous period
+rather than a sample: an operator asking "would this have worked last quarter"
+gets an answer about last quarter.
+
+Only the test window is replayed. The train window's job is to fix the split
+point and to describe the world triage is allowed to know about, and replaying
+it would place 70% more model calls to produce decisions nothing grades.
+
+Priors are not seeded from the train window's labels
+----------------------------------------------------
+:class:`ContextSnapshot` accepts priors and the runner uses them, but nothing
+here manufactures them out of the analyst labels in the train window. It
+would be easy and it would inflate the result: an outcome prior authored by a
+human suppresses a matching repeat immediately and without re-triage, so
+seeding priors from ground truth would score the ground truth. An operator who
+wants their real priors passes their real priors.
+"""
+
+from __future__ import annotations
+
+import time
+from collections.abc import Sequence
+from dataclasses import dataclass, field
+from datetime import datetime
+from typing import Any
+
+import structlog
+
+from app.agents.dispositions import normalize_disposition
+from app.replay.findings import HistoricalFinding
+from app.replay.normalize import ENVELOPE_LIMITS, FindingNormalizer, NormalizerUnavailable, to_fused_envelope
+from app.replay.shadow import ContextSnapshot, FrozenTriageContextReader, ShadowTriageWriter
+from app.workers.business_context import BusinessContextApplier
+from app.workers.fused_alert_consumer import FusedAlertTriageWorker
+
+logger = structlog.get_logger()
+
+__all__ = [
+ "DEFAULT_TRAIN_FRACTION",
+ "ReplayDecision",
+ "ReplayRun",
+ "ReplayRunner",
+ "TimeSplit",
+ "split_by_time",
+]
+
+#: 70/30, as the plan specifies. Exposed as a constant so the report can state
+#: the value it used rather than the value someone remembers the default being.
+DEFAULT_TRAIN_FRACTION = 0.7
+
+
+@dataclass(frozen=True)
+class TimeSplit:
+ """Where history was cut, and what fell either side."""
+
+ split_at: datetime
+ train: tuple[HistoricalFinding, ...]
+ test: tuple[HistoricalFinding, ...]
+ train_fraction: float
+
+ def as_method_note(self) -> dict[str, Any]:
+ return {
+ "split_at": self.split_at.isoformat(),
+ "train_fraction": self.train_fraction,
+ "train_findings": len(self.train),
+ "test_findings": len(self.test),
+ "train_window_start": self.train[0].closed_at.isoformat() if self.train else None,
+ "test_window_end": self.test[-1].closed_at.isoformat() if self.test else None,
+ }
+
+
+def split_by_time(
+ findings: Sequence[HistoricalFinding],
+ *,
+ train_fraction: float = DEFAULT_TRAIN_FRACTION,
+) -> TimeSplit:
+ """Order by close time and cut so the test window is the later period.
+
+ Ties at the boundary go to the **train** side. A finding closed at the
+ exact split instant is one the frozen context could legitimately have
+ known about, and putting it in the test window would ask the agent about
+ an alert whose answer is inside its own frozen memory.
+
+ Sorting is by ``(closed_at, finding_id)``. Close times collide routinely
+ when an analyst bulk-closes a queue, and a sort that is not total makes
+ the split, and therefore the report, depend on input order.
+ """
+ if not 0.0 < train_fraction < 1.0:
+ raise ValueError(f"train_fraction must be strictly between 0 and 1, got {train_fraction}")
+ ordered = sorted(findings, key=lambda f: (f.closed_at, f.finding_id))
+ if not ordered:
+ raise ValueError("cannot split an empty history")
+
+ index = max(1, min(len(ordered) - 1, int(round(len(ordered) * train_fraction))))
+ split_at = ordered[index - 1].closed_at
+ train = [f for f in ordered if f.closed_at <= split_at]
+ test = [f for f in ordered if f.closed_at > split_at]
+ return TimeSplit(
+ split_at=split_at,
+ train=tuple(train),
+ test=tuple(test),
+ train_fraction=train_fraction,
+ )
+
+
+@dataclass
+class ReplayDecision:
+ """One shadow triage decision, with everything needed to grade or audit it."""
+
+ finding_id: str
+ vendor: str
+ rule_id: str | None
+ closed_at: str
+
+ #: The analyst's label. ``unlabeled`` means excluded from accuracy.
+ expected_disposition: str
+ vendor_disposition: str
+ labelled: bool
+
+ #: What triage said, mapped onto the canonical taxonomy the analyst label
+ #: uses. Deterministic triage speaks a second dialect (``likely_benign``,
+ #: ``confirmed_incident``), so a raw comparison would score every
+ #: deterministic verdict as wrong against a vendor label that is already
+ #: canonical. The mapping is ``app.agents.dispositions``, the same one the
+ #: writeback and the memory store use, so replay cannot invent a third.
+ verdict: str | None = None
+
+ #: Exactly what triage emitted, before normalisation. Kept because a
+ #: verdict the taxonomy does not recognise is a finding about the product
+ #: and must not be laundered into a canonical one by the grader.
+ verdict_raw: str | None = None
+
+ confidence: float = 0.0
+ tier: str = ""
+
+ #: Reasoning, kept verbatim. Hallucination scoring reads indicators out
+ #: of this and checks them against the evidence the agent was given.
+ findings: list[str] = field(default_factory=list)
+ confidence_basis: list[str] = field(default_factory=list)
+
+ #: The evidence handed to triage, so a disputed score can be re-derived.
+ evidence: dict[str, Any] = field(default_factory=dict)
+
+ tool_calls: int = 0
+ resolved_models: list[str] = field(default_factory=list)
+ tokens: int = 0
+ measured_usd: float | None = None
+ estimated_usd: float | None = None
+ unpriced_calls: int = 0
+ latency_ms: int = 0
+
+ #: Set when triage refused the finding outright, which is neither a
+ #: verdict nor an abstention and must not be scored as either.
+ error: str | None = None
+
+ def as_dict(self) -> dict[str, Any]:
+ return {
+ "finding_id": self.finding_id,
+ "vendor": self.vendor,
+ "rule_id": self.rule_id,
+ "closed_at": self.closed_at,
+ "expected_disposition": self.expected_disposition,
+ "vendor_disposition": self.vendor_disposition,
+ "labelled": self.labelled,
+ "verdict": self.verdict,
+ "verdict_raw": self.verdict_raw,
+ "confidence": self.confidence,
+ "tier": self.tier,
+ "findings": list(self.findings),
+ "confidence_basis": list(self.confidence_basis),
+ "evidence": self.evidence,
+ "tool_calls": self.tool_calls,
+ "resolved_models": list(self.resolved_models),
+ "tokens": self.tokens,
+ "measured_usd": self.measured_usd,
+ "estimated_usd": self.estimated_usd,
+ "unpriced_calls": self.unpriced_calls,
+ "latency_ms": self.latency_ms,
+ "error": self.error,
+ }
+
+
+@dataclass
+class ReplayRun:
+ """A completed replay: the decisions, the split, and how it was produced."""
+
+ decisions: list[ReplayDecision]
+ split: TimeSplit
+ snapshot: ContextSnapshot
+ writes_attempted: dict[str, int]
+ tenant_id: str
+ connector_id: str
+
+ def method(self) -> dict[str, Any]:
+ """Everything a reader needs to decide whether to believe the numbers."""
+ return {
+ "tenant_id": self.tenant_id,
+ "connector_id": self.connector_id,
+ "split": self.split.as_method_note(),
+ "frozen_context": self.snapshot.as_method_note(),
+ "shadow_writes_attempted": dict(self.writes_attempted),
+ "envelope_limits": list(ENVELOPE_LIMITS),
+ }
+
+
+class ReplayRunner:
+ """Drives the production triage path over a frozen historical window."""
+
+ def __init__(
+ self,
+ *,
+ normalizer: FindingNormalizer,
+ tenant_id: str,
+ connector_id: str = "",
+ business_context: BusinessContextApplier | None = None,
+ ) -> None:
+ if normalizer is None: # pragma: no cover - defensive, the type says so
+ raise NormalizerUnavailable("replay requires the production connector normalizer")
+ self._normalizer = normalizer
+ self._tenant_id = tenant_id
+ self._connector_id = connector_id or getattr(normalizer, "connector_id", "") or ""
+ self._business_context = business_context
+
+ async def run(
+ self,
+ findings: Sequence[HistoricalFinding],
+ *,
+ snapshot: ContextSnapshot | None = None,
+ train_fraction: float = DEFAULT_TRAIN_FRACTION,
+ ) -> ReplayRun:
+ """Split, freeze, and replay the test window."""
+ split = split_by_time(findings, train_fraction=train_fraction)
+ frozen = snapshot or ContextSnapshot(split_at=split.split_at)
+ if frozen.split_at != split.split_at:
+ raise ValueError(
+ f"context snapshot was captured at {frozen.split_at.isoformat()} but the split is at "
+ f"{split.split_at.isoformat()}; a snapshot taken at a different instant is not a freeze"
+ )
+
+ writer = ShadowTriageWriter()
+ reader = FrozenTriageContextReader(frozen)
+ # The production worker, constructed exactly as the Kafka consumer
+ # constructs it apart from the two sinks. `bootstrap_servers` is
+ # required by the signature and unused: `triage()` never touches the
+ # consumer, and `start()` is not called.
+ worker = FusedAlertTriageWorker(
+ bootstrap_servers="",
+ business_context=self._business_context,
+ writer=writer,
+ context_reader=reader,
+ )
+
+ decisions: list[ReplayDecision] = []
+ for finding in split.test:
+ decisions.append(await self._replay_one(worker, finding))
+
+ logger.info(
+ "replay.completed",
+ tenant_id=self._tenant_id,
+ graded=sum(1 for d in decisions if d.labelled and d.error is None),
+ decisions=len(decisions),
+ writes_attempted=writer.total_writes,
+ )
+ return ReplayRun(
+ decisions=decisions,
+ split=split,
+ snapshot=frozen,
+ writes_attempted=dict(writer.calls),
+ tenant_id=self._tenant_id,
+ connector_id=self._connector_id,
+ )
+
+ async def _replay_one(self, worker: FusedAlertTriageWorker, finding: HistoricalFinding) -> ReplayDecision:
+ decision = ReplayDecision(
+ finding_id=finding.finding_id,
+ vendor=finding.vendor,
+ rule_id=finding.rule_id,
+ closed_at=finding.closed_at.isoformat(),
+ expected_disposition=finding.disposition,
+ vendor_disposition=finding.vendor_disposition,
+ labelled=finding.is_labelled,
+ )
+
+ try:
+ normalized = self._normalizer.normalize(dict(finding.raw))
+ except Exception as exc: # noqa: BLE001 — one bad row must not end the run
+ decision.error = f"normalize failed: {type(exc).__name__}: {exc}"
+ return decision
+
+ envelope = to_fused_envelope(
+ finding,
+ normalized,
+ tenant_id=self._tenant_id,
+ connector_id=self._connector_id,
+ )
+ decision.evidence = dict(envelope.get("alert") or {})
+
+ started = time.monotonic()
+ try:
+ summary = await worker.triage(envelope)
+ except Exception as exc: # noqa: BLE001 — recorded as a refusal, not a verdict
+ decision.error = f"triage failed: {type(exc).__name__}: {exc}"
+ decision.latency_ms = int(round((time.monotonic() - started) * 1000))
+ return decision
+ decision.latency_ms = int(round((time.monotonic() - started) * 1000))
+
+ if not isinstance(summary, dict):
+ decision.error = "triage returned no summary for this finding"
+ return decision
+
+ raw_verdict = summary.get("verdict")
+ decision.verdict_raw = raw_verdict
+ # An unrecognised verdict keeps its own spelling rather than falling
+ # back to a canonical one. The grader gives it its own confusion-matrix
+ # column, which is what makes it visible instead of scored as
+ # something the agent did not say.
+ decision.verdict = normalize_disposition(raw_verdict, default="") or (str(raw_verdict) if raw_verdict else None)
+ decision.confidence = float(summary.get("confidence") or 0.0)
+ decision.tier = str(summary.get("tier") or ("suppressed" if summary.get("suppressed") else ""))
+ decision.findings = [str(f) for f in (summary.get("findings") or [])]
+ decision.confidence_basis = [str(b) for b in (summary.get("confidence_basis") or [])]
+
+ cost = summary.get("cost") or {}
+ decision.tokens = int(cost.get("tokens") or 0)
+ decision.measured_usd = cost.get("measured_usd")
+ decision.estimated_usd = cost.get("estimated_usd")
+ decision.unpriced_calls = int(cost.get("unpriced_calls") or 0)
+ decision.resolved_models = [str(m) for m in (cost.get("resolved_models") or [])]
+ # Structurally zero rather than unmeasured: shadow mode declines
+ # escalation, and escalation is the only stage of this path that calls
+ # tools. Recorded so a future change that gives triage a tool shows up
+ # here as a number moving off zero.
+ decision.tool_calls = 0
+ return decision
diff --git a/services/agents/app/replay/shadow.py b/services/agents/app/replay/shadow.py
new file mode 100644
index 000000000..ea22395be
--- /dev/null
+++ b/services/agents/app/replay/shadow.py
@@ -0,0 +1,269 @@
+"""Shadow mode: the production triage path, writing nothing, reading a frozen world.
+
+Gap-closure Phase 1.2.
+
+These are the two halves of the replay's honesty, and they fail in opposite
+directions.
+
+:class:`ShadowTriageWriter` implements every method of
+``app.workers.triage_persistence.TriageWriter`` and performs none of them. It
+counts each call instead, so "wrote nothing" is a number a test can assert on
+rather than an absence a test has to prove. An absence proves nothing: a
+replay that never reached the writeback branch and a replay whose writeback
+was suppressed look identical from outside.
+
+:class:`FrozenTriageContextReader` implements the read port against a snapshot
+captured once, before the test window is replayed, and never refreshed. This is
+the leak the plan cares about. Without it the sequence is:
+
+ triage alert 140 -> verdict benign -> prior written -> triage alert 141
+ -> reads the prior -> auto-suppressed -> graded "correct"
+
+and the evaluation has graded the agent on an answer it supplied itself three
+seconds earlier. Freezing the reads closes the second half of that; the shadow
+writer closes the first. Both are needed: a frozen reader over a live writer
+still pollutes the store for the *next* run, and a null writer over a live
+reader still picks up whatever the console wrote while the replay was running.
+
+On the timestamps that are not there
+------------------------------------
+:func:`capture_context` drops any statement or prior carrying a recorded time
+after the split. Organisation-memory statements as served by
+``GET /feedback/context-statements`` carry no creation time at all: the query
+in ``app/services/analyst_feedback.py`` selects ``statement``, ``reason_code``,
+``scope``, ``scope_value``, ``observations`` and ``expires_at``, and nothing
+else. So for those rows the filter has nothing to test and they are kept.
+
+That is a real limit and it is counted rather than hidden:
+:attr:`ContextSnapshot.undated_statements` travels into the report, so a reader
+sees how much of the frozen context could actually be checked against the
+split. Silently dropping them would understate the context production gets;
+silently keeping them while claiming a clean point-in-time freeze would
+overstate the guarantee.
+"""
+
+from __future__ import annotations
+
+from collections import Counter
+from collections.abc import Iterable, Mapping
+from dataclasses import dataclass, field
+from datetime import UTC, datetime
+from typing import Any
+
+__all__ = [
+ "ContextSnapshot",
+ "FrozenTriageContextReader",
+ "ShadowTriageWriter",
+ "capture_context",
+]
+
+#: Keys that may carry a row's recorded time, newest-authority first. Both
+#: the outcome-prior shape (``first_seen`` / ``last_seen``) and the generic
+#: SQL shapes are covered so one function serves both stores.
+_TIME_KEYS: tuple[str, ...] = ("last_seen", "updated_at", "created_at", "first_seen", "recorded_at")
+
+
+class ShadowTriageWriter:
+ """A :class:`TriageWriter` that records what production would have written.
+
+ Every method returns the "nothing happened" value its live counterpart
+ returns on a no-op: ``None`` from :meth:`raise_approval` means no approval
+ id, and ``None`` from :meth:`write_back_disposition` means nothing was
+ attempted. The worker already handles both, because both occur in
+ production whenever the relevant feature is switched off, so shadow mode
+ exercises paths the worker takes anyway rather than novel ones.
+ """
+
+ def __init__(self) -> None:
+ self.calls: Counter[str] = Counter()
+
+ @property
+ def escalation_allowed(self) -> bool:
+ return False
+
+ @property
+ def persists_cost(self) -> bool:
+ return False
+
+ @property
+ def total_writes(self) -> int:
+ """How many writes production would have made. Zero is the claim under test."""
+ return sum(self.calls.values())
+
+ async def persist_auto_triage(self, **fields: Any) -> None:
+ self.calls["persist_auto_triage"] += 1
+
+ async def record_outcome(
+ self,
+ tenant_id: str,
+ signature: str,
+ *,
+ disposition: str,
+ confidence: float,
+ author: str,
+ alert_id: Any = None,
+ ) -> None:
+ self.calls["record_outcome"] += 1
+
+ async def record_suppression(
+ self,
+ *,
+ tenant_ref: str,
+ signature: str,
+ alert_id: Any,
+ disposition: str,
+ prior_author: str,
+ ) -> None:
+ self.calls["record_suppression"] += 1
+
+ async def raise_approval(self, **fields: Any) -> Any:
+ self.calls["raise_approval"] += 1
+ return None
+
+ async def write_back_disposition(
+ self,
+ *,
+ tenant_id: str,
+ alert_id: str,
+ disposition: str,
+ confidence: float | None = None,
+ rationale: str = "",
+ ) -> dict[str, Any] | None:
+ self.calls["write_back_disposition"] += 1
+ return None
+
+ def cache_verdict(
+ self,
+ governor: Any,
+ tenant_id: str,
+ fingerprint: str,
+ verdict: dict[str, Any],
+ *,
+ usd: float,
+ tokens: int,
+ ) -> None:
+ # Declining this one is not only about the write. The governor's cache
+ # is keyed on the evidence fingerprint, so a cached verdict is served
+ # back to the next alert with identical evidence as DEDUPLICATED. In a
+ # replay that is the test window answering itself through a third
+ # store, and it would not show up as a memory prior or a ledger row.
+ self.calls["cache_verdict"] += 1
+
+
+@dataclass(frozen=True)
+class ContextSnapshot:
+ """Durable state as it stood at the split point.
+
+ ``priors`` is keyed by evidence signature, which is what
+ ``CostGovernor.evidence_fingerprint`` produces and what the worker looks
+ up. Capturing by signature rather than by alert means the snapshot answers
+ the question the worker actually asks.
+ """
+
+ split_at: datetime
+ statements: tuple[Mapping[str, Any], ...] = ()
+ priors: Mapping[str, Mapping[str, Any]] = field(default_factory=dict)
+
+ #: Statements kept because they carry no recorded time to test against the
+ #: split. Reported so the freeze is not claimed to be tighter than it is.
+ undated_statements: int = 0
+
+ #: Rows dropped for carrying a time after the split. A non-zero value is
+ #: the freeze doing its job and is worth seeing.
+ dropped_statements: int = 0
+ dropped_priors: int = 0
+
+ def as_method_note(self) -> dict[str, Any]:
+ """The provenance block the replay report publishes about its own context."""
+ return {
+ "split_at": self.split_at.isoformat(),
+ "statements_frozen": len(self.statements),
+ "statements_without_timestamp": self.undated_statements,
+ "statements_dropped_after_split": self.dropped_statements,
+ "priors_frozen": len(self.priors),
+ "priors_dropped_after_split": self.dropped_priors,
+ }
+
+
+def _recorded_time(row: Mapping[str, Any]) -> datetime | None:
+ for key in _TIME_KEYS:
+ value = row.get(key)
+ if isinstance(value, datetime):
+ return value if value.tzinfo else value.replace(tzinfo=UTC)
+ if isinstance(value, str) and value.strip():
+ try:
+ parsed = datetime.fromisoformat(value.strip().replace("Z", "+00:00"))
+ except ValueError:
+ continue
+ return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC)
+ return None
+
+
+def capture_context(
+ *,
+ split_at: datetime,
+ statements: Iterable[Mapping[str, Any]] = (),
+ priors: Mapping[str, Mapping[str, Any]] | None = None,
+) -> ContextSnapshot:
+ """Freeze organisation memory and outcome priors as of ``split_at``."""
+ kept: list[Mapping[str, Any]] = []
+ undated = 0
+ dropped_statements = 0
+ for row in statements:
+ recorded = _recorded_time(row)
+ if recorded is None:
+ undated += 1
+ kept.append(dict(row))
+ continue
+ if recorded > split_at:
+ dropped_statements += 1
+ continue
+ kept.append(dict(row))
+
+ kept_priors: dict[str, Mapping[str, Any]] = {}
+ dropped_priors = 0
+ for signature, prior in (priors or {}).items():
+ recorded = _recorded_time(prior)
+ if recorded is not None and recorded > split_at:
+ dropped_priors += 1
+ continue
+ kept_priors[str(signature)] = dict(prior)
+
+ return ContextSnapshot(
+ split_at=split_at,
+ statements=tuple(kept),
+ priors=kept_priors,
+ undated_statements=undated,
+ dropped_statements=dropped_statements,
+ dropped_priors=dropped_priors,
+ )
+
+
+class FrozenTriageContextReader:
+ """A :class:`TriageContextReader` served entirely from a snapshot.
+
+ Holds no client, no pool and no URL, so there is nothing for it to refresh
+ from even if a caller wanted it to. That is deliberate: a reader that
+ *could* reach the live store would eventually be given a cache TTL, and a
+ TTL is a refresh with a delay on it.
+ """
+
+ def __init__(self, snapshot: ContextSnapshot) -> None:
+ self._snapshot = snapshot
+ self.reads: Counter[str] = Counter()
+
+ @property
+ def snapshot(self) -> ContextSnapshot:
+ return self._snapshot
+
+ async def fetch_statements(self, tenant_id: str | None) -> list[dict[str, Any]]:
+ self.reads["fetch_statements"] += 1
+ # Copied per call. The worker hands these to the prompt builder and
+ # a shared mutable list would let one alert's triage edit what the
+ # next one sees, which is the same leak by a shorter route.
+ return [dict(row) for row in self._snapshot.statements]
+
+ async def lookup_prior(self, tenant_id: str, signature: str) -> dict[str, Any] | None:
+ self.reads["lookup_prior"] += 1
+ prior = self._snapshot.priors.get(signature)
+ return dict(prior) if prior is not None else None
diff --git a/services/agents/app/workers/fused_alert_consumer.py b/services/agents/app/workers/fused_alert_consumer.py
index 35948fdd2..95b16bfc2 100644
--- a/services/agents/app/workers/fused_alert_consumer.py
+++ b/services/agents/app/workers/fused_alert_consumer.py
@@ -45,19 +45,23 @@ from app.agents.auto_triage_agent import run_auto_triage
from app.agents.dispositions import AUTO_CLOSEABLE_DISPOSITIONS, NEEDS_REVIEW, normalize_disposition
from app.agents.triage_agent import run_triage
from app.confidence.groundedness import score_groundedness
-from app.context.organisation_memory import fetch_statements
from app.core.cost_governor import Decision, get_governor
from app.core.cost_telemetry import CostSummary, CostTracker
from app.graph.runner import default_budget, run_escalation
from app.investigator import ledger as ledger_module
-from app.investigator import siem_writeback
from app.investigator.bundle_prompt import prefetch_context_bundle_dict
from app.llm.factory import llm_override
-from app.memory.outcomes import AI, lookup_prior, record_outcome, should_auto_suppress
+from app.memory.outcomes import AI, should_auto_suppress
from app.models.state import AgentStatus, InvestigationState
from app.routing.model_router import is_deterministic_mode
from app.security.llm_resolver import resolve_llm_config
from app.workers.business_context import BusinessContextApplier
+from app.workers.triage_persistence import (
+ LiveTriageContextReader,
+ LiveTriageWriter,
+ TriageContextReader,
+ TriageWriter,
+)
logger = structlog.get_logger()
@@ -155,6 +159,26 @@ def _coerce_uuid(value: Any, *, fallback: str) -> uuid.UUID:
return uuid.uuid5(_NAMESPACE, str(value or fallback))
+def _cost_payload(cost: CostSummary, tokens: int) -> dict[str, Any]:
+ """The spend a triage summary reports, with the provenance of every figure.
+
+ Added for Phase 1.2: replay has to record measured cost, tokens and the
+ model per decision, and the only place that knows them is this worker.
+ ``measured_usd`` stays ``None`` when no call reported a cost, because a
+ replay report that printed 0.0 there would be claiming a measurement of
+ free rather than the absence of one.
+ """
+ return {
+ "measured_usd": cost.measured_usd,
+ "measured_calls": cost.measured_calls,
+ "estimated_usd": cost.estimated_usd,
+ "estimated_calls": cost.estimated_calls,
+ "unpriced_calls": cost.unpriced_calls,
+ "resolved_models": list(cost.resolved_models),
+ "tokens": tokens,
+ }
+
+
def _rationale_of(state: InvestigationState) -> str:
"""Human-readable rationale for the verdict, for the ledger + alerts row."""
if state.confidence_basis:
@@ -224,6 +248,17 @@ def build_state(message: dict[str, Any]) -> InvestigationState | None:
class FusedAlertTriageWorker:
"""Consumes ``aisoc.alerts.fused`` and auto-triages each alert (copilot)."""
+ #: Production sinks, declared on the class rather than only assigned in
+ #: ``__init__``. Several tests build this worker with ``__new__`` to drive
+ #: one method without a broker, and a sink that existed only as an instance
+ #: attribute would leave those objects with no persistence at all —
+ #: silently, since every write here is fail-soft. A class-level default
+ #: makes "the default is production" true however the object was made.
+ #: Both implementations are stateless (``__slots__ = ()``), so one shared
+ #: instance carries nothing between workers.
+ _writer: TriageWriter = LiveTriageWriter()
+ _reader: TriageContextReader = LiveTriageContextReader()
+
def __init__(
self,
*,
@@ -233,10 +268,18 @@ class FusedAlertTriageWorker:
dlq_topic: str | None = None,
max_attempts: int = _MAX_ATTEMPTS,
business_context: BusinessContextApplier | None = None,
+ writer: TriageWriter | None = None,
+ context_reader: TriageContextReader | None = None,
) -> None:
self._bootstrap = bootstrap_servers
self._topic = topic
self._group_id = group_id
+ # Phase 1.2 — persistence is injected so replay evaluation can run this
+ # exact path writing nothing. The defaults are what the worker did
+ # inline before, so a production deployment that passes neither is
+ # byte-for-byte the same behaviour.
+ self._writer: TriageWriter = writer or LiveTriageWriter()
+ self._reader: TriageContextReader = context_reader or LiveTriageContextReader()
self._dlq_topic = dlq_topic or os.getenv("KAFKA_TOPIC_ALERTS_FUSED_DLQ", f"{topic}.dlq")
self._max_attempts = max(1, max_attempts)
self._consumer: Any | None = None
@@ -418,6 +461,7 @@ class FusedAlertTriageWorker:
return {
"incident_id": str(message.get("incident_id") or message.get("id") or ""),
"suppressed": True,
+ "cost": _cost_payload(CostSummary(), 0),
"business_context_rules": bc_matched,
"response_dispatched": False,
}
@@ -463,11 +507,15 @@ class FusedAlertTriageWorker:
# is on the worker's timeline and is skipped entirely when no
# LLM call is going to happen. Never raises; an empty list just
# means the prompt is what it was before.
- state.organisation_memory = await fetch_statements(str(state.tenant_id))
+ state.organisation_memory = await self._reader.fetch_statements(str(state.tenant_id))
# Bind a CostTracker so every LLM call on this path records its
# token/cost (safe_ainvoke -> record_llm_call) — previously the
# highest-volume LLM spend was recorded as $0 and invisible.
- async with CostTracker(run_id=str(state.run_id), tenant_id=str(state.tenant_id)) as tracker:
+ async with CostTracker(
+ run_id=str(state.run_id),
+ tenant_id=str(state.tenant_id),
+ persist=self._writer.persists_cost,
+ ) as tracker:
if use_llm and cfg is not None:
# Route the LLM call through the tenant's BYOK key/model so
# auto-triage actually honours per-tenant credentials.
@@ -517,7 +565,8 @@ class FusedAlertTriageWorker:
# spent. An unmeasured call contributes no dollars — it still
# contributes tokens, which is the cap that can be enforced
# honestly without a price.
- governor.record_verdict(
+ self._writer.cache_verdict(
+ governor,
str(state.tenant_id),
fingerprint,
{"verdict": verdict, "confidence": confidence},
@@ -541,7 +590,7 @@ class FusedAlertTriageWorker:
# autonomous closures compound (a later identical alert can suppress).
if _memory_writeback_enabled() and verdict:
with contextlib.suppress(Exception):
- await record_outcome(
+ await self._writer.record_outcome(
str(state.tenant_id),
fingerprint,
disposition=str(verdict),
@@ -578,6 +627,9 @@ class FusedAlertTriageWorker:
"verdict": verdict,
"confidence": confidence,
"tier": tier,
+ "cost": _cost_payload(cost, tokens),
+ "findings": list(state.findings),
+ "confidence_basis": list(state.confidence_basis),
"business_context_rules": bc_matched,
# Copilot default: triage is read-only, response requires approval.
"response_dispatched": False,
@@ -608,7 +660,7 @@ class FusedAlertTriageWorker:
if not alert_id or not verdict:
return None
try:
- report = await siem_writeback.write_back_disposition(
+ report = await self._writer.write_back_disposition(
tenant_id=str(state.tenant_id),
alert_id=alert_id,
disposition=str(verdict),
@@ -641,7 +693,7 @@ class FusedAlertTriageWorker:
if not action.requires_approval:
continue
risk = action.risk_level.value if hasattr(action.risk_level, "value") else str(action.risk_level)
- approval_id = await ledger_module.raise_approval(
+ approval_id = await self._writer.raise_approval(
tenant_ref=str(state.tenant_id),
run_id=state.run_id,
alert_id=(state.raw_alert or {}).get("id"),
@@ -673,7 +725,7 @@ class FusedAlertTriageWorker:
None. Best-effort: any lookup failure falls through to normal triage.
"""
try:
- prior = await lookup_prior(str(state.tenant_id), signature)
+ prior = await self._reader.lookup_prior(str(state.tenant_id), signature)
except Exception as exc: # noqa: BLE001 — memory read is advisory
logger.debug("auto_triage_worker.memory_lookup_failed", error=str(exc))
return None
@@ -699,7 +751,7 @@ class FusedAlertTriageWorker:
_METRICS["triaged"] += 1
await self._record(state, tier="memory", verdict=disposition, confidence=confidence)
with contextlib.suppress(Exception):
- await record_outcome(
+ await self._writer.record_outcome(
str(state.tenant_id),
signature,
disposition=disposition,
@@ -708,7 +760,7 @@ class FusedAlertTriageWorker:
alert_id=(state.raw_alert or {}).get("id"),
)
with contextlib.suppress(Exception):
- await ledger_module.record_suppression(
+ await self._writer.record_suppression(
tenant_ref=str(state.tenant_id),
signature=signature,
alert_id=(state.raw_alert or {}).get("id"),
@@ -729,6 +781,11 @@ class FusedAlertTriageWorker:
"verdict": disposition,
"confidence": confidence,
"tier": "memory",
+ # No model call was placed, so every figure here is a zero that was
+ # counted rather than a measurement that is missing.
+ "cost": _cost_payload(CostSummary(), 0),
+ "findings": list(state.findings),
+ "confidence_basis": list(state.confidence_basis),
"suppressed_by_memory": True,
"business_context_rules": bc_matched,
"response_dispatched": False,
@@ -743,7 +800,10 @@ class FusedAlertTriageWorker:
restart re-runs idempotently. Best-effort + budgeted — a failure or
timeout leaves the durable verdict intact and the alert escalated.
"""
- if not _escalation_enabled():
+ # Two independent switches. The env flag is the operator's; the
+ # writer's is the caller's, and a shadow run declines because every
+ # graph node records itself to the ledger.
+ if not _escalation_enabled() or not self._writer.escalation_allowed:
return
# Pre-fetch the same context an analyst-initiated investigation gets.
# The manual orchestrator has built a ContextBundle since T2.1; the
@@ -758,7 +818,11 @@ class FusedAlertTriageWorker:
raw_alert=state.raw_alert or {},
)
try:
- async with CostTracker(run_id=str(state.run_id), tenant_id=str(state.tenant_id)):
+ async with CostTracker(
+ run_id=str(state.run_id),
+ tenant_id=str(state.tenant_id),
+ persist=self._writer.persists_cost,
+ ):
await run_escalation(state, budget=default_budget(), seq_start=1)
_METRICS["escalated"] += 1
except Exception as exc: # noqa: BLE001 — escalation is best-effort over a durable verdict
@@ -875,7 +939,7 @@ class FusedAlertTriageWorker:
}
for a in state.proposed_actions
]
- await ledger_module.persist_auto_triage(
+ await self._writer.persist_auto_triage(
run_id=state.run_id,
alert_id=(state.raw_alert or {}).get("id"),
tenant_ref=str(state.tenant_id),
diff --git a/services/agents/app/workers/triage_persistence.py b/services/agents/app/workers/triage_persistence.py
new file mode 100644
index 000000000..b968e9838
--- /dev/null
+++ b/services/agents/app/workers/triage_persistence.py
@@ -0,0 +1,279 @@
+"""Every durable effect the triage path has, behind one injectable seam.
+
+Gap-closure Phase 1.2.
+
+Why this exists
+---------------
+Replay evaluation has to run triage against a customer's closed findings and
+measure the verdicts, while writing nothing: no ledger rows, no alert rows, no
+outcome-memory priors, no approvals, no disposition pushed back into their
+SIEM. The plan is explicit that the way to get there is to refactor the
+production path so persistence is injected, *not* to write a second triage
+implementation. A second implementation would measure itself.
+
+So :class:`FusedAlertTriageWorker` no longer calls the ledger, the memory
+store, the approvals queue, the writeback client or the cost governor's
+cache directly. It calls them through :class:`TriageWriter`, whose default is
+:class:`LiveTriageWriter` and whose default does exactly what the worker used
+to do inline. The worker is one object with one code path; only the sink
+differs.
+
+The reads are a separate port on purpose
+----------------------------------------
+:class:`TriageContextReader` covers the two reads that decide a verdict from
+durable state: the tenant's compiled organisation memory and the per-signature
+outcome prior. They are split from the writes because replay needs opposite
+things from the two halves. Writes must not happen at all. Reads must happen,
+but must see the world as it stood at the train/test split, or the test window
+influences its own triage: a verdict recorded for alert 140 becomes a prior
+that suppresses alert 141, and the evaluation grades the agent on answers it
+supplied itself.
+
+Keeping them apart means the leakage test can assert each property
+independently, which is what makes a failure legible. One combined port would
+let "wrote nothing" and "read nothing new" fail as the same assertion.
+
+What is deliberately *not* here
+-------------------------------
+Business-context rules are the third frozen-at-split input, and they are
+already injected: :class:`FusedAlertTriageWorker` takes a
+``BusinessContextApplier`` in its constructor. Adding a fourth port for
+something the constructor already parameterises would be ceremony.
+
+The cost governor's dedup cache is reached through
+:meth:`TriageWriter.cache_verdict` rather than being called on the governor
+singleton, because it is a write. The governor's ``check`` and
+``evidence_fingerprint`` stay direct calls: both are pure reads, and a shadow
+run wants the real fingerprint.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Protocol, runtime_checkable
+
+from app.context import organisation_memory
+from app.investigator import ledger as ledger_module
+from app.investigator import siem_writeback
+from app.memory import outcomes as outcomes_module
+
+__all__ = [
+ "LiveTriageContextReader",
+ "LiveTriageWriter",
+ "TriageContextReader",
+ "TriageWriter",
+]
+
+
+@runtime_checkable
+class TriageWriter(Protocol):
+ """Every write :meth:`FusedAlertTriageWorker.triage` performs.
+
+ An implementation that returns without doing anything is a complete
+ shadow mode. That is the property the replay runner depends on, and
+ ``test_replay_shadow_writes_nothing`` asserts it against the real modules
+ rather than against a mock of them.
+ """
+
+ @property
+ def escalation_allowed(self) -> bool:
+ """Whether the full investigation graph may run after the verdict.
+
+ Escalation is a write: :func:`app.graph.runner.run_escalation` records
+ every node it executes to the Investigation Ledger. It is also the one
+ effect that is not reachable through a method here, because it is a
+ whole subsystem rather than a row.
+
+ Declining it does not change what replay measures. ``triage()``
+ captures ``verdict`` and ``confidence`` into locals *before* calling
+ ``_maybe_escalate`` and returns those locals, so the graph cannot
+ alter the verdict being graded. ``test_escalation_cannot_change_the_
+ graded_verdict`` pins that rather than leaving it as a reading of the
+ code.
+ """
+
+ @property
+ def persists_cost(self) -> bool:
+ """Whether the :class:`CostTracker` bound to this run may flush to the database.
+
+ Cost is still *measured* in shadow mode, because tokens and latency
+ are two of the fields the replay report has to record. Only the write
+ is declined.
+ """
+
+ async def persist_auto_triage(self, **fields: Any) -> None:
+ """Write the verdict to the ledger and the ``alerts`` row.
+
+ Keyword-only and untyped by design: this mirrors
+ :func:`app.investigator.ledger.persist_auto_triage`, which carries
+ eighteen fields and gains more as the funnel does. Restating them
+ here would create a second signature to keep in step, and the one
+ that drifts is always the copy.
+ """
+
+ async def record_outcome(
+ self,
+ tenant_id: str,
+ signature: str,
+ *,
+ disposition: str,
+ confidence: float,
+ author: str,
+ alert_id: Any = None,
+ ) -> None:
+ """Write the per-signature outcome prior that lets a repeat alert suppress."""
+
+ async def record_suppression(
+ self,
+ *,
+ tenant_ref: str,
+ signature: str,
+ alert_id: Any,
+ disposition: str,
+ prior_author: str,
+ ) -> None:
+ """Record that a repeat alert was closed from memory without re-triage."""
+
+ async def raise_approval(self, **fields: Any) -> Any:
+ """Queue an approval-requiring proposed action. Returns its id, or ``None``."""
+
+ async def write_back_disposition(
+ self,
+ *,
+ tenant_id: str,
+ alert_id: str,
+ disposition: str,
+ confidence: float | None = None,
+ rationale: str = "",
+ ) -> dict[str, Any] | None:
+ """Project this verdict onto the finding in the source SIEM."""
+
+ def cache_verdict(
+ self,
+ governor: Any,
+ tenant_id: str,
+ fingerprint: str,
+ verdict: dict[str, Any],
+ *,
+ usd: float,
+ tokens: int,
+ ) -> None:
+ """Cache the verdict for dedup and account the spend against the tenant budget."""
+
+
+@runtime_checkable
+class TriageContextReader(Protocol):
+ """The durable state a verdict is allowed to depend on.
+
+ Both methods must be total. Production's implementations already are:
+ neither ``fetch_statements`` nor ``lookup_prior`` may take triage down
+ when the store behind it is unreachable.
+ """
+
+ async def fetch_statements(self, tenant_id: str | None) -> list[dict[str, Any]]:
+ """The tenant's active organisation-memory statements."""
+
+ async def lookup_prior(self, tenant_id: str, signature: str) -> dict[str, Any] | None:
+ """The durable outcome prior for an evidence signature, or ``None``."""
+
+
+class LiveTriageWriter:
+ """The production sink. Every method is the call the worker used to make inline.
+
+ Kept as a class with no state rather than a module of functions so the
+ default is an object the worker holds, and so a reader asking "what does
+ production do differently from shadow?" gets the answer by diffing two
+ classes that implement one protocol.
+ """
+
+ __slots__ = ()
+
+ @property
+ def escalation_allowed(self) -> bool:
+ return True
+
+ @property
+ def persists_cost(self) -> bool:
+ return True
+
+ async def persist_auto_triage(self, **fields: Any) -> None:
+ await ledger_module.persist_auto_triage(**fields)
+
+ async def record_outcome(
+ self,
+ tenant_id: str,
+ signature: str,
+ *,
+ disposition: str,
+ confidence: float,
+ author: str,
+ alert_id: Any = None,
+ ) -> None:
+ await outcomes_module.record_outcome(
+ tenant_id,
+ signature,
+ disposition=disposition,
+ confidence=confidence,
+ author=author,
+ alert_id=alert_id,
+ )
+
+ async def record_suppression(
+ self,
+ *,
+ tenant_ref: str,
+ signature: str,
+ alert_id: Any,
+ disposition: str,
+ prior_author: str,
+ ) -> None:
+ await ledger_module.record_suppression(
+ tenant_ref=tenant_ref,
+ signature=signature,
+ alert_id=alert_id,
+ disposition=disposition,
+ prior_author=prior_author,
+ )
+
+ async def raise_approval(self, **fields: Any) -> Any:
+ return await ledger_module.raise_approval(**fields)
+
+ async def write_back_disposition(
+ self,
+ *,
+ tenant_id: str,
+ alert_id: str,
+ disposition: str,
+ confidence: float | None = None,
+ rationale: str = "",
+ ) -> dict[str, Any] | None:
+ return await siem_writeback.write_back_disposition(
+ tenant_id=tenant_id,
+ alert_id=alert_id,
+ disposition=disposition,
+ confidence=confidence,
+ rationale=rationale,
+ )
+
+ def cache_verdict(
+ self,
+ governor: Any,
+ tenant_id: str,
+ fingerprint: str,
+ verdict: dict[str, Any],
+ *,
+ usd: float,
+ tokens: int,
+ ) -> None:
+ governor.record_verdict(tenant_id, fingerprint, verdict, usd=usd, tokens=tokens)
+
+
+class LiveTriageContextReader:
+ """The production reads: whatever the stores hold right now."""
+
+ __slots__ = ()
+
+ async def fetch_statements(self, tenant_id: str | None) -> list[dict[str, Any]]:
+ return await organisation_memory.fetch_statements(tenant_id)
+
+ async def lookup_prior(self, tenant_id: str, signature: str) -> dict[str, Any] | None:
+ return await outcomes_module.lookup_prior(tenant_id, signature)
diff --git a/services/agents/tests/test_replay_leakage.py b/services/agents/tests/test_replay_leakage.py
new file mode 100644
index 000000000..22ad5fd66
--- /dev/null
+++ b/services/agents/tests/test_replay_leakage.py
@@ -0,0 +1,393 @@
+"""The test window must not be able to influence its own triage.
+
+Gap-closure Phase 1.5, and the test the phase is really about. A replay report
+whose test window fed itself is worse than no report: it is a number with a
+method section that reads correctly and a result that is circular.
+
+Three routes exist from a test-window decision back into a later test-window
+decision, and they run through three different stores. Each gets its own test,
+because "no leakage" failing as one assertion tells a future reader nothing
+about which of the three broke.
+
+1. **Outcome memory.** Production writes every durable verdict back as a
+ per-signature prior, and looks a prior up *before* triage. Two findings
+ with the same canonical evidence therefore chain: the first alert's verdict
+ auto-closes the second without re-triage. Under replay that is the
+ evaluation answering its own next question.
+2. **The cost governor's dedup cache.** Keyed on the same evidence
+ fingerprint, and it short-circuits earlier still, returning the cached
+ verdict as tier ``cached``. It leaves no ledger row and no memory prior, so
+ the first two defences would both report clean while this one leaked.
+3. **Organisation memory.** Compiled from analyst feedback and read into the
+ triage prompt. Nothing in the test window writes a statement directly, but
+ the read is live in production and a mid-run refresh picks up whatever the
+ console wrote while the replay was running.
+
+Every test here is checked for sensitivity
+-------------------------------------------
+A leakage test that passes because nothing happens is the most likely thing to
+ship and the least useful. So each test also runs the *unprotected*
+configuration and asserts it leaks. If a future change makes the leak
+unreachable for some unrelated reason, the sensitivity half fails and says so,
+rather than the whole file going quietly green over a property it no longer
+tests.
+"""
+
+from __future__ import annotations
+
+import uuid
+from dataclasses import dataclass
+from datetime import UTC, datetime, timedelta
+from typing import Any
+
+import pytest
+from app.agents.dispositions import BENIGN, FALSE_POSITIVE, NEEDS_REVIEW, normalize_disposition
+from app.context import organisation_memory
+from app.memory import outcomes as outcomes_module
+from app.models.state import AgentStatus, InvestigationState
+from app.replay.findings import HistoricalFinding
+from app.replay.runner import ReplayRunner
+from app.replay.shadow import ContextSnapshot, FrozenTriageContextReader, ShadowTriageWriter, capture_context
+from app.workers import fused_alert_consumer as worker_module
+from app.workers.fused_alert_consumer import FusedAlertTriageWorker
+from app.workers.triage_persistence import LiveTriageContextReader
+
+_SPLIT = datetime(2026, 5, 1, 12, 0, tzinfo=UTC)
+
+#: Two findings, different ids, identical evidence. `canonical_evidence`
+#: excludes the row id and the raw payload, so both hash to one fingerprint,
+#: which is exactly the condition every leak below needs.
+_SHARED_EVIDENCE = {"search_name": "Repeated service logon", "src": "10.9.9.9", "host": "APPSRV-01"}
+
+
+class _Normalizer:
+ connector_id = "splunk"
+
+ def normalize(self, raw: dict[str, Any]) -> dict[str, Any]:
+ return {
+ "source": "splunk",
+ "title": raw.get("search_name"),
+ "severity": "low",
+ "src_ip": raw.get("src"),
+ "hostname": raw.get("host"),
+ "raw_event": raw,
+ }
+
+
+def _twin_findings() -> list[HistoricalFinding]:
+ """Two test-window findings that share a fingerprint, closed an hour apart."""
+ return [
+ HistoricalFinding(
+ vendor="splunk",
+ finding_id=f"ES-TWIN-{index}",
+ title="Repeated service logon",
+ disposition=FALSE_POSITIVE,
+ vendor_disposition="disposition:3",
+ closed_at=_SPLIT + timedelta(hours=index + 1),
+ rule_id="rule-service-logon",
+ raw=dict(_SHARED_EVIDENCE),
+ )
+ for index in range(2)
+ ]
+
+
+def _history() -> list[HistoricalFinding]:
+ """Enough train-window rows that a 70/30 split puts both twins in the test window."""
+ train = [
+ HistoricalFinding(
+ vendor="splunk",
+ finding_id=f"ES-OLD-{index:02d}",
+ title="Historic finding",
+ disposition=FALSE_POSITIVE,
+ vendor_disposition="disposition:3",
+ closed_at=_SPLIT - timedelta(days=index + 1),
+ rule_id="rule-historic",
+ raw={"search_name": "Historic finding", "host": f"OLD-{index:02d}"},
+ )
+ for index in range(5)
+ ]
+ return [*train, *_twin_findings()]
+
+
+@dataclass
+class _StubLlmConfig:
+ """Just enough of ``LlmConfig`` for the worker to take the LLM branch."""
+
+ allowed: bool = True
+ api_key: str = "test-key"
+ base_url: str = "http://gateway"
+ model: str = "aisoc-triage"
+ api_key_from_tenant: bool = False
+ base_url_from_tenant: bool = False
+ model_from_tenant: bool = False
+
+
+# --------------------------------------------------------------------------
+# 1. Outcome memory
+# --------------------------------------------------------------------------
+
+
+async def test_a_verdict_in_the_test_window_never_becomes_a_prior_for_a_later_one(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ monkeypatch.setenv("AISOC_DETERMINISTIC", "1")
+ runner = ReplayRunner(normalizer=_Normalizer(), tenant_id=str(uuid.uuid4()))
+
+ run = await runner.run(_history())
+
+ twins = [d for d in run.decisions if d.finding_id.startswith("ES-TWIN")]
+ assert len(twins) == 2, "the split did not put both twins in the test window; the fixture is stale"
+ # Neither is decided from memory, and in particular the second is not
+ # decided from the first. `memory` is the tier the suppression path sets.
+ assert [d.tier for d in twins] == ["deterministic", "deterministic"]
+ # Production *would* have written the prior. That it tried is what proves
+ # the leak was live and was refused, rather than never arising.
+ assert run.writes_attempted.get("record_outcome", 0) >= 2
+
+
+async def test_the_same_run_leaks_through_memory_once_the_shadow_sinks_are_removed(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ """Sensitivity check: the protection above is protecting against something real.
+
+ One stubbed model, one prior store, two configurations, everything else
+ held constant. With the shadow writer and the frozen reader, four repeats
+ of the same evidence are each triaged. Swap in the live reader and a writer
+ that records, and the fourth is auto-closed from the first three's prior at
+ tier ``memory``, without reaching triage at all.
+
+ The verdict is stubbed at ``benign`` / 0.95 rather than lowering the
+ corroboration thresholds, because those thresholds are what the leak needs
+ to clear and a test that relaxes them is testing a configuration nobody
+ runs. A confident benign on a repeat is what production's LLM tier
+ produces on exactly this shape of alert.
+ """
+ monkeypatch.delenv("AISOC_DETERMINISTIC", raising=False)
+
+ async def _confident_benign(state: InvestigationState) -> InvestigationState:
+ state.verdict = BENIGN
+ state.confidence = 0.95
+ state.status = AgentStatus.COMPLETED
+ return state
+
+ async def _resolve(tenant_id: str) -> _StubLlmConfig:
+ return _StubLlmConfig()
+
+ monkeypatch.setattr(worker_module, "run_auto_triage", _confident_benign)
+ monkeypatch.setattr(worker_module, "resolve_llm_config", _resolve)
+
+ store: dict[str, dict[str, Any]] = {}
+
+ async def _fake_record(tenant_id: str, signature: str, **kwargs: Any) -> dict[str, Any]:
+ prior = {
+ "signature": signature,
+ # Normalised exactly as the real ``record_outcome`` does. A stub
+ # that skipped this step would leave a disposition outside
+ # ``AUTO_CLOSEABLE_DISPOSITIONS``, the leak would look unreachable,
+ # and this whole file would disarm itself quietly.
+ "disposition": normalize_disposition(kwargs["disposition"], default=NEEDS_REVIEW),
+ "confidence": kwargs["confidence"],
+ "author": kwargs["author"],
+ "count": store.get(signature, {}).get("count", 0) + 1,
+ }
+ store[signature] = prior
+ return prior
+
+ async def _fake_lookup(tenant_id: str, signature: str) -> dict[str, Any] | None:
+ return store.get(signature)
+
+ monkeypatch.setattr(outcomes_module, "record_outcome", _fake_record)
+ monkeypatch.setattr(outcomes_module, "lookup_prior", _fake_lookup)
+
+ class _MemoryWritingShadow(ShadowTriageWriter):
+ async def record_outcome(self, tenant_id: str, signature: str, **kwargs: Any) -> None:
+ await outcomes_module.record_outcome(tenant_id, signature, **kwargs)
+
+ protected = await _tiers_for_repeats(
+ FusedAlertTriageWorker(
+ bootstrap_servers="",
+ writer=ShadowTriageWriter(),
+ context_reader=FrozenTriageContextReader(ContextSnapshot(split_at=_SPLIT)),
+ )
+ )
+ assert "memory" not in protected, protected
+
+ store.clear()
+ leaking = await _tiers_for_repeats(
+ FusedAlertTriageWorker(
+ bootstrap_servers="",
+ writer=_MemoryWritingShadow(),
+ context_reader=LiveTriageContextReader(),
+ )
+ )
+ assert leaking[-1] == "memory", (
+ "the last repeat was not suppressed from the earlier ones' prior, so the leak this "
+ f"file guards against is no longer reachable and the guard above proves nothing: {leaking}"
+ )
+
+
+async def _tiers_for_repeats(worker: FusedAlertTriageWorker, repeats: int = 4) -> list[str]:
+ """Triage the same evidence ``repeats`` times; return the tier each took."""
+ tenant = str(uuid.uuid4())
+ tiers: list[str] = []
+ for index in range(repeats):
+ summary = await worker.triage(
+ {
+ "id": f"ES-REPEAT-{index}",
+ "alert_row_id": f"ES-REPEAT-{index}",
+ "tenant_id": tenant,
+ "alert": {
+ "id": f"ES-REPEAT-{index}",
+ "title": "Repeated service logon",
+ "rule_id": "rule-service-logon",
+ "severity": "low",
+ "src_ip": _SHARED_EVIDENCE["src"],
+ "hostname": _SHARED_EVIDENCE["host"],
+ },
+ }
+ )
+ assert summary is not None
+ tiers.append(str(summary["tier"]))
+ return tiers
+
+
+# --------------------------------------------------------------------------
+# 2. The cost governor's dedup cache
+# --------------------------------------------------------------------------
+
+
+async def test_a_test_window_verdict_is_not_served_back_from_the_dedup_cache(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ """The cache leaks without touching memory or the ledger, so it needs its own test."""
+ monkeypatch.setenv("AISOC_DETERMINISTIC", "1")
+ runner = ReplayRunner(normalizer=_Normalizer(), tenant_id=str(uuid.uuid4()))
+
+ run = await runner.run(_history())
+
+ assert "cached" not in {d.tier for d in run.decisions}
+ # Declined rather than never reached: the worker asked the sink to cache
+ # every verdict it produced.
+ assert run.writes_attempted.get("cache_verdict", 0) == len(run.decisions)
+
+
+# --------------------------------------------------------------------------
+# 3. Organisation memory
+# --------------------------------------------------------------------------
+
+
+async def test_triage_reads_the_frozen_statements_not_whatever_the_store_holds_now(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ """The statements that reach the prompt are the snapshot's, at the split.
+
+ Organisation memory is only read on the LLM branch, so this drives that
+ branch directly and captures what the agent was handed. Asserting on the
+ prompt input rather than on the verdict is deliberate: a verdict can be
+ unchanged by a leak that nonetheless happened, and the property under test
+ is what the model was allowed to see.
+ """
+ captured: list[list[dict[str, Any]]] = []
+
+ async def _capture(state: InvestigationState) -> InvestigationState:
+ captured.append(list(state.organisation_memory or []))
+ state.verdict = FALSE_POSITIVE
+ state.confidence = 0.5
+ state.status = AgentStatus.RUNNING
+ return state
+
+ async def _resolve(tenant_id: str) -> _StubLlmConfig:
+ return _StubLlmConfig()
+
+ async def _live_statements(tenant_id: str | None) -> list[dict[str, Any]]:
+ return [{"statement": "POISON: written after the split", "observations": 99}]
+
+ monkeypatch.delenv("AISOC_DETERMINISTIC", raising=False)
+ monkeypatch.setattr(worker_module, "run_auto_triage", _capture)
+ monkeypatch.setattr(worker_module, "resolve_llm_config", _resolve)
+ monkeypatch.setattr(organisation_memory, "fetch_statements", _live_statements)
+
+ frozen = capture_context(
+ split_at=_SPLIT,
+ statements=[{"statement": "Frozen at the split", "observations": 3, "updated_at": "2026-04-01T00:00:00Z"}],
+ )
+ worker = FusedAlertTriageWorker(
+ bootstrap_servers="",
+ writer=ShadowTriageWriter(),
+ context_reader=FrozenTriageContextReader(frozen),
+ )
+
+ summary = await worker.triage(_envelope())
+
+ assert summary is not None
+ assert captured == [[{"statement": "Frozen at the split", "observations": 3, "updated_at": "2026-04-01T00:00:00Z"}]]
+
+ # Sensitivity: the same worker with the live reader does see the poison.
+ live_worker = FusedAlertTriageWorker(
+ bootstrap_servers="",
+ writer=ShadowTriageWriter(),
+ context_reader=LiveTriageContextReader(),
+ )
+ captured.clear()
+ await live_worker.triage(_envelope())
+ assert captured == [[{"statement": "POISON: written after the split", "observations": 99}]]
+
+
+def _envelope() -> dict[str, Any]:
+ return {
+ "id": "ES-PROMPT-1",
+ "alert_row_id": "ES-PROMPT-1",
+ "tenant_id": str(uuid.uuid4()),
+ "alert": {
+ "id": "ES-PROMPT-1",
+ "title": "Repeated service logon",
+ "rule_id": "rule-service-logon",
+ "severity": "low",
+ "hostname": "APPSRV-01",
+ },
+ }
+
+
+# --------------------------------------------------------------------------
+# The snapshot itself
+# --------------------------------------------------------------------------
+
+
+def test_rows_recorded_after_the_split_are_dropped_from_the_snapshot() -> None:
+ snapshot = capture_context(
+ split_at=_SPLIT,
+ statements=[
+ {"statement": "before", "updated_at": "2026-04-01T00:00:00Z"},
+ {"statement": "after", "updated_at": "2026-06-01T00:00:00Z"},
+ {"statement": "undated"},
+ ],
+ priors={
+ "sig-before": {"disposition": FALSE_POSITIVE, "last_seen": "2026-04-02T00:00:00Z"},
+ "sig-after": {"disposition": FALSE_POSITIVE, "last_seen": "2026-06-02T00:00:00Z"},
+ },
+ )
+
+ assert [s["statement"] for s in snapshot.statements] == ["before", "undated"]
+ assert set(snapshot.priors) == {"sig-before"}
+ assert snapshot.dropped_statements == 1
+ assert snapshot.dropped_priors == 1
+ # The one statement whose age could not be checked is counted, not hidden.
+ # Organisation memory as the API serves it carries no creation time at
+ # all, so this number is how a reader sees the limit of the freeze.
+ assert snapshot.undated_statements == 1
+ assert snapshot.as_method_note()["statements_without_timestamp"] == 1
+
+
+def test_a_snapshot_taken_at_a_different_instant_is_refused() -> None:
+ """A freeze from the wrong moment is not a freeze, and must not pass silently."""
+ runner = ReplayRunner(normalizer=_Normalizer(), tenant_id=str(uuid.uuid4()))
+ wrong = ContextSnapshot(split_at=_SPLIT + timedelta(days=30))
+
+ with pytest.raises(ValueError, match="not a freeze"):
+ import asyncio
+
+ asyncio.run(runner.run(_history(), snapshot=wrong))
diff --git a/services/agents/tests/test_replay_runner.py b/services/agents/tests/test_replay_runner.py
new file mode 100644
index 000000000..419f696fb
--- /dev/null
+++ b/services/agents/tests/test_replay_runner.py
@@ -0,0 +1,304 @@
+"""The replay runner: splitting, envelope construction, and reproducibility.
+
+Gap-closure Phase 1.2.
+"""
+
+from __future__ import annotations
+
+import uuid
+from datetime import UTC, datetime, timedelta
+from typing import Any
+
+import httpx
+import pytest
+from app.replay.connector_normalizer import PrenormalizedRows, fetch_normalized, row_key
+from app.replay.findings import HistoricalFinding
+from app.replay.normalize import NormalizerUnavailable, to_fused_envelope
+from app.replay.runner import DEFAULT_TRAIN_FRACTION, ReplayRunner, split_by_time
+
+_BASE = datetime(2026, 3, 1, tzinfo=UTC)
+
+#: The real Splunk ES notable shape, as ``list_closed_notables`` returns it.
+#: Recorded rather than invented: the field names are the ones the SPL in
+#: ``SplunkClient.list_closed_notables`` selects.
+_NOTABLE = {
+ "event_id": "ES-1",
+ "rule_id": "rule-42",
+ "rule_name": "Suspicious PowerShell",
+ "search_name": "Suspicious PowerShell",
+ "urgency": "high",
+ "disposition": "disposition:1",
+ "src": "192.0.2.10",
+ "host": "WS-01",
+ "_time": "1772323200",
+}
+
+
+class _SplunkLikeNormalizer:
+ """The mapping ``services/connectors`` applies to a Splunk row.
+
+ A stand-in, and the docstring says so rather than letting a reader assume
+ the real connector is under test here. The real one is reached over HTTP
+ in production (``app.replay.connector_normalizer``) and is exercised
+ against the registry in the connectors service's own suite.
+ """
+
+ connector_id = "splunk"
+
+ def normalize(self, raw: dict[str, Any]) -> dict[str, Any]:
+ return {
+ "source": "splunk",
+ "external_id": raw.get("event_id", ""),
+ "title": raw.get("search_name") or "Splunk Notable Event",
+ "severity": "high",
+ "src_ip": raw.get("src"),
+ "hostname": raw.get("host"),
+ "raw_event": raw,
+ "created_at": raw.get("_time"),
+ }
+
+
+def _finding(index: int, *, disposition: str = "false_positive") -> HistoricalFinding:
+ return HistoricalFinding(
+ vendor="splunk",
+ finding_id=f"ES-{index:03d}",
+ title="Suspicious PowerShell",
+ disposition=disposition,
+ vendor_disposition="disposition:1",
+ closed_at=_BASE + timedelta(hours=index),
+ rule_id="rule-42",
+ raw={**_NOTABLE, "event_id": f"ES-{index:03d}"},
+ )
+
+
+# --------------------------------------------------------------------------
+# Splitting
+# --------------------------------------------------------------------------
+
+
+def test_the_split_is_by_time_and_the_test_window_is_the_later_period() -> None:
+ findings = [_finding(i) for i in range(10)]
+
+ split = split_by_time(findings)
+
+ assert split.train_fraction == DEFAULT_TRAIN_FRACTION
+ assert len(split.train) == 7
+ assert len(split.test) == 3
+ assert max(f.closed_at for f in split.train) <= split.split_at
+ assert min(f.closed_at for f in split.test) > split.split_at
+
+
+def test_a_finding_closed_exactly_at_the_split_stays_in_the_train_window() -> None:
+ """A tie must not become a test case the frozen context already knows about."""
+ findings = [_finding(i) for i in range(10)]
+ # Give three findings the same close time as the boundary row.
+ tied = [
+ HistoricalFinding(
+ vendor="splunk",
+ finding_id=f"TIE-{i}",
+ title="Tied",
+ disposition="false_positive",
+ vendor_disposition="disposition:3",
+ closed_at=findings[6].closed_at,
+ raw=dict(_NOTABLE),
+ )
+ for i in range(3)
+ ]
+
+ split = split_by_time([*findings, *tied])
+
+ assert all(f.closed_at <= split.split_at for f in split.train)
+ assert not any(f.finding_id.startswith("TIE") for f in split.test)
+
+
+def test_splitting_is_stable_when_close_times_collide() -> None:
+ """A bulk close gives many findings one timestamp; the order must still be total."""
+ same_time = [
+ HistoricalFinding(
+ vendor="splunk",
+ finding_id=f"BULK-{i:02d}",
+ title="Bulk closed",
+ disposition="false_positive",
+ vendor_disposition="disposition:3",
+ closed_at=_BASE,
+ raw=dict(_NOTABLE),
+ )
+ for i in range(10)
+ ]
+ later = [_finding(i + 50) for i in range(10)]
+
+ first = split_by_time([*same_time, *later])
+ shuffled = split_by_time([*later[::-1], *same_time[::-1]])
+
+ assert [f.finding_id for f in first.test] == [f.finding_id for f in shuffled.test]
+
+
+def test_an_empty_history_is_refused_rather_than_split() -> None:
+ with pytest.raises(ValueError, match="empty history"):
+ split_by_time([])
+
+
[email protected]("fraction", [0.0, 1.0, -0.1, 1.5])
+def test_a_fraction_that_would_empty_a_window_is_refused(fraction: float) -> None:
+ with pytest.raises(ValueError, match="strictly between"):
+ split_by_time([_finding(0), _finding(1)], train_fraction=fraction)
+
+
+# --------------------------------------------------------------------------
+# Envelope
+# --------------------------------------------------------------------------
+
+
+def test_the_envelope_carries_the_fields_build_state_reads() -> None:
+ finding = _finding(1)
+ normalized = _SplunkLikeNormalizer().normalize(dict(finding.raw))
+
+ envelope = to_fused_envelope(finding, normalized, tenant_id="t-1", connector_id="splunk")
+
+ alert = envelope["alert"]
+ assert envelope["alert_row_id"] == finding.finding_id
+ assert alert["title"] == "Suspicious PowerShell"
+ assert alert["severity"] == "high"
+ # Lifted by the connector.
+ assert alert["src_ip"] == "192.0.2.10"
+ assert alert["hostname"] == "WS-01"
+ # Present only in the untouched vendor row; read from there rather than
+ # dropped, because a connector lifts only what it has a canonical home for.
+ assert alert["rule_id"] == "rule-42"
+ assert alert["connector_type"] == "splunk"
+ assert alert["raw_event"] == dict(finding.raw)
+
+
+def test_the_envelope_does_not_invent_a_fusion_confidence() -> None:
+ """Fusion computes it from correlated evidence a single finding does not have."""
+ finding = _finding(1)
+ envelope = to_fused_envelope(finding, _SplunkLikeNormalizer().normalize(dict(finding.raw)), tenant_id="t-1")
+
+ assert "confidence_score" not in envelope
+ assert envelope["alert"]["risk_score"] == 0.0
+
+
+# --------------------------------------------------------------------------
+# Running
+# --------------------------------------------------------------------------
+
+
+async def test_two_runs_over_one_history_produce_identical_decisions(monkeypatch: pytest.MonkeyPatch) -> None:
+ """The deterministic tier must be reproducible, which is the phase's acceptance bar.
+
+ Latency is excluded from the comparison and nothing else is. It is a wall
+ clock measurement and will never repeat; every other field is a property
+ of the input and the code.
+ """
+ monkeypatch.setenv("AISOC_DETERMINISTIC", "1")
+ findings = [_finding(i, disposition="true_positive" if i % 4 == 0 else "false_positive") for i in range(20)]
+ tenant = str(uuid.uuid4())
+
+ async def _run() -> list[dict[str, Any]]:
+ runner = ReplayRunner(normalizer=_SplunkLikeNormalizer(), tenant_id=tenant)
+ run = await runner.run(findings)
+ rows = []
+ for decision in run.decisions:
+ payload = decision.as_dict()
+ payload.pop("latency_ms")
+ rows.append(payload)
+ return rows
+
+ assert await _run() == await _run()
+
+
+async def test_a_row_the_normalizer_cannot_map_is_recorded_not_guessed(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setenv("AISOC_DETERMINISTIC", "1")
+
+ class _Refuses:
+ connector_id = "splunk"
+
+ def normalize(self, raw: dict[str, Any]) -> dict[str, Any]:
+ if raw.get("event_id") == "ES-009":
+ raise NormalizerUnavailable("no envelope for this row")
+ return _SplunkLikeNormalizer().normalize(raw)
+
+ runner = ReplayRunner(normalizer=_Refuses(), tenant_id=str(uuid.uuid4()))
+ run = await runner.run([_finding(i) for i in range(10)])
+
+ refused = [d for d in run.decisions if d.finding_id == "ES-009"]
+ assert len(refused) == 1
+ assert refused[0].verdict is None
+ assert "no envelope for this row" in (refused[0].error or "")
+
+
+# --------------------------------------------------------------------------
+# Reaching the production normalizer over HTTP
+# --------------------------------------------------------------------------
+
+
+async def test_the_connectors_service_supplies_the_normalized_envelopes() -> None:
+ rows = [dict(_NOTABLE), {**_NOTABLE, "event_id": "ES-2"}]
+ seen: dict[str, Any] = {}
+
+ def _handler(request: httpx.Request) -> httpx.Response:
+ seen["url"] = str(request.url)
+ seen["token"] = request.headers.get("Authorization")
+ seen["tenant"] = request.headers.get("X-AiSOC-Tenant-ID")
+ return httpx.Response(
+ 200,
+ json={
+ "connector_id": "splunk",
+ "row_count": 2,
+ "rows": [_SplunkLikeNormalizer().normalize(dict(row)) for row in rows],
+ },
+ )
+
+ transport = httpx.MockTransport(_handler)
+ original = httpx.AsyncClient
+
+ def _client(*args: Any, **kwargs: Any) -> httpx.AsyncClient:
+ kwargs["transport"] = transport
+ return original(*args, **kwargs)
+
+ with pytest.MonkeyPatch.context() as patch:
+ patch.setattr(httpx, "AsyncClient", _client)
+ lookup = await fetch_normalized("splunk", rows, tenant_id="t-1", service_url="http://connectors:8003", service_token="secret")
+
+ assert seen["url"] == "http://connectors:8003/connectors/splunk/normalize"
+ assert seen["token"] == "Bearer secret"
+ assert seen["tenant"] == "t-1"
+ assert isinstance(lookup, PrenormalizedRows)
+ assert lookup.normalize(dict(rows[1]))["external_id"] == "ES-2"
+
+
+async def test_a_short_batch_is_refused_rather_than_partially_graded() -> None:
+ """Fewer envelopes than rows would silently drop findings from the window."""
+ rows = [dict(_NOTABLE), {**_NOTABLE, "event_id": "ES-2"}]
+
+ def _handler(request: httpx.Request) -> httpx.Response:
+ return httpx.Response(200, json={"connector_id": "splunk", "row_count": 1, "rows": [{"source": "splunk"}]})
+
+ transport = httpx.MockTransport(_handler)
+ original = httpx.AsyncClient
+
+ def _client(*args: Any, **kwargs: Any) -> httpx.AsyncClient:
+ kwargs["transport"] = transport
+ return original(*args, **kwargs)
+
+ with pytest.MonkeyPatch.context() as patch:
+ patch.setattr(httpx, "AsyncClient", _client)
+ with pytest.raises(NormalizerUnavailable, match="silently drop findings"):
+ await fetch_normalized("splunk", rows, tenant_id="t-1", service_token="secret")
+
+
+async def test_a_missing_service_token_is_named_rather_than_retried() -> None:
+ with pytest.raises(NormalizerUnavailable, match="AISOC_SERVICE_TOKEN"):
+ await fetch_normalized("splunk", [dict(_NOTABLE)], tenant_id="t-1", service_token="")
+
+
+def test_a_row_outside_the_batch_is_refused_never_substituted() -> None:
+ lookup = PrenormalizedRows({row_key(_NOTABLE): {"source": "splunk"}}, connector_id="splunk")
+
+ with pytest.raises(NormalizerUnavailable, match="will not substitute"):
+ lookup.normalize({"event_id": "never-sent"})
diff --git a/services/agents/tests/test_replay_shadow_no_writes.py b/services/agents/tests/test_replay_shadow_no_writes.py
new file mode 100644
index 000000000..a12c34870
--- /dev/null
+++ b/services/agents/tests/test_replay_shadow_no_writes.py
@@ -0,0 +1,253 @@
+"""Shadow mode must not write anything, and must be the production path.
+
+Gap-closure Phase 1.2.
+
+Two claims, and they need different kinds of evidence.
+
+**"It writes nothing"** is proved against the *real* modules, not against
+mocks of them. Every write function in the triage path is monkeypatched to
+raise, and then a replay is run. A mock that records calls would pass while
+the worker called something nobody thought to mock; a module that raises
+cannot.
+
+**"It is the production path"** cannot be proved by a test that constructs the
+thing it is testing, so this file does not try. It pins the two properties that
+make the claim checkable instead: the worker's default sinks are the live ones
+(so a deployment passing neither argument gets production behaviour), and the
+service entrypoint constructs the worker without overriding them. The second is
+read off the source with ``ast`` rather than by importing ``main``, which pulls
+in Kafka.
+"""
+
+from __future__ import annotations
+
+import ast
+import uuid
+from datetime import UTC, datetime, timedelta
+from pathlib import Path
+
+import pytest
+from app.investigator import ledger as ledger_module
+from app.investigator import siem_writeback
+from app.memory import outcomes as outcomes_module
+from app.replay.findings import HistoricalFinding
+from app.replay.runner import ReplayRunner
+from app.replay.shadow import ContextSnapshot, FrozenTriageContextReader, ShadowTriageWriter
+from app.workers import fused_alert_consumer as worker_module
+from app.workers.fused_alert_consumer import FusedAlertTriageWorker
+from app.workers.triage_persistence import (
+ LiveTriageContextReader,
+ LiveTriageWriter,
+ TriageContextReader,
+ TriageWriter,
+)
+
+_BASE = datetime(2026, 4, 1, 9, 0, tzinfo=UTC)
+
+
+class _PassthroughNormalizer:
+ """Stands in for a connector. Named so a reader does not mistake it for one.
+
+ Replay refuses to normalize its own way, so the runner needs *some*
+ object with ``normalize``. Using a stand-in here keeps this file about
+ writes; ``test_replay_runner.py`` drives the real Splunk connector
+ mapping.
+ """
+
+ connector_id = "splunk"
+
+ def normalize(self, raw: dict) -> dict:
+ return {
+ "source": "splunk",
+ "title": raw.get("search_name") or "Splunk Notable Event",
+ "severity": "high",
+ "src_ip": raw.get("src"),
+ "hostname": raw.get("host"),
+ "raw_event": raw,
+ }
+
+
+def _findings(count: int = 10) -> list[HistoricalFinding]:
+ return [
+ HistoricalFinding(
+ vendor="splunk",
+ finding_id=f"ES-{index:03d}",
+ title="Suspicious PowerShell",
+ disposition="true_positive" if index % 3 == 0 else "false_positive",
+ vendor_disposition="disposition:1",
+ closed_at=_BASE + timedelta(hours=index),
+ rule_id="rule-powershell",
+ raw={"search_name": "Suspicious PowerShell", "src": "10.1.2.3", "host": f"WS-{index:03d}"},
+ )
+ for index in range(count)
+ ]
+
+
+def _every_write_raises(monkeypatch: pytest.MonkeyPatch) -> None:
+ """Make every durable write in the triage path fatal.
+
+ ``BaseException`` on purpose. The worker wraps most of its writes in
+ ``contextlib.suppress(Exception)`` or a fail-soft ``except Exception``,
+ which is correct production behaviour and is exactly what would let a
+ leaked write pass this test silently.
+ """
+
+ class WroteSomething(BaseException):
+ pass
+
+ def _refuse(name: str):
+ async def _fail(*args: object, **kwargs: object):
+ raise WroteSomething(f"shadow replay called {name}")
+
+ return _fail
+
+ monkeypatch.setattr(ledger_module, "persist_auto_triage", _refuse("ledger.persist_auto_triage"))
+ monkeypatch.setattr(ledger_module, "raise_approval", _refuse("ledger.raise_approval"))
+ monkeypatch.setattr(ledger_module, "record_suppression", _refuse("ledger.record_suppression"))
+ monkeypatch.setattr(outcomes_module, "record_outcome", _refuse("outcomes.record_outcome"))
+ monkeypatch.setattr(siem_writeback, "write_back_disposition", _refuse("siem_writeback.write_back_disposition"))
+
+
+async def test_a_replay_touches_no_write_path(_every_write_raises: None, monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setenv("AISOC_DETERMINISTIC", "1")
+ findings = _findings()
+ runner = ReplayRunner(normalizer=_PassthroughNormalizer(), tenant_id=str(uuid.uuid4()))
+
+ run = await runner.run(findings)
+
+ assert run.decisions, "the replay produced no decisions, so it proved nothing"
+ assert all(d.error is None for d in run.decisions), [d.error for d in run.decisions if d.error]
+
+
+async def test_the_writes_production_would_have_made_are_counted(monkeypatch: pytest.MonkeyPatch) -> None:
+ """Zero has to be a number the run reports, not an absence a reader infers.
+
+ A replay that never reached the persistence branch and a replay whose
+ persistence was suppressed both write nothing. Only the counter tells them
+ apart, and without it this suite could go green over a runner that silently
+ stopped replaying anything.
+ """
+ monkeypatch.setenv("AISOC_DETERMINISTIC", "1")
+ runner = ReplayRunner(normalizer=_PassthroughNormalizer(), tenant_id=str(uuid.uuid4()))
+
+ run = await runner.run(_findings())
+
+ writer_methods = {
+ "persist_auto_triage",
+ "record_outcome",
+ "record_suppression",
+ "raise_approval",
+ "write_back_disposition",
+ "cache_verdict",
+ }
+ attempted = {name for name, count in run.writes_attempted.items() if count}
+ # persist_auto_triage fires once per decision: production would have
+ # written a ledger row and an alerts row for each. That it was *attempted*
+ # is the proof the production path ran; that it was not *performed* is the
+ # previous test.
+ assert "persist_auto_triage" in attempted
+ assert run.writes_attempted["persist_auto_triage"] == len(run.decisions)
+ assert attempted <= writer_methods
+
+
+async def test_shadow_writer_implements_every_method_of_the_port() -> None:
+ """A sink missing a method is a write that falls through to production.
+
+ ``runtime_checkable`` protocols only check method *presence*, which is
+ precisely the property that matters here: a method added to
+ ``TriageWriter`` and not to the shadow implementation would raise
+ ``AttributeError`` mid-replay, or worse, be reached on the live module.
+ """
+ assert isinstance(ShadowTriageWriter(), TriageWriter)
+ assert isinstance(LiveTriageWriter(), TriageWriter)
+ assert isinstance(LiveTriageContextReader(), TriageContextReader)
+
+
+async def test_a_worker_built_without_sinks_is_the_production_one() -> None:
+ worker = FusedAlertTriageWorker(bootstrap_servers="")
+
+ assert isinstance(worker._writer, LiveTriageWriter)
+ assert isinstance(worker._reader, LiveTriageContextReader)
+ assert worker._writer.escalation_allowed is True
+ assert worker._writer.persists_cost is True
+
+
+def test_the_service_entrypoint_does_not_override_the_sinks() -> None:
+ """The deployed worker must be the default-sink one.
+
+ Read off the source rather than imported: ``app.main`` imports aiokafka
+ and the service's whole startup graph, and a test that has to boot the
+ service to check a constructor argument will be deleted the first time it
+ is slow.
+ """
+ source = Path(__file__).resolve().parents[1] / "app" / "main.py"
+ tree = ast.parse(source.read_text(encoding="utf-8"))
+
+ constructions = [
+ node
+ for node in ast.walk(tree)
+ if isinstance(node, ast.Call) and isinstance(node.func, ast.Name) and node.func.id == "FusedAlertTriageWorker"
+ ]
+ assert constructions, "app/main.py no longer constructs FusedAlertTriageWorker; this test is stale"
+ for call in constructions:
+ passed = {kw.arg for kw in call.keywords}
+ assert "writer" not in passed, "the deployed worker must use the live writer"
+ assert "context_reader" not in passed, "the deployed worker must use the live reader"
+
+
+async def test_escalation_cannot_change_the_graded_verdict(monkeypatch: pytest.MonkeyPatch) -> None:
+ """Shadow mode declines escalation, and that must not move the number.
+
+ ``ShadowTriageWriter.escalation_allowed`` is ``False`` because every node
+ of the investigation graph records itself to the ledger. That is only
+ sound if the verdict ``triage()`` returns is fixed before escalation runs.
+ This drives the real worker with escalation forced on and a graph runner
+ that rewrites the verdict, and asserts the returned summary is unmoved.
+ """
+ monkeypatch.setenv("AISOC_DETERMINISTIC", "1")
+ monkeypatch.setenv("AISOC_AGENT_ESCALATE_TO_GRAPH", "1")
+
+ async def _hostile_escalation(state, **kwargs: object):
+ state.verdict = "escalation_rewrote_this"
+ state.confidence = 0.01
+ return state
+
+ async def _no_bundle(**kwargs: object) -> dict:
+ return {}
+
+ monkeypatch.setattr(worker_module, "run_escalation", _hostile_escalation)
+ monkeypatch.setattr(worker_module, "prefetch_context_bundle_dict", _no_bundle)
+
+ class _EscalatingWriter(ShadowTriageWriter):
+ @property
+ def escalation_allowed(self) -> bool:
+ return True
+
+ worker = FusedAlertTriageWorker(
+ bootstrap_servers="",
+ writer=_EscalatingWriter(),
+ context_reader=FrozenTriageContextReader(ContextSnapshot(split_at=_BASE)),
+ )
+ envelope = {
+ "id": "ES-001",
+ "alert_row_id": "ES-001",
+ "tenant_id": str(uuid.uuid4()),
+ "alert": {"id": "ES-001", "title": "Ransomware detected", "severity": "critical", "hostname": "WS-001"},
+ }
+
+ summary = await worker.triage(envelope)
+
+ assert summary is not None
+ assert summary["verdict"] != "escalation_rewrote_this"
+
+
+def _frozen_reader():
+ from app.replay.shadow import FrozenTriageContextReader
+
+ return FrozenTriageContextReader(ContextSnapshot(split_at=_BASE))
diff --git a/services/connectors/app/api/router.py b/services/connectors/app/api/router.py
index 312596cd2..71287882c 100644
--- a/services/connectors/app/api/router.py
+++ b/services/connectors/app/api/router.py
@@ -110,6 +110,36 @@ class ResourceConfigRequest(BaseModel):
at_ts: str = PydField(default="", description="Optional ISO-8601 point-in-time.")
+#: A normalize request carrying more rows than this is refused rather than
+#: truncated. Truncation would return a short list the caller reads as "these
+#: are all of them", and replay would grade a window it did not ask for.
+MAX_NORMALIZE_ROWS = 2000
+
+
+class NormalizeRequest(BaseModel):
+ """Map raw vendor rows through a connector's production ``normalize()``.
+
+ Gap-closure Phase 1.2. Replay evaluation runs in ``services/agents`` and
+ has to normalize a customer's closed findings exactly as the live pipeline
+ does. It cannot import a connector: both services package their code as
+ top-level ``app``, so one process can hold one of them. The alternative to
+ this route is a second copy of every vendor's field mapping inside the
+ agents service, which is the one thing the replay work is not allowed to
+ do, because a mapping that drifts would grade the agent on inputs the
+ product never produces.
+
+ Carries no ``auth_config``, and that is a property rather than an
+ omission: the handler never builds a configured client, so this route
+ cannot make an outbound call to a customer's SIEM no matter what it is
+ sent.
+ """
+
+ rows: list[dict[str, Any]] = PydField(
+ ...,
+ description="Raw vendor rows, exactly as the vendor returned them.",
+ )
+
+
class FederatedQueryRequest(BaseModel):
"""Run a unified query against a single connector instance.
@@ -424,6 +454,58 @@ async def get_resource_config(connector_id: str, payload: ResourceConfigRequest)
return {"connector_id": connector_id, "resource_id": payload.resource_id, "config": config}
[email protected]("/connectors/{connector_id}/normalize")
+async def normalize_rows(connector_id: str, payload: NormalizeRequest):
+ """Run a connector's own ``normalize()`` over raw vendor rows.
+
+ The instance is built with ``__new__`` and no ``__init__``. That is
+ deliberate on two counts. ``normalize`` is a pure mapping in every
+ connector in this tree, so a configured client is not needed; and not
+ building one means this route holds no credential and has nothing to make
+ an outbound call with.
+
+ A connector whose ``normalize`` does reach for instance state raises
+ ``AttributeError``, and that becomes a 422 naming the connector rather
+ than a partially-mapped row. A half-normalized envelope would flow into
+ replay looking like a real one and quietly change what the agent was
+ graded on.
+ """
+ cls = CONNECTOR_REGISTRY.get(connector_id)
+ if cls is None:
+ raise HTTPException(status_code=404, detail=f"Connector '{connector_id}' not found")
+ if len(payload.rows) > MAX_NORMALIZE_ROWS:
+ raise HTTPException(
+ status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
+ detail=f"{len(payload.rows)} rows exceeds the {MAX_NORMALIZE_ROWS}-row limit for one normalize request",
+ )
+
+ connector = cls.__new__(cls)
+ normalized: list[dict[str, Any]] = []
+ for index, row in enumerate(payload.rows):
+ try:
+ normalized.append(cls.normalize(connector, dict(row)))
+ except AttributeError as exc:
+ raise HTTPException(
+ status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
+ detail=(
+ f"connector '{connector_id}' cannot normalize without a configured instance ({exc}); "
+ f"row {index} was not mapped and none are returned"
+ ),
+ ) from exc
+ except Exception as exc:
+ logger.exception("connector.normalize.runtime_error", connector_id=_safe_log_val(connector_id))
+ raise HTTPException(
+ status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
+ detail=f"connector '{connector_id}' failed to normalize row {index}: {type(exc).__name__}",
+ ) from exc
+
+ return {
+ "connector_id": connector_id,
+ "row_count": len(normalized),
+ "rows": normalized,
+ }
+
+
@router.post("/connectors/{connector_id}/query")
async def run_federated_query(connector_id: str, payload: FederatedQueryRequest):
"""Translate a ``UnifiedQuery`` and run it against the connector's backend.
diff --git a/services/connectors/tests/test_normalize_route.py b/services/connectors/tests/test_normalize_route.py
new file mode 100644
index 000000000..4268799b0
--- /dev/null
+++ b/services/connectors/tests/test_normalize_route.py
@@ -0,0 +1,115 @@
+"""``POST /connectors/{id}/normalize``: the production mapping, over HTTP.
+
+Gap-closure Phase 1.2. Replay evaluation lives in ``services/agents`` and
+cannot import a connector, because both services package their code as
+top-level ``app``. This route is how it reaches the real mapping instead of
+carrying a copy that would drift.
+
+The handler is driven directly rather than through a test client, which is how
+this suite already exercises router-level code. Authentication is a
+router-wide dependency (``require_console_or_service_auth``) covered by
+``test_tenant_scope.py`` and by ``scripts/check_route_auth.py``; repeating it
+here would test FastAPI's wiring rather than this route's behaviour.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+import pytest
+from app.api.router import MAX_NORMALIZE_ROWS, NormalizeRequest, normalize_rows
+from app.connectors import CONNECTOR_REGISTRY
+from fastapi import HTTPException
+
+#: A Splunk ES closed notable, in the shape ``list_closed_notables`` returns.
+#: Synthetic, and recorded against the field names that SPL selects.
+_NOTABLE: dict[str, Any] = {
+ "event_id": "ES-1",
+ "rule_id": "rule-42",
+ "search_name": "Suspicious PowerShell",
+ "urgency": "high",
+ "disposition": "disposition:1",
+ "src": "192.0.2.10",
+ "host": "WS-01",
+ "_time": "1772323200",
+}
+
+pytestmark = pytest.mark.asyncio
+
+
+async def test_the_route_returns_the_connectors_own_mapping() -> None:
+ body = await normalize_rows("splunk", NormalizeRequest(rows=[_NOTABLE]))
+
+ assert body["connector_id"] == "splunk"
+ assert body["row_count"] == 1
+ envelope = body["rows"][0]
+ assert envelope["source"] == "splunk"
+ assert envelope["title"] == "Suspicious PowerShell"
+ assert envelope["src_ip"] == "192.0.2.10"
+ assert envelope["hostname"] == "WS-01"
+ # The untouched vendor row travels with it, which is what lets replay hand
+ # fields the connector did not lift to the same triage production sees.
+ assert envelope["raw_event"] == _NOTABLE
+
+
+async def test_the_route_matches_the_connector_called_directly() -> None:
+ """The claim is "the same mapping", so it is compared against the class itself."""
+ cls = CONNECTOR_REGISTRY["splunk"]
+ direct = cls.normalize(cls.__new__(cls), dict(_NOTABLE))
+
+ body = await normalize_rows("splunk", NormalizeRequest(rows=[_NOTABLE]))
+
+ assert body["rows"][0] == direct
+
+
+async def test_order_is_preserved_so_a_caller_can_pair_rows_with_findings() -> None:
+ rows = [{**_NOTABLE, "event_id": f"ES-{index}"} for index in range(5)]
+
+ body = await normalize_rows("splunk", NormalizeRequest(rows=rows))
+
+ assert [e["external_id"] for e in body["rows"]] == [f"ES-{index}" for index in range(5)]
+
+
+async def test_an_unknown_connector_is_a_404() -> None:
+ with pytest.raises(HTTPException) as caught:
+ await normalize_rows("not-a-connector", NormalizeRequest(rows=[{}]))
+
+ assert caught.value.status_code == 404
+
+
+async def test_an_oversized_batch_is_refused_rather_than_truncated() -> None:
+ """A short list reads as "these are all of them" and would shrink the window."""
+ rows = [dict(_NOTABLE) for _ in range(MAX_NORMALIZE_ROWS + 1)]
+
+ with pytest.raises(HTTPException) as caught:
+ await normalize_rows("splunk", NormalizeRequest(rows=rows))
+
+ assert caught.value.status_code == 422
+ assert "exceeds" in str(caught.value.detail)
+
+
+async def test_the_request_model_carries_no_credential_field() -> None:
+ """Structural half of "this route cannot call a customer's SIEM".
+
+ The behavioural half is that the handler builds the connector with
+ ``__new__`` and never runs ``__init__``, so there is nothing configured to
+ call out with even if a credential were supplied.
+ """
+ assert set(NormalizeRequest.model_fields) == {"rows"}
+
+
[email protected]("connector_id", sorted(CONNECTOR_REGISTRY))
+async def test_every_registered_connector_either_normalizes_or_says_why(connector_id: str) -> None:
+ """No connector may half-map a row and return it looking complete.
+
+ A connector whose ``normalize`` reaches for instance state must produce a
+ 422 naming itself. The failure this forbids is a partially-populated
+ envelope flowing into replay indistinguishable from a real one.
+ """
+ try:
+ body = await normalize_rows(connector_id, NormalizeRequest(rows=[_NOTABLE]))
+ except HTTPException as exc:
+ assert exc.status_code == 422, exc.detail
+ assert connector_id in str(exc.detail)
+ return
+ assert body["row_count"] == 1
diff --git a/tests/test_readme_figures_gate.py b/tests/test_readme_figures_gate.py
index 40abcd0a8..7fefc5424 100644
--- a/tests/test_readme_figures_gate.py
+++ b/tests/test_readme_figures_gate.py
@@ -19,13 +19,19 @@ REPO_ROOT = Path(__file__).resolve().parent.parent
GATES = REPO_ROOT / "scripts" / "readme_gates.py"
-def _load_gates(tmp_root: Path):
- """Import readme_gates.py with REPO_ROOT pointed at a scratch tree."""
+def _import_gates():
+ """Import readme_gates.py exactly as it ships, pointed at the real tree."""
spec = importlib.util.spec_from_file_location("_readme_gates_under_test", GATES)
assert spec and spec.loader
module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = module
spec.loader.exec_module(module)
+ return module
+
+
+def _load_gates(tmp_root: Path):
+ """Import readme_gates.py with REPO_ROOT pointed at a scratch tree."""
+ module = _import_gates()
module.REPO_ROOT = tmp_root
module.README = tmp_root / "README.md"
@@ -111,8 +117,25 @@ def test_missing_sources_do_not_crash_the_gate(tmp_path: Path) -> None:
def test_live_repo_is_consistent() -> None:
- """The gate must pass against the real tree, not only fixtures."""
- module = _load_gates(REPO_ROOT)
+ """The gate must pass against the real tree, not only fixtures.
+
+ Loaded unmodified, which is the whole point of this test and was not true
+ of it before. ``_load_gates`` narrows ``FIGURE_DOCS`` to the one scratch
+ path its fixtures write, and passing the real root through it left that
+ narrowing in place: the "real tree" check read the compliance page and
+ neither ``ROADMAP.md`` nor ``RELEASES.md``. Both went stale, this test
+ stayed green, and ``scripts/readme_gates.py`` failed in CI over the same
+ repository this had just called consistent.
+
+ A test that reconfigures the gate before pointing it at production is not
+ testing production.
+ """
+ module = _import_gates()
+
+ assert module.REPO_ROOT == REPO_ROOT
+ # Pin the surface, so shrinking it is a visible edit rather than a silent
+ # one. Three documents publish the tally today.
+ assert len(module.FIGURE_DOCS) == 3
assert module.gate_readme_figures() == []