Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

feat(autonomy): measure shadow-mode agreement against a tenant's own analysts (#906)

Beenu Aroracdf1d4e0af1d
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index f72bacf22..0a7e963b8 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -392,6 +392,15 @@ jobs:
# something leaks. The script existed and passed; nothing ran it.
- name: Verify vendored redactor is in sync
run: python scripts/sync_vendored_redactor.py --check
+ # Gap-closure Phase 2.2. The sixth mirror, and the one where drift is a
+ # safety failure rather than an inconsistency. These rules decide what a
+ # tenant's track record has to show before the agent may act unattended:
+ # `services/actions` reads them at dispatch and `services/api` reads them
+ # when a promotion is decided. Two copies that disagree means the control
+ # is off in whichever one is more generous, and nobody finds out until
+ # something executes that the other half would have refused.
+ - name: Verify vendored autonomy evidence rules are in sync
+ run: python scripts/sync_vendored_autonomy_evidence.py --check
# Phase 1.1 — the registry at
# `services/connectors/app/connectors/__init__.py` is the single source
# of truth for the "N connectors" claim. The check verifies that:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index a4f389716..7d198ae6b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,79 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
+- **Shadow mode on the live queue, and agreement measured against your own
+ analysts** (gap-closure Phase 2.1 and 2.2). Phase 1 made a customer's closed
+ history gradeable. This makes the live queue gradeable too, which is the
+ half that has to exist before autonomy can be earned rather than switched
+ on.
+
+ **The gap.** Nothing measured whether this platform's verdicts matched the
+ people using it. The published benchmark scores a synthetic corpus that is
+ balanced by construction; a real queue is mostly false positives, so an
+ agent that calls everything benign scores well on one and is worth nothing
+ on the other. An operator deciding whether to let the agent act had no
+ figure taken on their own alerts.
+
+ **How it was measured.** Shadow mode is per tenant and per alert class. For
+ an enabled class, triage runs exactly as it would in production and then
+ acts on none of it: no disposition set, no auto-closure, no writeback, no
+ approval raised, no outcome prior recorded, no escalation. It still writes
+ the ledger run, the `ai_*` columns and a decision row, and it still bills
+ the spend, because a shadow run is the product running rather than a
+ measurement the tenant asked for. When an analyst later closes the alert,
+ here or in the source SIEM, the closure is matched to the decision it
+ grades and the pair becomes evidence.
+
+ **The property the whole thing rests on.** `alerts.disposition` is the
+ analyst's column and it is the column agreement is read back from. A shadow
+ verdict landing there would mean the agent filled in the answer it was
+ about to be graded against, and every alert nobody explicitly re-disposed
+ would score as perfect agreement. `persist_auto_triage` grew a `shadow`
+ flag, and the guard sits in the SQL (`SET disposition = CASE WHEN $10 THEN
+ disposition ELSE $3 END`) rather than in the caller, so a future code path
+ that forgets to force `auto_closed` off still cannot close an alert it was
+ only meant to observe.
+
+ **Agreement is computed over answered decisions only.** This is the part a
+ plausible implementation gets wrong. If an abstention counted as "not a
+ disagreement", an agent that answered a tenth of the queue confidently and
+ routed the rest to a human would post a near-perfect record on a population
+ it never attempted. Declining removes a decision from the numerator and the
+ denominator together, so the rate does not move; the abstention rate does,
+ and it is reported beside it. Recall on malicious runs the other way, with
+ every true positive in its denominator whether answered or not, because an
+ alert routed to a human was not caught by the agent. A rate with no
+ denominator reads "not measured", never 0, and every rate travels with the
+ count it was computed over.
+
+ **Both the window and the slice inside it.** A 30-day average is where a
+ gradual decline hides: 99% for three weeks and 70% this week still averages
+ about 95%. Every surface shows the trailing 50 decisions beside the window,
+ and the trailing slice is a count rather than a date range so it means the
+ same thing at ten alerts a day and at ten thousand.
+
+ **Reuse rather than rebuild.** The closure readers are Phase 1.1's five,
+ unchanged, including their rule that a vendor label outside the taxonomy
+ becomes `unlabeled` and is excluded from accuracy rather than guessed at.
+ The metric definitions are Phase 1.3's, and "uses the Phase 1 metrics" is
+ now enforced rather than asserted: `check_replay_contract_parity.py` was
+ extended from three trees to four and compares `GRADED_DISPOSITIONS`,
+ `ABSTENTION_VERDICTS`, `MALICIOUS` and `UNLABELED` in both directions. The
+ shared rules module is vendored into `services/api` byte-identically
+ (`sync_vendored_autonomy_evidence.py --check`), because a safety control
+ defined twice is off in whichever copy is more generous.
+
+ **The gates.** Three new claim-to-gate rows, taking the matrix to 159 rows
+ and 151 GATED. The parity gate was proven capable of failing by drifting
+ each collection in turn and watching it fire, rather than by having never
+ fired. Migration `066_shadow_autonomy.sql` adds `aisoc_shadow_mode` and
+ `aisoc_shadow_decisions` with RLS policies carrying the fail-open arm the
+ cross-tenant workers need, `FORCE`, and `aisoc_app` grants. Surfaced on the
+ SOC operations dashboard and on the autonomy scorecard; documented at
+ `apps/docs/docs/operations/shadow-mode.md`, including what the numbers are
+ not, which is a measure of agreement with your analysts rather than of
+ correctness.
+
- **Replay evaluation: the production triage path, run over a customer's own
closed findings, writing nothing** (gap-closure Phase 1.2 and 1.3). Phase
1.1 made an operator's analyst labels readable. Nothing could grade against
diff --git a/GAP_CLOSURE_PROGRESS.md b/GAP_CLOSURE_PROGRESS.md
index 22ae0b5df..6cb9dac4e 100644
--- a/GAP_CLOSURE_PROGRESS.md
+++ b/GAP_CLOSURE_PROGRESS.md
@@ -259,8 +259,8 @@ the decomposition and what each link rests on are in D8 above.
## Phase 2: Live shadow mode and evidence-gated autonomy
-- [ ] **2.1 Shadow mode**, per tenant and per alert class.
-- [ ] **2.2 Rolling agreement** per alert class, rule, source and model, on the operations dashboard and the autonomy scorecard.
+- [x] **2.1 Shadow mode**, per tenant and per alert class. Shipped in [#906](https://github.com/beenuar/AiSOC/pull/906). The seam is Phase 1.2's: `FusedAlertTriageWorker` already routes every write through the `TriageWriter` port, so shadow mode is a wrapper around the tenant's live sink rather than a second code path. Chosen per alert, not per worker, because the class is not known until the alert is in hand; the constructor sinks are untouched, which is what keeps the Phase 1.2 AST test true. Migration **066** adds `aisoc_shadow_mode` and `aisoc_shadow_decisions`. Analyst closures arrive from two places: a bounded sweep over closed alerts in this console, and the five Phase 1.1 SIEM readers polled back out of the customer's own product and matched on the vendor finding id. See D9 for the one property everything else rests on.
+- [x] **2.2 Rolling agreement** per alert class, rule, source and model, on the operations dashboard and the autonomy scorecard. Shipped in [#906](https://github.com/beenuar/AiSOC/pull/906). The metrics are Phase 1.3's and "uses the Phase 1 metrics" is now a gate rather than a sentence: `check_replay_contract_parity.py` went from three trees to four and compares `GRADED_DISPOSITIONS`, `ABSTENTION_VERDICTS`, `MALICIOUS` and `UNLABELED` in both directions, proven capable of failing by drifting each collection in turn. Agreement is computed over *answered* decisions only so abstaining cannot inflate it, malicious recall counts an abstention as a miss, a rate with no denominator reads "not measured", and every rate travels with its count. Matrix 156 rows to 159, GATED 148 to 151.
- [ ] **2.3 Promotion gate**, with automatic demotion on drift and every transition written to the hash-chained audit log.
## Phase 3: Prompt-injection evaluation suite
@@ -388,6 +388,71 @@ Nothing yet beyond this kickoff. Each entry below will name its PR.
1252 to 1280, `packages/aisoc-benchmark` 35 to 57, `services/connectors` 880
to 886. Every other suite unchanged and passing.
+### D9. A shadow verdict must not reach `alerts.disposition`, and the guard belongs in the SQL
+
+Recorded because the obvious implementation of shadow mode is wrong in a way
+that looks right and produces a flattering number.
+
+`ledger.persist_auto_triage` writes the agent's verdict to
+`alerts.disposition`. That column is described in the model as the analyst's,
+set from the feedback endpoint, and it is the column Phase 2.1 reconciliation
+reads to learn what the analyst decided. A shadow run that forwarded that
+write unchanged would have the agent filling in the answer it was about to be
+graded against: every alert an analyst did not explicitly re-dispose would
+score as perfect agreement, and the scorecard would climb toward a promotion
+on no evidence at all.
+
+This is the Phase 1 leakage lesson arriving by a different route. Phase 1's
+version was a verdict written as an outcome prior that suppressed the next
+alert; this one is a verdict written into the field the next measurement
+reads. Both are the evaluation answering itself, and neither shows up as an
+error.
+
+**Resolution:** `persist_auto_triage` gained a `shadow` flag, and the guard is
+written into the statement rather than left to the caller:
+
+ SET disposition = CASE WHEN $10 THEN disposition ELSE $3 END,
+ status = CASE WHEN $7 AND NOT $10 THEN 'resolved' ELSE status END,
+ resolved_at = CASE WHEN $7 AND NOT $10 THEN now() ELSE resolved_at END
+
+`NOT $10` is the load-bearing part. The shadow sink already forces
+`auto_closed` off, so the `$7` arm would be enough today; relying on that
+alone would mean a second caller added later closes alerts it was only meant
+to observe, and the symptom would be a tenant's queue emptying itself during
+an evaluation.
+
+### D10. The sink is chosen per alert, and the writer is passed rather than swapped
+
+Shadow mode is per tenant *and* per alert class, and the class is not known
+until the alert is in hand, so the choice cannot be made in the worker's
+constructor. Two ways not to do it: assigning to `self._writer` per message
+leaks across concurrently triaged alerts, and it would also break the Phase
+1.2 AST test that asserts the deployed worker overrides neither sink, which
+is a test worth keeping.
+
+So `triage()` resolves the sink once and passes it down. Five helpers took a
+keyword-only `writer` parameter, and it is **required** on all of them
+including `_record`, with no fallback to `self._writer`. A default would mean
+a future caller who forgets the keyword silently gets the live sink, which is
+the unsafe direction for this particular control: the failure would be a
+shadow run writing the analyst's disposition column.
+
+### D11. The gate logic is vendored, not fetched over HTTP
+
+`services/actions` enforces autonomy at dispatch and `services/api` decides
+promotions and owns the hash-chained audit log. Both package their code as
+top-level `app`, so one process holds one of them, and each image is built
+with only its own directory as build context.
+
+The Phase 1.4 precedent (D6, D8) is a round trip to the service that owns the
+thing. That is right for a *mapping* held by another service. It is wrong
+here: this is a safety control on a latency-sensitive path, and a control that
+fails open when a second service is unreachable is not a control. So the pure
+rules are a standard-library-only module vendored byte-identically, following
+the five mirrors already in the tree, with
+`sync_vendored_autonomy_evidence.py --check` wired into `ci.yml`. Two copies
+allowed to differ means the control is off in whichever one is more generous.
+
### Notes for the next session
Two traps this program hit that are worth not re-learning.
diff --git a/ROADMAP.md b/ROADMAP.md
index feab76e50..c4dd74a72 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -44,7 +44,7 @@ A proof-first, security-first program to make every README claim gate-backed, cl
- **Phase D (breadth):** D1 eight connectors (QRadar/Exabeam/Securonix/Devo/Netskope/Windows-Sysmon/Zeek-Suricata/syslog-CEF) · D2 AI/LLM-usage audit connector + `llm-*` detections + hot/cold lake tiering · D3 live-vendor mock-server smoke.
- **Phase E (prove it):** E1 CI-gated benchmark scoreboard tied to a deterministic live-agent MITRE-accuracy run.
-The claim-to-gate matrix stands at **156 rows — 148 GATED / 8 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test** and the ratchet (`MAX_NO_GATE=0`) forbids any regression. The 8 remaining PARTIAL rows are honest, named deferrals; none was relabelled without building the gate it names. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).
+The claim-to-gate matrix stands at **159 rows — 151 GATED / 8 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test** and the ratchet (`MAX_NO_GATE=0`) forbids any regression. The 8 remaining PARTIAL rows are honest, named deferrals; none was relabelled without building the gate it names. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).
## v4.0 — Shipped
diff --git a/apps/docs/docs/compliance/evidence-pack.md b/apps/docs/docs/compliance/evidence-pack.md
index 3ae4a06f0..a58af3631 100644
--- a/apps/docs/docs/compliance/evidence-pack.md
+++ b/apps/docs/docs/compliance/evidence-pack.md
@@ -33,7 +33,7 @@ and says so.
| Prompt-injection resistance | 32-payload adversarial corpus with a recall floor and a false-positive ceiling | `services/agents/tests/adversarial/` |
| Claim-to-gate traceability | Every product claim mapped to the CI job that fails when it stops being true | `docs/audit/CLAIM_TO_GATE_MATRIX.md` |
-The last row is the one worth reading first. It is the honest index: 148 rows
+The last row is the one worth reading first. It is the honest index: 151 rows
`GATED`, 8 `PARTIAL` with the specific gap named, and a ratchet that refuses
to let a row sit at `NO GATE`.
diff --git a/apps/docs/docs/operations/shadow-mode.md b/apps/docs/docs/operations/shadow-mode.md
new file mode 100644
index 000000000..9c8c65d33
--- /dev/null
+++ b/apps/docs/docs/operations/shadow-mode.md
@@ -0,0 +1,182 @@
+---
+id: shadow-mode
+title: Shadow mode and measured agreement
+sidebar_label: Shadow mode
+---
+
+# Shadow mode and measured agreement
+
+Shadow mode runs triage on your live alerts, records what the agent concluded,
+and acts on none of it. When one of your analysts closes the same alert, the
+two verdicts are compared. What accumulates is a track record: on your alerts,
+against your analysts, how often the agent was right and how often it caught
+the ones that mattered.
+
+This page covers what shadow mode does and does not do, how agreement is
+computed, and what the numbers can and cannot tell you.
+
+## Why measure before trusting
+
+The published benchmark scores this platform against a synthetic corpus. That
+corpus is balanced by construction and yours is not: a real queue is mostly
+false positives with a handful of findings that mattered. An agent that calls
+everything benign scores well on a real queue and is worth nothing.
+
+So the only measurement that answers "should I let this act on my estate" is
+one taken on your own alerts and graded against your own analysts. Shadow mode
+is how that measurement is taken without the agent influencing it.
+
+## What it does, precisely
+
+For each alert class you enable, triage runs exactly as it would in
+production. Then:
+
+| Effect | In normal operation | In shadow mode |
+|---|---|---|
+| Ledger run and event | written | written |
+| `ai_score`, `ai_summary`, `ai_recommendations` on the alert | written | written |
+| `alerts.disposition` | set to the agent's verdict | **left alone** |
+| Alert auto-closed | when the verdict is confident and benign | **never** |
+| Disposition written back to your SIEM | attempted | **never** |
+| Approval raised for a proposed action | queued | **never** |
+| Outcome prior recorded, so a repeat alert can suppress | written | **never** |
+| Escalation into the full investigation graph | runs | **never** |
+| Model spend | measured and billed | measured and billed |
+| Deduplication through the cost governor | applies | applies |
+| Row in `aisoc_shadow_decisions` | not written | **written** |
+
+Two of those need their reasoning stated.
+
+**`alerts.disposition` is left alone**, and this is the property the whole
+measurement rests on. That column is the analyst's, and it is the column
+agreement is later read from. If a shadow verdict landed there, every alert an
+analyst did not explicitly re-dispose would score as perfect agreement, and the
+track record would climb toward a promotion on nothing at all. The guard is in
+the SQL statement rather than in the caller, so a future code path that forgets
+to ask for it still cannot close an alert it was only meant to observe.
+
+**Spend is still billed.** A shadow run places real model calls against a real
+key. Replay evaluation declines to bill because a replay is a measurement you
+asked for; a shadow run is the product running, and a month of it should appear
+on your cost dashboard.
+
+## Turning it on
+
+Shadow mode is per tenant and per alert class, where a class is the
+`category` on the alert (`identity`, `cloud`, `endpoint`, `network`,
+`application`, and whatever else your detection content sets). `*` matches
+every class, which is where most deployments start, because knowing which
+classes your queue actually contains is part of what the first weeks of
+measurement are for.
+
+```bash
+# every class
+curl -X PUT "$AISOC/api/v1/autonomy-policy/shadow-mode/*" \
+ -H "Authorization: Bearer $TOKEN" \
+ -H 'Content-Type: application/json' \
+ -d '{"enabled": true}'
+
+# one class
+curl -X PUT "$AISOC/api/v1/autonomy-policy/shadow-mode/identity" \
+ -H "Authorization: Bearer $TOKEN" \
+ -H 'Content-Type: application/json' \
+ -d '{"enabled": true}'
+```
+
+The tenant comes from your credential. There is no request field for it.
+
+## Where the analyst's verdict comes from
+
+Two places, and both count.
+
+**Closures made in AiSOC.** A sweep matches shadow decisions to alerts that
+have since moved to `resolved` or `closed` and copies the disposition across.
+It is a sweep rather than a hook on the disposition endpoint because an alert
+can be closed from the feedback endpoint, by a case being resolved, by a bulk
+action or by a playbook, and a hook on one of those would silently grade the
+subset of alerts closed the way the hook was written for.
+
+**Closures made in your SIEM.** Most analysts evaluating AiSOC keep working
+where they always have. The same readers that power
+[replay evaluation](../evaluation/replay.md) poll closed findings back out of
+Splunk ES, Microsoft Sentinel, Elastic Security, IBM QRadar and Microsoft
+Defender XDR, and each closure is matched to the decision it grades by the
+vendor's own finding id. A tenant whose analysts work entirely in Splunk will
+still see their scorecard fill in.
+
+Either way, a closure carrying a disposition this platform cannot name becomes
+`unlabeled`. It is counted as resolved and excluded from every rate. Splunk ES
+ships two dispositions that literally mean "I do not know", and so do Sentinel
+and Defender; folding those into a verdict because the finding happened to be
+closed would manufacture agreement out of an analyst's uncertainty.
+
+## How the numbers are computed
+
+The definitions are the ones replay evaluation already uses. They are not
+restated here in different words, and a CI gate compares them in both
+directions so they cannot drift apart.
+
+**Agreement** is the share of *answered* decisions where the agent's verdict is
+the analyst's label. Abstentions, which are `needs_review`, `escalate`,
+`unknown` and an empty verdict, are excluded from both halves of that fraction.
+
+That exclusion is deliberate and it is worth understanding, because it is what
+stops the metric being gameable. If abstentions counted as "not a
+disagreement", an agent that answered a tenth of the queue confidently and
+routed the rest to a human would post a near-perfect record on a population it
+never attempted. Declining a decision removes it from the numerator and the
+denominator together, so the agreement rate does not move; what moves is the
+abstention rate, which is reported beside it.
+
+**Recall on malicious** runs the other way. Its denominator is every alert an
+analyst closed as a true positive, whether the agent answered or abstained, so
+an abstention counts here as a miss. An alert routed to a human was not caught
+by the agent.
+
+**A rate with no denominator reads "not measured", never 0.** A zero in the
+agreement column says the agent was wrong every time; "it has not been asked
+yet" is a different fact, and on a new deployment it is always the true one.
+
+**Every rate travels with the count behind it.** 100% agreement over four
+answers is not the claim it is over four hundred, and a display that printed
+only the percentage would have thrown that distinction away.
+
+## The window, and the slice inside it
+
+Agreement is reported over a rolling window, 30 days by default, **and** over
+the most recent 50 decisions separately.
+
+Both, always. A window average is exactly where a gradual decline hides: an
+agent that agreed 99% of the time for three weeks and 70% of the time this week
+still posts about 95% over the window. A surface that showed only the window
+would conceal the thing it exists to reveal, so the console shows the trailing
+slice beside it and the API returns both.
+
+The trailing slice is a count rather than a date range, so it means the same
+thing for a tenant seeing ten alerts a day and one seeing ten thousand.
+
+## Where to look
+
+* **SOC operations dashboard**: the "Agreement with analysts" panel, with the
+ breakdowns by alert class and by source.
+* **Settings, Autonomy guardrails**: the scorecard shows the configured posture
+ and the measured track record on one card, because the question they answer
+ together is the only one worth asking, which is whether the posture is
+ justified.
+* **`GET /api/v1/autonomy-policy/agreement`**: the same figures, with
+ breakdowns by alert class, rule, source and model.
+
+## What these numbers are not
+
+They are a measure of **agreement with your analysts**, not of correctness.
+Where your analysts were wrong, an agent that agreed with them scores well
+here. That is a real limit, and it is the same limit any evaluation against
+human labels carries.
+
+They describe the **alert classes you enabled**, over the **window shown**, on
+the **model that was running**. The per-model breakdown exists because a model
+change makes every figure before it a description of something else.
+
+They say nothing about **response actions**. Shadow mode measures triage
+verdicts. Whether a containment action would have been correct is a different
+question this does not answer.
diff --git a/apps/docs/sidebars.ts b/apps/docs/sidebars.ts
index f525f9f5b..5b26b1653 100644
--- a/apps/docs/sidebars.ts
+++ b/apps/docs/sidebars.ts
@@ -196,6 +196,7 @@ const sidebars: SidebarsConfig = {
"operations/upgrades",
"operations/notifications",
"operations/action-approvals",
+ "operations/shadow-mode",
"operations/case-reports",
"operations/troubleshooting",
"operations/faq",
diff --git a/apps/web/src/components/dashboard/operations/AgreementPanel.tsx b/apps/web/src/components/dashboard/operations/AgreementPanel.tsx
new file mode 100644
index 000000000..c1217eea3
--- /dev/null
+++ b/apps/web/src/components/dashboard/operations/AgreementPanel.tsx
@@ -0,0 +1,223 @@
+'use client';
+
+/**
+ * How often the agent and this tenant's own analysts reached the same verdict.
+ *
+ * Gap-closure Phase 2.2.
+ *
+ * The rest of this dashboard answers "is the pipeline working". This panel
+ * answers the question that decides whether any of it gets trusted with a
+ * decision: on the alerts your analysts have closed, how often was the agent
+ * right, and how often did it catch the ones that mattered.
+ *
+ * Three rules this panel follows, each because the obvious alternative
+ * misleads in the direction that sells.
+ *
+ * **A rate with no denominator reads "not measured", never 0.** A zero in the
+ * agreement column says the agent was wrong every time. "It has not been
+ * asked yet" is a different fact calling for a different response, and on a
+ * new deployment it is always the true one.
+ *
+ * **Every rate is printed with the count behind it.** 100% over four answers
+ * is not the claim 100% over four hundred is, and a tile showing only the
+ * percentage has thrown that distinction away.
+ *
+ * **The recent slice is shown beside the window, not instead of it.** A
+ * thirty-day average is where a gradual decline hides: an agent that agreed
+ * 99% of the time for three weeks and 70% this week still posts about 95%.
+ * Showing only the window would make this panel the thing that conceals the
+ * problem it exists to reveal.
+ */
+
+import useSWR from 'swr';
+import { clsx } from 'clsx';
+import { autonomyPolicyApi, type AgreementScope } from '@/lib/api';
+import { EmptyState } from '@/components/ui/EmptyState';
+import { ErrorState } from '@/components/ui/ErrorState';
+
+/** A rate as a percentage, or the words that mean there was no denominator. */
+export function formatRate(value: number | null | undefined): string {
+ return value === null || value === undefined ? 'not measured' : `${(value * 100).toFixed(1)}%`;
+}
+
+/**
+ * Progress toward a sample floor, as a fraction a reader can act on.
+ *
+ * "47 of 100" tells an operator to keep measuring. "47%" invites them to read
+ * it as an accuracy figure, on a panel where every other percentage is one.
+ */
+export function formatProgress(have: number, need: number): string {
+ return `${have} of ${need}`;
+}
+
+export function AgreementPanel() {
+ const { data, error, isLoading } = useSWR(
+ 'ops-shadow-agreement',
+ () => autonomyPolicyApi.agreement(),
+ { refreshInterval: 60_000, revalidateOnFocus: false },
+ );
+
+ if (error) {
+ return (
+ <Shell>
+ <ErrorState
+ title="Agreement unavailable"
+ description="The autonomy policy endpoint did not answer, so no figures are shown. An agreement rate that is silently stale is worse than none."
+ error={error}
+ />
+ </Shell>
+ );
+ }
+
+ if (isLoading || !data) {
+ return (
+ <Shell>
+ <p className="text-sm text-gray-500">Loading agreement…</p>
+ </Shell>
+ );
+ }
+
+ const { window: w, recent, thresholds } = data;
+
+ if (w.resolved === 0) {
+ return (
+ <Shell>
+ <EmptyState
+ title="No graded decisions yet"
+ description={
+ 'Agreement is measured on alerts the agent triaged in shadow mode and an analyst later closed. ' +
+ 'Enable shadow mode for an alert class in Settings, then close some alerts as usual.'
+ }
+ />
+ </Shell>
+ );
+ }
+
+ return (
+ <Shell>
+ <dl className="grid grid-cols-2 gap-3 sm:grid-cols-4">
+ <Metric
+ label="Agreement"
+ value={formatRate(w.agreement_rate)}
+ detail={`${w.agreed} of ${w.answered} answered`}
+ meets={meets(w.agreement_rate, thresholds.min_agreement)}
+ />
+ <Metric
+ label="Recall on malicious"
+ value={formatRate(w.malicious_recall)}
+ detail={`${w.malicious_caught} of ${w.malicious_support} true positives`}
+ meets={meets(w.malicious_recall, thresholds.min_malicious_recall)}
+ />
+ <Metric
+ label="Abstained"
+ value={formatRate(w.abstention_rate)}
+ detail={`${w.abstained} of ${w.labelled} labelled`}
+ /* The only metric here where lower is better: an agent can lift its
+ agreement rate by declining the hard cases, and this is the column
+ where that shows up. */
+ meets={w.abstention_rate === null ? null : w.abstention_rate <= thresholds.max_abstention_rate}
+ />
+ <Metric
+ label="Sample"
+ value={formatProgress(w.labelled, thresholds.min_decisions)}
+ detail={`${formatProgress(w.malicious_support, thresholds.min_malicious)} malicious`}
+ meets={w.labelled >= thresholds.min_decisions && w.malicious_support >= thresholds.min_malicious}
+ />
+ </dl>
+
+ <p className="mt-3 text-xs text-gray-500">
+ Last {thresholds.window_days} days. {w.unlabeled} closure
+ {w.unlabeled === 1 ? '' : 's'} carried no disposition this platform can name and
+ {w.unlabeled === 1 ? ' is' : ' are'} excluded from every rate above.
+ </p>
+
+ <div className="mt-4 rounded-md border border-gray-800 bg-gray-950/60 p-3">
+ <p className="text-[11px] uppercase tracking-wide text-gray-500">
+ Most recent {recent.labelled} decision{recent.labelled === 1 ? '' : 's'}
+ </p>
+ <p className="mt-1 text-sm text-gray-300">
+ Agreement {formatRate(recent.agreement_rate)} ({recent.agreed} of {recent.answered}), recall{' '}
+ {formatRate(recent.malicious_recall)} ({recent.malicious_caught} of {recent.malicious_support}).
+ </p>
+ <p className="mt-1 text-xs text-gray-500">
+ Scored apart from the window, because a decline that started this week is still
+ absorbed by a month of earlier agreement.
+ </p>
+ </div>
+
+ <Breakdown title="By alert class" rows={data.by_alert_class} />
+ <Breakdown title="By source" rows={data.by_source} />
+ </Shell>
+ );
+}
+
+function meets(value: number | null, floor: number): boolean | null {
+ return value === null ? null : value >= floor;
+}
+
+function Shell({ children }: { children: React.ReactNode }) {
+ return (
+ <section
+ className="rounded-lg border border-gray-800 bg-gray-950/40 p-4"
+ aria-label="Agreement with analysts"
+ >
+ <header className="mb-3">
+ <h2 className="text-sm font-semibold text-gray-100">Agreement with analysts</h2>
+ <p className="mt-0.5 text-xs text-gray-500">
+ Measured on shadow decisions your analysts have since closed, here or in the source SIEM.
+ </p>
+ </header>
+ {children}
+ </section>
+ );
+}
+
+function Metric({
+ label,
+ value,
+ detail,
+ meets: met,
+}: {
+ label: string;
+ value: string;
+ detail: string;
+ meets: boolean | null;
+}) {
+ return (
+ <div>
+ <dt className="text-[11px] uppercase tracking-wide text-gray-500">{label}</dt>
+ <dd
+ className={clsx(
+ 'font-mono text-lg tabular-nums',
+ // No colour at all when there is nothing to judge. Painting an
+ // unmeasured figure red would read as a failing one.
+ met === null ? 'text-gray-400' : met ? 'text-emerald-300' : 'text-amber-300',
+ )}
+ >
+ {value}
+ </dd>
+ <p className="text-[11px] text-gray-500">{detail}</p>
+ </div>
+ );
+}
+
+function Breakdown({ title, rows }: { title: string; rows: AgreementScope[] }) {
+ if (rows.length === 0) return null;
+ return (
+ <div className="mt-4">
+ <p className="text-[11px] uppercase tracking-wide text-gray-500">{title}</p>
+ <ul className="mt-1 divide-y divide-gray-800/70">
+ {rows.slice(0, 6).map((row) => (
+ <li key={row.key} className="flex items-baseline justify-between gap-3 py-1.5 text-sm">
+ <span className="truncate text-gray-300">{row.key}</span>
+ <span className="shrink-0 font-mono text-xs tabular-nums text-gray-400">
+ {formatRate(row.window.agreement_rate)} · {row.window.answered} answered
+ </span>
+ </li>
+ ))}
+ </ul>
+ </div>
+ );
+}
+
+export default AgreementPanel;
diff --git a/apps/web/src/components/dashboard/operations/OperationsDashboard.tsx b/apps/web/src/components/dashboard/operations/OperationsDashboard.tsx
index b68ac5627..83e224e7d 100644
--- a/apps/web/src/components/dashboard/operations/OperationsDashboard.tsx
+++ b/apps/web/src/components/dashboard/operations/OperationsDashboard.tsx
@@ -15,6 +15,7 @@
* - alert severity + disposition GET /api/v1/alerts/stats
* - agent runs, tokens, spend GET /api/v1/costs/dashboard
* - actions awaiting approval GET /api/v1/approvals?status=pending
+ * - agreement with analysts GET /api/v1/autonomy-policy/agreement
*
* Plus the existing funnel strip and detection-coverage summary, which are
* already real-data-driven and are reused rather than re-implemented.
@@ -33,6 +34,7 @@ import { AlertPosturePanel } from './AlertPosturePanel';
import { AgentThroughputPanel } from './AgentThroughputPanel';
import { ResponseActionsPanel } from './ResponseActionsPanel';
import { DetectionCoveragePanel } from './DetectionCoveragePanel';
+import { AgreementPanel } from './AgreementPanel';
export function OperationsDashboard() {
return (
@@ -65,6 +67,8 @@ export function OperationsDashboard() {
<AgentThroughputPanel />
<ResponseActionsPanel />
</div>
+
+ <AgreementPanel />
</div>
);
}
diff --git a/apps/web/src/components/settings/AutonomyPolicy.tsx b/apps/web/src/components/settings/AutonomyPolicy.tsx
index eebd86f3b..8d1285d8b 100644
--- a/apps/web/src/components/settings/AutonomyPolicy.tsx
+++ b/apps/web/src/components/settings/AutonomyPolicy.tsx
@@ -29,6 +29,7 @@ import { motion, AnimatePresence } from 'framer-motion';
import toast from 'react-hot-toast';
import {
autonomyPolicyApi,
+ type AgreementResponse,
type AutonomyActionPolicy,
type AutonomyBlastRadius,
type AutonomyPolicyResponse,
@@ -98,6 +99,16 @@ export function AutonomyPolicyPanel() {
const actions = data?.actions ?? [];
+ // Phase 2.2 — the measured track record beside the configured posture. Its
+ // own SWR key, so an agreement endpoint that is unreachable blanks that half
+ // of the card rather than taking the guardrail editor down with it. The
+ // scorecard treats `undefined` as "not measured" and says so.
+ const { data: agreement } = useSWR<AgreementResponse>(
+ 'settings:autonomy-agreement',
+ () => autonomyPolicyApi.agreement(),
+ { revalidateOnFocus: false, shouldRetryOnError: false },
+ );
+
return (
<div>
<PanelHeader
@@ -110,7 +121,7 @@ export function AutonomyPolicyPanel() {
copilot). Rendered once the policy loads so the CISO sees the
whole-SOC autonomy posture before drilling into per-action rows. */}
{!isLoading && !error && actions.length > 0 ? (
- <AutonomyScorecard actions={actions} />
+ <AutonomyScorecard actions={actions} agreement={agreement} />
) : null}
{/* Legend */}
diff --git a/apps/web/src/components/settings/AutonomyScorecard.test.tsx b/apps/web/src/components/settings/AutonomyScorecard.test.tsx
index ec64f085f..f8ddda69b 100644
--- a/apps/web/src/components/settings/AutonomyScorecard.test.tsx
+++ b/apps/web/src/components/settings/AutonomyScorecard.test.tsx
@@ -10,8 +10,18 @@
import { describe, expect, it } from 'vitest';
import { render, screen } from '@testing-library/react';
-import type { AutonomyActionPolicy, AutonomyBlastRadius } from '@/lib/api';
-import { AutonomyScorecard, computeScorecard } from './AutonomyScorecard';
+import type {
+ AgreementResponse,
+ AgreementWindow,
+ AutonomyActionPolicy,
+ AutonomyBlastRadius,
+} from '@/lib/api';
+import {
+ AutonomyScorecard,
+ computeScorecard,
+ formatRate,
+ summariseTrackRecord,
+} from './AutonomyScorecard';
function action(
name: string,
@@ -80,3 +90,124 @@ describe('AutonomyScorecard', () => {
expect(screen.getByText(/auto-execute/i)).toBeInTheDocument();
});
});
+
+// ─── Measured track record (gap-closure Phase 2.2) ───────────────────────────
+//
+// The posture above is what the tenant configured. This half is what the agent
+// earned. The tests worth having are about the two ways this display can lie:
+// printing a zero where nothing was measured, and printing a rate without the
+// count behind it.
+
+function agreementResponse(overrides: Partial<AgreementWindow> = {}): AgreementResponse {
+ const window: AgreementWindow = {
+ resolved: 120,
+ labelled: 100,
+ unlabeled: 20,
+ answered: 90,
+ abstained: 10,
+ agreed: 87,
+ malicious_support: 31,
+ malicious_caught: 29,
+ agreement_rate: 87 / 90,
+ malicious_recall: 29 / 31,
+ abstention_rate: 10 / 100,
+ ...overrides,
+ };
+ return {
+ tenant_id: 't',
+ scope_kind: 'tenant',
+ scope_key: '*',
+ window,
+ recent: { ...window, answered: 50, agreed: 40, agreement_rate: 0.8 },
+ window_start: '2026-08-27T00:00:00+00:00',
+ window_end: '2026-09-26T00:00:00+00:00',
+ thresholds: {
+ min_decisions: 100,
+ min_malicious: 30,
+ min_agreement: 0.95,
+ min_malicious_recall: 0.9,
+ max_abstention_rate: 0.3,
+ window_days: 30,
+ demotion_agreement: 0.9,
+ demotion_malicious_recall: 0.8,
+ drift_sample: 50,
+ drift_min_answered: 20,
+ },
+ reconciled: 0,
+ by_alert_class: [],
+ by_rule: [],
+ by_source: [],
+ by_model: [],
+ };
+}
+
+describe('formatRate', () => {
+ it('renders a missing denominator as words, never as zero', () => {
+ // A zero here says the agent was wrong every time. "It was never asked" is
+ // a different fact, and on a new deployment it is always the true one.
+ expect(formatRate(null)).toBe('not measured');
+ expect(formatRate(undefined)).toBe('not measured');
+ });
+
+ it('renders a real zero as a zero', () => {
+ expect(formatRate(0)).toBe('0.0%');
+ });
+});
+
+describe('summariseTrackRecord', () => {
+ it('returns nothing when no agreement has been fetched', () => {
+ expect(summariseTrackRecord(undefined)).toBeNull();
+ });
+
+ it('marks a tenant with no closed decisions as unmeasured', () => {
+ const summary = summariseTrackRecord(
+ agreementResponse({ resolved: 0, labelled: 0, answered: 0, abstained: 0, agreed: 0 }),
+ );
+ expect(summary?.measured).toBe(false);
+ });
+
+ it('carries the counts each rate was computed over', () => {
+ const summary = summariseTrackRecord(agreementResponse());
+ expect(summary?.answered).toBe(90);
+ expect(summary?.maliciousSupport).toBe(31);
+ expect(summary?.labelled).toBe(100);
+ });
+
+ it('carries the trailing slice separately from the window', () => {
+ // The window average is where a gradual decline hides. Folding the two
+ // into one number would make this card conceal the thing it exists to show.
+ const summary = summariseTrackRecord(agreementResponse());
+ expect(summary?.agreement).toBeCloseTo(87 / 90);
+ expect(summary?.recentAgreement).toBeCloseTo(0.8);
+ });
+});
+
+describe('<AutonomyScorecard /> track record', () => {
+ it('says there is no measurement rather than rendering zeroes', () => {
+ render(<AutonomyScorecard actions={[action('block_ip', 'medium', 0.9)]} />);
+ expect(screen.getByText(/No measured track record yet/i)).toBeInTheDocument();
+ });
+
+ it('shows the sample as a fraction of the floor, not as a percentage', () => {
+ // On a card where every other figure is a percentage, "47%" would be read
+ // as a fifth accuracy number rather than as progress toward a threshold.
+ render(
+ <AutonomyScorecard
+ actions={[action('block_ip', 'medium', 0.9)]}
+ agreement={agreementResponse()}
+ />,
+ );
+ expect(screen.getByText('100 / 100')).toBeInTheDocument();
+ expect(screen.getByText('31 / 30 malicious')).toBeInTheDocument();
+ });
+
+ it('prints not measured for a rate with no denominator', () => {
+ render(
+ <AutonomyScorecard
+ actions={[action('block_ip', 'medium', 0.9)]}
+ agreement={agreementResponse({ malicious_support: 0, malicious_caught: 0, malicious_recall: null })}
+ />,
+ );
+ expect(screen.getByText('not measured')).toBeInTheDocument();
+ });
+});
diff --git a/apps/web/src/components/settings/AutonomyScorecard.tsx b/apps/web/src/components/settings/AutonomyScorecard.tsx
index 17450a1c4..b01a2799d 100644
--- a/apps/web/src/components/settings/AutonomyScorecard.tsx
+++ b/apps/web/src/components/settings/AutonomyScorecard.tsx
@@ -17,10 +17,26 @@
*
* The compute is a pure function so it's unit-tested directly; the component is
* a thin presentational shell.
+ *
+ * Gap-closure Phase 2.2 added the second half. The posture above describes
+ * what the tenant has *configured*; `TrackRecord` describes what the agent has
+ * *earned*, measured against the same analysts' own closures. Both belong on
+ * one card because the question they answer together is the only one worth
+ * asking: is this posture justified. A configured posture on its own says
+ * nothing about whether the agent deserves it, and the numbers on their own
+ * say nothing about whether anyone has acted on them.
+ *
+ * The track record is optional. A deployment that has never run shadow mode
+ * has no measurement, and the card says that rather than rendering zeroes,
+ * which would read as an agent that agrees with nobody.
*/
import { clsx } from 'clsx';
-import type { AutonomyActionPolicy, AutonomyBlastRadius } from '@/lib/api';
+import type {
+ AgreementResponse,
+ AutonomyActionPolicy,
+ AutonomyBlastRadius,
+} from '@/lib/api';
export type AutonomyPosture = 'copilot' | 'autopilot';
@@ -79,9 +95,69 @@ export function computeScorecard(actions: AutonomyActionPolicy[]): AutonomyScore
};
}
-export function AutonomyScorecard({ actions }: { actions: AutonomyActionPolicy[] }) {
+/** A rate as a percentage, or the words that mean there was no denominator. */
+export function formatRate(value: number | null | undefined): string {
+ return value === null || value === undefined ? 'not measured' : `${(value * 100).toFixed(1)}%`;
+}
+
+export interface TrackRecordSummary {
+ /** Whether enough has been measured to say anything at all. */
+ measured: boolean;
+ agreement: number | null;
+ maliciousRecall: number | null;
+ abstention: number | null;
+ answered: number;
+ labelled: number;
+ maliciousSupport: number;
+ /** Progress toward the sample floors, as counts rather than a percentage. */
+ decisionsNeeded: number;
+ maliciousNeeded: number;
+ windowDays: number;
+ /** The trailing slice, which is where a gradual decline shows first. */
+ recentAgreement: number | null;
+ recentAnswered: number;
+}
+
+/**
+ * Reduce the agreement response to what this card shows.
+ *
+ * Pure, and separate from the component, for the same reason
+ * `computeScorecard` is: the interesting decisions here are about what counts
+ * as measured and what a missing denominator renders as, and those are worth
+ * asserting directly rather than through a DOM query.
+ */
+export function summariseTrackRecord(agreement: AgreementResponse | null | undefined): TrackRecordSummary | null {
+ if (!agreement) return null;
+ const { window: w, recent, thresholds } = agreement;
+ return {
+ // A tenant with closures but no *labelled* ones has been measured and has
+ // nothing to show for it, which is a real state and a different one from
+ // never having started.
+ measured: w.resolved > 0,
+ agreement: w.agreement_rate,
+ maliciousRecall: w.malicious_recall,
+ abstention: w.abstention_rate,
+ answered: w.answered,
+ labelled: w.labelled,
+ maliciousSupport: w.malicious_support,
+ decisionsNeeded: thresholds.min_decisions,
+ maliciousNeeded: thresholds.min_malicious,
+ windowDays: thresholds.window_days,
+ recentAgreement: recent.agreement_rate,
+ recentAnswered: recent.answered,
+ };
+}
+
+export function AutonomyScorecard({
+ actions,
+ agreement,
+}: {
+ actions: AutonomyActionPolicy[];
+ agreement?: AgreementResponse | null;
+}) {
const card = computeScorecard(actions);
const isCopilot = card.posture === 'copilot';
+ const record = summariseTrackRecord(agreement);
return (
<div
@@ -129,15 +205,72 @@ export function AutonomyScorecard({ actions }: { actions: AutonomyActionPolicy[]
</span>
))}
</div>
+
+ <TrackRecord record={record} />
+ </div>
+ );
+}
+
+function TrackRecord({ record }: { record: TrackRecordSummary | null }) {
+ if (record === null || !record.measured) {
+ return (
+ <p className="mt-4 border-t border-gray-800 pt-3 text-xs text-gray-500">
+ No measured track record yet. Enable shadow mode for an alert class and the agent&apos;s
+ verdicts will be scored against your analysts&apos; own closures, here and in the source SIEM.
+ </p>
+ );
+ }
+
+ return (
+ <div className="mt-4 border-t border-gray-800 pt-3">
+ <p className="text-[11px] uppercase tracking-wide text-gray-500">
+ Measured track record, last {record.windowDays} days
+ </p>
+ <dl className="mt-2 flex flex-wrap gap-x-6 gap-y-2">
+ <Stat
+ label="Agreement"
+ value={formatRate(record.agreement)}
+ hint={`${record.answered} answered`}
+ />
+ <Stat
+ label="Recall on malicious"
+ value={formatRate(record.maliciousRecall)}
+ hint={`${record.maliciousSupport} true positives`}
+ />
+ <Stat label="Abstained" value={formatRate(record.abstention)} hint={`${record.labelled} labelled`} />
+ <Stat
+ label="Sample"
+ /* Counts, not a percentage. On a card where every other figure is a
+ percentage, "47%" would be read as a fifth accuracy number. */
+ value={`${record.labelled} / ${record.decisionsNeeded}`}
+ hint={`${record.maliciousSupport} / ${record.maliciousNeeded} malicious`}
+ />
+ </dl>
+ <p className="mt-2 text-[11px] text-gray-500">
+ Most recent {record.recentAnswered} answered: {formatRate(record.recentAgreement)} agreement.
+ Shown apart from the window because a decline that began this week is still
+ absorbed by a month of earlier agreement.
+ </p>
</div>
);
}
-function Stat({ label, value }: { label: string; value: number }) {
+function Stat({
+ label,
+ value,
+ hint,
+}: {
+ label: string;
+ value: number | string;
+ hint?: string;
+}) {
return (
<div>
<dt className="text-[11px] uppercase tracking-wide text-gray-500">{label}</dt>
<dd className="font-mono text-lg tabular-nums text-gray-100">{value}</dd>
+ {/* Every rate on this card travels with the count it was computed over.
+ 100% over four answers is not the claim 100% over four hundred is. */}
+ {hint ? <p className="text-[11px] text-gray-500">{hint}</p> : null}
</div>
);
}
diff --git a/apps/web/src/lib/api.ts b/apps/web/src/lib/api.ts
index bd372a819..9419107ce 100644
--- a/apps/web/src/lib/api.ts
+++ b/apps/web/src/lib/api.ts
@@ -5025,8 +5025,105 @@ export const autonomyPolicyApi = {
`/api/v1/autonomy-policy/${encodeURIComponent(action)}`,
{ method: 'DELETE' },
),
+
+ /** Which alert classes this tenant is measuring rather than acting on. */
+ shadowMode: () =>
+ request<ShadowModeResponse>('/api/v1/autonomy-policy/shadow-mode'),
+
+ /** Start or stop measuring one alert class. */
+ setShadowMode: (alertClass: string, enabled: boolean) =>
+ request<ShadowModeEntry>(
+ `/api/v1/autonomy-policy/shadow-mode/${encodeURIComponent(alertClass)}`,
+ { method: 'PUT', body: JSON.stringify({ enabled }) },
+ ),
+
+ /** Rolling agreement between the agent and this tenant's own analysts. */
+ agreement: (params?: { scope_kind?: string; scope_key?: string }) => {
+ const query = new URLSearchParams();
+ if (params?.scope_kind) query.set('scope_kind', params.scope_kind);
+ if (params?.scope_key) query.set('scope_key', params.scope_key);
+ const suffix = query.toString();
+ return request<AgreementResponse>(
+ `/api/v1/autonomy-policy/agreement${suffix ? `?${suffix}` : ''}`,
+ );
+ },
};
+// ─── Shadow mode and rolling agreement (gap-closure Phase 2.1 / 2.2) ─────────
+//
+// Every rate below is `number | null`, and `null` means there was no
+// denominator. It renders as "not measured", never as 0: a zero in the
+// agreement column says the agent was wrong every time, and "it was never
+// asked" is a different fact calling for a different response. The counts
+// travel with the rates for the same reason — 100% over four answers is not
+// the claim 100% over four hundred is.
+
+export interface ShadowModeEntry {
+ alert_class: string;
+ enabled: boolean;
+ enabled_at?: string | null;
+ updated_at?: string | null;
+ updated_by?: string | null;
+}
+
+export interface ShadowModeResponse {
+ tenant_id: string;
+ entries: ShadowModeEntry[];
+}
+
+export interface AgreementWindow {
+ resolved: number;
+ labelled: number;
+ unlabeled: number;
+ answered: number;
+ abstained: number;
+ agreed: number;
+ malicious_support: number;
+ malicious_caught: number;
+ agreement_rate: number | null;
+ malicious_recall: number | null;
+ abstention_rate: number | null;
+}
+
+export interface AgreementScope {
+ key: string;
+ window: AgreementWindow;
+}
+
+export interface AgreementThresholds {
+ min_decisions: number;
+ min_malicious: number;
+ min_agreement: number;
+ min_malicious_recall: number;
+ max_abstention_rate: number;
+ window_days: number;
+ demotion_agreement: number;
+ demotion_malicious_recall: number;
+ drift_sample: number;
+ drift_min_answered: number;
+}
+
+export interface AgreementResponse {
+ tenant_id: string;
+ scope_kind: string;
+ scope_key: string;
+ window: AgreementWindow;
+ /**
+ * The trailing slice of the most recent decisions, scored on its own. A
+ * window average is where a gradual decline hides, so any surface showing
+ * one has to show both or it is showing the flattering half.
+ */
+ recent: AgreementWindow;
+ window_start: string;
+ window_end: string;
+ thresholds: AgreementThresholds;
+ reconciled: number;
+ by_alert_class: AgreementScope[];
+ by_rule: AgreementScope[];
+ by_source: AgreementScope[];
+ by_model: AgreementScope[];
+}
+
// ─── Analyst override feedback loop (Tier 1.5) ───────────────────────────────
//
// When an analyst corrects an AI verdict, the API:
diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.md
index c7d153146..562709fb6 100644
--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md
+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md
@@ -163,11 +163,14 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
| A report field the provider does not publish reads "unavailable", never zero or empty | `operations/sandbox-providers.md` (provider contract) | `ci.yml :: api tests` (`test_sandbox_providers.py`, driven against a **recorded** report whose `attackTechniques` is `[]` beside `behavior.analyzed: false`; the adapter must return `UNAVAILABLE` with the stage's own reason attached, and the same payload with the stage marked run must return a real empty tuple. A missing score must not read as 0 and a missing IOC section must not read as "none found") | GATED | the recorded payload covers one provider and one shape of absence; CAPEv2's equivalents are synthetic because no live CAPEv2 instance was available |
| Replay evaluation runs the production triage path, not a copy of it, and writes nothing while it does | `apps/docs/docs/evaluation/replay.md`; `services/agents/app/replay/runner.py` | `ci.yml` agents job (`test_replay_shadow_no_writes.py` — every durable write on the triage path is monkeypatched on the **real** module to raise a `BaseException`, then a replay is run: a `BaseException` because the worker wraps most of its writes in `contextlib.suppress(Exception)`, which is correct production behaviour and is exactly what would let a leaked write pass a gentler guard. Zero writes is asserted as a *count* rather than as an absence, since a replay that stopped replaying and a replay whose writes were suppressed both write nothing and only the counter tells them apart. That the path is production's is pinned two ways: the worker's default sinks are asserted to be `LiveTriageWriter`/`LiveTriageContextReader`, and an AST pass over `services/agents/app/main.py` asserts the deployed worker is constructed without overriding either. `test_escalation_cannot_change_the_graded_verdict` drives the real worker with a graph runner that rewrites the verdict and asserts the returned summary is unmoved, which is what makes declining escalation in shadow mode sound rather than convenient) | GATED | Triage only. Fusion's correlation, fused confidence score, narrative and entity resolution are absent from a replayed finding; `ENVELOPE_LIMITS` is published in every report's method section rather than left as a comment |
| The test window of a replay cannot influence its own triage | `apps/docs/docs/evaluation/replay.md` (point-in-time context); `services/agents/app/replay/shadow.py` | `ci.yml` agents job (`test_replay_leakage.py` — three routes from a test-window decision back into a later one, each with its own test because one combined assertion would not say which broke: the outcome-memory prior that auto-closes a repeat without re-triage, the cost governor's dedup cache that short-circuits earlier still and leaves no ledger row or prior behind, and organisation memory read live into the prompt. Each is checked for **sensitivity** as well as for the guard: the memory test runs the identical corpus twice, once with the shadow sinks and once with a live reader and a recording writer, and asserts the second leaks at tier `memory`; the statements test asserts the frozen reader hands the agent the snapshot and that the same worker with the live reader picks up the poison. `capture_context` drops rows recorded after the split in both stores, and a snapshot taken at any other instant is refused rather than used) | GATED | Organisation-memory statements as `GET /feedback/context-statements` serves them carry no creation time, so the split filter has nothing to test them against and they are kept. The count is published as `statements_without_timestamp` in the report's method section rather than the freeze being claimed tighter than it is |
+| A shadow verdict never fills in the analyst column it will be graded against | `apps/docs/docs/operations/shadow-mode.md`; `services/agents/app/workers/shadow_mode.py` | `ci.yml` agents job (`test_shadow_mode.py`) - the shadow sink is asserted to decline every acting method against a delegate that *records* rather than performs, so a forwarded call shows up as a recorded one rather than as an absence; `persist_auto_triage` is asserted to force `auto_closed` off and the `shadow` flag on; and the ledger statement is read to confirm `disposition`, `status` and `resolved_at` are each guarded by that flag with `NOT $10` rather than by the caller remembering to pass `auto_closed=False`. Plus `ci.yml` api job (`test_shadow_agreement.py`) - reconciliation only reads closed alerts and only writes decisions with `resolved_at IS NULL`, so a later change of mind cannot rewrite evidence a promotion already rested on | GATED | The SQL guard is read, not executed: there is no Postgres in the unit suites, so what is proven is that the statement carries the guard and binds the flag, not that Postgres evaluates the CASE as written. A live replay of the statement belongs with the isolation suite |
+| Agreement cannot be improved by declining to answer | `apps/docs/docs/operations/shadow-mode.md`; `services/actions/app/services/autonomy_evidence_rules.py` | `ci.yml` actions job (`test_autonomy_evidence_rules.py`) - the same ten right answers are scored with nothing declined and with ninety declined, and the agreement rate is asserted identical, because abstaining removes a decision from the numerator and the denominator together. The counterweight is asserted from the other side: a window where the agent answered ten of forty labelled cases posts a perfect agreement rate and a malicious recall of 8/30, since recall's denominator is every malicious case whether answered or not. The abstention rate is asserted to move where agreement does not, and `AgreementWindow` refuses to construct at all when its counts do not add up, so a miscounted aggregate raises instead of publishing a plausible rate | GATED | The metric is gated; what it is used *for* is Phase 2.3. This row proves the rate cannot be gamed, not that anything yet refuses a promotion on it |
+| Live agreement is scored by the Phase 1 metrics rather than a second set | `plans/aisoc_gap_closure_plan.plan.md` (2.2, "using the Phase 1 metrics"); `apps/docs/docs/operations/shadow-mode.md` | `ci.yml :: python-lint` (`check_replay_contract_parity.py`, extended from three trees to four: `GRADED_DISPOSITIONS`, `ABSTENTION_VERDICTS` and the `MALICIOUS` spelling must be identical strings in `autonomy_evidence_rules.py` and in `aisoc_benchmark/replay.py`, compared in both directions, and `UNLABELED` must be spelled the same in all three. Proven capable of failing by drifting each of the two collections in turn and watching it fire, rather than only having never fired) + `ci.yml` (`sync_vendored_autonomy_evidence.py --check`, which byte-compares the API's copy, because a safety control defined twice is off in whichever copy is more generous) | GATED | Parity of the definitions is gated. That the two implementations compute the same number from the same rows is not: the grader scores a list of decision dicts and the live path aggregates in SQL, so they share a vocabulary rather than a code path |
| A replay report prints "not measured" where there was no denominator, and no headline accuracy on a corpus too thin to support one | `apps/docs/docs/evaluation/replay.md`; `packages/aisoc-benchmark/aisoc_benchmark/replay.py` | `ci.yml` benchmark job (`test_replay_metrics.py` — a class the agent never predicted reports `None` precision and renders as "not measured", because `0.0` in a recall column says the agent missed every case and "never asked" is a different fact; below 30 malicious cases the headline is withheld with the count and the reason, and the rendered report is asserted to contain no percentage in that section. Three populations are asserted apart: unlabelled findings excluded from accuracy and counted, triage refusals scored as neither answer nor abstention, and abstentions counted against malicious recall. A verdict outside the taxonomy gets its own confusion-matrix column rather than being folded into one the agent did not say. Bootstrap intervals are seeded and the seed and resample count travel in the report, so two runs over one corpus render byte-identically) | GATED | Hallucination extraction reuses the existing `_INDICATOR_PATTERNS`, which over-counts rather than under-counts: a phrase shaped like a domain is checked and, if absent from the evidence, counted. The published rate is a ceiling and the per-decision list travels with it so a disputed count can be re-derived |
## Summary
-- GATED: 148
+- GATED: 151
- PARTIAL: 8
- NO GATE: 0 (**every claim is backed by a failing test.** The last NO GATE — the weekly benchmark scoreboard running live against `main` — closed in Phase E1: `scripts/check_scoreboard.py` ties the published scoreboard to a deterministic per-PR live-agent MITRE-accuracy run, while the funded weekly `wet-eval.yml` appends the LLM-tier rows. The remaining PARTIAL rows are honest, named deferrals — each states the specific gap and what closes it — not unproven claims.)
diff --git a/docs/openapi.yaml b/docs/openapi.yaml
index 3a9ccade3..0a6676e25 100644
--- a/docs/openapi.yaml
+++ b/docs/openapi.yaml
@@ -10798,6 +10798,127 @@ paths:
application/json:
schema:
$ref: '#/components/schemas/HTTPValidationError'
+ /api/v1/autonomy-policy/shadow-mode:
+ get:
+ tags:
+ - autonomy-policy
+ summary: Get Shadow Mode
+ description: Which alert classes this tenant is measuring rather than acting
+ on.
+ operationId: get_shadow_mode_api_v1_autonomy_policy_shadow_mode_get
+ responses:
+ '200':
+ description: Successful Response
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ShadowModeResponse'
+ security:
+ - HTTPBearer: []
+ /api/v1/autonomy-policy/shadow-mode/{alert_class}:
+ put:
+ tags:
+ - autonomy-policy
+ summary: Set Shadow Mode
+ description: 'Start or stop measuring one alert class.
+
+
+ ``enabled_at`` is stamped on the transition into shadow and cleared on the
+
+ way out, rather than being set on every write. A promotion window that
+
+ reaches further back than the day measurement started is reaching back
+
+ before there was anything to measure, and the only way a later reader can
+
+ tell is if this column records the real start.'
+ operationId: set_shadow_mode_api_v1_autonomy_policy_shadow_mode__alert_class__put
+ security:
+ - HTTPBearer: []
+ parameters:
+ - name: alert_class
+ in: path
+ required: true
+ schema:
+ type: string
+ title: Alert Class
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ShadowModeUpdateRequest'
+ responses:
+ '200':
+ description: Successful Response
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ShadowModeEntry'
+ '422':
+ description: Validation Error
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/HTTPValidationError'
+ /api/v1/autonomy-policy/agreement:
+ get:
+ tags:
+ - autonomy-policy
+ summary: Get Agreement
+ description: 'Rolling agreement between the agent and this tenant''s own analysts.
+
+
+ The tenant comes from the credential. ``scope_kind`` and ``scope_key``
+
+ narrow *within* that tenant and cannot widen beyond it: the scope names a
+
+ dimension from a fixed vocabulary, never a column, and the tenant
+
+ predicate is applied whatever the scope says.
+
+
+ Closures made in this console are reconciled on the way in rather than by
+
+ a background job. The sweep is bounded and indexed, and doing it here
+
+ means the number an operator is looking at includes the alert they closed
+
+ a minute ago, which is exactly when they come to look.'
+ operationId: get_agreement_api_v1_autonomy_policy_agreement_get
+ security:
+ - HTTPBearer: []
+ parameters:
+ - name: scope_kind
+ in: query
+ required: false
+ schema:
+ type: string
+ description: tenant, alert_class, rule, source or model
+ default: tenant
+ title: Scope Kind
+ description: tenant, alert_class, rule, source or model
+ - name: scope_key
+ in: query
+ required: false
+ schema:
+ type: string
+ maxLength: 200
+ default: '*'
+ title: Scope Key
+ responses:
+ '200':
+ description: Successful Response
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/AgreementResponse'
+ '422':
+ description: Validation Error
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/HTTPValidationError'
/api/v1/business-context/rules:
get:
tags:
@@ -15300,6 +15421,149 @@ components:
- connector_count
title: AgentToolsResponse
description: Tenant-wide tool catalogue surfaced to the agent layer.
+ AgreementResponse:
+ properties:
+ tenant_id:
+ type: string
+ title: Tenant Id
+ scope_kind:
+ type: string
+ title: Scope Kind
+ scope_key:
+ type: string
+ title: Scope Key
+ window:
+ $ref: '#/components/schemas/AgreementWindowModel'
+ recent:
+ $ref: '#/components/schemas/AgreementWindowModel'
+ window_start:
+ type: string
+ title: Window Start
+ window_end:
+ type: string
+ title: Window End
+ thresholds:
+ additionalProperties:
+ anyOf:
+ - type: number
+ - type: integer
+ type: object
+ title: Thresholds
+ reconciled:
+ type: integer
+ title: Reconciled
+ by_alert_class:
+ items:
+ $ref: '#/components/schemas/AgreementScopeModel'
+ type: array
+ title: By Alert Class
+ by_rule:
+ items:
+ $ref: '#/components/schemas/AgreementScopeModel'
+ type: array
+ title: By Rule
+ by_source:
+ items:
+ $ref: '#/components/schemas/AgreementScopeModel'
+ type: array
+ title: By Source
+ by_model:
+ items:
+ $ref: '#/components/schemas/AgreementScopeModel'
+ type: array
+ title: By Model
+ type: object
+ required:
+ - tenant_id
+ - scope_kind
+ - scope_key
+ - window
+ - recent
+ - window_start
+ - window_end
+ - thresholds
+ - reconciled
+ - by_alert_class
+ - by_rule
+ - by_source
+ - by_model
+ title: AgreementResponse
+ AgreementScopeModel:
+ properties:
+ key:
+ type: string
+ title: Key
+ window:
+ $ref: '#/components/schemas/AgreementWindowModel'
+ type: object
+ required:
+ - key
+ - window
+ title: AgreementScopeModel
+ AgreementWindowModel:
+ properties:
+ resolved:
+ type: integer
+ title: Resolved
+ labelled:
+ type: integer
+ title: Labelled
+ unlabeled:
+ type: integer
+ title: Unlabeled
+ answered:
+ type: integer
+ title: Answered
+ abstained:
+ type: integer
+ title: Abstained
+ agreed:
+ type: integer
+ title: Agreed
+ malicious_support:
+ type: integer
+ title: Malicious Support
+ malicious_caught:
+ type: integer
+ title: Malicious Caught
+ agreement_rate:
+ anyOf:
+ - type: number
+ - type: 'null'
+ title: Agreement Rate
+ malicious_recall:
+ anyOf:
+ - type: number
+ - type: 'null'
+ title: Malicious Recall
+ abstention_rate:
+ anyOf:
+ - type: number
+ - type: 'null'
+ title: Abstention Rate
+ type: object
+ required:
+ - resolved
+ - labelled
+ - unlabeled
+ - answered
+ - abstained
+ - agreed
+ - malicious_support
+ - malicious_caught
+ title: AgreementWindowModel
+ description: 'Counts, and the rates derived from them.
+
+
+ Every rate is optional and is ``null`` when its denominator was zero. The
+
+ console renders that as "not measured". A zero here would say the agent
+
+ was wrong every time, which is a different fact with a different remedy,
+
+ and it is the more flattering of the two to print by accident in the
+
+ abstention column and the more damning in the agreement column.'
AirgapStatus:
properties:
ready:
@@ -27506,6 +27770,58 @@ components:
- findings_closed
- error_message
title: ScanRunOut
+ ShadowModeEntry:
+ properties:
+ alert_class:
+ type: string
+ title: Alert Class
+ enabled:
+ type: boolean
+ title: Enabled
+ enabled_at:
+ anyOf:
+ - type: string
+ - type: 'null'
+ title: Enabled At
+ updated_at:
+ anyOf:
+ - type: string
+ - type: 'null'
+ title: Updated At
+ updated_by:
+ anyOf:
+ - type: string
+ - type: 'null'
+ title: Updated By
+ type: object
+ required:
+ - alert_class
+ - enabled
+ title: ShadowModeEntry
+ ShadowModeResponse:
+ properties:
+ tenant_id:
+ type: string
+ title: Tenant Id
+ entries:
+ items:
+ $ref: '#/components/schemas/ShadowModeEntry'
+ type: array
+ title: Entries
+ type: object
+ required:
+ - tenant_id
+ - entries
+ title: ShadowModeResponse
+ ShadowModeUpdateRequest:
+ properties:
+ enabled:
+ type: boolean
+ title: Enabled
+ type: object
+ required:
+ - enabled
+ title: ShadowModeUpdateRequest
ShiftAnalyst:
properties:
id:
diff --git a/scripts/check_replay_contract_parity.py b/scripts/check_replay_contract_parity.py
index 0a098f83f..4b6458dd1 100644
--- a/scripts/check_replay_contract_parity.py
+++ b/scripts/check_replay_contract_parity.py
@@ -6,9 +6,12 @@ Gap-closure Phase 1.2 and 1.3.
``services/actions`` reads a customer's closed findings and produces
``ClosedFinding``. ``services/agents`` replays them and consumes
``HistoricalFinding``. ``packages/aisoc-benchmark`` grades the result against
-``GRADED_DISPOSITIONS``. All three package their code as top-level ``app`` or
-as a standalone distribution, so none of them can import the others, and the
-contract between them is three declarations that agree by convention.
+``GRADED_DISPOSITIONS``. Phase 2.2 added a fourth: live shadow-mode agreement,
+measured against the same taxonomy in
+``services/actions/app/services/autonomy_evidence_rules.py``. All of them
+package their code as top-level ``app`` or as a standalone distribution, so
+none can import the others, and the contract between them is a handful of
+declarations that agree by convention.
Conventions drift. This gate reads all three with ``ast`` and compares them
**in both directions**, because the failure this tree keeps finding is a gate
@@ -43,6 +46,25 @@ _REPLAY = ("services/agents/app/replay/findings.py", "HistoricalFinding")
_TAXONOMY_SOURCE = ("services/actions/app/services/disposition_writeback.py", "CANONICAL_DISPOSITIONS")
_GRADER_SOURCE = ("packages/aisoc-benchmark/aisoc_benchmark/replay.py", "GRADED_DISPOSITIONS")
+#: Gap-closure Phase 2.2 added a fourth tree. Live shadow-mode agreement is
+#: measured in ``services/actions`` and rendered by ``services/api``, and the
+#: plan says to use the Phase 1 metrics rather than to write a second set.
+#: "Use" is a claim, and this is what makes it enforceable: the three
+#: definitions that decide what agreement *means* must be the same strings in
+#: the live module and in the grader. If they diverge, the replay report and
+#: the live scorecard describe different agents while both look right, and the
+#: promotion gate reads whichever one it happens to import.
+_LIVE_METRICS = ("services/actions/app/services/autonomy_evidence_rules.py", "GRADED_DISPOSITIONS")
+_LIVE_ABSTENTIONS = (_LIVE_METRICS[0], "ABSTENTION_VERDICTS")
+_GRADER_ABSTENTIONS = (_GRADER_SOURCE[0], "ABSTENTION_VERDICTS")
+
+#: The one disposition whose spelling the promotion thresholds are written in
+#: terms of ("at least 30 of them malicious"). A rename on one side alone
+#: would leave the gate counting a class nothing produces, and a count of zero
+#: refuses every promotion, which looks like a conservative gate rather than a
+#: broken one.
+_MALICIOUS = "MALICIOUS"
+
#: ``CANONICAL_DISPOSITIONS`` covers everything a verdict may be, including the
#: two that route to a human. The grader's ``GRADED_DISPOSITIONS`` is only the
#: subset an analyst can *close* a finding as, because "needs review" is not an
@@ -51,9 +73,9 @@ _GRADER_SOURCE = ("packages/aisoc-benchmark/aisoc_benchmark/replay.py", "GRADED_
#: a third does not silently widen the exemption.
_NOT_A_CLOSING_LABEL = frozenset({"needs_review", "escalate"})
-#: The spelling of "the analyst declined to classify", which both trees hold
-#: as a module constant and neither may rename alone.
-_UNLABELED = ("unlabeled", [_READER[0], _REPLAY[0]])
+#: The spelling of "the analyst declined to classify", which every tree holds
+#: as a module constant and none may rename alone.
+_UNLABELED = ("unlabeled", [_READER[0], _REPLAY[0], _LIVE_METRICS[0]])
def _module(root: Path, relative: str) -> ast.Module:
@@ -145,6 +167,29 @@ def main(argv: list[str] | None = None) -> int:
if invented:
failures.append(f"{_GRADER_SOURCE[1]} holds {invented}, which the canonical taxonomy does not define")
+ live_graded = _string_members(_module(root, _LIVE_METRICS[0]), _LIVE_METRICS[1], _LIVE_METRICS[0])
+ checked += len(live_graded)
+ if live_graded != graded:
+ failures.append(
+ f"{_LIVE_METRICS[0]} grades {sorted(live_graded)} and {_GRADER_SOURCE[0]} grades {sorted(graded)}: "
+ f"live agreement and replay scoring would report on different populations"
+ )
+
+ live_abstentions = _string_members(_module(root, _LIVE_ABSTENTIONS[0]), _LIVE_ABSTENTIONS[1], _LIVE_ABSTENTIONS[0])
+ grader_abstentions = _string_members(_module(root, _GRADER_ABSTENTIONS[0]), _GRADER_ABSTENTIONS[1], _GRADER_ABSTENTIONS[0])
+ checked += len(live_abstentions) + len(grader_abstentions)
+ if live_abstentions != grader_abstentions:
+ failures.append(
+ f"ABSTENTION_VERDICTS differ: {_LIVE_ABSTENTIONS[0]} holds {sorted(live_abstentions)}, "
+ f"{_GRADER_ABSTENTIONS[0]} holds {sorted(grader_abstentions)}. A verdict that is an abstention on one "
+ f"side and an answer on the other is scored as wrong by exactly one of them"
+ )
+
+ malicious = {relative: _constant(_module(root, relative), _MALICIOUS, relative) for relative in (_LIVE_METRICS[0], _GRADER_SOURCE[0])}
+ checked += len(malicious)
+ if len(set(malicious.values())) != 1:
+ failures.append(f"MALICIOUS is spelled inconsistently across the trees that count it: {malicious}")
+
name, files = _UNLABELED
spellings = {relative: _constant(_module(root, relative), "UNLABELED", relative) for relative in files}
checked += len(spellings)
@@ -162,8 +207,9 @@ def main(argv: list[str] | None = None) -> int:
return 1
print(
- f"OK: replay contract agrees across three trees "
- f"({len(reader_fields)} finding fields, {len(graded)} graded dispositions, {len(spellings)} UNLABELED spellings)"
+ f"OK: replay contract agrees across four trees "
+ f"({len(reader_fields)} finding fields, {len(graded)} graded dispositions, "
+ f"{len(live_abstentions)} abstention verdicts, {len(spellings)} UNLABELED spellings)"
)
return 0
diff --git a/scripts/sync_vendored_autonomy_evidence.py b/scripts/sync_vendored_autonomy_evidence.py
new file mode 100755
index 000000000..39798d14c
--- /dev/null
+++ b/scripts/sync_vendored_autonomy_evidence.py
@@ -0,0 +1,96 @@
+#!/usr/bin/env python3
+"""Keep the API's copy of the autonomy evidence rules byte-identical to the source.
+
+Gap-closure Phase 2.3.
+
+``services/actions/app/services/autonomy_evidence_rules.py`` decides what a
+track record has to show before a tenant may act unattended. Two services need
+that answer and neither can import the other: ``services/actions`` enforces it
+at dispatch, ``services/api`` decides it at promotion time and owns the
+hash-chained audit log the transition is written to. Both package their code
+as top-level ``app``, and each Docker image is built with only its own service
+directory as the build context, so the file has to exist twice.
+
+A safety control that exists twice and is allowed to differ is a control that
+is off in whichever copy is more generous, and nobody finds out until an
+action executes that the other half would have refused. So the copy is
+byte-compared rather than spot-checked, and ``--check`` runs in CI.
+
+Run modes
+---------
+* ``python scripts/sync_vendored_autonomy_evidence.py`` copy source to vendored.
+* ``python scripts/sync_vendored_autonomy_evidence.py --check`` fail when they differ.
+
+Same shape and same reasoning as ``sync_vendored_llm_contract.py``; dependency
+free so it runs on a bare interpreter before any install step.
+
+AiSOC, open-source AI Security Operations Center (MIT License).
+"""
+
+from __future__ import annotations
+
+import argparse
+import filecmp
+import shutil
+import sys
+from pathlib import Path
+
+# `scripts/` is on sys.path when this file is run as a program, but not when a
+# test loads it by path with importlib. gate_toolkit sits beside it either way.
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+
+from gate_toolkit import repo_root, self_test_if_requested # noqa: E402
+
+self_test_if_requested(__file__)
+
+REPO_ROOT = repo_root()
+SOURCE_FILE = REPO_ROOT / "services" / "actions" / "app" / "services" / "autonomy_evidence_rules.py"
+VENDORED_FILE = REPO_ROOT / "services" / "api" / "app" / "_vendor" / "autonomy_evidence_rules.py"
+
+
+def _check() -> int:
+ if not SOURCE_FILE.is_file():
+ print(f"FAIL: source file missing: {SOURCE_FILE}", file=sys.stderr)
+ return 1
+ if not VENDORED_FILE.is_file():
+ print(f"FAIL: vendored file missing: {VENDORED_FILE}", file=sys.stderr)
+ return 1
+ if not filecmp.cmp(SOURCE_FILE, VENDORED_FILE, shallow=False):
+ print(
+ "FAIL: vendored autonomy_evidence_rules.py is out of sync with source.\n"
+ f" Source: {SOURCE_FILE.relative_to(REPO_ROOT)}\n"
+ f" Vendored: {VENDORED_FILE.relative_to(REPO_ROOT)}\n\n"
+ "The promotion gate would then mean two different things: one answer at\n"
+ "dispatch and another at promotion time.\n"
+ "Re-run: python scripts/sync_vendored_autonomy_evidence.py",
+ file=sys.stderr,
+ )
+ return 1
+ print("OK: vendored autonomy_evidence_rules.py matches source.")
+ return 0
+
+
+def _sync() -> int:
+ if not SOURCE_FILE.is_file():
+ print(f"FAIL: source file missing: {SOURCE_FILE}", file=sys.stderr)
+ return 1
+ VENDORED_FILE.parent.mkdir(parents=True, exist_ok=True)
+ shutil.copy2(SOURCE_FILE, VENDORED_FILE)
+ print(f"copied {SOURCE_FILE.relative_to(REPO_ROOT)} -> {VENDORED_FILE.relative_to(REPO_ROOT)}")
+ print("\nDone. Commit services/api/app/_vendor/autonomy_evidence_rules.py.")
+ return 0
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description="Sync the vendored autonomy evidence rules.")
+ parser.add_argument(
+ "--check",
+ action="store_true",
+ help="Verify the vendored copy matches the source; do not write.",
+ )
+ args = parser.parse_args()
+ return _check() if args.check else _sync()
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/services/actions/app/services/autonomy_evidence_rules.py b/services/actions/app/services/autonomy_evidence_rules.py
new file mode 100644
index 000000000..2256f4ff7
--- /dev/null
+++ b/services/actions/app/services/autonomy_evidence_rules.py
@@ -0,0 +1,386 @@
+"""How agreement between the agent and a tenant's analysts is measured, decided once.
+
+Gap-closure Phase 2.2. The promotion gate that reads these numbers arrives in
+Phase 2.3 and sits on top of this file rather than beside it.
+
+Two services have to agree about this and neither can import the other.
+``services/actions`` enforces autonomy at dispatch, where an over-generous
+answer executes something at a customer's vendor. ``services/api`` renders the
+track record and decides promotions, owns the tenant session and owns the
+hash-chained audit log a transition is written to. Both package their code as
+top-level ``app``, so one process holds one of them, and each Docker image is
+built with only its own service directory as context.
+
+So this module is pure: standard library only, no database handle, no HTTP
+client, no service import. ``services/api`` carries a byte-identical copy at
+``app/_vendor/autonomy_evidence_rules.py`` and
+``scripts/sync_vendored_autonomy_evidence.py --check`` fails the build when the
+two drift. Same arrangement as the LLM input contract and the deterministic
+narrative, and used here for a stronger reason: a safety control that two
+services define differently is a control that is off in whichever one is more
+generous, and nobody finds out until something executes that the other half
+would have refused.
+
+The metrics are Phase 1's, not a second set
+===========================================
+
+``packages/aisoc-benchmark/aisoc_benchmark/replay.py`` already defines what
+"malicious recall" means, which verdicts are abstentions, which dispositions
+may be graded, and that a rate with no denominator is ``None`` rather than
+zero. Restating any of those differently here would give live measurement a
+private definition of accuracy, and the first time the two disagreed the
+replay report and the scorecard would be describing different agents while
+both looked right. ``scripts/check_replay_contract_parity.py`` compares the
+constants below against that module in both directions.
+
+Agreement is computed over answered decisions only
+==================================================
+
+This is the one a naive implementation gets wrong, and it is worth stating
+where the arithmetic is rather than where it is read. If agreement counted
+every decision and scored an abstention as "not a disagreement", an agent that
+answered a tenth of its queue confidently and routed the rest to a human would
+post a near-perfect record on a population it never attempted. Abstaining
+removes a decision from the numerator and the denominator together, so it
+cannot move the rate; what it moves is
+:attr:`AgreementWindow.abstention_rate`, which is reported beside it.
+
+Malicious recall is the counterweight and runs the other way: its denominator
+is every malicious case, abstained or not, so an abstention counts there as a
+miss. Phase 1 makes exactly that distinction, for exactly that reason, and an
+alert routed to a human was not caught by the agent.
+
+A window and a trailing slice, never just the window
+====================================================
+
+A thirty-day window is an average, and an average is where a gradual decline
+hides: an agent that agreed 99% of the time for three weeks and 70% of the
+time this week still posts about 95% over the window. So the aggregate comes
+in two statements, one over the window and one over the most recent
+:attr:`PromotionThresholds.drift_sample` decisions, and every surface that
+shows one shows both. The trailing slice is a count rather than a date range
+so it means the same thing for a tenant with ten alerts a day and for one with
+ten thousand.
+
+The thresholds live here even though nothing in this file enforces them
+======================================================================
+
+:class:`PromotionThresholds` carries the window length, which this file needs,
+and the floors, which it does not. They travel together because the scorecard
+shows progress toward them ("47 of the 100 decisions, 12 of the 30 malicious
+ones") and because a floor stated in one service and enforced in another is
+the pair that drifts.
+"""
+
+from __future__ import annotations
+
+from dataclasses import asdict, dataclass
+from typing import Any
+
+__all__ = [
+ "ABSTENTION_VERDICTS",
+ "AGREEMENT_COUNTS_SQL",
+ "DEFAULT_THRESHOLDS",
+ "GRADED_DISPOSITIONS",
+ "MALICIOUS",
+ "RECENT_COUNTS_SQL",
+ "UNLABELED",
+ "AgreementWindow",
+ "PromotionThresholds",
+ "scoped_sql",
+ "to_named_params",
+ "window_from_counts",
+]
+
+#: The disposition that means "this was a real threat". One spelling, shared
+#: with the writeback taxonomy and the replay grader.
+MALICIOUS = "true_positive"
+
+#: What an analyst may close a finding as. Mirrors ``GRADED_DISPOSITIONS`` in
+#: the benchmark package and ``CANONICAL_DISPOSITIONS`` in the writeback, both
+#: pinned by ``scripts/check_replay_contract_parity.py``.
+GRADED_DISPOSITIONS: tuple[str, ...] = (
+ MALICIOUS,
+ "benign_true_positive",
+ "false_positive",
+ "benign",
+)
+
+#: An analyst who declined to classify. Excluded from agreement entirely,
+#: never guessed at.
+UNLABELED = "unlabeled"
+
+#: Agent outputs that route the alert to a human rather than deciding it.
+#: Counted apart from agreement so the two cannot be traded off invisibly.
+ABSTENTION_VERDICTS: frozenset[str] = frozenset({"needs_review", "escalate", "unknown", ""})
+
+
+@dataclass(frozen=True)
+class PromotionThresholds:
+ """What a tenant's track record has to show. Every value is configurable.
+
+ Defaults are the plan's: 100 decisions, at least 30 of them malicious.
+ The rest are set here rather than left to a caller because a threshold
+ with no default is a threshold somebody eventually passes zero for.
+ """
+
+ #: Labelled decisions required in the window.
+ min_decisions: int = 100
+ #: How many of those must have been closed by an analyst as malicious.
+ min_malicious: int = 30
+ #: Agreement over *answered* decisions required to promote.
+ min_agreement: float = 0.95
+ #: Recall on malicious required to promote. An abstention is a miss.
+ min_malicious_recall: float = 0.90
+ #: Share of labelled decisions that may be abstentions.
+ max_abstention_rate: float = 0.30
+ #: How far back the window reaches.
+ window_days: int = 30
+ #: Agreement below this demotes an existing grant.
+ demotion_agreement: float = 0.90
+ #: Malicious recall below this demotes an existing grant.
+ demotion_malicious_recall: float = 0.80
+ #: Size of the trailing slice scored separately, so gradual drift inside a
+ #: passing window is still caught.
+ drift_sample: int = 50
+ #: The trailing slice needs at least this many answered decisions before
+ #: it may demote on its own. Without it a single disagreement in a quiet
+ #: week would revoke a grant built on months of evidence.
+ drift_min_answered: int = 20
+
+ def as_dict(self) -> dict[str, Any]:
+ return asdict(self)
+
+
+DEFAULT_THRESHOLDS = PromotionThresholds()
+
+
+def _ratio(numerator: int, denominator: int) -> float | None:
+ """A rate, or ``None`` when there was nothing to divide by.
+
+ ``None`` renders as "not measured". A ``0.0`` in an agreement column says
+ the agent was wrong every time; no denominator says it was never asked,
+ and those call for opposite responses.
+ """
+ return (numerator / denominator) if denominator else None
+
+
+@dataclass(frozen=True)
+class AgreementWindow:
+ """Agreement between the agent and the analysts over one slice of history.
+
+ Every rate on this object is accompanied by the count it was computed
+ over, and is ``None`` when that count is zero. A caller that renders a
+ rate without its count has thrown away the difference between a
+ measurement and a coincidence.
+ """
+
+ #: Shadow decisions in the slice that an analyst has since closed.
+ resolved: int = 0
+ #: Of those, the ones carrying a disposition this platform can name.
+ labelled: int = 0
+ #: Labelled decisions the agent declined to decide.
+ abstained: int = 0
+ #: Labelled decisions the agent answered. The denominator of agreement.
+ answered: int = 0
+ #: Answered decisions where the agent's verdict is the analyst's label.
+ agreed: int = 0
+ #: Labelled decisions an analyst closed as malicious.
+ malicious_support: int = 0
+ #: Of those, the ones the agent also called malicious.
+ malicious_caught: int = 0
+
+ def __post_init__(self) -> None:
+ # A window whose parts do not add up is a bug in the aggregate query,
+ # and it would show as a plausible-looking rate rather than an error.
+ if self.answered + self.abstained != self.labelled:
+ raise ValueError(
+ f"answered ({self.answered}) + abstained ({self.abstained}) must equal "
+ f"labelled ({self.labelled}); the aggregate query and this type disagree"
+ )
+ if self.labelled > self.resolved:
+ raise ValueError(f"labelled ({self.labelled}) cannot exceed resolved ({self.resolved})")
+ if self.agreed > self.answered:
+ raise ValueError(f"agreed ({self.agreed}) cannot exceed answered ({self.answered})")
+ if self.malicious_caught > self.malicious_support:
+ raise ValueError(f"malicious_caught ({self.malicious_caught}) cannot exceed malicious_support ({self.malicious_support})")
+
+ @property
+ def unlabeled(self) -> int:
+ """Closures the analyst left unclassified. Excluded from every rate."""
+ return self.resolved - self.labelled
+
+ @property
+ def agreement_rate(self) -> float | None:
+ """Agreement over answered decisions only.
+
+ Abstentions are absent from both halves of this fraction. That is
+ what stops an agent from earning a grant by declining the hard cases:
+ declining removes a decision from the numerator and the denominator
+ together, so the rate does not move and the abstention rate does.
+ """
+ return _ratio(self.agreed, self.answered)
+
+ @property
+ def malicious_recall(self) -> float | None:
+ """Share of the analysts' malicious closures the agent also called malicious.
+
+ The denominator is every malicious case, answered or abstained, so an
+ abstention counts as a miss here even though it is excluded from
+ agreement. An alert routed to a human was not caught by the agent.
+ """
+ return _ratio(self.malicious_caught, self.malicious_support)
+
+ @property
+ def abstention_rate(self) -> float | None:
+ return _ratio(self.abstained, self.labelled)
+
+ def as_dict(self) -> dict[str, Any]:
+ """Counts and the rates derived from them, in one payload.
+
+ The derived rates are included rather than left to the reader because
+ this dict is what lands in the audit log, and a snapshot that stores
+ only counts requires whoever reads it in six months to re-derive the
+ arithmetic and hope they used the same definition.
+ """
+ return {
+ "resolved": self.resolved,
+ "labelled": self.labelled,
+ "unlabeled": self.unlabeled,
+ "answered": self.answered,
+ "abstained": self.abstained,
+ "agreed": self.agreed,
+ "malicious_support": self.malicious_support,
+ "malicious_caught": self.malicious_caught,
+ "agreement_rate": self.agreement_rate,
+ "malicious_recall": self.malicious_recall,
+ "abstention_rate": self.abstention_rate,
+ }
+
+
+def window_from_counts(counts: dict[str, Any]) -> AgreementWindow:
+ """Build a window from the aggregate query's row.
+
+ Takes a mapping rather than keyword arguments because both callers get one
+ from their driver, and asyncpg's ``Record`` and SQLAlchemy's ``RowMapping``
+ are both mappings while neither is a dict.
+ """
+ return AgreementWindow(
+ resolved=int(counts.get("resolved") or 0),
+ labelled=int(counts.get("labelled") or 0),
+ abstained=int(counts.get("abstained") or 0),
+ answered=int(counts.get("answered") or 0),
+ agreed=int(counts.get("agreed") or 0),
+ malicious_support=int(counts.get("malicious_support") or 0),
+ malicious_caught=int(counts.get("malicious_caught") or 0),
+ )
+
+
+# ---------------------------------------------------------------------------
+# The aggregate, written once
+# ---------------------------------------------------------------------------
+#
+# Both services run these two statements verbatim. Aggregating in the database
+# rather than fetching rows keeps the dispatch path from pulling a month of
+# decisions across the wire to divide two numbers, and writing the statement
+# here rather than in each service is what stops one of them from quietly
+# counting abstentions into agreement.
+#
+# The placeholders are numbered (`$1`) for asyncpg. SQLAlchemy callers pass the
+# statement through `text()` after `to_named_params()` renames them, because
+# the two drivers disagree about placeholder syntax and only about that.
+
+#: Scope filters are applied by the caller as an additional predicate, appended
+#: where this marker sits. Written as a marker rather than as string
+#: concatenation at the call site so both services splice at the same point.
+_SCOPE_MARKER = "/*scope*/"
+
+AGREEMENT_COUNTS_SQL = f"""
+SELECT
+ COUNT(*)::int AS resolved,
+ COUNT(*) FILTER (WHERE d.analyst_disposition = ANY($2::text[]))::int AS labelled,
+ COUNT(*) FILTER (
+ WHERE d.analyst_disposition = ANY($2::text[])
+ AND COALESCE(d.verdict, '') = ANY($3::text[])
+ )::int AS abstained,
+ COUNT(*) FILTER (
+ WHERE d.analyst_disposition = ANY($2::text[])
+ AND NOT (COALESCE(d.verdict, '') = ANY($3::text[]))
+ )::int AS answered,
+ COUNT(*) FILTER (
+ WHERE d.analyst_disposition = ANY($2::text[])
+ AND NOT (COALESCE(d.verdict, '') = ANY($3::text[]))
+ AND d.verdict = d.analyst_disposition
+ )::int AS agreed,
+ COUNT(*) FILTER (WHERE d.analyst_disposition = $4)::int AS malicious_support,
+ COUNT(*) FILTER (WHERE d.analyst_disposition = $4 AND d.verdict = $4)::int AS malicious_caught
+FROM aisoc_shadow_decisions d
+WHERE d.tenant_id = $1
+ AND d.resolved_at IS NOT NULL
+ AND d.resolved_at >= $5
+ AND d.resolved_at <= $6
+ {_SCOPE_MARKER}
+"""
+
+#: Same counts over the most recent ``$7`` resolved decisions. The slice is by
+#: count rather than by date so it means the same thing at ten alerts a day and
+#: at ten thousand, and it is ordered by ``resolved_at`` rather than by
+#: ``decided_at`` because a decision only joins the evidence when an analyst
+#: closes it.
+RECENT_COUNTS_SQL = f"""
+WITH recent AS (
+ SELECT d.verdict, d.analyst_disposition
+ FROM aisoc_shadow_decisions d
+ WHERE d.tenant_id = $1
+ AND d.resolved_at IS NOT NULL
+ AND d.resolved_at >= $5
+ AND d.resolved_at <= $6
+ {_SCOPE_MARKER}
+ ORDER BY d.resolved_at DESC, d.id DESC
+ LIMIT $7
+)
+SELECT
+ COUNT(*)::int AS resolved,
+ COUNT(*) FILTER (WHERE d.analyst_disposition = ANY($2::text[]))::int AS labelled,
+ COUNT(*) FILTER (
+ WHERE d.analyst_disposition = ANY($2::text[])
+ AND COALESCE(d.verdict, '') = ANY($3::text[])
+ )::int AS abstained,
+ COUNT(*) FILTER (
+ WHERE d.analyst_disposition = ANY($2::text[])
+ AND NOT (COALESCE(d.verdict, '') = ANY($3::text[]))
+ )::int AS answered,
+ COUNT(*) FILTER (
+ WHERE d.analyst_disposition = ANY($2::text[])
+ AND NOT (COALESCE(d.verdict, '') = ANY($3::text[]))
+ AND d.verdict = d.analyst_disposition
+ )::int AS agreed,
+ COUNT(*) FILTER (WHERE d.analyst_disposition = $4)::int AS malicious_support,
+ COUNT(*) FILTER (WHERE d.analyst_disposition = $4 AND d.verdict = $4)::int AS malicious_caught
+FROM recent d
+"""
+
+
+def scoped_sql(statement: str, predicate: str = "") -> str:
+ """Splice a scope predicate into one of the statements above.
+
+ The predicate is written by the caller from a fixed vocabulary of column
+ names and placeholder numbers; nothing derived from a request reaches it.
+ Passing an empty predicate scores every class the tenant runs, which is
+ the tenant-wide row the scorecard leads with.
+ """
+ return statement.replace(_SCOPE_MARKER, predicate)
+
+
+def to_named_params(statement: str, names: list[str]) -> str:
+ """Rewrite ``$1 … $n`` into ``:name`` for the SQLAlchemy caller.
+
+ Rewriting beats keeping a second copy of a forty-line aggregate: the copy
+ is the thing that drifts, and this way the statement the API runs is
+ provably the statement ``services/actions`` runs with the placeholders
+ spelled the other way.
+ """
+ # Highest index first, so `$1` does not match inside `$10`.
+ for index in range(len(names), 0, -1):
+ statement = statement.replace(f"${index}", f":{names[index - 1]}")
+ return statement
diff --git a/services/actions/app/services/shadow_reconcile.py b/services/actions/app/services/shadow_reconcile.py
new file mode 100644
index 000000000..bf9fdd8b6
--- /dev/null
+++ b/services/actions/app/services/shadow_reconcile.py
@@ -0,0 +1,216 @@
+"""Grade a shadow decision against the closure an analyst made in their own SIEM.
+
+Gap-closure Phase 2.1.
+
+Half the closures that matter never happen in AiSOC. A customer evaluating the
+agent runs their queue where they always have: an analyst dispositions the
+notable in Splunk ES, classifies the incident in Sentinel, closes the offense
+in QRadar. If agreement were measured only against closures made in this
+console, a tenant whose analysts work in their SIEM would show a scorecard
+that never fills in, and the natural reading of that is "the agent is not
+being evaluated" rather than "we are looking in the wrong place".
+
+Phase 1.1 already built the readers. ``SplunkClient.list_closed_notables``,
+``SentinelClient.list_closed_incidents``, ``ElasticClient.list_closed_signals``,
+``QRadarClient.list_closed_offenses`` and ``DefenderClient.list_resolved_alerts``
+each return :class:`~app.services.alert_history.ClosedFinding` rows carrying
+the analyst's disposition already mapped onto the canonical taxonomy, with a
+vendor label outside it landing on ``unlabeled`` rather than being guessed at.
+This module is the sink those rows flow into: it is the one piece Phase 2
+needed that Phase 1 did not build, and it deliberately reuses the readers
+rather than adding a sixth way to ask a SIEM what its analysts decided.
+
+The join key
+============
+
+``ClosedFinding.finding_id`` is the vendor's own id for the finding, and it
+reaches the shadow decision row as ``external_id`` by way of
+``alerts.external_id``, which ``services/ingest`` already carries through from
+the vendor payload. Matching on anything else, a title or a timestamp, would
+silently grade one alert against another's closure, and the symptom would be
+an agreement rate that is merely wrong rather than obviously broken.
+
+What it refuses to do
+=====================
+
+It never invents a label. A finding whose disposition is ``unlabeled`` is
+still recorded as resolved, because "an analyst looked at this and declined to
+classify it" is a fact the scorecard must show; it is excluded from every rate
+by the aggregate, not by being dropped here. Dropping it would make a tenant
+whose history is mostly unlabeled look like a tenant with a small clean
+sample.
+
+It never overwrites a decision that already carries a closure. A tenant whose
+analysts work in both places would otherwise have whichever reconciliation ran
+last win, and a promotion would rest on evidence that changes depending on
+scheduling.
+"""
+
+from __future__ import annotations
+
+import os
+from collections.abc import Iterable, Sequence
+from dataclasses import dataclass
+from typing import Any
+from uuid import UUID
+
+import structlog
+
+from app.services.alert_history import UNLABELED, ClosedFinding
+from app.services.disposition_writeback import CANONICAL_DISPOSITIONS
+
+logger = structlog.get_logger(__name__)
+
+__all__ = ["ReconcileResult", "reconcile_findings"]
+
+
+@dataclass(frozen=True)
+class ReconcileResult:
+ """What one reconciliation pass did, in numbers an operator can act on.
+
+ ``unmatched`` is the interesting one. A large unmatched count against a
+ healthy ``matched`` means the tenant's analysts are closing findings the
+ agent never triaged, which is ordinary. A total of zero matched with a
+ large unmatched means the join key is not arriving, which is a wiring
+ fault, and the two must not be reported as one number.
+ """
+
+ considered: int = 0
+ matched: int = 0
+ unmatched: int = 0
+ already_resolved: int = 0
+ skipped_no_key: int = 0
+
+ def as_dict(self) -> dict[str, Any]:
+ return {
+ "considered": self.considered,
+ "matched": self.matched,
+ "unmatched": self.unmatched,
+ "already_resolved": self.already_resolved,
+ "skipped_no_key": self.skipped_no_key,
+ }
+
+
+def _dsn() -> str | None:
+ """An asyncpg-compatible DSN, or None when no database is configured.
+
+ Same resolution as ``tenant_policy._dsn`` and ``action_store._dsn``:
+ both the SQLAlchemy spelling the API uses and the plain pgx spelling
+ ``services/ingest`` uses are already present in deployed environments.
+ """
+ raw = (os.environ.get("DATABASE_DSN") or os.environ.get("DATABASE_URL") or "").strip()
+ if not raw:
+ return None
+ return raw.replace("postgresql+asyncpg://", "postgresql://").replace("postgres+asyncpg://", "postgresql://")
+
+
+async def _connect(dsn: str) -> Any:
+ import asyncpg # noqa: PLC0415 - optional at import time; only needed with a DSN
+
+ return await asyncpg.connect(dsn, timeout=10.0)
+
+
+def _disposition_of(finding: ClosedFinding) -> str:
+ """The label to record: canonical, or the literal ``unlabeled``.
+
+ ``ClosedFinding.__post_init__`` already refuses anything else, so this is
+ belt and braces rather than a second mapping. It exists so that a future
+ reader who changes the reader layer finds the invariant restated at the
+ point it is written to the database.
+ """
+ return finding.disposition if finding.disposition in CANONICAL_DISPOSITIONS else UNLABELED
+
+
+async def reconcile_findings(
+ tenant_id: str | UUID,
+ findings: Iterable[ClosedFinding],
+ *,
+ connection: Any = None,
+) -> ReconcileResult:
+ """Attach each closure to the shadow decision it grades, if there is one.
+
+ ``connection`` is injectable so a test drives the real SQL against a real
+ connection, and so a caller reconciling several vendors in one pass does
+ not open five. Production passes nothing and gets one connection for the
+ call.
+ """
+ rows: Sequence[ClosedFinding] = list(findings)
+ if not rows:
+ return ReconcileResult()
+
+ owned = connection is None
+ if owned:
+ dsn = _dsn()
+ if dsn is None:
+ logger.debug("shadow_reconcile.no_database")
+ return ReconcileResult(considered=len(rows), skipped_no_key=0)
+ connection = await _connect(dsn)
+
+ result = ReconcileResult(considered=len(rows))
+ try:
+ await connection.execute("SELECT set_config('app.current_tenant_id', $1, true)", str(tenant_id))
+ for finding in rows:
+ key = (finding.finding_id or "").strip()
+ if not key:
+ result = _bump(result, skipped_no_key=1)
+ continue
+ updated = await connection.fetchval(
+ """
+ UPDATE aisoc_shadow_decisions
+ SET analyst_disposition = $3,
+ vendor_disposition = $4,
+ resolution_source = $5,
+ resolved_at = $6,
+ resolved_by = $7
+ WHERE tenant_id = $1
+ AND external_id = $2
+ AND resolved_at IS NULL
+ RETURNING id
+ """,
+ UUID(str(tenant_id)),
+ key,
+ _disposition_of(finding),
+ finding.vendor_disposition or None,
+ finding.vendor,
+ finding.closed_at,
+ finding.closed_by,
+ )
+ if updated is not None:
+ result = _bump(result, matched=1)
+ continue
+ # No row updated means either no decision carries this id, or the
+ # one that does was already graded. The two are told apart
+ # explicitly rather than folded together, because one is ordinary
+ # and one means the join key never arrives.
+ existing = await connection.fetchval(
+ """
+ SELECT 1 FROM aisoc_shadow_decisions
+ WHERE tenant_id = $1 AND external_id = $2 AND resolved_at IS NOT NULL
+ LIMIT 1
+ """,
+ UUID(str(tenant_id)),
+ key,
+ )
+ result = _bump(result, already_resolved=1) if existing else _bump(result, unmatched=1)
+ finally:
+ if owned:
+ await connection.close()
+
+ logger.info("shadow_reconcile.pass_complete", tenant_id=str(tenant_id), **result.as_dict())
+ return result
+
+
+def _bump(result: ReconcileResult, **deltas: int) -> ReconcileResult:
+ """Return ``result`` with the named counters advanced.
+
+ The dataclass is frozen because it is reported and logged, and a counter
+ somebody can mutate after the log line is written is a counter that
+ eventually disagrees with it.
+ """
+ return ReconcileResult(
+ considered=result.considered + deltas.get("considered", 0),
+ matched=result.matched + deltas.get("matched", 0),
+ unmatched=result.unmatched + deltas.get("unmatched", 0),
+ already_resolved=result.already_resolved + deltas.get("already_resolved", 0),
+ skipped_no_key=result.skipped_no_key + deltas.get("skipped_no_key", 0),
+ )
diff --git a/services/actions/tests/test_autonomy_evidence_rules.py b/services/actions/tests/test_autonomy_evidence_rules.py
new file mode 100644
index 000000000..f695cd9a6
--- /dev/null
+++ b/services/actions/tests/test_autonomy_evidence_rules.py
@@ -0,0 +1,283 @@
+"""The arithmetic that decides whether an agent has earned anything.
+
+Gap-closure Phase 2.2.
+
+Most of these are about one property, stated three ways, because it is the
+property a plausible implementation gets wrong: **an agent must not be able to
+improve its agreement rate by declining to answer.** That is not a hypothetical
+failure. Routing the hard cases to a human is the single easiest thing for a
+triage agent to do, it looks like caution from every angle, and under a naive
+metric it produces a perfect record on the alerts nobody was worried about.
+"""
+
+from __future__ import annotations
+
+import pytest
+from app.services.autonomy_evidence_rules import (
+ ABSTENTION_VERDICTS,
+ AGREEMENT_COUNTS_SQL,
+ DEFAULT_THRESHOLDS,
+ GRADED_DISPOSITIONS,
+ MALICIOUS,
+ RECENT_COUNTS_SQL,
+ UNLABELED,
+ AgreementWindow,
+ PromotionThresholds,
+ scoped_sql,
+ to_named_params,
+ window_from_counts,
+)
+
+
+class TestAbstainingCannotBuyAgreement:
+ def test_declining_the_hard_cases_does_not_move_the_rate(self):
+ """Ten right out of ten answered is 100% whether or not 90 were declined."""
+ answered_everything = AgreementWindow(
+ resolved=10,
+ labelled=10,
+ answered=10,
+ abstained=0,
+ agreed=10,
+ malicious_support=4,
+ malicious_caught=4,
+ )
+ declined_most = AgreementWindow(
+ resolved=100,
+ labelled=100,
+ answered=10,
+ abstained=90,
+ agreed=10,
+ malicious_support=4,
+ malicious_caught=4,
+ )
+ assert answered_everything.agreement_rate == 1.0
+ assert declined_most.agreement_rate == 1.0
+
+ def test_what_declining_does_move_is_the_abstention_rate(self):
+ """The cost of abstaining is visible, and it is visible in its own column."""
+ declined_most = AgreementWindow(
+ resolved=100,
+ labelled=100,
+ answered=10,
+ abstained=90,
+ agreed=10,
+ malicious_support=4,
+ malicious_caught=4,
+ )
+ assert declined_most.abstention_rate == 0.9
+ assert declined_most.abstention_rate > DEFAULT_THRESHOLDS.max_abstention_rate
+
+ def test_an_abstention_on_a_malicious_case_counts_as_a_miss(self):
+ """Recall's denominator is every malicious case, answered or not.
+
+ An alert routed to a human was not caught by the agent, and this is
+ the metric that says so. Phase 1 makes the same call in
+ ``aisoc_benchmark.replay``, and the two have to agree or the replay
+ report and the live scorecard describe different agents.
+ """
+ window = AgreementWindow(
+ resolved=40,
+ labelled=40,
+ answered=10,
+ abstained=30,
+ agreed=10,
+ malicious_support=30,
+ malicious_caught=8,
+ )
+ # Agreement is perfect over what it answered.
+ assert window.agreement_rate == 1.0
+ # Recall is not, because 22 malicious cases went to a human.
+ assert window.malicious_recall == pytest.approx(8 / 30)
+
+
+class TestAbsentIsNotZero:
+ def test_a_rate_with_no_denominator_is_none(self):
+ empty = AgreementWindow()
+ assert empty.agreement_rate is None
+ assert empty.malicious_recall is None
+ assert empty.abstention_rate is None
+
+ def test_no_malicious_cases_is_not_zero_recall(self):
+ """A tenant whose queue held no true positives has an unmeasured recall.
+
+ Zero would say the agent missed every one of them. There were none to
+ miss, and the two call for opposite responses: one is a reason to
+ refuse autonomy, the other is a reason to keep measuring.
+ """
+ window = AgreementWindow(
+ resolved=50,
+ labelled=50,
+ answered=50,
+ abstained=0,
+ agreed=50,
+ malicious_support=0,
+ malicious_caught=0,
+ )
+ assert window.malicious_recall is None
+ assert window.agreement_rate == 1.0
+
+
+class TestUnlabelledIsResolvedButNotGraded:
+ def test_an_analyst_who_declined_to_classify_is_counted_and_excluded(self):
+ window = AgreementWindow(
+ resolved=100,
+ labelled=40,
+ answered=40,
+ abstained=0,
+ agreed=38,
+ malicious_support=12,
+ malicious_caught=11,
+ )
+ assert window.unlabeled == 60
+ # The rate is over the 40 that carried a label, not the 100 closed.
+ assert window.agreement_rate == pytest.approx(38 / 40)
+
+ def test_unlabeled_is_not_a_gradeable_disposition(self):
+ """Anything treating it as a verdict must fail a membership check."""
+ assert UNLABELED not in GRADED_DISPOSITIONS
+
+
+class TestTheCountsHaveToAddUp:
+ """A window whose parts disagree is a bug in the aggregate query.
+
+ It would surface as a plausible-looking rate rather than as an error, and
+ a plausible-looking rate is what a promotion gets granted on.
+ """
+
+ def test_answered_plus_abstained_must_equal_labelled(self):
+ with pytest.raises(ValueError, match="must equal"):
+ AgreementWindow(resolved=10, labelled=10, answered=5, abstained=2, agreed=5)
+
+ def test_agreed_cannot_exceed_answered(self):
+ with pytest.raises(ValueError, match="agreed"):
+ AgreementWindow(resolved=10, labelled=10, answered=5, abstained=5, agreed=6)
+
+ def test_caught_cannot_exceed_support(self):
+ with pytest.raises(ValueError, match="malicious_caught"):
+ AgreementWindow(
+ resolved=10,
+ labelled=10,
+ answered=10,
+ abstained=0,
+ agreed=10,
+ malicious_support=2,
+ malicious_caught=3,
+ )
+
+ def test_labelled_cannot_exceed_resolved(self):
+ with pytest.raises(ValueError, match="labelled"):
+ AgreementWindow(resolved=5, labelled=10, answered=10, abstained=0, agreed=0)
+
+
+class TestWindowFromCounts:
+ def test_it_reads_a_driver_row(self):
+ window = window_from_counts(
+ {
+ "resolved": 120,
+ "labelled": 100,
+ "abstained": 10,
+ "answered": 90,
+ "agreed": 87,
+ "malicious_support": 31,
+ "malicious_caught": 29,
+ }
+ )
+ assert window.unlabeled == 20
+ assert window.agreement_rate == pytest.approx(87 / 90)
+ assert window.malicious_recall == pytest.approx(29 / 31)
+
+ def test_missing_keys_read_as_zero_not_as_a_crash(self):
+ """An aggregate that returned no row at all is an empty window.
+
+ ``fetchrow`` on a tenant with no decisions returns ``None``, and the
+ callers pass ``dict(row or {})``. That has to produce a window whose
+ rates are all "not measured", not a ``KeyError`` three frames up.
+ """
+ assert window_from_counts({}) == AgreementWindow()
+
+
+class TestTheSqlIsSplicedNotConcatenated:
+ def test_both_statements_carry_the_scope_marker(self):
+ """Neither statement may be scoped by appending to the end.
+
+ The window aggregate ends in a ``WHERE`` chain and the trailing-slice
+ one ends in ``ORDER BY … LIMIT`` inside a CTE. A caller appending to
+ either would produce valid SQL for one and a syntax error for the
+ other, which is how a scope silently stops being applied on the half
+ that still parses.
+ """
+ for statement in (AGREEMENT_COUNTS_SQL, RECENT_COUNTS_SQL):
+ assert "/*scope*/" in statement
+
+ def test_an_empty_predicate_scores_the_whole_tenant(self):
+ spliced = scoped_sql(AGREEMENT_COUNTS_SQL, "")
+ assert "/*scope*/" not in spliced
+ assert "d.tenant_id = $1" in spliced
+
+ def test_the_tenant_predicate_survives_every_scope(self):
+ spliced = scoped_sql(RECENT_COUNTS_SQL, "AND d.alert_class = :scope_key")
+ assert "d.tenant_id = $1" in spliced
+ assert "AND d.alert_class = :scope_key" in spliced
+
+ def test_named_params_rewrites_every_placeholder(self):
+ names = ["tenant_id", "graded", "abstentions", "malicious", "window_start", "window_end", "recent_limit"]
+ rewritten = to_named_params(AGREEMENT_COUNTS_SQL, names)
+ assert ":tenant_id" in rewritten
+ assert ":recent_limit" not in rewritten # only the recent statement binds $7
+ for index in range(1, len(names) + 1):
+ assert f"${index}" not in rewritten
+
+ def test_double_digit_placeholders_are_rewritten_before_single_digit_ones(self):
+ """``$1`` must not match inside ``$10``.
+
+ Neither statement binds ten parameters today, so this is about the
+ helper rather than about them: the next caller that does would
+ otherwise get ``:tenant_id0`` and a runtime error far from here.
+ """
+ names = [f"p{n}" for n in range(1, 12)]
+ rewritten = to_named_params("SELECT $1, $10, $11", names)
+ assert rewritten == "SELECT :p1, :p10, :p11"
+
+
+class TestTheDefaultsArePlansDefaults:
+ def test_the_plan_numbers_are_the_defaults(self):
+ assert DEFAULT_THRESHOLDS.min_decisions == 100
+ assert DEFAULT_THRESHOLDS.min_malicious == 30
+
+ def test_demotion_floors_sit_below_promotion_thresholds(self):
+ """Deliberate hysteresis.
+
+ Equal values would flip a grant on every decision that moved the rate
+ across the line, and the audit log would fill with churn nobody could
+ read, which is how a real demotion gets missed.
+ """
+ limits = PromotionThresholds()
+ assert limits.demotion_agreement < limits.min_agreement
+ assert limits.demotion_malicious_recall < limits.min_malicious_recall
+
+ def test_thresholds_serialise_by_value(self):
+ """The snapshot records numbers, not the name of a constant.
+
+ A threshold retuned next quarter must not rewrite the justification
+ for every promotion granted under the old one.
+ """
+ payload = PromotionThresholds(min_decisions=7).as_dict()
+ assert payload["min_decisions"] == 7
+ assert payload["window_days"] == DEFAULT_THRESHOLDS.window_days
+
+
+class TestTheTaxonomyMatchesPhaseOne:
+ def test_malicious_is_the_writeback_spelling(self):
+ assert MALICIOUS == "true_positive"
+ assert GRADED_DISPOSITIONS[0] == MALICIOUS
+
+ def test_an_empty_verdict_is_an_abstention(self):
+ """A null verdict reaches the aggregate as ``''`` through COALESCE.
+
+ If that were not an abstention it would be scored as a wrong answer,
+ and a run that crashed before producing a verdict would count against
+ agreement as though the agent had decided something.
+ """
+ assert "" in ABSTENTION_VERDICTS
+ assert "needs_review" in ABSTENTION_VERDICTS
+ assert "escalate" in ABSTENTION_VERDICTS
diff --git a/services/actions/tests/test_shadow_reconcile.py b/services/actions/tests/test_shadow_reconcile.py
new file mode 100644
index 000000000..af81c462a
--- /dev/null
+++ b/services/actions/tests/test_shadow_reconcile.py
@@ -0,0 +1,211 @@
+"""Closures made in the customer's own SIEM, matched to the verdicts they grade.
+
+Gap-closure Phase 2.1.
+
+Half the closures that matter never happen in AiSOC. An analyst evaluating the
+agent dispositions the notable in Splunk, classifies the incident in Sentinel,
+closes the offense in QRadar. If agreement were measured only against closures
+made in this console, a tenant whose analysts work in their SIEM would see a
+scorecard that never fills in, and the natural reading of an empty scorecard
+is "the agent is not being evaluated" rather than "we are looking in the wrong
+place".
+
+The readers this consumes are Phase 1.1's, unchanged. These tests drive the
+real ``ClosedFinding`` rows those readers produce, including the
+``unlabeled`` ones, because the interesting behaviour is what happens to a
+closure the platform cannot name.
+"""
+
+from __future__ import annotations
+
+import uuid
+from datetime import UTC, datetime
+from typing import Any
+
+import pytest
+from app.services.alert_history import UNLABELED, parse_qradar_offense, parse_splunk_notable
+from app.services.shadow_reconcile import ReconcileResult, reconcile_findings
+
+TENANT = uuid.UUID("11111111-1111-1111-1111-111111111111")
+
+
+class _FakeConnection:
+ """An asyncpg connection that records statements and answers from a map.
+
+ ``matches`` holds the external ids that have an ungraded decision row;
+ ``graded`` holds the ones already carrying a closure. Everything else is
+ unmatched, which is the case the result type exists to keep separate.
+ """
+
+ def __init__(self, matches: set[str] | None = None, graded: set[str] | None = None) -> None:
+ self.matches = matches or set()
+ self.graded = graded or set()
+ self.updates: list[tuple[Any, ...]] = []
+ self.rls: list[str] = []
+ self.closed = False
+
+ async def execute(self, sql: str, *args: Any) -> None:
+ if "set_config" in sql:
+ self.rls.append(args[0])
+
+ async def fetchval(self, sql: str, *args: Any) -> Any:
+ if sql.strip().startswith("UPDATE"):
+ self.updates.append(args)
+ return uuid.uuid4() if args[1] in self.matches else None
+ return 1 if args[1] in self.graded else None
+
+ async def close(self) -> None:
+ self.closed = True
+
+
+def _splunk(finding_id: str, disposition: str) -> Any:
+ return parse_splunk_notable(
+ {
+ "event_id": finding_id,
+ "rule_name": "Suspicious login",
+ "disposition": disposition,
+ "review_time": "1790000000",
+ "reviewer": "alice",
+ "rule_id": "splunk-rule-1",
+ }
+ )
+
+
+class TestItReusesThePhaseOneReaders:
+ @pytest.mark.asyncio
+ async def test_a_mapped_disposition_is_written_through_unchanged(self):
+ conn = _FakeConnection(matches={"notable-1"})
+ result = await reconcile_findings(TENANT, [_splunk("notable-1", "disposition:1")], connection=conn)
+
+ assert result.matched == 1
+ # `disposition:1` is Splunk ES's "True Positive - Suspicious Activity",
+ # mapped by the Phase 1 taxonomy rather than by anything here.
+ assert conn.updates[0][2] == "true_positive"
+
+ @pytest.mark.asyncio
+ async def test_the_vendors_own_label_travels_alongside(self):
+ """A mapping somebody later disputes is only arguable if the raw label survived."""
+ conn = _FakeConnection(matches={"notable-1"})
+ await reconcile_findings(TENANT, [_splunk("notable-1", "disposition:2")], connection=conn)
+ assert conn.updates[0][3] == "disposition:2"
+
+ @pytest.mark.asyncio
+ async def test_an_i_do_not_know_closure_is_recorded_as_unlabeled(self):
+ """Splunk ES disposition 6 is literally named "Undetermined".
+
+ Folding it into a verdict because the finding happened to be closed
+ would manufacture agreement out of an analyst's admission of
+ uncertainty. It is still recorded, because a tenant whose history is
+ mostly unlabeled has to see that rather than a confident rate over a
+ remnant.
+ """
+ conn = _FakeConnection(matches={"notable-9"})
+ result = await reconcile_findings(TENANT, [_splunk("notable-9", "disposition:6")], connection=conn)
+
+ assert result.matched == 1
+ assert conn.updates[0][2] == UNLABELED
+
+ @pytest.mark.asyncio
+ async def test_a_qradar_non_issue_keeps_the_readers_distinction(self):
+ """Phase 1 maps "Non-Issue" to `benign`, not `benign_true_positive`.
+
+ It makes no claim about whether the rule was right, and that is the
+ distinction `benign` exists to carry. Re-deciding it here would be a
+ second taxonomy.
+ """
+ offense = parse_qradar_offense(
+ {"id": "77", "description": "Port scan", "closing_reason_name": "Non-Issue", "close_time": 1790000000000}
+ )
+ conn = _FakeConnection(matches={"77"})
+ await reconcile_findings(TENANT, [offense], connection=conn)
+ assert conn.updates[0][2] == "benign"
+
+
+class TestItTellsTheThreeOutcomesApart:
+ @pytest.mark.asyncio
+ async def test_unmatched_and_already_graded_are_separate_counters(self):
+ """One is ordinary and one means the join key never arrives.
+
+ An analyst closing findings the agent never triaged is normal. Zero
+ matched against a large unmatched is a wiring fault, and reporting
+ both as one number hides the difference.
+ """
+ conn = _FakeConnection(matches={"a"}, graded={"b"})
+ result = await reconcile_findings(
+ TENANT,
+ [_splunk("a", "disposition:1"), _splunk("b", "disposition:1"), _splunk("c", "disposition:1")],
+ connection=conn,
+ )
+
+ assert (result.matched, result.already_resolved, result.unmatched) == (1, 1, 1)
+ assert result.considered == 3
+
+ @pytest.mark.asyncio
+ async def test_a_finding_with_no_id_is_counted_rather_than_dropped(self):
+ """Nothing to join on. Reported, because a run of these is a fault.
+
+ The reader itself falls back from ``event_id`` to ``rule_id``, so this
+ is built without either: a vendor payload that carried neither is the
+ only way the id is genuinely absent.
+ """
+ anonymous = parse_splunk_notable({"rule_name": "Suspicious login", "disposition": "disposition:1", "review_time": "1790000000"})
+ assert anonymous.finding_id == ""
+
+ conn = _FakeConnection()
+ result = await reconcile_findings(TENANT, [anonymous], connection=conn)
+
+ assert result.skipped_no_key == 1
+ assert conn.updates == []
+
+ @pytest.mark.asyncio
+ async def test_nothing_to_do_is_an_empty_result_not_a_connection(self):
+ assert await reconcile_findings(TENANT, []) == ReconcileResult()
+
+
+class TestItNeverRegradesADecision:
+ @pytest.mark.asyncio
+ async def test_the_update_refuses_a_decision_that_already_carries_a_closure(self):
+ """A tenant whose analysts work in both places would otherwise have
+ whichever reconciliation ran last win, and a promotion would rest on
+ evidence that changes with the scheduler."""
+ conn = _FakeConnection(matches={"x"})
+ await reconcile_findings(TENANT, [_splunk("x", "disposition:1")], connection=conn)
+ # The statement itself carries the guard; the fake does not enforce it.
+ assert conn.updates, "expected the UPDATE to have been attempted"
+
+ @pytest.mark.asyncio
+ async def test_the_tenant_context_is_bound_before_any_read(self):
+ """RLS engages only on a session that set ``app.current_tenant_id``.
+
+ This connection is opened by a worker rather than by a request, so
+ nothing else sets it, and without it these statements would fall
+ through the policy's fail-open arm.
+ """
+ conn = _FakeConnection(matches={"x"})
+ await reconcile_findings(TENANT, [_splunk("x", "disposition:1")], connection=conn)
+ assert conn.rls == [str(TENANT)]
+
+ @pytest.mark.asyncio
+ async def test_an_injected_connection_is_not_closed_by_the_callee(self):
+ """A caller reconciling five vendors in one pass opens one connection."""
+ conn = _FakeConnection(matches={"x"})
+ await reconcile_findings(TENANT, [_splunk("x", "disposition:1")], connection=conn)
+ assert conn.closed is False
+
+
+class TestTheCloseTimeIsTheVendorsNotNow:
+ @pytest.mark.asyncio
+ async def test_the_recorded_resolution_time_comes_from_the_finding(self):
+ """The window a promotion is measured over is keyed on this column.
+
+ Stamping ``now()`` would put a closure made three weeks ago into this
+ week's trailing drift slice, which is the slice that exists to catch a
+ recent decline.
+ """
+ conn = _FakeConnection(matches={"notable-1"})
+ finding = _splunk("notable-1", "disposition:1")
+ await reconcile_findings(TENANT, [finding], connection=conn)
+
+ recorded = conn.updates[0][5]
+ assert recorded == finding.closed_at
+ assert recorded == datetime.fromtimestamp(1790000000, tz=UTC)
diff --git a/services/agents/app/investigator/ledger.py b/services/agents/app/investigator/ledger.py
index 37cef0f52..b0ace3522 100644
--- a/services/agents/app/investigator/ledger.py
+++ b/services/agents/app/investigator/ledger.py
@@ -438,6 +438,7 @@ async def persist_auto_triage(
unpriced_call_count: int = 0,
groundedness: float | None = None,
ungrounded: bool | None = None,
+ shadow: bool = False,
) -> bool:
"""Durably record an auto-triage outcome (issue #571) in ONE transaction:
@@ -455,6 +456,16 @@ async def persist_auto_triage(
configured, or an unknown tenant — neither is retryable). Raises
:class:`LedgerPersistError` on a real DB error so the caller can retry /
dead-letter. A completed run therefore always carries a non-null verdict.
+
+ ``shadow`` (gap-closure Phase 2.1) leaves the analyst's own columns alone:
+ ``disposition``, ``status`` and ``resolved_at`` keep whatever they held,
+ while the run, the event and the ``ai_*`` columns are written exactly as
+ they would be otherwise. ``alerts.disposition`` is the field an analyst
+ fills in and the field agreement is later measured against, so a shadow
+ verdict landing there would have the agent answering the question it is
+ about to be graded on. The guard is in the statement rather than left to
+ the caller, so a future caller that forgets to force ``auto_closed`` off
+ still cannot close an alert it was only meant to observe.
"""
pool = await get_pool()
if pool is None:
@@ -564,12 +575,12 @@ async def persist_auto_triage(
await conn.execute(
"""
UPDATE alerts
- SET disposition = $3,
+ SET disposition = CASE WHEN $10 THEN disposition ELSE $3 END,
ai_score = $4,
ai_summary = $5,
ai_recommendations = $6::jsonb,
- status = CASE WHEN $7 THEN 'resolved' ELSE status END,
- resolved_at = CASE WHEN $7 THEN now() ELSE resolved_at END,
+ status = CASE WHEN $7 AND NOT $10 THEN 'resolved' ELSE status END,
+ resolved_at = CASE WHEN $7 AND NOT $10 THEN now() ELSE resolved_at END,
-- Nullable on purpose: NULL is "not scored",
-- which is a different fact from "scored zero".
-- The deterministic path never assesses
@@ -589,12 +600,14 @@ async def persist_auto_triage(
auto_closed,
float(groundedness) if groundedness is not None else None,
ungrounded,
+ shadow,
)
logger.info(
"ledger.auto_triage_persisted",
run_id=str(run_id),
verdict=verdict,
auto_closed=auto_closed,
+ shadow=shadow,
alert_id=str(alert_id or ""),
)
return True
diff --git a/services/agents/app/workers/fused_alert_consumer.py b/services/agents/app/workers/fused_alert_consumer.py
index 95b16bfc2..67ba7391a 100644
--- a/services/agents/app/workers/fused_alert_consumer.py
+++ b/services/agents/app/workers/fused_alert_consumer.py
@@ -56,6 +56,13 @@ from app.models.state import AgentStatus, InvestigationState
from app.routing.model_router import is_deterministic_mode
from app.security.llm_resolver import resolve_llm_config
from app.workers.business_context import BusinessContextApplier
+from app.workers.shadow_mode import (
+ LiveShadowModePolicy,
+ ShadowDecision,
+ ShadowModePolicy,
+ ShadowModeTriageWriter,
+ alert_class_of,
+)
from app.workers.triage_persistence import (
LiveTriageContextReader,
LiveTriageWriter,
@@ -98,6 +105,11 @@ _METRICS = {
# rather than a broken integration.
"writeback_attempted": 0,
"writeback_executed": 0,
+ # Phase 2.1. Two counters for the same reason as the writeback pair: a
+ # shadow run whose decision row never lands measures nothing, and the
+ # symptom of that is a scorecard that stays empty rather than an error.
+ "shadow_triaged": 0,
+ "shadow_decisions_recorded": 0,
"errors": 0,
}
@@ -179,6 +191,17 @@ def _cost_payload(cost: CostSummary, tokens: int) -> dict[str, Any]:
}
+def _opt_text(value: Any) -> str | None:
+ """A trimmed string, or ``None`` for absent and empty alike.
+
+ The shadow decision row distinguishes the two: a null rule id means the
+ alert carried none, and an empty string would become a segment key named
+ after nothing in the per-rule breakdown.
+ """
+ text = str(value).strip() if value is not None else ""
+ return text or None
+
+
def _rationale_of(state: InvestigationState) -> str:
"""Human-readable rationale for the verdict, for the ledger + alerts row."""
if state.confidence_basis:
@@ -258,6 +281,10 @@ class FusedAlertTriageWorker:
#: instance carries nothing between workers.
_writer: TriageWriter = LiveTriageWriter()
_reader: TriageContextReader = LiveTriageContextReader()
+ #: Phase 2.1. Not a sink: it decides *which* sink each alert gets. Stated
+ #: at class level for the same reason as the two above, so a worker built
+ #: with ``__new__`` still resolves a policy rather than raising.
+ _shadow_policy: ShadowModePolicy = LiveShadowModePolicy()
def __init__(
self,
@@ -270,6 +297,7 @@ class FusedAlertTriageWorker:
business_context: BusinessContextApplier | None = None,
writer: TriageWriter | None = None,
context_reader: TriageContextReader | None = None,
+ shadow_policy: ShadowModePolicy | None = None,
) -> None:
self._bootstrap = bootstrap_servers
self._topic = topic
@@ -280,6 +308,7 @@ class FusedAlertTriageWorker:
# byte-for-byte the same behaviour.
self._writer: TriageWriter = writer or LiveTriageWriter()
self._reader: TriageContextReader = context_reader or LiveTriageContextReader()
+ self._shadow_policy: ShadowModePolicy = shadow_policy or LiveShadowModePolicy()
self._dlq_topic = dlq_topic or os.getenv("KAFKA_TOPIC_ALERTS_FUSED_DLQ", f"{topic}.dlq")
self._max_attempts = max(1, max_attempts)
self._consumer: Any | None = None
@@ -477,12 +506,20 @@ class FusedAlertTriageWorker:
governor = get_governor()
fingerprint = governor.evidence_fingerprint(str(state.tenant_id), state.raw_alert)
+ # Phase 2.1 — which sink this alert's writes go to is a per-alert
+ # question, because shadow mode is per tenant *and* per alert class.
+ # The constructor sinks stay untouched: `writer` is the tenant's live
+ # sink or a shadow wrapper around it, chosen here and passed down
+ # explicitly rather than swapped onto `self`, which would leak across
+ # concurrently triaged alerts.
+ writer = await self._sink_for(state, fingerprint)
+
# Wave 1 — forward auto-suppression: if this signature has a trusted
# prior benign/FP outcome, auto-resolve the repeat WITHOUT re-triage
# (this is what makes alert volume actually shrink over time). Gated on
# trust (human prior, or corroborated high-confidence AI prior).
if _memory_suppression_enabled():
- suppressed = await self._maybe_suppress_from_memory(state, fingerprint, bc_matched)
+ suppressed = await self._maybe_suppress_from_memory(state, fingerprint, bc_matched, writer=writer)
if suppressed is not None:
return suppressed
@@ -514,7 +551,7 @@ class FusedAlertTriageWorker:
async with CostTracker(
run_id=str(state.run_id),
tenant_id=str(state.tenant_id),
- persist=self._writer.persists_cost,
+ persist=writer.persists_cost,
) as tracker:
if use_llm and cfg is not None:
# Route the LLM call through the tenant's BYOK key/model so
@@ -565,7 +602,7 @@ class FusedAlertTriageWorker:
# spent. An unmeasured call contributes no dollars — it still
# contributes tokens, which is the cap that can be enforced
# honestly without a price.
- self._writer.cache_verdict(
+ writer.cache_verdict(
governor,
str(state.tenant_id),
fingerprint,
@@ -584,13 +621,13 @@ class FusedAlertTriageWorker:
verdict, confidence = self._apply_groundedness_gate(state, verdict, confidence)
_METRICS["triaged"] += 1
- await self._record(state, tier=tier, verdict=verdict, confidence=confidence, tokens=tokens, cost=cost)
+ await self._record(state, tier=tier, verdict=verdict, confidence=confidence, tokens=tokens, cost=cost, writer=writer)
# Wave 1 — write the durable outcome back as a per-signature prior so
# autonomous closures compound (a later identical alert can suppress).
if _memory_writeback_enabled() and verdict:
with contextlib.suppress(Exception):
- await self._writer.record_outcome(
+ await writer.record_outcome(
str(state.tenant_id),
fingerprint,
disposition=str(verdict),
@@ -606,21 +643,30 @@ class FusedAlertTriageWorker:
# skip enrichment. Best-effort: the verdict is already durable, so an
# enrichment/investigation failure never fails the triage outcome.
if state.status is not AgentStatus.COMPLETED:
- await self._maybe_escalate(state)
+ await self._maybe_escalate(state, writer=writer)
# Queue every action that needs sign-off so a human can actually give
# it. Until v9.0 proposed actions were persisted as JSON and stopped
# there, so "requires_approval" described an approval nobody could
# grant — the responder app's queue had no producer at all.
- approvals = await self._raise_approvals(state)
+ approvals = await self._raise_approvals(state, writer=writer)
# Close the loop with the SIEM that raised this alert. Runs last and
# fails soft: the verdict is already durable, and an unreachable
# Splunk must not re-drive the retry path and dead-letter an alert
# that was triaged correctly.
- source_writeback = await self._write_back_to_source(state, verdict, confidence, rationale=_rationale_of(state))
+ source_writeback = await self._write_back_to_source(state, verdict, confidence, rationale=_rationale_of(state), writer=writer)
+
+ shadow = isinstance(writer, ShadowModeTriageWriter)
+ if isinstance(writer, ShadowModeTriageWriter) and writer.recorded:
+ _METRICS["shadow_decisions_recorded"] += 1
return {
+ # Phase 2.1. Stated on the summary rather than inferred from the
+ # empty writeback and approval lists, because "shadow mode" and
+ # "nothing needed doing" produce the same empty lists and lead an
+ # operator to opposite conclusions.
+ "shadow": shadow,
"run_id": str(state.run_id),
"incident_id": str(state.incident_id),
"tenant_id": str(state.tenant_id),
@@ -641,6 +687,41 @@ class FusedAlertTriageWorker:
"proposed_actions": [{"action_type": a.action_type, "requires_approval": a.requires_approval} for a in state.proposed_actions],
}
+ async def _sink_for(self, state: InvestigationState, fingerprint: str) -> TriageWriter:
+ """The sink this alert's writes go to: the tenant's live one, or a shadow wrapper.
+
+ Resolved per alert because shadow mode is per tenant *and* per alert
+ class, and the class is not known until the alert is in hand. The
+ wrapper delegates to whatever sink the worker was constructed with, so
+ a replay passing its own sink keeps it and a production worker wraps
+ the live one.
+
+ A policy read that fails is treated as shadow by
+ :class:`~app.workers.shadow_mode.LiveShadowModePolicy`, which is the
+ conservative direction: the cost of being wrong is a few seconds of
+ suppressed writeback, against acting on an alert a tenant asked us
+ only to observe.
+ """
+ raw = state.raw_alert or {}
+ alert_class = alert_class_of(raw)
+ if not await self._shadow_policy.is_shadow(str(state.tenant_id), alert_class):
+ return self._writer
+
+ _METRICS["shadow_triaged"] += 1
+ return ShadowModeTriageWriter(
+ self._writer,
+ ShadowDecision(
+ tenant_ref=str(state.tenant_id),
+ alert_class=alert_class,
+ alert_id=raw.get("id"),
+ external_id=_opt_text(raw.get("external_id")),
+ rule_id=_opt_text(raw.get("rule_id")),
+ source=_opt_text(raw.get("connector_type") or raw.get("source")),
+ model=_opt_text(getattr(state, "model_used", None)),
+ evidence_signature=fingerprint,
+ ),
+ )
+
async def _write_back_to_source(
self,
state: InvestigationState,
@@ -648,6 +729,7 @@ class FusedAlertTriageWorker:
confidence: float,
*,
rationale: str,
+ writer: TriageWriter,
) -> dict[str, Any] | None:
"""Ask the API to project this verdict onto the source finding.
@@ -660,7 +742,7 @@ class FusedAlertTriageWorker:
if not alert_id or not verdict:
return None
try:
- report = await self._writer.write_back_disposition(
+ report = await writer.write_back_disposition(
tenant_id=str(state.tenant_id),
alert_id=alert_id,
disposition=str(verdict),
@@ -677,7 +759,7 @@ class FusedAlertTriageWorker:
_METRICS["writeback_executed"] += 1
return report
- async def _raise_approvals(self, state: InvestigationState) -> list[str]:
+ async def _raise_approvals(self, state: InvestigationState, *, writer: TriageWriter) -> list[str]:
"""Queue each approval-requiring proposed action; return the ids.
Only actions the agent itself marked ``requires_approval`` are
@@ -693,7 +775,7 @@ class FusedAlertTriageWorker:
if not action.requires_approval:
continue
risk = action.risk_level.value if hasattr(action.risk_level, "value") else str(action.risk_level)
- approval_id = await self._writer.raise_approval(
+ approval_id = await writer.raise_approval(
tenant_ref=str(state.tenant_id),
run_id=state.run_id,
alert_id=(state.raw_alert or {}).get("id"),
@@ -718,11 +800,20 @@ class FusedAlertTriageWorker:
state: InvestigationState,
signature: str,
bc_matched: list[str],
+ *,
+ writer: TriageWriter,
) -> dict[str, Any] | None:
"""Auto-resolve a repeat alert from a trusted prior outcome (Wave 1).
Returns a summary dict (and short-circuits triage) when suppressed, else
None. Best-effort: any lookup failure falls through to normal triage.
+
+ Under a shadow sink this still runs and still reaches a verdict, and
+ still resolves nothing: suppression is what production would have done
+ with this alert, so it is exactly the decision worth measuring. The
+ sink declines the outcome and suppression writes, and the shadow flag
+ keeps the alert row open, so the analyst closes it themselves and the
+ agreement is real.
"""
try:
prior = await self._reader.lookup_prior(str(state.tenant_id), signature)
@@ -749,9 +840,9 @@ class FusedAlertTriageWorker:
_METRICS["outcome_suppressed"] += 1
_METRICS["triaged"] += 1
- await self._record(state, tier="memory", verdict=disposition, confidence=confidence)
+ await self._record(state, tier="memory", verdict=disposition, confidence=confidence, writer=writer)
with contextlib.suppress(Exception):
- await self._writer.record_outcome(
+ await writer.record_outcome(
str(state.tenant_id),
signature,
disposition=disposition,
@@ -760,7 +851,7 @@ class FusedAlertTriageWorker:
alert_id=(state.raw_alert or {}).get("id"),
)
with contextlib.suppress(Exception):
- await self._writer.record_suppression(
+ await writer.record_suppression(
tenant_ref=str(state.tenant_id),
signature=signature,
alert_id=(state.raw_alert or {}).get("id"),
@@ -792,7 +883,7 @@ class FusedAlertTriageWorker:
"proposed_actions": [],
}
- async def _maybe_escalate(self, state: InvestigationState) -> None:
+ async def _maybe_escalate(self, state: InvestigationState, *, writer: TriageWriter) -> None:
"""Run the full investigation graph for an escalated alert (issue #569).
Shares the same graph runner as the manual investigations API. Every
@@ -803,7 +894,7 @@ class FusedAlertTriageWorker:
# Two independent switches. The env flag is the operator's; the
# writer's is the caller's, and a shadow run declines because every
# graph node records itself to the ledger.
- if not _escalation_enabled() or not self._writer.escalation_allowed:
+ if not _escalation_enabled() or not writer.escalation_allowed:
return
# Pre-fetch the same context an analyst-initiated investigation gets.
# The manual orchestrator has built a ContextBundle since T2.1; the
@@ -821,7 +912,7 @@ class FusedAlertTriageWorker:
async with CostTracker(
run_id=str(state.run_id),
tenant_id=str(state.tenant_id),
- persist=self._writer.persists_cost,
+ persist=writer.persists_cost,
):
await run_escalation(state, budget=default_budget(), seq_start=1)
_METRICS["escalated"] += 1
@@ -919,6 +1010,7 @@ class FusedAlertTriageWorker:
confidence: float,
tokens: int = 0,
cost: CostSummary | None = None,
+ writer: TriageWriter,
) -> None:
"""Durably persist the triage outcome (issue #571).
@@ -939,7 +1031,7 @@ class FusedAlertTriageWorker:
}
for a in state.proposed_actions
]
- await self._writer.persist_auto_triage(
+ await writer.persist_auto_triage(
run_id=state.run_id,
alert_id=(state.raw_alert or {}).get("id"),
tenant_ref=str(state.tenant_id),
diff --git a/services/agents/app/workers/shadow_mode.py b/services/agents/app/workers/shadow_mode.py
new file mode 100644
index 000000000..26f713e7d
--- /dev/null
+++ b/services/agents/app/workers/shadow_mode.py
@@ -0,0 +1,410 @@
+"""Shadow mode on the live queue: triage runs, and nothing it says is acted on.
+
+Gap-closure Phase 2.1.
+
+Phase 1 built shadow execution for replay, over a customer's *closed* history:
+:class:`app.replay.shadow.ShadowTriageWriter` implements every write the
+triage path makes and performs none of them. This is the same seam pointed at
+live alerts, and it differs in exactly one way that matters.
+
+A replay writes nothing at all, because its findings are already closed and
+anything it wrote would be noise in a store somebody else reads. A live shadow
+run has to write two things: the verdict, so an analyst can see what the agent
+thought, and a decision row, so agreement can be measured when that analyst
+closes the alert. What it must not do is *act*: no disposition set on the
+alert, no auto-closure, no writeback to the source SIEM, no approval queued,
+no outcome prior recorded, no escalation into the investigation graph.
+
+The one that is easy to get wrong
+=================================
+
+``ledger.persist_auto_triage`` writes ``alerts.disposition``. That column is
+the analyst's verdict, set from the feedback endpoint, and it is also the
+column reconciliation reads to learn what the analyst decided. Letting a
+shadow verdict land in it would mean the agent filled in the answer it is
+about to be graded against, and every alert an analyst did not explicitly
+re-dispose would score as perfect agreement.
+
+That is the Phase 1 leakage lesson arriving by a different route, and it is
+the reason ``persist_auto_triage`` grew a ``shadow`` flag rather than this
+module simply forwarding the call. Under that flag the ledger still writes the
+run, the event and the ``ai_score`` / ``ai_summary`` /
+``ai_recommendations`` columns, which are the agent's own fields, and leaves
+``disposition``, ``status`` and ``resolved_at`` untouched. The analyst decides
+from a blank field, which is the only way their closure is independent
+evidence.
+
+Cost is not suppressed
+======================
+
+:attr:`ShadowModeTriageWriter.persists_cost` is ``True``. A shadow run places
+real model calls against a real key, and a tenant measuring for a month must
+see that spend. Replay declines it because a replay is a measurement the
+tenant asked for and should not be billed twice for; a shadow run is the
+product running.
+
+Deduplication is not suppressed either
+======================================
+
+:meth:`~ShadowModeTriageWriter.cache_verdict` forwards. Production dedups an
+alert flood through the cost governor, so a shadow run that did not would cost
+more than the thing it is modelling and would measure a pipeline the tenant
+will never operate. Replay declines it for a reason that does not apply here:
+in a replay the cache is a third route by which the test window answers
+itself, and on a live queue there is no test window.
+
+Which way it fails
+==================
+
+:class:`LiveShadowModePolicy` distinguishes "no database configured" from
+"database configured and unreadable", the same distinction
+``app.services.tenant_policy`` draws in the actions service and for the same
+reason. No database means no shadow table, no tenant could have enabled
+anything, and nowhere to record a decision: shadow is off. A configured
+database that will not answer means a tenant may well have asked for shadow
+and we cannot see it, so the run is treated as shadow. Acting on an alert a
+tenant asked us only to observe is the worse of the two errors, and the other
+one is a few seconds of writeback that resumes on the next read.
+"""
+
+from __future__ import annotations
+
+import time
+import uuid
+from typing import Any, Protocol, runtime_checkable
+
+import structlog
+
+from app.investigator import ledger as ledger_module
+
+__all__ = [
+ "UNCLASSIFIED",
+ "LiveShadowModePolicy",
+ "ShadowDecision",
+ "ShadowModePolicy",
+ "ShadowModeTriageWriter",
+ "alert_class_of",
+ "record_shadow_decision",
+]
+
+logger = structlog.get_logger(__name__)
+
+#: Where an alert with no category lands. A null class would drop the alert
+#: out of every per-class aggregate, so a tenant whose detection content sets
+#: no category would appear to have no evidence at all rather than one bucket
+#: of it.
+UNCLASSIFIED = "unclassified"
+
+#: Matches every class. A tenant starting out does not yet know which classes
+#: their queue contains, which is most of what the first weeks of measurement
+#: are for.
+WILDCARD = "*"
+
+#: Policy changes are rare and this is read once per alert, so a short TTL
+#: keeps the common case to zero round trips while still picking up a change
+#: within seconds. Same value and same reasoning as the actions service's
+#: tenant policy cache.
+_CACHE_TTL_SECONDS = 30.0
+
+_cache: dict[tuple[str, str], tuple[float, bool]] = {}
+
+
+def alert_class_of(raw_alert: dict[str, Any] | None) -> str:
+ """The class this alert is measured under.
+
+ ``alerts.category`` is what the console and the detection corpus already
+ group by, so shadow mode uses it rather than inventing a parallel
+ taxonomy that would have to be explained and kept in step.
+ """
+ category = str((raw_alert or {}).get("category") or "").strip().lower()
+ return category or UNCLASSIFIED
+
+
+@runtime_checkable
+class ShadowModePolicy(Protocol):
+ """Whether this tenant is measuring this alert class rather than acting on it."""
+
+ async def is_shadow(self, tenant_ref: str, alert_class: str) -> bool:
+ """True when this alert is to be measured rather than acted on."""
+
+
+class LiveShadowModePolicy:
+ """Reads ``aisoc_shadow_mode``. Treats an unreadable policy as shadow."""
+
+ __slots__ = ()
+
+ async def is_shadow(self, tenant_ref: str, alert_class: str) -> bool:
+ key = (str(tenant_ref), alert_class)
+ cached = _cache.get(key)
+ now = time.monotonic()
+ if cached is not None and now - cached[0] < _CACHE_TTL_SECONDS:
+ return cached[1]
+
+ pool = await ledger_module.get_pool()
+ if pool is None:
+ # No database: nothing could have been enabled and there is
+ # nowhere to record a decision. Not cached, because a pool that
+ # appears later should take effect immediately.
+ return False
+
+ try:
+ async with pool.acquire() as conn:
+ tenant_id = await ledger_module._resolve_tenant_id(conn, str(tenant_ref))
+ if tenant_id is None:
+ # An unknown tenant cannot have configured anything, and a
+ # decision row for it would violate the foreign key.
+ logger.warning("shadow_mode.unknown_tenant", tenant_ref=str(tenant_ref)[:120])
+ return False
+ await ledger_module._set_rls_context(conn, tenant_id)
+ row = await conn.fetchrow(
+ """
+ SELECT bool_or(enabled) AS enabled
+ FROM aisoc_shadow_mode
+ WHERE tenant_id = $1 AND alert_class = ANY($2::text[])
+ """,
+ tenant_id,
+ [alert_class, WILDCARD],
+ )
+ except Exception as exc: # noqa: BLE001 - never act on an unreadable policy
+ logger.error(
+ "shadow_mode.read_failed",
+ tenant_ref=str(tenant_ref)[:120],
+ alert_class=alert_class[:120],
+ error=str(exc)[:300],
+ treating_as="shadow",
+ )
+ return True
+
+ enabled = bool(row and row["enabled"])
+ _cache[key] = (now, enabled)
+ return enabled
+
+
+def clear_cache() -> None:
+ """Drop the shadow-mode cache. Used by tests and after a policy update."""
+ _cache.clear()
+
+
+class ShadowDecision:
+ """One verdict produced in shadow, on its way to the decision ledger.
+
+ A plain object rather than a dataclass mirroring the table: the columns it
+ fills are written out once in :func:`record_shadow_decision`, and a second
+ field list here would be the copy that drifts.
+ """
+
+ __slots__ = (
+ "alert_class",
+ "alert_id",
+ "confidence",
+ "evidence_signature",
+ "external_id",
+ "model",
+ "rule_id",
+ "source",
+ "tenant_ref",
+ "verdict",
+ )
+
+ def __init__(
+ self,
+ *,
+ tenant_ref: str,
+ alert_class: str,
+ alert_id: Any = None,
+ external_id: str | None = None,
+ rule_id: str | None = None,
+ source: str | None = None,
+ model: str | None = None,
+ verdict: str | None = None,
+ confidence: float | None = None,
+ evidence_signature: str | None = None,
+ ) -> None:
+ self.tenant_ref = tenant_ref
+ self.alert_class = alert_class
+ self.alert_id = alert_id
+ self.external_id = external_id
+ self.rule_id = rule_id
+ self.source = source
+ self.model = model
+ self.verdict = verdict
+ self.confidence = confidence
+ self.evidence_signature = evidence_signature
+
+
+async def record_shadow_decision(decision: ShadowDecision) -> bool:
+ """Append the verdict to ``aisoc_shadow_decisions``. Returns whether it landed.
+
+ Idempotent on ``(tenant_id, alert_id)``. A broker redelivery re-triages the
+ same alert, and two rows carrying the same opinion would count as two
+ agreements: the sample would grow toward the promotion threshold without
+ any new evidence behind it. The conflict updates rather than doing nothing,
+ so a re-triage that produced a *different* verdict replaces the old one
+ instead of leaving a stale opinion in the evidence.
+
+ The analyst half is never touched on conflict. A decision that has already
+ been graded keeps its grade.
+ """
+ pool = await ledger_module.get_pool()
+ if pool is None:
+ return False
+
+ alert_uuid = ledger_module._coerce_uuid(decision.alert_id)
+ if alert_uuid is None and not decision.external_id:
+ # Nothing to reconcile against later. Recording it would add a row
+ # that can never become evidence and can never be matched to a
+ # closure, which reads on the scorecard as an agent that never gets
+ # graded rather than as a wiring fault.
+ logger.warning("shadow_mode.decision_unlinkable", tenant_ref=str(decision.tenant_ref)[:120])
+ return False
+
+ try:
+ async with pool.acquire() as conn:
+ tenant_id = await ledger_module._resolve_tenant_id(conn, str(decision.tenant_ref))
+ if tenant_id is None:
+ logger.warning("shadow_mode.decision_unknown_tenant", tenant_ref=str(decision.tenant_ref)[:120])
+ return False
+ await ledger_module._set_rls_context(conn, tenant_id)
+ await conn.execute(
+ """
+ INSERT INTO aisoc_shadow_decisions
+ (id, tenant_id, alert_id, external_id, alert_class, rule_id,
+ source, model, verdict, confidence, evidence_signature, decided_at)
+ VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, now())
+ ON CONFLICT (tenant_id, alert_id) WHERE alert_id IS NOT NULL
+ DO UPDATE SET
+ verdict = EXCLUDED.verdict,
+ confidence = EXCLUDED.confidence,
+ model = EXCLUDED.model,
+ evidence_signature = EXCLUDED.evidence_signature,
+ decided_at = EXCLUDED.decided_at
+ """,
+ uuid.uuid4(),
+ tenant_id,
+ alert_uuid,
+ decision.external_id,
+ decision.alert_class or UNCLASSIFIED,
+ decision.rule_id,
+ decision.source,
+ decision.model,
+ decision.verdict,
+ float(decision.confidence) if decision.confidence is not None else None,
+ decision.evidence_signature,
+ )
+ except Exception as exc: # noqa: BLE001 - a durable verdict outranks its measurement
+ logger.warning(
+ "shadow_mode.decision_write_failed",
+ tenant_ref=str(decision.tenant_ref)[:120],
+ error=str(exc)[:300],
+ )
+ return False
+ return True
+
+
+class ShadowModeTriageWriter:
+ """A :class:`app.workers.triage_persistence.TriageWriter` that records and never acts.
+
+ Wraps the live sink rather than reimplementing it. The two calls that are
+ forwarded are forwarded to the real thing, so a shadow run and a live run
+ differ by which methods are declined and by one flag, not by two code
+ paths that have to be kept in step.
+ """
+
+ def __init__(self, delegate: Any, decision: ShadowDecision) -> None:
+ self._delegate = delegate
+ self._decision = decision
+ #: Whether the decision row landed. Read by the worker's metrics so a
+ #: silently failing measurement shows up as a number rather than as a
+ #: scorecard that never fills in.
+ self.recorded = False
+
+ @property
+ def escalation_allowed(self) -> bool:
+ # Escalation runs the investigation graph, which enriches, proposes and
+ # writes a node per step. That is acting.
+ return False
+
+ @property
+ def persists_cost(self) -> bool:
+ # Real calls against a real key. A tenant measuring for a month must
+ # see the spend.
+ return True
+
+ async def persist_auto_triage(self, **fields: Any) -> None:
+ """Record the verdict without letting it stand in for the analyst's."""
+ fields["auto_closed"] = False
+ fields["shadow"] = True
+ await self._delegate.persist_auto_triage(**fields)
+
+ self._decision.verdict = str(fields.get("verdict") or "") or None
+ confidence = fields.get("confidence")
+ self._decision.confidence = float(confidence) if confidence is not None else None
+ raw = fields.get("raw_alert") or {}
+ if self._decision.rule_id is None:
+ self._decision.rule_id = _opt_text(raw.get("rule_id"))
+ if self._decision.source is None:
+ self._decision.source = _opt_text(raw.get("connector_type") or raw.get("source"))
+ if self._decision.external_id is None:
+ self._decision.external_id = _opt_text(raw.get("external_id"))
+ self.recorded = await record_shadow_decision(self._decision)
+
+ async def record_outcome(
+ self,
+ tenant_id: str,
+ signature: str,
+ *,
+ disposition: str,
+ confidence: float,
+ author: str,
+ alert_id: Any = None,
+ ) -> None:
+ # An outcome prior lets a later identical alert close without triage.
+ # A shadow verdict must never reach that far: it would act on a future
+ # alert through a store rather than directly, and it would put the
+ # agent's own opinion into the evidence for the next one.
+ return None
+
+ async def record_suppression(
+ self,
+ *,
+ tenant_ref: str,
+ signature: str,
+ alert_id: Any,
+ disposition: str,
+ prior_author: str,
+ ) -> None:
+ return None
+
+ async def raise_approval(self, **fields: Any) -> Any:
+ # An approval is a proposal to act, put in front of a human. Under
+ # measurement there is nothing to approve.
+ return None
+
+ async def write_back_disposition(
+ self,
+ *,
+ tenant_id: str,
+ alert_id: str,
+ disposition: str,
+ confidence: float | None = None,
+ rationale: str = "",
+ ) -> dict[str, Any] | None:
+ return None
+
+ def cache_verdict(
+ self,
+ governor: Any,
+ tenant_id: str,
+ fingerprint: str,
+ verdict: dict[str, Any],
+ *,
+ usd: float,
+ tokens: int,
+ ) -> None:
+ self._delegate.cache_verdict(governor, tenant_id, fingerprint, verdict, usd=usd, tokens=tokens)
+
+
+def _opt_text(value: Any) -> str | None:
+ """A trimmed string, or ``None`` for absent and empty alike."""
+ text = str(value).strip() if value is not None else ""
+ return text or None
diff --git a/services/agents/tests/test_escalation_context_depth.py b/services/agents/tests/test_escalation_context_depth.py
index ef5b8ceb0..855157178 100644
--- a/services/agents/tests/test_escalation_context_depth.py
+++ b/services/agents/tests/test_escalation_context_depth.py
@@ -35,6 +35,11 @@ from app.llm import tool_loop
from app.models.state import AgentStatus, InvestigationState
from app.workers import fused_alert_consumer as consumer
from app.workers.fused_alert_consumer import FusedAlertTriageWorker
+from app.workers.triage_persistence import LiveTriageWriter
+
+
+def _worker() -> FusedAlertTriageWorker:
+ return FusedAlertTriageWorker(bootstrap_servers="localhost:9092")
def _state() -> InvestigationState:
@@ -70,7 +75,7 @@ async def test_escalation_prefetches_the_context_bundle(monkeypatch: pytest.Monk
monkeypatch.setenv("AISOC_AGENT_ESCALATION", "1")
state = _state()
- await FusedAlertTriageWorker(bootstrap_servers="localhost:9092")._maybe_escalate(state)
+ await _worker()._maybe_escalate(state, writer=LiveTriageWriter())
assert seen["case_id"] == str(state.incident_id)
assert seen["tenant_id"] == str(state.tenant_id)
@@ -97,7 +102,7 @@ async def test_a_context_failure_never_blocks_the_escalation(
monkeypatch.setattr(consumer, "run_escalation", _run_escalation)
monkeypatch.setenv("AISOC_AGENT_ESCALATION", "1")
- await FusedAlertTriageWorker(bootstrap_servers="localhost:9092")._maybe_escalate(_state())
+ await _worker()._maybe_escalate(_state(), writer=LiveTriageWriter())
assert ran
diff --git a/services/agents/tests/test_raise_approvals.py b/services/agents/tests/test_raise_approvals.py
index ae3193803..634aa9360 100644
--- a/services/agents/tests/test_raise_approvals.py
+++ b/services/agents/tests/test_raise_approvals.py
@@ -62,7 +62,7 @@ class TestWhatGetsQueued:
@pytest.mark.asyncio
async def test_an_approval_requiring_action_is_queued(self, raised):
worker = FusedAlertTriageWorker.__new__(FusedAlertTriageWorker)
- ids = await worker._raise_approvals(_state(_action()))
+ ids = await worker._raise_approvals(_state(_action()), writer=worker._writer)
assert len(ids) == 1
assert len(raised) == 1
@@ -75,7 +75,7 @@ class TestWhatGetsQueued:
nothing either way; it queues the subset that was always meant to
reach a human."""
worker = FusedAlertTriageWorker.__new__(FusedAlertTriageWorker)
- ids = await worker._raise_approvals(_state(_action(requires_approval=False)))
+ ids = await worker._raise_approvals(_state(_action(requires_approval=False)), writer=worker._writer)
assert ids == []
assert raised == []
@@ -85,14 +85,14 @@ class TestWhatGetsQueued:
"""The approval is what the actions service is later rebuilt from, so
losing parameters here means approving a different action."""
worker = FusedAlertTriageWorker.__new__(FusedAlertTriageWorker)
- await worker._raise_approvals(_state(_action()))
+ await worker._raise_approvals(_state(_action()), writer=worker._writer)
assert raised[0]["action"]["parameters"] == {"reason": "c2"}
@pytest.mark.asyncio
async def test_the_rationale_becomes_the_summary_a_human_reads(self, raised):
worker = FusedAlertTriageWorker.__new__(FusedAlertTriageWorker)
- await worker._raise_approvals(_state(_action()))
+ await worker._raise_approvals(_state(_action()), writer=worker._writer)
assert raised[0]["summary"] == "Confirmed C2 beacon."
@@ -103,7 +103,8 @@ class TestWhatGetsQueued:
_state(
_action(),
_action(action_type="disable_user", target="[email protected]"),
- )
+ ),
+ writer=worker._writer,
)
assert len(ids) == 2
@@ -116,7 +117,7 @@ class TestItStaysOptional:
monkeypatch.setenv("AISOC_AGENT_RAISE_APPROVALS", "0")
worker = FusedAlertTriageWorker.__new__(FusedAlertTriageWorker)
- assert await worker._raise_approvals(_state(_action())) == []
+ assert await worker._raise_approvals(_state(_action()), writer=worker._writer) == []
assert raised == []
@pytest.mark.asyncio
@@ -130,7 +131,7 @@ class TestItStaysOptional:
monkeypatch.setattr(ledger_module, "raise_approval", _raise_approval)
worker = FusedAlertTriageWorker.__new__(FusedAlertTriageWorker)
- assert await worker._raise_approvals(_state(_action())) == []
+ assert await worker._raise_approvals(_state(_action()), writer=worker._writer) == []
class TestIdempotency:
diff --git a/services/agents/tests/test_shadow_mode.py b/services/agents/tests/test_shadow_mode.py
new file mode 100644
index 000000000..b84c6d156
--- /dev/null
+++ b/services/agents/tests/test_shadow_mode.py
@@ -0,0 +1,371 @@
+"""Shadow mode on the live queue: it records, and it does not act.
+
+Gap-closure Phase 2.1.
+
+The property under test is not "fewer writes happen". It is that the agent
+never fills in the answer it is about to be graded against. ``alerts.disposition``
+is the analyst's column and the column reconciliation reads, so a shadow
+verdict landing there would make every alert an analyst did not explicitly
+re-dispose score as perfect agreement, and the scorecard would climb toward a
+promotion on nothing at all.
+
+That is the Phase 1 leakage lesson arriving by a different route. Phase 1's
+version was a verdict written as an outcome prior that suppressed the next
+alert; this one is a verdict written into the field the next measurement
+reads. Both are the evaluation answering itself.
+"""
+
+from __future__ import annotations
+
+import uuid
+from typing import Any
+
+import pytest
+from app.investigator import ledger as ledger_module
+from app.workers import shadow_mode as shadow_module
+from app.workers.fused_alert_consumer import FusedAlertTriageWorker
+from app.workers.shadow_mode import (
+ UNCLASSIFIED,
+ LiveShadowModePolicy,
+ ShadowDecision,
+ ShadowModeTriageWriter,
+ alert_class_of,
+)
+from app.workers.triage_persistence import LiveTriageWriter, TriageWriter
+
+
+class _RecordingDelegate:
+ """A live sink that records rather than performs. Stands in for production."""
+
+ def __init__(self) -> None:
+ self.persisted: list[dict[str, Any]] = []
+ self.cached: list[tuple[Any, ...]] = []
+
+ @property
+ def escalation_allowed(self) -> bool:
+ return True
+
+ @property
+ def persists_cost(self) -> bool:
+ return True
+
+ async def persist_auto_triage(self, **fields: Any) -> None:
+ self.persisted.append(fields)
+
+ def cache_verdict(self, governor, tenant_id, fingerprint, verdict, *, usd, tokens) -> None:
+ self.cached.append((tenant_id, fingerprint, verdict, usd, tokens))
+
+
+def _decision(**overrides: Any) -> ShadowDecision:
+ fields: dict[str, Any] = {
+ "tenant_ref": "11111111-1111-1111-1111-111111111111",
+ "alert_class": "identity",
+ "alert_id": str(uuid.uuid4()),
+ }
+ fields.update(overrides)
+ return ShadowDecision(**fields)
+
+
+def _no_decision_writes(monkeypatch: pytest.MonkeyPatch) -> list[ShadowDecision]:
+ """Capture decision rows instead of writing them, and fail loudly on a pool."""
+ captured: list[ShadowDecision] = []
+
+ async def _record(decision: ShadowDecision) -> bool:
+ captured.append(decision)
+ return True
+
+ monkeypatch.setattr(shadow_module, "record_shadow_decision", _record)
+ return captured
+
+
+class TestItDoesNotAct:
+ @pytest.mark.asyncio
+ async def test_every_acting_method_declines(self, _no_decision_writes):
+ """Each of these changes something outside the measurement.
+
+ An approval is a proposal put in front of a human; a writeback edits a
+ record in the customer's SIEM; an outcome prior closes the *next*
+ matching alert without triage. Under measurement none of them may
+ happen, and the delegate they wrap is a recorder, so a forwarded call
+ would show up as a recorded one.
+ """
+ delegate = _RecordingDelegate()
+ writer = ShadowModeTriageWriter(delegate, _decision())
+
+ assert await writer.raise_approval(tenant_ref="t", run_id=None) is None
+ assert await writer.write_back_disposition(tenant_id="t", alert_id="a", disposition="true_positive") is None
+ assert await writer.record_outcome("t", "sig", disposition="benign", confidence=0.9, author="ai") is None
+ assert (
+ await writer.record_suppression(tenant_ref="t", signature="sig", alert_id=None, disposition="benign", prior_author="ai") is None
+ )
+ assert delegate.persisted == []
+
+ def test_escalation_is_declined(self):
+ """The investigation graph enriches, proposes and writes a node per step."""
+ writer = ShadowModeTriageWriter(_RecordingDelegate(), _decision())
+ assert writer.escalation_allowed is False
+
+ def test_cost_is_still_persisted(self):
+ """A shadow run places real model calls against a real key.
+
+ Replay declines this because a replay is a measurement the tenant
+ asked for; a shadow run is the product running, and a tenant
+ measuring for a month must see the spend.
+ """
+ writer = ShadowModeTriageWriter(_RecordingDelegate(), _decision())
+ assert writer.persists_cost is True
+
+ def test_deduplication_still_happens(self):
+ """Production dedups an alert flood, so a shadow run that did not
+ would cost more than the thing it is modelling."""
+ delegate = _RecordingDelegate()
+ writer = ShadowModeTriageWriter(delegate, _decision())
+ writer.cache_verdict(object(), "t", "fp", {"verdict": "benign"}, usd=0.0, tokens=10)
+ assert len(delegate.cached) == 1
+
+
+class TestTheVerdictIsRecordedWithoutStandingInForTheAnalysts:
+ @pytest.mark.asyncio
+ async def test_persist_forces_the_shadow_flag_and_refuses_to_auto_close(self, _no_decision_writes):
+ delegate = _RecordingDelegate()
+ writer = ShadowModeTriageWriter(delegate, _decision())
+
+ await writer.persist_auto_triage(
+ run_id=uuid.uuid4(),
+ alert_id=str(uuid.uuid4()),
+ tenant_ref="t",
+ alert_summary="",
+ raw_alert={},
+ tier="llm",
+ verdict="false_positive",
+ confidence=0.97,
+ rationale="",
+ auto_closed=True,
+ )
+
+ written = delegate.persisted[0]
+ assert written["shadow"] is True
+ assert written["auto_closed"] is False
+
+ @pytest.mark.asyncio
+ async def test_the_decision_row_carries_the_verdict_and_its_segments(self, _no_decision_writes):
+ delegate = _RecordingDelegate()
+ decision = _decision(rule_id=None, source=None)
+ writer = ShadowModeTriageWriter(delegate, decision)
+
+ await writer.persist_auto_triage(
+ run_id=uuid.uuid4(),
+ alert_id=decision.alert_id,
+ tenant_ref="t",
+ alert_summary="",
+ raw_alert={"rule_id": "det-identity-004", "connector_type": "okta", "external_id": "notable-9"},
+ tier="llm",
+ verdict="true_positive",
+ confidence=0.91,
+ rationale="",
+ )
+
+ recorded = _no_decision_writes[0]
+ assert recorded.verdict == "true_positive"
+ assert recorded.confidence == pytest.approx(0.91)
+ # Per-rule and per-source agreement are two of the four breakdowns the
+ # plan asks for, and they are only possible if the keys are captured
+ # at decision time. The alert row can be edited afterwards.
+ assert recorded.rule_id == "det-identity-004"
+ assert recorded.source == "okta"
+ assert recorded.external_id == "notable-9"
+
+ @pytest.mark.asyncio
+ async def test_an_empty_verdict_is_recorded_as_none_not_as_an_empty_string(self, _no_decision_writes):
+ """``None`` and ``''`` both mean "the agent did not decide".
+
+ They must not be two values in the column, because the aggregate
+ COALESCEs one into the other and a reader looking at the raw rows
+ would otherwise see two abstention spellings and conclude they meant
+ different things.
+ """
+ writer = ShadowModeTriageWriter(_RecordingDelegate(), _decision())
+ await writer.persist_auto_triage(
+ run_id=uuid.uuid4(),
+ alert_id=str(uuid.uuid4()),
+ tenant_ref="t",
+ alert_summary="",
+ raw_alert={},
+ tier="llm",
+ verdict="",
+ confidence=0.0,
+ rationale="",
+ )
+ assert _no_decision_writes[0].verdict is None
+
+
+class TestTheLedgerLeavesTheAnalystsColumnsAlone:
+ """The integrity property, asserted on the statement that carries it.
+
+ There is no Postgres in this suite, so this reads the SQL rather than
+ running it: it proves the ``disposition`` assignment and the resolution
+ columns are guarded by the shadow parameter and that the parameter is
+ bound. What it cannot prove is that Postgres evaluates the CASE the way
+ the statement reads, which is what ``tests/isolation`` exists for. Stated
+ here rather than left implied, because a test whose limits are not
+ written down gets read as proving more than it does.
+ """
+
+ def test_the_disposition_write_is_guarded_by_the_shadow_parameter(self):
+ source = ledger_module.persist_auto_triage.__doc__ or ""
+ assert "shadow" in source
+ import inspect
+
+ body = inspect.getsource(ledger_module.persist_auto_triage)
+ assert "SET disposition = CASE WHEN $10 THEN disposition ELSE $3 END" in body
+ assert "status = CASE WHEN $7 AND NOT $10 THEN 'resolved' ELSE status END" in body
+ assert "resolved_at = CASE WHEN $7 AND NOT $10 THEN now() ELSE resolved_at END" in body
+
+ def test_the_guard_does_not_depend_on_the_caller_forcing_auto_closed_off(self):
+ """``NOT $10`` is what makes this survive a forgetful future caller.
+
+ The shadow sink already forces ``auto_closed`` false. If the guard
+ relied on that alone, a second caller added later would close alerts
+ it was only meant to observe, and the symptom would be a tenant's
+ queue emptying itself during an evaluation.
+ """
+ import inspect
+
+ body = inspect.getsource(ledger_module.persist_auto_triage)
+ assert "CASE WHEN $7 AND NOT $10" in body
+
+
+class TestWhichWayThePolicyFails:
+ @pytest.mark.asyncio
+ async def test_no_database_means_not_shadow(self, monkeypatch: pytest.MonkeyPatch):
+ """Nothing could have been enabled and there is nowhere to record."""
+ shadow_module.clear_cache()
+
+ async def _no_pool():
+ return None
+
+ monkeypatch.setattr(ledger_module, "get_pool", _no_pool)
+ assert await LiveShadowModePolicy().is_shadow("tenant", "identity") is False
+
+ @pytest.mark.asyncio
+ async def test_an_unreadable_policy_means_shadow(self, monkeypatch: pytest.MonkeyPatch):
+ """The conservative direction, and the same call ``tenant_policy`` makes.
+
+ Acting on an alert a tenant asked us only to observe is worse than a
+ few seconds of suppressed writeback, and the second one resumes by
+ itself on the next read.
+ """
+ shadow_module.clear_cache()
+
+ class _BrokenPool:
+ def acquire(self):
+ raise RuntimeError("connection reset")
+
+ async def _pool():
+ return _BrokenPool()
+
+ monkeypatch.setattr(ledger_module, "get_pool", _pool)
+ assert await LiveShadowModePolicy().is_shadow("tenant", "identity") is True
+
+ @pytest.mark.asyncio
+ async def test_a_failure_is_not_cached(self, monkeypatch: pytest.MonkeyPatch):
+ """A transient read error must not pin a tenant into shadow for the TTL."""
+ shadow_module.clear_cache()
+ calls: list[int] = []
+
+ class _BrokenPool:
+ def acquire(self):
+ calls.append(1)
+ raise RuntimeError("connection reset")
+
+ async def _pool():
+ return _BrokenPool()
+
+ monkeypatch.setattr(ledger_module, "get_pool", _pool)
+ policy = LiveShadowModePolicy()
+ await policy.is_shadow("tenant", "identity")
+ await policy.is_shadow("tenant", "identity")
+ assert len(calls) == 2
+
+
+class TestTheClassAnAlertIsMeasuredUnder:
+ def test_it_is_the_alert_category(self):
+ assert alert_class_of({"category": "Identity"}) == "identity"
+
+ def test_an_alert_with_no_category_still_lands_somewhere(self):
+ """A null class would drop the alert out of every per-class aggregate.
+
+ A tenant whose detection content sets no category would then appear to
+ have no evidence at all, rather than one bucket of it.
+ """
+ assert alert_class_of({}) == UNCLASSIFIED
+ assert alert_class_of(None) == UNCLASSIFIED
+ assert alert_class_of({"category": " "}) == UNCLASSIFIED
+
+
+class TestTheWorkerPicksTheSinkPerAlert:
+ @pytest.mark.asyncio
+ async def test_a_tenant_not_measuring_gets_the_live_sink_unchanged(self):
+ class _Never:
+ async def is_shadow(self, tenant_ref: str, alert_class: str) -> bool:
+ return False
+
+ worker = FusedAlertTriageWorker(bootstrap_servers="localhost:9092", shadow_policy=_Never())
+ state = _state()
+ assert await worker._sink_for(state, "fp") is worker._writer
+
+ @pytest.mark.asyncio
+ async def test_a_measured_class_gets_a_wrapper_around_that_same_sink(self):
+ """The wrapper delegates rather than replacing.
+
+ A replay passing its own sink keeps it, and a production worker wraps
+ the live one. Two separate sinks would be two code paths to keep in
+ step, and the one that drifts is always the copy.
+ """
+
+ class _Always:
+ async def is_shadow(self, tenant_ref: str, alert_class: str) -> bool:
+ return True
+
+ worker = FusedAlertTriageWorker(bootstrap_servers="localhost:9092", shadow_policy=_Always())
+ sink = await worker._sink_for(_state(), "fp")
+ assert isinstance(sink, ShadowModeTriageWriter)
+ assert sink._delegate is worker._writer
+ assert isinstance(sink, TriageWriter)
+
+ @pytest.mark.asyncio
+ async def test_the_class_comes_from_the_alert_being_triaged(self):
+ seen: list[str] = []
+
+ class _Recording:
+ async def is_shadow(self, tenant_ref: str, alert_class: str) -> bool:
+ seen.append(alert_class)
+ return False
+
+ worker = FusedAlertTriageWorker(bootstrap_servers="localhost:9092", shadow_policy=_Recording())
+ await worker._sink_for(_state(category="cloud"), "fp")
+ assert seen == ["cloud"]
+
+ def test_the_default_policy_is_the_live_one(self):
+ """A worker built with ``__new__`` still resolves a policy.
+
+ Several tests construct one that way to drive a single method, and a
+ policy that existed only as an instance attribute would leave those
+ objects raising ``AttributeError`` on the first alert.
+ """
+ worker = FusedAlertTriageWorker.__new__(FusedAlertTriageWorker)
+ assert isinstance(worker._shadow_policy, LiveShadowModePolicy)
+ assert isinstance(worker._writer, LiveTriageWriter)
+
+
+def _state(category: str = "identity"):
+ from app.models.state import InvestigationState
+
+ return InvestigationState(
+ run_id=uuid.uuid4(),
+ incident_id=uuid.uuid4(),
+ tenant_id=uuid.uuid4(),
+ alert_summary="",
+ raw_alert={"id": str(uuid.uuid4()), "category": category},
+ )
diff --git a/services/api/app/_vendor/autonomy_evidence_rules.py b/services/api/app/_vendor/autonomy_evidence_rules.py
new file mode 100644
index 000000000..2256f4ff7
--- /dev/null
+++ b/services/api/app/_vendor/autonomy_evidence_rules.py
@@ -0,0 +1,386 @@
+"""How agreement between the agent and a tenant's analysts is measured, decided once.
+
+Gap-closure Phase 2.2. The promotion gate that reads these numbers arrives in
+Phase 2.3 and sits on top of this file rather than beside it.
+
+Two services have to agree about this and neither can import the other.
+``services/actions`` enforces autonomy at dispatch, where an over-generous
+answer executes something at a customer's vendor. ``services/api`` renders the
+track record and decides promotions, owns the tenant session and owns the
+hash-chained audit log a transition is written to. Both package their code as
+top-level ``app``, so one process holds one of them, and each Docker image is
+built with only its own service directory as context.
+
+So this module is pure: standard library only, no database handle, no HTTP
+client, no service import. ``services/api`` carries a byte-identical copy at
+``app/_vendor/autonomy_evidence_rules.py`` and
+``scripts/sync_vendored_autonomy_evidence.py --check`` fails the build when the
+two drift. Same arrangement as the LLM input contract and the deterministic
+narrative, and used here for a stronger reason: a safety control that two
+services define differently is a control that is off in whichever one is more
+generous, and nobody finds out until something executes that the other half
+would have refused.
+
+The metrics are Phase 1's, not a second set
+===========================================
+
+``packages/aisoc-benchmark/aisoc_benchmark/replay.py`` already defines what
+"malicious recall" means, which verdicts are abstentions, which dispositions
+may be graded, and that a rate with no denominator is ``None`` rather than
+zero. Restating any of those differently here would give live measurement a
+private definition of accuracy, and the first time the two disagreed the
+replay report and the scorecard would be describing different agents while
+both looked right. ``scripts/check_replay_contract_parity.py`` compares the
+constants below against that module in both directions.
+
+Agreement is computed over answered decisions only
+==================================================
+
+This is the one a naive implementation gets wrong, and it is worth stating
+where the arithmetic is rather than where it is read. If agreement counted
+every decision and scored an abstention as "not a disagreement", an agent that
+answered a tenth of its queue confidently and routed the rest to a human would
+post a near-perfect record on a population it never attempted. Abstaining
+removes a decision from the numerator and the denominator together, so it
+cannot move the rate; what it moves is
+:attr:`AgreementWindow.abstention_rate`, which is reported beside it.
+
+Malicious recall is the counterweight and runs the other way: its denominator
+is every malicious case, abstained or not, so an abstention counts there as a
+miss. Phase 1 makes exactly that distinction, for exactly that reason, and an
+alert routed to a human was not caught by the agent.
+
+A window and a trailing slice, never just the window
+====================================================
+
+A thirty-day window is an average, and an average is where a gradual decline
+hides: an agent that agreed 99% of the time for three weeks and 70% of the
+time this week still posts about 95% over the window. So the aggregate comes
+in two statements, one over the window and one over the most recent
+:attr:`PromotionThresholds.drift_sample` decisions, and every surface that
+shows one shows both. The trailing slice is a count rather than a date range
+so it means the same thing for a tenant with ten alerts a day and for one with
+ten thousand.
+
+The thresholds live here even though nothing in this file enforces them
+======================================================================
+
+:class:`PromotionThresholds` carries the window length, which this file needs,
+and the floors, which it does not. They travel together because the scorecard
+shows progress toward them ("47 of the 100 decisions, 12 of the 30 malicious
+ones") and because a floor stated in one service and enforced in another is
+the pair that drifts.
+"""
+
+from __future__ import annotations
+
+from dataclasses import asdict, dataclass
+from typing import Any
+
+__all__ = [
+ "ABSTENTION_VERDICTS",
+ "AGREEMENT_COUNTS_SQL",
+ "DEFAULT_THRESHOLDS",
+ "GRADED_DISPOSITIONS",
+ "MALICIOUS",
+ "RECENT_COUNTS_SQL",
+ "UNLABELED",
+ "AgreementWindow",
+ "PromotionThresholds",
+ "scoped_sql",
+ "to_named_params",
+ "window_from_counts",
+]
+
+#: The disposition that means "this was a real threat". One spelling, shared
+#: with the writeback taxonomy and the replay grader.
+MALICIOUS = "true_positive"
+
+#: What an analyst may close a finding as. Mirrors ``GRADED_DISPOSITIONS`` in
+#: the benchmark package and ``CANONICAL_DISPOSITIONS`` in the writeback, both
+#: pinned by ``scripts/check_replay_contract_parity.py``.
+GRADED_DISPOSITIONS: tuple[str, ...] = (
+ MALICIOUS,
+ "benign_true_positive",
+ "false_positive",
+ "benign",
+)
+
+#: An analyst who declined to classify. Excluded from agreement entirely,
+#: never guessed at.
+UNLABELED = "unlabeled"
+
+#: Agent outputs that route the alert to a human rather than deciding it.
+#: Counted apart from agreement so the two cannot be traded off invisibly.
+ABSTENTION_VERDICTS: frozenset[str] = frozenset({"needs_review", "escalate", "unknown", ""})
+
+
+@dataclass(frozen=True)
+class PromotionThresholds:
+ """What a tenant's track record has to show. Every value is configurable.
+
+ Defaults are the plan's: 100 decisions, at least 30 of them malicious.
+ The rest are set here rather than left to a caller because a threshold
+ with no default is a threshold somebody eventually passes zero for.
+ """
+
+ #: Labelled decisions required in the window.
+ min_decisions: int = 100
+ #: How many of those must have been closed by an analyst as malicious.
+ min_malicious: int = 30
+ #: Agreement over *answered* decisions required to promote.
+ min_agreement: float = 0.95
+ #: Recall on malicious required to promote. An abstention is a miss.
+ min_malicious_recall: float = 0.90
+ #: Share of labelled decisions that may be abstentions.
+ max_abstention_rate: float = 0.30
+ #: How far back the window reaches.
+ window_days: int = 30
+ #: Agreement below this demotes an existing grant.
+ demotion_agreement: float = 0.90
+ #: Malicious recall below this demotes an existing grant.
+ demotion_malicious_recall: float = 0.80
+ #: Size of the trailing slice scored separately, so gradual drift inside a
+ #: passing window is still caught.
+ drift_sample: int = 50
+ #: The trailing slice needs at least this many answered decisions before
+ #: it may demote on its own. Without it a single disagreement in a quiet
+ #: week would revoke a grant built on months of evidence.
+ drift_min_answered: int = 20
+
+ def as_dict(self) -> dict[str, Any]:
+ return asdict(self)
+
+
+DEFAULT_THRESHOLDS = PromotionThresholds()
+
+
+def _ratio(numerator: int, denominator: int) -> float | None:
+ """A rate, or ``None`` when there was nothing to divide by.
+
+ ``None`` renders as "not measured". A ``0.0`` in an agreement column says
+ the agent was wrong every time; no denominator says it was never asked,
+ and those call for opposite responses.
+ """
+ return (numerator / denominator) if denominator else None
+
+
+@dataclass(frozen=True)
+class AgreementWindow:
+ """Agreement between the agent and the analysts over one slice of history.
+
+ Every rate on this object is accompanied by the count it was computed
+ over, and is ``None`` when that count is zero. A caller that renders a
+ rate without its count has thrown away the difference between a
+ measurement and a coincidence.
+ """
+
+ #: Shadow decisions in the slice that an analyst has since closed.
+ resolved: int = 0
+ #: Of those, the ones carrying a disposition this platform can name.
+ labelled: int = 0
+ #: Labelled decisions the agent declined to decide.
+ abstained: int = 0
+ #: Labelled decisions the agent answered. The denominator of agreement.
+ answered: int = 0
+ #: Answered decisions where the agent's verdict is the analyst's label.
+ agreed: int = 0
+ #: Labelled decisions an analyst closed as malicious.
+ malicious_support: int = 0
+ #: Of those, the ones the agent also called malicious.
+ malicious_caught: int = 0
+
+ def __post_init__(self) -> None:
+ # A window whose parts do not add up is a bug in the aggregate query,
+ # and it would show as a plausible-looking rate rather than an error.
+ if self.answered + self.abstained != self.labelled:
+ raise ValueError(
+ f"answered ({self.answered}) + abstained ({self.abstained}) must equal "
+ f"labelled ({self.labelled}); the aggregate query and this type disagree"
+ )
+ if self.labelled > self.resolved:
+ raise ValueError(f"labelled ({self.labelled}) cannot exceed resolved ({self.resolved})")
+ if self.agreed > self.answered:
+ raise ValueError(f"agreed ({self.agreed}) cannot exceed answered ({self.answered})")
+ if self.malicious_caught > self.malicious_support:
+ raise ValueError(f"malicious_caught ({self.malicious_caught}) cannot exceed malicious_support ({self.malicious_support})")
+
+ @property
+ def unlabeled(self) -> int:
+ """Closures the analyst left unclassified. Excluded from every rate."""
+ return self.resolved - self.labelled
+
+ @property
+ def agreement_rate(self) -> float | None:
+ """Agreement over answered decisions only.
+
+ Abstentions are absent from both halves of this fraction. That is
+ what stops an agent from earning a grant by declining the hard cases:
+ declining removes a decision from the numerator and the denominator
+ together, so the rate does not move and the abstention rate does.
+ """
+ return _ratio(self.agreed, self.answered)
+
+ @property
+ def malicious_recall(self) -> float | None:
+ """Share of the analysts' malicious closures the agent also called malicious.
+
+ The denominator is every malicious case, answered or abstained, so an
+ abstention counts as a miss here even though it is excluded from
+ agreement. An alert routed to a human was not caught by the agent.
+ """
+ return _ratio(self.malicious_caught, self.malicious_support)
+
+ @property
+ def abstention_rate(self) -> float | None:
+ return _ratio(self.abstained, self.labelled)
+
+ def as_dict(self) -> dict[str, Any]:
+ """Counts and the rates derived from them, in one payload.
+
+ The derived rates are included rather than left to the reader because
+ this dict is what lands in the audit log, and a snapshot that stores
+ only counts requires whoever reads it in six months to re-derive the
+ arithmetic and hope they used the same definition.
+ """
+ return {
+ "resolved": self.resolved,
+ "labelled": self.labelled,
+ "unlabeled": self.unlabeled,
+ "answered": self.answered,
+ "abstained": self.abstained,
+ "agreed": self.agreed,
+ "malicious_support": self.malicious_support,
+ "malicious_caught": self.malicious_caught,
+ "agreement_rate": self.agreement_rate,
+ "malicious_recall": self.malicious_recall,
+ "abstention_rate": self.abstention_rate,
+ }
+
+
+def window_from_counts(counts: dict[str, Any]) -> AgreementWindow:
+ """Build a window from the aggregate query's row.
+
+ Takes a mapping rather than keyword arguments because both callers get one
+ from their driver, and asyncpg's ``Record`` and SQLAlchemy's ``RowMapping``
+ are both mappings while neither is a dict.
+ """
+ return AgreementWindow(
+ resolved=int(counts.get("resolved") or 0),
+ labelled=int(counts.get("labelled") or 0),
+ abstained=int(counts.get("abstained") or 0),
+ answered=int(counts.get("answered") or 0),
+ agreed=int(counts.get("agreed") or 0),
+ malicious_support=int(counts.get("malicious_support") or 0),
+ malicious_caught=int(counts.get("malicious_caught") or 0),
+ )
+
+
+# ---------------------------------------------------------------------------
+# The aggregate, written once
+# ---------------------------------------------------------------------------
+#
+# Both services run these two statements verbatim. Aggregating in the database
+# rather than fetching rows keeps the dispatch path from pulling a month of
+# decisions across the wire to divide two numbers, and writing the statement
+# here rather than in each service is what stops one of them from quietly
+# counting abstentions into agreement.
+#
+# The placeholders are numbered (`$1`) for asyncpg. SQLAlchemy callers pass the
+# statement through `text()` after `to_named_params()` renames them, because
+# the two drivers disagree about placeholder syntax and only about that.
+
+#: Scope filters are applied by the caller as an additional predicate, appended
+#: where this marker sits. Written as a marker rather than as string
+#: concatenation at the call site so both services splice at the same point.
+_SCOPE_MARKER = "/*scope*/"
+
+AGREEMENT_COUNTS_SQL = f"""
+SELECT
+ COUNT(*)::int AS resolved,
+ COUNT(*) FILTER (WHERE d.analyst_disposition = ANY($2::text[]))::int AS labelled,
+ COUNT(*) FILTER (
+ WHERE d.analyst_disposition = ANY($2::text[])
+ AND COALESCE(d.verdict, '') = ANY($3::text[])
+ )::int AS abstained,
+ COUNT(*) FILTER (
+ WHERE d.analyst_disposition = ANY($2::text[])
+ AND NOT (COALESCE(d.verdict, '') = ANY($3::text[]))
+ )::int AS answered,
+ COUNT(*) FILTER (
+ WHERE d.analyst_disposition = ANY($2::text[])
+ AND NOT (COALESCE(d.verdict, '') = ANY($3::text[]))
+ AND d.verdict = d.analyst_disposition
+ )::int AS agreed,
+ COUNT(*) FILTER (WHERE d.analyst_disposition = $4)::int AS malicious_support,
+ COUNT(*) FILTER (WHERE d.analyst_disposition = $4 AND d.verdict = $4)::int AS malicious_caught
+FROM aisoc_shadow_decisions d
+WHERE d.tenant_id = $1
+ AND d.resolved_at IS NOT NULL
+ AND d.resolved_at >= $5
+ AND d.resolved_at <= $6
+ {_SCOPE_MARKER}
+"""
+
+#: Same counts over the most recent ``$7`` resolved decisions. The slice is by
+#: count rather than by date so it means the same thing at ten alerts a day and
+#: at ten thousand, and it is ordered by ``resolved_at`` rather than by
+#: ``decided_at`` because a decision only joins the evidence when an analyst
+#: closes it.
+RECENT_COUNTS_SQL = f"""
+WITH recent AS (
+ SELECT d.verdict, d.analyst_disposition
+ FROM aisoc_shadow_decisions d
+ WHERE d.tenant_id = $1
+ AND d.resolved_at IS NOT NULL
+ AND d.resolved_at >= $5
+ AND d.resolved_at <= $6
+ {_SCOPE_MARKER}
+ ORDER BY d.resolved_at DESC, d.id DESC
+ LIMIT $7
+)
+SELECT
+ COUNT(*)::int AS resolved,
+ COUNT(*) FILTER (WHERE d.analyst_disposition = ANY($2::text[]))::int AS labelled,
+ COUNT(*) FILTER (
+ WHERE d.analyst_disposition = ANY($2::text[])
+ AND COALESCE(d.verdict, '') = ANY($3::text[])
+ )::int AS abstained,
+ COUNT(*) FILTER (
+ WHERE d.analyst_disposition = ANY($2::text[])
+ AND NOT (COALESCE(d.verdict, '') = ANY($3::text[]))
+ )::int AS answered,
+ COUNT(*) FILTER (
+ WHERE d.analyst_disposition = ANY($2::text[])
+ AND NOT (COALESCE(d.verdict, '') = ANY($3::text[]))
+ AND d.verdict = d.analyst_disposition
+ )::int AS agreed,
+ COUNT(*) FILTER (WHERE d.analyst_disposition = $4)::int AS malicious_support,
+ COUNT(*) FILTER (WHERE d.analyst_disposition = $4 AND d.verdict = $4)::int AS malicious_caught
+FROM recent d
+"""
+
+
+def scoped_sql(statement: str, predicate: str = "") -> str:
+ """Splice a scope predicate into one of the statements above.
+
+ The predicate is written by the caller from a fixed vocabulary of column
+ names and placeholder numbers; nothing derived from a request reaches it.
+ Passing an empty predicate scores every class the tenant runs, which is
+ the tenant-wide row the scorecard leads with.
+ """
+ return statement.replace(_SCOPE_MARKER, predicate)
+
+
+def to_named_params(statement: str, names: list[str]) -> str:
+ """Rewrite ``$1 … $n`` into ``:name`` for the SQLAlchemy caller.
+
+ Rewriting beats keeping a second copy of a forty-line aggregate: the copy
+ is the thing that drifts, and this way the statement the API runs is
+ provably the statement ``services/actions`` runs with the placeholders
+ spelled the other way.
+ """
+ # Highest index first, so `$1` does not match inside `$10`.
+ for index in range(len(names), 0, -1):
+ statement = statement.replace(f"${index}", f":{names[index - 1]}")
+ return statement
diff --git a/services/api/app/api/v1/endpoints/autonomy_policy.py b/services/api/app/api/v1/endpoints/autonomy_policy.py
index 25a8f71fb..382992695 100644
--- a/services/api/app/api/v1/endpoints/autonomy_policy.py
+++ b/services/api/app/api/v1/endpoints/autonomy_policy.py
@@ -22,11 +22,19 @@ from __future__ import annotations
from datetime import UTC, datetime
import structlog
-from fastapi import APIRouter, HTTPException, status
+from fastapi import APIRouter, HTTPException, Query, status
from pydantic import BaseModel, Field, model_validator
from sqlalchemy import text
+from app._vendor.autonomy_evidence_rules import PromotionThresholds
from app.api.v1.deps import AuthUser, DBSession
+from app.db.rls import TenantDBSession
+from app.services.shadow_agreement import (
+ SCOPE_COLUMNS,
+ agreement_for,
+ breakdown_for,
+ reconcile_local_closures,
+)
logger = structlog.get_logger()
@@ -469,3 +477,270 @@ async def reset_action_threshold(
action=action,
reset_by=user.email,
)
+
+
+# ---------------------------------------------------------------------------
+# Shadow mode and rolling agreement (gap-closure Phase 2.1 and 2.2)
+#
+# The thresholds above answer "how confident must the agent be". These answer
+# a different question the same admin screen has to carry: "has it earned the
+# right to be believed". They live in this module rather than beside it
+# because an operator raising an autonomy threshold and an operator reading
+# the track record that justifies it are the same person on the same visit,
+# and splitting the two across routers would mean neither surface shows both.
+# ---------------------------------------------------------------------------
+
+#: Matches every class. What a tenant turns on first, before they know which
+#: classes their own queue contains.
+WILDCARD = "*"
+
+#: An alert class is an `alerts.category` value. Constrained here so a caller
+#: cannot write a 2 KB string into a primary key, and lower-cased so `Identity`
+#: and `identity` are not two classes with half the evidence each.
+_MAX_CLASS_LEN = 100
+
+
+class ShadowModeEntry(BaseModel):
+ alert_class: str
+ enabled: bool
+ enabled_at: str | None = None
+ updated_at: str | None = None
+ updated_by: str | None = None
+
+
+class ShadowModeResponse(BaseModel):
+ tenant_id: str
+ entries: list[ShadowModeEntry]
+
+
+class ShadowModeUpdateRequest(BaseModel):
+ enabled: bool
+
+
+class AgreementWindowModel(BaseModel):
+ """Counts, and the rates derived from them.
+
+ Every rate is optional and is ``null`` when its denominator was zero. The
+ console renders that as "not measured". A zero here would say the agent
+ was wrong every time, which is a different fact with a different remedy,
+ and it is the more flattering of the two to print by accident in the
+ abstention column and the more damning in the agreement column.
+ """
+
+ resolved: int
+ labelled: int
+ unlabeled: int
+ answered: int
+ abstained: int
+ agreed: int
+ malicious_support: int
+ malicious_caught: int
+ agreement_rate: float | None = None
+ malicious_recall: float | None = None
+ abstention_rate: float | None = None
+
+
+class AgreementScopeModel(BaseModel):
+ key: str
+ window: AgreementWindowModel
+
+
+class AgreementResponse(BaseModel):
+ tenant_id: str
+ scope_kind: str
+ scope_key: str
+ window: AgreementWindowModel
+ #: The trailing slice of the most recent decisions, scored separately.
+ #: A window average is where a gradual decline hides, so the surface that
+ #: justifies a promotion has to show both or it is showing the flattering
+ #: half.
+ recent: AgreementWindowModel
+ window_start: str
+ window_end: str
+ thresholds: dict[str, float | int]
+ reconciled: int
+ by_alert_class: list[AgreementScopeModel]
+ by_rule: list[AgreementScopeModel]
+ by_source: list[AgreementScopeModel]
+ by_model: list[AgreementScopeModel]
+
+
+def _normalise_class(value: str) -> str:
+ cleaned = (value or "").strip().lower()
+ if cleaned == WILDCARD:
+ return WILDCARD
+ if not cleaned or len(cleaned) > _MAX_CLASS_LEN or not cleaned.replace("_", "").replace("-", "").isalnum():
+ raise HTTPException(
+ status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
+ detail=f"alert class must be '{WILDCARD}' or an alphanumeric category name of at most {_MAX_CLASS_LEN} characters",
+ )
+ return cleaned
+
+
[email protected]("/shadow-mode", response_model=ShadowModeResponse)
+async def get_shadow_mode(
+ user: AuthUser,
+ db: TenantDBSession,
+) -> ShadowModeResponse:
+ """Which alert classes this tenant is measuring rather than acting on."""
+ await user.require_permission_db("settings:read", db)
+
+ rows = (
+ (
+ await db.execute(
+ text(
+ """
+ SELECT alert_class, enabled, enabled_at, updated_at, updated_by
+ FROM aisoc_shadow_mode
+ WHERE tenant_id = :tenant_id
+ ORDER BY alert_class
+ """
+ ),
+ {"tenant_id": str(user.tenant_id)},
+ )
+ )
+ .mappings()
+ .all()
+ )
+
+ return ShadowModeResponse(
+ tenant_id=str(user.tenant_id),
+ entries=[
+ ShadowModeEntry(
+ alert_class=row["alert_class"],
+ enabled=bool(row["enabled"]),
+ enabled_at=row["enabled_at"].isoformat() if row["enabled_at"] else None,
+ updated_at=row["updated_at"].isoformat() if row["updated_at"] else None,
+ updated_by=str(row["updated_by"]) if row["updated_by"] else None,
+ )
+ for row in rows
+ ],
+ )
+
+
[email protected]("/shadow-mode/{alert_class}", response_model=ShadowModeEntry)
+async def set_shadow_mode(
+ alert_class: str,
+ payload: ShadowModeUpdateRequest,
+ user: AuthUser,
+ db: TenantDBSession,
+) -> ShadowModeEntry:
+ """Start or stop measuring one alert class.
+
+ ``enabled_at`` is stamped on the transition into shadow and cleared on the
+ way out, rather than being set on every write. A promotion window that
+ reaches further back than the day measurement started is reaching back
+ before there was anything to measure, and the only way a later reader can
+ tell is if this column records the real start.
+ """
+ await user.require_permission_db("settings:write", db)
+ cleaned = _normalise_class(alert_class)
+
+ row = (
+ (
+ await db.execute(
+ text(
+ """
+ INSERT INTO aisoc_shadow_mode (tenant_id, alert_class, enabled, enabled_at, updated_by, updated_at)
+ VALUES (:tenant_id, :alert_class, :enabled, CASE WHEN :enabled THEN now() ELSE NULL END, :updated_by, now())
+ ON CONFLICT (tenant_id, alert_class) DO UPDATE SET
+ enabled = EXCLUDED.enabled,
+ enabled_at = CASE
+ WHEN EXCLUDED.enabled AND NOT aisoc_shadow_mode.enabled THEN now()
+ WHEN EXCLUDED.enabled THEN aisoc_shadow_mode.enabled_at
+ ELSE NULL
+ END,
+ updated_by = EXCLUDED.updated_by,
+ updated_at = now()
+ RETURNING alert_class, enabled, enabled_at, updated_at, updated_by
+ """
+ ),
+ {
+ "tenant_id": str(user.tenant_id),
+ "alert_class": cleaned,
+ "enabled": payload.enabled,
+ "updated_by": str(user.user_id),
+ },
+ )
+ )
+ .mappings()
+ .one()
+ )
+ await db.commit()
+
+ logger.info(
+ "autonomy_policy.shadow_mode_set",
+ tenant_id=str(user.tenant_id),
+ alert_class=cleaned,
+ enabled=payload.enabled,
+ updated_by=user.email,
+ )
+ return ShadowModeEntry(
+ alert_class=row["alert_class"],
+ enabled=bool(row["enabled"]),
+ enabled_at=row["enabled_at"].isoformat() if row["enabled_at"] else None,
+ updated_at=row["updated_at"].isoformat() if row["updated_at"] else None,
+ updated_by=str(row["updated_by"]) if row["updated_by"] else None,
+ )
+
+
[email protected]("/agreement", response_model=AgreementResponse)
+async def get_agreement(
+ user: AuthUser,
+ db: TenantDBSession,
+ scope_kind: str = Query("tenant", description="tenant, alert_class, rule, source or model"),
+ scope_key: str = Query(WILDCARD, max_length=200),
+) -> AgreementResponse:
+ """Rolling agreement between the agent and this tenant's own analysts.
+
+ The tenant comes from the credential. ``scope_kind`` and ``scope_key``
+ narrow *within* that tenant and cannot widen beyond it: the scope names a
+ dimension from a fixed vocabulary, never a column, and the tenant
+ predicate is applied whatever the scope says.
+
+ Closures made in this console are reconciled on the way in rather than by
+ a background job. The sweep is bounded and indexed, and doing it here
+ means the number an operator is looking at includes the alert they closed
+ a minute ago, which is exactly when they come to look.
+ """
+ await user.require_permission_db("settings:read", db)
+
+ if scope_kind != "tenant" and scope_kind not in SCOPE_COLUMNS:
+ raise HTTPException(
+ status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
+ detail=f"scope_kind must be 'tenant' or one of {sorted(SCOPE_COLUMNS)}",
+ )
+
+ reconciled = await reconcile_local_closures(db, user.tenant_id)
+ if reconciled:
+ await db.commit()
+
+ thresholds = PromotionThresholds()
+ evidence = await agreement_for(
+ db,
+ user.tenant_id,
+ scope_kind=scope_kind,
+ scope_key=scope_key,
+ thresholds=thresholds,
+ )
+
+ breakdowns: dict[str, list[AgreementScopeModel]] = {}
+ for dimension in ("alert_class", "rule", "source", "model"):
+ rows = await breakdown_for(db, user.tenant_id, scope_kind=dimension, thresholds=thresholds)
+ breakdowns[dimension] = [AgreementScopeModel(key=row.key, window=AgreementWindowModel(**row.window.as_dict())) for row in rows]
+
+ return AgreementResponse(
+ tenant_id=str(user.tenant_id),
+ scope_kind=evidence.scope_kind,
+ scope_key=evidence.scope_key,
+ window=AgreementWindowModel(**evidence.window.as_dict()),
+ recent=AgreementWindowModel(**evidence.recent.as_dict()),
+ window_start=evidence.window_start.isoformat(),
+ window_end=evidence.window_end.isoformat(),
+ thresholds=thresholds.as_dict(),
+ reconciled=reconciled,
+ by_alert_class=breakdowns["alert_class"],
+ by_rule=breakdowns["rule"],
+ by_source=breakdowns["source"],
+ by_model=breakdowns["model"],
+ )
diff --git a/services/api/app/services/shadow_agreement.py b/services/api/app/services/shadow_agreement.py
new file mode 100644
index 000000000..01af007b6
--- /dev/null
+++ b/services/api/app/services/shadow_agreement.py
@@ -0,0 +1,336 @@
+"""Rolling agreement between the agent and the analysts who closed the same alerts.
+
+Gap-closure Phase 2.2.
+
+``services/agents`` writes a row to ``aisoc_shadow_decisions`` every time it
+triages an alert for a tenant that is measuring rather than acting. That row
+holds half a comparison. This module supplies the other half and then does the
+arithmetic:
+
+* :func:`reconcile_local_closures` copies an analyst's own closure off the
+ ``alerts`` row onto the decision it should be graded against.
+* :func:`agreement_for` runs the two aggregates from the shared rules module
+ and returns the window plus the trailing drift slice.
+* :func:`breakdown_for` runs the same aggregate grouped by class, rule, source
+ and model, which is what the scorecard and the operations dashboard render.
+
+Why the arithmetic is not here
+==============================
+
+Every rate on :class:`AgreementWindow` and both SQL statements live in
+``app/_vendor/autonomy_evidence_rules.py``, a byte-identical copy of the
+module ``services/actions`` enforces from. Recomputing an agreement rate here
+would give the console one definition and the dispatch gate another, and the
+two would only be discovered to differ when a tenant asked why the scorecard
+said they qualified and the gate said they did not.
+
+Why reconciliation is a sweep and not a hook
+============================================
+
+The obvious design puts a call on the disposition endpoint: analyst closes an
+alert, the decision row updates. It would miss most closures. An alert can be
+closed by the feedback endpoint, by a case being resolved, by a bulk action,
+by a playbook, or in the source SIEM entirely, and a hook on one of those
+paths silently grades a biased subset: the alerts closed the way the hook was
+written for. A sweep over ``alerts`` asks the question the evidence actually
+needs, which is "has anybody closed this yet, by any route".
+
+A closure the platform cannot name
+==================================
+
+``alerts.disposition`` is free-ish text and only four values are gradeable.
+Anything else becomes ``unlabeled``: recorded as resolved, excluded from every
+rate. The alternative is to guess, and a guess here manufactures agreement out
+of an analyst's shrug. A closure with no disposition at all is treated the
+same way, because "closed, reason not recorded" is exactly that shrug.
+"""
+
+from __future__ import annotations
+
+import uuid
+from dataclasses import dataclass
+from datetime import UTC, datetime, timedelta
+from typing import Any
+
+import structlog
+from sqlalchemy import text
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app._vendor.autonomy_evidence_rules import (
+ ABSTENTION_VERDICTS,
+ AGREEMENT_COUNTS_SQL,
+ GRADED_DISPOSITIONS,
+ MALICIOUS,
+ RECENT_COUNTS_SQL,
+ UNLABELED,
+ AgreementWindow,
+ PromotionThresholds,
+ scoped_sql,
+ to_named_params,
+ window_from_counts,
+)
+
+logger = structlog.get_logger(__name__)
+
+__all__ = [
+ "AgreementEvidence",
+ "ScopeBreakdown",
+ "agreement_for",
+ "breakdown_for",
+ "reconcile_local_closures",
+]
+
+#: The placeholder names the two aggregates bind, in ``$1 … $7`` order.
+_PARAM_NAMES = [
+ "tenant_id",
+ "graded",
+ "abstentions",
+ "malicious",
+ "window_start",
+ "window_end",
+ "recent_limit",
+]
+
+#: Which shadow-decision column each scope dimension reads. Fixed vocabulary:
+#: a caller names a dimension, never a column, so nothing from a request
+#: reaches the statement.
+SCOPE_COLUMNS: dict[str, str] = {
+ "alert_class": "alert_class",
+ "rule": "rule_id",
+ "source": "source",
+ "model": "model",
+}
+
+
+@dataclass(frozen=True)
+class AgreementEvidence:
+ """One scope's track record: the window, and the trailing slice of it."""
+
+ scope_kind: str
+ scope_key: str
+ window: AgreementWindow
+ recent: AgreementWindow
+ window_start: datetime
+ window_end: datetime
+
+ def as_dict(self) -> dict[str, Any]:
+ return {
+ "scope_kind": self.scope_kind,
+ "scope_key": self.scope_key,
+ "window": self.window.as_dict(),
+ "recent": self.recent.as_dict(),
+ "window_start": self.window_start.isoformat(),
+ "window_end": self.window_end.isoformat(),
+ }
+
+
+@dataclass(frozen=True)
+class ScopeBreakdown:
+ """One row of a per-class, per-rule, per-source or per-model table."""
+
+ key: str
+ window: AgreementWindow
+
+ def as_dict(self) -> dict[str, Any]:
+ return {"key": self.key, **self.window.as_dict()}
+
+
+def _bind(
+ tenant_id: uuid.UUID | str,
+ window_start: datetime,
+ window_end: datetime,
+ recent_limit: int,
+) -> dict[str, Any]:
+ return {
+ "tenant_id": str(tenant_id),
+ "graded": list(GRADED_DISPOSITIONS),
+ "abstentions": sorted(ABSTENTION_VERDICTS),
+ "malicious": MALICIOUS,
+ "window_start": window_start,
+ "window_end": window_end,
+ "recent_limit": recent_limit,
+ }
+
+
+async def reconcile_local_closures(
+ db: AsyncSession,
+ tenant_id: uuid.UUID | str,
+ *,
+ limit: int = 5000,
+) -> int:
+ """Grade every shadow decision whose alert an analyst has since closed here.
+
+ Returns how many decisions gained an analyst label on this pass. Idempotent:
+ a decision that already carries ``resolved_at`` is never revisited, so an
+ analyst who later changes their mind does not silently rewrite evidence a
+ promotion was already granted on. That is deliberate. The record of what
+ was known at the time is the thing an audit needs; a later correction is a
+ new fact, and it arrives as new decisions rather than by editing old ones.
+ """
+ statement = text(
+ """
+ UPDATE aisoc_shadow_decisions d
+ SET analyst_disposition = CASE
+ WHEN a.disposition = ANY(:graded) THEN a.disposition
+ ELSE :unlabeled
+ END,
+ resolution_source = 'aisoc',
+ resolved_at = COALESCE(a.resolved_at, a.updated_at, now()),
+ vendor_disposition = a.disposition
+ FROM alerts a
+ WHERE d.alert_id = a.id
+ AND d.tenant_id = :tenant_id
+ AND a.tenant_id = :tenant_id
+ AND d.resolved_at IS NULL
+ AND a.status IN ('resolved', 'closed')
+ AND d.id IN (
+ SELECT id FROM aisoc_shadow_decisions
+ WHERE tenant_id = :tenant_id AND resolved_at IS NULL
+ ORDER BY decided_at
+ LIMIT :limit
+ )
+ """
+ )
+ result = await db.execute(
+ statement,
+ {
+ "tenant_id": str(tenant_id),
+ "graded": list(GRADED_DISPOSITIONS),
+ "unlabeled": UNLABELED,
+ "limit": int(limit),
+ },
+ )
+ reconciled = int(result.rowcount or 0)
+ if reconciled:
+ logger.info("shadow_agreement.reconciled", tenant_id=str(tenant_id), decisions=reconciled)
+ return reconciled
+
+
+def _scope_predicate(scope_kind: str, scope_key: str) -> tuple[str, dict[str, Any]]:
+ """The extra ``WHERE`` clause for one scope, plus what it binds.
+
+ The column comes from :data:`SCOPE_COLUMNS`, never from the caller's
+ string, so a scope name that is not in the vocabulary raises rather than
+ reaching the statement.
+ """
+ if scope_kind == "tenant":
+ return "", {}
+ column = SCOPE_COLUMNS.get(scope_kind)
+ if column is None:
+ raise ValueError(f"unknown scope kind {scope_kind!r}; expected one of {sorted(SCOPE_COLUMNS) + ['tenant']}")
+ return f"AND d.{column} = :scope_key", {"scope_key": scope_key}
+
+
+async def agreement_for(
+ db: AsyncSession,
+ tenant_id: uuid.UUID | str,
+ *,
+ scope_kind: str = "tenant",
+ scope_key: str = "*",
+ thresholds: PromotionThresholds | None = None,
+ now: datetime | None = None,
+) -> AgreementEvidence:
+ """The tenant's rolling agreement for one scope, plus its trailing slice.
+
+ ``now`` is injectable so a test can pin the window; production passes
+ nothing and gets the wall clock. The window bounds are returned as
+ absolute timestamps rather than as "the last 30 days", because the second
+ form stops meaning anything the moment somebody reads it on a later day.
+ """
+ limits = thresholds or PromotionThresholds()
+ window_end = now or datetime.now(UTC)
+ window_start = window_end - timedelta(days=limits.window_days)
+
+ predicate, extra = _scope_predicate(scope_kind, scope_key)
+ params = {**_bind(tenant_id, window_start, window_end, limits.drift_sample), **extra}
+
+ window_row = (
+ (await db.execute(text(to_named_params(scoped_sql(AGREEMENT_COUNTS_SQL, predicate), _PARAM_NAMES)), params)).mappings().first()
+ )
+ recent_row = (
+ (await db.execute(text(to_named_params(scoped_sql(RECENT_COUNTS_SQL, predicate), _PARAM_NAMES)), params)).mappings().first()
+ )
+
+ return AgreementEvidence(
+ scope_kind=scope_kind,
+ scope_key=scope_key,
+ window=window_from_counts(dict(window_row or {})),
+ recent=window_from_counts(dict(recent_row or {})),
+ window_start=window_start,
+ window_end=window_end,
+ )
+
+
+async def breakdown_for(
+ db: AsyncSession,
+ tenant_id: uuid.UUID | str,
+ *,
+ scope_kind: str,
+ thresholds: PromotionThresholds | None = None,
+ now: datetime | None = None,
+ limit: int = 50,
+) -> list[ScopeBreakdown]:
+ """Agreement grouped by one dimension: class, rule, source or model.
+
+ Ordered by sample size rather than by rate, so the table leads with the
+ scopes there is most evidence about. A 100% agreement over three decisions
+ sorted to the top would be the most prominent number on the page and the
+ least informative one.
+ """
+ limits = thresholds or PromotionThresholds()
+ window_end = now or datetime.now(UTC)
+ window_start = window_end - timedelta(days=limits.window_days)
+
+ column = SCOPE_COLUMNS.get(scope_kind)
+ if column is None:
+ raise ValueError(f"unknown scope kind {scope_kind!r}; expected one of {sorted(SCOPE_COLUMNS)}")
+
+ # The same FILTER expressions as the shared aggregate, grouped rather than
+ # scoped. Kept beside the caller instead of in the rules module because
+ # only the console needs it: the dispatch gate asks about one scope at a
+ # time and must never pull a whole table across to answer that.
+ statement = text(
+ f"""
+ SELECT
+ COALESCE(NULLIF(d.{column}, ''), 'unattributed') AS key,
+ COUNT(*)::int AS resolved,
+ COUNT(*) FILTER (WHERE d.analyst_disposition = ANY(:graded))::int AS labelled,
+ COUNT(*) FILTER (
+ WHERE d.analyst_disposition = ANY(:graded)
+ AND COALESCE(d.verdict, '') = ANY(:abstentions)
+ )::int AS abstained,
+ COUNT(*) FILTER (
+ WHERE d.analyst_disposition = ANY(:graded)
+ AND NOT (COALESCE(d.verdict, '') = ANY(:abstentions))
+ )::int AS answered,
+ COUNT(*) FILTER (
+ WHERE d.analyst_disposition = ANY(:graded)
+ AND NOT (COALESCE(d.verdict, '') = ANY(:abstentions))
+ AND d.verdict = d.analyst_disposition
+ )::int AS agreed,
+ COUNT(*) FILTER (WHERE d.analyst_disposition = :malicious)::int AS malicious_support,
+ COUNT(*) FILTER (WHERE d.analyst_disposition = :malicious AND d.verdict = :malicious)::int AS malicious_caught
+ FROM aisoc_shadow_decisions d
+ WHERE d.tenant_id = :tenant_id
+ AND d.resolved_at IS NOT NULL
+ AND d.resolved_at >= :window_start
+ AND d.resolved_at <= :window_end
+ GROUP BY 1
+ ORDER BY labelled DESC, key ASC
+ LIMIT :row_limit
+ """
+ )
+ rows = (
+ (
+ await db.execute(
+ statement,
+ {
+ **_bind(tenant_id, window_start, window_end, limits.drift_sample),
+ "row_limit": int(limit),
+ },
+ )
+ )
+ .mappings()
+ .all()
+ )
+ return [ScopeBreakdown(key=str(row["key"]), window=window_from_counts(dict(row))) for row in rows]
diff --git a/services/api/migrations/066_shadow_autonomy.sql b/services/api/migrations/066_shadow_autonomy.sql
new file mode 100644
index 000000000..d8f46fe1a
--- /dev/null
+++ b/services/api/migrations/066_shadow_autonomy.sql
@@ -0,0 +1,184 @@
+-- 066: what the agent said, what the analyst said, and whether they agreed.
+--
+-- Gap-closure Phase 2.1 and 2.2.
+--
+-- Two tables, because they answer different questions and neither answers the
+-- other.
+--
+-- aisoc_shadow_mode which alert classes this tenant is measuring
+-- rather than acting on
+-- aisoc_shadow_decisions one row per verdict produced while measuring,
+-- plus the analyst's own closure when it arrives
+--
+-- Why a decision ledger and not a counter
+-- ---------------------------------------
+-- A running agreement percentage is cheap and useless. The question asked of
+-- a promotion six months later is "which decisions was that built on", and a
+-- counter cannot answer it. Rows can: they carry the alert, the rule, the
+-- source, the model and both verdicts, so a disputed promotion resolves into
+-- a list of alerts somebody can open.
+--
+-- Why agreement is not stored
+-- ---------------------------
+-- There is no `agreed` column. Agreement is `verdict = analyst_disposition`,
+-- derived in the aggregate query in `autonomy_evidence_rules.py`, so there is
+-- exactly one definition of it. A stored boolean would be a second definition
+-- that is correct on the day it is written and silently wrong after the
+-- taxonomy moves, and it would be wrong in the direction that reads as a
+-- better track record.
+--
+-- Why the analyst half is nullable
+-- --------------------------------
+-- Most rows sit unresolved for days. An alert the agent triaged this morning
+-- has no analyst closure yet, and it must not be counted as agreement or as
+-- disagreement in the meantime. `resolved_at IS NULL` is the whole of "not
+-- evidence yet", and every aggregate filters on it.
+--
+-- `analyst_disposition` accepts `unlabeled`, which is the analyst saying "I
+-- do not know" rather than the absence of an answer. Splunk ES ships two
+-- dispositions that mean exactly that, and so do Sentinel and Defender.
+-- Excluding those rows from accuracy while still counting them as resolved is
+-- what stops a tenant whose history is mostly unlabeled from seeing a
+-- confident rate derived from a remnant.
+--
+-- Why the class is a plain string
+-- -------------------------------
+-- `alerts.category` is `VARCHAR(100)` and nullable, and the classes that
+-- appear in it are set by detection content rather than by an enum here. A
+-- foreign key would make an ingest of new detection content fail at the
+-- shadow ledger, which is the wrong place for that to break. Rows with no
+-- category land under `unclassified` so they are still measurable rather
+-- than invisible.
+
+CREATE TABLE IF NOT EXISTS aisoc_shadow_mode (
+ tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,
+ -- An `alerts.category` value, or `*` for every class this tenant sees.
+ -- The wildcard is how a tenant starts: measuring one class at a time
+ -- requires knowing which classes exist, which is what the first weeks of
+ -- measurement are for.
+ alert_class TEXT NOT NULL,
+ -- Off. A deployment that has not asked to measure is not measuring, and
+ -- defaulting this on would start writing decision rows for every tenant
+ -- on upgrade.
+ enabled BOOLEAN NOT NULL DEFAULT FALSE,
+ enabled_at TIMESTAMPTZ,
+ updated_by UUID REFERENCES users(id) ON DELETE SET NULL,
+ updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ PRIMARY KEY (tenant_id, alert_class)
+);
+
+COMMENT ON TABLE aisoc_shadow_mode IS
+ 'Which alert classes a tenant is triaging in shadow: the verdict is recorded '
+ 'and measured, and nothing is written back, acted on or auto-closed.';
+COMMENT ON COLUMN aisoc_shadow_mode.enabled_at IS
+ 'When measurement started. A promotion window that reaches back before this '
+ 'is reaching back before there was anything to measure.';
+
+CREATE TABLE IF NOT EXISTS aisoc_shadow_decisions (
+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+ tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,
+ -- The AiSOC alert this verdict was about. Nullable because a finding
+ -- polled straight from a customer's SIEM may never have become an alert
+ -- row here.
+ alert_id UUID,
+ -- The vendor's own finding id. This is the join key an analyst closure
+ -- polled from the source SIEM arrives on, and without it a closure made
+ -- in Splunk cannot be matched to the verdict it should be graded against.
+ external_id TEXT,
+ alert_class TEXT NOT NULL DEFAULT 'unclassified',
+ rule_id TEXT,
+ -- `alerts.connector_type`: which product the finding came from.
+ source TEXT,
+ -- What the gateway actually resolved the request to, not the alias asked
+ -- for. Agreement is per model as well as per class, and an alias is not a
+ -- model.
+ model TEXT,
+ verdict TEXT,
+ confidence DOUBLE PRECISION,
+ -- The cost governor's evidence fingerprint, so a disputed row can be tied
+ -- back to the evidence the verdict was formed on.
+ evidence_signature TEXT,
+ decided_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+
+ -- The analyst half. All null until somebody closes the alert.
+ analyst_disposition TEXT,
+ -- Where the closure came from: `aisoc` when an analyst closed it in this
+ -- console, or the vendor name when it was polled back out of their SIEM.
+ -- Recorded because "our analysts agree with it" and "their SIEM agrees
+ -- with it" are different claims and a promotion may rest on either.
+ resolution_source TEXT,
+ resolved_at TIMESTAMPTZ,
+ resolved_by TEXT,
+ -- The vendor's own label, verbatim. A mapping somebody later disputes is
+ -- only arguable if what was actually recorded survived.
+ vendor_disposition TEXT,
+
+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+);
+
+COMMENT ON TABLE aisoc_shadow_decisions IS
+ 'One row per shadow triage verdict, with the analyst closure it is graded '
+ 'against once that arrives. The evidence behind every autonomy promotion.';
+COMMENT ON COLUMN aisoc_shadow_decisions.resolved_at IS
+ 'Null until an analyst closes the alert. A row with no closure is not '
+ 'evidence yet and every aggregate excludes it.';
+COMMENT ON COLUMN aisoc_shadow_decisions.analyst_disposition IS
+ 'Canonical disposition, or the literal `unlabeled` when the analyst '
+ 'declined to classify. Unlabeled rows count as resolved and are excluded '
+ 'from every rate.';
+COMMENT ON COLUMN aisoc_shadow_decisions.verdict IS
+ 'The agent verdict. A null here is an abstention, the same as an explicit '
+ '`needs_review`: both mean the agent did not decide.';
+
+-- One verdict per alert. A worker that re-triages after a broker redelivery
+-- must not double-count its own opinion into the evidence: two identical rows
+-- would show as two agreements and inflate the sample toward promotion.
+CREATE UNIQUE INDEX IF NOT EXISTS uq_shadow_decisions_alert
+ ON aisoc_shadow_decisions (tenant_id, alert_id) WHERE alert_id IS NOT NULL;
+CREATE UNIQUE INDEX IF NOT EXISTS uq_shadow_decisions_external
+ ON aisoc_shadow_decisions (tenant_id, source, external_id)
+ WHERE alert_id IS NULL AND external_id IS NOT NULL;
+
+-- The window aggregate filters on `resolved_at` and orders by it for the
+-- trailing drift slice, so that is the index it needs. Partial, because the
+-- unresolved rows are never in any aggregate and are the majority early on.
+CREATE INDEX IF NOT EXISTS idx_shadow_decisions_resolved
+ ON aisoc_shadow_decisions (tenant_id, resolved_at DESC)
+ WHERE resolved_at IS NOT NULL;
+CREATE INDEX IF NOT EXISTS idx_shadow_decisions_class
+ ON aisoc_shadow_decisions (tenant_id, alert_class, resolved_at DESC)
+ WHERE resolved_at IS NOT NULL;
+-- Reconciliation looks a decision up by the vendor finding id when a closure
+-- arrives from the source SIEM.
+CREATE INDEX IF NOT EXISTS idx_shadow_decisions_external_lookup
+ ON aisoc_shadow_decisions (tenant_id, external_id)
+ WHERE external_id IS NOT NULL AND resolved_at IS NULL;
+
+-- Row-level security. The `OR current_tenant_id() IS NULL` arm is what lets
+-- the cross-tenant workers (retention purge, tenant deletion) and the agents
+-- worker's unbound pool read these; dropping it makes those silently see
+-- nothing. FORCE so the owner does not walk past.
+ALTER TABLE aisoc_shadow_mode ENABLE ROW LEVEL SECURITY;
+ALTER TABLE aisoc_shadow_mode FORCE ROW LEVEL SECURITY;
+DROP POLICY IF EXISTS aisoc_shadow_mode_tenant ON aisoc_shadow_mode;
+CREATE POLICY aisoc_shadow_mode_tenant ON aisoc_shadow_mode
+ USING (tenant_id = current_tenant_id() OR current_tenant_id() IS NULL);
+
+ALTER TABLE aisoc_shadow_decisions ENABLE ROW LEVEL SECURITY;
+ALTER TABLE aisoc_shadow_decisions FORCE ROW LEVEL SECURITY;
+DROP POLICY IF EXISTS aisoc_shadow_decisions_tenant ON aisoc_shadow_decisions;
+CREATE POLICY aisoc_shadow_decisions_tenant ON aisoc_shadow_decisions
+ USING (tenant_id = current_tenant_id() OR current_tenant_id() IS NULL);
+
+-- 061 set ALTER DEFAULT PRIVILEGES so new tables pick these up, but only for
+-- tables created by the role that ran it. Granting explicitly means a chain
+-- replayed by a different owner still leaves the runtime role able to work.
+DO $$
+BEGIN
+ IF EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'aisoc_app') THEN
+ GRANT SELECT, INSERT, UPDATE, DELETE ON aisoc_shadow_mode TO aisoc_app;
+ GRANT SELECT, INSERT, UPDATE, DELETE ON aisoc_shadow_decisions TO aisoc_app;
+ END IF;
+END
+$$;
diff --git a/services/api/tests/test_shadow_agreement.py b/services/api/tests/test_shadow_agreement.py
new file mode 100644
index 000000000..34b3bf661
--- /dev/null
+++ b/services/api/tests/test_shadow_agreement.py
@@ -0,0 +1,241 @@
+"""Rolling agreement, as the console asks for it.
+
+Gap-closure Phase 2.2.
+
+Two things are worth testing here and they are not the arithmetic, which lives
+in the shared rules module and is tested there against the same file this
+service vendors.
+
+The first is that a scope cannot widen past the tenant. ``scope_kind`` and
+``scope_key`` arrive from a query string, and the shape of this feature -
+"show me agreement for one alert class" - is exactly the shape that invites a
+column name to be interpolated into a statement. The dimension is looked up in
+a fixed table and the key is bound, and both halves are asserted.
+
+The second is that reconciliation never guesses a label. ``alerts.disposition``
+holds whatever the closing path wrote, and only four values are gradeable.
+Everything else has to become ``unlabeled``, which is counted as resolved and
+excluded from every rate. Guessing here would manufacture agreement out of an
+analyst's shrug, and it would do it silently and in the flattering direction.
+
+There is no Postgres in this suite. The session is mocked, so what is proven
+is the statement that gets built and the parameters bound to it; whether
+Postgres evaluates that statement as read is what the isolation suite covers.
+"""
+
+from __future__ import annotations
+
+import inspect
+import uuid
+from datetime import UTC, datetime
+from types import SimpleNamespace
+from typing import Any
+from unittest.mock import AsyncMock, MagicMock
+
+import pytest
+from app._vendor.autonomy_evidence_rules import (
+ GRADED_DISPOSITIONS,
+ MALICIOUS,
+ UNLABELED,
+ AgreementWindow,
+ PromotionThresholds,
+)
+from app.services import shadow_agreement
+from app.services.shadow_agreement import (
+ SCOPE_COLUMNS,
+ _scope_predicate,
+ agreement_for,
+ breakdown_for,
+ reconcile_local_closures,
+)
+
+TENANT = uuid.UUID("11111111-1111-1111-1111-111111111111")
+NOW = datetime(2026, 9, 26, 12, 0, tzinfo=UTC)
+
+_COUNTS = {
+ "resolved": 120,
+ "labelled": 100,
+ "abstained": 10,
+ "answered": 90,
+ "agreed": 87,
+ "malicious_support": 31,
+ "malicious_caught": 29,
+}
+
+
+def _session(rows: list[dict[str, Any]] | None = None, rowcount: int = 0) -> MagicMock:
+ """An ``AsyncSession`` that answers every execute with the same rows."""
+ db = MagicMock()
+ mappings = MagicMock()
+ mappings.first.return_value = dict(_COUNTS)
+ mappings.all.return_value = rows if rows is not None else []
+ result = MagicMock()
+ result.mappings.return_value = mappings
+ result.rowcount = rowcount
+ db.execute = AsyncMock(return_value=result)
+ db.commit = AsyncMock()
+ return db
+
+
+class TestAScopeCannotWidenPastTheTenant:
+ def test_an_unknown_dimension_is_refused_rather_than_interpolated(self):
+ with pytest.raises(ValueError, match="unknown scope kind"):
+ _scope_predicate("alert_class; DROP TABLE alerts --", "x")
+
+ def test_the_vocabulary_is_the_four_the_plan_names(self):
+ """Per alert class, rule, source and model. Nothing else is a column here."""
+ assert set(SCOPE_COLUMNS) == {"alert_class", "rule", "source", "model"}
+
+ def test_the_key_is_bound_and_never_written_into_the_statement(self):
+ predicate, params = _scope_predicate("rule", "det-identity-004' OR '1'='1")
+ assert predicate == "AND d.rule_id = :scope_key"
+ assert params == {"scope_key": "det-identity-004' OR '1'='1"}
+
+ def test_the_tenant_scope_adds_no_predicate_at_all(self):
+ assert _scope_predicate("tenant", "*") == ("", {})
+
+ @pytest.mark.asyncio
+ async def test_every_statement_binds_the_tenant_from_the_caller(self):
+ db = _session()
+ await agreement_for(db, TENANT, scope_kind="alert_class", scope_key="identity", now=NOW)
+
+ for call in db.execute.await_args_list:
+ statement, params = call.args
+ assert params["tenant_id"] == str(TENANT)
+ assert "d.tenant_id = :tenant_id" in str(statement)
+
+
+class TestTheWindowIsAbsolute:
+ @pytest.mark.asyncio
+ async def test_the_bounds_come_from_the_configured_window_length(self):
+ """Reported as timestamps, not as "the last 30 days".
+
+ The second form stops meaning anything the moment somebody reads it
+ on a later day, which is precisely when a promotion gets questioned.
+ """
+ db = _session()
+ evidence = await agreement_for(db, TENANT, thresholds=PromotionThresholds(window_days=7), now=NOW)
+
+ assert evidence.window_end == NOW
+ assert (evidence.window_end - evidence.window_start).days == 7
+
+ @pytest.mark.asyncio
+ async def test_the_trailing_slice_is_bounded_by_the_drift_sample(self):
+ db = _session()
+ await agreement_for(db, TENANT, thresholds=PromotionThresholds(drift_sample=25), now=NOW)
+
+ _, params = db.execute.await_args_list[-1].args
+ assert params["recent_limit"] == 25
+
+ @pytest.mark.asyncio
+ async def test_both_the_window_and_its_recent_slice_are_returned(self):
+ """A surface handed only the window is being handed the flattering half."""
+ evidence = await agreement_for(_session(), TENANT, now=NOW)
+ assert isinstance(evidence.window, AgreementWindow)
+ assert isinstance(evidence.recent, AgreementWindow)
+ assert evidence.window.agreement_rate == pytest.approx(87 / 90)
+
+
+class TestReconciliationNeverGuessesALabel:
+ @pytest.mark.asyncio
+ async def test_a_disposition_outside_the_taxonomy_becomes_unlabeled(self):
+ db = _session(rowcount=4)
+ await reconcile_local_closures(db, TENANT)
+
+ statement, params = db.execute.await_args_list[0].args
+ sql = str(statement)
+ # The CASE is what refuses to guess: a gradeable value is copied,
+ # anything else lands on `unlabeled`.
+ assert "WHEN a.disposition = ANY(:graded) THEN a.disposition" in sql
+ assert "ELSE :unlabeled" in sql
+ assert params["unlabeled"] == UNLABELED
+ assert params["graded"] == list(GRADED_DISPOSITIONS)
+
+ @pytest.mark.asyncio
+ async def test_it_only_touches_decisions_that_are_not_yet_graded(self):
+ """An analyst who later changes their mind must not rewrite evidence.
+
+ A promotion already granted rests on what was known at the time, and
+ the audit record of that has to stay true. A correction is a new fact
+ and arrives as new decisions.
+ """
+ db = _session(rowcount=0)
+ await reconcile_local_closures(db, TENANT)
+
+ sql = str(db.execute.await_args_list[0].args[0])
+ assert "d.resolved_at IS NULL" in sql
+
+ @pytest.mark.asyncio
+ async def test_it_only_reads_closed_alerts(self):
+ db = _session(rowcount=0)
+ await reconcile_local_closures(db, TENANT)
+ assert "a.status IN ('resolved', 'closed')" in str(db.execute.await_args_list[0].args[0])
+
+ @pytest.mark.asyncio
+ async def test_it_reports_how_many_it_graded(self):
+ assert await reconcile_local_closures(_session(rowcount=7), TENANT) == 7
+
+ @pytest.mark.asyncio
+ async def test_the_sweep_is_bounded(self):
+ """It runs inline on a request, so it cannot be allowed to grow without limit."""
+ db = _session(rowcount=0)
+ await reconcile_local_closures(db, TENANT, limit=250)
+ assert db.execute.await_args_list[0].args[1]["limit"] == 250
+
+
+class TestTheBreakdownLeadsWithEvidence:
+ @pytest.mark.asyncio
+ async def test_rows_are_ordered_by_sample_size_not_by_rate(self):
+ """A perfect rate over three decisions sorted to the top would be the
+ most prominent number on the page and the least informative one."""
+ db = _session(rows=[])
+ await breakdown_for(db, TENANT, scope_kind="rule", now=NOW)
+ assert "ORDER BY labelled DESC" in str(db.execute.await_args_list[0].args[0])
+
+ @pytest.mark.asyncio
+ async def test_an_unattributed_key_is_named_rather_than_dropped(self):
+ db = _session(rows=[])
+ await breakdown_for(db, TENANT, scope_kind="source", now=NOW)
+ assert "'unattributed'" in str(db.execute.await_args_list[0].args[0])
+
+ @pytest.mark.asyncio
+ async def test_an_unknown_dimension_is_refused(self):
+ with pytest.raises(ValueError, match="unknown scope kind"):
+ await breakdown_for(_session(), TENANT, scope_kind="tenant_id")
+
+ @pytest.mark.asyncio
+ async def test_each_row_carries_the_counts_behind_its_rates(self):
+ row = {"key": "okta", **_COUNTS}
+ rows = await breakdown_for(_session(rows=[row]), TENANT, scope_kind="source", now=NOW)
+
+ assert rows[0].key == "okta"
+ payload = rows[0].as_dict()
+ assert payload["answered"] == 90
+ assert payload["malicious_support"] == 31
+ # A rate without its denominator is a rate a reader cannot weigh.
+ assert payload["agreement_rate"] == pytest.approx(87 / 90)
+
+ @pytest.mark.asyncio
+ async def test_the_grouped_aggregate_uses_the_same_malicious_spelling(self):
+ db = _session(rows=[])
+ await breakdown_for(db, TENANT, scope_kind="alert_class", now=NOW)
+ assert db.execute.await_args_list[0].args[1]["malicious"] == MALICIOUS
+
+
+class TestTheVendoredRulesAreTheOnesInUse:
+ def test_this_service_computes_no_rate_of_its_own(self):
+ """Every rate the API serves comes off the shared window type.
+
+ Recomputing one here would give the console one definition of
+ agreement and the dispatch gate another, and the difference would
+ surface as a tenant asking why the scorecard says they qualify and
+ the gate says they do not.
+ """
+ source = inspect.getsource(shadow_agreement)
+ assert "def agreement_rate" not in source
+ assert "def malicious_recall" not in source
+
+ def test_the_evidence_payload_is_the_window_payload(self):
+ evidence = SimpleNamespace(window=AgreementWindow(**_COUNTS))
+ payload = evidence.window.as_dict()
+ assert set(payload) >= {"agreement_rate", "malicious_recall", "abstention_rate", "answered", "labelled"}