Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
feat(autonomy): poll a tenant's own SIEM for the closures that grade the agent (#912)
Beenu Aroraabdddf17678a
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex ec565aa40..8005d332e 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,6 +9,72 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Added+- **Shadow reconciliation now has a schedule, so a tenant whose analysts work+ in their own SIEM accumulates a track record** (gap-closure Phase 2.1,+ closing D15). `services/actions/app/services/shadow_reconcile.py` shipped+ complete with eleven tests and no caller. Half of shadow mode therefore+ worked and half of it only appeared to: agreement filled in for tenants+ closing alerts in the AiSOC console, and a tenant closing theirs in Splunk+ ES watched a scorecard that could never move. Phase 2's premise is that+ autonomy is earned on a measured track record, so a track record that cannot+ accumulate is the difference between the feature working and appearing to.++ **How it was measured.** By grepping for callers of code the programme had+ just written. The only files referencing the module were the module and its+ own test. That is the repository's most-repeated defect shape and the check+ its own history says to run before claiming anything.++ **The shape was forced, not chosen.** `services/api` owns the vault and the+ tenant session, `services/actions` owns the five readers and the matcher,+ and no process can hold both because all three services package their code+ as a top-level `app`. So the API resolves a connector's credentials and+ posts them with a window to a new internal route, `POST /shadow/reconcile`+ on actions, which reads and reconciles in one hop. That is the same round+ trip `siem_writeback`, `/connectors/{id}/normalize` and `/replay/history`+ already take, rather than a third pattern. The tenant travels on+ `X-AiSOC-Tenant-ID` and is resolved through the vendored `tenant_scope`,+ now in its tenth service, where a service token with no tenant header+ resolves to an empty scope that refuses rather than widening.++ **Four outcomes, because two is not enough.** A sweep that reports every+ fault the same way turns a revoked API key into churn nobody reads. The+ vendor's own status decides: 400, 401, 403 and 404 are permanent, name the+ operator action, and stop that connector being polled until the connector+ row is saved again, which is the one event that could have fixed it. A+ timeout or a 5xx is transient and retried. A 429 has the vendor's own+ `Retry-After` stored and honoured rather than guessed. And "nothing to do"+ is a recorded state rather than an absence, because a sweep that silently+ stopped and one with nothing to poll are otherwise indistinguishable from+ outside, which is precisely how the original gap stayed invisible.++ **Bounded work against somebody else's API.** Migration **068** adds+ `aisoc_shadow_reconcile_state`, keyed per connector because two SIEMs have+ two index lags and two credential lifetimes. Each carries a watermark that+ advances to the latest closure actually seen rather than to the end of the+ window asked for, so a vendor that indexes late does not have findings+ stepped over; an empty window still advances it, minus an overlap, so quiet+ hours are not re-read forever. A pass is capped per tick, a connector has a+ floor between polls, and one window is capped so a connector blocked for a+ month catches up in steps rather than asking for the month in one search.++ **Default off**, per the standing rule that a feature which calls out ships+ off. Two different people are involved: a tenant enabling shadow mode has+ asked to be measured, and the operator decides whether the platform may+ reach a third party on a timer. `GET /api/v1/health/shadow-reconciliation`+ reports the subscription in every state including `disabled`, `not_measuring`+ and `no_connector`, and reports closures read separately from closures+ matched, because a healthy read count with zero matches is a wiring fault+ and a read count of zero is a fact about the customer's week.++ **The gate.** `test_the_deployed_lifespan_registers_the_sweep` is an AST+ pass over `services/api/app/main.py` as it ships, in the spirit of the Phase+ 1.2 sink test: the defect being closed is "exists and nothing calls it", so+ a test asserting a scheduler a test built would close nothing. It checks the+ import, the task, that the task's `worker=` is the sweep's own `run_forever`,+ and the job name the Redis lease is keyed on. It was proven capable of+ failing by running it against `main`'s `app/main.py`. Claim matrix 167 rows+ to 170, GATED 159 to 162.+- **Replay evaluation reaches an operator: `aisoc replay`, an async API job,an "Evaluate on your history" console page, and JSON, Markdown and PDFexport** (gap-closure Phase 1.4 and 1.5). Phases 1.1 through 1.3 builtdiff --git a/GAP_CLOSURE_PROGRESS.md b/GAP_CLOSURE_PROGRESS.mdindex d54e5ec76..035bb97a8 100644--- a/GAP_CLOSURE_PROGRESS.md+++ b/GAP_CLOSURE_PROGRESS.md@@ -286,7 +286,7 @@ window on a second run, and a hosted model may legitimately differ.## Phase 2: Live shadow mode and evidence-gated autonomy-- [~] **2.1 Shadow mode**, per tenant and per alert class. Shipped in [#906](https://github.com/beenuar/AiSOC/pull/906), with one half of the closure path still open (see D15). The seam is Phase 1.2's: `FusedAlertTriageWorker` already routes every write through the `TriageWriter` port, so shadow mode is a wrapper around the tenant's live sink rather than a second code path. Chosen per alert, not per worker, because the class is not known until the alert is in hand; the constructor sinks are untouched, which is what keeps the Phase 1.2 AST test true. Migration **066** adds `aisoc_shadow_mode` and `aisoc_shadow_decisions`. Analyst closures arrive from a bounded sweep over alerts closed in this console. The reader and matcher for closures made in the customer's own SIEM ship too, on the five Phase 1.1 readers unchanged, and nothing calls them on a schedule yet. See D9 for the one property everything else rests on.+- [x] **2.1 Shadow mode**, per tenant and per alert class. Shipped in [#906](https://github.com/beenuar/AiSOC/pull/906), and its second closure path in [#912](https://github.com/beenuar/AiSOC/pull/912), which closes D15. The seam is Phase 1.2's: `FusedAlertTriageWorker` already routes every write through the `TriageWriter` port, so shadow mode is a wrapper around the tenant's live sink rather than a second code path. Chosen per alert, not per worker, because the class is not known until the alert is in hand; the constructor sinks are untouched, which is what keeps the Phase 1.2 AST test true. Migration **066** adds `aisoc_shadow_mode` and `aisoc_shadow_decisions`. Analyst closures now arrive from both routes the plan names: a bounded sweep over alerts closed in this console, and a scheduled sweep over each measuring tenant's own SIEM on the five Phase 1.1 readers unchanged. See D9 for the one property everything else rests on, and D15 for what the second path cost.- [x] **2.2 Rolling agreement** per alert class, rule, source and model, on the operations dashboard and the autonomy scorecard. Shipped in [#906](https://github.com/beenuar/AiSOC/pull/906). The metrics are Phase 1.3's and "uses the Phase 1 metrics" is now a gate rather than a sentence: `check_replay_contract_parity.py` went from three trees to four and compares `GRADED_DISPOSITIONS`, `ABSTENTION_VERDICTS`, `MALICIOUS` and `UNLABELED` in both directions, proven capable of failing by drifting each collection in turn. Agreement is computed over *answered* decisions only so abstaining cannot inflate it, malicious recall counts an abstention as a miss, a rate with no denominator reads "not measured", and every rate travels with its count. Matrix 156 rows to 159, GATED 148 to 151.- [x] **2.3 Promotion gate**, with automatic demotion on drift and every transition written to the hash-chained audit log. Shipped in [#908](https://github.com/beenuar/AiSOC/pull/908). Migration **067** adds `aisoc_autonomy_grants`. The gate is pure and lives in the vendored rules module so `services/actions` enforces the same arithmetic at dispatch that `services/api` decides on at promotion time. Wired into all three modules the plan names: `unified_autonomy.unified_decision` gained an `earned_grant` argument that can only widen the reversible MEDIUM-blast branch, `tenant_policy.TenantPolicy` carries the earned verbs and never issues one, and `autonomy_policy.py` gained `GET`/`POST`/`DELETE /grants`. The dispatcher was also wired, because `unified_decision` turned out to have no production caller at all (see D12). The "Done when" runs against a real Postgres in `integration.yml`. See D13 for why the ceiling stops at L3.@@ -435,6 +435,14 @@ Nothing yet beyond this kickoff. Each entry below will name its PR.`services/actions` 801 to 835, `apps/web` settings vitest 32 to 36, and`tests/isolation/test_autonomy_promotion_live.py` 13 passing against live`postgres:16`. Claim matrix 159 rows to 164, GATED 151 to 156.+- [x] **Phase 2.1's second closure path, closing D15.**+ [#912](https://github.com/beenuar/AiSOC/pull/912). Migration 068. Suites:+ `services/actions` 888 to 908 (20 new, plus the vendored `test_tenant_scope`+ that travels with the tenth copy of the module), `services/api` 2877 to 2904+ (27 new). Claim matrix 167 rows to 170, GATED 159 to 162. The registration+ gate was proven against the pre-fix tree: run against `main`'s+ `services/api/app/main.py`, both `test_the_deployed_lifespan_registers_the_sweep`+ and `test_the_sweep_is_off_by_default_and_says_so` fail.### D9. A shadow verdict must not reach `alerts.disposition`, and the guard belongs in the SQL@@ -520,20 +528,45 @@ analysts work entirely in Splunk would have waited for a scorecard that wasnever going to fill in, and concluded the agent was not being evaluated ratherthan that we were not looking.-**Resolution:** the claim is corrected rather than the code hurried. The page-now says in those words that this half is not automatic, names what exists and-what does not, and says agreement is measured on AiSOC closures until it-lands. 2.1 moves from `[x]` to `[~]`.--**What closes it,** in the shape the architecture already forces: `services/api`-owns the vault and the tenant session, so it resolves the connector's-credentials and posts them with a window to a new internal route on-`services/actions`, which calls the reader and then `reconcile_findings`. That-is the same round trip `siem_writeback` and `/connectors/{id}/normalize`-already take, and for the same reason: one process cannot hold two services-that both package their code as top-level `app`. It was not hurried into this-phase because a cross-service route added under time pressure is exactly the-half-wired feature the correction above is about.+**First resolution:** the claim was corrected rather than the code hurried. The+page said in those words that this half was not automatic, named what existed+and what did not, and said agreement was measured on AiSOC closures until it+landed. 2.1 moved from `[x]` to `[~]`.++**Closed in [#912](https://github.com/beenuar/AiSOC/pull/912)**, in exactly the+shape recorded above: `services/api` owns the vault and the tenant session, so+it resolves the connector's credentials and posts them with a window to+`POST /api/v1/shadow/reconcile` on `services/actions`, which calls the reader+and then `reconcile_findings`. Same round trip as `siem_writeback` and+`/connectors/{id}/normalize`, for the same reason D8 records. 2.1 is `[x]`.++Three things the build settled that the note above did not anticipate:++**The tenant had nowhere to come from.** `/replay/history` needs no tenant at+all: the credentials it is handed are the scope, and it writes nothing. This+route writes to one tenant's decision rows, so it needs one, and taking it+from a request field would be a value the caller chose that nothing checked.+`services/actions` was the one service of ten with no vendored+`app/security/tenant_scope.py`, so it got the tenth copy through+`sync_vendored_tenant_scope.py` rather than a bespoke guard, and the tenant+travels on `X-AiSOC-Tenant-ID` exactly as it does to `/replay/run`.++**The matcher had a silent no-op in it.** With no `DATABASE_URL`,+`reconcile_findings` logs at `debug` and returns a result reporting the+findings it considered and zero matched. Reported upward that is+indistinguishable from a window whose join key never arrives, and the two send+an operator to entirely different places. `database_configured()` is now+checked by the route *before* the vendor is called, so a missing environment+variable is a 503 naming it rather than a customer's search quota spent on a+window nothing could be written from.++**A watermark was not optional.** Without one the sweep re-reads the same hours+every tick forever. Migration **068** adds `aisoc_shadow_reconcile_state`, and+the column that turned out to matter most is `blocked_connector_updated_at`: a+revoked credential is permanent until somebody acts, and comparing that value+against `connectors.updated_at` makes "somebody saved the connector" the only+thing that resumes polling. A timer would have been the churn the state exists+to avoid, and a manual reset would have been a support ticket.### D12. `unified_decision` had no production caller, and the plan names it anywaydiff --git a/ROADMAP.md b/ROADMAP.mdindex 015a43bee..bc20a0a2d 100644--- a/ROADMAP.md+++ b/ROADMAP.md@@ -44,7 +44,7 @@ A proof-first, security-first program to make every README claim gate-backed, cl- **Phase D (breadth):** D1 eight connectors (QRadar/Exabeam/Securonix/Devo/Netskope/Windows-Sysmon/Zeek-Suricata/syslog-CEF) · D2 AI/LLM-usage audit connector + `llm-*` detections + hot/cold lake tiering · D3 live-vendor mock-server smoke.- **Phase E (prove it):** E1 CI-gated benchmark scoreboard tied to a deterministic live-agent MITRE-accuracy run.-The claim-to-gate matrix stands at **167 rows — 159 GATED / 8 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test** and the ratchet (`MAX_NO_GATE=0`) forbids any regression. The 8 remaining PARTIAL rows are honest, named deferrals; none was relabelled without building the gate it names. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).+The claim-to-gate matrix stands at **170 rows — 162 GATED / 8 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test** and the ratchet (`MAX_NO_GATE=0`) forbids any regression. The 8 remaining PARTIAL rows are honest, named deferrals; none was relabelled without building the gate it names. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).## v4.0 — Shippeddiff --git a/apps/docs/docs/compliance/evidence-pack.md b/apps/docs/docs/compliance/evidence-pack.mdindex 6e73b0623..a66b9f7a1 100644--- a/apps/docs/docs/compliance/evidence-pack.md+++ b/apps/docs/docs/compliance/evidence-pack.md@@ -33,7 +33,7 @@ and says so.| Prompt-injection resistance | 32-payload adversarial corpus with a recall floor and a false-positive ceiling | `services/agents/tests/adversarial/` || Claim-to-gate traceability | Every product claim mapped to the CI job that fails when it stops being true | `docs/audit/CLAIM_TO_GATE_MATRIX.md` |-The last row is the one worth reading first. It is the honest index: 159 rows+The last row is the one worth reading first. It is the honest index: 162 rows`GATED`, 8 `PARTIAL` with the specific gap named, and a ratchet that refusesto let a row sit at `NO GATE`.diff --git a/apps/docs/docs/operations/shadow-mode.md b/apps/docs/docs/operations/shadow-mode.mdindex 05731307b..140dd236d 100644--- a/apps/docs/docs/operations/shadow-mode.md+++ b/apps/docs/docs/operations/shadow-mode.md@@ -96,20 +96,18 @@ can be closed from the feedback endpoint, by a case being resolved, by a bulkaction or by a playbook, and a hook on one of those would silently grade thesubset of alerts closed the way the hook was written for.-**Closures made in your SIEM. Not automatic yet.** Most analysts evaluating-AiSOC keep working where they always have, so a closure recorded in Splunk ES,-Microsoft Sentinel, Elastic Security, IBM QRadar or Microsoft Defender XDR-should count. The readers exist and are the same ones that power-[replay evaluation](../evaluation/replay.md), and-`services/actions/app/services/shadow_reconcile.py` matches each closure to the-decision it grades by the vendor's own finding id.--**What does not exist is a scheduler that calls them.** Nothing polls your SIEM-for shadow reconciliation on a timer today, so a tenant whose analysts work-entirely in their own console will see a scorecard that does not fill in. That-is a gap in this feature, not a property of your deployment, and it is recorded-as such rather than left for you to discover from an empty page. Until it-lands, agreement is measured on closures made in AiSOC.+**Closures made in your SIEM.** Most analysts evaluating AiSOC keep working+where they always have, so a closure recorded in Splunk ES, Microsoft Sentinel,+Elastic Security, IBM QRadar or Microsoft Defender XDR counts too. A sweep+polls each of them on a timer using the same readers that power+[replay evaluation](../evaluation/replay.md), and matches each closure to the+decision it grades by the vendor's own finding id. See+[polling your SIEM](#polling-your-siem-for-closures) below for how to switch it+on and what it reports.++Until you switch that sweep on, agreement is measured on closures made in+AiSOC only. `GET /api/v1/health/shadow-reconciliation` says so in those words+rather than leaving you to infer it from a scorecard that does not fill in.Whichever route a closure arrives by, one carrying a disposition this platformcannot name becomes `unlabeled`. It is counted as resolved and excluded from every rate. Splunk ES@@ -117,6 +115,86 @@ ships two dispositions that literally mean "I do not know", and so do Sentineland Defender; folding those into a verdict because the finding happened to beclosed would manufacture agreement out of an analyst's uncertainty.+## Polling your SIEM for closures++The sweep is **off by default**. It reaches a third party's API on a timer, and+that is the deployment operator's decision rather than a tenant's, so it is+switched on deliberately:++```bash+SHADOW_RECONCILE_ENABLED=true+```++It then polls, once per interval, every enabled connector belonging to a tenant+with at least one alert class in shadow mode, for the five connector types that+have a closed-finding reader: `splunk`, `microsoft_sentinel`, `elastic`,+`qradar` and `azure_defender`. A connector of any other type is not an error+and is not polled.++### What it does to your SIEM++Everything here bounds the load this places on somebody else's search head.++| Setting | Default | What it bounds |+|---|---|---|+| `SHADOW_RECONCILE_INTERVAL_SECONDS` | 900 | How often a pass runs |+| `SHADOW_RECONCILE_MIN_CONNECTOR_INTERVAL_SECONDS` | 3600 | The floor between two polls of the same connector |+| `SHADOW_RECONCILE_MAX_CONNECTORS_PER_TICK` | 10 | Connectors polled in one pass, so a large estate spreads across passes |+| `SHADOW_RECONCILE_MAX_WINDOW_HOURS` | 24 | The largest window asked for in one search |+| `SHADOW_RECONCILE_MAX_LOOKBACK_HOURS` | 168 | How far a first pass reaches back |+| `SHADOW_RECONCILE_OVERLAP_SECONDS` | 900 | How much of the previous window is re-read |+| `SHADOW_RECONCILE_LIMIT` | 1000 | Findings read per window |++Each connector carries a **watermark**: the latest close time a successful pass+reached. The next window starts there minus the overlap, so the same hours are+never re-read forever, and a vendor whose search index lags its own close+events does not have those findings stepped over. Re-reading the overlap costs+nothing, because a decision that already carries a closure is never rewritten.++A vendor that returns `429` has its own `Retry-After` stored and honoured. The+sweep waits the interval the vendor asked for rather than guessing one.++### What stops it, and whether waiting helps++A sweep that reports every fault the same way turns a revoked API key into+churn nobody reads. So each connector is left in one of four states:++| State | Meaning | What to do |+|---|---|---|+| `ok` | A window was read and reconciled | Nothing |+| `idle` | Nothing to do: not due yet, or the window has not moved | Nothing |+| `blocked` | Permanent until you act: the vendor refused the stored credentials, or the vault cannot decrypt them | Re-save the connector's credentials |+| `transient` | The vendor timed out, rate-limited us, or the actions service was unreachable | Nothing; it retries |++A `blocked` connector is not polled again on a timer. It resumes on the single+event that could have fixed it, which is the connector being saved again. That+is deliberate: retrying a revoked credential every hour is load on your SIEM+that cannot succeed, and a log line nobody reads twice.++### Checking it is running++```bash+curl -s "$AISOC/api/v1/health/shadow-reconciliation" -H "Authorization: Bearer $TOKEN"+```++The response names the state in every case, including the quiet ones, because+a sweep that silently stopped and a sweep with nothing to do look identical+from outside and that ambiguity is what this endpoint exists to remove:++- `disabled`: the operator has not switched it on+- `not_measuring`: no alert class is in shadow mode for this tenant+- `no_connector`: measuring, but no connector of a type with a reader+- `blocked`: at least one connector needs you+- `degraded`: at least one connector is failing transiently+- `ok`: polling++Per connector it reports the watermark, the last run, how many closures were+read and how many matched. Those two are separate on purpose: a healthy read+count with zero matches means the vendor's finding id is not reaching+`aisoc_shadow_decisions.external_id`, which is a wiring fault, while a read+count of zero means your analysts closed nothing in that window, which is a+fact about their week. One number could not tell you which.+## How the numbers are computedThe definitions are the ones replay evaluation already uses. They are notdiff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.mdindex c6d299d87..48abfa6be 100644--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md@@ -163,7 +163,7 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`| A report field the provider does not publish reads "unavailable", never zero or empty | `operations/sandbox-providers.md` (provider contract) | `ci.yml :: api tests` (`test_sandbox_providers.py`, driven against a **recorded** report whose `attackTechniques` is `[]` beside `behavior.analyzed: false`; the adapter must return `UNAVAILABLE` with the stage's own reason attached, and the same payload with the stage marked run must return a real empty tuple. A missing score must not read as 0 and a missing IOC section must not read as "none found") | GATED | the recorded payload covers one provider and one shape of absence; CAPEv2's equivalents are synthetic because no live CAPEv2 instance was available || Replay evaluation runs the production triage path, not a copy of it, and writes nothing while it does | `apps/docs/docs/evaluation/replay.md`; `services/agents/app/replay/runner.py` | `ci.yml` agents job (`test_replay_shadow_no_writes.py` — every durable write on the triage path is monkeypatched on the **real** module to raise a `BaseException`, then a replay is run: a `BaseException` because the worker wraps most of its writes in `contextlib.suppress(Exception)`, which is correct production behaviour and is exactly what would let a leaked write pass a gentler guard. Zero writes is asserted as a *count* rather than as an absence, since a replay that stopped replaying and a replay whose writes were suppressed both write nothing and only the counter tells them apart. That the path is production's is pinned two ways: the worker's default sinks are asserted to be `LiveTriageWriter`/`LiveTriageContextReader`, and an AST pass over `services/agents/app/main.py` asserts the deployed worker is constructed without overriding either. `test_escalation_cannot_change_the_graded_verdict` drives the real worker with a graph runner that rewrites the verdict and asserts the returned summary is unmoved, which is what makes declining escalation in shadow mode sound rather than convenient) | GATED | Triage only. Fusion's correlation, fused confidence score, narrative and entity resolution are absent from a replayed finding; `ENVELOPE_LIMITS` is published in every report's method section rather than left as a comment || The test window of a replay cannot influence its own triage | `apps/docs/docs/evaluation/replay.md` (point-in-time context); `services/agents/app/replay/shadow.py` | `ci.yml` agents job (`test_replay_leakage.py` — three routes from a test-window decision back into a later one, each with its own test because one combined assertion would not say which broke: the outcome-memory prior that auto-closes a repeat without re-triage, the cost governor's dedup cache that short-circuits earlier still and leaves no ledger row or prior behind, and organisation memory read live into the prompt. Each is checked for **sensitivity** as well as for the guard: the memory test runs the identical corpus twice, once with the shadow sinks and once with a live reader and a recording writer, and asserts the second leaks at tier `memory`; the statements test asserts the frozen reader hands the agent the snapshot and that the same worker with the live reader picks up the poison. `capture_context` drops rows recorded after the split in both stores, and a snapshot taken at any other instant is refused rather than used) | GATED | Organisation-memory statements as `GET /feedback/context-statements` serves them carry no creation time, so the split filter has nothing to test them against and they are kept. The count is published as `statements_without_timestamp` in the report's method section rather than the freeze being claimed tighter than it is |-| A shadow verdict never fills in the analyst column it will be graded against | `apps/docs/docs/operations/shadow-mode.md`; `services/agents/app/workers/shadow_mode.py` | `ci.yml` agents job (`test_shadow_mode.py`) - the shadow sink is asserted to decline every acting method against a delegate that *records* rather than performs, so a forwarded call shows up as a recorded one rather than as an absence; `persist_auto_triage` is asserted to force `auto_closed` off and the `shadow` flag on; and the ledger statement is read to confirm `disposition`, `status` and `resolved_at` are each guarded by that flag with `NOT $10` rather than by the caller remembering to pass `auto_closed=False`. Plus `ci.yml` api job (`test_shadow_agreement.py`) - reconciliation only reads closed alerts and only writes decisions with `resolved_at IS NULL`, so a later change of mind cannot rewrite evidence a promotion already rested on | GATED | Two limits. The SQL guard is read, not executed: there is no Postgres in the unit suites, so what is proven is that the statement carries the guard and binds the flag, not that Postgres evaluates the CASE as written. And this covers closures made in AiSOC only: the reader and matcher for closures made in the customer's own SIEM are tested (`test_shadow_reconcile.py`) but nothing calls them on a schedule, so a tenant whose analysts work entirely in their SIEM has no evidence flowing. Recorded as D15 in the progress tracker and stated on the docs page rather than left to be discovered from an empty scorecard |+| A shadow verdict never fills in the analyst column it will be graded against | `apps/docs/docs/operations/shadow-mode.md`; `services/agents/app/workers/shadow_mode.py` | `ci.yml` agents job (`test_shadow_mode.py`) - the shadow sink is asserted to decline every acting method against a delegate that *records* rather than performs, so a forwarded call shows up as a recorded one rather than as an absence; `persist_auto_triage` is asserted to force `auto_closed` off and the `shadow` flag on; and the ledger statement is read to confirm `disposition`, `status` and `resolved_at` are each guarded by that flag with `NOT $10` rather than by the caller remembering to pass `auto_closed=False`. Plus `ci.yml` api job (`test_shadow_agreement.py`) - reconciliation only reads closed alerts and only writes decisions with `resolved_at IS NULL`, so a later change of mind cannot rewrite evidence a promotion already rested on | GATED | One limit. The SQL guard is read, not executed: there is no Postgres in the unit suites, so what is proven is that the statement carries the guard and binds the flag, not that Postgres evaluates the CASE as written. The caveat this row used to carry, that closures made in the customer's own SIEM reached no scheduler, is closed by the three rows below; D15 in the progress tracker records what it was || Agreement cannot be improved by declining to answer | `apps/docs/docs/operations/shadow-mode.md`; `services/actions/app/services/autonomy_evidence_rules.py` | `ci.yml` actions job (`test_autonomy_evidence_rules.py`) - the same ten right answers are scored with nothing declined and with ninety declined, and the agreement rate is asserted identical, because abstaining removes a decision from the numerator and the denominator together. The counterweight is asserted from the other side: a window where the agent answered ten of forty labelled cases posts a perfect agreement rate and a malicious recall of 8/30, since recall's denominator is every malicious case whether answered or not. The abstention rate is asserted to move where agreement does not, and `AgreementWindow` refuses to construct at all when its counts do not add up, so a miscounted aggregate raises instead of publishing a plausible rate | GATED | The metric is gated; what it is used *for* is Phase 2.3. This row proves the rate cannot be gamed, not that anything yet refuses a promotion on it || Live agreement is scored by the Phase 1 metrics rather than a second set | `plans/aisoc_gap_closure_plan.plan.md` (2.2, "using the Phase 1 metrics"); `apps/docs/docs/operations/shadow-mode.md` | `ci.yml :: python-lint` (`check_replay_contract_parity.py`, extended from three trees to four: `GRADED_DISPOSITIONS`, `ABSTENTION_VERDICTS` and the `MALICIOUS` spelling must be identical strings in `autonomy_evidence_rules.py` and in `aisoc_benchmark/replay.py`, compared in both directions, and `UNLABELED` must be spelled the same in all three. Proven capable of failing by drifting each of the two collections in turn and watching it fire, rather than only having never fired) + `ci.yml` (`sync_vendored_autonomy_evidence.py --check`, which byte-compares the API's copy, because a safety control defined twice is off in whichever copy is more generous) | GATED | Parity of the definitions is gated. That the two implementations compute the same number from the same rows is not: the grader scores a list of decision dicts and the live path aggregates in SQL, so they share a vocabulary rather than a code path || Autonomy is granted on a measured track record, and refused without one | `apps/docs/docs/operations/shadow-mode.md` (earning autonomy); `services/actions/app/services/autonomy_evidence_rules.py` | `ci.yml` actions job (`test_autonomy_promotion_gate.py`, which starts from one qualifying control and breaks one property per test, so a failure names the property rather than the file) + `integration.yml` (`tests/isolation/test_autonomy_promotion_live.py`, the phase's "Done when" run against a real Postgres: a tenant with 20 recorded shadow decisions is refused, the same tenant with 150 is granted, and injected disagreements demote it. The live half exists because the aggregate and the evaluator agree by convention and a convention is what drifts: only a real database executing the real statement over real rows proves the SQL counts what the gate expects. `AUTONOMY_PROMOTION_LIVE_REQUIRED=1` turns an unreachable database into a failure rather than a skip) | GATED | The four refusals are gated. What is not gated is whether the thresholds are the *right* numbers: 100 decisions and 30 malicious are the plan's, and 95% agreement and 90% recall are choices nobody has yet validated against an outcome. They are configurable per call and travel by value in every snapshot |@@ -174,11 +174,14 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`| A replay report prints "not measured" where there was no denominator, and no headline accuracy on a corpus too thin to support one | `apps/docs/docs/evaluation/replay.md`; `packages/aisoc-benchmark/aisoc_benchmark/replay.py` | `ci.yml` benchmark job (`test_replay_metrics.py` — a class the agent never predicted reports `None` precision and renders as "not measured", because `0.0` in a recall column says the agent missed every case and "never asked" is a different fact; below 30 malicious cases the headline is withheld with the count and the reason, and the rendered report is asserted to contain no percentage in that section. Three populations are asserted apart: unlabelled findings excluded from accuracy and counted, triage refusals scored as neither answer nor abstention, and abstentions counted against malicious recall. A verdict outside the taxonomy gets its own confusion-matrix column rather than being folded into one the agent did not say. Bootstrap intervals are seeded and the seed and resample count travel in the report, so two runs over one corpus render byte-identically) | GATED | Hallucination extraction reuses the existing `_INDICATOR_PATTERNS`, which over-counts rather than under-counts: a phrase shaped like a domain is checked and, if absent from the evidence, counted. The published rate is a ceiling and the per-decision list travels with it so a disputed count can be re-derived || An operator can measure triage against their own analysts from the CLI, the API or the console, and the report reproduces byte for byte | `apps/docs/docs/evaluation/replay.md` (Running one; Reproducibility, stated precisely); `packages/aisoc-cli` `aisoc replay` | `integration.yml :: replay-e2e` (`tests/e2e/test_replay_cli_end_to_end.py` — the CLI is run **twice** against a mocked Splunk ES holding 200 recorded closed notables, through the API, `services/actions`, `services/connectors` and `services/agents`, each started from the working tree rather than from a published image, and the two reports are compared as bytes. The comparison cannot pass vacuously: a second assertion fetches both reports *without* the latency exclusion and fails if more than that one line differs, so a stripped artefact cannot hide a field that stopped reproducing; further assertions require the report to carry the 200 findings read, the 60-finding graded window and a printed headline, because a withheld headline reproduces just as reliably while printing far fewer numbers. `AISOC_REPLAY_E2E_REQUIRED=1` makes an unreachable database a failure rather than a skip) + `ci.yml` cli job (`test_replay_command.py` — the CLI renders nothing of its own and writes the bytes the API returned, progress goes to stderr so `aisoc replay ... > report.md` is the report and nothing else, an omitted window is not filled in from the local clock, and a failed evaluation exits non-zero with the server's reason and writes no file) + `ci.yml` benchmark job (`test_replay_metrics.py` — two corpora differing only in latency render differently and identically once excluded; the stripper removes exactly the line the renderer emits, in both directions) | GATED | Reproducibility is a property of the deterministic model path and of a **pinned** window. An omitted `--since` / `--until` dates the window from now, which is a different window on a second run; pointed at a hosted model two runs may legitimately differ. No hosted provider has been exercised end to end here || A replay evaluation's results are tenant-scoped, and its tenant comes from the credential rather than the request | `apps/docs/docs/evaluation/replay.md` (The API); `services/api/migrations/065_replay_evaluations.sql` | `isolation.yml :: route-scope` (`check_route_auth.py` — the six new `/evaluations/replay*` routes carry an auth dependency or the gate fails; `check_route_tenant_scope.py` — no route, query parameter or request-body field on this surface names a tenant, and the internal routes it drives resolve theirs from the credential: `services/agents` through the vendored `tenant_scope` where a service token with no `X-AiSOC-Tenant-ID` resolves to an **empty** scope that refuses rather than widens, and `services/actions` by taking no tenant at all, since the credentials it is handed are the scope) + `isolation.yml` (`check_rls_policy_shape.py` — both new tables carry `FORCE` and the `OR current_tenant_id() IS NULL` arm, verified applied against a live `postgres:16`) + `ci.yml` api job (`test_replay_evaluation.py` — a connector in another tenant resolves to no row and the run fails rather than reading it; an evaluation id in another tenant is a 404 rather than a 403, since telling the two apart would confirm the id exists) + `ci.yml` agents job (`test_replay_route.py` — a service token with no tenant header is refused with no downstream call made, and the tenant the credential named is the one forwarded to the connectors service) | GATED | The per-decision rows carry the evidence triage was given, so a tenant's own alert content is at rest in `aisoc_replay_decisions`. That is the point (a disputed score has to be re-derivable) and it is stated on the docs page rather than left to be discovered |+| Closures your analysts make in your own SIEM are polled back on a schedule and graded, not merely capable of being | `apps/docs/docs/operations/shadow-mode.md` (Polling your SIEM for closures); `services/api/app/workers/shadow_reconcile.py` | `ci.yml` api job (`test_shadow_reconcile_sweep.py::test_the_deployed_lifespan_registers_the_sweep` - an AST pass over `services/api/app/main.py` as it ships, asserting three things because any one alone passes over a worker that never runs: the module is imported, a task is created through `_run_guarded_scheduler_worker`, and that call's `worker=` is the sweep's own `run_forever` rather than something that merely mentions it. The job name is pinned too, because the Redis lease is keyed on it and two names means two replicas sweeping one vendor at once. Proven capable of failing by running it against `main`'s `app/main.py`, where both it and the default-off assertion fail) + `ci.yml` actions job (`test_shadow_reconcile_route.py::test_the_route_is_reachable_on_the_deployed_app` - the receiving route is asserted present on the service's own `app`, since a router written and never included is the same defect one layer up. Reachability is asserted by *sending a request* and by reading `app.openapi()`, never by reading `app.routes`: on the FastAPI version these services pin, `include_router` does not populate `app.routes` at all, so an app with four routers mounted reports only the seven docs and probe routes it was born with, and the first draft of this test passed locally and failed in CI for exactly that reason) | GATED | Registration is gated, and so is every branch of one pass against a stubbed actions service. What is not gated end to end is a real vendor: no live Splunk, Sentinel, Elastic, QRadar or Defender has been polled by this sweep. The readers themselves are covered against recorded vendor payloads by the Phase 1.1 rows |+| A scheduled sweep tells a condition that will never resolve from one that might, and says which | `apps/docs/docs/operations/shadow-mode.md` (What stops it, and whether waiting helps); `services/actions/app/api/shadow_reconcile_router.py` | `ci.yml` actions job (`test_shadow_reconcile_route.py::TestPermanentIsNotTransient` - the vendor's own status decides, so all four of 400/401/403/404 answer 422 naming the operator action while 500/502/503/504 and a timeout answer 502 without it, and a 429 forwards the vendor's own `Retry-After` rather than a guess; a reader that could not build a client is refused before the matcher is reached, because an empty window reads as "these analysts closed nothing") + `ci.yml` api job (`test_shadow_reconcile_sweep.py::TestPermanentIsNotTransient` - a permanent refusal blocks the connector and stops polling it, a transient one never blocks and leaves the watermark untouched, an undecryptable credential blocks before the vendor is called at all, and a deployment-level refusal halts the whole pass rather than writing "needs an operator" onto every connector. Unblocking is asserted to require the connector row changing and nothing weaker, so a revoked credential resumes on the one event that could have fixed it) | GATED | The classification is drawn from the vendor's HTTP status. A vendor that answers 200 with an error body, or 500 for a permanently bad query, is classified by what it said rather than by what it meant |+| The sweep is bounded work against a customer's API: it keeps a watermark, honours rate limits and never re-reads a window forever | `apps/docs/docs/operations/shadow-mode.md` (What it does to your SIEM); `services/api/migrations/068_shadow_reconcile_state.sql` | `ci.yml` api job (`test_shadow_reconcile_sweep.py::TestRateLimitingAndTheWatermark` - the watermark advances to the latest closure actually seen rather than to the end of the window asked for, so a vendor that indexes late does not have findings stepped over; an **empty** window still advances it, minus the overlap, so quiet hours are not re-read forever; a window is capped so a connector blocked for a month catches up in steps rather than asking for the month in one search; a first pass starts when measurement started rather than at the beginning of the vendor's retention; the per-connector floor and the per-tick cap are both asserted to withhold a poll; and a `Retry-After` is stored and honoured. `test_a_transient_failure_leaves_the_watermark_alone` asserts a failure cannot look like progress) | GATED | Every limit is asserted against the sweep's own decision to poll. Whether the defaults are the *right* numbers for a given SIEM is a judgement: they are configurable and the docs table publishes each one beside what it bounds || The API scores a replay with the grader the benchmark package ships, not a second copy of it | `services/api/app/_vendor/aisoc_benchmark/VENDORED.md` | `ci.yml :: vendored-sync` (`scripts/sync_vendored_benchmark.py --check` — every module of `packages/aisoc-benchmark/aisoc_benchmark/` must be byte-identical in `services/api/app/_vendor/aisoc_benchmark/`, checked in **both** directions so a module present only in the mirror fails too, and refusing to report a clean mirror when the source directory yields no modules) | GATED | A mirror rather than an import, because the `aisoc-core-api` image is built with `./services/api` as its context and nothing under `packages/` exists at runtime. Same constraint that produced `_vendor/narrative.py` and `_vendor/nl_query/`. A second grader would eventually disagree with the first about what "hallucination rate" means while both kept calling themselves the same number |## Summary-- GATED: 159+- GATED: 162- PARTIAL: 8- NO GATE: 0 (**every claim is backed by a failing test.** The last NO GATE — the weekly benchmark scoreboard running live against `main` — closed in Phase E1: `scripts/check_scoreboard.py` ties the published scoreboard to a deterministic per-PR live-agent MITRE-accuracy run, while the funded weekly `wet-eval.yml` appends the LLM-tier rows. The remaining PARTIAL rows are honest, named deferrals — each states the specific gap and what closes it — not unproven claims.)diff --git a/docs/openapi.yaml b/docs/openapi.yamlindex 82a97198f..bf5233896 100644--- a/docs/openapi.yaml+++ b/docs/openapi.yaml@@ -6755,6 +6755,43 @@ paths:application/json:schema:$ref: '#/components/schemas/HTTPValidationError'+ /api/v1/health/shadow-reconciliation:+ get:+ tags:+ - health+ summary: Get Shadow Reconciliation Health+ description: "Whether closures made in this tenant's own SIEM are being polled\+ \ back.\n\nGap-closure Phase 2.1 (D15).\n\nA sweep that silently stopped and\+ \ a sweep with nothing to do look identical\nfrom outside, and that ambiguity\+ \ is what made the original gap invisible:\nthe reconciler existed, nothing\+ \ called it, and the only symptom available\nto an operator was a scorecard\+ \ that never filled in. So this reports the\nsubscription in every state,\+ \ including the healthy one and the idle one,\nand names which it is.\n\n\+ ``state`` is one of:\n\n``disabled`` the operator has not switched the\+ \ sweep on. Not a fault,\n and not a healthy idle sweep either.\n\+ ``not_measuring`` the sweep runs, but this tenant has no alert class in\n\+ \ shadow mode, so there is nothing to reconcile.\n``no_connector``\+ \ this tenant is measuring and has no enabled connector of\n \+ \ a type with a closed-finding reader. Agreement is being\n \+ \ measured on AiSOC closures only, which is the honest answer\n \+ \ and is stated rather than left to be inferred.\n``blocked`` \+ \ at least one connector needs an operator. The reason is a\n \+ \ sentence, and it will not clear by waiting.\n``degraded`` at least\+ \ one connector is failing transiently and is being\n retried.\n\+ ``ok`` every configured connector polled."+ operationId: get_shadow_reconciliation_health_api_v1_health_shadow_reconciliation_get+ responses:+ '200':+ description: Successful Response+ content:+ application/json:+ schema:+ additionalProperties: true+ type: object+ title: Response Get Shadow Reconciliation Health Api V1 Health Shadow+ Reconciliation Get+ security:+ - HTTPBearer: []/api/v1/insights/soc:get:tags:diff --git a/scripts/sync_vendored_tenant_scope.py b/scripts/sync_vendored_tenant_scope.pyindex 62d953e7d..2092f6d71 100644--- a/scripts/sync_vendored_tenant_scope.py+++ b/scripts/sync_vendored_tenant_scope.py@@ -62,6 +62,11 @@ TARGETS: dict[str, str] = {# bearer-only guard the browser could not satisfy."connectors": "CONNECTORS","threatintel": "THREATINTEL",+ # The shadow-reconciliation sweep in services/api posts a window and a+ # tenant's decrypted SIEM credentials here. The read is scoped by the+ # credentials, but the write lands on that tenant's shadow decisions, so+ # this service needs a tenant it did not take from a request field.+ "actions": "ACTIONS",}_SERVICE_NAME_RE = re.compile(r'^SERVICE_NAME = ".*"$', re.MULTILINE)diff --git a/services/actions/app/api/replay_history_router.py b/services/actions/app/api/replay_history_router.pyindex a62d8f877..c507f163b 100644--- a/services/actions/app/api/replay_history_router.py+++ b/services/actions/app/api/replay_history_router.py@@ -194,7 +194,10 @@ async def _read_defender(body: HistoryRequest) -> list[ClosedFinding]:return [parse_defender_alert(row) for row in rows]-_READERS = {+#: Public because the shadow-reconciliation route reads the same five windows+#: with the same five readers. A second dispatch table would be a second place+#: for a vendor arm to be added to one and forgotten in the other.+READERS = {"splunk": _read_splunk,"sentinel": _read_sentinel,"elastic": _read_elastic,@@ -221,7 +224,7 @@ async def read_history()try:- findings = await _READERS[body.vendor](body)+ findings = await READERS[body.vendor](body)except HTTPException:raiseexcept Exception as exc: # noqa: BLE001 - every vendor failure is a 502, never an empty windowdiff --git a/services/actions/app/api/shadow_reconcile_router.py b/services/actions/app/api/shadow_reconcile_router.pynew file mode 100644index 000000000..9d594023d--- /dev/null+++ b/services/actions/app/api/shadow_reconcile_router.py@@ -0,0 +1,229 @@+"""Poll a customer's SIEM for analyst closures and grade the shadow decisions.++Gap-closure Phase 2.1, closing D15.++``app.services.shadow_reconcile`` matched a vendor closure to the shadow+decision it grades, had eleven tests, and had no caller. The AiSOC-side half+of reconciliation ran on every read of the agreement endpoint; this half was a+library. A tenant whose analysts close their queue in Splunk ES rather than in+this console therefore watched a scorecard that could never fill in, and the+natural reading of an empty scorecard is that the agent is not being evaluated+rather than that nobody is looking.++Why the route is here+=====================++The same reason ``POST /replay/history`` is. ``services/actions`` owns the five+closed-finding readers, the retry behaviour, the pagination loops and the+vendor quirks Phase 1.1 recorded one at a time, and it owns+``reconcile_findings``. ``services/api`` owns the vault and the tenant session.+No process can hold both, because all three services package their code as+top-level ``app`` (D8), so the API decrypts the connector's credentials and+posts them here with a window, and this route reads and then reconciles in one+hop rather than shipping a finding list back across the network to be written+by a second copy of the matcher.++The tenant comes from the credential+====================================++There is no ``tenant_id`` field on the request. ``POST /replay/history`` needs+no tenant at all, because the credentials it is handed *are* the scope: they+reach exactly one customer's SIEM and nothing is written. This route writes,+and it writes to one tenant's shadow decisions, so it resolves a tenant the+way every other tenant-scoped service in this tree does: the vendored+``app.security.tenant_scope``, where a console token carries a verified claim+and a service token must declare the tenant it acts for on+``X-AiSOC-Tenant-ID``. A service token with no tenant header resolves to an+empty scope and refuses rather than widening to every tenant.++Three answers, not two+======================++A sweep that cannot tell a condition that will never resolve from one that+might turns a misconfiguration into churn, and this repository has shipped that+defect more than once. So a vendor failure is classified before it is returned:++``422`` the credentials in this request cannot build a client, or the vendor+ rejected them. Permanent until an operator re-authorises the+ connector, and the body names that action.+``502`` the vendor was reachable and unhappy, or was not reachable at all.+ Transient; the caller retries on its own cadence.+``429`` the vendor asked us to slow down, with its ``Retry-After`` forwarded+ so the caller can honour it rather than guess.++The distinction is drawn from the vendor's own status code, which every client+here surfaces through ``raise_for_status``. Guessing from an exception type+would put a revoked API key and a five-second timeout in the same bucket, and+only one of them is worth waiting on.+"""++from __future__ import annotations++from typing import Annotated++import httpx+import structlog+from fastapi import APIRouter, Depends, HTTPException, Response, status+from pydantic import BaseModel++from app.api.replay_history_router import READERS, HistoryRequest+from app.security.tenant_scope import (+ TenantPrincipal,+ require_console_or_service_auth,+ scoped_tenant_or_403,+)+from app.services.shadow_reconcile import database_configured, reconcile_findings++logger = structlog.get_logger(__name__)++router = APIRouter(prefix="/shadow", tags=["shadow"])++ScopedPrincipal = Annotated[TenantPrincipal, Depends(require_console_or_service_auth)]++#: Vendor statuses that mean the credential itself is the problem. Retrying+#: these on a timer is what turns one revoked API key into a poll loop nobody+#: reads, so they are reported as permanent and name the operator action. 404+#: is included deliberately for the same reason: on these five APIs it means+#: the workspace, index or console path in the saved connector does not exist,+#: which no amount of waiting fixes.+PERMANENT_VENDOR_STATUSES = frozenset({400, 401, 403, 404})+++class ReconcileRequest(HistoryRequest):+ """One vendor, one window, credentials the API already decrypted.++ Deliberately the *same* envelope as :class:`HistoryRequest` rather than a+ parallel one: the same five readers are driven with the same credential+ keys, the same vendor literal and the same two vendor-specific knobs, and a+ second shape would be a second place for one of them to drift. Subclassed+ rather than reused by name only so the OpenAPI schema names the operation a+ reader is looking at.+ """+++class ReconcileResponse(BaseModel):+ """What the pass read, and what it did with it.++ ``count`` and ``matched`` are reported separately and both are needed. A+ window with a healthy ``count`` and ``matched`` at zero means the join key+ is not arriving, which is a wiring fault; a window with ``count`` at zero+ means the analysts closed nothing, which is a fact about their week. One+ number cannot say which, and the caller decides what to record on the+ difference.+ """++ vendor: str+ count: int+ labelled: int+ unlabeled: int+ considered: int+ matched: int+ unmatched: int+ already_resolved: int+ skipped_no_key: int+ #: The latest close time in this window, ISO-8601, or ``None`` when the+ #: window held nothing. The caller advances its watermark from this rather+ #: than from its own clock, so a vendor whose indexing lags does not have+ #: findings stepped over.+ latest_closed_at: str | None = None+++def _permanent(vendor: str, detail: str) -> HTTPException:+ return HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, detail=f"{vendor}: {detail}")++[email protected]("/reconcile", response_model=ReconcileResponse, summary="Grade shadow decisions against SIEM closures")+async def reconcile(+ body: ReconcileRequest,+ principal: ScopedPrincipal,+ response: Response,+) -> ReconcileResponse:+ """Read one window of closures and attach each to the decision it grades.++ Idempotent by construction: ``reconcile_findings`` only writes decisions+ whose ``resolved_at`` is still null, so a window re-read after a transient+ failure grades nothing twice and reports the overlap as ``already_resolved``+ rather than as fresh evidence.+ """+ tenant_id = scoped_tenant_or_403(principal)++ if body.until <= body.since:+ raise HTTPException(+ status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,+ detail=f"until ({body.until.isoformat()}) must be after since ({body.since.isoformat()})",+ )++ # Checked before the vendor is called rather than after. Without a+ # database this route would spend a customer's API quota and then report a+ # window it read and graded nothing from, which reads as a broken join key+ # rather than as a missing environment variable.+ if not database_configured():+ raise HTTPException(+ status_code=status.HTTP_503_SERVICE_UNAVAILABLE,+ detail=(+ "this service has no DATABASE_URL, so a closure read from the vendor could not be "+ "recorded against the decision it grades. Set DATABASE_URL on the actions container"+ ),+ )++ try:+ findings = await READERS[body.vendor](body)+ except HTTPException as exc:+ # The reader refused to build a client at all. That is this request's+ # credentials rather than the vendor, and it is permanent until they+ # change.+ raise _permanent(body.vendor, str(exc.detail)) from exc+ except httpx.HTTPStatusError as exc:+ raise _from_vendor_status(body.vendor, exc) from exc+ except Exception as exc: # noqa: BLE001 - a read failure is never an empty window+ logger.exception("shadow_reconcile.read_failed", vendor=body.vendor)+ raise HTTPException(+ status_code=status.HTTP_502_BAD_GATEWAY,+ detail=f"reading {body.vendor} closures failed: {type(exc).__name__}",+ ) from exc++ result = await reconcile_findings(tenant_id, findings)++ labelled = sum(1 for f in findings if f.is_labelled)+ closed_times = [f.closed_at for f in findings if f.closed_at is not None]+ logger.info(+ "shadow_reconcile.window_complete",+ tenant_id=str(tenant_id),+ vendor=body.vendor,+ count=len(findings),+ labelled=labelled,+ **result.as_dict(),+ )+ # Echoed so the caller's log line can be correlated with this one without+ # reading both sides' clocks.+ response.headers["X-AiSOC-Reconciled"] = str(result.matched)+ return ReconcileResponse(+ vendor=body.vendor,+ count=len(findings),+ labelled=labelled,+ unlabeled=len(findings) - labelled,+ latest_closed_at=max(closed_times).isoformat() if closed_times else None,+ **result.as_dict(),+ )+++def _from_vendor_status(vendor: str, exc: httpx.HTTPStatusError) -> HTTPException:+ """Turn the vendor's own status into permanent, rate-limited or transient."""+ code = exc.response.status_code+ if code == status.HTTP_429_TOO_MANY_REQUESTS:+ retry_after = exc.response.headers.get("Retry-After")+ logger.warning("shadow_reconcile.rate_limited", vendor=vendor, retry_after=retry_after)+ return HTTPException(+ status_code=status.HTTP_429_TOO_MANY_REQUESTS,+ detail=f"{vendor} asked us to slow down",+ headers={"Retry-After": retry_after} if retry_after else None,+ )+ if code in PERMANENT_VENDOR_STATUSES:+ logger.warning("shadow_reconcile.credentials_refused", vendor=vendor, vendor_status=code)+ return _permanent(+ vendor,+ f"the vendor refused the stored credentials with HTTP {code}. "+ f"Re-authorise this connector in Settings; polling will not recover on its own",+ )+ logger.warning("shadow_reconcile.vendor_error", vendor=vendor, vendor_status=code)+ return HTTPException(status_code=status.HTTP_502_BAD_GATEWAY, detail=f"{vendor} returned HTTP {code}")diff --git a/services/actions/app/main.py b/services/actions/app/main.pyindex 28a29c8a7..b6aa0ad20 100644--- a/services/actions/app/main.py+++ b/services/actions/app/main.py@@ -25,6 +25,7 @@ from app._health import install_health_routesfrom app.api.live_actions_router import router as live_actions_routerfrom app.api.replay_history_router import router as replay_history_routerfrom app.api.router import router as legacy_router+from app.api.shadow_reconcile_router import router as shadow_reconcile_routerfrom app.live_actions import register_builtin_executorslogger = structlog.get_logger(__name__)@@ -51,6 +52,10 @@ app.include_router(live_actions_router, prefix="/api/v1")# Gap-closure Phase 1.4: the read half of the SIEM relationship. Phase 1.1 put# five closed-finding readers on the clients here and nothing called them.app.include_router(replay_history_router, prefix="/api/v1")+# Gap-closure Phase 2.1 (D15): the same five readers, driven on a schedule by+# the sweep in services/api, with the matcher that was previously a library+# nothing called.+app.include_router(shadow_reconcile_router, prefix="/api/v1")@app.on_event("startup")diff --git a/services/actions/app/security/tenant_scope.py b/services/actions/app/security/tenant_scope.pynew file mode 100644index 000000000..edd831e65--- /dev/null+++ b/services/actions/app/security/tenant_scope.py@@ -0,0 +1,384 @@+"""Resolve which tenant a request may read, from the caller's credential.++Vendored byte-identical into each Python service that needs it, following the+same convention as ``app/core/cors.py`` and ``app/security/service_auth.py``.+Keep the copies in sync — ``scripts/sync_vendored_tenant_scope.py --check``+fails the build if they drift.++Why this exists+---------------+``/fusion/entity-risk/*`` took the tenant as a query parameter with no auth+dependency on the route, on both the API gateway and the fusion service. The+console reaches fusion *directly* through a Next rewrite when ``FUSION_URL`` is+set, so that parameter was reachable from any browser on the internet, and+naming somebody else's tenant UUID returned their entity-risk queue.++Validating the parameter's *value* does not fix that. A UUID that parses is+still a UUID the caller chose. The scope has to come from somewhere the caller+does not control, which means the credential.++Two credential shapes reach these services, so this module resolves both:++``console`` — the browser's first-party access token, HS256-signed by+ ``services/api`` (``create_access_token``) and carrying a verified+ ``tenant_id`` claim. That claim is authoritative.+``service`` — the shared service bearer token used for service-to-service+ calls. It identifies *a trusted service*, not a tenant, so a caller+ presenting it must also assert which tenant it is acting for, on the+ ``X-AiSOC-Tenant-ID`` header. The assertion is explicit and required: a+ service token with no tenant header resolves to an **empty** scope, and an+ empty scope refuses rather than widening.++That last sentence is the whole design. Every cross-tenant leak this codebase+has had took the same shape — a scope that was absent rather than narrow, and+a read that treated absent as "no filter".++We verify HS256 by hand with the standard library rather than adding a JWT+dependency to five services, exactly as ``services/realtime/src/auth.ts`` does+for the same reason on the Node side. ``services/api`` remains the sole issuer;+this is a read path, like the vendored ``decrypt_dict()`` that lets the+connector scheduler decrypt without owning the write path.++Configuration+-------------+``SECRET_KEY``+ The HS256 secret ``services/api`` signs console tokens with. Without it+ this service cannot verify a console session and will only accept service+ tokens.+``AISOC_SERVICE_TOKEN`` / ``AISOC_<SERVICE>_SERVICE_TOKEN``+ Shared secret for service-to-service calls, resolved by+ ``app.security.service_auth.resolve_token`` where that module exists.+``AISOC_DEV_MODE``+ When set and *no* credential material is configured at all, requests+ resolve to the demo tenant so a local ``docker compose up`` works. This is+ the single canonical dev-mode flag;+ ``services/api/tests/test_security_defaults.py`` asserts no shortcut is+ reachable when it is unset.+"""++from __future__ import annotations++import base64+import binascii+import hashlib+import hmac+import json+import logging+import os+import time+import uuid+from dataclasses import dataclass, field++from fastapi import Header, HTTPException, status++#: Set by each vendored copy so the per-service token override resolves.+SERVICE_NAME = "ACTIONS"++logger = logging.getLogger("aisoc.tenant_scope")++_TRUTHY = {"1", "true", "yes", "on"}++#: Mirrors ``INSECURE_SECRET_KEY_DEFAULTS`` in services/api/app/core/config.py+#: so a placeholder secret is treated as "unset" on this side too. Verifying a+#: token signed with a well-known literal would authenticate anybody who read+#: the repository.+INSECURE_SECRET_DEFAULTS = frozenset(+ {+ "change-me-in-production-at-least-32-chars",+ "dev_secret_key_change_in_production",+ "changeme",+ "secret",+ }+)++#: Deterministic demo tenant, matching ``DEMO_TENANT_ID`` in+#: services/api/app/api/v1/dev_auth.py. Only ever used under AISOC_DEV_MODE.+DEV_TENANT_ID = uuid.UUID("00000000-0000-0000-0000-000000000001")++#: Header a trusted service uses to declare which tenant it is acting for.+TENANT_HEADER = "X-AiSOC-Tenant-ID"++#: 30s clock-skew leeway, matching the Node verifier and common JWT defaults.+_CLOCK_SKEW_SECONDS = 30++#: Refs that mean "the caller did not name a tenant" rather than naming one.+#: Several request models default ``tenant_id`` to the string ``"default"``,+#: which is a placeholder, not a tenant — migration 001 seeds the canonical+#: tenant with that *slug* and the demo seed renames it, so the literal+#: identifies nothing anywhere. Treating it as a request for a tenant called+#: "default" would refuse every caller who simply left the field alone.+PLACEHOLDER_TENANT_REFS = frozenset({"", "default", "none", "null"})+++class TenantScopeError(Exception):+ """A read was attempted without a resolved tenant scope.++ Raised by :func:`resolve_scoped_tenant`. Routes let it surface as a 403;+ it means a query was about to run with no tenant filter, or with one the+ caller is not authorised for.+ """+++@dataclass(frozen=True)+class TenantPrincipal:+ """The tenants one credential may read, and what kind of credential it is.++ ``tenant_ids`` is authoritative and exhaustive. Nothing downstream may+ widen it, and no code path may substitute "all tenants" for an empty one.+ """++ tenant_ids: frozenset[uuid.UUID] = field(default_factory=frozenset)+ subject: str = "anonymous"+ #: True when a service token asserted the tenant rather than a verified+ #: console claim carrying it. Surfaced so logs can tell the two apart.+ delegated: bool = False++ @property+ def is_empty(self) -> bool:+ return not self.tenant_ids++ def covers(self, tenant_id: uuid.UUID) -> bool:+ return tenant_id in self.tenant_ids++ def ordered_ids(self) -> list[uuid.UUID]:+ return sorted(self.tenant_ids)+++EMPTY_PRINCIPAL = TenantPrincipal()+++def resolve_scoped_tenant(+ principal: TenantPrincipal,+ requested: uuid.UUID | str | None = None,+) -> uuid.UUID:+ """The one tenant this request may read, or refuse to name one.++ Intersection only. A ``requested`` tenant inside the principal's scope is+ honoured — that is an MSSP operator narrowing to one managed customer. A+ ``requested`` tenant outside it narrows to nothing and raises, rather than+ reaching outside. Passing ``None`` means "the caller's own tenant", which+ is the common case and the safe default.+ """+ if principal.is_empty:+ raise TenantScopeError("refusing to read with an empty tenant scope")++ if isinstance(requested, str) and requested.strip().lower() in PLACEHOLDER_TENANT_REFS:+ requested = None++ if requested is None:+ if len(principal.tenant_ids) != 1:+ raise TenantScopeError(f"caller reaches {len(principal.tenant_ids)} tenants; the request must name one")+ return next(iter(principal.tenant_ids))++ try:+ wanted = requested if isinstance(requested, uuid.UUID) else uuid.UUID(str(requested))+ except (ValueError, AttributeError, TypeError) as exc:+ raise TenantScopeError("requested tenant is not a UUID") from exc++ if not principal.covers(wanted):+ # Logged at warning with the subject, because a caller reaching for a+ # tenant they do not hold is a security event, not routine noise.+ logger.warning(+ "tenant_scope.refused subject=%s requested=%s scope_size=%d",+ # Sanitised inline rather than through a helper: CodeQL does not+ # track a helper across the call boundary, and `subject` really is+ # attacker-influenced — it carries the JWT `sub` claim.+ str(principal.subject).replace("\r", "").replace("\n", " ")[:128],+ str(wanted).replace("\r", "").replace("\n", " ")[:64],+ len(principal.tenant_ids),+ )+ raise TenantScopeError("requested tenant is outside the caller's authorised scope")+ return wanted+++# ---------------------------------------------------------------------------+# Credential resolution+# ---------------------------------------------------------------------------+++def _dev_mode() -> bool:+ return os.getenv("AISOC_DEV_MODE", "").strip().lower() in _TRUTHY+++def resolve_console_secret() -> str:+ """The HS256 secret console tokens are signed with, or "" when unusable."""+ secret = (os.getenv("SECRET_KEY") or "").strip()+ if not secret or secret in INSECURE_SECRET_DEFAULTS:+ return ""+ return secret+++def resolve_service_token() -> str:+ """Per-service override wins, then the shared platform token."""+ specific = os.getenv(f"AISOC_{SERVICE_NAME}_SERVICE_TOKEN", "").strip()+ if specific:+ return specific+ return os.getenv("AISOC_SERVICE_TOKEN", "").strip()+++def _b64url_decode(segment: str) -> bytes:+ padding = "=" * (-len(segment) % 4)+ return base64.urlsafe_b64decode(segment + padding)+++def verify_console_token(token: str, secret: str) -> dict | None:+ """Verify a first-party HS256 access token. Return claims, or None.++ Mirrors ``verifyRealtimeTicket`` in services/realtime/src/auth.ts. The+ signing side is ``create_access_token`` in services/api/app/core/security.py+ and must stay in sync with the claims checked here.+ """+ if not token or not secret:+ return None+ parts = token.split(".")+ if len(parts) != 3:+ return None+ header_b64, payload_b64, sig_b64 = parts++ try:+ header = json.loads(_b64url_decode(header_b64))+ except (ValueError, binascii.Error, UnicodeDecodeError):+ return None+ # Reject "alg: none" and asymmetric algorithms so a caller cannot downgrade+ # the verification into no verification at all.+ if not isinstance(header, dict) or header.get("alg") != "HS256":+ return None++ expected = hmac.new(+ secret.encode("utf-8"),+ f"{header_b64}.{payload_b64}".encode(),+ hashlib.sha256,+ ).digest()+ try:+ provided = _b64url_decode(sig_b64)+ except (ValueError, binascii.Error):+ return None+ if not hmac.compare_digest(expected, provided):+ return None++ try:+ claims = json.loads(_b64url_decode(payload_b64))+ except (ValueError, binascii.Error, UnicodeDecodeError):+ return None+ if not isinstance(claims, dict):+ return None++ # A refresh token must not buy access; the API sets type="access" on the+ # short-lived one and type="refresh" on the long-lived one.+ if claims.get("type") != "access":+ return None+ exp = claims.get("exp")+ if not isinstance(exp, int | float):+ return None+ if exp + _CLOCK_SKEW_SECONDS < time.time():+ return None+ if not claims.get("tenant_id"):+ return None+ return claims+++def _principal_from_claims(claims: dict) -> TenantPrincipal | None:+ try:+ tenant = uuid.UUID(str(claims["tenant_id"]))+ except (KeyError, ValueError, TypeError):+ return None+ return TenantPrincipal(+ tenant_ids=frozenset({tenant}),+ subject=f"console:{claims.get('sub', 'unknown')}",+ delegated=False,+ )+++async def require_console_or_service_auth(+ authorization: str | None = Header(default=None),+ x_aisoc_tenant_id: str | None = Header(default=None, alias=TENANT_HEADER),+) -> TenantPrincipal:+ """FastAPI dependency. Resolve the caller's tenant scope, or fail closed.++ Order matters only for clarity — the two credential shapes are disjoint. A+ console token carries its own tenant; a service token must declare one.+ """+ console_secret = resolve_console_secret()+ service_token = resolve_service_token()++ if not console_secret and not service_token:+ if _dev_mode():+ return TenantPrincipal(+ tenant_ids=frozenset({DEV_TENANT_ID}),+ subject="dev",+ delegated=False,+ )+ # No credential material configured at all: refuse to serve rather+ # than serve openly. This is the same fail-closed posture as+ # ``require_service_auth``.+ raise HTTPException(+ status_code=status.HTTP_503_SERVICE_UNAVAILABLE,+ detail=(+ "tenant-scoped auth is not configured; set SECRET_KEY and/or "+ f"AISOC_SERVICE_TOKEN (or AISOC_{SERVICE_NAME}_SERVICE_TOKEN) "+ "before exposing this service"+ ),+ )++ if not authorization or not authorization.startswith("Bearer "):+ raise HTTPException(+ status_code=status.HTTP_401_UNAUTHORIZED,+ detail="missing bearer credential",+ headers={"WWW-Authenticate": "Bearer"},+ )+ token = authorization[len("Bearer ") :].strip()++ # --- console session ---+ if console_secret:+ claims = verify_console_token(token, console_secret)+ if claims is not None:+ principal = _principal_from_claims(claims)+ if principal is not None:+ return principal++ # --- trusted service acting for a declared tenant ---+ if service_token and hmac.compare_digest(token, service_token):+ if not x_aisoc_tenant_id or not x_aisoc_tenant_id.strip():+ # An absent tenant is an empty scope, never every scope.+ logger.warning(+ "tenant_scope.service_token_without_tenant service=%s header=%s",+ SERVICE_NAME,+ TENANT_HEADER,+ )+ raise HTTPException(+ status_code=status.HTTP_403_FORBIDDEN,+ detail=f"service token must declare the tenant it acts for on {TENANT_HEADER}",+ )+ try:+ tenant = uuid.UUID(x_aisoc_tenant_id.strip())+ except ValueError:+ raise HTTPException(+ status_code=status.HTTP_400_BAD_REQUEST,+ detail=f"{TENANT_HEADER} must be a UUID",+ ) from None+ return TenantPrincipal(+ tenant_ids=frozenset({tenant}),+ subject="service",+ delegated=True,+ )++ raise HTTPException(+ status_code=status.HTTP_401_UNAUTHORIZED,+ detail="invalid or missing credential",+ headers={"WWW-Authenticate": "Bearer"},+ )+++def scoped_tenant_or_403(+ principal: TenantPrincipal,+ requested: uuid.UUID | str | None = None,+) -> uuid.UUID:+ """:func:`resolve_scoped_tenant`, surfaced to HTTP as 403.++ Routes call this so the refusal reaches the client as a refusal rather+ than a 500, without every route repeating the try/except.+ """+ try:+ return resolve_scoped_tenant(principal, requested)+ except TenantScopeError as exc:+ raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail=str(exc)) from excdiff --git a/services/actions/app/services/shadow_reconcile.py b/services/actions/app/services/shadow_reconcile.pyindex bf9fdd8b6..7f9140d27 100644--- a/services/actions/app/services/shadow_reconcile.py+++ b/services/actions/app/services/shadow_reconcile.py@@ -61,7 +61,7 @@ from app.services.disposition_writeback import CANONICAL_DISPOSITIONSlogger = structlog.get_logger(__name__)-__all__ = ["ReconcileResult", "reconcile_findings"]+__all__ = ["ReconcileResult", "database_configured", "reconcile_findings"]@dataclass(frozen=True)@@ -104,6 +104,19 @@ def _dsn() -> str | None:return raw.replace("postgresql+asyncpg://", "postgresql://").replace("postgres+asyncpg://", "postgresql://")+def database_configured() -> bool:+ """Whether this service can reach the database the decisions live in.++ Asked by the route before it reads a vendor window, because the no-database+ path below returns a result rather than raising: it reports ``considered``+ findings and ``matched`` zero, which is indistinguishable from a window+ whose join key never arrives. One of those is a wiring fault in ingest and+ the other is a missing ``DATABASE_URL`` on this container, and a sweep that+ reported them the same way would send an operator to debug the wrong one.+ """+ return _dsn() is not None++async def _connect(dsn: str) -> Any:import asyncpg # noqa: PLC0415 - optional at import time; only needed with a DSNdiff --git a/services/actions/tests/test_shadow_reconcile_route.py b/services/actions/tests/test_shadow_reconcile_route.pynew file mode 100644index 000000000..4952212b5--- /dev/null+++ b/services/actions/tests/test_shadow_reconcile_route.py@@ -0,0 +1,311 @@+"""The route that gives ``shadow_reconcile`` a caller.++Gap-closure Phase 2.1, closing D15.++The module under test existed with eleven tests and no caller. These cover the+route that changes that, and the four answers it has to be able to give: a+window it read and graded, credentials that will never work, a vendor asking us+to slow down, and a vendor that was merely unhappy. Telling the second apart+from the fourth is the whole point, because one of them is worth retrying on a+timer and the other turns into churn nobody reads.+"""++from __future__ import annotations++import uuid+from datetime import UTC, datetime, timedelta+from typing import Any++import httpx+import pytest+from app.api import shadow_reconcile_router as route_mod+from app.security.tenant_scope import TenantPrincipal, require_console_or_service_auth+from app.services.alert_history import ClosedFinding+from app.services.shadow_reconcile import ReconcileResult+from fastapi import FastAPI+from fastapi.testclient import TestClient++TENANT = uuid.UUID("11111111-1111-1111-1111-111111111111")+OTHER_TENANT = uuid.UUID("22222222-2222-2222-2222-222222222222")++_NOW = datetime(2026, 9, 20, 12, 0, tzinfo=UTC)+++def _finding(+ finding_id: str = "notable-1",+ disposition: str = "true_positive",+ *,+ closed_at: datetime | None = None,+) -> ClosedFinding:+ return ClosedFinding(+ vendor="splunk",+ finding_id=finding_id,+ title="Suspicious PowerShell on FIN-WS-04",+ disposition=disposition,+ vendor_disposition="disposition:1",+ closed_at=closed_at or _NOW,+ closed_by="[email protected]",+ )+++def _body(**overrides: Any) -> dict[str, Any]:+ payload: dict[str, Any] = {+ "vendor": "splunk",+ "credentials": {"splunk_url": "https://splunk.example.com:8089", "splunk_token": "t"},+ "since": (_NOW - timedelta(hours=1)).isoformat(),+ "until": _NOW.isoformat(),+ "limit": 500,+ }+ payload.update(overrides)+ return payload+++def client(monkeypatch: pytest.MonkeyPatch) -> TestClient:+ """The route mounted with a resolved principal and a configured database.++ The principal is overridden rather than a token minted, because what these+ tests are about is the route's behaviour once a tenant is resolved; the+ resolution itself is covered by the vendored ``test_tenant_scope.py`` that+ travels with the module in all ten services.+ """+ monkeypatch.setattr(route_mod, "database_configured", lambda: True)+ app = FastAPI()+ app.include_router(route_mod.router, prefix="/api/v1")+ app.dependency_overrides[require_console_or_service_auth] = lambda: TenantPrincipal(+ tenant_ids=frozenset({TENANT}), subject="service", delegated=True+ )+ return TestClient(app)+++def _patch_reader(monkeypatch: pytest.MonkeyPatch, result: Any) -> list[Any]:+ """Replace the splunk arm. ``result`` is a list to return or an exception to raise."""+ seen: list[Any] = []++ async def _reader(body: Any) -> list[ClosedFinding]:+ seen.append(body)+ if isinstance(result, BaseException):+ raise result+ return result++ monkeypatch.setitem(route_mod.READERS, "splunk", _reader)+ return seen+++def _patch_reconcile(monkeypatch: pytest.MonkeyPatch, **counts: int) -> list[tuple[Any, list[ClosedFinding]]]:+ calls: list[tuple[Any, list[ClosedFinding]]] = []++ async def _reconcile(tenant_id: Any, findings: Any, **_kw: Any) -> ReconcileResult:+ rows = list(findings)+ calls.append((tenant_id, rows))+ return ReconcileResult(considered=len(rows), **counts)++ monkeypatch.setattr(route_mod, "reconcile_findings", _reconcile)+ return calls+++def _no_scope() -> TenantPrincipal:+ """A credential that resolved to no tenant at all.++ A named function rather than a lambda so FastAPI overrides it as a+ zero-argument dependency; handing it the dataclass directly would have the+ framework read the constructor signature and try to populate the fields+ from the request, which is the opposite of what an empty scope means.+ """+ return TenantPrincipal()+++def _status_error(code: int, headers: dict[str, str] | None = None) -> httpx.HTTPStatusError:+ request = httpx.Request("GET", "https://splunk.example.com:8089/services/search/jobs")+ response = httpx.Response(code, headers=headers or {}, request=request)+ return httpx.HTTPStatusError("vendor said no", request=request, response=response)+++class TestAWindowItRead:+ def test_it_reconciles_and_reports_both_halves(self, client: TestClient, monkeypatch: pytest.MonkeyPatch) -> None:+ _patch_reader(monkeypatch, [_finding("a"), _finding("b", "benign")])+ calls = _patch_reconcile(monkeypatch, matched=2)++ response = client.post("/api/v1/shadow/reconcile", json=_body())++ assert response.status_code == 200+ body = response.json()+ assert body["count"] == 2+ assert body["matched"] == 2+ assert body["labelled"] == 2+ # The tenant reached the matcher from the credential, not the body.+ assert calls and str(calls[0][0]) == str(TENANT)++ def test_the_watermark_comes_from_the_latest_closure_not_the_clock(self, client: TestClient, monkeypatch: pytest.MonkeyPatch) -> None:+ """A vendor that indexes late must not have its findings stepped over.++ The caller advances its watermark from this field, so it has to be the+ latest close time actually seen rather than the end of the window the+ caller asked for.+ """+ latest = _finding("a")+ earlier = _finding("b", "benign", closed_at=_NOW - timedelta(minutes=5))+ _patch_reader(monkeypatch, [earlier, latest])+ _patch_reconcile(monkeypatch, matched=2)++ body = client.post("/api/v1/shadow/reconcile", json=_body()).json()++ assert body["latest_closed_at"] == _NOW.isoformat()++ def test_an_empty_window_is_a_success_with_no_watermark(self, client: TestClient, monkeypatch: pytest.MonkeyPatch) -> None:+ _patch_reader(monkeypatch, [])+ _patch_reconcile(monkeypatch)++ body = client.post("/api/v1/shadow/reconcile", json=_body()).json()++ assert body["count"] == 0+ assert body["latest_closed_at"] is None+++class TestPermanentIsNotTransient:+ """The distinction the sweep depends on, asserted from both sides."""++ @pytest.mark.parametrize("code", sorted(route_mod.PERMANENT_VENDOR_STATUSES))+ def test_a_refused_credential_is_permanent_and_names_the_action(+ self, client: TestClient, monkeypatch: pytest.MonkeyPatch, code: int+ ) -> None:+ _patch_reader(monkeypatch, _status_error(code))++ response = client.post("/api/v1/shadow/reconcile", json=_body())++ assert response.status_code == 422+ detail = response.json()["detail"]+ assert "Re-authorise this connector" in detail+ assert "will not recover on its own" in detail++ @pytest.mark.parametrize("code", [500, 502, 503, 504])+ def test_an_unhappy_vendor_is_transient(self, client: TestClient, monkeypatch: pytest.MonkeyPatch, code: int) -> None:+ _patch_reader(monkeypatch, _status_error(code))++ response = client.post("/api/v1/shadow/reconcile", json=_body())++ assert response.status_code == 502+ assert "Re-authorise" not in response.json()["detail"]++ def test_a_timeout_is_transient(self, client: TestClient, monkeypatch: pytest.MonkeyPatch) -> None:+ _patch_reader(monkeypatch, httpx.ReadTimeout("too slow"))++ response = client.post("/api/v1/shadow/reconcile", json=_body())++ assert response.status_code == 502++ def test_rate_limiting_forwards_the_vendors_own_retry_after(self, client: TestClient, monkeypatch: pytest.MonkeyPatch) -> None:+ """Obeyed rather than estimated. A guess is either rude or slow."""+ _patch_reader(monkeypatch, _status_error(429, {"Retry-After": "120"}))++ response = client.post("/api/v1/shadow/reconcile", json=_body())++ assert response.status_code == 429+ assert response.headers["Retry-After"] == "120"++ def test_unbuildable_credentials_are_permanent_rather_than_an_empty_window(+ self, client: TestClient, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ """The reader's own 422 must not become a quiet zero-closure pass.++ An empty window reads as "these analysts closed nothing", which is a+ statement about the customer's week rather than about their config.+ """+ calls = _patch_reconcile(monkeypatch)++ response = client.post("/api/v1/shadow/reconcile", json=_body(credentials={}))++ assert response.status_code == 422+ assert "client factory reads" in response.json()["detail"]+ assert calls == [], "a refused read must not reach the matcher at all"+++class TestRefusalsBeforeTheVendorIsCalled:+ def test_no_database_refuses_instead_of_spending_vendor_quota(self, client: TestClient, monkeypatch: pytest.MonkeyPatch) -> None:+ """Without a DSN the matcher returns zero matches and raises nothing.++ Reported as a success that number is indistinguishable from a broken+ join key, and the two send an operator to different places. So it is+ refused up front, before a customer's search quota is spent on a+ window nothing can be written from.+ """+ monkeypatch.setattr(route_mod, "database_configured", lambda: False)+ seen = _patch_reader(monkeypatch, [_finding()])++ response = client.post("/api/v1/shadow/reconcile", json=_body())++ assert response.status_code == 503+ assert "DATABASE_URL" in response.json()["detail"]+ assert seen == [], "the vendor must not be called when nothing could be recorded"++ def test_an_inverted_window_is_refused(self, client: TestClient, monkeypatch: pytest.MonkeyPatch) -> None:+ seen = _patch_reader(monkeypatch, [_finding()])++ response = client.post(+ "/api/v1/shadow/reconcile",+ json=_body(since=_NOW.isoformat(), until=(_NOW - timedelta(hours=1)).isoformat()),+ )++ assert response.status_code == 422+ assert seen == []+++class TestTheTenantIsNotARequestField:+ def test_the_request_model_carries_no_tenant(self) -> None:+ """A tenant the caller typed is a tenant nothing checked.++ Asserted against the model rather than by sending one, because a field+ Pydantic ignores and a field Pydantic honours both produce a 200 on the+ happy path and only one of them is safe.+ """+ fields = set(route_mod.ReconcileRequest.model_fields)+ assert not {f for f in fields if "tenant" in f}, f"ReconcileRequest grew a tenant field: {sorted(fields)}"++ def test_an_empty_scope_refuses_rather_than_widening(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """A service token with no tenant header must reach nothing, not everything."""+ monkeypatch.setattr(route_mod, "database_configured", lambda: True)+ app = FastAPI()+ app.include_router(route_mod.router, prefix="/api/v1")+ app.dependency_overrides[require_console_or_service_auth] = _no_scope+ seen = _patch_reader(monkeypatch, [_finding()])++ response = TestClient(app).post("/api/v1/shadow/reconcile", json=_body())++ assert response.status_code == 403+ assert seen == []++ def test_the_resolved_tenant_is_the_one_written(self, client: TestClient, monkeypatch: pytest.MonkeyPatch) -> None:+ """Not the one in any header the handler could have read for itself."""+ _patch_reader(monkeypatch, [_finding()])+ calls = _patch_reconcile(monkeypatch, matched=1)++ client.post(+ "/api/v1/shadow/reconcile",+ json=_body(),+ headers={"X-AiSOC-Tenant-ID": str(OTHER_TENANT)},+ )++ assert str(calls[0][0]) == str(TENANT)+++def test_the_route_is_reachable_on_the_deployed_app() -> None:+ """The failure being closed is "exists and nothing calls it".++ A router that is written and never included is the same defect one layer+ up, so the assertion is against the service's own ``app`` rather than+ against a FastAPI instance this test built.++ Reachability is asserted by *sending a request*, not by reading+ ``app.routes``. On the FastAPI version this service pins, ``include_router``+ does not populate ``app.routes`` at all: an app with four routers mounted+ reports only the seven docs and probe routes it was born with, and a test+ that reads that list would pass over a service with no API whatsoever.+ Anything other than 404 proves the path resolves to a handler; the body is+ not the point and is refused here because no credential is presented.+ """+ from app.main import app as deployed # noqa: PLC0415 - importing the service graph is the point++ assert "/api/v1/shadow/reconcile" in deployed.openapi()["paths"]++ response = TestClient(deployed).post("/api/v1/shadow/reconcile", json=_body())+ assert response.status_code != 404, "the router is declared but not mounted on the deployed app"diff --git a/services/actions/tests/test_tenant_scope.py b/services/actions/tests/test_tenant_scope.pynew file mode 100644index 000000000..f2c30dded--- /dev/null+++ b/services/actions/tests/test_tenant_scope.py@@ -0,0 +1,274 @@+"""Unit tests for the vendored tenant-scope resolver.++Vendored byte-identical into each service that carries+``app/security/tenant_scope.py``, and kept in lockstep by+``scripts/sync_vendored_tenant_scope.py``. The module is the thing that+decides which tenant a request may read, so every copy is tested where it+lives rather than trusting that the one covered by ``tests/isolation`` stands+in for the other five.++The cross-service two-tenant replay lives in+``tests/isolation/test_route_tenant_scope.py``; this file covers the+credential and intersection logic on its own.+"""++from __future__ import annotations++import base64+import hashlib+import hmac+import json+import time+import uuid++import pytest+from app.security import tenant_scope as ts+from fastapi import HTTPException++SECRET = "unit-test-secret-key-at-least-32-characters"+SERVICE_TOKEN = "unit-test-service-token"++TENANT_A = uuid.UUID("aaaaaaaa-0000-0000-0000-00000000000a")+TENANT_B = uuid.UUID("bbbbbbbb-0000-0000-0000-00000000000b")+++def _b64(raw: bytes) -> str:+ return base64.urlsafe_b64encode(raw).rstrip(b"=").decode()+++def _token(+ tenant: uuid.UUID | None = TENANT_A,+ *,+ secret: str = SECRET,+ alg: str = "HS256",+ token_type: str = "access",+ ttl: int = 600,+ sub: str = "user-1",+) -> str:+ header = _b64(json.dumps({"alg": alg, "typ": "JWT"}, separators=(",", ":")).encode())+ claims: dict[str, object] = {"sub": sub, "type": token_type, "exp": int(time.time()) + ttl}+ if tenant is not None:+ claims["tenant_id"] = str(tenant)+ payload = _b64(json.dumps(claims, separators=(",", ":")).encode())+ sig = _b64(hmac.new(secret.encode(), f"{header}.{payload}".encode(), hashlib.sha256).digest())+ return f"{header}.{payload}.{sig}"++[email protected](autouse=True)+def _configured(monkeypatch: pytest.MonkeyPatch):+ monkeypatch.setenv("SECRET_KEY", SECRET)+ monkeypatch.setenv("AISOC_SERVICE_TOKEN", SERVICE_TOKEN)+ monkeypatch.delenv("AISOC_DEV_MODE", raising=False)+ monkeypatch.delenv(f"AISOC_{ts.SERVICE_NAME}_SERVICE_TOKEN", raising=False)+++# ---------------------------------------------------------------------------+# Token verification+# ---------------------------------------------------------------------------+++def test_valid_token_yields_its_tenant() -> None:+ claims = ts.verify_console_token(_token(), SECRET)+ assert claims is not None and claims["tenant_id"] == str(TENANT_A)+++ ("label", "kwargs"),+ [+ # An `alg: none` header is the classic downgrade: a verifier that+ # reads the header to decide how to verify must refuse anything else.+ ("alg none", {"alg": "none"}),+ ("wrong key", {"secret": "a-completely-different-secret-key-32ch"}),+ # A refresh token is long-lived by design; buying access with it would+ # make the short access-token expiry meaningless.+ ("refresh token", {"token_type": "refresh"}),+ ("expired", {"ttl": -3600}),+ ("no tenant claim", {"tenant": None}),+ ],+)+def test_unacceptable_tokens_are_refused(label: str, kwargs: dict) -> None:+ assert ts.verify_console_token(_token(**kwargs), SECRET) is None, label++[email protected]("malformed", ["", "not-a-jwt", "a.b", "a.b.c.d", "...", "%%%.%%%.%%%"])+def test_malformed_tokens_are_refused(malformed: str) -> None:+ assert ts.verify_console_token(malformed, SECRET) is None+++def test_no_secret_means_no_verification() -> None:+ """An unconfigured secret must not verify anything, least of all everything."""+ assert ts.verify_console_token(_token(), "") is None+++def test_placeholder_secrets_count_as_unset() -> None:+ for placeholder in ts.INSECURE_SECRET_DEFAULTS:+ assert ts.resolve_console_secret.__doc__ # module contract exists+ assert placeholder in ts.INSECURE_SECRET_DEFAULTS+++# ---------------------------------------------------------------------------+# Intersection+# ---------------------------------------------------------------------------+++def _principal(*tenants: uuid.UUID) -> ts.TenantPrincipal:+ return ts.TenantPrincipal(tenant_ids=frozenset(tenants), subject="console:test")+++def test_no_request_tenant_means_the_callers_own() -> None:+ assert ts.resolve_scoped_tenant(_principal(TENANT_A)) == TENANT_A+++def test_a_tenant_inside_scope_is_honoured() -> None:+ assert ts.resolve_scoped_tenant(_principal(TENANT_A, TENANT_B), TENANT_B) == TENANT_B+++def test_a_tenant_outside_scope_narrows_to_nothing() -> None:+ with pytest.raises(ts.TenantScopeError):+ ts.resolve_scoped_tenant(_principal(TENANT_A), TENANT_B)+++def test_empty_scope_refuses_rather_than_widening() -> None:+ """The invariant every cross-tenant leak in this codebase has broken."""+ with pytest.raises(ts.TenantScopeError):+ ts.resolve_scoped_tenant(ts.EMPTY_PRINCIPAL)+ with pytest.raises(ts.TenantScopeError):+ ts.resolve_scoped_tenant(ts.EMPTY_PRINCIPAL, TENANT_A)+++def test_a_multi_tenant_caller_must_name_one() -> None:+ """Silently picking one of several would be a coin toss over customers."""+ with pytest.raises(ts.TenantScopeError):+ ts.resolve_scoped_tenant(_principal(TENANT_A, TENANT_B))+++def test_a_non_uuid_request_tenant_is_refused() -> None:+ with pytest.raises(ts.TenantScopeError):+ ts.resolve_scoped_tenant(_principal(TENANT_A), "not-a-uuid")++[email protected]("placeholder", ["", " ", "default", "DEFAULT", "none", "null"])+def test_a_placeholder_means_the_caller_did_not_ask(placeholder: str) -> None:+ """A placeholder is the absence of a request, not a tenant named "default".++ Several request models default `tenant_id` to the literal `"default"`.+ That string names no tenant anywhere — migration 001 seeds the canonical+ tenant with it as a *slug* and the demo seed renames that to `demo` — so+ treating it as a requested tenant would refuse every caller who left the+ field at its default.+ """+ assert ts.resolve_scoped_tenant(_principal(TENANT_A), placeholder) == TENANT_A+++def test_a_placeholder_still_cannot_widen_an_empty_scope() -> None:+ with pytest.raises(ts.TenantScopeError):+ ts.resolve_scoped_tenant(ts.EMPTY_PRINCIPAL, "default")+++def test_scoped_tenant_or_403_surfaces_a_refusal_as_403() -> None:+ with pytest.raises(HTTPException) as exc:+ ts.scoped_tenant_or_403(_principal(TENANT_A), TENANT_B)+ assert exc.value.status_code == 403+++# ---------------------------------------------------------------------------+# The dependency+# ---------------------------------------------------------------------------+++# `require_console_or_service_auth` is a FastAPI dependency: its parameters+# are normally resolved by FastAPI from the request. Calling it directly means+# supplying both, including the tenant header as an explicit `None` — leaving+# it out hands the function FastAPI's `Header(...)` sentinel rather than the+# absent value the test means to describe.+++async def test_console_session_resolves_to_its_own_tenant() -> None:+ principal = await ts.require_console_or_service_auth(authorization=f"Bearer {_token()}", x_aisoc_tenant_id=None)+ assert principal.tenant_ids == frozenset({TENANT_A})+ assert principal.delegated is False+++async def test_service_token_must_declare_the_tenant_it_acts_for() -> None:+ """A service credential identifies a service, not a tenant."""+ with pytest.raises(HTTPException) as exc:+ await ts.require_console_or_service_auth(authorization=f"Bearer {SERVICE_TOKEN}", x_aisoc_tenant_id=None)+ assert exc.value.status_code == 403+++async def test_service_token_with_a_declared_tenant_is_delegated() -> None:+ principal = await ts.require_console_or_service_auth(+ authorization=f"Bearer {SERVICE_TOKEN}",+ x_aisoc_tenant_id=str(TENANT_B),+ )+ assert principal.tenant_ids == frozenset({TENANT_B})+ assert principal.delegated is True+++async def test_a_declared_tenant_must_be_a_uuid() -> None:+ with pytest.raises(HTTPException) as exc:+ await ts.require_console_or_service_auth(+ authorization=f"Bearer {SERVICE_TOKEN}",+ x_aisoc_tenant_id="the-big-customer",+ )+ assert exc.value.status_code == 400++[email protected]("header", [None, "", "Basic abc", "Bearer ", "Bearer nonsense"])+async def test_a_missing_or_unrecognised_credential_is_401(header: str | None) -> None:+ with pytest.raises(HTTPException) as exc:+ await ts.require_console_or_service_auth(authorization=header, x_aisoc_tenant_id=None)+ assert exc.value.status_code == 401+++async def test_unconfigured_credentials_fail_closed(monkeypatch: pytest.MonkeyPatch) -> None:+ """No credential material and no dev mode serves 503, never the route."""+ monkeypatch.setenv("SECRET_KEY", "")+ monkeypatch.setenv("AISOC_SERVICE_TOKEN", "")+ monkeypatch.delenv("AISOC_DEV_MODE", raising=False)+ with pytest.raises(HTTPException) as exc:+ await ts.require_console_or_service_auth(authorization="Bearer anything", x_aisoc_tenant_id=None)+ assert exc.value.status_code == 503+++async def test_dev_mode_only_applies_with_nothing_configured(monkeypatch: pytest.MonkeyPatch) -> None:+ """Dev mode is a fallback for an unconfigured stack, not an auth bypass.++ With a secret configured, `AISOC_DEV_MODE` must not let an unauthenticated+ caller through — otherwise setting one flag in the wrong environment opens+ every tenant-scoped route.+ """+ monkeypatch.setenv("AISOC_DEV_MODE", "1")+ monkeypatch.setenv("SECRET_KEY", SECRET)+ monkeypatch.setenv("AISOC_SERVICE_TOKEN", SERVICE_TOKEN)+ with pytest.raises(HTTPException) as exc:+ await ts.require_console_or_service_auth(authorization=None, x_aisoc_tenant_id=None)+ assert exc.value.status_code == 401++ monkeypatch.setenv("SECRET_KEY", "")+ monkeypatch.setenv("AISOC_SERVICE_TOKEN", "")+ principal = await ts.require_console_or_service_auth(authorization=None, x_aisoc_tenant_id=None)+ assert principal.tenant_ids == frozenset({ts.DEV_TENANT_ID})+++async def test_a_per_service_token_override_wins(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setenv(f"AISOC_{ts.SERVICE_NAME}_SERVICE_TOKEN", "service-specific")+ principal = await ts.require_console_or_service_auth(+ authorization="Bearer service-specific",+ x_aisoc_tenant_id=str(TENANT_A),+ )+ assert principal.delegated is True+ # The shared token must no longer be accepted once an override exists.+ with pytest.raises(HTTPException):+ await ts.require_console_or_service_auth(+ authorization=f"Bearer {SERVICE_TOKEN}",+ x_aisoc_tenant_id=str(TENANT_A),+ )diff --git a/services/api/app/api/v1/endpoints/health.py b/services/api/app/api/v1/endpoints/health.pyindex 21a44a226..9490548e3 100644--- a/services/api/app/api/v1/endpoints/health.py+++ b/services/api/app/api/v1/endpoints/health.py@@ -87,6 +87,7 @@ from app.models.alert import Alertfrom app.models.connector import Connectorfrom app.services.connector_freshness import compute_freshnessfrom app.services.fleet_health import assess_fleet+from app.services.replay_evaluation.vendors import replayable_connector_idslogger = logging.getLogger(__name__)@@ -618,3 +619,151 @@ async def get_dead_letters(for r in rows],}++[email protected]("/shadow-reconciliation")+async def get_shadow_reconciliation_health(+ user: AuthUser,+ db: DBSession,+) -> dict[str, Any]:+ """Whether closures made in this tenant's own SIEM are being polled back.++ Gap-closure Phase 2.1 (D15).++ A sweep that silently stopped and a sweep with nothing to do look identical+ from outside, and that ambiguity is what made the original gap invisible:+ the reconciler existed, nothing called it, and the only symptom available+ to an operator was a scorecard that never filled in. So this reports the+ subscription in every state, including the healthy one and the idle one,+ and names which it is.++ ``state`` is one of:++ ``disabled`` the operator has not switched the sweep on. Not a fault,+ and not a healthy idle sweep either.+ ``not_measuring`` the sweep runs, but this tenant has no alert class in+ shadow mode, so there is nothing to reconcile.+ ``no_connector`` this tenant is measuring and has no enabled connector of+ a type with a closed-finding reader. Agreement is being+ measured on AiSOC closures only, which is the honest answer+ and is stated rather than left to be inferred.+ ``blocked`` at least one connector needs an operator. The reason is a+ sentence, and it will not clear by waiting.+ ``degraded`` at least one connector is failing transiently and is being+ retried.+ ``ok`` every configured connector polled.+ """+ enabled = bool(settings.SHADOW_RECONCILE_ENABLED)++ measuring = (+ (+ await db.execute(+ text(+ """+ SELECT alert_class, COALESCE(enabled_at, updated_at) AS since+ FROM aisoc_shadow_mode+ WHERE tenant_id = :tid AND enabled IS TRUE+ ORDER BY alert_class+ """+ ),+ {"tid": user.tenant_id},+ )+ )+ .mappings()+ .all()+ )++ rows = (+ (+ await db.execute(+ text(+ """+ SELECT s.connector_id, s.vendor, s.watermark_at, s.last_run_at, s.last_status,+ s.last_detail, s.last_considered, s.last_matched, s.consecutive_failures,+ s.blocked_reason, s.blocked_at, s.retry_after, c.name AS connector_name+ FROM aisoc_shadow_reconcile_state s+ LEFT JOIN connectors c ON c.id = s.connector_id AND c.tenant_id = s.tenant_id+ WHERE s.tenant_id = :tid+ ORDER BY s.last_run_at DESC NULLS LAST+ """+ ),+ {"tid": user.tenant_id},+ )+ )+ .mappings()+ .all()+ )++ # Which of this tenant's connectors the sweep *could* poll, asked of the+ # same table the sweep asks so the two cannot disagree about what counts.+ pollable = int(+ (+ await db.execute(+ text(+ """+ SELECT COUNT(*)::int FROM connectors+ WHERE tenant_id = :tid AND is_enabled IS TRUE AND connector_type = ANY(:replayable)+ """+ ),+ {"tid": user.tenant_id, "replayable": replayable_connector_ids()},+ )+ ).scalar_one()+ or 0+ )++ blocked = [r for r in rows if r["blocked_reason"]]+ failing = [r for r in rows if r["last_status"] == "transient"]++ if not enabled:+ state = "disabled"+ summary = (+ "The shadow-reconciliation sweep is switched off, so closures your analysts make in your own "+ "SIEM are not being polled back. Agreement is measured on closures made in AiSOC only."+ )+ elif not measuring:+ state = "not_measuring"+ summary = "No alert class is in shadow mode for this tenant, so there is nothing to reconcile."+ elif pollable == 0:+ state = "no_connector"+ summary = (+ "This tenant is measuring but has no enabled connector of a type with a closed-finding reader "+ f"({', '.join(replayable_connector_ids())}). Agreement is measured on closures made in AiSOC only."+ )+ elif blocked:+ state = "blocked"+ summary = f"{len(blocked)} connector(s) need an operator before reconciliation can resume."+ elif failing:+ state = "degraded"+ summary = f"{len(failing)} connector(s) are failing transiently and are being retried."+ else:+ state = "ok"+ summary = f"Polling {pollable} connector(s) for closures made in your own SIEM."++ return {+ "state": state,+ "summary": summary,+ "enabled": enabled,+ "interval_seconds": int(settings.SHADOW_RECONCILE_INTERVAL_SECONDS) if enabled else None,+ "measuring_classes": [r["alert_class"] for r in measuring],+ "pollable_connectors": pollable,+ "supported_connector_types": replayable_connector_ids(),+ "connectors": [+ {+ "connector_id": str(r["connector_id"]),+ "connector_name": r["connector_name"],+ "vendor": r["vendor"],+ "status": r["last_status"],+ "detail": r["last_detail"],+ "watermark_at": r["watermark_at"].isoformat() if r["watermark_at"] else None,+ "last_run_at": r["last_run_at"].isoformat() if r["last_run_at"] else None,+ "closures_read": int(r["last_considered"] or 0),+ "closures_matched": int(r["last_matched"] or 0),+ "consecutive_failures": int(r["consecutive_failures"] or 0),+ "needs_operator": bool(r["blocked_reason"]),+ "blocked_reason": r["blocked_reason"],+ "blocked_at": r["blocked_at"].isoformat() if r["blocked_at"] else None,+ "retry_after": r["retry_after"].isoformat() if r["retry_after"] else None,+ }+ for r in rows+ ],+ }diff --git a/services/api/app/core/config.py b/services/api/app/core/config.pyindex c4e9d2da8..9d683f01b 100644--- a/services/api/app/core/config.py+++ b/services/api/app/core/config.py@@ -322,6 +322,42 @@ class Settings(BaseSettings):)RETENTION_WORKER_INTERVAL_SECONDS: int = 21600 # 6h+ # Shadow-reconciliation sweep (gap-closure Phase 2.1, D15). Polls each+ # measuring tenant's own SIEM for the closures their analysts made there,+ # so a tenant whose queue lives in Splunk ES still accumulates the track+ # record autonomy is earned on.+ #+ # Default **off**, per the standing rule that a feature which calls out+ # ships off by default. Two different people are involved: a tenant+ # enabling shadow mode has asked to be measured, and the operator of the+ # deployment is the one who decides whether the platform may reach a+ # third-party API on a timer. `GET /api/v1/health/shadow-reconciliation`+ # reports "disabled" in those words rather than looking like a healthy+ # idle sweep.+ #+ # Every other value here bounds what this deployment does to somebody+ # else's SIEM: the tick cadence, a per-connector floor so a short cadence+ # cannot become a poll storm, a cap on connectors per pass so a large+ # estate is spread across passes, and a cap on one window so a connector+ # that was blocked for a month catches up in steps rather than asking for+ # the month in a single search.+ SHADOW_RECONCILE_ENABLED: bool = Field(+ default=False,+ validation_alias=AliasChoices("SHADOW_RECONCILE_ENABLED", "AISOC_SHADOW_RECONCILE_ENABLED"),+ )+ SHADOW_RECONCILE_INTERVAL_SECONDS: int = 900 # 15m+ SHADOW_RECONCILE_MIN_CONNECTOR_INTERVAL_SECONDS: int = 3600 # 1h+ SHADOW_RECONCILE_MAX_CONNECTORS_PER_TICK: int = 10+ # Re-read this much of the previous window. A vendor's search index lags+ # its own close events, so a window starting exactly where the last one+ # ended steps over anything indexed late. Re-reading is free of+ # consequence: an already-graded finding comes back as already_resolved.+ SHADOW_RECONCILE_OVERLAP_SECONDS: int = 900 # 15m+ # How far a first pass may reach back when a connector has no watermark.+ SHADOW_RECONCILE_MAX_LOOKBACK_HOURS: int = 168 # 7d+ SHADOW_RECONCILE_MAX_WINDOW_HOURS: int = 24+ SHADOW_RECONCILE_LIMIT: int = 1000+# Database# The default points at the bundled compose Postgres with its dev password.# In docker-compose.yml and .env.example the password is parameterised viadiff --git a/services/api/app/main.py b/services/api/app/main.pyindex 611ba18fb..c4ffe7ebf 100644--- a/services/api/app/main.py+++ b/services/api/app/main.py@@ -34,6 +34,7 @@ from app.services.plugin_manager import get_plugin_managerfrom app.workers.hunt_scheduler import run_forever as run_hunt_schedulerfrom app.workers.oauth_refresh import run_forever as run_oauth_refreshfrom app.workers.retention_purge import run_forever as run_retention_purge+from app.workers.shadow_reconcile import run_forever as run_shadow_reconcilefrom app.workers.weekly_digest_task import run_forever as run_weekly_digest_metrics_bearer = HTTPBearer(auto_error=False)@@ -55,6 +56,11 @@ _HUNT_SCHEDULER_LOCK_TTL_SECONDS = 300# ClickHouse mutation with mutations_sync=1 blocks until it lands. 30m keeps# a second replica from starting a concurrent sweep mid-delete._RETENTION_PURGE_LOCK_TTL_SECONDS = 1800+# A shadow-reconciliation pass makes up to SHADOW_RECONCILE_MAX_CONNECTORS_PER_TICK+# vendor searches, each bounded at 120s. 30m covers a slow pass without letting+# a second replica start one on top of it, which would double a customer's+# search load for no extra evidence.+_SHADOW_RECONCILE_LOCK_TTL_SECONDS = 1800async def _run_guarded_scheduler_worker(@@ -432,6 +438,37 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:except Exception as exc:logger.warning("hunt_scheduler worker failed to start", error=str(exc))+ # Gap-closure Phase 2.1 (D15). Polls each measuring tenant's own SIEM for+ # the closures their analysts made there. Without this the shadow-agreement+ # scorecard only ever fills in for tenants whose analysts close alerts in+ # this console, and a tenant working entirely in Splunk ES sees an empty+ # page that reads as "the agent is not being evaluated".+ #+ # Default off: this reaches a third party's API on a timer, and the plan's+ # standing rule is that a feature which calls out ships off by default.+ shadow_reconcile_task: asyncio.Task | None = None+ if settings.SHADOW_RECONCILE_ENABLED:+ try:+ shadow_reconcile_task = asyncio.create_task(+ _run_guarded_scheduler_worker(+ job_name="shadow_reconcile",+ ttl_seconds=_SHADOW_RECONCILE_LOCK_TTL_SECONDS,+ worker=run_shadow_reconcile,+ ),+ name="shadow_reconcile_worker",+ )+ logger.info("shadow_reconcile worker started")+ except Exception as exc:+ logger.warning("shadow_reconcile worker failed to start", error=str(exc))+ else:+ # Said out loud, because the state this closes is one where an operator+ # reads an empty scorecard and cannot tell a sweep that is off from one+ # that is broken.+ logger.info(+ "shadow_reconcile worker disabled; closures made in a tenant's own SIEM will not be "+ "reconciled. Set SHADOW_RECONCILE_ENABLED=true to poll them"+ )+# Phase 2.6 — flip /readyz to 200. All lifespan-managed# dependencies have been touched at this point (DB, Redis,# Neo4j, schedulers); the load balancer can route traffic@@ -482,6 +519,15 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:except Exception as exc:logger.warning("retention_purge worker shutdown error", error=type(exc).__name__)+ if shadow_reconcile_task is not None and not shadow_reconcile_task.done():+ shadow_reconcile_task.cancel()+ try:+ await shadow_reconcile_task+ except asyncio.CancelledError:+ logger.debug("shadow_reconcile worker cancelled during shutdown")+ except Exception as exc:+ logger.warning("shadow_reconcile worker shutdown error", error=type(exc).__name__)+if demo_bootstrap_task is not None and not demo_bootstrap_task.done():demo_bootstrap_task.cancel()try:diff --git a/services/api/app/workers/shadow_reconcile.py b/services/api/app/workers/shadow_reconcile.pynew file mode 100644index 000000000..662512777--- /dev/null+++ b/services/api/app/workers/shadow_reconcile.py@@ -0,0 +1,729 @@+"""Poll every measuring tenant's SIEM for the closures their analysts made there.++Gap-closure Phase 2.1, closing D15.++``services/actions/app/services/shadow_reconcile.py`` shipped complete, with+eleven tests, and nothing called it. Half of shadow mode therefore worked and+half of it only appeared to: a tenant whose analysts close their queue in+AiSOC accumulated a track record on every read of the agreement endpoint, and+a tenant whose analysts close their queue in Splunk ES watched a scorecard+that could not fill in. Phase 2's premise is that autonomy is earned on a+measured track record, so a track record that cannot accumulate is the+difference between the feature working and the feature appearing to.++This is the caller. It runs in ``services/api`` because that is the only+service that can drive it: the vault and the tenant session are here, the five+readers and the matcher are in ``services/actions``, and no process can hold+both because all three services package their code as top-level ``app`` (D8).+So a pass resolves a connector's credentials here and posts them with a window+to ``POST /api/v1/shadow/reconcile`` there, which is the same round trip+``siem_writeback``, ``/connectors/{id}/normalize`` and ``/replay/history``+already take.++Four outcomes, because two is not enough+========================================++A scheduled sweep is a consumer, and the rule this repository learned from the+UEBA consumer and the graph broadcaster applies: it has to tell a condition+that will never resolve from one that might, and say which. Reporting both as+"tick failed" turns a revoked API key into churn nobody reads, and reporting+neither leaves a stopped sweep looking exactly like an idle one.++``ok`` a window was read and reconciled. The watermark advances.+``idle`` there was nothing to do. The tenant is measuring but has no+ replayable connector, or the window has not moved since the+ last pass. Recorded rather than skipped, because "nothing to+ do" and "stopped" are the two states an outside observer+ cannot otherwise tell apart.+``blocked`` permanent until an operator acts: the vault cannot decrypt the+ stored credentials, or the vendor refused them. Polling stops+ for that connector and resumes on the single event that could+ have fixed it, which is the connector row being saved again.+ A timer would be the churn this state exists to avoid.+``transient`` the vendor timed out, rate-limited us, or the actions service+ was unreachable. Retried on the next tick, with the run length+ recorded so a fault that has outlived any explanation a retry+ would fix is visible as such.++Bounded work against somebody else's API+========================================++Every limit here is a limit on what this deployment does to a customer's SIEM.+The watermark stops a window being re-read forever, the per-connector floor+stops a short tick cadence becoming a poll storm, the per-tick cap spreads a+large estate across passes rather than fanning out at once, a vendor's own+``Retry-After`` is stored and honoured, and one window is capped so a connector+that was blocked for a month catches up in steps rather than asking for the+month in a single search.++Default off+===========++``SHADOW_RECONCILE_ENABLED`` defaults to false. The plan's standing rule is+that a feature which calls out ships off by default, and the two people+involved are different: a tenant enabling shadow mode has asked to be+measured, and the operator of the deployment is the one who decides whether+the platform may reach a third-party API on a timer. The health endpoint says+"disabled" in those words rather than reporting a healthy idle sweep.+"""++from __future__ import annotations++import asyncio+import logging+import uuid+from dataclasses import asdict, dataclass, field+from datetime import UTC, datetime, timedelta+from typing import Any++import httpx+from sqlalchemy import text+from sqlalchemy.ext.asyncio import AsyncSession++from app.core.airgap import AirgapViolation, enforce_airgap_for_url+from app.core.config import settings+from app.db.cross_tenant import assert_cross_tenant_session+from app.db.database import AsyncSessionLocal+from app.security.credential_vault import CredentialVaultError, get_vault+from app.services.actions_client import base_url as actions_base_url+from app.services.actions_client import service_headers as actions_headers+from app.services.replay_evaluation.vendors import (+ UnsupportedConnector,+ credentials_for,+ replayable_connector_ids,+ vendor_for,+)+from app.workers._tick_failures import TickFailures++logger = logging.getLogger("aisoc.shadow_reconcile")++__all__ = [+ "ConnectorOutcome",+ "SweepRun",+ "run_forever",+ "run_once",+]++#: A vendor search over a window of closures, not a containment. Generous+#: enough that a slow Splunk dispatch is not recorded as a fault, bounded so a+#: hung vendor cannot hold the tick open behind it.+_RECONCILE_TIMEOUT_S = 120.0++#: Statuses the actions route uses for a condition that will not clear on its+#: own: 422 is "these credentials cannot build a client, or the vendor refused+#: them". Kept as a set rather than a comparison so adding one is one edit.+_PERMANENT_STATUSES = frozenset({422})++#: Deployment-level refusals. These are not about any one connector, so they+#: halt the pass and are reported once instead of being written onto every+#: connector's row as if each had its own problem.+_DEPLOYMENT_STATUSES = frozenset({401, 403, 404, 503})+++@dataclass(frozen=True)+class ConnectorOutcome:+ """What one pass did for one connector, in the words the state row stores."""++ tenant_id: uuid.UUID+ connector_id: uuid.UUID+ vendor: str+ status: str+ detail: str+ considered: int = 0+ matched: int = 0+ unmatched: int = 0+ #: Present only on ``ok``. The close time the watermark advances to.+ watermark_at: datetime | None = None+ #: Present only when a vendor asked us to wait.+ retry_after: datetime | None = None++ def as_dict(self) -> dict[str, Any]:+ out = asdict(self)+ out["tenant_id"] = str(self.tenant_id)+ out["connector_id"] = str(self.connector_id)+ out["watermark_at"] = self.watermark_at.isoformat() if self.watermark_at else None+ out["retry_after"] = self.retry_after.isoformat() if self.retry_after else None+ return out+++@dataclass+class SweepRun:+ """One pass over every due connector.++ ``measuring_tenants`` and ``candidates`` are both reported, and the pair is+ the point: zero measuring tenants means nobody asked for this, while+ measuring tenants with zero candidates means somebody asked and has no+ connector this can poll. Only the second is worth telling them about.+ """++ started_at: datetime+ measuring_tenants: int = 0+ candidates: int = 0+ outcomes: list[ConnectorOutcome] = field(default_factory=list)+ #: Set when the pass stopped for a reason that applies to every connector.+ halted_reason: str | None = None++ def count(self, status: str) -> int:+ return sum(1 for o in self.outcomes if o.status == status)++ def as_dict(self) -> dict[str, Any]:+ return {+ "started_at": self.started_at.isoformat(),+ "measuring_tenants": self.measuring_tenants,+ "candidates": self.candidates,+ "polled": len(self.outcomes),+ "ok": self.count("ok"),+ "idle": self.count("idle"),+ "blocked": self.count("blocked"),+ "transient": self.count("transient"),+ "halted_reason": self.halted_reason,+ "outcomes": [o.as_dict() for o in self.outcomes],+ }+++class _Halt(Exception):+ """A refusal that applies to every connector, so the pass stops."""+++@dataclass(frozen=True)+class _Candidate:+ """A connector this pass may poll, and everything needed to decide whether to."""++ tenant_id: uuid.UUID+ connector_id: uuid.UUID+ connector_type: str+ auth_config: dict[str, Any]+ connector_config: dict[str, Any]+ connector_updated_at: datetime | None+ measuring_since: datetime | None+ watermark_at: datetime | None+ last_run_at: datetime | None+ retry_after: datetime | None+ blocked_reason: str | None+ blocked_connector_updated_at: datetime | None+ consecutive_failures: int+++#: One statement, because the decision "may this connector be polled now" reads+#: four tables and splitting it would mean a candidate list assembled from rows+#: that were true at four different instants.+#:+#: The connector must be enabled and of a type with a reader; the tenant must+#: have at least one alert class in shadow mode, since reconciling closures for+#: a tenant that is not measuring grades nothing. ``measuring_since`` is the+#: earliest ``enabled_at`` across that tenant's classes, which is the first+#: moment there was anything to grade.+_CANDIDATES_SQL = """+SELECT+ c.id AS connector_id,+ c.tenant_id AS tenant_id,+ c.connector_type AS connector_type,+ c.auth_config AS auth_config,+ c.connector_config AS connector_config,+ c.updated_at AS connector_updated_at,+ m.measuring_since AS measuring_since,+ s.watermark_at AS watermark_at,+ s.last_run_at AS last_run_at,+ s.retry_after AS retry_after,+ s.blocked_reason AS blocked_reason,+ s.blocked_connector_updated_at AS blocked_connector_updated_at,+ COALESCE(s.consecutive_failures, 0) AS consecutive_failures+FROM connectors c+JOIN (+ SELECT tenant_id, MIN(COALESCE(enabled_at, updated_at, created_at)) AS measuring_since+ FROM aisoc_shadow_mode+ WHERE enabled IS TRUE+ GROUP BY tenant_id+) m ON m.tenant_id = c.tenant_id+LEFT JOIN aisoc_shadow_reconcile_state s+ ON s.tenant_id = c.tenant_id AND s.connector_id = c.id+WHERE c.is_enabled IS TRUE+ AND c.connector_type = ANY(:replayable)+ORDER BY s.last_run_at ASC NULLS FIRST, c.id ASC+"""++_MEASURING_TENANTS_SQL = "SELECT COUNT(DISTINCT tenant_id)::int FROM aisoc_shadow_mode WHERE enabled IS TRUE"++_UPSERT_STATE_SQL = """+INSERT INTO aisoc_shadow_reconcile_state (+ tenant_id, connector_id, vendor, watermark_at, last_run_at, last_status, last_detail,+ last_considered, last_matched, last_unmatched, consecutive_failures,+ blocked_reason, blocked_at, blocked_connector_updated_at, retry_after, updated_at+) VALUES (+ :tenant_id, :connector_id, :vendor, :watermark_at, :last_run_at, :last_status, :last_detail,+ :considered, :matched, :unmatched, :consecutive_failures,+ :blocked_reason, :blocked_at, :blocked_connector_updated_at, :retry_after, now()+)+ON CONFLICT (tenant_id, connector_id) DO UPDATE SET+ vendor = EXCLUDED.vendor,+ -- Never moves backwards. A transient pass leaves it where the last+ -- successful one put it, so a failure cannot make the sweep re-read a+ -- window it already graded, and a clock that jumps cannot make it skip one.+ watermark_at = GREATEST(+ COALESCE(EXCLUDED.watermark_at, aisoc_shadow_reconcile_state.watermark_at),+ COALESCE(aisoc_shadow_reconcile_state.watermark_at, EXCLUDED.watermark_at)+ ),+ last_run_at = EXCLUDED.last_run_at,+ last_status = EXCLUDED.last_status,+ last_detail = EXCLUDED.last_detail,+ last_considered = EXCLUDED.last_considered,+ last_matched = EXCLUDED.last_matched,+ last_unmatched = EXCLUDED.last_unmatched,+ consecutive_failures = EXCLUDED.consecutive_failures,+ blocked_reason = EXCLUDED.blocked_reason,+ blocked_at = EXCLUDED.blocked_at,+ blocked_connector_updated_at = EXCLUDED.blocked_connector_updated_at,+ retry_after = EXCLUDED.retry_after,+ updated_at = now()+"""+++def _interval() -> int:+ """Tick cadence, floored so a misconfigured value cannot become a poll storm."""+ return max(int(getattr(settings, "SHADOW_RECONCILE_INTERVAL_SECONDS", 900)), 60)+++def _min_connector_interval() -> int:+ return max(int(getattr(settings, "SHADOW_RECONCILE_MIN_CONNECTOR_INTERVAL_SECONDS", 3600)), 60)+++def _max_per_tick() -> int:+ return max(int(getattr(settings, "SHADOW_RECONCILE_MAX_CONNECTORS_PER_TICK", 10)), 1)+++def _overlap() -> timedelta:+ return timedelta(seconds=max(int(getattr(settings, "SHADOW_RECONCILE_OVERLAP_SECONDS", 900)), 0))+++def _max_lookback() -> timedelta:+ return timedelta(hours=max(int(getattr(settings, "SHADOW_RECONCILE_MAX_LOOKBACK_HOURS", 168)), 1))+++def _max_window() -> timedelta:+ return timedelta(hours=max(int(getattr(settings, "SHADOW_RECONCILE_MAX_WINDOW_HOURS", 24)), 1))+++def _limit() -> int:+ return max(min(int(getattr(settings, "SHADOW_RECONCILE_LIMIT", 1000)), 5000), 1)+++def _safe(value: object, cap: int = 300) -> str:+ """One-line, length-capped rendering for a log record and for the state row."""+ return str(value).replace("\r", "").replace("\n", " ")[:cap]+++def _aware(value: datetime | None) -> datetime | None:+ """Postgres hands back aware values; a test fixture may not. Compare like for like."""+ if value is None:+ return None+ return value if value.tzinfo is not None else value.replace(tzinfo=UTC)+++def _window_for(candidate: _Candidate, now: datetime) -> tuple[datetime, datetime] | None:+ """The window to ask the vendor for, or ``None`` when there is nothing to ask.++ The start is the watermark less the overlap, because a vendor's search+ index lags its own close events and a window beginning exactly where the+ last one ended steps over anything indexed late. Re-reading the overlap is+ free of consequence: ``reconcile_findings`` only writes decisions whose+ ``resolved_at`` is null, so an already-graded finding comes back as+ ``already_resolved``.++ With no watermark the start is when this tenant began measuring, floored at+ the maximum lookback. Reaching back further would grade closures made+ before there was a decision to grade them against.+ """+ watermark = _aware(candidate.watermark_at)+ floor = now - _max_lookback()+ if watermark is not None:+ start = watermark - _overlap()+ else:+ measuring_since = _aware(candidate.measuring_since)+ start = measuring_since if measuring_since is not None else floor+ start = max(start, floor)++ # One window at a time. A connector that was blocked for a month catches up+ # over several passes rather than asking a customer's SIEM for the month in+ # a single search, which is the request most likely to be refused or to time+ # out and so the one least likely to ever succeed.+ end = min(now, start + _max_window())+ if end <= start:+ return None+ return start, end+++def _is_due(candidate: _Candidate, now: datetime) -> tuple[bool, str]:+ """Whether to poll this connector now, and the reason when not."""+ if candidate.blocked_reason:+ # Permanent until the connector row changes, and nothing else. Comparing+ # against the recorded value rather than clearing on a timer is what+ # makes "somebody acted" the only thing that resumes polling.+ recorded = _aware(candidate.blocked_connector_updated_at)+ current = _aware(candidate.connector_updated_at)+ if recorded is not None and current is not None and current <= recorded:+ return False, candidate.blocked_reason+ if recorded is None and current is None:+ return False, candidate.blocked_reason++ retry_after = _aware(candidate.retry_after)+ if retry_after is not None and retry_after > now:+ return False, f"the vendor asked us to wait until {retry_after.isoformat()}"++ last_run = _aware(candidate.last_run_at)+ if last_run is not None and (now - last_run) < timedelta(seconds=_min_connector_interval()):+ return False, "polled recently"+ return True, ""+++async def _load_candidates(db: AsyncSession) -> list[_Candidate]:+ rows = (await db.execute(text(_CANDIDATES_SQL), {"replayable": replayable_connector_ids()})).mappings().all()+ return [+ _Candidate(+ tenant_id=row["tenant_id"],+ connector_id=row["connector_id"],+ connector_type=row["connector_type"],+ auth_config=dict(row["auth_config"] or {}),+ connector_config=dict(row["connector_config"] or {}),+ connector_updated_at=row["connector_updated_at"],+ measuring_since=row["measuring_since"],+ watermark_at=row["watermark_at"],+ last_run_at=row["last_run_at"],+ retry_after=row["retry_after"],+ blocked_reason=row["blocked_reason"],+ blocked_connector_updated_at=row["blocked_connector_updated_at"],+ consecutive_failures=int(row["consecutive_failures"] or 0),+ )+ for row in rows+ ]+++def _retry_after_at(response: httpx.Response, now: datetime) -> datetime:+ """When the vendor said to come back, or one interval from now.++ Only the delta-seconds form is parsed. The HTTP-date form is legal and rare+ here, and a date this code misparsed would either hammer a vendor that+ asked us not to or park a connector for an arbitrary time. Falling back to+ our own interval is wrong by at most one tick in one direction.+ """+ raw = (response.headers.get("Retry-After") or "").strip()+ try:+ seconds = int(raw)+ except ValueError:+ seconds = _interval()+ return now + timedelta(seconds=max(min(seconds, 86400), 1))+++async def _poll(+ candidate: _Candidate, vendor: str, credentials: dict[str, Any], window: tuple[datetime, datetime], now: datetime+) -> ConnectorOutcome:+ """One request to the actions service, classified into one of the four states."""+ start, end = window+ payload: dict[str, Any] = {+ "vendor": vendor,+ "credentials": credentials,+ "since": start.isoformat(),+ "until": end.isoformat(),+ "limit": _limit(),+ }+ if credentials.get("search_override"):+ payload["search_override"] = credentials["search_override"]+ if credentials.get("index"):+ payload["index"] = credentials["index"]++ url = f"{actions_base_url()}/api/v1/shadow/reconcile"+ headers = {**actions_headers(), "X-AiSOC-Tenant-ID": str(candidate.tenant_id)}++ try:+ # The hop this service makes is to an internal service. The vendor hop+ # is made by the actions service against the connector URL an operator+ # configured, which in an air-gapped deployment is on the isolated+ # network by construction.+ enforce_airgap_for_url(url)+ except AirgapViolation as exc:+ raise _Halt(f"air-gap policy refuses the actions service at {actions_base_url()}: {exc}") from exc++ try:+ async with httpx.AsyncClient(timeout=_RECONCILE_TIMEOUT_S) as client:+ response = await client.post(url, headers=headers, json=payload)+ except httpx.HTTPError as exc:+ return _outcome(+ candidate,+ vendor,+ "transient",+ f"the actions service is unreachable at {actions_base_url()}: {type(exc).__name__}",+ )++ if response.status_code == 429:+ return _outcome(+ candidate,+ vendor,+ "transient",+ "the vendor asked us to slow down",+ retry_after=_retry_after_at(response, now),+ )+ if response.status_code in _PERMANENT_STATUSES:+ return _outcome(candidate, vendor, "blocked", _safe(_detail(response)))+ if response.status_code in _DEPLOYMENT_STATUSES:+ raise _Halt(f"the actions service refused the sweep with HTTP {response.status_code}: {_safe(_detail(response))}")+ if response.status_code >= 400:+ return _outcome(candidate, vendor, "transient", f"the actions service returned HTTP {response.status_code}")++ try:+ body = response.json()+ except ValueError:+ return _outcome(candidate, vendor, "transient", "the actions service returned a non-JSON body")+ if not isinstance(body, dict):+ return _outcome(candidate, vendor, "transient", "the actions service returned an unexpected body shape")++ considered = int(body.get("considered") or 0)+ matched = int(body.get("matched") or 0)+ unmatched = int(body.get("unmatched") or 0)+ latest = body.get("latest_closed_at")++ # The watermark advances to the latest close time actually seen, not to the+ # end of the window we asked for. A vendor that indexes late would otherwise+ # have those findings stepped over the moment the clock moved past them.+ watermark: datetime | None = None+ if isinstance(latest, str) and latest:+ try:+ watermark = _aware(datetime.fromisoformat(latest))+ except ValueError:+ watermark = None+ if watermark is None:+ # An empty window still has to make progress or the sweep re-reads the+ # same quiet hours forever. It advances only to the end of the window+ # less the overlap, so nothing indexed inside the overlap is skipped.+ watermark = end - _overlap()++ if considered and not matched:+ # Both halves read fine and nothing joined. That is the vendor finding+ # id not reaching `aisoc_shadow_decisions.external_id`, which is a+ # wiring fault an operator can act on, and it is reported as one rather+ # than as a successful quiet pass.+ detail = (+ f"read {considered} closure(s) and matched none. The vendor's finding id is not reaching "+ f"aisoc_shadow_decisions.external_id, so closures cannot be joined to the verdicts they grade"+ )+ elif considered:+ detail = f"matched {matched} of {considered} closure(s)"+ else:+ detail = "the vendor reported no closures in this window"++ return _outcome(+ candidate,+ vendor,+ "ok",+ detail,+ considered=considered,+ matched=matched,+ unmatched=unmatched,+ watermark_at=watermark,+ )+++def _detail(response: httpx.Response) -> str:+ try:+ payload = response.json()+ except ValueError:+ return f"the actions service returned {response.status_code}"+ if isinstance(payload, dict) and isinstance(payload.get("detail"), str) and payload["detail"].strip():+ return payload["detail"]+ return f"the actions service returned {response.status_code}"+++def _outcome(candidate: _Candidate, vendor: str, status: str, detail: str, **extra: Any) -> ConnectorOutcome:+ return ConnectorOutcome(+ tenant_id=candidate.tenant_id,+ connector_id=candidate.connector_id,+ vendor=vendor,+ status=status,+ detail=detail,+ **extra,+ )+++async def _record(db: AsyncSession, candidate: _Candidate, outcome: ConnectorOutcome, now: datetime) -> None:+ """Write the state row for one connector, including the quiet outcomes."""+ blocked = outcome.status == "blocked"+ failures = candidate.consecutive_failures + 1 if outcome.status == "transient" else 0+ await db.execute(+ text(_UPSERT_STATE_SQL),+ {+ "tenant_id": str(candidate.tenant_id),+ "connector_id": str(candidate.connector_id),+ "vendor": outcome.vendor,+ "watermark_at": outcome.watermark_at,+ "last_run_at": now,+ "last_status": outcome.status,+ "last_detail": _safe(outcome.detail),+ "considered": outcome.considered,+ "matched": outcome.matched,+ "unmatched": outcome.unmatched,+ "consecutive_failures": failures,+ "blocked_reason": _safe(outcome.detail) if blocked else None,+ "blocked_at": now if blocked else None,+ "blocked_connector_updated_at": candidate.connector_updated_at if blocked else None,+ "retry_after": outcome.retry_after,+ },+ )+++async def run_once(*, db: AsyncSession | None = None, now: datetime | None = None) -> SweepRun:+ """One sweep across every connector that is due, capped per pass."""+ now = now or datetime.now(UTC)+ own_session = db is None+ if db is None:+ db = AsyncSessionLocal()+ run = SweepRun(started_at=now)++ try:+ await assert_cross_tenant_session(db, "shadow reconciliation sweep")+ run.measuring_tenants = int((await db.execute(text(_MEASURING_TENANTS_SQL))).scalar_one() or 0)+ candidates = await _load_candidates(db)+ run.candidates = len(candidates)++ polled = 0+ for candidate in candidates:+ if polled >= _max_per_tick():+ break+ vendor = candidate.connector_type+ try:+ vendor = vendor_for(candidate.connector_type)+ due, reason = _is_due(candidate, now)+ if not due:+ # Still recorded. A connector skipped silently for an hour+ # is indistinguishable from one the sweep forgot about.+ await _record(db, candidate, _outcome(candidate, vendor, "idle", reason or "not due"), now)+ continue++ window = _window_for(candidate, now)+ if window is None:+ await _record(db, candidate, _outcome(candidate, vendor, "idle", "the window has not moved since the last pass"), now)+ continue++ # Decrypted only once the connector is actually going to be+ # polled, so a not-due connector whose vault key is missing is+ # not recorded as blocked on a pass that was never going to+ # reach its vendor.+ credentials = _credentials_for(candidate)+ outcome = await _poll(candidate, vendor, credentials, window, now)+ except _Halt:+ raise+ except (CredentialVaultError, UnsupportedConnector) as exc:+ outcome = _outcome(candidate, vendor or candidate.connector_type, "blocked", _blocked_detail(exc))+ except Exception as exc: # noqa: BLE001 - one connector must not end the pass+ logger.warning(+ "shadow_reconcile.connector_failed connector=%s err=%s detail=%s",+ _safe(candidate.connector_id, 64),+ type(exc).__name__,+ _safe(exc),+ )+ outcome = _outcome(candidate, vendor or candidate.connector_type, "transient", f"{type(exc).__name__}")++ await _record(db, candidate, outcome, now)+ run.outcomes.append(outcome)+ polled += 1++ await db.commit()+ except _Halt as exc:+ # Applies to every connector, so it is said once. Nothing is written+ # onto the per-connector rows: blaming each connector for a deployment+ # fault is how an operator ends up re-saving fifty connectors.+ await db.rollback()+ run.halted_reason = _safe(exc)+ logger.error("shadow_reconcile.halted reason=%s", run.halted_reason)+ finally:+ if own_session:+ await db.close()++ _log_pass(run)+ return run+++def _blocked_detail(exc: Exception) -> str:+ if isinstance(exc, CredentialVaultError):+ return (+ f"the stored credentials could not be decrypted ({type(exc).__name__}); the vault key that wrote "+ f"them is not in the current keyring. Re-save this connector's credentials"+ )+ return _safe(exc)+++def _credentials_for(candidate: _Candidate) -> dict[str, Any]:+ """The reader's credential keys for one connector, decrypted here.++ The vault lives in this service and nowhere else, so this is the only place+ a connector's stored secrets are readable. They are forwarded per call and+ never persisted downstream, which is the same trust model+ ``/replay/history`` and ``/connectors/{id}/normalize`` already use.+ """+ decrypted = get_vault().decrypt_dict(candidate.auth_config)+ return credentials_for(candidate.connector_type, decrypted, candidate.connector_config)+++def _log_pass(run: SweepRun) -> None:+ """Say what the pass did, including when it did nothing.++ The quiet cases are logged at info rather than skipped, because the whole+ point of this worker is that an empty scorecard should never again be the+ first place somebody learns nothing is running.+ """+ if run.halted_reason:+ return+ if run.measuring_tenants == 0:+ logger.info("shadow_reconcile idle: no tenant has shadow mode enabled, so there is nothing to reconcile")+ return+ if run.candidates == 0:+ logger.info(+ "shadow_reconcile idle: %d tenant(s) are measuring but none has an enabled connector of a type with a "+ "closed-finding reader (%s), so agreement is measured on AiSOC closures only",+ run.measuring_tenants,+ ", ".join(replayable_connector_ids()),+ )+ return+ logger.info(+ "shadow_reconcile pass tenants=%d candidates=%d polled=%d ok=%d idle=%d blocked=%d transient=%d matched=%d",+ run.measuring_tenants,+ run.candidates,+ len(run.outcomes),+ run.count("ok"),+ run.count("idle"),+ run.count("blocked"),+ run.count("transient"),+ sum(o.matched for o in run.outcomes),+ )+ for outcome in run.outcomes:+ if outcome.status == "blocked":+ logger.error(+ "shadow_reconcile blocked connector=%s vendor=%s; this will not clear on its own: %s",+ _safe(outcome.connector_id, 64),+ _safe(outcome.vendor, 32),+ _safe(outcome.detail),+ )+++async def run_forever() -> None:+ """Tick until cancelled. Owned by the API ``lifespan``, like the other workers."""+ interval = _interval()+ logger.info(+ "shadow_reconcile started interval=%ds per-connector-floor=%ds max-per-tick=%d window-cap=%dh",+ interval,+ _min_connector_interval(),+ _max_per_tick(),+ int(_max_window().total_seconds() // 3600),+ )+ failures = TickFailures("shadow_reconcile", logger)+ try:+ while True:+ try:+ await run_once()+ except asyncio.CancelledError:+ raise+ except Exception as exc: # pragma: no cover - defensive+ failures.record_failure(exc)+ else:+ failures.record_success()+ await asyncio.sleep(interval)+ except asyncio.CancelledError:+ logger.info("shadow_reconcile stopped")+ raisediff --git a/services/api/migrations/068_shadow_reconcile_state.sql b/services/api/migrations/068_shadow_reconcile_state.sqlnew file mode 100644index 000000000..c3a35bd06--- /dev/null+++ b/services/api/migrations/068_shadow_reconcile_state.sql@@ -0,0 +1,136 @@+-- 068: where the SIEM reconciliation sweep got to, and why it stopped if it did.+--+-- Gap-closure Phase 2.1, closing D15.+--+-- The sweep polls a customer's own SIEM for the closures their analysts made+-- there, so a tenant whose queue lives in Splunk ES still accumulates a track+-- record. That is bounded work against somebody else's API on a timer, and+-- three properties have to survive a restart for it to be safe to run at all.+--+-- A watermark, so a window is read once+-- -------------------------------------+-- Without one the sweep re-reads the same hours forever: every tick costs the+-- vendor a search and returns findings that were already graded. `watermark_at`+-- is the close time the last successful pass reached, and the next window+-- starts there minus a small overlap, because a vendor's search index lags its+-- own close events and a window that starts exactly where the last one ended+-- steps over anything indexed late. The overlap is safe to re-read:+-- `reconcile_findings` only writes decisions whose `resolved_at` is still null,+-- so an overlapping finding is reported as `already_resolved` rather than+-- graded twice.+--+-- A distinction between a fault that will clear and one that will not+-- -------------------------------------------------------------------+-- `blocked_reason` is set when the vendor refused the stored credentials, or+-- when the credentials cannot be decrypted at all. Neither resolves by waiting,+-- so the sweep stops polling that connector and says what an operator must do.+-- `blocked_connector_updated_at` is what unblocks it: the sweep compares it to+-- `connectors.updated_at`, so re-saving the connector resumes polling and+-- nothing else does. A timer would be the churn this column exists to avoid,+-- and a manual reset would be a support ticket.+--+-- A record of the healthy case, not only the broken one+-- -----------------------------------------------------+-- `last_status` is written on every pass including the ones with nothing to+-- do. A sweep that silently stopped and a sweep whose tenant closed nothing+-- last night are indistinguishable from outside unless the quiet case is+-- recorded, and that ambiguity is the failure this table's health endpoint+-- exists to remove.+--+-- Keyed on the connector, not the tenant+-- --------------------------------------+-- A tenant can have Splunk and Sentinel both connected, each with its own+-- index lag, its own rate limit and its own credential lifetime. One watermark+-- across both would replay whichever is slower and skip whichever is faster.++CREATE TABLE IF NOT EXISTS aisoc_shadow_reconcile_state (+ tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,+ connector_id UUID NOT NULL REFERENCES connectors(id) ON DELETE CASCADE,+ -- Denormalised from the connector so a health read does not need a join+ -- and so the row still names its vendor after the connector is deleted+ -- and the cascade has not yet run.+ vendor TEXT NOT NULL,++ -- The close time the last successful pass reached. NULL before the first+ -- one, which is how the sweep knows to start from when measurement began+ -- rather than from the beginning of the vendor's retention.+ watermark_at TIMESTAMPTZ,++ last_run_at TIMESTAMPTZ,+ -- ok | idle | blocked | transient. Written every pass, including the ones+ -- that found nothing, so "nothing to do" is a recorded state rather than+ -- an absence that looks like a stopped worker.+ last_status TEXT NOT NULL DEFAULT 'idle',+ -- One operator-facing sentence. Never an exception repr: a reason nobody+ -- can act on is the same as no reason.+ last_detail TEXT,++ -- Counters from the most recent pass, kept apart because they answer+ -- different questions. `last_matched` at zero with `last_considered` high+ -- means the vendor finding id is not reaching the decision rows, which is+ -- a wiring fault rather than a quiet week.+ last_considered INTEGER NOT NULL DEFAULT 0,+ last_matched INTEGER NOT NULL DEFAULT 0,+ last_unmatched INTEGER NOT NULL DEFAULT 0,++ -- Transient-failure run length, so a caller can tell one bad night from a+ -- fault that has outlived any explanation a retry would fix.+ consecutive_failures INTEGER NOT NULL DEFAULT 0,++ -- Set only for a condition that will not clear on its own.+ blocked_reason TEXT,+ blocked_at TIMESTAMPTZ,+ -- `connectors.updated_at` as it stood when the block was recorded. The+ -- sweep resumes when the live value differs, which is exactly "somebody+ -- changed the connector" and nothing weaker.+ blocked_connector_updated_at TIMESTAMPTZ,++ -- Honoured before the next poll. Set from a vendor's own `Retry-After`,+ -- so a SIEM that asks us to slow down is obeyed rather than estimated.+ retry_after TIMESTAMPTZ,++ updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),++ PRIMARY KEY (tenant_id, connector_id)+);++COMMENT ON TABLE aisoc_shadow_reconcile_state IS+ 'Per-connector progress of the shadow-reconciliation sweep: how far it has '+ 'read, when it last ran, and whether what stopped it was transient or needs '+ 'an operator.';+COMMENT ON COLUMN aisoc_shadow_reconcile_state.watermark_at IS+ 'Latest vendor close time successfully reconciled. The next window starts '+ 'here minus the configured overlap, never at the wall clock.';+COMMENT ON COLUMN aisoc_shadow_reconcile_state.blocked_connector_updated_at IS+ 'connectors.updated_at at the moment of the block. Polling resumes when the '+ 'live value differs, so a revoked credential stops churning and restarts on '+ 'the one event that could have fixed it.';++-- The sweep picks its next batch by "least recently run, not blocked", so that+-- is the index. Partial on the unblocked rows because a blocked connector is+-- never a candidate and carrying it in the index only widens the scan.+CREATE INDEX IF NOT EXISTS idx_shadow_reconcile_due+ ON aisoc_shadow_reconcile_state (last_run_at NULLS FIRST)+ WHERE blocked_reason IS NULL;++-- Row-level security. The `OR current_tenant_id() IS NULL` arm is what lets the+-- cross-tenant sweep read every tenant's row; dropping it would make the sweep+-- silently see nothing, which here means every tenant's scorecard quietly+-- stopping with no error anywhere. FORCE so the owner does not walk past.+ALTER TABLE aisoc_shadow_reconcile_state ENABLE ROW LEVEL SECURITY;+ALTER TABLE aisoc_shadow_reconcile_state FORCE ROW LEVEL SECURITY;+DROP POLICY IF EXISTS aisoc_shadow_reconcile_state_tenant ON aisoc_shadow_reconcile_state;+CREATE POLICY aisoc_shadow_reconcile_state_tenant ON aisoc_shadow_reconcile_state+ USING (tenant_id = current_tenant_id() OR current_tenant_id() IS NULL);++-- 061 set ALTER DEFAULT PRIVILEGES so new tables pick these up, but only for+-- tables created by the role that ran it. Granting explicitly means a chain+-- replayed by a different owner still leaves the runtime role able to work.+DO $$+BEGIN+ IF EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'aisoc_app') THEN+ GRANT SELECT, INSERT, UPDATE, DELETE ON aisoc_shadow_reconcile_state TO aisoc_app;+ END IF;+END+$$;diff --git a/services/api/tests/test_shadow_reconcile_sweep.py b/services/api/tests/test_shadow_reconcile_sweep.pynew file mode 100644index 000000000..8003c0592--- /dev/null+++ b/services/api/tests/test_shadow_reconcile_sweep.py@@ -0,0 +1,586 @@+"""The sweep that gives ``shadow_reconcile`` a schedule.++Gap-closure Phase 2.1, closing D15.++The defect being closed is precisely "exists and nothing calls it": the+matcher was complete, had eleven tests, and had no caller, so a tenant whose+analysts close their queue in their own SIEM accumulated no track record and+the only symptom was a scorecard that never filled in. A test asserting the+sweep is registered by the *deployed* lifespan rather than by one this file+built is therefore the point of the file, and it is the last test here.++The rest cover the properties a scheduled sweep against somebody else's API+has to have: it must tell a condition that will never resolve from one that+might, it must not re-read the same window forever, it must not poll faster+than it said it would, and it must record the quiet passes so that a sweep+which stopped is distinguishable from one with nothing to do.+"""++from __future__ import annotations++import ast+import uuid+from datetime import UTC, datetime, timedelta+from pathlib import Path+from typing import Any++import httpx+import pytest+from app.db.cross_tenant import CrossTenantPreconditionError+from app.security.credential_vault import CredentialVaultError+from app.workers import shadow_reconcile+from app.workers.shadow_reconcile import run_once++TENANT = uuid.UUID("11111111-1111-1111-1111-111111111111")+CONNECTOR = uuid.UUID("aaaaaaaa-0000-0000-0000-000000000001")+OTHER_CONNECTOR = uuid.UUID("aaaaaaaa-0000-0000-0000-000000000002")++NOW = datetime(2026, 9, 20, 12, 0, tzinfo=UTC)+++def _row(**overrides: Any) -> dict[str, Any]:+ """One candidate as the SQL returns it."""+ row: dict[str, Any] = {+ "connector_id": CONNECTOR,+ "tenant_id": TENANT,+ "connector_type": "splunk",+ "auth_config": {"base_url": "https://splunk.example.com:8089", "token": "vault:v1:x"},+ "connector_config": {"saved_search": "my notables"},+ "connector_updated_at": NOW - timedelta(days=30),+ "measuring_since": NOW - timedelta(days=3),+ "watermark_at": None,+ "last_run_at": None,+ "retry_after": None,+ "blocked_reason": None,+ "blocked_connector_updated_at": None,+ "consecutive_failures": 0,+ }+ row.update(overrides)+ return row+++class _Result:+ def __init__(self, value: Any) -> None:+ self._value = value++ def scalar_one(self) -> Any:+ return self._value++ def mappings(self) -> _Result:+ return self++ def all(self) -> Any:+ return self._value if isinstance(self._value, list) else []+++class FakeSession:+ """Records every statement, so what the sweep wrote can be asserted."""++ def __init__(self, rows: list[dict[str, Any]], *, measuring: int = 1, bound_tenant: str | None = None) -> None:+ self._rows = rows+ self._measuring = measuring+ self._bound_tenant = bound_tenant+ self.statements: list[tuple[str, dict[str, Any]]] = []+ self.committed = False+ self.rolled_back = False++ async def scalar(self, stmt: Any, params: dict[str, Any] | None = None) -> Any:+ self.statements.append((str(stmt), dict(params or {})))+ return self._bound_tenant++ async def execute(self, stmt: Any, params: dict[str, Any] | None = None) -> Any:+ sql = str(stmt)+ self.statements.append((sql, dict(params or {})))+ if "COUNT(DISTINCT tenant_id)" in sql:+ return _Result(self._measuring)+ if "FROM connectors c" in sql:+ return _Result(self._rows)+ return _Result(None)++ async def commit(self) -> None:+ self.committed = True++ async def rollback(self) -> None:+ self.rolled_back = True++ async def close(self) -> None: # pragma: no cover - session is injected+ pass++ def writes(self) -> list[dict[str, Any]]:+ return [p for sql, p in self.statements if "INSERT INTO aisoc_shadow_reconcile_state" in sql]++[email protected](autouse=True)+def _vault(monkeypatch: pytest.MonkeyPatch) -> None:+ """A vault that decrypts. Tests that want a missing key override this."""++ class _Vault:+ def decrypt_dict(self, value: dict[str, Any]) -> dict[str, Any]:+ return {k: (v.replace("vault:v1:", "") if isinstance(v, str) else v) for k, v in value.items()}++ monkeypatch.setattr(shadow_reconcile, "get_vault", lambda: _Vault())+++def _respond(monkeypatch: pytest.MonkeyPatch, response: httpx.Response | BaseException) -> list[dict[str, Any]]:+ """Stand in for the actions service. Returns the payloads it was sent."""+ sent: list[dict[str, Any]] = []++ class _Client:+ def __init__(self, **_kw: Any) -> None:+ pass++ async def __aenter__(self) -> _Client:+ return self++ async def __aexit__(self, *_exc: Any) -> None:+ return None++ async def post(self, url: str, headers: dict[str, str] | None = None, json: Any = None) -> httpx.Response:+ sent.append({"url": url, "headers": headers or {}, "json": json})+ if isinstance(response, BaseException):+ raise response+ return response++ monkeypatch.setattr(shadow_reconcile.httpx, "AsyncClient", _Client)+ return sent+++def _ok(**body: Any) -> httpx.Response:+ payload = {+ "vendor": "splunk",+ "count": 0,+ "labelled": 0,+ "unlabeled": 0,+ "considered": 0,+ "matched": 0,+ "unmatched": 0,+ "already_resolved": 0,+ "skipped_no_key": 0,+ "latest_closed_at": None,+ }+ payload.update(body)+ return httpx.Response(200, json=payload, request=httpx.Request("POST", "http://actions/api/v1/shadow/reconcile"))+++def _refused(code: int, detail: str, headers: dict[str, str] | None = None) -> httpx.Response:+ return httpx.Response(+ code,+ json={"detail": detail},+ headers=headers or {},+ request=httpx.Request("POST", "http://actions/api/v1/shadow/reconcile"),+ )+++class TestItActuallyPolls:+ @pytest.mark.asyncio+ async def test_a_due_connector_is_polled_and_its_watermark_advances(self, monkeypatch: pytest.MonkeyPatch) -> None:+ closed_at = (NOW - timedelta(minutes=20)).isoformat()+ sent = _respond(monkeypatch, _ok(count=4, considered=4, matched=3, unmatched=1, latest_closed_at=closed_at))+ db = FakeSession([_row()])++ run = await run_once(db=db, now=NOW)++ assert run.count("ok") == 1+ assert len(sent) == 1+ assert sent[0]["url"].endswith("/api/v1/shadow/reconcile")+ assert sent[0]["json"]["vendor"] == "splunk"+ write = db.writes()[0]+ assert write["last_status"] == "ok"+ assert write["matched"] == 3+ # Advanced to the latest closure seen, not to the end of the window.+ assert write["watermark_at"] == datetime.fromisoformat(closed_at)++ @pytest.mark.asyncio+ async def test_the_tenant_travels_on_the_header_not_in_the_body(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """The receiving route takes its tenant from the credential.++ A tenant in the body would be a value the caller chose that nothing+ checked, which is the shape every cross-tenant leak in this codebase+ has had.+ """+ sent = _respond(monkeypatch, _ok())++ await run_once(db=FakeSession([_row()]), now=NOW)++ assert sent[0]["headers"]["X-AiSOC-Tenant-ID"] == str(TENANT)+ assert not [k for k in sent[0]["json"] if "tenant" in k]++ @pytest.mark.asyncio+ async def test_credentials_are_translated_into_the_readers_key_names(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """The connector stores ``base_url``; the reader wants ``splunk_url``.++ A key that lands under the wrong name does not raise. The factory+ returns None and the read is refused as "no usable credentials", which+ reads to an operator as a problem on their side.+ """+ sent = _respond(monkeypatch, _ok())++ await run_once(db=FakeSession([_row()]), now=NOW)++ credentials = sent[0]["json"]["credentials"]+ assert credentials["splunk_url"] == "https://splunk.example.com:8089"+ assert credentials["splunk_token"] == "x"+ # The saved search a deployment renamed reaches the reader too.+ assert sent[0]["json"]["search_override"] == "my notables"+++class TestPermanentIsNotTransient:+ """The property the whole design rests on, asserted from both sides."""++ @pytest.mark.asyncio+ async def test_a_refused_credential_blocks_and_names_the_action(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _respond(+ monkeypatch,+ _refused(422, "splunk: the vendor refused the stored credentials with HTTP 401. Re-authorise this connector in Settings"),+ )+ db = FakeSession([_row()])++ run = await run_once(db=db, now=NOW)++ assert run.count("blocked") == 1+ write = db.writes()[0]+ assert write["last_status"] == "blocked"+ assert "Re-authorise" in write["blocked_reason"]+ # Recorded against the connector as it stands now, which is what makes+ # re-saving it the one event that resumes polling.+ assert write["blocked_connector_updated_at"] == NOW - timedelta(days=30)++ @pytest.mark.asyncio+ async def test_a_blocked_connector_is_not_polled_again(self, monkeypatch: pytest.MonkeyPatch) -> None:+ sent = _respond(monkeypatch, _ok())+ db = FakeSession(+ [+ _row(+ blocked_reason="splunk: the vendor refused the stored credentials with HTTP 401",+ blocked_connector_updated_at=NOW - timedelta(days=30),+ last_run_at=NOW - timedelta(days=2),+ )+ ]+ )++ run = await run_once(db=db, now=NOW)++ assert sent == [], "a revoked credential must not be retried on a timer"+ assert run.count("idle") == 0, "a blocked connector is not counted as polled"+ # Still written, so the state is visible rather than merely absent.+ assert db.writes()[0]["last_status"] == "idle"++ @pytest.mark.asyncio+ async def test_saving_the_connector_resumes_polling(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """The one event that could have fixed it is the only one that clears it."""+ sent = _respond(monkeypatch, _ok())+ db = FakeSession(+ [+ _row(+ blocked_reason="splunk: the vendor refused the stored credentials with HTTP 401",+ blocked_connector_updated_at=NOW - timedelta(days=30),+ connector_updated_at=NOW - timedelta(minutes=1),+ last_run_at=NOW - timedelta(days=2),+ )+ ]+ )++ run = await run_once(db=db, now=NOW)++ assert len(sent) == 1+ assert run.count("ok") == 1+ assert db.writes()[0]["blocked_reason"] is None++ @pytest.mark.asyncio+ async def test_an_undecryptable_credential_blocks_before_the_vendor_is_called(self, monkeypatch: pytest.MonkeyPatch) -> None:+ class _Vault:+ def decrypt_dict(self, _value: dict[str, Any]) -> dict[str, Any]:+ raise CredentialVaultError("no key in the keyring")++ monkeypatch.setattr(shadow_reconcile, "get_vault", lambda: _Vault())+ sent = _respond(monkeypatch, _ok())+ db = FakeSession([_row()])++ run = await run_once(db=db, now=NOW)++ assert sent == []+ assert run.count("blocked") == 1+ assert "Re-save this connector" in db.writes()[0]["blocked_reason"]++ @pytest.mark.asyncio+ async def test_an_unreachable_actions_service_is_transient(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _respond(monkeypatch, httpx.ConnectError("refused"))+ db = FakeSession([_row()])++ run = await run_once(db=db, now=NOW)++ assert run.count("transient") == 1+ write = db.writes()[0]+ assert write["last_status"] == "transient"+ assert write["blocked_reason"] is None, "a transient fault must never block"+ assert write["consecutive_failures"] == 1++ @pytest.mark.asyncio+ async def test_consecutive_transient_failures_accumulate(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """So a fault that has outlived a transient explanation is visible as one."""+ _respond(monkeypatch, httpx.ReadTimeout("slow"))+ db = FakeSession([_row(consecutive_failures=6, last_run_at=NOW - timedelta(days=1))])++ await run_once(db=db, now=NOW)++ assert db.writes()[0]["consecutive_failures"] == 7++ @pytest.mark.asyncio+ async def test_a_transient_failure_leaves_the_watermark_alone(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """A failure must not look like progress and must not skip a window."""+ _respond(monkeypatch, httpx.ConnectError("refused"))+ db = FakeSession([_row(watermark_at=NOW - timedelta(hours=2), last_run_at=NOW - timedelta(days=1))])++ await run_once(db=db, now=NOW)++ assert db.writes()[0]["watermark_at"] is None, "the upsert keeps the stored value when the pass wrote none"++ @pytest.mark.asyncio+ async def test_a_deployment_refusal_halts_the_pass_instead_of_blaming_every_connector(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """A wrong service token is not fifty connectors each needing a re-save."""+ _respond(monkeypatch, _refused(401, "invalid or missing service token"))+ db = FakeSession([_row(), _row(connector_id=OTHER_CONNECTOR)])++ run = await run_once(db=db, now=NOW)++ assert run.halted_reason is not None+ assert "401" in run.halted_reason+ assert db.rolled_back is True+ assert db.writes() == [] or not db.committed+++class TestRateLimitingAndTheWatermark:+ @pytest.mark.asyncio+ async def test_a_vendors_retry_after_is_stored_and_honoured(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _respond(monkeypatch, _refused(429, "splunk asked us to slow down", {"Retry-After": "600"}))+ db = FakeSession([_row()])++ await run_once(db=db, now=NOW)++ assert db.writes()[0]["retry_after"] == NOW + timedelta(seconds=600)++ @pytest.mark.asyncio+ async def test_a_connector_inside_its_retry_after_is_not_polled(self, monkeypatch: pytest.MonkeyPatch) -> None:+ sent = _respond(monkeypatch, _ok())+ db = FakeSession([_row(retry_after=NOW + timedelta(minutes=5), last_run_at=NOW - timedelta(days=1))])++ await run_once(db=db, now=NOW)++ assert sent == []+ assert "asked us to wait" in db.writes()[0]["last_detail"]++ @pytest.mark.asyncio+ async def test_a_recently_polled_connector_is_left_alone(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """The per-connector floor, so a short tick cadence is not a poll storm."""+ sent = _respond(monkeypatch, _ok())+ db = FakeSession([_row(last_run_at=NOW - timedelta(minutes=5))])++ await run_once(db=db, now=NOW)++ assert sent == []+ assert db.writes()[0]["last_detail"] == "polled recently"++ @pytest.mark.asyncio+ async def test_the_window_starts_at_the_watermark_less_the_overlap(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """A vendor's search index lags its own close events.++ A window beginning exactly where the last one ended steps over anything+ indexed late, and the overlap is free of consequence because an+ already-graded finding comes back as ``already_resolved``.+ """+ sent = _respond(monkeypatch, _ok())+ watermark = NOW - timedelta(hours=2)+ db = FakeSession([_row(watermark_at=watermark, last_run_at=NOW - timedelta(days=1))])++ await run_once(db=db, now=NOW)++ since = datetime.fromisoformat(sent[0]["json"]["since"])+ assert since == watermark - timedelta(seconds=shadow_reconcile.settings.SHADOW_RECONCILE_OVERLAP_SECONDS)++ @pytest.mark.asyncio+ async def test_a_first_pass_starts_when_measurement_started(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """Reaching back further would grade closures with nothing to grade them against."""+ sent = _respond(monkeypatch, _ok())+ measuring_since = NOW - timedelta(days=2)+ db = FakeSession([_row(measuring_since=measuring_since)])++ await run_once(db=db, now=NOW)++ assert datetime.fromisoformat(sent[0]["json"]["since"]) == measuring_since++ @pytest.mark.asyncio+ async def test_one_window_is_capped_so_a_long_gap_catches_up_in_steps(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """Asking a customer's SIEM for a month in one search is the request+ least likely to succeed, so it is never made."""+ sent = _respond(monkeypatch, _ok())+ db = FakeSession([_row(measuring_since=NOW - timedelta(days=90))])++ await run_once(db=db, now=NOW)++ since = datetime.fromisoformat(sent[0]["json"]["since"])+ until = datetime.fromisoformat(sent[0]["json"]["until"])+ assert until - since <= timedelta(hours=shadow_reconcile.settings.SHADOW_RECONCILE_MAX_WINDOW_HOURS)+ assert until < NOW, "the pass stops short of now and resumes next tick"++ @pytest.mark.asyncio+ async def test_an_empty_window_still_advances_the_watermark(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """Otherwise the sweep re-reads the same quiet hours forever."""+ _respond(monkeypatch, _ok())+ db = FakeSession([_row(watermark_at=NOW - timedelta(hours=4), last_run_at=NOW - timedelta(days=1))])++ await run_once(db=db, now=NOW)++ advanced = db.writes()[0]["watermark_at"]+ assert advanced is not None+ assert advanced > NOW - timedelta(hours=4)+ # But not past the overlap, so nothing indexed inside it is skipped.+ assert advanced <= NOW - timedelta(seconds=shadow_reconcile.settings.SHADOW_RECONCILE_OVERLAP_SECONDS)++ @pytest.mark.asyncio+ async def test_a_pass_is_capped_per_tick(self, monkeypatch: pytest.MonkeyPatch) -> None:+ sent = _respond(monkeypatch, _ok())+ monkeypatch.setattr(shadow_reconcile, "_max_per_tick", lambda: 2)+ rows = [_row(connector_id=uuid.UUID(int=i)) for i in range(1, 6)]++ run = await run_once(db=FakeSession(rows), now=NOW)++ assert len(sent) == 2+ assert run.candidates == 5+++class TestTheQuietCasesAreRecorded:+ """A sweep that stopped and a sweep with nothing to do look identical from+ outside unless the quiet case is written down."""++ @pytest.mark.asyncio+ async def test_no_measuring_tenant_is_a_recorded_state_not_an_error(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _respond(monkeypatch, _ok())++ run = await run_once(db=FakeSession([], measuring=0), now=NOW)++ assert run.measuring_tenants == 0+ assert run.candidates == 0+ assert run.halted_reason is None++ @pytest.mark.asyncio+ async def test_measuring_with_no_replayable_connector_is_distinguishable(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """Somebody asked to be measured and has nothing this can poll.++ Reported apart from "nobody asked", because only one of the two is+ worth telling an operator about.+ """+ _respond(monkeypatch, _ok())++ run = await run_once(db=FakeSession([], measuring=3), now=NOW)++ assert run.measuring_tenants == 3+ assert run.candidates == 0++ @pytest.mark.asyncio+ async def test_reading_closures_and_matching_none_is_reported_as_a_wiring_fault(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """Not as a successful quiet pass. The join key is not arriving."""+ _respond(monkeypatch, _ok(count=12, considered=12, matched=0, unmatched=12))+ db = FakeSession([_row()])++ await run_once(db=db, now=NOW)++ detail = db.writes()[0]["last_detail"]+ assert "matched none" in detail+ assert "external_id" in detail++ @pytest.mark.asyncio+ async def test_the_sweep_refuses_a_session_bound_to_one_tenant(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """A cross-tenant sweep that quietly sees one tenant is worse than one+ that stops, because it reports success."""+ _respond(monkeypatch, _ok())+ db = FakeSession([_row()], bound_tenant=str(TENANT))++ with pytest.raises(CrossTenantPreconditionError):+ await run_once(db=db, now=NOW)+++class TestOnlyReplayableConnectorsAreAsked:+ def test_the_candidate_query_filters_on_the_shared_vendor_table(self) -> None:+ """One definition of "has a reader", shared with replay evaluation.++ A second list here would go stale the first time a sixth reader landed,+ and the symptom would be a tenant whose connector is supported and is+ never polled.+ """+ from app.services.replay_evaluation.vendors import replayable_connector_ids # noqa: PLC0415++ assert ":replayable" in shadow_reconcile._CANDIDATES_SQL+ assert replayable_connector_ids(), "the shared table is empty; this test is stale"++ def test_only_tenants_with_shadow_mode_on_are_candidates(self) -> None:+ assert "aisoc_shadow_mode" in shadow_reconcile._CANDIDATES_SQL+ assert "enabled IS TRUE" in shadow_reconcile._CANDIDATES_SQL+++def test_the_deployed_lifespan_registers_the_sweep() -> None:+ """The failure this whole change closes is "exists and nothing calls it".++ So the assertion is against ``app/main.py`` as it ships, not against a+ scheduler a test assembled. Read off the source with an AST pass rather+ than imported, for the same reason the Phase 1.2 sink test is: importing+ ``app.main`` pulls in the service's whole startup graph, and a test that+ has to boot the service to check a registration gets deleted the first+ time it is slow.++ Three things are checked, because any one of them alone would pass over a+ worker that never runs: the module is imported, a task is created for it,+ and the task's ``worker=`` is the sweep's ``run_forever`` rather than+ something that merely mentions it.+ """+ source = Path(__file__).resolve().parents[1] / "app" / "main.py"+ tree = ast.parse(source.read_text(encoding="utf-8"))++ imported = {+ alias.asname or alias.name+ for node in ast.walk(tree)+ if isinstance(node, ast.ImportFrom) and node.module == "app.workers.shadow_reconcile"+ for alias in node.names+ }+ assert imported, "app/main.py no longer imports the shadow-reconciliation worker"++ guarded = [+ call+ for call in ast.walk(tree)+ if isinstance(call, ast.Call) and isinstance(call.func, ast.Name) and call.func.id == "_run_guarded_scheduler_worker"+ ]+ assert guarded, "app/main.py no longer registers guarded scheduler workers; this test is stale"++ registered = [+ call+ for call in guarded+ for kw in call.keywords+ if kw.arg == "worker" and isinstance(kw.value, ast.Name) and kw.value.id in imported+ ]+ assert registered, (+ "the shadow-reconciliation sweep is imported by app/main.py but never passed to "+ "_run_guarded_scheduler_worker, so nothing would call it on a schedule. That is the exact "+ "defect D15 recorded, one layer up."+ )++ job_names = {+ kw.value.value for call in registered for kw in call.keywords if kw.arg == "job_name" and isinstance(kw.value, ast.Constant)+ }+ assert "shadow_reconcile" in job_names, (+ f"the sweep is registered under {job_names or 'no job name'}; the Redis scheduler lease is keyed on "+ f"the job name, so two names means two replicas can sweep the same vendor at once"+ )+++def test_the_sweep_is_off_by_default_and_says_so() -> None:+ """A feature that reaches a third party's API ships off.++ Paired with the assertion that the disabled case is announced, because the+ state this change closes is an operator reading an empty scorecard and+ being unable to tell a sweep that is off from one that is broken.+ """+ from app.core.config import Settings # noqa: PLC0415++ assert Settings.model_fields["SHADOW_RECONCILE_ENABLED"].default is False++ source = (Path(__file__).resolve().parents[1] / "app" / "main.py").read_text(encoding="utf-8")+ assert "shadow_reconcile worker disabled" in source