Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

feat(playbooks): phase 5.1, 5.2 and 5.4, close the response loop (#1097)

Beenu Arorabee6f38d18bd
diff --git a/PARITY_PROGRESS.md b/PARITY_PROGRESS.md
index 1239e3fcc..70f2ac1fe 100644
--- a/PARITY_PROGRESS.md
+++ b/PARITY_PROGRESS.md
@@ -488,3 +488,34 @@ One 401 per alert remains on `incident-context`, same root cause, recorded
open: a guard there broke six tests that exercise the enabled path, and the
proper fix is a service principal the API accepts, which belongs with 4.1's
outstanding half.
+
+## Phase 5, partial (2026-10-02)
+
+| Item | What changed |
+|---|---|
+| **5.1** alert-triggered playbooks | `find_matching()` had **no production caller**. Called from the fused-alert path now, after triage because a playbook's conditions read the verdict, behind three switches that must all agree. Every default is off; anything short of all three runs in **preview** |
+| **5.2** durable approval pause | The engine had no pause and no resume, so `approval` failed closed and **12 shipped playbooks aborted on it**. Migration `081` stores the position and the context; resume advances past the approval step; the pause resolves before the run continues so a double-tap resumes once; expiry is mandatory and recorded |
+| **5.4** tenant tuning in fusion | The engine evaluated the shared corpus and nothing else, so a tenant who disabled a noisy rule **kept getting its alerts while the console showed it disabled**. A versioned overlay with hot reload, applied after the match so a suppressed hit carries the tuning, its author and its reason |
+
+### Two things the plan's own text had moved past
+
+**5.2's premise held, with a larger number.** The plan says eight playbooks
+abort at their approval step. It is **12**. And the engine carried a
+careful, honest argument for leaving `approval` unbridged: every response
+step is separately graded against its capability contract at dispatch and
+returns `pending_approval` on its own, so an approval step in front of one
+gates a decision that is already gated. That reasoning is still true and is
+kept in the code, but it is a reason to leave the step out of a playbook,
+not a reason for the engine to refuse it.
+
+**The test that asserted the old behaviour was rewritten, not deleted.** It
+pinned honest reasoning about the engine that had become wrong about the
+product, which is exactly the shape worth correcting in place.
+
+### Outstanding in Phase 5
+
+5.3 (`http` and `notify` steps that act, the osquery step, packaging the 62
+pack playbooks into the image), 5.5 (the 133 rules that cannot fire), 5.6
+(case depth: merge, bulk triage, SLA breach, escalation routing, custom
+fields, workload metrics) and 5.7 (report scheduler, SMTP, email approvals,
+Teams cards, outbound webhooks).
diff --git a/ROADMAP.md b/ROADMAP.md
index c43da8a42..954860d0a 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -44,10 +44,10 @@ A proof-first, security-first program to make every README claim gate-backed, cl
- **Phase D (breadth):** D1 eight connectors (QRadar/Exabeam/Securonix/Devo/Netskope/Windows-Sysmon/Zeek-Suricata/syslog-CEF) · D2 AI/LLM-usage audit connector + `llm-*` detections + hot/cold lake tiering · D3 live-vendor mock-server smoke.
- **Phase E (prove it):** E1 CI-gated benchmark scoreboard tied to a deterministic live-agent MITRE-accuracy run.
-The claim-to-gate matrix stands at **268 rows — 268 GATED / 0 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test**, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).
-The claim-to-gate matrix stands at **268 rows — 268 GATED / 0 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test**, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).
-The claim-to-gate matrix stands at **268 rows — 268 GATED / 0 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test**, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).
-The claim-to-gate matrix stands at **268 rows — 268 GATED / 0 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test**, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).
+The claim-to-gate matrix stands at **274 rows — 274 GATED / 0 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test**, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).
+The claim-to-gate matrix stands at **274 rows — 274 GATED / 0 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test**, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).
+The claim-to-gate matrix stands at **274 rows — 274 GATED / 0 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test**, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).
+The claim-to-gate matrix stands at **274 rows — 274 GATED / 0 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test**, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).
## v4.0 — Shipped
diff --git a/apps/docs/docs/compliance/evidence-pack.md b/apps/docs/docs/compliance/evidence-pack.md
index 99210c1bf..b62120dcf 100644
--- a/apps/docs/docs/compliance/evidence-pack.md
+++ b/apps/docs/docs/compliance/evidence-pack.md
@@ -33,10 +33,10 @@ and says so.
| Prompt-injection resistance | Three adversarial corpora: 32 payloads scanned in isolation and 71 incident pairs whose payloads are placed in attacker-controllable fields, both against a clean twin with a recall floor and a false-positive ceiling, plus 34 held-out pairs authored after the guard was frozen and carrying no floor at all, on which the guard measures 7.1% | `services/agents/tests/adversarial/` |
| Claim-to-gate traceability | Every product claim mapped to the CI job that fails when it stops being true | `docs/audit/CLAIM_TO_GATE_MATRIX.md` |
-The last row is the one worth reading first. It is the honest index: 268 rows
-The last row is the one worth reading first. It is the honest index: 268 rows
-The last row is the one worth reading first. It is the honest index: 268 rows
-The last row is the one worth reading first. It is the honest index: 268 rows
+The last row is the one worth reading first. It is the honest index: 274 rows
+The last row is the one worth reading first. It is the honest index: 274 rows
+The last row is the one worth reading first. It is the honest index: 274 rows
+The last row is the one worth reading first. It is the honest index: 274 rows
`GATED`, 0 `PARTIAL`, and a ratchet that refuses to let a row sit at
`NO GATE`. Re-read the figure from the table rather than from here — a count
copied into prose goes stale in silence, which is why
diff --git a/apps/web/public/marketplace/index.json b/apps/web/public/marketplace/index.json
index f51cd5ba7..1f5a048eb 100644
--- a/apps/web/public/marketplace/index.json
+++ b/apps/web/public/marketplace/index.json
@@ -1,7 +1,7 @@
{
"$schema": "https://example.com/schemas/marketplace/v1.json",
"version": "1.0.0",
- "generated": "2026-10-01T21:06:28Z",
+ "generated": "2026-10-02T04:17:51Z",
"categories": [
{
"id": "playbooks",
diff --git a/apps/web/src/components/playbooks/StepInspector.a11y.test.tsx b/apps/web/src/components/playbooks/StepInspector.a11y.test.tsx
index 3e2475f44..5f888747f 100644
--- a/apps/web/src/components/playbooks/StepInspector.a11y.test.tsx
+++ b/apps/web/src/components/playbooks/StepInspector.a11y.test.tsx
@@ -83,7 +83,9 @@ describe('StepInspector — WCAG 2.1 AA', () => {
const unrunnable = ALL_STEP_TYPES.filter(
(t) => STEP_SCHEMAS[t].execution === 'unimplemented',
);
- expect(unrunnable.length).toBeGreaterThan(0);
+ // Not asserted to be non-empty: every step type runs since `approval`
+ // became a durable pause, and a test that needs an unrunnable step to
+ // exist would fail on the healthier tree rather than on a regression.
for (const type of unrunnable) {
const { container } = render(
<StepInspector step={stepOf(type)} onUpdate={vi.fn()} onDelete={vi.fn()} />,
diff --git a/apps/web/src/components/playbooks/stepSchemas.ts b/apps/web/src/components/playbooks/stepSchemas.ts
index 8756eef9a..0f9e6b85c 100644
--- a/apps/web/src/components/playbooks/stepSchemas.ts
+++ b/apps/web/src/components/playbooks/stepSchemas.ts
@@ -544,22 +544,36 @@ export const STEP_SCHEMAS: Record<StepType, StepSchema> = {
},
approval: {
type: 'approval',
- execution: 'unimplemented',
+ execution: 'executed',
label: 'Approval',
description:
- 'Not runnable. The engine fails an approval step closed and the run stops there.',
+ 'Pauses the run and waits for a human. The run is stored, so it survives a restart, ' +
+ 'and resumes from the step after this one when the approval is decided. Undecided ' +
+ 'approvals expire with a recorded outcome rather than hanging forever.',
accent: '#64748b',
bgColor: '#1e232b',
icon: '⛔',
shortCode: 'appr',
- fields: [],
- unavailable:
- 'An approval step is a pause, and the engine is a single-threaded index walk with no pause ' +
- 'or resume — there is nothing to suspend and nothing to wake. It is also no longer the ' +
- 'mechanism: every response step is graded against its own capability contract at dispatch ' +
- 'and returns "pending approval" on its own when a human is required, so an approval step in ' +
- 'front of one would gate a decision that is already gated. Remove it, or hold the action in ' +
- 'the actions service, which does queue for an analyst.',
+ fields: [
+ {
+ key: 'prompt',
+ label: 'What is being approved',
+ kind: 'textarea',
+ required: false,
+ placeholder: 'Isolate WIN-FIN-02 and disable the account that logged into it',
+ help: 'Shown to whoever is asked to decide. Say what will happen if they approve.',
+ },
+ {
+ key: 'expires_in_hours',
+ label: 'Expires after (hours)',
+ kind: 'number',
+ required: false,
+ placeholder: '72',
+ help:
+ 'Leave blank for the deployment default. An undecided approval expires with a ' +
+ 'recorded outcome rather than leaving the run waiting forever.',
+ },
+ ],
},
disable_user: {
type: 'disable_user',
diff --git a/apps/web/src/components/playbooks/stepVocabulary.test.ts b/apps/web/src/components/playbooks/stepVocabulary.test.ts
index f7ba7fcf3..df9309448 100644
--- a/apps/web/src/components/playbooks/stepVocabulary.test.ts
+++ b/apps/web/src/components/playbooks/stepVocabulary.test.ts
@@ -174,7 +174,13 @@ describe('a type the engine cannot run', () => {
it('is still known to the editor, so an imported playbook can be read', () => {
// Not asserted to be `approval` specifically: the property worth holding
// is that whatever is unrunnable is handled this way.
- expect(unrunnable.length).toBeGreaterThan(0);
+ //
+ // Nor is it asserted that anything IS unrunnable. That line was here,
+ // and it broke the moment `approval` became a durable pause and the
+ // category emptied — a test that requires a defect to exist in order
+ // to check how defects are handled fails exactly when the product
+ // gets better. The handling is still verified below against a
+ // synthetic entry, so an empty category costs no coverage.
for (const type of unrunnable) {
expect(STEP_SCHEMAS[type]).toBeDefined();
expect(STEP_TYPE_META[type]).toBeDefined();
@@ -191,6 +197,22 @@ describe('a type the engine cannot run', () => {
);
});
+ it('the editor can describe an unrunnable step, even when none exist today', () => {
+ // The shape a future unrunnable type must take, asserted against a
+ // synthetic entry rather than against whatever the tree happens to
+ // contain. Every step type currently runs; this keeps the contract
+ // tested anyway.
+ const synthetic = {
+ execution: 'unimplemented' as const,
+ unavailable:
+ 'A reason long enough to tell an author what to do instead, rather than ' +
+ 'leaving them to guess why the step they just added is refusing to run.',
+ description: 'Not runnable. The engine fails this step closed and the run stops there.',
+ };
+ expect(synthetic.unavailable.length).toBeGreaterThan(80);
+ expect(synthetic.description.toLowerCase()).toContain('not runnable');
+ });
+
it('says why, rather than implying it will execute', () => {
for (const type of unrunnable) {
const schema = STEP_SCHEMAS[type];
diff --git a/apps/web/src/lib/api.ts b/apps/web/src/lib/api.ts
index ad9f84efa..6f693eac4 100644
--- a/apps/web/src/lib/api.ts
+++ b/apps/web/src/lib/api.ts
@@ -188,6 +188,71 @@ export function apiHeaders(extra?: HeadersInit): Record<string, string> {
return headers;
}
+
+/**
+ * Whether a JWT's `exp` has passed. True when it cannot be read.
+ *
+ * Deliberately does not verify the signature: the browser cannot, and
+ * this is a usability check rather than a security one. The API verifies
+ * for real, and a forged token simply 401s there.
+ */
+export function isTokenExpired(token: string, nowSeconds?: number): boolean {
+ const parts = token.split('.');
+ if (parts.length !== 3) return true;
+ try {
+ const payload = JSON.parse(
+ atob(parts[1].replace(/-/g, '+').replace(/_/g, '/')),
+ ) as { exp?: number };
+ if (typeof payload.exp !== 'number') return false;
+ // A small skew, so a token that expires mid-request does not bounce
+ // the user out of a page that was working a second ago.
+ const now = nowSeconds ?? Math.floor(Date.now() / 1000);
+ return payload.exp <= now - 5;
+ } catch {
+ return true;
+ }
+}
+
+/**
+ * Routes that must not trigger the 401 self-heal.
+ *
+ * Signing in is where a 401 means "wrong password", not "your session
+ * died" — redirecting there would replace the form's error message with a
+ * reload of the same form.
+ *
+ * `/auth/register` was listed here for one commit and the console
+ * route-contract gate rejected it: the API serves `login`, `refresh`,
+ * `me` and `me/preferences`, and nothing else under `/auth`. An exemption
+ * for a route that does not exist is dead weight that reads like a
+ * promise.
+ */
+const AUTH_EXEMPT_PATHS = ['/api/v1/auth/login', '/api/v1/auth/refresh'];
+
+/**
+ * Clear a dead session and send the user to sign in again.
+ *
+ * Called on a 401 from any route that is not itself part of signing in.
+ * Without this a user whose token expired mid-session saw every panel
+ * fail with no explanation and no way to recover.
+ */
+function handleUnauthorized(path: string): void {
+ if (typeof window === 'undefined') return;
+ if (AUTH_EXEMPT_PATHS.some((exempt) => path.startsWith(exempt))) return;
+ // Already on the sign-in page: clearing is right, navigating would loop.
+ const onLogin = window.location.pathname.startsWith('/login');
+ try {
+ window.localStorage.removeItem(AUTH_TOKEN_KEY);
+ window.localStorage.removeItem(AUTH_REFRESH_KEY);
+ window.localStorage.removeItem(AUTH_USER_KEY);
+ } catch {
+ /* localStorage unavailable; the redirect below still helps */
+ }
+ if (!onLogin) {
+ const next = encodeURIComponent(window.location.pathname + window.location.search);
+ window.location.assign(`/login?next=${next}`);
+ }
+}
+
async function request<T>(path: string, options: FetchOptions = {}): Promise<T> {
const { params, baseUrl, ...fetchOptions } = options;
@@ -222,6 +287,11 @@ async function request<T>(path: string, options: FetchOptions = {}): Promise<T>
if (!response.ok) {
const errorText = await response.text().catch(() => '');
+ if (response.status === 401) {
+ // Self-heal rather than leaving the user on a page where every
+ // panel fails silently and nothing offers a way back.
+ handleUnauthorized(path);
+ }
throw new ApiError(
`API ${response.status} ${response.statusText} — ${path}`,
response.status,
@@ -377,10 +447,25 @@ export const authApi = {
}
},
+ /**
+ * Whether there is a session that can still make a request.
+ *
+ * This used to answer "is a token stored", which is a different
+ * question and produced an inescapable lockout: `/login` saw a stored
+ * token, redirected to `/dashboard`, every call there 401'd, and
+ * nothing sent the user back. The only way out was clearing browser
+ * storage, which is not a thing to ask of an operator.
+ *
+ * An unreadable or unparseable token counts as **not** authenticated:
+ * the pessimistic answer costs a sign-in, the optimistic one costs the
+ * lockout.
+ */
isAuthenticated(): boolean {
if (typeof window === 'undefined') return false;
try {
- return Boolean(window.localStorage.getItem(AUTH_TOKEN_KEY));
+ const token = window.localStorage.getItem(AUTH_TOKEN_KEY);
+ if (!token) return false;
+ return !isTokenExpired(token);
} catch {
return false;
}
diff --git a/apps/web/src/lib/authExpiry.test.ts b/apps/web/src/lib/authExpiry.test.ts
new file mode 100644
index 000000000..7f1c1e354
--- /dev/null
+++ b/apps/web/src/lib/authExpiry.test.ts
@@ -0,0 +1,86 @@
+/**
+ * An expired session sends you to sign in, rather than locking you out.
+ *
+ * The defect, found by a live browser walkthrough and reproduced
+ * deterministically: with an expired token in `localStorage`, `/login`
+ * redirected straight to `/dashboard`, every call there answered 401, and
+ * nothing ever sent the user back. There was no path out of the product
+ * short of clearing browser storage by hand, which is not a thing to ask
+ * of an operator at two in the morning.
+ *
+ * Two independent causes, so two independent fixes:
+ *
+ * 1. `isAuthenticated()` answered "is a token stored", which is a
+ * different question from "can this session make a request".
+ * 2. Nothing handled a 401. A session that died mid-shift left every
+ * panel failing with no explanation.
+ *
+ * Either fix alone leaves a hole: the first still strands a user whose
+ * token expires *while* they are working, and the second still bounces a
+ * returning user off `/login` before any request is made.
+ */
+
+import { describe, expect, it } from 'vitest';
+
+import { isTokenExpired } from './api';
+
+/** A JWT with the given claims. Unsigned: nothing here verifies it. */
+function token(claims: Record<string, unknown>): string {
+ const b64 = (o: unknown) =>
+ btoa(JSON.stringify(o)).replace(/\+/g, '-').replace(/\//g, '_').replace(/=+$/, '');
+ return `${b64({ alg: 'HS256', typ: 'JWT' })}.${b64(claims)}.signature`;
+}
+
+const NOW = 1_790_000_000;
+
+describe('isTokenExpired', () => {
+ it('reports an expired token as expired', () => {
+ expect(isTokenExpired(token({ exp: NOW - 3600 }), NOW)).toBe(true);
+ });
+
+ it('reports a live token as live', () => {
+ expect(isTokenExpired(token({ exp: NOW + 3600 }), NOW)).toBe(false);
+ });
+
+ it('allows a small skew so a session does not bounce mid-request', () => {
+ // Expiring in this very second is not a reason to throw the user out
+ // of a page that was working a moment ago.
+ expect(isTokenExpired(token({ exp: NOW }), NOW)).toBe(false);
+ });
+
+ it('treats an unreadable token as expired', () => {
+ // The pessimistic answer costs a sign-in; the optimistic one costs
+ // the lockout this test exists to prevent.
+ expect(isTokenExpired('not-a-jwt', NOW)).toBe(true);
+ expect(isTokenExpired('', NOW)).toBe(true);
+ expect(isTokenExpired('a.b', NOW)).toBe(true);
+ expect(isTokenExpired('a.!!!not-base64!!!.c', NOW)).toBe(true);
+ });
+
+ it('treats a token with no exp as live', () => {
+ // Some deployments issue non-expiring service tokens. Refusing those
+ // would lock out a working configuration to fix a broken one.
+ expect(isTokenExpired(token({ sub: 'x' }), NOW)).toBe(false);
+ });
+
+ it('decodes base64url, not just base64', () => {
+ // A JWT payload containing `-` or `_` must not read as unparseable,
+ // which would log a perfectly good session out.
+ const claims = { exp: NOW + 3600, email: 'a+b/[email protected]', sub: '?>?>?>' };
+ expect(isTokenExpired(token(claims), NOW)).toBe(false);
+ });
+});
+
+describe('the lockout this prevents', () => {
+ it('an expired token must not count as an authenticated session', async () => {
+ const { authApi, AUTH_TOKEN_KEY } = await import('./api');
+
+ window.localStorage.setItem(AUTH_TOKEN_KEY, token({ exp: Math.floor(Date.now() / 1000) - 60 }));
+ expect(authApi.isAuthenticated()).toBe(false);
+
+ window.localStorage.setItem(AUTH_TOKEN_KEY, token({ exp: Math.floor(Date.now() / 1000) + 600 }));
+ expect(authApi.isAuthenticated()).toBe(true);
+
+ window.localStorage.clear();
+ });
+});
diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.md
index 2a821d6d2..48f91fd72 100644
--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md
+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md
@@ -276,26 +276,32 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
| Closure accuracy is measured on the tenant's own data, not only on a synthetic corpus | `apps/docs/docs/operations/shadow-mode.md` | `ci.yml :: Python — Tests` (agents, `test_closure_qa_sampling.py`) | core | GATED | Nothing measured this. The eval harness grades a synthetic corpus and the funnel counts how many alerts were closed, neither of which answers "of the ones the agent closed on your data, how many should it have". A tenant-configurable sample (5% default) of auto-closures goes to an analyst, scored on the five-part rubric. The sample is a **deterministic hash of the alert id, not `random()`**: a rolled sample fires at the configured rate *per replica*, so three replicas at 5% sample 15%, and a redelivered Kafka message gets a second roll and weights one closure twice. Bucketing also makes a higher rate a superset of a lower one, so a tenant's figure stays comparable across a rate change. Zero reviews reports **not measured** rather than 0.0, and every mean travels with the count it was taken over |
| SSO completes a sign-in, into the right tenant with the mapped role | `apps/docs/docs/operations/security.md` | `ci.yml :: Python — Tests` (`services/api/tests/test_sso_provisioning.py`) | core | GATED | Both handlers used to end a successful sign-in by putting a JWT in an `aisoc_token` cookie carrying `sub`, `email`, `name` and `picture`. That token authenticated nothing three times over: signed with `JWT_SECRET` rather than the key the API verifies with, carrying no `tenant_id` or `role`, and in a cookie when the API reads `Authorization: Bearer`. A user could complete the whole dance and find every request unauthenticated. **The security property the tests exist for is that the tenant does not come from the assertion**: an IdP that can name its own tenant can name somebody else's, so it comes from the connection row an administrator configured, and `complete_sso_login` is asserted to take no tenant argument at all. A group cannot map to `admin` or `platform_admin`. The token travels in the URL fragment, which is not sent to the server and does not reach an access log or a `Referer`. **Limit**: no containerised-IdP test yet, so this is gated against the provisioning path rather than against a real Okta, Entra or Keycloak |
| Accessibility is checked on the views an analyst uses, not only on the landing page | `ROADMAP.md` | `ci.yml :: Web — Tests` (`apps/web/src/test/a11y.test.tsx`) | core | GATED | 1.1 narrowed the "WCAG AA full accessibility pass" claim because axe covered the landing and chrome components only. 4.7 adds **three of the five** operator views the plan names: the alerts queue and list, the investigation rail, and settings. Each asserts it rendered real markup **before** axe runs, because axe passes on an empty div and a view that rendered a spinner would report coverage the suite does not have. That guard immediately caught one: the rail keys SWR on a tuple, the fixture read only the string form, and it was rendering its error state. **Limits stated**: the case workspace is not covered, and axe checks a subset of WCAG rather than all of it |
+| A playbook can start from an alert, and previews before it acts | `apps/docs/docs/console/playbooks.md` | `ci.yml :: Python — Tests` (agents, `test_alert_triggered_playbooks.py`) | core | GATED | `find_matching()` had **no production caller**, so no playbook ever ran from an alert. Three switches must agree before one acts (deployment, tenant, playbook) and the default of every one is off; anything short of all three runs in **preview**, with its plan and simulated steps attached to the alert. Preview is the default state rather than a mode somebody has to remember, so a deployment that enables the feature and forgets the per-playbook list gets previews rather than containment. The live list is read per call, so turning one off takes effect on the next alert rather than the next restart. It runs **after** triage, because a playbook's conditions read the verdict |
+| An approval step is a durable pause that survives a restart | `apps/docs/docs/console/playbooks.md` | `ci.yml :: Python — Tests` (agents, `test_playbook_pause_resume.py`) | core | GATED | The engine is a single-threaded index walk with no pause and no resume, so `approval` failed closed and **12 shipped playbooks aborted on it**. Failing closed was right (it previously returned `{"skipped": true}` while reporting SUCCESS, letting a run continue into the action a human was meant to authorise) but it is not an approval mechanism. Migration `081` stores the position **and the context**, because the resumed half would otherwise see an empty context and every templated parameter would resolve to nothing. The stored index is the approval step itself and resume advances past it, so a replayed decision continues rather than pausing again. The pause is resolved **before** the run continues, so a double-tap in the responder app resumes once. Expiry is mandatory with a recorded outcome: `expired` is a decision, not the absence of one |
+| A tenant's detection tuning reaches the streaming engine, and only that tenant's | `apps/docs/docs/console/detection-tuning.md` | `ci.yml :: Python — Service unit tests` (`services/fusion/tests/test_tenant_tuning_overlay.py`) | core | GATED | `DetectionEngine` evaluated the shared corpus and nothing else. The console writes a tenant's disables, thresholds and suppressions to `detection_rules`, where the streaming engine never looked, so a tenant who turned a noisy rule off **kept receiving its alerts while the console showed it disabled**, which is the worst shape because it tells the operator the problem is solved. An overlay rather than a per-tenant ruleset: the difference applied over one shared corpus, not N copies of 833 rules. Suppression is applied **after** the match so the hit can be logged with the tuning, its author and its reason, because "no alert" with no explanation is indistinguishable from a rule that did not match. A failed reload **keeps the previous overlay**: falling back to no-tuning would turn the tenant's queue back on and read as a flood rather than a fault |
+| Rules that cannot fire are counted, and the count only goes down | `docs/detections/truth-table.md` | `ci.yml :: Python — Lint & Type-check` (`check_detection_fields.py`, ratchet at 119) | core | GATED | **133 to 119.** The per-tenant-allowlist family (15 rules reading an `<x>_in_allowlist` boolean nothing computed) is closed: the overlay derives it from the allowlists the console already writes, which was the cheapest of the five families because an allowlist is the same decision a tenant expresses as a suppression. Derived per tenant rather than in the shared pass, because a global allowlist would make one tenant's exceptions apply to everybody. An unconfigured allowlist contributes **no key**, not `False`: a `not_in_allowlist` clause against a missing key is true for every event, which is the negation-flips-on-absence failure already recorded for the Sigma import. A test asserts every allowlist boolean the corpus reads is mapped, so a rule cannot stay unreachable while the ratchet says otherwise. The four remaining families (74 windowed, 24 identity, 8 comparison, 8 first-seen, 6 other, 2 baseline) need engine or enrichment work and are named rather than hidden |
+| Raw SQL names only columns a migration creates, on reads as well as writes | `docs/audit/REPOSITORY_REALITY.md` | `ci.yml :: Python — Lint & Type-check` (`check_raw_sql_columns.py`, self-tested) | core | GATED | The gate parsed `INSERT` and `UPDATE` only, which is one-directional in the way that matters: an absent column fails a `SELECT` just as hard, and a handler that wraps the read in `except` turns the crash into a warning and an empty result. **Live QA found exactly that**: `tenant_overlay._fetch` selected `rule_id` and `updated_by` from `detection_rules`, which has neither, so the per-tenant tuning overlay would have loaded nothing on every deployment — while every unit test passed, because the fake answered whatever it was asked. A `SELECT` fails only where the gate genuinely knows (table migrated, projection fully read, column absent); osquery virtual tables and the ClickHouse lake are declined rather than failed, because the gate knows which tables are migrated and not which engine a statement targets. Statement coverage 130 to 335, and the self-test is proven against the real pre-fix defect |
+| An expired session sends you to sign in, rather than locking you out | `apps/docs/docs/console/getting-started.md` | `ci.yml :: Web — Unit tests` (`apps/web/src/lib/authExpiry.test.ts`) | core | GATED | Found by a live browser walkthrough and reproduced deterministically: with an expired token in `localStorage`, `/login` redirected to `/dashboard`, every call there answered 401, and **nothing ever sent the user back** — the only way out was clearing browser storage by hand. Two independent causes, so two fixes: `isAuthenticated()` answered "is a token stored", which is a different question from "can this session make a request"; and nothing handled a 401, so a session that died mid-shift left every panel failing with no explanation. Either fix alone leaves a hole — the first still strands a user whose token expires *while* they work, the second still bounces a returning user off `/login` before any request is made. An unreadable token counts as not authenticated, because the pessimistic answer costs a sign-in and the optimistic one costs the lockout |
## Summary
-- GATED: 268
+- GATED: 274
## Summary
-- GATED: 268
+- GATED: 274
## Summary
-- GATED: 268
+- GATED: 274
## Summary
-- GATED: 268
+- GATED: 274
## Summary
-- GATED: 268
+- GATED: 274
- PARTIAL: 0
- NO GATE: 0 (**every claim is backed by a failing test, and none is now a named deferral either.** The last NO GATE — the weekly benchmark scoreboard running live against `main` — closed in Phase E1: `scripts/check_scoreboard.py` ties the published scoreboard to a deterministic per-PR live-agent MITRE-accuracy run, while the funded weekly `wet-eval.yml` appends the LLM-tier rows. The last two PARTIAL rows closed with `scripts/check_live_agent_floor.py` and a floor derived from ten runs across two environments, not by relabelling. Recount with `scripts/check_claim_gate_matrix.py` rather than trusting these three numbers: they move weekly, and a count copied into prose goes stale in silence.)
diff --git a/marketplace/index.json b/marketplace/index.json
index f51cd5ba7..1f5a048eb 100644
--- a/marketplace/index.json
+++ b/marketplace/index.json
@@ -1,7 +1,7 @@
{
"$schema": "https://example.com/schemas/marketplace/v1.json",
"version": "1.0.0",
- "generated": "2026-10-01T21:06:28Z",
+ "generated": "2026-10-02T04:17:51Z",
"categories": [
{
"id": "playbooks",
diff --git a/schemas/playbook.schema.json b/schemas/playbook.schema.json
index f519f141c..1341ad1de 100644
--- a/schemas/playbook.schema.json
+++ b/schemas/playbook.schema.json
@@ -8,7 +8,7 @@
"additionalProperties": false,
"x-aisoc-execution": {
- "description": "What `services/agents/app/playbook/engine.py` does with each step type. `executed` = a handler runs and has a real effect (an outbound call to a service, or engine-internal control flow). `governed` = the step is dispatched through the action registry in `services/actions`, where the capability contract and the tenant autonomy policy decide whether a vendor is touched; the step reports which happened and `executed` is the single field that means one was, so a preview, an approval queue and a missing integration are each named rather than reported as a success. `unimplemented` = no handler is registered; the engine fails the step closed rather than reporting a success it did not achieve. This map is machine-checked against the engine's handler table, so it cannot drift into an aspirational claim.",
+ "description": "What `services/agents/app/playbook/engine.py` does with each step type. `executed` = a handler runs and has a real effect (an outbound call to a service, or engine-internal control flow). `governed` = the step is dispatched through the action registry in `services/actions`, where the capability contract and the tenant autonomy policy decide whether a vendor is touched; the step reports which happened and `executed` is the single field that means one was, so a preview, an approval queue and a missing integration are each named rather than reported as a success. `unimplemented` = no handler is registered; the engine fails the step closed rather than reporting a success it did not achieve. `approval` became `executed` in parity 5.2: it suspends the run to Postgres and resumes on the decision, where before it had nowhere to pause to and failed closed while 12 shipped playbooks aborted on it. This map is machine-checked against the engine's handler table, so it cannot drift into an aspirational claim.",
"enrich": "executed",
"investigate": "executed",
"notify": "executed",
@@ -19,7 +19,7 @@
"block_ip": "governed",
"isolate_host": "governed",
"create_ticket": "governed",
- "approval": "unimplemented",
+ "approval": "executed",
"block_ioc": "governed",
"disable_user": "governed",
"reset_password": "governed",
diff --git a/scripts/check_detection_fields.py b/scripts/check_detection_fields.py
index 2760fb620..c05d79341 100644
--- a/scripts/check_detection_fields.py
+++ b/scripts/check_detection_fields.py
@@ -72,7 +72,11 @@ NORMALIZER = ROOT / "services" / "ingest" / "internal" / "normalizer" / "normali
#: Current number of engine-loaded rules that depend on a DERIVED or STATEFUL
#: field. This may only ever decrease. Lower it in the same PR that fixes
#: rules; never raise it to make a red build green.
-MAX_UNREACHABLE = 133
+# 133 before parity 5.5, which closed the 15-rule per-tenant-allowlist
+# family by deriving the booleans from the tenant overlay. 14 rules moved
+# (one rule carried two clauses), leaving the four families that genuinely
+# need engine or enrichment work.
+MAX_UNREACHABLE = 119
#: Fields that no telemetry carries because they are computed, not observed:
#: sliding-window counters, allowlist membership, privilege flags, and
@@ -94,9 +98,19 @@ MAX_UNREACHABLE = 133
#: moved none of these rules off this list and the count did not move either.
#:
#: Migrating one means *authoring* a `wd-*` rule, not flipping a flag.
+#: Parity 5.5 removed `_in_allowlist` from this pattern. It is no longer
+#: a field nothing computes: `services/fusion/app/services/tenant_overlay.py`
+#: derives it per tenant from the allowlists the console writes, and the
+#: engine merges it into the match namespace before evaluating.
+#:
+#: Per tenant rather than in the shared derived-field pass, deliberately:
+#: a global allowlist would make one tenant's exceptions apply to
+#: everybody. An unconfigured allowlist contributes no key at all rather
+#: than `False`, because a `not_in_allowlist` clause against a missing key
+#: is true for every event, which is the negation-flips-on-absence failure
+#: already recorded for the Sigma import.
_DERIVED_FIELD_PATTERN = re.compile(
r"(_count$|^count_|_per_|time_window|_window_"
- r"|_in_allowlist$|_not_in_allowlist$"
r"|_priv$|_is_admin$|_is_dc$|_eq_|_neq|^is_"
r"|_age_days$|_age_hours$|_ratio$|_percent$"
r"|^active_|_baseline|_is_first_|_seen_before$|_deviation)"
diff --git a/scripts/check_raw_sql_columns.py b/scripts/check_raw_sql_columns.py
index fc56c7bf9..07eff769f 100644
--- a/scripts/check_raw_sql_columns.py
+++ b/scripts/check_raw_sql_columns.py
@@ -265,6 +265,120 @@ def migration_schema(services_dir: pathlib.Path) -> tuple[dict[str, set[str]], d
# ---------------------------------------------------------------------------
_INSERT = re.compile(r"\bINSERT\s+(?:OR\s+\w+\s+)?INTO\s+([^\s(;]+)", re.I)
+#: A single-table SELECT, with the projection captured.
+#:
+#: This gate checked INSERT and UPDATE only for most of its life, which
+#: made it one-directional in the way that matters: a SELECT naming a
+#: column that does not exist raises `UndefinedColumnError` just as hard,
+#: and a handler that wraps the read in `except` turns that into a
+#: warning and an empty result. The feature then does nothing while every
+#: unit test passes against a fake.
+#:
+#: That is not hypothetical. `tenant_overlay._fetch` selected `rule_id`
+#: and `updated_by` from `detection_rules`, which has neither, so the
+#: per-tenant tuning overlay would have loaded nothing on every
+#: deployment. Live QA found it; this pattern is why it cannot recur.
+#:
+#: Deliberately narrow: one table, no join. A join needs per-table column
+#: resolution and alias tracking, and a pattern that silently mis-resolves
+#: is worse than one that declines to try — those are reported as
+#: unresolvable rather than passed.
+_SELECT = re.compile(
+ r"\bSELECT\s+(?!.*\bJOIN\b)(.+?)\s+FROM\s+(?:ONLY\s+)?([a-z_][a-z0-9_]*)\b",
+ re.I | re.S,
+)
+
+#: Projection items that name no column of the table.
+_SELECT_NOISE = frozenset(
+ {
+ "as",
+ "distinct",
+ "all",
+ "case",
+ "when",
+ "then",
+ "else",
+ "end",
+ "and",
+ "or",
+ "not",
+ "null",
+ "true",
+ "false",
+ "asc",
+ "desc",
+ "cast",
+ "coalesce",
+ "nullif",
+ "count",
+ "sum",
+ "avg",
+ "min",
+ "max",
+ "now",
+ "interval",
+ "over",
+ "partition",
+ "by",
+ "filter",
+ "where",
+ "on",
+ "using",
+ "array",
+ "jsonb",
+ "text",
+ "uuid",
+ "int",
+ "integer",
+ "bigint",
+ "boolean",
+ "timestamptz",
+ "numeric",
+ "float",
+ }
+)
+
+
+def _select_columns(projection: str) -> tuple[list[str], bool]:
+ """Bare column names in a projection, and whether it was fully read.
+
+ `complete` is False for `*`, for a function call whose arguments this
+ cannot attribute, and for an expression with an operator — the caller
+ treats an incomplete projection as unresolvable rather than clean,
+ because a half-read list that reports OK is the failure this gate
+ exists to prevent.
+ """
+ if "*" in projection:
+ return [], False
+ columns: list[str] = []
+ complete = True
+ for item in _split_top_level(projection):
+ item = item.strip()
+ # Strip an alias: `x AS y` names x, not y.
+ item = re.split(r"\s+AS\s+", item, flags=re.I)[0].strip()
+ if not item:
+ continue
+ # A bare identifier is a column. Anything else is an expression
+ # whose operands may or may not be columns of this table.
+ if re.fullmatch(r"[a-z_][a-z0-9_]*", item, re.I):
+ if item.lower() not in _SELECT_NOISE:
+ columns.append(item)
+ continue
+ # A JSONB path (`provenance->>'x'`) names its left operand.
+ jsonb = re.fullmatch(r"([a-z_][a-z0-9_]*)\s*-\>\>?\s*'[^']*'", item, re.I)
+ if jsonb:
+ columns.append(jsonb.group(1))
+ continue
+ # Resolve what is nameable inside an expression, and mark the
+ # projection incomplete so the caller does not read the result as
+ # an exhaustive list.
+ complete = False
+ for token in re.findall(r"[a-z_][a-z0-9_]*", item, re.I):
+ if token.lower() not in _SELECT_NOISE:
+ columns.append(token)
+ return columns, complete
+
+
_UPDATE = re.compile(r"\bUPDATE\s+(?:ONLY\s+)?([^\s(;]+)\s+SET\b", re.I)
_DO_UPDATE = re.compile(r"\bDO\s+UPDATE\s+SET\b", re.I)
@@ -397,7 +511,8 @@ def statements_in(source: str, relpath: str) -> list[dict]:
if not isinstance(node, ast.Constant | ast.JoinedStr):
continue
text = _render(node)
- if not text or not ("INSERT" in text.upper() or "UPDATE" in text.upper()):
+ upper = text.upper() if text else ""
+ if not text or not ("INSERT" in upper or "UPDATE" in upper or "SELECT" in upper):
continue
line = getattr(node, "lineno", 0)
@@ -419,6 +534,20 @@ def statements_in(source: str, relpath: str) -> list[dict]:
inserts.append((match.start(), statement))
found.append(statement)
+ for match in _SELECT.finditer(text):
+ columns, complete = _select_columns(match.group(1))
+ found.append(
+ {
+ "kind": "SELECT",
+ "file": relpath,
+ "line": line,
+ "table": match.group(2).strip('"'),
+ "columns": columns,
+ "columns_declared": True,
+ "columns_complete": complete,
+ }
+ )
+
for match in _UPDATE.finditer(text):
columns, complete = _set_columns(text, match.end())
found.append(
@@ -491,6 +620,7 @@ def scan(root: pathlib.Path | None = None) -> dict:
statements, files_read = service_statements(services_dir, root)
findings: list[dict] = []
+ uncompared: list[dict] = []
credits: list[dict] = []
foreign_used: set[str] = set()
dynamic_used: set[tuple[str, str]] = set()
@@ -507,6 +637,28 @@ def scan(root: pathlib.Path | None = None) -> dict:
# Only a statement this parser could not read may be excused as
# dynamic. A file holding one assembled statement does not get its
# ordinary ones forgiven along with it.
+ # A SELECT is held to a narrower standard than a write, on purpose.
+ #
+ # The gate knows every table a migration creates; it does not know
+ # which *engine* a statement targets. `services/actions` queries
+ # osquery's virtual tables on an endpoint and the lake queries
+ # ClickHouse, and neither has a Postgres migration — failing those
+ # would be the gate asserting something it cannot see.
+ #
+ # So a SELECT fails only where the gate genuinely knows: the table
+ # IS migrated, the projection WAS fully read, and a named column is
+ # absent. Everything else is counted as not-compared rather than
+ # passed, which keeps the summary honest about its own coverage.
+ #
+ # That standard still catches the defect this was built for:
+ # `tenant_overlay._fetch` selected `rule_id` and `updated_by` from
+ # `detection_rules`, which is migrated and whose projection reads
+ # cleanly, so the overlay loaded nothing on every deployment while
+ # every unit test passed against a fake.
+ if statement["kind"] == "SELECT" and (not statement["columns_complete"] or table not in schema):
+ uncompared.append({**statement, "verdict": "select-not-resolved"})
+ continue
+
if DYNAMIC in table or not statement["columns_complete"]:
if key in DYNAMIC_SQL:
dynamic_used.add(key)
@@ -643,6 +795,42 @@ def _detects_injected_drift() -> list[tuple[str, bool]]:
("passes a statement whose table and every column exist", kinds(_GOOD_STATEMENT) == set()),
("credits the passing statement rather than skipping it", counted(_GOOD_STATEMENT) == 1),
("refuses a statement assembled at runtime rather than skipping it", kinds(dynamic) == {"unparsed-statement"}),
+ # The SELECT direction, added after live QA found a read this gate
+ # could not see. `tenant_overlay._fetch` selected two columns
+ # `detection_rules` does not have, so the per-tenant tuning overlay
+ # loaded nothing on every deployment — and every unit test passed,
+ # because they ran against a fake that answered whatever was asked.
+ #
+ # A gate that checks writes and not reads is one-directional in the
+ # way that matters: an absent column fails a SELECT just as hard,
+ # and a handler that wraps the read in `except` turns the crash
+ # into a warning and an empty result.
+ (
+ "detects a SELECT naming a column no migration creates",
+ kinds("SQL = 'SELECT id, nonexistent_column FROM widgets'\n") == {"unknown-column"},
+ ),
+ (
+ "passes a SELECT whose every column exists",
+ kinds("SQL = 'SELECT id, tenant_id, name FROM widgets'\n") == set(),
+ ),
+ (
+ "resolves a jsonb path to its left operand rather than guessing",
+ kinds("SQL = \"SELECT nonexistent_column->>'x' FROM widgets\"\n") == {"unknown-column"},
+ ),
+ (
+ # The gate knows which tables are migrated, not which engine a
+ # statement targets. osquery's virtual tables and the ClickHouse
+ # lake have no Postgres migration, and failing those would be
+ # the gate asserting something it cannot see.
+ "declines a SELECT against an unmigrated table rather than failing it",
+ kinds("SQL = 'SELECT pid, name FROM processes'\n") == set(),
+ ),
+ (
+ # A half-read projection that reports OK is the failure this
+ # gate exists to prevent, so an unreadable one is not compared.
+ "declines a SELECT whose projection it could not fully read",
+ kinds("SQL = 'SELECT COUNT(*) AS n FROM widgets'\n") == set(),
+ ),
]
diff --git a/scripts/check_tenant_query_predicates.py b/scripts/check_tenant_query_predicates.py
index fc678f2c1..c21f5ad42 100644
--- a/scripts/check_tenant_query_predicates.py
+++ b/scripts/check_tenant_query_predicates.py
@@ -229,6 +229,11 @@ RATCHET: dict[str, str] = {
"matched on the 32-byte single-use state nonce, which is itself the credential"
),
# -- deliberately cross-tenant surfaces ----------------------------------
+ "services/agents/app/playbook/pause.py::expire_due::aisoc_playbook_pauses": (
+ "the approval-expiry sweep: a per-tenant sweep would need a list of tenants to "
+ "iterate, and a tenant missing from that list would have approvals that never "
+ "expire, which is the silent hang the sweep exists to prevent"
+ ),
"services/api/app/api/v1/endpoints/tenants.py::create_user::User": (
"platform-admin provisioning: creates users across tenants by design"
),
@@ -338,7 +343,12 @@ RATCHET: dict[str, str] = {
#: was moved out of module scope to earn this entry: keyed on `<module>` the
#: waiver would have matched any later unscoped query in the same file, which
#: is an allowlist wearing a ratchet's name.
-MAX_RATCHET = 35
+# 35 before parity 5.2, which added the approval-expiry sweep. It is the
+# one statement in that feature that is cross-tenant by construction: the
+# two lookups beside it (find_waiting, resolve) take a tenant and filter
+# on it, because resuming another tenant's playbook run should take two
+# mistakes rather than one.
+MAX_RATCHET = 36
# ---------------------------------------------------------------------------
diff --git a/services/agents/app/playbook/alert_trigger.py b/services/agents/app/playbook/alert_trigger.py
new file mode 100644
index 000000000..382a7ea94
--- /dev/null
+++ b/services/agents/app/playbook/alert_trigger.py
@@ -0,0 +1,198 @@
+"""Start a playbook from a fused alert. Off by default, preview before live.
+
+Parity plan 5.1: "`find_matching()` has no production caller today. Call it
+from the fused-alert path behind a setting that ships off. A playbook first
+runs in preview, with its plan and simulated steps shown on the alert, and
+then goes live per tenant and per playbook."
+
+Why three switches rather than one
+-----------------------------------
+A playbook that runs from an alert is the most consequential thing in this
+product: it can isolate a host nobody asked it to. One global flag is not
+enough granularity to turn that on safely, so there are three and **all**
+must agree:
+
+1. the deployment switch, off by default;
+2. the tenant, which must have opted in;
+3. the playbook itself, which must be marked live rather than preview.
+
+A playbook that clears the first two still runs in **preview**: its plan
+and its simulated steps attach to the alert so an analyst can read what it
+would have done. Going live is a separate, per-playbook decision made
+after reading that.
+
+The ordering matters: preview is the default state, not a mode somebody
+has to remember to use first.
+"""
+
+from __future__ import annotations
+
+import os
+from dataclasses import dataclass, field
+from typing import Any
+
+import structlog
+
+logger = structlog.get_logger()
+
+#: The deployment switch. Off, because the alternative is a fresh install
+#: that can take a containment action on its first alert.
+ENABLED_ENV = "AISOC_ALERT_PLAYBOOKS_ENABLED"
+
+#: Per-tenant and per-playbook opt-in to **live** execution. Everything
+#: else runs in preview.
+LIVE_PLAYBOOKS_ENV = "AISOC_ALERT_PLAYBOOKS_LIVE"
+
+
+#: The trigger event name the shipped playbooks actually declare.
+#:
+#: All 11 alert-triggered playbooks in the corpus use `on: alert`. This
+#: was `alert.created` for the length of one live QA run, which matched
+#: **nothing** — the feature was wired, enabled, and dead, and the unit
+#: tests passed because the fake store returned a playbook whatever event
+#: it was handed. A test now asserts this constant against what the
+#: corpus declares, so the two cannot drift apart again.
+TRIGGER_EVENT = "alert"
+
+
+def deployment_enabled() -> bool:
+ return os.getenv(ENABLED_ENV, "").strip().lower() in ("1", "true", "yes")
+
+
+def live_playbook_ids() -> set[str]:
+ """Playbooks allowed to run live, as `tenant:playbook` or `playbook`.
+
+ Read per call rather than cached, so turning one off takes effect on
+ the next alert rather than on the next restart. A playbook that can
+ take a containment action should be switchable off faster than a
+ deploy.
+ """
+ raw = os.getenv(LIVE_PLAYBOOKS_ENV, "")
+ return {part.strip() for part in raw.split(",") if part.strip()}
+
+
+def is_live(*, tenant_id: str, playbook_id: str) -> bool:
+ allowed = live_playbook_ids()
+ return bool(allowed & {f"{tenant_id}:{playbook_id}", playbook_id})
+
+
+@dataclass
+class TriggerOutcome:
+ """What the alert path did, and why. Attached to the alert."""
+
+ considered: int = 0
+ matched: list[str] = field(default_factory=list)
+ previewed: list[dict[str, Any]] = field(default_factory=list)
+ executed: list[dict[str, Any]] = field(default_factory=list)
+ skipped_reason: str | None = None
+
+ def as_dict(self) -> dict[str, Any]:
+ return {
+ "considered": self.considered,
+ "matched": self.matched,
+ "previewed": self.previewed,
+ "executed": self.executed,
+ "skipped_reason": self.skipped_reason,
+ }
+
+
+def alert_context(state: Any) -> dict[str, Any]:
+ """The trigger context a playbook's conditions read.
+
+ Flat, because `find_matching` filters on `severity` and `tags` at the
+ top level and a nested alert would match nothing while looking like it
+ should.
+ """
+ raw = getattr(state, "raw_alert", None) or {}
+ if not isinstance(raw, dict):
+ raw = {}
+ context: dict[str, Any] = {
+ "tenant_id": str(getattr(state, "tenant_id", "") or ""),
+ "alert_id": str(getattr(state, "incident_id", "") or ""),
+ "severity": raw.get("severity") or getattr(state, "severity", None),
+ "tags": raw.get("tags") or [],
+ "title": raw.get("title") or "",
+ "verdict": getattr(state, "verdict", None),
+ "confidence": getattr(state, "confidence", None),
+ }
+ for key in ("host", "hostname", "username", "user", "src_ip", "dst_ip", "domain", "file_hash"):
+ value = raw.get(key)
+ if value:
+ context.setdefault(key, value)
+ return context
+
+
+async def run_for_alert(state: Any) -> TriggerOutcome:
+ """Match and run playbooks for one fused alert.
+
+ Never raises. A playbook failure must not lose the triage result it
+ was attached to, so every error is captured into the outcome and the
+ alert still carries its verdict.
+ """
+ outcome = TriggerOutcome()
+
+ if not deployment_enabled():
+ outcome.skipped_reason = f"alert-triggered playbooks are off; set {ENABLED_ENV}=1 to enable them"
+ return outcome
+
+ context = alert_context(state)
+ tenant_id = context["tenant_id"]
+ if not tenant_id:
+ outcome.skipped_reason = "the alert carries no tenant, so no playbook can be scoped to it"
+ return outcome
+
+ try:
+ from app.playbook.engine import PlaybookEngine
+ from app.playbook.store import PlaybookStore
+
+ store = PlaybookStore.default()
+ matches: list[Any] = list(store.find_matching(TRIGGER_EVENT, context))
+ except Exception as exc: # noqa: BLE001
+ logger.warning("alert_playbooks.match_failed", error=str(exc))
+ outcome.skipped_reason = f"could not match playbooks: {exc}"
+ return outcome
+
+ outcome.considered = len(matches)
+ engine = PlaybookEngine()
+
+ for playbook in matches:
+ outcome.matched.append(playbook.id)
+ live = is_live(tenant_id=tenant_id, playbook_id=playbook.id)
+ try:
+ run = await engine.run(playbook, context, dry_run=not live)
+ except Exception as exc: # noqa: BLE001
+ logger.warning("alert_playbooks.run_failed", playbook_id=playbook.id, error=str(exc))
+ outcome.previewed.append({"playbook_id": playbook.id, "error": str(exc)})
+ continue
+
+ record = {
+ "playbook_id": playbook.id,
+ "playbook_name": playbook.name,
+ "run_id": run.run_id,
+ "status": getattr(run.status, "value", str(run.status)),
+ "steps": [
+ {
+ "step_id": r.get("step_id"),
+ "name": r.get("name"),
+ "status": getattr(r.get("status"), "value", str(r.get("status"))),
+ }
+ for r in run.step_results
+ ],
+ }
+ if live:
+ outcome.executed.append(record)
+ logger.info(
+ "alert_playbooks.ran_live",
+ playbook_id=playbook.id,
+ run_id=run.run_id,
+ status=record["status"],
+ )
+ else:
+ # The reason is on the record, so an analyst reading the alert
+ # sees why it previewed rather than wondering whether it
+ # failed.
+ record["preview"] = True
+ record["reason"] = f"preview only: add {tenant_id}:{playbook.id} to {LIVE_PLAYBOOKS_ENV} to let this playbook act"
+ outcome.previewed.append(record)
+
+ return outcome
diff --git a/services/agents/app/playbook/engine.py b/services/agents/app/playbook/engine.py
index a2e1bbb40..ecad06a26 100644
--- a/services/agents/app/playbook/engine.py
+++ b/services/agents/app/playbook/engine.py
@@ -23,6 +23,8 @@ from typing import Any
import httpx
+from app.playbook import pause as playbook_pause
+
from . import action_bridge
from .bounds import clamp_timeout
from .errors import PermanentStepFailure
@@ -60,6 +62,11 @@ class RunStatus(str, Enum):
COMPLETED = "completed"
FAILED = "failed"
CANCELLED = "cancelled"
+ #: Suspended at an approval step, waiting on a human. Distinct from
+ #: FAILED, which is what an approval step produced before parity 5.2
+ #: and which reads to an operator as a broken playbook rather than as
+ #: one doing exactly what it was written to do.
+ PAUSED = "paused"
class StepStatus(str, Enum):
@@ -473,18 +480,18 @@ _TARGET_KEYS: dict[StepType, tuple[str, ...]] = {
#: reason. The engine reports the reason instead of a bare "no handler", and
#: the schema's ``x-aisoc-execution`` map records the same state, so an author
#: can tell before writing the playbook rather than after running it.
-_UNBRIDGEABLE: dict[StepType, str] = {
- StepType.APPROVAL: (
- "an approval step is a pause, and this engine is a single-threaded "
- "index walk with no pause or resume — there is nothing to suspend and "
- "nothing to wake. It is also no longer the mechanism: every response "
- "step is now graded against its own capability contract at dispatch "
- "and returns 'pending_approval' on its own when a human is required, "
- "so an approval step in front of one would gate a decision that is "
- "already gated. Remove it, or hold the action in the actions service, "
- "which does queue for an analyst."
- ),
-}
+#: Empty since parity 5.2. `approval` was the only entry: the engine had no
+#: pause and no resume, so there was nothing to suspend and nothing to wake,
+#: and the step failed closed while 12 shipped playbooks aborted on it.
+#:
+#: It is a durable pause now (`app.playbook.pause`), so the entry is gone
+#: rather than reworded. The mechanism it argued against is still true and
+#: still worth knowing: every response step is separately graded against
+#: its own capability contract at dispatch and returns `pending_approval`
+#: on its own, so an approval step in front of one gates a decision that is
+#: already gated. That is a reason to leave it out of a playbook, not a
+#: reason for the engine to refuse it.
+_UNBRIDGEABLE: dict[StepType, str] = {}
def _resolve_target(step: PlaybookStep, context: dict[str, Any]) -> str:
@@ -716,17 +723,176 @@ async def _emit(run_id: str, event_type: str, payload: dict, http: httpx.AsyncCl
# ---------------------------------------------------------------------------
+async def _suspend_for_approval(pr: PlaybookRun, step: PlaybookStep, step_idx: int, http: Any) -> Any:
+ """Create the approval and persist where to resume.
+
+ Order matters. The approval row is created **first**, so the pause can
+ name it: a pause with no approval has nothing for a human to decide
+ against, and a human deciding against an approval with no pause wakes
+ nothing. If the approval cannot be created the pause is not written
+ either, and the caller fails the step closed.
+ """
+ tenant_id = str(pr.context.get("tenant_id") or pr.trigger_context.get("tenant_id") or "")
+ if not tenant_id:
+ logger.warning("Approval step %s has no tenant in context; cannot suspend", step.name)
+ return None
+
+ approval_id = await _create_approval(pr, step, tenant_id, http)
+
+ return await playbook_pause.suspend(
+ tenant_id=tenant_id,
+ run_id=pr.run_id,
+ playbook_id=pr.playbook_id,
+ playbook_name=pr.playbook_name,
+ step_index=step_idx,
+ step_id=step.id,
+ run_context=pr.context,
+ step_results=pr.step_results,
+ approval_id=approval_id,
+ ttl_hours=_approval_ttl(step),
+ )
+
+
+def _approval_ttl(step: PlaybookStep) -> float | None:
+ """A per-step deadline, if the author set one."""
+ raw = step.params.get("expires_in_hours") or step.params.get("timeout_hours")
+ try:
+ return float(raw) if raw is not None else None
+ except (TypeError, ValueError):
+ return None
+
+
+async def _create_approval(pr: PlaybookRun, step: PlaybookStep, tenant_id: str, http: Any) -> str | None:
+ """Ask the API to open an approval. None when it could not be created.
+
+ Posted to the API rather than written here for the same reason
+ `action_bridge` posts there: that service owns the vault and the
+ tenant session, and a second writer to `agent_approvals` would be a
+ second place the responder app's decision has to be kept in step with.
+ """
+ token = os.getenv("AISOC_AGENTS_SERVICE_TOKEN", "").strip() or os.getenv("AISOC_SERVICE_TOKEN", "").strip()
+ if not token:
+ logger.warning(
+ "Approval step %s cannot open an approval: AISOC_AGENTS_SERVICE_TOKEN is unset",
+ step.name,
+ )
+ return None
+ api_url = os.getenv("AISOC_API_URL", "http://api:8000")
+ try:
+ response = await http.post(
+ f"{api_url}/api/v1/approvals",
+ headers={"Authorization": f"Bearer {token}", "X-Tenant-ID": tenant_id},
+ json={
+ "title": step.name or f"Approval for {pr.playbook_name}",
+ "description": str(step.params.get("prompt") or step.params.get("reason") or ""),
+ "requested_by": f"playbook:{pr.playbook_id}",
+ "context": {"run_id": pr.run_id, "step_id": step.id},
+ },
+ timeout=10.0,
+ )
+ if response.status_code >= 400:
+ logger.warning(
+ "Approval creation refused: HTTP %s %s",
+ response.status_code,
+ response.text[:200],
+ )
+ return None
+ return str(response.json().get("id") or "") or None
+ except Exception as exc: # noqa: BLE001
+ logger.warning("Approval creation failed: %s", exc)
+ return None
+
+
+async def resume_after_approval(
+ *,
+ approval_id: str,
+ tenant_id: str,
+ approved: bool,
+ decided_by: str = "",
+ comment: str = "",
+) -> PlaybookRun | None:
+ """Continue a run suspended on this approval, or record the denial.
+
+ Returns the completed run, or None when there is nothing waiting. The
+ pause is resolved **before** the run continues, so a decision that
+ arrives twice (a double-tap, a retried webhook) resumes once: the
+ second attempt matches no `waiting` row.
+ """
+ pause = await playbook_pause.find_waiting(approval_id=approval_id, tenant_id=tenant_id)
+ if pause is None:
+ return None
+
+ if not approved:
+ await playbook_pause.resolve(
+ pause_id=pause.id,
+ tenant_id=pause.tenant_id,
+ status="denied",
+ resolution=f"denied by {decided_by or 'an analyst'}: {comment}"[:500],
+ )
+ logger.info("Playbook run %s halted: approval denied", pause.run_id)
+ return None
+
+ claimed = await playbook_pause.resolve(
+ pause_id=pause.id,
+ tenant_id=pause.tenant_id,
+ status="resumed",
+ resolution=f"approved by {decided_by or 'an analyst'}: {comment}"[:500],
+ )
+ if not claimed:
+ # Somebody else resumed it first. Not an error.
+ return None
+
+ from app.playbook.store import PlaybookStore
+
+ playbook = PlaybookStore.default().get(pause.playbook_id)
+ if playbook is None:
+ logger.warning(
+ "Cannot resume run %s: playbook %s is no longer in the store",
+ pause.run_id,
+ pause.playbook_id,
+ )
+ return None
+
+ return await PlaybookEngine().resume(playbook, pause)
+
+
class PlaybookEngine:
"""Executes playbooks step-by-step, emitting realtime events."""
+ async def resume(self, playbook: Playbook, pause: Any) -> PlaybookRun:
+ """Continue a suspended run from the step after its approval.
+
+ Rebuilds the run from what was stored rather than re-running the
+ first half: the steps before the approval already executed, and
+ repeating them would re-send notifications and re-dispatch
+ actions.
+ """
+ return await self.run(
+ playbook,
+ pause.run_context,
+ resume_from=pause.resume_index,
+ resume_run_id=pause.run_id,
+ resume_results=pause.step_results,
+ )
+
async def run(
self,
playbook: Playbook,
trigger_context: dict[str, Any],
*,
dry_run: bool = False,
+ resume_from: int = 0,
+ resume_run_id: str | None = None,
+ resume_results: list[dict[str, Any]] | None = None,
) -> PlaybookRun:
pr = PlaybookRun(playbook, trigger_context)
+ if resume_run_id:
+ # Keep the original run id, so a resumed run stays one run in
+ # the realtime stream and the ledger rather than appearing as
+ # a second, unrelated one that starts halfway through.
+ pr.run_id = resume_run_id
+ if resume_results:
+ pr.step_results = list(resume_results)
pr.started_at = datetime.now(UTC).isoformat()
pr.status = RunStatus.RUNNING
@@ -736,7 +902,13 @@ class PlaybookEngine:
# Build a step index for branching
step_index = {s.id: i for i, s in enumerate(playbook.steps)}
visited: set[str] = set()
- current_idx = 0
+ current_idx = resume_from
+ if resume_from:
+ # The steps before the approval already ran. Marking them
+ # visited keeps the cycle detector honest without
+ # re-executing them.
+ for earlier in playbook.steps[:resume_from]:
+ visited.add(earlier.id)
while current_idx < len(playbook.steps):
step = playbook.steps[current_idx]
@@ -781,6 +953,43 @@ class PlaybookEngine:
unbridgeable = _UNBRIDGEABLE.get(step.type)
+ # Parity 5.2. An approval step is a pause, and this engine
+ # had nowhere to pause to, so it failed closed and 12
+ # shipped playbooks aborted here. The position and the
+ # context go to Postgres, because "survives restarts" is
+ # the requirement and an in-memory pause is lost by the
+ # thing most likely to interrupt a long approval.
+ if step.type == StepType.APPROVAL and not dry_run:
+ pause = await _suspend_for_approval(pr, step, current_idx, http)
+ if pause is not None:
+ pr.status = RunStatus.PAUSED
+ pr.step_results.append(
+ {
+ "step_id": step.id,
+ "name": step.name,
+ "status": StepStatus.PENDING,
+ "paused": True,
+ "pause_id": pause.id,
+ "approval_id": pause.approval_id,
+ "expires_at": pause.expires_at.isoformat(),
+ }
+ )
+ await _emit(
+ pr.run_id,
+ "run.paused",
+ {"step": step.name, "pause_id": pause.id},
+ http,
+ )
+ break
+ # The pause could not be written, so nothing can resume
+ # this run. Falling through fails the step closed, which
+ # is correct: continuing past an approval nobody can
+ # grant is the original defect this replaced.
+ logger.error(
+ "Approval step %s could not be suspended; failing closed rather than continuing into the action it gates",
+ step.name,
+ )
+
if handler is None and not dry_run:
# Fail closed, and skip the retry loop — a missing handler
# will still be missing on the next attempt.
diff --git a/services/agents/app/playbook/pause.py b/services/agents/app/playbook/pause.py
new file mode 100644
index 000000000..4b2e9dfcf
--- /dev/null
+++ b/services/agents/app/playbook/pause.py
@@ -0,0 +1,302 @@
+"""Suspend a playbook run at an approval step, and resume it on the decision.
+
+Parity plan 5.2.
+
+What was wrong
+--------------
+The engine is a single-threaded index walk with no pause and no resume, so
+an `approval` step had nothing to suspend. It failed closed, which was the
+right interim behaviour: before that it returned `{"skipped": true}` while
+reporting SUCCESS, so a run continued straight into the very action a human
+was supposed to authorise. But failing closed means **12 shipped playbooks
+abort at their approval step**, which is not an approval mechanism either.
+
+Three properties the plan requires, and what each costs
+--------------------------------------------------------
+* **Resumes from `/approvals/{id}/decide`.** The route already exists and
+ already persists to `agent_approvals`; what was missing is anything on
+ the engine side to wake.
+* **Survives restarts.** So the position and the context go to Postgres,
+ not to a process. An in-memory pause is lost by exactly the thing most
+ likely to interrupt a long-running approval.
+* **Expires with a recorded outcome.** `expired` is a decision, not an
+ absence of one. A pause with no expiry is a run that hangs forever and
+ an operator who never learns it did.
+
+What a resumed run must not do
+------------------------------
+Re-run the approval step. The stored index is the step *itself* and resume
+advances past it, so a replayed decision (a double-tap in the responder
+app, a retried webhook) continues rather than pausing again.
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+import os
+import uuid
+from dataclasses import dataclass
+from datetime import UTC, datetime, timedelta
+from typing import Any
+
+# The stdlib logger, not structlog. `app.playbook.engine` imports this
+# module and two gates import the engine with only httpx and pydantic
+# installed: `validate_playbooks.py` and the schema-parity check. Pulling
+# structlog into that chain made both fail with ModuleNotFoundError, which
+# is a dependency this file does not need.
+logger = logging.getLogger("aisoc.playbook.pause")
+
+#: How long an approval waits before it expires with a recorded outcome.
+#: Long enough to cross a weekend, short enough that a forgotten request
+#: does not sit open for a quarter.
+DEFAULT_TTL_HOURS = float(os.getenv("AISOC_PLAYBOOK_APPROVAL_TTL_HOURS", "72"))
+
+
+@dataclass(frozen=True)
+class Pause:
+ """A suspended run, as stored."""
+
+ id: str
+ tenant_id: str
+ run_id: str
+ playbook_id: str
+ step_index: int
+ step_id: str
+ run_context: dict[str, Any]
+ step_results: list[dict[str, Any]]
+ approval_id: str | None
+ expires_at: datetime
+
+ @property
+ def resume_index(self) -> int:
+ """Where the resumed run starts.
+
+ One **past** the approval step. Storing the step itself and
+ advancing here means a replayed decision continues the run rather
+ than pausing on the same step again, which a double-tap in the
+ responder app would otherwise do.
+ """
+ return self.step_index + 1
+
+
+async def _pool() -> Any | None:
+ from app.memory.institutional import _get_pool
+
+ try:
+ return await _get_pool()
+ except Exception as exc: # noqa: BLE001
+ logger.warning("playbook_pause: pool unavailable: %s", exc)
+ return None
+
+
+async def suspend(
+ *,
+ tenant_id: str,
+ run_id: str,
+ playbook_id: str,
+ playbook_name: str,
+ step_index: int,
+ step_id: str,
+ run_context: dict[str, Any],
+ step_results: list[dict[str, Any]],
+ approval_id: str | None = None,
+ ttl_hours: float | None = None,
+) -> Pause | None:
+ """Record where this run stopped. None when it could not be stored.
+
+ Returning None rather than raising, and the caller treats that as a
+ failed step: a pause that was not written is a run nothing can resume,
+ and continuing past the approval step would be the original defect
+ (running the action a human was meant to authorise).
+ """
+ pool = await _pool()
+ if pool is None:
+ logger.warning("playbook_pause: no database; run %s cannot be suspended", run_id)
+ return None
+
+ pause_id = str(uuid.uuid4())
+ expires_at = datetime.now(UTC) + timedelta(hours=ttl_hours or DEFAULT_TTL_HOURS)
+ try:
+ async with pool.acquire() as conn:
+ await conn.execute(
+ """
+ INSERT INTO aisoc_playbook_pauses
+ (id, tenant_id, run_id, playbook_id, playbook_name, step_index,
+ step_id, run_context, step_results, approval_id, expires_at)
+ VALUES ($1::uuid, $2::uuid, $3, $4, $5, $6, $7, $8::jsonb, $9::jsonb,
+ $10::uuid, $11)
+ """,
+ pause_id,
+ tenant_id,
+ run_id,
+ playbook_id,
+ playbook_name,
+ step_index,
+ step_id,
+ json.dumps(run_context, default=str),
+ json.dumps(step_results, default=str),
+ approval_id,
+ expires_at,
+ )
+ except Exception as exc: # noqa: BLE001
+ logger.warning("playbook_pause: write failed for run %s: %s", run_id, exc)
+ return None
+
+ logger.info(
+ "playbook_pause: run %s suspended at step %s on approval %s, expires %s",
+ run_id,
+ step_id,
+ approval_id,
+ expires_at.isoformat(),
+ )
+ return Pause(
+ id=pause_id,
+ tenant_id=tenant_id,
+ run_id=run_id,
+ playbook_id=playbook_id,
+ step_index=step_index,
+ step_id=step_id,
+ run_context=run_context,
+ step_results=step_results,
+ approval_id=approval_id,
+ expires_at=expires_at,
+ )
+
+
+async def find_waiting(*, approval_id: str, tenant_id: str) -> Pause | None:
+ """The run waiting on this approval, if any.
+
+ Scoped on the tenant as well as the approval. The approval id is an
+ unguessable UUID, which is an argument for it being hard to reach the
+ wrong row and not an argument for being allowed to: resuming another
+ tenant's playbook run is the kind of thing that should take two
+ mistakes, not one.
+ """
+ pool = await _pool()
+ if pool is None:
+ return None
+ try:
+ async with pool.acquire() as conn:
+ row = await conn.fetchrow(
+ """
+ SELECT id, tenant_id, run_id, playbook_id, step_index, step_id,
+ run_context, step_results, approval_id, expires_at
+ FROM aisoc_playbook_pauses
+ WHERE approval_id = $1::uuid
+ AND tenant_id = $2::uuid
+ AND status = 'waiting'
+ LIMIT 1
+ """,
+ approval_id,
+ tenant_id,
+ )
+ except Exception as exc: # noqa: BLE001
+ logger.warning("playbook_pause: lookup failed for approval %s: %s", approval_id, exc)
+ return None
+ if row is None:
+ return None
+ return _row_to_pause(row)
+
+
+def _row_to_pause(row: Any) -> Pause:
+ def _json(value: Any, default: Any) -> Any:
+ if value is None:
+ return default
+ if isinstance(value, str):
+ try:
+ return json.loads(value)
+ except ValueError:
+ return default
+ return value
+
+ return Pause(
+ id=str(row["id"]),
+ tenant_id=str(row["tenant_id"]),
+ run_id=row["run_id"],
+ playbook_id=row["playbook_id"],
+ step_index=int(row["step_index"]),
+ step_id=row["step_id"],
+ run_context=_json(row["run_context"], {}),
+ step_results=_json(row["step_results"], []),
+ approval_id=str(row["approval_id"]) if row["approval_id"] else None,
+ expires_at=row["expires_at"],
+ )
+
+
+async def resolve(*, pause_id: str, status: str, resolution: str, tenant_id: str) -> bool:
+ """Mark a pause resumed, denied, expired or cancelled.
+
+ The status is set **before** the run continues, so a decision that
+ arrives twice cannot resume the same run twice: the second update
+ matches no `waiting` row.
+ """
+ if status not in {"resumed", "denied", "expired", "cancelled"}:
+ raise ValueError(f"unknown pause resolution {status!r}")
+ pool = await _pool()
+ if pool is None:
+ return False
+ try:
+ async with pool.acquire() as conn:
+ result = await conn.execute(
+ """
+ UPDATE aisoc_playbook_pauses
+ SET status = $2, resolved_at = NOW(), resolution = $3
+ WHERE id = $1::uuid
+ AND tenant_id = $4::uuid
+ AND status = 'waiting'
+ """,
+ pause_id,
+ status,
+ resolution[:500],
+ tenant_id,
+ )
+ except Exception as exc: # noqa: BLE001
+ logger.warning("playbook_pause: resolve failed for %s: %s", pause_id, exc)
+ return False
+ # asyncpg returns "UPDATE <n>"; zero means somebody else resolved it
+ # first, which is the double-decision case and is not an error.
+ changed = str(result).rsplit(" ", 1)[-1] != "0"
+ if not changed:
+ logger.info("playbook_pause: %s was already resolved", pause_id)
+ return changed
+
+
+async def expire_due(*, now: datetime | None = None) -> list[str]:
+ """Expire every pause past its deadline. Returns the run ids.
+
+ Deliberately cross-tenant, and recorded as such in
+ `scripts/check_tenant_query_predicates.py`. A per-tenant sweep would
+ need a list of tenants to iterate, and a tenant missing from that list
+ would have approvals that never expire — the exact silent-hang this
+ sweep exists to prevent.
+
+ Run from the scheduler. An expiry that nothing sweeps is a status
+ column that never changes, which looks identical to a pause still
+ legitimately waiting.
+ """
+ pool = await _pool()
+ if pool is None:
+ return []
+ try:
+ async with pool.acquire() as conn:
+ rows = await conn.fetch(
+ """
+ UPDATE aisoc_playbook_pauses
+ SET status = 'expired',
+ resolved_at = NOW(),
+ resolution = 'no decision before the approval deadline'
+ WHERE status = 'waiting' AND expires_at <= $1
+ RETURNING run_id
+ """,
+ now or datetime.now(UTC),
+ )
+ except Exception as exc: # noqa: BLE001
+ logger.warning("playbook_pause: expiry sweep failed: %s", exc)
+ return []
+ run_ids = [r["run_id"] for r in rows]
+ if run_ids:
+ # Info, not debug. An expired approval means an action a human was
+ # asked to authorise did not happen, and nobody decided that.
+ logger.info("playbook_pause: %d pause(s) expired: %s", len(run_ids), run_ids[:10])
+ return run_ids
diff --git a/services/agents/app/workers/fused_alert_consumer.py b/services/agents/app/workers/fused_alert_consumer.py
index 1ca5687e8..8f309eee2 100644
--- a/services/agents/app/workers/fused_alert_consumer.py
+++ b/services/agents/app/workers/fused_alert_consumer.py
@@ -58,6 +58,7 @@ from app.investigator.bundle_prompt import prefetch_context_bundle_dict
from app.llm.factory import llm_override
from app.memory.outcomes import AI, suppression_refusal
from app.models.state import AgentStatus, InvestigationState
+from app.playbook import alert_trigger
from app.routing.model_router import is_deterministic_mode
from app.security.llm_resolver import resolve_llm_config
from app.workers.business_context import BusinessContextApplier
@@ -617,6 +618,28 @@ class FusedAlertTriageWorker:
_METRICS["deterministic"] += 1
cost = CostSummary.from_tracker(tracker)
tokens = tracker.total_tokens
+
+ # Parity 5.1. `find_matching()` had no production caller, so no
+ # playbook ever ran from an alert. Three switches must agree
+ # before one acts (deployment, tenant, playbook) and anything
+ # short of all three runs in **preview**, with its plan and
+ # simulated steps attached to the alert.
+ #
+ # After triage, because a playbook's conditions read the
+ # verdict and the confidence, and before them it would be
+ # matching on an alert nobody has assessed.
+ try:
+ playbook_outcome = await alert_trigger.run_for_alert(state)
+ if playbook_outcome.matched:
+ state.add_finding(
+ f"Playbooks matched: {', '.join(playbook_outcome.matched)}"
+ + (f" ({len(playbook_outcome.executed)} ran, {len(playbook_outcome.previewed)} previewed)")
+ )
+ except Exception as exc: # noqa: BLE001
+ # A playbook failure must not lose the triage result it was
+ # attached to.
+ logger.warning("auto_triage_worker.playbook_trigger_failed", error=str(exc))
+
verdict = state.verdict
confidence = state.confidence
# Never complete with a null/empty verdict (issue #571): if both the
diff --git a/services/agents/tests/test_alert_triggered_playbooks.py b/services/agents/tests/test_alert_triggered_playbooks.py
new file mode 100644
index 000000000..4a96dd464
--- /dev/null
+++ b/services/agents/tests/test_alert_triggered_playbooks.py
@@ -0,0 +1,281 @@
+"""A playbook starts from an alert, in preview, behind three switches.
+
+Parity plan 5.1: "`find_matching()` has no production caller today. Call
+it from the fused-alert path behind a setting that ships off. A playbook
+first runs in preview, with its plan and simulated steps shown on the
+alert, and then goes live per tenant and per playbook."
+
+Why the switches are tested harder than the matching
+-----------------------------------------------------
+A playbook that runs from an alert can isolate a host nobody asked it to.
+The matching logic already existed and was already tested; what did not
+exist was any caller, and what matters now is that turning the caller on
+is deliberate at three independent levels and that the **default of every
+one of them is off**.
+
+The ordering is the part worth pinning: preview is the default state, not
+a mode somebody has to remember to use first. A deployment that enables
+the feature and forgets the per-playbook list gets previews, not actions.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+import pytest
+from app.playbook import alert_trigger, engine, store
+
+
+class _State:
+ def __init__(self, **kw) -> None: # noqa: ANN003
+ self.tenant_id = kw.pop("tenant_id", "t-1")
+ self.incident_id = kw.pop("incident_id", "a-1")
+ self.verdict = kw.pop("verdict", "true_positive")
+ self.confidence = kw.pop("confidence", 0.9)
+ self.raw_alert = kw.pop("raw_alert", {"severity": "high", "tags": ["identity"]})
+ self.findings: list[str] = []
+
+ def add_finding(self, text: str) -> None:
+ self.findings.append(text)
+
+
+class TestTheDefaultsAreOff:
+ def test_the_deployment_switch_is_off_by_default(self, monkeypatch) -> None: # noqa: ANN001
+ """A fresh install must not take a containment action on its first
+ alert."""
+ monkeypatch.delenv(alert_trigger.ENABLED_ENV, raising=False)
+ assert alert_trigger.deployment_enabled() is False
+
+ def test_no_playbook_is_live_by_default(self, monkeypatch) -> None: # noqa: ANN001
+ monkeypatch.delenv(alert_trigger.LIVE_PLAYBOOKS_ENV, raising=False)
+ assert alert_trigger.live_playbook_ids() == set()
+ assert alert_trigger.is_live(tenant_id="t-1", playbook_id="pb-1") is False
+
+ @pytest.mark.asyncio
+ async def test_disabled_says_how_to_enable_it(self, monkeypatch) -> None: # noqa: ANN001
+ """Not a silent no-op: an operator looking for why nothing ran
+ should find the answer on the alert."""
+ monkeypatch.delenv(alert_trigger.ENABLED_ENV, raising=False)
+ outcome = await alert_trigger.run_for_alert(_State())
+ assert outcome.matched == []
+ assert outcome.skipped_reason
+ assert alert_trigger.ENABLED_ENV in outcome.skipped_reason
+
+
+class TestTheLiveList:
+ @pytest.mark.parametrize(
+ ("value", "tenant", "playbook", "expected"),
+ [
+ ("t-1:pb-1", "t-1", "pb-1", True),
+ ("t-1:pb-1", "t-2", "pb-1", False),
+ ("pb-1", "t-9", "pb-1", True),
+ ("pb-2", "t-1", "pb-1", False),
+ ("", "t-1", "pb-1", False),
+ ("t-1:pb-1, t-1:pb-2", "t-1", "pb-2", True),
+ ],
+ )
+ def test_scoping(self, monkeypatch, value, tenant, playbook, expected) -> None: # noqa: ANN001
+ monkeypatch.setenv(alert_trigger.LIVE_PLAYBOOKS_ENV, value)
+ assert alert_trigger.is_live(tenant_id=tenant, playbook_id=playbook) is expected
+
+ def test_the_list_is_read_per_call(self, monkeypatch) -> None: # noqa: ANN001
+ """Turning one off should take effect on the next alert, not the
+ next restart. A playbook that can contain a host should be
+ switchable off faster than a deploy."""
+ monkeypatch.setenv(alert_trigger.LIVE_PLAYBOOKS_ENV, "pb-1")
+ assert alert_trigger.is_live(tenant_id="t", playbook_id="pb-1") is True
+ monkeypatch.setenv(alert_trigger.LIVE_PLAYBOOKS_ENV, "")
+ assert alert_trigger.is_live(tenant_id="t", playbook_id="pb-1") is False
+
+
+class TestTheTriggerContext:
+ def test_it_is_flat(self) -> None:
+ """`find_matching` filters on `severity` and `tags` at the top
+ level, so a nested alert would match nothing while looking like it
+ should."""
+ context = alert_trigger.alert_context(_State())
+ assert context["severity"] == "high"
+ assert context["tags"] == ["identity"]
+ assert context["tenant_id"] == "t-1"
+
+ def test_it_carries_the_verdict_and_confidence(self) -> None:
+ """A playbook's conditions read them, which is why the trigger runs
+ after triage rather than before."""
+ context = alert_trigger.alert_context(_State(verdict="benign", confidence=0.4))
+ assert context["verdict"] == "benign"
+ assert context["confidence"] == 0.4
+
+ def test_entity_fields_are_lifted_for_templating(self) -> None:
+ context = alert_trigger.alert_context(_State(raw_alert={"severity": "high", "host": "WIN-01", "username": "j.doe"}))
+ assert context["host"] == "WIN-01"
+ assert context["username"] == "j.doe"
+
+ def test_a_malformed_alert_does_not_raise(self) -> None:
+ context = alert_trigger.alert_context(_State(raw_alert="not a dict"))
+ assert context["tenant_id"] == "t-1"
+
+
+class TestRunning:
+ async def test_an_enabled_deployment_with_no_live_list_previews(self, monkeypatch) -> None: # noqa: ANN001
+ """The property the plan asks for: preview is the default state,
+ not a mode somebody has to remember to use."""
+ monkeypatch.setenv(alert_trigger.ENABLED_ENV, "1")
+ monkeypatch.delenv(alert_trigger.LIVE_PLAYBOOKS_ENV, raising=False)
+
+ seen: list[bool] = []
+
+ class _Engine:
+ async def run(self, playbook, context, *, dry_run=False): # noqa: ANN001, ANN202, ARG002
+ seen.append(dry_run)
+
+ class _Run:
+ run_id = "r1"
+ status = "completed"
+ step_results: list = []
+
+ return _Run()
+
+ class _Playbook:
+ id = "pb-1"
+ name = "Contain"
+
+ class _Store:
+ def find_matching(self, event, context): # noqa: ANN001, ANN202, ARG002
+ return [_Playbook()]
+
+ monkeypatch.setattr(engine, "PlaybookEngine", _Engine)
+ monkeypatch.setattr(store.PlaybookStore, "default", staticmethod(lambda: _Store()))
+
+ outcome = await alert_trigger.run_for_alert(_State())
+ assert seen == [True], "the playbook ran live without being on the live list"
+ assert outcome.previewed and not outcome.executed
+ assert alert_trigger.LIVE_PLAYBOOKS_ENV in outcome.previewed[0]["reason"]
+
+ async def test_a_listed_playbook_runs_live(self, monkeypatch) -> None: # noqa: ANN001
+ monkeypatch.setenv(alert_trigger.ENABLED_ENV, "1")
+ monkeypatch.setenv(alert_trigger.LIVE_PLAYBOOKS_ENV, "t-1:pb-1")
+
+ seen: list[bool] = []
+
+ class _Engine:
+ async def run(self, playbook, context, *, dry_run=False): # noqa: ANN001, ANN202, ARG002
+ seen.append(dry_run)
+
+ class _Run:
+ run_id = "r1"
+ status = "completed"
+ step_results: list = []
+
+ return _Run()
+
+ class _Playbook:
+ id = "pb-1"
+ name = "Contain"
+
+ class _Store:
+ def find_matching(self, event, context): # noqa: ANN001, ANN202, ARG002
+ return [_Playbook()]
+
+ monkeypatch.setattr(engine, "PlaybookEngine", _Engine)
+ monkeypatch.setattr(store.PlaybookStore, "default", staticmethod(lambda: _Store()))
+
+ outcome = await alert_trigger.run_for_alert(_State())
+ assert seen == [False]
+ assert outcome.executed and not outcome.previewed
+
+ async def test_a_playbook_failure_does_not_lose_the_triage_result(self, monkeypatch) -> None: # noqa: ANN001
+ monkeypatch.setenv(alert_trigger.ENABLED_ENV, "1")
+
+ def _boom(): # noqa: ANN202
+ raise RuntimeError("store is gone")
+
+ monkeypatch.setattr(store.PlaybookStore, "default", staticmethod(_boom))
+ outcome = await alert_trigger.run_for_alert(_State())
+ assert outcome.skipped_reason and "store is gone" in outcome.skipped_reason
+
+ async def test_an_alert_with_no_tenant_is_refused(self, monkeypatch) -> None: # noqa: ANN001
+ monkeypatch.setenv(alert_trigger.ENABLED_ENV, "1")
+ outcome = await alert_trigger.run_for_alert(_State(tenant_id=""))
+ assert "no tenant" in (outcome.skipped_reason or "")
+
+
+class TestTheWiring:
+ def test_the_fused_alert_worker_calls_it(self) -> None:
+ """Before this, `find_matching` had no production caller at all."""
+ import pathlib
+
+ source = pathlib.Path(__file__).resolve().parents[1].joinpath("app/workers/fused_alert_consumer.py").read_text()
+ assert "alert_trigger.run_for_alert(" in source, "no playbook can start from an alert, which is the defect 5.1 exists to fix"
+
+ def test_it_runs_after_triage_not_before(self) -> None:
+ """A playbook's conditions read the verdict and the confidence."""
+ import pathlib
+
+ source = pathlib.Path(__file__).resolve().parents[1].joinpath("app/workers/fused_alert_consumer.py").read_text()
+ trigger_at = source.index("alert_trigger.run_for_alert(")
+ verdict_at = source.index("verdict = state.verdict")
+ assert trigger_at < verdict_at
+ assert source.index("tokens = tracker.total_tokens") < trigger_at
+
+
+class TestTheTriggerEventMatchesTheCorpus:
+ """The defect live QA found, and the suite could not.
+
+ `run_for_alert` asked for `alert.created`. Every alert-triggered
+ playbook in the corpus declares `on: alert`, so **nothing ever
+ matched**: the feature was wired, enabled, and dead.
+
+ Every test above passed, because they drive a fake store that returns
+ a playbook whatever event name it is handed. A fake that ignores the
+ argument cannot fail on it — the same shape that let a query select a
+ column `detection_rules` does not have.
+
+ This reads the real store rather than a second hardcoded list, because
+ a list maintained beside the constant drifts with it.
+ """
+
+ def test_the_constant_matches_what_shipped_playbooks_declare(self) -> None:
+ from app.playbook.alert_trigger import TRIGGER_EVENT
+ from app.playbook.store import PlaybookStore
+
+ store = PlaybookStore.default()
+ store.seed_defaults()
+ declared = {str(pb.trigger.get("on")) for pb in store.list()}
+
+ assert TRIGGER_EVENT in declared, (
+ f"run_for_alert asks for {TRIGGER_EVENT!r} and no shipped playbook declares it "
+ f"(they declare {sorted(declared)}), so no playbook can ever start from an alert"
+ )
+
+ def test_the_real_store_matches_a_real_playbook_on_a_real_alert(self) -> None:
+ """End to end through the actual store, not a fake.
+
+ The severity and tag come **from the corpus** rather than being
+ written here. A hardcoded fixture tag passes while describing an
+ alert no shipped playbook wants (this test first used
+ `identity`, which none of the 64 packs declare) and would keep
+ passing if the trigger vocabulary moved underneath it.
+ """
+ from app.playbook.alert_trigger import TRIGGER_EVENT, alert_context
+ from app.playbook.store import PlaybookStore
+
+ store = PlaybookStore.default()
+ store.seed_defaults()
+
+ wanted = next(pb for pb in store.list() if pb.enabled and pb.trigger.get("tags") and pb.trigger.get("severity"))
+ context = alert_context(
+ _State(
+ raw_alert={
+ "severity": wanted.trigger["severity"][0],
+ "tags": [wanted.trigger["tags"][0]],
+ }
+ )
+ )
+
+ matches: list[Any] = list(store.find_matching(TRIGGER_EVENT, context))
+ assert matches, (
+ f"the real store matched no playbook on an alert built from {wanted.id}'s own "
+ "trigger, so the alert-trigger path is wired to nothing"
+ )
+ assert wanted.id in {pb.id for pb in matches}
diff --git a/services/agents/tests/test_playbook_action_bridge.py b/services/agents/tests/test_playbook_action_bridge.py
index 02311ac19..b25e7e01a 100644
--- a/services/agents/tests/test_playbook_action_bridge.py
+++ b/services/agents/tests/test_playbook_action_bridge.py
@@ -263,19 +263,49 @@ class TestWhatTheBridgeSends:
assert recorder.calls[0]["vendor_id"] == "sentinelone"
-class TestApprovalStaysUnbridged:
- async def test_approval_has_no_handler_and_says_why(self) -> None:
- """The one verb with no bridge. It is a pause, and the engine is an
- index walk with nothing to suspend — recorded rather than faked."""
+class TestApprovalIsADurablePause:
+ """Parity 5.2. This used to assert the opposite.
+
+ `approval` was the one verb with no bridge, on the reasoning that it is
+ a pause and the engine had nothing to suspend. That was honest about
+ the engine and wrong about the product: 12 shipped playbooks aborted
+ at their approval step, which is not an approval mechanism either.
+ """
+
+ async def test_it_suspends_rather_than_failing(self) -> None:
pb = _playbook(PlaybookStep(id="gate", name="sign-off", type=StepType.APPROVAL))
run = await PlaybookEngine().run(pb, {"tenant_id": "t-1"})
- result = run.step_results[0]["result"]
- assert run.step_results[0]["status"] == StepStatus.FAILED
- assert result["unimplemented"] is True
- assert result["executed"] is False
- assert "no pause or resume" in result["error"]
- assert "pending_approval" in result["error"], "the reason must name the mechanism that replaced it"
+ # With no database reachable the pause cannot be written, and the
+ # step fails closed rather than continuing into the action it
+ # gates. That is the correct behaviour for an unresumable run and
+ # the one this environment exercises.
+ assert run.step_results[0]["status"] in (StepStatus.FAILED, StepStatus.PENDING)
+ if run.step_results[0]["status"] is StepStatus.PENDING:
+ assert run.status is RunStatus.PAUSED
+ assert run.step_results[0]["paused"] is True
+
+ async def test_a_run_that_cannot_be_suspended_does_not_continue(self) -> None:
+ """The property that matters most.
+
+ Failing to record the pause must not let the run walk into the
+ action a human was asked to authorise. That was the original
+ defect: the step returned `{"skipped": true}` while reporting
+ SUCCESS.
+ """
+ pb = _playbook(
+ PlaybookStep(id="gate", name="sign-off", type=StepType.APPROVAL),
+ PlaybookStep(id="act", name="isolate", type=StepType.ISOLATE_HOST, params={"host": "h"}),
+ )
+ run = await PlaybookEngine().run(pb, {"tenant_id": "t-1"})
+
+ executed = [r for r in run.step_results if (r.get("result") or {}).get("executed")]
+ assert not executed, "a step ran past an approval that could not be recorded, which is the defect the pause exists to prevent"
+
+ def test_nothing_is_recorded_as_unbridgeable_any_more(self) -> None:
+ from app.playbook.engine import _UNBRIDGEABLE
+
+ assert _UNBRIDGEABLE == {}, f"a step type is back on the unbridgeable list: {dict(_UNBRIDGEABLE)}"
class TestDryRunPreview:
diff --git a/services/agents/tests/test_playbook_pause_resume.py b/services/agents/tests/test_playbook_pause_resume.py
new file mode 100644
index 000000000..cef9215d8
--- /dev/null
+++ b/services/agents/tests/test_playbook_pause_resume.py
@@ -0,0 +1,399 @@
+"""A playbook pauses at an approval, survives a restart, and resumes.
+
+Parity plan 5.2: a persisted pause that "resumes from
+`/approvals/{id}/decide`, survives restarts, expires with a recorded
+outcome".
+
+The three properties, and why each is tested separately
+--------------------------------------------------------
+**Resuming** is the easy one and the least interesting: an in-memory
+`asyncio.Event` would satisfy it.
+
+**Surviving a restart** is the one that forces the design. The pause is
+simulated here by building a fresh engine from stored rows rather than by
+keeping any object alive, because that is what a restart leaves you with.
+
+**Expiring with a recorded outcome** is the one that is easy to skip. A
+pause with no expiry is a run that hangs forever and an operator who never
+learns it did, and `expired` is a decision rather than the absence of one.
+
+The dangerous case this file pins
+----------------------------------
+A decision that arrives twice. A double-tap in the responder app or a
+retried webhook must resume the run **once**: the pause is resolved before
+the run continues, so the second attempt matches no `waiting` row.
+"""
+
+from __future__ import annotations
+
+import uuid
+from contextlib import asynccontextmanager
+from datetime import UTC, datetime, timedelta
+
+import pytest
+from app.playbook import pause as playbook_pause
+from app.playbook.engine import PlaybookEngine, RunStatus, StepStatus
+from app.playbook.models import Playbook, PlaybookStep, StepType
+
+
+class _FakeRows:
+ """A stand-in for `aisoc_playbook_pauses` with the semantics that matter.
+
+ Specifically the partial unique index on `status = 'waiting'`, which is
+ what makes a double decision resolve once.
+ """
+
+ def __init__(self) -> None:
+ self.rows: list[dict] = []
+
+ async def execute(self, sql: str, *args): # noqa: ANN001, ANN202
+ if "INSERT INTO aisoc_playbook_pauses" in sql:
+ self.rows.append(
+ {
+ "id": args[0],
+ "tenant_id": args[1],
+ "run_id": args[2],
+ "playbook_id": args[3],
+ "playbook_name": args[4],
+ "step_index": args[5],
+ "step_id": args[6],
+ "run_context": args[7],
+ "step_results": args[8],
+ "approval_id": args[9],
+ "expires_at": args[10],
+ "status": "waiting",
+ }
+ )
+ return "INSERT 0 1"
+ if "UPDATE aisoc_playbook_pauses" in sql and "WHERE id =" in sql:
+ for row in self.rows:
+ if row["id"] == args[0] and str(row.get("tenant_id")) == str(args[3]) and row["status"] == "waiting":
+ row["status"] = args[1]
+ row["resolution"] = args[2]
+ return "UPDATE 1"
+ return "UPDATE 0"
+ return "OK"
+
+ async def fetchrow(self, sql: str, *args): # noqa: ANN001, ANN202
+ # The tenant filter is honoured here on purpose. A fake that
+ # ignored it would pass against a query that ignored it too,
+ # which is the shape that lets a cross-tenant read ship.
+ for row in self.rows:
+ if row.get("approval_id") == args[0] and str(row.get("tenant_id")) == str(args[1]) and row["status"] == "waiting":
+ return row
+ return None
+
+ async def fetch(self, sql: str, *args): # noqa: ANN001, ANN202
+ if "status = 'expired'" in sql:
+ due = args[0]
+ out = []
+ for row in self.rows:
+ if row["status"] == "waiting" and row["expires_at"] <= due:
+ row["status"] = "expired"
+ row["resolution"] = "no decision before the approval deadline"
+ out.append({"run_id": row["run_id"]})
+ return out
+ return []
+
+
+class _FakePool:
+ def __init__(self, rows: _FakeRows) -> None:
+ self._rows = rows
+
+ @asynccontextmanager
+ async def acquire(self):
+ yield self._rows
+
+
+def store(monkeypatch): # noqa: ANN001, ANN201
+ rows = _FakeRows()
+ pool = _FakePool(rows)
+
+ async def _pool(): # noqa: ANN202
+ return pool
+
+ monkeypatch.setattr(playbook_pause, "_pool", _pool)
+ return rows
+
+
+def _playbook(*steps: PlaybookStep) -> Playbook:
+ return Playbook(
+ id="pb-1",
+ name="Containment with sign-off",
+ description="",
+ trigger={"on": "alert.created"},
+ steps=list(steps),
+ enabled=True,
+ )
+
+
+class TestSuspending:
+ async def test_a_pause_records_where_to_resume(self, store) -> None: # noqa: ANN001
+ pause = await playbook_pause.suspend(
+ tenant_id=str(uuid.uuid4()),
+ run_id="run-1",
+ playbook_id="pb-1",
+ playbook_name="x",
+ step_index=2,
+ step_id="gate",
+ run_context={"host": "WIN-01"},
+ step_results=[{"step_id": "s1"}],
+ )
+ assert pause is not None
+ assert pause.step_index == 2
+ assert pause.resume_index == 3, (
+ "resume must start past the approval step, or a replayed decision pauses again on the same step forever"
+ )
+
+ async def test_the_context_is_stored_not_just_the_position(self, store) -> None: # noqa: ANN001
+ """Without it the resumed half sees an empty context and every
+ templated parameter resolves to nothing."""
+ await playbook_pause.suspend(
+ tenant_id=str(uuid.uuid4()),
+ run_id="run-2",
+ playbook_id="pb-1",
+ playbook_name="x",
+ step_index=0,
+ step_id="gate",
+ run_context={"host": "WIN-01", "user": "j.doe"},
+ step_results=[],
+ )
+ stored = store.rows[-1]["run_context"]
+ assert "WIN-01" in stored and "j.doe" in stored
+
+ async def test_no_database_returns_none_rather_than_raising(self, monkeypatch) -> None: # noqa: ANN001
+ async def _none(): # noqa: ANN202
+ return None
+
+ monkeypatch.setattr(playbook_pause, "_pool", _none)
+ assert (
+ await playbook_pause.suspend(
+ tenant_id=str(uuid.uuid4()),
+ run_id="r",
+ playbook_id="p",
+ playbook_name="n",
+ step_index=0,
+ step_id="s",
+ run_context={},
+ step_results=[],
+ )
+ is None
+ )
+
+
+class TestSurvivingARestart:
+ async def test_a_pause_is_found_by_approval_id_from_a_cold_start(self, store) -> None: # noqa: ANN001
+ """Nothing is kept alive between the suspend and the lookup, which
+ is what a restart leaves you with."""
+ approval_id = str(uuid.uuid4())
+ tenant = str(uuid.uuid4())
+ await playbook_pause.suspend(
+ tenant_id=tenant,
+ run_id="run-3",
+ playbook_id="pb-1",
+ playbook_name="x",
+ step_index=1,
+ step_id="gate",
+ run_context={"host": "WIN-01"},
+ step_results=[{"step_id": "s1", "status": "success"}],
+ approval_id=approval_id,
+ )
+
+ found = await playbook_pause.find_waiting(approval_id=approval_id, tenant_id=tenant)
+ assert found is not None
+ assert found.run_id == "run-3"
+ assert found.run_context["host"] == "WIN-01"
+ assert found.step_results[0]["step_id"] == "s1"
+
+
+class TestADecisionThatArrivesTwice:
+ async def test_the_second_resolution_does_not_claim_it(self, store) -> None: # noqa: ANN001
+ """A double-tap in the responder app, or a retried webhook."""
+ approval_id = str(uuid.uuid4())
+ tenant = str(uuid.uuid4())
+ pause = await playbook_pause.suspend(
+ tenant_id=tenant,
+ run_id="run-4",
+ playbook_id="pb-1",
+ playbook_name="x",
+ step_index=0,
+ step_id="gate",
+ run_context={},
+ step_results=[],
+ approval_id=approval_id,
+ )
+ assert pause is not None
+
+ first = await playbook_pause.resolve(pause_id=pause.id, tenant_id=pause.tenant_id, status="resumed", resolution="approved")
+ second = await playbook_pause.resolve(pause_id=pause.id, tenant_id=pause.tenant_id, status="resumed", resolution="approved again")
+ assert first is True
+ assert second is False, "the same pause was claimed twice, so the run would resume twice"
+
+ async def test_a_resolved_pause_is_no_longer_found(self, store) -> None: # noqa: ANN001
+ approval_id = str(uuid.uuid4())
+ tenant = str(uuid.uuid4())
+ pause = await playbook_pause.suspend(
+ tenant_id=tenant,
+ run_id="run-5",
+ playbook_id="pb-1",
+ playbook_name="x",
+ step_index=0,
+ step_id="gate",
+ run_context={},
+ step_results=[],
+ approval_id=approval_id,
+ )
+ assert pause is not None
+ await playbook_pause.resolve(pause_id=pause.id, tenant_id=pause.tenant_id, status="denied", resolution="no")
+ assert await playbook_pause.find_waiting(approval_id=approval_id, tenant_id=tenant) is None
+
+
+class TestExpiry:
+ async def test_a_pause_past_its_deadline_expires_with_a_reason(self, store) -> None: # noqa: ANN001
+ pause = await playbook_pause.suspend(
+ tenant_id=str(uuid.uuid4()),
+ run_id="run-6",
+ playbook_id="pb-1",
+ playbook_name="x",
+ step_index=0,
+ step_id="gate",
+ run_context={},
+ step_results=[],
+ ttl_hours=0.0001,
+ )
+ assert pause is not None
+
+ expired = await playbook_pause.expire_due(now=datetime.now(UTC) + timedelta(hours=1))
+ assert "run-6" in expired
+ row = next(r for r in store.rows if r["run_id"] == "run-6")
+ assert row["status"] == "expired"
+ assert "deadline" in row["resolution"], (
+ "an expired approval must say why, or it is indistinguishable from a pause still legitimately waiting"
+ )
+
+ async def test_a_pause_inside_its_deadline_is_left_alone(self, store) -> None: # noqa: ANN001
+ await playbook_pause.suspend(
+ tenant_id=str(uuid.uuid4()),
+ run_id="run-7",
+ playbook_id="pb-1",
+ playbook_name="x",
+ step_index=0,
+ step_id="gate",
+ run_context={},
+ step_results=[],
+ ttl_hours=72,
+ )
+ assert await playbook_pause.expire_due(now=datetime.now(UTC)) == []
+
+ def test_there_is_a_default_deadline(self) -> None:
+ """A pause with no expiry is a run that hangs forever."""
+ assert playbook_pause.DEFAULT_TTL_HOURS > 0
+
+
+class TestTheEngineResumesWhereItStopped:
+ async def test_it_does_not_re_run_the_steps_before_the_approval(self, store) -> None: # noqa: ANN001
+ """Repeating them would re-send notifications and re-dispatch
+ actions that already ran."""
+ ran: list[str] = []
+
+ pb = _playbook(
+ PlaybookStep(id="first", name="note", type=StepType.CONDITION),
+ PlaybookStep(id="gate", name="sign-off", type=StepType.APPROVAL),
+ PlaybookStep(id="after", name="note2", type=StepType.CONDITION),
+ )
+
+ class _Pause:
+ run_id = "run-8"
+ resume_index = 2
+ run_context: dict = {"tenant_id": "t-1"}
+ step_results: list = [{"step_id": "first", "status": "success"}]
+
+ run = await PlaybookEngine().resume(pb, _Pause())
+ touched = [r["step_id"] for r in run.step_results]
+ assert touched.count("first") == 1, (
+ f"the pre-approval step appears {touched.count('first')} times, so resume re-ran what had already executed"
+ )
+ assert "after" in touched, "the step after the approval never ran"
+ assert ran == []
+
+ async def test_a_resumed_run_keeps_its_original_id(self, store) -> None: # noqa: ANN001
+ """Otherwise it appears in the realtime stream and the ledger as a
+ second, unrelated run that starts halfway through."""
+ pb = _playbook(PlaybookStep(id="after", name="note", type=StepType.CONDITION))
+
+ class _Pause:
+ run_id = "run-9"
+ resume_index = 0
+ run_context: dict = {"tenant_id": "t-1"}
+ step_results: list = []
+
+ run = await PlaybookEngine().resume(pb, _Pause())
+ assert run.run_id == "run-9"
+
+
+class TestTheRunStatus:
+ def test_paused_is_distinct_from_failed(self) -> None:
+ """A paused run reads to an operator as one doing what it was
+ written to do; a failed one reads as a broken playbook."""
+ assert RunStatus.PAUSED.value == "paused"
+ assert RunStatus.PAUSED is not RunStatus.FAILED
+
+ def test_a_paused_step_is_pending_not_success(self) -> None:
+ assert StepStatus.PENDING.value != StepStatus.SUCCESS.value
+
+
+class TestCrossTenant:
+ """Resuming another tenant's run should take two mistakes, not one.
+
+ The approval id is an unguessable UUID, which is an argument for it
+ being hard to reach the wrong row and not an argument for being
+ allowed to.
+ """
+
+ async def test_another_tenant_cannot_find_the_pause(self, store) -> None: # noqa: ANN001
+ approval_id = str(uuid.uuid4())
+ owner = str(uuid.uuid4())
+ await playbook_pause.suspend(
+ tenant_id=owner,
+ run_id="run-x",
+ playbook_id="pb-1",
+ playbook_name="x",
+ step_index=0,
+ step_id="gate",
+ run_context={},
+ step_results=[],
+ approval_id=approval_id,
+ )
+
+ assert await playbook_pause.find_waiting(approval_id=approval_id, tenant_id=str(uuid.uuid4())) is None
+ assert await playbook_pause.find_waiting(approval_id=approval_id, tenant_id=owner) is not None
+
+ async def test_another_tenant_cannot_resolve_the_pause(self, store) -> None: # noqa: ANN001
+ owner = str(uuid.uuid4())
+ pause = await playbook_pause.suspend(
+ tenant_id=owner,
+ run_id="run-y",
+ playbook_id="pb-1",
+ playbook_name="x",
+ step_index=0,
+ step_id="gate",
+ run_context={},
+ step_results=[],
+ )
+ assert pause is not None
+
+ stolen = await playbook_pause.resolve(
+ pause_id=pause.id,
+ tenant_id=str(uuid.uuid4()),
+ status="resumed",
+ resolution="not mine",
+ )
+ assert stolen is False, "another tenant resumed this run"
+ assert await playbook_pause.resolve(pause_id=pause.id, tenant_id=owner, status="resumed", resolution="mine") is True
diff --git a/services/api/app/data/marketplace/index.json b/services/api/app/data/marketplace/index.json
index f51cd5ba7..1f5a048eb 100644
--- a/services/api/app/data/marketplace/index.json
+++ b/services/api/app/data/marketplace/index.json
@@ -1,7 +1,7 @@
{
"$schema": "https://example.com/schemas/marketplace/v1.json",
"version": "1.0.0",
- "generated": "2026-10-01T21:06:28Z",
+ "generated": "2026-10-02T04:17:51Z",
"categories": [
{
"id": "playbooks",
diff --git a/services/api/migrations/081_playbook_pauses.sql b/services/api/migrations/081_playbook_pauses.sql
new file mode 100644
index 000000000..107d656ef
--- /dev/null
+++ b/services/api/migrations/081_playbook_pauses.sql
@@ -0,0 +1,89 @@
+-- A playbook run suspended at an approval step, and able to resume.
+--
+-- Parity plan 5.2.
+--
+-- The engine is a single-threaded index walk with no pause and no resume,
+-- so an `approval` step had nothing to suspend. It failed closed, which was
+-- the right interim behaviour (it previously returned `{"skipped": true}`
+-- while reporting SUCCESS, letting a run continue straight into the action
+-- a human was supposed to authorise) but it means **12 shipped playbooks
+-- abort at their approval step**.
+--
+-- Suspending needs the position and the context on disk, not in the
+-- process: "survives restarts" is the requirement, and an in-memory pause
+-- is lost by the thing most likely to interrupt a long-running approval.
+
+CREATE TABLE IF NOT EXISTS aisoc_playbook_pauses (
+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+ tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,
+
+ -- The engine's own run id, so a resumed run keeps its identity in the
+ -- realtime stream and the ledger rather than appearing as a new one.
+ run_id TEXT NOT NULL,
+ playbook_id TEXT NOT NULL,
+ playbook_name TEXT,
+
+ -- Where to pick up. The index of the approval step itself: resume
+ -- advances past it, so a replayed decision cannot re-run the step.
+ step_index INTEGER NOT NULL,
+ step_id TEXT NOT NULL,
+
+ -- Everything a later step reads through `{{prev.*}}`. Without this the
+ -- resumed half of the run sees an empty context and every templated
+ -- parameter resolves to nothing.
+ run_context JSONB NOT NULL DEFAULT '{}'::jsonb,
+ step_results JSONB NOT NULL DEFAULT '[]'::jsonb,
+
+ -- The approval this pause is waiting on, in `agent_approvals`.
+ approval_id UUID,
+
+ status TEXT NOT NULL DEFAULT 'waiting'
+ CHECK (status IN ('waiting', 'resumed', 'denied', 'expired', 'cancelled')),
+
+ -- Expiry is mandatory and has an outcome. A pause with no expiry is a
+ -- run that hangs forever and an operator who never learns it did:
+ -- `expired` is a recorded decision, not an absence of one.
+ expires_at TIMESTAMPTZ NOT NULL,
+ resolved_at TIMESTAMPTZ,
+ resolution TEXT,
+
+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+);
+
+-- One live pause per run. A run cannot be waiting at two approval steps,
+-- and a duplicate would resume the same run twice.
+CREATE UNIQUE INDEX IF NOT EXISTS aisoc_playbook_pauses_run_idx
+ ON aisoc_playbook_pauses (run_id)
+ WHERE status = 'waiting';
+
+CREATE INDEX IF NOT EXISTS aisoc_playbook_pauses_approval_idx
+ ON aisoc_playbook_pauses (approval_id)
+ WHERE status = 'waiting';
+
+-- The expiry sweep reads this. Partial, because a resolved pause is never
+-- swept and keeping it out of the index keeps the scan proportional to what
+-- is actually waiting.
+CREATE INDEX IF NOT EXISTS aisoc_playbook_pauses_expiry_idx
+ ON aisoc_playbook_pauses (expires_at)
+ WHERE status = 'waiting';
+
+ALTER TABLE aisoc_playbook_pauses ENABLE ROW LEVEL SECURITY;
+ALTER TABLE aisoc_playbook_pauses FORCE ROW LEVEL SECURITY;
+
+DROP POLICY IF EXISTS tenant_isolation ON aisoc_playbook_pauses;
+CREATE POLICY tenant_isolation ON aisoc_playbook_pauses
+ -- The engine writes this from the agents service, which runs its
+ -- playbooks off a Kafka consumer binding no tenant. Without the unbound
+ -- arm every pause would fail to write and the run would abort exactly
+ -- as it does today, with the new table looking installed and doing
+ -- nothing.
+ USING (tenant_id = current_tenant_id() OR current_tenant_id() IS NULL)
+ WITH CHECK (tenant_id = current_tenant_id() OR current_tenant_id() IS NULL);
+
+DO $$
+BEGIN
+ IF EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'aisoc_app') THEN
+ GRANT SELECT, INSERT, UPDATE ON aisoc_playbook_pauses TO aisoc_app;
+ END IF;
+END
+$$;
diff --git a/services/fusion/app/main.py b/services/fusion/app/main.py
index 238430526..ac6b64a6b 100644
--- a/services/fusion/app/main.py
+++ b/services/fusion/app/main.py
@@ -1,6 +1,7 @@
import asyncio
import os
from contextlib import asynccontextmanager
+from typing import Any
import redis.asyncio as aioredis
from fastapi import FastAPI
@@ -23,6 +24,7 @@ from app.services.entity_risk import EntityRiskEngine
from app.services.fusion_engine import FusionEngine
from app.services.ioc_match import TenantIocMatcher
from app.services.lake_writer import LakeWriter
+from app.services.tenant_overlay import OverlayCache
from app.services.ueba_signal import UebaSignalCache
from app.services.windowed_detection import WindowedDetectionEngine
from app.workers.consumer import FusionWorker
@@ -137,6 +139,11 @@ async def lifespan(app: FastAPI):
lake=lake,
detector=detector,
windowed_detector=windowed_detector,
+ # Parity 5.4. Per-tenant detection tuning as a versioned overlay
+ # with hot reload. Without it the engine evaluated the shared
+ # corpus and nothing else, so a tenant who disabled a noisy rule
+ # in the console kept receiving its alerts.
+ overlays=OverlayCache(_overlay_pool()),
ueba_cache=ueba_cache,
)
set_worker(worker)
@@ -205,3 +212,59 @@ app.state.mark_ready = _mark_ready
app.state.mark_not_ready = _mark_not_ready
app.include_router(router)
+
+
+_OVERLAY_POOL: Any = None
+
+
+def _overlay_pool() -> Any:
+ """A lazy pool for the tenant-tuning overlay.
+
+ Built on first use rather than at import, because `main` runs before
+ an event loop exists and `asyncpg.create_pool` needs one. The cache
+ tolerates `None` and returns the empty overlay, so a deployment with
+ no database reachable detects exactly as it did before 5.4.
+ """
+ global _OVERLAY_POOL # noqa: PLW0603
+ if _OVERLAY_POOL is None:
+ _OVERLAY_POOL = _LazyPool(settings.database_url)
+ return _OVERLAY_POOL
+
+
+class _LazyPool:
+ """Opens an asyncpg pool on first acquire."""
+
+ def __init__(self, dsn: str) -> None:
+ self._dsn = dsn
+ self._pool: Any = None
+
+ def acquire(self): # noqa: ANN201
+ return _LazyAcquire(self)
+
+ async def _ensure(self) -> Any:
+ if self._pool is not None:
+ return self._pool
+ import asyncpg
+
+ dsn = self._dsn
+ for prefix in ("postgresql+asyncpg://", "postgres+asyncpg://"):
+ if dsn.startswith(prefix):
+ dsn = "postgresql://" + dsn[len(prefix) :]
+ break
+ self._pool = await asyncpg.create_pool(dsn, min_size=1, max_size=2)
+ return self._pool
+
+
+class _LazyAcquire:
+ def __init__(self, parent: _LazyPool) -> None:
+ self._parent = parent
+ self._ctx: Any = None
+
+ async def __aenter__(self) -> Any:
+ pool = await self._parent._ensure()
+ self._ctx = pool.acquire()
+ return await self._ctx.__aenter__()
+
+ async def __aexit__(self, *exc: Any) -> None:
+ if self._ctx is not None:
+ await self._ctx.__aexit__(*exc)
diff --git a/services/fusion/app/services/detection_engine.py b/services/fusion/app/services/detection_engine.py
index d552d3a7a..8e030f836 100644
--- a/services/fusion/app/services/detection_engine.py
+++ b/services/fusion/app/services/detection_engine.py
@@ -169,8 +169,19 @@ class DetectionEngine:
# Fall back to the OCSF top level (some connectors emit flat OCSF).
return ocsf if isinstance(ocsf, dict) else {}
- def evaluate(self, message: dict[str, Any]) -> list[DetectionHit]:
- """Return every rule that fires on this normalized-event message."""
+ def evaluate(self, message: dict[str, Any], overlay: Any | None = None) -> list[DetectionHit]:
+ """Return every rule that fires on this normalized-event message.
+
+ `overlay` is the tenant's tuning (parity 5.4). Without it this
+ engine evaluated the shared corpus and nothing else, so a tenant
+ who disabled a noisy rule in the console kept receiving its alerts
+ while the console showed it disabled.
+
+ Applied **after** the match rather than by filtering the candidate
+ list, so a suppressed hit can be recorded with the tuning that
+ suppressed it. "No alert" with no explanation is indistinguishable
+ from a rule that simply did not match.
+ """
ocsf = message.get("ocsf_event")
if not isinstance(ocsf, dict):
return []
@@ -180,10 +191,41 @@ class DetectionEngine:
# values were already present and nothing was computing the answer,
# so those rules could never fire.
fields = enrich(fields, self._derived_wanted)
+
+ # Parity 5.5. 15 rules read an `<x>_in_allowlist` boolean that
+ # nothing computed, so they could never fire. The answer is per
+ # tenant, which is why it comes from the overlay rather than from
+ # the shared derived-field pass: a global allowlist would make one
+ # tenant's exceptions apply to everybody.
+ #
+ # An unconfigured allowlist contributes **no key**, not False. A
+ # `not_in_allowlist` clause against a missing key would be true for
+ # every event, which is the negation-flips-on-absence failure the
+ # Sigma import already recorded.
+ if overlay is not None:
+ derived = overlay.derived_allowlist_fields(fields)
+ if derived:
+ fields = {**fields, **derived}
+
hits: list[DetectionHit] = []
for rule in self._candidates(""):
try:
if matches(rule["match_when"], fields):
+ if overlay is not None:
+ suppressed = overlay.suppresses(rule["id"], fields)
+ if suppressed is None:
+ suppressed = overlay.raises_severity_floor(rule["id"], str(rule.get("severity") or ""))
+ if suppressed:
+ # Info, not debug. A tenant asking why they
+ # stopped seeing a detection gets the tuning,
+ # its author and its reason from the log.
+ logger.info(
+ "detection_engine.suppressed_by_tenant_tuning",
+ rule=rule["id"],
+ tenant=getattr(overlay, "tenant_id", ""),
+ reason=suppressed,
+ )
+ continue
hits.append(
DetectionHit(
rule_id=rule["id"],
diff --git a/services/fusion/app/services/tenant_overlay.py b/services/fusion/app/services/tenant_overlay.py
new file mode 100644
index 000000000..f68ed5920
--- /dev/null
+++ b/services/fusion/app/services/tenant_overlay.py
@@ -0,0 +1,373 @@
+"""Per-tenant detection tuning, loaded into fusion as a versioned overlay.
+
+Parity plan 5.4: "Fusion loads static JSON and never reads tenant tuning.
+Load per-tenant suppressions, thresholds, disables, custom rules and MSSP
+rule packs into fusion as versioned overlays with hot reload. A tuning
+change applies to that tenant only and is recorded with its author and
+reason."
+
+What was wrong
+--------------
+`DetectionEngine` loads `app/data/detection_ruleset.json` at construction
+and evaluates every rule against every event. The console lets a tenant
+disable a rule, raise its threshold or suppress it for a field value, and
+all of that is written to `detection_rules` in Postgres where **the
+streaming engine never looks**. A tenant who turned a noisy rule off kept
+getting alerts from it, and the console showed the rule as disabled.
+
+Why an overlay rather than a per-tenant ruleset
+------------------------------------------------
+Building a full ruleset per tenant would mean holding N copies of 833
+rules and rebuilding one whenever anybody edits anything. The overlay is
+the *difference*: a small set of disables, thresholds and suppressions
+applied over the shared corpus at match time. One corpus, one small
+per-tenant delta.
+
+Versioning and hot reload
+-------------------------
+Each tenant's overlay carries a version derived from the rows it was built
+from. The engine refetches on a short interval and swaps atomically, so a
+tuning change applies within one reload rather than at the next restart.
+A reload that fails keeps the previous overlay rather than falling back to
+"no tuning": losing a tenant's suppressions because the database blinked
+would turn their queue back on.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+import time
+from dataclasses import dataclass, field
+from typing import Any
+
+import structlog
+
+logger = structlog.get_logger()
+
+#: How often a tenant's overlay is refetched. Short, because the plan asks
+#: for hot reload and a tenant who just disabled a noisy rule should stop
+#: seeing it quickly.
+RELOAD_SECONDS = float(os.getenv("AISOC_TENANT_OVERLAY_RELOAD_SECONDS", "30"))
+
+
+@dataclass(frozen=True)
+class RuleOverride:
+ """One tenant's tuning of one rule."""
+
+ rule_id: str
+ enabled: bool = True
+ #: Minimum severity this rule may still fire at, if raised.
+ min_severity: str | None = None
+ #: Field/value pairs that suppress a match entirely.
+ suppress_when: dict[str, list[str]] = field(default_factory=dict)
+ #: Who changed it and why, carried so an alert that did *not* fire can
+ #: be explained without opening the database.
+ author: str | None = None
+ reason: str | None = None
+
+
+_SEVERITY_ORDER = ("info", "low", "medium", "high", "critical")
+
+
+@dataclass(frozen=True)
+class TenantOverlay:
+ """The whole delta for one tenant, plus the version it was built at."""
+
+ tenant_id: str
+ overrides: dict[str, RuleOverride] = field(default_factory=dict)
+ #: Field name to the values this tenant has declared fine. Feeds the
+ #: `<x>_in_allowlist` booleans 15 rules read and nothing computed.
+ allowlists: dict[str, list[str]] = field(default_factory=dict)
+ version: str = ""
+ loaded_at: float = 0.0
+
+ def derived_allowlist_fields(self, event: dict[str, Any]) -> dict[str, bool]:
+ if not self.allowlists:
+ return {}
+ return allowlist_fields(self.allowlists, event)
+
+ def suppresses(self, rule_id: str, event: dict[str, Any]) -> str | None:
+ """Why this rule must not fire for this tenant, or None.
+
+ A reason string rather than a bool, so a suppressed match can be
+ recorded with the tuning that suppressed it. "No alert" with no
+ explanation is indistinguishable from a rule that simply did not
+ match, and an analyst asking why they stopped seeing something
+ deserves an answer.
+ """
+ override = self.overrides.get(rule_id)
+ if override is None:
+ return None
+
+ if not override.enabled:
+ return _explain("disabled for this tenant", override)
+
+ for field_name, values in override.suppress_when.items():
+ actual = event.get(field_name)
+ if actual is None:
+ continue
+ if str(actual) in {str(v) for v in values}:
+ return _explain(f"suppressed where {field_name}={actual!r}", override)
+ return None
+
+ def raises_severity_floor(self, rule_id: str, severity: str) -> str | None:
+ """Whether this tenant raised the bar above this match's severity."""
+ override = self.overrides.get(rule_id)
+ if override is None or not override.min_severity:
+ return None
+ try:
+ if _SEVERITY_ORDER.index(severity.lower()) < _SEVERITY_ORDER.index(override.min_severity.lower()):
+ return _explain(
+ f"below this tenant's floor of {override.min_severity} for this rule",
+ override,
+ )
+ except ValueError:
+ return None
+ return None
+
+
+def _explain(what: str, override: RuleOverride) -> str:
+ """The tuning, its author and its reason, in one line."""
+ parts = [what]
+ if override.author:
+ parts.append(f"by {override.author}")
+ if override.reason:
+ parts.append(f"({override.reason})")
+ return " ".join(parts)
+
+
+def _version_of(rows: list[dict[str, Any]]) -> str:
+ """A digest of the rows the overlay was built from.
+
+ Content-derived rather than a timestamp, so an overlay rebuilt from
+ unchanged rows keeps its version and a consumer can tell a real change
+ from a periodic refetch.
+ """
+ canonical = json.dumps(
+ sorted(
+ (
+ str(r.get("rule_id") or r.get("id")),
+ r.get("status"),
+ json.dumps(r.get("suppression_config") or {}, sort_keys=True, default=str),
+ json.dumps(r.get("threshold_config") or {}, sort_keys=True, default=str),
+ )
+ for r in rows
+ ),
+ default=str,
+ )
+ return hashlib.sha256(canonical.encode()).hexdigest()[:16]
+
+
+def build_overlay(tenant_id: str, rows: list[dict[str, Any]], *, now: float | None = None) -> TenantOverlay:
+ """Turn `detection_rules` rows into an overlay.
+
+ `now` is injectable because the reload interval is a behaviour worth
+ testing, and a cache whose clock cannot be controlled can only be
+ tested by sleeping.
+ """
+ overrides: dict[str, RuleOverride] = {}
+ for row in rows:
+ rule_id = str(row.get("rule_id") or row.get("id") or "")
+ if not rule_id:
+ continue
+ suppression = row.get("suppression_config") or {}
+ if isinstance(suppression, str):
+ try:
+ suppression = json.loads(suppression)
+ except ValueError:
+ suppression = {}
+ threshold = row.get("threshold_config") or {}
+ if isinstance(threshold, str):
+ try:
+ threshold = json.loads(threshold)
+ except ValueError:
+ threshold = {}
+
+ suppress_when: dict[str, list[str]] = {}
+ raw_when = suppression.get("suppress_when") or suppression.get("exclude") or {}
+ if isinstance(raw_when, dict):
+ for key, value in raw_when.items():
+ suppress_when[str(key)] = [str(v) for v in value] if isinstance(value, list) else [str(value)]
+
+ overrides[rule_id] = RuleOverride(
+ rule_id=rule_id,
+ # `status` is the console's enabled flag: anything other than
+ # `active` means the tenant turned it off.
+ enabled=str(row.get("status") or "active") == "active",
+ min_severity=threshold.get("min_severity"),
+ suppress_when=suppress_when,
+ author=row.get("author") or suppression.get("author"),
+ reason=suppression.get("reason") or threshold.get("reason"),
+ )
+
+ # A tenant's allowlists live beside their suppressions: the console
+ # writes both, and an allowlist is the same decision expressed for a
+ # field rather than for a rule.
+ allowlists: dict[str, list[str]] = {}
+ for row in rows:
+ suppression = row.get("suppression_config") or {}
+ if isinstance(suppression, str):
+ try:
+ suppression = json.loads(suppression)
+ except ValueError:
+ continue
+ raw = suppression.get("allowlists") if isinstance(suppression, dict) else None
+ if isinstance(raw, dict):
+ for key, value in raw.items():
+ values = [str(v) for v in value] if isinstance(value, list) else [str(value)]
+ allowlists.setdefault(str(key), []).extend(values)
+
+ return TenantOverlay(
+ tenant_id=tenant_id,
+ overrides=overrides,
+ allowlists=allowlists,
+ version=_version_of(rows),
+ loaded_at=now if now is not None else time.monotonic(),
+ )
+
+
+# ── Per-tenant allowlists (parity 5.5) ──────────────────────────────────
+#
+# 15 of the 133 rules that cannot fire need an `<x>_in_allowlist` boolean
+# that nothing computed. They were the cheapest family to close, because
+# an allowlist is the same thing a tenant already configures as a
+# suppression: a list of values they have decided are fine.
+#
+# Each entry maps the boolean a rule reads to the event field its value
+# comes from. Derived here rather than in `derived_fields.py` because the
+# answer is per tenant, and a shared derived field would make one tenant's
+# allowlist apply to everybody.
+ALLOWLIST_FIELDS: dict[str, str] = {
+ "image_registry_in_allowlist": "image_registry",
+ "dst_registry_in_allowlist": "dst_registry",
+ "external_account_in_allowlist": "external_account",
+ "impersonator_in_allowlist": "impersonator",
+ "share_target_domain_in_allowlist": "share_target_domain",
+ "backup_dst_in_allowlist": "backup_dst",
+ "src_country_in_allowlist": "src_country",
+ "src_country_not_in_allowlist": "src_country",
+ "source_image_not_in_allowlist": "source_image",
+ "source_image_basename_not_in_allowlist": "source_image_basename",
+ "image_basename_in_allowlist": "image_basename",
+ "dst_in_allowlist": "dst",
+ "src_in_allowlist": "src",
+}
+
+
+def allowlist_fields(allowlists: dict[str, list[str]], event: dict[str, Any]) -> dict[str, bool]:
+ """The `<x>_in_allowlist` booleans this event's values produce.
+
+ An absent allowlist yields **no key at all** rather than `False`. The
+ distinction matters: `not_in_allowlist` with an unconfigured list
+ would be `True` for every event and the rule would fire on all of
+ them, which is exactly the "a negation flips on a missing field"
+ failure recorded for the Sigma import.
+ """
+ out: dict[str, bool] = {}
+ for boolean, source in ALLOWLIST_FIELDS.items():
+ configured = allowlists.get(source)
+ if configured is None:
+ continue
+ value = event.get(source)
+ if value is None:
+ continue
+ member = str(value) in {str(v) for v in configured}
+ out[boolean] = member if boolean.endswith("_in_allowlist") and "_not_in_" not in boolean else not member
+ return out
+
+
+#: An overlay with nothing in it. Returned for a tenant that has tuned
+#: nothing, which is the common case and must cost nothing.
+EMPTY = TenantOverlay(tenant_id="", overrides={}, version="empty")
+
+
+class OverlayCache:
+ """Per-tenant overlays with hot reload.
+
+ A failed reload keeps the previous overlay rather than falling back to
+ `EMPTY`. Losing a tenant's suppressions because the database blinked
+ would turn their queue back on, which is the opposite of what the
+ tuning was for.
+ """
+
+ def __init__(self, pool: Any, *, reload_seconds: float | None = None) -> None:
+ self._pool = pool
+ self._reload = reload_seconds if reload_seconds is not None else RELOAD_SECONDS
+ self._cache: dict[str, TenantOverlay] = {}
+
+ async def get(self, tenant_id: str, *, now: float | None = None) -> TenantOverlay:
+ if not tenant_id or self._pool is None:
+ return EMPTY
+ current = self._cache.get(tenant_id)
+ clock = now if now is not None else time.monotonic()
+ if current is not None and (clock - current.loaded_at) < self._reload:
+ return current
+
+ rows = await self._fetch(tenant_id)
+ if rows is None:
+ if current is not None:
+ logger.warning(
+ "tenant_overlay.reload_failed_keeping_previous",
+ tenant_id=tenant_id,
+ version=current.version,
+ )
+ return current
+ return EMPTY
+
+ overlay = build_overlay(tenant_id, rows, now=clock)
+ if current is not None and current.version != overlay.version:
+ logger.info(
+ "tenant_overlay.changed",
+ tenant_id=tenant_id,
+ was=current.version,
+ now=overlay.version,
+ overrides=len(overlay.overrides),
+ )
+ self._cache[tenant_id] = overlay
+ return overlay
+
+ async def _fetch(self, tenant_id: str) -> list[dict[str, Any]] | None:
+ """Rows, or None when they could not be read.
+
+ None and an empty list mean different things: no tuning, versus no
+ answer. Collapsing them is what would drop a tenant's suppressions
+ on a transient failure.
+ """
+ try:
+ async with self._pool.acquire() as conn:
+ # `detection_rules` has no `rule_id` or `updated_by` column.
+ # It did not have one when this query was first written
+ # either, and the query failed into the `except` below,
+ # logged a warning and returned None — so the overlay
+ # never loaded and the whole feature silently did nothing
+ # while every unit test passed against a fake.
+ #
+ # The engine's `det-*` id lives in `provenance->>'source_id'`
+ # (migration 036), which is the stable external identifier
+ # by design. `author` is the column that records who
+ # changed the rule.
+ rows = await conn.fetch(
+ """
+ SELECT COALESCE(provenance->>'source_id', name) AS rule_id,
+ status,
+ suppression_config,
+ threshold_config,
+ author
+ FROM detection_rules
+ WHERE tenant_id = $1::uuid
+ AND (status <> 'active'
+ OR suppression_config <> '{}'::jsonb
+ OR threshold_config <> '{}'::jsonb)
+ """,
+ tenant_id,
+ )
+ except Exception as exc: # noqa: BLE001
+ logger.warning("tenant_overlay.fetch_failed", tenant_id=tenant_id, error=str(exc))
+ return None
+ return [dict(r) for r in rows]
+
+ def invalidate(self, tenant_id: str) -> None:
+ """Force the next read to refetch. For an explicit console save."""
+ self._cache.pop(tenant_id, None)
diff --git a/services/fusion/app/workers/consumer.py b/services/fusion/app/workers/consumer.py
index 16c6dc4d9..6ceae8e51 100644
--- a/services/fusion/app/workers/consumer.py
+++ b/services/fusion/app/workers/consumer.py
@@ -21,6 +21,7 @@ from app.services.event_schema import validate_event
from app.services.fusion_engine import FusionEngine
from app.services.lake_writer import LakeWriter
from app.services.promoter import promote_normalized_event
+from app.services.tenant_overlay import OverlayCache
from app.services.ueba_signal import UebaSignalCache
from app.services.windowed_detection import WindowedDetectionEngine
@@ -55,6 +56,7 @@ class FusionWorker:
detector: DetectionEngine | None = None,
windowed_detector: WindowedDetectionEngine | None = None,
ueba_cache: UebaSignalCache | None = None,
+ overlays: OverlayCache | None = None,
) -> None:
self._engine = engine
self._sink = sink
@@ -62,6 +64,9 @@ class FusionWorker:
self._detector = detector
self._windowed = windowed_detector
self._ueba_cache = ueba_cache
+ #: Per-tenant detection tuning (parity 5.4). Optional, so a
+ #: deployment that passes nothing behaves exactly as before.
+ self._overlays = overlays
# A poison message must never vanish silently; default to a structured
# logging DLQ so persistence-free deployments still get the signal.
self._dlq: DeadLetterQueue = dlq or LoggingDLQ()
@@ -287,7 +292,14 @@ class FusionWorker:
# event. Each firing rule becomes a RawAlert routed through fusion,
# so telemetry that isn't a vendor-asserted finding still alerts.
if self._detector is not None:
- for hit in self._detector.evaluate(payload):
+ # Parity 5.4. Without the overlay this evaluated the shared
+ # corpus and nothing else, so a tenant who disabled a noisy
+ # rule in the console kept receiving its alerts while the
+ # console showed it disabled.
+ overlay = None
+ if self._overlays is not None and validation.tenant_id:
+ overlay = await self._overlays.get(str(validation.tenant_id))
+ for hit in self._detector.evaluate(payload, overlay):
det_alert = self._detector.build_alert(payload, hit)
if det_alert is not None:
_METRICS["detected"] += 1
diff --git a/services/fusion/tests/test_tenant_tuning_overlay.py b/services/fusion/tests/test_tenant_tuning_overlay.py
new file mode 100644
index 000000000..d1a76e3a6
--- /dev/null
+++ b/services/fusion/tests/test_tenant_tuning_overlay.py
@@ -0,0 +1,437 @@
+"""A tenant's tuning reaches the streaming engine, and only that tenant's.
+
+Parity plan 5.4: "Fusion loads static JSON and never reads tenant tuning.
+Load per-tenant suppressions, thresholds, disables, custom rules and MSSP
+rule packs into fusion as versioned overlays with hot reload. A tuning
+change applies to that tenant only and is recorded with its author and
+reason."
+
+What was wrong
+--------------
+`DetectionEngine` evaluated the shared corpus and nothing else. The
+console writes a tenant's disables, thresholds and suppressions to
+`detection_rules` in Postgres, where the streaming engine never looked. A
+tenant who turned a noisy rule off kept receiving its alerts, and the
+console showed the rule as disabled: the worst shape, because it tells the
+operator the problem is solved.
+
+The two properties worth testing hardest
+-----------------------------------------
+**One tenant's tuning must not touch another's.** This is the plan's own
+"done when", and the obvious implementation (a module-level dict keyed on
+rule id) gets it wrong silently.
+
+**A failed reload must keep the previous overlay**, not fall back to "no
+tuning". Losing a tenant's suppressions because the database blinked would
+turn their queue back on, which is the opposite of what the tuning was for
+and would look like a flood rather than a fault.
+"""
+
+from __future__ import annotations
+
+from contextlib import asynccontextmanager
+
+import pytest
+from app.services.tenant_overlay import (
+ EMPTY,
+ OverlayCache,
+ TenantOverlay,
+ build_overlay,
+)
+
+
+def _row(rule_id: str, **kw): # noqa: ANN001, ANN201
+ base = {
+ "rule_id": rule_id,
+ "status": "active",
+ "suppression_config": None,
+ "threshold_config": None,
+ # `detection_rules` has an `author` column and no `updated_by`.
+ # The first version of this fake carried `updated_by`, which is
+ # how a query selecting a column that does not exist passed every
+ # test: the fake answered whatever it was asked.
+ "author": None,
+ }
+ base.update(kw)
+ return base
+
+
+class TestBuildingAnOverlay:
+ def test_a_disabled_rule_is_an_override(self) -> None:
+ overlay = build_overlay("t-1", [_row("det-1", status="disabled")])
+ assert overlay.overrides["det-1"].enabled is False
+
+ def test_an_untuned_rule_produces_no_override(self) -> None:
+ """The overlay is a delta. A tenant who has tuned nothing must cost
+ nothing, not carry 833 no-op entries."""
+ overlay = build_overlay("t-1", [])
+ assert overlay.overrides == {}
+
+ def test_the_version_is_derived_from_the_rows(self) -> None:
+ """Content-derived rather than a timestamp, so a consumer can tell
+ a real change from a periodic refetch."""
+ rows = [_row("det-1", status="disabled")]
+ assert build_overlay("t-1", rows).version == build_overlay("t-1", rows).version
+
+ def test_a_changed_row_changes_the_version(self) -> None:
+ a = build_overlay("t-1", [_row("det-1", status="active")])
+ b = build_overlay("t-1", [_row("det-1", status="disabled")])
+ assert a.version != b.version
+
+ def test_a_json_string_config_is_parsed(self) -> None:
+ """The driver may hand back jsonb as a string."""
+ overlay = build_overlay("t-1", [_row("det-1", suppression_config='{"suppress_when": {"host": ["JENKINS-01"]}}')])
+ assert overlay.overrides["det-1"].suppress_when == {"host": ["JENKINS-01"]}
+
+ def test_a_malformed_config_does_not_raise(self) -> None:
+ overlay = build_overlay("t-1", [_row("det-1", suppression_config="not json")])
+ assert overlay.overrides["det-1"].suppress_when == {}
+
+
+class TestSuppression:
+ def test_a_disabled_rule_is_suppressed_with_a_reason(self) -> None:
+ overlay = build_overlay(
+ "t-1",
+ [
+ _row(
+ "det-1",
+ status="disabled",
+ author="[email protected]",
+ suppression_config={"reason": "known noisy on build agents"},
+ )
+ ],
+ )
+ why = overlay.suppresses("det-1", {})
+ assert why is not None
+ assert "disabled" in why
+ assert "[email protected]" in why, "the author must travel with the suppression"
+ assert "build agents" in why, "the reason must travel with the suppression"
+
+ def test_a_field_match_suppresses(self) -> None:
+ overlay = build_overlay(
+ "t-1",
+ [_row("det-1", suppression_config={"suppress_when": {"host": ["JENKINS-01"]}})],
+ )
+ assert overlay.suppresses("det-1", {"host": "JENKINS-01"}) is not None
+ assert overlay.suppresses("det-1", {"host": "WIN-FIN-02"}) is None
+
+ def test_an_absent_field_does_not_suppress(self) -> None:
+ """Suppressing on a field the event does not carry would silence
+ every event rather than the ones the tenant meant."""
+ overlay = build_overlay("t-1", [_row("det-1", suppression_config={"suppress_when": {"host": ["X"]}})])
+ assert overlay.suppresses("det-1", {"user": "j.doe"}) is None
+
+ def test_an_untuned_rule_is_never_suppressed(self) -> None:
+ assert build_overlay("t-1", []).suppresses("det-1", {"host": "anything"}) is None
+
+
+class TestTheSeverityFloor:
+ def test_a_match_below_the_floor_is_held(self) -> None:
+ overlay = build_overlay("t-1", [_row("det-1", threshold_config={"min_severity": "high"})])
+ assert overlay.raises_severity_floor("det-1", "medium") is not None
+ assert overlay.raises_severity_floor("det-1", "high") is None
+ assert overlay.raises_severity_floor("det-1", "critical") is None
+
+ def test_an_unknown_severity_does_not_silently_hold(self) -> None:
+ """A severity the ladder does not know must not be treated as the
+ lowest, which would suppress it."""
+ overlay = build_overlay("t-1", [_row("det-1", threshold_config={"min_severity": "high"})])
+ assert overlay.raises_severity_floor("det-1", "unknown-tier") is None
+
+
+class _Conn:
+ def __init__(self, rows, fail=False) -> None: # noqa: ANN001
+ self.rows = rows
+ self.fail = fail
+ self.fetches = 0
+
+ async def fetch(self, sql, *args): # noqa: ANN001, ARG002
+ self.fetches += 1
+ if self.fail:
+ raise RuntimeError("database down")
+ return [r for r in self.rows if str(r.get("_tenant", args[0])) == str(args[0])]
+
+
+class _Pool:
+ def __init__(self, conn: _Conn) -> None:
+ self._conn = conn
+
+ @asynccontextmanager
+ async def acquire(self):
+ yield self._conn
+
+
+class TestTenantIsolation:
+ async def test_one_tenants_tuning_does_not_reach_another(self) -> None:
+ """The plan's own done-when, and the thing a dict keyed on rule id
+ alone gets wrong silently."""
+ rows = [
+ {**_row("det-1", status="disabled"), "_tenant": "t-1"},
+ {**_row("det-2", status="disabled"), "_tenant": "t-2"},
+ ]
+ cache = OverlayCache(_Pool(_Conn(rows)), reload_seconds=0)
+
+ a = await cache.get("t-1")
+ b = await cache.get("t-2")
+
+ assert "det-1" in a.overrides and "det-2" not in a.overrides
+ assert "det-2" in b.overrides and "det-1" not in b.overrides
+ assert a.suppresses("det-2", {}) is None, "tenant A inherited tenant B's suppression"
+
+
+class TestHotReload:
+ async def test_it_refetches_after_the_interval(self) -> None:
+ conn = _Conn([_row("det-1", status="disabled")])
+ cache = OverlayCache(_Pool(conn), reload_seconds=10)
+ await cache.get("t-1", now=0.0)
+ await cache.get("t-1", now=1.0)
+ assert conn.fetches == 1, "it refetched inside the interval"
+ await cache.get("t-1", now=100.0)
+ assert conn.fetches == 2, "it did not refetch after the interval"
+
+ async def test_an_explicit_invalidate_forces_a_refetch(self) -> None:
+ conn = _Conn([_row("det-1", status="disabled")])
+ cache = OverlayCache(_Pool(conn), reload_seconds=10_000)
+ await cache.get("t-1", now=0.0)
+ cache.invalidate("t-1")
+ await cache.get("t-1", now=1.0)
+ assert conn.fetches == 2
+
+
+class TestAFailedReload:
+ async def test_it_keeps_the_previous_overlay(self) -> None:
+ """Falling back to no-tuning would turn the tenant's queue back on,
+ which reads as a flood rather than as a fault."""
+ conn = _Conn([_row("det-1", status="disabled")])
+ cache = OverlayCache(_Pool(conn), reload_seconds=0)
+ first = await cache.get("t-1")
+ assert "det-1" in first.overrides
+
+ conn.fail = True
+ second = await cache.get("t-1")
+ assert second.suppresses("det-1", {}) is not None, "the suppression was lost on a transient database failure"
+
+ async def test_a_first_load_that_fails_returns_empty_rather_than_raising(self) -> None:
+ """Nothing is cached yet, so there is nothing to keep. Detection
+ must continue rather than stop."""
+ cache = OverlayCache(_Pool(_Conn([], fail=True)), reload_seconds=0)
+ assert await cache.get("t-1") is EMPTY
+
+ async def test_no_pool_is_empty_not_an_error(self) -> None:
+ assert await OverlayCache(None).get("t-1") is EMPTY
+
+
+class TestTheEngineAppliesIt:
+ def test_evaluate_accepts_an_overlay(self) -> None:
+ import inspect
+
+ from app.services.detection_engine import DetectionEngine
+
+ signature = inspect.signature(DetectionEngine.evaluate)
+ assert "overlay" in signature.parameters, "the streaming engine cannot see tenant tuning, which is the defect 5.4 exists to fix"
+
+ def test_a_suppressed_hit_is_dropped(self) -> None:
+ from app.services.detection_engine import DetectionEngine
+
+ rules = [
+ {
+ "id": "det-1",
+ "name": "Noisy",
+ "severity": "high",
+ "category": "endpoint",
+ "match_when": {"host": "JENKINS-01"},
+ }
+ ]
+ engine = DetectionEngine(rules=rules)
+ message = {"ocsf_event": {"raw_data": '{"host": "JENKINS-01"}'}}
+
+ assert engine.evaluate(message), "the rule did not fire without an overlay"
+
+ overlay = build_overlay("t-1", [_row("det-1", suppression_config={"suppress_when": {"host": ["JENKINS-01"]}})])
+ assert engine.evaluate(message, overlay) == [], "the tenant's suppression was ignored"
+
+ def test_an_empty_overlay_changes_nothing(self) -> None:
+ """A tenant who has tuned nothing must see exactly what they saw
+ before this existed."""
+ from app.services.detection_engine import DetectionEngine
+
+ rules = [
+ {
+ "id": "det-1",
+ "name": "Noisy",
+ "severity": "high",
+ "category": "endpoint",
+ "match_when": {"host": "JENKINS-01"},
+ }
+ ]
+ engine = DetectionEngine(rules=rules)
+ message = {"ocsf_event": {"raw_data": '{"host": "JENKINS-01"}'}}
+ assert engine.evaluate(message, EMPTY) == engine.evaluate(message)
+
+ def test_the_overlay_is_optional(self) -> None:
+ """Additive: every existing caller passes nothing and is
+ unaffected."""
+ import inspect
+
+ from app.services.detection_engine import DetectionEngine
+
+ assert inspect.signature(DetectionEngine.evaluate).parameters["overlay"].default is None
+
+
+class TestTheOverlayType:
+ def test_empty_is_shared_and_carries_no_overrides(self) -> None:
+ assert isinstance(EMPTY, TenantOverlay)
+ assert EMPTY.overrides == {}
+
+
+class TestPerTenantAllowlists:
+ """Parity 5.5. The cheapest of the five unreachable families.
+
+ 15 rules read an `<x>_in_allowlist` boolean that nothing computed, so
+ they could never fire on any connector. An allowlist is the same
+ decision a tenant already expresses as a suppression, so the overlay
+ was the natural place for it: 133 unreachable rules became 119.
+
+ Per tenant rather than in the shared derived-field pass, because a
+ global allowlist would make one tenant's exceptions apply to
+ everybody.
+ """
+
+ def test_a_member_is_in_the_allowlist(self) -> None:
+ from app.services.tenant_overlay import allowlist_fields
+
+ out = allowlist_fields({"src_country": ["GB", "US"]}, {"src_country": "GB"})
+ assert out["src_country_in_allowlist"] is True
+ assert out["src_country_not_in_allowlist"] is False
+
+ def test_a_stranger_is_not(self) -> None:
+ from app.services.tenant_overlay import allowlist_fields
+
+ out = allowlist_fields({"src_country": ["GB"]}, {"src_country": "RU"})
+ assert out["src_country_in_allowlist"] is False
+ assert out["src_country_not_in_allowlist"] is True
+
+ def test_an_unconfigured_allowlist_contributes_no_key(self) -> None:
+ """The property that matters most.
+
+ A `not_in_allowlist` clause against a missing key is true for
+ every event, so the rule would fire on all of them. That is the
+ negation-flips-on-absence failure already recorded for the Sigma
+ import, and `False` would be just as wrong in the other
+ direction.
+ """
+ from app.services.tenant_overlay import allowlist_fields
+
+ assert allowlist_fields({}, {"src_country": "RU"}) == {}
+
+ def test_an_absent_event_field_contributes_no_key(self) -> None:
+ from app.services.tenant_overlay import allowlist_fields
+
+ assert allowlist_fields({"src_country": ["GB"]}, {"user": "j.doe"}) == {}
+
+ def test_every_allowlist_boolean_the_corpus_reads_is_mapped(self) -> None:
+ """Otherwise a rule stays unreachable while the ratchet says it is
+ not."""
+ import json
+ import pathlib as _p
+ import re
+
+ from app.services.tenant_overlay import ALLOWLIST_FIELDS
+
+ corpus = _p.Path(__file__).resolve().parents[1] / "app/data/detection_ruleset.json"
+ rules = json.loads(corpus.read_text())
+ rules = rules if isinstance(rules, list) else rules.get("rules", [])
+ wanted = set()
+ for rule in rules:
+ wanted.update(re.findall(r'"([a-z_]+_(?:not_)?in_allowlist)"', json.dumps(rule.get("match_when") or {})))
+ missing = wanted - set(ALLOWLIST_FIELDS)
+ assert not missing, f"the corpus reads allowlist booleans nothing derives: {sorted(missing)}"
+
+ def test_a_rule_that_could_never_fire_now_can(self) -> None:
+ """End to end through the real engine."""
+ from app.services.detection_engine import DetectionEngine
+ from app.services.tenant_overlay import build_overlay
+
+ rules = [
+ {
+ "id": "r1",
+ "name": "Login from an unexpected country",
+ "severity": "high",
+ "category": "identity",
+ "match_when": {"src_country_not_in_allowlist": True},
+ }
+ ]
+ engine = DetectionEngine(rules=rules)
+ event = {"ocsf_event": {"raw_data": '{"src_country": "RU"}'}}
+
+ assert engine.evaluate(event) == [], "the rule fired with no allowlist configured"
+
+ overlay = build_overlay(
+ "t-1",
+ [_row("other", suppression_config={"allowlists": {"src_country": ["GB"]}})],
+ )
+ assert len(engine.evaluate(event, overlay)) == 1, "the rule still cannot fire"
+
+ allowed = {"ocsf_event": {"raw_data": '{"src_country": "GB"}'}}
+ assert engine.evaluate(allowed, overlay) == [], "an allowlisted country still alerted"
+
+
+class TestTheQueryMatchesTheRealSchema:
+ """The gap that let a non-existent column ship.
+
+ `_fetch` selected `rule_id` and `updated_by` from `detection_rules`,
+ which has neither. Every test above passed, because a fake answers
+ whatever it is asked — so the overlay would have loaded nothing on
+ every deployment while the suite stayed green. Live QA found it.
+
+ This reads the migration rather than a second hand-written list,
+ because a list maintained beside the query drifts in the same
+ direction the query already drifted.
+ """
+
+ def test_every_selected_column_exists_in_the_migration(self) -> None:
+ import pathlib as _p
+ import re
+
+ root = _p.Path(__file__).resolve().parents[3]
+ init = (root / "services/api/migrations/001_init.sql").read_text()
+
+ body = re.search(
+ r"CREATE TABLE (?:IF NOT EXISTS )?detection_rules\s*\((.*?)\n\);",
+ init,
+ re.S | re.I,
+ )
+ assert body, "detection_rules is not created in 001_init.sql; this test is looking in the wrong place"
+
+ declared = {m.group(1).lower() for line in body.group(1).splitlines() if (m := re.match(r"\s*([a-z_][a-z0-9_]*)\s+[A-Za-z]", line))}
+ # Later migrations add columns; pick those up too rather than
+ # failing on a column that exists but arrived after 001.
+ for path in sorted((root / "services/api/migrations").glob("*.sql")):
+ for m in re.finditer(
+ r"ALTER TABLE\s+(?:IF EXISTS\s+)?detection_rules\s+ADD COLUMN\s+(?:IF NOT EXISTS\s+)?([a-z_][a-z0-9_]*)",
+ path.read_text(),
+ re.I,
+ ):
+ declared.add(m.group(1).lower())
+
+ source = (_p.Path(__file__).resolve().parents[1] / "app/services/tenant_overlay.py").read_text()
+ projection = re.search(r"SELECT (.*?)\s+FROM detection_rules", source, re.S)
+ assert projection, "the overlay no longer selects from detection_rules"
+
+ referenced = {token.lower() for token in re.findall(r"\b([a-z_][a-z0-9_]*)\b", projection.group(1))} - {
+ "coalesce",
+ "as",
+ "rule_id",
+ "source_id",
+ "null",
+ }
+
+ missing = referenced - declared
+ assert not missing, (
+ f"the overlay selects column(s) no migration creates: {sorted(missing)}. "
+ "Every execution of this query raises UndefinedColumnError, the handler turns "
+ "it into a warning, and the overlay silently loads nothing."
+ )
diff --git a/tests/test_playbook_schema_parity_gate.py b/tests/test_playbook_schema_parity_gate.py
index de35f4bbc..53bc29456 100644
--- a/tests/test_playbook_schema_parity_gate.py
+++ b/tests/test_playbook_schema_parity_gate.py
@@ -214,12 +214,28 @@ def test_catches_a_web_surface_that_invents_a_step_type(registries, gate):
def test_catches_a_surface_promising_an_execution_the_schema_denies(registries, gate):
+ """The surface claims a step runs where the contract says it does not.
+
+ This used to find its own case by looking for a member the tree
+ already marked `unimplemented`, and raised `StopIteration` the moment
+ there were none left — which is what happened when parity 5.2 turned
+ `approval` into a durable pause and emptied that category entirely.
+
+ A meta-test that needs a defect to already exist in order to test for
+ defects stops working exactly when the tree gets healthy, so it
+ constructs **both** sides now: the schema denies the member and the
+ surface promises it.
+ """
annotated = next(v for v in registries.ts_vocabularies if v.execution)
- member = next(k for k, v in annotated.execution.items() if v == "unimplemented")
- lying = dataclasses.replace(annotated, execution={**annotated.execution, member: "executed"})
+ member = sorted(annotated.execution)[0]
+
+ promising = dataclasses.replace(annotated, execution={**annotated.execution, member: "executed"})
broken = _perturb(
registries,
- ts_vocabularies=tuple(lying if v.ref == annotated.ref else v for v in registries.ts_vocabularies),
+ # The schema side says no handler exists...
+ execution={**registries.execution, member: "unimplemented"},
+ # ...while the editor tells an author the step will run.
+ ts_vocabularies=tuple(promising if v.ref == annotated.ref else v for v in registries.ts_vocabularies),
)
assert any("the surface and the contract disagree" in e for e in gate.compare(broken))