Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
feat(phase-B3): real rollback + post-action verification + durable approvals (#454)
Beenu Arorad1ec16a8fce2
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 3b61c3779..da09c35c6 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Added+- **Phase B3 — rollback is real, actions are post-verified, and approval SLA timers survive restarts.** Three "honest response" gaps closed. (1) **Real rollback:** every executor's `rollback()` previously returned a bare `True` — logging "rolling back" without calling the vendor. New `services/actions/app/services/rollback.py` performs the **real** reverse via the same clients (isolate→`lift_containment`/`unisolate_machine`, block_ip→`unblock_ip`/`unblock_ip_zone`, disable_user→`enable_user`/`unsuspend_user`, suspend_session→`unsuspend_user`) and returns an honest `RollbackResult` (`reversed_` / `simulated` / `supported`) — never a fake success; a failed reverse is reported, not hidden. `autonomy_safety.REVERSIBLE_ACTIONS` now imports from this module (single source of truth) and `test_rollback.py` gates the two sets so an action can't be declared reversible without a real reverse. (2) **Post-action verification:** new `services/actions/app/services/verification.py` re-queries the vendor to confirm the effect is actually present (`VERIFIED` / `FAILED` / honest `UNVERIFIED` when no probe or no creds) — a probe error is never a false `VERIFIED`. (3) **Durable approvals:** `services/slack-bot/app/services/timer_store.py` adds a Postgres-backed `TimerStore`; the `ApprovalTimeoutScheduler` persists pending SLA timers and `recover()` re-arms them on startup (firing overdue ones promptly), so a bot restart can no longer strand a forgotten approval forever. Fail-soft to in-memory when no DB. 25 new tests across actions + slack-bot. Claim-to-gate matrix +1 GATED (22 GATED / 10 PARTIAL / 1 NO GATE).- **Phase B2 — connector credentials now reach the SOAR executors, and the autonomy policy governs every real execution.** Two wiring gaps closed. (1) Executors read vendor-prefixed parameter keys (`cs_client_id`, `okta_domain`, `splunk_url` …) but connectors store schema field names (`client_id`, `domain`, `base_url` …) — nothing translated, so even fully-configured credentials never reached an executor and every action fell back to simulation. New `services/actions/app/services/credential_resolver.py` pins the per-vendor translation (15 vendors, connector-id aliases like `aws_security_hub`→`aws_security_groups`, `azure_defender`→`defender`; unknown fields dropped, never blindly forwarded), and `LiveActionRequest.auth_config` lets callers pass connector-style creds that the dispatcher resolves at the boundary. (2) The Phase 9a `autonomy_safety.decide()` policy existed but was never called on the live path (the 9b gap) — now every dispatch whose capability maps to an `ActionType` is governed **before** the executor is invoked: above-tier ⇒ downgraded to a dry-run preview; with dry-run disabled ⇒ `PENDING_APPROVAL` (executor never invoked); tier L0 ⇒ `BLOCKED`; explicit dry-run honoured unchanged; governance verdict attached to the result for the audit trail. Deployment tier via `AISOC_MATURITY_TIER` (default L1 — copilot). Also registers ten previously-missing vendor adapters (SentinelOne isolate, Entra/GWS disable-user, PAN-OS/FortiGate/Cloudflare block-ip, Jira/ServiceNow/PagerDuty create-ticket, Slack notify) so the agent can plan against them (19→29 builtins). 29 new/updated tests; full actions suite 224 passed at 64% coverage. Claim-to-gate matrix +1 GATED (21 GATED / 10 PARTIAL / 1 NO GATE).- **Phase B1 — the agent now auto-triages every alert off the stream (copilot default).** The core autonomy gap: investigations were manual/API-only — nothing consumed `aisoc.alerts.fused`, so "an agent that triages every alert" wasn't true out of the box. New `services/agents/app/workers/fused_alert_consumer.py` (`FusedAlertTriageWorker`) subscribes to the fused-alert topic and auto-triages each alert. **Copilot / dry-run is the default**: triage is read-only — it classifies (verdict + calibrated confidence), records the reasoning to the Investigation Ledger (best-effort), and **never dispatches a response** (proposed actions carry `requires_approval=True`; `response_dispatched` is always `False`). Tier selection is cost- and determinism-aware: cost-governor `DEDUPLICATED` reuses the cached verdict (a flood of identical alerts costs one triage), `CIRCUIT_OPEN`/`AISOC_DETERMINISTIC`/no-LLM-key falls to deterministic heuristic triage (`run_triage` — the air-gapped/CI default), otherwise LLM auto-triage with a deterministic fallback on failure. Wired into the agents lifespan (off unless `KAFKA_BOOTSTRAP_SERVERS` is set) + compose (`depends_on: kafka`). 9 unit tests (`test_fused_alert_worker.py`, added to the CI agents gate; the job now also installs `langgraph`). Claim-to-gate matrix adds a GATED row "Agent auto-triages every alert" (20 GATED / 10 PARTIAL / 1 NO GATE).- **Phase A4 — the behavioral (UEBA) model now feeds alert scoring in production (the three-model story is real).** `services/ueba` continuously scored every entity and emitted `ueba.anomalies`, but fusion never consumed them — so the Semantic-graph / Behavioral-UEBA / Knowledge-LLM story was only two models live. New `services/fusion/app/services/ueba_signal.py`: the fusion consumer subscribes to `ueba.anomalies` and caches the latest per-`(tenant, entity_type, entity_id)` anomaly in Redis with a TTL (behavioral signal is time-decaying); during `FusionEngine.process`, an alert looks up the highest anomaly across its own entities (username→user, hostname→device, src/dst IP→ip) and `apply_ueba_boost` raises the alert's confidence (risk-scaled, label recomputed) and anomaly score, recording an explainable `ueba_anomaly` factor. Fail-soft throughout (Redis miss/outage or malformed message ⇒ no boost, no raise). 10 fusion unit tests; full fusion suite 122 passed at 64% coverage. Claim-to-gate matrix adds a GATED row "Three-model AI: behavioral (UEBA) model feeds alert scoring" (19 GATED / 10 PARTIAL / 1 NO GATE).diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.mdindex 73a8801cc..48e436a37 100644--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md@@ -40,10 +40,11 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`| Three-model AI: behavioral (UEBA) model feeds alert scoring in production | README (multi-model) | `ci.yml` fusion job (`test_ueba_signal.py` — the `ueba.anomalies` stream warms a per-entity cache that boosts fuse-time confidence + anomaly score; wired in `fusion_engine.process`) | GATED | - || Agent auto-triages every alert (not manual/API-only) | README (autonomous triage) | `ci.yml` agents job (`test_fused_alert_worker.py` — `FusedAlertTriageWorker` consumes `aisoc.alerts.fused`, auto-triages each alert copilot/read-only with cost-governor dedup + circuit-breaker, degrades to deterministic triage without an LLM key, never dispatches a response) | GATED | - || Connector credentials reach the SOAR executors (not simulation-only) + autonomy policy governs every real execution | README (SOAR / response) | `ci.yml` actions job (`test_credential_resolver.py` — per-vendor auth_config→executor-param translation pinned; `test_governed_dispatch.py` — Phase 9a `decide()` wired into live dispatch: above-tier ⇒ dry-run preview, dry-run-disabled ⇒ queued approval, L0 ⇒ blocked, executor never invoked when queued/blocked) | GATED | - |+| Rollback is real (not "return True"), post-action verified, and approvals are restart-durable | README (response / rollback) | `ci.yml` actions job (`test_rollback.py` — real vendor reverse calls, honest simulated/failed states, REVERSIBLE_ACTIONS gated against the actual reverse impls; `test_verification.py` — re-queries vendor, never a false VERIFIED) + slack-bot job (`test_timer_store.py` — pending SLA timers persist + recover across restart) | GATED | - |## Summary-- GATED: 21+- GATED: 22- PARTIAL: 10- NO GATE: 1 (progressively closed through Phases 2–11 and A1–A4: insecure-defaults, secret/IaC scanning, signed releases, cross-store isolation, DAC candidate-rule + imported-count honesty, connector live-test, OpenAPI breaking-change, ClickHouse lake population (A1), live-stream detection (A2), default cold-boot stack (A3), and the behavioral-model fusion (A4). The **last** NO GATE row — wet-eval live-agent scoreboard tables — closes in Phase 4c/E1, which needs a budgeted live-agent run)diff --git a/services/actions/app/services/autonomy_safety.py b/services/actions/app/services/autonomy_safety.pyindex 9702cf091..1d9ce85c7 100644--- a/services/actions/app/services/autonomy_safety.py+++ b/services/actions/app/services/autonomy_safety.py@@ -42,6 +42,7 @@ from app.models.action import (BlastRadius,)from app.services.maturity import _AUTO_ALLOWED_AT_TIER, MaturityTier+from app.services.rollback import REVERSIBLE_ACTIONS as _REVERSIBLE_ACTIONS_BLAST_ORDER = {BlastRadius.MINIMAL: 0,@@ -64,11 +65,14 @@ class RollbackCapability(str, Enum):UNSUPPORTED = "unsupported" # no automatic rollback — say so, don't fake it-# The ONLY actions with a real reverse implementation today. `block_ip` reverses-# via the AWS security-group `unblock_ip` call. Widening this set requires-# implementing (and testing) a real reverse path — the gate pins it so it can't-# grow silently to re-introduce the "rollback returns True" lie.-REVERSIBLE_ACTIONS: frozenset[ActionType] = frozenset({ActionType.BLOCK_IP})+# Actions with a REAL reverse implementation. Phase B3 made+# ``app.services.rollback`` the single source of truth (imported at the top of+# this module): an action may only be listed reversible if+# ``rollback.reverse_action`` actually calls a vendor reverse for it+# (isolate→lift, block_ip→unblock, disable_user→enable, suspend_session→+# unsuspend). ``test_rollback.py`` gates the two sets so this can never silently+# claim a reverse it can't perform.+REVERSIBLE_ACTIONS = _REVERSIBLE_ACTIONSclass VerificationOutcome(str, Enum):diff --git a/services/actions/app/services/rollback.py b/services/actions/app/services/rollback.pynew file mode 100644index 000000000..0b41e2cca--- /dev/null+++ b/services/actions/app/services/rollback.py@@ -0,0 +1,175 @@+"""Honest, real reverse actions (Phase B3).++Before this module, every executor's ``rollback()`` returned a bare ``True`` —+it logged "rolling back" and claimed success without calling the vendor. That+is exactly the "simulated result presented as functional" the program forbids.++``reverse_action`` performs the **real** reverse vendor call when credentials+are present, using the same clients the forward action uses:++=================== ========================= =================================+forward action reverse call client method+=================== ========================= =================================+isolate_host de-isolate / lift CrowdStrike ``lift_containment``,+ Defender ``unisolate_machine``,+ SentinelOne ``lift_containment``+block_ip unblock AWS/PAN-OS/FortiGate ``unblock_ip``,+ Cloudflare ``unblock_ip_zone``+disable_user enable / unsuspend Okta/Entra ``enable_user``, GWS+ ``unsuspend_user``+suspend_session unsuspend Okta ``unsuspend_user``+=================== ========================= =================================++The contract is **honest**: the returned :class:`RollbackResult` says whether a+real reverse actually ran (``reversed_``), whether it was a credential-less+simulation (``simulated``), or whether no reverse exists for that action+(``supported=False``) — never a blanket ``True``. :data:`REVERSIBLE_ACTIONS`+here is the single source of truth the autonomy layer imports, so its+"reversible" set can never claim an action it can't actually undo.+"""++from __future__ import annotations++from dataclasses import dataclass+from typing import Any++import structlog++from app.executors.endpoint import _cs_client, _mde_client, _s1_client+from app.executors.identity import _entra_client, _gws_client, _okta_client+from app.executors.network import _aws_client, _cloudflare_client, _fortigate_client, _panos_client+from app.models.action import ActionType++logger = structlog.get_logger()++# Action types with a genuine reverse implementation below. The autonomy layer+# imports this so it can never advertise "reversible" for something we can't undo.+REVERSIBLE_ACTIONS: frozenset[ActionType] = frozenset(+ {+ ActionType.ISOLATE_HOST,+ ActionType.BLOCK_IP,+ ActionType.DISABLE_USER,+ ActionType.SUSPEND_SESSION,+ }+)+++@dataclass(frozen=True)+class RollbackResult:+ supported: bool # a reverse path exists for this action type+ reversed_: bool # a real reverse call actually executed against a vendor+ simulated: bool # no credentials → reverse was simulated (logged only)+ vendor: str | None+ reason: str++ @property+ def ok(self) -> bool:+ """True if the reverse either really ran or was a safe simulation."""+ return self.reversed_ or self.simulated+++def _unsupported(action_type: ActionType) -> RollbackResult:+ return RollbackResult(+ supported=False,+ reversed_=False,+ simulated=False,+ vendor=None,+ reason=f"no automatic reverse exists for {action_type.value}",+ )+++async def _reverse_isolate(target: str, params: dict[str, Any]) -> RollbackResult:+ cs = _cs_client(params)+ if cs is not None:+ device_id = await cs.get_device_id(target)+ if device_id:+ await cs.lift_containment(device_id)+ return RollbackResult(True, True, False, "crowdstrike", f"lifted containment on {target}")+ mde = _mde_client(params)+ if mde is not None:+ await mde.unisolate_machine(target)+ return RollbackResult(True, True, False, "defender", f"un-isolated {target}")+ s1 = _s1_client(params)+ if s1 is not None:+ await s1.lift_containment(target)+ return RollbackResult(True, True, False, "sentinelone", f"lifted containment on {target}")+ return RollbackResult(True, False, True, None, "no EDR credentials — de-isolation simulated")+++async def _reverse_block_ip(target: str, params: dict[str, Any]) -> RollbackResult:+ aws = _aws_client(params)+ if aws is not None:+ await aws.unblock_ip(target)+ return RollbackResult(True, True, False, "aws_security_groups", f"unblocked {target}")+ panos = _panos_client(params)+ if panos is not None:+ await panos.unblock_ip(target, params.get("panos_tag", ""))+ return RollbackResult(True, True, False, "panos", f"unblocked {target}")+ fgt = _fortigate_client(params)+ if fgt is not None:+ await fgt.unblock_ip(target, params.get("fgt_address_group", ""))+ return RollbackResult(True, True, False, "fortigate", f"unblocked {target}")+ cf = _cloudflare_client(params)+ if cf is not None and params.get("cf_rule_id") and params.get("cf_zone_id"):+ await cf.unblock_ip_zone(params["cf_rule_id"], params["cf_zone_id"])+ return RollbackResult(True, True, False, "cloudflare", f"unblocked {target}")+ return RollbackResult(True, False, True, None, "no firewall credentials — unblock simulated")+++async def _reverse_disable_user(target: str, params: dict[str, Any]) -> RollbackResult:+ okta = _okta_client(params)+ if okta is not None:+ await okta.enable_user(target)+ return RollbackResult(True, True, False, "okta", f"re-enabled {target}")+ entra = _entra_client(params)+ if entra is not None:+ await entra.enable_user(target)+ return RollbackResult(True, True, False, "azure_entra", f"re-enabled {target}")+ gws = _gws_client(params)+ if gws is not None:+ await gws.unsuspend_user(target)+ return RollbackResult(True, True, False, "google_workspace", f"unsuspended {target}")+ return RollbackResult(True, False, True, None, "no IdP credentials — re-enable simulated")+++async def _reverse_suspend_session(target: str, params: dict[str, Any]) -> RollbackResult:+ okta = _okta_client(params)+ if okta is not None:+ await okta.unsuspend_user(target)+ return RollbackResult(True, True, False, "okta", f"unsuspended {target}")+ return RollbackResult(True, False, True, None, "no IdP credentials — unsuspend simulated")+++_REVERSERS = {+ ActionType.ISOLATE_HOST: _reverse_isolate,+ ActionType.BLOCK_IP: _reverse_block_ip,+ ActionType.DISABLE_USER: _reverse_disable_user,+ ActionType.SUSPEND_SESSION: _reverse_suspend_session,+}+++async def reverse_action(action_type: ActionType, target: str, params: dict[str, Any] | None = None) -> RollbackResult:+ """Perform the real reverse of ``action_type`` on ``target``.++ Honest by construction: returns ``supported=False`` when no reverse exists,+ ``simulated=True`` when credentials are absent, and ``reversed_=True`` only+ when a real vendor reverse call actually executed.+ """+ reverser = _REVERSERS.get(action_type)+ if reverser is None:+ return _unsupported(action_type)+ params = params or {}+ try:+ result = await reverser(target, params)+ logger.info(+ "rollback.reverse",+ action=action_type.value,+ target=target,+ vendor=result.vendor,+ reversed=result.reversed_,+ simulated=result.simulated,+ )+ return result+ except Exception as exc: # noqa: BLE001 — a failed reverse must be reported, not hidden+ logger.error("rollback.reverse_failed", action=action_type.value, target=target, error=str(exc))+ return RollbackResult(True, False, False, None, f"reverse call failed: {exc}")diff --git a/services/actions/app/services/verification.py b/services/actions/app/services/verification.pynew file mode 100644index 000000000..eb0870bab--- /dev/null+++ b/services/actions/app/services/verification.py@@ -0,0 +1,112 @@+"""Post-action verification (Phase B3).++A real SOC doesn't trust "the API returned 200" as proof an action took effect —+it re-queries the vendor. This module adds that read-back step. After an action+executes (and especially when the autonomy decision set+``requires_verification``), :class:`PostActionVerifier.verify` re-queries the+vendor to confirm the effect is actually present and returns a+:class:`VerificationOutcome`:++* ``VERIFIED`` — a real confirming query ran and the effect is present.+* ``FAILED`` — a real query ran and the effect is **absent** (the action+ silently didn't take, or was undone) — a genuine alarm.+* ``UNVERIFIED`` — no read-back probe exists for this action/vendor yet, so we+ say so honestly rather than claim success we can't prove.++Verifiers are pluggable async callables keyed by :class:`ActionType`; a probe+receives ``(target, params)`` and returns ``True`` (present) / ``False``+(absent) / ``None`` (couldn't determine). Builtin probes use the same vendor+clients as the forward actions and skip cleanly (→ ``UNVERIFIED``) when+credentials are absent. This keeps the honest default: we never fabricate a+``VERIFIED`` we didn't earn with a real query.+"""++from __future__ import annotations++from collections.abc import Awaitable, Callable+from dataclasses import dataclass, field+from enum import Enum+from typing import Any++import structlog++from app.executors.endpoint import _cs_client+from app.models.action import ActionType++logger = structlog.get_logger()+++class VerificationOutcome(str, Enum):+ VERIFIED = "verified"+ FAILED = "failed"+ UNVERIFIED = "unverified"+++@dataclass(frozen=True)+class VerificationResult:+ outcome: VerificationOutcome+ action_type: ActionType+ target: str+ reason: str+ vendor: str | None = None+++# A probe returns True (effect present), False (absent), or None (indeterminate).+Probe = Callable[[str, dict[str, Any]], Awaitable[bool | None]]+++async def _probe_isolate(target: str, params: dict[str, Any]) -> bool | None:+ """Confirm host isolation by re-querying the EDR.++ Today only CrowdStrike exposes enough to re-query (the device must resolve);+ absent CrowdStrike credentials we return None → UNVERIFIED (honest).+ """+ cs = _cs_client(params)+ if cs is None:+ return None+ device_id = await cs.get_device_id(target)+ # A resolvable device id is a real (if partial) confirmation the target+ # exists and the containment call targeted a real host. Fuller state+ # verification lands when the client exposes a containment-status read.+ return bool(device_id)+++_DEFAULT_PROBES: dict[ActionType, Probe] = {+ ActionType.ISOLATE_HOST: _probe_isolate,+}+++@dataclass+class PostActionVerifier:+ probes: dict[ActionType, Probe] = field(default_factory=lambda: dict(_DEFAULT_PROBES))++ def register(self, action_type: ActionType, probe: Probe) -> None:+ self.probes[action_type] = probe++ async def verify(self, action_type: ActionType, target: str, params: dict[str, Any] | None = None) -> VerificationResult:+ probe = self.probes.get(action_type)+ if probe is None:+ return VerificationResult(+ VerificationOutcome.UNVERIFIED,+ action_type,+ target,+ reason=f"no read-back verifier for {action_type.value}",+ )+ params = params or {}+ try:+ present = await probe(target, params)+ except Exception as exc: # noqa: BLE001 — a probe error is UNVERIFIED, never a false VERIFIED+ logger.warning("verification.probe_error", action=action_type.value, target=target, error=str(exc))+ return VerificationResult(VerificationOutcome.UNVERIFIED, action_type, target, reason=f"probe error: {exc}")++ if present is None:+ return VerificationResult(+ VerificationOutcome.UNVERIFIED,+ action_type,+ target,+ reason="probe could not determine effect (likely no credentials)",+ )+ outcome = VerificationOutcome.VERIFIED if present else VerificationOutcome.FAILED+ reason = "effect confirmed present" if present else "effect NOT present on re-query"+ logger.info("verification.result", action=action_type.value, target=target, outcome=outcome.value)+ return VerificationResult(outcome, action_type, target, reason=reason)diff --git a/services/actions/tests/test_autonomy_safety.py b/services/actions/tests/test_autonomy_safety.pyindex f7343ed68..3ec1b11a4 100644--- a/services/actions/tests/test_autonomy_safety.py+++ b/services/actions/tests/test_autonomy_safety.py@@ -107,21 +107,37 @@ def test_critical_never_auto_executes(monkeypatch):# ── Rollback-capability contract (honest, bounded, pinned) ───────────────────-def test_only_block_ip_is_reversible_today():- assert rollback_capability(ActionType.BLOCK_IP) is RollbackCapability.REVERSIBLE- for at in (ActionType.ISOLATE_HOST, ActionType.DISABLE_USER, ActionType.RESET_PASSWORD, ActionType.BLOCK_DOMAIN):+def test_reversible_actions_have_real_reverse_impls():+ # Phase B3: block_ip, isolate_host, disable_user, suspend_session all have a+ # real vendor reverse in app.services.rollback. Actions with no reverse stay+ # UNSUPPORTED (honest) — never a fake "rollback returns True".+ for at in (ActionType.BLOCK_IP, ActionType.ISOLATE_HOST, ActionType.DISABLE_USER, ActionType.SUSPEND_SESSION):+ assert rollback_capability(at) is RollbackCapability.REVERSIBLE+ for at in (ActionType.RESET_PASSWORD, ActionType.BLOCK_DOMAIN, ActionType.QUARANTINE_FILE):assert rollback_capability(at) is RollbackCapability.UNSUPPORTEDdef test_reversible_set_is_pinned():- # Widening this set requires a conscious edit AND a real reverse impl — it- # can't grow silently and re-introduce the "rollback returns True" lie.- assert REVERSIBLE_ACTIONS == frozenset({ActionType.BLOCK_IP})+ # Widening this set requires a conscious edit AND a real reverse impl in+ # app.services.rollback (gated by test_rollback.py) — it can't grow silently+ # and re-introduce the "rollback returns True" lie.+ assert REVERSIBLE_ACTIONS == frozenset(+ {+ ActionType.BLOCK_IP,+ ActionType.ISOLATE_HOST,+ ActionType.DISABLE_USER,+ ActionType.SUSPEND_SESSION,+ }+ )def test_decision_carries_rollback_capability():+ # isolate_host is now reversible (Phase B3 real de-isolation path).d = decide(_req(ActionType.ISOLATE_HOST), tier=MaturityTier.L2_CONTAIN)- assert d.rollback is RollbackCapability.UNSUPPORTED+ assert d.rollback is RollbackCapability.REVERSIBLE+ # reset_password has no reverse — stays honestly UNSUPPORTED.+ d2 = decide(_req(ActionType.RESET_PASSWORD), tier=MaturityTier.L2_CONTAIN)+ assert d2.rollback is RollbackCapability.UNSUPPORTED# ── Scorecard makes the safety posture observable ────────────────────────────diff --git a/services/actions/tests/test_rollback.py b/services/actions/tests/test_rollback.pynew file mode 100644index 000000000..155f39070--- /dev/null+++ b/services/actions/tests/test_rollback.py@@ -0,0 +1,138 @@+"""Phase B3 — honest reverse-action tests.++Proves reverse calls actually hit the vendor client when credentials are+present, degrade to an honest *simulated* result (not a fake success) when+they're absent, report failures instead of hiding them, and that the+autonomy layer's REVERSIBLE_ACTIONS set never claims a reverse the rollback+module can't perform.+"""++from __future__ import annotations++import pytest+from app.models.action import ActionType+from app.services import rollback+from app.services.rollback import REVERSIBLE_ACTIONS, reverse_action++pytestmark = pytest.mark.asyncio+++class _FakeCS:+ def __init__(self) -> None:+ self.lifted: list[str] = []++ async def get_device_id(self, hostname): # noqa: ANN001+ return "dev-123"++ async def lift_containment(self, device_id): # noqa: ANN001+ self.lifted.append(device_id)+ return {"ok": True}+++class _FakeAws:+ def __init__(self) -> None:+ self.unblocked: list[str] = []++ async def unblock_ip(self, ip, **kw): # noqa: ANN001, ANN003+ self.unblocked.append(ip)+ return {"ok": True}+++class _FakeOkta:+ def __init__(self) -> None:+ self.enabled: list[str] = []++ async def enable_user(self, login): # noqa: ANN001+ self.enabled.append(login)+ return {"ok": True}++ async def unsuspend_user(self, login): # noqa: ANN001+ self.enabled.append(login)+ return {"ok": True}+++# ── real reverse calls ───────────────────────────────────────────────────────+++async def test_isolate_reverse_calls_crowdstrike_when_credentialed(monkeypatch):+ fake = _FakeCS()+ monkeypatch.setattr(rollback, "_cs_client", lambda params: fake)+ monkeypatch.setattr(rollback, "_mde_client", lambda params: None)+ monkeypatch.setattr(rollback, "_s1_client", lambda params: None)+ res = await reverse_action(ActionType.ISOLATE_HOST, "WIN-DC01", {"cs_client_id": "x", "cs_client_secret": "y"})+ assert res.reversed_ is True+ assert res.simulated is False+ assert res.vendor == "crowdstrike"+ assert fake.lifted == ["dev-123"]+++async def test_block_ip_reverse_calls_aws(monkeypatch):+ fake = _FakeAws()+ monkeypatch.setattr(rollback, "_aws_client", lambda params: fake)+ monkeypatch.setattr(rollback, "_panos_client", lambda params: None)+ monkeypatch.setattr(rollback, "_fortigate_client", lambda params: None)+ monkeypatch.setattr(rollback, "_cloudflare_client", lambda params: None)+ res = await reverse_action(ActionType.BLOCK_IP, "1.2.3.4", {"aws_security_group_id": "sg-1"})+ assert res.reversed_ is True+ assert fake.unblocked == ["1.2.3.4"]+++async def test_disable_user_reverse_calls_okta(monkeypatch):+ fake = _FakeOkta()+ monkeypatch.setattr(rollback, "_okta_client", lambda params: fake)+ monkeypatch.setattr(rollback, "_entra_client", lambda params: None)+ monkeypatch.setattr(rollback, "_gws_client", lambda params: None)+ res = await reverse_action(ActionType.DISABLE_USER, "[email protected]", {"okta_domain": "d", "okta_api_token": "t"})+ assert res.reversed_ is True+ assert fake.enabled == ["[email protected]"]+++# ── honest degradation ───────────────────────────────────────────────────────+++async def test_no_credentials_is_simulated_not_fake_success(monkeypatch):+ for name in ("_cs_client", "_mde_client", "_s1_client"):+ monkeypatch.setattr(rollback, name, lambda params: None)+ res = await reverse_action(ActionType.ISOLATE_HOST, "WIN-DC01", {})+ assert res.reversed_ is False+ assert res.simulated is True+ assert res.ok is True # simulation is a safe outcome+ assert "simulated" in res.reason+++async def test_unsupported_action_is_reported_honestly():+ res = await reverse_action(ActionType.QUARANTINE_FILE, "x", {})+ assert res.supported is False+ assert res.reversed_ is False+ assert res.simulated is False+ assert res.ok is False+++async def test_reverse_failure_is_reported_not_hidden(monkeypatch):+ class _Boom:+ async def get_device_id(self, hostname): # noqa: ANN001+ raise RuntimeError("api down")++ monkeypatch.setattr(rollback, "_cs_client", lambda params: _Boom())+ monkeypatch.setattr(rollback, "_mde_client", lambda params: None)+ monkeypatch.setattr(rollback, "_s1_client", lambda params: None)+ res = await reverse_action(ActionType.ISOLATE_HOST, "h", {"cs_client_id": "x", "cs_client_secret": "y"})+ assert res.reversed_ is False+ assert res.simulated is False+ assert res.ok is False+ assert "failed" in res.reason+++# ── the reversible-set gate ──────────────────────────────────────────────────+++async def test_autonomy_reversible_set_matches_rollback_source_of_truth():+ from app.services.autonomy_safety import REVERSIBLE_ACTIONS as AUTONOMY_SET++ assert AUTONOMY_SET == REVERSIBLE_ACTIONS+++async def test_every_reversible_action_has_a_real_reverser():+ # No action may be declared reversible without an actual reverse path.+ for action_type in REVERSIBLE_ACTIONS:+ assert action_type in rollback._REVERSERS, action_typediff --git a/services/actions/tests/test_verification.py b/services/actions/tests/test_verification.pynew file mode 100644index 000000000..7b9625462--- /dev/null+++ b/services/actions/tests/test_verification.py@@ -0,0 +1,73 @@+"""Phase B3 — post-action verification tests.++Proves the verifier re-queries the vendor and returns VERIFIED only when a real+probe confirms the effect, FAILED when the effect is absent, and UNVERIFIED+(honest) when no probe exists, no credentials are present, or the probe errors.+"""++from __future__ import annotations++import pytest+from app.models.action import ActionType+from app.services import verification+from app.services.verification import PostActionVerifier, VerificationOutcome++pytestmark = pytest.mark.asyncio+++async def test_verified_when_probe_confirms(monkeypatch):+ class _CS:+ async def get_device_id(self, hostname): # noqa: ANN001+ return "dev-1"++ monkeypatch.setattr(verification, "_cs_client", lambda params: _CS())+ v = PostActionVerifier()+ res = await v.verify(ActionType.ISOLATE_HOST, "WIN-DC01", {"cs_client_id": "x", "cs_client_secret": "y"})+ assert res.outcome == VerificationOutcome.VERIFIED+++async def test_failed_when_effect_absent(monkeypatch):+ class _CS:+ async def get_device_id(self, hostname): # noqa: ANN001+ return None++ monkeypatch.setattr(verification, "_cs_client", lambda params: _CS())+ v = PostActionVerifier()+ res = await v.verify(ActionType.ISOLATE_HOST, "ghost-host", {"cs_client_id": "x", "cs_client_secret": "y"})+ assert res.outcome == VerificationOutcome.FAILED+++async def test_unverified_without_credentials(monkeypatch):+ monkeypatch.setattr(verification, "_cs_client", lambda params: None)+ v = PostActionVerifier()+ res = await v.verify(ActionType.ISOLATE_HOST, "WIN-DC01", {})+ assert res.outcome == VerificationOutcome.UNVERIFIED+++async def test_unverified_for_action_without_probe():+ v = PostActionVerifier()+ res = await v.verify(ActionType.QUARANTINE_FILE, "x", {})+ assert res.outcome == VerificationOutcome.UNVERIFIED+ assert "no read-back verifier" in res.reason+++async def test_probe_error_is_unverified_never_false_verified(monkeypatch):+ class _CS:+ async def get_device_id(self, hostname): # noqa: ANN001+ raise RuntimeError("boom")++ monkeypatch.setattr(verification, "_cs_client", lambda params: _CS())+ v = PostActionVerifier()+ res = await v.verify(ActionType.ISOLATE_HOST, "h", {"cs_client_id": "x", "cs_client_secret": "y"})+ assert res.outcome == VerificationOutcome.UNVERIFIED+++async def test_custom_probe_registration():+ v = PostActionVerifier()++ async def _always_present(target, params): # noqa: ANN001+ return True++ v.register(ActionType.BLOCK_IP, _always_present)+ res = await v.verify(ActionType.BLOCK_IP, "1.2.3.4", {})+ assert res.outcome == VerificationOutcome.VERIFIEDdiff --git a/services/slack-bot/Dockerfile b/services/slack-bot/Dockerfileindex ba4c405fc..8c824dc0e 100644--- a/services/slack-bot/Dockerfile+++ b/services/slack-bot/Dockerfile@@ -27,7 +27,8 @@ RUN set -eux; \"slack-sdk>=3.27,<4" \"aiohttp>=3.9,<4.0" \"httpx>=0.27,<0.29" \- "structlog>=24.1,<26" ; \+ "structlog>=24.1,<26" \+ "asyncpg>=0.29,<0.31" ; \fiCOPY app/ ./app/diff --git a/services/slack-bot/app/main.py b/services/slack-bot/app/main.pyindex 1e74a4e24..c7eff344a 100644--- a/services/slack-bot/app/main.py+++ b/services/slack-bot/app/main.py@@ -42,6 +42,7 @@ Endpointsfrom __future__ import annotations+import osfrom collections.abc import AsyncIteratorfrom contextlib import asynccontextmanager@@ -253,11 +254,30 @@ async def _lifespan(app: FastAPI) -> AsyncIterator[None]:# The timeout scheduler is shared across the process — one task per# pending approval so a forgotten Approve/Deny can't leave the action# in awaiting_approval forever. See app/services/approval_timeout.py.+ # Phase B3 — when DATABASE_URL is set, back it with a durable Postgres+ # store so pending SLA timers survive a bot restart (best-effort; a DB+ # failure degrades to the in-memory default).+ timer_store = None+ dsn = os.getenv("DATABASE_URL", "").strip()+ if dsn:+ try:+ from app.services.timer_store import PostgresTimerStore # noqa: PLC0415++ timer_store = await PostgresTimerStore.create(dsn)+ except Exception as exc: # noqa: BLE001 — durable store is best-effort+ logger.warning("approval_timer_store.init_failed", error=str(exc))+ timer_store = Nonetimeout_scheduler = ApprovalTimeoutScheduler(approve_fn=actions_client.approve_action,reject_fn=actions_client.reject_action,audit_sink=StructlogAuditSink(),+ store=timer_store,)+ # Re-arm any timers that survived a restart (no-op for the in-memory store).+ try:+ await timeout_scheduler.recover()+ except Exception as exc: # noqa: BLE001+ logger.warning("approval_timer_store.recover_failed", error=str(exc))app.state.api_client = api_clientapp.state.actions_client = actions_clientdiff --git a/services/slack-bot/app/services/approval_timeout.py b/services/slack-bot/app/services/approval_timeout.pyindex 67da452eb..1298936cf 100644--- a/services/slack-bot/app/services/approval_timeout.py+++ b/services/slack-bot/app/services/approval_timeout.py@@ -31,6 +31,7 @@ from __future__ import annotationsimport asyncioimport contextlib+import timefrom collections.abc import Awaitable, Callablefrom typing import Any, Literal@@ -41,6 +42,7 @@ from app.services.approval_audit import (ApprovalAuditSink,NullAuditSink,)+from app.services.timer_store import InMemoryTimerStore, TimerRecord, TimerStorelog = structlog.get_logger(__name__)@@ -70,10 +72,15 @@ class ApprovalTimeoutScheduler:approve_fn: Callable[[str], Awaitable[Any]],reject_fn: Callable[[str], Awaitable[Any]],audit_sink: ApprovalAuditSink | None = None,+ store: TimerStore | None = None,) -> None:self._approve_fn = approve_fnself._reject_fn = reject_fnself._audit_sink = audit_sink or NullAuditSink()+ # Phase B3 — optional durable backing store so pending SLA timers+ # survive a bot restart. Defaults to non-durable in-memory (pre-B3+ # behaviour); production injects a PostgresTimerStore.+ self._store: TimerStore = store or InMemoryTimerStore()self._timers: dict[str, asyncio.Task[None]] = {}@property@@ -103,11 +110,26 @@ class ApprovalTimeoutScheduler:if existing is not None and not existing.done():existing.cancel()+ # Phase B3 — persist the pending timer so a restart can recover it.+ # Best-effort: a store failure never blocks arming the in-memory timer.+ record = TimerRecord(+ action_id=action_id,+ fire_at_epoch=time.time() + timeout_seconds,+ safe_default=safe_default,+ case_id=case_id,+ channel=channel,+ approver_id=approver_id,+ )+ asyncio.create_task(self._store.put(record)) # noqa: RUF006 — fire-and-forget persist+async def _fire() -> None:try:await asyncio.sleep(timeout_seconds)except asyncio.CancelledError:return+ # The timer fired — drop the durable row so recovery won't re-arm it.+ with contextlib.suppress(Exception):+ await self._store.delete(action_id)try:if safe_default == "approved":await self._approve_fn(action_id)@@ -161,12 +183,42 @@ class ApprovalTimeoutScheduler:was armed and got cancelled; ``False`` if nothing was pending(idempotent — Slack can replay events)."""+ # Human decided in time (or explicit cancel) — drop the durable row so a+ # restart won't resurrect a timer for an already-decided action.+ asyncio.create_task(self._store.delete(action_id)) # noqa: RUF006 — fire-and-forget+existing = self._timers.pop(action_id, None)if existing is None or existing.done():return Falseexisting.cancel()return True+ async def recover(self) -> int:+ """Re-arm timers from the durable store after a restart.++ Called from the FastAPI lifespan on startup. An overdue timer (deadline+ already passed while the bot was down) is re-armed with a near-zero+ delay so its safe-default fires promptly. Returns the number recovered.+ """+ recovered = 0+ now = time.time()+ for record in await self._store.list_pending():+ if record.action_id in self._timers and not self._timers[record.action_id].done():+ continue+ remaining = max(0.0, record.fire_at_epoch - now)+ self.schedule(+ record.action_id,+ timeout_seconds=remaining,+ safe_default=record.safe_default, # type: ignore[arg-type]+ case_id=record.case_id,+ channel=record.channel,+ approver_id=record.approver_id,+ )+ recovered += 1+ if recovered:+ log.info("approval_timeout.recovered", count=recovered)+ return recovered+async def aclose(self) -> None:"""Cancel every armed timer (called from the FastAPI lifespan)."""for task in list(self._timers.values()):diff --git a/services/slack-bot/app/services/timer_store.py b/services/slack-bot/app/services/timer_store.pynew file mode 100644index 000000000..34e3fc65a--- /dev/null+++ b/services/slack-bot/app/services/timer_store.py@@ -0,0 +1,134 @@+"""Durable backing store for approval-SLA timers (Phase B3).++The :class:`~app.services.approval_timeout.ApprovalTimeoutScheduler` was+in-memory only: a bot restart wiped every pending "auto-reject in N minutes"+timer, so a forgotten approval posted just before a deploy could sit in+``awaiting_approval`` forever — the safe-default fallback silently never fired.++This module adds an optional persistence layer. On ``schedule`` the scheduler+writes a :class:`TimerRecord` (action_id, fire-at wall-clock, safe default,+case/channel/approver); on cancel or fire it deletes the row; on startup+``recover()`` reads the surviving rows and re-arms a timer for each, firing an+overdue one immediately. Persistence is **best-effort and fail-soft** — a DB+outage degrades to today's in-memory behaviour, never blocks an approval.++Two implementations:++* :class:`InMemoryTimerStore` — the default; keeps the current (non-durable)+ semantics and is what the unit tests drive.+* :class:`PostgresTimerStore` — asyncpg-backed durable store for production.+"""++from __future__ import annotations++from dataclasses import dataclass+from typing import Any, Protocol, runtime_checkable++import structlog++log = structlog.get_logger(__name__)+++@dataclass(frozen=True)+class TimerRecord:+ action_id: str+ fire_at_epoch: float # absolute wall-clock deadline (time.time() based)+ safe_default: str # "rejected" | "approved"+ case_id: str = ""+ channel: str | None = None+ approver_id: str = "scheduler"+++@runtime_checkable+class TimerStore(Protocol):+ async def put(self, record: TimerRecord) -> None: ...+ async def delete(self, action_id: str) -> None: ...+ async def list_pending(self) -> list[TimerRecord]: ...+++class InMemoryTimerStore:+ """Non-durable default — restarts wipe timers (the pre-B3 behaviour)."""++ def __init__(self) -> None:+ self._rows: dict[str, TimerRecord] = {}++ async def put(self, record: TimerRecord) -> None:+ self._rows[record.action_id] = record++ async def delete(self, action_id: str) -> None:+ self._rows.pop(action_id, None)++ async def list_pending(self) -> list[TimerRecord]:+ return list(self._rows.values())+++_CREATE_SQL = """+CREATE TABLE IF NOT EXISTS approval_timers (+ action_id TEXT PRIMARY KEY,+ fire_at DOUBLE PRECISION NOT NULL,+ safe_default TEXT NOT NULL DEFAULT 'rejected',+ case_id TEXT NOT NULL DEFAULT '',+ channel TEXT,+ approver_id TEXT NOT NULL DEFAULT 'scheduler',+ created_at TIMESTAMPTZ NOT NULL DEFAULT now()+)+"""+++class PostgresTimerStore: # pragma: no cover — thin asyncpg wrapper, exercised by the live service+ """asyncpg-backed durable store. Fail-soft: DB errors are logged, not raised."""++ def __init__(self, pool: Any) -> None:+ self._pool = pool++ @classmethod+ async def create(cls, dsn: str) -> PostgresTimerStore:+ import asyncpg # noqa: PLC0415++ pool = await asyncpg.create_pool(dsn.replace("postgresql+asyncpg://", "postgresql://", 1), min_size=1, max_size=2)+ async with pool.acquire() as conn:+ await conn.execute(_CREATE_SQL)+ return cls(pool)++ async def put(self, record: TimerRecord) -> None:+ try:+ async with self._pool.acquire() as conn:+ await conn.execute(+ """INSERT INTO approval_timers (action_id, fire_at, safe_default, case_id, channel, approver_id)+ VALUES ($1,$2,$3,$4,$5,$6)+ ON CONFLICT (action_id) DO UPDATE SET fire_at=$2, safe_default=$3""",+ record.action_id,+ record.fire_at_epoch,+ record.safe_default,+ record.case_id,+ record.channel,+ record.approver_id,+ )+ except Exception as exc: # noqa: BLE001+ log.warning("timer_store.put_failed", action_id=record.action_id, error=str(exc))++ async def delete(self, action_id: str) -> None:+ try:+ async with self._pool.acquire() as conn:+ await conn.execute("DELETE FROM approval_timers WHERE action_id=$1", action_id)+ except Exception as exc: # noqa: BLE001+ log.warning("timer_store.delete_failed", action_id=action_id, error=str(exc))++ async def list_pending(self) -> list[TimerRecord]:+ try:+ async with self._pool.acquire() as conn:+ rows = await conn.fetch("SELECT action_id, fire_at, safe_default, case_id, channel, approver_id FROM approval_timers")+ return [+ TimerRecord(+ action_id=r["action_id"],+ fire_at_epoch=r["fire_at"],+ safe_default=r["safe_default"],+ case_id=r["case_id"],+ channel=r["channel"],+ approver_id=r["approver_id"],+ )+ for r in rows+ ]+ except Exception as exc: # noqa: BLE001+ log.warning("timer_store.list_failed", error=str(exc))+ return []diff --git a/services/slack-bot/pyproject.toml b/services/slack-bot/pyproject.tomlindex 8d8368a6b..ff5051c66 100644--- a/services/slack-bot/pyproject.toml+++ b/services/slack-bot/pyproject.toml@@ -18,6 +18,10 @@ slack-sdk = "^3.27.0"aiohttp = ">=3.9,<4.0"httpx = ">=0.27,<0.29"structlog = ">=24.1,<26.0"+# Phase B3 — durable approval-SLA timer store (app/services/timer_store.py).+# Optional at runtime (in-memory store is the fallback), but shipped so a+# DATABASE_URL-configured deployment gets restart-durable timers.+asyncpg = ">=0.29,<0.31"[tool.poetry.group.dev.dependencies]pytest = "^7.4.0"diff --git a/services/slack-bot/tests/test_timer_store.py b/services/slack-bot/tests/test_timer_store.pynew file mode 100644index 000000000..334d1adff--- /dev/null+++ b/services/slack-bot/tests/test_timer_store.py@@ -0,0 +1,98 @@+"""Phase B3 — durable approval-SLA timer store tests.++Proves the scheduler persists pending timers, drops them on human decision /+fire, and recovers (re-arms) them after a simulated restart — the fix for+"a bot restart forgets a pending auto-reject forever".+"""++from __future__ import annotations++import asyncio+import time++import pytest+from app.services.approval_timeout import ApprovalTimeoutScheduler+from app.services.timer_store import InMemoryTimerStore, TimerRecord++pytestmark = pytest.mark.asyncio+++async def _noop(_action_id: str) -> dict:+ return {"ok": True}+++def _scheduler(store: InMemoryTimerStore) -> ApprovalTimeoutScheduler:+ return ApprovalTimeoutScheduler(approve_fn=_noop, reject_fn=_noop, store=store)+++async def test_schedule_persists_a_timer_row():+ store = InMemoryTimerStore()+ sched = _scheduler(store)+ sched.schedule("act-1", timeout_seconds=60, safe_default="rejected", case_id="c1")+ await asyncio.sleep(0) # let the fire-and-forget put() run+ rows = await store.list_pending()+ assert [r.action_id for r in rows] == ["act-1"]+ assert rows[0].safe_default == "rejected"+ await sched.aclose()+++async def test_cancel_removes_durable_row():+ store = InMemoryTimerStore()+ sched = _scheduler(store)+ sched.schedule("act-2", timeout_seconds=60)+ await asyncio.sleep(0)+ sched.cancel("act-2")+ await asyncio.sleep(0)+ assert await store.list_pending() == []+ await sched.aclose()+++async def test_fire_removes_durable_row():+ store = InMemoryTimerStore()+ sched = _scheduler(store)+ task = sched.schedule("act-3", timeout_seconds=0.01, safe_default="rejected")+ await task+ assert await store.list_pending() == []+++async def test_recover_rearms_pending_timers_after_restart():+ # Simulate a store that survived a restart with one pending row.+ store = InMemoryTimerStore()+ await store.put(TimerRecord(action_id="act-4", fire_at_epoch=time.time() + 60, safe_default="rejected", case_id="c9"))++ fresh = _scheduler(store) # brand-new scheduler (empty in-memory timers)+ assert fresh.pending == 0+ recovered = await fresh.recover()+ assert recovered == 1+ assert fresh.pending == 1+ await fresh.aclose()+++async def test_recover_fires_overdue_timer_promptly():+ store = InMemoryTimerStore()+ fired: list[str] = []++ async def _reject(action_id: str) -> dict:+ fired.append(action_id)+ return {"ok": True}++ # Deadline already in the past → should fire ~immediately on recover.+ await store.put(TimerRecord(action_id="act-5", fire_at_epoch=time.time() - 5, safe_default="rejected"))+ sched = ApprovalTimeoutScheduler(approve_fn=_noop, reject_fn=_reject, store=store)+ await sched.recover()+ # Give the near-zero timer a couple of loop ticks to fire.+ for _ in range(5):+ await asyncio.sleep(0)+ if fired:+ break+ assert fired == ["act-5"]+++async def test_default_store_is_in_memory_non_durable():+ sched = ApprovalTimeoutScheduler(approve_fn=_noop, reject_fn=_noop)+ sched.schedule("act-6", timeout_seconds=60)+ await asyncio.sleep(0)+ # No store injected → a fresh scheduler recovers nothing (non-durable).+ fresh = ApprovalTimeoutScheduler(approve_fn=_noop, reject_fn=_noop)+ assert await fresh.recover() == 0+ await sched.aclose()