Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

feat(phase-B3): real rollback + post-action verification + durable approvals (#454)

Beenu Arorad1ec16a8fce2
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3b61c3779..da09c35c6 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
+- **Phase B3 — rollback is real, actions are post-verified, and approval SLA timers survive restarts.** Three "honest response" gaps closed. (1) **Real rollback:** every executor's `rollback()` previously returned a bare `True` — logging "rolling back" without calling the vendor. New `services/actions/app/services/rollback.py` performs the **real** reverse via the same clients (isolate→`lift_containment`/`unisolate_machine`, block_ip→`unblock_ip`/`unblock_ip_zone`, disable_user→`enable_user`/`unsuspend_user`, suspend_session→`unsuspend_user`) and returns an honest `RollbackResult` (`reversed_` / `simulated` / `supported`) — never a fake success; a failed reverse is reported, not hidden. `autonomy_safety.REVERSIBLE_ACTIONS` now imports from this module (single source of truth) and `test_rollback.py` gates the two sets so an action can't be declared reversible without a real reverse. (2) **Post-action verification:** new `services/actions/app/services/verification.py` re-queries the vendor to confirm the effect is actually present (`VERIFIED` / `FAILED` / honest `UNVERIFIED` when no probe or no creds) — a probe error is never a false `VERIFIED`. (3) **Durable approvals:** `services/slack-bot/app/services/timer_store.py` adds a Postgres-backed `TimerStore`; the `ApprovalTimeoutScheduler` persists pending SLA timers and `recover()` re-arms them on startup (firing overdue ones promptly), so a bot restart can no longer strand a forgotten approval forever. Fail-soft to in-memory when no DB. 25 new tests across actions + slack-bot. Claim-to-gate matrix +1 GATED (22 GATED / 10 PARTIAL / 1 NO GATE).
- **Phase B2 — connector credentials now reach the SOAR executors, and the autonomy policy governs every real execution.** Two wiring gaps closed. (1) Executors read vendor-prefixed parameter keys (`cs_client_id`, `okta_domain`, `splunk_url` …) but connectors store schema field names (`client_id`, `domain`, `base_url` …) — nothing translated, so even fully-configured credentials never reached an executor and every action fell back to simulation. New `services/actions/app/services/credential_resolver.py` pins the per-vendor translation (15 vendors, connector-id aliases like `aws_security_hub`→`aws_security_groups`, `azure_defender`→`defender`; unknown fields dropped, never blindly forwarded), and `LiveActionRequest.auth_config` lets callers pass connector-style creds that the dispatcher resolves at the boundary. (2) The Phase 9a `autonomy_safety.decide()` policy existed but was never called on the live path (the 9b gap) — now every dispatch whose capability maps to an `ActionType` is governed **before** the executor is invoked: above-tier ⇒ downgraded to a dry-run preview; with dry-run disabled ⇒ `PENDING_APPROVAL` (executor never invoked); tier L0 ⇒ `BLOCKED`; explicit dry-run honoured unchanged; governance verdict attached to the result for the audit trail. Deployment tier via `AISOC_MATURITY_TIER` (default L1 — copilot). Also registers ten previously-missing vendor adapters (SentinelOne isolate, Entra/GWS disable-user, PAN-OS/FortiGate/Cloudflare block-ip, Jira/ServiceNow/PagerDuty create-ticket, Slack notify) so the agent can plan against them (19→29 builtins). 29 new/updated tests; full actions suite 224 passed at 64% coverage. Claim-to-gate matrix +1 GATED (21 GATED / 10 PARTIAL / 1 NO GATE).
- **Phase B1 — the agent now auto-triages every alert off the stream (copilot default).** The core autonomy gap: investigations were manual/API-only — nothing consumed `aisoc.alerts.fused`, so "an agent that triages every alert" wasn't true out of the box. New `services/agents/app/workers/fused_alert_consumer.py` (`FusedAlertTriageWorker`) subscribes to the fused-alert topic and auto-triages each alert. **Copilot / dry-run is the default**: triage is read-only — it classifies (verdict + calibrated confidence), records the reasoning to the Investigation Ledger (best-effort), and **never dispatches a response** (proposed actions carry `requires_approval=True`; `response_dispatched` is always `False`). Tier selection is cost- and determinism-aware: cost-governor `DEDUPLICATED` reuses the cached verdict (a flood of identical alerts costs one triage), `CIRCUIT_OPEN`/`AISOC_DETERMINISTIC`/no-LLM-key falls to deterministic heuristic triage (`run_triage` — the air-gapped/CI default), otherwise LLM auto-triage with a deterministic fallback on failure. Wired into the agents lifespan (off unless `KAFKA_BOOTSTRAP_SERVERS` is set) + compose (`depends_on: kafka`). 9 unit tests (`test_fused_alert_worker.py`, added to the CI agents gate; the job now also installs `langgraph`). Claim-to-gate matrix adds a GATED row "Agent auto-triages every alert" (20 GATED / 10 PARTIAL / 1 NO GATE).
- **Phase A4 — the behavioral (UEBA) model now feeds alert scoring in production (the three-model story is real).** `services/ueba` continuously scored every entity and emitted `ueba.anomalies`, but fusion never consumed them — so the Semantic-graph / Behavioral-UEBA / Knowledge-LLM story was only two models live. New `services/fusion/app/services/ueba_signal.py`: the fusion consumer subscribes to `ueba.anomalies` and caches the latest per-`(tenant, entity_type, entity_id)` anomaly in Redis with a TTL (behavioral signal is time-decaying); during `FusionEngine.process`, an alert looks up the highest anomaly across its own entities (username→user, hostname→device, src/dst IP→ip) and `apply_ueba_boost` raises the alert's confidence (risk-scaled, label recomputed) and anomaly score, recording an explainable `ueba_anomaly` factor. Fail-soft throughout (Redis miss/outage or malformed message ⇒ no boost, no raise). 10 fusion unit tests; full fusion suite 122 passed at 64% coverage. Claim-to-gate matrix adds a GATED row "Three-model AI: behavioral (UEBA) model feeds alert scoring" (19 GATED / 10 PARTIAL / 1 NO GATE).
diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.md
index 73a8801cc..48e436a37 100644
--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md
+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md
@@ -40,10 +40,11 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
| Three-model AI: behavioral (UEBA) model feeds alert scoring in production | README (multi-model) | `ci.yml` fusion job (`test_ueba_signal.py` — the `ueba.anomalies` stream warms a per-entity cache that boosts fuse-time confidence + anomaly score; wired in `fusion_engine.process`) | GATED | - |
| Agent auto-triages every alert (not manual/API-only) | README (autonomous triage) | `ci.yml` agents job (`test_fused_alert_worker.py` — `FusedAlertTriageWorker` consumes `aisoc.alerts.fused`, auto-triages each alert copilot/read-only with cost-governor dedup + circuit-breaker, degrades to deterministic triage without an LLM key, never dispatches a response) | GATED | - |
| Connector credentials reach the SOAR executors (not simulation-only) + autonomy policy governs every real execution | README (SOAR / response) | `ci.yml` actions job (`test_credential_resolver.py` — per-vendor auth_config→executor-param translation pinned; `test_governed_dispatch.py` — Phase 9a `decide()` wired into live dispatch: above-tier ⇒ dry-run preview, dry-run-disabled ⇒ queued approval, L0 ⇒ blocked, executor never invoked when queued/blocked) | GATED | - |
+| Rollback is real (not "return True"), post-action verified, and approvals are restart-durable | README (response / rollback) | `ci.yml` actions job (`test_rollback.py` — real vendor reverse calls, honest simulated/failed states, REVERSIBLE_ACTIONS gated against the actual reverse impls; `test_verification.py` — re-queries vendor, never a false VERIFIED) + slack-bot job (`test_timer_store.py` — pending SLA timers persist + recover across restart) | GATED | - |
## Summary
-- GATED: 21
+- GATED: 22
- PARTIAL: 10
- NO GATE: 1 (progressively closed through Phases 2–11 and A1–A4: insecure-defaults, secret/IaC scanning, signed releases, cross-store isolation, DAC candidate-rule + imported-count honesty, connector live-test, OpenAPI breaking-change, ClickHouse lake population (A1), live-stream detection (A2), default cold-boot stack (A3), and the behavioral-model fusion (A4). The **last** NO GATE row — wet-eval live-agent scoreboard tables — closes in Phase 4c/E1, which needs a budgeted live-agent run)
diff --git a/services/actions/app/services/autonomy_safety.py b/services/actions/app/services/autonomy_safety.py
index 9702cf091..1d9ce85c7 100644
--- a/services/actions/app/services/autonomy_safety.py
+++ b/services/actions/app/services/autonomy_safety.py
@@ -42,6 +42,7 @@ from app.models.action import (
BlastRadius,
)
from app.services.maturity import _AUTO_ALLOWED_AT_TIER, MaturityTier
+from app.services.rollback import REVERSIBLE_ACTIONS as _REVERSIBLE_ACTIONS
_BLAST_ORDER = {
BlastRadius.MINIMAL: 0,
@@ -64,11 +65,14 @@ class RollbackCapability(str, Enum):
UNSUPPORTED = "unsupported" # no automatic rollback — say so, don't fake it
-# The ONLY actions with a real reverse implementation today. `block_ip` reverses
-# via the AWS security-group `unblock_ip` call. Widening this set requires
-# implementing (and testing) a real reverse path — the gate pins it so it can't
-# grow silently to re-introduce the "rollback returns True" lie.
-REVERSIBLE_ACTIONS: frozenset[ActionType] = frozenset({ActionType.BLOCK_IP})
+# Actions with a REAL reverse implementation. Phase B3 made
+# ``app.services.rollback`` the single source of truth (imported at the top of
+# this module): an action may only be listed reversible if
+# ``rollback.reverse_action`` actually calls a vendor reverse for it
+# (isolate→lift, block_ip→unblock, disable_user→enable, suspend_session→
+# unsuspend). ``test_rollback.py`` gates the two sets so this can never silently
+# claim a reverse it can't perform.
+REVERSIBLE_ACTIONS = _REVERSIBLE_ACTIONS
class VerificationOutcome(str, Enum):
diff --git a/services/actions/app/services/rollback.py b/services/actions/app/services/rollback.py
new file mode 100644
index 000000000..0b41e2cca
--- /dev/null
+++ b/services/actions/app/services/rollback.py
@@ -0,0 +1,175 @@
+"""Honest, real reverse actions (Phase B3).
+
+Before this module, every executor's ``rollback()`` returned a bare ``True`` —
+it logged "rolling back" and claimed success without calling the vendor. That
+is exactly the "simulated result presented as functional" the program forbids.
+
+``reverse_action`` performs the **real** reverse vendor call when credentials
+are present, using the same clients the forward action uses:
+
+=================== ========================= =================================
+forward action reverse call client method
+=================== ========================= =================================
+isolate_host de-isolate / lift CrowdStrike ``lift_containment``,
+ Defender ``unisolate_machine``,
+ SentinelOne ``lift_containment``
+block_ip unblock AWS/PAN-OS/FortiGate ``unblock_ip``,
+ Cloudflare ``unblock_ip_zone``
+disable_user enable / unsuspend Okta/Entra ``enable_user``, GWS
+ ``unsuspend_user``
+suspend_session unsuspend Okta ``unsuspend_user``
+=================== ========================= =================================
+
+The contract is **honest**: the returned :class:`RollbackResult` says whether a
+real reverse actually ran (``reversed_``), whether it was a credential-less
+simulation (``simulated``), or whether no reverse exists for that action
+(``supported=False``) — never a blanket ``True``. :data:`REVERSIBLE_ACTIONS`
+here is the single source of truth the autonomy layer imports, so its
+"reversible" set can never claim an action it can't actually undo.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any
+
+import structlog
+
+from app.executors.endpoint import _cs_client, _mde_client, _s1_client
+from app.executors.identity import _entra_client, _gws_client, _okta_client
+from app.executors.network import _aws_client, _cloudflare_client, _fortigate_client, _panos_client
+from app.models.action import ActionType
+
+logger = structlog.get_logger()
+
+# Action types with a genuine reverse implementation below. The autonomy layer
+# imports this so it can never advertise "reversible" for something we can't undo.
+REVERSIBLE_ACTIONS: frozenset[ActionType] = frozenset(
+ {
+ ActionType.ISOLATE_HOST,
+ ActionType.BLOCK_IP,
+ ActionType.DISABLE_USER,
+ ActionType.SUSPEND_SESSION,
+ }
+)
+
+
+@dataclass(frozen=True)
+class RollbackResult:
+ supported: bool # a reverse path exists for this action type
+ reversed_: bool # a real reverse call actually executed against a vendor
+ simulated: bool # no credentials → reverse was simulated (logged only)
+ vendor: str | None
+ reason: str
+
+ @property
+ def ok(self) -> bool:
+ """True if the reverse either really ran or was a safe simulation."""
+ return self.reversed_ or self.simulated
+
+
+def _unsupported(action_type: ActionType) -> RollbackResult:
+ return RollbackResult(
+ supported=False,
+ reversed_=False,
+ simulated=False,
+ vendor=None,
+ reason=f"no automatic reverse exists for {action_type.value}",
+ )
+
+
+async def _reverse_isolate(target: str, params: dict[str, Any]) -> RollbackResult:
+ cs = _cs_client(params)
+ if cs is not None:
+ device_id = await cs.get_device_id(target)
+ if device_id:
+ await cs.lift_containment(device_id)
+ return RollbackResult(True, True, False, "crowdstrike", f"lifted containment on {target}")
+ mde = _mde_client(params)
+ if mde is not None:
+ await mde.unisolate_machine(target)
+ return RollbackResult(True, True, False, "defender", f"un-isolated {target}")
+ s1 = _s1_client(params)
+ if s1 is not None:
+ await s1.lift_containment(target)
+ return RollbackResult(True, True, False, "sentinelone", f"lifted containment on {target}")
+ return RollbackResult(True, False, True, None, "no EDR credentials — de-isolation simulated")
+
+
+async def _reverse_block_ip(target: str, params: dict[str, Any]) -> RollbackResult:
+ aws = _aws_client(params)
+ if aws is not None:
+ await aws.unblock_ip(target)
+ return RollbackResult(True, True, False, "aws_security_groups", f"unblocked {target}")
+ panos = _panos_client(params)
+ if panos is not None:
+ await panos.unblock_ip(target, params.get("panos_tag", ""))
+ return RollbackResult(True, True, False, "panos", f"unblocked {target}")
+ fgt = _fortigate_client(params)
+ if fgt is not None:
+ await fgt.unblock_ip(target, params.get("fgt_address_group", ""))
+ return RollbackResult(True, True, False, "fortigate", f"unblocked {target}")
+ cf = _cloudflare_client(params)
+ if cf is not None and params.get("cf_rule_id") and params.get("cf_zone_id"):
+ await cf.unblock_ip_zone(params["cf_rule_id"], params["cf_zone_id"])
+ return RollbackResult(True, True, False, "cloudflare", f"unblocked {target}")
+ return RollbackResult(True, False, True, None, "no firewall credentials — unblock simulated")
+
+
+async def _reverse_disable_user(target: str, params: dict[str, Any]) -> RollbackResult:
+ okta = _okta_client(params)
+ if okta is not None:
+ await okta.enable_user(target)
+ return RollbackResult(True, True, False, "okta", f"re-enabled {target}")
+ entra = _entra_client(params)
+ if entra is not None:
+ await entra.enable_user(target)
+ return RollbackResult(True, True, False, "azure_entra", f"re-enabled {target}")
+ gws = _gws_client(params)
+ if gws is not None:
+ await gws.unsuspend_user(target)
+ return RollbackResult(True, True, False, "google_workspace", f"unsuspended {target}")
+ return RollbackResult(True, False, True, None, "no IdP credentials — re-enable simulated")
+
+
+async def _reverse_suspend_session(target: str, params: dict[str, Any]) -> RollbackResult:
+ okta = _okta_client(params)
+ if okta is not None:
+ await okta.unsuspend_user(target)
+ return RollbackResult(True, True, False, "okta", f"unsuspended {target}")
+ return RollbackResult(True, False, True, None, "no IdP credentials — unsuspend simulated")
+
+
+_REVERSERS = {
+ ActionType.ISOLATE_HOST: _reverse_isolate,
+ ActionType.BLOCK_IP: _reverse_block_ip,
+ ActionType.DISABLE_USER: _reverse_disable_user,
+ ActionType.SUSPEND_SESSION: _reverse_suspend_session,
+}
+
+
+async def reverse_action(action_type: ActionType, target: str, params: dict[str, Any] | None = None) -> RollbackResult:
+ """Perform the real reverse of ``action_type`` on ``target``.
+
+ Honest by construction: returns ``supported=False`` when no reverse exists,
+ ``simulated=True`` when credentials are absent, and ``reversed_=True`` only
+ when a real vendor reverse call actually executed.
+ """
+ reverser = _REVERSERS.get(action_type)
+ if reverser is None:
+ return _unsupported(action_type)
+ params = params or {}
+ try:
+ result = await reverser(target, params)
+ logger.info(
+ "rollback.reverse",
+ action=action_type.value,
+ target=target,
+ vendor=result.vendor,
+ reversed=result.reversed_,
+ simulated=result.simulated,
+ )
+ return result
+ except Exception as exc: # noqa: BLE001 — a failed reverse must be reported, not hidden
+ logger.error("rollback.reverse_failed", action=action_type.value, target=target, error=str(exc))
+ return RollbackResult(True, False, False, None, f"reverse call failed: {exc}")
diff --git a/services/actions/app/services/verification.py b/services/actions/app/services/verification.py
new file mode 100644
index 000000000..eb0870bab
--- /dev/null
+++ b/services/actions/app/services/verification.py
@@ -0,0 +1,112 @@
+"""Post-action verification (Phase B3).
+
+A real SOC doesn't trust "the API returned 200" as proof an action took effect —
+it re-queries the vendor. This module adds that read-back step. After an action
+executes (and especially when the autonomy decision set
+``requires_verification``), :class:`PostActionVerifier.verify` re-queries the
+vendor to confirm the effect is actually present and returns a
+:class:`VerificationOutcome`:
+
+* ``VERIFIED`` — a real confirming query ran and the effect is present.
+* ``FAILED`` — a real query ran and the effect is **absent** (the action
+ silently didn't take, or was undone) — a genuine alarm.
+* ``UNVERIFIED`` — no read-back probe exists for this action/vendor yet, so we
+ say so honestly rather than claim success we can't prove.
+
+Verifiers are pluggable async callables keyed by :class:`ActionType`; a probe
+receives ``(target, params)`` and returns ``True`` (present) / ``False``
+(absent) / ``None`` (couldn't determine). Builtin probes use the same vendor
+clients as the forward actions and skip cleanly (→ ``UNVERIFIED``) when
+credentials are absent. This keeps the honest default: we never fabricate a
+``VERIFIED`` we didn't earn with a real query.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Awaitable, Callable
+from dataclasses import dataclass, field
+from enum import Enum
+from typing import Any
+
+import structlog
+
+from app.executors.endpoint import _cs_client
+from app.models.action import ActionType
+
+logger = structlog.get_logger()
+
+
+class VerificationOutcome(str, Enum):
+ VERIFIED = "verified"
+ FAILED = "failed"
+ UNVERIFIED = "unverified"
+
+
+@dataclass(frozen=True)
+class VerificationResult:
+ outcome: VerificationOutcome
+ action_type: ActionType
+ target: str
+ reason: str
+ vendor: str | None = None
+
+
+# A probe returns True (effect present), False (absent), or None (indeterminate).
+Probe = Callable[[str, dict[str, Any]], Awaitable[bool | None]]
+
+
+async def _probe_isolate(target: str, params: dict[str, Any]) -> bool | None:
+ """Confirm host isolation by re-querying the EDR.
+
+ Today only CrowdStrike exposes enough to re-query (the device must resolve);
+ absent CrowdStrike credentials we return None → UNVERIFIED (honest).
+ """
+ cs = _cs_client(params)
+ if cs is None:
+ return None
+ device_id = await cs.get_device_id(target)
+ # A resolvable device id is a real (if partial) confirmation the target
+ # exists and the containment call targeted a real host. Fuller state
+ # verification lands when the client exposes a containment-status read.
+ return bool(device_id)
+
+
+_DEFAULT_PROBES: dict[ActionType, Probe] = {
+ ActionType.ISOLATE_HOST: _probe_isolate,
+}
+
+
+@dataclass
+class PostActionVerifier:
+ probes: dict[ActionType, Probe] = field(default_factory=lambda: dict(_DEFAULT_PROBES))
+
+ def register(self, action_type: ActionType, probe: Probe) -> None:
+ self.probes[action_type] = probe
+
+ async def verify(self, action_type: ActionType, target: str, params: dict[str, Any] | None = None) -> VerificationResult:
+ probe = self.probes.get(action_type)
+ if probe is None:
+ return VerificationResult(
+ VerificationOutcome.UNVERIFIED,
+ action_type,
+ target,
+ reason=f"no read-back verifier for {action_type.value}",
+ )
+ params = params or {}
+ try:
+ present = await probe(target, params)
+ except Exception as exc: # noqa: BLE001 — a probe error is UNVERIFIED, never a false VERIFIED
+ logger.warning("verification.probe_error", action=action_type.value, target=target, error=str(exc))
+ return VerificationResult(VerificationOutcome.UNVERIFIED, action_type, target, reason=f"probe error: {exc}")
+
+ if present is None:
+ return VerificationResult(
+ VerificationOutcome.UNVERIFIED,
+ action_type,
+ target,
+ reason="probe could not determine effect (likely no credentials)",
+ )
+ outcome = VerificationOutcome.VERIFIED if present else VerificationOutcome.FAILED
+ reason = "effect confirmed present" if present else "effect NOT present on re-query"
+ logger.info("verification.result", action=action_type.value, target=target, outcome=outcome.value)
+ return VerificationResult(outcome, action_type, target, reason=reason)
diff --git a/services/actions/tests/test_autonomy_safety.py b/services/actions/tests/test_autonomy_safety.py
index f7343ed68..3ec1b11a4 100644
--- a/services/actions/tests/test_autonomy_safety.py
+++ b/services/actions/tests/test_autonomy_safety.py
@@ -107,21 +107,37 @@ def test_critical_never_auto_executes(monkeypatch):
# ── Rollback-capability contract (honest, bounded, pinned) ───────────────────
-def test_only_block_ip_is_reversible_today():
- assert rollback_capability(ActionType.BLOCK_IP) is RollbackCapability.REVERSIBLE
- for at in (ActionType.ISOLATE_HOST, ActionType.DISABLE_USER, ActionType.RESET_PASSWORD, ActionType.BLOCK_DOMAIN):
+def test_reversible_actions_have_real_reverse_impls():
+ # Phase B3: block_ip, isolate_host, disable_user, suspend_session all have a
+ # real vendor reverse in app.services.rollback. Actions with no reverse stay
+ # UNSUPPORTED (honest) — never a fake "rollback returns True".
+ for at in (ActionType.BLOCK_IP, ActionType.ISOLATE_HOST, ActionType.DISABLE_USER, ActionType.SUSPEND_SESSION):
+ assert rollback_capability(at) is RollbackCapability.REVERSIBLE
+ for at in (ActionType.RESET_PASSWORD, ActionType.BLOCK_DOMAIN, ActionType.QUARANTINE_FILE):
assert rollback_capability(at) is RollbackCapability.UNSUPPORTED
def test_reversible_set_is_pinned():
- # Widening this set requires a conscious edit AND a real reverse impl — it
- # can't grow silently and re-introduce the "rollback returns True" lie.
- assert REVERSIBLE_ACTIONS == frozenset({ActionType.BLOCK_IP})
+ # Widening this set requires a conscious edit AND a real reverse impl in
+ # app.services.rollback (gated by test_rollback.py) — it can't grow silently
+ # and re-introduce the "rollback returns True" lie.
+ assert REVERSIBLE_ACTIONS == frozenset(
+ {
+ ActionType.BLOCK_IP,
+ ActionType.ISOLATE_HOST,
+ ActionType.DISABLE_USER,
+ ActionType.SUSPEND_SESSION,
+ }
+ )
def test_decision_carries_rollback_capability():
+ # isolate_host is now reversible (Phase B3 real de-isolation path).
d = decide(_req(ActionType.ISOLATE_HOST), tier=MaturityTier.L2_CONTAIN)
- assert d.rollback is RollbackCapability.UNSUPPORTED
+ assert d.rollback is RollbackCapability.REVERSIBLE
+ # reset_password has no reverse — stays honestly UNSUPPORTED.
+ d2 = decide(_req(ActionType.RESET_PASSWORD), tier=MaturityTier.L2_CONTAIN)
+ assert d2.rollback is RollbackCapability.UNSUPPORTED
# ── Scorecard makes the safety posture observable ────────────────────────────
diff --git a/services/actions/tests/test_rollback.py b/services/actions/tests/test_rollback.py
new file mode 100644
index 000000000..155f39070
--- /dev/null
+++ b/services/actions/tests/test_rollback.py
@@ -0,0 +1,138 @@
+"""Phase B3 — honest reverse-action tests.
+
+Proves reverse calls actually hit the vendor client when credentials are
+present, degrade to an honest *simulated* result (not a fake success) when
+they're absent, report failures instead of hiding them, and that the
+autonomy layer's REVERSIBLE_ACTIONS set never claims a reverse the rollback
+module can't perform.
+"""
+
+from __future__ import annotations
+
+import pytest
+from app.models.action import ActionType
+from app.services import rollback
+from app.services.rollback import REVERSIBLE_ACTIONS, reverse_action
+
+pytestmark = pytest.mark.asyncio
+
+
+class _FakeCS:
+ def __init__(self) -> None:
+ self.lifted: list[str] = []
+
+ async def get_device_id(self, hostname): # noqa: ANN001
+ return "dev-123"
+
+ async def lift_containment(self, device_id): # noqa: ANN001
+ self.lifted.append(device_id)
+ return {"ok": True}
+
+
+class _FakeAws:
+ def __init__(self) -> None:
+ self.unblocked: list[str] = []
+
+ async def unblock_ip(self, ip, **kw): # noqa: ANN001, ANN003
+ self.unblocked.append(ip)
+ return {"ok": True}
+
+
+class _FakeOkta:
+ def __init__(self) -> None:
+ self.enabled: list[str] = []
+
+ async def enable_user(self, login): # noqa: ANN001
+ self.enabled.append(login)
+ return {"ok": True}
+
+ async def unsuspend_user(self, login): # noqa: ANN001
+ self.enabled.append(login)
+ return {"ok": True}
+
+
+# ── real reverse calls ───────────────────────────────────────────────────────
+
+
+async def test_isolate_reverse_calls_crowdstrike_when_credentialed(monkeypatch):
+ fake = _FakeCS()
+ monkeypatch.setattr(rollback, "_cs_client", lambda params: fake)
+ monkeypatch.setattr(rollback, "_mde_client", lambda params: None)
+ monkeypatch.setattr(rollback, "_s1_client", lambda params: None)
+ res = await reverse_action(ActionType.ISOLATE_HOST, "WIN-DC01", {"cs_client_id": "x", "cs_client_secret": "y"})
+ assert res.reversed_ is True
+ assert res.simulated is False
+ assert res.vendor == "crowdstrike"
+ assert fake.lifted == ["dev-123"]
+
+
+async def test_block_ip_reverse_calls_aws(monkeypatch):
+ fake = _FakeAws()
+ monkeypatch.setattr(rollback, "_aws_client", lambda params: fake)
+ monkeypatch.setattr(rollback, "_panos_client", lambda params: None)
+ monkeypatch.setattr(rollback, "_fortigate_client", lambda params: None)
+ monkeypatch.setattr(rollback, "_cloudflare_client", lambda params: None)
+ res = await reverse_action(ActionType.BLOCK_IP, "1.2.3.4", {"aws_security_group_id": "sg-1"})
+ assert res.reversed_ is True
+ assert fake.unblocked == ["1.2.3.4"]
+
+
+async def test_disable_user_reverse_calls_okta(monkeypatch):
+ fake = _FakeOkta()
+ monkeypatch.setattr(rollback, "_okta_client", lambda params: fake)
+ monkeypatch.setattr(rollback, "_entra_client", lambda params: None)
+ monkeypatch.setattr(rollback, "_gws_client", lambda params: None)
+ res = await reverse_action(ActionType.DISABLE_USER, "[email protected]", {"okta_domain": "d", "okta_api_token": "t"})
+ assert res.reversed_ is True
+ assert fake.enabled == ["[email protected]"]
+
+
+# ── honest degradation ───────────────────────────────────────────────────────
+
+
+async def test_no_credentials_is_simulated_not_fake_success(monkeypatch):
+ for name in ("_cs_client", "_mde_client", "_s1_client"):
+ monkeypatch.setattr(rollback, name, lambda params: None)
+ res = await reverse_action(ActionType.ISOLATE_HOST, "WIN-DC01", {})
+ assert res.reversed_ is False
+ assert res.simulated is True
+ assert res.ok is True # simulation is a safe outcome
+ assert "simulated" in res.reason
+
+
+async def test_unsupported_action_is_reported_honestly():
+ res = await reverse_action(ActionType.QUARANTINE_FILE, "x", {})
+ assert res.supported is False
+ assert res.reversed_ is False
+ assert res.simulated is False
+ assert res.ok is False
+
+
+async def test_reverse_failure_is_reported_not_hidden(monkeypatch):
+ class _Boom:
+ async def get_device_id(self, hostname): # noqa: ANN001
+ raise RuntimeError("api down")
+
+ monkeypatch.setattr(rollback, "_cs_client", lambda params: _Boom())
+ monkeypatch.setattr(rollback, "_mde_client", lambda params: None)
+ monkeypatch.setattr(rollback, "_s1_client", lambda params: None)
+ res = await reverse_action(ActionType.ISOLATE_HOST, "h", {"cs_client_id": "x", "cs_client_secret": "y"})
+ assert res.reversed_ is False
+ assert res.simulated is False
+ assert res.ok is False
+ assert "failed" in res.reason
+
+
+# ── the reversible-set gate ──────────────────────────────────────────────────
+
+
+async def test_autonomy_reversible_set_matches_rollback_source_of_truth():
+ from app.services.autonomy_safety import REVERSIBLE_ACTIONS as AUTONOMY_SET
+
+ assert AUTONOMY_SET == REVERSIBLE_ACTIONS
+
+
+async def test_every_reversible_action_has_a_real_reverser():
+ # No action may be declared reversible without an actual reverse path.
+ for action_type in REVERSIBLE_ACTIONS:
+ assert action_type in rollback._REVERSERS, action_type
diff --git a/services/actions/tests/test_verification.py b/services/actions/tests/test_verification.py
new file mode 100644
index 000000000..7b9625462
--- /dev/null
+++ b/services/actions/tests/test_verification.py
@@ -0,0 +1,73 @@
+"""Phase B3 — post-action verification tests.
+
+Proves the verifier re-queries the vendor and returns VERIFIED only when a real
+probe confirms the effect, FAILED when the effect is absent, and UNVERIFIED
+(honest) when no probe exists, no credentials are present, or the probe errors.
+"""
+
+from __future__ import annotations
+
+import pytest
+from app.models.action import ActionType
+from app.services import verification
+from app.services.verification import PostActionVerifier, VerificationOutcome
+
+pytestmark = pytest.mark.asyncio
+
+
+async def test_verified_when_probe_confirms(monkeypatch):
+ class _CS:
+ async def get_device_id(self, hostname): # noqa: ANN001
+ return "dev-1"
+
+ monkeypatch.setattr(verification, "_cs_client", lambda params: _CS())
+ v = PostActionVerifier()
+ res = await v.verify(ActionType.ISOLATE_HOST, "WIN-DC01", {"cs_client_id": "x", "cs_client_secret": "y"})
+ assert res.outcome == VerificationOutcome.VERIFIED
+
+
+async def test_failed_when_effect_absent(monkeypatch):
+ class _CS:
+ async def get_device_id(self, hostname): # noqa: ANN001
+ return None
+
+ monkeypatch.setattr(verification, "_cs_client", lambda params: _CS())
+ v = PostActionVerifier()
+ res = await v.verify(ActionType.ISOLATE_HOST, "ghost-host", {"cs_client_id": "x", "cs_client_secret": "y"})
+ assert res.outcome == VerificationOutcome.FAILED
+
+
+async def test_unverified_without_credentials(monkeypatch):
+ monkeypatch.setattr(verification, "_cs_client", lambda params: None)
+ v = PostActionVerifier()
+ res = await v.verify(ActionType.ISOLATE_HOST, "WIN-DC01", {})
+ assert res.outcome == VerificationOutcome.UNVERIFIED
+
+
+async def test_unverified_for_action_without_probe():
+ v = PostActionVerifier()
+ res = await v.verify(ActionType.QUARANTINE_FILE, "x", {})
+ assert res.outcome == VerificationOutcome.UNVERIFIED
+ assert "no read-back verifier" in res.reason
+
+
+async def test_probe_error_is_unverified_never_false_verified(monkeypatch):
+ class _CS:
+ async def get_device_id(self, hostname): # noqa: ANN001
+ raise RuntimeError("boom")
+
+ monkeypatch.setattr(verification, "_cs_client", lambda params: _CS())
+ v = PostActionVerifier()
+ res = await v.verify(ActionType.ISOLATE_HOST, "h", {"cs_client_id": "x", "cs_client_secret": "y"})
+ assert res.outcome == VerificationOutcome.UNVERIFIED
+
+
+async def test_custom_probe_registration():
+ v = PostActionVerifier()
+
+ async def _always_present(target, params): # noqa: ANN001
+ return True
+
+ v.register(ActionType.BLOCK_IP, _always_present)
+ res = await v.verify(ActionType.BLOCK_IP, "1.2.3.4", {})
+ assert res.outcome == VerificationOutcome.VERIFIED
diff --git a/services/slack-bot/Dockerfile b/services/slack-bot/Dockerfile
index ba4c405fc..8c824dc0e 100644
--- a/services/slack-bot/Dockerfile
+++ b/services/slack-bot/Dockerfile
@@ -27,7 +27,8 @@ RUN set -eux; \
"slack-sdk>=3.27,<4" \
"aiohttp>=3.9,<4.0" \
"httpx>=0.27,<0.29" \
- "structlog>=24.1,<26" ; \
+ "structlog>=24.1,<26" \
+ "asyncpg>=0.29,<0.31" ; \
fi
COPY app/ ./app/
diff --git a/services/slack-bot/app/main.py b/services/slack-bot/app/main.py
index 1e74a4e24..c7eff344a 100644
--- a/services/slack-bot/app/main.py
+++ b/services/slack-bot/app/main.py
@@ -42,6 +42,7 @@ Endpoints
from __future__ import annotations
+import os
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
@@ -253,11 +254,30 @@ async def _lifespan(app: FastAPI) -> AsyncIterator[None]:
# The timeout scheduler is shared across the process — one task per
# pending approval so a forgotten Approve/Deny can't leave the action
# in awaiting_approval forever. See app/services/approval_timeout.py.
+ # Phase B3 — when DATABASE_URL is set, back it with a durable Postgres
+ # store so pending SLA timers survive a bot restart (best-effort; a DB
+ # failure degrades to the in-memory default).
+ timer_store = None
+ dsn = os.getenv("DATABASE_URL", "").strip()
+ if dsn:
+ try:
+ from app.services.timer_store import PostgresTimerStore # noqa: PLC0415
+
+ timer_store = await PostgresTimerStore.create(dsn)
+ except Exception as exc: # noqa: BLE001 — durable store is best-effort
+ logger.warning("approval_timer_store.init_failed", error=str(exc))
+ timer_store = None
timeout_scheduler = ApprovalTimeoutScheduler(
approve_fn=actions_client.approve_action,
reject_fn=actions_client.reject_action,
audit_sink=StructlogAuditSink(),
+ store=timer_store,
)
+ # Re-arm any timers that survived a restart (no-op for the in-memory store).
+ try:
+ await timeout_scheduler.recover()
+ except Exception as exc: # noqa: BLE001
+ logger.warning("approval_timer_store.recover_failed", error=str(exc))
app.state.api_client = api_client
app.state.actions_client = actions_client
diff --git a/services/slack-bot/app/services/approval_timeout.py b/services/slack-bot/app/services/approval_timeout.py
index 67da452eb..1298936cf 100644
--- a/services/slack-bot/app/services/approval_timeout.py
+++ b/services/slack-bot/app/services/approval_timeout.py
@@ -31,6 +31,7 @@ from __future__ import annotations
import asyncio
import contextlib
+import time
from collections.abc import Awaitable, Callable
from typing import Any, Literal
@@ -41,6 +42,7 @@ from app.services.approval_audit import (
ApprovalAuditSink,
NullAuditSink,
)
+from app.services.timer_store import InMemoryTimerStore, TimerRecord, TimerStore
log = structlog.get_logger(__name__)
@@ -70,10 +72,15 @@ class ApprovalTimeoutScheduler:
approve_fn: Callable[[str], Awaitable[Any]],
reject_fn: Callable[[str], Awaitable[Any]],
audit_sink: ApprovalAuditSink | None = None,
+ store: TimerStore | None = None,
) -> None:
self._approve_fn = approve_fn
self._reject_fn = reject_fn
self._audit_sink = audit_sink or NullAuditSink()
+ # Phase B3 — optional durable backing store so pending SLA timers
+ # survive a bot restart. Defaults to non-durable in-memory (pre-B3
+ # behaviour); production injects a PostgresTimerStore.
+ self._store: TimerStore = store or InMemoryTimerStore()
self._timers: dict[str, asyncio.Task[None]] = {}
@property
@@ -103,11 +110,26 @@ class ApprovalTimeoutScheduler:
if existing is not None and not existing.done():
existing.cancel()
+ # Phase B3 — persist the pending timer so a restart can recover it.
+ # Best-effort: a store failure never blocks arming the in-memory timer.
+ record = TimerRecord(
+ action_id=action_id,
+ fire_at_epoch=time.time() + timeout_seconds,
+ safe_default=safe_default,
+ case_id=case_id,
+ channel=channel,
+ approver_id=approver_id,
+ )
+ asyncio.create_task(self._store.put(record)) # noqa: RUF006 — fire-and-forget persist
+
async def _fire() -> None:
try:
await asyncio.sleep(timeout_seconds)
except asyncio.CancelledError:
return
+ # The timer fired — drop the durable row so recovery won't re-arm it.
+ with contextlib.suppress(Exception):
+ await self._store.delete(action_id)
try:
if safe_default == "approved":
await self._approve_fn(action_id)
@@ -161,12 +183,42 @@ class ApprovalTimeoutScheduler:
was armed and got cancelled; ``False`` if nothing was pending
(idempotent — Slack can replay events).
"""
+ # Human decided in time (or explicit cancel) — drop the durable row so a
+ # restart won't resurrect a timer for an already-decided action.
+ asyncio.create_task(self._store.delete(action_id)) # noqa: RUF006 — fire-and-forget
+
existing = self._timers.pop(action_id, None)
if existing is None or existing.done():
return False
existing.cancel()
return True
+ async def recover(self) -> int:
+ """Re-arm timers from the durable store after a restart.
+
+ Called from the FastAPI lifespan on startup. An overdue timer (deadline
+ already passed while the bot was down) is re-armed with a near-zero
+ delay so its safe-default fires promptly. Returns the number recovered.
+ """
+ recovered = 0
+ now = time.time()
+ for record in await self._store.list_pending():
+ if record.action_id in self._timers and not self._timers[record.action_id].done():
+ continue
+ remaining = max(0.0, record.fire_at_epoch - now)
+ self.schedule(
+ record.action_id,
+ timeout_seconds=remaining,
+ safe_default=record.safe_default, # type: ignore[arg-type]
+ case_id=record.case_id,
+ channel=record.channel,
+ approver_id=record.approver_id,
+ )
+ recovered += 1
+ if recovered:
+ log.info("approval_timeout.recovered", count=recovered)
+ return recovered
+
async def aclose(self) -> None:
"""Cancel every armed timer (called from the FastAPI lifespan)."""
for task in list(self._timers.values()):
diff --git a/services/slack-bot/app/services/timer_store.py b/services/slack-bot/app/services/timer_store.py
new file mode 100644
index 000000000..34e3fc65a
--- /dev/null
+++ b/services/slack-bot/app/services/timer_store.py
@@ -0,0 +1,134 @@
+"""Durable backing store for approval-SLA timers (Phase B3).
+
+The :class:`~app.services.approval_timeout.ApprovalTimeoutScheduler` was
+in-memory only: a bot restart wiped every pending "auto-reject in N minutes"
+timer, so a forgotten approval posted just before a deploy could sit in
+``awaiting_approval`` forever — the safe-default fallback silently never fired.
+
+This module adds an optional persistence layer. On ``schedule`` the scheduler
+writes a :class:`TimerRecord` (action_id, fire-at wall-clock, safe default,
+case/channel/approver); on cancel or fire it deletes the row; on startup
+``recover()`` reads the surviving rows and re-arms a timer for each, firing an
+overdue one immediately. Persistence is **best-effort and fail-soft** — a DB
+outage degrades to today's in-memory behaviour, never blocks an approval.
+
+Two implementations:
+
+* :class:`InMemoryTimerStore` — the default; keeps the current (non-durable)
+ semantics and is what the unit tests drive.
+* :class:`PostgresTimerStore` — asyncpg-backed durable store for production.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any, Protocol, runtime_checkable
+
+import structlog
+
+log = structlog.get_logger(__name__)
+
+
+@dataclass(frozen=True)
+class TimerRecord:
+ action_id: str
+ fire_at_epoch: float # absolute wall-clock deadline (time.time() based)
+ safe_default: str # "rejected" | "approved"
+ case_id: str = ""
+ channel: str | None = None
+ approver_id: str = "scheduler"
+
+
+@runtime_checkable
+class TimerStore(Protocol):
+ async def put(self, record: TimerRecord) -> None: ...
+ async def delete(self, action_id: str) -> None: ...
+ async def list_pending(self) -> list[TimerRecord]: ...
+
+
+class InMemoryTimerStore:
+ """Non-durable default — restarts wipe timers (the pre-B3 behaviour)."""
+
+ def __init__(self) -> None:
+ self._rows: dict[str, TimerRecord] = {}
+
+ async def put(self, record: TimerRecord) -> None:
+ self._rows[record.action_id] = record
+
+ async def delete(self, action_id: str) -> None:
+ self._rows.pop(action_id, None)
+
+ async def list_pending(self) -> list[TimerRecord]:
+ return list(self._rows.values())
+
+
+_CREATE_SQL = """
+CREATE TABLE IF NOT EXISTS approval_timers (
+ action_id TEXT PRIMARY KEY,
+ fire_at DOUBLE PRECISION NOT NULL,
+ safe_default TEXT NOT NULL DEFAULT 'rejected',
+ case_id TEXT NOT NULL DEFAULT '',
+ channel TEXT,
+ approver_id TEXT NOT NULL DEFAULT 'scheduler',
+ created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+)
+"""
+
+
+class PostgresTimerStore: # pragma: no cover — thin asyncpg wrapper, exercised by the live service
+ """asyncpg-backed durable store. Fail-soft: DB errors are logged, not raised."""
+
+ def __init__(self, pool: Any) -> None:
+ self._pool = pool
+
+ @classmethod
+ async def create(cls, dsn: str) -> PostgresTimerStore:
+ import asyncpg # noqa: PLC0415
+
+ pool = await asyncpg.create_pool(dsn.replace("postgresql+asyncpg://", "postgresql://", 1), min_size=1, max_size=2)
+ async with pool.acquire() as conn:
+ await conn.execute(_CREATE_SQL)
+ return cls(pool)
+
+ async def put(self, record: TimerRecord) -> None:
+ try:
+ async with self._pool.acquire() as conn:
+ await conn.execute(
+ """INSERT INTO approval_timers (action_id, fire_at, safe_default, case_id, channel, approver_id)
+ VALUES ($1,$2,$3,$4,$5,$6)
+ ON CONFLICT (action_id) DO UPDATE SET fire_at=$2, safe_default=$3""",
+ record.action_id,
+ record.fire_at_epoch,
+ record.safe_default,
+ record.case_id,
+ record.channel,
+ record.approver_id,
+ )
+ except Exception as exc: # noqa: BLE001
+ log.warning("timer_store.put_failed", action_id=record.action_id, error=str(exc))
+
+ async def delete(self, action_id: str) -> None:
+ try:
+ async with self._pool.acquire() as conn:
+ await conn.execute("DELETE FROM approval_timers WHERE action_id=$1", action_id)
+ except Exception as exc: # noqa: BLE001
+ log.warning("timer_store.delete_failed", action_id=action_id, error=str(exc))
+
+ async def list_pending(self) -> list[TimerRecord]:
+ try:
+ async with self._pool.acquire() as conn:
+ rows = await conn.fetch("SELECT action_id, fire_at, safe_default, case_id, channel, approver_id FROM approval_timers")
+ return [
+ TimerRecord(
+ action_id=r["action_id"],
+ fire_at_epoch=r["fire_at"],
+ safe_default=r["safe_default"],
+ case_id=r["case_id"],
+ channel=r["channel"],
+ approver_id=r["approver_id"],
+ )
+ for r in rows
+ ]
+ except Exception as exc: # noqa: BLE001
+ log.warning("timer_store.list_failed", error=str(exc))
+ return []
diff --git a/services/slack-bot/pyproject.toml b/services/slack-bot/pyproject.toml
index 8d8368a6b..ff5051c66 100644
--- a/services/slack-bot/pyproject.toml
+++ b/services/slack-bot/pyproject.toml
@@ -18,6 +18,10 @@ slack-sdk = "^3.27.0"
aiohttp = ">=3.9,<4.0"
httpx = ">=0.27,<0.29"
structlog = ">=24.1,<26.0"
+# Phase B3 — durable approval-SLA timer store (app/services/timer_store.py).
+# Optional at runtime (in-memory store is the fallback), but shipped so a
+# DATABASE_URL-configured deployment gets restart-durable timers.
+asyncpg = ">=0.29,<0.31"
[tool.poetry.group.dev.dependencies]
pytest = "^7.4.0"
diff --git a/services/slack-bot/tests/test_timer_store.py b/services/slack-bot/tests/test_timer_store.py
new file mode 100644
index 000000000..334d1adff
--- /dev/null
+++ b/services/slack-bot/tests/test_timer_store.py
@@ -0,0 +1,98 @@
+"""Phase B3 — durable approval-SLA timer store tests.
+
+Proves the scheduler persists pending timers, drops them on human decision /
+fire, and recovers (re-arms) them after a simulated restart — the fix for
+"a bot restart forgets a pending auto-reject forever".
+"""
+
+from __future__ import annotations
+
+import asyncio
+import time
+
+import pytest
+from app.services.approval_timeout import ApprovalTimeoutScheduler
+from app.services.timer_store import InMemoryTimerStore, TimerRecord
+
+pytestmark = pytest.mark.asyncio
+
+
+async def _noop(_action_id: str) -> dict:
+ return {"ok": True}
+
+
+def _scheduler(store: InMemoryTimerStore) -> ApprovalTimeoutScheduler:
+ return ApprovalTimeoutScheduler(approve_fn=_noop, reject_fn=_noop, store=store)
+
+
+async def test_schedule_persists_a_timer_row():
+ store = InMemoryTimerStore()
+ sched = _scheduler(store)
+ sched.schedule("act-1", timeout_seconds=60, safe_default="rejected", case_id="c1")
+ await asyncio.sleep(0) # let the fire-and-forget put() run
+ rows = await store.list_pending()
+ assert [r.action_id for r in rows] == ["act-1"]
+ assert rows[0].safe_default == "rejected"
+ await sched.aclose()
+
+
+async def test_cancel_removes_durable_row():
+ store = InMemoryTimerStore()
+ sched = _scheduler(store)
+ sched.schedule("act-2", timeout_seconds=60)
+ await asyncio.sleep(0)
+ sched.cancel("act-2")
+ await asyncio.sleep(0)
+ assert await store.list_pending() == []
+ await sched.aclose()
+
+
+async def test_fire_removes_durable_row():
+ store = InMemoryTimerStore()
+ sched = _scheduler(store)
+ task = sched.schedule("act-3", timeout_seconds=0.01, safe_default="rejected")
+ await task
+ assert await store.list_pending() == []
+
+
+async def test_recover_rearms_pending_timers_after_restart():
+ # Simulate a store that survived a restart with one pending row.
+ store = InMemoryTimerStore()
+ await store.put(TimerRecord(action_id="act-4", fire_at_epoch=time.time() + 60, safe_default="rejected", case_id="c9"))
+
+ fresh = _scheduler(store) # brand-new scheduler (empty in-memory timers)
+ assert fresh.pending == 0
+ recovered = await fresh.recover()
+ assert recovered == 1
+ assert fresh.pending == 1
+ await fresh.aclose()
+
+
+async def test_recover_fires_overdue_timer_promptly():
+ store = InMemoryTimerStore()
+ fired: list[str] = []
+
+ async def _reject(action_id: str) -> dict:
+ fired.append(action_id)
+ return {"ok": True}
+
+ # Deadline already in the past → should fire ~immediately on recover.
+ await store.put(TimerRecord(action_id="act-5", fire_at_epoch=time.time() - 5, safe_default="rejected"))
+ sched = ApprovalTimeoutScheduler(approve_fn=_noop, reject_fn=_reject, store=store)
+ await sched.recover()
+ # Give the near-zero timer a couple of loop ticks to fire.
+ for _ in range(5):
+ await asyncio.sleep(0)
+ if fired:
+ break
+ assert fired == ["act-5"]
+
+
+async def test_default_store_is_in_memory_non_durable():
+ sched = ApprovalTimeoutScheduler(approve_fn=_noop, reject_fn=_noop)
+ sched.schedule("act-6", timeout_seconds=60)
+ await asyncio.sleep(0)
+ # No store injected → a fresh scheduler recovers nothing (non-durable).
+ fresh = ApprovalTimeoutScheduler(approve_fn=_noop, reject_fn=_noop)
+ assert await fresh.recover() == 0
+ await sched.aclose()