Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
feat(phase-9a): autonomy-safety policy + honest rollback contract + scorecard (#434)
Beenu Arora97681fd8f381
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 14950edfa..bf899e9a9 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Added+- **Phase 9a — autonomy-safety policy + honest rollback contract + scorecard.** The reality audit found three holes behind "L0-L4 automation maturity gates every action": dry-run was opt-in (a mis-configured caller executes for real), ~15 executor `rollback()`s silently `return True` with no reverse vendor call (the platform claimed to reverse containment it never did), and there was no post-action verification. New `services/actions/app/services/autonomy_safety.py` closes them at the policy layer: `decide()` makes **dry-run the safe default** — anything not explicitly permitted to auto-execute is previewed (`DRY_RUN`), never silently executed; CRITICAL blast never auto-executes; HIGH only at L4 with a whitelist entry AND the `AISOC_ALLOW_HIGH_BLAST_AUTO` break-glass flag. `rollback_capability()` with a **pinned `REVERSIBLE_ACTIONS={block_ip}`** set makes the rollback claim honest and bounded — a caller learns "this cannot be auto-reversed" instead of a silent `True`, and the set can't grow without a conscious edit + real reverse implementation. Every unattended containment (`AUTO`, blast ≥ MEDIUM) sets `requires_verification`, and `AutonomyScorecard` counts executions that were never verified — and executions of irreversible actions — as **visible gaps** rather than assumed-away. 16 tests (`services/actions/tests/test_autonomy_safety.py`, in the actions coverage matrix). Enforcement-wiring of `decide()` into the live `/dispatch` + `submit_action` router, rewriting the silent-`return True` executor rollbacks, real vendor verifiers, and a durable approval-SLA timer table (replacing the in-memory `approval_timeout.py`) are tracked as 9b in `docs/audit/PROGRESS.md`.- **Phase 8 — LLMOps: prompt registry, model pins, response cache, structured-output validation.** A coherent `services/agents/app/llm/` LLMOps layer, all dependency-light and gated. (1) **Prompt registry** (`prompt_registry.py`) — every production prompt is a named, versioned, sha256-hashed artifact; the committed `prompts.lock.json` pins version→hash, and `scripts/check_prompt_lock.py --check` (wired into the CI lint job) fails when a prompt's text changes without a version bump + lock regeneration. This makes the `AGENTS.md` "prompt change ⇒ re-grade the eval harness" rule *enforceable* — you can no longer edit a prompt silently. (2) **Model pins + provider fallback** (`model_pins.py`) — logical roles pinned to concrete models with ordered fallback chains that ALWAYS terminate in the deterministic tier (env-overridable primaries, but the deterministic floor is gate-enforced), replacing the scattered `os.getenv("...", "gpt-4o-mini")` defaults. (3) **Content-addressed response cache** (`response_cache.py`) — keyed on sha256(model+prompt+input) with field separation and LRU eviction; safe under the determinism contract (a hit is byte-identical). (4) **Fail-closed structured-output validation** (`structured_output.py`) — strips code fences/prose, parses JSON, validates against a caller schema, and on ANY failure returns a deterministic fallback rather than propagating a half-parsed object into an autonomy decision. 15 tests (`services/agents/tests/test_llm_ops.py`, appended to the CI agents gate). Migration of the inline agent prompts to `registry.get()` is tracked as 8b. Doc: `apps/docs/docs/concepts/llmops.md`.- **Phase 7a — unified multi-model router with tier attribution + a determinism contract.** The reality audit confirmed the knowledge graph is already built at ingest (v8 T1.1, `services/ingest/internal/graph/`); the genuine Phase 7 delta was the *model router*. Before it, the deterministic→LLM fallback was reimplemented independently in NL query, playbook drafting, explain, copilot, and each sub-agent — no single audit point for which model answered or why. New `services/agents/app/routing/model_router.py` is that place: a `ModelRouter` that escalates deterministic → ML → LLM only when the cheaper tier is under-confident, and attributes every decision (`tier`, `model_used`, `attribution` trail, `tiers_considered`, `escalation_blocked_reason`). It **never silently uses the LLM** — a skipped or blocked LLM tier (no key, air-gap, deterministic mode, governor circuit open, or a tier error) is always recorded. Introduces the canonical **`AISOC_DETERMINISTIC`** flag and composes with the existing `CostGovernor` circuit breaker: either one forces deterministic-only, in which case the router is reproducible (same input → identical decision). 12 tests (`services/agents/tests/test_model_router.py`, appended to the CI agents gate) prove tier selection, attribution, the no-silent-LLM property, graceful LLM-failure degradation, and the determinism contract. Doc: `apps/docs/docs/concepts/model-router.md`. Remaining Phase 7 enrichments (posture collection, effective-permissions snapshot loader, bi-temporal validity, fusion-time ContextBundle) are tracked as 7b+ in `docs/audit/PROGRESS.md`.- **Phase 6 — performance + cost, both gated.** Two cheap, non-flaky gates in a new `.github/workflows/perf.yml`. (1) **Throughput** — `scripts/perf/throughput_harness.py` runs the *real* fusion hot path (`promote_normalized_event`) over a deterministic 20k-event synthetic corpus and reports events/sec + p50/p95/p99 per-event latency (measured ~250k eps on commodity hardware). The gate asserts a **generous regression floor** (1,000 eps — a >200× margin) so it fires only on a catastrophic regression such as an I/O call slipping onto the hot path, never on shared-runner jitter; it is explicitly a regression floor, not a production SLO. (2) **Cost** — `scripts/storage_cost_model.py` is a deterministic tiered-storage $/TB model (hot ClickHouse block 30d / warm object 60d / cold archive 275d, 8× ZSTD); the committed worked example (`docs/decisions/storage-cost-model.json`: ≈ $902/mo and ≈ $30/raw-TB at 1 TB/day) is drift-gated by `--check`, so the number can never silently diverge from the rate card. Rate-card values are labelled reference list prices to verify per provider/region — the value is the methodology, not the exact dollars. `docs/decisions/0005-storage-consolidation.md` (ADR) records the three-tier decision (keep ClickHouse-only hot; do not add a second hot engine) and cites the gated model.diff --git a/ROADMAP.md b/ROADMAP.mdindex b5c236709..cd89354f7 100644--- a/ROADMAP.md+++ b/ROADMAP.md@@ -31,7 +31,7 @@ A proof-first, security-first program to make every README claim gate-backed, cl- [x] Phase 6 — Performance + cost (fusion hot-path throughput harness with a generous regression-floor gate; deterministic storage $/TB cost model + drift gate; storage-consolidation ADR-0005)- [x] Phase 7 — Ingest-time graph + multi-model router — **graph-at-ingest already shipped** (v8 T1.1, `services/ingest/internal/graph/`); **7a landed**: unified deterministic→ML→LLM router with tier attribution + the `AISOC_DETERMINISTIC` determinism contract (`services/agents/app/routing/model_router.py`, gated). 7b+ (posture collection, effective-permissions snapshot loader, bi-temporal valid_from/valid_to, fusion-time ContextBundle) tracked in `docs/audit/PROGRESS.md`- [x] Phase 8 — LLMOps (version+hash-pinned prompt registry with a CI drift gate; model pins + deterministic-terminated provider-fallback chains; content-addressed response cache; fail-closed structured-output validation)-- [ ] Phase 9 — Autonomy safety (rollback, post-action verification, dry-run default, break-glass, approval SLA, scorecard)+- [x] Phase 9 — Autonomy safety (dry-run-by-default policy, honest bounded rollback-capability contract replacing silent `return True`, mandatory post-action verification for unattended containment, break-glass HIGH-blast flag, autonomy scorecard — all gated at the policy layer; live-router wiring + durable approval-SLA timer table tracked as 9b)- [ ] Phase 10 — Connector + content quality (conformance suite, published matrix, detection lifecycle)- [ ] Phase 11 — API / SDK / release engineering (oasdiff, SDK contract drift, versioning, release discipline)- [ ] Phase 12 — Observability + governance (SLOs, single trace, GOVERNANCE.md, DCO, committers, neutral home, trust docs)diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.mdindex 5b6426c7b..814cae6e0 100644--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md@@ -20,7 +20,7 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`| Detection-as-Code rejects candidates that regress MITRE accuracy | README L170 | `ci.yml :: python-test` (`test_detection_eval.py` — candidate `rule_body` run through the real engine vs its own positive/negative fixtures; approval requires it) + `p1-eval` w2-dac baseline | GATED | - || 800+ native detection rules | README L78, L170 | `validate-detections.yml` (strict fixture replay) | GATED | - || 6000+ imported detection rules | README L78 | `validate-detections.yml` (parse/provenance + `detection_truth_table.py --check`); README cites the executable figure (939) not the on-disk figure for coverage | GATED | - |-| L0-L4 automation maturity gates every action | README L171 | `ci.yml` autonomy-policy drift test | PARTIAL (thresholds gated; rollback/post-verify/dry-run ungated) | Phase 9 |+| L0-L4 automation maturity gates every action | README L171 | `ci.yml` autonomy-policy drift test + `test_autonomy_safety.py` (dry-run-default, rollback-capability contract, verification-mandate, scorecard) | PARTIAL (thresholds + dry-run-default + rollback-capability + verification-mandate gated at the policy layer; live-router wiring of `decide()` into `/dispatch` is 9b) | Phase 9b || Hunt-as-Code + `/hunt` | README L172 | `ci.yml :: p1-eval` (`hunt_corpus`) | GATED | - || Weekly benchmark scoreboard runs live against `main` | README L173 | `wet-eval.yml` (weekly) | NO GATE (no-ops without secret; live-agent tables are placeholders) | Phase 4 Tier 1 || MCP server exposes 13 tools | README L179 | `ci.yml :: mcp` | GATED | - |diff --git a/services/actions/app/services/autonomy_safety.py b/services/actions/app/services/autonomy_safety.pynew file mode 100644index 000000000..9702cf091--- /dev/null+++ b/services/actions/app/services/autonomy_safety.py@@ -0,0 +1,214 @@+"""Autonomy-safety policy + rollback-capability contract (Phase 9).++The reality audit found three holes behind the "L0-L4 automation maturity gates+every action" claim: (1) **dry-run is opt-in**, not the default, so a+mis-configured caller executes for real; (2) **~15 executors' `rollback()`+silently `return True`** without any reverse vendor call — the platform claims+to roll back containment it never reversed; (3) **there is no post-action+verification** that a containment actually took effect.++This module is the decision + contract layer that closes those holes honestly:++* :func:`decide` computes an :class:`AutonomyDecision` from the L0-L4 tier and+ blast radius. **Dry-run is the default**: anything not explicitly permitted to+ auto-execute is previewed (`DRY_RUN`), never silently executed. CRITICAL never+ auto-executes; HIGH only at L4 with a whitelist entry AND the+ ``AISOC_ALLOW_HIGH_BLAST_AUTO`` break-glass flag.+* :func:`rollback_capability` makes the rollback claim **honest and bounded**:+ only actions in :data:`REVERSIBLE_ACTIONS` (today, exactly ``block_ip``) may+ claim `REVERSIBLE`; everything else is `UNSUPPORTED`, so a caller learns "this+ cannot be auto-reversed" instead of a silent `True` lie. The gate pins the set+ so widening it requires a conscious edit (and a real reverse implementation).+* Every executed containment (`AUTO`, blast ≥ MEDIUM) sets+ ``requires_verification`` — post-action verification is mandated, and the+ :class:`AutonomyScorecard` counts executions that were never verified so the+ gap is visible rather than assumed-away.++Wiring these into the live REST router (`api/router.py`) is tracked as 9b; this+lands the policy + contract + gate so the semantics are enforced and honest.+"""++from __future__ import annotations++import os+from dataclasses import dataclass, field+from enum import Enum++from app.models.action import (+ ACTION_BLAST_RADIUS,+ APPROVAL_REQUIRED_ACTIONS,+ ActionRequest,+ ActionType,+ BlastRadius,+)+from app.services.maturity import _AUTO_ALLOWED_AT_TIER, MaturityTier++_BLAST_ORDER = {+ BlastRadius.MINIMAL: 0,+ BlastRadius.LOW: 1,+ BlastRadius.MEDIUM: 2,+ BlastRadius.HIGH: 3,+ BlastRadius.CRITICAL: 4,+}+++class AutonomyMode(str, Enum):+ BLOCKED = "blocked" # not permitted, not even a dry-run (e.g. L0 observe)+ DRY_RUN = "dry_run" # simulate only — no real effect (the SAFE default)+ QUEUED_APPROVAL = "queued_approval" # needs a human to execute+ AUTO = "auto" # permitted to execute unattended+++class RollbackCapability(str, Enum):+ REVERSIBLE = "reversible" # a real reverse vendor call exists+ UNSUPPORTED = "unsupported" # no automatic rollback — say so, don't fake it+++# The ONLY actions with a real reverse implementation today. `block_ip` reverses+# via the AWS security-group `unblock_ip` call. Widening this set requires+# implementing (and testing) a real reverse path — the gate pins it so it can't+# grow silently to re-introduce the "rollback returns True" lie.+REVERSIBLE_ACTIONS: frozenset[ActionType] = frozenset({ActionType.BLOCK_IP})+++class VerificationOutcome(str, Enum):+ VERIFIED = "verified" # re-queried the vendor; the effect is present+ UNVERIFIED = "unverified" # no verifier ran (the honest default today)+ FAILED = "failed" # re-queried; the effect is NOT present+++@dataclass(frozen=True)+class AutonomyDecision:+ mode: AutonomyMode+ blast_radius: BlastRadius+ tier: MaturityTier+ rollback: RollbackCapability+ requires_verification: bool+ reason: str++ @property+ def will_execute_for_real(self) -> bool:+ return self.mode is AutonomyMode.AUTO+++def rollback_capability(action_type: ActionType) -> RollbackCapability:+ return RollbackCapability.REVERSIBLE if action_type in REVERSIBLE_ACTIONS else RollbackCapability.UNSUPPORTED+++def default_dry_run() -> bool:+ """Dry-run-by-default unless an operator explicitly opts out."""+ raw = os.environ.get("AISOC_ACTIONS_DEFAULT_DRY_RUN")+ if raw is None:+ return True+ return raw.strip().lower() not in {"0", "false", "no", "off"}+++def allow_high_blast_auto() -> bool:+ """Break-glass: permit unattended HIGH-blast auto-execution. Default off."""+ raw = os.environ.get("AISOC_ALLOW_HIGH_BLAST_AUTO")+ if raw is None:+ return False+ return raw.strip().lower() in {"1", "true", "yes", "on"}+++def decide(+ request: ActionRequest,+ *,+ tier: MaturityTier,+ dry_run_default: bool | None = None,+ high_blast_auto: bool | None = None,+ whitelisted: bool = False,+) -> AutonomyDecision:+ """Compute the autonomy decision. Dry-run is the safe default."""+ if dry_run_default is None:+ dry_run_default = default_dry_run()+ if high_blast_auto is None:+ high_blast_auto = allow_high_blast_auto()++ blast = ACTION_BLAST_RADIUS.get(request.action_type, BlastRadius.HIGH)+ rb = rollback_capability(request.action_type)++ def _decision(mode: AutonomyMode, reason: str) -> AutonomyDecision:+ requires_verification = mode is AutonomyMode.AUTO and _BLAST_ORDER[blast] >= _BLAST_ORDER[BlastRadius.MEDIUM]+ return AutonomyDecision(+ mode=mode,+ blast_radius=blast,+ tier=tier,+ rollback=rb,+ requires_verification=requires_verification,+ reason=reason,+ )++ # L0 is observe-only: nothing executes, not even a dry-run preview.+ if tier == MaturityTier.L0_OBSERVE:+ return _decision(AutonomyMode.BLOCKED, "tier L0 (observe-only) blocks all execution")++ # CRITICAL blast never auto-executes at any tier.+ if blast == BlastRadius.CRITICAL:+ return _decision(AutonomyMode.QUEUED_APPROVAL, "CRITICAL blast radius always requires human approval")++ allowed = _AUTO_ALLOWED_AT_TIER.get(tier, set())+ if blast in allowed:+ # HIGH is the sensitive top of the ladder: L4 + whitelist + break-glass.+ if blast == BlastRadius.HIGH:+ if tier == MaturityTier.L4_AUTOMATE and whitelisted and high_blast_auto:+ return _decision(AutonomyMode.AUTO, "HIGH auto-executed: L4 + whitelisted + break-glass enabled")+ reasons = []+ if tier != MaturityTier.L4_AUTOMATE:+ reasons.append("not L4")+ if not whitelisted:+ reasons.append("not whitelisted")+ if not high_blast_auto:+ reasons.append("AISOC_ALLOW_HIGH_BLAST_AUTO off")+ return _decision(AutonomyMode.QUEUED_APPROVAL, f"HIGH blast requires approval ({', '.join(reasons)})")+ # Approval-required action types always gate on a human, even if the+ # tier would otherwise permit the blast radius.+ if request.action_type in APPROVAL_REQUIRED_ACTIONS:+ return _decision(AutonomyMode.QUEUED_APPROVAL, "action type always requires human approval")+ return _decision(AutonomyMode.AUTO, f"{blast.value} permitted to auto-execute at tier {tier.name}")++ # Not permitted to auto-execute at this tier → SAFE DEFAULT is a dry-run+ # preview (never a silent real execution), unless dry-run is disabled, in+ # which case it queues for approval.+ if dry_run_default:+ return _decision(AutonomyMode.DRY_RUN, f"{blast.value} above tier {tier.name} allowance → dry-run preview (safe default)")+ return _decision(AutonomyMode.QUEUED_APPROVAL, f"{blast.value} above tier {tier.name} allowance → queued for approval")+++@dataclass+class AutonomyScorecard:+ """Aggregates autonomy decisions + outcomes so the safety posture is+ observable: how many executed unattended, how many of those were verified,+ and how many touched irreversible actions."""++ total: int = 0+ by_mode: dict[str, int] = field(default_factory=dict)+ executed: int = 0+ executed_verified: int = 0+ executed_unverified: int = 0+ irreversible_executed: int = 0++ def record(self, decision: AutonomyDecision, *, verification: VerificationOutcome | None = None) -> None:+ self.total += 1+ self.by_mode[decision.mode.value] = self.by_mode.get(decision.mode.value, 0) + 1+ if decision.will_execute_for_real:+ self.executed += 1+ if verification is VerificationOutcome.VERIFIED:+ self.executed_verified += 1+ else:+ # Never verified (or verification failed) → count as a gap.+ self.executed_unverified += 1+ if decision.rollback is RollbackCapability.UNSUPPORTED:+ self.irreversible_executed += 1++ def summary(self) -> dict[str, object]:+ verified_rate = (self.executed_verified / self.executed) if self.executed else 1.0+ return {+ "total": self.total,+ "by_mode": dict(self.by_mode),+ "executed": self.executed,+ "executed_verified": self.executed_verified,+ "executed_unverified": self.executed_unverified,+ "irreversible_executed": self.irreversible_executed,+ "verified_rate": round(verified_rate, 4),+ }diff --git a/services/actions/tests/test_autonomy_safety.py b/services/actions/tests/test_autonomy_safety.pynew file mode 100644index 000000000..f7343ed68--- /dev/null+++ b/services/actions/tests/test_autonomy_safety.py@@ -0,0 +1,157 @@+"""Phase 9 — autonomy-safety policy + rollback-capability contract + scorecard.++Proves the three holes the reality audit flagged are closed at the policy level:+dry-run is the SAFE DEFAULT (never a silent real execution), the rollback claim+is honest and bounded (no silent `return True`), and every unattended+containment mandates post-action verification (with the gap counted, not hidden).+"""++from __future__ import annotations++from uuid import uuid4++from app.models.action import ActionRequest, ActionType, BlastRadius+from app.services.autonomy_safety import (+ REVERSIBLE_ACTIONS,+ AutonomyDecision,+ AutonomyMode,+ AutonomyScorecard,+ RollbackCapability,+ VerificationOutcome,+ decide,+ rollback_capability,+)+from app.services.maturity import MaturityTier+++def _req(action_type: ActionType) -> ActionRequest:+ return ActionRequest(incident_id=uuid4(), tenant_id=uuid4(), action_type=action_type, target="host-1")+++# ── Dry-run is the safe default ──────────────────────────────────────────────+++def test_over_tier_blast_defaults_to_dry_run_not_execution():+ # BLOCK_IP is MEDIUM; L2 permits only MINIMAL/LOW → must NOT auto-execute.+ d = decide(_req(ActionType.BLOCK_IP), tier=MaturityTier.L2_CONTAIN, dry_run_default=True)+ assert d.mode is AutonomyMode.DRY_RUN+ assert d.will_execute_for_real is False+++def test_dry_run_disabled_queues_for_approval_instead():+ d = decide(_req(ActionType.BLOCK_IP), tier=MaturityTier.L2_CONTAIN, dry_run_default=False)+ assert d.mode is AutonomyMode.QUEUED_APPROVAL+++def test_l0_blocks_everything():+ d = decide(_req(ActionType.NOTIFY_SLACK), tier=MaturityTier.L0_OBSERVE)+ assert d.mode is AutonomyMode.BLOCKED+++# ── Tiered auto-execution ────────────────────────────────────────────────────+++def test_minimal_auto_executes_at_l1_without_verification():+ d = decide(_req(ActionType.NOTIFY_SLACK), tier=MaturityTier.L1_NOTIFY)+ assert d.mode is AutonomyMode.AUTO+ assert d.requires_verification is False # MINIMAL < MEDIUM+++def test_medium_auto_executes_at_l3_and_requires_verification():+ d = decide(_req(ActionType.BLOCK_IP), tier=MaturityTier.L3_REMEDIATE)+ assert d.mode is AutonomyMode.AUTO+ assert d.requires_verification is True # any executed containment must be verified+++def test_medium_approval_required_action_queues_even_when_tier_permits():+ # RESET_PASSWORD is MEDIUM + approval-required.+ d = decide(_req(ActionType.RESET_PASSWORD), tier=MaturityTier.L3_REMEDIATE)+ assert d.mode is AutonomyMode.QUEUED_APPROVAL+++# ── HIGH blast: L4 + whitelist + break-glass ─────────────────────────────────+++def test_high_queues_without_break_glass():+ d = decide(_req(ActionType.ISOLATE_HOST), tier=MaturityTier.L4_AUTOMATE, whitelisted=True, high_blast_auto=False)+ assert d.mode is AutonomyMode.QUEUED_APPROVAL+ assert "AISOC_ALLOW_HIGH_BLAST_AUTO off" in d.reason+++def test_high_queues_without_whitelist():+ d = decide(_req(ActionType.ISOLATE_HOST), tier=MaturityTier.L4_AUTOMATE, whitelisted=False, high_blast_auto=True)+ assert d.mode is AutonomyMode.QUEUED_APPROVAL+ assert "not whitelisted" in d.reason+++def test_high_auto_only_at_l4_whitelisted_with_break_glass():+ d = decide(_req(ActionType.ISOLATE_HOST), tier=MaturityTier.L4_AUTOMATE, whitelisted=True, high_blast_auto=True)+ assert d.mode is AutonomyMode.AUTO+ assert d.requires_verification is True+++def test_high_never_auto_below_l4():+ d = decide(_req(ActionType.ISOLATE_HOST), tier=MaturityTier.L3_REMEDIATE, whitelisted=True, high_blast_auto=True)+ assert d.mode is not AutonomyMode.AUTO+++def test_critical_never_auto_executes(monkeypatch):+ from app.models import action as action_model++ monkeypatch.setitem(action_model.ACTION_BLAST_RADIUS, ActionType.BLOCK_IP, BlastRadius.CRITICAL)+ d = decide(_req(ActionType.BLOCK_IP), tier=MaturityTier.L4_AUTOMATE, whitelisted=True, high_blast_auto=True)+ assert d.mode is AutonomyMode.QUEUED_APPROVAL+ assert "CRITICAL" in d.reason+++# ── Rollback-capability contract (honest, bounded, pinned) ───────────────────+++def test_only_block_ip_is_reversible_today():+ assert rollback_capability(ActionType.BLOCK_IP) is RollbackCapability.REVERSIBLE+ for at in (ActionType.ISOLATE_HOST, ActionType.DISABLE_USER, ActionType.RESET_PASSWORD, ActionType.BLOCK_DOMAIN):+ assert rollback_capability(at) is RollbackCapability.UNSUPPORTED+++def test_reversible_set_is_pinned():+ # Widening this set requires a conscious edit AND a real reverse impl — it+ # can't grow silently and re-introduce the "rollback returns True" lie.+ assert REVERSIBLE_ACTIONS == frozenset({ActionType.BLOCK_IP})+++def test_decision_carries_rollback_capability():+ d = decide(_req(ActionType.ISOLATE_HOST), tier=MaturityTier.L2_CONTAIN)+ assert d.rollback is RollbackCapability.UNSUPPORTED+++# ── Scorecard makes the safety posture observable ────────────────────────────+++def test_scorecard_counts_unverified_executions_as_a_gap():+ card = AutonomyScorecard()+ auto = decide(_req(ActionType.BLOCK_IP), tier=MaturityTier.L3_REMEDIATE)+ dry = decide(_req(ActionType.ISOLATE_HOST), tier=MaturityTier.L2_CONTAIN) # DRY_RUN+ card.record(auto) # executed, no verification supplied → gap+ card.record(dry) # not executed+ card.record(auto, verification=VerificationOutcome.VERIFIED) # executed + verified+ s = card.summary()+ assert s["total"] == 3+ assert s["executed"] == 2+ assert s["executed_unverified"] == 1+ assert s["executed_verified"] == 1+ assert s["verified_rate"] == 0.5+++def test_scorecard_tracks_irreversible_executions():+ card = AutonomyScorecard()+ # An AUTO execution of an UNSUPPORTED-rollback action is the risky case.+ d = AutonomyDecision(+ mode=AutonomyMode.AUTO,+ blast_radius=BlastRadius.MEDIUM,+ tier=MaturityTier.L4_AUTOMATE,+ rollback=RollbackCapability.UNSUPPORTED,+ requires_verification=True,+ reason="test",+ )+ card.record(d)+ assert card.summary()["irreversible_executed"] == 1