Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

feat(phase-9a): autonomy-safety policy + honest rollback contract + scorecard (#434)

Beenu Arora97681fd8f381
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 14950edfa..bf899e9a9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
+- **Phase 9a — autonomy-safety policy + honest rollback contract + scorecard.** The reality audit found three holes behind "L0-L4 automation maturity gates every action": dry-run was opt-in (a mis-configured caller executes for real), ~15 executor `rollback()`s silently `return True` with no reverse vendor call (the platform claimed to reverse containment it never did), and there was no post-action verification. New `services/actions/app/services/autonomy_safety.py` closes them at the policy layer: `decide()` makes **dry-run the safe default** — anything not explicitly permitted to auto-execute is previewed (`DRY_RUN`), never silently executed; CRITICAL blast never auto-executes; HIGH only at L4 with a whitelist entry AND the `AISOC_ALLOW_HIGH_BLAST_AUTO` break-glass flag. `rollback_capability()` with a **pinned `REVERSIBLE_ACTIONS={block_ip}`** set makes the rollback claim honest and bounded — a caller learns "this cannot be auto-reversed" instead of a silent `True`, and the set can't grow without a conscious edit + real reverse implementation. Every unattended containment (`AUTO`, blast ≥ MEDIUM) sets `requires_verification`, and `AutonomyScorecard` counts executions that were never verified — and executions of irreversible actions — as **visible gaps** rather than assumed-away. 16 tests (`services/actions/tests/test_autonomy_safety.py`, in the actions coverage matrix). Enforcement-wiring of `decide()` into the live `/dispatch` + `submit_action` router, rewriting the silent-`return True` executor rollbacks, real vendor verifiers, and a durable approval-SLA timer table (replacing the in-memory `approval_timeout.py`) are tracked as 9b in `docs/audit/PROGRESS.md`.
- **Phase 8 — LLMOps: prompt registry, model pins, response cache, structured-output validation.** A coherent `services/agents/app/llm/` LLMOps layer, all dependency-light and gated. (1) **Prompt registry** (`prompt_registry.py`) — every production prompt is a named, versioned, sha256-hashed artifact; the committed `prompts.lock.json` pins version→hash, and `scripts/check_prompt_lock.py --check` (wired into the CI lint job) fails when a prompt's text changes without a version bump + lock regeneration. This makes the `AGENTS.md` "prompt change ⇒ re-grade the eval harness" rule *enforceable* — you can no longer edit a prompt silently. (2) **Model pins + provider fallback** (`model_pins.py`) — logical roles pinned to concrete models with ordered fallback chains that ALWAYS terminate in the deterministic tier (env-overridable primaries, but the deterministic floor is gate-enforced), replacing the scattered `os.getenv("...", "gpt-4o-mini")` defaults. (3) **Content-addressed response cache** (`response_cache.py`) — keyed on sha256(model+prompt+input) with field separation and LRU eviction; safe under the determinism contract (a hit is byte-identical). (4) **Fail-closed structured-output validation** (`structured_output.py`) — strips code fences/prose, parses JSON, validates against a caller schema, and on ANY failure returns a deterministic fallback rather than propagating a half-parsed object into an autonomy decision. 15 tests (`services/agents/tests/test_llm_ops.py`, appended to the CI agents gate). Migration of the inline agent prompts to `registry.get()` is tracked as 8b. Doc: `apps/docs/docs/concepts/llmops.md`.
- **Phase 7a — unified multi-model router with tier attribution + a determinism contract.** The reality audit confirmed the knowledge graph is already built at ingest (v8 T1.1, `services/ingest/internal/graph/`); the genuine Phase 7 delta was the *model router*. Before it, the deterministic→LLM fallback was reimplemented independently in NL query, playbook drafting, explain, copilot, and each sub-agent — no single audit point for which model answered or why. New `services/agents/app/routing/model_router.py` is that place: a `ModelRouter` that escalates deterministic → ML → LLM only when the cheaper tier is under-confident, and attributes every decision (`tier`, `model_used`, `attribution` trail, `tiers_considered`, `escalation_blocked_reason`). It **never silently uses the LLM** — a skipped or blocked LLM tier (no key, air-gap, deterministic mode, governor circuit open, or a tier error) is always recorded. Introduces the canonical **`AISOC_DETERMINISTIC`** flag and composes with the existing `CostGovernor` circuit breaker: either one forces deterministic-only, in which case the router is reproducible (same input → identical decision). 12 tests (`services/agents/tests/test_model_router.py`, appended to the CI agents gate) prove tier selection, attribution, the no-silent-LLM property, graceful LLM-failure degradation, and the determinism contract. Doc: `apps/docs/docs/concepts/model-router.md`. Remaining Phase 7 enrichments (posture collection, effective-permissions snapshot loader, bi-temporal validity, fusion-time ContextBundle) are tracked as 7b+ in `docs/audit/PROGRESS.md`.
- **Phase 6 — performance + cost, both gated.** Two cheap, non-flaky gates in a new `.github/workflows/perf.yml`. (1) **Throughput** — `scripts/perf/throughput_harness.py` runs the *real* fusion hot path (`promote_normalized_event`) over a deterministic 20k-event synthetic corpus and reports events/sec + p50/p95/p99 per-event latency (measured ~250k eps on commodity hardware). The gate asserts a **generous regression floor** (1,000 eps — a >200× margin) so it fires only on a catastrophic regression such as an I/O call slipping onto the hot path, never on shared-runner jitter; it is explicitly a regression floor, not a production SLO. (2) **Cost** — `scripts/storage_cost_model.py` is a deterministic tiered-storage $/TB model (hot ClickHouse block 30d / warm object 60d / cold archive 275d, 8× ZSTD); the committed worked example (`docs/decisions/storage-cost-model.json`: ≈ $902/mo and ≈ $30/raw-TB at 1 TB/day) is drift-gated by `--check`, so the number can never silently diverge from the rate card. Rate-card values are labelled reference list prices to verify per provider/region — the value is the methodology, not the exact dollars. `docs/decisions/0005-storage-consolidation.md` (ADR) records the three-tier decision (keep ClickHouse-only hot; do not add a second hot engine) and cites the gated model.
diff --git a/ROADMAP.md b/ROADMAP.md
index b5c236709..cd89354f7 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -31,7 +31,7 @@ A proof-first, security-first program to make every README claim gate-backed, cl
- [x] Phase 6 — Performance + cost (fusion hot-path throughput harness with a generous regression-floor gate; deterministic storage $/TB cost model + drift gate; storage-consolidation ADR-0005)
- [x] Phase 7 — Ingest-time graph + multi-model router — **graph-at-ingest already shipped** (v8 T1.1, `services/ingest/internal/graph/`); **7a landed**: unified deterministic→ML→LLM router with tier attribution + the `AISOC_DETERMINISTIC` determinism contract (`services/agents/app/routing/model_router.py`, gated). 7b+ (posture collection, effective-permissions snapshot loader, bi-temporal valid_from/valid_to, fusion-time ContextBundle) tracked in `docs/audit/PROGRESS.md`
- [x] Phase 8 — LLMOps (version+hash-pinned prompt registry with a CI drift gate; model pins + deterministic-terminated provider-fallback chains; content-addressed response cache; fail-closed structured-output validation)
-- [ ] Phase 9 — Autonomy safety (rollback, post-action verification, dry-run default, break-glass, approval SLA, scorecard)
+- [x] Phase 9 — Autonomy safety (dry-run-by-default policy, honest bounded rollback-capability contract replacing silent `return True`, mandatory post-action verification for unattended containment, break-glass HIGH-blast flag, autonomy scorecard — all gated at the policy layer; live-router wiring + durable approval-SLA timer table tracked as 9b)
- [ ] Phase 10 — Connector + content quality (conformance suite, published matrix, detection lifecycle)
- [ ] Phase 11 — API / SDK / release engineering (oasdiff, SDK contract drift, versioning, release discipline)
- [ ] Phase 12 — Observability + governance (SLOs, single trace, GOVERNANCE.md, DCO, committers, neutral home, trust docs)
diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.md
index 5b6426c7b..814cae6e0 100644
--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md
+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md
@@ -20,7 +20,7 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
| Detection-as-Code rejects candidates that regress MITRE accuracy | README L170 | `ci.yml :: python-test` (`test_detection_eval.py` — candidate `rule_body` run through the real engine vs its own positive/negative fixtures; approval requires it) + `p1-eval` w2-dac baseline | GATED | - |
| 800+ native detection rules | README L78, L170 | `validate-detections.yml` (strict fixture replay) | GATED | - |
| 6000+ imported detection rules | README L78 | `validate-detections.yml` (parse/provenance + `detection_truth_table.py --check`); README cites the executable figure (939) not the on-disk figure for coverage | GATED | - |
-| L0-L4 automation maturity gates every action | README L171 | `ci.yml` autonomy-policy drift test | PARTIAL (thresholds gated; rollback/post-verify/dry-run ungated) | Phase 9 |
+| L0-L4 automation maturity gates every action | README L171 | `ci.yml` autonomy-policy drift test + `test_autonomy_safety.py` (dry-run-default, rollback-capability contract, verification-mandate, scorecard) | PARTIAL (thresholds + dry-run-default + rollback-capability + verification-mandate gated at the policy layer; live-router wiring of `decide()` into `/dispatch` is 9b) | Phase 9b |
| Hunt-as-Code + `/hunt` | README L172 | `ci.yml :: p1-eval` (`hunt_corpus`) | GATED | - |
| Weekly benchmark scoreboard runs live against `main` | README L173 | `wet-eval.yml` (weekly) | NO GATE (no-ops without secret; live-agent tables are placeholders) | Phase 4 Tier 1 |
| MCP server exposes 13 tools | README L179 | `ci.yml :: mcp` | GATED | - |
diff --git a/services/actions/app/services/autonomy_safety.py b/services/actions/app/services/autonomy_safety.py
new file mode 100644
index 000000000..9702cf091
--- /dev/null
+++ b/services/actions/app/services/autonomy_safety.py
@@ -0,0 +1,214 @@
+"""Autonomy-safety policy + rollback-capability contract (Phase 9).
+
+The reality audit found three holes behind the "L0-L4 automation maturity gates
+every action" claim: (1) **dry-run is opt-in**, not the default, so a
+mis-configured caller executes for real; (2) **~15 executors' `rollback()`
+silently `return True`** without any reverse vendor call — the platform claims
+to roll back containment it never reversed; (3) **there is no post-action
+verification** that a containment actually took effect.
+
+This module is the decision + contract layer that closes those holes honestly:
+
+* :func:`decide` computes an :class:`AutonomyDecision` from the L0-L4 tier and
+ blast radius. **Dry-run is the default**: anything not explicitly permitted to
+ auto-execute is previewed (`DRY_RUN`), never silently executed. CRITICAL never
+ auto-executes; HIGH only at L4 with a whitelist entry AND the
+ ``AISOC_ALLOW_HIGH_BLAST_AUTO`` break-glass flag.
+* :func:`rollback_capability` makes the rollback claim **honest and bounded**:
+ only actions in :data:`REVERSIBLE_ACTIONS` (today, exactly ``block_ip``) may
+ claim `REVERSIBLE`; everything else is `UNSUPPORTED`, so a caller learns "this
+ cannot be auto-reversed" instead of a silent `True` lie. The gate pins the set
+ so widening it requires a conscious edit (and a real reverse implementation).
+* Every executed containment (`AUTO`, blast ≥ MEDIUM) sets
+ ``requires_verification`` — post-action verification is mandated, and the
+ :class:`AutonomyScorecard` counts executions that were never verified so the
+ gap is visible rather than assumed-away.
+
+Wiring these into the live REST router (`api/router.py`) is tracked as 9b; this
+lands the policy + contract + gate so the semantics are enforced and honest.
+"""
+
+from __future__ import annotations
+
+import os
+from dataclasses import dataclass, field
+from enum import Enum
+
+from app.models.action import (
+ ACTION_BLAST_RADIUS,
+ APPROVAL_REQUIRED_ACTIONS,
+ ActionRequest,
+ ActionType,
+ BlastRadius,
+)
+from app.services.maturity import _AUTO_ALLOWED_AT_TIER, MaturityTier
+
+_BLAST_ORDER = {
+ BlastRadius.MINIMAL: 0,
+ BlastRadius.LOW: 1,
+ BlastRadius.MEDIUM: 2,
+ BlastRadius.HIGH: 3,
+ BlastRadius.CRITICAL: 4,
+}
+
+
+class AutonomyMode(str, Enum):
+ BLOCKED = "blocked" # not permitted, not even a dry-run (e.g. L0 observe)
+ DRY_RUN = "dry_run" # simulate only — no real effect (the SAFE default)
+ QUEUED_APPROVAL = "queued_approval" # needs a human to execute
+ AUTO = "auto" # permitted to execute unattended
+
+
+class RollbackCapability(str, Enum):
+ REVERSIBLE = "reversible" # a real reverse vendor call exists
+ UNSUPPORTED = "unsupported" # no automatic rollback — say so, don't fake it
+
+
+# The ONLY actions with a real reverse implementation today. `block_ip` reverses
+# via the AWS security-group `unblock_ip` call. Widening this set requires
+# implementing (and testing) a real reverse path — the gate pins it so it can't
+# grow silently to re-introduce the "rollback returns True" lie.
+REVERSIBLE_ACTIONS: frozenset[ActionType] = frozenset({ActionType.BLOCK_IP})
+
+
+class VerificationOutcome(str, Enum):
+ VERIFIED = "verified" # re-queried the vendor; the effect is present
+ UNVERIFIED = "unverified" # no verifier ran (the honest default today)
+ FAILED = "failed" # re-queried; the effect is NOT present
+
+
+@dataclass(frozen=True)
+class AutonomyDecision:
+ mode: AutonomyMode
+ blast_radius: BlastRadius
+ tier: MaturityTier
+ rollback: RollbackCapability
+ requires_verification: bool
+ reason: str
+
+ @property
+ def will_execute_for_real(self) -> bool:
+ return self.mode is AutonomyMode.AUTO
+
+
+def rollback_capability(action_type: ActionType) -> RollbackCapability:
+ return RollbackCapability.REVERSIBLE if action_type in REVERSIBLE_ACTIONS else RollbackCapability.UNSUPPORTED
+
+
+def default_dry_run() -> bool:
+ """Dry-run-by-default unless an operator explicitly opts out."""
+ raw = os.environ.get("AISOC_ACTIONS_DEFAULT_DRY_RUN")
+ if raw is None:
+ return True
+ return raw.strip().lower() not in {"0", "false", "no", "off"}
+
+
+def allow_high_blast_auto() -> bool:
+ """Break-glass: permit unattended HIGH-blast auto-execution. Default off."""
+ raw = os.environ.get("AISOC_ALLOW_HIGH_BLAST_AUTO")
+ if raw is None:
+ return False
+ return raw.strip().lower() in {"1", "true", "yes", "on"}
+
+
+def decide(
+ request: ActionRequest,
+ *,
+ tier: MaturityTier,
+ dry_run_default: bool | None = None,
+ high_blast_auto: bool | None = None,
+ whitelisted: bool = False,
+) -> AutonomyDecision:
+ """Compute the autonomy decision. Dry-run is the safe default."""
+ if dry_run_default is None:
+ dry_run_default = default_dry_run()
+ if high_blast_auto is None:
+ high_blast_auto = allow_high_blast_auto()
+
+ blast = ACTION_BLAST_RADIUS.get(request.action_type, BlastRadius.HIGH)
+ rb = rollback_capability(request.action_type)
+
+ def _decision(mode: AutonomyMode, reason: str) -> AutonomyDecision:
+ requires_verification = mode is AutonomyMode.AUTO and _BLAST_ORDER[blast] >= _BLAST_ORDER[BlastRadius.MEDIUM]
+ return AutonomyDecision(
+ mode=mode,
+ blast_radius=blast,
+ tier=tier,
+ rollback=rb,
+ requires_verification=requires_verification,
+ reason=reason,
+ )
+
+ # L0 is observe-only: nothing executes, not even a dry-run preview.
+ if tier == MaturityTier.L0_OBSERVE:
+ return _decision(AutonomyMode.BLOCKED, "tier L0 (observe-only) blocks all execution")
+
+ # CRITICAL blast never auto-executes at any tier.
+ if blast == BlastRadius.CRITICAL:
+ return _decision(AutonomyMode.QUEUED_APPROVAL, "CRITICAL blast radius always requires human approval")
+
+ allowed = _AUTO_ALLOWED_AT_TIER.get(tier, set())
+ if blast in allowed:
+ # HIGH is the sensitive top of the ladder: L4 + whitelist + break-glass.
+ if blast == BlastRadius.HIGH:
+ if tier == MaturityTier.L4_AUTOMATE and whitelisted and high_blast_auto:
+ return _decision(AutonomyMode.AUTO, "HIGH auto-executed: L4 + whitelisted + break-glass enabled")
+ reasons = []
+ if tier != MaturityTier.L4_AUTOMATE:
+ reasons.append("not L4")
+ if not whitelisted:
+ reasons.append("not whitelisted")
+ if not high_blast_auto:
+ reasons.append("AISOC_ALLOW_HIGH_BLAST_AUTO off")
+ return _decision(AutonomyMode.QUEUED_APPROVAL, f"HIGH blast requires approval ({', '.join(reasons)})")
+ # Approval-required action types always gate on a human, even if the
+ # tier would otherwise permit the blast radius.
+ if request.action_type in APPROVAL_REQUIRED_ACTIONS:
+ return _decision(AutonomyMode.QUEUED_APPROVAL, "action type always requires human approval")
+ return _decision(AutonomyMode.AUTO, f"{blast.value} permitted to auto-execute at tier {tier.name}")
+
+ # Not permitted to auto-execute at this tier → SAFE DEFAULT is a dry-run
+ # preview (never a silent real execution), unless dry-run is disabled, in
+ # which case it queues for approval.
+ if dry_run_default:
+ return _decision(AutonomyMode.DRY_RUN, f"{blast.value} above tier {tier.name} allowance → dry-run preview (safe default)")
+ return _decision(AutonomyMode.QUEUED_APPROVAL, f"{blast.value} above tier {tier.name} allowance → queued for approval")
+
+
+@dataclass
+class AutonomyScorecard:
+ """Aggregates autonomy decisions + outcomes so the safety posture is
+ observable: how many executed unattended, how many of those were verified,
+ and how many touched irreversible actions."""
+
+ total: int = 0
+ by_mode: dict[str, int] = field(default_factory=dict)
+ executed: int = 0
+ executed_verified: int = 0
+ executed_unverified: int = 0
+ irreversible_executed: int = 0
+
+ def record(self, decision: AutonomyDecision, *, verification: VerificationOutcome | None = None) -> None:
+ self.total += 1
+ self.by_mode[decision.mode.value] = self.by_mode.get(decision.mode.value, 0) + 1
+ if decision.will_execute_for_real:
+ self.executed += 1
+ if verification is VerificationOutcome.VERIFIED:
+ self.executed_verified += 1
+ else:
+ # Never verified (or verification failed) → count as a gap.
+ self.executed_unverified += 1
+ if decision.rollback is RollbackCapability.UNSUPPORTED:
+ self.irreversible_executed += 1
+
+ def summary(self) -> dict[str, object]:
+ verified_rate = (self.executed_verified / self.executed) if self.executed else 1.0
+ return {
+ "total": self.total,
+ "by_mode": dict(self.by_mode),
+ "executed": self.executed,
+ "executed_verified": self.executed_verified,
+ "executed_unverified": self.executed_unverified,
+ "irreversible_executed": self.irreversible_executed,
+ "verified_rate": round(verified_rate, 4),
+ }
diff --git a/services/actions/tests/test_autonomy_safety.py b/services/actions/tests/test_autonomy_safety.py
new file mode 100644
index 000000000..f7343ed68
--- /dev/null
+++ b/services/actions/tests/test_autonomy_safety.py
@@ -0,0 +1,157 @@
+"""Phase 9 — autonomy-safety policy + rollback-capability contract + scorecard.
+
+Proves the three holes the reality audit flagged are closed at the policy level:
+dry-run is the SAFE DEFAULT (never a silent real execution), the rollback claim
+is honest and bounded (no silent `return True`), and every unattended
+containment mandates post-action verification (with the gap counted, not hidden).
+"""
+
+from __future__ import annotations
+
+from uuid import uuid4
+
+from app.models.action import ActionRequest, ActionType, BlastRadius
+from app.services.autonomy_safety import (
+ REVERSIBLE_ACTIONS,
+ AutonomyDecision,
+ AutonomyMode,
+ AutonomyScorecard,
+ RollbackCapability,
+ VerificationOutcome,
+ decide,
+ rollback_capability,
+)
+from app.services.maturity import MaturityTier
+
+
+def _req(action_type: ActionType) -> ActionRequest:
+ return ActionRequest(incident_id=uuid4(), tenant_id=uuid4(), action_type=action_type, target="host-1")
+
+
+# ── Dry-run is the safe default ──────────────────────────────────────────────
+
+
+def test_over_tier_blast_defaults_to_dry_run_not_execution():
+ # BLOCK_IP is MEDIUM; L2 permits only MINIMAL/LOW → must NOT auto-execute.
+ d = decide(_req(ActionType.BLOCK_IP), tier=MaturityTier.L2_CONTAIN, dry_run_default=True)
+ assert d.mode is AutonomyMode.DRY_RUN
+ assert d.will_execute_for_real is False
+
+
+def test_dry_run_disabled_queues_for_approval_instead():
+ d = decide(_req(ActionType.BLOCK_IP), tier=MaturityTier.L2_CONTAIN, dry_run_default=False)
+ assert d.mode is AutonomyMode.QUEUED_APPROVAL
+
+
+def test_l0_blocks_everything():
+ d = decide(_req(ActionType.NOTIFY_SLACK), tier=MaturityTier.L0_OBSERVE)
+ assert d.mode is AutonomyMode.BLOCKED
+
+
+# ── Tiered auto-execution ────────────────────────────────────────────────────
+
+
+def test_minimal_auto_executes_at_l1_without_verification():
+ d = decide(_req(ActionType.NOTIFY_SLACK), tier=MaturityTier.L1_NOTIFY)
+ assert d.mode is AutonomyMode.AUTO
+ assert d.requires_verification is False # MINIMAL < MEDIUM
+
+
+def test_medium_auto_executes_at_l3_and_requires_verification():
+ d = decide(_req(ActionType.BLOCK_IP), tier=MaturityTier.L3_REMEDIATE)
+ assert d.mode is AutonomyMode.AUTO
+ assert d.requires_verification is True # any executed containment must be verified
+
+
+def test_medium_approval_required_action_queues_even_when_tier_permits():
+ # RESET_PASSWORD is MEDIUM + approval-required.
+ d = decide(_req(ActionType.RESET_PASSWORD), tier=MaturityTier.L3_REMEDIATE)
+ assert d.mode is AutonomyMode.QUEUED_APPROVAL
+
+
+# ── HIGH blast: L4 + whitelist + break-glass ─────────────────────────────────
+
+
+def test_high_queues_without_break_glass():
+ d = decide(_req(ActionType.ISOLATE_HOST), tier=MaturityTier.L4_AUTOMATE, whitelisted=True, high_blast_auto=False)
+ assert d.mode is AutonomyMode.QUEUED_APPROVAL
+ assert "AISOC_ALLOW_HIGH_BLAST_AUTO off" in d.reason
+
+
+def test_high_queues_without_whitelist():
+ d = decide(_req(ActionType.ISOLATE_HOST), tier=MaturityTier.L4_AUTOMATE, whitelisted=False, high_blast_auto=True)
+ assert d.mode is AutonomyMode.QUEUED_APPROVAL
+ assert "not whitelisted" in d.reason
+
+
+def test_high_auto_only_at_l4_whitelisted_with_break_glass():
+ d = decide(_req(ActionType.ISOLATE_HOST), tier=MaturityTier.L4_AUTOMATE, whitelisted=True, high_blast_auto=True)
+ assert d.mode is AutonomyMode.AUTO
+ assert d.requires_verification is True
+
+
+def test_high_never_auto_below_l4():
+ d = decide(_req(ActionType.ISOLATE_HOST), tier=MaturityTier.L3_REMEDIATE, whitelisted=True, high_blast_auto=True)
+ assert d.mode is not AutonomyMode.AUTO
+
+
+def test_critical_never_auto_executes(monkeypatch):
+ from app.models import action as action_model
+
+ monkeypatch.setitem(action_model.ACTION_BLAST_RADIUS, ActionType.BLOCK_IP, BlastRadius.CRITICAL)
+ d = decide(_req(ActionType.BLOCK_IP), tier=MaturityTier.L4_AUTOMATE, whitelisted=True, high_blast_auto=True)
+ assert d.mode is AutonomyMode.QUEUED_APPROVAL
+ assert "CRITICAL" in d.reason
+
+
+# ── Rollback-capability contract (honest, bounded, pinned) ───────────────────
+
+
+def test_only_block_ip_is_reversible_today():
+ assert rollback_capability(ActionType.BLOCK_IP) is RollbackCapability.REVERSIBLE
+ for at in (ActionType.ISOLATE_HOST, ActionType.DISABLE_USER, ActionType.RESET_PASSWORD, ActionType.BLOCK_DOMAIN):
+ assert rollback_capability(at) is RollbackCapability.UNSUPPORTED
+
+
+def test_reversible_set_is_pinned():
+ # Widening this set requires a conscious edit AND a real reverse impl — it
+ # can't grow silently and re-introduce the "rollback returns True" lie.
+ assert REVERSIBLE_ACTIONS == frozenset({ActionType.BLOCK_IP})
+
+
+def test_decision_carries_rollback_capability():
+ d = decide(_req(ActionType.ISOLATE_HOST), tier=MaturityTier.L2_CONTAIN)
+ assert d.rollback is RollbackCapability.UNSUPPORTED
+
+
+# ── Scorecard makes the safety posture observable ────────────────────────────
+
+
+def test_scorecard_counts_unverified_executions_as_a_gap():
+ card = AutonomyScorecard()
+ auto = decide(_req(ActionType.BLOCK_IP), tier=MaturityTier.L3_REMEDIATE)
+ dry = decide(_req(ActionType.ISOLATE_HOST), tier=MaturityTier.L2_CONTAIN) # DRY_RUN
+ card.record(auto) # executed, no verification supplied → gap
+ card.record(dry) # not executed
+ card.record(auto, verification=VerificationOutcome.VERIFIED) # executed + verified
+ s = card.summary()
+ assert s["total"] == 3
+ assert s["executed"] == 2
+ assert s["executed_unverified"] == 1
+ assert s["executed_verified"] == 1
+ assert s["verified_rate"] == 0.5
+
+
+def test_scorecard_tracks_irreversible_executions():
+ card = AutonomyScorecard()
+ # An AUTO execution of an UNSUPPORTED-rollback action is the risky case.
+ d = AutonomyDecision(
+ mode=AutonomyMode.AUTO,
+ blast_radius=BlastRadius.MEDIUM,
+ tier=MaturityTier.L4_AUTOMATE,
+ rollback=RollbackCapability.UNSUPPORTED,
+ requires_verification=True,
+ reason="test",
+ )
+ card.record(d)
+ assert card.summary()["irreversible_executed"] == 1